“It doesn’t hurt to” is an idiom meaning it can be beneficial. It doesn’t mean you were arguing that it hurts to be fit. I’m saying it’s not necessary to be fit but it might still be worthwhile.