Funny, a girl just asked me a few weeks ago if are there any men left over 30 that doesn't have a pot belly. She's right, too.
And, yes, it does matter.
As you get older you do see most of your male counterparts getting way out of shape unless they have an exercise habit. I have bicycled and lifted weights since I was 14 and never stopped, simply because I love it. At worst i got 20 pounds overweight at times - mostly due to work commitments where I literally had zero time to exercise the way i wanted. But those periods were brief and I always got back into it quickly.
The thing with women is they really don't know what they fuck they want. But having money is probably the most important factor. I have seen time and time again hot women with out of shape fat guys. The eye candy guys who don't have money get treated like disposable items, just like a slutty woman would be.
I work out and stay fit for me. I don't care if anyone finds it attractive or not attractive. At the end of the day you need to find your own inner peace. Mine comes from feeling fit.