Exatcly. I used to say that I'm at the gym five days a week, but now I just mumble "Well.. quite a lot.. but you know - that's the drill of this game..", unless it's a fitness psycho chick that digs big muscles ("Oh... I like to watch how men lift heavy weights.." , literally heard this from one).
Another good one is - "I was born this way" and quickly change topic.
OTOH, it's kinda liberating to just let it fly with some folks. I was consulting with a small software company down in Boston this past summer and the VP Sales, who was a smoking hot, younger, smart, successful woman asked me, "I need to know how you stay so fit". I looked at her and said, "Mondays." She said, "what about Mondays?"
"On Mondays I have to force myself to take a day off"
She got the message and the determination. And she wanted the cock.