I can condense this down to one sentiment.
People who don't lift weight only ever think about the pain and time you go through in lifting weight.
That's all they seem to notice.
It doesn't help that people who lift weight, they don't really often evangelise what they are doing. They might complain "oh my legs are sore", but they won't say "Oh, I'm terribly satisfied at my recent efforts at the gym and this has had tremendous trickle-down ramifications for other parts of my life"
It's human nature, we rarely give good reviews. You're already doing the shit, why bother talking about it.
The problem with that is that the people who are left giving the good reviews are often scientists, who generally, as a rule - are presumed to be thin old men in white coats who have never touched a weight. IE, people who don't look like they have walked the walk.