Bodybuilders are not popular for a number of reasons.
Bodybuilders are not popular because for most people, big muscles and veins all over the body is a repulsive picture.
Bodybuilders are not popular because they spend too much time trying to look good. That sends the message that they are shallow or more shallow than the average person, someone you cannot form solid friendship or eventual love relationship with. Also, most perceive bodybuilders as insecure because of this obsession with muscularity.
Bodybuilders are not popular because for most people bodybuilding is more related to striptease than it is to sports. In fact, one just have to watch one single clip of bodybuilders competing to realise how erotically charged bodybuilding is. If anything, people see bodybuilders are sex objects rather than sportsmen. I'm sure most don't complain about it though. Obviously, your average heterosexual man is not interested in watching nearly naked men, if anything they would watch Miss Universe contests.
Bodybuilders are not popular because they, as a collective, are perceived as aggressive, sexists, misogynists, homophobes. Most of the time bodybuilding is related to these negative qualities which make people not find something to admire in bodybuilders. People adimre heros; not villains