1) It is completely gay. Don't get me wrong, I don't think calling something "gay" is an intelligent way to describe it, but it's the best I can do with this one. I mean, guys stressing out over their bodies, tanning, wearing speedos: this is all completely "gay"! Throw in the competition part of the sport, the prancing around like a fairy and having old people look at you and it is officially 30X more gay than even men's gymnastics.
2) Steroids. To compete in baseball, some guys do steroids. To compete in bodybuilding, all of the guys have to do steroids. Not exactly family friendly. No parent is going to encourage their kid to join a sport that requires injecting synth into your tricep to make it bigger.
3) Lack of talent. You aren't getting the best of the best. You're getting the best of the people who couldn't do something better. Plenty of pro athletes and pro wrestlers have physiques that could easily compete with pro bodybuilders, but they have something better to do.
Don't get me wrong, I love the sport, but I am completely out of my mind. Most people would probably cite the reasons above.