The drugs certainly don't help, but they're really not the main issue. What we're talking about here is a "sport" that consists of men shaving their bodies, oiling up, putting on a thong, and dancing around on a stage for an audience of mostly men, in what is for all intents and purposes a male beauty pageant. That is not a sport by most people's definition (sorry earl), or even an activity they really want to participate in or watch.
"Bodybuilding" as an activity (lifting weights and manipulating diet to improve your physique) will always be popular and something large numbers of people will participate in and enjoy, but competitive bodybuilding (the "oiled men in thongs" thing) will always be a fringe subculture.
It has less to do with what you mentioned, and more to do with what people can relate to, and what they've been brought up to participate in....mainstream sports also happen to be what we as kids are exposed to, and what most people can relate to, and it's relativity to participation....
Lets look at the most participated....running, soccer, golf, general weight training....sports almost anyone can do to a degree....the more up the scale in terms of skill/ athleticism/ etc...the less participants, and the more fan oriented...(Football/ baseball/ wrestling/ Tennis/ gymnastics/ boxing/ MMA/ track&Field, etc)...
You then get the nitche sports, which are too tough for most to participate in, thus the interest level pales...bodybuilding/ Ironman style sports/ collegient wrestling/ cliff diving/ extreme sports, etc....
It's the same ratio that the Olympics are televised and covered...as the mainstream sports are given primetime, mid-range level covered, but not primetime...and weightlifting/ luge/ steplechase/ and that funky cross-country-skiing-shooting a gun thing....all on at 3 a.m. and barely covered if at all...kind of like the awards they give "before" the Oscars are shown on T.V.