I think the ambivalence lies with what a bodybuilder has to do to prepare for a show. Their training regime is as demanding as any sport where strength and physical conditioning is required. In fact, I think more so because it is so all consuming. Truly a 24/7 commitment. What they put their bodies through is far more than what most sports require. Because it is so physical and such a high level of just pure exertion is required not to mentioned their extreme strength that few, save power/olympic lifters, can match. It is natural to think of them as athletes. But the actual event itself, the contest, is judge solely on appearence which has nothing to do with athletic perfromance. It's more of a pagent than a sport.
There's an inherent incongruity with bodybuilding. There's the training which is widely popular and the best overall way for a person to get into shape and appeals and is even respected among the masses. And then there is the bodybuilding contest itself which few are interested in and are generally replulsed by. It's an obscure subculture bordering on perverse and anyone interested in it are regarded as suspect. That's why whenever I'm perusing a bodybuilding mag I always make sure it's hidden inside another magazine like Granny Porn, Petite and Anal, Bukake Slutfest.... That way people don't think I'm weird or abnormal.