Hard to say really, but its more than likely The drug "Gurus" and the people who support it. The Chad Nichols, Palumbos, Rambods, Milos Sarcevs etc.
They create the underlying drug culture by pushing the boundaries. What springs up then is the seediness of obtaining drugs,selling, the entire sub-culture.
Last but not least, the health problems and the judges decisions. The judges obviously bear responsibility as well.
The Single most "THING" that is detrimental to bodybuilding are drugs.