The reason why most people think "steroids" when they see someone in really good shape or is pretty big, is because usually they are right.
Too many steroid douchebags in lifting, plain and simple. If I had a dollar for every "hardcore" "bodybuilder" I see every day, I'd have $36. It's sad man. So few build any kind of physique without drugs. Must be a bad feeling knowing you can't achieve much on your own. Of course, there are a few who get somewhere, then take gear. Chances are, you're not this guy.
These are the clowns that give bodybuilding a bad reputation, not McGwire and the baseball beasts.