Because people lie about it, a lot.
PED are widespread, every gym I've trained at more than a few times and it becomes obvious who is ready to supply the endless stream of guys looking for a short cut (or are just desperate). I am not talking about dungeons too, I mean health clubs and the like. The drug use is rife.
When you look at guys from before the steroid era then you realize just how hard it actually is to be big and muscular. People are so used to seeing hormonized physiques masquerading as natural that they've actually forgotten - or never knew - what is possible for a natural in terms of muscle mass retention at an extremely low bodyfat.
Does that mean that some gifted naturals will be accused of drugs ? Absolutely. However, usually, if it seems too good to be true, it is.