Exactly. It's kinda like eating candy and other garbage = cable curls, while eating clean is like doing barbell curls. Your biceps will grow from doing the former, but a lot more from the latter, going by that loose analogy.
Honestly, this is statement gets to the heart of the issue. Are barbell curls REALLY better than cable curls? How do you know? Have you observed long term growth with both, and compared the rates? And not just in yourself, but others too?
How about people clean vs dirty diet (keeping calories steady, ie "a calorie is just a calorie")? How many people espousing one method have even tried the other? Sure there are countless bodybuilders who have eaten clean while dieting down, but how many have EVEN TRIED eating dirty while dieting down? I'm sure 99.999999% of them, especially these days, just look at what everyone else is doing and follow suit.
I eat clean because I believe that there are substantial health benefits, and these will eventually have a significant impact on the appearance of your physique as well. But I definitely have my doubts, and wouldn't be surprised if the impact on one's appearance wasn't small enough to justify those in the "calorie is just a calorie" camp.