If you don't play basketball, but enjoy watching it, someone could say that you enjoy watching sweaty men run around. Then they could call you gay.
During the season basketball is on television almost every night and if you live in a city where there is a professional or college team then you can actually go to see live basketball quite a lot.
How many times can you see a bodybuilding show on television or live and in person?
Basketball is an athletic contest that can be debated and cheered on with friends, bodybuilding is an subjectively judged contest based on who has more muscle, symmetry used to be a criteria but isn't anymore.
How can you debate and cheer on with your friends anything that has to do with bodybuilding? How many chances do you have to see it live and in person?
It is easy to be a fan of basketball and not play it yourself but to be a fan of bodybuilding and not lift weights is quite different. Often fans of bodybuilding that don't actually lift have different reasons for looking at nearly naked men.