Yeah!
How is fishing a sport? You throw a line in the water and wait for a fish to bite. And golf? You hit a ball with a club and ride a cart to find it and hit it again...until it finally falls into a hole. Olypmic "sports": Syncronized swimming (people making shapes in the water), Equestrian (a person rides a horse and the horse does tricks), Bobsleigh (4 or 2 people in a closed sled going down an icey track), Luge (1 or 2 person in an open type sled), Skeleton (1 person head first in a sled, sliding down icey track) and Shooting (a gun...now there's a sport...ready, aim, fire. Which brings up Hunting...that's a sport? Hide in a tree and wait for an animal to come along and shoot at it?).
Those are sports because winning in them depends on the
performance of a physical skill.
Fishing requires precise technique; if you just throw the line in and wait for a fish to bite you won't get anywhere in competition (the "sport"). Golf definately requires physical technique as well. You know how horrible we all are when we start. The other sports you named also require considerable levels of physical skill as well, and winning in them is based on competition of the degree of that skill. So, they are "sports."
Winning in bodybuilding, on the other hand, depends about 80% on your build (how you look), only about 20% on the skill (posing). Therefore, it is not a sport and is instead a form of a beauty pageant, the aesthetics of the body being the "beauty" part.