There are some people who legitimately train for the different crossfit events, which are legitimately challenging.
And, in some ways, it's more of a "sport" than pure bodybuilding. At least there's a "performance contest" component to it.
It's just hard to legitimize a sport in which you see the athletes using 10lb plates that are purposely designed to look like 45's. You can't help but draw certain conclusions about that. In some ways it's a bit like pro-wrestling: most of those guys are in tremendous shape, it's hard to do, it carries a certain danger, and it's fun to watch, but... in the end, it's still just make-believe fighting.
My point is that Crossfit (proper) does require skill, strength, stamina, and coordination. But, there are certain aspects that will lead people to dismiss it from the ranks of its physical counterparts.