I guess it depends on how you define upper class, if you consider it to only be filthy rich, with owning a mansion worth well into the millions, several exotic cars, private jet, net worth in the tens of millions, etc then yeah you can say "X factor isnt upper class."
The way I look at it upper class is above what the majority of people can afford, so anything above average I consider upper class.
Having lots and lots of money is not the only - in fact, not even the main - criterion for being upper class. Many people have lots of money who you would still deem working class and vice versa.
From what I've seen he has a comfortable, successful middle class existence and is doing very well. There are certainly other factors involved in making a realistic judgement (schooling, family history) etc..
On a materialistic basis I would say that his lifestyle would be in the middle echelons of the middle classes - a great place to be.
However, it's hard to judge..PDiddy has the planes and flash cars but is most certainly not upper class. Maybe in North America the criteria are different.