I'm one of those who disagree. Silva's competition has been lackluster. In Jon Jones short time fighting in the UFC, he has faced better competition than Silva has faced almost his entire career in the UFC. What can actually be gleamed from beating the likes of Stephan Bonnar or a Travis Lutter? He is the king of a weak division, full smaller men of mid-level talent. You are free to disagree.
I partly agree with what you said, but I also think that Anderson is not physically big enough to mess with light heavyweights. He is tall enough, but his frame and ability carry that much muscle mass is not that great and would interfere with his ability to perform at his best if he were to try to gain the weight and face off in the light heavies.
I do agree in that the light heavyweight division has way better talent to offer than the middleweight division and that Anderson has not had the opportunity to face superior, all-around talent.
That said, I will say that these opponents (which he defeated) are pretty talented and are part of the reason why I say that he is that good:Chael Sonnen
(The guy is talented and can be a problem for any fighter & is also now a light heavyweight)Vitor Belfort
(He is a light heavyweight fighter and had a solid record)Dan Henderson
(Henderson is a legend and Anderson went through him and is also a light heavyweight)Rich Franklin
(Another light heavyweight that Anderson defeated)
These are the guys that I used to validate my reasoning for seeing him as the best. That, and not to mention that he has solid BJJ & Muay Thai skill and is entertaining to watch.
Ultimately, I do believe firmly that in order to determine who is truly the best, he would have to face Jon Jones to end the debate, but even then it would be very difficult.
Just as Anderson is seen as too superior for the middleweights, the argument can be made that Jon Jones is also like that of a heavyweight fighter fighting in the light heavyweight division due to both his height and frame, which can easily carry way more muscle mass and weight.
It's a tough call brother, I wouldn't be able to give you a perfect or even that good of an answer.
I also sway between thinking that GSP might be the best and then return to thinking that Silva is the best.