I am going to take a guess and say that this difference in size after competing directly relates to how much "supplementation" it took for that person to get to the size he had to begin with. (gear, GH, IGF, whatever)
If someone had to take "alot" to achieve his desired size, once coming off, he would shrink "alot" accordingly.
If someone only took a "little" then once coming off, should only shrink by the same amount.
Basically the natural genetics thing again.
Then again, if permanent damage were done to natural Test production, maybe some do not seek the HRT therapy, and others do. But all else being equal, I would still say its mostly genetics.