Vince Taylor is pushing 60. He seems to be fine. Dennis James is almost 50, as is Tony Freeman. They seem to be alive and well. And let's not forget Big Louie. He was at his biggest in his 40s, 315 lbs ripped.
Robby Robinson wasn't exactly a shrimp, when he was Masters Mr. Olympia. He looked awesome at 50.
Vince was big, but he was never huge, huge, and he never really developed the look a lot of the guys get. I'll give you Dennis James, Louie was huge, but he was also a good half a foot taller than the average, he also understood breaks, and got relatively small and lean afterward. Toney sort of is like Louie, he's taller and hasn't been playing the huge game for too long.
It isn't a perfect theory, but it's just what I've seen across all of the strength sports.