No, the standing of the Olympia then isn't the reason, it's because he was the best-built guy on the planet for a long while irrespective of contest.
No, historians will not say Pearl hands down because the evidence shows quite the contrary, despite your preconceptions. Obviously a big Pearl fan here, maybe the only one. 
Hmmm....at last count, Oliva won three Olympias (one unconstested); whereas. Pearl won four NABBA Mr. Universe titles (at the time when that contest, not the Olympia, was the premiere show). Also keep in mind that the Olympia wasn't even around until 1965. So, either way you slice it, Pearl won the top title in bodybuilding FOUR TIMES.
And, lest we forget, his last Universe came at the expense of several IFBB bodybuilders....INCLUDIN
G 3-time Mr. O, Sergio Oliva.