Arnold's a special case because of personality and acting, Ferrigno too, because of the acting. I think those two would still be at the top if you asked people to name a bodybuilder.
Powerlifters, and other strongmen have had fame, but it was because of other things like the Olympics, Televised Strongman, and Wrestling.
If you took away Arnold and Lou's acting gigs, Bodybuilder are at about the same level recognition wise more or less.