Just remember that people are followers. While everyone likes to think that they think for themselves and have their own beliefs, it just isn't true. Many are not "racist", because its not accepted now, but go to the 1950's and the N word would be dropped constantly is most households, by everyone from housewives to teenagers Many anti racist liberals who denounce the dreaded N word, will go to some Thai or viet vacation and readily ride in a rickshaw pulled by a human beast of burden.
Empiracal studies have been done where people will conform to whatever the group thinks. Look at the Stanley Milgram studies that tried to explain how so many in Nazi Germany kept torturing and doing unspeakable acts to prisoners "under orders". The Milgram study made people believe that they were giving an electric shock to test volunteers when they got a wrong answer to a test question. The more wrong answers, greater the shock. The fake test always had the actor get the answer wrong, and in the group control setting, nearly 70% kept giving increasing shocks, even when the actor pretended to scream in pain. About 5% were willing to go to death. Because it was accepted by the group.
So in turn of the century New York, expecting a black servant to shine your shoes was WIDELY accepted and expected. It was not right, I'm not justifying it, but be careful in saying how you would be the lone renegade with your own beliefs if you grew up in that culture.
It was a different time. Leave it at that. It doesn't mean Sandow was a miserable person. He was misguided like most of society usually is.