Boxing was dominated by blacks simply because it's usually a poor urban sport in America, so it attracts the poor and recent immigrants. You see this now, as the sport is becoming more Latin and Eastern European. Once a ethnic group achieves a certain degree of affluence/prosperity, they tend not to like being beat on as a way to earn money, and usually discourage their children from taking it up.
As far as one on one fighting? I doubt any one race has a stranglehold on the title of World's toughest, plenty of tough guys across all races.
