The guy isn't going to face criminal charges, so his first amendment rights aren't being compromised
However, the NBA might feel that having a (semi) openly racist owner may compromise profitability, so they have the right to take necessary actions in order to maintain their profitability.
Same thing with Mozilla, Paula Deen, Dog the Bounty Hunter, etc, etc