Not really, it's all about context.
Oprah introduced some dude last week as her "favorite rich negro man". he seemed a little pissed about it.