I honestly think its more than skin color for most of those people (some not, I agree)...but many non-blacks fear blacks associated in any way with black liberation theology (ie the Rev Wright types) or those who focus on past injustices in a more hostile way. If you have Condelezza or Powell, who are farther removed from these kinds of issues, you'd see less opposition. I'm not saying Obama is the same as Wright, Sharpton, jackson, etc...but looking at his past there is at least some documented influence/connection and I think that scares people considering the presidency is the most important position in the land. Even some of the things he writes in his book about race and his internal struggles, while I think they are insightful personally, we aren't used to hearing those things from presidents.
If he was more moderate and did not have past connections with people like Wright and others on the far left, it would be a hell of a lot easier for him in the general election.