The problem isn't that someone's "feelings" may get hurt. It's that you'd wish he would know better than to say certain things which have absolutely no advantage (and every disadvantage) in saying.
But then again, if he's in this for some other reason, you could figure he "knew better" than not say it. That's the problem. Because no doubt, he did say it.