My point is maybe we shouldn't, because when we do, we often make it out to be something its not.
Chanting USA USA USA is not racist. It can however be charged as nationalistic.
you said earlier that if we do worry about what people intend to mean, then we will not be able to talk to each other at all
I said that we do worry about what people intend to mean, and are still able to talk to each other, thereby invalidating your conditional.
Now you are saying we do, but that we shouldn't

. You probably meant "if we worry only about what people intend to mean" in your original conditional. In your original, you left open the possibility of also considering logical structure, but I think you intended to say that if we disregarded logical structure and only focused on intent, then we will not be able to talk to each other. This is allowed by my revised conditional adding the word "only", not by your original statement.
see how we need to focus on intent to even have this discussion? Intent is very important.