Ok, I'm still misunderstanding, last try: I'm ignoring the gun in my holster for said confrontation. Done. Even then, you're saying, I still cannot react. Anything less than a lethal situation - even though I have no intention of reaching for, using, or even acknowledging the gun's existence - I'm forced to tuck tail?
Dude pushes me, slaps my chick, throws drink, spits - but no intention of killing - I suck it up and split, right. Gun on my hip means I can't throw a punch. Damn. Thank God I could never get a CCW.
Apologize, I'm slow as fuck, but this interests me.
Doesn't matter. Here's a scenario: you're carrying, but a guy is being a real dick and you think, "I can kick his ass and I will never touch my gun". As soon as you get into it, though, you realize that the guy is better than you thought and you are about to get KTFO. What do you do? Pull your gun? That's illegal escalation to deadly force because you had the opportunity to escape, but allowed the fight to happen. Get KTFO and possibly lose your gun? Not good either. There's a felony there too.
Your only legal option is to avoid or de-escalate.
Now, if you are pearl-harbored and surprise attacked, you can (in some jurisdictions) "brandish" the weapon to get them to stop. However, the situation then becomes lethal, with all the laws attendant. You'd also best be prepared to use it or lose it.
Florida's SYG law is a notable exception as it has provisions for "peaceable movement" or something like that. It basically states that if someone is feloniously keeping you from moving forward in a public way, you can respond with deadly force. It was specific language added for those that didn't feel that the law abiding should be forced to retreat from a felony. To my knowledge, it hasn't been tested in court.
In general, American gun owners and the folks that choose to legally carry are the most sober and judicious citizens that our country has. To a man, they usually have more firearms training time than the average cop, and are certainly more level-headed.