You'll get called a "Nazi" by your political enemies regardless of how much you attempt to distance yourself from the ideology. It really makes no difference now. Spencer gets called a Nazi all the time. Heck, even flamboyant conservative homosexuals like Milo often get called one.
Besides, calling people "shitskin" is hardly going to go down well in polite company either.
Are you aware of a little thing called "rhetoric"? When they call you a Nazi, they're being rhetorical -- they don't mean that you're a member of the Nazi Party. What they're actually saying is that you're a superlatively disgusting Right-wing person. That you hold immoral views that should not be countenanced or considered.
Furthermore, they not actually talking
to you. They're
addressing the crowd while making it seem like they're talking to you. In effect, they're communicating to their fellow leftists that you're a person to be shunned and avoided.
By responding to this skillful rhetorical trick with "the Nazis really weren't evil, guys!" you make yourself into a useful idiot for the Left. Because (a) they're not buying it, (b) they don't give a shit, and (c) it actually vindicates their rhetorical trick by giving it a veneer of truth.
And this is why Nazi LARPers are doomed to total irrelevance forever. They are boogeymen and useful idiots. If they didn't exist, the ACLU and the ADL would have to create them.
The best response to their rhetorical trick is to foster community and family, be an honorable man, and hew closely to icons that still mean something and don't inspire immediate revulsion. The American flag is a particularly good option. I'm a White Nationalist and a traditionalist, and a real American. Nazi fetishism is abhorrent to me as something both alien and horribly counterproductive.