Honestly, I think I would.
Down just doesn't look like a man who would beat you to shit.
More like the guy who barks back, gets to your face, might even push your shoulder but that's it...wouldn't engage in real all-out.
I assume everyone knows the type