I am just shy of 300 pounds and built like any large powerlifter. No one bugs me ... except for drunk folks. As someone says, drunk people fear nothing. Even small women.
But let's be real. Even if a skilled smaller man can beat an unskilled larger man, the smaller guy will rarely pick a fight with the larger dude, or unless absolutely necessary, because it's human nature to not desire to tussle with an animal larger and more powerful than yourself.
Think about it. Would a UFC fighter weighing under 225 pounds really want to pursue an altercation with a 325-pound strongman-type? In a real world brawl, even with the skill and cardio advantage, the smaller opponent would seriously risk injury in such a scenario. Even if he "won" the fray.