If it's not important to others why do they mention it all the time?
Appearance is important to almost everyone, some of us just have the drive to take it all the way.
Yes, but physical size is only one facet of appearance. Someone who is 185 might be faster, more agile, and a better athlete than someone who is a completely jacked 260. Just being bigger does not equate into superiority.
The big guy feels superior because that's what's important to him. It's like when someone sees a 170 kid lifting in the gym and assumes the kid thinks he's "huge", but in reality, the kid is just in the gym exercising.