I suffer from that same dilemna all the time. Getting ripped sure does hurt and being bigger sure does feel as if you get more respect out in the real world. Me, I like the ladies so my motto has always been, "It doesn't matter what you look like in clothes, get down to your underwear in front of a hot stripper and if she isn't impressed, get back to the gym and dieting."
I will tell you that the pros suffer a great deal when they are big and fat. They hate that look and always talk about how many weeks they are out from starting their diet. The truth is, most pro bodybuilders that I know are happier when they are dieting. I think part of it has to do with all the time they spend eating and doing cardio which takes their mind off all the other things you would expect someone to care or worry about.