It is better to wear whatever you are comfortable in than to worry about how good you look. Women love men who are confident. Confident to wear what they like.
I am most comfortable when I'm just walking around in my underwear.
How you dress can make the difference between you getting that job.
Sweat pants and a T-shirt is probably the most universally comfortable clothes to walk around in public. Yeah, that will scream success and attract the hotties.
You really need to think more before you speak.