Hmmm... that's a hard question. Like you said it depends on the guy himself, a confident guy can pull off wearing jeans & a sweater just as well as a suit. Overall I'm a more casual girl myself... a good pair of broken in jeans on the right butt can be a wonderful thing
however a cleaned up guy in a well tailored suit is nothing to sneeze at. Gee... I guess I don't have a preference
but I will say that I can't stand the whole having my jeans falling off my ass, boxers hanging out, extremely oversized shirt look - bleh!