Casual clothes are just fine cheap. You're dealing with stretchy fabrics that conform. Anything with structure (jeans, dress shirts, other dress clothes) are things that will look bad if you don't get quality.
Anybody with more extreme bodies (bbers or women with a big difference between chest/waist, chest/hips/waist, hips/waist) is going to have to pay attention to fit or get stuff tailored.
But that's ok, go ahead and spend hours in the gym to get that V, and ruin it by wearing a shirt that gives you a gut