What kind of fag is worried about getting his clothes all dirty from a little bit of chalk? Seriously, any real weightlifting or powerlifting gym has guys who lift with chalk (you need it to help grip the bar due to moisture on the hands for lifts where you can't use straps).
The thing that bugs me a bit about chalk though is the guys who get all fired up and then clap their hands before a big lift, producing a huge chalk cloud. I actually sometimes get athsma from enhaling chalk dust particles, so that can affect my training.