I think it has more to do with who it is, rather than how big he really is. What I mean is, it's Brad Pitt, he could be fat and they'd love him. Women love celebrity worship (men do it as well), so if someone is famous, they're instantly hot. Shit, Mick Jagger gets laid for crying out loud.
Certain type of look is marketed, so women (mostly bimbos) buy into it.