Well, the main reason the smaller dudes get the roles is due to their acting ability / popularity at the time.
If they had a 6'3" actor with better acting skills and more popular than Damon at the time, i'm certain they would use him instead.
Sometimes the physically bigger actor doesn't have the right feel for the role either, but certainly casting is to blame for many roles as well, casting makes big mistakes as well, when there really are a better fit physically that also has the corresponding acting skills and a smaller one gets the role instead.