There will never be a fit all answer to this.
Seems to depend on the individual's digestive system and sensitivites etc, whether you can handle a lot of dairy in your diet (organic/non organic/raw etc).
I've never had a problem digesting dairy, so it doesnt slow me down or anything, but I have pals who drink milk and get stomach cramps so I guess just do what works for you.
Shame if you cant tolerate dairy as milk is a great way to get steady protein throughout the day. I drink about 3-4 pints of full fat milk per day, and just eat less carbs to balance out the extra calories/lactose i'm getting.