I used to have to pick through my loads at work, in -20* temps in the freezers at work.
I wore a pair of gloves to protect my hands, but usually that's all I wore for protective measures.
Otherwise, I would be in pants and a typical work shirt.
The one thing I found to make sure you didn't get cold, was just moving. It was only when you stopped moving/picking/stocking that the cold got to you.