Offensive Line.
Without good blocking the Qb doesn't get the ball to the skill player, who in turn doesn't get you the necessary yards much less the score.
Cornerback is a close second. One on one, on an island. On obvious passing downs, you have to mimic the movements of the receiver. Give him too much space, and he picks up chunks of yards at a time. Jam the receiver and he gets by you, it's 6 points. The advantage goes to the receiver every time. He knows where he's going, the cornerback doesn't. Easily the most athletic position on the field. Backpeddling, turning and opening your hips to match the receivers stride.(who in turn can make a sudden cut or stop altogether) Very tough.