I pig out every day and probably have clean fast food 5 days a week.
The secret isn't calorie restriction, it's busting your ass in the gym 5 times a week.
If you train hard enough, eat fairly clean but human and stay below too many total calories while getting enough protein and good carbs, and get your right sleep, you're fine.