I've gone through the extremes. At this point, I consume a "muscle building" diet, but it's what I'd call fairly moderate. It's relatively unprocessed, low in fat and high in protein, but it's not a chore to consume and I will err on the side of taste rather than nutritional value.
This is something that took me a while to get to. I kinda felt like a strict diet was all apart of the process.