Jason Ferrugia recommends taking something like 4-6 weeks off each year. One of the biggest guys at my gym back when I was in college would take something really long, like two months or even more off in the Summer and, not saying he didn't lose anything, but he bounced RIGHT back.
Seems like a lot of older guys who aren't obsessive about their workouts either take long breaks themselves (to little ill effect), or recommend it.