I did Japanese Jiu Jitsu for almost 2 years and it was a 60/40 split with the majority of what we did being judo.
Try to work with guys who know what they are doing-- I can't tell you how many injuries I saw where a novice was doing drills or randori with an advanced student and ended up injuring both of them by being overly agressive and tangling up feet, knees, arms-- or worse, not taking a throw and holding onto the gi.
The other most important thing you can do to avoid injury is learning how to fall properly. Whether its doing side falls, back falls, shoulder rolls, taking a throw, completing a throw, etc.-- figuring out how to minimize the level of impact when hitting the mat is the best skill you learn in judo next to throwing people. If you can master that, the likelihood of injury is relatively low.
As far as Judo being rough on your body-- it is. It's difficult for it not to be unless you are in a class full of sissys and don't get challeneged. I haven't done it in almost a year and I still have nagging injuries. It's fun, its useful, but its not the safest martial art there is.