Swimming is excellent. To another degree, ocean swimming will require more effort because of the extra resistance from the movement of the water/waves/wind, etc. So will place more demands on cardio. You really never get a complete rest, unless floating. This will also hold true for river swimming.
I have been told that the body will install/need a thin layer of fat as a thermo barrier against colder waters, as had been suggested. That does not mean you are fat, just have that light protective insulation of fatty tissue around you. Surfers seem to be more muscular than pool or swim event swimmers, from what I've seen.. Probably that extra work of swimming out to wait for the next set. And surfing it's self is a complete body workout. Good Luck.