If you think he would be well cared for and loved (which it sounds like he would be) and you think they would have more time for him, then it is something to think about. You would still be a part of his life and know he was okay.
IMO, Keeping him would only be selfish if you don't think you can meet all his needs.