Genetics play a huge role when it comes to aging esp. among females.
Losing sex drive as a dude seems unlikely unless you're entering oldest fart territory but suboptimal circumstances can surely hinder optimal drilling - mental/physical deterioration, saggy females, social playground, activities and spare time.
Imagine being 50 and seeking hot chicks around 40 - 45... this could be quite the hard endeavour depending on the country and culture.