Rule #1 of History:
Land belongs to those who can keep it.
This applies to all civilizations since the beginning of history. If your tribe/race/city/nation cannot defend and keep the land it resides on, it loses that land.
It's not a Parker Brothers game. It's not "I was here first so it's mine." That logic doesn't apply in the real world. Sorry.