It's simple. IF time travel can exist, it can only exist where the subject can go to the future. IF the subject could travel to the past, he/she would invariably alter events, effectively changing the future. Enough of this happening and our entire lives would be being revised on a continual basis.
Since I'm positive that my memories remain intact and constant, and my decisions today have consequences, I can plainly see that time travel to the past is impossible.