Does their "history and legacy" affect you at all? Can you speak for how everyone outside NE who gives a shit about football feels?
Does it affect me? No. Will I make fun of them incessantly about it? Yes.
Unlike you, though, I tend to have something to back up what I say. I've seen numerous polls (ESPN, etc...), with large samplings, that showed demographically, the feelings of fans on deflategate. To a state, every single one outside NE had the majority of people feeling they were guilty. This was before the Wells report. So, yeah, I can say with confidence the Pats are despised on the whole outside of NE. Unless, of course, you can prove otherwise.
If you want to argue semantics though, I'm not interested.