It's not the act of building a mosque, it's that it planned for ground zero.
They have been offered the ability to build in a neighboring area that isn't considered part of "ground zero," and they refused. At that point, it was no longer that they wanted to build a mosque. They only wanted to build a mosque at ground zero.
No one would make much of a fuss about a group simply wanting to erect a monument to Japanese culture, but people would make a fuss about that group wanting to erect that monument only at Pearl Harbor.
You could say the same thing about a monument to Germanic culture at Auschwitz or Birkenau.
There are also many sites devoted to nuclear technology, even historical monuments to nuclear weapons, but you're not going to find a monument celebrating nuclear weaponry at Hiroshima.
I don't understand how people feel they have to take some kind of absolute stance on things and then try to make every scenario either fit with or against those absolutes instead of looking at each situation individually.