Yes you can damage it if you are rough with it. The stuff is very fragile.
I've heard some guys say that when your mixing in the water, you don't want to shoot the water in to where it directly hits the GH pellet. The idea is that the hard pressure is too much, of course by hard pressure we're not talking a fire hose, but relative to the pellet. What some say you should do is to squirt the water in to where it hits the side walls of the vile and trickles down to the pellet, then mix slowly.
Maybe it's all overkill, I don't know, but it's not a big deal to do it that way...why chance it?