I was using public sector synonymously with the government. Your position is still an overstatement. The First Amendment does not require a government official to avoid all forms of religious expression.
One of the things that always puzzles me is how atheists get offended by a reference to something they don't believe exists.
I'll try to explain it to you....
I can't speak for all non believers but I can speak for myself and I get a general feel from others I know that this is the issue;
We would be perfectly happy to let people believe whatever they want. Elves, Leprechauns, Gods, Demons, Ghosts, Angels, Psychics etc etc. The list is probably endless. However we oppose people who do believe in those things forcing those personal beliefs on others. Simple as that. Whether it's in the form of a religious ceremonial prayer before a baseball game, promotion ceremony or council meeting to the other many ways they find to expose non believers to their belief.
The concern is, left unchecked, these people historically have been known to grow in confidence when others don't speak up, and before long they're burning witches, creating crusades and "saving" people who otherwise are just fine as they were. So reluctantly, the non believers have to be the bad guys and say "Hey Billy Bob, not everyone believes like you do, so put the torch down before you hurt someone"