I would still think the idea that the omnipresent polyfather only listens if you pray in a particular building was silly even if I did believe in gods.
If someone invites you to the mall and you don’t like consumerism you probably don’t decline the invitation in a way that makes it seem like you think the other person is irreconcilably wrong with the world for going shopping. That’s the message “No. I’m an atheist.” sends when someone invites you to their church socially.
“I’m an atheist.” is not a naturalistic viewpoint. It is not a statement of belief… except for perhaps the rather rude “I believe you are wrong.”
“What do you believe?”
“I don’t believe in your god.”
“I didn’t ask what you don’t believe.”
My beliefs aren’t dependent on gods not existing, so there’s no reason to mention them if I’m in a discussion about my beliefs. Even if someone else is making an argument based on their religion I can assert my (disagreeing) worldview without needing to attack their belief in gods.
Doing so, especially in response to a polite social invitation, is rude.
Probably has something to do with the inherent offensive negativity of asserting “non-belief” as equivalent to belief. You do have beliefs, right? But rather than sharing those on equal footing with the beliefs of others, you choose to make an issue of non-belief.
If a christian chimed in a discussion of beliefs with “I believe you’re all going to hell.”, you’d think that was rude and feel justified in giving them the cold shoulder, right? That’s basically what “I’m an atheist” is. “I won’t tell you what I do believe beyond that I believe you’re all wrong.”