You might want to actually try this. You will certainly encounter a diverse array of beliefs there, and nobody will push their beliefs on you. As atheists, you will be part of one of the more cohesive constituencies there. Additionally, your daughter may actually make some friends. My daughter positively loves it, but she’s much younger.
It’s tough to commit your Sunday mornings though, particularly when you don’t have God breathing down your neck to go.