No, they wouldn’t be able to handle it, but that isn’t my main concern. Last night, I asked CaveMan about how he would want to handle kids and church, and he just looked suprised and said that he just assumed he’d go with us until they were old enough to understand. 
I’ll think we’ll be just fine, oh yeah.
But you still haven’t addressed the question of what will happen if CaveGuy stops attending.
Also, this sounds like a relatively new and budding relationship and though you are clearly intelligent, I get the feeling still quite young and naive about things. Or maybe I’m just getting old and cynical. Nevertheless, I think it’s fair that you don’t underestimate the influence extended families have on a relationship.
Good luck to you. Really. 
Just keep taking your time as you are now and looking inside yourselves and talking openly as you are now. It’s your best strategy for having things work out in the long run.
I commend you both for thinking this through so clearly and recognizing this for the major issue that it can be if you want a long term relationship. I just had some thoughts on your having a happy church home for yourself.
I was raised as a Missouri Synod Lutheran (nearly the most conservative kind), then as a college student became involved with the southern California born again types. Now I’m probably more in Caveman’s camp.
As much as many of us may not like what Bibleman had to say, he’s done the discussion here a service. As Susan has noted, his attitude illustrates the response that is all too common to a relationship like hers.
I think it’s a fine thing that C wants to go to church with S. Also, to a point, the idea that’s been expressed in this thread that it’s no one’s business what C believes has merit. But where it gets fuzzy is that a church congregation is to a degree founded on the idea that it’s a group of people sharing common beliefs. How formally that’s defined varies from denomination to denomination and from congregation to congregation, and of course is in tension with the goal of bringing in new members.
What I’m trying to get at is that Susan leads me to believe that Caveman has enough integrity that he will not want to say that he believes something that he does not. Southern Baptist congregations have some degree of variability, but a large portion of them can make it very uncomfortable for a known unbeliever to remain within their midst – at the very least they will be subject to constant proselytizing. And if “yoked to an unbeliever” – Bibleman has given us a preview of the thoughts that will at least be lurking in some minds.
My experience with religious groups like this is that it becomes difficult to avoid situations where saying nothing doesn’t make it obvious that you’re not in agreement with the beliefs of the rest of the group. Failing that, being present and participating for a prolonged period without saying anything may lead people to lead people to believe you believe as the group does.
I certainly don’t recommend throwing it in the face of your new church home that your boyfriend is an atheist. But I think it may make for a bad relationship with the congregation to let them assume that he’s not only to find out much later that he is.
Well, here’s my experience with this:
I grew up going to church, usually Methodist. My mother took me and my siblings while Dad stayed home. At first she was doing it out of a sense of obligation as a responsible mother (she was raised in a church and thought it valuable, though her own beliefs were still under revision). Meanwhile Dad, who had been unchurched his whole life, didn’t see the point, and stayed home listening to music Sunday mornings while the rest of us dressed up and went out. I don’t remember any conflict about this (not in front of us children, anyway), and I don’t remember being particularly troubled by the discrepancy in my respective parents’ behavior. As with most aspects of childhood, it simply was what it was.
With time my mother became more devout, and eventually my father did too and began going to church as well. Suddenly (I was about 13) we were going to church as a family, and my parents were going because of belief, not convention. Interesting but not particularly earthshaking.
My point is, it’s not so unusual for people’s churchgoing patterns to change over time, and it’s not so unusual for children to witness an evolution in their parents’ beliefs. It is perfectly normal. In my opinion, anxiety about religious differences isn’t worth endangering a relationship over. Of course, Methodists aren’t Southern Baptists (yet). YMMV.