I don’t know how it works in churches with turnstiles, but the ticket booth attendants in Italian churches (usually large ones with famous artworks and monuments, e.g. Santa Croce in Florence) seem to rely on subjective judgment. Tourists are usually distinguishable by their style of dress, backpacks, bottles of mineral water, a generally “lost” look on their face, etc.–signs that indicate the person is going around sight-seeing. If you look like a tourist in any way, they’ll probably stop you at the entrance. If you don’t fit that description, you might get waved in, though to increase your odds you’d need to mutter something about “preghiere” and maybe make the sign of the cross.
But what if you use the “I’ll pretend to be a worshipper” ruse, and once you’re past the entrance, you resume your sightseeing intentions? This would be pretty obvious to the attendants–someone who’s visiting for prayer would walk straight to a particular chapel reserved for prayer, and wouldn’t walk around gazing at stuff. They typically have some kind of surveillance system set up around the church, and they’d probably see your non-worshipping activities on a CCTV monitor. In such a scenario, the attendant would probably ask you to pay the entrance/“tourist visit” fee, leave, or go to the chapel for prayer. Of course, this all would depend on various factors–how busy it is, the attendant’s mood, etc.
And then there’s the case of a Catholic tourist–can’t you go sightseeing AND stop for a while to pray? Of course you can, but if you’re doing any sightseeing at all, they’ll expect you to pay the entrance fee, if one is requested. In places like St. Peter’s (which doesn’t charge an entrance fee), the chapel that is reserved for prayer will usually be guarded (as flodnak points out), and security will prevent you from entering the chapel unless you’re going there specifically for prayer and not for sightseeing. What I said above for churches with ticket booth entrances applies to these scenarios. Speaking from personal experience, I’ve gained entry to such chapels by informing the guard that I’m Catholic, and nodding in agreement when the guard reminds me that no sightseeing is allowed, and I’ve behaved properly–walking straight to a pew, genuflecting, entering the pew, kneeling and looking straight ahead towards the altar (upon which the Blessed Sacrament is usually displayed in a monstrance). Even if you’re not Catholic (and aren’t into all that genuflection), you would probably be allowed in the chapel so long as you sit quietly and don’t look around gawking.
One last thing I’ll say about those churches that charge admittance–at first, it annoyed me, but I’ve come to appreciate it. Such churches really rely on the funds for restoration projects and general maintenance, and so I tend to regard it as a donation. Also, in Italy at least, most churches close down during the afternoon (anywhere from noon until 4:00pm)–UNLESS they charge admittance, in which case they can keep their doors open all day long. If you’ve ever had to plan your day of sightseeing around the opening hours of various churches (rather than their locations around town), then you can see how the entrance fees are more of a convenience than a hindrance.