Movies using Magic Tricks instead of Special Effects

Even if it’s all one unbroken shot, shortly before panning up to Newman’s face those hands we’ve been watching leave the shot.

Why?

It’d be slightly more impressive if the hands remained in the shot until the pan up — and I say “slightly” because, as you say, it forced you to conclude that it’s Newman, and, well, (a) that’s pretty good, by which I mean (b) it was good enough. But the only reason I can see for those hands to leave the shot is for a practical effect, like you’re kind of on about: they don’t need to stop the camera or anything, they can just have the other guy (wearing a shirt with similar sleeves) stop messing around with the cards and move away right as Newman swaps in.