The beginning of the problem is fairly simple: A system of two bodies will orbit around their mutual barycenter, which is a point defined by their respective masses. Often that point is inside the larger body, but sometimes it’s in space (i.e. Pluto and Charon, or binary stars).
Let’s add some more bodies to the equation, and my question: Why don’t man-made Earth satellites orbit the Earth-Moon barycenter too? Isn’t the E-M barycenter the functional Center of Mass of the entire Earth system? Why don’t we see our satellites orbiting way out of center with the Earth since the E-M barycenter is well away from the very middle of the Earth?
I suspect if we launched a satellite out to an orbit of a million miles or so, that satellite may orbit the Earth-Moon system barycenter and not the Earth’s physical center. But then the question is how far away until that takes place? Is there a transition?
Because satellites are (typically) much closer to the earth than to the Moon, and therefore the gravity from the Earth dominates. If you put a satellite close to the Moon, it would orbit the Moon (e.g. Apollo Command Modules).
That makes sense, but “much closer” is a pretty loose term in astrodynamics. Where does a satellite transition from orbiting one body in a multi-body system to orbiting the barycenter of that system? Is it a smooth transition or abrupt?
A related thought exercise might be 'could a theoretical satellite orbit the Pluto-Charon barycenter, even if unstable?
The scenario you have concocted (a body orbiting two other bodies without one being dominant) can be approximated as the restricted or Euler’s three-body solution. This solution assumes that the two central bodies are stationary, which of course isn’t true but us usually adequate for making state estimates. In reality, orbital trajectories in three-body (and more generically n-body) systems are generally chaotic but can be stable as long as the specific orbital energy of the orbiting body to both central bodies isn’t wildly different. In the case of a spacecraft orbiting the Moon, which is orbiting the Earth, which is orbiting the Sun, the spacecraft can be taken as being within the Moon’s dominant sphere of influence (SOI) and the orbit can be estimated as just purely around the Moon as a first approximation, with perturbations from the pull of the Earth and Sun being calculated as slight but regular and periodic adjustments.
The only configuration in which more than two objects can orbit a common barycenter is a Klemperer rosette (three or more bodies of identical mass and oriented on the vertices of a polygon which inscribes the orbit); such configurations are inherently unstable and do not exist in nature.
A satellite near Earth will orbit the Earth. I think a satellite in a much higher orbit (comparable to the orbit of the Moon) would either orbit the Earth, or stay in one of the Lagrange points, or go into a figure-8 orbit around both, or be in orbit around the Moon.
It’s a smooth transition; it’s basically a question of when a perturbation is so darn small, you can ignore it completely. The inverse-square-law means that the drop-off is very significant with distance.
For instance, in low-earth-orbit, terrestrial mass concentrations – mountain ranges and so on – can materially affect the stability of an orbit. But for very high-level orbits, such as geostrationary orbits, mountain ranges don’t really matter much, and even the oblateness (what is the correct word? Oblacity?) of the earth’s shape is of little consequence. Communication and weather satellites also ignore the presence of Mars and Jupiter.
Oblateness is only of little consequence for circular equatorial orbits. Those are admittedly the majority of geosynchronous orbits, but there are sometimes good reasons to put a satellite into an orbit that’s geosynchronous but not geostationary, and for any such orbit, the oblateness of the Earth will be significant over a timescale of months or so.
The concept is made a little less loose when you define the concept of the Hill sphere of a massive body. Basically, there’s a regime where a satellite is close enough to one of the massive bodies to be viewed as orbiting it; and there’s a regime where a satellite is far enough away from all of the bodies to be viewed as orbiting the common center of mass of the massive bodies. But in between, there’s just no way to orbit the two bodies in a stable way; you can get chaotic orbits and/or objects being ejected from the system and/or crashing together.