I have a bit of experience that is loosely related.

The scenario: Humpback whale researchers go to Alaska, stake out a cliff top overlooking one of the bays, and observe humpback whales and boats in the bay. The object of the research is to investigate whale-boat interactions. In particular, they want to know if the boats are scaring the whales away or otherwise disturbing them. The study will track whale movements around these bays, with and without boats nearby, and attempt to tell if the whales change there movements when boats are around.

Researchers observe whales and boats with a theodolite (basically, like a surveyors transit), which gives them an angle from north and a declination (angle below the horizontal, since they are looking down on the bay from a cliff top). They record these, enter data into a computer, and the program computes the actual x-y coordinates of the whale. A later program will analyze the whale and boat sightings to determine, specifically, the closest the whales and boats ever approach one another. I wrote those computer programs.

At first glance, using a “flat earth” model, the computations are trivial, as any trigonometry student would learn in the first week of the course. Over the rather short distances involved, one would think this would be just fine. But the lead investigator wanted to compute distances across the surface of the bay using a curved earth model – even though he acknowledged that the adjustments would be petty. Why? Because other researchers were doing that, and would snipe at his published results if he didn’t too, that’s why.

We both went to the library and dug through textbooks to find a relevant-looking formula for the computation. I looked in trig text books, and didn’t find any. He looked in surveying books, and didn’t find any that seemed quite to match the situation, although there were various formulas for vaguely similar computations.

I ended up having to devise the formula myself. It was moderately messy, and involved figures of such massively differing orders of magnitude as the radius of the earth and the height of the cliff. This was 30-some years ago. It’s just possible that I *might* still have the formula laying around somewhere, if I go digging through all my old stuff, although I doubt it.

One might wonder what kind of accuracy one could get. Note that the radius of the earth itself varies from place to place on the earth, and the variation alone is of the roughly the same order as the altitude of any cliff top from which observations were made. I had no data on the radius of the earth at those particular places, only a general figure from various general sources. (Equatorial radius, 6378 m; polar radius, 6357 m.)

So I ran the computations with a whole bunch of hypothetical test figures, varying the earth radius over a range of several hundred meters, and varying the cliff-top altitude over hundred or so meters. The results were just the way one would have liked: The formula was highly insensitive to rather large (but reasonable) variations in earth radius, but highly sensitive to small changes in cliff altitude. IIRC I also tested variations in the angle of declination, also with desirable results. So I considered our computations to be highly reliable.

To anyone out there who is up on your trigonometry, you could try to reconstruct the formula from what I’ve given here. I think I’ve described the scenario completely enough.