Apollo 11's chance of failure

[Mr.Kobayashi]

*'Fate has ordained that the men who went to the moon to explore in peace will stay on the moon to rest in peace.

These brave men, Neil Armstrong and Edwin Aldrin, know that there is no hope for their recovery. But they also know that there is hope for mankind in their sacrifice.'*

A speech Nixon never had to give - in the event of a catastrophic failure. Obviously everyone involved knew it was risky and there was a non-zero possibility that Armstrong, Aldrin and Collins could all be killed.

But what were the chances Nixon would have to deliver that speech - or that the mission would fail in another way and they’d have to abort the attempt? If the mission was flown 100,000 times, how many times does it go wrong?

[davidm]

We can start by looking at the statistics we know.
Out of 7 attempted landings, 1 failed and was damned close to fatal. So that’s a failure rate of ~ 14%. It’s not a very large sample but it’s all we have to go on.

The shuttle program had 2 total failures out of 135 flights.

The Space Shuttle, By the Numbers

Thats a failure rate of ~ 1.5%.

So just looking at those numbers, Moon landings are about 10 times more risky than trips to low Earth orbit, something that makes sense intuitively, at least to me.

[AK84]

At what point? At launch, in Earth Orbit, after TLI, in Lunar Orbit/surface, during TrasEarth Coast, landing etc etc?

Soyuz 11 completed most of its objectives. Its also killed its crew. Was it a “failure”?

[Dioptre]

During the Apollo program, NASA moved from being quite supportive of quantitative risk assessment to not using it as a management tool.

Mostly this was because they couldn’t get the numbers to add up. The original idea was that 1/100 for mission failure and 1/1000 for loss of crew would be acceptable.

Those targets were pretty unrealistic for space flight.

Chance of success
Saturn V launch vehicle: 0.88
Command module: 0.90
Lunar module: 0.95

That gives a 1/4 chance of mission failure.

(Cite for the above: Fragola, JR 1996 “Risk Management in US Manned Spacecraft: From Apollo to Alpha and Beyond”, Proc Product Assurance Symposium and Software Product Assurance Workshop)

NASA didn’t like these figures, and didn’t circulate them widely or use them in planning. The view at the time (and afterwards) was that they were too conservative, and didn’t take into account reliability growth as the USA gained experience during the program.

By today’s standards, the use of point estimates would make the numbers next to useless anyway. It was a case of “The chance of the mission failing is 25%, but the chance of our estimate being wrong is 90%”.

As others have said upthread, the actual probability is under-determined by the historical data. As a rule of thumb, with a low number of data points add or subtract one accident to give a rough idea of the probability range, and another to give a conservative estimate.

So if you count Apollo 13 as a potentially catastrophic failure (which is only fair if we consider Apollo 11 as an average moon trip instead of the very first one), the odds of Apollo 11 being a disaster were somewhere between 0 and 30%.

The reliability experts put it closer to 30%, the mission managers put it closer to 0%. Thus it ever was …

[SpeedwayRyan]

Chris Kraft apparently gave Apollo 8 (not 11) a 50/50 chance of returning to Earth safety:

Narrator: While everyone knew the stakes were high, few realized the full level of risk NASA was assuming on Apollo 8. Even its flight operations director privately set the chance of a safe return at just 50%. Across the country, Americans watched in awe as Apollo 8 set out on man’s first journey to the moon.

Susan Borman: Chris came by one night. I said, ‘hey Chris, you know, I’d really appreciate it if you’d level with me. I really, really want to know what you think their chances are of getting home. I really want to know.’ So he sat there with me and he pondered and he said, ‘You really mean that, don’t you?’ And I said, 'Yes, and you know I do. I really want to know." And he said, ‘OK, how’s 50/50?’ I said, “Good, that suits me fine.”

From http://www.pbs.org/wgbh/amex/moon/filmmore/pt.html

[Novelty_Bobble]

For some context regarding NASA assessments of risk generally, I’d recommend running through Richard Feynman’s famous “appendix F” from the report into the Challenger disaster.

One pernicious behaviour described that always sticks with me is

We have also found that certification criteria used in Flight
Readiness Reviews often develop a gradually decreasing strictness. The
argument that the same risk was flown before without failure is often
accepted as an argument for the safety of accepting it again. Because
of this, obvious weaknesses are accepted again and again, sometimes
without a sufficiently serious attempt to remedy them, or to delay a
flight because of their continued presence.

Now whether the relative success of the Apollo program contributed to that culture…who knows? It is interesting reading nonetheless and yet again highlights Feynman’s genius for cutting through the crap and stating complex issues in a layman-friendly way.

[ElvisL1ves]

It’s worth mentioning that there were incidents that could have led to disaster in the Apollo program, but didn’t due to intervention by the astronauts and the ground crews. Apollo 11’s LM was nearly out of fuel when it landed, with some skillful piloting by Armstrong saving it, and with only a little worse luck would have had to set down in a rock field and possibly tipped over. Apollo 12 was hit by lightning during liftoff and almost aborted. Later in the mission, the CM/LM docking mechanism failed and Conrad had to fix it manually. Even Apollo 13 returned the crew alive. In all cases, if you define success as returning the crews alive, which was never a given, the failure rate was zero. In the two shuttle crashes, there was nothing anyone could have done once the launch was committed.

[Just_Asking_Questions]

The chance for mission failure was quite high, but they managed to make it work despite that.

For instance, the crew left the tunnel between the LM and the CM pressurized, so when the spacecraft separated, the LM had a bit more velocity than planned. That led to it landing at the extreme downrange limit of the planned landing zone. The landing zone was full of large rocks, and the LM had to hover and transverse to find a good place to set down. At touchdown, the LM had probably less than 15 seconds of fuel remaining. So they were very close to having to do a “fire in the hole” abort (firing the ascent stage while still attached to the descent stage), at low altitude, and depending on the attitude of the spacecraft, might not have worked.

Plus, there were numerous 1201 and 1202 program alarms (you can hear them in the audio). Those were indications that the computer was overloaded. Fortunately, it was a smart computer, and ignored some processes. The landing radar was on, and giving input to the computer before it needed it. The computer continued to function, and gave good landing data. But, if it had really got overloaded, it would have shut down, and if that happened at a critical phase of the landing, it could have been catastrophic.

Also, in simulations just prior to the landing, the simulation supervisor gave the mission control folk a bunch of really tough errors, multiple failures plus 1201 error codes. MC made the decision to abort based on the 1201 errors, and were informed at the end of the simulation that that was not the correct decision. So it was a good thing they did those simulations, because if they hadn’t, Apollo 11 would have aborted the landing.

In Apollo 12, the spacecraft was hit by lightning on takeoff, and it played havoc with the on-board systems. Fortunately, the Saturn V continued to work reliably, and in orbit the crew reset all the instrumentation and proceeded with the mission. However, there was a fear that the lightning had set off the pyrotechnics for the parachutes. There was no way to check, and if they pyros had gone off during launch the chutes wouldn’t have opened at landing and the spacecraft would really have made a splashdown.

In Apollo 14, there were two problems with the LM , the biggest of which was a faulty abort switch. There was a bit of metal debris inside the switch, and it created a short and told the computer that an abort was called for. Fortunately, the engine was not running at the time, because if it had an abort would have been triggered. The MC folk studied the problem and wrote new code, which was painstakingly sent up to the crew, who manually entered it into the landing computer. This software upgrade was not proofed before use. The change had to tell the computer to ignore the abort code at some stages of the flight, but to make the switch available at the later stages of the landing (in case it was really needed).

One of the missions had one of the five second stage engines cut out, but they burned the remaining engines longer and achieved orbit.

Any one of these failures could have caused a mission failure (which means no landing, not necessarily death in the cold void of space, with the bodies of the crew perpetually orbiting, forever).

[ElvisL1ves]

Just remembered another one - the Apollo 11 LM ascent stage wouldn’t fire due to a faulty switch. Aldrin fired it manually by poking the switch with a pen.

[Dioptre]

@ElvisL1ves

It’s worth mentioning that there were incidents that could have led to disaster in the Apollo program, but didn’t due to intervention by the astronauts and the ground crews. Apollo 11’s LM was nearly out of fuel when it landed, with some skillful piloting by Armstrong saving it, and with only a little worse luck would have had to set down in a rock field and possibly tipped over. Apollo 12 was hit by lightning during liftoff and almost aborted. Later in the mission, the CM/LM docking mechanism failed and Conrad had to fix it manually. Even Apollo 13 returned the crew alive. In all cases, if you define success as returning the crews alive, which was never a given, the failure rate was zero. In the two shuttle crashes, there was nothing anyone could have done once the launch was committed.

Slightly off-topic nitpick: As part of the Columbia Accident Investigation Board they asked NASA to rerun the scenario if they had correctly assesssed the Thermal Protection System damage (which definitely was possible). The rerun team decided with hindsight that they would have considered two options: inflight repair or an Atlantis rush launch. Inflight repair would have been abandoned as too risky, but an early Atlantis launch was feasible, and would have overlapped with the time Columbia could stay in Orbit.

No guarantees that they would have done this, or that it would have been acceptable, but it was possible.

More on-topic nitpick: “Failure rate” refers to the probability of failure. Just because you roll a six doesn’t mean that the probability of rolling a six was 100%. Just because you have no accidents doesn’t mean that the failure rate was zero.

[ElvisL1ves]

My point was that human intervention, where possible, can alter outcomes substantially, as the above Apollo anecdotes illustrate. Armstrong on Gemini 8 was another example (one which convinced Slayton he was the man to command the first moon landing).

Counter-nitpick: Failure rate is data-derived. Failure *probability *is a calculation. And there are multiple levels that can define failure - even Apollo 13 was a success in some aspects.

[Hail_Ants]

@Mr.Kobayashi

*'Fate has ordained that the men who went to the moon to explore in peace will stay on the moon to rest in peace.

These brave men, Neil Armstrong and Edwin Aldrin, know that there is no hope for their recovery. But they also know that there is hope for mankind in their sacrifice.'*

A speech Nixon never had to give - in the event of a catastrophic failure. Obviously everyone involved knew it was risky and there was a non-zero possibility that Armstrong, Aldrin and Collins could all be killed.

But what were the chances Nixon would have to deliver that speech - or that the mission would fail in another way and they’d have to abort the attempt? If the mission was flown 100,000 times, how many times does it go wrong?

The only reason Nixon would have had to have given* that particular speech* was if the ascent engine on the LM failed to fire. And the chances of that happening were very, very slim.

On the one hand the LM’s ascent engine was a reasonably simply, solid-fueled, non-throttle-able rocket motor. It was much, ***much ***simpler & reliable than the LM’s descent engine, the command/service module’s main engine, or any of the Saturn V’s engines. But on the other hand, because of its design simplicity, each actual LM ascent engine that flew to the Moon could not, and was NOT tested (including Apollo 9’s which only flew in Earth orbit, but its failure would not have been a disaster because of this). Each was manufactured, sealed, and delivered to NASA in the ascent stage of the LM brand new & untested. Of course hundreds just like it were tested here on Earth. But still the Grumman engineers always held their breathe a little before each LM lunar ascent launch because, as the speech says, there was no backup if it didn’t work.

[buddy431]

@Hail_Ants

The only reason Nixon would have had to have given* that particular speech* was if the ascent engine on the LM failed to fire. And the chances of that happening were very, very slim.

On the one hand the LM’s ascent engine was a reasonably simply, solid-fueled, non-throttle-able rocket motor. It was much, ***much ***simpler & reliable than the LM’s descent engine, the command/service module’s main engine, or any of the Saturn V’s engines. But on the other hand, because of its design simplicity, each actual LM ascent engine that flew to the Moon could not, and was NOT tested (including Apollo 9’s which only flew in Earth orbit, but its failure would not have been a disaster because of this). Each was manufactured, sealed, and delivered to NASA in the ascent stage of the LM brand new & untested. Of course hundreds just like it were tested here on Earth. But still the Grumman engineers always held their breathe a little before each LM lunar ascent launch because, as the speech says, there was no backup if it didn’t work.

Nitpick: the ascent engine was liquid fueled (hydrazine mixture plus dinitrogen tetroxide). You’re right that it was relatively simple: pressure fed, hypergolic propellants (no ignitor necessary), non-throttleable.

[carnivorousplant]

@Hail_Ants

On the one hand the LM’s ascent engine was a reasonably simply, solid-fueled, non-throttle-able rocket motor.

For all these years, I believed it to be hypergolic liquid fuel.

NASA had to calculate the weight of the LEM including rocks to get it to orbit, and the Command Module, I presume, did all the maneuvering for rendezvous.

[Bosda_Di_Chi_of_Tricor]

@davidm

We can start by looking at the statistics we know.
Out of 7 attempted landings, 1 failed and was damned close to fatal. So that’s a failure rate of ~ 14%. It’s not a very large sample but it’s all we have to go on.

The shuttle program had 2 total failures out of 135 flights.
http://www.popularmechanics.com/science/space/nasa/us-space-shuttle-program-by-the-numbers
Thats a failure rate of ~ 1.5%.

So just looking at those numbers, Moon landings are about 10 times more risky than trips to low Earth orbit, something that makes sense intuitively, at least to me.

Bad analysis.
The Shuttle Program built on the experiences gained in part from Apollo.
With Apollo we had less experience.
Ergo the risk has much to do with inexperience, though not altogether.

[davidm]

@Bosda_Di_Chi_of_Tricor

Bad analysis.
The Shuttle Program built on the experiences gained in part from Apollo.
With Apollo we had less experience.
Ergo the risk has much to do with inexperience, though not altogether.

Like I said, I was starting with the numbers we know (or at least that I know). I never claimed high accuracy.

[Xema]

@ElvisL1ves

Apollo 11’s LM was nearly out of fuel when it landed, with some skillful piloting by Armstrong saving it, and with only a little worse luck would have had to set down in a rock field and possibly tipped over.

I think this is just slightly misleading.

They were low on fuel because Armstrong had seen they were headed toward a small crater without a good landing spot, and had to maneuver clear both of the crater and some rocks that bordered it. They had 30 seconds of fuel remaining when they touched down - which sounds like not much, but would have allowed them to maneuver clear of substantially worse obstacles than they actually encountered.

It certainly was a good thing that someone with Armstrong’s level of piloting skill was at the controls.

[Stranger_On_A_Train]

@davidm

We can start by looking at the statistics we know.
Out of 7 attempted landings, 1 failed and was damned close to fatal. So that’s a failure rate of ~ 14%. It’s not a very large sample but it’s all we have to go on.

The shuttle program had 2 total failures out of 135 flights.
http://www.popularmechanics.com/science/space/nasa/us-space-shuttle-program-by-the-numbers
Thats a failure rate of ~ 1.5%.

So just looking at those numbers, Moon landings are about 10 times more risky than trips to low Earth orbit, something that makes sense intuitively, at least to me.

This is not a correct way of assessing risk from prior history. The simplest way to assess probability of failure is to use a first level Bayesian estimate:

Probability of success p in the next flight for f failures in n attempts:

p = (n-f+1)/(n+2)

For 1 failure out of 7 attempts this gives a 78% chance of success for a subsequent flight. For 2 failures in 135 flights this gives a 98% chance of success.

Stranger

[carnivorousplant]

@Stranger_On_A_Train

Probability of success p in the next flight for f failures in n attempts:

p = (n-f+1)/(n+2)

From what is that derived?

[davidm]

@Stranger_On_A_Train

This is not a correct way of assessing risk from prior history. The simplest way to assess probability of failure is to use a first level Bayesian estimate:

Probability of success p in the next flight for f failures in n attempts:

p = (n-f+1)/(n+2)

For 1 failure out of 7 attempts this gives a 78% chance of success for a subsequent flight. For 2 failures in 135 flights this gives a 98% chance of success.

Stranger

Thanks. My stats experience consists of having taken Statistics 101 thirty years ago. It’s not one of my strong points.