Boeing software fix for 737 Max

Will the company’s engineers have access to the full set of data from the crashed plane’s black-box (and any relevant telemetry from other sources)?

I am wondering if they can replicate the exact set of variables the pilots in the fatal crash encountered and then run them on the new software. That would be the ultimate reassurance.

The airplane manufacturer participates in airliner crash investigations so they should have access to this data. But as to why this would be the ultimate reassurance I disagree–each crash is unique.

With the caveat that I have no special expertise but have been following these events out of interest, this is what I can offer from information that I’ve gleaned so far:

The software changes had already been in the works since long before the Ethiopian crash, but AIUI more aggressive changes were subsequently made, along with the AOA “disagree” warning, formerly optional, now being retrofitted on all 737 MAXs that don’t already have it.

I suspect the answer to your question is that between the voice recorder and the flight data recorder, probably “yes”, but the subtleties are probably not relevant because the big picture is becoming clearer (subject to confirmation, as the Ethiopian data has yet to be released, and the Lion Air data, too, is still being analyzed). But the bigger picture appears to be a combination of things, the most significant of which appears to be Boeing’s attempt to skimp on training costs for the MAX 8 by minimizing the differences from earlier generations, a factor that was said to have saved airlines from several hundred thousand dollars per plane up to potentially over a million per plane.

The most potentially damning result of this cost-cutting strategy appeared to be the complete failure to document the MCAS system in the MAX 8 flight manual. Especially odd was the fact that “MCAS” appears in the glossary, but nowhere else in the flight manual, suggesting it was part of some earlier draft but ultimately removed. And MCAS is the subsystem that behaves completely differently from earlier flight control systems in specific manual flight situations. Specifically, the mounting configuration necessary for the big CFM LEAP engines gave the MAX a tendency to pitch nose-up in certain situations which the software was designed to counter, a dynamic present in no other 737 model except the MAX, and moreover, the MCAS behavior was also highly unusual in creating strong but intermittent nose-down trim after brief pauses, a behavior that never happened in any previous models. And on top of all that, the behavior could be triggered by a faulty angle of attack (AOA) sensor, of which there are two, and Boeing in their wisdom never considered it necessary to provide, as standard equipment, a warning when the two sensors disagreed.

Maybe I should have said ‘best possible’ reassurance. Indeed, there will always be a butterfly flapping its wings somewhere.

Would putting a simulator through the same flight as the crash not show if the fault occurs again?

The best informed speculation on what probably happened, pending better data from the black boxes, is that it wasn’t really something that could be characterized as a “fault that occurred”, say in the sense of the software doing something that it wasn’t supposed to do. It appears to have been more like (1) an existing equipment fault or faults, affecting one AOA sensor and potentially an airspeed sensor as well, then (2) the MCAS doing exactly what it was supposed to do, given the bad inputs, and (3) neither pilot recognizing the behavior because they had never been trained on it, and therefore not making the connection to the relevant checklist.

The irony here is that Boeing claimed that pilots already had a documented procedure for dealing with runaway trim issues. This is true if read in a literal sense, since the documented checklist procedure would have fixed the problem, but disingenuous if one considers that this MCAS behavior looks nothing like runaway trim and the pilots would have to creatively make the connection themselves, as indeed apparently happened on a previous flight of that very same Ethiopian plane. But according to accounts of the Ethiopian cockpit voice recorder, in the final seconds the captain was still furiously looking through the flight manual for a solution, while the F/O apparently was mainly praying. Neither of them had a clue what to do (which would have been to flip two switches to totally disable the electric trim system, and then trim manually).

To expand a bit on what Wolfpup wrote: the problem isn’t a single “glitch” in the original software that either is or is not reproduced by the new software. Those flights crashed because of a cascading set of events including bad sensor input. A simulation would likely use all of the variables involved in one crash or the other, including the pilot control inputs and, if relevant, one or more failing angle-of-attack (AOA) sensors.

You’d likely run it that way (but with real pilots in the simulator) until the moment the MCAS system began putting the plane in an uncommanded dive. At that point, there are a couple of possible outcomes: maybe when the MCAS system sees disagreement between two AOA sensors, the revised software causes it to disengage while notifying the pilots that it has turned itself off. Maybe it begins an uncommnaded dive but the pilots’ control inputs (pulling back on the yoke or maybe fiddling with the pitch trim) causes it to turn off and stay off.

There are lots of other possible outcomes (including an uncommanded and uncontrollable dive into the ground, just like happened before). But it’s a lot more complicated than resolving a single-cause bug that produces a single fault.

Parenthetically, I’m shocked that the pilot of the Ethiopian flight didn’t know how to disable the MCAS system. He had no reason to know prior to the Lion Air crash, but I would imagine that nearly every 737 Max 8 driver in the world made a mental note about how to do just that after the likely cause of the Lion Air crash became known. (The first officer only had something like 200 hours, so I’m less surprised by his reaction). Do any of the commercial pilots on the board care to weigh in on that?

Are there any fragments of such airplane control software available for public viewing? Fragments large enough to get a feel for the decision making? It seems like a possible application for “fuzzy logic.”

IIRC I’ve never personally viewed software based on “fuzzy logic.” When we talked about it 30 years ago it seemed that Japanese countries loved it, but Americans computer scientists smirked at it!

I’m sure it is software based on fuzzy logic, that it, it accepts and produces inputs and outputs based on both analog and discrete signals. But I’m pretty sure that is uses general purpose processing units, not specialized control-systems logic units.

Let’s put the board of Boeing on board the first test flight.

I have supported and reviewed dynamic simulations on a software platform called Easy5 for aeroderivative / power gas turbines. The aeroderivatives are essentially airplane engines modified for power production.

I am guessing Easy5 is also used for airplane dynamic simulations. See their brochure here : https://documents.mscsoftware.com/cdn/farfuture/DnMn4fwh4JM42U7PpP8M4jhN3d5FwksPctO6BtTaLrg/mtime:1404847988/sites/default/files/sb_aircraft_equipment_ltr_w.pdf

There are two basic types of simulations:

  1. The “airplane” is simulated with mathematical models. The control system is simulated using an emulator that has the controller elements built in software.

  2. Same as above except that the real control system is used to interface with the “airplane” simulation. This is called hardware in the loop - Hardware-in-the-loop simulation - Wikipedia

You can save multiple states in the simulation like takeoff, 1 min from takeoff, engine malfunction, etc and then zoom in to see how the overall system reacts to those scenarios.

So assuming Boeing has the above dynamic simulations (pretty sure they do), it will be easy for them to check the new code by simulating the malfunction scenario. However, as the saying goes “All models are wrong, some are useful “. The malfunction is programmed by humans who may not understand it fully.

The general controllers used in the industry are simple PID (Proportional-Integral-Derivative) controllers with the derivative function disabled for most but a select few.

In the past, I’ve been involved in implementing Model Predictive Control (MPC) and nowadays I work on Artificial Intelligence (AI) implementations. The advanced controllers (MPC, AI, …) are NOT used for safety critical functions where traditional controllers have proven robustness and reliability.

Boeing is still flying B1 and B2 flights. I would guess no one that works for Boeing would have any issues being on one of those planes. I am currently sitting in the 737 assembly building, I haven’t heard anyone say they would not fly on our planes.

There’s a good read about airplane safety and software design here

In a nutshell the 737 Max is a plane that uses software (MCAS) to override an intrinsic design flaw, and that’s why it’s so hard to disable the MCAS. In all other planes the pilot is the ultimate arbiter of what’s going on. In the Max, it’s the computer.
Bigger engines mean the 737 Max is not the same plane as the 737 that has been in the skies for 50 years.
Those big engines, which are placed further forward on the wings, can make the plane’s nose rise up ridiculously fast, leading to a stall.
Software developers created code to prevent a stall, but the code relies on only one sensor, and never double-checks with any other sensor.

So if one sensor fails (such as being jammed by dirt) the computer thinks it’s nosing up and a super strong robot grabs the controls and overrides the pilot, smashing the plane into the ground while chirping ‘I’m a good boy preventing a stall’.
This is the ultimate problem of the ‘we’ll fix it in beta’ attitude.

As the OP’er I wanted to thank everyone for so many informative comments (and links).

Thanks!

This is what I came here to say. Them along with the top officials at the FAA.

And TV cameras, if they really want to assure the flying public. And not just the first test flight - make them ride all the parked planes out of the desert back into service.

Why didn’t the Boeing engineers also then design the software originally to gauge things like, nose attitude and plane attitude so that the “super strong robot” could also sense that, “I’m a good boy also sending this airplane plummeting nose-ground into the ground?”

I don’t think it’s been established that the computer nosed the plane into the ground. All that is known is that the tailplane jack screw indicates a downward trim. Unless it was in full deflection the pilots should have had elevator control of the plane.

This I believe and agree with you is the core problem. At least in the Chemical Industry, safety related functions like this will have triple redundant sensors with 2oo3 (2 out of 3 voting).

I am surprised to see Boeing only offering a “software fix”.

Because nose attitude isn’t related to angle of attack. You can have high angle of attack and a low nose attitude. In a steep turn the nose attitude is close to level but the angle of attack is high. You can have a very high nose attitude and a low angle of attack.

The most simple design change that Boeing could have done right at the start would be to just require the MCAS to have two AoA signals to operate. Had they done that then neither Lion Air or probably Ethiopian would have crashed and MCAS would remain a mystery function that no one knew about. It would still have inherent flaws but they would lie dormant.

It’s probably worth talking about how MCAS would operate in a situation it was designed for.

Let’s say a B737 MAX had been upset somehow, maybe it’s flown through some wake turbulence from an overflying A380. The autopilot doesn’t cope with that sort of thing so it’s disconnected itself and handed the pilot an aircraft with, for example, the nose pointing very high and the airspeed rapidly decreasing. The pilot, being well versed in unusual attitude recovery, finds the elevator quite ineffective with the low airspeed and rolls the aircraft into a steep bank which helps the nose drop to below the horizon. Now with the nose low and airspeed increasing, the control surfaces are becoming more effective and the pilot rolls the wings level and starts to recover from the resulting dive.

If the pilot pulls out too quickly the jet might stall and require further recovery which will cost altitude. On the other hand if he is too timid, the recovery will cost more altitude than necessary as well. The aim then is to pull out from the dive in a way that you are close to the stall but without actually stalling.

So our pilot is pulling out from the dive in the B737 MAX. The harder he pulls, the closer the aircraft gets to stalling but also the harder he has to pull to get even closer to the stall. When the angle of attack gets to a certain value something happens, the engine nacelles create some lift of their own which adds some extra pitch up effect without any increase in effort from the pilot. The pilot now has to decrease the pull effort to avoid stalling and has lost the sensory cue he had for how close he was to the limit. That’s what would happen without MCAS.

With MCAS, as the angle of attack reached the value where the engine nacelles started doing their weird little pitch up thing, the MCAS sends a nose down trim signal to the stabiliser. The leading edge of the stabiliser moves up causing the plane to want to pitch down and the pilot has to increase his pull force on the control column to keep the aircraft where he wants it, i.e., close to the stall. By doing this the pilot has kept the “feel” for what the aircraft is doing.

As the recovery is completed and the plane approaches level flight, the pilot relaxes the pressure on the column and starts flying like an airline pilot again rather than a fighter pilot. Meanwhile the MCAS has sensed the AoA has reduced to an acceptable level and has returned the stabiliser to its original position.

Note that nowhere in that scenario did the jet stall, nor did the MCAS help the pilot recover from a stall, the purpose of the MCAS was to mimic classic aerodynamic qualities of well behaved training aircraft and provide the pilot with sufficient feedback to let them manoeuvre close to the edge of the envelope.