Advanced technologies deliver benefits every day. But, sometimes interactions with technology can go awry and lead to disaster.
On March 10, the pilots aboard Ethiopian Airlines Flight 302 were unable to correct a failure in one of the Boeing 737 Max 8’s automated systems, resulting in a crash and the deaths of all passengers and crew. A year earlier, almost to the day, another automated vehicle – not an airplane but an Uber self-driving car – struck and killed Elaine Herzberg in Tempe, Arizona.
As experts in how humans and technologies interact, we know that it is impossible to completely eliminate risk in complex technological systems. These tragedies are the result of regulators and industry experts overlooking the complexities and risks of interactions between technologies and humans and increasingly relying on companies’ voluntary self-assessment, rather than objective, independent tests. Tragically, that appears to have happened with Boeing’s aircraft and the Uber car.
The crash of Ethiopian Airlines Flight 302, as well as that of Lion Air Flight 610 in 2018, happened despite oversight from one of the most technologically capable regulators in the world. Air travel is remarkably safe in light of the potential risks.
Before the 737 Max 8 took to the air, it had to pass a series of Federal Aviation Administration inspections. Over the course of that process, Boeing convinced the FAA that the automated system was safer than it actually was, and that pilots would need very little training on the new plane.
The FAA cleared the 737 Max 8 and its flight control system to fly – and retained that clearance not only after the Lion Air crash, but also for three days after the Ethiopian Airlines tragedy.
From airplanes to automobiles
As airplane automation is increasing, the same is true for cars. Various companies are testing autonomous vehicles on roads all around the country – and with far less oversight than the aviation industry. Local and federal rules are limited, often in the name of promoting innovation. Federal safety guidelines for autonomous vehicles require them to pass only the same performance tests as any other car, like minimum fuel economy standards, seat belt configurations and how well they’ll protect occupants in a rollover crash.
There’s no reliability testing of their sensors, much less their algorithms. Some states do require companies to report “disengagements” – when the so-called “safety driver” resumes control over the automated system. But mostly the self-driving car companies are allowed to do what they want, so long as there is a person behind the wheel.
In the months before the March 2018 collision, Uber was under pressure to catch up with GM Cruise and Waymo. Uber’s cars had a sensitive object-recognition system, which at times would be deceived by a shadow on the road and brake to avoid an obstacle that wasn’t actually there. That resulted in a rough, stop-and-start ride. To smooth things out, Uber’s engineers disabled the car’s emergency braking system. The company appears to have assumed the single safety driver would always be able to stop the car in time if there was really a danger of hitting something.
That’s not what happened as Elaine Herzberg crossed the road. The Uber self-driving car that hit and killed her did see her with its sensors and cameras, but was unable to stop on its own. The safety driver appears to have been distracted by her phone – in violation of Uber’s policies, though it’s unclear how the company briefed its safety drivers about the change to the automated system.