Stories about user interface failures with big consequences
December 21, 2019 5:29 PM   Subscribe

I am fascinated, both personally and professionally, by stories about complex systems that, in a crisis, fail primarily or partially because of poor interface design. Examples include this NY Times article about the Notre Dame fire detection system and this ProPublica story about the deadly touch screen steering system on U.S. Navy destroyers . Are there any other super-detailed stories in this vein?
posted by rockindata to Technology (28 answers total) 74 users marked this as a favorite
 
Therac-25
The system noticed that something was wrong and halted the X-ray beam, but merely displayed the word "MALFUNCTION" followed by a number from 1 to 64. The user manual did not explain or even address the error codes, so the operator pressed the P key to override the warning and proceed anyway.
posted by eruonna at 5:40 PM on December 21, 2019 [10 favorites]


You might enjoy the Kemeny Report on the accident at Three Mile Island.
posted by saladin at 5:53 PM on December 21, 2019 [1 favorite]


Air France Flight 447 where the flight system silently switched to "alternate law."
posted by dum spiro spero at 5:55 PM on December 21, 2019 [9 favorites]




If you dig into the 2018 Hawai'i false missle alert you learn that it was mostly because of bad UI (if that article isn't available this one has some pictures)
posted by jessamyn at 6:19 PM on December 21, 2019 [6 favorites]


The Lancaster Bomber (used during World War II) had an escape hatch that was too small for crew members to use, particularly because they were wearing parachute packs. Apparently, thousands of soldiers needlessly died because they couldn't exit their planes fast enough. There's some discussion of it here and here. (I'm assuming a pretty broad definition of "user interface".)
posted by alex1965 at 7:08 PM on December 21, 2019 [6 favorites]




There are many horror stories in the tech industry about somebody deleting or corrupting a database because they thought they were on a test system, but were actually on a production system.

I recall a Google outage a few years ago where someone had been editing a global blacklist file(?), and a simple typo caused the google to reject traffic from the whole internet. (or something like that. Can’t find the article now)

Death by PowerPoint chronicles stories of bullet-point presentations burying critical information that contributed to disasters.
posted by qxntpqbbbqxl at 7:15 PM on December 21, 2019


Something about corporate email and bananas.
posted by vrakatar at 7:32 PM on December 21, 2019 [5 favorites]


At the 2015 pagent for Miss Universe, host Steve Garvey announced the wrong winner at the end. The mistake was caused by the confusing design of the card he was reading.
posted by alex1965 at 7:45 PM on December 21, 2019 [3 favorites]


Check out this podcast: Cautionary Tales
https://feeds.megaphone.fm/cautionarytales

While not all directly related to the user interface , you'll probably enjoy all of the stories if you have that frame of mind
posted by McNulty at 8:06 PM on December 21, 2019


Years ago I read a gripping account of how the Challenger AND Colombia space shuttle tragedies both happened because of PowerPoint.

Boeing engineers used PowerPoint briefings to explain known problems with the spacecrafts to NASA, but PowerPoint is so badly designed that information hierarchies are muddled and basically the severity of the problem was made unclear by shitty PP slides that were impossible to decode sensibly. I don't remember the exact piece I read, nor where I found it (probably on Mefi actually), but the phenomenon is very google-able and much has been written about it.

PowerPoint is also implicated in some important military errors.
posted by nouvelle-personne at 8:37 PM on December 21, 2019


Florida ballot design in the 2000 election. On Googling to find a good link for this, I found out that it happened again in 2018.
posted by matildaben at 8:56 PM on December 21, 2019 [1 favorite]




Super detailed? Software Safety is an academic discipline. Nancy Leveson's personal webpage includes links to her Therac-25 report and lots of other case studies, and is a great place to start, as is her book Engineering a Safer World.
posted by caek at 9:28 PM on December 21, 2019 [1 favorite]


The grounding of Boeing 737 Max 8 planes due to problems with their Manoeuvring Characteristics Augmentation System (MCAS), is a high profile current example: 346 people dead and billions in lost revenue because of the company's decision to put a system live which could give planes unexpected handling patterns - without properly training or warning pilots. Here is a reasonably clear video talking about the issues.
posted by rongorongo at 12:30 AM on December 22, 2019 [2 favorites]


I work in healthcare technology and find this a really interesting subject for obvious reasons. Here’s a recent example of research showing prescribing errors correlated with the the ’electronic health record’ system used by GPs in the UK.
posted by mgrrl at 3:35 AM on December 22, 2019 [2 favorites]


In big industrial applications, you’ll want the phrase “abnormal situation management.” This ugly PowerPoint lists events caused by alarm floods/fatigue (the operator stops hearing alarms because there are so many useless ones) and other causes.
posted by chesty_a_arthur at 6:21 AM on December 22, 2019 [1 favorite]


Ok, that wasn’t the deck I thought it was . But it does cite the Texaco pembroke explosion as an example.
posted by chesty_a_arthur at 6:24 AM on December 22, 2019




A huge list here CAUSE MAPPING EXAMPLES AND CASE STUDIES
posted by lalochezia at 6:32 AM on December 22, 2019 [1 favorite]


The Overdose is a series of articles on Backchannel from 2015 about alert fatigue and poor user interface in medical software.
posted by spheniscus at 2:51 PM on December 22, 2019


If you listened to the Cautionary Tales podcast linked above, you should check out Galileo and Failure, a debunking of Tim Harford's use of Galileo's stone column example as a failure due to complexity.
posted by storybored at 9:16 PM on December 22, 2019 [2 favorites]


Fast Company released two stories on the typography/design issues that contributed to the Best Picture winner mix-up at the 2017 Oscars.
posted by helloimjennsco at 8:14 AM on December 23, 2019


Many of the above instances, and thousands more, are archived in the RISKS Digest archives. I did a FPP on the Blue about the digest when it turned 30 years old in 2015.

Not all of the incidents discussed in the digest are human-interface errors. Many are plain 'ol bugs. But it's all interesting and educational reading.
posted by JoeZydeco at 8:45 PM on December 23, 2019 [5 favorites]


One of my favorite FPPs is the Edge case saloon. A fine establishment. And one of my favorite comments belongs to k5.user: "When I started with a big defense contractor, they gave me the release build and said to play around with it to see how it worked etc. It crashed if you click the wrong button, etc., I say something and am told `this will be used by highly trained Marines, they won't click the wrong button, ever. Highly Trained'. Oh man, I didn't get out of that project fast enough."
posted by Melismata at 11:59 AM on December 24, 2019 [2 favorites]


This is sort of adjacent, but there is an emerging interdisciplinary quasi-academic community called The Maintainers that often deals with these kinds of stories. They have a mailing list where sometimes there are some interesting discussions related to design flaws and/or infrastructure failures/neglect.
posted by mostly vowels at 10:31 AM on December 27, 2019


Something about corporate email and bananas.

Free bananas in the kitchen!!!
posted by panama joe at 8:27 PM on December 29, 2019 [2 favorites]


« Older Mac Drive-to-Drive Transfer Question   |   "To Washing By Specialist Only"? Newer »
This thread is closed to new comments.


Tags

Share