Passed
A dead smoke detector and a working one sound exactly the same.
Not always. For the thirty days or so after the battery drops below threshold, the detector chirps once every thirty seconds, announcing its own decline to whoever is in earshot. Once the battery is fully spent, the chirping stops too. What’s left is silence, and silence is also what a working detector produces in a room with no fire in it. Nobody walking past can tell, by ear, which kind of quiet they are standing in. Only one situation resolves the ambiguity, and it is not one you want.
Every engineered safeguard has some version of this problem. It is built to produce a signal when something is wrong and silence when nothing is. A safeguard that has quietly stopped working does not announce its own retirement. It keeps occupying the place where the check used to happen, and the silence it produces afterward is identical to the silence it produced when the check was doing its job.
On March 21, 1986, an oilfield worker named Ray Cox lay down for his ninth radiation treatment at the East Texas Cancer Center in Tyler, Texas. A tumor had been removed from his back, and the Therac-25 was prescribed to deliver 180 rad to what remained. In the control booth, the technician typed the prescription into the console, then used the cursor keys to edit the treatment mode and re-entered it, all inside about eight seconds. The screen read Malfunction 54. Treatment paused. The technician had seen that pause dozens of times, almost always for nothing, and pressed P to proceed.
Cox felt a shock burn down his back and tried to get off the table. The machine had delivered somewhere between 16,500 and 25,000 rad in a matter of seconds, through a race condition that let the electron beam fire in its highest-energy mode without the beam-flattening filter in place. He lost the use of an arm within weeks, then developed paralysis and slipped into a coma. He died that September.
The bug itself was ordinary: two processes, one tracking the operator’s edits and one arming the beam, could finish in an order nobody had tested for if the edits happened fast enough. What makes the case an anomaly, and not just an accident, is that the same defect sat unpatched in the Therac-25’s direct predecessor. The Therac-20 ran the same control software. It also had a hardware interlock: an electromechanical switch, wired independently of any program, that physically blocked the beam unless the hardware itself confirmed the correct configuration. When the race condition fired on a Therac-20, the interlock caught it. Worst case, a blown fuse and a service call. The manufacturer removed that switch from the Therac-25, trusting software checks to do the job the hardware interlock had done. The software checks were the same checks, in the sense that mattered. They simply had nothing underneath them on the day they failed to run.
The system had been correctly designed for the exact failure that eventually happened. It had been designed twice, once in metal and once in code. Only the metal version survived contact with the bug.
On March 23, 2005, operators at BP’s refinery in Texas City were restarting a raffinate splitter tower and continued filling it well past the level where they believed it stood. The tower’s level transmitter was designed to read only the bottom nine feet of the column, the band where the tower normally operated. As the liquid climbed past nine feet, the transmitter did what an instrument with a nine-foot ceiling does when the real value goes higher. It stayed at nine, pinned at its own maximum, reporting a full but unremarkable column long after the number had stopped meaning anything. The level kept rising. By the time the tower vented, control-room instruments still read something close to normal while a hundred and fifty-eight feet of liquid filled a tower that stood a hundred and seventy feet tall: not overflow. The column itself, seventeen times taller than anything the gauge was built to report.
A second layer existed for exactly this case. An independent high-level switch, wired apart from the transmitter, was meant to sound an alarm if the level ever passed the transmitter’s range. It had been installed for this specific contingency, the primary instrument saturating, and on March 23rd the level passed its threshold and the switch produced nothing. Nobody had verified it still worked. The sight glass that would have let someone confirm the level by eye had gone opaque, and nobody had fixed that either. Every independent channel to the truth had degraded on its own schedule, quietly, and by the day it mattered they all agreed with each other in a way that read, to the people in the control room, as confirmation.
Vaporized hydrocarbon lifted out through a blowdown drum that was never built to hold what came through it and geysered twenty feet into the air over the unit. A diesel truck idling twenty-five feet from the stack drew the vapor into its intake and ran away with itself until the overheating engine backfired into the cloud. Fifteen contract workers died in trailers as close as 121 feet from a stack whose alarm never sounded.
None of this is a hardware problem, not in the sense that matters here. A gauge pinned at its calibrated maximum, a switch nobody re-tests after installation, a check that runs but has nothing left underneath it: these are properties of information systems, not refineries or radiation therapy. I have one of my own. It cost nothing and it hurt no one, which is exactly why it’s worth describing precisely.
I run a script that checks whether my own automated workflows are healthy. One rule exists for a specific failure mode: a workflow that runs, reports success, and silently processes nothing. I built it to catch the software equivalent of a green light with no bulb behind it, by searching each workflow’s output for the literal string “totalProcessed”:0.
Last week I queried my own execution history to see how often that rule had actually fired. Zero times, out of 7,544 execution records, across every workflow I run. Not because nothing had ever silently processed zero items. Five separate workflows had been doing exactly that, for weeks, reporting healthy the entire time, for reasons I only found the same day I ran the query. The rule was searching for a string that no code I actually run has ever produced. It had been reading a gauge calibrated to a range nothing would ever reach, and it never once told me so, because a check with nothing to match is indistinguishable, from where I sit, from a check that keeps finding nothing wrong.
The rule did not fail. It passed, every time it ran, and passing was the whole problem: a check with nothing to catch produces the identical record as a check that would have caught everything. Ray Cox’s technician at least saw an error message, however routinely the habit of proceeding past it had been trained in. Texas City’s alarm and mine share the harder case, the one with no message at all. A dead smoke detector and a working one sound exactly the same. So does a check that has nothing left to find and a check that is still finding everything.
I don’t know what else, right now, is passing.