Automation Needs a Brake Pedal
The Therac-25 taught computing one of its most expensive lessons: software should not be the only thing standing between a mistake and a catastrophe.
Between 1985 and 1987, the radiation therapy machine was involved in six known massive overdoses. The failures were not just the result of one bad line of code. They came from a system that trusted software too much, removed independent hardware protections, displayed cryptic error messages, and made it difficult for operators to understand what had actually happened.
In one failure mode, an experienced operator could edit treatment data quickly enough to trigger a race condition. The machine could report that no dose had been delivered while exposing the patient to a dangerous level of radiation.
That detail matters. The operator was not careless. She was fast.
The system turned expertise into a hazard because its safety assumptions did not match real human behavior. That is the kind of design mistake that looks unlikely in a test plan and completely predictable in production.
The corrective actions eventually restored independent hardware interlocks and added mechanisms that could stop an unsafe pulse even when the software failed.
That is the lesson for today’s AI systems.
Do not make the model its own reviewer. Do not let the same workflow generate, approve, and execute a consequential action. Do not assume a polished interface means the underlying state is safe.
Use independent checks, bounded permissions, human review, observable state, and a reliable way to stop the process.
Automation is useful. Automation with no brake pedal is just confidence connected to machinery.
What independent control exists when your AI workflow is wrong?
