Why Institutions Misdiagnose Failure

When systems fail, institutions reach reflexively for the same explanation: human error. An operator clicked the wrong button. A user misunderstood the screen. A staff member failed to follow procedure.

This diagnosis is comforting because it localizes responsibility. It preserves the system by isolating the failure to an individual. And it suggests a straightforward remedy: training, discipline, or replacement.

But across domains — nuclear power, medicine, aviation, elections, and now AI — “human error” has proven to be a poor explanatory category. It describes the final action, not the conditions that made that action likely, repeatable, or inevitable.

In design terms, “human error” is not a cause. It is a symptom.

The Pattern: When Interfaces Shift Cognitive Burden

Decades of safety engineering and human–computer interaction research converge on a consistent finding: systems fail most often when interfaces force humans to compensate for design ambiguity.

This happens when:

  • critical actions are visually indistinguishable from routine ones,
  • system states are hidden or poorly signaled,
  • prior actions cannot be easily reversed,
  • feedback is delayed or non-specific,
  • or operators must remember constraints rather than see them.

In these conditions, the system is effectively outsourcing governance to human memory and judgment — precisely where it is weakest under stress.

Calling the resulting failure “human error” mistakes the site of failure for its source.

Therac-25 and the Cost of Invisible State

The Therac-25 radiation therapy accidents are often summarized as a software bug story. But the deeper failure was interface-related.

Operators had no reliable way to see the machine’s internal state. Error messages were cryptic. Rapid data entry could place the machine in unsafe configurations without visible warning. The interface encouraged speed while obscuring consequence.

When patients were overdosed, investigators initially blamed operator mistakes. Only later did it become clear that the system was designed in a way that made safe operation dependent on perfect human behavior.

That is not a training problem. It is a design failure.

Three Mile Island: Control Without Comprehension

At Three Mile Island, operators faced a flood of alarms — hundreds activating simultaneously — without a coherent representation of system state. The interface conveyed activity but not meaning.

Operators acted rationally based on what they could see. Unfortunately, what they could see was misleading. Valves appeared closed when they were open. Indicators reflected commands, not physical reality.

Subsequent analyses showed that no amount of operator expertise could compensate for the interface’s failure to present causal structure. The system demanded interpretation under pressure without providing the tools to do so.

Again, “human error” described the moment of action, not the design that shaped it.

Ballots, EHRs, and the Politics of Blame

The same pattern appears in ostensibly non-safety-critical systems.

  • Poor ballot design has altered election outcomes by confusing voters.
  • Electronic health record interfaces have increased prescribing errors by burying dosage information or normalizing dangerous defaults.
  • Administrative systems routinely induce errors by forcing users to navigate inconsistent workflows.

In each case, the institutional response is similar: retrain users, issue guidance, blame inattentiveness.

What is rarely acknowledged is that these systems convert design decisions into behavioral risks, then assign responsibility downstream.

Why “Human Error” Persists

The persistence of the human-error frame is not accidental.

Design failure is institutionally expensive. It implicates procurement decisions, vendor relationships, standards bodies, and leadership accountability. Human error is cheap. It resolves the incident without reopening foundational choices.

But this misdiagnosis has a cost: systems do not improve, failures recur, and trust erodes.

From a governance perspective, “human error” is often a way of protecting the interface from scrutiny.

Implications for AI Systems

AI failures are already being narrated in this familiar register: hallucinations, misuse, overreliance, user misunderstanding. The risk is that the same misdiagnosis will harden into doctrine.

When an AI system produces an authoritative answer in a high-risk domain, and a human acts on it, the question is not whether the human should have known better. The question is:

What did the interface signal about authority, reliability, and permission to act?

If those signals were ambiguous, misleading, or absent, the failure belongs to design.

Reframing the Problem

To say that “human error” is a design smell is not to deny human responsibility. It is to locate responsibility where it can be exercised meaningfully.

Design is where institutions decide:

  • what users are allowed to do,
  • what they are warned against,
  • what they can see,
  • and what they are expected to infer.

Those decisions are governance decisions, whether acknowledged or not.

Where This Leaves the Design Arc

The design arc does not argue for perfect interfaces or error-free systems. It argues for something more modest and more demanding:

That institutions stop treating interface failures as behavioral deviations, and start treating them as structural choices with predictable consequences.

Only then does it become possible to talk seriously about responsibility, safety, and trust — in AI systems or any other.