04 Judgment 13 min read
The Ironies of Automation
Automation removes the easy 95% and leaves humans accountable for the hard 5% — using skills that only the removed 95% could have built.
In 1983 the psychologist Lisanne Bainbridge published a six-page paper called “Ironies of Automation.” It concerned industrial process control and it did not mention artificial intelligence, because there was none worth mentioning. It remains the single most important thing written about the situation we are now in, and almost nobody working on AI has read it.
Her argument has two moves. The first is that the designer of an automated system is a human being whose errors do not disappear — they are merely relocated from the operator to the design, where they are harder to see and arrive all at once. The second is the one that should keep people awake:
The designer, who tries to eliminate the operator, still leaves the operator to do the tasks which the designer cannot think how to automate.
Lisanne Bainbridge, 1983
Automation takes the routine, tractable, frequent portion of the work. What remains for the human is by construction the irregular, ambiguous and rare portion — the hardest part of the job. And the human is expected to handle it having lost, through disuse, the fluency that only routine practice maintains.
This is the ironies plural: we automate to reduce reliance on fallible humans, and in doing so we make the humans more fallible and their remaining responsibilities harder.
The aviation record is the closest thing we have to a controlled experiment
Aviation automated cognition earlier and more thoroughly than any other field, in a domain where every failure is investigated in public and in detail. It is the closest thing to a natural experiment on what we are about to do everywhere.
On 1 June 2009, Air France 447 was cruising over the Atlantic when its pitot tubes iced over. The airspeed readings became unreliable, the autopilot did what it is designed to do in the presence of bad data — it disconnected and handed control back — and three trained pilots had roughly four minutes to fly an aircraft that was, mechanically, entirely intact.
They stalled it into the ocean. Two hundred and twenty-eight people died. The pilot flying held the nose up for most of the descent, a control input that is correct in almost every situation a modern airline pilot ever encounters and catastrophically wrong in this one. The investigators found no mechanical failure that would have prevented recovery. What they found was a crew that had accumulated thousands of hours of supervising an aircraft and very few recent hours of flying one, confronted with the rare case the automation could not handle.
The industry’s response is instructive, because it is not what a naive reading would predict. Nobody proposed removing the automation — it had by then saved far more lives than it cost. The response was to mandate manual handling practice: periodic hand-flown approaches, upset recovery training, deliberate degradation of assistance in the simulator. The fix for over-reliance on automation was to make un-assisted competence a scheduled obligation.
That is the template. It is also, currently, what essentially no knowledge organisation does.
Automation bias has two distinct shapes
Two decades of human-factors research have separated the failure into two mechanisms, and they need different countermeasures.
| Failure | What it looks like | Why it happens | Countermeasure |
|---|---|---|---|
| Omission error | You miss a problem because the system did not flag it. | The aid becomes the perceptual channel. Anything outside its attention is outside yours. | Independent sampling: check things the tool was never asked about. |
| Commission error | You follow the system’s recommendation against evidence in front of you. | The output is fluent, confident and effortless; your own contrary reasoning is none of those. | Form and record your own judgment before exposure to the output. |
Commission errors are the more disturbing of the two, because they persist among people who know the tool is fallible. Fluency is a confidence signal that operates below argument. A well-written wrong answer is more persuasive than a badly-written right one, and language models are optimised, quite directly, for being well-written.
The jagged frontier
The most useful recent evidence is a 2023 field experiment run with several hundred Boston Consulting Group consultants. On tasks that fell inside the capability frontier of the model, consultants using it completed more work, faster, at higher quality — large effects, and the largest gains went to the lowest-performing consultants, which is a genuinely good news story about levelling.
On a task deliberately designed to fall outside that frontier — one where the model’s plausible answer was wrong — consultants using it were substantially more likely to reach the wrong conclusion than those working without it.
The frontier’s jaggedness is the whole problem. If capability degraded smoothly with difficulty, we could calibrate. Instead, two tasks that appear equally hard to you sit on opposite sides of the line, and the model’s confidence is approximately identical on both. The only thing that reliably locates the frontier is domain expertise — which is to say, the thing that is decaying.
Verification is not cheaper than generation
There is a comfortable story in circulation: humans will stop producing and start supervising. We become editors, reviewers, quality control. It sounds like a promotion.
In the domains that actually matter, it is not available, because of an asymmetry that gets very little attention. To reliably evaluate work in a domain, you need approximately the expertise required to have produced it. You can spot a wrong sum without being able to do the sum. You cannot spot a subtly wrong legal strategy, a plausible-but-flawed differential diagnosis, or an architecture that will fail under load in eighteen months, without the experience that would have let you generate the right one.
This is the verification gap, and it makes the supervision story circular: it proposes that humans retain the expert judgment while removing the practice by which expert judgment is acquired.
Which leads to the most serious structural problem of all.
The apprenticeship problem
Expertise is not transmitted by explanation. Aristotle’s word for what we are discussing is phronesis — practical wisdom, the capacity to perceive what a particular situation requires — and his claim was that it cannot be taught as rules, only developed through habituation in real situations with real consequences. Two and a half thousand years of professional training agree with him. Doctors do residencies. Lawyers do document review. Engineers fix other people’s bugs. Nobody becomes senior by reading about being senior.
The work that builds judgment is, precisely and without exception, the work that AI does best: the routine, high-volume, well-specified junior work. We are on course to automate the training pipeline for the expertise we have just established we cannot do without.
This is the best objection on the site and it is substantially correct. Aviation is dramatically safer. The right conclusion is not that automation is bad; it is that the safety came from a specific, expensive combination — automation plus mandated manual practice plus exhaustive public failure investigation plus enforced procedure. Aviation got the good outcome because it did the unpleasant work, after crashes, in public, under regulators.
The distinction that matters is between fungible skills and judgment skills. Long division is fungible: the answer is verifiable without the skill, so losing it costs nothing. Clinical reasoning is not: the output cannot be checked without the capability. Where the skill is required to evaluate its own output, atrophy is not a neutral trade — and the calculator analogy does not reach it.
Where judgment sits on the ladder
- 1
Cede Machine
The machine does it. You never see the work.
Acceptable only where errors are visible, cheap and self-correcting. Almost no consequential decision qualifies.
- 2
Approve Machine-led
The machine drafts. You review and sign.
Safe when verification is genuinely independent of the expertise being replaced. Rarer than it feels.
- 3
Collaborate Shared
You and the machine work the problem together.
- 4
Critique Human-led
You do the work first, unaided. Then ask the machine to attack it.
The default for any decision you would have to defend. Reach your own position, then use the machine as your sharpest adversary.
- 5
Reserve Human
You do it alone, on purpose, knowing help exists.
Reserved for the judgment reps that keep you competent — deliberately unaided work on problems you could have delegated.
Reach your own conclusion and write it down before you open the tool. This single sequencing rule neutralises commission error, which is the dominant failure mode. It costs minutes and it is the highest-return habit on this site.
Before delegating, record what you expect the output to say and what would make you reject it. If you cannot state a rejection criterion in advance, you are not supervising — you are watching.
Borrow the aviation fix outright. Block time — weekly, non-negotiable — for work you do unassisted that you could easily have delegated. Treat it as training load, not as productivity.
Regularly give the tool problems where you already know the answer is subtle. You are not testing the tool; you are building a map of where its confidence and its competence come apart.
If you lead people: protect some junior work from automation on the explicit grounds that it manufactures senior people. This is a real cost with a delayed return, which is exactly why nobody will do it unless it is named and defended out loud.