The Approve Button Is Not a Safeguard

A note for the developers shipping AI agents, from someone who keeps catching himself making the same mistake.

When we put an agent into production, the default safeguard is “a human reviews it.” Human-in-the-loop. It feels responsible, it satisfies the compliance people, and it feels safe. I reach for it every time. I have started to think the feeling is the problem.

This is not a new observation. It is a forty-three-year-old one that our industry has managed not to read.

Four ironies, 1983 edition

In 1983 Lisanne Bainbridge published a short paper in Automatica called “Ironies of Automation.” It was about industrial control rooms: chemical plants, power stations, the places where a process was automated and an operator was left to watch it. Every irony in it maps onto an agent pipeline today.

Irony one. We automate a task because we consider the human unreliable at it. Then we hand the human the hardest remaining job: catching whatever the automation could not handle. In my gloss on her point: the designer removed the person from the easy ninety-five percent and left them the five percent that defeated the automation, with less context than they had before.

In an agent pipeline that is the review queue. The model extracts, drafts, classifies, or routes. The human is asked to look at the residue. The residue is, by construction, the hard part.

Irony two. Skills you do not practice decay. Bainbridge’s operators lost manual control skill because they never exercised it, and then were expected to take over precisely when the automation failed. The engineer who stops writing code loses the ability to judge code. The analyst who stops doing the extraction by hand loses the feel for when an extraction is wrong.

Irony three. Nobody can stay vigilant watching a system that is right 199 times out of 200. Bainbridge cites the vigilance literature: sustained attention to a rare signal collapses within about half an hour. The 200th case is the one the gate exists for, and it arrives after the reviewer has approved 199 correct ones in a row.

Irony four. The better the automation gets, the worse all of this gets. More trust, less attention, deeper skill decay. Better engineering does not fix the problem. It intensifies it, right up until you cross into full autonomy and the human is no longer pretending to check.

That last one is the trap for people building agents right now. Every improvement in model quality makes the human gate weaker, not stronger, while making it look more reasonable.

What language models add

Bainbridge was writing about valves and pressure readings. Agents add three things she did not have to contend with.

Automation bias is measured, not speculated. Skitka, Mosier and Burdick showed in 1999 that people given an automated recommendation defer to it even against contrary evidence in front of them, committing both errors of commission (following a wrong recommendation) and omission (missing a problem the automation did not flag). Parasuraman and Manzey’s 2010 review put complacency and bias on the same attentional footing: the automation does not just get trusted, it gets watched less. A reviewer with a model’s suggested answer on screen is not a neutral second opinion. They are a second opinion with a thumb on the scale.

The reviewer is reading output framed by the model. The model chose what to summarize, what to surface, what to call the field. The human is judging a task the model now does better than they do day to day, through a lens the model built.

Language model errors are plausible by construction. A misread pressure gauge gives you an implausible number. A language model gives you a fluent, well-formatted, entirely reasonable wrong answer, because fluency is what it was optimized for. A plausibility check is the worst possible check for a system that fails plausibly.

Why it feels safe anyway

Moravec’s paradox explains the feeling. The things that are hard for humans (deliberate, effortful, step-by-step verification) are cheap for machines. The things humans are actually irreplaceable at feel effortless to us, so we do not count them as work: noticing that something is off against context the model does not have, deciding what is worth asking about, knowing what is at stake if this particular one is wrong.

Approval gating puts the human on their weakest task and starves their strongest one. The reviewer spends their attention verifying the thing the machine is good at, and has none left for the thing only they could do. It feels rigorous because it is effortful. Effort is not the same as coverage.

What has actually held up

None of this says remove the human. It says stop treating an approve button as a safety mechanism you never have to measure. Here is what has held up in the systems we run, with the parts that did not hold up included.

1. A checker that fails differently, and someone reading it in aggregate

The best example I have is one where the human gate worked on every single case and still missed the failure.

A learned override inside POAPP, one of our extraction systems, quietly mapped a generic charge line onto a real product code that did not belong on those orders. It survived ten months and about two hundred orders. The thing that had been flagging it the entire time was not a person and not a smarter model. It was QC-Check, a deterministic comparison downstream, asking one dumb question: is this line actually on the source document?

Here is the part the human-in-the-loop story gets backwards. When we audited the sixty-seven orders that had been through review, not one still carried the bad line. The reviewers caught it. Every one of them, every time, order by order, each fixing their own and seeing nothing systemic. The failure was never the humans. It was that nobody was aggregating what the humans kept correcting. A checker with a different error profile had been producing the signal for months; we had not built the cross-order view that turned sixty-seven individual corrections into one finding.

And the other two-thirds of the affected orders were never reviewed by a person at all. Some had no check coverage. Some were closed automatically with no human eyes on them. That is its own answer to whether a review queue is a safeguard.

Two lessons. Uncorrelated errors beat sustained attention, but only if someone reads the uncorrelated checker’s output as a population rather than a queue. And the cheapest decorrelated checker is usually the dumbest one. We wrote up the architecture of that principle separately, in No Learner Without a Decorrelated Checker, so I will not repeat it here.

2. Calibrate on outcomes, not approvals

Approving a plan is an anchoring exercise. The plan is in front of you, it is well-formed, and you are being asked whether it is fine. Confirming an outcome is different: the system committed to something falsifiable, it acted, and now you are being asked what actually happened. That question resists anchoring because the answer is in the world, not in the document.

This is what the Operational State Models layer on the SyscallAI platform is built around. Every response is reduced to a typed action contract, a commitment that can later be judged true or false, and every entry on that ledger has been confirmed by a person against the outcome, all one hundred and twenty-three of them. As of this week the model’s commitments matched what happened 89 percent of the time. That number can go up or down. The point is that it exists and that it is anchored to outcomes, not to whether the plan looked reasonable.

The same shape works for shipping changes. Before we shipped an address-matching fix to QC-Check in June, we replayed it against stored production data and predicted it would flip about 86 percent of a known failure class. We shipped it. The production mismatch rate went from 34 percent to under 3. The prediction was on the record before the outcome, which is the only way a prediction can teach you anything.

And the shape catches your own mistakes. When we audited QC-Check’s confidence score, we found it was anti-calibrated: the highest-confidence buckets needed corrections on 77 to 79 percent of approvals, the lowest on 53 to 59. We had shipped a number that pointed the wrong way. Only measuring against outcomes revealed that. An approve button never would have.

3. Put the rules in the substrate

Who can write to what, enforced at the database or the gateway, works while everyone is asleep. A prompt that says “do not modify the knowledge base” is a suggestion. A Postgres role that rejects writes is a fact.

FRIDAY, the memory substrate our systems share, hands autonomous agents a read-only database role. Not a read-only instruction: a role. We smoke-tested it: reads returned sixty-one rows, writes were rejected. The autonomous coding agent that runs overnight can query FRIDAY’s knowledge table and cannot write to it, by architectural decision, because one hallucinated summary written into the canonical memory would degrade every system that reads from it.

Athena, our content engine, learned the harder version of this lesson. It had a grounding gate that checks claims against evidence before anything publishes. It also had an API publish route that bypassed the gate entirely. The day we closed the bypass, the gate caught an identity hallucination on its first live run. A checker that can be bypassed is not a checker. It is a checker-shaped object.

4. Make actions reversible

Point-in-time restore has saved us where no review would have. Twice this year we overwrote records in FRIDAY that we did not mean to touch: once a terminal session that overwrote in bulk, once a routine that clobbered two entries by writing without reading first. Both times Neon’s point-in-time restore put the substrate back; the first time, the data restoration ran in about an hour. No reviewer was going to catch a bulk overwrite in progress, and none did. Reversibility caught it after the fact, which is the only time a bulk mistake can be caught.

The second incident had a footnote worth admitting. The backup table our own runbook described as a daily snapshot did not exist. We found out during the recovery. Our documentation had failed a ground-truth check that nobody had run because the documentation looked fine.

5. Have the automation escalate on surprise

The inversion that matters most: instead of the human watching the automation, the automation raises its hand when something does not fit.

Prometheus, our pre-execution judgment gate, returns graded verdicts (proceed, proceed with monitoring, pause for review, abort) rather than a binary. On one overnight autonomy run it returned pause-for-review, and, unasked, surfaced a routing path that could reach twenty-six client repositories with no allowlist, since fixed. Nobody had asked it about that. The gate found something outside the question.

Athena’s grounding gate does the same at publish time: it holds a piece with a stated reason rather than silently publishing or silently dropping. The standing rule when it holds is to add evidence, never to override.

The honest footnote here is that the surprise detector has to be calibrated too. Our fleet-health poller once declared two services down a minute after a deploy. Both were up. A ten-second probe timeout had met a fifteen-second cold start. An alarm that cries wolf trains humans to ignore it, which is automation bias running in the other direction. The fix was a backoff probe and a rule that nothing is “down” until two consecutive polls say so.

6. Gate humans on irreversibility and stakes, not frequency

Put the human where the action cannot be undone or where the cost of being wrong is high, not on every item because every item is where the queue is. And give every gate a removal criterion: it comes off when you have measured that the automated checker catches more than the human does.

I should be clear that the second half of that is a proposal, not a description of our current practice. We have added gates this year. I cannot find a single instance of us removing one, and none of our gates has a written criterion for its own retirement. That is the next thing to fix.

The tell

If you cannot state your human gate’s catch rate on injected errors, you do not have a safeguard. You have a feeling.

We have the instrument to measure it. When we chose the adversarial judge for Nemesis, our checker-of-checkers, we built twenty diff fixtures with exactly one injected bug each, wrote the passing bar down before the run, and put four candidate model judges through it. Three caught all twenty; the fourth caught nineteen. One of the perfect scorers was still eliminated, because catching bugs was not the only bar it had to clear, and we had written the other bars down first. That is a catch rate on injected errors, and it was cheap.

We have never pointed that instrument at a person. The ledger that is supposed to hold our checkers’ measured catch rates across six altitudes holds a handful of observations, single digits, and its drift-check field is null everywhere. The two-factor rule we wrote down, that an agent acts autonomously only when its own confidence and its checker’s measured catch rate both clear a bar and their product clears a third, is specified and not populated.

So the human gates in our pipelines are, today, exactly what I am warning you about. I have been running on the feeling more than I would like to admit. The difference between a feeling and a safeguard is a number, and I now know how to get it.