How Do We Make It Realize Something Went Wrong?

Judith gave a confident, badly wrong answer last night — and had no way, on her own, to tell it was wrong, let alone why. I had to point it out. That's the real problem with self-improving agents, and it isn't the one you'd expect: not that she failed to act on a lesson, but that she has no way to reach one — no tools to decide, by herself, why a response was inappropriate in the first place. Let me use an X post involving Jensen to illustrate this.

A meme — the punchline to a week of noise. A SemiAnalysis piece claimed Nvidia's CPO / Kyber roadmap had slipped; Nvidia refuted it with that exact line; Patrick Moorhead turned the refutation into a joke, Jensen next to him onstage.

Judith read it and answered like it was a research brief — a solemn, confident breakdown of who was ahead in the roadmap fight and what it all signaled. Competent, earnest, and completely beside the point: I'd shared a joke, not asked for analysis.

And once I'd pointed it out, she wrote it up perfectly — a real file, titled "Social Meme RSI Loop Miss", that got the diagnosis exactly right and even proposed the fix. That's the tell: handed the lesson, she could reproduce it flawlessly; on her own, she had no way to reach it at all. Unless a human labels the miss or supplies the missing context, Judith is blind to it — and that isn't a quirk of this agent, it's the default. The rest of this post is about how much that one dependency costs, at every step of trying to make the thing improve.

Anatomy of one reply

What answering this actually requires

Before the loop, look at the task. "React to this screenshot" is not one capability; it's a pipeline, and a wrong answer can come from a failure at any stage.

Several of these stages failed at once, and each needs a different intervention — a tool for perception, data for intent, retrieval for recency, reasoning for pragmatics. Hold onto that: it means even a perfect signal that "the answer was bad" leaves the loop with a second problem — which stage was bad? The reward is one scalar smeared across a seven-stage pipeline. Credit assignment within a single response, before you even get to credit assignment across the conversation.

The diagram

The loop everyone draws

Every RSI system draws a similar diagram: observe an outcome → rate it → diagnose → propose a fix → validate → curate it back into behavior. Six boxes, six arrows, back to the start. The mechanics of it are, by now, a solved engineering problem — reflect, propose, apply, gate, revert.

The mechanics are not where loops fail. As Jason Wei argues in Verifier's Law, and Song et al. formalize as the generation–verification gap, self-improvement is bottlenecked on verification — on cheaply and correctly knowing whether an output is good. Lilian Weng makes the same point for auto-research and coding agents in Harness Engineering for Self-Improvement: the ceiling is the evaluator. Where their work lives on verifiable tasks — code that compiles, math that checks, tests that pass — a deployed personal assistant has no verifier to fall back on. "Was that a good reply to a friend's meme" has no unit test. So the bottleneck they name for verifiable domains becomes, for a conversational agent, the whole game — and every arrow in that innocent diagram turns into a distinct measurement problem. Here they are, walked through the Jensen miss.

Difficulty 1 · The rater is the crux

"Making it realize" has a name — the rater — and it's the first place this breaks. The naive instinct is to score the response in isolation: was that a good answer? You can't. Nothing in the message tells you whether it landed. The signal is in what the conversation does next — a real idea, and not mine: continuation-as-reward is developed by Gooding & Grefenstette, and the older deployment-feedback line uses "a human turn followed → label it good."

The reason this is a genuine upgrade is worth stating precisely: scoring a message alone asks a model for a prior — a guess. Reading the next few turns gives it evidence — it doesn't guess whether the analysis was wanted, it observes me replying "you missed the point" and supplying the Kyber chain. That's the difference between an opinion and an outcome.

But the continuation has a floor, and the floor is where most failures live: often the conversation gives you nothing — it just moves on. Liu, Zhang & Choi put the empirical version bluntly: implicit follow-up feedback is useful for understanding users but noisy as a learning signal.

Silence after a good answer is identical to silence after a bad one.

So you build raters to grade the continuation — in practice not one but a bench of them, each asked a different question — and now you're in the single most-studied failure zone in this whole enterprise, and none of the news is good:

• The rater inherits the base model's blind spots. If it's the same family that made the error, its errors are correlated with the agent's — panels of judges collapse to two effective votes because they agree for the same wrong reasons. A judge that shares the judged's prior certifies consensus, not correctness.
• Judges are lenient and self-preferring by default. Our own calibration reproduced this exactly: the first rubric rated ~everything ≥ 2; 23 of 25 ignored messages were false positives.
• Reliability is not validity — a rater can be perfectly consistent and consistently wrong (2606.19544). And to know your rater is valid you must calibrate it against real human judgment — the exact scarce signal you built the rater to avoid needing. The validation is circular.

None of that is a dead end, though. You can build the continuation-rater — and, the harder half, calibrate it against a real outcome — on ground that already exists:

Difficulty 2 · Getting gold when the chat gives nothing

Say the rater flags a suspicion — I think that was off — but the chat stayed silent. Where does ground truth come from? Not the conversation: it isn't there, and it never was. You have to leave the chat. There are exactly three places, and the trick is routing the suspicion to the right one:

• The world, for anything checkable. The Jensen miss is mostly this — who is in the image and what the news chain was are facts. A reverse-image lookup, a vision call, a query for "Nvidia roadmap intact." No human, no chat signal; the gold is retrievable.
• A stronger reference model, for judgment calls reality can't settle. Better-calibrated than the model that erred. Silver, not gold.
• The human, rationed to the residue: unverifiable and high-stakes and high-suspicion. Everything else, log and let go.

Two traps. The correlated-error trap: if the agent "verifies" by re-examining with the same model that erred — especially where the model's own perception is the weak link — it cheerfully re-confirms the mistake. Verification has to be independent. And triage is the whole game — you cannot verify every turn, and most don't deserve it.

Each rung of that ladder — checking a claim, catching a fabrication, testing an assertion about an image — has a benchmark behind it:

Difficulty 3 · A diagnosis is not an intervention

Suppose you paid for the gold and have a hypothesis. Judith had a perfect one, in writing, and it improved nothing — because knowing what broke is not the same as fixing it, and picking the wrong kind of fix is how you fool yourself. Route by the gap:

• Knowledge gap (the Kyber chain) → a retrievable memory. Cheap; low-value, because news ages out.
• Behavior / routing gap (reflex earnest analysis) → a rule or routing step: classify intent before answering a shared image. The generalizable one.
• Capability gap (say, telling who's in a photo) → no rule fixes this. You cannot instruct a model into better perception. The honest fixes are a tool or "admit uncertainty." Pretending a prompt cures a capability gap is the most common self-deception here.

And a gate that keeps the loop from rotting: don't mint a permanent rule per mistake. Rules bloat context, contradict each other, and cause over-correction — fix the meme case and now she under-reads an image that really did want a serious answer. One-off → a decaying memory; a recurring class → earn a rule; real volume → maybe weights. You climb that ladder on repetition, not on one bad night — and because the failure was spread across four pipeline stages, one rule was never going to close it.

Of the three gaps, the intent one — is this a joke or a question — is the most trainable, and it's the one with data:

Difficulty 4 · An unvalidated fix is worse than none

This is the difficulty that makes the others dangerous instead of merely hard: a fix you didn't validate changes behavior in exactly the region you can't see. You will never watch the genuinely serious image she now under-reads because of the meme rule.

So a change survives three gates before it touches live behavior: held-out replay (does it get the failure case right now?), a regression set (does it break cases it used to get right? — where over-correction dies), and attribution (does the failure class actually stop recurring?). Judith wrote the held-out eval for the meme case herself. It was never run. That distance — between a written eval and a gated one — is the distance between a lessons file and a learning system.

Both gates run on the same fuel — held-out preference data, which is one of the few things this field has plenty of:

Difficulty 5 · The note is the tombstone

Which brings us back to the note.

Curate-back — promoting a validated change into the agent's live, always-loaded behavior — is where nearly every deployed loop quietly fails. Not because it's conceptually hard, but because writing a reflection feels like closing the loop, and it isn't. This is the paradigm Reflexion popularized — verbalize a lesson, store it — and its unspoken failure mode: a perfect post-mortem in a notes/ folder that no future turn reads is, behaviorally, identical to having learned nothing. The recent memory-systems survey names the same gap from the other side: agents are fine at write and read and neglect manage — accumulation without curation is where systems fail. Judith wrote a genuinely good analysis and filed it exactly where it could not affect her.

And even wired, promotion fights you: the live policy has to stay small and coherent. A lessons file that grows one entry per mistake becomes its own failure mode — internally contradictory, too long to follow, the exact context-bloat you were optimizing away. Improvement is not accumulation. It's maintaining a tight, validated policy — merging, generalizing, and retiring rules as aggressively as you add them.

Even the neglected half — maintaining the policy over time, not just writing to it — has benchmarks now:

Related work

What's actually new here

A few things here I haven't seen put plainly. That the "verification is the bottleneck" argument — made for code and math, where you can check the answer — bites hardest exactly where there's nothing to check against: "was that a good reply to a friend" has no test. That silence tells you nothing: a person gone quiet after a good answer looks identical to one gone quiet after a bad one — and quiet is the common case. That when the chat won't hand you the truth, there's a clean rule for where to go get it — check the world, escalate to a stronger model, or spend one of your rare human questions. And the one I'd stake the post on: an agent can write a flawless post-mortem of its own failure and change nothing. The note is the tombstone.

An agent that can write a flawless account of its own failure and still hand you the same failure tomorrow is not a curiosity. It's the current state of the art.

The honest part

Every arrow is a research problem

None of this is about the model. A smarter model writes a more articulate wrong analysis and a more confident wrong ID. And none of it is about the machinery — the bandits, the ledgers, the fail-closed gates. That's the easy half, and it's mostly done. The hard half is that every edge of the loop is a question about evidence you mostly don't have. A self-improving agent is only ever as good as its ability to know it was wrong — and on a real personal agent, in a real group chat, that knowledge is scarce, confounded, expensive to buy, and frequently just unavailable.

The Jensen photo is a good test precisely because it is so mundane. No adversary, no exotic capability, no safety cliff — a joke, a competent-but-wrong reply, a correction. If the loop can't close on that, it can't close.

The loop is easy to draw. Every arrow in it is a research problem.

---