Self-Improving Systems

What this lesson is about

A normal language model answers in a single forward pass: prompt in, answer out. A self-improving system wraps that model in a loop so it can look at its own work, judge it, and try again — at inference time, without retraining any weights. The agent drafts, critiques, and refines, ideally producing a better answer than it would on the first try.

This is a different idea from training-time self-improvement (self-supervised learning, self-play, generating your own training data). Here nothing about the model changes — we are just spending more compute and steps at deployment to get a better output from a fixed model. The central question of the whole lesson is: when does looking at your own work actually help, and when does it just spin?

The core loop: draft → critique → refine

Almost every method in this space is the same loop with different pieces swapped in:

The loop is appealing because it mirrors how people work — write a rough draft, reread it, fix the weak parts. But notice the load-bearing component is the critic in step 2. If the critic is right, the loop converges to a good answer. If the critic is wrong, the loop will happily “improve” the answer in the wrong direction.

Self-reflection / self-critique (Reflexion)

In the purest version, the model is its own critic. After an attempt, it is prompted to reflect: “What went wrong? What should I do differently?” Reflexion (Shinn et al., 2023) formalizes this for agents that act over multiple trials: after a failed episode, the agent writes a short verbal lesson (“I should have checked whether the file existed first”) and stores it in memory. On the next attempt, that lesson is added to the context so the agent does not repeat the mistake. Crucially, Reflexion needs a signal that the episode failed — typically an environment reward or a task that succeeds/fails — to know there is anything to reflect on.

Iterative refinement (Self-Refine)

Self-Refine (Madaan et al., 2023) is the same model playing all three roles: it generates, then critiques its own output with feedback, then refines, looping a few times. It helps most on tasks where the feedback is about preferences and style the model can recognize — making a response more polite, more concise, more readable — because recognizing “this is wordy” is genuinely easier than writing concisely on the first pass. It helps least on tasks requiring knowledge or reasoning the model simply does not have, where its self-assessment is no more reliable than its first answer.

External feedback beats self-critique

The most reliable self-improving systems do not rely on the model grading itself. They close the loop with an external, grounded signal — something whose verdict does not depend on the model's own (possibly mistaken) judgment:

• Execution results / test suites — write code, run it, feed the failing tests or stack trace back in. The compiler and the tests are not fooled by a confident-sounding answer.
• Tool and environment errors — an API returns a 400, a SQL query throws, a file is missing. These are objective and immediately actionable.
• Separate verifier models — a model trained specifically to score correctness, or a stronger model checking a weaker one. Useful, but only as trustworthy as the verifier itself.
• Human or reference feedback — ground truth when available, the strongest signal of all.

The reason external feedback works is that it breaks the circularity: the model is no longer the sole judge of its own correctness. This is why the strongest results for “agents that fix their own work” come from coding agents — the test suite is an external oracle. The demo below makes this contrast concrete.

Interactive demo: feedback quality governs improvement

Below, an agent tries to produce a number matching a hidden target. Each iteration is a full draft → critique → refine round. Toggle between a self-critique-only critic (a noisy, biased estimate of its own quality) and an external verifier (a grounded, accurate signal). Run the iterations and watch the score curve. Use New run (reseed) to see how erratic the self-critique runs are.

What to notice: with the verifier, true quality (green) climbs and the critic's reported score (amber) sits right on top of it. With self-critique only, the amber line wanders away from green — the agent is confident about scores that are wrong — and the green curve plateaus low or even dips. Same loop, same number of iterations; only the feedback source changed.

Learning from experience: writing lessons to memory

The loops above improve a single answer. A complementary kind of self-improvement happens across tasks: after each attempt, the agent distills what it learned into a short, reusable note and appends it to an external memory store. On future tasks, relevant notes are retrieved and pasted into the prompt. The agent gets better over time not because its weights changed, but because its context accumulates hard-won lessons.

This is exactly Reflexion's verbal-feedback memory, generalized: a growing scratchpad of “things that went wrong and how to avoid them.” It is cheap, interpretable, and editable. Its limits are the usual ones for retrieval — the right lesson has to actually be retrieved at the right moment, and a wrong lesson written to memory will be confidently repeated.

Recursive self-improvement — and its real limits

The dramatic version of this idea is recursive self-improvement: a system that improves its own ability to improve, bootstrapping toward ever-higher capability. It is worth being clear-eyed about what is and is not happening in the methods above. Inference-time loops do not increase the model's underlying capability — they spend extra compute to extract a better answer from a fixed model. There are hard ceilings:

• A model cannot reliably critique beyond its own competence. If it could recognize the correct answer, it could usually produce it. When verification is no easier than generation, self-critique adds noise, not signal.
• Reward hacking. Optimizing against an imperfect critic teaches the agent to satisfy the critic, not the goal — e.g. writing code that passes the visible tests by special-casing them, or phrasing answers to sound confident to a sycophantic judge.
• Reinforcing its own errors. A wrong self-assessment can send refinement the wrong way, and because the model then defends that change, errors can compound across iterations.
• Degeneration / collapse. Loops driven only by self-generated signals can drift — losing diversity, amplifying artifacts, or converging on confident nonsense. More iterations make this worse, not better.

None of this means self-improvement is fake — it means the gains come from grounding. Genuine recursive improvement requires a source of feedback the system cannot simply talk its way around: real execution, real environments, real tests, or real human judgment. Remove the ground truth and the loop optimizes its own reflection instead of the task.

When does self-improvement help?

Key Takeaways

• Self-improving systems wrap a fixed model in a draft → critique → refine loop at inference time; no weights change.
• The loop is only as good as its critic. Feedback quality — not iteration count — governs whether quality improves.
• Self-critique (Reflexion, Self-Refine) helps when verifying is easier than generating (style, format, catching obvious errors) and stores verbal lessons in memory to avoid repeating failures.
• External feedback — test results, execution errors, verifier models, human judgment — is far more reliable because it breaks the circularity of the model judging itself.
• A model cannot reliably critique beyond its own capability; pure self-loops risk reward hacking, reinforcing their own errors, and degeneration.
• Genuine recursive self-improvement needs grounding the system cannot talk around — real execution, environments, tests, or people.