Reflexion & Self-Correction

Reflexion, introduced by Noah Shinn and colleagues in March 2023, is the idea that an agent can learn from its own mistakes within a task, with no retraining. After a failed attempt, the model reflects in plain language on what went wrong, writes that reflection into agent memory, and tries again with the critique in context. The reinforcement signal is verbal, not a weight update, which is why the paper calls it verbal reinforcement learning. More broadly, self-correction is any verify-then-revise loop: generate a candidate, check it, turn the failure into specific feedback, and revise. The pattern is cheap to implement and genuinely powerful, but only under conditions worth being precise about.

The attempt, critique, retry loop#

A self-correction loop has three moves. Attempt: the agent produces a candidate output, say a function that should pass a test suite. Critique: the candidate is checked, ideally against something external (run the tests, compile the code, validate the JSON), and the failure is converted into a written diagnosis: "attempt 1 failed test_empty_input because the function indexes into an empty list; next time guard the empty case." Retry: that diagnosis goes into the prompt for attempt 2, so the model is no longer guessing blindly; it is responding to a concrete description of its own last error. A success gate ends the loop when the output passes, and a retry cap ends it when it will not.

Why it improves hard multi-step tasks#

On multi-step problems, the first attempt usually fails on a detail the model could recognize in hindsight: a missed edge case, an off-by-one, an unsupported claim. Critiquing a finished artifact is an easier job than producing a perfect one in a single forward pass, the same asymmetry that makes code review cheaper than writing the code. Self-correction exploits that gap, and unlike sampling five independent attempts, each retry is conditioned on a diagnosis of the previous failure, so attempts improve instead of just varying.

The numbers from 2023 still frame the field. Reflexion reached 91% pass@1 on the HumanEval coding benchmark when GPT-4 alone scored 80%, and on the ALFWorld decision-making benchmark the Reflexion-augmented ReAct agent completed 130 of 134 tasks across retries, a 22 point improvement over the non-reflective baseline. Self-Refine, a sibling pattern where one model drafts, critiques, and revises within a single attempt, averaged roughly 20 points of absolute improvement across seven tasks, even on GPT-4. The common thread: every headline result had either an external signal (unit tests, an environment saying "task failed") or a task like dialogue quality where critique is genuinely easier than generation.

Where self-correction goes wrong#

The failure modes are well documented and worth taking seriously. Huang and colleagues showed in "Large Language Models Cannot Self-Correct Reasoning Yet" (ICLR 2024) that intrinsic self-correction, where the model grades its own reasoning with no external feedback, often makes answers worse: prompted to "review your answer," models flip correct answers to incorrect ones at meaningful rates, because the critique comes from the same distribution that produced the error. A model that confused two API versions in its attempt will confuse them in its self-review too.

Two operational failure modes follow. Overconfident self-grading: the model declares a wrong answer correct and exits the loop early, which is worse than no loop because the retry budget bought false confidence. Looping: with no reliable stop signal, the model keeps "improving" output that was already fine, oscillating between two variants while burning tokens; a 5-retry loop multiplies cost per task by up to 6x, so an unanchored loop is an expensive way to get a slightly different wrong answer. Self-grading shares the known biases of LLM-as-judge setups, with the extra twist that the judge authored the work it is grading.

Anchor the loop in external verification#

The fix is to make the verify step something the model cannot talk its way past. Code is the best case: run the test suite, the compiler, the type checker, and feed the literal error output back as the critique. A failing traceback is ground truth; "I believe this code is correct" is not. The same principle generalizes: validate structured output against a schema, check extracted numbers against the source document, re-run a SQL query against a sandbox, verify a cited URL resolves. The CRITIC line of work made the point sharply: with tool-grounded feedback, self-correction gains hold up; remove the tools and the gains shrink toward zero or reverse.

A practical recipe for a coding agent: attempt, run tests, on failure prepend "previous attempt failed with: [error]" plus a one-line model-written reflection, retry, cap at 3 to 5 attempts, and escalate to a human or a fallback path on exhaustion. Most of the value arrives in the first retry; returns diminish fast after the second. Track pass rate by attempt number in your evals so you know what the loop actually buys you, and watch for tasks where attempt 3 does worse than attempt 1, the signature of an unanchored critique.

Practical takeaways#

Use self-correction wherever a hard external check exists: tests, compilers, schemas, validators. Treat the model's unaided self-grade as a hint, never a gate; Huang et al. is the citation to keep handy when someone proposes "just ask it to double-check." Cap retries at 3 to 5, log every reflection for debugging, and measure improvement per retry rather than assuming it. The pattern composes naturally with ReAct loops, where it turns a failed trajectory into a lesson for the next one. Verbal feedback is the cheapest learning signal in the stack, but only verification makes it trustworthy.