Critic and Reviser

Problem

Reflection works when there is an external signal, a schema, a test, a citation check. Some of the things you actually want to catch have no such check.

Does this reply address what the customer asked, or a nearby question? Is the tone right for someone on their third follow-up? Does this cite the current policy or one that reads similarly? No validator produces those findings. A person would catch all three in ten seconds.

The available substitute is another model call with a different job. That is the pattern, and its value depends almost entirely on one design choice that is easy to get wrong.

Forces

Some quality dimensions are judgment, and judgment is what models are for.
A critic and a generator can be given genuinely different inputs, unlike a self-review.
A critic reading only the output is not independent. Its errors correlate with the generator's.
Critics are lenient by default. A model asked whether text is good usually says yes.
Each critique-revise cycle costs two calls, and the revision can introduce new errors.
The critic is itself a model, with its own drift and its own failure modes.

Solution

A second call with a different prompt, a different role, and, critically, different inputs, producing structured findings rather than a verdict.

   GENERATOR                          CRITIC
   ────────────────────────           ──────────────────────────────
   sees: question + retrieved         sees: question + THE SOURCES
         context + tool results             + the draft
   job:  produce the answer           job:  find specific defects
                                            against a rubric
        │                                        ▲
        │  draft ────────────────────────────────┘
        ▼
   ┌────────────────┐   findings (structured, per-claim)
   │    REVISER     │ ◀────────────────────────────────
   │ targeted edits │
   └────────────────┘
        │
        ▼  bounded: 1 cycle by default, 2 at most
   ship · or escalate with the unresolved findings

   ✗ a critic given only the draft is a SECOND VOTE
   ✓ a critic given the sources is a CHECK

The critic reads the sources; the generator read the draft's own context. That asymmetry is where the value is.

Four rules:

Give the critic the sources, not the summary. This is the rule the pattern lives or dies on. A critic reading the draft alone shares every blind spot the generator had; a critic reading the retrieved chunks and tool results can see that the draft says something they do not support. The same argument appears in multi-agent verification and dual control: correlated reviewers do not catch correlated errors.

Demand structured findings, not a score. { claim, problem, evidence } per issue. A number is unactionable and a paragraph is unparseable; a list of specific defects can be counted, fed to the reviser, and asserted against. It also makes leniency visible: a critic returning zero findings on everything is a critic you can detect.

Give the critic a rubric and permission to fail things. Models asked to review are agreeable by default. Name the dimensions, and say plainly that finding nothing is a valid but unusual outcome. Calibrate against known-bad fixtures, a critic that passes your deliberately broken examples is not working.

One cycle by default. Critique, revise, ship. A second cycle when findings remain; after that, escalate with the findings attached rather than iterating.

Code

ts/src/control/critic.ts

const Finding = z.object({
  claim: z.string(),                 // the specific span at issue
  problem: z.enum(['unsupported', 'wrong_version', 'off_question', 'tone', 'omission']),
  evidence: z.string(),              // what in the SOURCES contradicts it
});
const Critique = z.object({ findings: z.array(Finding).max(8) });

export async function critique(draft: string, ctx: RunContext) {
  return model.structured({
    schema: Critique,
    system: CRITIC_PROMPT,           // rubric + explicit permission to fail
    input: {
      question: ctx.question,
      draft,
      // The asymmetry that makes this a check: the critic reads the
      // SOURCES, not the generator's rendering of them.
      sources: await loadChunks(ctx.retrievedIds, ctx),
      toolResults: ctx.scratchpad.toolResults(),
    },
  });
}

export async function criticReviser(draft: string, ctx: RunContext) {
  for (let cycle = 0; cycle < 2; cycle++) {
    const { findings } = await critique(draft, ctx);
    if (findings.length === 0) return { text: draft };

    draft = await model.complete({
      system: REVISE_PROMPT,
      input: { draft, findings },    // targeted; preserves what nobody flagged
    });
  }

  const { findings } = await critique(draft, ctx);
  return findings.length
    ? { text: draft, escalate: findings }   // a person gets the specifics
    : { text: draft };
}

The sources and toolResults lines are the pattern. Remove them and this becomes an expensive intrinsic self-review, which is measured not to help, the model count changed and the information did not.

Trade-offs

Two to four extra model calls per response. Critique, revise, re-critique, possibly again. On a path where the reply matters this is justifiable; across all traffic it is a large multiplier.

Critic leniency is the default failure. Without calibration you get a critic that finds nothing and a system that feels reviewed. Keep a small set of deliberately-broken fixtures in CI and assert the critic catches them, a critic that stops catching them has drifted.

A different prompt is weaker independence than a different model. Same weights, same blind spots, different instructions. Using a different model family for the critic buys more genuine independence and costs operational complexity plus a second drift surface.

The reviser can break what was right. Targeted findings limit the damage. Re-critiquing after revision catches the case where the fix introduced a new problem, and it is why the loop re-checks rather than trusting the revision.

When not to use it

When a deterministic check exists. Schema, citation resolution, amount grounding, use those first. They are free, total, and cannot be lenient. Reserve the critic for what they cannot express.

On high-volume, low-stakes paths. Doubling or tripling cost to polish a status update is a bad trade.

When the critic cannot be given independent inputs. If there are no sources to hand it, a purely generative task with no grounding, the critique is a second sample of the same judgment. Be honest that you are buying variance reduction rather than verification.

As a substitute for evals. A critic is a runtime check on one response and tells you nothing about whether quality is drifting. Sampled scoring is what answers that, and a team watching critic-finding rates is measuring their critic.

The critic is a model, so it needs everything a model needs

It is easy to treat the critic as infrastructure, a check, a filter, a piece of the harness. It is not. It is a second model call with a prompt, a version, and a failure mode.

Which means it belongs in the config bundle with a pinned model and a hashed prompt; it needs its own eval set, scored against known-good and known-bad fixtures; it drifts when the provider changes something underneath it, and a frozen canary is the only thing that will tell you; and its findings are prompt text that the reviser reasons over, so a badly-worded finding produces a badly-targeted revision.

A silently-degraded critic is worse than no critic, because the system continues to report that everything was reviewed.

Reflection: the cheaper version, and why the external signal is the whole thing
Dual Control: the same independence argument, with humans
When Not To: why a reviewer reading a conclusion is a second vote
Evaluator-Optimizer: the chapter, with bounded iteration and convergence
Scoring: rubrics, and calibrating a model that judges

References

Large Language Models Cannot Self-Correct Reasoning Yet, Huang et al., ICLR 2024, the result that makes an independent critic necessary rather than optional.

On this page