Judge With Rubric

Problem

Some things a golden set needs to check have no mechanical answer. Does the reply address what the customer actually asked? Is the tone right for someone on their third follow-up? Is the explanation complete enough to act on?

The available substitute is a model scoring the output. The version everyone builds first is:

   "Rate this support reply from 1 to 10."

It produces a number, the number moves between runs on identical input, and nobody can say what a 7 means or why yesterday's 8 is today's 6. The score is not wrong so much as uninterpretable: it cannot gate anything, cannot be debugged, and cannot be compared across a prompt change.

Forces

• Judgment dimensions are real and no deterministic check expresses them.
• A judge is a model, with the same nondeterminism as the thing it is judging.
• Judges have documented, specific biases: position, verbosity, and self-preference.
• Reproducibility is the requirement. A score that moves on identical input measures nothing.
• Free-form scoring hides disagreement inside a single number.
• A judge that is never calibrated drifts without anyone noticing.

Solution

Replace the score with a rubric of independent binary criteria, each with an explicit definition and an evidence requirement.

   ✗ "rate 1–10"                    ✓ RUBRIC
      → 7                              addresses_question      ✓
      → what changed?                  cites_current_policy    ✗  ← the finding
      → nobody knows                   amount_matches_tools    ✓
                                       states_next_step        ✓
                                       no_unsupported_claims   ✓
                                       tone_appropriate        ✓
                                       ────────────────────────────
                                       5/6 · and you know WHICH one

   each criterion carries:
     · a definition ("cites the version marked current at run time")
     · required evidence (the span, the chunk id)
     · a worked positive and negative example

   ORDER is randomized per call     ── position bias
   LENGTH is not a criterion        ── verbosity bias
   JUDGE ≠ GENERATOR where possible ── self-preference bias

Four rules:

Binary criteria, never a scale. Does the reply cite the policy version marked current at run time is answerable and checkable. How good is the citation, 1 to 5 is not, and the middle of any scale is where disagreement hides.

Define each criterion in the rubric, with examples. A criterion named tone_appropriate with no definition is the 1-to-10 problem with extra steps. One sentence of definition plus a worked positive and negative example is what makes two runs agree.

Require evidence per criterion. The judge returns the span it based each verdict on. That makes a wrong verdict debuggable rather than mysterious, and it is what lets you calibrate: you can see whether the judge failed the criterion or misread the text.

Mitigate the known biases structurally. Randomize criterion order per call. Never make length a criterion or let it correlate with one. Where the stakes justify it, use a different model family for the judge than for the generator.

Code

// Binary criteria with definitions. The definition IS the rubric.
const CRITERIA = [
  { id: 'addresses_question',
    definition: 'Every question the customer asked receives a direct answer or an explicit statement that it is out of scope.' },
  { id: 'cites_current_policy',
    definition: 'Any policy claim cites a chunk whose version was marked current at run time.' },
  { id: 'amount_matches_tools',
    definition: 'Every monetary figure appears in a tool result from this run.' },
  { id: 'states_next_step',
    definition: 'The reply says what happens next and who is doing it.' },
  { id: 'no_unsupported_claims',
    definition: 'No factual assertion lacks a retrieved source or tool result.' },
  { id: 'tone_appropriate',
    definition: 'Neither dismissive nor over-apologetic; matches the customer’s register.' },
] as const;

const Verdict = z.object({
  results: z.array(z.object({
    id: z.enum(CRITERIA.map(c => c.id) as [string, ...string[]]),
    pass: z.boolean(),
    evidence: z.string(),      // the span the verdict is based on
  })).length(CRITERIA.length),
});

export async function judge(reply: string, ctx: RunContext, seed: number) {
  const out = await model.structured({
    model: JUDGE_MODEL,          // pinned; ideally a different family
    temperature: 0,
    schema: Verdict,
    system: renderRubric(shuffle(CRITERIA, seed)),   // order randomized: position bias
    input: {
      question: ctx.question,
      reply,
      // The judge reads the SOURCES, not the generator's rendering of them.
      sources: await loadChunks(ctx.retrievedIds, ctx),
      toolResults: ctx.scratchpad.toolResults(),
      // Length is deliberately NOT provided and NOT a criterion.
    },
  });
  return { passed: out.results.filter(r => r.pass).length, results: out.results };
}

Two criteria in that list, amount_matches_tools and no_unsupported_claims, are also computable deterministically. Keep them in the rubric anyway: they are your calibration signal. When the judge disagrees with the mechanical check, the judge is wrong, and the rate of that disagreement is the cheapest ongoing measure of judge health you can get.

Trade-offs

Cost and latency per judged item. Six criteria in one structured call is one model pass, which is why they go in one call rather than six. On a golden set that runs on every commit this is a real per-build cost.

Calibration is required, not optional. A judge is only useful if it agrees with people on cases where people agree with each other. Score a few dozen items by hand, compare, and keep them as a calibration set. Reported practice puts strong judges above 80% agreement with human annotators, which is roughly the level humans achieve with each other, and is therefore the ceiling to aim at rather than a number to be disappointed by.

Judges drift. A pinned model still changes behaviour, so a moving score on a stable system is a judge problem until proven otherwise. The frozen canary should include judge fixtures, not just agent fixtures.

Rubric changes invalidate history. Adding a criterion changes what the score means, so trend lines break. Version the rubric with the config bundle and report per-criterion rates rather than only the total, so one added criterion does not erase the series.

When not to use it

When a deterministic check exists. Citation resolution, amount grounding, schema validity, forbidden retrievals. Those are free, total, and cannot be lenient. A judge is for what they cannot express.

As a gate on an interactive path. A judge is an eval, not a guardrail. It samples and reports; it does not sit between the agent and the customer.

Without calibration data. An uncalibrated judge produces numbers that feel like measurement. If there is no budget to hand-score a calibration set, be honest that the output is a signal rather than a score.

For preference between two outputs, without care. Pairwise comparison is where position bias is strongest. If you must, run both orderings and count only the cases where the verdict is consistent.

Related

• Golden Set: the cases the judge scores
• Trajectory Assertion: the deterministic checks to exhaust first
• Scoring: the chapter, including rubric design and inter-rater agreement
• Critic and Reviser: the same independence argument, applied at runtime rather than in evals
• Online Guardrail: why a judge is not a gate

References

• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, Zheng et al.: the judge biases a rubric has to be written against.