Golden Set

Problem

Someone changes a prompt. Is it better?

The honest answer, without a dataset, is that nobody knows. The available evidence is that the person who wrote it tried four examples and liked the results, which is the same four examples that motivated the change, so of course it does better on them.

The failure mode is not that changes ship untested. It is that they ship tested against the case that prompted them, so every change is an improvement on its own motivating example and a coin flip on everything else. Six months of that produces a prompt nobody can modify, because nobody can say what it currently handles.

Forces

• You cannot write expect(reply).toBe(...). The output is not deterministic.
• A number only means something comparatively, against yesterday's number, on the same set.
• Curation is expensive, so the set must be small enough to maintain and large enough to detect a real change.
• The set must resist overfitting, which is what happens when it becomes the target.
• Fixtures go stale as the corpus, the policy, and the product change.
• Failure cases are worth more than success cases, and are the ones nobody writes.

Solution

A small, curated, versioned set of labeled cases with a stated expected outcome, run on every change, reported as a distribution.

   GOLDEN SET  ~250 cases
   ┌─────────────────────────────────────────────────────────────┐
   │ 150  hand-labeled from real tickets                         │
   │      ── distribution matches production, deliberately       │
   │                                                             │
   │  40  ADVERSARIAL                                            │
   │      ── injection payloads, out-of-scope, multi-part        │
   │                                                             │
   │  30  NEGATIVE — must return nothing                         │
   │      ── the rare DETERMINISTIC assertion in this field      │
   │                                                             │
   │  30+ PROMOTED FROM PRODUCTION FAILURES  ◀── grows on its own│
   │      ── each one a bug that actually happened, replayable   │
   └─────────────────────────────────────────────────────────────┘

   each case:  { input, expected_outcome, expected_source,
                 must_escalate?, must_not_retrieve?, notes }

   INVARIANTS gate with no threshold · RATES report as a distribution

Four rules:

Write the expected outcome, not the expected text. For each case: the answer, the source it must come from, the action it should take, or the fact that it must escalate. Text comparison is meaningless; outcome comparison is checkable.

Include cases the agent should fail. Out-of-scope requests, ambiguous tickets, things that must escalate. A set of only-successes measures how well the agent does the easy thing, and escalation recall is the metric held highest precisely because missing one is the expensive error.

Promote every production failure. Replay turns an incident into a deterministic fixture. This is the source that compounds: the set grows at exactly the rate the system breaks, which beats any sampling strategy designed up front.

Separate what gates from what reports. Invariants, such as no cross-account data, no credit above cap, no uncited policy claim, fail the build with no threshold and no discussion. Rates are noisy on small samples, so they report as a distribution against a flake budget; gating a noisy metric on one run produces a suite people re-run until it passes.

Code

export interface GoldenCase {
  id: string;
  input: TicketInput;
  // The expected OUTCOME, never expected text.
  expect: {
    outcome: 'resolved' | 'escalated';
    source?: string;              // which document/tool must ground it
    mustEscalate?: boolean;
    mustNotRetrieve?: string[];   // negative: these ids must never appear
    coverage?: 'complete' | 'partial';
  };
  tags: ('real' | 'adversarial' | 'negative' | 'promoted')[];
  addedAt: string;
  provenance?: { runId: string; incident: string };   // for promoted cases
}

export async function runGolden(set: GoldenCase[], bundle: ConfigBundle) {
  const results = await Promise.all(set.map(c => runCase(c, bundle)));

  // INVARIANTS: no threshold, no discussion. Any violation fails the build.
  const violations = results.flatMap(r => r.invariantViolations);
  if (violations.length) throw new InvariantViolation(violations);

  // RATES: a distribution against a flake budget. Gating a noisy metric on
  // a single run produces a suite people re-run until it passes.
  return {
    resolution: rate(results, r => r.outcome === 'resolved'),
    escalationRecall: rate(
      results.filter(r => r.case.expect.mustEscalate),
      r => r.outcome === 'escalated',
    ),
    correctSource: rate(results, r => r.source === r.case.expect.source),
    byTag: groupBy(results, r => r.case.tags[0]),   // adversarial vs real
  };
}

mustNotRetrieve is the field worth copying. This document ID never appears in the retrieval trace for this principal is a hard, deterministic assertion, one of very few available in a probabilistic system, and you should take every one this field offers.

Trade-offs

Curation is the cost, and it is ongoing. 250 labeled cases is days of work, and they decay: policies change, the corpus is re-indexed, the product scope moves. Budget review time, and delete fixtures whose expected outcome is no longer correct rather than editing them until they pass.

Small sets are noisy. Detecting a three-point change on 250 cases at meaningful confidence is not always possible. That is why rates report rather than gate, and why the canary on real traffic exists: CI has the labels and production has the sample size.

Overfitting is real and quiet. A set used long enough becomes the target, and the agent gets better at the set without getting better. Keep a holdout slice that is never used for tuning, and rotate promoted cases in continuously so the set is never static.

Promoted cases carry customer data. Scrub and pseudonymize at extraction: the eval-fixture row in the compliance table is exactly this pipeline, and fixtures outlive the system they came from.

When not to use it

Never, once anything ships. A system without one cannot answer whether a change helped, which means every change is a guess. The genuine question is when to build it.

Before the problem is understood. Writing twenty cases is the cheapest available test of whether you understand the task, and failing to write them is the signal, not a reason to skip ahead.

The real limits are about scope:

Do not gate on rates too early. In the first weeks the numbers move for reasons unrelated to your changes. Report them, gate on invariants only, and start gating on rates when the baseline is stable.

Do not use it to measure production quality. A curated set measures your changes against a fixed reference. What real traffic is doing needs online scoring on a sample.

Related

• CI for Agents: set sizing, flake budgets, and what a gate actually needs
• Trajectory Assertion: asserting on the path, not only the outcome
• Judge With Rubric: scoring the cases whose expected outcome is not mechanical
• Replay-Driven Debugging: how a production failure becomes a fixture
• Canary Eval: the sample size CI cannot afford