Learning From Production Feedback

Production gives you more data than any lab and fewer clean labels than it appears to. A thumbs-down can mean the answer was wrong, too long, rude, slow, or correct but unwelcome. A human rewrite can fix tone while preserving the same factual error. A customer who quietly leaves provides the strongest signal and no explanation.

Feedback is evidence about an interaction. It becomes a label only after someone defines what the signal means.

Signals are not labels

The lesson is not to ignore weak signals. It is to store them with enough context to interpret later and never feed them directly into training or a golden set.

The feedback event

Keep the raw event immutable. Interpretation can change as the rubric changes.

export interface FeedbackEvent {
  id: string;
  runId: string;
  turnId?: string;
  actor: { type: 'customer' | 'reviewer' | 'system'; id?: string };
  kind: 'rating' | 'edit' | 'override' | 'reopen' | 'complaint' | 'outcome';
  value: unknown;
  occurredAt: string;
  exposure: { bundle: string; route: string; tenantId: string };
  source: string;
}

export interface Adjudication {
  feedbackId: string;
  rubricVersion: string;
  labels: string[];
  severity: 'none' | 'minor' | 'major' | 'critical';
  notes: string;
  reviewerId: string;
}

Separate actor from subject. A manager correcting a reply about a customer is the feedback actor; the customer remains the data subject. That distinction matters when the actor leaves the company or the subject requests erasure.

Join feedback to the run

A feedback event without the resolved bundle, retrieved evidence, tool trajectory, and final state is a complaint you cannot diagnose.

feedback event
      │ run_id
      ▼
run record
├── model and prompt bundle
├── input and context manifest
├── retrieval IDs and versions
├── tools, arguments, and results
├── policy and authority decisions
├── latency and cost
└── final business outcome

The join must survive compaction, Continue-As-New, and a UI reconnect. Use the stable run ID from What a Trace Must Answer, not a browser session ID.

Triage before labeling

Route feedback by potential harm and information value.

1. Immediate incident. Possible data leak, unauthorized effect, financial loss, or repeated dangerous behavior. Contain first; label later.
2. Deterministic defect. Wrong filter, stale version, broken unit conversion, missing idempotency. Open an engineering issue and add the lowest-layer regression test.
3. Behavioral failure. Poor answer, wrong tool choice, unnecessary escalation, incomplete reasoning. Candidate for the eval dataset.
4. Preference. Tone, format, verbosity, workflow friction. Product evidence, not necessarily a correctness case.
5. No defect found. Keep the event. The absence of a confirmed defect is a label too.

This triage prevents a prompt tweak from hiding an authorization bug and prevents every stylistic preference from bloating the golden set.

Corrections need a diff with reasons

Store the original, edited version, and a structured reason:

factual_correction
missing_evidence
wrong_action
policy_override
tone_or_style
too_long
too_short
changed_after_new_information
reviewer_preference

"Changed" is not a useful label. The reason decides whether the case belongs in retrieval, tool design, prompt behavior, policy, or nowhere.

When reviewers disagree, preserve both labels and send the case back to the rubric. Building a Dataset already established that disagreement often reveals an undefined success criterion. Do not erase that finding by picking the senior reviewer automatically.

The promotion pipeline

raw signal
    ↓
join to run and outcome
    ↓
privacy scrub + deduplication
    ↓
triage
    ↓
adjudication against versioned rubric
    ↓
candidate set
    ↓
review for representativeness and leakage
    ↓
golden set · report set · incident fixture · discard

Promotion requires:

• enough input and context to reproduce the task;
• an expected outcome that does not depend on hidden reviewer knowledge;
• source evidence and its version;
• tenant-safe pseudonymization;
• a stated slice and failure class;
• no duplicate or near-duplicate already dominating the set;
• a rubric version and reviewer record.

An incident fixture may preserve more internal detail than a portable golden case. Keep those stores separate. The fixture is for replay under restricted access; the golden case is for repeated evaluation and may reach CI workers and developer machines.

Sampling is part of the product

Feedback arrives from the users most willing to provide it, on the interactions where the control was visible, and after the outcomes that were obvious quickly. That is not the production distribution.

Use deliberate sampling alongside volunteered feedback:

• random baseline sample;
• all high-risk actions;
• escalations and human overrides;
• retries, loops, and cost outliers;
• new model, prompt, tool, or index versions;
• underrepresented tenants, languages, and routes;
• delayed outcomes such as reopened cases and reversed credits.

Track the denominator. "Twelve complaints" is uninterpretable without how many eligible interactions occurred and which users were offered the feedback control.

The NIST report on monitoring deployed AI systems identifies human-AI feedback loops and scaling human validation as open monitoring problems. That is the honest status. The mechanisms here create auditable evidence, not a magic unbiased data flywheel.

Feedback can be attacked

An attacker can flood negative ratings to force a rollback, insert secrets into free-text feedback, or craft corrections that later enter a training set. An internal reviewer can also encode policy changes through edits without authority to change policy.

Controls:

• authenticate the actor and record their role;
• rate-limit and deduplicate events;
• strip or quarantine secrets before dataset promotion;
• separate policy override from model correction;
• require a second reviewer for security-critical labels;
• never auto-train or auto-deploy from feedback;
• monitor label distribution by actor and tenant;
• retain raw evidence so suspicious adjudications can be audited.

The feedback store is untrusted input with a slower path to production. Treating it as trusted because a human typed it is the same mistake as trusting a retrieved document because an employee uploaded it.

Close the loop through releases

Improvement follows the same release path as any other change:

labeled failure
    ↓
root cause and lowest-layer fix
    ↓
offline regression
    ↓
shadow / canary
    ↓
production monitoring
    ↓
did the target slice improve without moving another one?

Record which feedback IDs motivated a change. After rollout, compare the affected slice and inspect for displaced failures. A prompt that reduces "too long" complaints may increase missing-evidence errors. Without lineage from signal to change to metric, the team remembers only that the prompt looked better in review.

Atlas, concretely

Atlas captures ratings, reviewer edits, approval overrides, reopened tickets, credit reversals, and complaints. Every event joins to a run and resolved bundle. Security and financial signals page the owner; behavioral cases enter a weekly adjudication queue.

No raw feedback enters the golden set. Two reviewers label high-risk cases. Promoted examples keep source IDs and versions, then the release manifest records which examples motivated the change. The dashboard reports feedback rate with its exposure denominator and slices it by route, bundle, and tenant.

References

• Challenges in Monitoring Deployed AI Systems, NIST, the monitoring gaps this chapter is a response to.

Takeaways

• Feedback is evidence about an interaction. It is not a label until interpreted against a rubric.
• Store raw events immutably and keep adjudication separate.
• Join every event to the resolved bundle, evidence, trajectory, policy decisions, and outcome.
• Triage incidents and deterministic defects before sending behavioral cases to an eval queue.
• Store edits with a reason. "Changed" does not identify the layer that failed.
• Promote cases through privacy review, deduplication, adjudication, and representativeness checks.
• Track exposure denominators and sample deliberately. Volunteered feedback is not the production distribution.
• Feedback can be poisoned. Authenticate actors, monitor distributions, and never auto-deploy from it.
• Link each change back to motivating cases, then verify the target slice improved after release.

Atlas is measured now: a golden set, a scoring ladder, gates that can stop a release, and production traffic feeding both. Every one of them can tell you a run was wrong without telling you where. Next: What a Trace Must Answer, the run record that makes every feedback event diagnosable.