The Operating Manual

At 03:12 the pager goes off. The alert says atlas: quality_score p50 down 4.1 over 45m (route=refund, bundle=7f2a91).

Nothing is down. Latency is normal, the error rate is zero, every pod is healthy, and the provider status page is green. The person holding the pager has to decide, at 03:12, whether this is a real regression, a slow news day for the sampler, or a thing that will look obvious at 09:00 and does not need anyone awake now.

That decision is the defining skill of operating an agentic system: is this broken, or is this the tail? No dashboard answers it by itself. This chapter is about building the ones that get close.

Three tiers, and your failures live in the third

   ┌───────────────────────────────────────────────────────────┐
   │  IS IT UP?        availability, latency, error rate,      │
   │                   saturation. every APM ships this        │
   ├───────────────────────────────────────────────────────────┤
   │  IS IT WORKING?   runs started vs. terminal, escalation   │
   │                   rate, turn count, cost per run,         │
   │                   tool error rate, queue age              │
   ├───────────────────────────────────────────────────────────┤
   │  IS IT RIGHT?     sampled quality score, grounding rate,  │
   │                   format compliance, resolution rate,     │
   │                   human override rate                     │
   │                   ── nothing in tier 1 or 2 moves when    │
   │                      this breaks                          │
   └───────────────────────────────────────────────────────────┘

The whole reason this chapter is not a generic SRE chapter is the bottom row. An agent feature can degrade steadily for six hours with every conventional SLO green, because the failure is behavioral rather than functional: the model became subtly more cautious, or more agreeable, or started omitting a field. Requests succeed. Latency is fine. The product is worse.

Which produces the rule that should govern the whole dashboard:

If your alerting can only see tiers 1 and 2, you are monitoring the availability of a system whose characteristic failure is being wrong while available.

Tier 2 is the useful middle. Those metrics move in minutes, cost nothing to compute, and are proxies for tier 3: an agent taking more turns, escalating more, or spending more per run is usually telling you something about quality before the quality score has enough samples to say it. Alert on tier 2, confirm with tier 3.

The alert set

Fewer alerts than you think, and the third one is the one people do not have.

The bolded row is the one worth copying precisely. Reported practice converges on a sustained drop of a few points over fifteen to sixty minutes, scored per route and per bundle version, and each qualifier is doing work:

• Sustained filters sampling noise, which on a small window is enormous.
• Per route stops a regression in one workflow from being averaged into invisibility by nine healthy ones.
• Per bundle version makes the rollout attribution automatic: the alert arrives already saying which configuration is responsible.

Two things deliberately absent: no alert on a single bad output, because in a probabilistic system one bad output is the expected tail and paging on it trains people to ignore the pager. And no alert on total spend, which is a lagging monthly artifact. Tokens per minute is the leading indicator.

Incident patterns

Most agent incidents are one of about eight things. Recognizing which, quickly, is most of the response.

The single highest-value habit is that the first question in every one of these is "what changed", and it should be answerable in one query across code deploys, config repoints, index builds, and provider incidents. A timeline that only shows code deploys will send you looking for a bug that was a re-index, which is why config changes have to be deploy events.

On-call for a probabilistic system

Four things are genuinely different, and each surprises someone coming from ordinary SRE.

You often cannot reproduce it. Running the failing input again gives a different answer. The substitute for reproduction is replay: the recorded run is deterministic even though the system is not, and "replay it" should be the first line of every runbook rather than "run it again."

The fix is usually not a deploy. A repointed manifest, a lowered k, a disabled fallback rung, a raised cap. This is good, it makes mitigation fast, and it means the config-change path needs to be safe at 3am, which is the reviewed-and-eval-gated discipline paying off precisely when someone wants to skip it.

Distinguishing a regression from noise is the core skill, and it needs a baseline distribution. Not a number, a distribution. "p50 quality is 4 points down" means nothing without knowing that the historical 45-minute-window standard deviation is 1.2 points. Publish the baseline variance next to the metric on the dashboard, or every responder re-derives it under pressure and half of them get it wrong.

Degradation is a legitimate mitigation. Unlike most systems, you can turn quality down to keep working: narrow the scope, drop to a cheaper rung, or route more to humans. Deciding in advance which lever to pull, and pre-approving it, is what keeps an incident from becoming a debate at 03:20.

The first ninety days

The order matters more than the contents, because each stage makes the next one measurable.

Days 1–14 · See it. Tracing with the eight questions answerable, config_hash on every run, cost attribution at the gateway, and a runs table with terminal states. Ship nothing else until a single run can be reconstructed.

Days 15–30 · Bound it. Run budget with a degrade rung, turn cap, risk tiers with tier 1+ gated, argument scoping on every write, and the dispatcher choke point if it is not already there. This is where the blast radius stops being unbounded.

Days 31–60 · Measure it. A CI eval suite with a real dataset, a frozen canary on a schedule, online scoring on a sample, and the tier-2 proxy metrics on a dashboard with their baseline variance. Now a change has a number.

Days 61–90 · Break it. Failure injection in CI, the fallback path evaluated as its own variant, a first game day, and the incident-pattern runbook written from what you actually saw in the first two months.

The sequence is deliberately the opposite of how these systems usually get built: capability first, legibility later. The reference architecture chapter named the reason: capability demos well and legibility does not, so capability gets funded first. Ninety days spent the other way round produces a system that is less impressive in month two and still operable in month twelve.

Where Part XVIII leaves you

Five chapters on everything that has to be true before you point users at it.

An architecture sized by two questions rather than by ambition. Versioning of the resolved bundle, because the git SHA does not identify what the model saw. Rollout that pins configuration per run and gates on blast radius rather than percentage. Cost engineering that starts with turn count rather than model choice. And an operating posture that watches the tier nothing else watches.

The through-line of the part: production readiness for an agent is mostly the ability to answer questions about it. What version, which tenant, how much, which chunk, whose authority, what changed. Every chapter here is a mechanism for making one of those answerable, and the systems that operate badly are almost never the ones with the worst models.

Part XIX asks whether you should have built one agent or several, and, having spent twenty parts on how much machinery one agent needs, the answer is less obvious than the diagrams suggest.

Takeaways

• Three dashboard tiers: is it up, is it working, is it right. Conventional monitoring covers the first two completely and the third not at all.
• An agent feature can degrade for six hours with every SLO green, because the failure is behavioral rather than functional.
• Tier-2 metrics: turn count, escalation rate, cost per run, queue age. They move in minutes and proxy for quality. Alert on tier 2, confirm with tier 3.
• The quality alert that works: a sustained drop of a few points over 15–60 minutes, scored per route and per bundle version. Sustained filters noise; per-route stops averaging; per-bundle makes attribution automatic.
• Never alert on a single bad output. In a probabilistic system that is the expected tail, and paging on it teaches people to ignore the pager.
• Never alert on total spend. It is a monthly lagging artifact. Tokens per minute is the leading indicator.
• Roughly eight incident patterns cover most of it. Recognizing which, fast, is most of the response.
• The first question in every incident is "what changed," and it must be answerable in one query spanning code deploys, config repoints, index builds, and provider events.
• You often cannot reproduce an agent failure. Replay is the substitute, and "replay it" belongs at the top of every runbook.
• The fix is usually a config change, not a deploy, which is why the config path has to be safe at 3am.
• Distinguishing regression from noise needs a baseline distribution, not a number. Publish the historical variance next to the metric.
• Degradation is a legitimate mitigation. Decide and pre-approve which lever to pull before the incident, not during it.
• Write runbooks as typed graphs with explicit branches. The benefit is not automation: it is that nothing is ambiguous at 3am.
• Ninety days: see it, bound it, measure it, break it. The opposite order to how these systems usually get built.
• Production readiness for an agent is mostly the ability to answer questions about it. The systems that operate badly are rarely the ones with the worst models.

Everything in Part XVIII so far describes a system already carrying traffic. Next: The Go-Live Readiness Review, the gate it should have passed first, and the evidence each answer needs behind it.