Evaluating the Path, Not Just the Answer

Atlas resolves ticket #8823 correctly. It also queried the warehouse to look up a single order, took seven steps where three would do, and silently ignored a tool error on the way. An answer-only score records: pass.

That is not a scoring bug. It is the limit of scoring answers: it cannot distinguish a system that works from one that got lucky, and a lucky path is one input away from an unlucky one. Sixty percent of production failures are not the model. They are retrieval, context, and tool behaviour, which is to say they live in the path.

The trap on the other side

State the counterweight immediately, because trajectory evals fail in a specific and self-inflicted way.

The entire reason to build an agent is that it chooses the sequence after seeing each result. A trajectory eval that demands one exact sequence has re-specified a fixed workflow and is now failing the agent for doing the thing you built it to do. And if the sequence really is fixed, Part VI already told you it should be a workflow rather than an agent.

So the design problem is precise:

Score the path without specifying it.

Four strictnesses, and when each is correct

Exact match returns 1 or 0 on an identical sequence, and the discipline for using it is one distinction:

Exact match belongs where order is a policy, never where order is an artifact.

verify_identity before issue_credit is policy in any system carrying both. Reversing it is a compliance failure regardless of outcome. Atlas's own version is the approval gate: the effect may not precede the decision, and a run that inverts them is wrong even when the amount is right. get_order before crm_account_risk_profile is an artifact of how this particular run happened to proceed, and demanding it penalises a model that found a better route.

Precision and recall carry the rest, and they measure different failures with different costs, the same asymmetry retrieval had:

Low recall means a required lookup never happened, so the answer was composed from incomplete data. It is the dangerous one, because it produces a fluent, confident, wrong reply.

Low precision means calls that were not needed. It costs money, latency, and context that is rented for the rest of the run. Expensive, and visible.

Optimise recall first. Precision failures show up on a cost dashboard; recall failures show up in a customer complaint.

Step efficiency

The ratio of minimum viable steps to steps actually taken, normalised per task class. A task needing three tool calls that took seven scores about 43%.

Two cautions make it usable rather than misleading.

The minimum is a judgement, not a fact. Someone has to decide that this ticket needs three calls, and that decision belongs in the golden set alongside the expected answer, which means it gets the same annotator-agreement discipline as everything else.

Normalise per task class, or the number is meaningless. A two-step order lookup and a nine-step account investigation averaged together produce a figure that describes neither. This is the per-route rule again, arriving in a metric.

Alongside it, the simpler and more actionable measure: the redundant call rate, the share of tool calls that contributed nothing to the outcome. Duplicated lookups, calls whose results were never referenced, a second query for something already in context. That number tends to be embarrassing the first time you compute it, and tool consolidation attacks it directly.

Recovery is the part nobody scores

Everything above measures a run that went well. The more informative question is what happens when something breaks, and it is almost never in an eval set, because production traffic mostly does not fail on demand.

Four measurable behaviours:

Loop escape is the branch-selection argument from Part VIII arriving as a number. That chapter said a repeated identical (tool, arguments, error) triple is a control-flow signal rather than a message problem; here it is a score, and a low one points at error text that selects no branch.

The construction requirement follows, and it is a recipe rather than a principle:

Inject failures deliberately for every tool in the schema: timeouts, 4xx, 5xx, and malformed payloads.

You cannot evaluate recovery on traffic that did not fail. This is also the one place where synthesis is unambiguously the right source: a fault-injection matrix is coverage of a space, not fidelity to a distribution, and no amount of production sampling will give you a malformed payload from every tool.

What the path catches that the answer does not

   FAILURE                                  CAUGHT BY
   ──────────────────────────────────       ─────────────────────
   Answered from incomplete data            recall
   Queried the warehouse for one order      precision · redundancy
   Took seven steps to do three             step efficiency
   Ignored a tool error and guessed         adaptation
   Retried the same bad call five times     loop escape
   Refunded before verifying identity       exact match (policy)
   Right answer, wrong document cited       argument assertions

Every one of those is a system that will fail on the next input, reported today as a success.

Atlas, concretely

The third row is the one that took discipline to leave empty. Atlas's expected trajectories record which calls a ticket requires and, with one exception, not the order, because a run that found the answer in a different sequence is a run that worked, and an eval that says otherwise is training the team to make the agent less of one.

Takeaways

• An answer-only score cannot distinguish a system that works from one that got lucky, and most production failures live in the path rather than the model.
• The opposite failure is demanding one exact sequence, which re-specifies a workflow and penalises the agent for the adaptivity you built it for.
• The design problem is to score the path without specifying it.
• Exact match belongs where order is policy, such as verify before refund, and never where order is an artifact of one run.
• Precision and recall measure different failures: low recall produces a confident answer from incomplete data; low precision costs money and context. Optimise recall first, because precision shows up on a dashboard and recall shows up in a complaint.
• Step efficiency is minimum viable steps over actual, normalised per task class. The minimum is a judgement that belongs in the golden set with the same annotator discipline as the answer.
• Averaging step efficiency across task classes produces a number describing neither.
• The redundant-call rate is the more actionable cousin, and tool consolidation attacks it directly.
• Recovery is the least-scored and most informative behaviour: adaptation, tuning, switching, and loop escape.
• Loop escape is the repeated-identical-error signal from Part VIII as a metric. A low score points at error text that selects no branch.
• You cannot evaluate recovery on traffic that did not fail. Inject timeouts, 4xx, 5xx, and malformed payloads for every tool. It is the one place synthesis is unambiguously right.
• Leaving order unscored where it is not policy takes discipline, and is what keeps the eval from making the agent less of one.

Those extra steps had a price, and nothing in the suite has charged for it yet. Next: Cost and Latency as Scores, where ninety seconds and a dollar is a failing result however right the answer was.