Cost Engineering

The first month Atlas works Meridian Supply's support queue at real volume, someone forecasts the annual bill from the pilot's per-ticket cost and gets a number nobody budgeted.

The instinct in the room is to downgrade the model. It is almost always the wrong first move, and the reason is arithmetic that most teams have never written down.

Where the money actually goes

One equation explains nearly every agentic bill, and it is not the one people carry in their heads.

   cost(run) ≈  Σ  ( context_at_turn(t) × input_price
                    + output_at_turn(t) × output_price )
               t=1..T

   and because the whole transcript is re-sent every turn:

        context_at_turn(t) ≈ system + tools + Σ (prior turns)
                             └─────────────┘
                             grows with t

   ⇒  total input tokens grow ~ T² / 2
      total output tokens grow ~ T

   so the run's cost against a 1-turn run is not T. It is

        T  +  (perTurn / prefix) × T(T-1)/2

   which for a 20-turn run is ~99× at Atlas's shape, and
   approaches 200× as the per-turn growth catches the prefix.

The chart is the same statement with the constant left in. Bars are what each turn sends; the line is the same conversation with history capped by compaction.

Three consequences follow, and they reorder the whole optimization list:

Input dominates, usually heavily. A run that emits two hundred output tokens per turn but carries a forty-thousand-token context is spending most of its money re-reading its own history. Output price gets the attention because it is the bigger number per token; input volume is what multiplies.

Turn count is the strongest lever you have. Because input cost is roughly quadratic in turns, a change that removes four turns from a twenty-turn run saves far more than the same percentage change in price-per-token. This is why tool granularity and result design are cost decisions and not only quality ones.

Retries and failures are billed at full price. A run that fails at step eighteen paid for eighteen steps. The retry budget is a cost control that happens to live in the reliability chapter.

So before touching the model, measure: tokens per run, split input/output, by turn index. Almost every team that does this for the first time finds that one tool returning a large payload every turn is a majority of their bill.

The levers, in order of leverage

The order is the argument. The top two are free of quality risk and are where the money usually is; the bottom row is the one people reach for first and the only one that degrades everything at once.

The effort row is the highest-leverage one that does carry a quality trade, and the one most teams have never touched, because it replaced the sampling knobs and arrived with a default. Omitting the effort parameter is identical to asking for the highest setting, so every run you have not thought about is running at the top of the scale, where the model generates several times the reasoning tokens of the lowest setting. It also makes more tool calls, which is why it belongs above context work rather than below it: it moves the turn count, and turn count is quadratic. The evals chapter's measured case, cost per attempt from $3.40 to $2.30 with no movement in resolution rate, is this lever, pulled once.

Two mechanisms deserve detail because their behavior is not obvious.

Cache reads and batch discounts stack. They apply to different axes: one to the repeated prefix, one to the whole request. So a large fan-out of latency-tolerant calls sharing one long system prompt gets both. On the shared portion the combined effect is dramatic, and it is the single best reason to move offline work (evals, backfills, dataset generation) to a batch endpoint rather than running it on the interactive path.

Routing is a bet, and the bet has a cost when it loses. A cascade sends the cheap model first and escalates on low confidence. That is genuinely 40–70% cheaper in reported practice, but only if the escalation signal is good, because a wrong "confident" from the cheap model is a bad answer at a discount, and a cascade that escalates too eagerly pays for both models and saves nothing.

Route on the task, not on the token

The routing decision most teams implement is a classifier that reads the request and guesses difficulty. It works about as well as that sounds.

The version that works is coarser and structural: different steps of the same run have different quality requirements, and you know which is which at design time.

Nothing here is a runtime guess. The node knows what it is, so the model is a property of the node, which also means the routing is testable, reviewable, and rollback-able in the gateway config rather than emergent from a classifier's mood.

The one runtime decision worth making is escalation: run small, and promote to large when a validator fails, a confidence check trips, or the risk tier says this call matters. That is a cascade with an honest gate, and the gate is the part to build carefully.

Enforce the budget in the loop

Cost controls that live in a dashboard are reports, not controls. The enforcing version is a counter in the run's state, checked where the spending happens.

export interface CostBudget {
  runId: string;
  capMicros: number;      // set from the run's value, not a global default
  spentMicros: number;
  softRatio: number;      // 0.7 — degrade before you fail
}

export function beforeCall(b: CostBudget, estimateMicros: number) {
  if (b.spentMicros + estimateMicros > b.capMicros) {
    // Not an error. The run finishes with what it has, or escalates.
    return { action: 'stop' as const, reason: 'run budget exhausted' };
  }
  if (b.spentMicros > b.capMicros * b.softRatio) {
    // Degrade first: smaller model, tighter retrieval, no speculative calls.
    return { action: 'degrade' as const };
  }
  return { action: 'proceed' as const };
}

// The cap is per run because that is the unit the business understands:
// a ticket worth $8 of margin does not get $40 of inference.

The degrade rung is what makes this usable. A budget with only a hard stop produces truncated runs at the worst moment; one that degrades first produces a cheaper, slightly worse completion, which is the same shape as a fallback, applied to money instead of availability.

And the cap is per run, derived from what the run is worth. A global monthly ceiling tells you nothing about whether a specific ticket deserved forty model calls. Unit economics are the frame: cost per resolved ticket, against the margin on a resolved ticket. If you cannot state that ratio, you are optimizing a number rather than a business.

The frontier is a choice, so make it once, explicitly

There is no cheapest-and-best. There is a curve, and every configuration sits on it or inside it, and inside is where most systems actually are, which is the encouraging part of this chapter.

The productive sequence is:

1 · Get onto the curve first. Turn reduction, prompt hygiene, cache prefix discipline, and not re-sending payloads you already summarized are pure wins with no quality cost. Do all of them before trading anything away.

2 · Then pick your position, per workflow. Not per company. The tier-0 automated refund and the escalation draft that a human will send have different stakes, and one number for both is a decision not to think about either.

3 · Measure the trade with evals, not with intuition. The eval suite is what makes "20% cheaper, 2% worse on the rubric" a sentence you can say. Without it, every cost change is a gamble settled by whoever complains loudest.

4 · Re-run the choice when prices move. They move a lot, always downward, and a routing table tuned eighteen months ago is leaving money on the table in a direction nobody notices.

The honest caution: cost optimization has a floor set by what the product is worth, and past a certain point you are choosing a worse product to save money that does not matter. A support agent whose replies get slightly worse to save two cents a ticket has made a bad trade, and the only way to know which side of that line you are on is to have the eval number and the margin number in the same sentence.

Where the first of those comes from is a chapter of its own, and it is worth reading before you believe any figure on this page: cost per resolved outcome is a different quantity from cost per run, and a cheap configuration that succeeds one time in five costs exactly what an expensive one that always works. The reported experience is that the first honest computation lands three to eight times above what the API arithmetic suggested, which is the gap between the number this chapter helps you lower and the number your business actually pays.

Atlas, concretely

The first two rows produced more saving than every price-level change combined, and neither of them touched a model. That is the ordinary result, and it is why "downgrade the model" is the wrong first move: it trades quality for the smallest lever on the list while the largest ones are still untouched.

References

• Prompt caching, cache-read pricing relative to input, and what invalidates a prefix.
• Message Batches, the batch discount, completion window, and request limits.

Takeaways

• The whole transcript is re-sent every turn, so input tokens grow roughly with the square of turn count. A 20-turn run costs on the order of a hundred times a 1-turn run on the input side, not 20×, and the multiple rises as the per-turn growth catches up with the fixed prefix.
• Input usually dominates the bill. Output price is the bigger per-token number; input volume is what multiplies.
• Turn count is therefore the strongest lever, which makes tool granularity and result design cost decisions.
• Failed and retried runs bill at full price. The retry budget is a cost control.
• Measure first: tokens per run, split input/output, by turn index. The usual finding is one fat tool result re-sent every turn.
• Lever order: cut turns, shrink what's re-sent, cache, batch, route, early-exit, and only then consider a cheaper model everywhere, which is the one that degrades everything at once.
• Cache-read and batch discounts stack, because they apply to different axes. Move evals, backfills, and dataset generation to batch.
• Route structurally, not with a difficulty classifier. Each node knows its own quality requirement at design time, which makes routing testable and rollback-able.
• Make escalation the only runtime routing decision, and build the gate carefully: a wrong confident answer from the cheap model is a bad answer at a discount.
• Enforce budgets in the loop, with a degrade rung before the hard stop. A cap that only fails produces truncated runs at the worst moment.
• Set the cap per run, from what the run is worth. Unit economics, cost per resolved ticket against margin on one, is the frame.
• Get onto the frontier before trading along it. Most systems are inside the curve, where savings cost nothing.
• Pick your position per workflow, measure the trade with evals, and re-run the choice when prices move.
• Treat every ratio here as relative, not absolute. Which lever is bigger than which is structural; the multipliers are not.
• None of it works without per-run, per-tenant cost attribution, recorded at the model gateway.

Deployed, versioned, rolled out and priced, and nobody has been woken up by it yet. Next: The Operating Manual, on dashboards, incident shapes, and being on call for something probabilistic.