Model Cascade

Problem

Every model call in the run uses the large model, because the hardest step needs it.

Most steps do not. Classifying a ticket into one of six labels, extracting three fields from a tool result, and deciding whether a reply mentions an order number are all tasks a much cheaper model does essentially as well, and they are the majority of calls in a twenty-turn run.

The obvious fix is to route each node to a model chosen at design time, and that is the right first move. It runs out where the difficulty varies within a node: most damage assessments are straightforward and some are genuinely ambiguous, and the node cannot know which it has until it looks.

Downgrading that node uniformly trades quality on the hard cases to save money on the easy ones. Keeping it large pays frontier prices for the easy ones. Neither is what you want.

Forces

• Difficulty varies per request, not just per node.
• Cheap models handle most instances of most tasks and fail on the tail.
• The saving is real: reported routing gains commonly land in the 40–70% range.
• An escalation costs both models, so a cascade that escalates too often is more expensive than not cascading.
• The confidence signal is the whole design, and a wrong "confident" is a bad answer at a discount.
• Two models mean two behaviours to evaluate, version, and watch for drift.

Solution

Run the cheap model first, and escalate on a signal you trust, preferring a deterministic check to the model's own self-assessment.

   request
      │
      ▼
   ┌──────────────┐  cheap model
   │   ATTEMPT    │  ~10× cheaper, ~90% of traffic ends here
   └──────┬───────┘
          │
          ▼
   ┌────────────────────────────────────────────────────────┐
   │  ESCALATION GATE — best signal available, in order:    │
   │                                                        │
   │  ① validator fails       schema, citation resolves,    │
   │     ── deterministic     amount matches a tool result  │
   │  ② out-of-distribution   label not in the closed set,  │
   │     ── deterministic     required field missing        │
   │  ③ risk tier ≥ 1         the CALL is consequential —   │
   │     ── from arguments    never decided by a cheap model│
   │  ④ self-reported low     weakest. calibrate or ignore  │
   │     confidence                                         │
   └──────────────────────┬─────────────────────────────────┘
                          │ escalate
                          ▼
                  ┌──────────────┐  large model
                  │   RETRY      │  full context, one attempt
                  └──────────────┘

   break-even at price ratio R: escalation rate below (R−1)/R
                                 R = 10  ──▶  90%

Four rules:

Prefer a deterministic gate. A failed schema validation, an unresolvable citation, or a label outside the closed set is a fact about the cheap model's output. Self-reported confidence is another guess from the model whose guess is in question: usable, and the weakest signal on the list.

Never let the cheap model handle a consequential call. Route by risk tier before routing by difficulty: anything above tier 0 goes to the large model regardless of how confident the cheap one was. A discount on the decision to move $2,500 is not a saving.

Watch the escalation rate, and know where break-even actually is. Every request pays the small model and an escalated fraction p pays both, so a cascade costs 1 + Rp against a flat R for never cascading, where R is the price ratio. Break-even is (R − 1) / R. At R = 10 that is an escalation rate of 90%, and at the ~10% this pattern targets the cascade costs about a fifth of the baseline. The margin is deliberately wide, which is why a rising escalation rate is an early warning about routing quality long before it is a cost problem.

Escalate with the original input, not the cheap model's output. The large model should answer the question, not review a draft. Passing the first attempt biases it toward agreeing, which is the correlated-reviewer failure in a new place.

Code

export async function cascade<T>(
  req: ModelRequest, schema: ZodSchema<T>, ctx: RunContext,
): Promise<{ value: T; servedBy: string }> {
  // Consequential calls never start on the cheap rung. A discount on the
  // decision to move money is not a saving.
  if (riskTier(req) > 0) {
    return { value: await callLarge(req, schema), servedBy: LARGE };
  }

  const first = await callSmall(req, schema);

  const gate = escalationGate(first, req, ctx);
  if (!gate.escalate) {
    ctx.trace.set('cascade.served_by', SMALL);
    return { value: first.value, servedBy: SMALL };
  }

  // Escalate with the ORIGINAL request. Passing the first attempt biases
  // the large model toward agreeing with it.
  ctx.metrics.inc('cascade.escalated', { reason: gate.reason });
  return { value: await callLarge(req, schema), servedBy: LARGE };
}

function escalationGate(first: Attempt, req: ModelRequest, ctx: RunContext) {
  // ① deterministic: a FACT about the output, not another opinion
  if (!first.parsed)                    return { escalate: true, reason: 'schema' };
  if (!citationsResolve(first, ctx))    return { escalate: true, reason: 'citation' };
  // ② out of distribution
  if (!LABELS.has(first.value.label))   return { escalate: true, reason: 'ood' };
  // ④ weakest signal, and only if calibrated against real outcomes
  if (first.confidence < THRESHOLD)     return { escalate: true, reason: 'confidence' };
  return { escalate: false as const };
}

cascade.served_by on the trace is the same field the fallback ladder requires, for the same reason: without it you cannot segment quality by which model actually answered, and a cascade whose cheap rung has quietly degraded looks exactly like a cascade that is working.

Trade-offs

A wrong "confident" is a bad answer at a discount. The failure mode is not an error, it is the cheap model being wrong and sure. This is why the deterministic gates come first and why the self-confidence threshold needs calibration against real outcomes rather than a number someone picked.

Latency increases on escalated requests. Two sequential calls. At a 10% escalation rate the p99 is the sum, which matters on interactive paths and not on background ones. Running both in parallel and discarding one removes the latency and removes the saving.

Two models to operate. Two versions to pin, two eval runs, two drift surfaces, two sets of formatting quirks. A cheap model that silently changes behaviour raises the escalation rate, which reads as a cost regression until someone checks the canary.

The threshold drifts with traffic. A gate tuned on last quarter's mix mis-escalates on this quarter's. Watch the escalation rate as a first-class metric, and treat a shift as a signal about the traffic, not just about the gate.

When not to use it

When per-node routing already covers it. If the node's difficulty does not vary, assign the model at design time: testable, reviewable, rollback-able, and with no runtime gate to be wrong.

When there is no trustworthy gate. Without a validator, a closed label set, or a citation check, you are left with self-reported confidence alone. That can work and it needs calibration data; shipping it uncalibrated produces confident cheap answers on exactly the cases that needed the large model.

When the escalation rate is high. Above the break-even implied by the price ratio, the cascade costs more than not having one. Measure before assuming it saves money.

On consequential calls. Anything above tier 0. The gate should route these before the cheap model is ever invoked.

Related

• Cost Engineering: the lever ordering, and why per-node routing comes first
• Two-Stage Rerank: the same cheap-then-expensive structure in retrieval
• Fallback Model Ladder: descending models for availability rather than cost
• Token Budget Enforcement: the ceiling a cascade helps you stay under
• Risk Tiers: the gate that runs before the cheap model is considered