Router

Problem

One agent handles every ticket. It carries nine tools, a system prompt covering policy questions, aggregations, account relationships, credits, and escalation, and a context assembly that pre-loads a bit of everything because any of it might be needed.

Every request pays for all of it. The tonnage question carries the refund policy language; the returns question carries the SQL schema description. Both pay the full tool catalogue on every turn, and both get a model choosing among nine options when three were plausible.

The failures follow the same shape. The agent semantically searches for a number that lives in a database. It reaches for issue_credit on a question that was only asking about timelines. It takes six turns to do what a specialized path does in two. Turn count is the strongest cost lever there is.

Forces

• Different request types need different tools, prompts, and context, and the union is worse at each than any specialization.
• Classification is cheap and reliable; open-ended tool selection is neither.
• A misroute is a whole-request failure, not a degraded one. The wrong path may have no way to recover.
• Routing adds a turn before any work begins.
• Some requests are genuinely mixed and belong to two paths at once.
• Each route is a separate thing to evaluate and maintain.

Solution

Classify first with a small model, then dispatch in code to a path with its own prompt, its own tool subset, and its own context budget.

   request
      │
      ▼
   ┌────────────────────┐   small model, low temperature,
   │   CLASSIFY         │   fixed label set, structured output
   │   → one of N       │   ── cheap, fast, evaluable
   └─────────┬──────────┘
             │  label
             ▼
   ┌────────────────────┐   ordinary code. a switch statement.
   │   DISPATCH         │   ── deterministic, testable, no model
   └──┬────┬────┬────┬──┘
      │    │    │    │
      ▼    ▼    ▼    ▼
   policy data account action        each path:
   3 tools 2    2       5             · its own system prompt
   ~2 turns 1   2       6-8           · its own tool subset
                                      · its own context budget
                                      · its own eval set

Four rules:

Classify into a closed set with structured output. A fixed enum, a small model, temperature at or near zero, where the model still takes one. Current frontier models reject the sampling parameters outright, so temperature: 0 is a setting for the small classifier you route with, not something to carry into the large model you route to. This is the one step where the model's job is narrow enough to be genuinely reliable, and a free-form label defeats the point.

Dispatch in code, not by prompting. The classifier returns policy; a switch picks the path. Asking one model to both classify and then behave differently is the undifferentiated agent again, wearing a routing table.

Include an unknown label, and route it to a human. A classifier forced to pick among five when the answer is none of them will pick confidently. The sixth label is what turns a misroute into an escalation.

Record the label on the trace. Routing accuracy is one of the rare deterministic assertions available in this field: which path did this request take has a right answer, and per-route metrics are how you find the path that is quietly failing.

Code

const Route = z.enum(['policy', 'data', 'account', 'action', 'mixed', 'unknown']);

// Small model, near-zero temperature, closed label set.
export async function classify(req: Request): Promise<z.infer<typeof Route>> {
  const { route } = await model.structured({
    model: SMALL,
    temperature: 0,
    schema: z.object({ route: Route, confidence: z.number() }),
    system: ROUTE_PROMPT,        // one line per label, plus the negative cases
    input: req.text,
  });
  return route;
}

// Each path is a different agent configuration, not a different prompt
// for the same one.
const PATHS: Record<string, PathConfig> = {
  policy:  { tools: ['search_policies'], prompt: POLICY, maxTurns: 3 },
  data:    { tools: ['query_warehouse', 'get_order'], prompt: DATA, maxTurns: 2 },
  account: { tools: ['get_account_graph', 'get_order'], prompt: ACCOUNT, maxTurns: 3 },
  action:  { tools: ACTION_TOOLS, prompt: ACTION, maxTurns: 8 },
};

export async function route(req: Request, ctx: RunContext) {
  const label = await classify(req);
  ctx.trace.set('route', label);        // routing accuracy is assertable

  // A classifier forced to choose will choose. The escape hatch is a label,
  // not a confidence threshold nobody calibrated.
  if (label === 'unknown') return escalate(ctx, 'unroutable request');
  if (label === 'mixed')   return decompose(req, ctx);   // see trade-offs

  return runAgent(PATHS[label], req, ctx);
}

The PATHS table is the pattern. Each entry is a different agent configuration, tools, prompt, and turn cap together, not one agent given different instructions. That is what buys the shorter loops and the smaller contexts.

Trade-offs

A turn before any work. A small classifier is fast, and it is still a round trip on the latency path. Skip it where the caller already knows the type: a form field, a queue name, or an API endpoint is a free, correct, deterministic route.

Misroutes are whole-request failures. The data path has no search_policies, so a policy question routed there cannot recover. It will produce a confident answer from nothing or fail. Design each path to detect that it is on the wrong one and hand back rather than improvise.

Mixed requests are the hard case and they are common. "Can we return these, and how long does the credit take?" is two routes. Options are decomposing into two runs and merging, routing to the broader path, or routing to a human. Decomposition is best and most work; picking the broader path silently is what most systems do and it forfeits the specialization for that request.

N paths is N eval sets. Each route needs its own fixtures and its own quality bar, plus a routing-accuracy set on top. This is real maintenance cost, and it is why five routes is usually better than twelve.

When not to use it

When the routing is knowable without a model. The determinism test applies directly: if a form field, a queue, or an endpoint already determines the type, route on that. A classifier re-deriving something you were told is pure cost and a new failure mode.

When the paths barely differ. If two routes share a prompt and a tool set, they are one route. Splitting for tidiness adds a classification decision that can be wrong in exchange for nothing.

When one path handles everything acceptably. A three-tool agent does not need a router. Measure whether the undifferentiated version is actually failing before adding a layer.

Below a volume where per-route evals are affordable. Five routes you cannot separately evaluate is five paths whose quality you are guessing at. Fewer routes, each measured, beats more routes, each assumed.

Related

• Deterministic Rails, the general form: code owns control flow, the model owns judgment
• Plan Then Execute: the other way to separate deciding from doing
• Retrieval as a Tool: routing expressed as tool descriptions instead of a classifier
• Model Cascade: routing on difficulty rather than on type
• Routing and Orchestration: the chapter this pattern comes from