Tool Result Truncation

Problem

query_warehouse returns 340 rows of line-item JSON, about 40,000 tokens. The model needed the total and the ship date.

Three things go wrong at once. The window takes a 40,000-token hit for two numbers. Because the transcript is re-sent every turn, that payload is billed again on every subsequent turn of the run. And the two numbers that mattered are now buried in the middle of a long context, which is where attention is weakest.

The naive fix is worse than the problem: slicing the JSON at 4,000 characters produces invalid JSON, and the model, reading a truncated array with no marker, treats what it can see as the complete result and confidently reports a total over 31 rows.

Forces

• Tool output is unbounded and its size is a property of the data, not the schema.
• The model usually needs a small projection of it, but which projection depends on the question.
• Silent truncation is a correctness bug, not a budget optimization: the model cannot distinguish "these are all the rows" from "these are the first thirty-one."
• Sometimes the whole result genuinely is needed, so the pattern must leave a route to it.
• Every turn pays again. A one-off cost at turn three is a recurring cost for the rest of the run.
• The right place to enforce this is the dispatcher, which is the only code that sees every tool result.

Solution

Cap tool output at the dispatcher, and make the cap shape the result rather than cut it.

   raw result (40k tokens)
        │
   ①  PROJECT   drop fields nobody asked for; the schema of the
        │       result is a design decision, not the API's default
        ▼
   ②  AGGREGATE where the answer is a number, compute it here
        │       340 rows → { count, sum, min, max, span }
        ▼
   ③  CAP       first N items, valid structure, plus:
        │         "showing 31 of 340"  ← the model must know
        │         a handle to fetch more  ← and be able to act
        ▼
   shaped result (~400 tokens)

Four rules:

Project before you cap. Result design is the primary fix and truncation is the backstop. A tool that returns thirty fields when the agent uses four has a schema problem, and capping the output is treating the symptom.

Aggregate in the tool, not in the model. If the question is "what did we ship to Iberia in Q2," the tool should return a number. Sending 340 rows so the model can add them up is expensive, slow, and wrong often enough to matter, because models are unreliable arithmetic engines.

Never truncate silently. The truncated result says what it is: "showing": 31, "of": 340. This is the same discipline as errors as instructions: the result is prompt text, and a result that misrepresents itself produces confident wrong reasoning with nothing to catch it.

Give it a way to get more. A continuation handle, a narrower query, or a get_full_result(ref) tool. A cap with no escape route turns a large result into a dead end, and the model's alternative is to guess.

Code

export interface ShapedResult {
  data: unknown;
  truncated?: { showing: number; of: number; continuation: string };
}

const MAX_RESULT_TOKENS = 2_000;

export function shapeForModel(
  raw: unknown, tool: ToolDef, ctx: RunContext,
): ShapedResult {
  // ① project — the tool declares what the model actually needs
  const projected = tool.project ? tool.project(raw) : raw;

  // ② aggregate — a summary beats rows whenever the answer is a number
  if (tool.aggregate) return { data: tool.aggregate(projected) };

  if (estimateTokens(projected) <= MAX_RESULT_TOKENS) return { data: projected };

  // ③ cap — structurally valid, explicitly labelled, with a way forward
  if (Array.isArray(projected)) {
    const kept = takeUntilTokens(projected, MAX_RESULT_TOKENS);
    return {
      data: kept,
      truncated: {
        showing: kept.length,
        of: projected.length,
        // Stored server-side; the model passes it back to page or refine.
        continuation: stash(ctx.runId, tool.name, projected, kept.length),
      },
    };
  }

  // Not a list: keep the head, and say so rather than pretending.
  return {
    data: headByTokens(projected, MAX_RESULT_TOKENS),
    truncated: { showing: 1, of: 1, continuation: stash(ctx.runId, tool.name, projected, 0) },
  };
}

stash keeps the full result outside the context, in object storage or the run's state, which means the data is not lost, it is merely not in the prompt. That distinction is the whole pattern: the model gets a projection and a handle, and the full result stays available to the trace, to a later call, and to a human.

Trade-offs

A round trip when the cap is wrong. If the model needed row 200, it pays a turn to page. Sizing the cap is an empirical question: look at how often continuations are actually followed, and raise the cap if the answer is "often."

Aggregation moves logic into tools. tool.aggregate is application code that has to be written, tested, and kept aligned with what users ask. This is real work, and it is the same work tool granularity asks for anyway.

Projection can drop what mattered. A fixed projection encodes an assumption about which fields matter. Where the question genuinely varies, prefer a field-selection argument on the tool over a hardcoded projection, and note that this is a schema decision, so the model chooses the fields and your code still enforces the cap.

One more thing that must not be forgotten at compaction. A continuation handle referencing a stashed result must survive a context rebuild, or the model is holding a pointer into nothing.

When not to use it

When results are naturally small. get_order returns one order. Adding a shaping layer to a tool whose output is bounded by its schema is ceremony.

When the tool should have been narrower. If truncation is firing constantly on one tool, that is a granularity signal, not a capping problem. query_warehouse that returns everything and gets truncated should probably be count_orders, sum_orders_by_region, and list_recent_orders.

When the full result is the deliverable. A tool whose output goes into a report or a file the user receives should write to object storage and return a reference: the model never needed the bytes at all.

When the caller is code, not the model. A node reading a tool result directly should get the full result. Shaping exists because the consumer is a context window; a function has no such constraint.

Related

• Result Design: the primary fix; truncation is the backstop
• Schema and Granularity: constant truncation on one tool means the tool is too broad
• Rolling Summary: compresses the turns; this compresses what goes into them
• Sub-Agent Isolation: when the processing of a large result should also stay out
• Batch Iterator: when the model genuinely must work through all 340 rows