Prompt Caching and Prefix Stability

The previous chapter ended with four levers for context, and a note that caching sits beside them rather than among them. Send less, compress, externalize, isolate. Those change how many tokens you send. Caching changes what those tokens cost, by roughly an order of magnitude, and changes nothing else.

That makes it the cheapest win in the book and the one most often left on the floor, because it has a single hard constraint that dictates the order of your prompt. Get the ordering right and caching mostly works by itself. Get it wrong and no amount of configuration will save it, and worse, nothing will tell you.

It is a prefix match, not a lookup

The whole feature follows from one sentence:

The cache key is the exact bytes of the prompt up to the marked point. Any change anywhere in that prefix invalidates everything after it.

This is not a cache in the sense of "look up this document by ID." It is a prefix match against the literal token sequence. There is no fuzzy matching, no per-block caching, no reordering. One flipped character at position 40 discards the work for positions 40 onward.

And the positions are not up to you. The provider renders your request in a fixed order:

   ┌──────────┬──────────────┬────────────────────────────┐
   │  tools   │    system    │          messages          │
   └──────────┴──────────────┴────────────────────────────┘
    position 0 ───────────────────────────────────▶ position N

    change a tool definition  ─────▶  ████████████████████  all of it
    change the system prompt  ─────────────▶  ████████████  from there on
    append a turn             ───────────────────────▶  ██  just the tail

Tool definitions render first. They sit at position zero, in front of everything. Adding a tool, removing one, or serializing your schemas in a different key order invalidates the entire request: system prompt, every turn of history, all of it. This is the single most expensive mistake available, and it is one line of code away in any codebase that builds its tool list per user.

The economics

Caching is not free. A cache write costs a premium over normal input; a cache read costs a fraction of it.

Which gives you a break-even you can compute:

Two consequences. Caching a prompt that is only used once is strictly worse than not caching it. You pay the write premium and never read. And the longer TTL is not simply better: it survives gaps in bursty traffic, but the doubled write means it needs more reads to pay for itself. Continuous traffic should use the short one, because each request re-warms the entry for the next.

For an agent this compounds the same way everything else does: the loop resends its whole history on every iteration, so the cached prefix is read once per model call, not once per user turn. A four-call turn reads the cache four times. The break-even arrives inside a single turn.

Placing the breakpoint

You mark the end of the cacheable prefix with cache_control on a content block.

const response = await client.messages.create({
  model: 'claude-opus-5',
  max_tokens: 1024,
  tools: TOOL_DEFINITIONS,          // rendered first — must be stable
  system: [
    {
      type: 'text',
      text: SYSTEM_PROMPT,          // frozen: no dates, no user names
      cache_control: { type: 'ephemeral' },
    },
  ],
  messages,                         // volatile: everything that changes
});

console.log(response.usage.cache_read_input_tokens);

Because system renders after tools, a marker on the last system block caches both, one breakpoint covering the entire fixed portion of the request. That is the default placement and it is right most of the time.

Three constraints worth knowing before you place a second one:

You get a small number of breakpoints, four on current Anthropic models. They are a budget, so spend them on genuine stability boundaries: the end of the tools-plus-system prefix, the end of a shared document set, the end of the most recent turn.

There is a minimum cacheable length, and it is model-specific. Below it, nothing caches: no error, no warning, just a write count of zero. Current minimums range from about 512 to 4,096 tokens depending on the model, and they are not ordered by generation: a newer model can have a lower minimum than an older one, so a prompt that caches fine on one model silently won't on another in the same family. Look yours up rather than assuming.

For multi-turn conversations, move the marker. Put it on the last content block of the most recent turn, each request. Earlier breakpoints stay valid as read points, so hits accumulate as the conversation grows and each turn extends the cached prefix by one turn.

The silent invalidators

Everything above is easy. This section is where caching actually fails, and it fails without saying so: the request succeeds, the answer is fine, and you pay full price forever.

Every one of these is a real pattern from real code, and every one lives in the prefix:

The fix is always the same shape: make it deterministic, move it after the breakpoint, or delete it. A timestamp the agent genuinely needs belongs in the latest user turn, where it invalidates nothing before it. A per-user fact belongs in the conversation, not the system prompt.

Notice what the last two rows imply. If your system supports "modes" by swapping the tool list, or routes to a cheaper model for sub-tasks, you have built a system that cannot cache. The escape hatches exist, but you have to design for them: dynamic tool discovery that appends schemas rather than replacing them; a mid-conversation system message that sits after the cached history rather than editing the prompt in front of it; giving a sub-task to a separate sub-agent instead of switching the main loop's model.

Verify, or you don't know

Because the failure is silent, an unverified cache is an assumed cache. The response tells you exactly what happened:

That last row catches people. input_tokens is not the size of your prompt; it is the part of it that neither hit nor wrote the cache. Total prompt size is the sum of all three. An agent that ran for an hour and reports 4,000 input tokens is not efficient. It is reading most of its prompt from cache, and the number you want is the sum.

The diagnostic is one line: if cache_read_input_tokens is zero across repeated requests that should share a prefix, an invalidator is at work. Dump the rendered prompt bytes from two consecutive requests and diff them. The culprit is always visible and is almost always in the table above.

Two failures specific to agents

Ordinary chat applications never hit these. Agent loops hit both.

The lookback window. A breakpoint searches backward a bounded number of content blocks, about twenty, to find a matching entry. A single agent turn that fires eight parallel tool calls appends sixteen blocks, and two such turns put the previous breakpoint out of reach. The cache is there; the next request simply can't see it. Long turns need an intermediate breakpoint every dozen or so blocks.

The fan-out race. An entry becomes readable only once the first response begins streaming. Fire ten identical-prefix requests concurrently and all ten pay full price, because none can read what the other nine are still writing. Send one, wait for its first token rather than its last, then release the other nine.

Both share a lesson worth generalizing: caching was designed for a conversation, and an agent is not a conversation. It is a burst of near-identical requests around a prefix that barely changes. That is the best possible case for caching and the easiest one to break.

The discipline

Everything in this chapter reduces to one ordering rule, applied when you build the prompt rather than after the bill arrives:

   stable ─────────────────────────────────────────▶ volatile

   tool schemas   frozen system   retrieved docs   history   this turn
   (sorted,       prompt (no      (per session)              (per request)
    fixed set)     interpolation)
                                 ▲
                                 └── breakpoint moves right as stability allows

Sort what you serialize. Freeze the system prompt. Push anything per-request as far right as it will go. Then mark the boundary and check cache_read_input_tokens to confirm you were right.

Do that and the discipline pays for itself twice: once in money, and once because a prompt built stable-to-volatile is a prompt whose structure you can reason about at all.

References

• Prompt caching, cache breakpoints, minimum cacheable prefix lengths, TTL behavior, and what invalidates a prefix.

Takeaways

• Caching is a prefix match on exact bytes. One changed byte invalidates everything after it.
• The render order is tools → system → messages. Tool definitions sit at position zero, so changing the tool set costs the entire cache.
• Reads cost roughly a tenth of normal input; writes cost a premium. Break-even is two requests on a short TTL, three on a long one, so caching a one-shot prompt is worse than not caching it.
• One breakpoint on the last system block covers tools and system together. That is the right default.
• There is a model-specific minimum length below which nothing caches, silently, and it is not ordered by model generation.
• The killers are invisible: a timestamp, a UUID, unsorted JSON, a per-user tool list, a mid-conversation model switch. All of them fail without an error.
• input_tokens is the uncached remainder, not the prompt size. Verify with cache_read_input_tokens or you are assuming.
• Agent-specific traps: a bounded lookback window that long tool-heavy turns overshoot, and concurrent fan-out where every request misses because none has finished writing.

Caching settles what the prompt costs on the way in. Next: Structured Output, and the other half of that contract: getting a usable value back out.