Timeouts, Retries, and Backoff

The provider degrades for ninety seconds. Latency triples; a fraction of requests return 529.

Atlas, working tickets through all of it, has a sixty-second timeout on model calls, three retries in the SDK, a node-level retry policy of three, and a graph that lets the model try again after a tool reports an error. None of those numbers looked unreasonable when they were written, and each was written by a different person.

Ninety seconds of provider degradation produces forty minutes of elevated traffic, a request volume roughly an order of magnitude above baseline aimed at a service that is already struggling, several thousand dollars of tokens spent on attempts whose answers were discarded, and a queue that takes until evening to drain. The provider recovers at second ninety-one. You do not.

You did not survive the outage. You amplified it.

This is the oldest failure in distributed systems, and agents make it worse in three specific ways that are worth naming before touching a single constant.

Why the standard advice is insufficient here

Exponential backoff with jitter and a retry budget is correct and every SRE text says so. It is also written for a world where a retry costs a socket and a few milliseconds. Three properties break that assumption.

Retries cost money, not just load. A retried database query costs microseconds. A retried model call costs the full input tokens again, and in an agent, the input is the entire accumulated context, which is largest exactly at the deep steps most likely to fail. Your retry policy is a line item.

The layers multiply, and there are more of them than you think. The SDK retries. The node retries. The workflow re-executes the activity. And then a fourth layer nobody counts: the model, reading a tool error in its transcript and deciding to call the tool again. That last one is not configured anywhere, does not appear in your retry metrics, and is the reason agent traffic during an incident looks nothing like the arithmetic predicted.

The unit of work is minutes, not milliseconds. Retrying a whole run because step fourteen failed discards thirteen successful steps and their tokens. Where to retry matters more than how often.

   ①  SDK              3 attempts   ┐
   ②  node policy      3 attempts   ├─ 3 × 3 × 3 = 27 provider calls
   ③  workflow retry   3 attempts   ┘   for one logical step
                                        …before the fourth layer
   ④  the model        reads "tool failed", calls it again
                       not configured · not counted · not bounded

   Nested retries multiply. They never add.

The first rule follows directly, and it is worth more than any tuning:

Retry at exactly one layer per failure class. Configure the others to zero.

Pick the layer that owns the recovery. Transport hiccups belong to the SDK, so let it retry and set the node's policy for that error to none. Anything needing durability across a process death belongs to the workflow, so the node does not retry it. Two layers both retrying the same failure is not defense in depth; it is multiplication.

Not everything should be retried

Retrying a permanent failure is pure cost, and half of a well-behaved policy is knowing what to abandon immediately. The error taxonomy draws the full map; the operational summary is this:

The two "no" rows people get wrong are the third and the fifth. A context-length error is deterministic: the same request will fail identically three more times, and each attempt bills you for a prompt the provider rejected. And a malformed-arguments error should not be retried by your code at all. Returning it to the model as an instruction is the recovery path, and wrapping a transport retry around it produces three identical failures before the model ever gets to see one.

Timeouts, plural

A single timeout on a streaming, variable-length call is the wrong instrument. Three are needed, and they answer different questions.

Time-to-first-token is the one usually missing and the one that matters most. It separates the provider is struggling from this is a long answer, and a single overall timeout conflates them. Set it long enough for a legitimate long generation and you wait a minute to discover a dead endpoint; set it short and you kill valid work.

For streaming, the useful bound is inactivity rather than total duration. If no token has arrived in fifteen seconds, the stream is dead regardless of how long it has been running. A total cap still belongs there as a backstop, because an agent that loops will happily consume any budget you leave open.

The deadline is the run's, not the call's

Per-call timeouts do not compose. A run with twenty steps at sixty seconds each has a worst case of twenty minutes, which nobody chose and nobody wants. And the user gave up at ninety seconds.

The fix is a deadline set once, at the start, and propagated: each call gets the smaller of its own timeout and the remaining budget.

export interface RunBudget {
  deadlineMs: number;      // absolute wall-clock, set once at run start
  maxAttempts: number;     // across the whole run, not per call
  attemptsUsed: number;
}

export function callTimeout(budget: RunBudget, preferredMs: number): number {
  const remaining = budget.deadlineMs - Date.now();
  if (remaining <= 0) throw new DeadlineExceeded();
  // Never let one call consume the whole remaining budget: leave room
  // for the run to finish cleanly and write its result.
  return Math.min(preferredMs, Math.floor(remaining * 0.8));
}

// Full jitter. Deterministic backoff synchronises every client that
// failed at the same instant into a second, sharper spike.
export function backoffMs(attempt: number, baseMs = 500, capMs = 30_000) {
  const ceiling = Math.min(capMs, baseMs * 2 ** attempt);
  return Math.random() * ceiling;
}

Two details carry the design. maxAttempts is per run, not per call. Otherwise a twenty-step run with three retries each has sixty attempts of headroom and no ceiling anyone reasoned about. And the backoff is full jitter: sampling uniformly from [0, ceiling] rather than adding a small wobble to a fixed delay. Deterministic backoff re-synchronizes every client that failed in the same second into a sharper spike one delay later, which is how a brief degradation becomes a sustained one.

Retry budgets, because per-call caps do not bound the system

Capping attempts per call bounds nothing globally. With everything failing, every call uses its full allowance and total load rises by the retry multiplier exactly when the dependency can least absorb it.

The control that actually bounds it is a retry budget, a ceiling on retries as a fraction of overall traffic, measured over a rolling window. Google's SRE practice pairs a per-request cap of three attempts with a per-client budget of 10%: the cap alone lets load grow to just under 3×, and the budget brings that back to roughly 1.1×. When the budget is exhausted, retries are refused and failures pass through.

That last clause is the part that feels wrong and is the point. During a widespread outage, retries are not helping. Everything is failing, so retrying mostly converts one failure into three. Passing the failure through immediately is both cheaper and faster to recover from, and it hands the decision to the layer that can actually do something useful: a fallback or a circuit breaker.

For agents, budget the money as well as the count. Retries consume the same token budget as productive work, and a run that has spent 80% of its allowance on retried attempts should stop rather than finish poorly at full price.

Retry the step, not the run

The agent-specific question, and the one with the largest practical payoff.

A run is not a request. When step fourteen fails, the thirteen completed steps are still valid: their tool results are known, their tokens are spent, and their conclusions are in the transcript. Retrying the run discards all of it and pays again.

This is what checkpointing and durable execution were for, arriving here as a cost argument rather than a correctness one:

The precondition for the third row is that steps be individually replayable, which means reads are safe to repeat and writes are idempotent, the property the next chapter is entirely about.

Atlas, concretely

Two lines are the whole chapter. Node retries are zero for transport failures, not because node retries are wrong, but because the SDK already owns that class and two owners multiply. And the retry budget fails through when exhausted, which converts the incident from an amplifying loop into a clean, fast, cheap set of failures that the next chapter's fallbacks can act on.

The provider recovers at second ninety-one. With these settings, so do you.

References

• Handling overload, Google SRE, per-request attempt caps, per-client retry budgets, and the amplification arithmetic behind both.

Takeaways

• The failure mode is amplification: ninety seconds of provider degradation becomes forty minutes of self-inflicted load, at full token price.
• Agents break the standard advice three ways: retries cost tokens, the layers multiply, and the unit of work is minutes.
• There is a fourth retry layer nobody configures: the model reading a tool error and calling it again. It is not in your retry metrics.
• Retry at exactly one layer per failure class and set the others to zero. Nested retries multiply; they never add.
• Do not retry deterministic failures. A context-length error will fail identically three more times and bill you each time.
• Malformed tool arguments are not a transport retry. Returning the error to the model is the recovery.
• Three timeouts, not one: connect, time-to-first-token, and an inactivity-based stall timeout. TTFT is the one usually missing and the best health signal.
• Set the deadline once, at the run, and propagate it. Per-call timeouts do not compose into anything anyone chose.
• Cap attempts per run rather than per call, or a twenty-step run has sixty attempts of unreasoned headroom.
• Use full jitter. Deterministic backoff re-synchronizes every failed client into a sharper second spike.
• Honor Retry-After over your own formula, and treat a large value as backpressure rather than as a wait.
• Add a retry budget, roughly 10% of traffic as a starting point. Let failures pass through when it is exhausted. During a real outage, retrying converts one failure into three.
• Budget the money too. Retries spend the same tokens as productive work.
• Retry from the last checkpoint, not from the start. Step fourteen should cost step fourteen, which requires the idempotency the next chapter builds.

Resuming from step fourteen is only sane if step fourteen can run again without paying twice. Next: Idempotency in Practice, where Part VIII's key meets dedup windows and the exactly-once story stops being true.