Tool Circuit Breaker

Problem

The carrier API goes down at 14:00. Every get_delivery_status call now takes thirty seconds to time out.

Each run makes the call, waits thirty seconds, retries with backoff, waits again, fails, and the agent, reading a tool error in its transcript, decides to try once more. A run that would normally take twelve seconds now takes four minutes and produces nothing.

Meanwhile every one of those attempts is load on a service that is trying to recover, and every one holds a worker slot and a chunk of the run's deadline budget waiting for a timeout that is guaranteed to fire.

A per-call retry policy cannot fix this, because it only ever sees one call. The information that would help, this dependency has failed the last forty times, exists only in aggregate, and nothing is looking at it.

Forces

• A per-call policy has no memory and cannot know the dependency is down.
• Timeouts against a dead dependency are the most expensive way to fail: full latency, zero information.
• Retries during an outage are load arriving when there is least room for it.
• Recovery must be detected, so the breaker cannot stay open forever.
• Probing with real work is expensive when a run costs money and carries a customer's ticket.
• Slow is a failure mode too. A dependency answering in twenty seconds is down from the agent's perspective.

Solution

A breaker per dependency with three states, tripping on latency as well as errors, and probing with synthetic calls rather than real ones.

                 failure rate OR p99 latency
                 crosses the threshold
        CLOSED ──────────────────────────────▶ OPEN
          ▲                                     │
          │                                     │ cooldown elapses
          │ probes succeed                      ▼
          └───────────────────────────── HALF-OPEN
                                                │
                                          a probe fails
                                                └──▶ back to OPEN

   CLOSED     calls flow, outcomes counted
   OPEN       fail instantly · no timeout waited · no retries generated
              → the caller takes the fallback, or escalates
   HALF-OPEN  a few SYNTHETIC probes, never a customer's run

   one breaker per (dependency, region) — not one per tool:
   three tools hitting the same carrier share one outage

Four rules:

Scope the breaker to the dependency, not the tool. get_delivery_status and get_tracking_events hit the same carrier; one breaker covers both. A per-tool breaker trips three times for one outage and takes three times as long to notice.

Trip on latency, not only on errors. A dependency answering in twenty seconds against a two-second normal is failing from the agent's perspective. It burns deadline and produces nothing. A p99 crossing a threshold should open the breaker as surely as a 5xx rate does.

Probe with synthetic calls. Half-open sends traffic into a dependency you believe is broken. When a run costs money and carries a customer's ticket, that traffic must be a throwaway health check, not the next real request.

An open breaker is a result, not an exception. The agent needs to know the tool is unavailable so it can take a fallback or escalate with what it has. Returning a stack trace produces a confused agent; returning "delivery tracking is temporarily unavailable; proceed without it or escalate" produces a useful one.

Code

interface BreakerConfig {
  failureRate: number;    // 0.5 over the window
  p99LatencyMs: number;   // slow IS down, from the agent's perspective
  minSamples: number;     // don't trip on 2 requests
  cooldownMs: number;
}

export class Breaker {
  private state: 'closed' | 'open' | 'half_open' = 'closed';
  private openedAt = 0;
  private window = new RollingWindow(60_000);

  isOpen(): boolean {
    if (this.state !== 'open') return false;
    if (Date.now() - this.openedAt < this.cfg.cooldownMs) return true;
    this.state = 'half_open';        // probes are scheduled separately
    return true;                     // real traffic still blocked
  }

  record(outcome: 'ok' | 'fail', latencyMs: number) {
    this.window.push(outcome, latencyMs);
    if (this.window.count < this.cfg.minSamples) return;

    // Slow is a failure mode. A dependency answering in 20s burns the
    // run's deadline and produces nothing.
    const shouldOpen =
      this.window.failureRate() > this.cfg.failureRate ||
      this.window.p99() > this.cfg.p99LatencyMs;

    if (shouldOpen && this.state === 'closed') {
      this.state = 'open';
      this.openedAt = Date.now();
      alerts.breakerOpened(this.dep, this.window.snapshot());
    }
  }
}

// The dispatcher turns an open breaker into an INSTRUCTION, not an error.
export async function callTool(tool: ToolDef, args: unknown, ctx: RunContext) {
  const b = breaker(tool.dependency);          // per DEPENDENCY, not per tool
  if (b.isOpen()) {
    return errorForModel('dependency_unavailable',
      `${tool.dependency} is temporarily unavailable. Proceed without this ` +
      `information if you can, or escalate with what you have.`);
  }

  const t0 = Date.now();
  try {
    const out = await tool.execute(args, ctx);
    b.record('ok', Date.now() - t0);
    return out;
  } catch (err) {
    b.record('fail', Date.now() - t0);
    throw err;
  }
}

Returning error_for_model rather than raising is the agent-specific part. An ordinary service breaker throws and the caller handles it; here the caller is a model, and telling it proceed without this or escalate is an instruction it can act on, which frequently turns a failed run into a partial answer plus a good escalation.

Trade-offs

A trip affects everyone. One breaker across the fleet means a threshold crossed by one bad worker denies the dependency to all runs. Scope per region where regions fail independently, and require a minimum sample count so a handful of local errors cannot trip it.

Thresholds are a real tuning problem. Too sensitive and transient blips cause self-inflicted outages; too lax and it never fires. Start conservative, half the calls failing over a meaningful sample, and tune from the observed distribution rather than from intuition.

Recovery detection costs something. Synthetic probes are cheap and are not free, and they must exercise the real path. A probe that hits a health endpoint while the actual API is broken is a breaker that closes into a still-dead dependency.

Half-open needs a concurrency limit. Allowing all queued traffic through the instant the cooldown expires re-floods a service that has just come back. One or two probes, then reopen the gate gradually.

When not to use it

For a dependency with no fallback and no partial answer. If the run simply cannot proceed without the tool, the breaker converts a slow failure into a fast one, genuinely worth something, and much less than when there is an alternative path.

When the dependency is elastic and yours. A service that scales under load does not need you deciding it is down. Give it backpressure instead.

When call volume is too low to measure. Three calls an hour cannot produce a meaningful failure rate. Use timeouts and bounded retries alone.

As a substitute for a rate limiter. A breaker reacts to failure; a limiter prevents it. If you are tripping breakers on 429s, the fix is upstream.

Related

• Fast and Slow Retries: the per-call policy this bounds in aggregate
• Delayed Retry with Jitter: spreading the load the breaker eventually stops
• Fallback Model Ladder: the per-rung breaker, and what an open one routes to
• Downstream Rate Limiting: preventing the failure rather than reacting to it
• Partial Result Return: what the agent does with a tool it can no longer call