Fan-Out Over Items

Problem

A quarterly review needs every ticket from the last three months classified, summarized, and checked against the current policy. Four thousand items, each needing a model call, each independent of the others.

The sequential version takes forty minutes and loses everything to a redeploy. The naive parallel version, starting four thousand concurrent calls, hits the provider's rate limit in the first second, produces a wall of 429s, triggers retry amplification, and takes down the interactive agent that shares the quota.

Between those is a large, unglamorous middle: run many at once, but not all at once, with failures isolated per item and progress that survives a worker dying.

Forces

• The items are independent. No item's result affects another's processing.
• Parallelism is the whole point: latency drops from sum to max.
• Provider quota is finite and shared with traffic that has users waiting.
• One poisoned item must not fail the batch. Item 3,112 having malformed data is not a reason to lose 3,999 results.
• The parent has limits too: a workflow tracking four thousand children accumulates history.
• Results have to be collected, and four thousand payloads through the parent is its own problem.

Solution

One child per item, a bounded concurrency window, and results written to storage rather than returned through the parent.

   PARENT
   ┌───────────────────────────────────────────────────────────┐
   │  4,000 items · semaphore of 20 in flight                  │
   │                                                           │
   │   ┌────┐ ┌────┐ ┌────┐        ┌────┐                      │
   │   │ c1 │ │ c2 │ │ c3 │  ...   │c20 │  ◀── window          │
   │   └─┬──┘ └─┬──┘ └─┬──┘        └─┬──┘                      │
   │     │      │      │             │                         │
   │     ▼      ▼      ✗ fails       ▼    one child finishing  │
   │   ok     ok    recorded,      ok    starts the next item  │
   │                batch continues                            │
   │                                                           │
   │  each child writes its OWN result to storage              │
   │  the parent collects REFERENCES, not payloads             │
   └────────────────────────┬──────────────────────────────────┘
                            ▼
                   { done: 3,987, failed: 13, manifest: s3://… }

Four rules:

Bound concurrency from the token budget, not from a thread count. The binding constraint is input tokens per minute, not CPU. Compute the window as available TPM divided by tokens per item per minute, and expect a number in the tens rather than the thousands.

Isolate failures per item. A child that fails records its failure and the batch continues. Fail the batch only on a systemic signal: a breaker opening, an auth error, a failure rate above a threshold, because those mean the next 3,000 items will fail too.

Return references, not payloads. Each child writes its result to object storage and returns a key. Four thousand result objects through the parent's history is how a fan-out hits the 50 MB history limit on a job that would otherwise have been fine.

Give the batch its own quota class. This is bulk work with no user waiting, so it belongs on the batch endpoint at half price and in a priority class that cannot starve interactive traffic. The 2am incident where an eval job took the interactive quota is this rule not being applied.

Code

export async function classifyBacklog(input: BatchInput): Promise<BatchResult> {
  // Sized from the TOKEN budget, not from a thread count. The binding
  // constraint is input TPM, and it usually yields tens, not thousands.
  const window = Math.max(1, Math.floor(input.tpmBudget / input.tokensPerItemPerMin));

  const inFlight = new Set<Promise<void>>();
  const results: string[] = [];      // storage keys, never payloads
  const failures: ItemFailure[] = [];

  for (const item of input.items) {
    if (inFlight.size >= window) await Promise.race(inFlight);

    const p = runChild(item)
      .then(key => { results.push(key); })
      .catch(err => {
        // Isolated: one bad item does not lose 3,999 good ones.
        failures.push({ id: item.id, error: String(err) });
      })
      .finally(() => { inFlight.delete(p); });

    inFlight.add(p);

    // Systemic signals DO stop the batch — the next 3,000 will fail too.
    if (failures.length / (results.length + failures.length || 1) > 0.2
        && results.length + failures.length > 50) {
      break;
    }
  }
  await Promise.all(inFlight);

  // A manifest, not four thousand payloads through the parent's history.
  return { done: results.length, failed: failures.length,
           manifest: await writeManifest(results, failures) };
}

The failure-rate break is the difference between a batch that isolates failures and one that ignores them. Per-item isolation is right for a malformed record; it is exactly wrong when the credential expired, and a 20% failure rate over a reasonable sample is the cheapest available signal that something systemic is happening.

Trade-offs

Window sizing is the whole performance story. Too small wastes wall clock; too large produces 429s, retries, and contention with interactive traffic. Derive it, measure the actual 429 rate, and treat any provider rate limiting as evidence the window is wrong rather than as something to retry through.

Children have their own overhead. Each is a workflow or activity with its own scheduling, history, and preamble. For four thousand tiny items, batching fifty per child is far cheaper than four thousand children: the unit of fan-out does not have to be the unit of work.

Partial success needs a product answer. 3,987 done, 13 failed is the normal outcome, and someone has to decide what that means: retry the thirteen, escalate them, or report the batch as complete with exceptions. Undefined means the manifest is a file nobody reads.

Progress visibility is worse than it looks. From outside, a fan-out is one long-running parent. Emit counts as it goes, or the only observable states are running and done, and a batch that will take three hours looks identical to one that is stuck.

When not to use it

When items are not independent. If item 12 needs item 11's result, this is a pipeline or a batch iterator, not a fan-out. Forcing parallelism onto sequential work produces subtly wrong results rather than an error.

When the set is small. Under a few dozen items, a loop inside one activity is simpler and the parallelism buys little.

When the set is enormous. Beyond a few thousand children, one parent becomes the bottleneck: its history grows and its scheduling saturates. Fan out to intermediate parents instead.

When work arrives continuously. Fan-out processes a known set in waves. A steady stream wants a sliding window, which keeps N in flight permanently rather than draining and refilling.

Related

• MapReduce Tree: when one parent cannot hold the fan-out
• Sliding Window: continuous arrival instead of a known set
• Batch Iterator: when the items must be walked in order
• Concurrency, Rate Limits, Backpressure: where the window size comes from
• Sub-Agent Context Isolation: the per-item isolation, viewed from the context side