Batch API Offload

Problem

Three workloads share one quota and one price: live ticket handling, the nightly eval suite, and a quarterly backfill re-classifying 340,000 historical tickets.

Only the first has anyone waiting. The other two run on the interactive endpoint at interactive prices, and at 02:04 they cause the incident where a job with no deadline starves the workload with customers in it.

Both problems have the same one-line fix, and most teams do not take it: providers sell a batch endpoint at 50% off both input and output tokens, in exchange for asynchronous completion within 24 hours. The eval suite does not care whether it finishes at 02:10 or 06:00. It has been paying double for a latency guarantee nobody needed.

Forces

• Half price is a large, unconditional discount on tokens with no quality difference.
• The 24-hour window is a hard constraint, not a soft target, though most batches complete inside an hour, and requests that do expire are not billed.
• Separate metering removes contention with interactive traffic, which is worth as much as the discount.
• Asynchrony changes the code shape: submit, poll, collect, rather than call and await.
• Partial failures need handling: a batch returns per-request outcomes, not one status.
• The discount stacks with prefix caching, and a shared preamble across thousands of items is exactly the case where it compounds.

Solution

Route every latency-tolerant workload to the batch endpoint, and treat batch-eligibility as a property of the workload rather than a per-call decision.

   WORKLOAD                    WAITING?   ENDPOINT     PRICE   QUOTA
   ─────────────────────────────────────────────────────────────────
   live ticket handling        customer   interactive  1.0×    shared
   drafted-reply suggestions   agent      interactive  1.0×    shared
   ─────────────────────────────────────────────────────────────────
   nightly eval suite          nobody     BATCH        0.5×    separate
   quarterly re-classification nobody     BATCH        0.5×    separate
   dataset generation          nobody     BATCH        0.5×    separate
   embedding backfill          nobody     BATCH        0.5×    separate

   ┌──────────────────────────────────────────────────────────────┐
   │ submit up to ~100k requests ──▶ poll ──▶ collect per-request │
   │ results within 24h · usually hours · per-item success/failure│
   └──────────────────────────────────────────────────────────────┘

   stacking: batch 0.5× on all tokens
           + cache read ~0.1× on the shared prefix
           ── a fan-out with one big system prompt gets both

Four rules:

Classify the workload, not the request. Is anyone waiting for this? has one answer per workload and it does not change per item. Deciding per call produces a system where some evals are batched and some are not, for no reason anyone can reconstruct.

Chunk submissions and record the mapping. A batch is capped at 100,000 requests or 256 MB, whichever comes first, so a job of 340,000 is several batches, and a job with large payloads may hit the size limit long before the count. Keep a durable map from your item ID to the batch request ID, because results come back keyed by the latter and a lost mapping is a completed batch you cannot use. Collect on a schedule too: results are retained for 29 days after the batch is created, so a job whose collector has been broken for a month is a batch you paid for and can no longer read.

Handle per-item outcomes. A batch does not succeed or fail as a unit: each request has its own result or error. Treat it exactly as a fan-out: isolate per-item failures, break on a systemic rate, and write a manifest saying what is missing.

Structure the batch to share a prefix. Ten thousand items with one large common system prompt get the batch discount on everything and the cache-read discount on the shared portion. Randomizing anything at the front of each request throws the second one away.

Code

// A property of the WORKLOAD, decided once. Not a per-call heuristic.
export const BATCH_ELIGIBLE = new Set(['eval', 'backfill', 'dataset_gen', 'embedding']);

export async function submitBatch(job: Job): Promise<BatchHandle[]> {
  const handles: BatchHandle[] = [];

  for (const chunk of chunkBy(job.items, MAX_REQUESTS_PER_BATCH)) {
    const requests = chunk.map(item => ({
      custom_id: item.id,                    // OUR id — the mapping is durable
      params: {
        model: job.model,
        // Identical prefix across every request in the batch, so the batch
        // discount and the cache-read discount both apply.
        system: [{ ...PREFIX.reference, cache_control: { type: 'ephemeral' } }],
        messages: [{ role: 'user', content: item.body }],
      },
    }));

    const batch = await provider.batches.create({ requests });
    // Persist the mapping BEFORE returning: a lost map is a completed batch
    // you cannot use.
    await store.recordBatch(job.id, batch.id, chunk.map(i => i.id));
    handles.push({ batchId: batch.id, count: chunk.length });
  }
  return handles;
}

export async function collect(job: Job, handle: BatchHandle) {
  const results = await provider.batches.results(handle.batchId);

  const failures: ItemFailure[] = [];
  for (const r of results) {
    // Per-item outcomes, exactly like a fan-out. Not one status.
    if (r.result.type === 'succeeded') await store.putResult(r.custom_id, r.result);
    else failures.push({ id: r.custom_id, error: r.result.error });
  }

  // A manifest that says what is missing, so the job is repairable.
  await store.writeManifest(job.id, { done: results.length - failures.length, failures });
}

custom_id carrying your item ID is what makes results joinable. Results return keyed by that field, and a batch of 100,000 responses you cannot map back to inputs is a completed job with no output.

Trade-offs

24 hours is a hard ceiling, not an estimate. Most batches complete within an hour, and nothing guarantees it: a batch that does not finish expires, though the expired requests are not billed. Any job on a schedule tighter than a day needs a fallback plan, usually running the remainder interactively, at full price, which is fine as an exception and expensive as a habit.

Asynchrony is a real code change. Submit, persist, poll, collect, handle partials. That is a durable workflow rather than a function call, and the polling loop must survive deploys.

The stacking is conditional on the cache TTL, and the default one loses. The discounts do compose: cache multipliers apply on top of the batch discount. But a batch runs for hours, and the short-TTL entry expires in about five minutes. Every request after the first few pays the write premium and never reads it, which is worse than not caching. Use the long TTL for batch work, keep the prefix identical across every request in the batch, and treat a low cache_read_input_tokens on a batch job as the signal that the stacking never happened.

Some capabilities are unavailable on the batch path. Provider feature parity is not guaranteed: certain modes are excluded. Check before assuming a workload can move, particularly if it depends on a specific serving configuration.

Not everything latency-tolerant is batch-shaped. A job needing item n's result to build item n+1's request cannot batch, because the whole submission is fixed at once. Independent items only.

When not to use it

When anyone is waiting. Interactive paths, agent turns inside a live run, anything with a user-visible deadline. The discount is not worth a 24-hour tail.

When the job must finish within hours, reliably. A nightly eval that must gate a morning deploy is at risk from the ceiling. Either move the deploy gate or accept interactive pricing for that suite.

When volume is small. A hundred items saves a trivial amount and costs you the submit-poll-collect machinery. The pattern earns its keep in the thousands.

When items are sequentially dependent. Batches are submitted whole; there is no way to feed one result into the next request.

Related

• Cache-First Prefix: the discount that stacks with this one on a shared preamble
• Priority Task Queues: the contention this removes rather than schedules
• Fan-Out Over Items: the per-item failure handling a batch result needs
• Batch Iterator: walking a job too large for a single submission
• Cost Engineering: where this sits in the lever ordering

References

• Message Batches, the window, discount, and request limits this pattern trades latency for.