The Reference Architecture

Twenty parts of this book have argued for components one at a time. Assembled naively that produces a diagram with fourteen boxes, which is an excellent way to spend six months not shipping.

So this chapter answers the opposite question. Not what could an agent platform contain. What does yours need, and what is the cost of adding a piece later rather than now?

The good news is that the answer is determined almost entirely by two questions, and you can answer both today.

Two questions decide your architecture

                    Does a run have side effects
                    the world can see?

                         NO              YES
                    ┌──────────────┬──────────────────┐
   Does a run       │              │                  │
   outlive one      │  TIER 1      │  TIER 2          │
   HTTP request?    │  request-    │  transactional   │
              NO    │  shaped      │  agent           │
                    │              │                  │
                    ├──────────────┼──────────────────┤
                    │              │                  │
                    │  TIER 2      │  TIER 3          │
              YES   │  async, but  │  durable         │
                    │  cheap to    │  agent           │
                    │  redo        │                  │
                    └──────────────┴──────────────────┘

Tier 1: request-shaped. A run completes inside one HTTP request and only reads. A support-answer drafter, a classifier, a summarizer, an internal Q&A bot. Your existing web application plus a model client, and nothing else on the list below. Teams who skip this tier because it "won't scale" spend a quarter building Tier 3 for a product that never needed it.

Tier 2: two cells, needing different things. Both demand machinery Tier 1 does not, for unrelated reasons, which is why the label covers a diagonal rather than a row.

Asynchronous is the common one: runs take minutes, so they cannot live in a request. You need somewhere to put the work and somewhere to keep the state. A queue and a state table get you here, and this is where most production agents actually belong.

Transactional is the cell teams forget, because it looks like Tier 1. The run finishes inside the request and still moves something in the world. It needs no queue and no state table. Nothing outlives the response. What it needs is everything that makes an effect safe to repeat: idempotency keys, the dispatcher choke point, and an audit record. A client retrying a timed-out POST does not know whether your agent finished, and neither does your agent.

Tier 3: durable. Runs take hours or days, pause for humans, and move money. Now a crash mid-run has to be survivable without repeating effects, and you want durable execution rather than a queue plus hope.

The honest observation about this diagram: the vertical axis is the one people get wrong. Teams size for volume and discover that duration is what broke them. A thousand short runs need a bigger box. Ten runs that each pause for two days need a different architecture.

The components, and when you can skip each

The last four rows are the chapter's actual content. Everything above them is a normal web service and can be added on a Tuesday. Those four are expensive later for different reasons, and it is worth being specific about each:

Tracing is retroactively impossible. The bug you are chasing happened last week and produced no record. Nothing you deploy today recovers it. This is the single cheapest thing on the list to add early and the most painful to lack.

Evals are worse. Not because they are hard to build, but because their value is comparative: the number only means something against yesterday's number. A team that adds evals in month six has an eval suite and no history, which tells them how they are doing and not whether it is getting better. Datasets start accumulating on day one or they start six months late.

Durable execution changes the code, not the deployment. Adopting Temporal is not an infrastructure decision you can defer behind an interface. It imposes determinism constraints on how nodes are written. Retrofitting it means rewriting the agent, which is why the tier question deserves a real answer up front even if you build Tier 2 first.

The dispatcher is where the whole security part lands. Taint, argument scoping, delegation, idempotency keys, risk tiers. Every one of those is a check in the path between the model's tool call and the effect. If tool calls dispatch from four different places in your codebase, you have four places to implement all of it, and you will implement it in three.

One choke point for tool execution, from the first write tool. It is the cheapest structural decision in this book and the most expensive to retrofit.

The shape of the thing

        ┌──────────┐
        │   API    │  auth, run creation, streaming out
        └────┬─────┘
             │
        ┌────▼─────────────────────────────────────────┐
        │  ORCHESTRATOR   graph / workflow definition   │
        └────┬─────────────────────────────┬───────────┘
             │                             │
        ┌────▼──────┐               ┌──────▼──────────┐
        │  MODEL    │               │   DISPATCHER    │ ◀── every check
        │  GATEWAY  │               │                 │     from Part XVII
        └────┬──────┘               └──────┬──────────┘
             │                             │
        ┌────▼──────┐          ┌───────────┼──────────┬────────────┐
        │ providers │          ▼           ▼          ▼            ▼
        └───────────┘      retrieval    database   external    sandbox
                                                    APIs
        ─────────────────────────────────────────────────────────────
        STATE          Postgres · object store · checkpoints · history
        OBSERVABILITY  traces · costs · evals · audit records

Two properties of this drawing matter more than the boxes.

Every arrow leaving toward the world passes through the dispatcher. That is not an aesthetic choice; it is the thing that makes the security part implementable and the trace complete.

State is a plane, not a box. The run's state lives in several stores with different durability and retention, and, as tenancy and compliance both showed, forgetting one of them is how the boundary leaks. Draw them together so the list stays visible.

The model gateway, because it is the cheapest lever here

One box deserves its own section, because it is small, nearly free, and quietly enables four later chapters.

A gateway is a thin internal service every model call goes through. Not a vendor product necessarily. A hundred lines and a config file will do. What it buys:

• One place for retries, timeouts, and the retry budget, which is exactly what the last part asked for, and impossible if six services each call the provider directly.
• One place for fallbacks to another model or provider.
• One place cost is attributed: per run, per tenant, per feature.
• One place model versions are pinned, so "we upgraded the model" is a config change with a rollback rather than a deploy across six repos.
• One place region and retention terms are enforced.

Five chapters' worth of control from one box, added at the start for almost nothing. If you take one thing from this chapter and you are Tier 1, take this.

What people build too early, and too late

Too early: a multi-agent topology (the next part is about why), a vector database, a fine-tuned model, a custom orchestration framework, and a Kubernetes install for a service handling four requests a minute.

Too late: tracing, evals, the dispatcher choke point, cost attribution, and a runs table with enough columns to reconstruct what happened.

The asymmetry is not a coincidence. The too-early list is capability, things that make the agent do more. The too-late list is legibility, things that let you find out what the agent did. Capability is visible in a demo and legibility is not, so capability gets funded first and legibility gets funded after the first incident.

Build the smallest architecture that can answer "what happened, and why." Then add capability as the product demands it.

Atlas, concretely

Nine of those twelve rows would be absent from a Tier 1 build, and that is the point of the tier question. Atlas needs them because a run pauses for two days and then moves money, not because agents in general need them.

Takeaways

• Two questions decide the architecture: does a run outlive one request, and does it have side effects the world can see.
• Duration is the axis teams get wrong. Volume needs a bigger box; long pauses need a different design.
• Tier 1 is a web app plus a model client, and skipping it to build for scale you don't have costs a quarter.
• Most production agents are Tier 2: usually a queue and a state table, sometimes a run that finishes inside the request and still needs idempotency and a dispatcher.
• Four things are expensive to add late: tracing, evals, durable execution, and the dispatcher choke point. Everything else is a normal Tuesday.
• Tracing is retroactively impossible. Last week's bad run left no record and nothing you deploy now recovers it.
• Evals are worse than tracing to defer, because their value is comparative. Month-six evals have no history to compare against.
• Durable execution changes how nodes are written, not just what you deploy. It cannot hide behind an interface.
• One choke point for tool execution from the first write tool. Every control in the security part is a check on that path, and four dispatch sites means implementing all of it four times.
• The vector index is the most over-provisioned box on the diagram: it commits you to a pipeline, an eval suite, a permissions model, and an attacker-writable surface.
• Build the model gateway on day one. It is ~200 lines and it is where retries, fallbacks, cost attribution, version pinning, and residency all land.
• The too-early list is capability; the too-late list is legibility. Capability demos well, which is why it gets funded first.
• Build the smallest architecture that can answer "what happened, and why."

An architecture that can say what happened still has to say which version of itself did it. Next: Versioning Prompts, Models, and Graphs, because the git SHA identifies your code and not what the model was given.