Versioning Prompts, Models, and Graphs

A customer disputes a reply Atlas sent on the 14th. You need to know what the agent was working from.

The prompt lives in the repo, so you check out the commit that was deployed that day. That gets you the template. It does not get you: which of nineteen tool descriptions were in the catalogue that afternoon, whether the retrieval k was still 8 or had already been lowered to 5, which corpus build was live, what the sampling temperature was, or which model version the provider was actually serving behind the string claude-sonnet-5.

Every one of those changed what the model saw. None of them is in the commit.

The git SHA identifies your code. It does not identify what the model was given, which is the only thing that determines what it did.

Version the resolved bundle, not the pieces

The mistake is treating the prompt as the artifact. The prompt is one input to an assembly step that runs at request time and produces the actual artifact: the token sequence the model received.

   system prompt template   ┐
   tool catalogue + schemas ┤
   sampling parameters      ┤        ┌──────────┐
   model id + provider      ┼──────▶ │ ASSEMBLE │ ──▶ what the model
   retrieval config (k, …)  ┤        └──────────┘     actually saw
   corpus build             ┤              │
   graph topology           ┘              │
                                           ▼
                                  config_hash: 7f2a91…
                                  ── recorded on every run

So the deployable artifact is the resolved configuration bundle, and it gets a content-addressed identity: hash the whole resolved set, record the hash on every run and every span, and store the bundle keyed by hash.

That single field answers the opening question in one query, and it is the join key that makes everything else in this part work: cost accounting by version, drift detection with a stable baseline, replay that knows whether a recording is still valid, and rollout comparisons that are comparing two known things.

export interface ConfigBundle {
  systemPromptId: string;      // immutable content id, not "v2"
  toolCatalogueHash: string;   // descriptions ARE prompt text
  model: { id: string; provider: string; region: string };
  // Whatever shapes generation on the model you actually call. On current
  // frontier models temperature/topP are rejected and `effort` is the lever
  // — record the one that exists, or the bundle stops naming the behaviour.
  sampling: { effort?: string; temperature?: number; topP?: number; maxTokens: number };
  retrieval: { k: number; indexBuild: string; rerank: boolean };
  graphVersion: string;
  corpusBuild: string;
}

// Content-addressed. Two deploys that produce the same bundle are the
// same configuration, whatever the git history says.
export function bundleHash(b: ConfigBundle): string {
  return sha256(canonicalJson(b)).slice(0, 16);
}

// Resolved ONCE, at run start, and carried for the whole run — because a
// run that changes configuration mid-execution is not one experiment; it
// is two, spliced, and neither is interpretable.
// The exception is policy: tier thresholds and authorization are NOT
// carried. They resolve at the moment of each action, or a run that
// paused on Friday acts on Friday's permissions. See /production/rollout/.
export function resolveForRun(runId: string, env: Env): ConfigBundle {
  const b = env.activeBundle();
  recordOnRun(runId, { configHash: bundleHash(b) });
  return b;
}

Three fields in that struct are the ones teams leave out, and each has cost someone a week:

toolCatalogueHash. A tool description is prompt text that happens to live in a schema. Part VIII said so for quality reasons and Part XVII said so for security reasons. Editing a description changes the agent's behavior exactly as editing the system prompt does, and it usually ships in a PR reviewed as a code change.

corpusBuild. The drift chapter's most common culprit. A re-index changes what the agent reads without changing a byte of your code.

region. Same model string, different endpoint, different serving stack, and a compliance-relevant difference besides.

Immutable identities, not semantic versions

The instinct is prompt-v2.1.3. It does not survive contact with reality, because there is no meaningful notion of a backward-compatible prompt change: a "patch" wording tweak can move behavior more than a "major" restructuring, and nobody can tell which in advance.

Use immutable content-addressed IDs plus a manifest. The prompt is stored by its own hash and never edited in place; the manifest says which ID is active in which environment. That gives three properties semantic versioning does not:

• Rollback is repointing, not reverting. The old artifact still exists, byte-identical.
• Two environments can be compared exactly, because "staging is on 7f2a91" is a fact rather than an inference.
• Nothing is ambiguous under concurrent edits. Two people changing the prompt produce two IDs, not a merge conflict resolved by whoever pushed last.

The human-readable label lives in the manifest as metadata: an author, a date, a change description, and, most importantly, its evaluation results.

A version without a score is not a version. It is a guess with a hash.

Every bundle promoted to production should carry the eval run that justified it, stored with the bundle. This is what makes six-months-later questions answerable: why did we ship this has an artifact, not a memory.

What rollback actually means here

Rollback is where the classical model breaks, because the system has four layers and you control three of them.

The corpus row is the one that needs a decision in advance: an index build that has been deleted is not a rollback target, so retain the previous build long enough to fall back to it, and treat "re-index in place" as a change with no undo.

The last row is the honest limit. You cannot roll back the provider's model. If behavior on a pinned model string shifts, and it does, your available moves are to switch to a different pinned version if one is offered, to adapt the prompt around the new behavior, or to fall back to another rung of the ladder. Which reframes what the version pin is for: it is not a guarantee, it is a record, and its value is that when behavior changes you can prove the change was not yours.

Decoupling deploy from release

Putting the bundle in a store rather than the binary is the right call. It makes rollback seconds instead of minutes and lets you change a prompt without a deploy pipeline. It also removes the safety rails that code review and CI were providing, so it comes with obligations:

A config change is a reviewed change. Same PR flow, same approvals. Otherwise you have built a production-mutation channel that bypasses every control in Part XVII, and a prompt is exactly the kind of thing an attacker or a well-meaning hurried colleague would want to edit.

A config change runs the eval suite. Promotion is gated on the score, not on someone's read of the diff.

A config change is a deploy event. It appears in the same timeline as code deploys, because drift diagnosis starts with "what changed" and a change invisible to that query is a change that will be blamed on the model.

One environment cannot silently diverge. The manifest per environment is itself versioned, so "what is production running" is a stored fact rather than a live read of a mutable key.

Atlas, concretely

The bottom two rows are the chapter. One hash names the whole configuration, one manifest says which hash is live, and one field on every run joins a customer complaint on the 14th to the exact bytes the model was given, including the tool description someone tightened that morning and did not think of as a prompt change.

Takeaways

• The git SHA identifies your code, not what the model was given, and only the latter determines what it did.
• The deployable artifact is the resolved configuration bundle: prompt, tool catalogue, model, sampling, retrieval config, graph, corpus build.
• Content-address the bundle and record config_hash on every run and span. That field is the join key for cost, drift, replay, and rollout comparisons.
• Tool descriptions are prompt text. Editing one changes behavior and usually ships in a PR reviewed as code.
• The corpus build belongs in the bundle. A re-index changes what the agent reads without changing a byte of code.
• Region belongs in the bundle: same model string, different serving stack, and a compliance difference.
• Use immutable content-addressed IDs plus a manifest, not semantic versions. There is no backward-compatible prompt change, and nobody can predict which edit moves behavior most.
• Immutable IDs give repoint-rollback, exact environment comparison, and no ambiguity under concurrent edits.
• A version without a score is a guess with a hash. Store the eval run that justified each promoted bundle.
• Four layers, three of them rollbackable. You cannot roll back the provider's model. The pin is a record, not a guarantee, and its value is proving the change was not yours.
• Retain the previous corpus build, or "roll back the index" is not an available move.
• Rollback does not un-send. Forty minutes of a bad prompt is forty minutes of real replies and real credits, plus durable state that needs its own cleanup.
• Config in a store beats config in the binary, but a config change must still be reviewed, eval-gated, and recorded as a deploy event, or you have built a control-bypassing mutation channel that drift diagnosis cannot see.

Every artifact is versioned and rollback is a real move. Neither says what becomes of a run that started before the change. Next: Rollout and In-Flight Migration, on shipping while thousands of them are mid-execution.