Tool as Activity

Problem

The agent loop is running inside a durable workflow, and someone calls the payments API directly from workflow code.

It works in development. In production it fails in a way that is hard to even describe: the worker crashes at step nine, Temporal replays the workflow from its history to rebuild state, and the replay calls the payments API again, because there is no record of the first call in the history, only the code that made it. The credit is issued twice, and the second one has no event anywhere saying why.

The same applies to the model call. A completion is nondeterministic by definition, so replaying workflow code that calls a model produces a different answer, a different tool choice, and a different branch, and the workflow diverges from its own history, which is a hard error rather than a subtle one.

Forces

• Replay must converge. Workflow code runs again from the top after any crash, and every non-recorded side effect runs again with it.
• Model calls are nondeterministic by construction, the one thing a workflow cannot contain.
• Tools have wildly different timing profiles: a 20 ms database read and a 90-second document generation should not share a timeout.
• Retry policy is per-operation. A read retries freely; an irreversible write does not.
• Every boundary costs a payload round trip through the history, which is size-limited.

Solution

Workflow code decides; activity code discovers. Every tool call and every model call is an activity.

   WORKFLOW  (replayed, must converge)
   ┌─────────────────────────────────────────────────┐
   │  the loop, the branches, the stopping rule      │
   │  the budget counters, the taint flag            │
   │  the approval wait                              │
   │                                                 │
   │   result = await callModel(ctx)      ───────┐   │
   │   if (result.toolCall) {                    │   │
   │     out = await runTool(name, args) ────┐   │   │
   │   }                                     │   │   │
   └─────────────────────────────────────────┼───┼───┘
                                             │   │
   ACTIVITY  (runs once, result recorded)    ▼   ▼
   ┌─────────────────────────────────────────────────┐
   │  the model call        nondeterministic         │
   │  the tool call         touches the world        │
   │                                                 │
   │  each with its OWN timeout and retry policy;    │
   │  the result lands in the history, so replay     │
   │  reads it instead of re-running it              │
   └─────────────────────────────────────────────────┘

Four rules:

One activity per tool, not one activity for "run a tool." A generic executeTool(name, args) activity gives every tool the same timeout and the same retry policy, which means the 20 ms read and the 90-second generation are configured identically and one of them is wrong. Separate activities let each carry its own.

Set the retry policy from the tool class. Class ① reads retry freely. Class ④ and ⑤ writes retry only if idempotent, and their permanent failures are listed as non-retryable so the workflow can decide rather than the SDK looping.

Keep the dispatcher's checks in the activity, not the workflow. Taint, scope, tier, delegation, and the idempotency key all live in the dispatcher, and the dispatcher runs inside the activity, because minting a token and hashing arguments are things you discover, and because the checks must run on the real call rather than on a replayed one.

Pass references, not payloads. Activity arguments and results are written to the event history verbatim. A 200 KB tool result crossing the boundary is 200 KB of permanent history per call, which is how a workflow hits its size limit in an afternoon.

Code

// Proxied with per-activity timeouts. Nothing here shares a policy.
const { callModel } = proxyActivities<typeof modelActivities>({
  startToCloseTimeout: '2 minutes',
  heartbeatTimeout: '20 seconds',        // streaming; see heartbeat pattern
  retry: { maximumAttempts: 3, nonRetryableErrorTypes: ['ContextTooLong'] },
});

const { getOrder } = proxyActivities<typeof readActivities>({
  startToCloseTimeout: '30 seconds',
  retry: { maximumAttempts: 5 },          // class ① — retry freely
});

const { issueCredit } = proxyActivities<typeof writeActivities>({
  startToCloseTimeout: '30 seconds',
  scheduleToCloseTimeout: '10 minutes',   // unsettled after 10m is a human's problem
  retry: {
    maximumAttempts: 3,                   // safe ONLY because it is idempotent
    nonRetryableErrorTypes: ['CreditLimitExceeded', 'AccountNotFound'],
  },
});

export async function ticketAgent(input: TicketInput): Promise<Outcome> {
  const state = initState(input);         // workflow-local: budgets, taint, scratchpad

  while (!state.done && state.turn < MAX_TURNS) {
    // Nondeterministic → activity. Its result is recorded, so replay
    // reads the same completion instead of sampling a new one.
    const step = await callModel(state.toRequest());

    if (step.toolCall) {
      // The dispatcher's eight checks run INSIDE the activity, on the
      // real call — not on a replay, and not in workflow code.
      const result = await dispatchActivityFor(step.toolCall.name)(
        step.toolCall.args, state.contextRef,   // a reference, not a payload
      );
      state.record(step, result);
    } else {
      state.done = true;
    }
    state.turn += 1;
  }
  return state.outcome();
}

policy_for(tool) deriving timeouts and retry rules from the tool class is what keeps this from becoming forty hand-tuned constants. The class is already recorded in the catalogue for security reasons; reuse it.

Trade-offs

Payload round trips. Every argument and result serializes into the history. This is the cost that bites first on agents, because the transcript is the payload. Passing the whole context to the model activity on every turn writes it to history on every turn. Pass a reference to state held outside, or you will hit the size limit long before the event count.

Latency per boundary. An activity dispatch is a queue round trip, typically single-digit to low tens of milliseconds. Negligible against a model call, meaningful if you make an activity out of something trivial.

Two places to look when debugging. The workflow shows the decisions; the activity shows what happened. Good tracing joins them under one run ID, and without that join an investigation crosses a boundary with nothing linking the sides.

Non-retryable classification is load-bearing. Getting it wrong in one direction burns money retrying a permanent failure; in the other it fails a run that would have succeeded on the second attempt. This is the error taxonomy with real consequences attached.

When not to use it

For pure computation. Formatting, arithmetic, parsing, and branching belong in workflow code. They are deterministic, they replay correctly, and making them activities adds latency and history for nothing.

For clocks, IDs, and randomness. Those have replay-safe SDK equivalents. Reaching for an activity to get the current time is a common first-week mistake with a one-line fix.

When you are not using durable execution. Outside a workflow, this is just "call your tools through a dispatcher", which you should do anyway, without the activity machinery.

For a tool you call fifty times in a tight loop. Fifty activities is fifty history entries and fifty round trips. Batch them into one activity that loops internally, and accept that the whole batch retries together.

Related

• Agent as Entity Workflow: the workflow this pattern lives inside
• Heartbeat for Long Tools: detecting a hung activity before its timeout
• Resumable Activity: restarting a long activity from a checkpoint
• Saga for Tool Side Effects: undoing activities that succeeded before a later failure
• Determinism and Retries: what replay actually compares, and the four sources of divergence