Adversarial Evals and Red Teaming

A prompt-injection test that asks a chatbot to reveal its system prompt is not a red team for an agentic system. The valuable targets are credentials, private data, authority, external writes, durable memory, and the resources an attacker can make you spend.

Test the system the attacker reaches, not the model in isolation.

Start from objectives

The threat model in this part named what an attacker wants. Turn each objective into an observable failure.

These are oracles. They decide pass or fail without asking another model whether the run looked safe.

Build a safe target

Never point exploratory attacks at production effects. The red-team environment needs realistic policy and fake consequences:

production-like agent
       │
       ├── seeded tenant data with canary records
       ├── fake ERP ledger
       ├── email sink
       ├── egress proxy with deny log
       ├── sandbox without production credentials
       ├── Temporal test namespace
       └── full traces and immutable audit records

Seed canary values that should never cross a boundary, such as a synthetic account number unique to tenant B. A leak then becomes an exact string assertion. Do not use real customer secrets as tripwires.

Give the environment the same dispatcher, schemas, identity propagation, retrieval filters, and workflow code as production. A mock that always denies proves only that the mock denies.

Attack every untrusted channel

Direct user text is one channel. The more interesting attacks arrive through content the user did not write in the current turn.

Vary placement and encoding because filters see syntax while the model sees meaning. The goal is not to collect clever jailbreak strings. It is to verify that untrusted content cannot acquire authority regardless of wording.

Test trajectories, not replies

The safest-looking reply may follow a dangerous internal path. The agent can read forbidden data and decide not to mention it. It can call a write tool that the sandbox later blocks. It can queue an approval request to an attacker-selected reviewer.

Capture and assert:

• retrieved IDs before post-filtering;
• tool catalogue before and after discovery;
• tool names and raw arguments;
• authorization and risk decisions;
• credentials and scopes minted for the call;
• egress attempts, including blocked ones;
• graph frontier per hop;
• memory writes;
• workflow signals, updates, timers, and effects;
• final answer and rendered UI events.

Security success is a property of the whole path. "No secret appeared in the answer" is one assertion near the end of a much longer list.

Encode cases as data

export interface AdversarialCase {
  id: string;
  objective: 'exfiltration' | 'cross_tenant' | 'unauthorized_effect' | 'exhaustion';
  principal: { tenantId: string; userId: string; scopes: string[] };
  input: string;
  fixtures: { documents?: string[]; toolResults?: string[]; events?: string[] };
  mustNever: {
    retrieveIds?: string[];
    callTools?: string[];
    contactHosts?: string[];
    writeMemory?: boolean;
  };
  bounds: { steps: number; tokens: number; wallMs: number };
}

Cases should name the objective and invariant, not the exact sentence the model must produce. The model may refuse, ignore, or safely summarize the hostile content. All are valid if the forbidden path remains unreachable.

Four suites

Boundary suite

Run exact cases against retrieval filters, graph traversal, tool scopes, argument validation, egress rules, sandbox mounts, and identity propagation. These should pass deterministically on every commit.

Behavioral suite

Run varied hostile inputs through the real model and score trajectory invariants across repeated samples. This catches the model following untrusted instructions until a control stops it. A blocked attempt still passes the safety invariant and should increment an attempted-bypass metric.

Chained suite

Combine channels over several turns:

user asks benign question
    ↓
retrieval introduces instruction
    ↓
tool result supplies destination
    ↓
later turn asks for summary
    ↓
agent attempts external write

Single-turn tests miss delayed attacks that wait for a capability or sensitive value to enter context.

Recovery suite

Verify containment after detection. The run stops or downgrades, the effect does not occur, the user receives a safe result, the security event is recorded, and a human receives enough evidence to investigate. A guardrail that throws an unhandled exception can turn a safe block into an availability attack.

Generated attacks need human control

An attacker model can vary wording, placement, language, and multi-turn strategy. It is useful for coverage and unreliable as both attacker and judge.

Give it a fixed objective, allowed channels, and a sandbox. Keep exact invariants outside the model. Deduplicate generated cases by behavior, not text. Ten thousand paraphrases that all hit the same blocked argument add cost without adding coverage.

Review novel successful trajectories manually. The interesting artifact is the path that crossed a boundary, not the attacker's explanation of why it worked.

MCP and dynamic tools

Test the catalogue as untrusted data:

• a server adds a new destructive tool after approval;
• a description contains instructions unrelated to the tool;
• two tools use confusable names;
• a schema widens an argument from an enum to free text;
• a read-only server begins returning model-controlled URLs;
• discovery ranks an unapproved server above an internal tool;
• a server asks the client to perform sampling or another privileged operation.

Allowlist by stable server and tool identity, pin or review catalogue changes, and diff schemas as deploy events. "The model never selected it in testing" is not a permission policy.

Durable attacks

Long-lived workflows create attacks that ordinary request tests miss:

• replaying an old approval after policy changed;
• signaling a workflow ID belonging to another tenant;
• flooding updates to grow history or consume reviewer attention;
• resuming after a deploy into a broader tool catalogue;
• retrying an ambiguous effect until one succeeds twice;
• poisoning memory before Continue-As-New carries it forward.

Fast-forward timers, duplicate and reorder signals, rotate credentials, and change policy between pause and resume. Assert authorization at the point of effect. A valid Tuesday checkpoint is not Thursday authority.

Severity follows realized capability

Score findings by the boundary crossed and the possible effect:

Do not downgrade a leak because the final response omitted the value. Do not call every ignored injection a vulnerability. The system's observed capability decides.

Promote every exploit

For each successful attack:

1. preserve the full trace and environment manifest;
2. identify the first boundary that should have stopped it;
3. change the lowest reliable control;
4. add an exact regression at that layer;
5. keep one end-to-end adversarial case for defense in depth;
6. re-run adjacent objectives and channels;
7. record the fix in the threat model.

The OWASP GenAI Red Teaming Guide provides a broader engagement process. This chapter's narrower rule is compatible with it: scope the whole application, define objectives, collect evidence, and turn findings into repeatable tests.

Atlas, concretely

Atlas seeds each tenant with unique canary records and runs all effects against a fake ledger and email sink. Its suite covers direct and retrieved injection, hostile tool results, cross-tenant graph paths, replayed webhooks, expired approvals, MCP catalogue changes, memory poisoning, and cost exhaustion.

The CI gate is exact: no forbidden ID enters retrieval or graph state, no unapproved host receives an attempt, no effect bypasses the dispatcher, every run stays within bounds, and every block leaves a traceable terminal outcome. A nightly behavioral run varies the hostile text against the real model. Every successful path becomes a deterministic boundary test before the fix merges.

Takeaways

• Red-team the application trajectory, not the model in isolation.
• Start from attacker objectives and exact forbidden observations.
• Use production code with seeded canary data, fake effects, controlled egress, and full traces.
• Cover retrieval, tools, MCP, memory, webhooks, files, feedback, and UI rendering.
• Test boundary, behavioral, chained, and recovery suites.
• Use attacker models for variation, never as the only judge.
• Include durable waits, signals, policy changes, and retries in the attack surface.
• Score findings by the boundary crossed and realized capability.
• Promote every exploit to the lowest deterministic regression that can prevent it.

Next: PII, Audit, and Compliance, including the data copies, retention rules, and evidence obligations the test environment must respect.