Computer-Using Agents

The supplier portal has no API. To answer one of Meridian Supply's order-status tickets, a person signs in, searches the order, opens a modal, copies a delivery date, and downloads a PDF. So someone gives Atlas, the agent that works those tickets, a browser and calls the missing integration solved.

The browser has turned one typed tool into hundreds of possible clicks. Every pixel can change, every page can contain hostile text, and the session holds credentials that may reach far beyond the task.

Computer use is a tool boundary with a much larger action space.

Choose the narrowest control plane

There are four ways to automate a screen, in descending structural reliability:

Use an API when one exists. For browser work, prefer semantic locators over coordinates. Reserve screenshot control for surfaces that expose no useful structure, such as a remote desktop or canvas application.

Playwright's official guidance recommends user-facing locators such as role, label, and text. It warns that long CSS or XPath chains break when DOM structure changes. Its actionability checks also wait for a target to be visible, stable, enabled, and able to receive events before clicking. Playwright locators and auto-waiting are good models even when another runtime drives the browser.

observe page
     |
     v
resolve target by role and name
     |
     v
check page, account, action, and risk
     |
     v
perform one action
     |
     v
verify expected state transition
     |
     +---- mismatch ----> stop or recover

One action, one postcondition

"Complete the return" is a goal. It is not a safe browser operation. The executor should expose small actions and verify each result.

export async function cancelOrder(page: Page, orderId: string) {
  await page.getByRole('textbox', { name: 'Order ID' }).fill(orderId);
  await page.getByRole('button', { name: 'Search' }).click();

  const result = page.getByRole('row', { name: new RegExp(orderId) });
  await expect(result).toBeVisible();
  await result.getByRole('link', { name: 'Open' }).click();

  await expect(page.getByRole('heading', { name: `Order ${orderId}` }))
    .toBeVisible();

  return {
    observedOrderId: orderId,
    cancel: async () => {
      await page.getByRole('button', { name: 'Cancel order' }).click();
      await expect(page.getByRole('dialog', { name: 'Confirm cancellation' }))
        .toBeVisible();
    },
  };
}

The code stops at the confirmation dialog. A separate policy gate approves the effect. The model never receives a generic click(x, y) capability for an action whose meaning the application can name.

Visual state is untrusted input

A webpage can contain instructions in prose, images, hidden text, filenames, notifications, or downloaded documents. The agent sees them in the same observation channel as legitimate UI labels.

The official Claude computer-use guidance warns that content on webpages or in images may override instructions or induce mistakes. It recommends a dedicated environment, minimal privileges, restricted network access, no sensitive data, and human confirmation for actions with meaningful consequences. Claude computer use security guidance

Apply the same taint model as retrieval:

trusted control state
  application route, expected domain, user scope

untrusted observation
  page text, pixels, downloads, tooltips, chat messages

policy decision
  deterministic code outside the computer session

Page text may help find a button. It never expands the allowed domain, account, recipient, amount, or operation.

A disposable session is the security boundary

Run each task in a dedicated browser profile or disposable virtual machine. Bind it to one principal, one tenant, and one operation class.

The session needs:

• an allowlist of domains and redirect targets;
• short-lived credentials scoped to the task;
• no access to personal browser cookies, password managers, or clipboard history;
• an isolated downloads directory;
• file scanning before any downloaded content enters context;
• limits on time, storage, network, and screenshots;
• destruction after completion or escalation.

Do not place a broad service-account password inside the browser image. Use a broker that mints a short-lived session after checking the run's delegated identity. The browser gets the session, not the reusable credential.

Recovery needs a state model

Pages drift. Sessions expire. A modal appears. The network returns an error after the remote system accepted the action.

Record after every meaningful step:

{
  "step": "cancel_confirmation_open",
  "url": "https://supplier.example/orders/481",
  "page_title": "Order 481",
  "semantic_snapshot_hash": "sha256:...",
  "screenshot_id": "shot_8823_07",
  "expected": ["dialog:Confirm cancellation"],
  "observed_at": "2026-08-09T14:22:11Z"
}

On mismatch, re-observe and classify the state. Never repeat the last click blindly. The remote action may already have happened.

For a write, follow the same rule as any other integration: read the result back through an independent route. A cancellation flow ends when the order status says canceled, not when the browser reports that it clicked a button.

Evaluate trajectories against controlled sites

A browser benchmark needs more than task completion. Score:

• target resolution accuracy;
• forbidden-domain and forbidden-action rate;
• duplicate-effect rate;
• successful recovery from overlays, stale state, and session expiry;
• number of actions and observations;
• accessibility-locator use versus coordinate fallback;
• evidence completeness;
• latency and cost;
• correct abstention when the page is outside the known state model.

Build deterministic variants of the site that move buttons, delay responses, insert hostile content, return ambiguous confirmation, and expire the session. A golden screenshot alone overfits the agent to one rendering.

Atlas, concretely

Atlas reads orders through the supplier API. Only claims and legacy delivery documents require the portal. The adapter exposes open_claim, upload_evidence, read_claim_status, and request_submit, not generic mouse and keyboard tools.

Submission is a tier-1 action. The adapter stops at a stored confirmation view, a reviewer approves it, and a fresh session re-authorizes and submits. A paired status read confirms the claim ID. Every run uses a disposable profile restricted to the supplier domain.

References

• Claude computer use tool, including environment design and security precautions.
• Claude tool reference, current tool types and versioning.
• Playwright locators, semantic locator guidance and strict target resolution.
• Playwright auto-waiting, actionability checks and retrying assertions.

Takeaways

• Prefer APIs, then semantic browser locators, then scripts, and use screenshot coordinates only when no structure exists.
• Expose task-specific browser operations rather than unrestricted clicks.
• Make every action prove a postcondition before continuing.
• Treat pixels, page text, downloads, and visual instructions as untrusted input.
• Run each task in a disposable, tenant-bound session with short-lived credentials and restricted egress.
• Consent, login, terms acceptance, and identity changes are effects with policy.
• Re-observe after uncertainty. Repeating the last click can duplicate a real action.
• Verify writes through an independent read and score the whole browser trajectory.

Next: Coding Agents and Workspaces, applying the same narrow-capability and isolated-session rules to a repository rather than a browser.