Towards Safe Autonomy for Computer Use

What it claims

SA-CUA is a dual-LLM defense against indirect prompt injection for computer-use agents. It works across hybrid web and OS environments because it labels trust from AT-SPI accessibility trees rather than the browser DOM. Prior work such as UCM uses the DOM, which limits it to browsers. The authors claim three things:

• Labeling leakage: zero leakage of untrusted free text (FNR_free 0%), against 44.1% for UCM, both with Qwen3.8-27B.
• Utility at zero attack success: 86.2% completion on RTC† against 18.5% for CaMeL-CUA, with Qwen3.8-27B.
• A new attack class: a benchmark, RTC-value, where the injected value has the same type as the expected benign one. The authors argue that bounding the quarantined model's output type, as CaMeL-CUA and UCM do, therefore does not stop these attacks. Even a single bit can steer the planner.

How it works

1. Mask everything by default. The screen starts fully masked, and the system unmasks only content it trusts. - First-party chrome is always visible. This includes menus, toolbars, the window frame and the URL bar. - A pixel region with no accessibility-tree node stays masked. Coverage gaps therefore cost utility, not security.
2. Derive rules offline. Rules are built once per URL template or application. - A rule-proposing LLM sees a sanitized subtree. Every name and text is replaced with [name:length:N], while role, id and class are kept. - It proposes rules, then drops any that cover too many elements or none. - Subtree claims are then checked against the unredacted tree. The page cuts off at this step.
3. Apply rules online. At each step the rules run against the live tree, and the privileged LLM sees only the unmasked screenshot.
4. Classify the task first. A specification classifier sends underspecified tasks back to the user to refine. It never asks the user to confirm an action.
5. Query the quarantined LLM. The quarantined LLM reads the full screen and answers within a requested type. The answer comes back as a redacted variable. - Redacted variables can be used in non-consequential actions, such as scrolling or answering the user. - Reading the value, or using it in a consequential action, requires endorsement.
6. Gate every action. A policy blocks actions such as clicks inside untrusted regions.

Trust is defined by influence, not authorship: content is user-generated if any party other than the operator can change what it displays. Under this definition commit hashes and timestamps count as untrusted. The authors chose it because it is decidable and reproducible.

flowchart LR
  Q[Quarantined LLM reads full screen] --> V[Redacted typed variable]
  V -->|non-consequential use| A[Scroll / answer user]
  V -->|read or consequential use| E{Endorse}
  E -->|verified via user task, LLM knowledge, trusted contacts, trusted domains| P[Privileged LLM]
  E -->|not verified| X[Stays redacted]
  P --> G{Policy gate} --> Act[click / type / scroll]

Endorsement follows the "robust endorsement" principle: the value being endorsed cannot influence its own check. The visible verification sources include:

• information in the user task
• the privileged LLM's parametric knowledge
• trusted contacts, such as e-mail addresses and GitLab handles
• trusted domains, such as registries, vendor docs and listings

Evidence

• Evaluated on WASP†, RTC† and the new RTC-value.
• SA-CUA has the highest completion rate among the defenses at 0% attack success.
• The specific numbers shown are the two above, both for Qwen3.8-27B.
• The fetched text does not include full tables, other models, or results per benchmark.

Limits

• Setup cost. The user must build an allowlist of trusted domains and e-mail contacts once. Rules must be derived offline for each app or domain before deployment.
• Threat model exclusions. Compromise of a trusted source is out of scope. File trust is tracked only per episode: downloaded files are untrusted.
• Accessibility-tree gaps. aria-hidden and aria-owns mean the tree can misrepresent what is rendered. By design this lowers utility rather than safety, but expect masked-out controls.
• Incomplete page. The text stops partway through the rule-derivation section. Full results, ablations and stated limitations were not visible.
• Not stated. No code, licence or run instructions appear in the fetched text. The paper is a preprint under review.