ASPI: Seeking Ambiguity Clarification Amplifies Prompt Injection Vulnerability in LLM Agents

What it is

ASPI (Ambiguous-State Prompt Injection), from Scale AI and collaborators, is a 728-scenario benchmark built on AgentDojo that measures how an LLM agent's prompt-injection vulnerability changes when it enters a clarification-seeking state. The summary gives the headline attack-success jumps; this document carries the mechanism, the domain split, the defenses tested, and their residual failure rates.

How it works

ASPI holds task, attacker goal, environment, and scoring fixed while varying only how adversarial content is delivered. From a fully specified AgentDojo task it removes exactly one user-provided slot (recipient, date, identifier, destination), forcing the agent to call ask_user. The attacker then injects instructions into the agent's own clarification reply channel.

The paper's key analytical result — absent from the summary — is a decomposition of the vulnerability gap into two factors: a state effect (how the model processes content once in clarification mode) and a channel effect (the ask_user solicited-response interface). The channel effect is the consistent primary driver across models; the state effect alone is heterogeneous. The mechanism: because the agent explicitly requested the input and expects it to be task-relevant, it collapses the trust boundary between instructions and untrusted data.

Preconditions / Affected

Applies to tool-using agents that ask clarifying questions. Scope is deliberately narrow: single-slot ambiguity with one-round ask_user. The authors note multi-slot, multi-turn, or open-ended ambiguity would expand the injection surface, so reported rates are an underestimate. Domain breakdown of the 728 instances: 407 Workspace, 97 Slack, 94 Travel, 130 Banking. An additional data point beyond the summary: Kimi K2.5 rises 11.1% → 63.1%.

What to do

The paper adapts and evaluates two lightweight defenses (rather than applying them naively):

• A segment-level prompt guard that scans both user and tool messages while preserving benign clarification content.
• An ask_user-aware tool filter that fires at the last safe boundary before agent action, preserving clarification ability.

Both reduce but do not eliminate the vulnerability. Residual attack success persists under defense — e.g., Gemini-3-Flash sits at 27.0% under prompt guard vs 35.7% undefended. The actionable takeaway: input filtering and tool restriction do not close the clarification attack surface, so treat the ask_user return channel as untrusted regardless of guardrails. Data and code: https://github.com/scaleapi/aspi.

Caveats

The 27.0% residual is one model under one defense; the paper frames these as mitigations, not fixes. Ambiguity is model-generated (Gemini-3.1-Pro proposes slot removals) then validated by a QC pipeline, so benchmark realism depends on that pipeline. Results are directly comparable to AgentDojo by construction but inherit its four-domain scope.