Understanding and Mitigating Role Spoofing in LLMs

What it is

A paper by Ezra Edelman, Benjamin Edelman, Surbhi Goel and Eran Malach (University of Pennsylvania, Princeton, Apple), arXiv 2610.40396v1, 29 October 2026. It studies role spoofing: text injected into a tool output that the model treats as a real user message. The page text is truncated partway through Section 2.2, so the probe defense and toy-setting results are described only at the level of the abstract and introduction.

How it works

Chat templates use special tokens that cannot appear in message content, so role should be easy to determine. The attack bypasses this with plain-text lookalikes. A tool response containing a string such as <|im_start|> user [...CONTENT...] is tokenized as ordinary text, yet models often accept it as a genuine user turn, especially when the wording reads like a user request.

The authors argue the cause is out-of-distribution generalization failure. In training data, correct role tracking, content cues and structure cues all agree, so a model can score perfectly while relying on a non-robust cue. Each model tested uses a different mixture of content-based and structure-based mechanisms.

flowchart TD
    A[Tool output with fake role tag] --> B{Model role mechanism}
    B -->|Content cues| C[Style and semantics look like user]
    B -->|Structure cues| D[Forged tokens look like turn boundary]
    C --> E[Treated as user]
    D --> E
    F[Adversarial training on content attacks] -.closes.-> C
    G[Role probe on activations] -.flags.-> E

Preconditions / Affected

The attacker must be able to place arbitrary text in a tool result, for example through a web page or an email. The authors use a permissive threat model with no escaping of the injected text. They note that tools can apply their own filtering or escaping of such strings.

AgentDojo workspace family results, as injection success divided by the rate at which the model follows the same task when the real user issues it:

The introduction also reports user impersonation over 95% of the time in the authors' own tool-use testbed for several models, including Claude Opus 5 and GPT-5.4. The authors' AgentDojo setup differs from the benchmark defaults (their Section B).

What to do

• Escape or strip strings that mimic chat-template markers (such as <|im_start|>user) in tool outputs before they reach the model. The paper suggests this as a tool-level defense but does not evaluate it in the visible text.
• Do not rely on adversarial training alone. In a toy setting and in fine-tuning Qwen3.5-27B, training against content-based injections reduced those attacks but left structure-based spoofing working.
• Consider linear probes on internal activations that predict each segment's role. Flag a message when the model's internal role for a segment differs from its true role (for example, a tool segment the model believes came from the user). The authors say probes trained on conversations with random content tokens separate clean from spoofed examples almost perfectly. They also report lower false positive rates than prior probe defenses (Abdelnabi et al., PIShield) and no need for injected or coherent training data. This needs access to model internals, so it applies to open-weight or self-hosted models.

Caveats

• Success is measured relative to what the model would do for a genuine user request. A refused task does not count as a defense failure, so the numbers show role confusion, not harm.
• A ratio near or above 100% (Claude Opus 5 at 103.6%) means injected requests were followed as often as genuine ones. Absolute rates stay below 62% because the baselines are themselves low.
• Architectural proposals (role embeddings, positional gaps, separate streams) report better injection resistance, but the authors say they were not tested against forged special tokens.
• The probe results come from the abstract and introduction. The visible text does not give the detection numbers or the models probed.