Passing the Test You Trained On: Re-evaluating Prompt-Injection Detectors for LLM Agents

What it is

This paper (arXiv:2610.03448) evaluates fifteen prompt-injection detectors plus two LLM judges by replaying ground-truth tool calls from AgentDojo and τ-bench without an LLM, producing certified-benign outputs, and labeling injected outputs via differential replay against clean environments. The full detector list and names (PIGuard, Prismor, Horizon-Labs, Prompt Guard 1/2, ProtectAI, Sentinel models, Sheltron, Wolf Defender, two task-aware judges) and the specific numbers behind the summary's claims are the actionable payload here.

How it works

Two detectors are named and quantified as opposite extremes: PIGuard catches 95.1% of BIPIA injections at 1% FPR but only 2.1% of AgentDojo injections; Horizon-Labs (trained on agent-style inputs, sharing no data with any benchmark) leads both agent benchmarks at 82.2% (AgentDojo) and 100.0% (τ-bench) with almost no false alarms. A training-data audit explains PIGuard's failure concretely: it saw 53 of 62 InjecAgent attack strings during training, but only as short standalone prompts — inside τ-bench tool outputs it detects those "seen" attacks no better than unseen ones (52.1% vs 55.8%), proving exposure to attack text doesn't transfer across input format. Two detectors (deepset/deberta-v3-base-injection and fmops/distilbert-prompt-injection) were excluded because they flag every benign README chunk. Task-aware LLM judges (Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, both 4-bit) score only 54-78% on agent benchmarks, below dedicated detectors, and the paper notes this gap stems partly from the judge's fixed question covering some attack goals (task-irrelevant) but not others (action/task-relevant goals).

Preconditions / Affected

Teams currently choosing detectors based on BIPIA, InjecAgent-prompt, or other non-agent benchmark leaderboards; any deployment using PIGuard, Prompt Guard 1/2, or similar DeBERTa-era classifiers against serialized JSON/YAML tool outputs rather than short prompt strings.

What to do

The paper states three concrete evaluation requirements for teams selecting a detector: (1) test on the agent's own serialized tool outputs, not prompt-string benchmarks; (2) report TPR at a low FPR operating point (1%), not aggregate accuracy; (3) audit what the detector's training data actually looked like (full documents vs. short prompts) before trusting benchmark rankings. Reproduction code and evaluation sets are published at the paper's linked GitHub repo (lzwhehe/benign-instruction-bench).

Caveats

The paper explicitly states its detection numbers are upper bounds — it tests only static attacks shipped by public benchmarks, and notes an adaptive attacker targeting a specific detector would lower detection further. The training-data audit was only possible for two detectors (PIGuard and Horizon-Labs) whose training data/assembly scripts are public; other detectors' overlap is unverified beyond model-card claims.