INFUSE: A Unified Surface Benchmark for Indirect Prompt Injection in Agentic Systems

What it is

INFUSE is a benchmark for indirect prompt injection (IPI) against terminal/coding agents. It pairs 40 long-horizon tasks with 56 attack scenarios, all run in isolated Docker sandboxes with full behavioral tracing. Code: https://github.com/judydnguyen/infuse. The authors are from Vanderbilt.

How it works

• Tasks: five categories: debugging, security/risk mitigation, file/data management, system deployment, application development. Each has a turn budget, tool allowlist and cost cap. A deterministic test.sh harness emits a binary success flag.
• Attack taxonomy (counts from the page): Data Exfiltration 11, Unauthorized Data Manipulation 13, Privilege Escalation 12, System Corruption/Safety Bypass 10, Supply Chain 10. Each attack has a payload template, compatible surfaces, a position strategy (APPEND, PREPEND, INLINE, REPLACE) and machine-verifiable success criteria.
• Injection engine: payloads are planted in five targeting categories (file names, file content, config files, terminal output, archives). Every observation is tagged with one of 12 read-side runtime surface types. Variants include single or multi-surface, persistent or one-shot, and plain or GCG-obfuscated. Surfaces are split into static (pre-placed) and dynamic (emerging from stdout, error traces, generated artifacts).
• Scoring: an LLM judge (ASR-L) plus heuristic verifiers, with utility measured on benign versus attacked runs.

Findings that go beyond the summary

• By category (mean ASR-L): Data Manipulation 22.8%, Malicious Dependency 20.8%, System Corruption 19.8%, Exfiltration 16.5%, Privilege Escalation 15.5%. Nothing is safe, and the spread is modest.
• Refusal is almost absent: the refusal rate is ≤1.0% for every model, so agents fail by complying, not by refusing.
• Model examples (ASR-L, Fig. 1b): Sonnet 4.6 3.2%, GPT-5.4 12.6%, Qwen2.5-32B 25.7%, GPT-5 26.3%, GPT-4o-mini 33.8%, Gemini 3.1 Flash-Lite 41.9%.
• Utility under attack: most models lose utility. Examples are Gemini 3.1 Flash-Lite −23.2 pp and Sonnet 4.6 −20.0 pp. A few gain slightly, e.g. GPT-5 (+1.8 pp, per the Fig. 1b label).
• Model-level predictors:
• ASR-L correlates with benign trajectory length across models (r≈0.50), but not within models (r≈0.00).
• Its correlation with MMLU-Pro is weak (r=−0.28).
• Benchmark capability is therefore a poor proxy for injection resistance. The authors advise reporting task, harness and budget alongside model identity.
• Guard models: external chat-safety classifiers transfer poorly to environment-embedded payloads, leaving ASR-L at 43.8–44.0% on a high-ASR stress subset.
• Prompt hardening: this cut ASR-L from 58.2% to 14.3% (−43.9 points), but utility trade-offs are uneven across models.

What to do

• If you ship a terminal or coding agent, run your own model and harness through the repo. Per-model results vary 3.2–41.9%, so published numbers for a different scaffold will not transfer.
• Do not rely on chat-oriented guard classifiers for tool output, file contents or filenames. Test them on environment-embedded payloads.
• Prompt-level hardening is the cheapest measured mitigation. Re-measure task utility per model after applying it.
• Treat dynamic surfaces (stdout, error traces, generated files) and filenames as untrusted input, since the benchmark injects there.

Caveats

• The fetched text is truncated mid-methodology. The evaluation protocol details, defense list and 12 surface types (Appendix A.2.2) are not visible here.
• The 58.2%→14.3% hardening result is on a high-ASR subset or configuration. The page excerpt does not say which models or attacks it covers.
• Headline ASR differs by evaluator: 19.6% LLM-judged versus 15.4% heuristic.
• The "first benchmark" claim is the authors' own.
• This is a v1 preprint and has not been independently reproduced.