Skip to content

Back to the catalog

Prompt Injection Detection for Email Agents Through Attack Chain Modeling
arxiv.orgpaper

Prompt Injection Detection for Email Agents Through Attack Chain Modeling

This paper proposes an attack-chain-based detection framework for indirect prompt injection in LLM email assistants, combining a text detector, stage-specific verifiers, rule-based risk signals, and user-intent/action consistency checks scored via a logistic decision policy, rather than treating detection as binary malicious-text classification. Evaluated under random splits, temporal and cross-dataset transfer, and ablations across five binary benchmarks, it achieves a mean F1 of 0.406 versus 0.216 for the best of five existing pretrained detectors, and shows random splits overestimate robustness under distribution shift while training on hard benign (attack-resembling) emails reduces false alarms.

TL;DR Attack-chain modeling (stage verifiers + intent-consistency checks) roughly doubles F1 over existing prompt-injection detectors for LLM email agents.
Open original ↗
Document

What it is

An academic paper (arXiv:2609.30657, Hashmi/Patel/Yin, Eastern Michigan University, Sep 2026) with code at github.com/AcaiLab/Email. It reframes indirect prompt-injection detection for email agents away from binary "is this text malicious" toward modeling the multi-stage attack chain, and reports what that decomposition reveals. The summary gives the headline F1 (0.406 vs 0.216); the page carries the stage structure, operating points, and the numbers an engineer needs to judge deployability.

How it works

Attack-chain labels are parsed from the Microsoft LLMail-Inject challenge data into six stages: (1) email retrieved, (2) defense bypassed, (3) tool called, (4) destination correct, (5) content correct, (6) full success. The framework stacks a broad TF-IDF attack detector (trained on LLMail+BIPIA+NotInject), per-stage verifiers, rule-based risk signals (instruction-override, tool-action, obfuscation patterns), and action/context consistency checks (comparing the user request against retrieved content) into a logistic meta-policy. Notably the base detector is a linear logistic classifier on TF-IDF n-grams, not a neural model.

Two findings the summary omits:

  • Later tool-argument stages are more predictable than earlier stages — destination/content correctness is easier to model than retrieval/bypass.
  • The action+context signals are what move the needle: ablation raises mean F1 from 0.348 to 0.406 when those signals are added to all stages.

Preconditions / Affected

Relevant to anyone building or defending an LLM email/tool agent that retrieves untrusted message content into context before tool use. The severe class imbalance is the operational reality: full-success attacks are 0.814% of submissions in LLMail Phase 1 and 0.337% in Phase 2 — high accuracy is trivial, precision/recall is not.

What to do

The paper defines two operating points you can adopt:

  • Review-oriented screening — threshold set to maximize validation F1; best config uses selective stage verification and drops the sparse full-success verifier (it acts as noise).
  • Strict block/confirm — threshold set on the NotInject hard-negative set for low FPR; best config uses all stage verifiers plus action/context consistency.

Adopt hard-negative (attack-resembling benign) training to cut false alarms, and evaluate with temporal phase transfer (train Phase 1, test Phase 2) rather than random splits, which overstate robustness. Baselines to beat/compare: ProtectAI DeBERTa v1/v2, deepset DeBERTa, TestSavant tiny, Arkaean DistilBERT, plus a gpt-4.1-mini zero-shot judge.

Caveats

The absolute number is the caveat: best mean F1 is 0.406. This is a research prototype demonstrating that stage modeling beats frozen binary detectors — not a production-ready detector. Evaluation is over public benchmarks (BIPIA, NotInject, NVIDIA Agentic IPI, PromptShield, ShieldLM, Neuralchemy), not a live agent.

Technologies named here

  • LLM email assistants / email agents
  • Proposed attack chain modeling detection framework
  • LLM

Techniques named here

  • Indirect prompt injection via email content

Related in AI Security