Skip to content

Back to the catalog

PIDS-Bench: Evaluating Prompt-Injection Detectors Under Over-Defense, Obfuscation, and Distribution Shift
arxiv.orgpaper

PIDS-Bench: Evaluating Prompt-Injection Detectors Under Over-Defense, Obfuscation, and Distribution Shift

PIDS-Bench is a frozen, multi-axis benchmark for prompt-injection detectors that jointly measures attack detection and benign false-positive rates across in-distribution inputs, hard-benign prompts mimicking injection structure, obfuscated attacks, and domain/structural distribution shifts. Testing seven detectors (learned baselines, external prompt-injection classifiers, broad-safety comparators) plus a rule-based baseline, the authors show a detector with F1 > 0.98 on held-out data still misclassifies roughly one-third of an externally-sourced benign security-adjacent corpus, and that hard-negative augmentation reduces over-defense only on curated stress inputs, not on externally-sourced ones — a pattern they call provenance-sensitive over-defense.

TL;DR A new benchmark shows prompt-injection detectors with near-perfect F1 still false-positive on ~1/3 of real-world benign security-adjacent text, and hard-negative augmentation doesn't fix it for externally-sourced prompts.
Open original ↗
Document

What it is

PIDS-Bench (Shire and Kim, Kookmin University; IEEE Access vol. 14, 2026, DOI 10.1109/ACCESS.2026.3728186) is a frozen, checksum-verified benchmark that evaluates prompt-injection gateway detectors on both error sides at fixed thresholds. The summary has the shape; what it omits is the specific detectors tested and the numbers that tell an engineer how bad the false-positive problem is.

The detectors and the numbers

The seven include named off-the-shelf components an engineer might actually deploy: ProtectAI DeBERTa-v3-base-prompt-injection-v2, DeBERTa-PI, PromptGuard 2 (86M), and Llama Guard 3-1B as a broad-safety comparator. Internal baselines are DeBERTa-v3-FT, DistilBERT, and TF-IDF + logistic regression.

Across a full threshold sweep (τ ∈ [0,1]) and five seeds, no internal detector reaches F1 ≥ 0.95 and externally-sourced hard-benign FPR ≤ 0.10 together. The floor on externally-sourced FPR, at each architecture's most favorable point:

  • DeBERTa-v3-FT: 0.226 ± 0.058
  • DistilBERT: 0.277 ± 0.099
  • TF-IDF + logistic regression: 0.241

The validation-tuned threshold is itself unstable — for DeBERTa-v3-FT, τ ranges from 0.01 to 0.904 across seeds — yet the constraint fails at every threshold and seed, which the authors read as a learned-feature problem, not a calibration one (stated as interpretation, not established mechanism).

The two distribution-shift failure modes, separated

The benchmark splits what a single shift score conflates:

  • Attack-side recall collapse: DistilBERT's recall on code-domain inputs falls to 0.435 while staying near-perfect elsewhere.
  • Benign-side FP inflation: TF-IDF hits a legal-domain FPR of 0.252 with recall unchanged.

Obfuscation is tested via leetspeak, homoglyph, and zero-width transforms. A further decomposition (Section 6.4) finds injection-like surface structure is the dominant driver of over-defense within every provenance, with source origin a smaller, statistically independent effect.

What to do

If you rely on any of the named classifiers as a blocking gateway, treat the ~one-third benign block rate on security-adjacent content as a real usability cost, and do not expect hard-negative augmentation or threshold tuning to fix it on real-world (externally-sourced) traffic — the paper shows both fail there. Splits and SHA-256 checksums are released in FREEZE_MANIFEST.txt for reproduction.

Caveats

The FPR figures are measured on stress distributions built to surface over-defense, not to estimate deployment FPR. Whether augmentation matched to the externally-sourced distribution would close the gap is untested — only curated-style augmentation and threshold calibration were shown to fail. The dataset instantiates the protocol and is not a general-purpose corpus.

Technologies named here

  • PIDS-Bench
  • Seven detectors
  • Rule-based lower-bound reference
  • LLM

Techniques named here

  • Prompt Injection
  • Obfuscated prompt-injection attacks
  • Hard-benign prompts
  • Hard-negative augmentation
  • Distribution shift evaluation

Related in AI Security