Skip to content

Back to the catalog

From Detection to Decision: Calibration and Deferral for Prompt-Injection Detection
arxiv.orgpaper

From Detection to Decision: Calibration and Deferral for Prompt-Injection Detection

Evaluates a native probabilistic decision model (Jev), Llama Prompt Guard 2 (22M and 86M) and an LLM judge on 47,611 held-out examples covering direct/indirect prompt injection, hard-benign inputs, obfuscation and distribution shift. The LLM judge has the best AUROC (0.897), followed by Jev (0.871), Prompt Guard 2 86M (0.757) and 22M (0.714). Jev's native probabilities do not consistently beat post-hoc calibrated classifier scores, and temperature scaling improves calibration but not discrimination. At a 1% false-negative-rate target, every system needs over 91.8% escalation, so probabilistic outputs alone do not give reliable security decisions under distribution shift.

TL;DR Prompt-injection detectors, even calibrated ones, can't support reliable allow/block/defer decisions under distribution shift; a 1% FNR target forces over 91.8% escalation.
Open original ↗
Document

What it is

An eBay evaluation of whether "calibrated" probabilities from a decision model (Jev, via the TypeSafe SDK, jev-latest, no post-hoc calibration) yield better allow/block/defer decisions for prompt-injection gating than a conventionally calibrated classifier. Comparators are Llama Prompt Guard 2 (22M, 86M) and a GPT-5.5 LLM judge. The confirmatory cohort is 47,611 examples, with calibration and prompt choices frozen beforehand.

What the numbers add beyond the summary

The 0.5 boundary is unusable for most systems. FNR at 0.5: Jev 64.9%, PG2-86M 93.2%, PG2-22M 90.8%, LLM judge 17.1%. The judge pays for that with 13.0% FPR (Jev 4.8%). TPR at 1% FPR is only 12.5% for Jev, 8.4% for the judge and about 1% for PG2, so none supports autonomous blocking at a low FPR.

Calibration is a mixed picture. Jev (ECE 0.541, Brier 0.440, NLL 1.432) is roughly tied with temperature-scaled PG2-86M (0.571, 0.439, 1.104). Jev wins ECE narrowly and loses NLL clearly. The raw judge has the best ECE and Brier (0.221, 0.163). The calibrated judge has the best NLL (0.550).

Confidently wrong on misses. Mean p(attack) on false negatives: Jev 0.163, raw PG2-86M 0.0107, calibrated PG2-86M 0.287. Temperature scaling softens these scores but recovers no missed attacks.

Deferral cost. Escalation needed to reach system FNR ≤ 1%:

GateEscalationSystem FPRMean latency
PG2-86M91.89%40.55%3,713 ms
PG2-22M92.45%43.69%3,734 ms
Jev92.55%45.84%3,885 ms

The spread is under 1 point, and the authors decline to rank the gates without uncertainty intervals.

Why escalation is so high. The reviewer (the judge) uses an FNR-first threshold of 0.0527, which gave FNR 0 but FPR 98.6% on its calibration split. Near-total escalation therefore makes the system behave like that reviewer. The deferral curves describe the gate–reviewer pair, not the gate alone.

Latency (mean / p95): PG2-86M 43.8 / 69.7 ms, PG2-22M 45.8 / 76.1 ms, Jev 190.9 / 301.7 ms, judge 3,890 / 6,604 ms. The API figures include network overhead.

What to do

  • Do not trust a vendor's "calibrated" claim. Fit temperature scaling on your own workload; it improved PG2's NLL from 5.064 to 1.104 for 86M.
  • Do not expect calibration to fix discrimination. Scaling is monotonic, so AUROC and decisions at 0.5 are unchanged.
  • Tune the reviewer's operating point together with the gate. An aggressive reviewer threshold drives the FPR.
  • Evaluate discrimination, calibration, reviewer behavior and cost jointly, as the paper recommends.

Caveats

  • The cohort is 88.2% attacks, mostly BIPIA (36,075 of 41,981). TensorTrust is attack-only and NotInject benign-only. Pooled AUPRC, ECE and Brier reflect this mixture, not natural prevalence.
  • The 91.8%+ crossings are descriptive test-set statistics, not deployable thresholds.
  • The study tests an off-the-shelf native probability against a target-calibrated classifier. It does not isolate the training objectives (RLCD).
  • The fetched text is truncated at the Threats to Validity section, so any further stated limitations are not covered here.

Technologies named here

  • Llama Prompt Guard 2 22M classifier
  • Llama Prompt Guard 2 86M classifier
  • Llama Prompt Guard 2
  • Jev native probabilistic decision model
  • LLM-based judge
  • Temperature scaling
  • LLaMa
  • LLM
  • Jev
  • Prompt Guard 2

Techniques named here

Related in AI Security