SecPO: Principled Adversarial Training for Prompt Injection Security

What it claims

SecPO (Sizhe Chen, Yibo Peng, Jaewon Chang, Chawin Sitawarin, David Wagner; arXiv, October 2026) argues that Meta-SecAlign stays breakable for two reasons. First, its DPO loss anchors training to the insecure undefended model. Second, it trains only on weak, static attack templates. The paper fixes each one separately and measures each fix. Adversarial training (AT) alone cuts ASR by 1.5 to 14.5× against Meta-SecAlign across three adaptive attacks. Swapping DPO for SecPO cuts it by a further 2.4 to 4.6×. The highest ASR from three state-of-the-art adaptive attacks is 13.3% for the defended Qwen3.6-27B, against 94.0% for Meta-SecAlign.

How it works

The paper defines an oracle reference model. On injected input, the oracle behaves as if the injection were absent. On clean input, it matches the undefended model. These two conditions combine to π_oracle(·|x_inj) = π_undef(·|x_clean). That quantity can be computed during training, because x_inj is built from x_clean. The SecPO loss is therefore DPO with one change: the reference log-probabilities for the preferred and rejected responses use the clean input instead of the injected one.

flowchart LR
  C[clean sample] --> I[injected sample]
  I --> P[policy log-probs]
  I -->|DPO reference| R[undefended model]
  C -->|SecPO reference| R
  P --> L[preference loss]
  R --> L

The gradient analysis shows that SecPO and DPO point in the same direction for each sample. SecPO rescales each sample by a factor that grows with Δ_undef, which measures how far the injection pushes the undefended model toward the attacker's response. Injection-sensitive samples therefore get more weight. The authors state this is not the same as DPO with a larger learning rate, because the scaling differs per sample.

Training data follows the Meta-SecAlign recipe. Another instruction sample is injected at the start or end of the data. The preferred response is the undefended model's output on the clean input, and the rejected response is its output on the injected instruction. AT replaces the static template with injections optimized by PAIR (best@240). Untrusted data sits in a dedicated "input" message role, separate from the "user" role.

Evidence

• In a Llama-3.1-8B-Instruct run without AT, using the same data and order, DPO drives the insecure response's sequence log-probability to about -400. SecPO drives it to about -700.
• AgentDojo ASR@800 under the AttackerMovesSecond attack is 8.8% for SecPO + AT, against 78.8% for Meta-SecAlign. ASR@800 means 800 attempts per task.
• DTaP-Bench ASR is 7.6%.
• Figure 1 places the standalone model at roughly GPT-5.5's security level. The authors note that GPT-5.5 combines several proprietary defenses and is at least two orders of magnitude larger.
• Utility stays close to the undefended model on nine benchmarks: agentic tool calling (three), instruction following (two), science (two) and math (two).
• The model also resists adaptive visual prompt injections, which the training did not target. Appendix Table 18 reports better robustness to GCG and to one other white-box attack.

How to use it

Code, model weights and the AT training set are released. The training set includes the optimized injections. Training on it skips the expensive attack-generation step and still gives 2.5 to 22.5× lower ASR in the authors' study. Implementing the loss on an existing DPO pipeline only changes the reference forward pass to use clean inputs. Deployment assumes the serving stack wraps untrusted content in a separate role that injected text cannot escape. The paper cites OpenAI Harmony as an example of a library that enforces this. The fetched text gives no install commands, repository URL or licence.

Limits

The threat model assumes the agent is told which inputs are trusted. It excludes three cases: trust labels inferred from context, instructions in data that should be followed because they match the user's intent, and malicious users (jailbreaks, direct injection). White-box gradient attacks are out of main scope, on the grounds that perplexity filters catch them, and appear only in an appendix. The paper notes that attackers can still use fake delimiters or fake responses inside the data role. Full AT is described as computationally expensive. The fetched page is cut off mid-Section 3, so ablations, compute costs and the authors' own limitations section were not available.