PISmith: Reinforcement Learning-based Red Teaming for Prompt Injection Defenses

What it is

PISmith is a red-teaming framework from Penn State (Chenlong Yin, Runpeng Geng, Yanting Wang, Jinyuan Jia), published at COLM 2026 (arXiv 2603.13026). It trains an attack LLM with a modified GRPO to write injected prompts against a defended target LLM. The attacker needs only query access and the target's outputs. Code is at https://github.com/albert-y1n/PISmith.

How it works

The attack policy takes a target task (instruction plus data context) and an injected task, such as embedding a phishing link in the response. It outputs an injected prompt that is inserted into the context. The reward is binary: 1 if the target's response fulfils the injected task, 0 otherwise. For each input, K prompts are sampled and scored.

The paper reports two failure modes when standard GRPO meets a strong defense:

• Almost every rollout fails, so the few successes get over-reinforced. Policy entropy collapses, rollouts become homogeneous, and gradients vanish.
• Even with exploration kept up, the rare successes are diluted by the many failures in the group.

Two fixes, both keyed to the current batch's mean reward r̄:

• Adaptive entropy regularization. An entropy bonus applies only when entropy falls below a cap (example value H_cap = 0.5). Its coefficient β rises toward β_max as r̄ drops below a threshold τ and decays to β_base above it. The cap exists because an unconditioned bonus dominates early training, when advantages are small, and produces incoherent prompts.
• Dynamic advantage weighting. Advantages of successful rollouts are multiplied by γ(r̄), which grows toward γ_max (example value 5.0) as r̄ falls below τ and equals 1 above it. At high r̄ the method reduces to plain GRPO.

The paper contrasts this with RL-Hammer, which handles sparsity by training jointly against undefended and defended targets. PISmith's authors suggest that mismatched signals between those targets explain RL-Hammer's weaker results.

Preconditions / Affected

Using it requires a way to query the defended model and a programmatic check for whether the injected task succeeded. The paper trains against Meta-SecAlign-8B in Figure 1, where standard GRPO's mean reward stays near zero over 500 steps while PISmith's rises. Defenses tested include filter-based ones (DataSentinel, PromptGuard) and prevention-based ones (Meta-SecAlign).

What to do

If you ship a prompt-injection defense, or rely on one, run PISmith from the repository against it instead of trusting static or template-based test results. The paper argues that near-zero attack success rates under non-adaptive attacks can give a false sense of security.

Caveats

The fetched text ends partway through Section 3.3.3, so it contains none of the results tables. The summary's claims of highest attack success across 13 benchmarks, 8 defenses and 7 baselines, and the InjecAgent and AgentDojo results on GPT-4o-mini and GPT-5-nano, could not be checked against numbers here. Training cost, the values of τ and β_max, and the attacker model size are not in the available text. The paper also states that the tested defenses cannot keep benign utility high and resist adaptive attacks at once, but the supporting data is not in the fetched portion.