Identify, Classify, Act: Per-Instruction Controlled Compliance for LLMs over External Data

What it is

EDIC (External-Data Instruction Classification) trains an LLM to handle documents or tool returns that mix useful directives with prompt injections. The model follows beneficial embedded instructions and rejects harmful ones, instead of rejecting or trusting everything. The paper is from Harbin Institute of Technology, Shenzhen (arXiv 2610.22874, 16 October 2026). Code is at github.com/yingyingxia666/EDIC.

How it works

Each embedded snippet is labeled beneficial, harmful or contextual. The model must identify instructions, classify them, then follow or reject each one.

Training data is built from real documents in four scene types: email (a Kaggle spam dataset), API docs (API-Bank), medical guidance (EPFL clinical guidelines) and knowledge QA (HotpotQA, Wiki-MultihopQA). Each source example gets three variants: beneficial-only, harmful-only and mixed. Harmful instructions carry an injection trigger such as "Ignore previous instructions". Each variant has one correct output and error outputs for over-conservative, injection-hijacked and partial-redirection behavior. Counterfactual checks drop samples where removing the beneficial instruction does not change the response, or where removing the harmful one does.

A generative judge takes the query, data, one candidate instruction and the model output, and returns a reasoning trace plus a binary "followed or not". It is trained with SFT, then GRPO using a correctness reward and a format reward.

The policy model gets SFT on chain-of-thought targets, then RL. The reward sums judge scores for beneficial instructions, subtracts λ times judge scores for harmful ones, and adds α times a format reward, with a KL penalty to the SFT checkpoint. Because the data mixes all three cases, always refusing or always obeying cannot maximize reward.

Results

• Qwen3-4B: 91.2% harmful-rejection and 89.5% benign-follow.
• Llama-3.1-8B: 92.8% harmful-rejection and 91.7% benign-follow.
• The 7.7 and 12.5 figures are percentage-point gains over ReasAlign, the strongest reasoning-based baseline, not relative gains. The Arena-Hard-Auto gap over ReasAlign shrinks to 4.6 points on Llama-3.1-8B.
• Baselines: SecAlign, StruQ, PromptGuard, Instruction Hierarchy, Meta-SecAlign (Llama only) and ReasAlign. The paper says SecAlign, StruQ and Meta-SecAlign gain safety by sharply cutting benign compliance.
• Evaluation covers MT-Bench, Arena-Hard-Auto, AgentDojo, CyberSecEval 2 (prompt injection) and InjecAgent.

Caveats

• On Llama-3.1-8B, EDIC leads on benign-follow but only matches the strongest baseline on harmful-rejection.
• Training uses 900 filtered examples per scene (150 validation, 250 judge training, 500 policy training). Cold-start reasoning traces were synthesized by GPT-5.4.
• The fetched text cuts off at Table 1, so per-benchmark numbers, the ablations and the appendices (hyperparameters, prompts) are not covered here.