Distillation Defenses Easily Break After Reinforcement Learning

What it claims

Prior work evaluates distillation defenses at the wrong checkpoint. They measure the attacker's model right after supervised distillation. The paper argues that a realistic attacker follows distillation with RL, because in its experiments distillation followed by RL beats either method alone. Two consequences follow:

• Antidistillation sampling (Savani et al., 2025) can look effective after distillation, but the gap can vanish after RL.
• Summaries instead of full reasoning traces, the defense GPT, Claude and Gemini currently deploy, is broken by a simple attack. The attacker reconstructs approximate traces from the summaries and answers, distills on them, then runs RL. This gives gains similar to distilling on the full hidden traces.

The authors' stated rule: any defense that leaks enough to approximately reconstruct traces is likely ineffective. Withholding steps instead would hurt legitimate users, such as a researcher who asked for a proof.

How it works

The setup uses the datasets, prompts and GRPO-based RL framework of Zeng et al. (2025). Distillation and RL use the same question set. Distillation is sequence-level KD, meaning SFT on the teacher's generated answers. The teacher is Qwen2.5-14B-RL, except Qwen2.5-1.5B-RL for the 0.5B student. The teacher is an open RL-trained model standing in for the closed model.

flowchart LR
  T[Defended teacher outputs] --> D[Distill student]
  D --> E1[Eval: prior work stops here]
  D --> R[GRPO RL]
  R --> E2[Eval: this paper]

The summary-reconstruction attack uses an expander that requires no training. Zhang et al. (2026) trained one. The authors say their version resembles publicly reported real attacks.

The paper's account of the mechanism:

• Distillation is mode-covering. It raises pass@k at high k, which the paper treats as the model's "reasoning boundary".
• RL is mode-seeking. It raises pass@1.
• So distillation bootstraps RL.

Evidence

• Figure 2: RL vs distillation. Students were Qwen2.5 0.5B/1.5B/3B/7B, Qwen2.5-Math-7B, Llama-3.2-3B, Llama-3.1-8B and Gemma-2B. Accuracy is averaged over GSM8K, Minerva Math and MATH500.
• RL beats distillation in almost all cases. The exception is Qwen2.5-Math-7B, which the paper treats as an outlier (Appendix A.3).
• Distillation followed by RL beats both.
• Exact bar values are not in the fetched text.
• Figure 3: pass@k for k=1–128. On Llama-3.2-3B, Qwen2.5-3B and Gemma-2B, distillation beats RL at high k. Distillation followed by RL is best at pass@1.
• Antidistillation case study. The student is Qwen2.5-0.5B and the teacher is Qwen2.5-1.5B-RL. Three poisoning levels were tested; they reduce the teacher's relative performance by 10%, 30% and 86%. The resulting student numbers are cut off in the fetched text.

Limits

• Small models only. Distillation followed by RL was run only on models of at most 3B parameters, because of compute limits.
• Math only. All results are on math benchmarks.
• Proxy teachers. Sections 3–4 use open proxy teachers, not the actual closed APIs.
• Model-family caveat. The authors note that some Qwen families give RL conclusions that don't generalize. They say they tested across families for this reason.
• Truncated source. The fetched text ends partway through Section 4. The following are absent, so verify them in the full paper before relying on them:
• the antidistillation post-RL numbers
• all Section 5 summary-attack results
• the batch-level defense discussion
• Disclosure. Results were disclosed to Google, OpenAI and Anthropic, who approved publication. The authors frame the work as threat modeling, not a new attack.