Distillation Defenses Easily Break After Reinforcement Learning

What it is

An arXiv paper (2609.35699v1) by Javaheri, Panfilov, Britton, Gal, and Gideoni (Oxford / ELLIS Tübingen) arguing that distillation-attack defenses are evaluated under a broken threat model. The concrete additions beyond the summary are the empirical mechanism, a specific broken defense, and the training-comparison result.

How it works

The core empirical finding (Figure 2, tested across Qwen2.5 0.5B–7B, Qwen2.5-Math-7B, Llama-3.2-3B, Llama-3.1-8B, Gemma-2B): RL alone usually outperforms distillation on accuracy, and distillation followed by RL beats both. So a serious attacker would distill to bootstrap, then run RL — not stop at distillation as current evaluations assume.

The mechanism is a pass@k argument. Distillation raises pass@k at high k (the "soft performance ceiling" / reasoning boundary) even when it doesn't help pass@1. RL then converts that expanded ceiling into pass@1 accuracy. Distillation's cross-entropy loss is mode-covering (spreads probability mass, widening what the model can eventually reach); RL is mode-seeking (concentrates mass on correct answers). Distillation thus expands the space RL then exploits.

Preconditions / Affected

• Deployed defenses that return reasoning-trace summaries instead of full traces — the paper names GPT, Claude, and Gemini as doing this. After RL, distilling on summary-reconstructed traces yields improvements similar to distilling on full traces.
• Antidistillation sampling (Savani et al., 2025) is the case study: it degrades a distilled student by poisoning teacher outputs, and looks effective post-distillation. Tested with poisoning that cut the teacher's relative performance by 10%, 30%, and 86% (Qwen2.5-0.5B student, Qwen2.5-1.5B-RL teacher) — the defense's advantage vanishes after the student gets subsequent RL.

What to do

The paper's actionable conclusion for defenders: any defense that leaks enough information to reconstruct approximate reasoning traces is likely ineffective, because RL closes the gap between approximate and full traces. Don't validate a defense with post-distillation numbers alone — evaluate after distillation plus RL, or it gives a false sense of security. The suggested promising direction is batch-level defenses (Section 6). Trace omission works only by damaging answer quality, which is dual-use (removing proof steps harms legitimate users as much as attackers).

Caveats

• Distillation→RL results are limited to models ≤3B parameters (compute limits on RL); larger-model claims rest on the theory, not experiments.
• Qwen2.5-Math-7B is an outlier where RL underperformed distillation (attributed to that model family).
• Related work: the paper's summary-expansion attack uses an untrained expander (simpler than Zhang et al., 2026's trained one); Panfilov et al. (2026) separately show encrypted/signed reasoning can be replayed into a cheaper same-family decoder that reads the hidden trace verbatim. Results were disclosed to Google, OpenAI, and Anthropic, who approved publication.