Meta’s AI-distilled model detector catches 79% at a 0.1% false-positive rate

What it claims

WM-Basin, from Meta FAIR and Meta Superintelligence Labs, detects student LLMs distilled from a watermarked teacher using only text the suspect model returns through an API. It reports a 79% true-positive rate at a 0.1% false-positive rate with n-gram=1, against 29% for the strongest prior black-box token-level detector under the same conditions. The paper is "Watermark Basins: Black-Box Detection of Distilled LLMs" (arXiv:2610.37266, October 2026).

How it works

The threat model is specific. Someone collects responses from a watermarked teacher through its API, then trains a student on them. The detector needs no student weights or logits.

1. Collect text generated by the suspect student.
2. Pass that text into a fixed, clean probe model.
3. Convert it into gradient representations, which show how training on the text would move the probe's parameters.
4. Separate ordinary fine-tuning directions from alignment with a key-specific "watermark basin" in parameter space.

The new element against token-level detectors is step 4. The signal is read from parameter-space movement in a probe, not from token statistics in the text itself.

Evidence

The evaluation covers 1,200 student models from 135 million to 8.2 billion parameters, across six student architecture families. Students were trained on mathematical reasoning traces from OpenR1-Math, with multiple teacher models and several watermarking schemes. The authors report that WM-Basin's lead over competing detectors widens as the detection threshold gets stricter and as the n-gram window grows.

The signal survived post-distillation quantization, pruning and reinforcement learning. WM-Basin also stayed more reliable than token-level detection where token signals weaken, including under low-temperature sampling.

Limits

All reported results come from a mathematical-reasoning setting (OpenR1-Math traces). The page gives no results for other domains or for students larger than 8.2 billion parameters. Extensive continued supervised fine-tuning on clean, unwatermarked data degrades detection. Paraphrasing the teacher's reasoning traces before training is another evasion route the authors identify. The method only applies when the teacher was watermarked and the key is known. The page is a news article that was written with AI assistance. It gives no code, compute cost, probe model identity or release details.