Linguistic Loopholes in LLM Unlearning: From a 174-Language Benchmark to Coverage-Aware Unlearning

What it claims

Standard unlearning methods close the fact in the language they were applied in, but often leave it open in other languages. On the Cross-Lingual Unlearning Tensor, fewer than 20% of alternative query–answer routes reach the same closure level as the unlearned language. RMU, for example, leaves 75.4% of access in its worst language. Choosing which K languages to unlearn in matters, and picking the individually strongest sources does not reliably produce a strong set. COVER picks the set by predicted coverage instead. The paper also finds that unlearning appears to suppress surface forms rather than meaning: changing script or paraphrase type reopens the fact.

How it works

• Residual-access metric. R = (p_U − p_O) / (p_FT − p_O), where p is the teacher-forced, length-normalized likelihood of the gold answer. U is the unlearned model, O is a retain-only oracle and FT is the fine-tuned model. R = 0 matches the oracle and R = 1 means nothing was removed.
• Matched-forgetting protocol. All methods are compared at the first checkpoint where source-language progress reaches a mean of at least 0.95 and a minimum of at least 0.90 across source languages. This stops a method from looking good by over-forgetting into gibberish, or by barely forgetting. Held-out languages are then scored on mean and max R, extraction strength, an LLM-judged generation label (full leakage, partial leakage, code-switching or gibberish) and chrF++.
• Routes. Each route is a query language → answer language pair. The paper scores ℓ→a (only the query language changes), a→ℓ (only the answer language changes), ℓ→en, en→ℓ and native ℓ→ℓ.
• COVER. 1. Measure how similarly the frozen model represents benign calibration data across languages. This draws on matched-versus-mismatched translation similarity (Zeng et al., 2025). 2. Score each candidate source set by its pairwise alignment to the held-out languages. 3. Coverage can be scored in two ways. Fixed coverage uses the similarities directly. Learned coverage fits a scorer to past unlearning outcomes, so it needs prior runs. 4. Pick K sources, then apply an ordinary unlearning objective.

The page text ends before the scoring formula.

Evidence

Benchmark scale.

• 174 language–script pairs across 18 scripts: 27 high-, 46 medium- and 101 low-resource.
• 25 ETPC paraphrase types; seven of them are translated into 40 languages, giving 1,690 rows per language.
• Built on TOFU (10% forget split) and LUME.
• 103 fine-tuned checkpoints on Tiny Aya, Qwen3 and Llama 3, each with a retain-only oracle.
• 287 unlearned checkpoints using SimNPO, RMU and GradDiff.

Acquisition. After fine-tuning in language a, knowledge transfers mostly on the query side. The mean likelihood gain is +0.47 on ℓ→a but only +0.09 on ℓ→ℓ. The share of ℓ→ℓ routes with sufficient access is 65.3% for Aya, 65.5% for Qwen and 29.6% for Llama.

Unlearning (oracle-normalized removal %, Table 2B).

Hardest languages. Most of the hardest-to-unlearn languages are low-resource, but Russian and Chinese also rank as consistent bottlenecks. Languages that gained more access during fine-tuning generally lose more in absolute terms during unlearning.

Surface-form effects.

• Unlearning with Romanized Hindi questions drops the gold-answer likelihood from 0.538 to below 0.001 for Romanized queries. Native-script Hindi queries still score 0.955, down from 0.984.
• Modal-verb paraphrases leave the most residual access in all three model families.

COVER results.

• Reduces mean held-out residual access by 7.8–27.3% versus uniform source selection.
• Can reduce generated disclosure more than taking the top-3 individual sources.
• Supervising all languages increases damage to retained facts and multilingual generation (Table 21, numbers not in the fetched text).

LORELEI transfer. A new LORELEI test: 100 news documents, human-translated into English, Hindi, Spanish, Russian, Chinese, Swahili, Bengali and Vietnamese. Validated on Aya and Qwen only.

How to use it

• Only a data subset and code are available now; the full dataset will be released after peer review. The fetched text gives no links, licence or commands.
• COVER needs the frozen model and benign calibration data at deployment, with no held-out forget examples. The learned-coverage variant also needs historical unlearning outcomes.
• If you evaluate an unlearning method, test across query and answer languages, scripts and paraphrases. Same-language evaluation overstates forgetting.

Limits

• The fetched text cuts off mid-method, so the COVER scoring details, the K values used and per-model gains are not visible here.
• Model sizes are not stated.
• Most results are on synthetic or fictitious QA (TOFU-based). The real-text LORELEI check is small: 100 documents, 8 languages, two model families.
• Only three unlearning methods are tested.