Gold vs.~Translation: Are We Measuring Language, Knowledge, or Locality in Translated Benchmarks?

What it claims

Translated MMLU is a usable proxy for native evaluation only when the native benchmark tests broadly shared curriculum content. Ranking agreement (Kendall's τb) across 29 models runs from 0.89 for curriculum-overlap benchmarks down to 0.51 for SwedishFacts, which covers Sweden-specific people and events. The main test is incremental proxy validity: whether the translated score still predicts native performance after controlling for model size and language proficiency. Across the seven pairings, four effects are robust, two are marginal and one is null. Adding English MMLU as a further control gives the same pattern, so the authors conclude that familiarity with MMLU content alone does not explain the differences.

How it works

Each native benchmark is paired with DeepL-translated MMLU in the same language. The proxies come from EU20-MMLU (Estonian, Lithuanian, Swedish, Danish) and m_mmlu (Norwegian Bokmål), restricted to G-MMLUX. G-MMLUX is a new subset that maps GlobalMMLU's culturally annotated items onto each translation and has 16,701 items across six languages. The seven native sets are:

• Exam-ET: 1,614 items
• INCLUDE-ET: 224 items
• Skolprov: 545 items
• SwedishFacts: 1,292 items, open-ended
• DanishCitizenTest: 668 items
• NRK-Quiz: 3,600 items
• INCLUDE-LT: 534 items

Native subjects are mapped to six MMLU categories rather than to individual subjects, because the taxonomies do not line up one to one. GPT-5.2 category labels agree with reference categories on 85.1% of 3,044 items.

Items are also tagged culturally sensitive (CS) or culturally agnostic (CA), with a required-knowledge type (none, cultural, regional, dialect). The CS share ranges from 14.3% (Skolprov) to 97.8% (SwedishFacts), against 27.8% in GlobalMMLU. Labels at scale come from a GPT-5.2 pipeline with structured JSON output. The prompt was tuned only on GlobalMMLU-Lite (dev/test split of 174/217) and checked against a native reference of 500 items: 100 per language, each rated by three professional native-speaker annotators.

Evidence

• Skolprov: the strongest incremental effect, ΔR² = .128 (95% CI [.038, .253]).
• SwedishFacts: no significant incremental value (p = .242).
• CS/CA pipeline accuracy: 90.6% against the native three-annotator majority (macro-F1 89.5%; CS recall 96.2%, CA recall 80.7%). On the held-out GlobalMMLU-Lite test split it reaches 73.3%, with CS recall of only 61.7%. Implicit cultural cues in English-origin items remain hard to detect.
• Human agreement on CS/CA: Fleiss' κ ranges from .25 (Danish) to .83 (Estonian). Gwet's AC1 ranges from .60 (Bokmål) to .91 (Estonian). The low Danish κ is a prevalence effect: almost all Danish items are CS, and annotators still reach 84% unanimity.

How to use it

The annotations, the annotation pipeline and the analysis code are released:

• Annotations and merged analysis data: https://github.com/gold-vs-translation/cs-annotation-data
• Analysis scripts and annotation pipeline: https://github.com/gold-vs-translation/annotation-and-analysis-scripts

The authors present the framework as extensible to other languages and translation systems. The page gives no licence or run instructions.

Limits

SwedishFacts is both the only open-ended benchmark and the most locally anchored one. The null result therefore cannot be attributed to local content rather than response format. Only DeepL translations are tested, and only five Nordic and Baltic languages. For required knowledge, 23 of the 500 native reference items are unresolved three-way ties, 13 of them Lithuanian. The fetched text ends partway through the meta-evaluation, so per-pairing results for the other six pairings are not shown here.