Language Unalignability: Why Some Concepts Resist Cross-Cultural Benchmark Evaluation

What it claims

Standard multilingual benchmarks (FLORES-200, MEGA/MEGAVERSE, XTREME-UP, Belebele) score against one or a few reference translations using chrF, BLEU, COMET or multiple-choice accuracy. The paper argues these metrics can only test whether a translation hits the dominant meaning of a concept. They cannot test whether the concept's internal distinctions of register, pragmatics or historical period survive.

The paper's example is Mandarin xīn kǔ le ("you've worked hard"). A model that renders it as "good job" in all four of its uses (superior-to-subordinate, peer, service worker, ironic) gets identical scores. The authors claim that any benchmark with a single reference point per source item "cannot, even in principle, detect" this loss.

How it works

• Usage cloud. A concept is the set of contextual embeddings an encoder produces for the token across a corpus of attested sentences in a given time window. The authors treat it as a point set, not a manifold. They use intrinsic dimensionality as a proxy for pragmatic complexity.
• Two faithfulness conditions for a map τ between the source and target clouds:
• M1 (lexical): cloud centroids correspond within ε. Dictionary-style benchmarks tacitly enforce only this.
• M2 (structural): local k-neighbourhood ranks are preserved within δ. The authors chose ranks over distances deliberately, so that the global anisotropy of contextual embedding spaces does not register as structural change.
• α-unalignability: no map satisfies both M1 within ε and M2 within δ.
• Three routes to unalignability:
• A, pragmatic-load asymmetry: honorifics, evidentials and case-marked information structure that the target language has no dimension for.
• B, diachronic non-stationarity: Mandarin qíng, "circumstance" in Pre-Qin to Han texts versus "emotion" today.
• C, categorial non-correspondence: the target cloud is a lower-dimensional projection of the source.
• Existing methods in these terms: Procrustes mapping optimises M1, optimal transport interpolates between M1 and M2, and Distributed Alignment Search is read as searching for a τ that satisfies M2 causally.
• Proposed clean test: compare the same word in the same language and encoder across two time windows. This removes language-distance, resource and tokenisation confounds, and is suggested as the best test of the Cycle-Consistency and Manifold-Curvature Mismatch metrics.

Evidence

The behavioural evidence reinterprets the authors' earlier work (Lian et al., 2025) rather than presenting new experiments.

• FLORES-200, Gemini-2.5-Flash: 204 languages, ten sentences per language, single run.
• Generation failures are about 10× more frequent English→target than target→English.
• In one-way ANOVAs on chrF, language family (ηp² = 0.41, E→T) and resource class (ηp² = 0.41, E→T) are both significant at p < .001.
• Script is not significant (ηp² = 0.17, p = .27, E→T).
• LOBSTER: 96 International Linguistics Olympiad problems covering more than 90 languages. Gemini-2.5-Pro mean scores by family:

Limits

• The FLORES analysis is explicitly preliminary: ten sentences per language, one run, one model. Script is acknowledged to be confounded with resource availability, which weakens the "script doesn't matter" argument.
• LOBSTER puzzles are meant to be self-contained, but models may have prior exposure to the languages.
• The diachronic test is proposed, not run in the text shown.
• The fetched text cuts off in §4. The Yami mechanistic study (§5) and the definitions of the four diagnostic metrics (§6) were not available, so this document does not cover how to compute them.
• No code, data or thresholds for ε and δ appear in the fetched text.
• This is a position-style preprint (v1, 6 Oct 2026).