Answering in the User's Variety: Calibrated Accommodation Pairs for Post-Training

What it claims

Across 10 open model lineages, post-training makes chat replies Standard English for every user. The update itself is not neutral, though. A per-token diagnostic shows the DPO stage favours matching British and Australian spelling over AAE and Singlish on all 10 lineages, pooled at 0.020 nats/token (95% CI [0.014, 0.027]). Adding naive "chosen = dialect rendering" preference pairs helps only at the largest dose tested, and it inserts dialect for Standard-English users. Calibrated accommodation pairs (CAP) penalize three failures in DPO data: under-accommodation, over-accommodation and imposition. CAP raises held-out convergence on OLMo-2-1B without that side effect.

How it works

The test set has 287 first-person requests in six task types. Native speakers rendered each one with content held fixed in six varieties: British (BRE), Australian (AUS), AAE, Colloquial Singapore English (SGE), Nigerian Pidgin (NGP) and Appalachian (APP). Each request also has Standard American English (SAE), informal and formal twins. Prestige tiers were pre-registered from attitude studies, with BRE at the top, AUS in the middle and the other four counted as stigmatized.

Each released stage (base, SFT, DPO, RLVR) answers every twin, with 2 samples per request and 3 seeds. Lineages are OLMo-2 (1B to 32B), OLMo-3, OLMo-3.1-32B, Tülu-2/3 and Zephyr-7B-β. Breadth checks add Qwen2.5/3/3.5, DeepSeek-V2-Lite, DeepSeek-LLM-7B, MiMo-7B, SmolLM3 and 5 deployed chatbots.

• Convergence ratio (CR): Only "novel spans" are scored: tokens outside any word 4-gram shared with the request, and outside quotations. Feature detectors count realizations against opportunities. CR is the reply density gap between twins divided by the request density gap. Spelling, SGE and NGP use a rate per 100 words instead.
• Matching bonus (MB): This is the length-normalized log-ratio between consecutive released checkpoints. It is computed on two content-matched replies, the SFT model's standard reply and its rendering in the variety. MB is positive when the update favours the matched reply more for a variety speaker than for an SAE speaker. Contrasts are adjusted for each reply's baseline likelihood.

Detectors agree with blind expert consensus on 28 of 30 replies (κ = 0.814). They over-fire on raw user requests, so they are applied only to replies.

Evidence

• Standardization by stage: On OLMo-2-7B, British-spelling convergence falls from 0.171 (base) to 0.120 (SFT) and 0.058 (DPO), then sits at 0.060 after RLVR. Informal register drops from 0.413 to 0.009 after DPO. Formal register is the exception: replies keep converging to it (0.141 after DPO).
• Stigmatized varieties: Base models barely mirror stigmatized grammar. OLMo-2-7B reproduces AAE at 0.022 and Singlish lexis at 0.051, both near zero after SFT.
• Prestige ordering: The order is already in the base model (ρ = 0.778), and DPO lowers levels without installing it (ρ = 0.845 after DPO).
• Ability on request: Models can still produce AAE when asked. OLMo-2-7B DPO reaches 0.567 of the authentic AAE feature rate versus 0.009 by default. Tülu-3-8B reaches 0.588 and OLMo-3-7B 0.411.
• Deployed chatbots: They answer Nigerian Pidgin in kind at 0.116 to 0.502, above any post-trained open model (at most 0.046). AAE stays at or below 0.009.
• Per-variety bonus: The AAE matching bonus is negative on 9 of 10 lineages. On OLMo-2-32B the raw contrast reverses (−0.031) but the adjusted one is positive (0.012). In popular families the adjusted contrast is positive in 12 of 14 updates and reversed in none.
• Released preference data: Only 0.81% of prompts in the OLMo-2-7B mix and 0.64% in Tülu-3 trigger the dialect recognisers.
• Dose re-run: DPO was re-run from the OLMo-2-1B SFT checkpoint with 1k (0.8%), 5k and 20k (14%) added AAE or Singlish pairs against matched standard controls. Held-out convergence stayed at the dose-0 level except at the largest dose.

How to use it

Code and data are released under the anonymous name "cap-accommodation". The page gives no licence, URL or run commands. The MB diagnostic needs consecutive public checkpoints, so it applies to lineages that release every stage. The CR metric needs only replies.

Limits

The fetched text ends partway through Section 5.2. It contains no CAP result numbers, no instruction-following scores and no detail on how CAP pairs are built beyond the three penalized failures.

CAP is shown only on OLMo-2-1B. The headline paired test postdates the pre-registration. The pre-registered mixed model gives a positive tier effect on 9 of 10 lineages, with a CI above zero on only 2. Per-lineage fits were made after seeing the 32B data.

Two pre-registered predictions failed: the prestige split growing at the preference stage, and a gradual dose response. The authors note that requested imitation increases stereotyping, so the on-request rates measure ability, not reply quality. They also state that whether a model should reply in a user's variety is for that variety's speakers to decide.