Benchmarking Arabic Dialects for Math Problems

What it claims

DAR-MGSM extends MGSM-Rev2 (250 GSM8K-derived math word problems) to Modern Standard Arabic and five dialects: Egyptian, Emirati, Jordanian, Moroccan and Saudi. The translated version has 250 problems per variety, 1,500 in total. A parallel localized version replaces names, locations, currencies and numerals with region-specific equivalents. The paper's central claim is that cultural localization exposes model fragility that direct translation hides. Flagship models such as GPT-5.4 and Gemma 4 show only slight degradation on the localized version.

How it works

Professional translators produced the translations, with one translator and one reviewer per variety. Native speakers then reviewed the output in-house. Correction rates ranged from 4.0% for Saudi to 49.6% for MSA. Most corrections came from misread English phrasings that encode mathematical relationships. Named entities were deliberately left unlocalized in this version, and translators were told to keep dialectal text natural, with minimal overlap with MSA.

Localization is semi-automated:

1. Entities in the MSA text are replaced with indexed, typed placeholders such as [person_m_1] and [person_m_2].
2. The placeholders are projected onto each dialect translation with a heuristic.
3. Each placeholder is mapped to a regional value, for example Lloyd/Toulouse/dollar becomes Metwally/Cairo/Egyptian pound for Egyptian. Currencies are pound, dinar, riyal and dirham.
4. Digits and percent signs are converted to Arabic-Indic numerals for every variety except Moroccan, which keeps Western Arabic numerals.
5. A final human validation pass checks the result.

51 problems contained no localizable entities.

Evidence

The models evaluated are Qwen-3.6, Gemma 4, GPT-5.4, Atlas-Chat, NileChat and Jais, eleven in total. Inference ran through OpenRouter or on a local GPU. The prompt was:

Solve the provided math problem. End with 'The answer is: [number]'.

Accuracy is computed on the extracted number after removing thousand separators. Moroccan uses periods as thousand separators; the other varieties use commas. The authors manually audited outputs to check whether answer-extraction failures inflated the error counts, with the breakdown in Appendix H. Degradation varies by model size and dialect. Localizing entity types one at a time changes accuracy only slightly compared with localizing all of them together.

Limits

The fetched text stops before the results table, so no per-model or per-dialect accuracy numbers are available here, and neither are the interpretability results. The page gives no licence or download location for the dataset. It also gives no model sizes for Qwen-3.6, Gemma 4, Atlas-Chat, NileChat or Jais.

Validation found 47 wrong name replacements across all dialects, under 1% of replacements. These came from the replacement script assuming each name appears only once in a problem. The benchmark covers grade-school arithmetic only.

Anyone scoring with their own harness should handle Arabic-Indic digits and the Moroccan separator convention, or extraction errors will look like reasoning errors. A separate MSA translation of the full GSM8K (AGSM8K, Alenezi and Salman) was published in May 2026, in parallel with this work.