The Open ASR Leaderboard Adds Its First Global South Language

What it claims

Monsoon is four speaker-disjoint test splits: Indian English and Hindi, each with a public and a private half. It is now on the Open ASR Leaderboard. Its purpose is to make error rates break down by who is speaking, not just by what was said.

The sets are small in hours and large in speakers. On the public en-IN split (roughly five hours), eight leaderboard models fall within 0.18 WER of each other (4.81–4.99). Split by region, their spreads differ sharply:

• openai/whisper-large-v3-turbo varies by 0.46 points across the five zones.
• mistralai/Voxtral-Mini-3B-2507 is 0.14 behind Whisper on the corpus but varies by 1.68 points, from 4.38 in the Central zone to 6.06 in the East.

The hardest zone also depends on the model:

The authors read this as a property of the models, not of the audio.

How it works

Source audio. Pairs held unscripted two-person conversations over a peer-to-peer interface, recorded dual-channel on their own handsets and connections. Contributors were recruited via the Voice Arena community, passed a proficiency screen, were paid, and consented to use in training and distribution. LLM-generated topics were localised by linguists and covered travel, healthcare, agriculture, education and digital services. Each conversation was seeded with a narrative cue and follow-up questions that were revealed progressively.

Duration cap. A per-speaker cap, calibrated for each language, keeps prolific contributors from dominating.

Gating before transcription:

flowchart LR
  A[Raw recording] --> B{Language ID match?}
  B -- no --> X[Rejected]
  B -- yes --> C{Gender classifier corroborates self-report?}
  C --> D{Live, not played-back?}
  D -- no --> X
  D -- yes --> E{SNR intelligible?}
  E -- no --> X
  E -- yes --> F[Per-channel VAD segmentation]
  F --> G[DNSMOS P.808 check]
  G --> H[5-level linguist transcription]

• Language check: a language-ID model covering 30+ languages.
• Gender check: a classifier corroborates the self-reported gender but does not replace it.
• Noise: background noise is kept deliberately. Only recordings unintelligible by SNR are dropped.
• Segmentation: splits happen at 2 s of silence, with a 15 s soft cap, done separately per channel. Every segment therefore has a single speaker.

Transcription.

• First drafts come from internal ASR models that do not appear on any public leaderboard, so no evaluated system influenced its own reference.
• Five levels of correction follow. Each correction round is verified by a different annotator.
• Numerals are written as words.
• Segments still flagged at the end go back for re-transcription.

Hindi lattice.

• For each span of a transcript, the reference lists every accepted spelling.
• Candidate spellings come from several ASR outputs plus LLM expansion. Native linguists then prune them.
• OIWER (from AI4Bharat) aligns the hypothesis against these accepted sets.

Evidence

• Speaker spread:
• 4,888 speakers in total.
• More than half of speakers contribute exactly one segment.
• The top ten speakers account for 2.8–6.8% of duration.
• Geography:
• Public en-IN covers 428 districts across 30 states and union territories.
• The Hindi splits cover 202 and 295 districts. Uttar Pradesh accounts for about 40% of Hindi speakers.
• In en-IN, no state exceeds 13%. Zone shares are South 35%, East 18%, Central 18%, North 16%, West 11%.
• Devices: 315–582 distinct device models per split, with no single model above 2.1% of segments.
• Metadata: 18 columns per segment, 12 of them metadata. Fields include age, gender, district, occupation, education, marital status, income band and handset. These are the fields needed to rerun the regional analysis by age, handset and so on.
• OIWER vs. single reference: Scoring against a flattened reference (the first variant per span) raised WER for every system, by uneven amounts. Two pairs of systems swapped rank. The page gives this only as a figure, with no numbers in the text.

How to use it

• Public splits: you can score models on these yourself.
• Private splits: open a PR on the Open ASR Leaderboard GitHub. Report your public-set results in the checklist; the HF team verifies them, computes the private metrics, and you confirm.
• Where en-IN shows up: it appears as "Voice Arena Monsoon" in the default column set, so it now counts toward every model's headline Average WER. Its private split feeds the Private (conversational) column, alongside Appen and DataoceanAI data.
• Where Hindi shows up: in the Multilingual tab. A model is ranked there only if it supports every selected language. You can also pick "Hindi" from the "Language dataset breakdown" dropdown.
• Scoring code: voi-oiwer is open-sourced for reproducing Hindi scores. The page gives no install command.

Limits

• The regional result is presented as an illustration, not a finding. At about five hours, a 0.18 corpus-level gap is within noise.
• The page does not give the regional result's per-zone sample sizes.
• The page states no licence for the data.
• The authors say the sets do not fix demographic or orthographic bias. They only make it visible.