Open TTS Leaderboard: Scalable Evaluation for Multilingual Text-to-Speech and Voice Cloning

What it claims

The Open TTS Leaderboard ranks open-source TTS and voice-cloning models with automatic metrics instead of pairwise human votes. Hugging Face says this cuts the time to evaluate a model from a couple of weeks of vote collection to a couple of hours. The motivation is coverage. As of Sep 30, 2026, only 16 of the 92 models on Artificial Analysis are open-weights, with a similar skew on Voice Arena, because arena operators must host open models themselves. The authors state that the leaderboard is a proxy and does not replace human preference.

How it works

• Default ranking: macro-average WER over the English splits of Seed TTS Eval and CV3 Eval (zero-shot).
• Other languages: Seed TTS Eval only has English and Chinese audio, so every other language is scored on CV3 Eval alone. Chinese, Japanese and Korean are scored by CER. The multilingual "Average WER" is a macro-average across the selected languages, so it mixes WER and CER.
• Voice cloning toggle: restricts the table to models that support cloning in the selected languages. It adds a SIM column and Pareto plots of SIM against RTFx and model size.
• Streaming tab (TTFA):
• For models with a streaming API, TTFA is the time until the first audio chunk arrives.
• For non-streaming models, TTFA is the time until the whole utterance is generated.
• Every model runs at batch size 1 in its default voice, on the same 50 English CV3-Eval prompts.
• The first 3 runs are discarded as warm-up, and the median of the remaining runs is reported.
• Results are on an H200, with CPU numbers for a small subset of models.

Evidence

The page gives rankings but no numeric scores.

• English WER: hexgrad/Kokoro-82M, Supertone/supertonic-3 and fishaudio/s2-pro lead.
• Multilingual: k2-fsa/OmniVoice, fishaudio/s2-pro and FunAudioLLM/Fun-CosyVoice3-0.5B-2512 are called strong.
• Voice cloning: WER improves for bosonai/higgs-tts-3-4b and openbmb/VoxCPM2 when a reference audio is provided. A model's default-voice WER can therefore understate it.
• Streaming: kyutai/pocket-tts is singled out as good on both GPU and CPU. This is the main lead the summary does not carry for anyone building a voice agent.

How to use it

• Use the Pareto plots to pick a model by trading WER or SIM against batched throughput (RTFx) and size, rather than by WER rank alone.
• In the Listen tab you can filter by language, dataset, voice cloning and models (or a random selection), then hear the actual outputs behind the metrics.
• Voting requires an HF login, and votes may later feed into the leaderboard.
• The evaluation scripts are not yet released. They are promised "soon," in the style of the Open ASR Leaderboard repo, with feedback via GitHub Issues and PRs.

Limits

• What the metrics miss: ASR-based WER measures intelligibility only, and WavLM SIM measures speaker identity only. Neither captures naturalness, expressiveness or listener preference, as the authors state.
• ASR dependence: WER depends on a single ASR model, Qwen3 ASR, so its errors are mixed into every score.
• Thin non-English coverage: languages other than English and Chinese rest on one dataset, CV3 Eval.
• TTFA comparability: TTFA is English-only, batch size 1 and default voice. For non-streaming models it is full-utterance latency, so their TTFA values do not measure the same thing as streaming models' values.
• CPU coverage: CPU results cover only a small set of models.
• Reproducibility: the evaluation code is not yet public, so you cannot reproduce or audit a score.