Recommended benchmark.

I expect voice to become one of the main ways people interact with robots and physical AI.

That makes speech recognition in real conversations much more important than it seems today.

Real conversations are hard to transcribe. People pause, talk over each other, and switch languages mid-sentence.

Robots also need to understand the languages people actually speak. More than 5.5 billion people across the Global South are non-English speakers.

The community needs a good way to measure these frontier capabilities.

@humynlabs built BRIDGE ASR 2.0 to measure exactly this.

It tests 23 speech recognition models on real two-person conversations, each 10 to 15 minutes long, in 18 Indic languages plus Spanish, Portuguese, and Vietnamese.

The metric I find most useful is code-switch F1.

It checks whether English words mixed into an Indic sentence stay in English. A model that writes “data backup” in Devanagari script scores zero.

You can also filter the leaderboard by language and by metric. The methodology and evaluation data are public, and the team wants researchers to test the benchmark and find where it breaks.

Check out the benchmark here: https://humynlabs.ai/bridge/ASR/2.0

BRIDGE Report — Humyn Labs

Source: https://humynlabs.ai/bridge/ASR/2.0

Why this Benchmark exists?

From speaker recruitment to evaluation pipeline — the decisions that make BRIDGE reproducible, auditable, and resistant to benchmark gaming.

Methodology

How the Benchmark was built ?

A dataset built for how people actually talk, and a metric stack built to show exactly where a model breaks, not just that it did. The scaffolding is simple to state: genuine, spontaneous dual-speaker conversations, tagged across seven cohort dimensions, normalised through the same three-step cleanup before anything is scored, then run identically across all 23 models.

6 Metric Evaluation Stack

WER — Word Error RateCER — Character Error RateSemSim — Semantic SimilarityCS F1 — Code-Switch F1PIER — Phoneme-Informed ERWIL — Word Information Lost

(A harder, more realistic dataset)

Voice AI’s requirements have shifted fast from clean, single-speaker dictation toward noisy, overlapping, real conversation and the dataset reflects where that need is now. Every recording is a genuine two-person conversation with real overlapping speech and cross-talk, the exact failure mode that breaks streaming ASR and speaker separation in production, captured across a wide range of everyday acoustic environments rather than a single clean studio.

(Normalised and tagged before a single score runs)

The scaffolding behind every score is the same regardless of model or language: genuine, spontaneous dual-speaker conversations, tagged across seven cohort dimensions, normalised through an identical three-step text cleanup before anything is scored, then run against every provider on the exact same set. That consistency is what makes a result attributable to a specific condition, a language, a noise level, a speaker overlap rather than lost inside a single average rank.

(Core accuracy, and where the errors actually are)

WER and CER anchor the stack, alongside MER and WIL — two bounded variants that stay stable even when a model hallucinates heavily, where plain WER can exceed 100% and break model-to-model comparison. Every error is also classified as a substitution, deletion, or insertion: high substitution means the model is misrecognising speech, high deletion means it’s dropping speech outright, usually a segmentation issue — and high insertion means it’s inventing words that were never spoken. A single WER collapses all three into one number and hides which is actually happening.

(Script fairness, meaning, and code-switching)

Standard WER counts a correctly recognised English word written in local script as a full error against a Latin-script reference — lwWER and script_penalty separate that script mismatch from a real transcription error. SemanticSim and LevenshteinSim ask a different question again: did the meaning survive even where the exact wording didn’t, and how close is the literal string. And because Indic conversation routinely drops English words into a native-language sentence, PIER, CS Recall, CS Precision, and CS F1 measure specifically whether that code-switching was handled — or missed, or fabricated — correctly.

One model leads by a clear margin.

ElevenLabs Scribe v2 leads with 10.99% lwWER, compared with 16.32% for Soniox stt-async-v4 — a 5.3pp gap and about a third fewer word errors. It ranks first in 14 of the 18 languages, one more than it wins on WER. Brand name does not guarantee performance. The bottom six Indic models range from 70.77% to 88.03% lwWER.

Multi-Dimensional Analysis

Cohort Performance Analysis

Choose a cohort dimension, a metric (WER, CER, SemSim, CS F1, lwWER, PIER, or WIL), and a model to see how performance shifts across conditions. All three filters work together — any combination is valid.

The usual assumption is that fast conversations are harder. Our data says the bigger problem is long silence.

After controlling for language and model, calls with gaps longer than 20 seconds lose an average of 5.05pp of lwWER, compared with just 1.33pp for rapid turn-taking. The effect shows up across almost every viable model: chirp_3 loses 13.22pp to long silences versus 6.69pp to speed; Gemini 2.5 Pro loses 10.87pp versus 2.16pp; even Scribe v2 loses 7.50pp versus 0.79pp. Controlling for language matters. Long pauses are concentrated in Marwari, which is already the hardest language in the set. Without that control, the apparent silence penalty rises to 16.33pp — overstating the effect of silence itself. The likely culprit is segmentation, not speech recognition.

Multi-Dimensional Analysis

The Hidden Quality Gap

CS F1 measures whether English vocabulary embedded in Indic speech is preserved — not dropped, not transliterated. A model that turns “data backup” into “डेटा बैकअप” scores 0 on CS F1. Invisible to WER. Fatal for downstream applications.

Strong code-switching — CS F1 ≥ 0.7

These models preserve English vocabulary in Indic speech. Both are viable for enterprise applications where English terminology appears in native-language conversation.

15

model

s

in this bucket

· ElevenLabs Scribe v2, Gemini 3 Pro (Preview), Gemini 3 Flash (Preview)

These models handle some code-switching. Performance varies by language and English density — verify on your specific use case before shipping.

7

model

s

in this bucket

· Speechmatics Melia, Gemini 2.5 Flash, Gladia v2

Zero code-switching — CS F1 < 0.2

These models systematically transliterate or drop English tokens. Not suitable for code-mixed enterprise Indic applications.

1

model

in this bucket

· AssemblyAI Universal

“Indic models drop English” is true for some models, but misleading for others.

Two nearly independent measures separate the failures: Script penalty: the English survives, but is written in the local script. saarika v2.5, saaras v3, AWS, chirp_3, and Soniox show 6–8pp penalties while maintaining ~90–95% code-switch precision. Code-switch recall: the English disappears entirely. Gemini 2.5 Flash and melia-1 show low recall (~51–52%) despite much smaller script penalties. The first problem is recoverable with transliteration-aware matching. The second is not: there is no token to recover. CS F1 collapses these into one number, making a fixable script error look identical to a fundamental recognition failure.

Access dataset & benchmark

Dataset access & citation

ASR benchmarks weren’t built for the languages you’re working on. BRIDGE was. Get access to the new 6 metric benchmark to evaluate your model.

If you use this benchmark in your research, please cite the following.