Voice of India: A Large-Scale Benchmark for Real-World Speech Recognition in India

# Voice of India: A Large-Scale Benchmark for Real-World Speech Recognition in India
Source: [https://arxiv.org/html/2604.19151](https://arxiv.org/html/2604.19151)
Bhogale Dhir Walecha Kaur Chhabra Pareek Sidh Jain Singh Singh Javed Banga Khapra

ManasAmritanshManmeetVanshikaAadityaHanumanSagarBhaskarUtkarshTahirShobhitMitesh M1Indian Institute of Technology, Madras, India 2Josh Talks, India[cs22d006@cse\.iitm\.ac\.in](https://arxiv.org/html/2604.19151v1/mailto:[email protected])

###### Abstract

Existing Indic ASR benchmarks often use scripted, clean speech and leaderboard driven evaluation that encourages dataset specific overfitting\. In addition, strict single reference WER penalizes natural spelling variation in Indian languages, including non standardized spellings of code\-mixed English origin words\. To address these limitations, we introduceVoice of India, a closed source benchmark built from unscripted telephonic conversations covering 15 major Indian languages across 139 regional clusters\. The dataset contains 306230 utterances, totaling 536 hours of speech from 36691 speakers with transcripts accounting for spelling variations\. We also analyze performance geographically at the district level, revealing disparities\. Finally, we provide detailed analysis across factors such as audio quality, speaking rate, gender, and device type, highlighting where current ASR systems struggle and offering insights for improving real world Indic ASR systems\.

###### keywords:

speech recognition, large\-scale evaluation, low\-resource

## 1Introduction

Recent progress in Indic Automatic Speech Recognition \(ASR\) has been driven by shared tasks and large scale benchmarks such as MUCS\[diwan2021\], IndicSUPERB\[javed2023indicsuperb\], Vistaar\[bhogale2023\], and datasets like IndicVoices\[javed2024indicvoices\], which have expanded coverage across languages, accents, orthographies, and code switching\. However, improvements on benchmark leaderboards often fail to translate to robust real world performance\. Existing benchmarks remain cleaner and more scripted than production audio\[likhomanenko2020rethinking\], and typically report only a single aggregate WER per language, masking large performance differences across regions and dialects\. Their public leaderboard structure further encourages dataset specific optimization, rewarding models that exploit benchmark artifacts rather than generalize to real conversational speech\. This issue is amplified by the reliance on a single reference transcript and strict WER scoring, which penalizes legitimate orthographic variation, including spelling differences and non standardized native script renderings of English origin words in code mixed spontaneous speech\.

![Refer to caption](https://arxiv.org/html/2604.19151v1/map.png)Figure 1:The WER map of India: Average Word Error Rate \(WER\) for ASR models for districts of IndiaTo address these gaps, we introduce Voice of India, a closed source evaluation benchmark built from unscripted, long form telephonic conversations that reflect how speech naturally occurs in everyday Indian interactions\. The benchmark is designed to evaluate ASR systems on spontaneous speech rather than scripted prompts, emphasizing semantic faithfulness over rigid string matching\. To avoid penalizing legitimate orthographic variation, the dataset includes multiple valid transcripts that capture natural spelling differences and alternative renderings commonly found in spontaneous and code mixed speech\. A central goal of the benchmark is to expose geographic disparities in performance\. Accordingly, we analyze results at a regional level: Figure[1](https://arxiv.org/html/2604.19151#S1.F1)visualizes district level WER, averaged across languages within each region and across four models that support all fifteen languages, providing a nationwide view of the error rates experienced by users\.

The dataset is constructed using a population proportional cluster sampling strategy to ensure balanced geographic representation\. Specifically, we consider 139 regional clusters covering nearly the entire country, and sample utterances from each cluster in proportion to its population\. The resulting corpus spans 15 major Indian languages and contains 306,230 utterances totaling 536 hours of speech from 36,691 speakers\. Beyond aggregate metrics, we perform detailed analyses across multiple factors including audio quality, speaking rate, segment length, geographic region, gender, recording device, and age group\. This fine grained evaluation is intended not only to benchmark current systems but also to identify the specific conditions and regions where existing ASR models lack robustness, thereby informing future research and model development\.

## 2Related Work

Benchmarks for Indian Language ASR\.

Early efforts to evaluate ASR for Indian languages include the Interspeech 2018 Low Resource ASR Challenge\[srivastava2018\], multilingual speech corpora released through OpenSLR\[he2020,butryna2020\], and the MUCS 2021 shared task\[diwan2021\]\.More recent benchmarks include IndicSUPERB\[javed2023indicsuperb\], Vistaar\[bhogale2023\], accent focused datasets such as Svarah\[javed2023svarah\]and Lahaja\[lahaja2024\], and large scale demographically diverse datasets such as IndicVoices\[javed2024indicvoices\], covering all 22 scheduled Indian languages\.

Large\-Scale Speech Data Collection\.

Mozilla Common Voice\[ardila2020\]pioneered crowdsourced multilingual speech collection, while Google throught their OpenSLR effort released multiple datasets for under\-resourced languages\[butryna2020,kjartansson2018\]\. FLEURS\[conneau2022\]expanded multilingual coverage with a standardized evaluation dataset, and the WAXAL initiative\[waxal2025\]collected speech resources for several Sub\-Saharan African languages\. Across these efforts, challenges include community mobilization, scalable quality control, and standardized data collection protocols\.

ASR Evaluation Beyond WER\.

Multi reference alignment methods\[arabic2015,arabic2019,japanese,style\_agnostic\]reduce spurious penalties from spelling variation but require expensive multiple transcriptions\. Rule based substitution frameworks such as SCLITE\[sclite\]support explicit variant mappings, yet rely on exhaustive enumeration that is impractical for languages with extensive spelling variation and code mixing\. Alternatives such as WERd\[werd\], normalization based evaluation\[whisper,malayalam\_normalization\], and phoneme based metrics\[power,snwer\]address some limitations but depend on incomplete external resources or normalization schemes\[malayalam\_complex,malayalam\_cer\]\.

Table 1:Overall statistics of the Voice of India Benchmark\.\(a\)Results of open source models and public APIs on the Voice of India BenchmarkModelasbnbhoguhihnekamaimlmrorpatateurElevenLabs Scribe v2\\cellcolor\[HTML\]FFFEFE15\.6\\cellcolor\[HTML\]C7E8D810\\cellcolor\[HTML\]FDF2F123\.5\\cellcolor\[HTML\]FEF6F521\.2\\cellcolor\[HTML\]ADDEC67\.7\\cellcolor\[HTML\]FEF7F720\.3\\cellcolor\[HTML\]FEF9F819\.2\-\\cellcolor\[HTML\]FDF3F223\\cellcolor\[HTML\]E8F5EF12\.9\\cellcolor\[HTML\]FEF7F620\.7\\cellcolor\[HTML\]FFFEFE15\.6\\cellcolor\[HTML\]FEF7F620\.4\\cellcolor\[HTML\]FDF3F223\\cellcolor\[HTML\]FCEFEE25\.5Amazon Transcribe\-\\cellcolor\[HTML\]BDE4D19\.1\\cellcolor\[HTML\]F9DFDD36\\cellcolor\[HTML\]FFFBFB17\.7\\cellcolor\[HTML\]A3DABF6\.8\\cellcolor\[HTML\]FAE4E232\.9\\cellcolor\[HTML\]FEFAF918\.6\-\\cellcolor\[HTML\]FCEBEA28\.2\\cellcolor\[HTML\]D6EEE211\.3\\cellcolor\[HTML\]FFFBFB17\.9\\cellcolor\[HTML\]FFFEFE15\.9\\cellcolor\[HTML\]FEF9F819\.3\\cellcolor\[HTML\]FEF8F819\.7\-AssemblyAI Universal\\cellcolor\[HTML\]E67C73104\.8\\cellcolor\[HTML\]E67C73103\.8\\cellcolor\[HTML\]F6CFCC46\.1\\cellcolor\[HTML\]E67C73101\.8\\cellcolor\[HTML\]FEF9F819\.3\\cellcolor\[HTML\]F7D3D043\.6\\cellcolor\[HTML\]EA8D8689\-\\cellcolor\[HTML\]E67C73107\.5\\cellcolor\[HTML\]EA908887\.6\-\\cellcolor\[HTML\]E67C73101\\cellcolor\[HTML\]F3BEBA57\.4\\cellcolor\[HTML\]E67C73105\\cellcolor\[HTML\]FBE5E431\.9Deepgram Nova 3\-\\cellcolor\[HTML\]FBEAE828\.9\\cellcolor\[HTML\]F6D0CD45\.8\-\\cellcolor\[HTML\]E9F6F013\\cellcolor\[HTML\]F7D5D242\.4\\cellcolor\[HTML\]F4C4C053\.7\-\-\\cellcolor\[HTML\]F7D3D043\.7\-\-\\cellcolor\[HTML\]F0AEA867\.8\\cellcolor\[HTML\]F7D4D143\.1\-Gemini 3 Pro\\cellcolor\[HTML\]FEF7F720\.1\\cellcolor\[HTML\]B6E1CC8\.5\\cellcolor\[HTML\]FEFAFA18\.4\\cellcolor\[HTML\]FFFEFE15\.8\\cellcolor\[HTML\]9AD6B96\\cellcolor\[HTML\]FFFCFC17\.2\\cellcolor\[HTML\]FEF8F719\.9\\cellcolor\[HTML\]FCEFEE25\.6\\cellcolor\[HTML\]FEF5F421\.7\\cellcolor\[HTML\]CFEBDE10\.7\\cellcolor\[HTML\]FEF6F620\.9\\cellcolor\[HTML\]F9FCFB14\.4\\cellcolor\[HTML\]FFFEFE15\.7\\cellcolor\[HTML\]FDF5F421\.9\\cellcolor\[HTML\]BDE4D19\.1Gemini 3 Flash\\cellcolor\[HTML\]FCEDEC26\.9\\cellcolor\[HTML\]E5F4EC12\.6\\cellcolor\[HTML\]FDF4F322\.6\\cellcolor\[HTML\]FDF4F322\.5\\cellcolor\[HTML\]B4E0CB8\.3\\cellcolor\[HTML\]FDF2F123\.9\\cellcolor\[HTML\]FDF4F322\.2\\cellcolor\[HTML\]FBE7E530\.8\\cellcolor\[HTML\]FCEDEB27\.1\\cellcolor\[HTML\]FFFEFE16\\cellcolor\[HTML\]FCEEED26\.1\\cellcolor\[HTML\]FEF9F819\.4\\cellcolor\[HTML\]FEF8F719\.9\\cellcolor\[HTML\]FCEBEA27\.9\\cellcolor\[HTML\]DDF1E711\.9GPT\-4o Transcribe\\cellcolor\[HTML\]E8857C94\.7\\cellcolor\[HTML\]F7D1CE44\.9\\cellcolor\[HTML\]F5CBC749\\cellcolor\[HTML\]E77F7698\.2\\cellcolor\[HTML\]FAE2E033\.9\\cellcolor\[HTML\]F7D1CE45\.2\\cellcolor\[HTML\]EB958D84\.2\\cellcolor\[HTML\]F2B9B560\.4\\cellcolor\[HTML\]E7817897\\cellcolor\[HTML\]F4C1BD55\.6\\cellcolor\[HTML\]EFA7A172\.5\\cellcolor\[HTML\]EFABA570\.1\\cellcolor\[HTML\]F1B4AE64\.2\\cellcolor\[HTML\]F0ACA669\.3\\cellcolor\[HTML\]F9E0DE35\.4GPT\-4o Mini Transcribe\\cellcolor\[HTML\]F9DDDA37\.6\\cellcolor\[HTML\]FEF6F521\.1\\cellcolor\[HTML\]F5CBC749\.1\\cellcolor\[HTML\]E67C73295\.9\\cellcolor\[HTML\]FEF8F819\.6\\cellcolor\[HTML\]F7D2CF44\.6\\cellcolor\[HTML\]E7807897\.5\\cellcolor\[HTML\]F6D0CD45\.6\\cellcolor\[HTML\]E67C73167\.8\\cellcolor\[HTML\]FBE7E630\.7\\cellcolor\[HTML\]F8D6D342\.1\\cellcolor\[HTML\]F9DCDA37\.9\\cellcolor\[HTML\]F5C7C351\.9\\cellcolor\[HTML\]EC999281\.2\\cellcolor\[HTML\]F5C6C252Indic Conformer\\cellcolor\[HTML\]F8FCFA14\.3\\cellcolor\[HTML\]CFEBDE10\.7\\cellcolor\[HTML\]F9E0DE35\.4\\cellcolor\[HTML\]FFFBFA18\\cellcolor\[HTML\]B3E0CA8\.2\\cellcolor\[HTML\]FBE6E431\.6\\cellcolor\[HTML\]FEF5F521\.4\\cellcolor\[HTML\]FDF0EF24\.7\\cellcolor\[HTML\]FCEEED26\\cellcolor\[HTML\]EAF6F013\.1\\cellcolor\[HTML\]F9FCFB14\.4\\cellcolor\[HTML\]FFFFFF14\.9\\cellcolor\[HTML\]FEF8F719\.9\\cellcolor\[HTML\]FDF2F123\.7\\cellcolor\[HTML\]B2DFC98\.1Microsoft Speech\-to\-Text\-\\cellcolor\[HTML\]FCEFEE25\.4\\cellcolor\[HTML\]F9DCD938\.1\-\\cellcolor\[HTML\]D7EFE311\.4\\cellcolor\[HTML\]FAE1DF34\.5\-\-\\cellcolor\[HTML\]F8D7D540\.9\\cellcolor\[HTML\]FBE5E431\.9\-\-\\cellcolor\[HTML\]FCEBEA28\-\\cellcolor\[HTML\]FCF0EF25\.2OmniASR LLM 1B\\cellcolor\[HTML\]FBE9E829\.2\\cellcolor\[HTML\]FBE9E729\.7\\cellcolor\[HTML\]FAE4E232\.8\\cellcolor\[HTML\]F8DBD838\.9\\cellcolor\[HTML\]FFFFFF14\.9\\cellcolor\[HTML\]FCECEB27\.3\\cellcolor\[HTML\]F6D0CD45\.7\\cellcolor\[HTML\]F6CDCA47\.7\\cellcolor\[HTML\]F3BDB858\.4\\cellcolor\[HTML\]FBE6E531\.2\\cellcolor\[HTML\]E98C8489\.8\\cellcolor\[HTML\]F9DEDC36\.5\\cellcolor\[HTML\]F5CBC749\\cellcolor\[HTML\]F3BEBA57\.3\\cellcolor\[HTML\]FFFCFC17\.2OmniASR LLM 7B\\cellcolor\[HTML\]FCEFEE25\.3\\cellcolor\[HTML\]FDF3F322\.8\\cellcolor\[HTML\]FBE6E431\.4\\cellcolor\[HTML\]FAE2E034\.1\\cellcolor\[HTML\]F1F9F513\.7\\cellcolor\[HTML\]FCEEED26\.3\\cellcolor\[HTML\]F8DAD839\.2\\cellcolor\[HTML\]F6CCC948\.2\\cellcolor\[HTML\]F5C6C252\\cellcolor\[HTML\]FCEEED26\.3\\cellcolor\[HTML\]EFA7A172\.6\\cellcolor\[HTML\]FAE3E133\.3\\cellcolor\[HTML\]F7D4D143\.1\\cellcolor\[HTML\]F5C8C550\.7\\cellcolor\[HTML\]FFFEFE16Sarvam Audio\\cellcolor\[HTML\]E6F4ED12\.7\\cellcolor\[HTML\]9BD6B96\.1\\cellcolor\[HTML\]FEF6F620\.9\\cellcolor\[HTML\]E7F5EE12\.8\\cellcolor\[HTML\]8FD1B15\\cellcolor\[HTML\]FFFBFB17\.6\\cellcolor\[HTML\]FFFDFD16\.3\\cellcolor\[HTML\]FDF0EF24\.8\\cellcolor\[HTML\]FEF9F918\.9\\cellcolor\[HTML\]C0E5D39\.4\\cellcolor\[HTML\]F4FAF714\\cellcolor\[HTML\]D5EEE111\.2\\cellcolor\[HTML\]F7FBF914\.2\\cellcolor\[HTML\]FFFAFA18\.2\\cellcolor\[HTML\]A5DAC07Saarika 2\.5\-\\cellcolor\[HTML\]B3E0CA8\.2\\cellcolor\[HTML\]FBE9E729\.6\\cellcolor\[HTML\]F4FAF714\\cellcolor\[HTML\]9CD7BA6\.2\\cellcolor\[HTML\]FCEEED26\\cellcolor\[HTML\]FFFDFD16\.4\-\\cellcolor\[HTML\]FEF9F918\.9\\cellcolor\[HTML\]C7E8D810\\cellcolor\[HTML\]FFFFFF15\.1\\cellcolor\[HTML\]E3F4EC12\.5\\cellcolor\[HTML\]FFFFFF14\.9\\cellcolor\[HTML\]FEF9F918\.9\-
![Refer to caption](https://arxiv.org/html/2604.19151v1/x1.png)\(b\)Averaged Language\-wise WER across models by age, gender, and income\.

## 3The Voice of India Benchmark

### 3\.1Speech Data Collection

Platform and Contributor Onboarding\.Speech data was collected through an online platform enabling large scale remote participation, where contributors across India recorded audio through a peer to peer interface\. Recruitment was conducted through a large nationwide digital community platform111To preserve anonymity, the name of our platform used for recruitment is not disclosed\.with millions of users distributed across the country, enabling outreach to speakers from diverse geographic regions, including rural and semi urban areas that are typically underrepresented in speech datasets\. The final dataset contains contributions from over 36,000 speakers across 15 languages\.

Collecting speech from such a geographically dispersed population introduced significant technical and logistical challenges\. Contributors often used low end smartphones and unstable internet connections, particularly in low bandwidth rural environments, requiring the recording infrastructure to operate reliably under such conditions\. To ensure recording quality, contributors first completed a screening task assessing language familiarity before being granted access to recording tasks\. Approved participants were compensated for their contributions\. All participants provided informed consent, and the data collection protocol was approved by the institute’s internal ethics committee\.

Topic Design and Prompt Generation\.Eliciting spontaneous speech at scale is challenging, as contributors often produce short responses without structured guidance\. To encourage natural, extended speech with diverse vocabulary, we curated a repository of conversational prompts spanning domains such as everyday life, personal experiences, travel, education, and social interactions\. Each topic is presented as an open ended narrative cue followed by progressively revealed follow up questions that guide speakers toward richer descriptions and reflections\.

To generate prompts across 15 languages, we used GPT 4\.5 to produce candidate topics across domains such as finance, healthcare, agriculture, and digital services\. All machine generated prompts were reviewed and refined by language experts to ensure linguistic naturalness and cultural relevance\. The final repository contains over 1,000 topics per language, translated and localized to preserve cross language comparability while retaining region specific characteristics\.

Audio Segmentation and Quality Control\.Raw recordings were segmented into utterances using WebRTC VAD, with adjacent speech regions merged based on short silences and duration limits\. Segments that were too short or excessively long were discarded\. Automated language identification using Meta MMS\[pratap2024scaling\]and SpeechBrain VoxLingua107\[speechbrain\]filtered mislabeled audio, yielding approximately 1,000 hours per language\. Finally, acoustic quality was enforced using DNSMOS\[reddy2021dnsmos\], removing segments with low perceptual quality scores\.

Demographically Stratified Sampling\.To ensure proportional representation, we used cluster\-based sampling where districts were grouped into geographically and dialectally coherent clusters\. Data volume per cluster was aligned with population proportions from the 2011 Census of India\[chandramouli2011census\]\. Within each demographic quota, segments were prioritized using inverse\-frequency word weights i\.e\., rare words \(1\-100 occurrences\) received the highest weight \(50\), while mid\-frequency \(101\-300\) and moderately frequent words \(301\-1000\) were assigned weights of 20 and 5 respectively, and common words \(\>\>1000\) receiving a weight of 0\.5\. Segment scores were computed as the mean weight of their constituent words, allowing the selection process to favor segments with richer and more diverse vocabulary while maintaining demographic balance\.

### 3\.2Transcription Process

High fidelity reference transcripts were produced using a machine\-assisted multi\-annotator pipeline\. Initial transcripts were generated using internally fine\-tunedWhispermodels for 11 languages and theIndic Conformer\[mahadhwani\]model for Assamese, Odia, Urdu, and Maithili\. These were initially verified against the audio by a native speaker and subsequently subjected to six rounds of cross\-validation by independent annotators\. Any segments flagged as inaccurate were re\-transcribed and underwent further accuracy verification, ultimately yielding highly accurate orthographic transcripts\.

#### 3\.2\.1Lattice Construction

Following\[bhogale2026oiwer\], Lexical and phonetic variations were generated and validated through a dedicated pipeline operating in parallel with human transcription\.

Variation generation\.We prompted Gemini 3 Flash to exhaustively enumerate valid word substitutions given both the ground\-truth transcript and all model hypotheses\. Segmentation variants \(e\.g\.,loginvs\.log in\) and named\-entity forms were captured by instructing the model to treat multi\-word phrases as atomic units\.

Pruning\.To remove invalid variations obtained from the first stage, we re\-prompted Gemini 3 Flash to re\-evaluate the generated lattice, by prompting the model to strictly align with the original ground\-truth semantics\.

Consensus alignment\.For contiguous error spans, cases where at least four of the eight top\-performing models agreed on a hypothesis absent from the lattice were flagged\. Flagged spans with semantic similarity below 0\.5 \(computed via fine\-tuned BERT\)\[deode2023l3cube\]were submitted for human review; acoustically ambiguous segments confirmed by annotators were incorporated into the lattice unconditionally\.

Disfluency handling\.Verbatim human transcripts retained half\-words and disfluencies\. To avoid penalizing intelligibility\-oriented models, such elements were made optional by merging adjacent lattice nodes\. Conversely, low\-amplitude sounds consistently transcribed by at least four models but absent from human references were added as optional nodes\.

## 4Experiment Setup

### 4\.1Models Evaluated

We evaluate 14 ASR systems, including 11 proprietary APIs and 3 open source models\. A model is evaluated for a language only if it provides explicit support through a native language tag or can be reliably conditioned through prompts indicating the target language\. For dialects such as Bhojpuri and Chhattisgarhi, dialect specific tags are used when available; otherwise prompt based conditioning is used, with Hindi as the fallback following the 2011 Census of India classification\.

The evaluated systems include proprietary APIs from Sarvam \(Sarvam Audio,Saarika 2\.5\), Google \(Gemini 3 Pro,Gemini 3 Flash\), OpenAI \(GPT\-4o Transcribe,GPT\-4o Mini Transcribe\),Amazon Transcribe,Deepgram Nova 3,AssemblyAI Universal,ElevenLabs Scribe v2, andMicrosoft Speech\-to\-Text\. We also include three open source models:Indic Conformer\(AI4Bharat\) and Meta'sOmniASR LLM 1BandOmniASR LLM 7B\. All models are evaluated using their default inference configurations\.

### 4\.2Evaluation Metric

We evaluate models using Orthographically\-Informed Word Error Rate \(OIWER\)\[bhogale2026oiwer\]metric\. Unlike standard WER, this accounts for permissible spelling variation between hypothesis and reference which reduces spurious errors caused by orthographic variation and better reflects recognition quality in languages with flexible spelling conventions\.

## 5Results and Discussion

### 5\.1Evaluation of models on the Voice of India Benchmark

Table[2\(a\)](https://arxiv.org/html/2604.19151#S2.F2.sf1)shows that most models exceed a WER of 20 \(highlighted in red\), a threshold often associated with practical usability, and no system meets this criterion consistently across all languages\. Even the best performing model,Sarvam Audio, exceeds this threshold on Bhojpuri \(20\.920\.9\) and Maithili \(24\.824\.8\)\.Sarvam Audioachieves the lowest WER in 13 of 15 languages, followed bySaarika 2\.5andGemini 3 Pro\.Indic ConformerandElevenLabs Scribe v2show moderate performance, while several models perform substantially worse; in some casesAssemblyAI Universalexceeds WER 100, indicating transcription failure\. Notable anomalies also appear:Gemini 3 Properforms best on dialectal varieties such as Bhojpuri and Chhattisgarhi, whileGPT\-4o Mini Transcribeshows severe degradation on Gujarati \(295\.9295\.9\) and Malayalam \(167\.8167\.8\) despite moderate performance on Hindi\. These results highlight the difficulty of robust ASR across Indian languages and dialects, reflecting their limited representation in large scale multilingual training and the challenges introduced by script diversity and orthographic flexibility\.

### 5\.2Does the performance vary across regions of India?

District level WER shows strong geographic variation, ranging from about 4% \(Nainital\) to 44% \(Mannarakkat\)\. Higher error rates appear in parts of South India, particularly Kerala and interior Karnataka, and in North Bihar, reflecting the presence of underrepresented languages such as Maithili and Bhojpuri\. In contrast, districts across the Hindi belt \(Uttar Pradesh, Delhi, Haryana, Rajasthan, and Madhya Pradesh\) cluster below 10% WER, indicating stronger alignment with standard Hindi speech\. Metropolitan districts also tend to show lower error rates\. Overall, the pattern reveals a clear geographic bias, with linguistically diverse or underrepresented regions exhibiting substantially higher WER than the Hindi belt and major urban centers\.

### 5\.3Are existing Indic ASR benchmarks reliable?

![Refer to caption](https://arxiv.org/html/2604.19151v1/x2.png)Figure 3:WER of six models across FLEURS \(public benchmark\), VoI Lattice, and Normal Transcription \(single\-reference\), with circled rank badges \(1 = lowest WER\)\.Public benchmarks such as FLEURS\[conneau2022\], though widely used for reporting performance, are vulnerable to overfitting due to their static and publicly accessible evaluation sets\. As shown in Figure[3](https://arxiv.org/html/2604.19151#S5.F3), models that achieve strong WER on FLEURS often perform substantially worse on our benchmark, particularly for morphologically richer languages\. Moreover, single reference WER is sensitive to transcription style, penalizing orthographic differences despite correct acoustic modeling\. The lattice based evaluation mitigates this issue by allowing multiple valid spelling variants, producing more stable system rankings\.

### 5\.4Does WER vary across different audio attributes?

Figure[4](https://arxiv.org/html/2604.19151#S5.F4)shows that deviations from ideal acoustic conditions consistently increase error rates across DNSMOS\[reddy2021dnsmos\]quality quartiles, speaking\-rate quartiles, and utterance duration bins \(<<2s, 2–5s,\>\>5s\)\. Audio degradation raises WER monotonically; ElevenLabs Scribe rises from15\.31%15\.31\\%to25\.20%25\.20\\%and Gemini\-3\-Pro from13\.42%13\.42\\%to23\.44%23\.44\\%between the highest and lowest quality quartiles\. Speaking rate exhibits a U\-shaped pattern, with Indic Conformer WER peaking at27\.57%27\.57\\%\(slow\) and27\.53%27\.53\\%\(very fast\) versus24\.75%24\.75\\%at moderate speeds\. Short utterances are most affected due to limited semantic context; Amazon STT degrades from10\.45%10\.45\\%\(\>\>5s\) to18\.74%18\.74\\%\(<<2s\), with Microsoft STT showing a similar trend \(10\.90%10\.90\\%to18\.59%18\.59\\%\)\.

### 5\.5Are models fair across demographics?

A fair model achieves comparable average WER across demographic groups \(Figure[2\(b\)](https://arxiv.org/html/2604.19151#S2.F2.sf2)\)\. While none of the models exhibit strict parity, the observed differences are small, indicating broadly consistent performance\. Models perform slightly better on female speech \(3\.1 to 4\.3 percent gap\), while younger speakers \(18 to 22\) show higher error rates than older speakers \(46 plus\)\. Income based differences are also minor, with slightly higher WER for higher income speakers, possibly due to increased linguistic complexity such as code mixing\. Notably,Gemini 3 ProandSarvam Audioshow less than 2 percent variance across income groups\.

### 5\.6Recommendations for Model Developers

Based on our analysis, we group evaluated systems into three tiers according to the nature of their failures, and provide targeted recommendations for each category\.

Tier I: Top\-performing systems \(Ranks 1–6, WER≤\\leq20%\)\.Tier I models \(Sarvam\-Audio, Gemini\-3\-Pro, IndicConformer\) largely solve general multilingual transcription but reveal three critical fault lines: \(1\) low\-resource languages like Bhojpuri and Maithili suffer WERs 4\-5×\\timeshigher than Hindi; \(2\) a systematic 19–21% male\-speaker penalty persists across all architectures; and \(3\) models fail catastrophically for out\-of\-region migrants \(e\.g\., Chattisgarhi speakers in Tamil Nadu face WERs of 55%\-65%\)\. Resolving these gaps demands targeted regional data collection, gender\-stratified training, and explicit cross\-regional evaluation metrics\.

Tier II: Competent but fragile systems \(Ranks 7–8, WER 21%–30%\)\.These systems perform well on canonical speech but degrade sharply under distributional shifts\. For example, Gemini 3 Flash exhibits a severe short\-utterance penalty compared to longer segments, as well as significant performance degradation on low\-quality audio, necessitating dedicated short\-audio pathways and multi\-condition training\. Concurrently, Microsoft STT struggles with underrepresented languages, demonstrating noticeably higher error rates on languages such as Bhojpuri and Malayalam\. Improving overall robustness requires targeted SNR augmentation and expanded data coverage, particularly for the Eastern Hindi belt and Dravidian language families\.

Tier III: Inadequate systems \(Ranks 9–16, WER≥\\geq35%\)\.These models face severe coverage limitations, especially for Dravidian languages\. Deepgram Nova\-3 and OmniASR yield elevated error rates in Tamil \(WER: 67\.8%\) and Odia \(WER: 89\.8%\), while GPT\-4o\-mini and AssemblyAI Universal produce extreme errors in Gujarati \(WER: 297%\) and Malayalam/Telugu \(WER≥\\geq100%\)\. Driven by failed language detection and generative hallucinations rather than standard transcription mistakes\. Consequently, substantial retraining and targeted language adaptations are essential prior to broad deployment\.

![Refer to caption](https://arxiv.org/html/2604.19151v1/x3.png)Figure 4:Performance of transcription models \(cross\-lingual average WER\) across speech characteristics, including MOS quality quartiles, speech rate, and audio duration\.

## 6Conclusion

We introduce Voice of India, a benchmark for evaluating ASR systems on real world Indian speech collected from unscripted telephonic conversations across multiple languages and regions\. The benchmark incorporates multiple transcription variants and evaluates systems using orthographically informed WER to better reflect natural spelling variation in spontaneous speech\. Evaluation across state of the art systems reveals substantial robustness gaps, with large performance differences across languages and regions and particularly high error rates in linguistically diverse areas\. We further show that public benchmarks can overestimate real world performance and that single reference WER exaggerates errors caused by orthographic variation\.

## References