@SarvamAI: We're open-sourcing two frameworks for evaluating Indian ASR, and a full guide on evaluation across 22 languages. WER (…
Summary
SarvamAI releases open-source evaluation frameworks and a guide tailored for 22 Indian languages, addressing limitations of standard WER/CER metrics.
Similar Articles
SamaVaani: Auditing and Debiasing Multilingual Clinical ASR for Indian Languages
This paper audits multilingual clinical ASR systems on psychiatric interviews in Indian languages and proposes SamaVaani, a unified debiasing technique to improve performance and fairness across demographic groups.
Voice of India: A Large-Scale Benchmark for Real-World Speech Recognition in India
Researchers introduce Voice of India, a 536-hour closed benchmark of unscripted telephonic conversations across 15 Indian languages and 139 regional clusters, exposing geographic and demographic ASR performance disparities.
Benchmarking Commercial ASR Systems on Code-Switching Speech: Arabic, Persian, and German
This paper presents a benchmark evaluating five commercial ASR systems on code-switching speech across Arabic-English, Persian-English, and German-English pairs, using a two-stage pipeline to select 300 samples per pair and assessing performance with WER and BERTScore. ElevenLabs Scribe v2 achieves the lowest overall WER (13.2%) and highest BERTScore (0.936), with public dataset available.
SCRIBE: Diagnostic Evaluation and Rich Transcription Models for Indic ASR
SCRIBE is a diagnostic evaluation framework for automatic speech recognition that provides categorical error decomposition for Indic languages, releasing benchmarks and open-weight rich transcription models for Hindi, Malayalam, and Kannada.
@MSFTResearch: We’ve significantly expanded coverage to support: ● 22 additional languages, now reaching communities across 38 African…
Microsoft Research announced expanded coverage for their ASR model, adding 22 languages across 38 African countries, with new datasets and test samples.