@SarvamAI: We're open-sourcing two frameworks for evaluating Indian ASR, and a full guide on evaluation across 22 languages. WER (…
Summary
SarvamAI releases open-source evaluation frameworks and a guide tailored for 22 Indian languages, addressing limitations of standard WER/CER metrics.
Similar Articles
Inspect India Evals: An Open Benchmarking Framework for Evaluating Large Language Models in the Indian Linguistic and Cultural Context
Introduces Inspect India Evals, an open-source framework for evaluating LLMs in Indian linguistic and cultural contexts, with six benchmarks testing multilingual ability, bias, safety, and cultural knowledge. Tests on five models show Sarvam-M 24B and Gemma 2 27B lead.
Seeds Before Objectives: Rethinking Evaluation for Low-Resource Garhwali ASR
This paper argues that single-run evaluations in low-resource ASR are unreliable and demonstrates with a new multi-seed Garhwali ASR benchmark that many reported gains vanish under seed-level testing, while standard CTC with w2v-BERT 2.0 remains the most robust approach.
Built an open-source gateway that lets existing ElevenLabs / OpenAI / Deepgram apps run on Sarvam AI by changing one line.
An open-source gateway, sarvam-bridge, lets existing ElevenLabs/OpenAI/Deepgram apps switch to Sarvam AI by changing only the base URL, handling Indic language quirks like chunking, audio reassembly, and language codes. The author details technical decisions, stress testing, and a crash bug fix.
SamaVaani: Auditing and Debiasing Multilingual Clinical ASR for Indian Languages
This paper audits multilingual clinical ASR systems on psychiatric interviews in Indian languages and proposes SamaVaani, a unified debiasing technique to improve performance and fairness across demographic groups.
Voice of India: A Large-Scale Benchmark for Real-World Speech Recognition in India
Researchers introduce Voice of India, a 536-hour closed benchmark of unscripted telephonic conversations across 15 Indian languages and 139 regional clusters, exposing geographic and demographic ASR performance disparities.