@rohanpaul_ai: VoiceArena launched Diarization Bench to compare 12 diarization systems on the same real-world audio and scoring rules.…
Summary
VoiceArena launched Diarization Bench, a benchmark comparing 12 diarization systems on real-world audio, highlighting how accuracy drops with overlap and the impact of collar effect on error rates.
View Cached Full Text
Cached at: 09/24/26, 02:16 AM
VoiceArena launched Diarization Bench to compare 12 diarization systems on the same real-world audio and scoring rules. It exposes how quickly accuracy drops with overlap and more speakers.
STT, TTS, conversational agents, and now diarization, all on @voicearena_ai
The number that got me is the collar effect: the collar is simply the timing tolerance around a speaker change.
NVIDIA Nemotron 3 scores 4.29% DER with a 250 ms collar. Score the exact same output at 0 ms and it becomes 14.72%.
Shobhit Banga (@shobhitbanga): Introducing Diarization Bench, @voicearena_ai’s benchmark for who spoke when.
The metric we measure is diarization error rate, or DER. It measures how much of a conversation a model attaches to the wrong person. Every second it misses, invents, or gives to another speaker counts
Similar Articles
Indic DiarBench: A Multilingual Joint Diarization and ASR Benchmark for Indian Languages
Indic DiarBench is a multilingual joint diarization and ASR benchmark covering all 22 scheduled languages of India with 108 hours of human-corrected multi-speaker audio, capturing conversational nuances like code-mixing and speaker overlap.
@rohanpaul_ai: I really like how @voicearena_ai designed this evaluation. Instead of asking whether one voice model is simply "better"…
The tweet highlights VoiceArena's evaluation method for voice models, which separates Task Completion and Naturalness via blind pairwise voting, and announces Jarvis Bench v0.5 as a conversational agent benchmark.
Voice Acting Arena
Launch of Voice Acting Arena, a platform for evaluating voice acting AI models through user comparisons to assess performance in acting, direction, and genuineness.
Voice of India: A Large-Scale Benchmark for Real-World Speech Recognition in India
Researchers introduce Voice of India, a 536-hour closed benchmark of unscripted telephonic conversations across 15 Indian languages and 139 regional clusters, exposing geographic and demographic ASR performance disparities.
@rohanpaul_ai: Voice AI is now good enough that preference and human-likeness can be scored as separate targets. @voicearena_ai launch…
Voice Arena launched Turing TTS, a benchmark that measures how often listeners mistake AI speech for human speech via a 'Fool Rate'; even the best model, Gemini 3.8 Flash TTS, fools listeners only about 10% of the time.