Who Said What, and Will It Be Remembered? Evaluating Persistent Speaker Attribution Across Meetings

Hugging Face Daily Papers Papers
speaker-attribution diarization speaker-verification speech-recognition benchmark pyannote

Summary

This paper introduces SI-cpWER, a corpus-level metric for persistent speaker attribution across meetings, and evaluates 5 commercial diarized-STT APIs plus 2 open baselines (and their own ThyVoice system) on NOTSOFAR and CHiME-6, showing that per-meeting ranking doesn't reflect cross-meeting identity consistency. A short version was accepted at IEEE SLT 2026 Demo Track.

Speech transcripts used as long-term memory must preserve both words and stable speaker identities. Existing meeting-transcription metrics either ignore speakers or remap anonymous speakers independently in each recording, so they cannot measure whether the same person retains one identity across meetings. We evaluate persistent speaker attribution with Speaker Identified cpWER (SI-cpWER), which scores a corpus under one global speaker-ID assignment. The benchmark covers five commercial diarize-then-identify cascades, two open academic baselines, and ThyVoice on the full 129-meeting CHiME-8 NOTSOFAR evaluation set in clean and noiseaugmented form, plus CHiME-6. ThyVoice is our end-to-end reference system; it repairs overlap and gates the evidence used to create and update voiceprints. Requiring persistent identity changes the commercial ranking: ThyVoice records lower SI-cpWER than every evaluated commercial cascade in all three conditions and the lowest mean in the full panel, 47.13 versus 54.75 for the next system. Complementary lexical, diarization, per-recording attribution, and speaker-clustering diagnostics characterize upstream error surfaces in the final attributed record. These results show why persistent attribution must be evaluated directly in systems that reuse conversations across time.
Original Article
View Cached Full Text

Cached at: 10/01/26, 08:24 PM

Paper page - Who Said What, and Will It Be Remembered? Evaluating Persistent Speaker Attribution Across Meetings

Source: https://huggingface.co/papers/2609.39344 Meeting transcripts are usually scored one recording at a time, so a system can look strong while assigning the same person different identities across meetings. We measure this with SI-cpWER: cpWER under one speaker mapping across the whole corpus.

We evaluate 5 commercial diarized-STT APIs and 2 open baselines, each paired with the same pyannoteAI voiceprint backend, plus our reference system ThyVoice, on NOTSOFAR (clean and noise-augmented) and CHiME-6. The commercial ranking changes: the best per-meeting cpWER does not give the best cross-meeting attribution.

The report also breaks errors down by layer (WER, DER/JER, speaker clustering) and tests how overlap handling and enrollment audio affect identity.

A short version is accepted at IEEE SLT 2026 (Demo Track).

Similar Articles

OSynth: A Longitudinal, Context-Sensitive Evaluation of Meeting Understanding

arXiv cs.CL

Otter.ai researchers present OSynth, a large-scale synthetic meeting dataset of 570 hours with latent ground-truth state, showing that state-of-the-art MLLMs gain significant quality on entity-sensitive dimensions when given broader workspace context. Human raters distinguished OSynth meetings from real transcripts with only 56% accuracy.

SPARC: Spoken Profile Authorship Recognition Corpus

Hugging Face Daily Papers

SPARC introduces a benchmark of over 12,500 speakers and 1.5M utterances to study how much speaker identity and demographic attributes can be verified from transcribed speech alone, evaluating attribution models and LLM adversaries. The work finds LLMs can distinguish same-speaker pairs and verify sex/accent labels above chance, while short 50-word excerpts reduce linkage risk.

ContextEcho: A Benchmark for Persona Drift in Long Agentic-Coding Sessions

arXiv cs.CL

This paper introduces ContextEcho, a benchmark and reusable harness for measuring persona drift in large language models during long, tool-using coding sessions, revealing that even frontier models like Anthropic Sonnet 4.5 exhibit significant drift that can persist across session compaction.