Multimodal Speaker Verification as a Threat to Speaker Anonymization

Hugging Face Daily Papers Papers
speaker-verification speaker-anonymization multimodal privacy aggregation audio-text

Summary

This paper investigates how multimodal speaker verification using audio, prosodic, and linguistic cues across multiple utterances can significantly degrade speaker anonymization, reducing equal error rate by over 15% with only five utterances.

Most automatic speaker verification (ASV) systems operate on individual utterances, despite real-world interactions typically consisting of multiple utterances. As speech accumulates, increasingly rich speaker information becomes available through acoustic, prosodic, and linguistic cues, potentially challenging speaker anonymization methods that primarily target vocal characteristics. We investigate ASV in a multi-utterance, multimodal setting and examine whether aggregating information across anonymized speech impacts privacy. We first study audio-only aggregation across multiple anonymized utterances and observe consistent performance improvements as more speech becomes available. We then incorporate prosodic and linguistic information, showing that multimodal systems outperform unimodal approaches. Finally, we compare aggregation strategies and find that frame-level aggregation yields the lowest EERs. Even with only five anonymized utterances, combining audio and text reduces EER by over 15% relative to audio-only aggregation, demonstrating that substantial speaker-discriminative information remains accessible despite anonymization.
Original Article
View Cached Full Text

Cached at: 07/27/26, 09:43 AM

Paper page - Multimodal Speaker Verification as a Threat to Speaker Anonymization

Source: https://huggingface.co/papers/2607.19636

Abstract

Mostautomaticspeakerverification(ASV)systemsoperateonindividualutterances,despitereal-worldinteractionstypicallyconsistingofmultipleutterances.Asspeechaccumulates,increasinglyrichspeakerinformationbecomesavailablethroughacoustic,prosodic,andlinguisticcues,potentiallychallengingspeakeranonymizationmethodsthatprimarilytargetvocalcharacteristics.WeinvestigateASVinamulti-utterance,multimodalsettingandexaminewhetheraggregatinginformationacrossanonymizedspeechimpactsprivacy.Wefirststudyaudio-onlyaggregationacrossmultipleanonymizedutterancesandobserveconsistentperformanceimprovementsasmorespeechbecomesavailable.Wethenincorporateprosodicandlinguisticinformation,showingthatmultimodalsystemsoutperformunimodalapproaches.Finally,wecompareaggregationstrategiesandfindthatframe-levelaggregationyieldsthelowestEERs.Evenwithonlyfiveanonymizedutterances,combiningaudioandtextreducesEERbyover15%relativetoaudio-onlyaggregation,demonstratingthatsubstantialspeaker-discriminativeinformationremainsaccessibledespiteanonymization.

View arXiv pageView PDFGitHubAdd to collection

Get this paper in your agent:

hf papers read 2607\.19636

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.19636 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.19636 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.19636 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Multimodal Speaker Identification in Classroom Environments

arXiv cs.CL

This paper evaluates a multimodal framework for speaker identification in K-12 classrooms by combining acoustic embeddings (ECAPA-TDNN) with LLM-derived semantic context from transcripts, improving accuracy from 39% to 50.3% overall and from 64.9% to 76.9% for longer utterances.

Your Multimodal Speech Model Says I Have a Face for Radio

arXiv cs.CL

This paper presents the first bias evaluation of multimodal speech recognition models, finding significant accuracy differences across gender and ethnicity when pairing faces with audio, with implications for fairness in AI systems.

How Reliable Are Multimodal Signals of Conversational State? Evidence from Remote Dyadic Collaborative Tasks

arXiv cs.CL

This paper evaluates the predictive accuracy, cross-task generalizability, and test-retest reliability of multimodal features for measuring conversational states like cognitive load and power in dyadic remote collaborative tasks. Findings show that linguistic features predict well but generalize poorly, acoustic reliability degrades when controlling for speaker identity, and interaction features provide the most reliable signal.