Speaker Identity in Non-Verbal Vocalizations: Conditional Distillation and Mixture of Experts Approach

Hugging Face Daily Papers Papers
speaker-verification non-verbal-vocalizations conditional-distillation mixture-of-experts domain-adaptation self-supervised-learning

Summary

This paper presents a novel speaker verification framework that combines frozen self-supervised features with ECAPA-TDNN and a Mixture of Experts module, using conditional distillation and contrastive loss to improve identity verification across both speech and non-verbal vocalizations while preventing catastrophic forgetting.

As expressive text-to-speech (TTS) and voice conversion (VC) systems increasingly generate non-verbal vocalizations (NVVs) to enhance naturalness, reliable speaker verification (SV) becomes essential to objectively assess identity consistency across both verbal and non-verbal segments. Yet current SV systems generalize poorly to NVVs, and fine-tuning on NVV data causes catastrophic forgetting of speech performance. We present the first systematic study across 10 NVV types and propose a framework combining frozen Data2Vec self-supervised features with ECAPA-TDNN, enhanced by a Mixture of Experts (MoE) module with learned domain-aware routing. A conditional distillation loss on speech inputs via a pretrained teacher retains speech-to-speech accuracy, while a contrastive loss bridges the speech-NVV domain gap. Our method reduces speech-NVV EER from 38.93% to 22.66% over a pretrained baseline, and improves speech EER from 13.17% to 9.24% via distillation.
Original Article
View Cached Full Text

Cached at: 06/25/26, 05:13 PM

Paper page - Speaker Identity in Non-Verbal Vocalizations: Conditional Distillation and Mixture of Experts Approach

Source: https://huggingface.co/papers/2606.21215

Abstract

A novel speaker verification framework combines frozen self-supervised features with ECAPA-TDNN and MoE modules to improve identity verification across both speech and non-verbal vocalizations while maintaining speech performance.

As expressive text-to-speech (TTS) and voice conversion (VC) systems increasingly generatenon-verbal vocalizations(NVVs) to enhance naturalness, reliablespeaker verification(SV) becomes essential to objectively assess identity consistency across both verbal and non-verbal segments. Yet current SV systems generalize poorly to NVVs, and fine-tuning on NVV data causes catastrophic forgetting of speech performance. We present the first systematic study across 10 NVV types and propose a framework combining frozenData2Vecself-supervised features withECAPA-TDNN, enhanced by aMixture of Experts(MoE) module with learneddomain-aware routing. Aconditional distillation losson speech inputs via a pretrained teacher retainsspeech-to-speech accuracy, while acontrastive lossbridges the speech-NVV domain gap. Our method reduces speech-NVV EER from 38.93% to 22.66% over a pretrained baseline, and improves speech EER from 13.17% to 9.24% via distillation.

View arXiv pageView PDFGitHubAdd to collection

Get this paper in your agent:

hf papers read 2606\.21215

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.21215 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2606.21215 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.21215 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Multimodal Speaker Verification as a Threat to Speaker Anonymization

Hugging Face Daily Papers

This paper investigates how multimodal speaker verification using audio, prosodic, and linguistic cues across multiple utterances can significantly degrade speaker anonymization, reducing equal error rate by over 15% with only five utterances.

Multimodal Speaker Identification in Classroom Environments

arXiv cs.CL

This paper evaluates a multimodal framework for speaker identification in K-12 classrooms by combining acoustic embeddings (ECAPA-TDNN) with LLM-derived semantic context from transcripts, improving accuracy from 39% to 50.3% overall and from 64.9% to 76.9% for longer utterances.

Consensus as Privileged Context for Label-Free Self-Distillation

arXiv cs.LG

A research paper introducing Canon, a label-free self-distillation method that uses consensus among sampled solutions to provide dense token-level supervision for training large language models on reasoning tasks, improving pass@1 by up to 12 points and outperforming label-free reinforcement learning at a fraction of the compute.

EchoDistill:Alignment Noisy-to-Clean Self-Distillation for Robust Audio LLMs

arXiv cs.CL

EchoDistill is an alignment-based noisy-to-clean self-distillation framework that improves the robustness of Audio Large Language Models (ALLMs) against real-world noise by using a frozen clean-audio teacher to guide the student model via group-relative policy optimization (GRPO). Experiments show significant improvements in semantic reliability and task performance under strong noise without additional inference costs.