Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes

Hugging Face Daily Papers Papers

Summary

Falcon is a causal audio-visual target speaker extraction model that uses lip video to isolate a target speaker's voice in noisy, overlapping speech, cutting parameters by 91.9% and latency by 11.1× versus causal AV-TFGridNet. The paper also introduces RealSSA, a benchmark spanning simulated and real-recorded acoustic scenes with varied interference conditions.

Streaming audio-visual target speaker extraction (AVTSE) is essential for latency-sensitive applications. However, current causal systems face two coupled gaps. First, standard benchmarks fix the number of speakers at two and ignore the near-/far-field interference, reverberation, and sudden noise of real scenes. Second, high-performance causal architectures are too heavy for edge deployment. Lightweight alternatives try to reclaim capacity by repeatedly invoking a small separator, which offsets the savings. To address the first gap, we release RealSSA, a realistic AVTSE benchmark consisting of RealSSA-Sim and RealSSA-Real. RealSSA-Sim provides controllable simulated scenes with 1--5 dynamic near-field interfering speakers; scenes with far-field interference additionally include 10--15 far-field interfering speakers. RealSSA-Real provides a held-out real-recorded evaluation set. To address the second gap, we propose Falcon, a causal AVTSE method. Its Separator performs multi-scale time-frequency separation in a single encoder--decoder pass. We train Falcon with an encoder-space speaker contrastive loss. This loss suppresses near-field leakage at zero inference cost and transfers across backbones. Falcon achieves state-of-the-art extraction quality on RealSSA-Sim, LRS2-2Mix, LRS3-2Mix, and VoxCeleb2-2Mix. Compared with the causal AV-TFGridNet baseline, it cuts parameters by 91.9%, computation by 20.2 times, and GPU inference latency by 11.1 times. Code and demo are available at https://cslikai.cn/Falcon/.
Original Article
View Cached Full Text

Cached at: 10/29/26, 05:31 AM

Paper page - Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes

Source: https://huggingface.co/papers/2610.32749 Falcon is a causal audio-visual target speaker extraction model that uses the target speaker’s mouth video to extract their voice from noisy, overlapping speech. It combines a single-pass multi-scale time-frequency separator with an encoder-space speaker contrastive loss that reduces interfering-speaker leakage without adding inference cost.

The paper also introduces RealSSA, a benchmark covering varied speaker counts, near- and far-field interference, reverberation, and sudden noise, with both simulated scenes and a held-out real-recorded evaluation set. Compared with causal AV-TFGridNet, Falcon reduces trainable parameters by 91.9%, computation by 20.2×, and GPU inference latency by 11.1× in the reported evaluation. Code, model weights, datasets, and interactive demos are available on the project page.

Similar Articles

AV-FABLE: Audio-Visual Manipulation Forensics in Long-Form Videos

Hugging Face Daily Papers

AV-FABLE introduces a long-form audio-visual benchmark for sparse manipulation detection, showing that frozen AV-LLM perception features contain forensic signals best recovered by separate temporal decoders (NFSR) rather than language outputs.

TaoMate-FVSR:Audio-Guided Facial Video Restoration

Hugging Face Daily Papers

TaoMate-FVSR proposes an audio-guided facial video restoration framework that combines reference-guided video flow modeling with audio synchronization to recover lip, dental, and facial details in AI-generated videos, achieving state-of-the-art perceptual similarity and synchronization on talking-head benchmarks.

FD-VAD: Semantic Endpoint Detection for Streaming Full-Duplex Speech

arXiv cs.CL

FD-VAD proposes an ASR-free, streaming semantic endpointer that maps causal audio windows directly to Continue/Stop decisions for full-duplex voice agents, combining a frozen speech encoder with a parameter-efficiently adapted language model and achieving state-of-the-art EOT recall on TurnBench.

Introducing the FFASR Leaderboard: Benchmarking ASR in the Real World

Hugging Face Blog

Introduces the FFASR Leaderboard, an open, community-driven benchmark for evaluating automatic speech recognition models under realistic far-field acoustic conditions, highlighting the significant performance gap between near-field and far-field scenarios.