Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes
Summary
Falcon is a causal audio-visual target speaker extraction model that uses lip video to isolate a target speaker's voice in noisy, overlapping speech, cutting parameters by 91.9% and latency by 11.1× versus causal AV-TFGridNet. The paper also introduces RealSSA, a benchmark spanning simulated and real-recorded acoustic scenes with varied interference conditions.
View Cached Full Text
Cached at: 10/29/26, 05:31 AM
Paper page - Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes
Source: https://huggingface.co/papers/2610.32749 Falcon is a causal audio-visual target speaker extraction model that uses the target speaker’s mouth video to extract their voice from noisy, overlapping speech. It combines a single-pass multi-scale time-frequency separator with an encoder-space speaker contrastive loss that reduces interfering-speaker leakage without adding inference cost.
The paper also introduces RealSSA, a benchmark covering varied speaker counts, near- and far-field interference, reverberation, and sudden noise, with both simulated scenes and a held-out real-recorded evaluation set. Compared with causal AV-TFGridNet, Falcon reduces trainable parameters by 91.9%, computation by 20.2×, and GPU inference latency by 11.1× in the reported evaluation. Code, model weights, datasets, and interactive demos are available on the project page.
Similar Articles
AV-FABLE: Audio-Visual Manipulation Forensics in Long-Form Videos
AV-FABLE introduces a long-form audio-visual benchmark for sparse manipulation detection, showing that frozen AV-LLM perception features contain forensic signals best recovered by separate temporal decoders (NFSR) rather than language outputs.
TaoMate-FVSR:Audio-Guided Facial Video Restoration
TaoMate-FVSR proposes an audio-guided facial video restoration framework that combines reference-guided video flow modeling with audio synchronization to recover lip, dental, and facial details in AI-generated videos, achieving state-of-the-art perceptual similarity and synchronization on talking-head benchmarks.
FD-VAD: Semantic Endpoint Detection for Streaming Full-Duplex Speech
FD-VAD proposes an ASR-free, streaming semantic endpointer that maps causal audio windows directly to Continue/Stop decisions for full-duplex voice agents, combining a frozen speech encoder with a parameter-efficiently adapted language model and achieving state-of-the-art EOT recall on TurnBench.
From 2.1M Hours of Raw Broadcast Audio to Finnish Speech SSL: Confidence Filtering and Architectural Analysis at Scale
Aalto University researchers present a large-scale Finnish speech SSL study using 2.1M hours of raw broadcast audio, showing that ASR confidence-based filtering outperforms LID-based filtering by 3-7% relative WER, along with controlled architectural comparisons of Transformer, Macaron-Net, Conformer, and E-Branchformer at matched parameter budgets.
Introducing the FFASR Leaderboard: Benchmarking ASR in the Real World
Introduces the FFASR Leaderboard, an open, community-driven benchmark for evaluating automatic speech recognition models under realistic far-field acoustic conditions, highlighting the significant performance gap between near-field and far-field scenarios.