Audio-Visual Intelligence in Large Foundation Models
Summary
This survey paper provides a comprehensive review of audio-visual intelligence within large foundation models, establishing a unified taxonomy, synthesizing core methodologies, and outlining key datasets, benchmarks, and open research challenges.
View Cached Full Text
Cached at: 05/08/26, 02:27 PM
Paper page - Audio-Visual Intelligence in Large Foundation Models
Source: https://huggingface.co/papers/2605.04045 Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
Audio-Visual Intelligence represents a multidisciplinary field integrating auditory and visual modalities through large foundation models, encompassing tasks from understanding and generation to interaction, with unified taxonomies and methodological foundations.
Audio-Visual Intelligence(AVI) has emerged as a central frontier in artificial intelligence, bridging auditory and visual modalities to enable machines that can perceive, generate, and interact in the multimodal real world. In the era oflarge foundation models, joint modeling of audio and vision has become increasingly crucial, i.e., not only for understanding but also for controllable generation and reasoning across dynamic, temporally grounded signals. Recent advances, such as Meta MovieGen and Google Veo-3, highlight the growing industrial and academic focus on unified audio-vision architectures that learn from massivemultimodal data. However, despite rapid progress, the literature remains fragmented, spanning diverse tasks, inconsistent taxonomies, and heterogeneous evaluation practices that impede systematic comparison and knowledge integration. This survey provides the first comprehensive review of AVI through the lens oflarge foundation models. We establish a unified taxonomy covering the broad landscape of AVI tasks, ranging from understanding (e.g.,speech recognition,sound localization) to generation (e.g.,audio-driven video synthesis,video-to-audio) and interaction (e.g.,dialogue, embodied, oragentic interfaces). We synthesize methodological foundations, includingmodality tokenization,cross-modal fusion, autoregressive anddiffusion-based generation,large-scale pretraining,instruction alignment, andpreference optimization. Furthermore, we curate representative datasets, benchmarks, and evaluation metrics, offering a structured comparison across task families and identifying open challenges insynchronization,spatial reasoning,controllability, andsafety. By consolidating this rapidly expanding field into a coherent framework, this survey aims to serve as a foundational reference for future research on large-scale AVI.
View arXiv pageView PDFProject pageGitHub15Add to collection
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.04045 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.04045 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.04045 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Audio-Visual Flamingo: Open Audio-Visual Intelligence for Long and Complex Videos
Audio-Visual Flamingo (AV-Flamingo) is a fully open audio-visual large language model designed for understanding and reasoning over long and complex videos, outperforming similarly sized open models and competitive with larger models.
LTX-2: Efficient Joint Audio-Visual Foundation Model
LTX-2 is introduced as an efficient joint audio-visual foundation model. The text includes a mix of the paper reference and a video script about countries facing existential threats, but the primary classification target is the AI model paper.
From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Models
This survey paper systematically reviews the paradigm evolution of unified vision-language perception in multimodal large language models (MLLMs), proposing a five-stage taxonomy and identifying open challenges toward general multimodal intelligence.
When Vision Speaks for Sound
This paper identifies that video-capable multimodal LLMs often appear to understand audio but actually rely on visual cues, a failure mode termed the audio-visual Clever Hans effect. It introduces Thud, an intervention-driven probing framework to diagnose this issue, and proposes an alignment recipe that improves audio-visual consistency by 28 percentage points.
Scalable Visual Pretraining for Language Intelligence
This paper demonstrates that unsupervised visual pretraining on documents, without text extraction, consistently outperforms text-only pretraining for language intelligence, providing an efficient and scalable approach for foundation models.