When Vision Speaks for Sound
Summary
This paper identifies that video-capable multimodal LLMs often appear to understand audio but actually rely on visual cues, a failure mode termed the audio-visual Clever Hans effect. It introduces Thud, an intervention-driven probing framework to diagnose this issue, and proposes an alignment recipe that improves audio-visual consistency by 28 percentage points.
View Cached Full Text
Cached at: 05/20/26, 06:36 AM
Paper page - When Vision Speaks for Sound
Source: https://huggingface.co/papers/2605.16403
Abstract
Video-capable multimodal large language models exhibit apparent audio understanding driven by visual cues rather than actual audio processing, necessitating intervention-based frameworks for diagnosing and improving audio-visual alignment.
Despite rapid progress invideo-capable MLLMs, we find that their apparent audio understanding in videos is often vision-driven: models rely on visual cues to infer or hallucinate acoustic information, rather than verifying the audio stream. This issue appears across both state-of-the-art open-source omni models and leading closed-source models from providers such as Google and OpenAI. We characterize this failure mode as anaudio-visual Clever Hans effect, in which models appear (falsely) audio-grounded, but actually exploit visual-acoustic correlations without verifying whether the audio and visual streams are truly aligned. To systematically study this behavior, we introduce Thud, anintervention-driven probing frameworkbased on threecounterfactual audio edits: Shift, which teststemporal synchronization; Mute, which testssound existence; and Swap, which testsaudio-visual consistency. Beyond diagnosis, we further study a two-stagealignment recipe: intervention-derivedpreference pairsteach audio verification, whileevent-level general video preferencesregularize the model against over-specialization. Our best 10K-sample recipe improves average performance across the three intervention dimensions by 28 percentage points, while slightly improving performance on general video and audio-visual QA benchmarks.
View arXiv pageView PDFProject pageGitHub1Add to collection
Get this paper in your agent:
hf papers read 2605\.16403
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.16403 in a model README.md to link it from this page.
Datasets citing this paper4
#### Rakancorle1/hans-10k Viewer• Updatedabout 15 hours ago • 20.8k • 15 #### Rakancorle1/hans-sft-4k Viewer• Updated1 day ago • 3.83k • 7 #### Rakancorle1/vggsync-3k Viewer• Updated1 day ago • 3k #### Rakancorle1/thud-eval Viewer• Updated1 day ago • 710
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.16403 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
Why Vision Fails as a Universal Bridge: Rectifying Modality Asynchrony in Multilingual MLLMs
The paper identifies the Ghost Anchor phenomenon in multilingual MLLMs, where visual signals are underutilized during early alignment, and proposes the ANCHOR training framework to improve visual semantic emergence and performance across languages.
Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models
This paper investigates why multimodal LLMs fail on vision-centric tasks when visual evidence conflicts with language priors, introducing the WhatIfVis benchmark and showing that models often encode visual evidence but cannot reliably control their reliance on it.
Do Multimodal LLMs See Before They Read? Diagnosing Contextual Sycophancy
The paper introduces multimodal contextual sycophancy in large language models, where external text overrides visual evidence, and proposes a diagnostic method using System-2 Visual Arbitration to improve performance.
From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMs
This paper studies how audio and visual information flow inside Audio-Visual Large Language Models (AVLLMs), revealing that AVLLMs follow sequential or parallel routing depending on input configuration, and that some tokens can be discarded after information transfer for efficiency.
Seeing Is Not Sharing: Some Vision-Language Models Overestimate Common Ground in Asymmetric Dialogue
This paper investigates a bias in vision-language models where they overestimate shared understanding in dialogue, confusing perceptual access with communicative grounding. The findings have implications for dialogue systems and VLM evaluation.