A Broader Look into MLLM's Retrieval Heads

Hugging Face Daily Papers Papers
multimodal-llm retrieval-heads attention-heads audio-visual document-retrieval ranking-objective

Summary

该论文研究多模态大模型（MLLM）中的稀疏检索头在新增音频-视觉模态下的分布与共享机制，并通过排序目标优化这些注意力头，将 Qwen3-VL-8B 在 MMDocIR 上的页面 Recall@1 提升 6.03 个点。

Locating relevant evidence in long inputs is a core capability of Multimodal Large Language Models. Prior studies attribute this capability to retrieval heads, a sparse set of attention heads that transfer information from the context to the output. However, these analyses are confined to a limited set of modalities and the corresponding models. Moreover, how retrieval heads can be leveraged to enhance the model calls for further exploration. In this paper, we ask whether this mechanism expands together with the input modalities of the model. Taking audio-visual input as a newly added modality, we construct paired inputs in which the same answer is either spoken or shown on screen. With the evidence content fixed, retrieval heads remain sparse in the new modality and are shared to a large extent across modalities. Across 18 language, vision-language, and omni-modal checkpoints, omni-modal models largely preserve the retrieval heads of their backbones. Ablating these heads eliminates nearly all audio-visual retrieval. The heads shared across modalities make up fewer than 2\% of all heads, yet ablating them also degrades document understanding, spatial perception, and visual mathematics. Building on these findings, we turn retrieval heads into a target of optimization by training their attention with a ranking objective. This raises the page Recall@1 of Qwen3-VL-8B on MMDocIR by 6.03 points and improves document retrieval across three model families. Functional attention heads can thus serve both as a lens on multimodal models and as a target for improving them.
Original Article
View Cached Full Text

Cached at: 10/30/26, 07:09 AM

Paper page - A Broader Look into MLLM’s Retrieval Heads

Source: https://huggingface.co/papers/2610.32411

Abstract

Retrieval heads remain sparse and shared across audio-visual modalities, and optimizing them improves document retrieval performance in multimodal models.

Locating relevant evidence in long inputs is a core capability ofMultimodal Large Language Models. Prior studies attribute this capability toretrieval heads, a sparse set ofattention headsthat transfer information from the context to the output. However, these analyses are confined to a limited set of modalities and the corresponding models. Moreover, howretrieval headscan be leveraged to enhance the model calls for further exploration. In this paper, we ask whether this mechanism expands together with the input modalities of the model. Takingaudio-visual inputas a newly added modality, we construct paired inputs in which the same answer is either spoken or shown on screen. With the evidence content fixed,retrieval headsremain sparse in the new modality and are shared to a large extent across modalities. Across 18 language, vision-language, and omni-modal checkpoints,omni-modal modelslargely preserve theretrieval headsof their backbones. Ablating these heads eliminates nearly all audio-visual retrieval. The heads shared across modalities make up fewer than 2\% of all heads, yet ablating them also degrades document understanding, spatial perception, and visual mathematics. Building on these findings, we turnretrieval headsinto a target of optimization by training their attention with aranking objective. This raises the page Recall@1 of Qwen3-VL-8B on MMDocIR by 6.03 points and improvesdocument retrievalacross three model families. Functionalattention headscan thus serve both as a lens on multimodal models and as a target for improving them.

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2610\.32411

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.32411 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.32411 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.32411 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

OneSearch-VL: Unified Multimodal Deep Research Agent for Image and Video

Hugging Face Daily Papers

该论文提出 OneSearch-VL，一个以视觉接地证据图（VGEG）为核心的统一多模态深度研究智能体，用于单图、多图和视频研究任务，并构建了 SFT/RL 数据集及细粒度评测基准，在多个图文与视频基准上显著超越 Qwen3-VL-8B。

Query-based Cross-Modal Projector Bolstering Mamba Multimodal LLM

arXiv cs.CL

This paper proposes a query-based cross-modal projector that compresses visual tokens via cross-attention to improve Mamba-based multimodal LLMs, boosting both performance and throughput on vision-language benchmarks while eliminating the need for manual 2D scan order design.