Beyond Text-Dominance: Understanding Modality Preference of Omni-modal Large Language Models

Hugging Face Daily Papers Papers
multimodal-llm modality-preference cross-modal-hallucination vision-language interpretability benchmarking omni-modal

Summary

This paper investigates modality preference in omni-modal large language models (OLLMs), revealing a paradigm shift from text-dominance to visual preference. The authors introduce a conflict-based benchmark and layer-wise probing to diagnose cross-modal hallucinations using internal model signals.

Native Omni-modal Large Language Models (OLLMs) have shifted from pipeline architectures to unified representation spaces. However, this native integration gives rise to a critical yet underexplored phenomenon: modality preference. To bridge this gap, we first systematically quantify modality preference of OLLMs using a newly-curated conflict-based benchmark and the modality selection rate metric. Our evaluation of ten representative OLLMs reveals a notable paradigm shift: unlike the ``text-dominance'' of traditional VLMs, most OLLMs exhibit a pronounced visual preference. To further understand the underlying mechanism, we conduct layer-wise probing and demonstrate that such modality preference is not static but emerges progressively in the mid-to-late layers. Building upon these insights, we leverage these internal signals to diagnose cross-modal hallucinations, achieving competitive performance across three downstream multi-modal benchmarks without task-specific data. Our work provides both a mechanistic understanding and a practical tool for building more trustworthy OLLMs. Our code and related resources are publicly available at: https://github.com/icip-cas/OmniPreference
Original Article
View Cached Full Text

Cached at: 04/21/26, 07:20 AM

Paper page - Beyond Text-Dominance: Understanding Modality Preference of Omni-modal Large Language Models

Source: https://huggingface.co/papers/2604.16902

Abstract

Research reveals that native omni-modal large language models exhibit visual preference over text, with modality preference emerging progressively in mid-to-late layers and enabling diagnosis of cross-modal hallucinations.

NativeOmni-modal Large Language Models(OLLMs) have shifted from pipeline architectures to unified representation spaces. However, this native integration gives rise to a critical yet underexplored phenomenon:modality preference. To bridge this gap, we first systematically quantifymodality preferenceof OLLMs using a newly-curatedconflict-based benchmarkand themodality selection ratemetric. Our evaluation of ten representative OLLMs reveals a notable paradigm shift: unlike the ``text-dominance’’ of traditional VLMs, most OLLMs exhibit a pronounced visual preference. To further understand the underlying mechanism, we conductlayer-wise probingand demonstrate that suchmodality preferenceis not static but emerges progressively in the mid-to-late layers. Building upon these insights, we leverage these internal signals to diagnosecross-modal hallucinations, achieving competitive performance across three downstream multi-modal benchmarks without task-specific data. Our work provides both a mechanistic understanding and a practical tool for building more trustworthy OLLMs. Our code and related resources are publicly available at: https://github.com/icip-cas/OmniPreference

View arXiv pageView PDFGitHub3Add to collection

Get this paper in your agent:

hf papers read 2604\.16902

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2604.16902 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2604.16902 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2604.16902 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Can Multimodal Large Language Models Understand OCT?

Hugging Face Daily Papers

This paper introduces OCT-Bench, a comprehensive benchmark for evaluating multimodal large language models on OCT image understanding, comprising 10,076 questions across 20 fine-grained tasks in perception, cognition, and reasoning. Evaluation of 20 models shows current MLLMs fall short of reliable OCT understanding.