Omni-Modal LLMs Know More Than They Say

Hugging Face Daily Papers Papers
omni-modal multimodal llm representation-alignment knowledge-extraction

Summary

该论文发现全模态大模型（Omni-LLMs）存在'知道却说不出'（knows-but-cannot-say）的现象，即能感知非文本模态信息却难以在输出中表达。作者提出Token级模态对齐（TMA）指标和OmniCatalyst方法，通过自生成文本补充模态输入，在七个Omni-LLM上平均提升7.29%性能并平衡模态偏好。

Omni-modal large language models (Omni-LLMs) extend LLMs to process heterogeneous modalities (e.g., vision, audio, and IMU signals) alongside language, enabling a broader range of multimodal tasks. Yet we find that these models often fail to express information from non-textual modalities even when they perceive it correctly. We term this disconnect knows-but-cannot-say. To investigate this phenomenon, we introduce Token-level Modality Alignment (TMA), which measures the alignment between individual modality representations and response representations. Our analysis shows that non-textual representations are less aligned with response tokens and receive less attention than textual representations, suggesting a potential explanation for uneven modality preferences. Motivated by these findings, we propose OmniCatalyst, a simple perceive rightarrow infer strategy that complements the original modality inputs with self-generated text, making non-textual knowledge more accessible. Across seven Omni-LLMs, OmniCatalyst consistently improves performance (+7.29%) and yields more balanced modality preferences, with further gains from diverse training strategies. Further analysis demonstrates that the knowledge extraction phases increase TMA, suggesting that improved alignment makes non-textual information more accessible during response generation. In summary, our findings uncover how Omni-LLMs can better express modality knowledge through language. Our code is available at https://github.com/naver-ai/OmniLLM.
Original Article
View Cached Full Text

Cached at: 10/14/26, 02:52 PM

Paper page - Omni-Modal LLMs Know More Than They Say

Source: https://huggingface.co/papers/2610.16135

Abstract

Omni-modallargelanguagemodels(Omni-LLMs)extendLLMstoprocessheterogeneousmodalities(e.g.,vision,audio,andIMUsignals)alongsidelanguage,enablingabroaderrangeofmultimodaltasks.Yetwefindthatthesemodelsoftenfailtoexpressinformationfromnon-textualmodalitiesevenwhentheyperceiveitcorrectly.Wetermthisdisconnectknows-but-cannot-say.Toinvestigatethisphenomenon,weintroduceToken-levelModalityAlignment(TMA),whichmeasuresthealignmentbetweenindividualmodalityrepresentationsandresponserepresentations.Ouranalysisshowsthatnon-textualrepresentationsarelessalignedwithresponsetokensandreceivelessattentionthantextualrepresentations,suggestingapotentialexplanationforunevenmodalitypreferences.Motivatedbythesefindings,weproposeOmniCatalyst,asimpleperceiverightarrowinferstrategythatcomplementstheoriginalmodalityinputswithself-generatedtext,makingnon-textualknowledgemoreaccessible.AcrosssevenOmni-LLMs,OmniCatalystconsistentlyimprovesperformance(+7.29%)andyieldsmorebalancedmodalitypreferences,withfurthergainsfromdiversetrainingstrategies.FurtheranalysisdemonstratesthattheknowledgeextractionphasesincreaseTMA,suggestingthatimprovedalignmentmakesnon-textualinformationmoreaccessibleduringresponsegeneration.Insummary,ourfindingsuncoverhowOmni-LLMscanbetterexpressmodalityknowledgethroughlanguage.Ourcodeisavailableathttps://github.com/naver-ai/OmniLLM.

View arXiv pageView PDFAdd to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.16135 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.16135 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.16135 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles