Anisotropic Modality Align
Summary
This paper proposes AnisoAlign, a framework that addresses the modality gap in multimodal models by applying anisotropic geometric correction to enable effective unpaired modality alignment.
View Cached Full Text
Cached at: 05/11/26, 07:20 AM
Paper page - Anisotropic Modality Align
Source: https://huggingface.co/papers/2605.07825 Published on May 8
·
Submitted byhttps://huggingface.co/Yu2020
Yu_xmon May 11
Authors:
,
,
,
,
,
,
,
,
,
Abstract
Research addresses the modality gap in multimodal models by proposing an anisotropic geometric correction framework that enables effective unpaired modality alignment through structured representation transformation.
Trainingmultimodal large language modelshas long been limited by the scarcity of high-quality paired multimodal data. Recent studies show that theshared representation spaceof pretrainedmultimodal contrastive modelscan serve as a bridge, enabling models to perform multimodal training with unimodal data. However, the key premise of this paradigm remains insufficiently understood: can representations from different modalities be reliably interchanged? The core obstacle lies in the persistentModality Gapin the shared space. In this work, we revisit the geometric nature of themodality gap. We find that modality representations already share compatible dominantsemantic geometry. What truly hinders modality interchangeability is not a simple global shift, but ananisotropic residual structureconcentrated along a small number of dominant directions. Based on this finding, we further propose the principle of anisotropicmodality gapalignment: effectivemodality alignmentshould align with the target-modality distribution while preserving the semantic structure of the source modality. Guided by this principle, we propose an anisotropicgeometric correction framework, AnisoAlign, for unpairedmodality alignment. This framework leverages theinternal geometric priorof the target modality and performs bounded correction on source-modality representations, thereby constructing substitute representations in the target modality. Experiments confirm its benefits in both geometric diagnostics and text-only MLLM training. Overall, this work recasts themodality gapfrom an empirical observation into a correctable, structured geometric phenomenon and provides a new representation alignment perspective for training multimodal models with unimodal data.
View arXiv pageView PDFGitHub62Add to collection
Get this paper in your agent:
hf papers read 2605\.07825
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.07825 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.07825 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.07825 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Beyond Binary Edits Robust Multimodal Knowledge Editing with Adversarial Subspace Alignment
This paper introduces ASAM (Adversarial Subspace Alignment for Multimodal Knowledge Editing), a method that improves the generality of intrinsic multimodal knowledge editing by generating adversarial yet semantically coherent variants and enforcing low-rank alignment of representations, addressing the limited generalization of existing methods.
Diagnosing and Mitigating Perception-Decision Misalignment in Omni-LLMs via Modality Subspace Activation
This paper diagnoses perceptual-decision misalignment in Omni-LLMs and proposes a training-free inference-time framework called Modality Subspace Activation to mitigate it by dynamically balancing modality strengths.
PolyAlign: Conditional Human-Distribution Alignment
PolyAlign is a distribution-aware alignment framework that aligns language models to context-specific human response distributions rather than a single global style, improving naturalness and faithfulness across bilingual settings.
Why Vision Fails as a Universal Bridge: Rectifying Modality Asynchrony in Multilingual MLLMs
The paper identifies the Ghost Anchor phenomenon in multilingual MLLMs, where visual signals are underutilized during early alignment, and proposes the ANCHOR training framework to improve visual semantic emergence and performance across languages.
Context-aware Modality-Topology Co-Alignment for Multimodal Attributed Graphs
Proposes CoMAG, a unified backbone for multimodal attributed graphs that learns task-adaptive reliable contexts and performs modality-preserving alignment, achieving state-of-the-art results on graph-level prediction, modality matching, and graph-conditioned generation.