Why Vision Fails as a Universal Bridge: Rectifying Modality Asynchrony in Multilingual MLLMs

# Why Vision Fails as a Universal Bridge: Rectifying Modality Asynchrony in Multilingual MLLMs
Source: [https://arxiv.org/html/2608.15085](https://arxiv.org/html/2608.15085)
Yihang DuAffiliation:The Chinese University of Hong Kong, ShenzhenAffiliation:Shenzhen Loop Area InstituteZhengzhao LaiAffiliation:The Chinese University of Hong Kong, ShenzhenSiyu LiAffiliation:The Chinese University of Hong Kong, ShenzhenYan HuAffiliation:The Chinese University of Hong Kong, ShenzhenAffiliation:National Health Data Institute \(Shenzhen\)yihangdu@slai\.edu\.cn,juhaoliang1@link\.cuhk\.edu\.cn,huyan@cuhk\.edu\.cnAffiliation:Equal contribution, Corresponding author

###### Abstract

Multimodal large language models \(MLLMs\) exhibit substantial performance degradation in non\-English visual reasoning, despite the strong multilingual competence of their text\-only backbones\. While mechanistic evidence from text\-only models suggests that non\-English inputs are routed through an English\-centric latent space, the multimodal implications of this phenomenon remain unexplored\. Through layer\-wise mechanistic analysis, we identify theGhost Anchorphenomenon: a temporal modality asynchrony where linguistic representations largely converge toward the English semantic manifold in early layers, while visual semanticization remains immature\. Consequently, visual signals are physically present yet exert limited influence during the early alignment window\. To address this, we proposeANCHOR, a training framework employing Proactive Visual Anchoring \(PVA\) to accelerate early visual semantic emergence and encourage visual representations to guide linguistic translation\. Mechanistic interventions indicate that ANCHOR increases the influence of visual signals during early translation\. Furthermore, experiments on xMMMU, MaXM, and CVQA show that ANCHOR improves aggregate performance over standard baselines across both fine\-tuned and zero\-shot languages\.

## 1Introduction

Multimodal Large Language Models \(MLLMs\) have significantly advanced vision\-language understanding by integrating LLMs with visual perception[25](https://arxiv.org/html/2608.15085#bib.bib20);[2](https://arxiv.org/html/2608.15085#bib.bib14)\. However, while MLLMs excel in English, achieving equitable understanding across diverse languages remains a fundamental challenge[30](https://arxiv.org/html/2608.15085#bib.bib3);[17](https://arxiv.org/html/2608.15085#bib.bib9)\. Empirical evaluations reveal a substantial performance degradation in non\-English contexts, indicating that current multimodal alignment remains highly English\-dependent and fails to ground visual concepts within a truly language\-agnostic space[34](https://arxiv.org/html/2608.15085#bib.bib23);[5](https://arxiv.org/html/2608.15085#bib.bib18)\.

A prevailing intuition attributes this disparity to data scarcity, motivating efforts to curate massive multilingual multimodal datasets[7](https://arxiv.org/html/2608.15085#bib.bib2);[30](https://arxiv.org/html/2608.15085#bib.bib3)\. Yet, this data\-centric approach incurs prohibitive costs and overlooks the internal representational mechanisms\. Instead of forcing alignment externally, an emerging consensus acknowledges that MLLMs inherently develop a shared semantic space where diverse languages converge[31](https://arxiv.org/html/2608.15085#bib.bib30)\. Ideally, heterogeneous inputs would converge toward a language\-neutral semantic space through synchronized cross\-modal interactions\. In contemporary MLLMs, however, this shared space often empirically manifests as English\-dominated because of language\-centric architectures and imbalanced pre\-training data[29](https://arxiv.org/html/2608.15085#bib.bib4)\. Within this empirically observed geometry, visual signals—grounded in physical reality—should act as active anchors that ground representations across linguistic boundaries[12](https://arxiv.org/html/2608.15085#bib.bib1);[15](https://arxiv.org/html/2608.15085#bib.bib5);[35](https://arxiv.org/html/2608.15085#bib.bib6)\.

![Refer to caption](https://arxiv.org/html/2608.15085v1/fig1.png)Figure 1:The Ghost Anchor phenomenon\.\(a\) Ideal synchronized cross\-modal alignment\. \(b\) The reality of modality asynchrony: rapid linguistic translation bypasses delayed visual semantic emergence in early layers\.However, the empirical reality is far from this synchronized ideal\. Despite the theoretical promise of vision as a universal anchor, visual signals provide limited assistance for cross\-lingual alignment: non\-English languages continue to exhibit systematic deficits even when visual grounding should theoretically facilitate comprehension[4](https://arxiv.org/html/2608.15085#bib.bib28);[33](https://arxiv.org/html/2608.15085#bib.bib29)\. This disconnect raises a critical question:Why does the visual modality fail to serve as the anticipated bridge?

We hypothesize that the answer lies in the temporal dynamics of the Transformer architecture\. Through a rigorous layer\-wise mechanistic analysis, we uncover a profoundModality Asynchrony: linguistic representations exhibit substantial convergence toward the English semantic manifold in early layers, while visual grounding remains semantically immature until much deeper in the network\. As illustrated in[Fig\.1](https://arxiv.org/html/2608.15085#S1.F1), this early linguistic convergence emerges well before visual semantics become available\. Consequently, the linguistic stream is forced to map to the English space “blindly”, bypassing visual guidance\.

Modality Asynchrony creates what we term theGhost Anchorphenomenon: the visual modality is physically present in the computation yet remains semantically invisible during the critical early alignment phase \([Fig\.1](https://arxiv.org/html/2608.15085#S1.F1)\)\. Consequently, the model defaults to a visual\-agnostic translation path, which fundamentally degrades its non\-English visual reasoning performance\.

To rectify this asynchrony, we proposeANCHOR, a training framework designed to synchronize dual\-stream convergence\. By accelerating visual semantic emergence through Proactive Visual Anchoring \(PVA\), ANCHOR encourages visual signals to become grounded early and influence the linguistic translation process\. Experiments on benchmarks such as xMMMU[30](https://arxiv.org/html/2608.15085#bib.bib3), MaXM[5](https://arxiv.org/html/2608.15085#bib.bib18), and CVQA[18](https://arxiv.org/html/2608.15085#bib.bib17)show that our approach improves cross\-lingual visual performance\.

In this work, we make the following principal contributions:

- •We systematically investigate the internal dynamics of multilingual MLLMs and formalize theGhost Anchorphenomenon, whereModality Asynchronyrenders visual signals semantically inaccessible during early alignment\.
- •We provide mechanistic evidence through layer\-wise analysis and controlled interventions, showing that visual information exerts negligible causal influence on the linguistic translation trajectory during the early alignment window\.
- •Building on these insights, we proposeANCHOR, a training framework that rectifies asynchrony by accelerating visual semantic emergence through Proactive Visual Anchoring \(PVA\), which explicitly supervises early\-layer visual representations using an external visual foundation model\.

## 2Related Work

##### Mechanistic Perspectives on Cross\-lingual Alignment

Cross\-lingual alignment has been primarily studied in text\-only multilingual models, where semantically equivalent expressions gradually converge in hidden space and support transfer[31](https://arxiv.org/html/2608.15085#bib.bib30);[34](https://arxiv.org/html/2608.15085#bib.bib23)\. Using logit\-lens and causal\-tracing style probes, mechanistic analyses further show that many multilingual LLMs process non\-English inputs through an English\-centric internal representation space, where non\-English tokens are implicitly translated into an English semantic pivot before being mapped to the target output[28](https://arxiv.org/html/2608.15085#bib.bib32);[19](https://arxiv.org/html/2608.15085#bib.bib11);[31](https://arxiv.org/html/2608.15085#bib.bib30)\. At the same time, strong global alignment can hurt language\-specific competence, especially for low\-resource or distant languages[8](https://arxiv.org/html/2608.15085#bib.bib31)\. This tension is still analyzed almost entirely without vision, leaving unclear how visual signals reshape \(or fail to reshape\) cross\-lingual alignment inside MLLMs[29](https://arxiv.org/html/2608.15085#bib.bib4);[24](https://arxiv.org/html/2608.15085#bib.bib8)\. Our work fills this gap by extending mechanistic analysis to multimodal settings and showing that visual signals can fail exactly when modality asynchrony delays visual semantic emergence during the critical alignment window\.

##### Multilingual Large Vision Language Models

Early multilingual vision\-language models such as M3P[15](https://arxiv.org/html/2608.15085#bib.bib5), UC2[35](https://arxiv.org/html/2608.15085#bib.bib6), and related BERT\-era frameworks[11](https://arxiv.org/html/2608.15085#bib.bib7)explicitly optimized cross\-lingual and cross\-modal objectives during pretraining\. In the LLM era, mainstream multilingual MLLMs rely more on scale\-first pipelines, where large translated or synthetic instruction mixtures are expected to induce multilingual visual competence implicitly[30](https://arxiv.org/html/2608.15085#bib.bib3);[7](https://arxiv.org/html/2608.15085#bib.bib2)\. This data\-centric strategy improves benchmarks but incurs high annotation cost and offers limited control over how multilingual text and visual semantics are aligned internally\. More targeted methods add alignment modules or objectives, including mBLIP[9](https://arxiv.org/html/2608.15085#bib.bib33), LRM\-LLaVA[12](https://arxiv.org/html/2608.15085#bib.bib1), and M2\-VLP[1](https://arxiv.org/html/2608.15085#bib.bib34)\. However, their supervision is still mainly applied at output or global\-distribution levels rather than explicitly constraining layer\-wise cross\-modal interaction during the critical alignment phase\. In contrast, our approach starts from a mechanistic diagnosis of modality asynchrony and directly targets the critical layer\-wise interaction window\.

## 3Unveiling Modality Asynchrony: A Mechanistic Analysis

Do visual signals truly function as language\-agnostic anchors in multilingual MLLMs?To answer this, we systematically investigate both the independent evolutionary trajectories of multilingual text and visual signals, and how they interact within the model’s internal representations\.

##### The English Semantic Manifold

To perform joint reasoning, MLLMs inherently project heterogeneous inputs \(e\.g\., multilingual text and visual signals\) into a shared representational space[23](https://arxiv.org/html/2608.15085#bib.bib24)\. During inference, representations from both diverse languages[31](https://arxiv.org/html/2608.15085#bib.bib30);[32](https://arxiv.org/html/2608.15085#bib.bib12)and cross\-modal visual inputs[29](https://arxiv.org/html/2608.15085#bib.bib4);[27](https://arxiv.org/html/2608.15085#bib.bib13)progressively converge into a unified geometry\. Driven by the language\-centric architecture of contemporary MLLMs[20](https://arxiv.org/html/2608.15085#bib.bib21);[21](https://arxiv.org/html/2608.15085#bib.bib22)and imbalanced pre\-training data, this shared hub pragmatically manifests as an English\-dominated space[19](https://arxiv.org/html/2608.15085#bib.bib11);[34](https://arxiv.org/html/2608.15085#bib.bib23)\. Thus, we formalize the English Semantic Manifold \(𝒮e​n\\mathcal\{S\}\_\{en\}\) as the target coordinate system where intermediate hidden stateshlh\_\{l\}progressively converge across layers:hl→layersh∗∈𝒮e​nh\_\{l\}\\xrightarrow\{\\text\{layers\}\}h^\{\*\}\\in\\mathcal\{S\}\_\{en\}\.

##### Dynamics of Multilingual and Modal Convergence\.

Let𝒮s​r​c\\mathcal\{S\}\_\{src\}denote the source\-language representational space and𝒮v​i​s\\mathcal\{S\}\_\{vis\}the initial perceptual space occupied by visual tokens\. Within this unified framework, the recurrencehl=fl​\(hl−1\)h\_\{l\}=f\_\{l\}\(h\_\{l\-1\}\)traces an incremental trajectory toward the English semantic manifold, wherehlh\_\{l\}denotes the hidden state at layerll\. We useFl=fl∘⋯∘f1F\_\{l\}=f\_\{l\}\\circ\\cdots\\circ f\_\{1\}to denote the cumulative transformation through layerll\. Both streams exhibit convergence toward𝒮e​n\\mathcal\{S\}\_\{en\}, yet follow distinct trajectories: the linguistic stream is hypothesized to undergo an implicit transition from𝒮s​r​c\\mathcal\{S\}\_\{src\}toward𝒮e​n\\mathcal\{S\}\_\{en\}underFlF\_\{l\}, while the visual stream undergoes visual semanticization from𝒮v​i​s\\mathcal\{S\}\_\{vis\}toward𝒮e​n\\mathcal\{S\}\_\{en\}\.

Crucially, for the MLLM to perform true multimodal reasoning, the implicit translation of the text stream should theoretically be grounded in the visual context\. Since Transformer architectures aggregate information via layer\-wise self\-attention, this grounding hinges on thetemporal coordinationbetween the two paths\. If visual tokens have not yet achieved visual semanticization during the specific layer window where translation occurs, the linguistic stream could default to a purely text\-based transformation\. Such a mechanistic asynchrony would result in a bypass of visual evidence, potentially explaining the reasoning failures observed in multilingual contexts\. In the following section, we empirically measure the precise layer\-wise timing of these two processes to test this asynchrony hypothesis\.

### 3\.1Modality Asynchrony: Early Translation, Delayed Semanticization

![Refer to caption](https://arxiv.org/html/2608.15085v1/fig2.png)Figure 2:Modality Asynchrony\.\(a\-b\)Layer\-wise English Similarity \(Sime​n\\text\{Sim\}\_\{en\}\) and English Translation Ratio \(RE​TR\_\{ET\}\) in text stream\.\(c\)Visual Grounding Score \(SV​GS\_\{VG\}\) in vision stream\. Shaded regions indicate the standard error of the mean \(SEM\)\.To investigate whether these pathways evolve synchronously, we extract intermediate hidden stateshlh\_\{l\}under a controlled input condition where visual tokens are paired with multilingual text descriptions\. All of the experimental details and metric definitions can be found in Appendix[A](https://arxiv.org/html/2608.15085#A1)\. We monitor the dual\-stream convergence using three complementary metrics\. All vocabulary\-level projections employ thelogit lenstechnique[16](https://arxiv.org/html/2608.15085#bib.bib15), which maps intermediate hidden states to probability distributions over the vocabulary via the pre\-trained unembedding matrix\.

- •Text Stream:We track the continuous geometric rotation and discrete lexical transition toward𝒮e​n\\mathcal\{S\}\_\{en\}usingEnglish Similarity \(Sime​n\\text\{Sim\}\_\{en\}\)and theEnglish Translation Ratio \(RE​TR\_\{ET\}\), respectively\. Specifically,Sime​n\\text\{Sim\}\_\{en\}measures the cosine similarity between the mean hidden states of the source and English reference sequences\.RE​TR\_\{ET\}extracts the top\-kkdecoded words from each text token via the logit lens and measures the proportion classified as English by a zero\-shot language classifier\.
- •Vision Stream:We directly probe the visual semanticization process using theVisual Grounding Score \(SV​GS\_\{VG\}\), tracking when visual tokens decode into ground\-truth object labels via the logit lens\.

##### Finding 1: Early Linguistic Translation\.

BothSime​n\\text\{Sim\}\_\{en\}andRE​TR\_\{ET\}reveal rapid linguistic convergence within the initial layers\. As shown in[Fig\.2](https://arxiv.org/html/2608.15085#S3.F2)\(a\-b\), source language representations efficiently rotate toward the English representation and decode into English vocabulary as early as layers 1–5\. This demonstrates that the English\-dominant linguistic transition becomes strongly detectable in the earliest stages of the trajectory\.

##### Finding 2: Delayed Visual Semanticization\.

In contrast, visual semantics emerge later\.[Fig\.2](https://arxiv.org/html/2608.15085#S3.F2)\(c\) shows thatSV​GS\_\{VG\}remains near\-zero throughout early\-to\-middle layers, only rising beyond layer 20\. Bridging the cross\-modal gap \(hl:𝒮v​i​s→𝒮e​nh\_\{l\}:\\mathcal\{S\}\_\{vis\}\\rightarrow\\mathcal\{S\}\_\{en\}\) appears more demanding than intra\-lingual translation, creating a pronounced Modality Asynchrony\.

##### The Ghost Anchor Phenomenon\.

The asynchrony suggests a mechanistic misalignment: by the time the linguistic stream settles into𝒮e​n\\mathcal\{S\}\_\{en\}, visual tokens remain close to their initial perceptual manifold𝒮v​i​s\\mathcal\{S\}\_\{vis\}\. At this stage, they primarily encode low\-level sensory features and provide limited assistance to the translation process\. We characterize this temporal separation as the Ghost Anchor phenomenon—visual tokens are physically present but exert limited measurable influence during early alignment\. We test this hypothesis through controlled intervention in[Section3\.2](https://arxiv.org/html/2608.15085#S3.SS2)\.

### 3\.2Causal Validation of the Ghost Anchor

WhileRE​TR\_\{ET\}is already high in early layers \([Fig\.2](https://arxiv.org/html/2608.15085#S3.F2)\(b\)\), the critical question is whether this translation trajectory is sensitive to visual context or predominantly driven by textual priors\. To test this, we replace each visual input with matched Gaussian noise—preserving the input dimensions, patch count, and token positions while substantially disrupting recognizable objects and scene structure \(see Appendix[A\.4](https://arxiv.org/html/2608.15085#A1.SS4)\)\. We therefore measure theVisual Causal Effecton the text stream via both the continuous geometric shift \(Δ​Sime​n\\Delta\\text\{Sim\}\_\{en\}\) and the discrete lexical transition \(Δ​RE​T\\Delta R\_\{ET\}\)\. As shown in[Fig\.3](https://arxiv.org/html/2608.15085#S3.F3), both metrics remain near\-zero across early layers for LLaVA\-1\.5\. These small changes indicate that the measured early translation trajectory has limited sensitivity to high\-level visual content under this intervention, providing evidence consistent with the Ghost Anchor phenomenon\. We further conduct experiments on a broader range of SOTA MLLMs in Appendix[C\.1](https://arxiv.org/html/2608.15085#A3.SS1), which show similarly small effects\.

![Refer to caption](https://arxiv.org/html/2608.15085v1/fig3.png)Figure 3:Visual Causal Effect\(LLaVA\-1\.5\-7B left, 13B right\)\. Difference between Normal and Noise inputs \(Δ=Normal−Noise\\Delta=\\text\{Normal\}\-\\text\{Noise\}\) across layers\. Shaded regions indicate the standard error of the mean \(SEM\)\.

## 4ANCHOR: Rectifying Modality Asynchrony through Proactive Visual Anchoring

### 4\.1Method Overview

Motivated by our mechanistic discovery of the Ghost Anchor effect, we propose ANCHOR to structurally synchronize the disparate evolutionary trajectories of text and vision\. Unlike data\-centric interventions that force external alignment, ANCHOR explicitly supervises the early\-layer evolution of visual representations\. Its core component, Proactive Visual Anchoring \(PVA\), leverages an external visual foundation model \(VFM\) to enforce visual semantic readiness before the critical alignment window closes, ensuring visual features actively guide linguistic translation\.

### 4\.2Proactive Visual Anchoring

Standard vision\-language training treats the vision encoder as a fixed feature extractor and only adapts the language side\. However, under text\-only supervision, fine\-grained visual information may not be preserved as visual tokens propagate through the LLM’s early layers\. By the time the semantic verbalization process completes, the cross\-modal alignment window has already closed\. PVA therefore uses an external VFM as a semantic target to encourage visually grounded representations in the early layers\.

##### Alignment Objective\.

Given the MLLM’s intermediate hidden stateshi,l∈ℝdMLLMh\_\{i,l\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{MLLM\}\}\}for visual tokeniiat layerll, we pass them through a lightweight alignment networkMLPa​l​i​g​n​\(⋅\)\\text\{MLP\}\_\{align\}\(\\cdot\)and compare the output against the semantic supervisor from a pre\-trained VFM\. Following the formulation in prior representation alignment work, the Proactive Visual Anchoring loss is defined as the negative mean cosine similarity across all visual tokens:

ℒa​n​c=−1\|𝒱v​i​s\|∑i∈𝒱v​i​scos\(MLPa​l​i​g​n\(hi,Le​a​r​l​y\),zi,V​F​M\),\\mathcal\{L\}\_\{anc\}=\-\\frac\{1\}\{\|\\mathcal\{V\}\_\{vis\}\|\}\\sum\_\{i\\in\\mathcal\{V\}\_\{vis\}\}\\text\{cos\}\\big\(\\text\{MLP\}\_\{align\}\(h\_\{i,L\_\{early\}\}\),z\_\{i,VFM\}\\big\),\(1\)where\|𝒱v​i​s\|\|\\mathcal\{V\}\_\{vis\}\|denotes the number of visual tokens, andzi,V​F​M∈ℝdVFMz\_\{i,VFM\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{VFM\}\}\}is the fine\-grained semantic feature extracted from SigLIP\-SO400M/14[26](https://arxiv.org/html/2608.15085#bib.bib35)for visual tokenii\. The alignment networkMLPa​l​i​g​n:ℝdMLLM→ℝdVFM\\text\{MLP\}\_\{align\}:\\mathbb\{R\}^\{d\_\{\\mathrm\{MLLM\}\}\}\\rightarrow\\mathbb\{R\}^\{d\_\{\\mathrm\{VFM\}\}\}consists of a two\-layer perceptron with GELU activation that projects the MLLM hidden states into the VFM feature space\.

The key design choices are: \(1\) we apply supervision at layerLe​a​r​l​yL\_\{early\}, where visual semantics should become available for cross\-modal alignment; and \(2\) we use a VFM \(SigLIP\-SO400M/14\) as the supervisor because it provides visually grounded semantic features that can serve as a shared reference across target languages\.

Combined Training Objective\.The full training objective combines the standard visual instruction tuning loss with the PVA anchoring loss:

ℒ=ℒS​F​T\+α⋅ℒa​n​c,\\mathcal\{L\}=\\mathcal\{L\}\_\{SFT\}\+\\alpha\\cdot\\mathcal\{L\}\_\{anc\},\(2\)whereℒS​F​T\\mathcal\{L\}\_\{SFT\}is the standard next\-token prediction loss on multilingual image\-text pairs, andα\\alphacontrols the strength of the anchoring regularization\. During training, we apply full\-parameter fine\-tuning \(FFT\) to both the multimodal projector and the firstLe​a​r​l​yL\_\{early\}transformer layers, freeze the remaining layers and the vision encoder, and train the alignment networkMLPa​l​i​g​n\\text\{MLP\}\_\{align\}\. The ablation in[SectionC\.3](https://arxiv.org/html/2608.15085#A3.SS3)suggests that restricting the intervention to this early window better preserves downstream performance than unconstrained full\-depth updates\.

By supervising the early\-layer trajectory of visual tokens, PVA effectively shifts the visual semantic emergence curve earlier in the model’s depth, ensuring that when the linguistic stream begins its cross\-modal alignment phase, grounded visual evidence is already available and can actively influence the alignment process\.

## 5Empirical Results and Analysis

To empirically validate the ANCHOR framework, we structure our evaluation around three core research questions:RQ1Multilingual Performance: Does ANCHOR improve visual reasoning across fine\-tuned and zero\-shot languages without degrading native English proficiency?RQ2Mechanistic Rectification: Does the framework mitigate the Ghost Anchor phenomenon by increasing the influence of visual signals during early translation?RQ3Intervention Design: Does explicit PVA supervision during the early alignment window improve performance over unanchored early\-layer adaptation?

### 5\.1Experiment Settings

##### Implementation Details

We implement ANCHOR using LLaVA\-1\.5 as our base MLLM, a classic architecture that is widely used for interpretability research\. For PVA, we employ SigLIP\-SO400M/14 as the external Visual Foundation Model\. Following our mechanistic analysis, we setLe​a​r​l​y=10L\_\{early\}=10as the early alignment window\. Models are fine\-tuned on 135K image\-text pairs \(15K × 9 languages\) from ShareGPT4V[6](https://arxiv.org/html/2608.15085#bib.bib16)\. See[SectionB\.1](https://arxiv.org/html/2608.15085#A2.SS1)for dataset details\.

##### Baselines

To isolate the efficacy of PVA, we compare ANCHOR against two baselines: \(1\)Std\. LoRA, which fine\-tunes without any visual\-side intervention; and \(2\)Early\-Layer FFT, which applies full\-parameter fine\-tuning to the firstLe​a​r​l​yL\_\{early\}transformer layers but omits the PVA objective\.

##### Benchmarks

We evaluate multilingual proficiency across three benchmarks targeting distinct dimensions: \(1\)xMMMU[30](https://arxiv.org/html/2608.15085#bib.bib3)for deep semantic reasoning in expert domains; \(2\)MaXM[5](https://arxiv.org/html/2608.15085#bib.bib18)for cross\-lingual transfer on low\-resource languages; and \(3\)CVQA[18](https://arxiv.org/html/2608.15085#bib.bib17)for culturally nuanced visual mappings\. Crucially, our evaluation covers both languages included in multilingual SFT and languages unseen during training, enabling us to assess generalization across both seen and zero\-shot settings\.

\(a\) Results on the xMMMU

SizeModelsenSFT LanguagesZero\-shotAvg\.arfrhiptidja7BLLaVA\-1\.535\.229\.934\.628\.233\.731\.632\.032\.2\+ Std\. LoRA34\.932\.936\.630\.236\.032\.030\.933\.4\+ Early\-Layer FFT32\.628\.932\.224\.131\.634\.332\.030\.8\+ ANCHOR35\.034\.839\.132\.639\.233\.934\.435\.613BLLaVA\-1\.534\.830\.935\.629\.237\.033\.730\.933\.2\+ Std\. LoRA34\.831\.535\.630\.236\.033\.031\.233\.2\+ Early\-Layer FFT33\.231\.235\.229\.633\.730\.628\.331\.7\+ ANCHOR35\.136\.337\.534\.237\.335\.233\.135\.5

\(b\) Results on the MaXM

SizeModelsenSFT LanguagesZero\-shotAvg\.frhithzhhero7BLLaVA\-1\.549\.432\.617\.717\.227\.812\.915\.124\.7\+ Std\. LoRA49\.033\.037\.730\.230\.316\.118\.330\.7\+ Early\-Layer FFT40\.530\.732\.326\.936\.519\.316\.929\.0\+ ANCHOR51\.336\.838\.438\.229\.314\.718\.332\.413BLLaVA\-1\.551\.830\.315\.820\.122\.014\.323\.625\.4\+ Std\. LoRA48\.634\.938\.332\.830\.316\.126\.832\.5\+ Early\-Layer FFT36\.635\.233\.534\.040\.013\.635\.632\.6\+ ANCHOR54\.139\.139\.337\.236\.920\.528\.136\.5

Table 1:Performance comparison on multilingual multimodal benchmarks: \(a\) xMMMU and \(b\) MaXM\. Best results are highlighted inbold\.

### 5\.2Quantitative Results

To systematically assess the macroscopic generalization of our framework, we evaluate ANCHOR on xMMMU, MaXM, and CVQA\. Results across all evaluations consistently exhibit three core trends that corroborate our hypotheses:

Retention of the English Semantic Manifold\.A persistent challenge in cross\-lingual alignment is degradation of the backbone’s native English proficiency\. Our evaluations indicate that ANCHOR generally preserves—and in certain cases, enhances—this capability\. For instance, while maintaining stable English performance on xMMMU, ANCHOR yields improvements on MaXM\. These results suggest that early\-layer visual grounding via PVA can integrate visual anchors without substantially interfering with the LLM’s established English\-centric reasoning pathways\.

Substantial Gains in Target SFT Languages\.For non\-English languages explicitly encountered during fine\-tuning, ANCHOR demonstrates pronounced improvements\. Compared with the degradation observed for the Early\-Layer FFT baseline, ANCHOR better preserves performance while updating early layers\. For example, on the 13B CVQA benchmark, standard early\-layer adaptation reduces Arabic accuracy from 38\.4% to 28\.1%; in contrast, ANCHOR improves it to 40\.1% and raises the average across the six displayed language–region pairs to 50\.3%, surpassing both the base model and standard LoRA on this displayed subset\. This contrast supports our mechanistic premise and indicates that explicit visual anchoring is beneficial when adapting early layers\.

Robust Zero\-Shot Cross\-Lingual Generalization\.The benefits of mitigating modality asynchrony extend to unseen linguistic spaces, reducing the cross\-lingual ”alignment tax” often incurred during SFT\. When transferring to zero\-shot languages, standard baselines can degrade because of overfitting to the target SFT distribution\. ANCHOR cushions this degradation on CVQA and achieves absolute zero\-shot improvements on xMMMU \(e\.g\., zero\-shot Japanese improving from 32\.0% to 34\.4% on the 7B model\)\. These results suggest that mitigating the Ghost Anchor phenomenon can produce a more transferable cross\-modal alignment space\.

SizeModelsSFT LanguagesZero\-shotAvg\.ar\-EGes\-ESzh\-CNam\-ETja\-JPro\-RO7BLLaVA\-1\.532\.567\.048\.626\.535\.051\.043\.4\+ Std\. LoRA31\.562\.044\.726\.135\.546\.741\.1\+ Early\-Layer FFT29\.650\.342\.827\.435\.544\.438\.3\+ ANCHOR37\.369\.349\.126\.436\.937\.442\.713BLLaVA\-1\.538\.470\.454\.325\.245\.857\.048\.5\+ Std\. LoRA36\.067\.950\.526\.142\.452\.345\.9\+ Early\-Layer FFT28\.152\.241\.836\.837\.044\.740\.1\+ ANCHOR40\.170\.859\.426\.047\.558\.050\.3

Table 2:Performance on CVQA\. We show six language–region pairs whose languages include both those used in SFT and those unseen during training; full results are reported in[Table8](https://arxiv.org/html/2608.15085#A3.T8)\. The Avg\. column reports the average accuracy across the six displayed language–region pairs\. Best results are highlighted inbold\.##### Generalization to Newer MLLMs\.

To examine whether ANCHOR extends beyond LLaVA\-1\.5, we additionally evaluate it on LLaVA\-NeXT \(7B/13B\) and InternVL3 \(8B/14B\), using the same training and evaluation settings as the corresponding Std\. LoRA baselines\. ANCHOR consistently improves over Std\. LoRA across the English, SFT\-language, and zero\-shot\-language groups\. Detailed gains on xMMMU, MaXM, and CVQA are reported in[Tables5](https://arxiv.org/html/2608.15085#A3.T5),[6](https://arxiv.org/html/2608.15085#A3.T6)and[7](https://arxiv.org/html/2608.15085#A3.T7)\.

![Refer to caption](https://arxiv.org/html/2608.15085v1/fig4.png)Figure 4:\(a\)Average visual causal effect \(Δ​RE​T\\Delta R\_\{ET\}andΔ​Sime​n\\Delta\\text\{Sim\}\_\{en\}\) across 7 languages\.\(b\)Layer\-wise hidden state dynamics in a cross\-lingual VQA case, showing how ANCHOR enables visual grounding to override textual priors during translation\.

### 5\.3Mechanistic Analysis: Validating Ghost Anchor Rectification

We mechanistically validate how the ANCHOR framework rectifies the Ghost Anchor phenomenon through \(1\) quantifying the visual causal influence on the early translation stage, and \(2\) examining a layer\-wise case study where visual anchoring successfully overrides incorrect linguistic priors\.

##### Enabling Visual Causal Effect

Following the methodology in[Section3\.2](https://arxiv.org/html/2608.15085#S3.SS2), we measure the visual causal effect \(Δ​RE​T\\Delta R\_\{ET\}andΔ​S​i​me​n\\Delta Sim\_\{en\}\) across the early layers \(l∈\[1,10\]l\\in\[1,10\]\)\. As shown in[Fig\.4](https://arxiv.org/html/2608.15085#S5.F4)\(a\), standard LLaVA\-1\.5 exhibits limited visual influence, with baselineΔ​RE​T\\Delta R\_\{ET\}averaging near zero \(−0\.016%\-0\.016\\%for 7B and1\.18%1\.18\\%for 13B\)\. This provides evidence consistent with the Ghost Anchor effect\. In contrast, ANCHOR produces a positive shift in bothΔ​RE​T\\Delta R\_\{ET\}andΔ​Sime​n\\Delta\\text\{Sim\}\_\{en\}across model scales, indicating increased visual influence on the measured early translation process\.

##### Case Study: Visual Anchoring Overriding Linguistic Priors

To contextualize this shift, we examine a cross\-lingual VQA case where the baseline follows a text\-induced prior and outputslaptoprather than the visual target, atable\(mesain Portuguese\)\. To trace this internal semantic evolution, we compute the layer\-wise cosine similarity between the final prompt token’s normalized hidden state and the static unembedding weight vectors of specific target concepts\. As shown in[Fig\.4](https://arxiv.org/html/2608.15085#S5.F4)\(b\), the baseline’s early\-layer hidden states exhibit stronger alignment with the misleading linguistic prior\. In contrast, ANCHOR shifts the internal trajectory toward the visually supported English concept \(table\), which is consistent with its final Portuguese output \(mesa\)\.

## 6Limitations

Our analysis operationalizes the English Semantic Manifold as an empirical property of the English\-dominant MLLMs studied in this work, rather than as a universal or normative semantic space\. Models trained with different language distributions or architectural designs may exhibit different intermediate semantic geometries\. In addition, our mechanistic metrics, including the logit lens, English Translation Ratio, and Visual Grounding Score, measure the decodability of semantic information from intermediate representations, which does not necessarily imply that the model uses all decoded information during generation\. Although the Gaussian\-noise intervention provides controlled evidence about the influence of visual content, it also introduces a distribution shift and may alter input properties beyond high\-level semantics\. Therefore, our results characterize modality asynchrony under these operationalizations rather than providing an exhaustive account of all cross\-modal causal pathways\.

## 7Conclusion

In this work, we investigate why visual signals provide limited assistance as cross\-lingual bridges for multilingual MLLMs\. Mechanistic analysis identifies the Ghost Anchor phenomenon associated with Modality Asynchrony: linguistic streams quickly converge to an English\-centric semantic manifold in early layers, whereas visual semantics emerge later, limiting the influence of visual inputs during early cross\-modal alignment\. To address this, we propose ANCHOR, which leverages Proactive Visual Anchoring \(PVA\) with an external Visual Foundation Model to accelerate early\-layer visual semantic emergence and encourage visual evidence to guide linguistic translation\. Experiments on xMMMU, MaXM, and CVQA show that ANCHOR improves cross\-lingual visual reasoning and mitigates the performance degradation associated with unanchored early\-layer adaptation\.

## Source Availability

## Acknowledgements

This work is supported by Longgang District Special Funds for Science and Technology Innovation under Grant LGXXKYJG2024001\.

## References

- Ahmatet al\.\(2025\)A\. Ahmat, L\. Wang, Y\. Yang, B\. Ma, R\. Dong, K\. Lu, R\. Ma, and X\. WangM2\-vlp: enhancing multilingual vision\-language pre\-training via multi\-grained alignment\.InProceedings of the ACM on Web Conference 2025,WWW ’25,New York, NY, USA,pp\. 3438–3450\.External Links:ISBN 9798400712746,[Link](https://doi.org/10.1145/3696410.3714861),[Document](https://dx.doi.org/10.1145/3696410.3714861)Cited by:[§2](https://arxiv.org/html/2608.15085#S2.SS0.SSS0.Px2.p1.1)\.
- Baiet al\.\(2025a\)S\. Bai, Y\. Cai, R\. Chen, K\. Chen, X\. Chen, Z\. Cheng, L\. Deng, W\. Ding, C\. Gao, C\. Ge,et al\.Qwen3\-vl technical report\.arXiv preprint arXiv:2511\.21631\.Cited by:[§1](https://arxiv.org/html/2608.15085#S1.p1.1)\.
- Baiet al\.\(2025b\)S\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge, S\. Song, K\. Dang, P\. Wang, S\. Wang, J\. Tang, H\. Zhong, Y\. Zhu, M\. Yang, Z\. Li, J\. Wan, P\. Wang, W\. Ding, Z\. Fu, Y\. Xu, J\. Ye, X\. Zhang, T\. Xie, Z\. Cheng, H\. Zhang, Z\. Yang, H\. Xu, and J\. LinQwen2\.5\-vl technical report\.arXiv preprint arXiv:2502\.13923\.Cited by:[§C\.1](https://arxiv.org/html/2608.15085#A3.SS1.p1.1)\.
- Bugliarelloet al\.\(2022\)E\. Bugliarello, F\. Liu, J\. Pfeiffer, S\. Reddy, D\. Elliott, E\. M\. Ponti, and I\. VulićIGLUE: a benchmark for transfer learning across modalities, tasks, and languages\.InInternational Conference on Machine Learning,pp\. 2370–2392\.Cited by:[§1](https://arxiv.org/html/2608.15085#S1.p3.1)\.
- Changpinyoet al\.\(2023\)S\. Changpinyo, L\. Xue, M\. Yarom, A\. Thapliyal, I\. Szpektor, J\. Amelot, X\. Chen, and R\. SoricutMaxm: towards multilingual visual question answering\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 2667–2682\.Cited by:[§B\.2](https://arxiv.org/html/2608.15085#A2.SS2.SSS0.Px2.p1.1),[§B\.2](https://arxiv.org/html/2608.15085#A2.SS2.p1.1),[§1](https://arxiv.org/html/2608.15085#S1.p1.1),[§1](https://arxiv.org/html/2608.15085#S1.p6.1),[§5\.1](https://arxiv.org/html/2608.15085#S5.SS1.SSS0.Px3.p1.1)\.
- Chenet al\.\(2024\)L\. Chen, J\. Li, X\. Dong, P\. Zhang, C\. He, J\. Wang, F\. Zhao, and D\. LinSharegpt4v: improving large multi\-modal models with better captions\.InEuropean Conference on Computer Vision,pp\. 370–387\.Cited by:[§B\.1](https://arxiv.org/html/2608.15085#A2.SS1.p1.1),[§5\.1](https://arxiv.org/html/2608.15085#S5.SS1.SSS0.Px1.p1.1)\.
- Dashet al\.\(2025\)S\. Dash, Y\. Nan, J\. Dang, A\. Ahmadian, S\. Singh, M\. Smith, B\. Venkitesh, V\. Shmyhlo, V\. Aryabumi, W\. Beller\-Morales,et al\.Aya vision: advancing the frontier of multilingual multimodality\.arXiv preprint arXiv:2505\.08751\.Cited by:[§B\.2](https://arxiv.org/html/2608.15085#A2.SS2.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2608.15085#S1.p2.1),[§2](https://arxiv.org/html/2608.15085#S2.SS0.SSS0.Px2.p1.1)\.
- Elshabrawyet al\.\(2025\)A\. Elshabrawy, H\. Kaing, H\. Song, A\. F\. Aji, H\. Tanaka, M\. Utiyama, and R\. DabreWhen alignment hurts: decoupling representational spaces in multilingual models\.arXiv preprint arXiv:2508\.12803\.Cited by:[§2](https://arxiv.org/html/2608.15085#S2.SS0.SSS0.Px1.p1.1)\.
- Geigleet al\.\(2024\)G\. Geigle, A\. Jain, R\. Timofte, and G\. GlavašMBLIP: efficient bootstrapping of multilingual vision\-LLMs\.InProceedings of the 3rd Workshop on Advances in Language and Vision Research \(ALVR\),J\. Gu, T\. \(\. Fu, D\. Hudson, A\. Celikyilmaz, and W\. Wang \(Eds\.\),Bangkok, Thailand,pp\. 7–25\.External Links:[Link](https://aclanthology.org/2024.alvr-1.2/),[Document](https://dx.doi.org/10.18653/v1/2024.alvr-1.2)Cited by:[§2](https://arxiv.org/html/2608.15085#S2.SS0.SSS0.Px2.p1.1)\.
- Gemma Team \(2025\)Gemma TeamGemma 3 technical report\.External Links:2503\.19786,[Link](https://arxiv.org/abs/2503.19786)Cited by:[§C\.1](https://arxiv.org/html/2608.15085#A3.SS1.p1.1)\.
- Huanget al\.\(2021\)P\. Huang, M\. Patrick, J\. Hu, G\. Neubig, F\. Metze, and A\. G\. HauptmannMultilingual multimodal pre\-training for zero\-shot cross\-lingual transfer of vision\-language models\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 2443–2459\.Cited by:[§2](https://arxiv.org/html/2608.15085#S2.SS0.SSS0.Px2.p1.1)\.
- Liet al\.\(2025\)J\. Li, Q\. Yang, B\. Jiang, S\. Zhu, and Q\. SunLRM\-llava: overcoming the modality gap of multilingual large language\-vision model for low\-resource languages\.InProceedings of the AAAI Conference on Artificial Intelligence,pp\. 24449–24457\.Cited by:[§1](https://arxiv.org/html/2608.15085#S1.p2.1),[§2](https://arxiv.org/html/2608.15085#S2.SS0.SSS0.Px2.p1.1)\.
- Linet al\.\(2014\)T\. Lin, M\. Maire, S\. Belongie, J\. Hays, P\. Perona, D\. Ramanan, P\. Dollár, and C\. L\. ZitnickMicrosoft coco: common objects in context\.InEuropean conference on computer vision,pp\. 740–755\.Cited by:[§A\.1](https://arxiv.org/html/2608.15085#A1.SS1.p1.1)\.
- Liuet al\.\(2024\)H\. Liu, C\. Li, Y\. Li, B\. Li, Y\. Zhang, S\. Shen, and Y\. J\. LeeLLaVA\-next: improved reasoning, ocr, and world knowledge\.External Links:[Link](https://llava-vl.github.io/blog/2024-01-30-llava-next/)Cited by:[§C\.1](https://arxiv.org/html/2608.15085#A3.SS1.p1.1)\.
- Niet al\.\(2021\)M\. Ni, H\. Huang, L\. Su, E\. Cui, T\. Bharti, L\. Wang, D\. Zhang, and N\. DuanM3p: learning universal representations via multitask multilingual multimodal pre\-training\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 3977–3986\.Cited by:[§1](https://arxiv.org/html/2608.15085#S1.p2.1),[§2](https://arxiv.org/html/2608.15085#S2.SS0.SSS0.Px2.p1.1)\.
- Nostalgebraist \(2020\)NostalgebraistInterpreting gpt: the logit lens\.Note:[https://www\.alignmentforum\.org/posts/AcKRB8wDpdaN6v6ru/interpreting\-gpt\-the\-logit\-lens](https://www.alignmentforum.org/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens)Accessed: 5 Jan 2026Cited by:[§A\.3](https://arxiv.org/html/2608.15085#A1.SS3.p1.1),[§3\.1](https://arxiv.org/html/2608.15085#S3.SS1.p1.1)\.
- Pfeifferet al\.\(2022\)J\. Pfeiffer, G\. Geigle, A\. Kamath, J\. O\. Steitz, S\. Roth, I\. Vulić, and I\. GurevychXGQA: cross\-lingual visual question answering\.InFindings of the association for computational linguistics: ACL 2022,pp\. 2497–2511\.Cited by:[§1](https://arxiv.org/html/2608.15085#S1.p1.1)\.
- Romeroet al\.\(2024\)D\. Romero, C\. Lyu, H\. A\. Wibowo, T\. Lynn, I\. Hamed, A\. N\. Kishore, A\. Mandal, A\. Dragonetti, A\. Abzaliev, A\. L\. Tonja,et al\.Cvqa: culturally\-diverse multilingual visual question answering benchmark\.arXiv preprint arXiv:2406\.05967\.Cited by:[§B\.2](https://arxiv.org/html/2608.15085#A2.SS2.SSS0.Px3.p1.1),[§B\.2](https://arxiv.org/html/2608.15085#A2.SS2.p1.1),[§1](https://arxiv.org/html/2608.15085#S1.p6.1),[§5\.1](https://arxiv.org/html/2608.15085#S5.SS1.SSS0.Px3.p1.1)\.
- Schutet al\.\(2025\)L\. Schut, Y\. Gal, and S\. FarquharDo multilingual llms think in english?\.arXiv preprint arXiv:2502\.15603\.Cited by:[§2](https://arxiv.org/html/2608.15085#S2.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2608.15085#S3.SS0.SSS0.Px1.p1.1)\.
- Shenet al\.\(2025\)S\. Shen, J\. Sun, Q\. Huang, and S\. WangVL\-sae: interpreting and enhancing vision\-language alignment with a unified concept set\.arXiv preprint arXiv:2510\.21323\.Cited by:[§3](https://arxiv.org/html/2608.15085#S3.SS0.SSS0.Px1.p1.1)\.
- Shuet al\.\(2025\)D\. Shu, H\. Zhao, J\. Hu, W\. Liu, A\. Payani, L\. Cheng, and M\. DuLarge vision\-language model alignment and misalignment: a survey through the lens of explainability\.arXiv preprint arXiv:2501\.01346\.Cited by:[§3](https://arxiv.org/html/2608.15085#S3.SS0.SSS0.Px1.p1.1)\.
- Singhet al\.\(2025\)A\. Singh, A\. Fry, A\. Perelman, A\. Tart, A\. Ganesh, A\. El\-Kishky, A\. McLaughlin, A\. Low, A\. Ostrow, A\. Ananthram,et al\.Openai gpt\-5 system card\.arXiv preprint arXiv:2601\.03267\.Cited by:[§A\.1](https://arxiv.org/html/2608.15085#A1.SS1.p1.1),[§B\.1](https://arxiv.org/html/2608.15085#A2.SS1.p1.1)\.
- Songet al\.\(2025\)S\. Song, X\. Li, S\. Li, S\. Zhao, J\. Yu, J\. Ma, X\. Mao, W\. Zhang, and M\. WangHow to bridge the gap between modalities: survey on multimodal large language model\.IEEE Transactions on Knowledge and Data Engineering37\(9\),pp\. 5311–5329\.Cited by:[§3](https://arxiv.org/html/2608.15085#S3.SS0.SSS0.Px1.p1.1)\.
- Sundaret al\.\(2025\)A\. Sundar, S\. Williamson, K\. Metcalf, B\. Theobald, S\. Seto, and M\. FedzechkinaSteering into new embedding spaces: analyzing cross\-lingual alignment induced by model interventions in multilingual language models\.arXiv preprint arXiv:2502\.15639\.Cited by:[§2](https://arxiv.org/html/2608.15085#S2.SS0.SSS0.Px1.p1.1)\.
- Tonget al\.\(2026\)S\. Tong, D\. Fan, J\. Nguyen, E\. Brown, G\. Zhou, S\. Qian, B\. Zheng, T\. Vallaeys, J\. Han, R\. Fergus,et al\.Beyond language modeling: an exploration of multimodal pretraining\.arXiv preprint arXiv:2603\.03276\.Cited by:[§1](https://arxiv.org/html/2608.15085#S1.p1.1)\.
- Tschannenet al\.\(2025\)M\. Tschannen, A\. Gritsenko, X\. Wang, M\. F\. Naeem, I\. Alabdulmohsin, N\. Parthasarathy, T\. Evans, L\. Beyer, Y\. Xia, B\. Mustafa,et al\.Siglip 2: multilingual vision\-language encoders with improved semantic understanding, localization, and dense features\.arXiv preprint arXiv:2502\.14786\.Cited by:[§4\.2](https://arxiv.org/html/2608.15085#S4.SS2.SSS0.Px1.p1.2)\.
- Venhoffet al\.\(2025\)C\. Venhoff, A\. Khakzar, S\. Joseph, P\. Torr, and N\. NandaHow visual representations map to language feature space in multimodal llms\.arXiv preprint arXiv:2506\.11976\.Cited by:[§3](https://arxiv.org/html/2608.15085#S3.SS0.SSS0.Px1.p1.1)\.
- Wendleret al\.\(2024\)C\. Wendler, V\. Veselovsky, G\. Monea, and R\. WestDo llamas work in English? on the latent language of multilingual transformers\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 15366–15394\.External Links:[Link](https://aclanthology.org/2024.acl-long.820/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.820)Cited by:[§2](https://arxiv.org/html/2608.15085#S2.SS0.SSS0.Px1.p1.1)\.
- Wuet al\.\(2024\)Z\. Wu, X\. V\. Yu, D\. Yogatama, J\. Lu, and Y\. KimThe semantic hub hypothesis: language models share semantic representations across languages and modalities\.arXiv preprint arXiv:2411\.04986\.Cited by:[§1](https://arxiv.org/html/2608.15085#S1.p2.1),[§2](https://arxiv.org/html/2608.15085#S2.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2608.15085#S3.SS0.SSS0.Px1.p1.1)\.
- Yueet al\.\(2024\)X\. Yue, Y\. Song, A\. Asai, S\. Kim, J\. de Dieu Nyandwi, S\. Khanuja, A\. Kantharuban, L\. Sutawika, S\. Ramamoorthy, and G\. NeubigPangea: a fully open multilingual multimodal llm for 39 languages\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§B\.2](https://arxiv.org/html/2608.15085#A2.SS2.SSS0.Px1.p1.1),[§B\.2](https://arxiv.org/html/2608.15085#A2.SS2.SSS0.Px2.p1.1),[§B\.2](https://arxiv.org/html/2608.15085#A2.SS2.p1.1),[§1](https://arxiv.org/html/2608.15085#S1.p1.1),[§1](https://arxiv.org/html/2608.15085#S1.p2.1),[§1](https://arxiv.org/html/2608.15085#S1.p6.1),[§2](https://arxiv.org/html/2608.15085#S2.SS0.SSS0.Px2.p1.1),[§5\.1](https://arxiv.org/html/2608.15085#S5.SS1.SSS0.Px3.p1.1)\.
- Zenget al\.\(2025\)H\. Zeng, S\. Han, L\. Chen, and K\. YuConverging to a lingua franca: evolution of linguistic regions and semantics alignment in multilingual large language models\.InProceedings of the 31st International Conference on Computational Linguistics,pp\. 10602–10617\.Cited by:[§1](https://arxiv.org/html/2608.15085#S1.p2.1),[§2](https://arxiv.org/html/2608.15085#S2.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2608.15085#S3.SS0.SSS0.Px1.p1.1)\.
- Zhaoet al\.\(2024\)Y\. Zhao, W\. Zhang, G\. Chen, K\. Kawaguchi, and L\. BingHow do large language models handle multilingualism?\.Advances in Neural Information Processing Systems37,pp\. 15296–15319\.Cited by:[§3](https://arxiv.org/html/2608.15085#S3.SS0.SSS0.Px1.p1.1)\.
- Zhenget al\.\(2025\)W\. Zheng, Z\. Liu, T\. Chakraborty, W\. Xu, X\. Gao, B\. C\. Z\. Tan, B\. Zou, C\. Liu, Y\. Hu, X\. Xie,et al\.MMA\-asia: a multilingual and multimodal alignment framework for culturally\-grounded evaluation\.arXiv preprint arXiv:2510\.08608\.Cited by:[§1](https://arxiv.org/html/2608.15085#S1.p3.1)\.
- Zhonget al\.\(2024\)C\. Zhong, F\. Cheng, Q\. Liu, J\. Jiang, Z\. Wan, C\. Chu, Y\. Murawaki, and S\. KurohashiBeyond english\-centric llms: what language do multilingual language models think in?\.arXiv preprint arXiv:2408\.10811\.Cited by:[§1](https://arxiv.org/html/2608.15085#S1.p1.1),[§2](https://arxiv.org/html/2608.15085#S2.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2608.15085#S3.SS0.SSS0.Px1.p1.1)\.
- Zhouet al\.\(2021\)M\. Zhou, L\. Zhou, S\. Wang, Y\. Cheng, L\. Li, Z\. Yu, and J\. LiuUc2: universal cross\-lingual cross\-modal vision\-and\-language pre\-training\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 4155–4165\.Cited by:[§1](https://arxiv.org/html/2608.15085#S1.p2.1),[§2](https://arxiv.org/html/2608.15085#S2.SS0.SSS0.Px2.p1.1)\.

## Appendix AExperimental Setup for Mechanistic Analysis

### A\.1Input Data Construction

We randomly sample 100 images from the COCO 2017 Validation Set[13](https://arxiv.org/html/2608.15085#bib.bib10)as our experimental stimuli\. To ensure strict semantic consistency across languages, we develop a high\-quality parallel corpus by translating all ground\-truth English captions into seven diverse target languages \([Table3](https://arxiv.org/html/2608.15085#A1.T3)\) using GPT\-5\.1[22](https://arxiv.org/html/2608.15085#bib.bib19)\.

LanguageAbbreviationsChinesezhSpanishesHindihiArabicarRussianruSwahiliswThaithTable 3:Selected languages used in the mechanistic experiments and their corresponding ISO 639\-1 codes\.To maintain a pure monolingual context and prevent any English prior leakage, the input sequence is formulated entirely in the target language:\[Visual Tokens\] \+ \[Target Language Instruction\] \+ \[Target Language Caption\]\. The instruction prompts the model to verify semantic consistency \(e\.g\.,”Please determine if the following description accurately describes the image\.”, translated accordingly\)\. An illustrative example of a constructed input sample across multiple languages is presented in[Fig\.5](https://arxiv.org/html/2608.15085#A1.F5)\.

![Refer to caption](https://arxiv.org/html/2608.15085v1/input_COCO_example.png)Figure 5:Illustrative example of the input construction\.This figure demonstrates the parallel input constructions across 7 structurally diverse target languages\. To enforce a strict monolingual context and prevent English prior leakage during inference, both the standardized verification instruction and the ground\-truth image caption are translated into the target language and paired with identical visual tokens from the COCO dataset\.
### A\.2Hidden State Extraction

To measure the linguistic and visual streams separately, we extract hidden states at each layerlland isolate two subsets from the full sequence:

- •Text tokens:All hidden states corresponding to the multilingual instruction and caption\.
- •Visual tokens:Only hidden states from image patches that contain labeled foreground objects \(identified via COCO segmentation masks\), excluding ambiguous background regions\.

Formally, we define the source\-language target set asℋt​e​x​t\(l\)=\{𝐡s​r​c,1,l,…,𝐡s​r​c,Ms​r​c,l\}\\mathcal\{H\}\_\{text\}^\{\(l\)\}=\\\{\\mathbf\{h\}\_\{src,1,l\},\\dots,\\mathbf\{h\}\_\{src,M\_\{src\},l\}\\\}, containing allMs​r​cM\_\{src\}source\-language instruction and caption tokens\. For English Similarity, the structurally matched English reference set isℋe​n\(l\)=\{𝐡e​n,1,l,…,𝐡e​n,Me​n,l\}\\mathcal\{H\}\_\{en\}^\{\(l\)\}=\\\{\\mathbf\{h\}\_\{en,1,l\},\\dots,\\mathbf\{h\}\_\{en,M\_\{en\},l\}\\\}, containing the correspondingMe​nM\_\{en\}English instruction and caption tokens\. The visual target set isℋv​i​s\(l\)=\{𝐡v​i​s,i,l∣i∈𝒩f​g\}\\mathcal\{H\}\_\{vis\}^\{\(l\)\}=\\\{\\mathbf\{h\}\_\{vis,i,l\}\\mid i\\in\\mathcal\{N\}\_\{fg\}\\\}, where𝒩f​g\\mathcal\{N\}\_\{fg\}denotes the index set of labeled object tokens\. These subsets are directly utilized to compute the metrics defined in Section[A\.3](https://arxiv.org/html/2608.15085#A1.SS3)\.

### A\.3Metrics Overview

We quantify layer\-wise dynamics through three complementary metrics: English Similarity \(Sime​n\\text\{Sim\}\_\{en\}\) and English Translation Ratio \(RE​TR\_\{ET\}\) for the text stream, and Visual Grounding Score \(SV​GS\_\{VG\}\) for the vision stream\. For vocabulary\-level projections, we employ thelogit lenstechnique[16](https://arxiv.org/html/2608.15085#bib.bib15), which maps an intermediate hidden state𝐡l\\mathbf\{h\}\_\{l\}to the probability distribution over the vocabulary𝒱\\mathcal\{V\}via the pre\-trained unembedding matrix𝐖U∈ℝ\|𝒱\|×d\\mathbf\{W\}\_\{U\}\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\|\\times d\}\.

#### A\.3\.1English Similarity \(Sime​n\\text\{Sim\}\_\{en\}\)

To measure continuous geometric alignment toward the English semantic manifold𝒮e​n\\mathcal\{S\}\_\{en\}, we first compute the sequence\-level mean representation for the source\-language input \(usingℋt​e​x​t\(l\)\\mathcal\{H\}\_\{text\}^\{\(l\)\}\) and its structurally matched English reference \(usingℋe​n\(l\)\\mathcal\{H\}\_\{en\}^\{\(l\)\}\)\. Both inputs contain the corresponding instruction and caption, while their token counts may differ across languages:

𝐡¯s​r​c,l=1Ms​r​c​∑j=1Ms​r​c𝐡s​r​c,j,l,𝐡¯e​n,l=1Me​n​∑j=1Me​n𝐡e​n,j,l\\overline\{\\mathbf\{h\}\}\_\{src,l\}=\\frac\{1\}\{M\_\{src\}\}\\sum\_\{j=1\}^\{M\_\{src\}\}\\mathbf\{h\}\_\{src,j,l\},\\quad\\overline\{\\mathbf\{h\}\}\_\{en,l\}=\\frac\{1\}\{M\_\{en\}\}\\sum\_\{j=1\}^\{M\_\{en\}\}\\mathbf\{h\}\_\{en,j,l\}\(3\)The layer\-wise English Similarity is then defined as their cosine similarity:

Sime​n​\(l\)=𝐡¯s​r​c,l⋅𝐡¯e​n,l‖𝐡¯s​r​c,l‖​‖𝐡¯e​n,l‖\\text\{Sim\}\_\{en\}\(l\)=\\frac\{\\overline\{\\mathbf\{h\}\}\_\{src,l\}\\cdot\\overline\{\\mathbf\{h\}\}\_\{en,l\}\}\{\\\|\\overline\{\\mathbf\{h\}\}\_\{src,l\}\\\|\\\|\\overline\{\\mathbf\{h\}\}\_\{en,l\}\\\|\}\(4\)A lowSime​n\\text\{Sim\}\_\{en\}in initial layers reflects the geometric separation between source language manifolds and the English semantic pivot\.

#### A\.3\.2English Translation Ratio \(RE​TR\_\{ET\}\)

To quantify vocabulary\-level convergence, we analyze the proportion of text tokens inℋt​e​x​t\(l\)\\mathcal\{H\}\_\{text\}^\{\(l\)\}that decode into English\. For thejj\-th source language token, we extract the top\-kkcandidate decoded words via the logit lens to form the set𝒯j,l\\mathcal\{T\}\_\{j,l\}, settingk=5k=5:

𝒯j,l=Topk​\(Softmax​\(𝐖U​𝐡s​r​c,j,l\)\)\\mathcal\{T\}\_\{j,l\}=\\text\{Top\}\_\{k\}\(\\text\{Softmax\}\(\\mathbf\{W\}\_\{U\}\\mathbf\{h\}\_\{src,j,l\}\)\)\(5\)To rigorously exclude language\-agnostic symbols \(e\.g\., punctuation, numbers\), we employ GPT\-4o as a zero\-shot language classifier, denoted asΦl​a​n​g​\(⋅\)\\Phi\_\{lang\}\(\\cdot\)\. A token is considered to have transitioned into the English space if at least one of its top\-kkdecoded words is classified as English\. The translation ratio is calculated only over tokens that bear actual linguistic content—that is, tokens for which at least one top\-kkdecoded word is classified as a natural language \(e\.g\., English, Chinese, Arabic\) rather than as punctuation, numbers, or other symbols\. Formally:

RE​T​\(l\)=∑j=1Ms​r​c𝕀\[∃w∈𝒯j,ls\.t\.Φl​a​n​g\(w\)=English\]∑j=1Ms​r​c𝕀\[∃w∈𝒯j,ls\.t\.Φl​a​n​g\(w\)∈\{Natural Languages\}\]R\_\{ET\}\(l\)=\\frac\{\\sum\_\{j=1\}^\{M\_\{src\}\}\\mathbb\{I\}\\Big\[\\exists w\\in\\mathcal\{T\}\_\{j,l\}\\text\{ s\.t\. \}\\Phi\_\{lang\}\(w\)=\\text\{English\}\\Big\]\}\{\\sum\_\{j=1\}^\{M\_\{src\}\}\\mathbb\{I\}\\Big\[\\exists w\\in\\mathcal\{T\}\_\{j,l\}\\text\{ s\.t\. \}\\Phi\_\{lang\}\(w\)\\in\\\{\\text\{Natural Languages\}\\\}\\Big\]\}\(6\)This metric directly quantifies the discrete lexical transition into the English space while accounting for the distributional nature of the output logits\.

#### A\.3\.3Visual Grounding Score \(SV​GS\_\{VG\}\)

We evaluate the emergence of visual semantics by measuring how visual tokens align with concrete linguistic concepts\. Using the labeled object hidden statesℋv​i​s\(l\)\\mathcal\{H\}\_\{vis\}^\{\(l\)\}extracted via COCO annotations, we monitor their convergence toward the set of ground\-truth object labels𝒞G​T\\mathcal\{C\}\_\{GT\}\.

For each object tokeni∈𝒩f​gi\\in\\mathcal\{N\}\_\{fg\}at layerll, we extract the top\-kkcandidate text tokens via the logit lens:𝒯i,l=Topk​\(Softmax​\(𝐖U​𝐡v​i​s,i,l\)\)\\mathcal\{T\}\_\{i,l\}=\\text\{Top\}\_\{k\}\(\\text\{Softmax\}\(\\mathbf\{W\}\_\{U\}\\mathbf\{h\}\_\{vis,i,l\}\)\), settingk=5k=5\. The grounding scoreSV​GS\_\{VG\}is defined as the successful semantic hit rate across these object tokens:

SV​G\(l\)=1\|𝒩f​g\|∑i∈𝒩f​g𝕀\[∃w∈𝒯i,l,∃c∈𝒞G​Ts\.t\.w⊆c\]S\_\{VG\}\(l\)=\\frac\{1\}\{\|\\mathcal\{N\}\_\{fg\}\|\}\\sum\_\{i\\in\\mathcal\{N\}\_\{fg\}\}\\mathbb\{I\}\\Big\[\\exists w\\in\\mathcal\{T\}\_\{i,l\},\\exists c\\in\\mathcal\{C\}\_\{GT\}\\text\{ s\.t\. \}w\\subseteq c\\Big\]\(7\)wherew⊆cw\\subseteq cdenotes that a decoded candidate word matches or is a valid substring of a ground\-truth object label\.SV​GS\_\{VG\}serves as a direct measurement for the cross\-modal semantic verbalization process\.

### A\.4Noise Intervention: Implementation Details

Motivation\.The noise intervention creates a control condition in which visual tokens remain present while recognizable high\-level visual content is substantially reduced\. This probes the sensitivity of linguistic translation to visual content while holding constant: \(i\) sequence length, \(ii\) token positions, and \(iii\) per\-channel first\- and second\-order pixel statistics\.

Implementation\.For each imageII, we compute per\-channel meanμc\\mu\_\{c\}and standard deviationσc\\sigma\_\{c\}from the original pixel values\. The noise imageInoiseI\_\{\\text\{noise\}\}is generated by sampling from a Gaussian distribution with matching statistics:

Inoise\(c\)​\(x,y\)∼𝒩⁡\(μc,σc2\),where​μc=1H​W​∑x,yI\(c\)​\(x,y\)I\_\{\\text\{noise\}\}^\{\(c\)\}\(x,y\)\\sim\\mathcal\{N\}\\left\(\\mu\_\{c\},\\sigma\_\{c\}^\{2\}\\right\),\\quad\\text\{where \}\\mu\_\{c\}=\\frac\{1\}\{HW\}\\sum\_\{x,y\}I^\{\(c\)\}\(x,y\)\(8\)The noise tensor is passed to the vision encoder viapixel\_values, preserving the input dimensions, patch count, and visual\-token positions while disrupting recognizable object and scene structure\.

![Refer to caption](https://arxiv.org/html/2608.15085v1/mask_example.png)Figure 6:\(a\)Original segmentation masks and semantic labels from the COCO dataset\.\(b\)The mapped24×2424\\times 24visual patches and their corresponding labels\.![Refer to caption](https://arxiv.org/html/2608.15085v1/figures/sup/noise_example.png)Figure 7:Visual comparison of intervention conditions\.Left:Normal RGB input with intact semantic content\.Right:Gaussian noise with matched per\-channel statistics \(μ\\mu,σ\\sigma\) but no recognizable objects or scenes\.Validation\.As shown in[Fig\.7](https://arxiv.org/html/2608.15085#A1.F7), the noise intervention removes recognizable object contours, textures, and scene structure from the input\. Because the matched noise preserves only per\-channel first\- and second\-order pixel statistics, it provides a control condition with substantially reduced high\-level visual semantics; we do not assume that it eliminates every decodable visual signal\.

## Appendix BDatasets

### B\.1Training Datasets

To construct a high\-quality multilingual visual instruction tuning dataset, we systematically sample and filter image\-text pairs from the ShareGPT4V dataset[6](https://arxiv.org/html/2608.15085#bib.bib16)\. To minimize potential noise and artifacts introduced during the machine translation process, we implement a strict filtering mechanism based on text length and annotation quality, prioritizing structurally straightforward sentences\. Specifically, we extract 9,000 image\-text pairs from the COCO subset and 6,000 from the GQA subset within ShareGPT4V\. These specific subsets are chosen because their concise and visually\-grounded question\-answering formats make them ideal anchors for cross\-lingual alignment\. Subsequently, we translate the selected English questions and answers into nine typologically diverse target languages using GPT\-5\.1\([22](https://arxiv.org/html/2608.15085#bib.bib19)\)\. These languages, namely Arabic \(ar\), Spanish \(es\), French \(fr\), Hindi \(hi\), Portuguese \(pt\), Russian \(ru\), Swahili \(sw\), Thai \(th\), and Chinese \(zh\), are specifically chosen to encompass a broad spectrum of language families and resource availability levels\. This pipeline ensures that the semantic consistency between the visual inputs and text representations is accurately preserved across different linguistic contexts\.

### B\.2Details of Evaluated Multilingual Multimodal Benchmarks

To comprehensively assess visual understanding and reasoning capabilities across diverse linguistic and cultural contexts, we evaluate the proposed method on three distinct multilingual multimodal benchmarks: xMMMU[30](https://arxiv.org/html/2608.15085#bib.bib3), MaXM[5](https://arxiv.org/html/2608.15085#bib.bib18), and CVQA[18](https://arxiv.org/html/2608.15085#bib.bib17)\.

##### Multilingual MMMU \(xMMMU\)

xMMMU[30](https://arxiv.org/html/2608.15085#bib.bib3)focuses on complex multimodal understanding and reasoning across multiple academic subjects, evaluating the capacity to understand specialized content across different languages and modalities\. The dataset features 183 subfields and 30 diverse image types, including charts, diagrams, and chemical structures\. It consists of 300 English questions randomly sampled from the MMMU validation set, which were subsequently translated into six languages \(ar, fr, hi, id, ja, pt\) using GPT\-4o\. The questions are presented in multiple\-choice and short\-answer formats\. We follow the evaluation protocol in[30](https://arxiv.org/html/2608.15085#bib.bib3)and report accuracy\.

##### Multilingual Visual Question Answering \(MaXM\)

MaXM[5](https://arxiv.org/html/2608.15085#bib.bib18)is a test\-only multilingual open\-ended visual question\-answering benchmark encompassing seven languages \(fr, hi, th, zh, en, he, ro\)\. To address the challenge of cultural diversity in multimodal understanding, the images used in MaXM were taken in regions where each language is spoken to better reflect cultural contexts\. It contains 2K questions, where answers are in the same language as the question\. We use accuracy as the evaluation metric following prior work[30](https://arxiv.org/html/2608.15085#bib.bib3);[7](https://arxiv.org/html/2608.15085#bib.bib2)\.

##### Culturally\-diverse Multilingual Visual Question Answering \(CVQA\)

CVQA[18](https://arxiv.org/html/2608.15085#bib.bib17)evaluates the ability to reason about culturally diverse visual content\. It consists of over 21K multiple\-choice questions designed to test multimodal reasoning across 39 distinct language\-region pairs, covering 31 unique languages and 30 countries\. To accurately reflect global cultural nuances, the dataset incorporates a wide spectrum of languages, ranging from high\-resource languages like Chinese and regional variations of Spanish, to lower\-resource indigenous languages such as Amharic, Javanese, Minangkabau, and Swahili\. We evaluate models under the local\-language question setting and report accuracy as the evaluation metric\.

## Appendix CAdditional Experiment Results

### C\.1Mechanistic Results on More MLLMs

[Table4](https://arxiv.org/html/2608.15085#A3.T4)reports the effect of visual intervention on the English Translation Ratio \(Δ​RET\\Delta R\_\{\\text\{ET\}\}\) and the English Similarity \(Δ​Simen\\Delta\\text\{Sim\}\_\{\\text\{en\}\}\), averaged over the early translation stage \(L∈\[1,10\]L\\in\[1,10\]\)\. Consistent with our main findings, all evaluated models, including Qwen2\.5\-VL[3](https://arxiv.org/html/2608.15085#bib.bib25), Gemma 3[10](https://arxiv.org/html/2608.15085#bib.bib27), and LLaVA\-NeXT[14](https://arxiv.org/html/2608.15085#bib.bib26), exhibit small changes under the intervention, providing evidence that the Ghost Anchor phenomenon extends across the evaluated architectures and scales\.

ModelΔ​RET↑\\Delta R\_\{\\text\{ET\}\}\\uparrowΔ​Simen↑\\Delta\\text\{Sim\}\_\{\\text\{en\}\}\\uparrowQwen2\.5\-VL 3B4\.12×10−34\.12\\times 10^\{\-3\}−2\.31×10−4\-2\.31\\times 10^\{\-4\}Qwen2\.5\-VL 7B5\.34×10−35\.34\\times 10^\{\-3\}−4\.51×10−3\-4\.51\\times 10^\{\-3\}Gemma\-3\-4B−1\.13×10−2\-1\.13\\times 10^\{\-2\}−4\.74×10−4\-4\.74\\times 10^\{\-4\}Gemma\-3\-13B6\.95×10−36\.95\\times 10^\{\-3\}7\.37×10−37\.37\\times 10^\{\-3\}LLaVA\-NeXT\-7B−3\.19×10−3\-3\.19\\times 10^\{\-3\}−4\.71×10−4\-4\.71\\times 10^\{\-4\}LLaVA\-NeXT\-13B8\.61×10−38\.61\\times 10^\{\-3\}−8\.28×10−3\-8\.28\\times 10^\{\-3\}Table 4:Visual Causal Effect on English Translation Ratio \(Δ​RET\\Delta R\_\{\\text\{ET\}\}\) and English Similarity \(Δ​Simen\\Delta\\text\{Sim\}\_\{\\text\{en\}\}\)\.BackboneSize𝚫\\boldsymbol\{\\Delta\}En𝚫\\boldsymbol\{\\Delta\}SFT Avg\.𝚫\\boldsymbol\{\\Delta\}Zero\-shot Avg\.𝚫\\boldsymbol\{\\Delta\}Avg\.LLaVA\-NeXT7B\+0\.4\+3\.1\+1\.7\+2\.3LLaVA\-NeXT13B\+0\.5\+4\.0\+1\.9\+2\.9InternVL38B\+0\.2\+2\.3\+1\.1\+1\.7InternVL314B\+0\.5\+2\.6\+1\.3\+1\.9Table 5:Generalization to newer MLLM backbones on xMMMU\. Each entry reports the accuracy difference in percentage points between ANCHOR and the corresponding Std\. LoRA baseline \(ANCHOR−Std\. LoRA\\text\{ANCHOR\}\-\\text\{Std\. LoRA\}\)\. SFT Avg\. and Zero\-shot Avg\. are macro\-averages over their respective language groups, while Avg\. is the macro\-average over all seven evaluated language subsets, including English\.BackboneSize𝚫\\boldsymbol\{\\Delta\}En𝚫\\boldsymbol\{\\Delta\}SFT Avg\.𝚫\\boldsymbol\{\\Delta\}Zero\-shot Avg\.𝚫\\boldsymbol\{\\Delta\}Avg\.LLaVA\-NeXT7B\+0\.8\+3\.4\+1\.2\+2\.4LLaVA\-NeXT13B\+1\.0\+3\.8\+1\.3\+2\.7InternVL38B\+0\.5\+2\.6\+0\.8\+1\.8InternVL314B\+0\.6\+3\.0\+0\.9\+2\.1Table 6:Generalization to newer MLLM backbones on MaXM\. Each entry reports the accuracy difference in percentage points between ANCHOR and the corresponding Std\. LoRA baseline \(ANCHOR−Std\. LoRA\\text\{ANCHOR\}\-\\text\{Std\. LoRA\}\)\. SFT Avg\. and Zero\-shot Avg\. are macro\-averages over their respective language groups, while Avg\. is the macro\-average over all seven evaluated language subsets, including English\.BackboneSize𝚫\\boldsymbol\{\\Delta\}SFT Avg\.𝚫\\boldsymbol\{\\Delta\}Zero\-shot Avg\.𝚫\\boldsymbol\{\\Delta\}Avg\.LLaVA\-NeXT7B\+3\.4\+1\.4\+2\.1LLaVA\-NeXT13B\+3\.8\+1\.8\+2\.5InternVL38B\+2\.6\+1\.0\+1\.6InternVL314B\+2\.9\+1\.1\+1\.7Table 7:Generalization to newer MLLM backbones on CVQA\. Each entry reports the accuracy difference in percentage points between ANCHOR and the corresponding Std\. LoRA baseline \(ANCHOR−Std\. LoRA\\text\{ANCHOR\}\-\\text\{Std\. LoRA\}\)\. SFT Avg\. and Zero\-shot Avg\. are macro\-averages over their respective language–region groups, while Avg\. is the macro\-average over all 39 language–region pairs\.
### C\.2Full Results on the CVQA Benchmark

Modelsar\-EGes\-ARes\-CLes\-COes\-ECes\-ESes\-MXes\-UYhi\-INpt\-BRLLaVA\-1\.5\-7B32\.554\.360\.756\.453\.667\.048\.343\.540\.357\.4\+ Std\. LoRA31\.551\.756\.455\.249\.562\.046\.140\.638\.856\.3\+ Early\-Layer FFT29\.641\.544\.946\.542\.350\.343\.737\.837\.345\.8\+ ANCHOR37\.355\.860\.358\.953\.769\.351\.246\.742\.559\.0LLaVA\-1\.5\-13B38\.457\.462\.061\.453\.670\.451\.139\.749\.363\.4\+ Std\. LoRA36\.054\.757\.761\.455\.067\.950\.839\.746\.859\.9\+ Early\-Layer FFT28\.141\.144\.044\.042\.552\.241\.834\.040\.347\.5\+ ANCHOR40\.159\.265\.363\.056\.170\.851\.542\.850\.364\.3Modelsru\-RUsw\-KEzh\-CNzh\-SGam\-ETbg\-BGbn\-INbr\-FRfil\-PHga\-IELLaVA\-1\.5\-7B57\.536\.348\.648\.626\.539\.429\.729\.646\.344\.2\+ Std\. LoRA57\.037\.444\.747\.626\.138\.833\.626\.942\.443\.9\+ Early\-Layer FFT41\.538\.542\.843\.927\.434\.833\.230\.639\.436\.8\+ ANCHOR57\.136\.349\.152\.526\.438\.034\.628\.344\.544\.0LLaVA\-1\.5\-13B57\.042\.954\.352\.825\.244\.734\.635\.848\.846\.6\+ Std\. LoRA53\.541\.850\.552\.826\.142\.332\.932\.147\.842\.0\+ Early\-Layer FFT46\.044\.741\.842\.036\.836\.727\.631\.438\.936\.8\+ ANCHOR57\.446\.059\.456\.026\.042\.835\.734\.749\.148\.5Modelsid\-IDig\-NGja\-JPjv\-IDko\-KRmin\-IDmn\-MNmr\-INms\-MYno\-NOLLaVA\-1\.5\-7B42\.232\.535\.036\.451\.036\.329\.836\.147\.952\.5\+ Std\. LoRA40\.132\.035\.535\.047\.236\.231\.134\.245\.451\.5\+ Early\-Layer FFT35\.727\.535\.531\.043\.531\.132\.132\.735\.942\.8\+ ANCHOR40\.431\.936\.937\.349\.337\.133\.237\.147\.253\.1LLaVA\-1\.5\-13B46\.636\.545\.836\.754\.135\.930\.838\.648\.658\.2\+ Std\. LoRA40\.831\.542\.437\.748\.636\.731\.039\.144\.852\.2\+ Early\-Layer FFT39\.630\.537\.032\.347\.235\.127\.235\.239\.445\.2\+ ANCHOR41\.234\.847\.539\.053\.237\.733\.940\.546\.756\.6Modelsom\-ETro\-ROrw\-RWsi\-LKsu\-IDta\-INte\-INur\-INur\-PKAvg\.LLaVA\-1\.5\-7B37\.451\.034\.524\.437\.531\.834\.027\.724\.141\.6\+ Std\. LoRA35\.546\.734\.923\.635\.026\.227\.029\.525\.940\.0\+ Early\-Layer FFT22\.044\.433\.625\.334\.024\.327\.028\.626\.436\.0\+ ANCHOR34\.937\.435\.324\.136\.429\.033\.232\.326\.642\.0LLaVA\-1\.5\-13B34\.657\.034\.528\.039\.026\.234\.041\.434\.344\.9\+ Std\. LoRA32\.252\.336\.630\.238\.032\.231\.537\.734\.743\.1\+ Early\-Layer FFT30\.444\.727\.226\.734\.022\.930\.030\.531\.037\.0\+ ANCHOR32\.058\.037\.130\.938\.733\.332\.639\.938\.345\.9

Table 8:CVQA benchmark performance\.Pinkcolumns denote language–region pairs whose languages are included in SFT, whileCyancolumns denote languages unseen during training\. TheAvg\.column reports the macro\-average accuracy across all 39 pairs\.
### C\.3Ablation Study

##### Target Layer \(Le​a​r​l​yL\_\{early\}\)

The choice ofLe​a​r​l​yL\_\{early\}is critical in our ANCHOR framework, as it defines the precise early translation window during which cross\-modal alignment must occur\. We conduct an ablation over different values ofLe​a​r​l​yL\_\{early\}to understand its impact on downstream reasoning performance\.

[Table9](https://arxiv.org/html/2608.15085#A3.T9)presents the results of varyingLe​a​r​l​yL\_\{early\}from 6 to 16 layers\. We evaluate on xMMMU and MaXM, reporting the macro\-averaged accuracy across all evaluated languages\.

The results reveal an inverted U\-shaped trend: choosingLe​a​r​l​yL\_\{early\}too small \(e\.g\., 6 layers\) may provide insufficient time for visual semantics to emerge\. Conversely, settingLe​a​r​l​yL\_\{early\}too large \(e\.g\., 16 layers\) may allow the linguistic trajectory to become strongly English\-dominant before effective visual guidance is available\. The best evaluated setting,Le​a​r​l​y=10L\_\{early\}=10, is consistent with our mechanistic analysis of visual semantic emergence and achieves the highest downstream performance among the tested values\. These results suggest that the timing of early\-layer intervention is important for cross\-modal alignment\.

##### Intervention Scope: Early\-Layer vs\. Full SFT

To further validate our layer\-specific design, we compare our early\-layer intervention \(Le​a​r​l​y=10L\_\{early\}=10\) against a full\-parameter fine\-tuning baseline augmented with the PVA objective\. Crucially, in this baseline, the PVA constraint remains applied at layer 10 to ensure a strictly controlled comparison\. As shown in[Table10](https://arxiv.org/html/2608.15085#A3.T10), applying Full SFT degrades the overall accuracy across both benchmarks compared to our targeted early\-layer approach\. This performance gap suggests that full\-depth updates may overfit to the SFT language distribution or interfere with representations that support cross\-lingual generalization\. Confining parameter updates to the early alignment window yields better aggregate performance in this comparison while preserving more of the LLM’s multilingual capability\.

Le​a​r​l​yL\_\{early\}xMMMUMaXM632\.828\.5834\.230\.71035\.632\.41234\.731\.81433\.530\.21632\.128\.9Table 9:Ablation study on the target layerLe​a​r​l​yL\_\{early\}\. xMMMU and MaXM results are macro\-averaged across all evaluated languages\. All experiments are conducted on LLaVA\-1\.5\-7B\.Intervention ScopexMMMUMaXMFull SFT \+ PVA32\.827\.5ANCHOR \(Ours\)35\.632\.4Table 10:Ablation on intervention scope using LLaVA\-1\.5\-7B\. Full\-parameter fine\-tuning augmented with PVA at layer 10 yields lower aggregate performance than the early\-layer intervention on both benchmarks, suggesting that targeted early\-layer updates better preserve cross\-lingual generalization in this setting\.

## Appendix DComplete Implementation Details

##### Model Architecture

The alignment network \(MLPalign\\text\{MLP\}\_\{\\text\{align\}\}\) consists of a two\-layer perceptron with GELU activation\.

##### Training Configuration

We train for 1 epoch on 135K image\-text pairs\. We use the AdamW optimizer with a peak learning rate of2×10−52\\times 10^\{\-5\}and a cosine decay schedule with a 3% warmup period\. The PVA loss weight is set toα=0\.4\\alpha=0\.4\. All experiments are conducted on NVIDIA A100 \(80GB\) GPUs with a global batch size of 128\. Input images are resized to336×336336\\times 336resolution, and the maximum sequence length is 2048 tokens\. During training, we apply full\-parameter fine\-tuning to the multimodal projector and the firstLe​a​r​l​y=10L\_\{early\}=10transformer layers, while freezing layers 11–32 \(for 7B\) or 11–40 \(for 13B\) and the vision encoder\. The trainable parameters thus include the vision\-language projector, the alignment networkMLPalign\\text\{MLP\}\_\{\\text\{align\}\}, and the initial FFT transformer layers\. For the standard LoRA baseline, we employ a rankr=128r=128and a scaling factorαlora=256\\alpha\_\{\\text\{lora\}\}=256, applying the adaptation to all linear layers \(includingq,k,v,oq,k,v,oand MLP gates\) of the LLM backbone to ensure a competitive and fair comparison\.