The Interlingua Hypothesis: LLMs Translate via a Latent Task-agnostic Feature Space

# LLMs Translatevia a Latent Task-agnostic Feature Space
Source: [https://arxiv.org/html/2609.00515](https://arxiv.org/html/2609.00515)
## The Interlingua Hypothesis: LLMs Translate via a Latent Task\-agnostic Feature Space

###### Abstract

Large language models \(LLMs\) have recently demonstrated improved machine translation performance over strong supervised baselines\. This raises questions as to what mechanisms underlie how LLMs perform machine translation between languages\. Motivated by recent interpretability findings—namely, that LLMs use massively multilingual latent feature representations to perform language modeling—we propose the interlingua hypothesis\. The hypothesis holds that language models translate by reading a source sentence into a latent feature space, and generate a target sentence by reading from the latent feature space\. We show three lines of evidence in support of this hypothesis: \(1\) variance in BLEU across language pairs is largely predictable from language\-specific competences with no language pair–specific interaction terms; \(2\) many model components are causally influential in both monolingual tasks and translation tasks; and \(3\) fine\-tuning on monolingual data recovers a large proportion of translation improvements relative to fine\-tuning on aligned documents\. Together, these provide convergent evidence in support of the interlingua hypothesis, and suggest new ways of understanding and improving how LLMs can be leveraged to perform translation tasks\.

## 1Introduction

Machine translation has long been an influential area of natural language processing\. The attention mechanism was first proposed to improve machine translation performance\([Bahdanau et al\., 2015](https://arxiv.org/html/2609.00515#bib.bib1);[Luong et al\., 2015](https://arxiv.org/html/2609.00515#bib.bib2);[Vaswani et al\., 2017](https://arxiv.org/html/2609.00515#bib.bib3)\); one consequence of attention was the emergence of language models that could be efficiently trained on massive corpora\([Devlin et al\., 2019](https://arxiv.org/html/2609.00515#bib.bib23);[Radford et al\., 2018](https://arxiv.org/html/2609.00515#bib.bib24)\)\. Recently, large language models \(LLMs\) have revolutionized many areas of natural language processing, but have been relatively slowly adopted in machine translation \(MT\)\.

A reason for the slow adoption of LLMs in MT has been a focus on low\-resource languages, where LLMs face significant challenges\([Hendy et al\., 2023](https://arxiv.org/html/2609.00515#bib.bib34);[Robinson et al\., 2023](https://arxiv.org/html/2609.00515#bib.bib44)\)\. Because LLMs require large training corpora, they are difficult to train effectively on low\-resource languages\. However, they also hold significant promise: even when not trained on a given language, LLMs can be prompted to translate from or into a language they have seen very little of in their training data\([Cahyawijaya et al\., 2024](https://arxiv.org/html/2609.00515#bib.bib25)\)—e\.g\., by prompting with a grammar book\([Tanzer et al\., 2024](https://arxiv.org/html/2609.00515#bib.bib22)\)\. Moreover, their representations of high\-resource languages could enable cross\-lingual transfer\([Conneau et al\., 2020](https://arxiv.org/html/2609.00515#bib.bib26)\)\.

Whether LLMs can deliver on this promise depends in part on the mechanisms underlying how they translate\. Specifically, do LLMs rely on task\- and language\-agnostic mechanisms to translate, or do they rely on more specialized translation mechanisms or language pair–specific mechanisms? If the former, this suggests clear paths toward improving MT performance, potentially without large parallel corpora\. Much recent work provides mechanistic evidence supporting the existence of massively multilingual representations in language modeling contexts\([Wendler et al\., 2024](https://arxiv.org/html/2609.00515#bib.bib5);[Brinkmann et al\., 2025](https://arxiv.org/html/2609.00515#bib.bib21);[Wu et al\., 2025](https://arxiv.org/html/2609.00515#bib.bib4)\)\.

We therefore hypothesize that LLMs perform MT in large part by reusing computational mechanisms for monolingual language modeling\. The “stages of inference” hypothesis\([Lad et al\., 2024](https://arxiv.org/html/2609.00515#bib.bib45)\)holds that LLMs devote the first half of their layers to reading an input and composing increasingly abstract concept representations\. Then, the latter half of the model reads these concept representations, and uses them to decide which token should be predicted given prior context\. If language models perform translation this way \(what we call theinterlingua hypothesis\), then translation would not necessarily require language pair–specific mechanisms; instead, it only requires that a model be capable of reading abstract features from the source language, and generating text for the target language conditioned on those features\.

If this hypothesis is true, it would entail the following predictions: \(1\) Given language pair\(S,T\)\(S,T\), translation performance should be predictable from monolingual capabilities inSSandTTin isolation without cross\-linguistic interaction terms\. \(2\) There should exist task\-agnostic representations that are causally relevant for predicting correct outputs in both machine translation*and*monolingual task settings\. Finally, \(3\) adding translation capabilities for a new language should primarily require improvements to monolingual capabilities for that language\-\-\-i\.e\., fine\-tuning on monolingual corpora should recover a substantial proportion of the improvement in translation performance as fine\-tuning on parallel corpora, assuming the model can already effectively handle the other language in the language pair\.111Note that our claims relate to the mechanisms a*fully trained*LLM uses to perform translation\. We do not make claims regarding what is contained in the pretraining data; it is possible that parallel pretraining data is necessary for multilingual representations to be learned\.

We investigate each of these three implications, and find positive evidence for each\. While no single experiment definitively confirms the interlingua hypothesis, each provides a different type of evidence in support of it\. These findings could provide a preliminary explanation for the value of exposure to \(largely monolingual\) documents in producing models more effective at translation\.

## 2Related Work

#### Massively multilingual feature representations\.

Our work is partially motivated by the observation that grammatical concept representations are highly multilingual in large language models, even across typologically distinct languages\([Brinkmann et al\., 2025](https://arxiv.org/html/2609.00515#bib.bib21)\)\. Similar evidence has been observed in[Wendler et al\. \(2024\)](https://arxiv.org/html/2609.00515#bib.bib5);[Dumas et al\. \(2025\)](https://arxiv.org/html/2609.00515#bib.bib35)\. Notably, intervening on grammatical concept representations has predictable effects on model behavior in both monolingual and machine translation settings\([Brinkmann et al\., 2025](https://arxiv.org/html/2609.00515#bib.bib21)\)\. These findings do not in themselves confirm the interlingua hypothesis, but they do suggest the existence of abstract feature representations not tied to particular tasks nor languages\.

#### Interlingua in multilingual NMT\.

A line of work in neural machine translation \(NMT\) seeks to engineer interlingua to improve cross\-lingual transfer via architectural and training improvements, largely to encoder\-decoder architectures \(e\.g\., XLM\-R,[Conneau et al\., 2020](https://arxiv.org/html/2609.00515#bib.bib26); M2M,[Fan et al\., 2020](https://arxiv.org/html/2609.00515#bib.bib48)\)\. Prior approaches introduce explicit shared representations via interlingua layers or networks \([Lu et al\., 2018](https://arxiv.org/html/2609.00515#bib.bib49);[Zhu et al\., 2020](https://arxiv.org/html/2609.00515#bib.bib13)\), or bottlenecks that naturally encourage language\-independent representations \([Vázquez et al\., 2019](https://arxiv.org/html/2609.00515#bib.bib20);[Mao et al\., 2023](https://arxiv.org/html/2609.00515#bib.bib19)\)\. These studies stipulate or engineer an interlingua and validate it behaviorally, whereas our study asks whether an interlingua emerges naturally and is causally relevant to MT performance in a decoder\-only LM trained on general language data with no such directly implemented incentives\.

Precedents to our mechanistic investigation include[Vázquez et al\. \(2020\)](https://arxiv.org/html/2609.00515#bib.bib14), who investigate whether sentence representations in NMT models with attention bottlenecks are language\-independent\. Others have investigated whether shared or language\-specific components are needed at all \([Escolano et al\., 2021](https://arxiv.org/html/2609.00515#bib.bib50);[Purason and Tättar, 2022](https://arxiv.org/html/2609.00515#bib.bib51)\); these studies investigate by controlling the degree of parameter sharing across languages, and find that fully shared representations underperformed representations with at least some language\-specific components\. Thus, in smaller\-scale settings using primarily parallel data, interlingua typically need to be imposed via the training objective, and this costs some performance\. We investigate whether this also holds in contemporary decoder\-only models trained on a larger quantity of task\- and domain\-general language data\.

#### LLMs for machine translation\.

Machine translation is difficult when parallel data is limited\([Koehn and Knowles, 2017](https://arxiv.org/html/2609.00515#bib.bib36)\), and large language models do not solve this problem\([Court and Elsner, 2024](https://arxiv.org/html/2609.00515#bib.bib37)\)\. Some hope that LLMs could enable cross\-lingual transfer\. For example,[Tanzer et al\. \(2024\)](https://arxiv.org/html/2609.00515#bib.bib22)showed that long\-context LLMs can translate a previously unseen low\-resource language when prompted with a grammar book containing linguistic descriptions and translation examples\. However,[Aycock et al\. \(2025\)](https://arxiv.org/html/2609.00515#bib.bib18)subsequently found that most of the improvement in this setting came from the book’s parallel examples rather than its grammatical explanations, highlighting the importance of parallel data for translation\. One of our experiments asks a complementary question: once an LLM has already acquired multilingual representations during pretraining, to what extent can improving its monolingual competence in a low\-resource language improve translation without additional parallel data? We provide preliminary evidence for cross\-lingual transfer in §[5](https://arxiv.org/html/2609.00515#S5)\.

#### Causal mediation analysis\.

Some of our evidence relies on estimates of the causal influence\([Lewis, 1973](https://arxiv.org/html/2609.00515#bib.bib41)\)of specific model components\. This relies on causal mediation analysis\([Pearl, 2001](https://arxiv.org/html/2609.00515#bib.bib40);[Vig et al\., 2020](https://arxiv.org/html/2609.00515#bib.bib42)\), a common technique in the mechanistic interpretability literature\([Finlayson et al\., 2021](https://arxiv.org/html/2609.00515#bib.bib7);[Geiger et al\., 2021](https://arxiv.org/html/2609.00515#bib.bib8);[Heimersheim and Nanda, 2024](https://arxiv.org/html/2609.00515#bib.bib12);[Marks et al\., 2025](https://arxiv.org/html/2609.00515#bib.bib39);[Mueller et al\., 2026](https://arxiv.org/html/2609.00515#bib.bib6), e\.g\.,\)\. While the purpose of this study is not to understand the exact functional role of particular model components, we wish to characterize whether the*same*representations influence a model’s behavior in multiple task settings and language pairs\.

## 3Modeling Translation Performance as a Function of Monolingual Capabilities

How much of a model’s ability to translate can be explained by its monolingual language modeling capabilities? If a model uses an interlingua to perform translation, then much of its translation capabilities should be explainable as a function of its ability to parse latent features from an input, and produce coherent text in that language\. In other words, one should not need language pair–specific terms to explain translation performance, unless a model deploys some separate translation mechanism that does not involve going through an interlingua\. To investigate, we define alinear modelthat predicts translation performance from monolingual competencies\.

We use several measures of monolingual competence\. First, to measure a model’s ability to distinguish grammatical from ungrammatical sentences, we use MultiBLiMP\([Jumelet et al\., 2026](https://arxiv.org/html/2609.00515#bib.bib11)\), a multilingual version of the BLiMP\([Warstadt et al\., 2020](https://arxiv.org/html/2609.00515#bib.bib9)\)benchmark\. For a given language, MultiBLiMP accuracy is the fraction of minimal pairs for which the model assigns higher full\-sentence log\-probability to the grammatical completion over its ungrammatical counterpart\. We average this over 200 randomly\-selected samples per language\. Because Llama and Aya saturate this benchmark for many of the languages in our analysis, we also compute the log\-probability*margin*, which is the meanlog⁡p⁡\(grammatical\)−log⁡p⁡\(nongrammatical\)\\log p\(\\text\{grammatical\}\)\-\\log p\(\\text\{nongrammatical\}\)per minimal pair\. To overcome issues related to benchmark saturation, we also include accuracy on GlobalMMLU\([Singh et al\., 2025](https://arxiv.org/html/2609.00515#bib.bib10)\), a multilingual multiple\-choice question answering dataset based on MMLU\([Hendrycks et al\., 2021](https://arxiv.org/html/2609.00515#bib.bib33)\)\. For each task, we filter the set of languages to those in common between the monolingual dataset and FLORES \(Table[1](https://arxiv.org/html/2609.00515#S3.T1)\)\.222The following languages are supported across all datasets we use in this section:ara, ces, deu, ell, eng, fas, fra, heb, hin, ita, nld, pol, por, ron, rus, spa, tur, ukr

For translation, we have language pair\(S,T\)\(S,T\), whereSSis the source andTTis the target language\. Translation competencetS​Tt\_\{ST\}is quantified as a BLEU score for\(S,T\)\(S,T\)\. To perform translation, we use a 2\-shot prompt containing two randomly sampled sentences from FLORES\.333We ensure that the examples do not overlap with the test example\.We usesacrebleu\([Post, 2018](https://arxiv.org/html/2609.00515#bib.bib27)\)to compute BLEUs\.

Letllbe a monolingual competence estimate using MultiBLiMP or GlobalMMLU\. We havelSl\_\{S\}andlTl\_\{T\}for each language pair\. We then learn coefficientsβS\\beta\_\{S\}andβT\\beta\_\{T\}as well as bias termβ0\\beta\_\{0\}to maximize predictive accuracy on BLEU scorestS​Tt\_\{ST\}in the following function:

βS​lS\+βT​lT\+β0=tS​T\\beta\_\{S\}l\_\{S\}\+\\beta\_\{T\}l\_\{T\}\+\\beta\_\{0\}=t\_\{ST\}\(1\)
We also train abilinear modelthat is nearly identical, but also contains a multiplicative interaction term:

βS​lS\+βT​lT\+βS​T​\(lS⋅lT\)\+β0=tS​T\\beta\_\{S\}l\_\{S\}\+\\beta\_\{T\}l\_\{T\}\+\\beta\_\{ST\}\(l\_\{S\}\\cdot l\_\{T\}\)\+\\beta\_\{0\}=t\_\{ST\}\(2\)If the interlingua hypothesis holds, then the bilinear model should not have significantly greater predictive power than the linear model\. This would imply that machine translation performance is better explained by monolingual terms, rather than by the existence of a translation mechanism \(which should use terms specific to particular language*pairs*\)\.

Table 1:Monolingual competence proxies\.NNis the number of the 24 FLORES languages on which the proxy is defined;↑\\uparrow/↓\\downarrowindicates the direction of higher monolingual competence\.#### Monolingual task performance predicts translation competence\.

Fitting the linear model on the common 18\-language subset, monolingual behavioral competence proxies are generally strong predictors of translation performance\. MultiBLiMP grammatical accuracy reachesR2=0\.294R^\{2\}=0\.294/0\.2350\.235for Llama/Aya\. While significant, this is relatively low; we find that this is largely because MultiBLiMP scores saturate at relatively low BLEU scores, such that it is a good predictor, but only up to middling BLEU scores\. In contrast, GlobalMMLU accuracy is a very strong predictor atR2=0\.739R^\{2\}=0\.739/0\.5100\.510for Llama/Aya\. Figure[1](https://arxiv.org/html/2609.00515#S3.F1)plots monolingual competencies against translation performance for each language shared across each evaluation dataset\. Qualitatively, the grammatical and MMLU proxies trend closely with translation quality\.

Table 2:Comparison of the power of monolingual tasks in predicting BLEU scores using the linear \(Rlin2R^\{2\}\_\{\\text\{lin\}\}\) and bilinear \(Rbil2R^\{2\}\_\{\\text\{bil\}\}\) models with several monolingual competence proxies on 18 languages\. GlobalMMLU is the strongest predictor\. The bilinear interaction term does not add significant predictive power over the linear model\.
#### Adding language pair interaction terms does not increase predictive power\.

Across each proxy and both models, the bilinear model has virtually the same predictive power as the linear model \(Table[2](https://arxiv.org/html/2609.00515#S3.T2); see also App\.[A](https://arxiv.org/html/2609.00515#A1)and App\.[A\.1](https://arxiv.org/html/2609.00515#A1.SS1)\)\.

The same trend holds when we predict BLEU scores from the language\-specific marginal BLEU scores, computed by averaging BLEU scores across all language pairs for a given source or target language\. Decomposing the full BLEU matrix into per\-language source and target main effects,BLEUS​T≈μ\+αS\+βT\\text\{BLEU\}\_\{ST\}\\approx\\mu\+\\alpha\_\{S\}\+\\beta\_\{T\}, explainsR2=0\.932R^\{2\}=0\.932\(Llama\) /0\.8790\.879\(Aya\) of the centered variance, and a rank\-1 multiplicative reconstruction recovers≈\\approx90%90\\%/83%83\\%of the matrix energy \(App\.[A\.2](https://arxiv.org/html/2609.00515#A1.SS2)\)\. Hence, pairwise translation quality is largely captured by per\-language competence, with pairwise interactions adding little predictive power over the simpler model\.

![Refer to caption](https://arxiv.org/html/2609.00515v1/figures/fig_proxy_marginal_all.png)Figure 1:Per\-language relationship between two monolingual proxies and the BLEU score for a given target language\. Both monolingual terms correlate significantly with BLEU scores, but GlobalMMLU accuracies are a stronger predictor than MultiBLiMP accuracies\.
#### Target language competence matters more than source language competence\.

Source and target competence contribute asymmetrically: changing the target language moves BLEU far more than changing the source\. The variance across target languages of mean BLEU exceeds the variance across source languages by9\.3×9\.3\\timesfor Llama and3\.9×3\.9\\timesfor Aya, and the target coefficient is1\.71\.7–5×5\\timesthe source coefficient \(MMLU target/source=2\.98=2\.98for Llama,1\.671\.67for Aya; MultiBLiMP\-margin4\.884\.88/2\.302\.30\)\. This may be because it is easier for models to extract meaning from the source language than produce fluent outputs in the target language; alternatively, it may be an artifact of relying on an n\-gram matching metric such as BLEU score\.

## 4Many Translation\-relevant Components Also Perform Monolingual Tasks

Our previous results show that monolingual capabilities are predictors of translation performance, but also that language pair interactions are not strong predictors\. This provides correlational evidence in support of our hypothesis, but not causal evidence\. To obtain causal evidence, we now perform an analysis based on causal mediation analysis\([Pearl, 2001](https://arxiv.org/html/2609.00515#bib.bib40);[Vig et al\., 2020](https://arxiv.org/html/2609.00515#bib.bib42)\)\.

Past work has found that language models use the first half of their layers to form progressively more abstract representations of the latent features in a given input\([Lad et al\., 2024](https://arxiv.org/html/2609.00515#bib.bib45)\)\. We hypothesize that language models could repurpose this language modeling machinery to perform translation by reading the source language into a latent feature space, and then using the latter half of its layers to read from this feature space and produce the translation in the output language\. If this is true, then we should be able to find model representations or components that are causally influential in both machine translation and language modeling settings\. To obtain causal evidence in support of this view, we now perform mechanistic experiments by patching model components\. We show that the most influential attention heads for producing correct translations also strongly mediate a language model’s ability to produce grammatical outputs in monolingual settings\.

### 4\.1Using GCM to Identify Translation\-relevant Components

![Refer to caption](https://arxiv.org/html/2609.00515v1/figures/promptswap_decomposition_heads.png)Figure 2:Llama\-3\.1\-8B mean\|IE^\|\|\\widehat\{\\mathrm\{IE\}\}\|for top attention heads under translation and control prompts, averaged over 56 translation directions\. The leading heads have much larger effects in the translation setting than in the controls, indicating that these heads are selective for matching translations and do not directly increase the probability of the given target sentence\.We first search for model components that are causally relevant in promoting the valid translation over an invalid translation\. We do so using generative causal mediation \(GCM;[Sankaranarayanan et al\., 2026](https://arxiv.org/html/2609.00515#bib.bib43)\)\. Intuitively, GCM measures how intervening on the activation of a model component \(e\.g\., an attention head\) influences the model’s relative preference for one continuation over another\.

We measure the model’s preference for a single gold translation by swapping the source sentence that the model reads\. Letrrbe a gold translation of a source sentence, held fixed throughout\. Letporigp\_\{\\text\{orig\}\}be a prompt whose final source sentence is the one thatrractually translates, and letpcfp\_\{\\text\{cf\}\}be the same prompt with a distinct final source sentence\. We define the preference metricMM:

M=log⁡π⁡\(r∣porig\)−log⁡π⁡\(r∣pcf\),M=\\log\\pi\(r\\mid p\_\{\\text\{orig\}\}\)\-\\log\\pi\(r\\mid p\_\{\\text\{cf\}\}\),\(3\)wherelog⁡π⁡\(r∣p\)=∑tlog⁡p⁡\(rt∣p,r<t\)\\log\\pi\(r\\mid p\)=\\sum\_\{t\}\\log p\(r\_\{t\}\\mid p,r\_\{<t\}\)is the teacher\-forced log\-probability of a continuation, summed over its tokens\.MMis the degree to which the model prefers to generate the gold translationrrwhen it has read the matching source, rather than a mismatched one; it isolates how much of the production ofrrdepends on having read the source content\.

We wish to know how strongly each attention head influences this preference\. To measure this, we take the activation of an attention head; we call thiszz, and letzorigz\_\{\\text\{orig\}\}andzcfz\_\{\\text\{cf\}\}be the values it takes when the model readsporigp\_\{\\text\{orig\}\}andpcfp\_\{\\text\{cf\}\}respectively\. LetIE​\(z\)\\text\{IE\}\(z\)be the causal contribution ofzztoMM, measured as the change inMMwhen we move the head from its mismatched\-source statezcfz\_\{\\text\{cf\}\}to its matched\-source statezorigz\_\{\\text\{orig\}\}while holding the rest of the computation fixed\. We source the counterfactual activationzcfz\_\{\\text\{cf\}\}by running a forward pass onpcfp\_\{\\text\{cf\}\}and cachingzzat the final source\-token position, then patch it into the corresponding position of the run we score:

IE⁡\(z\)=M⁡\(zorig\)−M⁡\(zcf\)\.\\mathrm\{IE\}\(z\)=M\(z\_\{\\text\{orig\}\}\)\-M\(z\_\{\\text\{cf\}\}\)\.\(4\)
Patching a single head and rerunning the model tells us that head’s exact contribution, but computingIE​\(z\)\\text\{IE\}\(z\)this way for everyzzis intractable: it would requireO⁡\(Z⋅n\)O\(Z\\cdot n\)forward passes, whereZZis the number of mediators andnnthe number of examples\. We instead use*attribution patching*\([Syed et al\., 2023](https://arxiv.org/html/2609.00515#bib.bib38)\), a first\-order linear approximation of the IE based on gradient attributions\([Simonyan et al\., 2013](https://arxiv.org/html/2609.00515#bib.bib32)\):

IE^​\(z\)=∇zM\|z=zorig⋅\(zorig−zcf\)\.\\widehat\{\\mathrm\{IE\}\}\(z\)=\\nabla\_\{z\}M\\big\|\_\{z=z\_\{\\text\{orig\}\}\}\\cdot\(z\_\{\\text\{orig\}\}\-z\_\{\\text\{cf\}\}\)\.\(5\)The gradient∇zM\\nabla\_\{z\}Mfor each component can be computed in a single backward pass, and the deltas\(zorig−zcf\)\(z\_\{\\text\{orig\}\}\-z\_\{\\text\{cf\}\}\)for each component can be computed in two forward passes by caching each component’s activation on the two source prompts\.

The magnitude of the indirect effectIE^\\widehat\{\\mathrm\{IE\}\}is how much of a model’s output behavior \(as quantified byMM\) flows through a component when all else is kept constant\. BecauseMMrewards producingrrunder the correct source, components with a positive indirect effect are those that cause the probability of the correct translationrorigr\_\{\\text\{orig\}\}to increase relative to the counterfactual translationrcfr\_\{\\text\{cf\}\}, and those with a negative indirect effect cause the probability of the correct translation to*decrease*relative to the counterfactual translation\.

We instantiaterras a gold translation from FLORES\([Goyal et al\., 2022](https://arxiv.org/html/2609.00515#bib.bib31)\), a massively parallel dataset that supports 101 languages\. Given a source and target language, we construct a 2\-shot prompt as follows:

> \{Source\}: \{shot 1 source\} \{Target\}: \{shot 1 target\} \{Source\}: \{shot 2 source\} \{Target\}: \{shot 2 target\} \{Source\}: \{query source\} \{Target\}:

and patch at the last source\-token position \(the final": "\)\. For each of the 56 ordered pairs where the source and target can be any language in \{English, Spanish, German, French, Turkish, Arabic, Hindi, Hebrew\} \(excluding pairs where the source and target are the same\), we compute theIE^\\widehat\{\\mathrm\{IE\}\}for all attention heads overn=100n=100uniformly sampled pairs\.

Naïvely, we may find heads that respond at least in part to variations in the source samples that are unrelated to the translation task\. To verify that the components we find are selective for correct source–target pairings, we compare indirect effects with three controls\. First, thesame\-language controlrefers to cases where the source language is the same as the target, such that the correct “translation” is the same sentence copied from the source, and the counterfactual translation is a randomly sampled sentence in the same language as the source\. Higher indirect effects for the translation setting than the same\-language control indicate that the component specifically causes correct translations to be more probable, and not copies of the source sequence\. Thenull cross\-language controlrefers to a setup similar to the machine translation setup, but where*neither*the original nor counterfactual completion are the correct translation\. We expect indirect effects here to be very small relative to the translation task\. Finally, we have thenull same\-language control, where both the original and counterfactual completions are in the same language as the source, but neither are the same as the source sentence\.

The heads we find have larger effects in the translation setting by far than in any control setting \(Figure[2](https://arxiv.org/html/2609.00515#S4.F2)\)\. The mean\|IE^\|\|\\widehat\{\\mathrm\{IE\}\}\|in the translation task exceeds that of the null cross\-language control by3\.0×3\.0\\timesover all heads, and5\.2×5\.2\\timesover the top\-10 heads for Llama \(and2\.5×2\.5\\timesfor all heads/4\.3×4\.3\\timesfor the top\-10 heads for Aya\)\. We observe a similar magnitude of increase relative to both same\-language controls\. This suggests that the heads we have found are responsible for performing machine translation, and that their effects cannot be explained by their general utility in generating fluent target sequences regardless of the source\.

The translation\-specific heads are localized to layers 13–14 in Llama and 15–20 in Aya\. With respect to the stages of inference hypothesis\([Lad et al\., 2024](https://arxiv.org/html/2609.00515#bib.bib45)\), this would correspond to the layers that refine concept representations into increasingly abstract representations\.

### 4\.2Translation Heads Have Similar Effects Across Language Pairs

An interlingua should have multilingual feature representations; this would allow a model to reuse the same representations when reading a source sequence and generating the target sequence\. Prior work has established the existence of massively multilingual representations\([Wendler et al\., 2024](https://arxiv.org/html/2609.00515#bib.bib5)\), including grammatical concept representations\([Brinkmann et al\., 2025](https://arxiv.org/html/2609.00515#bib.bib21)\); here, we confirm these findings for the model we study by investigating their indirect effects across language pairs\.

We follow the procedure of §[4\.1](https://arxiv.org/html/2609.00515#S4.SS1)to computeIE^\\widehat\{\\mathrm\{IE\}\}for each language pair in the translation test dataset\. We show the indirect effects for the top heads by absoluteIE^\\widehat\{\\mathrm\{IE\}\}across language pairs; if there is feature reuse across pairs, then the sign of the effect should be the same for many language pairs\. Note that the magnitude ofIE^\\widehat\{\\mathrm\{IE\}\}is not directly comparable across languages, as the initial probability of the target sequence and its translation capabilities are language pair–dependent\.

![Refer to caption](https://arxiv.org/html/2609.00515v1/figures/promptswap_llama_universal_heads_selected_directions.png)Figure 3:Signed mean IE for Llama\-3\.1\-8B on machine translation for selected language pairs\. Each column is 1 of the 15 top attention heads byIE^\\widehat\{\\mathrm\{IE\}\}across all languages\. The top axis shows for how many language pairs the head was in the top\-20 set\. Several heads have stable effect signs across language pairs, which suggests that the same heads are being reused across many language pairs\.The top attention heads byIE^\\widehat\{\\mathrm\{IE\}\}are largely shared across language pairs\. Figure[3](https://arxiv.org/html/2609.00515#S4.F3)shows the 15 heads that appear most often in a single direction’s top\-20 by\|IE^\|\|\\widehat\{\\mathrm\{IE\}\}\|: the most universal of these are in the top\-20 for all 56 directions\. The sign of their effect is also stable—a given head keeps the same sign \(favoring either correct or incorrect translations\) across virtually every direction, regardless of source and target languages \(For the full results from both models, see Figures[10](https://arxiv.org/html/2609.00515#A2.F10)and[11](https://arxiv.org/html/2609.00515#A2.F11)\)\. As predicted by our hypothesis, many of the translation mechanisms employed by the model do not depend on the choice of language pair; in fact, most of the top heads have the same directionality and general magnitude of effect on model performance for*all*language pairs\.

### 4\.3Ablating Translation Heads Degrades Performance on Translation and Monolingual Tasks

Another mechanistic prediction of our hypothesis is that the heads most responsible for performing machine translation should reuse computational machinery used for monolingual tasks\. To test this, we first verify that ablating the top heads by indirect effect harms machine translation performance\. Then, we show that ablating the same heads also harms performance on acceptability judgments and multiple\-choice question answering, suggesting that these heads are not selective for translation; rather, they may be reusing computational machinery from more general language modeling mechanisms\.

![Refer to caption](https://arxiv.org/html/2609.00515v1/figures/interventions_mmlu_bleu.png)Figure 4:Ablation experiments overview\. Ablating the top heads by causal influence on MT \(POS\-10, in red\) performance drives a significant decrease in BLEU, and more than ablating random heads \(Control, in grey\)\. For GlobalMMLU, ablating those top heads reduces the model’s ability to distinguish between the correct and incorrect answer more than ablating random heads does\. The Control involves ablating the same number of heads in the same layers\.For each target languageTT, we aggregate signed IE over the 7 language pairs in whichTTis the target, then select the top 10 positive\-IE heads \(POS\-10, the correct translation–favoring set\) by signed magnitude\. We mean\-ablate a head by replacing its output with its average activation over the translation prompts at all token positions\. As a control, we ablate the same number of randomly sampled heads from the same layers\. We then regenerate FLORES translations and measure the change in BLEU relative to the original model before ablations\.

Ablating these heads causes significant reductions in BLEU\. Ablating POS\-10 degrades BLEU in all 8 target languages for both models \(Figure[4](https://arxiv.org/html/2609.00515#S4.F4), bottom\) by roughly twice the amount as the random control\. This suggests that the heads we have found are causally relevant to translation performance\.

Table 3:Monolingual fine\-tuning recovers most of the Xhosa translation gains obtained with parallel fine\-tuning\. Both conditions also retain most performance on French and German translation \(despite their not appearing in the fine\-tuning data\), although monolingual fine\-tuning retrains slightly more performance\.Having shown the importance of these heads for translation quality, we now ask whether these same heads are responsible for monolingual capabilities\. For this, we again use the GlobalMMLU dataset\. We uniformly sample 400 examples per language\. Given promptppwith correct token completionrcorrectr\_\{\\text\{correct\}\}and a randomly chosen incorrect completionrincorrectr\_\{\\text\{incorrect\}\}, each minimal pair is scored by the acceptability margin

Δ=log⁡p⁡\(rcorrect∣p\)−log⁡p⁡\(rincorrect∣p\)\\Delta=\\log p\(r\_\{\\text\{correct\}\}\\mid p\)\-\\log p\(r\_\{\\text\{incorrect\}\}\\mid p\)\(6\)We report the*change in*Δ\\Deltaafter ablating the same heads ablated in the translation experiments:

Δchange=Δablation−Δoriginal,\\Delta\_\{\\text\{change\}\}=\\Delta\_\{\\text\{ablation\}\}\-\\Delta\_\{\\text\{original\}\},\(7\)where a negativeΔchange\\Delta\_\{\\text\{change\}\}means the ablation weakened the model’s ability to distinguish the correct from the incorrect answer\.

Ablating the POS\-10 heads from the MT task results in a negativeΔchange\\Delta\_\{\\text\{change\}\}, whereas ablating random heads has a smaller effect \(Figure[4](https://arxiv.org/html/2609.00515#S4.F4), top\)\. That the same heads affect performance in both tasks provides preliminary evidence that these heads implement computations that are reused across tasks\. Thus, these heads appear to implement computations that are not selective for translation alone\. This provides further support for our hypothesis\.444That said, applying the same ablations in MultiBLiMP results in a smaller effect; see Fig\.[14](https://arxiv.org/html/2609.00515#A2.F14)in App\.[B](https://arxiv.org/html/2609.00515#A2)\. Thus, these heads are not completely task\-agnostic\.

## 5Monolingual Fine\-tuning Recovers Most Gains from Parallel Data

If LLMs translate via task\-agnostic internal representations, translation quality should depend in part on the model’s monolingual competence in the source and target languages\. For example, if we wish to translate between a low\-resource and high\-resource language, then we should be able to improve translation performance given access only to data that improves language modeling quality on the low\-resource language, assuming that we preserve capabilities in the high\-resource language\. Under this view, low translation performance may reflect a failure to encode the source sentence into an adequate internal representation, or to decode the target sentence fluently from it\. Recent work has shown that bilingual or mixed\-language signals during pretraining can be important for acquiring translation capabilities in LLMs\([Briakou et al\., 2023](https://arxiv.org/html/2609.00515#bib.bib15);[Qorib et al\., 2025](https://arxiv.org/html/2609.00515#bib.bib16);[Shao et al\., 2026](https://arxiv.org/html/2609.00515#bib.bib17)\)\. We ask the complementary question of whether, given a fully pretrained model, improvements in monolingual language modeling can \(at least in part\) transfer to translation even without parallel data\.

We test this idea using Llama\-3\.1\-8B and TinyAya\-3B\.555We use TinyAya instead of Aya\-23\-8B because Aya\-23\-8B only provides an instruction\-tuned model, and no base model\. In pilot experiments, we found it difficult to achieve good performance after fine\-tuning in any language \(including high\-resource languages\) with instruction\-tuned models\.For each model, we compare how fine\-tuning on an underrepresented language \(Xhosa\) affects translation performance when the training data consist of either parallel data or monolingual text\. Specifically, in the monolingual setting, we train on unaligned text consisting of 80% Xhosa and 20% English, French, and German data to avoid catastrophic forgetting of the model’s existing language abilities\. In the parallel setting, we use paired Xhosa and English sentences from OPUS MT560\([Gowda et al\., 2021](https://arxiv.org/html/2609.00515#bib.bib29)\), presented in both translation directions\.

Both settings use a next\-token prediction loss and rank\-16 LoRA adapters\([Hu et al\., 2022](https://arxiv.org/html/2609.00515#bib.bib30)\)\. The reported runs use a learning rate of3×10−53\\times 10^\{\-5\}, a maximum sequence length of 512 tokens, and one epoch over 100 million tokens\. We evaluate translation with few\-shot prompting on the FLORES devtest split\([Goyal et al\., 2022](https://arxiv.org/html/2609.00515#bib.bib31)\), using examples from the dev split as in\-context demonstrations\. Thus, the models are prompted to translate at evaluation time even though the monolingual condition contains no translation examples during training\.

Table[3](https://arxiv.org/html/2609.00515#S4.T3)shows that monolingual fine\-tuning recovers most of the improvement obtained with parallel data\. For Llama, monolingual fine\-tuning raises BLEU from 16\.84 to 24\.81, compared with 24\.88 under parallel fine\-tuning\. This recovers 99% of the improvement over the base model\. For Aya, monolingual fine\-tuning raises BLEU from 23\.46 to 26\.64, compared with 27\.80 under parallel fine\-tuning, recovering 73% of the improvement\. For both models, both fine\-tuning conditions largely preserve translation performance from French and German into English, although monolingual fine\-tuning remains closer to the base performance\.

These results are consistent with the view that once an LLM has been pretrained and acquired multilingual representations, improving its ability to model an underrepresented language can produce most of the available translation gains\. Parallel data may still perform better because it can improve language modeling while also strengthening translation\-specific mechanisms\. Our results therefore do not imply that parallel data are unimportant, particularly during pretraining\. They show that, after pretraining, a large share of the gains from parallel fine\-tuning can be achieved using only unaligned data\.

We provide results for additional languages \(German and Thai\) and translation directions in Appendix[C](https://arxiv.org/html/2609.00515#A3)\. In short, changes in performance are smaller for higher\-resource languages, but parallel and monolingual fine\-tuning still achieve largely comparable results\.

## 6Discussion and Conclusions

We have provided evidence that machine translation capabilities in LLMs are mediated in significant part by multilingual representations that are also relevant for some monolingual tasks\. Across three complementary analyses, we find evidence consistent with this view: Translation performance is largely explained by source\- and target\-language competence, translation\-relevant components overlap with components used for monolingual grammatical behavior, and monolingual Xhosa fine\-tuning recovers most of the gains from parallel fine\-tuning across both models\. Together, these results provide support for the hypothesis that LLMs translate by mapping source\-language input into a latent feature space \(that can also in theory be used for monolingual next\-token predictions\), and then generating target\-language text from that shared representation\.

This interpretation helps explain why monolingual competence is strongly associated with translation performance\. If a model translates through a partially language\-agnostic feature space, then improving its ability to read or write a language should improve translation involving that language\. The observation that target\-language competence explains more variance than source\-language competence may suggest that fluent and grammatical generation is often the bottleneck\. Under this view, poor translation into a language need not imply the absence of a dedicated translation circuit for that language pair; instead, it may reflect weak target\-language decoding capabilities given otherwise usable latent representations of the source sentence\.

We emphasize that translation\-specific mechanisms not based on interlingua are also likely to exist; indeed,[Todd et al\. \(2024\)](https://arxiv.org/html/2609.00515#bib.bib46)find that there are components selective for word translation\. We do not claim that an interlingua is the only means by which LLMs translate, but rather, that it is a significant mechanism that controls a large proportion of model performance on MT tasks\. It is likely that LLMs use a mixture of mechanisms\([Gur\-Arieh et al\., 2026](https://arxiv.org/html/2609.00515#bib.bib28)\)to achieve their machine translation capabilities\.

## Limitations

The interlingua hypothesis is one plausible explanation for what we have observed, but our experiments do not rule out the existence of other mechanisms\. The finding that the target language yields more predictive power than the source language in predicting BLEU score may be an artifact of thenn\-gram\-based computation of BLEU scores\. It is also possible that the features underlying high\-quality translations are not well represented in the BLEU score\.

Our experiments were limited to two 8B\-parameter models and one 3B\-parameter model, and may not generalize to smaller or larger LLMs\. Future work should investigate whether similar trends hold for a wider variety of language models, and whether recent developments in thinking models affect these findings\.

For our GCM experiments, we focus on components that mediate at the last token, missing computations that happen at earlier positions\.

Finally, our translation experiments are limited to a relatively small number of language pairs\. Future work could scale up this experiment to investigate whether these trends hold across a large number of language pairs, and to what degree monolingual fine\-tuning can recover the performance of parallel fine\-tuning across many language pairs \(and what other factors explain when this works well and when it does not\)\.

## Acknowledgments

We are grateful to the members of the BAAIGL lab at Boston University for helpful comments on an earlier iteration of this work\. The computational work reported on in this paper was performed largely on the Shared Computing Cluster, which is administered by Boston University’s Research Computing Services\. Jannik Brinkmann is supported by the German Federal Ministry for Economic Affairs and the German Federal Ministry of Research, Technology and Space\.

## References

- S\. Aycock, D\. Stap, D\. Wu, C\. Monz, and K\. Sima’anCan LLMs really learn to translate a low\-resource language from one grammar book?\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=aMBSY2ebPw)Cited by:[§2](https://arxiv.org/html/2609.00515#S2.SS0.SSS0.Px3.p1.1)\.
- Bahdanauet al\.\(2015\)D\. Bahdanau, K\. Cho, and Y\. BengioNeural machine translation by jointly learning to align and translate\.Cited by:[§1](https://arxiv.org/html/2609.00515#S1.p1.1)\.
- Briakouet al\.\(2023\)E\. Briakou, C\. Cherry, and G\. FosterSearching for needles in a haystack: on the role of incidental bilingualism in PaLM’s translation capability\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 9432–9452\.External Links:[Link](https://aclanthology.org/2023.acl-long.524/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.524)Cited by:[§5](https://arxiv.org/html/2609.00515#S5.p1.1)\.
- Brinkmannet al\.\(2025\)J\. Brinkmann, C\. Wendler, C\. Bartelt, and A\. MuellerLarge language models share representations of latent grammatical concepts across typologically diverse languages\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 6131–6150\.External Links:[Link](https://aclanthology.org/2025.naacl-long.312/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.312),ISBN 979\-8\-89176\-189\-6Cited by:[§1](https://arxiv.org/html/2609.00515#S1.p3.1),[§2](https://arxiv.org/html/2609.00515#S2.SS0.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2609.00515#S4.SS2.p1.1)\.
- Cahyawijayaet al\.\(2024\)S\. Cahyawijaya, H\. Lovenia, and P\. FungLLMs are few\-shot in\-context low\-resource language learners\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 405–433\.External Links:[Link](https://aclanthology.org/2024.naacl-long.24/),[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.24)Cited by:[§1](https://arxiv.org/html/2609.00515#S1.p2.1)\.
- Conneauet al\.\(2020\)A\. Conneau, K\. Khandelwal, N\. Goyal, V\. Chaudhary, G\. Wenzek, F\. Guzmán, E\. Grave, M\. Ott, L\. Zettlemoyer, and V\. StoyanovUnsupervised cross\-lingual representation learning at scale\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 8440–8451\.External Links:[Link](https://aclanthology.org/2020.acl-main.747/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.747)Cited by:[§1](https://arxiv.org/html/2609.00515#S1.p2.1),[§2](https://arxiv.org/html/2609.00515#S2.SS0.SSS0.Px2.p1.1)\.
- Court and Elsner \(2024\)S\. Court and M\. ElsnerShortcomings of LLMs for low\-resource translation: retrieval and understanding are both the problem\.InProceedings of the Ninth Conference on Machine Translation,B\. Haddow, T\. Kocmi, P\. Koehn, and C\. Monz \(Eds\.\),Miami, Florida, USA,pp\. 1332–1354\.External Links:[Link](https://aclanthology.org/2024.wmt-1.125/),[Document](https://dx.doi.org/10.18653/v1/2024.wmt-1.125)Cited by:[§2](https://arxiv.org/html/2609.00515#S2.SS0.SSS0.Px3.p1.1)\.
- Devlinet al\.\(2019\)J\. Devlin, M\. Chang, K\. Lee, and K\. ToutanovaBERT: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Minneapolis, Minnesota,pp\. 4171–4186\.External Links:[Link](https://aclanthology.org/N19-1423/),[Document](https://dx.doi.org/10.18653/v1/N19-1423)Cited by:[§1](https://arxiv.org/html/2609.00515#S1.p1.1)\.
- Dumaset al\.\(2025\)C\. Dumas, C\. Wendler, V\. Veselovsky, G\. Monea, and R\. WestSeparating tongue from thought: activation patching reveals language\-agnostic concept representations in transformers\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 31822–31841\.External Links:[Link](https://aclanthology.org/2025.acl-long.1536/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1536),ISBN 979\-8\-89176\-251\-0Cited by:[§2](https://arxiv.org/html/2609.00515#S2.SS0.SSS0.Px1.p1.1)\.
- Escolanoet al\.\(2021\)C\. Escolano, M\. R\. Costa\-jussà, J\. A\. R\. Fonollosa, and M\. ArtetxeMultilingual machine translation: closing the gap between shared and language\-specific encoder\-decoders\.InProceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics,pp\. 944–948\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.eacl-main.80),[Link](https://aclanthology.org/2021.eacl-main.80/)Cited by:[§2](https://arxiv.org/html/2609.00515#S2.SS0.SSS0.Px2.p2.1)\.
- Fanet al\.\(2020\)A\. Fan, S\. Bhosale, H\. Schwenk, Z\. Ma, A\. El\-Kishky, S\. Goyal, M\. Baines, O\. Celebi, G\. Wenzek, V\. Chaudhary, N\. Goyal, T\. Birch, V\. Liptchinsky, S\. Edunov, E\. Grave, M\. Auli, and A\. JoulinBeyond english\-centric multilingual machine translation\.External Links:2010\.11125,[Document](https://dx.doi.org/10.48550/arXiv.2010.11125),[Link](https://arxiv.org/abs/2010.11125)Cited by:[§2](https://arxiv.org/html/2609.00515#S2.SS0.SSS0.Px2.p1.1)\.
- Finlaysonet al\.\(2021\)M\. Finlayson, A\. Mueller, S\. Gehrmann, S\. Shieber, T\. Linzen, and Y\. BelinkovCausal analysis of syntactic agreement mechanisms in neural language models\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),C\. Zong, F\. Xia, W\. Li, and R\. Navigli \(Eds\.\),Online,pp\. 1828–1843\.External Links:[Link](https://aclanthology.org/2021.acl-long.144/),[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.144)Cited by:[§2](https://arxiv.org/html/2609.00515#S2.SS0.SSS0.Px4.p1.1)\.
- Fiotto\-Kaufmanet al\.\(2025\)J\. F\. Fiotto\-Kaufman, A\. R\. Loftus, E\. Todd, J\. Brinkmann, K\. Pal, D\. Troitskii, M\. Ripa, A\. Belfki, C\. Rager, C\. Juang, A\. Mueller, S\. Marks, A\. S\. Sharma, F\. Lucchetti, N\. Prakash, C\. E\. Brodley, A\. Guha, J\. Bell, B\. C\. Wallace, and D\. BauNNsight and NDIF: democratizing access to open\-weight foundation model internals\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=MxbEiFRf39)Cited by:[Appendix E](https://arxiv.org/html/2609.00515#A5.p3.1)\.
- Geigeret al\.\(2021\)A\. Geiger, H\. Lu, T\. Icard, and C\. PottsCausal abstractions of neural networks\.InAdvances in Neural Information Processing Systems,M\. Ranzato, A\. Beygelzimer, Y\. Dauphin, P\.S\. Liang, and J\. W\. Vaughan \(Eds\.\),Vol\.34,pp\. 9574–9586\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2021/file/4f5c422f4d49a5a807eda27434231040-Paper.pdf)Cited by:[§2](https://arxiv.org/html/2609.00515#S2.SS0.SSS0.Px4.p1.1)\.
- Gowdaet al\.\(2021\)T\. Gowda, Z\. Zhang, C\. Mattmann, and J\. MayMany\-to\-english machine translation tools, data, and pretrained models\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations,pp\. 306–316\.External Links:[Link](http://dx.doi.org/10.18653/v1/2021.acl-demo.37),[Document](https://dx.doi.org/10.18653/v1/2021.acl-demo.37)Cited by:[Appendix D](https://arxiv.org/html/2609.00515#A4.p1.1),[§5](https://arxiv.org/html/2609.00515#S5.p2.1)\.
- Goyalet al\.\(2022\)N\. Goyal, C\. Gao, V\. Chaudhary, P\. Chen, G\. Wenzek, D\. Ju, S\. Krishnan, M\. Ranzato, F\. Guzmán, and A\. FanThe Flores\-101 evaluation benchmark for low\-resource and multilingual machine translation\.Transactions of the Association for Computational Linguistics10,pp\. 522–538\.External Links:[Link](https://aclanthology.org/2022.tacl-1.30/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00474)Cited by:[Appendix D](https://arxiv.org/html/2609.00515#A4.p1.1),[§4\.1](https://arxiv.org/html/2609.00515#S4.SS1.p6.1),[§5](https://arxiv.org/html/2609.00515#S5.p3.1)\.
- Gur\-Ariehet al\.\(2026\)Y\. Gur\-Arieh, M\. Geva, and A\. GeigerMixing mechanisms: how language models retrieve bound entities in\-context\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=UJ2UUjT2ko)Cited by:[§6](https://arxiv.org/html/2609.00515#S6.p3.1)\.
- Heimersheim and Nanda \(2024\)S\. Heimersheim and N\. NandaHow to use and interpret activation patching\.External Links:2404\.15255,[Link](https://arxiv.org/abs/2404.15255)Cited by:[§2](https://arxiv.org/html/2609.00515#S2.SS0.SSS0.Px4.p1.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. SteinhardtMeasuring massive multitask language understanding\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=d7KBjmI3GmQ)Cited by:[§3](https://arxiv.org/html/2609.00515#S3.p2.1)\.
- Hendyet al\.\(2023\)A\. Hendy, M\. Abdelrehim, A\. Sharaf, V\. Raunak, M\. Gabr, H\. Matsushita, Y\. J\. Kim, M\. Afify, and H\. H\. AwadallaHow good are gpt models at machine translation? a comprehensive evaluation\.External Links:2302\.09210,[Link](https://arxiv.org/abs/2302.09210)Cited by:[§1](https://arxiv.org/html/2609.00515#S1.p2.1)\.
- Huet al\.\(2022\)E\. J\. Hu, yelong shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[§5](https://arxiv.org/html/2609.00515#S5.p3.1)\.
- Jumeletet al\.\(2026\)J\. Jumelet, L\. Weissweiler, J\. Nivre, and A\. BisazzaMultiBLiMP 1\.0: a massively multilingual benchmark of linguistic minimal pairs\.Transactions of the Association for Computational Linguistics14,pp\. 193–216\.External Links:[Link](https://aclanthology.org/2026.tacl-1.10/),[Document](https://dx.doi.org/10.1162/tacl.a.600)Cited by:[Appendix D](https://arxiv.org/html/2609.00515#A4.p1.1),[§3](https://arxiv.org/html/2609.00515#S3.p2.1)\.
- Koehn and Knowles \(2017\)P\. Koehn and R\. KnowlesSix challenges for neural machine translation\.InProceedings of the First Workshop on Neural Machine Translation,T\. Luong, A\. Birch, G\. Neubig, and A\. Finch \(Eds\.\),Vancouver,pp\. 28–39\.External Links:[Link](https://aclanthology.org/W17-3204/),[Document](https://dx.doi.org/10.18653/v1/W17-3204)Cited by:[§2](https://arxiv.org/html/2609.00515#S2.SS0.SSS0.Px3.p1.1)\.
- Ladet al\.\(2024\)V\. Lad, W\. Gurnee, and M\. TegmarkThe remarkable robustness of LLMs: stages of inference?\.InICML 2024 Workshop on Mechanistic Interpretability,External Links:[Link](https://openreview.net/forum?id=R5unwb9KPc)Cited by:[§1](https://arxiv.org/html/2609.00515#S1.p4.1),[§4\.1](https://arxiv.org/html/2609.00515#S4.SS1.p9.1),[§4](https://arxiv.org/html/2609.00515#S4.p2.1)\.
- Lewis \(1973\)D\. LewisCausation\.The journal of philosophy70\(17\),pp\. 556–567\.Cited by:[§2](https://arxiv.org/html/2609.00515#S2.SS0.SSS0.Px4.p1.1)\.
- Luet al\.\(2018\)Y\. Lu, P\. Keung, F\. Ladhak, V\. Bhardwaj, S\. Zhang, and J\. SunA neural interlingua for multilingual machine translation\.InProceedings of the Third Conference on Machine Translation: Research Papers,pp\. 84–92\.External Links:[Document](https://dx.doi.org/10.18653/v1/W18-6309),[Link](https://aclanthology.org/W18-6309/)Cited by:[§2](https://arxiv.org/html/2609.00515#S2.SS0.SSS0.Px2.p1.1)\.
- Luonget al\.\(2015\)T\. Luong, H\. Pham, and C\. D\. ManningEffective approaches to attention\-based neural machine translation\.InProceedings of the 2015 Conference on Empirical Methods in Natural Language Processing,L\. Màrquez, C\. Callison\-Burch, and J\. Su \(Eds\.\),Lisbon, Portugal,pp\. 1412–1421\.External Links:[Link](https://aclanthology.org/D15-1166/),[Document](https://dx.doi.org/10.18653/v1/D15-1166)Cited by:[§1](https://arxiv.org/html/2609.00515#S1.p1.1)\.
- Maoet al\.\(2023\)Z\. Mao, H\. Song, R\. Dabre, C\. Chu, and S\. KurohashiVariable\-length neural interlingua representations for zero\-shot neural machine translation\.InProceedings of the 1st International Workshop on Multilingual, Multimodal and Multitask Language Generation,A\. Barreiro, M\. Silberztein, E\. Lloret, and M\. Paprzycki \(Eds\.\),Tampere, Finland,pp\. 16–25\.External Links:[Link](https://aclanthology.org/2023.multi3generation-1.3/)Cited by:[§2](https://arxiv.org/html/2609.00515#S2.SS0.SSS0.Px2.p1.1)\.
- Markset al\.\(2025\)S\. Marks, C\. Rager, E\. J\. Michaud, Y\. Belinkov, D\. Bau, and A\. MuellerSparse feature circuits: discovering and editing interpretable causal graphs in language models\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=I4e82CIDxv)Cited by:[§2](https://arxiv.org/html/2609.00515#S2.SS0.SSS0.Px4.p1.1)\.
- Muelleret al\.\(2026\)A\. Mueller, J\. Brinkmann, M\. Li, S\. Marks, K\. Pal, N\. Prakash, C\. Rager, A\. Sankaranarayanan, A\. S\. Sharma, J\. Sun, E\. Todd, D\. Bau, and Y\. BelinkovThe quest for the right mediator: surveying mechanistic interpretability for nlp through the lens of causal mediation analysis\.Computational Linguistics52\(1\),pp\. 331–378\.External Links:ISSN 0891\-2017,[Document](https://dx.doi.org/10.1162/COLI.a.572),[Link](https://doi.org/10.1162/COLI.a.572),https://direct\.mit\.edu/coli/article\-pdf/52/1/331/2554934/coli\.a\.572\.pdfCited by:[§2](https://arxiv.org/html/2609.00515#S2.SS0.SSS0.Px4.p1.1)\.
- Pearl \(2001\)J\. PearlDirect and indirect effects\.InProceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence,UAI’01,San Francisco, CA, USA,pp\. 411–420\.External Links:ISBN 1558608001Cited by:[§2](https://arxiv.org/html/2609.00515#S2.SS0.SSS0.Px4.p1.1),[§4](https://arxiv.org/html/2609.00515#S4.p1.1)\.
- Post \(2018\)M\. PostA call for clarity in reporting BLEU scores\.InProceedings of the Third Conference on Machine Translation: Research Papers,O\. Bojar, R\. Chatterjee, C\. Federmann, M\. Fishel, Y\. Graham, B\. Haddow, M\. Huck, A\. J\. Yepes, P\. Koehn, C\. Monz, M\. Negri, A\. Névéol, M\. Neves, M\. Post, L\. Specia, M\. Turchi, and K\. Verspoor \(Eds\.\),Brussels, Belgium,pp\. 186–191\.External Links:[Link](https://aclanthology.org/W18-6319/),[Document](https://dx.doi.org/10.18653/v1/W18-6319)Cited by:[Appendix D](https://arxiv.org/html/2609.00515#A4.p2.1),[§3](https://arxiv.org/html/2609.00515#S3.p3.1)\.
- Purason and Tättar \(2022\)T\. Purason and A\. TättarMultilingual neural machine translation with the right amount of sharing\.InProceedings of the 23rd Annual Conference of the European Association for Machine Translation,pp\. 91–100\.External Links:[Link](https://aclanthology.org/2022.eamt-1.12/)Cited by:[§2](https://arxiv.org/html/2609.00515#S2.SS0.SSS0.Px2.p2.1)\.
- Qoribet al\.\(2025\)M\. R\. Qorib, J\. Li, and H\. T\. NgJust go parallel: improving the multilingual capabilities of large language models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 33411–33424\.External Links:[Link](https://aclanthology.org/2025.acl-long.1602/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1602),ISBN 979\-8\-89176\-251\-0Cited by:[§5](https://arxiv.org/html/2609.00515#S5.p1.1)\.
- Radfordet al\.\(2018\)A\. Radford, K\. Narasimhan, T\. Salimans, I\. Sutskever,et al\.Improving language understanding by generative pre\-training\.Cited by:[§1](https://arxiv.org/html/2609.00515#S1.p1.1)\.
- Robinsonet al\.\(2023\)N\. R\. Robinson, P\. Ogayo, D\. R\. Mortensen, and G\. NeubigChatGPT mt: competitive for high\- \(but not low\-\) resource languages\.External Links:2309\.07423,[Link](https://arxiv.org/abs/2309.07423)Cited by:[§1](https://arxiv.org/html/2609.00515#S1.p2.1)\.
- Sankaranarayananet al\.\(2026\)A\. Sankaranarayanan, A\. Zur, A\. Geiger, and D\. Hadfield\-MenellActivation steering via generative causal mediation\.External Links:2602\.16080,[Link](https://arxiv.org/abs/2602.16080)Cited by:[§4\.1](https://arxiv.org/html/2609.00515#S4.SS1.p1.1)\.
- Shaoet al\.\(2026\)J\. Shao, R\. Tang, C\. Zhang, K\. Sevegnani, P\. Stenetorp, J\. Yang, and Y\. LuThe role of mixed\-language documents for multilingual large language model pretraining\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 36807–36818\.External Links:[Link](https://aclanthology.org/2026.acl-long.1706/),[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.1706),ISBN 979\-8\-89176\-390\-6Cited by:[§5](https://arxiv.org/html/2609.00515#S5.p1.1)\.
- Simonyanet al\.\(2013\)K\. Simonyan, A\. Vedaldi, and A\. ZissermanDeep inside convolutional networks: visualising image classification models and saliency maps\.arXiv preprint arXiv:1312\.6034\.Cited by:[§4\.1](https://arxiv.org/html/2609.00515#S4.SS1.p4.1)\.
- Singhet al\.\(2025\)S\. Singh, A\. Romanou, C\. Fourrier, D\. I\. Adelani, J\. G\. Ngui, D\. Vila\-Suero, P\. Limkonchotiwat, K\. Marchisio, W\. Q\. Leong, Y\. Susanto, R\. Ng, S\. Longpre, S\. Ruder, W\. Ko, A\. Bosselut, A\. Oh, A\. Martins, L\. Choshen, D\. Ippolito, E\. Ferrante, M\. Fadaee, B\. Ermis, and S\. HookerGlobal MMLU: understanding and addressing cultural and linguistic biases in multilingual evaluation\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 18761–18799\.External Links:[Link](https://aclanthology.org/2025.acl-long.919/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.919),ISBN 979\-8\-89176\-251\-0Cited by:[Appendix D](https://arxiv.org/html/2609.00515#A4.p1.1),[§3](https://arxiv.org/html/2609.00515#S3.p2.1)\.
- Syedet al\.\(2023\)A\. Syed, C\. Rager, and A\. ConmyAttribution patching outperforms automated circuit discovery\.External Links:2310\.10348,[Link](https://arxiv.org/abs/2310.10348)Cited by:[§4\.1](https://arxiv.org/html/2609.00515#S4.SS1.p4.1)\.
- Tanzeret al\.\(2024\)G\. Tanzer, M\. Suzgun, E\. Visser, D\. Jurafsky, and L\. Melas\-KyriaziA benchmark for learning to translate a new language from one grammar book\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=tbVWug9f2h)Cited by:[§1](https://arxiv.org/html/2609.00515#S1.p2.1),[§2](https://arxiv.org/html/2609.00515#S2.SS0.SSS0.Px3.p1.1)\.
- Toddet al\.\(2024\)E\. Todd, M\. L\. Li, A\. S\. Sharma, A\. Mueller, B\. C\. Wallace, and D\. BauFunction vectors in large language models\.InThe Twelfth International Conference on Learning Representations,Note:arXiv:2310\.15213External Links:[Link](https://openreview.net/forum?id=AwyxtyMwaG)Cited by:[§6](https://arxiv.org/html/2609.00515#S6.p3.1)\.
- Vaswaniet al\.\(2017\)A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. PolosukhinAttention is all you need\.Advances in neural information processing systems30\.Cited by:[§1](https://arxiv.org/html/2609.00515#S1.p1.1)\.
- Vázquezet al\.\(2020\)R\. Vázquez, A\. Raganato, M\. Creutz, and J\. TiedemannA systematic study of inner\-attention\-based sentence representations in multilingual neural machine translation\.Computational Linguistics46\(2\),pp\. 387–424\.External Links:[Link](https://aclanthology.org/2020.cl-2.5/),[Document](https://dx.doi.org/10.1162/coli%5Fa%5F00377)Cited by:[§2](https://arxiv.org/html/2609.00515#S2.SS0.SSS0.Px2.p2.1)\.
- Vázquezet al\.\(2019\)R\. Vázquez, A\. Raganato, J\. Tiedemann, and M\. CreutzMultilingual NMT with a language\-independent attention bridge\.InProceedings of the 4th Workshop on Representation Learning for NLP \(RepL4NLP\-2019\),I\. Augenstein, S\. Gella, S\. Ruder, K\. Kann, B\. Can, J\. Welbl, A\. Conneau, X\. Ren, and M\. Rei \(Eds\.\),Florence, Italy,pp\. 33–39\.External Links:[Link](https://aclanthology.org/W19-4305/),[Document](https://dx.doi.org/10.18653/v1/W19-4305)Cited by:[§2](https://arxiv.org/html/2609.00515#S2.SS0.SSS0.Px2.p1.1)\.
- Viget al\.\(2020\)J\. Vig, S\. Gehrmann, Y\. Belinkov, S\. Qian, D\. Nevo, Y\. Singer, and S\. ShieberInvestigating gender bias in language models using causal mediation analysis\.InAdvances in Neural Information Processing Systems,H\. Larochelle, M\. Ranzato, R\. Hadsell, M\.F\. Balcan, and H\. Lin \(Eds\.\),Vol\.33,pp\. 12388–12401\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2020/file/92650b2e92217715fe312e6fa7b90d82-Paper.pdf)Cited by:[§2](https://arxiv.org/html/2609.00515#S2.SS0.SSS0.Px4.p1.1),[§4](https://arxiv.org/html/2609.00515#S4.p1.1)\.
- Warstadtet al\.\(2020\)A\. Warstadt, A\. Parrish, H\. Liu, A\. Mohananey, W\. Peng, S\. Wang, and S\. R\. BowmanBLiMP: the benchmark of linguistic minimal pairs for English\.Transactions of the Association for Computational Linguistics8,pp\. 377–392\.External Links:[Link](https://aclanthology.org/2020.tacl-1.25/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00321)Cited by:[§3](https://arxiv.org/html/2609.00515#S3.p2.1)\.
- Wendleret al\.\(2024\)C\. Wendler, V\. Veselovsky, G\. Monea, and R\. WestDo llamas work in English? on the latent language of multilingual transformers\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 15366–15394\.External Links:[Link](https://aclanthology.org/2024.acl-long.820/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.820)Cited by:[§1](https://arxiv.org/html/2609.00515#S1.p3.1),[§2](https://arxiv.org/html/2609.00515#S2.SS0.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2609.00515#S4.SS2.p1.1)\.
- Wuet al\.\(2025\)Z\. Wu, X\. V\. Yu, D\. Yogatama, J\. Lu, and Y\. KimThe semantic hub hypothesis: language models share semantic representations across languages and modalities\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=FrFQpAgnGE)Cited by:[§1](https://arxiv.org/html/2609.00515#S1.p3.1)\.
- Zhuet al\.\(2020\)C\. Zhu, H\. Yu, S\. Cheng, and W\. LuoLanguage\-aware interlingua for multilingual neural machine translation\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 1650–1655\.External Links:[Link](https://aclanthology.org/2020.acl-main.150/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.150)Cited by:[§2](https://arxiv.org/html/2609.00515#S2.SS0.SSS0.Px2.p1.1)\.

## Appendix AAdditional Results for Modeling Translation Performance

Figure[5](https://arxiv.org/html/2609.00515#A1.F5)shows predicted vs\. actual BLEU scores for the linear and bilinear models\. In general, linear and bilinear models produce visually indistinguishable predictions\.

![Refer to caption](https://arxiv.org/html/2609.00515v1/figures/pred_vs_actual_grid_llama.png)

![Refer to caption](https://arxiv.org/html/2609.00515v1/figures/pred_vs_actual_grid_aya.png)

Figure 5:Predicted \(xx\) vs\. actual \(yy\) BLEU on the common 18\-language subset, for each proxy \(columns\) under the linear \(top row\) and bilinear \(bottom row\) models, Llama \(top\) and Aya \(bottom\)\. The two rows are visually indistinguishable for every proxy—the graphical form of the null interaction in Table[2](https://arxiv.org/html/2609.00515#S3.T2)\.### A\.1Language\-pair interaction tests

Table[4](https://arxiv.org/html/2609.00515#A1.T4)reports the nested\-modelFF\-test on thelS⋅lTl\_\{S\}\\\!\\cdot\\\!l\_\{T\}interaction coefficient, on the common 18\-language subset and the 17\-language no\-English subset\. The interaction is non\-significant in every cell\.

Table 4:NestedFF\-test for the language\-pair interaction term\. Non\-significant everywhere; largestΔ​R2=0\.0038\\Delta R^\{2\}=0\.0038\.
### A\.2Predicting BLEU from Language\-specific Translation Capabilities

Here, we show BLEU scores for all language pairs \(Figure[6](https://arxiv.org/html/2609.00515#A1.F6), left\)\. We take each source or target language’s average BLEU across language pairs, and fit linear models based on these terms to predict each language pair’s BLEU score \(see §[3](https://arxiv.org/html/2609.00515#S3)for details\)\. We observe that the error of a rank\-1 linear predictor \(Figure[6](https://arxiv.org/html/2609.00515#A1.F6), right\) is generally low at around 10–15%\. This provides further evidence that BLEU scores are predictable as a function of language\-specific capabilities\.

![Refer to caption](https://arxiv.org/html/2609.00515v1/figures/bleu_matrix_rank1_llama_masked.png)Figure 6:Left: the observed24×2424\\times 24Llama BLEU matrix \(self\-translation diagonal masked, never imputed\)\. Right: its rank\-1 reconstructionuS​vTu\_\{S\}v\_\{T\}, fit by masked alternating least squares over the 552 observed off\-diagonal cells\. Faithfulness=89\.8%=89\.8\\%\(Aya:83\.4%83\.4\\%\); a single source\-competence vector outer\-producted with a single target\-competence vector reconstructs most of the matrix\.
### A\.3Where the grammatical signal concentrates

Restricting the MultiBLiMP margin to subject–verb agreement phenomena*improves*BLEU prediction \(Llama, 18 languages: all\-phenomena marginR2=0\.324→0\.363R^\{2\}=0\.324\\to 0\.363for the SV\-agreement subset\)\. Per phenomenon, SV\-Person reachesR2=0\.688R^\{2\}=0\.688, SV\-Gender0\.5190\.519, SV\-Number0\.4120\.412, versus subject–predicate agreement at0\.1510\.151/0\.0370\.037\. The production\-side signal \(agreement on the generated verb\) is where translation predictiveness lives\. We caution that per\-phenomenon coverage is confounded with which languages each phenomenon is annotated for, so we report this as a strengthening analysis rather than a headline\.

### A\.4MMLU subject subsets

Unlike the MultiBLiMP phenomenon breakdown, the BLEU\-predictive signal in MMLU is*not*concentrated in any single subject category: aggregate MMLU \(R2=0\.598R^\{2\}=0\.598Llama /0\.4990\.499Aya, 23 languages\) is stronger than every individual category \(best single subset: Humanities0\.5600\.560for Llama, Social Sciences0\.4130\.413for Aya; STEM0\.3600\.360/0\.1900\.190\)\. As with phenomena, subject\-category accuracies are highly collinear across languages and largely track per\-language resource level\.

### A\.5Functional form

BLEU is bounded and right\-skewed, and grammatical accuracy saturates near 1\. Replacing the raw fit withlog⁡BLEU∼log⁡\(1−aS\)\+log⁡\(1−aT\)\\log\\text\{BLEU\}\\sim\\log\(1\-a\_\{S\}\)\+\\log\(1\-a\_\{T\}\)\(a logit\-like transform that un\-saturates accuracy\) improvesR2R^\{2\}from0\.2500\.250to0\.3390\.339\(Llama\) and0\.3940\.394to0\.4440\.444\(Aya\); the analogous log–log fit for perplexity improves0\.071→0\.1270\.071\\to 0\.127\(Llama\)\. We read these gains as variance\-stabilization addressing the same ceiling effect that motivates the log\-probability margin, rather than evidence of a specific \(e\.g\. exponential\) functional form, and so report the raw\-scale linear models in the main text\.

### A\.6Does grammatical competence add signal beyond Global MMLU?

Adding MultiBLiMP source and target terms on top of the Global MMLU\-only regression gives a small but significant gain \(Table[5](https://arxiv.org/html/2609.00515#A1.T5)\)\. Grammatical competence carries a little translation\-relevant signal that general task accuracy doesn’t capture\.

Table 5:Does grammatical competence add predictive value beyond general competence? We compareBLEU∼MMLUS\+MMLUT\\text\{BLEU\}\\sim\\text\{MMLU\}\_\{S\}\+\\text\{MMLU\}\_\{T\}to the same model alongside MultiBLiMP source and target terms\. Adding MultiBLiMP accuracy adds some small signal for Llama and Aya\.

## Appendix BAdditional GCM Ablation Results

### B\.1Translation\-specific head decompositions

Figure[7](https://arxiv.org/html/2609.00515#A2.F7)shows the attention head decompositions as in Fig\.[2](https://arxiv.org/html/2609.00515#S4.F2)in the main text\. The top heads concentrate in slightly later layers than Llama\.

![Refer to caption](https://arxiv.org/html/2609.00515v1/figures/promptswap_aya_decomposition_heads.png)Figure 7:Prompt\-swap GCM head IE under the four control tasks, top 20 heads sorted byreal\_cross−null\_cross\\textsc\{real\\\_cross\}\-\\textsc\{null\\\_cross\}, for Aya\-23\-8B \(cf\. Fig\.[2](https://arxiv.org/html/2609.00515#S4.F2)\)\. The top heads concentrate in layers 15–20\. Support and bar definitions as in Fig\.[2](https://arxiv.org/html/2609.00515#S4.F2)\.
### B\.2Top translation\-specific heads

Tables[6](https://arxiv.org/html/2609.00515#A2.T6)and[7](https://arxiv.org/html/2609.00515#A2.T7)show the details of the top 10 attention heads by translation\-specific effect\. A majority of the effect is concentrated in the top heads\.

Table 6:Top 10 attention heads by translation\-specific effectΔ=real\_cross−null\_cross\\Delta=\\textsc\{real\\\_cross\}\-\\textsc\{null\\\_cross\}under prompt\-swap GCM \(Llama\), with mean\|IE^\|\|\\widehat\{\\mathrm\{IE\}\}\|in each of the four control tasks\. \#dir is the number of the 56 cross\-language directions in which the head ranks in that direction’s top\-30 byreal\_crosseffect; all but one head appear in every direction\.Table 7:Top 10 Aya attention heads by translation\-specific effectΔ=real\_cross−null\_cross\\Delta=\\textsc\{real\\\_cross\}\-\\textsc\{null\\\_cross\}under prompt\-swap GCM \(cf\. Table[6](https://arxiv.org/html/2609.00515#A2.T6)\)\. \#dir is the number of the 56 cross\-language directions in which the head ranks top\-30 byreal\_crosseffect\.
### B\.3SAE\-feature decomposition analyses

Top features show the samereal\_cross\-dominant pattern as attention heads but with weaker separation\. A few top\-ranked features are same\-language features \(Fig\.[8](https://arxiv.org/html/2609.00515#A2.F8)and[9](https://arxiv.org/html/2609.00515#A2.F9)\)\.

![Refer to caption](https://arxiv.org/html/2609.00515v1/figures/promptswap_decomposition_sae.png)Figure 8:Llama\-3\.1\-8B mean\|IE^\|\|\\widehat\{\\mathrm\{IE\}\}\|for top SAE features under translation and control prompts\.![Refer to caption](https://arxiv.org/html/2609.00515v1/figures/promptswap_aya_decomposition_sae.png)Figure 9:Aya\-23\-8B mean\|IE^\|\|\\widehat\{\\mathrm\{IE\}\}\|for top SAE features under translation and control prompts\.
### B\.4Universality of translation\-relevant heads

Translation\-relevant heads are universal across translation language pairs; Hebrew and Hindi show more divergence from the rest of the languages \(Fig\.[10](https://arxiv.org/html/2609.00515#A2.F10)and[11](https://arxiv.org/html/2609.00515#A2.F11)\)\.

![Refer to caption](https://arxiv.org/html/2609.00515v1/figures/promptswap_llama_universal_heads_all_directions.png)Figure 10:Full Llama\-3\.1\-8B signed\-IE heatmap for the 15 most universal prompt\-swap heads across all 56 translation directions\.![Refer to caption](https://arxiv.org/html/2609.00515v1/figures/promptswap_aya_universal_heads_all_directions.png)Figure 11:Full Aya\-23\-8B signed\-IE heatmap for the 15 most universal prompt\-swap heads across all 56 translation directions\.
### B\.5Head\-IE concentration and sparsity

As further illustration of the head\-IE sparsity, Fig\.[12](https://arxiv.org/html/2609.00515#A2.F12)and[13](https://arxiv.org/html/2609.00515#A2.F13)show that the ratio of the largest mean \|IE\| to the median ranges around 2 orders of magnitude\.

![Refer to caption](https://arxiv.org/html/2609.00515v1/figures/promptswap_head_ie_top_median_ratio.png)Figure 12:Per\-direction head\-IE sparsity under prompt\-swap GCM: the ratio of the largest head’s mean\|IE^\|\|\\widehat\{\\mathrm\{IE\}\}\|to the median head’s, for each source→\\totarget direction \(Llama\-3\.1\-8B; diagonal omitted\)\. The ratio ranges from 60 to 162 \(median 116\)\. The top head is roughly two orders of magnitude above the median head in every direction, and is largest for directions into Spanish, German, and French and smallest into Hindi and Hebrew\.![Refer to caption](https://arxiv.org/html/2609.00515v1/figures/promptswap_aya_head_ie_top_median_ratio.png)Figure 13:Per\-direction head\-IE sparsity for Aya\-23\-8B under prompt\-swap GCM \(cf\. Fig\.[12](https://arxiv.org/html/2609.00515#A2.F12)\)\. The top\-head/median\-head ratio ranges from 51 to 104 \(median 83\)\.The sign of the translation task selection carries over to Multi\-BLiMP: ablating the POS\-10 heads lowers the grammaticality margin in all 8 target languages for both models, and ablating the NEG\-10 heads raises it in all 8 \(Fig\.[14](https://arxiv.org/html/2609.00515#A2.F14); Table[8](https://arxiv.org/html/2609.00515#A2.T8)\)\. The magnitude of the POS\-10 effect, however, is comparable to that of ablating random heads from the same layers, so it is the consistent direction of the effects, rather than their size, that suggests these heads are reused across tasks\.

![Refer to caption](https://arxiv.org/html/2609.00515v1/figures/interventions_multiblimp_smctrl.png)Figure 14:Change in the Multi\-BLiMP grammaticality margin from mean\-ablating the POS\-10 and NEG\-10 head sets and a random control\. Dotted lines mark cross\-language means\.Table 8:Multi\-BLiMP marginΔchange\\Delta\_\{\\text\{change\}\}relative to baseline from mean\-ablating the POS\-10 and NEG\-10 GCM head sets and their size\-matched controls \(ctrlPand ctrlN: 10 random heads drawn from the same layers as the corresponding set\)\. Negative means ablation weakened the grammaticality preference\. Ablating POS\-10 lowers the margin in all 8 targets and ablating NEG\-10 raises it in all 8, for both models; the size of the POS\-10 effect, however, is comparable to that of its random same\-layer control\.n=400n=400pairs per language except heb \(n=200n=200\) and hin \(n=100n=100\)\.The choice of the number of heads to ablate is unimportant to the overall effect; ablating different numbers of top heads has monotonic effect on performance \(Fig\.[15](https://arxiv.org/html/2609.00515#A2.F15)\)\.

![Refer to caption](https://arxiv.org/html/2609.00515v1/figures/head_count_dose_response.png)Figure 15:Head\-count dose response \(deu/eng/fra\) on Llama\. The effect is monotonic with respect to the number of heads ablated\.

## Appendix CAdditional Fine\-Tuning Results

This section reports the remaining fine\-tuning results\. We use the same models, training conditions, and evaluation procedure described in the main text\.

### C\.1Additional directions involving Xhosa

Table[9](https://arxiv.org/html/2609.00515#A3.T9)reports translation into Xhosa from English, French, and German\. Both fine\-tuning conditions improve performance across all three directions and both models\. For Llama, parallel fine\-tuning performs best when translating from English and French, while monolingual fine\-tuning performs best when translating from German\. For Aya, monolingual fine\-tuning performs slightly better in all three directions\. Thus, neither condition performs best in every setting\.

Table 9:BLEU for translation into Xhosa\. Column labels identify the source language\. Bold marks the best fine\-tuned result for each model and direction\.
### C\.2Fine\-tuning on higher\-resource languages

We also apply the same general pipeline to German and Thai using Llama\-3\.1\-8B\. Unlike Xhosa, both languages already have strong translation performance in the base model\. As shown in Table[10](https://arxiv.org/html/2609.00515#A3.T10), neither monolingual nor parallel fine\-tuning produces a consistent improvement\. Monolingual fine\-tuning leaves performance largely unchanged, while parallel fine\-tuning reduces performance in several directions\.

One possible explanation is that German and Thai were already well represented during pretraining, leaving less room for improvement from continued training\. Some of the fine\-tuning documents may also have appeared in the pretraining corpus\.

Table 10:BLEU after fine\-tuning Llama\-3\.1\-8B on German or Thai\. The first two result columns involve the adapted language; the final two measure other translation directions\.
### C\.3Preservation of monolingual abilities

We also measure MultiBLiMP accuracy after fine\-tuning Llama on Xhosa\. As shown in Table[11](https://arxiv.org/html/2609.00515#A3.T11), performance is largely preserved\. However, the base model is already near the maximum score on all three languages\. These results are therefore not strong enough to determine whether fine\-tuning causes cross\-lingual transfer of linguistic abilities\.

Table 11:MultiBLiMP accuracy after fine\-tuning Llama\-3\.1\-8B\. Scores are percentages\. The near\-ceiling base scores make small differences difficult to interpret\.Only the monolingual condition includes replay data from high\-resource languages\. Differences in retained performance therefore cannot be attributed only to the use of monolingual rather than parallel data\. A parallel condition with similar replay data could also reduce forgetting\.

## Appendix DArtifact Licenses

We use several existing scientific artifacts in our experiments\. FLORES\-101\([Goyal et al\., 2022](https://arxiv.org/html/2609.00515#bib.bib31)\)is released under a CC BY\-SA 4\.0 license\. MultiBLiMP\([Jumelet et al\., 2026](https://arxiv.org/html/2609.00515#bib.bib11)\)is released under a CC BY 4\.0 license\. GlobalMMLU\([Singh et al\., 2025](https://arxiv.org/html/2609.00515#bib.bib10)\)is released under an Apache 2\.0 license\. For the Xhosa–English fine\-tuning experiments, we use the English–Xhosa MT560 sentence\-pair dataset\([Gowda et al\., 2021](https://arxiv.org/html/2609.00515#bib.bib29)\); because OPUS MT560 aggregates data from multiple sources, downstream users should consult the original OPUS MT560 provenance information before redistributing or extending the dataset\.

We also use pretrained model checkpoints and evaluation software\. Llama\-3\.1\-8B is released under the Llama 3\.1 Community License, and Aya\-23\-8B is released under CC BY\-NC 4\.0 with Cohere’s acceptable\-use addendum\. SacreBLEU\([Post, 2018](https://arxiv.org/html/2609.00515#bib.bib27)\), which we use to compute BLEU scores, is released under an Apache 2\.0 license\.

In accordance with their licensing terms, all artifacts are used solely for research purposes\.

## Appendix EComputational Budget

All experiments use 8B\-parameter language models: Llama\-3\.1\-8B and Aya\-23\-8B\. We do not train any models from scratch\. The compute cost is dominated by translation generation and scoring, activation caching, gradient\-based attribution patching, ablation experiments, and LoRA fine\-tuning for the Xhosa adaptation experiments\.

The total computational budget was approximately 200 GPU hours on NVIDIA A100/H100 GPUs\. The largest individual runs were the causal mediation experiments, which require forward passes with activation caching and backward passes for attribution estimates across attention heads, and the LoRA fine\-tuning sweeps over learning rate, LoRA rank, and number of training examples\.

For activation caching, we usedNNsight\([Fiotto\-Kaufman et al\., 2025](https://arxiv.org/html/2609.00515#bib.bib47)\)\.