Why Pretraining Fails to Share Cross-Lingual Knowledge
Summary
This research paper investigates why pretraining in large language models fails to transfer knowledge across languages, identifies disjoint token spaces as a fundamental barrier, and proposes mapping languages to a shared token space to improve cross-lingual generalization.
arXiv:2609.19291v1 Announce Type: new
Abstract: Large Language Models (LLMs) have made remarkable progress in the processing and modeling of many languages. Yet, unlike human multilinguals, they exhibit surprisingly limited cross-lingual knowledge transfer. While this limitation is well documented, its origins during multilingual training remain unclear. We pretrain 360M- and 7B-parameter LLMs and show that poor cross-lingual knowledge generalization emerges during pretraining and persists under standard interventions. To isolate its cause, we employ a controlled bilingual pretraining setting using two copies of the same language, sharing identical text and token segmentation, but mapped to disjoint token spaces. We find that disjoint tokens alone are enough to induce knowledge compartmentalization, even between identical copies of the same language, establishing disjoint token spaces as a fundamental barrier to cross-lingual knowledge generalization. Guided by this understanding, we suggest mapping languages into a shared token space by simple word-wise translation and find it substantially improves cross-lingual knowledge generalization, recovering up to 12.6\% of native-language learning efficiency --- 14$\times$ the baseline.
View Cached Full Text
Cached at: 09/18/26, 08:53 AM
# Why Pretraining Fails to Share Cross-Lingual Knowledge Source: [https://arxiv.org/html/2609.19291](https://arxiv.org/html/2609.19291) CoLab Lab2026\-09\-16 Why Pretraining Fails to Share Cross\-Lingual Knowledge Adam Gaber1,†,Uriel Dolev2,Elisabeth Fittschen3,Bobby Cheng4,Yuval Marton5,Leshem Choshen1,6,7,∗ 1Weizmann Institute of Science,2Bar\-Ilan University,3Johns Hopkins University,4A\*STAR,5University of Washington,6MIT,7MIT\-IBM Watson AI Lab Large Language Models \(LLMs\) have made remarkable progress in the processing and modeling of many languages\. Yet, unlike human multilinguals, they exhibit surprisingly limited cross\-lingual knowledge transfer\. While this limitation is well documented, its origins during multilingual training remain unclear\. We pretrain 360M\- and 7B\-parameter LLMs and show that poor cross\-lingual knowledge generalization emerges during pretraining and persists under standard interventions\. To isolate its cause, we employ a controlled bilingual pretraining setting using two copies of the same language, sharing identical text and token segmentation, but mapped to disjoint token spaces\. We find that disjoint tokens alone are enough to induce knowledge compartmentalization, even between identical copies of the same language, establishing disjoint token spaces as a fundamental barrier to cross\-lingual knowledge generalization\. Guided by this understanding, we suggest mapping languages into a shared token space by simple word\-wise translation and find it substantially improves cross\-lingual knowledge generalization, recovering up to 12\.6% of native\-language learning efficiency — 14×\\timesthe baseline\. 22footnotetext:Corresponding author:[adam\.gaber@weizmann\.ac\.il](mailto:[email protected])11footnotetext:Corresponding author:[leshem\.choshen@weizmann\.ac\.il](mailto:[email protected])33footnotetext:Code publicly available[https://github\.com/AdamJaber03/torchtitan\-mulitlingual](https://github.com/AdamJaber03/torchtitan-mulitlingual)Figure 1:Merging token spaces via WWT yields substantial gains in cross\-lingual knowledge generalization\. Left:CLeqforEnglish–Arabicpretraining under various methods \(colors\), measuring cross\-lingual knowledge learning efficiency at 360M scale\. Right: left panel shows English perplexity at 360M; upper bound is a model trained on the English half alone, the lower bound is a model trained on twice as much English\. Right panel shows perplexity at 7B, bounds are proportional to 360M\.## 1 Introduction Modern LLMs exhibit impressive multilingual capabilities, yet unlike humans, they transfer surprisingly little knowledge across languages\. Understanding this limitation is important for building truly multilingual models\. Prior work has documented poor cross\-lingual generalization, characterizing it through model capacity and sample efficiency, and exploring factors such as tokenization, language imbalance, and activation alignment\([Chang et al\., 2024b](https://arxiv.org/html/2609.19291#bib.bib18);[Schäfer et al\., 2024](https://arxiv.org/html/2609.19291#bib.bib5);[Howe and Wingate, 2026](https://arxiv.org/html/2609.19291#bib.bib20);[Wu and Dredze, 2020](https://arxiv.org/html/2609.19291#bib.bib27);[Li et al\., 2024b](https://arxiv.org/html/2609.19291#bib.bib28)\)\. Recent work extends this generalization failure into the knowledge domain\([Qi et al\., 2023](https://arxiv.org/html/2609.19291#bib.bib31);[Ifergan et al\., 2025](https://arxiv.org/html/2609.19291#bib.bib30);[Goldman et al\., 2025](https://arxiv.org/html/2609.19291#bib.bib29)\)and tries to reduce it\([Huang et al\., 2023](https://arxiv.org/html/2609.19291#bib.bib33);[Diskind et al\., 2026](https://arxiv.org/html/2609.19291#bib.bib34);[Bandarkar et al\., 2026](https://arxiv.org/html/2609.19291#bib.bib10);[von Rad et al\., 2026a](https://arxiv.org/html/2609.19291#bib.bib32)\)\. However, these observations and fixes are made post\-hoc on fully trained models, which go through a sequence of training stages including pretraining, instruction tuning, and preference optimization, leaving open where this knowledge generalization failure, which we refer to as compartmentalization, originates\. We investigate compartmentalization directly at the pretraining stage\. However, causal analysis in this setting is complicated by two factors: First, pretraining corpora are vast, and tracking knowledge exposure across languages is intractable\. Second, natural languages simultaneously differ in vocabulary, scripts, syntax, frequency distributions and semantics, hindering controlled experiments\. We address these challenges through a two\-part methodology\. First, to reliably trace knowledge exposure, we introduce a controlled pretraining environment utilizing the injection of fictive knowledge \(§[2](https://arxiv.org/html/2609.19291#S2)\)\. Second, to strip away linguistic confounds, we evaluate generalization not only on natural bilingual pairs but also on structurally identical, cloned language copies, that differ in nothing but their token spaces \(§[5\.1](https://arxiv.org/html/2609.19291#S5.SS1)\)\. Using this combined framework, we pretrain 360M\- and 7B\-parameter models to measure and isolate the mechanisms of cross\-lingual knowledge generalization\. Our controlled measurements reveal that knowledgecompartmentalization already emerges during pretraining\(§[4](https://arxiv.org/html/2609.19291#S4)\)\. We further identifydisjoint token spaces as a fundamental barrierto the knowledge generalization mechanism that persists across model scales\. Two languages represented in disjoint regions of the model’s vocabulary, regardless of token segmentation, hinder sharing \(§[5](https://arxiv.org/html/2609.19291#S5)\)\. Guided by our newfound understanding of this mechanism, we suggest a word\-wise translation \(WWT\) intervention to efficiently unify the token space of vocabulary\-disjoint languages \(§[6\.1](https://arxiv.org/html/2609.19291#S6.SS1)\)\. This intervention proves beneficial, substantially improving cross\-lingual knowledge generalization while providing gains in general language modeling perplexity \(Fig\.[1](https://arxiv.org/html/2609.19291#S0.F1)\)\. This practical success provides a mechanistic validation of our findings on tokenization’s role in knowledge generalization \(§[6\.2](https://arxiv.org/html/2609.19291#S6.SS2)\)\. ## 2 Methods: Measuring Knowledge Generalization ### 2\.1 Controlling knowledge exposure \- Injecting fictive facts To measure if and to what extent exposure to a fact during training in one language yields knowledge of that fact in another language, we must know how often the fact has been observed in each language\. Tracing fact exposures in the vast pretraining corpora is often intractable\([Kandpal et al\., 2023](https://arxiv.org/html/2609.19291#bib.bib3);[Elazar et al\., 2024](https://arxiv.org/html/2609.19291#bib.bib4)\)\. We bypass this problem by injecting novel fictive knowledge, absent from the underlying pretraining data, with precise control over the number and language of exposures to each fact\. Our Fictional Knowledge Dataset \(FKD\) contains 2,048 facts about synthetic entities and associated attributes \(e\.g\., “Cultramive is a company that sells silver\-plated bicycles”\)\. Each fact is expanded into a per\-language set of diverse natural documents that paraphrase the same underlying knowledge — used for injection during pretraining — and held\-out multiple\-choice questions \(MCQs\) for evaluation\. We provide full details on FKD, including the generation pipeline and examples in App\.[B](https://arxiv.org/html/2609.19291#A2)\. Since prior work indicates that cross\-lingual generalization mechanisms can behave differently for high\- and low\-frequency tokens\([Schäfer et al\., 2024](https://arxiv.org/html/2609.19291#bib.bib5);[Feng et al\., 2025](https://arxiv.org/html/2609.19291#bib.bib71)\), we replicate our main experiments on a fictive\-persona dataset built around high\-frequency human names \(App\.[C](https://arxiv.org/html/2609.19291#A3)\)\. We validate FKD by pretraining monolingual models with entity facts injected at different rates \(App\.[B\.5](https://arxiv.org/html/2609.19291#A2.SS5)\)\. At 0\-injection, accuracy on the MCQs is around chance, injecting fact paraphrasings into pretraining increases accuracy in a log\-like manner, reaching substantial accuracy across all languages; FKD is learnable and controllable\. ### 2\.2 Evaluating Cross\-lingual knowledge generalization –CLeqScore While fictive knowledge allows us to control how it is introduced across languages, we’re still missing a direct measure, given a bilingual model, that quantifies how well it generalizes factual knowledge between languagesAAandBB\. To this end we introduce the Cross\-Lingual equivalence \(CLeq\) score\. CLeqB→A\\text\{CL\\textsubscript\{eq\}\}\_\{B\\rightarrow A\}measures how valuable an exposure to a fact in languageBBis for recalling that fact in languageAA, relative to a native exposure inAAitself\. A score of100%100\\%indicates that exposure inBBis as effective as exposure inAA, corresponding to perfect cross\-lingual equivalence\. A score of0%0\\%, in contrast, indicates that exposure inBBprovides no benefit for recall inAA, corresponding to complete knowledge compartmentalization across the two languages\. When reporting a single score for a given bilingual model trained on languagesAAandBB, we use the bidirectional average: CLeq=12\(CLeqA→B\+CLeqB→A\)\.\\text\{CL\\textsubscript\{eq\}\}=\\frac\{1\}\{2\}\\left\(\\text\{CL\\textsubscript\{eq\}\}\_\{A\\rightarrow B\}\+\\text\{CL\\textsubscript\{eq\}\}\_\{B\\rightarrow A\}\\right\)\. Crucially, ourCLeqdefinition normalizes cross\-lingual gains by native ones\. This makes our metric invariant to differences in baseline model competence \(raw accuracy\) that may arise from scale, training recipe, or data quality; a model that learns all facts better raises foreign and native exposure value together, leaving their ratio unchanged\. This separates gains in target\-language ability from gains in transfer, which measurements reported in absolute accuracy terms conflate\. #### EstimatingCLeqscore Controlling FKD exposure enables a direct estimate of a model’sCLeqscore\. For a fictive factffin FKD, letNA\(f\)N\_\{A\}\(f\)andNB\(f\)N\_\{B\}\(f\)denote the number of exposures to it during pretraining in languagesAAandBBrespectively, and letaccA\(f\)\\mathrm\{acc\}\_\{A\}\(f\)denote trained model accuracy onff’s language\-AAevaluation questions\. We fit a linear regression \(OLS\) over all facts, accA\(f\)=α\+βANA\(f\)\+βBNB\(f\)\+ε,\\mathrm\{acc\}\_\{A\}\(f\)=\\alpha\+\\beta\_\{A\}\\,N\_\{A\}\(f\)\+\\beta\_\{B\}\\,N\_\{B\}\(f\)\+\\varepsilon,\(1\)βA\\beta\_\{A\}captures the accuracy gained per native exposure,βB\\beta\_\{B\}captures the accuracy gained per cross\-lingual exposure, and the score is derived from their ratio: CLeqB→A=100×βBβA\.\\text\{CL\\textsubscript\{eq\}\}\_\{B\\rightarrow A\}=100\\times\\frac\{\\beta\_\{B\}\}\{\\beta\_\{A\}\}\.\(2\) Two features of this estimate warrant note\. First, the OLS regression provides a linear approximation to the relationship between exposure and accuracy \(§[2\.1](https://arxiv.org/html/2609.19291#S2.SS1)\); we experiment with an alternative curve\-aware estimator and find the simple linear form more informative\. Second, sampling and evaluation noise can occasionally pull estimates slightly below zero\. Negative values should be read as substantial or absolute compartmentalization rather than negative transfer\. App\.[D](https://arxiv.org/html/2609.19291#A4)covers these topics in detail, and reports significance measures confirming our headline comparisons are statistically robust\. ## 3 Experimental Setup Our experiments follow a common template\. We pretrain bilingual models from scratch, injecting FKD fictive facts at varying cross\-lingual rates, and estimateCLeqscore based on their acquisition\. We vary isolated aspects of the bilingual setup \(architecture, tokenization, and interventions\) and compare the resulting scores to reason about cross\-lingual knowledge generalization\. Each of the presentedCLeqresults requires a full model pretraining run corresponding to its unique condition\. ### 3\.1 Models and data We evaluate two scales: a 360M\-parameter model following the SmolLM2 architecture\([Allal et al\., 2025](https://arxiv.org/html/2609.19291#bib.bib9)\), and a 7B\-parameter adaptation of Llama\-3\([Grattafiori et al\., 2024](https://arxiv.org/html/2609.19291#bib.bib11)\)\. Both use a BPE tokenizer with a 65,536\-token vocabulary\. We train one tokenizer per language pair on a 50/50 split of 7M documents from each of its two languages, and share it across the matching runs\. Models whose perplexities we compare therefore read text under identical segmentation\. Models are trained to approximately Chinchilla\-optimal token budgets\([Hoffmann et al\., 2022](https://arxiv.org/html/2609.19291#bib.bib15)\)using TorchTitan\([Liang et al\., 2025](https://arxiv.org/html/2609.19291#bib.bib19)\)\. Architectures, hyperparameters, schedules, and hardware are detailed in App\.[A](https://arxiv.org/html/2609.19291#A1)\. We pretrain onFineWeb\-edufor English\([Penedo et al\., 2024](https://arxiv.org/html/2609.19291#bib.bib12)\),FineWeb\-edu\-arfor Arabic\([Alrashed et al\., 2024](https://arxiv.org/html/2609.19291#bib.bib14)\), a machine translation of FineWeb\-edu, andFineWeb2\-HQfor Russian\([Messmer et al\., 2025](https://arxiv.org/html/2609.19291#bib.bib13)\), native web text rather than translation\. Our bilingual experiments use a 50/50 token budget split between languages\. As the English and Arabic corpora are translations of one another, we enforce a strict document\-disjoint split to ensure models do not see parallel documents across languages\. ### 3\.2 Evaluation To calculate theCLeqmetric, we split FKD into 16 groups of 128 fictive entity facts\. Each of the 16 groups has all its fictive facts injected at a uniformNAN\_\{A\}andNBN\_\{B\}chosen from the cross\-product of \[0,20,100,1000\] with itself\. This range was selected to be denser on the critical ranges of the fact learning curve \(App\.[B\.5](https://arxiv.org/html/2609.19291#A2.SS5)\)\. Fact paraphrasing documents are injected stochastically into pretraining at a per\-language probability matching the target exposure countsNAN\_\{A\}andNBN\_\{B\}\(App\.[A\.5](https://arxiv.org/html/2609.19291#A1.SS5)\)\. We use the LM\-eval\-harness\([Gao et al\., 2024](https://arxiv.org/html/2609.19291#bib.bib37)\)evaluation protocol along with the accompanying FKD per\-language evaluation MCQs \(generated separately from the injection documents and never present in training\) to quantify the model’s fact acquisition across the different languages:accA\\mathrm\{acc\}\_\{A\}andaccB\\mathrm\{acc\}\_\{B\}\. Specifically, we use acc\-norm scoring from LM\-eval\-harness to rectify for possible variance in choice length\. With these values at hand, we fit the regression of Eq\.[1](https://arxiv.org/html/2609.19291#S2.E1)across all facts and takeCLeqas the ratio ofβA\\beta\_\{A\}andβB\\beta\_\{B\}\(§[2\.2](https://arxiv.org/html/2609.19291#S2.SS2.SSS0.Px1)\)\. We report fitted coefficients for representative runs in App\.[D](https://arxiv.org/html/2609.19291#A4) ## 4 Knowledge Compartmentalization Originates in Pretraining ### 4\.1 Compartmentalization emerges in standard bilingual pretraining Applying the above\-mentioned tools, we show that current pretraining practices strongly compartmentalize knowledge\. We pretrain a 360M\-parameterEnglish–ArabicLLM, injecting fictive facts at controlled cross\-lingual rates \(§[2\.1](https://arxiv.org/html/2609.19291#S2.SS1)\)\. With the trained bilingual model at hand, we measure fictive fact recall and deduce the relative value of foreign exposure to knowledge — theCLeqscore \(§[2\.2](https://arxiv.org/html/2609.19291#S2.SS2.SSS0.Px1)\)\. The pretrainedEnglish–Arabicmodel shows minuscule knowledge generalization, with aCLeqscore of 0\.9% \(Fig\.[1](https://arxiv.org/html/2609.19291#S0.F1)\)\. Severe knowledge compartmentalization is also apparent in anEnglish–Russiansetup withCLeq=1\.9%\\text\{CL\\textsubscript\{eq\}\}=1\.9\\%and at the 7B parameter scale withCLeq=5\.9%\\text\{CL\\textsubscript\{eq\}\}=5\.9\\%\(Tab\.[2](https://arxiv.org/html/2609.19291#S6.T2)\), suggesting knowledge compartmentalization in pretraining is not an artifact of our specific language pairing or scale\. Analyzing per\-language perplexity, we find that a bilingual setup is slightly better than pretraining on either of the monolingual halves of the corpus alone \(Fig\.[1](https://arxiv.org/html/2609.19291#S0.F1)\), consistent with previous works demonstrating positive transfer\([Conneau et al\., 2020a](https://arxiv.org/html/2609.19291#bib.bib21)\)\. However, it appears that whatever is driving this gain in language modeling, it is not knowledge generalization\. ### 4\.2 Standard interventions fail As bilingual pretraining naturally compartmentalizes knowledge, we experiment with pretraining interventions suggested for related purposes to resolve it, and find that they largely fail\. In particular, we experiment with two types of pretraining interventions in theEnglish–Arabicsetting at 360M\-parameter scale: language mixing and activation alignment \(App\.[E](https://arxiv.org/html/2609.19291#A5)\)\. Language mixing introduces multiple languages within a single input sequence, encouraging the model to utilize cross\-lingual information\. Our language mixing variant of choice is pretraining code\-switching \(named after linguistic code\-switching\), which randomly substitutes words with their cross\-lingual counterparts\([Zeng et al\., 2026](https://arxiv.org/html/2609.19291#bib.bib1), e\.g\.\)\. We use a comprehensive word\-level dictionary for the substitution operation, and sweep both the fraction of mixed documents and the within\-document substitution rate, finding no substantial benefit, with aCLeqscore of 2\.2%\. Activation alignment uses auxiliary losses to align pooled activations of semantically equivalent cross\-lingual units\. We experiment with InfoNCE\([van den Oord et al\., 2018](https://arxiv.org/html/2609.19291#bib.bib8);[Chi et al\., 2021](https://arxiv.org/html/2609.19291#bib.bib7)\)and L2 loss estimating sequence matches by chunking parallel documents\. Across various chunk sizes we find activation alignment offers no help in resolving compartmentalization:CLeqscore of 2\.5%\. Taken together, our results establish that knowledge compartmentalization emerges already in pretraining, persists under standard interventions, and is attenuated but not resolved by scale, suggesting that a fundamental solution should address the breakdown at this stage\. What causes this breakdown remains open: natural language pairs differ in script, grammar, segmentation, token vocabulary, and data distribution at once, so these measurements alone cannot say which difference is responsible\. ## 5 Disjoint Token Space is a Root Cause of Compartmentalization In this section, we isolate disjoint token spaces from other confounding variables of multilingual settings using a clone\-languages setup \(§[5\.1](https://arxiv.org/html/2609.19291#S5.SS1)\), and find that it alone is sufficient to induce knowledge compartmentalization in pretraining \(§[5\.2](https://arxiv.org/html/2609.19291#S5.SS2)\)\. We quantify the amount of token space sharing needed to resolve it \(§[5\.3](https://arxiv.org/html/2609.19291#S5.SS3)\), and show that even clone\-languages shared initialization surprisingly struggles, although to a much lesser extent, to completely resolve compartmentalization \(§[5\.4](https://arxiv.org/html/2609.19291#S5.SS4)\)\. This section’s results are presented in Figure[2](https://arxiv.org/html/2609.19291#S5.F2)\. ### 5\.1 Isolating disjoint tokenization:English1–English2setup Figure 2:Disjoint token spaces are severely compartmentalized\.CLeqforEnglish1–English2pretraining under various methods\. Soft\-mapping percentages refer to the tied embedding portionpp\. “Same data” refers to pretraining the language copies on identical corpora\.Poor multilingual generalization is frequently attributed to varying syntax, grammar, cultural nuances, data quantity and quality, or tokenizer fragmentation\([Rust et al\., 2021](https://arxiv.org/html/2609.19291#bib.bib23);[Petrov et al\., 2023](https://arxiv.org/html/2609.19291#bib.bib24);[Philippy et al\., 2023](https://arxiv.org/html/2609.19291#bib.bib25);[Lauscher et al\., 2020](https://arxiv.org/html/2609.19291#bib.bib26);[Hershcovich et al\., 2022](https://arxiv.org/html/2609.19291#bib.bib56);[Li et al\., 2024a](https://arxiv.org/html/2609.19291#bib.bib57)\)\. To strip away these confounding variables and isolate the pretraining process from linguistic aspects, we train on two copies of one, resource\-rich language, rather than on two different natural languages, which we name theEnglish1–English2pretraining setup\. By duplicating the model’s vocabulary, we create two structurally identical languages mapped to entirely disjoint token spaces\([K et al\., 2020](https://arxiv.org/html/2609.19291#bib.bib54)\)\. Formally, assuming a standard token vocabulary of sizeVV, any text is tokenized into a base sequenceT=\[t1,t2,…,tn\]T=\[t\_\{1\},t\_\{2\},\\dots,t\_\{n\}\]where each token indexti∈\{0,…,V−1\}t\_\{i\}\\in\\\{0,\\dots,V\-1\\\}\. We define the representations for our two language copies as perfectly parallel sequences shifted into mutually exclusive token spaces: English1\(T\)\\displaystyle\\text\{English\}\_\{1\}\(T\)=\[t1,t2,…,tn\]\\displaystyle=\[t\_\{1\},t\_\{2\},\\dots,t\_\{n\}\]ti∈\[0,V−1\]\\displaystyle t\_\{i\}\\in\[0,V\-1\]\(3\)English2\(T\)\\displaystyle\\text\{English\}\_\{2\}\(T\)=\[t1′,t2′,…,tn′\]\\displaystyle=\[t\_\{1\}^\{\\prime\},t\_\{2\}^\{\\prime\},\\dots,t\_\{n\}^\{\\prime\}\]ti′=ti\+V∈\[V,2V−1\]\\displaystyle t\_\{i\}^\{\\prime\}=t\_\{i\}\+V\\in\[V,2V\-1\]We double our model vocabulary to accommodate the disjoint token spaces\. Under this setup, the sole barrier to cross\-lingual generalization is the ability of the training process to bridge disjoint tokens\. ### 5\.2 Disjoint tokens alone suffice for compartmentalization Despite an identical underlying structure, we find severe knowledge compartmentalization between English1and English2:CLeqscore of−2\.6%\-2\.6\\%\. Scaling up doesn’t help, with a 7B\-parameter model trained in this setting yielding−1\.5%\-1\.5\\%\. Even pretraining on identical corpora \(except for the injected FKD a tiny fraction of training tokens\) does not offer any help \(−0\.1%\-0\.1\\%\); pretraining fails to generalize knowledge across 2 identical data distributions based on disjoint tokens alone\. We analyze the learned embedding spaces of English1and English2and find they are structurally similar and substantially alignable by a linear transformation, yet with no correspondence between matching tokens\. The model successfully learns the same underlying distribution independently in each space, yet fails to bridge them\. In contrast, models that do generalize learn to bridge the two token spaces, so that matching tokens have similar embeddings \(App\.[H](https://arxiv.org/html/2609.19291#A8)\)\. Together, this shows that structural similarity is not sufficient for knowledge generalization, which requires the model to learn to bridge the two token spaces, and that the standard pretraining process does not do so on its own\. ### 5\.3 How much sharing is needed? Disjoint Tokenization forms the barrier; we now quantify the barrier’s size when tokenization is only partially disjoint\. For this we introduce*soft\-mapping*, where for each token pairti,ti′t\_\{i\},t\_\{i\}^\{\\prime\}a fractionppof their embedding is tied and learned together\. Soft\-mapping experiments reveal a sharp threshold\. At 20% tying, generalization remains in the weak region withCLeq=8\.6%\\text\{CL\\textsubscript\{eq\}\}=8\.6\\%; at 50% tying it jumps to 82\.7%, and approaches 100% as tying becomes complete \(App\.[F](https://arxiv.org/html/2609.19291#A6)\)\. These results imply that a large amount of token sharing is required to induce knowledge generalization in pretraining\. We also experiment with vocabulary overlap, a subset of tokens completely shared between the 2 languages, and find that effectiveness is also dependent on the frequency of the shared tokens \(App\.[I](https://arxiv.org/html/2609.19291#A9)\)\. As a side effect, the results also confirm that our metric’s ceiling is reachable, so substantial shortfalls below it reflect genuine knowledge compartmentalization\. ### 5\.4 Pretraining drifts aligned tokens apart Token sharing is needed, but pretraining does not establish it on its own; we now test if pretraining can maintain and utilize a supplied matching\. We initialize each pair of parallel embeddings inEnglish1–English2to identical values — the same alignment that complete tying enforces \(§[5\.3](https://arxiv.org/html/2609.19291#S5.SS3)\), but provided as a starting point rather than a structural constraint\. We find that the training process fails to utilize these matches to their full extent, yielding aCLeqscore of 78\.6% — high, but substantially short of the reachable ceiling\. Identical data exposure remains short, with a score of 84\.6%\. This compounds the flaw established above: pretraining not only struggles to bridge disjoint tokens on its own, it also actively drives them apart when a matching is already provided\. Alignment between disjoint token spaces is thus not a stable state of pretraining but one it works against, rendering initialization\-based solutions to compartmentalization incomplete on their own\. Together, this section’s results establish disjoint token spaces as a fundamental barrier to knowledge generalization\. Pretraining neither converges to token sharing on its own, nor preserves it when supplied\. These findings motivate bypassing disjoint tokenization directly rather than aligning across it, but two questions remain\. Can token spaces be merged efficiently and practically? And while disjoint tokens are sufficient to cause compartmentalization, is removing them sufficient to resolve it in realistic settings, where many other variables are in play? ## 6 Unifying Token Spaces Reduces Compartmentalization Having established that disjoint token spaces are sufficient to induce knowledge compartmentalization, we now extend our mechanism diagnosis by introducing an efficient vocabulary mapping technique bypassing disjoint token spaces directly \(§[6\.1](https://arxiv.org/html/2609.19291#S6.SS1)\) and demonstrating it substantially improves generalization in the natural bilingual setting \(§[6\.2](https://arxiv.org/html/2609.19291#S6.SS2)\)\. We also verify the choice of a semantically meaningful mapping \(§[6\.4](https://arxiv.org/html/2609.19291#S6.SS4)\) and address practical aspects of this mapping strategy \(§[6\.3](https://arxiv.org/html/2609.19291#S6.SS3)\)\. ### 6\.1 Unifying token spaces via Word\-Wise Translation The WWT operation\.Word\-wise translation \(WWT\) maps text in languageAAto the token space of languageBBby replacing each word with its counterpart from a large\-scale invertible dictionary; out\-of\-dictionary words are transliterated via an invertible character\-level mapping \(dictionary curation and conflict resolution in App\.[G](https://arxiv.org/html/2609.19291#A7)\)\. Unlike full document translation, this mapping is context\-free and word\-wise, introducing minimal computational overhead\. It therefore can be applied and inverted on the fly during both pretraining and inference; users read and write native\-script text while the model trains and operates in the unified token space, making the intervention transparent at the interface\. WWT\-Ar\.We instantiate WWT for Arabic into English token space using an invertible Arabic\-to\-English dictionary, with the invertible Buckwalter transliteration as the out\-of\-dictionary fallback\. The result is WWT\-Ar — a new writing system for Arabic that retains every word choice, word order, and grammatical structure, while sharing a unified token space with English \(Table[2](https://arxiv.org/html/2609.19291#S6.T2)\)\. The dictionary covers 99\.7% of word occurrences in our corpus \(∼\\sim78% of unique word types\), so the mapped tokens are overwhelmingly translations rather than transliterations\. We also apply WWT to Russian, unifying its token space with English and creating WWT\-Ru\. Cost\.WWT increases sequence length: WWT\-Ar requires roughly 30% more tokens than native\-script Arabic to encode the same text \(App\.[G\.3](https://arxiv.org/html/2609.19291#A7.SS3)\), increasing the cost of LLM inference\. Under the fixed token budget of our experiments, this fertility overhead also means that a WWT\-Ar model is exposed to roughly 23% fewer Arabic documents than its native\-script counterpart, resulting in a data disadvantage relative to the models we compare it against\. We leave the overhead unhandled and address its origins and paths to reducing it in §[9](https://arxiv.org/html/2609.19291#S9)\. ### 6\.2 Unified token space improves knowledge generalization Table 1:An Arabic sentence, its WWT\-Ar rendering, and an English translation\. Arrows signal reading direction, color marks word origin\. WWT\-Ar maps every word into English tokens while preserving Arabic word order and grammar; it is Arabic in a different writing system, not a translation\.Table 2:Merging token spaces by applying WWT to the non\-English language raisesCLeqand improves English perplexity\. Mapping Arabic to WWT\-Ar yields substantial gains in knowledge generalization, raisingCLeqto 12\.6%, a14×14\\timesimprovement over theEnglish–Arabicbaseline at 360M\-parameter scale, and substantially higher than any intervention we test in that disjointly tokenized setting \(Fig\.[1](https://arxiv.org/html/2609.19291#S0.F1)\)\. Substantial gains appear across our tested settings: at 7B\-parameter scale, WWT boostsCLeqfrom 5\.9% to 12\.5%, in theEnglish–RussiansettingCLeqrises from 1\.9% to 23\.3% \(Tab\.[2](https://arxiv.org/html/2609.19291#S6.T2)\)\. We also experiment with partial token space overlap but find it to be ineffective \(App\.[I](https://arxiv.org/html/2609.19291#A9)\)\. Strikingly, even while training on substantially fewer Arabic documents, the WWT\-Ar model’s gain in knowledge generalization does not come at the expense of overall English language modeling, with perplexity improving from19\.5319\.53to18\.7418\.74\. The unified token space allows the model to extract more value from the cross\-lingual data it does process, actively driving better English language modeling\. Improvements appear across the tested settings \(Tab\.[2](https://arxiv.org/html/2609.19291#S6.T2)\)\. The knowledge generalization gain is asymmetric, withCLeqWWT\-Ar→English=17\.9%\\text\{CL\\textsubscript\{eq\}\}\_\{\\text\{WWT\-Ar\}\\rightarrow\\text\{English\}\}=17\.9\\%whileCLeqEnglish→WWT\-Ar=7\.3%\\text\{CL\\textsubscript\{eq\}\}\_\{\\text\{English\}\\rightarrow\\text\{WWT\-Ar\}\}=7\.3\\%\. Nonetheless, Arabic language modeling remains surprisingly strong \(App\.[J](https://arxiv.org/html/2609.19291#A10)\)\. The same asymmetry is present across our experimented scales and language\-pairs\. We attribute this to the one\-directional nature of WWT: WWT\-Ar’s vocabulary is entirely contained within English tokens, whereas much of English vocabulary never arises as a translation target and is therefore not part of WWT\-Ar’s vocabulary\. Crucially, WWT\-Ar is equivalent to Arabic, therefore, the gain cannot be attributed to WWT\-Ar being linguistically closer to English, because in every aspect except tokenization it is not\. We conclude that while linguistic differences might dampen cross\-lingual knowledge generalization in LLMs, disjoint token spaces are what gate it\. WWT produces substantial knowledge generalization between natural, vocabulary\-disjoint languages\. It does so without parallel corpora, auxiliary losses, or architectural changes: the intervention is a data\-level remapping, applied and inverted in real time\. Two limitations follow from fully merging the token spaces: the generation language must be fixed in advance, and code\-switched output is unsupported\. We address these next\. ### 6\.3 Restoring language identity — Soft Mapping WWT\-Ar tokens are indistinguishable from their English counterparts\. This is what enables knowledge sharing, but it also means the model has no mechanism for identifying which language it is producing, resulting in the above\-mentioned limitations\. Soft mapping relaxes the constraint\. Rather than merging tokens, WWT\-Ar retains distinct parallel to English token identities and shares a fractionppof its embedding dimensions with the corresponding English tokens, leaving the remainder language\-specific \(Appendix[F](https://arxiv.org/html/2609.19291#A6)\)\. Replacing WWT\-Ar with a highppsoft mapping version has a small cost on knowledge generalization with English, consistent with our earlier results on soft\-mapping[5\.3](https://arxiv.org/html/2609.19291#S5.SS3)\.CLeqscore increases withppand approaches the fully mapped WWT\-Ar, reachingCLeq=9\.60%\\text\{CL\\textsubscript\{eq\}\}=9\.60\\%atp=90%p=90\\%andCLeq=10\.47%\\text\{CL\\textsubscript\{eq\}\}=10\.47\\%atp=99%p=99\\%\(76% and 83% of the 12\.6% obtained under full mapping\)\. A small number of reserved dimensions therefore preserves most of the knowledge generalization while restoring a native mechanism for language identity, which can even be extended to a multilingual setting, and resolving the limitations of WWT\-Ar\. ### 6\.4 Semantics matter for token space merging We test whether the key to our previously introduced setup, mapping Arabic to WWT\-Ar, facilitating knowledge generalization lies in token sharing alone, or in the mapping being semantically faithful — each Arabic word mapped to an English word with the same meaning\. We evaluate three controls that share tokens without sharing meaning\. First, we randomly shuffle the WWT\-Ar dictionary, preserving the shared token inventory while destroying the word\-level correspondence:CLeqcollapses from 12\.6% to 3\.0% \. Second, we map Arabic into English script by transliteration alone\. This control is notable because character\-level romanization is itself a proposed method for unlocking multilingual capabilities\([Moosa et al\., 2023](https://arxiv.org/html/2609.19291#bib.bib39);[J et al\., 2024](https://arxiv.org/html/2609.19291#bib.bib63);[Saji et al\., 2025](https://arxiv.org/html/2609.19291#bib.bib65)\): it shares a token inventory without sharing token semantics, and we find that under itCLeqdrops to−0\.3\-0\.3%\. Third, returning to the confound\-freeEnglish1–English2setting, we let both copies share the same token space but permute token identities such that the same token index now corresponds to two different subwords:CLeqis−2\.6\-2\.6% — same as fully disjoint tokens\. Implementation and details for these setups are in App\.[K](https://arxiv.org/html/2609.19291#A11) Resolving disjoint tokenization is thus not a matter of token superposition: what knowledge generalization requires is not a shared token inventory but shared*meanings*of tokens\. Prior findings that transliteration helps mainly between related languages\([J et al\., 2024](https://arxiv.org/html/2609.19291#bib.bib63);[Jayakumar et al\., 2026](https://arxiv.org/html/2609.19291#bib.bib72)\)are what our mechanism predicts: related languages supply semantic matches through cognates, distant ones do not, making our semantic WWT mapping necessary\. ## 7 Related Work The Cross\-Lingual Knowledge Gap\.While LLMs easily acquire linguistic competence from modest data budgets\([Charpentier et al\., 2025](https://arxiv.org/html/2609.19291#bib.bib40)\), they struggle to transfer mathematical reasoning, coding, reading comprehension, and other learned skills outside their primary training language\([Shi et al\., 2023](https://arxiv.org/html/2609.19291#bib.bib41);[Peng et al\., 2024](https://arxiv.org/html/2609.19291#bib.bib42);[Bandarkar et al\., 2024](https://arxiv.org/html/2609.19291#bib.bib43);[Hu et al\., 2020](https://arxiv.org/html/2609.19291#bib.bib44);[Cheng et al\., 2026](https://arxiv.org/html/2609.19291#bib.bib74)\)\. This generalization failure has been shown to extend into the knowledge domain\([Goldman et al\., 2025](https://arxiv.org/html/2609.19291#bib.bib29);[Chang et al\., 2026](https://arxiv.org/html/2609.19291#bib.bib35);[Guo et al\., 2026](https://arxiv.org/html/2609.19291#bib.bib46)\), and widespread cross\-lingual factual inconsistencies have been documented\([Qi et al\., 2023](https://arxiv.org/html/2609.19291#bib.bib31);[Ifergan et al\., 2025](https://arxiv.org/html/2609.19291#bib.bib30)\)\. Relatedly,[Calderon et al\. \(2026\)](https://arxiv.org/html/2609.19291#bib.bib73)distinguish missing knowledge from encoded but inaccessible knowledge, finding recall rather than encoding to be the bottleneck\. Yet given no definitive answer on where these knowledge gaps originate within the training process, most attempts to mitigate them intervene post\-training via prompting and pre\-translation\([Huang et al\., 2023](https://arxiv.org/html/2609.19291#bib.bib33);[Wang et al\., 2025b](https://arxiv.org/html/2609.19291#bib.bib47);[Mondshine et al\., 2025](https://arxiv.org/html/2609.19291#bib.bib67);[Diskind et al\., 2026](https://arxiv.org/html/2609.19291#bib.bib34)\), knowledge editing\([Beniwal et al\., 2024](https://arxiv.org/html/2609.19291#bib.bib59)\), activation manipulation\([Wang et al\., 2025c](https://arxiv.org/html/2609.19291#bib.bib51);[Wang et al\., 2025a](https://arxiv.org/html/2609.19291#bib.bib50);[Manev, 2026](https://arxiv.org/html/2609.19291#bib.bib48)\), fine\-tuning\([Liu et al\., 2026](https://arxiv.org/html/2609.19291#bib.bib38);[Chua et al\., 2025](https://arxiv.org/html/2609.19291#bib.bib45)\)or reinforcment learning\([von Rad et al\., 2026b](https://arxiv.org/html/2609.19291#bib.bib49)\)\. We instead investigate knowledge generalization in pretraining, where multilingual limitations have been identified before, but in aggregated terms such as model capacity and perplexity\([Chang et al\., 2024b](https://arxiv.org/html/2609.19291#bib.bib18);[Howe and Wingate, 2026](https://arxiv.org/html/2609.19291#bib.bib20)\)\.[Liu et al\. \(2025\)](https://arxiv.org/html/2609.19291#bib.bib75)performed an observational study of multilingual knowledge acquisition on pretraining checkpoints, finding native frequency is the primary driver and characterizing cross\-lingual transfer as a second\-order effect dependent on script, but their confounded observational setting limits further isolation and understanding of it\.[Li et al\. \(2024b\)](https://arxiv.org/html/2609.19291#bib.bib28)explored cross\-lingual failures in pretraining, including knowledge gaps which they find narrow with scale, concluding that alignment emerges spontaneously in larger models\. Our controlled measurements challenge this: scaling from 360M to 7B attenuates compartmentalization but leaves the large majority of cross\-lingual value unrecovered, and in the confound\-freeEnglish1–English2setting scale provides no improvement at all \(§[5\.2](https://arxiv.org/html/2609.19291#S5.SS2)\)\. We therefore argue that compartmentalization is established in pretraining rather than dissolved by scale; we attribute this tension to our metric normalizing overall learning ability rather than reporting absolute accuracy \(§[2\.2](https://arxiv.org/html/2609.19291#S2.SS2.SSS0.Px1)\)\. Tokenization as a Generalization Barrier\.Typological distance and language family influence transfer\([Lauscher et al\., 2020](https://arxiv.org/html/2609.19291#bib.bib26)\), but script match is the primary predictor of whether parametric knowledge successfully crosses language boundaries\([Ifergan et al\., 2025](https://arxiv.org/html/2609.19291#bib.bib30);[Bandarkar et al\., 2026](https://arxiv.org/html/2609.19291#bib.bib10)\)\. Script reaches the model through tokenization, in two ways\. Through fragmentation: non\-Latin scripts suffer from over\-segmentation\([Rust et al\., 2021](https://arxiv.org/html/2609.19291#bib.bib23);[Petrov et al\., 2023](https://arxiv.org/html/2609.19291#bib.bib24)\), and vocabulary overlap:[Kallini et al\. \(2025\)](https://arxiv.org/html/2609.19291#bib.bib36)find that overlap, and especially natural semantic matches, aids generalization\. Following on this, existing methods transliterate or merge subwords\([Moosa et al\., 2023](https://arxiv.org/html/2609.19291#bib.bib39);[Zhang et al\., 2025](https://arxiv.org/html/2609.19291#bib.bib52);[Patil et al\., 2022](https://arxiv.org/html/2609.19291#bib.bib53);[J et al\., 2024](https://arxiv.org/html/2609.19291#bib.bib63);[Liu et al\., 2024](https://arxiv.org/html/2609.19291#bib.bib64)\), but their success is limited to closely related languages\. Countering both,[Conneau et al\. \(2020b\)](https://arxiv.org/html/2609.19291#bib.bib60);[Artetxe et al\. \(2020\)](https://arxiv.org/html/2609.19291#bib.bib58), and[Feng et al\. \(2025\)](https://arxiv.org/html/2609.19291#bib.bib71)show cross\-lingual structures emerge without any shared subwords\. We reconcile this: structural alignment does emerge without shared tokens, but knowledge does not follow it \(§[5\.2](https://arxiv.org/html/2609.19291#S5.SS2)\)\. Our work corroborates findings on the significance of script by establishing shared token space as the mechanism through which it acts and extends this by effectively applying this principle to distant language pairs\. Pretraining Causal Analysis Tools\.Analyzing isolated aspects of multilingual pretraining is confounded by lexical and tokenization\-related factors\. Prior work introduces synthetic duplicated vocabularies as a simplified test\-bed for multilingual analysis\([K et al\., 2020](https://arxiv.org/html/2609.19291#bib.bib54);[Dufter and Schütze, 2020](https://arxiv.org/html/2609.19291#bib.bib62);[Schäfer et al\., 2024](https://arxiv.org/html/2609.19291#bib.bib5);[Howe and Wingate, 2026](https://arxiv.org/html/2609.19291#bib.bib20)\)\. A separate line of work studies the dynamics of factual acquisition; this requires knowing exact exposure counts, which is intractable in natural corpora and is therefore estimated\([Kandpal et al\., 2023](https://arxiv.org/html/2609.19291#bib.bib3);[Elazar et al\., 2024](https://arxiv.org/html/2609.19291#bib.bib4)\)\. A more controlled yet synthetic approach addresses this by tracking acquisition per exposure to fictive facts injected during training\([Allen\-Zhu and Li, 2024](https://arxiv.org/html/2609.19291#bib.bib22);[Chang et al\., 2024a](https://arxiv.org/html/2609.19291#bib.bib55)\)\. We unify these approaches: we use a cloned\-vocabulary testbed, paired with fictive fact injection at known exposure rates, allowing us to measure the effect of isolated interventions and mechanisms on cross\-lingual knowledge transfer\. ## 8 Conclusions Our results converge on a single claim: cross\-lingual knowledge compartmentalization originates in pretraining and is in substantial part a tokenization phenomenon\. It appears even between languages that differ in nothing but their token inventory, and it is substantially reduced by an intervention that changes nothing but the tokens\. This origin matters for how compartmentalization should be addressed\. Most mitigations act after pretraining, restoring access to knowledge the model already holds\. They cannot recover what compartmentalization has already cost: a model that learns the same fact separately in each language has spent capacity and compute storing it twice, and output consistency does not undo that duplication\. Tokenization is a substantial part of the story, but not all of it\. While token space merging provides significant gains in knowledge generalization, it is far from resolving compartmentalization completely \(§[5\.3](https://arxiv.org/html/2609.19291#S5.SS3)\)\. The residual gap plausibly reflects imperfections of our mapping — transliterated out\-of\-dictionary words and naive conflict\-resolving — and the linguistic differences WWT leaves untouched by construction\. We leave understanding these factors, possibly utilizing our controlled environment, and improving upon them for future work\. If disjoint token spaces are the barrier, a natural question is whether tokenizer\-free architectures dissolve it by construction\([Xue et al\., 2022](https://arxiv.org/html/2609.19291#bib.bib66);[Hwang et al\., 2025](https://arxiv.org/html/2609.19291#bib.bib2)\)\. Our results do not settle this: a shared byte\-level interface removes disjoint token identities, but whether knowledge then generalizes across scripts remains untested, andCLeqoffers a direct way to test it\. Ultimately, cross\-lingual compartmentalization represents a fundamental flaw in LLM pretraining\. Our embedding analysis \(§[5\.2](https://arxiv.org/html/2609.19291#S5.SS2)\) shows that models successfully learn identical underlying distributions independently, yet completely fail to bridge them — structural similarity is not sufficient for knowledge generalization\. This failure has implications beyond text, directly challenging the assumption that multimodal systems will naturally generalize across parallel representations of the same concepts\([Huh et al\., 2024](https://arxiv.org/html/2609.19291#bib.bib69)\)\. Bridging disjoint interfaces, like languages, is a critical step toward disentangling underlying knowledge from its surface form, and moving toward models that learn unified representations of intelligence rather than isolated islands of modality\. ## 9 Limitations Corpus Artifacts\.Our Arabic pretraining corpus is machine\-translated from English\. This risks introducing “translationese”: structural and syntactic artifacts that make the Arabic text artificially resemble English grammar, potentially inflating the baseline structural alignment\([Doshi et al\., 2024](https://arxiv.org/html/2609.19291#bib.bib61)\)\. We mostly mitigate this by replicating our core compartmentalization and token\-space unification findings on native Russian data\. Linear Measurement of Non\-Linear Learning\.TheCLeqscore is estimated using a linear approximation\. This sits in tension with the log\-like shape of the knowledge acquisition curve \(App\.[B\.5](https://arxiv.org/html/2609.19291#A2.SS5)\)\. While we address this choice in §[2\.2](https://arxiv.org/html/2609.19291#S2.SS2.SSS0.Px1), it inherently oversimplifies the true dynamics of memorization\. Our FKD dataset is limited in complexity — relatively simple entity–attribute facts — and size — 2048 facts\. The latter limitation results in measurement noise, discussed extensively in App\.[D](https://arxiv.org/html/2609.19291#A4), where we also provide significance measurements of our headline comparisons\. Inference Overhead from Token Fertility\.While the WWT mapping operation itself is computationally cheap, it causes a substantial increase in sequence length, increasing the cost of inference\. We attribute this token fertility increase primarily to our conflict\-resolution strategy \(App\.[G](https://arxiv.org/html/2609.19291#A7)\)\. This overhead could potentially be mitigated by a more refined dictionary acquisition process or a better conflict resolution policy\. However, because the primary contribution of this mapping in our scope is mechanistic, we leave the engineering of a fully optimized mapping to future work\. Scalability to Massively Multilingual Settings\.We evaluate token space merging in a bilingual context\. We do not address the capacity limits of a single token space, leaving open the question of how many languages can be mapped into the English token space and how those would interfere\. ## References - L\. B\. Allal, A\. Lozhkov, E\. Bakouch, G\. M\. Blázquez, G\. Penedo, L\. Tunstall, A\. Marafioti, H\. Kydlíček, A\. P\. Lajarín, V\. Srivastav, J\. Lochner, C\. Fahlgren, X\. Nguyen, C\. Fourrier, B\. Burtenshaw, H\. Larcher, H\. Zhao, C\. Zakka, M\. Morlon, C\. Raffel, L\. von Werra, and T\. WolfSmolLM2: when smol goes big – data\-centric training of a small language model\.External Links:2502\.02737,[Link](https://arxiv.org/abs/2502.02737)Cited by:[§A\.1](https://arxiv.org/html/2609.19291#A1.SS1.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2609.19291#S3.SS1.p1.1)\. - Allen\-Zhu and Li \(2024\)Z\. Allen\-Zhu and Y\. LiPhysics of language models: part 3\.1, knowledge storage and extraction\.InForty\-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21\-27, 2024,External Links:2309\.14316,[Link](https://openreview.net/forum?id=5x788rqbcj)Cited by:[§B\.2](https://arxiv.org/html/2609.19291#A2.SS2.p2.1),[§7](https://arxiv.org/html/2609.19291#S7.p3.1)\. - Allen\-Zhu and Li \(2025\)Z\. Allen\-Zhu and Y\. LiPhysics of language models: part 3\.3, knowledge capacity scaling laws\.InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24\-28, 2025,External Links:2404\.05405,[Link](https://openreview.net/forum?id=FxNNiUgtfa)Cited by:[§B\.5](https://arxiv.org/html/2609.19291#A2.SS5.p3.1)\. - Alrashedet al\.\(2024\)S\. Alrashed, D\. Khizbullin, and D\. R\. PughFineweb\-edu\-ar: machine\-translated corpus to support arabic small language models\.External Links:2411\.06402,[Link](https://arxiv.org/abs/2411.06402)Cited by:[§3\.1](https://arxiv.org/html/2609.19291#S3.SS1.p2.1)\. - Artetxeet al\.\(2020\)M\. Artetxe, S\. Ruder, and D\. YogatamaOn the cross\-lingual transferability of monolingual representations\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 4623–4637\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.421),[Link](https://aclanthology.org/2020.acl-main.421/)Cited by:[§7](https://arxiv.org/html/2609.19291#S7.p2.1)\. - Bandarkaret al\.\(2026\)L\. Bandarkar, A\. Ansell, and T\. CohnLarge reasoning models struggle to transfer parametric knowledge across scripts\.External Links:2603\.17070,[Link](https://arxiv.org/abs/2603.17070)Cited by:[§B\.1](https://arxiv.org/html/2609.19291#A2.SS1.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2609.19291#S1.p2.1),[§7](https://arxiv.org/html/2609.19291#S7.p2.1)\. - Bandarkaret al\.\(2024\)L\. Bandarkar, D\. Liang, B\. Muller, M\. Artetxe, S\. N\. Shukla, D\. Husa, N\. Goyal, A\. Krishnan, L\. Zettlemoyer, and M\. KhabsaThe belebele benchmark: a parallel reading comprehension dataset in 122 language variants\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 749–775\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.44),[Link](https://aclanthology.org/2024.acl-long.44/)Cited by:[§7](https://arxiv.org/html/2609.19291#S7.p1.1)\. - Beniwalet al\.\(2024\)H\. Beniwal, K\. D, and M\. SinghCross\-lingual editing in multilingual language models\.InFindings of the Association for Computational Linguistics: EACL 2024,Y\. Graham and M\. Purver \(Eds\.\),St\. Julian’s, Malta,pp\. 2078–2128\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-eacl.140),[Link](https://aclanthology.org/2024.findings-eacl.140/)Cited by:[§7](https://arxiv.org/html/2609.19291#S7.p1.1)\. - Calderonet al\.\(2026\)N\. Calderon, E\. Ben\-David, Z\. Gekhman, E\. Ofek, and G\. YonaEmpty shelves or lost keys? recall is the bottleneck for parametric factuality\.External Links:2602\.14080,[Link](https://arxiv.org/abs/2602.14080)Cited by:[§7](https://arxiv.org/html/2609.19291#S7.p1.1)\. - Changet al\.\(2024a\)H\. Chang, J\. Park, S\. Ye, S\. Yang, Y\. Seo, D\. Chang, and M\. SeoHow do large language models acquire factual knowledge during pretraining?\.InAdvances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 \- 15, 2024,A\. Globersons, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. M\. Tomczak, and C\. Zhang \(Eds\.\),External Links:2406\.11813,[Link](http://papers.nips.cc/paper/_files/paper/2024/hash/6fdf57c71bc1f1ee29014b8dc52e723f-Abstract-Conference.html)Cited by:[§7](https://arxiv.org/html/2609.19291#S7.p3.1)\. - Changet al\.\(2026\)T\. A\. Chang, C\. Arnett, A\. Sadallah, A\. Eldesokey, A\. Kashar, A\. Daud, A\. G\. Olanihun, A\. L\. Mohammed, A\. Praise, A\. M\. Sharma, A\. Gupta, A\. P\. Merin, A\. Bremang, A\. Iyigun, A\. Simplício, A\. Essouaied, A\. Chorana, A\. Eppa, A\. Oladipo, A\. Kuri, A\. Ramesh, A\. Dorkin, A\. M\. Kondoro, A\. F\. Aji, A\. E\. Çetintaş, A\. Hanbury, A\. Dembele, A\. Niksarli, Á\. Arroyo, A\. Bajand, A\. Khanna, A\. Chkhaidze, A\. C\. Condez, A\. Hartl, A\. Mkhonto, A\. Hoblitzell, A\. Tran, A\. Poulis, A\. Majumder, A\. Chaudhary, A\. Vacalopoulou, A\. K\. K\. Wong, A\. Simonsen, A\. Kovalev, A\. Nayak, A\. S, A\. Lana, A\. Purwarianti, B\. Alhafni, B\. Busole, B\. Ghanem, B\. Nathani, B\. S\. Đurić, B\. Ogundipe, B\. Agbonile, B\. Bergsson, B\. T\. Fischer, B\. Tutar, B\. Çınar, C\. Kane, C\. Udomcharoenchaikit, C\. Helwe, C\. R\. Nerella, C\. C\. Liu, C\. Nwokolo, C\. Homan, C\. Sampebgo, C\. España\-Bonet, C\. Amol, D\. Lee, D\. S\. Smart, D\. Arad, D\. Dzenhaliou, D\. Choi, D\. Liu, D\. Semedo, D\. Anugraha, D\. Popoola, D\. Mataciunas, D\. Nyaboke, D\. Owusu, D\. K\. Kumar, D\. Tavares, D\. Glória\-Silva, D\. Goyal, D\. Lee, E\. K\. Buchanan, E\. N\. Anajemba, E\. N\. Grace, E\. Mickel, E\. Herranen, E\. Acharya, E\. Nisar, E\. Anand, E\. Habumuremyi, E\. M\. Ajiboye, E\. P\. Yulianrifat, E\. Adenuga, E\. Rudnicka, F\. Itiola, F\. T\. Butt, F\. F\. Sheikh, F\. Thekkekara, F\. Haouari, F\. Nsengiyumva, F\. A\. Ilasariya, F\. A\. Tjiaranata, F\. Laakom, F\. Grasso, F\. Periti, F\. Orabona, G\. K\. Solomon, G\. I\. Winata, G\. N\. Ngo, G\. Udhedhe\-oze, G\. Vinagre, G\. N\. S\. R\. Challagolla, G\. Urbizu\-Garmendia, G\. Vadithya, G\. Son, G\. Abdykadyrova, G\. S\. Mohapatra, H\. Ullah, H\. Einarsson, H\. Hu, H\. Saffari, H\. Zaidi, H\. Zhang, H\. A\. Shairah, H\. Vuong, H\. Kuulmets, H\. L\. Patel, H\. Bouamor, H\. Yu, I\. N\. Debess, İ\. E\. Deveci, I\. A\. Hanif, I\. Cho, I\. Vieira, I\. Calvo, I\. Manzi, I\. I\. Salifou, I\. Daud, I\. Yusuf, I\. Itzhak, I\. Zhelyazkov, I\. Belashkin, I\. Spada, J\. Brinton, J\. Isbarov, J\. Čibej, J\. Kocoń, J\. Cuhel, J\. Krito, J\. Purbey, J\. Za, J\. Mickel, J\. Kunz, J\. Ratovondranto, J\. Varsha, J\. Jeong, J\. T\. Dávalos, J\. Lee, J\. Magalhães, J\. S\. K\. Yi, J\. Kim, J\. Chataignon, J\. M\. Imperial, J\. Thevakumar, J\. Land, J\. Alekseenko, J\. Jiang, J\. Kim, K\. Sirts, K\. R, K\. V, K\. Tshinu, K\. Kukk, K\. Ponkshe, K\. Huseynova, K\. He, K\. Enevoldsen, K\. J\. Alvarez, K\. Zaman, K\. Mrini, K\. Kyars, K\. Gour, K\. Lainitha, K\. Kruusmaa, K\. Mukherjee, K\. Chouhan, L\. Castro, L\. M\. Porrino\-Moscoso, L\. S\. Z\. Nzambi, L\. Choshen, L\. Sencan, L\. Øvrelid, L\. Alazraki, L\. O\. Jones, L\. Ehimen\-Ugbede, L\. Thevakumar, L\. Thavarasa, M\. Malik, M\. K\. Keita, M\. Jangid, M\. D\. Santis, M\. Garcia, M\. Šuppa, M\. D’Ciofalo, M\. Ojastu, M\. Attaullah, M\. Sikander, M\. Narayan, M\. Skandalis, M\. Mehak, M\. İ\. Bozkurt, M\. Bayu, M\. Velayuthan, M\. Vizo, M\. Leventhal, M\. Marcińczuk, M\. Almasi, M\. Potočnjak, M\. Bangera, M\. Shafiei, M\. Ansari, M\. Sharma, M\. Indoria, M\. U\. Rehman, M\. R\. S\. Habibi, M\. Kolić, M\. B\. Kınay, N\. Galant, N\. S\. Rathore, N\. Permpredanun, N\. Maugin, N\. Norman, N\. K\. Corrêa, N\. Ljubešić, N\. Thomas, N\. de Silva, N\. Joshi, N\. Ponkshe, N\. Habash, N\. Udeze, N\. Thomas, N\. Ligeti\-Nagy, N\. Coulibaly, O\. Ogundepo, O\. K\. Buliaminu, O\. G\. Fejiro, O\. God’spraise, O\. Samuel, O\. D\. Oluwaseun, O\. Akindejoye, O\. Snissarenko, O\. A\. Chiemezie, O\. Kınay, O\. Tursun, O\. O\. Joshua, O\. Fiyinfoluwa, P\. Rodríguez, P\. Gamallo, P\. Arora, P\. Valente, P\. Rupnik, P\. O\. Ekiugbo, P\. Agarwal, P\. Sahoo, P\. Prokopidis, P\. Niau\-Puhipau, Q\. Yahya, R\. Mignone, R\. Singhal, R\. Raja, R\. M\. R\. Kadiyala, R\. Merx, R\. Larsen, R\. Rajalakshmi, R\. Ghosh, R\. Oji, R\. K\. Solis, R\. Guerra, R\. Zawar, S\. N\. Bashir, S\. Alzaabi, S\. Sandeep, S\. P\. Batchu, S\. S\. Kantareddy, S\. Muzammil, S\. Z\. Pranida, S\. Buchanan, S\. Rutunda, S\. Land, S\. Sulollari, S\. Ali, S\. Sapkota, S\. Kengatharaiyer, S\. Tautvaisas, S\. Sen, S\. Banerjee, S\. Diarra, S\. Afolayan, S\. M, S\. Lee, S\. Shah, S\. Venkitachalam, S\. Djurabaeva, S\. Ibejih, S\. S\. Dutta, S\. Gupta, S\. P\. Suárez, S\. Ahmadi, S\. Sukumar, S\. Song, S\. A, S\. Sofianopoulos, S\. E\. Simon, S\. Benčina, S\. Gvasalia, S\. More, S\. Dragazis, S\. Milosavljević, S\. P\. Kaufhold, S\. S, S\. Alrashed, S\. Ranathunga, T\. Someya, T\. K\. Pungeršek, T\. Haklay, T\. Jibril, T\. Aoyama, T\. Abashidze, T\. J\. D\. Cruz, T\. Blevins, T\. Nikas, T\. Idoko, T\. M\. Do, T\. Chubakov, T\. Munda, T\. Owoeye, T\. Gargiani, U\. Rathore, U\. Johannesen, U\. Ugwu, V\. A\. Putra, V\. B\. Kumar, V\. Arzt, V\. Konovalov, V\. Nedumpozhimana, V\. Ondrejova, V\. Horbik, V\. V\. R\. Kummitha, V\. Dinić, W\. Sewunetie, W\. Wu, X\. Zhao, Y\. Diarra, Y\. Nikankin, Y\. Mathur, Y\. Bagla, Y\. Bangera, Y\. Chen, Y\. Li, Y\. Xavier, Y\. Belinkov, Z\. Alyafeai, Z\. Batozargalova, Z\. Shan, Z\. R\. Tam, Z\. Tang, Z\. Nadova, B\. Abbasi, S\. Biderman, D\. Stap, D\. Ataman, F\. Schmidt, H\. Gonen, J\. Wang, and D\. I\. AdelaniGlobal piqa: evaluating commonsense reasoning across 100\+ languages and cultures\.External Links:2510\.24081,[Link](https://arxiv.org/abs/2510.24081)Cited by:[§7](https://arxiv.org/html/2609.19291#S7.p1.1)\. - Changet al\.\(2024b\)T\. A\. Chang, C\. Arnett, Z\. Tu, and B\. K\. BergenWhen is multilinguality a curse? language modeling for 250 high\- and low\-resource languages\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 4074–4096\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.236),2311\.09205,[Link](https://aclanthology.org/2024.emnlp-main.236/)Cited by:[§1](https://arxiv.org/html/2609.19291#S1.p2.1),[§7](https://arxiv.org/html/2609.19291#S7.p1.1)\. - Charpentieret al\.\(2025\)L\. Charpentier, L\. Choshen, R\. Cotterell, M\. O\. Gul, M\. Y\. Hu, J\. Liu, J\. Jumelet, T\. Linzen, A\. Mueller, C\. Ross, R\. S\. Shah, A\. Warstadt, E\. G\. Wilcox, and A\. WilliamsFindings of the third BabyLM challenge: accelerating language modeling research with cognitively plausible data\.InProceedings of the First BabyLM Workshop,L\. Charpentier, L\. Choshen, R\. Cotterell, M\. O\. Gul, M\. Y\. Hu, J\. Liu, J\. Jumelet, T\. Linzen, A\. Mueller, C\. Ross, R\. S\. Shah, A\. Warstadt, E\. G\. Wilcox, and A\. Williams \(Eds\.\),Suzhou, China,pp\. 399–420\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.babylm-main.28),ISBN TODO,[Link](https://aclanthology.org/2025.babylm-main.28/)Cited by:[§7](https://arxiv.org/html/2609.19291#S7.p1.1)\. - Chenget al\.\(2026\)B\. Cheng, A\. Gaber, Z\. Liu, C\. Arnett, O\. Goldman, C\. Tan, and L\. ChoshenSkill issue: are skills language\-invariant in llms?\.Cited by:[§7](https://arxiv.org/html/2609.19291#S7.p1.1)\. - Chiet al\.\(2021\)Z\. Chi, L\. Dong, F\. Wei, N\. Yang, S\. Singhal, W\. Wang, X\. Song, X\. Mao, H\. Huang, and M\. ZhouInfoXLM: an information\-theoretic framework for cross\-lingual language model pre\-training\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,K\. Toutanova, A\. Rumshisky, L\. Zettlemoyer, D\. Hakkani\-Tur, I\. Beltagy, S\. Bethard, R\. Cotterell, T\. Chakraborty, and Y\. Zhou \(Eds\.\),Online,pp\. 3576–3588\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.naacl-main.280),2007\.07834,[Link](https://aclanthology.org/2021.naacl-main.280/)Cited by:[§E\.2](https://arxiv.org/html/2609.19291#A5.SS2.p2.1),[§4\.2](https://arxiv.org/html/2609.19291#S4.SS2.p3.1)\. - Chuaet al\.\(2025\)L\. Chua, B\. Ghazi, Y\. Huang, P\. Kamath, R\. Kumar, P\. Manurangsi, A\. Sinha, C\. Xie, and C\. ZhangCrosslingual capabilities and knowledge barriers in multilingual large language models\.External Links:2406\.16135,[Link](https://arxiv.org/abs/2406.16135)Cited by:[§7](https://arxiv.org/html/2609.19291#S7.p1.1)\. - Conneauet al\.\(2020a\)A\. Conneau, K\. Khandelwal, N\. Goyal, V\. Chaudhary, G\. Wenzek, F\. Guzmán, E\. Grave, M\. Ott, L\. Zettlemoyer, and V\. StoyanovUnsupervised cross\-lingual representation learning at scale\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 8440–8451\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.747),1911\.02116,[Link](https://aclanthology.org/2020.acl-main.747/)Cited by:[§4\.1](https://arxiv.org/html/2609.19291#S4.SS1.p2.1)\. - Conneauet al\.\(2020b\)A\. Conneau, S\. Wu, H\. Li, L\. Zettlemoyer, and V\. StoyanovEmerging cross\-lingual structure in pretrained language models\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 6022–6034\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.536),[Link](https://aclanthology.org/2020.acl-main.536/)Cited by:[§7](https://arxiv.org/html/2609.19291#S7.p2.1)\. - Diskindet al\.\(2026\)E\. Diskind, I\. Trainin, U\. Shaham, L\. Choshen, I\. Szpektor, and O\. AbendCross\-lingual exploration for parametric knowledge\.External Links:2606\.24579,[Link](https://arxiv.org/abs/2606.24579)Cited by:[§1](https://arxiv.org/html/2609.19291#S1.p2.1),[§7](https://arxiv.org/html/2609.19291#S7.p1.1)\. - Doshiet al\.\(2024\)M\. Doshi, R\. Dabre, and P\. BhattacharyyaPretraining language models using translationese\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 5843–5862\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.334),[Link](https://aclanthology.org/2024.emnlp-main.334/)Cited by:[§9](https://arxiv.org/html/2609.19291#S9.p1.1)\. - Dufter and Schütze \(2020\)P\. Dufter and H\. SchützeIdentifying elements essential for BERT’s multilinguality\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 4423–4437\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.358),[Link](https://aclanthology.org/2020.emnlp-main.358/)Cited by:[§7](https://arxiv.org/html/2609.19291#S7.p3.1)\. - Elazaret al\.\(2024\)Y\. Elazar, A\. Bhagia, I\. Magnusson, A\. Ravichander, D\. Schwenk, A\. Suhr, E\. P\. Walsh, D\. Groeneveld, L\. Soldaini, S\. Singh, H\. Hajishirzi, N\. A\. Smith, and J\. DodgeWhat’s in my big data?\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:2310\.20707,[Link](https://openreview.net/forum?id=RvfPnOkPV4)Cited by:[§2\.1](https://arxiv.org/html/2609.19291#S2.SS1.p1.1),[§7](https://arxiv.org/html/2609.19291#S7.p3.1)\. - Fenget al\.\(2025\)Z\. Feng, H\. Cao, W\. Xu, and T\. ZhaoWord\-level cross\-lingual structure in large language models\.InProceedings of the 31st International Conference on Computational Linguistics,O\. Rambow, L\. Wanner, M\. Apidianaki, H\. Al\-Khalifa, B\. D\. Eugenio, and S\. Schockaert \(Eds\.\),Abu Dhabi, UAE,pp\. 2026–2037\.External Links:[Link](https://aclanthology.org/2025.coling-main.138/)Cited by:[§2\.1](https://arxiv.org/html/2609.19291#S2.SS1.p2.1),[§7](https://arxiv.org/html/2609.19291#S7.p2.1)\. - Gaoet al\.\(2024\)L\. Gao, J\. Tow, B\. Abbasi, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golding, J\. Hsu, A\. Le Noac’h, H\. Li, K\. McDonell, N\. Muennighoff, C\. Ociepa, J\. Phang, L\. Reynolds, H\. Schoelkopf, A\. Skowron, L\. Sutawika, E\. Tang, A\. Thite, B\. Wang, K\. Wang, and A\. ZouThe language model evaluation harness\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.12608602),[Link](https://zenodo.org/records/12608602)Cited by:[§3\.2](https://arxiv.org/html/2609.19291#S3.SS2.p2.1)\. - Goldmanet al\.\(2025\)O\. Goldman, U\. Shaham, D\. Malkin, S\. Eiger, A\. Hassidim, Y\. Matias, J\. Maynez, A\. M\. Gilady, J\. Riesa, S\. Rijhwani, L\. Rimell, I\. Szpektor, R\. Tsarfaty, and M\. EyalECLeKTic: a novel challenge set for evaluation of cross\-lingual knowledge transfer\.External Links:2502\.21228,[Link](https://arxiv.org/abs/2502.21228)Cited by:[§1](https://arxiv.org/html/2609.19291#S1.p2.1),[§7](https://arxiv.org/html/2609.19291#S7.p1.1)\. - Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan, A\. Yang, A\. Fan, A\. Goyal, A\. Hartshorn, A\. Yang, A\. Mitra, A\. Sravankumar, A\. Korenev, A\. Hinsvark, A\. Rao, A\. Zhang, A\. Rodriguez, A\. Gregerson, A\. Spataru, B\. Roziere, B\. Biron, B\. Tang, B\. Chern, C\. Caucheteux, C\. Nayak, C\. Bi, C\. Marra, C\. McConnell, C\. Keller, C\. Touret, C\. Wu, C\. Wong, C\. C\. Ferrer, C\. Nikolaidis, D\. Allonsius, D\. Song, D\. Pintz, D\. Livshits, D\. Wyatt, D\. Esiobu, D\. Choudhary, D\. Mahajan, D\. Garcia\-Olano, D\. Perino, D\. Hupkes, E\. Lakomkin, E\. AlBadawy, E\. Lobanova, E\. Dinan, E\. M\. Smith, F\. Radenovic, F\. Guzmán, F\. Zhang, G\. Synnaeve, G\. Lee, G\. L\. Anderson, G\. Thattai, G\. Nail, G\. Mialon, G\. Pang, G\. Cucurell, H\. Nguyen, H\. Korevaar, H\. Xu, H\. Touvron, I\. Zarov, I\. A\. Ibarra, I\. Kloumann, I\. Misra, I\. Evtimov, J\. Zhang, J\. Copet, J\. Lee, J\. Geffert, J\. Vranes, J\. Park, J\. Mahadeokar, J\. Shah, J\. van der Linde, J\. Billock, J\. Hong, J\. Lee, J\. Fu, J\. Chi, J\. Huang, J\. Liu, J\. Wang, J\. Yu, J\. Bitton, J\. Spisak, J\. Park, J\. Rocca, J\. Johnstun, J\. Saxe, J\. Jia, K\. V\. Alwala, K\. Prasad, K\. Upasani, K\. Plawiak, K\. Li, K\. Heafield, K\. Stone, K\. El\-Arini, K\. Iyer, K\. Malik, K\. Chiu, K\. Bhalla, K\. Lakhotia, L\. Rantala\-Yeary, L\. van der Maaten, L\. Chen, L\. Tan, L\. Jenkins, L\. Martin, L\. Madaan, L\. Malo, L\. Blecher, L\. Landzaat, L\. de Oliveira, M\. Muzzi, M\. Pasupuleti, M\. Singh, M\. Paluri, M\. Kardas, M\. Tsimpoukelli, M\. Oldham, M\. Rita, M\. Pavlova, M\. Kambadur, M\. Lewis, M\. Si, M\. K\. Singh, M\. Hassan, N\. Goyal, N\. Torabi, N\. Bashlykov, N\. Bogoychev, N\. Chatterji, N\. Zhang, O\. Duchenne, O\. Çelebi, P\. Alrassy, P\. Zhang, P\. Li, P\. Vasic, P\. Weng, P\. Bhargava, P\. Dubal, P\. Krishnan, P\. S\. Koura, P\. Xu, Q\. He, Q\. Dong, R\. Srinivasan, R\. Ganapathy, R\. Calderer, R\. S\. Cabral, R\. Stojnic, R\. Raileanu, R\. Maheswari, R\. Girdhar, R\. Patel, R\. Sauvestre, R\. Polidoro, R\. Sumbaly, R\. Taylor, R\. Silva, R\. Hou, R\. Wang, S\. Hosseini, S\. Chennabasappa, S\. Singh, S\. Bell, S\. S\. Kim, S\. Edunov, S\. Nie, S\. Narang, S\. Raparthy, S\. Shen, S\. Wan, S\. Bhosale, S\. Zhang, S\. Vandenhende, S\. Batra, S\. Whitman, S\. Sootla, S\. Collot, S\. Gururangan, S\. Borodinsky, T\. Herman, T\. Fowler, T\. Sheasha, T\. Georgiou, T\. Scialom, T\. Speckbacher, T\. Mihaylov, T\. Xiao, U\. Karn, V\. Goswami, V\. Gupta, V\. Ramanathan, V\. Kerkez, V\. Gonguet, V\. Do, V\. Vogeti, V\. Albiero, V\. Petrovic, W\. Chu, W\. Xiong, W\. Fu, W\. Meers, X\. Martinet, X\. Wang, X\. Wang, X\. E\. Tan, X\. Xia, X\. Xie, X\. Jia, X\. Wang, Y\. Goldschlag, Y\. Gaur, Y\. Babaei, Y\. Wen, Y\. Song, Y\. Zhang, Y\. Li, Y\. Mao, Z\. D\. Coudert, Z\. Yan, Z\. Chen, Z\. Papakipos, A\. Singh, A\. Srivastava, A\. Jain, A\. Kelsey, A\. Shajnfeld, A\. Gangidi, A\. Victoria, A\. Goldstand, A\. Menon, A\. Sharma, A\. Boesenberg, A\. Baevski, A\. Feinstein, A\. Kallet, A\. Sangani, A\. Teo, A\. Yunus, A\. Lupu, A\. Alvarado, A\. Caples, A\. Gu, A\. Ho, A\. Poulton, A\. Ryan, A\. Ramchandani, A\. Dong, A\. Franco, A\. Goyal, A\. Saraf, A\. Chowdhury, A\. Gabriel, A\. Bharambe, A\. Eisenman, A\. Yazdan, B\. James, B\. Maurer, B\. Leonhardi, B\. Huang, B\. Loyd, B\. D\. Paola, B\. Paranjape, B\. Liu, B\. Wu, B\. Ni, B\. Hancock, B\. Wasti, B\. Spence, B\. Stojkovic, B\. Gamido, B\. Montalvo, C\. Parker, C\. Burton, C\. Mejia, C\. Liu, C\. Wang, C\. Kim, C\. Zhou, C\. Hu, C\. Chu, C\. Cai, C\. Tindal, C\. Feichtenhofer, C\. Gao, D\. Civin, D\. Beaty, D\. Kreymer, D\. Li, D\. Adkins, D\. Xu, D\. Testuggine, D\. David, D\. Parikh, D\. Liskovich, D\. Foss, D\. Wang, D\. Le, D\. Holland, E\. Dowling, E\. Jamil, E\. Montgomery, E\. Presani, E\. Hahn, E\. Wood, E\. Le, E\. Brinkman, E\. Arcaute, E\. Dunbar, E\. Smothers, F\. Sun, F\. Kreuk, F\. Tian, F\. Kokkinos, F\. Ozgenel, F\. Caggioni, F\. Kanayet, F\. Seide, G\. M\. Florez, G\. Schwarz, G\. Badeer, G\. Swee, G\. Halpern, G\. Herman, G\. Sizov, Guangyi, Zhang, G\. Lakshminarayanan, H\. Inan, H\. Shojanazeri, H\. Zou, H\. Wang, H\. Zha, H\. Habeeb, H\. Rudolph, H\. Suk, H\. Aspegren, H\. Goldman, H\. Zhan, I\. Damlaj, I\. Molybog, I\. Tufanov, I\. Leontiadis, I\. Veliche, I\. Gat, J\. Weissman, J\. Geboski, J\. Kohli, J\. Lam, J\. Asher, J\. Gaya, J\. Marcus, J\. Tang, J\. Chan, J\. Zhen, J\. Reizenstein, J\. Teboul, J\. Zhong, J\. Jin, J\. Yang, J\. Cummings, J\. Carvill, J\. Shepard, J\. McPhie, J\. Torres, J\. Ginsburg, J\. Wang, K\. Wu, K\. H\. U, K\. Saxena, K\. Khandelwal, K\. Zand, K\. Matosich, K\. Veeraraghavan, K\. Michelena, K\. Li, K\. Jagadeesh, K\. Huang, K\. Chawla, K\. Huang, L\. Chen, L\. Garg, L\. A, L\. Silva, L\. Bell, L\. Zhang, L\. Guo, L\. Yu, L\. Moshkovich, L\. Wehrstedt, M\. Khabsa, M\. Avalani, M\. Bhatt, M\. Mankus, M\. Hasson, M\. Lennie, M\. Reso, M\. Groshev, M\. Naumov, M\. Lathi, M\. Keneally, M\. Liu, M\. L\. Seltzer, M\. Valko, M\. Restrepo, M\. Patel, M\. Vyatskov, M\. Samvelyan, M\. Clark, M\. Macey, M\. Wang, M\. J\. Hermoso, M\. Metanat, M\. Rastegari, M\. Bansal, N\. Santhanam, N\. Parks, N\. White, N\. Bawa, N\. Singhal, N\. Egebo, N\. Usunier, N\. Mehta, N\. P\. Laptev, N\. Dong, N\. Cheng, O\. Chernoguz, O\. Hart, O\. Salpekar, O\. Kalinli, P\. Kent, P\. Parekh, P\. Saab, P\. Balaji, P\. Rittner, P\. Bontrager, P\. Roux, P\. Dollar, P\. Zvyagina, P\. Ratanchandani, P\. Yuvraj, Q\. Liang, R\. Alao, R\. Rodriguez, R\. Ayub, R\. Murthy, R\. Nayani, R\. Mitra, R\. Parthasarathy, R\. Li, R\. Hogan, R\. Battey, R\. Wang, R\. Howes, R\. Rinott, S\. Mehta, S\. Siby, S\. J\. Bondu, S\. Datta, S\. Chugh, S\. Hunt, S\. Dhillon, S\. Sidorov, S\. Pan, S\. Mahajan, S\. Verma, S\. Yamamoto, S\. Ramaswamy, S\. Lindsay, S\. Lindsay, S\. Feng, S\. Lin, S\. C\. Zha, S\. Patil, S\. Shankar, S\. Zhang, S\. Zhang, S\. Wang, S\. Agarwal, S\. Sajuyigbe, S\. Chintala, S\. Max, S\. Chen, S\. Kehoe, S\. Satterfield, S\. Govindaprasad, S\. Gupta, S\. Deng, S\. Cho, S\. Virk, S\. Subramanian, S\. Choudhury, S\. Goldman, T\. Remez, T\. Glaser, T\. Best, T\. Koehler, T\. Robinson, T\. Li, T\. Zhang, T\. Matthews, T\. Chou, T\. Shaked, V\. Vontimitta, V\. Ajayi, V\. Montanez, V\. Mohan, V\. S\. Kumar, V\. Mangla, V\. Ionescu, V\. Poenaru, V\. T\. Mihailescu, V\. Ivanov, W\. Li, W\. Wang, W\. Jiang, W\. Bouaziz, W\. Constable, X\. Tang, X\. Wu, X\. Wang, X\. Wu, X\. Gao, Y\. Kleinman, Y\. Chen, Y\. Hu, Y\. Jia, Y\. Qi, Y\. Li, Y\. Zhang, Y\. Zhang, Y\. Adi, Y\. Nam, Yu, Wang, Y\. Zhao, Y\. Hao, Y\. Qian, Y\. Li, Y\. He, Z\. Rait, Z\. DeVito, Z\. Rosnbrick, Z\. Wen, Z\. Yang, Z\. Zhao, and Z\. MaThe llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[§A\.1](https://arxiv.org/html/2609.19291#A1.SS1.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2609.19291#S3.SS1.p1.1)\. - Grögeret al\.\(2026\)F\. Gröger, S\. Wen, and M\. BrbićRevisiting the Platonic representation hypothesis: an Aristotelian view\.InProceedings of the 43rd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.306\.External Links:[Link](https://openreview.net/forum?id=uz0gAAYydl)Cited by:[§H\.1](https://arxiv.org/html/2609.19291#A8.SS1.p3.1)\. - Guoet al\.\(2026\)P\. Guo, Y\. Tsai, C\. Hsu, K\. Chen, Y\. A\. Tsai, K\. Chang, N\. Peng, M\. Yeh, and S\. LinLiveCLKTBench: towards reliable evaluation of cross\-lingual knowledge transfer in multilingual LLMs\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 15193–15208\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.694),2511\.14774,ISBN 979\-8\-89176\-390\-6,[Link](https://aclanthology.org/2026.acl-long.694/)Cited by:[§7](https://arxiv.org/html/2609.19291#S7.p1.1)\. - Hershcovichet al\.\(2022\)D\. Hershcovich, S\. Frank, H\. Lent, M\. de Lhoneux, M\. Abdou, S\. Brandl, E\. Bugliarello, L\. Cabello Piqueras, I\. Chalkidis, R\. Cui, C\. Fierro, K\. Margatina, P\. Rust, and A\. SøgaardChallenges and strategies in cross\-cultural NLP\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 6997–7013\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.acl-long.482),[Link](https://aclanthology.org/2022.acl-long.482/)Cited by:[§5\.1](https://arxiv.org/html/2609.19291#S5.SS1.p1.1)\. - Hoffmannet al\.\(2022\)J\. Hoffmann, S\. Borgeaud, A\. Mensch, E\. Buchatskaya, T\. Cai, E\. Rutherford, D\. de Las Casas, L\. A\. Hendricks, J\. Welbl, A\. Clark, T\. Hennigan, E\. Noland, K\. Millican, G\. van den Driessche, B\. Damoc, A\. Guy, S\. Osindero, K\. Simonyan, E\. Elsen, J\. W\. Rae, O\. Vinyals, and L\. SifreTraining compute\-optimal large language models\.External Links:2203\.15556,[Link](https://arxiv.org/abs/2203.15556)Cited by:[§A\.4](https://arxiv.org/html/2609.19291#A1.SS4.p1.1),[§3\.1](https://arxiv.org/html/2609.19291#S3.SS1.p1.1)\. - Howe and Wingate \(2026\)T\. V\. Howe and D\. WingateLanguage models struggle with compartmentalization\.External Links:2605\.19284,[Link](https://arxiv.org/abs/2605.19284)Cited by:[§1](https://arxiv.org/html/2609.19291#S1.p2.1),[§7](https://arxiv.org/html/2609.19291#S7.p1.1),[§7](https://arxiv.org/html/2609.19291#S7.p3.1)\. - Huet al\.\(2020\)J\. Hu, S\. Ruder, A\. Siddhant, G\. Neubig, O\. Firat, and M\. JohnsonXTREME: a massively multilingual multi\-task benchmark for evaluating cross\-lingual generalisation\.InProceedings of the 37th International Conference on Machine Learning,H\. D\. III and A\. Singh \(Eds\.\),Proceedings of Machine Learning Research, Vol\.119,pp\. 4411–4421\.External Links:[Link](https://proceedings.mlr.press/v119/hu20b.html)Cited by:[§7](https://arxiv.org/html/2609.19291#S7.p1.1)\. - Huanget al\.\(2023\)H\. Huang, T\. Tang, D\. Zhang, X\. Zhao, T\. Song, Y\. Xia, and F\. WeiNot all languages are created equal in LLMs: improving multilingual capability by cross\-lingual\-thought prompting\.InFindings of the Association for Computational Linguistics: EMNLP 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 12365–12394\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.826),[Link](https://aclanthology.org/2023.findings-emnlp.826/)Cited by:[§1](https://arxiv.org/html/2609.19291#S1.p2.1),[§7](https://arxiv.org/html/2609.19291#S7.p1.1)\. - Huhet al\.\(2024\)M\. Huh, B\. Cheung, T\. Wang, and P\. IsolaPosition: the Platonic representation hypothesis\.InProceedings of the 41st International Conference on Machine Learning,R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research, Vol\.235,pp\. 20617–20642\.External Links:[Link](https://proceedings.mlr.press/v235/huh24a.html)Cited by:[§H\.1](https://arxiv.org/html/2609.19291#A8.SS1.p2.1),[§8](https://arxiv.org/html/2609.19291#S8.p5.1)\. - Hwanget al\.\(2025\)S\. Hwang, B\. Wang, and A\. GuDynamic chunking for end\-to\-end hierarchical sequence modeling\.External Links:2507\.07955,[Link](https://arxiv.org/abs/2507.07955)Cited by:[§8](https://arxiv.org/html/2609.19291#S8.p4.1)\. - Iferganet al\.\(2025\)M\. Ifergan, L\. Choshen, R\. Aharoni, I\. Szpektor, and O\. AbendBeneath the surface of consistency: exploring cross\-lingual knowledge representation sharing in LLMs\.InFindings of the Association for Computational Linguistics: NAACL 2025,L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 4630–4644\.External Links:2408\.10646,ISBN 979\-8\-89176\-195\-7,[Link](https://aclanthology.org/2025.findings-naacl.475/)Cited by:[Appendix I](https://arxiv.org/html/2609.19291#A9.p1.1),[§1](https://arxiv.org/html/2609.19291#S1.p2.1),[§7](https://arxiv.org/html/2609.19291#S7.p1.1),[§7](https://arxiv.org/html/2609.19291#S7.p2.1)\. - Jet al\.\(2024\)J\. J, R\. Dabre, A\. M, J\. Gala, T\. Jayakumar, R\. Puduppully, and A\. KunchukuttanRomanSetu: efficiently unlocking multilingual capabilities of large language models via Romanization\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 15593–15615\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.833),[Link](https://aclanthology.org/2024.acl-long.833/)Cited by:[§K\.4](https://arxiv.org/html/2609.19291#A11.SS4.p1.1),[§6\.4](https://arxiv.org/html/2609.19291#S6.SS4.p1.1),[§6\.4](https://arxiv.org/html/2609.19291#S6.SS4.p2.1),[§7](https://arxiv.org/html/2609.19291#S7.p2.1)\. - Jayakumaret al\.\(2026\)T\. Jayakumar, D\. Halder, and R\. DabreScripts through time: a survey of the evolving role of transliteration in nlp\.External Links:2604\.18722,[Link](https://arxiv.org/abs/2604.18722)Cited by:[§K\.4](https://arxiv.org/html/2609.19291#A11.SS4.p1.1),[§6\.4](https://arxiv.org/html/2609.19291#S6.SS4.p2.1)\. - Ket al\.\(2020\)K\. K, Z\. Wang, S\. Mayhew, and D\. RothCross\-lingual ability of multilingual BERT: an empirical study\.In8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26\-30, 2020,External Links:1912\.07840,[Link](https://openreview.net/forum?id=HJeT3yrtDr)Cited by:[§5\.1](https://arxiv.org/html/2609.19291#S5.SS1.p1.1),[§7](https://arxiv.org/html/2609.19291#S7.p3.1)\. - Kalliniet al\.\(2025\)J\. Kallini, D\. Jurafsky, C\. Potts, and M\. BarteldsFalse Friends are not foes: investigating vocabulary overlap in multilingual language models\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 21138–21154\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.1153),2509\.18750,ISBN 979\-8\-89176\-335\-7,[Link](https://aclanthology.org/2025.findings-emnlp.1153/)Cited by:[Appendix I](https://arxiv.org/html/2609.19291#A9.p1.1),[§7](https://arxiv.org/html/2609.19291#S7.p2.1)\. - Kandpalet al\.\(2023\)N\. Kandpal, H\. Deng, A\. Roberts, E\. Wallace, and C\. RaffelLarge language models struggle to learn long\-tail knowledge\.InInternational Conference on Machine Learning, ICML 2023, 23\-29 July 2023, Honolulu, Hawaii, USA,A\. Krause, E\. Brunskill, K\. Cho, B\. Engelhardt, S\. Sabato, and J\. Scarlett \(Eds\.\),Proceedings of Machine Learning Research, Vol\.202,pp\. 15696–15707\.External Links:2211\.08411,[Link](https://proceedings.mlr.press/v202/kandpal23a.html)Cited by:[§B\.5](https://arxiv.org/html/2609.19291#A2.SS5.p3.1),[§2\.1](https://arxiv.org/html/2609.19291#S2.SS1.p1.1),[§7](https://arxiv.org/html/2609.19291#S7.p3.1)\. - Kornblithet al\.\(2019\)S\. Kornblith, M\. Norouzi, H\. Lee, and G\. HintonSimilarity of neural network representations revisited\.InProceedings of the 36th International Conference on Machine Learning,K\. Chaudhuri and R\. Salakhutdinov \(Eds\.\),Proceedings of Machine Learning Research, Vol\.97,pp\. 3519–3529\.External Links:[Link](https://proceedings.mlr.press/v97/kornblith19a.html)Cited by:[§H\.1](https://arxiv.org/html/2609.19291#A8.SS1.p2.1)\. - Lauscheret al\.\(2020\)A\. Lauscher, V\. Ravishankar, I\. Vulić, and G\. GlavašFrom zero to hero: On the limitations of zero\-shot language transfer with multilingual Transformers\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 4483–4499\.External Links:[Link](https://aclanthology.org/2020.emnlp-main.363/),[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.363)Cited by:[§5\.1](https://arxiv.org/html/2609.19291#S5.SS1.p1.1),[§7](https://arxiv.org/html/2609.19291#S7.p2.1)\. - Liet al\.\(2024a\)C\. Li, M\. Chen, J\. Wang, S\. Sitaram, and X\. XieCultureLLM: incorporating cultural differences into large language models\.InAdvances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 \- 15, 2024,A\. Globersons, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. M\. Tomczak, and C\. Zhang \(Eds\.\),External Links:2402\.10946,[Link](http://papers.nips.cc/paper/_files/paper/2024/hash/9a16935bf54c4af233e25d998b7f4a2c-Abstract-Conference.html)Cited by:[§5\.1](https://arxiv.org/html/2609.19291#S5.SS1.p1.1)\. - Liet al\.\(2024b\)J\. Li, S\. Huang, A\. Ching, X\. Dai, and J\. ChenPreAlign: boosting cross\-lingual transfer by early establishment of multilingual alignment\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 10246–10257\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.572),[Link](https://aclanthology.org/2024.emnlp-main.572/)Cited by:[§1](https://arxiv.org/html/2609.19291#S1.p2.1),[§7](https://arxiv.org/html/2609.19291#S7.p1.1)\. - Lianget al\.\(2025\)W\. Liang, T\. Liu, L\. Wright, W\. Constable, A\. Gu, C\. Huang, I\. Zhang, W\. Feng, H\. Huang, J\. Wang, S\. Purandare, G\. Nadathur, and S\. IdreosTorchTitan: one\-stop pytorch native solution for production ready LLM pretraining\.InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24\-28, 2025,External Links:2410\.06511,[Link](https://openreview.net/forum?id=SFN6Wm7YBI)Cited by:[Appendix A](https://arxiv.org/html/2609.19291#A1.p1.1),[§3\.1](https://arxiv.org/html/2609.19291#S3.SS1.p1.1)\. - Liuet al\.\(2026\)T\. Liu, J\. Qi, M\. Sachan, R\. Cotterell, R\. Fernández, and A\. BisazzaPost\-training language models for crosslingual consistency\.External Links:2603\.04678,[Link](https://arxiv.org/abs/2603.04678)Cited by:[§7](https://arxiv.org/html/2609.19291#S7.p1.1)\. - Liuet al\.\(2024\)Y\. Liu, C\. Ma, H\. Ye, and H\. SchuetzeTransliCo: a contrastive learning framework to address the script barrier in multilingual pretrained language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 2476–2499\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.136),[Link](https://aclanthology.org/2024.acl-long.136/)Cited by:[§7](https://arxiv.org/html/2609.19291#S7.p2.1)\. - Liuet al\.\(2025\)Y\. Liu, M\. Wang, A\. H\. Kargaran, F\. Körner, E\. Nie, B\. Plank, F\. Yvon, and H\. SchützeTracing multilingual factual knowledge acquisition in pretraining\.External Links:2505\.14824,[Link](https://arxiv.org/abs/2505.14824)Cited by:[§7](https://arxiv.org/html/2609.19291#S7.p1.1)\. - Loshchilov and Hutter \(2019\)I\. Loshchilov and F\. HutterDecoupled weight decay regularization\.In7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6\-9, 2019,External Links:1711\.05101,[Link](https://openreview.net/forum?id=Bkg6RiCqY7)Cited by:[§A\.3](https://arxiv.org/html/2609.19291#A1.SS3.p1.1)\. - Manev \(2026\)A\. ManevInference\-time steering for cross\-lingual factual consistency in llms\.External Links:2607\.19243,[Link](https://arxiv.org/abs/2607.19243)Cited by:[§7](https://arxiv.org/html/2609.19291#S7.p1.1)\. - Messmeret al\.\(2025\)B\. Messmer, V\. Sabolčec, and M\. JaggiEnhancing multilingual llm pretraining with model\-based data selection\.InAdvances in Neural Information Processing Systems,D\. Belgrave, C\. Zhang, H\. Lin, R\. Pascanu, P\. Koniusz, M\. Ghassemi, and N\. Chen \(Eds\.\),Vol\.38, Main Conference\.External Links:[Document](https://dx.doi.org/10.52202/085713-2109),[Link](https://proceedings.neurips.cc/paper_files/paper/2025/file/5aee125f052c90e326dcf6f380df94f6-Paper-Datasets_and_Benchmarks_Track.pdf)Cited by:[§3\.1](https://arxiv.org/html/2609.19291#S3.SS1.p2.1)\. - Mondshineet al\.\(2025\)I\. Mondshine, T\. Paz\-Argaman, and R\. TsarfatyBeyond English: the impact of prompt translation strategies across languages and tasks in multilingual LLMs\.InFindings of the Association for Computational Linguistics: NAACL 2025,L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 1331–1354\.External Links:[Link](https://aclanthology.org/2025.findings-naacl.73/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-naacl.73),ISBN 979\-8\-89176\-195\-7Cited by:[§7](https://arxiv.org/html/2609.19291#S7.p1.1)\. - Moosaet al\.\(2023\)I\. M\. Moosa, M\. E\. Akhter, and A\. B\. HabibDoes transliteration help multilingual language modeling?\.InFindings of the Association for Computational Linguistics: EACL 2023,A\. Vlachos and I\. Augenstein \(Eds\.\),Dubrovnik, Croatia,pp\. 670–685\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.findings-eacl.50),[Link](https://aclanthology.org/2023.findings-eacl.50/)Cited by:[§K\.4](https://arxiv.org/html/2609.19291#A11.SS4.p1.1),[§6\.4](https://arxiv.org/html/2609.19291#S6.SS4.p1.1),[§7](https://arxiv.org/html/2609.19291#S7.p2.1)\. - Patilet al\.\(2022\)V\. Patil, P\. Talukdar, and S\. SarawagiOverlap\-based vocabulary generation improves cross\-lingual transfer among related languages\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 219–233\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.acl-long.18),[Link](https://aclanthology.org/2022.acl-long.18/)Cited by:[§7](https://arxiv.org/html/2609.19291#S7.p2.1)\. - Penedoet al\.\(2024\)G\. Penedo, H\. Kydlícek, L\. B\. Allal, A\. Lozhkov, M\. Mitchell, C\. A\. Raffel, L\. von Werra, and T\. WolfThe fineweb datasets: decanting the web for the finest text data at scale\.InAdvances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 \- 15, 2024,A\. Globersons, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. M\. Tomczak, and C\. Zhang \(Eds\.\),External Links:2406\.17557,[Link](http://papers.nips.cc/paper/_files/paper/2024/hash/370df50ccfdf8bde18f8f9c2d9151bda-Abstract-Datasets/_and/_Benchmarks/_Track.html)Cited by:[§3\.1](https://arxiv.org/html/2609.19291#S3.SS1.p2.1)\. - Penget al\.\(2024\)Q\. Peng, Y\. Chai, and X\. LiHumanEval\-XL: a multilingual code generation benchmark for cross\-lingual natural language generalization\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),N\. Calzolari, M\. Kan, V\. Hoste, A\. Lenci, S\. Sakti, and N\. Xue \(Eds\.\),Torino, Italia,pp\. 8383–8394\.External Links:[Link](https://aclanthology.org/2024.lrec-main.735/)Cited by:[§7](https://arxiv.org/html/2609.19291#S7.p1.1)\. - Petrovet al\.\(2023\)A\. Petrov, E\. L\. Malfa, P\. H\. S\. Torr, and A\. BibiLanguage model tokenizers introduce unfairness between languages\.InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 \- 16, 2023,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),External Links:2305\.15425,[Link](http://papers.nips.cc/paper/_files/paper/2023/hash/74bb24dca8334adce292883b4b651eda-Abstract-Conference.html)Cited by:[§5\.1](https://arxiv.org/html/2609.19291#S5.SS1.p1.1),[§7](https://arxiv.org/html/2609.19291#S7.p2.1)\. - Philippyet al\.\(2023\)F\. Philippy, S\. Guo, and S\. HaddadanTowards a common understanding of contributing factors for cross\-lingual transfer in multilingual language models: a review\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 5877–5891\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.323),2305\.16768,[Link](https://aclanthology.org/2023.acl-long.323/)Cited by:[§5\.1](https://arxiv.org/html/2609.19291#S5.SS1.p1.1)\. - Qiet al\.\(2023\)J\. Qi, R\. Fernández, and A\. BisazzaCross\-lingual consistency of factual knowledge in multilingual language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 10650–10666\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.658),[Link](https://aclanthology.org/2023.emnlp-main.658/)Cited by:[§1](https://arxiv.org/html/2609.19291#S1.p2.1),[§7](https://arxiv.org/html/2609.19291#S7.p1.1)\. - Qwenet al\.\(2025\)Qwen, :, A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. QiuQwen2\.5 technical report\.External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115)Cited by:[§G\.1](https://arxiv.org/html/2609.19291#A7.SS1.p1.1)\. - Rustet al\.\(2021\)P\. Rust, J\. Pfeiffer, I\. Vulić, S\. Ruder, and I\. GurevychHow good is your tokenizer? on the monolingual performance of multilingual language models\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),C\. Zong, F\. Xia, W\. Li, and R\. Navigli \(Eds\.\),Online,pp\. 3118–3135\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.243),2012\.15613,[Link](https://aclanthology.org/2021.acl-long.243/)Cited by:[§5\.1](https://arxiv.org/html/2609.19291#S5.SS1.p1.1),[§7](https://arxiv.org/html/2609.19291#S7.p2.1)\. - Sajiet al\.\(2025\)A\. Saji, J\. A\. Husain, T\. Jayakumar, R\. Dabre, A\. Kunchukuttan, and R\. PuduppullyRomanLens: the role of latent Romanization in multilinguality in LLMs\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 26410–26429\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.1354),ISBN 979\-8\-89176\-256\-5,[Link](https://aclanthology.org/2025.findings-acl.1354/)Cited by:[§K\.4](https://arxiv.org/html/2609.19291#A11.SS4.p1.1),[§6\.4](https://arxiv.org/html/2609.19291#S6.SS4.p1.1)\. - Schäferet al\.\(2024\)A\. Schäfer, S\. Ravfogel, T\. Hofmann, T\. Pimentel, and I\. SchlagThe role of language imbalance in cross\-lingual generalisation: insights from cloned language experiments\.External Links:2404\.07982,[Link](https://arxiv.org/abs/2404.07982)Cited by:[Appendix C](https://arxiv.org/html/2609.19291#A3.p1.1),[§1](https://arxiv.org/html/2609.19291#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.19291#S2.SS1.p2.1),[§7](https://arxiv.org/html/2609.19291#S7.p3.1)\. - Shiet al\.\(2023\)F\. Shi, M\. Suzgun, M\. Freitag, X\. Wang, S\. Srivats, S\. Vosoughi, H\. W\. Chung, Y\. Tay, S\. Ruder, D\. Zhou, D\. Das, and J\. WeiLanguage models are multilingual chain\-of\-thought reasoners\.InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1\-5, 2023,External Links:2210\.03057,[Link](https://openreview.net/pdf?id=fR3wGCk-IXp)Cited by:[§7](https://arxiv.org/html/2609.19291#S7.p1.1)\. - van den Oordet al\.\(2018\)A\. van den Oord, Y\. Li, and O\. VinyalsRepresentation learning with contrastive predictive coding\.External Links:1807\.03748,[Link](https://arxiv.org/abs/1807.03748)Cited by:[§E\.2](https://arxiv.org/html/2609.19291#A5.SS2.p2.1),[§4\.2](https://arxiv.org/html/2609.19291#S4.SS2.p3.1)\. - von Radet al\.\(2026a\)J\. von Rad, L\. Arts, G\. Burgess, E\. Kolokytha, H\. O’Donnell, E\. O\. Doumpas, E\. Sanchez, Y\. Lu, and P\. StenetorpImproving cross\-lingual factual recall via consistency\-driven reinforcement learning\.External Links:2606\.06586,[Link](https://arxiv.org/abs/2606.06586)Cited by:[§1](https://arxiv.org/html/2609.19291#S1.p2.1)\. - von Radet al\.\(2026b\)J\. von Rad, L\. Arts, G\. Burgess, E\. Kolokytha, H\. O’Donnell, E\. O\. Doumpas, E\. Sanchez, Y\. Lu, and P\. StenetorpPolyFact: comparing consistency\-driven post\-training methods for cross\-lingual factual recall\.External Links:2606\.06586,[Link](https://arxiv.org/abs/2606.06586)Cited by:[§7](https://arxiv.org/html/2609.19291#S7.p1.1)\. - Wanget al\.\(2025a\)M\. Wang, H\. Adel, L\. Lange, Y\. Liu, E\. Nie, J\. Strötgen, and H\. SchuetzeLost in multilinguality: dissecting cross\-lingual factual inconsistency in transformer language models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 5075–5094\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.253),ISBN 979\-8\-89176\-251\-0,[Link](https://aclanthology.org/2025.acl-long.253/)Cited by:[§7](https://arxiv.org/html/2609.19291#S7.p1.1)\. - Wanget al\.\(2025b\)Q\. Wang, S\. Pan, T\. Linzen, and E\. BlackMultilingual prompting for improving LLM generation diversity\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 6367–6389\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.324),ISBN 979\-8\-89176\-332\-6,[Link](https://aclanthology.org/2025.emnlp-main.324/)Cited by:[§7](https://arxiv.org/html/2609.19291#S7.p1.1)\. - Wanget al\.\(2025c\)W\. Wang, M\. Wu, B\. Haddow, and A\. BirchBridging the language gaps in large language models with inference\-time cross\-lingual intervention\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 5418–5433\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.270),2410\.12462,ISBN 979\-8\-89176\-251\-0,[Link](https://aclanthology.org/2025.acl-long.270/)Cited by:[§7](https://arxiv.org/html/2609.19291#S7.p1.1)\. - Wu and Dredze \(2020\)S\. Wu and M\. DredzeDo explicit alignments robustly improve multilingual encoders?\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 4471–4482\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.362),[Link](https://aclanthology.org/2020.emnlp-main.362/)Cited by:[§1](https://arxiv.org/html/2609.19291#S1.p2.1)\. - Xueet al\.\(2022\)L\. Xue, A\. Barua, N\. Constant, R\. Al\-Rfou, S\. Narang, M\. Kale, A\. Roberts, and C\. RaffelByT5: towards a token\-free future with pre\-trained byte\-to\-byte models\.Transactions of the Association for Computational Linguistics10,pp\. 291–306\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00461),[Link](https://aclanthology.org/2022.tacl-1.17/)Cited by:[§8](https://arxiv.org/html/2609.19291#S8.p4.1)\. - Zenget al\.\(2026\)L\. Zeng, S\. Y\. Feng, and M\. C\. FrankBringing up a bilingual babylm: investigating multilingual language acquisition using small\-scale models\.External Links:2603\.29552,[Link](https://arxiv.org/abs/2603.29552)Cited by:[§4\.2](https://arxiv.org/html/2609.19291#S4.SS2.p2.1)\. - Zhanget al\.\(2025\)C\. Zhang, J\. Lu, V\. Q\. Tran, T\. Schuster, D\. Metzler, and J\. LinTomato, tomahto, tomate: do multilingual language models understand based on subword\-level semantic concepts?\.InFindings of the Association for Computational Linguistics: NAACL 2025,L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 1821–1837\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-naacl.98),ISBN 979\-8\-89176\-195\-7,[Link](https://aclanthology.org/2025.findings-naacl.98/)Cited by:[§7](https://arxiv.org/html/2609.19291#S7.p2.1)\. ## Appendix ATraining Details All models are pretrained using the TorchTitan framework\([Liang et al\., 2025](https://arxiv.org/html/2609.19291#bib.bib19)\)\. An architectural breakdown and hyperparameters used can be found in Tab\.[A\.1](https://arxiv.org/html/2609.19291#A1.T1) ### A\.1 Architectures #### SmolLM2\-360M\. A standard dense transformer following the SmolLM2 architectural configuration\([Allal et al\., 2025](https://arxiv.org/html/2609.19291#bib.bib9)\)\. #### Llama\-3 7B\. A scaled\-down adaptation of the Llama\-3 8B architecture\([Grattafiori et al\., 2024](https://arxiv.org/html/2609.19291#bib.bib11)\)\. We reduce the number of hidden layers from 32 to 28 and the intermediate feed\-forward dimension from 14,336 to 13,312, giving approximately 7 billion parameters\. ### A\.2 Tokenizer We use a BPE tokenizer with a vocabulary of 65,536 tokens, trained on a 50/50 split of our bilingual data\. ForEnglish1–English2experiments, we double the vocabulary to 131,072 to accommodate two fully disjoint language copies\. ### A\.3 Optimisation All models are optimised with AdamW\([Loshchilov and Hutter, 2019](https://arxiv.org/html/2609.19291#bib.bib6)\)usingβ1=0\.9\\beta\_\{1\}=0\.9,β2=0\.95\\beta\_\{2\}=0\.95, and weight decay0\.10\.1\. The learning rate follows a warmup–decay, cosine decay, schedule\. ### A\.4 Scale\-Specific Configurations Compute budgets target Chinchilla\-optimal data scaling\([Hoffmann et al\., 2022](https://arxiv.org/html/2609.19291#bib.bib15)\)\. Learning rates are inspired by the original architecture configurations and tuned to our use case\. #### SmolLM2\-360M\. 4,600 steps, batch size 768, sequence length 2,048, for approximately 7\.2B tokens\. Warmup stage is 300 steps, peaking at lr5×10−45\\times 10^\{\-4\}then decaying to2\.5×10−52\.5\\times 10^\{\-5\}\. Trained on 8 RTX B6000 Pro GPUs\. #### Llama\-3 7B\. 133,600 steps, batch size 512, sequence length 2,048, for approximately 140B tokens\. Warmup stage is 1,000 steps peaking at lr3×10−43\\times 10^\{\-4\}then decaying to3×10−53\\times 10^\{\-5\}\. Models are trained on 128 H100 GPUs across 16 nodes\. Table A\.1:Pretraining configurations at both scales\. EveryCLeqresult in the paper corresponds to one full pretraining run under one of these two configurations\. TheEnglish1–English2runs instead use a doubled vocabulary of 131,072 tokens, to accommodate two disjoint token spaces \(see*Tokenizer*, above\)\. ### A\.5 Injection Mechanics To computeCLeqwe split FKD into 16 groups of 128 entities\. Each group is assigned an exposure count for languageAAand languageBBdrawn from the cross\-product of\{0,20,100,1000\}\\\{0,20,100,1000\\\}with itself\. Documents are injected stochastically during pretraining at a per\-language probability matching the target exposure count for that fact\. We report the realized injection counts and SDs in Tab\.[A\.2](https://arxiv.org/html/2609.19291#A1.T2)based on a representative run\. We find this pattern replicates across runs\. Table A\.2:Realised exposure counts against their targets, measured on a representative 360MEnglish–Arabicrun\. Deviations are consistent with the stochastic injection procedure; relative deviation is largest in the low\-exposure cells\. ### A\.6 Two\-Stage Mixing Schedule When language mixing intervention — code\-switching — is applied at 360M parameter scale, training is divided into two stages\. The first and longer stage \(3,600 steps\) applies the mixing at the chosen mixing rates; the second, shorter stage applies none, realigning the model to the natural data distribution and preventing excessive code\-switching in generation\. ## Appendix BFictive Knowledge Dataset \(FKD\) This appendix details the construction of the fictive knowledge dataset — FKD— we inject into pretraining to measure cross\-lingual knowledge generalization \(§[2\.1](https://arxiv.org/html/2609.19291#S2.SS1)\)\. Because the facts are fictive, they are guaranteed to be absent from the underlying corpus, so all recall is attributable to our injections and the number of exposures in each language is known exactly\. The dataset and generation code are publicly available[Why Pretraining Fails to Share Cross\-Lingual Knowledge](https://arxiv.org/html/2609.19291#footnotex3)\. FKD covers 3 languages that we evaluate in this paper — English, Arabic and Russian\. It is generated in three stages: seed facts, injection documents, and evaluation multi\-choice questions \(MCQs\)\. Tab\.[B\.1](https://arxiv.org/html/2609.19291#A2.T1)provides a representative example\. ### B\.1 Stage 1: Seed Facts A seed fact pairs a named entity with a single attribute \(e\.g\.,“Neltis is a podcast about the business of coffee”\)\. We manually author 64 fact templates spanning entity types such as companies, podcasts, and tools\. Each template is passed to Gemini 3\.1 Pro with instructions to produce 32 distinct facts using artificially generated entity names \(with the exception of entity types that are human, where generating some generic name\), yielding64×32=2,04864\\times 32=2\{,\}048seeds\. #### Translation and entity names\. Seeds are translated into all target languages with entity names transliterated\. We use GPT\-5\.5\-mini at a low reasoning setting for this end\. This means entity names share no tokens across scripts, avoiding this confounding variable in knowledge generalization\([Bandarkar et al\., 2026](https://arxiv.org/html/2609.19291#bib.bib10)\)\. #### Language balance\. Generated names inherit the phonotactics of the language they were generated in: prompting in English yields names that read as phonetically plausible in English but not necessarily elsewhere\. We verified this by manual inspection\. To prevent the resulting bias from favouring a single language, we generate seeds in equal proportion across English and Arabic, the pair on which most of our main experiments focus\. ### B\.2 Stage 2: Injection Documents Facts in real corpora appear scattered and paraphrased across heterogeneous contexts rather than in a single canonical statement\. To reproduce this, we expand each translated seed into many short documents that state or imply the fact in varied natural settings: casual blog posts, news items, advertisements, forum replies, and similar\. Expansion uses GPT\-5\.5\-mini at a low reasoning setting, with identical prompts across languages\. We generate 200–300 documents per entity per language\. The resulting documents’ average token length is∼\\sim20 tokens\. This diversity is deliberate;[Allen\-Zhu and Li \(2024\)](https://arxiv.org/html/2609.19291#bib.bib22)show that knowledge must be sufficiently augmented during pretraining—through paraphrasing and varied presentation—to acquire generalizable knowledge\. While their experiments conclude that even 5 augmentations are enough, we choose to target a higher number to better capture natural settings\. ### B\.3 Stage 3: Evaluation Questions For each fictive fact and language, we generate five four\-choice questions testing knowledge of it\. Distractors are generated to be as plausible as the correct choice given no prior knowledge but clearly incorrect given the fact\. These questions are generated separately from the injection documents and never appear in the training corpus\. We evaluate using the LM Evaluation Harness \(§[3\.2](https://arxiv.org/html/2609.19291#S3.SS2)\), averaging over the five questions per entity\. ### B\.4 FKD for WWT\-Ar FKD for WWT\-Ar introduced in §[6\.1](https://arxiv.org/html/2609.19291#S6.SS1)is obtained by applying the corresponding WWT mapping to the Arabic documents and evaluations, so that WWT\-Ar contains the same facts, documents and evaluations as natural Arabic\. ### B\.5 FKD Validation & Fact\-Acquisition Learning Curve Figure B\.1:Monolingual acquisition of fictive facts \(FKD\)\. The graph depicts the performance \(y\-axis\) per number of times a fact was seen \(x\-axis\) per input language \(line color\)\.We validate our fictive facts datasets \- FKD\- and injection pipeline by pretraining 360M\-parameter models with monolingual FKD injection\. We vary the exposure rates and measure fact acquisition using the accompanying MCQs; results in Fig\.[B\.1](https://arxiv.org/html/2609.19291#A2.F1)\. Our measurements reveal that fact acquisition \(measured through MCQ accuracy\) exhibits a log\-like dependence on exposure count\. Accuracy increases rapidly at the lower exposure range \(0\-300 range\) and plateaus around 1000 exposures\. Our evaluation injection grid is concentrated in the steep region, where learning efficiency is most measurable \(§[3\.2](https://arxiv.org/html/2609.19291#S3.SS2)\)\. While the overall learning\-curve trend is consistent across languages, Accuracy values differ, possibly reflecting a bias in our generation process\. Nonetheless, our metric’s definition as relative gain eliminates these differences from our measurements \(§[2\.2](https://arxiv.org/html/2609.19291#S2.SS2.SSS0.Px1)\)\. Our knowledge acquisition measurements are consistent with prior work on characterizing learning dynamics under controlled settings and extend them into the natural regime\.[Allen\-Zhu and Li \(2025\)](https://arxiv.org/html/2609.19291#bib.bib17)measure knowledge acquisition in synthetic setups, with many repetitions and a predefined document structure, and report that 1000 fact exposures reach near\-perfect memorization\. Extending this analysis to natural settings is challenging as knowledge exposure is intractable in enormous pretraining corpora\. Thus, natural\-corpus analyses evaluate fact acquisition and estimate exposure count based on entity frequency, reporting substantially slower acquisition\([Kandpal et al\., 2023](https://arxiv.org/html/2609.19291#bib.bib3)\), possibly due to overcounting when the entity is mentioned but the fact is not\. Our controlled setup using FKD provides precise measurement in a natural setting, confirming the synthetically measured dynamics\. ### B\.6 Examples Table B\.1:Representative example from FKD\. All seeds, documents, and MCQs in the three supported languages appear in the released data\. ## Appendix CHuman Fictional Knowledge Dataset \(HFKD\) FKD mostly uses artificially generated entity names, placing its entities names in the low\-frequency token regime\. Prior work indicates that cross\-lingual bridging mechanisms behave differently for high\- and low\-frequency tokens\([Schäfer et al\., 2024](https://arxiv.org/html/2609.19291#bib.bib5)\), so we construct a second dataset, Human Fictional Knowledge Dataset \(HFKD\), using human entities with common first and last names—tokens that appear frequently in any pretraining corpus\. All downstream processing \(document expansion, evaluation questions, injection\) is identical to FKD\. ### C\.1 Seed Generation To achieve diversity for human entities, we generate the fact space compositionally\. We define 50 professional domains \(cinema, science, etc\.\); GPT\-5\.5\-mini generates 5 roles per domain \(for cinema: actor, director, etc\.\) and 4 attributes per role \(for actor: awards, debut, genre, etc\.\), giving 1,000 role–attribute templates\. Each seed fact samples a first and last name from curated lists of 100 each, pairs them with a template, and prompts the model to fill in a plausible attribute value\. We sample 2,048 such combinations\. ### C\.2 Acquisition Characteristics HFKD is not directly comparable to FKD, Its monolingual acquisition curve occupies a substantially compressed range \(Fig\.[C\.1](https://arxiv.org/html/2609.19291#A3.F1)\), beginning at 60% and plateauing around 87%, against FKD’s 34% to 96%\. We suspect this is the result of contamination from real individuals, or generation quality\. We therefore interpret HFKD scores with caution and provide them only as an indication that our main results are consistent across frequency regimes\. Figure C\.1:Monolingual English acquisition of HFKD and FKD\. The graph depicts the performance \(y\-axis\) per number of times a fact was seen \(x\-axis\) per fictive knowledge dataset \(line color\)\. ### C\.3 Replication Results Every qualitative conclusion in the paper replicates on HFKD: bilingual pretraining is heavily compartmentalized\. theEnglish1–English2setup shows poor knowledge generalization as well, with embedding tying restoring it monotonically\. Finally, WWT substantially improves knowledge generalization between natural vocabulary disjoint languages \(Table[C\.1](https://arxiv.org/html/2609.19291#A3.T1)\)\. Table C\.1:CLeqon FKD and HFKD across the paper’s main settings on a 360M\-parameter scale\. Ordering is mostly preserved throughout, and headline comparisons hold;HFKD estimates are substantially noisier, appearing miscalibrated at the metric floor with one of the scores well into the negative region\. This follows from its compressed acquisition range: because accuracy spans roughly 30 points rather than 60, the native\-exposure slopeβA\\beta\_\{A\}is correspondingly smaller, and sinceCLeqis the ratioβB/βA\\beta\_\{B\}/\\beta\_\{A\}, a smaller denominator inflates the variance of the estimate while the entity\-based noise inβB\\beta\_\{B\}is unchanged\. The negative values should therefore be read as absolute compartmentalization against a wider noise floor than FKD’s, not as negative transfer\. ## Appendix DCross\-lingual Equivalence Score \(CLeq\) This appendix documents howCLeqis estimated, why the linear specification is preferred over a curve\-aware one, and how much sampling variance the estimates carry — establishing that the paper’s headline comparisons survive that variance\. ### D\.1 Estimation and Fit Diagnostics We estimateCLeqby estimating native and foreign exposure value to fact acquisition through ordinary least squares over all facts in FKD, regressing language\-AAaccuracy on the injection counts in both languages \(§[2\.2](https://arxiv.org/html/2609.19291#S2.SS2.SSS0.Px1)\)\. Native learning is strong across our experiments:βA\\beta\_\{A\}is positive and significant throughout, so the denominator of the ratio is bounded away from zero andCLeqestimates do not exhibit the instability characteristic of small\-denominator ratio estimators\. Precision is nonetheless limited by the numerator, as we discuss below\. Table[D\.1](https://arxiv.org/html/2609.19291#A4.T1)reports the fitted coefficients for two representative runs at 360M: theEnglish–Arabicbaseline and its WWT\-Ar counterpart, both evaluated in the Arabic→\{\\rightarrow\}English direction\. Native learning is comparable across the two \(βA≈4\.4\\beta\_\{A\}\\approx 4\.4–4\.9×10−44\.9\\times 10^\{\-4\}accuracy gain per exposure, accuracy range is 0–1\), so the difference inCLeqis driven almost entirely by the cross\-lingual coefficient\. In the baseline,βB\\beta\_\{B\}is statistically indistinguishable from zero: its 95% confidence interval spans zero \(CI across FKD fact samples\)\. Under WWT\-Ar,βB\\beta\_\{B\}is significantly positive and its interval lies entirely above the baseline’s upper bound\. The linear specification captures most but not all of the variation in accuracy \(R2R^\{2\}column in Tab\.[D\.1](https://arxiv.org/html/2609.19291#A4.T1)\)\. The residual misfit is systematic rather than random\. This is the expected consequence of fitting a saturating curve linearly over a non\-uniform exposure grid — the fit over\-predicts at zero exposures and under\-predicts at 100 — and it motivates a curve\-aware specification\. Table D\.1:Fitted coefficients of Eq\.[1](https://arxiv.org/html/2609.19291#S2.E1)for two representative 360M runs, in the Arabic→\{\\rightarrow\}English direction\.α\\alphais the intercept,βA\\beta\_\{A\}andβB\\beta\_\{B\}the accuracy gained per native and per cross\-lingual exposure\. Native learning is comparable across settings; the baseline’s cross\-lingual coefficient is indistinguishable from zero, while WWT\-Ar’s is significantly positive\. Notice that Eq\.[1](https://arxiv.org/html/2609.19291#S2.E1)fits accuracy between 0 and 1, hence the scale of the diffrent fitted parameters\. ### D\.2 Curve\-AwareCLeq Fact acquisition follows a log\-like rather than linear dependence on exposure count \(§[2\.1](https://arxiv.org/html/2609.19291#S2.SS1)\), pointing towards a misfit of our linear fit estimator\. We therefore also implement a curve\-aware estimator that models saturation directly: accA\(f\)=alog\(expsA\(f\)\+αexpsB\(f\)\+1\)\+b,\\mathrm\{acc\}\_\{A\}\(f\)=a\\log\\\!\\big\(exps\_\{A\}\(f\)\+\\alpha\\,exps\_\{B\}\(f\)\+1\\big\)\+b,\(4\)fitted by nonlinear least squares over three free parameters\. Hereα\\alphaadmits a direct reading as theCLeqquantity: it is the number of native exposures that one foreign exposure is worth, entering the curve on the same footing asexpsAexps\_\{A\}\. The curve\-aware score is therefore100×α100\\times\\alpha\. The curve\-aware specification fits the data substantially better, but itsα\\alphais not well identified\. Becauseα\\alphaenters only inside the logarithm alongsideexpsAexps\_\{A\}, and becauseaascales andbbshifts the resulting curve, the three parameters trade off against one another\. This additional flexibility improves fit without guaranteeing thatα\\alphaisolates the quantity of interest\. The linear specification has the opposite character\. It fits less well, and its residuals retain systematic curvature as we mentioned above, butβB\\beta\_\{B\}multipliesexpsBexps\_\{B\}alone and is therefore identified by construction: no combination of the remaining parameters can absorb variation along the foreign axis\. We prefer this trade—a worse fit with an interpretable parameter over a better fit with a confounded one—and report the linear estimator throughout\. Table[D\.2](https://arxiv.org/html/2609.19291#A4.T2)reports both, while absolute values differ, both specifications provide the same ranking among our main comparisons\. Table D\.2:CLequnder the linear and curve\-aware estimators\. The curve\-aware fit attains higherR2R^\{2\}throughout, while the ordering of conditions is mostly preserved under both\. ### D\.3 Sampling Variance and Significance CLeqis a ratio estimator computed over a finite set of entities, and its sampling variance is substantial\. Native learning is robust in every experiment we report, so the denominator contributes little noise; the variance is carried almost entirely by the numerator, which in most conditions estimates a very weak effect\. The suspect candidates for the noise sources are per\-fact variation in learnability, in the quality of its paraphrases, and in the difficulty of its evaluation questions\. Estimated independently, the resulting intervals are wide\. At 360M theEnglish–Arabicbaseline givesCLeq=0\.9%\\text\{CL\\textsubscript\{eq\}\}=0\.9\\%with a 95% CI of\[−5\.5,7\.4\]\[\-5\.5,7\.4\], while WWT\-Ar gives12\.6%12\.6\\%with\[5\.4,20\.0\]\[5\.4,20\.0\]\. These marginal intervals overlap, but they answer the wrong question\. Much of the variance is shared: a resampled set of entities that results in an extremeCLeqvalue for one model effects other models similarly, so entity variance inflates each interval without affecting the comparison between methods\. We therefore resample entities*jointly*across conditions, drawing one bootstrap sample per replicate \(within\-cell, entity\-wise, with replacement across our 16 groups of 128 entities\), evaluating all conditions on that same sample, and reporting the distribution of*differences*\(Table[D\.3](https://arxiv.org/html/2609.19291#A4.T3)\)\. Because every condition sees the same entities in a given replicate, shared entity variance cancels, and what remains is the variance of the comparison itself\. Under this procedure, the comparisons supporting our central claims separate cleanly\. WWT\-Ar exceeds theEnglish–Arabicbaseline by 11\.7 percentage points, with a 95% CI of\[7\.5,16\.1\]\[7\.5,16\.1\]that excludes zero; even its lower bound is several times the baseline point estimate\. The same holds at 7B and for the semantic controls of §[6\.4](https://arxiv.org/html/2609.19291#S6.SS4)\. Two consequences follow for reading the paper’s numbers\. First, small negativeCLeqestimates arise from noise inβB\\beta\_\{B\}when cross\-lingual learning is very weak; they are indistinguishable from zero and should be read as substantial or absolute compartmentalization rather than as negative transfer\. Second, differences of a few percentage points between conditions should not be interpreted as meaningful — the comparisons we draw conclusions from are those in Table[D\.3](https://arxiv.org/html/2609.19291#A4.T3)whose intervals exclude zero\. #### Scope of the bootstrap\. This procedure captures evaluation\-side variance only\. It does not capture variance across training runs: each configuration is a single pretraining run, and both the realized injection counts and their placement in the training order differ between runs\. Repeating every configuration across seeds is prohibitive at these scales\. We instead argue robustness through replication across the axes we can vary: model scale, language pair, and the close to monotone dose\-response in theEnglish1–English2tying \(§[5\.3](https://arxiv.org/html/2609.19291#S5.SS3)\) andEnglish–Arabicsoft\-mapping \(§[6\.3](https://arxiv.org/html/2609.19291#S6.SS3)\) sweeps, where noise alone would not produce a consistent trend across independent runs\. ComparisonΔ\\Delta95% CISignificant*English/Arabic, 360M*WWT\-Ar vs\. baseline\+11\.7\+11\.7\[7\.5,16\.1\]\[7\.5,16\.1\]∗WWT\-Ar vs\. Shuffled WWT\-Ar\+9\.6\+9\.6\[4\.5,14\.7\]\[4\.5,14\.7\]∗WWT\-Ar vs\. Transliterated\-Ar\+13\.0\+13\.0\[8\.9,17\.2\]\[8\.9,17\.2\]∗*English/Arabic, 7B*WWT\-Ar vs\. baseline\+6\.6\+6\.6\[1\.8, 11\.4\]∗*English/Russian, 360M*WWT\-Ru vs\. baseline21\.3%\[16\.8, 25\.9\]∗WWT\-Ru vs\. Shuffled WWT\-Ru19\.7%\[13\.3, 26\.1\]∗*English1−English2\\text\{English\}\_\{1\}\-\\text\{English\}\_\{2\}, 360M*99\.9% tying vs\. baseline\+101\.6\[91\.5, 113\.5\]∗99\.9% tying vs\. shared initialization\+20\.4\[12\.8, 28\.6\]∗Table D\.3:Paired\-bootstrap differences inCLeqfor the paper’s main comparisons\. Each replicate resamples entities once and evaluates all conditions on the same sample, so shared entity\-difficulty variance cancels\.∗marks intervals excluding zero\. ### D\.4 Target and Realized Exposure Because injection is stochastic, the realized number of exposures deviates from the target\. Deviations are consistent across our experiments, values reported in Tab\.[A\.2](https://arxiv.org/html/2609.19291#A1.T2)\. In relative terms the low\-exposure cells are the most affected—24\.3% at 20 exposures against 3\.2% at 1000—and these are precisely the cells in the steepest region of the acquisition curve \(App\.[B\.5](https://arxiv.org/html/2609.19291#A2.SS5)\), where accuracy is most sensitive to exposure count\. We estimateCLequsing target exposures\. To verify that this does not distort our measurements, we recomputeCLequsing the realized counts for two representative runs and find the difference negligible \(Table[D\.4](https://arxiv.org/html/2609.19291#A4.T4)\)\. Table D\.4:CLeqestimated from target versus realized exposure counts\. Using realized counts leaves both the values and their difference essentially unchanged\. ## Appendix EIntervention Implementation Details This appendix provides implementation details and detailed results for the interventions evaluated in §[4\.2](https://arxiv.org/html/2609.19291#S4.SS2)\. ### E\.1 Language Mixing — Code\-switching Inspired by lingual code\-switching, pretraining code\-switching introduces cross\-lingual context by randomly swapping words with their cross\-lingual counterparts during pretraining\. Because the language of the next token becomes stochastic, the model is pushed toward assigning shared representations across languages\. We vary two parameters: the*mixing ratio*—the proportion of documents subjected to code\-switching—and the*substitution probability*—the probability of replacing any given word within such a document\. We execute the substitution using a variant of our curated dictionary presented in \(App\.[G](https://arxiv.org/html/2609.19291#A7)\), which skips the conflict resolution stage to avoid the appended IDs on substituted words\. #### Data exclusion\. We strictly exclude FKD documents from the mixing pool, simulating the realistic case in which a limited aligned dataset is used alongside a much larger body of standard\-format data for which we would like to evaluate transfer\. #### Results\. We sweep the mixing ratio over\{0\.6,0\.1,0\.01\}\\\{0\.6,0\.1,0\.01\\\}and the substitution probability over\{0\.5,0\.1,0\.01\}\\\{0\.5,0\.1,0\.01\\\}\(Table[E\.1](https://arxiv.org/html/2609.19291#A5.T1)\)\. No setting produces meaningful knowledge generalization:CLeqranges from−1\.2%\-1\.2\\%to2\.2%2\.2\\%across the sweep, against a baseline of0\.9%0\.9\\%\(App\.[D](https://arxiv.org/html/2609.19291#A4)\)\. There is no clear trend in either parameter — the best and worst cells differ in mixing ratio by a factor of 60, but the ordering is not monotone in either direction, indicating the differences can be mainly attributed to noise\. Code\-switching in a naturalEnglish–Arabicvocabulary\-disjoint setting therefore does not appear to induce knowledge sharing at any mixing volume we test\. Table E\.1:CLequnder pretraining code\-switching in theEnglish–Arabicsetting at 360M, across mixing ratio and substitution probability\. The native\-script baseline is 0\.9%\. ### E\.2 Activation Alignment Loss Activation Alignment loss encourages the model to align inner representations of cross\-lingual semantically equivalent units\. We extract a global representation of a sequence by concatenating the max and average pooling of its layer\-4 activations\. We also experiment with multi\-depth aggregation of the loss on the SmolLM2\-360M architecture, where we average the losses from layers\{4,8,12,16,20,24,28\}\\\{4,8,12,16,20,24,28\\\}\(having normalized the losses to account for differences in scales across layers\)\. The auxiliary loss is weighted equally against the cross\-entropy objective and applied to 60%\. As with mixing, FKD documents are excluded from the alignment set\. We evaluate both an InfoNCE contrastive loss\([van den Oord et al\., 2018](https://arxiv.org/html/2609.19291#bib.bib8);[Chi et al\., 2021](https://arxiv.org/html/2609.19291#bib.bib7)\)and an L2 loss\. L2 lacks negative sampling and is in principle susceptible to representational collapse; we hypothesise that the primary language\-modeling objective provides sufficient regularization to prevent this\. #### Chunked matches\. Since only document\-level parallel data is available across ourEnglish–Arabicsetup, we utilize an assumption of structural similarity between paired documents — chunks from similar relative positions in the document carry similar semantics\. We choose a unit lengthnn, split the language\-AAdocument into chunks ofnnwords, and split the language\-BBdocument into the same number of equal\-length chunks\. The choice ofnnpresents a trade\-off: largenngives high confidence that paired chunks correspond but poor granularity, while smallnngives fine granularity at the risk of misalignment\. The structural\-similarity assumption is somewhat defensible for machine\-translated corpora such as ours due to the structural biases of machine translation\. #### Results\. We sweep the chunk sizennover\{4,20,100,250\}\\\{4,20,100,250\\\}words for both losses \(Table[E\.2](https://arxiv.org/html/2609.19291#A5.T2)\)\. No configuration offers substantial benefit to knowledge generalization:CLeqranges from 0\.3% to 2\.5% across the sweep, against a 0\.9% baseline\. Table E\.2:CLequnder activation alignment in theEnglish–Arabicsetting at 360M, across loss type and chunk size\. The native\-script baseline is 0\.9%\. ## Appendix FSoft Mapping Several experiments use*soft mapping*: a single mechanism for partially sharing representations between two token sets, applied in two settings\. In this appendix, we define it properly, describe each application, and present detailed results in Tab\.[F\.1](https://arxiv.org/html/2609.19291#A6.T1)\. ### F\.1 Mechanism Letddbe the embedding dimension of our model andℳ\\mathcal\{M\}be a correspondence between two token sets \(e\.g\.,English2\\text\{English\}\_\{2\}token and itsEnglish1\\text\{English\}\_\{1\}counterpart\)\. Under soft mapping with sharing ratepp, each pair\(t,t′\)∈ℳ\(t,t^\{\\prime\}\)\\in\\mathcal\{M\}shares the first⌊p⋅d⌋\\lfloor p\\cdot d\\rfloorembedding dimensions as tied parameters updated by gradients from both tokens; the remaining dimensions are independent\. Thusp=0p=0recovers disjoint embeddings andp=1p=1recovers full mapping\. Tokens without a counterpart inℳ\\mathcal\{M\}are unaffected\. ### F\.2 Diagnostic Use:English1–English2 Here soft mapping is purely a measurement instrument\. The two languages are structurally identical andℳ\\mathcal\{M\}is the exact shift correspondence between a token and its copy, soppcontrols the sole difference between them\. Sweepingppanswers how much sharing is required before generalization emerges, rather than merely whether it does \(§[5\.3](https://arxiv.org/html/2609.19291#S5.SS3)\)\. The sweep also calibrates the upper end of theCLeqscale, sincep→1p\\rightarrow 1approaches perfect sharing by construction\. ### F\.3 Practical Use: Soft WWT\-Ar Under full mapping, WWT\-Ar tokens are indistinguishable from their English counterparts\. This is what enables knowledge sharing, but it leaves the model no mechanism to represent which language it is generating: the output language must be fixed externally and code\-switched generation is unsupported\. Soft mapping resolves this—each language retains distinct token identities and shares only embedding dimensions\. We do not evaluate code\-switching ability in generation directly; we report only that the mechanism exists\. We additionally hypothesized that reserving language\-specific dimensions would*improve*transfer, on the grounds that translation equivalents differ in connotation and possible additional meanings and that merging them discards information the model needs\. We find no evidence for this: transfer increases withpp\(§[6\.3](https://arxiv.org/html/2609.19291#S6.SS3)\)\. Soft mapping is therefore best understood as trading a small amount of transfer for the practical properties above, rather than as an improvement over full mapping\. #### Construction\. Because WWT can change the token count and does not provide a strict one\-to\-one token correspondence, Soft WWT\-Ar requires an additional step: we first perform the WWT while shifting the resulting tokens byVVinto a disjoint range \(as inEnglish1–English2we double the model vocabulary size to accommodate this\), then tie a fractionppof each WWT\-Ar token’s dimensions to the corresponding English token\. Table F\.1:CLeqas a function of the sharing rateppin both settings\.*Free dims*is the number of language\-specific embedding dimensions remaining at each rate\. forEnglish1–English2p=1p=1corresponds to completely tied language copies —CLeq=100%\\text\{CL\\textsubscript\{eq\}\}=100\\%by definition; for English–WWT\-Ar it corresponds to standard \(not soft\) WWT\. ## Appendix GDictionary Curation and WWT\-Ar Construction This appendix describes how we generate the large\-scale word mapping dictionaries used for WWT operation in §[6\.1](https://arxiv.org/html/2609.19291#S6.SS1)\. We also provide more information about the costs of this mapping and where they originate\. ### G\.1 Dictionary Construction We parse the first 7M documents of our Arabic corpus and collect all unique words, then translate them with Qwen2\.5\-32B\-Instruct\([Qwen et al\., 2025](https://arxiv.org/html/2609.19291#bib.bib16)\)\. The result is a dictionary with 11M entries\. Any existing bilingual dictionary could be substituted, avoiding this step entirely\. For Russian, we curate a similar dictionary following the same procedure on the Russian corpora\. We execute the WWT operation \(§[6\.1](https://arxiv.org/html/2609.19291#S6.SS1)\) on Arabic and Russian using these dictionaries ### G\.2 Conflict Resolution The mapping must be reversible\. Where multiple Arabic words translate to the same English word, we differentiate them by appending an index: if two Arabic words both map todog, one is remapped todog1\. This guarantees that every distinct Arabic word form receives a distinct WWT\-Ar form, so no information is lost under the mapping\. Translation is many\-to\-one at scale — our dictionaries hold 11\.05M Arabic and 13\.23M Russian entries mapping onto 6\.51M and 6\.99M distinct English targets — but conflicted targets are not uniformly frequent in text, so most replacements need no index\. In the Arabic corpus 81\.9% of word replacements take the bare form and 99\.7% need at most a single\-digit index; Russian is heavier at 55\.8% and 94\.6% \(Table[G\.1](https://arxiv.org/html/2609.19291#A7.T1)\), which we attribute to its richer inflectional morphology, where case and number variants that are distinct surface forms frequently share a single English lemma\. Table G\.1:Share of word replacements taking each conflict\-index length\. ### G\.3 Tokenization Fertility Word\-wise mapping does not preserve sequence length\. Because dictionary targets and transliterated forms tokenize differently from their source words, mapped documents are longer: WWT\-Ar documents average 921 tokens against 709 for native Arabic, an increase of 29\.9%\. WWT\-Ru averages 1459 tokens against 1190, an increase of 22\.6% \(Table[G\.2](https://arxiv.org/html/2609.19291#A7.T2)\)\. Conflict resolution indices are the dominant source of WWT’s sequence\-length overhead, since they turn a word that would tokenize cleanly into a word\-plus\-digit token sequence\. Excluding them, WWT\-Ar’s fertility penalty falls from\+29\.9%\+29\.9\\%to\+17\.6%\+17\.6\\%and WWT\-Ru’s from\+22\.6%\+22\.6\\%to−2\.1%\-2\.1\\%\(Table[G\.2](https://arxiv.org/html/2609.19291#A7.T2)\)\. The overhead is therefore a property of our conflict\-resolution policy rather than an inherent property of word\-wise mapping itself\. ### G\.4 Out\-of\-Dictionary Words Words absent from the dictionary are handled by reversible transliteration \- a character level 1\-to\-1 mapping into English script\. for Arabic we use Buckwalter transliteration and for Russian we use GOST\. We measure that in the Arabic corpus 99\.7% of word occurrences are matched rather than transliterated \(78% of unique words\), and for Russian that number stands at 99\.8% \(92\.7% of unique words\)\. Table G\.2:Mean document length in tokens under the respectiveEnglish–ArabicandEnglish–RussianBPE tokenizers; deltas are reported relative to the native script tokenization\. Word\-wise mapping increases sequence length, more so for Russian than for Arabic\. A substantial part of the fertility increase originates in the appended IDs used for conflict resolution\.This carries a genuine inference\-time cost: generating a given amount of content under WWT requires proportionally more forward passes than in the native script\. It also has a consequence for how our results should be read\. Because we hold the token budget fixed rather than the document count, the WWT\-Ar and WWT\-Ru models see fewer documents than their native\-script counterparts—roughly 23% fewer for WWT\-Ar — and therefore cover less of the foreign\-language corpus\. The mapped models nonetheless achieve comparable or better English perplexity \(§[6](https://arxiv.org/html/2609.19291#S6)\)\. One might attribute that gain to reduced interference: with less foreign content competing for capacity, English would benefit\. Our results do not support this explanation\. English\-only pretraining yields*worse*English perplexity than in a bilingual configuration \(20\.51 against 19\.53 and 18\.99; §[4](https://arxiv.org/html/2609.19291#S4)\), so additional foreign content helps rather than competes at this scale, and the models are not capacity\-limited\. Reducing Arabic coverage should therefore reduce the benefit English receives, not increase it\. That WWT\-Ar delivers a larger benefit while contributing fewer documents indicates it contributes more per token—consistent with the increase in knowledge generalization characterized through theCLeqscore\. ## Appendix HEmbedding\-Space Analysis Models fail to generalize knowledge between two identical copies of English \(§[5\.2](https://arxiv.org/html/2609.19291#S5.SS2)\), yet both copies carry the same language and data distribution, so we should expect the model to learn the same structure for each\. To test this, we compare the two halves of each model’s embedding matrix and ask whether they are structurally similar, whether one linear map relates them, and whether matching tokens are placed together\. ### H\.1 Setup Corresponding tokens in the twoEnglish1–English2vocabularies are directly aligned by row in the embedding matrix\. To reduce noise from rarely updated rows, we retain tokens with at least1,0001\{,\}000expected exposures per language, estimated from corpus frequency and training budget\. This leaves36,17036\{,\}170rows at 360M and37,06237\{,\}062at 7B, out of65,53665\{,\}536per vocabulary\. To prevent a shared mean direction from inflating similarity, we normalize rows, mean\-center each matrix, and renormalize\. We report four measures:*matched cosine*, the mean cosine between corresponding rows;*linear CKA*\([Kornblith et al\., 2019](https://arxiv.org/html/2609.19291#bib.bib68)\)for global structure;*mutualkk\-NN*\([Huh et al\., 2024](https://arxiv.org/html/2609.19291#bib.bib69)\)atk=10k=10for local structure; and*linear\-map cosine*, the matched cosine on a held\-out20%20\\%after fitting a least\-squares map on the remaining80%80\\%\. Within\-model comparisons use all eligible rows at each scale; across\-model comparisons use the shared eligible set\. We analyze the tied embedding matrix at 360M and the LM head at 7B\.111The 360M architecture ties its input embedding and output projection; the 7B architecture separates them\. We use the 7B LM head because it receives the vocabulary\-wide prediction updates that also train the tied 360M matrix, whereas its input embedding remains close to initialization\.Following[Gröger et al\. \(2026\)](https://arxiv.org/html/2609.19291#bib.bib70), we calibrate each measure againstK=200K=200random row permutations atα=0\.05\\alpha=0\.05\. On the resulting00–11scale, higher means more similar:00marks the permutation baseline and11identical spaces\. Every within\-model score exceeds all permutations \(p=0\.005p=0\.005\)\. Table H\.1:Null\-calibrated embedding similarity andCLeqacrossEnglish1–English2interventions\. The baselines learn similar, linearly alignable spaces but show negligible direct token correspondence and no knowledge transfer; code\-switching and same initialization bridge the spaces and transfer knowledge\. ### H\.2 Results #### Within models\. At 360M, the baseline already has strong global and local structural similarity \(Table[H\.1](https://arxiv.org/html/2609.19291#A8.T1)\)\. Direct token correspondence is nevertheless negligible \(matched cosine0\.0120\.012\), while mapped cosine reaches0\.5600\.560on held\-out tokens\. A single linear map therefore recovers substantial correspondence absent from the raw space\. The 7B baseline repeats this pattern\. We further analyze the embeddings of two models that achieve high knowledge transfer\. The model trained with the same initialization for corresponding tokens \(§[5\.4](https://arxiv.org/html/2609.19291#S5.SS4)\) and substantial code\-switching in 60% of the documents, which transfers well in theEnglish1–English2setting, unlike the more realisticEnglish–Arabic\(§[4\.2](https://arxiv.org/html/2609.19291#S4.SS2)\)\. In both models, we observe a sharp rise in matched cosine similarity alongside the jump inCLeq\(Table[H\.1](https://arxiv.org/html/2609.19291#A8.T1)\)\. Global and local similarities also increase from already strong baselines, indicating that the main change is direct token alignment\. Figure H\.1:Pairwise null\-calibrated global \(linear CKA; left\) and local \(mutualkk\-NN; right\) embedding similarity\. All models learn a common English geometry; code\-switching and same initialization additionally bridge its two tokenized copies, yielding higher within\-model similarity\. #### Across models\. At 360M, the baseline’s two halves fall within the same CKA and mutualkk\-NN ranges as cross\-model pairs \(Figure[H\.1](https://arxiv.org/html/2609.19291#A8.F1)\)\. The baseline is thus no more structurally similar within one run than across runs: all 360M spaces learn a common English geometry\. Models that generalize add a stronger within\-model relation\. Across scales, global and local similarities are lower than within\-scale comparisons but remain substantial and well above the permutation baseline, despite differences in width and training budget\. Combined with high matched cosine in the transfer conditions \(Table[H\.1](https://arxiv.org/html/2609.19291#A8.T1)\), this shows that the interventions bridge corresponding tokens within a common geometry; shared geometry alone does not yield knowledge transfer\. ## Appendix IToken Overlap and Anchored Arabic Languages that share a writing system naturally exhibit partial token overlap through shared subwords, and this overlap correlates with better cross\-lingual generalization\([Ifergan et al\., 2025](https://arxiv.org/html/2609.19291#bib.bib30);[Kallini et al\., 2025](https://arxiv.org/html/2609.19291#bib.bib36)\)\. This appendix examines whether recreating that overlap artificially induces knowledge sharing\. It supports two claims in the main text: that the effectiveness of shared tokens depends on their frequency, not merely their number \(§[5\.3](https://arxiv.org/html/2609.19291#S5.SS3)\), and that partial token\-space overlap between natural languages does not deliver the gains of full merging \(§[6\.2](https://arxiv.org/html/2609.19291#S6.SS2)\)\. ### I\.1 Controlled Overlap forEnglish1–English2 We first sweep token overlap inEnglish1–English2, where a subset of tokens is shared outright between the two copies while the remainder stay disjoint\. Generalization rises steeply with the shared subset: sharing the 2% most frequent tokens already yieldsCLeq=59\.9%\\text\{CL\\textsubscript\{eq\}\}=59\.9\\%, reaching 89\.0% at 64% overlap \(Table[I\.1](https://arxiv.org/html/2609.19291#A9.T1)\)\. Sharing a subset of tokens completely is thus far more effective than sharing all tokens partially — 2% overlap outperforms 20% soft\-mapping by a wide margin \(App\.[F](https://arxiv.org/html/2609.19291#A6)\)\. Vocabulary share alone, however, does not determine the outcome\. We repeat the 8% condition with a different token subset — one covering 47% of running text rather than 69\.1% — andCLeqfalls from 66\.9% to 51\.7%\. The two conditions share the same fraction of the vocabulary and differ only in how frequently the shared tokens appear, so generalization tracks the*coverage*of the shared tokens rather than their count\. \(This second subset is the anchor set of the AnAr mapping introduced in §[I\.2](https://arxiv.org/html/2609.19291#A9.SS2)\.\) ### I\.2 Anchored Arabic Figure I\.1:An Arabic sentence under each representation\. AnAr replaces only words with a direct token\-to\-token dictionary match, leaving the rest in Arabic script; WWT\-Ar maps every word\. Both preserve Arabic word order \(notice native Arabic is written from right to left\) and grammar, unlike the English translation\.Following these results we construct AnAr, a mapping that replaces only those Arabic tokens with a direct token\-to\-token match in our dictionary, leaving the rest in Arabic script \(Figure[I\.1](https://arxiv.org/html/2609.19291#A9.F1)\)\. AnAr targets the same barrier as WWT\-Ar but far less aggressively, producing partial token overlap rather than a fully merged token space\. Practically, AnAr preserves sequence length, avoiding the fertility overhead of WWT\-Ar\. AnAr nonetheless fails to deliver meaningful knowledge generalization, withCLeq=1\.6%\\text\{CL\\textsubscript\{eq\}\}=1\.6\\%against the WWT\-Ar ceiling of 12\.6%\. The controlled sweep explains why: AnAr’s anchors comprise 8% of the vocabulary but cover only 47% of English and 34\.6% of Arabic running text, this low coverage, as we established, weakens the effect of partial vocabulary sharing \(App\.[I\.1](https://arxiv.org/html/2609.19291#A9.SS1)\), partially explaining the gap between effectiveness here and in theEnglish1–English2setup, more linguistic aspects may be in play\. OverlapCoverageCLeq*English1−English2\\text\{English\}\_\{1\}\-\\text\{English\}\_\{2\}*0%0%−2\.6\-2\.6%0\.1%11\.4%−1\.3\-1\.3%0\.5%29\.0%43\.1%2%50\.1%59\.9%4%58\.5%61\.2%8%69\.1%66\.9%8%47%51\.7%16%79\.2%68\.2%32%88\.5%77\.2%64%96\.1%89\.0%*English−Arabic\\text\{English\}\-\\text\{Arabic\}*0%0%0\.9%*English−AnAr\\text\{English\}\-\\text\{AnAr\}*8%47%/34\.6%1\.6%*English−WWT\-Ar\\text\{English\}\-\\text\{WWT\-Ar\}*100%100%12\.6%Table I\.1:Knowledge generalization as a function of token overlap\.*Overlap*is the share of vocabulary types shared between the two languages, and*coverage*is the share of training tokens that they account for, reported per side where the two differ\. The controlled setting achieves far higherCLeqthan the natural one, while within each settingCLeqincreases with overlap\. These experiments were performed at the 360M parameter scale\. ## Appendix JLanguage Modeling under WWT The main text reports that WWT improves English perplexity \(§[6\.2](https://arxiv.org/html/2609.19291#S6.SS2)\)\. This appendix provides the corresponding non\-English measurements, so that the gain can be read against what it costs on the mapped side, and reports the monolingual controls that isolate how much each representation extracts from the English half of the corpus \(Tab\.[J\.1](https://arxiv.org/html/2609.19291#A10.T1)\)\. ### J\.1 Measurement We report bits per byte \(based on Arabic script byte count\) — BPB — rather than perplexity\. WWT\-Ar and native\-script Arabic differ in token fertility \(App\.[G\.3](https://arxiv.org/html/2609.19291#A7.SS3)\), and a per\-token measure would mechanically favour the higher\-fertility representation: each WWT\-Ar token carries less of the underlying text and is correspondingly easier to predict\. ### J\.2 Arabic\-Side Modeling In the bilingual setting, mapping Arabic to WWT\-Ar costs little on the Arabic side: at 360M parameters bits per byte rises from0\.71290\.7129to0\.71600\.7160, a relative degredation of less the 0\.5%\. similarly, at 7B parameter scale BPB goesfrom0\.56790\.5679to0\.57410\.5741, degradation of1\.1%1\.1\\%\. This is a small cost for the14×14\\timesimprovement inCLeqreported in §[6\.2](https://arxiv.org/html/2609.19291#S6.SS2), and it is achieved while the WWT\-Ar model is exposed to roughly 23% fewer Arabic documents under our fixed token budget \(§[6\.1](https://arxiv.org/html/2609.19291#S6.SS1)\)\. ### J\.3 Monolingual Controls Bilingual numbers alone cannot say how much of each model’s Arabic\-side performance comes from the English half of the corpus\. We therefore train monolingual counterparts for both representations and compare each bilingual model against its monolingual baseline \(Table[J\.1](https://arxiv.org/html/2609.19291#A10.T1)\), trained only on the Arabic half of the data \(30% less data for WWT\-Ar\)\. The quantity of interest is the*improvement*from adding English: if the unified token space allows the model to draw more from English, WWT\-Ar should show the larger improvement\. Indeed, we find that the monolingual Arabic model outperforms the WWT\-Ar model with BPB of0\.71550\.7155compared to0\.73440\.7344\(2\.6% difference\), however, the WWT\-Ar model gains much more from the cross\-lingual data — aΔ\\Deltaof 0\.0184 compared to 0\.0026, roughly7×7\\timesmatching directionalCLeqmeasurement gain \(Tab\.[J\.1](https://arxiv.org/html/2609.19291#A10.T1)\)\. Table J\.1:Non\-English language modeling in bits per byte \(BPB\), normalized by native\-script bytes throughout \(§[J\.1](https://arxiv.org/html/2609.19291#A10.SS1)\); lower is better\.*Bilingual*models are trained on a 50/50 token split with English,*monolingual*models on the non\-English half alone\.Δ\\Deltais the improvement contributed by the English half\. Experiments performed at the 360M parameter scale\. ## Appendix KShuffled\-Mapping Controls Our WWT operation is overwhelmingly a translation rather than a transliteration operation \(§[6\.1](https://arxiv.org/html/2609.19291#S6.SS1)\), meaning it maps Arabic tokens or token sequences to their semantically matching English counterparts\. This appendix provides implementation details for the experiments performed in §[6\.4](https://arxiv.org/html/2609.19291#S6.SS4)to determine the importance of this property as opposed to plain token superposition\. We run three controls that preserve token sharing while destroying correspondence, and find they all fail\. We perform these experiments at the 360M parameter scale\. ### K\.1 Shuffled WWT Dictionary We destroy semantic correspondences by shuffling the keys and targets of our large\-scale dictionaries used for WWT, then proceeding with the WWT process as usual, mapping words to their shuffled dictionary mapping and transliterating out\-of\-vocabulary entries\. We find this process hurts knowledge generalization between the two languages compared to our unshuffled WWT baseline, with English–WWT\-ArCLeqfalling from 12\.6% to 3%, and English–WWT\-Ru falling from 23\.3% to 3\.6%\. ### K\.2 Transliteration Transliteration\-only mapping provides a similar effect: a unified token space, but tokens aren’t mapped to their semantic counterparts\. We perform the same reversible transliteration used for WWT but apply it to all words\. Essentially applying WWT with an empty dictionary\. We find that transliterating Arabic does not help with resolving compartmentalization between it and English, with aCLeqscore of−0\.3%\-0\.3\\%\. ### K\.3 PermutedEnglish1–English2 We additionally pretrain a model on two language clones —English1–English2— in which both languages occupy a single shared token space, but the token identities of one copy are permuted:σ:\{0,…,V−1\}→\{0,…,V−1\}\\sigma:\\\{0,\\dots,V\-1\\\}\\to\\\{0,\\dots,V\-1\\\}\. Formally, while a text sequence is tokenized in English1as an array of token IDs\(t1,t2,…,tn\)\(t\_\{1\},t\_\{2\},\\dots,t\_\{n\}\), English2tokenizes the parallel sequence into\(σ\(t1\),σ\(t2\),…,σ\(tn\)\)\(\\sigma\(t\_\{1\}\),\\sigma\(t\_\{2\}\),\\dots,\\sigma\(t\_\{n\}\)\), both share the same\{0,…,V−1\}\\\{0,\\dots,V\-1\\\}token space This provides a fully controlled environment where the entire token inventory is shared and the two languages remain structurally identical, with the only barrier to generalization being the mismatch between token idetities\. Yet, we observe no knowledge generalization with aCLeqscore of \-2\.6% — same as completely disjoint token spaces\. ### K\.4 Interpretation Shared tokens transfer knowledge only when they carry corresponding meanings\. Resolving disjoint tokenization is therefore not a matter of token superposition, and a semantically faithful mapping is necessary\. The transliteration control is notable, as previous work has suggested transliteration as a path to unlocking multilingual ability\([Moosa et al\., 2023](https://arxiv.org/html/2609.19291#bib.bib39);[J et al\., 2024](https://arxiv.org/html/2609.19291#bib.bib63);[Saji et al\., 2025](https://arxiv.org/html/2609.19291#bib.bib65);[Jayakumar et al\., 2026](https://arxiv.org/html/2609.19291#bib.bib72)\), we note that successful examples in past literature focus mostly on related languages where cognates are common and therefore transliteration can supply the semantic matches we find necessary\. SettingToken spaceSemantic mappingCLeq*English−Arabic\\text\{English\}\-\\text\{Arabic\}*Native Arabic \(baseline\)disjoint—0\.9%WWT\-Arsharedyes12\.6%Shuffled WWT\-Arsharedno3\.0%Transliterated Arabicsharedno−0\.3\-0\.3%*English−Russian\\text\{English\}\-\\text\{Russian\}*Native Russian \(baseline\)disjoint—1\.9%WWT\-Rusharedyes23\.3%Shuffled WWT\-Rusharedno3\.6%*English1−English2\\text\{English\}\_\{1\}\-\\text\{English\}\_\{2\}*Disjoint copies \(baseline\)disjoint—−2\.6\-2\.6%Fully shared copiessharedyes100%Permuted copiessharedno−2\.6\-2\.6%Table K\.1:Semantic controls at 360M\. Each block pairs a semantically faithful shared token space against controls that share tokens without sharing meanings\. Sharing a token inventory without semantic correspondence performs at or near the disjoint\-token baseline in every pair, while the semantically faithful mapping is the only condition that improves generalization\. forEnglish1–English2Fully shared copies are atCLeqof 100% by definition
Similar Articles
An In-Vitro Study on Cross-Lingual Generalization in Language Models
This paper introduces an in-vitro framework with two procedurally generated languages to study cross-lingual generalization in language models, finding that tokenization's preservation of reusable substructure is more critical than lexical similarity or data balance for transferring capabilities across languages.
Cross-Lingual Exploration for Parametric Knowledge
This paper explores cross-lingual prompting strategies to improve access to parametric knowledge in large language models, demonstrating significant gains in knowledge transfer and factual recall across 17 languages on multilingual benchmarks.
How to Use Parallel Data for Multilingual Pretraining
The paper proposes Parallel Pretraining (PPT), which concatenates documents with their translations in the same training sequence, and finds that a two-stage PPT-then-standard-pretraining approach consistently improves general language understanding and cross-lingual performance over standard pretraining at matched token budgets.
Instability of LLM Pre-Pretraining: It Doesn't Always Help. An Investigation on Multiple Languages
This paper investigates whether pretraining LLMs on artificial languages (pre-pretraining) consistently improves token efficiency across multiple natural languages, finding that gains are highly dependent on experimental setup and random seed, though stable gains appear for small models with the Llama tokenizer.
Notes on pretraining parallelisms and failed training runs (12 minute read)
A technical deep-dive into common causes of failed pretraining runs in large language models, including causality-breaking issues in expert routing and numerical precision bugs, with examples from Llama 4, Gemini 2 Pro, and GPT-4.