An In-Vitro Study on Cross-Lingual Generalization in Language Models

# An In-Vitro Study on Cross-Lingual Generalization in Language Models
Source: [https://arxiv.org/html/2605.26683](https://arxiv.org/html/2605.26683)
###### Abstract

Cross\-lingual transfer in language models is difficult to study in natural corpora because lexical overlap, morphology, data imbalance, and tokenization are entangled\. We introduce anin\-vitroframework with two procedurally generated languages that share the same ontology, typed grammar, and compositional structure, but differ in surface realization\. This lets us independently vary lexical distance, minority\-language proportion, tokenizer training regime, and vocabulary size, while evaluating transfer on a masked minority\-language condition whose lexical forms are never observed during training\. Across 700 controlled runs, we find that transfer is governed less by tokenizer balance or raw lexical similarity than by whether tokenization preserves reusable cross\-lingual substructure\. Smaller vocabularies often improve masked transfer by keeping words decomposable into shared fragments, whereas larger vocabularies can turn forms into language\-specific atoms\. We further show that transfer emerges as a staged process: grammatical and type\-level competence precede masked lexical generalization\. Finally, we attempt to explain this mechanism throughtokenizer bridgesand show that bridge strength correlates strongly with masked reachability\.

AnIn\-VitroStudy on Cross\-Lingual Generalization in Language Models

Adrian CosmaDalle Molle Institute for Artificial Intelligence \(IDSIA\)adrian\.cosma@idsia\.ch

## 1Introduction

Language models \(LMs\) exhibit cross\-lingual transfer: training on one language can improve performance in another even without direct supervision in the target languageZhaoet al\.\([2024](https://arxiv.org/html/2605.26683#bib.bib56)\); De Souzaet al\.\([2024](https://arxiv.org/html/2605.26683#bib.bib50)\)\. Yet the mechanism behind this transfer remains unclear\. In this work, we ask the question:How are semantic language capabilities transferred during training through surface forms?

Prior work suggests that multilingual models develop both language\-specific and shared language\-agnostic representationsZhaoet al\.\([2024](https://arxiv.org/html/2605.26683#bib.bib56)\); De Souzaet al\.\([2024](https://arxiv.org/html/2605.26683#bib.bib50)\), but existing evidence remains mixed on the role of language similarity and data compositionMalkinet al\.\([2022](https://arxiv.org/html/2605.26683#bib.bib30)\); Limisiewiczet al\.\([2023b](https://arxiv.org/html/2605.26683#bib.bib43)\), while tokenization has repeatedly emerged as a critical interface between surface form and meaningRustet al\.\([2021](https://arxiv.org/html/2605.26683#bib.bib25)\); Patilet al\.\([2022](https://arxiv.org/html/2605.26683#bib.bib32)\); Hämmerlet al\.\([2025](https://arxiv.org/html/2605.26683#bib.bib62)\)\. However, in natural multilingual corpora, lexical overlap, morphology, script, domain, corpus size, and tokenization all vary at once, making it difficult to determine which factors actually enable knowledge learned in one language to become useful in anotherBlevins and Zettlemoyer \([2022](https://arxiv.org/html/2605.26683#bib.bib28)\)\.

In this work, we study cross\-lingual generalizationin vitro\. We build a framework to procedurally generate two languages,𝐀\\mathbf\{A\}and𝐁\\mathbf\{B\}, that share the same underlying ontology, typed grammar, and compositional structure, but differ in their lexical realization\. This lets us parameterize the generation process with three factors that are usually confounded in natural data: lexical distance between languages, the relative amount of training data in each language, and the granularity induced by subword tokenization\. We further define a masked minority\-language condition,𝐁†\\mathbf\{B\}^\{\\dagger\}, by withholding a subset of lexical realizations from language𝐁\\mathbf\{B\}during training\. Success on𝐁†\\mathbf\{B\}^\{\\dagger\}therefore requires the model to transfer shared compositional structure across languages rather than merely memorize minority\-language surface forms\. Our setup enables the study of multilingual transfer in a controlled manner, pertaining to the larger context oflearning mechanicsMichaudet al\.\([2023](https://arxiv.org/html/2605.26683#bib.bib44)\); Liuet al\.\([2025](https://arxiv.org/html/2605.26683#bib.bib65)\); Allen\-Zhu and Li \([2025](https://arxiv.org/html/2605.26683#bib.bib38)\)\.

In particular, our setup enables us to test the following hypotheses from literature:

1. ℋ1\\mathcal\{H\}\_\{1\}Tokenizer Influence:Cross\-lingual transfer depends less on tokenizer balance itself than on whether tokenization preserves shared subword bridges between languagesRustet al\.\([2021](https://arxiv.org/html/2605.26683#bib.bib25)\); Patilet al\.\([2022](https://arxiv.org/html/2605.26683#bib.bib32)\); Hämmerlet al\.\([2025](https://arxiv.org/html/2605.26683#bib.bib62)\)
2. ℋ2\\mathcal\{H\}\_\{2\}Lexical Similarity:Lexically similar languages will result in easier cross\-lingual transferKunchukuttanet al\.\([2018](https://arxiv.org/html/2605.26683#bib.bib15)\); Gooskens and Swarte \([2017](https://arxiv.org/html/2605.26683#bib.bib13)\); Hernandezet al\.\([2021](https://arxiv.org/html/2605.26683#bib.bib23)\)
3. ℋ3\\mathcal\{H\}\_\{3\}Language Proportions:Increasing the amount of minority\-language data should improve target language capabilities, though evidence suggests that the benefits are condition\-dependentMalkinet al\.\([2022](https://arxiv.org/html/2605.26683#bib.bib30)\); Limisiewiczet al\.\([2023b](https://arxiv.org/html/2605.26683#bib.bib43)\)\.
4. ℋ4\\mathcal\{H\}\_\{4\}Skill Cascade:Cross\-lingual transfer is preceded by a cascade of skills that are required: learning the majority language \(grammar, type constraints\), learning the minority language \(grammar, type constraints\), then generalize to unseen conceptsBlevinset al\.\([2022](https://arxiv.org/html/2605.26683#bib.bib27)\); Lubanaet al\.\([2025](https://arxiv.org/html/2605.26683#bib.bib53)\); Wanget al\.\([2024](https://arxiv.org/html/2605.26683#bib.bib54)\)\.

We attempt to formalize the cross\-lingual transfer mechanism throughtokenizer bridges\. For each masked lexical item, we measure how strongly its tokenization overlaps with observed lexical forms under the same tokenizer\. We find that this bridge strength correlates with masked reachability, suggesting that transfer improves when unseen target\-language forms remain connected to observed forms through shared subword fragments\. The resulting view is that multilingual generalization is a property of the tokenized topology through which languages are presented to the model\.

Our contributions are as follows:

1. 1\.We introduce a controlledin\-vitroframework for studying cross\-lingual transfer under independently varied lexical distance, language proportion, tokenizer regime, and vocabulary size\.
2. 2\.We define a masked minority\-language condition that isolates cross\-lingual generalization from memorization of observed target\-language forms\.
3. 3\.We show that smaller vocabularies and reusable subword structure can improve masked transfer, while larger vocabularies may suppress it by isolating languages into more language\-specific units\.
4. 4\.We proposetokenizer bridgesas a potential explanation for masked cross\-lingual transfer and show that bridge strength predicts masked reachability\.

## 2Related Work

Prior work suggests that multilingual LMs develop both language\-specific and language\-agnostic representationsZhaoet al\.\([2024](https://arxiv.org/html/2605.26683#bib.bib56)\); De Souzaet al\.\([2024](https://arxiv.org/html/2605.26683#bib.bib50)\)\. Evidence for shared internal structure comes from probing studies identifying overlapping or language\-specific neuronsWendleret al\.\([2024](https://arxiv.org/html/2605.26683#bib.bib55)\); Zhaoet al\.\([2024](https://arxiv.org/html/2605.26683#bib.bib56)\), from synthetic pretraining experiments showing transfer across unrelated surface formsPapadimitriou and Jurafsky \([2020](https://arxiv.org/html/2605.26683#bib.bib20)\), and from studies of multilingual pretraining mixturesDe Souzaet al\.\([2024](https://arxiv.org/html/2605.26683#bib.bib50)\)\. These findings suggest that LMs can abstract away from surface realization and capture latent structure shared across languages\.

At the same time, cross\-lingual transfer is strongly shaped by the tokenization interface through which text enters the model\. Unlike humans, whose mutual intelligibility is mediated by vocabulary, phonology, and syntaxGooskens and Van Heuven \([2021](https://arxiv.org/html/2605.26683#bib.bib22)\); Escudero and Boersma \([2004](https://arxiv.org/html/2605.26683#bib.bib9)\); Gooskens and Swarte \([2017](https://arxiv.org/html/2605.26683#bib.bib13)\), LMs operate over subword units induced by data\-driven tokenizers\. This can introduce systematic unfairness across languages, especially when tokenizer training data is dominated by high\-resource languagesLimisiewiczet al\.\([2023a](https://arxiv.org/html/2605.26683#bib.bib42)\); Petrovet al\.\([2023](https://arxiv.org/html/2605.26683#bib.bib47)\); Arnettet al\.\([2026](https://arxiv.org/html/2605.26683#bib.bib58)\); Hämmerlet al\.\([2025](https://arxiv.org/html/2605.26683#bib.bib62)\); Patilet al\.\([2022](https://arxiv.org/html/2605.26683#bib.bib32)\)\. Prior work has therefore explored both tokenizer\-level interventionsAbagyanet al\.\([2025](https://arxiv.org/html/2605.26683#bib.bib57)\); Jabbar \([2023](https://arxiv.org/html/2605.26683#bib.bib41)\)and architecture\-level alternativesXueet al\.\([2022](https://arxiv.org/html/2605.26683#bib.bib37)\); Ahiaet al\.\([2024](https://arxiv.org/html/2605.26683#bib.bib48)\); Cosmaet al\.\([2025](https://arxiv.org/html/2605.26683#bib.bib61)\)\. However, the interaction between language similarity, data proportions, and tokenization remains difficult to isolate in natural multilingual corpora\.

This work\.We study multilingual transfer in a controlled synthetic setting that isolates language similarity, training proportions, and tokenizer parameters\. Instead of relying on natural languages, where domain, ontology, and lexical similarity are heavily confounded, we construct programmatic languages over a shared semantic spaceTaguchi and Sproat \([2026](https://arxiv.org/html/2605.26683#bib.bib69)\)\. Building onLubanaet al\.\([2025](https://arxiv.org/html/2605.26683#bib.bib53)\), we generate controlled lexical realizations on top of a Probabilistic Context\-Sensitive Grammar with type constraints\. Our evaluation tests generalization to concept realizations unseen during training, across multiple abstraction levels: word validity, grammaticality, constraint satisfaction, and lexical reachability of masked concepts\. This allows us to examine when multilingual transfer emerges during training, and how semantic structure is transferred across distinct lexical forms\. More broadly, our setup follows a line of work using formal grammars and synthetic data to expose learning dynamics that are difficult to identify in natural data with many confoundersPapadimitriou and Jurafsky \([2023](https://arxiv.org/html/2605.26683#bib.bib46)\); Lubanaet al\.\([2025](https://arxiv.org/html/2605.26683#bib.bib53)\); Allen\-Zhu and Li \([2025](https://arxiv.org/html/2605.26683#bib.bib38)\); Huet al\.\([2025](https://arxiv.org/html/2605.26683#bib.bib63)\); Blevins and Zettlemoyer \([2022](https://arxiv.org/html/2605.26683#bib.bib28)\)\.

Figure 1:Overview of the in\-vitro cross\-lingual setup\.Two procedurally generated languages,𝐀\\mathbf\{A\}and𝐁\\mathbf\{B\}, share the same underlying grammar, type constraints, and concept–property ontology, but differ in their lexical realizations\. Cross\-lingual difficulty is controlled through lexical distancedd, minority\-language proportionλ\\lambda, tokenizer type, and tokenizer vocabulary size\. A subset of minority\-language lexical forms is withheld to define the masked condition𝐁†\\mathbf\{B\}^\{\\dagger\}, enabling evaluation of whether models transfer shared compositional structure rather than memorizing surface forms\. Subword tokenization mediates overlap between the two languages\.
## 3In\-Vitro Cross\-Lingual Setup

### 3\.1Setup Formalization

Our objective is to isolate how cross\-lingual transfer depends on lexical similarity between languages, language proportions in the training corpus, and subword tokenization granularity\.

In our framework, we model language as a structured symbolic system specified by\(i\)a vocabulary of abstract symbols,\(ii\)a type signature over those symbols,\(iii\)a grammar that defines syntactically well\-formed sentences, and\(iv\)a realization function that maps abstract symbols to surface language\-specific word forms\. We extend the setup ofLubanaet al\.\([2025](https://arxiv.org/html/2605.26683#bib.bib53)\)to account for multi\-language realizations on top of the typed grammar\. Readers are referred to the original work for more details on the framework implementation\. As such, we do not treat language in the full sociolinguistic senseBallard \([1980](https://arxiv.org/html/2605.26683#bib.bib4)\), but rather as a finite generative system with a shared abstract layer and language\-specific realizations\.

We study cross\-lingual generalization in a controlled multilingual pretraining setting with two procedurally generated languages, denoted with𝐀\\mathbf\{A\}and𝐁\\mathbf\{B\}\. For notational convenience, we index them byℓ∈\{\\ell\\in\\\{𝐀\\mathbf\{A\},𝐁\\mathbf\{B\}\}\\\}\. We also define a masked evaluation condition, denoted𝐁†\\mathbf\{B\}^\{\\dagger\}, which is derived from𝐁\\mathbf\{B\}by withholding a subset of minority\-language lexical realizations during training\. All three conditions share the same underlying ontology and compositional rules, but differ in their lexical surface realization\. We show in Figure[1](https://arxiv.org/html/2605.26683#S2.F1)a diagram of our setup\.

#### Symbolic system and language realization\.

A language is a 6\-tuple:ℓ=\(𝒱,𝒯,τ,G,Γ,ρℓ\)\\ell=\(\\mathcal\{V\},\\mathcal\{T\},\\tau,G,\\Gamma,\\rho\_\{\\ell\}\), where:

- •𝒱\\mathcal\{V\}is a finite inventory of abstract lexical symbols;
- •𝒯\\mathcal\{T\}is a finite set of types;
- •τ:𝒱→𝒯\\tau:\\mathcal\{V\}\\to\\mathcal\{T\}assigns each symbol a type;
- •GGis a probabilistic grammar over𝒱\\mathcal\{V\}that generates well\-formed symbolic sequences;
- •Γ⊆𝒞×𝒫\\Gamma\\subseteq\\mathcal\{C\}\\times\\mathcal\{P\}is an ontology of semantically valid concept–property pairs, where𝒞,𝒫⊆𝒱\\mathcal\{C\},\\mathcal\{P\}\\subseteq\\mathcal\{V\}\.
- •ρℓ:𝒱→Σℓ∗\\rho\_\{\\ell\}:\\mathcal\{V\}\\to\\Sigma\_\{\\ell\}^\{\*\}, a realization function which maps abstract symbols to surface forms in languageℓ\\ellto a vocabularyΣℓ\\Sigma\_\{\\ell\}\.

In particular, we defineGGas a*Probabilistic Context\-Sensitive Grammar*\(PCSG\) \(see Appendix[A](https://arxiv.org/html/2605.26683#A1)\)\. For the ontology, let𝒞=\{c1,…,cNC\}\\mathcal\{C\}=\\\{c\_\{1\},\\dots,c\_\{N\_\{C\}\}\\\}and𝒫=\{p1,…,pNP\}\\mathcal\{P\}=\\\{p\_\{1\},\\dots,p\_\{N\_\{P\}\}\\\}be finite sets of*concepts*and*properties*, respectively\. We assume a typed relational structureΓ⊆𝒞×𝒫\\Gamma\\subseteq\\mathcal\{C\}\\times\\mathcal\{P\}, where\(c,p\)∈Γ\(c,p\)\\in\\Gammaindicates that propertyppis semantically valid for conceptcc\. This defines the ontology shared by all language conditions\. Intuitively,Γ\\Gammaspecifies which attributes may describe which entities\. For example, this encodes relationships such as\(hiker, climbs\)as valid but\(table, eats\)as invalid\.

This separates syntax from semantics and from lexical realization:GGdetermines which symbolic sequences are structurally well formed, whileΓ\\Gammaconstrains which concept\-property combinations are semantically valid\.

#### Symbolic sentence generation\.

A sentence is generated in three stages\. First, a symbolic templatez=\(v1,…,vn\)∈𝒱nz=\(v\_\{1\},\\dots,v\_\{n\}\)\\in\\mathcal\{V\}^\{n\}is sampled from the grammarGG\.

Second, the sampled symbols must satisfy the relevant type and ontology constraints\. For example, ifzzdescribes a conceptc∈𝒞c\\in\\mathcal\{C\}with a propertyp∈𝒫p\\in\\mathcal\{P\}, then\(c,p\)∈Γ\(c,p\)\\in\\Gammamust hold\. Third, the symbolic sentence is realized in languageLLby applyingρℓ\\rho\_\{\\ell\}tokenwise:ρℓ​\(z\)=\(ρℓ​\(v1\),…,ρℓ​\(vn\)\)\\rho\_\{\\ell\}\(z\)=\\bigl\(\\rho\_\{\\ell\}\(v\_\{1\}\),\\dots,\\rho\_\{\\ell\}\(v\_\{n\}\)\\bigr\)\.

This yields a corpora in different languages that are semantically aligned at the symbolic level but may differ arbitrarily in surface form\.

### 3\.2Multilingual Dataset Generation

#### Lexical realization and lexical distance\.

The two languages𝐀\\mathbf\{A\}and𝐁\\mathbf\{B\}share the same abstract vocabulary𝒱\\mathcal\{V\}, type systemτ\\tau, grammarGG, and ontologyΓ\\Gamma, but differ in their realization functionsρ𝐀\\rho\_\{\\color\[rgb\]\{0\.15625,0\.4765625,0\.72265625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.15625,0\.4765625,0\.72265625\}\\mathbf\{A\}\}andρ𝐁\\rho\_\{\\color\[rgb\]\{0\.85546875,0\.25390625,0\.40234375\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.85546875,0\.25390625,0\.40234375\}\\mathbf\{B\}\}\. We construct lexical realization by assuming languages are agglutinativeArkadiev \([2019](https://arxiv.org/html/2605.26683#bib.bib16)\)and constructed from concatenation of different morphemes: language\-specific prefixes, stems and suffixes\. Furthermore, each language has its specific pre\-defined frequency distribution of characters\.

Experimentally, we control cross\-lingual lexical similarity through a scalar parameterd∈\[0,1\]d\\in\[0,1\], where larger values ofddcorrespond to stronger perturbations of the latent stems and therefore greater lexical divergence between𝐀\\mathbf\{A\}and𝐁\\mathbf\{B\}\. This roughly corresponds to a lexical similarity distanceLevenshtein and others \([1966](https://arxiv.org/html/2605.26683#bib.bib2)\)used in linguistic studiesDinuet al\.\([2023a](https://arxiv.org/html/2605.26683#bib.bib40)\)\. For example, we could have the stem"caqujna", which leads to the two words"nawcaqjna"and"agcaqujna"in𝐀\\mathbf\{A\}and𝐁\\mathbf\{B\}, respectively\. The stem is being modified and each language added its own prefix\. In Appendix[A](https://arxiv.org/html/2605.26683#A1)we provide the pseudocode for word formation and show several examples of generated sentences\. We discuss limitations of this approach in the[Limitations](https://arxiv.org/html/2605.26683#Sx1)section\.

#### Training corpus mixture\.

Let𝒟A\\mathcal\{D\}\_\{A\}and𝒟B\\mathcal\{D\}\_\{B\}denote corpora of realized training sequences in𝐀\\mathbf\{A\}and𝐁\\mathbf\{B\}, respectively, both generated from the same underlying symbolic process\. We vary the language mixture by controlling the corpus ratio\|𝒟B\|\|𝒟A\|=λ\\frac\{\|\\mathcal\{D\}\_\{B\}\|\}\{\|\\mathcal\{D\}\_\{A\}\|\}=\\lambda\. We maintain𝐁\\mathbf\{B\}as the minority language, by ensuringλ∈\(0,0\.5\]\\lambda\\in\(0,0\.5\]\. Varyingλ\\lambdaallows us to separate effects of lexical similarity from effects of training language proportions\.

#### Masked minority\-language condition\.

To test genuine cross\-lingual transfer rather than direct memorization of minority\-language surface forms, we define a masked evaluation condition𝐁†\\mathbf\{B\}^\{\\dagger\}\. Let𝒱†⊂𝒱\\mathcal\{V\}^\{\\dagger\}\\subset\\mathcal\{V\}be a subset of lexical symbols whose realizations in language𝐁\\mathbf\{B\}are withheld during model training\. We then define

𝒟𝐁†=\{x∈𝒟𝐁:x​has​ρ𝐁​\(v\)​for some​v∈𝒱†\}\\mathcal\{D\}\_\{\{\\color\[rgb\]\{0\.85546875,0\.25390625,0\.40234375\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.85546875,0\.25390625,0\.40234375\}\\mathbf\{B\}^\{\\dagger\}\}\}=\\left\\\{x\\in\\mathcal\{D\}\_\{\{\\color\[rgb\]\{0\.85546875,0\.25390625,0\.40234375\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.85546875,0\.25390625,0\.40234375\}\\mathbf\{B\}\}\}:x\\text\{ has \}\\rho\_\{\{\\color\[rgb\]\{0\.85546875,0\.25390625,0\.40234375\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.85546875,0\.25390625,0\.40234375\}\\mathbf\{B\}\}\}\(v\)\\text\{ for some \}v\\in\\mathcal\{V\}^\{\\dagger\}\\right\\\}
The remaining minority\-language examples are

𝒟𝐁seen=𝒟𝐁∖𝒟𝐁†\\mathcal\{D\}\_\{\\color\[rgb\]\{0\.85546875,0\.25390625,0\.40234375\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.85546875,0\.25390625,0\.40234375\}\\mathbf\{B\}\}^\{\\mathrm\{seen\}\}=\\mathcal\{D\}\_\{\\color\[rgb\]\{0\.85546875,0\.25390625,0\.40234375\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.85546875,0\.25390625,0\.40234375\}\\mathbf\{B\}\}\\setminus\\mathcal\{D\}\_\{\{\\color\[rgb\]\{0\.85546875,0\.25390625,0\.40234375\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.85546875,0\.25390625,0\.40234375\}\\mathbf\{B\}^\{\\dagger\}\}\}, and the training corpus is𝒟train=𝒟𝐀∪𝒟𝐁seen\.\\mathcal\{D\}\_\{\\mathrm\{train\}\}=\\mathcal\{D\}\_\{\\color\[rgb\]\{0\.15625,0\.4765625,0\.72265625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.15625,0\.4765625,0\.72265625\}\\mathbf\{A\}\}\\cup\\mathcal\{D\}\_\{\\color\[rgb\]\{0\.85546875,0\.25390625,0\.40234375\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.85546875,0\.25390625,0\.40234375\}\\mathbf\{B\}\}^\{\\mathrm\{seen\}\}\.

Crucially, the underlying symbolic structures associated with𝒱†\\mathcal\{V\}^\{\\dagger\}may still be observed through𝐀\\mathbf\{A\}\. Success on𝐁†\\mathbf\{B\}^\{\\dagger\}therefore requires transferring shared compositional and semantic structure, rather than relying on direct exposure to the withheld𝐁\\mathbf\{B\}lexical forms\. We mask 25% of the property symbols for𝐁\\mathbf\{B\}\.

#### Tokenizer\.

Before model training, realized text is segmented by a subword tokenizerπ:Σ𝐀∗​⋃Σ𝐁∗→V∗\\pi:\\Sigma^\{\*\}\_\{\\color\[rgb\]\{0\.15625,0\.4765625,0\.72265625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.15625,0\.4765625,0\.72265625\}\\mathbf\{A\}\}\\bigcup\\Sigma^\{\*\}\_\{\\color\[rgb\]\{0\.85546875,0\.25390625,0\.40234375\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.85546875,0\.25390625,0\.40234375\}\\mathbf\{B\}\}\\to V^\{\*\}, whereVVis a learned BPE vocabulary of size\|V\|\|V\|\. We consider two tokenizer\-training regimes:*vanilla*\(1:λ1:\\lambda\) and*balanced*\(1:11:1\)\. In the more realistic*vanilla*condition, tokenizer training uses the same language mixture as model training; in the*balanced*condition, tokenizer training uses equal proportions of examples from both languages and is decoupled from the model\-training mixture\. In particular, in both cases, we train a BPE tokenizerGage \([1994](https://arxiv.org/html/2605.26683#bib.bib6)\)with whitespace pre\-tokenization on a dataset of fixed size\. Since tokenization determines the atomic units through which the model observes surface form, it acts as a bottleneck between lexical realization and cross\-lingual abstractionLimisiewiczet al\.\([2023a](https://arxiv.org/html/2605.26683#bib.bib42)\); Petrovet al\.\([2023](https://arxiv.org/html/2605.26683#bib.bib47)\); Patilet al\.\([2022](https://arxiv.org/html/2605.26683#bib.bib32)\)\. In contrast to other worksDe Souzaet al\.\([2024](https://arxiv.org/html/2605.26683#bib.bib50)\)that use a byte tokenizerXueet al\.\([2022](https://arxiv.org/html/2605.26683#bib.bib37)\), we opt to vary tokenization parameters to control for the effect of subword tokenization\.

#### Model Training\.

We train a small decoder\-only transformer LM on sentences generated using the procedure described above\. FollowingLubanaet al\.\([2025](https://arxiv.org/html/2605.26683#bib.bib53)\), we construct several tasks for model training and evaluation: free generation \(T0\), unscrambling \(T1\) and conditional generation \(T2\)\. For each task\-language pair, we have special tokens to condition the model on which task it is supposed to perform and in which language\. As such, we have a total of 6 such special tokens \(3 tasks×\\times2 languages\)\. Examples are shown in Figure[1](https://arxiv.org/html/2605.26683#S2.F1)\.

#### Problem statement\.

Given two languages with a shared symbolic system, a lexical\-distance parameterdd, a corpus\-mixture parameterλ\\lambda, and a tokenizerπ\\pi, we ask:under what conditions does knowledge acquired from the majority language𝐀\\mathbf\{A\}become available in the minority language𝐁\\mathbf\{B\}, especially on the masked condition𝐁†\\mathbf\{B\}^\{\\dagger\}?Because𝒟𝐁†\\mathcal\{D\}\_\{\{\\color\[rgb\]\{0\.85546875,0\.25390625,0\.40234375\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.85546875,0\.25390625,0\.40234375\}\\mathbf\{B\}^\{\\dagger\}\}\}is excluded from training, performance on𝐁†\\mathbf\{B\}^\{\\dagger\}is a direct measure of cross\-lingual generalization through shared structure\.

### 3\.3Evaluation Setup

Instead of simply looking at perplexity, which is an insufficient measure of downstream performanceVeličkovićet al\.\([2026](https://arxiv.org/html/2605.26683#bib.bib71)\), we primarily measure grammaticality and type constraints satisfactionLubanaet al\.\([2025](https://arxiv.org/html/2605.26683#bib.bib53)\)\. Our evaluation is split intointra\-languageperformance, looking individually at how performance on𝐀\\mathbf\{A\}and𝐁\\mathbf\{B\}progresses, andcross\-languagegeneralization by measuring performance on the masked lexical realizations in𝐁†\\mathbf\{B\}^\{\\dagger\}\. We measure performance for taskT1T\_\{1\}\(unscrambling\) for both maximal metric value and emergence stepWeiet al\.\([2022](https://arxiv.org/html/2605.26683#bib.bib35)\); Bertiet al\.\([2025](https://arxiv.org/html/2605.26683#bib.bib36)\)\. We consider the definition of emergence byAnderson \([1972](https://arxiv.org/html/2605.26683#bib.bib3)\)111”Emergence is when quantitative changes in a system result in qualitative changes in behavior\.”Anderson \([1972](https://arxiv.org/html/2605.26683#bib.bib3)\), and we compute emergence step as the first training step for which a particular metric achieves a value greater than 0\.02\.

Furthermore, we introduce a modified version ofconcept reachabilityLubanaet al\.\([2025](https://arxiv.org/html/2605.26683#bib.bib53)\)that is designed to accommodate tokenized concepts\. In particular, we measureTop\-KKReachability\.The model is conditioned with the templates "T0T\_\{0\}\-𝐀\\mathbf\{A\}<subject\_id\> is" or "T0T\_\{0\}\-𝐁\\mathbf\{B\}<subject\_id\> is" and is required to generate the tokens to a valid descriptive property to the specifiedsubject\_idin the target language\. Since such properties could be lexically realized in multiple tokens, we test whether there exists a path where every next token is in the top\-KKnext tokens under the evolving context\. We fixK=0\.1×\|V\|K=0\.1\\times\|V\|, to compare between tokenizer vocabulary sizes\. Subjects are chosen such that the realized versions of the associated properties are not seen during model training for𝐁\\mathbf\{B\}, but are seen in𝐀\\mathbf\{A\}\. This directly measures cross\-lingual semantic transfer through lexical realizations\. We provide pseudocode in Appendix[A](https://arxiv.org/html/2605.26683#A1)\.

## 4Results

\(\(a\)\)Tokenizer fertility for𝐀\\mathbf\{A\},𝐁\\mathbf\{B\}, and𝐁†\\mathbf\{B\}^\{\\dagger\}based on lexical similarity, language ratio, and vocabulary size\.\(\(b\)\)Vocabulary overlap increases with amount of𝐁\\mathbf\{B\}data\.\(\(c\)\)Vocabulary overlap varies with lexical similarity\.
Figure 2:Tokenizer fertility and vocabulary overlap under different multilingual conditions\.Figure 3:Masked top\-KKreachability across training\.We report Top\-KKreachability in the masked minority\-language condition𝐁†\\mathbf\{B\}^\{\\dagger\}across tokenizer vocabulary sizes, language proportions, lexical similarities, and tokenizer training regimes\. Smaller vocabularies generally yield higher masked reachability, while larger vocabularies suppress transfer by inducing more language\-specific units\. Balanced tokenization improves reachability in some low\-resource settings, but the effect is not uniform across vocabulary sizes\.\(\(a\)\)Grammaticality and type\-constraint satisfaction across proportions, lexical similarity, vocabulary size, and tokenizer type\.
\(\(b\)\)Average emergence time for grammaticality and type\-constraint satisfaction across languages for vanilla tokenizer and\|V\|=2048\|V\|=2048\(top\) and across vocabulary sizes for𝐁†\\mathbf\{B\}^\{\\dagger\}\(bottom\)\. Lower values indicate earlier emergence during training\.
\(\(c\)\)Training dynamics for validity, grammaticality, and type\-constraint satisfaction in the masked condition𝐁†\\mathbf\{B\}^\{\\dagger\}\. Capabilities emerge in a cascaded manner, before type constraints can be satisfied, the model learns to construct valid words and grammatically correct sentences\.

Figure 4:Structural capabilities emerge before masked lexical transfer\.The results support a staged view of cross\-lingual transfer: models first acquire shared structural constraints before generalizing to unseen minority\-language lexical realizations\.We study cross\-lingual generalization along three axes: tokenizer training regime, tokenizer vocabulary size, and the relation between language proportion and lexical similarity\. We first analyse the tokenizer\-induced structure of the two languages, then evaluate whether this structure supports transfer to the masked minority\-language condition𝐁†\\mathbf\{B\}^\{\\dagger\}\. We then compare the emergence of different capabilities to determine whether masked lexical generalization appears as an isolated effect or as the last stage of a broader learning trajectory\. Finally, we attempt to explain masked cross\-lingual transfer throughtokenizer bridges\.

### 4\.1Main results

#### Tokenization drives cross\-lingual structure \(ℋ1\\mathcal\{H\}\_\{1\}\)\.

In Fig\.[2](https://arxiv.org/html/2605.26683#S4.F2)we show fertilityRustet al\.\([2021](https://arxiv.org/html/2605.26683#bib.bib25)\)and vocabulary overlap between the two languages, across multiple language parameters \(see Appendix[A](https://arxiv.org/html/2605.26683#A1), Fig\.[6](https://arxiv.org/html/2605.26683#A1.F6)for similar plots with continuation rate as the primary metric\)\. The tokenizer does not expose the model to the two languages symmetrically\. The minority language is more fragmented than the majority language\. This effect is strongest at smaller language ratios, where fewer𝐁\\mathbf\{B\}examples are available during tokenizer training\. The overlap plots in Fig\.[2](https://arxiv.org/html/2605.26683#S4.F2)show that increasing the tokenizer vocabulary size generally reduces shared subword structure between the two languages\. As the vocabulary grows, BPE increasingly memorizes longer language\-specific units rather than preserving reusable pieces shared across𝐀\\mathbf\{A\}and𝐁\\mathbf\{B\}\. This creates a tension: larger vocabularies reduce fragmentation within each language, but also make the two languages less coupled at the subword level\. Since transfer to𝐁†\\mathbf\{B\}^\{\\dagger\}requires the model to use information acquired from𝐀\\mathbf\{A\}, this loss of shared substructure can harm cross\-lingual generalization\.

#### Balanced tokenizer training is not uniformly better \(ℋ1\\mathcal\{H\}\_\{1\}\)\.

In Fig\.[3](https://arxiv.org/html/2605.26683#S4.F3)we show a comparison between masked concept reachability across language proportions, tokenizer vocabulary sizes, and tokenizer types\. In most settings, vanilla tokenization matches or exceeds the balanced tokenizer despite being trained on a skewed mixture\. This indicates that the relevant factor is whether the resulting segmentation preserves subword units that can be reused across languages\. A balanced tokenizer can still fail if it allocates capacity to language\-specific whole\-word or near\-whole\-word tokens, while a vanilla tokenizer can remain useful if its skewed training distribution induces smaller shared fragments\.

Equal tokenizer exposure to both languages is not sufficient for transfer\. What matters is the structure of the induced vocabulary, especially whether the segmentation creates bridges between𝐀\\mathbf\{A\}and unseen lexical forms in𝐁†\\mathbf\{B\}^\{\\dagger\}\. In Figure[4\(a\)](https://arxiv.org/html/2605.26683#S4.F4.sf1), we show that maximum performance for grammaticality and type constraints satisfaction is consistently lower for the balanced tokenizer\.

#### Larger vocabularies suppress masked transfer\. \(ℋ1\\mathcal\{H\}\_\{1\}\)

Across tokenizer regimes, increasing vocabulary size tends to reduce masked concept reachability in𝐁†\\mathbf\{B\}^\{\\dagger\}\(Fig\.[3](https://arxiv.org/html/2605.26683#S4.F3)\)\. This pattern is clearest in the reachability curves, where smaller vocabularies produce higher top\-KKreachability than larger vocabularies under otherwise comparable conditions\. In our setting, the transfer problem depends on whether the model can connect unseen𝐁\\mathbf\{B\}lexical realizations to structure learned from𝐀\\mathbf\{A\}\. When the vocabulary is large, BPE tends to represent words as language\-specific atoms\. This reduces the number of shared intermediate units through which the model can generalize from one language to the other\. At the extreme, a word\-level vocabulary would eliminate almost all useful overlap between unrelated surface forms, leaving no subword path for transfer\.

Thus, vocabulary size has opposite effects on two objectives: it can make individual\-language modeling more efficient while making cross\-lingual abstraction harder\. For masked lexical transfer, the useful vocabulary is not necessarily the largest one, but the one that preserves reusable sublexical structure\.

#### Language proportion controls transfer more strongly than lexical similarity \(ℋ2\\mathcal\{H\}\_\{2\},ℋ3\\mathcal\{H\}\_\{3\}\)\.

The language\-mixture parameterλ\\lambdahas a consistent effect on𝐁†\\mathbf\{B\}^\{\\dagger\}performance\. Increasing the amount of minority\-language data improves grammaticality, type\-constraint satisfaction, and masked reachability \(Figure[4\(a\)](https://arxiv.org/html/2605.26683#S4.F4.sf1)\)\. This is expected: more𝐁\\mathbf\{B\}data gives the model more evidence about the target language’s surface regularities, even though the specific masked lexical forms are withheld\. In \.

By contrast, lexical similarity has a weaker and less consistent effect\. Although higher similarity sometimes improves reachability, the differences across similarity values are smaller than the differences induced by tokenizer vocabulary size and language proportion\. This suggests that surface similarity alone is not the dominant driver of transfer\. The model appears to rely more on abstract structure and tokenizer\-mediated overlap than on raw character\-level proximity between the two realization functions\.

Similarity aids the creation of shared subword structure, but it is not by itself sufficient\. Two languages may be character\-level similar yet poorly aligned under BPE, or less similar yet still connected through reusable fragments\. In this setting, tokenization is the mechanism through which lexical similarity becomes available to the model\.

#### Transfer emerges as a staged process \(ℋ4\\mathcal\{H\}\_\{4\}\)\.

In Fig\.[4\(b\)](https://arxiv.org/html/2605.26683#S4.F4.sf2)and[4\(c\)](https://arxiv.org/html/2605.26683#S4.F4.sf3)we show that cross\-lingual generalization emerges in a cascade\. The model first learns high\-level well\-formedness in the majority language𝐀\\mathbf\{A\}, it then improves on the observed minority language𝐁\\mathbf\{B\}, and only later shows meaningful generalization to the masked condition𝐁†\\mathbf\{B\}^\{\\dagger\}\. The same ordering appears across both grammaticality and type\-constraint satisfaction\. Grammaticality and type satisfaction rise earlierLubanaet al\.\([2025](https://arxiv.org/html/2605.26683#bib.bib53)\), while masked reachability remains lower and more sensitive to tokenizer design\.

The results support a cascade view of transfer\. Cross\-lingual lexical generalization depends on prior acquisition of several lower\-level capabilities: the model must learn the shared grammar, learn how the minority language realizes that grammar, and learn the ontology\-level type constraints before it can map unseen𝐁†\\mathbf\{B\}^\{\\dagger\}forms onto concepts observed through𝐀\\mathbf\{A\}\. Failure at any earlier stage limits masked transfer\. Additional experiments are shown in Appendix[A](https://arxiv.org/html/2605.26683#A1)\.

### 4\.2Token bridges facilitate cross\-lingual transfer

Our results suggest that transfer depends on whether the tokenizer leaves masked minority\-language forms connected to observed forms through shared subword fragments\. For a masked itemv∈𝒱†v\\in\\mathcal\{V\}^\{\\dagger\}, we define itstokenizer bridgeas

βπ​\(v\)=maxu∈𝒱∖𝒱†⁡sim​\(π​\(ρ𝐁​\(v\)\),π​\(ρ​\(u\)\)\),\\beta\_\{\\pi\}\(v\)=\\max\_\{u\\in\\mathcal\{V\}\\setminus\\mathcal\{V\}^\{\\dagger\}\}\\mathrm\{sim\}\\big\(\\pi\(\\rho\_\{\\color\[rgb\]\{0\.85546875,0\.25390625,0\.40234375\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.85546875,0\.25390625,0\.40234375\}\\mathbf\{B\}\}\(v\)\),\\pi\(\\rho\(u\)\)\\big\),
Wheresim\\mathrm\{sim\}is normalized token overlap between tokens of the masked symbols and any other token seen during training\. The average bridge strengthβ^π\\hat\{\\beta\}\_\{\\pi\}measures how isolated masked𝐁†\\mathbf\{B\}^\{\\dagger\}forms are under the tokenizer\.

This quantity explains the main empirical pattern\. Smaller vocabularies often preserve reusable fragments, while larger vocabularies merge frequent strings into language\-specific units, reducing bridge strength\. As shown in Fig\.[5](https://arxiv.org/html/2605.26683#S4.F5),β^π\\hat\{\\beta\}\_\{\\pi\}correlates strongly with peak masked reachability \(r=0\.62r=0\.62,p<0\.0001p<0\.0001\), suggesting that masked transfer is enabled by subword paths between observed and unseen lexical realizations\.

Figure 5:Tokenizer bridges correlate with masked cross\-lingual transfer\.We compare average tokenizer bridge strengthβ^π\\hat\{\\beta\}\_\{\\pi\}against peak Top\-KKreachability in the masked minority\-language condition𝐁†\\mathbf\{B\}^\{\\dagger\}\. The positive correlation suggests that masked transfer improves when the tokenizer preserves subword paths between observed and unseen lexical realizations\.

## 5Conclusions

We introduced anin\-vitroframework for studying cross\-lingual generalization under controlled multilingual conditions\. By procedurally generating two languages that share the same ontology, grammar, and compositional structure while differing in surface realization, we isolated factors that are normally entangled in natural corpora: lexical distance, language proportion, tokenizer training regime, and tokenizer vocabulary size\. The masked minority\-language condition further allowed us to test whether models can generalize to unseen lexical forms, rather than merely memorization\.

Our results suggest that transfer depends on whether the tokenizer preserves reusable subword structure that connects observed and unseen forms across languages\. Larger vocabularies can improve monolingual efficiency, but they may also collapse words into language\-specific atoms, weakening the subword bridges needed for cross\-lingual abstraction\. Conversely, smaller vocabularies can support transfer by keeping lexical forms decomposable into fragments that remain shared across languages\.

More broadly, our results support a staged view of multilingual learning\. Models first acquire structural competence, such as grammaticality and type constraints, before they can reliably generalize lexical knowledge to unseen minority\-language forms\. These findings imply that multilingual language models should not be trained with tokenizer design treated as a preprocessing detail\. The tokenizer defines the interface through which surface forms become available to the model, and therefore shapes which cross\-lingual generalizations are reachable\.

## Limitations

Our framework makes several simplifying assumptions about the nature of language that limit its generalizability\. First, we assume all languages conform to a recursive, context\-sensitive grammar\. This excludes languages such as \(for example\) Pirahã, which has been argued to lack recursive embedding, or languages commonly referred to Khoisan, whose structural properties fall outside our grammatical assumptions\(Everett,[2005](https://arxiv.org/html/2605.26683#bib.bib10); Witzlack\-Makarevich and Nakagawa,[2019](https://arxiv.org/html/2605.26683#bib.bib18)\)\. Similarly, Semitic languages such as Arabic and Hebrew employ non\-concatenative morphology \(root\-and\-pattern templatic systemsMcCarthy and Prince \([1990](https://arxiv.org/html/2605.26683#bib.bib5)\)\) which our agglutinative word formation model cannot capture\. We leave extension to these morphological types as future work\.

Second, our lexical realization model assumes words are formed by concatenating prefixes, stems, and suffixes, and that letter probabilities follow a Zipfian distributionZipf \([1935](https://arxiv.org/html/2605.26683#bib.bib1)\)\. This excludes fusional morphology, where morphemes merge and lose their boundaries, as well as tonal and phonological\. We further abstract away from phonetics entirely, meaning that surface similarity is measured purely at the character level rather than through phonological categories or syllable structure\. We instantiate our experiments with an SVO grammar, whereas natural languages exhibit several other word\-order types, including SOV, VSO, and VOS\. This is a limitation of the present configuration rather than the framework: other word orders can be supported by changing the grammar while keeping the ontology and evaluation protocol fixed\.

Third, we assume that both languages share exactly the same concept ontology and vocabulary size\. This rules out phenomena such as lexical gaps, where a concept expressible in one language has no direct equivalent in another, and false friendsBentivogli and Pianta \([2000](https://arxiv.org/html/2605.26683#bib.bib7)\); Dinuet al\.\([2023b](https://arxiv.org/html/2605.26683#bib.bib39)\), where surface\-similar forms carry different meanings across languages\. Natural multilingual settings rarely satisfy these assumptions, and the degree to which our findings extend to such settings remains an open question\.

Finally, language is inherently fluid and socially situated, and no finite generative grammar can fully capture its complexity\. Our setup should therefore be understood as a controlled approximation designed to isolate specific transfer mechanisms, rather than a faithful model of natural language acquisition or multilingual competence\.

## References

- D\. Abagyan, A\. R\. Salamanca, A\. F\. Cruz\-Salinas, K\. Cao, H\. Lin, A\. Locatelli, M\. Fadaee, A\. Üstün, and S\. Hooker \(2025\)One tokenizer to rule them all: Emergent language plasticity via multilingual tokenizers\.arXiv preprint arXiv:2506\.10766\.Cited by:[§2](https://arxiv.org/html/2605.26683#S2.p2.1)\.
- O\. Ahia, S\. Kumar, H\. Gonen, V\. Hofmann, T\. Limisiewicz, Y\. Tsvetkov, and N\. A\. Smith \(2024\)Magnet: Improving the multilingual fairness of language models with adaptive gradient\-based tokenization\.Advances in Neural Information Processing Systems37,pp\. 47790–47814\.Cited by:[§2](https://arxiv.org/html/2605.26683#S2.p2.1)\.
- Z\. Allen\-Zhu and Y\. Li \(2025\)Physics of language models: part 1, learning hierarchical language structures\.Transactions on Machine Learning Research\.Note:External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=mPQKyzkA1K)Cited by:[§1](https://arxiv.org/html/2605.26683#S1.p3.5),[§2](https://arxiv.org/html/2605.26683#S2.p3.1)\.
- P\. W\. Anderson \(1972\)More is different: broken symmetry and the nature of the hierarchical structure of science\.\.Science177\(4047\),pp\. 393–396\.Cited by:[§3\.3](https://arxiv.org/html/2605.26683#S3.SS3.p1.4),[footnote 1](https://arxiv.org/html/2605.26683#footnote1)\.
- P\. M\. Arkadiev \(2019\)Morphology in Typology: Historical Retrospect, State of the Art, and Prospects\.InOxford Research Encyclopedia of Linguistics,M\. Aronoff \(Ed\.\),External Links:[Document](https://dx.doi.org/10.1093/acrefore/9780199384655.013.626),ISBN 9780197851463,[Link](https://doi.org/10.1093/acrefore/9780199384655.013.626),https://academic\.oup\.com/book/0/chapter/547667735/chapter\-ag\-pdf/67715244/book\_61882\_section\_547667735\.ag\.pdfCited by:[§3\.2](https://arxiv.org/html/2605.26683#S3.SS2.SSS0.Px1.p1.8)\.
- C\. Arnett, T\. A\. Chang, S\. Biderman, and B\. Bergen \(2026\)Explaining and mitigating crosslingual tokenizer inequities\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=0Y7AxxNCYh)Cited by:[§2](https://arxiv.org/html/2605.26683#S2.p2.1)\.
- W\. L\. Ballard \(1980\)M\. A\. K\. Halliday, Language as social semiotic: The social interpretation of language and meaning\. London: Edward Arnold, 1978\. Pp\. 256\.\.Language in Society9\(1\),pp\. 84–89\.External Links:[Document](https://dx.doi.org/10.1017/S004740450000782X)Cited by:[§3\.1](https://arxiv.org/html/2605.26683#S3.SS1.p2.1)\.
- L\. Bentivogli and E\. Pianta \(2000\)Looking for lexical gaps\.InProceedings of the ninth EURALEX International Congress,pp\. 663–669\.Cited by:[Limitations](https://arxiv.org/html/2605.26683#Sx1.p3.1)\.
- L\. Berti, F\. Giorgi, and G\. Kasneci \(2025\)Emergent abilities in large language models: a survey\.arXiv preprint arXiv:2503\.05788\.Cited by:[§3\.3](https://arxiv.org/html/2605.26683#S3.SS3.p1.4)\.
- T\. Blevins, H\. Gonen, and L\. Zettlemoyer \(2022\)Analyzing the mono\-and cross\-lingual pretraining dynamics of multilingual language models\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,pp\. 3575–3590\.Cited by:[itemℋ4\\mathcal\{H\}\_\{4\}](https://arxiv.org/html/2605.26683#S1.I1.ix4.p1.1)\.
- T\. Blevins and L\. Zettlemoyer \(2022\)Language contamination helps explains the cross\-lingual capabilities of English pretrained models\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,pp\. 3563–3574\.Cited by:[§1](https://arxiv.org/html/2605.26683#S1.p2.1),[§2](https://arxiv.org/html/2605.26683#S2.p3.1)\.
- A\. Cosma, S\. Ruseti, E\. Radoi, and M\. Dascalu \(2025\)The strawberry problem: Emergence of character\-level understanding in tokenized language models\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 28240–28251\.Cited by:[§2](https://arxiv.org/html/2605.26683#S2.p2.1)\.
- L\. De Souza, T\. Almeida, R\. Lotufo, and R\. F\. Nogueira \(2024\)Measuring cross\-lingual transfer in bytes\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 7526–7537\.Cited by:[§1](https://arxiv.org/html/2605.26683#S1.p1.1),[§1](https://arxiv.org/html/2605.26683#S1.p2.1),[§2](https://arxiv.org/html/2605.26683#S2.p1.1),[§3\.2](https://arxiv.org/html/2605.26683#S3.SS2.SSS0.Px4.p1.5)\.
- L\. P\. Dinu, A\. Uban, A\. Cristea, A\. Dinu, I\. Iordache, S\. Georgescu, and L\. Zoicas \(2023a\)Robocop: A comprehensive romance borrowing cognate package and benchmark for multilingual cognate identification\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 7610–7629\.Cited by:[§3\.2](https://arxiv.org/html/2605.26683#S3.SS2.SSS0.Px1.p2.6)\.
- L\. Dinu, A\. Uban, A\. Cristea, A\. Dinu, I\. Iordache, S\. Georgescu, and L\. Zoicas \(2023b\)RoBoCoP: A Comprehensive ROmance BOrrowing COgnate Package and Benchmark for Multilingual Cognate Identification\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 7610–7629\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.473),[Link](https://aclanthology.org/2023.emnlp-main.473/)Cited by:[Limitations](https://arxiv.org/html/2605.26683#Sx1.p3.1)\.
- P\. Escudero and P\. Boersma \(2004\)Bridging the gap between L2 speech perception research and phonological theory\.Studies in second language acquisition26\(4\),pp\. 551–585\.Cited by:[§2](https://arxiv.org/html/2605.26683#S2.p2.1)\.
- D\. Everett \(2005\)Cultural constraints on grammar and cognition in Pirahã: Another look at the design features of human language\.Current anthropology46\(4\),pp\. 621–646\.Cited by:[Limitations](https://arxiv.org/html/2605.26683#Sx1.p1.1)\.
- P\. Gage \(1994\)A new algorithm for data compression\.The C Users Journal archive12,pp\. 23–38\.External Links:[Link](https://api.semanticscholar.org/CorpusID:59804030)Cited by:[§3\.2](https://arxiv.org/html/2605.26683#S3.SS2.SSS0.Px4.p1.5)\.
- C\. Gooskens and F\. Swarte \(2017\)Linguistic and extra\-linguistic predictors of mutual intelligibility between Germanic languages\.Nordic Journal of Linguistics40\(2\),pp\. 123–147\.External Links:[Document](https://dx.doi.org/10.1017/S0332586517000099)Cited by:[itemℋ2\\mathcal\{H\}\_\{2\}](https://arxiv.org/html/2605.26683#S1.I1.ix2.p1.1),[§2](https://arxiv.org/html/2605.26683#S2.p2.1)\.
- C\. Gooskens and V\. J\. Van Heuven \(2021\)Mutual intelligibility\.Similar languages, varieties, and dialects: A computational perspective,pp\. 51–95\.Cited by:[§2](https://arxiv.org/html/2605.26683#S2.p2.1)\.
- K\. Hämmerl, T\. Limisiewicz, J\. Libovickỳ, and A\. Fraser \(2025\)Beyond literal token overlap: Token alignability for multilinguality\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 2: Short Papers\),pp\. 756–767\.Cited by:[itemℋ1\\mathcal\{H\}\_\{1\}](https://arxiv.org/html/2605.26683#S1.I1.ix1.p1.1),[§1](https://arxiv.org/html/2605.26683#S1.p2.1),[§2](https://arxiv.org/html/2605.26683#S2.p2.1)\.
- D\. Hernandez, J\. Kaplan, T\. Henighan, and S\. McCandlish \(2021\)Scaling laws for transfer\.arXiv preprint arXiv:2102\.01293\.Cited by:[itemℋ2\\mathcal\{H\}\_\{2\}](https://arxiv.org/html/2605.26683#S1.I1.ix2.p1.1)\.
- M\. Y\. Hu, J\. Petty, C\. Shi, W\. Merrill, and T\. Linzen \(2025\)Between circuits and Chomsky: pre\-pretraining on formal languages imparts linguistic biases\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 9691–9709\.External Links:[Link](https://aclanthology.org/2025.acl-long.478/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.478),ISBN 979\-8\-89176\-251\-0Cited by:[§2](https://arxiv.org/html/2605.26683#S2.p3.1)\.
- H\. Jabbar \(2023\)Morphpiece: A linguistic tokenizer for large language models\.arXiv preprint arXiv:2307\.07262\.Cited by:[§2](https://arxiv.org/html/2605.26683#S2.p2.1)\.
- D\. P\. Kingma and J\. Ba \(2015\)Adam: a method for stochastic optimization\.\.InInternational Conference on Learning Representations,Cited by:[§A\.1](https://arxiv.org/html/2605.26683#A1.SS1.p3.3)\.
- A\. Kunchukuttan, M\. M\. Khapra, G\. Singh, and P\. Bhattacharyya \(2018\)Leveraging orthographic similarity for multilingual neural transliteration\.Transactions of the Association for Computational Linguistics6,pp\. 303–316\.Cited by:[itemℋ2\\mathcal\{H\}\_\{2\}](https://arxiv.org/html/2605.26683#S1.I1.ix2.p1.1)\.
- V\. I\. Levenshteinet al\.\(1966\)Binary codes capable of correcting deletions, insertions, and reversals\.InSoviet physics doklady,Vol\.10,pp\. 707–710\.Cited by:[§3\.2](https://arxiv.org/html/2605.26683#S3.SS2.SSS0.Px1.p2.6)\.
- T\. Limisiewicz, J\. Balhar, and D\. Mareček \(2023a\)Tokenization impacts multilingual language modeling: assessing vocabulary allocation and overlap across languages\.InFindings of the Association for Computational Linguistics: ACL 2023,A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 5661–5681\.External Links:[Link](https://aclanthology.org/2023.findings-acl.350/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-acl.350)Cited by:[§2](https://arxiv.org/html/2605.26683#S2.p2.1),[§3\.2](https://arxiv.org/html/2605.26683#S3.SS2.SSS0.Px4.p1.5)\.
- T\. Limisiewicz, D\. Malkin, and G\. Stanovsky \(2023b\)You can have your data and balance it too: Towards balanced and efficient multilingual models\.InProceedings of the 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP,pp\. 1–11\.Cited by:[itemℋ3\\mathcal\{H\}\_\{3\}](https://arxiv.org/html/2605.26683#S1.I1.ix3.p1.1),[§1](https://arxiv.org/html/2605.26683#S1.p2.1)\.
- Z\. Liu, Y\. Liu, E\. J\. Michaud, J\. Gore, and M\. Tegmark \(2025\)Physics of skill learning\.arXiv preprint arXiv:2501\.12391\.Cited by:[§1](https://arxiv.org/html/2605.26683#S1.p3.5)\.
- E\. S\. Lubana, K\. Kawaguchi, R\. Dick, and H\. Tanaka \(2025\)A percolation model of emergence: analyzing transformers trained on a formal language\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 37402–37460\.Cited by:[§A\.1](https://arxiv.org/html/2605.26683#A1.SS1.p1.3),[itemℋ4\\mathcal\{H\}\_\{4\}](https://arxiv.org/html/2605.26683#S1.I1.ix4.p1.1),[§2](https://arxiv.org/html/2605.26683#S2.p3.1),[§3\.1](https://arxiv.org/html/2605.26683#S3.SS1.p2.1),[§3\.2](https://arxiv.org/html/2605.26683#S3.SS2.SSS0.Px5.p1.4),[§3\.3](https://arxiv.org/html/2605.26683#S3.SS3.p1.4),[§3\.3](https://arxiv.org/html/2605.26683#S3.SS3.p2.9),[§4\.1](https://arxiv.org/html/2605.26683#S4.SS1.SSS0.Px5.p1.3)\.
- D\. Malkin, T\. Limisiewicz, and G\. Stanovsky \(2022\)A balanced data approach for evaluating cross\-lingual transfer: Mapping the linguistic blood bank\.InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 4903–4915\.Cited by:[itemℋ3\\mathcal\{H\}\_\{3\}](https://arxiv.org/html/2605.26683#S1.I1.ix3.p1.1),[§1](https://arxiv.org/html/2605.26683#S1.p2.1)\.
- J\. McCarthy and A\. Prince \(1990\)Prosodic morphology and templatic morphology\.InPerspectives on Arabic linguistics II: papers from the second annual symposium on Arabic linguistics,pp\. 1–54\.Cited by:[Limitations](https://arxiv.org/html/2605.26683#Sx1.p1.1)\.
- E\. Michaud, Z\. Liu, U\. Girit, and M\. Tegmark \(2023\)The quantization model of neural scaling\.Advances in Neural Information Processing Systems36,pp\. 28699–28722\.Cited by:[§1](https://arxiv.org/html/2605.26683#S1.p3.5)\.
- I\. Papadimitriou and D\. Jurafsky \(2020\)Learning Music Helps You Read: Using transfer to study linguistic structure in language models\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 6829–6839\.External Links:[Link](https://aclanthology.org/2020.emnlp-main.554/),[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.554)Cited by:[§2](https://arxiv.org/html/2605.26683#S2.p1.1)\.
- I\. Papadimitriou and D\. Jurafsky \(2023\)Injecting structural hints: using language models to study inductive biases in language learning\.InFindings of the Association for Computational Linguistics: EMNLP 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 8402–8413\.External Links:[Link](https://aclanthology.org/2023.findings-emnlp.563/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.563)Cited by:[§2](https://arxiv.org/html/2605.26683#S2.p3.1)\.
- V\. Patil, P\. Talukdar, and S\. Sarawagi \(2022\)Overlap\-based vocabulary generation improves cross\-lingual transfer among related languages\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 219–233\.Cited by:[itemℋ1\\mathcal\{H\}\_\{1\}](https://arxiv.org/html/2605.26683#S1.I1.ix1.p1.1),[§1](https://arxiv.org/html/2605.26683#S1.p2.1),[§2](https://arxiv.org/html/2605.26683#S2.p2.1),[§3\.2](https://arxiv.org/html/2605.26683#S3.SS2.SSS0.Px4.p1.5)\.
- A\. Petrov, E\. La Malfa, P\. Torr, and A\. Bibi \(2023\)Language Model Tokenizers Introduce Unfairness Between Languages\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 36963–36990\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/74bb24dca8334adce292883b4b651eda-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2605.26683#S2.p2.1),[§3\.2](https://arxiv.org/html/2605.26683#S3.SS2.SSS0.Px4.p1.5)\.
- P\. Rust, J\. Pfeiffer, I\. Vulić, S\. Ruder, and I\. Gurevych \(2021\)How good is your tokenizer? on the monolingual performance of multilingual language models\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),pp\. 3118–3135\.Cited by:[itemℋ1\\mathcal\{H\}\_\{1\}](https://arxiv.org/html/2605.26683#S1.I1.ix1.p1.1),[§1](https://arxiv.org/html/2605.26683#S1.p2.1),[§4\.1](https://arxiv.org/html/2605.26683#S4.SS1.SSS0.Px1.p1.5)\.
- N\. Shazeer \(2020\)Glu variants improve transformer\.arXiv preprint arXiv:2002\.05202\.Cited by:[§A\.1](https://arxiv.org/html/2605.26683#A1.SS1.p3.3)\.
- C\. Taguchi and R\. Sproat \(2026\)Creating ConLangs to Probe the Metalinguistic Grammatical Knowledge of LLMs\.External Links:[Link](https://arxiv.org/abs/2510.07591),2510\.07591Cited by:[§2](https://arxiv.org/html/2605.26683#S2.p3.1)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.Advances in neural information processing systems30\.Cited by:[§A\.1](https://arxiv.org/html/2605.26683#A1.SS1.p3.3)\.
- P\. Veličković, F\. Barbero, C\. Perivolaropoulos, S\. Osindero, and R\. Pascanu \(2026\)Perplexity Cannot Always Tell Right from Wrong\.arXiv preprint arXiv:2601\.22950\.Cited by:[§3\.3](https://arxiv.org/html/2605.26683#S3.SS3.p1.4)\.
- H\. Wang, P\. Minervini, and E\. Ponti \(2024\)Probing the emergence of cross\-lingual alignment during LLM training\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 12159–12173\.Cited by:[itemℋ4\\mathcal\{H\}\_\{4\}](https://arxiv.org/html/2605.26683#S1.I1.ix4.p1.1)\.
- J\. Wei, Y\. Tay, R\. Bommasani, C\. Raffel, B\. Zoph, S\. Borgeaud, D\. Yogatama, M\. Bosma, D\. Zhou, D\. Metzler, E\. H\. Chi, T\. Hashimoto, O\. Vinyals, P\. Liang, J\. Dean, and W\. Fedus \(2022\)Emergent abilities of large language models\.Transactions on Machine Learning Research\.Note:Survey CertificationExternal Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=yzkSU5zdwD)Cited by:[§3\.3](https://arxiv.org/html/2605.26683#S3.SS3.p1.4)\.
- C\. Wendler, V\. Veselovsky, G\. Monea, and R\. West \(2024\)Do Llamas Work in English? On the Latent Language of Multilingual Transformers\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 15366–15394\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.820),[Link](https://aclanthology.org/2024.acl-long.820/)Cited by:[§2](https://arxiv.org/html/2605.26683#S2.p1.1)\.
- A\. Witzlack\-Makarevich and H\. Nakagawa \(2019\)Linguistic Features and Typologies in Languages Commonly Referred to as ‘Khoisan’\.InThe Cambridge Handbook of African Linguistics,Cambridge Handbooks in Language and Linguistics,pp\. 382–416\.Cited by:[Limitations](https://arxiv.org/html/2605.26683#Sx1.p1.1)\.
- L\. Xue, A\. Barua, N\. Constant, R\. Al\-Rfou, S\. Narang, M\. Kale, A\. Roberts, and C\. Raffel \(2022\)ByT5: Towards a token\-free future with pre\-trained byte\-to\-byte models\.Transactions of the Association for Computational Linguistics10,pp\. 291–306\.Cited by:[§2](https://arxiv.org/html/2605.26683#S2.p2.1),[§3\.2](https://arxiv.org/html/2605.26683#S3.SS2.SSS0.Px4.p1.5)\.
- B\. Zhang and R\. Sennrich \(2019\)Root mean square layer normalization\.Advances in neural information processing systems32\.Cited by:[§A\.1](https://arxiv.org/html/2605.26683#A1.SS1.p3.3)\.
- Y\. Zhao, W\. Zhang, G\. Chen, K\. Kawaguchi, and L\. Bing \(2024\)How do Large Language Models Handle Multilingualism?\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=ctXYOoAgRy)Cited by:[§1](https://arxiv.org/html/2605.26683#S1.p1.1),[§1](https://arxiv.org/html/2605.26683#S1.p2.1),[§2](https://arxiv.org/html/2605.26683#S2.p1.1)\.
- G\. K\. Zipf \(1935\)The Psycho\-Biology of Language: An Introduction to Dynamic Philology\.Houghton Mifflin,Boston, MA\.Cited by:[§A\.1](https://arxiv.org/html/2605.26683#A1.SS1.p1.3),[Limitations](https://arxiv.org/html/2605.26683#Sx1.p2.1)\.

## Appendix AAppendix

Table 1:Paired sentence realizations from the synthetic bilingual corpus chosen to emphasize visible overlap under the 2048 tokenizer\. Each row comes from the same underlying symbolic sentence realized in𝐀\\mathbf\{A\}and𝐁\\mathbf\{B\}\. Shared subtokens receive the same color in both languages\. Legend:sharedshared subtoken;A\-only𝐀\\mathbf\{A\}\-only subtoken;B\-only𝐁\\mathbf\{B\}\-only subtoken;<eos\>control token\.Figure 6:Tokenizer fragmentation continuation rate for𝐀\\mathbf\{A\},𝐁\\mathbf\{B\}, and𝐁†\\mathbf\{B\}^\{\\dagger\}\. The continuation rate measures how often words are represented by multi\-token segmentations\. Higher continuation rates indicate greater fragmentation\. The masked condition follows the minority\-language tokenization pattern, showing that the model must generalize through the same subword interface used for observed𝐁\\mathbf\{B\}forms\.### A\.1Implementation details

We keep the number of concepts fixed, corresponding to 100 entities, 10 classes to divide over, 460 descriptive properties, 40 descriptive values, 100 relative properties \(seeLubanaet al\.\([2025](https://arxiv.org/html/2605.26683#bib.bib53)\)\)\. When forming symbolic sentences, we sampled concepts uniformly\. Words are created using the following syllable patterns: "CVC", "CCV", "CVCC", "CV", "VC", "V"\. The letter probabilities follow a Zipfian distributionZipf \([1935](https://arxiv.org/html/2605.26683#bib.bib1)\)and are slightly different for𝐀\\mathbf\{A\}and𝐁\\mathbf\{B\}, according to the lexical distancedd\. The average word length is nine characters\.

The PCSG used to generate sentences is presented in[7](https://arxiv.org/html/2605.26683#A1.F7)\. Each symbolic vocabulary item is assigned a latent proto\-stem and two surface realizations, one per language, obtained by applying language\-specific cognate transformations and optional category\-conditioned affixes; symbolic sequences are then realized by simple token substitution into the selected language \(Algorithm[1](https://arxiv.org/html/2605.26683#alg1)\. Here,ssdenotes a latent proto\-stem,c​\(v\)c\(v\)is the grammatical category of symbolic entryvv,π\\piandσ\\sigmaare an optional prefix and suffix, and∥\\\|denotes string concatenation\. The functionDeriveCognate​\(s,ℓ\)\\textsc\{DeriveCognate\}\(s,\\ell\)produces a language\-specific variant of the shared stemssfor languageℓ\\ell\.

Algorithm 1Bilingual lexical realization\.1:Symbolic vocabulary

𝒱\\mathcal\{V\}; symbolic sequence

X=\(x1,…,xn\)X=\(x\_\{1\},\\dots,x\_\{n\}\); language set

ℒ=\{A,B\}\\mathcal\{L\}=\\\{A,B\\\}; category function

c:𝒱→𝒞c:\\mathcal\{V\}\\to\\mathcal\{C\}
2:Bilingual lexicon

MMwith entries

M​\[v,ℓ\]M\[v,\\ell\]for

v∈𝒱v\\in\\mathcal\{V\}and

ℓ∈ℒ\\ell\\in\\mathcal\{L\}
3:functionMakeWord\(

s,c,ℓs,c,\\ell\)

4:

s\(ℓ\)←DeriveCognate​\(s,ℓ\)s^\{\(\\ell\)\}\\leftarrow\\textsc\{DeriveCognate\}\(s,\\ell\)
5:

π←SamplePrefix​\(c,ℓ\)\\pi\\leftarrow\\textsc\{SamplePrefix\}\(c,\\ell\)
6:

σ←SampleSuffix​\(c,ℓ\)\\sigma\\leftarrow\\textsc\{SampleSuffix\}\(c,\\ell\)
7:return

π​‖s\(ℓ\)‖​σ\\pi\\,\\\|\\,s^\{\(\\ell\)\}\\,\\\|\\,\\sigma
8:endfunction

9:for all

v∈𝒱v\\in\\mathcal\{V\}do

10:sample latent stem

ss
11:for all

ℓ∈ℒ\\ell\\in\\mathcal\{L\}do

12:

M​\[v,ℓ\]←MakeWord​\(s,c​\(v\),ℓ\)M\[v,\\ell\]\\leftarrow\\textsc\{MakeWord\}\(s,c\(v\),\\ell\)
13:endfor

14:endfor

15:return

MM

Each configuration is trained across nine seeds: three seeds for the dataset creation and three seeds for the model training\. Model is a small transformer decoderVaswaniet al\.\([2017](https://arxiv.org/html/2605.26683#bib.bib14)\)consisting of 4 layers, a model dimension of 256, using SwiGLUShazeer \([2020](https://arxiv.org/html/2605.26683#bib.bib21)\)and RMSNormZhang and Sennrich \([2019](https://arxiv.org/html/2605.26683#bib.bib19)\)layers\. We used a batch size of 64, learning rate of 0\.0001, adapted using a cosine decay with linear warmup of 256 steps\. We used AdamW optimizerKingma and Ba \([2015](https://arxiv.org/html/2605.26683#bib.bib12)\)withβ1=0\.9\\beta\_\{1\}=0\.9,β2=0\.95\\beta\_\{2\}=0\.95,ε=10−10\\varepsilon=10^\{\-10\}and weight decay of 0\.01\. Context size was set at 256 tokens, enough such that sequences are not truncated\. We train each model for 10000 steps\.We train a total of 700 models\. A training run takes around one hour\. Model training was performed on a consumer NVIDIA RTX 3060 GPU, making our experiments replicable on consumer hardware\.

Algorithm 2ReachableTopK1:model

ff, prompt

𝐱\\mathbf\{x\}, target sequences

𝒮\\mathcal\{S\}, top\-

kkvalue

kk, horizon

HH
2:whether any target sequence is reachable

3:

T←BuildTrie​\(𝒮\)T\\leftarrow\\textsc\{BuildTrie\}\(\\mathcal\{S\}\)
4:

ℱ←\{\(r,ϵ\)\}\\mathcal\{F\}\\leftarrow\\\{\(r,\\epsilon\)\\\}⊳\\trianglerightroot node and empty prefix

5:for

t=1t=1to

HHdo

6:if

ℱ=∅\\mathcal\{F\}=\\emptysetthenreturnfalse

7:endif

8:Batch all contexts

\[𝐱;𝐩\]\[\\mathbf\{x\};\\mathbf\{p\}\]for

\(n,𝐩\)∈ℱ\(n,\\mathbf\{p\}\)\\in\\mathcal\{F\}
9:Compute top\-

kknext\-token sets

𝒦\(n,𝐩\)\\mathcal\{K\}\_\{\(n,\\mathbf\{p\}\)\}
10:

ℱ′←∅\\mathcal\{F\}^\{\\prime\}\\leftarrow\\emptyset
11:for all

\(n,𝐩\)∈ℱ\(n,\\mathbf\{p\}\)\\in\\mathcal\{F\}do

12:for all

u∈Children​\(n\)∩𝒦\(n,𝐩\)u\\in\\mathrm\{Children\}\(n\)\\cap\\mathcal\{K\}\_\{\(n,\\mathbf\{p\}\)\}do

13:

n′←Next​\(n,u\)n^\{\\prime\}\\leftarrow\\mathrm\{Next\}\(n,u\)
14:if

n′n^\{\\prime\}is terminalthenreturntrue

15:endif

16:add

\(n′,𝐩∥u\)\(n^\{\\prime\},\\mathbf\{p\}\\\|u\)to

ℱ′\\mathcal\{F\}^\{\\prime\}
17:endfor

18:endfor

19:

ℱ←Deduplicate​\(ℱ′\)\\mathcal\{F\}\\leftarrow\\textsc\{Deduplicate\}\(\\mathcal\{F\}^\{\\prime\}\)
20:endfor

21:returnfalse

Top\-K ReachabilityAlgorithm[2](https://arxiv.org/html/2605.26683#alg2)checks whether any target sequence is reachable when decoding is restricted to tokens that are both valid trie continuations and present in the model’s top\-kkpredictions at each step\. By batching all active prefixes in the frontier, it evaluates many partial candidates with a single forward pass per decoding step\.

Production grammar``` S -> Ph NP VP EndOfSeq [1.0] | Ph NP VP SepSeq S [0.0] NP -> subjectID [0.8] | NP Conj NP [0.2] VP -> descPreP descV [0.4] | relV relPreP relNP [0.4] | VP Conj VP [0.2] relNP -> objectID [0.7] | objectID Conj relNP[0.3] Ph -> ’[P]’ [1.0] subjectID -> ’subjectID’[1.0] objectID -> ’objectID’ [1.0] relV -> ’relV’ [1.0] descV -> ’descV’ [1.0] descPreP -> ’descPreP’ [1.0] relPreP -> ’relPreP’ [1.0] Conj -> ’conj’ [1.0] SepSeq -> ’<sep>’ [1.0] EndOfSeq -> ’<eos>’ [1.0] ```

Figure 7:Probabilistic grammar used to generate symbolic sequences\. The grammar defines the abstract sentence structure shared by both languages before language\-specific lexical realization\. This shared symbolic layer allows us to isolate transfer through surface forms while holding syntax, type constraints, and ontology fixed\.
### A\.2Additional experiments\.

In Figure[8](https://arxiv.org/html/2605.26683#A1.F8)we should training dynamics using a balanced tokenizer\. Further in Figures[9](https://arxiv.org/html/2605.26683#A1.F9)and[10](https://arxiv.org/html/2605.26683#A1.F10)we show emergence steps across languages for vanilla and balanced tokenizers, respectively\. In Figures[11](https://arxiv.org/html/2605.26683#A1.F11),[12](https://arxiv.org/html/2605.26683#A1.F12),[13](https://arxiv.org/html/2605.26683#A1.F13),[14](https://arxiv.org/html/2605.26683#A1.F14),[15](https://arxiv.org/html/2605.26683#A1.F15), and[16](https://arxiv.org/html/2605.26683#A1.F16)we show the maximum performance reached across vanilla and balanced tokenizers\.

Figure 8:Training dynamics for validity, grammaticality, and type\-constraint satisfaction in the masked condition𝐁†\\mathbf\{B\}^\{\\dagger\}using a balanced tokenizer\.Figure 9:Average emergence time for grammaticality and type\-constraint satisfaction across languages for vanilla tokenizer and\|V\|∈\{2048,4096,8192\}\|V\|\\in\\\{2048,4096,8192\\\}\. Lower values indicate earlier emergence during training\.Figure 10:Average emergence time for grammaticality and type\-constraint satisfaction across languages for balanced tokenizer and\|V\|∈\{2048,4096,8192\}\|V\|\\in\\\{2048,4096,8192\\\}\. Lower values indicate earlier emergence during training\.Figure 11:Grammaticality in the vanilla tokenizer setting across vocabulary sizes, language ratios, and lexical similarities\. Grammaticality improves with additional minority\-language data and is comparatively robust across lexical similarity conditions, indicating that structural well\-formedness can be learned even when masked lexical transfer remains limited\.Figure 12:Grammaticality in the balanced tokenizer setting across vocabulary sizes, language ratios, and lexical similarities\. Balanced tokenizer training reduces some minority\-language asymmetries, but its effect depends on vocabulary size\. The results show that balanced tokenizer exposure does not automatically translate into uniformly better structural competence\.Figure 13:Type\-constraint satisfaction in the vanilla tokenizer setting\. Despite skewed tokenizer training, models acquire type\-level competence across a range of conditions\. The remaining gap between𝐁\\mathbf\{B\}and𝐁†\\mathbf\{B\}^\{\\dagger\}indicates that satisfying abstract constraints is easier than transferring those constraints to unseen target\-language surface forms\.Figure 14:Type\-constraint satisfaction in the balanced tokenizer setting\. Type\-level competence improves with greater minority\-language proportion and varies across vocabulary sizes\. Compared with masked reachability, type satisfaction emerges more reliably, suggesting that semantic constraints are learned before they become usable for generating unseen minority\-language lexical forms\.Figure 15:Masked Top\-KKreachability in the vanilla tokenizer setting\. Reachability is highest at smaller vocabulary sizes and generally decreases as vocabulary size increases\. This supports the view that larger BPE vocabularies can harm cross\-lingual transfer by replacing reusable subword fragments with language\-specific lexical units\.Figure 16:Masked Top\-KKreachability in the balanced tokenizer setting\. Balanced tokenization can improve transfer when the minority\-language proportion is small, but the benefit depends strongly on vocabulary size\. Larger vocabularies again reduce reachability, showing that equal tokenizer exposure is not sufficient when the induced vocabulary fails to preserve shared subword structure\.