There is No Theoretical Curse of Multilinguality For Embedding Space Structure

# There is No Theoretical Curse of Multilinguality For Embedding Space Structure
Source: [https://arxiv.org/html/2608.17088](https://arxiv.org/html/2608.17088)
Neha VermaVilém ZouharAffiliation:ETH Zürich\[0\.6em\]nbafna1@jhu\.eduPhilipp KoehnDavid Yarowsky\[0\.6em\] Johns Hopkins UniversityCenter for LanguageSpeech Processing

###### Abstract

A central goal of multilingual NLP is to achieve high monolingual performance per language and cross\-lingual alignment for large\-scale language coverage with a multilingual model\. The*curse of multilinguality*describes the phenomenon of degradation in multilingual model performance as we increase language coverage, posing a threat to the above goal\. This paper asks whether multilingual embedding spaces are inherently incapable of achieving perfect multilinguality without a prohibitive increase in required capacity\. We first formalize the goal of “perfect multilinguality”, embodied in two*multilinguality conditions*\. We then prove that the minimum dimensionality required for perfect multilinguality grows only logarithmically, asΘ⁡\(log⁡L\)\\Theta\(\\log L\), in the number of languagesLL\. That is, we show that there is no theoretical curse of multilinguality for embedding space structure\. This suggests that the empirical curse of multilinguality is a result of real world data and training conditions\. We back this understanding with a small\-scale empirical study\. Our paper provides the first theoretical and intrinsic perspective on the curse of multilinguality, with implications for the scientific understanding of this phenomenon\.111The code for our experiments is at[github\.com/niyatibafna/curse\-of\-multilinguality](https://github.com/niyatibafna/curse-of-multilinguality)\.

## 1Introduction

The ideal multilingual model exhibits high monolingual performance on individual languages as well as perfect cross\-lingual structure, or semantic sharing across languages, for arbitrary language coverage\. The well\-known*curse of multilinguality*\(COM,[9](https://arxiv.org/html/2608.17088#bib.bib5)\), also sometimes termed*negative interference*\([31](https://arxiv.org/html/2608.17088#bib.bib16)\), describes the phenomenon of observed degradation of multilingual performance as we train a fixed\-capacity model on increasingly more languages\. Folk wisdom generally understands fixed capacity to be an inherent bottleneck\([18](https://arxiv.org/html/2608.17088#bib.bib8)\), resulting in “less space” for each language, indicating that perfect multilinguality as described above may not be achievable\. Several previous works have demonstrated theCOMfrom the perspective of extrinsic task performance\([9](https://arxiv.org/html/2608.17088#bib.bib5);[1](https://arxiv.org/html/2608.17088#bib.bib1)\), and while other works have explored strategies to mitigate this issue\([35](https://arxiv.org/html/2608.17088#bib.bib17);[27](https://arxiv.org/html/2608.17088#bib.bib13)\), it remains a serious challenge\. This gives rise to a fundamental question: is it inherently impossible to accommodate increasing language coverage in a limited\-capacity model while maintaining desired monolingual and cross\-lingual performance on all languages? We provide a theoretical and intrinsic perspective on theCOM\. Specifically, we study the curse of multilinguality for embedding space structure \(COM\-EMB\), and show that multilingual representations can exhibit intrinsic quality while accommodating more languages without a prohibitive increase in capacity\.

First, we formulate two intrinsic requirements for a*perfect multilingual*embedding space, drawing from consensus in the field\([7](https://arxiv.org/html/2608.17088#bib.bib4);[19](https://arxiv.org/html/2608.17088#bib.bib9)\), and term them*multilinguality conditions*: \(a\) the space exhibits high\-quality shared monolingual semantics for each language, and \(b\) the space exhibits cross\-lingual alignment, meaning that translation equivalents across languages are “close” to each other in the space\. This is important for applications such as cross\-lingual information retrieval\([29](https://arxiv.org/html/2608.17088#bib.bib25);[28](https://arxiv.org/html/2608.17088#bib.bib14)\)\. In the context of embedding spaces, “capacity” can be understood as dimensionality\. Then, our theoretical result shows that the minimum dimensionality of a perfect multilingual space encodingLLlanguages grows asΘ⁡\(log⁡L\)\\Theta\(\\log L\)\. Specifically, we assume a baseDD\-dimensional concept space and show that the minimum sufficient extra dimensionality to accommodateLLlanguages while maintaining the multilinguality conditions grows asO⁡\(log⁡L\)O\(\\log L\), and that the dimensionality of any such space grows at least asΩ⁡\(log⁡L\)\\Omega\(\\log L\)\. This result shows that multilinguality is not entirely free: it requires an additional capacity cost depending onLL\. However, given that the number of languages in the world is finite and usually estimated at around∼\\sim6700, the log dependence of the cost indicates only a small multilinguality tax as we increase language coverage, provably as small as3030extra dimensions for70007000languages\. Thus, we show that there is no theoretical curse of multilinguality for embedding space structure\.

The above finding suggests that any empirical curse of multilinguality for embedding space structure is a consequence of real\-world conditions such as data and compute constraints, among other things\. We characterize the empiricalCOM\-EMBvia a controlled study of the empirical degradation of the multilinguality conditions with increasing language coverage in Transformer\-based models, in eight different configurations varying compute constraints, sampling conditions, and the set of languages being evaluated\. We show that while we do observe “cursed” behaviour in line with general wisdom, this behaviour depends on these configurations and is mitigated in favourable conditions\.

Our paper contributes the first theoretical and intrinsic grounding to the curse of multilinguality\. In showing that perfect intrinsic multilinguality is theoretically achievable at an arbitrary language scale, we set the bar for empirical multilingual spaces, and promote future work in exploring and closing the gap\.

## 2Background

#### Curse of multilinguality

The “curse of multilinguality” or phenomenon of “negative interference” is understood as the observed degradation in multilingual model performance as the number of training languages is increased for a fixed capacity model, commonly described in terms of capacity dilution over languages\([1](https://arxiv.org/html/2608.17088#bib.bib1);[27](https://arxiv.org/html/2608.17088#bib.bib13)\)\. While multilinguality provides benefits up to a point especially for low\-resource languages via cross\-lingual transfer from other language training data, high\-resource language performance usually suffers; further, after a point, all language performance suffers from including more training languages, constituting a “transfer\-dilution” tradeoff\([4](https://arxiv.org/html/2608.17088#bib.bib2);[7](https://arxiv.org/html/2608.17088#bib.bib4);[18](https://arxiv.org/html/2608.17088#bib.bib8)\)\.[31](https://arxiv.org/html/2608.17088#bib.bib16)show that the proportion of the target language with respect to total training data is an important factor, and that increasing model size helps, raising the question of the minimum model size required to accommodate a given language coverage\.[23](https://arxiv.org/html/2608.17088#bib.bib11)develop empirical scaling laws for model and data size while increasing language coverage\.

There are several works that attempt to mitigate the phenomenon of degradation, including strategies for better data sampling\([35](https://arxiv.org/html/2608.17088#bib.bib17);[21](https://arxiv.org/html/2608.17088#bib.bib10);[15](https://arxiv.org/html/2608.17088#bib.bib7)\), architectural innovations such as mixture\-of\-expert models\([27](https://arxiv.org/html/2608.17088#bib.bib13)\), and optimization techniques such as Gradient Vaccine\([36](https://arxiv.org/html/2608.17088#bib.bib18)\)\.

The above works explore the curse of multilinguality from a purely empirical lens\. To the best of our knowledge, we are the first to provide a theoretical lens on the curse of multilinguality\.

#### Intrinsic characterization of embedding spaces

Previous works in the curse of multilinguality study embedding space quality only indirectly by measuring extrinsic performance on tasks such as XNLI\([10](https://arxiv.org/html/2608.17088#bib.bib30)\)and machine translation\([1](https://arxiv.org/html/2608.17088#bib.bib1)\)\. However, there is broad general interest in investigating the structure of multilingual spaces, including assessing cross\-lingual overlap and alignment as studied by[41](https://arxiv.org/html/2608.17088#bib.bib27);[38](https://arxiv.org/html/2608.17088#bib.bib26);[30](https://arxiv.org/html/2608.17088#bib.bib15);[24](https://arxiv.org/html/2608.17088#bib.bib12);[19](https://arxiv.org/html/2608.17088#bib.bib9), among others, and the encoding of language versus semantic information\([8](https://arxiv.org/html/2608.17088#bib.bib3);[40](https://arxiv.org/html/2608.17088#bib.bib28)\)\. Tasks such as monolingual and cross\-lingual retrieval and bitext mining depend directly on representation structure\([28](https://arxiv.org/html/2608.17088#bib.bib14);[13](https://arxiv.org/html/2608.17088#bib.bib29)\), and inspire the formulation of our multilinguality conditions \([Section3](https://arxiv.org/html/2608.17088#S3)\) and metrics \([SectionB\.1](https://arxiv.org/html/2608.17088#A2.SS1)\)\. However, to the best of our knowledge, no previous work seeks to understand the curse of multilinguality from the intrinsic perspective of its impact on embedding space structure\.

## 3Multilinguality Conditions for Perfect Multilingual Spaces

In this section, we formulate conditions necessary for perfect multilinguality, given a reference high\-quality monolingual concept space\.

#### Preliminaries

We begin by defining key structures relevant to our argument\. Let𝒞\\mathcal\{C\}be a global set of concepts, and letℒ\\mathcal\{L\}be a set of languages\. LetZZbe the reference concept space, wherec∈𝒞c\\in\\mathcal\{C\}is embedded aszcz\_\{c\}with high\-quality semantic relationships\. For the multilingual embedding spaceXLX^\{L\}encodingL=\|ℒ\|L=\|\\mathcal\{L\}\|languages, letxc,ℓx\_\{c,\\ell\}represent the embedding for conceptc∈𝒞c\\in\\mathcal\{C\}in languageℓ∈ℒ\\ell\\in\\mathcal\{L\}\. LetXℓL⊂XLX^\{L\}\_\{\\ell\}\\subset X^\{L\}denote the monolingual subspace of languageℓ∈ℒ\\ell\\in\\mathcal\{L\}\. We usecos⁡\(⋅,⋅\)\\cos\(\\cdot,\\cdot\)to refer to the cosine similarity operator\. Finally, we useγ\\gammato refer to the resolution scale ofZZby which any two distinct points are minimally separated: i\.e\. we haveγ<1\\gamma<1such that∀c,c′∈𝒞,c≠c′,cos⁡\(zc,zc′\)<γ\\forall c,c^\{\\prime\}\\in\\mathcal\{C\},c\\neq c^\{\\prime\},\\cos\(z\_\{c\},z\_\{c^\{\\prime\}\}\)<\\gamma\.

#### Monolingual structure

A perfect multilingual space should encode high\-quality shared monolingual relationships for each encoded language\. We use the high\-quality concept spaceZZas reference\. Then, perfect monolingual structure requires that for each languageℓ\\ellrepresented in spaceXLX^\{L\}, its monolingual structureXℓLX^\{L\}\_\{\\ell\}exactly mimicsZZ\.222In practice, we do not know what perfect concept relationships are\. However, we can consider the monolingual space of a high\-resource language \(e\.g\. English\) as a reference space, making the monolingual structure condition equivalent to requiring parity for monolingual quality and shared semantics across languages\.

Monolingual structure can be understood in terms of relative similarity judgments between pairs of concepts\. Thus, perfect monolingual structure can be understood as the monolingual subspace of each language exhibiting identical similarity rankings as the reference concept space, resulting in high\-quality and shared semantics across languages\.

###### Definition 1\(Monolingual Structure\)\.

A multilingual spaceXLX^\{L\}satisfies the monolingual structure condition if each encoded monolingual subspace in it is identical in terms of similarity rankings \(∼R\\sim\_\{R\}\) to a high\-quality concept spaceZZ, i\.e\.∀ℓ∈ℒ,XLℓ∼RZ\\forall\\ell\\in\\mathcal\{L\},X^\{L\}\_\{\\ell\}\\sim\_\{R\}Z\. This condition requires that∀ℓ∈ℒ\\forall\\ell\\in\\mathcal\{L\}andc,c′,d,d′∈𝒞c,c^\{\\prime\},d,d^\{\\prime\}\\in\\mathcal\{C\},cos⁡\(zc,zc′\)≤cos⁡\(zd,zd′\)⇔cos⁡\(xc,ℓ,xc′,ℓ\)≤cos⁡\(xd,ℓ,xd′,ℓ\)\\cos\(z\_\{c\},z\_\{c^\{\\prime\}\}\)\\leq\\cos\(z\_\{d\},z\_\{d^\{\\prime\}\}\)\\iff\\cos\(x\_\{c,\\ell\},x\_\{c^\{\\prime\},\\ell\}\)\\leq\\cos\(x\_\{d,\\ell\},x\_\{d^\{\\prime\},\\ell\}\)\.

Figure 1:Illustration of[1](https://arxiv.org/html/2608.17088#Thmdff1)\.*Dog=Hund*,*Cat=Katze*,*Bark=Bellen*\. The concept space is depicted in English\. The German space on the left exhibits identical ordering of neighbours with the concept space while the German space on the right does not, because the nearest neighbour of*Katze\(=Cat\)*is*Bellen\(=Bark\)*and not*Hund\(=Dog\)*\.

#### Cross\-lingual alignment

A perfect multilingual space displays cross\-lingual alignment, understood as the property that language variants of a given concept are close together, and farther from other concepts\. Past literature distinguishes a “strong” and “weak” view of this idea\([28](https://arxiv.org/html/2608.17088#bib.bib14);[19](https://arxiv.org/html/2608.17088#bib.bib9)\)\. In the “strong” view, we require a concept in languageℓ\\ellto be closer to its translation equivalent in languageℓ′\\ell^\{\\prime\}than any different concept inanylanguage, including other concepts inℓ\\ell, whereas in the weak view, it is only required to be closer than any otherℓ′\\ell^\{\\prime\}\-language points\. We use the strong view, which also trivially implies the weak one\.

###### Definition 2\(Cross\-lingual alignment\)\.

A multilingual spaceXLX^\{L\}satisfies the cross\-lingual alignment condition if a concept encoding in a given language is closer to the encoding of the same concept expressed in any other language than to the encoding of any other concept in any language\. That is, this condition requires that for all conceptsc,c′∈𝒞c,c^\{\\prime\}\\in\\mathcal\{C\}wherec≠c′c\\neq c^\{\\prime\}and languagesℓ,m,n∈ℒ\\ell,m,n\\in\\mathcal\{L\},cos⁡\(xc,ℓ,xc,m\)\>cos⁡\(xc,ℓ,xc′,n\)\\cos\(x\_\{c,\\ell\},x\_\{c,m\}\)\>\\cos\(x\_\{c,\\ell\},x\_\{c^\{\\prime\},n\}\)\.

Figure 2:Illustration of[2](https://arxiv.org/html/2608.17088#Thmdff2), with two languages\. In the left space,*Hund\(=Dog\)*is closer to*Dog*than any different\-concept word, but in the right space, the different\-concept word*Cat*is closer to*Dog*than*Hund=\(Dog\)*\.

#### Non\-degeneracy\.

A multilingual space can place all concept translation equivalents as co\-incident at the same point in the reference high\-quality monolingual space, and thus trivially exhibit monolingual structure as well as cross\-lingual alignment\. However, this violates common practical assumptions of embedding spaces where an embedding space must be capable of distinguishing between any two embedded points with some fixed scale of resolution\([37](https://arxiv.org/html/2608.17088#bib.bib38)\)\. Thus, we require that our multilingual space have a resolution scale, or a maximum cosine similarityγX<1\\gamma\_\{X\}<1such that for any distinct\(c,ℓ\),\(d,m\)∈𝒞×ℒ\(c,\\ell\),\(d,m\)\\in\\mathcal\{C\}\\times\\mathcal\{L\},cos⁡\(xc,ℓ,xd,m\)<γX\\cos\(x\_\{c,\\ell\},x\_\{d,m\}\)<\\gamma\_\{X\}\.

## 4Theoretical Curse of Multilinguality for Embedding Space Structure

### 4\.1Defining the curse of multilinguality

TheCOMis described in the field for empirical spaces as the degradation of desired properties – usually, extrinsic task performance for some set of languages – with an increasing number of encoded languages, given fixed model capacity\. For theCOMwith respect to embedding space structure \(COM\-EMB\), “desired properties” correspond to perfect multilinguality, and “capacity” corresponds to dimensionality\.We formulate theCOM\-EMBas the inability of a multilingual embedding space to maintain perfect multilinguality given an increasing number of encoded languagesLLwithout a prohibitive increase in dimensionality\.

### 4\.2Theorem

In this section, we answer the following question:*how much dimensionality does a perfect multilingual space accommodatingLLlanguages theoretically require in terms ofLL?*If the required dimensionality grows prohibitively with the number of languages, e\.g\. linearly or exponentially, then this can be termed a theoreticalCOM\-EMB\.

###### Theorem 1\.

The minimum dimensionality of a perfect multilingual space encodingLLlanguages grows asΘ⁡\(log⁡L\)\\Theta\(\\log L\)\.

We prove this in two parts\. First, we upper\-bound the minimum*sufficient*dimensionality of a perfect multilingual space in terms of the number of encoded languages and the dimensionality of a fixed target concept space, in line with our setup in[Section3](https://arxiv.org/html/2608.17088#S3)\.

###### Theorem 1\.1\.

LetDZD\_\{Z\}be the dimensionality of the reference concept spaceZZ\. Then there exists a perfect multilingual spaceXLX^\{L\}encodingLLlanguages with minimum sufficient dimensionalityDXL=DZ\+O⁡\(log⁡L\)D\_\{X^\{L\}\}=D\_\{Z\}\+O\(\\log L\)\.

Next, we lower\-bound the dimensionality of the space, i\.e\. show the*necessary*dimensionality of a perfect multilingual space\.

###### Theorem 1\.2\.

Any perfect multilingual spaceXLX^\{L\}encodingLLlanguages must have dimensionalityDXL=Ω⁡\(log⁡L\)D\_\{X^\{L\}\}=\\Omega\(\\log L\)\.

SinceDZD\_\{Z\}is independent ofLL,[1\.1](https://arxiv.org/html/2608.17088#Thmthm1.Thmsubthm1)and[1\.2](https://arxiv.org/html/2608.17088#Thmthm1.Thmsubthm2)together establish[1](https://arxiv.org/html/2608.17088#Thmthm1)\.

The following proofs make a fixed unit norm assumption forZZandXLX^\{L\}\. This is standard in the use of text embeddings for applications such as retrieval\([37](https://arxiv.org/html/2608.17088#bib.bib38)\)\. Our proofs are also extendable to the case of variable norm as sketched in[SectionA\.4](https://arxiv.org/html/2608.17088#A1.SS4)\.

### 4\.3Proof of[1\.1](https://arxiv.org/html/2608.17088#Thmthm1.Thmsubthm1)

We proceed constructively\. GivenZZandLL, we describe a multilingual spaceXLX^\{L\}and show that \(a\) it maintains the multilinguality conditions, and that \(b\) its dimensionality follows the required bound\.

#### Multilingual space construction

Recall that for a conceptcc, we have its embeddingzcz\_\{c\}in the concept spaceZZ, with the resolution scaleγ\\gamma, and dimensionalityDZD\_\{Z\}\.

We now define a spaceUUof*language offsets*\. This space usesDLD\_\{L\}new dimensions, and is populated withLLlanguage offset vectorsuℓ∈ℝDLu\_\{\\ell\}\\in\\mathbb\{R\}^\{D\_\{L\}\}, with a fixed norm‖uℓ‖=r\\\|u\_\{\\ell\}\\\|=r, and with a maximum similarityρ<1\\rho<1between any two language offsets\. That is, we havecos⁡\(uℓ,um\)<ρ\\cos\(u\_\{\\ell\},u\_\{m\}\)<\\rhofor allℓ,m∈ℒ\\ell,m\\in\\mathcal\{L\}whereℓ≠m\\ell\\neq m\.

We then define our multilingual spaceXLX^\{L\}, by a direct sum construction of the concept space and the language offsets\. That is,∀c∈𝒞,ℓ∈ℒ,xc,ℓ:=11\+r2​zc⊕uℓ\\forall c\\in\\mathcal\{C\},\\ell\\in\\mathcal\{L\},x\_\{c,\\ell\}:=\\frac\{1\}\{\\sqrt\{1\+r^\{2\}\}\}z\_\{c\}\\oplus u\_\{\\ell\}, where⊕\\oplusrefers to vector concatenation\. As a result,xc,ℓ∈XLx\_\{c,\\ell\}\\in X^\{L\}has dimensionalityDXL=DZ\+DLD\_\{X^\{L\}\}=D\_\{Z\}\+D\_\{L\}\. Note the norm term11\+r2\\frac\{1\}\{\\sqrt\{1\+r^\{2\}\}\}\. Since‖zc‖=1\\\|z\_\{c\}\\\|=1and‖uℓ‖=r\\\|u\_\{\\ell\}\\\|=r, andzcz\_\{c\}anduℓu\_\{\\ell\}are orthogonal when embedded inXLX^\{L\}, we now haveXLX^\{L\}as a unit norm space with‖xc,ℓ‖=1\\\|x\_\{c,\\ell\}\\\|=1for allc,ℓc,\\elland dimensionalityDXLD\_\{X^\{L\}\}\.

#### Similarity bounds on offset vectors

We construct the language offset vectors in theDLD\_\{L\}\-dimensional spaceUUwith fixed normrrsuch that their minimum and maximum cosine similarity lies in the following interval, guaranteed to be non\-empty by a choosingr≥1r\\geq 1and a suitable choice ofρ\\rho:

1−1−γr2<cos⁡\(uℓ,um\)<ρ<1for all​ℓ≠m\.1\-\\frac\{1\-\\gamma\}\{r^\{2\}\}<\\cos\(u\_\{\\ell\},u\_\{m\}\)<\\rho<1\\qquad\\text\{for all \}\\ell\\neq m\.\(1\)

#### Similarity decomposition

Given thatXLX^\{L\}andZZare unit\-norm, we have the following relationship:cos⁡\(xc,ℓ,xd,m\)=⟨xc,ℓ,xd,m⟩\\cos\(x\_\{c,\\ell\},x\_\{d,m\}\)=\\langle x\_\{c,\\ell\},x\_\{d,m\}\\rangle; analogously forZZ\. Sincezcz\_\{c\}anduℓu\_\{\\ell\}are orthogonal when embedded inXLX^\{L\}, we make the following useful inner\-product decomposition for points inXLX^\{L\}, used throughout our proof:

⟨xc,ℓ,xd,m⟩\\displaystyle\\langle x\_\{c,\\ell\},x\_\{d,m\}\\rangle=⟨zc⊕uℓ1\+r2,zd⊕um1\+r2⟩=11\+r2​⟨zc⊕uℓ,zd⊕um⟩=⟨zc,zd⟩\+⟨uℓ,um⟩1\+r2\\displaystyle=\\left\\langle\\frac\{z\_\{c\}\\oplus u\_\{\\ell\}\}\{\\sqrt\{1\+r^\{2\}\}\},\\frac\{z\_\{d\}\\oplus u\_\{m\}\}\{\\sqrt\{1\+r^\{2\}\}\}\\right\\rangle=\\frac\{1\}\{1\+r^\{2\}\}\\left\\langle z\_\{c\}\\oplus u\_\{\\ell\},z\_\{d\}\\oplus u\_\{m\}\\right\\rangle=\\frac\{\\langle z\_\{c\},z\_\{d\}\\rangle\+\\langle u\_\{\\ell\},u\_\{m\}\\rangle\}\{1\+r^\{2\}\}\(2\)
###### Lemma 1\.

The spaceXLX^\{L\}satisfies the monolingual structure condition \([1](https://arxiv.org/html/2608.17088#Thmdff1)\)\.

###### Proof\.

We will show that for every languageℓ\\ell,XLℓ∼RZX^\{L\}\_\{\\ell\}\\sim\_\{R\}Z, i\.e\. each language subspace is identical toZZunder point similarity ranking\. Using[Equation2](https://arxiv.org/html/2608.17088#S4.E2)for points inXℓLX^\{L\}\_\{\\ell\}, we have

cos⁡\(xc,ℓ,xc′,ℓ\)\\displaystyle\\cos\(x\_\{c,\\ell\},x\_\{c^\{\\prime\},\\ell\}\)=⟨xc,ℓ,xc′,ℓ⟩=⟨zc,zc′⟩\+⟨uℓ,uℓ⟩1\+r2=⟨zc,zc′⟩\+r21\+r2=cos⁡\(zc,zc′\)\+r21\+r2\\displaystyle=\\langle x\_\{c,\\ell\},x\_\{c^\{\\prime\},\\ell\}\\rangle=\\frac\{\\langle z\_\{c\},z\_\{c^\{\\prime\}\}\\rangle\+\\langle u\_\{\\ell\},u\_\{\\ell\}\\rangle\}\{1\+r^\{2\}\}=\\frac\{\\langle z\_\{c\},z\_\{c^\{\\prime\}\}\\rangle\+r^\{2\}\}\{1\+r^\{2\}\}=\\frac\{\\cos\(z\_\{c\},z\_\{c^\{\\prime\}\}\)\+r^\{2\}\}\{1\+r^\{2\}\}=acos\(zc,zc′\)\+b\\displaystyle=a\\cos\(z\_\{c\},z\_\{c^\{\\prime\}\}\)\+bfor fixed and globalaaandbbdepending onlyrr\. Note that this automatically gives us, for any two languagesℓ,m\\ell,m,

cos⁡\(xc,ℓ,xc′,ℓ\)\\displaystyle\\cos\(x\_\{c,\\ell\},x\_\{c^\{\\prime\},\\ell\}\)=a⋅cos⁡\(zc,zc′\)\+b=cos⁡\(xc,m,xc′,m\)\\displaystyle=a\\cdot\\cos\(z\_\{c\},z\_\{c^\{\\prime\}\}\)\+b=\\cos\(x\_\{c,m\},x\_\{c^\{\\prime\},m\}\)
Given this affine behavior, and the language agnosticity of the result, it follows that the similarity ranking behavior is identical in all languagesℓ∈ℒ\\ell\\in\\mathcal\{L\}with respect toZZ, i\.e\.∀ℓ∈ℒ,XLℓ∼RZ\\forall\\ell\\in\\mathcal\{L\},X^\{L\}\_\{\\ell\}\\sim\_\{R\}Z\. Hence,XLX^\{L\}satisfies the monolingual structure condition\. ∎

###### Lemma 2\.

The spaceXLX^\{L\}satisfies the cross\-lingual alignment condition \([2](https://arxiv.org/html/2608.17088#Thmdff2)\)\.

###### Proof\.

We show that for everyc,c′∈𝒞c,c^\{\\prime\}\\in\\mathcal\{C\}wherec≠c′c\\neq c^\{\\prime\}and all languagesℓ,m,n∈ℒ\\ell,m,n\\in\\mathcal\{L\},cos⁡\(xc,ℓ,xc,m\)\>cos⁡\(xc,ℓ,xc′,n\)\\cos\(x\_\{c,\\ell\},x\_\{c,m\}\)\>\\cos\(x\_\{c,\\ell\},x\_\{c^\{\\prime\},n\}\)\. Recall that points inZZhave resolution scaleγ\\gammaand points inUUhave normrr\. From the similarity decomposition \([Equation2](https://arxiv.org/html/2608.17088#S4.E2)\), we can simplify and bound the two similarity expressions as follows for anyc≠c′,ℓ,m,nc\\neq c^\{\\prime\},\\ell,m,n:

cos⁡\(xc,ℓ,xc,m\)\\displaystyle\\cos\(x\_\{c,\\ell\},x\_\{c,m\}\)=⟨zc,zc⟩\+⟨uℓ,um⟩1\+r2=1\+⟨uℓ,um⟩1\+r2\\displaystyle=\\frac\{\\langle z\_\{c\},z\_\{c\}\\rangle\+\\langle u\_\{\\ell\},u\_\{m\}\\rangle\}\{1\+r^\{2\}\}=\\frac\{1\+\\langle u\_\{\\ell\},u\_\{m\}\\rangle\}\{1\+r^\{2\}\}cos⁡\(xc,ℓ,xc′,n\)\\displaystyle\\cos\(x\_\{c,\\ell\},x\_\{c^\{\\prime\},n\}\)=⟨zc,zc′⟩\+⟨uℓ,un⟩1\+r2<γ\+r21\+r2\.\\displaystyle=\\frac\{\\langle z\_\{c\},z\_\{c^\{\\prime\}\}\\rangle\+\\langle u\_\{\\ell\},u\_\{n\}\\rangle\}\{1\+r^\{2\}\}<\\frac\{\\gamma\+r^\{2\}\}\{1\+r^\{2\}\}\.
Now it suffices to show that

1\+⟨uℓ,um⟩\>γ\+r2\.1\+\\langle u\_\{\\ell\},u\_\{m\}\\rangle\>\\gamma\+r^\{2\}\.
Since⟨uℓ,um⟩=r2cos\(uℓ,um\)\\langle u\_\{\\ell\},u\_\{m\}\\rangle=r^\{2\}\\cos\(u\_\{\\ell\},u\_\{m\}\), the above expression is equivalent to

cos⁡\(uℓ,um\)\>1−1−γr2\.\\cos\(u\_\{\\ell\},u\_\{m\}\)\>1\-\\frac\{1\-\\gamma\}\{r^\{2\}\}\.which is guaranteed by our construction in[Equation1](https://arxiv.org/html/2608.17088#S4.E1)\. Thus,XLX^\{L\}satisfies the cross\-lingual alignment condition\. ∎

###### Lemma 3\.

The spaceXLX^\{L\}is non\-denegerate\.

###### Proof\.

The cosine similarity between any two distinct points inXLX^\{L\}is given by[Equation2](https://arxiv.org/html/2608.17088#S4.E2)\. We bound the maximum similarity between any two points based on the three cases: same concept with different language, different concept with same language, and different concept with different language\. Ifc=c′c=c^\{\\prime\}andℓ≠m\\ell\\neq m, thencos⁡\(xc,ℓ,xc,m\)<1\+r2​ρ1\+r2\\cos\(x\_\{c,\\ell\},x\_\{c,m\}\)<\\frac\{1\+r^\{2\}\\rho\}\{1\+r^\{2\}\}\. Ifc≠c′c\\neq c^\{\\prime\}andℓ=m\\ell=m, thencos⁡\(xc,ℓ,xc′,ℓ\)<γ\+r21\+r2\\cos\(x\_\{c,\\ell\},x\_\{c^\{\\prime\},\\ell\}\)<\\frac\{\\gamma\+r^\{2\}\}\{1\+r^\{2\}\}\. If bothc≠c′c\\neq c^\{\\prime\}andℓ≠m\\ell\\neq m, then

cos⁡\(xc,ℓ,xc′,m\)\\displaystyle\\cos\(x\_\{c,\\ell\},x\_\{c^\{\\prime\},m\}\)=⟨zc,zc′⟩\+⟨uℓ,um⟩1\+r2<γ\+r2​ρ1\+r2<γX:=max⁡\{γ\+r21\+r2,1\+r2​ρ1\+r2\}\.\\displaystyle=\\frac\{\\langle z\_\{c\},z\_\{c^\{\\prime\}\}\\rangle\+\\langle u\_\{\\ell\},u\_\{m\}\\rangle\}\{1\+r^\{2\}\}<\\frac\{\\gamma\+r^\{2\}\\rho\}\{1\+r^\{2\}\}<\\gamma\_\{X\}:=\\max\\left\\\{\\frac\{\\gamma\+r^\{2\}\}\{1\+r^\{2\}\},\\frac\{1\+r^\{2\}\\rho\}\{1\+r^\{2\}\}\\right\\\}\.
Note thatγX<1\\gamma\_\{X\}<1, sinceγ<1\\gamma<1andρ<1\\rho<1\.γX\\gamma\_\{X\}is independent ofLLand only depends on the constantsγ\\gamma,ρ\\rhoandrr; thus,XLX^\{L\}maintains a fixed cosine scale independent of the number of languages encoded\. Thus,XLX^\{L\}satisfies the non\-degeneracy condition at fixed resolution scaleγX<1\\gamma\_\{X\}<1\. ∎

###### Lemma 4\.

The minimum sufficient dimensionality ofXLX^\{L\}isO⁡\(log⁡L\)O\(\\log L\)\.

###### Proof\.

Recall thatDXL=DZ\+DLD\_\{X^\{L\}\}=D\_\{Z\}\+D\_\{L\}\. We will boundDLD\_\{L\}, the dimensionality of the language offset spaceUU\.

UUis constructed by placingLLfixed\-norm language offsets inUUsuch that each point pair has cosine similarity in the non\-empty interval defined in[Equation1](https://arxiv.org/html/2608.17088#S4.E1)\. BoundingDLD\_\{L\}therefore reduces to the following question: how many dimensionsDLD\_\{L\}are sufficient to packLLpoints on a sphere in aDLD\_\{L\}\-dimensional with specified maximum and minimum cosine similarity?

Using standard spherical cap packing bounds, we can show that the number of points that can be placed on a sphere in aDLD\_\{L\}\-dimensional space following these constraints grows exponentially inDLD\_\{L\}\.333This comes with the constraint on a choice ofρ\\rho\. Forr=1r=1, this isarccos⁡\(ρ\)<arccos⁡\(γ\)/2\\arccos\(\\rho\)<\\arccos\(\\gamma\)/2\.This is shown in detail in[SectionA\.1](https://arxiv.org/html/2608.17088#A1.SS1)\. Briefly, the maximum similarity constraint imposes an angular region within which all points must lie, and for a maximal packing, caps with angular radius equal to the minimum angular separation centered at the packed points must cover this containing region\. This allows us to lower\-bound the number of points in a maximal packing by a ratio of spherical cap measures, which grows exponentially with the dimensionality of the sphere\. Therefore, to representLLlanguage offsets, it is possible to choose some constantC\>0C\>0independent ofLLsuch that a sphere in aDLD\_\{L\}\-dimensional space withDL=⌈1C​log⁡L⌉D\_\{L\}=\\left\\lceil\\frac\{1\}\{C\}\\log L\\right\\rceilfitsexp⁡\(C​DL\)≥L\\exp\(CD\_\{L\}\)\\geq Lsuch points\. Therefore, the minimum sufficient dimension to packLLpoints maintaining the required constraints isDL=O⁡\(log⁡L\)D\_\{L\}=O\(\\log L\)\. SinceDZD\_\{Z\}is constant in terms ofLL,XLX^\{L\}has the minimum sufficient dimension

DXL=DZ\+DL=DZ\+O⁡\(log⁡L\)⟹DXL=O⁡\(log⁡L\)\\displaystyle D\_\{X^\{L\}\}=D\_\{Z\}\+D\_\{L\}=D\_\{Z\}\+O\(\\log L\)\\implies D\_\{X^\{L\}\}=O\(\\log L\)
Note that it suffices to exhibit a single perfect multilingual space with the above scaling to upper\-bound the minimum sufficient dimensionality for perfect multilinguality\. By the existence ofXLX^\{L\}, we establish the minimum sufficient dimension for perfect multilinguality grows asO⁡\(log⁡L\)O\(\\log L\)\. ∎

#### Fitting 7000 languages in 30 dimensions

For practical purposes, the constant involved in theO⁡\(log⁡L\)O\(\\log L\)bound may be of concern\. In[SectionA\.3](https://arxiv.org/html/2608.17088#A1.SS3), we prove that it is theoretically possible to fitL=7000L=7000language offset unit vectors intoDL=30D\_\{L\}=30dimensions while satisfying all required constraints on minimum and maximum similarity required for our constructive solution, with a realistic choice ofγ\\gamma\. Briefly, we show that a random construction ofLLpoints in aDLD\_\{L\}\-dimensional space has a non\-zero probability of satisfying our required constraints, proving existence\. Thus, our theoretical scaling factor is of minor practical concern given the typical dimensionality scale of representation spaces\.

### 4\.4Proof of[1\.2](https://arxiv.org/html/2608.17088#Thmthm1.Thmsubthm2)

###### Proof\.

Fix any conceptcc\. Then theLLlanguage variants of this concept,xc,1,…,xc,Lx\_\{c,1\},\\ldots,x\_\{c,L\}, are all distinct fixed\-norm vectors, or distinct points on a sphere\. SinceXLX^\{L\}is non\-degenerate as we assume it is perfectly multilingual, any two distinct points in it must remain separated at some fixed cosine resolution scaleγX<1\\gamma\_\{X\}<1, equivalent to a fixed angular separationθX:=arccos⁡\(γX\)\\theta\_\{X\}:=\\arccos\(\\gamma\_\{X\}\), independent ofLL\.444Fixing the resolution scale across the family of spacesXLX^\{L\}allows us to study the asymptotic behaviour of the necessary dimensionality ofXLX^\{L\}with increasingLLin a meaningful manner\. If we allow the resolution scale to shrink with increasing language coverage, our result may not hold\.Thus,XLX^\{L\}contains at leastLLpoints on a multidimensional sphere with a fixed angular separation, and its necessary dimensionality is lower bounded by the number of dimensions needed to place points in such a way\.

We now apply a standard spherical cap bound\. For any fixed angular separationθX\>0\\theta\_\{X\}\>0, the maximum number of vectors that can be placed on aDXLD\_\{X\_\{L\}\}\-dimensional sphere in with pairwise angular separation at leastθX\\theta\_\{X\}grows at most exponentially inDXLD\_\{X\_\{L\}\}\. We show this in detail in[SectionA\.2](https://arxiv.org/html/2608.17088#A1.SS2)\. Briefly, the minimum angular separation means that if we place a spherical cap with half the required angular separation around each point then these caps must be disjoint\. Their total measure cannot exceed that of the sphere\. This upper\-bounds the number of points by the reciprocal of a spherical cap measure, which grows at most exponentially with the dimensionality of the sphere\. That is, there exists a constantAθX\>0A\_\{\\theta\_\{X\}\}\>0, depending only onθX\\theta\_\{X\}, such that

L≤exp⁡\(AθX​DXL\)\\displaystyle L\\leq\\exp\(A\_\{\\theta\_\{X\}\}D\_\{X\_\{L\}\}\)⇒log⁡L≤AθX​DXL⇒DXL≥1AθX​log⁡L\\displaystyle\\Rightarrow\\log L\\leq A\_\{\\theta\_\{X\}\}D\_\{X\_\{L\}\}\\Rightarrow D\_\{X\_\{L\}\}\\geq\\frac\{1\}\{A\_\{\\theta\_\{X\}\}\}\\log L⇒DXL=Ω⁡\(log⁡L\)\\displaystyle\\Rightarrow D\_\{X\_\{L\}\}=\\Omega\(\\log L\)Thus, the dimensionality of any non\-degenerate multilingual space encodingLLlanguages has a logarithmic dependence onLL\. Note that if the original monolingual concept space “fully” requiresDZD\_\{Z\}dimensions to represent concept semantics, then theseΩ⁡\(log⁡L\)\\Omega\(\\log L\)dimensions required to construct language directions are additional to theseDZD\_\{Z\}dimensions\. Under this minimality assumption on the base concept space, the additional cost can be understood as a multilinguality tax:555In general, real embedding spaces may not use their ambient dimensionality\([3](https://arxiv.org/html/2608.17088#bib.bib37)\)\. Instead, this can be understood as placing pressure on true or effective dimensionality of such a space\.

DXL=DZ\+Ω⁡\(log⁡L\)D\_\{X^\{L\}\}=D\_\{Z\}\+\\Omega\(\\log L\)In general, the dimensionDXLD\_\{X^\{L\}\}of any perfect multilingual spaceXLX^\{L\}encodingLLlanguages must grow at least asΩ⁡\(log⁡L\)\\Omega\(\\log L\)\. ∎

## 5Empirical curse of multilinguality for embedding spaces

While our theorem shows that a multilingual space is theoretically capable of exhibiting perfect multilinguality under language scaling without a prohibitive cost, there may still exist an empiricalCOM\-EMBfor real\-world multilingual models\. In this section, we formulate metrics for monolingual structure and cross\-lingual alignment, and provide a small\-scale controlled study showing the empirical curse of multilinguality for embedding space structure under eight configurations mimicking various real\-world conditions\. Specifically, we train multilingual models from scratch on an increasing number of languages and evaluate embedding space structure under language scaling, in different training and evaluation configurations\.666Our characterization of the theoretical curse of multilinguality for embedding space structure allows for a small growth based on number of languages, while in our experiments, we keep the dimensionality fixed with language scale\. This is for simplicity, since it is difficult to estimate the permissible numerical inflation given its dependence on embedding space characteristics such asγ\\gammaas well as packing constants\. In practice, trained embedding spaces use very little of the ambient dimensionality\([14](https://arxiv.org/html/2608.17088#bib.bib6);[3](https://arxiv.org/html/2608.17088#bib.bib37)\), and logarithmic growth for around a hundred languages indicates a small extra constant as discussed in[SectionA\.3](https://arxiv.org/html/2608.17088#A1.SS3)\.We provide precise metric definitions and extra experimental details and results in[AppendixB](https://arxiv.org/html/2608.17088#A2)\.

#### Note on previous work

Previous works studying theCOMand related concepts generally differ with respect to their experimental conditions and assumptions\. Differences include \(i\) whether token compute is fixed\([31](https://arxiv.org/html/2608.17088#bib.bib16)\)or increases with more training languages\([1](https://arxiv.org/html/2608.17088#bib.bib1)\), \(ii\) whether the training data distribution is uniform over languages\([31](https://arxiv.org/html/2608.17088#bib.bib16)\)or follows a real\-world distribution\([1](https://arxiv.org/html/2608.17088#bib.bib1);[9](https://arxiv.org/html/2608.17088#bib.bib5)\), and \(iii\) whether performance metrics are tracked over all training languages\([9](https://arxiv.org/html/2608.17088#bib.bib5)\)or a fixed group such as high\-resource languages\([4](https://arxiv.org/html/2608.17088#bib.bib2)\), among other variables\.

#### Configurations

We study the empiricalCOM\-EMBunder eight configurations consisting of all combinations of the three above dimensions: \(i\)\{fixed\-compute, increasing\-compute\}, \(ii\)\{uniform\-sampling, realistic\-sampling\}, and \(iii\)\{aggregate\-all, aggregate\-fixed\}\.

#### Metrics

Our metrics are inspired from the multilinguality conditions\. We consider a “concept” to be a text, and compute the following using a fixed concept set across all target languages obtained from multiway parallel data\.

We evaluatemonolingual structureagainst a strong reference space, by measuring, for each concept, the overlap of the monolingual neighbourhood of that concept in each target language with the gold neighbourhood of that concept in a strong monolingual reference space\. We report this metric \(*k\-nearest\-neighbour overlap \(MS\-NNO\)*\) as a percentage ofk=20k=20, averaged over concepts and languages\. We also look at language mean over*token\-normalized masked language modeling loss*on a monolingual test set per language \(MS\-MLM\)as an indirect measure of monolingual structure quality\.

We evaluatecross\-lingual alignmentby checking for each languageℓ1\\ell\_\{1\}, whether, for each concept and each languageℓ2\\ell\_\{2\}, theℓ2\\ell\_\{2\}equivalent of that concept is closer to theℓ1\\ell\_\{1\}equivalent than otherℓ2\\ell\_\{2\}concepts \(weak view; similar trends for strong view shown in[SectionB\.3](https://arxiv.org/html/2608.17088#A2.SS3)\)\. This metric \(CLA\-WV\) is reported as a percentage of the concept set, averaged over language pairs in the target set\.

We also check for non\-degeneracy by checking for looking at the minimum, maximum, and mean cosine similarity over all point pairs in the space as a function of increasing language coverage\. We do not see evidence of degeneracy in any configuration \(shown in[SectionB\.3](https://arxiv.org/html/2608.17088#A2.SS3)\)\.

#### Experimental setup

We study a 4\-layer Transformer\-basedBERTencoder architecture with a masked\-language modeling training objective\([12](https://arxiv.org/html/2608.17088#bib.bib39)\)\. We use theMADLAD\-400corpus\([22](https://arxiv.org/html/2608.17088#bib.bib24)\)for monolingual training data, andFLORES\+\([16](https://arxiv.org/html/2608.17088#bib.bib20);[32](https://arxiv.org/html/2608.17088#bib.bib19)\),BOUQuET\([2](https://arxiv.org/html/2608.17088#bib.bib22);[33](https://arxiv.org/html/2608.17088#bib.bib23)\), andWMT24\+\+\([11](https://arxiv.org/html/2608.17088#bib.bib21)\)as multiway parallel data for evaluation\. We use the English concept space frome5\-large\-v2\([34](https://arxiv.org/html/2608.17088#bib.bib40)\)as a strong reference space forMS\-NNO\. We create nine language groups with incrementally more languages added in descending order of resourcedness, with the largest language group containing 100 languages\.777This is realistic in terms of how language coverage is expanded in the real world; we are also driven by data requirement constraints since initial smaller language groups require more data per language for some configurations\.The token count per language depends on the token\-compute and sampling settings:fixed\-computeexperiments are trained on a total budget of 500M tokens, andincreasing\-computewithuniform\-samplinguses 50M tokens per language\. Therealistic\-samplingsetting uses 500M as the maximum language token count, with other language token count proportionate to resource level in our training corpus\. Each model run consists of 5 seeds, with each seed sampling different data up to the budget, and random initialization of model\. The tokenizer is trained on a subset of the training corpus\. For theaggregate\-allsetting, we evaluate all training languages covered by each evaluation dataset\. Foraggregate\-fixed, the smallest language group is 10, and we evaluate these 10 fixed languages across all language groups\. We use the\[CLS\]token embedding as a text embedding for our trained models\.

#### Results and discussion

Figure 3:MS\-NNO,MS\-MLM, andCLA\-WVonBOUQuETfor encoder models with as the number of training languages increases, in eight configurations\. The top and bottom rows show cursed and non\-cursed configurations respectively\. Arrows indicate whether higher or lower metric values are better\. Error bars show standard deviation over five model seeds\.If all metrics exhibit deterioration for a configuration, we term it a “cursed” configuration\.[Figure3](https://arxiv.org/html/2608.17088#S5.F3)shows results for all eight configurations withBOUQuET, separating cursed from non\-cursed configurations\.888We see similar trends for the other two evaluation datasets and show them in[SectionB\.3](https://arxiv.org/html/2608.17088#A2.SS3)\.We observe four cursed configurations of eight, showing consistent degradation of monolingual structure and cross\-lingual alignment\.Therefore, we observe an empirical curse of multilinguality for embedding space structure, parallelling previous work in theCOM\.

In the other four non\-cursed configurations, we observe stability or improvements in one or more metrics\. We find that these are the configurations that generally maintain high token count for the target languages of interest as a consequence of compute and sampling choices\. For example, withincreasing\-compute,realistic\-sampling, andaggregate\-fixed\(\), token count for the 10 high\-resource target languages remains high and unchanged with more training languages, and metrics stay roughly stable\. We also see that metrics may actually show some improvements with more languages up to a point, e\.g\. withincreasing\-computeanduniform\-sampling\(\), which indicates that total token count and cross\-lingual transfer may also play a role\.In general, our findings suggest that the empirical curse of multilinguality for embedding space structure is sensitive to training and evaluation configurations and is mitigated in favourable conditions\.

#### Recommendation regarding empirical studies

The above finding regarding theCOM\-EMBparallels the fact that various previous empirical studies of the extrinsicCOMdiffer in their characterization of the phenomenon\. These often operate in different configurations based on their application of interest, overloading the catchall term “curse of multilinguality” and related concepts\. Given our findings, we recommend that future work make their configuration of interest explicit to promote scientific clarity regarding theCOM\.

## 6Conclusion

The curse of multilinguality describes the phenomenon of performance degradation in fixed capacity multilingual models with increasing language scale\. Our work provides the first theoretical and intrinsic grounding to this phenomenon\. Specifically, we look at whether multilingual representation spaces can theoretically can be scaled in language coverage while maintaining their quality\. We formulate the notion of*perfect multilinguality*for embedding spaces, embodied by two*multilinguality conditions*\. Our main contribution is a theoretical result that shows it is necessary and sufficient to scale representation space capacity only logarithmically in the number of languages while maintaining perfect multilinguality\. This shows that there is no theoretical curse of multilinguality for embedding space structure\. We also demonstrate the empirical curse of multilinguality for embedding space structure for the first time, and show that it is sensitive to data, compute, and evaluation conditions\. Our findings contribute to better formulating and understanding the curse of multilinguality, and raise questions regarding bridging the gap between theory and practice\.

## Limitations

#### Perfect multilinguality

Our formulation of perfect multilinguality for multilingual representation spaces is most directly relevant for applications such as embedding\-based retrieval\. Different multilingual representation spaces, such as intermediate representations of decoder models, may require adaptation based on needs from different downstream application scenarios\. Further, this conception uses a simplified scenario consisting of a global concept set with global semantics across languages\. This allows the cost of concept semantics to stay constant as language coverage grows in theory\. In practice, different languages may introduce language\-specific concepts, or have differing semantic relationships between concepts, introducing language\-specific needs and additional dimensionality costs\. We leave this to future work to incorporate into a theoretical model\.

#### Model scale

Our empirical studies are for small encoder models given the controlled nature and language scale of our study\. We leave it to future work to study the impact of model scaling and different architectures on the curse of multilinguality for embedding space structure\.

### AI use statement

In this work, we used generative AI tools as a soundboard for mathematical ideas, for ironing out details of and checking for errors in proofs, and for generating the idea for the proof in[SectionA\.3](https://arxiv.org/html/2608.17088#A1.SS3)\. All mathematical arguments in this paper have been written and verified by the authors\. We used AI\-assisted literature search tools in combination with manual search for discovering previous work, following by a manual review of relevance\. We used coding assistants for our experiments, with human review\. AI was used for typesetting of mathematical expressions, refining phrasing, and formatting scientific figures in the writing of this paper, and all AI\-generated phrasing has been checked by authors\. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI\.

## References

- Aharoniet al\.\(2019\)R\. Aharoni, M\. Johnson, and O\. FiratMassively Multilingual Neural Machine Translation\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Minneapolis, Minnesota,pp\. 3874–3884\.External Links:[Document](https://dx.doi.org/10.18653/v1/N19-1388),[Link](https://aclanthology.org/N19-1388/)Cited by:[§1](https://arxiv.org/html/2608.17088#S1.p1.1),[§2](https://arxiv.org/html/2608.17088#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2608.17088#S2.SS0.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2608.17088#S5.SS0.SSS0.Px1.p1.1)\.
- Andrewset al\.\(2025\)P\. Andrews, M\. Artetxe, M\. C\. Meglioli, M\. R\. Costa\-jussà, J\. Chuang, D\. Dale, M\. Duppenthaler, N\. P\. Ekberg, C\. Gao, D\. E\. Licht, J\. Maillard, A\. Mourachko, C\. Ropers, S\. Saleem, E\. Sánchez, I\. Tsiamas, A\. Turkatenko, A\. Ventayol\-Boada, and S\. YatesBOUQuET : dataset, benchmark and open initiative for universal quality evaluation in translation\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 27515–27535\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.1400/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1400),ISBN 979\-8\-89176\-332\-6Cited by:[§B\.2](https://arxiv.org/html/2608.17088#A2.SS2.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2608.17088#S5.SS0.SSS0.Px4.p1.1)\.
- Ansuiniet al\.\(2019\)A\. Ansuini, A\. Laio, J\. H\. Macke, and D\. ZoccolanIntrinsic dimension of data representations in deep neural networks\.Advances in Neural Information Processing Systems32\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2019/hash/cfcce0621b49c983991ead4c3d4d3b6b-Abstract.html)Cited by:[footnote 5](https://arxiv.org/html/2608.17088#footnote5),[footnote 6](https://arxiv.org/html/2608.17088#footnote6)\.
- Arivazhaganet al\.\(2019\)N\. Arivazhagan, A\. Bapna, O\. Firat, D\. Lepikhin, M\. Johnson, M\. Krikun, M\. X\. Chen, Y\. Cao, G\. Foster, C\. Cherry, W\. Macherey, Z\. Chen, and Y\. WuMassively Multilingual Neural Machine Translation in the Wild: Findings and Challenges\.arXiv\.External Links:1907\.05019,[Document](https://dx.doi.org/10.48550/arXiv.1907.05019),[Link](http://arxiv.org/abs/1907.05019)Cited by:[§2](https://arxiv.org/html/2608.17088#S2.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2608.17088#S5.SS0.SSS0.Px1.p1.1)\.
- Barmanet al\.\(2026\)S\. R\. Barman, A\. Starenky, S\. Bodnar, N\. Narasimhan, and A\. GopinathThe Geometry of Forgetting\.arXiv\.External Links:2604\.06222,[Document](https://dx.doi.org/10.48550/arXiv.2604.06222),[Link](http://arxiv.org/abs/2604.06222)Cited by:[§B\.4](https://arxiv.org/html/2608.17088#A2.SS4.SSS0.Px3.p1.1)\.
- Böröczky Jr and Wintsche \(2003\)K\. Böröczky Jr and G\. WintscheCovering the sphere by equal spherical balls\.InDiscrete and Computational Geometry: The Goodman\-Pollack Festschrift,pp\. 235–251\.External Links:[Link](https://link.springer.com/content/pdf/10.1007/978-3-642-55566-4.pdf#page=242)Cited by:[§A\.1](https://arxiv.org/html/2608.17088#A1.SS1.SSS0.Px1.p9.1.1),[§A\.2](https://arxiv.org/html/2608.17088#A1.SS2.SSS0.Px1.p5.1.1)\.
- Changet al\.\(2024\)T\. A\. Chang, C\. Arnett, Z\. Tu, and B\. K\. BergenWhen Is Multilinguality a Curse? Language Modeling for 250 High\- and Low\-Resource Languages\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 4074–4096\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.236),[Link](https://aclanthology.org/2024.emnlp-main.236/)Cited by:[§1](https://arxiv.org/html/2608.17088#S1.p2.1),[§2](https://arxiv.org/html/2608.17088#S2.SS0.SSS0.Px1.p1.1)\.
- Changet al\.\(2022\)T\. A\. Chang, Z\. Tu, and B\. K\. BergenThe Geometry of Multilingual Language Model Representations\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,Abu Dhabi, United Arab Emirates,pp\. 119–136\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.9),[Link](https://aclanthology.org/2022.emnlp-main.9)Cited by:[§B\.4](https://arxiv.org/html/2608.17088#A2.SS4.p1.1),[§2](https://arxiv.org/html/2608.17088#S2.SS0.SSS0.Px2.p1.1)\.
- Conneauet al\.\(2020\)A\. Conneau, K\. Khandelwal, N\. Goyal, V\. Chaudhary, G\. Wenzek, F\. Guzmán, E\. Grave, M\. Ott, L\. Zettlemoyer, and V\. StoyanovUnsupervised Cross\-lingual Representation Learning at Scale\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 8440–8451\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.747),[Link](https://aclanthology.org/2020.acl-main.747/)Cited by:[§1](https://arxiv.org/html/2608.17088#S1.p1.1),[§5](https://arxiv.org/html/2608.17088#S5.SS0.SSS0.Px1.p1.1)\.
- Conneauet al\.\(2018\)A\. Conneau, R\. Rinott, G\. Lample, A\. Williams, S\. R\. Bowman, H\. Schwenk, and V\. StoyanovXNLI: evaluating cross\-lingual sentence representations\.InProceedings of the 2018 conference on empirical methods in natural language processing,pp\. 2475–2485\.External Links:[Link](https://aclanthology.org/D18-1269/)Cited by:[§2](https://arxiv.org/html/2608.17088#S2.SS0.SSS0.Px2.p1.1)\.
- Deutschet al\.\(2025\)D\. Deutsch, E\. Briakou, I\. Caswell, M\. Finkelstein, R\. Galor, J\. Juraska, G\. Kovacs, A\. Lui, R\. Rei, J\. Riesa, S\. Rijhwani, P\. Riley, E\. Salesky, F\. Trabelsi, S\. Winkler, B\. Zhang, and M\. FreitagWMT24\+\+: expanding the language coverage of wmt24 to 55 languages and dialects\.External Links:2502\.12404,[Link](https://arxiv.org/abs/2502.12404)Cited by:[§B\.2](https://arxiv.org/html/2608.17088#A2.SS2.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2608.17088#S5.SS0.SSS0.Px4.p1.1)\.
- Devlinet al\.\(2019\)J\. Devlin, M\. Chang, K\. Lee, and K\. ToutanovaBert: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 \(long and short papers\),pp\. 4171–4186\.External Links:[Link](https://aclanthology.org/N19-1423/)Cited by:[§5](https://arxiv.org/html/2608.17088#S5.SS0.SSS0.Px4.p1.1)\.
- Enevoldsenet al\.\(2025\)K\. Enevoldsen, I\. Chung, I\. Kerboua, M\. Kardos, A\. Mathur, D\. Stap, J\. Gala, W\. Siblini, D\. Krzemiński, G\. I\. Winata,et al\.Mmteb: massive multilingual text embedding benchmark\.arXiv preprint arXiv:2502\.13595\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/hash/fc0e3f908a2116ba529ad0a1530a3675-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2608.17088#S2.SS0.SSS0.Px2.p1.1)\.
- Ethayarajh \(2019\)K\. EthayarajhHow contextual are contextualized word representations? comparing the geometry of BERT, ELMo, and GPT\-2 embeddings\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),K\. Inui, J\. Jiang, V\. Ng, and X\. Wan \(Eds\.\),Hong Kong, China,pp\. 55–65\.External Links:[Link](https://aclanthology.org/D19-1006/),[Document](https://dx.doi.org/10.18653/v1/D19-1006)Cited by:[footnote 6](https://arxiv.org/html/2608.17088#footnote6)\.
- Foroutanet al\.\(2025\)N\. Foroutan, P\. Teiletche, A\. K\. Tarun, and A\. BosselutRevisiting Multilingual Data Mixtures in Language Model Pretraining\.arXiv\.External Links:2510\.25947,[Document](https://dx.doi.org/10.48550/arXiv.2510.25947),[Link](http://arxiv.org/abs/2510.25947)Cited by:[§2](https://arxiv.org/html/2608.17088#S2.SS0.SSS0.Px1.p2.1)\.
- Goyalet al\.\(2022\)N\. Goyal, C\. Gao, V\. Chaudhary, P\. Chen, G\. Wenzek, D\. Ju, S\. Krishnan, M\. Ranzato, F\. Guzmán, and A\. FanThe flores\-101 evaluation benchmark for low\-resource and multilingual machine translation\.Transactions of the Association for Computational Linguistics10,pp\. 522–538\.External Links:[Link](https://direct.mit.edu/tacl/article-abstract/doi/10.1162/tacl_a_00474/110993)Cited by:[§B\.2](https://arxiv.org/html/2608.17088#A2.SS2.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2608.17088#S5.SS0.SSS0.Px4.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur,et al\.The llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[§B\.2](https://arxiv.org/html/2608.17088#A2.SS2.SSS0.Px1.p1.1)\.
- Gurgurovet al\.\(2024\)D\. Gurgurov, T\. Bäumel, and T\. AnikinaMultilingual Large Language Models and Curse of Multilinguality\.External Links:2406\.10602,[Document](https://dx.doi.org/10.48550/arXiv.2406.10602),[Link](http://arxiv.org/abs/2406.10602)Cited by:[§1](https://arxiv.org/html/2608.17088#S1.p1.1),[§2](https://arxiv.org/html/2608.17088#S2.SS0.SSS0.Px1.p1.1)\.
- Hämmerlet al\.\(2024\)K\. Hämmerl, J\. Libovický, and A\. FraserUnderstanding Cross\-Lingual Alignment—A Survey\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 10922–10943\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.649),[Link](https://aclanthology.org/2024.findings-acl.649/)Cited by:[§B\.1](https://arxiv.org/html/2608.17088#A2.SS1.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2608.17088#S1.p2.1),[§2](https://arxiv.org/html/2608.17088#S2.SS0.SSS0.Px2.p1.1),[§3](https://arxiv.org/html/2608.17088#S3.SS0.SSS0.Px3.p1.1)\.
- Johnsonet al\.\(1984\)W\. B\. Johnson J\. Lindenstrausset al\.Extensions of lipschitz mappings into a hilbert space\.Contemporary mathematics26\(189\-206\),pp\. 1\.External Links:[Link](https://d1wqtxts1xzle7.cloudfront.net/99288805/JL-Johnson-libre.pdf?1677690739=&response-content-disposition=inline%3B+filename%3DExtensions_of_Lipschitz_mappings_into_a.pdf)Cited by:[§A\.4](https://arxiv.org/html/2608.17088#A1.SS4.SSS0.Px2.p1.1)\.
- Kreutzeret al\.\(2021\)J\. Kreutzer, D\. Vilar, and A\. SokolovBandits Don’t Follow Rules: Balancing Multi\-Facet Machine Translation with Multi\-Armed Bandits\.InFindings of the Association for Computational Linguistics: EMNLP 2021,M\. Moens, X\. Huang, L\. Specia, and S\. W\. Yih \(Eds\.\),Punta Cana, Dominican Republic,pp\. 3190–3204\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.findings-emnlp.274),[Link](https://aclanthology.org/2021.findings-emnlp.274/)Cited by:[§2](https://arxiv.org/html/2608.17088#S2.SS0.SSS0.Px1.p2.1)\.
- Kuduguntaet al\.\(2023\)S\. Kudugunta, I\. Caswell, B\. Zhang, X\. Garcia, D\. Xin, A\. Kusupati, R\. Stella, A\. Bapna, and O\. FiratMadlad\-400: a multilingual and document\-level large audited dataset\.Advances in Neural Information Processing Systems36,pp\. 67284–67296\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/d49042a5d49818711c401d34172f9900-Abstract-Datasets_and_Benchmarks.html)Cited by:[§5](https://arxiv.org/html/2608.17088#S5.SS0.SSS0.Px4.p1.1)\.
- Longpreet al\.\(2026\)S\. Longpre, S\. Kudugunta, N\. Muennighoff, I\. Hsu, I\. Caswell, A\. Pentland, S\. Arik, C\. Lee, and S\. EbrahimiATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilinguality\.arXiv\.External Links:2510\.22037,[Document](https://dx.doi.org/10.48550/arXiv.2510.22037),[Link](http://arxiv.org/abs/2510.22037)Cited by:[§2](https://arxiv.org/html/2608.17088#S2.SS0.SSS0.Px1.p1.1)\.
- Mousiet al\.\(2024\)B\. Mousi, N\. Durrani, F\. Dalvi, M\. Hawasly, and A\. AbdelaliExploring Alignment in Shared Cross\-lingual Spaces\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 6326–6348\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.344),[Link](https://aclanthology.org/2024.acl-long.344/)Cited by:[§2](https://arxiv.org/html/2608.17088#S2.SS0.SSS0.Px2.p1.1)\.
- Naszódi \(2016\)M\. NaszódiOn some covering problems in geometry\.Proceedings of the American Mathematical Society144\(8\),pp\. 3555–3562\.External Links:[Link](https://pubs.ams.org/journals/proc/2016-144-08/S0002-9939-2016-12992-4)Cited by:[§A\.1](https://arxiv.org/html/2608.17088#A1.SS1.SSS0.Px1.p9.1.1),[§A\.2](https://arxiv.org/html/2608.17088#A1.SS2.SSS0.Px1.p5.1.1)\.
- Parket al\.\(2023\)K\. Park, Y\. J\. Choe, and V\. VeitchThe linear representation hypothesis and the geometry of large language models\.arXiv preprint arXiv:2311\.03658\.External Links:[Link](https://arxiv.org/abs/2311.03658)Cited by:[§B\.4](https://arxiv.org/html/2608.17088#A2.SS4.p1.1)\.
- Pfeifferet al\.\(2022\)J\. Pfeiffer, N\. Goyal, X\. V\. Lin, X\. Li, J\. Cross, S\. Riedel, and M\. ArtetxeLifting the Curse of Multilinguality by Pre\-training Modular Transformers\.InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,M\. Carpuat, M\. de Marneffe, and I\. V\. Meza Ruiz \(Eds\.\),Seattle, United States,pp\. 3479–3495\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.naacl-main.255),[Link](https://aclanthology.org/2022.naacl-main.255/)Cited by:[§1](https://arxiv.org/html/2608.17088#S1.p1.1),[§2](https://arxiv.org/html/2608.17088#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2608.17088#S2.SS0.SSS0.Px1.p2.1)\.
- Royet al\.\(2020\)U\. Roy, N\. Constant, R\. Al\-Rfou, A\. Barua, A\. Phillips, and Y\. YangLAReQA: Language\-Agnostic Answer Retrieval from a Multilingual Pool\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 5919–5930\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.477),[Link](https://aclanthology.org/2020.emnlp-main.477/)Cited by:[§B\.1](https://arxiv.org/html/2608.17088#A2.SS1.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2608.17088#S1.p2.1),[§2](https://arxiv.org/html/2608.17088#S2.SS0.SSS0.Px2.p1.1),[§3](https://arxiv.org/html/2608.17088#S3.SS0.SSS0.Px3.p1.1)\.
- Sasakiet al\.\(2018\)S\. Sasaki, S\. Sun, S\. Schamoni, K\. Duh, and K\. InuiCross\-lingual learning\-to\-rank with shared representations\.InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 \(Short Papers\),M\. Walker, H\. Ji, and A\. Stent \(Eds\.\),New Orleans, Louisiana,pp\. 458–463\.External Links:[Link](https://aclanthology.org/N18-2073/),[Document](https://dx.doi.org/10.18653/v1/N18-2073)Cited by:[§1](https://arxiv.org/html/2608.17088#S1.p2.1)\.
- Shahet al\.\(2023\)C\. Shah, Y\. Chandak, and M\. SuriThe Geometry of Multilingual Language Models: An Equality Lens\.arXiv\.External Links:2305\.07839,[Document](https://dx.doi.org/10.48550/arXiv.2305.07839),[Link](http://arxiv.org/abs/2305.07839)Cited by:[§2](https://arxiv.org/html/2608.17088#S2.SS0.SSS0.Px2.p1.1)\.
- Shahamet al\.\(2023\)U\. Shaham, M\. Elbayad, V\. Goswami, O\. Levy, and S\. BhosaleCauses and Cures for Interference in Multilingual Translation\.arXiv\.External Links:2212\.07530,[Document](https://dx.doi.org/10.48550/arXiv.2212.07530),[Link](http://arxiv.org/abs/2212.07530)Cited by:[§1](https://arxiv.org/html/2608.17088#S1.p1.1),[§2](https://arxiv.org/html/2608.17088#S2.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2608.17088#S5.SS0.SSS0.Px1.p1.1)\.
- Teamet al\.\(2024\)N\. Team, M\. R\. Costa\-jussà, J\. Cross, O\. Çelebi, M\. Elbayad, K\. Heafield, K\. Heffernan, E\. Kalbassi, J\. Lam, D\. Licht, J\. Maillard, A\. Sun, S\. Wang, G\. Wenzek, A\. Youngblood, B\. Akula, L\. Barrault, G\. M\. Gonzalez, P\. Hansanti, J\. Hoffman, S\. Jarrett, K\. R\. Sadagopan, D\. Rowe, S\. Spruit, C\. Tran, P\. Andrews, N\. F\. Ayan, S\. Bhosale, S\. Edunov, A\. Fan, C\. Gao, V\. Goswami, F\. Guzmán, P\. Koehn, A\. Mourachko, C\. Ropers, S\. Saleem, H\. Schwenk, and J\. WangScaling neural machine translation to 200 languages\.Nature630\(8018\),pp\. 841–846\.External Links:[Link](https://doi.org/10.1038/s41586-024-07335-x)Cited by:[§B\.2](https://arxiv.org/html/2608.17088#A2.SS2.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2608.17088#S5.SS0.SSS0.Px4.p1.1)\.
- Teamet al\.\(2026\)T\. O\. M\. Team, B\. Alastruey, N\. Bafna, A\. Caciolai, K\. Heffernan, A\. Kozhevnikov, C\. Ropers, E\. Sánchez, C\. Saint\-James, I\. Tsiamas, C\. Cheng, J\. Chuang, P\. Duquenne, M\. Duppenthaler, N\. Ekberg, C\. Gao, P\. L\. H\. Cabot, J\. M\. Janeiro, J\. Maillard, G\. M\. Gonzalez, H\. Schwenk, E\. Toledo, A\. Turkatenko, A\. Ventayol\-Boada, R\. Moritz, A\. Mourachko, S\. Parimi, M\. Williamson, S\. Yates, D\. Dale, and M\. R\. Costa\-jussàOmnilingual MT: machine translation for 1,600 languages\.External Links:[Link](https://arxiv.org/abs/2603.16309)Cited by:[§B\.2](https://arxiv.org/html/2608.17088#A2.SS2.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2608.17088#S5.SS0.SSS0.Px4.p1.1)\.
- Wanget al\.\(2024\)L\. Wang, N\. Yang, X\. Huang, B\. Jiao, L\. Yang, D\. Jiang, R\. Majumder, and F\. WeiText embeddings by weakly\-supervised contrastive pre\-training\.External Links:2212\.03533,[Link](https://arxiv.org/abs/2212.03533)Cited by:[§B\.2](https://arxiv.org/html/2608.17088#A2.SS2.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2608.17088#S5.SS0.SSS0.Px4.p1.1)\.
- Wanget al\.\(2020a\)X\. Wang, Y\. Tsvetkov, and G\. NeubigBalancing Training for Multilingual Neural Machine Translation\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 8526–8537\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.754),[Link](https://aclanthology.org/2020.acl-main.754/)Cited by:[§1](https://arxiv.org/html/2608.17088#S1.p1.1),[§2](https://arxiv.org/html/2608.17088#S2.SS0.SSS0.Px1.p2.1)\.
- Wanget al\.\(2020b\)Z\. Wang, Y\. Tsvetkov, O\. Firat, and Y\. CaoGradient Vaccine: Investigating and Improving Multi\-task Optimization in Massively Multilingual Models\.arXiv\.External Links:2010\.05874,[Document](https://dx.doi.org/10.48550/arXiv.2010.05874),[Link](http://arxiv.org/abs/2010.05874)Cited by:[§2](https://arxiv.org/html/2608.17088#S2.SS0.SSS0.Px1.p2.1)\.
- Welleret al\.\(2026\)O\. Weller, M\. Boratko, I\. Naim, and J\. LeeOn the theoretical limitations of embedding\-based retrieval\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=k9CzIvzfaA)Cited by:[§A\.3](https://arxiv.org/html/2608.17088#A1.SS3.p3.1),[§A\.4](https://arxiv.org/html/2608.17088#A1.SS4.SSS0.Px2.p1.1),[§3](https://arxiv.org/html/2608.17088#S3.SS0.SSS0.Px4.p1.1),[§4\.2](https://arxiv.org/html/2608.17088#S4.SS2.p5.1)\.
- Wen\-Yi and Mimno \(2023\)A\. W\. Wen\-Yi and D\. MimnoHyperpolyglot LLMs: cross\-lingual interpretability in token embeddings\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 1124–1131\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.71/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.71)Cited by:[§2](https://arxiv.org/html/2608.17088#S2.SS0.SSS0.Px2.p1.1)\.
- Xieet al\.\(2022a\)Z\. Xie, H\. Zhao, T\. Yu, and S\. LiDiscovering Low\-rank Subspaces for Language\-agnostic Multilingual Representations\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 5617–5633\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.379),[Link](https://aclanthology.org/2022.emnlp-main.379/)Cited by:[§B\.4](https://arxiv.org/html/2608.17088#A2.SS4.p1.1)\.
- Xieet al\.\(2022b\)Z\. Xie, H\. Zhao, T\. Yu, and S\. LiDiscovering low\-rank subspaces for language\-agnostic multilingual representations\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,pp\. 5617–5633\.External Links:[Link](https://aclanthology.org/2022.emnlp-main.379/)Cited by:[§2](https://arxiv.org/html/2608.17088#S2.SS0.SSS0.Px2.p1.1)\.
- Xuet al\.\(2023\)N\. Xu, Q\. Zhang, J\. Ye, M\. Zhang, and X\. HuangAre structural concepts universal in transformer language models? towards interpretable cross\-lingual generalization\.InFindings of the Association for Computational Linguistics: EMNLP 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 13951–13976\.External Links:[Link](https://aclanthology.org/2023.findings-emnlp.931/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.931)Cited by:[§2](https://arxiv.org/html/2608.17088#S2.SS0.SSS0.Px2.p1.1)\.

## Appendix AProof details

### A\.1Spherical cap bound for minimum sufficient dimensionality

In this section, we describe and prove the result necessary for the proof in[Section4\.3](https://arxiv.org/html/2608.17088#S4.SS3)\. Specifically, we will show that the number of points that can be placed on aDLD\_\{L\}\-dimensional sphere with radiusrrgiven a minimum and maximum cosine similarity following[Equation1](https://arxiv.org/html/2608.17088#S4.E1), and given a suitable choice ofρ\\rho, grows exponentially inDLD\_\{L\}\.

Recall thatγ<1\\gamma<1is the maximum allowed similarity in the non\-degenerate spaceZZ, given to us\. We can assume thatγ\>0\\gamma\>0; if observed maximum cosine similarity is negative, the non\-degeneracy condition would still hold for a largerγ\\gamma\.

We will user=1r=1in our construction for simplicity and without loss of generality\.[Equation1](https://arxiv.org/html/2608.17088#S4.E1)simplifies to

γ<cos⁡\(uℓ,um\)<ρ<1\\gamma<\\cos\(u\_\{\\ell\},u\_\{m\}\)<\\rho<1We will chooseρ\\rhosuch thatγ<ρ<1\\gamma<\\rho<1andarccos⁡\(ρ\)<arccos⁡\(γ\)/2\\arccos\(\\rho\)<\\arccos\(\\gamma\)/2\.

Now, we will show the following\.

###### Lemma 5\.

Fix constants0<γ<ρ<10<\\gamma<\\rho<1, such thatarccos⁡\(ρ\)<arccos⁡\(γ\)/2\\arccos\(\\rho\)<\\arccos\(\\gamma\)/2\. Thendddimensions are sufficient to packexp⁡\(Ω⁡\(d\)\)\\exp\(\\Omega\(d\)\)points on the unit sphereSd−1S^\{d\-1\}with maximum cosine similarityρ\\rhoand minimum cosine similarityγ\\gamma\. That is, there exists a setℳ⊂Sd−1\\mathcal\{M\}\\subset S^\{d\-1\}such that for all distinctu,u′∈ℳu,u^\{\\prime\}\\in\\mathcal\{M\},γ<⟨u,u′⟩<ρ\\gamma<\\langle u,u^\{\\prime\}\\rangle<\\rho, and\|ℳ\|=exp⁡\(Ω⁡\(d\)\)\|\\mathcal\{M\}\|=\\exp\(\\Omega\(d\)\)\.

#### Proof sketch

We will place points on a cap on a multi\-dimensional sphere bounded with some angular radius\. Since all points are within this cap, the maximum cosine similarity is bounded, letting us satisfy the maximum similarity constraint\. Now we will fit points on this cap such that any two points have a minimum angular radius from each other\. Imagine each such point making a small cap around itself with angular radius as the minimum required angular separation\. Now, consider the maximal set of fitting points in the big cap\. The union of the area of the small caps must cover the area of the big cap, since, if there was an uncovered point, that would mean that it was not within the minimum required separation of any other point, and we would be able to add in to our maximal set\. Since we can express the union of the area of the small caps in terms of the number of points, we therefore get a lower bound on the maximum number of fitting points in terms of the areas of the two caps\. By standard spherical cap results, the area of a spherical cap grows exponentially in the dimension of the sphere\. Thus, as long as the angular radius of the big cap is large enough relative to that of the small cap, guaranteed by our choice ofρ\\rho, the maximum number of points we can fit that satisfy the above constraints also grow exponentially in the dimension of the sphere, giving us the required result that the sufficient dimension to fitLLpoints is logarithmic inLL\.

###### Proof\.

Letθmin:=arccos⁡\(ρ\)\\theta\_\{\\min\}:=\\arccos\(\\rho\)andθmax:=arccos⁡\(γ\)\\theta\_\{\\max\}:=\\arccos\(\\gamma\)\. Sincearccos⁡\(ρ\)<arccos⁡\(γ\)/2\\arccos\(\\rho\)<\\arccos\(\\gamma\)/2, andγ\>0\\gamma\>0, we haveθmin<θmax/2<π/4\\theta\_\{\\min\}<\\theta\_\{\\max\}/2<\\pi/4\.

Chooseβ\\betasuch thatθmin<β<θmax2<π/4\\theta\_\{\\min\}<\\beta<\\frac\{\\theta\_\{\\max\}\}\{2\}<\\pi/4\. Fix a directionv∈Sd−1v\\in S^\{d\-1\}and define the spherical cap aroundvv:

𝒞β​\(v\):=\{u∈Sd−1:∠⁡\(u,v\)<β\}\.\\displaystyle\\mathcal\{C\}\_\{\\beta\}\(v\):=\\\{u\\in S^\{d\-1\}:\\angle\(u,v\)<\\beta\\\}\.
This means that any two points in this capuuandu′u^\{\\prime\}can have angular distance at most2​β<θmax2\\beta<\\theta\_\{\\max\}\(since each has angular distance at mostβ\\betafromvv\)\.

Now consider a set of points𝒬\\mathcal\{Q\}inside𝒞β​\(v\)\\mathcal\{C\}\_\{\\beta\}\(v\)such that every pair in this set has angular distance greater thanθmin\\theta\_\{\\min\}\. Equivalently,⟨u,u′⟩<ρ\\langle u,u^\{\\prime\}\\rangle<\\rho\. Thus, for any distinctu,u′∈𝒬u,u^\{\\prime\}\\in\\mathcal\{Q\}, we will have

θmin<∠⁡\(u,u′\)<2​β<θmax\\displaystyle\\theta\_\{\\min\}<\\angle\(u,u^\{\\prime\}\)<2\\beta<\\theta\_\{\\max\}⟹cos⁡\(θmax\)<⟨u,u′⟩<cos⁡\(θmin\)\\displaystyle\\implies\\cos\(\\theta\_\{\\max\}\)<\\langle u,u^\{\\prime\}\\rangle<\\cos\(\\theta\_\{\\min\}\)⟹γ<⟨u,u′⟩<ρ\\displaystyle\\implies\\gamma<\\langle u,u^\{\\prime\}\\rangle<\\rho
Thus, the set𝒬\\mathcal\{Q\}satisfies our required angular constraints\. In order to prove our lemma, it is enough to show that the maximal size of𝒬\\mathcal\{Q\}grows exponentially withdd; in other words, thatdddimensions are enough to pack some set of pointsℳ\\mathcal\{M\}with size growing exponentially indd\.

Letℳ=\{u1,…,uM\}⊂𝒞β​\(v\)\\mathcal\{M\}=\\\{u\_\{1\},\\ldots,u\_\{M\}\\\}\\subset\\mathcal\{C\}\_\{\\beta\}\(v\)with sizeM=\|ℳ\|M=\|\\mathcal\{M\}\|be a maximal set satisfying∠⁡\(ui,uj\)\>θmin​for all​i≠j\\angle\(u\_\{i\},u\_\{j\}\)\>\\theta\_\{\\min\}~\\text\{for all \}i\\neq j\.

Now, any point in𝒞β​\(v\)\\mathcal\{C\}\_\{\\beta\}\(v\)must be within angular distanceθmin\\theta\_\{\\min\}of some point inℳ\\mathcal\{M\}; otherwise this point could be added toℳ\\mathcal\{M\}\. Therefore,Cβ​\(v\)C\_\{\\beta\}\(v\)is a subset of the union of spherical caps created by defining a minimum angle radius around each point inℳ\\mathcal\{M\}\. In other words:

𝒞β​\(v\)⊆⋃i=1M𝒞θmin​\(ui\)\.\\mathcal\{C\}\_\{\\beta\}\(v\)\\subseteq\\bigcup\_\{i=1\}^\{M\}\\mathcal\{C\}\_\{\\theta\_\{\\min\}\}\(u\_\{i\}\)\.Now we will consider the surface area on the sphere\. Taking normalized spherical measureσ\\sigma, and sinceσ\\sigmaonly depends on angular radius,

σ​\(𝒞β​\(v\)\)\\displaystyle\\sigma\(\\mathcal\{C\}\_\{\\beta\}\(v\)\)≤∑i=1Mσ⁡\(𝒞θmin​\(ui\)\)=M​σ​\(𝒞θmin\)\\displaystyle\\leq\\sum\_\{i=1\}^\{M\}\\sigma\(\\mathcal\{C\}\_\{\\theta\_\{\\min\}\}\(u\_\{i\}\)\)=M\\sigma\(\\mathcal\{C\}\_\{\\theta\_\{\\min\}\}\)⟹M≥σ⁡\(𝒞β\)σ⁡\(𝒞θmin\)\.\\displaystyle\\implies M\\geq\\frac\{\\sigma\(\\mathcal\{C\}\_\{\\beta\}\)\}\{\\sigma\(\\mathcal\{C\}\_\{\\theta\_\{\\min\}\}\)\}\.
For fixed angular radii, spherical cap measure scales exponentially inddas per standard cap measure results\([25](https://arxiv.org/html/2608.17088#bib.bib31)\)\. Specifically, for0<α<π/20<\\alpha<\\pi/2, the normalized measure of a spherical cap of angular radiusα\\alphainSd−1S^\{d\-1\}given by[25](https://arxiv.org/html/2608.17088#bib.bib31), attributed to[6](https://arxiv.org/html/2608.17088#bib.bib34), satisfies

σ⁡\(𝒞α\)=Θα​\(sind−1⁡\(α\)d\)\\sigma\(\\mathcal\{C\}\_\{\\alpha\}\)=\\Theta\_\{\\alpha\}\\\!\\left\(\\frac\{\\sin^\{d\-1\}\(\\alpha\)\}\{\\sqrt\{d\}\}\\right\)
Applying this withα=β<π/2\\alpha=\\beta<\\pi/2andα=θmin<π/2\\alpha=\\theta\_\{\\min\}<\\pi/2gives

M≥σ⁡\(𝒞β\)σ⁡\(𝒞θmin\)=Ω⁡\(\(sin⁡βsin⁡θmin\)d−1\),M\\geq\\frac\{\\sigma\(\\mathcal\{C\}\_\{\\beta\}\)\}\{\\sigma\(\\mathcal\{C\}\_\{\\theta\_\{\\min\}\}\)\}=\\Omega\\\!\\left\(\\left\(\\frac\{\\sin\\beta\}\{\\sin\\theta\_\{\\min\}\}\\right\)^\{d\-1\}\\right\),
Sinceβ\>θmin\\beta\>\\theta\_\{\\min\}, we havesin⁡β\>sin⁡θmin\\sin\\beta\>\\sin\\theta\_\{\\min\}\. Sinceβ\\betaandθmin\\theta\_\{\\min\}are fixed, there exists a constantCM\>0C\_\{M\}\>0, independent ofdd, such that

M≥exp⁡\(CM​d\)⟹M=exp⁡\(Ω⁡\(d\)\)\\displaystyle M\\geq\\exp\(C\_\{M\}d\)\\implies M=\\exp\(\\Omega\(d\)\)That is, there exists some setℳ\\mathcal\{M\}containingexp⁡\(Ω⁡\(d\)\)\\exp\(\\Omega\(d\)\)points whose pairwise cosine similarities lie betweenγ\\gammaandρ\\rho, on add\-dimensional sphere\.

In other words, the minimum sufficient dimension to fitLLpoints whose pairwise cosine similarities lie betweenγ\\gammaandρ\\rhoonSd−1⊂ℝdS^\{d\-1\}\\subset\\mathbb\{R\}^\{d\}isd=O⁡\(log⁡L\)d=O\(\\log L\)\. ∎

### A\.2Spherical cap bound for necessary dimensionality

In this section, we describe and prove the result necessary for the proof in[Section4\.4](https://arxiv.org/html/2608.17088#S4.SS4)\. Specifically, we will show that the dimensionality required to placeLLfixed\-norm points on aDXLD\_\{X^\{L\}\}\-dimensional sphere with some minimum angular separation grows at least logarithmically inLL\.

Givenγ<1\\gamma<1as the maximum allowed cosine similarity, we will use minimum angular separationθ=arccos⁡\(γ\)\\theta=\\arccos\(\\gamma\), with0<θ<π0<\\theta<\\pi\. Now, we will show the following:

###### Lemma 6\.

Fix a constant0<θ<π0<\\theta<\\pi\. Supposeℳ⊂Sd−1\\mathcal\{M\}\\subset S^\{d\-1\}is a set ofM=\|ℳ\|M=\|\\mathcal\{M\}\|points on a unit sphere with pairwise angular separation greater thanθ\\theta\. That is, for all distinctu,u′∈ℳu,u^\{\\prime\}\\in\\mathcal\{M\},∠⁡\(u,u′\)\>θ\\angle\(u,u^\{\\prime\}\)\>\\theta\. Then there exists a constantAθ\>0A\_\{\\theta\}\>0, independent ofMManddd, such thatM≤exp⁡\(Aθ​d\)M\\leq\\exp\(A\_\{\\theta\}d\)\. Equivalently, any such packing ofMMpoints requiresd=Ω⁡\(log⁡M\)d=\\Omega\(\\log M\)\.

#### Proof sketch

We will placeMMpoints on a multidimensional sphere\. Think of each point inℳ\\mathcal\{M\}forming a spherical cap on the sphere with angular radius as half of the minimum required angular separation\. Since each pair of points must have the required angular separation, these spherical caps must be disjoint: if a point on the sphere belonged to caps associated with two points inℳ\\mathcal\{M\}, those two points would not satisfy the criterion by triangle inequality\. Thus, the surface area of the whole sphere must be larger than the union of the surface area of the spherical caps\. Since we haveMMspherical caps and can express the union of the surface area of the spherical caps in terms ofMM, we get an upper bound onMMin terms of an inverse spherical cap measure, which scales exponentially in the dimension of the sphere\. Thus, we get that the minimum dimension that can fitMMpoints is at least logarithmic inMM\.

###### Proof\.

Letℳ=\{u1,…,uM\}⊂Sd−1\\mathcal\{M\}=\\\{u\_\{1\},\\ldots,u\_\{M\}\\\}\\subset S^\{d\-1\}satisfy∠⁡\(ui,uj\)\>θfor all​i≠j\\angle\(u\_\{i\},u\_\{j\}\)\>\\theta\\qquad\\text\{for all \}i\\neq j\.

Around each pointuiu\_\{i\}, place a spherical cap of angular radiusθ/2\\theta/2:

𝒞θ/2​\(ui\):=\{u∈Sd−1:∠⁡\(u,ui\)≤θ/2\}\.\\mathcal\{C\}\_\{\\theta/2\}\(u\_\{i\}\):=\\\{u\\in S^\{d\-1\}:\\angle\(u,u\_\{i\}\)\\leq\\theta/2\\\}\.These caps are disjoint, since if some pointwwbelonged to both𝒞θ/2​\(ui\)\\mathcal\{C\}\_\{\\theta/2\}\(u\_\{i\}\)and𝒞θ/2​\(uj\)\\mathcal\{C\}\_\{\\theta/2\}\(u\_\{j\}\), then by the triangle inequality,∠⁡\(ui,uj\)≤∠⁡\(ui,w\)\+∠⁡\(w,uj\)≤θ\\angle\(u\_\{i\},u\_\{j\}\)\\leq\\angle\(u\_\{i\},w\)\+\\angle\(w,u\_\{j\}\)\\leq\\theta, contradicting the separation constraint\.

Taking normalized spherical measureσ\\sigma, we have

∑i=1Mσ⁡\(𝒞θ/2​\(ui\)\)≤σ⁡\(Sd−1\)=1\\sum\_\{i=1\}^\{M\}\\sigma\(\\mathcal\{C\}\_\{\\theta/2\}\(u\_\{i\}\)\)\\leq\\sigma\(S^\{d\-1\}\)=1Sinceσ\\sigmaonly depends on angular radius,

M​σ​\(𝒞θ/2\)≤1⟹M≤1σ⁡\(𝒞θ/2\)\.M\\sigma\(\\mathcal\{C\}\_\{\\theta/2\}\)\\leq 1\\implies M\\leq\\frac\{1\}\{\\sigma\(\\mathcal\{C\}\_\{\\theta/2\}\)\}\.
For fixed angular radius, spherical cap measures are at least exponentially small indd\. More concretely, for0<α<π/20<\\alpha<\\pi/2, the normalized measure of a spherical cap of angular radiusα\\alphainSd−1S^\{d\-1\}, as given by[25](https://arxiv.org/html/2608.17088#bib.bib31), attributed to[6](https://arxiv.org/html/2608.17088#bib.bib34)satisfies

σ⁡\(𝒞α\)≥12​π​d​sind−1⁡\(α\)\.\\sigma\(\\mathcal\{C\}\_\{\\alpha\}\)\\geq\\frac\{1\}\{\\sqrt\{2\\pi d\}\}\\sin^\{d\-1\}\(\\alpha\)\.Applying this withα=θ/2<π/2\\alpha=\\theta/2<\\pi/2gives

M\\displaystyle M≤1σ⁡\(𝒞θ/2\)≤2​π​d​\(1sin⁡\(θ/2\)\)d−1\\displaystyle\\leq\\frac\{1\}\{\\sigma\(\\mathcal\{C\}\_\{\\theta/2\}\)\}\\leq\\sqrt\{2\\pi d\}\\left\(\\frac\{1\}\{\\sin\(\\theta/2\)\}\\right\)^\{d\-1\}⟹log⁡M≤12​log⁡\(2​π​d\)\+\(d−1\)​log⁡\(1sin⁡\(θ/2\)\)\\displaystyle\\implies\\log M\\leq\\frac\{1\}\{2\}\\log\(2\\pi d\)\+\(d\-1\)\\log\\\!\\left\(\\frac\{1\}\{\\sin\(\\theta/2\)\}\\right\)
Sinceθ\\thetais fixed,sin⁡\(θ/2\)\\sin\(\\theta/2\)is a fixed constant in\(0,1\)\(0,1\)\. Sincelog⁡\(2​π​d\)=o⁡\(d\)\\log\(2\\pi d\)=o\(d\), there exists a constantAθ\>0A\_\{\\theta\}\>0depending only onθ\\theta, such that

log⁡M≤Aθ​d⟹d≥1Aθ​log⁡M⟹d=Ω⁡\(log⁡M\)\\displaystyle\\log M\\leq A\_\{\\theta\}d\\implies d\\geq\\frac\{1\}\{A\_\{\\theta\}\}\\log M\\implies d=\\Omega\(\\log M\)
Therefore any set ofLLpoints onSd−1⊂ℝdS^\{d\-1\}\\subset\\mathbb\{R\}^\{d\}with fixed pairwise cosine separationγ\\gammarequiresd=Ω⁡\(log⁡L\)d=\\Omega\(\\log L\)\. ∎

### A\.3Fitting 7000 languages in 30 extra dimensions

Our constructive proof in[Section4](https://arxiv.org/html/2608.17088#S4)and[SectionA\.1](https://arxiv.org/html/2608.17088#A1.SS1)constructs a language offset spaceUUwith dimensionalityDLD\_\{L\}, and shows that thatDL=O⁡\(log⁡L\)D\_\{L\}=O\(\\log L\)dimensions are sufficient to fitLLpoints on aDLD\_\{L\}\-dimensional unit sphere given minimum cosine similarityγ\\gammaand maximum cosine similarityρ\\rho\.

However, this bounds hides a constant which may be of concern for practical considerations\. In this section, we prove the following:

###### Lemma 7\.

It is possible to fitL=7000L=7000points on a3030\-dimensional sphere with minimum pairwise cosine similarityγ=0\.9\\gamma=0\.9and maximum cosine similarityρ=0\.99\\rho=0\.99\.

γ\\gammais set based on the choice of0\.10\.1as a standard scale of cosine separation for text embedding models\([37](https://arxiv.org/html/2608.17088#bib.bib38)\)\. Note that our proof requiresarccos⁡\(ρ\)<arccos⁡\(γ\)/2\\arccos\(\\rho\)<\\arccos\(\\gamma\)/2which holds for this choice ofρ\\rho\.

#### Sketch of proof

We will construct each point having 1 shared dimension with an appropriate length, which will guarantee minimum similarity\. We will use the remaining 29 dimensions to guarantee difference, i\.e\. ensure that any two points have atmost the maximum similarity\. In particular, we will prove this statement using an existence argument: if we choose 7000 points at random on a sphere in 29 dimensions, the probability that no pair of points crosses a particular threshold of similarity is non\-zero\. The existence of such a configuration gives us the above result\.

###### Proof\.

Consider the spaceℝ30=span​\(e0\)⊕ℝ29\\mathbb\{R\}^\{30\}=\\text\{span\}\(e\_\{0\}\)\\oplus\\mathbb\{R\}^\{29\}, wheree0e\_\{0\}is a unit vector orthogonal toℝ29\\mathbb\{R\}^\{29\}\. Now, construct

xi=0\.945​e0\+0\.055​yi,yi∈S28⊂ℝ29\.x\_\{i\}=\\sqrt\{0\.945\}\\,e\_\{0\}\+\\sqrt\{0\.055\}\\,y\_\{i\},\\qquad y\_\{i\}\\in S^\{28\}\\subset\\mathbb\{R\}^\{29\}\.Then∥xi∥2=1\\lVert x\_\{i\}\\rVert\_\{2\}=1and since all vectors have unit norm, cosine similarity is equal to the inner product\. Now, we consider the similarity between any two vectorsxix\_\{i\}andxjx\_\{j\}:

⟨xi,xj⟩\\displaystyle\\langle x\_\{i\},x\_\{j\}\\rangle=0\.945\+0\.055​⟨yi,yj⟩\.\\displaystyle=0\.945\+0\.055\\langle y\_\{i\},y\_\{j\}\\rangle\.
Therefore, if\|⟨yi,yj⟩\|≤911\\left\|\\langle y\_\{i\},y\_\{j\}\\rangle\\right\|\\leq\\frac\{9\}\{11\},for all​i≠j\\text\{for all \}i\\neq j, then

⟨xi,xj⟩\\displaystyle\\langle x\_\{i\},x\_\{j\}\\rangle∈\[0\.945−0\.055⋅911,0\.945\+0\.055⋅911\]\\displaystyle\\in\\left\[0\.945\-0\.055\\cdot\\frac\{9\}\{11\},\\,0\.945\+0\.055\\cdot\\frac\{9\}\{11\}\\right\]=\[0\.9,0\.99\]\.\\displaystyle=\[0\.9,0\.99\]\.
It remains to show that suchy1,…,y7000y\_\{1\},\\ldots,y\_\{7000\}exist\. Suppose that

y1,…,y7000​∼i\.i\.d\.​Unif⁡\(S28\)\.y\_\{1\},\\ldots,y\_\{7000\}\\overset\{\\mathrm\{i\.i\.d\.\}\}\{\\sim\}\\operatorname\{Unif\}\(S^\{28\}\)\.
Fix any pairi≠ji\\neq j\. Since the distribution on the sphere is rotationally invariant, we may fix one pointY=e1=\[1,0,0,…\]Y=e\_\{1\}=\[1,0,0,\.\.\.\]and choose only the second pointZ∼Unif⁡\(S28\)Z\\sim\\operatorname\{Unif\}\(S^\{28\}\)\. The distribution of the inner product is unchanged\. Thus, the dot product ofYYandZZis distributed identically to just the first coordinate ofZZ:

⟨Y,Z⟩=dZ1,\\langle Y,Z\\rangle\\stackrel\{\{\\scriptstyle d\}\}\{\{=\}\}Z\_\{1\},Z1Z\_\{1\}is the first coordinate of a uniformly random point onS28S^\{28\}\. The distribution of the coordinate’s value can be expressed in terms of a Beta distribution, as follows,999Briefly, this is because of the following: we generate a random point by choosing2929dimensions independently and normalizing all coordinates to result in unit norm\. The first coordinate can then be expressed asZ12∼AA\+BZ\_\{1\}^\{2\}\\sim\\frac\{A\}\{A\+B\}whereA∼χ12A\\sim\\chi^\{2\}\_\{1\}andB∼χ282B\\sim\\chi^\{2\}\_\{28\}andAAandBBare independent\. This standardly gives usZ12∼Beta⁡\(12,14\)Z\_\{1\}^\{2\}\\sim\\operatorname\{Beta\}\\\!\\left\(\\frac\{1\}\{2\},14\\right\)\.

⟨Y,Z⟩2∼Beta⁡\(12,14\)\.\\langle Y,Z\\rangle^\{2\}\\sim\\operatorname\{Beta\}\\\!\\left\(\\frac\{1\}\{2\},14\\right\)\.
Therefore, for any fixed pair,

p\\displaystyle p:=Pr⁡\(\|⟨Y,Z⟩\|2\>\(911\)2\)\\displaystyle:=\\Pr\\left\(\\left\|\\langle Y,Z\\rangle\\right\|^\{2\}\>\\left\(\\frac\{9\}\{11\}\\right\)^\{2\}\\right\)\(3\)=I40/121​\(14,12\)≈3\.3465×10−8,\\displaystyle=I\_\{40/121\}\\\!\\left\(14,\\frac\{1\}\{2\}\\right\)\\approx 3\.3465\\times 10^\{\-8\},\(4\)whereIx​\(a,b\)I\_\{x\}\(a,b\)denotes the regularized incomplete beta function\.

There are\(70002\)\\binom\{7000\}\{2\}pairs\. By the union bound,

Pr\(∃i<j:\|⟨yi,yj⟩\|\>911\)\\displaystyle\\Pr\\left\(\\exists\\,i<j:\\left\|\\langle y\_\{i\},y\_\{j\}\\rangle\\right\|\>\\frac\{9\}\{11\}\\right\)≤\(70002\)​p\\displaystyle\\leq\\binom\{7000\}\{2\}p≈0\.8198<1\.\\displaystyle\\approx 0\.8198<1\.Hence,

Pr⁡\(\|⟨yi,yj⟩\|≤911​for all​i≠j\)\>0\.\\Pr\\left\(\\left\|\\langle y\_\{i\},y\_\{j\}\\rangle\\right\|\\leq\\frac\{9\}\{11\}\\text\{ for all \}i\\neq j\\right\)\>0\.Therefore, there exists a realizationy1,…,y7000y\_\{1\},\\ldots,y\_\{7000\}satisfying the required bound, and the corresponding vectorsx1,…,x7000∈ℝ30x\_\{1\},\\ldots,x\_\{7000\}\\in\\mathbb\{R\}^\{30\}have pairwise cosine similarities in\[0\.9,0\.99\]\[0\.9,0\.99\]\. That is, it is possible to fit70007000points, or language offsets, satisfying the constraints from our proof such that the resulting space would maintain the multilinguality conditions, in3030additional dimensions\. ∎

### A\.4Fixed versus variable norm assumptions

Our proof of[1](https://arxiv.org/html/2608.17088#Thmthm1)assumes a fixed\-norm concept spaceZZand multilingual spaceXLX^\{L\}\. This proof is extendable to the variable norm case with a bounded norm\. We provide a brief sketch of the extension below\.

#### Extending[1\.1](https://arxiv.org/html/2608.17088#Thmthm1.Thmsubthm1)

We will still use fixed\-normUUin our construction, resulting in variable\-normXLX^\{L\}constructed asxc,ℓ=zc⊕‖zc‖​uℓx\_\{c,\\ell\}=z\_\{c\}\\oplus\\\|z\_\{c\}\\\|u\_\{\\ell\}\. Since‖uℓ‖=r\\\|u\_\{\\ell\}\\\|=r, we have‖xc,ℓ‖=‖zc‖​1\+r2\\\|x\_\{c,\\ell\}\\\|=\\\|z\_\{c\}\\\|\\sqrt\{1\+r^\{2\}\}, and therefore the similarity decomposition in[Equation2](https://arxiv.org/html/2608.17088#S4.E2)still works out as

cos⁡\(xc,ℓ,xd,m\)\\displaystyle\\cos\(x\_\{c,\\ell\},x\_\{d,m\}\)=⟨zc,zd⟩\+‖zc‖⋅‖zd‖​⟨uℓ,um⟩‖zc‖⋅‖zd‖​\(1\+r2\)\\displaystyle=\\frac\{\\langle z\_\{c\},z\_\{d\}\\rangle\+\\\|z\_\{c\}\\\|\\cdot\\\|z\_\{d\}\\\|\\langle u\_\{\\ell\},u\_\{m\}\\rangle\}\{\\\|z\_\{c\}\\\|\\cdot\\\|z\_\{d\}\\\|\(1\+r^\{2\}\)\}=cos⁡\(zc,zd\)\+⟨uℓ,um⟩1\+r2\\displaystyle=\\frac\{\\cos\(z\_\{c\},z\_\{d\}\)\+\\langle u\_\{\\ell\},u\_\{m\}\\rangle\}\{1\+r^\{2\}\}Now, our proofs showing thatXLX^\{L\}upholds the multilinguality conditions and is non\-degenerate follow as before\. The proof of the upper\-bound on the sufficient dimensionality ofUUin[SectionA\.1](https://arxiv.org/html/2608.17088#A1.SS1)also works as\-is, since we used fixed\-normUU\.

#### Extending[1\.2](https://arxiv.org/html/2608.17088#Thmthm1.Thmsubthm2)

IfXLX^\{L\}is variable\-norm, bounded by some maximum norm, then we must now placeLLpoints inside a sphere rather than on a sphere for this proof\. It can be shown that this also requires necessary dimensionality logarithmic inLL\. Analogously to the proof in[SectionA\.2](https://arxiv.org/html/2608.17088#A1.SS2): instead of fitting disjoint caps on a sphere, we will fit balls inside a sphere\. Since volume measures are also exponential in the sphere dimensionality, this gives us a similarly logarithmic bound for the necessary dimensionality as per sphere packing arguments\([20](https://arxiv.org/html/2608.17088#bib.bib36);[37](https://arxiv.org/html/2608.17088#bib.bib38)\)\.

## Appendix BAdditional details and results for the empirical curse

### B\.1Metrics for multilinguality conditions

We provide more details here on metrics used in[Section5](https://arxiv.org/html/2608.17088#S5)\. The concept set is identical across languages given multiparallel data

#### Monolingual structure

As per[1](https://arxiv.org/html/2608.17088#Thmdff1), the monolingual structure condition is concerned with the quality of monolingual relationships for a given language, and uses a theoretical ideal concept space as reference\. Given a reference spaceZZand target spaceXX, we use*k\-nearest\-neighbour overlap \(MS\-NNO\)*to measure the quality of monolingual neighbourhoods in the subspaceXt⊂XX\_\{t\}\\subset X, for each languagett\. Given𝒯\\mathcal\{T\}is the set of target languages,𝒞\\mathcal\{C\}is the set of language\-agnostic concepts,𝒩Xt​\(c,k\)\\mathcal\{N\}\_\{X\}^\{t\}\(c,k\)is the neighbourhood ofkkclosest concepts forccin languagettin spaceXX, and𝒩Z​\(c,k\)\\mathcal\{N\}\_\{Z\}\(c,k\)is the reference concept neighbourhood,

NNO⁡\(X,Z,k\)\\displaystyle\\mathrm\{NNO\}\(X,Z,k\)=1\|𝒯\|⋅\|𝒞\|⋅k∑t∈𝒯∑c∈𝒞∑nt∈𝒩Z​\(c,k\)𝟏\[nt∈𝒩Xt\(c,k\)\]\\displaystyle=\\frac\{1\}\{\|\\mathcal\{T\}\|\\cdot\|\\mathcal\{C\}\|\\cdot k\}\\sum\_\{t\\in\\mathcal\{T\}\}\\sum\_\{c\\in\\mathcal\{C\}\}\\sum\_\{n^\{t\}\\in\\mathcal\{N\}\_\{Z\}\(c,k\)\}\\mathbf\{1\}\[n^\{t\}\\in\\mathcal\{N\}\_\{X\}^\{t\}\(c,k\)\]
We consider the English concept space from a strong reference model as our theoretical ideal spaceZZand use mappings from multiparallel data to check set membership as in above\.

Note that monolingual text embedding quality in the field of retrieval is measured byrecall@kamong other metrics, given query and relevance judgments\. Our evaluation is an adjustment given the lack of multiparallel retrieval datasets and relevance judgments at our required language scale, and therefore treating monolingual neighbourhoods from a strong model as the gold target set\.

We usek=20k=20for reporting results, and observe similar trends for varyingk=5,10,20,50k=5,10,20,50\.

We also look at language mean over*normalized masked language modeling loss*on a monolingual test set per language \(MS\-MLM\)as an indirect measure of monolingual structure quality\.

#### Cross\-lingual alignment

Drawing from[2](https://arxiv.org/html/2608.17088#Thmdff2), and similar to previous work\([28](https://arxiv.org/html/2608.17088#bib.bib14);[19](https://arxiv.org/html/2608.17088#bib.bib9)\), we measure cross\-lingual alignment in embedding space structure by checking for each languageℓ1\\ell\_\{1\}, whether, for each concept and each languageℓ2\\ell\_\{2\}, theℓ2\\ell\_\{2\}equivalent of that concept is closer to theℓ1\\ell\_\{1\}equivalent than otherℓ2\\ell\_\{2\}concepts \(weak view;CLA\-WV\) or any any other concepts regardless of language \(strong view;CLA\-SV\)\. Given the set of target languages𝒯\\mathcal\{T\},ℓs,ℓt∈𝒯\\ell\_\{s\},\\ell\_\{t\}\\in\\mathcal\{T\}, and𝒢\\mathcal\{G\}as the comparison group depending on strong or weak view,

CLA\\displaystyle\\mathrm\{CLA\}=1\|𝒞\|⋅\|𝒯\|​\(\|𝒯\|−1\)∑ℓs∑ℓt≠ℓs∑c\[argmaxx′∈G\(cos\(xc,ℓs,x′\)\)=xc,ℓt\]\.\\displaystyle=\\frac\{1\}\{\|\\mathcal\{C\}\|\\cdot\|\\mathcal\{T\}\|\(\|\\mathcal\{T\}\|\-1\)\}\\sum\_\{\\ell\_\{s\}\}\\sum\_\{\\ell\_\{t\}\\neq\\ell\_\{s\}\}\\sum\_\{c\}\\mathbf\{1\}\\\!\\left\[\\mathrm\{argmax\}\_\{x^\{\\prime\}\\in G\}\(\\cos\(x\_\{c,\\ell\_\{s\}\},x^\{\\prime\}\)\)=x\_\{c,\\ell\_\{t\}\}\\right\]\.where𝒢=\{xc′,ℓt\|c′≠c\}∪\{xc,ℓt\}\\mathcal\{G\}=\\\{x\_\{c^\{\\prime\},\\ell\_\{t\}\}\|c^\{\\prime\}\\neq c\\\}\\cup\\\{x\_\{c,\\ell\_\{t\}\}\\\}in the weak view and𝒢=\{xc′,ℓm\|c′≠c,ℓm∈𝒯\}∪\{xc,ℓt\}\\mathcal\{G\}=\\\{x\_\{c^\{\\prime\},\\ell\_\{m\}\}\|c^\{\\prime\}\\neq c,\\ell\_\{m\}\\in\\mathcal\{T\}\\\}\\cup\\\{x\_\{c,\\ell\_\{t\}\}\\\}for the strong view\. We study both the weak view and strong view in our experiments and find similar trends\. We report the former\. Note that this evaluation is equivalent to measuring performance on a parallel text mining task\.

### B\.2Further experimental details

#### Models

For the reference model used for monolingual structure metric, we use a strong English text embedding model:e5\-large\-v2101010[huggingface\.co/intfloat/e5\-large\-v2](https://huggingface.co/intfloat/e5-large-v2)\([34](https://arxiv.org/html/2608.17088#bib.bib40)\), andLlama\-3\.1\-8B\-Instruct111111[huggingface\.co/meta\-llama/Llama\-3\.1\-8B\-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct)\([17](https://arxiv.org/html/2608.17088#bib.bib41)\)with last token pooling\. We obtain consistent trends\. Reported results use the former\.

#### Languages

Training languages were chosen in priority of token availability in theMADLAD\-400corpus, as well as availability in evaluation datasets\. We evaluate on all languages covered in the training language group:FLORES\+\(90 languages\)\([16](https://arxiv.org/html/2608.17088#bib.bib20);[32](https://arxiv.org/html/2608.17088#bib.bib19)\),BOUQuET\(90 languages\)\([2](https://arxiv.org/html/2608.17088#bib.bib22);[33](https://arxiv.org/html/2608.17088#bib.bib23)\), andWMT24\+\+\(55 languages\)\([11](https://arxiv.org/html/2608.17088#bib.bib21)\)\.

### B\.3Additional results

#### Other datasets

We show results for two other multiparallel datasets in[Figure4](https://arxiv.org/html/2608.17088#A2.F4)and[Figure5](https://arxiv.org/html/2608.17088#A2.F5)\. We also showCLA\-SV, which shows similar trends forWMT24\+\+, but is generally too low forFLORES\+andBOUQuETto show meaningful trends\.

These show similar trends as shown in[Figure3](https://arxiv.org/html/2608.17088#S5.F3)\.

Figure 4:MS\-NNO,MS\-MLM, andCLAonFLORES\+for encoder models with as the number of training languages increases, in eight configurations\. The top and bottom rows show cursed and non\-cursed configurations respectively\. Arrows indicate whether higher or lower metric values are better\. Error bars show standard deviation over five model seeds\.Figure 5:MS\-NNO,MS\-MLM, andCLAonWMT24\+\+for encoder models with as the number of training languages increases, in eight configurations\. The top and bottom rows show cursed and non\-cursed configurations respectively\. Arrows indicate whether higher or lower metric values are better\. Error bars show standard deviation over five model seeds\.
#### Non\-degeneracy

We plot the mean and standard deviation of pairwise cosine similarity across models forBOUQuETin[Figure6](https://arxiv.org/html/2608.17088#A2.F6)\(similar for other datasets\)\. This is stable, although generally high, which is evidence of high anisotropy in the space\.

Figure 6:Mean pairwise similarity onBOUQuETacross training and evaluation settings\. Error bars show standard deviation over five seeds\.
#### Maintaining token count

In[Section5](https://arxiv.org/html/2608.17088#S5), we discussed that the four settings in which metrics stay stable or improve are generally the settings that maintaintoken count for the target languages of interest\. We elaborate on this here\.

Withfixed\-compute, the token count for each language decreases as we add more languages; thus, mean conditions deteriorate withaggregate\-allfor both types of sampling\. However, when we havefixed\-computeand only look at high\-resource languages inaggregate\-fixedwithrealistic\-sampling, this effect is somewhat softened since subsequent LRLs consume a smaller part of token budget\. Withincreasing\-compute, token count for target languages remains unchanged with more training languages withuniform\-sampling, and metrics stay stable for both types of aggregation\. Withincreasing\-computeandrealistic\-sampling, similar to above, HRLs are able to maintain token count, allowing metrics to stay stable foraggregate\-fixed, whereasaggregate\-alloverrealistic\-samplingmeans that that performance means are driven down by low\-resource languages\.COM\-EMBtrends may potentially also be affected by other things such as model scale, optimization strategies, and data quality, the impact of which we leave to future work to investigate\.

### B\.4Additional exploration of language subspace use and dimensionality

Our construction achieving perfect multilinguality in[Section4\.3](https://arxiv.org/html/2608.17088#S4.SS3)uses compact language offsets that are reused across concepts\. This idea of low\-dimensional and reusable language feature representations is of interest to several works\([26](https://arxiv.org/html/2608.17088#bib.bib35);[39](https://arxiv.org/html/2608.17088#bib.bib32);[8](https://arxiv.org/html/2608.17088#bib.bib3)\)

We want to investigate whether, similarly to our constructive solution, language identity is compactly and reusably represented in our trained models that maintain the multilinguality conditions, and whether this differs by language scale\.

#### Defining language subspace efficiency

We quantify the efficiency of the model regarding language information as the effective dimensionality of the*language subspace*, or the space of the language offsets or language\-expressing directions\.

#### Constructing the language subspace

The language subspace is constructed as the space spanned by the difference vectors between pairs of translation equivalents, i\.e\. points expressing an identical concept in two different languages Formally,

U:=span\{xc,l−xc,l′\|c∈𝒞,l,l′∈ℒ,l≠l′\}U:=\\mathrm\{span\}\\\{x\_\{c,l\}\-x\_\{c,l^\{\\prime\}\}\|c\\in\\mathcal\{C\},l,l^\{\\prime\}\\in\\mathcal\{L\},l\\neq l^\{\\prime\}\\\}Here,xc,lx\_\{c,l\}is unit\-normalized\. We are now interested in the effective dimensionality ofUU\.

#### Computing effective dimensionality

We use participation ratio as a measure of effective dimensionality, in line with previous work\([5](https://arxiv.org/html/2608.17088#bib.bib33)\)\. LetMMstack the language offsetsxc,l−xc,l′x\_\{c,l\}\-x\_\{c,l^\{\\prime\}\}for allccand unordered pairs\(l,l′\)\(l,l^\{\\prime\}\), and letsis\_\{i\}be the singular values ofMMafter row\-centering\. We use

dlang=\(∑isi\)2∑isi2\.d\_\{\\mathrm\{lang\}\}=\\frac\{\\left\(\\sum\_\{i\}s\_\{i\}\\right\)^\{2\}\}\{\\sum\_\{i\}s\_\{i\}^\{2\}\}\.This behaves like a soft dimension count\. Intuitively, if all singular values of active directions are equal, it returns the number of active directions\. Instead, if the spectrum is dominated by a few directions, it returns a smaller value\. We then reporteffdimlang:=dlangD∈\(0,1\]\\mathrm\{effdim\_\{lang\}\}:=\\frac\{d\_\{\\mathrm\{lang\}\}\}\{D\}\\in\(0,1\], whereDDis the ambient dimensionality of the space\.

#### Trends

Figure 7:Effective dimensionality of the language subspace of all training languages onBOUQuETfor non\-cursed training setting\. Error bars show standard deviation over five seeds\.Figure 8:Overlap between concept and language subspace, usingBOUQuET\.
We are interested ineffdimlang\\mathrm\{effdim\_\{lang\}\}for models that maintain the multilinguality conditions\. We construct the language subspace over all training languages, and so we only look ataggregate\-allnon\-cursed configurations\. There is only one such configuration\. See[Figure8](https://arxiv.org/html/2608.17088#A2.F8)for trend ofeffdimlang\\mathrm\{effdim\_\{lang\}\}onBOUQuET\(similar for other datasets\) on this configuration\.

We find that the model consumes considerably more effective dimensionality to represent language information with higher language count, going up to 50% of ambient dimensionality\.

#### Interaction between language and concept subspace

The above experiment raises a natural next point of comparison with our constructive solution\. Namely: is the language subspace orthogonal to the concept subspace, or are semantic and language directions entangled? Note that in our constructive solution, the concept and language subspaces are orthogonal by construction\.

We construct the language subspace as above, and analogously construct a concept subspace using same\-language different\-concept pairs\. We then consider the minimal orthonormal bases of both subspaces that capturek=20%k=20\\%of energy with dimensionalitydcd\_\{c\}anddld\_\{l\}, and quantify the overlap between these directions using principal angle between the subspaces, adjusted for random overlap\. Givenm=min⁡\(dc,dl\)m=\\min\(d\_\{c\},d\_\{l\}\), we formulate this as:

OCL=1m​∑i=1mσi2−max⁡\(dc,dl\)D1−max⁡\(dc,dl\)D\.O\_\{\\mathrm\{CL\}\}=\\frac\{\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}\\sigma\_\{i\}^\{2\}\-\\frac\{\\max\(d\_\{c\},d\_\{l\}\)\}\{D\}\}\{1\-\\frac\{\\max\(d\_\{c\},d\_\{l\}\)\}\{D\}\}\.whereOCL≤1O\_\{\\mathrm\{CL\}\}\\leq 1and higher values mean more overlap\. Motivating this briefly: it can be shown that ifVconcept∈ℝD×dcV\_\{\\mathrm\{concept\}\}\\in\\mathbb\{R\}^\{D\\times d\_\{c\}\}andVlang∈ℝD×dlV\_\{\\mathrm\{lang\}\}\\in\\mathbb\{R\}^\{D\\times d\_\{l\}\}are the orthonormal bases capturingkkproportion of energy, then,σi=svd⁡\(Vconcept⊤​Vlang\)i=cos⁡θi\\sigma\_\{i\}=\\operatorname\{svd\}\(V\_\{\\mathrm\{concept\}\}^\{\\top\}V\_\{\\mathrm\{lang\}\}\)\_\{i\}=\\cos\\theta\_\{i\}, whereθi\\theta\_\{i\}are principal angles betweenVconceptV\_\{\\mathrm\{concept\}\}andVlangV\_\{\\mathrm\{lang\}\}\. It can also be shown that the expected overlap between two subspaces with randomly chosen dimensions given a total dimensionalityDDis:

𝔼⁡\[1m​∑i=1mcos2⁡θi\]=max⁡\(dc,dl\)D\\mathbb\{E\}\\\!\\left\[\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}\\cos^\{2\}\\theta\_\{i\}\\right\]=\\frac\{\\max\(d\_\{c\},d\_\{l\}\)\}\{D\}So our metricOCLO\_\{\\mathrm\{CL\}\}is a measure of the squared cosines of principal angles between the two subspaces of interest, adjusted for random\.

We plot this metric for all settings, shown in[Figure8](https://arxiv.org/html/2608.17088#A2.F8)\. We observe that the overlap is generally high across settings and number of training languages, including non\-cursed settings\. This is in contrast to the ideal constructive solution, where the concept and language subspace are orthogonal, with zero overlap\.

In general, our preliminary experiments indicate that empirical embedding spaces at our small scale do not behave similar to our ideal constructive solution even in non\-cursed conditions\. We leave it to future work to explore the underlying mechanisms of the \(non\-\)curse of multilinguality for embedding space structure in conducive settings and at scale, for various architectures and training configurations\.