Improving Cross-Lingual Token Representations by Adding a Pinch of SALT

# Improving Cross-Lingual Token Representations by Adding a Pinch of SALT
Source: [https://arxiv.org/html/2609.09953](https://arxiv.org/html/2609.09953)
rmTeXGyreTermesX \[\*japanese\]rmHaranoAjiMincho\-Regular\.otf \[\*devanagari\]rmLohit Devanagari \[\*arabic\]rmNoto Sans Arabic

###### Abstract

Cross\-lingual sentence encoders enable scalable transfer across hundreds of languages, powering applications such as translation mining and zero\-shot learning in low\-resource settings\. Although trained for sentence\-level alignment, they are increasingly also applied to token\-level tasks such as hallucination detection and sequence tagging, exposing a mismatch between training and usage\. We propose SALT, a lightweight post\-training method that improves token representations by injecting span\-level supervision into existing sentence encoders\. Across five multilingual token\-level benchmarks, SALT achieves the best overall results on four of them, outperforming alternative fine\-tuning strategies and competitive encoders\. It also improves sentence\-level performance on cross\-lingual retrieval and classification tasks\. These results demonstrate that span\-level supervision is an effective signal for improving both token and sentence representations\.

Pre\-processingSpan alignment extractionwith your age, all chest pain should be treated this waya su edad el dolor de pecho debe tratarse de esta manerawith your age, allchest painshould be treatedthis waya su edadel dolor de pechodebe tratarsede esta maneraEncoder![Refer to caption](https://arxiv.org/html/2609.09953v1/latex/figs/snowflake.png)Sentence\-levelrepresentation\+\+==TeacherembeddingTraining![Refer to caption](https://arxiv.org/html/2609.09953v1/latex/figs/salt.png)a su edad el dolor de pecho debe tratarse de esta manerawith your age, all chest pain should be treated this wayEncoder![Refer to caption](https://arxiv.org/html/2609.09953v1/latex/figs/fire.png)Encoder![Refer to caption](https://arxiv.org/html/2609.09953v1/latex/figs/fire.png)⋯\\cdotsasuedadeldolordepechomanera⋯\\cdotswithyourage,allchestpainwayAverage poolL𝐼𝑁𝑇=𝑀𝑆𝐸⁡\(CLOSEL\_\{\\mathit\{INT\}\}\{=\}\\mathit\{MSE\}\(\+\+,\)\)⋯\\cdotsSpan\-aggregatedembeddings⋯\\cdotsSpan\-basedcontrastive lossL𝐶𝑂𝑁𝑆𝑃𝐴𝑁L\_\{\\mathit\{CON\}\_\{\\mathit\{SPAN\}\}\}Decoder![Refer to caption](https://arxiv.org/html/2609.09953v1/latex/figs/snowflake.png)with your ageL𝑀𝑇𝑆𝑃𝐴𝑁L\_\{\\mathit\{MT\}\_\{\\mathit\{SPAN\}\}\}

Figure 1:Pre\-processing\(top\): extraction of aligned spans and teacher embeddings from a pre\-trained encoder\.Training SALT\(bottom\): we train the encoder with span\-level contrastive and translation losses to improve token\-level representations, besides an interpolation loss that preserves sentence embeddings\.## 1Introduction

Cross\-lingual sentence encoders have become a core building block of multilingual NLP\. Models such as SONAR\([Duquenne et al\., 2023](https://arxiv.org/html/2609.09953#bib.bib40)\)or MEXMA\([Janeiro et al\., 2025](https://arxiv.org/html/2609.09953#bib.bib41)\)learn a shared embedding space in which semantically equivalent sentences across languages are mapped to nearby representations, leading to strong performance on sentence\-level tasks such as cross\-lingual retrieval, sentence mining, and zero\-shot classification\.

Despite being trained for cross\-lingual sentence alignment, these models are increasingly used as*token\-level representations*in downstream applications, including word alignment\([Miao et al\., 2024](https://arxiv.org/html/2609.09953#bib.bib34)\), sequence tagging\([Mehta and Varma, 2023](https://arxiv.org/html/2609.09953#bib.bib35)\), hallucination detection\([Huang et al\., 2024](https://arxiv.org/html/2609.09953#bib.bib16)\), sentence segmentation\([Omnilingual MT Team et al\., 2026](https://arxiv.org/html/2609.09953#bib.bib22)\)and label projection\([Parekh et al\., 2024](https://arxiv.org/html/2609.09953#bib.bib15)\)\. This practice exposes a structural mismatch: sentence encoders are optimised to preserve global semantic similarity, yet are repurposed to support fine\-grained token\- and phrase\-level alignment\. As a result, they may yield well\-structured sentence embeddings but an inconsistent or noisy token representation\.

A promising way to address this mismatch is to introduce training signals at a finer granularity by identifying aligned words across languages and incorporating token\-level losses\([Alqahtani et al\., 2021](https://arxiv.org/html/2609.09953#bib.bib26);[Miao et al\., 2024](https://arxiv.org/html/2609.09953#bib.bib34)\)\. However, word\-level alignment is often too restrictive, as meaning is frequently distributed across multiple tokens and only fully preserved at the level of phrases rather than individual words\.

To this end, we propose using phrases or*spans*—contiguous sequences of tokens that express equivalent meaning across languages\. While this alignment signal is implicitly present in parallel data, previous sentence\- or token\-level objectives do not exploit it\. We introduce![[Uncaptioned image]](https://arxiv.org/html/2609.09953v1/latex/figs/salt.png)SALT\(Span\-AlignedLearning for cross\-lingualTokens\), a novel lightweight post\-training method for multilingual sentence encoders \(see Figure[1](https://arxiv.org/html/2609.09953#S0.F1)\)\. SALT augments standard sentence\-level training with span\-aligned contrastive and translation objectives that explicitly shape token\- and phrase\-level representations, while a sentence\-level interpolation objective preserves the original embedding space\. Importantly, SALT requires no architectural modifications and can be applied directly to existing encoders\.

We evaluate SALT on a suite of multilingual token\-level benchmarks, including word alignment, cross\-lingual sequence tagging, and word sense disambiguation\. Across tasks, SALT consistently improves token\-level performance while maintaining or improving sentence\-level quality\. In particular, it achieves the best overall performance among several fine\-tuning strategies on four out of five token\-level benchmarks, demonstrating that span\-level supervision provides a strong complementary signal rather than a competing objective\.

Our contributions are as follows:

- •We proposeSALT, a lightweight post\-training method that injects span\-level alignment into sentence encoders, bridging sentence\-level training and token\-level applications\.
- •We demonstrate that span\-aligned supervision consistently improves performance across multiple token\-level multilingual tasks, while preserving and sometimes improving sentence\-level embedding quality\.
- •We analyse the geometry of the word embedding space and find that SALT yields a better\-structured cross\-lingual representation, improving hubness while preserving isotropy and reducing language\-specific clustering\.

## 2Related work

##### Sentence encoders

Cross\-lingual sentence encoders aim to learn a shared semantic space in which sentences with equivalent meaning across languages are mapped to nearby embeddings\. Early approaches based on multilingual word embeddings have evolved into large\-scale sentence encoders trained on parallel corpora using contrastive, regression, or decoding objectives\. Models such as SONAR\([Duquenne et al\., 2023](https://arxiv.org/html/2609.09953#bib.bib40)\), MEXMA\([Janeiro et al\., 2025](https://arxiv.org/html/2609.09953#bib.bib41)\), OmniSONAR\([Omnilingual SONAR Team et al\., 2026](https://arxiv.org/html/2609.09953#bib.bib21)\)or LaBSE\([Feng et al\., 2022](https://arxiv.org/html/2609.09953#bib.bib39)\)exemplify this paradigm\.

##### Word and phrase alignment

Early statistical machine translation relied on word alignment models and derived phrase pairs as discrete translation units for decoding\([Brown et al\., 1993](https://arxiv.org/html/2609.09953#bib.bib30);[Koehn et al\., 2003](https://arxiv.org/html/2609.09953#bib.bib31)\), establishing a notion of cross\-lingual span correspondence\. While this is related in spirit to SALT, we use such span correspondences only as a training signal\. More recently, word alignment has been revisited in neural encoders, with methods extracting alignments from contextual representations\([Jalili Sabet et al\., 2020](https://arxiv.org/html/2609.09953#bib.bib36)\)\. Related work improves cross\-lingual alignment through objectives that encourage finer\-grained token\-level correspondence\([Chi et al\., 2021](https://arxiv.org/html/2609.09953#bib.bib32);[Dou and Neubig, 2021](https://arxiv.org/html/2609.09953#bib.bib13)\)\.

##### Bridging sentence\- and token\-level representations

Prior work improves cross\-lingual sentence encoders by incorporating auxiliary word\- or token\-level objectives during pre\-training\([Wei et al\., 2021](https://arxiv.org/html/2609.09953#bib.bib33);[Alqahtani et al\., 2021](https://arxiv.org/html/2609.09953#bib.bib26);[Li et al\., 2021](https://arxiv.org/html/2609.09953#bib.bib27);[Li et al\., 2023](https://arxiv.org/html/2609.09953#bib.bib28);[Miao et al\., 2024](https://arxiv.org/html/2609.09953#bib.bib34)\), typically relying on explicit alignment signals to enhance sentence\-level embeddings\. In contrast, we directly target token\- and span\-level representations, both in training and evaluation, explicitly focusing on improving their quality rather than treating them as a by\-product of sentence\-level learning, focusing on adapting existing sentence encoders for token\-centric tasks\.

## 3![[Uncaptioned image]](https://arxiv.org/html/2609.09953v1/latex/figs/salt.png)SALT: Span\-Aligned Learning for Cross\-Lingual Tokens

To improve cross\-lingual token representations produced by sentence encoders, we propose SALT, a training method that aligns semantically equivalent spans across languages\. SALT is lightweight and can be applied post\-training to any pre\-trained encoder without modifying its architecture\. As illustrated in Figure[1](https://arxiv.org/html/2609.09953#S0.F1), we first extract semantically equivalent multi\-word spans from parallel sentences and precompute their corresponding sentence representations\. During training, span\-level alignment objectives refine token and span embeddings while preserving the structure of the original sentence\-level representation space\.

### 3\.1Method

Letxxandyybe a translation pair\. A tokeniser𝒯\\mathcal\{T\}with vocabularyVVmaps them to token sequencestx=𝒯⁡\(x\)∈Vnt\_\{x\}=\\mathcal\{T\}\(x\)\\in V^\{n\}andty=𝒯⁡\(y\)∈Vmt\_\{y\}=\\mathcal\{T\}\(y\)\\in V^\{m\}\. A pretrained encoderℰ\\mathcal\{E\}then produces contextual embeddingsHx=ℰ⁡\(tx\)∈ℝn×dH\_\{x\}=\\mathcal\{E\}\(t\_\{x\}\)\\in\\mathbb\{R\}^\{n\\times d\}andHy=ℰ⁡\(ty\)∈ℝm×dH\_\{y\}=\\mathcal\{E\}\(t\_\{y\}\)\\in\\mathbb\{R\}^\{m\\times d\}\. These representations are typically trained such that their pooled sentence embeddings are aligned, i\.e\.,pool​\(Hx\)≈pool​\(Hy\)\\text\{pool\}\(H\_\{x\}\)\\approx\\text\{pool\}\(H\_\{y\}\)\. Following[Duquenne et al\. \(2023\)](https://arxiv.org/html/2609.09953#bib.bib40), we adopt average pooling, although our approach is compatible with alternative strategies, such as max pooling or the use of a designated \[CLS\] token \(as in[Janeiro et al\. \(2025\)](https://arxiv.org/html/2609.09953#bib.bib41)\)\.

We define aspan pairas a pair of contiguous token subsequences from a translation pair with similar contextual meaning\. Formally, it is given by\(tx\[i:j\],ty\[i′:j′\]\)\(t\_\{x\}\[i:j\],t\_\{y\}\[i^\{\\prime\}:j^\{\\prime\}\]\)\.tx\[i:j\]t\_\{x\}\[i:j\]denotes tokensi,…,j​−​1i,\\dots,j−1with0≤i<j≤n;0≤i′<j′<m0\\leq i<j\\leq n;\\;0\\leq i^\{\\prime\}<j^\{\\prime\}<m\. The span representations are obtained by pooling the token embeddings within each subsequence, i\.e\.,pool\(Hx\[i:j\]\)\\text\{pool\}\(H\_\{x\}\[i:j\]\)andpool\(Hy\[i′:j′\]\)\\text\{pool\}\(H\_\{y\}\[i^\{\\prime\}:j^\{\\prime\}\]\)\.

#### 3\.1\.1Span extraction

We require aligned spans across parallel sentences to construct training supervision\. We consider two approaches for span extraction: \(i\) an LLM\-based method, and \(ii\) a lightweight heuristic, CASE\.

In most of our experiments, we use LLaMA\-70B\([Team, 2024](https://arxiv.org/html/2609.09953#bib.bib29)\)to extract aligned spans \(see Appendix[B\.1](https://arxiv.org/html/2609.09953#A2.SS1)\)\. However, LLM\-based extraction can be computationally expensive and not always available\. To address this, we introduce CASE \(Constituent Alignment Span Extraction\), a heuristic that extracts aligned spans using token\-level alignments derived from encoderℰ\\mathcal\{E\}representations\.

##### CASE

Given a source sentencexx, we compute token\-level alignmentsM:\{0,…,n−1\}→\{0,…,m−1\}∪\{∅\}M:\\\{0,\\dots,n\-1\\\}\\rightarrow\\\{0,\\dots,m\-1\\\}\\cup\\\{\\varnothing\\\}using Argmax\([Jalili Sabet et al\., 2020](https://arxiv.org/html/2609.09953#bib.bib36)\), extended to multi\-token words\. For a target tokentt, we define the inverse mappingM−1​\(t\)=\{s∣M⁡\(s\)=t\}M^\{\-1\}\(t\)=\\\{s\\mid M\(s\)=t\\\}\.

We consider candidate source spanstx\[i:j\]t\_\{x\}\[i:j\]\. LetAi:j=\{M\(s\)∣s∈\{i,…,j−1\},M\(s\)≠∅\}A\_\{i:j\}=\\\{M\(s\)\\mid s\\in\\\{i,\\dots,j\-1\\\},\\;M\(s\)\\neq\\varnothing\\\}\. Then we define the aligned target span as the tightest span covering all aligned tokens:i′=minAi:ji^\{\\prime\}=\\min A\_\{i:j\},j′=maxAi:j\+1j^\{\\prime\}=\\max A\_\{i:j\}\+1\. To ensure syntactic validity, we restrict source spans to constituents identified using the Benepar parser\([Kitaev et al\., 2019](https://arxiv.org/html/2609.09953#bib.bib5)\)\. We discard span pairs for which alignment links are not fully contained within the span pair, i\.e\., no token in the target span is aligned to a source token outside the source span, and vice versa\.

Finally, we define the coverage of a span pair as the proportion of aligned tokens in both directions:

cover\\displaystyle\\mathrm\{cover\}=∣\{s∣s∈\{i,…,j−1\},M\(s\)≠∅\}∣2​\(\(j−i\)\+\(j′−i′\)\)\\displaystyle=\\frac\{\\mid\\\{s\\mid s\\in\\\{i,\\dots,j\-1\\\},M\(s\)\\neq\\varnothing\\\}\\mid\}\{2\(\(j\-i\)\+\(j^\{\\prime\}\-i^\{\\prime\}\)\)\}\(1\)\+∣\{t∣t∈\{i′,…,j′−1\},M−1\(t\)≠∅\}∣2​\(\(j−i\)\+\(j′−i′\)\)\\displaystyle\+\\frac\{\\mid\\\{t\\mid t\\in\\\{i^\{\\prime\},\\dots,j^\{\\prime\}\-1\\\},M^\{\-1\}\(t\)\\neq\\varnothing\\\}\\mid\}\{2\(\(j\-i\)\+\(j^\{\\prime\}\-i^\{\\prime\}\)\)\}
We discard any pair of source\-target spans whose coverage falls below a thresholdcc= 0\.7, ensuring that only well\-aligned spans are used\.

##### Evaluation of span extraction

We evaluate span quality via human evaluation on Chinese \(zho\_Hans\), French, Hindi, Russian, and Spanish\. Spans are extracted from English–target sentence pairs from our word alignment dataset \(Section[4\.1](https://arxiv.org/html/2609.09953#S4.SS1)\)\. We recruit 7 native speakers to annotate a total of 2,361 span pairs for both LLaMA\-70B and CASE \(with SONAR\), rating span semantic equivalence on a 4\-point scale: \(1\) unrelated, \(2\) somewhat related, \(3\) nearly equivalent, \(4\) equivalent\. See Appendix[G](https://arxiv.org/html/2609.09953#A7)for guidelines and full results\.

Figure[2](https://arxiv.org/html/2609.09953#S3.F2)shows the score distribution\. Both LLaMA\-based extraction and CASE yield high\-quality span pairs, with over 90% of annotations in the top two categories\. LLaMA performs better on Russian and French, while CASE matches or exceeds it in other languages\. We retain score\-4 spans to construct CrossSpan for the evaluation in Section[5\.1](https://arxiv.org/html/2609.09953#S5.SS1)\. Overall, both methods effectively identify semantically equivalent spans, supporting their use as supervision signals\. CASE offers an efficient alternative to LLM\-based extraction while preserving alignment quality\.

Figure 2:Human evaluation of extracted spans\. Score distributions for LLaMA\-70B and SONAR \(CASE\)\.

#### 3\.1\.2SALT training objective

SALT uses a translation \(MT\) and a contrastive \(CON\) objective at the span level, as well as an interpolation objective \(INT\) at the sentence level\.

ℒS​A​L​T=ℒC​O​NS​p​a​n\+α​ℒM​TS​p​a​n\+β​ℒI​N​T\\mathcal\{L\}\_\{SALT\}=\\mathcal\{L\}\_\{CON\_\{Span\}\}\+\\alpha\\mathcal\{L\}\_\{MT\_\{Span\}\}\+\\beta\\mathcal\{L\}\_\{INT\}\(2\)whereα,β∈ℝ\\alpha,\\beta\\in\\mathbb\{R\}are hyperparameters set to one unless otherwise specified\. For simplicity, we present the losses for a single translation pair\(x,y\)\(x,y\), though they are computed in batches during training\.

##### Contrastive objective

We apply the span extraction method \(LLM or CASE\) to obtain a collection ofppaligned spanstx\[i1:j1\],…,tx\[ip:jp\]t\_\{x\}\[i^\{1\}:j^\{1\}\],\\dots,t\_\{x\}\[i^\{p\}:j^\{p\}\]andty\[i′1:j′1\],…,ty\[i′p:j′p\]t\_\{y\}\[i^\{\\prime 1\}:j^\{\\prime 1\}\],\\dots,t\_\{y\}\[i^\{\\prime p\}:j^\{\\prime p\}\]\.

To increase coverage of both sentences, we augment this set with unaligned spans\. Specifically, for every maximal contiguous subsequence of tokens that does not belong to any aligned span, we form an additional span\. This yields additional spans\(tx\[ip\+1:jp\+1\],…,tx\[iq:jq\]\)\(t\_\{x\}\[i^\{p\+1\}:j^\{p\+1\}\],\\dots,t\_\{x\}\[i^\{q\}:j^\{q\}\]\)and\(ty\[i′p\+1:j′p\+1\],…,ty\[i′v:j′v\]\)\(t\_\{y\}\[i^\{\\prime p\+1\}:j^\{\\prime p\+1\}\],\\dots,t\_\{y\}\[i^\{\\prime v\}:j^\{\\prime v\}\]\)\.

We compute the corresponding span embeddingsHx1,…,HxqH\_\{x\}^\{1\},\\dots,H\_\{x\}^\{q\}andHy1,…,HyvH\_\{y\}^\{1\},\\dots,H\_\{y\}^\{v\}by averaging the token embeddings within each span\. We then define the contrastive loss as

ℒCONS​p​a​n\\displaystyle\\mathcal\{L\}\_\{\\text\{CON\}\_\{Span\}\}=−1p∑i=1p\[logexp⁡\(⟨Hxi,Hyi⟩/τ\)∑j=1vexp⁡\(⟨Hxi,Hyj⟩/τ\)\\displaystyle=\-\\frac\{1\}\{p\}\\sum\_\{i=1\}^\{p\}\\Bigg\[\\log\\frac\{\\exp\(\\langle H\_\{x\}^\{i\},H\_\{y\}^\{i\}\\rangle/\\tau\)\}\{\\sum\_\{j=1\}^\{v\}\\exp\(\\langle H\_\{x\}^\{i\},H\_\{y\}^\{j\}\\rangle/\\tau\)\}\(3\)\+logexp⁡\(⟨Hxi,Hyi⟩/τ\)∑j=1qexp⁡\(⟨Hxj,Hyi⟩/τ\)\]\\displaystyle\+\\log\\frac\{\\exp\(\\langle H\_\{x\}^\{i\},H\_\{y\}^\{i\}\\rangle/\\tau\)\}\{\\sum\_\{j=1\}^\{q\}\\exp\(\\langle H\_\{x\}^\{j\},H\_\{y\}^\{i\}\\rangle/\\tau\)\}\\Bigg\]
where⟨⋅,⋅⟩\\langle\\cdot,\\cdot\\rangledenotes the dot product,\(Hxi,Hyi\)\(H\_\{x\}^\{i\},H\_\{y\}^\{i\}\)corresponds to an aligned span pair, andτ\\tauis a temperature hyperparameter\. The first term applies a row\-wise softmax over target spans for each source span, while the second term applies a column\-wise softmax over source spans for each target span\. This loss encourages semantically equivalent spans to have similar embeddings while pushing apart spans with different meanings\.

##### Translation objective

Given a pre\-trained decoder𝒟\\mathcal\{D\}that we keep frozen, we define the translation loss as the cross\-entropy over the target span:

ℒMTSpan=−1p∑l=1p∑k=i′lj′l−1logP𝒟\(ty\[k\];ty\[i′l:k\],Hxl\)\\mathcal\{L\}\_\{\\text\{MT\}\_\{\\text\{Span\}\}\}\\\!=\\\!\-\\tfrac\{1\}\{p\}\\sum\_\{l=1\}^\{p\}\\sum\_\{k=i^\{\\prime l\}\}^\{j^\{\\prime l\}\\\!\-\\\!1\}\\\!\\log P\_\{\\mathcal\{D\}\}\\big\(t\_\{y\}\[k\];t\_\{y\}\[i^\{\\prime l\}:k\],H\_\{x\}^\{l\}\\big\)\(4\)where \(tx\[il:jl\]t\_\{x\}\[i^\{l\}:j^\{l\}\]andty\[i′l:j′l\]t\_\{y\}\[i^\{\\prime l\}:j^\{\\prime l\}\]\) is an extracted span pair,HxlH\_\{x\}^\{l\}is the pooled source span representation andP𝒟P\_\{\\mathcal\{D\}\}denotes the probability assigned by the decoder to the next target token, conditioned on the previous target tokens and the pooled representation of the source span\.

##### Interpolation objective

Following[Tsiamas et al\. \(2025\)](https://arxiv.org/html/2609.09953#bib.bib17), we introduce a sentence\-level interpolation objective that encourages the encoder to preserve the original sentence representation space\. We obtain a teacher representationzx​yz\_\{xy\}by averaging the source and target sentence embeddings produced by the frozen encoderℰf​r​o​z​e​n\\mathcal\{E\}^\{frozen\}\. The interpolation loss is then defined as

ℒI​N​T=MSE​\(zx​y,pool​\(Hx\)\+pool​\(Hy\)2\)\\mathcal\{L\}\_\{INT\}=\\text\{MSE\}\\Big\(z\_\{xy\},\\frac\{\\text\{pool\}\(H\_\{x\}\)\+\\text\{pool\}\(H\_\{y\}\)\}\{2\}\\Big\)\(5\)
whereMSEdenotes the mean squared error\. Intuitively, this loss constrains the encoder so that token\-level training objectives do not distort the sentence\-level embedding space, maintaining consistency with the initial frozen encoder\.

Task\(\#langs\)ExampleAER\(18\)Thenecessarycorrectionwillbemade\.EineentsprechendeAnderungwirdvorgenommenwerden\.Massive\(52\)What is today ’s forecast for BerlinDATEPLACEPAN\-X\(40\)REDIRECCIÓN Algarrobo \( Chile \)B\-LOCI\-LOCUDPOS\(51\)葬儀 の 最中 です よ \!NOUNADPNOUNAUXPARTPUNCTWiC\(6\)Bolivia holds a keyplayin this process of peace\.A musicalplayon the same subject was also staged\.\}​Differentmeaning\\left\.\\begin\{array\}\[\]\{@\{\}l@\{\}\}\\text\{Bolivia holds a key \\lx@text@underline\{play\} in this process of peace\.\}\\\\ \\text\{A musical \\lx@text@underline\{play\} on the same subject was also staged\.\}\\end\{array\}\\right\\\}\\ \\begin\{array\}\[\]\{@\{\}l@\{\}\}\\text\{Different\}\\\\ \\text\{meaning\}\\end\{array\}

Table 1:Summary of the tasks and number of languages used for the evaluation of cross\-lingual token representations\. Source:[Omnilingual SONAR Team et al\. \(2026\)](https://arxiv.org/html/2609.09953#bib.bib21)

## 4Experiment details

We initialiseℰ\\mathcal\{E\},𝒟\\mathcal\{D\}with the respective pre\-trained SONAR weights\([Duquenne et al\., 2023](https://arxiv.org/html/2609.09953#bib.bib40)\)\.

##### Data

We fine\-tune SONAR on a subset of the data originally used to train it, namely the publicly available NLLB Primary dataset\([Costa\-jussà et al\., 2022](https://arxiv.org/html/2609.09953#bib.bib20)\)\.111Downloaded using the script provided at[https://github\.com/gordicaleksa/Open\-NLLB](https://github.com/gordicaleksa/Open-NLLB/blob/nllb_replication/examples/nllb/data/README.md)\.This dataset comprises high\-quality parallel sentences spanning a diverse set of languages\. We begin by excluding all language pairs for which neither language belongs to the set of 57 designated test languages, yielding a total of 129 language pairs \(see Appendix[C\.1](https://arxiv.org/html/2609.09953#A3.SS1)\)\. The 29 test languages not observed during training can be used to evaluate the generalisation of our method \(see Appendix[D\.3](https://arxiv.org/html/2609.09953#A4.SS3)\)\.

For each retained language pair, we downsample the corpus to 40,000 sentence pairs, observing no degradation in performance as a result of this reduction\. In line with the procedure of[Omnilingual SONAR Team et al\. \(2026\)](https://arxiv.org/html/2609.09953#bib.bib21), we further refine the dataset by removing sentence pairs whose BLASER 2 score\([Dale and Costa\-jussà, 2024](https://arxiv.org/html/2609.09953#bib.bib19)\)deviates by more than one standard deviation from the mean score computed for the given language pair\.

##### Training hyperparameters

We train SONAR for 20,000 steps, with a maximum of 1400 tokens per batch \(roughly corresponding to 50 pairs of sentences per batch\) and a learning rate ofμ=10​e−5\\mu=10e\-5\. We use an AdamW optimiser\([Loshchilov and Hutter, 2019](https://arxiv.org/html/2609.09953#bib.bib38)\)and warm\-up of 1000 steps\. We do the training runs on three seeds and report average values in all experiments unless otherwise specified\. Standard deviations are reported in Appendix[F](https://arxiv.org/html/2609.09953#A6)\.

##### Baselines

We report the performance of the encoders XLM\-R\([Conneau and Lample, 2019](https://arxiv.org/html/2609.09953#bib.bib37)\), XLM\-Align\([Chi et al\., 2021](https://arxiv.org/html/2609.09953#bib.bib32)\), LaBSE\([Feng et al\., 2022](https://arxiv.org/html/2609.09953#bib.bib39)\), and MEXMA\([Janeiro et al\., 2025](https://arxiv.org/html/2609.09953#bib.bib41)\)\. In addition, we compare SALT against variants obtained by fine\-tuning SONAR with alternative training objectives\. Specifically, we train the encoder for longer using the original SONAR loss proposed by[Duquenne et al\. \(2023\)](https://arxiv.org/html/2609.09953#bib.bib40)\. We also include token\-level objectives: the Self\-Objective \(SO\) loss\([Dou and Neubig, 2021](https://arxiv.org/html/2609.09953#bib.bib13)\), a contrastive word\-alignment objective; the WordOT loss\([Alqahtani et al\., 2021](https://arxiv.org/html/2609.09953#bib.bib26)\), using Optimal Transport; the WACSE loss\([Miao et al\., 2024](https://arxiv.org/html/2609.09953#bib.bib34)\), which adds a masked language modeling head; and the OmniSONAR\-Token loss\([Omnilingual SONAR Team et al\., 2026](https://arxiv.org/html/2609.09953#bib.bib21)\), which extends SO loss with interpolation\. See Appendix[C\.4](https://arxiv.org/html/2609.09953#A3.SS4)for implementation details\.

##### Sentence evaluation

We evaluate the task of sentence mining using the FLORES\-200 devtest\([NLLB Team, 2024](https://arxiv.org/html/2609.09953#bib.bib18)\)\(80 languages\) and report xsim\([Feng et al\., 2022](https://arxiv.org/html/2609.09953#bib.bib39)\)and xsim\+\+\([Chen et al\., 2023](https://arxiv.org/html/2609.09953#bib.bib11)\)\. For classification, we report on tasks from MTEB\([Muennighoff et al\., 2023](https://arxiv.org/html/2609.09953#bib.bib12)\)\(English only\)\.

### 4\.1Cross\-lingual token evaluation

Standard benchmarks often evaluate sentence encoders through sequence\-level tasks, which can mask weaknesses in token representations\([Conneau et al\., 2018](https://arxiv.org/html/2609.09953#bib.bib6);[Hu et al\., 2020](https://arxiv.org/html/2609.09953#bib.bib2);[Enevoldsen et al\., 2025](https://arxiv.org/html/2609.09953#bib.bib3)\)\. Following[Omnilingual SONAR Team et al\. \(2026\)](https://arxiv.org/html/2609.09953#bib.bib21), we instead focus on tasks that isolate the embeddings of individual tokens, providing a more rigorous assessment of cross\-lingual token quality\. For word alignment, we test whether tokens across languages align semantically\. For sequence tagging, we feed the classifier head only the embedding of a single word to directly probe the quality of its token\-level representations across languages\.

##### Word alignment

Given a translation pairx,yx,y, the task of word alignment identifies which words correspond to each other semantically\. More formally, given a source–target sentence pairxx,yywith word lengthswxw\_\{x\},wyw\_\{y\}, respectively, we infer a binary matrixM∈\{0,1\}ws×wyM\\in\\\{0,1\\\}^\{w\_\{s\}\\times w\_\{y\}\}, whereMi​j=1M\_\{ij\}=1represents that theii\-th word in the source sentence aligns semantically with thejj\-th word in the target sentence\. We present an example in Table[1](https://arxiv.org/html/2609.09953#S3.T1)\.

Many current word alignment methods derive alignments based on the similarity of token embeddings\([Jalili Sabet et al\., 2020](https://arxiv.org/html/2609.09953#bib.bib36);[Azadi et al\., 2023](https://arxiv.org/html/2609.09953#bib.bib14)\)\. We define thetoken similarity matrixS∈ℝn×mS\\in\\mathbb\{R\}^\{n\\times m\};Si,j:=s​i​m​\(ℰ⁡\(tx​\[i\]\),ℰ⁡\(ty​\[j\]\)\)S\_\{i,j\}:=sim\(\\mathcal\{E\}\(t\_\{x\}\[i\]\),\\mathcal\{E\}\(t\_\{y\}\[j\]\)\), wheresimis a similarity measure \(we use cosine similarity\),tx​\[i\]t\_\{x\}\[i\]andty​\[j\]∈Vt\_\{y\}\[j\]\\in Vare theii\-th andjj\-th tokens of the source and target sentencesxxandyy\. Anextractionmethod is then applied to convert these similarities into a discrete alignment\. We use Itermax\([Jalili Sabet et al\., 2020](https://arxiv.org/html/2609.09953#bib.bib36)\)to extract alignments and report Alignment Error Rate \(AER\)\([Och and Ney, 2000](https://arxiv.org/html/2609.09953#bib.bib9)\)\. The full list of alignment datasets is in Appendix[C\.2](https://arxiv.org/html/2609.09953#A3.SS2)\.

##### Sequence tagging

We evaluate token\-level cross\-lingual representations using standard sequence labelling tasks, where each tokenxix\_\{i\}in an input sequencex=\(x1,…,xn\)x=\(x\_\{1\},\\dots,x\_\{n\}\)is assigned a discrete label\{1,…,N\}\\\{1,\\dots,N\\\}\. To isolate the contribution of the encoder, we train a linear classification headC∈ℝd×NC\\in\\mathbb\{R\}^\{d\\times N\}placed on top of the token embeddings while keeping the encoder parameters fixed\. The classifier is trained only on English data and subsequently evaluated in other languages \(zero\-shot\); see Appendix[C\.3](https://arxiv.org/html/2609.09953#A3.SS3)for implementation details\.

We consider a diverse set of benchmarks, including Massive \(Slot Filling\)\([FitzGerald et al\., 2023](https://arxiv.org/html/2609.09953#bib.bib1)\), PAN\-X \(Named Entity Recognition\)\([Pan et al\., 2017](https://arxiv.org/html/2609.09953#bib.bib25)\), UDPOS \(Part\-of\-Speech Tagging\)\([Nivre et al\., 2020](https://arxiv.org/html/2609.09953#bib.bib23)\), and WiC \(Word Sense Disambiguation\)\([Pilehvar and Camacho\-Collados, 2019](https://arxiv.org/html/2609.09953#bib.bib24)\)\. WiC is not technically a sequence tagging task; we adapt it by extracting the contextual embeddings of the target word in each sentence, concatenating the resulting vectors, and feeding them into a classifierC∈ℝ2​d×2C\\in\\mathbb\{R\}^\{2d\\times 2\}\. When a target word is split into multiple subword tokens, we represent it by averaging the corresponding embeddings\.

Finally, to handle mismatches between dataset annotations and tokeniser segmentation, we map labels to tokens based on maximal character overlap\. When multiple segments overlap a token equally, we resolve ties by selecting the first\.

AER \(↓\\downarrow\)PAN\-X \(↑\\uparrow\)Massive \(↑\\uparrow\)UDPOS \(↑\\uparrow\)WiC \(↑\\uparrow\)Pre\-trained encodersXLM\-R0\.3070\.5510\.2200\.5270\.538XLM\-Align0\.2460\.5320\.1870\.5220\.556LaBSE0\.2320\.5890\.4310\.5510\.538MEXMA0\.1710\.5930\.3960\.5530\.565SONAR0\.1830\.5820\.4020\.5480\.576SONAR \+ additional objectivesSONAR Loss\([Duquenne et al\., 2023](https://arxiv.org/html/2609.09953#bib.bib40)\)0\.1790\.5880\.4260\.5540\.581SO Loss\([Dou and Neubig, 2021](https://arxiv.org/html/2609.09953#bib.bib13)\)0\.1780\.5970\.4100\.5770\.574WordOT\([Alqahtani et al\., 2021](https://arxiv.org/html/2609.09953#bib.bib26)\)0\.3200\.6090\.4170\.5610\.585WACSE\([Miao et al\., 2024](https://arxiv.org/html/2609.09953#bib.bib34)\)0\.2050\.6000\.4090\.5690\.573OmniSONAR\-Token\([Omnilingual SONAR Team et al\., 2026](https://arxiv.org/html/2609.09953#bib.bib21)\)0\.1800\.6100\.4170\.5740\.583SALT![[Uncaptioned image]](https://arxiv.org/html/2609.09953v1/latex/figs/salt.png)\(ours\)0\.1710\.6220\.4520\.5710\.585

Table 2:Cross\-lingual token\-level evaluation across five benchmarks for frozen pre\-trained encoders and SONAR trained with additional objectives\. SALT achieves the best or joint\-best performance on four of five benchmarks\.Figure 3:Principal component projections of token embeddings for the same sentence across 80 languages\. Each colour corresponds to a language\. SONAR produces a more distributed and language\-agnostic embedding space\. Sentence:The great pyramid was created to honor the Pharaoh Khufu, and many of the smaller pyramids, tombs, and temples were built to honor Khufu’s wives and family members\.

## 5Results

Table[2](https://arxiv.org/html/2609.09953#S4.T2)compares SALT with pre\-trained encoders and alternative fine\-tuning objectives for SONAR across five cross\-lingual token benchmarks\. Among the pre\-trained models, MEXMA achieves the lowest AER \(0\.171\), while SONAR remains competitive on the other tasks\. Continued training with the SONAR loss improves all token\-level results, suggesting that SONAR was primarily optimised for sentence mining rather than token representations\.

Among the fine\-tuning objectives, SALT outperforms in word alignment \(AER\), PAN\-X, Massive, and WiC, showing that span\-level objectives provide complementary signals beyond sentence\- or token\-level training\. Gains are particularly pronounced on tasks requiring rich cross\-lingual semantics \(PAN\-X, Massive\) and on word alignment\. On UDPOS, SALT achieves a strong score of 0\.571 \(vs\. 0\.548 for the pre\-trained model\), but is slightly outperformed by SO Loss \(0\.577\)\. This may be due to UDPOS being more language\-specific, exhibiting limited cross\-lingual transfer relative to other tasks \(see Appendix[D\.2](https://arxiv.org/html/2609.09953#A4.SS2)\)

Taken together, the results show that SALT achieves the best overall performance among the considered strategies, and suggest that incorporating span\-level supervision is an effective and principled way to enrich multilingual sentence representations beyond sentence\-level objectives alone\.

Does improved token\-level representation come at the cost of sentence\-level quality? For cross\-lingual sentence mining, SALT consistently improves over SONAR on xsim and xsim\+\+ \(Table[3](https://arxiv.org/html/2609.09953#S5.T3)\)\. For MTEB classification tasks, SALT matches SONAR on XNLI and MIntent and improves on STS17\. Overall, SALT does not sacrifice sentence\-level quality and, in several cases, improves it\.

Mining \(↓\\downarrow\)Classification \(↑\\uparrow\)xsimxsim\+\+XNLISTS17MIntentSONAR0\.29\.90\.610\.650\.58SALT0\.18\.80\.610\.660\.58Table 3:Cross\-lingual sentence mining \(FLORES\-80\) and English\-only MTEB classification performance\.Figure 4:Left: cosine similarity of span pairs across layers\. SALT shows a better similarity than SONAR across all layers\. Right: translation performance for span embeddings\. Both experiments use CrossSpan\.### 5\.1Analysis

##### Span embeddings

SALT supports span\-level representations by averaging token embeddings within each span\. We find that these span representations are more semantically informative than their SONAR counterparts\. On the CrossSpan test set of high\-quality aligned cross\-lingual spans, SALT representations of aligned spans consistently exhibit higher cosine similarity across layers \(Figure[4](https://arxiv.org/html/2609.09953#S5.F4), left\), indicating stronger cross\-lingual alignment\.

A by\-product of our span\-translation objective is that the learned representations can be directly translated with afrozenSONAR decoder \(Figure[4](https://arxiv.org/html/2609.09953#S5.F4), right\)\. SALT span embeddings yield high\-quality translations, indicating that they preserve rich semantic information for downstream generation\.

##### Intermediate representations

Sentence encoders typically exhibit aparabolicbehaviour, where intermediate layers achieve the strongest cross\-lingual alignment\([Jalili Sabet et al\., 2020](https://arxiv.org/html/2609.09953#bib.bib36)\)\. This arises because cross\-lingual token representations emerge in mid\-layers but are not necessarily preserved in the final layers, which are more specialised for the pretraining objective\. The SALT loss shifts this pattern, with deeper layers becoming more cross\-lingually aligned for both spans \(Figure[4](https://arxiv.org/html/2609.09953#S5.F4), left\) and words \(Appendix[E\.1](https://arxiv.org/html/2609.09953#A5.SS1)\)\.

##### Geometry of the embedding space

We compare the geometry of word embeddings produced by SALT, SONAR, and MEXMA \(Table[4](https://arxiv.org/html/2609.09953#S5.T4); see Appendix[E\.2](https://arxiv.org/html/2609.09953#A5.SS2)for more details on experiments\)\.

We construct a word retrieval task by merging the datasets from the word alignment task\. For each English word, we retrieve the nearest neighbours with cosine similarity and evaluate whether the matches correspond to gold\-aligned targets\. SALT achieves the highest recall\.

A good cross\-lingual representation should be language\-agnostic, without inducing language\-specific subspaces\. We probe language separability by training a linear classifier to predict language ID from individual word embeddings\. SALT embeddings are significantly harder to classify, indicating more language\-invariant representations than SONAR and MEXMA\.

Isotropy captures whether the embedding space is dominated by privileged directions, leading to uneven use of representational capacity\. We measure it via mean cosine similarity over random word pairs, and via uniformity\([Wang and Isola, 2020](https://arxiv.org/html/2609.09953#bib.bib7)\)\. MEXMA is highly anisotropic\. SONAR and SALT produce more isotropic spaces by these measures, though as we show below, SONAR’s apparent isotropy masks a severe hubness problem\.

Hubness measures the tendency of a few embeddings to dominate nearest\-neighbour retrieval regardless of semantics\([Radovanovic et al\., 2010](https://arxiv.org/html/2609.09953#bib.bib10)\)\. We quantify it via the skewness of thekk\-occurrence distribution, which counts how often an embedding appears in another word’s nearest neighbours\. Zero skewness indicates a uniform space, while high skewness reflects hub collapse\. SONAR exhibits severe hubness \(11\.5 skewness\), while MEXMA and SALT are substantially better behaved\.

Taken together, these results reveal contrasting geometric failures in the baselines\. MEXMA produces an anisotropic, language\-stratified space, while SONAR suffers from severe hub collapse\. SALT avoids both, yielding a better\-distributed cross\-lingual token space \(Figure[3](https://arxiv.org/html/2609.09953#S4.F3)\)\. This suggests that isotropy, hubness and language inseparability are not independent but mutually reinforcing signatures of a well\-formed cross\-lingual geometry\.

Word Ret\.↑\\uparrowLang\. ID↓\\downarrowIsotropy↓\\downarrowHub↓\\downarrowsim / unifMEXMA63%0\.490\.44 / \-2\.24\.0SONAR62%0\.460\.04 /\-3\.811\.5SALT65%0\.350\.03/ \-3\.63\.5Table 4:Word retrieval \(Recall@5\), language identification \(F1\), isotropy \(mean cosine similarity and uniformity\) and hubness \(skewness\) for three encoders\.

### 5\.2Ablations

##### Impact of the different losses

We assess the contribution of each loss term of the SALT training objective by removing them and evaluating across two metrics: word alignment \(AER\) and averaging the four sequence tagging tasks\.

The results are presented in Table[6](https://arxiv.org/html/2609.09953#S5.T6)\. We see that post\-training onℒC​O​NS​p​a​n\\mathcal\{L\}\_\{\{CON\}\_\{Span\}\}or onℒI​N​T\\mathcal\{L\}\_\{INT\}improves on both tasks, being the latest the strongest single loss;ℒM​TS​p​a​n\\mathcal\{L\}\_\{\{MT\}\_\{Span\}\}worsens the pre\-trained model, and it worsens word alignment when used in conjunction withℒC​O​NS​p​a​n\\mathcal\{L\}\_\{\{CON\}\_\{Span\}\}or onℒI​N​T\\mathcal\{L\}\_\{INT\}; however, it adds complementary information not fully captured by the other two terms — its inclusion in SALT pushes SeqTag to the best score \(0\.557\), even asℒC​O​NS​p​a​n\+ℒI​N​T\\mathcal\{L\}\_\{\{CON\}\_\{Span\}\}\+\\mathcal\{L\}\_\{INT\}yields a marginally better AER\.

The full SALT objective achieves the best sequence tagging score and a highly competitive AER of 0\.171, confirming that all three terms are necessary for optimal performance\. No subset consistently dominates across both metrics, indicating that each loss captures complementary training signals\.

##### CASE span extraction

Do the SALT improvements stem from an additional external signal, given that LLaMA\-70B was used for span extraction? To investigate this, we compare results when training SALT with the self\-supervised span extraction heuristic CASE\. Results are shown in Table[5](https://arxiv.org/html/2609.09953#S5.T5)\. CASE yields better performance on Massive, whereas external supervision from LLaMA\-70B is superior on PAN\-X, UDPOS, and WiC\. Nevertheless, the difference is marginal: CASE would outperform all the baseline losses and encoders in Table[2](https://arxiv.org/html/2609.09953#S4.T2)except for UDPOS\. This suggests that the training regime is primarily responsible for SALT’s success, rather than the span extraction signal itself, and that comparable results can be achieved without access to a strong LLM\.

AERPAN\-XMassiveUDPOSWiCSALT \(LLaMA\)0\.1710\.6220\.4520\.5710\.585SALT \(CASE\)0\.1710\.6130\.4590\.5680\.584

Table 5:Cross\-lingual token\-level evaluation for SALT using LLaMA\-70B spans or the self\-supervised CASE\.AER↓\\downarrowSeqTag↑\\uparrowSONAR \(pre\-trained\)0\.1830\.527ℒC​O​NS​p​a​n\\mathcal\{L\}\_\{CON\_\{Span\}\}0\.1780\.538ℒM​TS​p​a​n\\mathcal\{L\}\_\{MT\_\{Span\}\}0\.1860\.522ℒI​N​T\\mathcal\{L\}\_\{INT\}0\.1740\.551ℒC​O​NS​p​a​n\+ℒM​TS​p​a​n\\mathcal\{L\}\_\{CON\_\{Span\}\}\+\\mathcal\{L\}\_\{MT\_\{Span\}\}0\.1790\.551ℒM​TS​p​a​n\+ℒI​N​T\\mathcal\{L\}\_\{MT\_\{Span\}\}\+\\mathcal\{L\}\_\{INT\}0\.1770\.554ℒC​O​NS​p​a​n\+ℒI​N​T\\mathcal\{L\}\_\{CON\_\{Span\}\}\+\\mathcal\{L\}\_\{INT\}0\.1700\.552ℒC​O​NS​p​a​n\+ℒM​TS​p​a​n\+ℒI​N​T\\mathcal\{L\}\_\{CON\_\{Span\}\}\+\\mathcal\{L\}\_\{MT\_\{Span\}\}\+\\mathcal\{L\}\_\{INT\}\(SALT\)0\.1710\.557

Table 6:Finetuning SONAR with different losses\.

## 6Conclusions

In this work, we introduced SALT, a lightweight post\-training method that improves cross\-lingual token representations in multilingual sentence encoders through span\-aligned supervision\. Across multilingual token\-level benchmarks, SALT achieved the strongest overall performance, consistently outperforming alternative fine\-tuning strategies and competitive encoders\. These gains did not come at the expense of sentence\-level quality: SALT preserved and, in several cases, improved performance on sentence mining and classification benchmarks\. Overall, our results suggest that span\-level supervision is a simple yet powerful inductive bias for multilingual representation learning\.

## Limitations

Despite the strong empirical performance of SALT, several limitations remain\.

SALT relies on parallel corpora to extract aligned spans, which may limit its applicability to truly low\-resource languages\. In addition, the self\-supervised CASE heuristic assumes that the underlying encoder already provides reasonably good cross\-lingual token representations, making adaptation to unseen or severely underrepresented languages more challenging; in those cases, LLM\-based spans should be used instead\.

Our method also uses average pooling to construct span embeddings\. While lightweight and architecture\-agnostic, this may fail to capture finer internal structure within longer or compositionally complex spans\.

Finally, we only evaluate SALT as a post\-training intervention\. Exploring span\-level supervision during large\-scale pre\-training could potentially yield larger improvements, but was beyond the computational scope of this work\.

## Ethics statement

This work aims to improve multilingual language representations for token\-level cross\-lingual tasks, which may help broaden language accessibility and support underrepresented languages in NLP\. However, multilingual representation models may also be used in sensitive applications such as surveillance or profiling, and we therefore encourage responsible deployment and evaluation\.

Our training data are derived from publicly available parallel corpora from the NLLB Primary dataset, which may contain societal or cultural biases present in web\-scale multilingual text\. As a result, SALT may inherit biases from the training data or pretrained models\.

We also conducted a human evaluation of span quality using native speakers\. Participation was voluntary, annotators were informed about the study and their right to withdraw, and no personally identifiable information was collected beyond anonymised annotator IDs\. The study received ethics approval from our institution\.

## References

- Alqahtaniet al\.\(2021\)S\. Alqahtani, G\. Lalwani, Y\. Zhang, S\. Romeo, and S\. MansourUsing optimal transport as alignment objective for fine\-tuning multilingual contextualized embeddings\.InFindings of the Association for Computational Linguistics: EMNLP 2021,M\. Moens, X\. Huang, L\. Specia, and S\. W\. Yih \(Eds\.\),Punta Cana, Dominican Republic,pp\. 3904–3919\.External Links:[Link](https://aclanthology.org/2021.findings-emnlp.329/),[Document](https://dx.doi.org/10.18653/v1/2021.findings-emnlp.329)Cited by:[§1](https://arxiv.org/html/2609.09953#S1.p3.1),[§2](https://arxiv.org/html/2609.09953#S2.SS0.SSS0.Px3.p1.1),[§4](https://arxiv.org/html/2609.09953#S4.SS0.SSS0.Px3.p1.1),[Table 2](https://arxiv.org/html/2609.09953#S4.T2.2.1.11.1)\.
- Azadiet al\.\(2023\)F\. Azadi, H\. Faili, and M\. J\. DoustiPMI\-align: word alignment with point\-wise mutual information without requiring parallel training data\.InFindings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9\-14, 2023,A\. Rogers, J\. L\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Findings of ACL,pp\. 12366–12377\.External Links:[Link](https://doi.org/10.18653/v1/2023.findings-acl.782),[Document](https://dx.doi.org/10.18653/V1/2023.FINDINGS-ACL.782)Cited by:[§4\.1](https://arxiv.org/html/2609.09953#S4.SS1.SSS0.Px1.p2.1)\.
- Brownet al\.\(1993\)P\. F\. Brown, S\. D\. Pietra, V\. J\. D\. Pietra, and R\. L\. MercerThe mathematics of statistical machine translation: parameter estimation\.Comput\. Linguistics19\(2\),pp\. 263–311\.Cited by:[§2](https://arxiv.org/html/2609.09953#S2.SS0.SSS0.Px2.p1.1)\.
- Chenet al\.\(2023\)M\. Chen, K\. Heffernan, O\. Çelebi, A\. Mourachko, and H\. SchwenkxSIM\+\+: An Improved Proxy to Bitext Mining Performance for Low\-Resource Languages\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 101–109\.External Links:[Link](https://aclanthology.org/2023.acl-short.10/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-short.10)Cited by:[§4](https://arxiv.org/html/2609.09953#S4.SS0.SSS0.Px4.p1.1)\.
- Chiet al\.\(2021\)Z\. Chi, L\. Dong, B\. Zheng, S\. Huang, X\. Mao, H\. Huang, and F\. WeiImproving pretrained cross\-lingual language models via self\-labeled word alignment\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),C\. Zong, F\. Xia, W\. Li, and R\. Navigli \(Eds\.\),Online,pp\. 3418–3430\.External Links:[Link](https://aclanthology.org/2021.acl-long.265/),[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.265)Cited by:[§2](https://arxiv.org/html/2609.09953#S2.SS0.SSS0.Px2.p1.1),[§4](https://arxiv.org/html/2609.09953#S4.SS0.SSS0.Px3.p1.1)\.
- Conneau and Lample \(2019\)A\. Conneau and G\. LampleCross\-lingual language model pretraining\.InAdvances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8\-14, 2019, Vancouver, BC, Canada,H\. M\. Wallach, H\. Larochelle, A\. Beygelzimer, F\. d’Alché\-Buc, E\. B\. Fox, and R\. Garnett \(Eds\.\),pp\. 7057–7067\.External Links:[Link](https://proceedings.neurips.cc/paper/2019/hash/c04c19c2c2474dbf5f7ac4372c5b9af1-Abstract.html)Cited by:[§4](https://arxiv.org/html/2609.09953#S4.SS0.SSS0.Px3.p1.1)\.
- Conneauet al\.\(2018\)A\. Conneau, R\. Rinott, G\. Lample, A\. Williams, S\. R\. Bowman, H\. Schwenk, and V\. StoyanovXNLI: Evaluating Cross\-lingual Sentence Representations\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,Cited by:[§4\.1](https://arxiv.org/html/2609.09953#S4.SS1.p1.1)\.
- Costa\-jussàet al\.\(2022\)M\. R\. Costa\-jussà, J\. Cross, O\. Çelebi, M\. Elbayad, K\. Heafield, K\. Heffernan, E\. Kalbassi, J\. Lam, D\. Licht, J\. Maillard, A\. Y\. Sun, S\. Wang, G\. Wenzek, A\. Youngblood, B\. Akula, L\. Barrault, G\. M\. Gonzalez, P\. Hansanti, J\. Hoffman, S\. Jarrett, K\. R\. Sadagopan, D\. Rowe, S\. Spruit, C\. Tran, P\. Andrews, N\. F\. Ayan, S\. Bhosale, S\. Edunov, A\. Fan, C\. Gao, V\. Goswami, F\. Guzmán, P\. Koehn, A\. Mourachko, C\. Ropers, S\. Saleem, H\. Schwenk, and J\. WangNo language left behind: scaling human\-centered machine translation\.CoRRabs/2207\.04672\.External Links:[Link](https://doi.org/10.48550/arXiv.2207.04672),[Document](https://dx.doi.org/10.48550/ARXIV.2207.04672),2207\.04672Cited by:[§4](https://arxiv.org/html/2609.09953#S4.SS0.SSS0.Px1.p1.1)\.
- Dale and Costa\-jussà \(2024\)D\. Dale and M\. R\. Costa\-jussàBLASER 2\.0: a metric for evaluation and quality estimation of massively multilingual speech and text translation\.InFindings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12\-16, 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Findings of ACL,pp\. 16075–16085\.External Links:[Link](https://doi.org/10.18653/v1/2024.findings-emnlp.943),[Document](https://dx.doi.org/10.18653/V1/2024.FINDINGS-EMNLP.943)Cited by:[§4](https://arxiv.org/html/2609.09953#S4.SS0.SSS0.Px1.p2.1)\.
- Dou and Neubig \(2021\)Z\. Dou and G\. NeubigWord alignment by fine\-tuning embeddings on parallel corpora\.InProceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 \- 23, 2021,P\. Merlo, J\. Tiedemann, and R\. Tsarfaty \(Eds\.\),pp\. 2112–2128\.External Links:[Link](https://doi.org/10.18653/v1/2021.eacl-main.181),[Document](https://dx.doi.org/10.18653/V1/2021.EACL-MAIN.181)Cited by:[§C\.2](https://arxiv.org/html/2609.09953#A3.SS2.p1.1),[§2](https://arxiv.org/html/2609.09953#S2.SS0.SSS0.Px2.p1.1),[§4](https://arxiv.org/html/2609.09953#S4.SS0.SSS0.Px3.p1.1),[Table 2](https://arxiv.org/html/2609.09953#S4.T2.2.1.10.1)\.
- Duquenneet al\.\(2023\)P\. Duquenne, H\. Schwenk, and B\. SagotSONAR: sentence\-level multimodal and language\-agnostic representations\.CoRRabs/2308\.11466\.External Links:[Link](https://doi.org/10.48550/arXiv.2308.11466),[Document](https://dx.doi.org/10.48550/ARXIV.2308.11466),2308\.11466Cited by:[§1](https://arxiv.org/html/2609.09953#S1.p1.1),[§2](https://arxiv.org/html/2609.09953#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2609.09953#S3.SS1.p1.1),[§4](https://arxiv.org/html/2609.09953#S4.SS0.SSS0.Px3.p1.1),[Table 2](https://arxiv.org/html/2609.09953#S4.T2.2.1.9.1),[§4](https://arxiv.org/html/2609.09953#S4.p1.1)\.
- Enevoldsenet al\.\(2025\)K\. C\. Enevoldsen, I\. Chung, I\. Kerboua, M\. Kardos, A\. Mathur, D\. Stap, J\. Gala, W\. Siblini, D\. Krzeminski, G\. I\. Winata, S\. Sturua, S\. Utpala, M\. Ciancone, M\. Schaeffer, D\. Misra, S\. Dhakal, J\. Rystrøm, R\. Solomatin, Ö\. V\. Çagatan, A\. Kundu, and et al\.MMTEB: massive multilingual text embedding benchmark\.InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24\-28, 2025,External Links:[Link](https://openreview.net/forum?id=zl3pfz4VCV)Cited by:[§4\.1](https://arxiv.org/html/2609.09953#S4.SS1.p1.1)\.
- Fenget al\.\(2022\)F\. Feng, Y\. Yang, D\. Cer, N\. Arivazhagan, and W\. WangLanguage\-agnostic BERT sentence embedding\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2022, Dublin, Ireland, May 22\-27, 2022,S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),pp\. 878–891\.External Links:[Link](https://doi.org/10.18653/v1/2022.acl-long.62),[Document](https://dx.doi.org/10.18653/V1/2022.ACL-LONG.62)Cited by:[§2](https://arxiv.org/html/2609.09953#S2.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2609.09953#S4.SS0.SSS0.Px3.p1.1),[§4](https://arxiv.org/html/2609.09953#S4.SS0.SSS0.Px4.p1.1)\.
- FitzGeraldet al\.\(2023\)J\. FitzGerald, C\. Hench, C\. Peris, S\. Mackie, K\. Rottmann, A\. Sanchez, A\. Nash, L\. Urbach, V\. Kakarala, R\. Singh, S\. Ranganath, L\. Crist, M\. Britan, W\. Leeuwis, G\. Tur, and P\. NatarajanMASSIVE: A 1M\-Example Multilingual Natural Language Understanding Dataset with 51 Typologically\-Diverse Languages\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 4277–4302\.External Links:[Link](https://aclanthology.org/2023.acl-long.235/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.235)Cited by:[§4\.1](https://arxiv.org/html/2609.09953#S4.SS1.SSS0.Px2.p2.1)\.
- Huet al\.\(2020\)J\. Hu, S\. Ruder, A\. Siddhant, G\. Neubig, O\. Firat, and M\. JohnsonXTREME: A massively multilingual multi\-task benchmark for evaluating cross\-lingual generalization\.CoRRabs/2003\.11080\.External Links:[Link](https://arxiv.org/abs/2003.11080),2003\.11080Cited by:[§4\.1](https://arxiv.org/html/2609.09953#S4.SS1.p1.1)\.
- Huanget al\.\(2024\)C\. Huang, A\. Ghaddar, I\. Kobyzev, M\. Rezagholizadeh, O\. Zaïane, and B\. ChenOTTAWA: optimal transport adaptive word aligner for hallucination and omission translation errors detection\.InFindings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11\-16, 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Findings of ACL,pp\. 6322–6334\.External Links:[Link](https://doi.org/10.18653/v1/2024.findings-acl.377),[Document](https://dx.doi.org/10.18653/V1/2024.FINDINGS-ACL.377)Cited by:[§1](https://arxiv.org/html/2609.09953#S1.p2.1)\.
- Jalili Sabetet al\.\(2020\)M\. Jalili Sabet, P\. Dufter, F\. Yvon, and H\. SchützeSimAlign: high quality word alignments without parallel training data using static and contextualized embeddings\.InFindings of the Association for Computational Linguistics: EMNLP 2020,T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 1627–1643\.External Links:[Link](https://aclanthology.org/2020.findings-emnlp.147/),[Document](https://dx.doi.org/10.18653/v1/2020.findings-emnlp.147)Cited by:[§C\.2](https://arxiv.org/html/2609.09953#A3.SS2.p1.1),[§C\.4](https://arxiv.org/html/2609.09953#A3.SS4.SSS0.Px1.p1.1),[§C\.4](https://arxiv.org/html/2609.09953#A3.SS4.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2609.09953#S2.SS0.SSS0.Px2.p1.1),[§3\.1\.1](https://arxiv.org/html/2609.09953#S3.SS1.SSS1.Px1.p1.1),[§4\.1](https://arxiv.org/html/2609.09953#S4.SS1.SSS0.Px1.p2.1),[§5\.1](https://arxiv.org/html/2609.09953#S5.SS1.SSS0.Px2.p1.1)\.
- Janeiroet al\.\(2025\)J\. M\. Janeiro, B\. Piwowarski, P\. Gallinari, and L\. BarraultMEXMA: token\-level objectives improve sentence representations\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2025, Vienna, Austria, July 27 \- August 1, 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),pp\. 23960–23995\.External Links:[Link](https://aclanthology.org/2025.acl-long.1168/)Cited by:[§1](https://arxiv.org/html/2609.09953#S1.p1.1),[§2](https://arxiv.org/html/2609.09953#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2609.09953#S3.SS1.p1.1),[§4](https://arxiv.org/html/2609.09953#S4.SS0.SSS0.Px3.p1.1)\.
- Kitaevet al\.\(2019\)N\. Kitaev, S\. Cao, and D\. KleinMultilingual constituency parsing with self\-attention and pre\-training\.InProceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28\- August 2, 2019, Volume 1: Long Papers,A\. Korhonen, D\. R\. Traum, and L\. Màrquez \(Eds\.\),pp\. 3499–3505\.External Links:[Link](https://doi.org/10.18653/v1/p19-1340),[Document](https://dx.doi.org/10.18653/V1/P19-1340)Cited by:[§3\.1\.1](https://arxiv.org/html/2609.09953#S3.SS1.SSS1.Px1.p2.1)\.
- Koehnet al\.\(2003\)P\. Koehn, F\. J\. Och, and D\. MarcuStatistical phrase\-based translation\.InHuman Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, HLT\-NAACL 2003, Edmonton, Canada, May 27 \- June 1, 2003,M\. A\. Hearst and M\. Ostendorf \(Eds\.\),External Links:[Link](https://aclanthology.org/N03-1017/)Cited by:[§2](https://arxiv.org/html/2609.09953#S2.SS0.SSS0.Px2.p1.1)\.
- Liet al\.\(2021\)S\. Li, P\. Yang, F\. Luo, and J\. XieMulti\-granularity contrasting for cross\-lingual pre\-training\.InFindings of the Association for Computational Linguistics: ACL\-IJCNLP 2021,C\. Zong, F\. Xia, W\. Li, and R\. Navigli \(Eds\.\),Online,pp\. 1708–1717\.External Links:[Link](https://aclanthology.org/2021.findings-acl.149/),[Document](https://dx.doi.org/10.18653/v1/2021.findings-acl.149)Cited by:[§2](https://arxiv.org/html/2609.09953#S2.SS0.SSS0.Px3.p1.1)\.
- Liet al\.\(2023\)Z\. Li, S\. Huang, Z\. Zhang, Z\. Deng, Q\. Lou, H\. Huang, J\. Jiao, F\. Wei, W\. Deng, and Q\. ZhangDual\-alignment pre\-training for cross\-lingual sentence embedding\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 3466–3478\.External Links:[Link](https://aclanthology.org/2023.acl-long.191/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.191)Cited by:[§2](https://arxiv.org/html/2609.09953#S2.SS0.SSS0.Px3.p1.1)\.
- Loshchilov and Hutter \(2019\)I\. Loshchilov and F\. HutterDecoupled weight decay regularization\.In7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6\-9, 2019,External Links:[Link](https://openreview.net/forum?id=Bkg6RiCqY7)Cited by:[§4](https://arxiv.org/html/2609.09953#S4.SS0.SSS0.Px2.p1.1)\.
- Martelliet al\.\(2023\)F\. Martelli, A\. S\. Bejgu, C\. Campagnano, J\. Cibej, R\. Costa, A\. Gantar, J\. Kallas, S\. P\. Koeva, K\. Koppel, S\. Krek, M\. Langemets, V\. Lipp, S\. Nimb, S\. Olsen, B\. S\. Pedersen, V\. Quochi, A\. Salgado, L\. Simon, C\. Tiberius, Rafael\-J\. Ureña\-Ruiz, and R\. NavigliXL\-WA: a gold evaluation benchmark for word alignment in 14 language pairs\.InProceedings of the 9th Italian Conference on Computational Linguistics, Venice, Italy, November 30 \- December 2, 2023,F\. Boschetti, G\. E\. Lebani, B\. Magnini, and N\. Novielli \(Eds\.\),CEUR Workshop Proceedings\.External Links:[Link](https://ceur-ws.org/Vol-3596/paper32.pdf)Cited by:[§C\.2](https://arxiv.org/html/2609.09953#A3.SS2.p1.1)\.
- Mehta and Varma \(2023\)R\. Mehta and V\. VarmaLLM\-RM at SemEval\-2023 task 2: multilingual complex NER using XLM\-RoBERTa\.InProceedings of the 17th International Workshop on Semantic Evaluation \(SemEval\-2023\),A\. Kr\. Ojha, A\. S\. Doğruöz, G\. Da San Martino, H\. Tayyar Madabushi, R\. Kumar, and E\. Sartori \(Eds\.\),Toronto, Canada,pp\. 453–456\.External Links:[Link](https://aclanthology.org/2023.semeval-1.62/),[Document](https://dx.doi.org/10.18653/v1/2023.semeval-1.62)Cited by:[§1](https://arxiv.org/html/2609.09953#S1.p2.1)\.
- Miaoet al\.\(2024\)Z\. Miao, Q\. Wu, K\. Zhao, Z\. Wu, and Y\. TsuruokaEnhancing cross\-lingual sentence embedding for low\-resource languages with word alignment\.InFindings of the Association for Computational Linguistics: NAACL 2024,K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 3225–3236\.External Links:[Link](https://aclanthology.org/2024.findings-naacl.204/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-naacl.204)Cited by:[§1](https://arxiv.org/html/2609.09953#S1.p2.1),[§1](https://arxiv.org/html/2609.09953#S1.p3.1),[§2](https://arxiv.org/html/2609.09953#S2.SS0.SSS0.Px3.p1.1),[§4](https://arxiv.org/html/2609.09953#S4.SS0.SSS0.Px3.p1.1),[Table 2](https://arxiv.org/html/2609.09953#S4.T2.2.1.12.1)\.
- Muennighoffet al\.\(2023\)N\. Muennighoff, N\. Tazi, L\. Magne, and N\. ReimersMTEB: massive text embedding benchmark\.InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics,A\. Vlachos and I\. Augenstein \(Eds\.\),Dubrovnik, Croatia,pp\. 2014–2037\.External Links:[Link](https://aclanthology.org/2023.eacl-main.148/),[Document](https://dx.doi.org/10.18653/v1/2023.eacl-main.148)Cited by:[§4](https://arxiv.org/html/2609.09953#S4.SS0.SSS0.Px4.p1.1)\.
- Neubig \(2011\)G\. NeubigThe kyoto free translation task\.Note:http://www\.phontron\.com/kfttCited by:[§C\.2](https://arxiv.org/html/2609.09953#A3.SS2.p1.1)\.
- Nivreet al\.\(2020\)J\. Nivre, M\. de Marneffe, F\. Ginter, J\. Hajic, C\. D\. Manning, S\. Pyysalo, S\. Schuster, F\. M\. Tyers, and D\. ZemanUniversal dependencies v2: an evergrowing multilingual treebank collection\.InProceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11\-16, 2020,N\. Calzolari, F\. Béchet, P\. Blache, K\. Choukri, C\. Cieri, T\. Declerck, S\. Goggi, H\. Isahara, B\. Maegaard, J\. Mariani, H\. Mazo, A\. Moreno, J\. Odijk, and S\. Piperidis \(Eds\.\),pp\. 4034–4043\.External Links:[Link](https://aclanthology.org/2020.lrec-1.497/)Cited by:[§4\.1](https://arxiv.org/html/2609.09953#S4.SS1.SSS0.Px2.p2.1)\.
- NLLB Team \(2024\)NLLB TeamScaling Neural Machine Translation to 200 Languages\.Nature630,pp\. 841–846\.External Links:[Document](https://dx.doi.org/10.1038/s41586-024-07335-x),[Link](https://www.nature.com/articles/s41586-024-07335-x)Cited by:[Appendix I](https://arxiv.org/html/2609.09953#A9.p2.1),[§4](https://arxiv.org/html/2609.09953#S4.SS0.SSS0.Px4.p1.1)\.
- Och and Ney \(2000\)F\. J\. Och and H\. NeyImproved statistical alignment models\.In38th Annual Meeting of the Association for Computational Linguistics, Hong Kong, China, October 1\-8, 2000,pp\. 440–447\.External Links:[Link](https://aclanthology.org/P00-1056/),[Document](https://dx.doi.org/10.3115/1075218.1075274)Cited by:[§4\.1](https://arxiv.org/html/2609.09953#S4.SS1.SSS0.Px1.p2.1)\.
- Omnilingual MT Teamet al\.\(2026\)Omnilingual MT Team, B\. Alastruey, N\. Bafna, A\. Caciolai, K\. Heffernan, A\. Kozhevnikov, C\. Ropers, E\. Sánchez, C\. Saint\-James, I\. Tsiamas, C\. Cheng, J\. Chuang, P\. Duquenne, M\. Duppenthaler, N\. Ekberg, C\. Gao, P\. L\. H\. Cabot, J\. M\. Janeiro, J\. Maillard, G\. M\. Gonzalez, H\. Schwenk, E\. Toledo, A\. Turkatenko, A\. Ventayol\-Boada, R\. Moritz, A\. Mourachko, S\. Parimi, M\. Williamson, S\. Yates, D\. Dale, and M\. R\. Costa\-jussàOmnilingual mt: machine translation for 1,600 languages\.External Links:2603\.16309,[Link](https://arxiv.org/abs/2603.16309)Cited by:[§1](https://arxiv.org/html/2609.09953#S1.p2.1)\.
- Omnilingual SONAR Teamet al\.\(2026\)Omnilingual SONAR Team, J\. M\. Janeiro, P\. H\. Cabot, I\. Tsiamas, Y\. Meng, V\. Iyer, G\. Ramírez, L\. Barrault, B\. Alastruey, Y\. Chung, M\. R\. Costa\-Jussa, D\. Dale, K\. Heffernan, J\. Jo, A\. Kozhevnikov, A\. Mourachko, C\. Ropers, H\. Schwenk, and P\. DuquenneOmnilingual sonar: cross\-lingual and cross\-modal sentence embeddings bridging massively multilingual text and speech\.CoRRabs/2603\.16606\.External Links:[Link](https://arxiv.org/abs/2603.16606),[Document](https://dx.doi.org/10.48550/arXiv.2603.16606),2603\.16606Cited by:[§C\.4](https://arxiv.org/html/2609.09953#A3.SS4.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2609.09953#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2609.09953#S3.T1),[§4](https://arxiv.org/html/2609.09953#S4.SS0.SSS0.Px1.p2.1),[§4](https://arxiv.org/html/2609.09953#S4.SS0.SSS0.Px3.p1.1),[§4\.1](https://arxiv.org/html/2609.09953#S4.SS1.p1.1),[Table 2](https://arxiv.org/html/2609.09953#S4.T2.2.1.13.1)\.
- Panet al\.\(2017\)X\. Pan, B\. Zhang, J\. May, J\. Nothman, K\. Knight, and H\. JiCross\-lingual name tagging and linking for 282 languages\.InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 \- August 4, Volume 1: Long Papers,R\. Barzilay and M\. Kan \(Eds\.\),pp\. 1946–1958\.External Links:[Link](https://doi.org/10.18653/v1/P17-1178),[Document](https://dx.doi.org/10.18653/V1/P17-1178)Cited by:[§4\.1](https://arxiv.org/html/2609.09953#S4.SS1.SSS0.Px2.p2.1)\.
- Parekhet al\.\(2024\)T\. Parekh, I\. Hsu, K\. Huang, K\. Chang, and N\. PengContextual label projection for cross\-lingual structured prediction\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\), NAACL 2024, Mexico City, Mexico, June 16\-21, 2024,K\. Duh, H\. Gómez\-Adorno, and S\. Bethard \(Eds\.\),pp\. 5738–5757\.External Links:[Link](https://doi.org/10.18653/v1/2024.naacl-long.321),[Document](https://dx.doi.org/10.18653/V1/2024.NAACL-LONG.321)Cited by:[§1](https://arxiv.org/html/2609.09953#S1.p2.1)\.
- Pilehvar and Camacho\-Collados \(2019\)M\. T\. Pilehvar and J\. Camacho\-ColladosWiC: the word\-in\-context dataset for evaluating context\-sensitive meaning representations\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL\-HLT 2019, Minneapolis, MN, USA, June 2\-7, 2019, Volume 1 \(Long and Short Papers\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),pp\. 1267–1273\.External Links:[Link](https://doi.org/10.18653/v1/n19-1128),[Document](https://dx.doi.org/10.18653/V1/N19-1128)Cited by:[§4\.1](https://arxiv.org/html/2609.09953#S4.SS1.SSS0.Px2.p2.1)\.
- Radovanovicet al\.\(2010\)M\. Radovanovic, A\. Nanopoulos, and M\. IvanovicHubs in space: popular nearest neighbors in high\-dimensional data\.J\. Mach\. Learn\. Res\.11,pp\. 2487–2531\.External Links:[Link](https://dl.acm.org/doi/10.5555/1756006.1953015),[Document](https://dx.doi.org/10.5555/1756006.1953015)Cited by:[§5\.1](https://arxiv.org/html/2609.09953#S5.SS1.SSS0.Px3.p5.1)\.
- Team \(2024\)L\. TeamThe llama 3 herd of models\.CoRRabs/2407\.21783\.External Links:[Link](https://doi.org/10.48550/arXiv.2407.21783),[Document](https://dx.doi.org/10.48550/ARXIV.2407.21783),2407\.21783Cited by:[§3\.1\.1](https://arxiv.org/html/2609.09953#S3.SS1.SSS1.p2.1)\.
- Tsiamaset al\.\(2025\)I\. Tsiamas, D\. Dale, and M\. R\. Costa\-jussàImproving language and modality transfer in translation by character\-level modeling\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2025, Vienna, Austria, July 27 \- August 1, 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),pp\. 20171–20187\.External Links:[Link](https://aclanthology.org/2025.acl-long.988/)Cited by:[§3\.1\.2](https://arxiv.org/html/2609.09953#S3.SS1.SSS2.Px3.p1.1)\.
- Wang and Isola \(2020\)T\. Wang and P\. IsolaUnderstanding contrastive representation learning through alignment and uniformity on the hypersphere\.InProceedings of the 37th International Conference on Machine Learning, ICML 2020, 13\-18 July 2020, Virtual Event,Proceedings of Machine Learning Research,pp\. 9929–9939\.External Links:[Link](http://proceedings.mlr.press/v119/wang20k.html)Cited by:[§5\.1](https://arxiv.org/html/2609.09953#S5.SS1.SSS0.Px3.p4.1)\.
- Weiet al\.\(2021\)X\. Wei, R\. Weng, Y\. Hu, L\. Xing, H\. Yu, and W\. LuoOn learning universal representations across languages\.In9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3\-7, 2021,External Links:[Link](https://openreview.net/forum?id=Uu1Nw-eeTxJ)Cited by:[§2](https://arxiv.org/html/2609.09953#S2.SS0.SSS0.Px3.p1.1)\.

## Appendix AA note on segmentation

We define words as whitespace\-delimited units of text\. For languages that do not use whitespace segmentation—namely scriptsHans, Hant, Jpan, Thai, Laoo, Khmr, Mymr, Tibtfollowing the SONAR notation—we treat tokeniser tokens as words\. When computing word embeddings, we obtain them by averaging the embeddings of the tokens that compose each word\.

## Appendix BSpans extraction

### B\.1Extracting spans with an LLM

For the LLM\-based method, we use LLaMA\-70B Instruct, quantised to 8\-bit precision using Bitsandbytes\. The prompt used for span extraction is provided in Figure[9](https://arxiv.org/html/2609.09953#A10.F9)\.

The generated spans are subsequently mapped back to SONAR tokens\. In the vast majority of cases, this mapping is unambiguous; when a SONAR token is split across multiple spans, we assign it to the span with the greatest string overlap\.

### B\.2Statistics on extracted spans

Table[14](https://arxiv.org/html/2609.09953#A7.T14)reports summary statistics of the extracted spans\. We observe that span lengths vary across languages, with CASE tending to extract longer spans in both source and target sides\. In particular, Chinese and Hindi exhibit relatively longer spans compared to other languages, often involving a larger number of target tokens\. Using spaCy’sen\_core\_web\_smmodel to analyse POS tags on the English side, we find that nouns are most frequent, followed by adpositions\. We also observe variation in POS tag distributions across languages and span sources; further details are provided in Table[15](https://arxiv.org/html/2609.09953#A7.T15)\.

## Appendix CExperiment details

##### Pre\-processing

### C\.1Languages studied

Table[16](https://arxiv.org/html/2609.09953#A10.T16)shows the language splits used for training and evaluation\. Training languages are drawn from the NLLB Primary dataset, while the test languages correspond to those covered by our cross\-lingual token\-level benchmark, including Word Alignment, PAN\-X, MASSIVE, UDPOS, and WiC\. For sentence retrieval, we use the 57 test languages; we only evaluate the MTEB tasks on English\.

### C\.2Alignment datasets

We evaluate a total of 18 language pairs\. As gold\-standard alignments, we use the XL\-WA dataset\([Martelli et al\., 2023](https://arxiv.org/html/2609.09953#bib.bib4)\)for 10 language pairs; we were unable to obtain the four additional subsets\. For en\-hi, en\-fr, en\-fa, and en\-cs, we adopt the same datasets as in[Jalili Sabet et al\. \(2020\)](https://arxiv.org/html/2609.09953#bib.bib36)\. For en\-zh, en\-ro, and en\-de, we use the datasets introduced by[Dou and Neubig \(2021\)](https://arxiv.org/html/2609.09953#bib.bib13), and for en\-ja we use the KFTT dataset\([Neubig, 2011](https://arxiv.org/html/2609.09953#bib.bib8)\)\.

### C\.3Sequence tagging evaluation

We use up to 10,000 training examples and 5,000 test examples per language, doubling these limits for WiC, while filtering out sequences exceeding 514 tokens\. A single linear classifier \(Linear\(dim, n\_classes\)\) is trained on frozen encoder embeddings using Adam with a learning rate of10−310^\{\-3\}, cross\-entropy loss, and a batch size of 4096\. We train with early stopping \(patience=10=10\) based on a 90/10 train\-validation split\. All embeddings are L2\-normalised before classification\.

### C\.4Losses implementations details

We refer readers to the original papers for detailed descriptions of the losses and their motivation\. Here, we describe implementation details and hyperparameters used in our setup\.

##### SO Loss

We follow[Omnilingual SONAR Team et al\. \(2026\)](https://arxiv.org/html/2609.09953#bib.bib21)and introduce a temperature hyperparameterτ=500\\tau=500\. We use Argmax\([Jalili Sabet et al\., 2020](https://arxiv.org/html/2609.09953#bib.bib36)\)for word alignment extraction\.

##### WordOT

We useτ=0\.05\\tau=0\.05, following the original paper\.

##### WACSE

We use Argmax\([Jalili Sabet et al\., 2020](https://arxiv.org/html/2609.09953#bib.bib36)\)for word alignment extraction \(WTR, AWP losses\)\. For AWP, we only use one mask per sentence pair\. We use the token embedding \(encoding\) matrix as the MLM head, and use a cross\-entropy loss\.

##### SALT

We useτ=5\.0\\tau=5\.0in our contrastive span loss\.

### C\.5Data sampling during training

Training uses dynamically constructed multilingual mini\-batches under a fixed token budget\. At each step, sentence pairs are sampled from source–target language distributions using temperature\-smoothed probabilities over language pairs\. Each pair is tokenised with the corresponding language\-specific encoder, and any example exceeding the maximum sequence length \(512\) on either side is discarded\.

A buffered packing strategy is then used to improve batch efficiency\. Examples are first ordered by sequence cost, defined asmax⁡\(\|x\|,\|y\|\)\\max\(\|x\|,\|y\|\), where\|x\|\|x\|and\|y\|\|y\|denote the number of tokens in the source and target sentences\. This reduces padding and increases the proportion of useful tokens per update\.

Optimisation follows the setup described in the main paper, using AdamW with a linear warm\-up over the first 1,000 steps and cosine annealing thereafter, decaying the learning rate smoothly to zero over the rest of training\.

## Appendix DOther results

### D\.1Training dynamics

We show the training dynamics in terms of sequence tagging and Word Alignment performance of the different losses studied in Figure[8](https://arxiv.org/html/2609.09953#A5.F8)\.

### D\.2Sequence tagging results

In this section, we provide an expanded analysis of the sequence tagging results\.

We evaluate sequence tagging under three different training settings:

- •English: the classifier is trained using only English data\.
- •All: the classifier is trained jointly on data from all languages\.
- •Single: for each target language, the classifier is trained exclusively on data from that language\.

Across all three settings, SALT consistently outperforms SONAR \(Figure[5](https://arxiv.org/html/2609.09953#A4.F5), Table[7](https://arxiv.org/html/2609.09953#A4.T7)\); Figure[12](https://arxiv.org/html/2609.09953#A10.F12)and[13](https://arxiv.org/html/2609.09953#A10.F13)show the performance per language on the English setting\)\. Notably, we observe that UDPOS exhibits limited cross\-lingual transfer: training on a single language yields substantially better performance than multilingual training \(Singlevs\.All\), suggesting that the task depends strongly on language\-specific characteristics\.

Figure 5:Performance of SALT vs SONAR on the sequence tagging tasks, on three training modes for the probe: only English data, all the data and data for every test language\. SALT improves all the results\.TaskSALT \(single\)SALT \(English\)SALT \(all\)SONAR \(single\)SONAR \(English\)SONAR \(all\)Massive0\.550\.450\.560\.540\.400\.54PAN\-X0\.790\.630\.730\.780\.580\.70UDPOS0\.760\.570\.620\.740\.550\.60WiC0\.600\.590\.590\.590\.580\.58

Table 7:F1 scores for the sequence tagging tasks on the three training settings\.
### D\.3Language generalisation

We evaluate the generalisation ability of our method on held\-out test languages unseen during training\. Results are consistent, with SALT outperforming the other losses on these languages \(see Table[8](https://arxiv.org/html/2609.09953#A4.T8)\) except on UDPOS\. Importantly, the training dynamics \(Figure[6](https://arxiv.org/html/2609.09953#A4.F6)\) show that held\-out languages benefit from SALT, particularly at earlier training stages, whereas trained languages continue to improve over a longer training horizon\.

PAN\-XMassiveUDPOSSONAR Loss0\.5620\.4110\.530SO Loss0\.5770\.3950\.549WordOT0\.5860\.4100\.535WACSE0\.5820\.4000\.538OmniSONAR\-Token0\.5900\.4020\.548SALT0\.5980\.4420\.544

Table 8:Sequence tagging performance on test languages not seen during training\.Figure 6:Sequence tagging performance throughout SALT training on trained \(left\) and held\-out \(right\) languages\.

## Appendix EAnalysis

### E\.1Word embeddings alignment

Figure[7](https://arxiv.org/html/2609.09953#A5.F7)shows the cosine similarity of spans and words\. We use high\-quality reference spans from CrossSpan and split them into words or multi\-word spans based on length\. Note that this applies only to this figure; in other experiments, spans may also consist of single words\.

### E\.2Geometry analysis

For word retrieval, we use our word alignment dataset \(Appendix[C\.2](https://arxiv.org/html/2609.09953#A3.SS2)\): first, we aggregate word embeddings by averaging token embeddings \(given space segmentation\)\. Second, for every English word that has a defined target, we find the 5 nearest neighbours using cosine similarity\.

We run the remainder of our experiments for geometry analysis on the FLORES\-80 devtest dataset\.

We evaluate language identification with a linear probe on L2\-normalised FLORES word embeddings\. For each model, we train a singlenn\.Linear\(1024, 80\)classifier to predict the language from individual word embeddings, using the first 500 FLORES sentences per language; 10% of these sentences are held out for validation and the remaining saved sentences are used for test\. Training uses AdamW with learning rate 10e\-3, batch size 8192, weight decay 0 and early stopping after 50 epochs without validation macro\-F1 improvement\. Duplicate training words are removed per language\.

For isotropy and hubness, we sample 10,000 random pairs of different sentences in different languages\. For hubness, we usek=10k=10nearest neighbours\.

Figure 7:Word and span similarity for SONAR and SALT across layers\.Figure 8:Sequence tagging and word alignment performance for different training losses\. In the word alignment plot, we omit WordOT to improve visualisation \(WordOT is poorly aligned\)\.
### E\.3Qualitative analysis

We take the sentence

> Most insects have the advantage of being able to fold their wings back along the body\.

and perform word\-level retrieval \(top\-5 nearest neighbours\) across sentences in 80 languages \(see Table[9](https://arxiv.org/html/2609.09953#A5.T9),[10](https://arxiv.org/html/2609.09953#A5.T10),[11](https://arxiv.org/html/2609.09953#A5.T11)\) for the full FLORES\-200 devtest\.

Overall, all three encoders are generally able to retrieve semantically equivalent words in other languages from the same sentence\. However, SALT more frequently retrieves tokens in non\-Latin scripts and, more importantly, places true translations closer in terms of absolute cosine similarity\. This is notable given our earlier geometric analysis, in which we observed that SALT yields a lower average cosine similarity across the embedding space than the other encoders\.

Another improvement is that SALT reduces a common failure mode in which the model retrieves the same word in the same language from different sentential contexts\. This suggests that SALT better preserves contextual meaning, rather than relying on surface\-form similarity\.

MEXMA RankScoreLangUnit10\.945hun\_Latnlegtobb20\.943heb\_Hebrrov30\.943nno\_Latnfleste40\.942fra\_Latnplupart50\.942ukr\_Cyrlbilshosti

SALT RankScoreLangUnit10\.988ckb\_Arabzorbey20\.987mar\_Devabahutek30\.986arz\_Arabmu’zem40\.986npi\_Devaadhikansh50\.984bul\_Cyrlpovecheto

SONAR RankScoreLangUnit10\.976fra\_Latnplupart20\.971por\_Latnmaioria30\.971kor\_Hangdaebubun\-ui40\.966spa\_Latnmayoria50\.965cat\_Latnmajoria

Table 9:Nearest neighbours across models for the query wordmost\(all non\-Latin scripts transliterated\)\.MEXMA RankScoreLangUnit10\.950ron\_Latninsecte20\.940afr\_Latninsekte30\.940spa\_Latninsectos40\.939swe\_Latninsekter50\.939epo\_Latninsektoj

SALT RankScoreLangUnit10\.978afr\_Latninsekte20\.970arz\_Arabal\-hasharat \(al\-ḥasharāt\)30\.968slv\_Latninsektov40\.967mkd\_Cyrlinsekti50\.967slk\_Latnhmyzu

SONAR RankScoreLangUnit10\.970epo\_Latninsektoj20\.964fra\_Latninsectes30\.960nob\_Latninsekter40\.954afr\_Latninsekte50\.954deu\_LatnInsekten

Table 10:Nearest neighbours for the query wordinsect\(all non\-Latin scripts transliterated\)\.MEXMA RankScoreLangUnitSent\. idx10\.949eng\_Latnfold62120\.872tha\_Thaia61930\.873tha\_Thaiph61940\.873tha\_Thaib61950\.849tha\_Thaia621

SALT RankScoreLangUnitSent\. idx10\.956ind\_Latnmelipat61920\.946epo\_Latnfaldi61930\.945ron\_Latnplieze61940\.937vie\_Latngap61950\.936bul\_Cyrlsgavat619

SONAR RankScoreLangUnitSent\. idx10\.915271epo\_Latnfaldi61920\.901712kor\_Hangjeobeul61930\.874322heb\_Hebrrov61940\.874098eng\_Latnfold62150\.872106ind\_Latnmelipat619

Table 11:Nearest neighbours across models for the query wordfold\(all non\-Latin scripts transliterated\)\. Note that MEXMA and SONAR retrieve words from other sentences\.

## Appendix FStandard deviations

Table[12](https://arxiv.org/html/2609.09953#A6.T12)shows the standard deviation of our results from Table[2](https://arxiv.org/html/2609.09953#S4.T2)\.

AERPAN\-XMassiveUDPOSWiCSONAR Loss0\.0002550\.0001200\.0006220\.0007350\.000361SO Loss0\.0003180\.0003110\.0018600\.0004450\.000064WordOT0\.0022770\.0005160\.0016760\.0001560\.000672WACSE0\.0001270\.0011460\.0007140\.0001130\.001513OmniSONAR\-Token0\.0000680\.0005490\.0006300\.0009380\.002037SALT0\.0001270\.0004560\.0012100\.0006770\.000210Table 12:Standard deviation of our results in Table[2](https://arxiv.org/html/2609.09953#S4.T2)\(we run training experiments in three seeds\)\.
## Appendix GHuman evaluation of spans

For each of Chinese, French, Hindi, Russian, and Spanish, a native speaker annotated spans extracted from a total of 2,361 sentence pairs generated by either LLaMA\-70B or CASE \(with SONAR\), without being informed of the originating system\. The annotation split per language and system can be seen in Table[13](https://arxiv.org/html/2609.09953#A7.T13)\. Annotators were presented with pairs of spans and asked to assess their semantic equivalence using a 4\-point scale \(see Appendix[G\.2](https://arxiv.org/html/2609.09953#A7.SS2)\)\. Further details on the recruitment process are provided in Appendix[G\.3](https://arxiv.org/html/2609.09953#A7.SS3)\.

### G\.1Results on human evaluation of extracted aligned spans

Table[13](https://arxiv.org/html/2609.09953#A7.T13)shows the score distribution of spans following human evaluation\. We observe that, in almost all languages, spans of acceptable quality \(Rating 3 or 4\) occur at a rate higher than 90%\. These results validate using spans as a training signal\. We also see that SONAR performance with CASE closely resembles LLM annotation\.

LanguageSource\#AnnotationsRating 1Rating 2Rating 3Rating 4Agg\. 3\+4SpanishLLaMA\-70B1290\.0%3\.1%0\.8%96\.1%96\.9%SpanishSONAR \(CASE\)2490\.4%2\.0%5\.6%92\.0%97\.6%RussianLLaMA\-70B1020\.0%7\.8%20\.6%71\.6%92\.2%RussianSONAR \(CASE\)2043\.4%8\.3%21\.1%67\.2%88\.3%HindiLLaMA\-70B3343\.9%7\.2%15\.3%73\.7%89\.0%HindiSONAR \(CASE\)3340\.9%3\.3%19\.5%76\.3%95\.8%FrenchLLaMA\-70B1350\.0%5\.2%11\.1%83\.7%94\.8%FrenchSONAR \(CASE\)2640\.4%8\.0%21\.6%70\.1%91\.7%ChineseLLaMA\-70B2801\.8%5\.4%13\.2%79\.6%92\.8%ChineseSONAR \(CASE\)3301\.5%3\.6%12\.7%82\.1%94\.8%

Table 13:Score distribution for the human evaluation of spans\.Lang PairSourceen \#words \(avg\)en \#words \(median\)XX \#words \(avg\)XX \#words \(median\)English\-ChineseLLaMA\-70B2\.612\.04\.173\.5English\-ChineseSONAR \(CASE\)5\.703\.08\.425\.5English\-FrenchLLaMA\-70B2\.532\.03\.133\.0English\-FrenchSONAR \(CASE\)2\.582\.03\.222\.0English\-HindiLLaMA\-70B2\.722\.04\.984\.0English\-HindiSONAR \(CASE\)2\.822\.05\.704\.0English\-RussianLLaMA\-70B2\.432\.02\.302\.0English\-RussianSONAR \(CASE\)2\.632\.02\.442\.0English\-SpanishLLaMA\-70B2\.642\.03\.153\.0English\-SpanishSONAR \(CASE\)2\.432\.02\.592\.0All English\-XXLLaMA\-70B2\.622\.03\.973\.0All English\-XXSONAR \(CASE\)3\.362\.04\.833\.0

Table 14:Average and median span word length\.Lang PairSource1st POS1st POS freq2nd POS2nd POS freq3rd POS3rd POS freqEnglish\-ChineseLLaMA\-70BNOUN55\.4ADP31\.8VERB26\.8English\-ChineseSONAR \(CASE\)NOUN71\.2ADP51\.2VERB38\.5English\-FrenchLLaMA\-70BNOUN54\.1VERB31\.1ADP26\.7English\-FrenchSONAR \(CASE\)NOUN49\.6ADP25\.8VERB23\.9English\-HindiLLaMA\-70BNOUN48\.8ADP35\.9VERB29\.6English\-HindiSONAR \(CASE\)NOUN50\.0DET32\.6PROPN28\.4English\-RussianLLaMA\-70BNOUN47\.1VERB38\.2DET25\.5English\-RussianSONAR \(CASE\)NOUN56\.9DET27\.9PRON24\.0English\-SpanishLLaMA\-70BNOUN57\.4ADP29\.5ADJ27\.1English\-SpanishSONAR \(CASE\)NOUN50\.2VERB26\.9DET26\.1All English\-XXLLaMA\-70BNOUN52\.3ADP31\.1VERB29\.5All English\-XXSONAR \(CASE\)NOUN56\.0ADP31\.3DET29\.8

Table 15:Most common POS tags of English spans\.
### G\.2Full instructions given to annotators

Figure[10](https://arxiv.org/html/2609.09953#A10.F10)shows the annotation guidelines\.

### G\.3Recruitment and demographics

Annotators were recruited through internal communication channels within our institution\. All annotators were residing in the country where the institution is based at the time of the study\. The annotator pool comprised five individuals, each a native speaker of one of the following languages: Chinese, French, Hindi, Russian and Spanish\.

Prior to the annotation process, all annotators participated in a training session in which we introduced the annotation interface, explained the evaluation rubric in detail, and jointly reviewed a set of example instances to ensure consistency in interpretation\. Annotators were compensated at a rate of $20\.5 per hour\.

### G\.4Participant Information Sheet

Figure[11](https://arxiv.org/html/2609.09953#A10.F11)shows the Participant Information Sheet that was shared with the annotators\.

## Appendix HCompute and model size

SONAR has 766M parameters, and SALT introduces no additional parameters\. We developed SALT using two NVIDIA A100 GPUs \(80 GB each\)\. Final training runs were conducted on two NVIDIA H200 GPUs \(90 GB each\) and required approximately two and a half days\.

## Appendix IUse of artifacts

We use the SONAR encoder and the open\-source subset of the NLLB Primary dataset, which are distributed under permissive licenses \(Creative Commons Attribution Non\-Commercial 4\.0 and MIT License, respectively\)\. These resources are used in accordance with their intended purpose of advancing research on multilingual sentence representations\. All other encoders and datasets are used solely for evaluation, respecting their respective licenses and intended uses\.

We do not apply additional filtering for offensive content or personally identifiable information \(PII\), as such filtering was already performed by the original dataset creators\([NLLB Team, 2024](https://arxiv.org/html/2609.09953#bib.bib18)\)\.

## Appendix JUsage of AI tools

We acknowledge using AI tools for grammar correction and some other language clarifications, as well as code writing assistance\.

Prompt for LLM span alignmentYou are tasked with extracting aligned spans\. You will be given two sentences, and need to identify spans of words/particles from the first one that appear in the second\. Copy the spans of words as they appear originally in both texts\. Only copy the extracted aligned spans, following the format of Example 1\. If there are hallucinations or omissions, don’t include them in the aligned spans\.
Example 1:Sentence 1 \[eng\]: Samples of body fluids and tissues from people with the disease should be handled with special caution \.Sentence 2 \[spa\]: Las muestras de tejidos y fluidos corporales de personas con la enfermedad deben manejarse con especial precaución \.
Aligned spans:\[Samples\] \- \[Las muestras\]\[of body fluids\] \- \[de fluidos corporales\]\[and tissues\] \- \[y tejidos\]\[from people\] \- \[de personas\]\[with the disease\] \- \[con la enfermedad\]\[should be handled\] \- \[deben manejarse\]\[with special caution\] \- \[con especial precaución\]Now, extract the alignments for the following pair:Sentence 1 \[\{src\_lang\}\]: \{src\_text\}Sentence 2 \[\{tgt\_lang\}\]: \{tgt\_text\}Aligned spans:Figure 9:Prompt that was used on LLaMA\-70B for span alignment\.Table 16:Full list of languages used\.Train\(93 languages\)ace\_Arabace\_Latnafr\_Latnaka\_Latnamh\_Ethiarb\_Arabary\_Arabarz\_Arabbam\_Latnban\_Latnben\_Bengbho\_Devabjn\_Arabbjn\_Latnbos\_Latnbug\_Latnbul\_Cyrlcrh\_Latndik\_Latndzo\_Tibteng\_Latnewe\_Latnfon\_Latnfra\_Latnfur\_Latnfuv\_Latngaz\_Latngrn\_Latnguj\_Gujrhat\_Latnhin\_Devahne\_Devahrv\_Latnind\_Latnkan\_Kndakas\_Arabkas\_Devakhm\_Khmrkin\_Latnknc\_Arabknc\_Latnlij\_Latnlim\_Latnlin\_Latnlmo\_Latnltg\_Latnlug\_Latnmag\_Devamal\_Mlymmar\_Devamin\_Latnmkd\_Cyrlmni\_Bengmri\_Latnmya\_Mymrnno\_Latnnob\_Latnnus\_Latnory\_Oryapan\_Gurupbt\_Arabpes\_Arabpor\_Latnprs\_Arabrus\_Cyrlscn\_Latnshn\_Mymrslv\_Latnsom\_Latnsot\_Latnspa\_Latnsrd\_Latnsrp\_Cyrlssw\_Latnswh\_Latnszl\_Latntam\_Tamltaq\_Latntaq\_Tfngtel\_Telutgl\_Latntir\_Ethitsn\_Latntso\_Latntzm\_Tfngukr\_Cyrlurd\_Arabvec\_Latnxho\_Latnyor\_Latnzho\_Hanszsm\_Latnzul\_LatnTest\(57 languages\)afr\_Latnals\_Latnamh\_Ethiarb\_Arabazj\_Latnben\_Bengbul\_Cyrlcat\_Latncym\_Latndan\_Latndeu\_Latnell\_Grekeng\_Latnest\_Latneus\_Latnfin\_Latnfra\_Latnheb\_Hebrhin\_Devahun\_Latnhye\_Armnind\_Latnisl\_Latnita\_Latnjav\_Latnjpn\_Jpankan\_Kndakat\_Georkaz\_Cyrlkhk\_Cyrlkhm\_Khmrkor\_Hanglvs\_Latnmal\_Mlymmar\_Devamya\_Mymrnld\_Latnnob\_Latnpes\_Arabpol\_Latnpor\_Latnron\_Latnrus\_Cyrlslv\_Latnspa\_Latnswe\_Latnswh\_Latntam\_Tamltel\_Telutgl\_Latntha\_Thaitur\_Latnurd\_Arabvie\_Latnyor\_Latnzho\_Hanszsm\_LatnTest∖\\setminusTrain\(29 languages\)als\_Latnazj\_Latncat\_Latncym\_Latndan\_Latndeu\_Latnell\_Grekest\_Latneus\_Latnfin\_Latnheb\_Hebrhun\_Latnhye\_Armnisl\_Latnita\_Latnjav\_Latnjpn\_Jpankat\_Georkaz\_Cyrlkhk\_Cyrlkor\_Hanglvs\_Latnnld\_Latnpol\_Latnron\_Latnswe\_Latntha\_Thaitur\_Latnvie\_Latn

Annotation instructionsPurpose: evaluate the quality of automatically generated alignments between words or short phrases across different languages\.Why you were invited: you are fluent in Hindi, Russian, French, Chinese, or Spanish\.Participation: one remote session \(about 120 minutes\), voluntary, and withdrawal is possible up to two months after participation\.Data collected: Alignment judgments, response times, an anonymized annotator ID
Explanation of RatingsRating 1 \- Completely Misaligned

The spans are unrelated in meaning and do not correspond to each other\. There is no meaningful semantic overlap\.ExampleEnglish: “These symptoms have been reported and \[might\] contribute to its rapid spreading”Spanish: “Se han informado de esos síntomas, \[lo que\] podría contribuir a su rápida propagación”Literal translation: “These symptoms have been reported, \[what\] may contribute to its rapid spreading”Explanation: The spans \(“might” vs\. “lo que”\) do not align in meaning or function\.Rating 2 \- Partial Overlap, Core Meaning MismatchThe spans share some overlap, but fail to capture the main meaning or function of each other\.

Example 1English: “\[Since cases\] have so far clustered in certain regions in EU/EEA countries”Spanish: “\[Dado que, hasta el momento, se han formado conglomerados de casos\] en ciertas regiones de países de la UE/EEE”Literal translation: “\[Since so far there have been clusters of cases\] in certain regions of countries”Explanation: There is overlap, but the English span \(“Since cases”\) is much shorter and does not capture the full meaning expressed in Spanish\.Example 2English: “What \[are the implications\] for public health practice?”Spanish: “¿Qué \[consecuencias\] tiene esto para la práctica de la salud pública?”Literal translation: “What \[consequences\] this has for public health practice?”Explanation: The Spanish span omits an important part of the meaning \(the verbal structure “are / has”\), so the alignment is incomplete\.Rating 3 \- Same Role, Minor Differences

The spans fulfill the same semantic role and largely match in meaning, but there are small differences \(e\.g\., missing words, slight shifts in nuance, or grammatical variation\)\.Example 1English: “The virus is transmitted through respiratory droplets and \[through objects\]\.”Spanish: “El virus se transmite por medio de las gotitas de la respiración y \[los objetos\]\.”Literal translation: “The virus is transmitted through respiratory droplets and \[the objects\]\.”Explanation: The meaning is almost identical, but the Spanish span lacks the preposition “through”\.Example 2English: “Cabin crew should wear disposable medical gloves, \[and possibly additional\] personal protective equipment”Spanish: “La tripulación de cabina debe usar guantes quirúrgicos desechables, \[y, quizás,\] equipos de protección individual”Literal translation: “Cabin crew should wear disposable medical gloves, \[and maybe\] personal protective equipment”Explanation: The meaning is very similar, but “additional” is not expressed in Spanish\.Rating 4 \- Fully Aligned \(Equivalent Meaning

The spans have the same meaning with no important differences\. They could be swapped between sentences without changing the meaning\.Example 1English: “\[We are not aware\] of any increased risk”Spanish: “\[No tenemos conocimiento\] de un aumento de riesgo”Literal translation: “\[\(We\) do not have knowledge\] of any increase in risk”Example 2English: “Particularly \[avoid\] cruise ship travel\.”Spanish: “En especial, \[se deben evitar\] los viajes en cruceros\.”Literal translation: “In particular, \[they must be avoided\] cruise ship travel”Explanation: The spans are fully equivalent in meaning despite minor grammatical differences\.Figure 10:Instructions shared with the annotators\.Participant Information Sheet \(PIS\)What is the purpose of the study?The purpose of this study is to evaluate the quality of automatically generated alignments between words or short phrases across different languages \(for example, “the dog” – “el perro”\)\. These alignments are used in research on multilingual language technologies\. By asking native speakers to judge whether these alignments are correct, we aim to better understand how accurate these automatic methods are and improve techniques for learning cross\-lingual language representations used in natural language processing systems\.Why have I been asked to take part?You are a native speaker of the studied language \(Russian, French, Chinese, Hindi or Spanish\)\.Do I have to take part?No – participation in this study is entirely up to you\. You can withdraw from the study at any time, up to 2 months, without giving a reason\. After this point, it will no longer be possible to withdraw because we are not collecting any data that would allow us to identify you\.What will happen if I decide to take part?•If you decide to take part, you will complete a short annotation task in which you review pairs of words or short phrases from two different languages \(for example, “the dog” – “el perro”\)\. You will be asked to indicate whether the alignment between the spans appears correct\.•The data collected will consist only of your judgments about the alignments and an anonymised annotator ID\. The task will be completed through a simple online questionnaire or annotation interface\.•Participation will involve a single session lasting approximately 120 minutes\. The task can be completed remotely at a time convenient for you\.Compensation\.Participants will receive financial compensation at a minimum rate of $20\.5 per hour for their time\.Are there any risks associated with taking part?There are no significant risks associated with participation\.What data are you collecting about me?The data we collect for our research is completely anonymous: We are not collecting any information that could, in our assessment, allow anyone to identify you\. Your signed participant consent form will be kept separately from your responses and destroyed two months after your participation\.What will happen to the results of this study?The results of this study may be summarised in published articles, reports and presentations\. Your anonymised data may be published and can also be used for future research\.Who can I contact?If you have any further questions about the study, please contact the lead researcher, \[PLACEHOLDER\]\. If you wish to make a complaint about the study, please contact\. \[PLACEHOLDER\]\. When you contact us, please provide the study title and detail the nature of your complaint\.Alternative formats\.To request this document in an alternative format, such as large print or on coloured paper, please contact \[PLACEHOLDER\]\.Figure 11:Participant Information Sheet that was shared with the annotators\.Figure 12:Performance of SALT vs SONAR on the sequence tagging tasks PAN\-X and Massive, on the English setting\.Figure 13:Performance of SALT vs SONAR on the sequence tagging tasks UDPOS and WiC, on the English setting\.