Apples to Apples? Towards Comparable Crosslingual Language Model Evaluation

# Apples to Apples? Towards Comparable Crosslingual Language Model Evaluation
Source: [https://arxiv.org/html/2608.25089](https://arxiv.org/html/2608.25089)
Ethan Gotlieb WilcoxCatherine ArnettGeorgetown UniversityEleutherAIAffiliation:\{xy236,ethan\.wilcox\}@georgetown\.edu, catherine@eleuther\.ai

###### Abstract

Crosslingual evaluation of language models that enables fair comparisons remains a fundamental challenge in multilingual NLP\. Existing studies adopt a variety of downstream tasks and intrinsic metrics with different theoretical justifications, yet there has been little empirical investigation into whether these approaches yield meaningful crosslingual conclusions\. We systematically examine crosslingual evaluation approaches using controlled monolingual language models trained on parallel data with varying tokenizer vocabulary sizes and model sizes, and further validate our findings on multilingual LLMs\. We further discuss challenges in achieving comparable downstream evaluation across languages\. Our results show that several widely used normalized metrics introduce crosslinguistic biases rooted in tokenization, encoding, and orthographic differences\. In contrast, sentence\-level negative log\-likelihood computed over semantically equivalent sequences provides more meaningful and consistent crosslingual comparisons\.111Code:[https://github\.com/xiulinyang/multilingual\-eval\.git](https://github.com/xiulinyang/multilingual-eval.git); models:[https://huggingface\.co/parallelm](https://huggingface.co/parallelm)\. To load the tokenizer, please usePreTrainedTokenizerFastrather thanAutoTokenizer\.

## 1Introduction

Fairly evaluating language models across typologically diverse languages is a fundamental yet underappreciated challenge in multilingual NLP\. Despite significant progress in extending language technology to an ever\-growing number of languages for both monolingual\([Aravinda et al\., 2025](https://arxiv.org/html/2608.25089#bib.bib29);[Zosa et al\., 2025](https://arxiv.org/html/2608.25089#bib.bib33), e\.g\.,\)and multilingual models\([BigScience Workshop et al\., 2023](https://arxiv.org/html/2608.25089#bib.bib7), e\.g\.,\), substantial performance gaps across languages persist\([Chang et al\., 2023](https://arxiv.org/html/2608.25089#bib.bib34);[Joshi et al\., 2020](https://arxiv.org/html/2608.25089#bib.bib14);[Shani et al\., 2026](https://arxiv.org/html/2608.25089#bib.bib48)\)\. Yet accurately measuring these gaps itself is a challenge: different metrics can yield substantially different conclusions about crosslingual model performance\([Shani et al\., 2026](https://arxiv.org/html/2608.25089#bib.bib48)\), raising the question of what a*fair*comparison even means\.

Evaluation approaches in multilingual NLP broadly fall into two categories\. Downstream task evaluations assess a model across a range of linguistic competencies, but fair crosslingual comparison using such evaluations is far from straightforward: conclusions are highly task\-dependent\([Rust et al\., 2021](https://arxiv.org/html/2608.25089#bib.bib30);[Limisiewicz et al\., 2023](https://arxiv.org/html/2608.25089#bib.bib3)\), and language\-specific benchmarks may not exist for many languages\. Intrinsic metrics, by contrast, directly quantify how well a model assigns probability to held\-out parallel texts, offering a more feasible and controlled signal for crosslingual comparison\. Yet, as we discuss below, “more controlled” does not necessarily mean “fair”\.

Figure 1:Suitability of intrinsic metrics for cross\-tokenizer and cross\-lingual comparison \(✓= suitable;✗= unsuitable;✓\\checkmark= conditionally suitable\)\. Sentence NLL computed on semantically equivalent parallel content is the only metric suitable for both comparisons\.Within intrinsic evaluations, there is no consensus on what probability\-based metric to use\. One line of work evaluates the probability assigned to sequences that express the same meaning across languages, typically using negative log\-likelihood, surprisal, or related derived metrics\([Mielke et al\., 2019](https://arxiv.org/html/2608.25089#bib.bib46);[Cotterell et al\., 2018](https://arxiv.org/html/2608.25089#bib.bib23);[Wan, 2022](https://arxiv.org/html/2608.25089#bib.bib27), e\.g\.,\)\. Another common approach relies on unit\-normalized variants such as perplexity \(PPL;[Shliazhko et al\., 2024](https://arxiv.org/html/2608.25089#bib.bib25);[Arnett and Bergen, 2025](https://arxiv.org/html/2608.25089#bib.bib19);[Thakur et al\., 2025](https://arxiv.org/html/2608.25089#bib.bib12)\), bits\-per\-byte \(BPB;[Xue et al\., 2022](https://arxiv.org/html/2608.25089#bib.bib36)\), or bits\-per\-character \(BPC;[Blevins et al\., 2022](https://arxiv.org/html/2608.25089#bib.bib37)\), which normalize likelihood by tokens, bytes, or characters, respectively\. Despite the intuitive justification for each of these metrics \(e\.g\., BPB is tokenizer\-agnostic;[Gao et al\., 2020](https://arxiv.org/html/2608.25089#bib.bib1)\), empirical evidence examining these claims about which metric to use has remained limited\. Recently,[Poelman and de Lhoneux \(2026\)](https://arxiv.org/html/2608.25089#bib.bib15)argue that current evaluation metrics may not be universally comparable for two reasons: \(i\) different metrics can produce different rankings across languages, and \(ii\) even within a single metric, paraphrases expressing the same meaning can lead to inconsistent crosslingual rankings\. These findings leave open the broader question of what constitutes a fair crosslingual comparison\.

In this paper, we argue that a fair metric is one satisfying three criteria: first, it should benot systematically driven by engineering choices, specifically tokenization, encoding, and orthography\. That is, language rankings should reflect language modeling quality rather than representation\-level factors such as the number of characters, bytes, or tokens\. Second, it should berobust to variation across semantically equivalent forms: language rankings should remain stable across translation alternatives expressing the same content\. Third, it should bemodel\-general, meaning that its behavior should generalize across monolingual and multilingual models of different scales\.

To isolate metric behavior from confounds inherent in multilingual models, we first use 50 monolingual language models trained on parallel corpora with five vocabulary size settings\([Yang et al\., 2026](https://arxiv.org/html/2608.25089#bib.bib10)\)and conduct a systematic empirical comparison of six intrinsic metrics, then extend this comparison to larger monolingual and multilingual models\. We find that five out of the six metrics violate at least one criterion above\. By contrast, sentence\-level NLL, which normalizes by meaning rather than by surface\-level units such as morphemes or characters, satisfies all three: it is \(i\) substantially less sensitive to tokenization and encoding choices \(§[4](https://arxiv.org/html/2608.25089#S4)\), \(ii\) more stable across translation alternatives \(§[5](https://arxiv.org/html/2608.25089#S5)\), and \(iii\) applicable to both monolingual and multilingual settings across scales \(§[6](https://arxiv.org/html/2608.25089#S6)\)\. Our results challenge both common practice and recent proposals in multilingual evaluation literature\. We show that both token\-normalized metrics such as perplexity and character\- and byte\-level alternatives, which have been explicitly advocated as more robust\([Shani et al\., 2026](https://arxiv.org/html/2608.25089#bib.bib48)\), introduce substantial and systematic biases that favor languages with longer sequences or more tokens\. At the same time, we empirically validate the theoretically motivated assumption underlying prior work that uses sentence\-level NLL or related transformed metrics\([Mielke et al\., 2019](https://arxiv.org/html/2608.25089#bib.bib46);[Cotterell et al\., 2018](https://arxiv.org/html/2608.25089#bib.bib23);[Chang et al\., 2023](https://arxiv.org/html/2608.25089#bib.bib34), e\.g\.,\), providing evidence for claims that had previously lacked direct empirical support\. We further discuss considerations for downstream evaluation, noting the challenges of achieving comparable assessment across languages and tasks\.

## 2Background & Related Work

### 2\.1Tokenization & Representation Biases

For crosslinguistic evaluation to be meaningful, it is necessary to disentangle intrinsic linguistic complexity from technical artifacts introduced by representation and preprocessing choices\([Shani et al\., 2026](https://arxiv.org/html/2608.25089#bib.bib48)\)\. Prior work has identified several sources of such artifacts, particularly those arising from tokenization and orthographic encoding\([Petrov et al\., 2023](https://arxiv.org/html/2608.25089#bib.bib44);[Ahia et al\., 2024](https://arxiv.org/html/2608.25089#bib.bib4), e\.g\.,\)\.

#### Tokenization

Language models are trained to predict the next token in a sequence, from a vocabulary of discrete tokens\. Consequently, how text is segmented into tokens directly affects how the model represents and learns linguistic structure\. Semantically equivalent content across languages can be segmented into very different numbers of tokens, even when the same tokenization algorithm is used\([Petrov et al\., 2023](https://arxiv.org/html/2608.25089#bib.bib44);[Arnett et al\., 2025](https://arxiv.org/html/2608.25089#bib.bib54)\)\. This disparity is often quantified withCorpus Token Count\(CTC;[Schmidt et al\., 2024](https://arxiv.org/html/2608.25089#bib.bib24)\), which measures the total number of tokens required to encode a corpus under a given tokenizer\.

Differences in token segmentation can influence both training dynamics and evaluation metrics\. Even under identical tokenization settings, languages may be split into substantially different numbers of segments, which can affect model performance\([Shani et al\., 2026](https://arxiv.org/html/2608.25089#bib.bib48)\)\. As a result, different languages may benefit from different tokenization strategies\([Fujii et al\., 2023](https://arxiv.org/html/2608.25089#bib.bib50);[Vemula et al\., 2025](https://arxiv.org/html/2608.25089#bib.bib53);[Reddy et al\., 2025](https://arxiv.org/html/2608.25089#bib.bib49);[Toraman et al\., 2023](https://arxiv.org/html/2608.25089#bib.bib47), e\.g\.,\)\. For instance,[Toraman et al\. \(2023\)](https://arxiv.org/html/2608.25089#bib.bib47)show that morphologically\-rich languages like Turkish require a larger vocab size to train and fine\-tune encoder models\.

#### Orthography and Encoding

Representation disparities also arise from differences in orthographic encoding\. The same semantic content written in different scripts may require different numbers of UTF\-8 bytes, a phenomenon known as thebyte premium\([Arnett et al\., 2024](https://arxiv.org/html/2608.25089#bib.bib45)\)\. Similarly, different writing systems may encode equivalent content using substantially different numbers of characters, resulting in what has been termed thelength premium\([Arnett et al\., 2024](https://arxiv.org/html/2608.25089#bib.bib45)\)\. These representational differences are independent of linguistic meaning but can systematically distort evaluation metrics that normalize by bytes or characters\.

### 2\.2Crosslinguistic Evaluation

The most widely adopted approach relies on negative log\-likelihood \(NLL\) computed over semantically equivalent sequences across languages\. Within this framework, studies differ in how they normalize the raw NLL scores\.[Cotterell et al\. \(2018\)](https://arxiv.org/html/2608.25089#bib.bib23)and[Mielke et al\. \(2019\)](https://arxiv.org/html/2608.25089#bib.bib46)normalize sentence\-level surprisals by the number of characters of a specific language \(usually English\), using the resulting scores to rank languages by difficulty\.[Limisiewicz et al\. \(2024\)](https://arxiv.org/html/2608.25089#bib.bib35)adopts a similar approach but normalizes by the number of bytes instead\.[Wan \(2022\)](https://arxiv.org/html/2608.25089#bib.bib27)simply aggregates sentence\-level NLL over an entire development set without normalization\. A more structured alternative is proposed by[Tsvetkov and Kipnis \(2024\)](https://arxiv.org/html/2608.25089#bib.bib26), who introduce Information Parity \(IP\), defined as the ratio of English NLL to the NLL of another language over parallel text\.

Other work evaluates multilingual models using metrics that operate at different granularities\.[Shliazhko et al\. \(2024\)](https://arxiv.org/html/2608.25089#bib.bib25)report perplexity scores across languages, while[Blevins et al\. \(2022\)](https://arxiv.org/html/2608.25089#bib.bib37)argue for bits per character \(BPC\) as a more language\-neutral unit\. Byte\-level language models further shift toward bits per byte \(BPB;[Xue et al\., 2022](https://arxiv.org/html/2608.25089#bib.bib36);[Zhang and Xu, 2022](https://arxiv.org/html/2608.25089#bib.bib80)\), which[Shani et al\. \(2026\)](https://arxiv.org/html/2608.25089#bib.bib48)suggest may yield fairer comparisons when scores are scaled to a shared unit\. Despite the apparent justification for each of these metrics,[Poelman and de Lhoneux \(2026\)](https://arxiv.org/html/2608.25089#bib.bib15)demonstrate that applying different metrics to the same model on the same data can produce conflicting language rankings\. This highlights that metric choice is not merely a technical detail but a substantive decision with interpretive consequences\. This work offers concrete empirical guidance for metric selection in crosslingual evaluation and contributes to ongoing efforts toward fair multilingual NLP\.

## 3Existing Intrinsic Metrics & Their Potential Confounds

In this section, we review mainstream probability\-based metrics used in language modeling and evaluate their suitability for crosslinguistic and cross\-model comparison\. Lett​\\xspacei​\\xspace\{\\color\[rgb\]\{0\.06,0\.46,1\}t\}\\xspace\_\{\{\\color\[rgb\]\{0\.06,0\.46,1\}i\}\\xspace\}\\xspacebe thei​\\xspacet​h\{\\color\[rgb\]\{0\.06,0\.46,1\}i\}\\xspace^\{th\}token in sequence𝐬\\mathbf\{s\}\\xspacewhich hasbb\\xspaceUTF\-8 bytes,cc\\xspacecharacters, and can be tokenized intonn\\xspacetokens\.

#### Bits per Byte

BPB is proposed by[Gao et al\. \(2020\)](https://arxiv.org/html/2608.25089#bib.bib1)because of “its invariance to different tokenization schemes and the ambiguity of measuring characters in Unicode\.”

BPB\(𝐬\\xspace\)=−1b​\\xspace∑i=1n​\\xspacelog2\(P\(t\\xspacei​\\xspace\\xspace∣t\\xspace<i​\\xspace\\xspace\)\)\\mathrm\{BPB\}\(\{\\color\[rgb\]\{0,0,0\}\\mathbf\{s\}\}\\xspace\)=\{\-\\frac\{1\}\{\{\\color\[rgb\]\{1,0\.5,0\}b\}\\xspace\}\\sum\_\{i=1\}^\{\{\\color\[rgb\]\{0\.9063,0\.1602,0\.543\}n\}\\xspace\}\\log\_\{2\}\(P\(\{\\color\[rgb\]\{0\.06,0\.46,1\}\{\\color\[rgb\]\{0\.06,0\.46,1\}t\}\\xspace\_\{\{\\color\[rgb\]\{0\.06,0\.46,1\}i\}\\xspace\}\}\\xspace\\mid\{\\color\[rgb\]\{0\.06,0\.46,1\}\{\\color\[rgb\]\{0\.06,0\.46,1\}\\textbf\{t\}\}\\xspace\_\{<\{\\color\[rgb\]\{0\.06,0\.46,1\}i\}\\xspace\}\}\\xspace\)\)\}\(1\)
It has been used in cross\-model evaluations\([Rae et al\., 2021](https://arxiv.org/html/2608.25089#bib.bib20), e\.g\.,\)for the same language\. However, we argue that BPB introduces bias when comparing across languages due to differences in orthographic encoding\. The same semantic content written in different scripts may require different numbers of UTF bytes \(i\.e\., differentbyte premiums\)\. For example,[Arnett et al\. \(2024\)](https://arxiv.org/html/2608.25089#bib.bib45)report that characters in scripts such as Khmer typically require three bytes per character \(excluding diacritics\), whereas Latin characters require only one byte\. As a result, if two models assign similar probabilities to semantically equivalent sequences in two languages, the language with the higher byte premium will necessarily obtain a lower BPB score because the total surprisal∑i=1n​\\xspacelog2⁡\(P⁡\(t​\\xspacei​\\xspace​\\xspace∣t\\xspace<i​\\xspace​\\xspace\)\)\\sum\_\{i=1\}^\{\{\\color\[rgb\]\{0\.9063,0\.1602,0\.543\}n\}\\xspace\}\\log\_\{2\}\(P\(\{\\color\[rgb\]\{0\.06,0\.46,1\}\{\\color\[rgb\]\{0\.06,0\.46,1\}t\}\\xspace\_\{\{\\color\[rgb\]\{0\.06,0\.46,1\}i\}\\xspace\}\}\\xspace\\mid\{\\color\[rgb\]\{0\.06,0\.46,1\}\{\\color\[rgb\]\{0\.06,0\.46,1\}\\textbf\{t\}\}\\xspace\_\{<\{\\color\[rgb\]\{0\.06,0\.46,1\}i\}\\xspace\}\}\\xspace\)\)is normalized by a larger number of bytes,bb\\xspace\.

#### Bits per Character

Bits per Character \(BPC\) measures the average sequence NLL per character:

BPC\(𝐬\\xspace\)=−1c​\\xspace∑i=1n​\\xspacelog2\(P\(t\\xspacei​\\xspace\\xspace∣t\\xspace<i​\\xspace\\xspace\)\)\\mathrm\{BPC\}\(\{\\color\[rgb\]\{0,0,0\}\\mathbf\{s\}\}\\xspace\)=\{\-\\frac\{1\}\{\{\\color\[rgb\]\{0,0\.88,0\}c\}\\xspace\}\\sum\_\{i=1\}^\{\{\\color\[rgb\]\{0\.9063,0\.1602,0\.543\}n\}\\xspace\}\\log\_\{2\}\(P\(\{\\color\[rgb\]\{0\.06,0\.46,1\}\{\\color\[rgb\]\{0\.06,0\.46,1\}t\}\\xspace\_\{\{\\color\[rgb\]\{0\.06,0\.46,1\}i\}\\xspace\}\}\\xspace\\mid\{\\color\[rgb\]\{0\.06,0\.46,1\}\{\\color\[rgb\]\{0\.06,0\.46,1\}\\textbf\{t\}\}\\xspace\_\{<\{\\color\[rgb\]\{0\.06,0\.46,1\}i\}\\xspace\}\}\\xspace\)\)\}\(2\)
It is mainly used in evaluations of character\-level models\([Al\-Rfou et al\., 2019](https://arxiv.org/html/2608.25089#bib.bib18), e\.g\.,\)\. While it mitigates tokenization bias, it may still introduce orthographic bias like BPB\. For example, Chinese translations are usually represented with fewer characters than English and thus it has a lower length premium\. Normalizing over the number of characters means that Chinese will almost always have a higher BPC than English, ascc\\xspaceis smaller\.

#### Mean Reciprocal Rank

MRR\([Limisiewicz et al\., 2023](https://arxiv.org/html/2608.25089#bib.bib3)\)evaluates models on the rank of the next token rather than its probability, making it probability\-agnostic\. However, since ranking is computed over the entire vocabulary, a larger vocabulary raises the upper bound of possible ranks, such that models with larger vocabularies tend to achieve lower MRR regardless of actual performance\.

MRR⁡\(𝐬​\\xspace\)=1n​\\xspace​∑t=1n​\\xspace1rank\(t\\xspacei​\\xspace\\xspace,P\(⋅∣t\\xspace<i​\\xspace\\xspace\)\)\\mathrm\{MRR\}\(\{\\color\[rgb\]\{0,0,0\}\\mathbf\{s\}\}\\xspace\)=\\frac\{1\}\{\{\\color\[rgb\]\{0\.9063,0\.1602,0\.543\}n\}\\xspace\}\\sum\_\{t=1\}^\{\{\\color\[rgb\]\{0\.9063,0\.1602,0\.543\}n\}\\xspace\}\\frac\{1\}\{\\mathrm\{rank\}\(\{\\color\[rgb\]\{0\.06,0\.46,1\}\{\\color\[rgb\]\{0\.06,0\.46,1\}t\}\\xspace\_\{\{\\color\[rgb\]\{0\.06,0\.46,1\}i\}\\xspace\}\}\\xspace,P\(\\cdot\\mid\{\\color\[rgb\]\{0\.06,0\.46,1\}\{\\color\[rgb\]\{0\.06,0\.46,1\}\\textbf\{t\}\}\\xspace\_\{<\{\\color\[rgb\]\{0\.06,0\.46,1\}i\}\\xspace\}\}\\xspace\)\)\}\(3\)

#### Perplexity

Held\-out perplexity measures the average number of nats \(or bits\) required to predict tokens in a corpus\. It has been the default metric for evaluating language models, with lower values typically indicating better performance\([Brown et al\., 2020](https://arxiv.org/html/2608.25089#bib.bib16), e\.g\.,\):

PPL\(𝐬\\xspace\)=exp\(−1n​\\xspace∑i=1n​\\xspaceln\(P\(t\\xspacei​\\xspace\\xspace∣t\\xspace<i​\\xspace\\xspace\)\)\)\\mathrm\{PPL\}\(\{\\color\[rgb\]\{0,0,0\}\\mathbf\{s\}\}\\xspace\)=\\mathrm\{exp\}\(\{\-\\frac\{1\}\{\{\\color\[rgb\]\{0\.9063,0\.1602,0\.543\}n\}\\xspace\}\\sum\_\{i=1\}^\{\{\\color\[rgb\]\{0\.9063,0\.1602,0\.543\}n\}\\xspace\}\\ln\(P\(\{\\color\[rgb\]\{0\.06,0\.46,1\}\{\\color\[rgb\]\{0\.06,0\.46,1\}t\}\\xspace\_\{\{\\color\[rgb\]\{0\.06,0\.46,1\}i\}\\xspace\}\}\\xspace\\mid\{\\color\[rgb\]\{0\.06,0\.46,1\}\{\\color\[rgb\]\{0\.06,0\.46,1\}\\textbf\{t\}\}\\xspace\_\{<\{\\color\[rgb\]\{0\.06,0\.46,1\}i\}\\xspace\}\}\\xspace\)\)\}\)\(4\)
Some previous studies report perplexity as the measure to compare models with different tokenizers and training data\([Thakur et al\., 2025](https://arxiv.org/html/2608.25089#bib.bib12), e\.g\.,\)\. Because perplexity is normalized by the number of tokens, it is highly sensitive to how the input is tokenized\. Consider a language model trained on LanguageAand evaluated on a parallel corpus containing both LanguageAand LanguageB, where the two languages use different scripts\. Since the model has never been exposed to LanguageB’s script, its tokenizer will decompose LanguageB’s text into smaller units \(down to bytes under standard BPE\) and produce longer token sequences\. As the denominator increases, the normalized log\-likelihood decreases, artificially deflating perplexity for LanguageBdespite the model having no genuine knowledge of it\.222We present a case study usingGPT2\([Radford et al\., 2019](https://arxiv.org/html/2608.25089#bib.bib72)\)in Appendix[E](https://arxiv.org/html/2608.25089#A5)to illustrate this point\.

#### Token\-level Negative log\-likelihood

Token\-level NLL \(Token NLL\) measures the NLL of the sequence normalized by the number of tokens:

Token NLL\(𝐬\\xspace\)=−1n​\\xspace∑i=1n​\\xspaceln\(P\(t\\xspacei​\\xspace\\xspace∣t\\xspace<i​\\xspace\\xspace\)\)\\text\{Token NLL\}\(\{\\color\[rgb\]\{0,0,0\}\\mathbf\{s\}\}\\xspace\)=\{\-\\frac\{1\}\{\{\\color\[rgb\]\{0\.9063,0\.1602,0\.543\}n\}\\xspace\}\\sum\_\{i=1\}^\{\{\\color\[rgb\]\{0\.9063,0\.1602,0\.543\}n\}\\xspace\}\\ln\(P\(\{\\color\[rgb\]\{0\.06,0\.46,1\}\{\\color\[rgb\]\{0\.06,0\.46,1\}t\}\\xspace\_\{\{\\color\[rgb\]\{0\.06,0\.46,1\}i\}\\xspace\}\}\\xspace\\mid\{\\color\[rgb\]\{0\.06,0\.46,1\}\{\\color\[rgb\]\{0\.06,0\.46,1\}\\textbf\{t\}\}\\xspace\_\{<\{\\color\[rgb\]\{0\.06,0\.46,1\}i\}\\xspace\}\}\\xspace\)\)\}\(5\)
Token NLL shares the same tokenization sensitivity as perplexity, as discussed above\.

#### Sentence\-level Negative log\-likelihood

Sentence\-level NLL \(Sent\-NLL\) is the sum of the NLL \(surprisal\) per token over a text sequence \(See[6](https://arxiv.org/html/2608.25089#S3.E6)\)\. Note that the sequence need not be a linguistic sentence; it can be any unit of parallel text that conveys equivalent meaning across languages, such as a paragraph or document\.

Sent\-NLL\(𝐬\\xspace\)=−∑i=1n​\\xspaceln\(P\(t\\xspacei​\\xspace\\xspace∣t\\xspace<i​\\xspace\\xspace\)\)\\text\{Sent\-NLL\}\(\{\\color\[rgb\]\{0,0,0\}\\mathbf\{s\}\}\\xspace\)=\{\-\\sum\_\{i=1\}^\{\{\\color\[rgb\]\{0\.9063,0\.1602,0\.543\}n\}\\xspace\}\\ln\(P\(\{\\color\[rgb\]\{0\.06,0\.46,1\}\{\\color\[rgb\]\{0\.06,0\.46,1\}t\}\\xspace\_\{\{\\color\[rgb\]\{0\.06,0\.46,1\}i\}\\xspace\}\}\\xspace\\mid\{\\color\[rgb\]\{0\.06,0\.46,1\}\{\\color\[rgb\]\{0\.06,0\.46,1\}\\textbf\{t\}\}\\xspace\_\{<\{\\color\[rgb\]\{0\.06,0\.46,1\}i\}\\xspace\}\}\\xspace\)\)\}\(6\)
Because Sent\-NLL sums the surprisal over the entire sequence without normalizing by token count, it measures the model’s surprisal for semantically equivalent content expressed in different languages\. Therefore, it is not directly affected by how a sequence is segmented\. For this reason, Sent\-NLL of parallel sentences has been more widely adopted in crosslinguistic comparisons\([Lin et al\., 2024](https://arxiv.org/html/2608.25089#bib.bib22);[Costa\-Jussà et al\., 2022](https://arxiv.org/html/2608.25089#bib.bib21);[Chang et al\., 2026](https://arxiv.org/html/2608.25089#bib.bib17)\), and we argue that it provides the most meaningful basis for crosslingual evaluation among the metrics considered here\.

#### Other metrics

There are other metrics such as Information Parity\([Tsvetkov and Kipnis, 2024](https://arxiv.org/html/2608.25089#bib.bib26), IP; \)and Bits per English Character\([Cotterell et al\., 2018](https://arxiv.org/html/2608.25089#bib.bib23), BPEC;\)\. As they are transformations of Sent\-NLL covered here, they should align with the results of Sent\-NLL\.

![Refer to caption](https://arxiv.org/html/2608.25089v1/heatmap_ranks_flores.png)Figure 2:Language model rankings across intrinsic metrics\. Each cell shows the rank of a language\-vocabulary size combination among all 50 settings \(10 languages \* 5 vocabulary sizes\) within each metric\. Lighter colors indicate better performance \(rank 1 = best\)\.
#### Summary

As summarized in Figure[1](https://arxiv.org/html/2608.25089#S1.F1), normalization over tokens, characters, or bytes introduces systematic biases and is unsuitable for cross\-tokenizer or cross\-lingual comparison\. BPB and BPC are appropriate for cross\-tokenizer comparison within the same language, but not cross\-lingually, as different writing systems yield different byte and character counts\. Perplexity and Token\-NLL are unsuitable for cross\-lingual or cross\-tokenizer comparisons because different tokenizers can segment the same content into different numbers of tokens across languages\. MRR requires constant vocabulary size across models\. Only Sent\-NLL over semantically equivalent content is suitable for both cross\-model and cross\-lingual comparison, as it normalizes by meaning rather than representation units\.

## 4Experiment 1: All metrics except for Sent\-NLL are biased in different ways

To ensure that observed metric differences reflect the metrics themselves rather than model\-specific confounds,333For example, a multilingual model trained predominantly on Chinese may assign lower BPC to Chinese text, masking BPC’s inherent bias against logographic writing systems: Chinese characters encode more information per character than alphabetic scripts, artificially shortening sentences and improving BPC scores\.we use monolingual models trained on parallel corpora with varying vocabulary sizes\.

#### Experiment Setup\.

We useGPT2models from concurrent work\([Yang et al\., 2026](https://arxiv.org/html/2608.25089#bib.bib10)\): monolingual models, each trained exclusively on one language’s portion of a parallel corpus\. This design ensures that cross\-linguistic differences are attributable to the languages themselves rather than to shared parameters or training data imbalances\.

Two model sizes are used\.GPT2\-smallmodels are trained onParallel\-10, a ten\-language parallel corpus spanning six language families \(∼\\sim20M words per language\),444Arabic, Chinese, English, French, German, Finnish, Polish, Russian, Korean, and Turkish\.with five vocabulary sizes,5558192, 32768, 49152, 65536, 81920\.yielding 50 models in total\.GPT2\-mediummodels are trained onParallel\-3, a trilingual corpus \(Chinese, English, Arabic;∼\\sim400M words per language\) with two vocabulary sizes \(32k and 65k\), yielding 6 models, allowing us to test whether findings hold at scale\. The details of the training corpora can be found in[AppendixA](https://arxiv.org/html/2608.25089#A1)\.

Table 1:Correlation Analysis between metrics and potential confounds\. PPL is scaled by log10\.
#### Evaluation\.

We evaluate the models using the metrics discussed in Section[3](https://arxiv.org/html/2608.25089#S3)on three parallel data sources in the corresponding languages: FLORES\-200\([NLLB Team et al\., 2022](https://arxiv.org/html/2608.25089#bib.bib57)\), a multi\-parallel machine translation benchmark covering more than 200 languages; the in\-domain test split of our training corpus \(40k sentences, see Table[3](https://arxiv.org/html/2608.25089#A1.T3)\); and 1,000 sentences from the Parallel Universal Dependencies treebanks\([Zeman et al\., 2017](https://arxiv.org/html/2608.25089#bib.bib79), PUD;\)\.

A fair metric should be less sensitive to encoding and tokenization choices\. Based on the discussion in[Section3](https://arxiv.org/html/2608.25089#S3), we expect language rankings based on BPB and BPC to be largely predictable from the number of bytes and characters, respectively, and rankings based on Token\-NLL, PPL, and MRR to be largely predictable from the number of tokens \(i\.e\., CTC\)\. In contrast, we do not expect language rankings based on Sent\-NLL to exhibit such relationships\. We examine these predictions in Figure[2](https://arxiv.org/html/2608.25089#S3.F2)and quantify them using Spearman’s rank correlation\.

#### Results

Figure[2](https://arxiv.org/html/2608.25089#S3.F2)reports rankings across 10 languages and 5 vocabulary sizes on FLORES\-200 forGPT2\-small\(results on other datasets are in Appendix[C\.1](https://arxiv.org/html/2608.25089#A3.SS1)\)\. Results forGPT2\-mediumcan be found in[SectionC\.2](https://arxiv.org/html/2608.25089#A3.SS2)\. Different metrics produce strikingly different crosslingual rankings\.[Poelman and de Lhoneux \(2026\)](https://arxiv.org/html/2608.25089#bib.bib15)attribute this to the general unreliability of probability\-based metrics\. While this may be true, in part, we argue that the inconsistencies are attributable to specific, identifiable biases in the normalization unit of each metric\.

We observe the expected patterns in Figure[2](https://arxiv.org/html/2608.25089#S3.F2)\. BPB and BPC consistently rank Chinese last, as Chinese has shorter character and byte sequences\. MRR, PPL, and Token NLL rankings are dominated by vocabulary size: smaller vocabularies yield better rankings regardless of language\. This ranking arises for different reasons\. For PPL and Token NLL, a tokenizer with a smaller vocabulary size produces more tokens \(i\.e\., a higher CTC\)\. As these metrics are averaged over tokens, a higher CTC can result in lower PPL and Token NLL \(i\.e\., better performance\), even when the underlying modeling quality is unchanged\. For MRR, a smaller vocabulary size reduces the upper bound of possible ranks, which can make models with smaller vocabulary sizes appear to perform better\. The same pattern holds in our scaling experiments\. These results suggest that rankings under surface\-normalized metrics are largely predictable from engineering choices alone, rather than reflecting genuine differences in language modeling quality\. In contrast, Sent\-NLL shows less sensitivity to either encoding or vocabulary size\.

Figure 3:Each row shows results for a different sample size\. Error bars represent 95% CI across 15 random seeds\.To substantiate this claim, we conduct Spearman’s rank correlation tests between each metric and the corresponding potential bias factors discussed in Section[3](https://arxiv.org/html/2608.25089#S3)\. For BPC and BPB, the number of characters and bytes for a given language remains constant regardless of tokenization, so we average model performance across tokenizers before computing correlations\. For other metrics, the potential confound is vocabulary size, which we operationalize as CTC and use in our analysis\.666Because for a given corpus tokenized by the same tokenization algorithm, vocabulary size and CTC are in a one\-to\-one correspondence, we use CTC for consistency in our following analysis\.As CTC varies across tokenization settings because different tokenizers produce different numbers of tokens for the same language, we therefore compute correlations between each model’s metric value and its corresponding CTC directly, without averaging across tokenizers\. For Sent\-NLL, we compute correlations with all three factors\.

Both the correlation coefficientρ\\rhoand p\-valueppof the correlation analysis are reported in Table[1](https://arxiv.org/html/2608.25089#S4.T1)\. We find that the potential confounds are strongly and significantly correlated with language rankings for all metrics \(ρ\\rho\>0\.8,pp<0\.01\) except Sent\-NLL, which shows no significant correlation with sequence length measured in bytes, characters, or tokens\.

## 5Experiment 2: Sent\-NLL is more robust for translation alternatives

One potential objection to Sent\-NLL is that different translations of the same text may yield different probability distributions, leading to inconsistent crosslingual rankings\([Poelman and de Lhoneux, 2026](https://arxiv.org/html/2608.25089#bib.bib15)\)\. However,[Poelman and de Lhoneux \(2026\)](https://arxiv.org/html/2608.25089#bib.bib15)demonstrate this instability using both translations and paraphrases within the same language rather than translations only\. Because paraphrases allow more flexible wording as long as the gist of the meaning is kept\([Neusner, 1986](https://arxiv.org/html/2608.25089#bib.bib75);[Bhagat and Hovy, 2013](https://arxiv.org/html/2608.25089#bib.bib2)\), they are more likely to encode different amounts of information\. In this case, their findings speak to within\-language variation rather than the validity of crosslingual comparison over parallel translations\. We provide additional discussion about this assumption in Section[D](https://arxiv.org/html/2608.25089#A4)\.

#### Experiment Setup\.

To test each metric’s sensitivity to meaning\-equivalent translations, we re\-evaluate[Poelman and de Lhoneux](https://arxiv.org/html/2608.25089#bib.bib15)’s \([2026](https://arxiv.org/html/2608.25089#bib.bib15)\) experiment using translations only and additionally evaluate Sent\-NLL\. Specifically, we use the English\-German translation pairs from WMT2019\([Barrault et al\., 2019](https://arxiv.org/html/2608.25089#bib.bib38)\)with an additional translation reference source created independently from[Freitag et al\. \(2020a\)](https://arxiv.org/html/2608.25089#bib.bib32);[Freitag et al\. \(2020b\)](https://arxiv.org/html/2608.25089#bib.bib13), denoted as DE11and DE22separately\. Following[Poelman and de Lhoneux \(2026\)](https://arxiv.org/html/2608.25089#bib.bib15), for these two alternative German translations, we construct two derived datasets: DElowestand DEhighest, consisting of sentences that receive the lower or higher metric value in each pairwise comparison, respectively\.

We use the same sets of models trained on English and German in Experiment 1 and compare Sent\-NLL for English and German across the five data sources: EN \(English source text\), DE11\(German translation\), DE22\(German alternative translation\), DElowest, and DEhighest\.

A fair metric should be robust to translation alternatives\. That is, for a given source text, evaluating models on different translations should yield consistent language rankings\. Based on the analysis in[Section3](https://arxiv.org/html/2608.25089#S3), we predict that Sent\-NLL will yield consistent rankings across translation alternatives, whereas the other metrics will be more sensitive to translation variation\.

One possible objection is that even for semantically equivalent translations, sentence\-level rankings may not align across languages, potentially affecting aggregated language rankings\([Poelman et al\., 2025](https://arxiv.org/html/2608.25089#bib.bib52)\)\. We argue that such sentence\-level mismatches introduce noise, and that their impact diminishes when averaged over a sufficiently large number of parallel sentences\. To test this, we conduct a follow\-up experiment under the same setting while varying the number of samples from 1 to the full dataset\. For each sample size, we repeat the experiment with 15 random seeds\.

#### Results\.

Figure[3](https://arxiv.org/html/2608.25089#S4.F3)shows the results of the replication experiment \(last row\) and the sampling experiment \(first three rows\)\.

The replication experiment reveals three major findings\. First, metrics including MRR, PPL, and Token NLL vary monotonically by vocabulary size, which is consistent with our findings in Section[4](https://arxiv.org/html/2608.25089#S4)\. Second, for these metrics, even with the same vocab size setting, using different translations in German yields different language rankings between English and German\. This pattern is especially salient when vocab size≥\\geq33k\. This suggests that these metrics are more sensitive to surface\-level differences\. Finally, we find that Sent\-NLL produces consistent language rankings across translation variants \(i\.e\., English scores lower than all four German datasets\)\. We observe the same consistency for BPB and BPC, but with different rankings\. However, as we show in Section[4](https://arxiv.org/html/2608.25089#S4), this apparent stability is driven in part by language\-specific encoding factors\.

Figure 4:Expected ranking and actual ranking across 6 metrics forBLOOM\-7b1andBLOOM\-1b1\.Regarding the sampling experiment, we find that when the sample size is small \(1–100\), the rankings are indeed unstable\. However, they become stable once the sample size reaches around 500\. We note that all other experiments in this paper use sample sizes greater than 500\.

## 6Experiment 3: Sent\-NLL generalizes to multilingual language models

The metric biases we identify are mathematical in nature and therefore model\-agnostic: they arise from how metrics normalize probability, not from properties of any particular model\.

#### Experiment Setup\.

To examine this, we evaluate nine multilingual LLMs on four languages \(Arabic, English, Russian, Chinese\) from FLORES\-200\. We select two groups of models based on different criteria\. First,BLOOM\([BigScience Workshop et al\., 2023](https://arxiv.org/html/2608.25089#bib.bib7)\),mGPT\([Shliazhko et al\., 2024](https://arxiv.org/html/2608.25089#bib.bib25)\), andXGLM\([Lin et al\., 2022](https://arxiv.org/html/2608.25089#bib.bib5)\)are selected because their training data composition is well\-documented, allowing us to derive expected language rankings\. Second,Teuken\([Ali et al\., 2024](https://arxiv.org/html/2608.25089#bib.bib6)\),Qwen\([Yang et al\., 2025](https://arxiv.org/html/2608.25089#bib.bib9)\), andLlama\([Grattafiori et al\., 2024](https://arxiv.org/html/2608.25089#bib.bib8)\)are included due to their wide usage in multilingual settings\. Training data composition is reported in Table[7](https://arxiv.org/html/2608.25089#A5.T7)\. For the first group, we compare expected language rankings, derived from training data proportions under the assumption that more data leads to better performance\([Hoffmann et al\., 2022](https://arxiv.org/html/2608.25089#bib.bib51)\), against actual metric rankings\. For the second group, whose full training data composition is undisclosed, we verify whether the dominant language reported in the technical documentation achieves the best ranking under each metric\. A reliable metric should produce rankings consistent with these expectations\.

#### Results\.

Results forBLOOM\-7BandBLOOM\-1Bare shown in Figure[4](https://arxiv.org/html/2608.25089#S5.F4)and confirm our predictions\. The leftmost column shows the expected ranking based on training data composition; the right two columns show actual rankings under each metric\. BPB and BPC are distorted by script differences: Chinese consistently ranks worst despite comprising the second largest portion ofBLOOM’s training data\. PPL and Token\-NLL are distorted by tokenization: Russian, which receives little training data inBLOOM, paradoxically ranks best because unseen languages are over\-segmented into more tokens, artificially deflating token\-normalized scores\. Only Sent\-NLL produces rankings that reflect actual training data composition\. We find the same pattern with other LLMs; raw results for all models are reported in Figure[14](https://arxiv.org/html/2608.25089#A6.F14), and ranking results forXGLM\-4\.5BandmGPTin Figure[13](https://arxiv.org/html/2608.25089#A6.F13)\.

## 7Considerations for Downstream Tasks

While intrinsic metrics are our primary focus, downstream task evaluations remain widely used in practice\. However, fair crosslingual comparison using downstream tasks is far from straightforward\. Below, we discuss several considerations for fair comparisons across languages using downstream evaluations\.

#### Translated Benchmarks\.

Some translated tasks may enable comparable evaluation across languages\. For example, a math benchmark translated into another language may allow a researcher to make an inference about the relative performance on math tasks in two languages\. Math and reasoning benchmarks are less likely to be influenced by translation quality and are more language\-agnostic\([Wu et al\., 2025](https://arxiv.org/html/2608.25089#bib.bib56)\)\. This relies on sufficiently high translation quality, however\. Machine translated benchmarks without human validation may contain translation artifacts and introduce noise into the evaluation\([Singh et al\., 2025](https://arxiv.org/html/2608.25089#bib.bib60)\)\. High\-quality benchmarks created through professional human translation, e\.g\. MMMLU\([OpenAI, 2024](https://arxiv.org/html/2608.25089#bib.bib61)\)or FLORES\([NLLB Team et al\., 2022](https://arxiv.org/html/2608.25089#bib.bib57)\)and derivatives like Belebele\([Bandarkar et al\., 2024](https://arxiv.org/html/2608.25089#bib.bib58)\)or SIB\-200\([Adelani et al\., 2024](https://arxiv.org/html/2608.25089#bib.bib59)\), are less likely to suffer from these issues\.

#### Culture\-specific tasks\.

Another consideration is cultural specificity\. Even when translated faithfully, some benchmarks may not be appropriate for comparable evaluation if their content is specific to a particular linguistic or cultural context\. Localized benchmarks, e\.g\. IndoMMLU \(Indonesian;[Koto et al\., 2023](https://arxiv.org/html/2608.25089#bib.bib66)\), CMMLU \(Chinese;[Li et al\., 2024](https://arxiv.org/html/2608.25089#bib.bib62)\), Turkish MMLU\([Yüksel et al\., 2024](https://arxiv.org/html/2608.25089#bib.bib65)\), Arabic MMLU\([Koto et al\., 2024](https://arxiv.org/html/2608.25089#bib.bib64)\), and KMMLU \(Korean;[Son et al\., 2025](https://arxiv.org/html/2608.25089#bib.bib63)\) evaluate not only target\-language performance but also cultural knowledge\. Localized benchmarks such as these correlate more strongly with human preferences than translated benchmarks\([Wu et al\., 2025](https://arxiv.org/html/2608.25089#bib.bib56)\)\. But achieving the same score on two localized MMLU\-style benchmarks does not signal comparable performance, as the questions are not difficulty\-matched or otherwise controlled\.

#### Language\-specific tasks\.

Language\-specific tasks, such as linguistic knowledge benchmarks like BLiMP\([Warstadt et al\., 2020](https://arxiv.org/html/2608.25089#bib.bib67)\)and MultiBLiMP\([Jumelet et al\., 2026](https://arxiv.org/html/2608.25089#bib.bib68)\), may not be comparable across languages for other reasons\. First, even benchmarks which evaluate the same relatively narrow tasks, such as subject\-verb agreement in MultiBLiMP\([Jumelet et al\., 2026](https://arxiv.org/html/2608.25089#bib.bib68)\), are based on non\-parallel data\. The items for each language differ in quantity and quality\. The number of items in MultiBLiMP per language ranges from 7 \(Gujarati\) to 4615 \(Old Russian\)\. For languages with very few items, like Gujarati, MultiBLiMP scores may be less reliable than those with much larger numbers of samples\. Additionally, as MultiBLiMP is based on UD Treebanks\([de Marneffe et al\., 2021](https://arxiv.org/html/2608.25089#bib.bib70);[Nivre et al\., 2020](https://arxiv.org/html/2608.25089#bib.bib28)\), which vary in their annotation schema\. These inconsistencies may lead to different conclusions about relative linguistic complexity, for example\([Rodríguez and López, 2025](https://arxiv.org/html/2608.25089#bib.bib81)\)\. Both of these issues can be addressed through the use of balanced parallel data with consistent annotation schema\.

These benchmarks are also sensitive to crosslinguistic differences\. In the development of CLAMS\([Mueller et al\., 2020](https://arxiv.org/html/2608.25089#bib.bib69)\), the authors discuss how differences in pronoun dropping, grammatical person distinction, and distribution of negative polarity items shaped the types of tests they could use in the comparison of agreement across languages\.

Subject\-verb agreement is inherently different across languages\. For some languages, the task is trivial as there is no agreement and the verb form is always the same \(e\.g\. Chinese and Tagalog;[Chao, 1968](https://arxiv.org/html/2608.25089#bib.bib82);[Reyes, 1969](https://arxiv.org/html/2608.25089#bib.bib83)\)\. Other languages have more complex agreement patterns due to higher number of distinctions between number \(singular, dual, and plural number, e\.g\. Arabic;[Bettega and D’Anna, 2022](https://arxiv.org/html/2608.25089#bib.bib84)\), gender/noun class \(about 10 noun classes in Swahili;[Moxley, 1998](https://arxiv.org/html/2608.25089#bib.bib85)\), verb class \(at least 4 verb classes in Georgian;[Makharoblidze, 2012](https://arxiv.org/html/2608.25089#bib.bib86)\), gender \(Tamil;[Venkatesan, 2024](https://arxiv.org/html/2608.25089#bib.bib87)\)\. It remains an open question how to fairly compare linguistic performance across languages, when the complexity of the phenomenon differs\.

## 8Discussion & Conclusion

As language models are increasingly deployed beyond English, fair crosslingual evaluation is a necessity\. How we evaluate models determines how we interpret results and allocate resources\. Prior work has noted that normalized metrics may be unreliable for cross\-lingual comparison in monolingual or multilingual settings\([Cotterell et al\., 2018](https://arxiv.org/html/2608.25089#bib.bib23);[Mielke et al\., 2019](https://arxiv.org/html/2608.25089#bib.bib46)\), but these observations have remained largely theoretical or have been based on intuition\. Problematic metrics continue to be used in different studies\([Shliazhko et al\., 2024](https://arxiv.org/html/2608.25089#bib.bib25);[Arnett and Bergen, 2025](https://arxiv.org/html/2608.25089#bib.bib19);[Thakur et al\., 2025](https://arxiv.org/html/2608.25089#bib.bib12)\)or explicitly advocated in recent proposals\([Shani et al\., 2026](https://arxiv.org/html/2608.25089#bib.bib48)\)\. Our contribution is to provide the first systematic empirical evidence across multiple metrics, languages, and tokenizer settings\. Such empirical grounding matters precisely because intuition alone has not been sufficient to change community norms\.

It is important to note that Sent\-NLL is not entirely free from confounds\. However, we argue that any remaining sensitivity reflects inherent linguistic complexity rather than arbitrary engineering choices, which is a qualitatively different kind of bias that is arguably unavoidable in any crosslingual comparison\. Our proposal implicitly assumes that all languages encode roughly comparable amounts of information, an assumption the literature both supports and challenges\([Koplenig et al\., 2017](https://arxiv.org/html/2608.25089#bib.bib31);[Crystal, 2010](https://arxiv.org/html/2608.25089#bib.bib55);[Fenk\-Oczlon and Fenk, 2014](https://arxiv.org/html/2608.25089#bib.bib71);[Bentz et al\., 2023](https://arxiv.org/html/2608.25089#bib.bib73)\)\. Even if languages systematically differ in information density, such differences reflect intrinsic linguistic properties rather than metric artifacts\. In this sense, Sent\-NLL may reflect both how well a model has learned a language and the intrinsic information density of that language\. Therefore, even if the linguistic equi\-complexity hypothesis, i\.e\., all natural languages are equally complex, is falsified, Sent\-NLL may not be a perfect measure of model performance in isolation, but it remains the least biased option among commonly used intrinsic metrics for crosslingual comparison\. Any remaining confound due to linguistic complexity should be viewed as a limitation shared by all crosslingual evaluation rather than a systematic bias of the metric itself\.

Finally, our results show that Sent\-NLL rankings show more variability across datasets than normalized metrics \(Figures[2](https://arxiv.org/html/2608.25089#S3.F2),[8](https://arxiv.org/html/2608.25089#A5.F8),[9](https://arxiv.org/html/2608.25089#A5.F9)\)\. We argue this reflects a strength rather than a weakness\. The apparent stability of metrics such as BPB, BPC, PPL, and Token\-NLL is artificial: their rankings are largely determined by engineering choices – script properties, tokenization granularity, and vocabulary size – rather than genuine differences in language modeling quality\. Sent\-NLL, by contrast, is sensitive to actual test data content, meaning its rankings reflect real differences in how well a model has learned each language\. Variability across datasets is therefore expected and desirable\.

## Limitations

Our evaluation only covers 10 languages due to limited parallel data\. A broader language sample would strengthen generalizability, although the significant correlations observed for all other metrics under the same sample size suggest that statistical power is sufficient to detect meaningful relationships\.

In the paraphrase experiments, only English and German translation pairs are tested due to the limited translation resources\. Extending this analysis to additional bilingual pairs with multiple translation references would further strengthen the conclusions\.

More broadly, our approach relies on high\-quality parallel sentences\. Although datasets such as FLORES provide relatively broad coverage, many languages still lack reliable parallel data, which limits the applicability of this evaluation framework\.

## Acknowledgments

We thank Tyler A\. Chang, Miryam de Lhoneux, Wessel Poelman, and anonymous reviewers for their helpful feedback\. We also thank CoreWeave for providing the GPU resources used for model training and evaluation\.

## References

- Adelaniet al\.\(2024\)D\. I\. Adelani, H\. Liu, X\. Shen, N\. Vassilyev, J\. O\. Alabi, Y\. Mao, H\. Gao, and E\. A\. LeeSIB\-200: a simple, inclusive, and big evaluation dataset for topic classification in 200\+ languages and dialects\.InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),Y\. Graham and M\. Purver \(Eds\.\),St\. Julian’s, Malta,pp\. 226–245\.External Links:[Link](https://aclanthology.org/2024.eacl-long.14/),[Document](https://dx.doi.org/10.18653/v1/2024.eacl-long.14)Cited by:[§7](https://arxiv.org/html/2608.25089#S7.SS0.SSS0.Px1.p1.1)\.
- Ahiaet al\.\(2024\)O\. Ahia, S\. Kumar, H\. Gonen, V\. Hofmann, T\. Limisiewicz, Y\. Tsvetkov, and N\. A\. SmithMagnet: improving the multilingual fairness of language models with adaptive gradient\-based tokenization\.Advances in neural information processing systems37,pp\. 47790–47814\.Cited by:[§2\.1](https://arxiv.org/html/2608.25089#S2.SS1.p1.1)\.
- Al\-Rfouet al\.\(2019\)R\. Al\-Rfou, D\. Choe, N\. Constant, M\. Guo, and L\. JonesCharacter\-level language modeling with deeper self\-attention\.InProceedings of the AAAI conference on artificial intelligence,Vol\.33,pp\. 3159–3166\.External Links:[Link](https://dl.acm.org/doi/10.1609/aaai.v33i01.33013159)Cited by:[§3](https://arxiv.org/html/2608.25089#S3.SS0.SSS0.Px2.p2.1)\.
- Aliet al\.\(2024\)M\. Ali, M\. Fromm, K\. Thellmann, J\. Ebert, A\. A\. Weber, R\. Rutmann, C\. Jain, M\. Lübbering, D\. Steinigen, J\. Leveling,et al\.Teuken\-7b\-base & teuken\-7b\-instruct: towards European LLMs\.arXiv preprint arXiv:2410\.03730\.Cited by:[Table 7](https://arxiv.org/html/2608.25089#A5.T7.2.6.2.1.1),[§6](https://arxiv.org/html/2608.25089#S6.SS0.SSS0.Px1.p1.1)\.
- Aravindaet al\.\(2025\)H\. W\. K\. Aravinda, R\. Sirajudeen, S\. Karunathilake, N\. de Silva, S\. Ranathunga, and R\. KaurSinLlama\- a large language model for Sinhala\.arXiv preprint arXiv:2508\.09115\.Cited by:[§1](https://arxiv.org/html/2608.25089#S1.p1.1)\.
- Arnett and Bergen \(2025\)C\. Arnett and B\. BergenWhy do language models perform worse for morphologically complex languages?\.InProceedings of the 31st International Conference on Computational Linguistics,O\. Rambow, L\. Wanner, M\. Apidianaki, H\. Al\-Khalifa, B\. D\. Eugenio, and S\. Schockaert \(Eds\.\),Abu Dhabi, UAE,pp\. 6607–6623\.External Links:[Link](https://aclanthology.org/2025.coling-main.441/)Cited by:[§1](https://arxiv.org/html/2608.25089#S1.p3.1),[§8](https://arxiv.org/html/2608.25089#S8.p1.1)\.
- Arnettet al\.\(2024\)C\. Arnett, T\. A\. Chang, and B\. BergenA bit of a problem: measurement disparities in dataset sizes across languages\.InProceedings of the 3rd Annual Meeting of the Special Interest Group on Under\-resourced Languages @ LREC\-COLING 2024,M\. Melero, S\. Sakti, and C\. Soria \(Eds\.\),Torino, Italia,pp\. 1–9\.External Links:[Link](https://aclanthology.org/2024.sigul-1.1/)Cited by:[§2\.1](https://arxiv.org/html/2608.25089#S2.SS1.SSS0.Px2.p1.1),[§3](https://arxiv.org/html/2608.25089#S3.SS0.SSS0.Px1.p2.1)\.
- Arnettet al\.\(2025\)C\. Arnett, T\. A\. Chang, S\. Biderman, and B\. BergenExplaining and mitigating crosslingual tokenizer inequities\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=0Y7AxxNCYh)Cited by:[§2\.1](https://arxiv.org/html/2608.25089#S2.SS1.SSS0.Px1.p1.1)\.
- Bandarkaret al\.\(2024\)L\. Bandarkar, D\. Liang, B\. Muller, M\. Artetxe, S\. N\. Shukla, D\. Husa, N\. Goyal, A\. Krishnan, L\. Zettlemoyer, and M\. KhabsaThe Belebele benchmark: A parallel reading comprehension dataset in 122 language variants\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 749–775\.External Links:[Link](https://aclanthology.org/2024.acl-long.44/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.44)Cited by:[§7](https://arxiv.org/html/2608.25089#S7.SS0.SSS0.Px1.p1.1)\.
- Barraultet al\.\(2019\)L\. Barrault, O\. Bojar, M\. R\. Costa\-jussà, C\. Federmann, M\. Fishel, Y\. Graham, B\. Haddow, M\. Huck, P\. Koehn, S\. Malmasi, C\. Monz, M\. Müller, S\. Pal, M\. Post, and M\. ZampieriFindings of the 2019 conference on machine translation \(WMT19\)\.InProceedings of the Fourth Conference on Machine Translation \(Volume 2: Shared Task Papers, Day 1\),O\. Bojar, R\. Chatterjee, C\. Federmann, M\. Fishel, Y\. Graham, B\. Haddow, M\. Huck, A\. J\. Yepes, P\. Koehn, A\. Martins, C\. Monz, M\. Negri, A\. Névéol, M\. Neves, M\. Post, M\. Turchi, and K\. Verspoor \(Eds\.\),Florence, Italy,pp\. 1–61\.External Links:[Link](https://aclanthology.org/W19-5301/),[Document](https://dx.doi.org/10.18653/v1/W19-5301)Cited by:[Table 2](https://arxiv.org/html/2608.25089#A1.T2.2.1.1.1.1.1.1.15.1),[§5](https://arxiv.org/html/2608.25089#S5.SS0.SSS0.Px1.p1.1)\.
- Barzilay and McKeown \(2001\)R\. Barzilay and K\. R\. McKeownExtracting paraphrases from a parallel corpus\.InProceedings of the 39th Annual Meeting of the Association for Computational Linguistics,Toulouse, France,pp\. 50–57\.External Links:[Link](https://aclanthology.org/P01-1008/),[Document](https://dx.doi.org/10.3115/1073012.1073020)Cited by:[Appendix D](https://arxiv.org/html/2608.25089#A4.p1.1)\.
- Bentzet al\.\(2023\)C\. Bentz, X\. Gutierrez\-Vasques, O\. Sozinova, and T\. SamardžićComplexity trade\-offs and equi\-complexity in natural languages: a meta\-analysis\.Linguistics Vanguard9\(s1\),pp\. 9–25\.Cited by:[§8](https://arxiv.org/html/2608.25089#S8.p2.1)\.
- Bettega and D’Anna \(2022\)S\. Bettega and L\. D’AnnaGender and number agreement in arabic\.Vol\.109,Brill\.Cited by:[§7](https://arxiv.org/html/2608.25089#S7.SS0.SSS0.Px3.p3.1)\.
- Bhagat and Hovy \(2013\)R\. Bhagat and E\. HovySquibs: what is a paraphrase?\.Computational Linguistics39\(3\),pp\. 463–472\.External Links:[Link](https://aclanthology.org/J13-3001/),[Document](https://dx.doi.org/10.1162/COLI%5Fa%5F00166)Cited by:[Appendix D](https://arxiv.org/html/2608.25089#A4.p6.1),[§5](https://arxiv.org/html/2608.25089#S5.p1.1)\.
- BigScience Workshopet al\.\(2023\)BigScience Workshop, T\. L\. Scao, A\. Fan, C\. Akiki, E\. Pavlick, S\. Ilić, D\. Hesslow, R\. Castagné, A\. S\. Luccioni, F\. Yvon, M\. Gallé, J\. Tow, A\. M\. Rush, S\. Biderman, A\. Webson, P\. S\. Ammanamanchi, T\. Wang, B\. Sagot, N\. Muennighoff, A\. V\. del Moral, O\. Ruwase, R\. Bawden, S\. Bekman, A\. McMillan\-Major, I\. Beltagy, H\. Nguyen, L\. Saulnier, S\. Tan, P\. O\. Suarez, V\. Sanh, H\. Laurençon, Y\. Jernite, J\. Launay, M\. Mitchell, C\. Raffel, A\. Gokaslan, A\. Simhi, A\. Soroa, A\. F\. Aji, A\. Alfassy, A\. Rogers, A\. K\. Nitzav, C\. Xu, C\. Mou, C\. Emezue, C\. Klamm, C\. Leong, D\. van Strien, D\. I\. Adelani, D\. Radev, E\. G\. Ponferrada, E\. Levkovizh, E\. Kim, E\. B\. Natan, F\. D\. Toni, G\. Dupont, G\. Kruszewski, G\. Pistilli, H\. Elsahar, H\. Benyamina, H\. Tran, I\. Yu, I\. Abdulmumin, I\. Johnson, I\. Gonzalez\-Dios, J\. de la Rosa, J\. Chim, J\. Dodge, J\. Zhu, J\. Chang, J\. Frohberg, J\. Tobing, J\. Bhattacharjee, K\. Almubarak, K\. Chen, K\. Lo, L\. V\. Werra, L\. Weber, L\. Phan, L\. B\. allal, L\. Tanguy, M\. Dey, M\. R\. Muñoz, M\. Masoud, M\. Grandury, M\. Šaško, M\. Huang, M\. Coavoux, M\. Singh, M\. T\. Jiang, M\. C\. Vu, M\. A\. Jauhar, M\. Ghaleb, N\. Subramani, N\. Kassner, N\. Khamis, O\. Nguyen, O\. Espejel, O\. de Gibert, P\. Villegas, P\. Henderson, P\. Colombo, P\. Amuok, Q\. Lhoest, R\. Harliman, R\. Bommasani, R\. L\. López, R\. Ribeiro, S\. Osei, S\. Pyysalo, S\. Nagel, S\. Bose, S\. H\. Muhammad, S\. Sharma, S\. Longpre, S\. Nikpoor, S\. Silberberg, S\. Pai, S\. Zink, T\. T\. Torrent, T\. Schick, T\. Thrush, V\. Danchev, V\. Nikoulina, V\. Laippala, V\. Lepercq, V\. Prabhu, Z\. Alyafeai, Z\. Talat, A\. Raja, B\. Heinzerling, C\. Si, D\. E\. Taşar, E\. Salesky, S\. J\. Mielke, W\. Y\. Lee, A\. Sharma, A\. Santilli, A\. Chaffin, A\. Stiegler, D\. Datta, E\. Szczechla, G\. Chhablani, H\. Wang, H\. Pandey, H\. Strobelt, J\. A\. Fries, J\. Rozen, L\. Gao, L\. Sutawika, M\. S\. Bari, M\. S\. Al\-shaibani, M\. Manica, N\. Nayak, R\. Teehan, S\. Albanie, S\. Shen, S\. Ben\-David, S\. H\. Bach, T\. Kim, T\. Bers, T\. Fevry, T\. Neeraj, U\. Thakker, V\. Raunak, X\. Tang, Z\. Yong, Z\. Sun, S\. Brody, Y\. Uri, H\. Tojarieh, A\. Roberts, H\. W\. Chung, J\. Tae, J\. Phang, O\. Press, C\. Li, D\. Narayanan, H\. Bourfoune, J\. Casper, J\. Rasley, M\. Ryabinin, M\. Mishra, M\. Zhang, M\. Shoeybi, M\. Peyrounette, N\. Patry, N\. Tazi, O\. Sanseviero, P\. von Platen, P\. Cornette, P\. F\. Lavallée, R\. Lacroix, S\. Rajbhandari, S\. Gandhi, S\. Smith, S\. Requena, S\. Patil, T\. Dettmers, A\. Baruwa, A\. Singh, A\. Cheveleva, A\. Ligozat, A\. Subramonian, A\. Névéol, C\. Lovering, D\. Garrette, D\. Tunuguntla, E\. Reiter, E\. Taktasheva, E\. Voloshina, E\. Bogdanov, G\. I\. Winata, H\. Schoelkopf, J\. Kalo, J\. Novikova, J\. Z\. Forde, J\. Clive, J\. Kasai, K\. Kawamura, L\. Hazan, M\. Carpuat, M\. Clinciu, N\. Kim, N\. Cheng, O\. Serikov, O\. Antverg, O\. van der Wal, R\. Zhang, R\. Zhang, S\. Gehrmann, S\. Mirkin, S\. Pais, T\. Shavrina, T\. Scialom, T\. Yun, T\. Limisiewicz, V\. Rieser, V\. Protasov, V\. Mikhailov, Y\. Pruksachatkun, Y\. Belinkov, Z\. Bamberger, Z\. Kasner, A\. Rueda, A\. Pestana, A\. Feizpour, A\. Khan, A\. Faranak, A\. Santos, A\. Hevia, A\. Unldreaj, A\. Aghagol, A\. Abdollahi, A\. Tammour, A\. HajiHosseini, B\. Behroozi, B\. Ajibade, B\. Saxena, C\. M\. Ferrandis, D\. McDuff, D\. Contractor, D\. Lansky, D\. David, D\. Kiela, D\. A\. Nguyen, E\. Tan, E\. Baylor, E\. Ozoani, F\. Mirza, F\. Ononiwu, H\. Rezanejad, H\. Jones, I\. Bhattacharya, I\. Solaiman, I\. Sedenko, I\. Nejadgholi, J\. Passmore, J\. Seltzer, J\. B\. Sanz, L\. Dutra, M\. Samagaio, M\. Elbadri, M\. Mieskes, M\. Gerchick, M\. Akinlolu, M\. McKenna, M\. Qiu, M\. Ghauri, M\. Burynok, N\. Abrar, N\. Rajani, N\. Elkott, N\. Fahmy, O\. Samuel, R\. An, R\. Kromann, R\. Hao, S\. Alizadeh, S\. Shubber, S\. Wang, S\. Roy, S\. Viguier, T\. Le, T\. Oyebade, T\. Le, Y\. Yang, Z\. Nguyen, A\. R\. Kashyap, A\. Palasciano, A\. Callahan, A\. Shukla, A\. Miranda\-Escalada, A\. Singh, B\. Beilharz, B\. Wang, C\. Brito, C\. Zhou, C\. Jain, C\. Xu, C\. Fourrier, D\. L\. Periñán, D\. Molano, D\. Yu, E\. Manjavacas, F\. Barth, F\. Fuhrimann, G\. Altay, G\. Bayrak, G\. Burns, H\. U\. Vrabec, I\. Bello, I\. Dash, J\. Kang, J\. Giorgi, J\. Golde, J\. D\. Posada, K\. R\. Sivaraman, L\. Bulchandani, L\. Liu, L\. Shinzato, M\. H\. de Bykhovetz, M\. Takeuchi, M\. Pàmies, M\. A\. Castillo, M\. Nezhurina, M\. Sänger, M\. Samwald, M\. Cullan, M\. Weinberg, M\. D\. Wolf, M\. Mihaljcic, M\. Liu, M\. Freidank, M\. Kang, N\. Seelam, N\. Dahlberg, N\. M\. Broad, N\. Muellner, P\. Fung, P\. Haller, R\. Chandrasekhar, R\. Eisenberg, R\. Martin, R\. Canalli, R\. Su, R\. Su, S\. Cahyawijaya, S\. Garda, S\. S\. Deshmukh, S\. Mishra, S\. Kiblawi, S\. Ott, S\. Sang\-aroonsiri, S\. Kumar, S\. Schweter, S\. Bharati, T\. Laud, T\. Gigant, T\. Kainuma, W\. Kusa, Y\. Labrak, Y\. S\. Bajaj, Y\. Venkatraman, Y\. Xu, Y\. Xu, Y\. Xu, Z\. Tan, Z\. Xie, Z\. Ye, M\. Bras, Y\. Belkada, and T\. WolfBLOOM: a 176b\-parameter open\-access multilingual language model\.External Links:2211\.05100,[Link](https://arxiv.org/abs/2211.05100)Cited by:[Table 7](https://arxiv.org/html/2608.25089#A5.T7.2.7.2.1.1),[Table 7](https://arxiv.org/html/2608.25089#A5.T7.2.8.2.1.1),[§1](https://arxiv.org/html/2608.25089#S1.p1.1),[§6](https://arxiv.org/html/2608.25089#S6.SS0.SSS0.Px1.p1.1)\.
- Blevinset al\.\(2022\)T\. Blevins, H\. Gonen, and L\. ZettlemoyerAnalyzing the mono\- and cross\-lingual pretraining dynamics of multilingual language models\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 3575–3590\.External Links:[Link](https://aclanthology.org/2022.emnlp-main.234/),[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.234)Cited by:[§1](https://arxiv.org/html/2608.25089#S1.p3.1),[§2\.2](https://arxiv.org/html/2608.25089#S2.SS2.p2.1)\.
- Boonthum \(2004\)C\. BoonthumISTART: paraphrase recognition\.InProceedings of the ACL Student Research Workshop,Barcelona, Spain,pp\. 31–36\.External Links:[Link](https://aclanthology.org/P04-2006/)Cited by:[Appendix D](https://arxiv.org/html/2608.25089#A4.p1.1)\.
- Bresnan and Nikitina \(2009\)J\. Bresnan and T\. NikitinaThe gradience of the dative alternation\.Reality exploration and discovery: Pattern interaction in language and life,pp\. 161–184\.Cited by:[Appendix D](https://arxiv.org/html/2608.25089#A4.p6.1)\.
- Brownet al\.\(2020\)T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.Language models are few\-shot learners\.Advances in neural information processing systems33,pp\. 1877–1901\.Cited by:[§3](https://arxiv.org/html/2608.25089#S3.SS0.SSS0.Px4.p1.1)\.
- Changet al\.\(2023\)T\. A\. Chang, C\. Arnett, Z\. Tu, and B\. K\. BergenWhen is multilinguality a curse? language modeling for 250 high\-and low\-resource languages\.arXiv preprint arXiv:2311\.09205\.Cited by:[§1](https://arxiv.org/html/2608.25089#S1.p1.1),[§1](https://arxiv.org/html/2608.25089#S1.p5.1)\.
- Changet al\.\(2026\)T\. A\. Chang, C\. Arnett, Z\. Tu, and B\. K\. BergenGoldfish: monolingual language models for 350 languages\.Language Resources and Evaluation Conference \(LREC\)\.Cited by:[§3](https://arxiv.org/html/2608.25089#S3.SS0.SSS0.Px6.p3.1)\.
- Chao \(1968\)Y\. R\. ChaoA grammar of spoken chinese\.University of California Press\.Cited by:[§7](https://arxiv.org/html/2608.25089#S7.SS0.SSS0.Px3.p3.1)\.
- Christodouloupoulos and Steedman \(2015\)C\. Christodouloupoulos and M\. SteedmanA massively parallel corpus: the bible in 100 languages\.Language resources and evaluation49\(2\),pp\. 375–395\.Cited by:[Table 2](https://arxiv.org/html/2608.25089#A1.T2.2.1.1.1.1.1.1.4.1)\.
- Costa\-Jussàet al\.\(2022\)M\. R\. Costa\-Jussà, J\. Cross, O\. Çelebi, M\. Elbayad, K\. Heafield, K\. Heffernan, E\. Kalbassi, J\. Lam, D\. Licht, J\. Maillard,et al\.No language left behind: scaling human\-centered machine translation\.arXiv preprint arXiv:2207\.04672\.Cited by:[§3](https://arxiv.org/html/2608.25089#S3.SS0.SSS0.Px6.p3.1)\.
- Cotterellet al\.\(2018\)R\. Cotterell, S\. J\. Mielke, J\. Eisner, and B\. RoarkAre all languages equally hard to language\-model?\.InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 \(Short Papers\),M\. Walker, H\. Ji, and A\. Stent \(Eds\.\),New Orleans, Louisiana,pp\. 536–541\.External Links:[Link](https://aclanthology.org/N18-2085/),[Document](https://dx.doi.org/10.18653/v1/N18-2085)Cited by:[§1](https://arxiv.org/html/2608.25089#S1.p3.1),[§1](https://arxiv.org/html/2608.25089#S1.p5.1),[§2\.2](https://arxiv.org/html/2608.25089#S2.SS2.p1.1),[§3](https://arxiv.org/html/2608.25089#S3.SS0.SSS0.Px7.p1.1),[§8](https://arxiv.org/html/2608.25089#S8.p1.1)\.
- Crystal \(2010\)D\. CrystalThe equality of languages\.InThe Cambridge Encyclopedia of Language,pp\. 6–7\.Cited by:[§8](https://arxiv.org/html/2608.25089#S8.p2.1)\.
- de Marneffeet al\.\(2021\)M\. de Marneffe, C\. D\. Manning, J\. Nivre, and D\. ZemanUniversal Dependencies\.Computational Linguistics47\(2\),pp\. 255–308\.External Links:[Link](https://aclanthology.org/2021.cl-2.11/),[Document](https://dx.doi.org/10.1162/coli%5Fa%5F00402)Cited by:[§7](https://arxiv.org/html/2608.25089#S7.SS0.SSS0.Px3.p1.1)\.
- Fenk\-Oczlon and Fenk \(2014\)G\. Fenk\-Oczlon and A\. FenkComplexity trade\-offs do not prove the equal complexity hypothesis\.Poznan Studies in Contemporary Linguistics50\(2\),pp\. 145\.External Links:[Link](https://doi.org/10.1515/psicl-2014-0010)Cited by:[§8](https://arxiv.org/html/2608.25089#S8.p2.1)\.
- Freitaget al\.\(2020a\)M\. Freitag, G\. Foster, D\. Grangier, and C\. CherryHuman\-paraphrased references improve neural machine translation\.InProceedings of the Fifth Conference on Machine Translation,L\. Barrault, O\. Bojar, F\. Bougares, R\. Chatterjee, M\. R\. Costa\-jussà, C\. Federmann, M\. Fishel, A\. Fraser, Y\. Graham, P\. Guzman, B\. Haddow, M\. Huck, A\. J\. Yepes, P\. Koehn, A\. Martins, M\. Morishita, C\. Monz, M\. Nagata, T\. Nakazawa, and M\. Negri \(Eds\.\),Online,pp\. 1183–1192\.External Links:[Link](https://aclanthology.org/2020.wmt-1.140/),[Document](https://dx.doi.org/10.18653/v1/2020.wmt-1.140)Cited by:[Appendix D](https://arxiv.org/html/2608.25089#A4.p1.1),[§5](https://arxiv.org/html/2608.25089#S5.SS0.SSS0.Px1.p1.1)\.
- Freitaget al\.\(2020b\)M\. Freitag, D\. Grangier, and I\. CaswellBLEU might be guilty but references are not innocent\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 61–71\.External Links:[Link](https://aclanthology.org/2020.emnlp-main.5/),[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.5)Cited by:[§5](https://arxiv.org/html/2608.25089#S5.SS0.SSS0.Px1.p1.1)\.
- Fujiiet al\.\(2023\)T\. Fujii, K\. Shibata, A\. Yamaguchi, T\. Morishita, and Y\. SogawaHow do different tokenizers perform on downstream tasks in scriptio continua languages?: a case study in Japanese\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 4: Student Research Workshop\),V\. Padmakumar, G\. Vallejo, and Y\. Fu \(Eds\.\),Toronto, Canada,pp\. 39–49\.External Links:[Link](https://aclanthology.org/2023.acl-srw.5/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-srw.5)Cited by:[§2\.1](https://arxiv.org/html/2608.25089#S2.SS1.SSS0.Px1.p2.1)\.
- Gaoet al\.\(2020\)L\. Gao, S\. Biderman, S\. Black, L\. Golding, T\. Hoppe, C\. Foster, J\. Phang, H\. He, A\. Thite, N\. Nabeshima,et al\.The pile: an 800gb dataset of diverse text for language modeling\.arXiv preprint arXiv:2101\.00027\.Cited by:[§1](https://arxiv.org/html/2608.25089#S1.p3.1),[§3](https://arxiv.org/html/2608.25089#S3.SS0.SSS0.Px1.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.The Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[Table 7](https://arxiv.org/html/2608.25089#A5.T7.2.2.2.1.1),[Table 7](https://arxiv.org/html/2608.25089#A5.T7.2.3.2.1.1),[Table 7](https://arxiv.org/html/2608.25089#A5.T7.2.4.2.1.1),[§6](https://arxiv.org/html/2608.25089#S6.SS0.SSS0.Px1.p1.1)\.
- Hoffmannet al\.\(2022\)J\. Hoffmann, S\. Borgeaud, A\. Mensch, E\. Buchatskaya, T\. Cai, E\. Rutherford, D\. de Las Casas, L\. A\. Hendricks, J\. Welbl, A\. Clark, T\. Hennigan, E\. Noland, K\. Millican, G\. van den Driessche, B\. Damoc, A\. Guy, S\. Osindero, K\. Simonyan, E\. Elsen, O\. Vinyals, J\. W\. Rae, and L\. SifreTraining compute\-optimal large language models\.InProceedings of the 36th International Conference on Neural Information Processing Systems,NIPS ’22,Red Hook, NY, USA\.External Links:ISBN 9781713871088Cited by:[§6](https://arxiv.org/html/2608.25089#S6.SS0.SSS0.Px1.p1.1)\.
- Joshiet al\.\(2020\)P\. Joshi, S\. Santy, A\. Budhiraja, K\. Bali, and M\. ChoudhuryThe state and fate of linguistic diversity and inclusion in the NLP world\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 6282–6293\.External Links:[Link](https://aclanthology.org/2020.acl-main.560/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.560)Cited by:[§1](https://arxiv.org/html/2608.25089#S1.p1.1)\.
- Jumeletet al\.\(2026\)J\. Jumelet, L\. Weissweiler, J\. Nivre, and A\. BisazzaMultiblimp 1\.0: a massively multilingual benchmark of linguistic minimal pairs\.Transactions of the Association for Computational Linguistics14,pp\. 193–216\.Cited by:[§7](https://arxiv.org/html/2608.25089#S7.SS0.SSS0.Px3.p1.1)\.
- Kopleniget al\.\(2017\)A\. Koplenig, P\. Meyer, S\. Wolfer, and C\. Müller\-SpitzerThe statistical trade\-off between word order and word structure–large\-scale evidence for the principle of least effort\.PloS one12\(3\),pp\. e0173614\.Cited by:[§8](https://arxiv.org/html/2608.25089#S8.p2.1)\.
- Kotoet al\.\(2023\)F\. Koto, N\. Aisyah, H\. Li, and T\. BaldwinLarge Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLU\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 12359–12374\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.760/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.760)Cited by:[§7](https://arxiv.org/html/2608.25089#S7.SS0.SSS0.Px2.p1.1)\.
- Kotoet al\.\(2024\)F\. Koto, H\. Li, S\. Shatnawi, J\. Doughman, A\. Sadallah, A\. Alraeesi, K\. Almubarak, Z\. Alyafeai, N\. Sengupta, S\. Shehata, N\. Habash, P\. Nakov, and T\. BaldwinArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 5622–5640\.External Links:[Link](https://aclanthology.org/2024.findings-acl.334/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.334)Cited by:[§7](https://arxiv.org/html/2608.25089#S7.SS0.SSS0.Px2.p1.1)\.
- Levin \(1993\)B\. LevinEnglish verb classes and alternations: a preliminary investigation\.University of Chicago Press\.Cited by:[Appendix D](https://arxiv.org/html/2608.25089#A4.p6.1)\.
- Liet al\.\(2024\)H\. Li, Y\. Zhang, F\. Koto, Y\. Yang, H\. Zhao, Y\. Gong, N\. Duan, and T\. BaldwinCMMLU: measuring massive multitask language understanding in Chinese\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 11260–11285\.External Links:[Link](https://aclanthology.org/2024.findings-acl.671/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.671)Cited by:[§7](https://arxiv.org/html/2608.25089#S7.SS0.SSS0.Px2.p1.1)\.
- Limisiewiczet al\.\(2023\)T\. Limisiewicz, J\. Balhar, and D\. MarečekTokenization impacts multilingual language modeling: assessing vocabulary allocation and overlap across languages\.InFindings of the Association for Computational Linguistics: ACL 2023,A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 5661–5681\.External Links:[Link](https://aclanthology.org/2023.findings-acl.350/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-acl.350)Cited by:[§1](https://arxiv.org/html/2608.25089#S1.p2.1),[§3](https://arxiv.org/html/2608.25089#S3.SS0.SSS0.Px3.p1.1)\.
- Limisiewiczet al\.\(2024\)T\. Limisiewicz, T\. Blevins, H\. Gonen, O\. Ahia, and L\. ZettlemoyerMYTE: morphology\-driven byte encoding for better and fairer multilingual language modeling\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 15059–15076\.External Links:[Link](https://aclanthology.org/2024.acl-long.804/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.804)Cited by:[§2\.2](https://arxiv.org/html/2608.25089#S2.SS2.p1.1)\.
- Linet al\.\(2024\)P\. Lin, S\. Ji, J\. Tiedemann, A\. F\. Martins, and H\. SchützeMala\-500: massive language adaptation of large language models\.arXiv preprint arXiv:2401\.13303\.Cited by:[§3](https://arxiv.org/html/2608.25089#S3.SS0.SSS0.Px6.p3.1)\.
- Linet al\.\(2022\)X\. V\. Lin, T\. Mihaylov, M\. Artetxe, T\. Wang, S\. Chen, D\. Simig, M\. Ott, N\. Goyal, S\. Bhosale, J\. Du, R\. Pasunuru, S\. Shleifer, P\. S\. Koura, V\. Chaudhary, B\. O’Horo, J\. Wang, L\. Zettlemoyer, Z\. Kozareva, M\. Diab, V\. Stoyanov, and X\. LiFew\-shot learning with multilingual generative language models\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 9019–9052\.External Links:[Link](https://aclanthology.org/2022.emnlp-main.616/),[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.616)Cited by:[Table 7](https://arxiv.org/html/2608.25089#A5.T7.2.9.2.1.1),[§6](https://arxiv.org/html/2608.25089#S6.SS0.SSS0.Px1.p1.1)\.
- Lison and Tiedemann \(2016\)P\. Lison and J\. TiedemannOpenSubtitles2016: extracting large parallel corpora from movie and TV subtitles\.InProceedings of the Tenth International Conference on Language Resources and Evaluation \(LREC’16\),N\. Calzolari, K\. Choukri, T\. Declerck, S\. Goggi, M\. Grobelnik, B\. Maegaard, J\. Mariani, H\. Mazo, A\. Moreno, J\. Odijk, and S\. Piperidis \(Eds\.\),Portorož, Slovenia,pp\. 923–929\.External Links:[Link](https://aclanthology.org/L16-1147/)Cited by:[Table 2](https://arxiv.org/html/2608.25089#A1.T2.2.1.1.1.1.1.1.3.1)\.
- Liuet al\.\(2026\)Y\. Liu, Y\. Shen, H\. Zhu, L\. Xu, Z\. Qian, S\. Song, K\. Zhang, J\. Tang, P\. Zhang, B\. Yang,et al\.A systematic assessment of language models with linguistic minimal pairs in chinese\.Transactions of the Association for Computational Linguistics14,pp\. 755–771\.Cited by:[footnote 8](https://arxiv.org/html/2608.25089#footnote8)\.
- Madnani and Dorr \(2010\)N\. Madnani and B\. J\. DorrGenerating phrasal and sentential paraphrases: a survey of data\-driven methods\.Computational Linguistics36\(3\),pp\. 341–387\.External Links:[Link](https://aclanthology.org/J10-3003/),[Document](https://dx.doi.org/10.1162/coli%5Fa%5F00002)Cited by:[Appendix D](https://arxiv.org/html/2608.25089#A4.p1.1)\.
- Makharoblidze \(2012\)T\. MakharoblidzeThe Georgian verb\.Lincom Europa Munich\.Cited by:[§7](https://arxiv.org/html/2608.25089#S7.SS0.SSS0.Px3.p3.1)\.
- Mielkeet al\.\(2019\)S\. J\. Mielke, R\. Cotterell, K\. Gorman, B\. Roark, and J\. EisnerWhat kind of language is hard to language\-model?\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,A\. Korhonen, D\. Traum, and L\. Màrquez \(Eds\.\),Florence, Italy,pp\. 4975–4989\.External Links:[Link](https://aclanthology.org/P19-1491/),[Document](https://dx.doi.org/10.18653/v1/P19-1491)Cited by:[§1](https://arxiv.org/html/2608.25089#S1.p3.1),[§1](https://arxiv.org/html/2608.25089#S1.p5.1),[§2\.2](https://arxiv.org/html/2608.25089#S2.SS2.p1.1),[§8](https://arxiv.org/html/2608.25089#S8.p1.1)\.
- Moxley \(1998\)J\. L\. MoxleySemantic structure of Swahili noun classes\.Language history and linguistic description in Africa2,pp\. 229\.Cited by:[§7](https://arxiv.org/html/2608.25089#S7.SS0.SSS0.Px3.p3.1)\.
- Muelleret al\.\(2020\)A\. Mueller, G\. Nicolai, P\. Petrou\-Zeniou, N\. Talmina, and T\. LinzenCross\-linguistic syntactic evaluation of word prediction models\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 5523–5539\.External Links:[Link](https://aclanthology.org/2020.acl-main.490/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.490)Cited by:[§7](https://arxiv.org/html/2608.25089#S7.SS0.SSS0.Px3.p2.1)\.
- Neusner \(1986\)J\. NeusnerTranslation and paraphrase: the differences and why they matter\.Hebrew studies,pp\. 26–37\.Cited by:[Appendix D](https://arxiv.org/html/2608.25089#A4.p7.1),[§5](https://arxiv.org/html/2608.25089#S5.p1.1)\.
- Nivreet al\.\(2020\)J\. Nivre, M\. de Marneffe, F\. Ginter, J\. Hajič, C\. D\. Manning, S\. Pyysalo, S\. Schuster, F\. Tyers, and D\. ZemanUniversal Dependencies v2: an evergrowing multilingual treebank collection\.InProceedings of the Twelfth Language Resources and Evaluation Conference,N\. Calzolari, F\. Béchet, P\. Blache, K\. Choukri, C\. Cieri, T\. Declerck, S\. Goggi, H\. Isahara, B\. Maegaard, J\. Mariani, H\. Mazo, A\. Moreno, J\. Odijk, and S\. Piperidis \(Eds\.\),Marseille, France,pp\. 4034–4043\(eng\)\.External Links:[Link](https://aclanthology.org/2020.lrec-1.497/),ISBN 979\-10\-95546\-34\-4Cited by:[§7](https://arxiv.org/html/2608.25089#S7.SS0.SSS0.Px3.p1.1)\.
- NLLB Teamet al\.\(2022\)NLLB Team, M\. R\. Costa\-jussà, J\. Cross, O\. Çelebi, M\. Elbayad, K\. Heafield, K\. Heffernan, E\. Kalbassi, J\. Lam, D\. Licht, J\. Maillard, A\. Sun, S\. Wang, G\. Wenzek, A\. Youngblood, B\. Akula, L\. Barrault, G\. Mejia\-Gonzalez, P\. Hansanti, J\. Hoffman, S\. Jarrett, K\. R\. Sadagopan, D\. Rowe, S\. Spruit, C\. Tran, P\. Andrews, N\. F\. Ayan, S\. Bhosale, S\. Edunov, A\. Fan, C\. Gao, V\. Goswami, F\. Guzmán, P\. Koehn, A\. Mourachko, C\. Ropers, S\. Saleem, H\. Schwenk, and J\. WangNo language left behind: scaling human\-centered machine translation\.External Links:arXiv:1902\.01382Cited by:[§4](https://arxiv.org/html/2608.25089#S4.SS0.SSS0.Px2.p1.1),[§7](https://arxiv.org/html/2608.25089#S7.SS0.SSS0.Px1.p1.1)\.
- OpenAI \(2024\)OpenAIMultilingual Massive Multitask Language Understanding \(MMMLU\)\.External Links:[Link](https://huggingface.co/datasets/openai/MMMLU)Cited by:[§7](https://arxiv.org/html/2608.25089#S7.SS0.SSS0.Px1.p1.1)\.
- Petrovet al\.\(2023\)A\. Petrov, E\. La Malfa, P\. Torr, and A\. BibiLanguage model tokenizers introduce unfairness between languages\.Advances in neural information processing systems36,pp\. 36963–36990\.Cited by:[§2\.1](https://arxiv.org/html/2608.25089#S2.SS1.SSS0.Px1.p1.1),[§2\.1](https://arxiv.org/html/2608.25089#S2.SS1.p1.1)\.
- Poelmanet al\.\(2025\)W\. Poelman, T\. Bauwens, and M\. de LhoneuxConfounding factors in relating model performance to morphology\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 7262–7287\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.369/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.369),ISBN 979\-8\-89176\-332\-6Cited by:[Appendix D](https://arxiv.org/html/2608.25089#A4.p3.1),[§5](https://arxiv.org/html/2608.25089#S5.SS0.SSS0.Px1.p4.1)\.
- Poelman and de Lhoneux \(2026\)W\. Poelman and M\. de LhoneuxForm and meaning in intrinsic multilingual evaluations\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),V\. Demberg, K\. Inui, and L\. Marquez \(Eds\.\),Rabat, Morocco,pp\. 2503–2521\.External Links:[Link](https://aclanthology.org/2026.eacl-long.113/),[Document](https://dx.doi.org/10.18653/v1/2026.eacl-long.113),ISBN 979\-8\-89176\-380\-7Cited by:[Appendix D](https://arxiv.org/html/2608.25089#A4.p1.1),[§1](https://arxiv.org/html/2608.25089#S1.p3.1),[§2\.2](https://arxiv.org/html/2608.25089#S2.SS2.p2.1),[§4](https://arxiv.org/html/2608.25089#S4.SS0.SSS0.Px3.p1.1),[§5](https://arxiv.org/html/2608.25089#S5.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2608.25089#S5.p1.1),[footnote 7](https://arxiv.org/html/2608.25089#footnote7)\.
- Radfordet al\.\(2019\)A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, and I\. SutskeverLanguage models are unsupervised multitask learners\.Cited by:[footnote 2](https://arxiv.org/html/2608.25089#footnote2)\.
- Raeet al\.\(2021\)J\. W\. Rae, S\. Borgeaud, T\. Cai, K\. Millican, J\. Hoffmann, F\. Song, J\. Aslanides, S\. Henderson, R\. Ring, S\. Young,et al\.Scaling language models: methods, analysis & insights from training gopher\.arXiv preprint arXiv:2112\.11446\.Cited by:[§3](https://arxiv.org/html/2608.25089#S3.SS0.SSS0.Px1.p2.1)\.
- Reddyet al\.\(2025\)V\. Reddy, C\. W\. Schmidt, Y\. Pinter, and C\. TannerHow much is enough? the diminishing returns of tokenization training data\.arXiv preprint arXiv:2502\.20273\.Cited by:[§2\.1](https://arxiv.org/html/2608.25089#S2.SS1.SSS0.Px1.p2.1)\.
- Reimers and Gurevych \(2020\)N\. Reimers and I\. GurevychMaking monolingual sentence embeddings multilingual using knowledge distillation\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 4512–4525\.External Links:[Link](https://aclanthology.org/2020.emnlp-main.365/),[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.365)Cited by:[Table 2](https://arxiv.org/html/2608.25089#A1.T2.2.1.1.1.1.1.1.8.1)\.
- Reyes \(1969\)E\. V\. ReyesSome problems of interference in the use of English verbs by native tagalog speakers\.Language Learning19\(1\-2\),pp\. 87–97\.Cited by:[§7](https://arxiv.org/html/2608.25089#S7.SS0.SSS0.Px3.p3.1)\.
- Rodríguez and López \(2025\)A\. B\. Rodríguez and M\. D\. J\. LópezBeyond the data: the impact of annotation inconsistencies in UD treebanks on typological universals and complexity assessment\.InProceedings of the 7th Workshop on Research in Computational Linguistic Typology and Multilingual NLP,M\. Hahn, P\. Rani, R\. Kumar, A\. Shcherbakov, A\. Sorokin, O\. Serikov, R\. Cotterell, and E\. Vylomova \(Eds\.\),Vienna, Austria,pp\. 43–51\.External Links:[Link](https://aclanthology.org/2025.sigtyp-1.5/),[Document](https://dx.doi.org/10.18653/v1/2025.sigtyp-1.5),ISBN 979\-8\-89176\-281\-7Cited by:[§7](https://arxiv.org/html/2608.25089#S7.SS0.SSS0.Px3.p1.1)\.
- Rustet al\.\(2021\)P\. Rust, J\. Pfeiffer, I\. Vulić, S\. Ruder, and I\. GurevychHow good is your tokenizer? on the monolingual performance of multilingual language models\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),C\. Zong, F\. Xia, W\. Li, and R\. Navigli \(Eds\.\),Online,pp\. 3118–3135\.External Links:[Link](https://aclanthology.org/2021.acl-long.243/),[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.243)Cited by:[§1](https://arxiv.org/html/2608.25089#S1.p2.1)\.
- Santos and Barreiro \(2025\)D\. Santos and A\. BarreiroParaphrase and translation: the importance of being close\.Open Research Europe5,pp\. 52\.Cited by:[Appendix D](https://arxiv.org/html/2608.25089#A4.p6.1)\.
- Schmidtet al\.\(2024\)C\. W\. Schmidt, V\. Reddy, H\. Zhang, A\. Alameddine, O\. Uzan, Y\. Pinter, and C\. TannerTokenization is more than compression\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 678–702\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.40/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.40)Cited by:[§2\.1](https://arxiv.org/html/2608.25089#S2.SS1.SSS0.Px1.p1.1)\.
- Shaniet al\.\(2026\)C\. Shani, Y\. Reif, N\. Roll, D\. Jurafsky, and E\. ShutovaThe roots of performance disparity in multilingual language models: intrinsic modeling difficulty or design choices?\.arXiv preprint arXiv:2601\.07220\.Cited by:[§1](https://arxiv.org/html/2608.25089#S1.p1.1),[§1](https://arxiv.org/html/2608.25089#S1.p5.1),[§2\.1](https://arxiv.org/html/2608.25089#S2.SS1.SSS0.Px1.p2.1),[§2\.1](https://arxiv.org/html/2608.25089#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2608.25089#S2.SS2.p2.1),[§8](https://arxiv.org/html/2608.25089#S8.p1.1)\.
- Shliazhkoet al\.\(2024\)O\. Shliazhko, A\. Fenogenova, M\. Tikhonova, A\. Kozlova, V\. Mikhailov, and T\. ShavrinaMGPT: few\-shot learners go multilingual\.Transactions of the Association for Computational Linguistics12,pp\. 58–79\.External Links:[Link](https://aclanthology.org/2024.tacl-1.4/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00633)Cited by:[Table 7](https://arxiv.org/html/2608.25089#A5.T7.2.10.2.1.1),[§1](https://arxiv.org/html/2608.25089#S1.p3.1),[§2\.2](https://arxiv.org/html/2608.25089#S2.SS2.p2.1),[§6](https://arxiv.org/html/2608.25089#S6.SS0.SSS0.Px1.p1.1),[§8](https://arxiv.org/html/2608.25089#S8.p1.1)\.
- Singhet al\.\(2025\)S\. Singh, A\. Romanou, C\. Fourrier, D\. I\. Adelani, J\. G\. Ngui, D\. Vila\-Suero, P\. Limkonchotiwat, K\. Marchisio, W\. Q\. Leong, Y\. Susanto, R\. Ng, S\. Longpre, S\. Ruder, W\. Ko, A\. Bosselut, A\. Oh, A\. Martins, L\. Choshen, D\. Ippolito, E\. Ferrante, M\. Fadaee, B\. Ermis, and S\. HookerGlobal MMLU: understanding and addressing cultural and linguistic biases in multilingual evaluation\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 18761–18799\.External Links:[Link](https://aclanthology.org/2025.acl-long.919/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.919),ISBN 979\-8\-89176\-251\-0Cited by:[§7](https://arxiv.org/html/2608.25089#S7.SS0.SSS0.Px1.p1.1)\.
- Sonet al\.\(2025\)G\. Son, H\. Lee, S\. Kim, S\. Kim, N\. Muennighoff, T\. Choi, C\. Park, K\. M\. Yoo, and S\. BidermanKMMLU: measuring massive multitask language understanding in Korean\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 4076–4104\.External Links:[Link](https://aclanthology.org/2025.naacl-long.206/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.206),ISBN 979\-8\-89176\-189\-6Cited by:[§7](https://arxiv.org/html/2608.25089#S7.SS0.SSS0.Px2.p1.1)\.
- Thakuret al\.\(2025\)A\. Thakur, A\. Nagpal, A\. Savarkar, K\. Pundalik, S\. Dosi, P\. Sawarkar, V\. Thakur, R\. Saluja, M\. S\. Desarkar, and G\. RamakrishnanThe art of breaking words: rethinking multilingual tokenizer design\.arXiv preprint arXiv:2508\.06533\.Cited by:[§1](https://arxiv.org/html/2608.25089#S1.p3.1),[§3](https://arxiv.org/html/2608.25089#S3.SS0.SSS0.Px4.p2.1),[§8](https://arxiv.org/html/2608.25089#S8.p1.1)\.
- Tiedemann \(23\-25\)J\. TiedemannParallel data, tools and interfaces in opus\.InProceedings of the Eight International Conference on Language Resources and Evaluation \(LREC’12\),N\. C\. \(\. Chair\), K\. Choukri, T\. Declerck, M\. U\. Dogan, B\. Maegaard, J\. Mariani, J\. Odijk, and S\. Piperidis \(Eds\.\),Istanbul, Turkey\(english\)\.External Links:ISBN 978\-2\-9517408\-7\-7Cited by:[Appendix A](https://arxiv.org/html/2608.25089#A1.p2.1)\.
- Toramanet al\.\(2023\)C\. Toraman, E\. H\. Yilmaz, F\. Şahinuç, and O\. OzcelikImpact of tokenization on language models: an analysis for Turkish\.ACM Transactions on Asian and Low\-Resource Language Information Processing22\(4\),pp\. 1–21\.Cited by:[§2\.1](https://arxiv.org/html/2608.25089#S2.SS1.SSS0.Px1.p2.1)\.
- Tsvetkov and Kipnis \(2024\)A\. Tsvetkov and A\. KipnisInformation parity: measuring and predicting the multilingual capabilities of language models\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 7971–7989\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.468/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.468)Cited by:[§2\.2](https://arxiv.org/html/2608.25089#S2.SS2.p1.1),[§3](https://arxiv.org/html/2608.25089#S3.SS0.SSS0.Px7.p1.1)\.
- Vemulaet al\.\(2025\)S\. R\. Vemula, S\. Dandapat, D\. Sharma, and P\. KrishnamurthyRethinking tokenization for rich morphology: the dominance of unigram over BPE and morphological alignment\.InThe 14th International Joint Conference on Natural Language Processing and The 4th Conference of the Asia\-Pacific Chapter of the Association for Computational Linguistics,S\. T\.y\.s\.s, S\. Shimizu, and Y\. Gong \(Eds\.\),Mumbai, India,pp\. 232–252\.External Links:[Link](https://aclanthology.org/2025.ijcnlp-srw.20/),ISBN 979\-8\-89176\-304\-3Cited by:[§2\.1](https://arxiv.org/html/2608.25089#S2.SS1.SSS0.Px1.p2.1)\.
- Venkatesan \(2024\)M\. VenkatesanGender agreement in a tamil\-hindi bilingual situation: the role of feature valuation\.Formal Approaches to South Asian Languages\.Cited by:[§7](https://arxiv.org/html/2608.25089#S7.SS0.SSS0.Px3.p3.1)\.
- Wan \(2022\)A\. WanFairness in representation for multilingual NLP: Insights from controlled experiments on conditional language modeling\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.25089#S1.p3.1),[§2\.2](https://arxiv.org/html/2608.25089#S2.SS2.p1.1)\.
- Warstadtet al\.\(2020\)A\. Warstadt, A\. Parrish, H\. Liu, A\. Mohananey, W\. Peng, S\. Wang, and S\. R\. BowmanBLiMP: the benchmark of linguistic minimal pairs for english\.Transactions of the Association for Computational Linguistics8,pp\. 377–392\.Cited by:[§7](https://arxiv.org/html/2608.25089#S7.SS0.SSS0.Px3.p1.1)\.
- Wuet al\.\(2025\)M\. Wu, W\. Wang, S\. Liu, H\. Yin, X\. Wang, Y\. Zhao, C\. Lyu, L\. Wang, W\. Luo, and K\. ZhangThe bitter lesson learned from 2,000\+ multilingual benchmarks\.arXiv preprint arXiv:2504\.15521\.Cited by:[§7](https://arxiv.org/html/2608.25089#S7.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2608.25089#S7.SS0.SSS0.Px2.p1.1)\.
- Xueet al\.\(2022\)L\. Xue, A\. Barua, N\. Constant, R\. Al\-Rfou, S\. Narang, M\. Kale, A\. Roberts, and C\. RaffelByT5: towards a token\-free future with pre\-trained byte\-to\-byte models\.Transactions of the Association for Computational Linguistics10,pp\. 291–306\.Cited by:[§1](https://arxiv.org/html/2608.25089#S1.p3.1),[§2\.2](https://arxiv.org/html/2608.25089#S2.SS2.p2.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[Table 7](https://arxiv.org/html/2608.25089#A5.T7.2.5.2.1.1),[§6](https://arxiv.org/html/2608.25089#S6.SS0.SSS0.Px1.p1.1)\.
- Yanget al\.\(2026\)X\. Yang, C\. Arnett, and E\. G\. WilcoxCrosslingual language models\.Open Science Framework\.External Links:[Link](https://osf.io/qdw42/overview)Cited by:[Table 2](https://arxiv.org/html/2608.25089#A1.T2),[Table 3](https://arxiv.org/html/2608.25089#A1.T3),[§1](https://arxiv.org/html/2608.25089#S1.p5.1),[§4](https://arxiv.org/html/2608.25089#S4.SS0.SSS0.Px1.p1.1)\.
- Yükselet al\.\(2024\)A\. Yüksel, A\. Köksal, L\. K\. Senel, A\. Korhonen, and H\. SchuetzeTurkishMMLU: measuring massive multitask language understanding in Turkish\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 7035–7055\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.413/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.413)Cited by:[§7](https://arxiv.org/html/2608.25089#S7.SS0.SSS0.Px2.p1.1)\.
- Zemanet al\.\(2017\)D\. Zeman, M\. Popel, M\. Straka, J\. Hajič, J\. Nivre, F\. Ginter, J\. Luotolahti, S\. Pyysalo, S\. Petrov, M\. Potthast, F\. Tyers, E\. Badmaeva, M\. Gokirmak, A\. Nedoluzhko, S\. Cinková, J\. Hajič jr\., J\. Hlaváčová, V\. Kettnerová, Z\. Urešová, J\. Kanerva, S\. Ojala, A\. Missilä, C\. D\. Manning, S\. Schuster, S\. Reddy, D\. Taji, N\. Habash, H\. Leung, M\. de Marneffe, M\. Sanguinetti, M\. Simi, H\. Kanayama, V\. de Paiva, K\. Droganova, H\. Martínez Alonso, Ç\. Çöltekin, U\. Sulubacak, H\. Uszkoreit, V\. Macketanz, A\. Burchardt, K\. Harris, K\. Marheinecke, G\. Rehm, T\. Kayadelen, M\. Attia, A\. Elkahky, Z\. Yu, E\. Pitler, S\. Lertpradit, M\. Mandl, J\. Kirchner, H\. F\. Alcalde, J\. Strnadová, E\. Banerjee, R\. Manurung, A\. Stella, A\. Shimada, S\. Kwak, G\. Mendonça, T\. Lando, R\. Nitisaroj, and J\. LiCoNLL 2017 shared task: multilingual parsing from raw text to Universal Dependencies\.InProceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies,J\. Hajič and D\. Zeman \(Eds\.\),Vancouver, Canada,pp\. 1–19\.External Links:[Link](https://aclanthology.org/K17-3001/),[Document](https://dx.doi.org/10.18653/v1/K17-3001)Cited by:[§4](https://arxiv.org/html/2608.25089#S4.SS0.SSS0.Px2.p1.1)\.
- Zhang and Xu \(2022\)M\. Zhang and J\. XuByte\-based multilingual NMT for endangered languages\.InProceedings of the 29th International Conference on Computational Linguistics,N\. Calzolari, C\. Huang, H\. Kim, J\. Pustejovsky, L\. Wanner, K\. Choi, P\. Ryu, H\. Chen, L\. Donatelli, H\. Ji, S\. Kurohashi, P\. Paggio, N\. Xue, S\. Kim, Y\. Hahm, Z\. He, T\. K\. Lee, E\. Santus, F\. Bond, and S\. Na \(Eds\.\),Gyeongju, Republic of Korea,pp\. 4407–4417\.External Links:[Link](https://aclanthology.org/2022.coling-1.388/)Cited by:[§2\.2](https://arxiv.org/html/2608.25089#S2.SS2.p2.1)\.
- Ziemskiet al\.\(2016\)M\. Ziemski, M\. Junczys\-Dowmunt, and B\. PouliquenThe United Nations parallel corpus v1\.0\.InProceedings of the Tenth International Conference on Language Resources and Evaluation \(LREC’16\),N\. Calzolari, K\. Choukri, T\. Declerck, S\. Goggi, M\. Grobelnik, B\. Maegaard, J\. Mariani, H\. Mazo, A\. Moreno, J\. Odijk, and S\. Piperidis \(Eds\.\),Portorož, Slovenia,pp\. 3530–3534\.External Links:[Link](https://aclanthology.org/L16-1561/)Cited by:[Table 2](https://arxiv.org/html/2608.25089#A1.T2.2.1.1.1.1.1.1.13.1)\.
- Zosaet al\.\(2025\)E\. Zosa, J\. Louma, K\. Hakala, A\. Virtanen, M\. Koistinen, R\. Luukkonen, A\. Reunamo, S\. Pyysalo, and J\. BurdgePoro 2: continued pretraining for language acquisition\.Note:LumiOpenCited by:[§1](https://arxiv.org/html/2608.25089#S1.p1.1)\.

## Appendix AParallel Multilingual Corpus

The training data sources and split can be found in Table[2](https://arxiv.org/html/2608.25089#A1.T2)and Table[3](https://arxiv.org/html/2608.25089#A1.T3)\. We report the typological features for each language in[Table8](https://arxiv.org/html/2608.25089#A5.T8)\.

Table 2:Sources of the parallel\-10 and parallel\-3 corpora, from[Yang et al\. \(2026\)](https://arxiv.org/html/2608.25089#bib.bib10)\.Table 3:Data splits for three experimental settings across ten languages\. Word counts are computed using whitespace\-separated tokens for all languages except Chinese, for which each character is treated as a word, from[Yang et al\. \(2026\)](https://arxiv.org/html/2608.25089#bib.bib10)\.Table 4:Training HyperparametersAll the sources follow the data license from OPUS\([Tiedemann, 23\-25](https://arxiv.org/html/2608.25089#bib.bib40)\)\.

## Appendix BModel Training Details

The training hyperparameters are reported in Table[4](https://arxiv.org/html/2608.25089#A1.T4)\.

## Appendix CSupplementary results for Experiment1

### C\.1Dataset specific results for monolingualGPT2\-small

The crosslingual rankings for Parallel\-10 test split, and PUD are shown in Figures[8](https://arxiv.org/html/2608.25089#A5.F8),[9](https://arxiv.org/html/2608.25089#A5.F9)separately\. The correlation test are visualized in Figures[10](https://arxiv.org/html/2608.25089#A5.F10),[11](https://arxiv.org/html/2608.25089#A5.F11),[12](https://arxiv.org/html/2608.25089#A5.F12)separately\. We observe similar patterns as discussed in the main text\. In the correlation test, we find significant correlation between each metric and their potential confounds across three datasets, but this is not the case for Sent\-NLL\.

### C\.2Dataset\-specific results for monolingualGPT2\-medium

Scaling experiment results on different datasets can be found in[Figures5](https://arxiv.org/html/2608.25089#A3.F5),[6](https://arxiv.org/html/2608.25089#A3.F6)and[7](https://arxiv.org/html/2608.25089#A3.F7)\.

![Refer to caption](https://arxiv.org/html/2608.25089v1/heatmap_ranks_flores_scale.png)Figure 5:Language model rankings in a scaled setting \(FLORES\)\.![Refer to caption](https://arxiv.org/html/2608.25089v1/heatmap_ranks_pud_scale.png)Figure 6:Language model rankings in a scaled setting \(PUD\)\.![Refer to caption](https://arxiv.org/html/2608.25089v1/heatmap_ranks_test_scale.png)Figure 7:Language model rankings in a scaled setting \(Parallel\-3\)\.

## Appendix DParaphrases encode different amounts of information

Another concern may arise from our exclusion of paraphrases in Section[5](https://arxiv.org/html/2608.25089#S5)\. Including paraphrases from[Freitag et al\. \(2020a\)](https://arxiv.org/html/2608.25089#bib.bib32)could indeed change the results\. For instance, DElowestmight obtain lower values and potentially rank below English, based on the results from[Poelman and de Lhoneux \(2026\)](https://arxiv.org/html/2608.25089#bib.bib15)\. However, this doubt relies on a common, but flawed, assumption that paraphrases within the same language should receive similar probabilities from a language model because they express thesame meaning, i\.e\., they are assumed to be semantically identical\([Barzilay and McKeown, 2001](https://arxiv.org/html/2608.25089#bib.bib78);[Boonthum, 2004](https://arxiv.org/html/2608.25089#bib.bib77);[Madnani and Dorr, 2010](https://arxiv.org/html/2608.25089#bib.bib74)\)and therefore language models should assign similar probabilities to paraphrases of the same text\([Poelman and de Lhoneux, 2026](https://arxiv.org/html/2608.25089#bib.bib15)\)\.777For example,[Poelman and de Lhoneux \(2026\)](https://arxiv.org/html/2608.25089#bib.bib15)assume that paraphrases have the same information content and therefore language models should assign “roughly the same \[Token NLL or BPC\] values for paraphrases within a single language”\.

We argue that this assumption does not necessarily hold\. Paraphrases are often semantically similar but not strictly identical, and they encode different amounts of information\. Consequently, there is no theoretical reason to expect that language models should assign more similar probabilities to two paraphrases than to translations of either sentence\.

To illustrate this point, we use two paraphrases of a sentence from[Poelman et al\. \(2025\)](https://arxiv.org/html/2608.25089#bib.bib52)as an example:

\\ex

\. Ḋo you know how to play chess? \.̱ Can you play chess?

“Know how to play chess” entails possessing the procedural knowledge about the rules of chess\. “Can play chess” entails knowledge of the rules, but it is more sensitive to context\. It could refer to possession of a chessboard or whether it is socially acceptable to play\.

While in many contexts these differences may not be important, paraphrases are rarely semantically equivalent\. Even grammatical alternations, e\.g\. dative alternation \(c\.f\. “I threw the box to John” and “I threw John the box\.”;[Levin, 1993](https://arxiv.org/html/2608.25089#bib.bib89)\), may not be considered semantically equivalent\([Bresnan and Nikitina, 2009](https://arxiv.org/html/2608.25089#bib.bib88)\)\. It may be more precise to label the sentences in Ex\.[D](https://arxiv.org/html/2608.25089#A4)asquasi\-paraphrases\([Bhagat and Hovy, 2013](https://arxiv.org/html/2608.25089#bib.bib2)\)ornear\-equivalence\([Santos and Barreiro, 2025](https://arxiv.org/html/2608.25089#bib.bib76)\), which may encompass a “broader, approximate, equivalence”\.

Admittedly, both translations and paraphrases may encode different amounts of information for the same source\. However, paraphrases typically permit substantially greater variation because they are not constrained by crosslinguistic correspondence\([Neusner, 1986](https://arxiv.org/html/2608.25089#bib.bib75), e\.g\.,\)\. As a result, inconsistencies observed with paraphrases do not necessarily undermine crosslingual evaluation\. Instead, they introduce an additional and less controlled source of variation\.

## Appendix ECase study withGPT2

To to illustrate the bias on perplexity introduced by differences in compression, we provide a case study usingopenai/GPT\-2model \(monolingual English\) evaluated on four languages from FLORES\. As shown in Table[6](https://arxiv.org/html/2608.25089#A5.T6), token\-level metrics \(PPL, token\-NLL, MRR\) all suggest that the model performs best on Russian\. This is counterintuitive given that the model is trained only on English, and MultiBLiMP results \(Table[5](https://arxiv.org/html/2608.25089#A5.T5)\)888For Chinese, it is evaluated on ZhoBLiMP\([Liu et al\., 2026](https://arxiv.org/html/2608.25089#bib.bib11)\)VerbPhrasecategory\.show that GPT2 performs well only on English \(0\.97\) and near chance on other languages \(in fact, below chance in Russian\)\. In contrast, Sentence\-level NLL assigns the lowest score to English, aligning better with expectations and the MultiBLiMP evaluations\.

Table 5:MultiBLiMP results of GPT\-2 across four languages\.Table 6:crosslingual evaluation results of an English language model on FLORES\.Table 7:Models used in this study and their pretraining data composition\.![Refer to caption](https://arxiv.org/html/2608.25089v1/heatmap_ranks_parallel.png)Figure 8:Language rankings across different intrinsic metrics for Parallel\-10 test split![Refer to caption](https://arxiv.org/html/2608.25089v1/heatmap_ranks_pud.png)Figure 9:Language rankings across different intrinsic metrics for PUDFigure 10:Spearman’s Rank Correlation between the intrinsic metrics and their potential confounding factors in FLORES \(perplexity is scaled by log10\)\.Figure 11:Spearman’s Rank Correlation between the intrinsic metrics and their potential confounding factors in Parallel10 \(perplexity is scaled by log10\)\.Figure 12:Spearman’s Rank Correlation between the intrinsic metrics and their potential confounding factors in PUD \(perplexity is scaled by log10\)\.Table 8:Typological properties of the languages used in our experiments\.
## Appendix FMultilingual Experiment

Results formGPTandXGLM\-4\.5Bcan be found in[Figure13](https://arxiv.org/html/2608.25089#A6.F13)and the raw results for all multilingual evaluated can be found in[Figure14](https://arxiv.org/html/2608.25089#A6.F14)\.

Figure 13:Expected ranking and actual ranking across 6 metrics formGPTandXGLM\-4\.5B\.Figure 14:Raw results for all models across six evaluation metrics