Language Models are not Equally Robust to Non-Canonical Tokenization across Languages

# Language Models are not Equally Robust to Non-Canonical Tokenization across Languages
Source: [https://arxiv.org/html/2607.26831](https://arxiv.org/html/2607.26831)
###### Abstract

Despite the existence of exponentially many valid tokenizations for a given string, language models operate on a single canonical sequence deterministically produced by the tokenizer, leaving the broader tokenization space largely uncharacterized\. In this paper, we investigate this overlooked space by studying the behavior of language models under non\-canonical tokenizations across diverse languages\. For English, prior work shows that models are largely invariant to alternative tokenizations that represent the same underlying string\. We ask whether this invariance generalizes to other languages beyond English\. We conduct a multilingual study across 27 languages spanning diverse scripts and evaluate LLM behavior under alternative tokenizations across six downstream tasks\. We find that tokenization invariance does not generalize: model behavior varies substantially across languages with instruction\-tuned models exhibiting an average relative performance drop of 23\.7% for Llama\-3\.1\-8B, 11\.4% for Qwen3\-8B, and 9\.9% for Gemma\-3\-12B\.The variation of tokenization invariance is systematic across languages\. Languages that exhibit higher token fragmentation show significantly greater sensitivity to non\-canonical tokenizations\. Our study of tokenization robustness serves as a diagnostic of how tightly a model is coupled to its tokenizer\. These results demonstrate that tokenization robustness is not a universal property of language models, but depends strongly on the language and its interaction with the tokenizer\. We show that LoRA fine\-tuning with multi\-tokenization training data provides an effective mitigation for tokenization sensitivity\. Fine\-tuning on English alone improves tokenization robustness across languages, while systematically sampling diverse non\-canonical tokenizations achieves the strongest overall performance\.

Language Models are not Equally Robust to Non\-Canonical Tokenization across Languages

Poulami Ghosh∗, Preethi Jyothi∗∗IIT Bombay, India\{poulami, pjyothi\}@cse\.iitb\.ac\.in

## 1Introduction

Tokenization plays a central role in modern language models, determining how raw text is segmented into the discrete units processed by the model\. Most contemporary systems rely on subword tokenization algorithms such asSennrichet al\.\([2016](https://arxiv.org/html/2607.26831#bib.bib15)\)\(Byte Pair Encoding\),Kudo and Richardson \([2018](https://arxiv.org/html/2607.26831#bib.bib16)\)\(SentencePiece\),Wuet al\.\([2016](https://arxiv.org/html/2607.26831#bib.bib17)\)\(WordPiece\),Kudo \([2018](https://arxiv.org/html/2607.26831#bib.bib19)\)\(UnigramLM\)\. Although tokenization is often treated as a preprocessing step, it directly shapes the model’s input distribution and therefore the representations learned during training\. The tokenizer typically produces a canonical tokenization, a deterministic segmentation of the input text according to its tokenization algorithm, which defines the input distribution seen during pretraining\. However, the same text can often be represented using many alternative segmentations composed entirely of valid vocabulary tokens\. These non\-canonical tokenizations preserve the surface text while altering only the sequence of input tokens\. Since contemporary LLMs are trained almost exclusively on canonical segmentations, such inputs lie outside the training distribution\. Consequently, one would generally expect degradation in model performance under such perturbation\. Surprisingly, however,Zhenget al\.\([2025](https://arxiv.org/html/2607.26831#bib.bib20)\)found that modern instruction\-tuned LLMs remain remarkably robust to non\-canonical tokenizations in English across a range of downstream tasks\. They hypothesize that this robustness emerges during post\-training, where models are encouraged to respond fluently regardless of variations in the tokenization of user instructions\.

![Refer to caption](https://arxiv.org/html/2607.26831v1/x1.png)Figure 1:LLM robustness to non\-canonical tokenization isNOTuniversal and varies substantially across languages, models, and tasks\. Models are "broken" in multilingual settings\.Whether this surprising robustness extends beyond English remains unclear\. English occupies a privileged position in multilingual tokenizers: it is heavily represented during tokenizer construction, exhibits relatively simple morphology, and benefits from whitespace\-delimited segmentation\. In contrast, multilingual tokenizers must allocate a fixed vocabulary across many languages with diverse scripts and morphological structures\. Prior work has shown that vocabulary allocation is highly uneven across languagesLimisiewiczet al\.\([2023](https://arxiv.org/html/2607.26831#bib.bib33)\); Petrovet al\.\([2023](https://arxiv.org/html/2607.26831#bib.bib38)\); Ahiaet al\.\([2023](https://arxiv.org/html/2607.26831#bib.bib39)\), particularly disadvantaging lower\-resource languages\. Taken together, these factors suggest that robustness findings derived from English cannot be assumed to generalize to other languages\.

Although non\-canonical tokenizations lie outside the training distribution, they are not merely synthetic perturbations\. They naturally arise in several practical settings where the tokenizer is modified after pretraining\. Recent work on vocabulary pruning and restructuringReifet al\.\([2026](https://arxiv.org/html/2607.26831#bib.bib40)\); Purasonet al\.\([2026](https://arxiv.org/html/2607.26831#bib.bib41)\); Kim and Kim \([2026](https://arxiv.org/html/2607.26831#bib.bib42)\)aims to improve efficiency and language coverage by removing redundant tokens or reallocating vocabulary capacity, causing the same text to be segmented differently after tokenizer adaptation\. Similarly, cross\-lingual tokenizer transfer methods replace or adapt the original tokenizer to better support new languages or domains through through embedding transfer or token alignmentRemyet al\.\([2024](https://arxiv.org/html/2607.26831#bib.bib43)\); Liet al\.\([2025](https://arxiv.org/html/2607.26831#bib.bib44),[2026](https://arxiv.org/html/2607.26831#bib.bib45)\)\. Such tokenizer replacement can also change how the same input would originally get segmented\. Beyond tokenizer adaptation,Gehet al\.\([2025](https://arxiv.org/html/2607.26831#bib.bib21)\)show that adversarially selected non\-canonical tokenizations of malicious strings can bypass LLM safety and alignment mechanisms while preserving the underlying text, although this has only been demonstrated for English\. These demonstrate practical scenarios where a model may encounter non\-canonical tokenization during inference and motivate understanding how robust multilingual language models are to alternative valid tokenizations\.

In this work, we systematically study how multilingual language models respond to non\-canonical tokenization across languages\. Rather than degrading uniformly, multilingual LLMs exhibit strikingly different levels of robustness to non\-canonical tokenization across models, languages, and tasks\. Our findings reveal that tokenization robustness is not simply a by\-product of stronger language models, but a distinct property of multilingual language models shaped by language\-specific tokenization characteristics such as token fragmentation\. Moreover, the effects extend beyond text generation\. Even purely semantic tasks like cross\-lingual sentence retrieval that rely on sentence\-level embeddings are significantly impacted by non\-canonical tokenizations\. Our contributions are: 1\.We present the first systematic study of non\-canonical tokenization in multilingual LLMs, spanning 27 languages, six downstream tasks, and three recent model families\. 2\.We analyze how non\-canonical tokenization affects multilingual language models differently across language families and scripts\. 3\.We show that non\-canonical tokenization serves as a natural source of data augmentation for improving multilingual generalization\.

## 2Related Work

#### Robustness to Non\-Canonical Tokenization\.

Recent work challenges the assumption that language models critically depend on a fixed canonical tokenization\.Zhenget al\.\([2025](https://arxiv.org/html/2607.26831#bib.bib20)\)show that modern LLMs retain strong performance even under heavily perturbed non\-canonical tokenizations, suggesting substantial robustness at inference time\. Similarly,Gehet al\.\([2025](https://arxiv.org/html/2607.26831#bib.bib21)\)demonstrate that adversarial tokenizations can significantly alter model behavior, including bypassing alignment safeguards, without changing the underlying text\. Together, these works establish that tokenization is not merely a preprocessing step but directly influences inference\. However, both studies are primarily conducted in English and do not examine how such robustness or sensitivity varies across languages with different scripts\.

#### Tokenization as a Latent Probabilistic Space\.

A complementary line of work treats tokenization as a source of latent variability in language modeling\.Cao and Rimell \([2021](https://arxiv.org/html/2607.26831#bib.bib23)\)andChirkovaet al\.\([2023](https://arxiv.org/html/2607.26831#bib.bib24)\)argue that evaluation should account for multiple tokenizations to better approximate the true likelihood of a sequence\.Gehet al\.\([2024](https://arxiv.org/html/2607.26831#bib.bib22)\)introduces the idea that tokenization space contains latent signal beyond the canonical sequence\. They show that aggregating probabilities across tokenizations can uncover useful signal and improve evaluation performance\. While these works highlight the importance of tokenization beyond a single canonical segmentation, they focus primarily on likelihood estimation rather than its effect on generative downstream performance, particularly in multilingual settings\.

#### Learning Under Tokenization Variability\.

Prior work has also explored exposing models to multiple tokenizations during training\.Cognettaet al\.\([2024](https://arxiv.org/html/2607.26831#bib.bib28)\)analyze subword regularization, showing that sampling alternative segmentations improves robustness and generalization\.Stegeret al\.\([2026](https://arxiv.org/html/2607.26831#bib.bib46)\)demonstrated that training with stochastically sampled non\-canonical tokenizations improves robustness to both random and adversarial tokenization perturbations, with uniform sampling strategies providing stronger robustness while preserving canonical performance\. At the extreme,Vieiraet al\.\([2024](https://arxiv.org/html/2607.26831#bib.bib29)\)eliminate tokenization entirely by modeling text at the character level\. These approaches suggest that models can benefit from tokenization variability, but they focus on training\-time interventions or alternative architectures, rather than inference\-time manipulation of tokenization in pretrained LLMs\.

However, all these directions remain largely disconnected and predominantly English\-centric\. In particular, there is limited understanding of how non\-canonical tokenization affects inference across languages from different language families with diverse scripts\. Our work addresses this gap by systematically studying the impact of non\-canonical tokenization on LLM inference in a multilingual setting\.

## 3Methodology

LetΣ\\Sigmabe a finite alphabet andΣ∗\\Sigma^\{\*\}the set of all finite strings overΣ\\Sigma\. LetVVdenote a tokenizer vocabulary, andV∗V^\{\*\}the set of finite token sequences\.

#### Tokenization and decoding\.

A tokenizer induces a*deterministic canonical tokenization*

Tcanon:Σ∗→V∗,tcanon=Tcanon​\(x\),T\_\{\\mathrm\{canon\}\}:\\Sigma^\{\*\}\\rightarrow V^\{\*\},\\quad t\_\{\\mathrm\{canon\}\}=T\_\{\\mathrm\{canon\}\}\(x\),which maps any stringx∈Σ∗x\\in\\Sigma^\{\*\}to a unique token sequence\.

A*decoding function*decode:V∗→Σ∗\\mathrm\{decode\}:V^\{\*\}\\rightarrow\\Sigma^\{\*\}maps token sequences back to strings, satisfying

decode​\(Tcanon​\(x\)\)=x\.\\mathrm\{decode\}\(T\_\{\\mathrm\{canon\}\}\(x\)\)=x\.
We define the set of all valid tokenizations of a stringxxas:

𝒯​\(x\)=\{t∈V∗:decode​\(t\)=x\}\.\\mathcal\{T\}\(x\)=\\\{\\,t\\in V^\{\*\}\\;:\\;\\mathrm\{decode\}\(t\)=x\\,\\\}\.By construction,Tcanon​\(x\)∈𝒯​\(x\)T\_\{\\mathrm\{canon\}\}\(x\)\\in\\mathcal\{T\}\(x\)\.

#### Tokenization invariance\.

Let a language model be a function

f:V∗→ℝd,f:V^\{\*\}\\rightarrow\\mathbb\{R\}^\{d\},mapping token sequences to outputs \(e\.g\., logits, representations, or task predictions\)\. We say the model satisfies*tokenization invariance*if for tokenizationst1,t2∈𝒯​\(x\)t\_\{1\},t\_\{2\}\\in\\mathcal\{T\}\(x\),

f​\(t1\)≈f​\(t2\)\.f\(t\_\{1\}\)\\approx f\(t\_\{2\}\)\.This captures the ideal requirement that model behavior should depend on the underlying string x, not on how it is segmented into tokens\. Given two tokenizationst1,t2∈𝒯​\(x\)t\_\{1\},t\_\{2\}\\in\\mathcal\{T\}\(x\), we quantify the deviation from tokenization invariance as:

Err​\(t1,t2\)=d​\(f​\(t1\),f​\(t2\)\),\\mathrm\{Err\}\(t\_\{1\},t\_\{2\}\)=d\\big\(f\(t\_\{1\}\),f\(t\_\{2\}\)\\big\),whereddmeasures tokenization\-induced performance gap\. In our evaluation setup, we compare a reference canonical tokenization with different non\-canonical tokenizations\.

![Refer to caption](https://arxiv.org/html/2607.26831v1/x2.png)\(a\)English
![Refer to caption](https://arxiv.org/html/2607.26831v1/x3.png)\(b\)Russian
![Refer to caption](https://arxiv.org/html/2607.26831v1/x4.png)\(c\)Thai
![Refer to caption](https://arxiv.org/html/2607.26831v1/x5.png)\(d\)Hindi

Figure 2:MDD of the word “time” across different languages\.
#### Tokenization types\.

We evaluate models under three tokenization settings\.

##### \(1\) Canonical tokenization\.

tc:=Tcanon​\(x\)\.t\_\{c\}:=T\_\{\\mathrm\{canon\}\}\(x\)\.
This is the default segmentation used at inference time\.

##### \(2\) Random non\-canonical tokenization\.

A non\-canonical tokenization is any tokenizationttwhere

t∈𝒯​\(x\),s\.t\.​t≠tct\\in\\mathcal\{T\}\(x\),\\quad\\text\{s\.t\. \}t\\neq t\_\{c\}FollowingGehet al\.\([2025](https://arxiv.org/html/2607.26831#bib.bib21)\), we represent the space of all tokenizations using an Multi\-valued Decision Diagram \(MDD\) as illustrated in Figure[2](https://arxiv.org/html/2607.26831#S3.F2)\. A Multi\-valued Decision Diagram \(MDD\) can be used to represent all possible tokenizations of a given string\.In this representation, each node corresponds to a position in the input string, while edges between nodes are labeled with the tokens they represent\. More specifically, an edge from nodeiito nodejjis associated with the tokenxi:j=\(xi,xi\+1,…,xj\)x\_\{i:j\}=\(x\_\{i\},x\_\{i\+1\},\\ldots,x\_\{j\}\)\. Any path from the root node to a terminal node corresponds to one valid tokenization of the stringxx\. Using an MDD, we can uniformly sample from the space of all tokenizations by randomly selecting edges weighted by the number of possible tokenizations in the corresponding sub\-MDD\. We generateNNsuch samples:

tn∼𝒰​\(𝒯​\(x\)\),n∈1,…,Nt\_\{n\}\\sim\\mathcal\{U\}\(\\mathcal\{T\}\(x\)\),n\\in\{1,\\ldots,N\}where𝒰\\mathcal\{U\}denotes sampling uniformly at random\.

##### \(3\) Character\-level tokenization\.

Letx=\(x1,…,x\|x\|\)x=\(x\_\{1\},\\dots,x\_\{\|x\|\}\)\. The character\-level tokenization is

tchar=\(x1,x2,…,x\|x\|\),t\_\{\\mathrm\{char\}\}=\(x\_\{1\},x\_\{2\},\\dots,x\_\{\|x\|\}\),where each symbol is treated as an individual token\. By construction,tchar∈𝒯​\(x\)t\_\{\\mathrm\{char\}\}\\in\\mathcal\{T\}\(x\)\.

#### Evaluation\.

For a given stringxx, we compare model outputs across tokenizations:

Err​\(tc,tn\),Err​\(tc,tchar\)\\mathrm\{Err\}\(t\_\{c\},t\_\{n\}\),\\quad\\mathrm\{Err\}\(t\_\{c\},t\_\{\\mathrm\{char\}\}\)All tokenizations are constrained to lie in𝒯​\(x\)\\mathcal\{T\}\(x\), ensuring the decoded string is identical and differences arise solely from token segmentation\. Thus, any observed differences inf\(\.\)f\(\.\)isolate the model’s sensitivity to tokenization\.

## 4Experimental Setup

![Refer to caption](https://arxiv.org/html/2607.26831v1/x6.png)\(a\)Llama
![Refer to caption](https://arxiv.org/html/2607.26831v1/x7.png)\(b\)Qwen
![Refer to caption](https://arxiv.org/html/2607.26831v1/x8.png)\(c\)Gemma

Figure 3:Fragmentation across languages under canonical and character\-level tokenization, measured on the FLORES\-200 benchmark\.We conduct experiments using the lm\-evaluation\-harness frameworkGaoet al\.\([2023](https://arxiv.org/html/2607.26831#bib.bib34)\), which provides a standardized interface for evaluating language models across a wide range of benchmarks\. We extend the framework to support alternative tokenization schemes by introducing preprocessing modules that generate non\-canonical tokenizations and character\-level tokenizations before model inference\. These perturbations preserve the original text string but alter the token segmentation supplied to the model\. For random non\-canonical tokenization, we sample uniformly at random from the space of all tokenizations across 5 different seeds and report the mean\.

#### Language Models\.

The main experiments in this paper are conducted on several instruction\-tuned models, namely Llama\-3\.1\-8B\-InstructGrattafioriet al\.\([2024](https://arxiv.org/html/2607.26831#bib.bib9)\), Qwen3\-8BYanget al\.\([2025](https://arxiv.org/html/2607.26831#bib.bib31)\), and gemma\-3\-12b\-itTeamet al\.\([2025](https://arxiv.org/html/2607.26831#bib.bib32)\), which are referred to throughout the paper asLlama,Qwen, andGemma, respectively\.

#### Fragmentation Rate\.

To quantify tokenization behavior, we compute the average number of tokens per word for each language\. This metric measures the degree of token fragmentation produced by the tokenizer\.

Before evaluating robustness to different tokenizations, we first analyze how the tokenizer segments text across languages under canonical and character\-level tokenization using the many\-to\-many multilingual FLORES\-200 benchmarkCosta\-Jussàet al\.\([2022](https://arxiv.org/html/2607.26831#bib.bib30)\), as illustrated in Figure[3](https://arxiv.org/html/2607.26831#S4.F3)\. Canonical tokenization represents the tokenizer’s default behavior, while character\-level tokenization provides the maximum possible fragmentation\. Together, they define the practical range of fragmentation exhibited by the tokenizer across languages\.111In principle, tokenizations shorter than the canonical tokenization may exist; however, such cases are small in number, and hence we treat canonical and character\-level tokenization as the practical lower and upper bounds of fragmentation\.

#### Languages\.

We conduct our main evaluation on languages that are supported across the models being compared to ensure fair cross\-model comparisons\. Since Gemma does not provide an official list of supported languages, we use the Qwen language set as a proxy\.222Looking at the canonical performance of Gemma on multilingual ARC, we do see that many of the languages have reasonably good support \(with ¿0\.8 task performance\)\. Also for every other task in Table[1](https://arxiv.org/html/2607.26831#S5.T1), the performance on this subset of languages using Gemma is better than with using Qwen\. This gives us confidence that these are indeed languages on which Gemma is reasonably proficient\.

As shown in Figure[3\(b\)](https://arxiv.org/html/2607.26831#S4.F3.sf2), several languages, namely Bengali, Gujarati, Armenian, Kannada, Malayalam, Marathi, Nepali, Tamil, and Telugu, exhibit canonical behavior that is nearly equivalent to character\-level tokenization for Qwen\. For these languages, canonical tokenization is already near the maximum fragmentation, leaving little room to construct meaningful non\-canonical variants and making the comparison with other languages qualitatively different\. Importantly, these languages are also not officially supported by Llama, leaving Gemma as the only model for which meaningful evaluation is possible\. Since our goal is to compare robustness consistently across model families, we exclude these languages from the main analysis to ensure a fair and interpretable evaluation across languages\. We present a separate analysis in Section[C](https://arxiv.org/html/2607.26831#A3)in the Appendix to highlight their distinctive tokenization behavior\. The final list of languages consists of 27 languages spanning diverse language families and scripts, as shown in Table[9](https://arxiv.org/html/2607.26831#A4.T9)\.

#### Task\.

We perform generation\-based evaluation on both short\-answer tasks \(MLQA, MGSM\) and multiple\-choice tasks \(Multi\-ARC, Multi\-HellaSwagLaiet al\.\([2023](https://arxiv.org/html/2607.26831#bib.bib35)\), Global\-MMLU\-LiteSinghet al\.\([2025](https://arxiv.org/html/2607.26831#bib.bib37)\), and BelebeleBandarkaret al\.\([2024](https://arxiv.org/html/2607.26831#bib.bib36)\)\)\. Details about tasks is present in Table[8](https://arxiv.org/html/2607.26831#A4.T8)\.

We retokenize the entire prompt string, including the instruction and the multiple choice answer choices\. We do not retokenize the system prompt or special tokens\.

![Refer to caption](https://arxiv.org/html/2607.26831v1/x9.png)\(a\)Llama
![Refer to caption](https://arxiv.org/html/2607.26831v1/x10.png)\(b\)Qwen
![Refer to caption](https://arxiv.org/html/2607.26831v1/x11.png)\(c\)Gemma

Figure 4:Impact of granularity of non\-canonical tokenization on performance on Multilingual ARC

## 5Results

Table 1:Generation\-based evaluation across multilingual benchmarks\. Scores are averaged across languages\. We report the relative performance drop from canonical tokenization under randomly sampled non\-canonical tokenization \(RandΔ\\Delta%\) and character\-level tokenization \(CharΔ\\Delta%\)\. MLQA and MGSM are short\-answer \(SA\) tasks; Multi\-HellaSwag, Multi\-ARC, Global\-MMLU\-Lite, and Belebele are multiple\-choice \(MC\)\. Performance degrades when inputs use non\-canonical tokenization\.### 5\.1Impact of non\-canonical tokenization

As observed in Table[1](https://arxiv.org/html/2607.26831#S5.T1), the results demonstrate that multilingual language models are highly sensitive to tokenization structure, even when the underlying input string remains semantically identical\. Across all benchmarks, canonical tokenization consistently yields the highest performance, while both random non\-canonical and character tokenizations lead to measurable degradation\. Character tokenization causes substantially larger performance drops than random non\-canonical tokenization for every model and task, indicating that increased segmentation fragmentation severely disrupts model representations\. Among the evaluated models, Qwen3\-8B exhibits the strongest robustness, with only 11\.39% average degradation under random tokenization and 16\.09% under character tokenization, suggesting stronger invariance to alternative token decompositions\. In contrast, Llama\-3\.1\-8B\-Instruct shows the highest vulnerability, with degradation increasing from 23\.77% to 40\.96%, revealing a strong dependence on canonical subword segmentation\. Gemma\-3\-12B\-IT achieves the highest canonical performance overall but displays intermediate robustness, remaining stable under random tokenization while degrading more substantially under character\-level fragmentation\. At task\-level, we observe that that short\-answer benchmarks such as MGSM and MLQA are more sensitive to tokenization variation, whereas multi\-choice benchmarks namely Global\-MMLU, Multi\-ARC, Multi\-HellaSwag and Belebele remains comparatively robust across models\.

For Multilingual\-ARC, we analyze whether fragmentation rate is predictive of model robustness under non\-canonical tokenization\. For each example, we compute the canonical fragmentation rate \(FR\) and evaluate model performance under both canonical and non\-canonical tokenizations\. We then fit a logistic regression model where the dependent variable is non\-canonical performance \(correct/incorrect\), and the predictors are canonical performance and canonical fragmentation rate\.

For English, canonical performance is a strong positive predictor of non\-canonical performance \(β=3\.42\\beta=3\.42,p<0\.001p<0\.001\), indicating that examples answered correctly under canonical tokenization are substantially more likely to remain correct under non\-canonical tokenizations\. Canonical fragmentation rate exhibits a significant negative correlation with non\-canonical performance \(β=−1\.11\\beta=\-1\.11,p=0\.021p=0\.021\), suggesting that examples with more fragmented canonical tokenizations are more vulnerable to performance degradation under non\-canonical tokenizations\. For analysis on more languages, please refer to Table[10](https://arxiv.org/html/2607.26831#A4.T10)\.

### 5\.2Impact of granularity

We study the impact of non\-canonical tokenization granularity by sampling non\-canonical tokenizations whose fragmentation ratio falls within a bucket relative to the canonical tokenization\. Lower buckets correspond to tokenizations close to the canonical form, while higher buckets contain increasingly fragmented tokenizations approaching character\-level segmentation\. This helps analyze how increasing token fragmentation affects model performance and robustness\. For each example, we compute the feasible range described in Section[4](https://arxiv.org/html/2607.26831#S4), which identifies the set of bucket ranges from which a valid non\-canonical tokenization can be sampled across different languages\. As shown in Figure[4](https://arxiv.org/html/2607.26831#S4.F4), the results reveal a strong and systematic relationship between tokenization granularity and model performance across all three language models\. As tokenization moves progressively farther from the canonical segmentation towards increasingly fragmented non\-canonical forms task performance generally declines across languages and models, demonstrating that multilingual performance is highly sensitive to token decomposition\. Llama\-3\.1\-8B\-Instruct exhibits the steepest degradation overall, with performance dropping sharply as granularity increases\. For example, languages such as German, French, Portuguese show substantial drop in performance as the granularity increases\. English is comparatively more stable, suggesting that models are substantially more robust for high\-resource languages that dominate pretraining corpora\.

Qwen3\-8B demonstrates the strongest robustness and smoothest degradation curves across most languages\. Several languages such as Portuguese, Russian, French, and English remain remarkably stable even at high granularity levels, with English preserving performance near canonical accuracy throughout the entire range\. In contrast, lower\-resource languages such as Basque, Dutch, and Hungarian show larger declines, indicating that robustness is correlated with language representation during training\. Gemma\-3\-12B\-IT achieves the highest canonical accuracies overall but exhibits mixed robustness patterns\. High\-resource languages including English, Spanish, German, French, and Russian remain relatively stable under moderate granularity increases, whereas languages such as Dutch, Swedish, Croatian, and Basque experience severe deterioration at higher bucket levels\.

#### Controlling for Sequence Length

One possible explanation for the observed degradation under non\-canonical tokenization is the increase in sequence length rather than changes in token boundaries themselves\. To control for this effect, we repeat the Multilingual\-ARC experiments using only non\-canonical tokenizations whose lengths closely match the canonical tokenization \(length\-ratio buckets of 0\.95–1 and 1–1\.05\)\. Results are present in Table[6](https://arxiv.org/html/2607.26831#A3.T6)in the Appendix\. Despite nearly identical sequence lengths, the performance gap largely persists across languages, indicating that the observed degradation cannot be explained solely by longer input sequences\. These findings suggest that changes in token boundaries themselves contribute substantially to the loss in performance\.

Table 2:Results on scaling experiments\. For multi\-choice tasks, namely MMLU and ARC, Qwen3\-4B model fails to follow system instruction and hence the results are arbitrary for small scale models\. For short\-answer tasks, we observe across tasks, that the robustness increases with scale\.

### 5\.3Impact of scale

As observed in Table[2](https://arxiv.org/html/2607.26831#S5.T2), the scaling experiments reveal several notable trends regarding the effect of model size on robustness\. For multilingual question answering \(MLQA\), canonical performance consistently improves with scale, increasing from 47\.56 for Qwen3\-4B to 65\.33 for Qwen3\-14B\. At the same time, the relative robustness degradation under randomization gradually decreases from 14\.65% to 10\.84%, indicating that larger models are not only more accurate but also modestly more robust to non\-canonically tokenized context\.

A similar trend is observed onMGSM, where canonical performance remains nearly constant across model scales \(44–45\), while RandΔ%\\Delta\\%steadily decreases from 21\.09% for the 4B model to 16\.21% for the 14B model\. This suggests that scaling primarily improves robustness on multilingual mathematical reasoning rather than baseline task performance\.

For multiple\-choice reasoning benchmarks such asGlobal\-MMLU\-LiteandMulti\-ARC, the 4B model exhibits unstable and unreliable behavior, often failing to follow system instructions, which results in negative or highly arbitrary robustness scores\. In contrast, the 8B and 14B models show more consistent behavior and achieve substantially higher canonical performance while maintaining low robustness degradation\. This indicates that scaling primarily improves instruction\-following and output consistency for smaller models rather than robustness itself\.

Overall, the results provide evidence that increasing model scale improves both baseline task performance and robustness\. The most significant gains occur when scaling from 4B to 8B parameters, whereas the improvements from 8B to 14B are comparatively modest, suggesting diminishing returns with further increases in model size\.

![Refer to caption](https://arxiv.org/html/2607.26831v1/x12.png)\(a\)Normal
![Refer to caption](https://arxiv.org/html/2607.26831v1/x13.png)\(b\)Random
![Refer to caption](https://arxiv.org/html/2607.26831v1/x14.png)\(c\)Character

Figure 5:Retrieval experiments on FLORES\-200 for Llama
### 5\.4Cross\-lingual experiments

In addition to benchmark evaluation, we further probe into how token segmentations influence the LLM’s internal representations\. We perform a cross\-lingual sentence retrieval experiment using the FLORES\-200 multilingual benchmark dataset\. Following the standard retrieval setup we construct bilingual sentence pairs and evaluate retrieval accuracy\. Given a query sentence in a source language, we compute its embedding and retrieve the closest sentence in the target language using cosine similarity,

sim​\(𝐱,𝐲\)=𝐱⋅𝐲‖𝐱‖​‖𝐲‖\\text\{sim\}\(\\mathbf\{x\},\\mathbf\{y\}\)=\\frac\{\\mathbf\{x\}\\cdot\\mathbf\{y\}\}\{\\\|\\mathbf\{x\}\\\|\\,\\\|\\mathbf\{y\}\\\|\}where𝐱\\mathbf\{x\}and𝐲\\mathbf\{y\}denote the sentence embeddings of the source sentencexxand target sentenceyy, respectively\. We report top\-1 retrieval accuracy, which measures the proportion of queries for which the correct translation is ranked highest among the candidates\. To evaluate robustness to tokenization perturbations, we compute sentence embeddings under different tokenization settings \(canonical, non\-canonical, and character\-level\)\. Because the underlying text remains unchanged across conditions, differences in retrieval performance reflect changes in how token segmentation influences the model’s multilingual representations\. Sentence embeddings are obtained by the final hidden state of the last token of the model\.

The results for Llama, Qwen, and Gemma are presented in Figures[5](https://arxiv.org/html/2607.26831#S5.F5),[6](https://arxiv.org/html/2607.26831#A4.F6), and[7](https://arxiv.org/html/2607.26831#A4.F7), respectively\. Across all three models, canonical tokenization consistently gives the highest cross\-lingual retrieval accuracy, while random and character\-level tokenization reduce performance\. High\-resource European languages achieve the best retrieval scores but also suffer the largest degradation under tokenization perturbations, showing strong dependence on subword segmentation\. In contrast, character\-based languages such as Chinese, Japanese, and Korean are generally more robust, with smaller drops under character tokenization\. Thai is the most sensitive language across models, while low\-resource languages such as Yoruba remain difficult regardless of tokenization setting\.

Among the models, Qwen3\-8B is the most robust to tokenization changes, maintaining relatively strong retrieval performance even under character tokenization\. Llama\-3\.1\-8B\-Instruct achieves high peak retrieval accuracy under canonical tokenization but shows degradation under non\-canonical tokenization\. Gemma\-3\-12B\-it performs worst overall and is especially sensitive to tokenization\. Overall, the experiments show that multilingual retrieval quality depends not only on semantics but also heavily on tokenizer robustness across scripts and language families\.

## 6Non\-Canonical Tokenization as a Source of Data Augmentation

In this section, our goal is to demonstrate a practical use of the exponentially large non\-canonical tokenization space\. We show that exposing the model to multiple tokenization variants of the same training examples acts as an effective form of data augmentation, improving multilingual performance and robustness through cross\-lingual transfer\. Non\-canonical tokenization serves as a natural source of data augmentation for improving multilingual generalization\.

### 6\.1Tokenization Schemes

We investigate a family of tokenization schemes for constructing the training datasets used for LoRA fine\-tuning\. Each scheme specifies how tokenized representations are generated for every sentence in the original training corpus\. For the multi\-copy schemes \(N×N\\times, whereN∈\{10,20,30\}N\\in\\\{10,20,30\\\}\), each original sentence is represented byNNtokenized variants, resulting in a proportionally larger training dataset\.

The tokenization schemes are defined as follows:

- •Canonical \(1×\\times\):Each sentence is represented once using the tokenizer’s canonical tokenization\.
- •Non\-Canonical \(1×\\times\):Each sentence is represented once using a randomly sampled valid non\-canonical tokenization\.
- •Character \(1×\\times\):Each sentence is represented once using the character\-level tokenization, corresponding to the longest valid tokenization\.
- •Canonical \(N×N\\times\):Each sentence is representedNNtimes, with every copy using the canonical tokenization\.
- •Mixed \(N×N\\times\):Each sentence is represented using repeated groups of ten tokenized variants\. Each group consists of one canonical tokenization, one character\-level tokenization, and eight independently sampled non\-canonical tokenizations\. For larger values ofNN, this ten\-copy pattern is repeated \(e\.g\., twice for20×20\\times, three times for30×30\\times, and so on\)\.
- •LR\-Bucketed \(N×N\\times\):Each sentence is represented using repeated groups of ten tokenized variants\. Each group consists of one canonical tokenization, one character\-level tokenization, and eight non\-canonical tokenizations sampled from eight disjoint token\-length\-ratio buckets\. The token\-length ratio is defined as the ratio of the number of tokens in the sampled tokenization to that of the canonical tokenization\. The eight buckets correspond to the intervals\[1\.00,1\.25\)\[1\.00,1\.25\),\[1\.25,1\.50\)\[1\.25,1\.50\),\[1\.50,1\.75\)\[1\.50,1\.75\),\[1\.75,2\.00\)\[1\.75,2\.00\),\[2\.00,2\.25\)\[2\.00,2\.25\),\[2\.25,2\.50\)\[2\.25,2\.50\),\[2\.50,2\.75\)\[2\.50,2\.75\), and\[2\.75,3\.00\]\[2\.75,3\.00\]\. For larger values ofNN, this ten\-copy pattern is repeated accordingly\.

### 6\.2Dataset

Among all the datasets considered in our study,Multilingual\-ARCis the only dataset that provides an official training split\. Consequently, all LoRA fine\-tuning experiments are conducted exclusively on Multilingual\-ARC\. The original training split consists of 1,116 training instances and 298 validation instances\. We apply the tokenization schemes described in Section[6\.1](https://arxiv.org/html/2607.26831#S6.SS1)to construct the corresponding training datasets for fine\-tuning\. For model selection, we construct a validation set in which each original validation instance is represented by its canonical, sampled non\-canonical, and character tokenizations, yielding a total of 894 validation instances\. Validation is performed on this unified validation set for all training schemes\.Details about hyperparamters are present in Table[7](https://arxiv.org/html/2607.26831#A4.T7)\.

Training SchemeCanonRandCharRandΔ\\Delta%CharΔ\\Delta%Baseline \(No Fine\-Tuning\)78\.371\.8668\.54215\.963Canonical \(1×\)79\.272\.366\.38\.9716\.502Non\-Canonical \(1×\)78\.872\.166\.38\.71316\.129Character \(1×\)78\.87266\.28\.93216\.271Canonical \(10 ×\)81\.173\.667\.79\.43616\.524Mixed \(10 ×\)81\.173\.767\.99\.216\.28LR\-Bucketed \(10 ×\)81\.173\.8689\.13116\.151Canonical \(20 ×\)81\.273\.767\.99\.3416\.399Mixed \(20 ×\)81\.574\.668\.78\.6615\.783LR\-Bucketed \(20 ×\)81\.674\.668\.98\.7815\.701Canonical \(30 ×\)81\.273\.767\.99\.40516\.49Mixed \(30 ×\)81\.27468\.28\.98716\.141LR\-Bucketed \(30 ×\)81\.874\.869\.28\.63715\.548

Table 3:Average accuracy \(%\) across all languages on the Multilingual\-ARC test set under canonical, randomly sampled non\-canonical, and character\-level tokenizations\.RandΔ\\DeltaandCharΔ\\Deltadenote the average relative percentage decrease in accuracy with respect to canonical evaluation for the corresponding training scheme\. Impact across languages can be observed in Figure[9](https://arxiv.org/html/2607.26831#A4.F9)in the Appendix\.

## 7Analysis

As observed in Table[3](https://arxiv.org/html/2607.26831#S6.T3), the results demonstrate that the proposed multi\-tokenization training schemes consistently improve cross\-lingual generalization and robustness to tokenization variation\. Although LoRA fine\-tuning is performed exclusively on the English training split of Multilingual\-ARC, performance improves consistently across all languages across all evaluation settings\. This indicates that the benefits of multi\-tokenization training transfer effectively across languages and are not confined to the language observed during fine\-tuning\.

We can observe that the choice of tokenization has only a limited effect when each training instance is represented by a single tokenization\. The Canonical \(1×\\times\), Non\-Canonical \(1×\\times\), and Character \(1×\\times\) schemes achieve nearly identical average performance across all three evaluation settings, suggesting that simply replacing one tokenization with another provides little additional robustness\.

In contrast, increasing the number of tokenized representations per training instance yields substantial improvements\. Relative to Canonical \(1×\\times\), Canonical \(10×\\times\) improves the average accuracy from 0\.792 to 0\.811 under Canonical evaluation, from 0\.723 to 0\.736 under Non\-Canonical evaluation, and from 0\.663 to 0\.677 under Character evaluation\. These gains demonstrate that exposing the model to multiple valid tokenizations of the same training instance is considerably more effective\.

Increasing the number of tokenized variants from 10×\\timesto 20×\\timesproduces additional but smaller improvements, indicating diminishing returns with increasing dataset expansion\. For example, the Mixed scheme improves from\(0\.811,0\.737,0\.679\)\(0\.811,0\.737,0\.679\)to\(0\.815,0\.746,0\.687\)\(0\.815,0\.746,0\.687\)under Canonical, Non\-Canonical, and Character evaluation, respectively, while LR\-Bucketed improves from\(0\.811,0\.738,0\.680\)\(0\.811,0\.738,0\.680\)to\(0\.816,0\.746,0\.689\)\(0\.816,0\.746,0\.689\)\. These improvements are modest compared to those obtained when moving from 1×\\timesto 10×\\times, suggesting that further dataset expansion provides only incremental improvements\.

It is important to note the comparison between the Mixed and LR\-Bucketed schemes highlights the importance of the distribution of non\-canonical tokenizations presented during training\. While both schemes consistently outperform Canonical training at the same dataset multiplier, LR\-Bucketed achieves the highest average performance under canonical \(0\.816\) and character \(0\.689\) evaluation, while matching mixed under non\-Canonical evaluation \(0\.746\)\. These results suggest that systematically covering the space of valid non\-canonical tokenizations through token\-length\-ratio buckets is more effective than repeatedly sampling arbitrary non\-canonical tokenizations\. Moreover, as observed in Table[3](https://arxiv.org/html/2607.26831#S6.T3), although Canonical scheme improves performance, it comes at the expense of robustness\. In contrast, LR\-Bucketed schemes achieve even greater performance gains while preserving robustness\. Taken together, the results indicate that robustness to tokenization variation is driven not only by increasing the number of tokenized training instances but also by increasing the diversity and coverage of the non\-canonical tokenization space\.

## 8Conclusion

In conclusion, although a given string can admit exponentially many valid tokenizations, language models process only a single deterministic canonical tokenization, leaving the broader tokenization space insufficiently explored\. This work systematically examines model behavior under non\-canonical tokenizations across 27 languages and six downstream tasks, revealing that tokenization invariance observed in English does not generalize universally\. Instead, language models exhibit substantial performance degradation under alternative tokenizations\.Our results demonstrate that LoRA fine\-tuning with multi\-tokenization training is an effective strategy for mitigating tokenization sensitivity\. Although fine\-tuning is performed solely on English data, it consistently improves robustness across languages, with systematically sampling diverse non\-canonical tokenizations yielding the best overall performance\.

## Limitations

Our study has two main limitations\. First, we evaluate non\-canonical tokenization as a training\-time data augmentation strategy only on Multilingual\-ARC, as it is the only benchmark among those considered in this work that provides a training split\. Evaluating this approach across a broader range of multilingual tasks would help establish the generality of the observed gains\.

Second, although our analysis reveals systematic differences in robustness across models and languages, it does not isolate the individual factors responsible for these differences\. Robustness may be influenced by multiple interacting factors, including vocabulary allocation, language resource level, pretraining representation etc\. Disentangling the independent contribution of these factors would require carefully controlled experiments that manipulate each variable in isolation\. We leave such analyses to future work\.

## Ethical Considerations

Non\-canonical tokenizations can affect model behavior and may increase the likelihood of incorrect, hallucinated, or harmful outputs\. Our work investigates the robustness of language models to alternative valid tokenizations in order to better understand their sensitivity to input segmentation across languages\. We do not intend these perturbation methods for harmful use; rather, we hope they help identify model vulnerabilities and motivate the development of more robust and reliable multilingual language models\.

## References

- O\. Ahia, S\. Kumar, H\. Gonen, J\. Kasai, D\. R\. Mortensen, N\. A\. Smith, and Y\. Tsvetkov \(2023\)Do all languages cost the same? tokenization in the era of commercial language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 9904–9923\.Cited by:[§1](https://arxiv.org/html/2607.26831#S1.p2.1)\.
- L\. Bandarkar, D\. Liang, B\. Muller, M\. Artetxe, S\. N\. Shukla, D\. Husa, N\. Goyal, A\. Krishnan, L\. Zettlemoyer, and M\. Khabsa \(2024\)The belebele benchmark: a parallel reading comprehension dataset in 122 language variants\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 749–775\.External Links:[Link](https://aclanthology.org/2024.acl-long.44/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.44)Cited by:[§4](https://arxiv.org/html/2607.26831#S4.SS0.SSS0.Px4.p1.1)\.
- K\. Cao and L\. Rimell \(2021\)You should evaluate your language model on marginal likelihood over tokenisations\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,pp\. 2104–2114\.Cited by:[§2](https://arxiv.org/html/2607.26831#S2.SS0.SSS0.Px2.p1.1)\.
- N\. Chirkova, G\. Kruszewski, J\. Rozen, and M\. Dymetman \(2023\)Should you marginalize over possible tokenizations?\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),pp\. 1–12\.Cited by:[§2](https://arxiv.org/html/2607.26831#S2.SS0.SSS0.Px2.p1.1)\.
- M\. Cognetta, V\. Zouhar, and N\. Okazaki \(2024\)Distributional properties of subword regularization\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 10753–10763\.Cited by:[§2](https://arxiv.org/html/2607.26831#S2.SS0.SSS0.Px3.p1.1)\.
- M\. R\. Costa\-Jussà, J\. Cross, O\. Çelebi, M\. Elbayad, K\. Heafield, K\. Heffernan, E\. Kalbassi, J\. Lam, D\. Licht, J\. Maillard,et al\.\(2022\)No language left behind: scaling human\-centered machine translation\.arXiv preprint arXiv:2207\.04672\.Cited by:[§4](https://arxiv.org/html/2607.26831#S4.SS0.SSS0.Px2.p2.1)\.
- L\. Gao, J\. Tow, B\. Abbasi, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golding, J\. Hsu, A\. Le Noac’h, H\. Li, K\. McDonell, N\. Muennighoff, C\. Ociepa, J\. Phang, L\. Reynolds, H\. Schoelkopf, A\. Skowron, L\. Sutawika, E\. Tang, A\. Thite, B\. Wang, K\. Wang, and A\. Zou \(2023\)A framework for few\-shot language model evaluation\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.10256836),[Link](https://zenodo.org/records/10256836)Cited by:[§4](https://arxiv.org/html/2607.26831#S4.p1.1)\.
- R\. Geh, Z\. Shao, and G\. Van den Broeck \(2025\)Adversarial tokenization\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 20738–20765\.Cited by:[§1](https://arxiv.org/html/2607.26831#S1.p3.1),[§2](https://arxiv.org/html/2607.26831#S2.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2607.26831#S3.SS0.SSS0.Px3.SPx2.p1.6)\.
- R\. Geh, H\. Zhang, K\. Ahmed, B\. Wang, and G\. Van den Broeck \(2024\)Where is the signal in tokenization space?\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 3966–3979\.Cited by:[§2](https://arxiv.org/html/2607.26831#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§4](https://arxiv.org/html/2607.26831#S4.SS0.SSS0.Px1.p1.1)\.
- H\. Kim and H\. Kim \(2026\)Optimizing korean\-centric llms via token pruning\.arXiv preprint arXiv:2604\.16235\.Cited by:[§1](https://arxiv.org/html/2607.26831#S1.p3.1)\.
- T\. Kudo and J\. Richardson \(2018\)SentencePiece: a simple and language independent subword tokenizer and detokenizer for neural text processing\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,E\. Blanco and W\. Lu \(Eds\.\),Brussels, Belgium,pp\. 66–71\.External Links:[Link](https://aclanthology.org/D18-2012/),[Document](https://dx.doi.org/10.18653/v1/D18-2012)Cited by:[§1](https://arxiv.org/html/2607.26831#S1.p1.1)\.
- T\. Kudo \(2018\)Subword regularization: improving neural network translation models with multiple subword candidates\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),I\. Gurevych and Y\. Miyao \(Eds\.\),Melbourne, Australia,pp\. 66–75\.External Links:[Link](https://aclanthology.org/P18-1007/),[Document](https://dx.doi.org/10.18653/v1/P18-1007)Cited by:[§1](https://arxiv.org/html/2607.26831#S1.p1.1)\.
- V\. Lai, C\. Nguyen, N\. Ngo, T\. Nguyen, F\. Dernoncourt, R\. Rossi, and T\. Nguyen \(2023\)Okapi: instruction\-tuned large language models in multiple languages with reinforcement learning from human feedback\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,Y\. Feng and E\. Lefever \(Eds\.\),Singapore,pp\. 318–327\.External Links:[Link](https://aclanthology.org/2023.emnlp-demo.28/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-demo.28)Cited by:[§4](https://arxiv.org/html/2607.26831#S4.SS0.SSS0.Px4.p1.1)\.
- C\. Li, Y\. Deng, W\. Yang, J\. Zhang, and C\. Zong \(2026\)TokAlign\+\+: advancing vocabulary adaptation via better token alignment\.arXiv preprint arXiv:2605\.13429\.Cited by:[§1](https://arxiv.org/html/2607.26831#S1.p3.1)\.
- C\. Li, J\. Zhang, and C\. Zong \(2025\)TokAlign: efficient vocabulary adaptation via token alignment\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 4109–4126\.Cited by:[§1](https://arxiv.org/html/2607.26831#S1.p3.1)\.
- T\. Limisiewicz, J\. Balhar, and D\. Mareček \(2023\)Tokenization impacts multilingual language modeling: assessing vocabulary allocation and overlap across languages\.InFindings of the Association for Computational Linguistics: ACL 2023,A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 5661–5681\.External Links:[Link](https://aclanthology.org/2023.findings-acl.350/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-acl.350)Cited by:[§1](https://arxiv.org/html/2607.26831#S1.p2.1)\.
- A\. Petrov, E\. La Malfa, P\. Torr, and A\. Bibi \(2023\)Language model tokenizers introduce unfairness between languages\.Advances in neural information processing systems36,pp\. 36963–36990\.Cited by:[§1](https://arxiv.org/html/2607.26831#S1.p2.1)\.
- T\. Purason, P\. Chizhov, I\. P\. Yamshchikov, and M\. Fishel \(2026\)Teaching old tokenizers new words: efficient tokenizer adaptation for pretrained models\.InFindings of the Association for Computational Linguistics: EACL 2026,V\. Demberg, K\. Inui, and L\. Marquez \(Eds\.\),Rabat, Morocco,pp\. 6492–6516\.External Links:[Link](https://aclanthology.org/2026.findings-eacl.341/),[Document](https://dx.doi.org/10.18653/v1/2026.findings-eacl.341),ISBN 979\-8\-89176\-386\-9Cited by:[§1](https://arxiv.org/html/2607.26831#S1.p3.1)\.
- Y\. Reif, G\. Kaplan, and R\. Schwartz \(2026\)Vocab diet: reshaping the vocabulary of LLMs via vector arithmetic\.InFindings of the Association for Computational Linguistics: ACL 2026,M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 32334–32352\.External Links:[Link](https://aclanthology.org/2026.findings-acl.1618/),[Document](https://dx.doi.org/10.18653/v1/2026.findings-acl.1618),ISBN 979\-8\-89176\-395\-1Cited by:[§1](https://arxiv.org/html/2607.26831#S1.p3.1)\.
- F\. Remy, P\. Delobelle, H\. Avetisyan, A\. Khabibullina, M\. de Lhoneux, and T\. Demeester \(2024\)Trans\-tokenization and cross\-lingual vocabulary transfers: language adaptation of llms for low\-resource nlp\.arXiv preprint arXiv:2408\.04303\.Cited by:[§1](https://arxiv.org/html/2607.26831#S1.p3.1)\.
- R\. Sennrich, B\. Haddow, and A\. Birch \(2016\)Neural machine translation of rare words with subword units\.InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),K\. Erk and N\. A\. Smith \(Eds\.\),Berlin, Germany,pp\. 1715–1725\.External Links:[Link](https://aclanthology.org/P16-1162/),[Document](https://dx.doi.org/10.18653/v1/P16-1162)Cited by:[§1](https://arxiv.org/html/2607.26831#S1.p1.1)\.
- S\. Singh, A\. Romanou, C\. Fourrier, D\. I\. Adelani, J\. G\. Ngui, D\. Vila\-Suero, P\. Limkonchotiwat, K\. Marchisio, W\. Q\. Leong, Y\. Susanto, R\. Ng, S\. Longpre, S\. Ruder, W\. Ko, A\. Bosselut, A\. Oh, A\. Martins, L\. Choshen, D\. Ippolito, E\. Ferrante, M\. Fadaee, B\. Ermis, and S\. Hooker \(2025\)Global MMLU: understanding and addressing cultural and linguistic biases in multilingual evaluation\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 18761–18799\.External Links:[Link](https://aclanthology.org/2025.acl-long.919/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.919),ISBN 979\-8\-89176\-251\-0Cited by:[§4](https://arxiv.org/html/2607.26831#S4.SS0.SSS0.Px4.p1.1)\.
- S\. Steger, R\. Li, S\. Ennadir, A\. Sims, A\. Solin, F\. Pernkopf, and M\. Trapp \(2026\)Stochasticity in tokenisation improves robustness\.arXiv preprint arXiv:2604\.16037\.Cited by:[§2](https://arxiv.org/html/2607.26831#S2.SS0.SSS0.Px3.p1.1)\.
- G\. Team, A\. Kamath, J\. Ferret, S\. Pathak, N\. Vieillard, R\. Merhej, S\. Perrin, T\. Matejovicova, A\. Ramé, M\. Rivière, L\. Rouillard, T\. Mesnard, G\. Cideron, J\. Grill, S\. Ramos, E\. Yvinec, M\. Casbon, E\. Pot, I\. Penchev, G\. Liu, F\. Visin, K\. Kenealy, L\. Beyer, X\. Zhai, A\. Tsitsulin, R\. Busa\-Fekete, A\. Feng, N\. Sachdeva, B\. Coleman, Y\. Gao, B\. Mustafa, I\. Barr, E\. Parisotto, D\. Tian, M\. Eyal, C\. Cherry, J\. Peter, D\. Sinopalnikov, S\. Bhupatiraju, R\. Agarwal, M\. Kazemi, D\. Malkin, R\. Kumar, D\. Vilar, I\. Brusilovsky, J\. Luo, A\. Steiner, A\. Friesen, A\. Sharma, A\. Sharma, A\. M\. Gilady, A\. Goedeckemeyer, A\. Saade, A\. Feng, A\. Kolesnikov, A\. Bendebury, A\. Abdagic, A\. Vadi, A\. György, A\. S\. Pinto, A\. Das, A\. Bapna, A\. Miech, A\. Yang, A\. Paterson, A\. Shenoy, A\. Chakrabarti, B\. Piot, B\. Wu, B\. Shahriari, B\. Petrini, C\. Chen, C\. L\. Lan, C\. A\. Choquette\-Choo, C\. Carey, C\. Brick, D\. Deutsch, D\. Eisenbud, D\. Cattle, D\. Cheng, D\. Paparas, D\. S\. Sreepathihalli, D\. Reid, D\. Tran, D\. Zelle, E\. Noland, E\. Huizenga, E\. Kharitonov, F\. Liu, G\. Amirkhanyan, G\. Cameron, H\. Hashemi, H\. Klimczak\-Plucińska, H\. Singh, H\. Mehta, H\. T\. Lehri, H\. Hazimeh, I\. Ballantyne, I\. Szpektor, I\. Nardini, J\. Pouget\-Abadie, J\. Chan, J\. Stanton, J\. Wieting, J\. Lai, J\. Orbay, J\. Fernandez, J\. Newlan, J\. Ji, J\. Singh, K\. Black, K\. Yu, K\. Hui, K\. Vodrahalli, K\. Greff, L\. Qiu, M\. Valentine, M\. Coelho, M\. Ritter, M\. Hoffman, M\. Watson, M\. Chaturvedi, M\. Moynihan, M\. Ma, N\. Babar, N\. Noy, N\. Byrd, N\. Roy, N\. Momchev, N\. Chauhan, N\. Sachdeva, O\. Bunyan, P\. Botarda, P\. Caron, P\. K\. Rubenstein, P\. Culliton, P\. Schmid, P\. G\. Sessa, P\. Xu, P\. Stanczyk, P\. Tafti, R\. Shivanna, R\. Wu, R\. Pan, R\. Rokni, R\. Willoughby, R\. Vallu, R\. Mullins, S\. Jerome, S\. Smoot, S\. Girgin, S\. Iqbal, S\. Reddy, S\. Sheth, S\. Põder, S\. Bhatnagar, S\. R\. Panyam, S\. Eiger, S\. Zhang, T\. Liu, T\. Yacovone, T\. Liechty, U\. Kalra, U\. Evci, V\. Misra, V\. Roseberry, V\. Feinberg, V\. Kolesnikov, W\. Han, W\. Kwon, X\. Chen, Y\. Chow, Y\. Zhu, Z\. Wei, Z\. Egyed, V\. Cotruta, M\. Giang, P\. Kirk, A\. Rao, K\. Black, N\. Babar, J\. Lo, E\. Moreira, L\. G\. Martins, O\. Sanseviero, L\. Gonzalez, Z\. Gleicher, T\. Warkentin, V\. Mirrokni, E\. Senter, E\. Collins, J\. Barral, Z\. Ghahramani, R\. Hadsell, Y\. Matias, D\. Sculley, S\. Petrov, N\. Fiedel, N\. Shazeer, O\. Vinyals, J\. Dean, D\. Hassabis, K\. Kavukcuoglu, C\. Farabet, E\. Buchatskaya, J\. Alayrac, R\. Anil, Dmitry, Lepikhin, S\. Borgeaud, O\. Bachem, A\. Joulin, A\. Andreev, C\. Hardin, R\. Dadashi, and L\. Hussenot \(2025\)Gemma 3 technical report\.External Links:2503\.19786,[Link](https://arxiv.org/abs/2503.19786)Cited by:[§4](https://arxiv.org/html/2607.26831#S4.SS0.SSS0.Px1.p1.1)\.
- T\. Vieira, B\. LeBrun, M\. Giulianelli, J\. L\. Gastaldi, B\. DuSell, J\. Terilla, T\. J\. O’Donnell, and R\. Cotterell \(2024\)From language models over tokens to language models over characters\.arXiv preprint arXiv:2412\.03719\.Cited by:[§2](https://arxiv.org/html/2607.26831#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Wu, M\. Schuster, Z\. Chen, Q\. V\. Le, M\. Norouzi, W\. Macherey, M\. Krikun, Y\. Cao, Q\. Gao, K\. Macherey,et al\.\(2016\)Google’s neural machine translation system: bridging the gap between human and machine translation\.arXiv preprint arXiv:1609\.08144\.Cited by:[§1](https://arxiv.org/html/2607.26831#S1.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§4](https://arxiv.org/html/2607.26831#S4.SS0.SSS0.Px1.p1.1)\.
- B\. S\. Zheng, A\. Liu, O\. Ahia, J\. Hayase, Y\. Choi, and N\. A\. Smith \(2025\)Broken tokens? your language model can secretly handle non\-canonical tokenizations\.arXiv preprint arXiv:2506\.19004\.Cited by:[§1](https://arxiv.org/html/2607.26831#S1.p1.1),[§2](https://arxiv.org/html/2607.26831#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix AModel Size And Budget

All experiments were conducted on NVIDIA RTX 6000 Ada Generation GPUs with 48 GB VRAM\.

## Appendix BUse of Large Language Models

We used large language models \(LLMs\) to improve the grammar, clarity, and readability of the paper\. All generated content was carefully reviewed and verified by the authors to ensure accuracy and consistency with the scientific contributions of the work\.

## Appendix CResults on Excluded Languages

As shown in Table[5](https://arxiv.org/html/2607.26831#A3.T5), the excluded languages have a substantially smaller tokenization space under the Qwen tokenizer because their canonical tokenization is already close to maximum fragmentation\. Consequently, non\-canonical tokenizations introduce only limited additional perturbation, which is reflected in the Multilingual\-ARC results \(Table[4](https://arxiv.org/html/2607.26831#A3.T4)\), where Qwen3\-8B exhibits only minor differences between canonical, non\-canonical, and character\-level tokenizations\. In contrast, Gemma3\-12B admits a much larger tokenization space for these languages, leading to greater variation in performance across tokenization schemes\.

For Armenian, many sentences in the dataset have only a single valid tokenization under the Qwen3 tokenizer\. Consequently, results for Armenian are omitted from these tables\.

Table 4:Results on Multilingual ARC for languages excluded from the main analysis\.Table 5:Average number of valid tokenizations \(log10\\log\_\{10\}scale\) for the excluded languages under the Qwen3\-8B and Gemma3\-12B tokenizers measured on the FLORES\-200 benchmark\. The substantially smaller tokenization space for Qwen reflects its near character\-level canonical tokenization for these languages\.Table 6:Multilingual\-ARC performance under canonical tokenization and non\-canonical tokenizations with sequence lengths closely matching the canonical tokenization \(length\-ratio buckets of 0\.95–1\.00 and 1\.00–1\.05\)\. Performance differences largely persist despite nearly identical sequence lengths\.
## Appendix DHyperparameter Details

The hyperparameter settings for the LoRA fine\-tuning experiments are provided in Table[7](https://arxiv.org/html/2607.26831#A4.T7)\.

Table 7:LoRA fine\-tuning hyperparameters used in all experiments\.TypeTaskNo\. of samplesSystem promptShort answerMLQA1000You are a helpful QA assistant\. Extract the answer span from the context\.MGSM250You are a helpful assistant\.Multiple choiceMulti\-ARC1000You are a helpful assistant\. For each of the following multiple\-choice questions, respond with the correct answer letter \(A, B, C, or D\) only\. Do not include any explanation, reasoning, or additional text\.Multi\-HellaSwag1000Global\-MMLU\-Lite400Belebele900Table 8:Tasks![Refer to caption](https://arxiv.org/html/2607.26831v1/x15.png)\(a\)Normal
![Refer to caption](https://arxiv.org/html/2607.26831v1/x16.png)\(b\)Random
![Refer to caption](https://arxiv.org/html/2607.26831v1/x17.png)\(c\)Character

Figure 6:Retrieval experiments on FLORES\-200 for Qwen![Refer to caption](https://arxiv.org/html/2607.26831v1/x18.png)\(a\)Normal
![Refer to caption](https://arxiv.org/html/2607.26831v1/x19.png)\(b\)Random
![Refer to caption](https://arxiv.org/html/2607.26831v1/x20.png)\(c\)Character

Figure 7:Retrieval experiments on FLORES\-200 for GemmaTable 9:LanguagesTable 10:Logistic regression coefficients on Multilingual\-ARC predicting non\-canonical performance from canonical performance and canonical fragmentation rate \(FR\)\. Positive coefficients for canonical performance indicate that examples solved under canonical tokenization are substantially more likely to remain correct under non\-canonical tokenizations\. Negative coefficients for canonical fragmentation rate indicate that more fragmented canonical tokenizations are associated with reduced robustness to non\-canonical tokenizations\.![Refer to caption](https://arxiv.org/html/2607.26831v1/x21.png)\(a\)Llama
![Refer to caption](https://arxiv.org/html/2607.26831v1/x22.png)\(b\)Qwen
![Refer to caption](https://arxiv.org/html/2607.26831v1/x23.png)\(c\)Gemma

Figure 8:Impact of granularity of non\-canonical tokenization on MGSM performance\.![Refer to caption](https://arxiv.org/html/2607.26831v1/x24.png)\(a\)Canonical
![Refer to caption](https://arxiv.org/html/2607.26831v1/x25.png)\(b\)Random Non\-canonical
![Refer to caption](https://arxiv.org/html/2607.26831v1/x26.png)\(c\)Character

Figure 9:Impact of LoRA fine\-tuning across languages\.