基于合并的词表扩展受阻于预分词器：一种保留英语能力的尼泊尔语修复方案，经持续预训练验证

# Merge-Based Vocabulary Extension Stalls at the Pre-Tokenizer: An English-Preserving Repair for Nepali, Tested in Continued Pretraining
Source: [https://arxiv.org/html/2610.33016](https://arxiv.org/html/2610.33016)
###### Abstract

Llama\-3, Qwen3 and many other open LLMs pre\-tokenize text with a regular expression whose word class admits only Unicode letters\. Devanagari vowel signs are combining marks, so this regex splits Nepali words at every vowel sign, and models that use it spend2\.62\.6to5\.85\.8times as many tokens on Nepali as on the same English text\. Merge\-based vocabulary extension cannot join the split pieces: for Llama\-3 and Qwen3 no number of new merges brings this premium below about2\.22\.2, and ten released extensions already end within 4% of this pre\-token floor\. We give a small edit to the deployed regex that provably keeps every token id of text without combining marks or Devanagari characters\. With 32,000 new merges it brings Nepali to1\.011\.01times English, about 54% fewer Nepali tokens than standard extension\. In small\-budget continued pretraining of Llama\-3\.2\-1B and Qwen3\-0\.6B, the original tokenizer gave the lowest held\-out Nepali bits per byte under every recipe we tested\. Matching the repair’s training steps to standard extension, with about 35% of the data repeated, reversed their order on Qwen but not on Llama\. Whether the token savings give a favourable latency and quality trade\-off in generation remains untested\. We release the tokenizers and code\.

## IIntroduction

On parallel FLORES\-200 text, Qwen3’s tokenizer needs4\.424\.42times as many tokens for a Nepali sentence as for its English translation, and thecl100ktokenizer used by Phi\-4, OLMo\-2 and Granite\-4\.1 needs4\.764\.76times as many \(§[IV](https://arxiv.org/html/2610.33016#S4)\)\. We call this ratio the*premium*\. Nepali speakers pay it on every request, in price, latency and usable context\.

Much of the premium has a mechanical cause\. GPT\-2\-style pre\-tokenizers split text with a regular expression whose word class\\p\{L\}matches one or more Unicode letters\[[41](https://arxiv.org/html/2610.33016#bib.bib46)\]; we call it the*letters\-only regex*\. In Unicode, Devanagari vowel signs, the virama, the anusvara and the candrabindu are combining marks \(\\p\{M\}\), so the regex starts a new pre\-token at each of them and byte\-pair encoding \(BPE\) can never merge across that boundary\.[Regmi et al\. \[50\]](https://arxiv.org/html/2610.33016#bib.bib13)formalised this cause and the resulting bound on token counts\. They showed that adding marks to the word class cuts Nepali from 4\.78 to 1\.58 tokens per word, and that 63% of the 1,345 text\-generation repositories they classified still use the letters\-only class\.

Regmi et al\. studied tokenizers trained from scratch\. Most practitioners instead adapt a model that already exists, usually by vocabulary extension: they add target\-language tokens, initialise their embeddings and continue pretraining\[[58](https://arxiv.org/html/2610.33016#bib.bib27),[62](https://arxiv.org/html/2610.33016#bib.bib35),[45](https://arxiv.org/html/2610.33016#bib.bib28),[56](https://arxiv.org/html/2610.33016#bib.bib39)\]\. We study that deployed\-model setting\. We show that the bound becomes a floor that released merge\-based extensions already hit; we give a repair of a deployed model’s regex that provably keeps English token ids; and we test it in controlled continued pretraining against the standard continued\-BPE recipe of[Purason et al\. \[45\]](https://arxiv.org/html/2610.33016#bib.bib28), with the tokenizers released\.

#### Findings\.

For Llama\-3 and Qwen3 the*pre\-token floor*, the lowest premium any merge\-based extension can reach, is about2\.22\.2\(§[III](https://arxiv.org/html/2610.33016#S3)\)\. Ten released extensions for six scripts end within 4% of their floors, and our own extension under the letters\-only regex stalls at it\. Registering our repair’s learned strings as added tokens\[[10](https://arxiv.org/html/2610.33016#bib.bib58), cf\.\]gets below the floor, but a fifth of the matches then start mid\-word\. Our repair adds marks to the word class with two string substitutions and, with new merges, brings Nepali to parity with English while leaving every token id of mark\-free, non\-Devanagari text unchanged \(§[V](https://arxiv.org/html/2610.33016#S5)\)\. The model\-quality result is less favourable\. Continued pretraining with no new tokens gave the best held\-out Nepali bits per byte on both models, so our main hypothesis failed\. At equal bytes the repair is worse than standard extension \(0\.0250\.025bits per byte on Llama and0\.0120\.012on Qwen, two\-seed means\), but it takes 26% fewer steps; at matched steps it is0\.0100\.010better on Qwen and0\.0120\.012worse on Llama, and a higher learning rate on Qwen improves all three arms’ Nepali scores without changing their order, at a cost on English \(§[VII\-B](https://arxiv.org/html/2610.33016#S7.SS2)\)\. For the same Nepali text the repair needs 54% fewer tokens than standard extension; whether this speeds up generation of equally good text is untested\.

#### Contributions\.

We make four contributions:

1. 1\.Evidence that the pre\-token floor binds released merge\-based extensions, and a test of the added\-token route around it \(§[III](https://arxiv.org/html/2610.33016#S3)\); an audit of 24 distinct deployed tokenizers with per\-tokenizer floors \(§[IV](https://arxiv.org/html/2610.33016#S4)\)\.
2. 2\.A repair of a deployed regex that provably keeps English token ids \(not model behaviour\), a Devanagari\-only variant with a stronger guarantee \(not trained\), and a practical pitfall in loading repaired tokenizers \(§[V](https://arxiv.org/html/2610.33016#S5)\)\.
3. 3\.A controlled comparison ofR0\(no extension\),R1\(standard extension\) andR2\(repair \+ extension\) in continued pretraining of two deployed models, reported against hypotheses written down in advance, including the one that failed \(§[VII\-B](https://arxiv.org/html/2610.33016#S7.SS2)\)\.
4. 4\.

## IIRelated work

#### Tokenization premiums\.

Tokenizers trained mostly on English encode other languages in more tokens, so those languages pay more for the same content\[[52](https://arxiv.org/html/2610.33016#bib.bib1),[44](https://arxiv.org/html/2610.33016#bib.bib9),[1](https://arxiv.org/html/2610.33016#bib.bib4),[31](https://arxiv.org/html/2610.33016#bib.bib7)\]\. For Nepali,[Kadariya \[25\]](https://arxiv.org/html/2610.33016#bib.bib22)reports4\.794\.79forcl100k, and[Srivastava \[57\]](https://arxiv.org/html/2610.33016#bib.bib21)finds thato200kcuts the mean Indic premium from8\.08\.0to2\.12\.1without attributing the change to the regex\. We measure sentence\-level token ratios on parallel FLORES\-200 text\[[21](https://arxiv.org/html/2610.33016#bib.bib2),[40](https://arxiv.org/html/2610.33016#bib.bib12)\], following[Petrov et al\. \[44\]](https://arxiv.org/html/2610.33016#bib.bib9)\. Fewer tokens do not guarantee a better model\[[53](https://arxiv.org/html/2610.33016#bib.bib5),[65](https://arxiv.org/html/2610.33016#bib.bib3),[30](https://arxiv.org/html/2610.33016#bib.bib64)\], so we also test the repair with continued pretraining\.

#### Pre\-tokenization of combining marks\.

A 2024 tiktoken issue reported that GPT\-2\-style regexes split Indic words at vowel signs and proposed the class\[\\p\{L\}\\p\{M\}\]\+\[[2](https://arxiv.org/html/2610.33016#bib.bib17)\]\. ZeTT\[[35](https://arxiv.org/html/2610.33016#bib.bib31)\]adjusted its regex to avoid over\-segmenting Hindi and Tamil,[Velayuthan and Sarveswaran \[60\]](https://arxiv.org/html/2610.33016#bib.bib6)showed that the GPT\-2, GPT\-4 and Llama\-3 pre\-tokenizers break Tamil, Sinhala and Hindi, and[Rana et al\. \[48\]](https://arxiv.org/html/2610.33016#bib.bib18)cut Hindi fertility from 3\.47 to 1\.36 tokens per word by swapping the GPT\-2 regex for Llama\-4’s; see also[Thakur et al\. \[59\]](https://arxiv.org/html/2610.33016#bib.bib60)\.[Regmi et al\. \[50\]](https://arxiv.org/html/2610.33016#bib.bib13)formalised the cause, stated the bound across 26 languages, and showed with 268M\-parameter models trained from scratch that the wider class lowers Nepali bits per byte by 4\.4%\. Qwen3\.5’s released tokenizer uses a mark\-aware class\[[46](https://arxiv.org/html/2610.33016#bib.bib50)\], a change also noted by[Land \[28\]](https://arxiv.org/html/2610.33016#bib.bib16), and[Shrestha and Pradhan \[55\]](https://arxiv.org/html/2610.33016#bib.bib15)study morphology\-guided pre\-tokenization for Nepali in models trained from scratch\. The edit itself is therefore known\. What has not been studied is applying it to an already trained model while keeping its English tokenization\.

#### Adapting the vocabulary of a pretrained model\.

Vocabulary extension adds tokens, initialises their embeddings from existing ones\[[20](https://arxiv.org/html/2610.33016#bib.bib25),[34](https://arxiv.org/html/2610.33016#bib.bib23),[16](https://arxiv.org/html/2610.33016#bib.bib24),[29](https://arxiv.org/html/2610.33016#bib.bib26),[51](https://arxiv.org/html/2610.33016#bib.bib63)\]and continues pretraining\[[12](https://arxiv.org/html/2610.33016#bib.bib62),[58](https://arxiv.org/html/2610.33016#bib.bib27),[27](https://arxiv.org/html/2610.33016#bib.bib34),[11](https://arxiv.org/html/2610.33016#bib.bib37),[38](https://arxiv.org/html/2610.33016#bib.bib43)\];[Duwal et al\. \[18\]](https://arxiv.org/html/2610.33016#bib.bib44)continue pretraining Llama\-3\-8B on Nepali with 4\-bit QLoRA and without new tokens\. Continued BPE appends new merges to a byte\-level model’s merge list;[Purason et al\. \[45\]](https://arxiv.org/html/2610.33016#bib.bib28)introduce it and report tokenizer\-only compression for many languages, Nepali included, under the unchanged regex\.[Yamaguchi et al\. \[62\]](https://arxiv.org/html/2610.33016#bib.bib35)and[Yamaguchi et al\. \[61\]](https://arxiv.org/html/2610.33016#bib.bib36)extend Llama\-3 and Qwen models for several abugidas,[Smith et al\. \[56\]](https://arxiv.org/html/2610.33016#bib.bib39)expand LFM2 and[Mehta et al\. \[32\]](https://arxiv.org/html/2610.33016#bib.bib61)Qwen3\.[Didenko \[15\]](https://arxiv.org/html/2610.33016#bib.bib47)adapts models to Ukrainian at fixed vocabulary size, keeping English token counts within 0\.05%\. All of these keep the original pre\-tokenizer\. Tokenizer transfer\[[35](https://arxiv.org/html/2610.33016#bib.bib31),[36](https://arxiv.org/html/2610.33016#bib.bib32),[17](https://arxiv.org/html/2610.33016#bib.bib38)\], dynamic tokenization\[[19](https://arxiv.org/html/2610.33016#bib.bib33)\]and grapheme\- or akshara\-aware pre\-tokenizers\[[26](https://arxiv.org/html/2610.33016#bib.bib19),[14](https://arxiv.org/html/2610.33016#bib.bib20)\]replace more of the tokenizer\.[Dagan et al\. \[13\]](https://arxiv.org/html/2610.33016#bib.bib11)replace the tokenizer of a pretrained model, regex included, during code fine\-tuning and find it recovers after about 50B tokens in their setting;[Rana et al\. \[48\]](https://arxiv.org/html/2610.33016#bib.bib18)likewise give Llama\-3\.2\-1B a new tokenizer with Llama\-4’s mark\-aware regex and train its embedding and output layers on 40B tokens\. Both alter English tokenization; ours does not\.

The closest prior system is TituLLM\[[39](https://arxiv.org/html/2610.33016#bib.bib29)\], which extends Llama\-3\.2 for Bengali; its released v2\.0 tokenizer uses a\\w\-based regex that keeps vowel signs attached\. It does not identify the regex as a cause and has no regex\-only or vocabulary\-only arm, and its regex also changes digit and whitespace pre\-tokens in English\. We add a controlled attribution on deployed models and a repair that leaves every input without marks or Devanagari characters unchanged\.

## IIIPre\-tokens bound merge\-based extension

A byte\-level BPE tokenizer\[[54](https://arxiv.org/html/2610.33016#bib.bib8),[47](https://arxiv.org/html/2610.33016#bib.bib10)\]first splits a stringsswith a regex into pre\-tokensP⁡\(s\)P\(s\)and then encodes each pre\-token on its own by applying merges in rank order\. Every pre\-token yields at least one token, so\|T⁡\(s\)\|≥\|P⁡\(s\)\|\|T\(s\)\|\\geq\|P\(s\)\|;[Regmi et al\. \[50\]](https://arxiv.org/html/2610.33016#bib.bib13)formalise this as a training\-free lower bound on fertility\. Appending merges, whatever they are and however many, changes neither the normalizer nor the regex, so the bound holds for every tokenizer extended this way\.

We turn this into a bound on the premium\. For a tokenizerTTwith pre\-tokenizerPP, the*pre\-token floor*is\|P⁡\(ne\)\|/\|T⁡\(en\)\|\|P\(\\text\{ne\}\)\|/\|T\(\\text\{en\}\)\|, the number of Nepali pre\-tokens divided by the number of English tokens, summed over the 1,012 FLORES\-200 devtest sentences\. A merge\-based extension that keepsPPand leaves English unchanged cannot bring the premium below this value, though it may stop above it\. Our standard extensionR1leaves English unchanged: every new merge produces a token that contains a complete Devanagari code point, and since UTF\-8 is self\-synchronising, no new merge can fire on text without one \(the same argument appears in the proof of Proposition[1](https://arxiv.org/html/2610.33016#Thmproposition1), Appendix[C](https://arxiv.org/html/2610.33016#A3)\)\.

For today’s regexes the floor is far from slack\. In Llama\-3 and Qwen the word branch is\[^\\r\\n\\p\{L\}\\p\{N\}\]?\\p\{L\}\+, a run of letters optionally preceded by one other character\. A Devanagari vowel sign ends the current run and becomes the leading character of the next one, so*nepālko sarkār*\(“government of Nepal”\) becomes six pre\-tokens, four of which start with a vowel sign that belongs to the previous consonant \(Figure[4](https://arxiv.org/html/2610.33016#A10.F4), Appendix[J](https://arxiv.org/html/2610.33016#A10)\)\. The floor is2\.192\.19for Llama\-3 and2\.172\.17for Qwen3\. That is above the premium ofo200k\(1\.611\.61\), and 2\.7 to 2\.8 times the floor under the repaired regex of §[V](https://arxiv.org/html/2610.33016#S5)\(0\.790\.79and0\.800\.80\)\. Llama\-3’s premium is already2\.582\.58, so standard extension can lower it by at most 15%\.

#### Released extensions end at the floor\.

Figure[1](https://arxiv.org/html/2610.33016#S3.F1)measures released merge\-based extensions of letters\-only models; Appendix[I](https://arxiv.org/html/2610.33016#A9)lists every number and how we found them\. Ten extensions by Yamaguchi et al\. of Llama\-3, Llama\-3\.1, Qwen2\.5 and Qwen3\[[62](https://arxiv.org/html/2610.33016#bib.bib35),[61](https://arxiv.org/html/2610.33016#bib.bib36)\]cover six scripts written with combining marks\. All ten end within 4% of their floor, having closed at least 98% of the distance from the base tokenizer\.222These extensions re\-order the base merges and change English token counts slightly \(\+6\.5% for the Llama\-3 Telugu release\), so for them we compare tokens with pre\-tokens of the target text directly\.The 61k\-token in\-place expansion of LFM2\-8B\-A1B into LFM2\.5\-8B\-A1B\[[56](https://arxiv.org/html/2610.33016#bib.bib39)\]ends further from its floor \(1\.071\.07times it for Hindi,1\.161\.16for Nepali,1\.241\.24for Bengali and1\.821\.82for Thai, which is written without spaces\), but its floors are just as high\. Under the repaired regex the same text has 2\.2 to 5\.9 times fewer pre\-tokens, so each of these extensions stopped well short of what new vocabulary could achieve\. Yamaguchi et al\.’s Amharic extension, for a script without combining marks, stays at1\.571\.57times its floor, and the repair does not change that floor\.

Fig\. 1:Released merge\-based extensions of letters\-only models on FLORES\-200 devtest of each target language, relative to the letters\-only pre\-token floor\. The ten extensions by Yamaguchi et al\. \(top\) move from the base tokenizer \(×\\times\) to within 4% of the floor \(∙\\bullet\)\. LFM2\.5 \(bottom\) stops further above it in all four languages\. The blue tick is the floor under the repaired regex\. Full numbers, including the Amharic control, are in Table[VI](https://arxiv.org/html/2610.33016#A9.T6)\.
#### Added tokens bypass the floor\.

Strings registered as Hugging Face*added tokens*are matched in the raw text before the regex runs, so the bound does not apply to them\. SinLlama\[[3](https://arxiv.org/html/2610.33016#bib.bib42)\]extends Llama\-3 for Sinhala this way and goes below its letters\-only floor\. To test this route fairly, we registered as added tokens the 31,753 whole Devanagari strings among the 32,000 learned forR2\(§[V](https://arxiv.org/html/2610.33016#S5)\), on the unchanged regex\. Nepali then costs1\.0251\.025times English on Llama\-3 and1\.0091\.009on Qwen3, close toR2’s1\.0141\.014, and no FLORES\-200 sentence without a Devanagari character changes its token ids\. The floor therefore binds merge\-based extension only\. The price is the segmentation\. Added tokens match the longest string anywhere, ignoring word boundaries, and 21% of the matches start inside a word and 28% end inside one\. Allowing only whole\-word matches \(single\_word=True\) removes those splits and also the gain: the premium returns to2\.492\.49on Llama\-3 and4\.224\.22on Qwen3 \(Appendix[J](https://arxiv.org/html/2610.33016#A10)\)\. We did not train models with added tokens\.

## IVAudit of deployed tokenizers

We measured 33 released tokenizers on FLORES\-200 devtest\. Nine produce token ids identical to another tokenizer’s on every sentence of five languages we checked \(for example, Phi\-4, OLMo\-2 and Granite\-4\.1 matchcl100k, and Qwen3 matches Qwen2\.5\), which leaves 24 distinct tokenizers \(Figure[2](https://arxiv.org/html/2610.33016#S4.F2), Appendix[F](https://arxiv.org/html/2610.33016#A6)\)\. We classify each by its own pre\-tokenizer configuration, since models in one family do not always share a regex\. Four SentencePiece and WordPiece tokenizers \(XLM\-R, mT5, NLLB\-200, IndicBERTv2\) do not round\-trip their input, so their low premiums are not comparable; we mark them and exclude them from comparisons\.

Fig\. 2:Nepali/English premium of 24 distinct tokenizers \(33 released; “\+nn” counts aliases\) on FLORES\-200 devtest, with 95% paired\-bootstrap intervals\. Black ticks mark the pre\-token floor of each letters\-only tokenizer: no merge\-based extension can move its bar to the left of the tick\. Dotted line: parity with English\.Eight distinct tokenizers \(13 released models, including GLM\-5\.3 and LFM2\) use the letters\-only class, and none of them can be extended below2\.152\.15\. Leaving aside GPT\-2 \(7\.557\.55\), which we include only for reference, their premiums range from2\.582\.58\(Llama\-3\) to5\.775\.77\(LFM2\)\. The seven lossless tokenizers with a mark\-aware regex range from1\.121\.12\(Arkios\[[49](https://arxiv.org/html/2610.33016#bib.bib14)\]\) to2\.972\.97\(DeepSeek\-V3\)\.

We split each premium into three factors \(Appendix[G](https://arxiv.org/html/2610.33016#A7)\)\. The base term,0\.770\.77to0\.800\.80for every tokenizer, is what the premium would be if every Nepali pre\-token under the repaired regex were a single token\. The regex factor,2\.772\.77to2\.792\.79for the Llama\-3, Qwen andcl100kpattern, counts the extra pre\-tokens the tokenizer’s own regex creates\. The vocabulary factor, tokens per pre\-token, is the only one that merge\-based extension can reduce, and it cannot go below 1\. For Llama\-3 it is already1\.181\.18, so most of the remaining premium sits in the regex factor\. The factors are an accounting identity\. The regex factor measures room for improvement, and repairing the regex alone does not lower the premium \(§[VII\-A](https://arxiv.org/html/2610.33016#S7.SS1)\)\. Among mark\-aware tokenizers the vocabulary factor still ranges from1\.411\.41\(Arkios\) to3\.713\.71\(DeepSeek\-V3\), so a good regex also needs Nepali merges\. Within families, the Nepali premium halves from Qwen3 to Qwen3\.5 \(4\.424\.42to2\.202\.20\), whose word branch is close to our repaired class\[[46](https://arxiv.org/html/2610.33016#bib.bib50)\], and falls from2\.582\.58to1\.831\.83from Llama\-3 to Llama\-4, which adoptso200k’s pattern\. Both vocabularies also grew, so these changes are consistent with our account without isolating the regex\.

A well\-formed Devanagari syllable never starts with a combining mark, yet 55% of Llama\-3’s Nepali tokens do\. Eveno200k’s mark\-aware regex leaves 26%, because BPE also splits inside words\.

## VRepairing a deployed tokenizer

The repair has three parts\. We edit the regex, learn new merges on top of the existing ones, and initialise the new embeddings so that the model behaves exactly as before on old tokens\.

#### Regex edit\.

We add\\p\{M\}to the word class and exclude it from the optional leading character and from the punctuation class\. A mark can then only be matched as part of a word, so it stays with the letters around it\. For the Llama\-3 and Qwen pattern this takes two string substitutions that edit three character classes \(Appendix[B](https://arxiv.org/html/2610.33016#A2)\); digits, contractions, whitespace and punctuation are handled as before\. We call the result the*repaired regex*\. Qwen3\.5 ships almost the same word branch, as its released tokenizer file shows\[[46](https://arxiv.org/html/2610.33016#bib.bib50)\]\.

#### Continued BPE\.

Following[Purason et al\. \[45\]](https://arxiv.org/html/2610.33016#bib.bib28), we pre\-tokenize Nepali text with the arm’s regex, segment every pre\-token with the original merges, as the deployed model sees it, and run BPE over these segmentations with each base token as an indivisible symbol\. TheKKnew merges are appended after the existing ones, so the original merges act first\. We discard any merge whose output contains no complete code point in U\+0900–U\+097F, together with merges that depend on it\.R1andR2run this procedure on the same text with the sameKKand differ only in the regex\.

###### Proposition 1\(English invariance\)\.

Letssbe a normalised string that contains no combining mark \(\\p\{M\}\) and no code point in U\+0900–U\+097F\. Then the repaired tokenizerR2and the original tokenizer produce identical token ids forss\.

The proof is in Appendix[C](https://arxiv.org/html/2610.33016#A3)\. The proposition covers ASCII English and code and every script whose normalised text has no combining marks\. It does not cover other scripts that use marks: the general repair changes the pre\-tokens of 61 of the 204 FLORES\-200 languages, including Arabic, Bengali and Thai, whose tokenization the base model has seen only in its original form\.

#### A Devanagari\-only repair\.

Admitting only the 34 combining marks of the Devanagari block, instead of all of\\p\{M\}, gives a stronger guarantee by the same argument: token ids are unchanged for every input without a code point in U\+0900–U\+097F\. On FLORES\-200 this variant changes pre\-tokens only in the ten Devanagari languages, and on FLORES Nepali itsR2token ids are identical to the general repair’s\. On the FineWeb\-2 Nepali test split \(275 MB, one document per line\), 24 of 49,462 lines are pre\-tokenized differently, all because of marks from other blocks, such as emoji variation selectors and Kannada vowel signs \(Appendix[B](https://arxiv.org/html/2610.33016#A2)\)\. We recommend this variant for deployment\. All experiments in this paper, and the released tokenizers, use the general repair\.

#### Embedding initialisation\.

Each new token’s input and output rows are set to the mean of the rows of its constituent base tokens\[[20](https://arxiv.org/html/2610.33016#bib.bib25)\], obtained by applying the original merges to its byte\-level string\. Original rows are untouched, so on any input whose ids are all original the resized model gives the same logits for every original token as the base model; on an English probe the largest difference is below10−410^\{\-4\}\. The new tokens do add terms to the softmax denominator, and continued pretraining is needed to train them\.

#### Loading pitfall\.

In transformers 5\.x, the classQwen2Tokenizerrebuilds the pre\-tokenizer from a hard\-coded pattern when it loads, which silently undoes a repaired regex stored intokenizer\.json\. Our released tokenizers therefore settokenizer\_classtoPreTrainedTokenizerFast, and we check after loading that the premium matches the paper\.

#### Arms\.

R0\(no extension\) keeps the original tokenizer\.R1\(standard extension\) appendsKKmerges learned under the original regex, as in[Purason et al\. \[45\]](https://arxiv.org/html/2610.33016#bib.bib28)and[Yamaguchi et al\. \[62\]](https://arxiv.org/html/2610.33016#bib.bib35)\.R2\(repair \+ extension\) switches to the repaired regex and appendsKKmerges learned under it\.

## VIExperimental setup

We committed an analysis plan to our own repository before training any model, then seven amendments: budget cuts and the primary analysis \(before any result\), warm\-up runs \(after the first Llama results\), a Llama rerun with the beginning\-of\-sequence \(BOS\) token in training, matched\-steps and second\-seed runs, further seeds with a learning\-rate check,R0at the higher rate, and a missing second seed \(each of the last five before running it\)\. The study up to the BOS rerun took about 11 hours, and several analysis choices changed after we saw results; Appendix[A](https://arxiv.org/html/2610.33016#A1)lists every commit and deviation, and Appendix[D](https://arxiv.org/html/2610.33016#A4)the training and evaluation details\.

#### Models and data\.

We use Llama\-3\.2\-1B \(base\)\[[22](https://arxiv.org/html/2610.33016#bib.bib53)\]and Qwen3\-0\.6B\-Base\[[63](https://arxiv.org/html/2610.33016#bib.bib54)\], whose tokenizers ship the letters\-only regex and which tie input and output embeddings\. The Qwen3 tokenizer is identical across model sizes; we built it on Qwen3\-1\.7B\-Base\. Nepali text is FineWeb\-2npi\_Deva\[[43](https://arxiv.org/html/2610.33016#bib.bib51)\]and English is FineWeb\-Edu\[[42](https://arxiv.org/html/2610.33016#bib.bib52)\]\. We normalise to NFC, deduplicate, and remove every document that shares a whitespace 10\-gram with FLORES\-200 dev or devtest\. A 30 MB held\-out slice of each language is never trained on\.R1andR2learn merges on the same 1 GB of Nepali, with the registeredK=32,000K\{=\}32\{,\}000\.

#### Continued pretraining\.

Each arm is trained for one epoch on the same documents, 1 GB of Nepali and 1 GB of English interleaved in one fixed order, with identical optimisation: a fixed, untuned peak learning rate of10−410^\{\-4\}and cosine decay \(later amendments add second seeds and, on Qwen, a rate of3×10−43\{\\times\}10^\{\-4\}\)\. BecauseR2encodes the same bytes in fewer tokens, it takes fewer steps\.R0andR1also save a checkpoint at the step whereR2ends, before their learning rate has fully decayed\. We call these the*main runs*\. The*warm\-up runs*\(Llama only\) repeat them after first training only the embedding matrix on 10% of the stream at a learning rate of10−310^\{\-3\}\.

#### Evaluation\.

All metrics are normalised by UTF\-8 bytes\. The primary metric is bits per byte \(BPB\) on held\-out native Nepali,*byte\-matched*: each document is split at whitespace into pieces of about 2,000 bytes, and every arm scores the same pieces\. Each piece is scored after one context token\. The*registered prefix*is the BOS token for Llama and the end\-of\-text token for Qwen, whose tokenizer defines no BOS token\. In the main and warm\-up runs, however, documents were separated only by the end\-of\-text token, so the Llama models never saw BOS\. We therefore also score them after that token, the*training separator*, as a diagnostic chosen after seeing results, and we reran the Llama main runs with BOS at every document start \(§[VII\-B](https://arxiv.org/html/2610.33016#S7.SS2)\)\. Secondary metrics \(held\-out English BPB, FLORES BPB, Belebele, 5\-shot translation\) are in Appendix[K](https://arxiv.org/html/2610.33016#A11)\.

#### Analysis\.

The registered main hypothesis was thatR2is non\-inferior to bothR1andR0on held\-out Nepali BPB, with a margin of 0\.01, while using fewer training tokens\. The labels for the 95% interval of \(R2−\-other\) are applied in a fixed order:*better*if it lies entirely below 0; otherwise*non\-inferior*if it lies entirely below\+0\.01\+0\.01; otherwise*worse*if it lies entirely above 0; otherwise*no detectable difference*\. So\[\+0\.004,\+0\.007\]\[\+0\.004,\+0\.007\]is non\-inferior although it excludes zero\. The label “worse” was added in the second amendment\. The registered test was a paired bootstrap over held\-out pieces\. Pieces of one document are correlated, so we report a bootstrap over documents instead \(B=10,000B\{=\}10\{,\}000\)\. Clustering by the 503 held\-out documents \(Appendix[D](https://arxiv.org/html/2610.33016#A4)\) widens the Nepali intervals 1\.4 to 2\.6 times and changes no verdict\. The same test on English pieces is the registered English control\. All intervals describe the sampling of evaluation text\. Run\-to\-run variance is measured only by second seeds \(Appendix[M](https://arxiv.org/html/2610.33016#A13)\)\.

## VIIResults

TABLE I:Continued pretraining, seed 0 of each arm, on the same 1 GB of Nepali and 1 GB of English\.Our results are the BOS\-rerun \(Llama\) and Qwen columns; two\-seed means and matched\-steps runs are in Table[IX](https://arxiv.org/html/2610.33016#A13.T9); the other Llama columns show the scoring artifact of §[VII\-B](https://arxiv.org/html/2610.33016#S7.SS2)\. BPB: held\-out native Nepali, in pieces of about 2,000 bytes, scored after the registered prefix \(BOS for Llama; end\-of\-text for Qwen, which has no BOS\) or the training separator \(end\-of\-text; a post hoc diagnostic\)\. Intervals resample 503 held\-out documents and reflect only the sampling of evaluation text\. Training tokens and decoding steps \(FLORES Nepali, relative toR2\) areR0/R1/R2\. Secondary metrics: Table[VIII](https://arxiv.org/html/2610.33016#A11.T8)\.### VII\-ACompression

Figure[3](https://arxiv.org/html/2610.33016#S7.F3)sweeps the number of new mergesKK\. Under the letters\-only regex \(R1\), extension saturates almost at once\. For Llama\-3 the first 1,000 merges take the premium from2\.582\.58to2\.322\.32, and the next 63,000 only to2\.202\.20, within0\.010\.01of the floor\. For Qwen3,R1reaches2\.192\.19byK=16K\{=\}16k and stops at2\.182\.18, against a floor of2\.172\.17\. It stops early because the letters\-only regex leaves few distinct Nepali pre\-tokens to merge \(Appendix[J](https://arxiv.org/html/2610.33016#A10)\)\. Under the repaired regex \(R2\), additional merges keep helping\. AtK=32K\{=\}32k Nepali reaches parity with English \(1\.011\.01\) on both models, and at6464k it is cheaper than English \(0\.950\.95\), belowo200k\. AtK=32K\{=\}32k,R2needs 46% ofR1’s tokens for FLORES Nepali\. All 28 extended tokenizers decode losslessly and reproduce the original token ids on all 1,012 English devtest sentences, as §[III](https://arxiv.org/html/2610.33016#S3)and Proposition[1](https://arxiv.org/html/2610.33016#Thmproposition1)predict\.

Fig\. 3:Nepali/English premium on FLORES\-200 devtest after addingKKmerges by continued BPE, under the letters\-only regex \(R1\) and the repaired regex \(R2\)\. Dashed lines: pre\-token floors under each model’s own pattern \(2\.192\.19and2\.172\.17letters\-only;0\.790\.79and0\.800\.80repaired, for Llama\-3 and Qwen3\)\. 95% paired\-bootstrap intervals \(half\-width at most 0\.03\) are narrower than the markers\.Both parts of the repair are needed\. Repairing the regex without new merges changes almost nothing \(2\.612\.61for Llama\-3,4\.424\.42for Qwen3\), because the old merges still cut Nepali words into the old pieces\. The repair also changes where tokens start\. The share of Nepali tokens that start with a combining mark, a rough sign of a split syllable, rises from 55% \(Llama\-3\) and 32% \(Qwen3\) to 65% and 64% underR1, since new merges can only build on mark\-initial fragments, and falls to 13% and 11% underR2\. The mark\-initial share does not predict model quality in §[VII\-B](https://arxiv.org/html/2610.33016#S7.SS2):R2has by far the fewest mark\-initial tokens yet trailsR0on both models\.

We repeated the sweep for seven other languages written with combining marks and two controls \(Table[V](https://arxiv.org/html/2610.33016#A8.T5), Appendix[H](https://arxiv.org/html/2610.33016#A8)\), learning merges on 25 to 361 MB of FineWeb\-2 text per language\. AtK=32K\{=\}32k,R2brings all seven to between0\.880\.88and1\.241\.24times English\.R1ends within 5% of its floor in six of them\. Thai is the exception: it is written without spaces, so its letters\-only pre\-tokens are long and leaveR1room\. For Amharic and Korean, whose scripts have no combining marks,R1andR2are identical at everyKK, as the mechanism predicts\.

### VII\-BContinued pretraining

#### The registered outcome\.

Table[I](https://arxiv.org/html/2610.33016#S7.T1)gives the results\. The main hypothesis fails againstR0: continued pretraining without new tokens gave the lowest held\-out Nepali BPB on both models\. On Qwen,R2was worse thanR0by0\.0700\.070, far beyond the 0\.01 margin, andR1was worse by0\.0600\.060\. Against standard extension,R2−\-R1=\+0\.0094=\+0\.0094\[\+0\.0084,\+0\.0104\]\[\+0\.0084,\+0\.0104\]\(registered piece\-level bootstrap:\[\+0\.0086,\+0\.0101\]\[\+0\.0086,\+0\.0101\]\), labelled “worse” although the point estimate lies inside the margin; averaged over two seeds it is\+0\.0122\+0\.0122\[\+0\.0112,\+0\.0132\]\[\+0\.0112,\+0\.0132\]\.

#### The BOS\-consistent rerun\.

The main Llama runs never saw BOS \(§[VI](https://arxiv.org/html/2610.33016#S6)\)\. Under the registered prefix they giveR2−\-R1=\+0\.0058=\+0\.0058\[\+0\.0043,\+0\.0073\]\[\+0\.0043,\+0\.0073\], non\-inferior although it excludes zero, but\+0\.027\+0\.027after the training separator\. A third amendment retrained the three Llama arms with BOS before every document and nothing else changed; we take this rerun as our Llama result\.R0,R1andR2reach0\.4710\.471,0\.4830\.483and0\.5090\.509:R1is worse thanR0by0\.01240\.0124\[0\.0114,0\.0135\]\[0\.0114,0\.0135\]andR2worse thanR1by0\.02560\.0256\[0\.0245,0\.0267\]\[0\.0245,0\.0267\], about 5% \(0\.02520\.0252over two seeds\)\. These match the separator diagnostic within0\.0020\.002, so the main runs’ registered\-prefix numbers reflect a scoring artifact of the unseen BOS token; after the end\-of\-text token the rerun scores0\.8750\.875to1\.6661\.666\(Appendix[K](https://arxiv.org/html/2610.33016#A11)\)\.

#### Matched steps\.

At equal bytes,R2takes 26% fewer steps thanR1\. A fourth amendment retrainedR2forR1’s step count \(737 for Llama with BOS, 748 for Qwen\), drawing the extra 35% of windows from a second pass over its stream, and a fifth added second seeds\. Averaged over two seeds, matched\-stepsR2is*better*thanR1on Qwen by0\.00990\.0099\[0\.0088,0\.0109\]\[0\.0088,0\.0109\]\(0\.5490\.549against0\.5590\.559\) and worse on Llama by0\.01200\.0120\[0\.0111,0\.0129\]\[0\.0111,0\.0129\]\(0\.4950\.495against0\.4830\.483\);R0stays best, by0\.0490\.049and0\.0240\.024\. Matching steps thus reverses the equal\-bytes gap on Qwen and halves it on Llama, though with 35% of the data repeated\. Seeds move each arm by about0\.0030\.003or less \(Appendix[M](https://arxiv.org/html/2610.33016#A13)\)\.

#### Learning rate\.

A fifth and a sixth amendment repeated the Qwen arms at3×10−43\{\\times\}10^\{\-4\}instead of10−410^\{\-4\}\(one seed,R2at matched steps\)\. Every arm improves,R0by0\.0210\.021,R1by0\.0520\.052andR2by0\.0460\.046, but the order holds:R00\.4790\.479,R20\.5040\.504,R10\.5090\.509\.R2beatsR1by0\.00570\.0057\[0\.0047,0\.0066\]\[0\.0047,0\.0066\]and trailsR0by0\.0240\.024\[0\.023,0\.026\]\[0\.023,0\.026\], so our predictions thatR0would gain little andR2would match it failed\. English gets worse:0\.8750\.875to0\.8790\.879BPB, against0\.8620\.862to0\.8640\.864at10−410^\{\-4\}and0\.8730\.873for the base model\.

#### What the repair buys\.

The clear gain is in token count\. For the same FLORES Nepali text,R2needs 46% ofR1’s tokens on both models, and 39% \(Llama\) and 23% \(Qwen\) ofR0’s\. We have not shown faster generation of equally good text: our timings forced 128 generated tokens per model on shared GPUs, used the original Llama runs and compared different outputs\. In themR2’s generations carry 1\.55 \(Llama\) and 1\.47 \(Qwen\) times as many bytes per token asR1’s, and throughput is 1\.48 and 1\.50 times higher, which we treat as indicative only\.

#### English\.

The registered English control holds:R2−\-R0on held\-out English is−0\.0010\-0\.0010\[−0\.0014,−0\.0007\]\[\-0\.0014,\-0\.0007\]\(Llama rerun\) and−0\.0028\-0\.0028\[−0\.0032,−0\.0024\]\[\-0\.0032,\-0\.0024\]\(Qwen\), andR2beatsR1by0\.00220\.0022on both\. “English\-preserving” refers to token ids only: every Llama arm ends worse on English than the base model \(0\.7980\.798to0\.8040\.804against0\.7880\.788\)\.

## VIIIDiscussion

#### Recommendations\.

Before extending a letters\-only model for a language with combining marks, compare the target text’s pre\-tokens with its tokens; if they are close, as for Llama\-3 on Nepali, merges have little room\. Under our recipes, plain continued pretraining gave the lowest held\-out Nepali BPB, as[Zhao et al\. \[64\]](https://arxiv.org/html/2610.33016#bib.bib56)found for Chinese, so extension needs another reason, such as token cost\. Then the repair is worth considering: at matched steps it was within about 0\.012 BPB of standard extension either way and needs 54% fewer tokens for the same Nepali text, though we have not shown a latency gain for equally good outputs\. We would use the Devanagari\-only repair \(untrained here; identical to ours on FLORES Nepali\), load it with a class that does not rebuild its pre\-tokenizer, and score models after the context they were trained with\.

#### Why our sign differs from Regmi et al\.

From scratch the wider class lowers Nepali BPB by 4\.4% at equal compute and still leads at equal bytes\[[50](https://arxiv.org/html/2610.33016#bib.bib13)\]; retrofitted at equal bytes it is worse, and the gap decreases under longer training with repeated data\[[55](https://arxiv.org/html/2610.33016#bib.bib15), cf\.\]: matched steps flip the sign on Qwen and halve the gap on Llama\.

## Limitations

- •The main hypothesis failed\.At our budget, neither extension recipe beat continued pretraining without new tokens on Nepali BPB\. At equal bytes the repair was also worse than standard extension on both models; at matched steps it was better on Qwen and still worse on Llama\.
- •Scale\.We adapt 1B\- and 0\.6B\-parameter models with 1 GB of text per language and one epoch; 1 GB of Nepali also holds less content than 1 GB of English\[[4](https://arxiv.org/html/2610.33016#bib.bib55)\]\. The registered 3 GB runs were dropped for budget reasons, so we cannot say whether the gap between arms closes with more data;[Dagan et al\. \[13\]](https://arxiv.org/html/2610.33016#bib.bib11)found recovery from a tokenizer change only after about 50B tokens in their code\-adaptation setting\.
- •Matched steps repeat data\.The matched\-stepsR2runs reachR1’s step count by seeing 35% of their stream twice; they hold compute equal, not data, though repeating data for a few epochs costs little\[[37](https://arxiv.org/html/2610.33016#bib.bib57)\]\. We did not runR0at matched steps, and Nepali is still a smaller share ofR2’s training tokens \(26% against 45% forR1\)\.
- •Scoring two tokenizations\.For the seed\-0 equal\-bytes models we add the base tokenization to the canonical one, a partial step towards the marginal likelihood\[[7](https://arxiv.org/html/2610.33016#bib.bib48),[8](https://arxiv.org/html/2610.33016#bib.bib49)\]\. It changes no score, because the models give the base tokenization almost no probability; this is a weak check, since nearer alternatives, such as splitting one new token, could still carry mass\.
- •Few runs\.Every interval reflects only the sampling of evaluation text\. In the Llama BOS rerun and on Qwen, every arm at10−410^\{\-4\}has two seeds, which differ by about0\.0030\.003or less per arm; the runs at3×10−43\{\\times\}10^\{\-4\}have one\.
- •Clustering\.Nepali intervals resample the 503 held\-out documents; English intervals still cluster pieces by a length heuristic\.
- •Learning rate\.We tried two learning rates, on Qwen only and with one seed at the higher one; continued\-pretraining schedules matter\[[23](https://arxiv.org/html/2610.33016#bib.bib59)\], and the English loss at the higher rate suggests forgetting\. Moving from10−410^\{\-4\}to3×10−43\{\\times\}10^\{\-4\}improved every arm by0\.020\.02to0\.050\.05without changing their order, so the best rate for each arm may differ from both and Llama was not tuned at all\. Mean initialisation may also suitR2’s long new tokens less well thanR1’s short ones\[[24](https://arxiv.org/html/2610.33016#bib.bib40)\]\.
- •BOS in training\.The main and warm\-up Llama runs never saw BOS, which the registered evaluation puts first, so their registered scores and all their secondary metrics carry this artifact; the warm\-up translation scores are unusable\. The rerun with BOS in training resolves the main comparison, but only on BPB: we did not run Belebele, translation or generation for it, nor a warm\-up version\. Qwen has no BOS token and was not affected\.
- •Weak secondary metrics\.Belebele accuracy is near chance for every model, translation scores come from a single greedy pass without intervals and keep only the first output line, which is empty whenever a model begins its answer with a newline \(we did not count how often\), and measured generation speed was taken on shared GPUs \(Appendix[K](https://arxiv.org/html/2610.33016#A11)\)\.
- •Equal compute\.The equal\-compute checkpoints ofR0andR1are taken before their learning rate has fully decayed, and are scored on 1 MB only\.
- •Languages\.We train models for Nepali only\. For other languages we report compression, and the general repair, which we trained and released, changes the tokenization of 61 FLORES\-200 languages whose quality we did not measure, including Hindi and Thai, which Llama\-3\.2 officially supports\. The cross\-script tokenizers use a relaxed merge filter not covered by Proposition[1](https://arxiv.org/html/2610.33016#Thmproposition1)\(Appendix[H](https://arxiv.org/html/2610.33016#A8)\)\.
- •Added tokens and segmentation\.We measured the added\-token route around the floor only at the tokenizer level and trained no model with it;[Churchill and Skiena \[10\]](https://arxiv.org/html/2610.33016#bib.bib58)add tokens post hoc to Llama 3\.2 1B for whole characters, not words, and[Balde et al\. \[5\]](https://arxiv.org/html/2610.33016#bib.bib65)change how added tokens are matched\. Our objection to it is segmentation, and we have no evidence that segmentation matters for quality\. Whether the regex repair beats added tokens is untested\.
- •Scoring context\.The BOS\-consistent rerun scores every piece after BOS, but 1,756 of the 2,259 pieces start mid\-document, a context these models never saw after BOS in training\.
- •What would settle it\.A learning\-rate sweep on both models, better embedding initialisation and a curve over data budgets would show whetherR0’s lead and the Llama gap persist\.

## Ethics and use of AI assistance

All text and evaluation data are public \(FineWeb\-2, FineWeb\-Edu, FLORES\-200, Belebele\) and used under their licences; we collected no data from people\. Released Llama\-derived tokenizers follow the Llama 3\.2 Community License\[[33](https://arxiv.org/html/2610.33016#bib.bib45)\]; Qwen3\-derived tokenizers are Apache\-2\.0\.

We used a large language model assistant extensively\. Under our direction it wrote and ran the experiment code, searched the literature and drafted the text\. We reviewed and approved the study design, the paper and all citations, and take responsibility for the content\. The tokenizer\-side numbers can be reproduced from public tokenizers and data with our released code, and every continued\-pretraining statistic from the released per\-piece scores\.

## References

- \[1\]O\. Ahia, S\. Kumar, H\. Gonen, J\. Kasai, D\. Mortensen, N\. Smith, and Y\. Tsvetkov\(2023\)Do all languages cost the same? Tokenization in the era of commercial language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 9904–9923\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.614/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.614)Cited by:[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px1.p1.1)\.
- \[2\]ajaykg\(2024\)Combining marks and Indic vowel marks within words are being split breaking all Indic languages and most languages except English and CJKs \(tiktoken issue \#292\)\.Note:GitHub issue, openai/tiktokenExternal Links:[Link](https://github.com/openai/tiktoken/issues/292)Cited by:[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px2.p1.1)\.
- \[3\]H\. W\. K\. Aravinda, R\. Sirajudeen, S\. Karunathilake, N\. de Silva, R\. Kaur, and S\. Ranathunga\(2025\)SinLlama – a large language model for Sinhala\.In2025 Moratuwa Engineering Research Conference \(MERCon\),pp\. 617–622\.External Links:[Document](https://dx.doi.org/10.1109/MERCon67903.2025.11217094),2508\.09115Cited by:[Appendix I](https://arxiv.org/html/2610.33016#A9.p1.1),[§III](https://arxiv.org/html/2610.33016#S3.SS0.SSS0.Px2.p1.1)\.
- \[4\]C\. Arnett, T\. A\. Chang, and B\. Bergen\(2024\)A bit of a problem: measurement disparities in dataset sizes across languages\.InProceedings of the 3rd Annual Meeting of the Special Interest Group on Under\-resourced Languages @ LREC\-COLING 2024,Torino, Italia,pp\. 1–9\.External Links:[Link](https://aclanthology.org/2024.sigul-1.1/)Cited by:[2nd item](https://arxiv.org/html/2610.33016#Sx1.I1.i2.p1.1)\.
- \[5\]G\. Balde, S\. Roy, M\. Mondal, and N\. Ganguly\(2024\)Adaptive BPE tokenization for enhanced vocabulary adaptation in finetuning pretrained language models\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Miami, Florida, USA,pp\. 14724–14733\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.863/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.863)Cited by:[12nd item](https://arxiv.org/html/2610.33016#Sx1.I1.i12.p1.1)\.
- \[6\]L\. Bandarkar, D\. Liang, B\. Muller, M\. Artetxe, S\. N\. Shukla, D\. Husa, N\. Goyal, A\. Krishnan, L\. Zettlemoyer, and M\. Khabsa\(2024\)The Belebele benchmark: a parallel reading comprehension dataset in 122 language variants\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 749–775\.External Links:[Link](https://aclanthology.org/2024.acl-long.44/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.44)Cited by:[2nd item](https://arxiv.org/html/2610.33016#A4.I1.i2.p1.1),[Appendix D](https://arxiv.org/html/2610.33016#A4.SS0.SSS0.Px1.p1.1)\.
- \[7\]K\. Cao and L\. Rimell\(2021\)You should evaluate your language model on marginal likelihood over tokenisations\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,pp\. 2104–2114\.External Links:[Link](https://aclanthology.org/2021.emnlp-main.161/),[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.161)Cited by:[4th item](https://arxiv.org/html/2610.33016#Sx1.I1.i4.p1.1)\.
- \[8\]N\. Chirkova, G\. Kruszewski, J\. Rozen, and M\. Dymetman\(2023\)Should you marginalize over possible tokenizations?\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),pp\. 1–12\.External Links:[Link](https://aclanthology.org/2023.acl-short.1/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-short.1)Cited by:[4th item](https://arxiv.org/html/2610.33016#Sx1.I1.i4.p1.1)\.
- \[9\]M\. Choudhury, S\. Chauhan, R\. J\. Das, D\. Sahnan, X\. Han, H\. Li, A\. Singh, A\. A\. Jadhav, U\. Agarwal, M\. Choudhary, D\. Banerjee, F\. Koto, J\. Bhat, A\. Shukla, S\. Ghosh, S\. Kamboj, O\. Pandit, L\. Pradhan, R\. Pal, S\. Sahu, S\. Doraiswamy, P\. Mullah, A\. El Filali, N\. Sengupta, G\. Ramakrishnan, R\. Joshi, G\. Gosal, A\. Sheinin, N\. Vassilieva, and P\. Nakov\(2025\)Llama\-3\-Nanda\-10B\-Chat: an open generative large language model for Hindi\.External Links:2504\.06011Cited by:[Appendix I](https://arxiv.org/html/2610.33016#A9.p1.1)\.
- \[10\]G\. Churchill and S\. Skiena\(2026\)Reducing tokenization premiums for low\-resource languages\.External Links:2601\.13328,[Link](https://arxiv.org/abs/2601.13328)Cited by:[§I](https://arxiv.org/html/2610.33016#S1.SS0.SSS0.Px1.p1.1),[12nd item](https://arxiv.org/html/2610.33016#Sx1.I1.i12.p1.1)\.
- \[11\]Z\. Csaki, B\. Li, J\. L\. Li, Q\. Xu, P\. Pawakapan, L\. Zhang, Y\. Du, H\. Zhao, C\. Hu, and U\. Thakker\(2024\)SambaLingo: teaching large language models new languages\.InProceedings of the Fourth Workshop on Multilingual Representation Learning \(MRL 2024\),J\. Sälevä and A\. Owodunni \(Eds\.\),Miami, Florida, USA,pp\. 1–21\.External Links:[Link](https://aclanthology.org/2024.mrl-1.1/),[Document](https://dx.doi.org/10.18653/v1/2024.mrl-1.1)Cited by:[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px3.p1.1)\.
- \[12\]Y\. Cui, Z\. Yang, and X\. Yao\(2023\)Efficient and effective text encoding for Chinese LLaMA and Alpaca\.External Links:2304\.08177,[Link](https://arxiv.org/abs/2304.08177)Cited by:[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px3.p1.1)\.
- \[13\]G\. Dagan, G\. Synnaeve, and B\. Rozière\(2024\)Getting the most out of your tokenizer for pre\-training and domain adaptation\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 9784–9805\.External Links:[Link](https://proceedings.mlr.press/v235/dagan24a.html)Cited by:[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px3.p1.1),[2nd item](https://arxiv.org/html/2610.33016#Sx1.I1.i2.p1.1)\.
- \[14\]K\. Darshana\(2026\)Separate before you compress: the WWHO tokenization architecture\.External Links:2603\.25309,[Link](https://arxiv.org/abs/2603.25309)Cited by:[Appendix L](https://arxiv.org/html/2610.33016#A12.p1.1),[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px3.p1.1)\.
- \[15\]B\. Didenko\(2026\)Writing\-system\-level tokenizer adaptation for byte\-level BPE\.Note:TokShop @ COLM 2026, non\-archivalExternal Links:2608\.00582,[Link](https://arxiv.org/abs/2608.00582)Cited by:[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px3.p1.1)\.
- \[16\]K\. Dobler and G\. de Melo\(2023\)FOCUS: effective embedding initialization for monolingual specialization of multilingual models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 13440–13454\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.829/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.829)Cited by:[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px3.p1.1)\.
- \[17\]K\. Dobler and G\. de Melo\(2024\)Language adaptation on a tight academic compute budget: tokenizer swapping works and pure bfloat16 is enough\.Note:WANT@ICML 2024External Links:2408\.15793Cited by:[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px3.p1.1)\.
- \[18\]S\. Duwal, S\. Prasai, and S\. Manandhar\(2025\)Domain\-adaptative continual learning for low\-resource tasks: evaluation on Nepali\.InProceedings of the First Workshop on Challenges in Processing South Asian Languages \(CHiPSAL 2025\),Abu Dhabi, UAE,pp\. 144–153\.External Links:[Link](https://aclanthology.org/2025.chipsal-1.14/)Cited by:[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px3.p1.1)\.
- \[19\]D\. Feher, I\. Vulić, and B\. Minixhofer\(2024\)Retrofitting large language models with dynamic tokenization\.External Links:2411\.18553Cited by:[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px3.p1.1)\.
- \[20\]L\. Gee, A\. Zugarini, L\. Rigutini, and P\. Torroni\(2022\)Fast vocabulary transfer for language model compression\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track,Y\. Li and A\. Lazaridou \(Eds\.\),Abu Dhabi, UAE,pp\. 409–416\.External Links:[Link](https://aclanthology.org/2022.emnlp-industry.41/),[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-industry.41)Cited by:[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px3.p1.1),[§V](https://arxiv.org/html/2610.33016#S5.SS0.SSS0.Px4.p1.1)\.
- \[21\]N\. Goyal, C\. Gao, V\. Chaudhary, P\. Chen, G\. Wenzek, D\. Ju, S\. Krishnan, M\. Ranzato, F\. Guzmán, and A\. Fan\(2022\)The Flores\-101 evaluation benchmark for low\-resource and multilingual machine translation\.Transactions of the Association for Computational Linguistics10,pp\. 522–538\.External Links:[Link](https://aclanthology.org/2022.tacl-1.30/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00474)Cited by:[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px1.p1.1)\.
- \[22\]A\. Grattafioriet al\.\(2024\)The Llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[§VI](https://arxiv.org/html/2610.33016#S6.SS0.SSS0.Px1.p1.1)\.
- \[23\]A\. Ibrahim, B\. Thérien, K\. Gupta, M\. L\. Richter, Q\. Anthony, T\. Lesort, E\. Belilovsky, and I\. Rish\(2024\)Simple and scalable strategies to continually pre\-train large language models\.Transactions on Machine Learning Research\.External Links:[Link](https://openreview.net/forum?id=DimPeeCxKO)Cited by:[7th item](https://arxiv.org/html/2610.33016#Sx1.I1.i7.p1.1)\.
- \[24\]R\. Joshi, U\. Vaidya, S\. S\. Chauhan, and N\. Wartikar\(2026\)Beyond initialization loss: a systematic study of token embedding initialization strategies for LLM vocabulary extension\.External Links:2608\.03494Cited by:[7th item](https://arxiv.org/html/2610.33016#Sx1.I1.i7.p1.1)\.
- \[25\]R\. Kadariya\(2026\)How well does AI understand Nepali? A multi\-task benchmark of large language models and embedding models on a low\-resource language\.Note:Preprint \(Zenodo/ResearchGate\)External Links:[Document](https://dx.doi.org/10.13140/RG.2.2.27979.55840),[Link](https://zenodo.org/records/21320729)Cited by:[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px1.p1.1)\.
- \[26\]S\. H\. Kapila and R\. Bagavathy\(2026\)Type\-driven tokenization for Brahmic scripts\.External Links:2609\.22125,[Link](https://arxiv.org/abs/2609.22125)Cited by:[Appendix L](https://arxiv.org/html/2610.33016#A12.p1.1),[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px3.p1.1)\.
- \[27\]S\. Kim, S\. Choi, and M\. Jeong\(2024\)Efficient and effective vocabulary expansion towards multilingual large language models\.External Links:2402\.14714Cited by:[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px3.p1.1)\.
- \[28\]S\. Land\(2026\)Small pre\-tokenization bugs with a big multilingual price\.Note:Substack blog postExternal Links:[Link](https://tokencontributions.substack.com/p/small-pre-tokenization-bugs-with)Cited by:[Appendix G](https://arxiv.org/html/2610.33016#A7.p2.1),[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px2.p1.1)\.
- \[29\]Y\. Liu, P\. Lin, M\. Wang, and H\. Schuetze\(2024\)OFA: a framework of initializing unseen subword embeddings for efficient large\-scale multilingual continued pretraining\.InFindings of the Association for Computational Linguistics: NAACL 2024,K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 1067–1097\.External Links:[Link](https://aclanthology.org/2024.findings-naacl.68/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-naacl.68)Cited by:[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px3.p1.1)\.
- \[30\]N\. Luitel, N\. Bekoju, A\. K\. Sah, and S\. Shakya\(2025\)Can perplexity predict fine\-tuning performance? an investigation of tokenization effects on sequential language models for Nepali\.External Links:2404\.18071,[Link](https://arxiv.org/abs/2404.18071)Cited by:[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px1.p1.1)\.
- \[31\]J\. M\. Lundin, A\. Zhang, N\. Karim, H\. Louzan, G\. Wei, D\. I\. Adelani, and C\. Carroll\(2026\)The token tax: systematic bias in multilingual tokenization\.InProceedings of the 7th Workshop on African Natural Language Processing \(AfricaNLP 2026\),E\. A\. Chimoto, C\. Lignos, S\. Muhammad, I\. Abdulmumin, C\. Siro, and D\. I\. Adelani \(Eds\.\),Rabat, Morocco,pp\. 103–112\.External Links:[Link](https://aclanthology.org/2026.africanlp-main.10/),[Document](https://dx.doi.org/10.18653/v1/2026.africanlp-main.10),ISBN 979\-8\-89176\-364\-7Cited by:[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px1.p1.1)\.
- \[32\]M\. Mehta, N\. Subramani, Z\. Xu, A\. Gupta, and V\. Srikumar\(2026\)Defragmenting language models: an interpretability\-based approach for vocabulary expansion\.External Links:2604\.16656,[Link](https://arxiv.org/abs/2604.16656)Cited by:[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px3.p1.1)\.
- \[33\]Meta Platforms, Inc\.\(2024\)Llama 3\.2 Community License Agreement\.Note:[https://github\.com/meta\-llama/llama\-models/blob/main/models/llama3\_2/LICENSE](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/LICENSE)Version release date 25 Sep 2024; accessed 2026\-10\-04Cited by:[Ethics and use of AI assistance](https://arxiv.org/html/2610.33016#Sx2.p1.1)\.
- \[34\]B\. Minixhofer, F\. Paischer, and N\. Rekabsaz\(2022\)WECHSEL: effective initialization of subword embeddings for cross\-lingual transfer of monolingual language models\.InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,M\. Carpuat, M\. de Marneffe, and I\. V\. Meza Ruiz \(Eds\.\),Seattle, United States,pp\. 3992–4006\.External Links:[Link](https://aclanthology.org/2022.naacl-main.293/),[Document](https://dx.doi.org/10.18653/v1/2022.naacl-main.293)Cited by:[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px3.p1.1)\.
- \[35\]B\. Minixhofer, E\. M\. Ponti, and I\. Vulić\(2024\)Zero\-shot tokenizer transfer\.InAdvances in Neural Information Processing Systems 37 \(NeurIPS 2024\),pp\. 46791–46818\.External Links:[Document](https://dx.doi.org/10.52202/079017-1484),2405\.07883Cited by:[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px2.p1.1),[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px3.p1.1)\.
- \[36\]B\. Minixhofer, I\. Vulić, and E\. M\. Ponti\(2025\)Universal cross\-tokenizer distillation via approximate likelihood matching\.InAdvances in Neural Information Processing Systems 38 \(NeurIPS 2025\),pp\. 88034–88063\.External Links:[Document](https://dx.doi.org/10.52202/085713-2653),2503\.20083Cited by:[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px3.p1.1)\.
- \[37\]N\. Muennighoff, A\. M\. Rush, B\. Barak, T\. Le Scao, A\. Piktus, N\. Tazi, S\. Pyysalo, T\. Wolf, and C\. Raffel\(2023\)Scaling data\-constrained language models\.InAdvances in Neural Information Processing Systems,Vol\.36\.External Links:[Link](https://arxiv.org/abs/2305.16264)Cited by:[3rd item](https://arxiv.org/html/2610.33016#Sx1.I1.i3.p1.1)\.
- \[38\]A\. Nag, S\. Chakrabarti, A\. Mukherjee, and N\. Ganguly\(2025\)Efficient continual pre\-training of LLMs for low\-resource languages\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 3: Industry Track\),Albuquerque, New Mexico,pp\. 304–317\.External Links:[Link](https://aclanthology.org/2025.naacl-industry.25/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-industry.25),ISBN 979\-8\-89176\-194\-0Cited by:[Appendix I](https://arxiv.org/html/2610.33016#A9.p1.1),[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px3.p1.1)\.
- \[39\]S\. K\. Nahin, R\. N\. Nandi, S\. Sarker, Q\. S\. Muhtaseem, M\. Kowsher, A\. C\. Shill, M\. Ibrahim, M\. H\. Menon, T\. A\. Muntasir, and F\. Alam\(2025\)TituLLMs: a family of Bangla LLMs with comprehensive benchmarking\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 24922–24940\.External Links:[Link](https://aclanthology.org/2025.findings-acl.1279/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.1279),ISBN 979\-8\-89176\-256\-5Cited by:[Appendix I](https://arxiv.org/html/2610.33016#A9.p1.1),[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px3.p2.1)\.
- \[40\]NLLB Team, M\. R\. Costa\-jussà, J\. Cross, O\. Çelebi, M\. Elbayad, K\. Heafield,et al\.\(2022\)No language left behind: scaling human\-centered machine translation\.arXiv preprint arXiv:2207\.04672\.External Links:[Link](https://arxiv.org/abs/2207.04672)Cited by:[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px1.p1.1)\.
- \[41\]OpenAI\(2019\)GPT\-2 source code:src/encoder\.py\.Note:GitHub repository openai/gpt\-2External Links:[Link](https://github.com/openai/gpt-2/blob/master/src/encoder.py)Cited by:[§I](https://arxiv.org/html/2610.33016#S1.p2.1)\.
- \[42\]G\. Penedo, H\. Kydlíček, L\. Ben Allal, A\. Lozhkov, M\. Mitchell, C\. Raffel, L\. Von Werra, and T\. Wolf\(2024\)The FineWeb datasets: decanting the web for the finest text data at scale\.InAdvances in Neural Information Processing Systems \(Datasets and Benchmarks Track\),External Links:[Link](https://arxiv.org/abs/2406.17557)Cited by:[§VI](https://arxiv.org/html/2610.33016#S6.SS0.SSS0.Px1.p1.1)\.
- \[43\]G\. Penedo, H\. Kydlíček, V\. Sabolčec, B\. Messmer, N\. Foroutan, A\. H\. Kargaran, C\. Raffel, M\. Jaggi, L\. Von Werra, and T\. Wolf\(2025\)FineWeb2: one pipeline to scale them all – adapting pre\-training data processing to every language\.InSecond Conference on Language Modeling \(COLM\),External Links:2506\.20920,[Link](https://arxiv.org/abs/2506.20920)Cited by:[§VI](https://arxiv.org/html/2610.33016#S6.SS0.SSS0.Px1.p1.1)\.
- \[44\]A\. Petrov, E\. La Malfa, P\. H\. S\. Torr, and A\. Bibi\(2023\)Language model tokenizers introduce unfairness between languages\.InAdvances in Neural Information Processing Systems 36 \(NeurIPS 2023\),External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/74bb24dca8334adce292883b4b651eda-Abstract-Conference.html),2305\.15425Cited by:[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px1.p1.1)\.
- \[45\]T\. Purason, P\. Chizhov, I\. P\. Yamshchikov, and M\. Fishel\(2026\)Teaching old tokenizers new words: efficient tokenizer adaptation for pretrained models\.InFindings of the Association for Computational Linguistics: EACL 2026,V\. Demberg, K\. Inui, and L\. Marquez \(Eds\.\),Rabat, Morocco,pp\. 6492–6516\.External Links:[Link](https://aclanthology.org/2026.findings-eacl.341/),[Document](https://dx.doi.org/10.18653/v1/2026.findings-eacl.341),ISBN 979\-8\-89176\-386\-9Cited by:[Appendix I](https://arxiv.org/html/2610.33016#A9.p1.1),[§I](https://arxiv.org/html/2610.33016#S1.p3.1),[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px3.p1.1),[§V](https://arxiv.org/html/2610.33016#S5.SS0.SSS0.Px2.p1.1),[§V](https://arxiv.org/html/2610.33016#S5.SS0.SSS0.Px6.p1.1)\.
- \[46\]Qwen Team\(2026\)Qwen3\.5\-9B tokenizer configuration \(tokenizer\.json\)\.Note:[https://huggingface\.co/Qwen/Qwen3\.5\-9B](https://huggingface.co/Qwen/Qwen3.5-9B)Pre\-tokenizer pattern read from the released fileCited by:[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px2.p1.1),[§IV](https://arxiv.org/html/2610.33016#S4.p3.1),[§V](https://arxiv.org/html/2610.33016#S5.SS0.SSS0.Px1.p1.1)\.
- \[47\]A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, and I\. Sutskever\(2019\)Language models are unsupervised multitask learners\.Technical reportOpenAI\.External Links:[Link](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)Cited by:[§III](https://arxiv.org/html/2610.33016#S3.p1.1)\.
- \[48\]S\. Rana, A\. Kulkarni, A\. Menezes, C\. Khatri, and S\. Agarwal\(2026\)MUTANT: a recipe for multilingual tokenizer design\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),San Diego, California, United States,pp\. 46261–46277\.External Links:[Link](https://aclanthology.org/2026.acl-long.2146/),[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.2146),ISBN 979\-8\-89176\-390\-6Cited by:[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px2.p1.1),[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px3.p1.1)\.
- \[49\]S\. Regmi, S\. Pudasaini, and C\. P\. Pun\(2026\)Arkios: an open bilingual English\-Nepali language model trained from scratch, with a Devanagari\-aware tokenizer\.External Links:2608\.30092,[Link](https://arxiv.org/abs/2608.30092)Cited by:[TABLE III](https://arxiv.org/html/2610.33016#A6.T3),[§IV](https://arxiv.org/html/2610.33016#S4.p2.1)\.
- \[50\]S\. Regmi, S\. Pudasaini, and C\. P\. Pun\(2026\)Vowel signs are not letters: a pre\-tokenization ceiling on multilingual tokenizer fertility\.External Links:2608\.26449,[Link](https://arxiv.org/abs/2608.26449)Cited by:[item 1](https://arxiv.org/html/2610.33016#A1.I1.i1.p1.1),[§I](https://arxiv.org/html/2610.33016#S1.p2.1),[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px2.p1.1),[§III](https://arxiv.org/html/2610.33016#S3.p1.1),[§VIII](https://arxiv.org/html/2610.33016#S8.SS0.SSS0.Px2.p1.1)\.
- \[51\]F\. Remy, P\. Delobelle, H\. Avetisyan, A\. Khabibullina, M\. de Lhoneux, and T\. Demeester\(2024\)Trans\-tokenization and cross\-lingual vocabulary transfers: language adaptation of LLMs for low\-resource NLP\.InFirst Conference on Language Modeling \(COLM\),External Links:[Link](https://arxiv.org/abs/2408.04303)Cited by:[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px3.p1.1)\.
- \[52\]P\. Rust, J\. Pfeiffer, I\. Vulić, S\. Ruder, and I\. Gurevych\(2021\)How good is your tokenizer? on the monolingual performance of multilingual language models\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),C\. Zong, F\. Xia, W\. Li, and R\. Navigli \(Eds\.\),Online,pp\. 3118–3135\.External Links:[Link](https://aclanthology.org/2021.acl-long.243/),[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.243)Cited by:[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px1.p1.1)\.
- \[53\]C\. W\. Schmidt, V\. Reddy, H\. Zhang, A\. Alameddine, O\. Uzan, Y\. Pinter, and C\. Tanner\(2024\)Tokenization is more than compression\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 678–702\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.40/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.40)Cited by:[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px1.p1.1)\.
- \[54\]R\. Sennrich, B\. Haddow, and A\. Birch\(2016\)Neural machine translation of rare words with subword units\.InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),K\. Erk and N\. A\. Smith \(Eds\.\),Berlin, Germany,pp\. 1715–1725\.External Links:[Link](https://aclanthology.org/P16-1162/),[Document](https://dx.doi.org/10.18653/v1/P16-1162)Cited by:[§III](https://arxiv.org/html/2610.33016#S3.p1.1)\.
- \[55\]K\. Shrestha and N\. Pradhan\(2026\)Preserving morphemes: morphology\-guided pre\-tokenization for Nepali\.External Links:2609\.33395,[Link](https://arxiv.org/abs/2609.33395)Cited by:[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px2.p1.1),[§VIII](https://arxiv.org/html/2610.33016#S8.SS0.SSS0.Px2.p1.1)\.
- \[56\]J\. T\. H\. Smith, T\. Dakhran, A\. Cabrera, S\. S\. Lee, P\. Pak, A\. Tadimeti, T\. Seyde, M\. Labonne, A\. Amini, and M\. Lechner\(2026\)In\-place tokenizer expansion for pre\-trained LLMs\.External Links:2607\.15232Cited by:[TABLE VI](https://arxiv.org/html/2610.33016#A9.T6),[§I](https://arxiv.org/html/2610.33016#S1.p3.1),[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px3.p1.1),[§III](https://arxiv.org/html/2610.33016#S3.SS0.SSS0.Px1.p1.1)\.
- \[57\]P\. Srivastava\(2026\)The tokenizer tax: quantifying and explaining the cross\-lingual cost of subword tokenization for Indian languages\.External Links:2607\.24276,[Link](https://arxiv.org/abs/2607.24276)Cited by:[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px1.p1.1)\.
- \[58\]A\. Tejaswi, N\. Gupta, and E\. Choi\(2024\)Exploring design choices for building language\-specific LLMs\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 10485–10500\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.614/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.614)Cited by:[§I](https://arxiv.org/html/2610.33016#S1.p3.1),[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px3.p1.1)\.
- \[59\]A\. Thakur, A\. Nagpal, A\. Savarkar, K\. Pundalik, S\. Dosi, P\. Sawarkar, V\. Thakur, R\. Saluja, M\. S\. Desarkar, and G\. Ramakrishnan\(2025\)The art of breaking words: rethinking multilingual tokenizer design\.External Links:2508\.06533,[Link](https://arxiv.org/abs/2508.06533)Cited by:[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px2.p1.1)\.
- \[60\]M\. Velayuthan and K\. Sarveswaran\(2025\)Egalitarian language representation in language models: it all begins with tokenizers\.InProceedings of the 31st International Conference on Computational Linguistics,O\. Rambow, L\. Wanner, M\. Apidianaki, H\. Al\-Khalifa, B\. D\. Eugenio, and S\. Schockaert \(Eds\.\),Abu Dhabi, UAE,pp\. 5987–5996\.External Links:[Link](https://aclanthology.org/2025.coling-main.400/)Cited by:[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px2.p1.1)\.
- \[61\]A\. Yamaguchi, T\. Morishita, A\. Villavicencio, and N\. Aletras\(2025\)Adapting chat language models using only target unlabeled language data\.Transactions on Machine Learning Research\.External Links:2412\.11704,[Link](https://mlanthology.org/tmlr/2025/yamaguchi2025tmlr-adapting/)Cited by:[TABLE VI](https://arxiv.org/html/2610.33016#A9.T6),[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px3.p1.1),[§III](https://arxiv.org/html/2610.33016#S3.SS0.SSS0.Px1.p1.1)\.
- \[62\]A\. Yamaguchi, A\. Villavicencio, and N\. Aletras\(2026\)How can we effectively expand the vocabulary of LLMs with 0\.01GB of target language text?\.Computational Linguistics52\(1\),pp\. 295–330\.External Links:[Document](https://dx.doi.org/10.1162/coli.a.581),2406\.11477Cited by:[TABLE VI](https://arxiv.org/html/2610.33016#A9.T6),[§I](https://arxiv.org/html/2610.33016#S1.p3.1),[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px3.p1.1),[§III](https://arxiv.org/html/2610.33016#S3.SS0.SSS0.Px1.p1.1),[§V](https://arxiv.org/html/2610.33016#S5.SS0.SSS0.Px6.p1.1)\.
- \[63\]A\. Yanget al\.\(2025\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§VI](https://arxiv.org/html/2610.33016#S6.SS0.SSS0.Px1.p1.1)\.
- \[64\]J\. Zhao, Z\. Zhang, L\. Gao, Q\. Zhang, T\. Gui, and X\. Huang\(2024\)LLaMA beyond English: an empirical study on language capability transfer\.External Links:2401\.01055,[Link](https://arxiv.org/abs/2401.01055)Cited by:[§VIII](https://arxiv.org/html/2610.33016#S8.SS0.SSS0.Px1.p1.1)\.
- \[65\]V\. Zouhar, C\. Meister, J\. Gastaldi, L\. Du, M\. Sachan, and R\. Cotterell\(2023\)Tokenization and the noiseless channel\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 5184–5207\.External Links:[Link](https://aclanthology.org/2023.acl-long.284/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.284)Cited by:[§II](https://arxiv.org/html/2610.33016#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix APre\-registration history

We committed the analysis plan to the project repository in nine steps between 2026\-10\-04 and 2026\-10\-06 \(IST\), and report all nine\. Times are commit times\.

1. 1\.Original registration\(7713fd2, 17:59\) registered hypotheses about the mechanism, a regex\-only tokenizer ablation, a cross\-script dose–response and training from scratch\. Our literature check then found that[Regmi et al\. \[50\]](https://arxiv.org/html/2610.33016#bib.bib13)had published the mechanism, the ablation, the cross\-script analysis and a from\-scratch comparison five weeks earlier\. We withdrew those hypotheses as contributions\.
2. 2\.Retrofit registration\(5430e86, 18:16\), before any model was trained or evaluated, registered the study reported here: armsR0,R1,R2and the untrained base model;K=32,000K\{=\}32\{,\}000unlessR2saturated earlier on FLORES dev; equal\-bytes continued pretraining of Llama\-3\.2\-1B and Qwen3\-1\.7B\-Base on a fixed byte budget split equally between Nepali and English \(3 GB each in our plan, a figure first written down in the first amendment\); a learning rate selected once per model onR0; two seeds\. Its hypotheses were P1 \(compression ordering, withR1above the letters\-only floor\), P2, the main hypothesis \(held\-out Nepali BPB ofR2at most 0\.01 above bothR1andR0, while using fewer training tokens\), P3 \(English BPB ofR2andR0within 0\.01\) and P4 \(R2decodes Nepali faster at equal bytes\)\. P1 had already been measured when it was written, so we report it as a measurement\.
3. 3\.First amendment\(aa8779a, 19:50; filePREREG\_AMENDMENT\.md, which numbers the amendments from 2 because it counts the retrofit registration as the first\), written after an internal review and before any continued\-pretraining run had produced an evaluation, recorded the deviations below, fixed four pipeline bugs, and fixed the primary analysis: byte\-matched BPB, a paired bootstrap over pieces and three claim labels \(better, non\-inferior, no detectable difference; §[VI](https://arxiv.org/html/2610.33016#S6)gives the later ordered set\)\. Earlier launches had been stopped by us because of the bugs, and none produced an evaluated checkpoint\. The only earlier evaluation, of the untouched base Llama\-3\.2\-1B with a since\-fixed prefix bug, is discarded\.
4. 4\.Second amendment\(9be7667, 21:46;PREREG\_AMENDMENT3\.md\) was written*after*observing the LlamaR0andR2main\-run results and before anyR1or Qwen result\. It registered the warm\-up runs, with the prediction that they shrink theR2−\-R0gap, and 3 GB runs that we did not run, and it added the “worse” category to the claim rules\.
5. 5\.Third amendment\(660fd50, 2026\-10\-05, 03:24;PREREG\_AMENDMENT4\.md\) was written after all main and warm\-up results and before running what it registered: the three Llama main runs repeated with every training document wrapped as BOS, document, end\-of\-text, scored with the registered protocol\. It predicted that scores after BOS and after the end\-of\-text token would agree within 0\.005 for every arm, and thatR2−\-R1after BOS would lie between the two earlier values \(\+0\.006\+0\.006and\+0\.027\+0\.027\)\. The second prediction held \(\+0\.026\+0\.026\), and the rerun matches the earlier separator diagnostic within 0\.002, which confirms that diagnostic in spirit\. The first prediction failed: after the end\-of\-text token the rerun models score far worse \(Appendix[K](https://arxiv.org/html/2610.33016#A11)\)\.
6. 6\.Fourth amendment\(9fb12e7, 2026\-10\-05, 16:08;PREREG\_AMENDMENT5\.md\) was written after all earlier results and two external reviews, and before running what it registered:R2retrained forR1’s step count on Llama \(BOS setting\) and Qwen, QwenR1andR2with a second seed, and two\-tokenization scoring\. It predicted that matched\-stepsR2would be non\-inferior toR1on Qwen \(it was better\), smaller\-gapped but still worse on Llama \(held\), still worse thanR0on both \(held\), that a second Qwen seed would moveR2−\-R1by less than 0\.005 \(it moved by 0\.0057, so this failed narrowly\), and that two\-tokenization scoring would change no verdict \(held\)\.
7. 7\.Fifth amendment\(c8bc301, 2026\-10\-06, 00:48;PREREG\_AMENDMENT6\.md\), written after the fourth amendment’s results and before running, added second seeds for matched\-stepsR2\(both models\), LlamaR0andR1and QwenR0, and QwenR1and matched\-stepsR2at a learning rate of3×10−43\{\\times\}10^\{\-4\}, under a budget cap\. It predicted that with seeds averaged matched\-stepsR2would be better thanR1on Qwen and worse on Llama, thatR0would stay best, and thatR2−\-R1would keep its sign at3×10−43\{\\times\}10^\{\-4\}\. All four held\.
8. 8\.Sixth amendment\(4bc56cc, 2026\-10\-06, 05:01;PREREG\_AMENDMENT7\.md\), written after the fifth amendment’s results and before running, added QwenR0at3×10−43\{\\times\}10^\{\-4\}\. It predicted thatR0would gain less than 0\.005 and that matched\-stepsR2would be non\-inferior to it\. Both failed:R0gained0\.0210\.021and stays ahead by0\.0240\.024\.
9. 9\.Seventh amendment\(6dddbeb, 2026\-10\-06, 07:30;PREREG\_AMENDMENT8\.md\), written after an external review and before running, added a second seed for equal\-bytes LlamaR2\. It predicted that the two\-seedR2−\-R1would stay worse and within 0\.005 of\+0\.0256\+0\.0256; it held \(\+0\.0252\+0\.0252\[\+0\.0241,\+0\.0263\]\[\+0\.0241,\+0\.0263\]\)\.

Two departures from these amendments: the fourth registered two\-tokenization scoring also for its new runs, but we applied it only to the six seed\-0 equal\-bytes models of the Llama rerun and Qwen; and the first amendment stated that we would not tune the learning rate on results, whereas the fifth and sixth test a second rate after seeing results\. We use that rate only as a robustness check, not to select one\.

In the repository these arePREREG\.md,PREREG\_RETROFIT\.md,PREREG\_AMENDMENT\.md, andPREREG\_AMENDMENT3\.mdtoPREREG\_AMENDMENT8\.md\. The first four commits lie within four hours, the fifth about six hours later, and the last three over the following day\. All are in a repository we control, so they document our plan without an external timestamp\. Apart from the author line ofPREREG\.md, no plan file was edited after its commit\.

#### Outcome of each registered hypothesis\.

P1 holds as a measurement \(§[VII\-A](https://arxiv.org/html/2610.33016#S7.SS1)\)\. P2 fails againstR0on both models\. AgainstR1it fails for Llama in the BOS\-consistent rerun \(R2is worse by0\.0260\.026\), and it also fails for Qwen \(R2is worse by0\.0090\.009,0\.0120\.012over two seeds\); it held for Llama only in the main runs under the registered prefix, a result of the BOS artifact \(§[VII\-B](https://arxiv.org/html/2610.33016#S7.SS2)\)\. P3 holds\. P4, faster generation in bytes per second at equal generated bytes, was not reliably evaluated: our timings ran on shared GPUs with a fixed number of generated tokens \(Appendix[K](https://arxiv.org/html/2610.33016#A11)\)\. We report token counts as a separate result\. The warm\-up prediction holds after the training separator; under the registered prefix the warm\-up scores are dominated by the BOS artifact\.

#### Deviations from the retrofit registration\.

The first amendment recorded these before any result:

- •Qwen3\-0\.6B\-Base replaces Qwen3\-1\.7B\-Base, for budget and time\. Their tokenizers are identical\.
- •1 GB of Nepali and 1 GB of English per arm replace 3 GB each, with identical documents in every arm\.
- •A fixed learning rate of10−410^\{\-4\}replaces selection onR0from\{3×10−5,10−4,3×10−4\}\\\{3\{\\times\}10^\{\-5\},10^\{\-4\},3\{\\times\}10^\{\-4\}\\\}by a 200\-step pilot, which was never completed \(the candidate set and pilot were first recorded in the first amendment\)\.
- •One seed per arm replaces two\.
- •Added:R0andR1checkpoints atR2’s final step \(equal\-compute view\); byte\-matched BPB as the primary metric, with the registered sliding\-window BPB as secondary; the paired bootstrap with claim rules as the primary test; a translation cap of 512 instead of 256 new tokens\.

Besides the runs added by the second to seventh amendments, we made four further changes after seeing results, all reported as such: intervals from a bootstrap over reconstructed documents instead of pieces \(§[VI](https://arxiv.org/html/2610.33016#S6)\); the training\-separator diagnostic; decoding steps instead of measured bytes per second for P4, whose implementation had fixed the number of generated tokens rather than bytes; and the “worse” label from the second amendment for the main runs\.

#### Pipeline bugs fixed before any reported run\.

\(1\) Documents had been stored one per line with their internal newlines, so “documents” were paragraphs; they are now stored as one JSON\-encoded document per line\. \(2\) New\-token embeddings had been initialised by decoding tokens to text, which turns partial UTF\-8 tokens into U\+FFFD; constituents now come from applying the original merges to the byte\-level string\. \(3\) The initialisation check had covered Qwen’s padding rows; it now covers only the original vocabulary\. \(4\) Evaluation had prefixed documents with the end\-of\-text token, while base Llama expects BOS; the registered evaluation uses BOS where it exists\.

## Appendix BRepair details

#### Regex edit\.

For the Llama\-3 and Qwen pattern we make two string substitutions\. The first replaces\\p\{L\}\+by\[\\p\{L\}\\p\{M\}\]\+in the word branch\. The second inserts\\p\{M\}after\\p\{L\}in the two negated classes, the optional leading character of the word branch and the punctuation branch\. Three character classes change:

\[^\\r\\n\\p\{L\}\\p\{N\}\]?\\p\{L\}\+ ↓\\downarrow \[^\\r\\n\\p\{L\}\\p\{M\}\\p\{N\}\]?\[\\p\{L\}\\p\{M\}\]\+ ␣?\[^\\s\\p\{L\}\\p\{N\}\]\+→\\;\\rightarrow\\;␣?\[^\\s\\p\{L\}\\p\{M\}\\p\{N\}\]\+

Nothing else in the pattern changes\.

#### Devanagari\-only variant\.

The variant replaces\\p\{M\}in the three classes by an explicit class of the 34 combining marks of the Devanagari block \(U\+0900–0903, U\+093A–093C, U\+093E–094F, U\+0951–0957, U\+0962–0963\)\. We checked that this class equals the set of\\p\{M\}code points in the block under Python’sunicodedata\(Unicode 15\.0\), theregexmodule and the Hugging Facetokenizersengine\. Every other character has the same class membership as in the original regex, so any text without a Devanagari mark is pre\-tokenized exactly as before\. Across the 204 FLORES\-200 devtest files, the general repair changes pre\-tokens in 61 languages for both models and the Devanagari\-only variant in exactly the ten Devanagari languages\. On FLORES Nepali the two variants give identical pre\-tokens and identicalR2token ids\. On the FineWeb\-2 Nepali test split \(275 MB, 49,462 documents, one per line\), 24 lines differ, in 47 segments; the marks involved are emoji variation selectors, Kannada, Bengali, Thai and Hebrew signs and one combining candrabindu in romanised text, and theR2token count of those lines is the same under both\.

#### Continued BPE\.

At inference BPE applies the lowest\-ranked applicable merge first, so the original merges act before the new ones\. The extended tokenizer reproduces the training procedure except in two cases: a new merge whose output string is already in the original vocabulary can let an original merge fire again afterwards, and under Hugging Face’signore\_mergesoption \(set for Llama\-3\) a pre\-token that is itself a vocabulary entry is emitted directly\. Neither case affects Proposition[1](https://arxiv.org/html/2610.33016#Thmproposition1)\. We run merge learning inside the Hugging Face Rust trainer by mapping each base token to a private\-use code point; learningK=32,000K\{=\}32\{,\}000merges on 1 GB of Nepali takes 1\.7 to 3\.4 minutes per tokenizer\.

#### Scope of Proposition[1](https://arxiv.org/html/2610.33016#Thmproposition1)\.

The proposition covers ASCII English and code and every script whose normalised text has no combining marks, including most Cyrillic, Han, Hangul and Ethiopic text\. It does not cover Devanagari, which changes by design, or text with marks, such as Hindi, Arabic with vowel points, Thai, or Latin written in decomposed form \(NFD\)\. The Llama\-3 tokenizer has no normalizer, so deployments should apply NFC first; Qwen3’s tokenizer applies NFC itself\. It also assumes that the regex engine’s\\p\{M\}table matches the one used to definess; for the Devanagari\-only variant we checked this directly\. Empirically,R1andR2atK=32,000K\{=\}32\{,\}000give token ids identical to the original tokenizer on all 1,012 FLORES devtest sentences in English, Russian and Amharic, for both models, including one Russian sentence with a combining acute that lies outside the proposition\.

## Appendix CProof of Proposition[1](https://arxiv.org/html/2610.33016#Thmproposition1)

###### Proof\.

*Pre\-tokens\.*The repaired regex differs from the original only in the three classes of Appendix[B](https://arxiv.org/html/2610.33016#A2), and each differs from its original only on\\p\{M\}characters\. A regex match onssconsults only characters ofss, and the order of the alternatives is unchanged\. Sincesshas no\\p\{M\}character, both regexes produce the same pre\-tokens\.*Merges and vocabulary\.*R2differs from the original tokenizer only by appended merges and the vocabulary entries they create, each of which contains a complete code point in U\+0900–U\+097F\. A merge\(a,b\)\(a,b\)applies only whereaaandbbare adjacent symbols of a pre\-token, so its output is a contiguous substring of that pre\-token’s bytes\. A vocabulary entry is emitted directly underignore\_mergesonly if it equals the whole pre\-token\. Neither can happen for a pre\-token ofss, because UTF\-8 is self\-synchronising andsscontains no such code point\. Each pre\-token ofssis therefore encoded with original merges and original vocabulary entries only, exactly as by the original tokenizer\.*Ids\.*New ids are appended above the original ids, so all ids coincide\. ∎

## Appendix DTraining and evaluation protocol

#### Data\.

We normalise to NFC, drop documents under 200 characters, remove exact duplicates, and drop every document that shares a whitespace 10\-gram with FLORES\-200 dev or devtest in Nepali or English\. This also covers the Belebele passages, which are FLORES passages\[[6](https://arxiv.org/html/2610.33016#bib.bib30)\], but not its questions or answers\. Documents are stored one JSON\-encoded document per line\. A 30 MB held\-out slice of each language is carved from the cleaned stream before training and never trained on\. The 1 GB of Nepali used to learn merges is taken from the start of the training stream\.

#### Continued pretraining\.

Documents are interleaved and separated by the model’s end\-of\-text token; in the main and warm\-up runs no BOS token is inserted, and in the Llama BOS\-consistent rerun every document is wrapped as BOS, document, end\-of\-text\. Optimisation is identical across arms: sequence length 2,048, global batch 256 sequences \(0\.52M tokens\), AdamW \(β=\(0\.9,0\.95\)\\beta=\(0\.9,0\.95\), weight decay 0\.1\), gradient clipping at 1\.0, 1% linear warm\-up, then cosine decay to 10% of peak, bf16 autocast with fp32 master weights, on 4×\\timesH200\. Initialisation is deterministic and there is no dropout, so a second seed would change only the order of training windows and bf16 non\-determinism\. In the warm\-up runs, stage 1 trains the whole tied embedding matrix, old rows included, at10−310^\{\-3\}on the first 10% of the stream; stage 2 is a main run started from the stage\-1 model\.

#### Evaluation\.

Every scored sequence is preceded by one context token \(§[VI](https://arxiv.org/html/2610.33016#S6)\), identically for every arm of a model\.

- •BPBon the held\-out native slices \(the first 4 MB of each\) and on FLORES\-200 devtest\. In the primary byte\-matched version, each held\-out document is split at whitespace into pieces of at most 2,000 bytes and every piece is scored independently, so every arm conditions on the same bytes\. Token windows would giveR2more bytes of context per scored token\. As a secondary metric, whole documents are scored with a sliding window of 2,048 tokens and stride 1,024\.
- •Belebele\[[6](https://arxiv.org/html/2610.33016#bib.bib30)\]npi\_Devaandeng\_Latn, zero\-shot, scoring each answer string by its log\-likelihood per byte given passage and question\.
- •Translation: FLORES\-200 devtest Nepali↔\\leftrightarrowEnglish, 5\-shot from FLORES dev, greedy decoding capped at 512 new tokens, first output line, chrF\+\+\.
- •Generation speed: UTF\-8 bytes generated per second in greedy continuation of FLORES dev Nepali sentences \(batch 64, 128 new tokens\)\.

Document ids were not stored with the pieces\. The Nepali held\-out slice holds 503 documents \(the first 503 of FineWeb\-2npi\_Devaafter cleaning;[results/revision/heldout\_ne\_doc\_ids\.json](https://results/revision/heldout_ne_doc_ids.json)\), recovered after an external review, and all Nepali intervals resample these documents\. An earlier version of this paper used a length heuristic \(a piece shorter than 1,900 bytes ends a document, 484 clusters\); switching moved interval endpoints by at most0\.00030\.0003\. English intervals still use that heuristic\. The margin of 0\.01 is 1\.3% of the base Llama model’s Nepali BPB and 1\.2% of base Qwen’s\. The equal\-compute checkpoints are scored on BPB only, on the first 1 MB of each held\-out slice \(567 Nepali pieces from 114 documents\)\.

## Appendix EData

Table[II](https://arxiv.org/html/2610.33016#A5.T2)gives the corpus cleaning and the token count of each arm’s training stream\.

TABLE II:Corpus cleaning \(FineWeb\-2npi\_Deva; FineWeb\-Edu; documents placed in the 30 MB held\-out slice are not counted as training documents\) and the token counts of each arm’s identical 2 GB training stream\. English token counts differ across arms of a model by less than 0\.012%, from the few English documents that contain Devanagari or combining marks\.FineWeb\-2 is already deduplicated and filtered for length, which is why our own filters remove almost nothing\.

## Appendix FFull audit

Table[III](https://arxiv.org/html/2610.33016#A6.T3)lists every distinct tokenizer\. For each of the nine aliases we compared token ids with its partner on all 1,012 devtest sentences of English, Nepali, Hindi, Chinese and Arabic; all were identical\. Kimi\-K3 \(requires executing remote code\) and Mistral\-Large\-3 \(its transformers load does not round\-trip; itstokenizer\.jsonuses Mistral\-NeMo’s mark\-aware regex\) are not included\. The four tokenizers that do not round\-trip belong to encoders and translation models; we keep them as reference points\.

TABLE III:All 24 distinct tokenizers \(33 released tokenizers\) on FLORES\-200 devtest\. Aliases: tokenizers whose token ids are identical on every devtest sentence of five languages \(Appendix[F](https://arxiv.org/html/2610.33016#A6)\)\. Regex: letters\-only, mark\-aware, other, or none \(no regex pre\-tokenizer\)\. ne/en: Nepali/English token premium with 95% paired\-bootstrap interval\. Floor: pre\-token floor under the tokenizer’s own regex, a lower bound on the premium of any merge\-based extension \(letters\-only and mark\-aware tokenizers only\)\. Mark\-init\.: share of Nepali tokens whose first character is a combining mark\. Lossless: decoding reproduces the input on 200 Nepali sentences\. Arkios is from[Regmi et al\. \[49\]](https://arxiv.org/html/2610.33016#bib.bib14)\. Hugging Face repositories are listed in the released code\.#### Broken syllables\.

The share of Nepali tokens that begin with a combining mark is 55% for Llama\-3, 32% for Qwen3 and 30% forcl100k\. Mark\-aware tokenizers show similar rates \(26% foro200k, 55% for TituLLM, which keeps Llama\-3’s merges\), because BPE also splits within words\.

## Appendix GPremium decomposition

We write the premium as

\|T⁡\(ne\)\|\|T⁡\(en\)\|=\|Prep​\(ne\)\|\|T⁡\(en\)\|⏟base⋅\|P⁡\(ne\)\|\|Prep​\(ne\)\|⏟regex factor⋅\|T⁡\(ne\)\|\|P⁡\(ne\)\|⏟vocabulary factor,\\frac\{\|T\(\\text\{ne\}\)\|\}\{\|T\(\\text\{en\}\)\|\}=\\underbrace\{\\frac\{\|P\_\{\\text\{rep\}\}\(\\text\{ne\}\)\|\}\{\|T\(\\text\{en\}\)\|\}\}\_\{\\text\{base\}\}\\cdot\\underbrace\{\\frac\{\|P\(\\text\{ne\}\)\|\}\{\|P\_\{\\text\{rep\}\}\(\\text\{ne\}\)\|\}\}\_\{\\text\{regex factor\}\}\\cdot\\underbrace\{\\frac\{\|T\(\\text\{ne\}\)\|\}\{\|P\(\\text\{ne\}\)\|\}\}\_\{\\text\{vocabulary factor\}\},wherePPis the tokenizer’s own pre\-tokenizer andPrepP\_\{\\text\{rep\}\}is the repaired Llama\-3 pattern, applied to every tokenizer so that the numerator of the base term is the same for all\. The base lies between0\.770\.77and0\.800\.80because English token counts barely differ\. Using one pattern means that Qwen’s regex factor also includes its single\-digit splitting\. The regex factor is0\.990\.99–1\.021\.02for mark\-aware tokenizers,2\.772\.77–2\.792\.79for the Llama\-3, Qwen andcl100kpattern,4\.064\.06for GPT\-2, whose pattern puts every run of marks in a pre\-token of its own, and5\.015\.01for Falcon3\. Falcon3’s pipeline splits digits after its byte\-level step, and its byte\-level alphabet writes byte0xBEas ¾, which Unicode classes as a number, so it cuts the vowel signU\+093Einside its UTF\-8 encoding\. All floors and factors here count pre\-tokens of each tokenizer’s full pre\-tokenizer pipeline; counting its last regex alone, as our audit script first did, understates Falcon3’s floor \(3\.213\.21instead of3\.963\.96\)\. The decomposition is an accounting identity and does not attribute cause\. The regex factor is the room that a better regex creates, which new merges must then fill\.

Table[IV](https://arxiv.org/html/2610.33016#A7.T4)shows the factors\. For Llama\-3 the vocabulary factor is1\.181\.18, the lowest in the audit: its 128k vocabulary already nearly covers Nepali pre\-tokens\. For Qwen andcl100kboth factors matter\. Qwen3\.5’s word branch is our repaired class with only the leading\-character class left unchanged;[Land \[28\]](https://arxiv.org/html/2610.33016#bib.bib16)reported its change for Hindi\.

TABLE IV:Premium = base×\\timesregex factor×\\timesvocabulary factor, on FLORES\-200 devtest, for all 15 letters\-only and mark\-aware tokenizers in the audit \(top: letters\-only; bottom: mark\-aware; “\(\+nn\)” counts aliases\)\. Merge\-based extension can only lower the last column, and not below 1\.
## Appendix HCross\-script sweep

TABLE V:Language/English premium on FLORES\-200 devtest,K=32,000K\{=\}32\{,\}000\. Marks: share of NFC code points that are combining marks; Amharic and Korean have none \(controls\)\.†Thai is written without spaces, so its pre\-tokens are long andR1is not yet at its floor\. ForR1, merge learning stopped more than 100 merges short of 32,000 for Tamil, Telugu, Sinhala, Burmese and Hindi \(Llama\) \(22,325–31,815 merges\), because no further merge was available under the letters\-only regex; Korean and Thai on Llama end at most 7 merges short\.For each language we used the FineWeb\-2*test*split, normalised to NFC, with documents that share a whitespace 10\-gram with that language’s FLORES\-200 dev or devtest removed \(for Thai we also checked 50\-character overlaps and found none\)\. The merge\-learning text ranges from 25 MB \(Amharic\) to 361 MB \(Bengali\), less than the 1 GB used for Nepali, so for languages whereR1ran out of merges we cannot fully separate the floor from data scarcity; Sinhala and AmharicR2may also be limited by data\.

#### A relaxed merge filter\.

The Nepali filter keeps a new merge only if its output contains a complete Devanagari character\. Scripts that the base vocabularies cover poorly have characters split into byte pieces, and this filter would drop the first byte\-level merge of each such character and every merge built on it \(for SinhalaR2on Llama\-3, only 13,530 of 32,000 merges survived\)\. For this sweep we therefore also keep merges whose output is a byte piece of a target\-script character\. Such merges can fire on other scripts that share those bytes: the lead byte 0xE0 begins every code point from U\+0800 to U\+0FFF, which covers all Indic scripts and Thai, and a continuation\-byte suffix can end code points in other blocks\. They cannot fire on ASCII text, which contains no such bytes, and we verified English token ids on all 1,012 sentences for every tokenizer\. Proposition[1](https://arxiv.org/html/2610.33016#Thmproposition1)does not cover these tokenizers, and we did not measure their effect on other non\-ASCII scripts\. Hindi shares the Nepali script, and Nepali merges also change Hindi tokenization\.

#### Thai and Burmese\.

Both are written without spaces between words, so a repaired pre\-token can span several words\. In practice noR2Thai\- or Burmese\-script token spans a space in either language\. In Thai, 28% \(Llama\-3\) and 25% \(Qwen3\) ofR2’s Thai\-script tokens cover more than one dictionary word \(PyThaiNLPnewmmsegmentation\), fewer than underR1\(33% and 31%\)\. In Burmese,R2’s tokens average 2\.2 and 1\.8 syllables, against 1\.3 underR1and 5\.1 syllables per space\-delimited chunk\.

## Appendix IReleased extensions

We did not follow a fixed search protocol\. Candidates came from the papers in §[II](https://arxiv.org/html/2610.33016#S2)and from manual searches of Hugging Face for Llama\-3, Qwen and LFM2 derivatives in Indic and Southeast Asian languages\. We kept releases whose tokenizer adds merges to a letters\-only base\. After inspection we excluded five releases that keep the base Llama\-3 tokenizer unchanged and three that do not extend a letters\-only base\. The Llama\-3 Hindi extension of Nanda\[[9](https://arxiv.org/html/2610.33016#bib.bib41)\]is gated and could not be measured, and we found no public tokenizer for[Nag et al\. \[38\]](https://arxiv.org/html/2610.33016#bib.bib43)or[Purason et al\. \[45\]](https://arxiv.org/html/2610.33016#bib.bib28)\. The Yamaguchi et al\. extensions re\-order the base merges and are not English\-invariant \(English tokens rise by 6\.5% for Llama\-3 Telugu and 7\.8% for Qwen3 Telugu\), so Table[VI](https://arxiv.org/html/2610.33016#A9.T6)compares tokens with pre\-tokens of the target text directly\. Two releases do not obey the letters\-only floor: TituLLM\[[39](https://arxiv.org/html/2610.33016#bib.bib29)\]changes the regex, and SinLlama\[[3](https://arxiv.org/html/2610.33016#bib.bib42)\]registers 11,080 strings as added tokens and reaches 0\.54 times its letters\-only floor\.

TABLE VI:Released merge\-based extensions that keep the letters\-only regex, on FLORES\-200 devtest of the target language\. Tokens/floor: extended\-tokenizer tokens divided by pre\-tokens under its own regex \(at least 1 by construction\)\. Gap closed: share of the base\-to\-floor gap that the extension removed\. Room: pre\-tokens under the letters\-only regex divided by pre\-tokens under the repaired regex\. First block:[Yamaguchi et al\. \[62\]](https://arxiv.org/html/2610.33016#bib.bib35); second:[Yamaguchi et al\. \[61\]](https://arxiv.org/html/2610.33016#bib.bib36), with the two Qwen3\-14B rows from a later release by the same authors; third: LFM2\.5\[[56](https://arxiv.org/html/2610.33016#bib.bib39)\]; last: the Amharic release of[Yamaguchi et al\. \[61\]](https://arxiv.org/html/2610.33016#bib.bib36), a script with no combining marks \(control\)\.
## Appendix JAdditional tokenizer arms

#### Regex alone\.

Repairing the regex without adding merges \(R2,K=0K\{=\}0\) gives2\.612\.61for Llama\-3 \(from2\.582\.58\) and4\.424\.42for Qwen3\. Under the repaired regex the old merges still tokenize Nepali words in old pieces; the repair only removes the bound that new merges would otherwise hit\.

#### Added tokens\.

Table[VII](https://arxiv.org/html/2610.33016#A10.T7)gives the added\-token arm of §[III](https://arxiv.org/html/2610.33016#S3)\. Default matching reaches almostR2’s compression and the lowest mark\-initial rate, but splits many words at arbitrary points: a fifth of the added\-token matches start inside a word and more than a quarter end inside one\. With whole\-word matching, few strings ever match and the premium returns close to the original\.

TABLE VII:Added tokens on the unchanged letters\-only regex, FLORES\-200 devtest\. We register the 31,753 ofR2’s 32,000 new strings that decode to whole Devanagari text \(with at most one leading space\) as added tokens\. Premium: Nepali tokens / English tokens\. Mark\-initial: share of Nepali tokens that start with a combining mark\. The bottom rows describe default matching, which ignores word boundaries: the share of tokens that are added tokens, and the share of added\-token matches that start or end inside a word\. No sentence without a Devanagari character changes its token ids \(191,460 FLORES\-200 sentences, both models, both matching modes\)\.
#### Broken syllables\.

A token that starts with a combining mark has certainly split a syllable, but the converse does not hold: a token can also end inside an akshara, for example just before a virama, so these shares are a lower bound on split syllables\. Counting only tokens that contain Devanagari, the share that begin with a combining mark rises from 57% \(Llama\-3\) and 33% \(Qwen3\) to 67% for both underR1, and falls to 15% and 12% underR2\. The letters\-only regex leaves few distinct Nepali pre\-tokens, so 5k to 30k merges cover nearly all frequent ones, which is whyR1saturates early\.

Fig\. 4:Pre\-tokens of*nepālko sarkār*\(“government of Nepal”\) under the letters\-only regex of Llama\-3 and Qwen and under the repaired regex\. We draw a dotted circle where a pre\-token starts with a vowel sign; it is not a duplicated character\.

## Appendix KAll continued\-pretraining runs

held\-out ne BPB↓\\downarrowen BPB↓\\downarrowFLORESBelebelechrF\+\+Gen\.ModelRunsArmTokensreg\.sep\.reg\.ne BPBne acc\.ne→\\toenen→\\toneB/sLlama\-3\.2\-1Bnonebase00\.766n/a0\.7880\.8630\.27732\.410\.85327mainR0417M0\.4740\.4710\.7990\.6110\.28732\.319\.85147R1386M0\.5340\.4830\.8030\.6480\.26827\.210\.46354R2286M0\.5390\.5100\.7990\.6710\.26627\.812\.39398warm\-upR0417M0\.7790\.4710\.8500\.8640\.2798\.89\.15349R1386M0\.8080\.4800\.8540\.9040\.2719\.94\.27521R2286M0\.5270\.5010\.8400\.6520\.28316\.215\.4110781 GB, BOSR0418M0\.4711\.0170\.7990\.607––––in trainingR1386M0\.4830\.8750\.8000\.617––––R2286M0\.5091\.6660\.7980\.626––––Qwen3\-0\.6Bnonebase00\.8520\.8520\.8730\.9450\.29026\.06\.82186mainR0574M0\.5010\.5010\.8640\.6310\.27020\.113\.63089R1392M0\.5610\.5610\.8640\.6890\.25311\.66\.94972R2292M0\.5700\.5700\.8620\.6880\.26617\.03\.97454TABLE VIII:Main, warm\-up and BOS rerun \(seed 0, learning rate10−410^\{\-4\}\), with secondary metrics; second seeds, matched\-steps and learning\-rate runs are in Table[IX](https://arxiv.org/html/2610.33016#A13.T9)\. Main runs: 1 GB of Nepali and 1 GB of English, documents separated by the end\-of\-text token only\. Warm\-up runs: the same, after training only the embedding matrix on the first 10% of the stream \(Tokens excludes that stage\)\. Held\-out BPB is scored in pieces of about 2,000 bytes, after the registered prefix \(reg\.: beginning\-of\-sequence token for Llama\) or the training separator \(sep\.: end\-of\-text token\); for Qwen the two coincide\. The base Llama model was never trained without its beginning\-of\-sequence token, so its separator score is not meaningful \(n/a\)\. All columns other than sep\. use the registered prefix; for the Llama warm\-up runs they are dominated by the scoring artifact of §[VII\-B](https://arxiv.org/html/2610.33016#S7.SS2)and should not be compared with the main runs\. 1 GB, BOS in training: the BOS\-consistent rerun, i\.e\. the main runs repeated with every training document wrapped as BOS, document, end\-of\-text \(third amendment\); its sep\. column is scored after the end\-of\-text token, which these models never saw directly before text\. Belebele, translation and generation speed were not run for it \(–\)\. Belebele: zero\-shot accuracy on 900 items \(chance 0\.25\); every value lies between 0\.25 and 0\.29 with 95% intervals of about±\\pm0\.03, so it cannot separate the arms\. chrF\+\+: 5\-shot FLORES\-200 devtest translation, single greedy pass, no intervals, keeping only the first line of output \(empty if the model starts with a newline\), so we draw no conclusions from it\. Gen\. B/s: UTF\-8 bytes of Nepali generated per second, measured on GPUs shared with concurrent training runs; it is noisy \(the base andR0Qwen models share tokenizer and architecture yet differ by 41%\) and we do not draw conclusions from it\.#### Equal compute and training time\.

AtR2’s final step count, theR0checkpoint of the main runs, whose learning rate had not yet decayed, is already better than the finalR2model by0\.0600\.060\[0\.056,0\.063\]\[0\.056,0\.063\]on Llama and0\.0530\.053\[0\.049,0\.057\]\[0\.049,0\.057\]on Qwen\. TheR1checkpoint at that step is better by0\.0060\.006\[0\.003,0\.008\]\[0\.003,0\.008\]on Llama and level on Qwen \(0\.00000\.0000\[−0\.0021,\+0\.0020\]\[\-0\.0021,\+0\.0020\]; finalR1is better than finalR2by0\.0090\.009on the same pieces, so on Qwen all ofR1’s advantage arrives in its last 192 steps, over which its learning rate falls the last 16% of the way to its floor\)\. These are scored on the first 1 MB of held\-out text with the registered prefix, so the Llama values carry the BOS artifact of §[VII\-B](https://arxiv.org/html/2610.33016#S7.SS2); no such checkpoints were scored for the rerun\. Measured training time fell by only 3% \(Llama\) and 11% \(Qwen\) relative toR1, in runs that shared GPUs with evaluation jobs\.

#### Warm\-up runs\.

Training only the embedding matrix first leavesR0where it was \(0\.4710\.471after the training separator\) and improvesR2from0\.5100\.510to0\.5010\.501\.R2’s deficit relative toR0shrinks from0\.0390\.039to0\.0310\.031, as the second amendment predicted, and its gap toR1from0\.0270\.027to0\.0210\.021\[0\.020,0\.022\]\[0\.020,0\.022\]\. These are differences between single runs, so we treat them as suggestive\. The amendment stated its prediction for the registered prefix, where the warm\-up makes Llama scores unusable:R0moves from0\.4740\.474to0\.7790\.779\. With tied embeddings, the BOS row is updated as an output class even though it never appears as input, and in the warm\-up runs its norm grows from0\.8630\.863to0\.8960\.896; this may contribute\.

#### English\.

In the main Llama runs under the registered prefix,R2−\-R0on English is\+0\.0003\+0\.0003\[−0\.0002,\+0\.0008\]\[\-0\.0002,\+0\.0008\]\(non\-inferior\)\. All three main\-run arms are slightly worse on English than the base model \(0\.7990\.799to0\.8030\.803against0\.7880\.788\)\.

#### Other BPB views\.

The registered sliding\-window BPB, which scores whole documents, orders the Nepali arms the same way \(Llama rerun0\.4240\.424,0\.4370\.437,0\.4640\.464; Qwen0\.4590\.459,0\.5150\.515,0\.5250\.525\)\. In the BOS\-consistent rerun, FLORES Nepali BPB is0\.6070\.607,0\.6170\.617and0\.6260\.626forR0,R1andR2; the rerun’s three held\-out differences match the separator diagnostic of the main runs \(\+0\.0121\+0\.0121,\+0\.0394\+0\.0394,\+0\.0273\+0\.0273\), andR2−\-R1lies between the two earlier values, as the third amendment predicted\. At10−410^\{\-4\}, every Qwen arm ends better on English than its base model \(0\.8620\.862to0\.8640\.864against0\.8730\.873\)\.

#### A failed prediction of the third amendment\.

The third amendment also predicted that scores after BOS and after the end\-of\-text token would agree within0\.0050\.005\. This failed: after the end\-of\-text token the rerun models score1\.0171\.017,0\.8750\.875and1\.6661\.666\. Score a model after the context it was trained with\.

#### English gains\.

R2is better thanR1on English by0\.00220\.0022on both models although English tokens are identical; a likely reason is that English is 74% ofR2’s training tokens but 55% ofR1’s, so it weighs more inR2’s loss\.

#### BOS diagnostic\.

On 40 held\-out Nepali documents \(first 1,500 characters\), Llama BPB after BOS and after the end\-of\-text token are 0\.720 and 1\.865 for the base model, 0\.441 and 0\.438 forR0of the main runs, and 0\.758 and 0\.438 forR0of the warm\-up runs\. The norm of the BOS embedding is 0\.863, 0\.860 and 0\.896 respectively \(mean row norm 0\.98–0\.99\)\. The base model, pretrained with BOS at sequence starts, scores far worse without it, while the continued\-pretraining models, trained on fixed\-length windows that start at arbitrary tokens, do not need it\. The BOS\-consistent rerun shows the converse: trained with BOS before every document, its models score far worse after the end\-of\-text token \(Table[VIII](https://arxiv.org/html/2610.33016#A11.T8), sep\. column\)\. We did not test the mechanism further\.

#### Secondary metrics\.

Belebele is near chance for every model\. Translation scores are erratic: QwenR2scores 3\.9 chrF\+\+ from English to Nepali, below the untrained base model \(6\.8\)\. We take the first output line as the hypothesis, which is empty when a model starts with a newline, and we did not measure how often that happens\. Qwen’s FLORES BPB is almost the same forR1andR2\(0\.689 and 0\.688\), althoughR2is worse on held\-out native text\. We draw no conclusions from these metrics\.

#### Measured generation speed\.

Generation was timed once per model, on GPUs that were also running training or evaluation jobs\. The Qwen base model and QwenR0share tokenizer and architecture, yet differ by 41% in bytes per second, and the generated text has fewer bytes per token than FLORES \(8\.1 against 12\.3 for LlamaR2\), so the measured speed also depends on what each model chose to write\. With these caveats,R2generates1\.481\.48\(Llama\) and1\.501\.50\(Qwen\) times as many bytes per second asR1, and its generations carry8\.078\.07and7\.477\.47bytes per token against5\.215\.21and5\.075\.07forR1\. Section[VII\-B](https://arxiv.org/html/2610.33016#S7.SS2)reports these together with the deterministic ratio of decoding steps on reference text\.

## Appendix LWhat the repair does not fix

A wider word class removes the splits that the regex forces, but BPE can still split inside a syllable: 26% ofo200k’s Nepali tokens and 11 to 13% ofR2’s start with a combining mark\. An akshara\-aware rule\[[26](https://arxiv.org/html/2610.33016#bib.bib19),[14](https://arxiv.org/html/2610.33016#bib.bib20)\]restricted to the Devanagari block would keep English token ids by the argument of Proposition[1](https://arxiv.org/html/2610.33016#Thmproposition1); testing one in a deployed model is future work\. Added tokens are a second route past the floor that needs no regex change; they split more words mid\-way \(§[III](https://arxiv.org/html/2610.33016#S3)\), but we have not measured their effect on model quality, and our training runs give no evidence that segmentation of this kind matters for it\.

## Appendix MAll runs of the later amendments

Table[IX](https://arxiv.org/html/2610.33016#A13.T9)lists every run of the Llama BOS setting and of Qwen\. Matched\-steps LlamaR2is slightly worse on English than the other Llama arms \(0\.8040\.804against0\.7980\.798to0\.8000\.800\), possibly because it sees part of its stream twice\.

ModelArmLRSeedStepsne BPBen BPBLlamaR010−410^\{\-4\}07970\.47090\.7985LlamaR010−410^\{\-4\}17970\.47090\.7984LlamaR110−410^\{\-4\}07370\.48330\.7998LlamaR110−410^\{\-4\}17370\.48320\.7993LlamaR210−410^\{\-4\}05460\.50890\.7975LlamaR210−410^\{\-4\}15460\.50800\.7974LlamaR2\-m10−410^\{\-4\}07370\.49570\.8044LlamaR2\-m10−410^\{\-4\}17370\.49480\.8043QwenR010−410^\{\-4\}010950\.50070\.8643QwenR010−410^\{\-4\}110950\.50030\.8631QwenR110−410^\{\-4\}07480\.56090\.8637QwenR110−410^\{\-4\}17480\.55780\.8638QwenR210−410^\{\-4\}05560\.57020\.8615QwenR210−410^\{\-4\}15560\.57290\.8617QwenR2\-m10−410^\{\-4\}07480\.54910\.8628QwenR2\-m10−410^\{\-4\}17480\.54990\.8626QwenR03×10−43\{\\times\}10^\{\-4\}010950\.47940\.8789QwenR13×10−43\{\\times\}10^\{\-4\}07480\.50920\.8748QwenR2\-m3×10−43\{\\times\}10^\{\-4\}07480\.50350\.8751TABLE IX:Every continued\-pretraining run of the Llama BOS setting and of Qwen, including the matched\-steps, second\-seed and learning\-rate runs of the fourth to seventh amendments\. BPB: held\-out text in pieces of about 2,000 bytes after the registered prefix\.R2\-m:R2trained forR1’s step count, with 35% of its stream seen twice\. Seeds change data order \(and bf16 non\-determinism\) only\.