TokEval: A Tokenizer Evaluation Suite

# TokEval: A Tokenizer Evaluation Suite
Source: [https://arxiv.org/html/2608.18062](https://arxiv.org/html/2608.18062)
###### Abstract

Language model tokenizers are typically selected with minimal evaluation, despite the fact that their design choices directly impact model capabilities\. This can be partly attributed to a limited understanding of which tokenizer properties affect which aspects of downstream performance\. We introduceTokEval, a framework of tokenizer evaluation metrics that goes beyond standard measures like fertility and compression rate to capture linguistically and structurally meaningful properties, e\.g\., UTF\-8 character boundary integrity and digit place\-value boundary alignment for mathematics\. To validate whether these metrics are predictive of downstream model performance, we conduct controlled language model pretraining experiments, varying solely the tokenizers’ training data mixture, pretokenization strategy, and training algorithm\. We evaluate the resulting models on bits\-per\-byte \(a tokenizer\-agnostic version of perplexity\) and several benchmarks, spanning linguistic understanding, mathematical reasoning, and code generation\. Our experiments suggest that different intrinsic properties have different impacts on model abilities: information\-theoretic metrics predict language modeling abilities \(Spearman\|ρ\|\|\\rho\|up to 0\.80\), while structure\-sensitive metrics, such as those measuring digit and line\-break handling, correlate with task accuracy\. We hopeTokEvalenables more principled tokenizer evaluation, replacing pretraining sweeps with intrinsic measurement wherever the two agree\.

[cimeister/tokenizer\-intrinsic\-evals](https://github.com/cimeister/tokenizer-intrinsic-evals)

[huggingface\.co/cmeister/tokenizer\-lm\-ablations](https://huggingface.co/cmeister/tokenizer-lm-ablations)

## 1Introduction

Subword tokenization is a ubiquitous component of modern language model pipelines\. Nearly all contemporary large language models \(LLMs\) rely on some variant of subword segmentation \(e\.g\., Byte Pair Encoding\([38](https://arxiv.org/html/2608.18062#bib.bib1), BPE;\)or the UnigramLM algorithm\([19](https://arxiv.org/html/2608.18062#bib.bib2)\)\) to map raw text to discrete token sequences, the format on which models can be efficiently trained and evaluated\. Despite this central role, tokenizer design often receives limited attention\. In contrast to the substantial efforts devoted to optimizing model architectures, training data, and learning algorithms, the tokenizer is typically selected using a small set of heuristics, such as vocabulary size, compression rate, or informal inspection of segmentation quality\. The choice is then rarely revisited over the course of model development\. This neglect persists despite growing evidence that tokenization choices shape downstream model behavior across various domains, including arithmetic accuracy, multilingual performance, and general task quality\([2](https://arxiv.org/html/2608.18062#bib.bib37);[39](https://arxiv.org/html/2608.18062#bib.bib4);[25](https://arxiv.org/html/2608.18062#bib.bib13);[35](https://arxiv.org/html/2608.18062#bib.bib5);[22](https://arxiv.org/html/2608.18062#bib.bib44)\)\. Taken together, these results show that tokenizer design is a substantive modeling choice with measurable consequences for downstream model capabilities\.

Perhaps a core reason for this lack of consideration is that testing out tokenizers is an expensive endeavor: the entire pipeline, from text processing to output formatting, must be reconfigured for each different design choice\. Furthermore, there are two clear issues with the current intrinsic evaluation protocols:111Intrinsic evaluation refers to evaluating a tokenizer in isolation, using only the tokenizer and a corpus, without training a downstream model\.Firstly, the current set of intrinsic evaluation metrics is very narrow\. For example, compression rate tells us only*how much*a tokenizer compresses, but not*how well*it preserves the structural properties that matter for specific downstream tasks\. Second, the relationship between intrinsic metrics and downstream performance remains contested\. As concrete examples,[46](https://arxiv.org/html/2608.18062#bib.bib7)found that Rényi efficiency correlates strongly with machine translation quality, and compression rate has long been treated as the default proxy for tokenizer quality\([35](https://arxiv.org/html/2608.18062#bib.bib5)\)\. Subsequent work, however, directly challenged both findings\([37](https://arxiv.org/html/2608.18062#bib.bib8);[10](https://arxiv.org/html/2608.18062#bib.bib14)\)\. We argue that these mixed results suggest not that intrinsic evaluation is hopeless, but rather that we need richer intrinsic metrics that capture domain\-specific structural tokenizer properties, and controlled experiments, isolating specific impacts of tokenizer characteristics\.

In this paper, we address both needs\. Our contributions are as follows:

- •We introduceTokEval, an open\-source library implementing a suite of intrinsic tokenizer evaluation metrics spanning text, math, code, and multilingual fairness, alongside tokenizer visualizations and integrity checks \([Section3](https://arxiv.org/html/2608.18062#S3)\)\.
- •We propose novel intrinsic metrics for mathematical content and code that target specific downstream capabilities \([Section3\.5](https://arxiv.org/html/2608.18062#S3.SS5)–[Section3\.6](https://arxiv.org/html/2608.18062#S3.SS6)\)\.
- •We conduct controlled pretraining experiments across a grid of tokenizer configurations, varying algorithm, pretokenization strategy, and training data mixture while holding all other variables constant \([Section5](https://arxiv.org/html/2608.18062#S5)\)
- •We use mixed\-effects regression to isolate the contribution of individual metrics while controlling for covariation in the design axes, showing that specific intrinsic metrics are predictive of specific downstream abilities \([Section5](https://arxiv.org/html/2608.18062#S5)\)\.

## 2Background and Related Work

##### Tokenization Algorithms\.

To enable a machine learning model to process text, the text must first be represented as a sequence of real\-valued vectors\. Tokenization is a key step in this preprocessing pipeline\. Formally, tokenization maps an input sequence of symbols \(typically raw text represented at the byte or character level\) to an output sequence of tokens drawn from a finite vocabulary\. Each token can then be associated with a real\-valued vector, for instance, through an embedding lookup table\. Notably, bytes or characters themselves could be used as the vocabulary symbols, in which case, the mapping is trivial\. However, byte\- and character\-level tokenization leads to substantially longer input sequences, which incurs significant computational overhead in both training and inference\([44](https://arxiv.org/html/2608.18062#bib.bib35);[42](https://arxiv.org/html/2608.18062#bib.bib34)\)\. Subword tokenization, where the vocabulary consists of units whose granularity spans between characters and full words, is the dominant approach, as it provides a favorable tradeoff between vocabulary coverage and sequence length\. It both provides a useful inductive bias for learning and improves training and inference efficiency\. Byte Pair Encoding\([38](https://arxiv.org/html/2608.18062#bib.bib1), BPE; \)is the most widely\-used subword tokenization algorithm\. While various other algorithms have been proposed and taken different approaches\([19](https://arxiv.org/html/2608.18062#bib.bib2), e\.g\., UnigramLM;\), many have built on BPE, proposing various modifications to address empirically observed issues, e\.g\., SuperBPE, BoundlessBPE and PickyBPE\([25](https://arxiv.org/html/2608.18062#bib.bib13);[36](https://arxiv.org/html/2608.18062#bib.bib46);[8](https://arxiv.org/html/2608.18062#bib.bib33)\)\.

##### Impacts of Tokenization\.

Several studies have investigated the relationship between tokenization choices and model capabilities\.[2](https://arxiv.org/html/2608.18062#bib.bib37)trained language models, ablating only tokenizer settings, and found that tokenizer choice significantly impacts both LLM downstream performance and training cost\. Notably, they saw that standard intrinsic metrics like fertility and parity were not reliably predictive of downstream performance\.[35](https://arxiv.org/html/2608.18062#bib.bib5)similarly performed tokenizer ablations for LLM training and observed that replacing a multilingual tokenizer with a dedicated monolingual tokenizer improves downstream performance across nearly all tasks and languages, with effect size comparable to that of a language’s share of data in the pretraining corpus\.[39](https://arxiv.org/html/2608.18062#bib.bib4)demonstrated that digit tokenization directly impacts models’ arithmetic capabilities, observing a∼20%\\sim 20\\%difference in mathematics task accuracy from changing tokenization strategy alone\. At a more fundamental level,[22](https://arxiv.org/html/2608.18062#bib.bib44)showed*causally*that including a subword in the vocabulary can increase the probability assigned to the corresponding string by up to 17×\\times, in comparison to if the string were represented by two tokens\.[30](https://arxiv.org/html/2608.18062#bib.bib43)similarly showed that the splitting of natural words into multiple tokens negatively impacts model performance\. Tokenizers have also been linked to discrepancies in model cost and accessibility\. Both[33](https://arxiv.org/html/2608.18062#bib.bib6)and[1](https://arxiv.org/html/2608.18062#bib.bib42)documented systematic cross\-lingual unfairness introduced at the tokenization stage, sometimes described as a “token tax” on under\-represented languages\([28](https://arxiv.org/html/2608.18062#bib.bib16)\)because of the economic consequences it imparts\. More recent work has extended this line of inquiry to investigate dialectal and representational disparities\([16](https://arxiv.org/html/2608.18062#bib.bib38)\)\.

##### Intrinsic Tokenizer Evaluation\.

Of particular interest to this work is the development of intrinsic tokenizer metrics and studies of how such metrics relate to downstream model performance\. An*intrinsic*metric evaluates a tokenizer in isolation, without training or querying a downstream model; it is a function of the tokenizer and a text corpus alone\. By contrast, an*extrinsic*metric measures the effect of a tokenizer on the performance of a model trained with it \(e\.g\., bits\-per\-byte, task accuracy\)\. Intrinsic metrics are attractive because they are cheap: evaluating a tokenizer often takes seconds, while training a language model to measure its extrinsic effect can take days or weeks\. The most widely used intrinsic metrics are compression\-based measures, such as corpus token count and fertility, which quantify the extent to which a tokenizer fragments text\. Prior work, however, has shown that compression\-based metrics alone are not a reliable objective for tokenizer design For example,[37](https://arxiv.org/html/2608.18062#bib.bib8)showed that minimizing corpus token count does not reliably improve downstream performance, and[10](https://arxiv.org/html/2608.18062#bib.bib14)constructed tokenizers whose Rényi efficiency can be increased arbitrarily while*degrading*model performance\. More recent work has therefore argued for evaluating tokenizers through multiple complementary intrinsic signals rather than through compression alone\([26](https://arxiv.org/html/2608.18062#bib.bib15)\)\. Another line of work studies tokenizer alignment with morphological units\.[3](https://arxiv.org/html/2608.18062#bib.bib9)introduce MorphScore, which measures the alignment between token boundaries and morpheme boundaries, and show that tokenization quality helps explain some cross\-linguistic differences in language\-model performance, though morphological boundary alignment itself is not consistently predictive\. Relatedly,[40](https://arxiv.org/html/2608.18062#bib.bib41)show that morphology\-preserving segmentations tend to yield better contextualized representations than segmentations that split roots and[16](https://arxiv.org/html/2608.18062#bib.bib38)show that tokenization parity is especially predictive of downstream performance on dialectal NLP tasks that rely on syntactic and morphological cues\.

## 3TheTokEvalFramework

TokEvalcomputes a suite of intrinsic metrics for a tokenizer over a user\-specified text corpus\. The library supports textual inputs on a per\-language basis for fine\-grained cross\-lingual analysis\. Concretely, let𝒯\\mathcal\{T\}be a tokenizer and𝒞=\{\(𝐱i,ℓi\)\}i=1N\\mathcal\{C\}=\\\{\(\\mathbf\{x\}\_\{i\},\\ell\_\{i\}\)\\\}\_\{i=1\}^\{N\}be a corpus of texts𝐱i\\mathbf\{x\}\_\{i\}with language labelsℓi\\ell\_\{i\}\. We use\|𝐱\|u\|\\mathbf\{x\}\|\_\{u\}to denote the length of the text𝐱\\mathbf\{x\}in unitu∈\{bytes,chars,words,lines\}u\\in\\\{\\texttt\{bytes\},\\texttt\{chars\},\\texttt\{words\},\\texttt\{lines\}\\\}, wherebytesis the default\.222The appropriate unit depends on the analysis\. Bytes require no language\-specific definition of word or character boundaries, but they penalize scripts that have larger per\-character UTF\-8 encodings \(1 byte for Latin, 3\-4 for CJK, 4 for emoji\)\. Line counts are appropriate for parallel corpora where each line is a translation, making the count constant across languages and isolating the tokenizer from both linguistic conventions and encoding artifacts\.Every text is encoded to a token sequence𝐭i=𝒯⁡\(𝐱i\)\\mathbf\{t\}\_\{i\}=\\mathcal\{T\}\(\\mathbf\{x\}\_\{i\}\), where each tokenttcomes from a tokenizer’s vocabulary𝒱\\mathcal\{V\}\. All metrics are reported per language and aggregated globally\.[Table1](https://arxiv.org/html/2608.18062#S3.T1)summarizes the metrics; additional details and design choice justifications are provided in the following subsections\.

TokEvalis implemented in Python\. It natively handles five tokenizer classes: HuggingFace, which loads both thetokenizers\(Rust\-accelerated\) andtransformersformats through one class, SentencePiece\([18](https://arxiv.org/html/2608.18062#bib.bib3)\), a pre\-tokenized\-corpus class for input that is already segmented into tokens, and ScriptBPE/MinGram, which both use the SCRIPT\-encoding classes\([20](https://arxiv.org/html/2608.18062#bib.bib39)\); aregister\_tokenizer\_classhook lets users register further tokenizer classes\. A unified wrapper ensures identical metric computation regardless of backend\.TokEvalalso includes a “sanity\-check” module that runs sixteen deterministic checks per tokenizer, e\.g\., byte coverage, combining\-mark handling, digit handling, roundtrip fidelity, and vocabulary reachability, and aggregates them into one pass/warn/fail severity per tokenizer\. A visualization module generates visualizations of tokenized text and standard plots from a results set, including per\-metric bar charts and per\-language and faceted breakdowns, across the tokenizers being compared\.

Table 1:Summary ofTokEvalintrinsic metrics\. Notation:p⁡\(t\)p\(t\)is the relative frequency of vocabulary entryttin the input corpus;ntn\_\{t\}is the count of all bigrams in the corpus withttin first position;nt,t′n\_\{t,t^\{\\prime\}\}is the count of all trigrams with\(t,t′\)\(t,t^\{\\prime\}\)in the first two positions;𝒜⁡\(c\)\\mathcal\{A\}\(c\)is the set of distinct vocabulary entries observed as successors of the contextcc;𝐱^=𝒯−1​\(𝒯​\(𝐱\)\)\\hat\{\\mathbf\{x\}\}=\\mathcal\{T\}^\{\-1\}\(\\mathcal\{T\}\(\\mathbf\{x\}\)\)is the round\-trip reconstruction;BobsB\_\{\\mathrm\{obs\}\}/BidealB\_\{\\mathrm\{ideal\}\}are observed/ideal digit boundary sets \(see[Section3\.5](https://arxiv.org/html/2608.18062#S3.SS5)\);𝒮\\mathcal\{S\}is the set of special tokens\.MetricFormulaInterpretationCompression & Information\-Theoretic\([Section3\.1](https://arxiv.org/html/2608.18062#S3.SS1)\)Compression rateCRℓ=∑i:ℓi=ℓ\|𝐱i\|u/∑i:ℓi=ℓ\|𝐭i\|\\mathrm\{CR\}\_\{\\ell\}=\\sum\_\{i:\\ell\_\{i\}=\\ell\}\|\\mathbf\{x\}\_\{i\}\|\_\{u\}\\,/\\,\\sum\_\{i:\\ell\_\{i\}=\\ell\}\|\\mathbf\{t\}\_\{i\}\|Multiplier by which tokenization compresses\.↑\\uparrow= more compression\.Unigram entropyH\(p\(⋅\)\)=−∑tp\(t\)log2p\(t\)H\(p\(\\cdot\)\)=\-\\sum\_\{t\}p\(t\)\\log\_\{2\}p\(t\)Uniformity of vocabulary usage\.↑\\uparrow= more even\.Vocab\. utilization\|\{t∈𝒱:p⁡\(t\)\>0\}\|/\|𝒱\|\|\\\{t\\in\\mathcal\{V\}:p\(t\)\>0\\\}\|\\,/\\,\|\\mathcal\{V\}\|Fraction of vocabulary entries the corpus uses at least once\.Token length∑i\|𝐱i\|chars/∑i\|𝐭i\|\\sum\_\{i\}\|\\mathbf\{x\}\_\{i\}\|\_\{\\texttt\{chars\}\}\\,/\\,\\sum\_\{i\}\|\\mathbf\{t\}\_\{i\}\|Mean characters per token occurrence\.Bigram entropyη\(t\)=H\(p\(⋅\|t\)\)/log2\|𝒜\(t\)\|\\eta\(t\)=H\(p\(\\cdot\|t\)\)\\,/\\,\\log\_\{2\}\|\\mathcal\{A\}\(t\)\|; η¯=∑tnt​η​\(t\)/∑tnt\\bar\{\\eta\}=\\sum\_\{t\}n\_\{t\}\\eta\(t\)/\\sum\_\{t\}n\_\{t\}Conditional entropy of the next token given the previous one, divided by the entropy the context’s own observed successor set could carry\.η¯∈\[0,1\]\\bar\{\\eta\}\\\!\\in\\\!\[0,1\];↓\\downarrow= predictable sequences\.Trigram entropyη\(t,t′\)=H\(p\(⋅\|t,t′\)\)/log2\|𝒜\(t,t′\)\|\\eta\(t,t^\{\\prime\}\)=H\(p\(\\cdot\|t,t^\{\\prime\}\)\)\\,/\\,\\log\_\{2\}\|\\mathcal\{A\}\(t,t^\{\\prime\}\)\|; η¯3=∑t,t′nt,t′​η​\(t,t′\)/∑t,t′nt,t′\\bar\{\\eta\}\_\{3\}=\\sum\_\{t,t^\{\\prime\}\}n\_\{t,t^\{\\prime\}\}\\eta\(t,t^\{\\prime\}\)/\\sum\_\{t,t^\{\\prime\}\}n\_\{t,t^\{\\prime\}\}As bigram entropy, with the conditioning context one token longer\.Rényi efficiencyH¯α​\(p⁡\(⋅\)\)=Hα​\(p⁡\(⋅\)\)/log2⁡\|𝒱\|\\bar\{H\}\_\{\\alpha\}\(p\(\\cdot\)\)=H\_\{\\alpha\}\(p\(\\cdot\)\)/\\log\_\{2\}\|\\mathcal\{V\}\|, Hα​\(p⁡\(⋅\)\)=11−α​log2⁡\(∑tp​\(t\)α\)H\_\{\\alpha\}\(p\(\\cdot\)\)=\\frac\{1\}\{1\-\\alpha\}\\log\_\{2\}\(\\sum\_\{t\}p\(t\)^\{\\alpha\}\)Generalized uniformity; computed forα∈\{1,2,2\.5,3\}\\alpha\\in\\\{1,2,2\.5,3\\\}, and reported here forα=2\\alpha=2\.↑\\uparrow= more uniform token usage\.Avg\. token rankr¯ℓ=∑i:ℓi=ℓ∑jr\(ti,j\)/∑i:ℓi=ℓ\|𝐭i\|\\bar\{r\}\_\{\\ell\}=\\sum\_\{i:\\ell\_\{i\}=\\ell\}\\sum\_\{j\}r\(t\_\{i,j\}\)\\,/\\,\\sum\_\{i:\\ell\_\{i\}=\\ell\}\|\\mathbf\{t\}\_\{i\}\|Token frequency rank averaged over all token occurrences, whererrranks by frequency in the corpus\.↓\\downarrow= few common tokens dominate\.Linguistic Alignment\([Section3\.2](https://arxiv.org/html/2608.18062#S3.SS2)\)FertilityFertℓ=meani:ℓi=ℓ\|𝐭i\|/\|𝐱i\|words\\mathrm\{Fert\}\_\{\\ell\}=\\text\{mean\}\_\{i:\\ell\_\{i\}=\\ell\}\\,\|\\mathbf\{t\}\_\{i\}\|/\|\\mathbf\{x\}\_\{i\}\|\_\{\\texttt\{words\}\}Tokens per word; defaultu=wordsu\{=\}\\texttt\{words\}\(mean of ratios\)\.1\.01\.0= one token per word\.MorphScoreMorpheme boundary P/R/F1 \(external library\)Token–morpheme alignment across 70 languages\.Multilingual Fairness\([Section3\.3](https://arxiv.org/html/2608.18062#S3.SS3)\)TFG \(Gini\)TFG=∑i,j\|ci−cj\|/2​n2​μ\\mathrm\{TFG\}=\\sum\_\{i,j\}\|c\_\{i\}\-c\_\{j\}\|\\,/\\,2n^\{2\}\\mu, cℓ=∑i\|𝐭i\|/∑i\|𝐱i\|uc\_\{\\ell\}=\\sum\_\{i\}\|\\mathbf\{t\}\_\{i\}\|/\\sum\_\{i\}\|\\mathbf\{x\}\_\{i\}\|\_\{u\}Cross\-lingual inequality of tokenization cost\.00= perfect parity\.Vocab\. util\. CoVuℓ=\|\{used vocab entries in​ℓ\}\|/\|𝒱\|u\_\{\\ell\}=\|\\\{\\text\{used vocab entries in \}\\ell\\\}\|/\|\\mathcal\{V\}\|; CoV=σ⁡\(uℓ\)/μ⁡\(uℓ\)\\mathrm\{CoV\}=\\sigma\(u\_\{\\ell\}\)/\\mu\(u\_\{\\ell\}\)over languagesℓ\\ellCross\-lingual dispersion of per\-language vocabulary utilization \(μ\\mu= mean,σ\\sigma= sample std\)\.↓\\downarrow= similar utilization across languages\.Encoding Fidelity\([Section3\.4](https://arxiv.org/html/2608.18062#S3.SS4)\)Exact match𝟙\[𝐱^=𝐱\]\\mathbb\{1\}\[\\hat\{\\mathbf\{x\}\}=\\mathbf\{x\}\], averaged over corpusStrict round\-trip reconstruction rate\.CERLev⁡\(𝐱,𝐱^\)/\|𝐱\|chars\\mathrm\{Lev\}\(\\mathbf\{x\},\\hat\{\\mathbf\{x\}\}\)\\,/\\,\|\\mathbf\{x\}\|\_\{\\texttt\{chars\}\}Character\-level Levenshtein distance, normalized; can exceed 1\.UNK ratenunk/ntotaln\_\{\\mathrm\{unk\}\}/n\_\{\\mathrm\{total\}\}Fraction of tokens mapped to unknown ID\.UTF\-8 completeness\|\{\(i,j\):bytes\(ti,j\)is valid UTF\-8 andti,j∉𝒮\}\|/\|\{\(i,j\):ti,j∉𝒮\}\|\\\{\(i,j\):\\text\{bytes\}\(t\_\{i,j\}\)\\,\\text\{is valid UTF\-8 and \}\\,t\_\{i,j\}\\notin\\mathcal\{S\}\\\}\|/\|\\\{\(i,j\):t\_\{i,j\}\\notin\\mathcal\{S\}\\\}Fraction of non\-special token occurrences that decode to complete characters\. Occurrence\-weighted in the corpus\.Char split rate\|\{c:multi\-byte, split\}\|/\|\{c:multi\-byte\}\|\|\\\{c:\\text\{multi\-byte, split\}\\\}\|\\,/\\,\|\\\{c:\\text\{multi\-byte\}\\\}\|Proportion of multi\-byte chars whose bytes span multiple tokens\.UTF\-8 boundary crossingFrac\. of non\-special token occurrences spanning\>\>1 character and leaving one of them incompleteFrequency of tokens produced by a merge across a character boundary\.Digit Tokenization\([Section3\.5](https://arxiv.org/html/2608.18062#S3.SS5)\)Digit boundaryF1F\_\{1\}F1F\_\{1\}ofBobsB\_\{\\mathrm\{obs\}\}vs\.BidealB\_\{\\mathrm\{ideal\}\}\(right\-aligned triples\)Place\-value alignment of digit span segmentation\.Digit split variabilityHpattern=−∑p\(np/N\)log2\(np/N\)H\_\{\\mathrm\{pattern\}\}=\-\\sum\_\{p\}\(n\_\{p\}/N\)\\log\_\{2\}\(n\_\{p\}/N\)Entropy of splitting patterns for same\-length digit spans\.↓\\downarrow= consistent\.Operator isolationFrac\. of operator occurrences whose covering tokens carry no non\-whitespace character from outside the operator spanWhether operators are fused with adjacent operands\. Computed separately on prose, code and math; reported here on the math corpus\.Code Tokenization\([Section3\.6](https://arxiv.org/html/2608.18062#S3.SS6)\)AST boundary alignmentFrac\. of AST leaf nodes fully aligned with token boundariesPer\-category \(identifier, keyword, operator, literal, delimiter\)\.Ident\. fragmentationFrac\. of identifiers split across\>\>1 token↑\\uparrow= more fragmentation of programmer\-defined names\.Indentation consistencyρ⁡\(nesting depth,whitespace token count\)\\rho\(\\text\{nesting depth\},\\text\{whitespace token count\}\)Spearman correlation;↑\\uparrow= monotonic depth signal\.

### 3\.1Compression and Information\-Theoretic Metrics

The library’s implementation ofCompression Rateuses a ratio of sums: total length of the corpus in the chosen base unit over total length in tokens\. This is done rather than a mean of per\-text ratios, avoiding bias toward short texts\. Unigram Entropy is the Shannon entropy of the empirical token frequency distribution, as computed with respect to the input corpus\.Rényi Efficiencygeneralizes this entropy to other entropy orders via a hyperparameterα\\alpha, recovering Shannon entropy whenα=1\\alpha=1\([46](https://arxiv.org/html/2608.18062#bib.bib7)\)\. To capture sequential structure,Bigram Entropycomputes theconditionalentropy of the next token given the previous one, frequency\-weighted by how often each bigram\-initial token occurs\. This metric is largely inspired by the work of[34](https://arxiv.org/html/2608.18062#bib.bib45), although it differs in several ways\.333This definition deviates from[34](https://arxiv.org/html/2608.18062#bib.bib45)in several ways: \(i\) each conditional entropy is normalized bylog2\\log\_\{2\}of the number of distinct successors \(next tokens\) that the context had in the corpus, i\.e\., by the maximum entropy achievable by a distribution over that same number of successors;\(ii\) the aggregation is frequency\-weighted rather than an unweighted mean over contexts; \(iii\) bigram\-initial tokens containing punctuation or digits are included; \(iv\) there is no windowing\. The library also computes the reference definition for comparison\.Low bigram entropy indicates highly predictable sequences, which can signal over\-fragmentation, a common symptom when a tokenizer was not trained on the language being evaluated\([34](https://arxiv.org/html/2608.18062#bib.bib45)\)\.Trigram Entropyuses the same setup, albeit conditioning on bigrams instead of single tokens\.Vocabulary Utilizationis the fraction of the total vocabulary entries that occur at least once in the corpus\.Token Lengthis the mean number of characters per token over all token occurrences in the corpus\. Finally,Average Token Rankassigns each vocabulary entry a rank by descending frequency \(rank 1 = most frequent\) and reports the mean frequency rank over all token occurrences in a given language\([24](https://arxiv.org/html/2608.18062#bib.bib36)\)\. A low value indicates that a few common tokens account for most occurrences, suggesting poor vocabulary utilization\.

### 3\.2Linguistic Alignment Metrics

Fertilityis approximately the inverse of Compression Rate, differing only in aggregation and default choice ofuu\. Cross\-lingual variation in fertility is a primary indicator of tokenization\-induced unfairness\([33](https://arxiv.org/html/2608.18062#bib.bib6)\): fragmenting low\-resource languages into many subwords per word degrades downstream performance\([22](https://arxiv.org/html/2608.18062#bib.bib44);[30](https://arxiv.org/html/2608.18062#bib.bib43)\)\.MorphScore\([3](https://arxiv.org/html/2608.18062#bib.bib9);[5](https://arxiv.org/html/2608.18062#bib.bib11)\)measures token–morpheme boundary alignment via precision, recall, and F1 over gold\-standard morphological segmentations, covering 70 languages\.TokEvalintegrates MorphScore directly, as a submodule of the fork\. By default, it uses the MorphScore library’s recommended settings\.

### 3\.3Multilingual Fairness

Tokenizer Fairness Gini \(TFG\)measure cross\-lingual encoding inequality via a Gini coefficient over per\-language tokenization costscℓ=∑i\|𝐭i\|/∑i\|xi\|uc\_\{\\ell\}=\\sum\_\{i\}\|\\mathbf\{t\}\_\{i\}\|/\\sum\_\{i\}\|x\_\{i\}\|\_\{u\}\. This metric is best computed withu=linesu=\\texttt\{lines\}on a parallel corpus, so that differences incℓc\_\{\\ell\}reflect tokenizer behavior rather than cross\-lingual variation in text length or encoding overhead\. Values near 1 indicate severe cross\-lingual inequality\.TokEvaladditionally provides Lorenz curve visualizations\.Vocabulary\-Utilization CoVtakes the coefficient of variation across per\-language vocabulary utilizationuℓu\_\{\\ell\}, quantifying differences across languages’ use of the vocabulary\. A lower CoV means the vocabulary is used to a similar degree across languages; a higher CoV means usage concentrates in some languages\. It is undefined when fewer than two languages are present\.

### 3\.4Encoding Fidelity

A tokenizer is not necessarily lossless: the round\-trip reconstruction𝐱^=𝒯−1​\(𝒯​\(𝐱\)\)\\hat\{\\mathbf\{x\}\}=\\mathcal\{T\}^\{\-1\}\(\\mathcal\{T\}\(\\mathbf\{x\}\)\)may differ from the original due to Unicode normalization, UNK substitution, or whitespace handling\.Reconstruction Exact Matchand, as a graded complement,Character Error Rate \(CER\)together characterize round\-trip fidelity\. CER uses the canonical definition of the metric and can exceed 1 when𝐱^\\hat\{\\mathbf\{x\}\}is substantially longer than𝐱\\mathbf\{x\}\(e\.g\., byte\-fallback expansion\)\. High exact\-match failure with near\-zero CER indicates minor systematic changes \(e\.g\., normalization\); high CER indicates fundamental encoding problems\. AUTF\-8 Token Completenessrate<1<\\\!1is not necessarily a defect: byte\-fallback mechanisms rely on incomplete byte tokens to represent unseen characters\.Character Split Rateis stratified and reported by byte width \(2\-, 3\-, 4\-byte\), since CJK characters and emoji are disproportionately affected\.UTF\-8 Boundary Crossing Rateis the fraction of non\-special token occurrences whose bytes span more than one character and leave at least one of those characters incomplete\. It captures the frequency of tokens produced by a merge across a character boundary, albeit not including \(at least one of\) the full characters on either side of the boundary\.

### 3\.5Digit Tokenization Quality

Tokenizer treatment of numbers directly affects arithmetic reasoning\([39](https://arxiv.org/html/2608.18062#bib.bib4)\)\. Motivated by these findings, we propose metrics that operationalize behaviors shown to benefit models’ math processing\.Three\-digit Boundary Alignmentevaluates whether digit spans are split at positions consistent with right\-aligned grouping into triples: for each contiguous digit span of lengthdd, we define*ideal*boundaries at positions\{d−3,d−6,…\}\\\{d\{\-\}3,d\{\-\}6,\\ldots\\\}from the left and compute precision, recall, andF1F\_\{1\}against the tokenizer’s observed boundaries\. For spans with≤3\\leq\\\!3digits, both boundary sets are empty; we setF1=1F\_\{1\}\\\!=\\\!1by convention iff the tokenizer keeps the span intact \(i\.e\.,\|Bobs\|=0\|B\_\{\\mathrm\{obs\}\}\|=0\)\. Results are bucketed by digit length\.Digit Split Variabilitycaptures the consistency of segmentation: inconsistent segmentation of same\-length digit spans may inhibit learning generalizable arithmetic patterns\. We group digit spans by length, record the splitting pattern of each occurrence \(e\.g\.,XX\|XXXvs\.X\|XXXX\), and compute the Shannon entropy of the resulting pattern distribution\.Operator Isolationmeasures whether operators are dedicated tokens rather than fused with adjacent operands \(e\.g\.,\+3\)\. The operator set is the 24 operators the library scores, in six categories: arithmetic, assignment, logical and bitwise, shift, comparison and ternary\. An occurrence counts as isolated when no token covering it carries a non\-whitespace character from outside the operator’s own span\. The metric is computed on three separate corpora, prose, code and math, and reported per domain; we report math results here\. The library also pools the three into an occurrence\-weighted micro\-average over operator occurrences\.

### 3\.6Code Tokenization Quality

Programming languages have deterministic grammars, so syntactic boundaries can be identified without manual annotation\. Parsing source code produces an abstract syntax tree \(AST\): a tree whose internal nodes represent grammatical constructs \(function definitions, loops, assignments\) and whose leaf nodes represent the atomic syntactic units that appear in the source text\. We propose novel metrics that evaluate whether tokenizers respect these boundaries\.444All parsing and identification is done using the python tree\-sitter library\.AST Boundary Alignmentchecks whether each AST leaf node’s character span\[s,e\)\[s,e\)coincides with token boundaries\.555A node is*fully aligned*if bothssandeefall on token boundaries\.The library also reports the two one\-sided rates separately; theEnd\-Alignment Rateis the fraction of AST leaf nodes whose end offseteefalls on a token boundary, irrespective ofss\. The alignment rate is reported per node category and aggregated across the 15 programming languages we measure: Bash, C, C\+\+, C\#, Go, Java, JavaScript, Lua, PHP, Python, R, Ruby, Rust, Scala, TypeScript\.Identifier Fragmentationis a complement to AST Boundary Alignment\. It is the occurrence\-weighted fraction of programmer\-defined identifiers that require more than one token to encode\. The relatedTokens per Identifierreports the mean number of tokens per programmer\-defined identifier occurrence rather than the fraction that take more than one\.Indentation Consistencytargets whitespace\-sensitive languages \(Python and Haskell\), computing the Spearman rank correlation between nesting depth and the number of whitespace\-only tokens in leading indentation per line\. Haskell is absent from all of our code results because its tree\-sitter grammar crashes the parser, so the values reported here are Python only\.

## 4Experimental Setup

We train 1\.27B\-parameter language models under 46 tokenizer configurations: 44 tokenizers trained using different configurations, plus two off\-the\-shelf reference tokenizers \(Mistral\-Nemo, LLaMA\-3\)\. We train two sets of models, differing only in their pretraining data: a natural\-language\-focused track and math\+code\-focused track\. Within each line of experiments, architecture, model training data, and hyperparameters are held constant; the sole experimental variable is the tokenizer\. Every model reported in this paper, the tokenizer each was trained with, and a per\-run table of the metrics reported here are released at[https://huggingface\.co/cmeister/tokenizer\-lm\-ablations](https://huggingface.co/cmeister/tokenizer-lm-ablations)\.

### 4\.1Tokenizer Configurations

For the tokenizers that we train, we focus on ablating three axes: algorithm, normalization/pretokenization strategy, and training data composition\. Vocabulary size is held constant at∼\\sim128K entries\. This paper uses different overlapping subsets of tokenizer–model pairs in different analyses\. The subsets are built from*tokenizer families*: a family groups tokenizers that share the same value on all three design axes and differ only in a training hyperparameter or preset of that shared configuration\.[SectionC\.1\.1](https://arxiv.org/html/2608.18062#A3.SS1.SSS1)gives the full definition with examples, and the Fam\. column of[Table5](https://arxiv.org/html/2608.18062#A1.T5)lists each row’s family, so the partition can be checked directly\. Theprimary panel\(P;n=29n=29\) takes one representative per family; it is the set on which the aggregate correlations \([Tables2](https://arxiv.org/html/2608.18062#S5.T2)and[16](https://arxiv.org/html/2608.18062#A6.T16)\) and the per\-language regressions \([Table3](https://arxiv.org/html/2608.18062#S5.T3)\) are computed\. Theextended panel\(E;n=39n=39\) contains every member of every family; it is used for the family\-mean correlations of[SectionF\.2](https://arxiv.org/html/2608.18062#A6.SS2)and the predictor fits of[SectionF\.1](https://arxiv.org/html/2608.18062#A6.SS1)\. Themath\+code panel\(M;n=20n=20\) holds the custom tokenizers with a math\+code model: the 19 primary\-panel members that have one, plus one English\-only build kept with a training defect flagged \([SectionC\.1\.1](https://arxiv.org/html/2608.18062#A3.SS1.SSS1)\); the math\+code columns of[Table2](https://arxiv.org/html/2608.18062#S5.T2)use only the 19 primary\-panel members\. Thereference panel\(R;n=2n=2\) holds the two off\-the\-shelf tokenizers, which are excluded from every ranking and aggregate statistic, because their training configurations are largely unknown and cannot be controlled for, and serve instead as held\-out prediction targets \([SectionF\.1](https://arxiv.org/html/2608.18062#A6.SS1)\); the one exception is the cross\-scale ranking\-stability check of[SectionE\.3](https://arxiv.org/html/2608.18062#A5.SS3), which measures how a fixed set of models reorders between training scales and therefore includes them, as its table states\. More detailed inclusion criteria are provided in[SectionC\.1\.1](https://arxiv.org/html/2608.18062#A3.SS1.SSS1)and the panel definitions are stated concisely in[Table4](https://arxiv.org/html/2608.18062#A1.T4)\.

##### Algorithms\.

We train tokenizers with five algorithms: BPE\([38](https://arxiv.org/html/2608.18062#bib.bib1)\), UnigramLM\([19](https://arxiv.org/html/2608.18062#bib.bib2)\), SuperBPE\([25](https://arxiv.org/html/2608.18062#bib.bib13)\), parity\-aware BPE\([12](https://arxiv.org/html/2608.18062#bib.bib31)\)and MinGram—a minimalist Unigram\-style tokenizer trained using an approximation of EM that more closely mimics inference\([21](https://arxiv.org/html/2608.18062#bib.bib40)\)\. BPE and UnigramLM tokenizers are trained with the HuggingFacetokenizerslibrary; the SuperBPE, parity\-aware BPE and MinGram tokenizers are trained using the authors’ published codebases\. All custom tokenizers are trained on 10 GB of text\. For the SuperBPE tokenizers, which use a two\-part training strategy, we use either plain BPE or parity\-aware BPE for the seed tokenizer, inheriting either 64K or 90K merges from this tokenizer before extending to 128K\.

##### Pretokenization and encoding\.

Simply put, pretokenization splits text before a vocabulary learning algorithm is applied based on a deterministic set of rules, determining what spans of text can become a token in the learned vocabulary\. Consequently, it is a design choice with large influence\([4](https://arxiv.org/html/2608.18062#bib.bib10)\)\. The main pretokenization strategies we explore are a minimal punctuation\-only baseline, the GPT\-4o regex \(camelCase splitting, left\-aligned 3\-digit groups, English contractions\), a Claude\-inspired regex, and a right\-aligned digit variant that groups numbers to place values\. In ablations, we try targeted variants of these pretokenization strategies\. Further details in[SectionA\.1](https://arxiv.org/html/2608.18062#A1.SS1)\. The majority of tokenizers we train use byte\-level encoding, which is standard practice\. We also train several tokenizers using SCRIPT byte encoding\([20](https://arxiv.org/html/2608.18062#bib.bib39)\), which groups characters by Unicode script into blocks before a subword algorithm is applied\.

##### Tokenizer training data compositions\.

The main suite of custom tokenizers are trained on one of three mixtures: English\-only, balanced multilingual \(35% English, 30% multilingual across 30 languages, 15% math, 15% code\), or code\-heavy \(50% English, 50% code\)\. We sample English training data from FineWeb\-Edu\([31](https://arxiv.org/html/2608.18062#bib.bib22)\), multilingual data from FineWeb2\([32](https://arxiv.org/html/2608.18062#bib.bib32)\), math data from FineMath\([27](https://arxiv.org/html/2608.18062#bib.bib23)\), and code data from StarCoderData\([23](https://arxiv.org/html/2608.18062#bib.bib24)\)\. The multilingual portion spans 30 languages across 11 writing systems \(Latin, Cyrillic, Arabic, Devanagari, Bengali, Tamil, Thai, Hangul, CJK, Greek, Hebrew\) and 11\+ language families\. Per\-language data is sampled proportionally to presence in the original FineWeb2 dataset\. Additional tokenizers are trained for ablations, using configurations that weight the 30 multilingual languages equally rather than proportionally, and that restrict the tokenizer’s multilingual training data to only a subset of languages\. More details can be found in[SectionA\.2](https://arxiv.org/html/2608.18062#A1.SS2)\.

### 4\.2Language Model Training

All models use the nanochat architecture\([17](https://arxiv.org/html/2608.18062#bib.bib26)\), a decoder\-only transformer with 24 layers\. The model has a total of 1\.27B parameters, of which 682M are in transformer weight matrices\. For thenatural\-language\-focused models, we use the same datasets as for tokenizer training, albeit in different proportions\. The token budget follows nanochat’s compute\-optimal scaling rules\. Because generative math and code abilities do not surface under the above natural\-language\-focused mixture at this model scale, we additionally trainmath\+code modelsfrom scratch on a∼\\sim20B\-token mixture, 50% math and 50% code by text bytes: math text from MegaMath\-Web\-Pro and code from The Stack v2’s educational subset\. Due to compute constraints, this second track covers a subset of the tokenizers rather than all of them: the math\+code panel \(M\) has 20 custom tokenizers, which with the 2 off\-the\-shelf references gives the 22 models of[Table18](https://arxiv.org/html/2608.18062#A6.T18)\.66619 of the 20 custom tokenizers are also members of P\. The remaining one,claude\-english\-bpe, is not: the math\+code models were trained before it was realized that this tokenizer lacks a full byte alphabet for fallback\. It is included only in the held\-out\-tokenizer code BPB and MBPP pass@1 fits\.Further math\+code models outside that panel are used in the appendix analyses \(the 24\-member extended math\+code roster of[SectionF\.3](https://arxiv.org/html/2608.18062#A6.SS3), the byte\-alphabet retrains, and the whitespace control of[Table14](https://arxiv.org/html/2608.18062#A5.T14)\)\. Full architectural details and hyperparameter values are in[AppendixB](https://arxiv.org/html/2608.18062#A2); training data details are in[SectionB\.3](https://arxiv.org/html/2608.18062#A2.SS3)\.

### 4\.3Evaluation

##### Downstream benchmarks and perplexity evaluations\.

For the natural\-language\-focused models, we consider bits\-per\-byte \(BPB\) on FLORES\+\([29](https://arxiv.org/html/2608.18062#bib.bib28)\), over all 215 FLORES\+ languages and, separately, restricted to the 31 languages present in our training data\. We also evaluate on two minimal\-pair agreement benchmarks: BLiMP\([43](https://arxiv.org/html/2608.18062#bib.bib17)\)and MultiBLiMP\([15](https://arxiv.org/html/2608.18062#bib.bib29)\)\(English and multilingual, respectively\)\. Both are loglikelihood\-scored under the BOS\-prefixed convention described in[AppendixC](https://arxiv.org/html/2608.18062#A3)\. Performance on Belebele \(four\-option multilingual reading comprehension\) was at chance, so we do not report those results\. The math\+code models \([Section4\.2](https://arxiv.org/html/2608.18062#S4.SS2)\) are evaluated on GSM8K\([9](https://arxiv.org/html/2608.18062#bib.bib18)\), HumanEval\([7](https://arxiv.org/html/2608.18062#bib.bib19)\), and on MBPP\([6](https://arxiv.org/html/2608.18062#bib.bib20)\), as well as on bits\-per\-byte held\-out StarCoderData across 7 programming languages \(Code BPB\), using data disjoint from the training mixture\. Generation tasks prepend the BOS token to the prompt, stop at the model’s EOS token, and heal the prompt boundary; shot counts, scoring, the healing procedure, and the BPB computation are in[AppendixC](https://arxiv.org/html/2608.18062#A3)\.

##### Intrinsic–downstream correlation\.

We measure the relationship between intrinsic tokenizer properties and downstream model quality via two complementary analyses\.*Aggregate correlation*computes one global intrinsic metric score and one downstream metric score per tokenizer, reporting the Spearman rank correlation between these values\. For the intrinsic metrics, this means intrinsic scores are averaged across languages, where applicable\. For most downstream metrics, it is already the case that each model receives only a single value, but otherwise \(e\.g\., in the case of FLORES BPB\), we likewise average across languages\.*Mixed\-effects regression*fitsDT,L∼β⋅IT,L\+\(1∣L\)D\_\{T,L\}\\sim\\beta\\cdot I\_\{T,L\}\+\(1\\mid L\)for each intrinsic metricIIand downstream metricDD, whereTTindexes tokenizers andLLindexes languages\. We reserve this analysis for metrics that provide scores per language\. The random intercept per language accounts for the fact that different languages have different baseline metric scores \(e\.g\., languages’ intrinsic difficulties differ, leading to differing baseline BPBs\) regardless of tokenizer choice\. The intrinsic metricIIis standardized \(zero mean, unit SD across tokenizers, within language\); the downstream metricDDis not\. This makes the magnitude ofβ\\betacomparable across metrics\. In plain terms,β\\betareports the change inDDassociated with a one\-SD change in the intrinsic metric, holding language constant\. As a concrete example, forβ=\+0\.01\\beta=\+0\.01: a tokenizer one SD higher on that metric is associated with FLORES BPB 0\.01 higher \(worse\), within a given language\. Alongside eachβ\\betawe report a standard error: the expected size of the estimate’s variation if the experiment were repeated\.[SectionC\.1\.2](https://arxiv.org/html/2608.18062#A3.SS1.SSS2)describes how these standard errors account for the dependence structure of the tokenizer panels\. We also report the median within\-language Spearmanρ~\\tilde\{\\rho\}across the 31 languages, as this can be more interpretable\. Allpp\-values are adjusted for multiple testing via Benjamini–Hochberg \(BH\) FDR correction\.

## 5Results

The results and following discussion focus on the intrinsic metric–downstream performance relationship\. The appendix contains material beyond the intrinsic–downstream relationship reported above: full per\-tokenizer downstream evaluations and intrinsic\-metric tables \([AppendicesE](https://arxiv.org/html/2608.18062#A5)and[D](https://arxiv.org/html/2608.18062#A4)\), a structural analysis of math\+code results \([SectionF\.3](https://arxiv.org/html/2608.18062#A6.SS3)\), and a per\-example validation of code structure metrics on five external code models \([SectionF\.4](https://arxiv.org/html/2608.18062#A6.SS4)\)\.

##### Aggregate intrinsic–downstream correlations\.

[Table2](https://arxiv.org/html/2608.18062#S5.T2)shows Spearman correlations between intrinsic metrics and downstream performance; significant, here and below, means BH\-adjustedpadj<0\.05p\_\{\\text\{adj\}\}<0\.05\. Five information\-theoretic metrics \([Section3\.1](https://arxiv.org/html/2608.18062#S3.SS1)\) are significant predictors of FLORES BPB on the trained languages, with\|ρ\|\|\\rho\|from0\.490\.49\(unigram entropy\) to0\.800\.80\(Rényi efficiency, the strongest correlation in the table\)\. Of the 14 intrinsic metrics, only digit boundaryF1F\_\{1\}significantly predicts Code BPB \(ρ=−0\.62\\rho=\-0\.62\), and only AST alignment predicts MBPP \(ρ=0\.61\\rho=0\.61\); we attribute the latter correlation to the vocabulary’s line\-break handling, which AST alignment partly reflects \([SectionF\.3](https://arxiv.org/html/2608.18062#A6.SS3)\)\. Digit boundaryF1F\_\{1\}is also the only significant BLiMP predictor \(ρ=−0\.51\\rho=\-0\.\{51\}\)\. However, the correlation’s sign coincides with an algorithm split rather than a digit\-segmentation gradient: the three UnigramLM tokenizers all have digit boundaryF1=0\.483F\_\{1\}=0\.483and BLiMP among the panel’s highest \(0\.828 to 0\.841\), while BPE RightAlign, whose place\-value pretokenization gives it the panel’s highest digit boundaryF1F\_\{1\}\(1\.0001\.000\), has BLiMP in the panel’s mid\-range \(0\.813 to 0\.816;[Tables10](https://arxiv.org/html/2608.18062#A4.T10)and[12](https://arxiv.org/html/2608.18062#A5.T12)\)\. Consistent with this, removing a single UnigramLM tokenizer pushes the digit\-boundary\-F1F\_\{1\}/BLiMP correlation just above the corrected significance threshold \([SectionF\.2](https://arxiv.org/html/2608.18062#A6.SS2)\)\.

Table 2:Aggregate Spearmanρ\\rhobetween intrinsic metrics and downstream performance \(n=29n=29\{\}for the natural\-language columns;n=19n=19\{\}for the math\+code columns\)\. Natural language intrinsic metrics are measured on the 31\-language training data subset in FLORES\+, while AST alignment and Operator isolation are measured on code and math data, respectively\.Boldindicates significance, with stars marking the \(BH\-adjusted\) thresholds:∗padj<0\.05p\_\{\\text\{adj\}\}<0\.05,∗∗padj<0\.01p\_\{\\text\{adj\}\}<0\.01,∗∗∗padj<0\.001p\_\{\\text\{adj\}\}<0\.001\.
##### Dependence on the measurement corpus\.

Repeating the analysis with intrinsic metrics measured on FineWeb\-2/\-Edu instead of FLORES\+ \([Table16](https://arxiv.org/html/2608.18062#A6.T16)\) changes the significance status of 7 of the 98 aggregate correlations\. The other 91 values keep their status; their coefficients still move, several even changing sign, but these values were not significant in either table\. Five of the seven values that do change significance status involve a metric whose value depends on the length unituu, which differs between the two tables \(u=linesu=\\texttt\{lines\}on FLORES\+,u=bytesu=\\texttt\{bytes\}on FineWeb\)\. Three of those five belong to the Gini coefficient: its correlation with FLORES\(all\) BPB is significant and negative under FLORES\+ measurement \(ρ=−0\.58\\rho=\-0\.58\) but positive and not significant under FineWeb measurement \(\+0\.38\+0\.38\)\. A reversal in the other direction happens as well: Gini becomes a significant predictor of FLORES\(trained\) BPB \(\+0\.62\+0\.62\) under FineWeb measurement while it was not under FLORES\+ measurement\. The difference in units can explain this: FineWeb is not parallel, so its Gini uses a byte\-normalized tokenization\-cost basis, which mixes per\-script byte\-length differences into the inequality measure\. The line\-normalized FLORES\+ Gini is arguably the more representative evaluation\. UTF\-8 boundary crossing and trigram entropy also become non\-significant predictors on FLORES\(trained\) BPB and MultiBLiMP accuracy, respectively, when the intrinsic metrics are measured on FineWeb data, but these metrics’ values do not depend onuu, so we do not have a good explanation\. Otherwise, the correlations discussed above also hold at the extended\-panel level and under leave\-one\-out, with two qualifications reported in[SectionF\.2](https://arxiv.org/html/2608.18062#A6.SS2): unigram entropy’s correlation with FLORES\(trained\) BPB falls just short of significance at the extended level, and removing a single tokenizer puts three of the four above the table\-wide corrected threshold\.

##### Fertility and FLORES\(all\) BPB\.

One surprising result is that fragmentation is associated with opposite directions of FLORES BPB change depending on which languages’ BPB is measured\. On the 31 trained languages, more fragmentation is associated with worse BPB: fertility’s within\-language mixed\-effects coefficient on FLORES\(trained\) BPB is\+0\.018\+0\.018BPB per SD, significant \([Table3](https://arxiv.org/html/2608.18062#S5.T3)\)\. On FLORES\(all\) BPB, computed mostly on the 184 of 215 FLORES\+ languages absent from both the tokenizer’s and the model’s training data, the association reverses: fertility’s aggregate correlation with FLORES\(all\) BPB isρ=−0\.57\\rho=\-0\.57and the Gini coefficient’s is−0\.58\-0\.58, both significant \([Table2](https://arxiv.org/html/2608.18062#S5.T2)\)\.777β\\betaandρ\\rhovalues are not directly comparable across the two analyses\.In our tokenizer ablations, this pattern repeats: restricting the tokenizer’s training languages from the 30\-language balanced mixture to the 6highreslanguages lowers FLORES\(all\) BPB by0\.02320\.0232, and switching the tokenizer’s training data from balanced to code\-heavy lowers it by0\.01530\.0153; both changes narrow the tokenizer’s vocabulary coverage of the FLORES\+ languages\. BPB is normalized by bytes, so fragmentation is not penalized by construction; the reversal instead reflects a mismatch between the vocabulary and the language being scored\. On the untrained languages, a vocabulary specialized toward the trained languages falls back to short, frequent byte\-level or near\-byte\-level fragments, and the model’s predictions there are high\-entropy but calibrated\. A broader multilingual vocabulary instead produces longer, rarer multi\-byte tokens on the same text, and the model’s conditional distribution over these tokens is fit to the trained languages they resemble, not to the language actually being scored\. Cross\-entropy penalizes a confident wrong prediction more than a hedged, uncertain one, so the specialized vocabulary’s byte\-level fallback scores lower \(better\) BPB even though it encodes the text less efficiently\. One could therefore argue that FLORES\(all\) BPB measures byte\-fallback robustness on unseen languages, not multilingual encoding quality on the trained languages\.

##### Within\-language predictors of FLORES BPB\.

When controlling for per\-language complexity with a random intercept via mixed\-effects regression, the predictiveness of several metrics changes in comparison to aggregate correlations \([Table3](https://arxiv.org/html/2608.18062#S5.T3)\)\. For per\-language FLORES BPB, the information\-theoretic metrics remain highly significant predictors, but UTF\-8 character split rate and fertility also become significant\. Fertility in particular illustrates the discrepancy that can arise between the two types of analyses\. Its within\-language coefficient is the largest in magnitude of any metric \(β=\+0\.018\\beta=\+0\.018,ρ~=\+0\.385\\tilde\{\\rho\}=\+0\.385\), yet its aggregate correlation with FLORES \(trained\) BPB in[Table2](https://arxiv.org/html/2608.18062#S5.T2)is not significant\. The likely explanation is that languages differ substantially in baseline FLORES BPB: Tamil and Bengali have far higher BPB than German or Spanish, regardless of tokenizer\. Within any given language, higher fertility reliably accompanies worse BPB, but the aggregate correlation pools this within\-language relationship together with the between\-language baseline differences, which makes the within\-language association harder to detect with onlyn=29n=29\{\}tokenizers\. Repeating this specification with per\-language MultiBLiMP accuracy as the outcome, five of the same nine metrics are significant after BH correction, each in the same direction as its FLORES BPB effect\.888Fertility’s coefficient is not significant when using clustered standard errors \([SectionC\.1\.2](https://arxiv.org/html/2608.18062#A3.SS1.SSS2)\)\.Of particular interest is that compression rate is not a significant predictor of MultiBLiMP accuracy, consistent with prior findings that compression alone is not a reliable objective for tokenizer quality\. The collective practical implication is that language, and the corpus an intrinsic metric is computed on, need to be controlled for in this kind of analysis\.

Table 3:Relationship between intrinsic tokenizer metrics \(FLORES\+\) and two per\-language downstream outcomes \(FLORES BPB and MultiBLiMP accuracy, 31 and 24 languages, respectively\), over the paper’s primary panel\.β\\beta: coefficient from mixed\-effects regression \(units: outcome units \(BPB or accuracy\) per SD of the metric\);±\\pmvalues are tokenizer\-family\-clustered standard errors \([SectionC\.1\.2](https://arxiv.org/html/2608.18062#A3.SS1.SSS2)\)\.ρ~\\tilde\{\\rho\}: median within\-language Spearman correlation\.Bold: indicates significance \(BH\-FDR on the family\-clusteredpp\-values\)\.∗padj<0\.05p\_\{\\text\{adj\}\}<0\.05,∗∗padj<0\.01p\_\{\\text\{adj\}\}<0\.01,∗∗∗padj<0\.001p\_\{\\text\{adj\}\}<0\.001\.
##### Predicting a held\-out tokenizer\.

Given a tokenizer’s intrinsic metrics, can we predict its downstream performance? To test this, we fit intrinsic\-metric predictors for each of four downstream metrics \(val BPB, FLORES \(trained\) BPB, code BPB, MBPP pass@1\) on the extended panels \(E and M\)\. Cross\-validation is leave\-one\-family\-out: each fold holds out one whole family, so no tokenizer is ever predicted by a fit trained on another member of its family \(see[SectionC\.1\.2](https://arxiv.org/html/2608.18062#A3.SS1.SSS2)\)\. We then use these fits to predict the two off\-the\-shelf reference tokenizers’ models, which were excluded from every fit\. Six of the eight resulting predictions \(four downstream metrics times two references\) fall inside the 90% prediction interval, the range calibrated on the cross\-validation errors to contain 90% of held\-out observations \([SectionF\.1](https://arxiv.org/html/2608.18062#A6.SS1)\); LLaMA\-3 falls outside it on two targets, FLORES \(trained\) BPB at\+2\.10\+2\.10standardized errors and code BPB at−2\.01\-2\.01\(prediction error divided by the cross\-validated root\-mean\-squared error\)\. Rank transfers less well than the point prediction: among the 39\-tokenizer extended panel plus the inserted reference, the val\-BPB fit places LLaMA\-3 at predicted rank 1, while its observed val BPB ranks 21st\. The practical use of these fits is therefore screening, not selection: a fit predicts a new tokenizer’s BPB to within a wide but calibrated interval before any model is trained, which is enough to discard clearly unsuitable configurations, but it does not order close candidates, so the final choice among them still requires training runs\.

## 6Discussion and Conclusion

##### What intrinsic metrics can and cannot predict\.

Our results support a division of labor among intrinsic metrics\. Information\-theoretic and compression\-based metrics are reliable for optimizing encoding efficiency: five of them are significant predictors of FLORES\(trained\) BPB \([Table2](https://arxiv.org/html/2608.18062#S5.T2)\), making them useful for narrowing the tokenizer search space before committing to pretraining\. They are not a sole proxy for task accuracy\([37](https://arxiv.org/html/2608.18062#bib.bib8);[10](https://arxiv.org/html/2608.18062#bib.bib14)\): the only significant task\-accuracy predictors in[Table2](https://arxiv.org/html/2608.18062#S5.T2)are structure\-sensitive quantities, digit boundaryF1F\_\{1\}and AST boundary alignment, and even these reflect differences between design\-axis groups \(algorithm for the BLiMP correlation, line\-break handling across pretokenizers for the MBPP correlation;[Sections5](https://arxiv.org/html/2608.18062#S5)and[F\.3](https://arxiv.org/html/2608.18062#A6.SS3.SSS0.Px1)\) rather than gradients within a single design choice\.

##### How to measure\.

There are three clear takeaways regarding protocol\. First, per\-language analysis is essential when evaluating over multilingual data: aggregate correlations can mask or even reverse within\-language relationships due to baseline differences across languages, as fertility illustrates \([Table3](https://arxiv.org/html/2608.18062#S5.T3)\); the same reasoning motivates tokenizer designs that give each language its own sub\-vocabulary\. Second, the corpus an intrinsic metric is measured on, and its normalization unit, are part of the metric’s definition: the Gini coefficient changes sign between line\-normalized FLORES\+ and byte\-normalized FineWeb measurement \([Table16](https://arxiv.org/html/2608.18062#A6.T16)\)\. Third, what a downstream metric measures depends on the evaluation population, and this must be accounted for when drawing conclusions about tokenizer impacts: FLORES\(all\) BPB, computed mostly on languages absent from training, measures byte\-fallback robustness of the language models rather than multilingual encoding quality on the trained languages\.

##### Limitations\.

Main results use a single architecture \(nanochat, decoder\-only transformer\) at 1\.27B parameters\. Tokenizer–performance relationships may differ at larger scales, where models have greater capacity to compensate for suboptimal tokenization, or under different architectures \(e\.g\., encoder\-decoder, state\-space models\)\. On the other hand, these models are likely too small for certain abilities to surface, implying our conclusions for these settings may be noisy\. Our tokenizer ablations leave many configurations untested\. The benchmark suite, though spanning perplexity, linguistic acceptability, math, and code, does not cover important capabilities like long\-context reasoning, instruction following, or open\-ended generation quality\. Further, withn=29n=29\{\}tokenizers, the aggregate correlation analysis has limited statistical power, especially for detecting moderate effect sizes\. Non\-significant correlations should be interpreted as “not detected” rather than “absent\.”

##### Conclusion\.

We introducedTokEval, an open\-source suite of intrinsic tokenizer metrics spanning text, mathematics, code, and multilingual fairness, and validated it with controlled pretraining across tokenizer configurations at two data mixtures\. Intrinsic measurement can replace pretraining sweeps where the two agree: information\-theoretic metrics screen for encoding efficiency without training a model, and the held\-out prediction analysis \([SectionF\.1](https://arxiv.org/html/2608.18062#A6.SS1)\) shows that such fits bound a new tokenizer’s BPB inside a calibrated interval but do not recover its rank among close candidates\. Task accuracy is predicted only by structural metrics targeted at the capability in question, and final selection among close candidates still requires training runs\. We hopeTokEvalmakes both steps, the intrinsic screening and the targeted structural checks, standard practice in tokenizer development\.

## Acknowledgments

We thank Tiago Pimentel, Amit Moryossef, and Craig Schmidt for discussions on the metrics presented in this paper, and Tiago Pimentel additionally for detailed feedback on the manuscript\. We thank Sander Land for feedback on the design of theTokEvallibrary and for several design improvements to the codebase\. The tokenizer language\-modeling ablations reported here were carried out as part of Apertus 2 tokenizer development, with compute provided by the Swiss AI Initiative\.

## Ethics Statement

This work is partly motivated by the observation that tokenization choices disproportionately affect low\-resource languages and non\-Latin scripts, leading to higher inference costs and degraded performance on already underserved communities\([33](https://arxiv.org/html/2608.18062#bib.bib6);[1](https://arxiv.org/html/2608.18062#bib.bib42)\)\. By providing tools to measure and compare these disparities \(e\.g\., the tokenizer fairness Gini coefficient and per\-language fertility\), we hopeTokEvalcan help practitioners identify and mitigate such inequities early in model development\. We note, however, that our multilingual coverage \(31 languages across 11 scripts\) still excludes the vast majority of the world’s languages, and our fairness metrics capture only tokenization\-level costs, not directly differences in models’ cross\-lingual abilities\.

## References

- Ahiaet al\.\(2023\)O\. Ahia, S\. Kumar, H\. Gonen, J\. Kasai, D\. Mortensen, N\. Smith, and Y\. TsvetkovDo all languages cost the same? tokenization in the era of commercial language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 9904–9923\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.614/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.614)Cited by:[§2](https://arxiv.org/html/2608.18062#S2.SS0.SSS0.Px2.p1.1),[Ethics Statement](https://arxiv.org/html/2608.18062#Sx2.p1.1)\.
- Aliet al\.\(2024\)M\. Ali, M\. Fromm, K\. Thellmann, R\. Rutmann, M\. Lübbering, J\. Leveling, K\. Klug, J\. Ebert, N\. Doll, J\. Buschhoff, C\. Jain, A\. Weber, L\. Jurkschat, H\. Abdelwahab, C\. John, P\. Ortiz Suarez, M\. Ostendorff, S\. Weinbach, R\. Sifa, S\. Kesselheim, and N\. Flores\-HerrTokenizer choice for LLM training: negligible or crucial?\.InFindings of the Association for Computational Linguistics: NAACL 2024,K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 3907–3924\.External Links:[Link](https://aclanthology.org/2024.findings-naacl.247/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-naacl.247)Cited by:[§1](https://arxiv.org/html/2608.18062#S1.p1.1),[§2](https://arxiv.org/html/2608.18062#S2.SS0.SSS0.Px2.p1.1)\.
- Arnett and Bergen \(2025\)C\. Arnett and B\. BergenWhy do language models perform worse for morphologically complex languages?\.InProceedings of the 31st International Conference on Computational Linguistics,Abu Dhabi, UAE,pp\. 6607–6623\.Cited by:[§2](https://arxiv.org/html/2608.18062#S2.SS0.SSS0.Px3.p1.1),[§3\.2](https://arxiv.org/html/2608.18062#S3.SS2.p1.1)\.
- Arnettet al\.\(2025a\)C\. Arnett, T\. A\. Chang, S\. Biderman, and B\. BergenExplaining and mitigating crosslingual tokenizer inequities\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=0Y7AxxNCYh)Cited by:[§4\.1](https://arxiv.org/html/2608.18062#S4.SS1.SSS0.Px2.p1.1)\.
- Arnettet al\.\(2025b\)C\. Arnett, M\. Hudspeth, and B\. O’ConnorEvaluating morphological alignment of tokenizers in 70 languages\.InProceedings of the ICML 2025 Tokenization Workshop \(TokShop\),External Links:[Link](https://arxiv.org/abs/2507.06378)Cited by:[§3\.2](https://arxiv.org/html/2608.18062#S3.SS2.p1.1)\.
- Austinet al\.\(2021\)J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le, and C\. SuttonProgram synthesis with large language models\.arXiv preprint arXiv:2108\.07732\.Cited by:[§4\.3](https://arxiv.org/html/2608.18062#S4.SS3.SSS0.Px1.p1.1)\.
- Chenet al\.\(2021\)M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. d\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman, A\. Ray, R\. Puri, G\. Krueger, M\. Petrov, H\. Khlaaf, G\. Sastry, P\. Mishkin, B\. Chan, S\. Gray, N\. Ryder, M\. Pavlov, A\. Power, L\. Kaiser, M\. Bavarian, C\. Winter, P\. Tillet, F\. P\. Ribeiro, X\. Yuan, W\. Zaremba, and I\. SutskeverEvaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[§4\.3](https://arxiv.org/html/2608.18062#S4.SS3.SSS0.Px1.p1.1)\.
- Chizhovet al\.\(2024\)P\. Chizhov, C\. Arnett, E\. Korotkova, and I\. P\. YamshchikovBPE gets picky: efficient vocabulary refinement during tokenizer training\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 16587–16604\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.925/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.925)Cited by:[§2](https://arxiv.org/html/2608.18062#S2.SS0.SSS0.Px1.p1.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. SchulmanTraining verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§4\.3](https://arxiv.org/html/2608.18062#S4.SS3.SSS0.Px1.p1.1)\.
- Cognettaet al\.\(2024\)M\. Cognetta, V\. Zouhar, S\. Moon, and N\. OkazakiTwo counterexamples to tokenization and the noiseless channel\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),Torino, Italia,pp\. 16897–16906\.External Links:[Link](https://aclanthology.org/2024.lrec-main.1469/)Cited by:[§1](https://arxiv.org/html/2608.18062#S1.p2.1),[§2](https://arxiv.org/html/2608.18062#S2.SS0.SSS0.Px3.p1.1),[§6](https://arxiv.org/html/2608.18062#S6.SS0.SSS0.Px1.p1.1)\.
- Dao \(2024\)T\. DaoFlashAttention\-2: faster attention with better parallelism and work partitioning\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§B\.1](https://arxiv.org/html/2608.18062#A2.SS1.p1.1)\.
- Foroutanet al\.\(2026\)N\. Foroutan, C\. Meister, D\. Paul, J\. Niklaus, S\. Ahmadi, A\. Bosselut, and R\. SennrichParity\-aware byte\-pair encoding: improving cross\-lingual fairness in tokenization\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 7514–7538\.External Links:[Link](https://aclanthology.org/2026.acl-long.342/),[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.342),ISBN 979\-8\-89176\-390\-6Cited by:[§4\.1](https://arxiv.org/html/2608.18062#S4.SS1.SSS0.Px1.p1.1)\.
- Gaoet al\.\(2023\)A framework for few\-shot language model evaluationExternal Links:[Link](https://github.com/EleutherAI/lm-evaluation-harness)Cited by:[§C\.1](https://arxiv.org/html/2608.18062#A3.SS1.SSS0.Px2.p1.1)\.
- Jordanet al\.\(2024\)K\. Jordan, Y\. Jin, V\. Boza, Y\. Jiacheng, F\. Cesista, L\. Newhouse, and J\. BernsteinMuon: an optimizer for hidden layers in neural networks\.Note:GitHub repository/blog post\.External Links:[Link](https://kellerjordan.github.io/posts/muon/)Cited by:[§B\.2](https://arxiv.org/html/2608.18062#A2.SS2.SSS0.Px2.p1.1)\.
- Jumeletet al\.\(2026\)J\. Jumelet, L\. Weissweiler, J\. Nivre, and A\. BisazzaMultiBLiMP 1\.0: a massively multilingual benchmark of linguistic minimal pairs\.Transactions of the Association for Computational Linguistics14,pp\. 193–216\.External Links:[Link](https://aclanthology.org/2026.tacl-1.10/),[Document](https://dx.doi.org/10.1162/tacl.a.600)Cited by:[§4\.3](https://arxiv.org/html/2608.18062#S4.SS3.SSS0.Px1.p1.1)\.
- Kanjirangatet al\.\(2025\)V\. Kanjirangat, T\. Samardzic, L\. Dolamic, and F\. RinaldiTokenization and representation biases in multilingual models on dialectal NLP tasks\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 23992–24010\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.1224/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1224),ISBN 979\-8\-89176\-332\-6Cited by:[§2](https://arxiv.org/html/2608.18062#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2608.18062#S2.SS0.SSS0.Px3.p1.1)\.
- Karpathy \(2025\)A\. KarpathyNanochat\.Note:Open\-source repository and model release\.External Links:[Link](https://github.com/karpathy/nanochat)Cited by:[§B\.1](https://arxiv.org/html/2608.18062#A2.SS1.p1.1),[§4\.2](https://arxiv.org/html/2608.18062#S4.SS2.p1.1)\.
- Kudo and Richardson \(2018\)T\. Kudo and J\. RichardsonSentencePiece: a simple and language independent subword tokenizer and detokenizer for neural text processing\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,Brussels, Belgium,pp\. 66–71\.External Links:[Link](https://aclanthology.org/D18-2012/),[Document](https://dx.doi.org/10.18653/v1/D18-2012)Cited by:[§3](https://arxiv.org/html/2608.18062#S3.p2.1)\.
- Kudo \(2018\)T\. KudoSubword regularization: improving neural network translation models with multiple subword candidates\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Melbourne, Australia,pp\. 66–75\.External Links:[Link](https://aclanthology.org/P18-1007/),[Document](https://dx.doi.org/10.18653/v1/P18-1007)Cited by:[§1](https://arxiv.org/html/2608.18062#S1.p1.1),[§2](https://arxiv.org/html/2608.18062#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2608.18062#S4.SS1.SSS0.Px1.p1.1)\.
- Land and Arnett \(2025\)S\. Land and C\. ArnettBPE stays on SCRIPT: structured encoding for robust multilingual pretokenization\.External Links:2505\.24689,[Link](https://arxiv.org/abs/2505.24689)Cited by:[§3](https://arxiv.org/html/2608.18062#S3.p2.1),[§4\.1](https://arxiv.org/html/2608.18062#S4.SS1.SSS0.Px2.p1.1)\.
- Land \(2026\)S\. LandMinGram: a minimalist unigram tokenizer with high compression and competitive morphological alignment\.External Links:2606\.27019,[Link](https://arxiv.org/abs/2606.27019)Cited by:[§4\.1](https://arxiv.org/html/2608.18062#S4.SS1.SSS0.Px1.p1.1)\.
- Lesciet al\.\(2025\)P\. Lesci, C\. Meister, T\. Hofmann, A\. Vlachos, and T\. PimentelCausal estimation of tokenisation bias\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 28325–28340\.External Links:[Link](https://aclanthology.org/2025.acl-long.1374/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1374),ISBN 979\-8\-89176\-251\-0Cited by:[§1](https://arxiv.org/html/2608.18062#S1.p1.1),[§2](https://arxiv.org/html/2608.18062#S2.SS0.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2608.18062#S3.SS2.p1.1)\.
- Liet al\.\(2023\)R\. Li, L\. B\. Allal, Y\. Zi, N\. Muennighoff, D\. Kocetkov, C\. Mou, M\. Marone, C\. Akiki, J\. Li, J\. Chim,et al\.StarCoder: may the source be with you\!\.arXiv preprint arXiv:2305\.06161\.Cited by:[§4\.1](https://arxiv.org/html/2608.18062#S4.SS1.SSS0.Px3.p1.1)\.
- Limisiewiczet al\.\(2023\)T\. Limisiewicz, J\. Balhar, and D\. MarečekTokenization impacts multilingual language modeling: assessing vocabulary allocation and overlap across languages\.InFindings of the Association for Computational Linguistics: ACL 2023,A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 5661–5681\.External Links:[Link](https://aclanthology.org/2023.findings-acl.350/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-acl.350)Cited by:[§3\.1](https://arxiv.org/html/2608.18062#S3.SS1.p1.1)\.
- Liuet al\.\(2025\)A\. Liu, J\. Hayase, V\. Hofmann, S\. Oh, N\. A\. Smith, and Y\. ChoiSuperBPE: space travel for language models\.InSecond Conference on Language Modeling \(COLM 2025\),External Links:[Link](https://arxiv.org/abs/2503.13423)Cited by:[§1](https://arxiv.org/html/2608.18062#S1.p1.1),[§2](https://arxiv.org/html/2608.18062#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2608.18062#S4.SS1.SSS0.Px1.p1.1)\.
- Lotzet al\.\(2025\)J\. F\. Lotz, A\. V\. Lopes, S\. Peitz, H\. Setiawan, and L\. EmiliBeyond text compression: evaluating tokenizers across scales\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 32155–32173\.External Links:[Link](https://aclanthology.org/2025.acl-long.1546/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1546),ISBN 979\-8\-89176\-251\-0Cited by:[§2](https://arxiv.org/html/2608.18062#S2.SS0.SSS0.Px3.p1.1)\.
- Lozhkovet al\.\(2024\)A\. Lozhkov, L\. Ben Allal, E\. Bakouch, L\. von Werra, and T\. WolfFineMath: the finest collection of mathematical content\.External Links:[Link](https://huggingface.co/datasets/HuggingFaceTB/finemath)Cited by:[§4\.1](https://arxiv.org/html/2608.18062#S4.SS1.SSS0.Px3.p1.1)\.
- Lundinet al\.\(2026\)J\. M\. Lundin, A\. Zhang, N\. Karim, H\. Louzan, G\. Wei, D\. I\. Adelani, and C\. CarrollThe token tax: systematic bias in multilingual tokenization\.InProceedings of the 7th Workshop on African Natural Language Processing \(AfricaNLP 2026\),E\. A\. Chimoto, C\. Lignos, S\. Muhammad, I\. Abdulmumin, C\. Siro, and D\. I\. Adelani \(Eds\.\),Rabat, Morocco,pp\. 103–112\.External Links:[Link](https://aclanthology.org/2026.africanlp-main.10/),[Document](https://dx.doi.org/10.18653/v1/2026.africanlp-main.10),ISBN 979\-8\-89176\-364\-7Cited by:[§2](https://arxiv.org/html/2608.18062#S2.SS0.SSS0.Px2.p1.1)\.
- NLLB Teamet al\.\(2024\)NLLB Team, M\. R\. Costa\-jussà, J\. Cross, O\. Çelebi, M\. Elbayad, K\. Heafield, K\. Heffernan, E\. Kalbassi, J\. Lam, D\. Licht, J\. Maillard, A\. Sun, S\. Wang, G\. Wenzek, A\. Youngblood, B\. Akula, L\. Barrault, G\. M\. Gonzalez, P\. Hansanti, J\. Hoffman, S\. Jarrett, K\. R\. Sadagopan, D\. Rowe, S\. Spruit, C\. Tran, P\. Andrews, N\. F\. Ayan, S\. Bhosale, S\. Edunov, A\. Fan, C\. Gao, V\. Goswami, F\. Guzmán, P\. Koehn, A\. Mourachko, C\. Ropers, S\. Saleem, H\. Schwenk, and J\. WangScaling neural machine translation to 200 languages\.Nature630\(8018\),pp\. 841–846\.External Links:ISSN 1476\-4687,[Document](https://dx.doi.org/10.1038/s41586-024-07335-x),[Link](https://doi.org/10.1038/s41586-024-07335-x)Cited by:[§4\.3](https://arxiv.org/html/2608.18062#S4.SS3.SSS0.Px1.p1.1)\.
- Pawaret al\.\(2025\)S\. Pawar, M\. Apte, K\. Jadhav, G\. K\. Palshikar, and N\. RamrakhiyaniBroken words, broken performance: effect of tokenization on performance of LLMs\.InProceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia\-Pacific Chapter of the Association for Computational Linguistics,K\. Inui, S\. Sakti, H\. Wang, D\. F\. Wong, P\. Bhattacharyya, B\. Banerjee, A\. Ekbal, T\. Chakraborty, and D\. P\. Singh \(Eds\.\),Mumbai, India,pp\. 372–385\.External Links:[Link](https://aclanthology.org/2025.ijcnlp-short.31/),[Document](https://dx.doi.org/10.18653/v1/2025.ijcnlp-short.31),ISBN 979\-8\-89176\-299\-2Cited by:[§2](https://arxiv.org/html/2608.18062#S2.SS0.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2608.18062#S3.SS2.p1.1)\.
- Penedoet al\.\(2024\)G\. Penedo, H\. Kydlícek, L\. B\. Allal, A\. Lozhkov, M\. Mitchell, C\. Raffel, L\. von Werra, and T\. WolfThe fineweb datasets: decanting the web for the finest text data at scale\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§4\.1](https://arxiv.org/html/2608.18062#S4.SS1.SSS0.Px3.p1.1)\.
- Penedoet al\.\(2025\)G\. Penedo, H\. Kydlíček, V\. Sabolčec, B\. Messmer, N\. Foroutan, A\. H\. Kargaran, C\. Raffel, M\. Jaggi, L\. V\. Werra, and T\. WolfFineWeb2: one pipeline to scale them all – adapting pre\-training data processing to every language\.External Links:2506\.20920,[Link](https://arxiv.org/abs/2506.20920)Cited by:[§4\.1](https://arxiv.org/html/2608.18062#S4.SS1.SSS0.Px3.p1.1)\.
- Petrovet al\.\(2023\)A\. Petrov, E\. La Malfa, P\. Torr, and A\. BibiLanguage model tokenizers introduce unfairness between languages\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 36963–36990\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/74bb24dca8334adce292883b4b651eda-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2608.18062#S2.SS0.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2608.18062#S3.SS2.p1.1),[Ethics Statement](https://arxiv.org/html/2608.18062#Sx2.p1.1)\.
- Poelmanet al\.\(2025\)W\. Poelman, T\. Bauwens, and M\. de LhoneuxConfounding factors in relating model performance to morphology\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,External Links:[Link](https://arxiv.org/abs/2511.01380)Cited by:[§3\.1](https://arxiv.org/html/2608.18062#S3.SS1.p1.1),[footnote 3](https://arxiv.org/html/2608.18062#footnote3)\.
- Rustet al\.\(2021\)P\. Rust, J\. Pfeiffer, I\. Vulić, S\. Ruder, and I\. GurevychHow good is your tokenizer? On the monolingual performance of multilingual language models\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),Online,pp\. 3118–3135\.External Links:[Link](https://aclanthology.org/2021.acl-long.243/),[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.243)Cited by:[§1](https://arxiv.org/html/2608.18062#S1.p1.1),[§1](https://arxiv.org/html/2608.18062#S1.p2.1),[§2](https://arxiv.org/html/2608.18062#S2.SS0.SSS0.Px2.p1.1)\.
- Schmidtet al\.\(2025\)C\. W\. Schmidt, V\. Reddy, C\. Tanner, and Y\. PinterBoundless byte pair encoding: breaking the pre\-tokenization barrier\.InSecond Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=oPAjXGV8qQ)Cited by:[§2](https://arxiv.org/html/2608.18062#S2.SS0.SSS0.Px1.p1.1)\.
- Schmidtet al\.\(2024\)C\. W\. Schmidt, V\. Reddy, H\. Zhang, A\. Alameddine, O\. Uzan, Y\. Pinter, and C\. TannerTokenization is more than compression\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Miami, Florida, USA,pp\. 678–702\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.40/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.40)Cited by:[§1](https://arxiv.org/html/2608.18062#S1.p2.1),[§2](https://arxiv.org/html/2608.18062#S2.SS0.SSS0.Px3.p1.1),[§6](https://arxiv.org/html/2608.18062#S6.SS0.SSS0.Px1.p1.1)\.
- Sennrichet al\.\(2016\)R\. Sennrich, B\. Haddow, and A\. BirchNeural machine translation of rare words with subword units\.InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Berlin, Germany,pp\. 1715–1725\.External Links:[Link](https://aclanthology.org/P16-1162/),[Document](https://dx.doi.org/10.18653/v1/P16-1162)Cited by:[§1](https://arxiv.org/html/2608.18062#S1.p1.1),[§2](https://arxiv.org/html/2608.18062#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2608.18062#S4.SS1.SSS0.Px1.p1.1)\.
- Singh and Strouse \(2024\)A\. K\. Singh and D\. StrouseTokenization counts: the impact of tokenization on arithmetic in frontier LLMs\.arXiv preprint arXiv:2402\.14903\.Cited by:[§1](https://arxiv.org/html/2608.18062#S1.p1.1),[§2](https://arxiv.org/html/2608.18062#S2.SS0.SSS0.Px2.p1.1),[§3\.5](https://arxiv.org/html/2608.18062#S3.SS5.p1.1)\.
- Soleret al\.\(2024\)A\. G\. Soler, M\. Labeau, and C\. ClavelThe impact of word splitting on the semantic content of contextualized word representations\.Transactions of the Association for Computational Linguistics12,pp\. 299–320\.External Links:[Link](https://aclanthology.org/2024.tacl-1.17/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00647)Cited by:[§2](https://arxiv.org/html/2608.18062#S2.SS0.SSS0.Px3.p1.1)\.
- Suet al\.\(2024\)J\. Su, M\. Ahmed, Y\. Lu, S\. Pan, W\. Bo, and Y\. LiuRoFormer: enhanced transformer with rotary position embedding\.Neurocomputing568,pp\. 127063\.External Links:ISSN 0925\-2312,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.neucom.2023.127063),[Link](https://www.sciencedirect.com/science/article/pii/S0925231223011864)Cited by:[§B\.1](https://arxiv.org/html/2608.18062#A2.SS1.p1.1)\.
- Tayet al\.\(2022\)Y\. Tay, V\. Q\. Tran, S\. Ruder, J\. Gupta, H\. W\. Chung, D\. Bahri, Z\. Qin, S\. Baumgartner, C\. Yu, and D\. MetzlerCharformer: fast character transformers via gradient\-based subword tokenization\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=JtBRnrlOEFN)Cited by:[§2](https://arxiv.org/html/2608.18062#S2.SS0.SSS0.Px1.p1.1)\.
- Warstadtet al\.\(2020\)A\. Warstadt, A\. Parrish, H\. Liu, A\. Mohananey, W\. Peng, S\. Wang, and S\. R\. BowmanBLiMP: the benchmark of linguistic minimal pairs for english\.Transactions of the Association for Computational Linguistics8,pp\. 377–392\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00321)Cited by:[§4\.3](https://arxiv.org/html/2608.18062#S4.SS3.SSS0.Px1.p1.1)\.
- Xueet al\.\(2022\)L\. Xue, A\. Barua, N\. Constant, R\. Al\-Rfou, S\. Narang, M\. Kale, A\. Roberts, and C\. RaffelByT5: towards a token\-free future with pre\-trained byte\-to\-byte models\.Transactions of the Association for Computational Linguistics10,pp\. 291–306\.External Links:[Link](https://aclanthology.org/2022.tacl-1.17/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00461)Cited by:[§2](https://arxiv.org/html/2608.18062#S2.SS0.SSS0.Px1.p1.1)\.
- Yanget al\.\(2021\)G\. Yang, E\. Hu, I\. Babuschkin, S\. Sidor, X\. Liu, D\. Farhi, N\. Ryder, J\. Pachocki, W\. Chen, and J\. GaoTuning large neural networks via zero\-shot hyperparameter transfer\.InAdvances in Neural Information Processing Systems,M\. Ranzato, A\. Beygelzimer, Y\. Dauphin, P\.S\. Liang, and J\. W\. Vaughan \(Eds\.\),Vol\.34,pp\. 17084–17097\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2021/file/8df7c2e3c3c3be098ef7b382bd2c37ba-Paper.pdf)Cited by:[§B\.2](https://arxiv.org/html/2608.18062#A2.SS2.SSS0.Px2.p1.1)\.
- Zouharet al\.\(2023\)V\. Zouhar, C\. Meister, J\. Gastaldi, L\. Du, M\. Sachan, and R\. CotterellTokenization and the noiseless channel\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Toronto, Canada,pp\. 5184–5207\.External Links:[Link](https://aclanthology.org/2023.acl-long.284/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.284)Cited by:[§1](https://arxiv.org/html/2608.18062#S1.p2.1),[§3\.1](https://arxiv.org/html/2608.18062#S3.SS1.p1.1)\.

## Appendix ATokenizer Training and Configuration Details

### A\.1Pretokenization Configurations

Before the tokenization algorithm \(BPE, UnigramLM, etc\.\) is applied, raw text is split into coarse segments called*pretokens*using a deterministic rule\. The tokenization algorithm then operates independently within each pretoken: merges \(in BPE\) or segmentation candidates \(in UnigramLM\) cannot cross pretoken boundaries\. Pretokenization thus defines the maximum granularity of the learned vocabulary; a token can be at most one pretoken long\. We look at several pretokenization strategies in this work, described below\.

##### Punctuation\.

This pretokenizer is defined using the HuggingFace pretokenizer class\. It is instantiated as:Sequence\(\[Punctuation\(behavior="Isolated"\), ByteLevel\(use\_regex=True\)\]\)\. It isolates punctuation characters, otherwise relying on the byte\-level encoder’s default regex \(GPT\-2\) for other split points\.

##### GPT\-4o\.

CamelCase\-aware splitting, left\-aligned 3\-digit groups \(`\\p\{N\}\{1,3\}`\), English contractions \(`’s`,`’t`,`’re`, etc\.\), preceding\-punctuation attachment\.

##### Claude\.

Differs from GPT\-4o in: whitespace separated by type \(spaces, tabs, newlines independently\), no punctuation attachment to preceding words, supports both straight and curly apostrophes\. We note that this is an approximation of the Claude pretokenization regex, derived using token counts from the Claude API\. We do not know definitively the pretokenization choices that Claude uses\.

##### Right\-aligned digits\.

Pretokenizes digit spans to groups of three according to place values \(e\.g\., “1234567”→\\to\[“1”, “234”, “567”\]\)\. Identical to GPT\-4o except the digit group regex uses a lookahead for place\-value alignment:`\\p\{N\}\{1,3\}\(?=\(?:\\p\{N\}\{3\}\)\*\(?:\\P\{N\}`$\)\)\|\. This ensures “123456” tokenizes as \[“123”, “456”\] \(thousands\-aligned\) rather than left\-aligned \[“12”, “345”, “6”\]\.

##### Clean, plus2, plus3\.

Cleanis a fifth, multilingual\-oriented regex, distinct from the four above: it does not split camelCase, does not explicitly attach contractions, and treats each digit as its own pretoken rather than grouping digits into runs\.plus2extendscleanby allowing leading apostrophes \(ASCII or curly\) to attach to the start of a pretoken, targeting French/Italian/Catalan contractions \(e\.g\.,*l’arbre*\);plus3additionally attaches a trailing apostrophe not followed by a letter, targeting Maltese*ta’*\.

### A\.2Training Data: Ablation Configurations

The intermediate\-language\-coverage ablations restrict the tokenizer’s training data to 5 of the 30 multilingual languages plus English \(highres: rus, spa, deu, fra, cmn, plus English; 6 languages total\) or 20 of the 30 plus English \(highmid: those same 5highreslanguages plus 15 mid\-resource languages, dropping the 10 lowest\-resource languages, plus English; 21 languages total\), so the language model still sees all 30 languages during training but the tokenizer never saw the omitted ones\. This meant to test impact on models’ generalization to languages absent from tokenizer training\. We also train a set of 5 tokenizers that weight the 30 languages equally rather than proportionally, one of which omits the repeat\-sampling the other four use to fill low\-resource tails\.

### A\.3Full Tokenizer Panel Specification

Table 4:The four tokenizer panels\. A tokenizer family groups configurations that share all three design\-axis values and differ only in a training hyperparameter or preset; the Fam\. column of[Table5](https://arxiv.org/html/2608.18062#A1.T5)lists each row’s family\.Table 5:Configurations for every tokenizer named in this paper: the 44 we train plus the 2 off\-the\-shelf references, the same 46 rows as[Table12](https://arxiv.org/html/2608.18062#A5.T12)\.Pretok: P=Punctuation, G=GPT\-4o, G2=GPT\-2, C=Claude, R=Right\-aligned, Cl=Clean\-multi, S=SCRIPT\-encoding, W=whitespace only\.Data: B=Balanced, E=English, Co=Code, Hr=Highres \(6 langs\), Hm=Highmid \(21 langs\), Eq=Equal\-weighted multilingual\.Panels: P=primary std\-1B panel member \(P, implies membership in the extended std\-1B panel too, so P is printed alone\), E=extended std\-1B panel member not in the primary panel \(E\), M=math\+code panel member \(M\), R=off\-the\-shelf reference \(R, never a panel member\), –=not a member of any panel; a row can show both a std\-1B marker and M\.Fam\.: the family index of[SectionC\.1\.1](https://arxiv.org/html/2608.18062#A3.SS1.SSS1); rows sharing an index are members of one family, and the 31 indices partition the 39\-member extended panel\. Rows outside that panel have no family\.Note: an axis not captured by the other columns, e\.g\. the SuperBPE stage\-1 base, or a property that disqualifies the row from panel membership\. The three English\-only builds marked “incomplete byte alphabet” represent 224 of 256 byte values \([SectionC\.1\.1](https://arxiv.org/html/2608.18062#A3.SS1.SSS1)\)\. BPE Claude \(eng\.\) shows M alone: it is excluded from the std\-1B panels for that defect but kept in the math\+code panel, with the defect flagged rather than dropped, because no defect\-free tokenizer for that configuration has a math\+code run at this scale \([Table18](https://arxiv.org/html/2608.18062#A6.T18)\)\.TokenizerAlgorithmPretokDataNFCPanelsFam\.NoteMistral\-NemoBPEG––R–LLaMA\-3BPE–––R–BPE Punct \(bal\.\)BPEPB✗P, M1BPE Punct \(eng\.\)BPEPE✗P, M2incomplete byte alphabetBPE GPT\-4o \(bal\.\)BPEGB✗P, M3BPE GPT\-4o NFCBPEGB✓P, M4BPE GPT\-4o \(eng\.\)BPEGE✗––incomplete byte alphabetBPE GPT\-4o \(eng\., full\-byte\)BPEGE✗P, M5full byte alphabetBPE GPT\-4o \(code\)BPEGCo✗P, M6BPE Claude \(bal\.\)BPECB✗P, M7BPE Claude NFCBPECB✓P, M8BPE Claude \(eng\.\)BPECE✗M–incomplete byte alphabetBPE RightAlign \(bal\.\)BPERB✗P, M9BPE RightAlign NFCBPERB✓P, M10Unigram GPT\-4oUnigramGB✗P, M11Unigram ClaudeUnigramCB✗P, M12Unigram RightAlignUnigramRB✗P, M13SuperBPE GPT\-4o \(bal\.\)SuperBPEGB✗P, M14PA\-BPE NFC GPT\-4oPA\-BPEGB✓P, M15Unigram GPT\-4o \(highmid\)UnigramGHm✗P1621 langsUnigram GPT\-4o \(highres\)UnigramGHr✗P176 langsMinGram SCRIPT\-encMinGramSB✓P, M18SCRIPT enc\.MinGram SCRIPT\-enc \(nl\-split\)MinGramSB✓E, M18SCRIPT enc\., splits line breaksBPE SCRIPT\-enc GPT\-4oBPESB✓P, M19SCRIPT enc\., GPT\-4o regexBPE GPT\-2 \(bal\.\)BPEG2B✗P20BPE GPT\-4o \(all\-eq\.\)BPEGEq✗P21BPE Claude \(all\-eq\.\)BPECEq✗P22BPE Punct \(all\-eq\.\)BPEPEq✗P23BPE GPT\-4o NFC \(all\-eq\.\)BPEGEq✓P24BPE GPT\-4o \(all\-eq\., no\-rep\.\)BPEGEq✗E21no repeat\-samplingBPE GPT\-4o \(bal\., mf50\)BPEGB✗––raised file cap, null controlBPE NFC clean \(bal\.\)BPEClB✓P, M25BPE NFC clean plus2 \(bal\.\)BPEClB✓E, M25plus2 apostropheBPE NFC clean plus3 \(bal\.\)BPEClB✓E, M25plus3 apostropheSuperBPE PA\-BPE GPT\-4oSuperBPEGB✓E26PA\-BPE baseSuperBPE PA\-BPE GPT\-4o t64kSuperBPEGB✓E26PA\-BPE baseSuperBPE clean C2 \(bal\.\)SuperBPEClB✓P, M27PA\-BPE base, C2SuperBPE clean C3 \(bal\.\)SuperBPEClB✓E, M27PA\-BPE base, C3SuperBPE Punct t64kSuperBPEPB✗E28PA\-BPE NFC GPT\-4o HWPA\-BPEGB✓E15hybrid windowPA\-BPE NFC cleanPA\-BPEClB✓P29Unigram GPT\-4o tunedUnigramGB✗E11tuned hyperparamsBPE GPT\-4o \(highres\)BPEGHr✗P306 langsBPE GPT\-4o \(highmid\)BPEGHm✗P3121 langsBPE Whitespace \(bal\.\)BPEWB✗––controlUnigram Whitespace tuned \(bal\.\)UnigramWB✗––control, SentencePiece, tuned

## Appendix BModel Architecture and Training Details

### B\.1Architecture

The model uses the nanochat d24 configuration\([17](https://arxiv.org/html/2608.18062#bib.bib26)\): 24 layers, hidden dimension 1536, 12 attention heads \(head dimension 128\), multi\-head attention, context length 2048\. Position information uses RoPE\([41](https://arxiv.org/html/2608.18062#bib.bib27)\)with base frequencyθ=100,000\\theta=100\{,\}000\. Activations are squared ReLU \(ReLU2\\text\{ReLU\}^\{2\}\)\. Normalization is parameterless RMSNorm\. QK\-norm applies a scale of 1\.2 to both queries and keys\. Logits are soft\-capped:15⋅tanh⁡\(logits/15\)15\\cdot\\tanh\(\\text\{logits\}/15\)\. Attention uses Flash Attention\([11](https://arxiv.org/html/2608.18062#bib.bib12)\)on Hopper GPUs with PyTorch SDPA as fallback\.

The sliding window pattern “SSSL” tiles across layers: three layers attend to⌈L/4⌉\\lceil L/4\\rceiltokens \(rounded to the Flash Attention tile size\), followed by one full\-context layer\. Alternating layers include value embeddings: per\-token learned vectors of dimension 128, projected to the full key\-value dimension \(1536\) via a learned linear layer, added to the value tensor through an input\-dependent gate\. Per\-layer learnable scalars control residual stream scaling and skip\-connection blending\.

##### Parameter counts\.

All models use a∼\\sim128K vocabulary \(Mistral\-Nemo differs slightly in vocab size at 131K, although∼\\sim1K of these are special tokens\)\. Under 128K vocabulary, the parameter counts for our architectures are: 682M for transformer matrices, 197M for input embeddings, 197M output projection, 197M for value embeddings \(12 layers×\\times128K×\\times128\), 74 scalar parameters\. This gives us a total of 1\.27B parameters\. Embedding parameters scale with vocabulary; of the two off\-the\-shelf tokenizers, LLaMA\-3 matches the standard 128,256\-token vocabulary exactly \(checked directly against the trained model weights: 0 parameter difference\), while Mistral\-Nemo’s larger 131,072\-token vocabulary adds 12,976,128 parameters, 1\.0% of the 1\.27B total\.

### B\.2Training Hyperparameters

We follow the training recommendations of the nanochat library\. We provide a summary of training details below, but refer the reader to the repository for further details and hyperparameter choice justifications\. Training is done on 4 NVIDIA GH200 GPUs inbfloat16\.

##### Training budget\.

For the natural\-language\-focused models, we train on10\.5×Nscale10\.5\\times N\_\{\\text\{scale\}\}tokens whereNscale=N\_\{\\text\{scale\}\}=transformer matrices\+\+output projection=879=879M\. This ratio is taken directly from nanochat’s scaling law analysis with Kaplan\-style parameter counting \(N∝C0\.54N\\propto C^\{0\.54\},D∝C0\.49D\\propto C^\{0\.49\}\)\.999See[https://github\.com/karpathy/nanochat/blob/master/dev/LOG\.md](https://github.com/karpathy/nanochat/blob/master/dev/LOG.md)This gives a target of 9\.23B tokens \(8,800 steps at batch size220=1,048,5762^\{20\}=1\{,\}048\{,\}576\)\. This budget was held fixed rather than extended, based on a check on five tokenizers \(four custom and LLaMA\-3\) whose models we continued training for a further 10B tokens on the same mixture, roughly twice the compute\-optimal budget: val BPB improved for all five with the cross\-tokenizer ranking preserved, but the math and code generation benchmarks stayed near floor \(MBPP pass@1 at most0\.0700\.070across the five continued models\) with unstable cross\-tokenizer rankings\. This motivated the separate, dedicated training of the math\+code models\.

##### Optimizer\.

Weight matrices use the Muon optimizer\([14](https://arxiv.org/html/2608.18062#bib.bib25)\); embeddings, projections and scalars use AdamW\. Learning rates are transferred across widths viaμ\\muP\([45](https://arxiv.org/html/2608.18062#bib.bib30)\)\. Base learning rates: 0\.02 \(Muon\), 0\.30 \(input embeddings\), 0\.008 \(output projection\), 0\.50 \(scalar parameters\)\. AdamW LRs scaled by\(dmodel/768\)−1=0\.5\(d\_\{\\text\{model\}\}/768\)^\{\-1\}=0\.5\(μ\\muP\([45](https://arxiv.org/html/2608.18062#bib.bib30)\)\)\. All LRs scaled byB/Bref=2≈1\.414\\sqrt\{B/B\_\{\\text\{ref\}\}\}=\\sqrt\{2\}\\approx 1\.414whereBref=524,288B\_\{\\text\{ref\}\}=524\{,\}288\. We usetorch\.compile\(dynamic=False\)and gradient accumulation of 8 micro\-batches\.[Table6](https://arxiv.org/html/2608.18062#A2.T6)lists the resulting effective learning rates and weight decay per parameter group\.

Table 6:Effective learning rates and optimizer settings per parameter group\.
##### Schedule\.

We use a linear LR warm\-up over 40 steps, holding it constant until 35% of training\. LR then decays linearly over the remaining 65% to 5% of peak\. Muon momentum: 0\.85→\\to0\.97 over 400 steps, constant at 0\.97, then 0\.97→\\to0\.90 during warmdown\. Weight decay:λref⋅B/Bref⋅\(Dref/D\)=0\.050\\lambda\_\{\\text\{ref\}\}\\cdot\\sqrt\{B/B\_\{\\text\{ref\}\}\}\\cdot\(D\_\{\\text\{ref\}\}/D\)=0\.050, with cosine decay to zero \(TepochT\_\{\\text\{epoch\}\}framework, which scales weight decay to holdTepoch=B/\(η​λ​D\)T\_\{\\text\{epoch\}\}=B/\(\\eta\\lambda D\)constant\)\.

### B\.3Training Data Composition

#### B\.3\.1Natural Language\-Focused Track

Our model training data consists of 36\.9% English web \(FineWeb\-Edu\), 33\.4% multilingual in 30 languages spanning 11 scripts \(FineWeb2\), 16% math \(FineMath\), and 13\.6% code \(StarCoderData\), where percentages are computed using text bytes\. Note that the multilingual portion creates a 266×\\timesdata scarcity gradient from Russian \(18% of bytes\) to Tamil \(0\.1%\), mimicking the drastic differences between languages’ representations observed in most real\-world datasets\. We use temperature sampling to create our dataset, with sampling weights derived from data proportions in the original dataset\. This produces a data scarcity gradient from Russian \(18\.1% of the multilingual data’s bytes, 531K docs\) to Tamil \(0\.1%, 1\.8K docs\)\.[Table7](https://arxiv.org/html/2608.18062#A2.T7)gives the overall domain composition \(English, multilingual, math, code\)\. The full per\-language breakdown is in[Table8](https://arxiv.org/html/2608.18062#A2.T8)\.

Table 7:LM training data composition \(5M documents, 25 GB UTF\-8 text\)\.Table 8:Multilingual training data by language, sorted by byte share\. 11 writing systems, 11\+ language families\. Data scarcity spans 266×\\timesfrom Russian to Tamil\.
#### B\.3\.2Math\+Code\-Focused Track

We build the math\+code mixture from two sources: MegaMath\-Web\-Pro \(LLM360 MegaMath;∼\\sim50 GB text,∼\\sim14M documents\) for math, and The Stack v2’s educational subset for code, restricted to seven languages weighted proportionally to their share of the source data: JavaScript \(12\.7 GB, 3\.4M docs\), Java \(11\.0 GB, 3\.4M docs\), Python \(11\.0 GB, 2\.3M docs\), C\+\+ \(7\.1 GB, 1\.4M docs\), TypeScript \(3\.5 GB, 1\.3M docs\), Go \(3\.4 GB, 0\.8M docs\), and Rust \(1\.3 GB, 0\.2M docs\), together∼\\sim50 GB and∼\\sim13M documents\. Eight furtherstackv2\-edulanguages \(C, C\#, PHP, Ruby, Shell, SQL, Swift, Markdown\) are excluded as less relevant to the code benchmarks of[Section4\.3](https://arxiv.org/html/2608.18062#S4.SS3)\. The combined mixture is∼\\sim100 GB of text over∼\\sim27M documents, 50% math and 50% code by bytes; documents are sampled with the same weighted\-random mechanism as the natural\-language mixture above, so any prefix of the mixture keeps this ratio\. We use a∼\\sim20B\-token prefix to train the math\+code models\.

## Appendix CEvaluation Details

### C\.1Downstream Evaluation Configurations

##### Loglikelihood scoring\.

BLiMP and MultiBLiMP are minimal\-pair benchmarks: each item pairs an acceptable sentence with a minimally differing unacceptable one, the model assigns each full sentence its total log\-likelihood, and the item counts as correct when the acceptable sentence receives the higher total\. Accuracy is the fraction of items scored correctly; no generation is involved\. Both benchmarks score the sentences from an empty context: the beginning\-of\-sequence token is prepended and every token of the sentence is scored \(the BOS\-prefixed convention\), matching how documents are packed during training, where every document is itself preceded by the beginning\-of\-sequence token\. The totals are unnormalized log\-likelihoods, so sentence pairs whose members tokenize to different lengths are compared on unequal token counts; this is the lm\-evaluation\-harness reference convention for these tasks, and[SectionE\.3](https://arxiv.org/html/2608.18062#A5.SS3)notes one consequence for cross\-scale BLiMP comparisons\.

##### Generation harness\.

GSM8K, HumanEval, and MBPP \([SectionF\.3](https://arxiv.org/html/2608.18062#A6.SS3)\) use the Language Model Evaluation Harness\([13](https://arxiv.org/html/2608.18062#bib.bib21), v0\.4\.11;\): GSM8K is 8\-shot chain\-of\-thought with exact\-match scoring on the full 1,319\-problem test set; HumanEval is pass@1 under greedy 0\-shot decoding; MBPP is pass@1 under greedy 3\-shot decoding\. Generation heals the prompt boundary: where the prompt’s own tokenization does not end on a token boundary the tokenizer would itself produce there, the decoder drops the affected trailing prompt token\(s\) and constrains generation until their content is re\-emitted, so scoring never starts from a token sequence absent from training\.

##### Bits\-per\-byte\.

BPB=∑iNLLi/\(ln⁡2⋅B\)\\text\{BPB\}=\\sum\_\{i\}\\text\{NLL\}\_\{i\}/\(\\ln 2\\cdot B\), whereNLLi\\text\{NLL\}\_\{i\}is the model’s negative log\-likelihood, in nats, for scored tokenii; dividing byln⁡2\\ln 2converts nats to bits; andBBis the UTF\-8 byte length of the decoded scored tokens\. Normalizing by byte length rather than by token count makes the metric comparable across tokenizers regardless of differences in vocabulary or compression rates\.

#### C\.1\.1Tokenizer Panel Inclusion Criteria

##### Eligibility\.

Membership in the primary correlation panel \(P\) is criteria\-defined\. A tokenizer is eligible if it meets all four of the following conditions:

- •it is trained on the paper’s standard source corpus; any composition, subset, or reweighting of that corpus is a value on the training\-data axis, so the language\-coverage and reweighting ablations remain eligible;
- •its vocabulary falls in the∼\\sim128K size band;
- •it carries no known tokenizer\-training defect;
- •it is not a deliberate minimal\-structure or null\-control probe\.

##### Tokenizer families\.

Two tokenizers belong to one family when they share the same value on all three design axes, algorithm, normalization/pretokenization strategy, and training data composition, and differ only in a training hyperparameter or preset of that shared configuration\. For example, BPE GPT\-4o \(all\-eq\.\) and its no\-repeat\-sampling variant differ only in a sampling option and form one family, whereas BPE GPT\-4o \(bal\.\) and BPE GPT\-4o NFC differ in a value on the normalization axis and form two families\. Options internal to one algorithm’s implementation also count as presets: the two MinGram builds, which differ only in whether merges may cross line breaks, form one family under this clause\. Treating line\-break crossing as a value on the pretokenization axis would instead place them in two families\. The 31 families partition the 39\-member extended panel, and the Fam\. column of[Table5](https://arxiv.org/html/2608.18062#A1.T5)lists every row’s family\.

##### Panel construction\.

Every member of every family is a member of theextendedpanel \(n=39n=39\)\. Theprimarypanel takes one representative per family, chosen at the family’s default preset, except two families, which contribute none\.101010Both are SuperBPE configurations\. SuperBPE trains in two stages: stage 1 trains a seed tokenizer, and stage 2 extends the seed with additional merges under a second pretokenization regex\. The stage\-2 regex is the regex that segments text when the finished tokenizer is applied, so it is the value the pretokenization axis considers for a SuperBPE configuration; the choice of seed tokenizer, including the regex the seed was trained under, is a training choice we consider internal to the algorithm, on the same footing as a hyperparameter\. Each stage\-2 regex is therefore represented once, at the default plain\-BPE seed and default preset: the plain\-BPE\-seeded build for the GPT\-4o regex, and the C2 build for the clean regex, whose family also holds the C3 preset variant\. The two families without a representative differ from an already\-represented configuration only in choices internal to the algorithm: the pair seeded from a parity\-aware BPE tokenizer under the GPT\-4o stage\-2 regex, and the punct\-seeded variant, which combines a non\-default seed with a non\-default transition point\. Adding a representative from either family would place a second point in a cell of the design grid that is already represented, without ablating any design axis\. Both remain their own families in the extended\-level analyses, which operate on them as units\.This gives the primary panel 29 members\. The primary panel’s members are therefore near\-independent design points by construction; strong dependencies between points would violate independence assumptions important to the validity of correlational analyses\. The extended panel’s same\-family members are not independent, so inference at that level treats families as units;[SectionC\.1\.2](https://arxiv.org/html/2608.18062#A3.SS1.SSS2)gives the procedure\.

##### Reference tokenizers\.

The two off\-the\-shelf reference tokenizers are held out of both panels\. We use them as prediction targets \([Section5](https://arxiv.org/html/2608.18062#S5)\)\.

##### Defect exception: three tokenizers without a full byte alphabet\.

Three of the tokenizers we trained cannot represent all 256 byte values: the English\-only builds under the punct, GPT\-4o and Claude pretokenizers each represent 224 of the 256, and none has a vocabulary entry containing\{\. The cause is that the BPE trainer seeds its single\-byte vocabulary only from byte values that occur in the tokenizer’s own training sample, and merges can only build on entries that exist, so a byte absent from that sample is unrepresentable\. This is arguably a violation of the defect\-free criterion that we stated earlier as necessary for inclusion, but tokenizers often do not contain the full byte\- or character\-alphabet \(for example, byte\-fallback is off by default in the HuggingFace Tokenizers library\)\. We thus did not label these tokenizers as defective\. We note that tokenizers and std\-1B models were retrained with the full byte alphabet for the GPT\-4o and Claude pretokenization builds\. A math\+code model was also trained for the full\-byte GPT\-4o variant\. The Claude pretokenizer math\+code variant is kept in the math\+code panel with the defect flagged, as are the std\-1B and math\+code models for the punct variant\. The sensitivity of the published correlations to the punct build, including the one result whose corrected significance depends on it, is quantified in[SectionF\.2](https://arxiv.org/html/2608.18062#A6.SS2)\.

#### C\.1\.2Statistical Testing Protocol

The tokenizer panels are designed grids, not random samples, and two sources of statistical dependence follow from that design\. First, members of one tokenizer family are close to duplicates, so treating every tokenizer as an independent observation would overstate the amount of evidence and understate uncertainty; the family partition of[SectionC\.1\.1](https://arxiv.org/html/2608.18062#A3.SS1.SSS1)is the unit that addresses this\. Second, the per\-language analyses observe each tokenizer once per language, so the rows contributed by one tokenizer are correlated with each other; clustering standard errors by tokenizer addresses this\. Throughout, a*standard error*is the expected size of an estimate’s variation if the experiment were repeated, tests are two\-sided, and significance means a Benjamini–Hochberg\-adjustedp<0\.05p<0\.05\([Section4\.3](https://arxiv.org/html/2608.18062#S4.SS3)\)\.[Table9](https://arxiv.org/html/2608.18062#A3.T9)maps each analysis to the panel it runs on and the uncertainty method it uses; the paragraphs below follow the same order\.

Table 9:The four inference layers, the panel each runs on, and the uncertainty method each uses\.##### Primary panel: analytic Spearman tests\.

On the primary panel, Spearman correlations are tested with the analyticpp\-value, adjusted within the correction families listed under Multiple comparisons below\.[SectionC\.1\.1](https://arxiv.org/html/2608.18062#A3.SS1.SSS1)states why the panel’s construction makes the independence approximation reasonable, and[SectionF\.2](https://arxiv.org/html/2608.18062#A6.SS2)probes the remaining shared\-design\-axis structure directly\.

##### Extended panel: family\-mean permutation tests\.

At the extended level, each family is first averaged to one point before ranking, so same\-family tokenizers cannot count as independent evidence\. Significance is then assessed with a permutation test: the outcome values are reshuffled among familiesB=10,000B=10\{,\}000times with a fixed seed, keeping each family intact, and the permutationpp\-value is the fraction of reshuffles whose\|ρ\|\|\\rho\|reaches the observed value, computed as\(h\+1\)/\(B\+1\)\(h\+1\)/\(B\+1\), wherehhcounts the qualifying reshuffles, so thepp\-value is never exactly zero\. The statistic is not computed below 5 families\.

##### Held\-out prediction intervals\.

The same family partition defines the cross\-validation folds of the held\-out prediction analysis \([SectionF\.1](https://arxiv.org/html/2608.18062#A6.SS1)\): each fold holds out one whole family, so no fit is scored on a family member of its own training data\. A 90% prediction interval is the range expected to contain 90% of new held\-out observations; here it is a closed\-formtt\-interval calibrated on these cross\-validation errors, not a bootstrap interval\.

##### Per\-language regressions: two fits, clustered standard errors\.

The cells of[Table3](https://arxiv.org/html/2608.18062#S5.T3)combine two fits\. The printedβ\\betais the mixed\-effects estimate \(outcome∼\\simmetric\+\(1\|\+\\ \(1\\,\|\\,language\)\), metric standardized\)\. The printed standard error and thepp\-value behind the stars come from a second fit of the same within\-language estimand: ordinary least squares with language fixed effects and standard errors clustered on the tokenizer, that is, computed while allowing the residuals contributed by one tokenizer to be correlated across the languages it appears in rather than treated as independent \(G=29G=29clusters; each family contributes one tokenizer to this panel, so tokenizer\-level and family\-level clustering coincide here\)\. The second fit is needed because the mixed\-effects standard errors do not account for that correlation; the two fits’ coefficients agree at the printed precision\.

Clustering increases the standard errors by factors of 1\.05 to 2\.77 and changes the significance status of one of the 18 metric\-outcome cells: fertility against MultiBLiMP accuracy, adjustedp=0\.00026p=0\.00026unclustered against0\.0630\.063clustered\. Six further cells keep their significance at a weaker star level, five in the MultiBLiMP column and UTF\-8 boundary crossing in the FLORES column\.

The clusteredpp\-values use a normal reference distribution\. Under the more conservativettreference withG−1G\-1degrees of freedom, the FLORES column is unchanged, while in the MultiBLiMP column the adjustedpp\-values of four of the five significant metrics rise just above the threshold \(0\.0550\.055\) and UTF\-8 boundary crossing stays significant \(0\.0360\.036\)\. Both inference variants are in the released artifacts\.

##### Multiple comparisons\.

Benjamini–Hochberg correction is applied within the following correction families:

- •the 98 printed cells of[Table2](https://arxiv.org/html/2608.18062#S5.T2)\(14 metrics×\\times7 downstream columns\);
- •the 98 printed cells of[Table16](https://arxiv.org/html/2608.18062#A6.T16), the parallel grid under FineWeb measurement;
- •the 160\-cell grid of the released correlation artifact \(20 metrics×\\times8 downstream targets\); this grid is the correction family behind the natural\-language code BPB correlations quoted in[SectionF\.3](https://arxiv.org/html/2608.18062#A6.SS3), a downstream column computed but not printed in the paper’s tables;
- •the 9 metrics of each outcome of[Table3](https://arxiv.org/html/2608.18062#S5.T3), corrected per outcome;
- •the 38 code\-structure tests of[SectionF\.3](https://arxiv.org/html/2608.18062#A6.SS3);
- •the per\-example families of[SectionF\.4](https://arxiv.org/html/2608.18062#A6.SS4), one per text basis and task, of 7 to 9 tests each\.

One association is tested in two of these families on two rosters: AST boundary alignment against MBPP pass@1 isρ=0\.61\\rho=0\.61over the 19 primary\-panel members with a math\+code model \([Table2](https://arxiv.org/html/2608.18062#S5.T2)\) andρ=0\.65\\rho=0\.65over the full 20\-member math\+code panel \([SectionF\.3](https://arxiv.org/html/2608.18062#A6.SS3)\); the two values describe the same relationship on nested rosters\. The line\-break statistics of[SectionF\.3](https://arxiv.org/html/2608.18062#A6.SS3.SSS0.Px1)and the cross\-model rank correlation of[SectionE\.4](https://arxiv.org/html/2608.18062#A5.SS4)are pre\-specified single comparisons rather than scans over a metric grid, and are reported without correction: the former as rawpp\-values, the latter as a bootstrap confidence interval, a range of values obtained by recomputing the statistic on many resamples of the data, expected to contain the true value in 95% of repetitions\.

##### Robustness and disclosure\.

Panel membership was frozen before any statistic was computed, and results are reported regardless of direction\. Each correlation discussed in[Section5](https://arxiv.org/html/2608.18062#S5)has a standing robustness artifact: single\-member leave\-one\-out over the primary panel, plus recomputation with each design\-axis group removed in turn \([SectionF\.2](https://arxiv.org/html/2608.18062#A6.SS2)\)\. Where a leverage check removes members, as in the operator\-isolation paragraph of[SectionF\.3](https://arxiv.org/html/2608.18062#A6.SS3), the full\-panel result is reported first and the removal is a labeled diagnostic, not a substitution\.

## Appendix DIntrinsic Evaluation Results

[Tables10](https://arxiv.org/html/2608.18062#A4.T10)and[11](https://arxiv.org/html/2608.18062#A4.T11)report the intrinsic metric values for 43 of the 44 trained tokenizers and for the two off\-the\-shelf references, measured on FLORES\+ and on FineWeb\-Edu/FineWeb2 respectively; the two tables’ length\-unit conventions differ and are stated in their captions\. The one tokenizer of[Table12](https://arxiv.org/html/2608.18062#A5.T12)absent from both, Unigram Whitespace tuned, has no intrinsic measurement; it is a minimal\-structure control and a member of no panel, so nothing correlated or ranked in this paper depends on it\.

Table 10:Intrinsic tokenizer evaluation results computed on a subset of the FLORES\+ validation set, in particular on the subset of languages in our training data\. Code and math metrics are computed on StarCoder and math data respectively\. Length\-unit\-dependent metrics useu=linesu=\\texttt\{lines\}, with one exception: Fertility keeps its defaultu=wordsu=\\texttt\{words\}in both this table and[Table11](https://arxiv.org/html/2608.18062#A4.T11)\([Table1](https://arxiv.org/html/2608.18062#S3.T1)\), so its two columns differ only by measurement corpus\. Compression is therefore measured in*lines*per token and is not comparable with the Compression column of[Table11](https://arxiv.org/html/2608.18062#A4.T11), which usesu=bytesu=\\texttt\{bytes\}\. Op Isolation is the math\-corpus domain, not the value pooled over prose, code and math\. Tokenizers are the 43 members of the[Table12](https://arxiv.org/html/2608.18062#A5.T12)set that have an intrinsic measurement \([AppendixD](https://arxiv.org/html/2608.18062#A4)\), plus the two off\-the\-shelf reference tokenizers \(marked with†\), excluded from every ranking and aggregate statistic except the cross\-scale check of[Table17](https://arxiv.org/html/2608.18062#A6.T17)\.Table 11:Intrinsic tokenizer evaluation results computed on a subset of FineWeb\-Edu and FineWeb2\. Concretely, we sample 1k documents per language for each of the languages present in our training data\. Length\-unit\-dependent metrics useu=bytesu=\\texttt\{bytes\}, with one exception: Fertility keeps its defaultu=wordsu=\\texttt\{words\}in both this table and[Table10](https://arxiv.org/html/2608.18062#A4.T10)\([Table1](https://arxiv.org/html/2608.18062#S3.T1)\)\. The value after Fertility’s±\\pmis the standard deviation of the per\-document tokens\-per\-word ratio across the sampled documents, which[Table10](https://arxiv.org/html/2608.18062#A4.T10)does not print because its unit of measurement is a single FLORES\+ line\. Compression is measured in*bytes*per token and is not comparable with the Compression column of[Table10](https://arxiv.org/html/2608.18062#A4.T10), which usesu=linesu=\\texttt\{lines\}\. Tokenizers are the same 43 members as[Table10](https://arxiv.org/html/2608.18062#A4.T10), plus the two off\-the\-shelf reference tokenizers \(marked with†\), excluded from every ranking and aggregate statistic except the cross\-scale check of[Table17](https://arxiv.org/html/2608.18062#A6.T17)\.
## Appendix ELanguage Model Results

### E\.1Language Model Evaluation Results \(Full\-Scale; 1\.27B\)

[Table12](https://arxiv.org/html/2608.18062#A5.T12)reports absolute scores per model on downstream benchmarks for every tokenizer named in this paper, which is a superset of P: it adds the extended\-panel\-only variants, the two defective English\-only retrains, the file\-cap null control and the two minimal\-structure controls\.[Table5](https://arxiv.org/html/2608.18062#A1.T5)gives each row’s panel membership and configuration\.

Table 12:Evaluation results for the 1\.27B models: 44 tokenizers trained on the standard corpus mixture at a∼\\sim128K vocabulary, plus two off\-the\-shelf reference tokenizers \(marked with†\) excluded from every ranking and aggregate statistic except the cross\-scale check of[Table17](https://arxiv.org/html/2608.18062#A6.T17)\. Panel membership per row is in[Table5](https://arxiv.org/html/2608.18062#A1.T5); the rows outside every panel are the two defective English\-only retrains, the file\-cap null control and the two minimal\-structure controls\. Lower is better for the BPB columns; higher is better for BLiMP and MultiBLiMP\. Code BPB for the 25 tokenizers in the cross\-model comparison appears in[Table14](https://arxiv.org/html/2608.18062#A5.T14), where the two model families’ ranks, not absolute scores, are compared\. ‘–’: not evaluated for that tokenizer\.
### E\.2Language Model Evaluation Results \(Pilot\-Scale; 300M\)

We also train 300M\-parameter models \(nanochat d16: 16 layers, 1024 hidden dimension, 8 heads\) on the same data;[Table13](https://arxiv.org/html/2608.18062#A5.T13)reports val BPB, FLORES BPB, and code BPB for this scale\. Note that these models cover only a subset of tokenizers from the entire panel, in particular, the ones that were part of the initial panel\. These use 3\.5B tokens following the same 10\.5×\\timesscaling ratio from the nanochat library\. Pilot models serve as a fast iteration loop; between the 300M and 1\.27B scales the tokenizer ranking by validation BPB has Kendallτ=0\.752\\tau=0\.752\([SectionE\.3](https://arxiv.org/html/2608.18062#A5.SS3)\)\.

Table 13:Pilot\-scale \(300M\) evaluation results; lower is better for all metrics\.†: off\-the\-shelf reference tokenizer, as in[Table12](https://arxiv.org/html/2608.18062#A5.T12)\. The roster is the 30 tokenizers of[Table12](https://arxiv.org/html/2608.18062#A5.T12)that also have a model at this scale, plus the same 2 references; the remaining configurations were added after the pilot sweep and have no 300M model\.
### E\.3Ranking Stability Across Training Scales

We report Kendall’sτ\\taubetween per\-tokenizer val BPB rank vectors at each pair of the four training scales \(d8∼\\sim222M, d12∼\\sim381M, d16∼\\sim596M, d24 1\.27B\), for the 18\-tokenizer panel, in[Table17](https://arxiv.org/html/2608.18062#A6.T17)\. All models at each scale were trained under one software configuration for that scale\. Adjacent\-scaleτ\\tauis0\.8820\.882\(d8\-d12\),0\.9220\.922\(d12\-d16\), and0\.7520\.752\(d16\-d24\); the full\-versus\-smallest \(d8\-d24\)τ\\tauis0\.7650\.765\. Notably, the d16\-d24 pair has the lowestτ\\tauof the three adjacent pairs, suggesting the danger of using results from smaller language models for larger language model tokenizer development\. The ranking changes most at the final doubling of scale, from 596M to the 1\.27B target:τ=0\.752\\tau=0\.752corresponds to about one tokenizer pair in eight ordered differently at the two scales\. Rank agreement between adjacent scales does not tighten as scale grows within our range, so a tokenizer selected on val BPB at a smaller scale can sit several rank positions away at the target scale, and a small\-scale sweep is a screening tool rather than a substitute for a target\-scale comparison\.

BLiMP carries a separate cross\-scale caveat: under the unnormalized total\-log\-likelihood scoring described in[AppendixC](https://arxiv.org/html/2608.18062#A3), aggregate BLiMP accuracy is lower for our 1\.27B models than for the 300M pilots, driven by a cluster of quantifier subtasks whose accuracy collapses under this scoring at the larger scale\. We therefore compare BLiMP only within a single scale\.

### E\.4Cross\-Model Rank Correlation of Code BPB

While the math\+code models \([SectionF\.3](https://arxiv.org/html/2608.18062#A6.SS3)\) share tokenizers with the natural\-language\-focused models, they are trained on very different data mixtures, so code BPB levels are not comparable between the two by construction: for a given tokenizer, a lower value under a math\+code model does not mean we should expect lower code BPB under the corresponding natural\-language\-focused model\.[Table14](https://arxiv.org/html/2608.18062#A5.T14)compares code BPB ranks between the two model families for the 25 tokenizers that have a model in both: because rank transfer is a per\-tokenizer comparison rather than a correlation panel, this set keeps every eligible tokenizer, including family variants that the primary panel collapses to one representative, both English\-only GPT\-4o variants \(with and without the full byte alphabet,[SectionC\.1\.1](https://arxiv.org/html/2608.18062#A3.SS1.SSS1)\), and the whitespace minimal\-structure probe as a labeled control\. The rank correlation between the natural\-language models’ code BPB rank and the math\+code models’ code BPB rank is Spearmanρ=0\.307\\rho=0\.\{307\}\(95% bootstrap CI−0\.191\-0\.\{191\}to0\.7150\.7\{15\},n=25n=25, with the labeled control included\) andρ=0\.359\\rho=\{0\.359\}\(95% CI−0\.147\-0\.147to0\.7640\.764,n=24n=2\{4\}\) with it excluded\. Both confidence intervals span zero, so the natural\-language models’ code BPB ranking does not transfer detectably to the math\+code models’ code BPB ranking\. We took this result as further justification of the choice to train dedicated math and code models\.

Table 14:Code BPB and code BPB rank for the natural\-language models and the math\+code models, for the 25 tokenizers with a model in both families, sorted by natural\-language\-model rank\. Code BPB levels are not comparable across the two columns; only the rank columns are\.†: labeled control \(whitespace minimal\-structure probe\), excluded from the without\-control correlation reported in the text\.∗: this row’s math\+code model was trained before a change to the cluster software environment that the other 24 postdate\. It is kept because it is the only math\+code run for that tokenizer\.

## Appendix FAdditional Analyses

### F\.1Held\-Out Prediction of the Reference Tokenizers

We test how far a fit estimated on the trained tokenizer panel predicts a tokenizer the panel never saw\. For each of four targets, val BPB and FLORES trained\-31 BPB \(fit on the 39\-tokenizer extended panel of[Sections4\.1](https://arxiv.org/html/2608.18062#S4.SS1)and[C\.1\.1](https://arxiv.org/html/2608.18062#A3.SS1.SSS1)\), and code BPB and MBPP pass@1 from the math\+code models \(both fit on the 20\-tokenizer math\+code grid of[Table18](https://arxiv.org/html/2608.18062#A6.T18), excluding the two references\), we take the best univariate intrinsic predictor and a multivariate fit on up to 3 standardized predictors selected by leave\-one\-family\-out cross\-validation inside the panel \(tokenizers from the same family, e\.g\., a base tokenizer and a preset variant, are held out together, so no fold trains on a tokenizer from its test point’s family\), then predict the two off\-the\-shelf reference tokenizers’ models, Mistral\-Nemo and LLaMA\-3, which share architecture and training data with the panel but were excluded from every panel fit\. We report each reference’s prediction error standardized by the fit’s cross\-validated root\-mean\-squared error \(CV\-RMSE\), and whether the observed value falls inside the CV\-RMSE\-calibrated 90% prediction interval \(a closed\-formtt\-interval, not a bootstrap interval\)\.

Table 15:Held\-out prediction of the two off\-the\-shelf reference tokenizers\. For each downstream target, the best univariate intrinsic predictor and the multivariate fit \(up to 3 standardized predictors, selected by leave\-one\-family\-out cross\-validation\), with the in\-sampleR2R^\{2\}, the cross\-validated root\-mean\-squared error \(CV\-RMSE\), and each reference’s prediction error divided by the CV\-RMSE \(standardized error\)\.†: the observed value falls outside the fit’s 90% prediction interval\. Intrinsic predictors are measured on FLORES\+ unless marked otherwise\. Val BPB and FLORES \(trained\) BPB are fit on the 39\-tokenizer extended std\-1B panel; the two math\+code targets are fit on the 20\-tokenizer math\+code panel\.[Table15](https://arxiv.org/html/2608.18062#A6.T15)reports all eight fits\. Under the best univariate fits, 6 of the 8 reference\-target predictions \(4 targets times 2 references\) fall inside their 90% prediction interval; both exceptions are LLaMA\-3, on FLORES \(trained\) BPB at\+2\.10\+2\.10standardized errors and on the math\+code models’ code BPB at−2\.01\-2\.01\. The multivariate fits have a higher in\-sampleR2R^\{2\}and a lower CV\-RMSE than the univariate fits on every target, yet 4 of their 8 reference predictions fall outside the 90% interval, against 2 of 8 for the univariate fits: the narrower multivariate intervals overstate the precision available for a tokenizer from outside the panel\.

The selected predictors differ by training mixture\. The natural\-language models’ val BPB and FLORES \(trained\) BPB are best predicted by tokens per identifier and identifier fragmentation; the math\+code models’ code BPB is best predicted by the Gini coefficient measured on FineWeb, and none of its selected predictors is an entropy metric \(unigram, bigram, or trigram entropy, or Rényi efficiency\), although those metrics correlate significantly with the natural\-language models’ code BPB \([SectionF\.3](https://arxiv.org/html/2608.18062#A6.SS3)\)\.[Table16](https://arxiv.org/html/2608.18062#A6.T16)shows the same Gini coefficient association with the math\+code models’ code BPB only under FineWeb measurement, not under FLORES\+ measurement\.

### F\.2Intrinsic–Downstream Correlations Robustness Checks

[Table16](https://arxiv.org/html/2608.18062#A6.T16)repeats the aggregate correlation analysis of[Table2](https://arxiv.org/html/2608.18062#S5.T2), with the FLORES\+\-measured intrinsic metrics replaced by their FineWeb\-Edu/FineWeb2\-measured counterparts, over the same primary panel and downstream targets\. Four rows \(identifier fragmentation, digit boundaryF1F\_\{1\}, operator isolation on the math corpus, and AST alignment\) are byte\-identical between the two tables, since those metrics are computed on code and math corpora rather than on the natural\-language corpus that varies between them; see the caption below\.

Table 16:Aggregate Spearman \(FineWeb\-2/\-Edu intrinsic\)ρ\\rhobetween intrinsic metrics and downstream performance \(n=29n=29\{\}primary\-panel tokenizers;n=19n=19\{\}for the math\+code Code BPB and MBPP columns, the primary\-panel members that also have a math\+code model\)\. Intrinsic metrics are measured on FineWeb\-Edu and FineWeb2 with length unitu=bytesu=\\texttt\{bytes\}, so unit\-dependent metrics \(compression rate, fertility\) are not comparable with the same column name in[Table2](https://arxiv.org/html/2608.18062#S5.T2), which usesu=linesu=\\texttt\{lines\}\. Four rows, Ident\. fragmentation, Digit boundary F1, Operator isolation \(math\), and AST alignment, are computed on code and math corpora rather than on FineWeb, so they are byte\-identical to the same rows of[Table2](https://arxiv.org/html/2608.18062#S5.T2)\.Boldindicates significance, with stars marking the \(BH\-adjusted\) thresholds, over the 98 printed cells and exactly as in[Table2](https://arxiv.org/html/2608.18062#S5.T2):∗padj<0\.05p\_\{\\text\{adj\}\}<0\.05,∗∗padj<0\.01p\_\{\\text\{adj\}\}<0\.01,∗∗∗padj<0\.001p\_\{\\text\{adj\}\}<0\.001\.As a robustness check, we repeat the correlation analysis at the extended\-panel level \([Section4\.3](https://arxiv.org/html/2608.18062#S4.SS3)\): each of the 31 tokenizer families in the 39\-tokenizer extended panel is collapsed to its family\-mean score before computing the Spearman correlation, so same\-family tokenizers cannot count as independent evidence, and significance is assessed by a permutation test that reshuffles whole families rather than individual tokenizers\. This extended\-level correction is computed over a larger set of comparisons than the 98 cells printed in[Table2](https://arxiv.org/html/2608.18062#S5.T2), so its adjustedpp\-values are not directly comparable to the table’s\. Under this check, the Rényi\-efficiency and digit\-boundary\-F1F\_\{1\}correlations reported above both remain significant \(padj=0\.016p\_\{\\text\{adj\}\}=0\.016and0\.0290\.029\), as do four of the five information\-theoretic correlations with FLORES\(trained\) BPB; the fifth, unigram entropy, falls just short of significance \(padj=0\.062p\_\{\\text\{adj\}\}=0\.062\)\. We checked whether any single tokenizer drives the four correlations discussed above \(Rényi efficiency with FLORES\(trained\) BPB, digit boundaryF1F\_\{1\}with BLiMP, digit boundaryF1F\_\{1\}with Code BPB, and AST alignment with MBPP\) by removing tokenizers one at a time from the relevant panel and recomputing each correlation\. Every one of these leave\-one\-out recomputations keeps the original sign andp<0\.05p<0\.05before correction for multiple testing: this holds across all 29 possible single\-tokenizer removals for the two correlations computed on the full primary panel \(Rényi efficiency/FLORES and digit boundaryF1F\_\{1\}/BLiMP\), and across all 19 possible removals for the two correlations restricted to primary\-panel tokenizers with a math\+code model \(digit boundaryF1F\_\{1\}/Code BPB and AST alignment/MBPP\); the worst\-casepp\-value across every removal of all four correlations is0\.01620\.0162\. Under the correction applied jointly across the full table \([Table2](https://arxiv.org/html/2608.18062#S5.T2)\), removing a single tokenizer, Unigram Claude \(Claude pretokenizer, UnigramLM algorithm\), is enough to push three of these four correlations, digit boundaryF1F\_\{1\}with BLiMP, digit boundaryF1F\_\{1\}with Code BPB and AST alignment with MBPP, just above the corrected significance threshold, and 12 of the 29 possible single removals move at least one of the four\.

We also checked robustness to removing whole groups of tokenizers that share a non\-default value on one design axis \(algorithm, pretokenization, or training\-data composition;[Section4\.1](https://arxiv.org/html/2608.18062#S4.SS1)\), keeping only the tokenizers that share this panel’s single most common value on that axis\. These axis\-based removals affect the four correlations less uniformly than single\-tokenizer removal: three weaken but keep their sign, while the AST\-alignment/MBPP correlation reverses sign, becoming small and not significant \(ρ=−0\.25\\rho=\-0\.25\), when restricted to the 7 tokenizers in this panel that use the GPT\-4o pretokenizer, this panel’s most common pretokenization choice\. This reversal is consistent with the line\-break attribution below: the AST\-alignment/MBPP association reflects a difference between pretokenizer choices rather than a relationship that holds within a single pretokenizer choice\.

Table 17:Kendall’sτ\\taubetween per\-tokenizer val BPB rank vectors across the four training scales, 18\-tokenizer panel \(n=18n=18for every cell; unlike the other tables in this paper, this panel includes the two off\-the\-shelf reference tokenizers\)\. All models at each scale were trained under one software configuration for that scale\.
### F\.3Math and Code Generation \(math\+code Models\)

Table 18:Evaluation results for the math\+code models: 20 LMs trained from scratch on a math and code mixture \([Section4\.2](https://arxiv.org/html/2608.18062#S4.SS2)\) using custom tokenizers at a∼\\sim128K vocabulary, plus two off\-the\-shelf reference tokenizers \(marked with†\) excluded from every ranking \([Section4\.1](https://arxiv.org/html/2608.18062#S4.SS1)\)\.b: this row’s tokenizer cannot represent all 256 byte values \([SectionC\.1\.1](https://arxiv.org/html/2608.18062#A3.SS1.SSS1)\)\. Both marked rows are English\-only builds kept in this panel with the defect flagged rather than dropped, because no defect\-free tokenizer for either configuration has a math\+code run at this scale; the GPT\-4o English build, which does have a defect\-free counterpart, is represented here by that counterpart instead\. Lower is better for Code BPB; higher is better for MBPP, HumanEval, and GSM8K\.[Table18](https://arxiv.org/html/2608.18062#A6.T18)reports the math\+code models’ generation benchmarks and code BPB\. Under identical training data, architecture, and budget, MBPP pass@1 ranges from0\.0000\.000to0\.2500\.250across the 20 custom tokenizers, while HumanEval pass@1 stays within0\.1340\.134to0\.1770\.177\. The paragraphs below trace the MBPP spread to the vocabularies’ line\-break handling, and then examine how the code\-structure metrics of[Section3](https://arxiv.org/html/2608.18062#S3)relate to it\.

##### Line\-break handling and code generation\.

Pretokenizers differ in whether the tokenizer may learn tokens that fuse punctuation with a following line break\. The GPT\-4o regex permits such merges: the balanced GPT\-4o BPE vocabulary contains 1,024 fused punctuation\-line\-break tokens, for example`\)\\n`and`;\\n`, among 1,091 tokens containing a line break overall\. The punct, Claude, and clean\-multi regexes split before a line break, so their vocabularies contain none\. This vocabulary property predicts code generation\. Across the 18 custom models with byte\-level vocabularies \(token\-byte counts are comparable only within this subset; the two SCRIPT\-encoding members and the two references are excluded\), the number of vocabulary tokens containing a line break correlates with MBPP pass@1 at Spearmanρ=−0\.717\\rho=\-0\.717\(significant,p=8\.1×10−4p=8\.1\\times 10^\{\-4\}\), and models whose tokenizer has at least one fused punctuation\-line\-break token average MBPP pass@10\.0750\.075, against0\.2260\.226for models whose tokenizer has none \(Mann–Whitneyp=4\.5×10−4p=4\.5\\times 10^\{\-4\}\)\. The difference is visible in the generated text itself: the fraction of a model’s MBPP generations containing a line break correlates with its MBPP pass@1 atρ=\+0\.593\\rho=\+0\.593\(significant,p=5\.8×10−3p=5\.8\\times 10^\{\-3\}; this measurement uses generated text rather than token bytes, so it covers all 20 custom models\)\. Models trained with vocabularies containing many fused punctuation\-line\-break tokens emit fewer line breaks in their generations, and MBPP scores Python, in which line breaks are syntactic\.

##### Code\-structure metrics in the math\+code models\.

In the math\+code models, AST boundary alignment correlates with MBPP pass@1 atρ=\+0\.65\\rho=\+0\.\{65\}\(significant,padj=0\.036p\_\{\\text\{adj\}\}=0\.036,n=20n=20\{\}; 38 tests in one Benjamini\-Hochberg family\), the only significant code\-structure correlation on this roster \(the 20 custom tokenizers of[Table18](https://arxiv.org/html/2608.18062#A6.T18), no references\)\. We do not interpret this correlation as an effect of AST boundary alignment itself, for two reasons\. First, this association is not separable from the line\-break handling described above: controlling for the number of vocabulary tokens containing a line break reduces it toρ=\+0\.26\\rho=\+0\.\{26\}\(not significant,p=0\.31p=0\.31,n=18n=18\{\}, this panel’s members with byte\-level vocabularies\), consistent with the pretokenizer\-axis robustness check in[Section5](https://arxiv.org/html/2608.18062#S5), where the same correlation reverses sign once restricted to tokenizers sharing one pretokenizer choice\. Second, the two MinGram tokenizers differ only in whether merges may cross line breaks \([Table5](https://arxiv.org/html/2608.18062#A1.T5)\)\. MBPP pass@1 is0\.0160\.016for the arm whose merges may cross them and0\.2160\.216for the arm that splits there,2\.32\.3panel standard deviations apart, while their AST boundary alignment differs by0\.00140\.0014standard deviations, too small a difference to explain that gap\. The splitting arm is a member of the extended math\+code roster rather than the 20\-member panel, so it has no row in[Table18](https://arxiv.org/html/2608.18062#A6.T18)\. At the extended level \(the 24\-custom\-tokenizer extended math\+code roster, every family variant rather than one representative per family\), the same association isρ=\+0\.66\\rho=\+0\.66\(significant,padj=0\.017p\_\{\\text\{adj\}\}=0\.017,n=24n=24\{\}\), the same direction and significance as on the primary roster above\. Which metrics predict code BPB also changes with the training mixture\. Under the natural\-language models, four entropy metrics correlate significantly with code BPB \(bigram entropyρ=−0\.55\\rho=\-0\.55, trigram entropy−0\.64\-0\.64, unigram entropy−0\.48\-0\.48, Rényi efficiency−0\.69\-0\.69, allpadj<0\.05p\_\{\\text\{adj\}\}<0\.05,n=29n=29\{\}\); under the math\+code models, none of the four is significant \([Table2](https://arxiv.org/html/2608.18062#S5.T2)\)\. The natural\-language models’ code BPB is a downstream column of the released correlation artifact that the paper’s tables do not print, and it is corrected within that artifact’s own family \([SectionC\.1\.2](https://arxiv.org/html/2608.18062#A3.SS1.SSS2)\); the same holds for the line\-break counts above\. A metric’s predictive value is therefore not a property of the metric alone but of the metric paired with a training distribution\.

##### Operator isolation\.

Operator isolation, whether operators are dedicated tokens rather than fused with adjacent operands \([Section3](https://arxiv.org/html/2608.18062#S3)\), is measured separately on the prose, math, and code corpora\. Its correlation with the math\+code models’ MBPP pass@1 is positive in all three measurement domains \(ρ=0\.52\\rho=0\.52prose,0\.340\.34code,0\.280\.28math,n=20n=20\{\}custom tokenizers\) but none survives BH correction \(smallestpadj=0\.058p\_\{\\text\{adj\}\}=0\.058, prose corpus\)\. The panel contains two UnigramLM tokenizers with MBPP pass@1 of exactly0\.0000\.000; removing them makes all three correlations significant \(padj=0\.012p\_\{\\text\{adj\}\}=0\.012,0\.0070\.007,0\.0070\.007;n=18n=18\)\. Because significance depends on those two models, we read the association as suggestive rather than established; and because the tokenizers that differ in operator isolation also differ in their whole pretokenization regex, the association cannot be attributed to operator handling specifically\.

### F\.4External\-Model Validation of Per\-Example Tokenization Metrics

As a check, we test whether per\-example code\-tokenization quality predicts per\-problem generation success on publicly released code models trained independently of this paper’s pipeline\.

We evaluate 5 publicly released code models \(StarCoder2\-3B, Qwen2\.5\-Coder\-3B, CodeGemma\-2B, phi\-2, stable\-code\-3b\), each with its own tokenizer, spanning 5 distinct tokenizer classes and vocabularies from 49,152 to 256,000 tokens, on HumanEval \(164 problems\) and MBPP \(500 problems\)\.

##### Design\.

For each \(model, problem\) pair we compute per\-example AST\-boundary and identifier\-fragmentation metrics, under that model’s own tokenizer, on the prompt and on the reference solution, and join them to the binary pass/fail outcome\. The primary estimator is a conditional logistic regression stratified on problem, which conditions out item difficulty by construction, with one ability term per model; a Bayesian generalized linear mixed model with a random intercept per problem serves as a companion specification\. Significance uses Benjamini\-Hochberg FDR correction within each text\-basis\-by\-task metric family\.

##### Result: null on both panels\.

No metric is FDR\-significant within either task on the external 5\-model panel: the smallest within\-task BH\-adjustedppover both tasks and both text bases is0\.2190\.219\(target text basis, HumanEval, end\-alignment rate\)\. Applying the identical design to this paper’s own 20 custom math\+code tokenizers \(the math\+code panel of[Table18](https://arxiv.org/html/2608.18062#A6.T18)\), joined to their own MBPP and HumanEval generations, gives the same null result: the smallest within\-task BH\-adjustedppis0\.0870\.\{087\}\(target text basis, MBPP, identifier fragmentation rate\), and 0 of 53 fitted pooled\-and\-within\-task cells are FDR\-significant\.

##### A specification artifact in the pooled fit\.

Pooling HumanEval and MBPP rows into one stratified fit requires one ability term per model per task; fitting a single ability term per model shared across both tasks instead produces an association in the pooled fit that is absent from both tasks individually\. On the internal 20\-tokenizer panel, a likelihood\-ratio test rejects the shared\-ability constraint for every one of the 18 pooled\-scope metric cells, significant in every case \(statistic137\.8\{137\.8\}to193\.5\{193\.5\}, 20 degrees of freedom\)\. The spurious pooled association is therefore a specification artifact of sharing one ability vector across two tasks with different pass rates, not a tokenization effect, and we do not report pooled\-scope results as findings on either panel\.