Tokka-Bench：在100种自然语言和20种编程语言上评估分词器

# Evaluating Tokenizers Across 100 Natural and 20 Programming Languages
Source: [https://arxiv.org/html/2610.08794](https://arxiv.org/html/2610.08794)
###### Abstract

Large language models rely on subword tokenizers whose quality varies across languages, yet no standardized multi\-metric framework exists for broad comparative evaluation\. We introduceTokka\-Bench, an open\-source framework that evaluates tokenizers on five complementary metrics—bytes per token, unique token coverage, subword fertility, word\-split rate, and vocabulary composition—across 100 natural languages \(30\+ scripts\) and 20 programming languages, using language\-aware segmentation adapted to each writing system\. Comparing seven BPE tokenizers \(GPT\-2, GPT\-4, gpt\-oss, Llama 3\.1, Gemma 3, Qwen3, and Kimi K2\) within individual languages, we find that vocabulary allocation strategy matters more than raw vocabulary size, and that programming\-language efficiency has converged among recent tokenizers despite divergent natural\-language profiles\. The framework, data, and interactive dashboard are publicly available\.

Tokka\-Bench: Evaluating Tokenizers Across 100 Natural and 20 Programming Languages

Ben Gublergithub\.com/bgub/tokka\-benchtokka\-bench\.streamlit\.app

## 1Introduction

Tokenization converts raw text into subword units and is the first step in every modern large language model \(LLM\)\. Byte Pair Encoding\(Sennrich et al\.,[2016](https://arxiv.org/html/2610.08794#bib.bib8)\)and its SentencePiece variant\(Kudo and Richardson,[2018](https://arxiv.org/html/2610.08794#bib.bib3)\)dominate, but vocabulary quality varies dramatically across languages\.

These disparities have far\-reaching consequences\. A tokenizer requiring many tokens for equivalent content reduces effective learning per training step\. At inference time, inefficient tokenization lowers throughput, increases cost, and accelerates context\-window exhaustion\(Ahia et al\.,[2023](https://arxiv.org/html/2610.08794#bib.bib2); Petrov et al\.,[2023](https://arxiv.org/html/2610.08794#bib.bib6)\)\.

Despite growing awareness, the community lacks a standardized, multi\-metric framework for comparing tokenizers across languages\. Prior work has focused on individual metrics such as subword fertility\(Ács,[2019](https://arxiv.org/html/2610.08794#bib.bib1); Rust et al\.,[2021](https://arxiv.org/html/2610.08794#bib.bib7)\)or cost parity\(Ahia et al\.,[2023](https://arxiv.org/html/2610.08794#bib.bib2)\), typically for a narrow set of languages\.

We presentTokka\-Bench, an open\-source framework for reproducible tokenizer comparison\. Our contributions are:

1. 1\.A reusable benchmark and toolcovering 100 natural languages \(30\+ scripts\) and 20 programming languages, with an interactive Streamlit dashboard\.
2. 2\.Five complementary metricscapturing compression efficiency, vocabulary coverage, subword granularity, word\-boundary alignment, and vocabulary composition\.
3. 3\.Language\-aware segmentationadapting the definition of a linguistic unit to each writing system \(Section[3\.4](https://arxiv.org/html/2610.08794#S3.SS4)\)\.
4. 4\.A demonstration analysisof seven BPE tokenizers across three generations\.

## 2Related Work

Ács \([2019](https://arxiv.org/html/2610.08794#bib.bib1)\)introduced subword fertility as a metric for comparing BERT tokenizers across languages\.Rust et al\. \([2021](https://arxiv.org/html/2610.08794#bib.bib7)\)expanded on this, showing that monolingual performance of multilingual models correlates with tokenizer quality across 22 typologically diverse languages\.

Petrov et al\. \([2023](https://arxiv.org/html/2610.08794#bib.bib6)\)demonstrated that tokenizer disparities introduce systematic unfairness, with low\-resource languages requiring up to 15×\\timesmore tokens for equivalent content\.Ahia et al\. \([2023](https://arxiv.org/html/2610.08794#bib.bib2)\)quantified the economic implications, showing that speakers of many languages are systematically overcharged by token\-based API pricing\.

Tokka\-Benchextends this line of work with a unified, multi\-metric framework covering an order of magnitude more languages, with language\-aware segmentation\.

## 3Methodology

### 3\.1Datasets

We draw evaluation text from three sources:FineWeb\(Penedo et al\.,[2024](https://arxiv.org/html/2610.08794#bib.bib5)\)for English web text,FineWeb\-2for 99 additional natural languages, andStarCoder\(Li et al\.,[2023](https://arxiv.org/html/2610.08794#bib.bib4)\)for 20 programming languages\. For each language, we sample a contiguous 2 MB UTF\-8 block\.

### 3\.2Tokenizers Evaluated

Table[1](https://arxiv.org/html/2610.08794#S3.T1)lists the seven tokenizers, spanning vocabulary sizes from 50K \(GPT\-2\) to 262K \(Gemma 3\)\.

Table 1:Tokenizers evaluated inTokka\-Bench\.
### 3\.3Metrics

Table[2](https://arxiv.org/html/2610.08794#S3.T2)summarizes our five per\-language metrics, all computed from a single tokenization pass over the 2 MB sample\.

Table 2:Per\-language metrics inTokka\-Bench\.#### Cross\-linguistic comparability\.

Because UTF\-8 byte widths and linguistic\-unit definitions vary by script, all metrics are most reliable when comparing tokenizers*within*a single language\. Cross\-linguistic comparisons are confounded by encoding overhead and differing unit granularity; see Section[6](https://arxiv.org/html/2610.08794#S6)for details\.

### 3\.4Language\-Aware Segmentation

Fertility and word\-split metrics require defining a “linguistic unit\.” Naively applying whitespace segmentation to all languages would produce meaningless fertility values for scripts like Chinese \(where words are not space\-delimited\) or Thai \(where spaces separate phrases, not words\)\. We therefore use three strategies:

- •Whitespace segmentationfor alphabetic scripts \(Latin, Cyrillic, Arabic, Devanagari, etc\.\): units are space\-delimited words\.
- •Character segmentationfor logographic and abugida scripts \(Chinese, Japanese, Thai, Khmer, Lao, Myanmar\): each non\-whitespace character is one unit\.
- •Syllable segmentationfor Tibetan\-family scripts: units are tsheg\-delimited syllables, with a character\-based fallback\.

## 4Results

All comparisons below are within\-language; cross\-linguistic differences are confounded as noted in Section[3\.3](https://arxiv.org/html/2610.08794#S3.SS3)\.

### 4\.1Tokenization Efficiency

Figure[1](https://arxiv.org/html/2610.08794#S4.F1)shows bytes\-per\-token across 10 typologically diverse languages\.

![Refer to caption](https://arxiv.org/html/2610.08794v1/x1.png)Figure 1:Tokenization efficiency \(bytes per token\) across 10 languages\.Clear generational patterns emerge\. GPT\-2 consistently trails modern tokenizers, often dramatically: for Hindi, Gemma 3 achieves 9\.3 bytes/token versus GPT\-2’s 1\.7 \(5\.5×\\times\), and for Khmer the gap reaches 5\.7×\\times\. Even for Russian, modern tokenizers achieve 2\.6–3\.8×\\timeshigher efficiency\. The gains are smallest for English \(<<1\.1×\\times\), reflecting GPT\-2’s English\-centric vocabulary\. Among recent models, Kimi K2 and Qwen3 show notably strong performance on Chinese, consistent with CJK\-heavy training data, while Gemma 3’s larger vocabulary does not uniformly translate to higher efficiency\.

### 4\.2Vocabulary Coverage

Figure[2](https://arxiv.org/html/2610.08794#S4.F2)plots unique token counts\. Within each language, coverage reflects how much vocabulary has been allocated to that script\.

![Refer to caption](https://arxiv.org/html/2610.08794v1/x2.png)Figure 2:Unique token count when encoding 2 MB per language\.Kimi K2 stands out with exceptionally high Chinese coverage \(42K unique tokens vs\. 6\.5K for GPT\-2\), reflecting dedicated vocabulary allocation\. However, larger vocabularies do not guarantee proportional utilization: averaged across natural languages, GPT\-2 activates 16\.8% of its 50K vocabulary per language, while Gemma 3 activates only 5\.3% of its 262K vocabulary—a consequence of spreading capacity across many scripts\.

### 4\.3Subword Fertility

Figure[3](https://arxiv.org/html/2610.08794#S4.F3)presents subword fertility across 17 languages and all seven tokenizers\.

![Refer to caption](https://arxiv.org/html/2610.08794v1/x3.png)Figure 3:Subword fertility \(tokens per linguistic unit\) across 17 languages and 7 tokenizers\. Values near 1\.0 are optimal; red cells indicate high fragmentation\. Units are language\-dependent: words for alphabetic scripts, characters for CJK/Thai/Khmer\.Western European languages show low fertility across modern tokenizers \(English 1\.3, French 1\.5, German 1\.7 for gpt\-oss\), while Eastern European and Indic languages exhibit higher fragmentation—Czech and Polish range from 2\.2 to 3\.7, and GPT\-2 reaches 7\.5 for Hindi, meaning each word becomes 7–8 tokens on average\. For CJK and Southeast Asian scripts, where units are characters rather than words, several modern tokenizers achieve sub\-1\.0 fertility \(e\.g\., Gemma 3 at 0\.37 for Thai, Kimi K2 at 0\.67 for Chinese\), meaning each token spans multiple characters\.

### 4\.4Programming\-Language Convergence

Figure[4](https://arxiv.org/html/2610.08794#S4.F4)compares bytes\-per\-token for 8 programming languages, contrasting GPT\-2 \(2019\) with three modern tokenizers \(Llama 3\.1, Kimi K2, gpt\-oss\)\.

![Refer to caption](https://arxiv.org/html/2610.08794v1/x4.png)Figure 4:Code tokenization efficiency\. Modern tokenizers show near\-identical performance despite differing natural\-language profiles\.Programming\-language efficiency has largely converged among post\-2023 models: Llama 3\.1, Kimi K2, and gpt\-oss achieve nearly identical bytes\-per\-token despite very different natural\-language profiles\. This likely reflects overlapping code corpora and the constrained ASCII character set of programming languages\. GPT\-2, trained before large\-scale code inclusion, lags by 29–51%\.

### 4\.5Vocabulary Size vs\. Efficiency

Figure[5](https://arxiv.org/html/2610.08794#S4.F5)plots mean bytes\-per\-token against vocabulary size\.

![Refer to caption](https://arxiv.org/html/2610.08794v1/x5.png)Figure 5:Vocabulary size vs\. mean efficiency across natural languages\.The relationship is noisy: Gemma 3 \(262K vocabulary\) and gpt\-oss \(200K\) achieve similar mean efficiency \(4\.30 and 4\.25 bytes/token respectively\), while Kimi K2 \(164K\) achieves 3\.19 by concentrating vocabulary on CJK subwords rather than maximizing breadth\. This suggests that*how*vocabulary is allocated matters as much as total size—targeted allocation for high\-value scripts can outperform a uniformly larger vocabulary\.

## 5Conclusion

Tokka\-Benchprovides a reproducible, open\-source benchmark for evaluating tokenizers across 100 natural and 20 programming languages with language\-aware segmentation, five complementary metrics, and an interactive dashboard\. Our demonstration analysis suggests that vocabulary allocation strategy matters more than raw vocabulary size and that programming\-language tokenization has converged among recent models\. The framework and data are available at[https://github\.com/bgub/tokka\-bench](https://github.com/bgub/tokka-bench), with an interactive dashboard at[https://tokka\-bench\.streamlit\.app/](https://tokka-bench.streamlit.app/)\.

#### Future work\.

Key directions include \(1\) correlating tokenizer metrics with downstream task performance, \(2\) developing normalization for fair cross\-linguistic comparison, \(3\) adding variance estimates via multiple samples per language, and \(4\) extending to non\-BPE approaches \(WordPiece, Unigram\)\.

## 6Limitations

#### Cross\-linguistic comparability\.

Each language is evaluated on a 2 MB UTF\-8 sample\. Because scripts vary in bytes per character \(1 for ASCII, 2–3 for Cyrillic/Arabic/Devanagari, 3–4 for CJK\), this fixed budget represents different amounts of text across languages, confounding cross\-linguistic metric comparisons\. Similarly, bytes\-per\-token conflates tokenization quality with UTF\-8 encoding properties, and fertility uses language\-dependent units \(words vs\. characters\), making cross\-linguistic values incommensurable\. All metrics are most reliable for within\-language tokenizer comparisons\.

#### Single sample, no variance estimates\.

Metrics are computed from a single 2 MB block per language without confidence intervals\. Topic or domain shifts could affect results, particularly unique token counts\.

#### Scope\.

All evaluated tokenizers use BPE variants; the demonstration does not cover WordPiece, Unigram, or byte\-level approaches\. We report tokenizer\-intrinsic metrics only and do not validate whether they predict downstream task performance\.

## Acknowledgments

Vin Howe, Sachin Raja, and Jacob Holloway reviewed early drafts and provided useful feedback\. Harsha Vardhan Khurdula helped gather relevant research and think through metrics systematically\.

AI assistance\.Claude \(Anthropic\) was used to assist with drafting, editing, and code generation during the development of this paper and the Tokka\-Bench framework\.

## References

- Ács \(2019\)Judit Ács\. 2019\.Exploring BERT’s vocabulary\.[https://juditacs\.github\.io/2019/02/19/bert\-tokenization\-stats\.html](https://juditacs.github.io/2019/02/19/bert-tokenization-stats.html)\.Blog post\.
- Ahia et al\. \(2023\)Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov\. 2023\.Do all languages cost the same? Tokenization in the era of commercial language models\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 9904–9923\. Association for Computational Linguistics\.
- Kudo and Richardson \(2018\)Taku Kudo and John Richardson\. 2018\.SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing\.In*Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations*, pages 66–71\. Association for Computational Linguistics\.
- Li et al\. \(2023\)Raymond Li, Loubna Ben Allal, Yangtian Zi, and 1 others\. 2023\.StarCoder: May the source be with you\!*Transactions on Machine Learning Research*\.
- Penedo et al\. \(2024\)Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, and 1 others\. 2024\.The FineWeb datasets: Decanting the web for the finest text data at scale\.In*Advances in Neural Information Processing Systems*, volume 37\.
- Petrov et al\. \(2023\)Aleksandar Petrov, Emanuele La Malfa, Philip Torr, and Adel Bibi\. 2023\.Language model tokenizers introduce unfairness between languages\.In*Advances in Neural Information Processing Systems*, volume 36\.
- Rust et al\. \(2021\)Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych\. 2021\.How good is your tokenizer? On the monolingual performance of multilingual language models\.In*Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\)*, pages 3118–3135\. Association for Computational Linguistics\.
- Sennrich et al\. \(2016\)Rico Sennrich, Barry Haddow, and Alexandra Birch\. 2016\.Neural machine translation of rare words with subword units\.In*Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 1715–1725\. Association for Computational Linguistics\.