Pruned BPE: Post-training Visibility Pruning and Token Reallocation for Byte Pair Encoding

# Pruned BPE: Post-training Visibility Pruning and Token Reallocation for Byte Pair Encoding
Source: [https://arxiv.org/html/2608.00837](https://arxiv.org/html/2608.00837)
\\setCJKmainfont

\[ BoldFont = FandolSong\-Bold\.otf \]FandolSong\-Regular\.otf\\setCJKsansfont\[ BoldFont = FandolHei\-Bold\.otf \]FandolHei\-Regular\.otf\\setCJKmonofontFandolFang\-Regular\.otf\\newCJKfontfamily\\cjkcodefont\[ Scale=0\.80, AutoFakeBold=false \]FandolSong\-Regular\.otf

Kenny Shao Department of Computer Science Florida International University kennyshao0919@gmail\.com

###### Abstract

Byte Pair Encoding \(BPE\) is widely used for subword tokenization, but standard BPE exposes every learned merge token to the downstream model, including tokens that mainly serve as intermediate construction units and rarely appear in the final encoded corpus\. This paper proposes Pruned BPE, a post\-training visibility\-pruning and token\-reallocation method that separates merge construction from model\-visible vocabulary selection\. After standard BPE training, tokens are evaluated by final exposure; low\-exposure tokens are retained as internal\-only merge nodes, while their model\-visible vocabulary slots are reassigned to additional better\-exposed candidates learned through resumed training, preserving the target model\-visible vocabulary size\. During encoding, internal\-only tokens are recursively expanded into visible descendants before token IDs are returned, while the original BPE merge order is preserved\. Experiments on two non\-overlapping corpora dominated by English and Chinese, as well as on their combination, show that Pruned BPE consistently reduces encoded length relative to Standard BPE at the same training corpus, evaluation corpus, and model\-visible vocabulary size\. At a 40% exposure threshold, Pruned BPE reduces encoded length by approximately 0\.27%–0\.36% on same\-corpus evaluations under the same model\-visible vocabulary budget\. In an additional vocabulary\-only evaluation using a shared exact minimum\-token dynamic\-programming encoder, Pruned BPE retains an advantage of approximately 0\.23%–0\.31%, providing evidence that the improvement arises from the more efficient composition of the model\-visible vocabulary\. Overall, these improvements represent a meaningful fraction of the approximately 1\.5%–3\.8% marginal reduction that would otherwise require adding another 2K tokens to the Standard BPE vocabulary\. Qualitative analysis shows that internal\-only tokens include reusable English word fragments, Chinese character and phrase components, partial UTF\-8 byte sequences, and structured\-text fragments\. These results indicate that post\-training visibility pruning can improve BPE vocabulary efficiency without increasing the number of tokens exposed to the language model\.

## 1Introduction

Byte Pair Encoding \(BPE\) was originally introduced by Gage in 1994 as a data\-compression algorithm that iteratively replaces frequent adjacent byte pairs with new symbols\[[1](https://arxiv.org/html/2608.00837#bib.bib1)\]\. It was later adapted as a subword\-tokenization method for neural machine translation\[[2](https://arxiv.org/html/2608.00837#bib.bib2)\]and became widely used in large\-scale language modeling\. Byte\-level BPE, used in GPT\-style tokenizers such as GPT\-2\[[3](https://arxiv.org/html/2608.00837#bib.bib3)\], is especially attractive for large language models \(LLMs\) because it provides a fixed vocabulary while retaining the ability to encode arbitrary Unicode text without an unknown\-token issue\.

BPE\-style tokenizers remain common in modern LLM systems\. For example, Meta’s Llama 3 tokenizer usestiktokenand BPE merge ranks\[[4](https://arxiv.org/html/2608.00837#bib.bib4)\]; Alibaba’s Qwen and Qwen3 models use byte\-level BPE tokenization\[[5](https://arxiv.org/html/2608.00837#bib.bib5),[6](https://arxiv.org/html/2608.00837#bib.bib6)\]; and Mistral’s v3 tokenizer is described as using BPE with SentencePiece\[[7](https://arxiv.org/html/2608.00837#bib.bib7)\]\. These examples show that BPE remains a practical tokenizer design, but they also motivate closer attention to how its limited vocabulary budget is allocated\.

The allocation limitation comes from a mismatch between how BPE learns tokens and how the final vocabulary is used\. During training, BPE repeatedly adds the most frequent adjacent pair as a new token\. However, a token that is useful at an intermediate merge stage is not necessarily suitable as a final model\-visible vocabulary entry\. Some learned tokens mainly act as construction units for later, longer tokens\. For example, fragments such asvironment,ournal, or\\cjkcodefont朗普 may be useful on the path toward longer tokens such asenvironment,environmental,journal,journalism,journalist, or\\cjkcodefont特朗普, but they may be poor choices as final tokens if they rarely appear independently after all merges are applied\. This problem is especially visible in byte\-level tokenization\. Since each Unicode character may be represented by multiple bytes, standard byte\-level BPE may create intermediate byte combinations that are necessary for constructing complete characters, words, or multi\-character expressions\. For Chinese, Japanese, Korean, and other non\-Latin scripts, some intermediate byte fragments may have little value as model\-visible units even though they remain necessary for reconstructing the BPE merge tree\.

Exposing such low\-exposure intermediate tokens to the language model may waste limited vocabulary capacity\. Moreover, because these tokens are rarely emitted in the final encoded corpus, their embeddings may receive fewer effective training updates and may therefore be less well trained, potentially affecting downstream model performance\.

This paper proposes*Pruned BPE*, a post\-training visibility\-pruning and token\-reallocation method for BPE tokenizers\. The key idea is to separate*merge construction*from*vocabulary exposure*\. BPE training is first allowed to build the merge tree normally\. After training, learned tokens are evaluated by their*final exposure*, defined as how often they remain visible in the fully encoded corpus\. Tokens with insufficient final exposure are retained as internal\-only construction tokens rather than exposed to the downstream language model\. Their visible vocabulary slots are then reallocated to better candidate tokens learned by training beyond the target visible vocabulary size\.

The method is therefore different from ordinary vocabulary pruning or trimming\. It is not intended to shrink the final vocabulary\. Instead, it preserves the target model\-visible vocabulary size while changing which learned tokens are exposed\. During encoding, standard BPE merging is preserved up to the final merged token sequence; any internal\-only token is recursively expanded into visible child tokens before token IDs are returned\. This design preserves the learned merge structure, allows visibility thresholds to be reapplied to a sufficiently large trained candidate merge structure, and frames the problem as vocabulary reallocation rather than simple token removal\.

Overall, the contributions of this work are twofold:

First, this work introduces final exposure as a post\-training criterion for separating BPE merge construction from model\-visible vocabulary selection\. It reallocates low\-exposure token slots while preserving the target model\-visible vocabulary size and the original merge order\. The implementation also integrates token\-ID remapping so that additional trained candidates can fill the target visible vocabulary size seamlessly and in a predictable manner\. A vocabulary\-only evaluation using a shared exact minimum\-token objective further separates the effect of visible\-vocabulary composition from native BPE merge\-order encoding and shows that the resulting advantage persists\.

Second, this work provides a practical reference implementation, including trainers and the tokenizer, which supports Unicode\-aware pre\-tokenization, token boundaries suitable for both natural languages and code\-oriented text, CPU\-parallel training, training\-data sharding, checkpointing, and separate export of visible and internal vocabularies\.

## 2Related Work

BPE was first proposed by Gage as a compression algorithm based on replacing frequent adjacent byte pairs\[[1](https://arxiv.org/html/2608.00837#bib.bib1)\]\. Sennrich et al\. adapted BPE for neural machine translation, showing that subword units can represent rare words compositionally and reduce open\-vocabulary problems\[[2](https://arxiv.org/html/2608.00837#bib.bib2)\]\. Later systems such as GPT\-2, GPT\-3, and RoBERTa adopted byte\-level BPE or closely related variants for large\-scale transformer models\[[3](https://arxiv.org/html/2608.00837#bib.bib3),[8](https://arxiv.org/html/2608.00837#bib.bib8),[9](https://arxiv.org/html/2608.00837#bib.bib9)\]\. These works establish BPE as a practical tokenizer baseline, but they do not distinguish between tokens needed for merge construction and tokens that should remain model\-visible after all merges are applied\.

SentencePiece provides a language\-independent tokenizer and detokenizer framework that can train subword models directly from raw text, including BPE and unigram language\-model tokenization\[[10](https://arxiv.org/html/2608.00837#bib.bib10)\]\. It is related to this work as a practical tokenizer\-construction framework, but its main focus is training and detokenization infrastructure rather than post\-training separation of internal merge tokens from model\-visible vocabulary entries\.

The following works are more directly related because they modify or reconsider how learned subword tokens are used after or during BPE\-style training\.

BPE\-Dropout\[[11](https://arxiv.org/html/2608.00837#bib.bib11)\]modifies the deterministic BPE segmentation process by randomly dropping merge operations during training, producing multiple possible segmentations for the same word and improving robustness in some neural machine translation settings\. This is different from Pruned BPE\. BPE\-Dropout changes the segmentation behavior during model training, while Pruned BPE preserves the standard BPE merge path and applies a post\-training visibility decision to determine which learned tokens should be exposed to the model\.

Scaffold\-BPE\[[12](https://arxiv.org/html/2608.00837#bib.bib12)\]is the most closely related work\. It similarly identifies tokens that mainly serve as components of longer tokens, but it makes visibility decisions dynamically during BPE training\. Pruned BPE instead preserves the standard BPE training path and applies post\-training visibility pruning based on final exposure, allowing different visibility criteria to be applied to the same sufficiently large trained candidate merge structure\. The main differences are summarized in Table[1](https://arxiv.org/html/2608.00837#S2.T1)\.

AspectScaffold\-BPEPruned BPEWhen token visibility is decidedDuring each BPE iteration when a new token is generatedAfter a Standard BPE training stage; final visibility is assigned after resumed candidate training\.Effect on visible\-vocabulary construction pathOnline scaffold decisions can affect later visible\-token membershipStandard BPE merge path is preservedRequired change to BPE training loopYesThe core pair\-selection and merge operations are unchanged, but an outer resumed\-training and dynamic\-stopping procedure is added\.Required change to BPE encoding implementationYesYes\. Standard BPE merging is followed by recursive expansion of internal\-only tokens, with merge ranks stored separately from exported IDs\.Ability to change token visibility after trainingLimited; usually requires rerunning the online procedureYes, provided that the trained candidate merge structure contains enough eligible tokens to fill the target model\-visible vocabulary\.Decision basisDynamic frequency comparison against current queue\-head frequencyFinal token exposureMain risk or tradeoffEarly online decisions may be aggressive and may affect later visible vocabulary compositionRequires training extra candidate tokens to refill the final visible vocabularyTable 1:Comparison between Scaffold\-BPE and Pruned BPEVocabulary\-trimming methods also reconsider whether every learned subword should remain in the final vocabulary\. Cognetta et al\. study post\-processing that replaces rare subwords with their components, but report no consistent performance improvement and possible substantial degradation\[[13](https://arxiv.org/html/2608.00837#bib.bib13)\]\. Unlike simple trimming, Pruned BPE refills the released slots with additional trained candidates so that the target model\-visible vocabulary size is preserved\. The main differences are summarized in Table[2](https://arxiv.org/html/2608.00837#S2.T2)\.

AspectVocabulary TrimmingPruned BPEMain motivation and goalRemove rare or low\-exposure subwords from the final vocabulary, mainly to reduce effective model vocabulary size and possibly improve robustness\.Move low\-exposure intermediate tokens out of the model\-visible vocabulary and reallocate visible\-token slots to other higher\-exposure candidate tokens\.Final visible vocabulary size and outcomeUsually reduces the effective visible vocabulary size; decomposed tokens may also increase tokenized sequence length\.Designed to preserve the target visible vocabulary size by refilling pruned slots with extra trained candidates; sequence length impact depends on the pruning criterion and replacement tokens\.Treatment of removed or pruned tokensRemoved from the final vocabulary and recursively decomposed when produced\.Low\-exposure tokens are retained internally, while visible slots are refilled from extra trained candidates\.Token ID handlingOften handled indirectly by downstream vocabulary construction; implementation details may depend on the toolkit\.Explicit visible/internal vocabulary export with token\-ID remapping\.Required extra training budgetNo; starts from an already trained BPE vocabulary\.Yes; resumed training continues until enough sufficiently exposed candidates are available to fill the target visible vocabulary\.Main risk or tradeoffCan shrink the effective vocabulary and increase sequence length; prior experiments report no consistent performance improvement and possible degradation\.Requires training extra candidate tokens and selecting an appropriate visibility criterion\.Empirical evidencePrior NMT experiments found no consistent improvement and possible heavy degradation\[[13](https://arxiv.org/html/2608.00837#bib.bib13)\]\.Evaluated in this work under fixed model\-visible vocabulary sizes with resumed candidate training\.Table 2:Comparison between Vocabulary Trimming and Pruned BPE
## 3Proposed Method: Pruned BPE

### 3\.1Standard BPE Baseline

The baseline in this work is standard byte\-level BPE\. Given a training corpusCC, each input string, or each chunk produced by the optional pretokenizerPP, is converted into a sequence of UTF\-8 byte tokens\. This initialization guarantees that any Unicode input can be represented without an unknown\-token mechanism\. The BPE training alphabet contains the 256 possible byte values\. Any predefined special tokens occupy reserved positions in the final model\-visible vocabulary but do not participate in merge learning\.

During BPE training, the corpus is represented as a multiset of token sequences\. At each step, the algorithm counts adjacent token pairs in the current corpus representation and selects the most frequent one\. LetSSdenote the current multiset of token sequences, letf​\(x,y\)f\(x,y\)denote the frequency of adjacent pair\(x,y\)\(x,y\)inSS, and let𝒫​\(S\)\\mathcal\{P\}\(S\)denote the set of adjacent token pairs appearing inSS\. The selected pair can be written as

\(a,b\)=arg⁡max⁡\{f​\(x,y\):\(x,y\)∈𝒫​\(S\)\}\.\(a,b\)=\\arg\\max\\\{f\(x,y\):\(x,y\)\\in\\mathcal\{P\}\(S\)\\\}\.
A new tokenttis created to represent the concatenation ofaaandbb\. The merge rule\(a,b\)→t\(a,b\)\\rightarrow tis appended to the learned merge list, and all occurrences of the adjacent pair\(a,b\)\(a,b\)in the corpus representation are replaced bytt\. This process is repeated until the desired target vocabulary size is reached\.

In standard BPE, every learned merge token is included in the final visible vocabulary\. For the byte\-level setting used in this work, the 256 base byte tokens andNreservedN\_\{\\mathrm\{reserved\}\}reserved special tokens occupy fixed positions in the model\-visible vocabulary\. Therefore, for a target model\-visible vocabulary sizeNvisibleN\_\{\\mathrm\{visible\}\}, the required number of learned tokens isNrequired=Nvisible−256−NreservedN\_\{\\mathrm\{required\}\}=N\_\{\\mathrm\{visible\}\}\-256\-N\_\{\\mathrm\{reserved\}\}\. Standard BPE performsNrequiredN\_\{\\mathrm\{required\}\}successful merge operations and exposes every resulting learned token\. The final vocabulary therefore contains the 256 base byte tokens, theNreservedN\_\{\\mathrm\{reserved\}\}reserved special tokens, and theNrequiredN\_\{\\mathrm\{required\}\}learned merge tokens\.

Standard BPE therefore does not distinguish between learned tokens that remain frequent in the final encoded corpus and those that mainly serve as intermediate components of later merges\. Pruned BPE retains the same merge process but changes how the final model\-visible vocabulary is selected\.

### 3\.2Final Exposure and Visibility Pruning

To distinguish these cases, the*final exposure*of a token is defined as the number of times it appears in the corpus after the full BPE merge sequence has been applied\. LetS∗S^\{\*\}denote the final encoded training corpus after all learned BPE merges have been performed\. For a tokentt, its final exposure is defined as

E​\(t\)=∑s∈S∗counts⁡\(t\),E\(t\)=\\sum\_\{s\\in S^\{\*\}\}\\operatorname\{count\}\_\{s\}\(t\),
wherecounts⁡\(t\)\\operatorname\{count\}\_\{s\}\(t\)is the number of occurrences of tokenttin the final token sequencess\. Final exposure is therefore different from the frequency of the pair that originally created the token\. A token may be created from a highly frequent pair at an early training step, but later be absorbed into longer tokens and appear only rarely in the final encoded corpus\.

Pruned BPE uses final exposure as a post\-training visibility criterion\. The merge tree is first trained normally using the standard BPE merge rule\. The algorithm does not modify how adjacent pairs are counted, how the most frequent pair is selected, or how the corpus representation is updated during training\. After training is complete, each learned token is evaluated according to its final exposure\. Tokens whose exposure is greater than or equal to a thresholdτ\\tauare considered eligible for the model\-visible vocabulary, while tokens whose exposure is belowτ\\tauare classified as internal\-only\. During final export, eligible learned tokens are admitted in original merge order until the target number of visible tokens is reached\.

An internal\-only token is not removed from the tokenizer\. It remains in the merge table and can still be used as an intermediate construction unit during encoding\. The difference is that it is never emitted as a model\-visible token ID to the downstream language model\. This separation allows the tokenizer to preserve the same merge structure learned by BPE while avoiding the use of model vocabulary slots for tokens that have little or no final exposure\.

### 3\.3Token Reallocation

If low\-exposure tokens were simply removed from a Standard BPE vocabulary, the final model\-visible vocabulary would become smaller than the target size\. This would make comparison with Standard BPE unfair and could introduce the disadvantages associated with vocabulary trimming, including increased sequence length and possible downstream degradation\[[13](https://arxiv.org/html/2608.00837#bib.bib13)\]\. Pruned BPE instead retains these tokens as internal\-only merge nodes and assigns their released model\-visible slots to sufficiently exposed candidates learned through resumed training\. The method therefore performs vocabulary reallocation rather than vocabulary reduction and preserves the requested model\-visible vocabulary size\.

A necessary implementation detail is that token reallocation changes exported token IDs but does not change the BPE merge order\. During training, each learned token has an original merge rank determined by the order in which its merge was created\. This rank must continue to control merge priority during encoding\. The exported token ID, by contrast, identifies the token in the final visible or internal vocabulary\. After pruning and reallocation, the exported token ID and original merge rank are not necessarily the same\. For example, an internal\-only token may be assigned an ID in the internal vocabulary while retaining an early merge rank\. The tokenizer must therefore preserve both values so that reallocation does not alter the learned sequence of BPE merges\.

In the reference implementation, token reallocation is performed in two stages\. First, Standard BPE is trained to the target model\-visible vocabulary sizeNvisibleN\_\{\\mathrm\{visible\}\}with the minimum exposure threshold set toτ=0\\tau=0\. Here,NvisibleN\_\{\\mathrm\{visible\}\}denotes the complete vocabulary exposed to the downstream model, including the 256 base byte tokens andNreservedN\_\{\\mathrm\{reserved\}\}reserved special tokens\. This produces the Standard BPE baseline and a checkpoint containing the merge structure learned up to the target size\.

Second, the checkpoint is reloaded with the desired exposure thresholdτ\>0\\tau\>0\. Visibility analysis is applied to the learned tokens to measure the current number whose final exposure is at leastτ\\tau\. Training then resumes from the checkpoint to generate additional candidate tokens\. Rather than using a fixed extra\-token budget, training continues until the number of sufficiently exposed learned tokens is large enough to fill all learned\-token positions in the model\-visible vocabulary:

Neligible≥Nrequired,Nrequired=Nvisible−256−Nreserved\.N\_\{\\mathrm\{eligible\}\}\\geq N\_\{\\mathrm\{required\}\},\\qquad N\_\{\\mathrm\{required\}\}=N\_\{\\mathrm\{visible\}\}\-256\-N\_\{\\mathrm\{reserved\}\}\.
Here,NeligibleN\_\{\\mathrm\{eligible\}\}is the number of learned tokens whose final exposure is at leastτ\\tau, andNrequiredN\_\{\\mathrm\{required\}\}is the number of learned\-token positions available in the model\-visible vocabulary\. The 256 base byte tokens and theNreservedN\_\{\\mathrm\{reserved\}\}reserved special tokens occupy the remaining fixed positions\. A maximum training vocabulary size may be supplied as a safety bound, but the required training size is otherwise determined dynamically by this stopping condition\.

After resumed training, final exposure is evaluated again on the resulting training corpus\. Learned tokens are then scanned in original merge order\. Low\-exposure tokens encountered before the visible target is filled are retained as internal\-only tokens, while sufficiently exposed tokens are assigned to the model\-visible vocabulary\. Once the required number of visible learned tokens has been reached, export stops and all remaining later candidates are discarded\. Exported token IDs are assigned separately, while the original merge ranks of all retained learned tokens are preserved for encoding\.

### 3\.4Encoding with Internal\-only Tokens

During encoding, Pruned BPE applies all retained merge rules—those associated with visible or internal\-only tokens—according to their original BPE merge ranks\. Therefore, the initial output of the merge process may contain both visible tokens and internal\-only tokens\. Since internal\-only tokens are not exposed to the language model, they must be expanded before the final token ID sequence is returned\.

Each learned BPE token has two children corresponding to the pair that created it\. If an encoded token is visible, its visible token ID is returned directly\. If the token is internal\-only, the tokenizer recursively expands it into its children\. Each child is then checked in the same way: visible children are emitted, while internal\-only children are expanded further\. This process continues until all emitted tokens are visible tokens\.

Formally, letchild⁡\(t\)=\(a,b\)\\operatorname\{child\}\(t\)=\(a,b\)denote the two children of a learned tokentt, and letidexport⁡\(t\)\\operatorname\{id\}\_\{\\mathrm\{export\}\}\(t\)denote the exported ID assigned to a visible tokentt\. The final emission function can be written recursively as

emit⁡\(t\)=\{\[idexport⁡\(t\)\],if​t∈V,emit⁡\(a\)∥emit⁡\(b\),if​t∈I​and​child⁡\(t\)=\(a,b\)\.\\operatorname\{emit\}\(t\)=\\begin\{cases\}\[\\operatorname\{id\}\_\{\\mathrm\{export\}\}\(t\)\],&\\text\{if \}t\\in V,\\\\\[2\.0pt\] \\operatorname\{emit\}\(a\)\\,\\\|\\,\\operatorname\{emit\}\(b\),&\\text\{if \}t\\in I\\text\{ and \}\\operatorname\{child\}\(t\)=\(a,b\)\.\\end\{cases\}
Here,VVis the visible vocabulary,IIis the internal\-only vocabulary, and∥\\\|denotes sequence concatenation\. Base byte tokens are always visible, so the recursion always terminates\. As a result, Pruned BPE can use internal\-only tokens during the merge process while guaranteeing that the final output sequence contains only model\-visible token IDs\.

This design preserves compatibility with ordinary downstream language\-model training\. The model only sees token IDs from the visible vocabulary\. Internal\-only tokens are implementation details of the tokenizer and are used only to support the merge structure needed to construct larger visible tokens\.

Algorithm 1Pruned BPE Training with Post\-training Visibility Pruning and Token Reallocation1:Corpus

CC; target model\-visible vocabulary size

NvisibleN\_\{\\mathrm\{visible\}\}; reserved\-token count

NreservedN\_\{\\mathrm\{reserved\}\}; exposure threshold

τ\\tau; optional maximum training size

NmaxN\_\{\\mathrm\{max\}\}; pre\-tokenizer

PP, if used by the training pipeline

2:Visible vocabulary

VV; internal\-only vocabulary

II; retained merge table

MexportM\_\{\\mathrm\{export\}\}; exported\-ID mapping

3:Pretokenize

CCusing

PP, if applicable, and convert the chunks to UTF\-8 byte sequences

SS
4:Initialize the 256 base byte tokens and the merge table

M←∅M\\leftarrow\\emptyset
5:

Nrequired←Nvisible−256−NreservedN\_\{\\mathrm\{required\}\}\\leftarrow N\_\{\\mathrm\{visible\}\}\-256\-N\_\{\\mathrm\{reserved\}\}
6:Stage 1: Standard BPE baseline

7:whilethe number of learned tokens is less than

NrequiredN\_\{\\mathrm\{required\}\}do

8:Perform one Standard BPE merge on

SS
9:Record the merge rule and its original merge rank in

MM
10:endwhile

11:Save a checkpoint containing

SS,

MM, and the learned\-token metadata

12:Export the corresponding Standard BPE tokenizer using

τ=0\\tau=0
13:Stage 2: Resumed training with visibility pruning

14:Reload the checkpoint and compute final exposure

E​\(t\)E\(t\)
15:

Neligible←N\_\{\\mathrm\{eligible\}\}\\leftarrowthe number of learned tokens

ttsatisfying

E​\(t\)≥τE\(t\)\\geq\\tau
16:while

Neligible<NrequiredN\_\{\\mathrm\{eligible\}\}<N\_\{\\mathrm\{required\}\}do

17:if

NmaxN\_\{\\mathrm\{max\}\}is specified and the training vocabulary has reached

NmaxN\_\{\\mathrm\{max\}\}then

18:terminate with an insufficient\-candidates error

19:endif

20:Perform a small batch of additional Standard BPE merges on

SS, without exceeding

NmaxN\_\{\\mathrm\{max\}\}when it is specified

21:Record the new merge rules and their original merge ranks in

MM
22:Recompute final exposure and update

NeligibleN\_\{\\mathrm\{eligible\}\}
23:endwhile

24:Final visibility analysis and export

25:Recompute final exposure

E​\(t\)E\(t\)on the resulting training corpus

26:Initialize

VVwith the 256 base byte tokens and

I←∅I\\leftarrow\\emptyset
27:foreach learned token

ttin original merge orderdo

28:ifthe number of visible learned tokens has reached

NrequiredN\_\{\\mathrm\{required\}\}then

29:break

30:elseif

E​\(t\)<τE\(t\)<\\tauthen

31:Add

ttto

II
32:else

33:Add

ttto

VV
34:endif

35:endfor

36:Discard all remaining later candidate tokens

37:Append the

NreservedN\_\{\\mathrm\{reserved\}\}special tokens to

VV
38:Retain in

MexportM\_\{\\mathrm\{export\}\}the merge rules for learned tokens retained in

V∪IV\\cup I
39:Assign exported IDs while preserving original merge ranks

40:return

VV,

II,

MexportM\_\{\\mathrm\{export\}\}, and the exported\-ID mapping

Algorithm[1](https://arxiv.org/html/2608.00837#alg1)summarizes the complete Pruned BPE training and export procedure\. Here,NmaxN\_\{\\mathrm\{max\}\}bounds the trainable vocabulary consisting of the 256 base byte tokens and the learned candidate tokens; reserved special tokens are excluded because they do not participate in BPE training\. Pretokenization is not essential to the core method; the shared pretokenizer used by the reference implementation during training and encoding is described in Subsection[4\.2](https://arxiv.org/html/2608.00837#S4.SS2)\.

## 4Experiments

### 4\.1Corpus Data

Two tokenizer\-training corpora were constructed for the experiments\. They use disjoint document samples and do not share documents, although some source categories, such as Reddit and Chinese Wikipedia, are represented in both corpora\. Both corpora were used during the development and testing of the proposed algorithm to verify the correctness of the trainer implementations and the tokenizer\.

#### Corpus I\.

Corpus I contains approximately 640 MB of UTF\-8 text\. It consists primarily of English and Chinese text, with a small amount of source code and multilingual data\. The English portion is approximately 430 MB and consists mainly of a locally collected sample from FineWeb\-Edu, an educational English web\-text dataset derived from the FineWeb/Common Crawl pipeline\[[14](https://arxiv.org/html/2608.00837#bib.bib14)\], together with a sample of Reddit data\[[15](https://arxiv.org/html/2608.00837#bib.bib15)\]\. The Chinese portion is approximately 204 MB and consists of conversational text collected from several Chinese social media platforms and text from Chinese Wikipedia pages, including both Simplified and Traditional Chinese\. In addition, Corpus I contains a small amount of code\-oriented text, including Java, Python, JavaScript, TypeScript, HTML, JSON, XML, and related formats, as well as a small multilingual component covering additional languages such as French, German, Portuguese, and Finnish\.

#### Corpus II\.

Corpus II contains approximately 1 GB of UTF\-8 text\. It was designed to provide a larger and more balanced mixture of English, Chinese, multilingual text, and code\-oriented text\. The English portion includes a 360 MB subset sampled from CC\-News and approximately 180 MB of randomly sampled Reddit data\[[15](https://arxiv.org/html/2608.00837#bib.bib15)\]\. The CC\-News subset was drawn from the Hugging Facevblagoje/cc\_newsversion of the Common Crawl News dataset\[[16](https://arxiv.org/html/2608.00837#bib.bib16),[17](https://arxiv.org/html/2608.00837#bib.bib17)\], which contains English\-language news articles published between January 2017 and December 2019\. To reduce ordering bias from source, crawl time, or topic clusters, the CC\-News articles were randomly shuffled before sampling\.

The Chinese portion of Corpus II includes a 382 MB subset sampled from THUCNews and approximately 30 MB of text from Chinese Wikipedia pages, including both Simplified and Traditional Chinese\. THUCNews is a Chinese news text classification corpus released with the THUCTC project by the Natural Language Processing Laboratory of Tsinghua University\[[18](https://arxiv.org/html/2608.00837#bib.bib18)\], and was generated from Sina News RSS subscription data from 2005 to 2011\. The selected subset covers 14 news categories: sports, entertainment, home, lottery, real estate, education, fashion, politics, horoscopes, gaming, society, technology, stocks, and finance\. To avoid category imbalance caused by directory ordering, THUCNews was sampled separately by category, with each category limited to the same maximum size before being added to the tokenizer\-training corpus\.

Corpus II also contains a small amount of code\-oriented text, including Java, Python, JavaScript, TypeScript, C\+\+, Markdown, and HTML\. In addition, it includes an 84 MB multilingual corpus covering 42 languages other than English and Chinese, with approximately equal amounts of text for each language\. These languages include Arabic, French, Spanish, Portuguese, Russian, Japanese, Korean, and many others\.

#### Preprocessing\.

All text sources were lightly cleaned before tokenizer training\. The preprocessing removed common boilerplate content, social\-media sharing widgets or sharing blocks, duplicated adjacent lines, visible encoding artifacts, and website\-specific templates or navigation texts\. The goal of this cleaning step was not to heavily normalize the text, but to remove obvious non\-content material that could distort token\-frequency statistics\. The corpora were otherwise kept as mixed natural UTF\-8 text so that the tokenizer would be trained on realistic English, Chinese, multilingual, and code\-oriented inputs\.

#### Training and evaluation use\.

No separate held\-out split was created\. Same\-corpus evaluations use the tokenizer\-training corpus itself, while the additional cross\-corpus evaluations are described in Subsection[4\.3](https://arxiv.org/html/2608.00837#S4.SS3)\.

#### Code and data availability\.

The Python and Cython implementations, the corpus data used in the experiments, and all trained tokenizer files are publicly available in the project repository\[[19](https://arxiv.org/html/2608.00837#bib.bib19)\]\. Each Standard BPE and Pruned BPE configuration includes avocab\.txtfile\. Pruned BPE configurations additionally include aninter\_vocab\.txtfile containing the internal\-only vocabulary\.

### 4\.2Pretokenization

Before applying BPE training or tokenization, input text is divided into boundary\-aware text chunks using a lightweight pretokenization step\. This step is Unicode\-aware and is intended to preserve natural boundaries between scripts, numbers, punctuation, whitespace, and code\-like substrings\. Pretokenization is used as an implementation choice to make the tokenizer behavior more stable on mixed English, Chinese, multilingual, and code\-oriented text\. It is not the main contribution of this work\.

The pretokenizer preserves the original input exactly: concatenating all produced chunks reconstructs the original string without loss\. Ordinary single spaces are attached to the following word\-like chunk when possible, following the common convention used by GPT\-style BPE tokenizers in which leading spaces are treated as part of the following token\. However, tabs, newlines, indentation, and runs of multiple spaces are kept as separate chunks so that formatting\-sensitive text, especially source code, is not normalized or destroyed\.

Characters are first classified into broad Unicode\-aware categories, including whitespace, Latin word characters, digits, non\-Latin Unicode word characters, punctuation, symbols, and other characters\. Latin word spans include ASCII letters, accented Latin letters, digits, and underscores, allowing terms such asabc123,GPT4,IPv6,user\_id,don’t,C’était,déjà, andMontréalto remain within a single pretokenized chunk when appropriate\. The implementation also keeps common code\- and technical\-expression patterns such asGPT\-4,COVID\-19,Node\.js,file\.java,C\+\+, andC\#as compact chunks when their internal punctuation functions as part of the expression rather than as a structural boundary\.

Digit\-first spans are handled separately\. Numeric expressions such as3\.14and10\-20are kept together, while mixed number\-first expressions such as123abcare split at the transition from number to letters\. This rule also helps avoid merging Chinese date\-like strings into large mixed chunks\. For example, a string such as\\cjkcodefont1949年6月24日 is segmented into alternating numeric and Chinese word chunks\.

Non\-Latin Unicode word spans, including Chinese, Japanese, Korean, Greek, Cyrillic, and other scripts, are grouped separately from Latin words and digits\. This prevents cross\-script chunks such as Chinese\-plus\-English or Japanese\-plus\-number sequences from being treated as a single initial unit\. For example, Chinese text followed by an English abbreviation is split at the script boundary, and Japanese or Korean text followed by digits is split at the transition to the numeric span\.

The pretokenizer also treats structural punctuation as boundaries\. Punctuation and symbols are generally emitted as separate chunks, while repeated identical punctuation or symbol characters are grouped together, such as repeated periods, dashes, equal signs, or closing parentheses\. A small number of code\-oriented structural cases are handled explicitly; for example, the opener of an HTML or XML closing tag is preserved as</, so that</div\>is pretokenized into</,div, and\>\. File\-extension\-like chunks such as\.js,\.java,\.py,\.html, and\.jsonare also kept together when they occur in appropriate contexts\.

This pretokenization step is therefore designed to reduce undesirable merges across obvious linguistic, numeric, punctuation, and structural boundaries while still preserving useful word\-like and code\-like units\. It does not change the BPE merge algorithm itself; it only defines the initial text chunks over which BPE is applied\. To faithfully compare Standard BPE with the proposed Pruned BPE method, the same pretokenization procedure is applied to both methods during training and tokenization in the experiments described below\.

### 4\.3Experimental Setup

Using Corpus I, Corpus II, and the combined Corpus I \+ Corpus II described above, three corpus\-specific tokenizer training configurations were evaluated\. All three configurations followed the same implementation and experimental procedure, differing only in the corpus used for tokenizer training\.

The target vocabulary sizes were selected using a marginal encoded\-length reduction heuristic\. During tokenizer training, multiple checkpoints were saved at increasing vocabulary sizes\. The Standard BPE tokenizer constructed from each checkpoint was used to encode the corresponding training corpus, and the marginal reduction in encoded length was measured between adjacent checkpoints\. For two adjacent candidate valuesNi<Ni\+1N\_\{i\}<N\_\{i\+1\}of the model\-visible vocabulary sizeNvisibleN\_\{\\mathrm\{visible\}\}, letLstd​\(Ni\)L\_\{\\mathrm\{std\}\}\(N\_\{i\}\)andLstd​\(Ni\+1\)L\_\{\\mathrm\{std\}\}\(N\_\{i\+1\}\)denote the numbers of tokens produced when the same corpus is encoded by the corresponding Standard BPE tokenizers\. The marginal encoded length reduction is defined as:

Rstd​\(Ni,Ni\+1\)=Lstd​\(Ni\)−Lstd​\(Ni\+1\)Lstd​\(Ni\)×100%\.R\_\{\\mathrm\{std\}\}\(N\_\{i\},N\_\{i\+1\}\)=\\frac\{L\_\{\\mathrm\{std\}\}\(N\_\{i\}\)\-L\_\{\\mathrm\{std\}\}\(N\_\{i\+1\}\)\}\{L\_\{\\mathrm\{std\}\}\(N\_\{i\}\)\}\\times 100\\%\.\(1\)
For each training corpus, four consecutive candidate target vocabulary sizes at 2K intervals were selected from the region immediately before the marginal reduction curve began to flatten\. This region continued to provide meaningful compression benefits while avoiding vocabulary sizes at which additional merges exhibited clear diminishing returns\. To calculate the marginal reduction associated with the largest selected target size, one additional Standard BPE checkpoint 2K above that size was retained\. This additional checkpoint was used only for the marginal\-reduction calculation and was not included as a Pruned BPE evaluation target\. This selection procedure was used only to identify reasonable experimental vocabulary budgets and is not part of the proposed pruning method\. The main evaluation compares Standard BPE and Pruned BPE at the same selected target vocabulary sizes\.

TrainingcorpusEvaluationcorpusModel\-visiblevocab\. sizeStandard BPElength \(M\)Pruned BPE \(20%\)Pruned BPE \(30%\)Pruned BPE \(40%\)Length \(M\)Δ20\\Delta\_\{20\}\(%\)Length \(M\)Δ30\\Delta\_\{30\}\(%\)Length \(M\)Δ40\\Delta\_\{40\}\(%\)Corpus ICorpus I8K194\.009193\.6930\.16193\.5520\.24193\.4030\.3110K186\.742186\.4610\.15186\.3210\.23186\.1990\.2912K181\.310181\.0530\.14180\.9350\.21180\.8030\.2814K177\.056176\.7930\.15176\.6660\.22176\.5660\.28Corpus II8K372\.997372\.5550\.12372\.2380\.20371\.9420\.2810K360\.776360\.3060\.13360\.0690\.20359\.7320\.2912K351\.264350\.8650\.11350\.6460\.18350\.4200\.2414K343\.806343\.2920\.15343\.1220\.20342\.9340\.25Corpus IICorpus I10K197\.549197\.3110\.12197\.2200\.17197\.1190\.2212K191\.933191\.6240\.16191\.4910\.23191\.4170\.2714K187\.445187\.1830\.14187\.0580\.21187\.0040\.2416K183\.931183\.6040\.18183\.4600\.26183\.3420\.32Corpus II10K319\.106318\.5800\.16318\.2970\.25318\.0480\.3312K308\.820308\.2710\.18308\.0190\.26307\.7870\.3314K300\.691300\.1480\.18299\.8740\.27299\.7000\.3316K294\.086293\.4920\.20293\.2370\.29293\.0140\.36Corpus I\+Corpus IICorpus I12K186\.137185\.9050\.12185\.7540\.21185\.6370\.2714K181\.753181\.5210\.13181\.3800\.21181\.2840\.2616K178\.266178\.0360\.13177\.9400\.18177\.8360\.2418K175\.320175\.1010\.13174\.9950\.19174\.9270\.22Corpus II12K310\.885310\.4930\.13310\.2370\.21310\.0340\.2714K302\.763302\.3330\.14302\.1190\.21301\.8790\.2916K296\.077295\.6010\.16295\.4100\.23295\.1850\.3018K290\.537290\.0490\.17289\.8560\.23289\.6660\.30Corpus I\+Corpus II12K497\.022496\.3990\.13495\.9920\.21495\.6710\.2714K484\.516483\.8540\.14483\.5000\.21483\.1640\.2816K474\.343473\.6370\.15473\.3500\.21473\.0220\.2818K465\.857465\.1490\.15464\.8510\.22464\.5930\.27Table 3:Encoded lengths produced by Standard BPE and Pruned BPE under different exposure thresholds\.For each selected target size, the Standard BPE checkpoint produced at that size was used both to export the baseline tokenizer and to initialize three Pruned BPE variants\. Lettbt\_\{b\}denote the last learned token produced during Stage 1, that is, the token created by theNrequiredN\_\{\\mathrm\{required\}\}\-th successful merge, and letE​\(tb\)E\(t\_\{b\}\)denote its final exposure count\. The minimum exposure thresholds were defined as

τr=⌈r⋅E​\(tb\)⌉,r∈\{0\.20,0\.30,0\.40\}\.\\tau\_\{r\}=\\left\\lceil r\\cdot E\(t\_\{b\}\)\\right\\rceil,\\qquad r\\in\\\{0\.20,0\.30,0\.40\\\}\.\(2\)
For each value ofrr, Algorithm[1](https://arxiv.org/html/2608.00837#alg1)is executed with its exposure\-threshold input set toτ=τr\\tau=\\tau\_\{r\}\. Thus, the 20%, 30% and 40% settings correspond to minimum exposure counts equal to 20%, 30% and 40%, respectively, of the final exposure of the Stage 1 boundary token, rounded upward to the nearest integer\. For each threshold, training was resumed from the corresponding checkpoint using the dynamic stopping and final\-export procedure defined in Subsection[3\.3](https://arxiv.org/html/2608.00837#S3.SS3)\.

In the experiments reported here,Nreserved=4N\_\{\\mathrm\{reserved\}\}=4\. The four reserved special tokens were<\_EOS\_\>,<\_SOS\_\>,<\_SEP\_\>, and<\_PAD\_\>\. They were appended to the model\-visible vocabulary during export and were excluded from BPE training and visibility pruning\. The model\-visible and internal\-only vocabularies were exported asvocab\.txtandinter\_vocab\.txt, respectively\.

The resulting Standard BPE tokenizer and its three Pruned BPE variants were evaluated on both the tokenizer training corpus and one or more additional evaluation corpora\. For tokenizers trained on a single corpus, the other corpus served as a cross\-corpus evaluation set\. Tokenizers trained on the combined corpus were evaluated on the combined corpus and on each component corpus separately\.

Forp∈\{20,30,40\}p\\in\\\{20,30,40\\\}, letLpruned,p​\(N\)L\_\{\\mathrm\{pruned\},p\}\(N\)denote the encoded length produced by the Pruned BPE tokenizer using thep%p\\%threshold setting at model\-visible vocabulary sizeNN\. Its reduction relative to the corresponding Standard BPE tokenizer is

Δp​\(N\)=Lstd​\(N\)−Lpruned,p​\(N\)Lstd​\(N\)×100%\.\\Delta\_\{p\}\(N\)=\\frac\{L\_\{\\mathrm\{std\}\}\(N\)\-L\_\{\\mathrm\{pruned\},p\}\(N\)\}\{L\_\{\\mathrm\{std\}\}\(N\)\}\\times 100\\%\.\(3\)
Each comparison uses the same training corpus, evaluation corpus, and model\-visible vocabulary size\.

Encoded length is the total number of tokens produced when a corpus is encoded by a given tokenizer\. Two reduction measures are used\. Equation \([1](https://arxiv.org/html/2608.00837#S4.E1)\) measures the marginal benefit of increasing the Standard BPE vocabulary between adjacent target sizes\. Equation \([3](https://arxiv.org/html/2608.00837#S4.E3)\) measures the additional reduction obtained by Pruned BPE relative to Standard BPE at the same training corpus, evaluation corpus, and model\-visible vocabulary size\.

For threshold settingpp, letNinternal,pN\_\{\\mathrm\{internal\},p\}denote the number of retained internal\-only tokens\. The internal\-only share is defined as

Sharep=Ninternal,pNrequired×100%\.\\mathrm\{Share\}\_\{p\}=\\frac\{N\_\{\\mathrm\{internal\},p\}\}\{N\_\{\\mathrm\{required\}\}\}\\times 100\\%\.
Thus, the share is measured relative to the number of learned\-token positions in the model\-visible vocabulary\.

Table[3](https://arxiv.org/html/2608.00837#S4.T3)reports the encoded lengths and corresponding Pruned BPE reduction percentages, with encoded lengths expressed in millions of tokens\. All reduction percentages were calculated from the original integer token counts before the encoded lengths were rounded for presentation in millions\. Table[4](https://arxiv.org/html/2608.00837#S4.T4)separately reports the counts and shares of tokens classified as internal\-only for each Pruned BPE tokenizer\.

TrainingcorpusModel\-visiblevocabulary size20% threshold30% threshold40% thresholdCountShare \(%\)CountShare \(%\)CountShare \(%\)Corpus I8K841\.091311\.691892\.4410K1011\.041621\.662272\.3312K1211\.031921\.642782\.3714K1541\.122461\.793352\.44Corpus II10K1011\.041671\.712382\.4412K1361\.162121\.812942\.5014K1681\.222721\.983502\.5516K2211\.403382\.154582\.91Corpus I\+Corpus II12K1000\.851791\.522562\.1814K1340\.982191\.593152\.2916K1701\.082581\.643712\.3618K2071\.173111\.754192\.36Table 4:Counts and shares of internal\-only tokens under different exposure thresholds\.
### 4\.4Effect of the Exposure Threshold

Together, Tables[3](https://arxiv.org/html/2608.00837#S4.T3)and[4](https://arxiv.org/html/2608.00837#S4.T4)show the effect of increasing the exposure threshold from 20% to 40%\. For a fixed training corpus and model\-visible vocabulary size, the internal\-only token sets are nested: every token classified as internal\-only at a lower threshold remains internal\-only at a higher threshold, while additional low\-exposure tokens may also be classified as internal\-only\. Consequently, the number of internal\-only tokens is non\-decreasing as the exposure threshold rises, while the internal\-only token share also generally increases\. With respect to the other experimental dimension, as the model\-visible vocabulary size grows, the number of internal\-only tokens is likewise non\-decreasing, while the share generally increases but may remain stable or fluctuate slightly because the total vocabulary size grows simultaneously\. One possible explanation for this overall upward trend is that continued training creates longer tokens that absorb occurrences previously represented by shorter intermediate tokens, thereby reducing the final exposure of those intermediate tokens\.

The encoded\-length results show a corresponding improvement\. For every reported combination of training corpus, evaluation corpus, and model\-visible vocabulary size, a higher exposure threshold produces a greater reduction in encoded length relative to the corresponding Standard BPE tokenizer\. Within the tested threshold range, pruning more low\-exposure tokens therefore allows more visible vocabulary slots to be reallocated and consistently improves encoded\-length efficiency\.

### 4\.5Comparison with Standard BPE

8K10K12K14K0112233443\.753\.752\.912\.912\.352\.351\.951\.950\.310\.310\.290\.290\.280\.280\.280\.28Model\-visible vocabulary sizeNvisibleN\_\{\\mathrm\{visible\}\}Encoded\-length reduction \(%\)\(a\) Corpus I10K12K14K16K3\.223\.222\.632\.632\.202\.201\.871\.870\.330\.330\.330\.330\.330\.330\.360\.36Model\-visible vocabulary sizeNvisibleN\_\{\\mathrm\{visible\}\}\(b\) Corpus IIStandard BPE:Nvisible→Nvisible\+2​KN\_\{\\mathrm\{visible\}\}\\\!\\rightarrow\\\!N\_\{\\mathrm\{visible\}\}\+2\\mathrm\{K\}Pruned BPE: FixedNvisibleN\_\{\\mathrm\{visible\}\}12K14K16K18K2\.522\.522\.102\.101\.791\.791\.541\.540\.270\.270\.280\.280\.280\.280\.270\.27Model\-visible vocabulary sizeNvisibleN\_\{\\mathrm\{visible\}\}\(c\) Corpus I \+ Corpus IIFigure 1:Comparison of Standard BPE and Pruned BPE for reducing encoded lengthFigure[1](https://arxiv.org/html/2608.00837#S4.F1)compares two different ways of reducing encoded length\. Each panel corresponds to one training corpus: Corpus I, Corpus II, or the combined Corpus I \+ Corpus II\. At each model\-visible vocabulary sizeNvisibleN\_\{\\mathrm\{visible\}\}, the Standard BPE bar shows the marginal encoded\-length reduction obtained by increasing the vocabulary fromNvisibleN\_\{\\mathrm\{visible\}\}toNvisible\+2​KN\_\{\\mathrm\{visible\}\}\+2\\mathrm\{K\}\. In contrast, the Pruned BPE bar shows the additional reduction obtained relative to Standard BPE at the same model\-visible vocabulary sizeNvisibleN\_\{\\mathrm\{visible\}\}, using a 40% exposure threshold\. Thus, the Standard BPE bars measure the benefit of adding 2K more model\-visible tokens, whereas the Pruned BPE bars measure the benefit of reallocating the existing vocabulary capacity without increasing its size\. Each tokenizer was evaluated on the same corpus used for its training\.

Although the absolute encoded\-length reduction achieved by Pruned BPE relative to Standard BPE may appear small, it should be interpreted in relation to the diminishing marginal gain obtained by expanding the Standard BPE vocabulary\. Across the configurations shown in Figure[1](https://arxiv.org/html/2608.00837#S4.F1), increasing the Standard BPE vocabulary by 2K tokens reduces encoded length by approximately 1\.5%–3\.8%, while Pruned BPE provides an additional reduction of approximately 0\.27%–0\.36% under the same model\-visible vocabulary budget\. In later vocabulary ranges, where the marginal benefit of adding another 2K Standard BPE tokens is often only about 2%, an additional reduction of roughly 0\.3% from pruning and token reallocation represents a meaningful fraction of that remaining gain\. The figure therefore shows that Pruned BPE can improve encoding efficiency without requiring a larger model\-visible vocabulary\.

### 4\.6Vocabulary\-only Minimum\-token Evaluation

The preceding experiments use each tokenizer’s native encoding procedure\. Standard BPE applies learned merge rules according to their merge ranks, while Pruned BPE applies the retained merge rules and then recursively expands internal\-only tokens\. However, vocabulary construction and tokenizer inference are distinct components of a tokenization system, and different inference procedures may produce different segmentations from the same vocabulary\[[20](https://arxiv.org/html/2608.00837#bib.bib20)\]\. Schmidt et al\. introduced PathPiece, which uses minimum\-token segmentation for a given vocabulary\[[21](https://arxiv.org/html/2608.00837#bib.bib21)\]\. Motivated by this line of work, an additional vocabulary\-only evaluation was performed to determine whether the advantage of Pruned BPE persists when both visible vocabularies are encoded using the same minimum\-token objective\.

The Standard BPE and Pruned BPE tokenizers trained on Corpus I \+ Corpus II at model\-visible vocabulary sizes of 12K, 14K, 16K, and 18K were evaluated using an independently implemented minimum\-token dynamic\-programming \(DP\) segmentation algorithm described below\. The 40% exposure\-threshold configurations were used for Pruned BPE, consistent with the main comparison in Subsection[4\.5](https://arxiv.org/html/2608.00837#S4.SS5)\. Each tokenizer was evaluated on Corpus I, Corpus II, and their combination\.

For this evaluation, only the token strings in the corresponding model\-visiblevocab\.txtfile were made available to the encoder\. Merge ranks, merge\-tree child relationships, token\-frequency statistics, and Pruned BPE’s internal\-only vocabulary were not used\. In particular, tokens frominter\_vocab\.txtwere excluded because they are not part of the vocabulary exposed to the downstream model\. The same pretokenization procedure described in Subsection[4\.2](https://arxiv.org/html/2608.00837#S4.SS2)was applied before both native BPE encoding and minimum\-token DP encoding\. The optimization is therefore exact within each shared pretokenization boundary rather than across boundaries that neither tokenizer is permitted to cross\.

For a pretokenized byte sequencexxof lengthnn, letD​\(i\)D\(i\)denote the minimum number of visible tokens required to encode the suffix beginning at byte positionii\. The base case isD​\(n\)=0D\(n\)=0, and, for0≤i<n0\\leq i<n, the recurrence is

D​\(i\)=mint∈Vxi:i\+\|t\|=t⁡\{1\+D​\(i\+\|t\|\)\}\.D\(i\)=\\min\_\{\\begin\{subarray\}\{c\}t\\in V\\\\ x\_\{i:i\+\|t\|\}=t\\end\{subarray\}\}\\left\\\{1\+D\\bigl\(i\+\|t\|\\bigr\)\\right\\\}\.
Here,VVis the tokenizer’s model\-visible vocabulary andxi:i\+\|t\|x\_\{i:i\+\|t\|\}is the byte sequence beginning at positioniiwith the same length as tokentt\. Because all 256 individual byte tokens are visible, at least one valid transition is available at every position\. The values are computed from right to left, and a trie is used to enumerate the vocabulary tokens beginning at each position without scanning the complete vocabulary\. Ifℓmax\\ell\_\{\\max\}is the maximum visible\-token length, the encoding requiresO​\(n​ℓmax\)O\(n\\ell\_\{\\max\}\)time andO​\(n\)O\(n\)DP storage, excluding the vocabulary trie\.

LetLstdDP​\(N\)L\_\{\\mathrm\{std\}\}^\{\\mathrm\{DP\}\}\(N\)andLpruned,40DP​\(N\)L\_\{\\mathrm\{pruned\},40\}^\{\\mathrm\{DP\}\}\(N\)denote the encoded lengths produced by this shared DP encoder using, respectively, the Standard BPE and 40%\-threshold Pruned BPE visible vocabularies of sizeNN\. The Pruned BPE reduction under the shared DP encoder is defined as

ΔDP​\(N\)=LstdDP​\(N\)−Lpruned,40DP​\(N\)LstdDP​\(N\)×100%\.\\Delta\_\{\\mathrm\{DP\}\}\(N\)=\\frac\{L\_\{\\mathrm\{std\}\}^\{\\mathrm\{DP\}\}\(N\)\-L\_\{\\mathrm\{pruned\},40\}^\{\\mathrm\{DP\}\}\(N\)\}\{L\_\{\\mathrm\{std\}\}^\{\\mathrm\{DP\}\}\(N\)\}\\times 100\\%\.
Figure[2](https://arxiv.org/html/2608.00837#S4.F2)compares this reduction with the reduction obtained using the tokenizers’ native BPE\-based encoding procedures\.

12K14K16K18K00\.10\.10\.20\.20\.30\.30\.40\.40\.270\.270\.260\.260\.240\.240\.220\.220\.290\.290\.260\.260\.250\.250\.230\.23Model\-visible vocabulary sizeNvisibleN\_\{\\mathrm\{visible\}\}Encoded\-length reduction \(%\)\(a\) Corpus I12K14K16K18K0\.270\.270\.290\.290\.300\.300\.300\.300\.290\.290\.300\.300\.310\.310\.310\.31Model\-visible vocabulary sizeNvisibleN\_\{\\mathrm\{visible\}\}\(b\) Corpus IINative BPE encodingMinimum\-token DP encoding12K14K16K18K0\.270\.270\.280\.280\.280\.280\.270\.270\.290\.290\.290\.290\.290\.290\.280\.28Model\-visible vocabulary sizeNvisibleN\_\{\\mathrm\{visible\}\}\(c\) Corpus I \+ Corpus IIFigure 2:Comparison of native BPE encoding and vocabulary\-only minimum\-token DP encodingThe minimum\-token DP encoder produced fewer tokens than the corresponding native encoder for every Standard BPE and Pruned BPE vocabulary\. Because this comparison is not directly related to the proposed pruning method, the detailed reductions are not reported here\.

More importantly, Pruned BPE continued to produce shorter sequences in all 12 comparisons under the shared minimum\-token objective\. On the combined evaluation corpus, the reduction relative to Standard BPE was approximately 0\.28%–0\.29%\. On Corpus I, it ranged from approximately 0\.23% to 0\.29%, while on Corpus II it ranged from approximately 0\.29% to 0\.31%\. Across all configurations, the DP\-based reduction ranged from approximately 0\.23% to 0\.31%, compared with approximately 0\.22% to 0\.30% under native BPE encoding\.

Because both visible vocabularies were evaluated using the same segmentation objective, and because the DP encoder did not use merge ranks or internal\-only merge nodes, the remaining improvement cannot be attributed solely to Standard BPE’s merge\-order encoding procedure\. Instead, the result provides evidence that visibility pruning and token reallocation improve the composition of the model\-visible vocabulary itself\. This evaluation does not establish independence from every possible encoding algorithm, but it shows that the improvement persists when native merge\-order effects are removed through the shared exact minimum\-token DP encoder\.

### 4\.7Qualitative Token Analysis

The Pruned BPE tokenizer trained on the combined corpus with an 18K target vocabulary and a 40% exposure threshold was selected for qualitative inspection because it contains the largest model\-visible vocabulary and a large set of internal\-only tokens among the evaluated configurations\. It therefore provides the broadest range of examples for examining the structural roles of internal\-only tokens\. All 419 internal\-only tokens in this configuration remain direct or indirect components of model\-visible tokens\. Thus, although these tokens are not exposed to the language model, they are retained as functional nodes in the merge structure\.

Many English examples are intermediate word fragments that are largely absorbed into longer visible tokens\. For example,“ecause”is internal\-only but is used directly to construct the visible tokens“Because”,“Because”, and“because”\. Similarly,“vironment”supports“environment”and“Environment”\. Some internal fragments are shared by several different words:“artment”contributes to“Department”,“department”, and“apartment”, while“ording”contributes to both“according”and“recording”\. These examples show that an internal\-only token may remain useful as a reusable merge component even when exposing it as an independent model token provides limited benefit\.

The Chinese tokens exhibit similar behavior at both the character and multi\-character levels\. For example, the internal token\\cjkcodefont人民共和 is extended into the internal tokens\\cjkcodefont人民共和国 and\\cjkcodefont人民共和國, which in turn support the visible tokens\\cjkcodefont中华人民共和国 and\\cjkcodefont中華人民共和國, respectively\. Other examples include\\cjkcodefont西班 in\\cjkcodefont西班牙,\\cjkcodefont澳大利 in\\cjkcodefont澳大利亚, and\\cjkcodefont俱乐 in\\cjkcodefont俱乐部\. Even complete characters may be internal\-only when they occur primarily as components of longer visible tokens\. For example, both\\cjkcodefont尴 and\\cjkcodefont尬 are internal\-only components of the visible token\\cjkcodefont尴尬\. A multi\-level example is provided by\\cjkcodefont葡萄牙: the internal\-only tokens\\cjkcodefont葡 and\\cjkcodefont萄 are first combined to form the visible token\\cjkcodefont葡萄, which is then used to construct the longer visible token\\cjkcodefont葡萄牙\. This example also illustrates that a token may remain model\-visible while serving as an intermediate component of another visible token\.

Because the tokenizer operates on bytes, some internal\-only tokens correspond to partial UTF\-8 sequences rather than independently decodable text\. For example, the internal byte sequence“83 BD”is shared by the visible Chinese characters\\cjkcodefont能, encoded as“E8 83 BD”, and\\cjkcodefont都, encoded as“E9 83 BD”\. Similarly, the byte prefix“E9 BC”is used in visible tokens representing several different Chinese characters, including\\cjkcodefont鼓,\\cjkcodefont鼠,\\cjkcodefont鼻, and\\cjkcodefont鼎\. The byte prefix“E2 98”likewise supports the black\-star and white\-star Unicode symbols, represented by“U\+2605”and“U\+2606”, respectively\. Such fragments are necessary for preserving the learned byte\-level merge structure but are poor candidates for occupying independent model\-visible vocabulary slots\.

A smaller number of internal tokens reflect structured web text\. For example,“htt”supports both“http”and“https”, while“www”supports the visible token“www\.”; the leading spaces and the final period shown inside the quotation marks are part of the corresponding tokens\. These examples indicate that visibility pruning is not limited to ordinary word morphology; it can also reclaim model\-visible vocabulary slots occupied by intermediate fragments arising from multilingual byte sequences and structured text\.

Not all internal\-only tokens are obviously uninformative fragments\. One example isournal, which was also noted in the Introduction\. It was not classified as an internal\-only token by the tokenizer examined above, but it was classified as internal\-only by the tokenizer trained on Corpus I with a 14K target vocabulary and a 40% exposure threshold, even though it is a recognizable component of words such asjournal,journalism, andjournalist\. This does not necessarily indicate an incorrect pruning decision: once longer tokens containing this fragment have been learned,ournalmay have little final exposure as an independently emitted token\. Moreover, its classification is configuration\-dependent; a token that is internal\-only for one training corpus, target vocabulary size, or exposure threshold may remain model\-visible under another configuration\. This example highlights that visibility pruning is determined by empirical final exposure rather than by whether a token appears linguistically meaningful in isolation\.

## 5Discussion and Future Work

Pruned BPE introduces several tradeoffs and limitations\. It requires additional tokenizer training to produce enough eligible replacement tokens, and its effectiveness depends on the selected exposure threshold and on how well the tokenizer\-training corpus represents downstream text\. A threshold that is too high may cause useful tokens to be decomposed more often than the replacement candidates can compensate for\. In addition, the present experiments evaluate tokenizer\-level encoded length only\. They do not yet establish whether Pruned BPE improves language\-model training efficiency, validation loss, downstream task performance, or generation quality\.

The vocabulary\-only minimum\-token evaluation provides additional evidence about the source of the encoded\-length improvement\. When the Standard BPE and Pruned BPE visible vocabularies were evaluated under the same exact minimum\-token objective, Pruned BPE continued to produce shorter sequences in every tested configuration\. Because this evaluation did not use native merge ranks or Pruned BPE’s internal\-only merge nodes, the result provides evidence that visibility pruning and token reallocation improve the composition of the model\-visible vocabulary itself, rather than producing an advantage solely through the native encoding procedure\. The minimum\-token DP encoder was used as a diagnostic evaluation rather than as the proposed deployment encoder\. The experiment therefore does not establish superiority under every possible inference method, nor does it address tokenizer runtime, language\-model quality, or downstream training efficiency\.

The most important direction for future work is controlled downstream language\-model evaluation\. Models with the same architecture, training data, optimization settings, and model\-visible vocabulary size could be trained using Standard BPE and Pruned BPE tokenizers\. Such experiments could compare validation loss, convergence behavior, training throughput, memory use, and downstream task performance\. They could also test whether replacing rarely emitted visible tokens with better\-exposed tokens leads to more effectively trained embeddings or a more balanced distribution of token usage\.

Future work should also investigate automatic threshold selection and broader evaluation settings\. Instead of selecting a fixed exposure ratio manually, the threshold could be chosen using held\-out data or by estimating the tradeoff between the expansion cost of making a token internal\-only and the compression benefit of the replacement token that occupies its visible slot\. Additional experiments should cover larger multilingual corpora, code\-focused datasets, and wider vocabulary ranges\. Because visibility decisions are made after the merge structure has been trained, another possible direction is to export different domain\-specific model\-visible vocabularies from the same underlying BPE merge tree\.

## 6Conclusion

This paper introduced Pruned BPE, a post\-training visibility\-pruning and token\-reallocation method that separates the BPE merge structure from the model\-visible vocabulary\. Low\-exposure learned tokens remain available as internal merge nodes, while their visible vocabulary positions are reassigned to better\-exposed candidates obtained through resumed Standard BPE training\. Experiments across two corpora and their combination showed consistent encoded\-length reductions relative to Standard BPE under the same model\-visible vocabulary budget\. The improvement also persisted when both visible vocabularies were evaluated under a shared exact minimum\-token objective, providing evidence that the gain arises from more efficient model\-visible vocabulary composition rather than solely from native BPE merge\-order encoding\. Although downstream language\-model evaluation remains necessary, the results demonstrate that final exposure provides a practical basis for improving the allocation efficiency of a fixed BPE vocabulary\.

## References

- \[1\]Philip Gage\.A new algorithm for data compression\.*The C Users Journal*, 12\(2\):23–38, 1994\.
- \[2\]Rico Sennrich, Barry Haddow, and Alexandra Birch\.Neural machine translation of rare words with subword units\.In*Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics*, pages 1715–1725, 2016\.
- \[3\]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever\.Language models are unsupervised multitask learners\.OpenAI technical report, 2019\.
- \[4\]Meta AI\.Llama 3 tokenizer implementation\.*GitHub repository: meta\-llama/llama3*, 2024\.
- \[5\]Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, and others\.Qwen technical report\.*arXiv preprint arXiv:2309\.16609*, 2023\.
- \[6\]Qwen Team\.Qwen3 technical report\.*arXiv preprint arXiv:2505\.09388*, 2025\.
- \[7\]Mistral AI\.Tokenization: v3 tokenizer\.Mistral AI documentation, 2024\.
- \[8\]Tom B\. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert\-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M\. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei\.Language models are few\-shot learners\.In*Advances in Neural Information Processing Systems*, 33:1877–1901, 2020\.
- \[9\]Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov\.RoBERTa: A robustly optimized BERT pretraining approach\.*arXiv preprint arXiv:1907\.11692*, 2019\.
- \[10\]Taku Kudo and John Richardson\.SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing\.In*Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations*, pages 66–71, 2018\.
- \[11\]Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita\.BPE\-Dropout: Simple and effective subword regularization\.In*Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 1882–1892, 2020\.
- \[12\]Haoran Lian, Yizhe Xiong, Jianwei Niu, Shasha Mo, Zhenpeng Su, Zijia Lin, Hui Chen, Jungong Han, and Guiguang Ding\.Scaffold\-BPE: Enhancing byte pair encoding for large language models with simple and effective scaffold token removal\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, volume 39, number 23, pages 24539–24548, 2025\.[https://doi\.org/10\.1609/aaai\.v39i23\.34633](https://doi.org/10.1609/aaai.v39i23.34633)\.
- \[13\]Marco Cognetta, Tatsuya Hiraoka, Naoaki Okazaki, Rico Sennrich, and Yuval Pinter\.An analysis of BPE vocabulary trimming in neural machine translation\.*arXiv preprint arXiv:2404\.00397*, 2024\.
- \[14\]Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro von Werra, and Thomas Wolf\.The FineWeb datasets: Decanting the web for the finest text data at scale\.In*Advances in Neural Information Processing Systems*, volume 37, pages 30811–30849, 2024\.
- \[15\]Dewarim\.Updated Reddit comment dataset as torrents\.Redditr/datasets, 2017\.Available at[https://www\.reddit\.com/r/datasets/comments/65o7py/updated\_reddit\_comment\_dataset\_as\_torrents/](https://www.reddit.com/r/datasets/comments/65o7py/updated_reddit_comment_dataset_as_torrents/)\.
- \[16\]Common Crawl Foundation\.News dataset available\.Common Crawl Blog, 2016\.Available at[https://commoncrawl\.org/blog/news\-dataset\-available](https://commoncrawl.org/blog/news-dataset-available)\.
- \[17\]Vladimir Blagojevic\.cc\_news\.Hugging Face Datasets\.Available at[https://huggingface\.co/datasets/vblagoje/cc\_news](https://huggingface.co/datasets/vblagoje/cc_news)\.
- \[18\]Zhipeng Guo, Yu Zhao, Yabin Zheng, Xiance Si, Zhiyuan Liu, and Maosong Sun\.THUCTC: An Efficient Chinese Text Classifier\.GitHub repository, 2016\.Available at[https://github\.com/thunlp/THUCTC](https://github.com/thunlp/THUCTC)\.
- \[19\]Kenny Shao\.Pruned BPE Tokenizer: Python/Cython implementation, corpus data, and trained tokenizers\.GitHub repository, 2026\.Available at[https://github\.com/Kshao09/Pruned\-BPE\-Tokenizer](https://github.com/Kshao09/Pruned-BPE-Tokenizer)\.
- \[20\]Omri Uzan, Craig W\. Schmidt, Chris Tanner, and Yuval Pinter\.Greed is all you need: An evaluation of tokenizer inference methods\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\)*, pages 813–822, 2024\.
- \[21\]Craig W\. Schmidt, Varshini Reddy, Haoran Zhang, Alec Alameddine, Omri Uzan, Yuval Pinter, and Chris Tanner\.Tokenization is more than compression\.In*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 678–702, 2024\.