A Pilot Study of Autocompleting Tokenizers

# A Pilot Study of Autocompleting Tokenizers
Source: [https://arxiv.org/html/2608.15080](https://arxiv.org/html/2608.15080)
## A Pilot Study of Autocompleting TokenizersThanks:Accepted to the Second Tokenization Workshop \(TokShop\) at COLM 2026\.

Samuel Wexler & Mark HopkinsAffiliation:Department of Computer ScienceAffiliation:Williams CollegeAffiliation:Williamstown, MA 01267, USAEmail:[\{saw9,mh24\}@williams\.edu](mailto:)

###### Abstract

Modern input methods routinely rely on autocomplete to omit information that can be recovered from local context\. Inspired by these autocomplete\-assisted writing systems, we investigate whether Transformer inputs can be compressed in a similar manner\. Byte\-level tokenization offers a simple and language\-independent alternative to subword tokenization, but its longer input sequences typically result in increased computational cost and reduced model quality\. We propose a compression scheme that employs a lightweight autoregressive byte language model to identify and remove bytes that are easily predictable from their surrounding context before Transformer processing\. The resulting compressed representation is then provided as input to a standard encoder–decoder Transformer\. Experiments on machine translation show that a substantial fraction of source\-language bytes can be omitted without degrading translation quality\. On English–French, our best method preserves translation performance while reducing source sequence length by nearly one\-third\. Additional experiments on Finnish–English, Russian–English, and Chinese–English demonstrate that the approach generalizes across diverse writing systems and morphological typologies, yielding comparable or improved translation quality at compression ratios between 0\.47 and 0\.67\. These findings suggest that many input bytes are predictable enough to be represented implicitly rather than explicitly, providing a simple mechanism for reducing the sequence\-length overhead associated with byte\-level models\.

## 1Introduction

Subword tokenization\([13](https://arxiv.org/html/2608.15080#bib.bib2)\)remains the predominant approach for Transformer\-based language models\([17](https://arxiv.org/html/2608.15080#bib.bib9);[3](https://arxiv.org/html/2608.15080#bib.bib5);[16](https://arxiv.org/html/2608.15080#bib.bib7)\), but there has been growing interest in byte\-level and tokenizer\-free alternatives\([18](https://arxiv.org/html/2608.15080#bib.bib10);[4](https://arxiv.org/html/2608.15080#bib.bib11);[15](https://arxiv.org/html/2608.15080#bib.bib12);[19](https://arxiv.org/html/2608.15080#bib.bib13);[8](https://arxiv.org/html/2608.15080#bib.bib14)\)\. Potential benefits include a smaller embedding matrix \(allowing those parameters to be reallocated to the core Transformer layers\) and a more uniform treatment of diverse writing systems\. However, byte tokenization produces considerably longer token sequences than subword tokenization, which typically results in inferior performance \(in terms of runtime, memory consumption, and model quality\)\.

This paper presents a pilot study of a new compression\-based input representation that seeks to retain the simplicity and language independence of byte\-level representations while mitigating their sequence\-length overhead\.The core idea is to leverage an auxiliary autocomplete model to remove predictable bytes from the original input sequence\.

Humans routinely communicate through autocomplete\-assisted input systems that omit information recoverable from context\. We investigate whether a similar principle can be applied to Transformer inputs\. The approach is specifically inspired by QWERTY keyboard input methods for Chinese\([7](https://arxiv.org/html/2608.15080#bib.bib4)\)and the smartphone typing habits of Latin\-script language speakers \-\-\- both of which typically rely on autocomplete mechanisms\. In both cases, the actual sequence of typed characters does not match standard orthography\. Instead, it looks like short bursts of Latin letters111Note that all Latin letters are represented as a single byte with UTF\-8\.punctuated by autocompletion triggers\.[7](https://arxiv.org/html/2608.15080#bib.bib4)refers to this as*hypography*— from the Greek roots*hypo*\(meaning “below”\) and*graphy*\(meaning “writing”\) — and defines it as “writing that operates in service of conventional writing or script but at a register beneath\.”

Figure[1](https://arxiv.org/html/2608.15080#S1.F1)shows one variant of our proposed approach\. The general approach has three stages:

1. 1\.Prediction:The original byte sequence is provided as input to a \(small\) autocomplete model, which auto\-regressively predicts the next byte given the previous bytes\.
2. 2\.Compression:Byte sequences that were confidently predicted by the autocomplete model are removed\. Sentinels are \(possibly\) inserted to indicate where bytes were removed\.
3. 3\.Translation:The compressed sequence is provided to a \(large\) Transformer and translated into a desired output\.

![Refer to caption](https://arxiv.org/html/2608.15080v1/demo1.png)Figure 1:One variant \(“length\-encoded sentinel”\) of our proposed technique\. The original byte sequence is provided as input to an autocomplete model, which auto\-regressively predicts the next byte given the previous bytes\. Then, byte sequences that were confidently predicted by the autocomplete model are removed\. Sentinels are \(possibly\) inserted to indicate where bytes were removed\. Finally, the compressed sequence is ingested by a Transformer\.Our experiments demonstrate that this approach can substantially reduce source sequence length while preserving translation quality\. On English–French, we achieve parity with uncompressed byte tokenization after removing nearly one\-third of the source bytes\. Across additional language pairs spanning agglutinative \(Finnish\), Cyrillic \(Russian\), and logographic \(Chinese\) writing systems, we observe comparable or improved translation performance at compression ratios ranging from 0\.47 to 0\.67\. These findings suggest that a substantial fraction of input bytes are predictable enough to be reconstructed from local context and therefore omitted prior to Transformer processing without degrading translation quality\.

Our contributions are as follows:

- •We introduce a compression technique that removes highly predictable bytes prior to Transformer processing\.
- •We compare multiple variants of this technique, identifying design choices that best preserve translation quality under aggressive sequence compression\.
- •We evaluate the proposed technique across language pairs spanning multiple writing systems and morphological typologies\.

## 2Related Work

Byte\-level language modeling has emerged as an attractive alternative to subword tokenization\. By operating directly on UTF\-8 bytes, byte\-level models avoid language\-specific tokenizers, eliminate out\-of\-vocabulary issues, and provide a unified representation across writing systems\. However, these benefits come at the cost of substantially longer sequences, increasing the computational burden of Transformer architectures\.

A substantial body of work has therefore focused on improving the efficiency of byte\-level Transformers\. ByT5\([18](https://arxiv.org/html/2608.15080#bib.bib10)\)demonstrated that competitive multilingual performance can be achieved using a pure byte\-level representation\. Subsequent approaches sought to mitigate the resulting sequence\-length explosion through architectural modifications\. CANINE\([4](https://arxiv.org/html/2608.15080#bib.bib11)\)employs learned downsampling to shorten sequences before deep Transformer processing, while Charformer\([15](https://arxiv.org/html/2608.15080#bib.bib12)\)learns latent segmentations directly from character sequences\. MegaByte\([19](https://arxiv.org/html/2608.15080#bib.bib13)\)and related hierarchical architectures process byte sequences at multiple temporal scales, enabling efficient modeling of long byte streams\.

Particularly relevant to our work is Byte Latent Transformer \(BLT\)\([8](https://arxiv.org/html/2608.15080#bib.bib14)\), which uses a lightweight byte\-level model to estimate local prediction difficulty and dynamically partition byte streams into variable\-length patches\. Regions that are easy to predict are grouped into larger patches, while more surprising regions are represented with finer granularity\. Both BLT and our approach exploit the observation that information density varies substantially across a byte sequence and that predictable regions require less explicit representation than unpredictable ones\. However, whereas BLT adaptively changes the granularity at which bytes are represented, our method directly removes highly predictable bytes before Transformer processing, yielding a compressed sequence that can be consumed by a standard encoder\-decoder architecture without architectural modification\.

The work most closely related to ours is the neural compression framework of[6](https://arxiv.org/html/2608.15080#bib.bib8)\. Their approach trains a byte\-level language model and uses the resulting probability estimates within an arithmetic coding scheme to generate compressed text representations\. To facilitate Transformer training on these compressed streams, they introduce equal\-information windows and periodic resetting of both the arithmetic coder and language\-model context\. Both their approach and our approach exploit the observation that many bytes are highly predictable from local context and therefore need not be represented explicitly\. But whereas neural compression seeks information\-theoretically efficient encodings that largely discard the original lexical structure, our approach performs selective byte removal while preserving an explicit correspondence between the compressed sequence and the original text\.

Our work is also related to approaches that allocate computation according to input difficulty or information content\. Prior work has shown that substantial computational savings can be achieved by identifying and removing less informative representations during inference or training\. For example, Power\-BERT\([5](https://arxiv.org/html/2608.15080#bib.bib16)\)progressively eliminates token representations within Transformer layers, while DynamicViT\([12](https://arxiv.org/html/2608.15080#bib.bib15)\)dynamically sparsifies token sequences based on estimated importance\. Similar to these approaches, our method seeks to identify portions of the input that can be handled by a lightweight model and reserve the capacity of a larger Transformer for less predictable content\. Unlike prior token\-pruning methods, however, our approach operates directly on raw byte sequences and performs compression as a preprocessing step, producing a shorter input sequence before any Transformer computation occurs\.

Viewed broadly, BLT\([8](https://arxiv.org/html/2608.15080#bib.bib14)\), neurally compressed text\([6](https://arxiv.org/html/2608.15080#bib.bib8)\), and our approach all leverage the predictive distribution of a lightweight byte\-level model to identify regions of text that require less explicit representation\. The three methods differ, however, in how they exploit this signal\. BLT preserves the original information content while adapting the granularity of the representation, grouping predictable regions into larger patches and allocating finer representations to less predictable regions\. At the opposite extreme, neurally compressed text uses the predictive distribution to construct an information\-theoretically compressed encoding that largely replaces the original byte sequence\. Our method occupies a middle ground between these approaches: rather than changing the representational granularity or replacing the input with a compressed code, we selectively remove highly predictable bytes while preserving an explicit correspondence between the compressed sequence and the original text\.This perspective places the three approaches along a continuum ranging from adaptive representation, to selective omission, to full entropy\-based compression\.

## 3Methodology

As discussed in the introduction \(and visualized in Figure[1](https://arxiv.org/html/2608.15080#S1.F1)\), our proposed method first performs autoregressive next\-bytepredictionon a byte sequence, thencompressesthe byte sequence based on these predictions, and finallytranslatesthe compressed sequence using a standard encoder\-decoder Transformer\.

### 3\.1Prediction

We train an autoregressive language modelP𝖺𝖼​\(bk\|b1,…,bk−1\)P\_\{\\mathsf\{ac\}\}\(b\_\{k\}\|b\_\{1\},\\dots,b\_\{k\-1\}\)that predicts thekkth token in a sequence given the previous tokens\. In this case, a token is either a byte or a designated start\-of\-sequence marker□\\squareor end\-of\-sequence marker■\\blacksquare\. In other words, the token vocabulary isℬ=\{0,1,…,255\}∪\{□,■\}\\mathcal\{B\}=\\\{0,1,\.\.\.,255\\\}\\cup\\\{\\square,\\blacksquare\\\}\. The language model can take many forms, but it should be small and fast – the main idea is that it will efficiently show us which parts of a document can be predicted without a deep understanding of the text\.

In this study, we train a two\-layer decoder\-only Transformer withd𝗆𝗈𝖽𝖾𝗅=1024d\_\{\\mathsf\{model\}\}=1024,d𝖿𝖿=512d\_\{\\mathsf\{ff\}\}=512and 16 attention heads per layer\.

Then, given input sequenceb1,…,bmb\_\{1\},\\dots,b\_\{m\}\(whereb1=□b\_\{1\}=\\squareandbm=■b\_\{m\}=\\blacksquare\), we compute the following quantities for all2≤i≤m2\\leq i\\leq m:

pi\\displaystyle p\_\{i\}=\\displaystyle=P𝖺𝖼​\(bi\|b<i\)\\displaystyle P\_\{\\mathsf\{ac\}\}\(b\_\{i\}\|b\_\{<i\}\)p^i\\displaystyle\\hat\{p\}\_\{i\}=\\displaystyle=maxb≠bi⁡P𝖺𝖼​\(b\|b<i\)\\displaystyle\\max\_\{b\\not=b\_\{i\}\}P\_\{\\mathsf\{ac\}\}\(b\|b\_\{<i\}\)In other words, we compute the probabilitypip\_\{i\}of the observed token and the probabilityp^i\\hat\{p\}\_\{i\}of the most probable token \(apart from the observed token\) according to the model\. Note that these probabilities can be computed in parallel using a causal mask\.

### 3\.2Compression

![Refer to caption](https://arxiv.org/html/2608.15080v1/compressions.png)Figure 2:Application of the four compression schemes to the segment “these green apples\.” From upper left \(clockwise\): Length\-Encoded Sentinel, Explicit Sentinel, Implicit Sentinel, Sentinel\-Free\.Next, we use the probabilitiespip\_\{i\}andp^i\\hat\{p\}\_\{i\}to determine which bytes are “easily predictable”\. We experiment with two thresholds:

1. 1\.Absolute Threshold:Bytebib\_\{i\}iseasily predictableifpi≥p^ip\_\{i\}\\geq\\hat\{p\}\_\{i\}andpi\>αp\_\{i\}\>\\alphafor hyperparameterα∈\(0,1\]\\alpha\\in\(0,1\]\. The conditionpi≥p^ip\_\{i\}\\geq\\hat\{p\}\_\{i\}ensures that the observed byte is the model’s most likely prediction\.
2. 2\.Relative Threshold:Bytebib\_\{i\}iseasily predictableifpi−p^i\>αp\_\{i\}\-\\hat\{p\}\_\{i\}\>\\alphafor hyperparameterα∈\(0,1\]\\alpha\\in\(0,1\]\.

Once we have determined the indicesℰ⊆\{2,…,m\}\\mathcal\{E\}\\subseteq\\\{2,\\dots,m\\\}of easily predictable bytes, we proceed to compress the original byte sequence\. We experiment with four compression schemes \(examples of each are provided by Figure[2](https://arxiv.org/html/2608.15080#S3.F2)\):

1. 1\.Sentinel\-free:For everyi∈ℰi\\in\\mathcal\{E\}, bytebib\_\{i\}is simply removed from the byte sequence\.
2. 2\.Explicit Sentinel:For every consecutive sequencei,i\+1,…,j∈ℰi,i\+1,\\dots,j\\in\\mathcal\{E\}, wherei−1∉ℰi\-1\\not\\in\\mathcal\{E\}andj\+1∉ℰj\+1\\not\\in\\mathcal\{E\}, we replace the byte subsequencebi,bi\+1,…​bjb\_\{i\},b\_\{i\+1\},\\dots b\_\{j\}with a special sentinel∇\\nabla\.
3. 3\.Length\-encoded Sentinel:For every consecutive sequencei,i\+1,…,j∈ℰi,i\+1,\\dots,j\\in\\mathcal\{E\}, wherei−1∉ℰi\-1\\not\\in\\mathcal\{E\}andj\+1∉ℰj\+1\\not\\in\\mathcal\{E\}, we replace the byte subsequencebi,bi\+1,…​bjb\_\{i\},b\_\{i\+1\},\\dots b\_\{j\}with a special sentinelk, wherek=j−i\+1k=j\-i\+1\(i\.e\., the number of bytes replaced\)\.
4. 4\.Implicit Sentinel:For everyi∈ℰi\\in\\mathcal\{E\}, bytebib\_\{i\}is removed from the byte sequence\. For everyi∉ℰi\\not\\in\\mathcal\{E\}such thati\+1∈ℰi\+1\\in\\mathcal\{E\}, we replacebib\_\{i\}withbi\+256b\_\{i\}\+256\.

### 3\.3Translation

We then provide the compressed token sequencex1,…,xnx\_\{1\},\\dots,x\_\{n\}as input to a standard encoder\-decoder Transformer\. The input token vocabulary depends on the compression scheme\. For thesentinel\-freecompression scheme, the token vocabulary𝒱𝗌𝖿=\{0,1,…,255\}∪\{□,■,Pad\}\\mathcal\{V\}\_\{\\mathsf\{sf\}\}=\\\{0,1,\.\.\.,255\\\}\\cup\\\{\\square,\\blacksquare,\\textsc\{Pad\}\\\}has 259 tokens: one for each possible byte, plus a start\-of\-sequence marker, an end\-of\-sequence marker, and a pad token\. Theexplicit sentineltoken vocabulary adds the sentinel∇\\nablato𝒱𝗌𝖿\\mathcal\{V\}\_\{\\mathsf\{sf\}\}\. Thelength\-encoded sentineltoken vocabulary adds sentinels1,2,3,4,5,6,7,8\\hbox to11\.37pt\{\\vbox to11\.37pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 5\.68657pt\\lower\-5\.68657pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\}\}\\lx@inpgf@ignorespaces\\hbox\{\\hbox\{\{\\lxSVG@begingroup@\{\_scopebegin\} \{\{\}\{\{\{\}\}\}\{\{\}\}\{\}\{\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\}\{\}\{\}\{\}\{\}\{\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 7\.59 0 C 7\.59 4\.19 4\.19 7\.59 0 7\.59 C \-4\.19 7\.59 \-7\.59 4\.19 \-7\.59 0 C \-7\.59 \-4\.19 \-4\.19 \-7\.59 0 \-7\.59 C 4\.19 \-7\.59 7\.59 \-4\.19 7\.59 0 Z M 0 0\}\{fill:none\} \\lx@inpgf@ignorespaces \}\{\{\{\{\\lx@inpgf@ignorespaces\}\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-2\.5pt\}\{\-3\.22221pt\}\\lxSVG@begingroup@\{transform\} \\pgfsys@hbox\{58\}\\lxSVG@closescope \}\}\} \\lxSVG@closescope \}\}\} \} \\lxSVG@closescope \{\{\{\}\}\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\},\\hbox to11\.37pt\{\\vbox to11\.37pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 5\.68657pt\\lower\-5\.68657pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\}\}\\lx@inpgf@ignorespaces\\hbox\{\\hbox\{\{\\lxSVG@begingroup@\{\_scopebegin\} \{\{\}\{\{\{\}\}\}\{\{\}\}\{\}\{\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\}\{\}\{\}\{\}\{\}\{\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 7\.59 0 C 7\.59 4\.19 4\.19 7\.59 0 7\.59 C \-4\.19 7\.59 \-7\.59 4\.19 \-7\.59 0 C \-7\.59 \-4\.19 \-4\.19 \-7\.59 0 \-7\.59 C 4\.19 \-7\.59 7\.59 \-4\.19 7\.59 0 Z M 0 0\}\{fill:none\} \\lx@inpgf@ignorespaces \}\{\{\{\{\\lx@inpgf@ignorespaces\}\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-2\.5pt\}\{\-3\.22221pt\}\\lxSVG@begingroup@\{transform\} \\pgfsys@hbox\{58\}\\lxSVG@closescope \}\}\} \\lxSVG@closescope \}\}\} \} \\lxSVG@closescope \{\{\{\}\}\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\},\\hbox to11\.37pt\{\\vbox to11\.37pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 5\.68657pt\\lower\-5\.68657pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\}\}\\lx@inpgf@ignorespaces\\hbox\{\\hbox\{\{\\lxSVG@begingroup@\{\_scopebegin\} \{\{\}\{\{\{\}\}\}\{\{\}\}\{\}\{\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\}\{\}\{\}\{\}\{\}\{\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 7\.59 0 C 7\.59 4\.19 4\.19 7\.59 0 7\.59 C \-4\.19 7\.59 \-7\.59 4\.19 \-7\.59 0 C \-7\.59 \-4\.19 \-4\.19 \-7\.59 0 \-7\.59 C 4\.19 \-7\.59 7\.59 \-4\.19 7\.59 0 Z M 0 0\}\{fill:none\} \\lx@inpgf@ignorespaces \}\{\{\{\{\\lx@inpgf@ignorespaces\}\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-2\.5pt\}\{\-3\.22221pt\}\\lxSVG@begingroup@\{transform\} \\pgfsys@hbox\{58\}\\lxSVG@closescope \}\}\} \\lxSVG@closescope \}\}\} \} \\lxSVG@closescope \{\{\{\}\}\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\},\\hbox to11\.37pt\{\\vbox to11\.37pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 5\.68657pt\\lower\-5\.68657pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\}\}\\lx@inpgf@ignorespaces\\hbox\{\\hbox\{\{\\lxSVG@begingroup@\{\_scopebegin\} \{\{\}\{\{\{\}\}\}\{\{\}\}\{\}\{\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\}\{\}\{\}\{\}\{\}\{\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 7\.59 0 C 7\.59 4\.19 4\.19 7\.59 0 7\.59 C \-4\.19 7\.59 \-7\.59 4\.19 \-7\.59 0 C \-7\.59 \-4\.19 \-4\.19 \-7\.59 0 \-7\.59 C 4\.19 \-7\.59 7\.59 \-4\.19 7\.59 0 Z M 0 0\}\{fill:none\} \\lx@inpgf@ignorespaces \}\{\{\{\{\\lx@inpgf@ignorespaces\}\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-2\.5pt\}\{\-3\.22221pt\}\\lxSVG@begingroup@\{transform\} \\pgfsys@hbox\{58\}\\lxSVG@closescope \}\}\} \\lxSVG@closescope \}\}\} \} \\lxSVG@closescope \{\{\{\}\}\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\},\\hbox to11\.37pt\{\\vbox to11\.37pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 5\.68657pt\\lower\-5\.68657pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\}\}\\lx@inpgf@ignorespaces\\hbox\{\\hbox\{\{\\lxSVG@begingroup@\{\_scopebegin\} \{\{\}\{\{\{\}\}\}\{\{\}\}\{\}\{\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\}\{\}\{\}\{\}\{\}\{\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 7\.59 0 C 7\.59 4\.19 4\.19 7\.59 0 7\.59 C \-4\.19 7\.59 \-7\.59 4\.19 \-7\.59 0 C \-7\.59 \-4\.19 \-4\.19 \-7\.59 0 \-7\.59 C 4\.19 \-7\.59 7\.59 \-4\.19 7\.59 0 Z M 0 0\}\{fill:none\} \\lx@inpgf@ignorespaces \}\{\{\{\{\\lx@inpgf@ignorespaces\}\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-2\.5pt\}\{\-3\.22221pt\}\\lxSVG@begingroup@\{transform\} \\pgfsys@hbox\{58\}\\lxSVG@closescope \}\}\} \\lxSVG@closescope \}\}\} \} \\lxSVG@closescope \{\{\{\}\}\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\},\\hbox to11\.37pt\{\\vbox to11\.37pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 5\.68657pt\\lower\-5\.68657pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\}\}\\lx@inpgf@ignorespaces\\hbox\{\\hbox\{\{\\lxSVG@begingroup@\{\_scopebegin\} \{\{\}\{\{\{\}\}\}\{\{\}\}\{\}\{\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\}\{\}\{\}\{\}\{\}\{\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 7\.59 0 C 7\.59 4\.19 4\.19 7\.59 0 7\.59 C \-4\.19 7\.59 \-7\.59 4\.19 \-7\.59 0 C \-7\.59 \-4\.19 \-4\.19 \-7\.59 0 \-7\.59 C 4\.19 \-7\.59 7\.59 \-4\.19 7\.59 0 Z M 0 0\}\{fill:none\} \\lx@inpgf@ignorespaces \}\{\{\{\{\\lx@inpgf@ignorespaces\}\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-2\.5pt\}\{\-3\.22221pt\}\\lxSVG@begingroup@\{transform\} \\pgfsys@hbox\{58\}\\lxSVG@closescope \}\}\} \\lxSVG@closescope \}\}\} \} \\lxSVG@closescope \{\{\{\}\}\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\},\\hbox to11\.37pt\{\\vbox to11\.37pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 5\.68657pt\\lower\-5\.68657pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\}\}\\lx@inpgf@ignorespaces\\hbox\{\\hbox\{\{\\lxSVG@begingroup@\{\_scopebegin\} \{\{\}\{\{\{\}\}\}\{\{\}\}\{\}\{\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\}\{\}\{\}\{\}\{\}\{\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 7\.59 0 C 7\.59 4\.19 4\.19 7\.59 0 7\.59 C \-4\.19 7\.59 \-7\.59 4\.19 \-7\.59 0 C \-7\.59 \-4\.19 \-4\.19 \-7\.59 0 \-7\.59 C 4\.19 \-7\.59 7\.59 \-4\.19 7\.59 0 Z M 0 0\}\{fill:none\} \\lx@inpgf@ignorespaces \}\{\{\{\{\\lx@inpgf@ignorespaces\}\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-2\.5pt\}\{\-3\.22221pt\}\\lxSVG@begingroup@\{transform\} \\pgfsys@hbox\{58\}\\lxSVG@closescope \}\}\} \\lxSVG@closescope \}\}\} \} \\lxSVG@closescope \{\{\{\}\}\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\},\\hbox to11\.37pt\{\\vbox to11\.37pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 5\.68657pt\\lower\-5\.68657pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\}\}\\lx@inpgf@ignorespaces\\hbox\{\\hbox\{\{\\lxSVG@begingroup@\{\_scopebegin\} \{\{\}\{\{\{\}\}\}\{\{\}\}\{\}\{\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\}\{\}\{\}\{\}\{\}\{\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 7\.59 0 C 7\.59 4\.19 4\.19 7\.59 0 7\.59 C \-4\.19 7\.59 \-7\.59 4\.19 \-7\.59 0 C \-7\.59 \-4\.19 \-4\.19 \-7\.59 0 \-7\.59 C 4\.19 \-7\.59 7\.59 \-4\.19 7\.59 0 Z M 0 0\}\{fill:none\} \\lx@inpgf@ignorespaces \}\{\{\{\{\\lx@inpgf@ignorespaces\}\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-2\.5pt\}\{\-3\.22221pt\}\\lxSVG@begingroup@\{transform\} \\pgfsys@hbox\{58\}\\lxSVG@closescope \}\}\} \\lxSVG@closescope \}\}\} \} \\lxSVG@closescope \{\{\{\}\}\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\}to𝒱𝗌𝖿\\mathcal\{V\}\_\{\\mathsf\{sf\}\}\(in this study, we use8to replace any consecutive sequence of easily predictable bytes of length 8 or greater\)\. Theimplicit sentineltoken vocabulary adds\{256,…,511\}\\\{256,\\dots,511\\\}to𝒱𝗌𝖿\\mathcal\{V\}\_\{\\mathsf\{sf\}\}\.

For this study, we continue to use subword tokenization for the output tokens\. Since our experiments focus on machine translation, we use the multilingual tokenizer provided with Meta’s NLLB\-200 model\. Extending the autocompleting tokenizer to the Transformer output is left for future work\.

## 4Experiments

### 4\.1Comparing Prediction Thresholds and Compression Schemes

Our first set of experiments used English\-to\-French machine translation as a testbed for comparing the effectiveness of different prediction thresholds \(absolute and relative\) and compression schemes \(sentinel\-free, explicit sentinel, length\-encoded sentinel, and implicit sentinel\)\. We retained 6,994,688 sentence pairs from the WMT14 benchmark\([2](https://arxiv.org/html/2608.15080#bib.bib1)\)whose English source lengths were at most 512 bytes, allowing training without sequence truncation222All experiments were run on a single Nvidia RTX A6000 GPU with 48GB of RAM\.while training with \(uncompressed\) byte tokenization\. We used the unfiltered WMT14 English\-French benchmarks for validation and testing\.

Table 1:Translation quality as a function of compression scheme and threshold\. Compression ratio is defined as compressed length divided by original length\.We trained a standard encoder\-decoder Transformer\([17](https://arxiv.org/html/2608.15080#bib.bib9)\)with 12 encoder layers, 12 decoder layers,d𝗆𝗈𝖽𝖾𝗅=1024d\_\{\\mathsf\{model\}\}=1024,d𝖿𝖿=4096d\_\{\\mathsf\{ff\}\}=4096, and 16 attention heads per layer\. Models were trained using Adafactor\([14](https://arxiv.org/html/2608.15080#bib.bib6)\)with relative\-step learning\-rate scaling, weight decay 0\.01, and a minibatch size of 32\. Training employed automatic mixed\-precision arithmetic and gradient\-norm clipping with threshold 1\.0\. Parameters were updated after every minibatch\. Validation cross\-entropy loss was evaluated every 500 training steps, and the checkpoint with the lowest validation loss was retained\. Training was run for 600,000 training steps\.

During inference, translations were generated using beam search with a beam width of 4\. The maximum output length was set dynamically as32\+3​n32\+3n, wherennis the source sequence length in tokens\. Translation quality was evaluated using BLEU\([9](https://arxiv.org/html/2608.15080#bib.bib17)\)and chrF\([10](https://arxiv.org/html/2608.15080#bib.bib18)\)\. Scores were computed using the Hugging Faceevaluate\(v0\.4\.3\) implementation of SacreBLEU\([11](https://arxiv.org/html/2608.15080#bib.bib19)\)and chrF, respectively\. We report corpus\-level BLEU and chrF scores on the test set\.

We conducted 26 experiments, each time selecting a different prediction threshold and compression scheme\. The results can be found in Table[1](https://arxiv.org/html/2608.15080#S4.T1)\. In that table, thecompression ratiois the compressed length divided by the original length of the test set \(in bytes\)\. The final row of the table shows the performance of uncompressed byte tokenization\.

![Refer to caption](https://arxiv.org/html/2608.15080v1/plot1.png)Figure 3:Translation quality versus compression ratio for the four compression schemes\. Compression ratio is defined as compressed length divided by original length\. This plot also includes three baselines: theBytebaseline is uncompressed byte tokenization\. TheNo Vowelsbaseline removes all vowels from the input sentences\. TheFirst Threebaseline preserves only the first three letters of each word\.![Refer to caption](https://arxiv.org/html/2608.15080v1/plot2.png)Figure 4:Translation quality versus compression ratio for the four compression schemes, with absolute and relative prediction thresholds isolated\. There is no clear advantage to absolute or relative prediction thresholds\.These numbers are visualized in Figure[3](https://arxiv.org/html/2608.15080#S4.F3)and Figure[4](https://arxiv.org/html/2608.15080#S4.F4), which plot each experiment’s chrF score versus its compression ratio\. Here are some takeaways:

The most successful compression scheme is Implicit Sentinel\.Using the Implicit Sentinel compression scheme, we achieve parity with uncompressed byte tokenization at a compression ratio of 0\.679, meaning that we can eliminate nearly a third of the English bytes without impacting the quality of the translation system\.

The autocomplete model is beneficial\.Figure[3](https://arxiv.org/html/2608.15080#S4.F3)includes two additional baselines besides uncompressed byte tokenization\. TheNo Vowelsbaseline simply removes all vowels from the input sentences, whereas theFirst Threebaseline preserves only the first three letters of each word\. WhileNo Vowelsis the more successful of the two, it underperforms theSentinel Freecompression scheme, which removes letters more strategically than these baselines by leveraging the autocomplete model\.

Encoding the number of removed bytes is beneficial\.TheLength\-Encoded Sentinelcompression scheme outperforms theExplicit Sentinelcompression scheme, at the minor cost of adding seven additional sentinels to the token vocabulary\.

There is no clear advantage to either absolute or relative prediction thresholds\.In Figure[4](https://arxiv.org/html/2608.15080#S4.F4), we plot the same data \(minus the baselines\) but we show which datapoints use absolute prediction thresholds, and which use relative prediction thresholds\. They appear to perform similarly\.

### 4\.2Additional Languages

Based on the findings from the previous experiments, we applied Implicit Sentinel compression to three WMT19\([1](https://arxiv.org/html/2608.15080#bib.bib3)\)language pairs: Finnish–English \(agglutinative morphology\), Russian–English \(Cyrillic script\), and Chinese–English \(logographic script\)\. Because these datasets are larger than the filtered WMT14 English–French subset used in prior experiments, we extended training to 1M steps to ensure convergence under the increased data scale\.

In this extended evaluation, some training examples exceed the maximum sequence length supported by the base Transformer architecture, leading to truncation of longer sentences\. While this is a standard constraint in training sequence models under fixed computational and memory budgets, it highlights a fundamental limitation of length\-bounded processing: when inputs exceed capacity, information must be discarded in a hard and unstructured manner\. Our compression\-based approach can be viewed as an alternative to this truncation mechanism, in which sequence length is reduced in a data\-dependent manner using model\-estimated predictability rather than a fixed cutoff\. This allows longer inputs to be mapped into the available computational budget in a more adaptive way, while still preserving a correspondence with the original sequence\. As such, the setting serves as a useful stress test for comparing hard truncation with learned or heuristic compression strategies under identical resource constraints\.

Table 2:Translation quality and compression ratio for three WMT19 language pairs\. Compression ratio is defined as compressed length divided by original length\.Experimental results are shown in Table[2](https://arxiv.org/html/2608.15080#S4.T2)\. We observe that Implicit Sentinel compression improves performance on Russian–English and Chinese–English while maintaining parity on Finnish–English\. One possible contributing factor is the interaction between compression and script or morphological structure: Russian text in UTF\-8 typically requires more bytes per character than Latin\-based languages, which may increase the potential benefit of byte\-level redundancy removal\. Similarly, Chinese text is represented with multi\-byte encodings but carries relatively high semantic density at the character level, which may affect the distribution of predictable versus informative regions\. Finnish, while morphologically rich, shows similar performance to the byte baseline under the tested compression ratio, suggesting that gains may depend on both script\-level encoding properties and language\-specific redundancy patterns\.

To better understand what information is removed by the autocomplete model, we also analyzed omission rates as a function of byte category\. This analysis is deferred to the appendix\.

## 5Conclusion

We introduced a compression\-based approach to byte\-level tokenization that leverages a lightweight autoregressive model to identify and remove highly predictable bytes before Transformer processing\. Inspired by autocomplete\-assisted writing systems, the proposed method shifts some of the burden of sequence modeling from a large translation model to a much smaller byte\-level predictor, allowing portions of the input sequence to be represented implicitly rather than explicitly\.

Across a range of compression schemes and prediction thresholds, we find that substantial reductions in source sequence length are possible without sacrificing translation quality\. In particular, an implicit\-sentinel encoding preserves English–French translation performance while removing nearly one\-third of the source bytes\. Additional experiments on Finnish–English, Russian–English, and Chinese–English demonstrate that the approach generalizes across languages with markedly different writing systems and morphological characteristics, in some cases improving translation quality while simultaneously shortening the input sequence\.

Viewed more broadly, our results suggest that byte\-level representations contain a considerable amount of predictable information that need not be processed directly by large Transformer models\. Rather than treating every byte as equally worthy of computation, future architectures may benefit from allocating computation according to predictability, reserving expensive processing for less predictable portions of the input\. In this sense, our work occupies a middle ground between adaptive\-representation approaches such as BLT and information\-theoretic compression approaches based on arithmetic coding, demonstrating that simple predictability\-based omission can be an effective mechanism for reducing sequence length\.

Several limitations remain\. Our experiments focus exclusively on machine translation and employ a fixed autoregressive predictor architecture\. Future work should explore alternative prediction models, applications beyond translation, and more systematic investigations of the relationship between language structure, writing systems, and compression effectiveness\. We also leave open the possibility of extending predictability\-based compression to both the source and target sides of sequence generation\. We hope this pilot study motivates further research into autocomplete\-inspired approaches to neural model input\.

## References

- L\. Barrault, O\. Bojar, M\. R\. Costa\-jussà, C\. Federmann, M\. Fishel, Y\. Graham, B\. Haddow, M\. Huck, P\. Koehn, S\. Malmasi, C\. Monz, M\. Müller, S\. Pal, M\. Post, and M\. ZampieriFindings of the 2019 conference on machine translation \(wmt19\)\.InProceedings of the Fourth Conference on Machine Translation \(WMT19\),Florence, Italy\.External Links:[Link](http://www.statmt.org/wmt19/)Cited by:[§4\.2](https://arxiv.org/html/2608.15080#S4.SS2.p1.1)\.
- Bojaret al\.\(2014\)O\. Bojar, C\. Buck, C\. Federmann, B\. Haddow, P\. Koehn, J\. Leveling, C\. Monz, P\. Pecina, M\. Post, H\. Saint\-Amand, R\. Soricut, L\. Specia, and A\. TamchynaFindings of the 2014 workshop on statistical machine translation\.InProceedings of the Ninth Workshop on Statistical Machine Translation,Baltimore, Maryland, USA,pp\. 12–58\.Cited by:[§4\.1](https://arxiv.org/html/2608.15080#S4.SS1.p1.1)\.
- Brownet al\.\(2020\)T\. B\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. Kaplan,et al\.Language models are few\-shot learners\.Advances in Neural Information Processing Systems33\.Cited by:[§1](https://arxiv.org/html/2608.15080#S1.p1.1)\.
- Clarket al\.\(2022\)J\. H\. Clark, D\. Garrette, I\. Turc, and J\. WietingCANINE: pre\-training an efficient tokenization\-free encoder for language representation\.Transactions of the Association for Computational Linguistics10,pp\. 73–91\.Cited by:[§1](https://arxiv.org/html/2608.15080#S1.p1.1),[§2](https://arxiv.org/html/2608.15080#S2.p2.1)\.
- Goyalet al\.\(2020\)S\. Goyal, A\. Raghunathan, M\. Jain, H\. Simhadri, and P\. JainPower\-bert: accelerating bert inference via progressive word\-vector elimination\.InInternational Conference on Machine Learning,Cited by:[§2](https://arxiv.org/html/2608.15080#S2.p5.1)\.
- Lesteret al\.\(2024\)B\. Lester, J\. Lee, A\. Alemi, J\. Pennington, A\. Roberts, J\. Sohl\-Dickstein, and N\. ConstantTraining llms over neurally compressed text\.Transactions on Machine Learning Research\.External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=pRvhMSV48t)Cited by:[§2](https://arxiv.org/html/2608.15080#S2.p4.1),[§2](https://arxiv.org/html/2608.15080#S2.p6.1)\.
- Mullaney \(2024\)T\. S\. MullaneyThe chinese computer: a global history of the information age\.The MIT Press,Cambridge, MA\.External Links:ISBN 9780262047517Cited by:[§1](https://arxiv.org/html/2608.15080#S1.p3.1)\.
- Pagnoniet al\.\(2025\)A\. Pagnoni, R\. Fam, D\. Hesslow, A\. Kumar, A\. Lozhkov,et al\.Byte latent transformer: patches scale better than tokens\.arXiv preprint arXiv:2412\.09871\.Cited by:[§1](https://arxiv.org/html/2608.15080#S1.p1.1),[§2](https://arxiv.org/html/2608.15080#S2.p3.1),[§2](https://arxiv.org/html/2608.15080#S2.p6.1)\.
- Papineniet al\.\(2002\)K\. Papineni, S\. Roukos, T\. Ward, and W\. ZhuBLEU: a method for automatic evaluation of machine translation\.InProceedings of the 40th Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 311–318\.External Links:[Document](https://dx.doi.org/10.3115/1073083.1073135)Cited by:[§4\.1](https://arxiv.org/html/2608.15080#S4.SS1.p3.1)\.
- Popović \(2015\)M\. PopovićChrF: character n\-gram f\-score for automatic mt evaluation\.InProceedings of the Tenth Workshop on Statistical Machine Translation \(WMT 2015\),pp\. 392–395\.External Links:[Document](https://dx.doi.org/10.18653/v1/W15-3049)Cited by:[§4\.1](https://arxiv.org/html/2608.15080#S4.SS1.p3.1)\.
- Post \(2018\)M\. PostA call for clarity in reporting bleu scores\.InProceedings of the Third Conference on Machine Translation \(WMT 2018\),pp\. 186–191\.External Links:[Document](https://dx.doi.org/10.18653/v1/W18-6319)Cited by:[§4\.1](https://arxiv.org/html/2608.15080#S4.SS1.p3.1)\.
- Raoet al\.\(2021\)Y\. Rao, W\. Zhao, Z\. Zhu, J\. Zhou, and J\. LuDynamicViT: efficient vision transformers with dynamic token sparsification\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2608.15080#S2.p5.1)\.
- Sennrichet al\.\(2016\)R\. Sennrich, B\. Haddow, and A\. BirchNeural machine translation of rare words with subword units\.InProceedings of ACL,Cited by:[§1](https://arxiv.org/html/2608.15080#S1.p1.1)\.
- Shazeer and Stern \(2018\)N\. Shazeer and M\. SternAdafactor: adaptive learning rates with sublinear memory cost\.InProceedings of the 35th International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.80,pp\. 4603–4611\.External Links:[Link](https://proceedings.mlr.press/v80/shazeer18a.html),[Document](https://dx.doi.org/10.48550/arXiv.1804.04235)Cited by:[§4\.1](https://arxiv.org/html/2608.15080#S4.SS1.p2.1)\.
- Tayet al\.\(2022\)Y\. Tay, M\. Dehghani, D\. Bahri, and D\. MetzlerCharformer: fast character transformers via gradient\-based subword tokenization\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.15080#S1.p1.1),[§2](https://arxiv.org/html/2608.15080#S2.p2.1)\.
- Touvronet al\.\(2023\)H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi,et al\.Llama 2: open foundation and fine\-tuned chat models\.arXiv preprint arXiv:2307\.09288\.Cited by:[§1](https://arxiv.org/html/2608.15080#S1.p1.1)\.
- Vaswaniet al\.\(2017\)A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, L\. Kaiser, and I\. PolosukhinAttention is all you need\.InAdvances in Neural Information Processing Systems,Vol\.30,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf)Cited by:[§1](https://arxiv.org/html/2608.15080#S1.p1.1),[§4\.1](https://arxiv.org/html/2608.15080#S4.SS1.p2.1)\.
- Xueet al\.\(2022\)L\. Xue, A\. Barua, N\. Constant, M\. Kale, R\. Al\-Rfou, A\. Siddhant, N\. Chirkova, and C\. RaffelByT5: towards a token\-free future with pre\-trained byte\-to\-byte models\.Transactions of the Association for Computational Linguistics10,pp\. 291–306\.Cited by:[§1](https://arxiv.org/html/2608.15080#S1.p1.1),[§2](https://arxiv.org/html/2608.15080#S2.p2.1)\.
- Yuet al\.\(2024\)L\. Yu, D\. Simig, C\. Flaherty, A\. Aghajanyan, L\. Zhang, V\. Smith,et al\.MegaByte: predicting million\-byte sequences with multiscale transformers\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§1](https://arxiv.org/html/2608.15080#S1.p1.1),[§2](https://arxiv.org/html/2608.15080#S2.p2.1)\.

## Appendix AByte Stream Analysis

![Refer to caption](https://arxiv.org/html/2608.15080v1/breakdown.png)Figure 5:Composition of source\-language byte streams and omission behavior of the compression step\. Each bar represents all source bytes in the test set for a language\. Colors indicate byte categories, while hatched regions denote bytes omitted by the autocomplete model\. Russian and Chinese contain a substantially larger proportion of UTF\-8 lead and continuation bytes than Finnish, and a considerable fraction of these bytes are removed during compression\.Figure[5](https://arxiv.org/html/2608.15080#A1.F5)reports the percentage of bytes removed for six categories: ASCII letters, ASCII punctuation, digits, spaces, UTF\-8 lead bytes, and UTF\-8 continuation bytes\.

Several patterns emerge\. First, omission rates are consistently highest for structural elements of the byte stream, such as spaces and UTF\-8 encoding bytes, rather than for ASCII letters themselves\. In Finnish, for example, over 90% of UTF\-8 continuation bytes are removed, while ASCII letters are removed only about 25% of the time\. Similarly, Russian exhibits high omission rates for both UTF\-8 lead bytes \(54%\) and continuation bytes \(42%\), compared to only 12% for ASCII letters\. These results suggest that the autocomplete model preferentially removes bytes that are highly constrained by local context\.

Second, the analysis highlights the interaction between our method and UTF\-8 encoding\. Russian and Chinese text contain large numbers of multi\-byte characters, causing UTF\-8 lead and continuation bytes to constitute a substantial fraction of the source sequence\. Many of these bytes appear highly predictable, likely because they encode orthographic information that is partially determined by neighboring bytes\. As a result, compression disproportionately targets encoding\-level redundancy in non\-Latin scripts\.

Third, omission patterns differ substantially across languages\. Russian exhibits high omission rates for both UTF\-8 lead and continuation bytes, whereas Chinese continuation bytes are removed much more frequently than Chinese lead bytes\. Finnish, in contrast, is dominated by ASCII letters and contains relatively few multi\-byte characters, limiting the potential gains obtainable through UTF\-8 redundancy reduction\. These observations are broadly consistent with the experimental results in Table[2](https://arxiv.org/html/2608.15080#S4.T2), where Russian benefits most from compression, Chinese shows moderate gains, and Finnish primarily maintains parity with the byte\-tokenization baseline\.

These findings suggest that the proposed method preferentially removes portions of the byte stream that are highly predictable from local context, including whitespace, punctuation, and components of multi\-byte UTF\-8 encodings\. Understanding the extent to which these effects arise from language structure versus properties of UTF\-8 itself remains an interesting direction for future work\.