Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models

# Distilling Smaller, Stronger Byte Models
Source: [https://arxiv.org/html/2609.12303](https://arxiv.org/html/2609.12303)
## Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models

Kalyani Marathe†\*, Artidoro Pagnoni†\*, Tomasz Limisiewicz, Margaret Li,Mike Lewis\#, Luke Zettlemoyer\#, Srinivasan Iyer†\#\*Meta FAIR†Joint first author\.\#Joint last author\.

###### Abstract

Small models are made more capable through distillation from a larger one that shares their tokenization scheme\. However, do distilled byte and token models behave similarly in terms of scaling trends as compute and data increases? To enable this comparison, we introduce two variants to efficiently convert token logits toByte Logits: 1\) approximate:Marginalize\-It, and 2\) exact:End\-Of\-Token\.We then present the first large scale study of overtraining decoder\-only dense transformer models varying two dimensions simultaneously: thetokenization scheme\(Tokens,Bytes,Bytes w/ <eot\>\) and thetraining objective\(Distillation vs\. Cross\-Entropy\), sweeping layer\-parameter\-matched models with≈\\approx1 billion parameters up to 1 trillion bytes of data\. Across eight benchmarks spanning three categories: Multiple Choice QA, Language Generation, and Machine Translation, we find thatToken\-1Bmodels outperform byte models \(End\-Of\-Token\-1BandBytes\-1B\) in the low\-FLOP regime but eventually plateau; byte models start worse yet surpassToken\-1Bmodels with more compute, reaching a higher downstream task performance ceiling\. Extrapolating the average top\-1 error vs\. validation BPB scaling laws predicts that, asymptotically, distilledEnd\-Of\-Token\-1Boutperforms distilledToken\-1Bby up to4%on averaged downstream task performance\. They are also far more data efficient, matching the performance of distilledToken\-1Busing onlyone\-sixthof the training data\. Moreover, by operating over a small vocabulary of≈256\\approx 256bytes instead of≈100\\approx 100K tokens, they circumvent the need for top\-kktruncation during logit dumping, while also reducing logit storage costs to roughlyone\-fifth\. Finally, our downstream performance scaling laws predict that our distilledEnd\-Of\-Token\-1Bmodels asymptotically surpass theLlama 3\.2\-1Bmodel\([Meta AI, \(2024\)](https://arxiv.org/html/2609.12303#bib.bib39)\),Gemma\-3\-1B\-pt\([Kamath et al\., \(2025\)](https://arxiv.org/html/2609.12303#bib.bib29)\), andGemma 2B\([Team et al\., \(2024\)](https://arxiv.org/html/2609.12303#bib.bib52)\)models on averaged downstream tasks by up to6\.5%, 8\.1%, and2\.1%, respectively\.

††footnotetext:∗Correspondence:kmarathe@uw\.edu,\{artidoro, sviyer\}@meta\.com## 1Introduction

Scaling has become the prevailing approach for improving language model performance\([Kaplan et al\., \(2020\)](https://arxiv.org/html/2609.12303#bib.bib30);[Hoffmann et al\., \(2022\)](https://arxiv.org/html/2609.12303#bib.bib26);[Henighan et al\., \(2020\)](https://arxiv.org/html/2609.12303#bib.bib23)\)\. However, deploying models in real\-world settings demands more than scale alone: the cost of serving can outweigh the benefits of scale\([Pope et al\., \(2023\)](https://arxiv.org/html/2609.12303#bib.bib48);[Dettmers & Zettlemoyer, \(2023\)](https://arxiv.org/html/2609.12303#bib.bib14)\)\. Models are often 1\) distilled into smaller yet capable models that, by training against the teacher’s next\-token distribution rather than ground\-truth tokens, outperform on downstream tasks\([Hinton et al\., \(2015\)](https://arxiv.org/html/2609.12303#bib.bib25);[Beyer et al\., \(2022\)](https://arxiv.org/html/2609.12303#bib.bib6)\), and 2\) overtrained to amortize inference costs over a model’s lifetime\([Sardana et al\., \(2023\)](https://arxiv.org/html/2609.12303#bib.bib49);[Gadre et al\., \(2024\)](https://arxiv.org/html/2609.12303#bib.bib17)\)\. Yet it remains unclear how the choice of student tokenization interacts with these techniques as compute scales\.

In this work we compare how bytes and tokens affect distillation scaling trends as the compute budget grows\. Bytes are a compelling alternative for two reasons: First, by compressing the vocabulary to just≈\\approx256 values, transformers operating on bytes can preserve the distribution of a large token language model exactly\([Hayase et al\., \(2025\)](https://arxiv.org/html/2609.12303#bib.bib21);[Phan et al\., \(2024\)](https://arxiv.org/html/2609.12303#bib.bib47)\)and potentially can bypass the need for top\-k logit truncation for offline distillation\. Second, they also possess interesting properties: they learn relationships between bytes from fewer samples than token models require for tokens\([Hestness et al\., \(2017\)](https://arxiv.org/html/2609.12303#bib.bib24)\), which makes them data efficient learners\([Xue et al\., \(2022\)](https://arxiv.org/html/2609.12303#bib.bib56);[Zheng et al\., \(2025\)](https://arxiv.org/html/2609.12303#bib.bib60)\)\. Distillation into such models, though, hinges on open questions: 1\) how to efficiently convert token logits to byte logits, 2\) how does tokenization and training objectives affect the BPB vs FLOPs power laws 3\) whether distillation outperforms supervised byte \(and token\) level training on downstream tasks, 4\) at what compute budget byte\-level surpasses token\-level distillation, 5\) how the trained models and the asymptotic predictions compare to the open\-weights model of the comparable model sizes and what are the data and storage efficiency implications\. This work addresses all five\. We summarize our key contributions in section[1\.1](https://arxiv.org/html/2609.12303#S1.SS1)below:

Figure 1:Downstream task performance scaling laws for lr=4e\-3 averaged across six benchmarks: Multiple Choice QA\(ARC\-Easy, ARC\-Challenge, HellaSwag, PIQA\) and Language Generation \(MBPP, Natural Questions\) tasks\.End\-Of\-Token Distilledmodel, by preserving the teacher distribution exactly as well as with the highest amount of compute spent on fixed amount of data raises the asymptotic performance ceiling\.End\-Of\-Token Distilledasymptotically outperforms theToken Distilledmodel by4%, theMarginalize\-It Distilledmodel by1\.9%, and the three open weights models: Llama 3\.2\-1B model\([Meta AI, \(2024\)](https://arxiv.org/html/2609.12303#bib.bib39)\), Gemma\-3\-1B\-pt\([Kamath et al\., \(2025\)](https://arxiv.org/html/2609.12303#bib.bib29)\), and Gemma 2B\([Team et al\., \(2024\)](https://arxiv.org/html/2609.12303#bib.bib52)\)models on averaged downstream tasks by upto6\.5%, 8\.1%, 2\.1%respectively\.### 1\.1Summary of Contributions

Token logits to Byte logits conversion in a single forward pass\.Distilling a smaller byte model from a large token model first requires converting the teacher’s token logits onto byte logits\. A token comprises several bytes, so the probability distribution must be split across byte positions\. Prior methods\([Hayase et al\., \(2025\)](https://arxiv.org/html/2609.12303#bib.bib21);[Phan et al\., \(2024\)](https://arxiv.org/html/2609.12303#bib.bib47)\)to convert token distribution to byte distribution require multiple forward passes which is prohibitive when teacher inference is computationally expensive\. We introduce two single\-pass variants for converting token logits toByte Logitsby marginalizing over byte positions: 1\) approximate:Marginalize\-Itre\-normalizes over the prefix\-matching tokens, while 2\) exact:End\-Of\-Tokenadds an<eot\>token to the vocabulary so no probability is lost\.

Bits\-Per\-Byte vs Training FLOPs Power Laws Across Tokenization and Training Objectives\.With a logit conversion in hand, we ask how the resulting byte models scale compared to token models and their supervised counterparts\. We present a large\-scale study of overtraining layer\-parameter matched decoder\-only dense transformers along two axes: the training objective \(Distillation loss vs Supervised/Cross\-Entropy loss\) and the tokenization scheme \(Tokens,Bytes,Bytes w/ <eot\>\): sweeping decoder\-only transformer models\([Vaswani et al\., \(2017\)](https://arxiv.org/html/2609.12303#bib.bib54)\)with fixed 1\.28 billion layer parameters up to 1 trillion bytes data\. We train multipleToken\-1B\(1\.81B\),Bytes\-1B\(1\.28B\), andEnd\-Of\-Token\-1B\(1\.28B\) models to fit Validation BPB vs Training FLOPs power laws to show that distilledEnd\-Of\-Token\-1Bmodels show better asymptotic behavior compared to distilledToken\-1BandBytes\-1B\.

Average Downstream Task Accuracy vs Training FLOPs Scaling Trends\.To understand how the scaling trends in validation loss translate into downstream task performance forMarginalize\-It\(approximate\) andEnd\-Of\-Token\(exact\) we now turn to benchmark evaluations\. Across three categories and eight benchmarks: Multiple Choice QA \(ARC\-Easy, ARC\-Challenge, HellaSwag, PIQA\), Language Generation \(MBPP, Natural Questions\), and Machine Translation \(Flores\) we find thatToken\-1Btransformers \(supervised and distilled\) with show strong performance at low compute regimes but plateau quickly\.Bytes\-1B\(1\.28B\) andEnd\-Of\-Token\-1B\(1\.28B\), lag initially behind theToken\-1B\(1\.81B\) models, continue to improve at a steeper rate with additional compute despite fewer total parameters due to the small vocabulary size of≈\\approx256 bytes\.

Comparing Distillation Scaling Trajectories Across Compute Budgets\.Downstream task performance vs training FLOPs scaling trend raises the practical question of which method to choose at a given compute budget\. We fit Downstream error vs\. Validation BPB scaling laws \(Figure[1](https://arxiv.org/html/2609.12303#S1.F1)\) and introduceFeather Plots\(Figure[11](https://arxiv.org/html/2609.12303#S7.F11)\), which connect IsoFLOP points on the downstream\-error vs\. Validation\-BPB curves\. These plots let us compare downstream performance trajectories across compute budgets for each of the tokenization scheme\. By extrapolating the downstream error vs validation BPB scaling laws, our experiments predict that distilledEnd\-Of\-Token\-1Btransformers raise the asymptotic ceiling of the averaged downstream task performance\. We then predict that by both preserving the teacher distribution exactly as well as by spending≈\\approx30\.94%more compute on a fixed amount of data compared toMarginalize\-It\(approximate\) method, distilledEnd\-Of\-Token\-1B\(exact\) improves over distilledToken\-1Bby up to4%and distilledBytes\-1Bby up to1\.9%asymptotically\.

End\-Of\-Token\-1B: Data and Storage Efficiency, Asymptotic win over Open\-Weight models\.The asymptotic accuracy gains fromEnd\-Of\-Tokenmethod also come with efficiency gains\. DistilledEnd\-Of\-Token\-1Bmodel also surpass distilledToken\-1Bmodels with up toone\-sixthof the training data with sufficient training despite having fewer total parameters\. By operating over a small vocabulary of≈256\\approx 256bytes instead of the≈100\\approx 100K tokens they circumvent the need for top\-k truncation during logit dumping, reducing logit storage costs to roughlyone\-fifth\. Finally, comparisons with the open\-weight models with the asymptotic predictions reveal our distilledEnd\-Of\-Token\-1Bmodel surpass the[Meta AI \(\(2024\)\)](https://arxiv.org/html/2609.12303#bib.bib39),[Kamath et al\. \(\(2025\)\)](https://arxiv.org/html/2609.12303#bib.bib29), and[Team et al\. \(\(2024\)\)](https://arxiv.org/html/2609.12303#bib.bib52)models on averaged downstream tasks by upto6\.5% and 8\.1%, 2\.1%respectively\.

Paper Outline: We introduce two logit conversion methods:Marginalize\-ItandEnd\-Of\-Tokento convert the token logits to byte logits \(Section[2](https://arxiv.org/html/2609.12303#S2)\)\. We then describe the experimental setup, the six scenarios used for our scaling study, and the benchmarks used for evaluations \(Section[3](https://arxiv.org/html/2609.12303#S3)\)\. We then study the validation BPB vs FLOPs scaling laws \(Section[4](https://arxiv.org/html/2609.12303#S4)\), the downstream task performance vs training FLOP scaling trends \(Section[5](https://arxiv.org/html/2609.12303#S5)\) and the downstream performance vs validation BPB scaling laws \(Section[6](https://arxiv.org/html/2609.12303#S6)\)\. We then use these scaling laws to find out when byte distillation will be advantageous over token distillation baseline \(Section[7](https://arxiv.org/html/2609.12303#S7)\)\. Through the lens of downstream error scaling laws, with the help ofFeather Plotswe demonstrate how the relative performance ordering between theToken Distilled,Marginalize\-It Distilled, andEnd\-Of\-Token Distilledmodels evolve with the training compute budget\. We then benchmark our trained checkpoints and our asymptotic predictions against the open weight models \(Section[8](https://arxiv.org/html/2609.12303#S8)\)\. Next, we contextualize our work within the existing literature \(Section[9](https://arxiv.org/html/2609.12303#S9)\)\. Lastly we reflect on the findings, limitations and opportunities for future work \(Section[10](https://arxiv.org/html/2609.12303#S10)\)\.

## 2Logit Conversion Methods

In this section we describe two methods:Marginalize\-ItandEnd\-Of\-Token, we use in our scaling study to efficiently convert token logits toByte Logits\. Figure[2](https://arxiv.org/html/2609.12303#S2)demonstrate the byte logits formation process with the help of an example\. We provide the pseudocode for the two methods in Appendix[F\.1](https://arxiv.org/html/2609.12303#A6.SS1)and discuss special cases in Appendix[F\.2](https://arxiv.org/html/2609.12303#A6.SS2)\.

### 2\.1Marginalize\-It Logits

For each token used in teacher inference, we have a\|V\|\|V\|\-sized logit tensor denoting the next\-token distribution, whereVVis the vocabulary of the BPE tokenizer\. We also have the decoded byte sequence of every BPE token inVV\. To compute the*first*byte distribution of each BPE token, we marginalize over the full vocabularyVVbased on the first byte\. For each*subsequent*byte, we restrict the vocabulary to the tokens whose prefixes match the ground\-truth bytes and compute the byte distribution conditioned on this subset\. We refer to this as theMarginalize\-Itmethod\. While this approach always yields an exact distribution for the first byte, the distributions over subsequent bytes are only*approximate*\. Figure[2](https://arxiv.org/html/2609.12303#S2.F2)\(column 1\) shows theMarginalize\-Itlogit conversion process step by step\. To predict the distributionB3, the continuations of tokenis\(whose probability is 0\.125\) are silently dropped and only the tokensisuandiskare carried forward\. While obtaining the next token distribution of tokeniscan fix this problem, it requires running additional teacher inference over the sequence:\[<bos\>, T, iram, is\]\. The number of additional inferences required to obtain exact conversions can get intractable for long sequences\. To avoid that problem, inMarginalize\-Itmethod remaining probability is redistributed among tokensisuandisk:

P⁡\(isu\)\\displaystyle P\(\\texttt\{isu\}\)=0\.50\.5\+0\.125=0\.8,\\displaystyle=\\frac\{0\.5\}\{0\.5\+0\.125\}=0\.8,P⁡\(isk\)\\displaystyle P\(\\texttt\{isk\}\)=0\.1250\.5\+0\.125=0\.2,\\displaystyle=\\frac\{0\.125\}\{0\.5\+0\.125\}=0\.2,During training, the byte sequence used is:\[<bos\>,T,i,r,a,m,i,s,u,<eos\>\]\. We provide additional details in Appendix[F\.2](https://arxiv.org/html/2609.12303#A6.SS2)\.Marginalize\-Itserves as the main baseline throughout the study\.

### 2\.2End\-Of\-Token Logits

TheEnd\-Of\-Tokenmethod is essentially theMarginalize\-Itmethod with two key changes\. First, while forming the byte logits after teacher inference, we append an<eot\>token to mark the end of each BPE token as shown in Figure[2](https://arxiv.org/html/2609.12303#S2.F2)\(column 2\)\. Second, we expand the byte vocabulary by one to store the<eot\>probabilities and then apply marginalization over byte positions to form the byte logits\. During training, we append<eot\>after every BPE token in the pretraining dataset:\[<bos\>, <eot\>,T,<eot\>,i,r,a,m,<eot\>,i,s,u,<eot\>,<eos\>,<eot\>\]\. This resolves the missing\-probability problem ofMarginalize\-It\. To computeB3distribution, unlikeMarginalize\-It,End\-Of\-Tokenconsiders*all*tokens with prefixis\. Since the total probability of tokens starting with prefixisis 0\.5 \+ 0\.125 \+ 0\.125 = 0\.75:

P⁡\(isu<eot\>\)\\displaystyle P\(\\texttt\{isu<eot\>\}\)=0\.50\.75=0\.667,\\displaystyle=\\frac\{0\.5\}\{0\.75\}=0\.667,P⁡\(isk<eot\>\)\\displaystyle P\(\\texttt\{isk<eot\>\}\)=0\.1250\.75=0\.167,\\displaystyle=\\frac\{0\.125\}\{0\.75\}=0\.167,P⁡\(is<eot\>\)\\displaystyle P\(\\texttt\{is<eot\>\}\)=0\.1250\.75=0\.167\.\\displaystyle=\\frac\{0\.125\}\{0\.75\}=0\.167\.In this way, the probability mass of the possible continuations ofisis absorbed into the<eot\>token\. This approach is exact, does not require additional inferences over teacher\. Inserting<eot\>tokens also has an unintended but remarkable consequence: the≈\\approx30\.94%additional compute it spends per unit of data \(one<eot\>every approx 4\.5 bytes\) turns out to further improve asymptotic performance relative toMarginalize\-Itmethod\.

Figure 2:A step\-by\-step illustration of the two logit conversion methods\. The first column demonstrates theMarginalize\-ItByte Logitsformation process\. Consider the tokenized promptTiramisu = \[<bos\>, T, iram, isu, <eos\>\], and in particular the next\-token distribution following the tokeniram, for which the ground truth isisu\. Each column represents the decoded byte sequence of a BPE token from the full vocabulary \(pruned here for clarity\) of the Llama 3\-8B model, with the token’s probability shown at the top of the column; columns sharing a color share a byte prefix\. To compute the first\-byte distribution, the columns sharing the byte prefixB1are aggregated\. Next, we subset the vocabulary by the ground\-truth byteiand aggregate the probabilities overB2\. Finally, we repeat this process forB3\. A limitation of theMarginalize\-Itmethod is that it does not account for all possible continuations after the BPE tokeniswhile predictingB3: here, for example, only the tokensisuandiskare considered\. TheEnd\-Of\-Tokenmethod \(second column\) addresses this problem by preserving the language modeling distribution exactly, assigning the residual probability mass to the<eot\>token, as illustrated for the prediction of byteB3\.

## 3Scaling Study Design

### 3\.1Experimental Setup

In this section we describe the Model Architecture, Pretraining Dataset and Training Details\. We provide specifics of these settings in Section[A](https://arxiv.org/html/2609.12303#A1)\.

Model Architecture\.We use the transformer architecture\([Vaswani et al\., \(2017\)](https://arxiv.org/html/2609.12303#bib.bib54)\)to train all of the layer\-parameter\-matched models with≈\\approx1 billion parameters\. We use three variants:Token\-1B,Bytes\-1B, andEnd\-Of\-Token\-1Blisted in Appendix \(Table[9](https://arxiv.org/html/2609.12303#A1.T9)\)\. While the three model architectures have equal total layer parameters \(1\.28B\), theToken\-1B\(1\.81B\) is in fact larger due to the larger vocabulary size\. We stick to the Llama\-3\([Grattafiori et al\., \(2024\)](https://arxiv.org/html/2609.12303#bib.bib20)\)implementation for the model design choices\.

Training Dataset\.We use the Llama\-2\([Meta AI, \(2024\)](https://arxiv.org/html/2609.12303#bib.bib39)\)training mixture for our supervised and distillation experiments\. Our study is designed such that both the token logits and byte logits occupy the same amount of memory for the same amount of data\. See section[A\.6](https://arxiv.org/html/2609.12303#A1.SS6)for the full storage cost breakdown\.

Training Details\.We train our models with the AdamW\([Loshchilov & Hutter, \(2017\)](https://arxiv.org/html/2609.12303#bib.bib36)\)optimizer and sweep three learning rates∈\\in\{1e\-3, 4e\-3, 8e\-3\}\. We use 10% warmup steps for each data scale\. We provide additional details on the loss function in Appendix \([A\.3](https://arxiv.org/html/2609.12303#A1.SS3)\)

### 3\.2Scaling Along Two Axes: Tokenization and Training Objectives

Our study seeks to explore scaling trends of layer\-parameter\-matched models with overtraining for six experiments described below\. This enables us to understand the complex relationships between the Supervised \(Cross\-Entropy Loss\) and Distillation pretraining as well as the effects of the tokenization schemes:Tokens,Bytes,Bytes w/ <eot\>on the training dynamics\. Table[1](https://arxiv.org/html/2609.12303#S3.T1)details the six experimental settings in our overtraining study\. Model size and FLOPs/unit values are obtained from Table[9](https://arxiv.org/html/2609.12303#A1.T9)\. ForMarginalize\-ItandEnd\-Of\-Tokendistillation we use the logit conversion process described in Section[2](https://arxiv.org/html/2609.12303#S2)\.

Table 1:Six experiments in our overtraining study across the training\-objective and tokenization axes\.
### 3\.3Benchmark Evaluations

Multiple Choice Question Answering\.We evaluate performance on four benchmarks which evaluate the ability of a model to solve multiple choice questions\. Specifically we use ARC\-Easy\([P\. Clark et al\., \(2018\)](https://arxiv.org/html/2609.12303#bib.bib12)\), ARC\-Challenge\([P\. Clark et al\., \(2018\)](https://arxiv.org/html/2609.12303#bib.bib12)\), HellaSwag\([Zellers et al\., \(2019\)](https://arxiv.org/html/2609.12303#bib.bib58)\), and PIQA\([Bisk et al\., \(2020\)](https://arxiv.org/html/2609.12303#bib.bib8)\)to benchmark the token and byte checkpoints with and without distillation\.

Language Generation Tasks\.We further evaluate the language modeling capabilities of the byte and token models on MBPP\([Austin et al\., \(2021\)](https://arxiv.org/html/2609.12303#bib.bib2)\)and Natural Questions\([Kwiatkowski et al\., \(2019\)](https://arxiv.org/html/2609.12303#bib.bib32)\)\.

Machine Translation\.Lastly we also evaluate our trained checkpoints on English to German and German to English translation using the Flores\([Goyal et al\., \(2022\)](https://arxiv.org/html/2609.12303#bib.bib19)\)benchmark\.

## 4Scaling Trends I: Validation BPB vs Training FLOPS

In this experiment we analyze the overtraining behavior of the six experiments described in[3\.2](https://arxiv.org/html/2609.12303#S3.SS2)\. We plot the BPB of the model on a held out validation dataset\.

### 4\.1Validation BPB vs Training FLOP Plots

We fit the power law for validation BPB with respect to FLOPs using equation \([1](https://arxiv.org/html/2609.12303#S4.E1)\) wherexxis the number of FLOPs and y is the BPB at a given compute budget \(Table[2](https://arxiv.org/html/2609.12303#S4.T2)\)\. Theyy, asymptotically approaches c\.:

y=b⋅xa\+cy=b\\cdot x^\{a\}\+c\(1\)
Figure 3:The figure shows our scaling curves for different data scales, for a learning rate of 4e\-3 and a fixed parameter count of approximately 1 billion parameter transformer models\(see Table[9](https://arxiv.org/html/2609.12303#A1.T9)for details\)\. Note that data scales plot serves as an alternative x axis to training flops\. Since the FLOPs per unit changes for the three models, we specify the number of units used for training\. The distilled models are trained using Llama 3\-8B as a teacher model\. The validation BPB of the teacher model is 0\.85511\.Table 2:Power\-law scaling equationsy=b⋅xa\+cy=b\\cdot x^\{a\}\+cfitted to BPB vs\. training FLOPs for LR=4e\-3\.Table 3:We observe that the amount of FLOPs spent on a fixed amount of data \(one BPE token in this case\) affects the asymptotic BPB\. Both supervised and distilled models asymptotically follow descending BPB trend as the Total FLOPs Per Token increase\. The FLOPs/Unit values are obtained from Table[9](https://arxiv.org/html/2609.12303#A1.T9)\. Depending on the model architecture, a unit can be a token or a byte\. We note thatEnd\-Of\-Token\-1Buses≈\\approx30\.94%extra FLOPs per token compared toBytes\-1B\.We notice several trends from these plots:

1. 1\.First, bothMarginalize\-It Distilled\(approximate\) andEnd\-Of\-Token Distilled\(exact\) methods demonstrate improvements over their supervised counterparts\.
2. 2\.Marginalize\-It Distilledtraining, despite being an approximation works very well in practice\.End\-Of\-Token Distilledmethod is exact and yields consistent improvements over the supervised training\.
3. 3\.Second, we also observe thatToken\-1Bmodels have the lowest BPB at lower compute budgets, and appear to saturate faster\. TheEnd\-Of\-Token\-1Bhave the worst BPB to begin with yet yield the best asymptote in the infinite compute regime\.Bytes\-1Btraining runs, appear to be sandwiched between the other two scenarios at the beginning and the end of the training runs\.
4. 4\.From Table[3](https://arxiv.org/html/2609.12303#S4.T3), we observe that asymptotically for both supervised and distilled models: 1\)End\-Of\-Token\-1BBPB<<Bytes\-1BBPB<<Token\-1BBPB\. We also observe that the supervised and distilled models with the same tokenization scheme exhibits small differences in the asymptotic value\.
5. 5\.We also observe interesting trends across all training runs\. At around1\.8×10211\.8\\times 10^\{21\}FLOPs, theToken Supervisedtraining run appears to have the highest BPB of all, with all other five curves demonstrating lower BPB values\. Does this mean that all other five models at that compute budget are better thanToken Supervised? To find out, we turn to benchmark evaluations in Section[5](https://arxiv.org/html/2609.12303#S5)\.

## 5Scaling Trends II: Task Performance vs Training FLOPs

In this experiment we seek to understand if the observations from the validation BPB curves from section[4](https://arxiv.org/html/2609.12303#S4)are consistent on a variety of benchmarks\. We evaluate the capabilities of our trained models on a total of eight benchmarks spanning three categories: Multiple choice question answering, Generative, and Machine Translation tasks \(Refer to section[3\.3](https://arxiv.org/html/2609.12303#S3.SS3)for more details\)\.

### 5\.1Results on Multiple Choice Question Answering Benchmarks

To evaluate the capabilities of trained models on four multiple choice question answering tasks we plot the % Accuracy vs Training FLOP trends for our six experiments in Figure[4](https://arxiv.org/html/2609.12303#S5.F4)\. In addition we also report the averaged accuracy across four benchmarks\. We observe that:

1. 1\.The averaged choice accuracy improves for all of the 6 experiments, and the distilled models consistently show better performance compared to the supervised models\. This trend also holds for the benchmarks individually\.
2. 2\.Token\-1Bmodels start best, outperforming all of the byte curves at lower compute budgets\. All of the byte curves start worse but appear to be improving faster than the token models, eventually catching up with them\.
3. 3\.If we compare within the four byte experiments,both supervised and distilledEnd\-Of\-Token\-1Bstart worse, and eventually surpass theBytes\-1Bmodels methods respectively\.
4. 4\.Token\-1Bmodels, both supervised and distilled have the best performance at the beginning and appear to be eventually saturating, a pattern also observed in Validation BPB vs Training FLOPs curves\.
5. 5\.Lastly we observe that at 220B token supervised budget only the distilledToken\-1Boutperforms its supervised counterpart curve but the byte curves, both supervised and distilledBytes\-1BandEnd\-Of\-Token\-1Bmodels are much worse, which is counterintuitive to the key observation in[5](https://arxiv.org/html/2609.12303#S4.I1.i5)\.

Figure 4:We plot Accuracy vs training FLOP curves for all the six scenarios described in section[3\.2](https://arxiv.org/html/2609.12303#S3.SS2)\. For all of the Multiple Choice Question Answering Benchmarks, we observe that the token models show better performance at low compute budgets and eventually plateau\.Bytes\-1BandEnd\-Of\-Token\-1Bmodels: both supervised and distilled, start worse yet improve at a much faster rate\.
### 5\.2Results on Language Generation Benchmarks

In this experiment we study the language generation capabilities of the trained models on two benchmarks \( Figure[5](https://arxiv.org/html/2609.12303#S5.F5)\) \(more details in section[3\.3](https://arxiv.org/html/2609.12303#S3.SS3)\)\. We observe that:

1. 1\.The trends are similar to Multiple Choice benchmarks\. However the saturation effect ofToken\-1Bmodels seems to be less pronounced, potentially because the tasks are much harder\.
2. 2\.In all cases, distilled models appear to be better for the same compute budget and the result about observation[5](https://arxiv.org/html/2609.12303#S4.I1.i5)at1\.8×10211\.8\\times 10^\{21\}FLOPs budget is observed in this case as well\. Only theToken Distilledmodel has better performance while all other four byte curves are much worse in most cases\.

Figure 5:We visualize%\\%Accuracy vs training FLOPs curves for language generation tasks\. On generative tasks, token models achieve better performance at lower compute budgets and appear to plateau with the training FLOPs\.Bytemodels andBytes w/ <eot\>models show a different scaling behavior, with much worse performance at a lower compute budget which continues to improve with FLOPs with no sign of saturation\.
### 5\.3Results on Machine Translation Benchmarks

We also evaluate models on machine translation benchmarks and plot BLEU score\([Papineni et al\., \(2002\)](https://arxiv.org/html/2609.12303#bib.bib45)\)vs Training FLOP curves\. The plots \(Figure[6](https://arxiv.org/html/2609.12303#S5.F6)\) show that:

1. 1\.Byte model BLEU scores appear to be following an S\-shaped trend with training FLOPs, improving slowly at low compute regimes, followed by a steeper improvement regions and then finally saturating at high compute budgets\.
2. 2\.Next, the results on these benchmarks contradict the observation[5](https://arxiv.org/html/2609.12303#S4.I1.i5)here as well\.
3. 3\.End\-Of\-Token Distilledmodels by preserving the teacher distribution exactly show significant improvements overBytes w/ <eot\> Supervisedmodels\.

Figure 6:We plot BLEU score for the translation tasks for different training budgets\. We observe clearly different scaling behaviors for token and byte models\. Token models can achieve significantly better performance at smaller compute budgets\. The BLEU score appears to saturate with overtraining\. Byte models both with and without the<eot\>demonstrate extremely low BLEU scores at lower compute budgets yet the rate of improvement with overtraining is better compared to token models\.
### 5\.4Concluding Remarks

In section[5\.1](https://arxiv.org/html/2609.12303#S5.SS1),[5\.2](https://arxiv.org/html/2609.12303#S5.SS2), and[5\.3](https://arxiv.org/html/2609.12303#S5.SS3), we consistently saw that despite achieving better validation BPB at≈1\.8×1021\\approx 1\.8\\times 10^\{21\}FLOPs, all four byte models, both distilled and supervised continue to be much worse compared toToken Supervisedmodel\. What could explain this behavior? In section[6](https://arxiv.org/html/2609.12303#S6)we set out to explain this phenomenon\.

For the data points we observe, the distilled models achieve lower BPB and better overall downstream task performance than their supervised counterparts\. However, this does not always hold when comparing theToken\-1Bagainst theBytes\-1BandEnd\-Of\-Token\-1Bmodels: aToken\-1Bcan have worse BPB yet better downstream performance, as observed at the 220B token data scale\.

## 6Scaling Trends III: Task Performance vs Validation BPB

We now turn to explaining how the observations in sections[4](https://arxiv.org/html/2609.12303#S4)and[5](https://arxiv.org/html/2609.12303#S5)fit together\. For all of the three downstream task categories we study the relationship between the downstream task performance and the validation BPB\. To do so, we use the formulation of scaling laws for downstream tasks in\([Gadre et al\., \(2024\)](https://arxiv.org/html/2609.12303#bib.bib17)\)\. We plot the Average top\-1 error, essentially,\(1−%Accuracy\)\(1\-\\%\\text\{ Accuracy\}\)against the validation BPB, for Multiple Choice Question Answering \(section[6\.1](https://arxiv.org/html/2609.12303#S6.SS1)\) and Language Generation \(section[6\.2](https://arxiv.org/html/2609.12303#S6.SS2)\)\. Lastly, since Machine Translation \(section[6\.3](https://arxiv.org/html/2609.12303#S6.SS3)\) uses BLEU as an evaluation metric, we plot the BLEU score vs the validation BPB\.

### 6\.1Results on Multiple Choice Question Answering Benchmarks

1. 1\.Across all benchmarks as well the averaged performance \(Figure[7](https://arxiv.org/html/2609.12303#S6.F7)\), all of the six scaling trends demonstrate significantly different behavior\. First, if we compare the performances for constant BPB, they imply wildly different error rates\. Second, the iso\-error lines intersect with theEnd\-Of\-TokenandTokencurves at very different compute budgets\.
2. 2\.Second: Changes in the tokenization scheme i\.e\. the amount of FLOPs spent to model a fixed amount of data can dramatically change the scaling behavior\. For instance,Bytes\-1BandEnd\-Of\-Token\-1Bmodels do not show significantly different scaling behaviors compared toToken\-1Bmodels\.
3. 3\.Lastly, the training objective \(supervised as well as distillation\) also appear to alter the scaling behavior\.

Figure 7:We visualize Average\-top\-1 error vs the validation BPB curves for the six scenarios for LR=4e\-3\. All of these curves including the averaged metric appear to follow an exponential decay\.Token\-1Bshows significantly different trend compared toBytes\-1BandEnd\-Of\-Token\-1B\. Both the training objective \(Cross\-Entropy vs Distillation\) as well as the tokenization scheme:Tokens,Bytes,Bytes w/ <eot\>appear to influence the Average Error vs Validation BPB trends\.
### 6\.2Results on Language Generation Benchmarks

We now visualize the Average top\-1 error vs Validation BPB curves for the language generation tasks described in section[3\.3](https://arxiv.org/html/2609.12303#S3.SS3)\. In addition, we also plot the averaged generation performance \(in row 1\)\.

1. 1\.We observe that for language generation as well, the same BPB can mean different averaged downstream performance for the three experiment categories \- \(Token,Bytes,Bytes w/ <eot\>\) as well as training objectives \- supervised and distillation\.

Figure 8:We plot the Average\-top\-1 error vs BPB curves for the MBPP and the NQ benchmarks\. We notice an exponential decay for all three scenarios, with significantly different trends for theToken\-1BvsBytes\-1BandEnd\-Of\-Token\-1Bmodels\. Distilled models as well demonstrate slightly different scaling behavior compared to the supervised counterparts\.
### 6\.3Results on Machine Translation Benchmarks

Lastly we plot BLEU score vs Validation BPB curves for Machine Translation tasks \(Figure[9](https://arxiv.org/html/2609.12303#S6.F9)\)\. We observe that:

1. 1\.TheToken\-1Bmodels,Bytes\-1Bmodels andEnd\-Of\-Token\-1Bmodels show different scaling behaviors\.
2. 2\.On average,End\-Of\-Tokendistillation appears to exhibit monotonic trend compared to the noisyBytes w/ <eot\> Supervisedcurves\.

Figure 9:We plot averaged BLEU score vs Validation BPB performance, and on the averaged BLEU performance for the six scenarios for LR=4e\-3\. We notice the differences in the scaling behavior across tokenization schemes as well as the training objectives\.End\-Of\-Tokendistillation appears to exhibit monotonic trend compared to the noisy supervised curves\.
### 6\.4Fitting the Downstream Task Error vs Validation BPB Scaling Law

We fit the equation[2](https://arxiv.org/html/2609.12303#S6.E2)for our six experiments on averaged top\-1 error calculated over total 4 Multiple Choice and 2 language generation benchmarks \(from[3\.3](https://arxiv.org/html/2609.12303#S3.SS3)\) whereyydenotes the BPB value,z⁡\(y\)z\(y\)is the Averaged Top\-1 Downstream Task Error, andϵ\\epsilon, k andγ\\gammaare fit using data\. The scaling law equations are detailed in Table[4](https://arxiv.org/html/2609.12303#S6.T4)\. The interpolations and extrapolations are visualized in Figure[10](https://arxiv.org/html/2609.12303#S6.F10)\. The curves are bounded by the asymptotic BPB c, obtained from the power law equations in Table[2](https://arxiv.org/html/2609.12303#S4.T2)\.

z⁡\(y\)=ϵ−k⋅e−γ​yz\(y\)=\\epsilon\-k\\cdot e^\{\-\\gamma y\}\(2\)
Table 4:Fitted downstream error scaling lawsz⁡\(y\)=ϵ−k⋅e−γ​yz\(y\)=\\epsilon\-k\\cdot e^\{\-\\gamma y\}\(equation[2](https://arxiv.org/html/2609.12303#S6.E2)\) for each model variant and learning rate of 4e\-3, whereyyis the validation BPB\.Figure 10:We fit the downstream error scaling laws for six benchmarks with respect to Validation BPB\. All six experiments follow a smooth trend\.Token\-1Bcurves show significantly different behavior compared to theBytes\-1BandEnd\-Of\-Token\-1Bcurves\. The training objective \(distillation vs supervised \(Cross Entropy\) also affects the scaling laws\.End\-Of\-Token Distilled, by preserving the teacher distribution exactly as well as with the highest amount of compute spent on fixed amount of data outperforms all of the other models asymptotically\.ByteandBytes w/ <eot\>models, despite having fewer total parameters asymptotically surpass theTokenmodels\.
### 6\.5Downstream Task Performance Upper Bound

To calculate the asymptotic downstream task performance upper bound, we use the Average top\-1 error at asymptotic validation BPB c, for each curve obtained from equation[2](https://arxiv.org/html/2609.12303#S4.T2)\. We then compute the asymptotic performance at that BPB as shown in Figure[10](https://arxiv.org/html/2609.12303#S6.F10)\. Table[4](https://arxiv.org/html/2609.12303#S6.T4)shows the asymptotic BPB as well as the task performance of the six experiments\.

We observe that:

1. 1\.We notice that distilledToken\-1Basymptotically outperforms over its supervised counterpart\.
2. 2\.End\-Of\-Token Distilledmethod outperforms bothToken DistilledandBytes w/ <eot\> Supervisedasymptotically\.
3. 3\.DistilledBytes\-1Bperforms on par or suboptimally compared to its supervised model potentially because it uses an approximation of the teacher distribution\. It however still outperforms asymptotic distilledToken\-1Baccuracy\.
4. 4\.Lastly, we note that by spending around≈\\approx30\.94%additional compute on a fixed amount of data,Bytes w/ <eot\> Supervisedshows better asymptotic behavior compared to theBytes Supervisedmodel\.
5. 5\.End\-Of\-Token Distilledshows better asymptotic downstream performance \(lowest Average top\-1 error\) compared to theBytes w/ <eot\> Supervisedmodel\.

Table 5:Asymptotic validation BPB \(c\\mathrm\{c\}\) and the corresponding downstream average accuracya⋆=1−%z⋆a^\{\\star\}=1\-\\%z^\{\\star\}wherez⋆z^\{\\star\}is the Asymptotic Average top\-1 error predicted by the scaling law equation:[2](https://arxiv.org/html/2609.12303#S6.E2),z⋆=ϵ−k⋅e−γ​cz^\{\\star\}=\\epsilon\-k\\cdot e^\{\-\\gamma c\}, whereccis obtained from the power law equations in Table[4](https://arxiv.org/html/2609.12303#S6.T4)for each model variant and learning rate\.
### 6\.6When Lower BPB Does Not \(Always\) Imply Better Models

Our observations show that BPB \(Bits\-Per\-Byte\) does not allow fair comparison across models with different training objectives \(Distillation and Cross\-Entropy\) and tokenization schemes:Tokens,Bytes,Bytes w/ <eot\>\. We show that the relationship between the BPB acrossTokens,Bytes,Bytes w/ <eot\>is nuanced in the context of benchmark evaluations\. These disparities potentially arise from the differences in probability distributions learned by spending more compute per unit data on a small amount of data vs small amount of compute spent per unit data on larger amount of data under a fixed compute budget\. Although all of theBytes\-1BandEnd\-Of\-Token\-1Bmodels surpass theToken SupervisedBPB training curve, they exhibit worse downstream task performance across benchmarks, falsifying observation[5](https://arxiv.org/html/2609.12303#S4.I1.i5)when the models being compared have different training objectives and tokenization schemes, emphasizing the importance of this final calibration step while determining the quality of a language model before real\-world deployment\. See appendix[G\.1](https://arxiv.org/html/2609.12303#A7.SS1)and section[10](https://arxiv.org/html/2609.12303#S10)for additional discussion\.

## 7The Case forEnd\-Of\-TokenLogits

To understand ifEnd\-Of\-Tokenlogits are advantageous, it is essential to find out when they will if at all surpass the distilled token models\. To achieve that we use the downstream error prediction scaling laws from[6\.4](https://arxiv.org/html/2609.12303#S6.SS4)as shown below:

### 7\.1Feather Plot: Visualizing Iso\-FLOP Point Connections

![Refer to caption](https://arxiv.org/html/2609.12303v1/feather_lr_4e3_zoomed_in.png)Figure 11:The figure shows aFeather Plotfor a learning rate of 4e\-3\. We show thatEnd\-Of\-Token Distilledtransformer model has the lowest Average top\-1 downstream task performance error asymptotically compared to theMarginalize\-ItandTokendistillation\. We also visualize the Validation BPB values and the Average top\-1 downstream task error at the compute budget of1020\.6310^\{20\.63\}FLOPs \(a randomly sampled value\)\. We note the differences in observed BPB value and the task performance by projecting iso\-FLOP points on the X and Y axes for the three curves\. At that compute budget,Tokendistillation leads significantly, while bothEnd\-Of\-Tokendistillation andMarginalize\-Itdistillation lag behind\.We visualize lines connecting Iso\-FLOP points which we callFeather Plotsto understand how the BPB\(Bits\-Per\-Byte\) value as well as the downstream task performance evolves with compute budget\. To plot them we sample points from FLOP rangef​1f1tof​2f2\. For each sampled pointxxwe computeyyusing equation[1](https://arxiv.org/html/2609.12303#S4.E1)\. Next we use the obtained validation loss in terms of BPByyto compute the downstream errorz⁡\(y\)z\(y\)using equation[2](https://arxiv.org/html/2609.12303#S6.E2)\. We connect the points obtained for three methods:Token,Marginalize\-It,End\-Of\-Tokendistillation to analyze the trend\. We observe for example the learnig rate of 4e\-3:

1. 1\.At low compute budgets, distilledToken\-1Bperformance leads significantly compared toBytes\-1BandEnd\-Of\-Token\-1Bmodels\.
2. 2\.As the compute budget progresses, distilledEnd\-Of\-Token\-1Bovertakes theBytes\-1Bmodels\.
3. 3\.Eventually, distilledEnd\-Of\-Token\-1Bcatches up with theBytes\-1Btoo and continues to improve until the asymptotic value of the performance\.

### 7\.2Calculating Downstream Error Cross\-Over Point with Token Distillation

Given the validation BPB prediction and downstream error prediction scaling laws for distilledToken\-1B,Bytes\-1B,End\-Of\-Token\-1Bmodels we want to find out at what compute budgetMarginalize\-It DistilledandEnd\-Of\-Token Distilledmethods will give the same asymptotic downstream error prediction as that of aToken Distilledmodel\.

Table 6:Training compute at which distilledBytes\-1BandEnd\-Of\-Token\-1Bmatch the asymptotic downstream performance ofToken\-1B\. We takeToken Distilledasymptotic BPB \(theccterm of its fitted BPB–FLOPs power law; column Token Dist\. BPB\) and map it throughToken Distilledbenchmark scaling law to obtain its asymptotic downstream accuracy \(column Accuracy\)\. For each byte method we then invert its own benchmark scaling law at that accuracy to get the BPB it must attain \(column Method BPB\), and invert its BPB–FLOPs power law at that BPB to get the required training compute \(column Training FLOPs\)\.Table 7:Effective training\-data volume at the compute budget required to achieve theTokendistillation asymptotic performance from Table[5](https://arxiv.org/html/2609.12303#S6.T5)forMarginalize\-ItandEnd\-Of\-Tokenmethods\.Bytes=Byte Units×Multiplier=\\text\{Byte Units\}\\times\\text\{Multiplier\}converts byte units into actual UTF\-8 bytes of unique text, accounting for the fraction of each unit that is genuine content \(Multiplier=1=1forMarginalize\-It Distilled;=4\.5/5\.5=4\.5/5\.5forEnd\-Of\-Token Distilled, whose sequences carry an extra<eot\>per4\.54\.5content bytes\)\.Data Savings=Token to Byte/Bytes=\\text\{Token to Byte\}/\\text\{Bytes\}is how much less unique text the byte method needs to reach the same performance\. To calculate storage savings we use our storage cost calculations from Table[10](https://arxiv.org/html/2609.12303#A1.T10)where we equate the storage cost of oneTokenwith 4\.5Bytesto obtain k \(for top\-k\)\.Storage Savings=\(Tokens×4\.5\)/Byte Units=\(\\text\{Tokens\}\\times 4\.5\)/\\text\{Byte Units\}compares the byte footprint ofToken Distilledapproach against the raw byte\-unit count\.
### 7\.3Results

1. 1\.We summarize our results in Table[7](https://arxiv.org/html/2609.12303#S7.T7)\. DistilledEnd\-Of\-Token\-1Btraining runs asymptotically outperformToken\-1Bmodel, hence we can compute the crossover points ofBytes\-1BandEnd\-Of\-Token\-1Bmodels withToken\-1Bmodel\.
2. 2\.SinceEnd\-Of\-Tokenuses the<eot\>tokens every 4\.5 bytes, the actual text data is less than the byte units used for training\. Taking this into account,End\-Of\-TokenByte Logitscan reduce the storage cost to approximatelyone\-fifthcompared to token distillation when trained for upto 6\.7 Trillion byte units or data equivalent to 1\.2 Trillion Llama 3\-8B tokens\.

End\-Of\-Token\-1Bdemonstrates better asymptotic downstream performance compared to distilledToken\-1B\. It requiresone\-sixthtext data and it can save storage costs uptoone\-fifth\.

## 8Comparisons with Open\-Weight models

We now compare our asymptotic predictions for each of the six experiments described in section[3\.2](https://arxiv.org/html/2609.12303#S3.SS2)with the open weight models:Llama 3\.2\-1Bmodel\([Meta AI, \(2024\)](https://arxiv.org/html/2609.12303#bib.bib39)\),Gemma\-3\-1B\-pt\([Kamath et al\., \(2025\)](https://arxiv.org/html/2609.12303#bib.bib29)\), andGemma 2B\([Team et al\., \(2024\)](https://arxiv.org/html/2609.12303#bib.bib52)\)in Table[8](https://arxiv.org/html/2609.12303#S8.T8)\. We evaluate these models using\([Gao et al\., \(2024\)](https://arxiv.org/html/2609.12303#bib.bib18)\)\.

Table 8:Predicted asymptotic downstream accuracy \(%\) in the for our LR=4e\-3 obtained from table[15](https://arxiv.org/html/2609.12303#A2.T15), obtained by extrapolating the scaling trend\. Comparisons with the open\-weights model with our asymptotic predictions reveal our distilledEnd\-Of\-Token\-1Bmodel surpass the[Meta AI \(\(2024\)\)](https://arxiv.org/html/2609.12303#bib.bib39),[Kamath et al\. \(\(2025\)\)](https://arxiv.org/html/2609.12303#bib.bib29), and[Team et al\. \(\(2024\)\)](https://arxiv.org/html/2609.12303#bib.bib52)models on averaged downstream tasks by upto6\.5%6\.5\\%and8\.1%8\.1\\%, 2\.1% respectively\. Additional details for llama and gemma models are provided in table[18](https://arxiv.org/html/2609.12303#A3.T18)ModelMethodAsymptotic Avg\. Accuracy\(%\)Token\-1BToken Supervised46\.0Token\-1BToken Distilled48\.4Bytes\-1BBytes Supervised51\.2Bytes\-1BMarginalize\-It Distilled50\.5End\-Of\-Token\-1BBytes w/ <eot\> Supervised51\.6End\-Of\-Token\-1BEnd\-Of\-Token Distilled52\.4Llama\-3\.2\-1B[Meta AI \(\(2024\)\)](https://arxiv.org/html/2609.12303#bib.bib39)45\.9 \(≈\\approx9T BPE tokens\)Gemma\-3\-1B\-pt[Kamath et al\. \(\(2025\)\)](https://arxiv.org/html/2609.12303#bib.bib29)44\.3 \(≈\\approx2T BPE tokens\)Gemma 2b[Team et al\. \(\(2024\)\)](https://arxiv.org/html/2609.12303#bib.bib52)50\.3 \(≈\\approx3T BPE tokens\)
## 9Related Work

Scaling Laws with Overtraining\.Which factors affect the power law exponent with overtraining has been a topic of great interest in deep learning\.[Kaplan et al\. \(\(2020\)\)](https://arxiv.org/html/2609.12303#bib.bib30)show that changing the architecture from transformers to LSTMs does not change the power law exponent for shorter contexts\.\([Bahri et al\., \(2024\)](https://arxiv.org/html/2609.12303#bib.bib3)\)show that the power law exponent changes with data distribution\.\([Henighan et al\., \(2020\)](https://arxiv.org/html/2609.12303#bib.bib23)\)show that the power law exponents change across modalities and also across image resolutions\.\([Bansal et al\., \(2022\)](https://arxiv.org/html/2609.12303#bib.bib4);[Bahri et al\., \(2024\)](https://arxiv.org/html/2609.12303#bib.bib3)\)show that the noise level in data can also significantly affect the exponents\. In the context of language modeling\([Hestness et al\., \(2017\)](https://arxiv.org/html/2609.12303#bib.bib24)\)show that character models learn relationships between characters with fewer samples compared to word level language models learn from words\. Several recent works focus on the overtraining of language models\.\([Sardana et al\., \(2023\)](https://arxiv.org/html/2609.12303#bib.bib49)\)demonstrate that when the inference budget is large enough, it is better to train a small model for longer\.[Gadre et al\. \(\(2024\)\)](https://arxiv.org/html/2609.12303#bib.bib17)derive scaling laws with overtraining for different model sizes and training datasets\.[Limisiewicz et al\. \(\(2026\)\)](https://arxiv.org/html/2609.12303#bib.bib34)analyze the impact of data compression on scaling laws\.[Lee et al\. \(\(2026\)\)](https://arxiv.org/html/2609.12303#bib.bib33)show that the autoregressive byte models approach performance parity with their BPE counterparts as compute scales\. Distinct from prior work, we study power laws for≈\\approxfixed model size with overtraining along two axes: training objective and the tokenization scheme\. We observe that both axes affect the exponent as seen from table \([11](https://arxiv.org/html/2609.12303#A2.T11)\) and table \([15](https://arxiv.org/html/2609.12303#A2.T15)\)\. We show that a simple addition of an<eot\>token every≈\\approx4\.5 bytes can improve a model’s asymptotic downstream performance: an effect explained by the additional FLOPs spent for a fixed amount of data from the increased sequence length at fixed context\.

Scaling Laws for Downstream Tasks\.[Schaeffer et al\. \(\(2023\)\)](https://arxiv.org/html/2609.12303#bib.bib50)argue that the capabilities that appear to be emergent in language models are due to the researcher’s choice of metric\. In the context of transfer learning[Isik et al\. \(\(2025\)\)](https://arxiv.org/html/2609.12303#bib.bib28)study how the choice of the pretraining data affects downstream cross entropy and Machine Translation performance \(BLEU\) score\.[Gadre et al\. \(\(2024\)\)](https://arxiv.org/html/2609.12303#bib.bib17)establish the relationship between the Average top\-1 downstream error across a number of benchmarks and the validation Cross\-Entropy Loss\. They demonstrate that using validation loss alone while comparing models trained on different data distributions can be misleading\.[Owen \(\(2024\)\)](https://arxiv.org/html/2609.12303#bib.bib43)show that the aggregate benchmark performance is decently predictable, though predicting individual performance is harder\.[Bhagia et al\. \(\(2024\)\)](https://arxiv.org/html/2609.12303#bib.bib7)establish a two stage approach where they use N \(\# of parameters\) and D \(\# of tokens\) to predict the intermediate task loss which is then mapped to the task performance\.[Lourie et al\. \(\(2025\)\)](https://arxiv.org/html/2609.12303#bib.bib37)conduct a meta\-analysis of the downstream tasks considered in[Gadre et al\. \(\(2024\)\)](https://arxiv.org/html/2609.12303#bib.bib17)to demonstrate that the task\-specific scaling trends may vary\. They further compare them with the findings from[Magnusson et al\. \(\(2025\)\)](https://arxiv.org/html/2609.12303#bib.bib38)to show that the scaling behavior depends on the experimental setup and that the choice pretraining corpus, validation corpus, or downstream task can affect the scaling laws\. Our work studies how the training objective and the tokenization scheme can affect the downstream performance with Training FLOPs\.

Pretraining Distillation\.[Hinton et al\. \(\(2015\)\)](https://arxiv.org/html/2609.12303#bib.bib25)introduced distillation and showed that a small model can be improved by using the output probabilities of a large model\.[Busbridge et al\. \(\(2025\)\)](https://arxiv.org/html/2609.12303#bib.bib9)build scaling laws to determine the optimal compute allocation between student and the teacher\.[Peng et al\. \(\(2025\)\)](https://arxiv.org/html/2609.12303#bib.bib46)investigate the effect of different design decisions in token to token distillation\. Beyond language modeling[Beyer et al\. \(\(2022\)\)](https://arxiv.org/html/2609.12303#bib.bib6)proposed the “patient and consistent teacher” hypothesis for ResNet\-50 models\([He et al\., \(2016\)](https://arxiv.org/html/2609.12303#bib.bib22)\)trained on Imagenet\([Deng et al\., \(2009\)](https://arxiv.org/html/2609.12303#bib.bib13)\)\. In this work we study pretraining distillation for both token and byte models\.

Token to Bytes transfer\.[Minixhofer et al\. \(\(2025\)\)](https://arxiv.org/html/2609.12303#bib.bib40)present an approach for adapting pretrained token\-level models to operate on raw bytes\.[Minixhofer et al\. \(\(2026\)\)](https://arxiv.org/html/2609.12303#bib.bib41)and[Bao et al\. \(\(2026\)\)](https://arxiv.org/html/2609.12303#bib.bib5)propose methods for distilling token models into byte models in the fine\-tuning stage\.[Hayase et al\. \(\(2025\)\)](https://arxiv.org/html/2609.12303#bib.bib21)and[Phan et al\. \(\(2024\)\)](https://arxiv.org/html/2609.12303#bib.bib47)study the exact BPE to bytes transfer to primarily tackle theprompt boundary problemwhere the prompt, when ends in the middle of a BPE token, results into distorted next token distributions at inference time\. These methods can be used to convert token logits to byte logits\. However they remain computationally expensive and require multiple inference passes over the teacher to obtain exact conversions\. We circumvent this problem by simply marginalizing the prefix\-matched token distributions inMarginalize\-Itmethod and introducing an additional<eot\>token in the vocabulary and the sequences during training to preserve the distribution in theEnd\-Of\-Tokenmethod\.

Byte Level Models\.[Al\-Rfou et al\. \(\(2018\)\)](https://arxiv.org/html/2609.12303#bib.bib1),[Choe et al\. \(\(2019\)\)](https://arxiv.org/html/2609.12303#bib.bib10),[El Boukkouri et al\. \(\(2020\)\)](https://arxiv.org/html/2609.12303#bib.bib15)laid the groundwork for byte level language modeling\.[J\.H\. Clark et al\. \(\(2022\)\)](https://arxiv.org/html/2609.12303#bib.bib11)introduce CANINE, an architecture with downsampling, followed by a deep transformer stack and upsampling layers for efficient character level modeling\.[Xue et al\. \(\(2022\)\)](https://arxiv.org/html/2609.12303#bib.bib56)train an encoder\-decoder model operating on UTF\-8 bytes and show that byte level transformers are data efficient learners, though take longer to train and are much expensive during inference time\.[Yu et al\. \(\(2023\)\)](https://arxiv.org/html/2609.12303#bib.bib57)introduce the concept of patching to enable sub\-quadratic self attention\.[Nawrot et al\. \(\(2023\)\)](https://arxiv.org/html/2609.12303#bib.bib42)propose a hierarchical architecture for byte level modeling jointly performs language modeling and token segmentation\.[Pagnoni et al\. \(\(2025\)\)](https://arxiv.org/html/2609.12303#bib.bib44)propose a new architecture with a local encoder and decoder that operates on patches of raw bytes\.[Hwang et al\. \(\(2026\)\)](https://arxiv.org/html/2609.12303#bib.bib27)propose dynamic chunking for end to end byte level sequence modeling\. Very recently EvaByte[Zheng et al\. \(\(2025\)\)](https://arxiv.org/html/2609.12303#bib.bib60)trained byte level transformers with various tricks like multi\-byte prediction and linearized attention\. We study three decoder\-only transformer variants\([Vaswani et al\., \(2017\)](https://arxiv.org/html/2609.12303#bib.bib54)\)with vocabulary sizes of 128256 \(Token\-1B\), 260 \(Bytes\-1B\), and 261 \(End\-Of\-Token\-1B\)\. They differ in both parameter count \(from the embedding layers\) and FLOPs \(by varying sequence lengths\)\. By preserving the teacher distribution exactly while spending≈\\approx30\.94%additional compute on a fixed amount of data, theEnd\-Of\-Token\-1Bmodel asymptotically outperforms the distilledToken\-1B\. We also note thatEnd\-Of\-Token\-1Bmodel despite having fewer parameters can asymptotically surpassToken\-1Bon downstream tasks\.

## 10Discussion

Marginalize\-It vs End Of Token\.Our findings show thatMarginalize\-It, despite being an approximation asymptotically outperforms theTokendistillation baseline\.End\-Of\-Tokenoffers two advantages: 1\) it preserves the teacher distribution exactly, and 2\) it improves asymptotic performance of the supervisedEnd\-Of\-Token\-1Bbaseline itself at the cost of an extra≈\\approx30\.94%compute per unit of data\. Due to these two properties,End\-Of\-Tokendistillation asymptotically outperforms all of the other five models on downstream tasks\.

The role of<eot\>tokens on power laws\.Our work shows that simply adding<eot\>token every≈4\.5\\approx 4\.5bytes on average affects the power law coefficients[3](https://arxiv.org/html/2609.12303#S4.F3)\. We also observe that supervisedEnd\-Of\-Token\-1Byields better asymptotic downstream performance compared to theBytes\-1Bbaseline\. This finding opens many exciting avenues for future work\.

Inference Costs\.While our scaling laws show that, distilling Llama 3\-8B\([Grattafiori et al\., \(2024\)](https://arxiv.org/html/2609.12303#bib.bib20)\)model intoEnd\-Of\-Token\-1Bmodels with 1\.28B total parameters is asymptotically better than distilling intoToken\-1Bmodels with 1\.81B total parameters,End\-Of\-Token\-1Bmodels remain significantly expensive at inference time\. Comparing the asymptotic performance of a largerToken\-1Bmodel against that of a smallerEnd\-Of\-Token\-1Btransformers with matched inference cost could reveal whether distilledEnd\-Of\-Token\-1Btransformers are also efficient at inference time; this is beyond the scope of this study\.

BPB vs\. Benchmark Performance Discrepancy\.Our observations reveal that BPB is not a metric that enables fair comparison across models with different training objectives and tokenization schemes\. Many other works observe the same phenomenon for different optimizers\([Zhang et al\., \(2026\)](https://arxiv.org/html/2609.12303#bib.bib59)\), data distributions\([Gadre et al\., \(2024\)](https://arxiv.org/html/2609.12303#bib.bib17)\), long\-context setting,\([Fang et al\., \(2025\)](https://arxiv.org/html/2609.12303#bib.bib16)\), byte modeling\([Lee et al\., \(2026\)](https://arxiv.org/html/2609.12303#bib.bib33)\), and in theoretical analyses\([Liu et al\., \(2023\)](https://arxiv.org/html/2609.12303#bib.bib35);[Veličković et al\., \(2026\)](https://arxiv.org/html/2609.12303#bib.bib55)\)\. We identify additional settings in which both the training objective \(Cross Entropy Supervised vs Distillation\) and the tokenization scheme \(Token,Bytes,Bytes w/ <eot\>\) can alter the downstream\-task\-performance vs\. validation\-BPB scaling trajectories, implying that a model with a lower BPB may not be necessarily a better model\.

### 10\.1Future work

Our scaling study focuses on dense transformer models\. It would be interesting to explore how introducing a third axis of sparsity, by training Mixture\-of\-Experts models affects these power laws\. Next, our study examines scaling trends across varying data budgets with approximately fixed model size of 1\.28 billion layer parameters, ranging up to 1 trillion bytes\. We also note that some of our larger training runs epoch\. Future work should explore training with larger model sizes and more data\. Next we report our findings based on eight benchmarks spanning three categories: Multiple Choice Question Answering, Language Generation, and Machine Translation\. Future work should explore evaluating models on more benchmarks\. Lastly, understanding the training dynamics of distilling large byte transformers into smaller ones remains an open question\.

## Acknowledgments

We thank Jonathan Hayase, Alisa Liu, and Benjamin Minixhofer for thoughtful discussions and feedback on this work\.

###### Contents

1. [1Introduction](https://arxiv.org/html/2609.12303#S1)1. [1\.1Summary of Contributions](https://arxiv.org/html/2609.12303#S1.SS1)
2. [2Logit Conversion Methods](https://arxiv.org/html/2609.12303#S2)1. [2\.1Marginalize\-It Logits](https://arxiv.org/html/2609.12303#S2.SS1) 2. [2\.2End\-Of\-Token Logits](https://arxiv.org/html/2609.12303#S2.SS2)
3. [3Scaling Study Design](https://arxiv.org/html/2609.12303#S3)1. [3\.1Experimental Setup](https://arxiv.org/html/2609.12303#S3.SS1) 2. [3\.2Scaling Along Two Axes: Tokenization and Training Objectives](https://arxiv.org/html/2609.12303#S3.SS2) 3. [3\.3Benchmark Evaluations](https://arxiv.org/html/2609.12303#S3.SS3)
4. [4Scaling Trends I: Validation BPB vs Training FLOPS](https://arxiv.org/html/2609.12303#S4)1. [4\.1Validation BPB vs Training FLOP Plots](https://arxiv.org/html/2609.12303#S4.SS1)
5. [5Scaling Trends II: Task Performance vs Training FLOPs](https://arxiv.org/html/2609.12303#S5)1. [5\.1Results on Multiple Choice Question Answering Benchmarks](https://arxiv.org/html/2609.12303#S5.SS1) 2. [5\.2Results on Language Generation Benchmarks](https://arxiv.org/html/2609.12303#S5.SS2) 3. [5\.3Results on Machine Translation Benchmarks](https://arxiv.org/html/2609.12303#S5.SS3) 4. [5\.4Concluding Remarks](https://arxiv.org/html/2609.12303#S5.SS4)
6. [6Scaling Trends III: Task Performance vs Validation BPB](https://arxiv.org/html/2609.12303#S6)1. [6\.1Results on Multiple Choice Question Answering Benchmarks](https://arxiv.org/html/2609.12303#S6.SS1) 2. [6\.2Results on Language Generation Benchmarks](https://arxiv.org/html/2609.12303#S6.SS2) 3. [6\.3Results on Machine Translation Benchmarks](https://arxiv.org/html/2609.12303#S6.SS3) 4. [6\.4Fitting the Downstream Task Error vs Validation BPB Scaling Law](https://arxiv.org/html/2609.12303#S6.SS4) 5. [6\.5Downstream Task Performance Upper Bound](https://arxiv.org/html/2609.12303#S6.SS5) 6. [6\.6When Lower BPB Does Not \(Always\) Imply Better Models](https://arxiv.org/html/2609.12303#S6.SS6)
7. [7The Case forEnd\-Of\-TokenLogits](https://arxiv.org/html/2609.12303#S7)1. [7\.1Feather Plot: Visualizing Iso\-FLOP Point Connections](https://arxiv.org/html/2609.12303#S7.SS1) 2. [7\.2Calculating Downstream Error Cross\-Over Point with Token Distillation](https://arxiv.org/html/2609.12303#S7.SS2) 3. [7\.3Results](https://arxiv.org/html/2609.12303#S7.SS3)
8. [8Comparisons with Open\-Weight models](https://arxiv.org/html/2609.12303#S8)
9. [9Related Work](https://arxiv.org/html/2609.12303#S9)
10. [10Discussion](https://arxiv.org/html/2609.12303#S10)1. [10\.1Future work](https://arxiv.org/html/2609.12303#S10.SS1)
11. [AExperimental Setup Details](https://arxiv.org/html/2609.12303#A1)1. [A\.1Model Architecture and FLOP Calculations\.](https://arxiv.org/html/2609.12303#A1.SS1) 2. [A\.2Pretraining Dataset, Tokenization, and Data Scales](https://arxiv.org/html/2609.12303#A1.SS2) 3. [A\.3Loss Function and Optimization Settings](https://arxiv.org/html/2609.12303#A1.SS3) 4. [A\.4Sequence Length and Global Batch Size\.](https://arxiv.org/html/2609.12303#A1.SS4) 5. [A\.5Bits Per Byte Calculation\.](https://arxiv.org/html/2609.12303#A1.SS5) 6. [A\.6Storage Cost Calculations of Token and Byte Logits](https://arxiv.org/html/2609.12303#A1.SS6)
12. [BAdditional Results using Validation Dataset 1](https://arxiv.org/html/2609.12303#A2)
13. [CComparisons with open\-weight models: Additional details](https://arxiv.org/html/2609.12303#A3)
14. [DOn Scaling Laws Sensitivity](https://arxiv.org/html/2609.12303#A4)1. [D\.1Leave One Out Error Analysis](https://arxiv.org/html/2609.12303#A4.SS1) 2. [D\.2Residual Weighting Scheme Sensitivity Analysis](https://arxiv.org/html/2609.12303#A4.SS2) 3. [D\.3Choice of the validation data](https://arxiv.org/html/2609.12303#A4.SS3)
15. [EAdditional Results using Validation Dataset 2](https://arxiv.org/html/2609.12303#A5)1. [E\.1BPB vs Training FLOPs power law equations](https://arxiv.org/html/2609.12303#A5.SS1) 2. [E\.2Average top\-1 Error Scaling Laws](https://arxiv.org/html/2609.12303#A5.SS2) 3. [E\.3Feather Plots: Comparisons using validation dataset 2](https://arxiv.org/html/2609.12303#A5.SS3) 4. [E\.4Predicting the asymptotic Average Downstream task performance](https://arxiv.org/html/2609.12303#A5.SS4)
16. [FDistillation Methods: Additional Details](https://arxiv.org/html/2609.12303#A6)1. [F\.1Marginalize\-ItandEnd\-Of\-TokenPseudocode](https://arxiv.org/html/2609.12303#A6.SS1) 2. [F\.2ComparingMarginalize\-Itmethod withEnd\-Of\-Token](https://arxiv.org/html/2609.12303#A6.SS2)
17. [GOn BPB vs Benchmark Performance Discrepancy](https://arxiv.org/html/2609.12303#A7)1. [G\.1Why the same BPB may imply different downstream performance?](https://arxiv.org/html/2609.12303#A7.SS1)
18. [References](https://arxiv.org/html/2609.12303#bib)

## Appendix AExperimental Setup Details

In this section we provide additional details on the design decisions for our experiments\.

### A\.1Model Architecture and FLOP Calculations\.

To train both the token and byte student models, we use the transformer\([Vaswani et al\., \(2017\)](https://arxiv.org/html/2609.12303#bib.bib54)\)architecture with 1 billion parameters\. We use SwiGLU\([Shazeer, \(2020\)](https://arxiv.org/html/2609.12303#bib.bib51)\)activation function\. The differences in the number of parameters and training FLOPs arise from the differences in the vocabulary size – byte models use fewer parameters and FLOPs per byte\.

We use Llama3\-8B\([Grattafiori et al\., \(2024\)](https://arxiv.org/html/2609.12303#bib.bib20)\)as our teacher model\.

Table 9:Model Architecture Comparison
### A\.2Pretraining Dataset, Tokenization, and Data Scales

We use the Llama\-2\([Touvron et al\., \(2023\)](https://arxiv.org/html/2609.12303#bib.bib53)\)pretraining mixture for all of our experiments\. We tokenize the data using two tokenizers: BPE\-Tiktoken \(Llama3\-8B\) tokenizer and the byte tokenizer\. Our byte tokenizer uses a vocabulary size of 260: 256 UTF\-8 bytes and 4 special tokens\. We train multiple 1B models with varying D where D is the number of tokens/byte units\.

### A\.3Loss Function and Optimization Settings

A typical knowledge distillation setup involves two components:α​LK​L\\alpha\{L\_\{KL\}\}, a loss term computed using the teacher’s logits and\(1−α\)​LC​E\(1\-\\alpha\)\{L\_\{CE\}\}, the next token prediction loss computed against the ground truth data\. Whenα=0\\alpha=0, the objective reduces to supervised training \(Cross\-Entropy loss with respect to data\) and whenα=1\\alpha=1, the model is trained purely via teacher distillation\. We use equation[3](https://arxiv.org/html/2609.12303#A1.E3)as our loss function and useα\\alpha, the mixing coefficient to control our supervised and distillation runs\.

LKD=α⋅LKL\+\(1−α\)⋅LCE\{L\}\_\{\\text\{KD\}\}=\\alpha\\cdot\{L\}\_\{\\text\{KL\}\}\+\(1\-\\alpha\)\\cdot\{L\}\_\{\\text\{CE\}\}\(3\)
LKL=1n​∑t=1n∑cpT​\(c∣x<t\)​log⁡pT​\(c∣x<t\)pS​\(c∣x<t\)L\_\{\\text\{KL\}\}=\\frac\{1\}\{n\}\\sum\_\{t=1\}^\{n\}\\sum\_\{c\}p\_\{T\}\(c\\mid x\_\{<t\}\)\\log\\frac\{p\_\{T\}\(c\\mid x\_\{<t\}\)\}\{p\_\{S\}\(c\\mid x\_\{<t\}\)\}\(4\)
LCE=−1n∑t=1n∑cyt,clogpS\(c∣x<t\)L\_\{\\text\{CE\}\}=\-\\frac\{1\}\{n\}\\sum\_\{t=1\}^\{n\}\\sum\_\{c\}y\_\{t,c\}\\log p\_\{S\}\(c\\mid x\_\{<t\}\)\(5\)
where:

- •ttis the position index in the sequence
- •nnis the total number of tokens in the sequence
- •ccis the token index over the vocabularyV\{V\}
- •yt,c∈\{0,1\}y\_\{t,c\}\\in\\\{0,1\\\}is a one\-hot indicator that the ground\-truth token at positionttiscc
- •pT​\(c∣x<t\)p\_\{T\}\(c\\mid x\_\{<t\}\)is the teacher model’s predicted probability distribution over the next token given all preceding tokensx<tx\_\{<t\}
- •pS​\(c∣x<t\)p\_\{S\}\(c\\mid x\_\{<t\}\)is the student model’s predicted probability distribution over the next token given all preceding tokensx<tx\_\{<t\}
- •α∈\[0,1\]\\alpha\\in\[0,1\]is a weighting factor balancing the KL divergence and cross\-entropy losses

We train our models using the AdamW\([Loshchilov & Hutter, \(2017\)](https://arxiv.org/html/2609.12303#bib.bib36)\)optimizer withβ1=0\.9\\beta\_\{1\}=0\.9andβ2=0\.95\\beta\_\{2\}=0\.95and a cosine learning rate schedule\. We set warmup to 10% of the maximum number of steps for each data scale use and three peak learning rates∈\\in\{1e\-3, 4e\-3, 8e\-3\}\. We use independent weight decay\([Kosson et al\., \(2025\)](https://arxiv.org/html/2609.12303#bib.bib31)\)of 1e\-4 and a grad clip norm of 1\.0\.

### A\.4Sequence Length and Global Batch Size\.

We set sequence length to 2048 for ourToken\-1Bmodels\. We train models using 64 H200s per training run and use a global batch size is 0\.5M tokens\. For training byte transformers, our initial explorations with the exact same context and data \(8K×\\times4×\\times64 = 2M bytes\) per optimization step resulted in loss spikes\. To stabilize the runs we use qk\-norm and set the sequence length to 9216 forBytes\-1Band 11264 forEnd\-Of\-Token\-1B, and maintain the global batch size as that of the token models\. If the byte sequence formed by the 2048 tokens exceeds the byte sequence length, we truncate it; if the sequence is shorter, we pad it with padding tokens to preserve the context length\. We use 64 H200s for eachBytes\-1BandEnd\-Of\-Token\-1Bas well, maintaining the global batch size of≈\\approx0\.5M byte units as that of the token models\. Hence, overall our byte models use much less data compared to token based models for the same compute budget\.

### A\.5Bits Per Byte Calculation\.

We use BPB \(Bits per Byte\) \(equation[6](https://arxiv.org/html/2609.12303#A1.E6)\), a smooth tokenizer independent metric on a held out validation set to plot the scaling curves\. The BPB is defined as the total cross\-entropy lossℒCE\\mathcal\{L\}\_\{\\text\{CE\}\}summed over the validation dataxxdivided by the total number of bytesBBinxxand scaled by a constant\. Intuitively, it captures the average number of bits a model uses to represent the data\.

BPB=LCE​\(x\)⋅log2⁡eB\\text\{BPB\}=\\frac\{\{L\}\_\{\\text\{CE\}\}\(x\)\\cdot\\log\_\{2\}e\}\{B\}\(6\)
Finally, We use Llama 3\-8B\([Grattafiori et al\., \(2024\)](https://arxiv.org/html/2609.12303#bib.bib20)\)and Llama3\.2\-1B\([Meta AI, \(2024\)](https://arxiv.org/html/2609.12303#bib.bib39)\)models as reference points to compare with our token and byte asymptotes\. Since these models are overtrained on several trillion tokens, they simulate the infinite data regimes\.

### A\.6Storage Cost Calculations of Token and Byte Logits

The biggest pain point of using offline distillation to train small language models is the storage cost of the teacher logits\. Each BPE token of a Llama3\-8B model, for instance, produces 128256 logit values\. A billion parameter model overtrained to 2T tokens, for instance, would require and2​T×4×1282562T\\times 4\\times 128256bytes = 1\.026048 Exabytes assumingfloat32datatype\. To circumvent this storage problem, researchers store only thetop\-klogits per token, wherekktypically is of the order of several hundreds\. On the contrary, byte transformers use a fixed vocabulary of only≈\\approx260 bytes, trading vocabulary size for increased sequence length\. Our controlled study is designed to characterize this tradeoff\.

Table 10:Storage cost analysis for top\-k logits per token\. We solve for k and found it to be≈\\approx600In table[10](https://arxiv.org/html/2609.12303#A1.T10)we detail the storage cost of storing one BPE token and theirtop\-klogits\. Note that for BPE tokens, we use the sparse representation of logits while for bytes logits we store the full distribution\. By equating the total cost of BPE\-tiktoken with bytes, we solve fork=4685/8k=4685/8, which turns out to be 585\.625≈\\approx600\. Therefore for our controlled comparison we store 600 logits per BPE\-tiktoken token and all of the byte logits i\.e\. 260 values for distillation\.

## Appendix BAdditional Results using Validation Dataset 1

In this section we provide additional results for the learning rates of 1e\-3 an 8e\-3 for validation dataset 1 \(which is used for analysis in the paper\)\.

Figure 12:The figure shows our scaling curves for different data scales and learning rates for a fixed parameter count of approximately 1 billion parameter transformer models\(see Table[9](https://arxiv.org/html/2609.12303#A1.T9)for details\)\. The distilled models are trained using Llama 3\-8B as a teacher model\. The validation BPB of the teacher model is 0\.85511\.Table 11:Power\-law equationsy=b⋅xa\+cy=b\\cdot x^\{a\}\+cfitted to BPB vs\. training FLOPs\.Table 12:We demonstrate that the amount of FLOPs spent on a fixed amount of data \(one BPE token in this case\) affects the asymptotic BPB\. Overall:Bytes w/ <eot\> SupervisedBPB<<Bytes SupervisedBPB<<Token SupervisedBPB\. andEnd\-Of\-Token DistilledBPB<<Marginalize\-It Distilled\) BPB<<Token DistilledBPB\. We also note that for all learning rates, within the groups \(based on training objective\), the supervised and distilled models exhibit small differences in the asymptotic value\. The FLOPs/Unit values are obtained from Table[9](https://arxiv.org/html/2609.12303#A1.T9)\.Table 13:Scaling\-law goodness of fit\. Each entry is the coefficient of determinationR2=1−SSres/SStotR^\{2\}=1\-\\mathrm\{SS\_\{res\}\}/\\mathrm\{SS\_\{tot\}\}for the fity=b⋅xa\+cy=b\\cdot x^\{a\}\+cof BPB versus training FLOPs\.Figure 13:We plot Accuracy vs training FLOP curves for all the six scenarios from section[3\.2](https://arxiv.org/html/2609.12303#S3.SS2)\. For all of the Multiple Choice Question Answering Benchmarks, we observe that the token models show better performance at low compute budgets and eventually plateau\. Byte models, start worse yet improve at a much faster rate\.Figure 14:We visualize%\\%Accuracy vs training FLOPs curves for language generation tasks\. On generative tasks, token models achieve better performance at lower compute budgets and appear to plateau with the training FLOPs\.Bytes\-1BandEnd\-Of\-Token\-1Bmodels show a different scaling behavior, with much worse performance at a lower compute budget which continues to improve with FLOPs with no sign of saturation\.Figure 15:We plot BLEU score for the translation tasks for different training budgets\. We observe clearly different scaling behaviors for token and byte models\. Token models can achieve significantly better performance at a smaller compute budgets\. The BLEU score appears to saturate with overtraining\. Byte models both with and without the<eot\>demonstrate extremely low BLEU scores at lower compute budgets yet the rate of improvement with overtraining is better compared to token models\.Figure 16:We visualize Average\-top\-1 error vs the validation BPB curves for the six scenarios for different learning rates and Multiple Choice Question Answering benchmarks\. All of these curves including the averaged metric appear to follow an exponential decay\. HellaSwag demonstrates smoothest scaling behavior\.Figure 17:We plot the Average\-top\-1 error vs BPB curves for the MBPP and the NQ benchmarks\. We notice an exponential decay for all three scenarios, with significantly different trends for the token and byte models\. MBPP shows noisier behavior compared to NQ\. We deduce that the same validation Bits\-Per\-Byte imply different downstream performace for different training objectives\(distillation vs cross\-entropy\) tokenization schemes\(tokens vs bytes\)\. Distilled models as well demonstrate slightly different scaling behavior compared to the supervised counterparts\.Figure 18:We plot BLEU score vs Validation BPB performance for English to German translation, German to English translation, and on the averaged BLEU performance for the six scenarios for three learning rates\. We notice the differences in the scaling behavior across tokenization schemes as well as the training objectives\. In some cases distillation appears to exhibit monotonic trend compared to the noisy supervised curves especially for English to German translation\.Figure 19:Downstream Error Scaling lawsTable 14:Fitted downstream error scaling lawsz⁡\(y\)=ϵ−k⋅e−γ​yz\(y\)=\\epsilon\-k\\cdot e^\{\-\\gamma y\}\(equation[2](https://arxiv.org/html/2609.12303#S6.E2)\) for each model variant and learning rate, whereyyis the validation BPB\.Table 15:Asymptotic validation BPB \(c\\mathrm\{c\}\) and the corresponding downstream average accuracya⋆=1−%z⋆a^\{\\star\}=1\-\\%z^\{\\star\}wherez⋆z^\{\\star\}is the Asymptotic Average top\-1 error predicted by the scaling law equation:[2](https://arxiv.org/html/2609.12303#S6.E2),z⋆=ϵ−k⋅e−γ​cz^\{\\star\}=\\epsilon\-k\\cdot e^\{\-\\gamma c\}, whereccis obtained from the power law equations in Table[4](https://arxiv.org/html/2609.12303#S6.T4)for each model variant and learning rate\.![Refer to caption](https://arxiv.org/html/2609.12303v1/val_1_row2_isoflop_lines.png)Figure 20:Feather plots for three learning rates on validation dataset 1Table 16:Training compute at whichMarginalize\-It DistilledandEnd\-Of\-Token Distilledmatch the asymptotic downstream performance of Token Distilled\. We take Token Distilledś asymptotic BPB \(theccterm of its fitted BPB–FLOPs power law; columnToken Dist\. BPB\) and map it through Token Distilledś benchmark scaling law to obtain its asymptotic downstream accuracy \(columnAccuracy\)\. For each byte method we then invert its own benchmark scaling law at that accuracy to get the BPB it must attain \(columnMethod BPB\), and invert its BPB–FLOPs power law at that BPB to get the required training compute \(columnTraining FLOPs\)\. The accuracy column is shared within each learning rate because both methods are evaluated against the same Token Distilled asymptote\.Table 17:Effective training\-data volume at the compute budget required to achieve theTokendistillation asymptotic performance from Table[5](https://arxiv.org/html/2609.12303#S6.T5)\. HereFPU\\mathrm\{FPU\}\(*FLOPs Per Unit*\) is the training FLOPs consumed per sequence unit, where a unit is one token forToken Distilledand one byte for the byte\-level methods; it is obtained from Table[9](https://arxiv.org/html/2609.12303#A1.T9)\.Byte Units=FLOPs/FPUbytes=\\text\{FLOPs\}/\\mathrm\{FPU\}\_\{\\text\{bytes\}\}is the number of byte units the method processes at its crossover compute\.Bytes=Byte Units×Multiplier=\\text\{Byte Units\}\\times\\text\{Multiplier\}converts byte units into actual UTF\-8 bytes of unique text, accounting for the fraction of each unit that is genuine content \(Multiplier=1=1forMarginalize\-It Distilled;=4\.5/5\.5=4\.5/5\.5forEnd\-Of\-Token Distilled, whose sequences carry an extra<eot\>per4\.54\.5content bytes\)\.Tokens=FLOPs/FPUtokens=\\text\{FLOPs\}/\\mathrm\{FPU\}\_\{\\text\{tokens\}\}is the token countToken Distilledwould process at the same compute\.Token to Byte=Tokens×4\.5=\\text\{Tokens\}\\times 4\.5expresses that token count in bytes using the average of4\.54\.5bytes per token\.Data Savings=Token to Byte/Bytes=\\text\{Token to Byte\}/\\text\{Bytes\}is how much less unique text the byte method needs to reach the same performance\. To calculate storage savings we use our storage cost calculations from Table[10](https://arxiv.org/html/2609.12303#A1.T10)where we equate the storage cost of oneTokenwith 4\.5Bytesto obtain k \(for top\-k\)\.Storage Savings=\(Tokens×4\.5\)/Byte Units=\(\\text\{Tokens\}\\times 4\.5\)/\\text\{Byte Units\}compares the byte footprint ofToken Distilledapproach against the raw byte\-unit count\.
## Appendix CComparisons with open\-weight models: Additional details

Table 18:Downstream benchmark accuracy \(%\) for the4​e−34\\text\{e\}\{\-\}3learning\-rate runs at their maximum available data scale, compared against the Llama and Gemma 1B baselines\. Avg\. / Asympt\. Avg\. is the mean over the six benchmarks\.
## Appendix DOn Scaling Laws Sensitivity

We now characterize the sensitivity of downstream error scaling laws\.

### D\.1Leave One Out Error Analysis

We analyze the Leave\-One\-Out sensitivity of the power laws\.

Figure 21:We visualize the sensitivity of the scaling law fits via leave\-one\-out analysis\. For each learning rate and for each experiment, we drop one of the datapoints and fit the power law using the rest and visualize the range of c\.Tokencurves on average demonstrate little spread compared toBytesandBytes w/ <eot\>methods\.
### D\.2Residual Weighting Scheme Sensitivity Analysis

Figure 22:We visualize the sensitivity of the residual weighting scheme on the power laws\. Three weighting schemes, each a choice of theσ\\sigmavector incurve\_fit’s objective∑i\(\(y^​\(xi\)−yi\)/σi\)2\\sum\_\{i\}\\big\(\(\\hat\{y\}\(x\_\{i\}\)\-y\_\{i\}\)/\\sigma\_\{i\}\\big\)^\{2\}:
•unweighted: ordinary least squares\.
•weighted y: minimizes relative error\(y^−y\)/y\(\\hat\{y\}\-y\)/y, down\-weighting high\-BPB low\-compute runs\.
•inverse x: shrinksσ\\sigmafor large\-compute points so they dominate the extrapolation to high compute points\.
We useunweightedmethod for all of our plots\. We note that other weighting schemes are more sensitive toBytesandBytes w/ <eot\>tokenizations\.
### D\.3Choice of the validation data

We observe that the choice of the validation dataset can alter the cross\-over points of the training curves and as a result the perception of a better model\. Consider for example in Figure[23](https://arxiv.org/html/2609.12303#A4.F23)\. In Validation Dataset 2, the token curves have not intersected the byte curves yet, while in Validation Dataset 1 they cross them quickly\. The nature of theFeather Plotsalso change with changes in the validation data\. Moreover, the amount of overtraining required to reliably predict the asymptotic behavior could also change with change in the validation dataset\. Hence, taking performance on multiple validation sets into account, when fitting scaling laws might be helpful in predicting the final model performance\.

Figure 23:We demonstrate how changes in the validation dataset can affect the validation BPB vs training FLOPs scaling plots by comparing power laws for LR=4e\-3\. At the 220BToken SupervisedFLOP budget, all of the other five models have a lower BPB for validation 1\. On the other hand, for validation dataset 2, only the distilled token model has a lower BPB\. The two datasets also have different asymptotic Bits\-Per\-Byte values for all six curves\.Figure 24:BPB vs FLOPs for validation dataset 2\.

## Appendix EAdditional Results using Validation Dataset 2

### E\.1BPB vs Training FLOPs power law equations

Table 19:Fitted BPB\-vs\-FLOPs power laws of the formy=b​xa\+cy=b\\,x^\{a\}\+c, wherexxis training FLOPs andyyis bits\-per\-byte on validation dataset 2\.
### E\.2Average top\-1 Error Scaling Laws

Table 20:Fitted downstream error scaling lawsz⁡\(y\)=ϵ−k⋅e−γ​yz\(y\)=\\epsilon\-k\\cdot e^\{\-\\gamma y\}for each model variant and learning rate, whereyyis the validation BPB calibrated using Validation dataset 2 BPB values\.Figure 25:Downstream error scaling laws calibrated using Validation dataset 2 BPB values\.Table 21:Asymptotic validation BPB \(BPB⋆\\mathrm\{BPB\}^\{\\star\}\) and the corresponding downstream average accuracy predicted by the scalingz⁡\(y\)=ϵ−k⋅e−γ​yz\(y\)=\\epsilon\-k\\cdot e^\{\-\\gamma y\}aty=cy=c, for each model variant and learning rate calibrated using Validation dataset 2 BPB values\.
### E\.3Feather Plots: Comparisons using validation dataset 2

![Refer to caption](https://arxiv.org/html/2609.12303v1/val_2_row2_isoflop_lines.png)Figure 26:Feather plots calibrated using Validation dataset 2 BPB values
### E\.4Predicting the asymptotic Average Downstream task performance

Table 22:Training compute at whichMarginalize\-It DistilledandEnd\-Of\-Token Distilledmatch the asymptotic downstream performance of Token Distilled using Validation Dataset 2\. We take Token Distilledś asymptotic BPB \(theccterm of its fitted BPB–FLOPs power law; columnToken Dist\. BPB\) and map it through Token Distilledś benchmark scaling law to obtain its asymptotic downstream accuracy \(columnAccuracy\)\. For each byte method we then invert its own benchmark scaling law at that accuracy to get the BPB it must attain \(columnMethod BPB\), and invert its BPB–FLOPs power law at that BPB to get the required training compute \(columnTraining FLOPs\)\. The accuracy column is shared within each learning rate because both methods are evaluated against the same Token Distilled asymptote\.Table 23:Effective training\-data volume at the compute budget of Table[22](https://arxiv.org/html/2609.12303#A5.T22)using Validation Dataset 2\. HereFPU\\mathrm\{FPU\}\(FLOPs Per Unit\) is the training FLOPs consumed per sequence unit, where a unit is one token for Token Distilled and one byte for the byte\-level methods; it is read directly from each run and used to convert a compute budget into a data count\.Byte Units=FLOPs/FPUbytes=\\text\{FLOPs\}/\\mathrm\{FPU\}\_\{\\text\{bytes\}\}is the number of byte units the method processes at its crossover compute\.Bytes=Byte Units×Multiplier=\\text\{Byte Units\}\\times\\text\{Multiplier\}converts byte units into actual UTF\-8 bytes of unique text, accounting for the fraction of each unit that is genuine content \(Multiplier=1=1forMarginalize\-It Distilled;=4\.5/5\.5=4\.5/5\.5forEnd\-Of\-Token Distilled, whose sequences carry an extra end\-of\-token symbol per4\.54\.5content bytes\)\.Tokens=FLOPs/FPUtokens=\\text\{FLOPs\}/\\mathrm\{FPU\}\_\{\\text\{tokens\}\}is the token count Token Distilled would process at the same compute\.Token to Byte=Tokens×4\.5=\\text\{Tokens\}\\times 4\.5expresses that token count in bytes using the average of4\.54\.5bytes per token\.Data Savings=Token to Byte/Bytes=\\text\{Token to Byte\}/\\text\{Bytes\}is how much less unique text the byte method needs to reach the same performance\.Storage Savings=\(Tokens×4\.5\)/Byte Units=\(\\text\{Tokens\}\\times 4\.5\)/\\text\{Byte Units\}compares the token pipelineś byte footprint against the raw byte\-unit count\.

## Appendix FDistillation Methods: Additional Details

### F\.1Marginalize\-ItandEnd\-Of\-TokenPseudocode

importnumpyasnp

EOT=256

defbyte\_probabilities\(probs,token\_to\_bytes,lengths,gold,use\_eot=False\):

"""BytedistributionsP\(b\|gold\[:k\],context\)fork=0\.\.len\(gold\)\-1\.

probs\(vocab\_size\):tokenprobabilities\.

token\_to\_bytes\(vocab\_size,max\_len\):per\-tokenbytes,\-1padded\.

lengths\(vocab\_size\):truebytelength\.

gold:bytesequencetoconditionon\.

use\_eot:ifTrue,alphabetis\{0\.\.255\}U\{EOT\}\(size257\)else256\.

"""

n\_bytes=257ifuse\_eotelse256

live=np\.ones\(len\(probs\),dtype=bool\)

dist=\[\]

forkinrange\(len\(gold\)\):

active=live&\(lengths\>k\)

bytes\_at\_k=token\_to\_bytes\[:,k\]

accum=np\.bincount\(bytes\_at\_k\[active\],probs\[active\],minlength=n\_bytes\)

total=accum\.sum\(\)

dist\.append\(accum/totaliftotal\>0elseaccum\)

live&=\(bytes\_at\_k==gold\[k\]\)&\(lengths\>k\+1\)

returndist

### F\.2ComparingMarginalize\-Itmethod withEnd\-Of\-Token

In this section we discuss four special cases of byte logit conversion:

Case I: Predicting token’s first byte distribution using token level teacher distribution \(Figure[27](https://arxiv.org/html/2609.12303#A6.F27)\):

\(a\)B1 is predicted directly by marginalization using next token distribution of tokeniramand is exact\.\(b\)B1 is predicted directly by marginalization using next token distribution of tokeniram<eot\>and is exact\.
Figure 27:We demonstrate the first byte prediction of the next token\. BothMarginalize\-ItandEnd\-Of\-Tokenmethods exactly preserve teacher distributions in this caseCase II: Predicting token’s intermediate byte distribution using token level teacher distribution when the ground truth token length matches with all other alternative tokens with non\-zero probability in the vocabulary \(Figure[28](https://arxiv.org/html/2609.12303#A6.F28)\): :

\(a\)B3 prediction is based on tokens with prefixiswith a length of at least 3 bytes\. Other alternative paths \(with non zero probability\) have exactly 3 bytes length and can be marginalized exactly\.\(b\)B3 prediction is based on tokens with the prefixisand a length of at least 3 bytes\. All alternative paths with non zero probabilities have byte length 4 and can be marginalized exactly\. Similarly, for B4 prediction, all paths have length of 4 and can be marginalized exactly\.
Figure 28:We demonstrate the intermediate byte prediction case when all other alternative paths with non zero probability in the vocabulary have exactly the same length as that of the ground truth token\. In this case, bothMarginalize\-ItandEnd\-Of\-Tokenmethods behave equally and give exact byte probabilities\.Case III: Predicting token’s intermediate byte distribution using token level teacher distribution when there exist other alternative tokens with non zero probabilities in the vocabulary sharing the same prefix but having shorter byte sequence length compared to the ground truth token \(Figure[29](https://arxiv.org/html/2609.12303#A6.F29)\):\.

\(a\)B3 prediction is based on tokens with prefixiswith a length of at least 3 bytes\. Therefore it misses other tokenization paths: \[is,ooo\] , \[is,oo\], \[is,uuu\]\(b\)B3 prediction is based on tokens with the prefixisand a length of at least 3 bytes\. Adding<eot\>preserves the distribution, as the byte length ofis<eot\>is now 3 and no alternative token paths are missed\. B4 prediction is conditioned on tokens starting with prefixisuand with a length of at least 4\. Since only the tokenisu<eot\>satisfies that condition, the teacher distribution is preserved exactly\.
Figure 29:We demonstrate the intermediate byte prediction when there exits other tokens in the vocabulary with shorter byte length with non zero probabilities\. SinceMarginalize\-Itcomputes probabilities based on the ground truth token it can miss these alternative paths\.End\-Of\-Tokenmethod on the other hand circumvents this problem by adding the new<eot\>token\.Case IV: Predicting token’s intermediate byte distribution using token level teacher distribution when there exist other alternative tokens with non zero probabilities in the vocabulary sharing the same prefix but having longer byte sequence length compared to the ground truth token\. \(Figure[30](https://arxiv.org/html/2609.12303#A6.F30)\):

\(a\)B3 prediction is based on the next token distribution ofisand therefore it misses contributions from other tokenization paths formed by tokensisuandisk\.\(b\)B3 prediction is based on tokens with prefixisand has a length of at least 3 bytes\. Adding<eot\>preserves the distribution as now byte length ofis<eot\>is 3 and no alternative token paths are missed\. B4 prediction is based on the next token distribution ofis<eot\>and preserves exact distribution as no other token paths have prefixis<eot\>\.
Figure 30:We demonstrate intermediate byte prediction when there exist other tokens in the vocabulary with longer byte lengths and non\-zero probabilities\. SinceMarginalize\-Itcomputes probabilities based on the ground\-truth token, it can miss the contributions from these alternative paths\. TheEnd\-Of\-Tokenmethod, on the other hand, circumvents this problem by adding the new<eot\>token\.

## Appendix GOn BPB vs Benchmark Performance Discrepancy

### G\.1Why the same BPB may imply different downstream performance?

BPB is a function of only one number per position: the probability the model places on the target token,pcorrectp\_\{\\text\{correct\}\}\. The per\-token loss is−log2⁡pcorrect\-\\log\_\{2\}p\_\{\\text\{correct\}\}, so oncepcorrectp\_\{\\text\{correct\}\}is fixed the loss is fixed—BPB is completely blind to how the remaining mass1−pcorrect1\-p\_\{\\text\{correct\}\}is arranged among the other vocabulary items\. Downstream accuracy, in contrast, depend on whether the target is thearg⁡max\\arg\\max, i\.e\. on the ranking of the full distribution\. Since the ranking is governed by the residual mass that BPB ignores, two models can share an identical BPB while decoding completely different text\.

Example:Concretely, take the target “Where’s my tiramisu?”, which the Llama3\-8b tokenizer segments into77tokens spanning2020bytes:

Where\|’s\|␣my\|␣tir\|am\|isu\|?

where␣denotes a leading space\. Suppose both models place probabilitypcorrect=0\.4p\_\{\\text\{correct\}\}=0\.4on the correct token at every position\. Then each per\-token loss is−log2⁡0\.4=1\.32\-\\log\_\{2\}0\.4=1\.32bits, the total is7×1\.32=9\.247\\times 1\.32=9\.24bits, and

BPB=9\.2420=0\.462\\text\{BPB\}\\;=\\;\\frac\{9\.24\}\{20\}\\;=\\;0\.462\(7\)for both models\. The models differ only in where they place the remaining0\.60\.6of mass—which BPB never observes:

Table 24:An illustration of how the relative ranking of tokens in the vocabulary can lead to worse generation performance while maintaining the same BPB value\.In Model A the target is rank 1 at every step, so greedy decoding produces “Where’s my tiramisu?”—a7/77/7exact match\. In Model B a single distractor sits at0\.50\.5, just above the target’s0\.40\.4, so the target is rank 2 at every step and the concatenatedarg⁡max\\arg\\maxtokens spell “There’d be calzone\!”—0/70/7\. The loss never observes the0\.50\.5distractor greater the0\.40\.4target; it considers only the0\.40\.4\. Both models therefore have an identical BPB of0\.4630\.463while achieving100%100\\%versus0%0\\%token accuracy\.Equal BPB constrains the probability of the target but does not guarantee a rank 1\.

## References

- Al\-Rfou et al\. \(\(2018\)\)Al\-Rfou, R\., Choe, D\., Constant, N\., Guo, M\. & Jones, L\.\(2018\)\.Character\-level language modeling with deeper self\-attention\.In Aaai conference on artificial intelligence\.[https://api\.semanticscholar\.org/CorpusID:52004855](https://api.semanticscholar.org/CorpusID:52004855)
- Austin et al\. \(\(2021\)\)Austin, J\., Odena, A\., Nye, M\., Bosma, M\., Michalewski, H\., Dohan, D\.others\(2021\)\.Program synthesis with large language models\.arXiv preprint arXiv:2108\.07732 \.
- Bahri et al\. \(\(2024\)\)Bahri, Y\., Dyer, E\., Kaplan, J\., Lee, J\. & Sharma, U\.\(2024\)\.Explaining neural scaling laws\.Proceedings of the National Academy of Sciences 121 27 e2311878121\.
- Bansal et al\. \(\(2022\)\)Bansal, Y\., Ghorbani, B\., Garg, A\., Zhang, B\., Cherry, C\., Neyshabur, B\. & Firat, O\.\(2022\)\.Data scaling laws in nmt: The effect of noise and architecture\.In International conference on machine learning \( 1466–1482\)\.
- Bao et al\. \(\(2026\)\)Bao, Z\., Leng, J\., Wang, J\., Peng, B\. & Lu, Y\.\(2026\)\.Distilling token\-trained models into byte\-level models\.arXiv preprint arXiv:2602\.01007 \.
- Beyer et al\. \(\(2022\)\)Beyer, L\., Zhai, X\., Royer, A\., Markeeva, L\., Anil, R\. & Kolesnikov, A\.\(2022\)\.Knowledge distillation: A good teacher is patient and consistent\.In Proceedings of the ieee/cvf conference on computer vision and pattern recognition \( 10925–10934\)\.
- Bhagia et al\. \(\(2024\)\)Bhagia, A\., Liu, J\., Wettig, A\., Heineman, D\., Tafjord, O\., Jha, A\.H\.others\(2024\)\.Establishing task scaling laws via compute\-efficient model ladders\.arXiv preprint arXiv:2412\.04403 \.
- Bisk et al\. \(\(2020\)\)Bisk, Y\., Zellers, R\., Gao, J\., Choi, Y\. et al\.\(2020\)\.Piqa: Reasoning about physical commonsense in natural language\.In Proceedings of the aaai conference on artificial intelligence \( 34, 7432–7439\)\.
- Busbridge et al\. \(\(2025\)\)Busbridge, D\., Shidani, A\., Weers, F\., Ramapuram, J\., Littwin, E\. & Webb, R\.\(2025\)\.Distillation scaling laws\.arXiv preprint arXiv:2502\.08606 \.
- Choe et al\. \(\(2019\)\)Choe, D\., Al\-Rfou, R\., Guo, M\., Lee, H\. & Constant, N\.\(2019\)\.Bridging the gap for tokenizer\-free language models\.arXiv e\-prints arXiv–1908\.
- J\.H\. Clark et al\. \(\(2022\)\)Clark, J\.H\., Garrette, D\., Turc, I\. & Wieting, J\.\(2022\)\.Canine: Pre\-training an efficient tokenization\-free encoder for language representation\.Transactions of the Association for Computational Linguistics 10 73–91\.
- P\. Clark et al\. \(\(2018\)\)Clark, P\., Cowhey, I\., Etzioni, O\., Khot, T\., Sabharwal, A\., Schoenick, C\. & Tafjord, O\.\(2018\)\.Think you have solved question answering? try arc, the ai2 reasoning challenge\.arXiv preprint arXiv:1803\.05457 \.
- Deng et al\. \(\(2009\)\)Deng, J\., Dong, W\., Socher, R\., Li, L\-J\., Li, K\. & Fei\-Fei, L\.\(2009\)\.Imagenet: A large\-scale hierarchical image database\.In 2009 ieee conference on computer vision and pattern recognition \(p\. 248\-255\)\.DOI: 10\.1109/CVPR\.2009\.5206848
- Dettmers & Zettlemoyer \(\(2023\)\)Dettmers, T\. & Zettlemoyer, L\.\(2023\)\.The case for 4\-bit precision: k\-bit inference scaling laws\.In International conference on machine learning \( 7750–7774\)\.
- El Boukkouri et al\. \(\(2020\)\)El Boukkouri, H\., Ferret, O\., Lavergne, T\., Noji, H\., Zweigenbaum, P\. & Tsujii, J\.\(2020\)\.Characterbert: Reconciling elmo and bert for word\-level open\-vocabulary representations from characters\.In Proceedings of the 28th international conference on computational linguistics \( 6903–6915\)\.
- Fang et al\. \(\(2025\)\)Fang, L\., Wang, Y\., Liu, Z\., Zhang, C\., Jegelka, S\., Gao, J\.Wang, Y\.\(2025\)\.What is wrong with perplexity for long\-context language modeling?In International conference on learning representations \( 2025, 94541–94563\)\.
- Gadre et al\. \(\(2024\)\)Gadre, S\.Y\., Smyrnis, G\., Shankar, V\., Gururangan, S\., Wortsman, M\., Shao, R\.others\(2024\)\.Language models scale reliably with over\-training and on downstream tasks\.arXiv preprint arXiv:2403\.08540 \.
- Gao et al\. \(\(2024\)\)Gao, L\., Tow, J\., Abbasi, B\., Biderman, S\., Black, S\., DiPofi, A\.Zou, A\.\(2024\)\.The language model evaluation harness\.: Zenodo\.[https://zenodo\.org/records/12608602](https://zenodo.org/records/12608602)DOI: 10\.5281/zenodo\.12608602
- Goyal et al\. \(\(2022\)\)Goyal, N\., Gao, C\., Chaudhary, V\., Chen, P\-J\., Wenzek, G\., Ju, D\.Fan, A\.\(2022\)\.The flores\-101 evaluation benchmark for low\-resource and multilingual machine translation\.Transactions of the Association for Computational Linguistics 10 522–538\.
- Grattafiori et al\. \(\(2024\)\)Grattafiori, A\., Dubey, A\., Jauhri, A\., Pandey, A\., Kadian, A\., Al\-Dahle, A\.others\(2024\)\.The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783 \.
- Hayase et al\. \(\(2025\)\)Hayase, J\., Liu, A\., Smith, N\.A\. & Oh, S\.\(2025\)\.Sampling from your language model one byte at a time\.arXiv preprint arXiv:2506\.14123 \.
- He et al\. \(\(2016\)\)He, K\., Zhang, X\., Ren, S\. & Sun, J\.\(2016\)\.Deep residual learning for image recognition\.In Proceedings of the ieee conference on computer vision and pattern recognition \( 770–778\)\.
- Henighan et al\. \(\(2020\)\)Henighan, T\., Kaplan, J\., Katz, M\., Chen, M\., Hesse, C\., Jackson, J\.others\(2020\)\.Scaling laws for autoregressive generative modeling\.arXiv preprint arXiv:2010\.14701 \.
- Hestness et al\. \(\(2017\)\)Hestness, J\., Narang, S\., Ardalani, N\., Diamos, G\., Jun, H\., Kianinejad, H\.Zhou, Y\.\(2017\)\.Deep learning scaling is predictable, empirically\.arXiv preprint arXiv:1712\.00409 \.
- Hinton et al\. \(\(2015\)\)Hinton, G\., Vinyals, O\. & Dean, J\.\(2015\)\.Distilling the knowledge in a neural network\.arXiv preprint arXiv:1503\.02531 \.
- Hoffmann et al\. \(\(2022\)\)Hoffmann, J\., Borgeaud, S\., Mensch, A\., Buchatskaya, E\., Cai, T\., Rutherford, E\.others\(2022\)\.Training compute\-optimal large language models\.arXiv preprint arXiv:2203\.15556 10 \.
- Hwang et al\. \(\(2026\)\)Hwang, S\., Wang, B\. & Gu, A\.\(2026\)\.Dynamic chunking for end\-to\-end hierarchical sequence modeling\.In International conference on learning representations \( 2026, 149273–149313\)\.
- Isik et al\. \(\(2025\)\)Isik, B\., Ponomareva, N\., Hazimeh, H\., Paparas, D\., Vassilvitskii, S\. & Koyejo, S\.\(2025\)\.Scaling laws for downstream task performance in machine translation\.In International conference on learning representations \( 2025, 88769–88790\)\.
- Kamath et al\. \(\(2025\)\)Kamath, G\.T\.A\., Ferret, J\., Pathak, S\., Vieillard, N\., Merhej, R\., Perrin, S\.Hussenot, L\.\(2025\)\.Gemma 3 technical report\.ArXiv abs/2503\.19786 \.[https://api\.semanticscholar\.org/CorpusID:277313563](https://api.semanticscholar.org/CorpusID:277313563)
- Kaplan et al\. \(\(2020\)\)Kaplan, J\., McCandlish, S\., Henighan, T\., Brown, T\.B\., Chess, B\., Child, R\.Amodei, D\.\(2020\)\.Scaling laws for neural language models\.arXiv preprint arXiv:2001\.08361 \.
- Kosson et al\. \(\(2025\)\)Kosson, A\., Welborn, J\., Liu, Y\., Jaggi, M\. & Chen, X\.\(2025\)\.Weight decay may matter more than mup for learning rate transfer in practice\.arXiv preprint arXiv:2510\.19093 \.
- Kwiatkowski et al\. \(\(2019\)\)Kwiatkowski, T\., Palomaki, J\., Redfield, O\., Collins, M\., Parikh, A\., Alberti, C\.Petrov, S\.\(2019\)\.Natural questions: A benchmark for question answering research\.Transactions of the Association for Computational Linguistics 7 452–466\.[https://aclanthology\.org/Q19\-1026/](https://aclanthology.org/Q19-1026/)
- Lee et al\. \(\(2026\)\)Lee, C\., Yan, J\.N\., Liang, C\., Shi, J\., Zhang, Y\., Liu, J\.others\(2026\)\.The efficiency gap in byte modeling\.arXiv preprint arXiv:2605\.12928 \.
- Limisiewicz et al\. \(\(2026\)\)Limisiewicz, T\., Pagnoni, A\., Iyer, S\., Lewis, M\., Mehta, S\., Liu, A\.Zettlemoyer, L\.\(2026\)\.Compute optimal tokenization\.arXiv preprint arXiv:2605\.01188 \.
- Liu et al\. \(\(2023\)\)Liu, H\., Xie, S\.M\., Li, Z\. & Ma, T\.\(2023\)\.Same pre\-training loss, better downstream: Implicit bias matters for language models\.In International conference on machine learning \( 22188–22214\)\.
- Loshchilov & Hutter \(\(2017\)\)Loshchilov, I\. & Hutter, F\.\(2017\)\.Decoupled weight decay regularization\.arXiv preprint arXiv:1711\.05101 \.
- Lourie et al\. \(\(2025\)\)Lourie, N\., Hu, M\.Y\. & Cho, K\.\(2025\)\.Scaling laws are unreliable for downstream tasks: A reality check\.arXiv preprint arXiv:2507\.00885 \.
- Magnusson et al\. \(\(2025\)\)Magnusson, I\., Tai, N\., Bogin, B\., Heineman, D\., Hwang, J\.D\., Soldaini, L\.others\(2025\)\.Datadecide: How to predict best pretraining data with small experiments\.arXiv preprint arXiv:2504\.11393 \.
- Meta AI \(\(2024\)\)Meta AI\.\(2024\)\.Llama 3\.2: Revolutionizing edge ai and vision\.[https://ai\.meta\.com/blog/llama\-3\-2\-connect\-2024\-vision\-edge\-mobile\-devices/](https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/)Accessed: 2024\-09\-25
- Minixhofer et al\. \(\(2025\)\)Minixhofer, B\., Murray, T\., Limisiewicz, T\., Korhonen, A\., Zettlemoyer, L\., Smith, N\.A\.Hofmann, V\.\(2025\)\.Bolmo: Byteifying the next generation of language models\.arXiv preprint arXiv:2512\.15586 \.
- Minixhofer et al\. \(\(2026\)\)Minixhofer, B\., Vulić, I\. & Ponti, E\.M\.\(2026\)\.Universal cross\-tokenizer distillation via approximate likelihood matching\.Advances in Neural Information Processing Systems 38 79297–79326\.
- Nawrot et al\. \(\(2023\)\)Nawrot, P\., Chorowski, J\., Lancucki, A\. & Ponti, E\.M\.\(2023\)\.Efficient transformers with dynamic token pooling\.In A\. Rogers, J\. Boyd\-Graber & N\. Okazaki \(Eds\.\), Proceedings of the 61st annual meeting of the association for computational linguistics \(volume 1: Long papers\) \( 6403–6417\)\.Toronto, Canada: Association for Computational Linguistics\.[https://aclanthology\.org/2023\.acl\-long\.353/](https://aclanthology.org/2023.acl-long.353/)DOI: 10\.18653/v1/2023\.acl\-long\.353
- Owen \(\(2024\)\)Owen, D\.\(2024\)\.How predictable is language model benchmark performance?arXiv preprint arXiv:2401\.04757 \.
- Pagnoni et al\. \(\(2025\)\)Pagnoni, A\., Pasunuru, R\., Rodriguez, P\., Nguyen, J\., Muller, B\., Li, M\.others\(2025\)\.Byte latent transformer: Patches scale better than tokens\.In Proceedings of the 63rd annual meeting of the association for computational linguistics \(volume 1: Long papers\) \( 9238–9258\)\.
- Papineni et al\. \(\(2002\)\)Papineni, K\., Roukos, S\., Ward, T\. & Zhu, W\-J\.\(2002\)\.Bleu: a method for automatic evaluation of machine translation\.In Proceedings of the 40th annual meeting on association for computational linguistics \(p\. 311–318\)\.USA: Association for Computational Linguistics\.[https://doi\.org/10\.3115/1073083\.1073135](https://doi.org/10.3115/1073083.1073135)DOI: 10\.3115/1073083\.1073135
- Peng et al\. \(\(2025\)\)Peng, H\., Lv, X\., Bai, Y\., Yao, Z\., Zhang, J\., Hou, L\. & Li, J\.\(2025\)\.Pre\-training distillation for large language models: A design space exploration\.In Proceedings of the 63rd annual meeting of the association for computational linguistics \(volume 1: Long papers\) \( 3603–3618\)\.
- Phan et al\. \(\(2024\)\)Phan, B\., Amos, B\., Gat, I\., Havasi, M\., Muckley, M\. & Ullrich, K\.\(2024\)\.Exact byte\-level probabilities from tokenized language models for fim\-tasks and model ensembles\.arXiv preprint arXiv:2410\.09303 \.
- Pope et al\. \(\(2023\)\)Pope, R\., Douglas, S\., Chowdhery, A\., Devlin, J\., Bradbury, J\., Heek, J\.Dean, J\.\(2023\)\.Efficiently scaling transformer inference\.Proceedings of machine learning and systems 5 606–624\.
- Sardana et al\. \(\(2023\)\)Sardana, N\., Portes, J\., Doubov, S\. & Frankle, J\.\(2023\)\.Beyond chinchilla\-optimal: Accounting for inference in language model scaling laws\.arXiv preprint arXiv:2401\.00448 \.
- Schaeffer et al\. \(\(2023\)\)Schaeffer, R\., Miranda, B\. & Koyejo, S\.\(2023\)\.Are emergent abilities of large language models a mirage?Advances in neural information processing systems 36 55565–55581\.
- Shazeer \(\(2020\)\)Shazeer, N\.\(2020\)\.Glu variants improve transformer\.arXiv preprint arXiv:2002\.05202 \.
- Team et al\. \(\(2024\)\)Team, G\., Mesnard, T\., Hardin, C\., Dadashi, R\., Bhupatiraju, S\., Pathak, S\.others\(2024\)\.Gemma: Open models based on gemini research and technology\.arXiv preprint arXiv:2403\.08295 \.
- Touvron et al\. \(\(2023\)\)Touvron, H\., Martin, L\., Stone, K\., Albert, P\., Almahairi, A\., Babaei, Y\.others\(2023\)\.Llama 2: Open foundation and fine\-tuned chat models\.arXiv preprint arXiv:2307\.09288 \.
- Vaswani et al\. \(\(2017\)\)Vaswani, A\., Shazeer, N\., Parmar, N\., Uszkoreit, J\., Jones, L\., Gomez, A\.N\.Polosukhin, I\.\(2017\)\.Attention is all you need\.Advances in neural information processing systems 30 \.
- Veličković et al\. \(\(2026\)\)Veličković, P\., Barbero, F\., Perivolaropoulos, C\., Osindero, S\. & Pascanu, R\.\(2026\)\.Perplexity cannot always tell right from wrong\.arXiv preprint arXiv:2601\.22950 \.
- Xue et al\. \(\(2022\)\)Xue, L\., Barua, A\., Constant, N\., Al\-Rfou, R\., Narang, S\., Kale, M\.Raffel, C\.\(2022\)\.Byt5: Towards a token\-free future with pre\-trained byte\-to\-byte models\.Transactions of the Association for Computational Linguistics 10 291–306\.
- Yu et al\. \(\(2023\)\)Yu, L\., Simig, D\., Flaherty, C\., Aghajanyan, A\., Zettlemoyer, L\. & Lewis, M\.\(2023\)\.Megabyte: Predicting million\-byte sequences with multiscale transformers\.Advances in Neural Information Processing Systems 36 78808–78823\.
- Zellers et al\. \(\(2019\)\)Zellers, R\., Holtzman, A\., Bisk, Y\., Farhadi, A\. & Choi, Y\.\(2019\)\.Hellaswag: Can a machine really finish your sentence?In Proceedings of the 57th annual meeting of the association for computational linguistics \( 4791–4800\)\.
- Zhang et al\. \(\(2026\)\)Zhang, T\.T\., Shah, A\., Zhang, Y\., Zhang, V\., Matni, N\. & Simchowitz, M\.\(2026\)\.Double preconditioning \(dopr\): Optimization for test\-time performance, not validation loss\.arXiv preprint arXiv:2606\.06418 \.
- Zheng et al\. \(\(2025\)\)Zheng, L\., Zhao, X\., Wang, G\., Wu, C\., Dong, D\., Wang, A\.Kong, L\.\(2025\)\.Evabyte: Efficient byte\-level language models at scale\.[https://hkunlp\.github\.io/blog/2025/evabyte](https://hkunlp.github.io/blog/2025/evabyte)