在计算与数据约束下重写合成数据的缩放定律

# Rewriting Scaling Laws for Synthetic Data with Compute and Data Constraints
Source: [https://arxiv.org/html/2610.31867](https://arxiv.org/html/2610.31867)
Nicholas RobertsAhmed Ahmed11footnotemark:1Suhas KothaPercy LiangPrinceton UniversityStanford UniversityStanford UniversityStanford University Yu WangAlex RendaNitya ManiJane StreetJane StreetJane Street††thanks:Equal contribution, order determined by coin flip\. Work done during an internship at Jane Street\. Correspondence to*nick11roberts@princeton\.edu*and*ahmedah@cs\.stanford\.edu*

###### Abstract

Synthetic rewrites, where a language model rewrites a natural document e\.g\. into a target style, are now a standard ingredient of pretraining corpora\. However, critical aspects of their use remain poorly understood, such as how much compute should be spent rewriting training data and whether a model trained on rewrites can outperform the rewriter\. We study these two questions by proposing a rewriting scaling law that predicts loss from model size, training tokens, the fraction of training tokens that are rewrites, and the number of epochs over the natural data, charging the compute spent generating rewrites\. We fit this scaling law in two settings\. In the compute\-limited setting, natural data is plentiful and rewrites come from a third\-party rewriter trained with more compute than our largest run\. We find that the estimated optimal rewrite fraction increases with compute, from under10%10\\%at3×10183\\times 10^\{18\}FLOPs to32%32\\%at102210^\{22\}FLOPs\. Our scaling forecast extrapolates33×33\\timesin FLOPs to train a 22B\-parameter model, whose final loss is within1\.1%1\.1\\%of our prediction\. This model achieves a better BPB than its rewriter on several web domains at one\-tenth the rewriter’s training compute\. In the data\-limited setting, we assume we only have a fixed seed set of natural data and that our rewriter is a model of the same size as the target, pretrained on that seed set\. Again, we find the optimal rewrite fraction increases with the compute budget\. We also demonstrate that a model trained on rewrites of its own pretraining data, generated by a rewriter of the same size, beats the best run on natural data alone at every model size up to 3B\. Our results show that synthetic rewriting can be studied through the scaling law lens: that optimal methodological choices are predictable and scaling does not necessarily break down beyond the rewriter’s capabilities\.

## 1Introduction

Large language models have a host of remarkable capabilities, many of which improve smoothly during pretraining and grow as model parameters and training tokens increase, in a manner that is predictable via scaling laws\([Kaplan et al\., 2020](https://arxiv.org/html/2610.31867#bib.bib9);[Hoffmann et al\., 2022](https://arxiv.org/html/2610.31867#bib.bib3)\)\. Early language models were pretrained using corpora of text that were primarily comprised ofnatural data, text generated either manually by a human or generated programmatically\. Common sources of such tokens included web pages collected from internet crawls, GitHub repositories, and academic publications\([Gao et al\., 2020](https://arxiv.org/html/2610.31867#bib.bib1)\)\. However, the amount of compute available for training has grown much faster than the amount of high quality, natural pretraining data\([Villalobos et al\., 2022](https://arxiv.org/html/2610.31867#bib.bib10);[Kim et al\., 2025](https://arxiv.org/html/2610.31867#bib.bib7)\)\.

These constraints on the availability of large\-scale natural corpora of text have motivated usingsynthetic rewrites, documents generated by prompting a language model with a natural document and an instruction to write a different version of the document\. Common instructions for synthetic rewrites include rewriting in the manner of a Wikipedia article\([Maini et al\., 2024](https://arxiv.org/html/2610.31867#bib.bib5)\), converting math web pages into synthetic math textbook\-style exposition\([Mahabad et al\., 2025](https://arxiv.org/html/2610.31867#bib.bib13)\), synthesizing or summarizing lengthy source material\([Maini et al\., 2025](https://arxiv.org/html/2610.31867#bib.bib17)\), and transforming a lower quality source into a higher quality output using the model’s knowledge\([Su et al\., 2024](https://arxiv.org/html/2610.31867#bib.bib12);[Nguyen et al\., 2025](https://arxiv.org/html/2610.31867#bib.bib14)\)\.

Synthetic rewrites of natural documents have become a standard ingredient in many modern pretraining corpora\([Maini et al\., 2025](https://arxiv.org/html/2610.31867#bib.bib17);[Su et al\., 2024](https://arxiv.org/html/2610.31867#bib.bib12);[Nguyen et al\., 2025](https://arxiv.org/html/2610.31867#bib.bib14);[Niklaus et al\., 2026](https://arxiv.org/html/2610.31867#bib.bib21)\)\. However, their use does have several drawbacks\. First, synthetic rewrites are costly to generate: for a given model size, rewriting a given document involves a significant fraction of the compute of training on that document\. Second, synthetic rewrites are necessarily generated by a model with a given amount of complexity and capability, and it is unclear the extent to which that bounds the capabilities of a model trained on these synthetic rewrites\.

In this paper we develop a*rewriting scaling law*that takes as input the*rewrite fraction*, the fraction of training tokens that are synthetic rewrites, and predicts the expected loss of a model trained at that rewrite fraction\. When extrapolating this scaling law to predict the loss\-minimizing fraction at a given compute budget, we account for the*cost\-adjusted training FLOPs*, the amount of FLOPs available for training after accounting for the cost of generating synthetic rewrites\.

With this scaling law, we investigate two specific research questions:

1. RQ1*Does our scaling law accurately predict the optimal rewrite fraction and its achieved loss?*
2. RQ2*Should we include rewrites when targeting capabilities that surpass those of the rewriter?*

Figure 1:*\(Left; compute\-limited\) Effective rewriting recipes are predictable via scaling laws:*best possible Paloma BPB \(lower is better; allowing third\-party rewrites\) vs FLOPs \(accounting for rewrite compute costs\) as compared to using natural data only\. Unfilled blue points are imputed via linear interpolations of the loss of the closest rewrite fractions that we evaluate, whereas solid points represent observed training runs\.*\(Right; data\-limited\) The trained model surpasses its rewriter:*final Paloma BPB vs FLOP for a33B target model trained using only natural data or synthetic rewrites coming from a fixed55B\-token natural data corpus\. Using natural data only, which necessitates epoching, results in worse loss after33epochs\. The best proportion of rewrites from a bootstrapped rewriter improves over epoching on the natural seed set alone at every budget\.We study these questions in distinct settings characteristic of language model training paradigms in which the primary constraint is either compute\([Hoffmann et al\., 2022](https://arxiv.org/html/2610.31867#bib.bib3)\)or data\([Kim et al\., 2025](https://arxiv.org/html/2610.31867#bib.bib7)\)\.

In the first setting, which we termcompute\-limited, we focus on a scenario where natural data is unbounded\. We assume an*external rewriter*, a rewriting model that is trained with more total compute than our largest training run\. This setting exemplifies the small\-scale scientific workflow\.

In the second setting, which we term thedata\-limitedsetting, we focus on a scenario in which natural data is bounded\. We also assume that we only have access to a*bootstrapped rewriter*which is trained on a subset of the available data and for fewer FLOPs than a given candidate training run\. This setting exemplifies the frontier\-scale workflow in which the objective is to train a model more capable than any previous model\.

We find the following results:

- •*Our scaling law accurately predicts the optimal rewrite fraction and achieved loss at large budgets\.*In the compute\-limited setting, we fit our rewriting scaling law at low budgets and extrapolate to a set of102210^\{22\}\-FLOP runs\. We find that the predicted optimal rewrite fraction results in the lowest held\-out loss among tested models at the102210^\{22\}\-FLOP scale, achieving loss within1\.1%1\.1\\%of the predicted loss\. In the data\-limited setting, a scaling law fit to low budgets predicts a rewrite fraction at large held\-out budgets whose loss is within1\.14%1\.14\\%of that of the best observed rewrite fraction\. In both settings, the optimal rewrite fractions increase with scale and are task\-specific, differing between math, code, question\-answering, and general web text\.
- •*Models can achieve and exceed their rewriter model’s performance\.*In the compute\-limited setting, pretraining a model for102210^\{22\}FLOPs—where32%32\\%of its training data is rewritten—outperforms the external rewriter \(SmolLM2\-1\.7B\-Instruct\) on 10 out of 16 general web text sources \(using roughly one\-tenth of the rewriter’s training compute\)\. In the data\-limited setting, where the bootstrapped rewriter is always trained with fewer FLOPs than the target model, models trained on a mixture containing rewrites consistently outperform the rewriter\.

Our results give evidence that synthetic rewriting can be studied through the scaling law lens, showing both that optimal methodological choices are predictable and that scaling does not necessarily break down beyond the capabilities of the rewriter\.

## 2Methodology

In this section we first present the rewriting scaling law we use to predict the expected loss of a model trained with a given amount of synthetic rewrites in the training set\. We then describe how we practically fit and use this scaling law to predict optimal rewrite fractions, including how we account for the cost of compute to generate the synthetic rewrites\.

### 2\.1A Rewriting Scaling Law

In a given experiment in both the compute\- and data\-limited settings, we pretrain anNN\-parameter*target model*on a*corpus*withDDtotal tokens composed of a combination of*natural tokens*and*synthetic rewrites*thereof\. The set of natural tokens in a given corpus is drawn from a*seed set*ofTTnatural tokens, potentially with replacement \(epoching\)\. The synthetic rewrite tokens are rewrites of the natural tokens selected for the corpus, without epoching\. We define the*rewrite fraction*R∈\[0,1\]R\\in\[0,1\]for a given corpus as the fraction of tokens in the corpus that are synthetic rewrites\.

Our*rewriting scaling law*predicts that anNN\-parameter target model trained on aDD\-token corpus with seed set sizeTTand rewrite fractionRRhas held\-out lossLLas follows:

L⁡\(N,D,R,T\)=ANα\+BDβ⏟Chinchilla terms\+E⁡\(R\)⏟rewrite\-fraction\-dependentirreducible loss\+F⁡\(R,D,T\)⏟repetition penalty\.L\(N,D,R,T\)=\\underbrace\{\\frac\{A\}\{N^\{\\alpha\}\}\+\\frac\{B\}\{D^\{\\beta\}\}\}\_\{\\text\{Chinchilla terms\}\}\+\\underbrace\{E\(R\)\}\_\{\\begin\{subarray\}\{c\}\\text\{rewrite\-fraction\-dependent\}\\\\ \\text\{irreducible loss\}\\end\{subarray\}\}\\hskip 6\.00006pt\+\\hskip 13\.99995pt\\underbrace\{F\(R,D,T\)\}\_\{\\mathclap\{\\text\{repetition penalty\}\}\}\.\(1\)
The termsA/Nα\+B/DβA/N^\{\\alpha\}\+B/D^\{\\beta\}are the same as in the classic Chinchilla\-style scaling law of[Hoffmann et al\. \(2022\)](https://arxiv.org/html/2610.31867#bib.bib3), which model the loss contributions of using a certain parameter count and dataset size\.

The termE⁡\(R\)E\(R\)represents the irreducible loss as a function of the rewrite fractionRR\. Appendix[L](https://arxiv.org/html/2610.31867#A12)defines this as an approximation of the KL divergence between the training and evaluation distributions\. The specific functional form for this*rewrite\-fraction\-dependent irreducible loss*is given asE⁡\(R\)=E0−λn​log⁡\(1−\(1−ρn\)​R\)−λs​log⁡\(ρs\+\(1−ρs\)​R\)E\(R\)=E\_\{0\}\-\\lambda\_\{n\}\\log\\\!\\left\(1\-\(1\-\\rho\_\{n\}\)R\\right\)\-\\lambda\_\{s\}\\log\\\!\\left\(\\rho\_\{s\}\+\(1\-\\rho\_\{s\}\)R\\right\), whereλn\\lambda\_\{n\},λs\\lambda\_\{s\},ρn\\rho\_\{n\},ρs\\rho\_\{s\}, andE0E\_\{0\}are all fitted parameters\. Additional details can be found in Appendix[L](https://arxiv.org/html/2610.31867#A12)\.

The termF⁡\(R,D,T\)F\(R,D,T\)represents the cost of epoching natural data multiple times, and is drawn from Equation \(7\) in[Lovelace et al\. \(2026\)](https://arxiv.org/html/2610.31867#bib.bib2)\. This*repetition penalty*term is defined asF⁡\(R,D,T\)=P​\(1−R\)s​\(\(1−R\)​D/T−1\)\+,F\(R,D,T\)=P\\,\(1\-R\)^\{s\}\\,\\bigl\(\(1\-R\)D/T\-1\\bigr\)\_\{\+\},wherePPandssare fitted parameters\. The repetition penalty is only nonzero when the natural tokens are epoched, so it appears only in the data\-limited setting\.

### 2\.2Predicting Optimal Rewrite Fractions at a Compute Budget

Once we fit our rewriting scaling law, our goal is to choose the rewrite fraction that minimizes loss for a fixed compute budget\. Since synthetic rewrites are not free, the compute budget must account for both the cost of training the target model and the cost of generating synthetic rewrites\.

We approximate the FLOPs for training a target model withNNparameters on a corpus of sizeDDwith rewrite fractionRRusing synthetic data generated by a rewriter withNrN\_\{r\}parameters with:

Ctotal:=6​N​D⏟target\-model training\+2​Nr​D​R⏟source prefill\+2​Nr​D​R⏟rewrite decoding\.C\_\{\\mathrm\{total\}\}:=\\underbrace\{6ND\}\_\{\\text\{target\-model training\}\}\+\\underbrace\{2N\_\{r\}DR\}\_\{\\text\{source prefill\}\}\+\\underbrace\{2N\_\{r\}DR\}\_\{\\text\{rewrite decoding\}\}\.\(2\)
ThisCtotalC\_\{\\mathrm\{total\}\}approximation makes several assumptions\. First, we use the standard assumptions that forward and backward passes are respectively2​N2Nand4​N4NFLOPs per token\([Kaplan et al\., 2020](https://arxiv.org/html/2610.31867#bib.bib9)\), giving6​N​D6NDtarget\-model training FLOPs\. Second, we assume that the length of a synthetic rewrite is the same as the length of the input document, and that these lengths are large relative to other elements of the prompt; this gives2​Nr​D​R2N\_\{r\}DRFLOPs for forward passes \(prefill\) on the natural tokens used for rewriting, and an additional2​Nr​D​R2N\_\{r\}DRFLOPs for forward passes \(decode\) when generating the synthetic rewrites\.111For FLOPs to be a reasonable proxy for the cost of generating synthetic rewrites, we must also assume that generation is FLOP\-bound and not memory\-bandwidth\-bound; this often holds with sufficiently large batch sizes and short context lengths\([Pope et al\., 2023](https://arxiv.org/html/2610.31867#bib.bib23)\), which is tractable when generating synthetic data\.We also assume no amortization of the cost of generating rewrites across multiple different target model training runs\.

The*optimal rewrite fraction*R∗R^\{\*\}minimizes the true loss at a given compute budget\. We denote our estimate of it asR^∗\\widehat\{R\}^\{\*\}, which is the minimizer of our fitted scaling law:

R^∗=arg​minR∈\[0,1\]⁡minN,D​L​\(N,D,R,T\)s\.t\.6​N​D\+4​Nr​RD=Ctotal\.\\widehat\{R\}^\{\*\}=\\argmin\_\{R\\in\[0,1\]\}\\;\\min\_\{N,\\,D\}\\;L\(N,D,R,T\)\\quad\\text\{s\.t\.\}\\quad 6ND\+4N\_\{r\}RD=C\_\{\\mathrm\{total\}\}\.\(3\)

## 3Experiments

We study our research questions in two different settings\.

First in thecompute\-limitedsetting, we assume a fixed FLOP budget, more natural data in our seed set than our largest target model requires \(T≫DT\\gg D\), and an*external rewriter*trained for more FLOPs than any member of our scaling ladder on a potentially different seed set of data\. This setting is motivated by the small\-scale scientific workflow\.

Next in thedata\-limitedsetting, we assume a fixed FLOP budget, a limited seed set \(T≪DT\\ll D\), and for each target model a corresponding*bootstrapped rewriter*trained for fewer FLOPs than that target model on the same seed set\. This setting is motivated by the frontier\-scale workflow in which the objective is to train a model more capable than any previous model\.

Consistent across both settings we find:\(RQ1\)scaling law based extrapolations well predict the optimal fraction of rewrites and the corresponding achieved loss; and\(RQ2\), in both the compute\-limited and data\-limited settings, the scaling law predicts and we empirically confirm that it is optimal to include rewrites \(i\.e\.,R^∗\>0\\widehat\{R\}^\{\*\}\>0\) for target models that surpass capabilities of the rewriter\.

### 3\.1Experimental Setup

##### Architecture and training\.

All runs use Qwen3 dense models\([Yang et al\., 2025a](https://arxiv.org/html/2610.31867#bib.bib32)\)with the Llama 3 tokenizer\([Grattafiori et al\., 2024](https://arxiv.org/html/2610.31867#bib.bib26)\), trained with Adam and Hyperball\([Wen et al\., 2026](https://arxiv.org/html/2610.31867#bib.bib22)\)under a linear warm\-up and linear\-decay schedule\. Appendix[B](https://arxiv.org/html/2610.31867#A2)gives the width\-to\-depth rule and Appendix[C](https://arxiv.org/html/2610.31867#A3)the hyperparameter tuning and transfer recipe\.

##### Data\.

In the compute\-limited setting, we train on FineWeb\-Edu\([Penedo et al\., 2024](https://arxiv.org/html/2610.31867#bib.bib19)\)with FinePhrase rewrites\([Niklaus et al\., 2026](https://arxiv.org/html/2610.31867#bib.bib21)\), which are FineWeb\-Edu rewrites from SmolLM2\-1\.7B\([Ben Allal et al\., 2025](https://arxiv.org/html/2610.31867#bib.bib16)\)\. In the data\-limited setting, we train on a55B\-token seed set from FineWeb\-Edu and rewrites of that seed set using a model that we train ourselves, also on the seed set\. We retain only documents from web\-archive \(WARC\) files dated before 2023 to reduce LM\-generated text in the natural corpus \(Appendix[A](https://arxiv.org/html/2610.31867#A1)\)\. For mixtures with rewrite fraction0<R<10<R<1we interleave natural and rewritten documents in proportion toRRthroughout training \(Appendix[I](https://arxiv.org/html/2610.31867#A9)\)\.

##### Rewriting protocol\.

Following FinePhrase\([Niklaus et al\., 2026](https://arxiv.org/html/2610.31867#bib.bib21)\), every rewrite converts one natural document into a math exposition, an FAQ, a table, or a tutorial\.

##### Evaluation\.

Our evaluations include Paloma\([Magnusson et al\., 2024](https://arxiv.org/html/2610.31867#bib.bib18)\)and the set of aggregations in the Olmo 3 base easy evaluation suite\([Ettinger and others, 2025](https://arxiv.org/html/2610.31867#bib.bib48)\): QA \(ARC, HellaSwag, and MMLU\), Math \(Minerva MATH, with GSM8K\-Platinum and TheoremQA added\), and Code \(HumanEval, MBPP, and multilingual MBPP\)\([Clark et al\., 2018](https://arxiv.org/html/2610.31867#bib.bib33);[Zellers et al\., 2019](https://arxiv.org/html/2610.31867#bib.bib34);[Hendrycks et al\., 2021a](https://arxiv.org/html/2610.31867#bib.bib35);[Cobbe et al\., 2021](https://arxiv.org/html/2610.31867#bib.bib36);[Vendrow et al\., 2025](https://arxiv.org/html/2610.31867#bib.bib37);[Chen et al\., 2023](https://arxiv.org/html/2610.31867#bib.bib38);[Hendrycks et al\., 2021b](https://arxiv.org/html/2610.31867#bib.bib39);[Lewkowycz et al\., 2022](https://arxiv.org/html/2610.31867#bib.bib40);[Chen et al\., 2021](https://arxiv.org/html/2610.31867#bib.bib41);[Austin et al\., 2021](https://arxiv.org/html/2610.31867#bib.bib42);[Cassano et al\., 2023](https://arxiv.org/html/2610.31867#bib.bib43)\)\. We list the suite membership and aggregation for all evaluations in Appendix[D](https://arxiv.org/html/2610.31867#A4)\.

On all evaluations, we report bits\-per\-byte \(BPB\) at the trained40964096\-token context\. For Paloma, we report the macro\-averaged BPB, BPB averaged within each Paloma domain then averaged across domains \(equal\-weighting all domains\)\. For the Olmo 3 suites we report gold\-answer BPB\.

### 3\.2Compute\-Limited Experiments

In thecompute\-limitedsetting, we assume a fixed FLOP budget, more natural data in our seed set than our largest target model requires \(T≫DT\\gg D\), and an*external rewriter*trained for more FLOPs than any member of our scaling ladder on a potentially different seed set of data\.

##### Scaling ladder methodology\.

We trainIsoFLOP ladders—sets of models with varying size and token counts, each trained at one of a range of fixed training budgets—on FineWeb\-Edu at eleven rewrite fractionsR∈\{0,0\.1,…,1\.0\}R\\in\\\{0,0\.1,\\ldots,1\.0\\\}\. SmolLM2\-1\.7B\([Ben Allal et al\., 2025](https://arxiv.org/html/2610.31867#bib.bib16)\)generates rewrites, which we draw from FinePhrase\([Niklaus et al\., 2026](https://arxiv.org/html/2610.31867#bib.bib21)\)\. We train target models with training budgets spanning3×10183\\times 10^\{18\}to9×10199\\times 10^\{19\}FLOPs at all eleven rewrite fractions, and natural\-data\-only \(R=0R=0\) ladders at1\.8×10201\.8\\times 10^\{20\}and3×10203\\times 10^\{20\}FLOPs\. We fit our scaling laws with these target models against performance on each of the QA, Math, Code, and Paloma evals\. We validate our rewriting scaling law’s predictions with validation runs at102110^\{21\}FLOPs across all eleven rewrite fractions and a set of102210^\{22\}runs including a run at the predictedR^∗\\widehat\{R\}^\{\*\}optimum \(and correspondingNNandDDoptima\) for the Paloma eval\.

##### Rewriting methodology\.

In the compute\-limited setting, we use the released FinePhrase SmolLM2\-1\.7B rewrites\. SmolLM2\-1\.7B was trained for roughly102310^\{23\}FLOPs\.

Figure 2:The estimated compute\-optimal rewrite fraction increases with compute on every evaluation\.The curves showR^∗\\widehat\{R\}^\{\*\}across evaluations and FLOP budgets\. Solid lines indicate that the compute budget is within the evaluated ladder, while dashed lines are extrapolations\. Points mark the fit optimum at each ladder budget; filled points land on a tested rewrite fraction and unfilled points fall between two\. Stars mark predicted break\-even with SmolLM2\-Instruct\. The vertical line marks the training FLOPs of SmolLM2\-Instruct\.Figure 3:The surface fit to budgets through3×10203\\times 10^\{20\}FLOPs forecasts the102210^\{22\}runs to within1\.4%1\.4\\%\.Each panel is one rewrite fractionRR\. Dots are runs and curves are the fit IsoFLOP slices, colored by training budget\. TheR=0R=0ladders reach3×10203\\times 10^\{20\}FLOPs and theR\>0R\>0ladders9×10199\\times 10^\{19\}FLOPs\. Open stars mark the fit optimum at each budget, filled blue stars mark the compute\-optimal runs at1\.8×10201\.8\\times 10^\{20\}and3×10203\\times 10^\{20\}FLOPs, and the dashed line extends the compute\-optimal forecast to102210^\{22\}FLOPs\. The orange star is the102210^\{22\}run, held out of the fit, labeled with its measured loss relative to the forecast\. TheR=0\.32R=0\.32run’s forecast error is−1\.1%\-1\.1\\%whether the surface is evaluated atR=0\.3R=0\.3or atR=0\.32R=0\.32\. Figure[14](https://arxiv.org/html/2610.31867#A14.F14)shows all eleven rewrite fractions\.
#### 3\.2\.1Compute\-Limited Experimental Results

##### The estimated optimal rewrite fraction increases in FLOP budget and varies between tasks \(RQ1\)\.

Figure[2](https://arxiv.org/html/2610.31867#S3.F2.fig1)shows the estimated optimal rewrite fractionR^∗\\widehat\{R\}^\{\*\}for each different eval as a function of FLOP budget\. For all evals,R^∗\\widehat\{R\}^\{\*\}monotonically increases with FLOP budget\. This is consistent with the observation that as the total compute budget \(and in particular the size of the target model\) grows, the cost of generating synthetic rewrites shrinks relative to the cost of training\.

We find thatR^∗\\widehat\{R\}^\{\*\}changes across target domains: on Code and Math,R^∗\\widehat\{R\}^\{\*\}rises more quickly than on QA and Paloma\. On Code and Math,R^∗\\widehat\{R\}^\{\*\}rises to1\.01\.0at FLOP budgets within our scaling ladder; on QA and Paloma, our scaling law fit extrapolates asymptotes of0\.770\.77and0\.320\.32respectively\. SmolLM2\-1\.7B was trained on FineMath, Stack\-Edu, and StarCoderData\([Ben Allal et al\., 2025](https://arxiv.org/html/2610.31867#bib.bib16);[Li et al\., 2023](https://arxiv.org/html/2610.31867#bib.bib31)\), which contain higher quality data for math and code capabilities than in the natural FineWeb\-Edu corpus\. The fact that math and code eventually require a rewrite fraction ofR=1R=1suggests that it is easier to learn math and code capabilities by proxy through SmolLM2\-1\.7B rewrites than from the natural FineWeb\-Edu data itself\.

The differences in loss between the predicted optimumR^∗\\widehat\{R\}^\{\*\}and suboptimalRRpoints are substantial: at102210^\{22\}FLOPs, training on rewrites alone \(R=1R=1\) instead of natural data alone \(R=0R=0\) reduces Math BPB from1\.331\.33to0\.800\.80and Code BPB from1\.041\.04to0\.530\.53\.

##### The estimated optimal rewrite fraction extrapolated from smaller budgets gives the lowest measured Paloma BPB at102210^\{22\}FLOPs, across all models we train \(RQ1\)\.

Our scaling law extrapolation predicts that Paloma BPB at a total budget of102210^\{22\}FLOPs will be minimized at rewrite fractionR=0\.32R=0\.32\(Figure[2](https://arxiv.org/html/2610.31867#S3.F2.fig1)\)\. We train four models at102210^\{22\}FLOPs that share\(N,D\)\(N,D\)and differ inR∈\{0,0\.32,0\.5,1\.0\}R\\in\\\{0,0\.32,0\.5,1\.0\\\}\. Of these,R=0\.32R=0\.32is indeed optimal \(Figure[3](https://arxiv.org/html/2610.31867#S3.F3)\)\.

##### The102210^\{22\}\-FLOP run at the predicted optimum reaches a lower Paloma BPB than its rewriter on1010of1616sources \(RQ2\)\.

SmolLM2\-1\.7B\-Instruct was trained on1111T tokens at≈1\.1×1023\{\\approx\}1\.1\\times 10^\{23\}FLOPs\. The102210^\{22\}\-FLOP run atR=0\.32R=0\.32reaches a lower Paloma BPB than its rewriter on1010of the1616Paloma sources and comes within0\.0120\.012BPB of it on the macro average,0\.9830\.983against0\.9720\.972\(Table[1](https://arxiv.org/html/2610.31867#S3.T1.fig1)\)\. The sources where the run does not break even with the rewriter are code \(which the rewriter’s training corpus covers more than our natural corpus FineWeb\-Edu\([Ben Allal et al\., 2025](https://arxiv.org/html/2610.31867#bib.bib16), Figure 2\)\), social media, and two web mixtures\. Table[1](https://arxiv.org/html/2610.31867#S3.T1.fig1)also compares against a102210^\{22\}run atR=0R=0which underperformsR=0\.32R=0\.32on most benchmarks, showing that including some rewrites helps surpass the rewriter at this compute budget\. The rewriter’s performance on a benchmark therefore does not represent a hard limit on the gains from its rewrites in the compute\-limited setting\.

Table 1:Our run at102210^\{22\}training FLOPs reaches a lower Paloma BPB than its rewriter on1010of1616sources\.Per\-source Paloma BPB for the natural\-only \(R=0R=0\) and predicted\-optimum \(R=0\.32R=0\.32\) runs at102210^\{22\}training FLOPs and for SmolLM2\-1\.7B\-Instruct, sorted by the predicted\-optimum run’s margin over the rewriter; the rule separates the sources where it is at or below the rewriter\. Bold marks the lowest BPB per row\. Appendix[N\.3](https://arxiv.org/html/2610.31867#A14.SS3)reports all four runs at102210^\{22\}training FLOPs and the SmolLM2 base model\.

### 3\.3Data\-Limited Experiments

In thedata\-limitedsetting, we assume a fixed FLOP budget, a limited natural seed set \(T≪DT\\ll D\), and for each target model a corresponding*bootstrapped rewriter*model trained for fewer FLOPs than that target model on the same fixed natural seed set\.

##### Scaling ladder methodology\.

Throughout, we assume we have access to a small seed set ofT=5T=5B tokens sampled from FineWeb\-Edu\. We train target models with550550M,837837M,11B, and33B parameters and measure each target model’s total compute budget in*epoch budgets*W=Ctotal/6​N​TW=C\_\{\\mathrm\{total\}\}/6NT, the compute required to train anNN\-parameter model, epoching the entire seed setWWtimes\. We train models with epoch budgetsW∈\{3,6,9,…,24\}W\\in\\\{3,6,9,\\ldots,24\\\}\. At each budget, we sweep rewrite fractionsR∈\{0,0\.13,0\.29,0\.47,0\.71,1\}R\\in\\\{0,0\.13,0\.29,0\.47,0\.71,1\\\}\.222The rewrite fractions are chosen so that the fraction of FLOPs spent generating and training on rewrites,5​R/\(3\+2​R\)5R/\(3\+2R\), runs from00to11in steps of0\.20\.2\(our rewriter has the same parameter count as our target model\)\.We report Paloma bits per byte at the final checkpoint \(best\-checkpoint results: Appendix[H\.2](https://arxiv.org/html/2610.31867#A8.SS2)\)\.

##### Rewriting methodology\.

For each target model, we pretrain a rewriter model with the same number of parameters on the seed set \(i\.e\.,W=1W=1; Appendix[E](https://arxiv.org/html/2610.31867#A5)\)\. Pretrained models typically do not follow instructions, so we then fine\-tune the rewriter on GLM\-5\.2\([Zhipu AI, 2026](https://arxiv.org/html/2610.31867#bib.bib45)\)rewrites of10001000seed\-set documents \(77M tokens\)\. We select the checkpoint with an LM judge and use it to rewrite the seed set under the same four FinePhrase prompts \(Appendix[F](https://arxiv.org/html/2610.31867#A6)\)\.333The small set of rewritten documents limits possible knowledge transfer from GLM\-5\.2 to the rewriter\. Appendix[G](https://arxiv.org/html/2610.31867#A7)compares this recipe with the SmolLM2 SFT\+DPO recipe\.For synthetic rewrites within each mixture, we sample the four rewriting styles in proportion to their token counts\.

Figure 4:TheR^∗\\widehat\{R\}^\{\*\}selected by the scaling law costs at most1\.14%1\.14\\%in loss at held\-out budgets\.The law \(Eq\.[1](https://arxiv.org/html/2610.31867#S2.E1)\) is fit per target size to budgets throughW=18W=18;W=21W=21andW=24W=24are held out \(shaded, open markers\)\.Top:measured final\-checkpoint Paloma macro BPB atR^∗\\widehat\{R\}^\{\*\}, as a percentage above the loss at the best observedRRon the same six\-value grid\.Bottom:R^∗\\widehat\{R\}^\{\*\}\(blue\), the best observedRR\(orange\), and the law’s optimum over continuousR∈\[0,1\]R\\in\[0,1\]\(dashed\)\.Figure 5:Final\-checkpoint Paloma macro BPB against the epoch budgetWWfor each target size\.Solid black:natural\-only epoching \(R=0R=0\) for the full budget ofWWepochs over the seed set\.Dotted black:the best natural\-only run at any budget, defined by the loss reached by stopping once further epochs raise it\.Blue:best mixture at eachWW, the run with the lowest final loss at that budget over all rewrite fractions, so it coincides with the natural\-only point wherever natural\-only is best\.
#### 3\.3\.1Data\-Limited Experimental Results

##### The best observed rewrite fraction increases in epoch budget and is predictable at unseen budgets \(RQ1\)\.

The best observed rewrite fraction—the testedRRresulting in the lowest final loss at each size and budget—increases with the epoch budgetWWat every target size \(Figure[4](https://arxiv.org/html/2610.31867#S3.F4)\)\. For the33B target, the best observedRRincreases from0\.130\.13atW=3W=3to0\.710\.71atW=24W=24\.

We predictR^∗\\widehat\{R\}^\{\*\}in the data limited setting by fitting the rewriting scaling law \(Eq\.[1](https://arxiv.org/html/2610.31867#S2.E1)\) for each target size to budgets throughW=18W=18and forecastR^∗\\widehat\{R\}^\{\*\}for the held\-outW=21W=21andW=24W=24runs\. Figure[4](https://arxiv.org/html/2610.31867#S3.F4)presents the results of this experiment\. AtR∗R^\{\*\}, the measured loss is within0\.74%0\.74\\%of the forecast at every target size for both held\-out budgets\. For the550550M,837837M, and11B target models,R^∗\\widehat\{R\}^\{\*\}empirically minimizes Paloma BPB among those testedRR\(except for550550M atW=21W=21, where it is within0\.01%0\.01\\%of the Paloma BPB at the best choice ofRR\)\. For the33B target model atW=21W=21, the scaling law selectsR^∗=0\.47\\widehat\{R\}^\{\*\}=0\.47and achieves a Paloma BPB of1\.1751\.175,0\.5%0\.5\\%above the best observedR=0\.71R=0\.71; atW=24W=24the scaling law selectsR^∗=0\.47\\widehat\{R\}^\{\*\}=0\.47and achieves a Paloma BPB of1\.1781\.178,1\.14%1\.14\\%above the loss at the best observedR=0\.71R=0\.71\.

Figure 6:Rewrites beat natural\-only epoching on Math and Code at every budget, but on QA only afterW=6W=6\.Final\-checkpoint BPB for the33B target against the epoch budgetWWon QA, Math, and Code, with the same three series as Figure[5](https://arxiv.org/html/2610.31867#S3.F5)\. Appendix[H\.1](https://arxiv.org/html/2610.31867#A8.SS1)shows every target size\.
##### Rewrites reach a lower loss than natural\-only epoching at every model size \(RQ2\)\.

Figure[5](https://arxiv.org/html/2610.31867#S3.F5)compares natural\-only \(R=0R=0\) models to those trained atR^∗\\widehat\{R\}^\{\*\}across a range of epoch budgetsWW; in particular, these plots show the final checkpoint’s Paloma BPB, with the result that for epoch budgetsW\>3W\>3the natural\-only runs overfit\. Training atR^∗\\widehat\{R\}^\{\*\}results in lower Paloma BPB than training atR=0R=0at every size\.

##### The best observed rewrite fraction depends on the task\.

Figure[6](https://arxiv.org/html/2610.31867#S3.F6)repeats Figure[5](https://arxiv.org/html/2610.31867#S3.F5)for the33B target on QA, Math, and Code\. On Math and Code the best mixture withR\>0R\>0beats the best natural\-only run at every budget\. On QA the natural\-only run is best throughW=6W=6, by0\.0140\.014BPB; beyond that, including synthetic rewrites outperforms\.

## 4Related Work

##### Data\-constrained pretraining\.

When the corpus is finite, additional compute means additional epochs, whose returns diminish\([Muennighoff et al\., 2023](https://arxiv.org/html/2610.31867#bib.bib4)\)\.[Lovelace et al\. \(2026\)](https://arxiv.org/html/2610.31867#bib.bib2)model this with a repetition penalty that activates beyond the first epoch, which we adopt as theF⁡\(R,D,T\)F\(R,D,T\)term of Eq\. \([1](https://arxiv.org/html/2610.31867#S2.E1)\)\.[Kim et al\. \(2025\)](https://arxiv.org/html/2610.31867#bib.bib7);[Kim et al\. \(2026\)](https://arxiv.org/html/2610.31867#bib.bib8)study the same regime from the perspective of a fixed data budget and unlimited compute, which is our data\-limited setting\.

##### Synthetic rewriting\.

Rewriting prompts a language model to restate a pretraining document, typically in a target style, and trains on the result alongside or instead of the original\([Maini et al\., 2024](https://arxiv.org/html/2610.31867#bib.bib5)\)\. Rewrites are now a component of several high\-quality pretraining corpora\([Su et al\., 2024](https://arxiv.org/html/2610.31867#bib.bib12);[Nguyen et al\., 2025](https://arxiv.org/html/2610.31867#bib.bib14);[Niklaus et al\., 2026](https://arxiv.org/html/2610.31867#bib.bib21);[Mahabad et al\., 2025](https://arxiv.org/html/2610.31867#bib.bib13);[Fujii et al\., 2025](https://arxiv.org/html/2610.31867#bib.bib15);[Ben Allal et al\., 2025](https://arxiv.org/html/2610.31867#bib.bib16)\), and[Maini et al\. \(2025\)](https://arxiv.org/html/2610.31867#bib.bib17)report that their benefit persists with scale\.

##### Rewriting as distillation\.

Whether rewrites help because the rewriter knows more than the corpus is a standing question\.[Maini et al\. \(2024\)](https://arxiv.org/html/2610.31867#bib.bib5)ablate the rewriter and find that a1\.81\.8B model retains most of the gain of a77B one\.[Maini et al\. \(2025\)](https://arxiv.org/html/2610.31867#bib.bib17)find at trillion\-token scale that returns to rewriter size saturate near33B parameters and that a rewriter’s benchmark scores are a poor guide to the value of its rewrites\.[Yang et al\. \(2025b\)](https://arxiv.org/html/2610.31867#bib.bib30)remove the external rewriter altogether, training a synthesizer on the pretraining corpus itself, and show that a33B model still gains from the documents it synthesizes\. Section[3\.2](https://arxiv.org/html/2610.31867#S3.SS2)and Section[3\.3](https://arxiv.org/html/2610.31867#S3.SS3)measure whether the rewriter’s BPB caps the gains \(RQ2\) in each setting\.

##### Scaling laws for synthetic pretraining data\.

Three recent studies fit scaling laws to synthetic pretraining data\.[Kang et al\. \(2025\)](https://arxiv.org/html/2610.31867#bib.bib29)train over a thousand models of100100M–33B parameters on mixtures of CommonCrawl with rephrased and textbook\-style synthetic data from a77B generator, fitting a separate Chinchilla law to each mixture: the best rephrased mixtures lie near30%30\\%synthetic, pure rephrased data underperforms natural text, and mixtures containing rephrased data are projected to reach a lower irreducible loss than natural text alone\.[Qin et al\. \(2026\)](https://arxiv.org/html/2610.31867#bib.bib27)unify compute\- and data\-optimal scaling through a token\-effectiveness function that discounts derived tokens, repetitions or SmolLM2\-1\.7B\-instruct paraphrases of a fixed fresh corpus, relative to fresh ones, fit jointly on models of1414M–600600M parameters\. They find that effectiveness falls with model size and tokens per parameter\.[Qin et al\. \(2025\)](https://arxiv.org/html/2610.31867#bib.bib28)scale synthetic math data alone, generated from FineWeb\-Edu math documents by a123123B model, in continued training of11B–88B base models, and find a rectified scaling law in synthetic tokens that plateaus near300300B tokens, with larger models needing fewer\. None of the three account for compute spent on generation, so their optimal mixtures answer a different question: how much to rewrite when rewrites are free\.[Qin et al\. \(2026\)](https://arxiv.org/html/2610.31867#bib.bib27)project that paraphrasing should stop helping beyond77B parameters\. However, our2222B\-parameter run at the predicted optimum reaches0\.9830\.983Paloma BPB against1\.0611\.061for natural data alone at the same budget\. All three also use an off\-the\-shelf generator stronger than the models they train\. In the data\-limited setting we pretrain the rewriter on the seed set and instruction\-tune it on only77M tokens of external rewrites, so gains from its rewrites can draw on little knowledge the corpus lacks\. Appendix[N\.1\.1](https://arxiv.org/html/2610.31867#A14.SS1.SSS1)tests the token\-effectiveness model of[Qin et al\. \(2026\)](https://arxiv.org/html/2610.31867#bib.bib27)against ours\.

## 5Conclusion and Limitations

We asked whether scaling laws predict the cost\-adjusted optimal rewrite fraction and associated loss \(RQ1\) and whether rewrites help when targeting capabilities beyond the rewriter \(RQ2\)\. Our rewriting scaling law answers both\. In the compute\-limited setting, the rewrite fraction fit at small budgets minimizes loss among the fractions we train at33×33\\timesthe compute, and a pretrained model with 32% rewritten training data outperforms the rewriter on 10/16 general web text domains\. In the data\-limited setting, we similarly find that the optimal rewrite fractions and losses of held\-out budgets can be predicted via scaling laws, and that a positive proportion of rewrites is additive even when the rewriter model is less capable than the target model\.

##### Limitations\.

Several choices limit the extent of our results\. One is the choice of the natural seed set: every optimal rewrite fraction we report is for FineWeb\-Edu\. We also study this question on the DCLM seed set, where rewrites still help but by less \(Appendix[N\.7](https://arxiv.org/html/2610.31867#A14.SS7)\), suggesting that a strong natural seed set benefits less from rewriting\. We also test only the four FinePhrase rewrite styles and one external rewriter in each setting, so we cannot say how the optimal fraction varies across rewrite prompt distributions or across a family of rewriters\. Finally, our compute accounting is approximate: we charge4​Nr4N\_\{r\}FLOPs per rewritten token, which approximates decoding as costing the same FLOPs as prefill and assumes the rewrite is the same length as its source; we do not account for costs associated with pretraining the rewriter model\.

### AI Use Statement

This paper studies scaling laws for synthetic data generated by language models\. Rewrites in the compute\-limited setting are FinePhrase rewrites generated by SmolLM2\-1\.7B, and rewrites in the data\-limited setting are generated by rewriters we pretrain ourselves and SFT on GLM\-5\.2 rewrites of10001000seed\-set documents\. Finally, an LM judge selects the rewriter checkpoint \(Section[3\.1](https://arxiv.org/html/2610.31867#S3.SS1), Appendices[F](https://arxiv.org/html/2610.31867#A6)and[J](https://arxiv.org/html/2610.31867#A10)\)\. We used AI assistants to discuss analysis choices, to implement and debug analysis code, for initial drafts of paper sections, and for feedback on prose\. We have reviewed, verified, and updated code and text originating from AI\.

#### Acknowledgments

We thank Lucas Torroba\-Hennigen, Howard Yen, and Abbas Zeitoun for the many thoughtful discussions during the internship—we learned so much from you all\. We also thank Rylan Schaeffer and Kaiyue Wen for their helpful feedback on drafts of this paper\. Finally, we are grateful to the members of Danqi Chen’s, Sanjeev Arora’s, and Frederic Sala’s groups for their valuable feedback\.

## References

- J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le, and C\. SuttonProgram synthesis with large language models\.External Links:2108\.07732,[Link](https://arxiv.org/abs/2108.07732)Cited by:[§3\.1](https://arxiv.org/html/2610.31867#S3.SS1.SSS0.Px4.p1.1)\.
- Ben Allalet al\.\(2025\)L\. Ben Allal, A\. Lozhkov, E\. Bakouch, G\. M\. Blázquez, G\. Penedo, L\. Tunstall, A\. Marafioti, H\. Kydlíček, A\. P\. Lajarín, V\. Srivastav, J\. Lochner, C\. Fahlgren, X\. Nguyen, C\. Fourrier, B\. Burtenshaw, H\. Larcher, H\. Zhao, C\. Zakka, M\. Morlon, C\. Raffel, L\. von Werra, and T\. WolfSmolLM2: when smol goes big – data\-centric training of a small language model\.External Links:2502\.02737,[Link](https://arxiv.org/abs/2502.02737)Cited by:[§N\.2](https://arxiv.org/html/2610.31867#A14.SS2.SSS0.Px1.p1.1),[Appendix G](https://arxiv.org/html/2610.31867#A7.p1.1),[§3\.1](https://arxiv.org/html/2610.31867#S3.SS1.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2610.31867#S3.SS2.SSS0.Px1.p1.1),[§3\.2\.1](https://arxiv.org/html/2610.31867#S3.SS2.SSS1.Px1.p2.1),[§3\.2\.1](https://arxiv.org/html/2610.31867#S3.SS2.SSS1.Px3.p1.1),[§4](https://arxiv.org/html/2610.31867#S4.SS0.SSS0.Px2.p1.1)\.
- Cassanoet al\.\(2023\)F\. Cassano, J\. Gouwar, D\. Nguyen, S\. Nguyen, L\. Phipps\-Costin, D\. Pinckney, M\. Yee, Y\. Zi, C\. J\. Anderson, M\. Q\. Feldman, A\. Guha, M\. Greenberg, and A\. JangdaMultiPL\-E: a scalable and polyglot approach to benchmarking neural code generation\.IEEE Transactions on Software Engineering\.External Links:[Link](https://arxiv.org/abs/2208.08227)Cited by:[§3\.1](https://arxiv.org/html/2610.31867#S3.SS1.SSS0.Px4.p1.1)\.
- Chenet al\.\(2021\)M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. de Oliveira Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman, A\. Ray, R\. Puri, G\. Krueger, M\. Petrov, H\. Khlaaf, G\. Sastry, P\. Mishkin, B\. Chan, S\. Gray, N\. Ryder, M\. Pavlov, A\. Power, L\. Kaiser, M\. Bavarian, C\. Winter, P\. Tillet, F\. P\. Such, D\. Cummings, M\. Plappert, F\. Chantzis, E\. Barnes, A\. Herbert\-Voss, W\. H\. Guss, A\. Nichol, A\. Paino, N\. Tezak, J\. Tang, I\. Babuschkin, S\. Balaji, S\. Jain, W\. Saunders, C\. Hesse, A\. N\. Carr, J\. Leike, J\. Achiam, V\. Misra, E\. Morikawa, A\. Radford, M\. Knight, M\. Brundage, M\. Murati, K\. Mayer, P\. Welinder, B\. McGrew, D\. Amodei, S\. McCandlish, I\. Sutskever, and W\. ZarembaEvaluating large language models trained on code\.External Links:2107\.03374,[Link](https://arxiv.org/abs/2107.03374)Cited by:[§3\.1](https://arxiv.org/html/2610.31867#S3.SS1.SSS0.Px4.p1.1)\.
- Chenet al\.\(2023\)W\. Chen, M\. Yin, M\. Ku, P\. Lu, Y\. Wan, X\. Ma, J\. Xu, X\. Wang, and T\. XiaTheoremQA: a theorem\-driven question answering dataset\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,External Links:[Link](https://arxiv.org/abs/2305.12524)Cited by:[§3\.1](https://arxiv.org/html/2610.31867#S3.SS1.SSS0.Px4.p1.1)\.
- Clarket al\.\(2018\)P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. TafjordThink you have solved question answering? try arc, the ai2 reasoning challenge\.External Links:1803\.05457,[Link](https://arxiv.org/abs/1803.05457)Cited by:[§3\.1](https://arxiv.org/html/2610.31867#S3.SS1.SSS0.Px4.p1.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. SchulmanTraining verifiers to solve math word problems\.External Links:2110\.14168,[Link](https://arxiv.org/abs/2110.14168)Cited by:[§3\.1](https://arxiv.org/html/2610.31867#S3.SS1.SSS0.Px4.p1.1)\.
- Ettingeret al\.\(2025\)A\. Ettingeret al\.Olmo 3\.External Links:2512\.13961,[Link](https://arxiv.org/abs/2512.13961)Cited by:[§3\.1](https://arxiv.org/html/2610.31867#S3.SS1.SSS0.Px4.p1.1)\.
- Fujiiet al\.\(2025\)K\. Fujii, Y\. Tajima, S\. Mizuki, M\. Kawamura, H\. Shimada, T\. Shiotani, K\. Saito, M\. Oi, T\. Nakamura, T\. Okamoto, S\. Ishida, K\. Hattori, Y\. Ma, H\. Takamura, R\. Yokota, J\. Sakuma, and N\. OkazakiRewriting pre\-training data boosts llm performance in math and code\.External Links:2505\.02881,[Link](https://arxiv.org/abs/2505.02881)Cited by:[§4](https://arxiv.org/html/2610.31867#S4.SS0.SSS0.Px2.p1.1)\.
- Gaoet al\.\(2020\)L\. Gao, S\. Biderman, S\. Black, L\. Golding, T\. Hoppe, C\. Foster, J\. Phang, H\. He, A\. Thite, N\. Nabeshima, S\. Presser, and C\. LeahyThe pile: an 800gb dataset of diverse text for language modeling\.External Links:2101\.00027,[Link](https://arxiv.org/abs/2101.00027)Cited by:[§1](https://arxiv.org/html/2610.31867#S1.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan, A\. Yang, A\. Fan, A\. Goyal, A\. Hartshorn, A\. Yang, A\. Mitra, A\. Sravankumar, A\. Korenev, A\. Hinsvark, A\. Rao, A\. Zhang, A\. Rodriguez, A\. Gregerson, A\. Spataru, B\. Roziere, B\. Biron, B\. Tang, B\. Chern, C\. Caucheteux, C\. Nayak, C\. Bi, C\. Marra, C\. McConnell, C\. Keller, C\. Touret, C\. Wu, C\. Wong, C\. C\. Ferrer, C\. Nikolaidis, D\. Allonsius, D\. Song, D\. Pintz, D\. Livshits, D\. Wyatt, D\. Esiobu, D\. Choudhary, D\. Mahajan, D\. Garcia\-Olano, D\. Perino, D\. Hupkes, E\. Lakomkin, E\. AlBadawy, E\. Lobanova, E\. Dinan, E\. M\. Smith, F\. Radenovic, F\. Guzmán, F\. Zhang, G\. Synnaeve, G\. Lee, G\. L\. Anderson, G\. Thattai, G\. Nail, G\. Mialon, G\. Pang, G\. Cucurell, H\. Nguyen, H\. Korevaar, H\. Xu, H\. Touvron, I\. Zarov, I\. A\. Ibarra, I\. Kloumann, I\. Misra, I\. Evtimov, J\. Zhang, J\. Copet, J\. Lee, J\. Geffert, J\. Vranes, J\. Park, J\. Mahadeokar, J\. Shah, J\. van der Linde, J\. Billock, J\. Hong, J\. Lee, J\. Fu, J\. Chi, J\. Huang, J\. Liu, J\. Wang, J\. Yu, J\. Bitton, J\. Spisak, J\. Park, J\. Rocca, J\. Johnstun, J\. Saxe, J\. Jia, K\. V\. Alwala, K\. Prasad, K\. Upasani, K\. Plawiak, K\. Li, K\. Heafield, K\. Stone, K\. El\-Arini, K\. Iyer, K\. Malik, K\. Chiu, K\. Bhalla, K\. Lakhotia, L\. Rantala\-Yeary, L\. van der Maaten, L\. Chen, L\. Tan, L\. Jenkins, L\. Martin, L\. Madaan, L\. Malo, L\. Blecher, L\. Landzaat, L\. de Oliveira, M\. Muzzi, M\. Pasupuleti, M\. Singh, M\. Paluri, M\. Kardas, M\. Tsimpoukelli, M\. Oldham, M\. Rita, M\. Pavlova, M\. Kambadur, M\. Lewis, M\. Si, M\. K\. Singh, M\. Hassan, N\. Goyal, N\. Torabi, N\. Bashlykov, N\. Bogoychev, N\. Chatterji, N\. Zhang, O\. Duchenne, O\. Çelebi, P\. Alrassy, P\. Zhang, P\. Li, P\. Vasic, P\. Weng, P\. Bhargava, P\. Dubal, P\. Krishnan, P\. S\. Koura, P\. Xu, Q\. He, Q\. Dong, R\. Srinivasan, R\. Ganapathy, R\. Calderer, R\. S\. Cabral, R\. Stojnic, R\. Raileanu, R\. Maheswari, R\. Girdhar, R\. Patel, R\. Sauvestre, R\. Polidoro, R\. Sumbaly, R\. Taylor, R\. Silva, R\. Hou, R\. Wang, S\. Hosseini, S\. Chennabasappa, S\. Singh, S\. Bell, S\. S\. Kim, S\. Edunov, S\. Nie, S\. Narang, S\. Raparthy, S\. Shen, S\. Wan, S\. Bhosale, S\. Zhang, S\. Vandenhende, S\. Batra, S\. Whitman, S\. Sootla, S\. Collot, S\. Gururangan, S\. Borodinsky, T\. Herman, T\. Fowler, T\. Sheasha, T\. Georgiou, T\. Scialom, T\. Speckbacher, T\. Mihaylov, T\. Xiao, U\. Karn, V\. Goswami, V\. Gupta, V\. Ramanathan, V\. Kerkez, V\. Gonguet, V\. Do, V\. Vogeti, V\. Albiero, V\. Petrovic, W\. Chu, W\. Xiong, W\. Fu, W\. Meers, X\. Martinet, X\. Wang, X\. Wang, X\. E\. Tan, X\. Xia, X\. Xie, X\. Jia, X\. Wang, Y\. Goldschlag, Y\. Gaur, Y\. Babaei, Y\. Wen, Y\. Song, Y\. Zhang, Y\. Li, Y\. Mao, Z\. D\. Coudert, Z\. Yan, Z\. Chen, Z\. Papakipos, A\. Singh, A\. Srivastava, A\. Jain, A\. Kelsey, A\. Shajnfeld, A\. Gangidi, A\. Victoria, A\. Goldstand, A\. Menon, A\. Sharma, A\. Boesenberg, A\. Baevski, A\. Feinstein, A\. Kallet, A\. Sangani, A\. Teo, A\. Yunus, A\. Lupu, A\. Alvarado, A\. Caples, A\. Gu, A\. Ho, A\. Poulton, A\. Ryan, A\. Ramchandani, A\. Dong, A\. Franco, A\. Goyal, A\. Saraf, A\. Chowdhury, A\. Gabriel, A\. Bharambe, A\. Eisenman, A\. Yazdan, B\. James, B\. Maurer, B\. Leonhardi, B\. Huang, B\. Loyd, B\. D\. Paola, B\. Paranjape, B\. Liu, B\. Wu, B\. Ni, B\. Hancock, B\. Wasti, B\. Spence, B\. Stojkovic, B\. Gamido, B\. Montalvo, C\. Parker, C\. Burton, C\. Mejia, C\. Liu, C\. Wang, C\. Kim, C\. Zhou, C\. Hu, C\. Chu, C\. Cai, C\. Tindal, C\. Feichtenhofer, C\. Gao, D\. Civin, D\. Beaty, D\. Kreymer, D\. Li, D\. Adkins, D\. Xu, D\. Testuggine, D\. David, D\. Parikh, D\. Liskovich, D\. Foss, D\. Wang, D\. Le, D\. Holland, E\. Dowling, E\. Jamil, E\. Montgomery, E\. Presani, E\. Hahn, E\. Wood, E\. Le, E\. Brinkman, E\. Arcaute, E\. Dunbar, E\. Smothers, F\. Sun, F\. Kreuk, F\. Tian, F\. Kokkinos, F\. Ozgenel, F\. Caggioni, F\. Kanayet, F\. Seide, G\. M\. Florez, G\. Schwarz, G\. Badeer, G\. Swee, G\. Halpern, G\. Herman, G\. Sizov, Guangyi, Zhang, G\. Lakshminarayanan, H\. Inan, H\. Shojanazeri, H\. Zou, H\. Wang, H\. Zha, H\. Habeeb, H\. Rudolph, H\. Suk, H\. Aspegren, H\. Goldman, H\. Zhan, I\. Damlaj, I\. Molybog, I\. Tufanov, I\. Leontiadis, I\. Veliche, I\. Gat, J\. Weissman, J\. Geboski, J\. Kohli, J\. Lam, J\. Asher, J\. Gaya, J\. Marcus, J\. Tang, J\. Chan, J\. Zhen, J\. Reizenstein, J\. Teboul, J\. Zhong, J\. Jin, J\. Yang, J\. Cummings, J\. Carvill, J\. Shepard, J\. McPhie, J\. Torres, J\. Ginsburg, J\. Wang, K\. Wu, K\. H\. U, K\. Saxena, K\. Khandelwal, K\. Zand, K\. Matosich, K\. Veeraraghavan, K\. Michelena, K\. Li, K\. Jagadeesh, K\. Huang, K\. Chawla, K\. Huang, L\. Chen, L\. Garg, L\. A, L\. Silva, L\. Bell, L\. Zhang, L\. Guo, L\. Yu, L\. Moshkovich, L\. Wehrstedt, M\. Khabsa, M\. Avalani, M\. Bhatt, M\. Mankus, M\. Hasson, M\. Lennie, M\. Reso, M\. Groshev, M\. Naumov, M\. Lathi, M\. Keneally, M\. Liu, M\. L\. Seltzer, M\. Valko, M\. Restrepo, M\. Patel, M\. Vyatskov, M\. Samvelyan, M\. Clark, M\. Macey, M\. Wang, M\. J\. Hermoso, M\. Metanat, M\. Rastegari, M\. Bansal, N\. Santhanam, N\. Parks, N\. White, N\. Bawa, N\. Singhal, N\. Egebo, N\. Usunier, N\. Mehta, N\. P\. Laptev, N\. Dong, N\. Cheng, O\. Chernoguz, O\. Hart, O\. Salpekar, O\. Kalinli, P\. Kent, P\. Parekh, P\. Saab, P\. Balaji, P\. Rittner, P\. Bontrager, P\. Roux, P\. Dollar, P\. Zvyagina, P\. Ratanchandani, P\. Yuvraj, Q\. Liang, R\. Alao, R\. Rodriguez, R\. Ayub, R\. Murthy, R\. Nayani, R\. Mitra, R\. Parthasarathy, R\. Li, R\. Hogan, R\. Battey, R\. Wang, R\. Howes, R\. Rinott, S\. Mehta, S\. Siby, S\. J\. Bondu, S\. Datta, S\. Chugh, S\. Hunt, S\. Dhillon, S\. Sidorov, S\. Pan, S\. Mahajan, S\. Verma, S\. Yamamoto, S\. Ramaswamy, S\. Lindsay, S\. Lindsay, S\. Feng, S\. Lin, S\. C\. Zha, S\. Patil, S\. Shankar, S\. Zhang, S\. Zhang, S\. Wang, S\. Agarwal, S\. Sajuyigbe, S\. Chintala, S\. Max, S\. Chen, S\. Kehoe, S\. Satterfield, S\. Govindaprasad, S\. Gupta, S\. Deng, S\. Cho, S\. Virk, S\. Subramanian, S\. Choudhury, S\. Goldman, T\. Remez, T\. Glaser, T\. Best, T\. Koehler, T\. Robinson, T\. Li, T\. Zhang, T\. Matthews, T\. Chou, T\. Shaked, V\. Vontimitta, V\. Ajayi, V\. Montanez, V\. Mohan, V\. S\. Kumar, V\. Mangla, V\. Ionescu, V\. Poenaru, V\. T\. Mihailescu, V\. Ivanov, W\. Li, W\. Wang, W\. Jiang, W\. Bouaziz, W\. Constable, X\. Tang, X\. Wu, X\. Wang, X\. Wu, X\. Gao, Y\. Kleinman, Y\. Chen, Y\. Hu, Y\. Jia, Y\. Qi, Y\. Li, Y\. Zhang, Y\. Zhang, Y\. Adi, Y\. Nam, Yu, Wang, Y\. Zhao, Y\. Hao, Y\. Qian, Y\. Li, Y\. He, Z\. Rait, Z\. DeVito, Z\. Rosnbrick, Z\. Wen, Z\. Yang, Z\. Zhao, and Z\. MaThe llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[Appendix B](https://arxiv.org/html/2610.31867#A2.p1.1),[§3\.1](https://arxiv.org/html/2610.31867#S3.SS1.SSS0.Px1.p1.1)\.
- Hallet al\.\(2023\)D\. Hall, P\. Liang, and Marin CommunityLevanter: legible, scalable, reproducible foundation models with JAX\.Note:[https://github\.com/marin\-community/levanter](https://github.com/marin-community/levanter)SoftwareCited by:[Appendix I](https://arxiv.org/html/2610.31867#A9.p1.1)\.
- Held \(2026\)W\. HeldScaling laws that extrapolate 300×\\timespast the fit\.Note:[https://www\.openathena\.ai/blog/delphi/](https://www.openathena.ai/blog/delphi/)Open Athena BlogCited by:[Appendix C](https://arxiv.org/html/2610.31867#A3.SS0.SSS0.Px1.p1.1),[Appendix C](https://arxiv.org/html/2610.31867#A3.SS0.SSS0.Px2.p1.1),[Appendix C](https://arxiv.org/html/2610.31867#A3.p1.1)\.
- Hendryckset al\.\(2021a\)D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. SteinhardtMeasuring massive multitask language understanding\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2009.03300)Cited by:[§3\.1](https://arxiv.org/html/2610.31867#S3.SS1.SSS0.Px4.p1.1)\.
- Hendryckset al\.\(2021b\)D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. SteinhardtMeasuring mathematical problem solving with the MATH dataset\.InAdvances in Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://arxiv.org/abs/2103.03874)Cited by:[§3\.1](https://arxiv.org/html/2610.31867#S3.SS1.SSS0.Px4.p1.1)\.
- Hoffmannet al\.\(2022\)J\. Hoffmann, S\. Borgeaud, A\. Mensch, E\. Buchatskaya, T\. Cai, E\. Rutherford, D\. de Las Casas, L\. A\. Hendricks, J\. Welbl, A\. Clark, T\. Hennigan, E\. Noland, K\. Millican, G\. van den Driessche, B\. Damoc, A\. Guy, S\. Osindero, K\. Simonyan, E\. Elsen, J\. W\. Rae, O\. Vinyals, and L\. SifreTraining compute\-optimal large language models\.External Links:2203\.15556,[Link](https://arxiv.org/abs/2203.15556)Cited by:[§L\.1](https://arxiv.org/html/2610.31867#A12.SS1.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2610.31867#S1.p1.1),[§1](https://arxiv.org/html/2610.31867#S1.p6.1),[§2\.1](https://arxiv.org/html/2610.31867#S2.SS1.p3.1)\.
- Kanget al\.\(2025\)F\. Kang, N\. Ardalani, M\. Kuchnik, Y\. Emad, M\. Elhoushi, S\. Sengupta, S\. Li, R\. Raghavendra, R\. Jia, and C\. WuDemystifying synthetic data in llm pre\-training: a systematic study of scaling laws, benefits, and pitfalls\.External Links:2510\.01631,[Link](https://arxiv.org/abs/2510.01631)Cited by:[§4](https://arxiv.org/html/2610.31867#S4.SS0.SSS0.Px4.p1.1)\.
- Kaplanet al\.\(2020\)J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. AmodeiScaling laws for neural language models\.External Links:2001\.08361,[Link](https://arxiv.org/abs/2001.08361)Cited by:[§1](https://arxiv.org/html/2610.31867#S1.p1.1),[§2\.2](https://arxiv.org/html/2610.31867#S2.SS2.p3.1)\.
- Kimet al\.\(2026\)K\. Kim, S\. Kotha, Y\. Choi, T\. Hashimoto, N\. Haber, and P\. LiangData\-efficient pre\-training by scaling synthetic megadocs\.External Links:2603\.18534,[Link](https://arxiv.org/abs/2603.18534)Cited by:[§4](https://arxiv.org/html/2610.31867#S4.SS0.SSS0.Px1.p1.1)\.
- Kimet al\.\(2025\)K\. Kim, S\. Kotha, P\. Liang, and T\. HashimotoPre\-training under infinite compute\.External Links:2509\.14786,[Link](https://arxiv.org/abs/2509.14786)Cited by:[§1](https://arxiv.org/html/2610.31867#S1.p1.1),[§1](https://arxiv.org/html/2610.31867#S1.p6.1),[§4](https://arxiv.org/html/2610.31867#S4.SS0.SSS0.Px1.p1.1)\.
- Levineet al\.\(2021\)Y\. Levine, N\. Wies, O\. Sharir, H\. Bata, and A\. ShashuaThe depth\-to\-width interplay in self\-attention\.External Links:2006\.12467,[Link](https://arxiv.org/abs/2006.12467)Cited by:[Appendix B](https://arxiv.org/html/2610.31867#A2.p1.1)\.
- Lewkowyczet al\.\(2022\)A\. Lewkowycz, A\. Andreassen, D\. Dohan, E\. Dyer, H\. Michalewski, V\. Ramasesh, A\. Slone, C\. Anil, I\. Schlag, T\. Gutman\-Solo, Y\. Wu, B\. Neyshabur, G\. Gur\-Ari, and V\. MisraSolving quantitative reasoning problems with language models\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://arxiv.org/abs/2206.14858)Cited by:[§3\.1](https://arxiv.org/html/2610.31867#S3.SS1.SSS0.Px4.p1.1)\.
- Liet al\.\(2024\)J\. Li, A\. Fang, G\. Smyrnis, M\. Ivgi, M\. Jordan, S\. Gadre, H\. Bansal, E\. Guha, S\. Keh, K\. Arora,et al\.DataComp\-lm: in search of the next generation of training sets for language models\.External Links:2406\.11794,[Link](https://arxiv.org/abs/2406.11794)Cited by:[Appendix A](https://arxiv.org/html/2610.31867#A1.SS0.SSS0.Px1.p1.1),[§N\.7](https://arxiv.org/html/2610.31867#A14.SS7.SSS0.Px1.p1.1)\.
- Liet al\.\(2023\)R\. Li, L\. Ben Allal, Y\. Zi, N\. Muennighoff, D\. Kocetkov, C\. Mou, M\. Marone, C\. Akiki, J\. Li, J\. Chim, Q\. Liu, E\. Zheltonozhskii, T\. Y\. Zhuo, T\. Wang, O\. Dehaene, M\. Davaadorj, J\. Lamy\-Poirier, J\. Monteiro, O\. Shliazhko, N\. Gontier, N\. Meade, A\. Zebaze, M\. Yee, L\. K\. Umapathi, J\. Zhu, B\. Lipkin, M\. Oblokulov, Z\. Wang, R\. Murthy, J\. Stillerman, S\. S\. Patel, D\. Abulkhanov, M\. Zocca, M\. Dey, Z\. Zhang, N\. Fahmy, U\. Bhattacharyya, W\. Yu, S\. Singh, S\. Luccioni, P\. Villegas, M\. Kunakov, F\. Zhdanov, M\. Romero, T\. Lee, N\. Timor, J\. Ding, C\. Schlesinger, H\. Schoelkopf, J\. Ebert, T\. Dao, M\. Mishra, A\. Gu, J\. Robinson, C\. J\. Anderson, B\. Dolan\-Gavitt, D\. Contractor, S\. Reddy, D\. Fried, D\. Bahdanau, Y\. Jernite, C\. M\. Ferrandis, S\. Hughes, T\. Wolf, A\. Guha, L\. von Werra, and H\. de VriesStarCoder: may the source be with you\!\.External Links:2305\.06161,[Link](https://arxiv.org/abs/2305.06161)Cited by:[§N\.2](https://arxiv.org/html/2610.31867#A14.SS2.SSS0.Px1.p1.1),[§3\.2\.1](https://arxiv.org/html/2610.31867#S3.SS2.SSS1.Px1.p2.1)\.
- Lovelaceet al\.\(2026\)J\. Lovelace, C\. Belardi, S\. Kundurthy, S\. Sudhakar, and K\. Q\. WeinbergerPrescriptive scaling laws for data constrained training\.External Links:2605\.01640,[Link](https://arxiv.org/abs/2605.01640)Cited by:[Appendix L](https://arxiv.org/html/2610.31867#A12.SS0.SSS0.Px2.p1.1),[§L\.2](https://arxiv.org/html/2610.31867#A12.SS2.SSS0.Px2.p1.2),[§N\.1\.5](https://arxiv.org/html/2610.31867#A14.SS1.SSS5.p1.1),[§N\.1\.5](https://arxiv.org/html/2610.31867#A14.SS1.SSS5.p2.1),[§2\.1](https://arxiv.org/html/2610.31867#S2.SS1.p5.1),[§4](https://arxiv.org/html/2610.31867#S4.SS0.SSS0.Px1.p1.1)\.
- Magnussonet al\.\(2024\)I\. Magnusson, A\. Bhagia, V\. Hofmann, L\. Soldaini, A\. H\. Jha, O\. Tafjord, D\. Schwenk, E\. P\. Walsh, Y\. Elazar, K\. Lo, D\. Groeneveld, I\. Beltagy, H\. Hajishirzi, N\. A\. Smith, K\. Richardson, and J\. DodgePaloma: a benchmark for evaluating language model fit\.InThe Thirty\-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=yjj8ele147)Cited by:[Appendix D](https://arxiv.org/html/2610.31867#A4.p1.1),[§3\.1](https://arxiv.org/html/2610.31867#S3.SS1.SSS0.Px4.p1.1)\.
- Mahabadet al\.\(2025\)R\. K\. Mahabad, S\. Satheesh, S\. Prabhumoye, M\. Patwary, M\. Shoeybi, and B\. CatanzaroNemotron\-cc\-math: a 133 billion\-token\-scale high quality math pretraining dataset\.External Links:2508\.15096,[Link](https://arxiv.org/abs/2508.15096)Cited by:[§1](https://arxiv.org/html/2610.31867#S1.p2.1),[§4](https://arxiv.org/html/2610.31867#S4.SS0.SSS0.Px2.p1.1)\.
- Mainiet al\.\(2025\)P\. Maini, V\. Dorna, P\. Doshi, A\. Carranza, F\. Pan, J\. Urbanek, P\. Burstein, A\. Fang, A\. Deng, A\. Abbas,et al\.BeyondWeb: lessons from scaling synthetic data for trillion\-scale pretraining\.External Links:2508\.10975,[Link](https://arxiv.org/abs/2508.10975)Cited by:[§1](https://arxiv.org/html/2610.31867#S1.p2.1),[§1](https://arxiv.org/html/2610.31867#S1.p3.1),[§4](https://arxiv.org/html/2610.31867#S4.SS0.SSS0.Px2.p1.1),[§4](https://arxiv.org/html/2610.31867#S4.SS0.SSS0.Px3.p1.1)\.
- Mainiet al\.\(2024\)P\. Maini, S\. Seto, H\. Bai, D\. Grangier, Y\. Zhang, and N\. JaitlyRephrasing the web: a recipe for compute and data\-efficient language modeling\.External Links:2401\.16380,[Link](https://arxiv.org/abs/2401.16380)Cited by:[§1](https://arxiv.org/html/2610.31867#S1.p2.1),[§4](https://arxiv.org/html/2610.31867#S4.SS0.SSS0.Px2.p1.1),[§4](https://arxiv.org/html/2610.31867#S4.SS0.SSS0.Px3.p1.1)\.
- MiniMax \(2026\)MiniMaxMiniMax\-M2\.7\.Note:[https://www\.minimax\.io/news/minimax\-m27\-en](https://www.minimax.io/news/minimax-m27-en)Cited by:[§F\.1](https://arxiv.org/html/2610.31867#A6.SS1.p2.1)\.
- Mlodozeniecet al\.\(2026\)B\. Mlodozeniec, P\. Ablin, L\. Béthune, D\. Busbridge, M\. Klein, J\. Ramapuram, and M\. CuturiCompleted hyperparameter transfer across modules, width, depth, batch and duration\.InICLR,External Links:[Link](https://arxiv.org/abs/2512.22382)Cited by:[Appendix C](https://arxiv.org/html/2610.31867#A3.p1.1)\.
- Muennighoffet al\.\(2023\)N\. Muennighoff, A\. M\. Rush, B\. Barak, T\. L\. Scao, A\. Piktus, N\. Tazi, S\. Pyysalo, T\. Wolf, and C\. RaffelScaling data\-constrained language models\.External Links:2305\.16264,[Link](https://arxiv.org/abs/2305.16264)Cited by:[§N\.1\.5](https://arxiv.org/html/2610.31867#A14.SS1.SSS5.p1.1),[§4](https://arxiv.org/html/2610.31867#S4.SS0.SSS0.Px1.p1.1)\.
- Nguyenet al\.\(2025\)T\. Nguyen, Y\. Li, O\. Golovneva, L\. Zettlemoyer, S\. Oh, L\. Schmidt, and X\. LiRecycling the web: a method to enhance pre\-training data quality and quantity for language models\.External Links:2506\.04689,[Link](https://arxiv.org/abs/2506.04689)Cited by:[§1](https://arxiv.org/html/2610.31867#S1.p2.1),[§1](https://arxiv.org/html/2610.31867#S1.p3.1),[§4](https://arxiv.org/html/2610.31867#S4.SS0.SSS0.Px2.p1.1)\.
- Niklauset al\.\(2026\)J\. Niklaus, A\. Yamaguchi, M\. Štefánik, G\. Penedo, H\. Kydlíček, E\. Bakouch, L\. Tunstall, E\. E\. Beeching, T\. Frere, C\. Raffel, L\. von Werra, and T\. WolfHow can we synthesize high\-quality pretraining data? a systematic study of prompt design, generator model, and source data\.External Links:2604\.13977,[Link](https://arxiv.org/abs/2604.13977)Cited by:[Appendix A](https://arxiv.org/html/2610.31867#A1.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2610.31867#S1.p3.1),[§3\.1](https://arxiv.org/html/2610.31867#S3.SS1.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2610.31867#S3.SS1.SSS0.Px3.p1.1),[§3\.2](https://arxiv.org/html/2610.31867#S3.SS2.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2610.31867#S4.SS0.SSS0.Px2.p1.1)\.
- Penedoet al\.\(2024\)G\. Penedo, H\. Kydlíček, L\. Ben Allal, A\. Lozhkov, M\. Mitchell, C\. Raffel, L\. von Werra, and T\. WolfThe fineweb datasets: decanting the web for the finest text data at scale\.External Links:2406\.17557,[Link](https://arxiv.org/abs/2406.17557)Cited by:[Appendix A](https://arxiv.org/html/2610.31867#A1.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2610.31867#S3.SS1.SSS0.Px2.p1.1)\.
- Popeet al\.\(2023\)R\. Pope, S\. Douglas, A\. Chowdhery, J\. Devlin, J\. Bradbury, A\. Levskaya, J\. Heek, K\. Xiao, S\. Agrawal, and J\. DeanEfficiently scaling transformer inference\.InProceedings of Machine Learning and Systems,Vol\.5,pp\. 606–624\.External Links:[Link](https://arxiv.org/abs/2211.05102)Cited by:[footnote 1](https://arxiv.org/html/2610.31867#footnote1)\.
- Qinet al\.\(2026\)T\. Qin, K\. Hamidieh, and D\. Alvarez\-MelisBridging compute\- and data\-optimal pretraining\.External Links:2607\.25271,[Link](https://arxiv.org/abs/2607.25271)Cited by:[§4](https://arxiv.org/html/2610.31867#S4.SS0.SSS0.Px4.p1.1)\.
- Qinet al\.\(2025\)Z\. Qin, Q\. Dong, X\. Zhang, L\. Dong, X\. Huang, Z\. Yang, M\. KHADEMI, D\. Zhang, H\. H\. Awadalla, Y\. R\. Fung, W\. Chen, M\. Cheng, and F\. WeiScaling laws of synthetic data for language model\.InSecond Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=UmUXPXHtdl)Cited by:[§4](https://arxiv.org/html/2610.31867#S4.SS0.SSS0.Px4.p1.1)\.
- Robertset al\.\(2025\)N\. Roberts, N\. Chatterji, S\. Narang, M\. Lewis, and D\. HupkesCompute optimal scaling of skills: knowledge vs reasoning\.External Links:2503\.10061,[Link](https://arxiv.org/abs/2503.10061)Cited by:[§N\.2](https://arxiv.org/html/2610.31867#A14.SS2.SSS0.Px2.p1.1)\.
- Suet al\.\(2024\)D\. Su, K\. Kong, Y\. Lin, J\. Jennings, B\. Norick, M\. Kliegl, M\. Patwary, M\. Shoeybi, and B\. CatanzaroNemotron\-cc: transforming common crawl into a refined long\-horizon pretraining dataset\.External Links:2412\.02595,[Link](https://arxiv.org/abs/2412.02595)Cited by:[§1](https://arxiv.org/html/2610.31867#S1.p2.1),[§1](https://arxiv.org/html/2610.31867#S1.p3.1),[§4](https://arxiv.org/html/2610.31867#S4.SS0.SSS0.Px2.p1.1)\.
- Vendrowet al\.\(2025\)J\. Vendrow, E\. Vendrow, S\. Beery, and A\. MadryDo large language model benchmarks test reliability?\.External Links:2502\.03461,[Link](https://arxiv.org/abs/2502.03461)Cited by:[§3\.1](https://arxiv.org/html/2610.31867#S3.SS1.SSS0.Px4.p1.1)\.
- Villaloboset al\.\(2022\)P\. Villalobos, A\. Ho, J\. Sevilla, T\. Besiroglu, L\. Heim, and M\. HobbhahnWill we run out of data? limits of llm scaling based on human\-generated data\.External Links:2211\.04325,[Link](https://arxiv.org/abs/2211.04325)Cited by:[§1](https://arxiv.org/html/2610.31867#S1.p1.1)\.
- Wenet al\.\(2026\)K\. Wen, X\. Dang, K\. Lyu, T\. Ma, and P\. LiangFantastic pretraining optimizers and where to find them ii: hyperball optimization\.External Links:2606\.16899,[Link](https://arxiv.org/abs/2606.16899)Cited by:[Appendix C](https://arxiv.org/html/2610.31867#A3.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2610.31867#S3.SS1.SSS0.Px1.p1.1)\.
- Yanget al\.\(2025a\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. QiuQwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§3\.1](https://arxiv.org/html/2610.31867#S3.SS1.SSS0.Px1.p1.1)\.
- Yanget al\.\(2025b\)Z\. Yang, A\. Zhang, H\. Liu, T\. Hashimoto, E\. Candès, C\. Wang, and R\. PangSynthetic bootstrapped pretraining\.InICLR,External Links:[Link](https://www.arxiv.org/abs/2509.15248)Cited by:[§4](https://arxiv.org/html/2610.31867#S4.SS0.SSS0.Px3.p1.1)\.
- Zellerset al\.\(2019\)R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. ChoiHellaSwag: can a machine really finish your sentence?\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,External Links:[Link](https://arxiv.org/abs/1905.07830)Cited by:[§3\.1](https://arxiv.org/html/2610.31867#S3.SS1.SSS0.Px4.p1.1)\.
- Zhipu AI \(2026\)Zhipu AIGLM\-5\.2\.Note:[https://github\.com/zai\-org](https://github.com/zai-org)Cited by:[§F\.1](https://arxiv.org/html/2610.31867#A6.SS1.p2.1),[Appendix F](https://arxiv.org/html/2610.31867#A6.p1.1),[§3\.3](https://arxiv.org/html/2610.31867#S3.SS3.SSS0.Px2.p1.1)\.
- Zhouet al\.\(2023\)C\. Zhou, P\. Liu, P\. Xu, S\. Iyer, J\. Sun, Y\. Mao, X\. Ma, A\. Efrat, P\. Yu, L\. Yu, S\. Zhang, G\. Ghosh, M\. Lewis, L\. Zettlemoyer, and O\. LevyLIMA: less is more for alignment\.External Links:2305\.11206,[Link](https://arxiv.org/abs/2305.11206)Cited by:[Appendix F](https://arxiv.org/html/2610.31867#A6.p1.1)\.

## Appendix Roadmap

Setup and evaluation

- •[Data sources](https://arxiv.org/html/2610.31867#A1)\(pageA\)
- •[Model architecture](https://arxiv.org/html/2610.31867#A2)\(pageB\)
- •[Hyper\-parameter recipe](https://arxiv.org/html/2610.31867#A3)\(pageC\)
- •[Evaluation tasks and subtasks](https://arxiv.org/html/2610.31867#A4)\(pageD\)
- •[Mixture construction](https://arxiv.org/html/2610.31867#A9)\(pageI\)

Data\-limited experiments

- •[Rewriter capacity](https://arxiv.org/html/2610.31867#A5)\(pageE\)
- •[Checkpoint selection with an LM judge](https://arxiv.org/html/2610.31867#A6.SS1)\(pageF\.1\)
- •[Alignment ablation](https://arxiv.org/html/2610.31867#A7)\(pageG\)
- •[Complete budget grid](https://arxiv.org/html/2610.31867#A8)\(pageH\)
- •[Full\-budget U\-curves](https://arxiv.org/html/2610.31867#A8.F11)\(Figure11, page11\)
- •[Domain loss across target sizes](https://arxiv.org/html/2610.31867#A8.SS1)\(AppendixH\.1, pageH\.1\)
- •[Best\-checkpoint evaluation](https://arxiv.org/html/2610.31867#A8.SS2)\(pageH\.2\)

Rewrite analysis

- •[Rewrite generation and cost](https://arxiv.org/html/2610.31867#A10)\(pageJ\)
- •[Style and line\-content ablations](https://arxiv.org/html/2610.31867#A11)\(pageK\)
- •[Qualitative rewrite examples](https://arxiv.org/html/2610.31867#A15)\(pageO\)
- •[Rewriting scaling law details](https://arxiv.org/html/2610.31867#A12)\(pageL\)
- •[Allocation dependence onRR](https://arxiv.org/html/2610.31867#A13)\(pageM\)

Compute\-limited experiments

- •[Experimental grid](https://arxiv.org/html/2610.31867#A14)\(pageN\)
- •[Full\-RRIsoFLOP surface](https://arxiv.org/html/2610.31867#A14.F14)\(Figure14, page14\)
- •[Per\-RRfrontier and checkpoint check](https://arxiv.org/html/2610.31867#A14.F15)\(Figures15–16, page15\)
- •[Infinite\-compute extrapolation](https://arxiv.org/html/2610.31867#A14.SS1.SSS1)\(pageN\.1\.1\)
- •[Fits and forward forecasts](https://arxiv.org/html/2610.31867#A14.SS1.SSS4)\(pageN\.1\.4\)
- •[Curvature ablation](https://arxiv.org/html/2610.31867#A14.SS1.SSS6)\(pageN\.1\.6\)
- •[Domain\-specific optima](https://arxiv.org/html/2610.31867#A14.SS2)\(pageN\.2\)
- •[Evaluations of the102210^\{22\}\-FLOP runs](https://arxiv.org/html/2610.31867#A14.SS3)\(pageN\.3\)
- •[Break\-even allocations](https://arxiv.org/html/2610.31867#A14.SS4)\(pageN\.4\)
- •[Loss\- versus accuracy\-based fits](https://arxiv.org/html/2610.31867#A14.SS5)\(pageN\.5\)
- •[Surface\-specification ablations](https://arxiv.org/html/2610.31867#A14.SS6)\(pageN\.6\)

DCLM diagnostics

- •[DCLM IsoFLOP ladders](https://arxiv.org/html/2610.31867#A14.F28)\(Figure28, page28\)
- •[DCLM vs\. FineWeb\-Edu \(R=0R=0\)](https://arxiv.org/html/2610.31867#A14.F29)\(Figure29, page29\)
- •[Source quality: DCLM rewrite sweep and transfer](https://arxiv.org/html/2610.31867#A14.SS7)\(pageN\.7\)

## Appendix AData Sources

##### Natural text\.

The primary experiments use FineWeb\-Edu\([Penedo et al\., 2024](https://arxiv.org/html/2610.31867#bib.bib19)\)\. We drop every document whose WARC file is dated 2023 or later, to limit LM\-generated text in the natural corpus, and shuffle the remaining documents\. The data\-limited experiments fix a55B\-token seed set from this corpus\. The compute\-limited experiments sample fresh documents at each training\-token budget\. The source\-quality control replaces FineWeb\-Edu with DCLM\([Li et al\., 2024](https://arxiv.org/html/2610.31867#bib.bib6)\)\.

##### Rewritten text\.

The compute\-limited experiments draw rewrites from FinePhrase\([Niklaus et al\., 2026](https://arxiv.org/html/2610.31867#bib.bib21)\), which contains approximately338338M rewrites of FineWeb\-Edu documents in each of the FAQ, math, table, and tutorial styles\. The data\-limited experiments generate rewrites of the seed set with size\-matched bootstrapped rewriters \(Appendix[J](https://arxiv.org/html/2610.31867#A10)\)\.

## Appendix BModel Architecture

All runs use the dense Qwen3 architecture, the Llama 3 tokenizer\([Grattafiori et al\., 2024](https://arxiv.org/html/2610.31867#bib.bib26)\), and a sequence length of40964096\. The hidden dimensiondddetermines each model size\. The MLP intermediate dimension is4​d4d\. Attention heads have dimension128128, givingd/128d/128query and key–value heads\. The log\-corrected depth\-to\-width ratio of[Levine et al\. \(2021\)](https://arxiv.org/html/2610.31867#bib.bib20)sets the number of layers:

L⁡\(d\)=round⁡\(d55\+4​log2​d\)\.\\displaystyle L\(d\)=\\operatorname\{round\}\\left\(\\frac\{d\}\{55\+4\\log\_\{2\}d\}\\right\)\.\(4\)This rule assigns1111layers atd=1024d=1024and2121layers atd=2048d=2048\. Appendix[C](https://arxiv.org/html/2610.31867#A3)specifies the candidate widths and selection rule\.

## Appendix CHyper\-parameter Tuning

The Completed\(d\)P recipe of[Mlodozeniec et al\. \(2026\)](https://arxiv.org/html/2610.31867#bib.bib25), implemented in Marin\([Held, 2026](https://arxiv.org/html/2610.31867#bib.bib24)\), scales one reference configuration to each run’s batch sizeBbatchB\_\{\\mathrm\{batch\}\}and training\-token countDD\. We measureBbatchB\_\{\\mathrm\{batch\}\}in sequences of40964096tokens\.

##### Optimizer and schedule\.

We train with Adam and Hyperball\([Wen et al\., 2026](https://arxiv.org/html/2610.31867#bib.bib22)\);[Wen et al\. \(2026\)](https://arxiv.org/html/2610.31867#bib.bib22)show that Hyperball makes the optimal learning rate robust across model sizes\. Following[Held \(2026\)](https://arxiv.org/html/2610.31867#bib.bib24), we tune the learning rates on the156156M\-parameter reference model with Paloma as the objective, then scale the learning rates and the remaining hyper\-parameters to each run’s batch size and token count\. Every run uses linear warm\-up over the first10%10\\%of steps and linear decay to zero\.

##### Reference point\.

The reference configuration is ad=512d=512\(156156M\-parameter\) model trained withBbatch,0=64B\_\{\\mathrm\{batch\},0\}=64andD0=2\.5×109D\_\{0\}=2\.5\\times 10^\{9\}tokens on FineWeb\-Edu\. Its hyper\-parameters are the best of a Gaussian\-process search over430430configurations at this scale, selected on Paloma validation macro cross\-entropy as in Delphi\([Held, 2026](https://arxiv.org/html/2610.31867#bib.bib24)\): Hyperball learning rateη0=6\.30×10−3\\eta\_\{0\}=6\.30\\times 10^\{\-3\}, Adam learning rateη0A=6\.56×10−4\\eta^\{\\mathrm\{A\}\}\_\{0\}=6\.56\\times 10^\{\-4\},ϵ0=1\.85×10−8\\epsilon\_\{0\}=1\.85\\times 10^\{\-8\},β1=0\.9\\beta\_\{1\}=0\.9, andβ2,0=0\.9999\\beta\_\{2,0\}=0\.9999\.

##### Scaling rules\.

Withr/r0=\(Bbatch​D0\)/\(Bbatch,0​D\)r/r\_\{0\}=\(B\_\{\\mathrm\{batch\}\}\\,D\_\{0\}\)/\(B\_\{\\mathrm\{batch\},0\}\\,D\), a run with batch sizeBbatchB\_\{\\mathrm\{batch\}\}andDDtokens uses

η\\displaystyle\\eta=min⁡\(0\.01,η0​Bbatch/Bbatch,0​\(D0/D\)0\.3\),\\displaystyle=\\min\\left\(0\.01,\\;\\eta\_\{0\}\\sqrt\{B\_\{\\mathrm\{batch\}\}/B\_\{\\mathrm\{batch\},0\}\}\\,\(D\_\{0\}/D\)^\{0\.3\}\\right\),\(5\)ηA\\displaystyle\\eta^\{\\mathrm\{A\}\}=min⁡\(0\.01,η0A​r/r0\),\\displaystyle=\\min\\left\(0\.01,\\;\\eta^\{\\mathrm\{A\}\}\_\{0\}\\sqrt\{r/r\_\{0\}\}\\right\),\(6\)ϵ\\displaystyle\\epsilon=ϵ0​r0/r,\\displaystyle=\\epsilon\_\{0\}\\sqrt\{r\_\{0\}/r\},\(7\)β2\\displaystyle\\beta\_\{2\}=clip⁡\(β2,0Bbatch/Bbatch,0,0\.9,0\.9999\)\.\\displaystyle=\\operatorname\{clip\}\\left\(\\beta\_\{2,0\}^\{\\,B\_\{\\mathrm\{batch\}\}/B\_\{\\mathrm\{batch\},0\}\},\\;0\.9,\\;0\.9999\\right\)\.\(8\)Theβ2\\beta\_\{2\}rule keeps its half\-life constant in tokens\. Every run uses gradient clipping at0\.10\.1,zz\-loss weight10−710^\{\-7\}, linear warm\-up over the first10%10\\%of steps, and linear decay to zero\.

##### Batch size\.

Each run targets2162^\{16\}steps\. We roundD/\(216⋅4096\)D/\(2^\{16\}\\cdot 4096\)up to a power of two, then halve the batch size untilη<0\.01\\eta<0\.01,ηA<0\.01\\eta^\{\\mathrm\{A\}\}<0\.01, andβ2\>0\.9\\beta\_\{2\}\>0\.9, subject to8≤Bbatch≤81928\\leq B\_\{\\mathrm\{batch\}\}\\leq 8192\.

##### Candidate models per budget\.

For each training budgetCC, we enumerate hidden dimensions from512512in increments of128128whenC≤2×1019C\\leq 2\\times 10^\{19\}FLOPs and256256otherwise\. Each architecture follows Appendix[B](https://arxiv.org/html/2610.31867#A2)and receivesD=C/\(3​Ffwd\)D=C/\(3F\_\{\\mathrm\{fwd\}\}\)tokens, whereFfwdF\_\{\\mathrm\{fwd\}\}denotes forward FLOPs per token\. Budgets are labeled by this nominalCC; the scaling\-law fits and all compute accounting useC=6​N​DC=6NDas in Section[2](https://arxiv.org/html/2610.31867#S2)\. We exclude models above12​B⋅max⁡\(1,C/\(3×1020\)\)12\\mathrm\{B\}\\cdot\\max\\\!\\left\(1,\\sqrt\{C/\(3\\times 10^\{20\}\)\}\\right\)parameters or250250tokens per parameter\.

## Appendix DEvaluation Details

We evaluate language\-model fit with Paloma\([Magnusson et al\., 2024](https://arxiv.org/html/2610.31867#bib.bib18)\)and question answering, mathematics, and code with offline evaluation harnesses\.444The SmolLM2 Paloma reference uses OLMES \([https://github\.com/allenai/olmes](https://github.com/allenai/olmes)\)\. The domain\-likelihood suites use OLMo Eval \([https://github\.com/allenai/olmo\-eval](https://github.com/allenai/olmo-eval)\)\.Unless stated otherwise, each metric is bits per byte \(BPB\): reference\-continuation negative log\-likelihood in bits divided by UTF\-8 byte length\. Lower is better\.

##### Paloma\.

Paloma macro BPB averages 16 sources uniformly: 4chan, C4 100 Domains, C4 English, Dolma v1\.5, Dolma 100 Programming Languages, Dolma 100 Subreddits, Falcon RefinedWeb, Gab, M2D2 S2ORC, M2D2 Wikipedia, Manosphere, mC4, Penn Treebank, RedPajama, TwitterAAE, and WikiText\-103\. Both settings use the training\-loop evaluator at the models’ trained40964096\-token context and pool bytes within each source\. We evaluate SmolLM2 at a forced40964096\-token context and match the within\-source pooling for the rewriter comparison\.

##### Question answering\.

The QA suite contains ARC\-Easy, ARC\-Challenge, HellaSwag, and all 57 MMLU subjects\. We first average the two ARC tasks and the 57 MMLU subjects within their families, then average ARC, HellaSwag, and MMLU uniformly\. The MMLU subjects are:

- •*STEM:*abstract algebra, anatomy, astronomy, college biology, college chemistry, college computer science, college mathematics, college physics, computer security, conceptual physics, electrical engineering, elementary mathematics, high\-school biology, high\-school chemistry, high\-school computer science, high\-school mathematics, high\-school physics, high\-school statistics, machine learning, and medical genetics;
- •*humanities:*formal logic, high\-school European history, high\-school US history, high\-school world history, international law, jurisprudence, logical fallacies, moral disputes, moral scenarios, philosophy, prehistory, professional law, and world religions;
- •*social sciences:*econometrics, high\-school geography, high\-school government and politics, high\-school macroeconomics, high\-school microeconomics, high\-school psychology, human sexuality, professional psychology, public relations, security studies, sociology, and US foreign policy; and
- •*other:*business ethics, clinical knowledge, global facts, human aging, management, marketing, college medicine, miscellaneous, nutrition, professional accounting, professional medicine, and virology\.

Rank classification scores every answer choice\. The scaling surfaces use gold\-answer BPB \(qa\_rc:bpb:base\_easy\)\. The secondary metric is the fraction of examples on which the gold answer ranks first \(qa\_rc:base\_easy\)\.

##### Mathematics\.

The Math metricmath\_bpb:alluniformly averages nine gold\-answer BPB tasks: GSM8K\-Platinum, TheoremQA, and the seven Minerva MATH subjects \(algebra, counting and probability, geometry, intermediate algebra, number theory, prealgebra, and precalculus\)\.

##### Code\.

The Code suite contains three\-shot HumanEval, three\-shot MBPP, and three\-shot multilingual MBPP\-v2fix in Bash, C, C\+\+, C\#, Go, Haskell, Java, JavaScript, MATLAB, PHP, Python, R, Ruby, Rust, Scala, Swift, and TypeScript\. We first average the 17 multilingual variants, then average HumanEval, MBPP, and multilingual MBPP uniformly \(code\_bpb:base\_easy\)\. The metric is reference\-code BPB, not execution accuracy\.

## Appendix EData Limited Base Rewriters

Figure 7:11B and33B targets at matched total compute\. Each panel pairs the nearest budgets on the grid, with FLOP ratios0\.970\.97,0\.970\.97, and1\.091\.09\.Figure 8:Rewriter capacity\.Left:11B target loss atW=12W=12with rewrites from the550550M,11B, and33B rewriters\.Right:share of the11B–33B target gap closed by training the11B target on33B rewrites, at each budget’s best mixture\.##### Effect of rewriter capacity\.

The capacity control trains the11B target on rewrites from the550550M and33B rewriters atW≤12W\\leq 12\. AtW=12W=12, the550550M rewriter sits0\.0110\.011BPB below the33B rewriter atR=0\.13R=0\.13and trails it by0\.0690\.069BPB atR=1R=1\(Figure[8](https://arxiv.org/html/2610.31867#A5.F8), left\)\. Training the11B target on33B rewrites closes52%52\\%of the11B–33B target gap atR=1R=1andW=12W=12, and between−4%\-4\\%and22%22\\%at each budget’s best mixture \(right\)\. Figure[7](https://arxiv.org/html/2610.31867#A5.F7)pairs the11B and33B targets at the nearest total\-compute budgets on the grid\. The33B target has lower BPB in1717of the1818pairs, by0\.0310\.031BPB on average at the smallest budget pair and0\.0880\.088BPB at the largest\.

## Appendix FRewriter Instruction Tuning

Each data\-limited rewriter receives supervised fine\-tuning \(SFT\) after pretraining on the seed set\. The SFT set contains40004000examples: GLM\-5\.2\([Zhipu AI, 2026](https://arxiv.org/html/2610.31867#bib.bib45)\)rewrites of10001000seed\-set documents under each of the four FinePhrase prompts,77M tokens in total, at sequence length81928192\. Following[Zhou et al\. \(2023\)](https://arxiv.org/html/2610.31867#bib.bib11), we train with a batch size of3232examples and linear decay of the learning rate to zero, with no warm\-up\. We sweep the learning rate and the number of epochs per rewriter size and select the checkpoint with the LM judge of Appendix[F\.1](https://arxiv.org/html/2610.31867#A6.SS1)\.

### F\.1Checkpoint Selection with an LM Judge

The in\-house rewriter generates the synthetic data for every data\-limited run \(Section[3\.3](https://arxiv.org/html/2610.31867#S3.SS3)\), so we choose its fine\-tuning learning rate and number of epochs by the quality of the rewrites each checkpoint generates\. At550550M and837837M the checkpoints are the1212from learning rates10−510^\{\-5\}to5×10−45\\times 10^\{\-4\}at1010or1515epochs, at33B the six from5×10−55\\times 10^\{\-5\}to2×10−42\\times 10^\{\-4\}, and at11B a shortlist of seven of3131fine\-tuning runs that also covers55epochs \(Table[2](https://arxiv.org/html/2610.31867#A6.T2)\)\. Every checkpoint rewrites the same100100randomly sampled FineWeb\-Edu documents under the four FinePhrase prompts,400400rewrites per checkpoint\. The reference for each document and prompt is its FinePhrase rewrite, generated by SmolLM2\-1\.7B, the rewriter of the compute\-limited setting\.

GLM\-5\.2\([Zhipu AI, 2026](https://arxiv.org/html/2610.31867#bib.bib45)\)and MiniMax\-M2\.7\([MiniMax, 2026](https://arxiv.org/html/2610.31867#bib.bib46)\)judge pairs of rewrites of the same document under the same prompt\. A pair is either a checkpoint’s rewrite and the SmolLM2\-1\.7B rewrite or the rewrites of two checkpoints, and not every pair is judged\. A judge receives the source document, the rewrite prompt, and the two rewrites, and returns\+1\+1if the first rewrite is better,−1\-1if the second is better, or00for a tie \(prompt below\)\. Each pair is judged in both presentation orders\. Elo ratings computed from each judge’s preferences place every checkpoint and SmolLM2\-1\.7B on one scale, and the rating difference gives each checkpoint’s expected win rate against SmolLM2\-1\.7B\. A checkpoint’s score is this win rate averaged over the two judges\.

Judge promptCompare how well the two rewrites rewrite the source document according to the style specified in the rewrite prompt\.Source document:\{source\}Rewrite prompt:\{prompt\}Rewrite A:\{rewrite\_a\}Rewrite B:\{rewrite\_b\}Return \+1 if A is better, \-1 if B is better, or 0 for a tie\.

Table[2](https://arxiv.org/html/2610.31867#A6.T2)ranks the checkpoints by score and marks the checkpoint that generates the seed\-set rewrites at each size: a learning rate of10−410^\{\-4\}for1515epochs at550550M and11B,2×10−42\\times 10^\{\-4\}for1010epochs at837837M, and5×10−55\\times 10^\{\-5\}for1515epochs at33B\. This is the highest\-scoring checkpoint at550550M,837837M, and33B\. At11B it scores0\.70\.7points below the highest\-scoring checkpoint, and the confidence intervals of the two overlap\. No checkpoint reaches a50%50\\%win rate against SmolLM2\-1\.7B\.

SizeSFT learning rateEpochsMean win rate \(%\)550M10−410^\{\-4\}15†28\.32×10−42\\times 10^\{\-4\}1527\.72×10−42\\times 10^\{\-4\}1026\.610−410^\{\-4\}1025\.02×10−52\\times 10^\{\-5\}1524\.82×10−52\\times 10^\{\-5\}1023\.85×10−55\\times 10^\{\-5\}1023\.25×10−45\\times 10^\{\-4\}1022\.65×10−45\\times 10^\{\-4\}1522\.35×10−55\\times 10^\{\-5\}1521\.910−510^\{\-5\}1519\.910−510^\{\-5\}1019\.4837M2×10−42\\times 10^\{\-4\}10†32\.52×10−42\\times 10^\{\-4\}1531\.510−410^\{\-4\}1530\.910−410^\{\-4\}1029\.75×10−55\\times 10^\{\-5\}1528\.02×10−52\\times 10^\{\-5\}1026\.45×10−55\\times 10^\{\-5\}1026\.32×10−52\\times 10^\{\-5\}1525\.35×10−45\\times 10^\{\-4\}1525\.05×10−45\\times 10^\{\-4\}1024\.410−510^\{\-5\}1523\.210−510^\{\-5\}1020\.91B2×10−42\\times 10^\{\-4\}1033\.310−410^\{\-4\}15†32\.610−410^\{\-4\}1031\.72×10−42\\times 10^\{\-4\}531\.710−410^\{\-4\}530\.72×10−42\\times 10^\{\-4\}1530\.65×10−55\\times 10^\{\-5\}1530\.33B5×10−55\\times 10^\{\-5\}15†38\.810−410^\{\-4\}1038\.55×10−55\\times 10^\{\-5\}1038\.410−410^\{\-4\}1537\.42×10−42\\times 10^\{\-4\}1034\.52×10−42\\times 10^\{\-4\}1533\.9Table 2:Win rates against the SmolLM2\-1\.7B FinePhrase rewrite, averaged over both judges\.Each row is a rewriter checkpoint\. Scores are the Elo\-derived win rate against the SmolLM2\-1\.7B rewrite of the same document and prompt, averaged over GLM\-5\.2 and MiniMax\-M2\.7\.†\\daggermarks the checkpoint that generates the seed\-set rewrites\. Rows are ordered by score within each size\. The11B results cover seven shortlisted configurations\.

## Appendix GAlignment

The data\-limited rewriters receive supervised fine\-tuning \(SFT\) on GLM\-5\.2 rewrites and LM\-judge checkpoint selection \(Appendix[F](https://arxiv.org/html/2610.31867#A6)\)\. The alignment control instead follows the SmolLM2 post\-training procedure\([Ben Allal et al\., 2025](https://arxiv.org/html/2610.31867#bib.bib16)\): SFT followed by direct preference optimization \(DPO\) on the11B rewriter, with no LM\-judge selection\. Across the1515runs withR\>0R\>0atW∈\{3,6,9\}W\\in\\\{3,6,9\\\}, SFT\+DPO lowers Paloma macro BPB in all1515and by0\.0460\.046BPB on average \(Figure[9](https://arxiv.org/html/2610.31867#A7.F9)\)\. The programming\-languages and RedPajama sources account for80%80\\%of the reduction, and BPB increases on88of the1616Paloma sources \(Figure[10](https://arxiv.org/html/2610.31867#A7.F10)\)\.

Figure 9:Final\-checkpoint Paloma macro BPB for the11B target trained on rewrites from the SFT and SFT\+DPO rewriters atW∈\{3,6,9\}W\\in\\\{3,6,9\\\}\. Both conditions share the natural\-onlyR=0R=0runs\.Figure 10:Mean Paloma BPB change per source from replacing the SFT rewriter with the SFT\+DPO rewriter across the1616mixed runs \(0<R<10<R<1,W≤12W\\leq 12\)\. Orange bars denote lower BPB under SFT\+DPO and blue bars lower BPB under the SFT rewriter, matching the colors of Figure[9](https://arxiv.org/html/2610.31867#A7.F9)\.
## Appendix HData\-Limited Experimental Results

Figure[11](https://arxiv.org/html/2610.31867#A8.F11)reports the full data\-limited grid: four target sizes \(550550M,837837M,11B, and33B parameters\), eight epoch budgetsW∈\{3,6,9,12,15,18,21,24\}W\\in\\\{3,6,9,12,15,18,21,24\\\}, and six rewrite fractionsR∈\{0,0\.13,0\.29,0\.47,0\.71,1\}R\\in\\\{0,0\.13,0\.29,0\.47,0\.71,1\\\}\. Table[3](https://arxiv.org/html/2610.31867#A8.T3)gives the total compute, accounting for rewrites, for every size and budget\. The rewriter generates every rewrite token a run trains on, and no rewrite token is trained on twice, so the repetition penalty in Eq\. \([14](https://arxiv.org/html/2610.31867#A12.E14)\) applies to the natural seed set only\. Figure[4](https://arxiv.org/html/2610.31867#S3.F4)in the main text showsW∈\{3,6,12,24\}W\\in\\\{3,6,12,24\\\}\.

Table 3:Total compute, accounting for rewrites, for the complete data\-limited grid, in units of102010^\{20\}FLOPs\. Each compute value has six rewrite\-fraction runs\.Figure 11:Final\-checkpoint Paloma macro BPB for all192192data\-limited runs\. Each panel shows one target size at all eight epoch budgetsWWand six rewrite fractionsRR\. Open pentagons mark the best observed mixture at each size and budget\.### H\.1Domain Loss Across Target Sizes

Figure[12](https://arxiv.org/html/2610.31867#A8.F12)repeats Figure[6](https://arxiv.org/html/2610.31867#S3.F6)for every target size\. The best observed rewrite fraction on QA is00atW≤6W\\leq 6for the33B target, atW≤9W\\leq 9for the11B and837837M targets, and at every budget but one for the550550M target\. On Math and Code it is interior at every size fromW=3W=3\. The QA, Math, and Code values come from our evaluation harness \(Appendix[D](https://arxiv.org/html/2610.31867#A4)\)\.

Figure 12:Final\-checkpoint BPB against the epoch budgetWWfor every target size \(rows\) on the QA, Math, and Code suites \(columns\)\. Series as in Figure[5](https://arxiv.org/html/2610.31867#S3.F5)\.
### H\.2Best\-Checkpoint Evaluation

Section[3\.3](https://arxiv.org/html/2610.31867#S3.SS3)reports final checkpoints\. The best\-checkpoint analysis instead selects, for each run, the saved checkpoint with the lowest Paloma validation loss whenever it improves on the final checkpoint by at least0\.010\.01nats, and evaluates Paloma, QA, Math, and Code at that checkpoint\. Paloma alone determines the selection\. Selection changes the best observed rewrite fraction on Paloma in44of the3232size–budget settings\.

Figure[13](https://arxiv.org/html/2610.31867#A8.F13)shows the33B target atW∈\{3,6,12,24\}W\\in\\\{3,6,12,24\\\}\. AtW=24W=24, selection lowers natural\-only Paloma BPB from1\.5121\.512to1\.2411\.241, below pure rewriting at1\.2641\.264\. The lowest Paloma BPB in the figure is atW=24W=24andR=0\.71R=0\.71\. QA and Code reach their minima atW=24W=24andR=0\.47R=0\.47, and Math ties atR=0\.13R=0\.13andR=0\.47R=0\.47after rounding to two decimals\.

![Refer to caption](https://arxiv.org/html/2610.31867v1/de_best_checkpoint_evals.png)Figure 13:Best\-checkpoint BPB for the33B target by epoch budgetWWand rewrite fractionRR\. Hatching marks runs re\-evaluated at an earlier checkpoint whose Paloma validation loss is at least0\.010\.01nats below the final checkpoint\. Magenta outlines mark row minima, including ties after rounding to two decimals\. Paloma alone selects each checkpoint\.

## Appendix IMixture Construction

A mixture at rewrite fractionRRdraws its natural tokens from the filtered FineWeb\-Edu documents \(Appendix[A](https://arxiv.org/html/2610.31867#A1)\) and its rewritten tokens from the FinePhrase rewrites of those documents, pooled across the math, FAQ, table, and tutorial styles and sampled by tokens\. We train with Levanter\([Hall et al\., 2023](https://arxiv.org/html/2610.31867#bib.bib44)\)and build every mixture with its mixture dataset\. The natural corpus and the rewrite pool enter as two components with weights1−R1\-RandRR\. Levanter partitions the training stream into fixed\-size blocks and assigns the examples of each block to the two components in proportion to their weights, so every block\-aligned prefix of the stream has rewrite fractionRRup to rounding\. The assignment is a deterministic function of the seed, so runs at the sameRRsee the same interleaving\.

## Appendix JGenerating Rewrites

##### Compute\-limited generation\.

SmolLM2\-1\.7B generates the FinePhrase rewrites\. We account for each rewritten training token as one input pass over its source document and one output pass over the rewrite,4​Nr4N\_\{r\}FLOPs withNr=1\.71N\_\{r\}=1\.71B \(Eq\.[12](https://arxiv.org/html/2610.31867#A12.E12)\)\.

##### Data\-limited generation\.

Each target size uses a rewriter of the same size, pretrained on the seed set and instruction\-tuned at sequence length81928192on GLM\-5\.2 rewrites of10001000seed\-set documents under the four FinePhrase prompts \(40004000examples\)\. The rewriter then rewrites the seed set under the same prompts at temperature1\.01\.0\. Appendix[L\.2](https://arxiv.org/html/2610.31867#A12.SS2)converts the generated volume to the rewrite fractionRR\.

We generate the synthetic pool before target\-model training by repeatedly sampling fresh rewrites of the same seed documents under the same four FinePhrase prompts at temperature1\.01\.0\. The pool remains fixed during training: we do not regenerate rewrites between training epochs\. Each synthetic token used by a run is seen once, while natural tokens may be repeated\.

## Appendix KRewrite\-Style and Line\-Content Ablations

We sample6,0006\{,\}000documents from each FinePhrase variant and classify every line as prose, list, table, header, or code \(Table[4](https://arxiv.org/html/2610.31867#A11.T4)\)\. Table rewrites are59%59\\%table lines\. Tutorial rewrites have the largest code share,0\.80%0\.80\\%of lines\. This classification measures surface form and not faithfulness to the source document\.

Table 4:Line composition of6,0006\{,\}000documents per FinePhrase variant, in percent of classified lines\.##### Style contrasts\.

The content ablation trains one run per arm and rewrite fraction, for2020arms and1010fractions\. Four style\-only arms keep one rewrite style, and four style\-out arms remove one style\. FAQ style\-only has higher Paloma BPB than FAQ style\-out at all1010fractions, by0\.02260\.0226BPB on average\. Math style\-only is higher at77of1010fractions \(\+0\.0032\+0\.0032BPB on average\)\. Table style\-only is lower at all1010fractions \(−0\.0039\-0\.0039BPB\), and tutorial style\-only is lower at88of1010\(−0\.0038\-0\.0038BPB\)\.

##### Line\-type contrasts\.

Five arms remove one line type from every rewrite: code, headers, lists, prose, or tables\. We compare each against the full mixture at the nine fractions belowR=0\.9R=0\.9and exclude the full\-mixture run atR=0\.9R=0\.9\. Every removal raises mean BPB, by0\.01200\.0120to0\.02550\.0255BPB, with the largest increase from removing prose\.

## Appendix LRewriting Scaling Law Details

This section collects the derivations and the per\-setting instantiations of the rewriting scaling law \(Section[2](https://arxiv.org/html/2610.31867#S2)\), Eq\. \([1](https://arxiv.org/html/2610.31867#S2.E1)\)\.

##### Mixture\-dependent loss floor\.

We fitLLin bits per byte; every statement below holds in any logarithmic base, withλn\\lambda\_\{n\}andλs\\lambda\_\{s\}in Eq\. \([10](https://arxiv.org/html/2610.31867#A12.E10)\) rescaled accordingly\.E⁡\(R\)E\(R\)is the irreducible loss of the mixture: the cross\-entropy that a model with unlimited capacity and data would reach on the evaluation distribution when trained on a mixture of\(1−R\)\(1\-R\)natural andRRrewritten data\. We writepep\_\{e\}for the evaluation distribution,pnp\_\{n\}andpsp\_\{s\}for the natural and rewrite distributions, andpR=\(1−R\)​pn\+R​psp\_\{R\}=\(1\-R\)\\,p\_\{n\}\+R\\,p\_\{s\}for the training mixture’s distribution\. A model with unlimited capacity and data converges topRp\_\{R\}, so

E\(R\)=H\(pe\)\+KL\(pe∥pR\)\.E\(R\)=H\(p\_\{e\}\)\+\\mathrm\{KL\}\\\!\\left\(p\_\{e\}\\,\\\|\\,p\_\{R\}\\right\)\.\(9\)Note that only the KL divergence depends onRR\. BecauseKL\(p∥q\)\\mathrm\{KL\}\(p\\,\\\|\\,q\)is convex inqqandpRp\_\{R\}is linear inRR,E⁡\(R\)E\(R\)is convex inRRfor every evaluation distribution\. Evaluated on natural text \(pe=pnp\_\{e\}=p\_\{n\}\) it is minimized atR=0R=0and increases withRR; evaluated on the rewrites themselves \(pe=psp\_\{e\}=p\_\{s\}\) the divergence falls to zero asR→1R\\to 1; evaluated on a distribution that neither source matches exactly, such as Paloma, the minimizer can be interior\. A two\-component approximation to the divergence \(Appendix[N\.1\.2](https://arxiv.org/html/2610.31867#A14.SS1.SSS2)\) gives the parametric form

E⁡\(R\)=E0−λn​log⁡\(1−\(1−ρn\)​R\)−λs​log⁡\(ρs\+\(1−ρs\)​R\)\.E\(R\)=E\_\{0\}\-\\lambda\_\{n\}\\log\\\!\\left\(1\-\(1\-\\rho\_\{n\}\)R\\right\)\-\\lambda\_\{s\}\\log\\\!\\left\(\\rho\_\{s\}\+\(1\-\\rho\_\{s\}\)R\\right\)\.\(10\)Appendix[N\.1\.2](https://arxiv.org/html/2610.31867#A14.SS1.SSS2)derives Eq\. \([10](https://arxiv.org/html/2610.31867#A12.E10)\) and interpretsλn,ρn,λs,ρs\\lambda\_\{n\},\\rho\_\{n\},\\lambda\_\{s\},\\rho\_\{s\}as the share of the evaluation distribution each source covers better and the other source’s relative density there\. The first term is the cost of*removing*natural text and the second the benefit of*adding*rewrites\. The minimizerR∞∗=arg​minR⁡E​\(R\)R^\{\*\}\_\{\\infty\}=\\argmin\_\{R\}E\(R\), the floor\-optimal rewrite fraction, is unique because Eq\. \([10](https://arxiv.org/html/2610.31867#A12.E10)\) is strictly convex for positiveλn,λs\\lambda\_\{n\},\\lambda\_\{s\}, and is interior whenE′​\(0\)<0<E′​\(1\)E^\{\\prime\}\(0\)<0<E^\{\\prime\}\(1\)\(Appendix[N\.1\.2](https://arxiv.org/html/2610.31867#A14.SS1.SSS2)\)\.

##### Repetition penalty\.

F⁡\(R,D,T\)=P​\(1−R\)s​\(\(1−R\)​D/T−1\)\+F\(R,D,T\)=P\\,\(1\-R\)^\{s\}\\,\\bigl\(\(1\-R\)D/T\-1\\bigr\)\_\{\+\}is the repetition penalty of[Lovelace et al\. \(2026\)](https://arxiv.org/html/2610.31867#bib.bib2), diluted by rewrites at ratess\. It is zero when\(1−R\)​D≤T\(1\-R\)D\\leq T, as in every compute\-limited run\.

##### Rewrites shift the floor only\.

RRenters Eq\. \([1](https://arxiv.org/html/2610.31867#S2.E1)\) throughE⁡\(R\)E\(R\)and, when\(1−R\)​D\>T\(1\-R\)D\>T, through the dilution factor\(1−R\)s\(1\-R\)^\{s\}and the natural epochs\(1−R\)​D/T\(1\-R\)D/Tof the repetition penalty; it does not enter the capacity and data terms\. This is the central modeling assumption of the paper: rewrites change*what*the model converges to, not*how fast*it gets there inNNandDD\. It implies that the best loss at any training budget decomposes as

L∗​\(C,R\)=Lbase∗​\(C\)\+g~​\(R\),g~​\(R\)=E⁡\(R\)−E⁡\(0\)\.L^\{\*\}\(C,R\)=L^\{\*\}\_\{\\mathrm\{base\}\}\(C\)\+\\tilde\{g\}\(R\),\\qquad\\tilde\{g\}\(R\)=E\(R\)\-E\(0\)\.\(11\)Here,L∗\(C,R\)=min6​N​D=CL\(N,D,R,T→∞\)L^\{\*\}\(C,R\)=\\min\_\{6ND=C\}L\(N,D,R,T\\to\\infty\)is the best loss over\(N,D\)\(N,D\)at training budgetCCandLbase∗​\(C\)≡L∗​\(C,0\)L^\{\*\}\_\{\\mathrm\{base\}\}\(C\)\\equiv L^\{\*\}\(C,0\)\. Figure[15](https://arxiv.org/html/2610.31867#A14.F15)\(Appendix[N\.1](https://arxiv.org/html/2610.31867#A14.SS1)\) tests how closelyN∗​\(C\)N^\{\*\}\(C\)andD∗​\(C\)D^\{\*\}\(C\)remain stable acrossRR\.

##### Accounting for rewrites\.

Section[2](https://arxiv.org/html/2610.31867#S2)gives the generation cost of4​Nr4N\_\{r\}FLOPs per rewritten training token for a rewriter withNrN\_\{r\}parameters, so

Ctotal=C\+4​Nr​R​D,C=6​N​D\.C\_\{\\mathrm\{total\}\}=C\+4N\_\{r\}RD,\\qquad C=6ND\.\(12\)We do not account for the cost of training the rewriter as it is amortized across all experiments\.

### L\.1Compute\-Limited Setting

##### Setup\.

The compute\-limited setting reflects the non\-frontier case in which natural data is plentiful and our goal is to characterize the strongest possible model subject to a fixed compute budget\. We use an external rewriter: one that we do not train ourselves, is stronger than any model that we train, and is trained with more compute than our largest training run, and we do not account for its training cost\. A mixture with rewrite fractionRRdraws a fractionRRof its training tokens from rewrites and the remaining1−R1\-Rfrom natural text\. We fit the law to IsoFLOP sweeps: at each training budgetCCwe train one grid of model sizesNNwithD=C/\(6​N\)D=C/\(6N\)tokens, and every rewrite fractionRRuses the same grid\.

##### Fitting the surface\.

We fit Eq\. \([1](https://arxiv.org/html/2610.31867#S2.E1)\) withF=0F=0to every run, since\(1−R\)​D≤T\(1\-R\)D\\leq Tfor every compute\-limited run, to obtain the floorE⁡\(R\)E\(R\), subtract the fitted shiftg~​\(R\)\\tilde\{g\}\(R\)from every run, and fit the\(N,D\)\(N,D\)backbone to the pooled runs with Chinchilla Approach 2\([Hoffmann et al\., 2022](https://arxiv.org/html/2610.31867#bib.bib3)\)\. A quadratic inlog⁡D\\log Dat each budget givesD∗​\(C\)D^\{\*\}\(C\)andLbase∗​\(C\)L^\{\*\}\_\{\\mathrm\{base\}\}\(C\), and power laws across budgets giveα\\alphaandβ\\beta\. We add an off\-frontier curvature term that vanishes with zero slope on the frontier and leavesN∗​\(C\)N^\{\*\}\(C\),D∗​\(C\)D^\{\*\}\(C\), andLbase∗​\(C\)L^\{\*\}\_\{\\mathrm\{base\}\}\(C\)unchanged\. Appendix[N\.1\.4](https://arxiv.org/html/2610.31867#A14.SS1.SSS4)gives the exponent conversion, the fitted parameters, the leave\-one\-budget\-out forecasts, and the curvature ablation\.

##### Compute\-optimal rewrite fraction\.

Under Eq\. \([11](https://arxiv.org/html/2610.31867#A12.E11)\) the floor\-optimal fractionR∞∗R^\{\*\}\_\{\\infty\}minimizes loss at any*training*budget\. Accounting for generation changes the answer: at a fixed total budget, every rewritten token is paid for twice, once to generate and once to train on, so raisingRRshrinks the training budget available\. Because natural data is plentiful in this setting, we assume an unlimited supply of natural tokens and defineL⁡\(N,D,R\)=limT→∞L⁡\(N,D,R,T\)L\(N,D,R\)=\\lim\_\{T\\to\\infty\}L\(N,D,R,T\)\. AsT→∞T\\to\\inftythe repetition penaltyF⁡\(R,D,T\)F\(R,D,T\)vanishes\. The compute\-optimal allocation minimizes this surface jointly over model size, token count, and rewrite fraction at a fixed total budget:

\(N∗,D∗,R^∗\)=arg​minN,D,R∈\[0,1\]⁡L​\(N,D,R\)s\.t\.6​N​D\+4​Nr​RD=Ctotal,\\bigl\(N^\{\*\},D^\{\*\},\\widehat\{R\}^\{\*\}\\bigr\)=\\argmin\_\{N,\\,D,\\,R\\in\[0,1\]\}L\(N,D,R\)\\quad\\text\{s\.t\.\}\\quad 6ND\+4N\_\{r\}RD=C\_\{\\mathrm\{total\}\},\(13\)where we evaluateL⁡\(N,D,R\)L\(N,D,R\)with the off\-frontier curvature term of Eq\. \([31](https://arxiv.org/html/2610.31867#A14.E31)\) added, as in the fitted surface\. Because each rewritten token also costs generation compute, the optimum withR\>0R\>0trains a slightly larger model on fewer tokens than the natural\-only allocation at the same training compute\. The generation cost is linear inDDwhile the training loss improves as a power law in compute, so the overhead of rewriting becomes relatively cheaper as budgets grow, andR^∗​\(Ctotal\)\\widehat\{R\}^\{\*\}\(C\_\{\\mathrm\{total\}\}\)increases from near00towardR^∞∗\\widehat\{R\}^\{\*\}\_\{\\infty\}\(Figure[2](https://arxiv.org/html/2610.31867#S3.F2.fig1)\)\.

### L\.2Data\-Limited Setting

##### Setup\.

The data\-limited setting reflects the frontier case in which no stronger rewriter exists, we target a fixed model size, and the amount of natural data is fixed, but compute is the plentiful resource\. A fixed seed set ofTTnatural tokens and a target model withNNparameters are given, and an internal rewriter of the same size, pretrained on the seed set, generates the rewrites\. We do not account for the rewriter’s training cost, which is shared across all data\-limited experiments at that size\. As above, budgets are marginal compute given the rewriter\. As in the main paper, we measure the total compute budget byW=Ctotal/\(6​N​T\)W=C\_\{\\mathrm\{total\}\}/\(6NT\)\. A run withDDtraining tokens and rewrite fractionRRtrains on\(1−R\)​D\(1\-R\)Dnatural tokens andR​DRDsynthetic tokens, each synthetic token seen once\. Training and generation together cost6​N​D\+4​N​R​D=6​N​T​W6ND\+4NRD=6NTW, givingD=3​T​W/\(3\+2​R\)D=3TW/\(3\+2R\)\. The run therefore passes over the natural seed set3​W​\(1−R\)/\(3\+2​R\)3W\(1\-R\)/\(3\+2R\)times\.

##### Data\-limited scaling model\.

With the target size fixed atN0N\_\{0\}, the capacity termA/N0αA/N\_\{0\}^\{\\alpha\}in Eq\. \([1](https://arxiv.org/html/2610.31867#S2.E1)\) is a constant and is absorbed into the floor\. At epoch budgetWWand rewrite fractionRR, the run seesD=3​T​W/\(3\+2​R\)D=3TW/\(3\+2R\)tokens, counting natural and synthetic tokens alike:3​W​\(1−R\)/\(3\+2​R\)3W\(1\-R\)/\(3\+2R\)epochs over the seed set and the rest fresh rewrites, none repeated\. Substituting into Eq\. \([1](https://arxiv.org/html/2610.31867#S2.E1)\) gives

LDE​\(R,W\)=E⁡\(R\)\+B\(3​T​W/\(3\+2​R\)\)β\+P​\(1−R\)s​\(3​W​\(1−R\)3\+2​R−1\)\+\.L\_\{\\mathrm\{DE\}\}\(R,W\)=E\(R\)\+\\frac\{B\}\{\\left\(3TW/\(3\+2R\)\\right\)^\{\\beta\}\}\+P\\,\(1\-R\)^\{s\}\\left\(\\frac\{3W\(1\-R\)\}\{3\+2R\}\-1\\right\)\_\{\+\}\.\(14\)AtR=0R=0the law reduces to the epoching law of[Lovelace et al\. \(2026\)](https://arxiv.org/html/2610.31867#bib.bib2)\. AtR=1R=1the repetition penalty vanishes\. In between, rewrites add fresh tokens and dilute repetition of the seed set\. We fit the law separately for each target–rewriter pair, pooling all epoch budgets and rewrite fractions, because the size\-matched rewriter changes the quality of the synthetic data with the target size \(Figure[8](https://arxiv.org/html/2610.31867#A5.F8)\)\. The minimizer of the fitted law overRRisR^∗​\(W\)\\widehat\{R\}^\{\*\}\(W\)\.

## Appendix MDependence of Efficient\-Frontier Allocation onRR

The shared surface fits one compute\-optimal token countD∗​\(C\)D^\{\*\}\(C\)across rewrite fractions\. We test this restriction on the583583runs of the five budgets with all eleven rewrite fractions\. Both fits give every budget–fraction cell its own loss level\. The restricted fit sharesD∗​\(C\)D^\{\*\}\(C\)across fractions\. The alternative adds ten fraction\-specific offsets tolog10⁡D∗\\log\_\{10\}D^\{\*\}, withR=0R=0as the reference\.

The offsets lower RMSE from0\.008310\.00831to0\.007620\.00762BPB, and the largest shift isD∗​\(R\)/D∗​\(0\)=1\.196D^\{\*\}\(R\)/D^\{\*\}\(0\)=1\.196atR=0\.8R=0\.8\. A wild bootstrap that resamples residual signs jointly within each architecture \(400400draws\) rejects equalD∗D^\{\*\}across fractions \(p=1/401p=1/401\)\. Figure[15](https://arxiv.org/html/2610.31867#A14.F15)shows that a shift of this size does not change the selected model size, and Appendix[N\.6](https://arxiv.org/html/2610.31867#A14.SS6)compares richer surface specifications on the same runs\.

## Appendix NCompute\-Limited Experimental Results

### N\.1Scaling Model Details

The FineWeb\-Edu ladders contain five training budgets from3×10183\\times 10^\{18\}to9×10199\\times 10^\{19\}FLOPs at eleven rewrite fractionsR∈\{0,0\.1,…,1\}R\\in\\\{0,0\.1,\\ldots,1\\\}, and natural\-only \(R=0R=0\) ladders at1\.8×10201\.8\\times 10^\{20\}and3×10203\\times 10^\{20\}FLOPs\. The surface fit uses these604604Paloma BPB observations and the102210^\{22\}\-FLOP hero run \(R=0\.321R=0\.321\) as a frontier anchor\. Nonzero\-RRruns at1\.8×10201\.8\\times 10^\{20\}and3×10203\\times 10^\{20\}FLOPs, the102110^\{21\}runs, and the remaining102210^\{22\}runs are validation runs outside the fit\. The forward forecasts in Appendix[N\.1\.4](https://arxiv.org/html/2610.31867#A14.SS1.SSS4)refit the surface without each held\-out budget\.

Figure 14:The sharedL⁡\(N,D,R\)L\(N,D,R\)surface as per\-RRIsoFLOP slices at all eleven rewrite fractions, in Paloma macro BPB\. Dots mark runs, open stars mark fitted frontier optima, and filled stars mark the102110^\{21\}and102210^\{22\}runs\. Labels report measured minus fitted loss as a percentage of fitted loss\.##### The frontier location does not depend onRR\.

The surface fit pools all rewrite fractions before locating the IsoFLOP optima \(Appendix[L\.1](https://arxiv.org/html/2610.31867#A12.SS1)\), so the fittedN∗​\(C\)N^\{\*\}\(C\)andD∗​\(C\)D^\{\*\}\(C\)cannot vary withRRby construction\. Figure[15](https://arxiv.org/html/2610.31867#A14.F15)tests the assumption by fitting a separate Approach\-2 parabola inlog⁡D\\log Dto each\(R,C\)\(R,C\)sweep and taking its vertex\. The5656per\-RRoptima over seven budgets lie within0\.850\.85–1\.28×1\.28\\timesof the sharedD∗​\(C\)D^\{\*\}\(C\)\(RMS10%10\\%\), and per\-RRpower laws give token exponents between0\.450\.45and0\.570\.57against the shared0\.460\.46\. The spread acrossRRat a fixed budget matches the spread of theR=0R=0ladder across budgets \(0\.850\.85–1\.03×1\.03\\times\)\.

Figure[16](https://arxiv.org/html/2610.31867#A14.F16)asks whether this spread changes the selected model size\. The grid steps by1\.21\.2–1\.4×1\.4\\timesinNN\. In2323of5454sweeps the per\-RRvertex selects a neighboring grid size, never one two steps away\. Most of these cases are at1\.8×10191\.8\\times 10^\{19\}and9×10199\\times 10^\{19\}FLOPs, where the sharedN∗N^\{\*\}falls midway between two grid sizes\. On each sweep’s own parabola, the shared choice costs at most0\.00150\.0015BPB \(0\.00020\.0002BPB at the median\), against the0\.0820\.082\-BPB floor gain atR^∞∗\\widehat\{R\}^\{\*\}\_\{\\infty\}\. The measured gap between the two grid runs \(median0\.00150\.0015, maximum0\.01080\.0108BPB\) matches the parabola residuals and does not grow withRR\.

![Refer to caption](https://arxiv.org/html/2610.31867v1/paper_frontier_by_r.png)Figure 15:Per\-RRIsoFLOP optima against the shared frontier\.Left:compute\-optimal token countDR∗​\(C\)D^\{\*\}\_\{R\}\(C\)from a separate Approach\-2 parabola for each rewrite fraction and budget, colored byRR, with the shared frontierD∗​\(C\)∝C0\.46D^\{\*\}\(C\)\\propto C^\{0\.46\}\.Right:ratio of each per\-RRoptimum to the shared frontier, by budget\.![Refer to caption](https://arxiv.org/html/2610.31867v1/paper_checkpoint_by_r.png)Figure 16:Whether fitting the frontier perRRchanges the selected model size\.Left:for each sweep with a full rewrite ladder, the grid size nearest the per\-RRvertex relative to the one nearest the sharedN∗​\(C\)N^\{\*\}\(C\)\. Cells report the BPB cost of the shared choice on that sweep’s own IsoFLOP parabola\.Right:the same cost by rewrite fraction, on the parabola \(filled\) and between the two measured grid runs \(open\)\.
#### N\.1\.1Do Rewrites Still Help in the Infinite\-Compute Limit?

Figure[17](https://arxiv.org/html/2610.31867#A14.F17.fig1)shows the measured evidence that rewrites shift the floor independently of scale\. After subtracting each budget’sR=0R=0value,8282IsoFLOP\-optimal observations from nine budgets collapse onto one curveg~​\(R\)\\tilde\{g\}\(R\)with RMSE0\.00440\.0044BPB\.

Figure 17:Rewrite gains collapse onto one curve across3,333×3\{,\}333\\timescompute\.IsoFLOP\-optimal Paloma BPB after subtracting each budget’sR=0R=0value\. The fittedg~​\(R\)\\tilde\{g\}\(R\)has RMSE0\.00440\.0044BPB over8282observations from nine budgets\.The final fit givesR^∞∗=0\.328\\widehat\{R\}^\{\*\}\_\{\\infty\}=0\.328and a floor shift of−0\.0825\-0\.0825BPB\. Extrapolating both frontiers, a natural\-only model at102210^\{22\}training FLOPs needs5\.01×5\.01\\timesas many tokens to match the estimated floor\-optimal mixture, and the natural\-only frontier stops reaching the mixture frontier at3\.61×10263\.61\\times 10^\{26\}FLOPs \(Figure[18](https://arxiv.org/html/2610.31867#A14.F18)\)\. These magnitudes depend on the fit specification\. An independent fit to all671671runs, including the validation runs but holding three102210^\{22\}\-FLOP runs out, gives a12\.17×12\.17\\timesmultiplier at102210^\{22\}FLOPs with95%95\\%bootstrap interval\[6\.39,45\.01\]\[6\.39,45\.01\]and divergence at1\.75×10241\.75\\times 10^\{24\}FLOPs\. No run exceeds102210^\{22\}FLOPs, so both multipliers are extrapolations\.

Figure 18:Natural tokens required to match the estimated floor\-optimal mixture, as a function of training compute\. The final fit gives5\.01×5\.01\\timesat102210^\{22\}FLOPs and diverges atCdiv=3\.61×1026C\_\{\\mathrm\{div\}\}=3\.61\\times 10^\{26\}FLOPs\. Dashed curves exceed the fit range\. The671671\-run fit gives12\.17×12\.17\\timesand1\.75×10241\.75\\times 10^\{24\}FLOPs\.
#### N\.1\.2E⁡\(R\)E\(R\)Derivation

Letpep\_\{e\}denote the evaluation distribution,pnp\_\{n\}the natural training distribution, andpsp\_\{s\}the rewrite distribution\. The mixture distribution is

pR​\(x\)=\(1−R\)​pn​\(x\)\+R​ps​\(x\)\.\\displaystyle p\_\{R\}\(x\)=\(1\-R\)p\_\{n\}\(x\)\+Rp\_\{s\}\(x\)\.\(15\)Partition the evaluation support into a natural\-favored region𝒜\\mathcal\{A\}and a rewrite\-favored regionℬ\\mathcal\{B\}\. On𝒜\\mathcal\{A\}, approximateps​\(x\)=ρn​pn​\(x\)p\_\{s\}\(x\)=\\rho\_\{n\}p\_\{n\}\(x\)withρn<1\\rho\_\{n\}<1\. Onℬ\\mathcal\{B\}, approximatepn​\(x\)=ρs​ps​\(x\)p\_\{n\}\(x\)=\\rho\_\{s\}p\_\{s\}\(x\)withρs<1\\rho\_\{s\}<1\. Letλn\\lambda\_\{n\}andλs\\lambda\_\{s\}denote the evaluation mass of the two regions in the chosen logarithmic unit\.

Equation[15](https://arxiv.org/html/2610.31867#A14.E15)then becomes

pR​\(x\)\\displaystyle p\_\{R\}\(x\)=pn​\(x\)​\[1−\(1−ρn\)​R\],\\displaystyle=p\_\{n\}\(x\)\\left\[1\-\(1\-\\rho\_\{n\}\)R\\right\],x∈𝒜,\\displaystyle x\\in\\mathcal\{A\},\(16\)pR​\(x\)\\displaystyle p\_\{R\}\(x\)=ps​\(x\)​\[ρs\+\(1−ρs\)​R\],\\displaystyle=p\_\{s\}\(x\)\\left\[\\rho\_\{s\}\+\(1\-\\rho\_\{s\}\)R\\right\],x∈ℬ\.\\displaystyle x\\in\\mathcal\{B\}\.\(17\)Substituting into the evaluation cross\-entropy−𝔼x∼pe​log⁡pR​\(x\)\-\\mathbb\{E\}\_\{x\\sim p\_\{e\}\}\\log p\_\{R\}\(x\)and absorbing terms independent ofRRintoE0E\_\{0\}gives

E⁡\(R\)=E0−λn​log⁡\[1−\(1−ρn\)​R\]−λs​log⁡\[ρs\+\(1−ρs\)​R\]\.\\displaystyle E\(R\)=E\_\{0\}\-\\lambda\_\{n\}\\log\\left\[1\-\(1\-\\rho\_\{n\}\)R\\right\]\-\\lambda\_\{s\}\\log\\left\[\\rho\_\{s\}\+\(1\-\\rho\_\{s\}\)R\\right\]\.\(18\)The derivatives are

E′​\(R\)\\displaystyle E^\{\\prime\}\(R\)=λn​\(1−ρn\)1−\(1−ρn\)​R−λs​\(1−ρs\)ρs\+\(1−ρs\)​R,\\displaystyle=\\frac\{\\lambda\_\{n\}\(1\-\\rho\_\{n\}\)\}\{1\-\(1\-\\rho\_\{n\}\)R\}\-\\frac\{\\lambda\_\{s\}\(1\-\\rho\_\{s\}\)\}\{\\rho\_\{s\}\+\(1\-\\rho\_\{s\}\)R\},\(19\)E′′​\(R\)\\displaystyle E^\{\\prime\\prime\}\(R\)=λn​\(1−ρn\)2\[1−\(1−ρn\)​R\]2\+λs​\(1−ρs\)2\[ρs\+\(1−ρs\)​R\]2\>0\.\\displaystyle=\\frac\{\\lambda\_\{n\}\(1\-\\rho\_\{n\}\)^\{2\}\}\{\[1\-\(1\-\\rho\_\{n\}\)R\]^\{2\}\}\+\\frac\{\\lambda\_\{s\}\(1\-\\rho\_\{s\}\)^\{2\}\}\{\[\\rho\_\{s\}\+\(1\-\\rho\_\{s\}\)R\]^\{2\}\}\>0\.\(20\)The floor is convex, with unconstrained minimizer

R∞∗=λs​\(1−ρs\)−λn​\(1−ρn\)​ρs\(1−ρn\)​\(1−ρs\)​\(λn\+λs\)\.\\displaystyle R^\{\*\}\_\{\\infty\}=\\frac\{\\lambda\_\{s\}\(1\-\\rho\_\{s\}\)\-\\lambda\_\{n\}\(1\-\\rho\_\{n\}\)\\rho\_\{s\}\}\{\(1\-\\rho\_\{n\}\)\(1\-\\rho\_\{s\}\)\(\\lambda\_\{n\}\+\\lambda\_\{s\}\)\}\.\(21\)The constrained solution clips Eq\.[21](https://arxiv.org/html/2610.31867#A14.E21)to\[0,1\]\[0,1\]\.

#### N\.1\.3Chinchilla Approach 2→\\toApproach 3 Conversion

Approach 2 fits the compute\-optimal token and loss frontiers

D∗​\(C\)\\displaystyle D^\{\*\}\(C\)=aD​Cp,\\displaystyle=a\_\{D\}C^\{p\},\(22\)L∗​\(C\)\\displaystyle L^\{\*\}\(C\)=E\+aL​C−γ\.\\displaystyle=E\+a\_\{L\}C^\{\-\\gamma\}\.\(23\)WithC=6​N​DC=6ND, the corresponding model frontier is

N∗​\(C\)=16​aD​C1−p\.\\displaystyle N^\{\*\}\(C\)=\\frac\{1\}\{6a\_\{D\}\}C^\{1\-p\}\.\(24\)For an Approach\-3 surfaceL⁡\(N,D\)=E\+A​N−α\+B​D−βL\(N,D\)=E\+AN^\{\-\\alpha\}\+BD^\{\-\\beta\}, both reducible\-loss terms share the frontier exponent\. Matching exponents gives

α=γ1−p,β=γp\.\\displaystyle\\alpha=\\frac\{\\gamma\}\{1\-p\},\\qquad\\beta=\\frac\{\\gamma\}\{p\}\.\(25\)The first\-order condition gives the capacity and data terms shares\(1−p\)\(1\-p\)andppof the frontier reducible loss, so

A\\displaystyle A=\(1−p\)​aL​\(16​aD\)α,\\displaystyle=\(1\-p\)a\_\{L\}\\left\(\\frac\{1\}\{6a\_\{D\}\}\\right\)^\{\\alpha\},\(26\)B\\displaystyle B=p​aL​aDβ\.\\displaystyle=p\\,a\_\{L\}a\_\{D\}^\{\\beta\}\.\(27\)For the final FineWeb\-Edu fit,aD=4\.5033a\_\{D\}=4\.5033,p=0\.46447p=0\.46447,aL=67\.2942a\_\{L\}=67\.2942, andγ=0\.10963\\gamma=0\.10963, givingα=0\.20472\\alpha=0\.20472andβ=0\.23604\\beta=0\.23604\.

#### N\.1\.4FittingL⁡\(N,D,R,T\)L\(N,D,R,T\)

##### Data\-limited case\.

Each target–rewriter pair contributes4848runs: eight epoch budgets by six rewrite fractions\. We fit Eq\. \([14](https://arxiv.org/html/2610.31867#A12.E14)\) to final\-checkpoint Paloma BPB separately for each pair, because the size\-matched rewriter changes the quality of the synthetic data with the target size \(Figure[8](https://arxiv.org/html/2610.31867#A5.F8)\)\. Each fit pools all epoch budgets and rewrite fractions and uses nine parameters: the five floor parameters,BB,β\\beta,PP, andss\(Table[5](https://arxiv.org/html/2610.31867#A14.T5)\)\. We minimize the unweighted sum of squared residuals with bounded least squares \(ρn,ρs∈\[10−4,1\]\\rho\_\{n\},\\rho\_\{s\}\\in\[10^\{\-4\},1\],β∈\[0\.05,3\]\\beta\\in\[0\.05,3\],s∈\[0,8\]s\\in\[0,8\]\) from six fixed starts and6060seeded random starts, keeping the lowest\-cost solution\. Table[5](https://arxiv.org/html/2610.31867#A14.T5)reports point estimates only; the fit script and its cached outputs are released with the code\.

Table 5:Data\-limited fits of Eq\. \([14](https://arxiv.org/html/2610.31867#A12.E14)\) to final\-checkpoint Paloma BPB, one fit per target–rewriter pair,4848runs per row\. RMSE,E0E\_\{0\},λn\\lambda\_\{n\},λs\\lambda\_\{s\}, andPPare in BPB;BBis in BPB⋅\\cdottokensβ, withDDin tokens as in Eq\. \([14](https://arxiv.org/html/2610.31867#A12.E14)\);ρn\\rho\_\{n\},ρs\\rho\_\{s\},β\\beta, andssare dimensionless\.ρs\\rho\_\{s\}sits at its10−410^\{\-4\}lower bound except for the11B pair\.Paloma BPB is U\-shaped in rewrite fraction\. The fitted optimum increases with epoch budget within every target–rewriter pair\. The repetition coefficientPPincreases across pairs\. The best\-mixture asymptote is no higher than the natural\-only asymptote at every pair and lies strictly below it at three; the11B fit places the two at parity and attributes its risingR^∗​\(W\)\\widehat\{R\}^\{\*\}\(W\)to the repetition penalty\. The fit does not isolate a target\-size effect, because the size\-matched rewriter changes the quality of the synthetic data with the target size\.

Figure 19:Estimated optimal rewrite fractionR^∗\\widehat\{R\}^\{\*\}and best observedRRas a function of epoch budgetWWfor each target–rewriter pair\. Orange squares mark the best observedRRon the tested grid\. Dashed blue curves showR^∗\\widehat\{R\}^\{\*\}from each pair’s scaling\-law fit toW≤18W\\leq 18, and the shaded budgetsW=21W=21andW=24W=24are held out\.R^∗\\widehat\{R\}^\{\*\}increases with budget within every pair\. Across pairs, the rewriter changes with the target size\.Out of sample, we refit Eq\. \([14](https://arxiv.org/html/2610.31867#A12.E14)\) for each target–rewriter pair to the3636runs atW≤18W\\leq 18and forecast the six rewrite fractions at each of the held\-out budgetsW=21W=21andW=24W=24\. Figure[4](https://arxiv.org/html/2610.31867#S3.F4)reports theRRthis fit selects at every budget and its measured cost relative to the best observedRR\. The mean error is within±1\.4%\\pm 1\.4\\%at every pair and budget, and the measured loss at the best observedRR, which isR=0\.71R=0\.71at every pair and both budgets, is within0\.74%0\.74\\%of its forecast \(Table[6](https://arxiv.org/html/2610.31867#A14.T6)\)\. The largest errors are atR=0\.13R=0\.13for the550550M,837837M, and11B pairs and atR=0\.29R=0\.29for the33B pair\. The error is\(Lmeasured−Lpredicted\)/Lpredicted\(L\_\{\\mathrm\{measured\}\}\-L\_\{\\mathrm\{predicted\}\}\)/L\_\{\\mathrm\{predicted\}\}, as in Table[8](https://arxiv.org/html/2610.31867#A14.T8)\.

Table 6:Forward data\-limited forecasts\. Each row fits the3636runs atW≤18W\\leq 18and forecasts the six rewrite fractions at the held\-out budgetWW\. RMSE is in BPB\.
##### Compute\-limited case\.

The surface fits the604604observations in three stages: the mixture\-KL floor, the pooled Approach\-2 frontiers promoted through Appendix[N\.1\.3](https://arxiv.org/html/2610.31867#A14.SS1.SSS3), and the curvature term of Appendix[N\.1\.6](https://arxiv.org/html/2610.31867#A14.SS1.SSS6)\. Table[7](https://arxiv.org/html/2610.31867#A14.T7)lists the parameters\. The RMSE over all runs is0\.014490\.01449BPB\. The promoted frontiers are

D∗​\(C\)\\displaystyle D^\{\*\}\(C\)=4\.5033​C0\.46447,\\displaystyle=4\.5033\\,C^\{0\.46447\},\(28\)L∗​\(C,0\)\\displaystyle L^\{\*\}\(C,0\)=0\.80569\+67\.2942​C−0\.10963,\\displaystyle=0\.80569\+67\.2942\\,C^\{\-0\.10963\},\(29\)N∗​\(C\)\\displaystyle N^\{\*\}\(C\)=C6​D∗​\(C\),\\displaystyle=\\frac\{C\}\{6D^\{\*\}\(C\)\},\(30\)which give\(N∗,D∗\)=\(0\.554​B,3\.009​B\)\(N^\{\*\},D^\{\*\}\)=\(0\.554\\text\{B\},3\.009\\text\{B\}\)at101910^\{19\}FLOPs and\(22\.39​B,74\.44​B\)\(22\.39\\text\{B\},74\.44\\text\{B\}\)at102210^\{22\}FLOPs\.

Table 7:Final FineWeb\-Edu surface parameters\.NNandDDare absolute parameter and token counts, andC=6​N​DC=6NDis in FLOPs\. The fittedρs\\rho\_\{s\}reaches its lower bound\. The fitted floor atR=0R=0isE⁡\(0\)=E0−λs​log⁡ρs=0\.805691E\(0\)=E\_\{0\}\-\\lambda\_\{s\}\\log\\rho\_\{s\}=0\.805691\(Eq\.[10](https://arxiv.org/html/2610.31867#A12.E10)\)\.
##### Surface fit\.

We fit the joint surfaceL⁡\(N,D,R\)L\(N,D,R\)to the604604runs of the IsoFLOP ladders, which include the102210^\{22\}\-FLOP hero run as the largest\-budget point\. In sample, the fit has RMSE0\.01450\.0145BPB \(R2=0\.95R^\{2\}=0\.95\) over all604604runs, a root\-mean\-square relative error of1\.1%1\.1\\%\. The residual concentrates in the over\-trained tail: on the296296runs within2×2\\timesof the fitted token optimumD∗​\(C\)D^\{\*\}\(C\), the RMSE is0\.00960\.0096BPB \(R2=0\.98R^\{2\}=0\.98\) and the relative error0\.7%0\.7\\%\. The quantities we use downstream are the compute\-optimal losses, the lowest measured loss at each of the5858pairs of budget and rewrite fraction\. The fitted frontier reproduces these to within0\.75%0\.75\\%\(median0\.19%0\.19\\%\)\. Stated in compute rather than loss, the largest of these errors is the amount by which the fitted compute\-optimal loss changes when the training FLOPs change by19%19\\%, and the median error is the amount from a4%4\\%change in training FLOPs\. Figure[14](https://arxiv.org/html/2610.31867#A14.F14)shows the fitted ladders at all eleven rewrite fractions\. Out of sample, we perform a forward leave\-one\-budget\-out analysis: we fit the surface to the budgets below each held\-out budget and forecast everyRR\. These forecasts have mean errors within±2%\\pm 2\\%at22–3×3\\timesextrapolation, and forecasting the102210^\{22\}runs from a fit range ending at3×10203\\times 10^\{20\}FLOPs gives a mean error of−1\.0%\-1\.0\\%, with every measured loss0\.90\.9–1\.4%1\.4\\%below its forecast \(Table[8](https://arxiv.org/html/2610.31867#A14.T8)\)\.

Table[8](https://arxiv.org/html/2610.31867#A14.T8)reports forward leave\-one\-budget\-out forecasts\. Each row fits only the budgets below the held\-out budget and forecasts every rewrite fraction at that budget\. The error is\(Lmeasured−Lpredicted\)/Lpredicted\(L\_\{\\mathrm\{measured\}\}\-L\_\{\\mathrm\{predicted\}\}\)/L\_\{\\mathrm\{predicted\}\}, so negative values are conservative forecasts\.

Table 8:Forward compute\-limited forecasts by training budget\. The102210^\{22\}row fits through3×10203\\times 10^\{20\}FLOPs and predicts the four rewrite fractions trained at the hero\-run allocation \(R∈\{0,0\.321,0\.5,1\}R\\in\\\{0,0\.321,0\.5,1\\\}\)\.

#### N\.1\.5Repetition\-Penalty Form Ablation

Eq\. \([14](https://arxiv.org/html/2610.31867#A12.E14)\) charges repeated natural tokens with the additive penaltyF=P​\(1−R\)s​\(\(1−R\)​D/T−1\)\+F=P\\,\(1\-R\)^\{s\}\(\(1\-R\)D/T\-1\)\_\{\+\}, with\(1−R\)​D/T=3​W​\(1−R\)/\(3\+2​R\)\(1\-R\)D/T=3W\(1\-R\)/\(3\+2R\)natural epochs\. The factor\(\(1−R\)​D/T−1\)\+\(\(1\-R\)D/T\-1\)\_\{\+\}is the one\-parameter penalty of[Lovelace et al\. \(2026\)](https://arxiv.org/html/2610.31867#bib.bib2)\(their Eq\. 6\), linear in the epochs beyond the first, with the capacity ratioN/TN/Tabsorbed intoPPbecause both are fixed within a data\-limited fit; the dilution factor\(1−R\)s\(1\-R\)^\{s\}is ours\. We refit the4848runs of every target–rewriter pair, with the floorE⁡\(R\)E\(R\),BB, andβ\\betafree and the protocol of Table[5](https://arxiv.org/html/2610.31867#A14.T5), under six penalties: \(i\) the paper’sP​\(1−R\)s​\(\(1−R\)​D/T−1\)\+P\(1\-R\)^\{s\}\(\(1\-R\)D/T\-1\)\_\{\+\}; \(ii\)P​\(1−R\)s​\(\(1−R\)​D/T−1\)\+δP\(1\-R\)^\{s\}\(\(1\-R\)D/T\-1\)\_\{\+\}^\{\\delta\}withδ∈\[0\.5,3\]\\delta\\in\[0\.5,3\]free, the superlinear exponent on the repetition count of their Eq\. 8; \(iii\) the same withδ\\deltafixed at1\.841\.84, the value that[Lovelace et al\. \(2026\)](https://arxiv.org/html/2610.31867#bib.bib2)fit for their Eq\. 8 on their own runs; we do not fit it, and this variant tests whether their exponent transfers to our setting \(labeled “δ=1\.84\\delta=1\.84\(Lovelace\)” in Figures[20](https://arxiv.org/html/2610.31867#A14.F20)–[22](https://arxiv.org/html/2610.31867#A14.F22)\); \(iv\)P​\(\(1−R\)​D/T−1\)\+P\(\(1\-R\)D/T\-1\)\_\{\+\}, their Eq\. 6 without dilution \(s=0s=0\); \(v\) no penalty,P=0P=0, so thatDDenters the Chinchilla term only; and \(vi\) no additive penalty but the effective\-data term of[Muennighoff et al\. \(2023\)](https://arxiv.org/html/2610.31867#bib.bib4)in place ofDD, with the saturating count applied to the natural epochs only, since rewrites are never repeated\. Their Eq\. 7 exponent onN/UDN/U\_\{D\}is not identifiable here, becauseNNandTTdo not vary within a pair\. Per pair,88runs have\(1−R\)​D/T=0\(1\-R\)D/T=0\(R=1R=1\), one has0<\(1−R\)​D/T<10<\(1\-R\)D/T<1\(W=3W=3,R=0\.71R=0\.71\), and3939have\(1−R\)​D/T\>1\(1\-R\)D/T\>1\(up to\(1−R\)​D/T=24\(1\-R\)D/T=24atR=0R=0\), so every penalty is identified on the same3939cells and the data cannot separate\(\(1−R\)​D/T−1\)\+\(\(1\-R\)D/T\-1\)\_\{\+\}from\(1−R\)​D/T\(1\-R\)D/T\(Section[N\.1\.4](https://arxiv.org/html/2610.31867#A14.SS1.SSS4)\)\.

Table 9:Repetition\-penalty form ablation for Eq\. \([14](https://arxiv.org/html/2610.31867#A12.E14)\), one fit per target–rewriter pair,4848runs per fit\. RMSE is in BPB andR2=1−SSres/SStotR^\{2\}=1\-\\mathrm\{SS\}\_\{\\mathrm\{res\}\}/\\mathrm\{SS\}\_\{\\mathrm\{tot\}\}over the4848runs\. The held\-out columns refit the3636runs atW≤18W\\leq 18and forecast the six rewrite fractions atW=21W=21andW=24W=24: mean error is the mean of\(Lmeasured−Lpredicted\)/Lpredicted\(L\_\{\\mathrm\{measured\}\}\-L\_\{\\mathrm\{predicted\}\}\)/L\_\{\\mathrm\{predicted\}\}over the six, error at best obs\.RRis the same quantity at the best observedRR\(R=0\.71R=0\.71at every pair and both budgets\), andRselR\_\{\\mathrm\{sel\}\}is the tested fraction with the lowest forecast loss, the selection rule of Figure[4](https://arxiv.org/html/2610.31867#S3.F4)\.R^24∗\\widehat\{R\}^\{\*\}\_\{24\}is the minimizer of the full4848\-run fit overR∈\[0,1\]R\\in\[0,1\]atW=24W=24\. The parameter count includes the five floor parameters,BB, andβ\\beta\. Fitted parameters are inrepetition\_penalty/de\_penalty\_ablation\.csv\.Figure 20:Fitted Paloma BPB against rewrite fraction under each repetition penalty \(full4848\-run fits\), atW=24W=24\(top\) andW=3W=3\(bottom\), with the measured runs\. The four penalties with an additive term are nearly coincident atW=24W=24between the tested fractions; the two forms without one fall0\.050\.05–0\.160\.16BPB below theR=0R=0run atW=24W=24and0\.070\.07–0\.150\.15BPB above it atW=3W=3\. The no\-dilution form places itsW=3W=3minimum below the measured points at smallRR\.Figure 21:Estimated optimal rewrite fractionR^∗​\(W\)\\widehat\{R\}^\{\*\}\(W\)under each repetition penalty \(full4848\-run fits\), with the best observedRRat each budget\. All forms with the dilution factor\(1−R\)s\(1\-R\)^\{s\}giveR^∗​\(24\)\\widehat\{R\}^\{\*\}\(24\)between0\.390\.39and0\.590\.59, below the best observedR=0\.71R=0\.71atW≥18W\\geq 18; the no\-dilution form rises faster \(0\.620\.62–0\.740\.74\) but at the cost of fit quality and a degenerate floor \(Table[9](https://arxiv.org/html/2610.31867#A14.T9)\)\. The forms without an additive penalty are flat inWW\.Figure 22:Observed against predicted Paloma macro BPB for every repetition penalty \(rows, in the order of Table[9](https://arxiv.org/html/2610.31867#A14.T9)\) and target–rewriter pair \(columns\), all4848runs of the full fit, coloured by epoch budgetWW, with the identity line\. Hollow markers are theW=21W=21andW=24W=24forecasts of theW≤18W\\leq 18refit\. Each panel reportsR2R^\{2\}and RMSE of the full fit; axis limits are shared within a column\. The three dilution forms and the no\-dilution form lie along the identity line; the two forms without an additive penalty flatten to horizontal bands at largeWW, where theR=0R=0runs cannot be separated from their neighbours by a term inDDalone\.Three conclusions follow \(Table[9](https://arxiv.org/html/2610.31867#A14.T9), Figures[20](https://arxiv.org/html/2610.31867#A14.F20)–[22](https://arxiv.org/html/2610.31867#A14.F22)\)\. Figure[22](https://arxiv.org/html/2610.31867#A14.F22)shows the fits run by run: the paper form and the twoδ\\deltavariants have indistinguishableR2R^\{2\}\(0\.9590\.959–0\.9850\.985across pairs for the paper form, within0\.0110\.011of it forδ\\deltafree or fixed\), the no\-dilution form loses0\.0060\.006–0\.0230\.023, and the two forms without an additive penalty fall toR2=0\.62R^\{2\}=0\.62–0\.870\.87, lowest at33B, where the penalty is largest\. First, an additive penalty is needed: dropping it raises RMSE by1\.81\.8–3\.4×3\.4\\times, pushesβ\\betato its upper bound of33, and under\-predicts the held\-outW=24W=24runs by0\.80\.8–3\.8%3\.8\\%on average because theR=0R=0run atW=24W=24cannot be reached by a term inDDalone; the effective\-data variant fits no better, since its saturation parameter goes to0\.850\.85–0\.920\.92withβ\\betaat its lower bound at three pairs and to its upper bound of200200, which is the no\-penalty form, at the fourth\. Second, the exponent on the repetition count is not identified by this grid\. Freeingδ\\deltalowers RMSE by at most0\.00050\.0005BPB and returnsδ=0\.77,1\.10,1\.05,1\.19\\delta=0\.77,1\.10,1\.05,1\.19across pairs, indistinguishable from the linear form on the tested fractions, whilessmoves to compensate \(s=8s=8, its bound, at550550M\)\. Fixingδ=1\.84\\delta=1\.84, the value[Lovelace et al\. \(2026\)](https://arxiv.org/html/2610.31867#bib.bib2)obtain from residuals against a single\-epoch Chinchilla fit withN/UDN/U\_\{D\}varied, is worse at every pair \(\+0\.0015\+0\.0015to\+0\.0023\+0\.0023BPB\), drivesβ\\betato its bound at11B, and doubles the held\-out mean error; their exponent does not transfer once the floor andssare free\. Third, the dilution factor\(1−R\)s\(1\-R\)^\{s\}is what the linear form needs to coexist with the mixture floor: without it \(s=0s=0\) RMSE rises by0\.00110\.0011–0\.00470\.0047BPB and the fit is degenerate, withE0→0E\_\{0\}\\to 0andβ\\betaat its lower bound at two pairs, so the Chinchilla term becomes a near\-constant and the floor absorbs the penalty’sRR\-dependence\. On the selection metric of Figure[4](https://arxiv.org/html/2610.31867#S3.F4)the picture is less flattering to the paper’s form: with theW≤18W\\leq 18fit it selectsR=0\.47R=0\.47instead of the best observedR=0\.71R=0\.71at33B \(measured cost0\.50%0\.50\\%atW=21W=21and1\.14%1\.14\\%atW=24W=24\) and at550550M,W=21W=21\(0\.01%0\.01\\%\), as do theδ\\deltavariants, whereas the no\-dilution form selects0\.710\.71at all eight held\-out cells despite its worse RMSE, because itsR^∗​\(W\)\\widehat\{R\}^\{\*\}\(W\)rises faster\. All forms with dilution place the continuousR^∗​\(24\)\\widehat\{R\}^\{\*\}\(24\)at0\.390\.39–0\.590\.59, below the best observedRR, and the two forms without a penalty are flat inWW\. We keep the linear form of Eq\. \([14](https://arxiv.org/html/2610.31867#A12.E14)\): it is the most parsimonious form that reproduces the U\-curves, and the extra exponent buys nothing the data can resolve\. The exponentss, notδ\\delta, governs how fastR^∗\\widehat\{R\}^\{\*\}grows with budget, and it is weakly identified \(s=1\.5s=1\.5–7\.07\.0across pairs\), so extrapolations ofR^∗​\(W\)\\widehat\{R\}^\{\*\}\(W\)beyondW=24W=24should be read with that in mind\.

#### N\.1\.6Ablating the Curvature Term

The compute\-limited surface adds to Eq\. \([1](https://arxiv.org/html/2610.31867#S2.E1)\) withF=0F=0a term fit to the excess curvature of the measured IsoFLOP parabolas,

Curv⁡\(N,D\)=HCη​\[log10⁡DD∗​\(C\)\]2,C=6​N​D,\\mathrm\{Curv\}\(N,D\)=\\frac\{H\}\{C^\{\\eta\}\}\\left\[\\log\_\{10\}\\frac\{D\}\{D^\{\*\}\(C\)\}\\right\]^\{2\},\\qquad C=6ND,\(31\)so that

LCE​\(N,D,R\)=E⁡\(R\)\+ANα\+BDβ\+Curv⁡\(N,D\)\.\\displaystyle L\_\{\\mathrm\{CE\}\}\(N,D,R\)=E\(R\)\+\\frac\{A\}\{N^\{\\alpha\}\}\+\\frac\{B\}\{D^\{\\beta\}\}\+\\mathrm\{Curv\}\(N,D\)\.\(32\)The term vanishes with zero slope on the frontier, so it changes off\-frontier predictions without movingN∗​\(C\)N^\{\*\}\(C\),D∗​\(C\)D^\{\*\}\(C\), orLbase∗​\(C\)L^\{\*\}\_\{\\mathrm\{base\}\}\(C\)\.

We ablate the term on the671671\-run fit of Appendix[N\.1\.1](https://arxiv.org/html/2610.31867#A14.SS1.SSS1), holding the floor and Chinchilla terms fixed\. Removing it raises the residual sum of squares from0\.06080\.0608to0\.27260\.2726, with likelihood\-ratio statistic1006\.61006\.6\. The architecture\-wise wild bootstrap of Appendix[M](https://arxiv.org/html/2610.31867#A13), over7979architectures and400400draws, rejects the restriction \(p=1/401p=1/401\)\.

### N\.2Domain\-Specific Compute Optima

##### Break\-even with SmolLM2\-Instruct by evaluation\.

On QA, the lowest fitted QA BPB over model size, token count, and rewrite fraction matches SmolLM2\-1\.7B\-Instruct at5\.3×10215\.3\\times 10^\{21\}total FLOPs withR^∗=0\.74\\widehat\{R\}^\{\*\}=0\.74, and the102210^\{22\}\-FLOP run trained at the QA optimum measures0\.6960\.696QA BPB against0\.7090\.709for SmolLM2\-Instruct\. On Math and Code the fitted minimum matches SmolLM2\-Instruct only at≈6×1022\{\\approx\}6\\times 10^\{22\}and≈4×1023\{\\approx\}4\\times 10^\{23\}total FLOPs,6×6\\timesand43×43\\timesthe largest budget we trained, and the Code crossing lies above SmolLM2’s≈1\.1×1023\{\\approx\}1\.1\\times 10^\{23\}training FLOPs\. SmolLM2’s training corpus includes FineMath and Stack\-Edu\([Ben Allal et al\., 2025](https://arxiv.org/html/2610.31867#bib.bib16)\)and StarCoderData\([Li et al\., 2023](https://arxiv.org/html/2610.31867#bib.bib31)\), math and code data that FineWeb\-Edu largely lacks\. For each domain, the102210^\{22\}\-FLOP run trained at that domain’s fitted optimum measures a higher loss than its surface forecasts, by4%4\\%on QA,14%14\\%on Math, and32%32\\%on Code, against1%1\\%for the Paloma\-optimal run\. We hypothesize two causes\. First, we tuned the training recipe on Paloma, whose macro average weights1616sources equally: ten web or encyclopedic text sources, five social\-media sources, one code source, and no math source \(Appendix[D](https://arxiv.org/html/2610.31867#A4)\)\. Web and encyclopedic text carries the knowledge that QA evaluations test, so the forecast error is smallest on Paloma and then QA\. Second, the rewrite styles include math and not code\. The rewrites contain at most0\.8%0\.8\\%code lines and add lists, tables, and headers rather than code \(Appendix[K](https://arxiv.org/html/2610.31867#A11)\), so code remains rare in the training data even atR=1R=1, and the Code surface is fit to the narrowest range of loss\.

Figure 23:The estimated floor\-optimal rewrite fractionR^∞∗\\widehat\{R\}^\{\*\}\_\{\\infty\}against two properties of each Paloma source\.Left:R^∞∗\\widehat\{R\}^\{\*\}\_\{\\infty\}has no association with the rewriter’s relative strength on the source \(Spearmanρ=0\.03\\rho=0\.03,p=0\.91p=0\.91\)\.Right:R^∞∗\\widehat\{R\}^\{\*\}\_\{\\infty\}increases with the natural\-data model’s BPB on the source \(Spearmanρ=0\.47\\rho=0\.47,p=0\.07p=0\.07\)\.Figures[24](https://arxiv.org/html/2610.31867#A14.F24)and[25](https://arxiv.org/html/2610.31867#A14.F25)decompose the aggregate trajectories of Figure[2](https://arxiv.org/html/2610.31867#S3.F2.fig1)into Paloma sources and domain\-suite tasks\. Across Paloma sources,R^∞∗\\widehat\{R\}^\{\*\}\_\{\\infty\}reaches≈0\.89\{\\approx\}0\.89for programming languages and0\.570\.57for RedPajama, while C4, RefinedWeb, and Wikipedia remain atR^∗=0\\widehat\{R\}^\{\*\}=0throughout the evaluated compute range \(Figure[24](https://arxiv.org/html/2610.31867#A14.F24)\)\. Every Math and Code task reachesR^∞∗=1\\widehat\{R\}^\{\*\}\_\{\\infty\}=1\. Within QA, ARC\-Easy, ARC\-Challenge, and MMLU reachR^∞∗≥0\.82\\widehat\{R\}^\{\*\}\_\{\\infty\}\\geq 0\.82, HellaSwag remains below0\.10\.1, and the QA aggregate reaches0\.770\.77\(Figure[25](https://arxiv.org/html/2610.31867#A14.F25)\)\.

Each trajectory comes from a separate surface fit to its aggregate\. The Paloma, QA, Math, and Code fits contain604604,597597,608608, and610610observations and have RMSE0\.01480\.0148,0\.01950\.0195,0\.03010\.0301, and0\.06810\.0681BPB\. Their estimated floor optima are0\.3290\.329,0\.7680\.768,1\.01\.0, and1\.01\.0\. The Math and Code estimates lie at the boundary, and their break\-even points lie beyond the measured ladders\.

Figure 24:Estimated compute\-optimal rewrite fractionR^∗​\(Ctotal\)\\widehat\{R\}^\{\*\}\(C\_\{\\mathrm\{total\}\}\)for each Paloma source\. Solid curves cover the measured ladders through≈2×1020\{\\approx\}2\\times 10^\{20\}total FLOPs, and dashed curves extrapolate\. The thick curve is Paloma macro BPB\. The vertical line marks SmolLM2’s training compute\.Figure 25:Estimated compute\-optimal rewrite fractionR^∗​\(Ctotal\)\\widehat\{R\}^\{\*\}\(C\_\{\\mathrm\{total\}\}\)for each task\. Legends reportR^∞∗\\widehat\{R\}^\{\*\}\_\{\\infty\}, and thick curves show the aggregate trajectories of Figure[2](https://arxiv.org/html/2610.31867#S3.F2.fig1)\. MMLU, Minerva, and multilingual MBPP average5757,77, and1717subtasks\. TheoremQA switches fromR=0R=0toR=1R=1because its fitted floor is nearly linear inRR; its fit excludes the two budgets with the fewest model widths\.
##### Compute\-optimal allocation by evaluation\.

Each per\-evaluation surface fit has its own compute\-optimal frontierN∗​\(C\)N^\{\*\}\(C\)andD∗​\(C\)D^\{\*\}\(C\), shared across rewrite fractions within the fit\. Figure[26](https://arxiv.org/html/2610.31867#A14.F26)shows each evaluation’s natural\-only IsoFLOP slices with its fitted optima and the Paloma frontier overlaid, and Table[10](https://arxiv.org/html/2610.31867#A14.T10)reports the frontier at two training budgets\. The QA frontier allocates the fewest tokens per parameter and the Code frontier the most, with Math between them: at102210^\{22\}FLOPs the QA optimum has1\.5×1\.5\\timesthe parameters of the Code optimum at the same training compute\. This reproduces the finding of[Roberts et al\. \(2025\)](https://arxiv.org/html/2610.31867#bib.bib47)that knowledge QA is capacity\-hungry while code is data\-hungry, and that the compute\-optimal parameter count depends on the skill composition of the validation set by up to≈50%\{\\approx\}50\\%\. Our fits reach the same conclusion on ladders that include rewrites at every fraction, so the allocation difference is a property of the evaluation rather than of the rewrite fraction\. Relative to the Paloma frontier, which increases more steeply inNN, the QA optimum lies at more parameters below≈1020\{\\approx\}10^\{20\}FLOPs and fewer above, while the Code optimum lies at fewer parameters at every budget\.

Figure 26:Knowledge QA is capacity\-hungry and Code is data\-hungry\.Left three panels:natural\-only \(R=0R=0\) IsoFLOP slices from each evaluation’s own surface fit, loss against parameter count, colored by training budget, with dots for runs and stars for the fitted optimum at each budget\. The dashed line is the Paloma frontierN∗​\(C\)N^\{\*\}\(C\)drawn on that evaluation’s surface\.Right:compute\-optimal parameter count of each evaluation’s fit against training compute\. The QA optimum has more parameters than the Code optimum at every budget,1\.5×1\.5\\timesat102210^\{22\}FLOPs\.Table 10:Compute\-optimal allocation from each evaluation’s surface fit at two training budgets\.D∗/N∗D^\{\*\}/N^\{\*\}is tokens per parameter\.

### N\.3Evaluations of the102210^\{22\}\-FLOP Runs

Table[11](https://arxiv.org/html/2610.31867#A14.T11)compares the102210^\{22\}\-FLOP runs with SmolLM2 using the matched Paloma reference and the offline QA, Math, and Code references\. Table[12](https://arxiv.org/html/2610.31867#A14.T12)reports the102210^\{22\}runs by Paloma source\. Tables[13](https://arxiv.org/html/2610.31867#A14.T13)and[14](https://arxiv.org/html/2610.31867#A14.T14)break the QA, Math, and Code aggregates into their tasks, subjects, and languages\.

Table 11:The102210^\{22\}\-FLOP hero run nearly matches SmolLM2\-Instruct on Paloma\.The Paloma column uses byte\-pooled BPB at a40964096\-token context for both model families\. The QA, Math, and Code columns retain their matched offline evaluations\. The first four rows share the Paloma\-optimal\(N,D\)\(N,D\), hence training computeC≈1022C\\approx 10^\{22\}FLOPs, and differ only inRR; their total computeCtotal=6​N​D\+4​Nr​R​DC\_\{\\mathrm\{total\}\}=6ND\+4N\_\{r\}RDvaries withRR\. SmolLM2 is trained on1111T tokens at≈1\.1×1023\{\\approx\}1\.1\\times 10^\{23\}FLOPs,11×11\\timesthe total budget of the hero run\.Optimized forModelRRNNDDBPB↓\\downarrowAcc\.↑\\uparrowPalomaQAMathCodeQA–Natural\-only0021\.9B76B1\.0610\.7311\.3281\.0420\.619PalomaHero0\.320\.3221\.9B76B0\.9830\.7000\.8420\.5870\.622–Grid0\.50\.521\.9B76B0\.9880\.6980\.8200\.5680\.620–Grid1\.01\.021\.9B76B1\.0520\.7190\.8000\.5290\.594QAGrid0\.740\.7413\.8B120B0\.9880\.6960\.8010\.5200\.620CodeGrid1\.01\.09\.7B172B1\.1070\.7100\.7880\.5070\.597–SmolLM2\-1\.7B–1\.7B11T0\.9430\.6670\.7070\.3990\.617–SmolLM2\-1\.7B\-Instruct–1\.7B11T0\.9720\.7090\.6970\.4260\.606Table 12:TheR=0\.321R=0\.321hero run minimizes Paloma macro BPB among the runs at102210^\{22\}training FLOPs\.The four2222B runs share\(N,D\)=\(21\.9​B,76​B\)\(N,D\)=\(21\.9\\mathrm\{B\},76\\mathrm\{B\}\); the best value among them per row is bold\. The SmolLM2 columns are evaluated on the same Paloma validation split at a40964096\-token context\. TheR=0\.321R=0\.321run reaches a lower BPB than SmolLM2\-1\.7B\-Instruct on1010of the1616sources\.Table 13:Task\-level scores behind the QA, Math, and Code aggregates of Table[11](https://arxiv.org/html/2610.31867#A14.T11)\.Columns are the runs at102210^\{22\}training FLOPs with downstream evaluations; best value per row in bold\. MMLU, Minerva MATH, and multilingual MBPP are macro averages over their subtasks \(Table[14](https://arxiv.org/html/2610.31867#A14.T14)\)\. SmolLM2 task\-level scores are not archived\.Table 14:Minerva MATH subjects and multilingual MBPP languages\(BPB, lower is better\) for the runs of Table[13](https://arxiv.org/html/2610.31867#A14.T13); best value per row in bold\.
### N\.4Breakeven Parameters

#### N\.4\.1Definition and Relation to Training Compute

For each comparison metricmm, break\-even is the smallest total compute, accounting for rewrites, at which the fitted compute\-optimal frontier reaches SmolLM2\-Instruct’s score:

CBE,m=min⁡\{Ctotal:Lm∗​\(Ctotal\)≤Lm,SmolLM2​\-​inst\}\.\\displaystyle C\_\{\\mathrm\{BE\},m\}=\\min\\left\\\{C\_\{\\mathrm\{total\}\}:L\_\{m\}^\{\*\}\(C\_\{\\mathrm\{total\}\}\)\\leq L\_\{m,\\mathrm\{SmolLM2\\text\{\-\}inst\}\}\\right\\\}\.\(33\)HereLm∗​\(Ctotal\)L\_\{m\}^\{\*\}\(C\_\{\\mathrm\{total\}\}\)is the loss at the joint optimum of Eq\. \([13](https://arxiv.org/html/2610.31867#A12.E13)\) on metricmm’s fitted surface\. SmolLM2\-Instruct enters only as the reference score\. Its own training compute is1\.1286×10231\.1286\\times 10^\{23\}FLOPs\. The crossing accounts for rewrite generation but not SmolLM2’s training\. Paloma uses the0\.97160\.9716BPB reference matched to our byte\-pooled aggregation and40964096\-token context\. QA, Math, and Code use the same offline evaluations for both model families\.

#### N\.4\.2BreakevenNN,DD, andRRper Domain

Table[15](https://arxiv.org/html/2610.31867#A14.T15)gives the fitted allocation at each matched crossing\. The training\-compute column is6​N∗​D∗6N^\{\*\}D^\{\*\}, and its difference from total compute is the generation cost\. Paloma, QA, and Math cross below SmolLM2’s training compute, and Code crosses above it\.

Table 15:Fitted break\-even allocations against SmolLM2\-Instruct\. Paloma crosses at2\.53×10222\.53\\times 10^\{22\}FLOPs withR^∗=0\.325\\widehat\{R\}^\{\*\}=0\.325, using matched within\-source aggregation and context length\. QA crosses at5\.32×10215\.32\\times 10^\{21\}FLOPs withR^∗=0\.739\\widehat\{R\}^\{\*\}=0\.739\. The Math and Code crossings lie at theR=1R=1boundary and beyond the measured ladders\.

### N\.5Loss\- vs Accuracy\-Based Fits

#### N\.5\.1Parametric Accuracy Model

As a check on the BPB criterion, we fit QA accuracy directly on all607607accuracy observations by mapping the surface through a logistic link:

a^​\(N,D,R\)\\displaystyle\\widehat\{a\}\(N,D,R\)=θ1\+exp⁡z⁡\(N,D,R\),\\displaystyle=\\frac\{\\theta\}\{1\+\\exp z\(N,D,R\)\},\(34\)z⁡\(N,D,R\)\\displaystyle z\(N,D,R\)=E⁡\(R\)\+ANα\+BDβ\+H​C−η​\[log10⁡DD∗​\(C\)\]2,\\displaystyle=E\(R\)\+\\frac\{A\}\{N^\{\\alpha\}\}\+\\frac\{B\}\{D^\{\\beta\}\}\+HC^\{\-\\eta\}\\left\[\\log\_\{10\}\\frac\{D\}\{D^\{\*\}\(C\)\}\\right\]^\{2\},\(35\)whereC=6​N​DC=6NDandE⁡\(R\)E\(R\)is the mixture\-dependent floor of Eq\. \([10](https://arxiv.org/html/2610.31867#A12.E10)\)\. The fit minimizes a soft\-L1L\_\{1\}objective with scale0\.020\.02, initialized from the QA\-BPB surface, and refits every parameter\.

#### N\.5\.2Accuracy Fits

The observed QA accuracies span0\.30410\.3041–0\.51290\.5129\. The fit hasR2=0\.98795R^\{2\}=0\.98795, RMSE0\.004530\.00453, and mean absolute error0\.003450\.00345\. Its ceilingθ\\thetareaches the upper bound11and its interceptE0E\_\{0\}the lower bound−10\-10, so the asymptotic parameters are weakly identified although the in\-grid fit is accurate\.

With the1\.711\.71B rewriter cost, the accuracy fit givesR^∗​\(1022\)=0\.638\\widehat\{R\}^\{\*\}\(10^\{22\}\)=0\.638\. It reaches the SmolLM2\-instruct and base QA accuracies,0\.6060\.606and0\.6170\.617, at3\.336×10213\.336\\times 10^\{21\}and4\.484×10214\.484\\times 10^\{21\}FLOPs\. Both references exceed every accuracy in the fitting grid, so both crossings are extrapolations\. The QA hero run obtains0\.620170\.62017accuracy against a forecast of0\.647340\.64734\. The QA\-BPB surface givesR^∗​\(1022\)=0\.749\\widehat\{R\}^\{\*\}\(10^\{22\}\)=0\.749and crosses the instruct BPB reference at5\.316×10215\.316\\times 10^\{21\}FLOPs\. Both criteria favor a large rewrite fraction, and the exact estimated optimum and crossing depend on the criterion\.

### N\.6Scaling Setup Ablations

#### N\.6\.1Alternative Surface Specifications

We compare three nested surface models on the583583runs of Appendix[M](https://arxiv.org/html/2610.31867#A13)\. M1 shares the compute\-optimal center across rewrite fractions and gives each fraction a floor shift\. M2 adds anRR\-specific center and curvature\. M3 fits an independent log\-quadratic to every\(C,R\)\(C,R\)cell\. Table[16](https://arxiv.org/html/2610.31867#A14.T16)reports the residual error and Bayesian information criterion \(BIC\) of each\.

Table 16:Surface\-model ladder on the same583583runs\. BIC favors the shared\-center model M1 although the richer models reduce residual error\.NestedFFtests reject M1 against M2 \(p=4\.73×10−12p=4\.73\\times 10^\{\-12\}\) and against M3 \(p=7\.79×10−10p=7\.79\\times 10^\{\-10\}\)\. TheRR\-dependent floor shifts nevertheless agree within≈0\.005\{\\approx\}0\.005BPB across the three models, and M1 attains99\.3%99\.3\\%of M3’sR2R^\{2\}with15%15\\%of its parameters\. We use M1’s shared center\.

### N\.7Optimal Rewrite Fraction across Source Quality

##### The estimated floor\-optimal rewrite fraction is smaller, but non\-zero, on a stronger source corpus\.

We replace FineWeb\-Edu with DCLM\([Li et al\., 2024](https://arxiv.org/html/2610.31867#bib.bib6)\)and sweepRRat fixedNNandDDat training budgets ofC=9×1018C=9\\times 10^\{18\}and1\.8×10191\.8\\times 10^\{19\}FLOPs\. These sweeps do not account for generation, so they estimate the floorE⁡\(R\)E\(R\)and its optimumR∞∗R^\{\*\}\_\{\\infty\}rather than the compute\-optimalR∗R^\{\*\}of Section[2](https://arxiv.org/html/2610.31867#S2)\. DCLM is the stronger corpus: at every budget with final evaluations for both corpora, the natural\-only DCLM run with the lowest Paloma macro BPB lies below its FineWeb\-Edu counterpart \(Figure[27](https://arxiv.org/html/2610.31867#A14.F27), right\), and DCLM has lower Paloma BPB in all7777matched width–depth cells over seven budgets \(Appendix[N\.7](https://arxiv.org/html/2610.31867#A14.SS7)\)\. On DCLM,R=0\.1R=0\.1minimizes Paloma BPB at both budgets, and its gain overR=0R=0is smaller than the gain of the best rewrite fraction overR=0R=0on the FineWeb\-Edu sweeps at the same budgets \(Figure[27](https://arxiv.org/html/2610.31867#A14.F27), left\)\. The rest of this appendix fits a surface to DCLM and reports its estimated floor\-optimal rewrite fraction and the matched comparison with FineWeb\-Edu\.

Figure 27:Left:Paloma macro BPB againstRRon DCLM at two training budgets:550550M parameters and2\.92\.9B tokens at9×10189\\times 10^\{18\}FLOPs, and669669M parameters and4\.74\.7B tokens at1\.8×10191\.8\\times 10^\{19\}FLOPs\.Right:final Paloma macro BPB of the observed compute\-optimal natural\-only \(R=0R=0\) DCLM and FineWeb\-Edu run at each training budget shown\. Each point selects the lowest\-Paloma run within its corpus and budget\.The DCLM sweep gives a fittedR^∞∗≈0\.08\\widehat\{R\}^\{\*\}\_\{\\infty\}\\approx 0\.08\(Figure[27](https://arxiv.org/html/2610.31867#A14.F27)\)\. This appendix gives the natural\-only ladder, the floor fit, and the matched comparison with FineWeb\-Edu\.

##### Natural\-only ladder\.

The DCLM ladder contains8181completedR=0R=0runs at eight budgets,3×10183\\times 10^\{18\},9×10189\\times 10^\{18\},1\.8×10191\.8\\times 10^\{19\},3×10193\\times 10^\{19\},9×10199\\times 10^\{19\},1\.8×10201\.8\\times 10^\{20\},3×10203\\times 10^\{20\}, and102210^\{22\}FLOPs, with55,1010,1515,1010,1414,1313,1313, and11runs \(Figure[28](https://arxiv.org/html/2610.31867#A14.F28)\)\. The DCLM runs follow the recipe of Appendix[C](https://arxiv.org/html/2610.31867#A3)with a reference point tuned on DCLM: a Gaussian\-process search over the Hyperball learning rate, Adam learning rate, andϵ\\epsilonatd=512d=512and2\.5×1092\.5\\times 10^\{9\}tokens, withβ1=0\.9\\beta\_\{1\}=0\.9andβ2,0=0\.9999\\beta\_\{2,0\}=0\.9999fixed, givesη0=1\.45×10−2\\eta\_\{0\}=1\.45\\times 10^\{\-2\},η0A=3\.28×10−3\\eta^\{\\mathrm\{A\}\}\_\{0\}=3\.28\\times 10^\{\-3\}, andϵ0=8\.20×10−12\\epsilon\_\{0\}=8\.20\\times 10^\{\-12\}\. An Approach\-2 fit to the budget frontiers gives

L⁡\(N,D\)=0\.67800\+46\.677N0\.26207\+29\.947D0\.21364,\\displaystyle L\(N,D\)=0\.67800\+\\frac\{46\.677\}\{N^\{0\.26207\}\}\+\\frac\{29\.947\}\{D^\{0\.21364\}\},\(36\)which impliesD∗​\(C\)∝C0\.55091D^\{\*\}\(C\)\\propto C^\{0\.55091\}and fits the frontier with RMSE0\.0009890\.000989BPB, againstα=0\.20\\alpha=0\.20andD∗∝C0\.46D^\{\*\}\\propto C^\{0\.46\}on FineWeb\-Edu\. The single102210^\{22\}run has no within\-budget width control and enters the fit as an anchor only\.

Figure 28:The seven width\-controlled DCLM IsoFLOP ladders\. Points are completed runs, curves are per\-budget quadratics in log model size, and stars mark their minima\. Each panel reports theR2R^\{2\}of its quadratic\. Every budget has an interior minimum over model size\. The single\-run102210^\{22\}anchor is omitted\.Figure 29:Final evaluations of the observed compute\-optimal natural\-only DCLM and FineWeb\-Edu runs\. At each training budget and corpus, we select theR=0R=0run with the lowest final Paloma macro BPB and show that run’s Paloma, Math, Code, and QA scores\. Lower is better for the three BPB panels and higher for QA accuracy\.
##### DCLM rewrite sweep\.

The sweep contains4040runs at the two configurations of Section[3\.2](https://arxiv.org/html/2610.31867#S3.SS2), one per\(C,R\)\(C,R\)cell\. Refitting the mixture\-KL floor givesR^∞∗=0\.078\\widehat\{R\}^\{\*\}\_\{\\infty\}=0\.078, a0\.005770\.00577\-BPB gain overR=0R=0, and RMSE0\.001620\.00162BPB\.

##### Cross\-corpus comparison\.

Matching DCLM runs to FineWeb\-EduR=0R=0runs by budget, width, and depth gives7777pairs over seven budgets\. DCLM has lower Paloma BPB in all7777pairs\. The pairs are not exact: the DCLM models have1\.021\.02–1\.16×1\.16\\timesthe parameters and0\.910\.91–0\.99×0\.99\\timesthe tokens, and the two corpora use different tuned reference points\.

## Appendix OQualitative Rewrite Examples

We selected the ten examples below to illustrate the four styles and the range of rewriter sizes, not to assess fidelity\. Each pairs a source document with its FinePhrase rewrite and one bootstrapped\-rewriter rewrite, spanning the FAQ, table, tutorial, and math styles and rewriters from550550M to33B parameters\. Several rewrites contain errors that the rewriter introduced: arithmetic mistakes in the math style, invented details in the FAQ and tutorial styles, and a contradiction of the source in one biology example\. We do not measure aggregate fidelity, and the loss results in Sections[3\.2](https://arxiv.org/html/2610.31867#S3.SS2)and[3\.3](https://arxiv.org/html/2610.31867#S3.SS3)do not depend on it\.

#### O\.0\.1550M: FAQ

Checkpoint:delphi\-glm\-550m\-sft\-llama3\-lr1e\-04\-ep15 Document:<urn:uuid:f719a404\-47c8\-4aab\-bc52\-a5ce965f66dc\>

##### Original\.

WhatdoesNEPADstandfor?

TheNewPartnershipforAfrica’sDevelopment\.

WhataretheoriginsofNEPAD?

TheNEPADstrategicframeworkdocumentwaspreparedbytheleadersofthefiveinitiatingstates\(Algeria,Egypt,Nigeria,Senegal,SouthAfrica\),inresponsetoamandategiventothembytheSummitoftheOrganisationofAfricanUnity\(OAU\)\.The37thSummitoftheOAUinJuly2001formallyadoptedthestrategicframeworkdocument\.

WhatisNEPAD?

NEPADisaVISIONandSTRATEGICFRAMEWORKFORAFRICA’SRENEWAL\.

NEPADisaprogrammeoftheAfricanUnion\(AU\),adoptedin2001byAfricanleaders,withtheprimaryobjectivesofpovertyeradication,promotionofsustainablegrowthanddevelopment,andtheempowermentofwomenthroughbuildinggenuinepartnershipsatcountry,regionalandgloballevels\.

NEPADisablueprintforAfrica’sdevelopmentinthe21stcenturyandprovidesuniqueopportunitiesforAfricatoaddressthecriticalchallengesfacingthecontinent,includingtheattainmentoftheMillenniumDevelopmentGoals\(MDGs\)andothercontinentalandinternationalagreedupongoals\.

WhyisthereaneedforNEPAD?

NEPADisdesignedtoaddressthecurrentchallengesfacingtheAfricancontinent\.IssuessuchastheescalatingpovertylevelsandunderdevelopmentofAfricancountriesandthecontinuedmarginalisationofAfricafromtheglobaleconomyneededaradicalnewintervention,spearheadedbyAfricanleaders\-thatwouldbringforthanewVisionforAfrica’sRenewal\.

WhatistheNEPADprogrammeofaction?

ItisadetailedactionplanderivedfromtheNEPADStrategicFrameworkdocumentandtheNEPADInitialActionPlan,adoptedbytheAfricanUnionSummitinDurbaninJune2002\.TheNEPADProgrammeofActionisaholistic,comprehensiveandintegratedsustainabledevelopmentinitiativefortherevivalofAfrica\.

WhataretheNEPADprimaryobjectives?

WhataretheprinciplesofNEPAD?

WhataretheelementsofNEPAD’sstrategicfocus?

Insum,NEPADcallsforadaptedpolicyreformsandincreasedinvestmentsintheareasofagricultureandfoodsecurity,scienceandtechnology,environment,tradeandmarketaccess,governance,infrastructure\(energy,transportandwatersanitation,andinformationandcommunicationtechnologies\),genderandcapacitydevelopment\.

HowistheNEPADprogrammegoverned?

TheworkoftheNEPADPlanningandCoordinatingAgency\-whichistaskedwiththeimplementationoftheNEPADProgramme\-isoverseenbytheNEPADHeadsofStateandGovernmentOrientationCommittee\(HSGOC\)andaSteeringCommittee\.

TheChairpersonoftheAfricanUnionCommissionexercisessupervisoryauthorityovertheAgency\.

Basically,themainfeaturesoftheAU/NEPADgovernancestructureincludetheAfricanUnionAssemblyofHeadsofStateandGovernment\(AUAssembly\);NEPADHeadsofStateandGovernmentOrientationCommittee\(HSGOC\)andaSteeringCommittee\.

WhatistheNEPADPlanningandCoordinatingAgency\(NEPADAgency\)?

TheNEPADAgencywasestablishedbythe14thAfricanUnionSummitdecisionastheinstitutionalvehicleforimplementingtheAfricanUnionDevelopmentagenda\.DesignatedasthetechnicalbodyoftheAfricanUnion,thecoremandateoftheNEPADAgencyistofacilitateandcoordinatetheimplementationofregionalandcontinentalpriorityprogrammesandprojectsandtopushforpartnerships,resourcemobilisationandresearchandknowledgemanagement\.

##### FinePhrase\.

NEPADstandsforNewPartnershipforAfrica’sDevelopment,aprogramoftheAfricanUnioncreatedbytheleadersoffiveinitiatingstates\(Algeria,Egypt,Nigeria,Senegal,SouthAfrica\)in1999andofficiallyadoptedbythe37thSummitoftheOAUin2001\.TheNEPADstrategicframeworkdocumentaddressescriticalissuesincludingpoverty,sustainablegrowth,andenhancingwomen’sempowermentthroughbuildinggenuinepartnershipsatalllevels\.ItisaimedatachievingMillenniumDevelopmentGoalsandglobal,continental,andinternationalagreedupongoals\.

##### 550M rewrite\.

\#FrequentlyAskedQuestions:NEPAD,VISION,andSTRATEGYforAfricanDevelopment

\#\#FoundationalConcepts

\*\*WhatisNEPAD?\*\*

TheNewPartnershipforAfrica’sDevelopment\(NEPAD\)isaglobalpartnershipdesignedforAfricandevelopment\.ThenamestandsforNewPartnershipforAfrica’sDevelopment,anditdefinestherationaleforNEPAD’sinterventionsandinitiatives\.

\*\*WhataretheoriginsofNEPAD?\*\*

Thestrategicframeworkdocumentwaspreparedbyfiveinitiatingstates\(Algeria,Egypt,Nigeria,Senegal,SouthAfrica\),whorespondedtoamandategiventothembytheSummitoftheOrganisationofAfricanUnity\(OAU\)\.ItwassignedbyleadersofthefivefoundingstatesandwasofficiallyadoptedbytheSummitinJuly2001\.

#### O\.0\.2837M: Table

Checkpoint:delphi\-glm\-837m\-sft\-llama3\-lr2e\-04\-ep10 Document:<urn:uuid:ccdd3db1\-ea0b\-4b2b\-885f\-f0b5243ec8b3\>

##### Original\.

"Predator"bacteria\(green\)surround"prey"bacteria\(red\)inthispetridishversionoftheSerengeti\.Ratherthaneatingtheirprey,however,predatorcellsreleaseachemicalthatactivatesasuicidegeneintheprey\.Preycellsalsoreleaseachemical,butonethatpromotessurvivalofthepredators\.Researchersgeneticallyprogrammedthecellsto"communicate"witheachotherinthiswayandfunctionasasyntheticecosystem\.Theartificialsystemactsasanexperimentalmodelandcanhelpusunderstandbehaviorsinmorecomplex,naturalecosystems\.July9,2008

CourtesyofHaoSong,DukeUniversity\.

FullStory:\(http://publications\.nigms\.nih\.gov/computinglife/predator\.htm\)

##### FinePhrase\.

\|CellType\|Predator\|Prey\|

\|\-\-\-\-\-\-\-\-\-\-\-\|\-\-\-\-\-\-\-\-\-\-\|\-\-\-\-\-\-\-\|

\|Green\|Red\|No\|

\|1Death\|1Survival\|0\|

\|1Death\|0Deactivation\|1\|

\|Deactivates\|Deactivates\|Deactivates\|

Question:Whatarethepotentialadvantagesanddisadvantagesofcreatinganengineeredsyntheticecosystem?

Answer:Thepotentialadvantagesincludegaininginsightsintocomplexbehaviorsinecosystems,testinghypotheses,anddevelopingmoreefficientmethodsforbioremediationandbiotechnology\.However,thedisadvantagesmaycompriserisksinunintendedconsequencesofintroducingalienlifeforms,unforeseensideeffectsonnativeecosystems,andchallengesinlong\-termsustainabilityandcontroloftheengineeredecosystem\.

##### 837M rewrite\.

\|Aspect\|Details\|

\|\-\-\-\-\-\-\-\-\|\-\-\-\-\-\-\-\-\-\|

\|\*\*Topic\*\*\|Predatorbacteria\(green\)surrounded"prey"bacteria\(red\)inapetridish\|

\|\*\*Purpose\*\*\|Studybehaviorofpredatorsinartificialecosystems\|

\|\*\*Technique\*\*\|Involvesreleasingachemicalthatactivatesasuicidegeneinprey\|

\|\*\*CellDesign\*\*\|Cellscommunicateviasuicidalgeneexpression\|

\|\*\*ExperimentalSetup\*\*\|Petridishwithpreyandpredatorcells\|

\|\*\*KeyFinding\*\*\|Predatorcellsreleasechemicalthatactivatessuicidegene\|

\|\*\*ResearchSignificance\*\*\|Helpsunderstandnaturalecosystembehaviors\|

\|\*\*Application\*\*\|Understandingnaturalecosystembehaviorincomplexenvironments\|

\*\*Question:\*\*Howdoesthe"Predator"bacteria’ssuicidegenefunctioncontributetoitsecosystem\-basedbehavior?

\*\*Answer:\*\*Thesuicidegeneinquestionfunctionsasachemicalthatactivatesasuicidegeneinthepreycells,butthechemicalitselfpromotessurvivalofthepredatorsratherthansuicide\.Thisdualroleallowsthebacteriatobotheatpreycellsandreleaseachemicalthatpromotestheirsurvival,promotingecosystemsustainability\.

#### O\.0\.31B: Table

Checkpoint:delphi\-glm\-sft\-llama3\-lr1e\-04\-ep15 Document:<urn:uuid:48cb395a\-8ea3\-4ee8\-a473\-4e98f269603f\>

##### Original\.

DefinitionfromWiktionary,thefreedictionary

Alternativeforms

\-acousticks\(obsolete\)

Audio\(US\)\(file\)

See\-icsregardingthetreatmentofsuchnounsassingular\.

\-Thephysicalqualityofaspaceforperformingmusic\.

\-\(physics\)Thescienceofsounds,teachingtheirnature,phenomenaandlaws\.

\-Acoustics,then,orthescienceofsound,isaveryconsiderablebranchofphysics\.\-SirJohnHerschel\.

Usagenotes

\-Thesciencewaspreviouslydividedbysomewritersintodiacoustics,whichexplainsthepropertiesofsoundscomingdirectlyfrom\(sic\!Webster\)theear;andcatacoustica,whichtreatsofreflectedsoundsorechoes\.Thisdivisionisnowobsolete\.

Relatedterms

qualityofaspacefordoingmusic

physics:ascienceofsounds

\-Thetranslationsbelowneedtobecheckedandinsertedaboveintotheappropriatetranslationtables,removinganynumbers\.Numbersdonotnecessarilymatchthoseindefinitions\.SeeinstructionsatHelp:Howtochecktranslations\.

##### FinePhrase\.

\*\*Table:\*\*

\|\*\*Subject\*\*\|\*\*Description\*\*\|

\|\-\-\-\-\-\-\-\-\-\-\-\|\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\|

\|Audio\|Definedasthecollectionofsounds\|

\|thatamancanhear\|intheuniverse\.Inthissense\|

\|\-\-\-\-\-\-\-\-\-\-\-\|\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\|

\|Physics\|Thescientificstudyofmotion\|

\|\-\-\-\-\-\-\-\-\-\-\-\|\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\|

\|Sound\|Thevibrationwhichoccurswhen\|

\|asoundwavehappens\|objectsofsufficientmass\|

\|\-\-\-\-\-\-\-\-\-\-\-\|\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\|

\|Reflected\|Theechoproducedwhenasound\|

\|wavebouncesback\|fromareflectivesurface\|

\|\-\-\-\-\-\-\-\-\-\-\-\|\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\|

\|Catacoustics\|Abranchofphysicsdealing\|

\|withreflectedsoundsorechoes\|

\*\*QuestionandAnswerPair:\*\*

\#\#Question:

Whatbranchofphysicsdealswiththestudyofreflectionsorechoesinspace,leadingtotheconceptofacoustics?

\#\#Answer:

Acoustics\.

##### 1B rewrite\.

\|Aspect\|Details\|

\|:\-\-\-\-\-\-\-\|:\-\-\-\-\-\-\-\-\|

\|Definition\|Thephysicalqualityofaspaceforperformingmusic\|

\|Alternativeforms\|Acoustics\(obsolete\)\|

\|Audio\(US\)\|File\|

\|Definition\|Thescienceofsounds,teachingtheirnature,phenomena,andlaws\|

\|Usage\|Theywerepreviouslydividedbysomewritersintodiacoustics;explainspropertiesofsoundscomingfromtheear\|

\|Relatedterms\|Qualityofaspacefordoingmusic\|

\|Definition\|Thescienceofsounds,teachingtheirnature,phenomena,andlaws\|

\|Usage\|Werepreviouslydividedbysomewritersintodiacoustics;explainsdivisionnowobsolete\|

\|Relatedterms\|Qualityofaspacefordoingmusic\|

\|Definition\|Thescienceofsounds,teachingtheirnature,phenomena,andlaws\|

\|Usage\|Wereformerlydividedbysomewritersintodiacoustics;explainsdivisionnowobsolete\|

\|Relatedterms\|Qualityofaspacefordoingmusic\|

\*\*Question:\*\*Howdoestheusagenotefor"sound"differfromhowitwashistoricallydefined?

\*\*Answer:\*\*Theusagenotestatesthat"sound"waspreviouslydividedintodiacoustics,explainsthatthisdivisionisnowobsolete,andreferences"qualityofaspacefordoingmusic"asarelatedterm\.

#### O\.0\.41B: FAQ

Checkpoint:delphi\-glm\-sft\-llama3\-lr1e\-04\-ep15 Document:<urn:uuid:4fd3037a\-b829\-4827\-8fa6\-c3aeb3e57890\>

##### Original\.

PainManagementandBioelectricTherapy

\-Introductiontobioelectrictherapy

\-Whatconditionsaretreatedwithbioelectrictherapy?

\-Howeffectiveisbioelectrictherapyforpainrelief?

\-Whathappensduringbioelectrictherapy?

\-Whatarethesideeffectsofbioelectrictherapy?

\-Howoftenshouldpeoplegetbioelectrictherapy?

\-Howdopeopleprepareforbioelectrictherapy?

IntroductiontoBioelectricTherapy

Bioelectrictherapyisasafe,drug\-freetreatmentoptionforpeopleinpain\.Itisusedtotreatsomechronicpainandacutepainconditions\.Itrelievespainbyblockingpainmessagestothebrain\.Whenyouareinjured,painreceptorssendamessagetothecentralnervoussystem\(thebrainandspinalcord\)\.Themessageisregisteredaspainbycertaincellsinthebody\.Usingbioelectriccurrents,bioelectrictherapyrelievespainbyinterruptingpainsignalsbeforetheyreachthebrain\.Bioelectrictherapyalsopromptsthebodytoproduceendorphinswhichhelptorelievepain\.

WhatConditionsAreTreatedWithBioelectricTherapy?

Bioelectrictherapycanbeusedtotreatchronicandacutepainconditionsincluding:

\-Complexregionalpainsyndrome,alsoknownasreflexsympatheticdystrophyorRSD\.

\-Backpain\.

\-Musclepain\.

\-Headachesandmigraines\.

\-Disordersofbloodflowintheupperandlowerlimbs\.

\-Temporomandibularjoint\(TMJ\)syndrome\(whichaffectsthejaw\)\.

\-Disordersofthenervoussystem,suchasdiabeticneuropathy\.

\-Painandulcersoftheskinresultingfrompoorcirculationorscleroderma\(achronicconditionthatcancausethickeningorhardeningoftheskin\)\.

Bioelectrictherapyisn’trightforeveryone\.Itisnotrecommendedforpeoplewho:

\-Haveapacemaker\.

\-Arepregnant\.

\-Havethrombosis\(bloodclotsinthearmsorlegs\)\.

\-Haveabacterialinfection\.

ChronicPain/BackPain

Findtipsandadvancesintreatment\.

##### FinePhrase\.

Bioelectrictherapyisanon\-invasive,drug\-freetreatmentthatcanhelpmanagechronicpainandbackpain,reduceinflammation,andspeedupthehealingprocess\.

##### 1B rewrite\.

\#FrequentlyAskedQuestions:PainManagementandBioelectricTherapy

\#\#UnderstandingBioelectricTherapy

\*\*Whatisbioelectrictherapy?\*\*

Bioelectrictherapyisasafe,drug\-freetreatmentoptionforpeopleinpain\.Itisusedtotreatsomechronicpainandacutepainconditions\.Itrelievespainbyblockingpainmessagestothebrain\.

\*\*Whatconditionsdoesbioelectrictherapytreat?\*\*

Bioelectrictherapytreatssomechronicpainconditionsandacutepainconditions\.Itrelievespainbyblockingpainmessagestothebrain\.

\*\*Howeffectiveisbioelectrictherapyforpainrelief?\*\*

Bioelectrictherapyishighlyeffectiveforpainrelief\.Itworksbyinterruptingpainsignalsbeforetheyreachthebrain,ratherthanstimulatingthebraintoproducemorphineorendorphinsthemselves\.

\*\*Whathappensduringbioelectrictherapy?\*\*

Duringbioelectrictherapy,electrodesareplacedontheskintodeliverelectricalcurrent\.Thiscurrentthenpassesthroughthebody,interruptingpainsignalsthatwouldotherwisereachthebrain\.Theaffectedareaisthenexposedtothetherapeuticbioelectriccurrents\.

\#\#SideEffectsandPrecautions

\*\*Whatarethesideeffectsofbioelectrictherapy?\*\*

Whilemostbioelectrictherapiesaresafeandeffective,somesideeffectscanoccur\.Thesecanincludeskinreactions,infections,andpotentialinjurytothebrainornervoussystem\.

\*\*Howoftenshouldpeoplegetbioelectrictherapy?\*\*

Peoplewhoreceivebioelectrictherapyshouldtakeitasdirectedtoensureproperpainrelief\.

\#\#TreatmentDuration

\*\*Whenshouldpeoplegetbioelectrictherapy?\*\*

Peoplewhoreceivebioelectrictherapyshouldtakeitasdirectedbecauseitinterruptspainsignalsbeforetheyreachthebrain\.Thetreatmentalsopromptsthebodytoproduceendorphins,whichhelptorelievepain\.

\#\#SpecialConsiderations

\*\*Arebioelectrictherapyandpainmanagementexactlythesamething?\*\*

No\.Whiletheysharemanytreatments,bioelectrictherapyandpainmanagementaredistinct\.Bioelectrictherapyuseselectricalcurrentstoblockpainmessages,whilepainmanagementusesmedicationtomanagethepain\.

\*\*Whoshouldnottakebioelectrictherapy?\*\*

Peoplewhohaveapacemakershouldnottakebioelectrictherapy\.Peoplewhoarepregnantshouldalsoavoidbioelectrictherapy\.Thesepeoples’bodiesmaynotreacttotheelectricalcurrentasitpassesthroughthem\.

\#\#AdditionalResources

\*\*WherecanIfindmoreinformationaboutbioelectrictherapy?\*\*

Additionalinformationcanbefoundintheresourcesforpainmanagementandbioelectrictherapy\.Topicsincludehistory,typesofpainconditionstreated,effectivetreatmentmethods,sideeffects,andsafetyconcerns\.

#### O\.0\.53B: Table

Checkpoint:delphi\-glm\-3b\-sft\-llama3\-lr5e\-05\-ep15 Document:<urn:uuid:3e09eef3\-39c3\-41eb\-a404\-bb93536daa7f\>

##### Original\.

WhenYourChildHasPharyngitisorTonsillitis

\|Painorrednesscanoccurintheseareas\.\|

WhatCausesPharyngitisorTonsillitis?

Mostcommonly,pharyngitisandtonsillitisarecausedbyaviralorbacterialinfection\.

WhatAretheSymptomsofPharyngitisorTonsillitis?

Themainsymptomofbothconditionsisasorethroat\.Yourchildmayalsohaveafever,rednessorswellingofthethroat,andtroubleswallowing\.

HowIsPharyngitisorTonsillitisDiagnosed?

Thehealthcareproviderwillexamineyourchild’sthroat\.Thehealthcareprovidermightswab\(wipe\)yourchild’sthroat\.Thisswabwillbetestedforthebacteriathatcausesaninfectioncalledstrepthroat\.Ifneeded,abloodtestcanbedonetocheckforaviralinfection,suchasmononucleosis\.

HowIsPharyngitisorTonsillitisTreated?

Ifyourchild’ssorethroatiscausedbyabacterialinfection,thehealthcareprovidermayprescribeantibiotics\.Otherwise,youcantreatyourchild’ssorethroatathome\.Todothis:

Giveyourchildacetaminophenoribuprofentoeasethepain\.Don’tgiveyourchildaspirintorelieveafever\.UsingaspirintotreatafeverinchildrencouldcauseaseriousconditioncalledReye’ssyndrome\.

Giveyourchildcoolliquidstodrink\.

Haveyourchildgarglewithwarmsaltwaterifithelpsrelievepain\.Anover\-the\-counterthroatnumbingspraymayalsohelp\.

WhatAretheLong\-TermConcerns?

Ifyourchildhasfrequentsorethroats,takehimorhertoseeahealthcareprovider\.Removingthetonsilsmayhelprelieveyourchild’srecurringproblems\.

Callyourchild’shealthcareproviderrightawayifyourchildhasanyofthefollowing:

##### FinePhrase\.

\|Heading\|Description\|

\|\-\-\-\-\-\-\-\-\-\|\-\-\-\-\-\-\-\-\-\-\-\-\-\|

\|WhenYourChildHasPharyngitisorTonsillitis\|

\|WhattoKnowAboutPharyngitisorTonsillitis\|

\|CausesofPharyngitisorTonsillitis\|

\|SymptomsofPharyngitisorTonsillitis\|

\|DiagnosisProcessforPharyngitisorTonsillitis\|

\|HowtoTreatPharyngitisorTonsillitis\|

\|Long\-TermConcerns\|

WhatAretheLong\-TermConcernsforPharyngitisorTonsillitis?

##### 3B rewrite\.

\|Category\|Details\|

\|\-\-\-\-\-\-\-\-\-\-\|\-\-\-\-\-\-\-\-\-\|

\|\*\*Condition\*\*\|Pharyngitis/tonsillitis\|

\|\*\*Cause\*\*\|Viralorbacterialinfection\|

\|\*\*Symptoms\*\*\|Sorethroat,fever,redness/swelling,difficultyswallowing\|

\|\*\*KeySymptom\*\*\|Sorethroat\|

\|\*\*AlternativeSymptoms\*\*\|Fever,redness/swelling,swallowingproblems\|

\|\*\*Diagnosis\*\*\|Medicalproviderexaminesthroatforstrepthroat;bloodtestforviralinfection\(mononucleosis\);ortonsilremovalifneeded\|

\|\*\*Treatment\*\*\|Antibioticsforbacterialinfections;acetaminophenoribuprofenforpain;coolliquidsforsorethroat;garglingwithsaltwaterforswallowing\|

\|\*\*Self\-CareSteps\*\*\|Removetonsilsifsorethroatiscausedbybacteria;avoidaspirinforfever;usethroatnumbingsprayifneeded;avoidthroatdrying\|

\|\*\*Long\-TermConcerns\*\*\|Frequentsorethroats;removaloftonsilsforrecurringproblems\|

\*\*Question:\*\*Whyisremovingthetonsilsrecommendedforchildrenwithrecurringpharyngitisortonsillitis?

\*\*Answer:\*\*Removingthetonsilshelpsrelievechildren’srecurringproblemsbecausethetonsilsserveasareservoirforbacteria,andremovingthemcreatesaclearpathforhealing\.

#### O\.0\.63B: Tutorial

Checkpoint:delphi\-glm\-3b\-sft\-llama3\-lr5e\-05\-ep15 Document:<urn:uuid:66166475\-112f\-43f3\-a5c1\-215ae0e2a518\>

##### Original\.

Nowmorethanfivedecadesold,HerbertHoover’s1959statementtotheBoardofTrusteesofStanfordUniversityonthepurposeandscopeoftheHooverInstitutioncontinuestoguideanddefineitsmissioninthetwenty\-firstcentury:

"ThisInstitutionsupportstheConstitutionoftheUnitedStates,itsBillofRightsanditsmethodofrepresentativegovernment\.Bothoursocialandeconomicsystemsarebasedonprivateenterprisefromwhichspringsinitiativeandingenuity\.\.\.\.OursisasystemwheretheFederalGovernmentshouldundertakenogovernmental,socialoreconomicaction,exceptwherelocalgovernment,orthepeople,cannotundertakeitforthemselves\.\.\.\.TheoverallmissionofthisInstitutionis,fromitsrecords,torecallthevoiceofexperienceagainstthemakingofwar,andbythestudyoftheserecordsandtheirpublication,torecallman’sendeavorstomakeandpreservepeace,andtosustainforAmericathesafeguardsoftheAmericanwayoflife\.ThisInstitutionisnot,andmustnotbe,amerelibrary\.Butwiththesepurposesasitsgoal,theInstitutionitselfmustconstantlyanddynamicallypointtheroadtopeace,topersonalfreedom,andtothesafeguardsoftheAmericansystem\."

Theprinciplesofindividual,economic,andpoliticalfreedom;privateenterprise;andrepresentativegovernmentwerefundamentaltothevisionoftheInstitution’sfounder\.Bycollectingknowledge,generatingideas,anddisseminatingboth,theInstitutionseekstosecureandsafeguardpeace,improvethehumancondition,andlimitgovernmentintrusionintothelivesofindividuals\.

##### FinePhrase\.

I\.Introduction

\-AbriefoverviewoftheimportanceoftheHooverInstitution

\-Themissionandpurposeoftheinstitution

II\.NumberedStepstoUnderstandtheInstitution’sMission

III\.UnderstandingIndividualFreedom

\-Whatitmeanstobearesponsiblecitizen

IV\.TheRoleofGovernmentinSociety

\-Providingpublicgoods

\-Encouraginginnovation

V\.EducationandItsImportance

\-Thevalueofknowledgeinpersonalandsocietaldevelopment

VI\.ALifeofLiberty

\-Fosteringcreativity

\-Promotingself\-reliance

VII\.Conclusion

\-Recapoftheinstitution’sprinciples

\-Importanceofindividualfreedom,economicfreedom,politicalfreedom,andpreservingpersonalfreedom

VIII\.FurtherResources

\-TheofficialwebsiteoftheHooverInstitution

\-Othersourcesofinformationontheinstitution

##### 3B rewrite\.

\#UnderstandingHerbertHoover’s1959StatementonthePurposeandScopeoftheHooverInstitution:ATutorial

\#\#BackgroundandContext

\-\*\*Date:\*\*December5,1959

\-\*\*Historicalcontext:\*\*Establishedjustoverfivedecadesearlier

\-\*\*Keyphrase:\*\*"\.\.\.supporttheConstitutionoftheUnitedStates,itsBillofRights,andmethodofrepresentativegovernment"

\-\*\*Historicalcontext:\*\*TheseprinciplesguidedtheInstitute’smissionthroughthetwentiethcentury\.

\#\#Step1:ReviewtheInstitution’sFoundingPrinciples

TheHooverInstitutionoperatesunderthefollowingprinciples:

1\.\*\*Individualfreedom:\*\*Ensuresindividualrightsandfreedomsareprotected

2\.\*\*Economicfreedom:\*\*Supportsprivateenterprisethroughfreeinquiryandknowledgegeneration

3\.\*\*Politicalfreedom:\*\*Supportsrepresentativegovernmentthroughtheballotboxandpublicdebate

4\.\*\*Respectforlaw:\*\*Guaranteeslegalauthorityandinstitutionalintegrity

5\.\*\*SustainingAmericanWayofLife:\*\*Emphasizeslocalself\-determinationandsustainablepractices

\#\#Step2:UnderstandtheInstitution’sMission

ThemissionoftheHooverInstitutionis:

\-\*\*Recollectthevoiceofexperienceagainstthemakingofwar\*\*

\-\*\*Recollectman’seffortstomakeandpreservepeace\*\*

\-\*\*SustainforAmericathesafeguardsoftheAmericanwayoflife\*\*

\#\#Step3:RecognizetheInstitute’sGoal

Themissionstatementsetsthefoundationforaction:

\-Preventingwar

\-Improvinghumanconditions

\-Limitinggovernmentintrusionintoindividuallives

\#\#Step4:UnderstandtheRoleoftheBoard

TheBoardofTrusteescontinuestodefinetheInstitution’spurpose:

\-\*\*Focus:\*\*Today,throughleadershipandexpertise

\-\*\*Mission:\*\*KeeptheInstitution’smissionclearthroughconsistentconsiderationandaction

\#\#Step5:AccesstheInstitute’sRecords

Tosupportthemission,theBoardofTrusteesmaintains:

\-\*\*Scope:\*\*TherecordsoftheHooverInstitution

\-\*\*Scope:\*\*Theycollectknowledge,generateideas,anddisseminatebothforeducationalandscholarlypurposes

\-\*\*Purpose:\*\*Secureandsafeguardpeace,improvethehumancondition,andlimitgovernmentintrusionintoindividuallives

\#\#Step6:ApplyThesePrinciplesinDailyLife

\-\*\*Individualfreedom:\*\*Protecttherightsandfreedomsyouenjoy

\-\*\*Economicfreedom:\*\*Supportprivateenterprisethroughfreeenterprise

\-\*\*Politicalfreedom:\*\*Supportrepresentativegovernmentthroughtheballotboxandpublicdebate

\-\*\*Respectforlaw:\*\*Ensurelegalauthorityandinstitutionalintegrity

\-\*\*SustainingtheAmericanwayoflife:\*\*Emphasizelocalself\-determinationandsustainablepractices

\#\#AdditionalResources

\-Source:HerbertHoover’s1959StatementonthePurposeandScopeoftheHooverInstitution:ATutorial

\-Source:Washington,D\.C\.\(December5,1959\)\-HerbertHoover’s1959AnnualReport,StanfordUniversity

\-Source:http://www\.hoover\.org/archive/statement/\(accessedDecember5,2019\)

#### O\.0\.71B: Math

Checkpoint:delphi\-glm\-sft\-llama3\-lr1e\-04\-ep15 Document:<urn:uuid:63c9a260\-1d93\-4db5\-bbb6\-9126d821ed64\>

##### Original\.

ZomeTools\-ZomeSystems

TheZomeTools\-ZomeSystem

,isafantasticGeometryEducationsystem\.MadeofprecisionmoldedABSplastic,theZomeToolsGeometrykitsareversatilemanipulativetoolsforproblemsolvingandhands\-onexplorationofmath,geometryandscienceconceptsforfirstgradersthroughgradstudents\.Colorandshapecodedpartsintegratetheconceptsofshape,numberandvectorintoonecoherentgeometrysystem\.Theresultisthebesteducationalgeometryconstructionsystemonthemarket\.ZomeSystem/ZomeToolscomponentsaredesignedtoincorporatetheFibonaccisequenceandtheGoldenProportion\-theverysameproportionsthatgovernthegrowthofcells,crystalsandothernaturalforms\.

connectornodeprovides62buildabledirectionsinspace\.ThisallowsthousandsofgeometricalZomestructures

thatcan’tbebuiltinanyothersystem\.

WiththehelpofZometoolsyoucancontinuetoexplore:Whysomanyeverydayitemslikeballs,pizzas,andcanlidsareround\.ThehowsandwhysofthedevelopmentofournumbersystembytheancientGreeks\.Whygeometricshapesshowupsoofteninart,scienceandboardgames\.Whatpropertiesattributedtothespiralmakeitthemostwidespreadshapeinnature,fromembryosandhaircurlstohurricanesandgalaxies\.Howthehumanbodysharesthedesignofacornplantandthesolarsystem\.HowasnowflakeislikeStonehenge,andabeehivelikeacalendar\.Howourtenfingersholdthesecretsofbothalobsterandacathedral\.Zometoolsprovideanintroductiontothegeometriccodeofnatureandtheuniverse\.

##### FinePhrase\.

Problem:

Jennafound2ZomeToolssetsfromZomeToolsatthestore\.Sheboughtthetotalamountofsetssoshebought10sets\.Then,shetraded5setswithDanielforStarLego\.Afterthat,shespent$20tobuymoreZomeToolssets\.IfJennahas2ZomeToolssetsleft,howmuchdideachZomeToolssetcostherafterpurchasingfromZomeToolsandtrading?

##### 1B rewrite\.

\*\*Problem:\*\*

AteacherisbuildingageometricZomeToolskitforherfirst\-gradeclass\.Theteacherhas62buildabledirectionsinspace,andeachdirectioncontains373built\-indirections\.Shewantstoprovideherstudentswithaccesstoasmanybuildabledirectionsaspossible,notingthat1directioncontains5built\-indirections\.Ifsheallocates4sectionsofspaceperdirection,howmanytotalbuilt\-indirectionswilltheclasshaveaccessto?

\*\*Solution:\*\*

Step1:Identifythetotalnumberofbuilt\-indirections\.

Totaldirections=62

Step2:Calculatethenumberofbuilt\-indirectionsavailable\.

62x5=252built\-indirections

Step3:Determinehowmanyofthesedirectionscanbeused\.

252x5=258directions

Step4:Calculatetheremainingnumberofdirectionstobeused\.

258\-258=48directions

Step5:Determinehowmanystudentscanaccessasmanybuilt\-indirectionsaspossible\.

48x5=144students

Answer:Theclasswillhave258built\-indirectionsavailable,and144studentswillbeabletoaccessasmanyofthesedirectionsaspossible\.

#### O\.0\.83B: Math

Checkpoint:delphi\-glm\-3b\-sft\-llama3\-lr5e\-05\-ep15 Document:<urn:uuid:d29bb74f\-45d0\-452c\-b599\-d774fedf25bd\>

##### Original\.

ColoredmapcompiledbymerginggeologicandtopographicmapsoftheUnitedStatesthroughcomputerprocessingandenhancementwithlinkstopanoramicmovieusingQuickTime,descriptionsoffeatures,andagameentitledPuzzleofRegions\.

TheUSGSprovidesmap\-on\-demand\(MOD\)printingforselectedinfrequentlyrequestedgeologicandotherthememapproductsprovidinghigh\-qualitypaper\-copymapsasneeded\.Sitelinkstoavailablemapproductswiththumbnailviewsandhowtoorder\.

Thispublication\(Open\-FileReport95\-479\)isateacher’sguidetotwoU\.S\.GeologicalSurveymapsthatshowgravityandmagneticdataincontourformsuperimposedonaLANDSATsatelliteimageoftheSanFranciscoBayarea\.

Techniquesofcomputer\-basedrepresentationsofarealgeologytoconstructthree\-dimensionalgeologicmapsthatretainalltheinformationoftraditionalgeologicmapswhileprovidingamoredetailedpictureofthesubsurface\.

MainpageforaccessinglinksforinformationanddataontheSanFranciscoBayestuaryanditswatershedwithlinkstohighlights,water,biology,wetlands,hazards,digitalmaps,geologicmapping,winds,bathymetryandoverviewoftheBay\.

##### FinePhrase\.

Whatisthevalueofdividing50by2x3andthenmultiplyingthisresultby15?

##### 3B rewrite\.

\*\*Problem:\*\*

AgeologicalmapshowsgeologicformationsandtopographicfeaturesoftheUnitedStates\.Themapshows24differenttypesofgeologicformationsand5differenttopographiczones\.Additionally,themapnames3majormountainsystemsand4majordrainagesystems\.Ifthemap’spublisherrequiresateacher’sguideto2U\.S\.GeologicalSurveymapsthatshowgravityandmagneticdataincontourformsuperimposedonaLANDSATsatelliteimageoftheSanFranciscoBayarea,andthe2ndmapcosts$24percopy,howmanytotalcopiesoftheteacher’sguidemustthepublisherordertosellenoughcopiestocoverthetotalcostofthe2maps?

\*\*Solution:\*\*

Step1:Identifythenumberofdifferentgeologicformationsonthe24typesofformationsmap\.

Totalformations=24

Step2:Identifythenumberofdifferenttopographiczonesonthe5differenttopographiczonesmap\.

Topographiczones=5

Step3:Calculatethetotalnumberofformationsandtopographiczonesonthe24typesofformationsmap\.

24typesx3formationspertype=72formationsandtopographiczones

Step4:Calculatethetotalnumberofcopiesofthe24typesofformationsmapneeded\.

72formationsx2copiesperformation=240copies

Step5:Calculatethetotalcostofthe2maps\.

240copiesx$24percopy=$720

Thepublishermustorder720copiesoftheteacher’sguidetosellenoughcopiestocoverthetotalcost\.

#### O\.0\.91B: Tutorial

Checkpoint:delphi\-glm\-sft\-llama3\-lr1e\-04\-ep15 Document:<urn:uuid:66166475\-112f\-43f3\-a5c1\-215ae0e2a518\>

##### Original\.

Nowmorethanfivedecadesold,HerbertHoover’s1959statementtotheBoardofTrusteesofStanfordUniversityonthepurposeandscopeoftheHooverInstitutioncontinuestoguideanddefineitsmissioninthetwenty\-firstcentury:

"ThisInstitutionsupportstheConstitutionoftheUnitedStates,itsBillofRightsanditsmethodofrepresentativegovernment\.Bothoursocialandeconomicsystemsarebasedonprivateenterprisefromwhichspringsinitiativeandingenuity\.\.\.\.OursisasystemwheretheFederalGovernmentshouldundertakenogovernmental,socialoreconomicaction,exceptwherelocalgovernment,orthepeople,cannotundertakeitforthemselves\.\.\.\.TheoverallmissionofthisInstitutionis,fromitsrecords,torecallthevoiceofexperienceagainstthemakingofwar,andbythestudyoftheserecordsandtheirpublication,torecallman’sendeavorstomakeandpreservepeace,andtosustainforAmericathesafeguardsoftheAmericanwayoflife\.ThisInstitutionisnot,andmustnotbe,amerelibrary\.Butwiththesepurposesasitsgoal,theInstitutionitselfmustconstantlyanddynamicallypointtheroadtopeace,topersonalfreedom,andtothesafeguardsoftheAmericansystem\."

Theprinciplesofindividual,economic,andpoliticalfreedom;privateenterprise;andrepresentativegovernmentwerefundamentaltothevisionoftheInstitution’sfounder\.Bycollectingknowledge,generatingideas,anddisseminatingboth,theInstitutionseekstosecureandsafeguardpeace,improvethehumancondition,andlimitgovernmentintrusionintothelivesofindividuals\.

##### FinePhrase\.

I\.Introduction

\-AbriefoverviewoftheimportanceoftheHooverInstitution

\-Themissionandpurposeoftheinstitution

II\.NumberedStepstoUnderstandtheInstitution’sMission

III\.UnderstandingIndividualFreedom

\-Whatitmeanstobearesponsiblecitizen

IV\.TheRoleofGovernmentinSociety

\-Providingpublicgoods

\-Encouraginginnovation

V\.EducationandItsImportance

\-Thevalueofknowledgeinpersonalandsocietaldevelopment

VI\.ALifeofLiberty

\-Fosteringcreativity

\-Promotingself\-reliance

VII\.Conclusion

\-Recapoftheinstitution’sprinciples

\-Importanceofindividualfreedom,economicfreedom,politicalfreedom,andpreservingpersonalfreedom

VIII\.FurtherResources

\-TheofficialwebsiteoftheHooverInstitution

\-Othersourcesofinformationontheinstitution

##### 1B rewrite\.

\#UnderstandingtheHerbertHooverInstitution:AGuidetoItsLegacyandCorePrinciples

\#\#BackgroundandContext

\*\*Step1:IdentifytheInstitution’sHistoricalSignificance\*\*

\-\*\*Established:\*\*1959

\-\*\*OriginalPurpose:\*\*Purposeedfor5decadestopreservetheInstitution’sroleinAmericandemocracy

\-\*\*CorePurpose:\*\*SupportforitsConstitution,BillofRights,andmethodofrepresentativegovernment

\-\*\*HistoricalContext:\*\*Foundedin1959byPresidentHerbertHoover,thisinstitutioncontinuestoserveAmericatoday

\#\#Step2:UnderstandtheMissionStatement\*\*

\-\*\*CorePurpose:\*\*Recallthevoiceofexperienceagainstthemakingofwar

\-\*\*MissionStatement:\*\*Avoidcreatingunnecessarygovernmental,social,oreconomicactions

\-\*\*Approach:\*\*Studytherecordstounderstandeffortstomakeandpreservepeace,andbystudy,toilluminatetheboundsofAmericanwayoflife

\#\#Step3:RecognizetheHistoricalSignificanceofIndividualLiberty\*\*

\-\*\*CorePrinciple:\*\*ThecoremissionofthisInstitutionistorecallthevoiceofexperienceagainstthemakingofwar

\-\*\*Approach:\*\*Avoidcreatingunnecessarygovernmental,social,oreconomicactions

\-\*\*HistoricalContext:\*\*Enablevoluntary,collaborativeengagementwithpeace

\-\*\*EmergencyResponse:\*\*AcknowledgethattheInstitutioncannotoperateeffectivelywithoutpropercommunitysupport

\#\#Step4:UnderstandtheRoleoftheIndividualinAmericanSociety\*\*

\-\*\*CorePrinciple:\*\*Theindividualmustplayameaningfulroleinshapinghisorherownsociety

\-\*\*Approach:\*\*FosterpersonalfreedomandthesafeguardsoftheAmericansystemasprimarygoals

\-\*\*HistoricalContext:\*\*Rememberthatgovernmentintrusionintopersonaldecisionsisnotacceptable

\#\#Step5:ApplyCorePrinciplestoContemporaryChallenges\*\*

\-\*\*Purpose:\*\*RestoretheInstitute’sroleinAmericandemocracyby:

\-Recreatingthevoicesofexperienceagainstthemakingofwar

\-Creatingpeacethroughresearch,education,andpublicengagement

\-Limitinggovernmentintrusionintoindividuallives

\-StrengtheningtheAmericanwayoflifethroughindividualfreedomandlimitedgovernment

\#\#KeyTakeaway

TheHerbertHooverInstitution’slegacygoesbeyondpreservingliterature\.Bycreatingaspaceforcollectiveknowledge,learning,andcollaborativeengagement,itseekstosecureandsafeguardpeace,improvethehumancondition,andlimitgovernmentintrusionintoindividuallives\.

#### O\.0\.103B: FAQ

Checkpoint:delphi\-glm\-3b\-sft\-llama3\-lr5e\-05\-ep15 Document:<urn:uuid:4fd3037a\-b829\-4827\-8fa6\-c3aeb3e57890\>

##### Original\.

PainManagementandBioelectricTherapy

\-Introductiontobioelectrictherapy

\-Whatconditionsaretreatedwithbioelectrictherapy?

\-Howeffectiveisbioelectrictherapyforpainrelief?

\-Whathappensduringbioelectrictherapy?

\-Whatarethesideeffectsofbioelectrictherapy?

\-Howoftenshouldpeoplegetbioelectrictherapy?

\-Howdopeopleprepareforbioelectrictherapy?

IntroductiontoBioelectricTherapy

Bioelectrictherapyisasafe,drug\-freetreatmentoptionforpeopleinpain\.Itisusedtotreatsomechronicpainandacutepainconditions\.Itrelievespainbyblockingpainmessagestothebrain\.Whenyouareinjured,painreceptorssendamessagetothecentralnervoussystem\(thebrainandspinalcord\)\.Themessageisregisteredaspainbycertaincellsinthebody\.Usingbioelectriccurrents,bioelectrictherapyrelievespainbyinterruptingpainsignalsbeforetheyreachthebrain\.Bioelectrictherapyalsopromptsthebodytoproduceendorphinswhichhelptorelievepain\.

WhatConditionsAreTreatedWithBioelectricTherapy?

Bioelectrictherapycanbeusedtotreatchronicandacutepainconditionsincluding:

\-Complexregionalpainsyndrome,alsoknownasreflexsympatheticdystrophyorRSD\.

\-Backpain\.

\-Musclepain\.

\-Headachesandmigraines\.

\-Disordersofbloodflowintheupperandlowerlimbs\.

\-Temporomandibularjoint\(TMJ\)syndrome\(whichaffectsthejaw\)\.

\-Disordersofthenervoussystem,suchasdiabeticneuropathy\.

\-Painandulcersoftheskinresultingfrompoorcirculationorscleroderma\(achronicconditionthatcancausethickeningorhardeningoftheskin\)\.

Bioelectrictherapyisn’trightforeveryone\.Itisnotrecommendedforpeoplewho:

\-Haveapacemaker\.

\-Arepregnant\.

\-Havethrombosis\(bloodclotsinthearmsorlegs\)\.

\-Haveabacterialinfection\.

ChronicPain/BackPain

Findtipsandadvancesintreatment\.

##### FinePhrase\.

Bioelectrictherapyisanon\-invasive,drug\-freetreatmentthatcanhelpmanagechronicpainandbackpain,reduceinflammation,andspeedupthehealingprocess\.

##### 3B rewrite\.

\#FrequentlyAskedQuestions:PainManagementandBioelectricTherapy

\#\#Foundations

\*\*Whatisbioelectrictherapy?\*\*

Bioelectrictherapyisadrug\-freetreatmentoptionforpeopleinpain\.Ituseselectricalcurrentstointerruptpainmessagesbeforetheyreachthebrain,whichhelpsrelievepain\.

\*\*Whatconditionsaretreatedwithbioelectrictherapy?\*\*

Bioelectrictherapyisusedtotreatcertainchronicpainandacutepainconditions,including:

\-Complexregionalpainsyndrome

\-Backpain

\-Musclepain

\-Headachesandmigraines

\-Disordersofbloodflowintheupperandlowerlimbs

\-Temporomandibularjoint\(TMJ\)syndrome

\-Disordersofthenervoussystem,suchasdiabeticneuropathy

\*\*Howdoesbioelectrictherapywork?\*\*

Bioelectrictherapyrelievespainbyblockingpainmessagestothebrain\.Whenyouareinjured,painreceptorssendpainmessagestothecentralnervoussystem\(thebrainandspinalcord\)\.Themessageisregisteredaspainbycertaincellsinthebody\.Thecellsinterpretthispainasthreateningandtriggertheprotectiveresponseofpainreliefmechanisms\.

\#\#HowBioelectricTherapyWorks

\*\*Howdoesbioelectrictherapyrelievepain?\*\*

Bioelectrictherapypromptsthebodytoproduceendorphins\.Thesearecompoundsthathelprelievepain\.Additionally,thetherapypromptsthebraintoproducemoreofitsownendorphins,whichhelpstofurtherrelievethepain\.

\*\*Whathappensduringbioelectrictherapy?\*\*

Bioelectrictherapyisanon\-invasiveprocedurethatuseselectricalcurrentstoblockpainmessagesbeforetheyreachthebrain\.Therepeatedstimulationofspecificpain\-relievingnervescausesthebodyproduceitsownendorphins,whichreducepain\.

\*\*Howoftenshouldpeoplegetbioelectrictherapy?\*\*

Peopleshouldreceivebioelectrictherapyregularly\-eitherasanoutpatienttreatmentorincertaincases,aspartofarehabilitationprogram\.

\#\#PreparationandLimitations

\*\*Isbioelectrictherapyrightforeveryone?\*\*

No\.Bioelectrictherapyisnotrecommendedforpeoplewhohaveapacemaker,arepregnant,havethrombosis\(bloodclotsinthelegsorarms\),orhaveabacterialinfection\.

\*\*Whatconditionsdon’trespondwelltobioelectrictherapy?\*\*

Bioelectrictherapyisnotrecommendedforconditionslikechronicpain,backpain,musclepain,orheadachesandmigrainesthatdon’trespondtoothertreatments\.

\#\#AdditionalInformation

\*\*WherecanIfindmoreinformationaboutbioelectrictherapy?\*\*

Additionalinformationisavailablethroughthecompanythatsuppliesthetreatmentequipment,orthroughmedicalwebsitesthatcoverpainmanagementandbioelectrictherapy\.