@AhmedSQRD: Synthetic rewrites are now standard to improve data quality and when data is limited. But how much compute should you s…

Synthetic rewrites are now standard to improve data quality and when data is limited. But how much compute should you spend rewriting? And can a model trained on rewrites beat the rewriting model?

We fit scaling laws to answer both:

1/🧵

We fit a rewriting scaling law over parameters N, tokens D, rewrite fraction R, which is the fraction of the training mixture composed of synthetic rewrites, accounting for the compute spent on rewriting. We answer two questions:

1. What is the optimal rewrite fraction, and is it predictable at scale?

2. Can a model trained on rewrites outperform the rewriter?

2/🧵

In the compute-limited setting we assume plentiful natural data, but limited compute. We’re given a natural dataset and a paired dataset of synthetic rewrites from an off-the-shelf rewriter, and we then train IsoFLOP ladders with varied training mixtures from fully synthetic to fully natural text.

3/🧵

In the data-limited setting we instead assume a limited seed set of data, but relatively plentiful compute. To generate synthetic data we train a bootstrapped rewriter on the seed set for one epoch and instruction tune it with a small set of documents to follow the same rewrite styles as in the compute efficient setting. We then train IsoFLOP ladders that trade off spending compute either on epoching the seed set or generating and training on synthetic data for a target model.

Notably, in the data efficient setting synthetic data is unique but natural data is repeated! We thus measure our experiments in terms of an epoch budget W, which allows a FLOP budget that can epoch over the seed set W times at R=0, or gradually spend more compute to generate and train on synthetic data as we approach R=1.

4/🧵

We use Fineweb-Edu as our natural dataset for both settings, and measure heldout loss on Paloma and the Math, Code and QA benchmarks from Olmo-Eval. In the compute-limited setting we use SmolLM2 as our external rewriter (trained on 1e23 FLOPs) through the Finephrase dataset. We train IsoFLOP sweeps at 11 rewrite fractions, R from 0 to 1 in steps of 0.1, over five training budgets from 3e18 to 9e19 FLOPs, and use them to predict heldout R* at 1e21 and 1e22 FLOPs

5/🧵

For the data efficient setting we use a 5B seedset from Fineweb Edu. We then fix a model size and train on one epoch of this seed set before SFT on 1000 documents of rewrites to mimic the style of finephrase generated from GLM 5.2 (about 7M tokens)

We then sweep five intervals from R=0 to 1 that evenly charge the cost of compute and evaluate how loss shifts as compute increases. We run experiments with model sizes at 550M, 837M, 1B and 3B, fixing both the capacity of the rewriter and the target model

6/🧵

In the compute-limited setting, the optimal rewrite fraction increases with compute: from under 10% at 3×10¹⁸ FLOPs to 32% at 10²² FLOPs. Rewriting with a fixed external model becomes relatively cheaper as the target grows.

We extrapolate 33× in FLOPs to a 22B model at the predicted 32% mix. It has the lowest Paloma loss of the four mixtures we test at 10²² FLOPs, within 1.1% of the forecast.

7/🧵

In the data-limited setting, we find that the optimal value of R increases with compute. For the 3B target, the best observed R rises from 0.13 at epoch budget W=3 to 0.71 at W=24, where again W measures compute in units of one pass over the seed set.

Fitting through W=18, the law selects mixtures at held-out W=21 and 24 whose measured Paloma loss is within 1.14% of the best tested mixture across all four sizes.

8/🧵

Can the target beat the rewriter? In the compute-limited setting, our 32% mixture beats SmolLM2 on 10/16 Paloma sources using roughly one-tenth of its pretraining compute. Its overall Paloma loss is still slightly higher: 0.983 vs 0.972 BPB

In the data-limited setting, mixtures beat the best natural-only run at every size up to 3B, even with a size-matched, less capable rewriter. I found this especially surprising since it suggests the possibility of ‘self improvement’ in pretraining for models much smaller than frontier scale!

9/🧵

We also find the best mixture also depends on the domain. In the compute-limited setting, Math and Code favor 100% rewrites within our scaling ladder, while the fitted optima approach 77% for QA and 32% for Paloma.

At 10²² FLOPs, all-rewrite training instead of natural-only lowers Math BPB from 1.33 to 0.80 and Code BPB from 1.04 to 0.53.

10/🧵

Domain differences persist in the data-limited setting. For the 3B target, mixtures beat the best natural-only run on Math and Code at every budget, by 0.34 and 0.31 BPB at W=24. For QA however, natural only wins after W=6.

11/🧵

Our work of course has a few limitations: we ran our main experiments on FineWeb-Edu. When compared to a ‘higher quality’ dataset such as DCLM (as measured by compute efficiency on right plot), you might expect that synthetic re-writing is less helpful and we find this to be the case with a lower optimal value of R (left plot).

We also limit our re-writing styles to the four listed in FinePhrase as compute optimal (rewrite in the style of MATH, FAQ, Table, Tutorial styles), and only evaluate one external rewriter in the compute-limited setting.

12/🧵

That’s all! Grateful to co-led this with @nick11roberts

Nick and I are grateful to have done this work with Jane Street and our mentors Yu Wang, @alex_renda_ and Nitya Mani! Also in collaboration with @kothasuhas and @percyliang

Please read the paper for more details :) Paper: https://arxiv.org/abs/2610.31867

Model checkpoints: https://huggingface.co/jane-street-internship-org/rewriting-scaling-laws/tree/main… 13/🧵

( P.S. I’m also on the industry job market, so please reach out if you find this work relevant :) )