Scaling Laws for Mixture Pretraining Under Data Constraints
Summary
This paper studies the trade-off between scarce target data and abundant generic data in mixture pretraining, finding that repetition is a key driver of performance and that mixture training tolerates 15-20 repetitions of target data. It introduces a repetition-aware scaling law to optimize mixture configurations under data constraints.
arXiv:2605.12715v1 Announce Type: new
Abstract: As language models scale, the amount of data they require grows -- yet many target data sources, such as low-resource languages or specialized domains, are inherently limited in size. A common strategy is to mix this scarce but valuable target data with abundant generic data, which presents a fundamental trade-off: too little target data in the mixture underexposes the model to the target domain, while too much target data repeats the same examples excessively, yielding diminishing returns and eventual overfitting. We study this trade-off across more than 2,000 language-model training runs spanning multiple model and target dataset sizes, as well as several data types, including multilingual, domain-specific, and quality-filtered mixtures. Across all settings, we find that repetition is a central driver of target-domain performance, and that mixture training tolerates much higher repetition than single-source training: scarce target corpora can be reused 15-20 times, with the optimal number of repetitions depending on the target data size, compute budget, and model scale. Next, we introduce a repetition-aware mixture scaling law that accounts for the decreasing value of repeated target tokens and the regularizing role of generic data. Optimizing the scaling law provides a principled way to compute effective mixture configurations, yielding practical mixture recommendations for pretraining under data constraints.
View Cached Full Text
Cached at: 05/14/26, 06:17 AM
# Scaling Laws for Mixture Pretraining Under Data Constraints
Source: [https://arxiv.org/html/2605.12715](https://arxiv.org/html/2605.12715)
\(May 12, 2026\)
###### Abstract
As language models scale, the amount of data they require grows – yet many target data sources, such as low\-resource languages or specialized domains, are inherently limited in size\. A common strategy is to mix this scarce but valuable target data with abundant generic data, which presents a fundamental trade\-off: too little target data in the mixture underexposes the model to the target domain, while too much target data repeats the same examples excessively, yielding diminishing returns and eventual overfitting\. We study this trade\-off across more than 2,000 language\-model training runs spanning multiple model and target dataset sizes, as well as several data types, including multilingual, domain\-specific, and quality\-filtered mixtures\. Across all settings, we find that repetition is a central driver of target\-domain performance, and that mixture training tolerates much higher repetition than single\-source training: scarce target corpora can be reused 15–20 times, with the optimal number of repetitions depending on the target data size, compute budget, and model scale\. Next, we introduce a repetition\-aware mixture scaling law that accounts for the decreasing value of repeated target tokens and the regularizing role of generic data\. Optimizing the scaling law provides a principled way to compute effective mixture configurations, yielding practical mixture recommendations for pretraining under data constraints\.
Figure 1:Repetition dynamics for 143M model, 50M German tokens mixed with English,hhdefining the weight of German data in the training mix\.\(a\)Repetition factorrrgrows with training tokens; beyond the repetition frontier, loss begins to increase\.\(b\)German validation loss vs\. training tokens\. Stars indicate overfitting onset\.## 1Introduction
Large language models \(LLMs\) have demonstrated remarkable performance across a wide range of language understanding, mathematics, science, and knowledge\-intensive tasks\(luong2025advanced;woodruff2026accelerating;singh2025openai;antropic2026\)\. While much of this success can be attributed to large scale pretraining corpora exceeding trillions of tokens\(hojel2025essential;li2024datacomp\), many language model pretraining scenarios involve data that cannot be freely scaled including low\-resource languages, specialized domains, and curated datasets, which offer far less unique data\. During pretraining, this data is mixed with abundant generic data, such as pairing low\-resource language text with English text, or a domain\-specific math corpus with generic web text\.
Figure 2:Best achievable German test loss by target data size, shown for the 539M model\.However, mixing introduces a new trade\-off\. The limited data must be repeated for enough of the total training to provide sufficient target\-domain signal, but high repetition leads to memorization and eventual overfitting\. Figure[1](https://arxiv.org/html/2605.12715#S0.F1)illustrates this challenge for a 143M model with 50M German tokens mixed with English\. Higher fractions of German data start overfitting \(Figure[1](https://arxiv.org/html/2605.12715#S0.F1)b\) as the number of repetitions crosses a frontier \(Figure[1](https://arxiv.org/html/2605.12715#S0.F1)a\)\. This raises the question:how do model scale, data size, and repetition jointly shape the outcome of mixture pretraining when target data in the mixture is constrained?
Prior work has studied these two dimensions separately\.muennighoff2023scalingderive scaling laws for data\-constrained training, showing that up to 4 repetitions over a fixed monolithic dataset are nearly as valuable as unique data\. However, this result, widely adopted as a practical ceiling for data repetition, applies to single\-source training where*all*tokens are repeated\. In mixture pretraining, only the constrained domain is repeated while the generic domain continues to supply fresh tokens, which is a structurally different regime\. Conversely, data mixing laws\(ye2024data;xie2024doremi;shukor2025scalinglawsoptimaldata\)optimize the composition of training mixtures to maximize downstream performance, but assume each data source has unlimited data\. Our work sits at the intersection: we study how to optimally mix data\-constrained sources with abundant ones, jointly optimizing mixture weights and repetition, a setting that arises in many real\-world pretraining scenarios\. Our contributions are:
- •A systematic empirical study of over 2000 training runs spanning model sizes from 101M to 805M parameters, different target data sizes, and diverse data types: multilingual \(German, French, Swahili\), multi\-domain \(mathematics, scientific papers, Wikipedia\), and quality\-filtered subsets\.
- •Empirical findings on repetition in data mixtures: repeated tokens follow diminishing returns predictably across all data types; optimal repetition scales with target dataset size and compute budget; and larger models consistently extract more from limited data despite overfitting faster\. Crucially, abundant generic data sustains learning and unlocks far higher repetition than reported for single\-source training without performance degradation, with optimal for target task performance repetitions reaching 15–20 times \(Figure[2](https://arxiv.org/html/2605.12715#S1.F2)\)\.
- •A scaling law at the intersection of data\-constrained training and mixture optimization that predicts target\-domain loss as a function of target data size, mixture ratio, and model size\. We demonstrate that this scaling law can be fitted at small scales and then extrapolates at larger scales, and that it can be used to estimate accurately the optimal number of repetitions\.
We also extend our findings to the case of two scarce target domains\. Together, our findings provide both empirical understanding of how repetition behaves in data mixtures and a predictive scaling law for training effectively on limited target data\.
## 2Related Work
### Neural scaling laws
The study of scaling laws for language models was popularized bykaplan2020scaling, who showed that loss decreases as a power law in model size, dataset size, and compute\.hoffmann2022trainingrefined these findings with the Chinchilla scaling laws, establishing compute\-optimal training recipes that balance model size and data\. Relating to our work,muennighoff2023scalinggeneralize Chinchilla to a single data\-constrained domain, finding that repeating data up to approximately four epochs causes negligible degradation, with meaningful gains extending to roughly 16 epochs and ending at 40\. They propose scaling laws that model the diminishing value of repeated tokens\. Subsequent work has extended scaling laws to downstream task performance\(wei2022emergent\), mixture\-of\-experts architectures\(abnar2025parameters;krajewski2024scaling;wang2024scaling\), continued pretraining\(que2024d;liew2025acceleration;seto2026optimal\), finetuning\(bethune2025scaling;zhang2024scaling\), and data quality\(chang2024scaling\)\. Our work extends this line of research providing a predictive scaling law for data constrained mixtures, an under\-explored regime where only one component of a data mixture is unlimited while others are severely constrained\.
### Data mixing for language models
Training on a mixture of data domains has become standard, with corpora aggregated from multiple web sources\(gao2020pile;cerebras2023slimpajama;soldaini2024dolma\)with varying data size per domain, and recent models are trained on carefully tuned domain mixtures\(bakouch2025smollm3;olmo2025olmo\)\. Prior work also focus on optimizing domain weights using distributionally robust optimization\(xie2024doremi;fan2024doge\), or by discovering domains through clustering the pretraining corpus\(diaonemotron;grangier2025task\)\. The domain mixture scaling law \(DMSL\) framework\(ye2024data;shukor2025scalinglawsoptimaldata\)models how per\-domain loss depends on mixture weights and total compute\. Our repetition\-aware scaling law builds on DMSL by incorporating the effect of data repetition, which becomes critical when any mixture component is data\-limited\. Finally,anonymous2026mixdonttunecompare data mixing against hyperparameter tuning \(weight decay, learning rate\) as competing strategies for data\-constrained pretraining\. The focus of our work is on how to set the mixture optimally once mixing is chosen by characterizing how the mixing ratio, repetition, and model size jointly determine target\-domain loss, and that eliminates the need for per\-configuration sweeps\.
### Data\-constrained pretraining
Several works study data\-constrained pretraining\.hernandez2022scalingpropose parametric models for the value of repeated data in multi\-epoch training, andgoyal2024scalingcompare repeating high\-quality data against training on fresh lower\-quality data, concluding that the benefit of filtering depends on the total compute budget\. When high\-quality data for the target domain is constrained, synthetic data has been used as an alternative to repetition\.gunasekar2023textbookspretrained on synthetic “textbook quality” data, andmaini2024rephrasingdemonstrate that rephrasing web data offer alternatives to repeating data\. For multi\-domain settings,seto2025trainingstudy pretraining bilingual models when target\-language data is constrained, showing that high\-quality auxiliary\-language data can partially substitute for target\-language data in typologically close languages, but that model scaling has diminishing returns\. Low\-resource language modelingfaces similar constraints, where mixing with high\-resource languages is a common strategy\(joshi2020state\)\. This work studies different scenarios where data is constrained in at least one target domain, and unconstrained in other generic data, and provides a recipe for determining the target domain loss\. This is tangential to other approaches which aim to improve the diversity of data by building more target data, or incorporating more generic data\.
## 3Methodology: Mixture Training and Datasets
We consider a mixture pretraining setup with two data sources, where onetargetsource has limited size and agenericsource provides effectively unlimited data\. LetDtargetD\_\{\\text\{target\}\}the target data size, i\.e\., the number of unique target\-domain tokens,DtotalD\_\{\\text\{total\}\}the total number of training tokens \(i\.e\., the total compute budget\),h∈\[0,1\]h\\in\[0,1\]the fraction ofDtotalD\_\{\\text\{total\}\}devoted to the target domain\. The number of repetitionsrris the number of time the unique target tokens are repeated throughout training:
r=h⋅Dtotal/Dtarget\.r=h\\cdot D\_\{\\text\{total\}\}/D\_\{\\text\{target\}\}\\kern 5\.0pt\.\(3\.1\)For a fixed compute budgetDtotalD\_\{\\text\{total\}\}and data poolDtargetD\_\{\\text\{target\}\}, increasing the target weighthhincreases the repetition factorrr, and vice versa\. The generic domain is assumed abundant enough to never repeat\. We extend the framework to multiple constrained target domains in Section[7](https://arxiv.org/html/2605.12715#S7)\.
To test whether the repetition–diversity trade\-off generalizes, we study three types of data\-constrained scenarios commonly encountered in practice: limited target\-language data \(multilingual\), limited specialised\-domain data \(multi\-domain\), and limited high\-quality data \(quality filtering\)\.
### Multilingual
The target domain is German text from FineWeb2\(penedo2025fineweb2\), artificially constrained to 50M, 100M, 500M, or 1B tokens111Across all our experimental setups, each smaller set is a subset of a larger one\., and an unlimited \(no constraint\) setting\. The generic domain is English web text from FineWeb\(penedo2024fineweb\), treated as effectively unlimited\. To verify that findings are not language\-specific, we run 50M, 100M, and 500M data size experiments with French and Swahili as target languages, also derived from FineWeb2\. Evaluation is done on a held\-out set of FineWeb2 for the corresponding language\.
Figure 3:German validation loss vs\. total training tokens for the 143M model across three target data budgets\. Each curve corresponds to a different target weighthh\.Figure 4:Empirical optimalrras a function of training tokens for three data constraint sizes across model sizes\. Optimal repetition grows steadily with the training budget\.
### Multi\-domain
The target domain is OpenWebMath\(paster2023openwebmath\), a collection of mathematical web pages, constrained to 10M, 100M, or 1B tokens\. The generic domain is a subset of DCLM\(gadre2024language\), a large\-scale curated English web corpus\. We additionally run three\-domain experiments using Wikipedia and peS2o\(soldaini2023pes2o\), a corpus of scientific papers derived from Semantic Scholar, both sourced from the OLMo data mix\(walsh2024olmo2\), with dataset sizes ranging from 10M–500M and 50M–2\.5B, respectively\. As the generic domain, we use Fineweb and Nemotron\-CC\(dash2024nemotroncc\)\. Evaluation is done on held\-out sets from the corresponding datasets\.
### Quality
We score DCLM\(li2024datacomp\)documents using the original fasttext quality classifier222[https://huggingface\.co/mlfoundations/fasttext\-oh\-eli5](https://huggingface.co/mlfoundations/fasttext-oh-eli5)and select the top \{1, 5, 10, 20\}% of the DCLM base corpus as the*high\-quality*\(HQ\) target set, with the full DCLM base data as thelow\-qualitygeneric set\. Unlike the different domain and language experiments, exploring data quality allows us to explore the trade\-off of amount of data from the data constrained domain, and closeness to the target domain\. Stricter filtering yields a lower set of HQ data and thus more repetition at a givenhh, but the data is expected to be closer on average to the target set\. In contrast, a lower filter percent will yield more data, but will be further from the target set\. Evaluation is performed on the data used to train the original classifier\.
Experiments use GPT\-2\-style autoregressive decoder\-only Transformer language models spanning sizes from 101M to 805M non\-embedding parameters \(see Appendix[10](https://arxiv.org/html/2605.12715#S10)for full architecture specifications\)\. Models are trained for approximately100×N100\\times Ntokens, whereNNis the non\-embedding parameter count, a sufficient horizon to observe both the benefits and eventual degradation from data repetition\. Data sources are mixed at the sample level: each mini\-batch contains examples sampled independently from the target set with probabilityhhand from the generic set with probability1−h1\-h\.
## 4Empirical Findings
We present results primarily through the German–English bilingual setup due to space constraints, simulating the low\-resource scenario with German\. Results for other languages and domains are consistent and discussed in Appendix[11](https://arxiv.org/html/2605.12715#S11)\.
### Data Repetition Leads to Predictable Overfitting
Figure[3](https://arxiv.org/html/2605.12715#S3.F3)shows target\-domain loss for the 143M model across three target dataset sizes, with each curve corresponding to a different target weighthh\. Whenhhforces high repetition, loss first improves then rises sharply, reflecting overfitting\. Target datasets of larger size delay this degradation because the samehhproduces less repetition \(Eq\.[3\.1](https://arxiv.org/html/2605.12715#S3.E1)\)\. Crucially, across all settings, overfitting onset is governed by the repetition factorrralone, i\.e\., the same target weight can be safe or harmful depending on the available dataset, but the samerrconsistently triggers degradation regardless of how it is reached\. This predictability of the same threshold holding across data sizes, model scales, and domains motivates the scaling law in Section[5](https://arxiv.org/html/2605.12715#S5)\.
### Mixture Training Unlocks High Repetition Tolerance
Figure[4](https://arxiv.org/html/2605.12715#S3.F4)shows the optimal repetition factor \(i\.e\., the value ofrrthat minimizes target loss\) as a function of training tokens\. Across all settings, optimalrrincreases steadily with the training budget, reaching up to 15–20 depending on model size and dataset size, substantially exceeding the widely used rule\-of\-thumb of<4<4epochs for single\-source data\-constrained training\(muennighoff2023scaling\)\. The difference stems from the generic domain, which provides a continuous supply of fresh tokens that effectively regularizes training and allows the model to absorb far more target\-domain repetition before degradation sets in\. Sincerrresults from the interplay ofhh, dataset size, and training budget \(Eq\.[3\.1](https://arxiv.org/html/2605.12715#S3.E1)\), the target weight needed to reach optimalrrvaries widely, e\.g\., from 9\.5% \(101M\) to 1\.9% \(539M\) for the 50M dataset\.
Figure 5:Target\-domain loss across model sizes at 9% and 20% target weights for fixed 100M German budget\. Dashed lines show the best\-loss envelope across allhhvalues\.
### Larger Models Overfit Faster, Yet Still Win
Figure[5](https://arxiv.org/html/2605.12715#S4.F5)shows target\-domain loss across five model sizes at two fixed target weights \(h=9%h=9\\%and20%20\\%\) with 100M German tokens; dashed lines show the best\-loss envelope, the minimum achievable loss at each training step across allhhvalues\. Ath=9%h=9\\%, larger models plateau earlier while smaller models are still improving\. Ath=20%h=20\\%, the larger models eventually overfit \(increasing target loss\), while the smallest models still improve\. This happens because larger models memorize the data more easily\. Yet in every case, larger models reach lower minima before degradation sets in, and the*best\-loss*envelope consistently favors larger models at every training budget\.*Scaling up remains beneficial*even under tight data constraints, but the optimal operating window narrows with model size\. This is consistent with Figure[4](https://arxiv.org/html/2605.12715#S3.F4), where the optimal repetition decreases with model size\.
Figure 6:Best achievable target loss by dataset size across different model sizes with optimal repetition\.
### Predictable Scaling
Figure[6](https://arxiv.org/html/2605.12715#S4.F6)shows*best*achievable loss as a function of unique data available among all weightshhand total training tokens, for different model sizes\. We see that these curves are remarkably regular: scaling up model size, or the number of available target data \(e\.g\., by collecting fresh target data\) both have a very predictable effect on the loss\.
### Broad Target Data Filters Can Beat More Repetitions\.
The preceding experiments treat the domain dataset as fixed and study how repetition degrades loss\. In some settings, however, the dataset size is a design choice as one can choose how to classify documents into the domain of interest\. For example, using a quality filter at varying quality levels determines how much data remains, and a practitioner can trade quality for quantity by adjusting the threshold\. We use the Quality Data Mixture to determine whether accepting lower quality data \(that is still data constrained and closer to the target domain\) to reduce repetition yields better outcomes than repeating the highest quality data333For many data constrained domains, selecting the amount to filter out is tunable\. We study data quality as existing work gives us an already established way of easily varying the amount of data and relevance to the target domainli2024datacomp\.
To ground the experiment, we first confirm in Figure[22](https://arxiv.org/html/2605.12715#S15.F22)with a high quality target dataset of the 99th percentile and show loss curves at three dataset sizes \(26M, 132M, 264M tokens\)\. At the smallest scale, onlyh≤0\.1h\\leq 0\.1avoids overfitting, and larger sets progressively unlock higher sustainable target weights\. This establishes the baseline that for a fixed quality threshold, the repetition penalty imposes the same constraint on the target loss\.
Figure 7:Best loss by quality band\. Dashed line marks the crossover where Q90–99 overtakes Q95–99\.We vary the quality filter threshold while holding the data source fixed\. We fix the data constrained set such that the top 1% \(Q99–100\) is 26M tokens, and add increasing amounts of data to the data constrained dataset at thresholds of 5%, 10%, and 20% up to 982M tokens\. Results are in Figure[8](https://arxiv.org/html/2605.12715#S4.F8)\. Relaxing the quality threshold and accepting slightly lower\-quality data in exchange for a larger set consistently outperforms repeating a narrow high\-quality slice\. At Q99–100, onlyh=0\.1h=0\.1avoids overfitting at up to 10B tokens of training, while Q90–99 supports weights up toh=0\.4h=0\.4and Q80–99 shows nearly all weights monotonically decreasing\.
Figure 8:Loss curves for four quality bands ranging from 25M tokens at 99th percentile to 982M at the 80th, and ordered by increasing dataset size\. Broader filters enable higher target weights without overfitting\.The best quality set also depends on the training budget as we further observe that the best quality filter threshold changes at around 5B tokens as seen in Figure[7](https://arxiv.org/html/2605.12715#S4.F7)where the top 5% high quality documents \(Q95–99\) performs best due to higher per\-token quality, but the top 10% set \(Q90–99\) overtakes it as training progresses and the smaller dataset saturates\. By 30B tokens, Q90–99 achieves loss 2\.33 versus 2\.80 for Q99–100 due to the repetition penalty\. This demonstrates that while repetition is an important factor, when high\-quality data is scarce, broadening the filter is preferable to over repeating data\.
## 5Repetition\-Aware Mixture Scaling Law
The goal of this section is to quantify the empirical findings of Section[4](https://arxiv.org/html/2605.12715#S4)\. We use scaling laws\(kaplan2020scaling\), which, in their original forms, are simple power laws that allow to predict the train loss of a model from its number of parametersNNand total number of tokens it has been trained onDtotalD\_\{\\text\{total\}\}\. In our setup, both the target data sizeDtargetD\_\{\\text\{target\}\}and the target weighthhinfluence the loss on the target domain\. Our objective is to obtain a simple and predictive formula forLtarget\(N,Dtotal,Dtarget,h\)L\_\{\\text\{target\}\}\(N,D\_\{\\text\{total\}\},D\_\{\\text\{target\}\},h\)\. We focus on the practically relevant regime where the target data is fully consumed at least once, i\.e\.r=h⋅Dtotal/Dtarget≥1r=h\\cdot D\_\{\\text\{total\}\}/D\_\{\\text\{target\}\}\\geq 1\. Whenr<1r<1, target tokens are never repeated and mixing reduces to standard mixture selection, as explored, e\.g\., in\(shukor2025scalinglawsoptimaldata\)\. In the following, we write scaling laws learned parameters inblue\.
### Effective data
The core building block of our law is the*effective data*DeffD\_\{\\mathrm\{eff\}\}, which accounts for the diminishing value of repeated tokens\. Followingmuennighoff2023scaling, we define the effective contribution of the*target*domain as
DT=Dtarget\(1\+ρ\(r\)\),withρ\(r\)=r1\(1−e−\(r−1\)/r1\),D\_\{T\}=D\_\{\\text\{target\}\}\\bigl\(1\+\\rho\(r\)\\bigr\)\\,,\\qquad\\text\{with\}\\quad\\rho\(r\)=\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}r\_\{1\}\}\\\!\\left\(1\-e^\{\-\(r\-1\)/\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}r\_\{1\}\}\}\\right\),\(5\.1\)wherer1\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}r\_\{1\}\}is a parameter that controls the effect repeated data\. For smallr≪r1\+1r\\ll\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}r\_\{1\}\}\+1, we haveρ\(r\)≈r−1\\rho\(r\)\\approx r\-1so thatDT≈r⋅DtargetD\_\{T\}\\approx r\\cdot D\_\{\\text\{target\}\}: each pass counts fully\. For largerr,DTD\_\{T\}saturates at\(1\+r1\)Dtarget\(1\+\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}r\_\{1\}\}\)\\,D\_\{\\text\{target\}\}, reflecting the diminishing value of further repetitions\. The total effective data is then
Deff=\(1−h\)Dtotal\+τDT,D\_\{\\mathrm\{eff\}\}=\(1\-h\)\\,D\_\{\\text\{total\}\}\+\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}\\tau\}\\,D\_\{T\}\\,,\(5\.2\)where the\(1−h\)Dtotal\(1\-h\)\\,D\_\{\\text\{total\}\}term corresponds to the number of tokens seen from the generic dataset andτ\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}\\tau\}controls the relative value of target\-domain tokens compared to generic\-domain tokens\. This formulation directly encodes two observations from Section[4](https://arxiv.org/html/2605.12715#S4)\. The saturation ofρ\(r\)\\rho\(r\)atr1\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}r\_\{1\}\}is the quantitative form of the diminishing value of repeated target tokens: forr≫r1r\\gg\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}r\_\{1\}\}, further passes add essentially nothing toDeffD\_\{\\mathrm\{eff\}\}\. Conversely, the unsaturated\(1−h\)Dtotal\(1\-h\)\\,D\_\{\\text\{total\}\}term formalises why mixture training tolerates far more repetition than single\-source training \(Figure[4](https://arxiv.org/html/2605.12715#S3.F4)\): generic tokens never saturate, so they keepDeffD\_\{\\mathrm\{eff\}\}growing even when the target contribution has plateaued\.
### Scaling law formulas
We propose two loss formulas, for fixed and variable model size:
Lfix=E\+ADeffα\+γh,Lsize=E\+CNβ\+BNδDeffα\+γh\.L\_\{\\mathrm\{fix\}\}=\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}E\}\+\\frac\{\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}A\}\}\{D\_\{\\mathrm\{eff\}\}^\{\\,\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}\\alpha\}\}\}\+\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}\\gamma\}\\,h\\,,\\qquad L\_\{\\mathrm\{size\}\}=\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}E\}\+\\frac\{\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}C\}\}\{N^\{\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}\\beta\}\}\}\+\\frac\{\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}B\}\\,N^\{\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}\\delta\}\}\}\{D\_\{\\mathrm\{eff\}\}^\{\\,\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}\\alpha\}\}\}\+\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}\\gamma\}\\,h\\,\.\(5\.3\)InLfixL\_\{\\mathrm\{fix\}\}, the irreducible lossE\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}E\}is a constant baseline; the data termA/Deffα\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}A\}/D\_\{\\mathrm\{eff\}\}^\{\\,\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}\\alpha\}\}is a Chinchilla\-style power\-law relating loss to effective data; and the weight penaltyγh\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}\\gamma\}\\,his a linear cost on the target weight\.LsizeL\_\{\\mathrm\{size\}\}additionally includes a Chinchilla\-style capacity termC/Nβ\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}C\}/N^\{\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}\\beta\}\}and a data\-size couplingNδN^\{\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}\\delta\}\}\(δ\>0\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}\\delta\}\>0\)\(hoffmann2022training\): the capacity term captures the intrinsic capability gain from scale, while the coupling means that for the sameDeffD\_\{\\mathrm\{eff\}\}, the data\-limited loss component is amplified at largerNN, encoding the fact that larger models yield better losses but overfit faster \(Figure[5](https://arxiv.org/html/2605.12715#S4.F5)\)\. At fixedNN,LsizeL\_\{\\mathrm\{size\}\}reduces exactly toLfixL\_\{\\mathrm\{fix\}\}withA=BNδ\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}A\}=\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}B\}\\,N^\{\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}\\delta\}\}andEfix=E\+C/NβE\_\{\\mathrm\{fix\}\}=\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}E\}\+\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}C\}/N^\{\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}\\beta\}\}, ensuring consistency\.LfixL\_\{\\mathrm\{fix\}\}has six fitted parameters \(E\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}E\},A\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}A\},α\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}\\alpha\},r1\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}r\_\{1\}\},τ\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}\\tau\},γ\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}\\gamma\}\);LsizeL\_\{\\mathrm\{size\}\}has nine \(E\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}E\},C\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}C\},β\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}\\beta\},B\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}B\},δ\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}\\delta\},α\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}\\alpha\},r1\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}r\_\{1\}\},τ\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}\\tau\},γ\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}\\gamma\}\)\.
### Fitting
Given a collection of training runs indexed byii, each characterised by a total token budgetDtotal,iD\_\{\\text\{total\},i\}, a target weighthih\_\{i\}, an available target datasetDtarget,iD\_\{\\text\{target\},i\}, and optionally a model sizeNiN\_\{i\}, we observe the resulting loss on the target domainℓi\\ell\_\{i\}\. The parametersθ\\thetaof the scaling law are estimated by minimising the reweighted Huber loss
θ^=argminθ∑iωi⋅ℋ\(ℓi−Lθ\(Dtotal,i,hi,Dtarget,i,Ni\)\),\\hat\{\\theta\}=\\arg\\min\_\{\\theta\}\\sum\_\{i\}\\omega\_\{i\}\\cdot\\mathcal\{H\}\\\!\\bigl\(\\ell\_\{i\}\-L\_\{\\theta\}\(D\_\{\\text\{total\},i\},\\,h\_\{i\},\\,D\_\{\\text\{target\},i\},\\,N\_\{i\}\)\\bigr\)\\,,\(5\.4\)whereℋ\\mathcal\{H\}is the Huber loss and the weightsωi=max\(ri⋅hi,ϵ\)\\omega\_\{i\}=\\max\(r\_\{i\}\\cdot h\_\{i\},\\,\\epsilon\)withϵ=0\.01\\epsilon=0\.01emphasise the high\-repetition, high\-fraction regime:hhalone would under\-weight high\-repetition runs at moderate fractions, whilerralone would under\-weight high\-fraction runs with large datasets \(whererris low\)\. The qualitative ranking of methods is unchanged under alternative weights; see Appendix[20](https://arxiv.org/html/2605.12715#S20)\. Followingshukor2025scalinglawsoptimaldata, we minimise[5\.4](https://arxiv.org/html/2605.12715#S5.E4)using basin\-hopping optimisation with 100 random restarts to avoid poor local minima in the non\-convex loss landscape\.
## 6Scaling Law Results
We evaluate the scaling law on experimental setups described in Section[3](https://arxiv.org/html/2605.12715#S3)\. For each setup,LfixL\_\{\\mathrm\{fix\}\}is fitted independently per model size, andLsizeL\_\{\\mathrm\{size\}\}is fitted jointly across all model sizes\. To test extrapolation forLfixL\_\{\\mathrm\{fix\}\}, we fit on the first 50% of training steps and test on second half\. ForLsizeL\_\{\\mathrm\{size\}\}, we fit on all but the largest model scale, and test on the held\-out largest model scale\.
Table 1:Test weightedR2R^\{2\}\.Left:fixed\-size formulas, fitted independently per model size\.Right:multi\-size formulas, fitted on smaller model sizes and evaluated on the held\-out largest model\.GermanMathsQualityWiki/peS2oGermanMathsLfixL\_\{\\mathrm\{fix\}\}0\.950\.880\.710\.80LsizeL\_\{\\mathrm\{size\}\}0\.650\.73Repetition\-agnostic0\.780\.780\.140\.72Repetition\-agnostic\+NN0\.590\.71Utility decay0\.720\.55−\-0\.640\.79Utility decay\+NN0\.560\.69Domain\-agnostic−\-40\.7−\-0\.49−\-2\.19−\-1\.16Domain\-agnostic\+NN−\-0\.23−\-0\.77### Baselines
We compare against three other scaling laws formulas inspired by the literature\.*Repetitions\-agnostic*\(shukor2025scalinglawsoptimaldata\)replacesDeffD\_\{\\mathrm\{eff\}\}with\(1−h\)Dtotal\+τhDtotal\(1\-h\)\\,D\_\{\\text\{total\}\}\+\{\\color\[rgb\]\{0\.1171875,0\.390625,0\.78515625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.1171875,0\.390625,0\.78515625\}\\tau\}\\,h\\,D\_\{\\text\{total\}\}, treating repeated tokens as unique\.*Domain\-agnostic*\(muennighoff2023scaling\)uses a single saturating function of total tokens without distinguishing domains\.*Utility decay*\(goyal2024scaling\)models repetition through a decaying exponent on the data term\. The formulas are detailed in Appendix[18](https://arxiv.org/html/2605.12715#S18)\.
### Optimal mixture prediction
Figure 9:Predicted vs\. empirical optimal repetitionr∗r^\{\*\}for the German target dataset of 1B tokens\. Solid: empirical optimum; dashed:LfixL\_\{\\mathrm\{fix\}\}prediction\.
### Loss prediction
Table[1](https://arxiv.org/html/2605.12715#S6.T1)reports the weighted coefficient of determination \(wR2wR^\{2\}\) on the held\-out test split \(trainwR2wR^\{2\}in Appendix[19](https://arxiv.org/html/2605.12715#S19)\), where each observation is weighted bymax\(r⋅h,ϵ\)\\max\(r\\cdot h,\\,\\epsilon\)to emphasise the high\-repetition regime\.LfixL\_\{\\mathrm\{fix\}\}consistently achieves the best testwR2wR^\{2\}, extrapolating beyond the fitting range\.*Repetitions\-agnostic*fits the training range reasonably but fails to extrapolate to high repetitions\.*Domain agnostic*fails completely: treating all tokens as interchangeable does not capture the loss dynamics\.*Utility decay*performs honorably on several datasets\.
MedianMeanp90LfixL\_\{\\mathrm\{fix\}\}263476Rep\-agn\.887399Util\. dec\.314195Dom\.\-agn\.475998Table 2:Fraction of training tokens wasted by following each formula’s predicted optimal mixture instead of the oracle\.
The primary use of the scaling law is to predict the optimal target weighth∗h^\{\*\}for a given total token budgetDtotalD\_\{\\text\{total\}\}and available target domain datasetDtargetD\_\{\\text\{target\}\}\. Once the scaling law parameters are estimated, we can simply solve
h∗=argminh∈\[0,1\]Lfix\(Dtotal,h,Dtarget\)h^\{\*\}=\\arg\\min\_\{h\\in\[0,1\]\}L\_\{\\mathrm\{fix\}\}\(D\_\{\\text\{total\}\},\\,h,\\,D\_\{\\text\{target\}\}\)\(6\.1\)by grid search overhh\. Figure[9](https://arxiv.org/html/2605.12715#S6.F9)shows an example of empirical vs\. estimated optimal repetition\.
To quantify the practical cost of suboptimal mixture predictions, we measure the fraction of training tokens*wasted*by following each formula’s recommendation instead of the oracle \(Table[2](https://arxiv.org/html/2605.12715#S6.T2)\)\. At a given budgetDtotalD\_\{\\text\{total\}\}, the formula recommends a fractionhpred∗h^\{\*\}\_\{\\text\{pred\}\}; we interpolate the empirical loss at that fraction and find the budgetD′D^\{\\prime\}at which the run using the empirically best fraction at each step first reached the same loss\. The wasted fraction\(Dtotal−D′\)/Dtotal\(D\_\{\\text\{total\}\}\-D^\{\\prime\}\)/D\_\{\\text\{total\}\}directly measures how much compute could have been saved with perfect knowledge of the optimal mixture\. Detailed results are in Appendix[19](https://arxiv.org/html/2605.12715#S19)\.
## 7Multiple Domain Experiments
In order to test whether our findings and scaling law formula extend to settings with more than one constrained domain, we set up experiments with a mixture of three domains: one unconstrained generic source and two constrained target domains\. We use two configurations: FineWeb as the generic source with peS2o and Wikipedia as targets, and Nemotron as the generic source with Wikipedia and peS2o as targets\.444Subplot \(a\) of Figure[10](https://arxiv.org/html/2605.12715#S7.F10)uses FineWeb \+ peS2o \+ Wikipedia; subplots \(b\) and \(c\) use Nemotron \+ Wikipedia \+ peS2o\.We vary the dataset sizes across three configurations: 5×\\timesless, base, and 5×\\timesmore data\. Our primary metric is the average loss across both target domains, analogous to evaluating on a test set that combines text from both domains\.
Figure 10:Multi\-domain experiments \(101M model, English \+ Wikipedia \+ peS2o\)\.\(a\)Best achievable average target loss under equal vs\. proportional weighting across three set\-size configurations\.\(b\)Optimal repetitions under proportional weighting with 10% compute confidence band \(shaded\)\.\(c\)Independently optimizingrrper domain using the bilingual scaling law outperforms the best grid\-searched proportional weighting\.### Weighting Strategies
First, we compared two weighting strategies:*equal weighting*\(hwiki=hpeS2oh\_\{\\text\{wiki\}\}=h\_\{\\text\{peS2o\}\}\) and*proportional weighting*, where fractions are set proportional to dataset size \(hwiki/hpeS2o=Dwiki/DpeS2oh\_\{\\text\{wiki\}\}/h\_\{\\text\{peS2o\}\}=D\_\{\\text\{wiki\}\}/D\_\{\\text\{peS2o\}\}\)\. Our experiments demonstrate \(Figure[10](https://arxiv.org/html/2605.12715#S7.F10)a\) that by average loss, proportional weighting outperforms equal weighting when data is scarce, while equal weighting is better in the data\-rich regime\. The two strategies cross near the base configuration, suggesting that proportional weighting becomes increasingly beneficial as repetition grows\. For the rest of this section, we focus on the proportional weighting, and provide the results for equal weighting in Appendix[17](https://arxiv.org/html/2605.12715#S17)\.
### Empirical Findings
The key findings from the bilingual experiments carry over to the multi\-domain setting\. Figure[10](https://arxiv.org/html/2605.12715#S7.F10)b shows that optimal repetition grows steadily with training tokens across all set\-size configurations, with smaller sets requiring substantially higher repetition—the 5×\\timesless variant reachesr≈30r\\approx 30while 5×\\timesmore stays belowr≈7r\\approx 7\. This mirrors the bilingual experiments: the generic domain continues to regularize training even with multiple constrained domains present, enabling high repetition tolerance\. Importantly, the choice ofrrneed not be exact\. We quantify this through a 10% compute confidence band \(shaded regions in Figure[10](https://arxiv.org/html/2605.12715#S7.F10)b\): at each checkpoint, we identify all values ofrrthat achieve loss no worse than the best loss achievable with 10% less compute\. The bands are wide, particularly for smaller sets, indicating that even rough estimates of optimalrryield near\-optimal target loss\.
### Scaling Law Generalization
The similarity of these findings to the bilingual setting raises the question of whether the scaling law transfers directly to multi\-domain mixtures\. We test this by setting each domain’s repetition factor independently using the optimalrrpredicted by the bilingual scaling law for the corresponding set size \(rwiki=35r\_\{\\text\{wiki\}\}=35,rpeS2o=17r\_\{\\text\{peS2o\}\}=17\)\. Figure[10](https://arxiv.org/html/2605.12715#S7.F10)c shows that this independently optimized configuration consistently outperforms the best proportional weighting and narrows the gap to the 2\-domain oracle \(the average loss achieved by training each domain separately with English\)\. While a gap to the oracle remains, this result suggests that costly multi\-domain sweeps over all possible combinations of domain weights can be avoided: our two\-domain scaling law can independently estimate the optimalrrfor each constrained domain, and these can be combined into a single mixture—replacing an entire experimental grid with a single, better\-performing run\.
## 8Conclusion
We studied mixture pretraining under data constraints across diverse data types – multilingual, multi\-domain, and quality\-filtered setups\. Our experiments reveal consistent patterns across all settings: repeated target\-domain tokens yield diminishing returns, optimal repetition scales predictably with data availability, and larger models extract more from limited data\. Crucially, mixture training tolerates substantially higher repetition than single\-source training, with generic data acting as an implicit regularizer\. We formalized these dynamics in a scaling law that accurately predicts the optimal mixture across experimental setups, outperforming baselines that ignore either repetition or domain structure\. In practice, this allows training on scarce target data without expensive sweeps: given only the target pool size and compute budget, the law prescribes the mixture ratio that maximizes target\-domain performance in a single run\.
## References
\\beginappendix
## 9Experimental Setup
### Multilingual setup
For each combination of model size and pool size, we sweephhover a fine grid of 19–27 values \(see Table[4](https://arxiv.org/html/2605.12715#S10.T4)for a summary\), with denser sampling at low repetition levels where the loss landscape changes most rapidly\. The grid spans number of repetitions from below 1 to above 30, though configurations where the resultinghhexceeds 1 are excluded\. For larger models, the feasible range ofhhis narrower because the longer training duration \(100×N100\\times Ntokens\) produces higher repetition at the samehh; for example, the 539M model on the 50M pool coversrrup to approximately 20\. Evaluation is performed on 10,000 samples from held\-out test sets of both FineWeb2 \(target language\) and FineWeb \(English\), allowing us to measure both target\-domain and generic\-domain loss at each checkpoint\.
### Multi\-domain setup
For the three\-domain experiments \(Section[7](https://arxiv.org/html/2605.12715#S7)\), we train 101M models with a total training budget of 30B tokens \(approximately300×N300\\times N\), increased relative to the bilingual setup to allow sufficient repetition of both target domains\. The three pool\-size configurations correspond to: 5×\\timesless \(Wiki 20M, peS2o 100M\), base \(Wiki 100M, peS2o 500M\), and 5×\\timesmore \(Wiki 500M, peS2o 2\.5B\)\. We sweep over total target fractionshtotal=hwiki\+hpeS2oh\_\{\\text\{total\}\}=h\_\{\\text\{wiki\}\}\+h\_\{\\text\{peS2o\}\}ranging from 0\.02 to 0\.60 with step 0\.02, under both equal weighting \(hwiki=hpeS2oh\_\{\\text\{wiki\}\}=h\_\{\\text\{peS2o\}\}\) and proportional weighting \(hwiki/hpeS2o=Dwiki/DpeS2oh\_\{\\text\{wiki\}\}/h\_\{\\text\{peS2o\}\}=D\_\{\\text\{wiki\}\}/D\_\{\\text\{peS2o\}\}\)\.
### Math setup
For the OpenWebMath experiments, we sweephhover a linear grid whose density depends on pool size: 26 values for the 10M and 100M pools and 8 values for the 1B pool \(60 total per model size\)\. The 805M model uses a denser grid—35, 35, and 20 values for the 10M, 100M, and 1B pools respectively \(90 total\)—because its longer training horizon \(1M vs\. 200K steps\) produces higher repetition counts that require finer resolution\. Thehhrange is adapted to each pool size, spanning up toh=0\.078h=0\.078for the 10M pool,h=0\.78h=0\.78for the 100M pool, andh=1\.0h=1\.0for the 1B pool\. The maximum number of repetitions reached is approximately 41 for the 130–630M models and 92 for 936M\. The 101M–539M models are trained for 200K steps \(13\.1B tokens\) and the 805M model for 1M steps \(65\.5B tokens\), with a reduced batch size of 32 sequences\. The resulting tokens\-to\-parameters ratios range from approximately130×130\\times\(101M\) down to24×24\\times\(539M\), with81×81\\timesfor 805M\. These experiments use a SentencePiece tokenizer\(kudo2018sentencepiece\)trained on C4\(raffel2020exploring\)with a vocabulary of 32,000 tokens, rather than the GPT\-2 BPE tokenizer used in other setups\. Evaluation is performed on 10,240 sequences \(10\.5M tokens\) held out from OpenWebMath\.
### Quality setup
For the quality filter experiments, we train 100M parameter models ranging from 1–30B tokens, sweeping overh∈\{0\.1,0\.2,…,1\.0\}h\\in\\\{0\.1,0\.2,\\dots,1\.0\\\}\. We train models with varying quality pools primarily either 1%, 5%, and 10% with non\-overlapping bins between the 1% and 5% and 10% runs to explicitly measure the impact of having lower quality data, but with lower repetition\.
## 10Training Details
### Model architecture
All models are GPT\-2\-style autoregressive decoder\-only Transformers with learned positional embeddings\. Models are scaled along a depth\-based rule where the hidden dimensiondmodel=128×Ld\_\{\\text\{model\}\}=128\\times LforLLlayers, with an FFN intermediate size of4×dmodel4\\times d\_\{\\text\{model\}\}, attention head dimension of 64, anddmodel/64d\_\{\\text\{model\}\}/64attention heads\. All models use a context length of 1,024 tokens\.
### Training
Multilingual models are trained for approximately100×N100\\times Ntokens, whereNNis the non\-embedding parameter count\. The three\-domain experiments are trained for300×N300\\times N, the quality experiments are trained for 30K steps; the details about the OpenWebMath experiments are provided above\. All models are optimized with Adam,\(kingma2014adam\)a constant learning rate following a linear warmup, weight decay of10−210^\{\-2\}, gradient clipping at 0\.1, and no dropout\. The multilingual and Wiki/peS2o experiments use a learning rate of10−310^\{\-3\}\(selected based on initial experiments for 101M, 143M, and 192M models on English\-German data\), a warmup of 1% of training steps, a batch size of 256 sequences, and a GPT\-2 BPE tokenizer\(radford2019language\)with a vocabulary of 50,257 tokens\. The quality experiments use a learning rate of3×10−43\\times 10^\{\-4\}, a warmup of 1% of training steps, a batch size of 1,024 sequences, and the same GPT\-2 BPE tokenizer\. The OpenWebMath experiments use a learning rate of10−410^\{\-4\}withμ\\muP,a 2,000\-step warmup, a batch size of 32 sequences, and a SentencePiece tokenizer\(kudo2018sentencepiece\)trained on C4\(raffel2020exploring\)with a vocabulary of 32,000 tokens\.
### Learning Rate
We use a constant learning rate \(after warmup\) rather than a cosine or linear decay schedule\. Constant or near\-constant schedules are common practice in pretraining work\(hagele2024scaling;porian2025resolving\)\. Our experimental design also requires it\. The central object of analysis is a loss\-vs\-repetition curve read along a single training run \(e\.g\., Figures[3](https://arxiv.org/html/2605.12715#S3.F3),[5](https://arxiv.org/html/2605.12715#S4.F5)\)\. A decay schedule couples the learning rate to the training horizon, so checkpoints at different repetition counts would not be directly comparable within a run\. Replicating our sweep under a decay schedule would require a separate run for each evaluation point, multiplying compute by the number of checkpoints\.
### Compute resources
All experiments were run on a mix of A100 and H100 GPUs\. A single training run requires approximately 40 GPU\-hours for the 101M model, 64 for 143M, 100 for 192M, 264 for 340M, 600 for 539M, and 1200 GPU\-hours for 805M\.
Table 3:Model architectures\. Parameters refer to non\-embedding parameters\. All models use a sequence length of 1,024, a batch size of 256 sequences, and are trained onD≈100×ND\\approx 100\\times Ntotal tokens\.ParamsLayersHiddenFFNHeads101M81024409616143M91152460818192M101280512020340M121536614424539M141792716828805M162048819232Table 4:Experimental setup summary\.DtargetD\_\{\\text\{target\}\}denotes the range of target data pool sizes; \#hhis the total number of target fraction configurations per model size\.NNDtargetD\_\{\\text\{target\}\}\#hhMultilingualGerman101M50M–1B72143M50M–1B75192M50M–1B75340M50M–1B77539M50M–1B82French101M50M–500M63143M50M–500M64192M50M–500M63340M50M–500M55539M50M–500M57Swahili101M50M–500M63143M50M–500M64192M50M–500M63340M50M–500M55539M50M–500M57
NNDtargetD\_\{\\text\{target\}\}\#hhDomainMath \(OpenWebMath\)101M10M–1B60197M10M–1B60340M10M–1B60539M10M–1B60805M10M–1B90Wiki \+ peS2o \(3\-domain\)101M10M–500M153Qual\.Quality filtering101M25M–264M10
## 11Consistency of the Findings Across Languages and Domains
The core findings from the German–English bilingual experiments generalize across all settings tested\. We replicate the experimental grid for French and Swahili – languages that differ substantially from German in morphology, script frequency, and available web data – and observe qualitatively identical behavior in all cases\. The same U\-shaped overfitting emerges at highhhwith small data pools, the same dependence of optimalrron pool size holds, and larger models consistently achieve lower best\-case loss despite earlier overfitting onset\. We further verify these patterns on non\-language domain splits \(Wikipedia, peS2o, OpenWebMath mixed with generic English data\), confirming that the repetition dynamics are a general property of mixture training under data constraints rather than an artifact of any particular language pair\.
Below we provide supporting evidence organized as follows:
- •Optimal repetition\(Section[12](https://arxiv.org/html/2605.12715#S12)\): French, Swahili, and OpenWebMath all exhibit the same steady growth of optimalrrwith training budget, governed by data pool size rather than domain identity\.
- •Loss curves\(Section[13](https://arxiv.org/html/2605.12715#S13)\): Full loss\-vs\-tokens curves for all model sizes and languages confirm the overfitting at highhhwith small pools and monotonic improvement with large pools\.
- •Cross\-model behavior\(Section[14](https://arxiv.org/html/2605.12715#S14)\): Larger models overfit faster yet achieve lower minima across all languages – the same pattern as German\.
- •Quality experiments\(Section[15](https://arxiv.org/html/2605.12715#S15)\): The repetition–diversity trade\-off extends to quality\-filtered data at multiple data scales\.
- •Comparison to unlimited data\(Section[16](https://arxiv.org/html/2605.12715#S16)\): When target data is unconstrained, mixture training provides no benefit over single\-domain training – confirming that our findings are specific to the data\-constrained regime\.
## 12Optimal Repetition Across Languages and Domains
Figures[11](https://arxiv.org/html/2605.12715#S12.F11)–[12](https://arxiv.org/html/2605.12715#S12.F12)show the optimal number of repetitions as a function of training tokens for French and Swahili, matching the German plot in the main text \(Figure[4](https://arxiv.org/html/2605.12715#S3.F4)\)\. The same steady growth of optimalrrwith training budget is observed across all languages\.
Figure 11:Optimal repetition as a function of training tokens for three data constraint sizes across different model sizes \(French\)\.Figure 12:Optimal repetition as a function of training tokens for three data constraint sizes across different model sizes \(Swahili\)\.Figure[13](https://arxiv.org/html/2605.12715#S12.F13)extends this analysis to a non\-language domain: OpenWebMath mixed with DCLM as the generic source\. Despite the very different nature of mathematical text, the same qualitative pattern holds—optimalrrincreases with training tokens, decreases with pool size, and smaller models tolerate higher repetition \(since they are trained on more tokens relative to their capacity\)\. This confirms that the repetition dynamics are a general property of mixture training under data constraints, independent of the specific domain or language\.
Figure 13:Optimal repetition as a function of training tokens for three data constraint sizes across different model sizes \(OpenWebMath\)\.
## 13Loss Curves Across Languages, Model Sizes, and Domains
Figures[14](https://arxiv.org/html/2605.12715#S13.F14)–[16](https://arxiv.org/html/2605.12715#S13.F16)show target\-domain validation loss as a function of training tokens for all model sizes \(101M–539M\) and all three target languages \(German, French, Swahili\)\. Each column corresponds to a different target data pool size; each row to a different model size; each curve within a panel corresponds to a different target weighthh\(color\)\. The same qualitative pattern—U\-shaped overfitting at highhhwith small pools, monotonic improvement with large pools—holds consistently across all configurations\.
Figure[17](https://arxiv.org/html/2605.12715#S13.F17)shows the same analysis for OpenWebMath, confirming that the loss curve dynamics extend beyond natural language to mathematical text\.
Figure 14:German target\-domain loss curves across all model sizes \(rows\) and data pool sizes \(columns\)\.Figure 15:French target\-domain loss curves across all model sizes \(rows\) and data pool sizes \(columns\)\.Figure 16:Swahili target\-domain loss curves across all model sizes \(rows\) and data pool sizes \(columns\)\.Figure 17:Math \(OpenWebMath\) target\-domain loss curves across all model sizes \(rows\) and data pool sizes \(columns\)\.
## 14Cross\-Model Loss Curves
Figure[5](https://arxiv.org/html/2605.12715#S4.F5)shows that larger models overfit faster yet achieve lower best\-case loss, demonstrated for German with 100M tokens ath=9%h=9\\%and20%20\\%\. Here we extend this analysis to five target fractions \(5–20%\) and verify the pattern across all languages and domains\. Figure[18](https://arxiv.org/html/2605.12715#S14.F18)shows the full German panel, Figures[19](https://arxiv.org/html/2605.12715#S14.F19)–[20](https://arxiv.org/html/2605.12715#S14.F20)show French and Swahili, and Figure[21](https://arxiv.org/html/2605.12715#S14.F21)shows the OpenWebMath domain\. In all cases, the same ordering persists: larger models reach lower loss minima despite earlier overfitting onset, and the best\-loss envelope \(dashed\) consistently favors larger models at every training budget\. The pattern is particularly pronounced for the mathematics domain \(Figure[21](https://arxiv.org/html/2605.12715#S14.F21)\), where the separation between model sizes is larger and the overfitting onset for larger models is sharper\.
Figure 18:German target\-domain loss across model sizes at five target fractions, for the 100M token pool\. Dashed lines show the best\-loss envelope\.Figure 19:French target\-domain loss across model sizes at five target fractions, for the 100M token pool\. Dashed lines show the best\-loss envelope\.Figure 20:Swahili target\-domain loss across model sizes at five target fractions, for the 100M token pool\. Dashed lines show the best\-loss envelope\.Figure 21:OpenWebMath target loss across model sizes at different mixture fractionshh, for the 100M token pool\. Dashed lines show the best\-loss envelope across allhhvalues\.
## 15Quality Experiments: Additional Target Data Scales and Overlapping Quality Pools
To ground the experiments, we first confirm that a data constrained fixed quality level follows the same pattern as multilingual data\. Figure[22](https://arxiv.org/html/2605.12715#S15.F22)shows Q99–100 loss curves at three pool sizes \(26M, 132M, 264M tokens\)\. At the smallest scale, onlyh≤0\.1h\\leq 0\.1avoids overfitting, and larger pools progressively unlock higher sustainable target weights\. This establishes the baseline that for a fixed quality threshold, the repetition penalty imposes the same constraint on the target loss\.
Figure 22:Loss vs\. training tokens for Q99–100 at three data pool sizes\. Each line corresponds to a target weighthh\. Smaller pools force lower weights to avoid overfitting from repetition\.[Section˜4](https://arxiv.org/html/2605.12715#S4)further establishes that when training with a data\-constrained high\-quality domain, relaxing the quality filter to increase pool size consistently outperforms repeating the highest\-quality slice starting from the 26M 99th percentile data pool\. A crossover at 5B tokens shows the optimal threshold shifts from Q95–99 to Q90–99 as training progresses\. Here we present an expanded experimental suite, including additional data scales that support and extend these findings\. We conduct three families of experiments, all using 100M\-parameter models trained on a two\-domain mixture of high\-quality \(HQ\) filtered text and unlimited general DCLM\. Each experiment sweeps target weightw∈\{0\.1,0\.2,…,1\.0\}w\\in\\\{0\.1,0\.2,\\ldots,1\.0\\\}\.
### Non\-overlapping quality buckets at multiple data scales
We define non\-overlapping quality bands \(Q90–99, Q95–99, Q99–100\) and train at three data scales:
- •1×\\times: pools range from 26M to 982M tokens including severe data constraint, and high repetition especially for high quality data\.
- •5×\\times: pools range from 132M to 2\.1B tokens\. This imposes a moderate constraint where the highest qualitybuckets have high repetition counts, but lower quality buckets do not\.
- •10×\\times: pools range from 264M to 4\.2B tokens\. Most quality buckets avoid significant repetition, and only the highest quality bucket still overfits with too much repetition\.
\(a\)Pool sizes: Q99–100 \(132M\), Q95–99 \(821M\), Q90–99 \(2\.1B\), Q90–95 \(1\.2B\)\.
\(b\)Pool sizes: Q99–100 \(264M\), Q95–99 \(1\.7B\), Q90–99 \(4\.2B\), Q90–95 \(2\.5B\)\.
Figure 23:Loss curves for four quality bands at increasing data scales\. Each line corresponds to a mixture weighthh\. Broader quality filters enable higher mixture weights without overfitting, with the effect diminishing as pool size increases\.These experiments disentangle quality from quantity by ensuring no data overlap between bands, and reveal how the quality–quantity tradeoff evolves with different data constrained data scales\. Scaling up from the 26M target data pool to 5×\\timesand 10×\\timesthe pool size confirms the main findings and reveals how the quality–quantity tradeoff evolves with data availability\. At 5×\\times, Q95–99 dominates for the first∼\{\\sim\}14B tokens before Q90–99 overtakes it \([Figure˜23\(a\)](https://arxiv.org/html/2605.12715#S15.F23.sf1)\)\. The same crossover is observed at 1×\\timesbut shifted later due to the larger pool delaying saturation\. At 10×\\times, Q95–99 remains the best band throughout 30B tokens of training \([Figure˜23\(b\)](https://arxiv.org/html/2605.12715#S15.F23.sf2)\): with data pools exceeding 1\.6B tokens, no band experiences meaningful repetition at reasonable values ofhh, and higher per\-token quality is the dominant factor\. This demonstrates that the quality–quantity tradeoff is scale\-dependent: at small scales, breadth wins; at large scales, quality wins; and the crossover token count increases predictably with pool size\. We include results with Q90\-95 as a control to show that simply increasing the amount of data but decreasing quality performs worse\. The best\-loss per quality over allhhis shown in[Figure˜24](https://arxiv.org/html/2605.12715#S15.F24), and confirms this quantitatively: at 1×\\timesstarting from 132M target data size, a crossover from Q95–99 to Q90–99 occurs at∼\{\\sim\}14B tokens \(later than the∼\{\\sim\}5B crossover at 1×\\times\), while at 10×\\timesQ95–99 leads throughout\. As pool size grows, the quality–quantity tradeoff vanishes and per\-token quality is most important\.
\(a\)5×\\timesdata scale\.
\(b\)10×\\timesdata scale\.
Figure 24:Best achievable target loss by quality band at \(a\) 5×\\timesand \(b\) 10×\\timestarget data scales\. Unlike the 1×\\timessetting \([figure˜7](https://arxiv.org/html/2605.12715#S4.F7)\), at larger pool sizes no single quality band dominates—the curves largely overlap, confirming that the quality–quantity tradeoff diminishes when pools are large enough to avoid severe repetition\.
### Experiments with overlapping buckets
We use the full untruncated pool at each quality threshold \(Q80–100: 5B, Q90–100: 2B, Q95–100: 954M, Q99–100: 132M\) and train for 10B tokens\. In this expeirment, the quality bands are \(e\.g\., all contain the Q99\-100 data\), representing the practical approach of setting a single lower\-bound quality threshold\. The results are consistent\. The 95th percentile, Q95–100, achieves the best loss consistent with the Q95\-99 results for up to 10B tokens in[Figure˜25](https://arxiv.org/html/2605.12715#S15.F25)\.
Figure 25:Loss curves for inclusive quality thresholds \(each band includes all data above the given percentile\)\. Pool sizes increase from left to right as broader thresholds include more data\.
### Equal\-size data constrained size experiments\.
We truncate each quality band \(Q80–100, Q90–100, Q95–100, Q99–100\) to approximately the same pool size \(∼\{\\sim\}132–139M tokens\) and train for 10B tokens\. This isolates the effect of per\-token quality by controlling for pool size as all bands have approximately the same number of repetitions\.[26](https://arxiv.org/html/2605.12715#S15.F26)shows results when the data constrained pools are equal size\. With the impact of repetition equalized across quality, the ordering is strict: Q99–100\>\>Q95–100\>\>Q90–100\>\>Q80–100 at all training steps\.
Figure 26:Best achievable target loss with all quality bands truncated to the same pool size\. With repetition equalized, higher per\-token quality consistently achieves lower loss\.
### Final remarks
When pool size is equalized, quality improves validation loss\. When pools vary naturally, the repetition penalty from small high\-quality pools outweighs the quality advantage\. The non\-overlapping band experiments isolate the mechanism: it is the data in broader bands \(Q90–99 beyond Q95–99\) that helps at larger training budgets, not merely reduced repetition of the top slice\. The scale dependence of the crossover has a practical implication that for any given training budget and data pool, there exists an optimal quality threshold that balances per\-token quality against repetition\. Our scaling law captures this tradeoff enabling practitioners to extrapolate which data pool to use and for how many repetitions\.
## 16Comparison to Unlimited Target Data
To establish an upper bound on target\-domain performance, we train all model sizes with a∼60\{\\sim\}60B German token pool – large enough that repetition never occurs\. Figure[27](https://arxiv.org/html/2605.12715#S16.F27)shows target loss for each mixture fractionhhin this unlimited setting\. Without data constraints, higher target fractions monotonically achieve lower target loss at every training budget: there is no overfitting, no U\-shaped behavior, and no quality–quantity tradeoff\.
Figure 27:German target loss with unlimited data \(∼60\{\\sim\}60B pool,r<1r<1throughout\) for all mixture fractionshh\. Without data constraints, higher target fractions always achieve lower target loss—no overfitting occurs\.Figure[28](https://arxiv.org/html/2605.12715#S16.F28)then compares the best achievable loss under constrained pools \(50M–1B\) against this unlimited baseline\. The gap between each constrained curve and the unlimited line quantifies the cost of the data constraint\. Constrained pools plateau as training progresses while the unlimited baseline continues to improve, confirming that the performance ceiling is imposed by data exhaustion rather than model capacity\. The gap widens with training budget, underscoring the importance of optimal mixture design when target data is scarce\.
Figure 28:Best achievable German target loss vs\. training tokens for each model size, comparing constrained data pools against an unlimited \(∼60\{\\sim\}60B\) baseline where target data is never repeated\.
## 17Analysis of Three\-Domain Experiments
Section[7](https://arxiv.org/html/2605.12715#S7)presents results under proportional weighting; here we report the complementary equal\-weighting experiments \(hwiki=hpeS2oh\_\{\\text\{wiki\}\}=h\_\{\\text\{peS2o\}\}\)\.
Figure[29](https://arxiv.org/html/2605.12715#S17.F29)shows optimal repetition under equal weighting with the 10% compute confidence band\. The core pattern from the proportional case carries over: optimalrrgrows steadily with training budget, and smaller data pools require substantially higher repetition before the optimum is reached\. Under equal weighting, the smaller domain \(Wiki\) is repeated more aggressively than under proportional weighting—since equal weights allocate the same fraction to both domains regardless of pool size, the smaller pool exhausts its unique tokens earlier\. For example, at 10B training tokens in the base configuration \(Wiki 100M, peS2o 500M\), equal weighting reachesr≈14r\\approx 14while proportional weighting stays atr≈6r\\approx 6\(measured with respect to the Wiki pool\)\. The confidence bands remain wide, confirming that approximate repetition estimates suffice for near\-optimal performance under either strategy\.
Figure 29:Optimal repetitions under equal weighting with 10% compute confidence band \(shaded\) in the three\-domain setup \(101M model\)\.Figure 30:Per\-domain best loss under equal vs\. proportional weighting across five pool\-size configurations\.Additionally, we examined how the choice of weighting strategy affects each target domain individually\. Beyond the configurations reported in the main text, we include a 10×\\timesless experiment \(Wiki 10M, peS2o 50M\) to stress\-test both strategies under severe data constraints\. Figure[30](https://arxiv.org/html/2605.12715#S17.F30)breaks down the per\-domain loss under both strategies across all five configurations\. Proportional weighting consistently favors the larger pool \(peS2o\), achieving lower peS2o loss at the expense of Wiki\. Equal weighting reverses this tradeoff: it assigns more relative weight to the smaller domain, improving Wiki loss but sacrificing peS2o performance\. The effect is most pronounced when pool sizes differ substantially—in the 5×\\timesmore configuration, proportional weighting achieves 0\.12 lower peS2o loss than equal weighting, while equal weighting gains only 0\.03 on Wiki\. The*swapped*configuration \(where Wiki becomes the larger pool\) confirms this mechanism: when pool sizes are reversed, proportional weighting now favors Wiki instead\.
Figure[31](https://arxiv.org/html/2605.12715#S17.F31)shows the best achievable per\-domain loss throughout training for all four pool\-size configurations\.
Figure 31:Best achievable per\-domain loss throughout training under equal \(left\) and proportional \(right\) weighting for four pool\-size configurations\. Blue: Wikipedia; red: peS2o; dashed purple: average\.
## 18Baseline Scaling Law Formulas
We describe the baseline formulas used in Section[6](https://arxiv.org/html/2605.12715#S6)\. All formulas predict the target\-domain loss as a function of the total tokensDtotalD\_\{\\text\{total\}\}, the target weighthh, the number of repetitionsr=hDtotal/Dtargetr=h\\,D\_\{\\text\{total\}\}/D\_\{\\text\{target\}\}, and optionally the model sizeNN\.
### Repetitions\-agnostic\(shukor2025scalinglawsoptimaldata\)\.
This baseline treats all tokens as unique regardless of repetition:
L=E\+ADeffα\+γh,Deff=\(1−h\)Dtotal\+τhDtotal\.L=E\+\\frac\{A\}\{D\_\{\\mathrm\{eff\}\}^\{\\alpha\}\}\+\\gamma\\,h\\,,\\qquad D\_\{\\mathrm\{eff\}\}=\(1\-h\)\\,D\_\{\\text\{total\}\}\+\\tau\\,h\\,D\_\{\\text\{total\}\}\\,\.\(18\.1\)The effective data is a weighted sum of generic and target tokens with no repetition discounting—repeated target tokens are valued the same as fresh ones\. Parameters:EE,AA,α\\alpha,τ\\tau,γ\\gamma\(5 parameters\)\.
### Domain\-agnostic\(muennighoff2023scaling\)\.
This baseline uses a single saturating function of total tokens without distinguishing domains:
L=E\+ADeffα,Deff=C\(1−e−μR\),L=E\+A\\,D\_\{\\mathrm\{eff\}\}^\{\\,\\alpha\}\\,,\\qquad D\_\{\\mathrm\{eff\}\}=C\\bigl\(1\-e^\{\-\\mu\\,R\}\\bigr\)\\,,\(18\.2\)whereC=\(1−h\)Dtotal\+hDtotal/rC=\(1\-h\)\\,D\_\{\\text\{total\}\}\+h\\,D\_\{\\text\{total\}\}/ris the total number of unique tokens andR=Dtotal/CR=D\_\{\\text\{total\}\}/Cis the overall repetition ratio\. Parameters:EE,AA,α\\alpha\(<0<0\),μ\\mu\(4 parameters\)\.
### Utility decay\(goyal2024scaling\)\.
This baseline models repetition through an exponentially decaying exponent:
L=E\+aDtotalbeff,beff=\(1−h\)b0\+hb1δr−1,L=E\+a\\,D\_\{\\text\{total\}\}^\{\\,b\_\{\\mathrm\{eff\}\}\}\\,,\\qquad b\_\{\\mathrm\{eff\}\}=\(1\-h\)\\,b\_\{0\}\+h\\,b\_\{1\}\\,\\delta^\{r\-1\}\\,,\(18\.3\)whereδ=0\.51/τ\\delta=0\.5^\{1/\\tau\}is a half\-life parameter\. As repetitions increase, the exponent on the target\-domain contribution decays toward zero, reducing the data\-scaling benefit\. Parameters:EE,aa,b0b\_\{0\},b1b\_\{1\},τ\\tau\(5 parameters\)\.
### Multi\-size extensions\.
Each baseline extends to variable model size by adding a capacity termC/NβC/N^\{\\beta\}\(and for the repetitions\-agnostic and domain\-agnostic formulas, a data\-size couplingNδN^\{\\delta\}\), following the same pattern asLsizeL\_\{\\mathrm\{size\}\}\(Eq\.[5\.3](https://arxiv.org/html/2605.12715#S5.E3)\)\.
## 19Extended Scaling Law Results
This section reports the full results underlying Section[6](https://arxiv.org/html/2605.12715#S6), including the train/test breakdown for the loss prediction tables, the per\-dataset optimal\-mixture prediction errors, and the size\-extrapolation wasted\-token metric\.
### Loss prediction: train and testwR2wR^\{2\}\(fixed\-size formula\)\.
Table[5](https://arxiv.org/html/2605.12715#S19.T5)reports train and testwR2wR^\{2\}on all four experimental setups for the fixed\-size formulaLfixL\_\{\\mathrm\{fix\}\}and the three baselines\. Train corresponds to the first 50% of training steps, test to the held\-out second half\.
Table 5:Train and test weightedR2R^\{2\}\(wR2wR^\{2\}\) for the fixed\-size formulaLfixL\_\{\\mathrm\{fix\}\}, fitted independently per model size \(or per quality level / domain\)\. Each observation is weighted bymax\(r⋅h,ϵ\)\\max\(r\\cdot h,\\epsilon\)to emphasise the high\-repetition regime\. Train: first 50% of steps; test: second half\.GermanMathsQualityWiki/peS2oTrainTestTrainTestTrainTestTrainTestLfixL\_\{\\mathrm\{fix\}\}0\.850\.950\.990\.880\.910\.710\.680\.80Rep\-agnostic0\.810\.780\.950\.780\.860\.140\.670\.72Utility decay0\.520\.720\.770\.550\.94−\-0\.640\.690\.79Domain\-agnostic0\.03−\-40\.70\.10−\-0\.490\.46−\-2\.190\.42−\-1\.16
### Loss prediction: train and testwR2wR^\{2\}\(size extrapolation\)\.
Table[6](https://arxiv.org/html/2605.12715#S19.T6)reports the same metric for the multi\-size formulaLsizeL\_\{\\mathrm\{size\}\}, fitted on all model sizes except the largest and evaluated on the held\-out largest model\.
Table 6:Size extrapolation: weightedR2R^\{2\}\(wR2wR^\{2\}\) for the multi\-size formulaLsizeL\_\{\\mathrm\{size\}\}, fitted on smaller model sizes and evaluated on the held\-out largest model \(539M for German, 936M for Maths\)\. Train:wR2wR^\{2\}on fitting sizes; Test:wR2wR^\{2\}on held\-out size\.German \(539M\)Maths \(936M\)TrainTestTrainTestLsizeL\_\{\\mathrm\{size\}\}0\.950\.650\.970\.73Rep\-agnostic\+NN0\.880\.590\.940\.71Utility decay\+NN0\.760\.560\.850\.69Domain\-agnostic\+NN0\.25−\-0\.230\.46−\-0\.77
### Optimal mixture prediction error\.
For each experiment \(a specific model size and target dataset size\) and each held\-out training checkpointDtotalD\_\{\\text\{total\}\}, we compare the predictedh∗h^\{\*\}to the empirically best weighthemp∗h^\{\*\}\_\{\\text\{emp\}\}at that checkpoint\. Table[7](https://arxiv.org/html/2605.12715#S19.T7)reports the median of\|log10\(hpred∗\)−log10\(hemp∗\)\|\|\\log\_\{10\}\(h^\{\*\}\_\{\\text\{pred\}\}\)\-\\log\_\{10\}\(h^\{\*\}\_\{\\text\{emp\}\}\)\|per dataset over all such pairs on held\-out test steps\. A value of0\.10\.1corresponds to a∼1\.26×\{\\sim\}1\.26\\timesmultiplicative error in the predicted weight\.
Table 7:Optimal mixture prediction on held\-out test steps: median\|log10\(hpred∗\)−log10\(hemp∗\)\|\|\\log\_\{10\}\(h^\{\*\}\_\{\\text\{pred\}\}\)\-\\log\_\{10\}\(h^\{\*\}\_\{\\text\{emp\}\}\)\|\(lower is better\)\. A value of0\.10\.1corresponds to a∼1\.26×\{\\sim\}1\.26\\timeserror in the predicted fraction\.Ger\.MathQual\.WikiAvgLfixL\_\{\\mathrm\{fix\}\}0\.070\.130\.470\.100\.19Rep\-agnostic0\.601\.010\.610\.500\.68Utility decay0\.100\.200\.380\.120\.20Domain\-agn\.2\.571\.995\.303\.073\.23LsizeL\_\{\\mathrm\{size\}\}0\.150\.18——0\.17Rep\-agn\.\+NN0\.731\.44——1\.09Util\. dec\.\+NN0\.150\.17——0\.16Dom\.\-agn\.\+NN3\.271\.56——2\.42
### Size extrapolation: wasted tokens\.
A stronger test of the multi\-size formula is whether it can predict the behaviour of a model size never seen during fitting\. We fitLsizeL\_\{\\mathrm\{size\}\}and all multi\-size baselines on all model sizes*except the largest*and evaluate exclusively on the held\-out largest model \(539M for German, 936M for Maths\)\. Table[8](https://arxiv.org/html/2605.12715#S19.T8)reports the weighted median wasted token fraction on the held\-out largest model for each setup\.LsizeL\_\{\\mathrm\{size\}\}achieves a testwR2wR^\{2\}of0\.650\.65on German and0\.730\.73on Maths \(Table[6](https://arxiv.org/html/2605.12715#S19.T6)\), demonstrating that the capacity and data\-size coupling terms extrapolate meaningfully to larger scales\. The wasted token metric is even more discriminating: most baselines fail catastrophically on German, producing predictions outside the empirical range \(counted as100%100\\%\)\.LsizeL\_\{\\mathrm\{size\}\}wastes58%58\\%on German and24%24\\%on Maths, the only formula that avoids catastrophic failure on both\. Results for French and Swahili are reported in Appendix[19](https://arxiv.org/html/2605.12715#S19.SS0.SSS0.Px6)\.
Table 8:Size extrapolation: weighted median wasted token fraction \(lower is better\)\. Predictions outside the empirical weight range are counted as100%100\\%\.German \(539M\)Maths \(936M\)LsizeL\_\{\\mathrm\{size\}\}58%24%Rep\-agnostic\+NN100%97%Utility decay\+NN100%22%Domain\-agnostic\+NN100%84%
### Limitations
The scaling law is descriptive: it summarises observed training dynamics but does not prescribe an optimal allocation from first principles\. Size extrapolation is encouraging on both German \(wR2=0\.65wR^\{2\}=0\.65at 539M\) and Maths \(wR2=0\.73wR^\{2\}=0\.73at 936M\), though the wasted\-token metric reveals that predictions remain imperfect at unseen scales\. The quality\-filtering setup is harder to predict than language/domain splits \(testwR2=0\.71wR^\{2\}=0\.71, log\-error=0\.47=0\.47\), likely because quality tiers interact with the model in ways not fully captured by the effective data abstraction\. Finally, our experiments span∼100M\{\\sim\}100\\text\{M\}–900M900\\text\{M\}parameters; whether the fitted exponents extrapolate to multi\-billion\-parameter models remains an open question\.
### Size Extrapolation: French and Swahili
We repeat the size\-extrapolation experiment of Section[6](https://arxiv.org/html/2605.12715#S6)on French and Swahili\. For each language, we fit the multi\-size formulas on model sizes 101M–340M and evaluate on the held\-out 539M model\. Table[9](https://arxiv.org/html/2605.12715#S19.T9)reports weightedR2R^\{2\}\(test / train\)\.
On French,LsizeL\_\{\\mathrm\{size\}\}achieves a testwR2wR^\{2\}of0\.930\.93, even exceeding its trainwR2wR^\{2\}of0\.920\.92—indicating that the 539M model’s behaviour is well predicted by trends from smaller scales\. On Swahili, extrapolation is harder:LsizeL\_\{\\mathrm\{size\}\}achieves0\.690\.69on the held\-out 539M model, while the utility decay baseline reaches0\.710\.71\. The domain\-agnostic baseline yields near\-zero or negativewR2wR^\{2\}on both languages\.
For wasted tokens, all formulas \(includingLsizeL\_\{\\mathrm\{size\}\}\) produce predictions outside the empirical weight range for French and Swahili at 539M, resulting in≥94%\{\\geq\}94\\%waste\. This reflects the narrower range of target fractions tested at this scale for these languages rather than a fundamental failure of the formula\.
Table 9:Size extrapolation on French and Swahili: weightedR2R^\{2\}\(test / train\) when fitting on 101M–340M and evaluating on 539M\.French \(539M\)Swahili \(539M\)LsizeL\_\{\\mathrm\{size\}\}\(9p\)0\.93/0\.920\.69 /0\.92Rep\-agnostic\+NN\(8p\)0\.87 / 0\.900\.65 / 0\.90Utility decay\+NN\(7p\)0\.69 / 0\.620\.71/ 0\.69Domain\-agnostic\+NN\(7p\)−\-0\.54 / 0\.08−\-0\.00 / 0\.02
## 20Robustness to Weighting Scheme
We verify that the qualitative ranking of formulas is robust to the choice of observation weights in thewR2wR^\{2\}metric\. Table[10](https://arxiv.org/html/2605.12715#S20.T10)reports testwR2wR^\{2\}on the German dataset under four weighting schemes:max\(r⋅h,ϵ\)\\max\(r\\cdot h,\\,\\epsilon\)\(our default\),rronly,hhonly, and uniform \(standardR2R^\{2\}\)\. The rankingLfixL\_\{\\mathrm\{fix\}\}\>\>rep\-agnostic\>\>utility decay\>\>domain\-agnostic is preserved across all schemes, confirming that the advantage of our formula is not an artifact of the weighting\.
Table 10:TestwR2wR^\{2\}on German under different weighting schemes\. The formula ranking is stable across all choices\.r⋅hr\\cdot h\(default\)rrhhUniformLfixL\_\{\\mathrm\{fix\}\}\(6p\)0\.960\.950\.780\.90Rep\-agnostic \(5p\)0\.810\.820\.520\.85Utility decay \(5p\)0\.780\.850\.390\.71Domain\-agnostic \(4p\)<<−\-1<<−\-1<<−\-1<<−\-1
## 21Downstream Benchmark Evaluation
We evaluate downstream task performance for the best\-loss configurations from Figure[6](https://arxiv.org/html/2605.12715#S4.F6)\(one checkpoint per model size and data pool combination, selected at the training step that minimizes German test loss\)\. We use seven benchmarks covering reasoning, commonsense, and language understanding: ARC\-Challenge and ARC\-Easy\(clark2018think\)\(science question answering\), HellaSwag\(zellers2019hellaswag\)\(commonsense sentence completion\), LAMBADA\(paperno2016lambada\)\(long\-range word prediction\), PIQA\(bisk2020piqa\)\(physical intuition reasoning\), SciQ\(welbl2017sciq\)\(science multiple\-choice\), and WinoGrande\(sakaguchi2020winogrande\)\(coreference resolution\)\. Each benchmark is evaluated in both English \(original\) and German \(translated\), allowing us to assess both target\-domain and non\-target\-domain capabilities from the same checkpoint\.
Tables[11](https://arxiv.org/html/2605.12715#S21.T11)and[12](https://arxiv.org/html/2605.12715#S21.T12)report accuracy on all benchmarks\. German benchmark performance improves consistently with both model size and target data pool size, confirming that the perplexity gains translate to downstream improvements\. English performance decreases as more training budget is allocated to German \(higherhh\), reflecting the expected trade\-off between target and non\-target domains\. Larger models partially mitigate this cost: the 539M model retains 48\.6% English average even at the highest German allocation, compared to 39\.7% for 101M\.
Table 11:German downstream benchmark accuracy \(%\) at the best target\-loss checkpoint\. More target data consistently improves German performance across all model sizes\.ModelDataset sizeARC\-CARC\-EHSwagLambadaPIQASciQWinoAvg101M50M24\.031\.028\.013\.752\.160\.450\.837\.2101M100M24\.531\.228\.415\.253\.063\.250\.237\.9101M500M23\.432\.629\.817\.655\.360\.150\.338\.5101M1B25\.730\.929\.819\.157\.059\.148\.938\.6143M50M23\.229\.928\.615\.352\.461\.750\.137\.3143M100M23\.731\.928\.716\.354\.462\.351\.538\.4143M500M23\.733\.529\.819\.356\.563\.250\.839\.6143M1B24\.333\.130\.520\.756\.062\.850\.739\.7192M50M24\.131\.028\.915\.453\.762\.149\.737\.8192M100M24\.232\.729\.616\.953\.963\.250\.538\.7192M500M24\.132\.331\.220\.057\.662\.751\.339\.9192M1B24\.235\.331\.220\.857\.562\.252\.940\.6340M50M24\.231\.229\.215\.752\.862\.651\.438\.2340M100M25\.033\.030\.418\.453\.764\.249\.739\.2340M500M24\.833\.832\.224\.157\.665\.353\.041\.5340M1B25\.336\.132\.924\.657\.264\.352\.541\.8539M50M24\.231\.029\.819\.052\.964\.650\.038\.8539M100M26\.131\.831\.619\.754\.765\.351\.540\.1539M500M25\.135\.334\.323\.657\.464\.552\.541\.8539M1B25\.237\.335\.124\.359\.365\.251\.442\.6Table 12:English downstream benchmark accuracy \(%\) at the same checkpoints\. More target data \(higherhh\) reduces English performance, reflecting the trade\-off between target and non\-target domains\.ModelPoolARC\-CARC\-EHSwagLambadaPIQASciQWinoAvg101M50M23\.140\.232\.931\.464\.762\.750\.743\.7101M100M23\.140\.232\.630\.864\.965\.349\.843\.8101M500M21\.335\.328\.824\.661\.657\.851\.540\.1101M1B21\.835\.528\.022\.560\.757\.851\.539\.7143M50M23\.240\.434\.636\.665\.264\.050\.544\.9143M100M24\.641\.234\.932\.566\.567\.751\.045\.5143M500M23\.338\.832\.229\.964\.666\.048\.943\.4143M1B22\.438\.930\.328\.863\.062\.851\.942\.6192M50M24\.741\.337\.935\.567\.366\.651\.946\.4192M100M23\.941\.336\.835\.867\.767\.451\.246\.3192M500M23\.441\.035\.032\.065\.866\.150\.744\.9192M1B22\.138\.631\.329\.463\.062\.751\.842\.7340M50M24\.245\.241\.939\.868\.969\.752\.948\.9340M100M24\.344\.642\.340\.669\.269\.951\.248\.9340M500M25\.444\.240\.140\.068\.570\.151\.748\.6340M1B24\.342\.338\.538\.566\.267\.651\.347\.0539M50M26\.446\.646\.545\.470\.371\.452\.851\.3539M100M24\.747\.546\.545\.469\.572\.453\.751\.4539M500M24\.646\.645\.243\.169\.270\.152\.450\.2539M1B24\.345\.442\.342\.167\.569\.049\.748\.6
## 22Limitations
We did not tune training hyperparameters jointly with the mixture and repetition settings, instead using standard values throughout\. It is possible that interactions between learning rate, batch size, or schedule and the number of repetitions could shift the optimal operating points we report\. Additionally, all experiments use a single architecture family \(GPT\-2\-style decoder\-only transformers\); whether the observed scaling behavior transfers to other architectures remains an open question\. We leave both explorations to future work\. Our largest model is 939M parameters; while we observe consistent trends across model sizes suggesting the scaling law extrapolates, verifying this at scales of tens or hundreds of billions of parameters would require substantially more compute than available for this study\.
††Apple and the Apple logo are trademarks of Apple Inc\., registered in the U\.S\. and other countries and regions\.Similar Articles
Prescriptive Scaling Laws for Data Constrained Training
A modified scaling law accounting for data repetition effects provides compute-optimal training strategies for data-constrained scenarios, showing that beyond a point further repetition is counterproductive and compute is better spent on model capacity.
Repetition Mismatch: Why Data Mixture Experiments Don't Scale and How to Fix Them
The paper identifies repetition mismatch as a primary cause for data mixture experiments failing to scale, and proposes a repetition-controlled subsampling procedure that allows small-scale experiments to recover near-optimal mixtures using far fewer tokens.
InfoLaw: Information Scaling Laws for Large Language Models with Quality-Weighted Mixture Data and Repetition
InfoLaw is a data-aware scaling framework that predicts model loss based on token consumption, model size, data mixture weights, and repetition, enabling efficient data-recipe selection under varying compute budgets.
Predictable Scaling Laws of Optimal Hyperparameters for LLM Continued Pre-training
This paper discovers predictable scaling laws for optimal hyperparameters (learning rate, batch size) in LLM continued pre-training, proposing a two-stage framework that reduces hyperparameter search overhead by up to 90% while maintaining performance.
Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws
This paper studies data-constrained language model pretraining, proposing masked-input regularization (MIR) to improve validation loss and downstream performance, and SoftQ, a scaling law that better captures model-data interaction under repeated data.