Scaling Repeated-Data Pretraining: Batch Size, Regularization, and GPU-Time Tradeoffs

Hugging Face Daily Papers Papers
pretraining repeated-data batch-size regularization compute-scaling gpu-efficiency training-recipes

Summary

该论文研究在数据有限但算力充裕场景下的重复数据预训练，发现小批量通过梯度噪声提供隐式正则化，并在固定时间预算下小批量+少GPU可取得更低验证损失，挑战了常见地增大数据并行度以最大化GPU利用率的做法。

We study pretraining when compute is abundant but unique training data are finite, requiring repeated training on the same data. We first revisit infinite-compute pretraining (Kim et al. 2026) and show that its frontier depends critically on choices previously fixed or studied only in isolation. In particular, optimizing batch size together with epoch counts reveals a U-shaped relationship between validation loss and batch size, favoring smaller batches than typical pretraining, likely because small-batch gradient noise provides implicit regularization. We further find that training recipes, previously optimized under a single-pass assumption, can be tuned, e.g., repacking documents into new sequences every epoch, weight tying, and positive embedding weight decay. We then relax the infinite-compute assumption. In particular, smaller-than-typical batch sizes can require prohibitively long sequential training, motivating to treat GPUs and time as separate resources. Tighter deadlines favor larger batches, and latency-oriented parallelism can trade additional GPUs for shorter training. Nevertheless, even under a fixed time budget, a smaller batch with fewer GPUs can achieve lower validation loss than a larger batch with more GPUs, challenging the common practice of increasing data parallelism to maximize GPU utilization under a deadline. Together, our results identify underexplored factors in repeated-data training and provide concrete guidance on how training recipes and resource allocation should change from single-pass pretraining.
Original Article
View Cached Full Text

Cached at: 10/14/26, 02:52 PM

Paper page - Scaling Repeated-Data Pretraining: Batch Size, Regularization, and GPU-Time Tradeoffs

Source: https://huggingface.co/papers/2610.15965

Abstract

Westudypretrainingwhencomputeisabundantbutuniquetrainingdataarefinite,requiringrepeatedtrainingonthesamedata.Wefirstrevisitinfinite-computepretraining(Kimetal.2026)andshowthatitsfrontierdependscriticallyonchoicespreviouslyfixedorstudiedonlyinisolation.Inparticular,optimizingbatchsizetogetherwithepochcountsrevealsaU-shapedrelationshipbetweenvalidationlossandbatchsize,favoringsmallerbatchesthantypicalpretraining,likelybecausesmall-batchgradientnoiseprovidesimplicitregularization.Wefurtherfindthattrainingrecipes,previouslyoptimizedunderasingle-passassumption,canbetuned,e.g.,repackingdocumentsintonewsequenceseveryepoch,weighttying,andpositiveembeddingweightdecay.Wethenrelaxtheinfinite-computeassumption.Inparticular,smaller-than-typicalbatchsizescanrequireprohibitivelylongsequentialtraining,motivatingtotreatGPUsandtimeasseparateresources.Tighterdeadlinesfavorlargerbatches,andlatency-orientedparallelismcantradeadditionalGPUsforshortertraining.Nevertheless,evenunderafixedtimebudget,asmallerbatchwithfewerGPUscanachievelowervalidationlossthanalargerbatchwithmoreGPUs,challengingthecommonpracticeofincreasingdataparallelismtomaximizeGPUutilizationunderadeadline.Together,ourresultsidentifyunderexploredfactorsinrepeated-datatrainingandprovideconcreteguidanceonhowtrainingrecipesandresourceallocationshouldchangefromsingle-passpretraining.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2610\.15965

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.15965 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.15965 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.15965 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Scaling Domain Data Repetition in LLM Pretraining

Hugging Face Daily Papers

This paper studies the trade-off in repeating high-quality domain data during LLM pretraining to maintain performance as models scale, finding that optimal repetition counts increase with model size and are negatively correlated with domain validation loss.

Scaling Laws for Mixture Pretraining Under Data Constraints

arXiv cs.LG

This paper studies the trade-off between scarce target data and abundant generic data in mixture pretraining, finding that repetition is a key driver of performance and that mixture training tolerates 15-20 repetitions of target data. It introduces a repetition-aware scaling law to optimize mixture configurations under data constraints.