Beyond Data Schedule: Fast LLM Pretraining on Data Mixtures by Loss-Reweighting
Summary
This paper proposes LARE (Loss-Adaptive Reweighting under fixed Exposure), a method that adjusts domain loss weights from running loss statistics during LLM pretraining, achieving an average +7.84 pp improvement across five benchmarks for a 1.5B model over a matched data mixture.
View Cached Full Text
Cached at: 10/28/26, 01:36 AM
# Beyond Data Schedule: Fast LLM Pretraining on Data Mixtures by Loss-Reweighting
Source: [https://arxiv.org/abs/2610.32141](https://arxiv.org/abs/2610.32141)
[View PDF](https://arxiv.org/pdf/2610.32141)
> Abstract:Large language models often acquire specialized capabilities through successive pretraining stages, but these gains can come at the cost of forgetting earlier knowledge\. In this work, we investigate whether such trade\-offs can be improved by changing how strongly each domain contributes to learning while keeping its data exposure fixed\. We find that, in controlled knowledge\-\-mathematics experiments, loss reweighting produces a single checkpoint that surpasses the best knowledge and reasoning scores attained along either stage trajectory\. This reveals an independent control that data\-mixture ratios alone do not capture\. Building on this finding, we propose LARE \(Loss\-Adaptive Reweighting under fixed Exposure\), which automatically adjusts domain weights from running loss statistics\. Motivated by a theoretical analysis of gradient noise, its normalized inverse\-square\-root rule balances relative loss scales without changing the sampled mixture or requiring additional runs to search for weights\. Across knowledge, mathematics, and code, \\Method\{\} improves all five evaluated benchmarks for a 1\.5B model trained on 240B tokens, averaging \+7\.84 pp over a matched direct mixture, including gains of \+17\.06 pp on GSM8K and \+9\.00 pp on MATH\. These findings demonstrate that adjusting domain loss scales can improve joint capabilities without additional domain exposure\.
## Submission history
From: Jiazheng Li \[[view email](https://arxiv.org/show-email/379b5198/2610.32141)\] **\[v1\]**Fri, 23 Oct 2026 04:09:46 UTC \(822 KB\)Similar Articles
Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods
This paper introduces ADAPT, an online reweighting framework for LLM data curation that dynamically adjusts sample importance during training via loss weighting, outperforming offline selection and mixing methods in cross-benchmark generalization.
AC-ODM: Actor--Critic Online Data Mixing for Sample-Efficient LLM Pretraining
AC-ODM uses reinforcement learning to dynamically optimize pretraining data composition for LLMs, achieving faster convergence and higher downstream accuracy with negligible computational overhead.
Data-Efficient Adaptation of LLMs via Attention Head Reweighting
Introduces Attention Head Reweighting (AHR), a data-efficient method for adapting LLMs to text classification tasks by learning a single scalar per attention head, drastically reducing trainable parameters while outperforming LoRA in limited data settings.
DomainPilot: Domain-Level Loss-Guided Two-Stage Data Mixture Optimization for Efficient Language Model Fine-Tuning
DomainPilot introduces a domain-level loss-guided two-stage framework for data mixture optimization in LLM fine-tuning, achieving improvements on MMLU-Redux, AIME24, LiveCodeBench v5, and BFCL v3 without increasing data volume or training cost.
Data Mixing for Large Language Models Pretraining: A Survey and Outlook
This paper presents a comprehensive survey of data mixing methods for LLM pretraining, formalizing the problem as bilevel optimization and introducing a taxonomy that distinguishes static (rule-based and learning-based) from dynamic (adaptive and externally guided) mixing approaches. The authors analyze trade-offs, identify cross-cutting challenges, and outline future research directions including finer-grained domain partitioning and pipeline-aware designs.