Holistic Data Scheduler for LLM Pre-training via Multi-Objective Reinforcement Learning
Summary
Introduces Holistic Data Scheduler (HDS), a reinforcement learning-based framework that dynamically adjusts data mixtures during LLM pre-training using a multi-objective reward function, achieving 44% fewer iterations to reach target perplexity and a 7.2% improvement on MMLU.
View Cached Full Text
Cached at: 06/24/26, 05:46 AM
Paper page - Holistic Data Scheduler for LLM Pre-training via Multi-Objective Reinforcement Learning
Source: https://huggingface.co/papers/2606.24133
Abstract
A novel online data mixing framework called Holistic Data Scheduler uses reinforcement learning with a multi-objective reward function to optimize large language model pre-training efficiency and performance.
The composition of training data, governed by the diversity of sources and their mixing strategy, is a cornerstone ofLarge Language Model(LLM) pre-training.Online Data Mixing(ODM), the technique of adaptively adjusting data mixtures during training, has emerged as a promising direction to improve efficiency. However, existing methods are constrained by their reliance on a singular optimization perspective, which fundamentally overlooks the need for complex LLM pre-training to consider the dynamic data composition from multiple dimensions. To overcome this limitation, we introduce the Holistic Data Scheduler (HDS), a novelonline data mixingframework. HDS formulates the data scheduling challenge as areinforcement learningproblem in a continuous control space and leverages theSoft Actor-Critic(SAC) algorithm for its stability and sample efficiency in exploring the high-dimensional policy space. At the core of HDS lies a novel multi-objective, holistic reward function that integrates three critical perspectives: adata-driven rewardfor quality, aloss-driven rewardcapturing inter-domain influence, and amodel-driven rewardbased on weight norms. To validate our design and determine its optimal configuration, we conducted systematic experiments on LLMs of various sizes. On The Pile benchmark, HDS reaches the final validation perplexity of the next best method with 44% fewer training iterations. Furthermore, it achieves a 7.2% improvement on the MMLU 0-shot task along with consistent gains on other benchmarks, showcasing its ability to enhance bothtraining efficiencyand finalmodel capability.
View arXiv pageView PDFProject pageGitHub199Add to collection
Get this paper in your agent:
hf papers read 2606\.24133
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.24133 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.24133 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.24133 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
AC-ODM: Actor--Critic Online Data Mixing for Sample-Efficient LLM Pretraining
AC-ODM uses reinforcement learning to dynamically optimize pretraining data composition for LLMs, achieving faster convergence and higher downstream accuracy with negligible computational overhead.
RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training
Harvard researchers challenge the standard LLM training pipeline by showing RL can be effectively applied during pre-training rather than only after SFT, finding that data composition matters more than model scale, and proposing parallel averaging of RL and SFT objectives that outperforms sequential approaches while preserving general capabilities.
DHRCL:Training Code LLMs with Dense Hierarchical Rewards and Curriculum Learning
DHRCL proposes a reinforcement learning framework with dense hierarchical rewards and curriculum learning for training code LLMs, using syntax validation, execution success, unit-test pass rate, and AST structural similarity in a three-stage curriculum.
PPO-HSC: An Exploratory Reinforcement Learning Framework Based on Wide-Area Policy Coverage Optimization
PPO-HSC introduces a High-order Sampling Coverage reward to encourage exploration of diverse reasoning patterns in RL fine-tuning of LLMs, improving solution diversity and state-space coverage on math and code tasks.
LLMZero: Discovering Adaptive Training Strategies for RL Post-Training via LLM Agents
LLMZero uses LLM agents to search over training trajectories via tree search, discovering adaptive multi-parameter transitions for RL post-training that outperform fixed schedules and grid search across diverse tasks.