AC-ODM: Actor--Critic Online Data Mixing for Sample-Efficient LLM Pretraining
Summary
AC-ODM uses reinforcement learning to dynamically optimize pretraining data composition for LLMs, achieving faster convergence and higher downstream accuracy with negligible computational overhead.
View Cached Full Text
Cached at: 06/23/26, 05:43 PM
Paper page - AC-ODM: Actor–Critic Online Data Mixing for Sample-Efficient LLM Pretraining
Source: https://huggingface.co/papers/2505.23878
Abstract
AC-ODM optimizes pretraining data composition for LLMs using reinforcement learning to improve convergence speed and downstream accuracy while maintaining computational efficiency.
Optimizingpretraining data compositionis pivotal forLLM generalization. Whiledynamic mixingoutperformsstatic strategiesby capturing evolving training dynamics, current methods fail to reconcile computational efficiency with sample efficiency and structural flexibility for diverse pipelines.We introduce Actor--Critic Online Data Mixing (AC-ODM), which approaches data mixing from areinforcement learningperspective with aparameterized policythat we theoretically prove to act as a dynamic linear surrogate maximizing theconstructive interferenceof gradients. To enhance practical flexibility, AC-ODM supports two operational modes: (i) aproxy modefor fixed, pre-prepared corpora, where a policy learned on a small model is transferred to a larger target; and (ii) anon-proxy modefor direct end-to-end training from scratch without priors. Empirically, AC-ODM significantly outperforms prior methods inconvergence speedanddownstream accuracyacross various architectures. OnPythia-1B, it reaches optimalvalidation perplexityusing up to 66% fewer training steps than competitive baselines, delivering a 27.5% relative improvement inMMLU accuracyand a 2.23 x higherpass@1onHumanEval, all while incurring a virtually negligible (0.4%) per-step wall-clock increase and only 2% additional memory overhead. Code is available at https://github.com/DANG-ai/AC-ODM.
View arXiv pageView PDFProject pageGitHub1Add to collection
Get this paper in your agent:
hf papers read 2505\.23878
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2505.23878 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2505.23878 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2505.23878 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Data Mixing for Large Language Models Pretraining: A Survey and Outlook
This paper presents a comprehensive survey of data mixing methods for LLM pretraining, formalizing the problem as bilevel optimization and introducing a taxonomy that distinguishes static (rule-based and learning-based) from dynamic (adaptive and externally guided) mixing approaches. The authors analyze trade-offs, identify cross-cutting challenges, and outline future research directions including finer-grained domain partitioning and pipeline-aware designs.
Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Time
This paper introduces OP-Mix, a data mixing algorithm that uses low-rank adapters trained on the current model to cheaply simulate candidate data mixtures, enabling efficient and unified data mixing across pretraining, continual midtraining, and continual instruction tuning. OP-Mix consistently finds near-optimal mixtures while using a fraction of the compute of baselines, improving pretraining perplexity by 6.3% and reducing compute by 66-95% in continual learning settings.
@songhan_mit: Explore lightening OPD for efficient LLM post training:
The article introduces a method to lighten OPD for efficient post-training of Large Language Models.
Data Mixing as Mixture Experiment: Response Surface Methodology and Optimal Design for Large Language Model Pretraining
The paper formulates data mixing for large language model pretraining as a mixture experiment, applying response surface methodology and optimal experimental design to improve the efficiency and interpretability of proxy training runs.
ProactiveLLM: Learning Active Interaction for Streaming Large Language Models
ProactiveLLM introduces a method for streaming LLMs to actively decide when to generate output based on endogenous cues, using mask-based streaming modeling and synchronized privileged self-distillation, reducing latency without external annotations.