data-mixing

Carnegie Mellon paper shows that pretraining, fine-tuning, and RL can collapse into a single training rule (MaxRL as likelihood training on correct answers), with blended data runs outperforming staged training on reasoning and coding benchmarks while preserving general skills.

This paper proposes optimized data mixing for supervised fine-tuning to enable reasoning models to generalize in-language reasoning across 60 languages, with a model achieving over 93% L2 reasoning rate on multiple benchmarks.

The paper formulates data mixing for large language model pretraining as a mixture experiment, applying response surface methodology and optimal experimental design to improve the efficiency and interpretability of proxy training runs.

CausalMix formulates data mixture optimization as a causal inference problem for LLM training, enabling dynamic adaptation to shifting data distributions without costly retraining, and demonstrates improved performance on Qwen2.5-0.5B and Qwen3-4B-Base.

AC-ODM uses reinforcement learning to dynamically optimize pretraining data composition for LLMs, achieving faster convergence and higher downstream accuracy with negligible computational overhead.

This paper introduces OP-Mix, a data mixing algorithm that uses low-rank adapters trained on the current model to cheaply simulate candidate data mixtures, enabling efficient and unified data mixing across pretraining, continual midtraining, and continual instruction tuning. OP-Mix consistently finds near-optimal mixtures while using a fraction of the compute of baselines, improving pretraining perplexity by 6.3% and reducing compute by 66-95% in continual learning settings.

InfoLaw is a data-aware scaling framework that predicts model loss based on token consumption, model size, data mixture weights, and repetition, enabling efficient data-recipe selection under varying compute budgets.

This paper presents a comprehensive survey of data mixing methods for LLM pretraining, formalizing the problem as bilevel optimization and introducing a taxonomy that distinguishes static (rule-based and learning-based) from dynamic (adaptive and externally guided) mixing approaches. The authors analyze trade-offs, identify cross-cutting challenges, and outline future research directions including finer-grained domain partitioning and pipeline-aware designs.