@rohanpaul_ai: The separate stages of AI training can collapse into a single method. New Carnegie Mellon paper shows that pretraining,…
Summary
Carnegie Mellon paper shows that pretraining, fine-tuning, and RL can collapse into a single training rule (MaxRL as likelihood training on correct answers), with blended data runs outperforming staged training on reasoning and coding benchmarks while preserving general skills.
View Cached Full Text
Cached at: 10/31/26, 01:35 PM
The separate stages of AI training can collapse into a single method.
New Carnegie Mellon paper shows that pretraining, fine-tuning, and RL can share a single training rule, and blending their data often beats separate stages.
LLMs are usually trained in stages: pretraining, midtraining, fine-tuning, then RL, each with its own algorithm. Hard switches between stages can also erase earlier skills.
The authors recast a form of RL called MaxRL as likelihood training on the model’s own correct answers. That lets one trainer mix pretraining text, curated data, and RL samples in any proportion over time.
On reasoning puzzles with a 124M model, the best blended runs beat the best staged run and kept general skills better. After later coding RL, a 0.76B model that blended in 5% midtraining data during math RL reached 32.69% on MBPP+, versus 29.81% for staged training.
Treat your training pipeline as a data-mixing schedule, and try blending earlier data into RL instead of switching stages abruptly.
Similar Articles
Understanding Reasoning from Pretraining to Post-Training
This paper studies how pretraining choices (model size, data) affect returns from RL post-training on reasoning tasks, using chess as a controlled testbed. It finds that post-RL performance is well-predicted by pretraining loss and that RL amplifies correct moves on easy puzzles while surfacing correct moves on hard puzzles, with findings transferring to math domains.
Training Foundation Models with a Single Algorithm
A paper unifying foundation-model training stages (pretraining, midtraining, SFT, RL) under a single likelihood-based algorithm that blends static data and RL generations, improving accuracy-compute tradeoff and capability retention.
Understanding Reasoning from Pretraining to Post-Training
This paper investigates the relationship between pretraining and reinforcement learning (RL) post-training for large language models using chess as a controlled testbed. It establishes a scaling law connecting pretraining loss to post-RL performance and shows that RL amplifies correct moves on easy puzzles while surfacing new correct moves on hard ones.
@rohanpaul_ai: What if you could pay the reasoning cost once, then reuse what the model learned across future tasks? New Microsoft pap…
A Microsoft paper proposes a method to replace expensive test-time reasoning with distilled skills from past agent runs, showing competitive performance on some benchmarks while reducing output token usage.
@rohanpaul_ai: A Primer paper about how reasoning models improve after training Shows that better reasoning models depend less on raw …
This primer paper explores how reasoning models improve after training, arguing that effective reasoning data relies more on checkable training evidence than raw data size. It categorizes reasoning data by verification methods and emphasizes preserving messy agent data for learning signals.