@DeFiMinty: Can Reinforcement Learning compensate for weak pretraining? Researchers tested this with chess puzzles, where every pro…
Summary
Researchers investigate how reinforcement learning (RL) interacts with pretraining quality, showing that stronger pretrained models benefit more from RL and that RL cannot fully compensate for weak pretraining. A joint scaling law is identified for pretraining and post-training compute allocation.
View Cached Full Text
Cached at: 07/22/26, 02:31 PM
Can Reinforcement Learning compensate for weak pretraining?
Researchers tested this with chess puzzles, where every proposed move can be checked against a known solution.
They pretrained models from 5M to 1B parameters on human games before applying supervised fine-tuning and RL.
Better pretrained models performed better after receiving the same amount of RL compute and improved faster as RL continued. Weakly pretrained models saw limited gains, suggesting that RL could not fully compensate for insufficient pretraining in this setting.
RL behaved differently depending on puzzle difficulty. On easier puzzles, it mostly increased confidence in correct moves the model already favored. On harder puzzles, it sometimes made a previously overlooked correct move much more likely, but it could also increase confidence in an incorrect move.
A smaller experiment with a 1B math model found the same pattern with longer-pretrained checkpoints performing better under RL.
RL appears to get more leverage from a strong foundation.
Pavel Izmailov (@Pavel_Izmailov): New paper: Understanding Reasoning from Pretraining to Post-Training!
We study the full LLM training pipeline from pretraining to post-training, find a joint scaling law, figure out how the compute should be allocated, and study what RL is doing to the policy.
🧵
Similar Articles
Understanding Reasoning from Pretraining to Post-Training
This paper investigates the relationship between pretraining and reinforcement learning (RL) post-training for large language models using chess as a controlled testbed. It establishes a scaling law connecting pretraining loss to post-RL performance and shows that RL amplifies correct moves on easy puzzles while surfacing new correct moves on hard ones.
Understanding Reasoning from Pretraining to Post-Training
This paper studies how pretraining choices (model size, data) affect returns from RL post-training on reasoning tasks, using chess as a controlled testbed. It finds that post-RL performance is well-predicted by pretraining loss and that RL amplifies correct moves on easy puzzles while surfacing correct moves on hard puzzles, with findings transferring to math domains.
@sebkrier: Training models with RL can often lead to the reward signal being gamed; for example when you use an LLM judge for fuzz…
Debate training can mitigate reward hacking in reinforcement learning from AI feedback (RLAIF) by using a debate opponent to improve ground-truth accuracy for fuzzy tasks.
RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training
Harvard researchers challenge the standard LLM training pipeline by showing RL can be effectively applied during pre-training rather than only after SFT, finding that data composition matters more than model scale, and proposing parallel averaging of RL and SFT objectives that outperforms sequential approaches while preserving general capabilities.
@ickma2311: CMU Advanced NLP: Reinforcement Learning I had been curious about how RL works on top of LLMs, and this CMU lecture mad…
CMU Advanced NLP lecture clarifies how reinforcement learning optimizes whole-output rewards (correctness, helpfulness, safety) rather than next-token prediction used in pretraining/fine-tuning.