Training Foundation Models with a Single Algorithm
Summary
A paper unifying foundation-model training stages (pretraining, midtraining, SFT, RL) under a single likelihood-based algorithm that blends static data and RL generations, improving accuracy-compute tradeoff and capability retention.
View Cached Full Text
Cached at: 10/30/26, 07:12 PM
Paper page - Training Foundation Models with a Single Algorithm
Source: https://huggingface.co/papers/2610.40564 Published on Oct 29
·
Submitted byhttps://huggingface.co/AZH04
Alvinon Oct 30
Abstract
Foundation-model training stages can be unified under a single likelihood-based algorithm that blends static data and reinforcement learning generations, improving efficiency and capability retention.
Foundation modelsare typically trained in stages: pretraining, midtraining, supervised fine-tuning (SFT), andreinforcement learning(RL), with changes in both data and learning algorithms. We show that this algorithmic separation is not fundamental. By deriving suitableRL objectivesasmaximum-likelihood fittingto model-generated target distributions, we place all of these stages under a single likelihood-based training algorithm,unified training. This perspective turns the foundation-model training lifecycle into a problem of choosing which data to train on, in what proportions, and when, with conventional staged training as a special case. We instantiate this framework in aunified trainingsystem that mixes static data and RL generations within a single trainer. Across controlled and real-world experiments,unified trainingimproves the accuracy-compute tradeoff and task coverage while preserving broader capabilities, and provides a stronger starting point forcontinual learning. This work lays the groundwork for a unified science of foundation-model training, connecting mathematical principles, training infrastructure, and the development of model capabilities.
View arXiv pageView PDFProject pageGitHub5Add to collection
Get this paper in your agent:
hf papers read 2610\.40564
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2610.40564 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2610.40564 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2610.40564 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
@rohanpaul_ai: The separate stages of AI training can collapse into a single method. New Carnegie Mellon paper shows that pretraining,…
Carnegie Mellon paper shows that pretraining, fine-tuning, and RL can collapse into a single training rule (MaxRL as likelihood training on correct answers), with blended data runs outperforming staged training on reasoning and coding benchmarks while preserving general skills.
RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training
Harvard researchers challenge the standard LLM training pipeline by showing RL can be effectively applied during pre-training rather than only after SFT, finding that data composition matters more than model scale, and proposing parallel averaging of RL and SFT objectives that outperforms sequential approaches while preserving general capabilities.
Unsupervised Post-Training of Foundation Models: A Survey
This survey paper categorizes 80 unsupervised post-training methods for foundation models, organizing them by the internal update signals and presenting a unified framework for selection and evaluation.
Training Fair Tabular Foundation Models
This paper proposes FairTFM, a training strategy to incorporate fairness constraints into tabular foundation models, enabling fair predictions via in-context learning without task-specific retraining.
When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoff
This paper investigates the loss of model plasticity after excessive supervised fine-tuning (SFT) in the SFT-then-RL pipeline for LLMs, and proposes Rejuvenation, a method that restores plasticity via base-anchored model fusion and targeted neuron reset, consistently improving RL performance.