@NielsRogge: What is mid-training? The stage between pre-training and post-training A base model is continued on a smaller, curated …
Summary
Explains mid-training as a stage between pre-training and post-training, where a base model is continued on curated data to strengthen specific capabilities before instruction tuning.
View Cached Full Text
Cached at: 06/02/26, 03:43 PM
What is mid-training?
The stage between pre-training and post-training
A base model is continued on a smaller, curated data mixture chosen to strengthen capabilities that the original pre-training run undercovered, such as multilinguality, domain knowledge, or long-context extension.
It usually keeps a pre-training-like objective, but uses higher-quality or more targeted data so later instruction tuning, preference tuning, or RL can shape behavior on top of stronger capabilities.
Learn more here: https://paperswithcode.co/methods/mid-training…
Similar Articles
Anthropic researchers detail “model spec midtraining”, which adds a stage between pretraining and fine-tuning to improve generalization from alignment training
Anthropic researchers introduce Model Spec Midtraining (MSM), a new training stage between pretraining and fine-tuning designed to improve how models generalize from alignment training and reduce agentic misalignment.
UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing
UltraX proposes a function-calling refinement framework for large-scale pre-training data that introduces insertion alongside deletion and modification, enabling fine-grained instance-level editing. It builds a reliable program-supervision generation pipeline and demonstrates improved data efficiency and model performance when pretraining 1B models from scratch.
@anyscalecompute: LLM post-training is the new baseline. Picking the wrong method or GPU config is how you waste a 36-hour run. Introduci…
Anyscale introduces a new Agent Skill for LLM post-training that automatically selects the optimal fine-tuning method (SFT, DPO, GRPO, etc.) and generates ready-to-launch configs, helping avoid wasted GPU runs.
How Post-Training Shapes Biological Reasoning Models
This paper investigates how post-training stages such as continued pre-training, supervised fine-tuning, and reinforcement learning affect generalization in biological reasoning models, finding that these stages have distinct impacts on in-domain and out-of-domain performance.
Early Data Exposure Improves Robustness to Subsequent Fine-Tuning
This paper shows that mixing post-training data into pretraining (early exposure) improves how robustly a model retains capabilities after subsequent fine-tuning, challenging the notion that immediate post-training performance predicts retention. Controlled experiments on 135M and 1B models demonstrate that early exposure consistently improves the trade-off between upstream retention and downstream performance.