@SergioPaniego: Frontier models use distillation as a step of their post-training pipelines! In 2026 it has three jobs: - compress a bi…
Summary
Explains three uses of distillation in frontier model post-training pipelines: compressing large models into small ones, merging RL experts, and self-teaching. Includes a write-up detailing which models use each method.
View Cached Full Text
Cached at: 07/09/26, 07:59 AM
Frontier models use distillation as a step of their post-training pipelines!
In 2026 it has three jobs:
- compress a big model into a small one
- merge RL experts into a single model
- let a model teach itself
I wrote up which models use each one and how
Similar Articles
@SergioPaniego: https://x.com/SergioPaniego/status/2074863503312044499
An article surveying how frontier AI models in 2026 use distillation techniques, covering off-policy, on-policy, and self-distillation stages, with examples from Gemma, DeepSeek, GLM, Nemotron, and Qwen3.
@SergioPaniego: before model distillation was an attack vector. it was. pretty handy way of improving model performance on a task you c…
The article provides a brief history of model distillation in AI and announces an upcoming live stream class on distilling open models using TRL (Transformer Reinforcement Learning).
@dlwh: Frontier AI models are built from thousands of small decisions: data sourcing, filtering, mixtures, curricula, scaling …
Frontier AI models are built from thousands of small decisions, emphasizing the importance of process knowledge in their development.
@ben_burtenshaw: before model distillation was an attack vector. it was. pretty handy way of improving model performance on a task you c…
Ben Burtenshaw announces a live stream on July 7th covering knowledge distillation in post-training, showing how to implement it using small models to approach large model performance.
@SergioPaniego: frontier agents are this good partly because the model was trained inside the very harness it ships with great to see t…
Sergio Paniego highlights that frontier agents' performance is due to models being trained inside their deployment harness. The new work 'Polar: Agentic RL on Any Harness at Scale' by NVIDIA AI enables turning harnesses like Codex, Claude Code, Qwen Code, or Pi into RL training environments without modifying their internals.