@michalwols: Amazing talk on on policy self distillation on @yacinelearning 's channel

Amazing talk on on policy self distillation on @yacinelearning ’s channel https://t.co/mCCxcZsO6O

TL;DR: This video introduces a post-training method called “self-distillation” that combines the model’s own rollouts with rich environmental feedback (e.g., compiler errors) to generate dense token-level learning signals, significantly improving efficiency. SDPO achieves GRPO-level accuracy 6x faster with reasoning trajectories up to 11x shorter, while SDFT enables a single model to sequentially learn multiple skills without forgetting.

Background: The Signal Sparsity Problem in Reinforcement Learning

Currently, most post-training of large language models is done via reinforcement learning methods similar to GRPO. The workflow is: pose a question, let the model generate multiple rollouts, then use a verifier to score each rollout—usually just a binary correct/incorrect judgment. Weights are then updated based on which attempts were good or bad. The issue is that this reward signal is extremely sparse: each complete rollout gets only a single score. The model must figure out for itself which tokens in a thousand-token chain of thought actually matter—a notoriously difficult credit assignment problem.

Self-Distillation: Bypassing the Sparse Reward Bottleneck

This video explores a class of methods that completely bypass this bottleneck using “self-distillation.” The core idea: in many environments, there already exists rich textual feedback—compiler errors, runtime exceptions, judge evaluations, etc. This information explains why an attempt failed, yet current RL methods don’t use it at all—in extreme cases, they compress everything into a binary reward.

Self-distillation works by taking the model’s own rollout and feeding it back into the same model, but now conditioned on whatever feedback is available from the environment. It makes the model re-evaluate its own tokens. In this distillation setup, the teacher and student are actually the same model, but the teacher sees more context. This produces dense token-level learning signals while being cheap—it only requires one forward pass on existing rollouts.

The method was independently developed by two collaborating teams:

• Jonas Kubatier’s group at ETH Zurich, who developed SDPO for the reinforcement learning setting.
• Eden Shenfield (should be spelled Idan Shenfeld) at MIT, who developed SDFT for continual learning (learning from demonstrations).

Both papers were released simultaneously in January 2026; in fact, they had met earlier in 2025 to discuss the idea.

Simplicity and Effectiveness of the Method

The paradigm’s core lies in its simplicity: using the model’s own in-context learning ability to bootstrap its own learning. The results are impressive:

• SDPO: achieves GRPO-level accuracy up to six times faster in wall-clock time, and produces reasoning trajectories up to 11 times shorter.
• SDFT: lets a single model sequentially learn multiple skills without forgetting previous ones, whereas standard SFT fails completely.

These methods are already being adopted in production. For example, the Open Claw RL system and the cutting-edge open-source model GLM 5 both use similar approaches in their post-training pipelines.

Guest Introductions and Discussion

Idan Shenfeld’s Background and Perspective

Idan is a PhD student at MIT, advised by Pulkit Agrawal, researching reinforcement learning algorithms (primarily for LLMs, but also for robotics). Before his PhD, he studied RL as an undergraduate, worked on autonomous driving (participating in GM’s large-scale autonomous driving project), and interned at DeepMind (part of the post-training team in Summer 2024). He shared his mindset on the difference between product work and research:

“When you’re working on an actual product or application, you really care about performance. Understanding is just a tool to get the right performance. Sometimes, even though understanding is interesting and research is fun, just going through a dataset and cleaning it can give you much bigger performance gains than anything else. So you need to restrain yourself from focusing too much on cool, interesting, or novel things, and first do the basics. Then you invest your energy… As researchers, we don’t care about the specific application; what we bring to the world is new understanding, new knowledge. That requires a completely different perspective. I don’t care about being state-of-the-art on some benchmark… The key is what new understanding, new knowledge, new perspective you bring to the community.”

Yonas Sebuter’s Background and Motivation

Yonas is pursuing his PhD at ETH Zurich, advised by Andreas Krause. His research mainly focuses on “test-time training”—once a model is given a task, it can continue training during inference. The main motivation is to deploy models into new environments at inference time, let them continue learning and exploring in that environment, and eventually solve difficult tasks that were initially completely outside the model’s distribution. His previous work centered on how to explore effectively in new environments.

From Test-Time Learning to Self-Distillation

Yonas explained how SDPO is a natural extension of his prior work: when you want to continue training and improving a language model via gradient descent at test time (with limited compute), two aspects become critical: 1) how to efficiently convert data into gradient updates (compute-constrained, must maximize signal); 2) you generally need better signals, extracting as much as possible from the environment. Idan added that the intuition comes from coding scenarios: after getting ChatGPT to generate code, you paste the error back and ask it to improve—it’s already very good at understanding where its own error comes from. Self-distillation formalizes this process.

On the “RL Razor” and Forgetting

Idan mentioned how his paper “RL Razor: Why Online Reinforcement Learning Forgets Less” influenced him: online methods tend to forget less than offline methods (like SFT fine-tuning). But that doesn’t justify “just use RL and throw away all other learning signals.” Self-distillation provides an online algorithm that can learn from very diverse signals.

Coincidental Independent Discovery

The two met at NeurIPS 2024, showed each other what they were working on—Yonas presented slides on SDPO, Idan opened his Overleaf draft—and realized they had hit upon the same core algorithmic innovation from different angles: Yonas focused on textual feedback (SDPO), Idan on learning from expert demonstrations (SDFT). They decided to collaborate. This coincidence is exciting because such convergence of ideas often heralds the birth of a learning paradigm capable of learning from extremely rich and diverse data.

Summary and Future

The simplicity and effectiveness of the self-distillation paradigm have been independently validated by multiple teams and are already being deployed in real-world systems. It addresses the signal sparsity problem in RL post-training while supporting continual learning, offering a new direction for language models to improve themselves.

Source: YouTube video link (https://www.youtube.com/watch?v=OgEGV7apEzI)