@BhavinJawade: 𝗢𝗻-𝗽𝗼𝗹𝗶𝗰𝘆 𝗱𝗶𝘀𝘁𝗶𝗹𝗹𝗮𝘁𝗶𝗼𝗻 𝗶𝘀𝗻'𝘁 𝗮 𝗳𝗿𝗲𝗲-𝗹𝘂𝗻𝗰𝗵 On-policy distillation has become a default…

On-policy distillation isn’t a free-lunch

On-policy distillation has become a default post-training tool in many open-source frontier model training recipes. Recent releases lean on it heavily: DeepSeek v4, MiMO, and Nemotron-Cascade-2 use MOPD, and GLM 5.x uses on-policy cross-stage self-distillation. It provides RL’s on-policy nature reducing exposure bias, while providing token level supervision like SFT. But OPD and OPSD have their own failure modes. In this post I discuss a few of them:

1. Early mistakes are structurally uncorrectable. When the student samples a rollout and takes an early wrong turn, the per-token KL computed along that frozen rollout cannot pull it back onto a correct path. TRD proves that this failure is built into the objective rather than being a matter of noisy gradients. Even with a perfect teacher, the gradient obtained from token-level KL on the student’s own rollout agrees with the ideal corrective gradient at exactly one point, the token where the student first diverged, and disagrees everywhere after it. Every later supervision target is therefore anchored to a context that the student should never have entered. Because reweighting or clipping only rescales the magnitude of each token’s gradient, and here the terms point in the wrong direction, no per-token adjustment can recover the correct update. TRD’s proposed fix is to distill along a teacher-refined trajectory rather than the raw student rollout, which restores a target the student can actually follow.

2. A stronger teacher can be a worse teacher. On-policy distillation can only teach the student at states the student itself visits, and the usable signal at each of those states lives in the overlap between the student’s and teacher’s next-token distributions. Rethinking OPD shows that a bigger, higher-scoring teacher can fail to move a student while a weaker one succeeds, because if the teacher’s token distribution places its mass on tokens the student rarely produces, the overlap is small and almost nothing transfers, no matter how capable the teacher is in absolute terms. What actually predicts success is early top-k thinking-pattern overlap. In runs that work, the shared top-k tokens carry 97 to 99% of the probability mass and the overlap ratio climbs steadily during training, whereas a run that starts with low overlap never recovers it. A teacher trained on the same recipe as the student also converges toward the student’s own distribution, so its higher benchmark score does not correspond to any new knowledge it can transfer. The practical rule is to pick teachers by distributional closeness to the student, not by leaderboard rank.

3. Privileged-information-conditioned OPSD can fail to transfer. In OPSD you distill a teacher that was conditioned on privileged information, such as the gold answer, into a student that will never have it. The Many Faces of OPD shows what goes wrong when that information is instance-specific. The student cannot recover the teacher’s per-instance reasoning, since it never sees the answer, so it instead learns a single answer-free policy that effectively averages the teacher’s behavior across all problems, and that averaged policy is too generic to solve any particular one. The signature is initial gains followed by collapse: rollouts grow long, fill with hedging tokens, and accuracy craters toward zero. The approach works only when the privileged information is a shared rule that applies across all instances, such as a system prompt or an alignment preference, and not when it is a per-problem answer.

4. Thinking collapse: dense supervision suppresses the model’s own deliberation. A teacher conditioned on the answer has no reason to hesitate, backtrack, or explore, so its per-token targets quietly push down the student’s deliberation tokens. Diagnosing and Mitigating Thinking Collapse names this phenomenon thinking collapse: over training, the student’s native reasoning behavior erodes as the exploratory tokens that carry it, words like wait, maybe, and alternatively, become progressively less frequent. The mechanism is local rather than global. The damage concentrates at high-entropy decision forks, the branch points where the student is genuinely uncertain and would normally deliberate. Exactly there, the student’s top-1 token is often an exploratory marker while the answer-conditioned teacher’s top-1 token is not, so the mismatch produces a strong gradient that suppresses the very tokens that make reasoning work. The result is a model whose native reasoning behavior is measurably suppressed, and downstream reasoning accuracy falls in step with it.

Trajectory-Refined Distillation (TRD) https://arxiv.org/abs/2606.08432 Rethinking On-Policy Distillation of Large Language Models: https://arxiv.org/abs/2604.13016 The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes https://arxiv.org/abs/2605.11182 Diagnosing and Mitigating Thinking Collapse in On-Policy Self-Distillation https://arxiv.org/abs/2607.10805

When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning (http://arxiv.org/abs/2605.21606) shows that OPSD’s standard objective weights all tokens equally, treating the privileged teacher’s target as equally reliable at every student-visited prefix. But high teacher entropy is ambiguous, it can signal a non-viable branch or just benign solution diversity.

I think the method in the thinking collapse paper (https://arxiv.org/pdf/2607.10805) to fix this is quite interesting, probably not the cleanest, but the idea is neat. the intuition is not to trust the privileged self teacher everywhere. The collapse only happens at high-entropy decision forks, the spots where the student wants to emit a deliberation token (wait, maybe, alternatively) but the teacher, already knowing the answer, wants to push it down. In the paper they find that these spots are the tokens inside the high-entropy region where the student assigns higher probability to its own sampled token than the teacher does. Only at those tokens, they soft anchor the target back towards the frozen base model (original base student policy) instead of the teacher. Everywhere else, teachers supervision is used without this regularization.