Denser neq Better: Limits of On-Policy Self-Distillation for Continual Post-Training

Hugging Face Daily Papers Papers
self-distillation continual-learning post-training on-policy foundation-models

Summary

This paper investigates on-policy self-distillation in continual post-training, finding it accelerates in-domain specialization but fails to prevent forgetting and can collapse in out-of-distribution scenarios, suggesting that on-policy data alone is insufficient for continual learning.

Continual post-training enables foundation models to acquire new knowledge while preserving existing capabilities. Recent work suggests that on-policy learning can mitigate forgetting, with on-policy self-distillation emerging as a particularly attractive approach. In this work, we revisit this optimistic view through self-distillation policy optimization (SDPO). Our experiments show that SDPO can accelerate in-domain specialization when teacher signals are stable and well aligned, but it struggles to generalize to out-of-distribution scenarios. In continual post-training, SDPO exhibits stronger forgetting and can even collapse, whereas on-policy reinforcement learning methods such as GRPO adapt more conservatively and better preserve prior capabilities. Further analyses reveal that denser self-distillation induces larger drift in both parameter space and response space, and can amplify high-frequency formatting artifacts through a self-reinforcing teacher--student loop. These findings suggest that on-policy data alone is insufficient for continual learning. Dense self-distillation can accelerate specialization when teacher targets are stable and token-level supervision is reliable, but it should not be treated as a default stabilizer for continual post-training. Our code is available at https://github.com/Moenupa/SDPO-CL.
Original Article
View Cached Full Text

Cached at: 07/03/26, 03:52 AM

Paper page - Denser neq Better: Limits of On-Policy Self-Distillation for Continual Post-Training

Source: https://huggingface.co/papers/2607.01763

Abstract

On-policy self-distillation in continual post-training accelerates in-domain specialization but fails to prevent forgetting and can collapse in out-of-distribution scenarios, indicating that on-policy data alone is insufficient for continual learning.

Continual post-trainingenables foundation models to acquire new knowledge while preserving existing capabilities. Recent work suggests thaton-policy learningcan mitigate forgetting, withon-policy self-distillationemerging as a particularly attractive approach. In this work, we revisit this optimistic view throughself-distillationpolicy optimization(SDPO). Our experiments show that SDPO can accelerate in-domain specialization when teacher signals are stable and well aligned, but it struggles to generalize to out-of-distribution scenarios. Incontinual post-training, SDPO exhibits stronger forgetting and can even collapse, whereas on-policyreinforcement learningmethods such asGRPOadapt more conservatively and better preserve prior capabilities. Further analyses reveal that denserself-distillationinduces larger drift in bothparameter spaceandresponse space, and can amplify high-frequency formatting artifacts through a self-reinforcing teacher--student loop. These findings suggest that on-policy data alone is insufficient forcontinual learning. Denseself-distillationcan accelerate specialization when teacher targets are stable and token-level supervision is reliable, but it should not be treated as a default stabilizer forcontinual post-training. Our code is available at https://github.com/Moenupa/SDPO-CL.

View arXiv pageView PDFProject pageGitHub0Add to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.01763 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.01763 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.01763 in a Space README.md to link it from this page.

Collections including this paper3

Similar Articles

Self-Distillation Enables Continual Learning [pdf]

Hacker News Top

Introduces Self-Distillation Fine-Tuning (SDFT), a method that enables on-policy learning from demonstrations to achieve continual learning without catastrophic forgetting, outperforming supervised fine-tuning.

Gains and Collapse in On-Policy Distillation:A Reinforcement Learning Perspective

Hugging Face Daily Papers

This paper analyzes on-policy distillation (OPD) for language model post-training through a reinforcement learning lens, explaining why OPD both improves performance and can collapse into repetitive generation. It finds that masking unhealthy responses during training and SFT initialization can mitigate collapse, shifting focus from teacher generation quality to its ability to evaluate student rollouts.

Adaptive Supervised Anchoring for On-Policy Self-Distillation

arXiv cs.LG

This paper proposes an adaptive supervised anchoring framework for on-policy self-distillation, addressing the problem of rollout-conditioned signal degradation in language model training. The method separates rollout-conditioned distribution matching from canonical-context supervision, improving task acquisition while preserving general capabilities.

On-Policy Distillation (5 minute read)

TLDR AI

This paper introduces on-policy distillation, which trains a student model on its own trajectories with teacher token-level KL supervision to fix train-inference mismatch, unifying forward-KL, reverse-KL, and JSD losses, with reverse-KL favored for smaller students.