On-Policy Self-Distillation without Any Supervision
Summary
Introduces U-OPSD, an unsupervised on-policy self-distillation method that uses internal consistency and majority-vote pseudo-solutions to improve LLMs without external supervision, matching or exceeding supervised methods on math benchmarks.
View Cached Full Text
Cached at: 08/12/26, 08:22 AM
Paper page - On-Policy Self-Distillation without Any Supervision
Source: https://huggingface.co/papers/2608.06296
Abstract
Unsupervised on-policy self-distillation improves large language models by using internal consistency and majority-vote pseudo-solutions to correct confident errors without external supervision.
On-policy (Self-)Distillation (OPD / OPSD) has shown strong potential for post-training large language models (LLMs). However, existing methods still rely heavily on external supervision, including ground-truth signals, environmental feedback, or guidance from larger models, and therefore fall short of genuine “self”-distillation. In this study, we show thaton-policy self-distillationcan be achieved using only a model’s own generations via internal consistency. We proposeunsupervised on-policy self-distillation(U-OPSD).U-OPSDfirst samples multiple rollouts and constructs a pseudo solution bymajority voteunder aself-consistencythreshold. It then conditions the model’s distribution on thepseudo-solutionand distills itself on the disagreeing completions, allowing the model to correct itself precisely where it is confidently wrong. Across diverse benchmarks, base models, and training settings,U-OPSDconsistently improves over the base models and matches or surpasses supervised methods with ground truth (GT) such as OPSD andGRPO. On five mathematicalreasoning benchmarks, i.e., AIME24, AIME25, HMMT25, MATH500, and AMC23,U-OPSDimproves over the base model by 8.5% and 10.7% on Qwen3 non-thinking mode at 4B and 8B scales, and outperforms OPSD by 3.2% and 2.3% on average, respectively. In thinking mode,U-OPSDstays on par with OPSD, ahead by 0.9% at 4B and level at 8B and surpassingGRPOby 0.7% and 1.1%, respectively. Code is available at [https://github.com/williamium3000/u-opsd](https://github.com/williamium3000/u-opsd).
View arXiv pageView PDFProject pageAdd to collection
Get this paper in your agent:
hf papers read 2608\.06296
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.06296 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.06296 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.06296 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Learning from the Self-future: On-policy Self-distillation for dLLMs
Introduces d-OPSD, the first on-policy self-distillation framework for diffusion large language models, using suffix conditioning and step-level supervision to outperform RLVR and SFT baselines on reasoning benchmarks.
β-OPSD: Deriving with Policy Optimization, Training with Self-Distillation
This paper introduces β-OPSD, a generalization of on-policy self-distillation that frames it as a policy optimization family with a controllable KL penalty. The method uses distillation to approximate expensive policy optimization, improving stability and reasoning performance on mathematical benchmarks.
Reducing the Safety Tax in LLM Safety Alignment with On-Policy Self-Distillation
This paper introduces OPSA, an on-policy self-distillation method for LLM safety alignment that reduces the safety tax by training on the model's own rollouts and using a teacher flip rate to activate latent safety reasoning, achieving stronger safety-reasoning tradeoffs across multiple model scales.
dOPSD: On-Policy Self-Distillation for Diffusion Language Models
This paper introduces dOPSD, an on-policy self-distillation method for diffusion language models that leverages internal denoising trajectories to improve mathematical reasoning and code generation.
Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoning
The paper proposes EGRSD and CL-EGRSD, on-policy self-distillation methods that weight token-level supervision by teacher entropy to improve reasoning accuracy-length tradeoff in LLMs, evaluated on Qwen3-4B and Qwen3-8B.