Self-Distilled Policy Gradient

Hugging Face Daily Papers Papers
self-distillation policy-gradient reinforcement-learning kl-regularization verifier-advantages

Summary

This paper proposes SDPG, a self-distilled policy-gradient framework that combines on-policy self-distillation with verifier advantages and KL regularization to improve reinforcement learning stability and performance.

On-policy self-distillation, where a language model conditions on privileged context to supervise its own generations, is a promising source of dense supervision for sparse-reward reinforcement learning. Actually, it can be instantiated as an auxiliary full-vocabulary student-to-teacher reverse Kullback-Leibler divergence loss. We therefore propose SDPG, a self-distilled policy-gradient framework that combines group-relative verifier advantages with normalized standard deviation, exact full-vocabulary on-policy self-distillation, as well as reference-policy KL regularization. Empirically, SDPG improves stability and performance over RLVR and self-distillation baselines. The code is available at https://github.com/lauyikfung/SDPG.
Original Article
View Cached Full Text

Cached at: 06/04/26, 03:41 AM

Paper page - Self-Distilled Policy Gradient

Source: https://huggingface.co/papers/2606.04036

Abstract

A self-distilled policy-gradient framework combines on-policy self-distillation with verifier advantages and KL regularization to improve reinforcement learning stability and performance.

On-policyself-distillation, where a language model conditions on privileged context to supervise its own generations, is a promising source of dense supervision for sparse-rewardreinforcement learning. Actually, it can be instantiated as an auxiliary full-vocabulary student-to-teacherreverse Kullback-Leibler divergenceloss. We therefore propose SDPG, a self-distilledpolicy-gradientframework that combines group-relativeverifier advantageswith normalized standard deviation, exact full-vocabulary on-policyself-distillation, as well as reference-policyKL regularization. Empirically, SDPG improves stability and performance over RLVR andself-distillationbaselines. The code is available at https://github.com/lauyikfung/SDPG.

View arXiv pageView PDFProject pageGitHub0Add to collection

Get this paper in your agent:

hf papers read 2606\.04036

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.04036 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2606.04036 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.04036 in a Space README.md to link it from this page.

Collections including this paper1

Similar Articles

Self-Distilled Policy Gradient

arXiv cs.LG

SDPG (Self-Distilled Policy Gradient) is a new RL training framework for LLMs that combines group-relative verifier advantages with on-policy self-distillation and KL regularization to address sparse rewards and instability in RLVR training. The method uses a shared model as both student and teacher by conditioning on privileged context, showing improved stability and performance over RLVR and self-distillation baselines.

β-OPSD: Deriving with Policy Optimization, Training with Self-Distillation

Hugging Face Daily Papers

This paper introduces β-OPSD, a generalization of on-policy self-distillation that frames it as a policy optimization family with a controllable KL penalty. The method uses distillation to approximate expensive policy optimization, improving stability and reasoning performance on mathematical benchmarks.

I-SDPO: Instance-Level Adaptive Self-Distillation Policy Optimization

arXiv cs.LG

Introduces I-SDPO, an instance-level adaptive self-distillation policy optimization method that switches between GRPO and privileged self-distillation based on whether a rollout group contains any successful responses, improving reasoning performance on SciKnowEval.

Self-Distillation Enables Continual Learning [pdf]

Hacker News Top

Introduces Self-Distillation Fine-Tuning (SDFT), a method that enables on-policy learning from demonstrations to achieve continual learning without catastrophic forgetting, outperforming supervised fine-tuning.