Self-Distilled Policy Gradient

# Self-Distilled Policy Gradient
Source: [https://arxiv.org/html/2606.04036](https://arxiv.org/html/2606.04036)
Yifeng Liu Shiyuan Zhang11footnotemark:1Yifan Zhang11footnotemark:1Quanquan Gu22footnotemark:2Equal contributionDepartment of Computer Science, University of California, Los Angeles, CA, USA; email: liuyifeng@cs\.ucla\.eduDepartment of Computer Science, University of California, Los Angeles, CA, USA; email: zsy25ucla@ucla\.eduPrinceton AI Laboratory, Princeton University, Princeton, NJ, USA; email: yifzhang@princeton\.eduCorresponding Author, Department of Computer Science, University of California, Los Angeles, CA, USA; email: qgu@cs\.ucla\.edu

###### Abstract

On\-policy self\-distillation, where a language model conditions on privileged context to supervise its own generations, is a promising source of dense supervision for sparse\-reward reinforcement learning\. Actually, it can be instantiated as an auxiliary full\-vocabulary student\-to\-teacher reverse Kullback\-Leibler divergence loss\. We therefore propose SDPG, a self\-distilled policy\-gradient framework that combines group\-relative verifier advantages with normalized standard deviation, exact full\-vocabulary on\-policy self\-distillation, as well as reference\-policy KL regularization\. Empirically, SDPG improves stability and performance over RLVR and self\-distillation baselines\. The code is available at[https://github\.com/lauyikfung/SDPG](https://github.com/lauyikfung/SDPG)\.

PromptxxPrivileged contextcc𝝅𝜽\\bm\{\\pi\_\{\\theta\}\}shared modelStudentpt=πθ\(⋅∣x,y<t\)p\_\{t\}\\;=\\;\\pi\_\{\\theta\}\(\\,\\cdot\\mid x,\\,y\_\{<t\}\\,\)Teacherqt=πθ\(⋅∣c,x,y<t\)q\_\{t\}\\;=\\;\\pi\_\{\\theta\}\(\\,\\cdot\\mid c,\\,x,\\,y\_\{<t\}\\,\)Rollouts\{y\(i\)\}i=1G∼pt\\\{y^\{\(i\)\}\\\}\_\{i=1\}^\{G\}\\sim p\_\{t\}VerifierR​\(x,y\(i\)\)R\(x,y^\{\(i\)\}\)Aout\(i\)=R\(i\)−μGσG\+ϵstdA^\{\(i\)\}\_\{\\text\{out\}\}\\;=\\;\\dfrac\{R^\{\(i\)\}\-\\mu\_\{G\}\}\{\\sigma\_\{G\}\+\\epsilon\_\{\\mathrm\{std\}\}\}Full\-Vocab OPD KLℓtOPD=DKL​\(pt∥SG​\[qt\]\)\\ell^\{\\text\{OPD\}\}\_\{t\}\\;=\\;D\_\{\\text\{KL\}\}\\\!\\bigl\(p\_\{t\}\\,\\big\\\|\\,\\mathrm\{SG\}\[\\,q\_\{t\}\\,\]\\bigr\)gatemi=1​\[Aout\(i\)\>0\]m\_\{i\}\\;=\\;\\mathbf\{1\}\\\!\\bigl\[A^\{\(i\)\}\_\{\\text\{out\}\}\>0\\bigr\]ℒout\\mathcal\{L\}\_\{\\text\{out\}\}on\-policy policy gradientβ​\(k\)​ℒOPD\+\\beta\(k\)\\,\\mathcal\{L\}^\{\+\}\_\{\\text\{OPD\}\}gated \+ scheduledα​ℒ𝒦​\(πθ,πref\)\\alpha\\,\\mathcal\{L\}\_\{\\mathcal\{K\}\}\(\\pi\_\{\\theta\},\\pi\_\{\\text\{ref\}\}\)reference KL regularization \(UFKL/URKL\)πref\\pi\_\{\\text\{ref\}\}fixed𝓛SDPG=ℒout\+β​\(k\)​ℒOPD\+\+α​ℒ𝒦​\(πθ,πref\)\\bm\{\\mathcal\{L\}\_\{\\text\{SDPG\}\}\}\\;=\\;\\mathcal\{L\}\_\{\\text\{out\}\}\\;\+\\;\\beta\(k\)\\,\\mathcal\{L\}^\{\+\}\_\{\\text\{OPD\}\}\\;\+\\;\\alpha\\,\\mathcal\{L\}\_\{\\mathcal\{K\}\}\\bigl\(\\pi\_\{\\theta\},\\pi\_\{\\text\{ref\}\}\\bigr\)withoutccwithccmodulatesinputspolicydistributionssignalslossesobjective

Figure 1:Overview of the Self\-Distilled Policy Gradient \(SDPG\) objective, combining rollout\-based outcome optimization, gated On\-Policy Distillation \(OPD\) from privileged context, and a KL regularization to a fixed reference policy\. Note that OPD is also a form of policy gradient\.## 1Introduction

With the development of Reinforcement Learning with Verifiable Rewards \(RLVR\), Large Language Models \(LLMs\) have demonstrated remarkable capabilities in complex reasoning tasks such as mathematics and code generation\. Algorithms in this family, such as Group Relative Policy Optimization \(GRPO\)\(Shaoet al\.,[2024](https://arxiv.org/html/2606.04036#bib.bib4)\), optimize against rule\-based outcome rewards and have become the standard recipe for post\-training reasoning models, eliminating the cost and bias of human preference annotation\.

Despite its success, RLVR encounters several limitations, including sparse sequence\-level reward across tokens and instability under negative advantages during the early stages of training\. Although recent works such as Dr\.GRPO\(Liuet al\.,[2025](https://arxiv.org/html/2606.04036#bib.bib30)\), DAPO\(Yuet al\.,[2025](https://arxiv.org/html/2606.04036#bib.bib12)\), and GSPO\(Zhenget al\.,[2025](https://arxiv.org/html/2606.04036#bib.bib31)\)address the latter through asymmetric dual\-clip thresholds and sequence\-level advantage, the sparsity issue remains unresolved\.

Recently, on\-policy distillation \(OPD\) approaches have been proposed to yield dense token\-level signals\(Agarwalet al\.,[2024](https://arxiv.org/html/2606.04036#bib.bib44); Lu and Lab,[2025](https://arxiv.org/html/2606.04036#bib.bib47); Fuet al\.,[2026](https://arxiv.org/html/2606.04036#bib.bib50)\)\. Such methods maintain two models: a student model to be optimized that rolls out trajectories, and a teacher model that produces token\-level guidance via Kullback–Leibler divergence \(KL\) regularization or related objectives\(Guet al\.,[2024](https://arxiv.org/html/2606.04036#bib.bib46); Xuet al\.,[2025](https://arxiv.org/html/2606.04036#bib.bib45); Yanget al\.,[2025](https://arxiv.org/html/2606.04036#bib.bib49)\)\. However, traditional distillation approaches use a much larger and stronger teacher, which imposes a considerable memory burden when optimizing student models\. Moreover, heterogeneous teacher signals may hurt the smoothness of the training process\.

A recent line of work addresses such limitations through on\-policy self\-distillation\. In these methods, the teacher model is exactly the same model as the student model, but with additional knowledge such as demonstrations, direct answers, and reasoning paths\(Hübotteret al\.,[2026](https://arxiv.org/html/2606.04036#bib.bib18); Shenfeldet al\.,[2026](https://arxiv.org/html/2606.04036#bib.bib5); Penalozaet al\.,[2026](https://arxiv.org/html/2606.04036#bib.bib51)\)\. This converts sparse and inconsistent outcome rewards into dense, per\-token, and homogeneous supervision\. In detail, OPCD\(Yeet al\.,[2026](https://arxiv.org/html/2606.04036#bib.bib1)\)involves in\-context knowledge in the teacher model and internalizes it into the student model through KL divergence; OPSD\(Zhaoet al\.,[2026](https://arxiv.org/html/2606.04036#bib.bib17)\)applies a full, vocabulary\-wise KL divergence for better reasoning performance; and TRRD\(Zhanget al\.,[2026b](https://arxiv.org/html/2606.04036#bib.bib52)\)incorporates trust regions in distillation\.

However, the phrase “self\-distillation” can obscure a useful policy\-gradient interpretation\. For a fixed rollout prefix\(x,y<t\)\(x,y\_\{<t\}\), let

qt​\(a\)=πθ​\(a∣c,x,y<t\)andpt​\(a\)=πθ​\(a∣x,y<t\),q\_\{t\}\(a\)=\\pi\_\{\\theta\}\(a\\mid c,x,y\_\{<t\}\)\\quad\\text\{and\}\\quad p\_\{t\}\(a\)=\\pi\_\{\\theta\}\(a\\mid x,y\_\{<t\}\),whereqtq\_\{t\}is the privileged distribution induced by the same model under contextcc, andptp\_\{t\}is the deployable distribution without privileged context\. We use the full\-vocabulary reverse KLDKL​\(pt∥SG​\[qt\]\)D\_\{\\mathrm\{KL\}\}\(p\_\{t\}\\\|\\mathrm\{SG\}\[q\_\{t\}\]\)between these two distributions\. Conditioned on the sampled prefix and with the privileged branch detached, its student\-side gradient is locally identical to a detached\-sampling policy\-gradient update whose token advantage is a centered log teacher/student ratio\. This is a gradient identity, not a replacement of the full\-vocabulary KL objective\. It also suggests a failure mode of pure self\-distillation: without a verifier, a privileged but imperfect distribution can reinforce locally plausible tokens on globally wrong trajectories\. A natural remedy is to combine the advantages of both self\-distillation and RLVR methods by keeping the GRPO reward objective with self\-distillation signals\. RLSD\(Yanget al\.,[2026a](https://arxiv.org/html/2606.04036#bib.bib8)\)incorporates self\-distillation in the clipped importance ratio of GRPO loss function to moderate the teacher signal\. Nevertheless,\(Xiaoet al\.,[2026](https://arxiv.org/html/2606.04036#bib.bib48)\)still notices that strong teacher signals may hurt the potential of the student models\.

In order to address the above challenge, we proposeSDPG\(Self\-Distilled Policy Gradient\), which integrates exact full\-vocabulary privileged OPD into KL\-regularized policy optimization\. The resulting objective has two complementary sources of supervision: a sparse binary outcome signal from the verifier and a dense full\-vocabulary distillation signal from the context\-conditioned teacher\. Based on the RPG framework\(Zhanget al\.,[2026a](https://arxiv.org/html/2606.04036#bib.bib6)\), we focus on unnormalized reference\-policy KL regularizations in the main text, with normalized variants and all anchor derivations deferred to the appendix\. To control noise in the privileged teacher signal, we apply positive\-advantage gating and a warmup\-decay schedule to the distillation coefficient\. Our contributions are:

1. 1\.We derive an exact local policy\-gradient form of reverse\-KL full\-vocabulary OPD: for privileged teacherqtq\_\{t\}and deployable studentptp\_\{t\}, the student\-side gradient ofDKL​\(pt∥SG​\[qt\]\)D\_\{\\mathrm\{KL\}\}\(p\_\{t\}\\\|\\mathrm\{SG\}\[q\_\{t\}\]\)equals a detached\-sampling update with centered log\-ratio advantageSG​\[D¯t−log⁡\(p¯t​\(a\)/q¯t​\(a\)\)\]\\mathrm\{SG\}\[\\bar\{D\}\_\{t\}\-\\log\(\\bar\{p\}\_\{t\}\(a\)/\\bar\{q\}\_\{t\}\(a\)\)\], whereD¯t=DKL​\(p¯t∥q¯t\)\\bar\{D\}\_\{t\}=D\_\{\\mathrm\{KL\}\}\(\\bar\{p\}\_\{t\}\\\|\\bar\{q\}\_\{t\}\)\.
2. 2\.We propose a KL\-regularized policy optimization objective that combines binary outcome rewards with exact full\-vocabulary privileged distillation, and instantiate it with rollout\-policy\-sampled unnormalized forward and reverse KL regularizations to a fixed reference policy\.
3. 3\.We develop two stabilizers, positive\-advantage gating and a warmup\-decay schedule for the distillation coefficient, and show that SDPG improves over GRPO and self\-distillation baselines\.

## 2Background

Before introducing the SDPG framework, we briefly review GRPO, one of the standard paradigms in RLVR and introduce on\-policy self\-distillation\. After that, we formally define the divergence measures, including the unnormalized KL divergence, which serves as the theoretical foundation for our main KL regularizations\.

### 2\.1Group Relative Policy Optimization

In reasoning tasks, such as mathematics or coding, models are typically trained using verifiable outcome\-based rewards\. Given a promptxx, the model generates a sequence of tokensy=\{y1,y2,…,y\|y\|\}y=\\\{y\_\{1\},y\_\{2\},\\dots,y\_\{\|y\|\}\\\}\. A rule\-based verifier assigns a scalar rewardR​\(x,y\)R\(x,y\), e\.g\.,11for correct and0for incorrect\.

Group Relative Policy Optimization \(GRPO\)\(Shaoet al\.,[2024](https://arxiv.org/html/2606.04036#bib.bib4)\)is the current standard for RLVR\. For each promptxx, GRPO samples a group ofGGoutputs\{y\(1\),…,y\(G\)\}\\\{y^\{\(1\)\},\\dots,y^\{\(G\)\}\\\}from a frozen rollout policyπold\\pi\_\{\\mathrm\{old\}\}, usually the current policy snapshot at the beginning of the update\. Instead of training a separate value network, GRPO computes a sequence\-level advantageA\(i\)A^\{\(i\)\}by normalizing the rewards within the group:

A\(i\)=R​\(x,y\(i\)\)−μGσG\+εstd,\\displaystyle A^\{\(i\)\}=\\frac\{R\(x,y^\{\(i\)\}\)\-\\mu\_\{G\}\}\{\\sigma\_\{G\}\+\\varepsilon\_\{\\mathrm\{std\}\}\},\(2\.1\)whereμG\\mu\_\{G\}andσG\\sigma\_\{G\}are the mean and standard deviation of the group’s rewards, andεstd\>0\\varepsilon\_\{\\mathrm\{std\}\}\>0avoids division by zero\. Equivalently, implementations often setA\(i\)=0A^\{\(i\)\}=0when all rewards in the group are identical\. The policy is then optimized using a PPO\-style\(Schulmanet al\.,[2017](https://arxiv.org/html/2606.04036#bib.bib19)\)clipped surrogate objective:

ℒGRPO​\(θ\)=−𝔼x,\{y\(i\)\}i=1G∼πold\(⋅∣x\)​\[1∑i\|y\(i\)\|​∑i=1G∑t=1\|y\(i\)\|min⁡\(ri,t​A\(i\),clip​\(ri,t,1−ϵ,1\+ϵ\)​A\(i\)\)\],\\begin\{aligned\} \\mathcal\{L\}\_\{\\text\{GRPO\}\}\(\\theta\)=\-\\mathbb\{E\}\_\{x,\\;\\\{y^\{\(i\)\}\\\}\_\{i=1\}^\{G\}\\sim\\pi\_\{\\mathrm\{old\}\}\(\\cdot\\mid x\)\}\\bigg\[\\frac\{1\}\{\\sum\_\{i\}\|y^\{\(i\)\}\|\}\\sum\_\{i=1\}^\{G\}\\sum\_\{t=1\}^\{\|y^\{\(i\)\}\|\}\\min\\Bigl\(r\_\{i,t\}A^\{\(i\)\},\\;\\mathrm\{clip\}\(r\_\{i,t\},1\-\\epsilon,1\+\\epsilon\)A^\{\(i\)\}\\Bigr\)\\bigg\],\\end\{aligned\}

\(2\.2\)whereri,tr\_\{i,t\}is the importance ratio defined as follows:

ri,t=πθ​\(yt\(i\)∣x,y<t\(i\)\)πold​\(yt\(i\)∣x,y<t\(i\)\)r\_\{i,t\}=\\frac\{\\pi\_\{\\theta\}\(y^\{\(i\)\}\_\{t\}\\mid x,y^\{\(i\)\}\_\{<t\}\)\}\{\\pi\_\{\\text\{old\}\}\(y^\{\(i\)\}\_\{t\}\\mid x,y^\{\(i\)\}\_\{<t\}\)\}is the importance sampling ratio\. During the policy update,πold\\pi\_\{\\mathrm\{old\}\}is fixed and gradients are taken only throughπθ\\pi\_\{\\theta\}\.

While GRPO is computationally efficient, the scalar advantageA\(i\)A^\{\(i\)\}is applied equally to every token in the sequence\. This sparse, sequence\-level credit assignment makes it difficult for the model to identify which specific reasoning steps were correct or flawed, leading to inefficient exploration\. Furthermore, standard PPO clipping struggles with the overwhelming number of negative advantages \(A<0A<0\) early in training, which can degrade the model’s foundational language capabilities\.

### 2\.2On\-Policy Self\-Distillation

In standard off\-policy knowledge distillation, a smaller student model is trained to mimic the behavior of a more capable, distinct teacher model\. The conventional objective minimizes the divergence between the teacher’s and student’s output distributions over trajectories generated by the teacher\. While effective, this off\-policy paradigm suffers from exposure bias and distribution mismatch, as the student is trained on the teacher’s distribution but evaluated on its own rollouts during inference\.

On\-policy self\-distillation reduces distribution mismatch by evaluating the distillation loss on prefixes sampled from the student’s rollout distribution, usually a frozen snapshot of the current policy\. It also eliminates the need for an external teacher model: a single modelπθ\\pi\_\{\\theta\}acts as the deployable student when conditioned onxxand as the privileged teacher when additionally conditioned oncc, such as ground\-truth reasoning traces or verified reference answers\. At prefix\(x,y<t\)\(x,y\_\{<t\}\), the two distributions are

qt​\(a\)=πθ​\(a∣c,x,y<t\),pt​\(a\)=πθ​\(a∣x,y<t\)\.\\displaystyle q\_\{t\}\(a\)=\\pi\_\{\\theta\}\(a\\mid c,x,y\_\{<t\}\),\\qquad p\_\{t\}\(a\)=\\pi\_\{\\theta\}\(a\\mid x,y\_\{<t\}\)\.Prior work usually aligns these distributions through a full\-vocabulary KL divergence\. In this work we use the student\-to\-teacher reverse KLDKL​\(pt∥SG​\[qt\]\)D\_\{\\mathrm\{KL\}\}\(p\_\{t\}\\\|\\mathrm\{SG\}\[q\_\{t\}\]\)\. Proposition[3\.1](https://arxiv.org/html/2606.04036#S3.Thmtheorem1)shows that, on a fixed sampled prefix and with the teacher branch detached, its student\-side gradient has an equivalent local policy\-gradient form\.

### 2\.3Unnormalized KL Divergence

The standard Normalized Kullback\-Leibler \(KL\) divergence between two distributionsPPandQQis defined as:

DKL​\(P∥Q\)=∫P​\(x\)​log⁡P​\(x\)Q​\(x\)​d​x\.\\displaystyle D\_\{\\text\{KL\}\}\(P\\\|Q\)=\\int P\(x\)\\log\\frac\{P\(x\)\}\{Q\(x\)\}dx\.However, in the scenario when reference measures may not be perfectly normalized\(Zhanget al\.,[2026a](https://arxiv.org/html/2606.04036#bib.bib6)\)\(i\.e\.,∫xP​\(x\)\\int\_\{x\}P\(x\)and∫xQ​\(x\)\\int\_\{x\}Q\(x\)may be not equal to 1\), to yield a more elegant and symmetric gradient, we employ the Unnormalized KL \(UKL\) divergence\(Zhu and Rohwer,[1995](https://arxiv.org/html/2606.04036#bib.bib2); Minka,[2005](https://arxiv.org/html/2606.04036#bib.bib7)\)\. UKL introduces a mass correction term to handle scenarios where the distributions sum over sub\-vocabularies or are otherwise unnormalized\. It is defined as:

UKL​\(P∥Q\)=∫P​\(x\)​log⁡P​\(x\)Q​\(x\)​d​x⏟DKL​\(P∥Q\)\+∫\(Q​\(x\)−P​\(x\)\)​𝑑x⏟Mass Correction\.\\displaystyle\\text\{UKL\}\(P\\\|Q\)=\\underbrace\{\\int P\(x\)\\log\\frac\{P\(x\)\}\{Q\(x\)\}dx\}\_\{D\_\{\\text\{KL\}\}\(P\\\|Q\)\}\+\\underbrace\{\\int\\big\(Q\(x\)\-P\(x\)\\big\)dx\}\_\{\\text\{Mass Correction\}\}\.~\(2\.3\)Actually, it is equivalent tok3k\_\{3\}estimator\(Schulman,[2020](https://arxiv.org/html/2606.04036#bib.bib20)\), which is an unbiased estimator of KL divergence but with lower variance\. While we will employ the exact token\-level KL over the full vocabulary for the self\-distillation objective, we will use this unnormalized KL \(UKL\) framework for the policy regularization against the reference modelπref\\pi\_\{\\mathrm\{ref\}\}\.

## 3Self\-Distilled Policy Gradient

SDPG has three components\. First, it uses on\-policy objective as used in RLVR, with rewards supplied by a binary verifier\. Second, it adds exact full\-vocabulary OPD on prefixes sampled from the unprivileged rollout policy, so the privileged signal is dense but remains tied to the student’s own state distribution\. Third, it anchors the updated policy to a fixed reference policy through a rollout\-policy\-sampled KL surrogate\. Algorithm[1](https://arxiv.org/html/2606.04036#alg1)summarizes the training loop\.

Algorithm 1SDPG: Self\-Distilled Policy Gradient with Full\-Vocabulary OPDTraining data

𝒟=\{\(x,c\)\}\\mathcal\{D\}=\\\{\(x,c\)\\\}, where

xxis the input and

ccis privileged in\-context knowledge; language model

πθ\\pi\_\{\\theta\}; fixed reference policy

πref\\pi\_\{\\mathrm\{ref\}\}; total training steps

TT;

ϵstd\>0\\epsilon\_\{\\mathrm\{std\}\}\>0\.

foreach training step

k=1,…,Tk=1,\\ldots,Tdo

Sample a batch of prompts and privileged contexts

\{\(xj,cj\)\}j=1B∼𝒟\\\{\(x\_\{j\},c\_\{j\}\)\\\}\_\{j=1\}^\{B\}\\sim\\mathcal\{D\}
foreach prompt

xjx\_\{j\}do

// Rollout from the frozen unprivileged behavior policy

Sample a group of

GGresponses

\{yj\(i\)\}i=1G∼πθ\(⋅∣xj\)\\\{y\_\{j\}^\{\(i\)\}\\\}\_\{i=1\}^\{G\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\_\{j\}\)
// Compute outcome rewards and group\-relative advantages

Obtain binary verifier rewards

Rj\(i\)=R​\(xj,yj\(i\)\)R\_\{j\}^\{\(i\)\}=R\(x\_\{j\},y\_\{j\}^\{\(i\)\}\)
Compute

Aout\(i\)=Rj\(i\)−μjσj\+ϵstdA^\{\(i\)\}\_\{\\mathrm\{out\}\}=\\frac\{R\_\{j\}^\{\(i\)\}\-\\mu\_\{j\}\}\{\\sigma\_\{j\}\+\\epsilon\_\{\\mathrm\{std\}\}\}, where

μj\\mu\_\{j\}and

σj\\sigma\_\{j\}are the mean and standard deviation of

\{Rj\(i\)\}i=1G\\\{R\_\{j\}^\{\(i\)\}\\\}\_\{i=1\}^\{G\}
Set

mj\(i\)=𝟏​\[Aout\(i\)\>0\]m\_\{j\}^\{\(i\)\}=\\mathbf\{1\}\[A^\{\(i\)\}\_\{\\mathrm\{out\}\}\>0\]
foreach response

yj\(i\)y\_\{j\}^\{\(i\)\}and token position

ttdo

Define the prefix state

sj,i,t=\(xj,yj,<t\(i\)\)s\_\{j,i,t\}=\(x\_\{j\},y\_\{j,<t\}^\{\(i\)\}\)
// Student, privileged\-teacher, behavior, and reference distributions

Compute

pj,i,t=πθ\(⋅∣sj,i,t\)p\_\{j,i,t\}=\\pi\_\{\\theta\}\(\\cdot\\mid s\_\{j,i,t\}\),

qj,i,t=SG\[πθ\(⋅∣cj,sj,i,t\)\]q\_\{j,i,t\}=\\mathrm\{SG\}\[\\pi\_\{\\theta\}\(\\cdot\\mid c\_\{j\},s\_\{j,i,t\}\)\]and

πref\(⋅∣sj,i,t\)\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\\mid s\_\{j,i,t\}\)
// Exact full\-vocabulary OPD loss on this sampled prefix

ℒj,i,tOPD=∑a∈𝒱pj,i,t​\(a\)​log⁡pj,i,t​\(a\)qj,i,t​\(a\)\\mathcal\{L\}^\{\\mathrm\{OPD\}\}\_\{j,i,t\}=\\sum\_\{a\\in\\mathcal\{V\}\}p\_\{j,i,t\}\(a\)\\log\\frac\{p\_\{j,i,t\}\(a\)\}\{q\_\{j,i,t\}\(a\)\}
endfor

endfor

Update

θ\\thetaby minimizing

ℒSDPG​\(θ\)\\mathcal\{L\}\_\{\\mathrm\{SDPG\}\}\(\\theta\)in Eq\. \([3\.1](https://arxiv.org/html/2606.04036#S3.E1)\)

endfor

return

πθ\\pi\_\{\\theta\}

### 3\.1KL\-Regularized Policy Optimization with Outcome and OPD

SDPG minimizes a KL\-regularized policy optimization objective with two sources of supervision:

ℒSDPG​\(θ\)=ℒout​\(θ\)\+β​\(k\)​ℒOPD\+​\(θ\)\+α​ℒ𝒦​\(πθ,πref\),\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{SDPG\}\}\(\\theta\)=\\mathcal\{L\}\_\{\\mathrm\{out\}\}\(\\theta\)\+\\beta\(k\)\\mathcal\{L\}\_\{\\mathrm\{OPD\}\}^\{\+\}\(\\theta\)\+\\alpha\\mathcal\{L\}\_\{\\mathcal\{K\}\}\(\\pi\_\{\\theta\},\\pi\_\{\\mathrm\{ref\}\}\),\(3\.1\)whereℒout\\mathcal\{L\}\_\{\\mathrm\{out\}\}is the reward\-based policy\-gradient loss,ℒOPD\+\\mathcal\{L\}\_\{\\mathrm\{OPD\}\}^\{\+\}is the gated full\-vocabulary OPD loss,β​\(k\)\\beta\(k\)is the distillation coefficient at training stepkk, andℒ𝒦\\mathcal\{L\}\_\{\\mathcal\{K\}\}is a KL regularization against the fixed reference policy\. Whenβ=0\\beta=0, SDPG reduces to the corresponding RPG\-style objective\. However, the forms of𝒦\\mathcal\{K\}for the on\-policy loss are quite different fromZhanget al\.\([2026a](https://arxiv.org/html/2606.04036#bib.bib6)\), which is under a one\-step off\-policy setting\. Whenα=0\\alpha=0, SDPG becomes outcome\-reward policy optimization with full\-vocabulary OPD but without a reference\-policy anchor\.

### 3\.2On\-policy Reward\-based Loss

We first derive the reward\-based lossℒout\\mathcal\{L\}\_\{\\mathrm\{out\}\}in Eq\. \([3\.1](https://arxiv.org/html/2606.04036#S3.E1)\)\. For full on\-policy specialization, the rollout policy is exactly the current policyπθ\\pi\_\{\\theta\}\. Therefore,yi∼SG\[πθ\(⋅∣x\)\]y\_\{i\}\\sim\\mathrm\{SG\}\[\\pi\_\{\\theta\}\(\\cdot\\mid x\)\], whereSG\\mathrm\{SG\}is the stop\-gradient operator\. For the objective

Jout​\(θ\)=𝔼x,y∼πθ\(⋅\|x\)​\[R​\(y\)\],\\displaystyle J\_\{\\mathrm\{out\}\}\(\\theta\)=\\mathbb\{E\}\_\{x,y\\sim\\pi\_\{\\theta\}\(\\cdot\|x\)\}\[R\(y\)\],to generate gradient signals from reward, we should change to REINFORCE\-style surrogate lossWilliams \([1992](https://arxiv.org/html/2606.04036#bib.bib27)\):

ℒout​\(xi,yi,θ\)=−SG​\[R​\(yi\)\]​log⁡πθ​\(xi\)\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{out\}\}\(x\_\{i\},y\_\{i\},\\theta\)=\-\\mathrm\{SG\}\[R\(y\_\{i\}\)\]\\log\\pi\_\{\\theta\}\(x\_\{i\}\)so that𝔼x∼πθ​\[∇θℒout​\(x,y,θ\)\]=−∇θJout​\(θ\)\\mathbb\{E\}\_\{x\\sim\\pi\_\{\\theta\}\}\[\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\mathrm\{out\}\}\(x,y,\\theta\)\]=\-\\nabla\_\{\\theta\}J\_\{\\mathrm\{out\}\}\(\\theta\)\. Therefore, no PPO\-style importance\-ratio clipping is needed\. Moreover, similar as GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2606.04036#bib.bib4)\), to mitigate the variance and neutralize baseline bias, we use the same group\-relative advantageAout​\(y\)A\_\{\\mathrm\{out\}\}\(y\)as in Eq\. \([2\.1](https://arxiv.org/html/2606.04036#S2.E1)\) instead of the original rewardR​\(y\)R\(y\)\. Therefore, the verifier\-grounded objective can be written directly as

ℒout​\(θ\)\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{out\}\}\(\\theta\)=−𝔼\(x,c\)∼𝒟,\{yi\}i=1G∼πθ\(⋅∣x\)​\[1∑i=1G\|yi\|​∑i=1G∑t=1\|yi\|SG​\[Aout​\(yi\)\]​log⁡πθ​\(yi,t∣x,yi,<t\)\]\.\\displaystyle=\-\\mathbb\{E\}\_\{\(x,c\)\\sim\\mathcal\{D\},\\;\\\{y\_\{i\}\\\}\_\{i=1\}^\{G\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\bigg\[\\frac\{1\}\{\\sum\_\{i=1\}^\{G\}\|y\_\{i\}\|\}\\sum\_\{i=1\}^\{G\}\\sum\_\{t=1\}^\{\|y\_\{i\}\|\}\\mathrm\{SG\}\[A\_\{\\mathrm\{out\}\}\(y\_\{i\}\)\]\\,\\log\\pi\_\{\\theta\}\(y\_\{i,t\}\\mid x,y\_\{i,<t\}\)\\bigg\]\.

### 3\.3Full\-Vocabulary Distillation on Sampled Prefixes

Instead of using a sampled\-token approximation to the privileged teacher signal, SDPG uses the exact full\-vocabulary student\-to\-teacher KL on each sampled prefix similar toZhaoet al\.\([2026](https://arxiv.org/html/2606.04036#bib.bib17)\)\. For a sampled responseyiy\_\{i\}and token positiontt, define the prefixsi,t=\(x,yi,<t\)s\_\{i,t\}=\(x,y\_\{i,<t\}\)and the two next\-token distributions

pi,t​\(a\)\\displaystyle p\_\{i,t\}\(a\)=πθ​\(a∣x,yi,<t\),\\displaystyle=\\pi\_\{\\theta\}\(a\\mid x,y\_\{i,<t\}\),qi,t​\(a\)\\displaystyle q\_\{i,t\}\(a\)=πθ​\(a∣c,x,yi,<t\)\.\\displaystyle=\\pi\_\{\\theta\}\(a\\mid c,x,y\_\{i,<t\}\)\.With the sampled prefix and teacher branch detached, the per\-token OPD loss is

ℓi,tOPD​\(θ\)=DKL​\(pi,t∥SG​\[qi,t\]\)=∑a∈𝒱pi,t​\(a\)​log⁡pi,t​\(a\)SG​\[qi,t​\(a\)\]\.\\displaystyle\\ell^\{\\mathrm\{OPD\}\}\_\{i,t\}\(\\theta\)=D\_\{\\mathrm\{KL\}\}\(p\_\{i,t\}\\\|\\mathrm\{SG\}\[q\_\{i,t\}\]\)=\\sum\_\{a\\in\\mathcal\{V\}\}p\_\{i,t\}\(a\)\\log\\frac\{p\_\{i,t\}\(a\)\}\{\\mathrm\{SG\}\[q\_\{i,t\}\(a\)\]\}\.\(3\.2\)
###### Proposition 3\.1\(Fixed\-prefix reverse\-KL OPD gradient as a policy gradient\)\.

Fix a rollout prefixst=\(x,y<t\)s\_\{t\}=\(x,y\_\{<t\}\)and writept​\(a\)=πθ​\(a∣x,y<t\)p\_\{t\}\(a\)=\\pi\_\{\\theta\}\(a\\mid x,y\_\{<t\}\)andqt​\(a\)=πθ​\(a∣c,x,y<t\)q\_\{t\}\(a\)=\\pi\_\{\\theta\}\(a\\mid c,x,y\_\{<t\}\)\. Letp¯t=SG​\[pt\]\\bar\{p\}\_\{t\}=\\mathrm\{SG\}\[p\_\{t\}\]andq¯t=SG​\[qt\]\\bar\{q\}\_\{t\}=\\mathrm\{SG\}\[q\_\{t\}\], and assumeq¯t​\(a\)\>0\\bar\{q\}\_\{t\}\(a\)\>0wheneverp¯t​\(a\)\>0\\bar\{p\}\_\{t\}\(a\)\>0\. With the teacher branch detached, the reverse\-KL full\-vocabulary OPD loss

ℒOPD,t​\(θ\)=DKL​\(pt∥q¯t\)\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{OPD\},t\}\(\\theta\)=D\_\{\\mathrm\{KL\}\}\(p\_\{t\}\\\|\\bar\{q\}\_\{t\}\)has the same student\-side gradient, at the current iterate, as the detached\-sampling policy\-gradient surrogate

ℒ~OPD,tPG​\(θ\)=−𝔼a∼p¯t​\[Atdist​\(a\)​log⁡pt​\(a\)\],Atdist​\(a\)=SG​\[D¯t−log⁡p¯t​\(a\)q¯t​\(a\)\],\\displaystyle\\widetilde\{\\mathcal\{L\}\}^\{\\mathrm\{PG\}\}\_\{\\mathrm\{OPD\},t\}\(\\theta\)=\-\\mathbb\{E\}\_\{a\\sim\\bar\{p\}\_\{t\}\}\\left\[A^\{\\mathrm\{dist\}\}\_\{t\}\(a\)\\log p\_\{t\}\(a\)\\right\],\\qquad A^\{\\mathrm\{dist\}\}\_\{t\}\(a\)=\\mathrm\{SG\}\\left\[\\bar\{D\}\_\{t\}\-\\log\\frac\{\\bar\{p\}\_\{t\}\(a\)\}\{\\bar\{q\}\_\{t\}\(a\)\}\\right\],\(3\.3\)whereD¯t=DKL​\(p¯t∥q¯t\)\\bar\{D\}\_\{t\}=D\_\{\\mathrm\{KL\}\}\(\\bar\{p\}\_\{t\}\\\|\\bar\{q\}\_\{t\}\)\. Moreover,AtdistA^\{\\mathrm\{dist\}\}\_\{t\}is centered under the detached student distribution:𝔼a∼p¯t​\[Atdist​\(a\)\]=0\\mathbb\{E\}\_\{a\\sim\\bar\{p\}\_\{t\}\}\[A^\{\\mathrm\{dist\}\}\_\{t\}\(a\)\]=0\.

The proof is given in Appendix[A\.1](https://arxiv.org/html/2606.04036#A1.SS1)\. Proposition[3\.1](https://arxiv.org/html/2606.04036#S3.Thmtheorem1)is a gradient identity, not an implementation change: SDPG minimizes the explicit full\-vocabulary KL in Eq\. \([3\.2](https://arxiv.org/html/2606.04036#S3.E2)\), because this leads to more accurate estimation of the gradient\. The total distillation loss over sampled sequences is

ℒOPD​\(θ\)=𝔼\(x,c\)∼𝒟,\{yi\}i=1G∼πθ\(⋅∣x\)​\[1∑i=1G\|yi\|​∑i=1G∑t=1\|yi\|ℓi,tOPD​\(θ\)\]\.\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{OPD\}\}\(\\theta\)=\\mathbb\{E\}\_\{\(x,c\)\\sim\\mathcal\{D\},\\;\\\{y\_\{i\}\\\}\_\{i=1\}^\{G\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\bigg\[\\frac\{1\}\{\\sum\_\{i=1\}^\{G\}\|y\_\{i\}\|\}\\sum\_\{i=1\}^\{G\}\\sum\_\{t=1\}^\{\|y\_\{i\}\|\}\\ell^\{\\mathrm\{OPD\}\}\_\{i,t\}\(\\theta\)\\bigg\]\.\(3\.4\)

### 3\.4On\-policy Unnormalized KL for SDPG

Now we focus on the KL regularization termℒ𝒦\\mathcal\{L\}\_\{\\mathcal\{K\}\}in Eq\. \([3\.1](https://arxiv.org/html/2606.04036#S3.E1)\)\. We postpone the derivations for general forward and reverse KL regularization to Appendix[A\.2](https://arxiv.org/html/2606.04036#A1.SS2)\. From the derivation there, it is worth noting thatπref=πθ\\pi\_\{\\mathrm\{ref\}\}=\\pi\_\{\\theta\}is not sufficient to minimize the surrogate loss for forward and reserve KL, which is due to the inherent biases within \(normalized\) forward and backward KL regularization\. To tackle this mismatching issue, we apply the unnormalized KL term as introduced in Section[2\.3](https://arxiv.org/html/2606.04036#S2.SS3)\. For simplicity, we denoteJR&D=Jout\+β​\(k\)​JOPDJ\_\{\\mathrm\{R\\&D\}\}=J\_\{\\mathrm\{out\}\}\+\\beta\(k\)J\_\{\\mathrm\{OPD\}\}andℒR&D=ℒout\+β​\(k\)​ℒOPD\\mathcal\{L\}\_\{\\mathrm\{R\\&D\}\}=\\mathcal\{L\}\_\{\\mathrm\{out\}\}\+\\beta\(k\)\\mathcal\{L\}\_\{\\mathrm\{OPD\}\}for training stepkkas the objective and loss functions for reward\-based and distillation terms, respectively\.

In detail, consider the objective using unnormalized forward KL regularization as follows:

JSDPG−UFKL​\(θ\)=JR&D​\(θ\)−α​UKL​\(πref∥πθ\),\\displaystyle J\_\{\\mathrm\{SDPG\-UFKL\}\}\(\\theta\)=J\_\{\\mathrm\{R\\&D\}\}\(\\theta\)\-\\alpha\\mathrm\{UKL\}\(\\pi\_\{\\mathrm\{ref\}\}\\\|\\pi\_\{\\theta\}\),whereJOPDJ\_\{\\mathrm\{OPD\}\}is the objective of on\-policy distillation implicitly involved in Eq\. \([3\.4](https://arxiv.org/html/2606.04036#S3.E4)\)\. The gradient, expressed as an expectation overπθ\\pi\_\{\\theta\}usingwT​\(x\)=πθ​\(x\)/πteacher​\(x\)w\_\{T\}\(x\)=\\pi\_\{\\theta\}\(x\)/\\pi\_\{\\mathrm\{teacher\}\}\(x\),wR=πθ​\(x\)/πref​\(x\)w\_\{R\}=\\pi\_\{\\theta\}\(x\)/\\pi\_\{\\mathrm\{ref\}\}\(x\),πteacher​\(x\)=πθ​\(x,c\)\\pi\_\{\\mathrm\{teacher\}\}\(x\)=\\pi\_\{\\theta\}\(x,c\)is:

∇θJSDPG−UFKL​\(θ\)=∇θJR&D​\(θ\)−α​𝔼x∼πθ​\[\(1−wR​\(x\)−1\)​∇θlog⁡πθ​\(x\)\]\.\\displaystyle\\nabla\_\{\\theta\}J\_\{\\mathrm\{SDPG\-UFKL\}\}\(\\theta\)=\\nabla\_\{\\theta\}J\_\{\\mathrm\{R\\&D\}\}\(\\theta\)\-\\alpha~\\mathbb\{E\}\_\{x\\sim\\pi\_\{\\theta\}\}\\left\[\(1\-w\_\{R\}\(x\)^\{\-1\}\)\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(x\)\\right\]\.A corresponding differentiable surrogate loss term for minimization via gradient descent is \(ignoring the prefixy<ty\_\{<t\}already generated\):

ℒSDPG−UFKL​\(x,θ\)=ℒR&D​\(x,θ\)\+α​\(wR​\(x\)−1\+log⁡wR​\(x\)\),\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{SDPG\-UFKL\}\}\(x,\\theta\)=\\mathcal\{L\}\_\{\\mathrm\{R\\&D\}\}\(x,\\theta\)\+\\alpha\(w\_\{R\}\(x\)^\{\-1\}\+\\log w\_\{R\}\(x\)\),such that𝔼x∼πθ​\[∇θℒSDPG−UFKL​\(x,θ\)\]=−∇θJSDPG−UFKL​\(θ\)\\mathbb\{E\}\_\{x\\sim\\pi\_\{\\theta\}\}\[\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\mathrm\{SDPG\-UFKL\}\}\(x,\\theta\)\]=\-\\nabla\_\{\\theta\}J\_\{\\mathrm\{SDPG\-UFKL\}\}\(\\theta\)\. Therefore, the total surrogate loss function can be written as

ℒSDPG−UFKL​\(θ\)=ℒR&D​\(θ\)\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{SDPG\-UFKL\}\}\(\\theta\)=\\mathcal\{L\}\_\{\\mathrm\{R\\&D\}\}\(\\theta\)\+α​𝔼\(x,c\)∼𝒟,y∼πθ\(⋅∣x\)​\[1∑i=1G\|yi\|​∑i=1G∑t=1\|yi\|\(πref​\(yi,t∣x,yi,<t\)πθ​\(yi,t∣x,yi,<t\)\+log⁡πθ​\(yi,t∣x,yi,<t\)πref​\(yi,t∣x,yi,<t\)\)\]\.\\displaystyle\\qquad\+\\alpha~\\mathbb\{E\}\_\{\(x,c\)\\sim\\mathcal\{D\},y\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\bigg\[\\frac\{1\}\{\\sum\_\{i=1\}^\{G\}\|y\_\{i\}\|\}\\sum\_\{i=1\}^\{G\}\\sum\_\{t=1\}^\{\|y\_\{i\}\|\}\\bigg\(\\frac\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{i,t\}\\mid x,y\_\{i,<t\}\)\}\{\\pi\_\{\\theta\}\(y\_\{i,t\}\\mid x,y\_\{i,<t\}\)\}\+\\log\\frac\{\\pi\_\{\\theta\}\(y\_\{i,t\}\\mid x,y\_\{i,<t\}\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{i,t\}\\mid x,y\_\{i,<t\}\)\}\\bigg\)\\bigg\]\.
Moreover, we can also apply the unnormalized reverse KL regularization as follows:

JSDPG−URKL​\(θ\)=JR&D​\(θ\)−α​UKL​\(πθ∥πref\)\.\\displaystyle J\_\{\\mathrm\{SDPG\-URKL\}\}\(\\theta\)=J\_\{\\mathrm\{R\\&D\}\}\(\\theta\)\-\\alpha\\mathrm\{UKL\}\(\\pi\_\{\\theta\}\\\|\\pi\_\{\\mathrm\{ref\}\}\)\.The gradient, expressed as an expectation overπθ\\pi\_\{\\theta\}is:

∇θJSDPG−URKL​\(θ\)=∇θJR&D​\(θ\)−α​𝔼x∼πθ​\[\(log⁡wR​\(x\)\)​∇θlog⁡πθ​\(x\)\]\.\\displaystyle\\nabla\_\{\\theta\}J\_\{\\mathrm\{SDPG\-URKL\}\}\(\\theta\)=\\nabla\_\{\\theta\}J\_\{\\mathrm\{R\\&D\}\}\(\\theta\)\-\\alpha~\\mathbb\{E\}\_\{x\\sim\\pi\_\{\\theta\}\}\\left\[\(\\log w\_\{R\}\(x\)\)\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(x\)\\right\]\.A corresponding differentiable surrogate loss term for minimization via gradient descent is \(ignoring the prefixy<ty\_\{<t\}already generated\):

ℒSDPG−URKL​\(x,θ\)=ℒR&D​\(x,θ\)\+α2​log2⁡wR​\(x\),\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{SDPG\-URKL\}\}\(x,\\theta\)=\\mathcal\{L\}\_\{\\mathrm\{R\\&D\}\}\(x,\\theta\)\+\\frac\{\\alpha\}\{2\}\\log^\{2\}w\_\{R\}\(x\),such that𝔼x∼πθ​\[∇θℒSDPG−URKL​\(x,θ\)\]=−∇θJSDPG−URKL​\(θ\)\\mathbb\{E\}\_\{x\\sim\\pi\_\{\\theta\}\}\[\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\mathrm\{SDPG\-URKL\}\}\(x,\\theta\)\]=\-\\nabla\_\{\\theta\}J\_\{\\mathrm\{SDPG\-URKL\}\}\(\\theta\)\. And the total surrogate loss function can be written as

ℒSDPG−URKL​\(θ\)\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{SDPG\-URKL\}\}\(\\theta\)=ℒR&D​\(θ\)\+α​𝔼\(x,c\)∼𝒟,y∼πθ\(⋅∣x\)​\[1∑i=1G\|yi\|​∑i=1G∑t=1\|yi\|12​log2⁡πθ​\(yi,t∣x,yi,<t\)πref​\(yi,t∣x,yi,<t\)\]\.\\displaystyle=\\mathcal\{L\}\_\{\\mathrm\{R\\&D\}\}\(\\theta\)\+\\alpha~\\mathbb\{E\}\_\{\(x,c\)\\sim\\mathcal\{D\},y\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\bigg\[\\frac\{1\}\{\\sum\_\{i=1\}^\{G\}\|y\_\{i\}\|\}\\sum\_\{i=1\}^\{G\}\\sum\_\{t=1\}^\{\|y\_\{i\}\|\}\\frac\{1\}\{2\}\\log^\{2\}\\frac\{\\pi\_\{\\theta\}\(y\_\{i,t\}\\mid x,y\_\{i,<t\}\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{i,t\}\\mid x,y\_\{i,<t\}\)\}\\bigg\]\.

### 3\.5Additional Stabilizers for SDPG

Because the current model induces the privileged OPD target under a richer context, it can be noisy early in training and over\-constrained late in training\. We therefore add two lightweight controls to prevent the privileged OPD signal from overwhelming verifier\-grounded learning, including the positive advantage gating and a warmup\-then\-decay scheduler for the distillation term\. These choices trust privileged distillation only on verifier\-endorsed rollouts and phase out the privileged signal near the end of training\.

#### 3\.5\.1Positive Advantage Gating

When a rollout is incorrect \(Aout\(i\)<0A^\{\(i\)\}\_\{\\mathrm\{out\}\}<0\), the privileged teacher can still assign high probability to locally plausible tokens on the sampled wrong prefix\. Applying full\-vocabulary OPD on such prefixes may conflict with the verifier signal: the outcome objective suppresses the trajectory, whereas the distillation objective can still imitate the privileged teacher around that prefix\.

We therefore gate the OPD loss by the outcome advantage:

mi=𝟏​\[Aout\(i\)\>0\],ℒOPD\+​\(θ\)=𝔼\(x,c\)∼𝒟,\{yi\}i=1G∼πθ\(⋅∣x\)​\[1∑i=1G\|yi\|​∑i=1G∑t=1\|yi\|mi​ℓi,tOPD\]\.\\displaystyle m\_\{i\}=\\mathbf\{1\}\[A^\{\(i\)\}\_\{\\mathrm\{out\}\}\>0\],\\qquad\\mathcal\{L\}\_\{\\mathrm\{OPD\}\}^\{\+\}\(\\theta\)=\\mathbb\{E\}\_\{\(x,c\)\\sim\\mathcal\{D\},\\;\\\{y\_\{i\}\\\}\_\{i=1\}^\{G\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\left\[\\frac\{1\}\{\\sum\_\{i=1\}^\{G\}\|y\_\{i\}\|\}\\sum\_\{i=1\}^\{G\}\\sum\_\{t=1\}^\{\|y\_\{i\}\|\}m\_\{i\}\\ell^\{\\mathrm\{OPD\}\}\_\{i,t\}\\right\]\.\(3\.5\)This relies on the full\-vocabulary OPD signal only for trajectories that the verifier endorses within the group\. If all rewards in a group are identical, both the mean\-centered outcome advantage and the OPD gate vanish, avoiding unvalidated distillation on uninformative groups\. In the initial stage, the gate may often be inactive, and the binary outcome reward dominates\. A training dataset with moderate difficulty or curriculum learning\(Wanget al\.,[2021](https://arxiv.org/html/2606.04036#bib.bib13); Leeet al\.,[2024](https://arxiv.org/html/2606.04036#bib.bib14); Wenet al\.,[2025](https://arxiv.org/html/2606.04036#bib.bib15); Shiet al\.,[2025](https://arxiv.org/html/2606.04036#bib.bib16)\)is therefore useful for activating the distillation signal\. Ifmi=1m\_\{i\}=1for all responses, Eq\. \([3\.5](https://arxiv.org/html/2606.04036#S3.E5)\) reduces to standard full\-vocabulary OPD on all sampled prefixes\.

#### 3\.5\.2β\\betaScheduler

![Refer to caption](https://arxiv.org/html/2606.04036v1/x1.png)Figure 2:The illustration of theβ\\betaschedule\.Early misalignment betweenptp\_\{t\}and the privileged distributionqtq\_\{t\}can make the OPD target noisy\. To prevent privileged distillation from destabilizing exploration, we warm upβ\\beta\. The OPD term then takes effect gradually after the outcome policy has begun to find correct trajectories\.

Moreover, under an idealized privileged\-information model, distilling a teacher conditioned on information unavailable to the deployable student can leave an irreducible conditional mutual\-information gap, e\.g\.,I​\(Yt;C∣X,Y<t\)\>0I\(Y\_\{t\};C\\mid X,Y\_\{<t\}\)\>0whenCCdenotes the privileged variable\(Yanget al\.,[2026a](https://arxiv.org/html/2606.04036#bib.bib8)\)\. Under our formulation, this means the privileged OPD target may remain biased by information unavailable at inference\. Therefore, to release the student and encourage exploration, we decayβ\\betaat the end of training, phasing out the distillation signal after the student has internalized its useful information\.

The effective distillation coefficient follows a warmup\-decay schedule, illustrated in Figure[2](https://arxiv.org/html/2606.04036#S3.F2):

β​\(k\)=βbase×min⁡\(1,kTwarm\)⏟warmup×min⁡\(1,T−kTdecay\)⏟decay,\\displaystyle\\beta\(k\)=\\beta\_\{\\mathrm\{base\}\}\\times\\underbrace\{\\min\\left\(1,\\frac\{k\}\{T\_\{\\mathrm\{warm\}\}\}\\right\)\}\_\{\\text\{warmup\}\}\\times\\underbrace\{\\min\\left\(1,\\frac\{T\-k\}\{T\_\{\\mathrm\{decay\}\}\}\\right\)\}\_\{\\text\{decay\}\},whereTwarmT\_\{\\mathrm\{warm\}\}andTdecayT\_\{\\mathrm\{decay\}\}are the warmup and decay step counts, andTTis the total number of training steps\. If the warmup and decay windows overlap, the maximum coefficient can be belowβbase\\beta\_\{\\mathrm\{base\}\}\.

## 4Experiments

In this section, we empirically evaluate our proposed SDPG algorithm and compare the performance against baselines on challenging mathematical reasoning tasks based on pretrained LLMs, including GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2606.04036#bib.bib4)\)and RLSD\(Yanget al\.,[2026a](https://arxiv.org/html/2606.04036#bib.bib8)\)\.

![Refer to caption](https://arxiv.org/html/2606.04036v1/x2.png)

![Refer to caption](https://arxiv.org/html/2606.04036v1/x3.png)

\(a\) AIME24

![Refer to caption](https://arxiv.org/html/2606.04036v1/x4.png)

\(b\) AIME25

![Refer to caption](https://arxiv.org/html/2606.04036v1/x5.png)

\(c\) AMC23

![Refer to caption](https://arxiv.org/html/2606.04036v1/x6.png)

\(d\) Reward

![Refer to caption](https://arxiv.org/html/2606.04036v1/x7.png)

\(e\) Entropy

![Refer to caption](https://arxiv.org/html/2606.04036v1/x8.png)

\(f\) Response Length

Figure 3:Training dynamics and benchmark performance on Qwen3\-4B trained with baseline algorithms and SDPG variants\.Top row:\(a\) AIME24, \(b\) AIME25, \(c\) AMC23\.Bottom row:\(d\) group\-relative reward, \(e\) actor entropy, \(f\) average response length\. Both SDPG\-URKL and SDPG\-UFKL reach higher final accuracies than baselines\. SDPG\-UFKL also avoids the entropy collapse observed in RLSD\.### 4\.1Experiment Settings

We conduct experiments primarily on Qwen3\-4B\(Yanget al\.,[2025](https://arxiv.org/html/2606.04036#bib.bib49)\); additional results on Qwen3\-1\.7B with the further baseline, OPCD, are reported in Appendix[D\.2](https://arxiv.org/html/2606.04036#A4.SS2)\. For training, we utilize the DAPO\-Math\-17k dataset\(Yuet al\.,[2025](https://arxiv.org/html/2606.04036#bib.bib12)\)with 13\.9k English samples and generate the privileged information using Gemini 2\.5 Pro\(Comaniciet al\.,[2025](https://arxiv.org/html/2606.04036#bib.bib26)\)\. And the prompts for teacher and student models are shown in Figure[4](https://arxiv.org/html/2606.04036#S4.F4)\. We evaluate the fine\-tuned models on the benchmarks of AIME2024\(MAA,[2024a](https://arxiv.org/html/2606.04036#bib.bib21),[b](https://arxiv.org/html/2606.04036#bib.bib22)\), AIME2025\(MAA,[2025a](https://arxiv.org/html/2606.04036#bib.bib23),[b](https://arxiv.org/html/2606.04036#bib.bib24)\), and AMC23\(MAA,[2023](https://arxiv.org/html/2606.04036#bib.bib25)\)\. Experiments are implemented using the verl framework\(Shenget al\.,[2025](https://arxiv.org/html/2606.04036#bib.bib10)\)with the vLLM engine\(Kwon,[2025](https://arxiv.org/html/2606.04036#bib.bib11)\)for efficient LLM serving and inference\.

All experiments use the AdamW optimizer\(Loshchilov and Hutter,[2018](https://arxiv.org/html/2606.04036#bib.bib9)\)with a learning rate of1×10−61\\times 10^\{\-6\}, a weight decay of0\.10\.1\(β1,β2\)=\(0\.9,0\.999\)\(\\beta\_\{1\},\\beta\_\{2\}\)=\(0\.9,0\.999\), and gradient clipping at1\.01\.0\. Training proceeds for 400 steps with 10 warmup steps in the beginning\. The global training batch size is 128, with 8 responses per prompt and a temperature of1\.01\.0\. We use FSDP with bfloat16 mixed precision and the vLLM rollout engine, and all experiments are conducted on 8 NVIDIA H100 GPUs\. The maximum prompt and response lengths are set to 2,048 and 4,096, respectively, with dynamic batching enabled\. For all the baselines, we useϵstd=1​e−6\\epsilon\_\{\\mathrm\{std\}\}=1e\-6in Eq\. \([2\.1](https://arxiv.org/html/2606.04036#S2.E1)\), and use a clipping threshold of\(ϵ1,ϵ2\)=\(0\.2,0\.2\)\(\\epsilon\_\{1\},\\epsilon\_\{2\}\)=\(0\.2,0\.2\), and the KL regularization coefficient is set to1×10−31\\times 10^\{\-3\}\. For SDPG, we useα=1×10−3\\alpha=1\\times 10^\{\-3\},βbase=1×10−3\\beta\_\{\\mathrm\{base\}\}=1\\times 10^\{\-3\},Twarm=50T\_\{\\mathrm\{warm\}\}=50, andTdecay=350T\_\{\\mathrm\{decay\}\}=350\.

Solve the following math problem step by step\. Present your final answer inside\\\\backslashboxed\{\}, for example\\\\backslashboxed\{42\}\.\{question\}Remember to put your final answer inside\\\\backslashboxed\{\}\.Student PromptSolve the following math problem step by step\. Present your final answer inside\\\\backslashboxed\{\}, for example\\\\backslashboxed\{42\}\.\{question\}Remember to put your final answer inside\\\\backslashboxed\{\}\. \[TEACHER\_CONTEXT\_TOKEN\]\[Hint\] The correct answer is \{answer\}\. A common way to solve this is: \{solution\}\[Instruction\] If possible, derive the answer \{answer\} using an alternative, equally rigorous mathematical approach \(e\.g\., algebraic vs geometric, or different substitution\)\. If no alternative exists, articulate the standard approach with exceptional clarity\. Do NOT state that you were given the answer or reference\.Teacher PromptFigure 4:Prompt templates for the student and teacher models, where the “\{question\}”, “\{answer\}” are from the dataset and the “\{solution\}” is generated by Gemini 2\.5 Pro\.
### 4\.2Experiment Results

The quantitative results in Table[1](https://arxiv.org/html/2606.04036#S4.T1)demonstrate the competitive performance of the proposed SDPG framework\. Both SDPG\-URKL and SDPG\-UFKL outperform GRPO and RLSD across all the benchmarks, with SDPG\-UFKL achieving the top score in five and SDPG\-URKL in the remaining one\. Figure[3](https://arxiv.org/html/2606.04036#S4.F3)complements these results by illustrating evaluation scores and training dynamics\. The accuracy gap between SDPG and GRPO opens within the first 50 steps and persists throughout training \(Figures[3](https://arxiv.org/html/2606.04036#S4.F3)a–c\), and SDPG reaches the high\-reward plateau \(Figure[3](https://arxiv.org/html/2606.04036#S4.F3)d\) several hundred steps earlier than GRPO\. Notably, SDPG\-UFKL maintains substantially higher actor entropy throughout training \(Figure[3](https://arxiv.org/html/2606.04036#S4.F3)e\), in contrast to RLSD whose entropy collapses toward zero by step 250, a known signature of mode collapse in pure self\-distillation\. We attribute this stability to the combination of positive\-advantage gating and the warmup\-decayβ\\betaschedule, which together prevent privileged distillation from over\-constraining the policy after the student has internalized its useful signal\. Response lengths \(Figure[3](https://arxiv.org/html/2606.04036#S4.F3)f\) for SDPG methods stabilize at intermediate values, sufficient for multi\-step reasoning while remaining shorter than GRPO’s verbose outputs\. Ablation studies isolating the contributions of the OPD distillation term and the KL regularization, as well as additional results on Qwen3\-1\.7B, are reported in Appendix[D](https://arxiv.org/html/2606.04036#A4)\.

Table 1:Performance of models trained with baselines and SDPG variants on AIME24, AIME25, and AMC23 \(pass@1, mean@32\) with Qwen3\-4B trained for 400 steps\.*Last*shows the score at step 400;*Best*shows the peak across training\. Column maximum inbold, second\-best inunderlined\.AIME24AIME25AMC23MethodLastBestLastBestLastBestGRPO0\.2800\.3160\.2420\.2790\.7140\.739RLSD0\.3780\.3950\.3000\.3040\.8130\.813SDPG\-URKL \(ours\)0\.3800\.4010\.3070\.3080\.8630\.863SDPG\-UFKL \(ours\)0\.3800\.4080\.3270\.3350\.8580\.870

## 5Related Work

##### Reinforcement Learning with Verifiable Rewards\.

Reinforcement Learning with Verifiable Rewards \(RLVR\)\(Wanget al\.,[2025b](https://arxiv.org/html/2606.04036#bib.bib29)\)has become one of the dominant techniques in post\-training stages of LLMs, which reward models with an automatic verifier for achieving correct final answers regardless of intermediate reasoning steps\(Srivastava and Aggarwal,[2025](https://arxiv.org/html/2606.04036#bib.bib28)\)\. It eliminates the need for human preference labels in previous algorithms like PPO\(Schulmanet al\.,[2017](https://arxiv.org/html/2606.04036#bib.bib19)\)\. One classic paradigm is GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2606.04036#bib.bib4)\), which utilizes group\-relative advantage, PPO\-style clipped surrogate, as well as KL regularization to achieve competitive performances when training LLMs\. However, GRPO has limitations: advantages collapse and learning stalls when all rollouts in the group have identical rewards, and token\-wise credit assignment remains sparse in long generations\. Therefore, a lot of improved algorithms have been proposed\. Dr\. GRPO\(Liuet al\.,[2025](https://arxiv.org/html/2606.04036#bib.bib30)\)removes biased normalization to improve token efficiency\. VinePPO\(Kazemnejadet al\.,[2024](https://arxiv.org/html/2606.04036#bib.bib53)\)utilizes Monte\-Carlo\-based value estimation to improve credit assignment\. Moreover, DAPO\(Yuet al\.,[2025](https://arxiv.org/html/2606.04036#bib.bib12)\)introduces techniques including clip\-higher, dynamic sampling, token\-level loss, and soft overlong punishment to enhance downstream performance, while GSPO\(Zhenget al\.,[2025](https://arxiv.org/html/2606.04036#bib.bib31)\)uses sequence likelihood in the importance ratio to further stabilize training\. RPG\(Zhanget al\.,[2026a](https://arxiv.org/html/2606.04036#bib.bib6)\)provides a unified framework for different types of KL regularization in GRPO\.

However, these approaches may suffer from signal homogeneity among all tokens in a sequence\. Some research utilizes process reward models\(Lightmanet al\.,[2023](https://arxiv.org/html/2606.04036#bib.bib32); Wanget al\.,[2024](https://arxiv.org/html/2606.04036#bib.bib33); Chenet al\.,[2024](https://arxiv.org/html/2606.04036#bib.bib34); Zhanget al\.,[2025](https://arxiv.org/html/2606.04036#bib.bib35); Daiet al\.,[2025](https://arxiv.org/html/2606.04036#bib.bib36); Yanget al\.,[2026b](https://arxiv.org/html/2606.04036#bib.bib42); Luoet al\.,[2024](https://arxiv.org/html/2606.04036#bib.bib43)\)or step\-level value estimators for intermediate signals\. However, expensive human step annotations may make these methods less affordable\. Beyond these, other works try to use per\-token signals such as entropy, key\-token statistics, uncertainty, or attention dynamics to circumvent expensive annotations\(Xieet al\.,[2025](https://arxiv.org/html/2606.04036#bib.bib37); Liet al\.,[2026](https://arxiv.org/html/2606.04036#bib.bib38); Chenget al\.,[2026](https://arxiv.org/html/2606.04036#bib.bib39); Wanget al\.,[2025a](https://arxiv.org/html/2606.04036#bib.bib40); Chenet al\.,[2025](https://arxiv.org/html/2606.04036#bib.bib41)\)\. However, such utilization of intrinsic signals may be heuristic\. Our method instead incorporates full\-vocabulary privileged self\-distillation while retaining the binary outcome verifier\.

##### On\-Policy Distillation and Self\-Distillation\.

To deal with sparse rewards in reinforcement learning, on\-policy distillation \(OPD\) techniques have been proposed to provide dense token\-level supervision signals\(Agarwalet al\.,[2024](https://arxiv.org/html/2606.04036#bib.bib44); Lu and Lab,[2025](https://arxiv.org/html/2606.04036#bib.bib47); Fuet al\.,[2026](https://arxiv.org/html/2606.04036#bib.bib50)\)\. They train a student model based on trajectories sampled from its own policy, while another teacher model provides token\-level targets via KL regularization or related objectives\(Guet al\.,[2024](https://arxiv.org/html/2606.04036#bib.bib46); Xuet al\.,[2025](https://arxiv.org/html/2606.04036#bib.bib45); Yanget al\.,[2025](https://arxiv.org/html/2606.04036#bib.bib49); Xiaoet al\.,[2026](https://arxiv.org/html/2606.04036#bib.bib48)\)\. However, these OPD methods often require a large external teacher, which requires much larger memory usage and may provide mismatched guidance due to model heterogeneity\.

To address such issues, self\-distillation methods sample from the same student policy but evaluate the model under privileged knowledge, including ground\-truth solution paths and environmental feedback\(Hübotteret al\.,[2026](https://arxiv.org/html/2606.04036#bib.bib18); Shenfeldet al\.,[2026](https://arxiv.org/html/2606.04036#bib.bib5); Penalozaet al\.,[2026](https://arxiv.org/html/2606.04036#bib.bib51)\)\. In detail, OPSD\(Zhaoet al\.,[2026](https://arxiv.org/html/2606.04036#bib.bib17)\)utilizes full KL divergence between teacher and student models to enhance the reasoning ability of LLMs; OPCD\(Yeet al\.,[2026](https://arxiv.org/html/2606.04036#bib.bib1)\)adds flexibility by decoupling the on\-policy strategy; and TRRD\(Zhanget al\.,[2026b](https://arxiv.org/html/2606.04036#bib.bib52)\)involves the teacher policy in the importance ratio to alleviate the conflict between reward function and distillation term\. However, pure on\-policy self\-distillation approaches suffer from limited exploration and mode collapse\. RLSD\(Yanget al\.,[2026a](https://arxiv.org/html/2606.04036#bib.bib8)\)instead uses the privileged teacher\-student likelihood ratio as a token\-level credit reweighting signal inside a GRPO\-style objective; we summarize this distinction in Section[B](https://arxiv.org/html/2606.04036#A2)and provide the full loss in Appendix[B](https://arxiv.org/html/2606.04036#A2)\. By taking advantage of RLVR techniques, the training process becomes smoother\. However, strong teacher signals can still impose a lower ceiling on the student\(Xiaoet al\.,[2026](https://arxiv.org/html/2606.04036#bib.bib48)\)\. SDPG differs from these methods by preserving the exact full\-vocabulary OPD objective while coupling it with verifier\-based policy optimization and reference\-policy KL regularizationing\.

## 6Conclusion

We presented SDPG, a Self\-Distilled Policy Gradient framework that combines verifier\-based RLVR with exact full\-vocabulary OPD\. In this view, the reverse\-KL OPD term remains a full\-vocabulary distillation objective, while its fixed\-prefix student\-side gradient admits an equivalent policy\-gradient form with a centered log\-ratio token advantage\. Combining this OPD\-equivalent distillation signal with binary verifier rewards improves credit assignment while retaining the exploration and selection benefits of RLVR\. On LLM reasoning tasks, the proposed algorithms achieve better performance and stability than baseline algorithms\.

## Acknowledgement

Thank Fetch Compute program for their support of compute resources\.

## References

- On\-policy distillation of language models: learning from self\-generated mistakes\.InThe twelfth international conference on learning representations,Cited by:[§1](https://arxiv.org/html/2606.04036#S1.p3.1),[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px2.p1.1)\.
- G\. Chen, M\. Liao, C\. Li, and K\. Fan \(2024\)Step\-level value preference optimization for mathematical reasoning\.InFindings of the Association for Computational Linguistics: EMNLP 2024,pp\. 7889–7903\.Cited by:[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px1.p2.1)\.
- M\. Chen, G\. Chen, W\. Wang, and Y\. Yang \(2025\)Seed\-grpo: semantic entropy enhanced grpo for uncertainty\-aware policy optimization\.arXiv preprint arXiv:2505\.12346\.Cited by:[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px1.p2.1)\.
- D\. Cheng, S\. Huang, X\. Zhu, B\. Dai, X\. Zhao, Z\. Zhang, and F\. Wei \(2026\)Reasoning with exploration: an entropy perspective\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 30377–30385\.Cited by:[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px1.p2.1)\.
- G\. Comanici, E\. Bieber, M\. Schaekermann,et al\.\(2025\)Gemini 2\.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities\.arXiv preprint arXiv:2507\.06261\.External Links:[Link](https://arxiv.org/abs/2507.06261)Cited by:[§4\.1](https://arxiv.org/html/2606.04036#S4.SS1.p1.1)\.
- M\. Dai, C\. Yang, and Q\. Si \(2025\)S\-grpo: early exit via reinforcement learning in reasoning models\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,Cited by:[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px1.p2.1)\.
- Y\. Fu, H\. Huang, K\. Jiang, Y\. Zhu, and D\. Zhao \(2026\)Revisiting on\-policy distillation: empirical failure modes and simple fixes\.arXiv preprint arXiv:2603\.25562\.Cited by:[§1](https://arxiv.org/html/2606.04036#S1.p3.1),[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px2.p1.1)\.
- Y\. Gu, L\. Dong, F\. Wei, and M\. Huang \(2024\)Minillm: knowledge distillation of large language models\.InThe twelfth international conference on learning representations,Cited by:[§1](https://arxiv.org/html/2606.04036#S1.p3.1),[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px2.p1.1)\.
- J\. Hübotter, F\. Lübeck, L\. Behric, A\. Baumann, M\. Bagatella, D\. Marta, I\. Hakimi, I\. Shenfeld, T\. K\. Buening, C\. Guestrin,et al\.\(2026\)Reinforcement learning via self\-distillation\.arXiv preprint arXiv:2601\.20802\.Cited by:[§1](https://arxiv.org/html/2606.04036#S1.p4.1),[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px2.p2.1)\.
- A\. Kazemnejad, M\. Aghajohari, E\. Portelance, A\. Sordoni, S\. Reddy, A\. Courville, and N\. L\. Roux \(2024\)Vineppo: refining credit assignment in rl training of llms\.arXiv preprint arXiv:2410\.01679\.Cited by:[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px1.p1.1)\.
- W\. Kwon \(2025\)VLLM: an efficient inference engine for large language models\.Ph\.D\. Thesis,UC Berkeley\.Cited by:[§4\.1](https://arxiv.org/html/2606.04036#S4.SS1.p1.1)\.
- B\. W\. Lee, H\. Cho, and K\. M\. Yoo \(2024\)Instruction tuning with human curriculum\.InFindings of the Association for Computational Linguistics: NAACL 2024,pp\. 1281–1309\.Cited by:[§3\.5\.1](https://arxiv.org/html/2606.04036#S3.SS5.SSS1.p2.1)\.
- Z\. Li, L\. Kang, F\. Xiao, L\. Xing, Q\. Si, Z\. Li, W\. Gong, D\. Yang, Y\. Xiao, and H\. Guo \(2026\)Outcome\-grounded advantage reshaping for fine\-grained credit assignment in mathematical reasoning\.arXiv preprint arXiv:2601\.07408\.Cited by:[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px1.p2.1)\.
- H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe \(2023\)Let’s verify step by step\.InThe twelfth international conference on learning representations,Cited by:[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px1.p2.1)\.
- Z\. Liu, C\. Chen, W\. Li, P\. Qi, T\. Pang, C\. Du, W\. S\. Lee, and M\. Lin \(2025\)Understanding r1\-zero\-like training: a critical perspective\.arXiv preprint arXiv:2503\.20783\.Cited by:[§1](https://arxiv.org/html/2606.04036#S1.p2.1),[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px1.p1.1)\.
- I\. Loshchilov and F\. Hutter \(2018\)Decoupled weight decay regularization\.InInternational Conference on Learning Representations,Cited by:[§4\.1](https://arxiv.org/html/2606.04036#S4.SS1.p2.12)\.
- K\. Lu and T\. M\. Lab \(2025\)On\-policy distillation\.Thinking Machines Lab: Connectionism\.External Links:[Document](https://dx.doi.org/10.64434/tml.20251026),[Link](https://thinkingmachines.ai/)Cited by:[§1](https://arxiv.org/html/2606.04036#S1.p3.1),[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px2.p1.1)\.
- L\. Luo, Y\. Liu, R\. Liu, S\. Phatale, M\. Guo, H\. Lara, Y\. Li, L\. Shu, Y\. Zhu, L\. Meng,et al\.\(2024\)Improve mathematical reasoning in language models by automated process supervision\.arXiv preprint arXiv:2406\.06592\.Cited by:[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px1.p2.1)\.
- M\. A\. o\. A\. A\. M\. C\. MAA \(2023\)2023 AMC\.External Links:[Link](https://artofproblemsolving.com/wiki/index.php/AMC_12_Problems_and_Solutions)Cited by:[§4\.1](https://arxiv.org/html/2606.04036#S4.SS1.p1.1)\.
- M\. A\. o\. A\. A\. M\. C\. MAA \(2024a\)2024 AIME\-I\.Note:Accessed: 2025\-05\-08External Links:[Link](https://artofproblemsolving.com/wiki/index.php/2024_AIME_I)Cited by:[§4\.1](https://arxiv.org/html/2606.04036#S4.SS1.p1.1)\.
- M\. A\. o\. A\. A\. M\. C\. MAA \(2024b\)2024 AIME\-II\.Note:Accessed: 2025\-05\-08External Links:[Link](https://artofproblemsolving.com/wiki/index.php/2024_AIME_II)Cited by:[§4\.1](https://arxiv.org/html/2606.04036#S4.SS1.p1.1)\.
- M\. A\. o\. A\. A\. M\. C\. MAA \(2025a\)2025 AIME\-I\.External Links:[Link](https://artofproblemsolving.com/wiki/index.php/2025_AIME_I)Cited by:[§4\.1](https://arxiv.org/html/2606.04036#S4.SS1.p1.1)\.
- M\. A\. o\. A\. A\. M\. C\. MAA \(2025b\)2025 AIME\-II\.External Links:[Link](https://artofproblemsolving.com/wiki/index.php/2025_AIME_II)Cited by:[§4\.1](https://arxiv.org/html/2606.04036#S4.SS1.p1.1)\.
- T\. Minka \(2005\)Divergence measures and message passing\.Cited by:[§2\.3](https://arxiv.org/html/2606.04036#S2.SS3.p1.4)\.
- E\. Penaloza, D\. Vattikonda, N\. Gontier, A\. Lacoste, L\. Charlin, and M\. Caccia \(2026\)Privileged information distillation for language models\.arXiv preprint arXiv:2602\.04942\.Cited by:[§1](https://arxiv.org/html/2606.04036#S1.p4.1),[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px2.p2.1)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§2\.1](https://arxiv.org/html/2606.04036#S2.SS1.p2.9),[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px1.p1.1)\.
- J\. Schulman \(2020\)Approximating kl divergence\.Note:[http://joschu\.net/blog/kl\-approx\.html](http://joschu.net/blog/kl-approx.html)Accessed on June 2, 2026Cited by:[§2\.3](https://arxiv.org/html/2606.04036#S2.SS3.p1.6)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.\(2024\)Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§A\.2](https://arxiv.org/html/2606.04036#A1.SS2.p1.1),[§A\.3](https://arxiv.org/html/2606.04036#A1.SS3.p4.4),[§1](https://arxiv.org/html/2606.04036#S1.p1.1),[§2\.1](https://arxiv.org/html/2606.04036#S2.SS1.p2.5),[§3\.2](https://arxiv.org/html/2606.04036#S3.SS2.p1.7),[§4](https://arxiv.org/html/2606.04036#S4.p1.1),[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px1.p1.1)\.
- I\. Shenfeld, M\. Damani, J\. Hübotter, and P\. Agrawal \(2026\)Self\-distillation enables continual learning\.arXiv preprint arXiv:2601\.19897\.Cited by:[§1](https://arxiv.org/html/2606.04036#S1.p4.1),[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px2.p2.1)\.
- G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. Wu \(2025\)Hybridflow: a flexible and efficient rlhf framework\.InProceedings of the Twentieth European Conference on Computer Systems,pp\. 1279–1297\.Cited by:[§A\.3](https://arxiv.org/html/2606.04036#A1.SS3.p1.2),[§4\.1](https://arxiv.org/html/2606.04036#S4.SS1.p1.1)\.
- T\. Shi, Y\. Wu, L\. Song, T\. Zhou, and J\. Zhao \(2025\)Efficient reinforcement finetuning via adaptive curriculum learning\.InNeurIPS 2025 Workshop on Efficient Reasoning,Cited by:[§3\.5\.1](https://arxiv.org/html/2606.04036#S3.SS5.SSS1.p2.1)\.
- S\. S\. Srivastava and V\. Aggarwal \(2025\)A technical survey of reinforcement learning techniques for large language models\.arXiv preprint arXiv:2507\.04136\.Cited by:[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px1.p1.1)\.
- P\. Wang, L\. Li, Z\. Shao, R\. Xu, D\. Dai, Y\. Li, D\. Chen, Y\. Wu, and Z\. Sui \(2024\)Math\-shepherd: verify and reinforce llms step\-by\-step without human annotations\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 9426–9439\.Cited by:[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px1.p2.1)\.
- S\. Wang, L\. Yu, C\. Gao, C\. Zheng, S\. Liu, R\. Lu, K\. Dang, X\. Chen, J\. Yang, Z\. Zhang,et al\.\(2025a\)Beyond the 80/20 rule: high\-entropy minority tokens drive effective reinforcement learning for llm reasoning\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,Cited by:[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px1.p2.1)\.
- X\. Wang, Y\. Chen, and W\. Zhu \(2021\)A survey on curriculum learning\.IEEE transactions on pattern analysis and machine intelligence44\(9\),pp\. 4555–4576\.Cited by:[§3\.5\.1](https://arxiv.org/html/2606.04036#S3.SS5.SSS1.p2.1)\.
- Y\. Wang, Q\. Yang, Z\. Zeng, L\. Ren, L\. Liu, B\. Peng, H\. Cheng, X\. He, K\. Wang, J\. Gao,et al\.\(2025b\)Reinforcement learning for reasoning in large language models with one training example\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,Cited by:[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px1.p1.1)\.
- L\. Wen, Y\. Cai, F\. Xiao, X\. He, Q\. An, Z\. Duan, Y\. Du, J\. Liu, T\. Tanglifu, X\. Lv,et al\.\(2025\)Light\-r1: curriculum sft, dpo and rl for long cot from scratch and beyond\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 6: Industry Track\),pp\. 318–327\.Cited by:[§3\.5\.1](https://arxiv.org/html/2606.04036#S3.SS5.SSS1.p2.1)\.
- R\. J\. Williams \(1992\)Simple statistical gradient\-following algorithms for connectionist reinforcement learning\.Machine learning8\(3\),pp\. 229–256\.Cited by:[§3\.2](https://arxiv.org/html/2606.04036#S3.SS2.p1.8)\.
- B\. Xiao, B\. Xia, B\. Yang, B\. Gao, B\. Shen, C\. Zhang, C\. He, C\. Lou, F\. Luo, G\. Wang,et al\.\(2026\)Mimo\-v2\-flash technical report\.arXiv preprint arXiv:2601\.02780\.Cited by:[§1](https://arxiv.org/html/2606.04036#S1.p5.5),[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px2.p2.1)\.
- C\. Xie, R\. Pan, X\. Wu, Y\. Zhang, J\. Fu, T\. Gao, and G\. Zhou \(2025\)Unlocking exploration in rlvr: uncertainty\-aware advantage shaping for deeper reasoning\.arXiv preprint arXiv:2510\.10649\.Cited by:[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px1.p2.1)\.
- W\. Xu, R\. Han, Z\. Wang, L\. Le, D\. Madeka, L\. Li, W\. Y\. Wang, R\. Agarwal, C\. Lee, and T\. Pfister \(2025\)Speculative knowledge distillation: bridging the teacher\-student gap through interleaved sampling\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2606.04036#S1.p3.1),[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px2.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§1](https://arxiv.org/html/2606.04036#S1.p3.1),[§4\.1](https://arxiv.org/html/2606.04036#S4.SS1.p1.1),[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px2.p1.1)\.
- C\. Yang, C\. Qin, Q\. Si, M\. Chen, N\. Gu, D\. Yao, Z\. Lin, W\. Wang, J\. Wang, and N\. Duan \(2026a\)Self\-distilled rlvr\.arXiv preprint arXiv:2604\.03128\.Cited by:[§A\.3](https://arxiv.org/html/2606.04036#A1.SS3.p4.4),[Appendix B](https://arxiv.org/html/2606.04036#A2.p1.4),[§1](https://arxiv.org/html/2606.04036#S1.p5.5),[§3\.5\.2](https://arxiv.org/html/2606.04036#S3.SS5.SSS2.p2.3),[§4](https://arxiv.org/html/2606.04036#S4.p1.1),[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px2.p2.1)\.
- C\. Yang, Q\. Si, M\. Dai, D\. Yao, M\. Zheng, M\. Chen, Z\. Lin, and W\. Wang \(2026b\)Test\-time prompt intervention\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 34223–34231\.Cited by:[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px1.p2.1)\.
- T\. Ye, L\. Dong, X\. Wu, S\. Huang, and F\. Wei \(2026\)On\-policy context distillation for language models\.arXiv preprint arXiv:2602\.12275\.Cited by:[Appendix C](https://arxiv.org/html/2606.04036#A3.p1.2),[§D\.2](https://arxiv.org/html/2606.04036#A4.SS2.p1.1),[§1](https://arxiv.org/html/2606.04036#S1.p4.1),[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px2.p2.1)\.
- Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, L\. Liu,et al\.\(2025\)Dapo: an open\-source llm reinforcement learning system at scale\.arXiv preprint arXiv:2503\.14476\.Cited by:[§A\.3](https://arxiv.org/html/2606.04036#A1.SS3.p4.4),[§1](https://arxiv.org/html/2606.04036#S1.p2.1),[§4\.1](https://arxiv.org/html/2606.04036#S4.SS1.p1.1),[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px1.p1.1)\.
- L\. Zhang, A\. Hosseini, H\. Bansal, M\. Kazemi, A\. Kumar, and R\. Agarwal \(2025\)Generative verifiers: reward modeling as next\-token prediction\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px1.p2.1)\.
- Y\. Zhang, Y\. Liu, H\. Yuan, Y\. Yuan, Q\. Gu, and A\. C\. Yao \(2026a\)On the design of kl\-regularized policy gradient algorithms for llm reasoning\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2606.04036#S1.p6.1),[§2\.3](https://arxiv.org/html/2606.04036#S2.SS3.p1.4),[§3\.1](https://arxiv.org/html/2606.04036#S3.SS1.p1.8),[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px1.p1.1)\.
- Z\. Zhang, S\. Jiang, Y\. Shen, Y\. Zhang, D\. Ram, S\. Yang, Z\. Tu, W\. Xia, and S\. Soatto \(2026b\)Reinforcement\-aware knowledge distillation for llm reasoning\.arXiv preprint arXiv:2602\.22495\.Cited by:[§1](https://arxiv.org/html/2606.04036#S1.p4.1),[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px2.p2.1)\.
- S\. Zhao, Z\. Xie, M\. Liu, J\. Huang, G\. Pang, F\. Chen, and A\. Grover \(2026\)Self\-distilled reasoner: on\-policy self\-distillation for large language models\.arXiv preprint arXiv:2601\.18734\.Cited by:[§1](https://arxiv.org/html/2606.04036#S1.p4.1),[§3\.3](https://arxiv.org/html/2606.04036#S3.SS3.p1.3),[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px2.p2.1)\.
- C\. Zheng, S\. Liu, M\. Li, X\. Chen, B\. Yu, C\. Gao, K\. Dang, Y\. Liu, R\. Men, A\. Yang,et al\.\(2025\)Group sequence policy optimization\.arXiv preprint arXiv:2507\.18071\.Cited by:[§1](https://arxiv.org/html/2606.04036#S1.p2.1),[§5](https://arxiv.org/html/2606.04036#S5.SS0.SSS0.Px1.p1.1)\.
- H\. Zhu and R\. Rohwer \(1995\)Information geometric measurements of generalisation\.Preprint\.Cited by:[§2\.3](https://arxiv.org/html/2606.04036#S2.SS3.p1.4)\.

\\appendixpage

## Appendix AMore on SDPG Loss

### A\.1Proof of Proposition[3\.1](https://arxiv.org/html/2606.04036#S3.Thmtheorem1)

For a rollout prefixst=\(x,y<t\)s\_\{t\}=\(x,y\_\{<t\}\), recall the unprivileged student distribution and the privileged teacher distribution:

pt​\(a\)\\displaystyle p\_\{t\}\(a\)=πθ​\(a∣st\)=πθ​\(a∣x,y<t\),\\displaystyle=\\pi\_\{\\theta\}\(a\\mid s\_\{t\}\)=\\pi\_\{\\theta\}\(a\\mid x,y\_\{<t\}\),qt​\(a\)\\displaystyle q\_\{t\}\(a\)=πteacher​\(a∣st\)=πθ​\(a∣c,x,y<t\)\.\\displaystyle=\\pi\_\{\\mathrm\{teacher\}\}\(a\\mid s\_\{t\}\)=\\pi\_\{\\theta\}\(a\\mid c,x,y\_\{<t\}\)\.All identities here are local to a fixed prefixsts\_\{t\}and to the current iterate used to form the detached quantities\. Write

p¯t=SG​\[pt\],q¯t=SG​\[qt\],D¯t=DKL​\(p¯t∥q¯t\)\.\\bar\{p\}\_\{t\}=\\mathrm\{SG\}\[p\_\{t\}\],\\qquad\\bar\{q\}\_\{t\}=\\mathrm\{SG\}\[q\_\{t\}\],\\qquad\\bar\{D\}\_\{t\}=D\_\{\\mathrm\{KL\}\}\(\\bar\{p\}\_\{t\}\\\|\\bar\{q\}\_\{t\}\)\.The sampled\-prefix distribution, the teacher branch, and any policy\-gradient coefficients are treated as detached, as in standard surrogate optimization\. Reverse\-KL full\-vocabulary on\-policy distillation minimizes

ℒOPD,t​\(θ\)=DKL​\(pt∥q¯t\)=∑a∈𝒱pt​\(a\)​log⁡pt​\(a\)q¯t​\(a\)\.\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{OPD\},t\}\(\\theta\)=D\_\{\\mathrm\{KL\}\}\(p\_\{t\}\\\|\\bar\{q\}\_\{t\}\)=\\sum\_\{a\\in\\mathcal\{V\}\}p\_\{t\}\(a\)\\log\\frac\{p\_\{t\}\(a\)\}\{\\bar\{q\}\_\{t\}\(a\)\}\.\(A\.1\)The negative student\-side gradient of Eq\. \([A\.1](https://arxiv.org/html/2606.04036#A1.E1)\) is

−∇θℒOPD,t​\(θ\)\\displaystyle\-\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\mathrm\{OPD\},t\}\(\\theta\)=−∑a∈𝒱pt​\(a\)​\(log⁡pt​\(a\)q¯t​\(a\)\+1\)​∇θlog⁡pt​\(a\)\\displaystyle=\-\\sum\_\{a\\in\\mathcal\{V\}\}p\_\{t\}\(a\)\\left\(\\log\\frac\{p\_\{t\}\(a\)\}\{\\bar\{q\}\_\{t\}\(a\)\}\+1\\right\)\\nabla\_\{\\theta\}\\log p\_\{t\}\(a\)=𝔼a∼p¯t​\[−pt​\(a\)p¯t​\(a\)​\(log⁡pt​\(a\)q¯t​\(a\)\+1\)​∇θlog⁡pt​\(a\)\],\\displaystyle=\\mathbb\{E\}\_\{a\\sim\\bar\{p\}\_\{t\}\}\\left\[\-\\frac\{p\_\{t\}\(a\)\}\{\\bar\{p\}\_\{t\}\(a\)\}\\left\(\\log\\frac\{p\_\{t\}\(a\)\}\{\\bar\{q\}\_\{t\}\(a\)\}\+1\\right\)\\nabla\_\{\\theta\}\\log p\_\{t\}\(a\)\\right\],where gradients are evaluated at the iterate satisfyingp¯t=pt\\bar\{p\}\_\{t\}=p\_\{t\}\. At the iterate wherep¯t=pt\\bar\{p\}\_\{t\}=p\_\{t\},

𝔼a∼p¯t​\[∇θlog⁡pt​\(a\)\]=∑a∈𝒱pt​\(a\)​∇θlog⁡pt​\(a\)=∇θ​∑a∈𝒱pt​\(a\)=0\.\\mathbb\{E\}\_\{a\\sim\\bar\{p\}\_\{t\}\}\[\\nabla\_\{\\theta\}\\log p\_\{t\}\(a\)\]=\\sum\_\{a\\in\\mathcal\{V\}\}p\_\{t\}\(a\)\\nabla\_\{\\theta\}\\log p\_\{t\}\(a\)=\\nabla\_\{\\theta\}\\sum\_\{a\\in\\mathcal\{V\}\}p\_\{t\}\(a\)=0\.Adding the state\-dependent baseline1\+D¯t1\+\\bar\{D\}\_\{t\}therefore leaves the gradient unchanged and yields

−∇θℒOPD,t​\(θ\)=𝔼a∼p¯t​\[\(D¯t−log⁡p¯t​\(a\)q¯t​\(a\)\)​∇θlog⁡pt​\(a\)\],\\displaystyle\-\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\mathrm\{OPD\},t\}\(\\theta\)=\\mathbb\{E\}\_\{a\\sim\\bar\{p\}\_\{t\}\}\\left\[\\left\(\\bar\{D\}\_\{t\}\-\\log\\frac\{\\bar\{p\}\_\{t\}\(a\)\}\{\\bar\{q\}\_\{t\}\(a\)\}\\right\)\\nabla\_\{\\theta\}\\log p\_\{t\}\(a\)\\right\],which is the negative gradient of the detached\-sampling surrogate in Eq\. \([3\.3](https://arxiv.org/html/2606.04036#S3.E3)\)\. Centering follows from

𝔼a∼p¯t​\[Atdist​\(a\)\]=D¯t−∑a∈𝒱p¯t​\(a\)​log⁡p¯t​\(a\)q¯t​\(a\)=0\.\\mathbb\{E\}\_\{a\\sim\\bar\{p\}\_\{t\}\}\[A^\{\\mathrm\{dist\}\}\_\{t\}\(a\)\]=\\bar\{D\}\_\{t\}\-\\sum\_\{a\\in\\mathcal\{V\}\}\\bar\{p\}\_\{t\}\(a\)\\log\\frac\{\\bar\{p\}\_\{t\}\(a\)\}\{\\bar\{q\}\_\{t\}\(a\)\}=0\.□\\square

Thus, on a fixed sampled prefix and with the teacher branch detached, reverse\-KL full\-vocabulary OPD has a local policy\-gradient interpretation with centered log\-ratio advantage\. SDPG nevertheless implements the explicit full\-vocabulary KL in Eq\. \([A\.1](https://arxiv.org/html/2606.04036#A1.E1)\); the policy\-gradient form is an interpretation of its gradient, not a sampled\-token replacement\. The same identity also yields sampled\-token Monte Carlo estimators by replacing the full\-vocabulary expectation with samples from the detached student distribution\.

### A\.2Normalized KL terms

In common implementations such as GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2606.04036#bib.bib4)\), a forward or reverse KL term is directly applied to the final loss functions\. However, if such KL terms are only rollout\-based estimation, they could be biased and here we derive the correct loss functions for these KL regularizations\.

Firstly, we consider the objective for forward KL regularization as follows:

JFKL=JR&D−α​DKL​\(πref∥πθ\)\.\\displaystyle J\_\{\\mathrm\{FKL\}\}=J\_\{\\mathrm\{R\\&D\}\}\-\\alpha D\_\{\\mathrm\{KL\}\}\(\\pi\_\{\\mathrm\{ref\}\}\\\|\\pi\_\{\\theta\}\)\.The gradient, expressed as an expectation overπθ\\pi\_\{\\theta\}usingwT​\(x\)=πθ​\(x\)/πteacher​\(x\)w\_\{T\}\(x\)=\\pi\_\{\\theta\}\(x\)/\\pi\_\{\\mathrm\{teacher\}\}\(x\),wR=πθ​\(x\)/πref​\(x\)w\_\{R\}=\\pi\_\{\\theta\}\(x\)/\\pi\_\{\\mathrm\{ref\}\}\(x\),πteacher​\(x\)=πθ​\(x,c\)\\pi\_\{\\mathrm\{teacher\}\}\(x\)=\\pi\_\{\\theta\}\(x,c\)is:

∇θJFKL​\(θ\)=∇θJR&D​\(θ\)\+α​𝔼x∼πθ​\[wR​\(x\)−1​∇θlog⁡πθ​\(x\)\]\.\\displaystyle\\nabla\_\{\\theta\}J\_\{\\mathrm\{FKL\}\}\(\\theta\)=\\nabla\_\{\\theta\}J\_\{\\mathrm\{R\\&D\}\}\(\\theta\)\+\\alpha~\\mathbb\{E\}\_\{x\\sim\\pi\_\{\\theta\}\}\[w\_\{R\}\(x\)^\{\-1\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(x\)\]\.A corresponding differentiable surrogate loss term for minimization via gradient descent is \(ignoring the prefixy<ty\_\{<t\}already generated\):

ℒFKL​\(x,θ\)=ℒR&D​\(x,θ\)\+α​wR​\(x\)−1,\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{FKL\}\}\(x,\\theta\)=\\mathcal\{L\}\_\{\\mathrm\{R\\&D\}\}\(x,\\theta\)\+\\alpha w\_\{R\}\(x\)^\{\-1\},such that𝔼x∼πθ​\[∇θℒFKL​\(x,θ\)\]=−∇θJFKL​\(θ\)\\mathbb\{E\}\_\{x\\sim\\pi\_\{\\theta\}\}\[\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\mathrm\{FKL\}\}\(x,\\theta\)\]=\-\\nabla\_\{\\theta\}J\_\{\\mathrm\{FKL\}\}\(\\theta\)\. And the total surrogate loss function can be written as

ℒFKL​\(θ\)\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{FKL\}\}\(\\theta\)=ℒR&D​\(θ\)\+α​𝔼\(x,c\)∼𝒟,y∼πθ\(⋅∣x\)​\[1∑i=1G\|yi\|​∑i=1G∑t=1\|yi\|πref​\(yi,t∣x,yi,<t\)πθ​\(yi,t∣x,yi,<t\)\]\\displaystyle=\\mathcal\{L\}\_\{\\mathrm\{R\\&D\}\}\(\\theta\)\+\\alpha~\\mathbb\{E\}\_\{\(x,c\)\\sim\\mathcal\{D\},y\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\bigg\[\\frac\{1\}\{\\sum\_\{i=1\}^\{G\}\|y\_\{i\}\|\}\\sum\_\{i=1\}^\{G\}\\sum\_\{t=1\}^\{\|y\_\{i\}\|\}\\frac\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{i,t\}\\mid x,y\_\{i,<t\}\)\}\{\\pi\_\{\\theta\}\(y\_\{i,t\}\\mid x,y\_\{i,<t\}\)\}\\bigg\]
For the reverse KL regularization, consider the objective as follows:

JRKL=JR&D−α​DKL​\(πθ∥πref\)\.\\displaystyle J\_\{\\mathrm\{RKL\}\}=J\_\{\\mathrm\{R\\&D\}\}\-\\alpha D\_\{\\mathrm\{KL\}\}\(\\pi\_\{\\theta\}\\\|\\pi\_\{\\mathrm\{ref\}\}\)\.The gradient, expressed as an expectation overπθ\\pi\_\{\\theta\}usingwT​\(x\)=πθ​\(x\)/πteacher​\(x\)w\_\{T\}\(x\)=\\pi\_\{\\theta\}\(x\)/\\pi\_\{\\mathrm\{teacher\}\}\(x\),wR=πθ​\(x\)/πref​\(x\)w\_\{R\}=\\pi\_\{\\theta\}\(x\)/\\pi\_\{\\mathrm\{ref\}\}\(x\),πteacher​\(x\)=πθ​\(x,c\)\\pi\_\{\\mathrm\{teacher\}\}\(x\)=\\pi\_\{\\theta\}\(x,c\)is:

∇θJRKL​\(θ\)=∇θJR&D​\(θ\)−α​𝔼x∼πθ​\[\(log⁡wR​\(x\)\+1\)​∇θlog⁡πθ​\(x\)\]\.\\displaystyle\\nabla\_\{\\theta\}J\_\{\\mathrm\{RKL\}\}\(\\theta\)=\\nabla\_\{\\theta\}J\_\{\\mathrm\{R\\&D\}\}\(\\theta\)\-\\alpha~\\mathbb\{E\}\_\{x\\sim\\pi\_\{\\theta\}\}\[\(\\log w\_\{R\}\(x\)\+1\)\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(x\)\]\.A corresponding differentiable surrogate loss term for minimization via gradient descent is \(ignoring the prefixy<ty\_\{<t\}already generated\):

ℒRKL​\(x,θ\)=ℒR&D​\(x,θ\)\+α2​\(log⁡wR​\(x\)\+1\)2,\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{RKL\}\}\(x,\\theta\)=\\mathcal\{L\}\_\{\\mathrm\{R\\&D\}\}\(x,\\theta\)\+\\frac\{\\alpha\}\{2\}\(\\log w\_\{R\}\(x\)\+1\)^\{2\},such that𝔼x∼πθ​\[∇θℒRKL​\(x,θ\)\]=−∇θJRKL​\(θ\)\\mathbb\{E\}\_\{x\\sim\\pi\_\{\\theta\}\}\[\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\mathrm\{RKL\}\}\(x,\\theta\)\]=\-\\nabla\_\{\\theta\}J\_\{\\mathrm\{RKL\}\}\(\\theta\)\. And the total surrogate loss function can be written as

ℒRKL​\(θ\)\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{RKL\}\}\(\\theta\)=ℒR&D​\(θ\)\+α​𝔼\(x,c\)∼𝒟,y∼πθ\(⋅∣x\)​\[1∑i=1G\|yi\|​∑i=1G∑t=1\|yi\|12​\(1\+log⁡πθ​\(yi,t∣x,yi,<t\)πref​\(yi,t∣x,yi,<t\)\)2\]\.\\displaystyle=\\mathcal\{L\}\_\{\\mathrm\{R\\&D\}\}\(\\theta\)\+\\alpha~\\mathbb\{E\}\_\{\(x,c\)\\sim\\mathcal\{D\},y\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\bigg\[\\frac\{1\}\{\\sum\_\{i=1\}^\{G\}\|y\_\{i\}\|\}\\sum\_\{i=1\}^\{G\}\\sum\_\{t=1\}^\{\|y\_\{i\}\|\}\\frac\{1\}\{2\}\\bigg\(1\+\\log\\frac\{\\pi\_\{\\theta\}\(y\_\{i,t\}\\mid x,y\_\{i,<t\}\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{i,t\}\\mid x,y\_\{i,<t\}\)\}\\bigg\)^\{2\}\\bigg\]\.
It can be observed that due to expectation overπθ\\pi\_\{\\theta\}, which is differentiable, although both gradient and expectation operations are linear\-operator, they cannot commute and the gradient of expectation is not the same as the expectation of the gradient\. Therefore, for rollout\-based KL regularization, the original KL loss forms are actually biased\.

### A\.3Analysis in one\-step off\-policy settings

In the implementation of modern RL frameworks, such as verl\(Shenget al\.,[2025](https://arxiv.org/html/2606.04036#bib.bib10)\), the rollout modelπrollout\\pi\_\{\\mathrm\{rollout\}\}is often different from the current modelπθ\\pi\_\{\\theta\}\. For example, there would be some sub\-iterations of gradient updates for mini\-batches in each step\. Therefore, there could be issues of within\-step off\-policy drift and stale importance weights\. Therefore, the loss would be different from the full on\-policy implementations, and an importance sampling ratio factor should be applied in the loss function\. Based on Eq\.[3\.1](https://arxiv.org/html/2606.04036#S3.E1), we have

ℒSDPG​\(θ\)\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{SDPG\}\}\(\\theta\)=ℒout​\(θ\)\+β​\(k\)​ℒOPD​\(θ\)\+α​ℒ𝒦​\(πθ,πref\)\\displaystyle=\\mathcal\{L\}\_\{\\mathrm\{out\}\}\(\\theta\)\+\\beta\(k\)\\mathcal\{L\}\_\{\\mathrm\{OPD\}\}\(\\theta\)\+\\alpha\\mathcal\{L\}\_\{\\mathcal\{K\}\}\(\\pi\_\{\\theta\},\\pi\_\{\\mathrm\{ref\}\}\)=𝔼\(x,c\)∼𝒟,\{yi\}i=1G∼πθ\(⋅∣x\)\[1∑i=1G\|yi\|∑i=1G∑t=1\|yi\|−SG\[Aout\(yi\)\]logπθ\(yi,t∣x,yi,<t\)\\displaystyle=\\mathbb\{E\}\_\{\(x,c\)\\sim\\mathcal\{D\},\\;\\\{y\_\{i\}\\\}\_\{i=1\}^\{G\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\bigg\[\\frac\{1\}\{\\sum\_\{i=1\}^\{G\}\|y\_\{i\}\|\}\\sum\_\{i=1\}^\{G\}\\sum\_\{t=1\}^\{\|y\_\{i\}\|\}\-\\mathrm\{SG\}\[A\_\{\\mathrm\{out\}\}\(y\_\{i\}\)\]\\,\\log\\pi\_\{\\theta\}\(y\_\{i,t\}\\mid x,y\_\{i,<t\}\)\+βℒOPD\(x,θ\)\+αf𝒦\(πθ,πR;x,yi\)\]\\displaystyle\\qquad\+\\beta\\mathcal\{L\}\_\{\\mathrm\{OPD\}\}\(x,\\theta\)\+\\alpha f\_\{\\mathcal\{K\}\}\(\\pi\_\{\\theta\},\\pi\_\{R\};x,y\_\{i\}\)\\bigg\]=𝔼\(x,c\)∼𝒟,\{yi\}i=1G∼πrollout\(⋅∣x\)\[1∑i=1G\|yi\|∑i=1G∑t=1\|yi\|−SG\[ρi,tAi\]logπθ\(yi,t∣x,yi,<t\)\\displaystyle=\\mathbb\{E\}\_\{\(x,c\)\\sim\\mathcal\{D\},\\;\\\{y\_\{i\}\\\}\_\{i=1\}^\{G\}\\sim\\pi\_\{\\mathrm\{rollout\}\}\(\\cdot\\mid x\)\}\\bigg\[\\frac\{1\}\{\\sum\_\{i=1\}^\{G\}\|y\_\{i\}\|\}\\sum\_\{i=1\}^\{G\}\\sum\_\{t=1\}^\{\|y\_\{i\}\|\}\-\\mathrm\{SG\}\[\\rho\_\{i,t\}A\_\{i\}\]\\,\\log\\pi\_\{\\theta\}\(y\_\{i,t\}\\mid x,y\_\{i,<t\}\)\+βSG\[ρi,t\]ℒOPD\(x,θ\)\+αSG\[ρi,t\]f𝒦\(πθ,πR;x,yi\)\],\\displaystyle\\qquad\+\\beta\\mathrm\{SG\}\[\\rho\_\{i,t\}\]\\mathcal\{L\}\_\{\\mathrm\{OPD\}\}\(x,\\theta\)\+\\alpha\\mathrm\{SG\}\[\\rho\_\{i,t\}\]f\_\{\\mathcal\{K\}\}\(\\pi\_\{\\theta\},\\pi\_\{R\};x,y\_\{i\}\)\\bigg\],where

ρi,t=ρ​\(yi,t∣x,yi,<t\)=πθ​\(yi,t∣x,yi,<t\)πrollout​\(yi,t∣x,yi,<t\)\\displaystyle\\rho\_\{i,t\}=\\rho\(y\_\{i,t\}\\mid x,y\_\{i,<t\}\)=\\frac\{\\pi\_\{\\theta\}\(y\_\{i,t\}\\mid x,y\_\{i,<t\}\)\}\{\\pi\_\{\\mathrm\{rollout\}\}\(y\_\{i,t\}\\mid x,y\_\{i,<t\}\)\}is the importance ratio during sampling andAi=Aout​\(yi\)A\_\{i\}=A\_\{\\mathrm\{out\}\}\(y\_\{i\}\)\. Andf𝒦f\_\{\\mathcal\{K\}\}is the corresponding KL surrogate loss\.

For the reward\-based term, usually we can apply a PPO\-style clip to stabilize the training process\. In detail, the clipped advantage with the importance ratio is

ℓt,iclip=\{min⁡\(max⁡\(−Ai​ρt,i,−Ai​clip​\(ρt,i,1−εl,1\+εh\)\),−Ai​c\)Ai<0max⁡\(−Ai​ρt,i,−Ai​clip​\(ρt,i,1−εl,1\+εh\)\)Ai≥0,\\displaystyle\\ell\_\{t,i\}^\{\\mathrm\{clip\}\}=\\begin\{cases\}\\min\\\!\\bigl\(\\max\(\-A\_\{i\}\\rho\_\{t,i\},\\,\-A\_\{i\}\\,\\text\{clip\}\(\\rho\_\{t,i\},1\{\-\}\\varepsilon\_\{l\},1\{\+\}\\varepsilon\_\{h\}\)\),\\;\-A\_\{i\}c\\bigr\)&A\_\{i\}<0\\\\ \\max\(\-A\_\{i\}\\rho\_\{t,i\},\\,\-A\_\{i\}\\,\\text\{clip\}\(\\rho\_\{t,i\},1\{\-\}\\varepsilon\_\{l\},1\{\+\}\\varepsilon\_\{h\}\)\)&A\_\{i\}\\geq 0\\end\{cases\},
whereϵl,ϵh\\epsilon\_\{l\},\\epsilon\_\{h\}as well asccare the clipping hyperparameters\. Here we can setϵh=ϵl\\epsilon\_\{h\}=\\epsilon\_\{l\}as in GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2606.04036#bib.bib4)\)and RLSD\(Yanget al\.,[2026a](https://arxiv.org/html/2606.04036#bib.bib8)\), orϵh\>ϵl\\epsilon\_\{h\}\>\\epsilon\_\{l\}as in DAPO\(Yuet al\.,[2025](https://arxiv.org/html/2606.04036#bib.bib12)\)\. Therefore, we achieve the final loss in near on\-policy \(one\-step off\-policy\) approximation implementation:

ℒSDPGapprox​\(θ\)\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{SDPG\}\}^\{\\mathrm\{approx\}\}\(\\theta\)=𝔼\(x,c\)∼𝒟,\{yi\}i=1G∼πrollout\(⋅∣x\)\[1∑i=1G\|yi\|∑i=1G∑t=1\|yi\|SG\[ℓt,iclip\]logπθ\(yi,t∣x,yi,<t\)\\displaystyle=\\mathbb\{E\}\_\{\(x,c\)\\sim\\mathcal\{D\},\\;\\\{y\_\{i\}\\\}\_\{i=1\}^\{G\}\\sim\\pi\_\{\\mathrm\{rollout\}\}\(\\cdot\\mid x\)\}\\bigg\[\\frac\{1\}\{\\sum\_\{i=1\}^\{G\}\|y\_\{i\}\|\}\\sum\_\{i=1\}^\{G\}\\sum\_\{t=1\}^\{\|y\_\{i\}\|\}\\mathrm\{SG\}\[\\ell\_\{t,i\}^\{\\mathrm\{clip\}\}\]\\,\\log\\pi\_\{\\theta\}\(y\_\{i,t\}\\mid x,y\_\{i,<t\}\)\+βSG\[ρi,t\]ℒOPD\(x,θ\)\+αSG\[ρi,t\]f𝒦\(πθ,πR;x,yi\)\]\.\\displaystyle\\qquad\+\\beta\\mathrm\{SG\}\[\\rho\_\{i,t\}\]\\mathcal\{L\}\_\{\\mathrm\{OPD\}\}\(x,\\theta\)\+\\alpha\\mathrm\{SG\}\[\\rho\_\{i,t\}\]f\_\{\\mathcal\{K\}\}\(\\pi\_\{\\theta\},\\pi\_\{R\};x,y\_\{i\}\)\\bigg\]\.

### A\.4Discussion about OPSD\-style distillation

While we use OPSD\-style full\-vocabulary on\-policy distillation term, deploying this objective in modern distributed training frameworks \(e\.g\., FSDP with vLLM rollouts\) requires specific implementation\. In this section, we provide a rigorous analysis of the exact loss computed in our implementation, analyzing the discrepancies between the theoretical gradients and the practical approximations\.

The full OPSD\-style loss function is given by:

ℒOPD​\(θ\)=𝔼\(x,c\)∼𝒟\{yi\}i=1G∼πθ\(⋅∣x\)​\[1∑i\|yi\|​∑i,t𝔼v∼πθ\(⋅∣x,yi,<t\)​\[log⁡πθ​\(v∣x,yi,<t\)SG​\[πθ​\(v∣c,x,yi,<t\)\]\]\]\.\\mathcal\{L\}\_\{\\mathrm\{OPD\}\}\(\\theta\)=\\mathbb\{E\}\_\{\\begin\{subarray\}\{c\}\(x,c\)\\sim\\mathcal\{D\}\\\\ \\\{y\_\{i\}\\\}\_\{i=1\}^\{G\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\\end\{subarray\}\}\\left\[\\frac\{1\}\{\\sum\_\{i\}\|y\_\{i\}\|\}\\sum\_\{i,t\}\\mathbb\{E\}\_\{v\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x,y\_\{i,<t\}\)\}\\left\[\\log\\frac\{\\pi\_\{\\theta\}\(v\\mid x,y\_\{i,<t\}\)\}\{\\mathrm\{SG\}\[\\pi\_\{\\theta\}\(v\\mid c,x,y\_\{i,<t\}\)\]\}\\right\]\\right\]\.
And the corresponding objective is:

J​\(θ\)=𝔼y∼πθ​\[∑tDKL​\(πθ∥SG​\[πteacher\]\)\],J\(\\theta\)=\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\theta\}\}\\left\[\\sum\_\{t\}D\_\{\\mathrm\{KL\}\}\(\\pi\_\{\\theta\}\\\|\\mathrm\{SG\}\[\\pi\_\{\\text\{teacher\}\}\]\)\\right\],
whereπteacher=πθ\(⋅\|c,⋅\)\\pi\_\{\\mathrm\{teacher\}\}=\\pi\_\{\\theta\}\(\\cdot\|c,\\cdot\)\. When differentiating this objective, it yields two distinct gradient paths for the distillation term:

∇θ𝔼y∼πθ​\[∑tDKL\]=𝔼y​\[∇θ​∑tDKL\]⏟\(1\) Direct Path\-wise Gradient\+𝔼y​\[∑tDKL⋅∇θlog⁡πθ​\(yt\)\]⏟\(2\) Score Function Gradient\\nabla\_\{\\theta\}\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\theta\}\}\\left\[\\sum\_\{t\}D\_\{\\mathrm\{KL\}\}\\right\]=\\underbrace\{\\mathbb\{E\}\_\{y\}\\left\[\\nabla\_\{\\theta\}\\sum\_\{t\}D\_\{\\mathrm\{KL\}\}\\right\]\}\_\{\\text\{\(1\) Direct Path\-wise Gradient\}\}\+\\underbrace\{\\mathbb\{E\}\_\{y\}\\left\[\\sum\_\{t\}D\_\{\\mathrm\{KL\}\}\\cdot\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{t\}\)\\right\]\}\_\{\\text\{\(2\) Score Function Gradient\}\}
The direct path\-wise gradient term corresponds toℒOPD​\(θ\)\\mathcal\{L\}\_\{\\mathrm\{OPD\}\}\(\\theta\), which is the actual implementation in most modern RL frameworks\. However, the score function term is usually omitted, which actually treats the computed KL divergence as an additional negative reward signal\. It is worth noting that the magnitude of this term \(β⋅DKL\\beta\\cdot D\_\{\\mathrm\{KL\}\}\) is negligible compared to the primary sequence\-level outcome advantageAiA\_\{i\}\. And the variance introduced by estimating it would outweigh its marginal theoretical benefit\. Therefore, the distillation term acts almost as a local shaping constraint as in term \(1\)\. Therefore, we employ the OPSD\-style on\-policy self\-distillation loss as a suitable approximation\.

## Appendix BReinforcement Learning with Self\-Distillation

RLSD\(Yanget al\.,[2026a](https://arxiv.org/html/2606.04036#bib.bib8)\)combines verifier\-grounded RLVR with privileged self\-distillation, but it uses the privileged model only to redistribute token\-level credit rather than to define an auxiliary distribution\-matching objective\. For a sampled tokenyi,ty\_\{i,t\}, RLSD computes a stop\-gradient privileged information gainΔi,t=SG​\[log⁡qi,t​\(yi,t\)−log⁡pi,t​\(yi,t\)\]\\Delta\_\{i,t\}=\\mathrm\{SG\}\[\\log q\_\{i,t\}\(y\_\{i,t\}\)\-\\log p\_\{i,t\}\(y\_\{i,t\}\)\], whereqi,tq\_\{i,t\}is the privileged teacher distribution andpi,tp\_\{i,t\}is the deployable student distribution\. This gain is exponentiated with the sign of the sequence\-level verifier advantage and used to reweight the GRPO token advantage\. Consequently, tokens favored by the privileged context receive larger positive credit on successful trajectories, while the ratio is inverted for negative\-advantage trajectories\.

The key distinction from SDPG is that RLSD does not optimize a separate full\-vocabulary distribution\-matching OPD loss\. The privileged model only changes the magnitude of the verifier\-grounded policy\-gradient update, whereas SDPG keeps the exact full\-vocabulary reverse\-KL OPD objective and combines it with outcome\-reward optimization\.

For a sampled responsey\(i\)y^\{\(i\)\}and prefix\(x,yi,<t\)\(x,y\_\{i,<t\}\), define the deployable student distribution and privileged teacher distribution as

pi,t​\(a\)\\displaystyle p\_\{i,t\}\(a\)=πθ​\(a∣x,yi,<t\),\\displaystyle=\\pi\_\{\\theta\}\(a\\mid x,y\_\{i,<t\}\),qi,t​\(a\)\\displaystyle q\_\{i,t\}\(a\)=πθ​\(a∣c,x,yi,<t\)\.\\displaystyle=\\pi\_\{\\theta\}\(a\\mid c,x,y\_\{i,<t\}\)\.RLSD computes the stop\-gradient privileged information gain on the sampled token:

Δi,t=SG​\[log⁡qi,t​\(yi,t\)−log⁡pi,t​\(yi,t\)\]\.\\displaystyle\\Delta\_\{i,t\}=\\mathrm\{SG\}\\left\[\\log q\_\{i,t\}\(y\_\{i,t\}\)\-\\log p\_\{i,t\}\(y\_\{i,t\}\)\\right\]\.The gain is converted into a direction\-aware evidence weight by using the sign of the sequence\-level verifier advantage:

ui,t\\displaystyle u\_\{i,t\}=exp⁡\(sign⁡\(A\(i\)\)​Δi,t\)\\displaystyle=\\exp\\left\(\\operatorname\{sign\}\(A^\{\(i\)\}\)\\Delta\_\{i,t\}\\right\)=\(qi,t​\(yi,t\)pi,t​\(yi,t\)\)sign⁡\(A\(i\)\)\.\\displaystyle=\\left\(\\frac\{q\_\{i,t\}\(y\_\{i,t\}\)\}\{p\_\{i,t\}\(y\_\{i,t\}\)\}\\right\)^\{\\operatorname\{sign\}\(A^\{\(i\)\}\)\}\.The evidence weight is clipped and interpolated with the original GRPO advantage:

A^i,tRLSD=A\(i\)​\[\(1−λrlsd\)\+λrlsd​clip⁡\(ui,t,1−ϵw,1\+ϵw\)\],\\displaystyle\\widehat\{A\}^\{\\mathrm\{RLSD\}\}\_\{i,t\}=A^\{\(i\)\}\\left\[\(1\-\\lambda\_\{\\mathrm\{rlsd\}\}\)\+\\lambda\_\{\\mathrm\{rlsd\}\}\\operatorname\{clip\}\(u\_\{i,t\},1\-\\epsilon\_\{w\},1\+\\epsilon\_\{w\}\)\\right\],whereλrlsd∈\[0,1\]\\lambda\_\{\\mathrm\{rlsd\}\}\\in\[0,1\]controls the strength of self\-distilled credit redistribution andϵw\\epsilon\_\{w\}bounds the per\-token credit deviation\. Settingλrlsd=0\\lambda\_\{\\mathrm\{rlsd\}\}=0recovers the uniform GRPO advantage, whileλrlsd=1\\lambda\_\{\\mathrm\{rlsd\}\}=1gives the fully reweighted RLSD advantage\.

Under the minimization convention used in this paper, the corresponding GRPO\-style RLSD surrogate is

ℒRLSD​\(θ\)=−𝔼\(x,c\)∼𝒟,\{y\(i\)\}i=1G∼πold\(⋅∣x\)​\[1∑i=1G\|y\(i\)\|​∑i=1G∑t=1\|y\(i\)\|min⁡\(ρi,t​A^i,tRLSD,clip⁡\(ρi,t,1−ϵ,1\+ϵ\)​A^i,tRLSD\)\],\\begin\{aligned\} \\mathcal\{L\}\_\{\\mathrm\{RLSD\}\}\(\\theta\)=\-\\mathbb\{E\}\_\{\(x,c\)\\sim\\mathcal\{D\},\\;\\\{y^\{\(i\)\}\\\}\_\{i=1\}^\{G\}\\sim\\pi\_\{\\mathrm\{old\}\}\(\\cdot\\mid x\)\}\\left\[\\frac\{1\}\{\\sum\_\{i=1\}^\{G\}\|y^\{\(i\)\}\|\}\\sum\_\{i=1\}^\{G\}\\sum\_\{t=1\}^\{\|y^\{\(i\)\}\|\}\\min\\left\(\\rho\_\{i,t\}\\widehat\{A\}^\{\\mathrm\{RLSD\}\}\_\{i,t\},\\operatorname\{clip\}\(\\rho\_\{i,t\},1\-\\epsilon,1\+\\epsilon\)\\widehat\{A\}^\{\\mathrm\{RLSD\}\}\_\{i,t\}\\right\)\\right\],\\end\{aligned\}

where

ρi,t=πθ​\(yi,t∣x,yi,<t\)πold​\(yi,t∣x,yi,<t\)\.\\displaystyle\\rho\_\{i,t\}=\\frac\{\\pi\_\{\\theta\}\(y\_\{i,t\}\\mid x,y\_\{i,<t\}\)\}\{\\pi\_\{\\mathrm\{old\}\}\(y\_\{i,t\}\\mid x,y\_\{i,<t\}\)\}\.Equivalently, RLSD replaces the uniform sequence\-level GRPO advantageA\(i\)A^\{\(i\)\}in Eq\. \([2\.2](https://arxiv.org/html/2606.04036#S2.E2)\) with the token\-dependent advantageA^i,tRLSD\\widehat\{A\}^\{\\mathrm\{RLSD\}\}\_\{i,t\}\. No separate full\-vocabulary OPD KL loss is optimized; the privileged teacher affects only the magnitude of token\-level credit and not the sign of the verifier\-grounded update\.

## Appendix CAnalysis on On\-policy Context Distillation

Recent advancements like On\-Policy Context Distillation \(OPCD\)\(Yeet al\.,[2026](https://arxiv.org/html/2606.04036#bib.bib1)\)attempt to distill in\-context knowledgeccthrough on\-policy KL matching\. In our notation, the student distribution at prefixst=\(x,y<t\)s\_\{t\}=\(x,y\_\{<t\}\)is

pt​\(a\)=πθ​\(a∣x,y<t\),p\_\{t\}\(a\)=\\pi\_\{\\theta\}\(a\\mid x,y\_\{<t\}\),and the privileged distribution is

qt​\(a\)=πθ​\(a∣c,x,y<t\)\.q\_\{t\}\(a\)=\\pi\_\{\\theta\}\(a\\mid c,x,y\_\{<t\}\)\.Letq¯t=SG​\[qt\]\\bar\{q\}\_\{t\}=\\mathrm\{SG\}\[q\_\{t\}\]andp¯t=SG​\[pt\]\\bar\{p\}\_\{t\}=\\mathrm\{SG\}\[p\_\{t\}\]denote the detached distributions at the current iterate, and setD¯t=DKL​\(p¯t∥q¯t\)\\bar\{D\}\_\{t\}=D\_\{\\mathrm\{KL\}\}\(\\bar\{p\}\_\{t\}\\\|\\bar\{q\}\_\{t\}\)\. The reverse\-KL full\-vocabulary OPD objective used by SDPG is

ℒOPD,t=DKL​\(pt∥q¯t\)=∑a∈𝒱pt​\(a\)​log⁡pt​\(a\)q¯t​\(a\)\.\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{OPD\},t\}=D\_\{\\mathrm\{KL\}\}\(p\_\{t\}\\\|\\bar\{q\}\_\{t\}\)=\\sum\_\{a\\in\\mathcal\{V\}\}p\_\{t\}\(a\)\\log\\frac\{p\_\{t\}\(a\)\}\{\\bar\{q\}\_\{t\}\(a\)\}\.The corresponding negative student\-side gradient at the fixed prefix is

−∇θℒOPD,t=−∑a∈𝒱pt​\(a\)​\(log⁡pt​\(a\)q¯t​\(a\)\+1\)​∇θlog⁡pt​\(a\)\.\\displaystyle\-\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\mathrm\{OPD\},t\}=\-\\sum\_\{a\\in\\mathcal\{V\}\}p\_\{t\}\(a\)\\left\(\\log\\frac\{p\_\{t\}\(a\)\}\{\\bar\{q\}\_\{t\}\(a\)\}\+1\\right\)\\nabla\_\{\\theta\}\\log p\_\{t\}\(a\)\.Equivalently, at the same fixed prefix and current iterate, the gradient can be written as an expectation over a detached student\-token sample:

−∇θℒOPD,t=𝔼a∼p¯t​\[SG​\[D¯t−log⁡p¯t​\(a\)q¯t​\(a\)\]​∇θlog⁡pt​\(a\)\],\-\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\mathrm\{OPD\},t\}=\\mathbb\{E\}\_\{a\\sim\\bar\{p\}\_\{t\}\}\\left\[\\mathrm\{SG\}\\left\[\\bar\{D\}\_\{t\}\-\\log\\frac\{\\bar\{p\}\_\{t\}\(a\)\}\{\\bar\{q\}\_\{t\}\(a\)\}\\right\]\\nabla\_\{\\theta\}\\log p\_\{t\}\(a\)\\right\],where the added baseline1\+D¯t1\+\\bar\{D\}\_\{t\}is valid becausep¯t=pt\\bar\{p\}\_\{t\}=p\_\{t\}at the iterate where the surrogate is formed\. SDPG uses the explicit full\-vocabulary objectiveDKL​\(pt∥q¯t\)D\_\{\\mathrm\{KL\}\}\(p\_\{t\}\\\|\\bar\{q\}\_\{t\}\)in implementation, so its distillation component is exactly reverse\-KL OPD, while the centered log\-ratio expression clarifies the corresponding local policy\-gradient signal\.

However, pure self\-distillation differs from SDPG in two important ways\. First, it optimizes the privileged teacher signal without a binary outcome verifier, so it has no mechanism to prefer globally correct trajectories over locally plausible but incorrect ones\. Second, it usually applies the teacher signal on all trajectories, including trajectories that the verifier would reject\. SDPG addresses these issues by combining binary outcome rewards, positive\-advantage gating, and reference\-policy KL regularizationing\.

## Appendix DAblation Studies

### D\.1Effect of the Full\-Vocabulary OPD Term and KL regularization

To isolate the individual contributions of the two non\-outcome components in the SDPG objective \(Eq\. \([3\.1](https://arxiv.org/html/2606.04036#S3.E1)\)\), we run two ablations on Qwen3\-4B: settingα=0\\alpha=0removes the policy KL regularization, leaving only the binary outcome reward and full\-vocabulary OPD term, and settingβ=0\\beta=0removes the OPD term, recovering RPG without self\-distillation\.

![Refer to caption](https://arxiv.org/html/2606.04036v1/x9.png)

![Refer to caption](https://arxiv.org/html/2606.04036v1/x10.png)

\(a\) AIME24

![Refer to caption](https://arxiv.org/html/2606.04036v1/x11.png)

\(b\) AIME25

![Refer to caption](https://arxiv.org/html/2606.04036v1/x12.png)

\(c\) AMC23

![Refer to caption](https://arxiv.org/html/2606.04036v1/x13.png)

\(d\) Reward

![Refer to caption](https://arxiv.org/html/2606.04036v1/x14.png)

\(e\) Entropy

![Refer to caption](https://arxiv.org/html/2606.04036v1/x15.png)

\(f\) Response Length

Figure 5:Ablation study on Qwen3\-4B isolating the KL regularization \(α\\alpha\) and the full\-vocabulary OPD term \(β\\beta\)\. The top row are the result of \(a\) AIME24, \(b\) AIME25, \(c\) AMC23 \(all pass@1 mean@32\)\. The bottom row shows the \(d\) reward, \(e\) entropy, \(f\) response length\. URKL \(default\) and UFKL are the full SDPG variants, while “α=0\\alpha=0” removes the policy KL regularization and “β=0\\beta=0” removes the OPD term\.The two ablations reveal complementary roles\. Removing the full\-vocabulary OPD term \(β=0\\beta=0\) preserves the reward and length profiles of URKL but loses the early\-training accuracy advantage on AIME24 and AIME25 \(Figures[5](https://arxiv.org/html/2606.04036#A4.F5)a, b\), confirming that privileged distillation is the primary driver of fast convergence on harder benchmarks\. Removing the policy KL regularization \(α=0\\alpha=0\), in contrast, achieves comparable or slightly higher accuracy on AIME24/25 but at the cost of severely shortened response length \(around 2,000 tokens, Figure[5](https://arxiv.org/html/2606.04036#A4.F5)f\) and rising entropy \(Figure[5](https://arxiv.org/html/2606.04036#A4.F5)e\), suggesting that without the anchor the student begins to deviate from coherent reasoning patterns\. These observations indicate that full\-vocabulary OPD provides dense supervision, while the KL regularization stabilizes policy updates, so that we choose to retain both components\.

### D\.2Robustness Across Model Scales: Qwen3\-1\.7B

To assess whether the design of SDPG generalizes beyond the 4B scale, we additionally run experiments on the smaller Qwen3\-1\.7B base model under the same training and evaluation protocol, with one additional baseline: OPCD\(Yeet al\.,[2026](https://arxiv.org/html/2606.04036#bib.bib1)\)\. Results are shown in Figure[6](https://arxiv.org/html/2606.04036#A4.F6)and Table[2](https://arxiv.org/html/2606.04036#A4.T2)\.

![Refer to caption](https://arxiv.org/html/2606.04036v1/x16.png)

![Refer to caption](https://arxiv.org/html/2606.04036v1/x17.png)

\(a\) AIME24

![Refer to caption](https://arxiv.org/html/2606.04036v1/x18.png)

\(b\) AIME25

![Refer to caption](https://arxiv.org/html/2606.04036v1/x19.png)

\(c\) AMC23

![Refer to caption](https://arxiv.org/html/2606.04036v1/x20.png)

\(d\) Reward

![Refer to caption](https://arxiv.org/html/2606.04036v1/x21.png)

\(e\) Entropy

![Refer to caption](https://arxiv.org/html/2606.04036v1/x22.png)

\(f\) Response Length

Figure 6:Training dynamics and benchmark performance on Qwen3\-1\.7B trained with baseline algorithms and SDPG variants\. The top row are the results of: \(a\) AIME24, \(b\) AIME25, \(c\) AMC23 \(all pass@1 mean@32\)\. The bottom row are the results of \(d\) reward, \(e\) entropy, \(f\) response length\. SDPG\-URKL and SDPG\-UFKL outperform GRPO and RLSD across all three benchmarks\. OPCD, a pure self\-distillation baseline without reward, exhibits training instability after step 250, with accuracy on AIME and response length both collapsing\.Table 2:Performance on Qwen3\-1\.7B \(pass@1, mean@32\)\.*Last*is the score at step 400;*Best*is the peak across training\. Column maximum inbold, second\-bestunderlined\.AIME24AIME25AMC23MethodLastBestLastBestLastBestGRPO0\.0960\.1150\.1180\.1180\.4670\.467RLSD0\.1770\.1990\.1580\.1850\.5730\.582OPCD0\.0170\.1250\.0130\.1400\.2450\.474SDPG\-URKL \(ours\)0\.1910\.1970\.1890\.1920\.6200\.620SDPG\-UFKL \(ours\)0\.1920\.2120\.1820\.1880\.6370\.666The results on the 1\.7B model corroborate the main findings at the 4B scale\. SDPG\-UFKL achieves the highest score on five of the six*Last*/*Best*columns, with SDPG\-URKL taking the lead on AIME25, and the two SDPG variants together hold the top\-two positions in every column except AIME24*Best*\. The accuracy gap over GRPO and RLSD opens within the first 50 steps and persists throughout training \(Figures[6](https://arxiv.org/html/2606.04036#A4.F6)a–c\)\. RLSD’s entropy collapses below0\.10\.1by step 200 \(Figure[6](https://arxiv.org/html/2606.04036#A4.F6)e\), mirroring the pattern observed at the 4B scale, while SDPG\-URKL and SDPG\-UFKL maintain entropy above0\.40\.4throughout training\. Notably, OPCD, as a pure self\-distillation baseline, exhibits sharp degradation after step 250, where the AIME24 accuracy drops from0\.130\.13to0\.020\.02, response length collapses to under 300 tokens, and reward turns sharply negative \(Figures[6](https://arxiv.org/html/2606.04036#A4.F6)a, d, f\)\. This instability supports the central design hypothesis of SDPG that full\-vocabulary privileged distillation must be coupled with a binary outcome objective and a policy anchor to remain stable, especially on smaller models where pure imitation amplifies teacher imperfections\.