Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance

# Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance
Source: [https://arxiv.org/html/2608.00782](https://arxiv.org/html/2608.00782)
First Author Affiliation / Address line 1 Affiliation / Address line 2 Affiliation / Address line 3 email@domain &Second Author Affiliation / Address line 1 Affiliation / Address line 2 Affiliation / Address line 3 email@domain Zhuowen Han1,Jinwei Xiao2,Zhengxi Lu2,Renren Jin1,Zhiyuan Yao2,Yuxin Liu2, Hongyan Hao2,Yueqing Sun2,Yu Yang2,Qi Gu2,Xunliang Cai2,Deyi Xiong1,†\\dagger

1TJUNLP Lab, School of Computer Science and Technology, Tianjin University, 2Meituan Longcat Team \{zwhan, dyxiong\}@tju\.edu\.cn guqi03@meituan\.com

###### Abstract

Reinforcement learning with verifiable rewards \(RLVR\) has become a standard paradigm for post\-training large language models \(LLMs\)\. While Group Relative Policy Optimization \(GRPO\) is widely adopted, it suffers from sparse reward signals and loses gradients entirely when all responses within a group receive identical rewards\. On\-policy distillation \(OPD\) offers a natural remedy by providing dense, token\-level supervision from a teacher model\. However, naively combining GRPO with OPD leads to degraded performance, due to three underlying causes: not all samples benefit from distillation; fitting too quickly to the teacher undermines the exploratory capacity of RL; and OPD’s advantages are asymmetric, suppressing most tokens\. To address these challenges, we proposeRSTG\(Recovering LearningSignals via AdaptiveTeacherGuidance\), which applies distillation selectively and precisely where it matters most\. At the sample level, OPD is restricted to negative zero\-variance prompts with each sample weighted by the teacher’s confidence score\. At the token level, distillation targets only tokens with high student entropy or large teacher\-student divergence\. We further augment training with SFT on correct trajectories generated by the teacher model, injecting positive gradient signals where RL yields none\. Experiments demonstrate that RSTG substantially outperforms naive GRPO\+OPD by \+4\.02% on math and \+3\.05% on code\.

Distill Where You Fail: Recovering Learning Signals of Negative RL\-Groups from Adaptive Teacher Guidance

Zhuowen Han1, Jinwei Xiao2, Zhengxi Lu2, Renren Jin1, Zhiyuan Yao2, Yuxin Liu2,Hongyan Hao2,Yueqing Sun2,Yu Yang2,Qi Gu2,Xunliang Cai2,Deyi Xiong1,†\\dagger1TJUNLP Lab, School of Computer Science and Technology, Tianjin University,2Meituan Longcat Team\{zwhan, dyxiong\}@tju\.edu\.cn guqi03@meituan\.com

††footnotetext:†\\daggerCorresponding author## 1Introduction

Post\-training large language models via reinforcement learning with verifiable rewards \(RLVR\) has emerged as a standard approach for improving reasoning capabilities\(Team,[2025b](https://arxiv.org/html/2608.00782#bib.bib14); DeepSeek\-AI,[2025](https://arxiv.org/html/2608.00782#bib.bib16); Team,[2025a](https://arxiv.org/html/2608.00782#bib.bib18)\)\. Among RLVR methods, Group Relative Policy Optimization \(GRPO;Shaoet al\.,[2024](https://arxiv.org/html/2608.00782#bib.bib17)\) is widely adopted for its simplicity and stability\. GRPO normalizes outcome rewards across a group of rollouts to estimate a scalar advantage applied uniformly to every token, resulting in sparse rewards\(Liet al\.,[2026c](https://arxiv.org/html/2608.00782#bib.bib38)\)and vanishing gradients when all rollouts are correct or incorrect; we refer to such prompts aspositiveandnegative zero\-variance prompts, respectively\(Zhenget al\.,[2025](https://arxiv.org/html/2608.00782#bib.bib19); Fenget al\.,[2025](https://arxiv.org/html/2608.00782#bib.bib20)\)\.

![Refer to caption](https://arxiv.org/html/2608.00782v1/x1.png)Figure 1:Training Dynamics on MATH\.Naively GRPO \+ OPD proves ineffective, while oursRSTGachieves higher performance\.Recent work has shifted toward on\-policy distillation \(OPD;Agarwalet al\.,[2024](https://arxiv.org/html/2608.00782#bib.bib21); Guet al\.,[2024](https://arxiv.org/html/2608.00782#bib.bib22)\), which has the student generate its own rollouts and leverages the teacher’s per\-token log\-probabilities as a dense reward signal, naturally addressing the limitations of GRPO\. A straightforward approach is to complement GRPO with OPD by summing their losses\(Agarwalet al\.,[2024](https://arxiv.org/html/2608.00782#bib.bib21)\)\. However, as shown in Figure[1](https://arxiv.org/html/2608.00782#S1.F1), this naive combination underperforms standard GRPO, and even with an annealing schedule on the OPD coefficient\(Xuet al\.,[2025](https://arxiv.org/html/2608.00782#bib.bib11)\), the results remain unsatisfactory\.

We identify three key reasons why this naive combination fails:\(1\) Sample\-level Uniformity:Not all samples benefit from OPD; the quality of the teacher’s guidance depends on its proficiency on the given sample\.\(2\) Teacher\-boundedness:OPD converges quickly with performance capped at the teacher’s level\. Premature convergence to the teacher severely undermines RL’s exploratory capacity\.\(3\) Advantage Asymmetry:Student\-generated tokens are typically assigned low probability by the teacher, causing most token\-level advantages to be negative\(Fuet al\.,[2026](https://arxiv.org/html/2608.00782#bib.bib2); Jiaet al\.,[2026](https://arxiv.org/html/2608.00782#bib.bib3); Koet al\.,[2026](https://arxiv.org/html/2608.00782#bib.bib27); Luet al\.,[2026](https://arxiv.org/html/2608.00782#bib.bib5)\), suppressing the learning signal\. Furthermore, OPD is inherently local, conditioning on a potentially erroneous student\-generated prefix and thus producing unreliable gradients\(Fuet al\.,[2026](https://arxiv.org/html/2608.00782#bib.bib2)\)\.

To address the aforementioned limitations, we proposeRSTG\(Recovering LearningSignals via AdaptiveTeacherGuidance\)\.\(1\) Sample Selection\.Applying OPD exclusively on prompts where the student fails but the teacher succeeds outperforms standard OPD on all samples, using only3\.63%3\.63\\%of the full data\. We therefore apply OPD solely onnegative zero\-variance prompts\(Leet al\.,[2025](https://arxiv.org/html/2608.00782#bib.bib10); Zhenget al\.,[2025](https://arxiv.org/html/2608.00782#bib.bib19)\)and weight the distillation signal by the teacher’s degree of mastery over each prompt\.\(2\) Token Selection\.We restrict OPD gradient updates to tokens where the student exhibits high entropy or where the teacher\-student divergence is large, slowing convergence and reducing gradient noise\.\(3\) Auxiliary SFT\.On negative zero\-variance prompts, we perform SFT on correct trajectories pre\-generated by the teacher, injecting positive gradient signals and providing a global perspective to alleviate the locality of OPD\.

We validateRSTGacross the Qwen2\.5 and Qwen3 model families on mathematical and code benchmarks\.RSTGachieves substantial improvements over naive GRPO\+OPD \(\+4\.02%\+4\.02\\%on mathematics,\+3\.05%\+3\.05\\%on code\), while also mitigating advantage asymmetry, slowing convergence toward the teacher, and preventing abrupt response length inflation\.

In summary, our contributions are as follows:

- •We identify the applicable scope of OPD and demonstrate the impact of teacher proficiency on its effectiveness\.
- •We proposeRSTG, enabling effective integration of GRPO and OPD\.

## 2Related Work

#### On\-Policy Distillation \(OPD\)\.

Knowledge distillation\(Hintonet al\.,[2015](https://arxiv.org/html/2608.00782#bib.bib24)\)transfers capabilities from a teacher to a student model\. OPD\(Agarwalet al\.,[2024](https://arxiv.org/html/2608.00782#bib.bib21); Guet al\.,[2024](https://arxiv.org/html/2608.00782#bib.bib22); Koet al\.,[2026](https://arxiv.org/html/2608.00782#bib.bib27)\)samples trajectories from the student and aligns it with the teacher’s token\-level logit distribution, providing dense on\-policy supervision that naturally complements GRPO\. Two dominant paradigms exist: GKD\-style OPD uses the token\-level KL divergence directly as the training loss\(Agarwalet al\.,[2024](https://arxiv.org/html/2608.00782#bib.bib21)\), while PG\-style OPD treats the per\-token reverse KL as a dense reward signal and updates the student via policy gradient, with the advantage defined as the negation of the reverse KL\(Koet al\.,[2026](https://arxiv.org/html/2608.00782#bib.bib27); Yanget al\.,[2026a](https://arxiv.org/html/2608.00782#bib.bib25)\)\. We adopt the latter, which unifies naturally with the RL framework\(Yanget al\.,[2026b](https://arxiv.org/html/2608.00782#bib.bib23)\)\.

#### Combining RL and OPD\.

Prior work has explored integrating RL with knowledge distillation to jointly leverage reward signals and teacher supervision\. GKD\(Agarwalet al\.,[2024](https://arxiv.org/html/2608.00782#bib.bib21)\)first explores this unification for text summarization, and KDRL\(Xuet al\.,[2025](https://arxiv.org/html/2608.00782#bib.bib11)\)further extends this direction\. More recently, a growing body of work has investigated combining RL with on\-policy self\-distillation \(OPSD\)\(Yanget al\.,[2026a](https://arxiv.org/html/2608.00782#bib.bib25); Liet al\.,[2026a](https://arxiv.org/html/2608.00782#bib.bib26); Luet al\.,[2026](https://arxiv.org/html/2608.00782#bib.bib5)\)\. However, this paradigm still faces significant challenges, and existing methods tend to fail across a broader range of models and settings\.RSTGaims to make this combination more robust and effective\.

## 3Preliminaries

LetDDdenote the input distribution, and letπ𝜽\\pi\_\{\\bm\{\\theta\}\}andπ∗\\pi^\{\*\}denote the student and teacher policies, respectively\. We unify the following three methods under a common reinforcement learning framework\.

### 3\.1Group Relative Policy Optimization

The RL objective can be formulated as

𝒥RL​\(𝜽\)=max𝜽⁡𝔼𝒙∼D,𝒚∼π𝜽\(⋅∣x\)​r​\(𝒙,𝒚\),\\mathcal\{J\}\_\{\\mathrm\{RL\}\}\(\\bm\{\\theta\}\)=\\max\_\{\\bm\{\\theta\}\}\\mathbb\{E\}\_\{\\bm\{x\}\\sim D,\\bm\{y\}\\sim\\pi\_\{\\bm\{\\theta\}\}\(\\cdot\\mid x\)\}r\(\\bm\{x\},\\bm\{y\}\),\(1\)where the trajectories𝒚\\bm\{y\}are sampled from the current policy modelπ𝜽\\pi\_\{\\bm\{\\theta\}\},r​\(𝒙,𝒚\)r\(\\bm\{x\},\\bm\{y\}\)is the reward function that measures the quality of a response sequence𝒚=\(y1,⋯,yT\)\\bm\{y\}=\(y\_\{1\},\\cdots,y\_\{T\}\)to a query𝒙\\bm\{x\}\. A common approach to solving this objective is to apply policy gradient, updating the policy parameters using an estimated gradient of the form:

∇𝜽𝒥RL​\(𝜽\)\\displaystyle\\nabla\_\{\\bm\{\\theta\}\}\\mathcal\{J\}\_\{\\mathrm\{RL\}\}\(\\bm\{\\theta\}\)=𝔼𝒙∼D,𝒚∼π𝜽\(⋅∣𝒙\)\\displaystyle=\\mathbb\{E\}\_\{\\bm\{x\}\\sim D,\\bm\{y\}\\sim\\pi\_\{\\bm\{\\theta\}\}\(\\cdot\\mid\\bm\{x\}\)\}\(2\)\[∑t=1TAt​∇𝜽log⁡π𝜽​\(yt∣𝒙,𝒚<t\)\],\\displaystyle\\quad\\left\[\\sum\_\{t=1\}^\{T\}A\_\{t\}\\nabla\_\{\\bm\{\\theta\}\}\\log\\pi\_\{\\bm\{\\theta\}\}\\left\(y\_\{t\}\\mid\\bm\{x\},\\bm\{y\}\_\{<t\}\\right\)\\right\],whereAtA\_\{t\}is the relative advantage of tokenyty\_\{t\}over a baseline value\. Traditional methods such as PPO estimateAtA\_\{t\}via a learned critic, introducing substantial overhead\. Group Relative Policy Optimization \(GRPO\) obviates the need for an additional value function approximator as required in PPO\. Instead, it uses the average reward of multiple sampled outputs generated in response to the same question as the baseline\. Formally, letπ𝜽\\pi\_\{\\bm\{\{\\theta\}\}\}sample G responses\{y1,y2,⋯,yG\}\\\{y\_\{1\},y\_\{2\},\\cdots,y\_\{G\}\\\}for each promptxx\. The optimization objective of GRPO with token\- level loss and without the KL penalty term is:

𝒥GRPO​\(𝜽\)=\\displaystyle\\mathcal\{J\}\_\{\\text\{GRPO\}\}\(\\bm\{\\theta\}\)=𝔼𝒙∼D,\{𝒚\(i\)\}i=1G∼π𝜽old\(⋅∣𝒙\)\\displaystyle\\mathbb\{E\}\_\{\\bm\{x\}\\sim D,\\\{\\bm\{y\}^\{\(i\)\}\\\}\_\{i=1\}^\{G\}\\sim\\pi\_\{\\bm\{\\theta\}\_\{\\mathrm\{old\}\}\}\(\\cdot\\mid\\bm\{x\}\)\}\(3\)\[1G∑i=1G1\|yi\|∑t=1\|yi\|min\(ri,t\(𝜽\)Ai,t,\\displaystyle\\left\[\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\\frac\{1\}\{\|y\_\{i\}\|\}\\sum\_\{t=1\}^\{\|y\_\{i\}\|\}\\min\\left\(r\_\{i,t\}\(\\bm\{\\theta\}\)\{A\}\_\{i,t\},\\right\.\\right\.clip\(ri,t\(𝜽\),1−ε,1\+ε\)Ai,t\)\],\\displaystyle\\left\.\\left\.\\operatorname\{clip\}\(r\_\{i,t\}\(\\bm\{\\theta\}\),1\-\\varepsilon,1\+\\varepsilon\)\{A\}\_\{i,t\}\\right\)\\right\],whereri,t​\(𝜽\)=π𝜽​\(yi,t\|x,yi,<t\)π𝜽old​\(yi,t\|x,yi,<t\)r\_\{i,t\}\(\\bm\{\\theta\}\)=\\frac\{\\pi\_\{\\bm\{\\theta\}\}\(y\_\{i,t\}\|x,y\_\{i,<t\}\)\}\{\\pi\_\{\\bm\{\\theta\}\_\{\\text\{old\}\}\}\(y\_\{i,t\}\|x,y\_\{i,<t\}\)\}, the advantage is given byAi,t=r​\(𝒙,𝒚\(j\)\)−mean​\(\{r​\(𝒙,𝒚\(j\)\)\}j=1G\)std​\(\{r​\(𝒙,𝒚\(j\)\)\}j=1G\)\{A\}\_\{i,t\}=\\frac\{r\(\\bm\{x\},\\bm\{y\}^\{\(j\)\}\)\-\\mathrm\{mean\}\\left\(\\left\\\{r\(\\bm\{x\},\\bm\{y\}^\{\(j\)\}\)\\right\\\}\_\{j=1\}^\{G\}\\right\)\}\{\\mathrm\{std\}\\left\(\\left\\\{r\(\\bm\{x\},\\bm\{y\}^\{\(j\)\}\)\\right\\\}\_\{j=1\}^\{G\}\\right\)\}, andε\\varepsilonis a clipping hyperparameter\.

### 3\.2On\-Policy Distillation

The main idea of OPD is to let the student generate its own trajectories, and then minimize the reverse KL divergence between the student and the teacher on those student\-generated trajectories:

𝒥OPD​\(𝜽\)\\displaystyle\\mathcal\{J\}\_\{\\mathrm\{OPD\}\}\(\\bm\{\\theta\}\)=min𝜽⁡𝔼𝒙∼D,𝒚∼π𝜽\(⋅∣𝒙\)\\displaystyle=\\min\_\{\\bm\{\\theta\}\}\\,\\mathbb\{E\}\_\{\\bm\{x\}\\sim D,\\,\\bm\{y\}\\sim\\pi\_\{\\bm\{\\theta\}\}\(\\cdot\\mid\\bm\{x\}\)\}\(4\)\[𝒟KL\(π𝜽\(𝒚∣𝒙\)∥π∗\(𝒚∣𝒙\)\)\]\.\\displaystyle\\quad\\left\[\\mathcal\{D\}\_\{\\mathrm\{KL\}\}\\\!\\left\(\\pi\_\{\\bm\{\\theta\}\}\(\\bm\{y\}\\mid\\bm\{x\}\)\\,\\\|\\,\\pi^\{\*\}\(\\bm\{y\}\\mid\\bm\{x\}\)\\right\)\\right\]\.
Then, we can get the gradient of OPD as:

∇𝜽𝒥OPD​\(𝜽\)=𝔼𝒙∼D,𝒚∼π𝜽\(⋅∣𝒙\)\\displaystyle\\nabla\_\{\\bm\{\\theta\}\}\\mathcal\{J\}\_\{\\mathrm\{OPD\}\}\(\\bm\{\\theta\}\)=\\mathbb\{E\}\_\{\\bm\{x\}\\sim D,\\,\\bm\{y\}\\sim\\pi\_\{\\bm\{\\theta\}\}\(\\cdot\\mid\\bm\{x\}\)\}\(5\)\[∑t=1T\(logπ𝜽\(yt∣𝒙,𝒚<t\)−logπ∗\(yt∣𝒙,𝒚<t\)\)\\displaystyle\\left\[\\sum\_\{t=1\}^\{T\}\\left\(\\log\\pi\_\{\\bm\{\\theta\}\}\\left\(y\_\{t\}\\mid\\bm\{x\},\\bm\{y\}\_\{<t\}\\right\)\-\\log\\pi^\{\*\}\\left\(y\_\{t\}\\mid\\bm\{x\},\\bm\{y\}\_\{<t\}\\right\)\\right\)\\right\.⋅∇𝜽logπ𝜽\(yt∣𝒙,𝒚<t\)\]\.\\displaystyle\\quad\\left\.\\cdot\\nabla\_\{\\bm\{\\theta\}\}\\log\\pi\_\{\\bm\{\\theta\}\}\\left\(y\_\{t\}\\mid\\bm\{x\},\\bm\{y\}\_\{<t\}\\right\)\\right\]\.
Eq\. \([17](https://arxiv.org/html/2608.00782#A1.E17)\) takes the same form as Eq\. \([2](https://arxiv.org/html/2608.00782#S3.E2)\), unifying OPD within the RL framework, whereAtOPD=−\(log⁡π𝜽​\(yt∣𝒙,𝒚<t\)−log⁡π∗​\(yt∣𝒙,𝒚<t\)\)A\_\{t\}^\{\\text\{OPD\}\}=\-\\left\(\\log\\pi\_\{\\bm\{\\theta\}\}\\left\(y\_\{t\}\\mid\\bm\{x\},\\bm\{y\}\_\{<t\}\\right\)\-\\log\\pi^\{\*\}\\left\(y\_\{t\}\\mid\\bm\{x\},\\bm\{y\}\_\{<t\}\\right\)\\right\)serves as the advantage in OPD, enabling token\-level credit assignment\. The detailed derivation is provided in Appendix[A\.1](https://arxiv.org/html/2608.00782#A1.SS1)\.

### 3\.3Supervised Fine\-Tuning

Supervised Fine\-Tuning \(SFT\) trains the student policyπ𝜽\\pi\_\{\\bm\{\\theta\}\}to mimic the teacherπ∗\\pi^\{\*\}using demonstration data\. From an information\-theoretic perspective, SFT minimizes the forward KL divergence between the teacher and the student, which contrasts with the reverse KL divergence used in OPD:

𝒥SFT​\(𝜽\)\\displaystyle\\mathcal\{J\}\_\{\\mathrm\{SFT\}\}\(\\bm\{\\theta\}\)=min𝜽⁡𝔼𝒙∼D\\displaystyle=\\min\_\{\\bm\{\\theta\}\}\\,\\mathbb\{E\}\_\{\\bm\{x\}\\sim D\}\(6\)\[𝒟KL\(π∗\(𝒚∣𝒙\)∥π𝜽\(𝒚∣𝒙\)\)\]\.\\displaystyle\\quad\\left\[\\mathcal\{D\}\_\{\\mathrm\{KL\}\}\\\!\\left\(\\pi^\{\*\}\(\\bm\{y\}\\mid\\bm\{x\}\)\\,\\\|\\,\\pi\_\{\\bm\{\\theta\}\}\(\\bm\{y\}\\mid\\bm\{x\}\)\\right\)\\right\]\.
Ignoring the entropy ofπ∗\\pi^\{\*\}\(independent of𝜽\\bm\{\\theta\}\), this is equivalent to maximizing the expected log\-likelihood of teacher trajectories:

max𝜽⁡𝔼𝒙∼D,𝒚∼π∗\(⋅∣𝒙\)​\[∑t=1Tlog⁡π𝜽​\(yt∣𝒙,𝒚<t\)\]\.\\max\_\{\\bm\{\\theta\}\}\\mathbb\{E\}\_\{\\bm\{x\}\\sim D,\\bm\{y\}\\sim\\pi^\{\*\}\(\\cdot\\mid\\bm\{x\}\)\}\\left\[\\sum\_\{t=1\}^\{T\}\\log\\pi\_\{\\bm\{\\theta\}\}\(y\_\{t\}\\mid\\bm\{x\},\\bm\{y\}\_\{<t\}\)\\right\]\.\(7\)
Taking the gradient with respect to𝜽\\bm\{\\theta\}yields:

∇𝜽𝒥SFT​\(𝜽\)\\displaystyle\\nabla\_\{\\bm\{\\theta\}\}\\mathcal\{J\}\_\{\\mathrm\{SFT\}\}\(\\bm\{\\theta\}\)=𝔼𝒙∼D,𝒚∼π∗\(⋅∣𝒙\)\\displaystyle=\\mathbb\{E\}\_\{\\bm\{x\}\\sim D,\\,\\bm\{y\}\\sim\\pi^\{\*\}\(\\cdot\\mid\\bm\{x\}\)\}\(8\)\[∑t=1T1⋅∇𝜽log⁡π𝜽​\(yt∣𝒙,𝒚<t\)\]\.\\displaystyle\\quad\\left\[\\sum\_\{t=1\}^\{T\}1\\cdot\\nabla\_\{\\bm\{\\theta\}\}\\log\\pi\_\{\\bm\{\\theta\}\}\\left\(y\_\{t\}\\mid\\bm\{x\},\\bm\{y\}\_\{<t\}\\right\)\\right\]\.
Comparing Eq\. \([8](https://arxiv.org/html/2608.00782#S3.E8)\) with the general policy gradient formulation in Eq\. \([2](https://arxiv.org/html/2608.00782#S3.E2)\), SFT can be elegantly unified within the RL framework as anoff\-policyalgorithm\(Zhuet al\.,[2026](https://arxiv.org/html/2608.00782#bib.bib36); Wuet al\.,[2025](https://arxiv.org/html/2608.00782#bib.bib37)\)\. Specifically, the trajectories𝒚\\bm\{y\}are sampled from the teacher policyπ∗\\pi^\{\*\}rather than the active student policyπ𝜽\\pi\_\{\\bm\{\\theta\}\}, and the advantage function is implicitly set to a constantAtSFT=1A\_\{t\}^\{\\mathrm\{SFT\}\}=1\. This implies that every token generated by the teacher is treated as a gold standard, receiving a uniform, positive credit\.

![Refer to caption](https://arxiv.org/html/2608.00782v1/x2.png)Figure 2:Illustration ofRSTGframework\.For negative zero\-variance prompts,RSTGis applied; otherwise, standard GRPO is used\.RSTGweights OPD advantages by the teacher’s confidence, restricts gradient updates to high\-quality tokens, and performs SFT on teacher\-generated reference answers\.

## 4Method

The framework ofRSTGis illustrated in Figure[2](https://arxiv.org/html/2608.00782#S3.F2)\. When a negative zero\-variance prompt is encountered, theRSTGbranch is activated; otherwise, the standard GRPO branch is followed\. WithinRSTG, we weight the OPD advantages according to the teacher’s confidence on the given prompt \(§[4\.1](https://arxiv.org/html/2608.00782#S4.SS1)\), restrict OPD gradient updates to high\-quality tokens \(§[4\.2](https://arxiv.org/html/2608.00782#S4.SS2)\), and perform SFT using reference answers generated by the teacher model \(§[4\.3](https://arxiv.org/html/2608.00782#S4.SS3)\)\.

### 4\.1Data Selection and Teacher\-Guided Advantage Weighting

To study the effects of data difficulty and teacher proficiency on OPD, we partition the full 57k\-sample dataset𝒟\\mathcal\{D\}into two nested subsets based on model performance: \(1\)𝒟sw\\mathcal\{D\}\_\{\\text\{sw\}\}\(9k samples\), where the student fails all 8 rollouts \(mean@8=0\\text\{mean@8\}=0\), and \(2\)𝒟swtr\\mathcal\{D\}\_\{\\text\{swtr\}\}\(2k samples\), a subset of𝒟sw\\mathcal\{D\}\_\{\\text\{sw\}\}where the teacher achieves a perfect success rate \(mean@8=1\\text\{mean@8\}=1\)\. We evaluate OPD across these three data scales on mathematical benchmarks \(Figure[3](https://arxiv.org/html/2608.00782#S4.F3)\)\. The results yield two key insights:

Finding 1\.Training on tasks challenging for the student \(𝒟sw\\mathcal\{D\}\_\{\\text\{sw\}\}\) outperforms training on the full dataset \(𝒟\\mathcal\{D\}\)\.

Finding 2\.Restricting training to samples where the teacher excels \(𝒟swtr\\mathcal\{D\}\_\{\\text\{swtr\}\}\) yields the best performance using only 3\.63% of𝒟\\mathcal\{D\}\.

![Refer to caption](https://arxiv.org/html/2608.00782v1/x3.png)Figure 3:Performance on the mathmatical benchmark for Qwen3\-1\.7B\-Instruct trained via OPD on different data partitions: Vanilla𝒟\\mathcal\{D\}, Student Wrong𝒟sw\\mathcal\{D\}\_\{\\text\{sw\}\}, Student Wrong and Teacher Right𝒟swtr\\mathcal\{D\}\_\{\\text\{swtr\}\}\.This indicates that the complementary effect of OPD on RL should be channeled specifically into this category of samples, which naturally correspond tonegative zero\-variance promptsin GRPO, where all rollouts fail and the advantage collapses to zero, leaving no training signal\. OPD effectively recovers these lost gradient signals by providing token\-level advantages\. Furthermore, we incorporate teacher confidence as a fine\-grained weighting mechanism into this hybrid objective\.

Formally, for sampleii, letωi∈\[0,1\]\\omega\_\{i\}\\in\[0,1\]denote the teacher’smean@8score, serving as a proxy for teacher confidence to dynamically scale the token\-level OPD advantage\. The hybrid advantage function is defined as:

Ai,tHybrid=\{β⋅ωi⋅Ai,tOPD,if​rj=0,∀j∈GAi,tGRPO,otherwise\{A\}\_\{i,t\}^\{\\text\{Hybrid\}\}=\\begin\{cases\}\\beta\\cdot\\omega\_\{i\}\\cdot\{A\}\_\{i,t\}^\{\\text\{OPD\}\},&\\text\{if \}r\_\{j\}=0,\\ \\forall j\\in G\\\\ \{\{A\}\}\_\{i,t\}^\{\\text\{GRPO\}\},&\\text\{otherwise\}\\end\{cases\}\(9\)
whereAi,tGRPO\{A\}\_\{i,t\}^\{\\text\{GRPO\}\}denotes the standard GRPO advantage,Ai,tOPD\{A\}\_\{i,t\}^\{\\text\{OPD\}\}the OPD advantage, andβ\\betaa scaling hyperparameter controlling the magnitude of the OPD signal\. The weighting coefficientωi\\omega\_\{i\}assigns greater optimization weight to tokens where the teacher is more confident, leading to more reliable policy optimization\.

### 4\.2Mitigating Premature Convergence via Token Selection

We select high\-quality tokens to slow the student’s convergence toward the teacher and reduce gradient noise\. Inspired by TIP\(Xuet al\.,[2026](https://arxiv.org/html/2608.00782#bib.bib1)\), we identify two categories of valuable tokens: \(1\)high\-entropy tokens, where the student is uncertain, typically representing critical reasoning junctures; and \(2\)tokens with large teacher\-student discrepancy, which tend to carry richer information\.

We define the student’s entropy at positionttas:

ht=−∑vpt,v​log⁡pt,v,h\_\{t\}=\-\\sum\_\{v\}p\_\{t,v\}\\log p\_\{t,v\},\(10\)wherept,vp\_\{t,v\}is the student’s predicted probability for vocabulary tokenvvat positiontt\. The teacher\-student discrepancy is defined as:

dt=\|Ai,tOPD\|\.d\_\{t\}=\|\{A\}\_\{i,t\}^\{\\text\{OPD\}\}\|\.\(11\)
Tokens with largehth\_\{t\}ordtd\_\{t\}are considered high\-value\. We combine these two criteria via the Soft\-OR formulation \(see Appendix[A\.2](https://arxiv.org/html/2608.00782#A1.SS2.SSS0.Px1)for details\)\. Given min\-max normalizedh^t,d^t∈\[0,1\]\\hat\{h\}\_\{t\},\\hat\{d\}\_\{t\}\\in\[0,1\], the selection score is:

st=h^t\+d^t−h^t⋅d^t\.s\_\{t\}=\\hat\{h\}\_\{t\}\+\\hat\{d\}\_\{t\}\-\\hat\{h\}\_\{t\}\\cdot\\hat\{d\}\_\{t\}\.\(12\)
The token\-masked OPD advantage is then defined as:

A^i,tOPD=\{Ai,tOPD,if​t∈𝒮k0,otherwise\\hat\{A\}\_\{i,t\}^\{\\text\{OPD\}\}=\\begin\{cases\}\{A\}\_\{i,t\}^\{\\text\{OPD\}\},&\\text\{if \}t\\in\\mathcal\{S\}\_\{k\}\\\\\[4\.0pt\] 0,&\\text\{otherwise\}\\end\{cases\}\(13\)where𝒮k\\mathcal\{S\}\_\{k\}denotes the top\-k%k\\%tokens ranked bysts\_\{t\}, enabling gradual distillation of the teacher’s capabilities while reducing gradient noise\.

### 4\.3Complementary Supervision on Negative Zero\-Variance Prompts

This advantage asymmetry is further exacerbated on negative zero\-variance prompts\. As illustrated in Figure[4](https://arxiv.org/html/2608.00782#S4.F4), the advantage values on negative zero\-variance prompts are consistently more negative than those on positive zero\-variance prompts throughout training, reflecting a severely suppressive optimization signal that penalizes the student without providing any constructive guidance toward correct solutions\.

![Refer to caption](https://arxiv.org/html/2608.00782v1/x4.png)Figure 4:OPD advantage curves during training on positive and negative zero\-variance prompts\.To mitigate this advantage asymmetry of OPD, we augment the training objective for negative zero\-variance prompts with an auxiliary SFT loss computed on correct trajectories pre\-generated out by the teacherπ∗\\pi^\{\*\}\. Formally, the final training objective is defined as:

𝒥Final​\(𝜽\)=\{𝒥GRPO​\(𝜽;β⋅ωi⋅A^i,tOPD\)\+β⋅𝒥SFT​\(𝜽\),if​rj=0,∀j∈G𝒥GRPO​\(𝜽;Ai,tGRPO\),otherwise\\mathcal\{J\}\_\{\\text\{Final\}\}\(\\bm\{\\theta\}\)=\\begin\{cases\}\\mathcal\{J\}\_\{\\text\{GRPO\}\}\\left\(\\bm\{\\theta\};\\beta\\cdot\\omega\_\{i\}\\cdot\\hat\{A\}\_\{i,t\}^\{\\text\{OPD\}\}\\right\)\\\\ \+\\beta\\cdot\\mathcal\{J\}\_\{\\text\{SFT\}\}\(\\bm\{\\theta\}\),\\quad\\text\{if \}r\_\{j\}=0,\\ \\forall j\\in G\\\\\[6\.0pt\] \\mathcal\{J\}\_\{\\text\{GRPO\}\}\\left\(\\bm\{\\theta\};\{\{A\}\}\_\{i,t\}^\{\\text\{GRPO\}\}\\right\),\\quad\\text\{otherwise\}\\end\{cases\}

\(14\)
where𝒥SFT\\mathcal\{J\}\_\{\\text\{SFT\}\}is computed exclusively on negative zero\-variance prompts, andβ\\betais shared with the OPD term in Eq\. \([9](https://arxiv.org/html/2608.00782#S4.E9)\), ensuring consistent regularization across both components\.

As established in Section[3\.3](https://arxiv.org/html/2608.00782#S3.SS3), this objective is equivalent to assigning a uniform positive advantageAtSFT=1A\_\{t\}^\{\\text\{SFT\}\}=1to every token in the teacher trajectory, which injects purely positive gradient signals into the optimization on prompts where RL yields no learning signal, thereby shifting the OPD advantage space toward the positive direction, alleviating the advantage asymmetry issue, while simultaneously introducing a global perspective signal and guiding the student toward correct solution trajectories demonstrated by the teacher\.

## 5Experiments

Table 1:Main results on MATH and CODE benchmarks across three Student\-Teacher model pairs\.Bestandsecond\-bestare highlighted\.![Refer to caption](https://arxiv.org/html/2608.00782v1/x5.png)

![Refer to caption](https://arxiv.org/html/2608.00782v1/x6.png)

![Refer to caption](https://arxiv.org/html/2608.00782v1/x7.png)

Figure 5:Performance on MATH500 for Qwen3\-1\.7B\-Instruct→\\rightarrowQwen3\-4B\-Instruct\-2507, Qwen3\-4B\-Instruct→\\rightarrowQwen3\-4B\-Instruct\-2507, and Qwen2\.5\-3B\-Instruct→\\rightarrowQwen2\.5\-14B\-Instruct from left to right, respectively\.### 5\.1Setup

#### Models

We employ three teacher\-student model pairs across two model families:

- •Qwen3\(Team,[2025b](https://arxiv.org/html/2608.00782#bib.bib14)\): Qwen3\-1\.7B\-Instruct→\\rightarrowQwen3\-4B\-Instruct\-2507 \(Pair ❶\); Qwen3\-4B\-Instruct→\\rightarrowQwen3\-4B\-Instruct\-2507 \(Pair ❷\)\.
- •Qwen2\.5\(Yanget al\.,[2024](https://arxiv.org/html/2608.00782#bib.bib29)\): Qwen2\.5\-3B\-Instruct→\\rightarrowQwen2\.5\-14B\-Instruct \(Pair ❸\)\.

#### Dataset and Metrics\.

For training, we filter the DeepMath\(Heet al\.,[2025](https://arxiv.org/html/2608.00782#bib.bib7)\)dataset to select 57K samples with difficulty≥6\\geq 6as math RL data, and use Eurus\-RL\-Code\(Cuiet al\.,[2025](https://arxiv.org/html/2608.00782#bib.bib8)\)\(25K samples\) as code RL data\. For evaluation, we use AIME 2024\(MAA,[2024](https://arxiv.org/html/2608.00782#bib.bib33)\), AIME 2025\(MAA,[2025](https://arxiv.org/html/2608.00782#bib.bib34)\), MATH\-500\(Gaoet al\.,[2025](https://arxiv.org/html/2608.00782#bib.bib30)\), and OLMPIAD Bench\(Heet al\.,[2024](https://arxiv.org/html/2608.00782#bib.bib31)\)for mathematical reasoning, and APPS\(Hendryckset al\.,[2021](https://arxiv.org/html/2608.00782#bib.bib12)\)and MBPP\+\(Liuet al\.,[2023](https://arxiv.org/html/2608.00782#bib.bib35)\)for code generation\. We set temperature to 1\.0, top\-p to 1\.0, and maximum generation length to 8,192, sampling 16 solutions per math problem and 4 per code problem\. Further details are in Appendix[A\.3](https://arxiv.org/html/2608.00782#A1.SS3)\.

#### Implementation Details\.

We set the batch size to 256, maximum response length to 8192, number of rollouts to 8, and learning rate to1×10−61\\times 10^\{\-6\}, with thinking mode disabled for all models\. For mathematics, training runs for 550 steps \(222 steps/epoch\); for code, 400 steps \(98 steps/epoch\), both reaching convergence\. The coefficientβ\\betain Eq\.[14](https://arxiv.org/html/2608.00782#S4.E14)is linearly annealed withβinit=5×10−3\\beta\_\{\\text\{init\}\}=5\\times 10^\{\-3\},δ=5×10−5\\delta=5\\times 10^\{\-5\}, andβmin=1×10−3\\beta\_\{\\text\{min\}\}=1\\times 10^\{\-3\}\. To construct the SFT dataset, we pre\-samplen=8n=8responses per prompt from the teacher, retaining the shortest correct response as reference\. More details and cost analysis are provided in Appendix[A\.5](https://arxiv.org/html/2608.00782#A1.SS5)\.

#### Baselines

We compare our method against five baselines: GRPO, OPD,GRPO\+OPD, which directly combines the GRPO and OPD losses following the same configuration as KDRL\(Xuet al\.,[2025](https://arxiv.org/html/2608.00782#bib.bib11)\)\. See Appendix[A\.4](https://arxiv.org/html/2608.00782#A1.SS4)for more details\. Since our method specifically targets negative zero\-variance prompts, we also include two representative baselines from the line of work on learning from negatives:ReLIFT\(Maet al\.,[2025](https://arxiv.org/html/2608.00782#bib.bib9)\), which applies SFT using ground\-truth answers on negative zero\-variance prompts, andRL\-ZVP\(Leet al\.,[2025](https://arxiv.org/html/2608.00782#bib.bib10)\), which designs an asymmetric advantage formulation based on token\-level entropy for both positive and negative zero\-variance prompts to extract effective learning signals\. As our primary objective is to push the upper bound of reinforcement learning, all baselines above are built upon GRPO, with the exception of OPD\.

### 5\.2Main Results

Table[1](https://arxiv.org/html/2608.00782#S5.T1)presents the performance of all methods across four mathematical reasoning benchmarks and two code generation benchmarks\.

For mathematics, naive GRPO\+OPD tends to slightly underperform standard GRPO, with changes of−0\.2%\-0\.2\\%,−1\.57%\-1\.57\\%, and\+0\.36%\+0\.36\\%across the three model pairs\. In contrast,RSTGachieves substantial improvements over naive GRPO\+OPD, with gains of\+4\.02%\+4\.02\\%,\+3\.52%\+3\.52\\%, and\+0\.89%\+0\.89\\%respectively, surpassing nearly all baselines\.

For code generation, naive GRPO\+OPD yields notable improvements over standard GRPO, with gains of\+4\.52%\+4\.52\\%,\+7\.44%\+7\.44\\%, and\+9\.64%\+9\.64\\%across the three model pairs\. Nevertheless,RSTGachieves consistent further improvements over naive GRPO\+OPD, with gains of\+2\.56%\+2\.56\\%,\+3\.05%\+3\.05\\%, and\+1\.97%\+1\.97\\%across the three model pairs, surpassing almost all baselines\.

Additionally, we observe that ReLIFT achieves relatively strong performance among the baselines, as directly supervising the model with ground\-truth answers is effective for inherently challenging negative zero\-variance prompts\. However, it is confined to offline imitation learning\. OPD complements this by providing online corrective signals at every token generation step, and with our carefully designed components \(§[5\.5](https://arxiv.org/html/2608.00782#S5.SS5.SSS0.Px2)\), the two objectives are organically integrated to reinforce each other, leading to more effective learning\.

Figure[5](https://arxiv.org/html/2608.00782#S5.F5)illustrates the training dynamics of all methods on MATH500 as a representative benchmark, whereRSTGoutperforms nearly all baselines at every training step\. Notably, for Model Pair 2, where both the teacher and student are 4B models with a relatively small capability gap, the performance gains are less pronounced compared to the other two model pairs\. More complete training curves are provided in the Appendix[A\.6](https://arxiv.org/html/2608.00782#A1.SS6)\.

### 5\.3The Asymmetry of Advantage

We compared the evolution of advantages during training betweenRSTGand naive GRPO\+OPD\. As shown in Figure[6](https://arxiv.org/html/2608.00782#S5.F6),RSTGyields higher advantage values and alleviates the advantage asymmetry introduced by OPD, thereby allowing more tokens to receive positive learning signals\.

![Refer to caption](https://arxiv.org/html/2608.00782v1/x8.png)Figure 6:Comparison of advantage values during training betweenRSTGand naive GRPO\+OPD, using model pair 2 as a representative example\.
### 5\.4Convergence Speed and Response Length

#### Convergence Speed\.

We use the overlapMoverlapM\_\{\\text\{overlap\}\}between the top\-kktokens \(k=16k=16\) of the teacher and the student\(Liet al\.,[2026b](https://arxiv.org/html/2608.00782#bib.bib13)\)to quantify the alignment between their candidate spaces, where a higher value indicates that the student more closely fits the teacher\. We monitor this metric to track the rate at which the student converges to the teacher\. At step 200, standard OPD reaches69\.7%69\.7\\%, naive GRPO\+OPD reaches67\.6%67\.6\\%, whileRSTGachieves only65\.81%65\.81\\%, demonstrating thatRSTGslows convergence toward the teacher and preserves a larger exploration space for RL\.

#### Response Length\.

OPD suffers from abrupt length inflation\(Luoet al\.,[2026](https://arxiv.org/html/2608.00782#bib.bib28); Fuet al\.,[2026](https://arxiv.org/html/2608.00782#bib.bib2)\), as illustrated in Figure[7](https://arxiv.org/html/2608.00782#S5.F7)\. OurRSTGeffectively mitigates this issue, maintaining response length comparable to that of GRPO\.

![Refer to caption](https://arxiv.org/html/2608.00782v1/x9.png)Figure 7:Comparison of response length during training betweenRSTGand OPD\.![Refer to caption](https://arxiv.org/html/2608.00782v1/x10.png)Figure 8:Performance on mathematical benchmarks as components ofRSTGare progressively added\. MATH500 is excluded from the figure due to its larger score range, but is included in the AVG computation\.Table 2:Average accuracy on mathematical benchmarks under differentβ\\betastrategies\. Best values arebolded\.

### 5\.5Ablation Studies

#### Each component plays a crucial role\.

Taking model pair1 as an example, we start from a baseline that naively combines the GRPO and OPD losses over all samples, and progressively incorporate each component of ourRSTG\. As shown in Figure[8](https://arxiv.org/html/2608.00782#S5.F8), each component contributes meaningfully to the overall performance\. Detailed results are provided in the Appendix[A\.7](https://arxiv.org/html/2608.00782#A1.SS7)\.

#### The careful design of OPD is indispensable\.

When the carefully designed OPD components are removed fromRSTG, specifically teacher\-guided advantage weighting and token selection, the average accuracy on the math benchmarks for Pair 1 drops to 51\.74%, which is even lower than applying SFT on top of GRPO alone \(i\.e\., ReLIFT in Table[1](https://arxiv.org/html/2608.00782#S5.T1), 52\.52%\)\. This outcome is expected: without these carefully designed components, OPD fails to operate on the appropriate samples, and the absence of token selection causes the model to converge prematurely toward the teacher distribution, thereby undermining its exploratory capacity during RL training\. This demonstrates that our OPD design choices are indispensable\. Only with these designs in place can OPD and SFT be effectively integrated, enabling the model to simultaneously imitate the teacher’s correct responses via SFT and receive fine\-grained corrective signals at every token generation step via OPD, thus achieving more effective and efficient training\. Without such elaborate design, OPD may even exert a detrimental effect on overall performance\.

#### The coefficientβ\\betaof OPD and SFT\.

We investigate two strategies forβ\\beta: constant and linear annealing\. For the constant strategy, we evaluate several fixed values ofβ\\beta; for the linear annealing strategy, we evaluate several different initial valuesβinit\\beta\_\{\\text\{init\}\}\. In both cases, OPD is applied solely on negative zero\-variance prompts without any additional components\. Results are reported in Table[2](https://arxiv.org/html/2608.00782#S5.T2)\.

A constantβ\\betaconsistently degrades performance, as the performance upper bound of OPD is lower than that of GRPO\. A largeβ\\betacauses the model to overfit to the teacher and converge prematurely to this lower bound\. The linear annealing strategy suffers from the same issue when the initial value is too large, while too small an initial value results in a distillation signal too weak to be effective\. We therefore adoptedβinit=5×10−3\\beta\_\{\\text\{init\}\}=5\\times 10^\{\-3\}as our final choice\. Since the SFT intensity is aligned with that of OPD, both terms share the same coefficientβ\\beta\.

## 6Conclusion

To address the performance degradation caused by the naive combination of GRPO and OPD, we proposeRSTG, which selectively applies OPD on difficult samples that the student has not yet mastered, weighted by the teacher’s degree of proficiency\. Furthermore,RSTGemploys fine\-grained token selection to slow down convergence and reduce gradient noise, and incorporates an auxiliary SFT objective to inject positive gradient signals and provide a global perspective on the solution space\. Experiments across benchmarks in two domains and three model pairs confirm consistent improvements over both standard GRPO and naive GRPO\+OPD baselines\.

## Limitations

We conduct experiments across two domains, math and code, which demonstrate the generalizability of our method\. Nevertheless, we look forward to exploring its applicability in other settings, such as agentic tasks\. Due to computational constraints and the strict requirement of OPD for a meaningful capability gap between the teacher and student, our experiments are conducted at the largest scale feasible within our resources\. We expectRSTGto remain effective at larger model scales, and leave this for future work\.

## References

- On\-Policy Distillation of Language Models: Learning from Self\-Generated Mistakes\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=3zKtaqxLhW)Cited by:[§1](https://arxiv.org/html/2608.00782#S1.p2.1),[§2](https://arxiv.org/html/2608.00782#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2608.00782#S2.SS0.SSS0.Px2.p1.1)\.
- G\. Cui, L\. Yuan, Z\. Wang, H\. Wang, W\. Li, B\. He, Y\. Fan, T\. Yu, Q\. Xu, W\. Chen, J\. Yuan, H\. Chen, K\. Zhang, X\. Lv, S\. Wang, Y\. Yao, X\. Han, H\. Peng, Y\. Cheng, Z\. Liu, M\. Sun, B\. Zhou, and N\. Ding \(2025\)Process Reinforcement through Implicit Rewards\.CoRRabs/2502\.01456\.External Links:[Link](https://doi.org/10.48550/arXiv.2502.01456),[Document](https://dx.doi.org/10.48550/ARXIV.2502.01456),2502\.01456Cited by:[§5\.1](https://arxiv.org/html/2608.00782#S5.SS1.SSS0.Px2.p1.1)\.
- DeepSeek\-AI \(2025\)DeepSeek\-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning\.CoRRabs/2501\.12948\.External Links:[Link](https://doi.org/10.48550/arXiv.2501.12948),[Document](https://dx.doi.org/10.48550/ARXIV.2501.12948),2501\.12948Cited by:[§1](https://arxiv.org/html/2608.00782#S1.p1.1)\.
- Y\. Feng, P\. Jain, A\. Hartshorn, Y\. Duan, and J\. Kempe \(2025\)Don’t Waste Mistakes: Leveraging Negative RL\-Groups via Confidence Reweighting\.CoRRabs/2510\.08696\.External Links:[Link](https://doi.org/10.48550/arXiv.2510.08696),[Document](https://dx.doi.org/10.48550/ARXIV.2510.08696),2510\.08696Cited by:[§1](https://arxiv.org/html/2608.00782#S1.p1.1)\.
- Y\. Fu, H\. Huang, K\. Jiang, J\. Liu, Z\. Jiang, Y\. Zhu, and D\. Zhao \(2026\)Revisiting on\-policy distillation: empirical failure modes and simple fixes\.arXiv preprint arXiv:2603\.25562\.Cited by:[§1](https://arxiv.org/html/2608.00782#S1.p3.1),[§5\.4](https://arxiv.org/html/2608.00782#S5.SS4.SSS0.Px2.p1.1)\.
- B\. Gao, F\. Song, Z\. Yang, Z\. Cai, Y\. Miao, Q\. Dong, L\. Li, C\. Ma, L\. Chen, R\. Xu, Z\. Tang, B\. Wang, D\. Zan, S\. Quan, G\. Zhang, L\. Sha, Y\. Zhang, X\. Ren, T\. Liu, and B\. Chang \(2025\)Omni\-MATH: A Universal Olympiad Level Mathematic Benchmark for Large Language Models\.InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24\-28, 2025,External Links:[Link](https://openreview.net/forum?id=yaqPf0KAlN)Cited by:[§5\.1](https://arxiv.org/html/2608.00782#S5.SS1.SSS0.Px2.p1.1)\.
- Y\. Gu, L\. Dong, F\. Wei, and M\. Huang \(2024\)MiniLLM: Knowledge Distillation of Large Language Models\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=5h0qf7IBZZ)Cited by:[§1](https://arxiv.org/html/2608.00782#S1.p2.1),[§2](https://arxiv.org/html/2608.00782#S2.SS0.SSS0.Px1.p1.1)\.
- C\. He, R\. Luo, Y\. Bai, S\. Hu, Z\. L\. Thai, J\. Shen, J\. Hu, X\. Han, Y\. Huang, Y\. Zhang, J\. Liu, L\. Qi, Z\. Liu, and M\. Sun \(2024\)OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad\-Level Bilingual Multimodal Scientific Problems\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2024, Bangkok, Thailand, August 11\-16, 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),pp\. 3828–3850\.External Links:[Link](https://doi.org/10.18653/v1/2024.acl-long.211),[Document](https://dx.doi.org/10.18653/V1/2024.ACL-LONG.211)Cited by:[§5\.1](https://arxiv.org/html/2608.00782#S5.SS1.SSS0.Px2.p1.1)\.
- Z\. He, T\. Liang, J\. Xu, Q\. Liu, X\. Chen, Y\. Wang, L\. Song, D\. Yu, Z\. Liang, W\. Wang, Z\. Zhang, R\. Wang, Z\. Tu, H\. Mi, and D\. Yu \(2025\)DeepMath\-103K: A Large\-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning\.CoRRabs/2504\.11456\.External Links:[Link](https://doi.org/10.48550/arXiv.2504.11456),[Document](https://dx.doi.org/10.48550/ARXIV.2504.11456),2504\.11456Cited by:[§5\.1](https://arxiv.org/html/2608.00782#S5.SS1.SSS0.Px2.p1.1)\.
- D\. Hendrycks, S\. Basart, S\. Kadavath, M\. Mazeika, A\. Arora, E\. Guo, C\. Burns, S\. Puranik, H\. He, D\. Song, and J\. Steinhardt \(2021\)Measuring Coding Challenge Competence With APPS\.InProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual,J\. Vanschoren and S\. Yeung \(Eds\.\),External Links:[Link](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/c24cd76e1ce41366a4bbe8a49b02a028-Abstract-round2.html)Cited by:[§A\.3](https://arxiv.org/html/2608.00782#A1.SS3.p1.1),[§5\.1](https://arxiv.org/html/2608.00782#S5.SS1.SSS0.Px2.p1.1)\.
- G\. E\. Hinton, O\. Vinyals, and J\. Dean \(2015\)Distilling the knowledge in a neural network\.CoRRabs/1503\.02531\.External Links:[Link](http://arxiv.org/abs/1503.02531),1503\.02531Cited by:[§2](https://arxiv.org/html/2608.00782#S2.SS0.SSS0.Px1.p1.1)\.
- N\. Jia, H\. Yang, X\. Ma, J\. Lian, S\. Zhang, W\. Zhang, K\. Zeng, X\. Cai, and Z\. Sun \(2026\)Asymmetric on\-policy distillation: bridging exploitation and imitation at the token level\.arXiv preprint arXiv:2605\.06387\.Cited by:[§1](https://arxiv.org/html/2608.00782#S1.p3.1)\.
- J\. Ko, S\. Abdali, Y\. J\. Kim, T\. Chen, and P\. Cameron \(2026\)Scaling Reasoning Efficiently via Relaxed On\-Policy Distillation\.CoRRabs/2603\.11137\.External Links:[Link](https://doi.org/10.48550/arXiv.2603.11137),[Document](https://dx.doi.org/10.48550/ARXIV.2603.11137),2603\.11137Cited by:[§1](https://arxiv.org/html/2608.00782#S1.p3.1),[§2](https://arxiv.org/html/2608.00782#S2.SS0.SSS0.Px1.p1.1)\.
- T\. V\. Le, M\. Jeon, K\. Vu, V\. Lai, and E\. Yang \(2025\)No Prompt Left Behind: Exploiting Zero\-Variance Prompts in LLM Reinforcement Learning via Entropy\-Guided Advantage Shaping\.CoRRabs/2509\.21880\.External Links:[Link](https://doi.org/10.48550/arXiv.2509.21880),[Document](https://dx.doi.org/10.48550/ARXIV.2509.21880),2509\.21880Cited by:[§1](https://arxiv.org/html/2608.00782#S1.p4.1),[§5\.1](https://arxiv.org/html/2608.00782#S5.SS1.SSS0.Px4.p1.1)\.
- G\. Li, T\. Yang, J\. Fang, M\. Song, M\. Zheng, H\. Guo, D\. Zhang, J\. Wang, and T\. Chua \(2026a\)Unifying Group\-Relative and Self\-Distillation Policy Optimization via Sample Routing\.CoRRabs/2604\.02288\.External Links:[Link](https://doi.org/10.48550/arXiv.2604.02288),[Document](https://dx.doi.org/10.48550/ARXIV.2604.02288),2604\.02288Cited by:[§2](https://arxiv.org/html/2608.00782#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Li, Y\. Zuo, B\. He, J\. Zhang, C\. Xiao, C\. Qian, T\. Yu, H\. Gao, W\. Yang, Z\. Liu, and N\. Ding \(2026b\)Rethinking On\-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe\.CoRRabs/2604\.13016\.External Links:[Link](https://doi.org/10.48550/arXiv.2604.13016),[Document](https://dx.doi.org/10.48550/ARXIV.2604.13016),2604\.13016Cited by:[§5\.4](https://arxiv.org/html/2608.00782#S5.SS4.SSS0.Px1.p1.6)\.
- Z\. Li, L\. Kang, F\. Xiao, L\. Xing, Q\. Si, Z\. Li, W\. Gong, D\. Yang, Y\. Xiao, and H\. Guo \(2026c\)Outcome\-Grounded Advantage Reshaping for Fine\-Grained Credit Assignment in Mathematical Reasoning\.CoRRabs/2601\.07408\.External Links:[Link](https://doi.org/10.48550/arXiv.2601.07408),[Document](https://dx.doi.org/10.48550/ARXIV.2601.07408),2601\.07408Cited by:[§1](https://arxiv.org/html/2608.00782#S1.p1.1)\.
- J\. Liu, C\. S\. Xia, Y\. Wang, and L\. Zhang \(2023\)Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation\.InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 \- 16, 2023,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2023/hash/43e9d647ccd3e4b7b5baab53f0368686-Abstract-Conference.html)Cited by:[§5\.1](https://arxiv.org/html/2608.00782#S5.SS1.SSS0.Px2.p1.1)\.
- Z\. Lu, Z\. Yao, Z\. Han, Z\. Wang, J\. Wu, Q\. Gu, X\. Cai, W\. Lu, J\. Xiao, Y\. Zhuang,et al\.\(2026\)Self\-Distilled Agentic Reinforcement Learning\.arXiv preprint arXiv:2605\.15155\.Cited by:[§1](https://arxiv.org/html/2608.00782#S1.p3.1),[§2](https://arxiv.org/html/2608.00782#S2.SS0.SSS0.Px2.p1.1)\.
- F\. Luo, Y\. Chuang, G\. Wang, Z\. Xu, X\. Han, T\. Zhang, and V\. Braverman \(2026\)Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models\.CoRRabs/2604\.08527\.External Links:[Link](https://doi.org/10.48550/arXiv.2604.08527),[Document](https://dx.doi.org/10.48550/ARXIV.2604.08527),2604\.08527Cited by:[§5\.4](https://arxiv.org/html/2608.00782#S5.SS4.SSS0.Px2.p1.1)\.
- L\. Ma, H\. Liang, M\. Qiang, L\. Tang, X\. Ma, Z\. H\. Wong, J\. Niu, C\. Shen, R\. He, B\. Cui, and W\. Zhang \(2025\)Learning What Reinforcement Learning Can’t: Interleaved Online Fine\-Tuning for Hardest Questions\.CoRRabs/2506\.07527\.External Links:[Link](https://doi.org/10.48550/arXiv.2506.07527),[Document](https://dx.doi.org/10.48550/ARXIV.2506.07527),2506\.07527Cited by:[§5\.1](https://arxiv.org/html/2608.00782#S5.SS1.SSS0.Px4.p1.1)\.
- MAA \(2024\)American invitational mathematics examination \- aime 2024\.\.Cited by:[§5\.1](https://arxiv.org/html/2608.00782#S5.SS1.SSS0.Px2.p1.1)\.
- MAA \(2025\)American invitational mathematics examination \- aime 2025\.\.Cited by:[§5\.1](https://arxiv.org/html/2608.00782#S5.SS1.SSS0.Px2.p1.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. Guo \(2024\)DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models\.CoRRabs/2402\.03300\.External Links:[Link](https://doi.org/10.48550/arXiv.2402.03300),[Document](https://dx.doi.org/10.48550/ARXIV.2402.03300),2402\.03300Cited by:[§1](https://arxiv.org/html/2608.00782#S1.p1.1)\.
- G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. Wu \(2025\)HybridFlow: A Flexible and Efficient RLHF Framework\.InProceedings of the Twentieth European Conference on Computer Systems, EuroSys 2025, Rotterdam, The Netherlands, 30 March 2025 \- 3 April 2025,pp\. 1279–1297\.External Links:[Link](https://doi.org/10.1145/3689031.3696075),[Document](https://dx.doi.org/10.1145/3689031.3696075)Cited by:[§A\.5](https://arxiv.org/html/2608.00782#A1.SS5.p1.3)\.
- G\. Team \(2025a\)Gemini 2\.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities\.CoRRabs/2507\.06261\.External Links:[Link](https://doi.org/10.48550/arXiv.2507.06261),[Document](https://dx.doi.org/10.48550/ARXIV.2507.06261),2507\.06261Cited by:[§1](https://arxiv.org/html/2608.00782#S1.p1.1)\.
- Q\. Team \(2025b\)Qwen3 Technical Report\.CoRRabs/2505\.09388\.External Links:[Link](https://doi.org/10.48550/arXiv.2505.09388),[Document](https://dx.doi.org/10.48550/ARXIV.2505.09388),2505\.09388Cited by:[§1](https://arxiv.org/html/2608.00782#S1.p1.1),[1st item](https://arxiv.org/html/2608.00782#S5.I1.i1.p1.2)\.
- Y\. Wu, Y\. Zhou, Z\. Ziheng, Y\. Peng, X\. Ye, X\. Hu, W\. Zhu, L\. Qi, M\. Yang, and X\. Yang \(2025\)On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification\.CoRRabs/2508\.05629\.External Links:[Link](https://doi.org/10.48550/arXiv.2508.05629),[Document](https://dx.doi.org/10.48550/ARXIV.2508.05629),2508\.05629Cited by:[§3\.3](https://arxiv.org/html/2608.00782#S3.SS3.p7.4)\.
- H\. Xu, Q\. Zhu, H\. Deng, J\. Li, L\. Hou, Y\. Wang, L\. Shang, R\. Xu, and F\. Mi \(2025\)KDRL: Post\-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning\.CoRRabs/2506\.02208\.External Links:[Link](https://doi.org/10.48550/arXiv.2506.02208),[Document](https://dx.doi.org/10.48550/ARXIV.2506.02208),2506\.02208Cited by:[§A\.4](https://arxiv.org/html/2608.00782#A1.SS4.p1.1),[§1](https://arxiv.org/html/2608.00782#S1.p2.1),[§2](https://arxiv.org/html/2608.00782#S2.SS0.SSS0.Px2.p1.1),[§5\.1](https://arxiv.org/html/2608.00782#S5.SS1.SSS0.Px4.p1.1)\.
- Y\. Xu, H\. Sang, Z\. Zhou, R\. He, Z\. Wang, and A\. Geramifard \(2026\)Tip: token importance in on\-policy distillation\.arXiv preprint arXiv:2604\.14084\.Cited by:[§4\.2](https://arxiv.org/html/2608.00782#S4.SS2.p1.1)\.
- A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Qiu \(2024\)Qwen2\.5 Technical Report\.CoRRabs/2412\.15115\.External Links:[Link](https://doi.org/10.48550/arXiv.2412.15115),[Document](https://dx.doi.org/10.48550/ARXIV.2412.15115),2412\.15115Cited by:[2nd item](https://arxiv.org/html/2608.00782#S5.I1.i2.p1.1)\.
- C\. Yang, C\. Qin, Q\. Si, M\. Chen, N\. Gu, D\. Yao, Z\. Lin, W\. Wang, J\. Wang, and N\. Duan \(2026a\)Self\-Distilled RLVR\.CoRRabs/2604\.03128\.External Links:[Link](https://doi.org/10.48550/arXiv.2604.03128),[Document](https://dx.doi.org/10.48550/ARXIV.2604.03128),2604\.03128Cited by:[§2](https://arxiv.org/html/2608.00782#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2608.00782#S2.SS0.SSS0.Px2.p1.1)\.
- W\. Yang, W\. Liu, R\. Xie, K\. Yang, S\. Yang, and Y\. Lin \(2026b\)Learning beyond Teacher: Generalized On\-Policy Distillation with Reward Extrapolation\.CoRRabs/2602\.12125\.External Links:[Link](https://doi.org/10.48550/arXiv.2602.12125),[Document](https://dx.doi.org/10.48550/ARXIV.2602.12125),2602\.12125Cited by:[§2](https://arxiv.org/html/2608.00782#S2.SS0.SSS0.Px1.p1.1)\.
- H\. Zheng, Y\. Zhou, B\. R\. Bartoldson, B\. Kailkhura, F\. Lai, J\. Zhao, and B\. Chen \(2025\)Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts\.CoRRabs/2506\.02177\.External Links:[Link](https://doi.org/10.48550/arXiv.2506.02177),[Document](https://dx.doi.org/10.48550/ARXIV.2506.02177),2506\.02177Cited by:[§1](https://arxiv.org/html/2608.00782#S1.p1.1),[§1](https://arxiv.org/html/2608.00782#S1.p4.1)\.
- W\. Zhu, R\. Xie, R\. Wang, and P\. Liu \(2026\)Hybrid Policy Distillation for LLMs\.CoRRabs/2604\.20244\.External Links:[Link](https://doi.org/10.48550/arXiv.2604.20244),[Document](https://dx.doi.org/10.48550/ARXIV.2604.20244),2604\.20244Cited by:[§3\.3](https://arxiv.org/html/2608.00782#S3.SS3.p7.4)\.

## Appendix AAppendix

### A\.1Preliminaries

The main idea of OPD is to let the student generate its own trajectories, and then minimize the reverse KL divergence between the student and the teacher on those student\-generated trajectories:

𝒥OPD​\(𝜽\)\\displaystyle\\mathcal\{J\}\_\{\\mathrm\{OPD\}\}\(\\bm\{\\theta\}\)=min𝜽⁡𝔼𝒙∼D,𝒚∼π𝜽\(⋅∣𝒙\)\\displaystyle=\\min\_\{\\bm\{\\theta\}\}\\,\\mathbb\{E\}\_\{\\bm\{x\}\\sim D,\\,\\bm\{y\}\\sim\\pi\_\{\\bm\{\\theta\}\}\(\\cdot\\mid\\bm\{x\}\)\}\(15\)\[𝒟KL\(π𝜽\(𝒚∣𝒙\)∥π∗\(𝒚∣𝒙\)\)\]\.\\displaystyle\\quad\\left\[\\mathcal\{D\}\_\{\\mathrm\{KL\}\}\\\!\\left\(\\pi\_\{\\bm\{\\theta\}\}\(\\bm\{y\}\\mid\\bm\{x\}\)\\,\\\|\\,\\pi^\{\*\}\(\\bm\{y\}\\mid\\bm\{x\}\)\\right\)\\right\]\.
Expanding the KL divergence and applying the chain rule of probability, we have:

𝒥OPD​\(𝜽\)\\displaystyle\\mathcal\{J\}\_\{\\mathrm\{OPD\}\}\(\\bm\{\\theta\}\)=𝔼𝒙,𝒚∼π𝜽​\[log⁡π𝜽​\(𝒚∣𝒙\)π∗​\(𝒚∣𝒙\)\]\\displaystyle=\\mathbb\{E\}\_\{\\bm\{x\},\\,\\bm\{y\}\\sim\\pi\_\{\\bm\{\\theta\}\}\}\\left\[\\log\\frac\{\\pi\_\{\\bm\{\\theta\}\}\(\\bm\{y\}\\mid\\bm\{x\}\)\}\{\\pi^\{\*\}\(\\bm\{y\}\\mid\\bm\{x\}\)\}\\right\]\(16\)=𝔼𝒙,𝒚∼π𝜽​\[∑t=1Tlog⁡π𝜽​\(yt∣𝒙,𝒚<t\)π∗​\(yt∣𝒙,𝒚<t\)\]\.\\displaystyle=\\mathbb\{E\}\_\{\\bm\{x\},\\,\\bm\{y\}\\sim\\pi\_\{\\bm\{\\theta\}\}\}\\left\[\\sum\_\{t=1\}^\{T\}\\log\\frac\{\\pi\_\{\\bm\{\\theta\}\}\(y\_\{t\}\\mid\\bm\{x\},\\bm\{y\}\_\{<t\}\)\}\{\\pi^\{\*\}\(y\_\{t\}\\mid\\bm\{x\},\\bm\{y\}\_\{<t\}\)\}\\right\]\.
Taking the gradient with respect to𝜽\\bm\{\\theta\}and applying the log\-derivative trick∇𝜽𝔼𝒚∼π𝜽​\[f​\(𝒚\)\]=𝔼𝒚∼π𝜽​\[f​\(𝒚\)​∇𝜽log⁡π𝜽​\(𝒚\)\]\\nabla\_\{\\bm\{\\theta\}\}\\mathbb\{E\}\_\{\\bm\{y\}\\sim\\pi\_\{\\bm\{\\theta\}\}\}\[f\(\\bm\{y\}\)\]=\\mathbb\{E\}\_\{\\bm\{y\}\\sim\\pi\_\{\\bm\{\\theta\}\}\}\[f\(\\bm\{y\}\)\\nabla\_\{\\bm\{\\theta\}\}\\log\\pi\_\{\\bm\{\\theta\}\}\(\\bm\{y\}\)\]:

∇𝜽𝒥OPD​\(𝜽\)=𝔼𝒙∼D,𝒚∼π𝜽\(⋅∣𝒙\)\\displaystyle\\nabla\_\{\\bm\{\\theta\}\}\\mathcal\{J\}\_\{\\mathrm\{OPD\}\}\(\\bm\{\\theta\}\)=\\mathbb\{E\}\_\{\\bm\{x\}\\sim D,\\,\\bm\{y\}\\sim\\pi\_\{\\bm\{\\theta\}\}\(\\cdot\\mid\\bm\{x\}\)\}\(17\)\[∑t=1T\(logπ𝜽\(yt∣𝒙,𝒚<t\)−logπ∗\(yt∣𝒙,𝒚<t\)\)\\displaystyle\\left\[\\sum\_\{t=1\}^\{T\}\\left\(\\log\\pi\_\{\\bm\{\\theta\}\}\\left\(y\_\{t\}\\mid\\bm\{x\},\\bm\{y\}\_\{<t\}\\right\)\-\\log\\pi^\{\*\}\\left\(y\_\{t\}\\mid\\bm\{x\},\\bm\{y\}\_\{<t\}\\right\)\\right\)\\right\.⋅∇𝜽logπ𝜽\(yt∣𝒙,𝒚<t\)\]\.\\displaystyle\\quad\\left\.\\cdot\\nabla\_\{\\bm\{\\theta\}\}\\log\\pi\_\{\\bm\{\\theta\}\}\\left\(y\_\{t\}\\mid\\bm\{x\},\\bm\{y\}\_\{<t\}\\right\)\\right\]\.
Eq\. \([17](https://arxiv.org/html/2608.00782#A1.E17)\) takes the same form as Eq\. \([2](https://arxiv.org/html/2608.00782#S3.E2)\), unifying OPD within the RL framework, whereAtOPD=−\(log⁡π𝜽​\(yt∣𝒙,𝒚<t\)−log⁡π∗​\(yt∣𝒙,𝒚<t\)\)A\_\{t\}^\{\\text\{OPD\}\}=\-\\left\(\\log\\pi\_\{\\bm\{\\theta\}\}\\left\(y\_\{t\}\\mid\\bm\{x\},\\bm\{y\}\_\{<t\}\\right\)\-\\log\\pi^\{\*\}\\left\(y\_\{t\}\\mid\\bm\{x\},\\bm\{y\}\_\{<t\}\\right\)\\right\)serves as the advantage in OPD, enabling token\-level credit assignment\.

### A\.2Method

#### Mitigating Premature Convergence via Token Selection

Tokens with largehth\_\{t\}ordtd\_\{t\}are considered high\-value\. We combine these two criteria via the Soft\-OR formulation, which approximates the logical OR operation in a differentiable manner: if eitherh^t\\hat\{h\}\_\{t\}ord^t\\hat\{d\}\_\{t\}is large,sts\_\{t\}will be large, while avoiding double\-counting when both are large\. Given min\-max normalizedh^t,d^t∈\[0,1\]\\hat\{h\}\_\{t\},\\hat\{d\}\_\{t\}\\in\[0,1\], the selection score is:

st=h^t\+d^t−h^t⋅d^t\.s\_\{t\}=\\hat\{h\}\_\{t\}\+\\hat\{d\}\_\{t\}\-\\hat\{h\}\_\{t\}\\cdot\\hat\{d\}\_\{t\}\.\(18\)

### A\.3Experiment Details

The APPS dataset contains 10,000 samples in total\(Hendryckset al\.,[2021](https://arxiv.org/html/2608.00782#bib.bib12)\), spanning three difficulty levels: Introductory, Interview, and Competition\. To accelerate evaluation, we uniformly sample 500 instances as our test set, with an equal number of samples drawn from each difficulty level\.

### A\.4Baselines

For the configuration of naive GRPO\+OPD, we follow KDRL\(Xuet al\.,[2025](https://arxiv.org/html/2608.00782#bib.bib11)\), which considers three variants: \(1\) applying OPD on all samples, \(2\) applying OPD only on samples where the student fails, and \(3\) applying OPD only on negative zero\-variance prompts\. For each model pair, we evaluate all three variants and select the best\-performing one as the naive GRPO\+OPD baseline\. The OPD coefficient is kept consistent with our method\.

### A\.5Implementation Details\.

We implementRSTGbased on the VeRL framework\(Shenget al\.,[2025](https://arxiv.org/html/2608.00782#bib.bib39)\)\. For mathematics, we train for 550 steps on 8 A100 GPUs; the wall\-clock time for Qwen3\-1\.7B\-Instruct→\\rightarrowQwen3\-4B\-Instruct\-2507, Qwen3\-4B\-Instruct→\\rightarrowQwen3\-4B\-Instruct\-2507, and Qwen2\.5\-3B\-Instruct→\\rightarrowQwen2\.5\-14B\-Instruct is approximately 3, 4, and 2\.5 days, respectively\. For code, we train for 400 steps, taking approximately 4, 5, and 3 days for the three pairs, respectively\.

#### Cost Analysis

During GRPO training, taking Qwen2\.5\-3B\-Instruct→\\rightarrowQwen2\.5\-14B\-Instruct as an example, negative zero\-variance prompts account for approximately15\.6%15\.6\\%of steps initially, and this proportion decreases as the model improves\. SinceRSTGapplies OPD and SFT exclusively on negative zero\-variance prompts, the additional overhead over standard GRPO is minimal\. On 8 A100 GPUs for 550 training steps, the extra cost amounts to approximately 12 hours, which is acceptable\.

Furthermore, the SFT data is pre\-generated offline and incurs no cost during training\. This pre\-generation process is efficient: for the 57K training set with a single rollout per prompt, it takes only about 2 hours on 2 A100 GPUs\. Overall, the computational cost ofRSTGis fully manageable\.

Table 3:Ablation study results on mathematical reasoning benchmarks\.

### A\.6Training Dynamics

We select one model pair from each of the Qwen3 and Qwen2\.5 families and present their performance on mathematical benchmarks throughout training in Figures[9](https://arxiv.org/html/2608.00782#A1.F9)and[10](https://arxiv.org/html/2608.00782#A1.F10)\.

![Refer to caption](https://arxiv.org/html/2608.00782v1/x11.png)Figure 9:Performance of Qwen3\-1\.7B\-Instruct→\\rightarrowQwen3\-4B\-Instruct\-2507 on mathematical benchmarks throughout training, compared against standard GRPO\.![Refer to caption](https://arxiv.org/html/2608.00782v1/x12.png)Figure 10:Performance of Qwen2\.5\-3B\-Instruct→\\rightarrowQwen2\.5\-14B\-Instruct\-2507 on mathematical benchmarks throughout training, compared against standard GRPO\.
### A\.7Ablation Studies

Taking Qwen3\-1\.7B\-Instruct→\\rightarrowQwen3\-4B\-Instruct as an example, we start from a baseline that naively combines the GRPO and OPD losses over all samples, and progressively incorporate each component of ourRSTG\. Detailed results are provided in Table[3](https://arxiv.org/html/2608.00782#A1.T3), where each component is shown to contribute meaningfully to the overall performance\.