Qwen Tongyi Lab proposes RLCSD, a simple but important critique of on-policy self-distillation.

Their key observation is that the distillation signal often concentrates on stylistic tokens rather than task critical reasoning tokens. As a result, the model may learn differences in phrasing, brevity, or tone instead of the reasoning process that actually determines correctness.

RLCSD addresses this with contrastive supervision that focuses learning on task relevant tokens. A nice example of a hidden failure mode in a technique that many people are rapidly adopting.

RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation Paper: http://arxiv.org/abs/2606.11709

Reinforcement Learning with Contrastive On-Policy Self-Distillation

Source: https://arxiv.org/html/2606.11709 Leyi Pan1,211footnotemark:1,Shuchang Tao2,Yunpeng Zhai2,Lingzhe Zhang2,3,Zhaoyang Liu2, Bolin Ding2,Aiwei Liu1†,Lijie Wen1† 1Tsinghua University,2Tongyi Lab, Alibaba Group,3Peking University [email protected], [email protected], [email protected]

††footnotetext:∗This work is done during Leyi Pan’s internship at Tongyi Lab, Alibaba Group.†Corresponding authors.AbstractOn-policy self-distillation (OPSD) provides dense, token-level supervision for reasoning models by aligning a model’s own distribution with the distribution it produces under privileged context, typically a verified solution. However, we show that the learning signal drawn from this distributional gap concentrates on style tokens rather than task-bearing ones, as the hinted model tends to produce more direct, shorter outputs. We term this pathologyprivilege-induced style drift, which destabilizes training or causes response length to shrink. To address this, we proposeRLCSD(Reinforcement Learning with Contrastive on-policy Self-Distillation), which mitigates this drift by contrasting the teacher-student gap under a correct hint against that under a wrong hint, suppressing the style shift that conditioning on a hint tends to induce regardless of correctness, and yielding a signal that is more concentrated on task-bearing tokens. Experiments on Qwen3 (1.7B/4B/8B) and Olmo-3-7B-Think across mathematical and logical reasoning show that RLCSD consistently outperforms GRPO and prior OPSD methods. We further show that the contrastive principle is general: it plugs into existing OPSD methods to improve them, and its underlying insight extends to the broader cross-model on-policy distillation setting.

(a)Top-10 tokens by|ec||e_{c}|and|ectr||e_{\mathrm{ctr}}|. (b)Response length over training. (c)Performance comparison across benchmarks on Qwen3-8B.

Figure 1:RLCSD overview.(a) Motivation.Existing OPSD methods use|ec||e_{c}|as the optimization signal, which concentrates on style tokens, whereas RLCSD’s contrastive signal|ectr||e_{\mathrm{ctr}}|shifts the signal onto task-bearing tokens (e.g., mathematical content in the math reasoning task).(b) Response length over training.OPSD, SDPO, and SRPO suffer from training instability (response length explosion), while RLSD exhibits pronounced length shrinkage on math reasoning; RLCSD remains stable and preserves response length throughout.(c) Performance across benchmarks.RLCSD consistently outperforms GRPO and prior OPSD methods on both mathematical and logical reasoning.## 1Introduction

Reinforcement learning with verifiable rewards (RLVR), exemplified by GRPO(Shaoet al.,2024), has emerged as a dominant paradigm for training large reasoning models(Guoet al.,2025; Liuet al.,2024; Yanget al.,2025; Zenget al.,2026; Honget al.,2025). However, GRPO derives its entire learning signal from a single outcome-level reward at the end of each rollout, which becomes prohibitively sparse in long-horizon reasoning tasks where credit must be assigned across thousands of intermediate tokens. On-policy distillation (OPD)(Agarwalet al.,2024; Song and Zheng,2026; Guet al.,2024)addresses this limitation by drawing dense, token-level supervision from the logits of a stronger teacher along trajectories sampled by the student itself. The on-policy formulation mitigates the exposure bias inherent in standard supervised fine-tuning, while the refinement of supervision from the trajectory level to the token level substantially accelerates convergence. Recent studies indicate that OPD attains performance comparable to or exceeding that of RLVR, and the technique has been incorporated into the post-training pipelines of several flagship large language models, including Qwen3(Yanget al.,2025), GLM-5(Zenget al.,2026), the MiMo series(Xiaoet al.,2026), and DeepSeek-V4(DeepSeek-AI,2026).

The practical applicability of OPD, however, is constrained by several requirements. The method requires white-box access to the teacher’s token-level logits, precluding strong closed-source teachers; the teacher and student must share an identical vocabulary, a condition rarely satisfied across model families; and serving a separate, typically larger teacher alongside the student throughout RL training imposes substantial memory and latency overhead. On-policy self-distillation (OPSD)(Zhaoet al.,2026; Hübotteret al.,2026; Yanget al.,2026; Heet al.,2026; Liet al.,2026a; Jinet al.,2026; Kimet al.,2026; Shenet al.,2026a;b; Keet al.,2026; Wanget al.,2026)has recently emerged as a means of circumventing these obstacles. Under this paradigm, a single model assumes both roles, with the teacher granted access to privileged context (such as a verified reference solution) unavailable to the student.

Existing OPSD methods(Zhaoet al.,2026; Hübotteret al.,2026; Yanget al.,2026)derive token-level supervision fromec,t=log⁡πT​(yt∣x,yc∗,y<t)−log⁡πS​(yt∣x,y<t)e_{c,t}=\log\pi_{T}(y_{t}\mid x,y_{c}^{*},y_{<t})-\log\pi_{S}(y_{t}\mid x,y_{<t}), wherexxis the query,yyis a student-sampled rollout,TTandSSrefer to the same model under different conditioning, andyc∗y^{*}_{c}is the privileged context, typically a ground truth referenced solution provided in the dataset or a correct same-group rollout. Conditioning on a verified solution does sharpen teacher confidence on task-bearing tokens, but it also induces systematic, correctness-orthogonal shifts in generative behavior: the privileged teacher favors shorter, more assertive phrasings, suppresses exploratory hedges, and reallocates probability mass toward formatting and discourse tokens. We call this parasitic componentprivilege-induced style drift. As Figure1(a)shows, the learning signal is dominated by such stylistic tokens while the signal on task-bearing tokens is diluted, yielding concrete pathologies shown in Figure1(b):training instability, often accompanied by entropy and length explosion, and premature response-length shrinkage, which limits later-stage reasoning performance..

To address these issues, we proposeRLCSD:ReinforcementLearning withContrastive on-policySelf-Distillation. Rather than relying onec,te_{c,t}alone as the learning signal, we form a contrastive estimate by conditioning the teacher on a correct and an incorrect solution in parallel, yielding

ec,t=log⁡πT​(yt∣x,yc∗,y<t)−log⁡πS​(yt∣x,y<t),\displaystyle e_{c,t}=\log\pi_{T}(y_{t}\mid x,y_{c}^{*},y_{<t})-\log\pi_{S}(y_{t}\mid x,y_{<t}),(1)ew,t=log⁡πT​(yt∣x,yw∗,y<t)−log⁡πS​(yt∣x,y<t),\displaystyle e_{w,t}=\log\pi_{T}(y_{t}\mid x,y_{w}^{*},y_{<t})-\log\pi_{S}(y_{t}\mid x,y_{<t}),(2)ectr,t=ec,t−ew,t,\displaystyle e_{\text{ctr},t}=e_{c,t}-e_{w,t},(3)whereyc∗y_{c}^{*}andyw∗y_{w}^{*}denote a correct and an incorrect reference solution, respectively. As standard datasets seldom provide negative references, we draw both from the student’s own rollouts of the same query, taking a correct and an incorrect trajectory from the same group. The contrastive differenceectr,t=ec,t−ew,te_{\text{ctr},t}=e_{c,t}-e_{w,t}then serves as the token-level supervision signal. Provided that the correct and incorrect hints are presented under an identical prompt template, the subtraction can suppress the stylistic component shared across the two conditionings, thereby making the remaining signal more reflective of task correctness. As illustrated in Figure1(a), tokens ranked by|ec||e_{c}|on math problems are dominated by stylistic markers like “Wait” and “Therefore”, whereas those ranked by|ectr||e_{\text{ctr}}|more often correspond to task-bearing tokens, such as mathematical content tokens in the math reasoning task. Given the contrastive token-level signalectr,te_{\mathrm{ctr},t}, we combine it with the query-level advantageAORMA_{\mathrm{ORM}}from GRPO by treatingectr,te_{\mathrm{ctr},t}as a per-token modulation ofAORMA_{\mathrm{ORM}}rather than as a substitute for it. Several further design choices, detailed in Section3, ensure that the modulation neither reverses the direction ofAORMA_{\mathrm{ORM}}nor allows the token-level signal to be diluted in the aggregated loss.

We evaluate RLCSD on Qwen3(Yanget al.,2025)at three scales (1.7B, 4B, 8B) and Olmo-3-7B-Think(Olmoet al.,2025)across two task families: mathematical reasoning (AMC23(MAA,2023), AIME24(Zhang and Math-AI,2024), AIME25(Zhang and Math-AI,2025)) and logical reasoning (Knights & Knaves(Xieet al.,2025), at both in-distribution and out-of-distribution difficulty). As shown in Figure1(c), RLCSD consistently outperforms GRPO and the on-policy self-distillation baselines (OPSD(Zhaoet al.,2026), SDPO(Hübotteret al.,2026), SRPO(Liet al.,2026a), RLSD(Yanget al.,2026)) across the tested benchmarks. Figure1(b)further reveals that RLCSD maintains stable training and preserves response length throughout optimization, whereas competing OPSD baselines either destabilize or suffer length shrinkage.

Our contributions are summarized as follows:

• •We characterizeprivilege-induced style drift, the pathology in which conditioning the teacher on a privileged solution shifts its per-token signal toward stylistic rather than correctness-bearing tokens. To address it, we proposeRLCSD, which is designed to suppress this drift through a symmetric contrast between correct and incorrect privileged hints, and integrates the cleaned signal into RLVR as a verifier-anchored modulation of the GRPO advantage.
• •We conduct extensive experiments on Qwen3 at three scales (1.7B/4B/8B) and Olmo-3-7B-Think across mathematical and logical reasoning, showing that RLCSDconsistently outperformsGRPO and prior on-policy self-distillation methods while sustaining stable entropy and response length, whereas prior OPSD baselines either become unstable or prematurely shrink their responses.
• •Extensive ablation studies show thatthe contrastive principle is general: it can be plugged into existing on-policy self-distillation methods to improve them, and our analysis extends the insight to the broader on-policy distillation setting.

2Related Work

2.1RLVR and On-Policy Distillation

Reinforcement learning with verifiable rewards (RLVR)(Shaoet al.,2024; Yuet al.,2026a; Liuet al.,2025; Zhenget al.,2025; Panet al.,2025)has emerged in recent years as a central post-training paradigm for large language models, achieving substantial gains on reasoning-intensive benchmarks. Representative algorithms such as GRPO(Shaoet al.,2024)dispense with the critic network of classical actor-critic methods and instead estimate advantages by normalizing rewards across a group of rollouts sampled from the same query. The reward itself is supplied by an external verifier and is therefore noise-free on the tasks it covers. The granularity of supervision, however, is at the level of the entire response: a single scalar advantage is broadcast uniformly to every token in the rollout, providing no fine-grained credit assignment to the intermediate reasoning steps that determine correctness.

On-policy distillation (OPD)(Guet al.,2024; Agarwalet al.,2024)has emerged in response to this limitation. By computing a per-token distribution gap between a teacher and the student along the student’s own sampled trajectories, OPD yields a dense, token-level learning signal that complements the sparse trajectory-level reward of RLVR. The dominant choice for measuring this gap is the reverse-KL divergence between teacher and student, giving rise to the canonical OPD objective:

ℒOPD(θ)=𝔼y∼πS[1|y|∑t=1|y|KL(πS(⋅∣x,y<t)∥sg[πT(⋅∣x,y<t)])],\mathcal{L}_{\text{OPD}}(\theta)=\mathbb{E}_{y\sim\pi_{S}}\left[\frac{1}{|y|}\sum_{t=1}^{|y|}\mathrm{KL}\left(\pi_{S}(\cdot\mid x,y_{<t})\,\|\,\mathrm{sg}[\pi_{T}(\cdot\mid x,y_{<t})]\right)\right],(4)wheresg​[⋅]\mathrm{sg}[\cdot]denotes the stop-gradient operator and|y||y|is the length of the rollout in tokens. It has been shown that this objective is equivalent, in the policy-gradient sense, to REINFORCE(Williams,1992)with per-token rewardrt=log⁡πT​(yt∣x,y<t)−log⁡πS​(yt∣x,y<t)r_{t}=\log\pi_{T}(y_{t}\mid x,y_{<t})-\log\pi_{S}(y_{t}\mid x,y_{<t})(Guet al.,2024; Lu and Thinking Machines Lab,2025).

A line of work has explored alternative divergence choices to address the trade-off between mode coverage and mode seeking inherent in reverse-KL: forward-KL offers stronger mode coverage at the cost of mass-spreading, while Jensen-Shannon Divergence(Agarwalet al.,2024)and skew KL(Koet al.,2024)formulations interpolate between the two extremes. Other extensions consider OPD under black-box access to the teacher(Yeet al.,2025), and propose training-efficient variants that reduce the per-step distillation cost(Wuet al.,2026). At industrial scale, MiMo-V2-Flash(Xiaoet al.,2026)integrates the dense token-level advantagelog⁡πT​(yt∣x,y<t)−log⁡πS​(yt∣x,y<t)\log\pi_{T}(y_{t}\mid x,y_{<t})-\log\pi_{S}(y_{t}\mid x,y_{<t})from OPD with the outcome-based verifier rewardAORMA_{\mathrm{ORM}}of RLVR in its post-training pipeline. More recent investigations(Liet al.,2026b; Fuet al.,2026; Zhuet al.,2026)have shifted attention from objective design to the conditions under which OPD itself succeeds: the teacher and student should share compatible reasoning styles, the teacher must possess capabilities genuinely absent from the student, response length must fall within a productive regime, and approximations such as top-kktruncation should preserve unbiasedness of the gradient estimator.

2.2On-Policy Self-Distillation

Despite the rapid progress of on-policy distillation, the paradigm faces a series of practical obstacles. First, most OPD methods require white-box access to the teacher’s token-level logits, which precludes the use of strong closed-source models as teachers. Second, the teacher and student must share an identical vocabulary, a condition rarely satisfied across model families and one that restricts OPD in practice to the distillation of smaller models from larger counterparts within the same series. Third, serving a separate teacher model throughout RL training imposes substantial memory and latency overhead. To address these limitations, on-policy self-distillation (OPSD) has emerged(Zhaoet al.,2026; Hanet al.,2026; Yuet al.,2026b; Haoet al.,2026). In OPSD, the teacher and the student are the same model; the teacher is simply granted access to privileged context unavailable to the student, such as a ground-truth solution or terminal feedback from the environment. The standard OPSD objective takes the following dense distillation form:

ℒOPSD(θ)=𝔼y∼πS[1|y|∑t=1|y|KL(πS(⋅∣x,y<t)∥sg[πT(⋅∣x,yc∗,y<t)])],\mathcal{L}_{\text{OPSD}}(\theta)=\mathbb{E}_{y\sim\pi_{S}}\left[\frac{1}{|y|}\sum_{t=1}^{|y|}\mathrm{KL}\left(\pi_{S}(\cdot\mid x,y_{<t})\,\|\,\mathrm{sg}[\pi_{T}(\cdot\mid x,y_{c}^{*},y_{<t})]\right)\right],(5)whereTTandSSdenotes the same model, andyc∗y_{c}^{*}is the privileged context. Inherited from OPD, the OPSD objective is equivalent to reinforcement learning with per-token advantageAt=log⁡πT​(yt∣x,yc∗,y<t)−log⁡πS​(yt∣x,y<t)A_{t}=\log\pi_{T}(y_{t}\mid x,y_{c}^{*},y_{<t})-\log\pi_{S}(y_{t}\mid x,y_{<t}).

Recent work has explored different instantiations of the OPSD framework, most of which adoptdenseper-token distillation. OPSD(Zhaoet al.,2026)performs dense distillation using the canonical forward-KL form. While it observes the dominance of stylistic tokens in the per-token signal, it addresses this only through an ad-hoc per-token pointwise KL-clipping mechanism, which fails to resolve the issue at its root and leaves training unstable. SDPO(Hübotteret al.,2026)likewise uses dense distillation, but replaces forward KL with the Jensen–Shannon divergence to balance mode-seeking and mode-covering behavior in the per-token gap. SRPO(Liet al.,2026a)observes that the privileged context contributes little additional signal on already-correct rollouts, and proposes a sample-level routing mechanism in which correct samples are optimized via GRPO while incorrect ones are optimized via SDPO’s dense objective. In contrast to all of the above, RLSD(Yanget al.,2026)departs from dense distillation entirely: rather than matching full token-level distributions, it relies on asampled-tokensignal (which RLCSD also adopts) and argues that this OPSD-like signal should serve only as a modulation ofAORMA_{\text{ORM}}rather than a replacement. Accordingly, it designs an algorithm in which the verifier-derivedAORMA_{\text{ORM}}determines the direction of every update while the sampled-token signal modulates only its magnitude.

However, all of these methods rely exclusively onone-sided positive privileged contexts(e.g., dataset ground-truth solutions or correct in-group rollouts) when constructing the teacher distribution, and therefore inherit the privilege-induced style drift identified earlier as a systematic confound in their token-level signal.To address this limitation, we propose RLCSD, which removes the drift by construction through a symmetric contrastive cancellation between correct and incorrect privileged contexts.

3Method

3.1Problem: Privilege-Induced Style Drift

Figure 2:Per-rollout mean|ec,t||e_{c,t}|on style vs. task tokens. The privilege shifts most of its influence onto style tokens (0.2630.263vs.0.0830.083).Existing OPSD methods derive their signal fromec,t=log⁡πT​(yt∣x,yc∗,y<t)−log⁡πS​(yt∣x,y<t)e_{c,t}=\log\pi_{T}(y_{t}\mid x,y^{*}_{c},y_{<t})-\log\pi_{S}(y_{t}\mid x,y_{<t}), whereTTandSSare the same model with and without the privileged contextyc∗y^{*}_{c}. Conditioning on a verified solution does sharpen the teacher on correctness-bearing tokens, but it simultaneously induces a systematic, correctness-orthogonal shift in generative behavior: the privileged teacher adopts shorter, more assertive phrasings, suppresses exploratory hedges, and reallocates probability mass toward formatting and discourse tokens. We call this parasitic componentprivilege-induced style drift.

The signal is dominated by style tokens.

To quantify the drift, we take math reasoning as a representative case and partition the vocabulary into ataskset of content-bearing tokens (numerals, operators, and mathematical symbols) and astyleset of formatting and discourse markers (e.g. “Wait”, “Therefore”, “\n\n”). The full partitioning procedure is detailed in AppendixC. For each sampled rollout we compute the per-rollout mean of|ec,t||e_{c,t}|separately over its task tokens and over its style tokens, and plot the two resulting distributions in Figure2. The distributions are cleanly separated: the style-token mean (0.2630.263) exceeds the task-token mean (0.0830.083) by roughly3×3\times, with almost no overlap. In other words, the privileged conditioning concentrates the overwhelming majority of its influence on stylistic tokens while diluting the signal on the correctness-bearing tokens that actually determine the answer. As we show empirically in Section4, this skew is not benign: in existing OPSD methods it manifests as two distinct failure modes—entropy-driven training instability, often accompanied by response-length explosion, and premature response-length shrinkage.

3.2Overview of RLCSD

Building on the diagnosis of privilege-induced style drift in Section3.1, RLCSD constructs a clean, token-level supervision signal by contrastive cancellation across symmetrically-formatted privileged contexts and uses it to modulate the verifier-derived advantage in a GRPO objective. Figure3illustrates the full pipeline, which consists of three stages:

• •Stage 1: Rollout Sampling and Partitioning.For each query, the model samples a group of rollouts, which are then partitioned by the verifier into a correct subset and an incorrect subset.
• •Stage 2: Constructing the Contrastive Token-Level Signal.From the subsets produced in Stage 1, we draw a positive hint and a set of negative hints, and feed each hint to the teacher under an identical prompt template. The contrastive difference between the positive and the negative branches yields a per-token signalectr,te_{\text{ctr},t}on sampled tokens in which the shared stylistic component is substantially reduced, leaving a signal that is more strongly tied to task correctness. Moreover, we introduce two design refinements, marginalizing overKKnegative hints and excluding the current student rollout from the hint pool, both detailed in §3.4.
• •Stage 3: Modulating the Outcome Advantage and Aggregating the Loss.We convertectr,te_{\text{ctr},t}into a bounded per-token modulationrtr_{t}and apply it toAORMA_{\text{ORM}}through clamping and masking, yielding a two-path PPO-style clipped aggregated loss(Schulmanet al.,2017).

The remainder of this section details each stage: privileged context construction (§3.3), construction of the contrastive token-level signal (§3.4), verifier-anchored modulation of the advantage (§3.5), and the final two-path loss (§3.6).

Figure 3:Overview of RLCSD.Stage 1samples a group of rollouts and partitions them by the verifier into correct (𝒢+\mathcal{G}^{+}) and incorrect (𝒢−\mathcal{G}^{-}) subsets.Stage 2feeds a positive hint andKKnegative hints to the teacher under an identical template; their difference yields the contrastive signalectr,te_{\mathrm{ctr},t}, their difference yields the contrastive signalectr,te_{\mathrm{ctr},t}, in which the shared privilege-induced style component is substantially suppressed.Stage 3convertsectr,te_{\mathrm{ctr},t}into a bounded modulationrtr_{t}, applies it to the verifier advantageAORMA_{\mathrm{ORM}}with a sign-preserving clamp, and aggregates a two-path clipped loss.

3.3Constructing Privileged Contexts from Group Rollouts

For each queryxx, we follow the GRPO sampling protocol and draw a group ofGGrollouts{y(g)}g=1G\{y^{(g)}\}_{g=1}^{G}from the student policyπS\pi_{S}. A rule-based verifier assigns each rollout a binary reward:r=1r=1if the extracted answer matches the ground truth andr=0r=0otherwise***All training tasks in our experiments use binary (0/1) rewards. For math tasks the verifier extracts the\\backslashboxed{}answer and checks equivalence; for Knights-and-Knaves it parses the assignment list.. This partitions the group into a correct subset𝒢+\mathcal{G}^{+}and an incorrect subset𝒢−\mathcal{G}^{-}. Groups for which either subset is empty are discarded; this discards no useful training signal, because a uniform-outcome group already yields zero advantage under GRPO’s group-relative normalization.

Symmetric prompt templates for positive and negative hints.

From each non-discarded group we form apositive privileged contextfrom a rollout in𝒢+\mathcal{G}^{+}and anegative privileged contextfrom a rollout in𝒢−\mathcal{G}^{-}. Each hint is wrapped in an identical prompt template consisting of (i) the original problem statement, (ii) the sibling rollout’s full reasoning trace, (iii) the extracted final answer, and (iv) a continuation instruction that transitions from the reference block to the teacher’s own response. The template used in our main experiments is shown below. Critically, the template makes no distinction between correct and incorrect hints: a negative hint is presented with the same “Reference Solution” framing and “Correct final answer” label as a positive one, so the teacher treats both as ground-truth references and produces distributions that differonlyin what the reference says. Because the template is byte-for-byte identical across the two branches, the privilege-induced stylistic shift is more likely to be shared between them and therefore partially cancels in the subtractionectr,te_{\text{ctr},t}, leaving the signal more task-bearing in practice.

Teacher Privileged Prompt TemplateProblem: {problem} Here is a reference solution to this problem: === Reference Solution Begin === {sibling rollout text} Correct final answer: {extracted answer} === Reference Solution End === After reading the reference solution above, make sure you understand the reasoning behind each step.Please reason step by step, and put your final answer within\\backslashboxed{}.

3.4Contrastive Token-Level Signal

The vanilla form ofectr,te_{\text{ctr},t}.

The most vanilla instantiation of the contrastive signal samples a positive hintyc∗y_{c}^{*}uniformly from𝒢+\mathcal{G}^{+}and a single negative hintyw∗y_{w}^{*}uniformly from𝒢−\mathcal{G}^{-}, and takes the difference of the two teacher–student log-probability gaps:

ectr,t(vanilla)=ec,t−ew,t=log⁡πT​(yt∣x,yc∗,y<t)−log⁡πT​(yt∣x,yw∗,y<t),yc∗∼𝒢+,yw∗∼𝒢−.e_{\text{ctr},t}^{{\color[rgb]{0.7265625,0.109375,0.109375}\definecolor[named]{pgfstrokecolor}{rgb}{0.7265625,0.109375,0.109375}(\text{vanilla})}}=e_{c,t}-e_{w,t}=\log\pi_{T}(y_{t}\mid x,y_{c}^{*},y_{<t})-\log\pi_{T}(y_{t}\mid x,y_{w}^{*},y_{<t}),\quad y_{c}^{*}\sim\mathcal{G}^{+},\;y_{w}^{*}\sim\mathcal{G}^{-}.(6)This vanilla form suffers from two structural problems, which motivate two corresponding refinements.

Figure 4:Task/style analysis on|ectr,t||e_{\mathrm{ctr},t}|. The style–task gap shrinks markedly relative to|ec||e_{c}|(Figure2), indicating the contrast suppresses the style component and yields a cleaner signal.

Refinement 1:KK-marginalized negative hints.

A direct use of the vanilla contrast is unstable because incorrect solutions are highly heterogeneous: for an incorrect target rolloutyy, a randomly sampled negative hintyw∗y_{w}^{*}may correspond to a very different error type, in which case the teacher conditioned onyw∗y_{w}^{*}does not reliably endorse the tokens ofyy. The negative branch then ceases to provide a stable counter-signal, weakening the directional consistency ofectr,t(vanilla)e^{(\mathrm{vanilla})}_{\mathrm{ctr},t}and injecting noise into training. To address this, instead of conditioning on a single incorrect rollout, we sampleKKnegative hints independently fromG−G^{-}and average their probabilities in the negative branch. This replaces a brittle one-to-one contrast with a more stable estimate against the error distribution represented by the group. Empirical evidence is provided in AppendixA.1.

Refinement 2: excluding the target rollout from the hint pool.

Sampling a hint uniformly from𝒢±\mathcal{G}^{\pm}may select the target rolloutyyitself. Conditioning the teacher onyyas its own hint shifts its distribution toward extreme over-confidence, making the signal overly sharp and destroying the token-level uniquenessectr,te_{\text{ctr},t}is meant to capture. We thereforeexcludeyyfrom the hint pool, drawing both positive and negative hints from its siblings𝒢±∖{y}\mathcal{G}^{\pm}\setminus\{y\}; AppendixA.2quantifies the over-confidence effect that motivates this choice.

Finalectr,te_{\mathrm{ctr},t}and its effect.

Combining the two refinements, the final contrastive signal is

ectr,t=log⁡πT​(yt∣x,yc∗,y<t)−log⁡1K​∑k=1KπT​(yt∣x,yw,k∗,y<t),yc∗∼𝒢+∖{y},{yw,k∗}k=1K∼𝒢−∖{y}.e_{\mathrm{ctr},t}=\log\pi_{T}(y_{t}\mid x,y_{c}^{*},y_{<t})-\log\frac{1}{K}\sum_{k=1}^{K}\pi_{T}(y_{t}\mid x,y_{w,k}^{*},y_{<t}),\quad y_{c}^{*}\sim\mathcal{G}^{+}\!\setminus\!\{y\},\;\{y_{w,k}^{*}\}_{k=1}^{K}\sim\mathcal{G}^{-}\!\setminus\!\{y\}.(7)Since the shared template makes the privilege-induced style component more similar across the two branches, the subtraction suppresses this shared component, leaving the signal more task-bearing. Repeating the task/style analysis of Section3.1on|ectr,t||e_{\mathrm{ctr},t}|(Figure4) confirms this: the style–task separation seen under|ec||e_{c}|(Figure2) is substantially reduced, shifting weight onto the task-bearing tokens that determine the answer.

3.5Verifier-Anchored Modulation of the Advantage

Outcome-level GRPO advantage.

For each queryxx, GRPO samples a group ofGGrollouts{y(g)}g=1G\{y^{(g)}\}_{g=1}^{G}from the old policyπθold\pi_{\theta_{\text{old}}}. A verifier assigns each rollout an outcome rewardR(g)∈{0,1}R^{(g)}\in\{0,1\}. The outcome-level advantage is then computed by group-relative normalization:

AORM(g)=R(g)−1G​∑j=1GR(j)1G​∑j=1G(R(j)−1G​∑ℓ=1GR(ℓ))2+ϵadv.A_{\text{ORM}}^{(g)}=\frac{R^{(g)}-\frac{1}{G}\sum_{j=1}^{G}R^{(j)}}{\sqrt{\frac{1}{G}\sum_{j=1}^{G}\left(R^{(j)}-\frac{1}{G}\sum_{\ell=1}^{G}R^{(\ell)}\right)^{2}}+\epsilon_{\text{adv}}}.(8)This scalar advantage is broadcast to all tokens in rollouty(g)y^{(g)}. Thus,AORM(g)A_{\text{ORM}}^{(g)}provides a verifier-grounded trajectory-level update direction, but it does not distinguish which intermediate tokens are responsible for the final outcome.

Drawing on the ideas behind Xiaomi MiMo’s OPD algorithm(Xiaoet al.,2026)and RLSD(Yanget al.,2026), we do not treat the teacher–student log-probability gap as the sole source of learning signal. Instead, we use the verifier-derivedAORMA_{\text{ORM}}as the anchor that determines the update direction, and use the contrastive token-level signalectr,te_{\text{ctr},t}only to modulate its magnitude at selected tokens. The specific procedure is as follows.

Soft normalization and rescaling.

Rawectr,te_{\text{ctr},t}values can be heavy-tailed, with occasional outlier tokens producing values that would dominate the optimization. We first squash them through a hyperbolic tangent and then rescale the result by a factorλ\lambda:

rt=λ​tanh⁡(ectr,tτ)∈(−λ,λ),r_{t}=\lambda\tanh\!\left(\frac{e_{\text{ctr},t}}{\tau}\right)\in(-\lambda,\lambda),(9)whereτ\taucontrols the soft-threshold slope andλ\lambdasets the overall scale.

Token-level mask.

Many tokens carry anrtr_{t}whose absolute value is very small; these can be regarded as noise. We therefore apply a token-level mask to single out the subset of tokens carrying contrastive signal strong enough to warrant modulation, and only these tokens have their advantage modulated byrtr_{t}:

mt=𝟙​(|rt|>δ).m_{t}=\mathbb{1}(|r_{t}|>\delta).(10)Empirically,mt=1m_{t}=1for roughly 20% to 30% of tokens, consistent with the observation that a small fraction of high-influence tokens determines the outcome of long reasoning chains.

Sign-preserving clamp.

For the tokens selected for modulation, we addrtr_{t}to the GRPO advantageAORMA_{\text{ORM}}broadcast to tokenttand clamp the result so that it retains the sign ofAORMA_{\text{ORM}}:

A~t={max⁡(0,AORM+rt)if​AORM≥0,min⁡(0,AORM+rt)if​AORM<0.\tilde{A}_{t}=\begin{cases}\max(0,\,A_{\text{ORM}}+r_{t})&\text{if }A_{\text{ORM}}\geq 0,\\ \min(0,\,A_{\text{ORM}}+r_{t})&\text{if }A_{\text{ORM}}<0.\end{cases}(11)This safeguard prevents the contrastive modulation from reversing the verifier-determined direction at any individual token.

3.6Two-Path Loss Aggregation

We write the RLCSD optimization target as a PPO-style clipped surrogate objective to be maximized; in implementation, we minimize its negative as the training loss. The objective contains two paths: an unmodulated path for tokens withmt=0m_{t}=0, which uses the original outcome-level advantageAORMA_{\text{ORM}}, and a modulated path for tokens withmt=1m_{t}=1, which uses the sign-preserving modulated advantageA~t\tilde{A}_{t}.

For rollouty(g)y^{(g)}, let

ρg,t​(θ)=πθ​(yt(g)∣x,y<t(g))πθold​(yt(g)∣x,y<t(g))\rho_{g,t}(\theta)=\frac{\pi_{\theta}(y^{(g)}_{t}\mid x,y^{(g)}_{<t})}{\pi_{\theta_{\text{old}}}(y^{(g)}_{t}\mid x,y^{(g)}_{<t})}(12)denote the per-token importance ratio. For a token with advantageAA, we define the clipped surrogate contribution as

gg,t​(A;θ)=min⁡(ρg,t​(θ)​A,clip⁡(ρg,t​(θ),1−ϵ,1+ϵ)​A).g_{g,t}(A;\theta)=\min\!\Big(\rho_{g,t}(\theta)A,\;\operatorname{clip}\!\big(\rho_{g,t}(\theta),1-\epsilon,1+\epsilon\big)A\Big).(13) For each rollouty(g)y^{(g)}, define the unmodulated and modulated token sets as

𝒰(g)={t:mt(g)=0},ℳ(g)={t:mt(g)=1}.\mathcal{U}^{(g)}=\{t:m^{(g)}_{t}=0\},\qquad\mathcal{M}^{(g)}=\{t:m^{(g)}_{t}=1\}.(14)The rollout-level RLCSD surrogate objective averages the two paths independently:

JRLCSD(g)​(θ)=1|𝒰(g)|​∑t∈𝒰(g)gg,t​(AORM(g);θ)+η⋅1|ℳ(g)|​∑t∈ℳ(g)gg,t​(A~t(g);θ),J_{\text{RLCSD}}^{(g)}(\theta)=\frac{1}{|\mathcal{U}^{(g)}|}\sum_{t\in\mathcal{U}^{(g)}}g_{g,t}\!\left(A_{\text{ORM}}^{(g)};\theta\right)+\eta\cdot\frac{1}{|\mathcal{M}^{(g)}|}\sum_{t\in\mathcal{M}^{(g)}}g_{g,t}\!\left(\tilde{A}^{(g)}_{t};\theta\right),(15)whereη\etacontrols the relative weight of the modulated path. In the rare case that either token set is empty, the corresponding path is omitted from the rollout-level objective.

Finally, the full query-level objective averages over the sampled group, and the population objective averages over training queries:

JRLCSD​(θ)=𝔼x∼𝒟​[1G​∑g=1GJRLCSD(g)​(θ)].J_{\text{RLCSD}}(\theta)=\mathbb{E}_{x\sim\mathcal{D}}\left[\frac{1}{G}\sum_{g=1}^{G}J_{\text{RLCSD}}^{(g)}(\theta)\right].(16)The training loss minimized in implementation is the negative of this maximization objective:

ℒRLCSD​(θ)=−JRLCSD​(θ).\mathcal{L}_{\text{RLCSD}}(\theta)=-J_{\text{RLCSD}}(\theta).(17) The choice to normalize the two paths independently within each rollout, rather than taking a single global mean over all tokens, is essential for preserving the influence of the contrastively modulated tokens. In practice, only about 20%–30% of tokens satisfymt=1m_{t}=1; under a single global average, the modulated path would therefore contribute only in proportion to this small fraction and be easily overwhelmed by the unmodulated tokens. As a result, the token-level contrastive signal would be diluted, and the objective would drift toward plain GRPO, losing the fine-grained credit assignment that motivates RLCSD. Independent normalization avoids this dilution: it decouples the strength of the modulated path from the masked-token ratio|ℳ(g)|/|y(g)||\mathcal{M}^{(g)}|/|y^{(g)}|, and makesη\etathe sole, explicit control over how much the contrastive signal contributes to training.

4Experiments

We conduct comprehensive experiments to answer the following research questions:

• •RQ1:How does RLCSD compare with GRPO and other OPSD baselines in terms of task performance and training stability? (§4.2)
• •RQ2:Can the proposed contrastive-hint signal improve not only RLCSD itself but also other OPSD baselines as a general-purpose component? (§4.3)
• •RQ3:How important are the individual design choices in RLCSD, including reference-hint selection, integration withAORMA_{\text{ORM}}, and the two-path loss aggregation? (§4.4)
• •RQ4:What broader insights does RLCSD offer for on-policy distillation (OPD)? (§4.5)

4.1Experimental Setup

Models and Datasets.

We experiment with four reasoning models,all with thinking mode enabled during both training and validation: Qwen3-1.7B, Qwen3-4B, Qwen3-8B(Yanget al.,2025), and Olmo-3-7B(Olmoet al.,2025). We train on two categories of tasks. (1)Math reasoning, for which the training data is drawn from DeepMath-103K(Heet al.,2025): we use difficulty levels 5–7 for the 1.7B model, levels 6–8 for the 4B model, and levels 7–10 for the 8B model. The test sets are AMC23(MAA,2023), AIME24(Zhang and Math-AI,2024), and AIME25(Zhang and Math-AI,2025). (2)Logical reasoning, for which we train on Knight-and-Knaves(Xieet al.,2025)with 4–8 roles. The test set is also Knight-and-Knaves, comprising an in-domain test (4–8 roles, 500 questions) and an out-of-domain test (9, 10, and 11 roles, 100 questions each).

Baselines.

We compare RLCSD against (1)GRPO(Shaoet al.,2024), group relative policy optimization with binary outcome rewards verified against ground-truth answers; and the following on-policy self-distillation baselines: (2)OPSD(Zhaoet al.,2026), dense-logit distillation using forward KL with a per-token pointwise KL-clipping mechanism; (3)SDPO(Hübotteret al.,2026), dense-logit distillation using Jensen–Shannon divergence; (4)SRPO(Liet al.,2026a), which routes successful samples to GRPO and failed samples to SDPO; and (5)RLSD(Yanget al.,2026), which uses the teacher–student probability gap on sampled tokens to modulateAORMA_{\text{ORM}}, whereAORMA_{\text{ORM}}determines the update direction and the probability gap determines its magnitude.

Implementation Details.

All experiments are conducted on 8 H20 GPUs with full-parameter training. We use a learning rate of1×10−61\times 10^{-6}for math reasoning and5×10−65\times 10^{-6}for logical reasoning, with the teacher instantiated as a snapshot of the student model that is refreshed every 10 student training steps. Other hyperparameters are set toK=4,τ=0.02K=4,\tau=0.02,λ=0.5\lambda=0.5,δ=0.02\delta=0.02, andη=1.0\eta=1.0. For the construction of the privileged context, all on-policy self-distillation baselines use the ground-truth answers provided by the dataset, whereas RLCSD uses correct and incorrect student rollouts from the same group†††Ablations in Section4.4show that the choice of hint source (dataset CoT versus the model’s own correct rollout) has little effect on its own.. In all methods, the privileged context consists of the CoT solution together with the answer. Training and evaluation settings are kept identical across all methods, with a maximum generation length of1638416384during training and3891238912during validation. More experiment details are provided in AppendixB.

4.2Main Results

Table 1:Main results across model scales. Math reasoning (AMC23, AIME24, AIME25) is reported as mean@12; logical reasoning (Knight-and-Knaves) is reported as pass@1, with 4–8 Roles as the in-domain (ID) setting and 9 Roles, 10 Roles, and 11 Roles as out-of-domain (OOD) settings.Boldmarks the best result in each block, andunderlinemarks the second-highest result;green subscriptsdenote the gain of RLCSD over the Base model.(a)Math: entropy (b)Math: length (c)Math: reward (d)Math: accuracy (e)K&K: entropy (f)K&K: length (g)K&K: reward (h)K&K: accuracy

Figure 5:Training dynamics of Qwen3-4B on math reasoning (top) and Knight-and-Knaves (bottom), comparing entropy, response length, training reward, and validation performance. Existing OPSD methods exhibit two characteristic failure modes: entropy explosion followed by training collapse (OPSD, SDPO, SRPO), and response-length shrinkage (RLSD). In contrast, RLCSD remains stable throughout training, preserving both entropy and response length while achieving stronger final validation performance.#### Performance Comparison of RLCSD against Other Baselines.

As shown in Table1, RLCSD consistently outperforms GRPO and prior OPSD methods across model scales and task settings, attaining the best result on almost every benchmark. The gains over the Base model are consistent across model families: on average, RLCSD improves by+4.3+4.3on math and+10.9+10.9on logical reasoning for Qwen3-1.7B,+2.5+2.5and+6.8+6.8for Qwen3-4B,+2.7+2.7and+14.4+14.4for Qwen3-8B, and+1.8+1.8and+9.9+9.9for Olmo-3-7B. The advantage is especially pronounced on the out-of-distribution Knight-and-Knaves splits, reaching+21.0+21.0on the 11-role setting for Qwen3-8B and+13.0+13.0for Olmo-3-7B, suggesting that the cleaned token-level signal improves generalization rather than merely fitting the training difficulty.

Failure Mode Analysis of Other On-policy Self-distillation Methods.

Figure5reveals two distinct failure modes in existing on-policy self-distillation methods:

• •Failure Mode 1: Entropy explosion and training instability.As shown in Figures5(a)and5(e), methods such as OPSD, SDPO, and SRPO suffer from rapid entropy growth during training. This unbounded increase destabilizes optimization and eventually causes collapse, manifesting as abrupt response-length explosion (SDPO and SRPO in Figure5(b); SDPO, SRPO, and OPSD in Figure5(f)) together with sharp drops in training reward and validation performance (SDPO and SRPO in Figures5(c)and5(d); SDPO, SRPO, and OPSD in Figures5(g)and5(h)).
• •Failure Mode 2: Premature response-length shrinkage.In contrast, RLSD exhibits a substantial decrease in response length as training proceeds, especially on math reasoning (Figure5(b)). Although optimization remains stable, the shortened responses limit later-stage performance and make the method less suitable for long-horizon reasoning tasks that require extended chains of thought, as reflected in the validation curve in Figure5(d).

Stable Dynamics of RLCSD.

In contrast, RLCSD maintains stable training dynamics throughout optimization, keeping both policy entropy and response length stable (Figures5(a),5(b),5(e), and5(f)), on par with GRPO. The key difference is that RLCSD augments GRPO’s verifier-derived outcome signal with a dense token-level contrastive modulation, yielding finer-grained credit assignment without sacrificing optimization stability. As a result, RLCSD retains the same stable optimization behavior as GRPO while converging to consistently better validation performance, as shown in Figures5(d)and5(h).

Table 2:Contrastive hints as a plug-in component. For each on-policy self-distillation method, we isolate the effect of the contrastive construction by varying the hint source: the dataset-provided CoT (as in the main experiments), the model’s own correct rollout, and the model’s own correct/incorrect rollouts in contrast. Math reasoning (AMC23, AIME24, AIME25) is mean@12; Knight-and-Knaves (pass@1) uses roles 4–8 as in-domain (ID) and roles 9/10/11 as out-of-domain (OOD). All results use Qwen3-4B;boldmarks the best within each method block.Δ\Deltais the gain of the contrastive variant over the better of the two non-contrastive hint sources.(a)Entropy (b)Training rewardFigure 5:Effect of the contrast mechanism on OPSD for the K&K task (Qwen3-4B). Without contrast, the actor entropy explodes and training destabilizes; adding contrast keeps the entropy bounded (a) and yields a stable training reward (b).

(a)RLSD (b)RLCSDFigure 6:Effect of the contrast mechanism on response length for math tasks (Qwen3-4B). ontrast mitigates premature response-length shrinkage for both RLSD (a) and one-sided RLCSD ablation (b), preserving longer reasoning traces that benefit late-stage training.

4.3Contrastive Hints as a General-Purpose Component

A central claim of this work is that the contrastive construction is not specific to RLCSD, but rather a general principle for purifying privileged token-level signals. To test this, we apply it to two representative on-policy self-distillation methods from different families: OPSD(Zhaoet al.,2026), a dense full-distribution distillation method, and RLSD(Yanget al.,2026), an advantage-modulation method. For each method, we vary only the source of the privileged signal while keeping all other components fixed, and compare three settings:(1) dataset CoT, where the privileged context is the ground-truth solution provided by the dataset, as in the original methods and our main experiments;(2) own rollout, where the privileged context is replaced by a correct rollout sampled by the model itself from the same group, isolating the effect of switching the hint source from dataset-provided to self-generated; and(3) own rollout + contrast, where the contrastive construction is applied on top of self-generated hints by contrasting a correct sibling rollout against an incorrect one.

How the contrast enters each method.

The two method families incorporate the contrast differently. For advantage-modulation methods (RLSD and RLCSD), the contrast acts at the scalar level: the per-token signal

ectr,t=log⁡πT​(yt∣x,yc∗,y<t)−log⁡1K​∑kπT​(yt∣x,yw,k∗,y<t)e_{\mathrm{ctr},t}=\log\pi_{T}(y_{t}\mid x,y_{c}^{*},y_{<t})-\log\frac{1}{K}\sum_{k}\pi_{T}(y_{t}\mid x,y_{w,k}^{*},y_{<t})replaces the one-sidedec,te_{c,t}as the modulation ofAORMA_{\mathrm{ORM}}. OPSD, by contrast, incorporates the signal at thedistributionlevel. Following a classifier-free-guidance formulation(Ho and Salimans,2022), we replace the OPSD targetπT(⋅∣x,yc∗,y<t)\pi_{T}(\cdot\mid x,y_{c}^{*},y_{<t})with a contrastively amplified target:

qtarget=softmax​((1+α)​ℓc−α​ℓw),q_{\text{target}}=\mathrm{softmax}\big((1+\alpha)\,\ell_{c}-\alpha\,\ell_{w}\big),(18)whereℓc\ell_{c}andℓw\ell_{w}are the teacher logits under the correct and incorrect hints, respectively, andα≥0\alpha\geq 0controls the guidance strength (α=0\alpha=0recovers vanilla OPSD). The per-token loss is further gated by a soft mask

wt=DtDt+τ,Dt=KL(πT(⋅∣x,yw∗,y<t)∥πT(⋅∣x,yc∗,y<t)),w_{t}\;=\;\frac{D_{t}}{D_{t}+\tau},\qquad D_{t}\;=\;\mathrm{KL}\!\left(\pi_{T}(\cdot\mid x,y_{w}^{*},y_{<t})\,\big\|\,\pi_{T}(\cdot\mid x,y_{c}^{*},y_{<t})\right),(19)withτ≥0\tau\!\geq\!0(τ=0\tau\!=\!0disables the mask), and the final objective is∑twtKL(qtarget,t∥πθ(⋅∣x,y<t))\sum_{t}w_{t}\,\mathrm{KL}(q_{\text{target},t}\,\|\,\pi_{\theta}(\cdot\mid x,y_{<t})).

Results.

(1) Switching the one-sided hint source barely matters.Replacing the dataset CoT with the model’s own correct rollout leaves performance essentially unchanged across all three methods.

(2) Using the contrastive signal yields consistent improvements.Building a correct/incorrect contrast on top of self-generated hints improves every method on nearly every metric relative to the better one-sided source. OPSD gains+2.3+2.3on the K&K average and RLSD gains+2.2+2.2on the math average, with the largest single improvements reaching+3.6+3.6on AIME24 and+6.0+6.0on the 11-role split for RLSD. For RLCSD, replacing the contrastive signal with a one-sided hint reduces the math average by3.03.0and the K&K average by5.45.4. This ablation establishes the contrastive signal as a method-agnostic component that improves performance.

(3) The contrastive signal directly mitigates the two failure modes identified in Section4.2.On the K&K task, adding contrast to OPSD keeps the actor entropy bounded instead of letting it explode in the late stage, restoring a stable training reward (Figure6) and thereby mitigating theentropy-explosionmode. On math, the contrast preserves longer reasoning traces rather than letting them collapse: this holds both for RLSD (Figure7(a)) and for RLCSD, whose one-sided ablation reproduces the samepremature length-shrinkagemode that contrast removes (Figure7(b)). Together, these results show that the contrastive construction is a general, method-agnostic way to improve the quality of privileged token-level signals.

4.4Ablation of Key Design Choices

Table 3:Ablation of key design choices in RLCSD (Qwen3-4B). Each variant removes a single component from the full method. Math reasoning (AMC23, AIME24, AIME25) is reported as mean@12; Knight-and-Knaves (pass@1) uses roles 4–8 as in-domain (ID) and roles 9/10/11 as out-of-domain (OOD). Bold marks the best result in each column.We ablate the key design choices of RLCSD introduced in Section3, focusing on two aspects: how reference hints are selected for the privileged context, and how the final loss incorporates the contrastive token-level signal. All experiments in this section are conducted on Qwen3-4B, and the results are reported in Table3.

Ablation on reference-hint selection.

We first examine the two refinements introduced in Section3.4for constructing the contrastive signal. The variant−-KK-marginal negative hintreplaces the marginalized negative branch with a single negative hint, making the contrast sensitive to error-type mismatch between the target rollout and the sampled negative hint. The variant++self-rollout in hint poolallows the target rollout itself to be used as a hint, inducing self-referential over-confidence in the teacher and weakening the token-level distinctiveness thatectr,te_{\mathrm{ctr},t}is intended to capture. The mechanisms behind both effects are analyzed in Section3.4and AppendixA.

Table3confirms that both refinements are important in practice. RemovingKK-marginalization reduces the math and K&K averages by 1.2 and 2.5 points, respectively, while allowing the self-rollout into the hint pool reduces them by 1.2 and 3.2 points. These drops are consistent with the analysis in Section3.4: the former introduces noisy negative contrasts, and the latter pushes the teacher toward trivially over-confident token distributions. Together, these results confirm that reliable hint selection is necessary for keeping the contrastive signal aligned with task correctness.

Ablation on the final loss formulation.

We next ablate the two design choices that determine how the contrastive signal is integrated into optimization. By design,ectr,te_{\mathrm{ctr},t}is used only as amodulationofAORMA_{\mathrm{ORM}}under the sign-preserving clamp in Eq.11, rather than as a replacement for it. This design reflects the complementary roles of the two signals:AORMA_{\mathrm{ORM}}provides a noise-free, verifier-grounded update direction from the outcome reward, whileectr,te_{\mathrm{ctr},t}refines its magnitude at the token level. The variant−-AORMA_{\mathrm{ORM}}anchoringremoves this verifier anchor and instead usesectr,te_{\mathrm{ctr},t}directly as the token-level advantage. As shown in Table3, this causes the largest degradation among all ablations, reducing the math and K&K averages by 2.3 and 4.5 points, respectively. The reason is that, althoughectr,te_{\mathrm{ctr},t}is substantially cleaner than the one-sided signal, it still retains a residual wrong-sign rate. Anchoring onAORMA_{\mathrm{ORM}}helps prevent such reversals from changing the update direction, allowing the contrastive signal to sharpen credit assignment without overriding the outcome reward.

We further compare against “−-two-path loss aggregation”, which replaces the two-path objective in Eq.15with a single global average over all tokens. Under this formulation, the modulated path contributes only in proportion to the masked-token fraction|ℳ|/N|\mathcal{M}|/N. Since only about 20%–30% of tokens satisfy the mask in practice, the contrastive signal is heavily diluted and the objective moves closer to plain GRPO. Consistent with this intuition, Table3shows that removing the two-path aggregation reduces the math and K&K averages by 1.6 and 3.5 points, respectively. This confirms that independent normalization is necessary to preserve the influence of the modulated tokens and fully realize the benefit of token-level contrastive supervision.

4.5Broader Insights for On-Policy Distillation

The same decomposition governs cross-model OPD.

The privilege-induced style drift we identify inself-distillation may not be specific to using the student as its own teacher. A similar decomposition into task-bearing and surface-style components may also help explaincross-modelon-policy distillation (OPD). Standard OPD uses the per-token reverse KL between teacherTTand studentSSon student rollouts as the learning signal(Lu and Thinking Machines Lab,2025; Agarwalet al.,2024). Part of thisTT–SSgap reflects genuine differences in reasoning ability, while another part reflects stylistic differences in discourse, formatting, and thinking pattern. When the latter dominates, the gradient transfers register rather than capability, and OPD degrades into stylistic mimicry. Consistent with this view,Liet al.(2026b)show that a stronger teacher can fail where a weaker but stylistically aligned one succeeds. To make this intuition concrete, we analyze which tokens dominate the learning signal across teacher–student pairs.

Table 4:Top-15 response tokens by per-tokenKLt\mathrm{KL}_{t}for each teacher, on Qwen3-1.7B-Base rollouts over DeepMath.

Setup.

We adopt the same teacher–student configuration asLiet al.(2026b). The student is Qwen3-1.7B-Base. We compare two teachers with similar benchmark accuracy but very different stylistic distance from the student: Qwen3-4B-Instruct, a heavily post-trained instruct model in non-thinking mode, and Qwen3-4B-Base-GRPO‡‡‡https://huggingface.co/lllyx/Qwen3-4B-Base-GRPO, a lightly post-trained math model obtained by zero-RL GRPO on Qwen3-4B-Base. The former is stylistically far from a base student, while the latter remains much closer in style. We sample 200 prompts from DeepMath(Heet al.,2025), draw two student rollouts for each prompt, and compute the per-token reverse KL,

KLt=KL(πS(⋅∣x,y<t)∥πT(⋅∣x,y<t)),\mathrm{KL}_{t}=\mathrm{KL}\!\left(\pi_{S}(\cdot\mid x,y_{<t})\,\|\,\pi_{T}(\cdot\mid x,y_{<t})\right),between the student and each teacher.

Results.

Table4reveals a clear separation in which tokens dominate the learning signal. For the instruct teacher, the highest-KLt\mathrm{KL}_{t}positions are dominated by discourse and structural tokens, including connectives (“The”, “If”, “Thus”), paragraph breaks, and LaTeX delimiters. Its gradient is therefore concentrated on stylistic choices—roughly, “how should I phrase the next step?”—rather than on mathematical content. In contrast, the GRPO teacher’s highest-KLt\mathrm{KL}_{t}positions are dominated by digits, operators, and task-bearing content words such as “subsets”, “ex”, and “=”. This is the kind of signal that directly teaches mathematics rather than register. Training results further support this interpretation (Table5): although Qwen3-4B-Instruct has the stronger standalone math performance, distilling it into Qwen3-1.7B-Base yields a worse student than distilling the stylistically closer GRPO teacher.

Takeaways.

This analysis yields two implications. First, the style drift characterized for OPSD may not be unique to self-distillation: a similar task-bearing versus surface-style decomposition appears relevant to cross-model OPD, where a teacher whose KL mass falls on discourse tokens may transfer register rather than capability. Second, because OPD conditions on no privileged hint, RLCSD’s contrastive cancellation does not directly apply. Instead, the per-token KL ranking offers a cheap, training-free diagnostic for teacher selection.

Table 5:Cross-model OPD with teachers of different stylistic distance from the student. The student is Qwen3-1.7B-Base, and both runs use the same OPD training setup (DeepMath(Heet al.,2025)as training set, 300 steps). Although Qwen3-4B-Instruct has stronger standalone math performance, distilling from the stylistically closer Qwen3-4B-Base-GRPO yields a better student on all three math benchmarks.

5Conclusion

We studied on-policy self-distillation through the lens ofprivilege-induced style drift, a pathology in which privileged conditioning shifts the token-level learning signal toward stylistic rather than task-bearing tokens. To address this, we proposed RLCSD, which mitigates the drift by contrasting correct and incorrect privileged hints under a shared template, and integrates the resulting signal into RLVR as a verifier-anchored token-level modulation ofAORMA_{\mathrm{ORM}}. Experiments across Qwen3 and Olmo models on mathematical and logical reasoning show that RLCSD consistently outperforms GRPO and prior OPSD methods while maintaining stable training dynamics. Beyond RLCSD itself, our results show that the contrastive principle is general: it improves existing OPSD methods and offers a broader perspective on how token-level distillation signals should be constructed and interpreted in both self-distillation and cross-model OPD.

References

• On-policy distillation of language models: learning from self-generated mistakes.InInternational Conference on Learning Representations,Vol.2024,pp. 21246–21263.Cited by:§1,§2.1,§2.1,§4.5.
• DeepSeek-AI (2026)DeepSeek-v4: towards highly efficient million-token context intelligence.Cited by:§1.
• Y. Fu, H. Huang, K. Jiang, J. Liu, Z. Jiang, Y. Zhu, and D. Zhao (2026)Revisiting on-policy distillation: empirical failure modes and simple fixes.arXiv preprint arXiv:2603.25562.Cited by:§2.1.
• Y. Gu, L. Dong, F. Wei, and M. Huang (2024)Minillm: knowledge distillation of large language models.InInternational Conference on Learning Representations,Vol.2024,pp. 32694–32717.Cited by:§1,§2.1,§2.1.
• D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi,et al.(2025)Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948.Cited by:§1.
• Z. Han, T. Zhang, H. Wang, and Y. Sun (2026)Adaptive teacher exposure for self-distillation in llm reasoning.arXiv preprint arXiv:2605.11458.Cited by:§2.2.
• G. Hao, Y. Shang, Y. Long, Z. Zhao, and H. Liang (2026)Self-policy distillation via capability-selective subspace projection.arXiv preprint arXiv:2605.22675.Cited by:§2.2.
• Y. He, S. Kaur, A. Bhaskar, Y. Yang, J. Liu, N. Ri, L. Fowl, A. Panigrahi, D. Chen, and S. Arora (2026)Self-distillation zero: self-revision turns binary rewards into dense supervision.arXiv preprint arXiv:2604.12002.Cited by:§1.
• Z. He, T. Liang, J. Xu, Q. Liu, X. Chen, Y. Wang, L. Song, D. Yu, Z. Liang, W. Wang,et al.(2025)Deepmath-103k: a large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning.arXiv preprint arXiv:2504.11456.Cited by:§4.1,§4.5,Table 5.
• J. Ho and T. Salimans (2022)Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598.Cited by:§4.3.
• W. Hong, W. Yu, X. Gu, G. Wang, G. Gan, H. Tang, J. Cheng, J. Qi, J. Ji, L. Pan,et al.(2025)Glm-4.5 v and glm-4.1 v-thinking: towards versatile multimodal reasoning with scalable reinforcement learning.arXiv preprint arXiv:2507.01006.Cited by:§1.
• J. Hübotter, F. Lübeck, L. Behric, A. Baumann, M. Bagatella, D. Marta, I. Hakimi, I. Shenfeld, T. K. Buening, C. Guestrin,et al.(2026)Reinforcement learning via self-distillation.arXiv preprint arXiv:2601.20802.Cited by:§1,§1,§1,§2.2,§4.1.
• Y. Jin, Y. Wang, L. Fu, Y. Xiao, Y. Luo, H. Liu, B. A. Prakash, J. Hester, J. Wang, and S. Kumar (2026)UniSD: towards a unified self-distillation framework for large language models.arXiv preprint arXiv:2605.06597.Cited by:§1.
• J. Ke, Z. Wen, W. Li, C. He, and L. Zhang (2026)Respecting self-uncertainty in on-policy self-distillation for efficient llm reasoning.arXiv preprint arXiv:2605.13255.Cited by:§1.
• J. Kim, J. Jeon, D. Li, and Y. Yang (2026)Rebellious student: reversing teacher signals for reasoning exploration with self-distilled rlvr.arXiv preprint arXiv:2605.10781.Cited by:§1.
• J. Ko, S. Kim, T. Chen, and S. Yun (2024)Distillm: towards streamlined distillation for large language models.arXiv preprint arXiv:2402.03898.Cited by:§2.1.
• G. Li, T. Yang, J. Fang, M. Song, M. Zheng, H. Guo, D. Zhang, J. Wang, and T. Chua (2026a)Unifying group-relative and self-distillation policy optimization via sample routing.arXiv preprint arXiv:2604.02288.Cited by:§1,§1,§2.2,§4.1.
• Y. Li, Y. Zuo, B. He, J. Zhang, C. Xiao, C. Qian, T. Yu, H. Gao, W. Yang, Z. Liu,et al.(2026b)Rethinking on-policy distillation of large language models: phenomenology, mechanism, and recipe.arXiv preprint arXiv:2604.13016.Cited by:§2.1,§4.5,§4.5.
• A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan,et al.(2024)Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437.Cited by:§1.
• Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, C. Du, W. S. Lee, and M. Lin (2025)Understanding r1-zero-like training: a critical perspective.arXiv preprint arXiv:2503.20783.Cited by:§2.1.
• K. Lu and Thinking Machines Lab (2025)On-policy distillation.Note:https://thinkingmachines.ai/blog/on-policy-distillation/Cited by:§2.1,§4.5.
• MAA (2023)AMC 2023.Note:https://huggingface.co/datasets/math-ai/amc23Dataset compiled from the 2023 American Mathematics CompetitionsCited by:§1,§4.1.
• T. Olmo, A. Ettinger, A. Bertsch, B. Kuehl, D. Graham, D. Heineman, D. Groeneveld, F. Brahman, F. Timbers, H. Ivison, J. Morrison, J. Poznanski, K. Lo, L. Soldaini, M. Jordan, M. Chen, M. Noukhovitch, N. Lambert, P. Walsh, P. Dasigi, R. Berry, S. Malik, S. Shah, S. Geng, S. Arora, S. Gupta, T. Anderson, T. Xiao, T. Murray, T. Romero, V. Graf, A. Asai, A. Bhagia, A. Wettig, A. Liu, A. Rangapur, C. Anastasiades, C. Huang, D. Schwenk, H. Trivedi, I. Magnusson, J. Lochner, J. Liu, L. J. V. Miranda, M. Sap, M. Morgan, M. Schmitz, M. Guerquin, M. Wilson, R. Huff, R. L. Bras, R. Xin, R. Shao, S. Skjonsberg, S. Z. Shen, S. S. Li, T. Wilde, V. Pyatkin, W. Merrill, Y. Chang, Y. Gu, Z. Zeng, A. Sabharwal, L. Zettlemoyer, P. W. Koh, A. Farhadi, N. A. Smith, and H. Hajishirzi (2025)Olmo 3.External Links:2512.13961,LinkCited by:§1,§4.1.
• L. Pan, S. Tao, Y. Zhai, Z. Fu, L. Fang, M. He, L. Zhang, Z. Liu, B. Ding, A. Liu,et al.(2025)D-treerpo: towards more reliable policy optimization for diffusion language models.arXiv preprint arXiv:2512.09675.Cited by:§2.1.
• J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017)Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347.Cited by:3rd item.
• Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu,et al.(2024)Deepseekmath: pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300.Cited by:§1,§2.1,§4.1.
• G. Shen, X. Cheng, C. Zhao, L. Huang, J. Li, D. Zhao, and X. Yu (2026a)Anti-self-distillation for reasoning rl via pointwise mutual information.arXiv preprint arXiv:2605.11609.Cited by:§1.
• G. Shen, L. Huang, X. Cheng, C. Zhao, J. Li, D. Zhao, and X. Yu (2026b)From generic correlation to input-specific credit in on-policy self distillation.arXiv preprint arXiv:2605.11613.Cited by:§1.
• G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu (2024)HybridFlow: a flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256.Cited by:§B.1.
• M. Song and M. Zheng (2026)A survey of on-policy distillation for large language models.arXiv preprint arXiv:2604.00626.Cited by:§1.
• J. Wang, X. Ouyang, Z. Chen, Y. Hu, Z. Pan, X. Li, and L. Guo (2026)TRACE: distilling where it matters via token-routed self on-policy alignment.arXiv preprint arXiv:2605.10194.Cited by:§1.
• R. J. Williams (1992)Simple statistical gradient-following algorithms for connectionist reinforcement learning.Machine learning8(3),pp. 229–256.Cited by:§2.1.
• Y. Wu, S. Han, and H. Cai (2026)Lightning opd: efficient post-training for large reasoning models with offline on-policy distillation.arXiv preprint arXiv:2604.13010.Cited by:§2.1.
• B. Xiao, B. Xia, B. Yang, B. Gao, B. Shen, C. Zhang, C. He, C. Lou, F. Luo, G. Wang,et al.(2026)Mimo-v2-flash technical report.arXiv preprint arXiv:2601.02780.Cited by:§1,§2.1,§3.5.
• T. Xie, Z. Gao, Q. Ren, H. Luo, Y. Hong, B. Dai, J. Zhou, K. Qiu, Z. Wu, and C. Luo (2025)Logic-rl: unleashing llm reasoning with rule-based reinforcement learning.arXiv preprint arXiv:2502.14768.Cited by:§1,§4.1.
• A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv,et al.(2025)Qwen3 technical report.arXiv preprint arXiv:2505.09388.Cited by:§1,§1,§4.1.
• C. Yang, C. Qin, Q. Si, M. Chen, N. Gu, D. Yao, Z. Lin, W. Wang, J. Wang, and N. Duan (2026)Self-distilled rlvr.arXiv preprint arXiv:2604.03128.Cited by:§1,§1,§1,§2.2,§3.5,§4.1,§4.3.
• T. Ye, L. Dong, Z. Chi, X. Wu, S. Huang, and F. Wei (2025)Black-box on-policy distillation of large language models.arXiv preprint arXiv:2511.10643.Cited by:§2.1.
• Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu,et al.(2026a)Dapo: an open-source llm reinforcement learning system at scale.Advances in Neural Information Processing Systems38,pp. 113222–113244.Cited by:§2.1.
• X. Yu, L. Liao, Y. Zhang, Y. Yu, L. Xue, and Q. Guo (2026b)Preference-based self-distillation: beyond kl matching via reward regularization.arXiv preprint arXiv:2605.05040.Cited by:§2.2.
• A. Zeng, X. Lv, Z. Hou, Z. Du, Q. Zheng, B. Chen, D. Yin, C. Ge, C. Huang, C. Xie,et al.(2026)Glm-5: from vibe coding to agentic engineering.arXiv preprint arXiv:2602.15763.Cited by:§1.
• Y. Zhang and T. Math-AI (2024)American invitational mathematics examination (aime) 2024.Cited by:§1,§4.1.
• Y. Zhang and T. Math-AI (2025)American invitational mathematics examination (aime) 2025.Cited by:§1,§4.1.
• S. Zhao, Z. Xie, M. Liu, J. Huang, G. Pang, F. Chen, and A. Grover (2026)Self-distilled reasoner: on-policy self-distillation for large language models.arXiv preprint arXiv:2601.18734.Cited by:§1,§1,§1,§2.2,§2.2,§4.1,§4.3.
• C. Zheng, S. Liu, M. Li, X. Chen, B. Yu, C. Gao, K. Dang, Y. Liu, R. Men, A. Yang,et al.(2025)Group sequence policy optimization.arXiv preprint arXiv:2507.18071.Cited by:§2.1.
• S. Zhu, X. Ye, H. Lu, W. Shi, and G. Liu (2026)The many faces of on-policy distillation: pitfalls, mechanisms, and fixes.arXiv preprint arXiv:2605.11182.Cited by:§2.1.

Appendix AAdditional Analysis of Reference-Hint Selection

This appendix expands on the two refinements introduced in Section3.4:KK-marginalization of negative hints (AppendixA.1) and excluding the target rollout from the hint pool (AppendixA.2). Together they detail why the vanilla contrastive signal is unreliable and how each refinement restores a clean, verifier-aligned signal.

A.1Negative-Hint Selection: Case Analysis andKK-Marginalization

Why a single negative hint is unreliable.

Correct solutions tend to converge to a small set of canonical strategies, whereas incorrect solutions diverge across many distinct error modes. Whether the optimization direction induced by the vanilla contrastive signal aligns with the verifier therefore depends on whether the target rolloutyyand the sampled negative hintyw∗y_{w}^{*}share a comparable error type. Table6enumerates the three resulting cases:

• •Correctyy.The teacher endorses each token inyyunderyc∗y_{c}^{*}(drivingec,te_{c,t}up) and conflicts underyw∗y_{w}^{*}(drivingew,te_{w,t}down), giving a strongly positiveectr,te_{\text{ctr},t}that matches the verifier.
• •Incorrectyywith similar-erroryw∗y_{w}^{*}.Symmetrically,ec,te_{c,t}drops andew,te_{w,t}rises, giving a strongly negativeectr,te_{\text{ctr},t}that again matches the verifier.
• •Incorrectyywith dissimilar-erroryw∗y_{w}^{*}.ec,te_{c,t}still drops, but becauseyw∗y_{w}^{*}commits a different mistake fromyy, the teacher does not endorseyy’s tokens under it either;ew,te_{w,t}loses its expected positive sign, andectr,te_{\text{ctr},t}can carry either sign.

The third case is the failure mode: a substantial fraction of incorrect rollouts end up with their tokens overall encouraged rather than penalized, injecting substantial noise into the optimization. Actively selecting an error-matched negative hint via online LLM-as-judge scoring is prohibitively expensive at RL-training scale, so we instead marginalize overKKindependently sampled negative hints, taking their probability-mean as the negative branch. This spreads the teacher’s mass across the error modes of𝒢−\mathcal{G}^{-}rather than betting on a single one.

Table 6:Behavior of the naive contrastive signalectr,t(Naive)e_{\text{ctr},t}^{(\text{Naive})}across three sampling cases. Arrows indicate the sign of each quantity:↑\uparrowpositive,↓\downarrownegative. The first two cases yield aectr,te_{\text{ctr},t}that aligns with the verifier (✓), but the third case is the failure mode (✗): whenyw∗y_{w}^{*}does not shareyy’s error type, the negative branchew,te_{w,t}no longer reliably endorsesyy’s tokens, andectr,te_{\text{ctr},t}loses its directional anchor.Figure 7:Rollout-level mean ofectr,te_{\text{ctr},t}, computed with one positive hint and one or more negative hints. The positive hint is a randomly sampled correct sibling in all three panels; the panels differ in the rollout type and the negative-hint strategy.Left:correct rollouts with a single randomly sampled negative hint, illustrating the expected positive signal of Case 1.Middle:incorrect rollouts with a single LLM-judged least-similar negative hint (the worst case, Case 3).Right:incorrect rollouts withKK-marginalized negative hints.

Empirical validation.

Figure7confirms the analysis through rollout-level means ofectr,te_{\text{ctr},t}. Correct rollouts produce a cleanly positive signal (left, frac>0>0= 0.989), validating Case 1. To isolate Case 3, the middle panel uses the least-similar negative hint per incorrectyy(selected offline by an LLM judge): the signal is centered at zero, with roughly half the rollouts bearing the wrong sign.KK-marginalization (right) restores a cleanly negative-skewed signal, reducing the wrong-sign rate from51%51\%to under8%8\%.

A.2Self-Referential Collapse and Excluding the Target Rollout

Self-conditioning induces over-confidence.

Sampling a hint uniformly from𝒢+\mathcal{G}^{+}or𝒢−\mathcal{G}^{-}may select the target rolloutyyitself. Conditioning the teacher onyyas its own hint shifts its distribution toward extreme over-confidence, making the training signal overly sharp and destroying the token-level uniqueness thatectr,te_{\mathrm{ctr},t}is meant to capture. As shown in Figure8, for correct rollouts a sibling hint leaves the overall distribution largely unchanged, whereas using the rollout itself as the hint raises the fraction of high-probability tokens (PT>1−10−5P_{T}>1-10^{-5}) by roughly30%30\%. For incorrect rollouts,KK-marginalization mitigates this effect, but including the rollout itself among theKKcandidates still increases the high-probability fraction slightly (by about10%10\%). To avoid this, we exclude the target rolloutyyfrom the hint pool, drawing both positive and negative hints from its siblings𝒢±∖{y}\mathcal{G}^{\pm}\setminus\{y\}.

Figure 8:Distribution of teacher token probabilitiesPT​(yt∣x,hint,y<t)P_{T}(y_{t}\mid x,\text{hint},y_{<t})under different hint sources, for correct (top) and incorrect (bottom) rollouts. Using a sibling hint leaves the distribution close to the no-hint case, whereas including the rollout itself as the hint shifts mass toward over-confident, high-probability tokens, motivating its exclusion from the hint pool.

Appendix BImplementation Details

This section presents the full implementation details that were not fully described in Section4.1of the main text.

B.1Shared training setup

All methods are implemented on top of the veRL(Shenget al.,2024)PPO trainer with FSDP2 actor sharding and vLLM rollouts. All runs use full-parameter training (no LoRA) on8×8\timesH20 GPUs. The settings in Table7are identical across GRPO, OPSD, SDPO, SRPO, RLSD, and RLCSD within a given (task, backbone) configuration. Only the per-step loss differs across methods.

Table 7:Hyperparameters shared by all six methods.GroupHyperparameterValueOptimizationOptimizerAdamWWeight decay0.010.01LR warmup (linear)5050stepsGPUs8×8\timesH20BatchingPer-device batch88Group sizeGG88PPO mini-batch1616Rollout IS modetoken-levelRollout IS threshold2.02.0Rollout samplingTemperature1.01.0Top-pp/ Top-kk0.95/200.95\ /\ 20Max prompt length2,0482{,}048Max completion length16,38416{,}384Evaluation samplingTemperature0.60.6Top-pp/ Top-kk0.95/200.95\ /\ 20Max completion length38,91238{,}912Eval batch size1616Samples per query1212(math, mean@12) /11(K&K, pass@1)#### Note on the Learning rate.

For the mathematics tasks, all six methods use a learning rate of1×10−61\times 10^{-6}. For the logical reasoning tasks, RLCSD performs better with a learning rate of5×10−65\times 10^{-6}. In contrast, the other methods experience more severe training collapse at5×10−65\times 10^{-6}, and therefore use1×10−61\times 10^{-6}.

B.2Method-specific hyperparameters

Table8lists the hyperparameters that vary across methods. We follow theoriginal paperswherever values are specified there.

Table 8:Method-specific hyperparameters.MethodHyperparameterValueDescriptionGRPOKL-to-ref coef.10−310^{-3}coef. onKL​(πθ∥πref)\mathrm{KL}(\pi_{\theta}\|\pi_{\text{ref}}).PPO clipϵ\epsilon0.20.2PPO-clip range.OPSDKL-to-ref coef.0disabled.JSD mixingβ\beta0.00.00= forward KL,11= reverse KL.Vocab clip0.050.05per-(token, vocab) divergence cap.Full-logit distillyesdense top-kk+ tail-bucket target.Top-kk100100vocab indices per token.Tail bucketyesresidual mass outside top-kk.Teacher modefixedfrozen pretrained checkpoint.SDPO / SRPOKL-to-ref coef.0disabled.Mixing coef.α\alpha0.50.5student/teacher mix in JSD target.Full-logit distillyesdense top-kk+ tail-bucket.Top-kk100100vocab indices per token.PPO clip ratio2.02.0IS-ratio clip on distill term.EMA decay0.950.95EMA self-teacher decay.Teacher modeEMAEMA of policy weights.RLSDKL-to-ref coef.0disabled.PPO clipϵ\epsilon0.20.2outer PPO-clip onπθ/πold\pi_{\theta}/\pi_{\text{old}}.Evidence clampϵw\epsilon_{w}0.20.2clamp onwt=exp⁡(sign​(A)​(log⁡πT−log⁡πS))w_{t}=\exp(\mathrm{sign}(A)\,(\log\pi_{T}\!-\!\log\pi_{S})).Modulatorλ\lambda0.50.5λ=0\lambda{=}0: GRPO;λ=1\lambda{=}1: full modulation.λ\lambdaschedule0.5→00.5\!\to\!0in5050stepslinear decay.Teacher modesnapshot (τsync=10\tau_{\text{sync}}{=}10)hard copy every1010steps.RLCSD (ours)KL-to-ref coef.0disabled.PPO clipϵ\epsilon0.20.2both paths of Eq.17.Slopeτ\tau0.020.02tanh\tanhtemperature, Eq.9.Scaleλ\lambda0.50.5overall scale ofrtr_{t}, Eq.9.Mask thresholdδ\delta0.020.02mt=𝟙​[|rt|>δ]m_{t}{=}\mathbb{1}[|r_{t}|{>}\delta], Eq.10.Two-path weightη\eta1.01.0modulated-path weight, Eq.17.Neg. hintsKK44negative siblings averaged, Eq.7.Teacher modesnapshot (τsync=10\tau_{\text{sync}}{=}10)hard copy every1010steps.

Appendix CVocabulary Partitioning for Task / Style Statistics

This section specifies the token partition of math reasoning tasks used in Figures2and4.

For each decoded token, we first strip the leading-space marker used by the Qwen3 tokenizer and lowercase the result. The token is then sent through the following rules in order, returning at the first match:

1. (1)empty or whitespace-only→\rightarrowstyle;
2. (2)matches any of the math regexes (a digit\d; an arithmetic operator in+ - = * / < > × ÷≤\leq≥\geq≠\neq; a LaTeX command\[A-Za-z]+; a double backslash; or one of$ ^ _)→\rightarrowtask;
3. (3)the normalized form is in the math wordlist{mod, prime, factor, gcd, lcm, log, ln, sin, cos, tan, exp, integral, sqrt, boxed, frac, sum, prod, pi, alpha, beta, gamma, theta, delta, lambda, mu, sigma, infty, leq, geq, neq, cdot, times, div}→\rightarrowtask;
4. (4)pure punctuation or a literal newline token (\n,\\n)→\rightarrowstyle;
5. (5)the normalized form is in the discourse wordlist (connectivestherefore, so, thus, hence, then, because, since; hedgeswait, maybe, perhaps, seems, okay, ok, well, now, first, next, finally, actually, alternatively, however; scaffoldingstep, answer, let, lets; closed-class function wordsis, are, us, we, the, a, an, of, to, for, in, on, by, at, as, and, or, but, if, yes, no, this, that, these, those, it, its, be, been, being, have, has, had, do, does, did, will, would, should, could, can, may)→\rightarrowstyle;
6. (6)otherwise→\rightarrowneutral.

Appendix DComputational Cost Analysis

RLCSD introduces additional teacher-side computation compared with one-sided self-distillation methods, since it evaluates the teacher on both a correct hint and multiple wrong hints to form the contrastive signal. This extra cost is reflected in theteacher_log_probtime in Table9: RLCSD spends13.4613.46s per step on teacher scoring, compared with about99s for one-sided methods such as RLSD and OPSD.

However, this overhead is small compared with the dominant cost of autoregressive rollout generation. RLCSD spends473.89473.89s per step ongen, so the extra teacher scoring is only a minor fraction of the generation cost. In other words, although RLCSD requires more teacher forwards, these forward-only evaluations add relatively little wall-clock overhead compared with sampling long reasoning trajectories.

More importantly, the end-to-end step time remains favorable. As shown in Table9, RLCSD is thethird fastestmethod overall, behind only GRPO and RLSD. In particular, it is faster than dense-distillation baselines such as OPSD, SDPO, and SRPO. A likely reason is that these methods perform dense token-level distillation over the vocabulary during training, which makes the optimization stage itself much more expensive than the sampled-token modulation used in RLCSD and RLSD.

Table 9:Average wall-clock time per training step (seconds) on math reasoning task, using Qwen3-4B as the base model. RLCSD spends more time onteacher_log_probbecause it uses additional wrong-hint teacher forwards, but this overhead is small relative to rollout generation (gen). In total step time, RLCSD remains the third fastest method overall.