Multi-Rollout On-Policy Distillation via Peer Successes and Failures

# Multi-Rollout On-Policy Distillation via Peer Successes and Failures
Source: [https://arxiv.org/html/2605.12652](https://arxiv.org/html/2605.12652)
Weichen Yu1,2, Xiaomin Li2, Yizhou Zhao1, Xiaoze Liu3, Ruowang Zhang3, Haixin Wang2, Yinyi Luo1, Chen Henry Wu1, Gaurav Mittal2, Matt Fredrikson1, Yu Hu2 1Carnegie Mellon University2Microsoft3Purdue University wyu3@andrew\.cmu\.edu

###### Abstract

Large language models are often post\-trained with sparse verifier rewards, which indicate whether a sampled trajectory succeeds but provide limited guidance about where reasoning succeeds or fails\. On\-policy distillation \(OPD\) offers denser token\-level supervision by training on student\-generated trajectories, yet existing methods typically distill each rollout independently and ignore the other attempts sampled for the same prompt\. We introduce Multi\-Rollout On\-Policy Distillation \(MOPD\), a peer\-conditioned distillation framework that uses the student’s local rollout group to construct more informative teacher signals\. MOPD conditions the teacher on both successful and failed peer rollouts: successes provide positive evidence for valid reasoning patterns, while failures provide structured negative evidence about plausible mistakes to avoid\. We study two peer\-context constructions: positive peer imitation and contrastive success–failure conditioning\. Experiments on competitive programming, mathematical reasoning, scientific question answering, and tool\-use benchmarks show that MOPD consistently improves over standard on\-policy baselines\. Further teacher\-signal analysis shows that mixed success–failure contexts better align teacher scores with verifier rewards, indicating that the gains arise from more faithful, instance\-adaptive supervision\. These results indicate that effective on\-policy distillation should exploit the student’s multi\-rollout trial\-and\-error behavior rather than treating rollouts as isolated samples\. Code is available at[https://github\.com/viviable/mopd\_code](https://github.com/viviable/mopd_code)\.

## 1Introduction

Large language models \(LLMs\) are often post\-trained with reinforcement learning from verifiable rewards, where sampled solutions are scored by answer checkers, unit tests, or task\-specific verifiersGuoet al\.\([2025](https://arxiv.org/html/2605.12652#bib.bib2)\); Schulmanet al\.\([2017](https://arxiv.org/html/2605.12652#bib.bib56)\); Shaoet al\.\([2024](https://arxiv.org/html/2605.12652#bib.bib55)\)\. Although effective, these rewards are usually sparse: they indicate whether an entire trajectory succeeds or fails, but provide little guidance about which tokens or reasoning steps caused the outcomeYueet al\.\([2025](https://arxiv.org/html/2605.12652#bib.bib64)\); Chanet al\.\([2024](https://arxiv.org/html/2605.12652#bib.bib54)\)\. Distillation and self\-distillation provide a complementary form of supervision by converting teacher or privileged\-model judgments into dense token\-level signalsZelikmanet al\.\([2022](https://arxiv.org/html/2605.12652#bib.bib12)\); Gulcehreet al\.\([2023](https://arxiv.org/html/2605.12652#bib.bib13)\); Singhet al\.\([2024](https://arxiv.org/html/2605.12652#bib.bib37)\); Chenet al\.\([2024](https://arxiv.org/html/2605.12652#bib.bib38)\); Yanget al\.\([2024](https://arxiv.org/html/2605.12652#bib.bib39)\)\. On\-policy distillation further aligns this supervision with the student’s own behavior by training on trajectories sampled from the student policyAgarwalet al\.\([2024](https://arxiv.org/html/2605.12652#bib.bib1)\); Guet al\.\([2024](https://arxiv.org/html/2605.12652#bib.bib21)\); Koet al\.\([2024](https://arxiv.org/html/2605.12652#bib.bib22)\)\.

Despite this advantage, existing OPD and on\-policy self\-distillation methods underuse a key source of information already available during training: multiple rollouts generated for the same problem instance\. In many pipelines, each rollout is distilled independently, so the teacher signal for one trajectory is computed without access to the other attempts sampled from the same student\. This design discards the local structure of the rollout group\. For reasoning\-intensive tasks, such local structure is highly informative: successful rollouts reveal valid solution strategies, while failed rollouts expose plausible but incorrect reasoning paths, missing constraints, formatting errors, or execution mistakes\. Treating rollouts independently prevents the teacher from comparing these alternatives and identifying where an unsuccessful trajectory diverges from a successful one\.

We proposeMulti\-Rollout On\-Policy Distillation \(MOPD\), a peer\-conditioned distillation framework that explicitly uses the student’s rollout group to construct teacher signals\. For each prompt, the student samples multiple on\-policy trajectories, which are scored by a verifier and partitioned into successful and failed sets\. MOPD then conditions the teacher on peer rollouts from the same problem instance when supervising a target trajectory\. In this formulation, successful rollouts act as peer experts that provide positive evidence about valid reasoning patterns, while failed rollouts serve as structured negative evidence that identifies misleading solution paths the student should avoid\.

The key insight is thatmulti\-rollout sampling creates a local trial\-and\-error space around each problem\. Instead of viewing each trajectory in isolation, the teacher can compare the current rollout against peer successes and failures, using this contrast to produce more targeted supervision\. This turns OPD from independent imitation of individual trajectories into a comparative learning process: the teacher is no longer only a global expert, but also a local diagnostic model that can recognize instance\-specific errors and distinguish superficially plausible failures from correct solutions\. We instantiate this idea through two peer\-context construction strategies: positive peer imitation and contrastive success–failure conditioning\.

Moreover, prior methods often assume that a self\-teacher becomes a reliable source of supervision once it is conditioned on privileged information, such as ground\-truth outcomes, verified successful answers, hints, or environment feedback\. However, this assumption is usually evaluated only indirectly through downstream training performance\. Moreover, directly asking whether the privileged teacher can recover the correct answer is not a reliable diagnostic: if the privileged context already contains answer\-related information or verifier feedback, the self\-teacher may appear accurate by exploiting shortcuts rather than by producing a faithful supervision signal over the student’s own trajectories\.What matters for self distillation is not merely whether the self\-teacher knows the answer, but whether its token\-level or trajectory\-level preferences are aligned with correctness among the rollouts the student actually generates\.

![Refer to caption](https://arxiv.org/html/2605.12652v1/x1.png)Figure 1:MOPD Illustration\.To directly examine whether peer conditioning improves the self\-teacher signal itself, we introduce an analysis of self\-teacher signal quality\. For each prompt, we fix a set of student\-generated rollouts containing both successful and failed attempts, vary only the context shown to the self\-teacher, and compare the self\-teacher’s normalized logits or scores with ground\-truth verifier rewards\. A better self\-teacher signal should rank successful rollouts above failed ones, correlate more strongly with reward, and better separate correct from incorrect candidates\. This analysis avoids the confound of answer\-recovery tests and provides a direct measure of whether privileged contexts produce supervision that is actually aligned with correctness\.

Empirically, MOPD improves over standard on\-policy baselines across most scenarios on competitive programming, mathematical reasoning, scientific question answering, and tool\-use benchmarks\. The strongest gains arise from mixed peer contexts that combine successful and failed rollouts, supporting the hypothesis that contrastive evidence from peer rollout is more useful than positive demonstrations alone\. Our self\-teacher signal analysis further shows that the two\-success\-plus\-one\-failure context achieves the strongest alignment with verifier rewards across ranking and discrimination metrics, and our ablations show the same context yields the best compact downstream performance\. These results indicate that effective distillation should exploit the student’s multi\-rollout trial\-and\-error behavior, not only the strength of the teacher\.

## 2Related Work

#### On\-Policy Distillation

Knowledge distillation transfers a teacher’s predictive distribution to a smaller student, but off\-policy training creates a distribution mismatch: the student is supervised on states it never visits at inferenceHintonet al\.\([2015](https://arxiv.org/html/2605.12652#bib.bib15)\); Kim and Rush \([2016](https://arxiv.org/html/2605.12652#bib.bib16)\); Rosset al\.\([2011](https://arxiv.org/html/2605.12652#bib.bib19)\)\. On\-policy distillation addresses this by sampling trajectories from the student’s own policy and computing teacher supervision on those same trajectoriesAgarwalet al\.\([2024](https://arxiv.org/html/2605.12652#bib.bib1)\); Guet al\.\([2024](https://arxiv.org/html/2605.12652#bib.bib21)\); Koet al\.\([2024](https://arxiv.org/html/2605.12652#bib.bib22)\)\. Within this framework, work has focused on divergence objectives, token reweighting, and interpolation between on\- and off\-policy regimesWenet al\.\([2023](https://arxiv.org/html/2605.12652#bib.bib24)\); Huanget al\.\([2025](https://arxiv.org/html/2605.12652#bib.bib28)\); Zhanget al\.\([2026a](https://arxiv.org/html/2605.12652#bib.bib31)\); Jinet al\.\([2026](https://arxiv.org/html/2605.12652#bib.bib36)\)\. A parallel thread replaces an external teacher with self\-distillation: one copy of the model is conditioned on privileged context—solutions, hints, or environment feedback—and supervises a second copy that lacks itHübotteret al\.\([2026](https://arxiv.org/html/2605.12652#bib.bib8)\); Yeet al\.\([2026](https://arxiv.org/html/2605.12652#bib.bib4)\); Zhaoet al\.\([2026b](https://arxiv.org/html/2605.12652#bib.bib5)\); Penalozaet al\.\([2026](https://arxiv.org/html/2605.12652#bib.bib41)\); Yanget al\.\([2026](https://arxiv.org/html/2605.12652#bib.bib6)\)\. Information can also be aggregated across multiple sampled solutions through consistency, voting, or committee teachersWanget al\.\([2023](https://arxiv.org/html/2605.12652#bib.bib45)\); Muennighoffet al\.\([2025](https://arxiv.org/html/2605.12652#bib.bib46)\); Liet al\.\([2025b](https://arxiv.org/html/2605.12652#bib.bib47)\)\. However, the teacher signal for each rollout is constructed independently, leaving cross\-rollout structure within a sampling group unexploited\.

#### Multi\-Rollout Reinforcement Learning and Verifier\-Guided Improvement

Post\-training with verifiable rewards replaces learned reward models with programmatic checkers and updates the policy from group\-relative advantages computed over multiple sampled rollouts per promptShaoet al\.\([2024](https://arxiv.org/html/2605.12652#bib.bib55)\); Wenet al\.\([2026](https://arxiv.org/html/2605.12652#bib.bib62)\); Ouyanget al\.\([2022](https://arxiv.org/html/2605.12652#bib.bib53)\); Rafailovet al\.\([2023](https://arxiv.org/html/2605.12652#bib.bib58)\)\. Process reward models provide denser credit assignment by grading intermediate reasoning stepsCobbeet al\.\([2021](https://arxiv.org/html/2605.12652#bib.bib71)\); Setluret al\.\([2025](https://arxiv.org/html/2605.12652#bib.bib72)\); Zhanget al\.\([2025c](https://arxiv.org/html/2605.12652#bib.bib73)\), and hybrid pipelines couple these signals with on\-policy distillationXuet al\.\([2025b](https://arxiv.org/html/2605.12652#bib.bib76);[a](https://arxiv.org/html/2605.12652#bib.bib77)\); Zhanget al\.\([2026b](https://arxiv.org/html/2605.12652#bib.bib79)\)\. While these methods exploit multiple rollouts per prompt through advantage normalization or rejection sampling, the supervisory signal for any single trajectory is computed without conditioning on the contents of the others\. The work most adjacent to ours treats successes and failures as complementary evidence at the level of advantages or filters, but stops short of routing both into a single peer\-conditioned distillation target\. Extended related work for both topics is in[appendix˜F](https://arxiv.org/html/2605.12652#A6)\.

## 3Preliminaries

We consider the problem of distilling reasoning capabilities from a teacher model into a student model under an on\-policy sampling regime\. Letx∈𝒳x\\in\\mathcal\{X\}denote an input problem or prompt, and lety=\(y1,…,yT\)y=\(y\_\{1\},\\ldots,y\_\{T\}\)denote an output trajectory generated autoregressively by a student policyπθ\\pi\_\{\\theta\}\. At decoding steptt, we write the prefix asy<ty\_\{<t\}and the token\-level context asct=\(x,y<t\)c\_\{t\}=\(x,y\_\{<t\}\)\. The student distribution isπθ\(⋅∣ct\)\\pi\_\{\\theta\}\(\\cdot\\mid c\_\{t\}\), while the teacher distribution is denoted byqϕ\(⋅∣ct\)q\_\{\\phi\}\(\\cdot\\mid c\_\{t\}\), possibly conditioned on additional privileged information\. In prior self\-distillation methods, the privileged information available to the teacher typically includes ground\-truth outcomes, verified successful answersYeet al\.\([2025](https://arxiv.org/html/2605.12652#bib.bib3);[2026](https://arxiv.org/html/2605.12652#bib.bib4)\); Zhaoet al\.\([2026b](https://arxiv.org/html/2605.12652#bib.bib5)\); Yanget al\.\([2026](https://arxiv.org/html/2605.12652#bib.bib6)\), or environment feedbackHübotteret al\.\([2026](https://arxiv.org/html/2605.12652#bib.bib8)\)\.

#### On\-policy distillation\.

Offline distillation trains the student on trajectories sampled from a teacher or from a fixed dataset\. This creates a distribution mismatch: during inference, the student may visit states that were not observed during training\. On\-policy distillation addresses this issue by first sampling trajectories from the student itself and then obtaining teacher supervision on those same trajectories\. Formally, for each promptxx, the student generatesy∼πθ\(⋅∣x\)\.y\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\.The teacher then provides token\-level targets along the student\-induced trajectory\. A standard on\-policy distillation loss can be written as

ℒOPD=𝔼x∼𝒟,y∼πθ\(⋅∣x\)\[∑t=1TD\(πθ\(⋅∣x,y<t\)∥qϕ\(⋅∣x,y<t\)\)\],\\mathcal\{L\}\_\{\\mathrm\{OPD\}\}=\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\},\\,y\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\left\[\\sum\_\{t=1\}^\{T\}D\\\!\\left\(\\pi\_\{\\theta\}\(\\cdot\\mid x,y\_\{<t\}\)\\,\\\|\\,q\_\{\\phi\}\(\\cdot\\mid x,y\_\{<t\}\)\\right\)\\right\],\(1\)whereD​\(⋅\)D\(\\cdot\)denotes a divergence loss, either KL, reverse KL, or Jensen\-Shannon \(JS\), as detailed in[appendix˜B](https://arxiv.org/html/2605.12652#A2)\. This formulation provides dense token\-level supervision on the student’s visited states\.

Nevertheless, standard OPD treats each sampled trajectory independently\. When multiple rollouts are generated for the same prompt, the teacher distribution for one rollout is usually computed without access to the successes and failures observed in the other rollouts\. This prevents the teacher from exploiting local, instance\-specific evidence contained in the rollout group\.

## 4Multi\-Rollout On\-Policy Distillation

![Refer to caption](https://arxiv.org/html/2605.12652v1/x2.png)Figure 2:MOPD Pipeline\.We proposeMulti\-Rollout On\-Policy Distillation\(MOPD\), a peer\-conditioned distillation framework that exploits the local structure of multiple on\-policy rollouts generated for the same question as in[fig\.˜2](https://arxiv.org/html/2605.12652#S4.F2)\. Standard on\-policy distillation supervises each sampled trajectory independently: the teacher evaluates the student’s current trajectory without direct access to the alternative attempts produced for the same instance\. This ignores a useful source of instance\-specific information\. In reasoning tasks, successful rollouts reveal valid solution paths, while failed rollouts expose plausible but incorrect branches\. MOPD uses this local contrast to construct a more informative teacher signal\.

At a high level, MOPD first samples a group of student rollouts for each prompt, scores them with a verifier, partitions them into successful and failed trajectories, and then conditions the teacher on peer information from the same rollout group\. The resulting teacher distribution is no longer only a global distribution over next tokens; it becomes a*peer\-conditioned*distribution that can compare the current trajectory against other local attempts\. This allows the teacher to provide sharper token\-level supervision, especially near branching points where failed trajectories diverge from successful ones\.

### 4\.1Multi\-rollout peer conditioning

For each promptxx, the student samples a group ofNNon\-policy trajectories,

Y\(x\)=\{y\(i\)\}i=1N,y\(i\)∼πθ\(⋅∣x\)\.Y\(x\)=\\\{y^\{\(i\)\}\\\}\_\{i=1\}^\{N\},\\qquad y^\{\(i\)\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\.\(2\)Each trajectory is evaluated by a verifier or reward functionRR, and the rollout group is partitioned according to a success thresholdτ\\tau:

Y\+​\(x\)=\{y\(i\):R​\(x,y\(i\)\)≥τ\},Y−​\(x\)=\{y\(i\):R​\(x,y\(i\)\)<τ\}\.Y^\{\+\}\(x\)=\\\{y^\{\(i\)\}:R\(x,y^\{\(i\)\}\)\\geq\\tau\\\},\\qquad Y^\{\-\}\(x\)=\\\{y^\{\(i\)\}:R\(x,y^\{\(i\)\}\)<\\tau\\\}\.\(3\)HereY\+​\(x\)Y^\{\+\}\(x\)contains peer successes andY−​\(x\)Y^\{\-\}\(x\)contains peer failures\. For a target rollouty\(i\)y^\{\(i\)\}, we exclude the target itself and define

Y−i\+​\(x\)=Y\+​\(x\)∖\{y\(i\)\},Y−i−​\(x\)=Y−​\(x\)∖\{y\(i\)\}\.Y^\{\+\}\_\{\-i\}\(x\)=Y^\{\+\}\(x\)\\setminus\\\{y^\{\(i\)\}\\\},\\qquad Y^\{\-\}\_\{\-i\}\(x\)=Y^\{\-\}\(x\)\\setminus\\\{y^\{\(i\)\}\\\}\.\(4\)
MOPD constructs a peer context from these two sets:Ci​\(x\)=𝒞v​\(x,y\(i\),Y−i\+​\(x\),Y−i−​\(x\)\),C\_\{i\}\(x\)=\\mathcal\{C\}\_\{v\}\\left\(x,y^\{\(i\)\},Y^\{\+\}\_\{\-i\}\(x\),Y^\{\-\}\_\{\-i\}\(x\)\\right\),where𝒞v\\mathcal\{C\}\_\{v\}denotes a context construction rule\. The indexvvspecifies which peer\-conditioning strategy is used\. Given this context, the teacher distribution for the target trajectory becomesqϕpeer\(⋅∣x,y<t\(i\)\)=qϕ\(⋅∣x,Ci\(x\),y<t\(i\)\)\.q\_\{\\phi\}^\{\\mathrm\{peer\}\}\(\\cdot\\mid x,y^\{\(i\)\}\_\{<t\}\)=q\_\{\\phi\}\(\\cdot\\mid x,C\_\{i\}\(x\),y^\{\(i\)\}\_\{<t\}\)\.Compared with a standard teacher distributionqϕ\(⋅∣x,y<t\(i\)\)q\_\{\\phi\}\(\\cdot\\mid x,y^\{\(i\)\}\_\{<t\}\), the peer\-conditioned teacher can use other rollouts from the same instance as local evidence\. Successful peers provide positive support for correct reasoning patterns, while failed peers provide negative evidence about misleading or invalid trajectories\.

The rollout\-level MOPD loss is

ℒMOPD\(i\)=∑t=1TiD\(πθ\(⋅∣x,y<t\(i\)\)∥qϕpeer\(⋅∣x,y<t\(i\)\)\),\\mathcal\{L\}^\{\(i\)\}\_\{\\mathrm\{MOPD\}\}=\\sum\_\{t=1\}^\{T\_\{i\}\}D\\\!\\left\(\\pi\_\{\\theta\}\(\\cdot\\mid x,y^\{\(i\)\}\_\{<t\}\)\\,\\\|\\,q\_\{\\phi\}^\{\\mathrm\{peer\}\}\(\\cdot\\mid x,y^\{\(i\)\}\_\{<t\}\)\\right\),\(5\)whereDDis a token\-level divergence\. In our implementation,DDis instantiated as reverse KL for math tasks, and Jensen–Shannon divergence for other tasks\.

### 4\.2Peer\-context construction

The main design choice in MOPD is the construction ofCi​\(x\)C\_\{i\}\(x\)\. We compare two natural choices: a positive\-only context that exposes the teacher to additional successful peers, and a contrastive context that adds failed peers as structured negative evidence\. This comparison isolates whether failure\-as\-negative\-evidence sharpens the teacher signal beyond success\-only conditioning\.

Algorithm 1MOPD Algorithm1:

𝒟\\mathcal\{D\},

πθ\\pi\_\{\\theta\},

qϕq\_\{\\phi\},

RR,

NN,

τ\\tau,

𝒞v\\mathcal\{C\}\_\{v\},

α\\alpha
2:Updated student

πθ\\pi\_\{\\theta\}
3:foreach training iterationdo

4:foreach

x∈ℬx\\in\\mathcal\{B\}do

5:Sample rollout group

Y​\(x\)Y\(x\)as in Eq\. \([2](https://arxiv.org/html/2605.12652#S4.E2)\)\.

6:Score and partition rollouts as in Eq\. \([3](https://arxiv.org/html/2605.12652#S4.E3)\)\.

7:Construct

Ci​\(x\)C\_\{i\}\(x\), query

qϕpeerq\_\{\\phi\}^\{\\mathrm\{peer\}\},

8:

9:compute

ℒMOPD\(i\)\\mathcal\{L\}^\{\(i\)\}\_\{\\mathrm\{MOPD\}\}\.

10:endfor

11:Update

πθ\\pi\_\{\\theta\}by minimizing

12:

13:

ℒMOPD=1N​∑iℒMOPD\(i\)\\mathcal\{L\}\_\{\\mathrm\{MOPD\}\}=\\tfrac\{1\}\{N\}\\sum\_\{i\}\\mathcal\{L\}^\{\(i\)\}\_\{\\mathrm\{MOPD\}\}\.

14:endfor

15:return

πθ\\pi\_\{\\theta\}

#### Positive peer imitation\.

The first strategy conditions the teacher on successful peer trajectories only\. Lety⋆y^\{\\star\}denote a primary successful rollout and lety\+y^\{\+\}denote an additional successful peer, both selected fromY−i\+​\(x\)Y^\{\+\}\_\{\-i\}\(x\)when available\. The context is

Ci\(1\)​\(x\)=Template​\(y⋆\)⊕SuccessTemplate​\(y\+\),C\_\{i\}^\{\(1\)\}\(x\)=\\mathrm\{Template\}\(y^\{\\star\}\)\\oplus\\mathrm\{SuccessTemplate\}\(y^\{\+\}\),\(6\)where⊕\\oplusdenotes text concatenation\. If only one successful peer is available, the context reduces to the primary successful trajectory:Ci\(1\)​\(x\)=Template​\(y⋆\)\.C\_\{i\}^\{\(1\)\}\(x\)=\\mathrm\{Template\}\(y^\{\\star\}\)\.

This variant treats successful rollouts as peer experts\. It encourages the teacher to evaluate the target trajectory in light of alternative correct reasoning paths\. Compared with using a single reference solution, positive peer imitation broadens the set of correct trajectories the teacher conditions on, so the distilled student is anchored to multiple valid derivation styles rather than one\.

#### Contrastive success–failure conditioning\.

The second strategy conditions the teacher on both successful and failed peer trajectories\. Given successful peersy⋆,y\+∈Y−i\+​\(x\)y^\{\\star\},y^\{\+\}\\in Y^\{\+\}\_\{\-i\}\(x\)and a failed peery−∈Y−i−​\(x\)y^\{\-\}\\in Y^\{\-\}\_\{\-i\}\(x\), we define

Ci\(2\)​\(x\)=Template​\(y⋆\)⊕SuccessTemplate​\(y\+\)⊕FailureTemplate​\(y−\)\.C\_\{i\}^\{\(2\)\}\(x\)=\\mathrm\{Template\}\(y^\{\\star\}\)\\oplus\\mathrm\{SuccessTemplate\}\(y^\{\+\}\)\\oplus\\mathrm\{FailureTemplate\}\(y^\{\-\}\)\.\(7\)
This variant provides the teacher with contrastive local evidence\. Successful rollouts indicate what the target trajectory should resemble, while failed rollouts identify plausible but incorrect reasoning patterns that should be avoided\. The contrastive context can therefore sharpen token\-level supervision by helping the teacher distinguish between superficially similar correct and incorrect trajectories\. In tasks with sparse terminal rewards, this is especially useful because the failed peer exposes where an otherwise plausible solution path becomes invalid\.

#### Unified view

The two peer\-context variants share a unified gated form:Ci​\(x\)=Template​\(y⋆\)⊕λ\+​A\+​\(x\)⊕λ−​A−​\(x\)C\_\{i\}\(x\)=\\mathrm\{Template\}\(y^\{\\star\}\)\\oplus\\lambda\_\{\+\}A\_\{\+\}\(x\)\\oplus\\lambda\_\{\-\}A\_\{\-\}\(x\), whereA\+​\(x\)A\_\{\+\}\(x\)denotes additional successful demonstrations,A−​\(x\)A\_\{\-\}\(x\)denotes failed demonstrations, and the gatesλ\+,λ−∈\{0,1\}\\lambda\_\{\+\},\\lambda\_\{\-\}\\in\\\{0,1\\\}determine which type of peer evidence is included\. Algorithm[1](https://arxiv.org/html/2605.12652#alg1)summarizes the MOPD training procedure\.

## 5Experiments

We evaluate whether augmenting the teacher with peer rollouts improves post\-training in reasoning\-intensive domains\. Our method is compared against the base model, GRPO, and SDPO\. SDPO conditions its self\-teacher on environment feedback when available; our method instead conditions the teacher on peer rollouts—both successful and failed—from the same prompt\. This allows us to test whether peer\-augmented teacher information provides additional benefits beyond scalar\-reward RL and feedback\-conditioned self\-distillation\.

### 5\.1Experimental Setup

#### Benchmarks\.

We consider four families of tasks that require multi\-step reasoning or precise execution\.Science QA\.We use the reasoning subsets \(L3\) from SciKnowEval\(Feng and et al\.,[2024](https://arxiv.org/html/2605.12652#bib.bib9)\), covering undergraduate\-level questions in biology, chemistry, physics, and materials science\.Tool Use\.We evaluate tool\-use reasoning on ToolAlpaca\(Tang and et al\.,[2023](https://arxiv.org/html/2605.12652#bib.bib10)\), where the model must map a user request and an API specification to the correct tool call\.Code\.We evaluate on LiveCodeBench v6 \(LCBv6\)\(Jainet al\.,[2025](https://arxiv.org/html/2605.12652#bib.bib11)\), which contains 131 contest\-style programming problems released between February and May 2025\. This split provides a recent and challenging evaluation of generalization to unseen coding problems\.Math\.For math training, we use a filtered subset of DeepMath\(Heet al\.,[2025](https://arxiv.org/html/2605.12652#bib.bib7)\), selecting 57K examples with difficulty level at least 6 following\(Yanget al\.,[2026](https://arxiv.org/html/2605.12652#bib.bib6)\)\. We evaluate on AIME2024, AIME2025, and HMMT25\.

#### Metrics\.

For code and math, we report mean@8 and pass@8, following standard multi\-sample evaluation\. All results are reported as percentages\.Training protocol\.For each prompt, we sample a group of on\-policy rollouts and evaluate them using a task\-specific verifier or reward function\. The resulting rollouts are partitioned into successful and failed sets\. We then construct the privileged teacher context from peer rollouts, using two successful rollouts and one failed rollout unless otherwise specified\. This design exposes the teacher to both correct solution patterns and informative failure modes\. Across methods, we keep the rollout generation budget and optimization protocol fixed to ensure a fair comparison\.

Implementation details\.Training is implemented inverlwith asynchronous vLLM rollouts and FSDP\-based optimization\. We use a prompt batch size of 32 and sample 8 rollouts per prompt during training\. The actor learning rate is set to1×10−51\\times 10^\{\-5\}with 10 warmup steps\. For generation, we use a maximum prompt length of 2048 tokens and a maximum response length of 4096 tokens \(for math with 8192\)\. Because SDPO conditions on additional solution context, we reserve an extra 2048\-token headroom, yielding a total rollout context budget of 8192 tokens\. Validation evaluates 8 samples per prompt\. The configuration uses self\-distillation with mixing coefficientα=0\.5\\alpha=0\.5anddistillation\_topk=100\. We use FlashAttention\-2 with bfloat16 model weights for both actor and critic\. For training stability, we use reverse KL divergence for math tasks, and JS divergence otherwise\.

### 5\.2Main Results for Self\-Distillation

#### LiveCodeBench\.

Table[5](https://arxiv.org/html/2605.12652#S5.T5)reports results on LCBv6 for Qwen3\-4B and Qwen3\-8B\. LCBv6 provides structured environment feedback—such as compiler errors and test\-case verdicts—at the end of each rollout\. Our method consistently improves over the base model, GRPO, and SDPO at both model scales\. On Qwen3\-4B, our method improves over SDPO by large margin and on Qwen3\-8B, our method reaches 61\.82 mean@8 and 67\.23 pass@8\.Math reasoning\.Table[1](https://arxiv.org/html/2605.12652#S5.T1)shows results on AIME2024, AIME2025, and HMMT25 using Qwen3\-4B as the base model\. Our method substantially improves mean@8 compared to SDPO\. SDPO is designed for settings with rich environment feedback, but math tasks provide only a single binary reward signal from a verifier, which likely explains why it underperforms the baseline in this domain\. Peer\-augmented supervision is particularly effective for mathematical reasoning in self\-distillation, where alternative successful derivations and failed attempts provide complementary evidence about solution structure that a single verifier reward cannot\.Science QA\.Table[4](https://arxiv.org/html/2605.12652#S5.T4)shows MOPD achieves the best performance across all four domains, on science multi\-choice questions benchmarks\.Tool Use\.Tool Use requires precise mapping from natural\-language requests to structured API calls, our method outperforms\. Peer conditioning therefore extends beyond reasoning\-rich settings to tasks where the primary challenge is accurate structured\-output prediction\.Time Cost\.MOPD introduces acceptable computational overhead over SDPO, with total per\-step time increasing by only 9\.1% on LCB and 33\.3% on QA\-Chemistry, detailed breakdown results in[table˜9](https://arxiv.org/html/2605.12652#A5.T9)\.Number of ever\-success questions during training\.As in[fig\.˜3](https://arxiv.org/html/2605.12652#S5.F3), MOPD reaches a higher count earlier and finally than SDPO, indicating faster exploration of the solution space across training prompts\.

![Refer to caption](https://arxiv.org/html/2605.12652v1/figs/compare_self_distillation_uid_ever_success.png)Figure 3:Number of training data that have ever generated a correct answer in theNNrollout during training\.
#### Case Study\.

During training, we save the generated rollouts and compare them on the same question across training steps to provide a case study\. Additionally, after training for the same number of steps, we save checkpoints from both SDPO and MOPD, then sample from these checkpoints to evaluate whether each model can successfully solve the target problem and avoid failure patterns\. The details in[appendix˜C](https://arxiv.org/html/2605.12652#A3), showing thatMOPD and SDPO start from the same failure pattern early in training, but by repeatedly exposing the teacher to failed peers of the same bug family and explicitly signaling ’avoid this mistake,’ MOPD suppresses the error by checkpoint time— while baseline method, lacking this channel, does not\.

Table 1:Math reasoning results on AIME2024/2025 and HMMT25\. All methods start from the same Qwen3\-4B model and use the same generation budget\. Bold marks the best result in each column\.AIME2025AIME2024HMMT25 Feb\.HMMT25 Nov\.Methodmean@8pass@8mean@8pass@8mean@8pass@8mean@8pass@8Qwen3\-4B17\.9232\.5716\.8931\.605\.0610\.009\.5813\.91GRPO25\.8036\.6617\.0932\.0516\.9220\.9413\.1619\.53SDPO7\.8116\.667\.2916\.298\.5913\.6112\.0118\.49MOPD25\.4136\.2828\.5433\.1218\.5019\.5815\.8322\.47

### 5\.3Analysis of Self\-Teacher Signal Quality

![Refer to caption](https://arxiv.org/html/2605.12652v1/x3.png)Figure 4:Self\-teacher\-signal quality across seven context conditions\. Each panel reports an averaged prompt\-level metric\. Higher is better for all metrics except the Brier score\.Table 2:Science QA results\.MethodBio\.Chem\.Phys\.Mat\.Qwen3\-8B30\.8942\.9858\.4465\.59GRPO47\.3264\.2464\.1272\.09SDPO50\.6062\.9167\.3672\.34MOPD55\.6974\.2976\.2578\.59
Table 3:ToolUse results\.MethodQwen3\-8BGRPOSDPOMOPDTool Use59\.1160\.3263\.4566\.73

Table 4:Ablation on peer\-context construction for MOPD: which combination of successful and failed peer rollouts to inject into the self\-teacher context\.QA\-ChemistryLiveCodeBenchPeer context𝒞​\(u\)\\mathcal\{C\}\(u\)mean@8pass@8mean@8pass@81 successful solution51\.9058\.9249\.4964\.342 successful solutions52\.6259\.0255\.6264\.021 failure solution35\.6737\.6446\.7254\.861 success \+ 1 failure60\.5068\.1258\.9062\.012 success \+ 1 failure74\.2983\.5361\.8267\.238 solutions \(2 hours\)73\.5783\.1158\.4363\.688 solutions \(4 hours\)81\.2584\.4958\.9564\.71

#### Setup: self\-teacher signal evaluation\.

To understand why peer information improves training, we directly analyze the quality of the self\-teacher signal induced by different context constructions\. For each questionxx, we sample a set of rollouts\{y\(i\)\}i=1n\\\{y^\{\(i\)\}\\\}\_\{i=1\}^\{n\}from the policy of the base model and evaluate each response using both the task verifier and the self\-teacher model\. Each response is associated with three quantities:\(y\(i\),si,ri\),\(y^\{\(i\)\},s\_\{i\},r\_\{i\}\),wheresi∈ℝs\_\{i\}\\in\\mathbb\{R\}is the self\-teacher average token logprobsi=1Ti​∑t=1Tilog⁡qϕ​\(yt\(i\)∣x,Ci​\(x\),y<t\(i\)\)s\_\{i\}=\\frac\{1\}\{T\_\{i\}\}\\sum\_\{t=1\}^\{T\_\{i\}\}\\log q\_\{\\phi\}\\\!\\left\(y\_\{t\}^\{\(i\)\}\\mid x,\\,C\_\{i\}\(x\),\\,y\_\{<t\}^\{\(i\)\}\\right\)assigned to response\(y\(i\)\(y^\{\(i\)\},ri∈\[0,1\]r\_\{i\}\\in\[0,1\]is the ground truth reward\. The self\-teacher scoresis\_\{i\}is computed under each specific teacher privilege knowledge, whilerir\_\{i\}is a fixed property of the candidate rollout\. Thus, by holding the prompt and candidate responses fixed and varying only the teacher\-visible context, we can isolate how different types of peer information affect the reliability of the self\-teacher signal\. We consider several context conditions, including base \(no peer information\), successful peer rollouts, failed peer rollouts, combinations of successful and failed rollouts, and all the rollouts\. The analysis uses Qwen3\-4B model generated on the LiveCodeBench\-v6 benchmark\.

We evaluate each self\-teacher context condition with six complementary metrics covering ranking, discrimination, and calibration; full definitions are in[appendix˜A](https://arxiv.org/html/2605.12652#A1)\.Mean Spearman CorrelationandMean Kendall’sτ\\tauboth measure ordinal agreement between teacher scores and ground\-truth rewards across rollouts\.Pairwise Accuracydirectly reports the fraction of reward\-distinguishable rollout pairs that the teacher orders correctly\.Success AUCmeasures the probability that a successful rollout receives a higher teacher score than a failed one;Success Point\-Biserial Correlationquantifies the linear association between teacher scores and binary success labels; andSuccess Brier Score \(Sigmoid\)assesses probabilistic calibration by mapping teacher scores through a sigmoid and computing MSE against rewards\.

Table 5:LCBv6 results\.Qwen3\-4BQwen3\-8BMethodmean@8pass@8mean@8pass@8Base28\.8049\.3630\.9753\.02GRPO40\.7555\.4343\.6558\.72SDPO48\.8463\.2349\.4964\.34Ours57\.0165\.4861\.8267\.23![Refer to caption](https://arxiv.org/html/2605.12652v1/x4.png)
Figure 5:Diversity Analysis\.

#### Results and interpretation\.

Figure[4](https://arxiv.org/html/2605.12652#S5.F4)and Table[4](https://arxiv.org/html/2605.12652#S5.T4)together reveal how peer context composition affects teacher signal quality and downstream performance\. 1\) Successful peer rollouts are the primary driver of teacher\-signal quality: any condition containing at least one successful peer lifts pairwise accuracy from0\.130\.13at the no\-peer baseline to above0\.320\.32\. A failed peer alone barely moves the discrimination metrics\. 2\) Successful peers have a stronger effect on correctness\-related metrics—Spearmanρ\\rhorises from 0\.00 to above 0\.10 and Kendallτ\\taufrom 0\.00 to above 0\.13 when at least one successful solution is included—suggesting that seeing a correct peer helps the teacher better assess the target rollout’s quality\. 3\) Failed peers complement successful ones: adding a failure to a success\-only context lifts pairwise accuracy from0\.320\.32to0\.400\.40, showing that negative evidence sharpens decision boundaries that positive evidence alone leaves blurred\. 4\) Combining both types yields the best results: the “2 success \+ 1 failure” context achieves the highest score on55of the66ranking and discrimination metrics in the signal\-quality analysis, with a competitive Brier score, and the highest LCB downstream mean@88among the compact peer\-context settings\. 5\) We notice that the results for base are very low, for example, the pairwise accuracy and success AUC is only 0\.13\. We look into the generations of Qwen3\-4B and find that57%of prompts have no successful rollout \(Y\+​\(x\)=∅Y^\{\+\}\(x\)=\\emptyset\), explaining the near\-zero metrics in the hard bucket\. This confirms that correct solutions occupy extremely low\-probability regions of the model’s output distribution—the self\-teacher finds them genuinely surprising, which directly explains why model accuracy remains below 30% on lcb\-v6\. 6\) On LiveCodeBench, providing all available rollouts yields no additional gains beyond the compact two\-success\-plus\-one\-failure context; more peer rollouts do not strictly dominate a balanced contrastive pair on every benchmark\.

### 5\.4Solution Diversity Analysis

We investigate whether MOPD’s exposure to multiple peer rollouts during training leads to greater diversity in generated solutions\. We measure two axes of diversity that are well defined for code: Distinct\-1 and Distinct\-2 for within\-response lexical richness \(the vocabulary and bigram coverage inside a single program\), and AST\-node Jaccard distance for between\-response structural variation \(the difference between programs in their parsed syntax\-tree composition\)\. Both are computed on LiveCodeBench\-v6 checkpoints at step 50, with each model generating 8 responses per problem\. Detailed implementation and metric definitions are provided in[appendix˜D](https://arxiv.org/html/2605.12652#A4)\.

Figure[5](https://arxiv.org/html/2605.12652#S5.F5)shows that MOPD produces substantially more diverse outputs than both the base model and SDPO across all three metrics\. On Distinct\-1 and Distinct\-2, SDPO remains close to the untrained base model \(0\.027 vs\. 0\.025; 0\.110 vs\. 0\.111\), suggesting that its training signal does not encourage lexical variation\. MOPD, by contrast, achieves nearly double the Distinct\-1 score of SDPO \(0\.052 vs\. 0\.027\) and a notable gain in Distinct\-2 \(0\.141 vs\. 0\.110\)\. A similar pattern holds at the structural level: AST Jaccard distance increases from 0\.110 \(SDPO\) to 0\.129 \(MOPD\), indicating that MOPD generates solutions with more structurally distinct implementations\. Conditioning the teacher on failed peer rollouts therefore improves both correctness and the lexical and structural diversity of the student’s outputs\.

### 5\.5Does Peer Information Improve Teacher–Student Distillation?

We next study whether peer information remains useful in a teacher–student setting, where the supervision signal is provided by a larger teacher model and used to train a smaller student\. Our general knowledge distillation \(GKD\) pipeline follows an offline teacher\-forcing paradigm: for each input prompt, a larger teacher model generates 8 completions, which are then converted into supervised fine\-tuning examples\. The student is trained by maximum\-likelihood learning on these teacher responses\. In on\-policy distillation \(OPD\), the teacher provides supervision conditioned on on\-policy student responses\. Ours TS method further augments this teacher context with peer information: successful and failed peer rollouts\.

Table 6:Teacher–student on\-policy distillation results\. All methods share the same rollout budget and hyperparameter\.ToolUseLiveCodeBenchMethodmean@8pass@8mean@8pass@8Qwen3\-4B \(S\)53\.2758\.5828\.8049\.36Qwen3\-14B \(T\)56\.0260\.8946\.7565\.16GKD \(TS\)55\.9160\.0546\.0556\.67OPD \(TS\)59\.6361\.5860\.9267\.30SDPO \(Self\)62\.0465\.9748\.8463\.23Ours \(TS\)66\.4468\.6161\.9267\.35Ours \(Self\)64\.6166\.8157\.0165\.48Table[6](https://arxiv.org/html/2605.12652#S5.T6)reveals several trends in peer\-conditioned distillation\. 1\) On ToolUse and LiveCodeBench, traditional teacher\-student setups with on\-policy strategies still outperform self\-distillation\. 2\) When the teacher is given richer context through peer information, distillation can surpass the teacher model’s own performance—a result that underscores the value of privileged observations at inference time\. 3\) Adding peer information consistently improves both the self\-distillation and teacher–student settings, confirming that the benefit is not specific to a particular pairing of model sizes\. 4\) On\-policy methods outperform their off\-policy counterparts across both domains, consistent with the distribution\-shift argument motivating on\-policy training\.

## 6Conclusion

We introduced MOPD, a peer\-conditioned on\-policy distillation framework that uses multiple student rollouts from the same prompt to construct more informative teacher signals\. By conditioning the teacher on successful and failed peer rollouts, MOPD turns independent trajectory supervision into a comparative learning process: successes provide positive evidence for valid reasoning, while failures expose error patterns to avoid\. Experiments across code, math, science QA, and tool use show that MOPD consistently improves over standard on\-policy baselines\. Both ablations and teacher\-signal analysis indicate that mixed success–failure contexts produce the most faithful supervision, better aligning teacher scores with verifier rewards and downstream performance\. Effective distillation should exploit the student’s rollout group as a structured collection of successes and failures, rather than treating each trajectory in isolation\.

## References

- On\-policy distillation of language models: learning from self\-generated mistakes\.InThe twelfth international conference on learning representations,Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2605.12652#S1.p1.1),[§2](https://arxiv.org/html/2605.12652#S2.SS0.SSS0.Px1.p1.1)\.
- R\. Bao, D\. Yu, K\. Fan, and M\. Liao \(2025\)Fixing distribution shifts of llm self\-critique via on\-policy self\-play training\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 17680–17700\.Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px2.p1.1)\.
- N\. Boizard, K\. E\. Haddad, C\. Hudelot, and P\. Colombo \(2025\)Towards Cross\-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs\.Transactions on Machine Learning Research\.External Links:[Link](https://arxiv.org/abs/2402.12030)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px1.p1.1)\.
- D\. Busbridge, A\. Shidani, F\. Weers, J\. Ramapuram, E\. Littwin, and R\. Webb \(2025\)Distillation Scaling Laws\.InProceedings of the International Conference on Machine Learning \(ICML\),External Links:[Link](https://arxiv.org/abs/2502.08606)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px1.p1.1)\.
- S\. Casper, X\. Davies, C\. Shi, T\. K\. Gilbert, J\. Scheurer, J\. Rando, R\. Freedman, T\. Korbak, D\. Lindner, P\. Freire,et al\.\(2023\)Open problems and fundamental limitations of reinforcement learning from human feedback\.Transactions on Machine Learning Research\.Note:Survey Certification, Featured CertificationExternal Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=bx24KpJ4Eb)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px3.p1.1)\.
- A\. J\. Chan, H\. Sun, S\. Holt, and M\. Van Der Schaar \(2024\)Dense reward for free in reinforcement learning from human feedback\.arXiv preprint arXiv:2402\.00782\.Cited by:[§1](https://arxiv.org/html/2605.12652#S1.p1.1)\.
- Z\. Chen, Y\. Deng, H\. Yuan, K\. Ji, and Q\. Gu \(2024\)Self\-play fine\-tuning converts weak language models to strong language models\.InInternational Conference on Machine Learning,pp\. 6621–6642\.Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2605.12652#S1.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2605.12652#S2.SS0.SSS0.Px2.p1.1)\.
- Q\. Deng, K\. Chen, M\. Zhang, and Z\. Xu \(2026\)HiPO: self\-hint policy optimization for RLVR\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=rcb20pHmT1)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px3.p1.1)\.
- K\. Ding \(2026\)HDPO: Hybrid Distillation Policy Optimization via Privileged Self\-Distillation\.arXiv preprint arXiv:2603\.23871\.External Links:[Link](https://arxiv.org/abs/2603.23871)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px2.p1.1)\.
- A\. Dixit, T\. Liang, and J\. Telang \(2026\)Project aletheia: verifier\-guided distillation of backtracking for small language models\.InLogical and Symbolic Reasoning in Language Models @ AAAI 2026,External Links:[Link](https://openreview.net/forum?id=4zupWb2xmE)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px3.p1.1)\.
- Y\. Feng and et al\. \(2024\)SciKnowEval: evaluating multi\-level scientific knowledge of large language models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Cited by:[§5\.1](https://arxiv.org/html/2605.12652#S5.SS1.SSS0.Px1.p1.1)\.
- Y\. Fu, T\. Chen, J\. Chai, X\. Wang, S\. Tu, G\. Yin, W\. Lin, Q\. Zhang, Y\. Zhu, and D\. Zhao \(2026\)SRFT: a single\-stage method with supervised and reinforcement fine\-tuning for reasoning\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=n6E0r6kQWQ)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px3.p1.1)\.
- Y\. Gu, L\. Dong, F\. Wei, and M\. Huang \(2024\)MiniLLM: On\-Policy Distillation of Large Language Models\.Proceedings of ICLR\.External Links:[Link](https://arxiv.org/abs/2306.08543)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2605.12652#S1.p1.1),[§2](https://arxiv.org/html/2605.12652#S2.SS0.SSS0.Px1.p1.1)\.
- X\. Guan, Y\. Liu, X\. Lu, B\. Cao, B\. He, X\. Han, L\. Sun, J\. Lou, B\. Yu, Y\. Lu,et al\.\(2024\)Search, verify and feedback: towards next generation post\-training paradigm of foundation models via verifier engineering\.arXiv preprint arXiv:2411\.11504\.Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px3.p1.1)\.
- A\. Gudibande, E\. Wallace, C\. Snell, X\. Geng, H\. Liu, P\. Abbeel, S\. Levine, and D\. Song \(2023\)The False Promise of Imitating Proprietary LLMs\.arXiv preprint arXiv:2305\.15717\.External Links:[Link](https://arxiv.org/abs/2305.15717)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px1.p1.1)\.
- C\. Gulcehre, Z\. Wang, A\. Novikov, S\. Bechtle, N\. Heess, S\. Pankov, N\. R\. Ke, J\. Alayrac, V\. Bapst, R\. Hoffmann,et al\.\(2023\)ReST: reinforced self\-training \(rest\) for language models\.arXiv preprint arXiv:2308\.08998\.Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2605.12652#S1.p1.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.\(2025\)Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§1](https://arxiv.org/html/2605.12652#S1.p1.1)\.
- Z\. He, T\. Liang, J\. Xu, Q\. Liu, X\. Chen, Y\. Wang, L\. Song, D\. Yu, Z\. Liang, W\. Wang,et al\.\(2025\)Deepmath\-103k: a large\-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning\.arXiv preprint arXiv:2504\.11456\.Cited by:[§5\.1](https://arxiv.org/html/2605.12652#S5.SS1.SSS0.Px1.p1.1)\.
- G\. Hinton, O\. Vinyals, and J\. Dean \(2015\)Distilling the knowledge in a neural network\.arXiv preprint arXiv:1503\.02531\.Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2605.12652#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Hong, N\. Lee, and J\. Thorne \(2024\)ORPO: monolithic preference optimization without reference model\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 11170–11189\.Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px3.p1.1)\.
- H\. Huang, J\. Song, Y\. Zhang, and P\. Ren \(2025\)SelecTKD: Selective Token\-Weighted Knowledge Distillation for LLMs\.arXiv preprint arXiv:2510\.24021\.External Links:[Link](https://arxiv.org/abs/2510.24021)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2605.12652#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Hübotter, F\. Lübeck, L\. Behric, A\. Baumann, M\. Bagatella, D\. Marta, I\. Hakimi, I\. Shenfeld, T\. K\. Buening, C\. Guestrin, and A\. Krause \(2026\)Reinforcement learning via self\-distillation\.arXiv preprint arXiv:2601\.20802\.Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2605.12652#S2.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2605.12652#S3.p1.8)\.
- N\. Jain, K\. Han, A\. Gu, W\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. Stoica \(2025\)LiveCodeBench: holistic and contamination free evaluation of large language models for code\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§5\.1](https://arxiv.org/html/2605.12652#S5.SS1.SSS0.Px1.p1.1)\.
- Y\. Jiang, C\. Chan, M\. Chen, and W\. Wang \(2023\)Lion: Adversarial Distillation of Proprietary Large Language Models\.Proceedings of EMNLP\.External Links:[Link](https://arxiv.org/abs/2305.12870)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px1.p1.1)\.
- W\. Jin, T\. Min, Y\. Yang, S\. R\. Kadhe, Y\. Zhou, D\. Wei, N\. Baracaldo, and K\. Lee \(2026\)Entropy\-Aware On\-Policy Distillation of Language Models\.arXiv preprint arXiv:2603\.07079\.External Links:[Link](https://arxiv.org/abs/2603.07079)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2605.12652#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Jung, S\. Yoon, D\. Kim, and H\. Lee \(2025\)ToDi: Token\-wise Distillation via Fine\-Grained Divergence Control\.Proceedings of EMNLP\.External Links:[Link](https://arxiv.org/abs/2505.16297)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px1.p1.1)\.
- J\. Kim, X\. Luo, M\. Kim, S\. Lee, D\. Kim, J\. Jeon, D\. Li, and Y\. Yang \(2026\)Why Does Self\-Distillation \(Sometimes\) Degrade the Reasoning Capability of LLMs?\.arXiv preprint arXiv:2603\.24472\.External Links:[Link](https://arxiv.org/abs/2603.24472)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px2.p1.1)\.
- Y\. Kim and A\. M\. Rush \(2016\)Sequence\-level knowledge distillation\.InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing,pp\. 1317–1327\.Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2605.12652#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Ko, S\. Abdali, Y\. J\. Kim, T\. Chen, and P\. Cameron \(2026\)Scaling Reasoning Efficiently via Relaxed On\-Policy Distillation\.arXiv preprint arXiv:2603\.11137\.External Links:[Link](https://arxiv.org/abs/2603.11137)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px2.p1.1)\.
- J\. Ko, T\. Chen, S\. Kim, T\. Ding, L\. Liang, I\. Zharkov, and S\. Yun \(2025\)DistiLLM\-2: A Contrastive Approach Boosts the Distillation of LLMs\.Proceedings of ICML\.External Links:[Link](https://arxiv.org/abs/2503.07067)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px1.p1.1)\.
- J\. Ko, S\. Kim, T\. Chen, and S\. Yun \(2024\)DistiLLM: Towards Streamlined Distillation for Large Language Models\.Proceedings of ICML\.External Links:[Link](https://arxiv.org/abs/2402.03898)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2605.12652#S1.p1.1),[§2](https://arxiv.org/html/2605.12652#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Li, M\. Galley, C\. Brockett, J\. Gao, and W\. B\. Dolan \(2016\)A diversity\-promoting objective function for neural conversation models\.InProceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies,pp\. 110–119\.Cited by:[Appendix D](https://arxiv.org/html/2605.12652#A4.SS0.SSS0.Px1.p1.3)\.
- Y\. Li, Y\. Zuo, B\. He, J\. Zhang, C\. Xiao, C\. Qian, T\. Yu, H\. Gao, W\. Yang, Z\. Liu, and N\. Ding \(2026\)Rethinking on\-policy distillation of large language models: phenomenology, mechanism, and recipe\.External Links:2604\.13016,[Link](https://arxiv.org/abs/2604.13016)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px1.p1.1)\.
- Y\. Li, Z\. Li, and J\. Liu \(2025a\)A Note on Hybrid Online Reinforcement and Imitation Learning for LLMs: Formulations and Algorithms\.arXiv preprint arXiv:2512\.23097\.External Links:[Link](https://arxiv.org/abs/2512.23097)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px3.p1.1)\.
- Z\. Li, Y\. Ji, R\. Meng, and D\. He \(2025b\)Learning from committee: reasoning distillation from a mixture of teachers with peer\-review\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 4190–4205\.Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2605.12652#S2.SS0.SSS0.Px1.p1.1)\.
- H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe \(2024\)Let’s Verify Step by Step\.Proceedings of ICLR\.External Links:[Link](https://arxiv.org/abs/2305.20050)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px3.p1.1)\.
- Y\. Lyu, C\. Wang, J\. Huang, and T\. Xu \(2025\)From Correction to Mastery: Reinforced Distillation of Large Language Model Agents\.arXiv preprint arXiv:2509\.14257\.External Links:[Link](https://arxiv.org/abs/2509.14257)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px3.p1.1)\.
- Y\. Meng, M\. Xia, and D\. Chen \(2024\)SimPO: simple preference optimization with a reference\-free reward\.Advances in Neural Information Processing Systems37,pp\. 124198–124235\.Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px3.p1.1)\.
- N\. Muennighoff, Z\. Yang, W\. Shi, X\. L\. Li, L\. Fei\-Fei, H\. Hajishirzi, L\. Zettlemoyer, P\. Liang, E\. Candès, and T\. Hashimoto \(2025\)S1: simple test\-time scaling\.arXiv preprint arXiv:2501\.19393\.Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2605.12652#S2.SS0.SSS0.Px1.p1.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2605.12652#S2.SS0.SSS0.Px2.p1.1)\.
- E\. Penaloza, D\. Vattikonda, N\. Gontier, A\. Lacoste, L\. Charlin, and M\. Caccia \(2026\)Privileged Information Distillation for Language Models\.arXiv preprint arXiv:2602\.04942\.External Links:[Link](https://arxiv.org/abs/2602.04942)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2605.12652#S2.SS0.SSS0.Px1.p1.1)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn \(2023\)Direct preference optimization: your language model is secretly a reward model\.Advances in neural information processing systems36,pp\. 53728–53741\.Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2605.12652#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Ross, G\. J\. Gordon, and J\. A\. Bagnell \(2011\)A reduction of imitation learning and structured prediction to no\-regret online learning\.InProceedings of the 14th International Conference on Artificial Intelligence and Statistics,pp\. 627–635\.Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2605.12652#S2.SS0.SSS0.Px1.p1.1)\.
- H\. Sang, Y\. Xu, Z\. Zhou, R\. He, Z\. Wang, and J\. Sun \(2026\)On\-Policy Self\-Distillation for Reasoning Compression\.arXiv preprint arXiv:2603\.05433\.External Links:[Link](https://arxiv.org/abs/2603.05433)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px2.p1.1)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§1](https://arxiv.org/html/2605.12652#S1.p1.1)\.
- A\. Setlur, C\. Nagpal, A\. Fisch, X\. Geng, J\. Eisenstein, R\. Agarwal, A\. Agarwal, J\. Berant, and A\. Kumar \(2025\)Rewarding progress: scaling automated process verifiers for LLM reasoning\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=A6Y7AqlzLW)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2605.12652#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.\(2024\)Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2605.12652#S1.p1.1),[§2](https://arxiv.org/html/2605.12652#S2.SS0.SSS0.Px2.p1.1)\.
- I\. Shenfeld, M\. Damani, J\. Hübotter, and P\. Agrawal \(2026\)Self\-Distillation Enables Continual Learning\.arXiv preprint arXiv:2601\.19897\.External Links:[Link](https://arxiv.org/abs/2601.19897)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px2.p1.1)\.
- A\. Singh, J\. D\. Co\-Reyes, R\. Agarwal, A\. Anand, P\. Patil, X\. Garcia, P\. J\. Liu, J\. Harrison, J\. Lee, K\. Xu,et al\.\(2024\)Beyond human data: scaling self\-training for problem\-solving with language models\.Transactions on Machine Learning Research\.Note:Expert CertificationExternal Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=lNAyUngGFK)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2605.12652#S1.p1.1)\.
- C\. Snell, J\. Lee, K\. Xu, and A\. Kumar \(2024\)Scaling llm test\-time compute optimally can be more effective than scaling model parameters\.arXiv preprint arXiv:2408\.03314\.Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px2.p1.1)\.
- M\. Song and M\. Zheng \(2026\)A survey of on\-policy distillation for large language models\.External Links:2604\.00626,[Link](https://arxiv.org/abs/2604.00626)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px1.p1.1)\.
- Q\. Tang and et al\. \(2023\)ToolAlpaca: generalized tool learning for language models with 3000 simulated cases\.arXiv preprint arXiv:2306\.05301\.Cited by:[§5\.1](https://arxiv.org/html/2605.12652#S5.SS1.SSS0.Px1.p1.1)\.
- X\. Wang, J\. Wei, D\. Schuurmans, Q\. V\. Le, E\. H\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou \(2023\)Self\-consistency improves chain of thought reasoning in language models\.InThe Eleventh International Conference on Learning Representations,Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2605.12652#S2.SS0.SSS0.Px1.p1.1)\.
- X\. Wen, Z\. Liu, S\. Zheng, S\. Ye, Z\. Wu, Y\. Wang, Z\. Xu, X\. Liang, J\. Li, Z\. Miao, J\. Bian, and M\. Yang \(2026\)Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base LLMs\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=jGbRWwIidy)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2605.12652#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Wen, Z\. Li, W\. Du, and L\. Mou \(2023\)f\-Divergence Minimization for Sequence\-Level Knowledge Distillation\.Proceedings of ACL\.External Links:[Link](https://arxiv.org/abs/2307.15190)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2605.12652#S2.SS0.SSS0.Px1.p1.1)\.
- F\. Wu, W\. Xuan, X\. Lu, M\. Liu, Y\. Dong, Z\. Harchaoui, and Y\. Choi \(2025a\)The invisible leash: why rlvr may or may not escape its origin\.arXiv preprint arXiv:2507\.14843\.Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px3.p1.1)\.
- T\. Wu, C\. Tao, J\. Wang, R\. Yang, Z\. Zhao, and N\. Wong \(2025b\)Rethinking Kullback\-Leibler Divergence in Knowledge Distillation for Large Language Models\.Proceedings of COLING\.External Links:[Link](https://arxiv.org/abs/2404.02657)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px1.p1.1)\.
- H\. Xu, Q\. Zhu, H\. Deng, J\. Li, L\. Hou, Y\. Wang,et al\.\(2025a\)KDRL: Post\-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning\.arXiv preprint arXiv:2506\.02208\.External Links:[Link](https://arxiv.org/abs/2506.02208)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2605.12652#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Xu, L\. Pang, Y\. Zhu, J\. Gu, Z\. Wei, J\. Deng, F\. Pan, H\. Shen, and X\. Cheng \(2025b\)RLKD: Distilling LLMs’ Reasoning via Reinforcement Learning\.arXiv preprint arXiv:2505\.16142\.External Links:[Link](https://arxiv.org/abs/2505.16142)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2605.12652#S2.SS0.SSS0.Px2.p1.1)\.
- W\. Xu, R\. Han, Z\. Wang, L\. T\. Le, D\. Madeka, L\. Li, W\. Y\. Wang, R\. Agarwal, C\. Lee, and T\. Pfister \(2025c\)Speculative Knowledge Distillation: Bridging the Teacher\-Student Gap Through Interleaved Sampling\.Proceedings of ICLR\.External Links:[Link](https://arxiv.org/abs/2410.11325)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px1.p1.1)\.
- X\. Xu, M\. Li, C\. Tao, T\. Shen, R\. Cheng, J\. Li, C\. Xu, D\. Tao, and T\. Zhou \(2024\)A Survey on Knowledge Distillation of Large Language Models\.arXiv preprint arXiv:2402\.13116\.External Links:[Link](https://arxiv.org/abs/2402.13116)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px1.p1.1)\.
- J\. Yan, Y\. Li, Z\. Hu, Z\. Wang, G\. Cui, X\. Qu, Y\. Cheng, and Y\. Zhang \(2025\)Learning to Reason under Off\-Policy Guidance\.arXiv preprint arXiv:2504\.14945\.External Links:[Link](https://arxiv.org/abs/2504.14945)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px3.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu,et al\.\(2025\)Qwen3 Technical Report\.arXiv preprint arXiv:2505\.09388\.External Links:[Link](https://arxiv.org/abs/2505.09388)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px1.p1.1)\.
- W\. Yang, W\. Liu, R\. Xie, K\. Yang, S\. Yang, and Y\. Lin \(2026\)Learning beyond teacher: generalized on\-policy distillation with reward extrapolation\.arXiv preprint arXiv:2602\.12125\.Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2605.12652#S2.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2605.12652#S3.p1.8),[§5\.1](https://arxiv.org/html/2605.12652#S5.SS1.SSS0.Px1.p1.1)\.
- Z\. Yang, T\. Pang, H\. Feng, H\. Wang, W\. Chen, M\. Zhu, and Q\. Liu \(2024\)Self\-distillation bridges distribution gap in language model fine\-tuning\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1028–1043\.Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2605.12652#S1.p1.1)\.
- T\. Ye, L\. Dong, Z\. Chi, X\. Wu, S\. Huang, and F\. Wei \(2025\)Black\-box on\-policy distillation of large language models\.arXiv preprint arXiv:2511\.10643\.Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2605.12652#S3.p1.8)\.
- T\. Ye, L\. Dong, X\. Wu, S\. Huang, and F\. Wei \(2026\)On\-policy context distillation for language models\.arXiv preprint arXiv:2602\.12275\.Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2605.12652#S2.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2605.12652#S3.p1.8)\.
- Y\. Yue, Z\. Chen, R\. Lu, A\. Zhao, Z\. Wang, Y\. Yue, S\. Song, and G\. Huang \(2025\)Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model?\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=4OsgYD7em5)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2605.12652#S1.p1.1)\.
- E\. Zelikman, Y\. Wu, J\. Laskin, and N\. D\. Goodman \(2022\)STaR: bootstrapping reasoning with reasoning\.Advances in Neural Information Processing Systems35,pp\. 15476–15488\.Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2605.12652#S1.p1.1)\.
- D\. Zhang, S\. Zhoubian, Z\. Hu, Y\. Yue, Y\. Dong, and J\. Tang \(2024\)ReST\-mcts\*: llm self\-training via process reward guided tree search\.Advances in Neural Information Processing Systems37,pp\. 64735–64772\.Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px2.p1.1)\.
- D\. Zhang, Z\. Yang, S\. Janghorbani, J\. Han, A\. Ressler, Q\. Qian, G\. D\. Lyng, S\. S\. Batra, and R\. E\. Tillman \(2026a\)Fast and Effective On\-policy Distillation from Reasoning Prefixes\.arXiv preprint arXiv:2602\.15260\.External Links:[Link](https://arxiv.org/abs/2602.15260)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2605.12652#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Zhang, X\. Zhang, T\. Zhang, B\. Hu, Y\. Chen, and J\. Xu \(2025a\)AlignDistil: Token\-Level Language Model Alignment as Adaptive Policy Distillation\.Proceedings of ACL\.External Links:[Link](https://arxiv.org/abs/2503.02832)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px1.p1.1)\.
- X\. Zhang, Y\. Zhang, H\. Sun, K\. Feng, C\. Lu, C\. Yang, and H\. Meng \(2025b\)Critique\-grpo: advancing llm reasoning with natural language and numerical feedback\.arXiv preprint arXiv:2506\.03106\.Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px2.p1.1)\.
- Z\. Zhang, S\. Jiang, Y\. Shen, Y\. Zhang, D\. Ram, S\. Yang,et al\.\(2026b\)Reinforcement\-aware Knowledge Distillation for LLM Reasoning\.arXiv preprint arXiv:2602\.22495\.External Links:[Link](https://arxiv.org/abs/2602.22495)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2605.12652#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Zhang, C\. Zheng, Y\. Wu, B\. Zhang, R\. Lin, B\. Yu, D\. Liu, J\. Zhou, and J\. Lin \(2025c\)The lessons of developing process reward models in mathematical reasoning\.arXiv preprint arXiv:2501\.07301\.Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2605.12652#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Zhao, Z\. Wang, X\. Zhao, J\. Zhou, C\. Xu, C\. Liu, L\. Zhang, Y\. Jia, Y\. Zhang, H\. Yu, Z\. Xu, Q\. Li, and Y\. Qin \(2026a\)Large language model post\-training: a unified view of off\-policy and on\-policy learning\.External Links:2604\.07941,[Link](https://arxiv.org/abs/2604.07941)Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px3.p1.1)\.
- S\. Zhao, Z\. Xie, M\. Liu, J\. Huang, G\. Pang, F\. Chen, and A\. Grover \(2026b\)Self\-distilled reasoner: on\-policy self\-distillation for large language models\.arXiv preprint arXiv:2601\.18734\.Cited by:[Appendix F](https://arxiv.org/html/2605.12652#A6.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2605.12652#S2.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2605.12652#S3.p1.8)\.

## Appendix ATeacher Signal Quality Metric

#### Mean Spearman Correlation\.

Spearman correlation measures the agreement between the rank ordering induced by teacher scores and the rank ordering induced by rewards\. For a prompt, letrank​\(si\)\\mathrm\{rank\}\(s\_\{i\}\)andrank​\(ri\)\\mathrm\{rank\}\(r\_\{i\}\)denote the average ranks under ties\. The prompt\-level Spearman correlation is

ρS=corr​\(rank​\(s1\),…,rank​\(sn\),rank​\(r1\),…,rank​\(rn\)\),\\rho\_\{\\mathrm\{S\}\}\\;=\\;\\mathrm\{corr\}\\\!\\bigl\(\\mathrm\{rank\}\(s\_\{1\}\),\\dots,\\mathrm\{rank\}\(s\_\{n\}\),\\;\\mathrm\{rank\}\(r\_\{1\}\),\\dots,\\mathrm\{rank\}\(r\_\{n\}\)\\bigr\),wherecorr​\(⋅,⋅\)\\mathrm\{corr\}\(\\cdot,\\cdot\)is the Pearson correlation coefficient\. We report the mean of this quantity across prompts\. Higher values indicate that the teacher better preserves the reward\-induced ordering\.

#### Mean Kendall’sτ\\tau\.

Kendall’sτ\\tauevaluates pairwise ranking consistency between teacher scores and rewards\. For each unordered pair\(i,j\)\(i,j\)withi<ji<j, let the pair be*concordant*if\(si−sj\)​\(ri−rj\)\>0\(s\_\{i\}\-s\_\{j\}\)\(r\_\{i\}\-r\_\{j\}\)\>0and*discordant*if\(si−sj\)​\(ri−rj\)<0\(s\_\{i\}\-s\_\{j\}\)\(r\_\{i\}\-r\_\{j\}\)<0\. Pairs tied in either score or reward are ignored\. The prompt\-level metric is

τ=C−DC\+D,\\tau\\;=\\;\\frac\{C\-D\}\{C\+D\},whereCCandDDare the numbers of concordant and discordant pairs, respectively\. We then averageτ\\tauacross prompts\. Higher values indicate stronger ordinal agreement\.

#### Pairwise Accuracy\.

Pairwise accuracy measures the fraction of unequal\-reward pairs that are ordered correctly by the teacher\. Formally,

PairAcc=∑i<j𝟏​\[ri≠rj\]​𝟏​\[\(si−sj\)​\(ri−rj\)\>0\]∑i<j𝟏​\[ri≠rj\]\.\\mathrm\{PairAcc\}\\;=\\;\\frac\{\\sum\_\{i<j\}\\mathbf\{1\}\[r\_\{i\}\\neq r\_\{j\}\]\\mathbf\{1\}\\\!\\left\[\(s\_\{i\}\-s\_\{j\}\)\(r\_\{i\}\-r\_\{j\}\)\>0\\right\]\}\{\\sum\_\{i<j\}\\mathbf\{1\}\[r\_\{i\}\\neq r\_\{j\}\]\}\.Unlike Kendall’sτ\\tau, this metric does not penalize incorrect pairs symmetrically around zero; instead, it directly reports the proportion of reward\-distinguishable pairs ranked correctly\. Higher is better\.

#### Success AUC\.

Success AUC treats the teacher score as a binary classifier score for success\. Let𝒫=\{i:yi=1\}\\mathcal\{P\}=\\\{i:y\_\{i\}=1\\\}and𝒩=\{j:yj=0\}\\mathcal\{N\}=\\\{j:y\_\{j\}=0\\\}\. The prompt\-level AUC is computed by comparing all positive–negative pairs:

AUC=1\|𝒫\|​\|𝒩\|​∑i∈𝒫∑j∈𝒩\(𝟏​\[si\>sj\]\+12​𝟏​\[si=sj\]\)\.\\mathrm\{AUC\}\\;=\\;\\frac\{1\}\{\|\\mathcal\{P\}\|\|\\mathcal\{N\}\|\}\\sum\_\{i\\in\\mathcal\{P\}\}\\sum\_\{j\\in\\mathcal\{N\}\}\\left\(\\mathbf\{1\}\[s\_\{i\}\>s\_\{j\}\]\+\\frac\{1\}\{2\}\\mathbf\{1\}\[s\_\{i\}=s\_\{j\}\]\\right\)\.This is the probability that a randomly chosen successful response receives a higher teacher score than a randomly chosen unsuccessful one, with ties receiving half credit\. Higher is better\.

#### Success Point\-Biserial Correlation\.

To quantify the linear association between teacher scores and binary success labels, we compute the point\-biserial correlation, which is simply the Pearson correlation between the continuous teacher scores and the binary labels:

ρPB=corr​\(s1,…,sn,r1,…,rn\)\.\\rho\_\{\\mathrm\{PB\}\}\\;=\\;\\mathrm\{corr\}\(s\_\{1\},\\dots,s\_\{n\},\\;r\_\{1\},\\dots,r\_\{n\}\)\.A larger positive value indicates that successful responses tend to receive higher teacher scores\.

#### Success Brier Score \(Sigmoid\)\.

To assess probabilistic calibration of teacher scores with respect to success, we first map each score to a pseudo\-probability using the sigmoid transform

pi=σ​\(si\)=11\+e−si,p\_\{i\}\\;=\\;\\sigma\(s\_\{i\}\)\\;=\\;\\frac\{1\}\{1\+e^\{\-s\_\{i\}\}\},and then compute the Brier score

Brier=1n​∑i=1n\(pi−ri\)2\.\\mathrm\{Brier\}\\;=\\;\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\(p\_\{i\}\-r\_\{i\}\)^\{2\}\.This metric measures the mean squared error between predicted success probabilities and observed binary outcomes\. Lower values indicate better calibration\.

ConditionMeanMeanPairwiseSuccessSuccessSuccessSpearmanKendall’sτ\\tauAcc\.AUCPoint\-BiserialBrier \(Sig\.\)Base\-0\.0024\-0\.00310\.12750\.1283\-0\.00370\.2071Primary Sol\.0\.09880\.12620\.32110\.32170\.08590\.2986Second Sol\.0\.11900\.16320\.35280\.35470\.10650\.3139Failure Sol\.0\.00970\.00950\.13260\.13320\.00520\.1728Sol\.\+Failure0\.11220\.13260\.40450\.40450\.10380\.27012 Suc\.\+1 Failure0\.15950\.18790\.47330\.47330\.15200\.2908All Solutions0\.12600\.16230\.35500\.35560\.11070\.3124Table 7:Prompt\-level offline teacher\-signal metrics averaged over prompts for the six context conditions\. Higher is better for all metrics except Success Brier \(Sigmoid\), where lower is better\.Table[7](https://arxiv.org/html/2605.12652#A1.T7)reports prompt\-level teacher\-signal metrics across the six context conditions\. The2 Suc\.\+1 Failurecondition consistently achieves the strongest performance, attaining the highest Mean Spearman \(0\.15950\.1595\), Mean Kendall’sτ\\tau\(0\.18790\.1879\), Pairwise Accuracy \(0\.47330\.4733\), Success AUC \(0\.47330\.4733\), and Success Point\-Biserial \(0\.15200\.1520\), indicating that a mixed context of two successful and one failed solution provides the richest teacher signal\. In contrast, theFailure Sol\.condition performs the worst across nearly all ranking and discrimination metrics, with a Mean Spearman of only0\.00970\.0097and a Pairwise Accuracy of0\.13260\.1326, suggesting that failure demonstrations alone offer little useful signal\.Failure Sol\.attains the lowest Brier score \(0\.17280\.1728\), but its discrimination metrics also collapse to near\-zero, so this calibration value reflects a degenerate near\-uniform output rather than a useful signal\. Among the success\-only conditions,Second Sol\.slightly outperformsPrimary Sol\.andAll Solutionsacross most metrics, whileSol\.\+Failureimproves pairwise accuracy over either success\-only condition alone\.

## Appendix BPreliminary on Divergence\-based distillation

#### Divergence\-based distillation\.

A common approach to distillation is to minimize a divergence between the teacher distributionqϕq\_\{\\phi\}and the student distributionπθ\\pi\_\{\\theta\}at each token position\. In language model distillation, the direction of the KL divergence induces qualitatively different behavior\.

The*forward KL*objective,

DKL\(qϕ\(⋅∣ct\)∥πθ\(⋅∣ct\)\),D\_\{\\mathrm\{KL\}\}\\\!\\left\(q\_\{\\phi\}\(\\cdot\\mid c\_\{t\}\)\\,\\\|\\,\\pi\_\{\\theta\}\(\\cdot\\mid c\_\{t\}\)\\right\),\(8\)penalizes the student for assigning low probability to tokens favored by the teacher\. It is therefore mode\-covering: when the teacher assigns mass to multiple plausible continuations, forward KL encourages the student to preserve this diversity\.

The*reverse KL*objective,

DKL\(πθ\(⋅∣ct\)∥qϕ\(⋅∣ct\)\),D\_\{\\mathrm\{KL\}\}\\\!\\left\(\\pi\_\{\\theta\}\(\\cdot\\mid c\_\{t\}\)\\,\\\|\\,q\_\{\\phi\}\(\\cdot\\mid c\_\{t\}\)\\right\),\(9\)penalizes the student for placing probability mass on tokens that the teacher considers unlikely\. This objective is mode\-seeking and is often attractive in on\-policy settings because the expectation is taken over the student distribution\. However, reverse KL may over\-concentrate probability mass on a subset of teacher\-preferred modes, especially when the teacher distribution is uncertain or when the supervision signal is incomplete\.

A symmetric alternative is the Jensen–Shannon divergence,

DJS​\(p∥q\)=12​DKL​\(p∥m\)\+12​DKL​\(q∥m\),m=12​\(p\+q\)\.D\_\{\\mathrm\{JS\}\}\(p\\,\\\|\\,q\)=\\frac\{1\}\{2\}D\_\{\\mathrm\{KL\}\}\(p\\,\\\|\\,m\)\+\\frac\{1\}\{2\}D\_\{\\mathrm\{KL\}\}\(q\\,\\\|\\,m\),\\qquad m=\\frac\{1\}\{2\}\(p\+q\)\.\(10\)DJSD\_\{\\mathrm\{JS\}\}is symmetric and bounded\. In principle, it can provide a more stable compromise between mode\-covering and mode\-seeking behavior\. In practice, however, computing such distribution\-level objectives over the full vocabulary may be expensive for large language models, motivating approximations based on sampled tokens, top\-kksupport, or teacher\-provided log\-probabilities\.

## Appendix CCase Studies

We present three complementary forms of evidence for MOPD’s qualitative advantage\. Case Studies 1 and 2 isolate checkpoint\-level semantic failures: both models see the same task, but MOPD preserves the crucial constraints that SDPO drops\. Case Study 1 additionally provides direct training\-time evidence linking the checkpoint gap to MOPD’s peer\-conditioning mechanism\. Case Study 3 examines training dynamics, showing that MOPD internalizes correct behavior earlier while SDPO remains unstable late in training\.

### C\.1Case Study 1: Identifier\-space composition \(checkpoint gap \+ training\-time mechanism\)

#### Candidate pair\.

The bib\-mapping task provides a direct SDPO\-versus\-MOPD comparison\. On this prompt, the SDPO checkpoint at training step 50 reaches accuracy0\.1750\.175, while the MOPD checkpoint at the same step solves it perfectly\.

#### SDPO failure\.

The SDPO trajectory mixes up the role of the permutation arrays:

```
bib_to_person = [0] * (N + 1)
for i in range(N):
    bib_to_person[Q[i]] = P[i]
result = [str(bib_to_person[Q[i]]) for i in range(N)]
```

This is incorrect in two ways:P\[i\]is the stared\-at person, not the wearer of bibQ\[i\], and the final answer stops in person\-ID space instead of projecting back to the target person’s bib number\. The program preserves the rough shape of the mapping task but composes the wrong relations\.

#### MOPD success\.

The corresponding MOPD sample keeps the two identifier spaces separate throughout:

```
bib_to_person = [0] * (N + 1)
for i in range(N):
    bib_to_person[Q[i]] = i + 1
for i in range(1, N + 1):
    person = bib_to_person[i]
    target_person = P[person - 1]
    result.append(str(Q[target_person - 1]))
```

This trajectory performs the full chain correctly:bib→\\rightarrowwearing person→\\rightarrowstared\-at person→\\rightarrowtarget bib\.

#### Training\-time context: MOPD encounters the same failure pattern during rollout\.

The same bib\-mapping prompt appears in the MOPD rollout stream at training step 1 with seven failed peers and one successful peer; the failed peers exhibit the same identifier\-space ambiguity that SDPO retains at checkpoint time\. Because MOPD injects failed peers directly into the teacher context, the teacher supervising the successful target on this prompt is exposed to a concrete instance of this bug paired with an explicit “avoid this mistake” instruction\. SDPO has no equivalent channel\.

### C\.2Case Study 2: Dropped middle\-character constraint

#### Candidate pair\.

The evenly spacedA\-B\-Ccounting task gives a second clean contrast\. On this prompt, the SDPO checkpoint at training step 50 reaches accuracy0\.8250\.825, while the MOPD checkpoint at the same step is fully correct\.

#### SDPO failure\.

The SDPO code enforces equal spacing and checks the two endpoints, but drops the requirement that the middle character must beB:

```
for j in range(1, n-1):
    for i in range(j):
        if S[i] == ’A’:
            k = 2 * j - i
            if k < n and S[k] == ’C’:
                count += 1
```

This overcounts invalid triples such asA\-R\-Cin the test caseARC, where the correct answer is0\. The failure is a local semantic omission: the positional structure is preserved, but one label constraint from the specification is forgotten\.

#### MOPD success\.

The MOPD sample reinstates the missing check explicitly:

```
for j in range(1, n - 1):
    if s[j] == ’B’:
        for i in range(j):
            if s[i] == ’A’:
                k = 2 * j - i
                if k < n and s[k] == ’C’:
                    count += 1
```

The program is structurally close to the SDPO attempt but preserves the full symbolic constraint set\.

#### Takeaway\.

Case Studies 1 and 2 together show that MOPD’s advantage in avoiding the failure pattern is not tied to a single error type\. In Case Study 1, SDPO fails by composing the wrong identifier relations; here it fails by dropping one clause from the specification\. MOPD succeeds in both settings; the contrast holds across two distinct error families rather than depending on a single failure mode\.

### C\.3Case Study 3: Training dynamics — MOPD stabilizes early, SDPO fails late

#### Candidate pair\.

The direction\-opposite task illustrates the training\-time gap\. The SDPO checkpoint fails on this prompt at both step 50 and step 100 with accuracy0\.00\.0: the step\-50 sample is truncated, and the step\-100 sample is flagged for a missing‘‘‘pythonblock\. The MOPD checkpoint at step 50 already solves the same prompt with accuracy1\.01\.0\.

#### Early MOPD signal\.

In the MOPD rollout stream, a correct solution first appears at rollout step 10 with score1\.01\.0—the earliest observed point at which the repaired behavior is present in training\-time rollouts for this prompt\.

#### Late SDPO failure\.

At rollout step 80, SDPO still fails with accuracy0\.00\.0, receiving feedback:Incorrect Format: Put your code inside a ‘‘‘python \.\.\. ‘‘‘ block\.

#### Takeaway\.

The gap here is not algorithmic: MOPD internalizes basic output\-format discipline by approximately step 10, while SDPO has not stabilized that behavior even late in training\.

### C\.4Overall Pattern

Taken together, the three case studies in lcv benchmark reveal a consistent picture\. The baseline’s failures are narrow and consequential—a dropped constraint, a confused identifier space, an unresolved format requirement—rather than wholesale algorithmic breakdowns\. These are precisely the errors that peer conditioning is designed to surface: by exposing the teacher to failed peers that exhibit the same bug, MOPD produces a sharper supervision signal at the critical decision points\. Case Study 1 compared through training logs; Case Studies 2 and 3 provide corroborating evidence across a different failure type and a training\-dynamics lens\. The combined evidence supports the conclusion that MOPD’s gains arise from a more targeted teacher signal, not from discovering fundamentally different solution strategies\.

## Appendix DDiversity

#### Distinct\-nn\.

Distinct\-nn\[Liet al\.,[2016](https://arxiv.org/html/2605.12652#bib.bib57)\]is the ratio of uniquenn\-grams to totalnn\-grams across the concatenated token stream of all responses:

Distinct​\-​n=\|\{unique​n​\-grams\}\|total​n​\-grams\.\\mathrm\{Distinct\\text\{\-\}\}n=\\frac\{\|\\\{\\text\{unique \}n\\text\{\-grams\}\\\}\|\}\{\\text\{total \}n\\text\{\-grams\}\}\.\(11\)We report Distinct\-1 and Distinct\-2\.

#### AST node Jaccard distance \(code only\)\.

Each response is parsed into a Python AST and its node\-type multiset𝐚i\\mathbf\{a\}\_\{i\}is extracted\. Pairwise Jaccard distance is

di​jAST=1−∑tmin⁡\(ai​t,aj​t\)∑tmax⁡\(ai​t,aj​t\),d^\{\\mathrm\{AST\}\}\_\{ij\}=1\-\\frac\{\\sum\_\{t\}\\min\(a\_\{it\},\\,a\_\{jt\}\)\}\{\\sum\_\{t\}\\max\(a\_\{it\},\\,a\_\{jt\}\)\},\(12\)wherettranges over all AST node types present in either response\. Responses that fail to parse are excluded; the metric is omitted for a prompt if fewer than two responses parse successfully\. The per\-prompt score is the mean over all valid pairs\.

#### Code Extraction for Diversity Measurement\.

For code\-focused diversity analysis, we do not compute metrics on the raw response text directly\. Instead, each response is first converted into a code view using a simple fenced\-code\-block extraction rule\. Specifically, we match Markdown code blocks of the form‘‘‘python \.\.\. ‘‘‘or‘‘‘ \.\.\. ‘‘‘with a regular expression, and if multiple code blocks appear in one response, we keep only the last matched block\. The extracted block is then stripped of leading and trailing whitespace and used as the canonical code content for subsequent diversity computation\. If no fenced code block is found, the entire response text is used as a fallback\. This procedure is intentionally lightweight: it does not perform full Markdown parsing, and only thepythonlanguage tag is explicitly removed\. As a result, responses with other language tags may retain the tag text inside the extracted content\.

#### Results\.

Table[8](https://arxiv.org/html/2605.12652#A4.T8)summarises the diversity metrics across the three candidate sets evaluated on 131 shared prompts with up to eight rollouts each\.

Table 8:Diversity metrics \(mean over prompts,n=131n\{=\}131\)\. Higher is better for all columns\.Boldmarks the best value per column\.MethodDistinct\-1Distinct\-2AST Jaccarddist\.MOPD0\.0520\.1410\.129SDPO0\.0270\.1100\.110Baseline0\.0250\.1110\.092MOPD produces rollouts with substantially higher lexical and structural diversity: Distinct\-1 is roughly2×2\{\\times\}that of SDPO and the baseline, and the mean pairwise AST node\-type Jaccard distance is 17% higher than SDPO\.

## Appendix EMore Training Details and Training Statistics

#### Compute Resources\.

All self\-distillation experiments are conducted on a single node of 8×\\timesH100 GPUs and a single node of 8×\\timesA100 GPUs\. Teacher–student distillation experiments are conducted on 2 nodes each equipped with 8×\\timesH100 GPUs\.

Table 9:Per\-step teacher\-query cost comparison\. Teacher context length is measured in tokens \(mean over training steps\)\. Wall\-clock time is measured per gradient update step on 8×\\timesH100 GPUs with batch size 32 and 8 rollouts per prompt\.DatasetMethodTime GenTime RewardTime AdvTime UpdateTotalLCBSDPO37400178550MOPD37400290600QA\-ChemistrySDPO50\.117590MOPD50\.12110120
#### Time Cost\.

We report several wall\-clock timing metrics in seconds to break down the cost of each training iteration\. To avoid startup overhead, we report the average timing measured after the first five training steps, since earlier steps may include additional one\-time loading and initialization costs\.timing\_s/genmeasures the time spent generating sampled responses from the current policy\.timing\_s/rewardmeasures the time required to obtain training rewards, including reward\-model scoring or task\-specific reward computation\.timing\_s/update\_actormeasures the wall\-clock time of the actor optimization stage\. In the SDPO setting, this stage includes the student forward pass, the teacher forward pass used to produce teacher logits, the self\-distillation loss computation, and the subsequent backward pass and optimizer update\. Finally,timing\_s/advmeasures the time spent in the advantage\-processing stage, which includes reward post\-processing and the computation of training advantages used for policy optimization\.timing\_s/stepmeasures the total duration of one training step, covering the full pipeline from rollout generation to optimization\. The reported stage times do not sum exactly to the full step time because they only cover the main timed components of the training loop\. The total step time also includes additional overhead such as data movement, batch reorganization, mask construction, metric logging, coordination between workers, and other small bookkeeping operations that are not exposed as separate timing entries\. As a result, the difference between the sum of the listed stage times and the total step time can be understood as uncategorized system and orchestration overhead\.

Table[9](https://arxiv.org/html/2605.12652#A5.T9)reports the per\-step wall\-clock time breakdown for SDPO and MOPD across two datasets\. The dominant cost in both methods is reward computation and parameter update; rollout generation and advantage estimation are comparatively negligible\. MOPD introduces additional overhead primarily in the update phase, as the teacher must process longer peer\-conditioned contexts\. Despite this, the total per\-step time increases modestly: from 550s to 600s on LCB\(a 9\.1% increase\)and from 90s to 120s on QA\-Chemistry\(a 33\.3% increase\)\. These results indicate that the computational overhead of peer conditioning is small relative to the overall training cost, and that MOPD’s consistent performance gains are not simply an artifact of increased compute budget\.

Table 10:Science QA results with standard deviation over 3 runs\. Bold marks the best result in each column\.MethodBiologyChemistryPhysicsMaterialsSDPO50\.60±2\.150\.60\_\{\\pm 2\.1\}73\.22±2\.573\.22\_\{\\pm 2\.5\}67\.36±1\.967\.36\_\{\\pm 1\.9\}72\.34±2\.672\.34\_\{\\pm 2\.6\}MOPD55\.69±1\.6\{55\.69\}\_\{\\pm 1\.6\}74\.29±2\.0\{74\.29\}\_\{\\pm 2\.0\}76\.25±2\.8\{76\.25\}\_\{\\pm 2\.8\}78\.59±1\.9\{78\.59\}\_\{\\pm 1\.9\}
#### Training stability\.

Table[10](https://arxiv.org/html/2605.12652#A5.T10)reports the mean and standard deviation of MOPD across 3 independent runs on the Science QA benchmarks\. While MOPD consistently achieves the best performance across all four domains, the standard deviations reveal moderate variance, particularly in Physics \(±2\.8\\pm 2\.8\) and Chemistry \(±2\.0\\pm 2\.0\), suggesting that training is not fully stabilized across runs\. We attribute this to the stochastic nature of on\-policy rollout sampling and the sensitivity of peer\-context construction to the specific success/failure partition observed in each run\. Improving training stability through more robust peer selection or variance reduction techniques remains an avenue for future work\.

## Appendix FMore Related Work

#### On\-Policy Distillation of Large Language Models

Knowledge distillation classically transfers a teacher’s predictive distribution to a smaller student through supervision on teacher\- or human\-curated sequencesHintonet al\.\[[2015](https://arxiv.org/html/2605.12652#bib.bib15)\], Kim and Rush \[[2016](https://arxiv.org/html/2605.12652#bib.bib16)\], Xuet al\.\[[2024](https://arxiv.org/html/2605.12652#bib.bib17)\]\. For autoregressive language models, this off\-policy regime exhibits a well\-known limitation: the student is trained on states it never visits at inference, and prediction errors compound over long generationsGudibandeet al\.\[[2023](https://arxiv.org/html/2605.12652#bib.bib18)\], Rosset al\.\[[2011](https://arxiv.org/html/2605.12652#bib.bib19)\], Song and Zheng \[[2026](https://arxiv.org/html/2605.12652#bib.bib20)\]\. The central response is to move the supervision signal onto trajectories drawn from the student’s own evolving policy, anchoring training on the distribution that will be operated over at deploymentAgarwalet al\.\[[2024](https://arxiv.org/html/2605.12652#bib.bib1)\], Guet al\.\[[2024](https://arxiv.org/html/2605.12652#bib.bib21)\], Koet al\.\[[2024](https://arxiv.org/html/2605.12652#bib.bib22),[2025](https://arxiv.org/html/2605.12652#bib.bib23)\]\. Within this framework, a substantial body of work focuses on how to construct and weight the per\-token learning signal\. Forward\-, reverse\-, and skewed\-divergence objectives offer different trade\-offs between mode coverage and mode seekingGuet al\.\[[2024](https://arxiv.org/html/2605.12652#bib.bib21)\], Koet al\.\[[2024](https://arxiv.org/html/2605.12652#bib.bib22)\], Wenet al\.\[[2023](https://arxiv.org/html/2605.12652#bib.bib24)\], Wuet al\.\[[2025b](https://arxiv.org/html/2605.12652#bib.bib25)\], Junget al\.\[[2025](https://arxiv.org/html/2605.12652#bib.bib26)\]\. Other works adapt the supervision to vocabulary mismatches between teacher and studentBoizardet al\.\[[2025](https://arxiv.org/html/2605.12652#bib.bib27)\], reweight tokens by uncertainty or salienceHuanget al\.\[[2025](https://arxiv.org/html/2605.12652#bib.bib28)\], Zhanget al\.\[[2025a](https://arxiv.org/html/2605.12652#bib.bib29)\], or interpolate between fully on\- and off\-policy regimes via teacher\-prefixed rolloutsXuet al\.\[[2025c](https://arxiv.org/html/2605.12652#bib.bib30)\], Zhanget al\.\[[2026a](https://arxiv.org/html/2605.12652#bib.bib31)\]\. Black\-box variants drop the teacher\-logit requirement and supervise the student through sampled outputs, sequence\-level rewards, or adversarial discriminationYeet al\.\[[2025](https://arxiv.org/html/2605.12652#bib.bib3)\], Jianget al\.\[[2023](https://arxiv.org/html/2605.12652#bib.bib32)\]\. The paradigm now underlies widely deployed post\-training pipelines for instruction\-following, code generation, and mathematical reasoningYanget al\.\[[2025](https://arxiv.org/html/2605.12652#bib.bib33)\]\. A complementary thread documents stability and failure modes of these objectives, including capacity\-induced learnability gaps, divergence pathologies, and entropy collapse, and proposes adaptive losses or curricula to mitigate themBusbridgeet al\.\[[2025](https://arxiv.org/html/2605.12652#bib.bib34)\], Liet al\.\[[2026](https://arxiv.org/html/2605.12652#bib.bib35)\], Jinet al\.\[[2026](https://arxiv.org/html/2605.12652#bib.bib36)\]\. Despite this breadth, the dominant assumption is that the teacher signal for each rollout is constructed independently, leaving cross\-rollout structure within a sampling group unexploited\.

#### Self\-Distillation and Self\-Improvement for Reasoning

A second line of work treats the same model as both teacher and student, exploiting either privileged conditioning or repeated sampling to manufacture supervision without an external teacher\. Iterative self\-training procedures generate candidate solutions, filter by verification or reward, and fine\-tune on the surviving trajectories, yielding a coarse form of self\-distillation for reasoning and alignmentZelikmanet al\.\[[2022](https://arxiv.org/html/2605.12652#bib.bib12)\], Gulcehreet al\.\[[2023](https://arxiv.org/html/2605.12652#bib.bib13)\], Singhet al\.\[[2024](https://arxiv.org/html/2605.12652#bib.bib37)\], Chenet al\.\[[2024](https://arxiv.org/html/2605.12652#bib.bib38)\], Yanget al\.\[[2024](https://arxiv.org/html/2605.12652#bib.bib39)\]\. Subsequent work develops soft self\-distillation: one copy of the model is conditioned on additional context—ground\-truth solutions, hints, demonstrations, or environment feedback—and produces per\-token targets for a second copy that lacks that conditioningYeet al\.\[[2026](https://arxiv.org/html/2605.12652#bib.bib4)\], Zhaoet al\.\[[2026b](https://arxiv.org/html/2605.12652#bib.bib5)\], Hübotteret al\.\[[2026](https://arxiv.org/html/2605.12652#bib.bib8)\], Shenfeldet al\.\[[2026](https://arxiv.org/html/2605.12652#bib.bib40)\], Penalozaet al\.\[[2026](https://arxiv.org/html/2605.12652#bib.bib41)\], Ding \[[2026](https://arxiv.org/html/2605.12652#bib.bib42)\], Yanget al\.\[[2026](https://arxiv.org/html/2605.12652#bib.bib6)\]\. Self\-distillation has been particularly active in long chain\-of\-thought reasoning, where supervision is sparse and individual traces are expensive\. One direction compresses or reshapes long reasoning traces during distillationSanget al\.\[[2026](https://arxiv.org/html/2605.12652#bib.bib43)\], Koet al\.\[[2026](https://arxiv.org/html/2605.12652#bib.bib44)\]\. Another aggregates information across multiple sampled solutions through consistency, voting, or committee teachers, transforming an ensemble of attempts into a richer learning signalWanget al\.\[[2023](https://arxiv.org/html/2605.12652#bib.bib45)\], Muennighoffet al\.\[[2025](https://arxiv.org/html/2605.12652#bib.bib46)\], Liet al\.\[[2025b](https://arxiv.org/html/2605.12652#bib.bib47)\], Snellet al\.\[[2024](https://arxiv.org/html/2605.12652#bib.bib48)\]\. A third direction interleaves rollouts with structured self\-critique or search before constructing supervision, expanding the teacher’s effective view of the problemZhanget al\.\[[2024](https://arxiv.org/html/2605.12652#bib.bib49)\], Baoet al\.\[[2025](https://arxiv.org/html/2605.12652#bib.bib50)\], Zhanget al\.\[[2025b](https://arxiv.org/html/2605.12652#bib.bib51)\]\. Recent analyses also document the limitations of these procedures, noting that self\-distillation can degrade reasoning when the student is asked to imitate signals beyond its competence or when the privileged context misaligns with what the student can reproduceKimet al\.\[[2026](https://arxiv.org/html/2605.12652#bib.bib52)\]\. Across these directions, the common thread is that a single model can sharpen its own training signal when given structured access to additional information or to alternative attempts; whether and how those alternative attempts should be jointly conditioned on within a single distillation step remains underexplored\.

#### Multi\-Rollout Reinforcement Learning and Verifier\-Guided Improvement

A third line of related work concerns post\-training paradigms that exploit groups of rollouts and external verification rather than per\-token teacher matching\. Reinforcement learning from human or AI feedback aligns model behavior with preference labels collected over pairs or groups of completions, with subsequent advances exploring offline, reference\-free, and online variantsOuyanget al\.\[[2022](https://arxiv.org/html/2605.12652#bib.bib53)\], Rafailovet al\.\[[2023](https://arxiv.org/html/2605.12652#bib.bib58)\], Honget al\.\[[2024](https://arxiv.org/html/2605.12652#bib.bib59)\], Menget al\.\[[2024](https://arxiv.org/html/2605.12652#bib.bib60)\], Casperet al\.\[[2023](https://arxiv.org/html/2605.12652#bib.bib61)\]\. Reinforcement learning with verifiable rewards extends this idea by replacing learned reward models with programmatic checkers—answer matching, unit tests, type\-checkers—and updating the policy from group\-relative advantages computed over multiple sampled rollouts per promptShaoet al\.\[[2024](https://arxiv.org/html/2605.12652#bib.bib55)\], Wenet al\.\[[2026](https://arxiv.org/html/2605.12652#bib.bib62)\], Wuet al\.\[[2025a](https://arxiv.org/html/2605.12652#bib.bib63)\], Yueet al\.\[[2025](https://arxiv.org/html/2605.12652#bib.bib64)\], Fuet al\.\[[2026](https://arxiv.org/html/2605.12652#bib.bib65)\], Denget al\.\[[2026](https://arxiv.org/html/2605.12652#bib.bib66)\]\. A unified perspective on off\- and on\-policy post\-training places these methods alongside on\-policy distillation and clarifies when each provides the most useful supervision signalZhaoet al\.\[[2026a](https://arxiv.org/html/2605.12652#bib.bib67)\], Yanet al\.\[[2025](https://arxiv.org/html/2605.12652#bib.bib68)\], Liet al\.\[[2025a](https://arxiv.org/html/2605.12652#bib.bib69)\]\. Process\- and outcome\-level verifiers refine the supervision beyond a single sparse trajectory reward\. Step\-level verifiers and process reward models grade intermediate states, providing denser credit assignment for reasoningLightmanet al\.\[[2024](https://arxiv.org/html/2605.12652#bib.bib70)\], Cobbeet al\.\[[2021](https://arxiv.org/html/2605.12652#bib.bib71)\], Setluret al\.\[[2025](https://arxiv.org/html/2605.12652#bib.bib72)\], Zhanget al\.\[[2025c](https://arxiv.org/html/2605.12652#bib.bib73)\], Guanet al\.\[[2024](https://arxiv.org/html/2605.12652#bib.bib74)\], Dixitet al\.\[[2026](https://arxiv.org/html/2605.12652#bib.bib75)\]\. Hybrid post\-training pipelines couple on\-policy distillation with reinforcement learning from verifiers, using teacher\-derived signals as an auxiliary objective alongside reward optimizationXuet al\.\[[2025b](https://arxiv.org/html/2605.12652#bib.bib76),[a](https://arxiv.org/html/2605.12652#bib.bib77)\], Lyuet al\.\[[2025](https://arxiv.org/html/2605.12652#bib.bib78)\], Zhanget al\.\[[2026b](https://arxiv.org/html/2605.12652#bib.bib79)\]\. While these methods do leverage multiple rollouts per prompt—typically through advantage normalization, rejection sampling, or success\-conditioned filtering—the supervisory signal for any single trajectory is computed without conditioning on the contents of the others\. The work most adjacent to ours therefore treats successes and failures from the same group as complementary streams of evidence at the level of advantages or filters, but stops short of routing both into a single, peer\-conditioned distillation target\.

## Appendix GLimitation

MOPD relies on the availability of multiple rollouts and a verifier or reward function that can partition them into successful and failed trajectories\. This makes the method most directly applicable to verifiable domains such as code, math, tool use, and structured QA, while extending it to open\-ended generation tasks may require reliable preference models or process\-level evaluators\. Peer conditioning increases context length and teacher\-query cost when raw rollouts are concatenated, especially for long reasoning traces; selecting the most informative peers under context\-length constraints is an open question\. Finally, the quality of the teacher signal depends on the correctness of the success/failure labels: noisy verifiers or mislabeled rollouts may introduce misleading positive or negative evidence\. Future work should study more robust peer selection, better handling of noisy feedback, and extensions to less easily verifiable tasks\.

## Appendix HSocietal Impacts

MOPD may have positive societal impacts by improving the efficiency of post\-training and making stronger reasoning capabilities more accessible in smaller language models\. Better distillation can reduce inference cost and energy use, and may help deploy capable models in resource\-constrained settings\. In domains such as code generation, tool use, scientific question answering, and mathematical reasoning, more reliable small models could support education, research assistance, and productivity tools\.

At the same time, improving the reasoning and tool\-use capabilities of smaller models may also increase risks\. More capable and cheaper models could be misused for generating harmful code, automating deceptive content, or executing tool\-use workflows in unintended ways\. Because MOPD learns from both successful and failed trajectories, poor verifier design or biased feedback could also reinforce undesirable behavior if incorrect success/failure labels are treated as reliable supervision\. These risks suggest that peer\-conditioned distillation should be paired with careful verifier design, safety evaluations, misuse monitoring, and appropriate deployment safeguards, especially when applied to open\-ended, high\-impact, or tool\-using systems\.

## Appendix IDeclaration of LLM Usage

We used large language models as assistance tools during the preparation of this paper\. Specifically, LLMs were used to help draft, revise, and polish parts of the manuscript, including improving clarity, grammar, and organization\. We also used LLM assistance for code writing and debugging, such as drafting utility scripts, checking implementation details, and improving code readability\.