Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress

# Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress
Source: [https://arxiv.org/html/2608.19408](https://arxiv.org/html/2608.19408)
###### Abstract

On\-policy distillation \(OPD\) has emerged as an effective framework for post\-training language models by pairing student\-generated trajectories with dense token\-level supervision from a teacher\. However, OPD implicitly assumes that teacher\-derived rewards are an appropriate proxy for reasoning progress, and therefore treats all teacher feedback equally during policy optimization\. While in practice, this assumption does not always hold\. We observe that teacher\-derived rewards often conflict with genuine reasoning progress, as reasoning steps with clear reasoning advancement may still receive lower distillation rewards, simply due to deviation from teacher’s outputs\. To address this mismatch, we propose Reasoning\-Progress\-Aware Reward Filtering for On\-Policy Distillation \(R2\-OPD\), which constructs two within\-trajectory rankings of reasoning spans, one from teacher\-derived rewards and the other from independently estimated progress reward\. Distillation rewards are selectively suppressed whenever the two rankings disagree, reducing supervision that conflicts with reasoning progress while preserving effective teacher guidance\. Our approach shows consistent improvement over standard OPD especially regarding reasoning performances\.

## Introduction

Knowledge distillation transfers the predictive behavior of a capable teacher model to a naive student model, offering a practical route to improving language models without the full expense of training or deploying the teacher\([12](https://arxiv.org/html/2608.19408#bib.bib1);[18](https://arxiv.org/html/2608.19408#bib.bib2)\)\. For autoregressive generation, conventional distillation is commonly performed on static teacher\-generated datasets, leading to an exposure bias between the training trajectories and inference trajectories\([3](https://arxiv.org/html/2608.19408#bib.bib3)\)\. On\-policy distillation \(OPD\) addresses this mismatch by taking student\-generated responses as training inputs and querying the teacher at the specific states the student encounters\. Combined with token\-level distribution matching and policy optimization, OPD provides dense supervision across the entire response, serving as a promising paradigm for post\-training reasoning models and model merging\([29](https://arxiv.org/html/2608.19408#bib.bib16);[35](https://arxiv.org/html/2608.19408#bib.bib4)\)\.

Despite these advantages, token\-level teacher supervision does not directly*measure whether a reasoning span advances the solution\.*At every decoding step, OPD rewards the student for staying close to the teacher distribution, implicitly treating teacher similarity as a proxy for reasoning and answer quality\([11](https://arxiv.org/html/2608.19408#bib.bib19)\)\. This proxy can be unreliable because a student\-generated span may increase the probability of arriving at the correct answer even if it departs from the teacher trajectory\([16](https://arxiv.org/html/2608.19408#bib.bib20)\)\. Conversely, a teacher\-like span does not necessarily make meaningful, concise progress toward the ultimate solution\. Consequently, a constructive reasoning step may receive unfavorable distillation guidance simply for deviating from the teacher\. Applying such uncalibrated signals uniformly risks suppressing valid reasoning pathways rather than effectively transferring functional knowledge\([24](https://arxiv.org/html/2608.19408#bib.bib21)\)\.

This observation motivates a distinction between*distillation compatibility*and*reasoning progress*\. While the former measures agreement with the teacher, the latter measures how an intermediate state changes the likelihood of solving the task\. Process rewards provide a natural teacher\-independent surrogate for reasoning progress by sampling continuations from successive reasoning states and comparing their solve probabilities to estimate the marginal contribution of each span\([15](https://arxiv.org/html/2608.19408#bib.bib18);[20](https://arxiv.org/html/2608.19408#bib.bib7);[30](https://arxiv.org/html/2608.19408#bib.bib11)\)\. However, directly combining raw process rewards with token\-level divergence can be ineffective, as these two signals operate on different scales and exhibit fine\-grained noise\. The former inherits Monte Carlo noise from a finite number of continuations, while the latter can fluctuate with local lexical choices and teacher uncertainty\.

In this work, we proposeReasoning\-Progress\-Aware Reward Filtering for On\-Policy Distillation \(R2\-OPD\), a novel reward\-filtering framework for reasoning\-oriented OPD\. Across consecutive reasoning spans, R2\-OPD first merges adjacent reasoning spans with sign\-consistent process rewards\. Such a procedure removes dependence on noisy internal boundary estimates and produces more stable units for reward calibration\. It then aggregates token\-level divergence within each merged span, and compares the relative ordering induced by reasoning progress with that induced by teacher divergence\. When a higher\-progress span is penalized more strongly for departing from the teacher, the two signals exhibit a local ranking conflict\. Rather than replacing the OPD reward or adding process reward as a new optimization objective, R2\-OPD utilizes this conflict only as a reliability test and masks the distillation rewards of the most inconsistent spans\. In this way, the method preserves teacher guidance where it agrees with reasoning progress, while reducing supervision that may discourage productive reasoning\. This approach not only makes teacher guidance more effective, but also significantly boosts OPD performance in complex reasoning scenarios\. Our contributions are as follows:

- •We identify a common failure mode of reasoning\-oriented OPD and show that uniformly applied distillation rewards may provide misleading supervision\. By introducing R2\-OPD, it adopts independently estimated process rewards to construct a teacher\-independent reference, detects local progress–distillation ranking conflicts, and selectively masks unreliable segment\-level supervision\.
- •We develop sign\-consistent process\-reward merging and segment\-level divergence averaging, and provide theoretical results characterizing the cancellation of internal boundary\-estimation errors and the reduction of local divergence variance under weak dependence\.
- •We empirically demonstrate that filtering progress\-conflicting distillation signals consistently improves reasoning performance over standard OPD\.

## Preliminaries

### On\-Policy Distillation

Letxxandy=\(y1,…,yT\)∼πS\(⋅∣x\)y=\(y\_\{1\},\\dots,y\_\{T\}\)\\sim\\pi\_\{S\}\(\\cdot\\mid x\)denote a prompt and a response sampled on\-policy from the student model, respectively\. Letht=\(x,y<t\)h\_\{t\}=\(x,y\_\{<t\}\)denote the decoding context at steptt\. In this work, reverse Kullback–Leibler \(KL\) divergence\-based On\-Policy Distillation \(OPD\) is taken into investigation\. Specifically, OPD optimizes the student over its self\-generated trajectories by minimizing the reverse KL divergence between the student and teacher token distributions at each decoding step\([1](https://arxiv.org/html/2608.19408#bib.bib5);[7](https://arxiv.org/html/2608.19408#bib.bib6)\)\. Given teacher policyπT\\pi\_\{T\}, evaluating this reverse KL divergence over the full vocabulary effectively weighs each token by student policyπS\\pi\_\{S\}, leading to the following objective:

ℒOPD\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{OPD\}\}=𝔼x,y∼πS\(⋅∣x\)​\[1T​∑t=1TℓtKL\],\\displaystyle=\\mathbb\{E\}\_\{\\begin\{subarray\}\{c\}x,y\\sim\\pi\_\{S\}\(\\cdot\\mid x\)\\end\{subarray\}\}\\left\[\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\ell\_\{t\}^\{\\mathrm\{KL\}\}\\right\],\(1\)ℓtKL\\displaystyle\\ell\_\{t\}^\{\\mathrm\{KL\}\}=DKL\(πS\(⋅∣ht\)∥πT\(⋅∣ht\)\)\\displaystyle=D\_\{\\mathrm\{KL\}\}\\\!\\left\(\\pi\_\{S\}\(\\cdot\\mid h\_\{t\}\)\\,\\\|\\,\\pi\_\{T\}\(\\cdot\\mid h\_\{t\}\)\\right\)=∑v∈𝒱πS​\(v∣ht\)​log⁡πS​\(v∣ht\)πT​\(v∣ht\)\.\\displaystyle=\\sum\_\{v\\in\\mathcal\{V\}\}\\pi\_\{S\}\(v\\mid h\_\{t\}\)\\log\\frac\{\\pi\_\{S\}\(v\\mid h\_\{t\}\)\}\{\\pi\_\{T\}\(v\\mid h\_\{t\}\)\}\.Directly computing the KL divergence over the entire vocabulary𝒱\\mathcal\{V\}at every step is computationally prohibitive, and extremely low\-probability tokens offer negligible supervisory signals\. Consequently, practical OPD implementations approximate Eq\. \([1](https://arxiv.org/html/2608.19408#Sx2.E1)\) using a support set𝒮t⊆𝒱\\mathcal\{S\}\_\{t\}\\subseteq\\mathcal\{V\}of sizeHH\. The support can be constructed from theHHhighest\-probability tokens of the student, the teacher, or their union\.

For the student\-derived support used in our implementation, we define the normalized student weight

wt,v=πS​\(v∣ht\)∑u∈𝒮tπS​\(u∣ht\),v∈𝒮t,w\_\{t,v\}=\\frac\{\\pi\_\{S\}\(v\\mid h\_\{t\}\)\}\{\\sum\_\{u\\in\\mathcal\{S\}\_\{t\}\}\\pi\_\{S\}\(u\\mid h\_\{t\}\)\},\\qquad v\\in\\mathcal\{S\}\_\{t\},\(2\)And then we compute the support\-restricted approximation to the token\-level KL loss as

ℓtKL,𝒮=∑v∈𝒮twt,v​log⁡πS​\(v∣ht\)πT​\(v∣ht\)\.\\ell\_\{t\}^\{\\mathrm\{KL\},\\mathcal\{S\}\}=\\sum\_\{v\\in\\mathcal\{S\}\_\{t\}\}w\_\{t,v\}\\log\\frac\{\\pi\_\{S\}\(v\\mid h\_\{t\}\)\}\{\\pi\_\{T\}\(v\\mid h\_\{t\}\)\}\.\(3\)

### Process Reward Estimation

![Refer to caption](https://arxiv.org/html/2608.19408v1/Figures/bar_score_breakdown.png)Figure 1:Distribution of evaluation outcomes for DeepSeek\-R1\-Distill\-Qwen\-1\.5B and Qwen3\-1\.7B on DAPO \(500 samples, max 7,168 response tokens\)\.While the token\-level supervision introduced above measures distributional agreement with the teacher, it fails to explicitly reflect the correctness of the final outcome\. To bridge this gap, recent OPD methods for reasoning tasks have explored the incorporation of outcome\-level correctness to calibrate or filter token\-level teacher supervision\([13](https://arxiv.org/html/2608.19408#bib.bib25);[2](https://arxiv.org/html/2608.19408#bib.bib32);[42](https://arxiv.org/html/2608.19408#bib.bib31)\)\. However, outcome feedback indicates only whether a completed response is correct and provides limited information about which intermediate steps advance the solution, introduce an error, or recover from an earlier mistake\. Furthermore, as illustrated in Figure[1](https://arxiv.org/html/2608.19408#Sx2.F1), a third of the responses contain correct reasoning paths, but are truncated due to overthinking\. Relying solely on outcome\-based filtering thus fails to fully exploit such valuable trajectories\. Process rewards address this temporal credit\-assignment problem by evaluating intermediate reasoning states or transitions, thereby providing a finer\-grained account of how the likelihood of task success evolves along a trajectory\([20](https://arxiv.org/html/2608.19408#bib.bib7);[30](https://arxiv.org/html/2608.19408#bib.bib11)\)\.

This form of supervision has become useful across broader post\-training workflows\([5](https://arxiv.org/html/2608.19408#bib.bib17)\)\. The process reward can guide reinforcement learning by assigning credit to individual reasoning steps, support rejection sampling or reranking among candidate solutions, and steer test\-time search toward promising partial trajectories\([21](https://arxiv.org/html/2608.19408#bib.bib12);[40](https://arxiv.org/html/2608.19408#bib.bib15)\)\. When manual step\-level annotation is unavailable, prior work has constructed automatic process supervision from the success rate of sampled continuations, grounding the value of an intermediate state in its probability of eventually reaching a correct answer\([31](https://arxiv.org/html/2608.19408#bib.bib8);[25](https://arxiv.org/html/2608.19408#bib.bib10);[26](https://arxiv.org/html/2608.19408#bib.bib9)\)\.

Following this intuition, we estimate the solve probability of intermediate reasoning states through on\-policy rollouts, and design process rewards as the incremental change in solve probability across consecutive reasoning states\. Given a responsey=\(y1,…,yT\)y=\(y\_\{1\},\\dots,y\_\{T\}\), we partition it intoMMcontiguous reasoning segmentsΣ⁡\(y\)=\{σ1,…,σM\}\\Sigma\(y\)=\\\{\\sigma\_\{1\},\\dots,\\sigma\_\{M\}\\\}\. Segment boundaries are determined in two stages\. We first identify candidate boundary positions inyyby matching its tokens against a fixed lexicon𝒯\\mathcal\{T\}of self\-reflective discourse markers\. To prevent over\-segmentation caused by consecutive discourse markers with minimal reasoning content, we accept a candidate boundarybm−1b\_\{m\-1\}only whenNsent\(ybm−1\+1:t\)≥SminN\_\{\\mathrm\{sent\}\}\(y\_\{b\_\{m\-1\}\+1:t\}\)\\geq S\_\{\\min\}\. Here,Nsent​\(⋅\)N\_\{\\mathrm\{sent\}\}\(\\cdot\)measures the sentence count, andSminS\_\{\\min\}specifies the minimum number of sentences required between successive accepted boundaries\. The complete lexicon and matching procedure are provided in Appendix[B\.5](https://arxiv.org/html/2608.19408#A2.SS5.SSSx1)\.

To quantify the reasoning progress at boundary statepm=\(x,y1:bm\)p\_\{m\}=\(x,y\_\{1:b\_\{m\}\}\), we estimate its solve probability viaNevalN\_\{\\mathrm\{eval\}\}on\-policy Monte Carlo rollouts:

cm\(1\),…,cm\(Neval\)∼πS\(⋅∣p¯m,ℐans\),c\_\{m\}^\{\(1\)\},\\dots,c\_\{m\}^\{\(N\_\{\\mathrm\{eval\}\}\)\}\\;\\sim\\;\\pi\_\{S\}\(\\cdot\\mid\\bar\{p\}\_\{m\},\\mathcal\{I\}\_\{\\text\{ans\}\}\),\(4\)wherep¯m\\bar\{p\}\_\{m\}is the format\-standardized prefix andℐans\\mathcal\{I\}\_\{\\text\{ans\}\}is an answer\-eliciting prompt \(Appendix[B\.5](https://arxiv.org/html/2608.19408#A2.SS5.SSSx2)\)\. For each prefix statepmp\_\{m\}, the corresponding solve probability is estimated by Monte Carlo sampling using the ground\-truth answerggand verifiable reward functionℛ⁡\(⋅,⋅\)\\mathcal\{R\}\(\\cdot,\\cdot\),

S^m=\{0,m=0,1Neval​∑ℓ=1Nevalℛ⁡\(pm∥cm\(ℓ\),g\),1≤m≤M−1,ℛ⁡\(y,g\),m=M\.\\hat\{S\}\_\{m\}=\\begin\{cases\}0,&m=0,\\\\\[2\.0pt\] \\frac\{1\}\{N\_\{\\mathrm\{eval\}\}\}\\sum\_\{\\ell=1\}^\{N\_\{\\mathrm\{eval\}\}\}\\mathcal\{R\}\\big\(p\_\{m\}\\mathbin\{\\\|\}c\_\{m\}^\{\(\\ell\)\},\\,g\\big\),&1\\leq m\\leq M\-1,\\\\\[2\.0pt\] \\mathcal\{R\}\(y,g\),&m=M\.\\end\{cases\}\(5\)Eq\. \([5](https://arxiv.org/html/2608.19408#Sx2.E5)\) assigns zero credit to the initial state, and reuses the actual response’s correctness form=Mm=M, avoiding unnecessary generation passes for the terminal state\.

Theprocess rewardP​RmPR\_\{m\}for segmentσm\\sigma\_\{m\}is defined as the marginal gain in solve probability across its boundary:

PRm≜S^m−S^m−1,m=1,…,M\.PR\_\{m\}\\;\\triangleq\\;\\hat\{S\}\_\{m\}\-\\hat\{S\}\_\{m\-1\},\\qquad m=1,\\dots,M\.\(6\)To avoid redundant computation,P​RmPR\_\{m\}is evaluated only when segmentation succeeds and a string\-level check confirmsg∈yg\\in y\. Otherwise, the trajectory is tagged as*PR\-unavailable*and passed through unfiltered\. This procedure ensures R2\-OPD never penalizes trajectories which lack explicit process signals\.

## Reasoning\-Progress\-Aware Reward Filtering

The token\-level OPD reward measures how closely the student matches the teacher at each decoding state, but it does not indicate whether a reasoning span actually improves the student’s ability to solve the problem\. Conversely, terminal correctness provides only trajectory\-level supervision and cannot attribute success or failure to individual reasoning spans\. As a result, a reasoning span that genuinely increases the student’s probability of solving the problem may still receive a low distillation reward simply because its token distribution differs from that of the teacher\. To address this limitation, we use the process reward defined in Eq\. \([6](https://arxiv.org/html/2608.19408#Sx2.E6)\) as an independent reference when evaluating the reliability of the OPD supervision for each reasoning span\. Importantly, the process reward neither replaces the OPD reward nor serves as an additional optimization objective\. Instead, it provides a teacher\-independent estimate of reasoning progress, allowing us to assess whether the distillation signal is aligned with the estimated contribution of each reasoning span\. Our method consists of three stages, which are described in detail in the following subsections\. This algorithm is illustrated in Figure[3](https://arxiv.org/html/2608.19408#Sx3.F3)\. For clarity, we describe the segment\-level operations for a single response and omit the response index unless multiple responses must be distinguished\.

##### Noise Reduction via Sign\-Consistent merging\.

![Refer to caption](https://arxiv.org/html/2608.19408v1/Figures/process_reward_example.png)Figure 2:representative mismatch between reasoning progress and the OPD divergence signal\. A reasoning regressive segment \(J9J\_\{9\}\) receives a smaller KL divergence under Equation \([9](https://arxiv.org/html/2608.19408#Sx3.E9)\) than segments that derive or recover the correct solution, respectively\(J2J\_\{2\}andJ10J\_\{10\}\)\.In practice, both process rewards and token\-level distillation signals can be very noisy at a fine granularity, posing challenges for policy optimization\. Process rewards are estimated from a finite number of continuations, while token\-level divergences can fluctuate substantially because of local lexical choices, teacher uncertainty, and the top\-kkapproximation\. Directly comparing these signals over short reasoning spans may therefore frequently produce unstable and even conflicting guidance\. To overcome this challenge, R2\-OPD first aggregates adjacent spans that exhibit a consistent direction of estimated progress, obtaining coarser units on which both signals can be evaluated more reliably\.

Specifically, because estimation with a smallNevalN\_\{\\mathrm\{eval\}\}renders individualP​RmPR\_\{m\}noisy, we merge adjacent segments whose nonzero process rewards have the same sign\. A zero\-valued process reward is absorbed into the current run and does not create a new boundary; a new run begins only when the nonzero sign changes from positive to negative or vice versa\. This partitions\{1,…,M\}\\\{1,\\dots,M\\\}intonnmaximal contiguous index sets𝒥1,…,𝒥n\\mathcal\{J\}\_\{1\},\\dots,\\mathcal\{J\}\_\{n\}\. Each index set𝒥j\\mathcal\{J\}\_\{j\}induces a merged segmentσ~j\\tilde\{\\sigma\}\_\{j\}by concatenating its constituent segments, with aggregated process reward:

P​R~j≜∑m∈𝒥jPRm,j=1,…,n\.\\widetilde\{PR\}\_\{j\}\\;\\triangleq\\;\\sum\_\{m\\in\\mathcal\{J\}\_\{j\}\}PR\_\{m\},\\qquad j=1,\\dots,n\.\(7\)Because process rewards are defined as differences between consecutive solve\-probability estimates, summing them over a merged run produces a telescoping sum\. Consequently, the aggregated reward depends only on the estimates at the two endpoints rather than on those at the internal boundaries\. We formalize this property below\.

![Refer to caption](https://arxiv.org/html/2608.19408v1/Figures/reasoning_progress_aware_reward_filtering_for_OPD.png)Figure 3:Overview of the proposed Reasoning\-Progress\-Aware Reward Filtering OPD\. Adjacent reasoning spans with consistent progress signals are first merged\. For each merged segment, we sum its process rewards and average its token\-level KL divergences\. Rank conflicts between the resulting segment\-level signals are then used to identify unreliable spans, which are masked during the OPD policy update\.###### Lemma 1\(Telescoping of Aggregated Process Rewards\)\.

Let𝒥j=\{a,…,b\}\\mathcal\{J\}\_\{j\}=\\\{a,\\dots,b\\\}be a merged run, and write each estimated solve probability asS^m=Sm\+ϵm\\hat\{S\}\_\{m\}=S\_\{m\}\+\\epsilon\_\{m\}, whereSmS\_\{m\}is the underlying solve probability andϵm\\epsilon\_\{m\}is its Monte Carlo estimation error\. Then

P​R~j=\(Sb−Sa−1\)\+\(ϵb−ϵa−1\)\.\\widetilde\{PR\}\_\{j\}=\(S\_\{b\}\-S\_\{a\-1\}\)\+\(\\epsilon\_\{b\}\-\\epsilon\_\{a\-1\}\)\.\(8\)Hence, the aggregated reward is independent of the internal boundary errorsϵa,…,ϵb−1\\epsilon\_\{a\},\\dots,\\epsilon\_\{b\-1\}and retains only the errors at the two endpoints\.

###### Proof\.

SubstitutingP​Rm=S^m−S^m−1PR\_\{m\}=\\hat\{S\}\_\{m\}\-\\hat\{S\}\_\{m\-1\}into Eq\. \([7](https://arxiv.org/html/2608.19408#Sx3.E7)\) gives

∑m=abP​Rm=∑m=ab\(S^m−S^m−1\)=S^b−S^a−1,\\sum\_\{m=a\}^\{b\}PR\_\{m\}=\\sum\_\{m=a\}^\{b\}\(\\hat\{S\}\_\{m\}\-\\hat\{S\}\_\{m\-1\}\)=\\hat\{S\}\_\{b\}\-\\hat\{S\}\_\{a\-1\},because all internal terms telescope\. SubstitutingS^m=Sm\+ϵm\\hat\{S\}\_\{m\}=S\_\{m\}\+\\epsilon\_\{m\}yields Eq\. \([8](https://arxiv.org/html/2608.19408#Sx3.E8)\)\. ∎

Restricting merging to sign\-consistent runs avoids canceling progress signals that reflect genuine changes in reasoning direction\. The resulting segments therefore capture progress across major turning points while eliminating dependence on noisy internal boundary estimates\.

### Rank\-Based Conflict Detection and Masking

For each merged segmentj≥1j\\geq 1, letℐj⊆\{1,…,T\}\\mathcal\{I\}\_\{j\}\\subseteq\\\{1,\\dots,T\\\}denote the token positions belonging toσ~j\\tilde\{\\sigma\}\_\{j\}\. In order to compare token\-level distillation feedback with segment\-level process rewards at a common granularity while avoiding a systematic dependence on segment length, we average the token\-level signals within each merged segment\. We refer to the resulting mean as the average segment\-level distillation lossκj\\kappa\_\{j\}:

κj=1\|ℐj\|​∑t∈ℐjℓtKL,𝒮,\\kappa\_\{j\}\\;=\\;\\frac\{1\}\{\|\\mathcal\{I\}\_\{j\}\|\}\\sum\_\{t\\in\\mathcal\{I\}\_\{j\}\}\\ell\_\{t\}^\{\\mathrm\{KL\},\\mathcal\{S\}\},\(9\)where a largerκj\\kappa\_\{j\}indicates stronger disagreement with the teacher\. Here, token distance refers to the difference\|t−s\|\|t\-s\|between two token positions in the response sequence\.

###### Proposition 2\(Variance Reduction by Segment Averaging\)\.

LetLj=\|ℐj\|L\_\{j\}=\|\\mathcal\{I\}\_\{j\}\|, and suppose that within merged segmentjjthe support\-restricted token\-level losses satisfy

ℓtKL,𝒮=μj\+ϵt,t∈ℐj,\\ell\_\{t\}^\{\\mathrm\{KL\},\\mathcal\{S\}\}=\\mu\_\{j\}\+\\epsilon\_\{t\},\\qquad t\\in\\mathcal\{I\}\_\{j\},whereμj\\mu\_\{j\}is locally constant,𝔼⁡\[ϵt\]=0\\mathbb\{E\}\[\\epsilon\_\{t\}\]=0, and

\|Cov⁡\(ϵt,ϵs\)\|≤σ2​ρ\|t−s\|for some​0≤ρ<1\.\\left\|\\operatorname\{Cov\}\(\\epsilon\_\{t\},\\epsilon\_\{s\}\)\\right\|\\leq\\sigma^\{2\}\\rho^\{\|t\-s\|\}\\qquad\\text\{for some \}0\\leq\\rho<1\.Thenκj\\kappa\_\{j\}is an unbiased estimator ofμj\\mu\_\{j\}, and

Var⁡\(κj\)≤σ2Lj​1\+ρ1−ρ\.\\operatorname\{Var\}\(\\kappa\_\{j\}\)\\leq\\frac\{\\sigma^\{2\}\}\{L\_\{j\}\}\\frac\{1\+\\rho\}\{1\-\\rho\}\.\(10\)Thus, provided thatσ2\\sigma^\{2\}andρ\\rhoare bounded independently ofLjL\_\{j\}, its variance decreases asO⁡\(1/Lj\)O\(1/L\_\{j\}\)\.

A detailed proof is provided in Appendix[A\.1](https://arxiv.org/html/2608.19408#A1.SS1)\. Proposition[2](https://arxiv.org/html/2608.19408#Thmproposition2)formalizes how averaging reduces sensitivity to isolated token\-level loss fluctuations when correlations decay with token distance\. This merging trades some temporal resolution for a more stable estimate of the teacher–student discrepancy over a coherent progress interval\.

We do not assume that proximity to the teacher is universally equivalent to reasoning quality\. Instead, we use process rewards to identify local ranking conflicts between the teacher\-imitation signal and the estimated contribution of reasoning segments\. Specifically, if segmentaahas a larger estimated process contribution than segmentbb, but is assigned a larger average distillation loss, then the distillation signal penalizes the empirically more useful segment more strongly\. We regard such an ordering as locally inconsistent:

P​R~a\>P​R~bandκa\>κb\.\\widetilde\{PR\}\_\{a\}\>\\widetilde\{PR\}\_\{b\}\\quad\\text\{and\}\\quad\\kappa\_\{a\}\>\\kappa\_\{b\}\.\(11\)Rather than treated as an assumption that teacher proximity directly measures segment quality, this criterion is used as an operational diagnostic\. A representative case that violates this expected relationship is illustrated in Figure[2](https://arxiv.org/html/2608.19408#Sx3.F2)\.

The process reward and segment\-level average distillation loss have different scales and need not be calibrated across trajectories\. We therefore compare their relative order within each response instead of requiring their absolute values to be directly comparable\([4](https://arxiv.org/html/2608.19408#bib.bib13);[23](https://arxiv.org/html/2608.19408#bib.bib14)\)\. This within\-response ranking is insensitive to trajectory\-wide offsets and positive rescaling, and it reduces the influence of absolute\-score calibration noise\. The ranking determines the expected direction of the relation between progress and average distillation loss, while the original score gaps in Eq\. \([9](https://arxiv.org/html/2608.19408#Sx3.E9)\) quantify the severity of each detected conflict\.

For each responseii, we exclude the initial prefix segment \(j=1j=1\) and sort the remaining segments by decreasing process\-reward progressPR~j\\widetilde\{\\mathrm\{PR\}\}\_\{j\}, breaking ties by increasing lossκj\\kappa\_\{j\}\. Letρi\\rho\_\{i\}denote this sorted sequence of segment indices\. We form adjacent pairs𝒫i=\{\(ρi,r,ρi,r\+1\)∣PR~ρi,r\>PR~ρi,r\+1\}\\mathcal\{P\}\_\{i\}=\\\{\(\\rho\_\{i,r\},\\rho\_\{i,r\+1\}\)\\mid\\widetilde\{\\mathrm\{PR\}\}\_\{\\rho\_\{i,r\}\}\>\\widetilde\{\\mathrm\{PR\}\}\_\{\\rho\_\{i,r\+1\}\}\\\}, where adjacency is defined on the ranked sequence rather than temporal order\.

For each pair\(a,b\)∈𝒫i\(a,b\)\\in\\mathcal\{P\}\_\{i\}, an order violation occurs if the higher\-progress segmentaaincurs a higher distillation loss, quantified asvi​\(a,b\)=\[\(PR~a−PR~b\)​\(κa−κb\)\]\+v\_\{i\}\(a,b\)=\\left\[\(\\widetilde\{\\mathrm\{PR\}\}\_\{a\}\-\\widetilde\{\\mathrm\{PR\}\}\_\{b\}\)\(\\kappa\_\{a\}\-\\kappa\_\{b\}\)\\right\]\_\{\+\}\. The inconsistency score of segmentjjis accumulated across all adjacent pairs it participates in:

Incj=∑\(a,b\)∈𝒫i:j∈\{a,b\}vi\(a,b\)\.\\mathrm\{Inc\}\_\{j\}=\\sum\_\{\(a,b\)\\in\\mathcal\{P\}\_\{i\}:\\,j\\in\\\{a,b\\\}\}v\_\{i\}\(a,b\)\.\(12\)
##### Segment Masking\.

Masking is applied to eligible responses containing at leastmax⁡\(3,nmin\)\\max\(3,n\_\{\\min\}\)segments with\|𝒫i\|\>0\|\\mathcal\{P\}\_\{i\}\|\>0\. Given a masking ratioq%q\\%, the budget for responseiiisbi=⌈q100​ni⌉b\_\{i\}=\\lceil\\frac\{q\}\{100\}n\_\{i\}\\rceil\. We rank candidate segments \(j≥2j\\geq 2\) byIncj\\mathrm\{Inc\}\_\{j\}in descending order, and select the top\-bib\_\{i\}segments with strictly positive inconsistency scores for masking\.

Letℳi\\mathcal\{M\}\_\{i\}denote the set of selected segment indices in responseii\. The token\-level mask is defined asMt\(i\)=0M\_\{t\}^\{\(i\)\}=0if tokenttbelongs to any segmentj∈ℳij\\in\\mathcal\{M\}\_\{i\}, andMt\(i\)=1M\_\{t\}^\{\(i\)\}=1otherwise\. LetZi=∑t=1TiMt\(i\)Z\_\{i\}=\\sum\_\{t=1\}^\{T\_\{i\}\}M\_\{t\}^\{\(i\)\}denote the number of unmasked tokens in responseii\. Applying this mask yields the R2\-OPD objective:

ℒR2​\-​OPD=𝔼xi,yi∼πS\(⋅∣xi\)\[1Zi∑t=1TiMt\(i\)ℓtKL,𝒮,\(i\)\]\.\\mathcal\{L\}\_\{\\mathrm\{R^\{2\}\\text\{\-\}OPD\}\}=\\mathbb\{E\}\_\{x\_\{i\},\\,y\_\{i\}\\sim\\pi\_\{S\}\(\\cdot\\mid x\_\{i\}\)\}\\left\[\\frac\{1\}\{Z\_\{i\}\}\\sum\_\{t=1\}^\{T\_\{i\}\}M\_\{t\}^\{\(i\)\}\\,\\ell\_\{t\}^\{\\mathrm\{KL\},\\mathcal\{S\},\(i\)\}\\right\]\.\(13\)Masked tokens therefore contribute neither to the loss numerator nor to the normalization denominator, keeping the per\-response loss scale comparable across different masking ratios\. Apart from masking and the corresponding renormalization, the policy\-optimization procedure remains otherwise unchanged\.

## Experiments

Table 1:OPD performance on DeepSeek\-R1\-Distill\-Qwen\-1\.5B with JustRL as the teacher model\.### Experimental Setup

Our primary experiments employ DeepSeek\-R1\-Distill\-Qwen\-1\.5B as the student and JustRL\-1\.5B as the teacher\. As highlighted by[19](https://arxiv.org/html/2608.19408#bib.bib30), effective distillation requires a teacher with complementary knowledge rather than merely a larger parameter scale\. The former is a compact model distilled from DeepSeek\-R1 and inherits its reasoning\-oriented training recipe\([8](https://arxiv.org/html/2608.19408#bib.bib27)\), while the latter already demonstrates that a simple RL recipe can elicit strong reasoning performance from a 1\.5B model\([9](https://arxiv.org/html/2608.19408#bib.bib26)\)\. To assess whether our approach generalizes across different model families, we further evaluate on a heterogeneous setup using Qwen3\-1\.7B\([37](https://arxiv.org/html/2608.19408#bib.bib29)\)as the student and e3\-1\.7B\([27](https://arxiv.org/html/2608.19408#bib.bib28)\)as the teacher\.

All models are trained on the deduplicated DAPO\-Math\-17K dataset\([19](https://arxiv.org/html/2608.19408#bib.bib30)\)for one epoch using the AdamW optimizer with a learning rate of5×10−65\\times 10^\{\-6\}and a global batch size of 64\. The maximum prompt and response lengths are set to 1,024 and 7,168 tokens, respectively\. For the support\-restricted reverse\-KL objective, we setH=16H=16and construct𝒮t\\mathcal\{S\}\_\{t\}from the student’s 16 highest\-probability tokens at each decoding step\. Downstream reasoning performance is evaluated on AIME 2024\([14](https://arxiv.org/html/2608.19408#bib.bib33)\), AIME 2025\([22](https://arxiv.org/html/2608.19408#bib.bib34)\), and OlympiadBench\([10](https://arxiv.org/html/2608.19408#bib.bib35)\)\. For Rank\-Based Conflict Detection and Segment Masking, we generateNeval=8N\_\{\\mathrm\{eval\}\}=8answer\-eliciting rollouts per evaluated boundary using a sampling temperature of 0\.7,top\-​k=50\\text\{top\-\}k=50,top\-​p=1\.0\\text\{top\-\}p=1\.0, and a maximum rollout length of 300 tokens\. Hyperparameters for conflict detection are set toSmin=3S\_\{\\min\}=3andnmin=3n\_\{\\min\}=3, with a segment\-level masking ratio ofq=30%q=30\\%\. Complete optimization, rollout\-construction, and filtering details are provided in Appendix[B](https://arxiv.org/html/2608.19408#A2)\.

### Evaluation Baselines

We compare R2\-OPD with standard OPD and four recent research\.OPDapplies dense token\-level reverse\-KL supervision from the teacher to trajectories sampled on\-policy from the student\([1](https://arxiv.org/html/2608.19408#bib.bib5)\)\.E\-OPDaugments reverse\-KL training with forward KL at positions where the teacher distribution has high entropy, aiming to preserve plausible alternatives and avoid mode collapse\([17](https://arxiv.org/html/2608.19408#bib.bib22)\)\.TIP\-OPDcharacterizes token importance using student entropy and the divergence between teacher and student, retaining both uncertain positions and low\-entropy positions where the student is confidently misaligned with the teacher\([36](https://arxiv.org/html/2608.19408#bib.bib23)\)\.Uni\-OPDaddresses insufficient student exploration through data balancing and unreliable teacher supervision using outcome\-guided margin calibration\([13](https://arxiv.org/html/2608.19408#bib.bib25)\)\.IW\-OPDstudies position bias in OPD and weights tokens according to accumulated student\-teacher discrepancy, emphasizing earlier positions while downweighting later positions of less reliable supervision\([34](https://arxiv.org/html/2608.19408#bib.bib24)\)\.

We report task accuracy using the task\-specific answer verifier\. For each benchmark, avg@4 averages accuracy over four sampled responses, whereas pass@4 measures whether at least one of the four responses is correct\. During evaluation, responses are sampled with a temperature of1\.01\.0, top\-kkof5050, and top\-ppof0\.950\.95, with the maximum response length set to8,1928\{,\}192tokens, which aligns with the literature\.

Table 2:Transfer performance of R2\-OPD on Qwen3\-1\.7B with e3\-1\.7B as the teacher model\.
### Main Results

We summarize results comparison on DeepSeek\-R1\-Distill\-Qwen\-1\.5B model in Table[1](https://arxiv.org/html/2608.19408#Sx4.T1)\. It shows that R2\-OPD achieves the best aggregate performance, with 35\.06 avg@4 and 51\.83 pass@4\. It outperforms standard OPD by 2\.51 and 4\.46 points, respectively, with the largest gains observed on the two AIME benchmarks\. Among the recent OPD variants, Uni\-OPD is the strongest overall competitor, yet R2\-OPD exceeds it by 4\.28 avg@4 and 5\.17 pass@4 points\. Uni\-OPD remains stronger on OlympiadBench, suggesting the benefit of progress\-aware filtering varies with the benchmark\.

The relatively weaker AIME results of E\-OPD and TIP\-OPD reflect an interaction between entropy\-sensitive supervision and the response\-length constraint\. Emphasizing uncertain positions can preserve alternative reasoning, but may also increase the likelihood of lengthy or unfinished reasoning on challenging problems\. Uni\-OPD may likewise be affected because its correctness\-aware calibration relies on complete, verifiable trajectories, whereas our maximum training response length is consistently shorter than that used in its original study, suggesting that R2\-OPD can result in more concise answers for reasoning tasks\.

Table[2](https://arxiv.org/html/2608.19408#Sx4.T2)further evaluates transfer to the Qwen3\-1\.7B/e3\-1\.7B pair\. Although the avg@4 gain over OPD is modest, R2\-OPD consistently improves the aggregate pass@4 from 45\.70 to 48\.19 and obtains higher pass@4 on all three benchmarks\. These results suggest that progress\-aware filtering transfers across model families, while its effective size may vary across datasets and metrics\.

### Ablations

We conduct ablation studies to examine two design choices central to R2\-OPD\. First, we vary the masking ratioqqto assess how the amount of filtered supervision affects downstream performance\. Second, we remove sign\-consistent segment merging to isolate its contribution to the reliability of PR–KL rank comparisons and the resulting task accuracy\. All ablations use the main DeepSeek\-R1\-Distill\-Qwen\-1\.5B/JustRL\-1\.5B configuration, with all other training and evaluation settings held fixed unless otherwise specified\.

![Refer to caption](https://arxiv.org/html/2608.19408v1/Figures/q-ablition.jpg)Figure 4:Sensitivity of R2\-OPD to the masking ratioqq\. A moderate ratio ofq=30q=30provides the best overall performance, whereas overly aggressive masking substantially degrades accuracy on AIME 2024 and AIME 2025\.##### Sensitivity to the Masking Ratio\.

Figure[4](https://arxiv.org/html/2608.19408#Sx4.F4)examines the effect of the masking ratioqq, which controls the proportion of merged reasoning segments considered for filtering\. A moderate masking ratio ofq=30q=30achieves the strongest performance across all three benchmarks, reaching 32\.50 on AIME 2024, 25\.83 on AIME 2025, and 46\.86 on OlympiadBench\. Reducing the ratio toq=10q=10yields lower performance on the two AIME benchmarks, suggesting that overly conservative filtering may retain a substantial amount of progress\-conflicting supervision\. Conversely, increasingqqto 50 causes a pronounced degradation, particularly on AIME 2024 and AIME 2025, where accuracy falls to 19\.17 and 12\.15, respectively\. This indicates that overly aggressive masking can remove otherwise useful teacher guidance\. Overall,q=30q=30provides a favorable balance between suppressing unreliable distillation signals and preserving informative supervision\. Performance on OlympiadBench remains comparatively stable across the three settings, indicating lower sensitivity to the masking ratio on this benchmark\.

##### Effectiveness of Sign\-Consistent Merging\.

![Refer to caption](https://arxiv.org/html/2608.19408v1/Figures/consistency_compare.png)Figure 5:PR–KL rank agreement during training with and without sign\-consistent segment merging\. Light lines show per\-step values, and dark lines show 15\-step moving averages\. The two vertical axes use the same numerical span\.To verify if sign\-consistent merging improves unit reliability for conflict detection, we measure PR–KL rank agreement at each step\. An adjacent segment pair is deemed consistent if higher process reward corresponds to a non\-greater distillation lossκ\\kappa\. We record the average proportion of consistent pairs per response, where 0\.5 represents a random baseline and higher values indicate stronger alignment between estimated progress and teacher–student agreement\.

As shown in Figure[5](https://arxiv.org/html/2608.19408#Sx4.F5), sign\-consistent merging raises the smoothed PR–KL rank agreement from approximately0\.200\.20–0\.270\.27to0\.550\.55–0\.730\.73and keeps it above the random baseline for nearly the entire training trajectory\. This persistent gap indicates that merging reduces boundary\-level noise and yields more reliable units for conflict detection\. Consistently, Table[3](https://arxiv.org/html/2608.19408#Sx4.T3)shows that merging improves avg@4 by 15\.00 and 14\.16 points on AIME 2024 and AIME 2025, respectively, while yielding a smaller decrease of 0\.96 points on OlympiadBench\. Together, these results show that improved rank reliability translates into more effective OPD supervision, particularly on the more challenging AIME benchmarks\.

Table 3:Effect of sign\-consistent segment merging on avg@4 accuracy\.Δ\\Deltadenotes the improvement of the full R2\-OPD over its no\-merging variant\.

## Related works

##### Process Rewards for Reasoning Language Models

Process rewards evaluate intermediate reasoning states and provide finer\-grained credit assignment than terminal outcome rewards\([30](https://arxiv.org/html/2608.19408#bib.bib11);[20](https://arxiv.org/html/2608.19408#bib.bib7)\)\. To reduce the cost of human annotation, subsequent methods automatically construct process supervision from sampled continuations, outcome verification, or model\-based judgments\([31](https://arxiv.org/html/2608.19408#bib.bib8);[21](https://arxiv.org/html/2608.19408#bib.bib12);[26](https://arxiv.org/html/2608.19408#bib.bib9);[41](https://arxiv.org/html/2608.19408#bib.bib37);[38](https://arxiv.org/html/2608.19408#bib.bib36)\)\. These signals have been used both training\-time policy optimization and inference\-time candidate selection or search\([40](https://arxiv.org/html/2608.19408#bib.bib15);[5](https://arxiv.org/html/2608.19408#bib.bib17);[39](https://arxiv.org/html/2608.19408#bib.bib38);[33](https://arxiv.org/html/2608.19408#bib.bib39)\)\. However, process rewards can be affected by finite\-rollout noise, verifier errors, and distribution shift\([15](https://arxiv.org/html/2608.19408#bib.bib18);[28](https://arxiv.org/html/2608.19408#bib.bib40);[6](https://arxiv.org/html/2608.19408#bib.bib41)\)\. Unlike prior work that directly optimizes or searches against process rewards, we use reasoning progress as an independent reliability test for OPD supervision and aggregate adjacent progress\-consistent spans to reduce estimation noise\.

##### Reliability and Selectivity of OPD Signals

Recent work has investigated the conditions under which dense token\-level supervision in OPD supports effective capability transfer\. Large student–teacher discrepancies can misdirect exploration, while teachers may prefer plausible but incorrect solutions over correct alternative reasoning paths\([32](https://arxiv.org/html/2608.19408#bib.bib42);[2](https://arxiv.org/html/2608.19408#bib.bib32)\)\. At a coarser granularity, aggregated token\-level guidance can rank correct trajectories below incorrect ones, motivating calibration against outcome rewards\. Together, these findings suggest that teacher supervision on student rollouts should be assessed rather than applied uniformly\([13](https://arxiv.org/html/2608.19408#bib.bib25)\)\. At the token level, high teacher entropy has motivated selectively combining reverse and forward KL to preserve plausible alternatives, while student entropy and teacher–student divergence have been used to identify informative training positions\([17](https://arxiv.org/html/2608.19408#bib.bib22);[36](https://arxiv.org/html/2608.19408#bib.bib23)\)\. Supervision quality has also been shown to deteriorate at later positions as the discrepancy between student and teacher accumulates along a rollout, motivating position\-sensitive importance weighting\([34](https://arxiv.org/html/2608.19408#bib.bib24)\)\. Existing methods assess OPD signals using token uncertainty, teacher–student discrepancy, sequence position, or trajectory correctness\. Our work instead evaluates teacher supervision at the reasoning\-span level by examining its consistency with estimated reasoning progress\.

## Conclusion

This work identifies a limitation of reasoning\-oriented on\-policy distillation, where agreement with the teacher distribution does not necessarily reflect whether an intermediate step advances the solution\. To this end, we introduce R2\-OPD, which estimates reasoning progress from on\-policy continuations, merges adjacent segments with sign\-consistent progress, and compares the resulting ranking with that induced by teacher–student divergence\. Rather than replacing the OPD objective with an additional reward, R2\-OPD utilizes ranking conflicts as a reliability test and selectively masks supervision that may discourage productive reasoning\. Experiments show that this selective treatment of teacher supervision improves aggregate reasoning performance over standard OPD, with particularly strong gains on the more challenging AIME benchmarks and consistent improvements in pass@4\. Future work will investigate approaches to reduce the cost and variance of process reward estimation and examine whether progress\-aware filtering generalizes to larger models and other diverse domains\.

## References

- Agarwalet al\.\(2024\)R\. Agarwal, N\. Vieillard, Y\. Zhou, P\. Stanczyk, S\. Ramos Garea, M\. Geist, and O\. BachemOn\-policy distillation of language models: learning from self\-generated mistakes\.InInternational Conference on Learning Representations,B\. Kim, Y\. Yue, S\. Chaudhuri, K\. Fragkiadaki, M\. Khan, and Y\. Sun \(Eds\.\),Vol\.2024,pp\. 21246–21263\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/file/5be69a584901a26c521c2b51e40a4c20-Paper-Conference.pdf)Cited by:[On\-Policy Distillation](https://arxiv.org/html/2608.19408#Sx2.SSx1.p1.2),[Evaluation Baselines](https://arxiv.org/html/2608.19408#Sx4.SSx2.p1.1),[Table 1](https://arxiv.org/html/2608.19408#Sx4.T1.1.5.1)\.
- Akhondzadehet al\.\(2026\)M\. S\. Akhondzadeh, V\. Lingam, A\. Tejaswi, C\. Ekbote, S\. Sanghavi, and A\. BojchevskiReward\-gated on\-policy distillation\.arXiv preprint arXiv:2607\.04037\.Cited by:[Process Reward Estimation](https://arxiv.org/html/2608.19408#Sx2.SSx2.p1.1),[Reliability and Selectivity of OPD Signals](https://arxiv.org/html/2608.19408#Sx5.SSx4.SSS0.Px2.p1.1)\.
- Bengioet al\.\(2015\)S\. Bengio, O\. Vinyals, N\. Jaitly, and N\. ShazeerScheduled sampling for sequence prediction with recurrent neural networks\.Advances in neural information processing systems28\.Cited by:[Introduction](https://arxiv.org/html/2608.19408#Sx1.p1.1)\.
- Christianoet al\.\(2017\)P\. F\. Christiano, J\. Leike, T\. Brown, M\. Martic, S\. Legg, and D\. AmodeiDeep reinforcement learning from human preferences\.InAdvances in Neural Information Processing Systems,I\. Guyon, U\. V\. Luxburg, S\. Bengio, H\. Wallach, R\. Fergus, S\. Vishwanathan, and R\. Garnett \(Eds\.\),Vol\.30,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf)Cited by:[Rank\-Based Conflict Detection and Masking](https://arxiv.org/html/2608.19408#Sx3.SSx1.p4.1)\.
- Cuiet al\.\(2025\)G\. Cui, L\. Yuan, Z\. Wang, H\. Wang, Y\. Zhang, J\. Chen, W\. Li, B\. He, Y\. Fan, T\. Yu,et al\.Process reinforcement through implicit rewards\.arXiv preprint arXiv:2502\.01456\.Cited by:[Process Reward Estimation](https://arxiv.org/html/2608.19408#Sx2.SSx2.p2.1),[Process Rewards for Reasoning Language Models](https://arxiv.org/html/2608.19408#Sx5.SSx4.SSS0.Px1.p1.1)\.
- Dontsovet al\.\(2026\)A\. Dontsov, A\. Korznikov, A\. V\. Galichin, and E\. TutubalinaOut of distribution, out of luck: process rewards misguide reasoning models\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 2: Short Papers\),V\. Demberg, K\. Inui, and L\. Marquez \(Eds\.\),Rabat, Morocco,pp\. 421–435\.External Links:[Link](https://aclanthology.org/2026.eacl-short.31/),[Document](https://dx.doi.org/10.18653/v1/2026.eacl-short.31),ISBN 979\-8\-89176\-381\-4Cited by:[Process Rewards for Reasoning Language Models](https://arxiv.org/html/2608.19408#Sx5.SSx4.SSS0.Px1.p1.1)\.
- Guet al\.\(2024\)Y\. Gu, L\. Dong, F\. Wei, and M\. HuangMinillm: knowledge distillation of large language models\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 32694–32717\.Cited by:[On\-Policy Distillation](https://arxiv.org/html/2608.19408#Sx2.SSx1.p1.2)\.
- Guoet al\.\(2025\)D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[Experimental Setup](https://arxiv.org/html/2608.19408#Sx4.SSx1.p1.1)\.
- Heet al\.\(2025\)B\. He, Z\. Qu, Z\. Liu, Y\. Chen, Y\. Zuo, C\. Qian, K\. Zhang, W\. Chen, C\. Xiao, G\. Cui,et al\.Justrl: scaling a 1\.5 b llm with a simple rl recipe\.arXiv preprint arXiv:2512\.16649\.Cited by:[Experimental Setup](https://arxiv.org/html/2608.19408#Sx4.SSx1.p1.1)\.
- Heet al\.\(2024\)C\. He, R\. Luo, Y\. Bai, S\. Hu, Z\. Thai, J\. Shen, J\. Hu, X\. Han, Y\. Huang, Y\. Zhang, J\. Liu, L\. Qi, Z\. Liu, and M\. SunOlympiadBench: a challenging benchmark for promoting AGI with olympiad\-level bilingual multimodal scientific problems\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 3828–3850\.External Links:[Link](https://aclanthology.org/2024.acl-long.211/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.211)Cited by:[Experimental Setup](https://arxiv.org/html/2608.19408#Sx4.SSx1.p2.1)\.
- Heoet al\.\(2026\)B\. Heo, J\. Hwang, S\. Yun, and D\. HanOn\-policy delta distillation\.arXiv preprint arXiv:2607\.15161\.Cited by:[Introduction](https://arxiv.org/html/2608.19408#Sx1.p2.1)\.
- Hintonet al\.\(2015\)G\. Hinton, O\. Vinyals, and J\. DeanDistilling the knowledge in a neural network\.arXiv preprint arXiv:1503\.02531\.Cited by:[Introduction](https://arxiv.org/html/2608.19408#Sx1.p1.1)\.
- Houet al\.\(2026\)W\. Hou, S\. Peng, W\. Wang, Z\. Ruan, Y\. Zhang, Z\. Zhou, M\. Gao, Y\. Chen, K\. Wang, H\. Yang,et al\.Uni\-opd: unifying on\-policy distillation with a dual\-perspective recipe\.arXiv preprint arXiv:2605\.03677\.Cited by:[Process Reward Estimation](https://arxiv.org/html/2608.19408#Sx2.SSx2.p1.1),[Evaluation Baselines](https://arxiv.org/html/2608.19408#Sx4.SSx2.p1.1),[Table 1](https://arxiv.org/html/2608.19408#Sx4.T1.1.9.1),[Reliability and Selectivity of OPD Signals](https://arxiv.org/html/2608.19408#Sx5.SSx4.SSS0.Px2.p1.1)\.
- Jia \(2024\)M\. JiaAIME problem set 2024\.Huggingface\.External Links:[Link](https://huggingface.co/datasets/Maxwell-Jia/AIME_2024)Cited by:[Experimental Setup](https://arxiv.org/html/2608.19408#Sx4.SSx1.p2.1)\.
- Jiaet al\.\(2025\)Z\. Jia, A\. Rakhlin, and T\. XieDo we need to verify step by step? rethinking process supervision from a theoretical perspective\.arXiv preprint arXiv:2502\.10581\.Cited by:[Introduction](https://arxiv.org/html/2608.19408#Sx1.p3.1),[Process Rewards for Reasoning Language Models](https://arxiv.org/html/2608.19408#Sx5.SSx4.SSS0.Px1.p1.1)\.
- Jianget al\.\(2026\)L\. Jiang, H\. Xu, Y\. Ding, and A\. ZhangTrajectory\-refined distillation\.External Links:2606\.08432,[Link](https://arxiv.org/abs/2606.08432)Cited by:[Introduction](https://arxiv.org/html/2608.19408#Sx1.p2.1)\.
- Jinet al\.\(2026\)W\. Jin, T\. Min, Y\. Yang, D\. Wei, Y\. Zhou, S\. R\. Kadhe, N\. Baracaldo, and K\. LeeEntropy\-aware on\-policy distillation of language models\.InForty\-third International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=J5i09faOOf)Cited by:[Evaluation Baselines](https://arxiv.org/html/2608.19408#Sx4.SSx2.p1.1),[Table 1](https://arxiv.org/html/2608.19408#Sx4.T1.1.6.1),[Reliability and Selectivity of OPD Signals](https://arxiv.org/html/2608.19408#Sx5.SSx4.SSS0.Px2.p1.1)\.
- Kim and Rush \(2016\)Y\. Kim and A\. M\. RushSequence\-level knowledge distillation\.InProceedings of the 2016 conference on empirical methods in natural language processing,pp\. 1317–1327\.Cited by:[Introduction](https://arxiv.org/html/2608.19408#Sx1.p1.1)\.
- Liet al\.\(2026\)Y\. Li, Y\. Zuo, B\. He, J\. Zhang, C\. Xiao, C\. Qian, T\. Yu, H\. Gao, W\. Yang, Z\. Liu,et al\.Rethinking on\-policy distillation of large language models: phenomenology, mechanism, and recipe\.arXiv preprint arXiv:2604\.13016\.Cited by:[Experimental Setup](https://arxiv.org/html/2608.19408#Sx4.SSx1.p1.1),[Experimental Setup](https://arxiv.org/html/2608.19408#Sx4.SSx1.p2.1)\.
- Lightmanet al\.\(2024\)H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. CobbeLet’s verify step by step\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 39578–39601\.Cited by:[Introduction](https://arxiv.org/html/2608.19408#Sx1.p3.1),[Process Reward Estimation](https://arxiv.org/html/2608.19408#Sx2.SSx2.p1.1),[Process Rewards for Reasoning Language Models](https://arxiv.org/html/2608.19408#Sx5.SSx4.SSS0.Px1.p1.1)\.
- Luoet al\.\(2024\)L\. Luo, Y\. Liu, R\. Liu, S\. Phatale, H\. Lara, Y\. Li, L\. Shu, Y\. Zhu, L\. Meng, J\. Sun, and A\. RastogiImprove mathematical reasoning in language models by automated process supervision\.ArXivabs/2406\.06592\.External Links:[Link](https://api.semanticscholar.org/CorpusID:270379625)Cited by:[Process Reward Estimation](https://arxiv.org/html/2608.19408#Sx2.SSx2.p2.1),[Process Rewards for Reasoning Language Models](https://arxiv.org/html/2608.19408#Sx5.SSx4.SSS0.Px1.p1.1)\.
- math\-ai \(2025\)math\-aiAIME problem set 2025\.Huggingface\.External Links:[Link](https://huggingface.co/datasets/math-ai/aime25)Cited by:[Experimental Setup](https://arxiv.org/html/2608.19408#Sx4.SSx1.p2.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[Rank\-Based Conflict Detection and Masking](https://arxiv.org/html/2608.19408#Sx3.SSx1.p4.1)\.
- Plyusovet al\.\(2026\)D\. Plyusov, A\. Gorbatovski, A\. Malakhov, N\. Balagansky, B\. Shaposhnikov, D\. Korotyshova, and D\. GavrilovTrust\-region behavior blending for on\-policy distillation\.arXiv preprint arXiv:2605\.31159\.Cited by:[Introduction](https://arxiv.org/html/2608.19408#Sx1.p2.1)\.
- Quet al\.\(2025\)Y\. Qu, M\. Y\. Yang, A\. Setlur, L\. Tunstall, E\. E\. Beeching, R\. Salakhutdinov, and A\. KumarOptimizing test\-time compute via meta reinforcement fine\-tuning\.arXiv preprint arXiv:2503\.07572\.Cited by:[Process Reward Estimation](https://arxiv.org/html/2608.19408#Sx2.SSx2.p2.1)\.
- Setluret al\.\(2025a\)A\. Setlur, C\. Nagpal, A\. Fisch, X\. Geng, J\. Eisenstein, R\. Agarwal, A\. Agarwal, J\. Berant, and A\. KumarRewarding progress: scaling automated process verifiers for llm reasoning\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 60808–60838\.Cited by:[Process Reward Estimation](https://arxiv.org/html/2608.19408#Sx2.SSx2.p2.1),[Process Rewards for Reasoning Language Models](https://arxiv.org/html/2608.19408#Sx5.SSx4.SSS0.Px1.p1.1)\.
- Setluret al\.\(2025b\)A\. Setlur, M\. Y\. Yang, C\. Snell, J\. Greer, I\. Wu, V\. Smith, M\. Simchowitz, and A\. KumarE3: learning to explore enables extrapolation of test\-time compute for llms\.arXiv preprint arXiv:2506\.09026\.Cited by:[Experimental Setup](https://arxiv.org/html/2608.19408#Sx4.SSx1.p1.1)\.
- Songet al\.\(2025\)M\. Song, Z\. Su, X\. Qu, J\. Zhou, and Y\. ChengPRMBench: a fine\-grained and challenging benchmark for process\-level reward models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 25299–25346\.External Links:[Link](https://aclanthology.org/2025.acl-long.1230/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1230),ISBN 979\-8\-89176\-251\-0Cited by:[Process Rewards for Reasoning Language Models](https://arxiv.org/html/2608.19408#Sx5.SSx4.SSS0.Px1.p1.1)\.
- Song and Zheng \(2026\)M\. Song and M\. ZhengA survey of on\-policy distillation for large language models\.arXiv preprint arXiv:2604\.00626\.Cited by:[Introduction](https://arxiv.org/html/2608.19408#Sx1.p1.1)\.
- Uesatoet al\.\(2022\)J\. Uesato, N\. Kushman, R\. Kumar, F\. Song, N\. Siegel, L\. Wang, A\. Creswell, G\. Irving, and I\. HigginsSolving math word problems with process\- and outcome\-based feedback\.ArXivabs/2211\.14275\.External Links:[Link](https://api.semanticscholar.org/CorpusID:254017497)Cited by:[Introduction](https://arxiv.org/html/2608.19408#Sx1.p3.1),[Process Reward Estimation](https://arxiv.org/html/2608.19408#Sx2.SSx2.p1.1),[Process Rewards for Reasoning Language Models](https://arxiv.org/html/2608.19408#Sx5.SSx4.SSS0.Px1.p1.1)\.
- Wanget al\.\(2024\)P\. Wang, L\. Li, Z\. Shao, R\. Xu, D\. Dai, Y\. Li, D\. Chen, Y\. Wu, and Z\. SuiMath\-shepherd: verify and reinforce llms step\-by\-step without human annotations\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 9426–9439\.Cited by:[Process Reward Estimation](https://arxiv.org/html/2608.19408#Sx2.SSx2.p2.1),[Process Rewards for Reasoning Language Models](https://arxiv.org/html/2608.19408#Sx5.SSx4.SSS0.Px1.p1.1)\.
- Wanget al\.\(2026\)R\. Wang, H\. Wang, Y\. Chen, B\. Xue, T\. Fang, W\. Yu, and K\. WongDemystifying on\-policy distillation: roles, pathologies, and regulations\.arXiv preprint arXiv:2607\.13399\.Cited by:[Reliability and Selectivity of OPD Signals](https://arxiv.org/html/2608.19408#Sx5.SSx4.SSS0.Px2.p1.1)\.
- Xieet al\.\(2026a\)W\. Xie, H\. Zhao, W\. Liu, Y\. Zhu, L\. Chen, M\. Ye, Z\. Chen, Y\. Xu, S\. Dong, Z\. Wang,et al\.Step\-wise rubric rewards for llm reasoning\.arXiv preprint arXiv:2605\.17291\.Cited by:[Process Rewards for Reasoning Language Models](https://arxiv.org/html/2608.19408#Sx5.SSx4.SSS0.Px1.p1.1)\.
- Xieet al\.\(2026b\)Y\. Xie, S\. Zhu, T\. Wen, B\. Chen, and Y\. WangOn the position bias of on\-policy distillation\.arXiv preprint arXiv:2606\.22600\.Cited by:[Evaluation Baselines](https://arxiv.org/html/2608.19408#Sx4.SSx2.p1.1),[Table 1](https://arxiv.org/html/2608.19408#Sx4.T1.1.8.1),[Reliability and Selectivity of OPD Signals](https://arxiv.org/html/2608.19408#Sx5.SSx4.SSS0.Px2.p1.1)\.
- Xuet al\.\(2026a\)A\. Xu, B\. Lin, B\. Xue, B\. Wang, B\. Xu, B\. Wu, B\. Zhang, C\. Lin, C\. Dong, C\. Ling,et al\.Deepseek\-v4: towards highly efficient million\-token context intelligence\.arXiv preprint arXiv:2606\.19348\.Cited by:[Introduction](https://arxiv.org/html/2608.19408#Sx1.p1.1)\.
- Xuet al\.\(2026b\)Y\. Xu, H\. Sang, Z\. Zhou, R\. He, Z\. Wang, and A\. GeramifardTip: token importance in on\-policy distillation\.arXiv preprint arXiv:2604\.14084\.Cited by:[Evaluation Baselines](https://arxiv.org/html/2608.19408#Sx4.SSx2.p1.1),[Table 1](https://arxiv.org/html/2608.19408#Sx4.T1.1.7.1),[Reliability and Selectivity of OPD Signals](https://arxiv.org/html/2608.19408#Sx5.SSx4.SSS0.Px2.p1.1)\.
- Yanget al\.\(2025a\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[Experimental Setup](https://arxiv.org/html/2608.19408#Sx4.SSx1.p1.1)\.
- Yanget al\.\(2025b\)Z\. Yang, C\. He, X\. Shi, S\. Deng, L\. Li, Q\. Yin, and D\. JiangBeyond the first error: process reward models for reflective mathematical reasoning\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 4711–4728\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.253/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.253),ISBN 979\-8\-89176\-335\-7Cited by:[Process Rewards for Reasoning Language Models](https://arxiv.org/html/2608.19408#Sx5.SSx4.SSS0.Px1.p1.1)\.
- Yuanet al\.\(2026\)H\. Yuan, Z\. Xu, H\. Wang, X\. Yi, J\. Gao, X\. Zhang, Y\. Wang, C\. Yu, and Y\. WuVerifiable process rewards for agentic reasoning\.arXiv preprint arXiv:2605\.10325\.Cited by:[Process Rewards for Reasoning Language Models](https://arxiv.org/html/2608.19408#Sx5.SSx4.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2024\)D\. Zhang, S\. Zhoubian, Z\. Hu, Y\. Yue, Y\. Dong, and J\. TangReST\-mcts\*: llm self\-training via process reward guided tree search\.arXiv preprint arXiv:2406\.03816\.Cited by:[Process Reward Estimation](https://arxiv.org/html/2608.19408#Sx2.SSx2.p2.1),[Process Rewards for Reasoning Language Models](https://arxiv.org/html/2608.19408#Sx5.SSx4.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2025\)Z\. Zhang, C\. Zheng, Y\. Wu, B\. Zhang, R\. Lin, B\. Yu, D\. Liu, J\. Zhou, and J\. LinThe lessons of developing process reward models in mathematical reasoning\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 10495–10516\.Cited by:[Process Rewards for Reasoning Language Models](https://arxiv.org/html/2608.19408#Sx5.SSx4.SSS0.Px1.p1.1)\.
- Zhenget al\.\(2026\)B\. Zheng, X\. Ma, Y\. Liang, J\. Ruan, X\. Fu, K\. Lin, B\. Zhu, K\. Zeng, and X\. CaiScope: signal\-calibrated on\-policy distillation enhancement with dual\-path adaptive weighting\.arXiv preprint arXiv:2604\.10688\.Cited by:[Process Reward Estimation](https://arxiv.org/html/2608.19408#Sx2.SSx2.p1.1)\.

## Appendix AProofs

### A\.1Proof of Proposition[2](https://arxiv.org/html/2608.19408#Thmproposition2)

###### Proof\.

Fix a merged segmentjjand condition on its contiguous token\-index setℐj=\{u,u\+1,…,u\+Lj−1\}\\mathcal\{I\}\_\{j\}=\\\{u,u\+1,\\ldots,u\+L\_\{j\}\-1\\\}\. For notational simplicity, writeL=LjL=L\_\{j\}\. By Eq\. \([9](https://arxiv.org/html/2608.19408#Sx3.E9)\) and the decompositionℓtKL,𝒮=μj\+ϵt\\ell\_\{t\}^\{\\mathrm\{KL\},\\mathcal\{S\}\}=\\mu\_\{j\}\+\\epsilon\_\{t\},

κj\\displaystyle\\kappa\_\{j\}=1L​∑t∈ℐjℓtKL,𝒮\\displaystyle=\\frac\{1\}\{L\}\\sum\_\{t\\in\\mathcal\{I\}\_\{j\}\}\\ell\_\{t\}^\{\\mathrm\{KL\},\\mathcal\{S\}\}=1L​∑t∈ℐj\(μj\+ϵt\)\\displaystyle=\\frac\{1\}\{L\}\\sum\_\{t\\in\\mathcal\{I\}\_\{j\}\}\(\\mu\_\{j\}\+\\epsilon\_\{t\}\)=μj\+1L​∑t∈ℐjϵt\.\\displaystyle=\\mu\_\{j\}\+\\frac\{1\}\{L\}\\sum\_\{t\\in\\mathcal\{I\}\_\{j\}\}\\epsilon\_\{t\}\.Taking expectations and using𝔼⁡\[ϵt\]=0\\mathbb\{E\}\[\\epsilon\_\{t\}\]=0for everyt∈ℐjt\\in\\mathcal\{I\}\_\{j\}gives

𝔼⁡\[κj\]=μj\+1L​∑t∈ℐj𝔼⁡\[ϵt\]=μj\.\\mathbb\{E\}\[\\kappa\_\{j\}\]=\\mu\_\{j\}\+\\frac\{1\}\{L\}\\sum\_\{t\\in\\mathcal\{I\}\_\{j\}\}\\mathbb\{E\}\[\\epsilon\_\{t\}\]=\\mu\_\{j\}\.Therefore,κj\\kappa\_\{j\}is an unbiased estimator of the locally constant segment\-average distillation lossμj\\mu\_\{j\}\.

It remains to bound the variance\. Adding the constantμj\\mu\_\{j\}does not change variance, so

Var⁡\(κj\)\\displaystyle\\operatorname\{Var\}\(\\kappa\_\{j\}\)=Var⁡\(1L​∑t∈ℐjϵt\)\\displaystyle=\\operatorname\{Var\}\\\!\\left\(\\frac\{1\}\{L\}\\sum\_\{t\\in\\mathcal\{I\}\_\{j\}\}\\epsilon\_\{t\}\\right\)=1L2​∑t∈ℐj∑s∈ℐjCov⁡\(ϵt,ϵs\)\.\\displaystyle=\\frac\{1\}\{L^\{2\}\}\\sum\_\{t\\in\\mathcal\{I\}\_\{j\}\}\\sum\_\{s\\in\\mathcal\{I\}\_\{j\}\}\\operatorname\{Cov\}\(\\epsilon\_\{t\},\\epsilon\_\{s\}\)\.The covariance sum containsLLdiagonal terms witht=st=s\. The assumed covariance bound implies

Var⁡\(ϵt\)=Cov⁡\(ϵt,ϵt\)≤σ2\.\\operatorname\{Var\}\(\\epsilon\_\{t\}\)=\\operatorname\{Cov\}\(\\epsilon\_\{t\},\\epsilon\_\{t\}\)\\leq\\sigma^\{2\}\.For every distanced∈\{1,…,L−1\}d\\in\\\{1,\\ldots,L\-1\\\}, contiguity ofℐj\\mathcal\{I\}\_\{j\}implies that exactlyL−dL\-dunordered pairs of token positions satisfy\|t−s\|=d\|t\-s\|=d\. Each unordered pair occurs twice in the double sum, once as\(t,s\)\(t,s\)and once as\(s,t\)\(s,t\)\. Since a covariance may be negative, we upper\-bound each off\-diagonal term by its absolute value\. Hence,

Var⁡\(κj\)\\displaystyle\\operatorname\{Var\}\(\\kappa\_\{j\}\)≤1L2​\[L​σ2\+2​σ2​∑d=1L−1\(L−d\)​ρd\]\\displaystyle\\leq\\frac\{1\}\{L^\{2\}\}\\left\[L\\sigma^\{2\}\+2\\sigma^\{2\}\\sum\_\{d=1\}^\{L\-1\}\(L\-d\)\\rho^\{d\}\\right\]=σ2L2​\[L\+2​∑d=1L−1\(L−d\)​ρd\]\.\\displaystyle=\\frac\{\\sigma^\{2\}\}\{L^\{2\}\}\\left\[L\+2\\sum\_\{d=1\}^\{L\-1\}\(L\-d\)\\rho^\{d\}\\right\]\.UsingL−d≤LL\-d\\leq L, extending the finite nonnegative sum to an infinite one, and applying the geometric\-series identity∑d=1∞ρd=ρ/\(1−ρ\)\\sum\_\{d=1\}^\{\\infty\}\\rho^\{d\}=\\rho/\(1\-\\rho\), which is valid because0≤ρ<10\\leq\\rho<1, we obtain

Var⁡\(κj\)\\displaystyle\\operatorname\{Var\}\(\\kappa\_\{j\}\)≤σ2L2​\[L\+2​L​∑d=1∞ρd\]\\displaystyle\\leq\\frac\{\\sigma^\{2\}\}\{L^\{2\}\}\\left\[L\+2L\\sum\_\{d=1\}^\{\\infty\}\\rho^\{d\}\\right\]=σ2L​\(1\+2​ρ1−ρ\)\\displaystyle=\\frac\{\\sigma^\{2\}\}\{L\}\\left\(1\+\\frac\{2\\rho\}\{1\-\\rho\}\\right\)=σ2L​1\+ρ1−ρ\.\\displaystyle=\\frac\{\\sigma^\{2\}\}\{L\}\\frac\{1\+\\rho\}\{1\-\\rho\}\.RestoringL=LjL=L\_\{j\}proves Eq\. \([10](https://arxiv.org/html/2608.19408#Sx3.E10)\)\. Ifσ2\\sigma^\{2\}andρ\\rhoare bounded independently ofLjL\_\{j\}, the multiplicative factorσ2​\(1\+ρ\)/\(1−ρ\)\\sigma^\{2\}\(1\+\\rho\)/\(1\-\\rho\)is constant with respect to segment length\. Consequently, the variance upper bound scales asO⁡\(1/Lj\)O\(1/L\_\{j\}\)\. ∎

## Appendix BTraining Details

This section provides the optimization and sequence settings, training data, on\-policy response generation procedure, model\-specific reasoning templates, process\-reward rollout construction, eligibility and fallback rules, and complete pseudocode of R2\-OPD\. Model configurations are described in the main paper and are therefore not repeated here\.

### B\.1Optimization and Sequence Settings

Except for its progress\-aware masking stage, R2\-OPD follows the standard OPD training loop: the student generates responses on\-policy, the teacher supplies token\-level distributional supervision on the resulting student contexts, and the student is updated using the reverse\-KL objective\. Table[4](https://arxiv.org/html/2608.19408#A2.T4)separates the settings of the OPD update from those used only for process\-reward estimation and segment filtering\.

Table 4:Optimization, process\-reward estimation, and filtering settings used in all R2\-OPD experiments\.
### B\.2Training Data

All models are trained for one epoch on the same deduplicated DAPO\-Math\-17K corpus used in the main experiments\. All reported configurations use the same prompt set, so their differences do not arise from configuration\-specific data selection\. Each training example consists of a mathematical promptxix\_\{i\}and a verifier\-compatible ground\-truth answergig\_\{i\}\. Responses are generated by the current student during training rather than precomputed or replaced by teacher\-generated solutions\.

### B\.3On\-Policy Response Generation

For each promptxix\_\{i\}, the current student samples an on\-policy responseyiy\_\{i\}with a maximum length of 7,168 tokens\. This response is the training trajectory on which the OPD loss is defined\. The teacher is queried on the same token contexts to compute the support\-restricted reverse\-KL loss\. At each position, the approximate support contains the 16 tokens assigned the highest probability by the student, as specified byH=16H=16\.

The token mask is initialized to one over the entire response\. R2\-OPD subsequently changes this mask only when the response satisfies all process\-reward and conflict\-detection requirements described below\. The short answer\-eliciting rollouts used for boundary evaluation are separate generations: they estimate intermediate solve probabilities, but their tokens are not included in the KL objective and are not used as replacement training trajectories\.

### B\.4Model\-Specific Reasoning Templates

We use the chat template shipped with each model checkpoint rather than imposing a single wrapper across model families\. The student and teacher in each configuration share the same family\-specific format: DeepSeek\-R1\-Distill\-Qwen\-1\.5B and JustRL\-DeepSeek\-1\.5B use the DeepSeek reasoning template, whereas Qwen3\-1\.7B and e3\-1\.7B use the Qwen3 thinking\-mode template\. This distinction is important because the former is based on Qwen2\.5 but does not use the standard Qwen2\.5 ChatML wrapper\.

Across the two configurations, the benchmark problem is inserted into the following common task\-level instruction:

Solvethefollowingmathproblemstepbystep\.

Putyouranswerinside\\boxed\{\}\.

\{problem\}

Remembertoputyouranswerinside\\boxed\{\}\.

Only the model\-specific special\-token wrapper differs\. The schematics below give the single\-turn path used in our experiments and preserve the ordering and line breaks of the corresponding tokenizer templates\. For the DeepSeek template, we use ASCII aliases for its Unicode special tokens to ensure compatibility with the paper’sLaTeXengine:<DS\-BOS\>,<DS\-User\>,<DS\-Assistant\>, and<DS\-EOS\>denote the checkpoint’s beginning\-of\-sequence, user, assistant, and end\-of\-sequence tokens, respectively\. These aliases are used only for typesetting; the implementation uses the literal tokens from the tokenizer file\.

##### Qwen2\.5\-based DeepSeek group\.

For both the DeepSeek\-R1\-Distill\-Qwen\-1\.5B student and the JustRL\-DeepSeek\-1\.5B teacher, applying the checkpoint template withadd\_generation\_prompt=Trueproduces a prefix ending in the assistant marker followed immediately by an opening<think\>tag:

<DS\-BOS\><DS\-User\>\{taskprompt\}<DS\-Assistant\><think\>

\{reasoning\}

</think\>

\{finalanswer\}<DS\-EOS\>

Thus, the opening reasoning tag is part of the formatted generation prefix\. The generated continuation supplies the reasoning content, closes the tag, emits the final answer, and terminates with the checkpoint’s end\-of\-sequence token\. We retain this DeepSeek formatting throughout response generation and teacher scoring; we do not replace it with Qwen2\.5’s<\|im\_start\|\>and<\|im\_end\|\>markers\.

##### Qwen3 group\.

For the Qwen3\-1\.7B student and e3\-1\.7B teacher, we apply the Qwen3 template in thinking mode, i\.e\., withenable\_thinking=True\. The corresponding single\-turn sequence is:

<\|im\_start\|\>user

\{taskprompt\}<\|im\_end\|\>

<\|im\_start\|\>assistant

<think\>

\{reasoning\}

</think\>

\{finalanswer\}<\|im\_end\|\>

Unlike the DeepSeek template, the Qwen3 generation prefix ends after the assistant\-role newline; in thinking mode, the model itself generates the<think\>block before the final response\. Settingenable\_thinking=Falsewould instead insert an empty<think\></think\>block into the prompt, but this non\-thinking path is not used in our reasoning experiments\.

These templates format the original on\-policy responses used by the OPD objective\. They are distinct from the additional answer\-eliciting instructionℐans\\mathcal\{I\}\_\{\\mathrm\{ans\}\}used only to construct boundary\-evaluation rollouts in Appendix[B\.5](https://arxiv.org/html/2608.19408#A2.SS5.SSSx2)\.

### B\.5Process\-Reward Rollout Construction

For an eligible response, process rewards are estimated by evaluating whether the student can still reach the ground\-truth answer from selected intermediate reasoning states\. We call the generations used for this evaluation*answer\-eliciting rollouts*; after this definition, we refer to them simply as rollouts\. Each rollout is sampled from the current student after conditioning on a standardized reasoning prefix and a fixed answer\-eliciting instruction\. The construction of the evaluated boundaries and standardized prompts is detailed next\.

#### Segmentation Lexicon and Boundary Matching

Table[6](https://arxiv.org/html/2608.19408#A2.T6)lists the complete discourse\-marker lexicon used by the segmentation procedure\. The categories are descriptive; matching is performed only against the literal expressions shown in the table\.

Before matching, the decoded response is treated as plain text\. Matching is case\-insensitive, uses word boundaries, and permits one or more whitespace characters between the words of a multiword marker\. Matches are processed from left to right\. A candidate cut is placed immediately before the matched marker, so the marker begins the next segment\. The cut is retained only when the text since the previous accepted boundary contains at leastSminS\_\{\\min\}sentences, where the implementation approximates the sentence count by the number of periods\. If no cut is accepted, the complete response is treated as a single segment\.

#### Answer\-Eliciting Instruction and Prefix Standardization

The symbolℐans\\mathcal\{I\}\_\{\\mathrm\{ans\}\}denotes a fixed textual suffix, rather than a learned component\. It instructs the student to stop extending the reasoning chain and emit a verifier\-compatible final answer based on the reasoning already contained in the prefix\. Its exact wording is:

> Based on the reasoning above, directly give the final answer\. Put the final answer within \\boxed\{\}\. If no definitive answer can be derived from the existing reasoning, output \\boxed\{no answer\}\.

A segment boundary commonly falls inside an open<think\>block\. If the truncated response prefix contains more occurrences of<think\>than</think\>, we remove trailing newline characters and append</think\>followed by a blank line beforeℐans\\mathcal\{I\}\_\{\\mathrm\{ans\}\}\. Otherwise, the prefix is left unchanged\. Closing the unmatched tag supplies the mode\-transition cue needed for the answer\-eliciting rollout to follow the answer instruction instead of treating it as additional reasoning\. This preprocessing is applied only to the rollout prompt; it does not alter the original student trajectory used for training\.

#### Boundary Rollouts and Solve\-Probability Estimation

Letpm=\(x,y1:bm\)p\_\{m\}=\(x,y\_\{1:b\_\{m\}\}\)denote the state at the end of segmentmm\. For every nonterminal boundary of a PR\-available response, we drawNeval=8N\_\{\\mathrm\{eval\}\}=8rollouts from the student conditioned on the standardized prefixp¯m\\bar\{p\}\_\{m\}andℐans\\mathcal\{I\}\_\{\\mathrm\{ans\}\}\. Rollouts use temperature 0\.7, top\-k=50k=50, top\-p=1\.0p=1\.0, and a maximum generation length of 300 tokens\. The task\-specific verifier then evaluates whether the completed sequence reaches the ground\-truth answergig\_\{i\}\.

The intermediate solve probabilityS^m\\hat\{S\}\_\{m\}is the empirical fraction of successful rollouts\. Following Eq\. \([5](https://arxiv.org/html/2608.19408#Sx2.E5)\), we setS^0=0\\hat\{S\}\_\{0\}=0and reuse the verifier result of the original on\-policy response for the terminal stateS^M\\hat\{S\}\_\{M\}\. Consequently, additional rollouts are required only for theM−1M\-1nonterminal boundaries\. The process reward of segmentmmis the finite differenceP​Rm=S^m−S^m−1PR\_\{m\}=\\hat\{S\}\_\{m\}\-\\hat\{S\}\_\{m\-1\}\.

### B\.6Eligibility and Fallback Rules

The implementation uses conservative fallback behavior: whenever a response does not provide enough information to define a reliable ranking conflict, its original OPD supervision is retained\. Table[5](https://arxiv.org/html/2608.19408#A2.T5)lists the complete decision logic\.

Table 5:Eligibility checks and conservative fallback actions used by R2\-OPD\.Table 6:Complete lexicon of discourse markers used to propose reasoning\-segment boundaries\.
### B\.7Sign\-Consistent Filtering and Mask Construction

The segment\-level finite differences are first merged into maximal runs with the same sign, with zero\-valued rewards absorbed into the current run\. For each merged segmentjj, we sum its process rewards to obtainP​R~i,j\\widetilde\{PR\}\_\{i,j\}and average its token\-level KL losses to obtainκi,j\\kappa\_\{i,j\}\. Candidate segments exclude the first merged segment, which is retained to avoid removing the initial problem interpretation and setup\.

The remaining segments are ordered by decreasingP​R~i,j\\widetilde\{PR\}\_\{i,j\}, with increasingκi,j\\kappa\_\{i,j\}used to break equal\-reward ties\. Only adjacent pairs with a strict process\-reward difference are compared\. A positive inconsistency contribution occurs when the segment with greater estimated progress also has greater KL loss, indicating that teacher–student disagreement is larger on the segment that contributes more to solving the problem\. Contributions incident to each segment are accumulated intoInci,j\\mathrm\{Inc\}\_\{i,j\}\.

For responseii, the masking budget isbi=⌈\(q/100\)​ni⌉b\_\{i\}=\\lceil\(q/100\)n\_\{i\}\\rceil\. Among segmentsj≥2j\\geq 2withInci,j\>0\\mathrm\{Inc\}\_\{i,j\}\>0, we mask at most thebib\_\{i\}highest\-scoring segments\. If fewer thanbib\_\{i\}candidates have positive scores, all positive\-score candidates are masked and the unused budget is not reassigned\. All unselected segments retain their original supervision\. The final loss is normalized by the number of unmasked tokens, as in Eq\. \([13](https://arxiv.org/html/2608.19408#Sx3.E13)\), so responses with different retained lengths remain comparable within the batch\.

### B\.8Training Procedure

Algorithm[1](https://arxiv.org/html/2608.19408#alg1)summarizes the complete batch\-level procedure\. The student first generates on\-policy responses and receives token\-level reverse\-KL supervision from the teacher\. Eligible responses are then evaluated using answer\-eliciting rollouts, merged into sign\-consistent progress units, checked for conflicts with segment\-average KL losses, and selectively masked before the normalized OPD update\.

Algorithm 1Training procedure of R2\-OPD for a single batch\.1:Batch

ℬ=\{\(xi,gi\)\}\\mathcal\{B\}=\\\{\(x\_\{i\},g\_\{i\}\)\\\}, student

πS\\pi\_\{S\}, teacher

πT\\pi\_\{T\}
2:

H,Neval,q,Smin,nmin,MmaxH,N\_\{\\mathrm\{eval\}\},q,S\_\{\\min\},n\_\{\\min\},M\_\{\\max\}
3:Updated student

πS\\pi\_\{S\}
4:foreach

\(xi,gi\)∈ℬ\(x\_\{i\},g\_\{i\}\)\\in\\mathcal\{B\}do

5:Sample

yi∼πS\(⋅∣xi\)y\_\{i\}\\sim\\pi\_\{S\}\(\\cdot\\mid x\_\{i\}\)and set

Mt\(i\)←1M\_\{t\}^\{\(i\)\}\\leftarrow 1for all

tt
6:Construct each student top\-

HHsupport

𝒮t\(i\)\\mathcal\{S\}\_\{t\}^\{\(i\)\}and compute

ℓtKL,𝒮,\(i\)\\ell\_\{t\}^\{\\mathrm\{KL\},\\mathcal\{S\},\(i\)\}⊳\\trianglerightEq\. \([3](https://arxiv.org/html/2608.19408#Sx2.E3)\)

7:

Σi←Segment​\(yi,Smin,Mmax\)\\Sigma\_\{i\}\\leftarrow\\textsc\{Segment\}\(y\_\{i\};S\_\{\\min\},M\_\{\\max\}\)
8:if

gi∉yig\_\{i\}\\notin y\_\{i\}or

\|Σi\|<2\|\\Sigma\_\{i\}\|<2then

9:continue⊳\\trianglerightPR\-unavailable; retain all supervision

10:endif

11:

Mi←\|Σi\|M\_\{i\}\\leftarrow\|\\Sigma\_\{i\}\|,

S^i,0←0\\hat\{S\}\_\{i,0\}\\leftarrow 0, and

S^i,Mi←ℛ⁡\(yi,gi\)\\hat\{S\}\_\{i,M\_\{i\}\}\\leftarrow\\mathcal\{R\}\(y\_\{i\},g\_\{i\}\)
12:for

m=1m=1to

Mi−1M\_\{i\}\-1do

13:Sample

\{ci,m\(ℓ\)\}ℓ=1Neval∼πS\(⋅∣p¯i,m,ℐans\)\\\{c\_\{i,m\}^\{\(\\ell\)\}\\\}\_\{\\ell=1\}^\{N\_\{\\mathrm\{eval\}\}\}\\sim\\pi\_\{S\}\(\\cdot\\mid\\bar\{p\}\_\{i,m\},\\mathcal\{I\}\_\{\\mathrm\{ans\}\}\)
14:

S^i,m←1Neval​∑ℓ=1Nevalℛ⁡\(pi,m∥ci,m\(ℓ\),gi\)\\hat\{S\}\_\{i,m\}\\leftarrow\\frac\{1\}\{N\_\{\\mathrm\{eval\}\}\}\\sum\_\{\\ell=1\}^\{N\_\{\\mathrm\{eval\}\}\}\\mathcal\{R\}\(p\_\{i,m\}\\mathbin\{\\\|\}c\_\{i,m\}^\{\(\\ell\)\},g\_\{i\}\)
15:endfor

16:

P​Ri,m←S^i,m−S^i,m−1PR\_\{i,m\}\\leftarrow\\hat\{S\}\_\{i,m\}\-\\hat\{S\}\_\{i,m\-1\}for

m=1,…,Mim=1,\\ldots,M\_\{i\}
17:

\(\{𝒥i,j\},\{ℐi,j\}\)j=1ni←MergeSameSignRuns​\(Σi,P​Ri\)\(\\\{\\mathcal\{J\}\_\{i,j\}\\\},\\\{\\mathcal\{I\}\_\{i,j\}\\\}\)\_\{j=1\}^\{n\_\{i\}\}\\leftarrow\\textsc\{MergeSameSignRuns\}\(\\Sigma\_\{i\},PR\_\{i\}\)⊳\\trianglerightZeros are absorbed

18:for

j=1j=1to

nin\_\{i\}do

19:

P​R~i,j←∑m∈𝒥i,jP​Ri,m\\widetilde\{PR\}\_\{i,j\}\\leftarrow\\sum\_\{m\\in\\mathcal\{J\}\_\{i,j\}\}PR\_\{i,m\}⊳\\trianglerightEq\. \([7](https://arxiv.org/html/2608.19408#Sx3.E7)\)

20:

κi,j←\|ℐi,j\|−1​∑t∈ℐi,jℓtKL,𝒮,\(i\)\\kappa\_\{i,j\}\\leftarrow\|\\mathcal\{I\}\_\{i,j\}\|^\{\-1\}\\sum\_\{t\\in\\mathcal\{I\}\_\{i,j\}\}\\ell\_\{t\}^\{\\mathrm\{KL\},\\mathcal\{S\},\(i\)\}⊳\\trianglerightEq\. \([9](https://arxiv.org/html/2608.19408#Sx3.E9)\)

21:endfor

22:if

ni<max⁡\(3,nmin\)n\_\{i\}<\\max\(3,n\_\{\\min\}\)then

23:continue⊳\\trianglerightResponse is ineligible for masking

24:endif

25:

ρi←Sort​\(\{2,…,ni\}\)\\rho\_\{i\}\\leftarrow\\textsc\{Sort\}\(\\\{2,\\ldots,n\_\{i\}\\\}\)by decreasing

P​R~i,j\\widetilde\{PR\}\_\{i,j\}, then increasing

κi,j\\kappa\_\{i,j\}
26:

𝒫i←\{\(ρi,r,ρi,r\+1\):P​R~i,ρi,r\>P​R~i,ρi,r\+1\}\\mathcal\{P\}\_\{i\}\\leftarrow\\\{\(\\rho\_\{i,r\},\\rho\_\{i,r\+1\}\):\\widetilde\{PR\}\_\{i,\\rho\_\{i,r\}\}\>\\widetilde\{PR\}\_\{i,\\rho\_\{i,r\+1\}\}\\\}
27:if

\|𝒫i\|=0\|\\mathcal\{P\}\_\{i\}\|=0then

28:continue

29:endif

30:

Inci,j←0\\mathrm\{Inc\}\_\{i,j\}\\leftarrow 0for

j=1,…,nij=1,\\ldots,n\_\{i\}
31:foreach

\(a,b\)∈𝒫i\(a,b\)\\in\\mathcal\{P\}\_\{i\}do

32:

vi​\(a,b\)←\[\(P​R~i,a−P​R~i,b\)​\(κi,a−κi,b\)\]\+v\_\{i\}\(a,b\)\\leftarrow\[\(\\widetilde\{PR\}\_\{i,a\}\-\\widetilde\{PR\}\_\{i,b\}\)\(\\kappa\_\{i,a\}\-\\kappa\_\{i,b\}\)\]\_\{\+\}
33:

Inci,a←Inci,a\+vi​\(a,b\)\\mathrm\{Inc\}\_\{i,a\}\\leftarrow\\mathrm\{Inc\}\_\{i,a\}\+v\_\{i\}\(a,b\);

Inci,b←Inci,b\+vi​\(a,b\)\\mathrm\{Inc\}\_\{i,b\}\\leftarrow\\mathrm\{Inc\}\_\{i,b\}\+v\_\{i\}\(a,b\)
34:endfor

35:

bi←⌈\(q/100\)​ni⌉b\_\{i\}\\leftarrow\\lceil\(q/100\)n\_\{i\}\\rceil
36:

ℳi←\\mathcal\{M\}\_\{i\}\\leftarrowtop\-

bib\_\{i\}indices in

\{j≥2:Inci,j\>0\}\\\{j\\geq 2:\\mathrm\{Inc\}\_\{i,j\}\>0\\\}by decreasing

Inci,j\\mathrm\{Inc\}\_\{i,j\}
37:Set

Mt\(i\)←0M\_\{t\}^\{\(i\)\}\\leftarrow 0iff

t∈ℐi,jt\\in\\mathcal\{I\}\_\{i,j\}for some

j∈ℳij\\in\\mathcal\{M\}\_\{i\}
38:endfor

39:

Zi←∑t=1TiMt\(i\)Z\_\{i\}\\leftarrow\\sum\_\{t=1\}^\{T\_\{i\}\}M\_\{t\}^\{\(i\)\}for each

ii
40:

ℒ^R2​\-​OPD←1\|ℬ\|​∑i1Zi​∑t=1TiMt\(i\)​ℓtKL,𝒮,\(i\)\\widehat\{\\mathcal\{L\}\}\_\{\\mathrm\{R^\{2\}\\text\{\-\}OPD\}\}\\leftarrow\\frac\{1\}\{\|\\mathcal\{B\}\|\}\\sum\_\{i\}\\frac\{1\}\{Z\_\{i\}\}\\sum\_\{t=1\}^\{T\_\{i\}\}M\_\{t\}^\{\(i\)\}\\ell\_\{t\}^\{\\mathrm\{KL\},\\mathcal\{S\},\(i\)\}
41:Update

πS\\pi\_\{S\}by minimizing

ℒ^R2​\-​OPD\\widehat\{\\mathcal\{L\}\}\_\{\\mathrm\{R^\{2\}\\text\{\-\}OPD\}\}⊳\\trianglerightEq\. \([13](https://arxiv.org/html/2608.19408#Sx3.E13)\)

All mask entries are initialized to one\. Therefore, every earlycontinuein Algorithm[1](https://arxiv.org/html/2608.19408#alg1)implements the conservative fallback described in Table[5](https://arxiv.org/html/2608.19408#A2.T5)\.

## Appendix CQualitative Case Studies

We provide two AIME 2024 examples in which R2\-OPD produces the verified answer\. This section uses the full width of the two\-column page so that the complete response boxes remain readable\. Unlike a two\-column float, each box is part of the normal document flow and can therefore break automatically across pages while respecting the page margins\. Each box reproduces the complete evaluator\-facing response recorded for the selected sample; blank lines are suppressed uniformly for compact typesetting, and the two non\-ASCII mathematical symbols are transliterated asomegaandsqrtto ensure robustLaTeXcompilation\. Responses that reach the generation limit are reproduced through their recorded endpoint and marked as truncated\.

##### Model setting\.

Both cases use the Qwen2\.5\-based configuration from the main experiments\. The base\-model responses are generated by the original DeepSeek\-R1\-Distill\-Qwen\-1\.5B checkpoint\. For standard OPD and R2\-OPD, the student is initialized from this same checkpoint and distilled from the same JustRL\-DeepSeek\-1\.5B teacher on DAPO\-Math\-17K\. The two distilled students therefore share the model initialization, teacher, training data, and evaluation protocol; their difference is that standard OPD retains token\-level reverse\-KL supervision throughout the response, whereas R2\-OPD filters supervision on progress\-conflicting segments\. All displayed responses use the same AIME 2024 prompt format and an 8,192\-token evaluation limit\.

Within each incorrect response, the bold label and underlined passages identify the erroneous transition together with its immediate consequence\. Longer annotations are split across adjacent lines so that standard underlining remains within the page boundary\. We use no color or background shading\.

### C\.1Case 1: Hyperbola\-Constrained Rhombus

Problem\.LetAA,BB,CC, andDDbe points on the hyperbolax220−y224=1\\frac\{x^\{2\}\}\{20\}\-\\frac\{y^\{2\}\}\{24\}=1such thatA​B​C​DABCDis a rhombus whose diagonals intersect at the origin\. Find the greatest real number that is less thanB​D2BD^\{2\}for all such rhombi\.Reference answer:480\\boxed\{480\}\.

Base\-model response \(incorrect:80\\boxed\{80\}\)\.

Giventhehyperbola\\\(\\frac\{x^2\}\{20\}\-\\frac\{y^2\}\{24\}=1\\\),weneedtofindthegreatestrealnumberthatislessthan\\\(BD^2\\\)forallrhombi\\\(ABCD\\\)whosediagonalsintersectattheorigin\.

1\.Thehyperbolahasverticesat\\\(\(\\pm\\sqrt\{20\},0\)\\\)and\\\(\(0,\\pm\\sqrt\{24\}\)\\\)\.

2\.Points\\\(A\\\),\\\(B\\\),\\\(C\\\),and\\\(D\\\)formarhombuswithdiagonalsintersectingattheorigin,implying\\\(A\\\)and\\\(B\\\)aresymmetricwithrespecttotheorigin\.

3\.Theslopesofthediagonals\\\(AC\\\)and\\\(BD\\\)arenegativereciprocals,leadingtothecondition\\\(\\frac\{y\_1\}\{x\_1\}\\times\\frac\{y\_2\}\{x\_2\}=\-1\\\)\.

4\.Expressing\\\(BD^2\\\)intermsofcoordinatesof\\\(B\\\)and\\\(D\\\),weget\\\(BD^2=4\(x\_2^2\+y\_2^2\)\\\)\.

5\.Usingthehyperbolaequationandtheslopecondition,weexpress\\\(x\_2^2\\\)and\\\(y\_2^2\\\)intermsofaparameter\\\(k=\\frac\{x\_1\}\{y\_1\}\\\)\.

6\.Substitutingandsimplifying,wefind\\\(BD^2\\\)asafunctionof\\\(k^2\\\),denoted\\\(t\\\)\.

7\.Thefunction\\\(f\(t\)=\\frac\{960\(1\+t\)\}\{12\-10t\}\\\)isanalyzed,showingitisincreasingfor\\\(0\\leqt<\\frac\{6\}\{5\}\\\)\.

\[ERRONEOUS STEP\]8\. The minimum value ofB​D2BD^\{2\}occurs att=0t=0, givingB​D2=80BD^\{2\}=80\.

Thus,thegreatestrealnumberthatislessthan\\\(BD^2\\\)forallsuchrhombiis\\\(\\boxed\{80\}\\\)\.

OPD response \(incorrect:80\\boxed\{80\}\)\.

Giventhehyperbola\\\(\\frac\{x^2\}\{20\}\-\\frac\{y^2\}\{24\}=1\\\),points\\\(A\\\),\\\(B\\\),\\\(C\\\),and\\\(D\\\)lieonthishyperbolasuchthat\\\(ABCD\\\)isarhombuswithdiagonalsintersectingattheorigin\.Weneedtofindthegreatestrealnumberlessthan\\\(BD^2\\\)forallsuchrhombi\.

1\.Thehyperbolaequationis\\\(\\frac\{x^2\}\{20\}\-\\frac\{y^2\}\{24\}=1\\\)\.

2\.Therhombusdiagonalsintersectattheoriginandareperpendicular,implyingtheirslopesmultiplyto\-1\.

3\.Points\\\(A\\\),\\\(B\\\),\\\(C\\\),and\\\(D\\\)lieonthehyperbolaandaresymmetricwithrespecttotheorigin\.

4\.Thecoordinatesofpoints\\\(A\\\),\\\(B\\\),\\\(C\\\),and\\\(D\\\)satisfythehyperbolaequationandtheperpendicularitycondition\.

5\.Thecondition\\\(abcd\+bcd=0\\\)andthehyperbolaconstraintsleadtotheexpressionfor\\\(BD^2\\\)\.

6\.Usingtheconditionsandparametrization,wefind\\\(BD^2=4\(c^2\+d^2\)\\\)\.

7\.Expressing\\\(BD^2\\\)intermsof\\\(a\\\)and\\\(b\\\),wefind

\[ERRONEOUS STEP\]B​D2=80\+440011​t−720BD^\{2\}=80\+\\frac\{4400\}\{11t\-720\}, wheret=a2t=a^\{2\}\.

8\. Asttapproaches infinity,B​D2BD^\{2\}approaches 80 from above\.

9\.Thegreatestrealnumberlessthan\\\(BD^2\\\)forallsuchrhombiistheinfimum,whichis80\.

Thus,theansweris\\\(\\boxed\{80\}\\\)\.

R2\-OPD response \(correct:480\\boxed\{480\}\)\.

Giventhehyperbola\\\(\\frac\{x^2\}\{20\}\-\\frac\{y^2\}\{24\}=1\\\),weneedtofindthegreatestrealnumberthatislessthan\\\(BD^2\\\)forallrhombi\\\(ABCD\\\)withdiagonalsintersectingattheorigin\.

1\.Theverticesoftherhombusare\\\(A,B,C,D\\\)onthehyperbola,withdiagonalsintersectingattheorigin\.

2\.Thecoordinatesoftheverticescanberepresentedas\\\(A\(a,b\)\\\),\\\(B\(c,d\)\\\),\\\(C\(\-a,\-b\)\\\),and\\\(D\(\-c,\-d\)\\\)\.

3\.Theconditionforthediagonalstobeperpendicularis\\\(ac\+bd=0\\\)\.

4\.Thedistances\\\(AB\\\)and\\\(BC\\\)mustbeequal,leadingtotheequation\\\(ac\+bd=0\\\)\.

5\.Usingthehyperbolaequation,weexpress\\\(b^2\\\)and\\\(d^2\\\)intermsof\\\(a\\\)and\\\(c\\\)\.

6\.Thedistance\\\(BD^2\\\)iscalculatedas\\\(4\(c^2\+d^2\)\\\)\.

7\.Bysubstitutingandsimplifying,wefindthat\\\(BD^2=480\+\\frac\{288000\}\{u\}\\\)where\\\(u=11t\-720\\\)\.

8\.As\\\(u\\\)approachesinfinity,\\\(BD^2\\\)approaches480fromabove,makingtheinfimumof\\\(BD^2\\\)equalto480\.

Thus,thegreatestrealnumberthatislessthan\\\(BD^2\\\)forallsuchrhombiis\\\(\\boxed\{480\}\\\)\.

The base model and OPD both admit the boundary valuet=0t=0without establishing that it corresponds to a valid rhombus on the hyperbola, yielding 80\. In contrast, R2\-OPD preserves the geometric constraints through the limiting argument and correctly identifies 480 as the infimum ofB​D2BD^\{2\}\.

### C\.2Case 2: Product over Roots of Unity

Problem\.Letω≠1\\omega\\neq 1be a 13th root of unity\. Find the remainder when∏k=012\(2−2​ωk\+ω2​k\)\\prod\_\{k=0\}^\{12\}\(2\-2\\omega^\{k\}\+\\omega^\{2k\}\)is divided by 1000\.Reference answer:321\\boxed\{321\}\.

Base\-model response \(incorrect:191\\boxed\{191\}\)\.

Given\\\(\\omega\\neq1\\\)asaprimitive13throotofunity,weneedtofindtheremainderwhen\\\(\\prod\_\{k=0\}^\{12\}\(2\-2\\omega^k\+\\omega^\{2k\}\)\\\)isdividedby1000\.

First,werecognizethat\\\(\\omega\\\)satisfies\\\(\\omega^\{13\}=1\\\)andtheminimalpolynomialfor\\\(\\omega\\\)isthe13thcyclotomicpolynomial,whichis\\\(\\Phi\_\{13\}\(x\)=x^\{12\}\+x^\{11\}\+\\dots\+x\+1\\\)\.

Weneedtoevaluatetheproduct\\\(\\prod\_\{k=0\}^\{12\}\(2\-2\\omega^k\+\\omega^\{2k\}\)\\\)\.Bysubstituting\\\(j=2k\\mod13\\\),wenotethatsince2iscoprimeto13,thesubstitutionisbijective\.

\[ERRONEOUS STEP\]Thus, the product simplifies to∏j=012\(2−ωj\)\\prod\_\{j=0\}^\{12\}\(2\-\\omega^\{j\}\)\.

Werecognizethatthisproductisequivalenttoevaluatingthepolynomial\\\(P\(x\)=2\-2x\+x^2\\\)at\\\(x=\\omega^j\\\)forall\\\(j\\\)\.

It then concludes that∏j=012\(2−ωj\)=213−1\\prod\_\{j=0\}^\{12\}\(2\-\\omega^\{j\}\)=2^\{13\}\-1\.

Calculatingthis,weget:

\\\[2^\{13\}\-1=8192\-1=8191\\\]

Finally,wefindtheremainderwhen8191isdividedby1000:

\\\[8191\\mod1000=191\\\]

Thus,theremainderis\\\(\\boxed\{191\}\\\)\.

OPD response \(incorrect; truncated at 8,192 tokens, extracted answer11\)\.

<think\>

Okay,soI’vegotthisproblemherewhereIneedtofindtheremainderwhentheproduct\\\(\\prod\_\{k=0\}^\{12\}\(2\-2\\omega^k\+\\omega^\{2k\}\)\\\)isdividedby1000,where\\\(\\omega\\neq1\\\)isa13throotofunity\.Hmm,let’strytobreakthisdownstepbystep\.

First,let’srecallthat\\\(\\omega\\\)isaprimitive13throotofunity,meaning\\\(\\omega^\{13\}=1\\\)and\\\(\\omega^k\\neq1\\\)for\\\(k=1,2,\.\.\.,12\\\)\.So,allthepowersof\\\(\\omega\\\)cycleevery13terms\.

Theexpressioninsidetheproductis\\\(2\-2\\omega^k\+\\omega^\{2k\}\\\)\.MaybeIcanrewritethisexpressioninamoremanageableform\.Letmefactororsimplifyit\.

Letmenotethat\\\(\\omega^\{2k\}\\\)isthesameas\\\(\(\\omega^k\)^2\\\)\.So,thetermbecomes\\\(2\-2\\omega^k\+\(\\omega^k\)^2\\\)\.Wait,thatlooksfamiliar\.IfIlet\\\(x=\\omega^k\\\),thentheexpressionbecomes\\\(2\-2x\+x^2\\\)\.Whichisaquadraticinx\.

So,\\\(x^2\-2x\+2\\\)\.Hmm,maybethatcanbefactoredorexpressedintermsofrootsofunity\.

Alternatively,perhapsit’saquadraticinx,somaybeeachtermisaquadraticpolynomialevaluatedat\\\(x=\\omega^k\\\)\.Since\\\(\\omega^k\\\)arethe13throotsofunity,perhapswecanrelatetheproductoverallktosomepolynomialevaluatedatspecificpoints\.

Irememberthattheproductoverrootsofunitycansometimesbeconnectedtocyclotomicpolynomialsorsomethingsimilar\.MaybeIcanexpressthisproductasaproductoverkofaquadraticin\\\(\\omega^k\\\),andthenrelatethattothevalueofapolynomialatspecificpoints\.

Letmeconsidertheproduct\\\(\\prod\_\{k=0\}^\{12\}\(2\-2\\omega^k\+\\omega^\{2k\}\)\\\)\.Since\\\(\\omega\\\)isaprimitive13throot,theminimalpolynomialisthe13thcyclotomicpolynomial,whichis\\\(\\Phi\_\{13\}\(x\)=x^\{12\}\+x^\{11\}\+\\dots\+x\+1\\\),butI’mnotsureifthat’sdirectlyhelpfulhere\.

Alternatively,maybeconsiderthateachterm\\\(2\-2\\omega^k\+\\omega^\{2k\}\\\)canberewrittenas\\\(\(\\sqrt\{2\}\)^2\-2\\omega^k\+\(\\omega^k\)^2\\\)\.Wait,thatresemblestheexpressionforaquadraticin\\\(\\omega^k\\\)\.Alternatively,perhapsfactoritas\\\(\(1\+\\omega^k\)^2\+1\\\),butletmecheck\.

Wait,let’scompute\\\(2\-2\\omega^k\+\\omega^\{2k\}\\\)\.

Letmedenote\\\(z=\\omega^k\\\),thenthetermbecomes\\\(2\-2z\+z^2=z^2\-2z\+2\\\)\.Hmm,maybewecanfactorthisquadratic?

Thediscriminantis\\\(4\-8=\-4\\\),sotherootsare\\\(\[2\\pm\\sqrt\{\-4\}\]/2=1\\pmi\\\)\.Hmm,so\\\(z^2\-2z\+2=\(z\-\(1\+i\)\)\(z\-\(1\-i\)\)\\\)\.So,\\\(2\-2z\+z^2=\(z\-\(1\+i\)\)\(z\-\(1\-i\)\)\\\)\.

Therefore,substitutingback\\\(z=\\omega^k\\\),weget:

\\\(\(\\omega^k\-\(1\+i\)\)\(\\omega^k\-\(1\-i\)\)\\\)\.

Therefore,theproductbecomes:

\\\(\\prod\_\{k=0\}^\{12\}\(\\omega^k\-\(1\+i\)\)\(\\omega^k\-\(1\-i\)\)=\\left\(\\prod\_\{k=0\}^\{12\}\(\\omega^k\-\(1\+i\)\)\\right\)\\left\(\\prod\_\{k=0\}^\{12\}\(\\omega^k\-\(1\-i\)\)\\right\)\\\)\.

Now,thatseemspromisingbecausetheproductover\(z\-\\omega^k\)isrelatedtothecyclotomicpolynomialevaluatedatz\.

Ingeneral,foraprimitiventhrootofunity,theproduct\\\(\\prod\_\{k=0\}^\{n\-1\}\(x\-\\omega^k\)=x^n\-1\\\),butwait,actuallyforprimitiveroots,it’sthecyclotomicpolynomial\.Wait,let’srecallthatforanyn,\\\(\\prod\_\{k=0\}^\{n\-1\}\(x\-\\omega^k\)=x^n\-1\\\),butthat’strueif\\\(\\omega\\\)isaprimitiventhrootofunity\.

Wait,actually,moreprecisely,if\\\(\\omega\\\)isaprimitiventhrootofunity,then\\\(\\prod\_\{k=0\}^\{n\-1\}\(x\-\\omega^k\)=x^n\-1\\\)\.Soyes,that’scorrect\.

Therefore,inourcase,since\\\(\\omega\\\)isaprimitive13throot,then\\\(\\prod\_\{k=0\}^\{12\}\(x\-\\omega^k\)=x^\{13\}\-1\\\)\.Therefore,theproduct\\\(\\prod\_\{k=0\}^\{12\}\(\\omega^k\-a\)=\\prod\_\{k=0\}^\{12\}\(\-\(a\-\\omega^k\)\)=\(\-1\)^\{13\}\\prod\_\{k=0\}^\{12\}\(a\-\\omega^k\)=\(\-1\)^\{13\}\(a^\{13\}\-1\)\\\)\.

Wait,let’sverifythatstep\.

Giventhat\\\(\\prod\_\{k=0\}^\{12\}\(x\-\\omega^k\)=x^\{13\}\-1\\\),sosubstitutingx=a,wehave\\\(\\prod\_\{k=0\}^\{12\}\(a\-\\omega^k\)=a^\{13\}\-1\\\)\.Therefore,\\\(\\prod\_\{k=0\}^\{12\}\(\\omega^k\-a\)=\(\-1\)^\{13\}\\prod\_\{k=0\}^\{12\}\(a\-\\omega^k\)=\(\-1\)^\{13\}\(a^\{13\}\-1\)\\\)\.

Since13isodd,\(\-1\)^13=\-1,so\\\(\\prod\_\{k=0\}^\{12\}\(\\omega^k\-a\)=\-\(a^\{13\}\-1\)=1\-a^\{13\}\\\)\.

Wait,sothatmeans\\\(\\prod\_\{k=0\}^\{12\}\(\\omega^k\-a\)=1\-a^\{13\}\\\)\.

So,inourcase,forthefirstproduct,\\\(\\prod\_\{k=0\}^\{12\}\(\\omega^k\-\(1\+i\)\)=1\-\(1\+i\)^\{13\}\\\)\.

Similarly,\\\(\\prod\_\{k=0\}^\{12\}\(\\omega^k\-\(1\-i\)\)=1\-\(1\-i\)^\{13\}\\\)\.

Therefore,theoriginalproductbecomes:

\\\(\[1\-\(1\+i\)^\{13\}\]\\times\[1\-\(1\-i\)^\{13\}\]\\\)\.

So,letmecomputethisexpression:

Letmedenote\\\(A=1\-\(1\+i\)^\{13\}\\\)and\\\(B=1\-\(1\-i\)^\{13\}\\\)\.ThentheproductisA\*B\.

So,computeA\*B=\[1\-\(1\+i\)^\{13\}\]\[1\-\(1\-i\)^\{13\}\]\.

Letmecompute\(1\+i\)^\{13\}and\(1\-i\)^\{13\}\.

First,let’scompute\(1\+i\)^13\.

Weknowthat1\+iisacomplexnumberwithmagnitudesqrt\(2\)andanglepi/4\.So,inpolarform,1\+i=sqrt\(2\)e^\{ipi/4\}\.

Therefore,\(1\+i\)^13=\(sqrt\(2\)\)^13e^\{i13pi/4\}\.

Simplifytheangle:13pi/4\.Subtract2piuntilit’swithin0to2pi\.

13pi/4=3pi\+pi/4=3pi\+45degrees,whichismorethan2pi\.

Subtract2pi\(whichis8pi/4\)from13pi/4:13pi/4\-8pi/4=5pi/4\.So,angleis5pi/4\.

Therefore,\(1\+i\)^13=\(sqrt\(2\)\)^13e^\{i5pi/4\}\.

Similarly,\(1\-i\)^13=\(sqrt\(2\)\)^\{13\}e^\{\-i5pi/4\},since1\-iissqrt\(2\)e^\{\-ipi/4\}\.

So,compute\(1\+i\)^13=\(sqrt\(2\)\)^13\[cos\(5pi/4\)\+isin\(5pi/4\)\]\.

Similarly,\(1\-i\)^13=\(sqrt\(2\)\)^13\[cos\(\-5pi/4\)\+isin\(\-5pi/4\)\]=\(sqrt\(2\)\)^13\[cos\(5pi/4\)\-isin\(5pi/4\)\]\.

Compute\(sqrt\(2\)\)^13:

sqrt\(2\)=2^\{1/2\},so\(sqrt\(2\)\)^\{13\}=2^\{13/2\}=2^6\*2^\{1/2\}=64\*sqrt\(2\)\.

So,\(1\+i\)^13=64sqrt\(2\)\[cos\(5pi/4\)\+isin\(5pi/4\)\]\.

Similarly,\(1\-i\)^13=64sqrt\(2\)\[cos\(5pi/4\)\-isin\(5pi/4\)\]\.

Computecos\(5pi/4\)andsin\(5pi/4\):

cos\(5pi/4\)=\-sqrt\(2\)/2,sin\(5pi/4\)=\-sqrt\(2\)/2\.

So,substituting:

\(1\+i\)^13=64sqrt\(2\)\[\-sqrt\(2\)/2\-isqrt\(2\)/2\]=64sqrt\(2\)\*\(\-sqrt\(2\)/2\)\(1\+i\)=64sqrt\(2\)\*\(\-sqrt\(2\)/2\)\*\(1\+i\)\.

Compute64sqrt\(2\)\*\(\-sqrt\(2\)/2\)=64\*\(\-2\)/2=64\*\(\-1\)=\-64\.

Therefore,\(1\+i\)^13=\-64\(1\+i\)\.

Similarly,\(1\-i\)^13=64sqrt\(2\)\[\-sqrt\(2\)/2\+isqrt\(2\)/2\]=64sqrt\(2\)\*\(\-sqrt\(2\)/2\)\(1\-i\)\.

Compute64sqrt\(2\)\*\(\-sqrt\(2\)/2\)=\-64,so\(1\-i\)^13=\-64\(1\-i\)\.

Therefore,puttingbackintoAandB:

A=1\-\(1\+i\)^13=1\-\(\-64\(1\+i\)\)=1\+64\(1\+i\)=1\+64\+64i=65\+64i\.

Similarly,B=1\-\(1\-i\)^13=1\-\(\-64\(1\-i\)\)=1\+64\(1\-i\)=1\+64\-64i=65\-64i\.

Therefore,theproductA\*B=\(65\+64i\)\(65\-64i\)\.

Thisisaproductofacomplexconjugatepair:\(a\+b\)\(a\-b\)=a^2\-b^2\.

Computethis:

\(65\)^2\-\(64i\)^2=4225\-\(64^2\)\(i^2\)=4225\-\(4096\)\(\-1\)=4225\+4096=8321\.

So,theproductis8321\.

Therefore,theoriginalproductis8321\.Now,theproblemsaystofindtheremainderwhenthisproductisdividedby1000\.

So,compute8321mod1000\.

Divide8321by1000:1000\*8=8000,8321\-8000=321\.Sotheremainderis321\.

Wait,letmeconfirmthestepsagainbecauseImighthavemadeamistakesomewhere\.

Wait,let’sgoback\.

Wehadtheproductas\[1\-\(1\+i\)^13\]\[1\-\(1\-i\)^13\]=\(65\+64i\)\(65\-64i\)=65^2\+64^2\.

Wait,holdon\!Wait,IthinkImadeamistakeincomputingA\*B\.Wait,let’sre\-examine:

Earlier,IhadA=1\-\(1\+i\)^13=1\-\(\-64\(1\+i\)\)=1\+64\(1\+i\)=65\+64i\.

Similarly,B=1\-\(1\-i\)^13=1\-\(\-64\(1\-i\)\)=1\+64\(1\-i\)=65\-64i\.

Therefore,A=65\+64iandB=65\-64i\.

ThenA\*B=\(65\+64i\)\(65\-64i\)=65^2\-\(64i\)^2=4225\-\(64^2\)\(i^2\)=4225\-\(4096\)\(\-1\)=4225\+4096=8321\.

Yes,that’scorrect\.Sotheproductis8321\.

Now,8321dividedby1000\.1000\*8=8000,so8321\-8000=321\.Sotheremainderis321\.

Wait,butwaitasecond,isthatcorrect?Because8321dividedby1000is8withremainder321,yes\.

Butwait,letmemakesurethattheinitialsubstitutionwascorrect\.Let’sdouble\-check\.

Westartedwiththeproductoverk=0to12of\(2\-2omega^k\+omega^\{2k\}\)\.

Werewroteitastheproductoverkof\(omega^k\-\(1\+i\)\)\(omega^k\-\(1\-i\)\),whichledustotheproductover\(1\-\(1\+i\)^\{13\}\)\(1\-\(1\-i\)^\{13\}\)\.

Wait,let’sconfirmthestepwherewefactoredthequadratic\.

Wehadtheterm2\-2omega^k\+omega^\{2k\}=\(omega^k\-\(1\+i\)\)\(omega^k\-\(1\-i\)\)\.

Yes,that’scorrectbecauseexpanding\(x\-\(1\+i\)\)\(x\-\(1\-i\)\)=x^2\-2x\+\(1\+i\)\(1\-i\)=x^2\-2x\+\(1\-i^2\)=x^2\-2x\+2,whichmatchestheoriginalexpression\.

So,thatstepiscorrect\.

Then,theproductoverk=0to12of\(omega^k\-\(1\+i\)\)isequalto\(\-1\)^13\*\(\(1\+i\)^13\-1\)?Wait,waitearlierreasoningwasthatproduct\_\{k=0\}^\{12\}\(a\-omega^k\)=a^\{13\}\-1\.Wait,butwaitlet’sthinkagain\.

Giventhattheproduct\_\{k=0\}^\{12\}\(x\-omega^k\)=x^\{13\}\-1\.

So,ifwesubstitutex=a,thenproduct\_\{k=0\}^\{12\}\(a\-omega^k\)=product\_\{k=0\}^\{12\}\(a\-omega^k\)=\(\-1\)^13product\_\{k=0\}^\{12\}\(omega^k\-a\)=\(\-1\)^13\(a^\{13\}\-1\)\.

So,product\_\{k=0\}^\{12\}\(omega^k\-a\)=\(\-1\)^\{13\}\(a^\{13\}\-1\)=\-\(a^\{13\}\-1\)=1\-a^\{13\}\.

Yes,sothatwascorrect\.

Therefore,product\_\{k=0\}^\{12\}\(omega^k\-a\)=1\-a^\{13\}\.

Therefore,product\_\{k=0\}^\{12\}\(omega^k\-\(1\+i\)\)=1\-\(1\+i\)^\{13\}\.

Similarlyfor\(1\-i\)\.

So,thatstepiscorrect\.

Then,wecomputed\(1\+i\)^13=\-64\(1\+i\),and\(1\-i\)^13=\-64\(1\-i\)\.ThensubstitutingbackintoAandB:

A=1\-\(1\+i\)^13=1\-\(\-64\(1\+i\)\)=1\+64\(1\+i\)=65\+64i\.

Similarly,B=1\-\(1\-i\)^13=1\-\(\-64\(1\-i\)\)=1\+64\(1\-i\)=65\-64i\.

Therefore,A\*B=\(65\+64i\)\(65\-64i\)=65^2\+64^2=4225\+4096=8321\.

Sothatseemscorrect\.Then8321dividedby1000is8witharemainderof321\.

Wait,butholdonasecond\.Letmecheckthecalculationof\(1\+i\)^13and\(1\-i\)^13again\.

Wesaid\(1\+i\)hasmagnitudesqrt\(2\),anglepi/4\.So,\(1\+i\)^13=\(sqrt\(2\)\)^13e^\{i\*13\*pi/4\}\.

Compute\(sqrt\(2\)\)^13:2^\(13/2\)=2^6\*2^\(1/2\)=64\*sqrt\(2\)\.

Angle:13\*pi/4\.Subtractmultiplesof2piuntilit’sbetween0and2pi\.

13pi/4=3pi\+pi/4=3\.75pi\.

Subtract2pionce:3\.75pi\-2pi=1\.75pi=7pi/4\.

Soangleis7pi/4\.

So,\(1\+i\)^13=64sqrt\(2\)e^\{i7pi/4\}\.

e^\{i7pi/4\}=cos\(7pi/4\)\+isin\(7pi/4\)=sqrt\(2\)/2\-isqrt\(2\)/2\.

Therefore,\(1\+i\)^13=64sqrt\(2\)\(sqrt\(2\)/2\-isqrt\(2\)/2\)=64sqrt\(2\)\*sqrt\(2\)/2\(1\-i\)=64\*\(2\)/2\(1\-i\)=64\*\(1\-i\)=64\-64i\.

Wait,waitthatcontradictsourearliercomputationwherewegot\-64\(1\+i\)\.Hmm,maybeImadeamistakeinanglesubtraction\.

Wait,letmecompute13\*pi/4\.

13dividedby4is3\.25pi,whichis3pi\+pi/4,whichis3\.75pi\.

Subtract2pionce:3\.75pi\-2pi=1\.75pi,whichis7pi/4\.

Soangleis7pi/4,whichis315degrees,socos\(7pi/4\)=sqrt\(2\)/2,sin\(7pi/4\)=\-sqrt\(2\)/2\.

Therefore,\(1\+i\)^13=\(sqrt\(2\)\)^13\[cos\(7pi/4\)\+isin\(7pi/4\)\]=64sqrt\(2\)\[sqrt\(2\)/2\-isqrt\(2\)/2\]=64sqrt\(2\)\*sqrt\(2\)/2\(1\-i\)=64\*\(2\)/2\(1\-i\)=64\*\(1\-i\)=64\-64i\.

Wait,sothiscontradictsmyearliercalculationwhereIthought\(1\+i\)^13=\-64\(1\+i\)\.Hmm,sowhichiscorrect?

Wait,let’srecalculate\(1\+i\)^13stepbystep\.

Express\(1\+i\)inpolarform:modulussqrt\(2\),anglepi/4\.

Therefore,\(1\+i\)^13=\(sqrt\(2\)\)^13\*e^\{i\*13\*pi/4\}\.

Computemodulus:\(sqrt\(2\)\)^13=2^\(13/2\)=2^6\*2^\(1/2\)=64\*sqrt\(2\)\.

Angle:13\*pi/4=3\.25pi\.

Subtract2piuntilangleisbetween0and2pi\.

3\.25pi\-2pi=1\.25pi,whichis5pi/4\.

Wait,wait,13\*pi/4=3\.25pi=3pi\+pi/4\.

3pi\+pi/4=3\.75pi\.

Subtract2pi:3\.75pi\-2pi=1\.75pi=7pi/4\.

Wait,soangleis7pi/4,not5pi/4\.So,angleis7pi/4\.

So,e^\{i7pi/4\}=cos\(7pi/4\)\+isin\(7pi/4\)=sqrt\(2\)/2\-isqrt\(2\)/2\.

Therefore,\(1\+i\)^13=64sqrt\(2\)\*\[sqrt\(2\)/2\-isqrt\(2\)/2\]=64sqrt\(2\)\*sqrt\(2\)/2\(1\-i\)=64\*\(2\)/2\(1\-i\)=64\*\(1\-i\)=64\-64i\.

Wait,sothiscontradictsmyearlierconclusionthat\(1\+i\)^13=\-64\(1\+i\)\.SowheredidIgowrongearlier?

Wait,let’scompute\(1\+i\)^13usingbinomialexpansionoranothermethod\.

Alternatively,maybecompute\(1\+i\)^13usingbinomialtheoremorusingDeMoivre’stheorem\.

Wait,usingDeMoivre’stheorem,\(1\+i\)=sqrt\(2\)e^\{ipi/4\},so\(1\+i\)^13=\(sqrt\(2\)\)^13e^\{i13pi/4\}\.

Whichis2^\(13/2\)e^\{i\(13pi/4\)\}\.

Simplify13pi/4:subtract2pi\(8pi/4\)toget5pi/4\.

So,e^\{i5pi/4\}=cos\(5pi/4\)\+isin\(5pi/4\)=\-sqrt\(2\)/2\-isqrt\(2\)/2\.

Therefore,\(1\+i\)^13=2^\(13/2\)e^\{i5pi/4\}=2^\(13/2\)\(\-sqrt\(2\)/2\-isqrt\(2\)/2\)=2^\(13/2\)\*\(\-sqrt\(2\)/2\)\(1\+i\)\.

Compute2^\(13/2\)\*\(\-sqrt\(2\)/2\)=2^\(6\.5\)\*\(\-2^\(1/2\)/2\)=2^6\*sqrt\(2\)\*\(\-sqrt\(2\)/2\)=64\*\(sqrt\(2\)\*\-sqrt\(2\)\)/2=64\*\(\-2\)/2=64\*\(\-1\)=\-64\.

Therefore,\(1\+i\)^13=\-64\(1\+i\)\.

Ah,soearlierstepwhereIthoughtanglewas7pi/4waswrong\.Itshouldbe5pi/4\.

So,let’scorrectthat\.

So,\(1\+i\)^13=2^\(13/2\)e^\{i5pi/4\}=2^\(13/2\)\[cos\(5pi/4\)\+isin\(5pi/4\)\]\.

Computecos\(5pi/4\)=\-sqrt\(2\)/2,sin\(5pi/4\)=\-sqrt\(2\)/2\.

So,\(1\+i\)^13=2^\(13/2\)\(\-sqrt\(2\)/2\-isqrt\(2\)/2\)=2^\(13/2\)\*\(\-sqrt\(2\)/2\)\(1\+i\)\.

Simplify2^\(13/2\)\*\(\-sqrt\(2\)/2\)=2^\(6\.5\)\*\(\-2^\(1/2\)/2\)=2^6\*sqrt\(2\)\*\(\-sqrt\(2\)/2\)=64\*\(2\)/2\*\(\-1\)=64\*1\*\(\-1\)=\-64\.

Therefore,\(1\+i\)^13=\-64\(1\+i\)\.

Similarly,\(1\-i\)^13:let’scompute\.

\(1\-i\)hasmodulussqrt\(2\),angle\-pi/4\.

So,\(1\-i\)^13=\(sqrt\(2\)\)^13e^\{i13\*\(\-pi/4\)\}=2^\(13/2\)e^\{\-i13pi/4\}\.

Simplifyangle:\-13pi/4\.Add2piuntilwithin0to2pi\.

\-13pi/4\+2pi\*3=\-13pi/4\+24pi/4=11pi/4\.

11pi/4\-2pi=11pi/4\-8pi/4=3pi/4\.

Soangleis3pi/4\.

Thus,e^\{\-i13pi/4\}=e^\{\-i3pi/4\}=cos\(3pi/4\)\-isin\(3pi/4\)=\-sqrt\(2\)/2\-isqrt\(2\)/2\.

Therefore,\(1\-i\)^13=2^\(13/2\)\[\-sqrt\(2\)/2\-isqrt\(2\)/2\]=2^\(13/2\)\*\(\-sqrt\(2\)/2\)\(1\+i\)\.

Again,2^\(13/2\)\*\(\-sqrt\(2\)/2\)=2^\(6\.5\)\*\(\-2^\(0\.5\)/2\)=64\*sqrt\(2\)\*\(\-sqrt\(2\)/2\)=64\*\(\-2\)/2=64\*\(\-1\)=\-64\.

Therefore,\(1\-i\)^13=\-64\(1\+i\)and\-64\(1\-i\)?

Wait,waitlet’scomputeitstepbystep\.

\(1\-i\)^13=\(sqrt\(2\)\)^13e^\{\-i13pi/4\}\.

Computeangle:\-13pi/4\.

Add2piuntilit’swithin0to2pi\.

\-13pi/4\+8pi/4=\(\-13\+8\)pi/4=\(\-5pi\)/4\.

But\-5pi/4isequivalentto3pi/4\(since\-5pi/4\+2pi=3pi/4\)\.

Wait,maybeanotherapproach\.Let’scomputee^\{\-i13pi/4\}\.

\-13pi/4=\-3pi\-pi/4=\-3pi\-pi/4\.Add4pitogetwithin0to4pi\.

\-13pi/4\+4pi=\-13pi/4\+16pi/4=3pi/4\.

Therefore,e^\{\-i13pi/4\}=e^\{\-i3pi/4\}=cos\(\-3pi/4\)\+isin\(\-3pi/4\)=cos\(3pi/4\)\-isin\(3pi/4\)=\-sqrt\(2\)/2\-isqrt\(2\)/2\.

Therefore,\(1\-i\)^13=2^\(13/2\)\*\[\-sqrt\(2\)/2\-isqrt\(2\)/2\]\.

Whichis2^\(13/2\)\*\(\-sqrt\(2\)/2\)\(1\+i\)\.

Compute2^\(13/2\)\*\(\-sqrt\(2\)/2\)=2^\(6\.5\)\*\(\-2^\(0\.5\)/2\)=64\*sqrt\(2\)\*\(\-sqrt\(2\)/2\)=64\*\(\-2\)/2=64\*\(\-1\)=\-64\.

Therefore,\(1\-i\)^13=\-64\(1\+i\)\.

Wait,soboth\(1\+i\)^13and\(1\-i\)^13equalto\-64\(1\+i\)and\-64\(1\-i\)respectively?

Wait,holdon,no\.

Wait,\(1\+i\)^13=\-64\(1\+i\)\.

Similarly,\(1\-i\)^13=\-64\(1\-i\)\.

Wait,waitthatconflictswithearlier\.Wait,perhapsnot\.

Wait,let’srecast:

\(1\+i\)^13=\-64\(1\+i\)\.

\(1\-i\)^13=\-64\(1\-i\)\.

Solet’ssubstitutebackintoAandB\.

So,A=1\-\(1\+i\)^13=1\-\(\-64\(1\+i\)\)=1\+64\(1\+i\)=1\+64\+64i=65\+64i\.

Similarly,B=1\-\(1\-i\)^13=1\-\(\-64\(1\-i\)\)=1\+64\(1\-i\)=1\+64\-64i=65\-64i\.

Therefore,A=65\+64iandB=65\-64i\.

Therefore,A\*B=\(65\+64i\)\(65\-64i\)=65^2\+64^2=4225\+4096=8321\.

So,theproductis8321,whichmodulo1000is8321\-8\*1000=8321\-8000=321\.

Wait,butwait,let’sconfirm\.

Wait,8321dividedby1000is8\*1000=8000,8321\-8000=321\.Soyes,theremainderis321\.

Wait,butearlierwhenIthought\(1\+i\)^13was\-64\(1\+i\),whichledtoA=65\+64iandB=65\-64i,whichwhenmultipliedgive65^2\+64^2=8321\.

Butwait,letmechecktheangleagain\.

Wait,whenIcomputed\(1\+i\)^13,Iinitiallythoughtanglewas5pi/4,butthencorrecteditto7pi/4,butthenanotherwayitwas3pi/4\.

Wait,perhapsImadeamistakeincomputingtheangle\.

Letmecompute\(1\+i\)^13\.

Express1\+iinpolarform:modulussqrt\(2\),anglepi/4\.

So\(1\+i\)^13=\(sqrt\(2\)\)^13e^\{i\*13\*pi/4\}\.

13\*pi/4=3pi\+pi/4=3\.75pi\.

Subtract2pitwice:3\.75pi\-2pi=1\.75pi,whichis7pi/4\.

Wait,1\.75piis7pi/4,whichisinthefourthquadrant\.

Soangleis7pi/4\.

So,e^\{i7pi/4\}=cos\(7pi/4\)\+isin\(7pi/4\)=sqrt\(2\)/2\-isqrt\(2\)/2\.

Therefore,\(1\+i\)^13=\(sqrt\(2\)\)^13\[sqrt\(2\)/2\-isqrt\(2\)/2\]\.

Compute\(sqrt\(2\)\)^13\*sqrt\(2\)/2=\(2^\(1/2\)\)^13\*2^\(1/2\)/2=2^\(13/2\+1/2\)/2=2^\(14/2\)/2=2^7/2=128/2=64\.

Similarly,imaginarypart:\(sqrt\(2\)\)^13\*\(\-sqrt\(2\)/2\)=64\*\(\-sqrt\(2\)/2\)=64\*\(\-1/sqrt2\)=\-64\*sqrt\(2\)/2=\-32sqrt\(2\)\.

Wait,waitno:

Wait,let’scomputestepbystep\.

\(1\+i\)^13=\(sqrt\(2\)\)^13\*e^\{i7pi/4\}=\(2^\(1/2\)\)^13\*\(sqrt\(2\)/2\-isqrt\(2\)/2\)=2^\(13/2\)\*\[sqrt\(2\)/2\-isqrt\(2\)/2\]\.

Compute2^\(13/2\)\*sqrt\(2\)/2=2^\(13/2\+1/2\)/2^\(1\)\)Wait,wait:

Wait,2^\(13/2\)\*sqrt\(2\)/2=2^\(13/2\)\*2^\(1/2\)/2=2^\(14/2\)/2=2^7/2=64/2=32\.

Similarly,theimaginarypart:2^\(13/2\)\*\(\-sqrt\(2\)/2\)=\-2^\(13/2\+1/2\)/2=\-2^\(14/2\)/2=\-2^7/2=\-128/2=\-64\.

\[ERRONEOUS STEP\]Therefore,\(1\+i\)13=32−64​i\(1\+i\)^\{13\}=32\-64i\.

Wait, that’s different from the previously correct value, so the response restarts the derivation\.

Wait,letmecompute\(1\+i\)^13stepbystep\.

First,\(1\+i\)^1=1\+i

\(1\+i\)^2=\(1\+i\)\(1\+i\)=1\+2i\+i^2=1\+2i\-1=2i\.

\(1\+i\)^3=\(1\+i\)^2\*\(1\+i\)=2i\(1\+i\)=2i\+2i^2=2i\-2=\-2\+2i\.

\(1\+i\)^4=\(\-2\+2i\)\(1\+i\)=\-2\(1\+i\)\+2i\(1\+i\)=\-2\-2i\+2i\+2i^2=\-2\+0i\-2=\-4\.

\(1\+i\)^4=\-4\.

Therefore,\(1\+i\)^13=\(1\+i\)^4\*\(1\+i\)^9=\(\-4\)\*\(1\+i\)^9\.

Compute\(1\+i\)^9:

Wesawthat\(1\+i\)^1=1\+i

R2\-OPD response \(correct:321\\boxed\{321\}\)\.

Let\\\(\\omega\\neq1\\\)bea13throotofunity\.Weneedtofindtheremainderwhentheproduct

\\\[

\\prod\_\{k=0\}^\{12\}\(2\-2\\omega^k\+\\omega^\{2k\}\)

\\\]

isdividedby1000\.

First,wenotethatwhen\\\(k=0\\\),thetermis1\.For\\\(k\\\)from1to12,eachtermis\\\(2\-2\\omega^k\+\\omega^\{2k\}\\\)\.Thiscanbefactoredas\\\(\(\\omega^k\-\(1\+i\)\)\(\\omega^k\-\(1\-i\)\)\\\)\.

Thus,theproductbecomes:

\\\[

1\\times\\prod\_\{k=1\}^\{12\}\(\\omega^k\-\(1\+i\)\)\(\\omega^k\-\(1\-i\)\)

\\\]

Thisproductisequivalenttoevaluatingthepolynomial\\\(\\Phi\_\{13\}\(x\)\\\)at\\\(x=1\+i\\\)and\\\(x=1\-i\\\),where\\\(\\Phi\_\{13\}\(x\)\\\)isthe13thcyclotomicpolynomial\.

Thecyclotomicpolynomial\\\(\\Phi\_\{13\}\(x\)\\\)isgivenby:

\\\[

\\Phi\_\{13\}\(x\)=x^\{12\}\+x^\{11\}\+\\cdots\+x\+1

\\\]

Thus,weneedtocompute:

\\\[

\\Phi\_\{13\}\(1\+i\)\\times\\Phi\_\{13\}\(1\-i\)

\\\]

Calculating\\\(\\Phi\_\{13\}\(1\+i\)\\\)and\\\(\\Phi\_\{13\}\(1\-i\)\\\):

\-\\\(\\Phi\_\{13\}\(1\+i\)=\-64\+65i\\\)

\-\\\(\\Phi\_\{13\}\(1\-i\)=\-64\-65i\\\)

Multiplyingtheseresults:

\\\[

\(\-64\+65i\)\(\-64\-65i\)=\(\-64\)^2\-\(65i\)^2=4096\-\(\-4225\)=4096\+4225=8321

\\\]

Finally,theremainderwhen8321isdividedby1000is:

\\\[

8321\\mod1000=321

\\\]

Thus,theremainderis:

\\\[

\\boxed\{321\}

\\\]

The base model incorrectly reduces the quadratic factor to2−ωj2\-\\omega^\{j\}, changing the product and yielding 191\. OPD explores several algebraic routes but repeatedly revises its calculation and terminates without a complete solution; the evaluator extracts 1 from the truncated generation\. R2\-OPD instead factors the quadratic over1±i1\\pm i, evaluates the conjugate cyclotomic\-polynomial terms, and obtains8321≡321\(mod1000\)8321\\equiv\\boxed\{321\}\\pmod\{1000\}\.

These examples are qualitative rather than causal evidence\. They complement the aggregate results by showing concrete trajectories in which the competing responses either lose a necessary constraint or fail to complete a consistent derivation, whereas R2\-OPD reaches the verified answer\.