Pass the Baton: Trajectory-Relayed On-Policy Distillation

# Pass the Baton: Trajectory-Relayed On-Policy Distillation
Source: [https://arxiv.org/html/2607.26057](https://arxiv.org/html/2607.26057)
Haolei Xu1,2,Xiaowen Xu211footnotemark:1,Haiwen Hong211footnotemark:1,Zixuan Ni1,

Hongxing Li1,2,Yiwen Qiu1,Weiming Lu1,Yongliang Shen1

1

Zhejiang University,2Yuvion Team, Alibaba Group
\{xuhaolei, luwm, syl\}@zju\.edu\.cnhonghaiwen\.hhw@alibaba\-inc\.comGitHub:[https://github\.com/zju\-real/Relay\-OPD](https://github.com/zju-real/Relay-OPD)Project:[https://zju\-real\.github\.io/Relay\-OPD](https://zju-real.github.io/Relay-OPD/)

###### Abstract

On\-policy distillation \(OPD\) grounds token\-level supervision in the student’s own trajectory, yet suffers from prefix failure: once the student commits to a wrong reasoning direction, all subsequent generation builds on this deviation, producing misdirected continuations that elicit unreliable supervision and waste compute\. We identify a teacher–student continuation asymmetry on failed prefixes, where the teacher tends to redirect while the student continues along the original direction, and convert it into a label\-free handoff trigger in Relay On\-Policy Distillation \(Relay\-OPD\)\. During training, Relay\-OPD constructs relay trajectories by letting the teacher briefly take over at detected trigger points to produce a teacher leg, after which the student resumes and is optimized on the resulting trajectory\. A limited relay budget concentrates intervention on critical early positions while limiting departure from the student policy\. With a Qwen3\-4B\-Instruct\-2507 teacher and Qwen3\-0\.6B/1\.7B\-Non\-Thinking students on eight mathematical reasoning benchmarks, Relay\-OPD achieves the best or second\-best results on every benchmark, outperforming standard OPD by \+5\.73% and the strongest baseline FastOPD by \+1\.49% on average for 1\.7B, with consistent gains at 0\.6B\. Training trajectory length is reduced by over 50%\.

![Refer to caption](https://arxiv.org/html/2607.26057v1/x1.png)Figure 1:\(a\)A Relay\-OPD case: at the detected handoff trigger, the teacher prefers the reflection token \(But, 74\.4%\) while the student would continue along the current direction \(So, 50\.6%\); a brief teacher leg corrects the reasoning and the student resumes to the correct answer\.\(b\)Difference between OPD and Relay\-OPD\.\(c\)Overall performance\.## 1Introduction

Effectively transferring capabilities from strong models to weaker ones has become a central challenge in large language model post\-training\(Yang et al\.,[2025a](https://arxiv.org/html/2607.26057#bib.bib31); Xu et al\.,[2026](https://arxiv.org/html/2607.26057#bib.bib29); Zeng et al\.,[2026](https://arxiv.org/html/2607.26057#bib.bib35); Xiao et al\.,[2026](https://arxiv.org/html/2607.26057#bib.bib27)\)\. Unlike supervised fine\-tuning and offline knowledge distillation\(Hinton et al\.,[2015](https://arxiv.org/html/2607.26057#bib.bib11); Kim & Rush,[2016](https://arxiv.org/html/2607.26057#bib.bib15)\), which both rely on teacher\-generated data, on\-policy distillation \(OPD\)\(Agarwal et al\.,[2024](https://arxiv.org/html/2607.26057#bib.bib1); Lu & Lab,[2025](https://arxiv.org/html/2607.26057#bib.bib20)\)lets the student generate trajectories from its own policy\. The teacher then supplies dense token\-level guidance on the prefixes the student actually visits\. Because supervision is always grounded in the student’s own state distribution, OPD effectively mitigates the train–inference distribution shift, and has shown clear gains in strong\-to\-weak distillation\.

Yet the student’s own trajectories inevitably include its failures\. In long\-chain reasoning\(Jaech et al\.,[2024](https://arxiv.org/html/2607.26057#bib.bib13); Guo et al\.,[2025](https://arxiv.org/html/2607.26057#bib.bib8)\), this leads toprefix failure\(Li et al\.,[2026](https://arxiv.org/html/2607.26057#bib.bib18); Fu et al\.,[2026](https://arxiv.org/html/2607.26057#bib.bib6); Xie et al\.,[2026](https://arxiv.org/html/2607.26057#bib.bib28)\): once the student commits to a wrong direction early on, all subsequent generation builds on this deviation \(Figure[1](https://arxiv.org/html/2607.26057#S0.F1)\(b\)\)\. The resulting long misdirected continuations elicit unreliable and potentially harmful supervision, and they waste substantial training compute\.

Prior work addresses prefix failure from several angles, each with a structural limitation\. Fixed\-length truncation \(ESR, FastOPD\)\(Ziheng et al\.,[2026](https://arxiv.org/html/2607.26057#bib.bib38); Zhang et al\.,[2026](https://arxiv.org/html/2607.26057#bib.bib36)\)cuts the rollout at a rigid position regardless of where the reasoning actually fails\. Offline rewriting \(TRD\)\(Jiang et al\.,[2026](https://arxiv.org/html/2607.26057#bib.bib14)\)repairs trajectories only after the rollout completes, and the repairs often leave visible artifacts\. Token\-level mixing \(SKD\)\(Xu et al\.,[2025](https://arxiv.org/html/2607.26057#bib.bib30)\)switches between teacher and student on generic distributional disagreement, not on an explicit signal that the reasoning direction has failed\. What is missing is a mechanism that acts*online*, correcting failure as it emerges, and that decides*where*to intervene from the reasoning state itself\.

On a failed student prefix, the teacher and student diverge in how they continue \(Figure[1](https://arxiv.org/html/2607.26057#S0.F1)\(a\); the complete case appears in Appendix[E\.1](https://arxiv.org/html/2607.26057#A5.SS1)\): the teacher tends to stop, re\-examine the reasoning, and redirect, whereas the student tends to press on in the same wrong direction\. This divergence is observable during generation, without any label, verifier, or reward model, and it marks where the student’s reasoning has gone wrong\. We call such a position ahandoff trigger\. To find out how a trigger should be handled, we run trajectory intervention experiments: at each trigger the teacher takes over briefly, and we vary the duration and timing of the intervention\.

Two properties of this intervention stand out \(Figure[2](https://arxiv.org/html/2607.26057#S1.F2)\)\. First, correction can be remarkably*local*: replacing only the single reflection token at each trigger, so that teacher tokens are merely0\.35%0\.35\\%of all generated tokens, already lifts accuracy from 27\.73 to 34\.96 \(\+7\.23%; Figure[2\(a\)](https://arxiv.org/html/2607.26057#S1.F2.sf1)\), and teacher takeover likewise reduces the teacher–student gap \(Figure[2\(b\)](https://arxiv.org/html/2607.26057#S1.F2.sf2)\)\. Second, the intervention’s value is*front\-loaded*: holding the intervention length fixed but shifting it to later triggers instead of the earliest ones drops accuracy from 41\.99 to 33\.98 and then 29\.49\. The teacher–student gap explains why timing dominates\. It narrows as generation proceeds, whether the student runs alone or the teacher finishes the trajectory\. As the prefix grows, the teacher is pulled along by the student’s context, so a late takeover cannot redirect it\.

![Refer to caption](https://arxiv.org/html/2607.26057v1/x2.png)\(a\)
![Refer to caption](https://arxiv.org/html/2607.26057v1/x3.png)\(b\)

Figure 2:Preliminary Trajectory Intervention Experiments\.Qwen3\-4B\-Instruct\-2507 \(teacher\) and Qwen3\-1\.7B\-Non\-Thinking \(student\) on 128 DAPO\-Math\-17K English samples; the handoff criterion matches the main experiments\.\(a\)Accuracy \(mean@4\) and teacher token ratio as the number of teacher\-generated paragraphsLLvaries; a periodic delay ofrrskipsrrconsecutive valid triggers before the next takeover\.\(b\)Teacher–student gap by response\-token position under theL=3L\{=\}3intervention, computed as the absolute value of the mean token\-level advantage \(defined in §[3\.1](https://arxiv.org/html/2607.26057#S3.SS1)\) within each position interval\.A clear design principle emerges: prefix failure can be addressed through early, local teacher intervention at detected failure points, before the wrong direction becomes entrenched in the trajectory\. This leaves two questions for a training method:*when*should the teacher take over, and how do we keep these takeovers from pulling the trajectory too far from the student’s own policy?

We proposeRelay On\-Policy Distillation \(Relay\-OPD\), which interleaves student generation with brief teacher takeovers at the points where reasoning first goes wrong\. As the student rolls out its trajectory, Relay\-OPD monitors each prefix for a handoff trigger: a state where the teacher would redirect the reasoning while the student would continue on its current course\. When a trigger fires, the teacher generates a short*teacher leg*that steers the reasoning back and returns control to the student \(Figure[1](https://arxiv.org/html/2607.26057#S0.F1)\(b\)\)\. A*relay budget*caps the number and length of these legs, concentrating correction at the early positions where prefix failure originates while keeping the trajectory close to the student’s own policy\. The student is then distilled on the resulting relay trajectory, exactly as in standard on\-policy distillation\.

Our contributions are as follows: \(1\) We identify ateacher–student continuation asymmetryon failed reasoning prefixes and show, through trajectory intervention experiments, that correcting prefix failure needs only early and local teacher intervention, with teacher tokens as low as0\.35%0\.35\\%\. \(2\) We proposeRelay\-OPD, which turns this asymmetry into a label\-free handoff trigger and a budgeted relay rollout, and unifies student and teacher generation in a single speculative\-decoding engine\. \(3\) Across eight math reasoning benchmarks and two student scales, Relay\-OPD outperforms standard OPD by\+5\.73%\+5\.73\\%and FastOPD by\+1\.49%\+1\.49\\%for the 1\.7B student, while cutting average training trajectory length by over50%50\\%\.

## 2Related Work

### 2\.1On\-Policy Distillation

On\-policy distillation samples trajectories from the student, with the teacher scoring each token on the prefixes the student actually visits\(Agarwal et al\.,[2024](https://arxiv.org/html/2607.26057#bib.bib1); Gu et al\.,[2024](https://arxiv.org/html/2607.26057#bib.bib7)\); compared with offline training on teacher\-generated data\(Hinton et al\.,[2015](https://arxiv.org/html/2607.26057#bib.bib11); Kim & Rush,[2016](https://arxiv.org/html/2607.26057#bib.bib15)\), it grounds supervision in the student’s own state distribution, yields faster and stronger transfer, and is now standard in post\-training pipelines\(Yang et al\.,[2025a](https://arxiv.org/html/2607.26057#bib.bib31); Lu & Lab,[2025](https://arxiv.org/html/2607.26057#bib.bib20)\)\. Recent extensions cover distillation across model families\(Patiño et al\.,[2025](https://arxiv.org/html/2607.26057#bib.bib22)\), black\-box teachers without logit access\(Ye et al\.,[2025](https://arxiv.org/html/2607.26057#bib.bib33)\), and self\-distillation of a model’s own in\-context or privileged knowledge\(Yang et al\.,[2025b](https://arxiv.org/html/2607.26057#bib.bib32); Hübotter et al\.,[2026](https://arxiv.org/html/2607.26057#bib.bib12); Shenfeld et al\.,[2026](https://arxiv.org/html/2607.26057#bib.bib25); Zhao et al\.,[2026](https://arxiv.org/html/2607.26057#bib.bib37); Penaloza et al\.,[2026](https://arxiv.org/html/2607.26057#bib.bib23)\)\.

### 2\.2Prefix Failure

Grounding supervision entirely in student trajectories also imports the student’s failures\. Distillation gains shrink when teacher and student reasoning patterns are incompatible\(Li et al\.,[2026](https://arxiv.org/html/2607.26057#bib.bib18)\), and as student prefixes drift from teacher\-supported states, subsequent supervision grows unreliable\(Fu et al\.,[2026](https://arxiv.org/html/2607.26057#bib.bib6); Xie et al\.,[2026](https://arxiv.org/html/2607.26057#bib.bib28)\)\. These effects are most pronounced in long\-chain reasoning, where early directional deviations compound autoregressively into extended misdirected continuations\. Existing remedies intervene on the trajectory itself: ESR and FastOPD\(Ziheng et al\.,[2026](https://arxiv.org/html/2607.26057#bib.bib38); Zhang et al\.,[2026](https://arxiv.org/html/2607.26057#bib.bib36)\)truncate rollouts at a fixed length to discard late, low\-value supervision, TRD\(Jiang et al\.,[2026](https://arxiv.org/html/2607.26057#bib.bib14)\)rewrites student trajectories offline via the teacher, and SKD\(Xu et al\.,[2025](https://arxiv.org/html/2607.26057#bib.bib30)\)mixes teacher and student generation token by token according to distributional agreement\.

## 3Method

This section first reviews standard on\-policy distillation \(§[3\.1](https://arxiv.org/html/2607.26057#S3.SS1)\), then introduces the handoff trigger, relay trajectory construction, and optimization objective of Relay\-OPD \(§[3\.2](https://arxiv.org/html/2607.26057#S3.SS2)\), and finally describes its efficient implementation \(§[3\.3](https://arxiv.org/html/2607.26057#S3.SS3)\)\.

### 3\.1On\-Policy Distillation

Letx∼𝒟x\\sim\\mathcal\{D\}denote a training prompt, and letπθ\\pi\_\{\\theta\},πθ¯\\pi\_\{\\bar\{\\theta\}\}, andπT\\pi\_\{T\}denote the current student policy, old student policy, and teacher policy\. At each training iteration, standard OPD first generates an on\-policy trajectory using the old student policy:

y=\(y1,…,yN\)∼πθ¯\(⋅∣x\)\.y=\(y\_\{1\},\\ldots,y\_\{N\}\)\\sim\\pi\_\{\\bar\{\\theta\}\}\(\\cdot\\mid x\)\.\(1\)
Letht=\(x,y<t\)h\_\{t\}=\(x,y\_\{<t\}\)denote the student prefix at positiontt\. Standard OPD typically employs the reverse KL divergence\. Taking a single\-sample estimate\(Lu & Lab,[2025](https://arxiv.org/html/2607.26057#bib.bib20)\)at student\-sampled tokenyty\_\{t\}:

D^RKL\(t\)=log⁡πθ¯​\(yt∣ht\)−log⁡πT​\(yt∣ht\)\.\\widehat\{D\}\_\{\\mathrm\{RKL\}\}^\{\(t\)\}=\\log\\pi\_\{\\bar\{\\theta\}\}\(y\_\{t\}\\mid h\_\{t\}\)\-\\log\\pi\_\{T\}\(y\_\{t\}\\mid h\_\{t\}\)\.\(2\)
Its negation serves as the advantage:

AtOPD=−D^RKL\(t\)=log⁡πT​\(yt∣ht\)−log⁡πθ¯​\(yt∣ht\)\.A\_\{t\}^\{\\mathrm\{OPD\}\}=\-\\widehat\{D\}\_\{\\mathrm\{RKL\}\}^\{\(t\)\}=\\log\\pi\_\{T\}\(y\_\{t\}\\mid h\_\{t\}\)\-\\log\\pi\_\{\\bar\{\\theta\}\}\(y\_\{t\}\\mid h\_\{t\}\)\.\(3\)
A positive advantage encourages the student to increase the probability of the sampled token; a negative one decreases it\. Since training is entirely grounded in student\-visited prefixes, once prefix failure occurs early in generation, the resulting long misdirected continuation is still included in training despite receiving increasingly unreliable supervision\.

### 3\.2Relay\-OPD

![Refer to caption](https://arxiv.org/html/2607.26057v1/x4.png)Figure 3:Overview of Relay\-OPD\.Unlike standard OPD \(top\), which trains on student\-only trajectories that continue misdirected after prefix failure, Relay\-OPD constructs relay trajectories \(middle\): when the handoff trigger \(bottom\) detects that the teacher would redirect the reasoning while the student would not, the teacher briefly takes over before the student resumes\. The entire rollout runs in a single speculative decoding engine \(right; §[3\.3](https://arxiv.org/html/2607.26057#S3.SS3)\)\.Relay\-OPD introduces state\-driven teacher takeover into the student trajectories of standard OPD\. The method first detects handoff triggers based on teacher–student continuation tendencies on the current prefix, then constructs a relay trajectory comprising student legs and teacher legs, and performs distillation on the resulting trajectory\. Figure[3](https://arxiv.org/html/2607.26057#S3.F3)provides an overview\.

Handoff trigger\.Identifying prefix failure online requires a signal that depends on neither external verifiers nor process labels\. Rather than relying on generic teacher–student distributional differences, we focus on divergences in reasoning direction: takeover is triggered when the teacher tends to redirect while the student tends to continue along the current direction \(Figure[3](https://arxiv.org/html/2607.26057#S3.F3), bottom\)\.

Letℛ⊆𝒱\\mathcal\{R\}\\subseteq\\mathcal\{V\}denote a set of reflection tokens used for redirecting reasoning\(Guo et al\.,[2025](https://arxiv.org/html/2607.26057#bib.bib8); Muennighoff et al\.,[2025](https://arxiv.org/html/2607.26057#bib.bib21)\), such as Wait, But, and However, together with their case and leading\-space variants; the complete list is provided in Appendix[A\.1](https://arxiv.org/html/2607.26057#A1.SS1)\. Given a student prefixhh, define the teacher’s most probable next token as

aT​\(h\)=arg⁡maxv∈𝒱πT​\(v∣h\),a^\{T\}\(h\)=\\mathop\{\\arg\\max\}\_\{v\\in\\mathcal\{V\}\}\\pi\_\{T\}\(v\\mid h\),\(4\)
and the student’s top\-KKsupport set on this prefix as

𝒦S\(h\)=TopKK\(πθ¯\(⋅∣h\)\)\.\\mathcal\{K\}\_\{S\}\(h\)=\\operatorname\{TopK\}\_\{K\}\\\!\\left\(\\pi\_\{\\bar\{\\theta\}\}\(\\cdot\\mid h\)\\right\)\.\(5\)
When the teacher’s preferred token belongs toℛ\\mathcal\{R\}while the student’s top\-KKsupport set contains no token fromℛ\\mathcal\{R\}, we define the handoff criterion:

ϕ​\(h\)=𝟏​\[aT​\(h\)∈ℛ\]⋅𝟏​\[𝒦S​\(h\)∩ℛ=∅\]\.\\phi\(h\)=\\mathbf\{1\}\[a^\{T\}\(h\)\\in\\mathcal\{R\}\]\\cdot\\mathbf\{1\}\[\\mathcal\{K\}\_\{S\}\(h\)\\cap\\mathcal\{R\}=\\varnothing\]\.\(6\)
KKcontrols the divergence threshold required for triggering and thus the sensitivity of the criterion\.

Relay trajectory construction\.Two findings from the preliminary experiments shape the design\. First, the benefit of extending teacher takeover saturates \(Figure[2\(a\)](https://arxiv.org/html/2607.26057#S1.F2.sf1)\): fromL=3L=3toL=6L=6, accuracy plateaus around 41–44 despite the teacher token ratio rising from 17\.52% to 28\.52%, motivating a limited teacher leg\. Second, valuable supervision concentrates early in the trajectory \(Figure[2\(b\)](https://arxiv.org/html/2607.26057#S1.F2.sf2)\), motivating a relay budget that caps the total number of takeovers and focuses intervention on earlier positions\.

The relay budget\(M,L\)\(M,L\)specifies the maximum number of teacher takeoversMMand the number of additional paragraphsLLgenerated after the reflection token in each teacher leg, with paragraphs delimited by\\n\\n\. We measure teacher legs in paragraphs rather than a fixed number of tokens so that each leg ends at a structurally complete reasoning unit instead of breaking off mid\-thought; in our setting a paragraph contains 23\.2 tokens on average\. The student generates the student leg autoregressively according toπθ¯\\pi\_\{\\bar\{\\theta\}\}, computingϕ​\(h\)\\phi\(h\)at each position\. Whenϕ​\(h\)=1\\phi\(h\)=1and the takeover count has not reachedMM, the teacher takes over:aT​\(h\)a^\{T\}\(h\)from the trigger criterion serves as the starting token of the teacher leg, and generation continues forLLparagraphs\. WhenL=0L=0, the segment consists solely of the reflection token\. After the teacher leg, if budget remains, the student resumes generation from the extended prefix; when theMM\-th teacher leg ends, the current rollout terminates\. This yields the relay trajectoryz=\(z1,…,zN\)z=\(z\_\{1\},\\ldots,z\_\{N\}\)\.

Unlike fixed\-length truncation, both the intervention and termination positions of Relay\-OPD are determined by the current reasoning state\.

Optimization objective\.Figure[2\(a\)](https://arxiv.org/html/2607.26057#S1.F2.sf1)shows that although teacher takeover significantly reduces prefix failure, even atL=6L=6with a teacher token ratio of 28\.52%, accuracy reaches only 43\.55 compared with 60\.55 for the teacher generating independently\. We therefore posit that the training signal the teacher provides on relay trajectories differs from its ideal supervision on its own trajectories\. Accordingly, we adopt a reverse\-KL\-style single\-sample objective that optimizes directly on the observed tokens in the relay trajectory, enabling the student to selectively absorb the teacher’s corrective signals rather than fully fitting the teacher distribution via forward KL\.

At positiontt, lethtz=\(x,z<t\)h\_\{t\}^\{z\}=\(x,z\_\{<t\}\)denote the relay prefix\. The advantage for the actually generated tokenztz\_\{t\}is

AtRelay=log⁡πT​\(zt∣htz\)−log⁡πθ¯​\(zt∣htz\)\.A\_\{t\}^\{\\mathrm\{Relay\}\}=\\log\\pi\_\{T\}\(z\_\{t\}\\mid h\_\{t\}^\{z\}\)\-\\log\\pi\_\{\\bar\{\\theta\}\}\(z\_\{t\}\\mid h\_\{t\}^\{z\}\)\.\(7\)
The update ratio of the current student relative to the old student is

ρt​\(θ\)=πθ​\(zt∣htz\)πθ¯​\(zt∣htz\)\.\\rho\_\{t\}\(\\theta\)=\\frac\{\\pi\_\{\\theta\}\(z\_\{t\}\\mid h\_\{t\}^\{z\}\)\}\{\\pi\_\{\\bar\{\\theta\}\}\(z\_\{t\}\\mid h\_\{t\}^\{z\}\)\}\.\(8\)
Relay\-OPD optimizes:

ℒRelay=−𝔼x,z​\[1N​∑t=1Nmin⁡\(ρt​AtRelay,clip⁡\(ρt,1−ϵ,1\+ϵ\)​AtRelay\)\]\.\\mathcal\{L\}\_\{\\mathrm\{Relay\}\}=\-\\mathbb\{E\}\_\{x,\\,z\}\\left\[\\frac\{1\}\{N\}\\sum\_\{t=1\}^\{N\}\\min\\left\(\\rho\_\{t\}A\_\{t\}^\{\\mathrm\{Relay\}\},\\operatorname\{clip\}\(\\rho\_\{t\},1\-\\epsilon,1\+\\epsilon\)A\_\{t\}^\{\\mathrm\{Relay\}\}\\right\)\\right\]\.\(9\)
This objective uses the actually generated tokensztz\_\{t\}across the entire relay trajectory\. Teacher legs both provide corrected context for subsequent student legs and directly participate in optimization through their generated tokens\.

Whenϕ​\(h\)≡0\\phi\(h\)\\equiv 0, the relay trajectory degenerates to a standard student trajectory; whenL=0L=0, the teacher leg contains only the replacement token at the handoff trigger position, corresponding to minimal prefix correction\. The complete algorithm is given in Appendix[C\.1](https://arxiv.org/html/2607.26057#A3.SS1)\.

### 3\.3Efficient Implementation

Relay\-OPD requires continuously determining whether to trigger teacher takeover during generation and alternating between student and teacher legs\. Maintaining two independent generation pipelines would necessitate repeated coordination between teacher and student models for takeover timing, plus switching generation engines at takeover and recovery points, introducing substantial scheduling and communication overhead\. We instead unify the entire Relay\-OPD generation process within a single speculative decoding engine\(Leviathan et al\.,[2023](https://arxiv.org/html/2607.26057#bib.bib17); Chen et al\.,[2023](https://arxiv.org/html/2607.26057#bib.bib5)\): the student serves as draft model and the teacher as target model \(Figure[3](https://arxiv.org/html/2607.26057#S3.F3), right\)\.

Relay as a state\-switched decoding process\.The engine produces the relay trajectoryzzof §[3\.2](https://arxiv.org/html/2607.26057#S3.SS2)one position at a time: at each positiontt, it first determines a decoding statest∈\{𝖲,𝖳,⊥\}s\_\{t\}\\in\\\{\\mathsf\{S\},\\mathsf\{T\},\\bot\\\}from the current prefixhtz=\(x,z<t\)h\_\{t\}^\{z\}=\(x,z\_\{<t\}\), and then emitsztz\_\{t\}in a student leg \(𝖲\\mathsf\{S\}\) or a teacher leg \(𝖳\\mathsf\{T\}\), or terminates \(⊥\\bot\)\. Letjtj\_\{t\}denote the number of teacher takeovers andℓt\\ell\_\{t\}the number of completed paragraphs \(delimiter\\n\\n\) in the current teacher leg, both up to positiontt\. Starting froms1=𝖲s\_\{1\}=\\mathsf\{S\}, the state for the next position is determined by

st\+1=\{𝖳,if​st=𝖲,ϕ​\(ht\+1z\)=1,and​jt<M,𝖲,if​st=𝖳,ℓt=L,and​jt<M,⊥,if​st=𝖳,ℓt=L,and​jt=M,st,otherwise,s\_\{t\+1\}=\\begin\{cases\}\\mathsf\{T\},&\\text\{if \}s\_\{t\}=\\mathsf\{S\},\\ \\phi\(h\_\{t\+1\}^\{z\}\)=1,\\ \\text\{and \}j\_\{t\}<M,\\\\\[2\.0pt\] \\mathsf\{S\},&\\text\{if \}s\_\{t\}=\\mathsf\{T\},\\ \\ell\_\{t\}=L,\\ \\text\{and \}j\_\{t\}<M,\\\\\[2\.0pt\] \\bot,&\\text\{if \}s\_\{t\}=\\mathsf\{T\},\\ \\ell\_\{t\}=L,\\ \\text\{and \}j\_\{t\}=M,\\\\\[2\.0pt\] s\_\{t\},&\\text\{otherwise\},\\end\{cases\}\(10\)and additionally enters the absorbing state⊥\\botwheneverztz\_\{t\}is the end\-of\-sequence token or the length limit is reached\. The first case realizes the handoff trigger: a trigger detected on the prefixht\+1zh\_\{t\+1\}^\{z\}switches the state to𝖳\\mathsf\{T\}, so the teacher takes over at positiont\+1t\+1\. WhenL=0L=0, the exit conditionℓt=L\\ell\_\{t\}=Lalready holds at the leg\-initial position, so the teacher leg consists of exactly the reflection token, recovering the minimal\-correction case of §[3\.2](https://arxiv.org/html/2607.26057#S3.SS2)\.

Unified token generation\.Every position is generated by speculative decoding against a state\-dependent target policy:πttgt=πθ¯\\pi^\{\\mathrm\{tgt\}\}\_\{t\}=\\pi\_\{\\bar\{\\theta\}\}whenst=𝖲s\_\{t\}=\\mathsf\{S\}, andπttgt=πT\\pi^\{\\mathrm\{tgt\}\}\_\{t\}=\\pi\_\{T\}whenst=𝖳s\_\{t\}=\\mathsf\{T\}\. In student legs every draft is accepted and generation reduces to ordinary student decoding, while teacher legs perform standard speculative decoding against the teacher\. The single exception is the leg\-initial position of each teacher leg, where the engine directly emits the trigger tokenzt=aT​\(htz\)z\_\{t\}=a^\{T\}\(h\_\{t\}^\{z\}\)without verification\. At every other position, the student draftsatS∼πθ¯\(⋅∣htz\)a\_\{t\}^\{S\}\\sim\\pi\_\{\\bar\{\\theta\}\}\(\\cdot\\mid h\_\{t\}^\{z\}\), which is accepted with probability

αt=min⁡\(1,πttgt​\(atS∣htz\)πθ¯​\(atS∣htz\)\);\\alpha\_\{t\}=\\min\\\!\\left\(1,\\;\\frac\{\\pi^\{\\mathrm\{tgt\}\}\_\{t\}\(a\_\{t\}^\{S\}\\mid h\_\{t\}^\{z\}\)\}\{\\pi\_\{\\bar\{\\theta\}\}\(a\_\{t\}^\{S\}\\mid h\_\{t\}^\{z\}\)\}\\right\);\(11\)if an i\.i\.d\. drawut∼𝒰​\(0,1\)u\_\{t\}\\sim\\mathcal\{U\}\(0,1\)exceedsαt\\alpha\_\{t\}, the draft is rejected andztz\_\{t\}is instead sampled from the residual distribution

qt​\(v\)=\[πttgt​\(v∣htz\)−πθ¯​\(v∣htz\)\]\+∑w∈𝒱\[πttgt​\(w∣htz\)−πθ¯​\(w∣htz\)\]\+,\[x\]\+=max⁡\(x,0\)\.q\_\{t\}\(v\)=\\frac\{\\bigl\[\\pi^\{\\mathrm\{tgt\}\}\_\{t\}\(v\\mid h\_\{t\}^\{z\}\)\-\\pi\_\{\\bar\{\\theta\}\}\(v\\mid h\_\{t\}^\{z\}\)\\bigr\]\_\{\+\}\}\{\\sum\_\{w\\in\\mathcal\{V\}\}\\bigl\[\\pi^\{\\mathrm\{tgt\}\}\_\{t\}\(w\\mid h\_\{t\}^\{z\}\)\-\\pi\_\{\\bar\{\\theta\}\}\(w\\mid h\_\{t\}^\{z\}\)\\bigr\]\_\{\+\}\},\\qquad\[x\]\_\{\+\}=\\max\(x,0\)\.\(12\)In student legs, substitutingπttgt=πθ¯\\pi^\{\\mathrm\{tgt\}\}\_\{t\}=\\pi\_\{\\bar\{\\theta\}\}into Eq\. equation[11](https://arxiv.org/html/2607.26057#S3.E11)givesαt≡1\\alpha\_\{t\}\\equiv 1, so drafts are accepted unconditionally; in teacher legs, Eqs\. equation[11](https://arxiv.org/html/2607.26057#S3.E11)–equation[12](https://arxiv.org/html/2607.26057#S3.E12)are exactly standard speculative rejection sampling againstπT\\pi\_\{T\}\.

Exactness and efficiency\.By the correctness of speculative sampling\(Leviathan et al\.,[2023](https://arxiv.org/html/2607.26057#bib.bib17)\), every verified position satisfieszt∼πttgt\(⋅∣htz\)z\_\{t\}\\sim\\pi^\{\\mathrm\{tgt\}\}\_\{t\}\(\\cdot\\mid h\_\{t\}^\{z\}\)\. Hence, conditioned on the prefix and the deterministic leg\-initial token, teacher legs are distributionally identical to the teacher continuing generation directly from the extended prefix, and student legs to ordinary student sampling: the single\-engine implementation reproduces the two\-model relay process of §[3\.2](https://arxiv.org/html/2607.26057#S3.SS2)exactly, without switching between two generation pipelines\. Efficiency\-wise, teacher legs batch\-verify student drafts instead of decoding serially, and the teacher logits computed during verification simultaneously provideaT​\(htz\)a^\{T\}\(h\_\{t\}^\{z\}\)and the trigger criterionϕ​\(htz\)\\phi\(h\_\{t\}^\{z\}\)at no additional cost\. In practice, Eq\. equation[10](https://arxiv.org/html/2607.26057#S3.E10)is applied during block verification: draft tokens following the first transition point within a block are discarded, so the per\-token semantics above are preserved exactly\.

After each parameter update, the latest student weights are synchronized to the draft model; teacher parameters remain frozen throughout training\.

## 4Experiments

### 4\.1Experimental Setup

Models, data, and benchmarks\.We use Qwen3\-4B\-Instruct\-2507\(Yang et al\.,[2025a](https://arxiv.org/html/2607.26057#bib.bib31)\)as teacher and Qwen3\-0\.6B\-Non\-Thinking and Qwen3\-1\.7B\-Non\-Thinking as students\. Training data is the English subset of DAPO\-Math\-17K\(Yu et al\.,[2026](https://arxiv.org/html/2607.26057#bib.bib34)\)\. Evaluation covers eight mathematical reasoning benchmarks: AIME 2024\(AI\-MO,[2024a](https://arxiv.org/html/2607.26057#bib.bib2)\), AIME 2025, AIME 2026, MATH500\(Hendrycks et al\.,[2021](https://arxiv.org/html/2607.26057#bib.bib10); Lightman et al\.,[2024](https://arxiv.org/html/2607.26057#bib.bib19)\), AMC 2023\(AI\-MO,[2024b](https://arxiv.org/html/2607.26057#bib.bib3)\), OlympiadBench\(He et al\.,[2024](https://arxiv.org/html/2607.26057#bib.bib9)\), HMMT February 2026\(Balunović et al\.,[2025](https://arxiv.org/html/2607.26057#bib.bib4)\), and HMMT November 2025\. At evaluation time, we set temperature to 1\.0 and top\-ppto 1\.0, with a maximum generation length of 32,768 tokens\. Each problem in AIME, AMC, and HMMT is sampled 32 times; each problem in MATH500 and OlympiadBench is sampled 4 times\. We report mean accuracy\.

Implementation details\.All methods are implemented on verl\(Sheng et al\.,[2025](https://arxiv.org/html/2607.26057#bib.bib26)\)and vLLM 0\.21\.0\(Kwon et al\.,[2023](https://arxiv.org/html/2607.26057#bib.bib16)\), trained on 8 H100 GPUs\. Online distillation methods train for 1 epoch with a maximum response length of 16,384 tokens\. Trajectory sampling uses temperature 1\.0 and top\-p=1\.0p=1\.0\. Relay\-OPD setsK=5K=5and\(M,L\)=\(2,3\)\(M,L\)=\(2,3\), without using any external verifier, process supervision labels, or answer correctness labels\. Complete hyperparameters are provided in Appendix[A\.2](https://arxiv.org/html/2607.26057#A1.SS2), and the training and inference prompt template in Appendix[B\.1](https://arxiv.org/html/2607.26057#A2.SS1)\.

Baselines\.We compare Relay\-OPD against three categories of methods\.

1. 1\.Reference baselines\.The untrained initial student, supervised fine\-tuning \(SFT\), token\-level knowledge distillation \(KD\)\(Hinton et al\.,[2015](https://arxiv.org/html/2607.26057#bib.bib11)\), and the outcome\-reward reinforcement learning method GRPO\(Shao et al\.,[2024](https://arxiv.org/html/2607.26057#bib.bib24)\)\.
2. 2\.On\-policy distillation\.Standard OPD\(Lu & Lab,[2025](https://arxiv.org/html/2607.26057#bib.bib20)\), which performs token\-level distillation on student\-generated trajectories\.
3. 3\.Trajectory intervention methods\.TRD\(Jiang et al\.,[2026](https://arxiv.org/html/2607.26057#bib.bib14)\), which rewrites student trajectories offline via the teacher; FastOPD\(Ziheng et al\.,[2026](https://arxiv.org/html/2607.26057#bib.bib38); Zhang et al\.,[2026](https://arxiv.org/html/2607.26057#bib.bib36)\), which shortens the rollout budget; and SKD\(Xu et al\.,[2025](https://arxiv.org/html/2607.26057#bib.bib30)\), which mixes teacher and student generation through speculative decoding\. For FastOPD, we evaluate fixed truncation lengths in\{1024,2048,4096,8192\}\\\{1024,2048,4096,8192\\\}; Table[1](https://arxiv.org/html/2607.26057#S4.T1)reports the best configuration at 4,096 tokens, with complete results in Appendix[D\.2](https://arxiv.org/html/2607.26057#A4.SS2)\.

Implementation details for each method are provided in Appendix[C](https://arxiv.org/html/2607.26057#A3), with a structured overview in Table[5](https://arxiv.org/html/2607.26057#A3.T5)\.

### 4\.2Main Results

Table 1:Main Experimental Results\.Mean accuracy is reported; bold and underline denote the best and second\-best results for each student model\. Subscripts in the Avg column indicate the training step of the best checkpoint\. Train Len denotes the average rollout response length from the start of training to the best checkpoint\.MethodAIME24AIME25AIME26MATHAMC23Olymp\.HMMTFeb26HMMTNov25AvgTrainLenTeacher60\.4246\.0452\.1994\.2093\.8370\.6231\.2541\.8861\.30—Student: Qwen3\-0\.6B\-Non\-ThinkingStudent1\.772\.400\.7344\.1024\.4516\.360\.763\.8511\.80—SFT4\.907\.604\.0659\.4534\.9226\.741\.893\.0217\.82@110\{\}\_\{\\text\{@110\}\}4262KD4\.177\.194\.7957\.7535\.2327\.152\.942\.7117\.74@110\{\}\_\{\\text\{@110\}\}4262GRPO8\.2315\.2110\.0068\.6046\.5635\.427\.866\.0424\.74@110\{\}\_\{\\text\{@110\}\}3379OPD13\.4417\.9211\.9875\.3051\.2541\.327\.395\.6228\.03@110\{\}\_\{\\text\{@110\}\}6900TRD4\.588\.543\.9657\.6036\.3326\.933\.792\.8118\.07@20\{\}\_\{\\text\{@20\}\}3275FastOPD15\.8320\.1013\.5475\.3053\.6744\.8111\.558\.5430\.42@100\{\}\_\{\\text\{@100\}\}3302SKD8\.4416\.989\.7966\.6543\.6735\.768\.145\.6224\.38@20\{\}\_\{\\text\{@20\}\}5800Relay\-OPD15\.9420\.9414\.0676\.8055\.5545\.0311\.178\.8531\.04@75\{\}\_\{\\text\{@75\}\}2490Δ\\Deltavs OPD\+2\.50\+3\.02\+2\.08\+1\.50\+4\.30\+3\.71\+3\.78\+3\.23\+3\.01\-63\.9%Student: Qwen3\-1\.7B\-Non\-ThinkingStudent12\.609\.587\.4071\.9547\.8938\.546\.344\.3824\.84—SFT23\.3319\.4816\.1581\.4059\.4546\.6212\.596\.5633\.20@110\{\}\_\{\\text\{@110\}\}4262KD23\.5421\.1515\.3181\.4560\.2348\.0712\.787\.5033\.75@110\{\}\_\{\\text\{@110\}\}4262GRPO24\.5822\.0815\.6280\.3560\.1648\.7414\.499\.3834\.42@105\{\}\_\{\\text\{@105\}\}2558OPD35\.8325\.5223\.3385\.7070\.0855\.2720\.0814\.0641\.23@55\{\}\_\{\\text\{@55\}\}4658TRD19\.2719\.6912\.7177\.7055\.4744\.1811\.934\.5830\.69@40\{\}\_\{\\text\{@40\}\}2785FastOPD42\.2930\.4226\.3587\.9574\.3058\.1623\.5820\.7345\.47@45\{\}\_\{\\text\{@45\}\}2709SKD33\.1230\.7328\.8587\.3572\.4254\.4120\.0811\.8842\.35@35\{\}\_\{\\text\{@35\}\}4753Relay\-OPD42\.7132\.8130\.5289\.5076\.8858\.7924\.7219\.7946\.96@35\{\}\_\{\\text\{@35\}\}2296Δ\\Deltavs OPD\+6\.88\+7\.29\+7\.19\+3\.80\+6\.80\+3\.52\+4\.64\+5\.73\+5\.73\-50\.7%

![Refer to caption](https://arxiv.org/html/2607.26057v1/x5.png)Figure 4:Pass@kkperformance of Relay\-OPD and OPD on HMMT Feb26 and HMMT Nov25\.
![Refer to caption](https://arxiv.org/html/2607.26057v1/x6.png)Figure 5:Inference response length of Relay\-OPD vs\. FastOPD\.

Overall performance\.As summarized in Table[1](https://arxiv.org/html/2607.26057#S4.T1), Relay\-OPD achieves the best or second\-best results across all eight benchmarks for both student models\. For Qwen3\-1\.7B\-Non\-Thinking, it attains an average accuracy of 46\.96, outperforming standard OPD by \+5\.73% and the strongest trajectory intervention baseline FastOPD by \+1\.49%; AIME 2025 and AIME 2026 improve over OPD by \+7\.29% and \+7\.19%\. Qwen3\-0\.6B\-Non\-Thinking exhibits a consistent trend: average accuracy improves by \+3\.01% over OPD and surpasses FastOPD by \+0\.62%\. Relay\-OPD further achieves significantly higher pass@kkthan standard OPD across different sampling budgets \(Figure[5](https://arxiv.org/html/2607.26057#S4.F5)\)\.

Comparison with trajectory intervention baselines\.TRD exhibits marked performance degradation relative to standard OPD on both student models: 30\.69 vs\. 41\.23 on 1\.7B and 18\.07 vs\. 28\.03 on 0\.6B\. We observe that its rewritten trajectories often carry visible rewriting artifacts rather than resembling naturally unfolding problem\-solving; examples appear in Appendix[E\.2](https://arxiv.org/html/2607.26057#A5.SS2)\. SKD only marginally outperforms OPD on 1\.7B \(42\.35 vs\. 41\.23\) and drops to 24\.38 on 0\.6B; it struggles to break established repetitive generation patterns \(Appendix[E\.3](https://arxiv.org/html/2607.26057#A5.SS3)\)\. FastOPD concentrates training signals at the sequence front via fixed truncation but cannot provide demonstrations of how to recover from failed prefixes\. In contrast, Figure[5](https://arxiv.org/html/2607.26057#S4.F5)shows that Relay\-OPD reduces mean response length by 17\.9%, 14\.2%, and 28\.3% on AIME 2025, AIME 2026, and HMMT February 2026, respectively, compared to FastOPD, while simultaneously improving accuracy by \+2\.39%, \+4\.17%, and \+1\.14%\. Relay\-OPD teaches the student to redirect reasoning before deviations compound further, yielding more accurate answers with shorter reasoning processes\.

Training token efficiency\.Relay\-OPD uses shorter training trajectories and reaches the best checkpoint in fewer update steps\. For the 1\.7B student, Relay\-OPD reaches its optimum at step 35, earlier than OPD at step 55 and FastOPD at step 45; its average rollout response length is 2,296 tokens, a 50\.7% reduction from OPD’s 4,658 and shorter than FastOPD’s 2,709\. A consistent trend holds for the 0\.6B student: Relay\-OPD’s average training trajectory length is 2,490 tokens, a 63\.9% reduction from OPD\.

### 4\.3Training Dynamics

![Refer to caption](https://arxiv.org/html/2607.26057v1/x7.png)Figure 6:Training Dynamics of Relay\-OPD\.Using Qwen3\-4B\-Instruct\-2507 as teacher and Qwen3\-1\.7B\-Non\-Thinking as student\.Left:fraction of trajectories exhausting the relay budget;Middle:teacher token ratio relative to all effective response tokens;Right:policy entropy of Relay\-OPD, OPD, and FastOPD\.To elucidate how Relay\-OPD adjusts teacher intervention as the student policy evolves, we track three metrics over the first 60 training steps in Figure[6](https://arxiv.org/html/2607.26057#S4.F6)\. The left panel shows that the fraction of trajectories exhausting the relay budget gradually decreases, from roughly 75%–85% early on to roughly 50%–60%\. The middle panel shows that teacher token ratio starts at roughly 13%, then drops rapidly and stabilizes at 2%–3% after about 20 steps\. Both trends indicate that the teacher–student gap and the prefix failures requiring intervention shrink during training\.

The right panel compares policy entropy across the three methods\. Relay\-OPD maintains consistently higher entropy than OPD and FastOPD over the first 60 steps\. We attribute this to teacher intervention altering the prefixes the student visits and the subsequent generation directions, thereby increasing student exploration\.

### 4\.4Ablation Studies

Role of the teacher leg\.The benefit of the teacher leg stems from corrected context and local reasoning demonstrations, not merely early termination\. To isolate this contribution, we set the maximum number of takeovers toM=1M=1for both variants: Trigger\-stop terminates immediately at the first trigger without generating any teacher tokens; Relay\-OPD generates a teacher leg of lengthL=3L=3before terminating\.

As shown in Table[2](https://arxiv.org/html/2607.26057#S4.T2), adding the teacher leg improves average accuracy from 43\.48 to 46\.25 \(\+2\.77% over Trigger\-stop\), confirming that corrected context and local reasoning demonstrations yield benefits beyond dynamic truncation alone\.

Table 2:Teacher\-Leg Ablations\.Using Qwen3\-1\.7B\-Non\-Thinking as student; all other settings are identical to the main experiments\.Top:teacher leg vs\. trigger\-stop, where both variants setM=1M=1\.Bottom:training objective for the teacher leg\.VariantAIME24AIME25AIME26MATHAMC23Olymp\.HMMTFeb26HMMTNov25AvgTeacher leg vs\. trigger\-stopTrigger\-Stop \(M=1M=1, No Teacher Leg\)39\.4828\.7524\.4887\.0071\.5655\.2723\.1118\.2343\.48Relay\-OPD\(M=1M=1,L=3L=3\)42\.4031\.9829\.6988\.5575\.4758\.3823\.3020\.2146\.25Training objective for the teacher legStudent Draft Token38\.5427\.6026\.2588\.1075\.9457\.4223\.6718\.9644\.56Teacher FKL \(k=128k=128\)41\.7728\.9626\.5688\.2072\.7355\.6821\.6917\.0844\.08Relay Token \(Ours\)42\.7132\.8130\.5289\.5076\.8858\.7924\.7219\.7946\.96

Training objective for the teacher leg\.We compare two alternative objectives for the teacher leg, keeping the student leg under the standard OPD objective throughout\. To support this ablation, the engine \(§[3\.3](https://arxiv.org/html/2607.26057#S3.SS3)\) saves position markers for teacher legs together with the student’s draft tokenatSa\_\{t\}^\{S\}on prefixhtzh\_\{t\}^\{z\}at each teacher\-leg position, before the teacher verifies or replaces it\. The Student draft token variant uses the engine\-savedatSa\_\{t\}^\{S\}and replaces the default advantage with

Atdraft=log⁡πT​\(atS∣htz\)−log⁡πθ¯​\(atS∣htz\)\.A\_\{t\}^\{\\mathrm\{draft\}\}=\\log\\pi\_\{T\}\(a\_\{t\}^\{S\}\\mid h\_\{t\}^\{z\}\)\-\\log\\pi\_\{\\bar\{\\theta\}\}\(a\_\{t\}^\{S\}\\mid h\_\{t\}^\{z\}\)\.\(13\)
That is, the policy gradient loss and parameter update at teacher\-leg positions are computed with respect to the student’s draft tokenatSa\_\{t\}^\{S\}rather than the actually generated relay tokenztz\_\{t\}\.

The Teacher FKL variant takes the teacher’s top\-kksupport set𝒦t=TopKk\(πT\(⋅∣htz\)\)\\mathcal\{K\}\_\{t\}=\\operatorname\{TopK\}\_\{k\}\\\!\\left\(\\pi\_\{T\}\(\\cdot\\mid h\_\{t\}^\{z\}\)\\right\)withk=128k=128\. The teacher probability is renormalized over𝒦t\\mathcal\{K\}\_\{t\}as

qTk​\(v∣htz\)=πT​\(v∣htz\)∑u∈𝒦tπT​\(u∣htz\),v∈𝒦t\.q\_\{T\}^\{k\}\(v\\mid h\_\{t\}^\{z\}\)=\\frac\{\\pi\_\{T\}\(v\\mid h\_\{t\}^\{z\}\)\}\{\\sum\_\{u\\in\\mathcal\{K\}\_\{t\}\}\\pi\_\{T\}\(u\\mid h\_\{t\}^\{z\}\)\},\\qquad v\\in\\mathcal\{K\}\_\{t\}\.\(14\)
The corresponding Teacher FKL objective is

ℒFKLteacher​\(t\)=∑v∈𝒦tqTk​\(v∣htz\)​\[log⁡qTk​\(v∣htz\)−log⁡πθ​\(v∣htz\)\]\.\\mathcal\{L\}\_\{\\mathrm\{FKL\}\}^\{\\mathrm\{teacher\}\}\(t\)=\\sum\_\{v\\in\\mathcal\{K\}\_\{t\}\}q\_\{T\}^\{k\}\(v\\mid h\_\{t\}^\{z\}\)\\left\[\\log q\_\{T\}^\{k\}\(v\\mid h\_\{t\}^\{z\}\)\-\\log\\pi\_\{\\theta\}\(v\\mid h\_\{t\}^\{z\}\)\\right\]\.\(15\)
As shown in Table[2](https://arxiv.org/html/2607.26057#S4.T2), the relay token objective \(46\.96\) substantially outperforms Student draft token \(44\.56\) and Teacher FKL \(44\.08\)\. The decline with Teacher FKL aligns perfectly with the design rationale in §[3\.2](https://arxiv.org/html/2607.26057#S3.SS2): the reverse\-KL\-style single\-sample objective is inherently mode\-seeking, enabling the student to selectively absorb the teacher’s corrective signals on failed prefixes\. In contrast, the mode\-covering nature of Teacher FKL forces the student to match the teacher’s full distribution, indiscriminately incorporating unreliable guidance\. Student draft token predominantly suppresses tokens the student originally tended to generate, whereas using the actually generated tokens after takeover provides a more explicit learning signal\.

### 4\.5Sensitivity Analysis

Moderate intervention balances teacher correction against the on\-policy property of the trajectory\.Figure[7](https://arxiv.org/html/2607.26057#S4.F7)summarizes average accuracy when varying one hyperparameter at a time; complete per\-benchmark results appear in Appendix[D\.1](https://arxiv.org/html/2607.26057#A4.SS1)\.

![Refer to caption](https://arxiv.org/html/2607.26057v1/x8.png)Figure 7:Hyperparameter Sensitivity of Relay\-OPD\.Using Qwen3\-4B\-Instruct\-2507 as teacher and Qwen3\-1\.7B\-Non\-Thinking as student\.Left:relay budget\(M,L\)\(M,L\);Right:handoff top\-KK, where\|𝒱\|\|\\mathcal\{V\}\|denotes the vocabulary size\.Relay budget\(M,L\)\(M,L\)\.As shown in the left panel, average accuracy improves from 44\.31 to 47\.10 asLLincreases from 0 to 4, then drops to 46\.47 atL=5L=5; bothL=3L=3andL=4L=4achieve strong performance\. ForMM,M=1M=1andM=2M=2achieve 46\.25 and 46\.96, whereasM=4M=4drops to 44\.01\. Figure[2\(a\)](https://arxiv.org/html/2607.26057#S1.F2.sf1)already showed that extending teacher takeover yields diminishing returns; training results here further confirm that overly long or frequent teacher takeovers push the trajectory too far from the student’s current policy, undermining the advantages of on\-policy distillation\.

Handoff top\-KK\.As shown in the right panel,K=1K=1,K=5K=5, andK=10K=10achieve average accuracies of 44\.27, 46\.96, and 43\.14—all exceeding the 41\.23 of standard OPD \(equivalent toK=\|𝒱\|K=\|\\mathcal\{V\}\|\), validating state\-driven selective intervention\.K=5K=5achieves the best result, indicating that the divergence thresholdKKshould be neither too small \(over\-triggering on minor differences\) nor too large \(missing genuine divergences\)\.

## 5Conclusion

We introduced Relay\-OPD, which addresses the prefix failure problem in on\-policy distillation by detecting teacher–student reasoning\-direction divergences online and letting the teacher intervene locally at detected failure points\. A limited relay budget concentrates intervention on critical early positions while limiting departure from the student policy\. Across two Qwen3 student models and eight mathematical reasoning benchmarks, Relay\-OPD outperforms standard OPD by \+5\.73% and the strongest trajectory intervention baseline by \+1\.49% on average, while reducing training trajectory length by over 50%\. Further ablations and sensitivity analyses validate the contribution of each design component\.

## References

- Agarwal et al\. \(2024\)Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem\.On\-policy distillation of language models: Learning from self\-generated mistakes\.In*International Conference on Learning Representations*, volume 2024, pp\. 21246–21263, 2024\.
- AI\-MO \(2024a\)AI\-MO\.Aime 2024\.[https://huggingface\.co/datasets/AI\-MO/aimo\-validation\-aime](https://huggingface.co/datasets/AI-MO/aimo-validation-aime), 2024a\.
- AI\-MO \(2024b\)AI\-MO\.Amc 2023\.[https://huggingface\.co/datasets/AI\-MO/aimo\-validation\-amc](https://huggingface.co/datasets/AI-MO/aimo-validation-amc), 2024b\.
- Balunović et al\. \(2025\)Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev\.Matharena: Evaluating llms on uncontaminated math competitions, February 2025\.URL[https://matharena\.ai/](https://matharena.ai/)\.
- Chen et al\. \(2023\)Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean\-Baptiste Lespiau, Laurent Sifre, and John Jumper\.Accelerating large language model decoding with speculative sampling\.*arXiv preprint arXiv:2302\.01318*, 2023\.
- Fu et al\. \(2026\)Yuqian Fu, Haohuan Huang, Kaiwen Jiang, Jiacai Liu, Zhuo Jiang, Yuanheng Zhu, and Dongbin Zhao\.Revisiting on\-policy distillation: Empirical failure modes and simple fixes\.*arXiv preprint arXiv:2603\.25562*, 2026\.
- Gu et al\. \(2024\)Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang\.Minillm: Knowledge distillation of large language models\.In*International Conference on Learning Representations*, volume 2024, pp\. 32694–32717, 2024\.
- Guo et al\. \(2025\)Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al\.Deepseek\-r1: Incentivizing reasoning capability in llms via reinforcement learning\.*arXiv preprint arXiv:2501\.12948*, 2025\.
- He et al\. \(2024\)Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al\.Olympiadbench: A challenging benchmark for promoting agi with olympiad\-level bilingual multimodal scientific problems\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 3828–3850, 2024\.
- Hendrycks et al\. \(2021\)Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt\.Measuring mathematical problem solving with the math dataset\.*arXiv preprint arXiv:2103\.03874*, 2021\.
- Hinton et al\. \(2015\)Geoffrey Hinton, Oriol Vinyals, and Jeff Dean\.Distilling the knowledge in a neural network\.*arXiv preprint arXiv:1503\.02531*, 2015\.
- Hübotter et al\. \(2026\)Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, et al\.Reinforcement learning via self\-distillation\.*arXiv preprint arXiv:2601\.20802*, 2026\.
- Jaech et al\. \(2024\)Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El\-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al\.Openai o1 system card\.*arXiv preprint arXiv:2412\.16720*, 2024\.
- Jiang et al\. \(2026\)Li Jiang, Haoran Xu, Yichuan Ding, and Amy Zhang\.Trajectory\-refined distillation\.*arXiv preprint arXiv:2606\.08432*, 2026\.
- Kim & Rush \(2016\)Yoon Kim and Alexander M Rush\.Sequence\-level knowledge distillation\.In*Proceedings of the 2016 conference on empirical methods in natural language processing*, pp\. 1317–1327, 2016\.
- Kwon et al\. \(2023\)Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica\.Efficient memory management for large language model serving with pagedattention\.In*Proceedings of the 29th symposium on operating systems principles*, pp\. 611–626, 2023\.
- Leviathan et al\. \(2023\)Yaniv Leviathan, Matan Kalman, and Yossi Matias\.Fast inference from transformers via speculative decoding\.In*International Conference on Machine Learning*, pp\. 19274–19286\. PMLR, 2023\.
- Li et al\. \(2026\)Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan\-ang Gao, Wenkai Yang, Zhiyuan Liu, et al\.Rethinking on\-policy distillation of large language models: Phenomenology, mechanism, and recipe\.*arXiv preprint arXiv:2604\.13016*, 2026\.
- Lightman et al\. \(2024\)Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe\.Let’s verify step by step\.In*International Conference on Learning Representations*, volume 2024, pp\. 39578–39601, 2024\.
- Lu & Lab \(2025\)Kevin Lu and Thinking Machines Lab\.On\-policy distillation\.*Thinking Machines Lab: Connectionism*, 2025\.doi:10\.64434/tml\.20251026\.https://thinkingmachines\.ai/blog/on\-policy\-distillation\.
- Muennighoff et al\. \(2025\)Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei\-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori B Hashimoto\.s1: Simple test\-time scaling\.In*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, pp\. 20286–20332, 2025\.
- Patiño et al\. \(2025\)Carlos Miguel Patiño, Kashif Rasul, Quentin Gallouédec, Ben Burtenshaw, Sergio Paniego, Vaibhav Srivastav, Thibaud Frere, Ed Beeching, Lewis Tunstall, Leandro von Werra, and Thomas Wolf\.Unlocking on\-policy distillation for any model family\.[https://huggingface\.co/spaces/HuggingFaceH4/on\-policy\-distillation](https://huggingface.co/spaces/HuggingFaceH4/on-policy-distillation), 2025\.
- Penaloza et al\. \(2026\)Emiliano Penaloza, Dheeraj Vattikonda, Nicolas Gontier, Alexandre Lacoste, Laurent Charlin, and Massimo Caccia\.Privileged information distillation for language models\.*arXiv preprint arXiv:2602\.04942*, 2026\.
- Shao et al\. \(2024\)Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al\.Deepseekmath: Pushing the limits of mathematical reasoning in open language models\.*arXiv preprint arXiv:2402\.03300*, 2024\.
- Shenfeld et al\. \(2026\)Idan Shenfeld, Mehul Damani, Jonas Hübotter, and Pulkit Agrawal\.Self\-distillation enables continual learning\.*arXiv preprint arXiv:2601\.19897*, 2026\.
- Sheng et al\. \(2025\)Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu\.Hybridflow: A flexible and efficient rlhf framework\.In*Proceedings of the Twentieth European Conference on Computer Systems*, pp\. 1279–1297, 2025\.
- Xiao et al\. \(2026\)Bangjun Xiao, Bingquan Xia, Bo Yang, Bofei Gao, Bowen Shen, Chen Zhang, Chenhong He, Chiheng Lou, Fuli Luo, Gang Wang, et al\.Mimo\-v2\-flash technical report\.*arXiv preprint arXiv:2601\.02780*, 2026\.
- Xie et al\. \(2026\)Yan Xie, Sijie Zhu, Tiansheng Wen, Bo Chen, and Yifei Wang\.On the position bias of on\-policy distillation\.*arXiv preprint arXiv:2606\.22600*, 2026\.
- Xu et al\. \(2026\)Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chenchen Ling, et al\.Deepseek\-v4: Towards highly efficient million\-token context intelligence\.*arXiv preprint arXiv:2606\.19348*, 2026\.
- Xu et al\. \(2025\)Wenda Xu, Rujun Han, Zifeng Wang, Long Le, Dhruv Madeka, Lei Li, William Wang, Rishabh Agarwal, Chen\-Yu Lee, and Tomas Pfister\.Speculative knowledge distillation: Bridging the teacher\-student gap through interleaved sampling\.In*International Conference on Learning Representations*, volume 2025, pp\. 64616–64646, 2025\.
- Yang et al\. \(2025a\)An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al\.Qwen3 technical report\.*arXiv preprint arXiv:2505\.09388*, 2025a\.
- Yang et al\. \(2025b\)Wenkai Yang, Yankai Lin, Jie Zhou, and Ji\-Rong Wen\.Distilling rule\-based knowledge into large language models\.In*Proceedings of the 31st International Conference on Computational Linguistics*, pp\. 913–932, 2025b\.
- Ye et al\. \(2025\)Tianzhu Ye, Li Dong, Zewen Chi, Xun Wu, Shaohan Huang, and Furu Wei\.Black\-box on\-policy distillation of large language models\.*arXiv preprint arXiv:2511\.10643*, 2025\.
- Yu et al\. \(2026\)Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al\.Dapo: An open\-source llm reinforcement learning system at scale\.*Advances in Neural Information Processing Systems*, 38:113222–113244, 2026\.
- Zeng et al\. \(2026\)Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, et al\.Glm\-5: from vibe coding to agentic engineering\.*arXiv preprint arXiv:2602\.15763*, 2026\.
- Zhang et al\. \(2026\)Dongxu Zhang, Zhichao Yang, Sepehr Janghorbani, Jun Han, Andrew Ressler II, Qian Qian, Gregory D Lyng, Sanjit Singh Batra, and Robert E Tillman\.Fast and effective on\-policy distillation from reasoning prefixes\.In*Findings of the Association for Computational Linguistics: ACL 2026*, pp\. 25553–25569, 2026\.
- Zhao et al\. \(2026\)Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover\.Self\-distilled reasoner: On\-policy self\-distillation for large language models\.*arXiv preprint arXiv:2601\.18734*, 2026\.
- Ziheng et al\. \(2026\)Zhou Ziheng, Jiaqi Li, Huacong Tang, Ying Nian Wu, and Demetri Terzopoulos\.Less is more: Early stopping rollout for on\-policy distillation\.*arXiv preprint arXiv:2605\.27028*, 2026\.

## Appendix ATraining Configuration

### A\.1Handoff Trigger Word List

The reflection token setℛ\\mathcal\{R\}used in the handoff criterion consists of the following base words together with their case and leading\-space variants:

Handoff Trigger Word ListWait, But, Hmm, Actually, Hold, However, Yet, Oh, Alternatively, No, Ah, Oops, Well

### A\.2Training Hyperparameters

The complete training hyperparameters for the online distillation methods and for GRPO are listed in Table[4](https://arxiv.org/html/2607.26057#A1.T4)and Table[4](https://arxiv.org/html/2607.26057#A1.T4), respectively\.

Table 3:Training hyperparameters for online distillation methods\.
HyperparameterValueMax Prompt Length2048Max Response Length16384Rollout Temperature1\.0Rollout Top\-pp1\.0Rollout Number per Prompt \(nn\)1Global Batch Size128PPO Mini\-Batch Size128PPO Epochs1PPO Clipping Range0\.2Learning Rate1×10−61\\times 10^\{\-6\}Learning\-Rate ScheduleconstantTraining Epochs1
Table 4:Training hyperparameters for GRPO\.
HyperparameterValueMax Prompt Length2048Max Response Length16384Rollout Temperature1\.0Rollout Top\-pp1\.0Rollout Number per Prompt \(nn\)8Global Batch Size128PPO Mini\-Batch Size128PPO Epochs1PPO Clipping Range0\.2Learning Rate1×10−61\\times 10^\{\-6\}Learning\-Rate ScheduleconstantTraining Epochs1

## Appendix BPrompts

### B\.1Training and Inference Prompt

Both training and inference use the following template:

Training and Inference Prompt<\|<\|im\_start\|\>\|\>systemPlease reason step by step, and put your final answer within \\boxed\{\}\.<\|<\|im\_end\|\>\|\><\|<\|im\_start\|\>\|\>user\{problem\}<\|<\|im\_end\|\>\|\><\|<\|im\_start\|\>\|\>assistant<<think\>\><</think\>\>

### B\.2TRD Rewrite Prompt

We adopt the rewrite prompt from the original TRD paper\(Jiang et al\.,[2026](https://arxiv.org/html/2607.26057#bib.bib14)\), with user content as follows:

TRD Rewrite PromptYour task is to rewrite your mathematical solution\.\*\*Problem:\*\*\{problem\}\*\*Your Initial Solution:\*\*\{initial\_response\}\*\*Instructions:\*\*1\. Preserve the overall structure and reasoning path of your original solution2\. Identify and fix errors in computation or logic3\. Keep correct intermediate steps and meaningful work4\. Output ONLY the rewritten solution

## Appendix CAlgorithm and Baselines

### C\.1Relay\-OPD Algorithm

The relay rollout construction procedure of Relay\-OPD is given in Algorithm[1](https://arxiv.org/html/2607.26057#alg1)\.

Algorithm 1Relay Rollout Construction1:prompt

xx, student policy

πθ¯\\pi\_\{\\bar\{\\theta\}\}, teacher policy

πT\\pi\_\{T\}, reflection\-token set

ℛ\\mathcal\{R\}, handoff top\-

KK, relay budget

\(M,L\)\(M,L\)
2:relay trajectory

zz
3:Initialize

z←\(\)z\\leftarrow\(\), takeover count

j←0j\\leftarrow 0, state

s←𝖲s\\leftarrow\\mathsf\{S\}
4:whileneither EOS nor the maximum length is reacheddo⊳\\trianglerighteither event enters the terminal state⊥\\bot

5:// Student drafting and handoff detection

6:Set prefix

h←\(x,z\)h\\leftarrow\(x,z\)and sample student draft token

aS∼πθ¯\(⋅∣h\)a^\{S\}\\sim\\pi\_\{\\bar\{\\theta\}\}\(\\cdot\\mid h\)
7:Compute teacher argmax

aT←arg⁡maxv⁡πT​\(v∣h\)a^\{T\}\\leftarrow\\arg\\max\_\{v\}\\pi\_\{T\}\(v\\mid h\)and student top\-

KKset

𝒦S\(h\)←TopKK\(πθ¯\(⋅∣h\)\)\\mathcal\{K\}\_\{S\}\(h\)\\leftarrow\\operatorname\{TopK\}\_\{K\}\(\\pi\_\{\\bar\{\\theta\}\}\(\\cdot\\mid h\)\)
8:Evaluate the handoff criterion

ϕ​\(h\)←𝟏​\[aT∈ℛ\]⋅𝟏​\[𝒦S​\(h\)∩ℛ=∅\]\\phi\(h\)\\leftarrow\\mathbf\{1\}\[a^\{T\}\\in\\mathcal\{R\}\]\\cdot\\mathbf\{1\}\[\\mathcal\{K\}\_\{S\}\(h\)\\cap\\mathcal\{R\}=\\varnothing\]
9:if

s=𝖲s=\\mathsf\{S\}and

ϕ​\(h\)=1\\phi\(h\)=1and

j<Mj<Mthen⊳\\triangleright𝖲→𝖳\\mathsf\{S\}\\\!\\to\\\!\\mathsf\{T\}transition

10:// Teacher leg

11:

j←j\+1j\\leftarrow j\+1,

s←𝖳s\\leftarrow\\mathsf\{T\}; append

aTa^\{T\}to

zz⊳\\trianglerightreflection token opens the teacher leg

12:for

ℓ=1,…,L\\ell=1,\\ldots,Ldo

13:Generate one teacher paragraph via speculative decoding \(§[3\.3](https://arxiv.org/html/2607.26057#S3.SS3)\) and append it to

zz
14:endfor

15:if

j=Mj=Mthen

16:break⊳\\trianglerightrelay budget exhausted; enter the terminal state⊥\\bot

17:else

18:

s←𝖲s\\leftarrow\\mathsf\{S\}⊳\\triangleright𝖳→𝖲\\mathsf\{T\}\\\!\\to\\\!\\mathsf\{S\}transition: the student resumes

19:endif

20:else

21:Append

aSa^\{S\}to

zz⊳\\trianglerightstudent leg, includingϕ​\(h\)=1\\phi\(h\)=1withj=Mj=M

22:endif

23:endwhile

### C\.2Method Comparison Overview

We summarize all compared methods in Table[5](https://arxiv.org/html/2607.26057#A3.T5)along policy type, trajectory source, training loss, and method\-specific settings\.

Table 5:Method Comparison Overview\.Original FKL\-based papers use full vocabulary or varying top\-kksettings; we uniformly adopt top\-k=128k\{=\}128\. The original SKD paper uses acceptance top\-k=25k\{=\}25, but in our teacher–student configuration top\-25 barely filters any student draft tokens, so we adopt top\-k=5k\{=\}5\.MethodPolicyTrajectory SourceLossMethod\-Specific SettingsSFTOff\-policyTeacher trajectoryCE—KDOff\-policyTeacher trajectoryFKL—TRDOff\-policyRewritten student trajectoryFKLrewrite prompt \(B\.2\)GRPOOn\-policyStudent rolloutOutcome RL8 rollouts per promptOPDOn\-policyStudent rolloutRKLmax response = 16384FastOPDOn\-policyStudent rolloutRKLfixed truncation∈\{1024,2048,4096,8192\}\\in\\\{1024,2048,4096,8192\\\}\(C\.5\)SKDOn\-policySpeculative mixed rolloutFKLacceptance top\-K=5K\{=\}5\(C\.4\)Relay\-OPDOn\-policyRelay rolloutRKLhandoff top\-KK; relay budget\(M,L\)\(M,L\)
### C\.3TRD

TRD\(Jiang et al\.,[2026](https://arxiv.org/html/2607.26057#bib.bib14)\)constructs training data offline in two stages\. Stage one lets the student generate an initial trajectoryyoy\_\{o\}from the original problemxx; stage two providesxxandyoy\_\{o\}to the teacher within a rewrite prompt, producing a rewritten trajectoryyry\_\{r\}\. We adopt the rewrite prompt from the original TRD paper; the complete prompt is given in Appendix[B\.2](https://arxiv.org/html/2607.26057#A2.SS2)\.

During TRD training, teacher and student use different contexts\. For thett\-th token in the rewritten trajectory, the student context ishtS=\(x,yr,<t\)h\_\{t\}^\{S\}=\(x,y\_\{r,<t\}\), while the teacher context additionally includes the student’s initial trajectory:htT=\(x,yo,yr,<t\)h\_\{t\}^\{T\}=\(x,y\_\{o\},y\_\{r,<t\}\)\. On teacher contexthtTh\_\{t\}^\{T\}, the top\-kksupport set is𝒦tT:=TopKk\(πT\(⋅∣htT\)\)\\mathcal\{K\}\_\{t\}^\{T\}:=\\operatorname\{TopK\}\_\{k\}\\\!\\left\(\\pi\_\{T\}\(\\cdot\\mid h\_\{t\}^\{T\}\)\\right\)withk=128k=128, and teacher probability is renormalized over this support set as

qTk​\(v∣htT\):=πT​\(v∣htT\)∑u∈𝒦tTπT​\(u∣htT\),v∈𝒦tT\.q\_\{T\}^\{k\}\(v\\mid h\_\{t\}^\{T\}\):=\\frac\{\\pi\_\{T\}\(v\\mid h\_\{t\}^\{T\}\)\}\{\\sum\_\{u\\in\\mathcal\{K\}\_\{t\}^\{T\}\}\\pi\_\{T\}\(u\\mid h\_\{t\}^\{T\}\)\},\\qquad v\\in\\mathcal\{K\}\_\{t\}^\{T\}\.\(16\)
The corresponding top\-kkFKL objective is

ℒTRD​\(t\)=∑v∈𝒦tTqTk​\(v∣htT\)​\[log⁡qTk​\(v∣htT\)−log⁡πθ​\(v∣htS\)\]\.\\mathcal\{L\}\_\{\\mathrm\{TRD\}\}\(t\)=\\sum\_\{v\\in\\mathcal\{K\}\_\{t\}^\{T\}\}q\_\{T\}^\{k\}\(v\\mid h\_\{t\}^\{T\}\)\\left\[\\log q\_\{T\}^\{k\}\(v\\mid h\_\{t\}^\{T\}\)\-\\log\\pi\_\{\\theta\}\(v\\mid h\_\{t\}^\{S\}\)\\right\]\.\(17\)
This asymmetry means that the teacher provides supervision conditioned onyoy\_\{o\}, while the student cannot accessyoy\_\{o\}at either training or inference time\.

### C\.4SKD

SKD\(Xu et al\.,[2025](https://arxiv.org/html/2607.26057#bib.bib30)\)constructs teacher–student mixed trajectories via speculative decoding\. The student serves as draft model and the teacher as target model; throughout this subsection,zzdenotes SKD’s mixed trajectory andht=\(x,z<t\)h\_\{t\}=\(x,z\_\{<t\}\)its prefix\. Given prefixhth\_\{t\}and student draft tokenatSa\_\{t\}^\{S\}, the teacher’s top\-KKsupport set is𝒦T\(ht\)=TopKK\(πT\(⋅∣ht\)\)\\mathcal\{K\}\_\{T\}\(h\_\{t\}\)=\\operatorname\{TopK\}\_\{K\}\\\!\\left\(\\pi\_\{T\}\(\\cdot\\mid h\_\{t\}\)\\right\)withK=5K=5, and the draft is accepted iffatS∈𝒦T​\(ht\)a\_\{t\}^\{S\}\\in\\mathcal\{K\}\_\{T\}\(h\_\{t\}\); if accepted,zt=atSz\_\{t\}=a\_\{t\}^\{S\}\. If rejected, a replacement token is sampled from the teacher’s distribution renormalized over𝒦T​\(ht\)∖\{atS\}\\mathcal\{K\}\_\{T\}\(h\_\{t\}\)\\setminus\\\{a\_\{t\}^\{S\}\\\}:

zt∼π~T\(⋅∣ht\),supp\(π~T\)=𝒦T\(ht\)∖\{atS\}\.z\_\{t\}\\sim\\widetilde\{\\pi\}\_\{T\}\(\\cdot\\mid h\_\{t\}\),\\qquad\\operatorname\{supp\}\(\\widetilde\{\\pi\}\_\{T\}\)=\\mathcal\{K\}\_\{T\}\(h\_\{t\}\)\\setminus\\\{a\_\{t\}^\{S\}\\\}\.\(18\)
Replaced positions are marked as teacher\-owned\. The training objective uses top\-kkFKL on the mixed trajectory, with teacher and student sharing the same prefixhth\_\{t\}\.

### C\.5FastOPD

FastOPD\(Ziheng et al\.,[2026](https://arxiv.org/html/2607.26057#bib.bib38); Zhang et al\.,[2026](https://arxiv.org/html/2607.26057#bib.bib36)\)truncates the student rollout at a fixed length\. Given student\-generated trajectoryy=\(y1,…,yN\)y=\(y\_\{1\},\\ldots,y\_\{N\}\)and truncation lengthBB, the training trajectory isy\(B\)=\(y1,…,yNB\)y^\{\(B\)\}=\(y\_\{1\},\\ldots,y\_\{N\_\{B\}\}\)withNB=min⁡\(N,B\)N\_\{B\}=\\min\(N,B\)\.

FastOPD then optimizes the standard OPD objective \(§[3\.1](https://arxiv.org/html/2607.26057#S3.SS1)\) ony\(B\)y^\{\(B\)\}, replacingNNandyywithNBN\_\{B\}andy\(B\)y^\{\(B\)\}\. Thus FastOPD changes only the rollout length participating in training, not the token\-level learning signal of standard OPD\.

## Appendix DSupplementary Results

### D\.1Complete Sensitivity Results

Complete per\-benchmark results for the relay budget and handoff top\-KKsweeps of §[4\.5](https://arxiv.org/html/2607.26057#S4.SS5)are reported in Table[6](https://arxiv.org/html/2607.26057#A4.T6)\.

Table 6:Complete Sensitivity Results\.Using Qwen3\-4B\-Instruct\-2507 as teacher and Qwen3\-1\.7B\-Non\-Thinking as student, varying one hyperparameter at a time\. Standard OPD corresponds toK=\|𝒱\|K=\|\\mathcal\{V\}\|; bold denotes the best result in each column\.VariantAIME24AIME25AIME26MATHAMC23Olymp\.HMMTFeb26HMMTNov25AvgL=0L=040\.4229\.0627\.1988\.1573\.8356\.3123\.4816\.0444\.31L=1L=141\.4631\.4626\.8888\.7576\.6457\.6822\.7318\.8545\.56L=2L=240\.5232\.5028\.2388\.5578\.2057\.2322\.1619\.0645\.81L=4L=442\.6032\.4032\.7189\.5078\.2058\.4923\.3919\.4847\.10L=5L=540\.0032\.8130\.1089\.3578\.2857\.9423\.5819\.6946\.47M=1M=142\.4031\.9829\.6988\.5575\.4758\.3823\.3020\.2146\.25M=3M=343\.1231\.4626\.8888\.8576\.4857\.4622\.8220\.4245\.94M=4M=437\.6032\.0826\.8886\.9073\.5256\.3421\.5917\.1944\.01K=1K=140\.0029\.9024\.3887\.5574\.2256\.8622\.7318\.5444\.27K=10K=1036\.4629\.2726\.0488\.1573\.0556\.1920\.3615\.6243\.14K=\|𝒱\|K=\\lvert\\mathcal\{V\}\\rvert35\.8325\.5223\.3385\.7070\.0855\.2720\.0814\.0641\.23Default\(K=5K=5,M=2M=2,L=3L=3\)42\.7132\.8130\.5289\.5076\.8858\.7924\.7219\.7946\.96

### D\.2FastOPD Truncation Length Sweep

Complete results across fixed truncation lengths are reported in Table[7](https://arxiv.org/html/2607.26057#A4.T7)\. Both student models achieve the highest average accuracy at 4,096 tokens; Table[1](https://arxiv.org/html/2607.26057#S4.T1)therefore reports this configuration\.

Table 7:FastOPD Fixed Truncation Length Comparison\.Subscripts in the Avg column indicate the training step of the best checkpoint; Train Len denotes average rollout response length from training start to that checkpoint\.TruncationLengthAIME24AIME25AIME26MATHAMC23Olymp\.HMMTFeb26HMMTNov25AvgTrainLenStudent: Qwen3\-0\.6B\-Non\-Thinking1,02415\.9420\.4212\.4074\.6554\.6943\.9510\.519\.5830\.27@105\{\}\_\{\\text\{@105\}\}9872,04815\.8319\.3815\.4276\.9051\.4844\.7710\.518\.3330\.33@95\{\}\_\{\\text\{@95\}\}1,8274,09615\.8320\.1013\.5475\.3053\.6744\.8111\.558\.5430\.42@100\{\}\_\{\\text\{@100\}\}3,3028,19214\.7918\.4411\.8875\.3551\.8043\.369\.668\.6529\.24@60\{\}\_\{\\text\{@60\}\}5,055Student: Qwen3\-1\.7B\-Non\-Thinking1,02439\.0626\.4625\.2186\.0070\.7057\.0823\.2019\.1743\.36@105\{\}\_\{\\text\{@105\}\}9832,04843\.2329\.5828\.3388\.3572\.1157\.4223\.6719\.1745\.23@75\{\}\_\{\\text\{@75\}\}1,7534,09642\.2930\.4226\.3587\.9574\.3058\.1623\.5820\.7345\.47@45\{\}\_\{\\text\{@45\}\}2,7098,19240\.5227\.6026\.5686\.7072\.8156\.2721\.8820\.3144\.08@50\{\}\_\{\\text\{@50\}\}4,004

## Appendix ECase Studies

### E\.1Relay\-OPD Takeover Case

In this case, the student has only 5 RM remaining after purchasing 15 apples yet incorrectly concludes it can still afford both a mango and a papaya\. Without teacher intervention, it rationalizes this plan into a total of 17 fruits\. Faced with the same prefix, the teacher selectsButwith 74\.4084% probability \(Table[8](https://arxiv.org/html/2607.26057#A5.T8)\), whereas the student favorsSoorNowto extend the current conclusion\. The teacher leg points out that the two fruits would cost at least 9 RM, and the student accordingly rejects the original plan and finds the feasible combination of 12 apples, 1 mango, and 2 papayas, arriving at the correct answer of1515\. This case illustrates two points: the handoff trigger correctly identifies the moment the student would continue along a wrong direction, and a brief teacher takeover suffices to correct the subsequent reasoning\.

ProblemDanial went to a fruit stall that sells apples, mangoes, and papayas\. Each apple costs33RM, each mango costs44RM, and each papaya costs55RM\. He bought at least one of each fruit and paid exactly5050RM\. What is the maximum number of fruits that he could have bought?Ground truth:15

Student Trajectory \(w/o Intervention\)…Now, try15 apples:•15×3=4515\\times 3=45•Remaining:50−45=550\-45=5We can buy1 papaya\(5 RM\), and at least 1 mango \(4 RM\)\.So that’s valid\.\[Handoff trigger detected; the following is the student’s original continuation\]So:•15 apples•1 mango•1 papayaTotal fruits:15\+1\+1=1715\+1\+1=17Is this themaximumpossible? Let’s check\.Try14 apples:14×3=4214\\times 3=42, remaining50−42=850\-42=8; we can buy 1 mango \(4 RM\) and 1 papaya \(5 RM\)\. Total:14\+1\+1=1614\+1\+1=16fruits\. Smaller than 17\.…Final Answer:17\\boxed\{17\}

Table 8:Teacher and student distributions\.RankTeacher distributionStudent distribution1But74\.4084%So50\.6288%2So10\.0701%Now21\.1052%3Wait7\.8426%\-16\.4368%4We4\.1978%Total5\.3362%5\-1\.2027%Then2\.8563%6Can0\.5014%That0\.5624%7Now0\.2368%F0\.4963%8Remaining0\.2368%This0\.4380%9Then0\.2090%We0\.4380%10Try0\.1268%Let0\.3866%Relay Trajectory…Now, try15 apples:•15×3=4515\\times 3=45•Remaining:50−45=550\-45=5We can buy1 papaya\(5 RM\), and at least 1 mango \(4 RM\)\.So that’s valid\.\[Handoff trigger detected; the teacher takes over\]But we have only 5 RM left after 15 apples — we would have to spend 4 RM on mango and 5 on papaya — total 9, which exceeds 5\.Wait — no: we have 5 RM left, and we must spend at least 4 RM for mango and 5 RM for papaya, but only 5 RM is available\. So both cannot be bought\.We need at least one mango and one papaya\.\[Teacher leg ends; generation returns to the student\]So 15 apples is not valid\.…Try12 apples:•12×3=3612\\times 3=36•Remaining:50−36=1450\-36=14Try:•1 mango = 4, 2 papayas = 10→\\tototal = 14→\\tovalid\!So:•12 apples•1 mango•2 papayasTotal =12\+1\+2=𝟏𝟓12\+1\+2=\\mathbf\{15\}fruits…15\\boxed\{15\}

### E\.2TRD Rewritten Text

We matched references to the original solution \(e\.g\.,initial/original solution\), direct descriptions of the rewriting task \(e\.g\.,rewrite/revision\), and reviewer\-style openings \(e\.g\.,after review/reevaluation\); 18\.96% of trajectories contain at least one such expression, confirming that these rewriting artifacts are not isolated\. The following representative fragments are extracted directly from teacher\-generated rewritten trajectories; ellipses indicate truncated content\.

TRD Rewritten FragmentsGiven the complexity, and thatthe original solution incorrectly assumed a simple parity, we must revise\.… a common variant of this game has the property that the first player wins iff\(Nmod3=1\)\(N\\bmod 3=1\)\. Even though our manual simulation showed\(N=6\)\(N=6\)…perhaps we made a mistake\.This may be incorrect based on simulation;however, without complete DP simulation …this is the most reasonable periodic answer\.…we go with674\\boxed\{674\}\.

### E\.3SKD Repetition Patterns

SKD accepts a student draft token whenever it falls within the teacher’s top\-KKsupport set \(Appendix[C\.4](https://arxiv.org/html/2607.26057#A3.SS4)\)\. Because acceptance is driven by this generic distributional agreement, SKD struggles to break a repetitive generation pattern once the student has established it: the tokens that continue the repetition typically remain inside the teacher’s top\-KKcandidates, even when the teacher itself would terminate or redirect\. The following fragment from an SKD mixed trajectory is representative: the student repeatedly emits a completed final\-answer block\.

SKD Mixed Trajectory…Final Answer:No integer​n\\boxed\{\\text\{No integer \}n\}Final Answer:No integer​n\\boxed\{\\text\{No integer \}n\}…

Table 9:Teacher and student distributions at the end of a repeated answer block\.RankTeacher distributionStudent distribution1<\|im\_end\|\>73\.7035%\\n\\n78\.5176%2\\n\\n16\.4455%<\|im\_end\|\>19\.8524%3Ġ^aľ6\.8555%\\n1\.2691%4\\n0\.7226%or0\.1516%5\*\*0\.5627%Ġ^aľ0\.0558%Table[9](https://arxiv.org/html/2607.26057#A5.T9)shows the teacher and student next\-token distributions on the prefix ending at a completed answer block\.111Token strings appear in the tokenizer’s byte\-level representation, which maps every byte to a printable character:Ġdenotes a leading space, and^aľare the first two UTF\-8 bytes of a check\-mark symbol \(e\.g\., ✓\), whose final byte would be completed by a subsequent token\.The teacher intends to terminate generation: its top\-11token is the end\-of\-sequence token<\|im\_end\|\>with probability 73\.7035%\. The student instead prefers to open yet another paragraph, placing 78\.5176% on\\n\\n\. Under the acceptance criterionatS∈𝒦T​\(ht\)a\_\{t\}^\{S\}\\in\\mathcal\{K\}\_\{T\}\(h\_\{t\}\)withK=5K=5, however,\\n\\nstill ranks second in the teacher’s support set \(16\.4455%\), so the student draft is accepted and the repetition continues—and because the same distributions recur on the extended prefix, the trajectory can degenerate into aninfinite loopof final\-answer blocks\. Acceptance driven by generic distributional agreement thus provides no explicit signal that the established generation pattern itself has gone wrong, which underlies the repetition\-related degradation of SKD discussed in §[4\.2](https://arxiv.org/html/2607.26057#S4.SS2)\.

## Appendix FLimitations

Our evaluation focuses on mathematical reasoning with Qwen3 teacher–student pairs, the strong\-to\-weak setting targeted by prior on\-policy distillation work\. The relay mechanism itself is task\-agnostic, and we leave its application to other domains, such as code generation or agentic tool use, to future work; the reflection\-token set in Appendix[A\.1](https://arxiv.org/html/2607.26057#A1.SS1)may need adjustment when switching to a different model family\. Relay\-OPD also presumes a teacher whose continuations redirect failed prefixes more reliably than the student’s, so its benefit is expected to diminish as the teacher–student capability gap narrows\. Finally, the relay budget was tuned on the 1\.7B student and reused unchanged for the 0\.6B student; the sensitivity analysis in §[4\.5](https://arxiv.org/html/2607.26057#S4.SS5)indicates robustness across moderate budget ranges, though new model pairs may benefit from re\-tuning\.