照我做的做，别照我说的做：视觉语言模型中的同策略蒸馏会继承教师模型的特性

# Do As I Do, Not As I Say: On-Policy Distillation Inherits the Teacher in Vision-Language Models
Source: [https://arxiv.org/html/2610.20989](https://arxiv.org/html/2610.20989)
Kefei Yan11footnotemark:1Yiren Wang11footnotemark:1Yiyang Wang11footnotemark:1Affiliation:Georgia Institute of TechnologyAffiliation:Atlanta, GA, USAEmail:[\{frank\.yang,kyan66,ywang4361,ywang3420\}@gatech\.edu](mailto:)

###### Abstract

Improving the reasoning capabilities of vision\-language models \(VLMs\) often relies on post\-training with supervision from stronger models\. Two common strategies are supervised fine\-tuning \(SFT\), which trains on teacher\-generated demonstrations, and on\-policy distillation \(OPD\), which provides teacher feedback on trajectories generated by the student itself\. While OPD can be more data\-efficient by supervising student\-generated trajectories, it is unclearhow the teacher’s supervision at each state affects the student’s learned behavior\. To study this question, we conduct a controlled comparison of OPD and filtered SFT across data budgets and tasks where the teacher exhibits different levels of reliability, and analyze how these differences shape student accuracy and completion behavior\. Across the two tasks, we find that OPD is effective only when the teacher remains reliable on student\-generated trajectories\. On ChartQA, where the teacher completes 99% of its solutions, OPD trained on only 10% of the data comes within 1\.4 accuracy points of full\-data SFT\. In contrast, on Geometry3K, where the teacher fails to complete 22% of its solutions, OPD underperforms SFT by 7\.9 points, with most of the gap explained by incomplete responses\. More strikingly, starting OPD from an SFT student reduces its completion rate from 76% to 62%, effectively erasing the completion behavior learned through SFT\. Filtering teacher demonstrations for completeness alone recovers much of SFT’s advantage\. These results show that on\-policy supervision can transfer not only teacher knowledge but also persistent failure modes on student\-visited states\. Code is available at[https://github\.com/ffy208/opd\-inherits\-teacher\-vlm](https://github.com/ffy208/opd-inherits-teacher-vlm)\.

## 1Introduction

Distilling a large vision\-language model \(VLM\) into a small one is the standard way to make multimodal reasoning cheap enough to deploy\. The common recipe is supervised fine\-tuning on teacher\-written solutions\([Magister et al\., 2023](https://arxiv.org/html/2610.20989#bib.bib30);[Ho et al\., 2023](https://arxiv.org/html/2610.20989#bib.bib31)\)\(SFT\), which is behavior cloning: the student sees only states the teacher visited, drifts into others at inference, and its errors compound with sequence length\([Ross and Bagnell, 2010](https://arxiv.org/html/2610.20989#bib.bib36);[Ross et al\., 2011](https://arxiv.org/html/2610.20989#bib.bib1)\)\. On\-policy distillation \(OPD\) is the DAgger\-style remedy\([Agarwal et al\., 2024](https://arxiv.org/html/2610.20989#bib.bib3);[Gu et al\., 2024](https://arxiv.org/html/2610.20989#bib.bib4)\): the student samples its own solution, the teacher scores every generated token, and the student minimises a per\-token divergence to the teacher on the states it actually reaches\. For text\-only language models, on\-policy distillation on a twentieth of the training set has been reported to outperform supervised distillation on the full set\([Agarwal et al\., 2024](https://arxiv.org/html/2610.20989#bib.bib3)\), and it is now a standard post\-training stage\([Yang et al\., 2025](https://arxiv.org/html/2610.20989#bib.bib33);[Lu and Thinking Machines Lab, 2025](https://arxiv.org/html/2610.20989#bib.bib37)\)\. A dense 2026 literature now adapts the objective to VLMs by re\-weighting the teacher’s signal toward visually grounded tokens\([Liu et al\., 2026](https://arxiv.org/html/2610.20989#bib.bib12);[Yoon et al\., 2026](https://arxiv.org/html/2610.20989#bib.bib13);[Xue et al\., 2026](https://arxiv.org/html/2610.20989#bib.bib14)\)\.

While these methods change the objective and report the gains of the change, they leave open the question that comes first in practice: given a teacher and a few hundred to a few thousand training questions, when does plain on\-policy feedback improve on filtered teacher demonstrations, and which behavior of the teacher does the student acquire along the way? Related work observes that teacher feedback degrades on flawed student prefixes\([Qiu et al\., 2026](https://arxiv.org/html/2610.20989#bib.bib27);[Sun et al\., 2026](https://arxiv.org/html/2610.20989#bib.bib23)\); our experiments add completion as the behavior at stake, a matched comparison with filtered demonstrations, and a student that loses the completion gain it already had\.

The two recipes differ in cost and in what they transfer: OPD keeps the teacher in the loop and supervises the student on its own states with whatever the teacher does there; SFT pays once for teacher solutions and transfers only those that survive a correctness filter\. Which difference dominates depends on the task, and in our experiments the wrong choice costs up to7\.97\.9points\. Accuracy at a single budget cannot tell whether a gap comes from the amount of data or the number of training steps, from reasoning errors or from solutions that were never completed, from the objective or from the filter\.

To investigate these questions, we conducted \(Section[2](https://arxiv.org/html/2610.20989#S2)\) a controlled comparison of the two channels across nested data budgets with each channel’s update count fixed, token\-role divergence logging at every training step, and completion diagnostics, applied to a Qwen3\-VL 2B student and 8B teacher\([Bai and others, 2025](https://arxiv.org/html/2610.20989#bib.bib11)\)on ChartQA\([Masry et al\., 2022](https://arxiv.org/html/2610.20989#bib.bib7)\)and Geometry3K\([Lu et al\., 2021](https://arxiv.org/html/2610.20989#bib.bib10)\)\.

Our contributions are threefold:

- •A controlled empirical comparison of OPD and filtered SFT\.We systematically evaluate both training recipes across four data budgets, four seeds, and two vision\-language reasoning tasks with different teacher completion rates\. We further examine the sensitivity of their relative performance to SFT training length and evaluation distribution\.
- •Identification of a completion\-specific failure mode\.We show that final\-answer accuracy can obscure substantial differences in completion behavior\. By decomposing accuracy differences according to whether each method produces a final answer, we characterize how incomplete responses contribute to the performance gap\. We further demonstrate that continued OPD can undo completion behavior previously acquired through SFT\.
- •An empirical investigation of the failure\.We examine teacher behavior on student\-generated prefixes, compare demonstrations filtered by correctness and completeness, and evaluate five OPD training variants\. These analyses characterize the persistence of the completion failure and investigate the possible role of teacher feedback on student\-visited states\.

Two findings stand out\.

- •A persistent failure to complete solutions\.On Geometry3K, where the teacher itself does not finish 22% of its solutions within the token budget, OPD students at 900 problems conclude 63% of test responses against 76% for SFT and trail SFT by7\.97\.9points \(−10\.5\-10\.5to−5\.3\-5\.3\), of which6\.66\.6points come from problems that only one method concludes\. OPD started from the SFT student returns its rollout termination rate to plain OPD’s within 40 steps and ends at 62% test completion, the zero\-shot level\.
- •A small, non\-robust data\-efficiency advantage with a reliable teacher\.On ChartQA, OPD with 300 training questions reaches0\.8260\.826accuracy, within1\.41\.4points of SFT trained on 3,000 \(0\.8400\.840\), and leads SFT by2\.72\.7points at the same budget under the protocol’s fixed SFT length \(per seed\+0\.6\+0\.6to\+5\.8\+5\.8\) and by1\.11\.1to1\.51\.5points, with intervals that include zero, against an SFT whose training length is tuned\.

Together, these results characterize a previously underexplored failure mode of on\-policy distillation and give a working rule and a target: on these tasks OPD was competitive with filtered SFT at small budgets only where the teacher reliably completed its solutions; where it did not, the student’s completion failures concentrated on the problems the teacher does not finish, and from those prefixes the teacher rarely concludes, even with the answer in its prompt\. Whether that causes the failure or accompanies it awaits a teacher trained to conclude from such states \(Section[4\.3](https://arxiv.org/html/2610.20989#S4.SS3)\)\. Our findings motivate evaluating completion alongside accuracy and examining the reliability of teacher feedback on student\-visited states\.

## 2Method: a controlled comparison of two supervision channels

### 2\.1Two supervision channels

Letxxbe a prompt \(an image and a question\),πT\\pi\_\{T\}the teacher andπθ\\pi\_\{\\theta\}the student\([Hinton et al\., 2015](https://arxiv.org/html/2610.20989#bib.bib2), distillation as matching the teacher’s outputs,\)\. SFT samples solutionsy∼πT\(⋅∣x\)y\\sim\\pi\_\{T\}\(\\cdot\\mid x\)once, keeps those whose final answer is correct\([Ho et al\., 2023](https://arxiv.org/html/2610.20989#bib.bib31);[Yuan et al\., 2023](https://arxiv.org/html/2610.20989#bib.bib32)\), and minimises the token\-level cross\-entropy−∑tlogπθ\(yt∣x,y<t\)\-\\sum\_\{t\}\\log\\pi\_\{\\theta\}\(y\_\{t\}\\mid x,y\_\{<t\}\)on the kept set\. OPD instead samples a batch of rolloutsy\(i\)∼πθ\(⋅∣x\(i\)\)y^\{\(i\)\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x^\{\(i\)\}\)at every step and minimises the mean, over all generated tokens of the batch, of a divergence between the two next\-token distributions on the student’s own prefix,

ℒ\(θ\)=1∑i\|y\(i\)\|∑i∑t=1\|y\(i\)\|D\(πθ\(⋅∣x\(i\),y<t\(i\)\)∥πT\(⋅∣x\(i\),y<t\(i\)\)\),\\mathcal\{L\}\(\\theta\)=\\frac\{1\}\{\\sum\_\{i\}\|y^\{\(i\)\}\|\}\\sum\_\{i\}\\sum\_\{t=1\}^\{\|y^\{\(i\)\}\|\}D\\big\(\\pi\_\{\\theta\}\(\\cdot\\mid x^\{\(i\)\},y^\{\(i\)\}\_\{<t\}\)\\,\\\|\\,\\pi\_\{T\}\(\\cdot\\mid x^\{\(i\)\},y^\{\(i\)\}\_\{<t\}\)\\big\),\(1\)withDDthe reverse KL by default\([Agarwal et al\., 2024](https://arxiv.org/html/2610.20989#bib.bib3);[Gu et al\., 2024](https://arxiv.org/html/2610.20989#bib.bib4)\)\. The divergence is taken over the full vocabulary, including the end\-of\-turn token, at every generated position up to and including the first end\-of\-turn token or the generation budget; a rollout that runs to the budget contributes every one of its positions\.

The case for on\-policy supervision is DAgger’s\([Ross and Bagnell, 2010](https://arxiv.org/html/2610.20989#bib.bib36);[Ross et al\., 2011](https://arxiv.org/html/2610.20989#bib.bib1)\): querying the expert on the learner’s own states removes the compounding error of behavior cloning, provided the expert’s action on those states is worth imitating\. The two channels therefore differ in two things at once: the states on which the teacher is queried \(its own prefixes against the student’s\) and the teacher behavior that reaches the student \(only correct solutions against everything the teacher does\)\. Our design separates these\.

### 2\.2Design: what is held fixed and what varies

Rather than propose a new objective, we hold everything but the supervision channel and the amount of data fixed: one frozen teacher and one student initialisation, with LoRA on the language\-model projections and the vision encoder frozen, so the channels differ in language\-side supervision and not in visual representation; one prompt, answer format, generation budget, scorer, and held\-out test set per task; budgets that are nested prefixes of one seeded shuffle, so both channels see the same questions and a larger budget adds questions rather than replacing them; and a number of optimizer updates that is fixed within each channel whatever the budget, so that along a curve only the amount of data changes \(the counts are not matched between channels; a training\-length control on SFT checks this\)\. The teacher is neither prompted for completeness nor retrained: its reliability varies only through the task, and two tasks on which the same teacher is reliable and unreliable turn that dependence into a difference between tasks rather than between teachers\.

The channel \(filtered demonstrations against on\-policy token\-level feedback\) is crossed with four budgets, two tasks, and four paired seeds\. Because the channel bundles the state distribution with the filter, two further SFT data sets \(no filter; every teacher sample with an answer line, correct or not\) separate the two\. Five single\-component interventions on OPD \(the loss on terminated rollouts only, the teacher’s sampling temperature, the forward KL, a sequence\-mean loss, initialisation from the SFT student\) change one part of the recipe each, and a larger teacher and two out\-of\-distribution test sets check the dependence on teacher size and test distribution\.

### 2\.3What we measure

Beyond accuracy, three complementary analyses characterize the observed differences and probe possible explanations\.*Completion diagnostics*: a response is*complete*if it contains an answer line \(one without counts as wrong\),*terminates*if it emits the end\-of\-turn token before the budget, and is*truncated*if it reaches the budget; paired accuracy differences are decomposed by which method concluded the problem\.*Feedback logging*: at every OPD step we record each generated token’s divergence and a heuristic role \(*answer line*,*numeric value*,*arithmetic*,*text*\); a role’s*KL concentration*is its share of the batch’s divergence divided by its share of tokens\.*Teacher probes*: the teacher is continued greedily from prefixes of the student’s truncated and concluded outputs and its end\-of\-turn probability is read along them, to ask what it does on the student’s states rather than on its own\.

## 3Experimental setup

#### Models and tasks\.

The student is Qwen3\-VL\-2B\-Instruct, trained with LoRA\([Hu et al\., 2022](https://arxiv.org/html/2610.20989#bib.bib5)\)of rank 64 on the language\-model projections, and the teacher Qwen3\-VL\-8B\-Instruct\([Bai and others, 2025](https://arxiv.org/html/2610.20989#bib.bib11)\), frozen; a 32B teacher is used in Section[4\.4](https://arxiv.org/html/2610.20989#S4.SS4)\. ChartQA\([Masry et al\., 2022](https://arxiv.org/html/2610.20989#bib.bib7)\)is restricted to human\-written questions \(3,000 training and 500 held\-out test questions sampled with a fixed seed\), where the 8B teacher answers 84% correctly and concludes 99% of its solutions with an answer line\. Geometry3K\([Lu et al\., 2021](https://arxiv.org/html/2610.20989#bib.bib10)\)has 2,101 training and 500 of the 601 official test problems with numeric answers, where the teacher answers 57% correctly and concludes only 78% of its solutions within the 1,536\-token budget\. The correctness filter of SFT keeps 2,402 of the 3,000 ChartQA teacher samples \(80%\) and 1,163 of the 2,101 Geometry3K samples \(55%\), so an SFT budget ofbbquestions trains on about0\.8​b0\.8bor0\.55​b0\.55bdemonstrations\. Two out\-of\-distribution chart sets, 461 numeric questions from CharXiv\([Wang et al\., 2024](https://arxiv.org/html/2610.20989#bib.bib9)\)and 500 factoid questions from ChartQAPro\([Masry et al\., 2025](https://arxiv.org/html/2610.20989#bib.bib8)\), score every ChartQA\-trained model\.

#### Protocol\.

Every model sees the same prompt: solve step by step and end with a lineAnswer: <value\>\. Accuracy is measured using relaxed match with 5% numeric tolerance\([Methani et al\., 2020](https://arxiv.org/html/2610.20989#bib.bib34)\), the ChartQA convention, which we also apply to Geometry3K \(the Geometry3K ordering is unchanged at 1% and 0\.1%; Appendix[I](https://arxiv.org/html/2610.20989#A9)\)\. The stored records predate the evaluator’s stop\-reason field, so truncation is inferred by re\-tokenising the output \(within 8 tokens of the budget; Appendix[D](https://arxiv.org/html/2610.20989#A4)\)\. For every correctness\-filtered SFT and OPD model in the paper fewer than 1% of responses stop without an answer line, so an incomplete response is a truncated one \(exceptions: the two SFT data controls of Section[4\.3](https://arxiv.org/html/2610.20989#S4.SS3), at 14% and 2% of responses at 300 problems\)\. SFT runs 302 optimizer steps \(two epochs of the full data\) at every budget; OPD runs 150 steps of 16 rollouts at temperature 1\.0 with a 512\-token budget on charts and 1,536 on geometry; the step count is fixed within each method, not matched between them\. Budgetsb∈\{100,300,900,3000\}b\\in\\\{100,300,900,3000\\\}questions \(charts\) or\{100,300,900,2101\}\\\{100,300,900,2101\\\}problems \(geometry\) are nested prefixes of a seeded shuffle\. Seeds change the data order, rollout sampling, and LoRA initialisation; the test set is fixed; every point and every variant has four seeds\. Full hyperparameters are in Appendix[A](https://arxiv.org/html/2610.20989#A1)\.

#### Intervals\.

Every interval is a 95% question\-clustered bootstrap interval: the 500 test questions are resampled with replacement, every seed’s outcome on a resampled question is kept together, and differences are paired by question within each seed and averaged over seeds; this treats the four trained models as fixed\. For the headline comparisons we also give each seed’s difference and attinterval over the four seeds, whose target is the method rather than these models \(Appendix[H](https://arxiv.org/html/2610.20989#A8)\), and we call a difference robust only if the clustered and the seed\-level interval both exclude zero\.

## 4Experiments and Results

### 4\.1With a reliable teacher, OPD is competitive at small budgets

Figure 1:Accuracy against training questions at a fixed number of optimizer updates per method \(mean over four seeds\) on ChartQA \(a\), where the teacher concludes 99% of its solutions, and Geometry3K \(b\), where it concludes 78%; \(c\) share of Geometry3K test responses with an answer line\. Shaded bands: 95% question\-clustered bootstrap intervals; grey dashed: zero\-shot teacher; grey dotted: zero\-shot student\. \(d\) Rollout termination rate during OPD on Geometry3K at 900 problems \(five\-step windows, mean and±1\\pm 1s\.d\. over four seeds\) from the base and from the filtered\-SFT student, which starts at0\.790\.79and is in plain OPD’s band within 40 steps \(shaded\)\.Table 1:ChartQA: accuracy on 500 held\-out questions, mean over four seeds \(across\-seed sd in brackets; every mean has a question\-clustered 95% interval of±0\.03\\pm 0\.03\), with the paired OPD minus SFT difference, its clustered interval, and thettinterval over the four per\-seed differences\. The last two rows use the SFT length that scored best in Appendix[K](https://arxiv.org/html/2610.20989#A11)\. Bold: the higher accuracy where the clustered interval excludes zero, and intervals that exclude zero\. Zero\-shot student0\.6680\.668, teacher0\.8440\.844\.AccuracyOPD−\-SFTQuestionsSFTOPDpaired, clustered 95%ttover seeds100 \(3%\)0\.7930\.793\(\.022\)0\.8110\.811\(\.009\)\+0\.018\+0\.018\[−0\.001\-0\.001,\+0\.038\+0\.038\]\[−0\.006\-0\.006,\+0\.043\+0\.043\]300 \(10%\)0\.7990\.799\(\.015\)0\.8260\.826\(\.007\)\+0\.027\+0\.027\[\+0\.008\+0\.008,\+0\.047\+0\.047\]\[−0\.008\-0\.008,\+0\.062\+0\.062\]900 \(30%\)0\.8080\.808\(\.011\)0\.8230\.823\(\.004\)\+0\.015\+0\.015\[−0\.004\-0\.004,\+0\.035\+0\.035\]\[\+0\.003\+0\.003,\+0\.027\+0\.027\]3000 \(100%\)0\.8400\.840\(\.007\)0\.8250\.825\(\.008\)−0\.015\-0\.015\[−0\.032\-0\.032,\+0\.002\+0\.002\]\[−0\.033\-0\.033,\+0\.003\+0\.003\]100, SFT 38 steps0\.7970\.797\(\.012\)0\.8110\.811\(\.009\)\+0\.015\+0\.015\[−0\.005\-0\.005,\+0\.035\+0\.035\]\[\+0\.001\+0\.001,\+0\.028\+0\.028\]300, SFT 38 steps0\.8160\.816\(\.005\)0\.8260\.826\(\.007\)\+0\.011\+0\.011\[−0\.006\-0\.006,\+0\.028\+0\.028\]\[\+0\.001\+0\.001,\+0\.020\+0\.020\]Table[1](https://arxiv.org/html/2610.20989#S4.T1)and Figure[1](https://arxiv.org/html/2610.20989#S4.F1)a give the ChartQA curve\. OPD’s mean is above SFT’s at 100, 300, and 900 questions by1\.51\.5to2\.72\.7points, and eleven of the twelve per\-seed differences are positive, but no lead is robust: the 300\-question lead excludes zero under the clustered interval and not under the seed\-level one \(per seed\+0\.6\+0\.6to\+5\.8\+5\.8\), the 900\-question lead the reverse, and the 100\-question lead under neither\. With a tenth of the questions, OPD at 300 \(0\.8260\.826\) comes within1\.41\.4points of SFT on all 3,000 \(0\.8400\.840\), a paired difference of−1\.4\-1\.4\[−3\.1\-3\.1,\+0\.4\+0\.4\]; we did not pre\-specify a non\-inferiority margin, so we say OPD approaches full\-data SFT at a data ratio of the kind reported for text\-only OPD\([Agarwal et al\., 2024](https://arxiv.org/html/2610.20989#bib.bib3)\)rather than matches it\. At 3,000 questions the ordering reverses by1\.51\.5points, all four seeds negative and the clustered interval touching zero\. The flat OPD curve is a property of the fixed\-update design: 2,400 rollouts are 24 passes over 100 questions and 0\.8 over 3,000, so OPD gains0\.80\.8points per decade of data against3\.13\.1for SFT \(Appendix[L](https://arxiv.org/html/2610.20989#A12)\)\.

#### Control: the training length of SFT\.

The protocol’s 302 steps are two epochs of the full data but about 20 epochs at 300 questions and 60 at 100, so the design could be handicapping SFT at low budgets\. It is: retrained for 19, 38, 75, and 150 steps \(Appendix[K](https://arxiv.org/html/2610.20989#A11)\), SFT peaks at 38 steps with0\.8160\.816at 300 questions against0\.7990\.799, and against it OPD’s lead shrinks to\+1\.1\+1\.1\[−0\.6\-0\.6,\+2\.8\+2\.8\] at 300 questions and\+1\.5\+1\.5\[−0\.5\-0\.5,\+3\.5\+3\.5\] at 100 \(Table[1](https://arxiv.org/html/2610.20989#S4.T1)\): OPD’s mean is above every SFT length, but its lead over a length\-tuned SFT is 1 to 1\.5 points and not robust, the tuned SFT’s seed spread is no larger than OPD’s, and the SFT length was selected on the test set, since the protocol has no validation split\.

### 4\.2The teacher’s feedback lands on the answer line first

Figure 2:Where the teacher’s per\-token feedback lands on ChartQA\. KL concentration is a token role’s share of the batch’s reverse KL divided by its share of tokens \(dotted line:11\); it measures where the divergence concentrates, not the size of the gradient\. \(a\) Over training, nine OPD runs with a complete per\-role log \(100 questions: four seeds; 300: seeds 1–3; 900: seeds 1–2\), 10\-step windows, mean and±1\\pm 1s\.d\. over runs\. \(b\) One model per condition \(zero\-shot student; 300\-question seed\-1 OPD student\) on 100 test questions, 11,625 and 12,842 tokens \(Appendix[C](https://arxiv.org/html/2610.20989#A3)\)\.Figure[2](https://arxiv.org/html/2610.20989#S4.F2)shows where the per\-role feedback lands \(the full training curves for both tasks are in Appendix[B](https://arxiv.org/html/2610.20989#A2)\)\. On the zero\-shot student’s outputs the answer line, 6% of tokens, holds 14% of the KL \(Appendix[C](https://arxiv.org/html/2610.20989#A3)\); over the first ten OPD steps on ChartQA it is1\.3×1\.3\\times, by steps 11–200\.42×0\.42\\times, and at the end0\.27×0\.27\\times, while numeric\-value tokens stay at0\.29×0\.29\\timesto0\.43×0\.43\\timestheir share and the mean per\-token KL falls from0\.410\.41to0\.150\.15: the feedback is spent first on how the student ends its solution and is diffuse afterwards\. The log measures agreement, not correctness: a digit is not a perceptual token, and a low divergence means agreement whether both models are right or wrong, so it does not show whether the student’s reading of the chart improves\.

### 4\.3With an unreliable teacher, the student does not complete its solutions

Table 2:Geometry3K: accuracy on 500 held\-out problems, four seeds at every budget \(means; across\-seed sd between0\.0060\.006and0\.0250\.025, per seed in Appendix[M](https://arxiv.org/html/2610.20989#A13); every mean has a question\-clustered 95% interval of±0\.035\\pm 0\.035to±0\.038\\pm 0\.038\), the paired OPD minus SFT difference with its clustered and seed\-levelttintervals, “Completion”, the fraction of responses with an answer line \(every response without one ran to the 1,536\-token budget; Appendix[D](https://arxiv.org/html/2610.20989#A4)\), and “Acc\.\|\|compl\.”, accuracy among those \(range over seeds\)\. Bold as in Table[1](https://arxiv.org/html/2610.20989#S4.T1)\. Zero\-shot student0\.3660\.366\(completion0\.650\.65\), teacher0\.5700\.570\(completion0\.780\.78\)\.On Geometry3K the ordering reverses \(Table[2](https://arxiv.org/html/2610.20989#S4.T2), Figure[1](https://arxiv.org/html/2610.20989#S4.F1)b\)\. SFT leads by1\.91\.9points at 100 problems \(intervals include zero\), the two methods tie at 300, and SFT leads by7\.97\.9points at 900 and6\.76\.7at 2,101, every per\-seed difference between−5\.2\-5\.2and−10\.6\-10\.6and both intervals excluding zero\. An OPD student concludes 62–64% of test problems against 73–78% for SFT, a completion gap of 11 to 14 points at every budget, and every missing answer is a response that ran to the token budget \(186 of 186 at 900 problems; Appendix[D](https://arxiv.org/html/2610.20989#A4)\)\. Among responses that do conclude, the OPD student’s accuracy is higher \(0\.610\.61–0\.700\.70against0\.520\.52–0\.670\.67\), but the two methods conclude different problems\. Decomposing the paired difference by which method concluded a problem \(Figure[3](https://arxiv.org/html/2610.20989#S4.F3)c; Appendix[D](https://arxiv.org/html/2610.20989#A4); every contribution in points of the 500\-problem accuracy\): at 900 problems the 95 problems per 500 that only SFT concludes cost OPD8\.88\.8points, the 29 that only OPD concludes gain it2\.22\.2, and the 285 both conclude cost it1\.41\.4, so completion accounts for6\.66\.6of the7\.97\.9points; at 2,101 it accounts for4\.34\.3of6\.76\.7\. Within the subset both methods conclude, OPD’s accuracy is2\.42\.4points below SFT’s at 900 \[−6\.0\-6\.0,\+1\.2\+1\.2\],4\.04\.0below at 2,101 \[−7\.7\-7\.7,−0\.3\-0\.3\], and4\.34\.3above at 300 \[\+2\.7\+2\.7,\+9\.7\+9\.7\]: the higher conditional accuracy of OPD in Table[2](https://arxiv.org/html/2610.20989#S4.T2)is largely a selection effect, and the completion gap explains most, not all, of the accuracy gap\.

#### Is the teacher the source?

Two observations point at the teacher\. First, the teacher’s own solutions do not conclude: 78% of its training samples and 78% of its greedy test solutions contain an answer line\. Second, the student’s failures concentrate on the problems the teacher does not finish \(Appendix[D](https://arxiv.org/html/2610.20989#A4)\): on the 109 test problems the greedy teacher leaves incomplete, the zero\-shot, SFT, and OPD students conclude 35%, 45%, and 20%; on the 391 it concludes, 73%, 85%, and 75%\. Problem difficulty could drive both failures, so this is an association\. The most direct evidence is what the teacher does when it takes over a student’s state \(Table[4](https://arxiv.org/html/2610.20989#A4.T4), Appendix[D](https://arxiv.org/html/2610.20989#A4)\): continued greedily from a 256\-token prefix of an OPD student’s truncated test output with the rest of the budget, it concludes 34%, and 3% from the truncation point, against 90% from a 256\-token prefix of a concluded output\. That budget shrinks with the cut \(1,280, 1,024, 512, and 512 tokens\), so the probe was repeated with 512 tokens at every cut \(lower block\): the within\-trajectory difference between the 256\-token cut and the truncation point falls from\+30\.6\+30\.6\[\+25\.4\+25\.4,\+35\.8\+35\.8\] to\+12\.2\+12\.2\[\+8\.5\+8\.5,\+16\.2\+16\.2\] points for the OPD student and to\+0\.8\+0\.8\[−5\.1\-5\.1,\+7\.4\+7\.4\] for the SFT student\. At equal budget the teacher recovers few failed prefixes of either student \(14% of SFT’s, 16% of OPD’s from 256 tokens\) and most prefixes of concluded outputs \(77 to 81%\), but those are different problems and trajectories, so the probe describes the states rather than identifies what produced them\. The teacher\-forced probe agrees: along outputs that ran to the budget the teacher’s end\-of\-turn probability stays below10−310^\{\-3\}at every position, against0\.980\.98at the position where a concluded output stops, and its mass on answer\-opening tokens \(a proxy\) peaks at0\.120\.12to0\.130\.13, against0\.630\.63to0\.820\.82where a concluded output’s answer line begins \(Appendix[D](https://arxiv.org/html/2610.20989#A4)\)\. Changing the teacher’s prompt does not supply what is missing \(Table[5](https://arxiv.org/html/2610.20989#A4.T5), on the zero\-shot student’s failed prefixes from 300 training problems rather than the trained students’ test outputs\): completion guidance leaves its completion at 19% against 18% \(\+1\.1\+1\.1\[−3\.3\-3\.3,\+5\.5\+5\.5\] points\), and the gold answer lowers it to 7% \(−11\.0\-11\.0\[−18\.7\-18\.7,−4\.4\-4\.4\]\), so no tested prompt establishes the improvement a training\-level intervention would need\.

#### Does a larger generation budget close the gap?

It does not\. Re\-scoring the same four 900\-problem models per method with 4,096 generated tokens instead of 1,536 \(Appendix[E](https://arxiv.org/html/2610.20989#A5)\), the OPD students’ incomplete responses fall only from 186 to 168 of 500 \(accuracy\+0\.8\+0\.8\[−0\.4\-0\.4,\+1\.9\+1\.9\]\), whereas SFT’s fall from 120 to 50 \(accuracy\+4\.0\+4\.0\[\+2\.5\+2\.5,\+5\.5\+5\.5\]\), so the gap widens from7\.67\.6to10\.810\.8points \[−13\.5\-13\.5,−8\.2\-8\.2\]\.

#### Which part of the filter matters: three SFT data sets\.

We retrained SFT on the teacher’s first sample for every problem, correct or not, concluded or not \(Figure[3](https://arxiv.org/html/2610.20989#S4.F3)a,b; Table[12](https://arxiv.org/html/2610.20989#A6.T12)in Appendix[F](https://arxiv.org/html/2610.20989#A6)\)\. At 300 problems, where filtered SFT and OPD tie, removing the filter drops SFT’s completion rate to OPD’s0\.620\.62and its accuracy1\.81\.8points below both \(intervals include zero\)\. At 900 problems the unfiltered SFT is3\.63\.6points below the filtered one \(per seed−1\.2\-1\.2to−8\.6\-8\.6, so the seed\-level interval includes zero\), its completion rate \(0\.690\.69\) lies between SFT’s and OPD’s, and it stays4\.24\.2points above OPD \[\+1\.8\+1\.8,\+6\.7\+6\.7\]\. Removing the filter changes the demonstrations’ correctness, completeness, length, difficulty mix, and number at once\. A third SFT, trained on the teacher’s first sample whenever it contains an answer line, correct or not \(1,645 of 2,101 samples; the criterion is the answer line, not an emitted end\-of\-turn token\), separates these: at 900 problems it reaches completion0\.740\.74and accuracy0\.4730\.473, within2\.12\.1\[−4\.4\-4\.4,\+0\.2\+0\.2\] and0\.80\.8\[−3\.2\-3\.2,\+1\.6\+1\.6\] points of the correctness\-filtered SFT and4\.74\.7\[\+2\.5\+2\.5,\+6\.9\+6\.9\] and2\.92\.9\[\+0\.7\+0\.7,\+5\.0\+5\.0\] points above the unfiltered one \(Table[12](https://arxiv.org/html/2610.20989#A6.T12); Appendix[H](https://arxiv.org/html/2610.20989#A8)\); at 300 problems it matches the filtered SFT \(0\.720\.72/0\.4320\.432against0\.730\.73/0\.4200\.420\)\. Selecting completed demonstrations thus recovers much of the observed SFT advantage under the tested recipes\. The difference between the two filtering schemes \(correctness\-filtered minus complete\-only\) is\+0\.8\+0\.8points \[−1\.6\-1\.6,\+3\.2\+3\.2\], so a benefit of correctness of up to about three points is not excluded, and the two data sets differ in size \(1,163 against 1,645 demonstrations\) as well as in correctness, so this is a difference between recipes rather than an identified contribution of correctness; a count\-matched control was not run\.

![Refer to caption](https://arxiv.org/html/2610.20989v1/figures/lab_fig_completion.png)Figure 3:Geometry3K, 900 problems, four seeds per row\. \(a\) Paired accuracy difference of the three SFT data sets \(filtered, the reference; unfiltered; complete\-only, defined in the text\) and of plain OPD and its five variants against the filtered SFT; bars: 95% question\-clustered bootstrap intervals with the four models per method as fixed \(seed\-levelttintervals in Table[12](https://arxiv.org/html/2610.20989#A6.T12), Appendix[F](https://arxiv.org/html/2610.20989#A6)\)\. \(b\) Share of test responses with an answer line; dashed: zero\-shot teacher; dotted: zero\-shot student\. \(c\) OPD minus SFT accuracy at each budget, in points of the 500 test problems, split by which method concluded the problem \(a problem only one method concludes counts against the other\); black tick: total\. From the per\-problem records; printed totals rounded to one decimal \(Table[10](https://arxiv.org/html/2610.20989#A4.T10)\)\.
#### Three on\-policy variants, and what their failure does and does not say about the objective\.

An on\-policy analogue of the filter computes the loss only on rollouts that terminated \(OPD\-term\); rollouts at the teacher’s temperature test the sampling distribution \(OPD\-t07\); the forward KL tests whether the reverse KL attenuates the stopping signal \(OPD\-fwd\)\. None of the three restores completion \(Figure[3](https://arxiv.org/html/2610.20989#S4.F3)a,b\): it stays at0\.600\.60to0\.640\.64, accuracy within a point of plain OPD, and all three match plain OPD on ChartQA \(Appendix[J](https://arxiv.org/html/2610.20989#A10)\)\. The gradient shows why the divergence alone cannot be blamed: the loss compares the two next\-token distributions over the whole vocabulary at every position, so the teacher’s probability of stopping enters the loss whether or not the student sampled a stop\. Writingpep\_\{e\}andqeq\_\{e\}for the student’s and the teacher’s end\-of\-turn probabilities at a position,zez\_\{e\}for the student’s end\-of\-turn logit, andLLfor the position’s reverse KL,∂L/∂ze=pe​\(log⁡\(pe/qe\)−L\)\\partial L/\\partial z\_\{e\}=p\_\{e\}\\,\(\\log\(p\_\{e\}/q\_\{e\}\)\-L\), which pusheszez\_\{e\}up wheneverlog⁡\(pe/qe\)<L\\log\(p\_\{e\}/q\_\{e\}\)<L, that ispe<qe​eLp\_\{e\}<q\_\{e\}e^\{L\}\(a studentpep\_\{e\}belowqeq\_\{e\}suffices but is not necessary\), but is scaled bypep\_\{e\}, so a student that rarely stops receives a small push; the forward KL has gradientpe−qep\_\{e\}\-q\_\{e\}, without the attenuation\. That OPD\-fwd fails as plain OPD does \(0\.3940\.394, completion0\.600\.60\) argues against attenuation and leaves two candidates: the teacher, conditioned on the student’s overlong prefixes, may itself assign little probability to stopping; and the token\-mean loss of Equation[1](https://arxiv.org/html/2610.20989#S2.E1)is dominated by rollouts that ran to the budget, which hold about three quarters of the positions in the 900\-problem runs \(mean rollout length 892 tokens, termination rate0\.550\.55\)\. The second does not by itself explain the failure: a sequence\-mean loss, in which every rollout counts once \(OPD\-seqmean; Appendix[A](https://arxiv.org/html/2610.20989#A1)\), does not restore completion, with0\.4070\.407accuracy and0\.620\.62completion at 900 problems against0\.4020\.402/0\.630\.63for plain OPD \(\+0\.4\+0\.4\[−1\.4\-1\.4,\+2\.1\+2\.1\] and−0\.7\-0\.7\[−2\.6\-2\.6,\+1\.2\+1\.2\]; per seed in Appendix[H](https://arxiv.org/html/2610.20989#A8)\), the same termination trajectory during training, and0\.8250\.825against0\.8260\.826on ChartQA \(Table[12](https://arxiv.org/html/2610.20989#A6.T12), Appendix[J](https://arxiv.org/html/2610.20989#A10)\)\. The first is what the probe above measures:qeq\_\{e\}is below10−310^\{\-3\}on these prefixes, so the target the objective sets at these positions is a student that does not stop either\. Whether the gradient at a position pusheszez\_\{e\}up depends on the student’spep\_\{e\}and on the position’s full KL, neither of which we recorded, so the direction of the push is not measured: the probe establishes the target, not the gradient\. Filtered SFT instead concentrates its supervision on demonstrations that contain an answer line\.

#### On\-policy feedback undoes the completion gain the student obtained from SFT\.

We started OPD from the merged filtered\-SFT student of the same seed \(OPD\-sftinit\), which concludes 76% of its test solutions, and trained it for the usual 150 steps\. The training log \(Figure[1](https://arxiv.org/html/2610.20989#S4.F1)d\) shows the behaviour being undone: the rollout termination rate is0\.790\.79over the first five steps,0\.540\.54by steps 21–40, and0\.600\.60over the rest of the run, against0\.630\.63,0\.480\.48, and0\.560\.56for plain OPD from the base model\. At evaluation it concludes 62% of test responses, indistinguishable from plain OPD \(−0\.5\-0\.5\[−2\.8\-2\.8,\+1\.7\+1\.7\]\) and13\.813\.8points below the SFT model it started from \(\[−16\.4\-16\.4,−11\.2\-11\.2\];tt\[−17\.7\-17\.7,−9\.8\-9\.8\]\)\. On the problems both it and SFT conclude no difference is detected \(−0\.5\-0\.5\[−3\.6\-3\.6,\+3\.1\+3\.1\]\); it ends at0\.4220\.422,5\.95\.9points below SFT \(\[−8\.2\-8\.2,−3\.8\-3\.8\];tt\[−10\.9\-10\.9,−1\.0\-1\.0\]\) and1\.91\.9above plain OPD \(\[−0\.2\-0\.2,\+4\.2\+4\.2\], not robust\)\. The objective thus undoes a completion gain the student already had rather than merely failing to teach one; the experiment identifies the training process as the cause, not which part of it\.

### 4\.4Distribution shift and teacher size

Off distribution \(Appendix[G](https://arxiv.org/html/2610.20989#A7)\), both methods move the student by only 2 to 5 points on CharXiv and ChartQAPro, against a teacher that is itself at0\.410\.41there; the only paired difference that excludes zero is OPD’s\+3\.0\+3\.0\[\+0\.5\+0\.5,\+5\.4\+5\.4\] on CharXiv at 100 questions, the other seven are within±0\.7\\pm 0\.7points, and overlapping intervals do not establish equivalence\. A 32B teacher on the 300\-question ChartQA run \(four seeds\) is itself1\.41\.4points more accurate \(0\.8580\.858against0\.8440\.844, an interval that includes zero\) and its student is not more accurate \(0\.8120\.812against0\.8260\.826,−1\.5\-1\.5\[−3\.1\-3\.1,\+0\.2\+0\.2\];\+1\.2\+1\.2\[−0\.9\-0\.9,\+3\.4\+3\.4\] against SFT\)\.

## 5Related work

#### On\-policy distillation\.

Building on imitation\-learning distillation\([Lin et al\., 2020](https://arxiv.org/html/2610.20989#bib.bib28)\), generalised knowledge distillation\([Agarwal et al\., 2024](https://arxiv.org/html/2610.20989#bib.bib3)\), MiniLLM\([Gu et al\., 2024](https://arxiv.org/html/2610.20989#bib.bib4)\), and DistiLLM\([Ko et al\., 2024](https://arxiv.org/html/2610.20989#bib.bib29)\)established student\-sampled, teacher\-scored token\-level distillation, and[Agarwal et al\. \(2024\)](https://arxiv.org/html/2610.20989#bib.bib3)reported the data\-efficiency advantage that Section[4\.1](https://arxiv.org/html/2610.20989#S4.SS1)measures for a VLM; text\-only analyses\([Li et al\., 2026](https://arxiv.org/html/2610.20989#bib.bib20);[Xie et al\., 2026](https://arxiv.org/html/2610.20989#bib.bib21)\)study its phenomenology\. Concurrent text\-only work argues that teacher reliability must be verified before dense on\-policy supervision\([Zhang et al\., 2026](https://arxiv.org/html/2610.20989#bib.bib22)\), that token\-level corrections cannot unfold a complete repair\([Sun et al\., 2026](https://arxiv.org/html/2610.20989#bib.bib23)\), and that the teacher’s supervision is noisy\([Ding and Zhang, 2026](https://arxiv.org/html/2610.20989#bib.bib24)\);[Qiu et al\. \(2026\)](https://arxiv.org/html/2610.20989#bib.bib27)show for visual hallucination that when teacher and student condition on the same flawed prefix the teacher’s signal collapses toward the student’s error\.[Ma \(2026\)](https://arxiv.org/html/2610.20989#bib.bib38)shows that low teacher–student token\-level divergence can coexist with an incorrect final answer in text\-only OPD\. We study a distinct failure mode in vision\-language distillation: responses that never reach an answer line\. We decompose the resulting accuracy gap by completion status and show that five on\-policy variants fail to restore completion; OPD also erases the completion gain of an SFT\-initialised student\. Changing the divergence\([Xiao et al\., 2026](https://arxiv.org/html/2610.20989#bib.bib25), see also\)or weighting rollouts\(as[Ai and Yan, 2026](https://arxiv.org/html/2610.20989#bib.bib26), propose offline\)did not restore completion\.

#### On\-policy distillation for vision\-language models\.

A rapid 2026 literature adapts the objective to VLMs: visual\-advantage weighting\([Liu et al\., 2026](https://arxiv.org/html/2610.20989#bib.bib12)\)up\-weights the tokens whose teacher prediction depends on fine visual detail and, trained on Geometry3K and ViRL39K, reports gains over plain OPD; decomposed gradient steering\([Yoon et al\., 2026](https://arxiv.org/html/2610.20989#bib.bib13)\)and Fisher\-projected distillation\([Xue et al\., 2026](https://arxiv.org/html/2610.20989#bib.bib14)\)reshape the update toward grounding; others transfer reasoning from a text\-only teacher\([Bousselham et al\., 2026](https://arxiv.org/html/2610.20989#bib.bib16)\)or replace the teacher with the same model given privileged input\([Yuan et al\., 2026](https://arxiv.org/html/2610.20989#bib.bib15);[Zhu et al\., 2026](https://arxiv.org/html/2610.20989#bib.bib17);[Xiang et al\., 2026](https://arxiv.org/html/2610.20989#bib.bib18);[Wang et al\., 2026](https://arxiv.org/html/2610.20989#bib.bib19)\)\. All change the objective and compare the change against plain OPD; we keep the objective fixed and ask how it compares with filtered demonstrations and which behaviour of the teacher the student acquires\.

## 6Conclusion

We study when OPD improves over filtered SFT and how teacher behavior on student\-generated trajectories affects the student\. OPD is data\-efficient when the teacher reliably completes its solutions, but degrades when the teacher frequently fails to complete\. On Geometry3K, most of the gap comes from incomplete responses, and OPD even erases completion gains previously learned through SFT\. Filtering teacher demonstrations for completeness alone recovers much of SFT’s advantage, while teacher probes show that the teacher itself rarely completes from failed student trajectories\. These results highlight teacher reliability on student\-visited states as a critical factor in the success of OPD\.

### Limitations

Our evaluation covers one model family, two tasks, one student size below 4B, and a second teacher size on one task; other families, tasks, and scales remain untested\. The attribution of the completion failure to the teacher rests on the teacher’s own trajectories, the per\-problem overlap with the student’s failures, and the probe on the student’s prefixes; no tested teacher prompt clearly improved its recovery on the student’s failed prefixes \(one lowered it\), so a teacher trained to conclude from these states remains untested; we did not record the student’s own end\-of\-turn probability on those prefixes, and the continuation probe compares truncated and concluded outputs on different problems\. With four seeds per point, several secondary differences depend on how the interval is formed \(Appendix[H](https://arxiv.org/html/2610.20989#A8)\), and the SFT training length was selected on the test set\. The unfiltered control removes both filters and its data carry an appended end\-of\-turn token; the completeness\-only control separates the filters, not the token, and has more demonstrations\. Truncation is inferred by re\-tokenising \(validated in Appendix[D](https://arxiv.org/html/2610.20989#A4)\); token roles are heuristic\.

### AI use statement

We used AI coding assistants \(Claude, Anthropic\) to assist in this project\. It assisted with writing and refactoring training, evaluation, and analysis code under the authors’ direction, drafted the documentation and the prose of this manuscript, which the authors revised, and assisted with literature search\. Every experiment was designed by the authors and run by them on their own compute; every number in the paper is read from the released per\-question evaluation records, and the statistical procedures were checked against independent implementations in the test suite\. The authors reviewed all generated code and text and take full responsibility for the content\.

### Ethics statement

We strictly adhere to the ICLR Code of Ethics\. This work distills publicly released models on public benchmarks and introduces no new data collection\. The distilled students inherit the biases and failure modes of the teacher, which is one of the paper’s findings; we do not release them as general\-purpose assistants\.

### Reproducibility statement

All code, configurations, Slurm scripts, per\-step training logs, and per\-question evaluation records \(prediction, gold answer, correctness, and completion for every test question of every model\) are released at[https://github\.com/ffy208/opd\-inherits\-teacher\-vlm](https://github.com/ffy208/opd-inherits-teacher-vlm); every mean, interval, and paired difference in the paper is recomputed from the records by one script\. Data order, rollout sampling, and LoRA initialisation are seeded, so a rerun follows the same recipe, but GPU kernels and vLLM decoding are not bit\-reproducible across hardware, and re\-evaluating the same weights can move a single seed’s accuracy by a few tenths of a point on ChartQA and by up to2\.82\.8points on Geometry3K \(Section[3](https://arxiv.org/html/2610.20989#S3)\)\. The released evaluator records each generation’s stop reason and token count, which the records used here lack \(Appendix[D](https://arxiv.org/html/2610.20989#A4)\)\. Appendix[A](https://arxiv.org/html/2610.20989#A1)lists every hyperparameter, the test\-subset construction, the answer parser, and the end\-of\-turn convention; Appendix[N](https://arxiv.org/html/2610.20989#A14)lists the engineering choices needed to train a VLM student and teacher on one 40 GB GPU\.

### Acknowledgments

We thank Geyang Guo for her detailed feedback and helpful suggestions for improving the manuscript\. We acknowledge computational resources and services provided by the AI Makerspace of the College of Engineering \(RRID:SCR\_028058\) and the Partnership for an Advanced Computing Environment \(PACE\) at the Georgia Institute of Technology, Atlanta, Georgia, USA \(RRID:SCR\_027619\)\.

## References

- Agarwalet al\.\(2024\)R\. Agarwal, N\. Vieillard, Y\. Zhou, P\. Stanczyk, S\. Ramos, M\. Geist, and O\. BachemOn\-policy distillation of language models: learning from self\-generated mistakes\.InInternational Conference on Learning Representations \(ICLR\),External Links:2306\.13649Cited by:[§1](https://arxiv.org/html/2610.20989#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.20989#S2.SS1.p1.2),[§4\.1](https://arxiv.org/html/2610.20989#S4.SS1.p1.1),[§5](https://arxiv.org/html/2610.20989#S5.SS0.SSS0.Px1.p1.1)\.
- Ai and Yan \(2026\)Y\. Ai and W\. YanTrajectory learnability for offline on\-policy distillation with imperfect teachers\.arXiv preprint arXiv:2609\.18321\.Cited by:[§5](https://arxiv.org/html/2610.20989#S5.SS0.SSS0.Px1.p1.1)\.
- Baiet al\.\(2025\)S\. Baiet al\.Qwen3\-VL technical report\.arXiv preprint arXiv:2511\.21631\.Cited by:[§1](https://arxiv.org/html/2610.20989#S1.p4.1),[§3](https://arxiv.org/html/2610.20989#S3.SS0.SSS0.Px1.p1.1)\.
- Bousselhamet al\.\(2026\)W\. Bousselham, H\. Kuehne, and C\. SchmidVOLD: reasoning transfer from LLMs to vision\-language models via on\-policy distillation\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 26209–26218\.Cited by:[§5](https://arxiv.org/html/2610.20989#S5.SS0.SSS0.Px2.p1.1)\.
- Ding and Zhang \(2026\)Y\. Ding and R\. ZhangDoes on\-policy distillation really distill? From noisy teacher to self\-improvement\.arXiv preprint arXiv:2608\.31046\.Cited by:[§5](https://arxiv.org/html/2610.20989#S5.SS0.SSS0.Px1.p1.1)\.
- Guet al\.\(2024\)Y\. Gu, L\. Dong, F\. Wei, and M\. HuangMiniLLM: knowledge distillation of large language models\.InInternational Conference on Learning Representations \(ICLR\),External Links:2306\.08543Cited by:[§1](https://arxiv.org/html/2610.20989#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.20989#S2.SS1.p1.2),[§5](https://arxiv.org/html/2610.20989#S5.SS0.SSS0.Px1.p1.1)\.
- Hintonet al\.\(2015\)G\. Hinton, O\. Vinyals, and J\. DeanDistilling the knowledge in a neural network\.arXiv preprint arXiv:1503\.02531\.Cited by:[§2\.1](https://arxiv.org/html/2610.20989#S2.SS1.p1.1)\.
- Hoet al\.\(2023\)N\. Ho, L\. Schmid, and S\. YunLarge language models are reasoning teachers\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),External Links:2212\.10071Cited by:[§1](https://arxiv.org/html/2610.20989#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.20989#S2.SS1.p1.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations \(ICLR\),External Links:2106\.09685Cited by:[§3](https://arxiv.org/html/2610.20989#S3.SS0.SSS0.Px1.p1.1)\.
- Koet al\.\(2024\)J\. Ko, S\. Kim, T\. Chen, and S\. YunDistiLLM: towards streamlined distillation for large language models\.InInternational Conference on Machine Learning \(ICML\),External Links:2402\.03898Cited by:[§5](https://arxiv.org/html/2610.20989#S5.SS0.SSS0.Px1.p1.1)\.
- Kwonet al\.\(2023\)W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. StoicaEfficient memory management for large language model serving with PagedAttention\.InProceedings of the 29th Symposium on Operating Systems Principles \(SOSP\),External Links:2309\.06180Cited by:[Appendix N](https://arxiv.org/html/2610.20989#A14.p1.1)\.
- Liet al\.\(2026\)Y\. Li, Y\. Zuo, B\. He, J\. Zhang, C\. Xiao, C\. Qian, T\. Yu, H\. Gao, W\. Yang, Z\. Liu, and N\. DingRethinking on\-policy distillation of large language models: phenomenology, mechanism, and recipe\.arXiv preprint arXiv:2604\.13016\.Cited by:[§5](https://arxiv.org/html/2610.20989#S5.SS0.SSS0.Px1.p1.1)\.
- Linet al\.\(2020\)A\. Lin, J\. Wohlwend, H\. Chen, and T\. LeiAutoregressive knowledge distillation through imitation learning\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),External Links:[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.494),2009\.07253Cited by:[§5](https://arxiv.org/html/2610.20989#S5.SS0.SSS0.Px1.p1.1)\.
- Liuet al\.\(2026\)R\. Liu, X\. Lv, G\. Li, X\. Zhu, Z\. Wang, Z\. Zhang, J\. Chen, Z\. Li, B\. Li, J\. Gao, and S\. WuVisual\-advantage on\-policy distillation for vision\-language models\.arXiv preprint arXiv:2605\.21924\.Cited by:[§1](https://arxiv.org/html/2610.20989#S1.p1.1),[§5](https://arxiv.org/html/2610.20989#S5.SS0.SSS0.Px2.p1.1)\.
- Loshchilov and Hutter \(2019\)I\. Loshchilov and F\. HutterDecoupled weight decay regularization\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[Table 3](https://arxiv.org/html/2610.20989#A1.T3.2.6.2.1.1)\.
- Lu and Thinking Machines Lab \(2025\)K\. Lu and Thinking Machines LabOn\-policy distillation\.Thinking Machines Lab: Connectionism\.Note:Blog post, 27 October 2025External Links:[Document](https://dx.doi.org/10.64434/tml.20251026),[Link](https://thinkingmachines.ai/blog/on-policy-distillation/)Cited by:[§1](https://arxiv.org/html/2610.20989#S1.p1.1)\.
- Luet al\.\(2021\)P\. Lu, R\. Gong, S\. Jiang, L\. Qiu, S\. Huang, X\. Liang, and S\. ZhuInter\-GPS: interpretable geometry problem solving with formal language and symbolic reasoning\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics \(ACL\),External Links:2105\.04165Cited by:[§1](https://arxiv.org/html/2610.20989#S1.p4.1),[§3](https://arxiv.org/html/2610.20989#S3.SS0.SSS0.Px1.p1.1)\.
- Ma \(2026\)G\. MaOutcome\-confounded local supervision in on\-policy distillation\.arXiv preprint arXiv:2607\.23731\.Cited by:[§5](https://arxiv.org/html/2610.20989#S5.SS0.SSS0.Px1.p1.1)\.
- Magisteret al\.\(2023\)L\. C\. Magister, J\. Mallinson, J\. Adamek, E\. Malmi, and A\. SeverynTeaching small language models to reason\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),External Links:2212\.08410Cited by:[§1](https://arxiv.org/html/2610.20989#S1.p1.1)\.
- Masryet al\.\(2025\)A\. Masry, M\. S\. Islam, M\. Ahmed, A\. Bajaj, F\. Kabir, A\. Kartha, M\. T\. R\. Laskar, M\. Rahman, S\. Rahman, M\. Shahmohammadi, M\. Thakkar, M\. R\. Parvez, E\. Hoque, and S\. JotyChartQAPro: a more diverse and challenging benchmark for chart question answering\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 19123–19151\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.978)Cited by:[§3](https://arxiv.org/html/2610.20989#S3.SS0.SSS0.Px1.p1.1)\.
- Masryet al\.\(2022\)A\. Masry, D\. X\. Long, J\. Q\. Tan, S\. Joty, and E\. HoqueChartQA: a benchmark for question answering about charts with visual and logical reasoning\.InFindings of the Association for Computational Linguistics: ACL 2022,External Links:2203\.10244Cited by:[§1](https://arxiv.org/html/2610.20989#S1.p4.1),[§3](https://arxiv.org/html/2610.20989#S3.SS0.SSS0.Px1.p1.1)\.
- Methaniet al\.\(2020\)N\. Methani, P\. Ganguly, M\. M\. Khapra, and P\. KumarPlotQA: reasoning over scientific plots\.InIEEE Winter Conference on Applications of Computer Vision \(WACV\),External Links:[Document](https://dx.doi.org/10.1109/WACV45572.2020.9093523),1909\.00997Cited by:[§3](https://arxiv.org/html/2610.20989#S3.SS0.SSS0.Px2.p1.1)\.
- Qiuet al\.\(2026\)C\. Qiu, D\. Li, Y\. Zhang, L\. Gong, and Z\. TanOPD\-Aha: from linguistic momentum to visual reflection in multimodal on\-policy distillation\.arXiv preprint arXiv:2609\.16459\.Cited by:[§1](https://arxiv.org/html/2610.20989#S1.p2.1),[§5](https://arxiv.org/html/2610.20989#S5.SS0.SSS0.Px1.p1.1)\.
- Ross and Bagnell \(2010\)S\. Ross and D\. BagnellEfficient reductions for imitation learning\.InProceedings of the 13th International Conference on Artificial Intelligence and Statistics \(AISTATS\),Proceedings of Machine Learning Research, Vol\.9,pp\. 661–668\.Cited by:[§1](https://arxiv.org/html/2610.20989#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.20989#S2.SS1.p2.1)\.
- Rosset al\.\(2011\)S\. Ross, G\. J\. Gordon, and J\. A\. BagnellA reduction of imitation learning and structured prediction to no\-regret online learning\.InProceedings of the 14th International Conference on Artificial Intelligence and Statistics \(AISTATS\),Proceedings of Machine Learning Research, Vol\.15,pp\. 627–635\.External Links:1011\.0686Cited by:[§1](https://arxiv.org/html/2610.20989#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.20989#S2.SS1.p2.1)\.
- Sunet al\.\(2026\)C\. Sun, L\. Liu, H\. Lei, T\. Ling, J\. Xie, Z\. Zheng, Y\. Wang, H\. Liu, F\. Xiao, L\. Liu, Y\. Du, Z\. Cheng, Z\. Jiang, and Q\. GuStep\-level on\-policy distillation: interpolating between on\-policy distillation and supervised fine\-tuning\.arXiv preprint arXiv:2608\.16333\.Cited by:[§1](https://arxiv.org/html/2610.20989#S1.p2.1),[§5](https://arxiv.org/html/2610.20989#S5.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2024\)Z\. Wang, M\. Xia, L\. He, H\. Chen, Y\. Liu, R\. Zhu, K\. Liang, X\. Wu, H\. Liu, S\. Malladi, A\. Chevalier, S\. Arora, and D\. ChenCharXiv: charting gaps in realistic chart understanding in multimodal LLMs\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:2406\.18521Cited by:[§3](https://arxiv.org/html/2610.20989#S3.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2026\)Z\. Wang, S\. Huang, W\. Xu, B\. Wen, and X\. JiangVideo\-OPSD: exploiting privileged visual evidence for on\-policy self\-distillation in video large language models\.arXiv preprint arXiv:2608\.27065\.Cited by:[§5](https://arxiv.org/html/2610.20989#S5.SS0.SSS0.Px2.p1.1)\.
- Xianget al\.\(2026\)S\. Xiang, K\. An, W\. Yu, Y\. Liu, J\. Luan, P\. Fu, and Q\. WangTeaching the way, not the answer: privileged tutoring distillation for multimodal policy optimization\.arXiv preprint arXiv:2606\.07000\.Cited by:[§5](https://arxiv.org/html/2610.20989#S5.SS0.SSS0.Px2.p1.1)\.
- Xiaoet al\.\(2026\)H\. Xiao, Y\. Niu, D\. Liu, C\. Luo, and J\. LiTV\-Regulated OPD: direction matters in on\-policy distillation\.arXiv preprint arXiv:2609\.08341\.Cited by:[§5](https://arxiv.org/html/2610.20989#S5.SS0.SSS0.Px1.p1.1)\.
- Xieet al\.\(2026\)Y\. Xie, S\. Zhu, T\. Wen, B\. Chen, and Y\. WangOn the position bias of on\-policy distillation\.arXiv preprint arXiv:2606\.22600\.Cited by:[§5](https://arxiv.org/html/2610.20989#S5.SS0.SSS0.Px1.p1.1)\.
- Xueet al\.\(2026\)L\. Xue, F\. Xiong, M\. Ma, and C\. ZhangDistill what the student can see: fisher\-projected on\-policy distillation for vision\-language models\.arXiv preprint arXiv:2608\.01263\.Cited by:[§1](https://arxiv.org/html/2610.20989#S1.p1.1),[§5](https://arxiv.org/html/2610.20989#S5.SS0.SSS0.Px2.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§1](https://arxiv.org/html/2610.20989#S1.p1.1)\.
- Yoonet al\.\(2026\)H\. S\. Yoon, E\. Yoon, J\. Jang, S\. Eom, J\. W\. Hong, M\. Hasegawa\-Johnson, Q\. Dai, C\. Luo, and C\. D\. YooDecomposed on\-policy distillation for vision\-language reasoning: steering gradients for visual grounding\.InProceedings of the 43rd International Conference on Machine Learning \(ICML\),Note:arXiv:2606\.00564Cited by:[§1](https://arxiv.org/html/2610.20989#S1.p1.1),[§5](https://arxiv.org/html/2610.20989#S5.SS0.SSS0.Px2.p1.1)\.
- Yuanet al\.\(2026\)Q\. Yuan, J\. Lou, X\. Yu, H\. Lin, L\. Sun, X\. Han, and Y\. LuVision\-OPD: learning to see fine details for multimodal LLMs via on\-policy self\-distillation\.arXiv preprint arXiv:2605\.18740\.Cited by:[§5](https://arxiv.org/html/2610.20989#S5.SS0.SSS0.Px2.p1.1)\.
- Yuanet al\.\(2023\)Z\. Yuan, H\. Yuan, C\. Li, G\. Dong, K\. Lu, C\. Tan, C\. Zhou, and J\. ZhouScaling relationship on learning mathematical reasoning with large language models\.arXiv preprint arXiv:2308\.01825\.Cited by:[§2\.1](https://arxiv.org/html/2610.20989#S2.SS1.p1.1)\.
- Zhanget al\.\(2026\)Z\. Zhang, Z\. Sun, F\. Zhao, K\. Peng, B\. Liang, H\. Deng, Y\. Hu, K\. Wong, and M\. ChuanVerify before you distill: prompt\-level teacher gating for on\-policy distillation\.arXiv preprint arXiv:2609\.02998\.Cited by:[§5](https://arxiv.org/html/2610.20989#S5.SS0.SSS0.Px1.p1.1)\.
- Zhuet al\.\(2026\)Q\. Zhu, Y\. Wang, Z\. Wen, T\. Zhang, M\. Zhang, Y\. Liu, S\. Chen, S\. Wu, J\. Yang, and X\. JiangRP\-OPSD: resolution\-privileged on\-policy self\-distillation for multimodal large language models\.arXiv preprint arXiv:2607\.24447\.Cited by:[§5](https://arxiv.org/html/2610.20989#S5.SS0.SSS0.Px2.p1.1)\.

## Appendix AHyperparameters and evaluation details

Table[3](https://arxiv.org/html/2610.20989#A1.T3)lists every training, sampling, and evaluation setting; the end\-of\-turn convention and the answer parser are the ones the loss mask and the scorer use\.

Table 3:Hyperparameters, data construction, and evaluation settings\.Student / teacherQwen3\-VL\-2B\-Instruct / Qwen3\-VL\-8B\-Instruct \(bf16\); Qwen3\-VL\-32B\-Instruct in Section[4\.4](https://arxiv.org/html/2610.20989#S4.SS4)LoRArank 64,α=128\\alpha=128, language\-model q/k/v/o and MLP projections, vision encoder and projector frozenData500 test questions sampled with seed 42 from the official test split \(human\-written ChartQA questions; the 601 Geometry3K test problems\); training budgets are nested prefixes of a seeded shuffle of the training split; images resized to a longest side of 768 px \(300 to 600 visual tokens\)SFT302 steps, batch 4×\\timesaccumulation 4, lr10−410^\{\-4\}, cosine, max sequence length 2,048; loss on the response tokens only; an end\-of\-turn token is appended to every demonstration before the sequence is cut to 2,048 tokens \(the prompt is never cut\)SFT dataone teacher sample per question atT=0\.7T\{=\}0\.7, kept if its answer is correct \(2,402 of 3,000 on ChartQA; 1,163 of 2,101 on Geometry3K, of whose samples 78% contain an answer line\); the unfiltered control keeps every first sample; the complete\-only control keeps the first sample when it contains an answer line, correct or not \(1,645 of 2,101\)OPD150 steps, 16 rollouts per step, micro\-batch 4, lr5⋅10−55\\cdot 10^\{\-5\}, 20 warmup steps, cosine, AdamW\([Loshchilov and Hutter, 2019](https://arxiv.org/html/2610.20989#bib.bib35)\)without weight decay, grad clip 1\.0Rollouts<\|im\_end\|\>is the end\-of\-turn token; a rollout ends at its first occurrence or at the budget of 512 new tokens \(charts\) / 1,536 \(geometry\);T=1\.0T\{=\}1\.0, top\-kkoffDivergencereverse KL over the full vocabulary \(end\-of\-turn token included\) in fp32 chunks, at every generated position up to and including the end\-of\-turn token; loss is the mean over all such positions of the batch \(Equation[1](https://arxiv.org/html/2610.20989#S2.E1)\); teacher and student score the same prefixes with the same attention maskVariantsone change each: OPD\-term \(loss on rollouts that emitted end\-of\-turn only\), OPD\-t07 \(rollouts atT=0\.7T\{=\}0\.7\), OPD\-fwd \(forward KL\), OPD\-sftinit \(student initialised from the merged SFT model of the same seed\), OPD\-seqmean \(sequence\-mean loss1B​∑i1\|y\(i\)\|​∑tDt\(i\)\\frac\{1\}\{B\}\\sum\_\{i\}\\frac\{1\}\{\|y^\{\(i\)\}\|\}\\sum\_\{t\}D^\{\(i\)\}\_\{t\}over theB=16B\{=\}16rollouts of a step, so that every rollout counts once instead of once per token, in place of the token mean of Equation[1](https://arxiv.org/html/2610.20989#S2.E1)\), SFT complete\-only \(the data above\), SFT\-nn\(nnsteps instead of 302\)EvaluationvLLM greedy decoding, the task’s token budget \(4,096 in Appendix[E](https://arxiv.org/html/2610.20989#A5)\); the answer is the last line matchingAnswer:; relaxed accuracy with 5% relative numeric tolerance \(exact match for a gold value of 0; normalised string match for text answers\); radicals, fractions, andπ\\piare evaluated numericallyComputeone A100 40 GB or L40S 48 GB per point; OPD 13\.5 s/step \(charts\), 85 s/step \(geometry\)#### Prompts\.

Every model receives one image and the following user message \(the ChartQA version replaces the first two sentences by “Look at the chart and answer the question below\. Reason step by step: first read the relevant values from the chart, then do any calculation needed\.”\); the student always sees the geometry version\.

> Look at the geometry figure and solve the problem below\. Reason step by step: first state the given measures and relations you can read from the figure, then apply the relevant theorems and compute\. Give the final answer as a number \(decimals are fine; simplify radicals or fractions to a decimal if needed\)\. \[*guidance or paraphrase inserted here*\]Keep the reasoning concise\. On the last line, write the final answer in exactly this format: Answer: <your answer\> Question: \{question\}\[*privileged suffix appended here*\]

The completion guidance inserted before “Keep the reasoning concise” is: “Do not repeat a step you have already completed or restart the solution; once the value is determined, stop reasoning and commit to it, and always finish with the answer line\.” The length\-matched paraphrase replaces the first two sentences by “Study the geometry figure and work out the problem below\. Proceed step by step: begin with the measures and relationships shown in the figure, then use the appropriate theorems to compute the result\. Report the final answer as a number \(decimals are acceptable; convert radicals or fractions to a decimal when needed\)\.” and inserts “Name each quantity you use as you introduce it, so that the intermediate values of the solution can be read directly from the reasoning, and give the answer in the units of the figure\.” at the same place\. The privileged suffix is “The correct final answer is known to be: \{answer\}\. Write out the reasoning that leads to it, and finish with the same ‘Answer: ¡your answer¿’ line\.” In the continuation probe the prompt is followed by the assistant turn containing the student’s prefix, which the teacher continues greedily\.

## Appendix BFeedback dynamics

Figure[4](https://arxiv.org/html/2610.20989#A2.F4)gives the per\-role feedback log of Section[4\.2](https://arxiv.org/html/2610.20989#S4.SS2)on ChartQA and Figure[5](https://arxiv.org/html/2610.20989#A2.F5)repeats the analysis for the Geometry3K runs\. We also tested whether the end\-of\-training KL share on numeric tokens predicts a run’s remaining gap to the teacher across runs: with 9 runs on ChartQA and 11 on Geometry3K the Spearman correlations are−0\.13\-0\.13\[−0\.83\-0\.83,\+0\.73\+0\.73\] and\+0\.66\+0\.66\[\+0\.07\+0\.07,\+0\.89\+0\.89\]; the Geometry3K interval excludes zero narrowly, but it is one of five statistics we examined and the runs’ accuracies within a task are too close for this test to have power, so we do not claim it\.

Figure 4:Feedback dynamics on ChartQA over the nine OPD runs with a complete per\-role training log \(100 questions: four seeds; 300: seeds 1–3; 900: seeds 1–2; the 300\-question seed\-42 run predates per\-role logging and the other logs stop at an earlier checkpoint on the Hub; windows of 10 steps, shaded:±1\\pm 1s\.d\. over runs\)\. \(a\) KL concentration per token role \(share of the batch’s KL divided by share of tokens; dotted line:11\); \(b\) mean per\-token KL; \(c\) rollout termination rate\.Figure 5:As Figure[4](https://arxiv.org/html/2610.20989#A2.F4), for the eleven OPD runs on Geometry3K with a complete per\-role log \(300 and 900 problems: four seeds each; 100 problems: seeds 1 and 42; 2,101 problems: seed 42; the Hub copies of the other 100\- and 2,101\-problem logs stop at an earlier checkpoint\)\. The answer\-line concentration falls from0\.450\.45to0\.280\.28; numeric values stay between0\.480\.48and0\.600\.60; the rollout termination rate \(c\) stays between0\.470\.47and0\.610\.61over 150 steps\.
## Appendix CToken\-level feedback before and after OPD

Figure[6](https://arxiv.org/html/2610.20989#A3.F6)shows the per\-token divergence on one test question before and after training, and Figure[7](https://arxiv.org/html/2610.20989#A3.F7)the role breakdown over 100 test questions that the concentration statistic of Section[3](https://arxiv.org/html/2610.20989#S3)summarises\.

![Refer to caption](https://arxiv.org/html/2610.20989v1/lab_fig_heatmap.png)Figure 6:Per\-token reverse KL between teacher and student on one ChartQA test question \(rollouts at temperature 1\.0\), for the zero\-shot student \(top\) and the OPD student trained on 300 questions \(bottom\), same colour scale; the underline gives the token’s role\. Before training the opening of the answer line carries a large divergence; after training it carries almost none and the residual disagreement is diffuse \(here largest on a wording choice\)\. Numbers are tokenised digit by digit and the disagreement on a number sits on its first digit\.Figure 7:Share of tokens against share of the teacher’s KL mass per role over the rollouts on 100 ChartQA test questions \(11,625 and 12,842 tokens\), for the zero\-shot student \(a\) and the OPD\-300 student \(b, seed 1\); the number above each bar is the concentration \(KL share divided by token share\)\. Mean per\-token KL falls from0\.470\.47to0\.230\.23\.
## Appendix DCompletion, termination, and truncation

#### How truncation is identified\.

The evaluation records used in this paper store each model’s output text but not the generator’s stop reason\. Truncation is therefore inferred: the output is re\-tokenised with the student’s tokenizer and counted as truncated if it lies within 8 tokens of the generation budget \(the re\-tokenised lengths of truncated outputs cluster at the budget minus 0 to 2 tokens; only one output in the 900\-problem OPD runs falls between 8 and 40 tokens below it\)\. “Stops” in Table[8](https://arxiv.org/html/2610.20989#A4.T8)means not truncated by this criterion; it is an approximation of an emitted end\-of\-turn token, not a direct observation of one\. The released evaluator now records the stop reason and output token count of every generation, so future records need no inference\. Re\-evaluating the 900\-problem OPD and SFT students of seed 1 with that evaluator \(models rebuilt from the Hub checkpoint and adapter; accuracy0\.4220\.422and0\.4860\.486against0\.4060\.406and0\.4720\.472in the recorded pass\), the criterion agrees with the recorded stop reason on 500 of 500 and 498 of 500 responses: it calls 183 and 120 responses truncated, the generator 183 and 118; the two disagreements are SFT responses that stopped within 8 tokens of the budget, and no truncated response is missed\.

#### Teacher probe\.

Table[6](https://arxiv.org/html/2610.20989#A4.T6)gives the teacher\-forced probe: a pass of the 8B teacher over every stored test output of a student \(900\-problem models, four seeds each; the zero\-shot student is one model\), recording at every prefix length its probability of the end\-of\-turn token and of a token that opens an answer line: the three tokensAnswer,Answerwith a leading space, and\*\*, which are the first tokens ofAnswer,Answer,\*\*Answer, andAnswer:and the token after the newline in\\nAnswer\. This mass is a proxy for opening an answer line, not the probability of one:\*\*also opens ordinary bold text, and an answer line needs the following tokens as well\. The probabilities are computed in bf16 and summarised \(maximum, mean, value at the stop position\) at full precision; only the stored per\-position series are rounded to four decimals\. “Max EOS” in Table[6](https://arxiv.org/html/2610.20989#A4.T6)is the maximum over the positions of one output, averaged over outputs; the largest value at any position of any truncated output is4⋅10−54\\cdot 10^\{\-5\}\(zero\-shot\),8⋅10−68\\cdot 10^\{\-6\}\(SFT\),5⋅10−55\\cdot 10^\{\-5\}\(OPD\), and6⋅10−46\\cdot 10^\{\-4\}\(OPD\-sftinit\), so no truncated output has a position above10−310^\{\-3\}\.

Table 4:Continuation probe on Geometry3K: share of the teacher’s greedy continuations that reach an answer line, from prefixes of a student’s truncated outputs \(cut at 256, 512, and 1,024 tokens and at the truncation point\) and, for comparison, from 256\-token prefixes of its concluded outputs; the rest of the 1,536\-token budget is 1,280, 1,024, 512, and 512 tokens\. Seeds 1 and 42 of the 900\-problem students pooled;nn: truncated outputs in parentheses, and 203, 435, and 320 concluded outputs of the zero\-shot, SFT, and OPD students; brackets: 95% question\-clustered bootstrap intervals\.Table[4](https://arxiv.org/html/2610.20989#A4.T4)and Table[7](https://arxiv.org/html/2610.20989#A4.T7)give the continuation probe: the teacher continues each output greedily from a prefix of 256, 512, or 1,024 tokens with the rest of the 1,536\-token budget, at least 512 tokens \(so 1,280, 1,024, and 512\), and from the truncation point with 512 more tokens, and a second pass gives every cut the same 512 tokens \(lower block of Table[4](https://arxiv.org/html/2610.20989#A4.T4)\); prefix plus continuation is scored like a response\. Per seed, the fixed\-budget completion rate from 256 tokens is0\.160\.16/0\.160\.16for the two OPD students and0\.120\.12/0\.160\.16for the two SFT students\. Prefixes that already contain an answer line are not continued \(nine prefixes of concluded outputs; none of a truncated output\)\. The intervals of Table[4](https://arxiv.org/html/2610.20989#A4.T4)are question\-clustered percentile bootstraps over the pooled seeds \(the two outputs of a problem are resampled together\), and the paired differences between cuts keep each trajectory’s pairing; per seed, the completion rate from the truncation point is0\.020\.02and0\.040\.04for the two OPD students and0\.080\.08and0\.130\.13for the two SFT students, and from 256 tokens0\.340\.34/0\.330\.33and0\.260\.26/0\.370\.37\.

#### Prompt\-level interventions on the teacher\.

Table[5](https://arxiv.org/html/2610.20989#A4.T5)repeats the fixed\-budget continuation with the teacher under three other prompts, on the outputs of the zero\-shot student for the first 300 training problems \(91 truncated, 209 concluded; the test set is not used to design an intervention, and these are not the trained students’ test outputs of Table[4](https://arxiv.org/html/2610.20989#A4.T4)\): the geometry prompt with explicit completion guidance appended, a length\-matched paraphrase of the geometry prompt without that guidance, and the geometry prompt with the gold answer appended \(privileged information\); the full texts are in Appendix[A](https://arxiv.org/html/2610.20989#A1)\. The student’s prompt is unchanged in every case\. Paired over the same prefixes, completion guidance minus the original prompt is\+1\.1\+1\.1\[−3\.3\-3\.3,\+5\.5\+5\.5\] points of completion at the 256\-token cut and\+2\.2\+2\.2\[−2\.2\-2\.2,\+6\.6\+6\.6\] at 512, and−1\.1\-1\.1\[−3\.3\-3\.3,\+0\.0\+0\.0\] and\+1\.1\+1\.1\[\+0\.0\+0\.0,\+3\.3\+3\.3\] points of accuracy; guidance minus the paraphrase is\+1\.1\+1\.1\[−7\.7\-7\.7,\+9\.9\+9\.9\] and\+2\.2\+2\.2\[\+0\.0\+0\.0,\+5\.5\+5\.5\] of completion,\+1\.1\+1\.1\[−2\.2\-2\.2,\+4\.4\+4\.4\] and\+1\.1\+1\.1\[\+0\.0\+0\.0,\+3\.3\+3\.3\] of accuracy: no improvement is detected; at the 256\-token cut the interval against the original prompt excludes improvements above 5\.5 points, and the less precise comparison with the paraphrase excludes only improvements above 9\.9 points\. The gold answer lowers completion by11\.011\.0points \[−18\.7\-18\.7,−4\.4\-4\.4\] at 256 tokens and from0\.780\.78to0\.540\.54on the concluded outputs; 85 of the 91 continuations run to the budget \(75 under the original prompt\)\. Reading the continuations, 37 of the 91 re\-examine an earlier step \(22 under the original prompt\), 20 note a contradiction between the prefix and what they derive \(15\), and 6 refer to the given answer; for example, one continuation computes the altitude of a right triangle, then writes “Wait, that’s not correct\. The altitude to the hypotenuse is the geometric mean of the two segments only if the altitude is drawn to the hypotenuse” and restarts\. Re\-derivation is thus a plausible reading of the drop, not a demonstrated cause\. Because no tested prompt established the improvement in teacher recovery that the training\-level intervention \(the same students trained under such a teacher\) was to build on, that intervention was not run; it remains open whether a teacher trained to conclude from these states transmits completion through on\-policy distillation\.

Table 5:Fixed\-budget continuation \(512 tokens\) by the teacher under four prompts, from prefixes of the zero\-shot student’s outputs on the first 300 training problems of Geometry3K: completion / accuracy on the 91 truncated outputs \(brackets: 95% bootstrap interval of the completion rate over outputs, one per problem\) and on the outputs that concluded \(nnin parentheses; concluded outputs shorter than a cut are not continued\)\. Paired differences between prompts are in the text\.Table 6:Teacher\-forced probe on Geometry3K test outputs \(mean over seeds;nnper 500\)\. For outputs that conclude: the teacher’s end\-of\-turn probability at the position where the student stopped \(“EOS at stop”\), the share of outputs on which it exceeds0\.50\.5, and its probability mass on answer\-opening tokens \(a proxy, see text\) where the student’s answer line begins \(“answer at onset”\)\. For outputs that ran to the budget: the maximum end\-of\-turn probability over the positions of an output, averaged over outputs \(no single position exceeds10−310^\{\-3\}\), and the mean over outputs of the maximum answer\-opening probability \(above0\.50\.5on 10 to 12% of outputs\)\.Table 7:Continuation probe on Geometry3K \(900\-problem SFT and OPD students, seeds 1 and 42 pooled; the zero\-shot student\): completion / accuracy of the teacher’s continuation from a prefix of the student’s output, split by whether the student’s own output concluded, with the number of outputs continued at each cut \(concluded outputs shorter than a cut, and prefixes that already contain an answer line, are not continued\)\. The truncated rows are those of the upper block of Table[4](https://arxiv.org/html/2610.20989#A4.T4)\.
#### Cross\-tabulation\.

Table[8](https://arxiv.org/html/2610.20989#A4.T8)cross\-tabulates, for every model, whether a test response contains an answer line and whether it was truncated\. For every correctness\-filtered SFT and OPD model, including the OPD variants, a response without an answer line is a truncated one and a response that stops has an answer line, up to a few responses per 500 that give an answer and then continue to the budget; the exceptions are the two SFT data controls, which stop without an answer on 70 \(unfiltered\) and 10 \(complete\-only\) of 500 responses at 300 problems and on 7 and 0\.5 at 900 \(Section[4\.3](https://arxiv.org/html/2610.20989#S4.SS3)\)\. Table[9](https://arxiv.org/html/2610.20989#A4.T9)splits each student’s completion rate by whether the greedy teacher concludes the same test problem, and Table[10](https://arxiv.org/html/2610.20989#A4.T10)decomposes each paired accuracy difference by which method concluded a problem\.

Table 8:Geometry3K, mean counts per 500 test responses over four seeds \(single models for the zero\-shot rows\)\. “Stops” means not truncated by the re\-tokenisation criterion; “at budget” means truncated\.Table 9:Geometry3K, 900 problems: each student’s completion rate on the 391 test problems the greedy teacher concludes within 1,536 tokens and on the 109 it does not \(mean over four seeds; the zero\-shot student is one model\)\. At 4,096 tokens the teacher concludes 422 problems\.Table 10:Geometry3K: decomposition of the paired accuracy difference \(percentage points of the 500 test problems, mean over four seeds\) by which method concluded a problem\. A problem only one method concludes counts against the other, which scores it wrong;nnis the mean number of problems in each group\. The three components and the total are rounded independently, so the sum of the components may differ from the total by0\.10\.1\.

## Appendix EGeneration budget

Table[11](https://arxiv.org/html/2610.20989#A5.T11)re\-scores the same four 900\-problem models per method at 4,096 generated tokens\. The 1,536 column is a separate evaluation pass of the same weights as Table[2](https://arxiv.org/html/2610.20989#S4.T2); the four\-seed means differ by0\.10\.1\(SFT\) and0\.40\.4\(OPD\) points and individual seeds by up to2\.82\.8points, because greedy decoding of long solutions is not bit\-reproducible across hardware and batch composition\.

Table 11:Geometry3K, 900 problems, the same four models per method scored at two generation budgets \(means with question\-clustered intervals\)\. The SFT minus OPD gap grows with the budget because OPD students do not finish when given more room\.
## Appendix FControl and variants at 900 and 300 problems

Table 12:Geometry3K, 900 problems, four seeds each \(300\-problem rows in Appendix[F](https://arxiv.org/html/2610.20989#A6)\): the three SFT data sets \(filtered: correct teacher samples, the reference; unfiltered: every teacher sample; complete\-only: the samples that contain an answer line, correct or not\) and plain OPD with its five variants \(defined in Section[4\.3](https://arxiv.org/html/2610.20989#S4.SS3)and Appendix[A](https://arxiv.org/html/2610.20989#A1)\)\. Paired differences against the filtered SFT, with clustered and seed\-levelttintervals; bold: intervals that exclude zero\.Table 13:Geometry3K, 300 problems, four seeds each: the three SFT data sets, plain OPD, and OPD\-term, as in Table[12](https://arxiv.org/html/2610.20989#A6.T12)\. No interval excludes zero\.vs\. filtered SFTVariantAccuracyCompletionpaired, clustered 95%ttover seedsSFTfiltered0\.4200\.420\(\.013\)0\.730\.73unfiltered0\.4020\.402\(\.016\)0\.620\.62−0\.018\-0\.018\[−0\.042\-0\.042,\+0\.007\+0\.007\]\[−0\.064\-0\.064,\+0\.028\+0\.028\]complete\-only0\.4320\.432\(\.011\)0\.720\.72\+0\.012\+0\.012\[−0\.010\-0\.010,\+0\.036\+0\.036\]\[−0\.021\-0\.021,\+0\.046\+0\.046\]OPDplain0\.4190\.419\(\.019\)0\.620\.62−0\.001\-0\.001\[−0\.024\-0\.024,\+0\.022\+0\.022\]\[−0\.020\-0\.020,\+0\.019\+0\.019\]term0\.4010\.401\(\.009\)0\.620\.62−0\.019\-0\.019\[−0\.040\-0\.040,\+0\.003\+0\.003\]\[−0\.039\-0\.039,\+0\.001\+0\.001\]
## Appendix GOut\-of\-distribution results and the 32B teacher

With Qwen3\-VL\-32B\-Instruct as the teacher on 300 ChartQA questions \(four seeds, same protocol\), the teacher’s zero\-shot accuracy is0\.8580\.858\(completion0\.9860\.986\) against0\.8440\.844for the 8B, a paired difference of\+1\.4\+1\.4\[−0\.8\-0\.8,\+3\.6\+3\.6\]; its student reaches0\.8120\.812\(sd0\.0090\.009; per seed0\.814/0\.812/0\.822/0\.8000\.814/0\.812/0\.822/0\.800\)\. The per\-token KL to the 32B teacher starts at0\.590\.59against0\.400\.40to0\.450\.45for the 8B and is still0\.230\.23against0\.170\.17to0\.180\.18at steps 91–100, while the answer\-line concentration collapses as in Figure[4](https://arxiv.org/html/2610.20989#A2.F4); we read this as a description rather than an explanation, since a KL between different teachers is not a distance on a common scale\. The 4B student was not run: it is already at the 8B teacher’s level zero\-shot \(0\.8360\.836against0\.8440\.844\)\.

Table 14:Out\-of\-distribution accuracy of ChartQA\-trained models \(four seeds at every budget\) on CharXiv \(461 numeric questions\) and ChartQAPro \(500 factoid questions\), with paired OPD minus SFT and its question\-clustered interval\. Zero\-shot student0\.2580\.258/0\.2760\.276, teacher0\.4120\.412/0\.4060\.406\.
## Appendix HSensitivity of the intervals to the resampling unit

Tables[15](https://arxiv.org/html/2610.20989#A8.T15)and[16](https://arxiv.org/html/2610.20989#A8.T16)give, for every headline comparison, the four per\-seed paired differences and three 95% intervals:*questions*, the question\-clustered bootstrap used throughout the paper, which resamples questions and keeps every seed’s outcome on a question together, treating the four trained models as fixed;*two\-way*, a bootstrap that resamples seeds with replacement as well as questions; and*seeds*, thettinterval over the four per\-seed differences\. A bootstrap that resamples the4×5004\\times 500per\-question outcomes as if they were independent gives intervals about a quarter narrower than the question\-clustered ones \(for example \[\+1\.3\+1\.3,\+4\.2\+4\.2\] against \[\+0\.8\+0\.8,\+4\.7\+4\.7\] for OPD minus SFT at 300 ChartQA questions\) and is not used\. The Geometry3K losses at 900 and 2,101 problems, at both generation budgets, exclude zero under every convention; most of the ChartQA differences do not\.

Table 15:ChartQA and out\-of\-distribution comparisons: paired differences in percentage points \(seeds 1/2/3/42\) and their intervals under three conventions\.Table 16:Geometry3K comparisons, as in Table[15](https://arxiv.org/html/2610.20989#A8.T15)\.
## Appendix INumeric tolerance

Relaxed accuracy accepts a numeric answer within 5% of the gold value\. Table[17](https://arxiv.org/html/2610.20989#A9.T17)re\-scores every Geometry3K model from its stored predictions at 1% and 0\.1%\. Absolute accuracies fall by 1 to 4 points; the ordering of the methods is unchanged, and every interval that excludes zero at 5% does so at 1% and 0\.1% as well, with endpoints moving by up to a point\.

Table 17:Geometry3K accuracy \(mean over four seeds\) at three numeric tolerances, with the paired OPD minus SFT difference in points and its question\-clustered interval\. Zero\-shot student / teacher:0\.3660\.366/0\.5700\.570at 5%,0\.3540\.354/0\.5520\.552at 1%,0\.3300\.330/0\.5340\.534at 0\.1%\.
## Appendix JChartQA variants

Table[18](https://arxiv.org/html/2610.20989#A10.T18)scores the OPD variants of Section[4\.3](https://arxiv.org/html/2610.20989#S4.SS3)on ChartQA, where the teacher concludes 99% of its solutions\.

Table 18:OPD variants on ChartQA \(four seeds each, across\-seed sd in brackets\): no variant differs from plain OPD\. Reference points: SFT0\.7930\.793/0\.7990\.799and OPD0\.8110\.811/0\.8260\.826at 100 / 300 questions\. Intervals are question\-clustered\.
## Appendix KTraining length of SFT on ChartQA

Table[19](https://arxiv.org/html/2610.20989#A11.T19)gives the SFT training\-length control of Section[4\.1](https://arxiv.org/html/2610.20989#S4.SS1)\.

Table 19:SFT trained for fewer steps than the protocol’s 302, four seeds each \(across\-seed sd in brackets\), and the paired OPD minus SFT difference against each length \(question\-clustered intervals\)\. 150 steps of 16 sequences equals OPD’s number of training sequences\. The best length was selected on the test set\.
## Appendix LData\-scaling slopes

Table[20](https://arxiv.org/html/2610.20989#A12.T20)fits accuracy againstlog10\\log\_\{10\}of the training budget, per seed, over the four budgets of Figure[1](https://arxiv.org/html/2610.20989#S4.F1)\. SFT gains 3 to 5 points per decade of questions; OPD gains none, because its compute is fixed at 150 steps of 16 rollouts, or 2,400 rollouts, which is 24 passes over 100 questions, 8 over 300, 2\.7 over 900, and 0\.8 over 3,000\. The slopes describe this protocol: with 1\.5 decades of data and one model size they are not scaling laws, and whether OPD’s slope rises when its step count grows with the data is the experiment the protocol leaves open, which we leave to future work\.

Table 20:Accuracy \(mean over four seeds\) at the four budgets and the slope of a per\-seed least\-squares fit of accuracy onlog10\\log\_\{10\}\(budget\), in points per decade, with thettinterval over the four seeds and theR2R^\{2\}of the fit to the seed means\.
## Appendix MPer\-seed results

Table[21](https://arxiv.org/html/2610.20989#A13.T21)lists the accuracy of every trained model behind the paper’s means, seed by seed; the unfiltered and complete\-only SFT controls, the OPD variants, and the re\-scored evaluations are included\.

Table 21:Per\-seed accuracy on the 500 test questions \(seeds 1, 2, 3, and 42\) and the four\-seed mean\.
## Appendix NEngineering notes

Training an 8B teacher and a 2B student together on one 40 GB GPU with full\-vocabulary KL required: left\-padded prompts so that generation positions align across the batch;logits\_to\_keepso that neither model materialises logits over the prompt or image tokens; the KL computed in fp32 in chunks over the sequence; micro\-batching of the scoring pass with gradient accumulation; and rollouts at batch 16 \(the rollout is 90% of step time and doubling its batch doubled throughput at constant peak memory of 27 GB\)\. Checkpoints \(adapter, optimizer, scheduler, data cursor\) go to a remote repository every 25 steps and a killed job resumes from the latest one\. vLLM\([Kwon et al\., 2023](https://arxiv.org/html/2610.20989#bib.bib6)\)evaluates 500 questions in about 20 seconds\. For a 32B teacher the student is pinned to one GPU and the teacher sharded over two more\.