EasyOPD: An Easy-to-use On-Policy Distillation Framework for Large Language Models

# An Easy-to-use On-Policy Distillation Framework for Large Language Models
Source: [https://arxiv.org/html/2607.11012](https://arxiv.org/html/2607.11012)
Jie Sun1,3,∗,§, Mao Zheng2,∗, Mingyang Song2,∗, Qiyong Zhong1,∗,§ Gengsheng Li1,∗,§, Zhepei Hong1, Chang Wu1, Pengfei Liu3 Junfeng Fang4,†, Xiang Wang1,† 1University of Science and Technology of China2LLM Department, Tencent 3Shanghai Innovation Institute4National University of Singapore [https://github\.com/lds\-ustc/EasyOPD](https://github.com/lds-ustc/EasyOPD)

###### Abstract

Conventional language\-model distillation often relies on fixed teacher\-generated data, which may not cover the states encountered by an evolving student policy\. On\-policy distillation \(OPD\) instead collects teacher or evaluator supervision on student\-generated rollouts\. However, existing OPD methods differ substantially in supervision form, tokenizer compatibility, teacher access, and supervision granularity, leading to fragmented implementations that are difficult to reproduce and extend\. We presentEasyOPD, an on\-policy distillation framework built on verl, a distributed reinforcement\-learning framework for large language models\.EasyOPDseparates user\-side configuration, method\-specific supervision logic, and verl\-based execution\. Its method modules connect to the shared backend through extension boundaries for loss construction, rollout metadata, reward processing, tokenizer alignment, and teacher\-side computation\. We instantiate representative methods for three OPD settings—cross\-tokenizer OPD, on\-policy self\-distillation, and step\-wise OPD\. Experiments on reasoning, code\-generation, scientific\-knowledge, and tool\-use benchmarks show that these implementations can be executed through the same verl\-based backend while retaining their method\-specific objectives and task\-dependent performance profiles\. We releaseEasyOPDwith runnable YAML configurations, documentation, and an installable demonstration package and video\.

EasyOPD: An Easy\-to\-use On\-Policy Distillation Framework for Large Language Models

Jie Sun1,3,∗,§, Mao Zheng2,∗, Mingyang Song2,∗, Qiyong Zhong1,∗,§Gengsheng Li1,∗,§, Zhepei Hong1, Chang Wu1, Pengfei Liu3Junfeng Fang4,†, Xiang Wang1,†1University of Science and Technology of China2LLM Department, Tencent3Shanghai Innovation Institute4National University of Singapore[https://github\.com/lds\-ustc/EasyOPD](https://github.com/lds-ustc/EasyOPD)

\\NoHyper††footnotetext:∗Equal Contribution\.†Corresponding Authors\.§Work done during internship at Tencent\.\\endNoHyper

## 1Introduction

Large language model distillation transfers capabilities from a stronger teacher to a smaller, more efficient student, aiming to reduce deployment costs while retaining reasoning, coding, and instruction\-following abilities\(Alkhulaifiet al\.,[2021](https://arxiv.org/html/2607.11012#bib.bib4)\)\. Conventional distillation uses teacher responses or token distributions on fixed contexts\(Yanget al\.,[2024b](https://arxiv.org/html/2607.11012#bib.bib49)\)\. At inference, however, the student conditions on its own prefixes and may encounter contexts poorly covered by offline supervision\(Guet al\.,[2024](https://arxiv.org/html/2607.11012#bib.bib5); Koet al\.,[2024](https://arxiv.org/html/2607.11012#bib.bib9)\)\. On\-policy distillation \(OPD\) mitigates this mismatch through a common loop: the current student generates rollouts, a teacher or evaluator supervises them, and the resulting signals guide student updates\(Song and Zheng,[2026](https://arxiv.org/html/2607.11012#bib.bib11); Lu and Thinking Machines Lab,[2025](https://arxiv.org/html/2607.11012#bib.bib7)\)\. OPD methods vary by supervision interface, tokenizer compatibility, and granularity, with overlapping settings including logit\-based distillation, cross\-tokenizer alignment\(Sunet al\.,[2026](https://arxiv.org/html/2607.11012#bib.bib87)\), on\-policy self\-distillation, black\-box or rubric\-based feedback, and step\-wise or multi\-turn agentic supervision\(Zhonget al\.,[2026](https://arxiv.org/html/2607.11012#bib.bib72); Liet al\.,[2026a](https://arxiv.org/html/2607.11012#bib.bib99)\)\.

![Refer to caption](https://arxiv.org/html/2607.11012v1/x1.png)Figure 1:Architecture ofEasyOPD\. The user layer provides configuration\-based invocation, theEasyOPDlayer encapsulates method\-local OPD logic, and verl provides distributed execution\. Hooks for loss, rollout, reward, alignment, and teacher\-side computation connect method modules to verl, allowing supported methods to share the same execution backend and be selected via YAML configurations\.A shared algorithmic paradigm does not yield a common system interface\. Depending on the supervision, OPD may span rollout generation, teacher inference, cross\-tokenizer alignment, reward construction, distributed data flow, and optimization\. For example, cross\-tokenizer OPD requires alignment across teacher and student tokenizations before constructing compatible supervision, whereas black\-box or rubric\-based OPD may convert textual judgments or scores into rewards; these patterns require different extension points\. Existing frameworks address complementary parts of the OPD workflow: TRL\(von Werraet al\.,[2020](https://arxiv.org/html/2607.11012#bib.bib88)\)provides trainer\-centric distillation APIs; verl\(Shenget al\.,[2024](https://arxiv.org/html/2607.11012#bib.bib89)\)and slime\(Zhuet al\.,[2025](https://arxiv.org/html/2607.11012#bib.bib90)\)support scalable on\-policy rollout and distributed execution; and KDFlow\(Zhanget al\.,[2026](https://arxiv.org/html/2607.11012#bib.bib70)\)provides a distillation\-oriented stack\. However, as summarized in Table[1](https://arxiv.org/html/2607.11012#S3.T1), we find no single reviewed framework that documents method\-local extension boundaries across the heterogeneous supervision interfaces considered here while reusing a common distributed backend\. This gap calls for a lightweight OPD abstraction linking method\-specific supervision to a shared execution substrate\.

To fill this gap, we presentEasyOPD, a unified OPD framework built on verl\.EasyOPDreuses verl for rollout generation, model execution, and optimization while adding a lightweight abstraction layer for method\-local supervision\. Method modules connect to the shared training flow through explicit extension points: cross\-tokenizer methods localize alignment and supervision construction, whereas black\-box or rubric\-based methods localize feedback acquisition and training\-signal construction\. This design turns OPD integration from framework\-wide modification into method\-local extension\.

To realize this design,EasyOPDadopts the three\-layer architecture shown in Figure[1](https://arxiv.org/html/2607.11012#S1.F1)\. The user layer provides Python and CLI entry points and YAML configurations for methods, models, datasets, and hyperparameters; theEasyOPDlayer manages method registration, method\-local modules, and supervision diagnostics, connecting to verl through dispatch hooks for loss, rollout metadata, reward, alignment, and teacher\-side computation; and the verl layer handles distributed rollout, model execution, worker orchestration, reward computation, and optimization\. Users select methods through configuration, while developers add a method by registering a method\-local module with the required hooks, so that the core trainer and workers reach method\-specific supervision only through the shared dispatch layer\.

We instantiate this design with representative methods across three settings: cross\-tokenizer OPD, on\-policy self\-distillation, and step\-wise OPD; its extension boundaries are further designed to accommodate additional supervision forms such as black\-box or rubric\-based feedback\. Within each setting, the compared methods share the model initialization, training data, execution backend, and evaluation procedure where applicable, while retaining their original supervision designs\. We release the code under the Apache License 2\.0 on GitHub with reproducible YAML configurations, documentation, an installable demonstration package, and a demonstration video\.

## 2Background and Motivation

#### Previous Solutions\.

Knowledge distillation \(KD\) typically trains a smaller student model to imitate the predictive behavior of a larger or more capable teacher\(Hintonet al\.,[2015](https://arxiv.org/html/2607.11012#bib.bib3); Koet al\.,[2024](https://arxiv.org/html/2607.11012#bib.bib9)\)\. For autoregressive language models, common offline approaches include sequence\-level distillation, in which the student is trained on teacher\-generated sequences\(Kim and Rush,[2016](https://arxiv.org/html/2607.11012#bib.bib91)\), and supervised token\-level KD, in which the student matches the teacher’s next\-token distributions along prefixes induced by a fixed offline set of output sequences\(Agarwalet al\.,[2024](https://arxiv.org/html/2607.11012#bib.bib8)\)\. These output sequences may be ground\-truth or teacher\-generated\. In their purely offline form, both approaches optimize over an output\-prefix distribution that does not track the current student’s rollout distribution\. During autoregressive inference, however, the student conditions on its own previous predictions\. A deviation can therefore shift subsequent prefixes away from those observed during offline distillation, creating a training–inference distribution mismatch and allowing errors to compound across decoding steps\(Linet al\.,[2020](https://arxiv.org/html/2607.11012#bib.bib59); Agarwalet al\.,[2024](https://arxiv.org/html/2607.11012#bib.bib8)\)\.

#### On\-Policy Distillation Mechanism\.

On\-policy distillation \(OPD\) collects teacher supervision on outputs generated by the current student, rather than relying only on fixed offline sequences\. During training, the student generates responses or trajectories, and the teacher provides supervision on the resulting prefixes or states\. The student is then updated, and new rollouts are generated with the updated policy\. In a common white\-box setting, the teacher provides next\-token distributions on student\-generated prefixes, and the student minimizes a token\-level divergence from the teacher\(Agarwalet al\.,[2024](https://arxiv.org/html/2607.11012#bib.bib8)\)\. The defining property of OPD is therefore that supervision is collected on states induced by the current student, rather than the use of any particular distillation objective\(Lu and Thinking Machines Lab,[2025](https://arxiv.org/html/2607.11012#bib.bib7); Song and Zheng,[2026](https://arxiv.org/html/2607.11012#bib.bib11)\)\.

Training on student\-generated states helps reduce the mismatch between offline training data and the prefixes encountered during autoregressive inference, while providing guidance on errors made by the student itself\(Agarwalet al\.,[2024](https://arxiv.org/html/2607.11012#bib.bib8); Koet al\.,[2024](https://arxiv.org/html/2607.11012#bib.bib9)\)\. However, OPD also turns distillation into an iterative pipeline that must coordinate student rollout, teacher inference, supervision construction, and model optimization\. Since OPD methods differ in teacher access, feedback form, and training objectives\(Song and Zheng,[2026](https://arxiv.org/html/2607.11012#bib.bib11)\), a practical framework should support this shared pipeline while allowing method\-specific supervision to be implemented consistently\.

## 3Design and Implementation

Table 1:Comparison of OPD supervision interfaces and framework properties\.✓denotes native or first\-class support,∘\\circdenotes partial or experimental support, and×\\timesdenotes no documented support\.*On\-policy self\-distillation*refers to settings in which the teacher is derived from the student model and supervises student\-generated rollouts\. The comparison reflects publicly documented functionality as of July 2026, using the versions or commits listed in Appendix[A](https://arxiv.org/html/2607.11012#A1), where per\-cell evidence is also provided\.OPD supervision\-interface coverageFramework propertiesFrameworkOPDCross\-tok\.OPDOn\-policyself\-distill\.Step\-wiseOPDReleasedYAML/CLIDocumented methodextension APICompatible w/multi\-nodeTRL✓∘\\circ∘\\circ×\\times✓×\\times∘\\circverl✓×\\times×\\times∘\\circ✓×\\times✓slime✓×\\times∘\\circ∘\\circ✓×\\times✓KDFlow✓✓✓×\\times✓∘\\circ∘\\circOurs✓✓✓✓✓✓✓\(via verl\)One entry point for every OPD setting inEasyOPD[⬇](data:text/plain;base64,ZnJvbSBlYXN5b3BkIGltcG9ydCBFYXN5T1BECnByaW50KEVhc3lPUEQubGlzdF9tZXRob2RzKCkpICAgIyBbLi4uLCAnc2ltY3QnLCd1bGQnLCdhbG0nLCdkc2tkJywnc2RwbycsJ3NvZCcsIC4uLl0KIyBUaGUgU0FNRSBjYWxsIHNlbGVjdHMgYW55IG1ldGhvZDsgb25seSB0aGUgbmFtZSArIFlBTUwgY2hhbmdlLgptID0gRWFzeU9QRC5mcm9tX2hwYXJhbXMoInNpbWN0IiwgY29uZmlnX3BhdGg9ImVhc3lvcGQvY29uZmlnL3NpbWN0LnlhbWwiKSAgIyBjcm9zcy10b2tlbml6ZXIgT1BEICh1bGR8YWxtfGRza2QpCm0gPSBFYXN5T1BELmZyb21faHBhcmFtcygic2RwbyIsICBjb25maWdfcGF0aD0iZWFzeW9wZC9jb25maWcvc2Rwby55YW1sIikgICAjIG9uLXBvbGljeSBzZWxmLWRpc3RpbGxhdGlvbiAodnMgZ3JwbykKbSA9IEVhc3lPUEQuZnJvbV9ocGFyYW1zKCJzb2QiLCAgIGNvbmZpZ19wYXRoPSJlYXN5b3BkL2NvbmZpZy9zb2QueWFtbCIpICAgICMgc3RlcC13aXNlIE9QRCAodnMgcmVzcG9uc2UtbGV2ZWwgb3BkLCBncnBvKQ==)fromeasyopdimportEasyOPDprint\(EasyOPD\.list\_methods\(\)\)m=EasyOPD\.from\_hparams\("simct",config\_path="easyopd/config/simct\.yaml"\)m=EasyOPD\.from\_hparams\("sdpo",config\_path="easyopd/config/sdpo\.yaml"\)m=EasyOPD\.from\_hparams\("sod",config\_path="easyopd/config/sod\.yaml"\)[⬇](data:text/plain;base64,IyBFYWNoIG1ldGhvZCAoYW5kIGl0cyBiYXNlbGluZXMpIGlzIG9uZSBsYXVuY2ggc2NyaXB0OyB0aGUgY29uZmlnIHNlbGVjdHMgdGhlIG1ldGhvZCBhbmQgdGhlIGhvb2tzIGl0IGFjdGl2YXRlcy4KJCBiYXNoIGV4YW1wbGVzL3NpbWN0L3J1bl9zaW1jdC5zaCAgICMgY3Jvc3MtdG9rZW5pemVyOiB1bGR8YWxtfGRza2QKJCBiYXNoIGV4YW1wbGVzL3NkcG8vcnVuX3NkcG8uc2ggICAgICMgc2VsZi1kaXN0aWxsYXRpb246IGdycG8KJCBiYXNoIGV4YW1wbGVzL3NvZC9ydW5fc29kLnNoICAgICAgICMgc3RlcC13aXNlOiByZXNwb25zZS1sZXZlbCBvcGQsIGdycG8=)$bashexamples/simct/run\_simct\.sh$bashexamples/sdpo/run\_sdpo\.sh$bashexamples/sod/run\_sod\.sh[⬇](data:text/plain;base64,IyBUaGUgcmVnaXN0cnkgYWN0aXZhdGVzIG9ubHkgdGhlIGhvb2tzIGEgbWV0aG9kIG5lZWRzIGFuZCBsb2dzIHN1cGVydmlzaW9uLXNwZWNpZmljIGRpYWdub3N0aWNzOgojICAgc2ltY3QgLT4gYWxpZ25tZW50ICsgdGVhY2hlciBzaWRlY2FyICsgbG9zcyAgOiBzaW1jdC94dG9rX2tkX2xvc3MsIHNpbXBsZS90ZWFjaGVyX2xvc3NfdG9rZW5zX21lYW4KIyAgIHNkcG8gIC0+IHJlcHJvbXB0IHNlbGYtdGVhY2hlciArIGxvc3MgICAgICAgIDogYWN0b3Ivc2Rwby9sb3NzLCBzZWxmX2Rpc3RpbGxhdGlvbi9yZXByb21wdF9zYW1wbGVfZnJhY3Rpb24KIyAgIHNvZCAgIC0+IHN0ZXAtd2lzZSBLTCByZXdlaWdodGluZyAgICAgICAgICAgIDogYWN0b3IvdG9rZW5fa2wsIHN0ZXB3aXNlX29wZF9jb2Vm)\#Theregistryactivatesonlythehooksamethodneedsandlogssupervision\-specificdiagnostics:\#simct\-\>alignment\+teachersidecar\+loss:simct/xtok\_kd\_loss,simple/teacher\_loss\_tokens\_mean\#sdpo\-\>repromptself\-teacher\+loss:actor/sdpo/loss,self\_distillation/reprompt\_sample\_fraction\#sod\-\>step\-wiseKLreweighting:actor/token\_kl,stepwise\_opd\_coefFigure 2:A running example ofEasyOPD\. The three studied settings—cross\-tokenizer OPD, on\-policy self\-distillation, and step\-wise OPD—and their baselines are all reached through the*same*from\_hparamsentry point or a single launch script, changing only the method name and its YAML config\. The registry then activates only the hooks each method needs, verl performs rollout and optimization, andEasyOPDreports supervision\-specific diagnostics\. All API calls, method names, launch commands, and diagnostic metric names are taken from the released repository\.EasyOPD provides a unified OPD training process on top of verl\. Instead of treating each OPD method as an isolated training script, it decomposes an OPD run into four reusable stages: method invocation, configuration resolution, supervision construction, and distributed execution\. This section describes how these stages form a unified workflow that connects diverse OPD methods to the same verl execution substrate\.

### 3\.1A Unified OPD Workflow

Although OPD methods differ in their supervision interfaces, they share a common training flow\. Given a teacher model, a student model, and training data, the student first generates on\-policy rollouts; the selected method then constructs supervision on these rollouts, such as teacher logits, aligned token distributions, judge rewards, or step\-level feedback; finally, the training system consumes this supervision through loss computation, reward aggregation, or trajectory\-level weighting\. EasyOPD makes this shared process explicit and standardizes invocation and dispatch, without requiring all methods to share a single loss, teacher interface, or evaluation metric\. As shown in Figure[1](https://arxiv.org/html/2607.11012#S1.F1), the user layer specifies what to run, the EasyOPD layer defines how supervision is constructed, and the verl layer executes the resulting training process on top of its distributed runtime\(Shenget al\.,[2024](https://arxiv.org/html/2607.11012#bib.bib89)\)\.

### 3\.2Declarative Invocation and Configuration

EasyOPD exposes OPD methods through declarative configurations and launch scripts\. Released methods can be selected through method\-specific YAML configurations without manually editing rollout workers, reward managers, logit processors, or trainer\-side loss assembly, which turns method selection into a configuration\-level operation and makes experiments easier to reproduce\. Figure[2](https://arxiv.org/html/2607.11012#S3.F2)shows a concrete running example in which the samefrom\_hparamsentry point and a single launch command drive all three studied settings—cross\-tokenizer OPD, on\-policy self\-distillation, and step\-wise OPD—together with their baselines, changing only the method name and its YAML configuration\.

Different OPD methods require different options: logit\-based methods may specify divergence directions or top\-kksupport; cross\-tokenizer methods may specify alignment or overlap\-vocabulary behavior; rubric\-based methods may specify judge prompts or reward aggregation rules; step\-wise methods may specify trajectory segmentation or weighting strategies\. EasyOPD keeps these choices in a unified configuration space and resolves them through method registration, so new methods can be added without hard\-coding method choices into the trainer\.

### 3\.3Method\-Local Supervision Logic

The core of EasyOPD is to localize OPD\-specific supervision logic: a method defines how supervision is obtained, transformed, consumed, and diagnosed\. Standard OPD may only require a token\-level distillation objective, cross\-tokenizer OPD may require tokenizer or span alignment and a common supervision space, rubric\-based OPD may convert judge feedback into rewards, and step\-wise OPD may attach feedback to intermediate reasoning states or multi\-turn trajectories\. EasyOPD does not force these methods into a single loss\-only abstraction; it gives each a local space to implement the supervision interface it needs\.

Method\-local logic also includes diagnostics\. Implemented method modules expose supervision\-specific diagnostics in addition to task performance, so users can check whether the intended supervision signal is active, not only the final loss\. For example, the released cross\-tokenizer implementation reports the overlap\-vocabulary size, the number of valid and span\-level aligned segments, the count of skipped samples, and the valid\-span ratio, while the step\-wise implementation reports the mean step\-level weight and the number of supervised steps\. These metrics help users debug failures that final task performance alone cannot explain\.

### 3\.4Execution through Thin verl Hooks

EasyOPD reuses verl as the execution substrate instead of reimplementing distributed training\. verl provides the system\-heavy components required by OPD, including rollout generation, reward computation, model forwarding, logit processing, distributed workers, trainer\-side optimization, and parallel execution, and EasyOPD connects method\-local supervision logic to this substrate through a small set of thin hooks\.

A hook is a dispatch boundary rather than a method implementation: it exposes a stable interaction point in the training pipeline and routes the corresponding data or control flow to the selected method module\. The hook set covers the main entry points of OPD supervision: loss hooks for token\- or distribution\-level supervision, rollout hooks for trajectory metadata, reward hooks for judge or rubric feedback, alignment hooks for token\- or span\-level mapping, and teacher\-sidecar hooks for teacher\-side signals such as logits, hidden states, or judge outputs, as used respectively by logit\-based, feature\-based\(Zhanget al\.,[2024](https://arxiv.org/html/2607.11012#bib.bib55)\), and rubric\-based methods\.

This design balances expressiveness and maintainability\. Writing every method directly into verl would create method\-specific branches, while exposing only a single fixed interface would make alignment\-, reward\-, or trajectory\-based OPD methods difficult to express\. Thin hooks keep method logic local while verl remains the shared backend\. EasyOPD thus inherits rollout generation, worker orchestration, optimization, and distributed execution from verl; its contribution is the dispatch layer connecting these components to method\-local supervision logic\.

## 4Experiments

### 4\.1Experimental Setup

We evaluateEasyOPDin three settings that exercise different method\-specific interfaces\. Cross\-tokenizer OPD requires supervision across different teacher and student tokenizations\. On\-policy self\-distillation instantiates teacher and student policies from the same model under different conditioning contexts\. Step\-wise OPD assigns supervision to intermediate reasoning or tool\-interaction steps\. These settings cover variation in supervision space, teacher conditioning, and supervision granularity, and are not intended to form a single cross\-regime leaderboard across methods\.

#### Cross\-tokenizer OPD\.

Standard OPD directly compares teacher and student next\-token distributions, which are not naturally aligned under different tokenizers; cross\-tokenizer methods address this by constructing a shared supervision space\. We compare SimCT\(Sunet al\.,[2026](https://arxiv.org/html/2607.11012#bib.bib87)\), ULD\(Boizardet al\.,[2025](https://arxiv.org/html/2607.11012#bib.bib12)\), ALM\(Minixhoferet al\.,[2025](https://arxiv.org/html/2607.11012#bib.bib14)\), and DSKD\(Zhanget al\.,[2024](https://arxiv.org/html/2607.11012#bib.bib55)\), distilling Qwen2\.5\-7B\-Instruct\(Yanget al\.,[2024a](https://arxiv.org/html/2607.11012#bib.bib38)\)into Phi\-4\-mini\-Instruct\(Aboueleninet al\.,[2025](https://arxiv.org/html/2607.11012#bib.bib40)\)\. All methods start from the same student checkpoint, warm\-started on a shared 10K teacher\-generated math and code corpus, and use the same training prompts and optimization budget\. We evaluate on MATH500\(Lightmanet al\.,[2024](https://arxiv.org/html/2607.11012#bib.bib25)\), GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2607.11012#bib.bib28)\), MBPP\(Austinet al\.,[2021](https://arxiv.org/html/2607.11012#bib.bib29)\), and LiveCodeBench v6\(Jainet al\.,[2025](https://arxiv.org/html/2607.11012#bib.bib26)\), reporting exact\-match accuracy for reasoning and pass@1 for code\.

#### On\-policy self\-distillation\.

On\-policy self\-distillation \(OPSD\) uses the same base model under different conditioning contexts: the student generates a response from the original prompt, while the self\-teacher re\-evaluates it with additional feedback\. We instantiate this setting with SDPO\(Hübotteret al\.,[2026](https://arxiv.org/html/2607.11012#bib.bib93)\), using Qwen3\-8B\(Qwen Team,[2025](https://arxiv.org/html/2607.11012#bib.bib96)\)as both the student and self\-teacher, and compare it with the base model and GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2607.11012#bib.bib97)\)\. We evaluate Chemistry from SciKnowEval\(Fenget al\.,[2024](https://arxiv.org/html/2607.11012#bib.bib94)\), Biology from SciKnowEval\(Fenget al\.,[2024](https://arxiv.org/html/2607.11012#bib.bib94)\), Tool Use from ToolAlpaca\(Tanget al\.,[2023](https://arxiv.org/html/2607.11012#bib.bib95)\), and mathematical reasoning on GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2607.11012#bib.bib28)\)\. We report the highest Average@16 reached within one\-hour and five\-hour wall\-clock budgets, where Average@16 is the mean correctness over 16 independently sampled responses per question\.

#### Step\-wise OPD\.

Response\-level OPD\(Agarwalet al\.,[2024](https://arxiv.org/html/2607.11012#bib.bib8)\)applies teacher supervision uniformly across a trajectory, although supervision on later steps may become less reliable after erroneous tool interactions, a common failure mode as LLMs are deployed as long\-horizon, tool\-use, and research agents\(Xiaoet al\.,[2025](https://arxiv.org/html/2607.11012#bib.bib75); Liet al\.,[2026b](https://arxiv.org/html/2607.11012#bib.bib74); Fenget al\.,[2026](https://arxiv.org/html/2607.11012#bib.bib78); Wuet al\.,[2025](https://arxiv.org/html/2607.11012#bib.bib76); Fuet al\.,[2026b](https://arxiv.org/html/2607.11012#bib.bib80); Zenget al\.,[2026](https://arxiv.org/html/2607.11012#bib.bib77); Fuet al\.,[2026a](https://arxiv.org/html/2607.11012#bib.bib79)\)\. SOD\(Zhonget al\.,[2026](https://arxiv.org/html/2607.11012#bib.bib72)\)instead reweights the distillation loss by step\-level student–teacher divergence\. We use Qwen3\-1\.7B\(Qwen Team,[2025](https://arxiv.org/html/2607.11012#bib.bib96)\)as the student and a GRPO\-optimized Qwen3\-4B\(Zhonget al\.,[2026](https://arxiv.org/html/2607.11012#bib.bib72)\)as the teacher, comparing the base model, GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2607.11012#bib.bib97)\), response\-level OPD, and SOD under a shared SFT initialization, training data, and optimization budget\(Zhonget al\.,[2026](https://arxiv.org/html/2607.11012#bib.bib72)\)\. We evaluate on AIME 2024, AIME 2025\(Mathematical Association of America,[2026](https://arxiv.org/html/2607.11012#bib.bib98)\), and LiveCodeBench v6\(Jainet al\.,[2025](https://arxiv.org/html/2607.11012#bib.bib26)\); each score is Average@32 over 32 sampled responses per problem, and Avg\. is their unweighted mean\.

#### Shared implementation\.

All methods are implemented inEasyOPDand run on the same verl\-based infrastructure\(Shenget al\.,[2024](https://arxiv.org/html/2607.11012#bib.bib89)\)\. Within each setting we fix the model, training data, backend, and evaluation procedure where applicable, retaining each method’s original supervision design\. Optimization, sampling, hardware, and checkpoint details are in Appendix[B](https://arxiv.org/html/2607.11012#A2)\.

### 4\.2Results

#### Cross\-tokenizer OPD\.

Table[2](https://arxiv.org/html/2607.11012#S4.T2)shows that all four evaluated cross\-tokenizer methods improve the unweighted average over the base student\. SimCT obtains the highest average \(53\.5, \+3\.3 over the base\) and the best MATH500, ALM leads on MBPP and LiveCodeBench v6, and DSKD is best on GSM8K\. This confirms that the evaluated cross\-tokenizer objectives run throughEasyOPD’s alignment and loss interfaces under a matched setup, while retaining distinct task\-dependent profiles\.

Table 2:Cross\-tokenizer OPD results\. Reasoning scores are exact\-match accuracy, code scores are pass@1, and Avg\. is their unweighted mean\. Best per column in bold\.Table 3:SDPO and GRPO in the OPSD setting under one\-hour and five\-hour wall\-clock budgets\. Average@16 is the mean correctness over 16 sampled responses per question; Base is the untrained checkpoint, reported once per task\. We evaluate every five steps and report the best Average@16 observed within each budget; the best per task and budget is in bold\.Table 4:Step\-wise OPD with Qwen3\-1\.7B as the student\. OPD and SOD use a GRPO\-optimized Qwen3\-4B teacher\. Values are percentages; Avg\. is the unweighted mean across AIME 2024, AIME 2025, and LiveCodeBench v6\. Best per column in bold\.
#### On\-policy self\-distillation\.

Table[3](https://arxiv.org/html/2607.11012#S4.T3)compares SDPO and GRPO under the two wall\-clock budgets\. SDPO’s gains are largest on Chemistry and Biology \(\+19\.2 points at one hour, \+6\.7 and \+6\.0 at five hours\), while GRPO is stronger on GSM8K and on the five\-hour Tool Use result\. SDPO’s relative performance is thus task\-dependent in the evaluated setting\.

#### Step\-wise OPD\.

Table[4](https://arxiv.org/html/2607.11012#S4.T4)shows that response\-level OPD exceeds GRPO on all three benchmarks, and SOD obtains the highest observed score on each, for an unweighted average of 39\.95 \(\+5\.1 over response\-level OPD\)\. These results are consistent with the intended role of step\-wise weighting, but do not by themselves establish that every step\-wise OPD method will outperform response\-level OPD\.

#### Overall findings\.

Across the three case studies,EasyOPDruns the evaluated methods on the same verl backend while preserving their distinct supervision requirements; results should be read within each setting rather than as a cross\-regime ranking\.

## 5Conclusion and Future Work

We present EasyOPD, a method\-oriented framework that integrates on\-policy distillation methods on a shared verl backend, separating user\-side configuration, method\-local supervision logic, and distributed execution through thin dispatch hooks\. Across cross\-tokenizer OPD, on\-policy self\-distillation, and step\-wise OPD, methods with different supervision interfaces share this backend while keeping their own objectives and diagnostics; future work will broaden method coverage and larger\-model evaluation\.

## References

- M\. A\. Abouelenin, A\. Ashfaq, A\. Atkinson, H\. Awadalla, N\. Bach, J\. Bao, A\. Benhaim, M\. Cai, V\. Chaudhary, C\. Chen, D\. Chen, D\. Chen, J\. Chen, W\. Chen, Y\. Chen, Y\. Chen, Q\. Dai, X\. Dai, R\. Fan, M\. Gao, M\. Gao, A\. Garg, A\. Goswami, J\. Hao, A\. Hendy, Y\. Hu, X\. Jin, M\. Khademi, D\. Kim, Y\. J\. Kim, G\. Lee, J\. Li, Y\. Li, C\. Liang, X\. Lin, Z\. Lin, M\. Liu, Y\. Liu, G\. Lopez, C\. Luo, P\. Madan, V\. Mazalov, A\. Mitra, A\. Mousavi, A\. Nguyen, J\. Pan, D\. Perez\-Becker, J\. Platin, T\. Portet, K\. Qiu, B\. Ren, L\. Ren, S\. Roy, N\. Shang, Y\. Shen, S\. Singhal, S\. Som, X\. Song, T\. Sych, P\. Vaddamanu, S\. Wang, Y\. Wang, Z\. Wang, H\. Wu, H\. Xu, W\. Xu, Y\. Yang, Z\. Yang, D\. Yu, I\. Zabir, J\. Zhang, L\. L\. Zhang, Y\. Zhang, and X\. Zhou \(2025\)Phi\-4\-mini technical report: compact yet powerful multimodal language models via mixture\-of\-LoRAs\.arXiv preprintarXiv:2503\.01743\.Cited by:[§4\.1](https://arxiv.org/html/2607.11012#S4.SS1.SSS0.Px1.p1.1)\.
- On\-policy distillation of language models: learning from self\-generated mistakes\.InProceedings of the 12th International Conference on Learning Representations \(ICLR 2024\),Vienna, Austria\.Cited by:[§2](https://arxiv.org/html/2607.11012#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2607.11012#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2607.11012#S2.SS0.SSS0.Px2.p2.1),[§4\.1](https://arxiv.org/html/2607.11012#S4.SS1.SSS0.Px3.p1.1)\.
- A\. Alkhulaifi, F\. Alsahli, and I\. Ahmad \(2021\)Knowledge distillation in deep learning and its applications\.PeerJ Computer Science7,pp\. e474\.Cited by:[§1](https://arxiv.org/html/2607.11012#S1.p1.1)\.
- J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le, and C\. Sutton \(2021\)Program synthesis with large language models\.arXiv preprintarXiv:2108\.07732\.Cited by:[§4\.1](https://arxiv.org/html/2607.11012#S4.SS1.SSS0.Px1.p1.1)\.
- N\. Boizard, K\. E\. Haddad, C\. Hudelot, and P\. Colombo \(2025\)Towards cross\-tokenizer distillation: the universal logit distillation loss for LLMs\.Transactions on Machine Learning Research \(TMLR\)\.Cited by:[§4\.1](https://arxiv.org/html/2607.11012#S4.SS1.SSS0.Px1.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman \(2021\)Training verifiers to solve math word problems\.arXiv preprintarXiv:2110\.14168\.Cited by:[§4\.1](https://arxiv.org/html/2607.11012#S4.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2607.11012#S4.SS1.SSS0.Px2.p1.1)\.
- K\. Feng, K\. Ding, W\. Wang, X\. Zhuang, Z\. Wang, M\. Qin, Y\. Zhao, J\. Yao, Q\. Zhang, and H\. Chen \(2024\)SciKnowEval: evaluating multi\-level scientific knowledge of large language models\.arXiv preprintarXiv:2406\.09098\.External Links:2406\.09098Cited by:[§4\.1](https://arxiv.org/html/2607.11012#S4.SS1.SSS0.Px2.p1.1)\.
- Y\. Feng, J\. Sun, Z\. Yang, J\. Ai, C\. Li, Z\. Li, F\. Zhang, K\. He, R\. Ma, J\. Lin, J\. Sun, Y\. Xiao, S\. Zhou, W\. Wu, Y\. Liu, P\. Liu, Y\. Qiao, S\. Zhang, and K\. Zhang \(2026\)LongCLI\-bench: a preliminary benchmark and study for long\-horizon agentic programming in command\-line interfaces\.arXiv preprintarXiv:2602\.14337\.Cited by:[§4\.1](https://arxiv.org/html/2607.11012#S4.SS1.SSS0.Px3.p1.1)\.
- D\. Fu, S\. Wu, Y\. Wu, Z\. Peng, Y\. Huang, J\. Sun, J\. Zeng, M\. Jiang, L\. Zhang, Y\. Li, J\. Hu, L\. Liu, J\. Hou, and P\. Liu \(2026a\)DaVinci\-env: open swe environment synthesis at scale\.arXiv preprintarXiv:2603\.13023\.Cited by:[§4\.1](https://arxiv.org/html/2607.11012#S4.SS1.SSS0.Px3.p1.1)\.
- D\. Fu, Y\. Wu, X\. Cai, L\. Ye, S\. Xia, Z\. Huang, W\. Si, T\. Xu, J\. Sun, K\. Li, M\. Jiang, J\. Wang, Q\. Hua, P\. Lu, X\. Hu, Y\. Xiao, and P\. Liu \(2026b\)ARGO: asynchronous rollout with human guidance for research agent optimization\.OpenReview preprint\.Cited by:[§4\.1](https://arxiv.org/html/2607.11012#S4.SS1.SSS0.Px3.p1.1)\.
- Y\. Gu, L\. Dong, F\. Wei, and M\. Huang \(2024\)MiniLLM: knowledge distillation of large language models\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=5h0qf7IBZZ)Cited by:[§1](https://arxiv.org/html/2607.11012#S1.p1.1)\.
- G\. E\. Hinton, O\. Vinyals, and J\. Dean \(2015\)Distilling the knowledge in a neural network\.arXiv preprintarXiv:1503\.02531\.Cited by:[§2](https://arxiv.org/html/2607.11012#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Hübotter, F\. Lübeck, L\. Behric, A\. Baumann, M\. Bagatella, D\. Marta, I\. Hakimi, I\. Shenfeld, T\. K\. Buening, C\. Guestrin, and A\. Krause \(2026\)Reinforcement learning via self\-distillation\.arXiv preprintarXiv:2601\.20802\.External Links:2601\.20802Cited by:[Appendix B](https://arxiv.org/html/2607.11012#A2.SS0.SSS0.Px3.p1.4),[§4\.1](https://arxiv.org/html/2607.11012#S4.SS1.SSS0.Px2.p1.1)\.
- N\. Jain, K\. Han, A\. Gu, W\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. Stoica \(2025\)LiveCodeBench: holistic and contamination free evaluation of large language models for code\.InProceedings of the 13th International Conference on Learning Representations \(ICLR 2025\),Cited by:[§4\.1](https://arxiv.org/html/2607.11012#S4.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2607.11012#S4.SS1.SSS0.Px3.p1.1)\.
- Y\. Kim and A\. M\. Rush \(2016\)Sequence\-level knowledge distillation\.InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing,J\. Su, K\. Duh, and X\. Carreras \(Eds\.\),Austin, Texas,pp\. 1317–1327\.External Links:[Document](https://dx.doi.org/10.18653/v1/D16-1139),[Link](https://aclanthology.org/D16-1139/)Cited by:[§2](https://arxiv.org/html/2607.11012#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Ko, S\. Kim, T\. Chen, and S\. Yun \(2024\)DistiLLM: towards streamlined distillation for large language models\.InProceedings of the 41st International Conference on Machine Learning \(ICML 2024\),Vol\.235,Vienna, Austria\.Cited by:[§1](https://arxiv.org/html/2607.11012#S1.p1.1),[§2](https://arxiv.org/html/2607.11012#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2607.11012#S2.SS0.SSS0.Px2.p2.1)\.
- G\. Li, M\. Zheng, M\. Song, R\. Liu, T\. Yang, J\. Sun, Q\. Zhong, H\. Guo, and J\. Fang \(2026a\)On\-policy distillation with curriculum turn\-level guidance for multi\-turn agents\.arXiv preprintarXiv:2606\.15912\.Cited by:[§1](https://arxiv.org/html/2607.11012#S1.p1.1)\.
- K\. Li, J\. Shi, Y\. Xiao, M\. Jiang, J\. Sun, Y\. Wu, D\. Fu, S\. Xia, X\. Cai, T\. Xu, W\. Si, W\. Li, D\. Wang, and P\. Liu \(2026b\)AgencyBench: benchmarking the frontiers of autonomous agents in 1m\-token real\-world contexts\.arXiv preprintarXiv:2601\.11044\.Cited by:[§4\.1](https://arxiv.org/html/2607.11012#S4.SS1.SSS0.Px3.p1.1)\.
- H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe \(2024\)Let’s verify step by step\.InProceedings of the 12th International Conference on Learning Representations \(ICLR 2024\),Cited by:[§4\.1](https://arxiv.org/html/2607.11012#S4.SS1.SSS0.Px1.p1.1)\.
- A\. Lin, J\. Wohlwend, H\. Chen, and T\. Lei \(2020\)Autoregressive knowledge distillation through imitation learning\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP 2020\),Online Event,pp\. 6121–6133\.Cited by:[§2](https://arxiv.org/html/2607.11012#S2.SS0.SSS0.Px1.p1.1)\.
- K\. Lu and Thinking Machines Lab \(2025\)On\-policy distillation\.Thinking Machines Lab: Connectionism\.Note:[https://thinkingmachines\.ai/blog/on\-policy\-distillation](https://thinkingmachines.ai/blog/on-policy-distillation)External Links:[Document](https://dx.doi.org/10.64434/tml.20251026)Cited by:[§1](https://arxiv.org/html/2607.11012#S1.p1.1),[§2](https://arxiv.org/html/2607.11012#S2.SS0.SSS0.Px2.p1.1)\.
- Mathematical Association of America \(2026\)MAA invitational competitions: american invitational mathematics examination \(AIME\)\.Note:[https://maa\.org/maa\-invitational\-competitions/](https://maa.org/maa-invitational-competitions/)Accessed: 2026\-07\-01Cited by:[§4\.1](https://arxiv.org/html/2607.11012#S4.SS1.SSS0.Px3.p1.1)\.
- B\. Minixhofer, I\. Vulić, and E\. M\. Ponti \(2025\)Universal cross\-tokenizer distillation via approximate likelihood matching\.InAdvances in Neural Information Processing Systems 38 \(NeurIPS 2025\),Cited by:[§4\.1](https://arxiv.org/html/2607.11012#S4.SS1.SSS0.Px1.p1.1)\.
- Qwen Team \(2025\)Qwen3 technical report\.arXiv preprintarXiv:2505\.09388\.External Links:2505\.09388Cited by:[§4\.1](https://arxiv.org/html/2607.11012#S4.SS1.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2607.11012#S4.SS1.SSS0.Px3.p1.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. Guo \(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.arXiv preprintarXiv:2402\.03300\.External Links:2402\.03300Cited by:[§4\.1](https://arxiv.org/html/2607.11012#S4.SS1.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2607.11012#S4.SS1.SSS0.Px3.p1.1)\.
- G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. Wu \(2024\)HybridFlow: a flexible and efficient rlhf framework\.arXiv preprint arXiv: 2409\.19256\.Cited by:[§1](https://arxiv.org/html/2607.11012#S1.p2.1),[§3\.1](https://arxiv.org/html/2607.11012#S3.SS1.p1.1),[§4\.1](https://arxiv.org/html/2607.11012#S4.SS1.SSS0.Px4.p1.1)\.
- M\. Song and M\. Zheng \(2026\)A survey of on\-policy distillation for large language models\.arXiv preprintarXiv:2604\.00626\.Cited by:[§1](https://arxiv.org/html/2607.11012#S1.p1.1),[§2](https://arxiv.org/html/2607.11012#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2607.11012#S2.SS0.SSS0.Px2.p2.1)\.
- J\. Sun, M\. Zheng, M\. Song, Q\. Zhong, Y\. Cheng, B\. Feng, P\. Liu, J\. Fang, and X\. Wang \(2026\)SimCT: recovering lost supervision for cross\-tokenizer on\-policy distillation\.Note:arXiv preprint arXiv:2605\.07711External Links:2605\.07711,[Link](https://arxiv.org/abs/2605.07711)Cited by:[§1](https://arxiv.org/html/2607.11012#S1.p1.1),[§4\.1](https://arxiv.org/html/2607.11012#S4.SS1.SSS0.Px1.p1.1)\.
- Q\. Tang, Z\. Deng, H\. Lin, X\. Han, Q\. Liang, B\. Cao, and L\. Sun \(2023\)ToolAlpaca: generalized tool learning for language models with 3000 simulated cases\.arXiv preprintarXiv:2306\.05301\.External Links:2306\.05301Cited by:[§4\.1](https://arxiv.org/html/2607.11012#S4.SS1.SSS0.Px2.p1.1)\.
- L\. von Werra, Y\. Belkada, L\. Tunstall, E\. Beeching, T\. Thrush, N\. Lambert, S\. Huang, K\. Rasul, and Q\. Gallouédec \(2020\)TRL: transformers reinforcement learning\.Note:[https://github\.com/huggingface/trl](https://github.com/huggingface/trl)GitHub repositoryCited by:[§1](https://arxiv.org/html/2607.11012#S1.p2.1)\.
- Y\. Wu, D\. Fu, W\. Si, Z\. Huang, M\. Jiang, K\. Li, S\. Xia, J\. Sun, T\. Xu, X\. Hu, P\. Lu, X\. Cai, L\. Ye, W\. Zhu, Y\. Xiao, and P\. Liu \(2025\)InnovatorBench: evaluating agents’ ability to conduct innovative LLM research\.arXiv preprintarXiv:2510\.27598\.Cited by:[§4\.1](https://arxiv.org/html/2607.11012#S4.SS1.SSS0.Px3.p1.1)\.
- Y\. Xiao, M\. Jiang, J\. Sun, K\. Li, J\. Lin, Y\. Zhuang, J\. Zeng, S\. Xia, Q\. Hua, X\. Li, X\. Cai, T\. Wang, Y\. Zhang, L\. Liu, X\. Wu, J\. Hou, Y\. Cheng, W\. Li, X\. Wang, D\. Wang, and P\. Liu \(2025\)LIMI: less is more for agency\.arXiv preprintarXiv:2509\.17567\.Cited by:[§4\.1](https://arxiv.org/html/2607.11012#S4.SS1.SSS0.Px3.p1.1)\.
- A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Qiu \(2024a\)Qwen2\.5 technical report\.arXiv preprintarXiv:2412\.15115\.Cited by:[§4\.1](https://arxiv.org/html/2607.11012#S4.SS1.SSS0.Px1.p1.1)\.
- C\. Yang, W\. Lu, Y\. Zhu, Y\. Wang, Q\. Chena, C\. Gao, B\. Yan, and Y\. Chen \(2024b\)Survey on knowledge distillation for large language models: methods, evaluation, and application\.arXiv preprintarXiv:2407\.01885\.Cited by:[§1](https://arxiv.org/html/2607.11012#S1.p1.1)\.
- J\. Zeng, D\. Fu, T\. Mi, Y\. Zhuang, Y\. Huang, X\. Li, L\. Ye, M\. Xie, Q\. Hua, Z\. Huang, M\. Jiang, H\. Wang, J\. Lin, Y\. Xiao, J\. Sun, Y\. Wu, and P\. Liu \(2026\)DaVinci\-dev: agent\-native mid\-training for software engineering\.arXiv preprintarXiv:2601\.18418\.Cited by:[§4\.1](https://arxiv.org/html/2607.11012#S4.SS1.SSS0.Px3.p1.1)\.
- S\. Zhang, X\. Zhang, Z\. Sun, Y\. Chen, and J\. Xu \(2024\)Dual\-space knowledge distillation for large language models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing \(EMNLP 2024\),Miami, Florida, USA,pp\. 18164–18181\.Cited by:[§3\.4](https://arxiv.org/html/2607.11012#S3.SS4.p2.1),[§4\.1](https://arxiv.org/html/2607.11012#S4.SS1.SSS0.Px1.p1.1)\.
- S\. Zhang, X\. Zhang, T\. Zhang, B\. Hu, Y\. Chen, and J\. Xu \(2026\)KDFlow: a user\-friendly and efficient knowledge distillation framework for large language models\.arXiv preprintarXiv:2603\.01875\.Cited by:[Table 5](https://arxiv.org/html/2607.11012#A1.T5),[§1](https://arxiv.org/html/2607.11012#S1.p2.1)\.
- Q\. Zhong, M\. Zheng, M\. Song, X\. Lin, J\. Sun, H\. Jiang, X\. Wang, and J\. Fang \(2026\)SOD: step\-wise on\-policy distillation for small language model agents\.arXiv preprintarXiv:2605\.07725\.Cited by:[§1](https://arxiv.org/html/2607.11012#S1.p1.1),[§4\.1](https://arxiv.org/html/2607.11012#S4.SS1.SSS0.Px3.p1.1)\.
- Z\. Zhu, C\. Xie, X\. Lv, and slime Contributors \(2025\)Slime: an llm post\-training framework for rl scaling\.Note:[https://github\.com/THUDM/slime](https://github.com/THUDM/slime)GitHub repository\. Corresponding author: Xin LvCited by:[§1](https://arxiv.org/html/2607.11012#S1.p2.1)\.

## Appendix AFramework Comparison Details

OPD supervision\-interface coverageFramework propertiesFrameworkOPDCross\-tok\.OPDOn\-policyself\-distill\.Step\-wiseOPDReleasedYAML/CLIDocumented methodextension APICompatible w/multi\-nodeTRL✓∘\\circa∘\\circb×\\times✓×\\timesc∘\\circdverl✓×\\times×\\times∘\\circe✓×\\timesf✓slime✓g×\\times∘\\circh∘\\circe,g✓×\\timesf✓KDFlow✓✓✓i×\\times✓∘\\circj∘\\circkOurs✓✓✓✓✓✓✓
Table 5:Framework\-specific evidence supporting the ratings in Table[1](https://arxiv.org/html/2607.11012#S3.T1)\. Ratings reflect publicly documented functionality of the following versions: EasyOPD builds on verl 0\.5\.0, and we compare against TRL 0\.27\.0, the slime repository \(accessed 2025\-06\-19\), and the KDFlow implementation\(Zhanget al\.,[2026](https://arxiv.org/html/2607.11012#bib.bib70)\)\. Where a capability could not be verified from public documentation or code, it is marked as not documented rather than inferred as absent\.aCross\-tokenizer support is provided through experimental distillation components rather than the standard GKD path\.

bTRL provides an experimental SDPO trainer with live, base, and EMA teacher options, but the support is tied to a specialized trainer\.

cOPD variants are implemented as separate trainer classes rather than composable method\-level extension points\.

dMulti\-node trainer execution is available, but a disaggregated asynchronous rollout–teacher–trainer pipeline is not provided\.

eAgentic or step\-wise supervision is expressible through general reward/RLVR and multi\-turn infrastructure, but is not exposed as a dedicated OPD interface\.

fNew variants require method\-specific logic across internal rollout, reward/advantage, logit\-processing, or loss components\.

gslime provides OPD through its RL reward/advantage pipeline and offers strong agentic and multi\-turn rollout infrastructure\.

hA released slime example uses the original model as the teacher for self\-distillation, but does not expose a general live or EMA self\-teacher interface\.

iKDFlow supports student\-to\-teacher weight synchronization and EMA teacher updates for on\-policy self\-distillation\.

jKDFlow provides decoupled algorithm APIs, but they remain tied to its own distillation stack\.

kKDFlow supports distributed execution, while its documented on\-policy training loop is primarily synchronous\.

Table[5](https://arxiv.org/html/2607.11012#A1.T5)provides the framework\-specific evidence underlying the ratings in Table[1](https://arxiv.org/html/2607.11012#S3.T1)\.

## Appendix BExperimental Setup Details

We summarize the optimization, sampling, and hardware settings for the three case studies\. Unless noted otherwise, all settings are taken from the released configuration files and launch scripts\.

#### Software environment\.

All experiments run on a node with8×8\\timesNVIDIA H20 96GB GPUs \(driver 535\.161\.08, CUDA 12\.4\) under Ubuntu 22\.04, using the releasedOpenAgentRLconda environment \(Python 3\.11, PyTorch 2\.6\.0\+cu124, verl 0\.5\.0, Ray 2\.47\.1, Transformers 4\.51\.1, vLLM 0\.8\.5\.post1, SGLang 0\.4\.6\.post1, FlashAttention 2\.7\.4\.post1, and xFormers 0\.0\.29\.post2\)\. The full pinned dependency list and a one\-click installer are released with the code\.

#### Cross\-tokenizer OPD\.

We distill Qwen2\.5\-7B\-Instruct \(vocabulary size 151,936\) into a Phi\-4\-mini\-Instruct student \(3\.8B, vocabulary size 200,064\) that is first SFT\-warm\-started on a shared 10K mixed math\-and\-code corpus derived from the single\-turn prompt–answer pairs of themixed\_math\_code\_10kdataset\. Training uses GRPO with a cross\-tokenizer KL reward, a learning rate of5×10−75\\times 10^\{\-7\}with a0\.050\.05warmup ratio, a train batch size of6464, one epoch \(154154steps over the∼\\sim9\.9K training prompts\), maximum prompt and response lengths of40964096tokens, and single\-sample rollouts at temperature0\.60\.6\. All compared methods share this training\-prompt set and optimization budget\. We save a checkpoint every2020steps and, after merging, evaluate each checkpoint with an SGLang\-based inference server, reporting exact\-match accuracy for reasoning \(verified withmath\_verify\) and pass@1 for code on MATH500, GSM8K, MBPP, and LiveCodeBench v6\.

#### On\-policy self\-distillation\.

SDPO uses Qwen3\-8B as both the student and the self\-teacher, with an EMA self\-teacher \(update rate0\.050\.05\), top\-kklogit distillation withk=100k=100, and a symmetric KL interpolation \(α=0\.5\\alpha=0\.5\)\. Following the SDPO setup\(Hübotteret al\.,[2026](https://arxiv.org/html/2607.11012#bib.bib93)\), we evaluate a checkpoint every five training steps and report the best Average@16 observed within the one\-hour and five\-hour wall\-clock budgets, where Average@16 is the mean correctness over 16 independently sampled responses per question\.

#### Step\-wise OPD\.

We use Qwen3\-1\.7B as the student and a GRPO\-optimized Qwen3\-4B as the teacher, with a shared SFT initialization, training data, and optimization budget across compared methods\. Rollouts are multi\-turn tool\-interaction trajectories, and SOD applies step\-wise adaptive weighting \(with numerical\-stability offsetϵ=10−6\\epsilon=10^\{\-6\}and bound offsetδ=0\.2\\delta=0\.2\) on top of the response\-level OPD objective\. Each benchmark score is Average@32 over 32 independently sampled responses per problem\.

## Appendix CReleased Methods and Hooks

Table[6](https://arxiv.org/html/2607.11012#A3.T6)lists the released methods used in this work and the dispatch hooks each one registers\. Every method generates on\-policy rollouts through verl’s shared rollout engine; the hooks in the table are the*additional*method\-local extension points a method registers on top of this shared pipeline\. Because a hook only routes data or control to a method module, a method activates only the hooks it needs: a logit\-based objective such as GKD registers a single loss hook, whereas methods that require teacher\-side computation or per\-step trajectory metadata additionally register a teacher\-sidecar or rollout\-metadata hook\. All other pipeline stages, including rollout generation, worker orchestration, and optimization, are reused from verl unchanged\.

Table 6:Method\-local dispatch hooks registered by each released method, beyond the shared rollout–optimization pipeline that all methods reuse from verl\.✓marks an activated hook; the rollout\-metadata hook attaches per\-step or trajectory annotations and is distinct from rollout generation itself\. Cross\-tokenizer alignment in SimCT reuses the sharedsimpleteacher sidecar and is applied inside its loss path\.