DASH-OPD: Discrepancy-Aware Switching with Hysteresis for On-Policy Distillation

# DASH-OPD: Discrepancy-Aware Switching with Hysteresis for On-Policy Distillation
Source: [https://arxiv.org/html/2607.29078](https://arxiv.org/html/2607.29078)
Yuchen Xia1,2Qianguo Sun2Chao Song2Junlong Wu3Yiyan Qi2Yunjian Xu1††footnotemark: 1The Chinese University of Hong Kong2IDEA Research3Emdoor Research Institute 1\{ycxia,yjxu\}@mae\.cuhk\.edu\.hk 2\{xiayuchen,sunqianguo,songchao,qiyiyan\}@idea\.edu\.cn 3junlong\.wu@emdoor\.com

###### Abstract

On\-policy distillation \(OPD\) trains student models on their own rollouts to reduce exposure bias\. However, in multi\-turn agent scenarios, early student errors can lead a trajectory away from the teacher’s familiar domain\. Existing curriculum learning methods regulate how much teacher support is used according to training progress, but cannot determine when it is needed\. In light of this, we propose DASH\-OPD, Discrepancy\-Aware Switching with Hysteresis for OPD, a new agentic OPD method that can switch executors adaptively and bidirectionally\. On each turn, DASH\-OPD calculates a mean log\-probability ratio between the two executors over action tokens as their discrepancy\. Student\-to\-teacher ratios on student turns form drift signals, while teacher\-to\-student ratios on teacher turns form recovery signals\. These signals are normalized and accumulated over multiple turns into drift and recovery evidence\. DASH\-OPD switches executors when the evidence exceeds its corresponding switching threshold\. This multi\-turn accumulation makes the switching hysteretic, preventing high\-frequency switches caused by transient fluctuations\. On ALFWorld, DASH\-OPD outperforms all the baselines and demonstrates superior training and deployment efficiency\. This paper is a work in progress\. Code, training logs, and model checkpoints will be released later\.

## 1Introduction

![Refer to caption](https://arxiv.org/html/2607.29078v1/figs/fig1.png)Figure 1:Existing rollout curricula use fixed or random schedules for teacher intervention\.Large language model \(LLM\) agents solve long\-horizon tasks by interleaving reasoning, actions, and environment feedback\[yao2023react,wang2022scienceworld,yao2022webshop\], but their strongest capabilities often reside in models with excessively high deployment costs\[chen2024frugalgpt\]\. Knowledge distillation\[hinton2015distilling,kim2016sequence,gu2024minillm\]can transfer such capabilities to smaller student models\. On\-policy distillation \(OPD\)\[agarwal2024onpolicy\]further trains on student\-generated outputs to reduce the training\-inference mismatch \(i\.e\., exposure bias\)\. However, the advantage of OPD becomes fragile in multi\-turn interactions: early student errors can push the trajectory outside the teacher’s familiar domain, thereby decreasing the teacher’s instruction quality\[wang2026tcod,li2026guided\]\.

Teacher\-generated turns can repair a collapsing trajectory, but excessive intervention reduces student state coverage and recreates the exposure bias\. Recent agentic OPD methods seek to balance teacher and student control through rollout scheduling\. As shown in[Fig\.˜1](https://arxiv.org/html/2607.29078#S1.F1), TCOD\[wang2026tcod\]progressively expands student\-controlled segments in trajectories, while Guided\-OPD\[li2026guided\]samples each turn’s executor randomly and gradually decays the teacher\-intervention probability with training progress\. These methods regulate*how much*teacher support is used, but not*when*it is needed\. The need for teacher support can vary across different trajectories of the same training step, and across different segments of the same trajectory\[hoque2022thrifty,cai2025adaptive\]\.

![Refer to caption](https://arxiv.org/html/2607.29078v1/figs/fig2.png)Figure 2:DASH\-OPD accumulates teacher–student discrepancy into drift and recovery evidence, enabling adaptive and bidirectional switching between these two models\.To address this issue, we propose DASH\-OPD: Discrepancy\-Aware Switching with Hysteresis for OPD\. The overall pipeline of DASH\-OPD is shown in[Fig\.˜2](https://arxiv.org/html/2607.29078#S1.F2)\. After the current executor generates one turn, the other model scores the action in response, yielding a mean log\-probability ratio between the two executors of this turn\. The ratio quantifies teacher\-student discrepancy\. On student turns, the student\-to\-teacher ratios form drift signals, measuring how the student departs from the teacher’s familiar domain\. On teacher turns, the teacher\-to\-student ratios form recovery signals, measuring how the student re\-enters the teacher’s familiar domain\. These two types of signals are normalized and accumulated over multiple turns into two kinds of evidence, respectively\. The drift signals form drift evidence, building a support debt for the teacher\. The recovery signals form recovery evidence, building a return credit for the student\. DASH\-OPD switches between the teacher and the student by comparing the drift and recovery evidence with their corresponding switching thresholds\. Following Guided\-OPD, we train DASH\-OPD over all response tokens with a reverse\-KL distillation loss on student turns and a forward\-KL distillation loss on teacher turns\.

The switching mechanism of DASH\-OPD is hysteretic because it acts on the evidence accumulated over multiple turns rather than on an instantaneous signal\. The teacher support is only provided when sustained drift signals push the drift evidence beyond the support threshold\. Likewise, control returns to the student only when sustained recovery signals push the recovery evidence beyond the return threshold\. Therefore, transient fluctuations are insufficient to change the executor, preventing high\-frequency switches\.

To the best of our knowledge, DASH\-OPD is the*first*agentic OPD method that can switch executors adaptively and bidirectionally\. This capability directs teacher support to where it is most needed and yields improved performance\. On ALFWorld\[shridhar2021alfworld\], DASH\-OPD achieves the highest success rate across all evaluation splits and student scales, while requiring the fewest interaction rounds in five of six settings \(ranking second in the remaining one\)\. DASH\-OPD also demonstrates substantially greater training and deployment efficiency: it reduces teacher\-generated turns by 35\.4% and executor switches by 77\.1% relative to Guided\-OPD, and it Pareto\-dominates every OPD baseline in success rate and generation cost\.

## 2Related work

#### Agentic OPD\.

Turn\-level rollout scheduling has become a research frontier in agentic OPD\. TCOD\[wang2026tcod\]controls how far the student can advance in a trajectory as training progresses\. It progressively expands student\-controlled segments according to a fixed temporal curriculum\. Guided\-OPD\[li2026guided\]instead schedules who generates each turn\. It samples the teacher or the student at turn boundaries, using a gradually decreasing teacher\-intervention probability\. Both methods regulate teacher support according to a predefined curriculum\[bengio2009curriculum\], rather than switching executors adaptively\. TurnOPD\[zhou2026turnopd\]uses probe\-based turn statistics to adapt the rollout depth, stopping once later turns are unlikely to provide useful supervision\. This improves the flexibility of rollout scheduling, but it does not leverage the benefits of teacher support during rollout\.

#### General OPD\.

Token\-level rollout scheduling has also been studied in general OPD\[koo2025switch\]\. AdaSwitch\[peng2026adaswitch\]replaces a fixed switching threshold with a context\-dependent one, but permits only a single student\-to\-teacher handoff, after which the teacher completes the response\. Other work schedules how long the student rollout lasts\. POPD and TOPD\[zhang2026fullrollouts\]both control the student rollout length through predetermined horizons: POPD progressively expands the horizon with training progress, whereas TOPD keeps it fixed\. ESR\[zhou2026earlystopping\]similarly distills only the first fixed number of response tokens\.

## 3Preliminaries

### 3\.1Multi\-turn agents

Unlike static generation, a multi\-turn agent operates in a dynamic environment where each response influences the next\[liu2024agentbench\]\. At turntt, an environment emits observationoto\_\{t\}, and the agent conditions on historyht=\(x,o1,y1,…,ot\)h\_\{t\}=\(x,o\_\{1\},y\_\{1\},\\ldots,o\_\{t\}\), wherexxis the task prompt andyty\_\{t\}is a response whose parsed action producesot\+1o\_\{t\+1\}\. The responseyt=\(yt,1,…,yt,Lt\)y\_\{t\}=\(y\_\{t,1\},\\ldots,y\_\{t,L\_\{t\}\}\)is generated autoregressively by a policyπ\\pi:

π​\(yt∣ht\)=∏i=1Ltπ​\(yt,i∣ht,yt,<i\),\\pi\(y\_\{t\}\\mid h\_\{t\}\)=\\prod\_\{i=1\}^\{L\_\{t\}\}\\pi\(y\_\{t,i\}\\mid h\_\{t\},y\_\{t,<i\}\),\(1\)whereLtL\_\{t\}is the response length andyt,<i=\(yt,1,…,yt,i−1\)y\_\{t,<i\}=\(y\_\{t,1\},\\ldots,y\_\{t,i\-1\}\)\.

### 3\.2On\-policy distillation \(OPD\)

Unlike conventional distillation, OPD directly optimizes the policy on the data it generates, mitigating the exposure bias\. A trainable student policyπθ\\pi\_\{\\theta\}and a frozen teacher policyπT\\pi\_\{T\}are used in the training process \(evaluation is student\-only\)\.

Conventional distillation minimizes forward KL on teacher\-induced contexts\[hinton2015distilling,kullback1951information\]:

ℒforward=∑t𝔼ht∼qπTt​DKL​\(πT∥πθ\),\\mathcal\{L\}\_\{\\mathrm\{forward\}\}=\\sum\_\{t\}\\mathbb\{E\}\_\{h\_\{t\}\\sim q\_\{\\pi\_\{T\}\}^\{t\}\}D\_\{\\mathrm\{KL\}\}\(\\pi\_\{T\}\\\|\\pi\_\{\\theta\}\),\(2\)whereas a common OPD minimizes reverse KL on student\-induced contexts\[agarwal2024onpolicy\]:

ℒreverse=∑t𝔼ht∼qπθt​DKL​\(πθ∥πT\)\.\\mathcal\{L\}\_\{\\mathrm\{reverse\}\}=\\sum\_\{t\}\\mathbb\{E\}\_\{h\_\{t\}\\sim q\_\{\\pi\_\{\\theta\}\}^\{t\}\}D\_\{\\mathrm\{KL\}\}\(\\pi\_\{\\theta\}\\\|\\pi\_\{T\}\)\.\(3\)Here,qπTtq\_\{\\pi\_\{T\}\}^\{t\}andqπθtq\_\{\\pi\_\{\\theta\}\}^\{t\}denote the history distributions induced by the teacher and student policies, respectively\. In practice, training trajectories are generated by a frozen snapshotπθ¯\\pi\_\{\\bar\{\\theta\}\}of the student\.

## 4Method

### 4\.1Overview

DASH\-OPD switches the rollout executor adaptively and bidirectionally according to the discrepancy between the teacher and the student\. Letmt∈\{S,T\}m\_\{t\}\\in\\\{S,T\\\}denote the executor at turntt\. Exactly one model generates the response:

yt∼\{πθ¯\(⋅∣ht\),mt=S,πT\(⋅∣ht\),mt=T\.y\_\{t\}\\sim\\begin\{cases\}\\pi\_\{\\bar\{\\theta\}\}\(\\cdot\\mid h\_\{t\}\),&m\_\{t\}=S,\\\\ \\pi\_\{T\}\(\\cdot\\mid h\_\{t\}\),&m\_\{t\}=T\.\\end\{cases\}\(4\)The other model scores the action in this response under the same context, yielding a mean log\-probability ratio\. This ratio forms a discrepancy signal of turnttand updates an evidence variable used to select the executor at turnt\+1t\+1\.

The switching process has two directions\. On student turns, DASH\-OPD accumulates drift signals into drift evidence, which represents the teacher’s support debt\. The teacher provides support when the student progressively departs from the teacher’s familiar domain\. On teacher turns, DASH\-OPD accumulates recovery signals into recovery evidence, which represents the student’s return credit\. The control returns to the student when it re\-enters the teacher’s familiar domain\. Because a switch depends on evidence accumulated over multiple turns, rather than on an instantaneous discrepancy, the switching mechanism of DASH\-OPD is hysteretic\.

### 4\.2Directional discrepancy signals

To detect the teacher\-student discrepancy, DASH\-OPD compares the two models on their responses\. On student turntt, a student\-to\-teacher mean log\-probability ratio is computed as:

dtS=1NtA​∑iAt,i​\[log⁡πθ¯​\(yt,i∣ht,yt,<i\)−log⁡πT​\(yt,i∣ht,yt,<i\)\],yt∼πθ¯,d\_\{t\}^\{S\}=\\frac\{1\}\{N\_\{t\}^\{A\}\}\\sum\_\{i\}A\_\{t,i\}\[\\log\\pi\_\{\\bar\{\\theta\}\}\(y\_\{t,i\}\\mid h\_\{t\},y\_\{t,<i\}\)\-\\log\\pi\_\{T\}\(y\_\{t,i\}\\mid h\_\{t\},y\_\{t,<i\}\)\],\\quad y\_\{t\}\\sim\\pi\_\{\\bar\{\\theta\}\},\(5\)whereAt,iA\_\{t,i\}equals one when tokeniibelongs to the parsed action content, andNtA=∑iAt,iN\_\{t\}^\{A\}=\\sum\_\{i\}A\_\{t,i\}is the number of action tokens\. Thus, the discrepancy only considers the action portion of the response\. On teacher turns, the ratio is computed in the opposite direction using the same action\-only mask:

dtT=1NtA​∑iAt,i​\[log⁡πT​\(yt,i∣ht,yt,<i\)−log⁡πθ¯​\(yt,i∣ht,yt,<i\)\],yt∼πT\.d\_\{t\}^\{T\}=\\frac\{1\}\{N\_\{t\}^\{A\}\}\\sum\_\{i\}A\_\{t,i\}\[\\log\\pi\_\{T\}\(y\_\{t,i\}\\mid h\_\{t\},y\_\{t,<i\}\)\-\\log\\pi\_\{\\bar\{\\theta\}\}\(y\_\{t,i\}\\mid h\_\{t\},y\_\{t,<i\}\)\],\\quad y\_\{t\}\\sim\\pi\_\{T\}\.\(6\)
In expectation,dtSd\_\{t\}^\{S\}anddtTd\_\{t\}^\{T\}estimate reverse and forward KL, respectively\. A largedtSd\_\{t\}^\{S\}indicates that a student response is less familiar to the teacher and hence forms a drift signal\. A decreasingdtTd\_\{t\}^\{T\}indicates that the student re\-enters the teacher’s familiar domain and hence forms a recovery signal\. These two signals are kept separate because they are observed from different executors and describe different switching evidence\.

### 4\.3Drift and recovery evidence

On student turntt, DASH\-OPD updates drift\-signal statisticsℛtS=\(μtS,vtS\)\\mathcal\{R\}^\{S\}\_\{t\}=\(\\mu^\{S\}\_\{t\},v^\{S\}\_\{t\}\), whereμtS\\mu^\{S\}\_\{t\}is the mean of the drift signals observed previously, andvtSv^\{S\}\_\{t\}is the variance\. On teacher turns, the recovery\-signal statisticsℛtT=\(μtT,vtT\)\\mathcal\{R\}^\{T\}\_\{t\}=\(\\mu^\{T\}\_\{t\},v^\{T\}\_\{t\}\)are also updated\[welford1962corrected\]\. To detect multi\-turn trends in the drift and recovery signals, DASH\-OPD normalizes each signal by the corresponding statistics:

d¯tmt=clip⁡\(dtmt−μt−1mtmax⁡\(vt−1mt,vmin\),−d¯max,d¯max\),mt∈\{S,T\},\\bar\{d\}\_\{t\}^\{m\_\{t\}\}=\\operatorname\{clip\}\\\!\\left\(\\frac\{d\_\{t\}^\{m\_\{t\}\}\-\\mu\_\{t\-1\}^\{m\_\{t\}\}\}\{\\sqrt\{\\max\(v\_\{t\-1\}^\{m\_\{t\}\},v\_\{\\min\}\)\}\},\-\\bar\{d\}\_\{\\max\},\\bar\{d\}\_\{\\max\}\\right\),\\quad m\_\{t\}\\in\\\{S,T\\\},\(7\)wherevminv\_\{\\min\}denotes a small positive constant that prevents division by zero, andd¯max\\bar\{d\}\_\{\\max\}bounds extreme signals\. The first turn of each executor in a trajectory initializes the mean and receives a normalized signal of zero\. The statistics are retained when the same executor appears in multiple segments of a trajectory\.

The two types of normalized signals are accumulated into two evidence variables, drift evidenceDtD\_\{t\}and recovery evidenceRtR\_\{t\}, respectively\[page1954continuous\]\. On student turntt, DASH\-OPD updatesDtD\_\{t\}by:

Dt=max⁡\(ρ​Dt−1\+d¯tS−κS,0\),mt=S,D\_\{t\}=\\max\(\\rho D\_\{t\-1\}\+\\bar\{d\}\_\{t\}^\{S\}\-\\kappa\_\{S\},0\),\\quad m\_\{t\}=S,\(8\)whereρ∈\[0,1\]\\rho\\in\[0,1\]retains historical evidence andκS,κT≥0\\kappa\_\{S\},\\kappa\_\{T\}\\geq 0filter small fluctuations\. On teacher turns, DASH\-OPD updatesRtR\_\{t\}by:

Rt=max⁡\(ρ​Rt−1−d¯tT−κT,0\),mt=T\.R\_\{t\}=\\max\(\\rho R\_\{t\-1\}\-\\bar\{d\}\_\{t\}^\{T\}\-\\kappa\_\{T\},0\),\\quad m\_\{t\}=T\.\(9\)
Positive normalized drift signalsd¯tS\\bar\{d\}\_\{t\}^\{S\}increase drift evidence, and negative normalized recovery signalsd¯tT\\bar\{d\}\_\{t\}^\{T\}increase recovery evidence\. The rectification removes evidence in the opposite direction\. As a result, several moderate but consistent signals can trigger a switch, whereas a transient fluctuation is usually insufficient\.

### 4\.4Switching mechanism

The drift and recovery evidence mainly control the two switching directions\. The teacher\-support and student\-return thresholds are denoted byτon\\tau\_\{\\mathrm\{on\}\}andτoff\\tau\_\{\\mathrm\{off\}\}, respectively\. A trajectory with maximum lengthTmaxT\_\{\\max\}has a teacher budgetHmax=⌊q​Tmax⌋H\_\{\\max\}=\\lfloor qT\_\{\\max\}\\rfloor, whereq∈\[0,1\]q\\in\[0,1\]\. When the teacher budget is exhausted, control returns to the student\. A teacher segment has a minimum lengthℓmin\\ell\_\{\\min\}\. A teacher intervention is disallowed when the remaining teacher budget cannot support a complete segment ofℓmin\\ell\_\{\\min\}turns\.

After a student turn, the condition for teacher support is:

supportt=\(Dt\>τon∨gt\)∧\(Hmax−Ht≥ℓmin\),\\mathrm\{support\}\_\{t\}=\(D\_\{t\}\>\\tau\_\{\\mathrm\{on\}\}\\ \\lor\\ g\_\{t\}\)\\land\(H\_\{\\max\}\-H\_\{t\}\\geq\\ell\_\{\\min\}\),\(10\)whereHtH\_\{t\}denotes the total number of teacher\-generated turns, andgtg\_\{t\}denotes a flag that detects stagnation\. This flag is activated when an action or an observation repeats forKKconsecutive student turns\. Ifsupportt\\mathrm\{support\}\_\{t\}holds, DASH\-OPD setsmt\+1=Tm\_\{t\+1\}=Tand resets the drift evidence to zero\. After a teacher turn, the condition for returning control to the student is:

returnt=\(Rt\>τoff∨Ht≥Hmax\)∧ℓt≥ℓmin,\\mathrm\{return\}\_\{t\}=\(R\_\{t\}\>\\tau\_\{\\mathrm\{off\}\}\\lor H\_\{t\}\\geq H\_\{\\max\}\)\\land\\ell\_\{t\}\\geq\\ell\_\{\\min\},\(11\)whereℓt\\ell\_\{t\}denotes the length of the ongoing teacher segment\. Ifreturnt\\mathrm\{return\}\_\{t\}holds, DASH\-OPD setsmt\+1=Sm\_\{t\+1\}=Sand resets the recovery evidence to zero\.

#### Rollout initialization\.

For each trajectory, the executor of the first turn is selected randomly\. The probability of selecting the teacher decreases linearly as training progresses:

pT​\(s\)=1−s/Smax,p\_\{T\}\(s\)=1\-s/S\_\{\\max\},\(12\)wheressis the current training step andSmaxS\_\{\\max\}is the maximum number of training steps\. This warm start gives more teacher\-generated initial states early in training and decays linearly as the student improves\. It affects only the first turn, and all later switches follow[Eqs\.˜10](https://arxiv.org/html/2607.29078#S4.E10)and[11](https://arxiv.org/html/2607.29078#S4.E11)\.

### 4\.5Role\-aware distillation

Following Guided\-OPD\[li2026guided\], we use reverse\-KL distillation on student turns and forward\-KL distillation on teacher turns\. Letℒt,iS\\mathcal\{L\}\_\{t,i\}^\{S\}andℒt,iT\\mathcal\{L\}\_\{t,i\}^\{T\}denote the corresponding per\-token losses, and let𝕀tS=𝕀​\[mt=S\]\\mathbb\{I\}\_\{t\}^\{S\}=\\mathbb\{I\}\[m\_\{t\}=S\],𝕀tT=𝕀​\[mt=T\]\\mathbb\{I\}\_\{t\}^\{T\}=\\mathbb\{I\}\[m\_\{t\}=T\]\. The overall objective is:

ℒoverall=∑t,iMt,i​\(𝕀tS​ℒt,iS\+λT​𝕀tT​ℒt,iT\)max⁡\(∑t,iMt,i,1\),\\mathcal\{L\}\_\{\\mathrm\{overall\}\}=\\frac\{\\sum\_\{t,i\}M\_\{t,i\}\\left\(\\mathbb\{I\}\_\{t\}^\{S\}\\mathcal\{L\}\_\{t,i\}^\{S\}\+\\lambda\_\{T\}\\mathbb\{I\}\_\{t\}^\{T\}\\mathcal\{L\}\_\{t,i\}^\{T\}\\right\)\}\{\\max\\\!\\left\(\\sum\_\{t,i\}M\_\{t,i\},1\\right\)\},\(13\)whereλT\\lambda\_\{T\}controls the contribution of teacher\-generated tokens, andMt,iM\_\{t,i\}denotes the mask for theii\-th token in the response of thett\-th turn\.Mt,iM\_\{t,i\}covers all valid response tokens and is distinct from the action\-only discrepancy maskAt,iA\_\{t,i\}\. The executor selected at each turn also determines the distillation direction\.

### 4\.6Algorithm and cost

Algorithm 1DASH\-OPD for one training trajectory1:Student

πθ\\pi\_\{\\theta\}and its snapshot

πθ¯\\pi\_\{\\bar\{\\theta\}\}, teacher

πT\\pi\_\{T\}, environment

ℰ\\mathcal\{E\}, training step

ss, and controller hyperparameters

2:Initialize

Hmax←⌊q​Tmax⌋H\_\{\\max\}\\leftarrow\\lfloor qT\_\{\\max\}\\rfloor,

D,R,H,ℓ←0D,R,H,\\ell\\leftarrow 0, and

ℛS,ℛT←∅\\mathcal\{R\}^\{S\},\\mathcal\{R\}^\{T\}\\leftarrow\\emptyset
3:Sample

m1m\_\{1\}by[Eq\.˜12](https://arxiv.org/html/2607.29078#S4.E12)if

Hmax≥ℓminH\_\{\\max\}\\geq\\ell\_\{\\min\}; otherwise set

m1←Sm\_\{1\}\\leftarrow S
4:for

t=1,…,Tmaxt=1,\\ldots,T\_\{\\max\}do

5:Generate

yty\_\{t\}by[Eq\.˜4](https://arxiv.org/html/2607.29078#S4.E4)and obtain its log\-probabilities under

πθ¯\\pi\_\{\\bar\{\\theta\}\}and

πT\\pi\_\{T\}
6:Compute

dtmtd\_\{t\}^\{m\_\{t\}\}and

d¯tmt\\bar\{d\}\_\{t\}^\{m\_\{t\}\}by[Eqs\.˜5](https://arxiv.org/html/2607.29078#S4.E5),[6](https://arxiv.org/html/2607.29078#S4.E6)and[7](https://arxiv.org/html/2607.29078#S4.E7); update

ℛmt\\mathcal\{R\}^\{m\_\{t\}\}
7:Parse action

ata\_\{t\}from

yty\_\{t\}; record

\(ht,yt,mt\)\(h\_\{t\},y\_\{t\},m\_\{t\}\)and token log\-probabilities

8:if

mt=Sm\_\{t\}=Sthen

9:Update

gtg\_\{t\}from repetitions within the current student segment

10:Update

DDby[Eq\.˜8](https://arxiv.org/html/2607.29078#S4.E8)and evaluate

supportt\\mathrm\{support\}\_\{t\}by[Eq\.˜10](https://arxiv.org/html/2607.29078#S4.E10)

11:

\(mt\+1,D\)←\(T,0\)\(m\_\{t\+1\},D\)\\leftarrow\(T,0\)if

supportt\\mathrm\{support\}\_\{t\}; otherwise

mt\+1←Sm\_\{t\+1\}\\leftarrow S
12:else

13:

H←H\+1H\\leftarrow H\+1;

ℓ←ℓ\+1\\ell\\leftarrow\\ell\+1; update

RRby[Eq\.˜9](https://arxiv.org/html/2607.29078#S4.E9)

14:Evaluate

returnt\\mathrm\{return\}\_\{t\}by[Eq\.˜11](https://arxiv.org/html/2607.29078#S4.E11)

15:

\(mt\+1,R,ℓ\)←\(S,0,0\)\(m\_\{t\+1\},R,\\ell\)\\leftarrow\(S,0,0\)if

returnt\\mathrm\{return\}\_\{t\}; otherwise

mt\+1←Tm\_\{t\+1\}\\leftarrow T
16:endif

17:Advance

ℰ\\mathcal\{E\}with

ata\_\{t\};breakif terminal

18:endfor

19:Update

πθ\\pi\_\{\\theta\}by[Eq\.˜13](https://arxiv.org/html/2607.29078#S4.E13)

[Algorithm˜1](https://arxiv.org/html/2607.29078#alg1)presents the pseudocode of DASH\-OPD\. Computing the directional discrepancy costsO​\(Lt\)O\(L\_\{t\}\)for a response of lengthLtL\_\{t\}\. All subsequent controller operations takeO​\(1\)O\(1\)time per turn andO​\(1\)O\(1\)trajectory\-local memory\.

## 5Experiments

### 5\.1Experimental setup

#### Environment and metrics\.

We evaluate DASH\-OPD and baselines on ALFWorld\[shridhar2021alfworld\], a text\-based household environment that requires an agent to complete compositional tasks through multi\-turn interaction\. Following Guided\-OPD\[li2026guided\], we use all 140 IID tasks and all 134 OOD tasks in ALFWorld\. Each task is attempted once with a maximum of 30 interaction turns\. We report success rate \(SR, higher is better\) and the mean number of interaction turns \(Round, lower is better\)\.

#### Models and optimization\.

We study Qwen3\-1\.7B and Qwen3\-4B students with a frozen Qwen3\-30B\-A3B teacher\[yang2025qwen3\]\. At each student scale, every distilled policy is trained for 250 optimizer steps with a rollout batch size of 16, a training batch size of 64, a learning rate of10−610^\{\-6\}, and a KL coefficient of 1\.0\. Training rollouts use temperature 1\.0 and a maximum response length of 512 tokens\. All experiments are conducted on 8 NVIDIA A100 GPUs with 80GB memory each\.

#### DASH\-OPD configuration\.

We useρ=1\\rho=1,vmin=10−6v\_\{\\min\}=10^\{\-6\}, no signal bound, and no slack \(κS=κT=0\\kappa\_\{S\}=\\kappa\_\{T\}=0\)\. The switching thresholds are\(τon,τoff\)=\(3,1\)\(\\tau\_\{\\mathrm\{on\}\},\\tau\_\{\\mathrm\{off\}\}\)=\(3,1\), the minimum teacher\-segment length isℓmin=2\\ell\_\{\\min\}=2, and the stagnation threshold isK=3K=3\. The teacher budget spans the full 30\-turn horizon \(q=1q=1\)\. The same controller configuration is used at both student model scales\.

#### Evaluation protocol\.

All distilled policies are evaluated student\-only, without access to the teacher\. We use eight parallel workers, temperature 0\.4, top\-p=1p=1, no top\-kkor min\-pptruncation, a 4,096\-token response limit, a two\-turn observation–action history, and seed 42\. All the methods use the same evaluation configuration\.

#### Baselines and reporting convention\.

We compare against the zero\-shot student, vanilla OPD\[agarwal2024onpolicy\], TCOD\[wang2026tcod\], and Guided\-OPD\[li2026guided\]\. The zero\-shot teacher is included as a reference\. Vanilla OPD generates complete training trajectories with the student\. TCOD progressively expands the student\-controlled suffix according to training progress\. Guided\-OPD samples the executor at each turn with a decaying teacher\-intervention probability\. Each baseline uses its official implementation and training configuration\.

### 5\.2Main results

Table 1:Full\-set ALFWorld success rates \(SR, %\)\. Overall combines the IID and OOD splits and excludes the ALFWorld hard set\. Bold and underline mark the best and second\-best students, respectively\. The teacher reference is not included in this ranking\.MethodIID \(140 tasks\)OOD \(134 tasks\)Overall \(274 tasks\)SR↑\\uparrowRound↓\\downarrowSR↑\\uparrowRound↓\\downarrowSR↑\\uparrowRound↓\\downarrowQwen3\-30B\-A3B teacherZero\-shot45\.00%23\.1336\.57%24\.2740\.88%23\.69Qwen3\-1\.7B studentZero\-shot2\.86%29\.280\.00%30\.001\.46%29\.63Vanilla OPD23\.57%25\.4729\.10%25\.0126\.28%25\.25TCOD22\.86%25\.6431\.34%24\.9927\.01%25\.32Guided\-OPD22\.86%26\.1729\.10%25\.1025\.91%25\.65DASH\-OPD \(ours\)31\.43%24\.9834\.33%24\.6932\.85%24\.84Qwen3\-4B studentZero\-shot31\.43%24\.4936\.57%23\.8333\.94%24\.16Vanilla OPD34\.29%24\.3838\.81%24\.4036\.50%24\.39TCOD37\.86%23\.5932\.84%24\.2935\.40%23\.93Guided\-OPD35\.00%23\.8641\.04%23\.1037\.96%23\.49DASH\-OPD \(ours\)41\.43%22\.8641\.04%23\.4641\.24%23\.15#### Overall performance\.

[Table˜1](https://arxiv.org/html/2607.29078#S5.T1)shows that DASH\-OPD attains the highest or tied\-highest SR on every split at both student model scales\. With the 1\.7B student, DASH\-OPD achieves 32\.85% overall SR, exceeding the strongest baseline, TCOD, by 5\.84 percentage points\. With the 4B student, it reaches 41\.24% overall SR, improving over the strongest baseline, Guided\-OPD, by 3\.28 points\. The 4B DASH\-OPD student even exceeds the 30B\-A3B zero\-shot teacher on the overall SR\. DASH\-OPD also requires the fewest mean interaction rounds in five of the six settings and ranks second in the remaining one\.

![Refer to caption](https://arxiv.org/html/2607.29078v1/x1.png)Figure 3:Deployment efficiency on the full ALFWorld validation set\. Overall SR is plotted against the mean number of generated response tokens per trajectory over 274 tasks\. The star denotes the zero\-shot teacher reference, and the upper\-left direction is preferable\. DASH\-OPD lies above and to the left of every OPD baseline at both student model scales\.
#### Deployment efficiency\.

[Figure˜3](https://arxiv.org/html/2607.29078#S5.F3)shows that DASH\-OPD strictly Pareto\-dominates every distilled baseline at both student scales: it achieves higher SR while generating fewer response tokens per evaluation trajectory\. For the 1\.7B student, DASH\-OPD reaches 32\.85% SR with 1,433\.1 tokens per trajectory\. Relative to Guided\-OPD, the strongest baseline in[Figure˜3](https://arxiv.org/html/2607.29078#S5.F3), this corresponds to a 6\.94 percentage point SR gain with 89\.9 fewer tokens\. For the 4B student, DASH\-OPD obtains 41\.24% SR with 1,218\.2 tokens, improving over Guided\-OPD by 3\.28 points on SR while using 12\.1 fewer tokens\. The 4B DASH\-OPD student also exceeds the teacher reference by 0\.36 points on SR while generating 68\.5 fewer tokens\.

![Refer to caption](https://arxiv.org/html/2607.29078v1/x2.png)Figure 4:Teacher intervention profiles for the Qwen3\-4B student\. We plot the cumulative number of teacher\-generated turns \(left\) and the number of teacher–student switches \(right\) over 250 training steps\. Both vertical axes are in thousands\. DASH\-OPD requires substantially fewer teacher turns and executor switches than Guided\-OPD by the end of training\.![Refer to caption](https://arxiv.org/html/2607.29078v1/x3.png)Figure 5:Persistence after returning control to the student during DASH\-OPD training\. Left: the fraction of returns followed by at least three or five student turns \(task completion before the respective horizon also counts as persistence\)\. Right: the mean number of consecutive student turns following each return, measured until the next teacher takeover or trajectory termination\.
#### Teacher intervention efficiency\.

[Figure˜4](https://arxiv.org/html/2607.29078#S5.F4)compares the teacher intervention profiles of DASH\-OPD and Guided\-OPD with 4B students\. Guided\-OPD accumulates teacher turns throughout training rapidly, while its switching count increases markedly after approximately 50 steps\. DASH\-OPD follows a lower curve for teacher turns and a more gradual switching curve that remains below Guided\-OPD after approximately 75 steps\. By the end of training, DASH\-OPD uses 23,543 teacher turns, compared with 36,444 for Guided\-OPD, reducing teacher intervention by 12,901 turns \(35\.4%\)\. DASH\-OPD also reduces executor switches from 17,254 for Guided\-OPD to 3,947, eliminating 13,307 events \(77\.1%\)\. Hence, DASH\-OPD’s discrepancy\-aware switching utilizes teacher support more efficiently than Guided\-OPD’s random sampling strategy\.

#### Persistence after returning student control\.

[Figure˜5](https://arxiv.org/html/2607.29078#S5.F5)examines whether a return to the student initiates a sustained segment or an immediate reversal\. After control returns to the 1\.7B student, the student remains in control for at least three turns in 84% of cases, and for at least five turns in 54% of cases\. These rates rise to 91% and 73% for the 4B student\. The mean post\-return student span similarly increases from 8\.0 turns for 1\.7B to 11\.7 turns for 4B\. Together with the 77\.1% reduction in total switches for the 4B run, these persistent student segments are consistent with the intended hysteretic behavior: evidence\-based returns restore student control without frequent teacher–student oscillation\. Moreover, the higher persistence of the student suggests stronger student capabilities and more effective teacher intervention\.

## 6Conclusion and limitations

#### Conclusion\.

We presented DASH\-OPD, an adaptive, bidirectional switching method for multi\-turn OPD\. It accumulates directional teacher–student discrepancy into drift and recovery evidence, then applies thresholds to decide teacher intervention and student return hysteretically\. On ALFWorld, DASH\-OPD attains the best success rate across all splits and student scales\. It also demonstrates substantially greater training and deployment efficiency\. These results show that DASH\-OPD establishes a new state\-of\-the\-art for agentic OPD\.

#### Limitations\.

Student\-teacher discrepancy is an imperfect proxy for risk: shared errors may remain undetected, whereas benign differences may trigger intervention\. Future work could explore more sophisticated risk metrics, such as those based on model uncertainty or environment feedback\.

## References