TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Training
Summary
TurnOPD introduces turn-level budgeting for on-policy distillation of long-horizon agents, addressing inefficiencies in vanilla OPD by adaptive rollout-depth and progressive turn-normalized loss budgeting, achieving better accuracy under equal training budgets.
View Cached Full Text
Cached at: 07/08/26, 04:38 AM
# TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Training Source: [https://arxiv.org/abs/2607.05804](https://arxiv.org/abs/2607.05804) [View PDF](https://arxiv.org/pdf/2607.05804) > Abstract:On\-policy distillation \(OPD\) trains a student policy by matching a stronger teacher on the student's own trajectories, offering a promising framework for language agent training\. However, its application to long\-horizon agentic tasks remains insufficiently explored\. We identify two key inefficiencies in vanilla agent OPD: \(1\) full\-horizon rollouts often waste wall\-clock resources on tail turns that provide weak and noisy KL supervision, and \(2\) trajectory\-level KL objectives concentrate most of the loss on shallow tokens, leaving deeper decision turns under\-trained once initial behaviors are aligned\. To address these challenges, we propose TurnOPD, a turn\-level budgeting strategy for efficient on\-policy distillation of long\-horizon agents\. TurnOPD consists of two budget controllers: adaptive rollout\-depth budgeting, which uses probe\-based turn statistics to determine rollout length, and progressive turn\-normalized loss budgeting, which gradually shifts KL weighting from token\-level to turn\-balanced supervision\. Experiments on ALFWorld, WebShop, and Multi\-Hop Search with task\-specialized teacher models show that TurnOPD achieves superior validation accuracy under equal wall\-clock training budgets and advances the accuracy\-\-time frontier beyond vanilla OPD\. ## Submission history From: Yuhang Zhou \[[view email](https://arxiv.org/show-email/dca82c47/2607.05804)\] **\[v1\]**Tue, 7 Jul 2026 03:56:35 UTC \(2,259 KB\)
Similar Articles
Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation
This paper investigates the parameter-level mechanisms behind the efficiency of On-Policy Distillation (OPD) for large language models, attributing it to early 'foresight' in module allocation and update direction. It proposes EffOPD, a plug-and-play method that accelerates OPD training by 3x without compromising final performance.
ATOD: Annealed Turn-aware On-policy Distillation for Multi-turn Autonomous Agents
The paper introduces ATOD, a hybrid online distillation algorithm combining on-policy distillation and reinforcement learning for training small language model agents in multi-turn tasks, featuring an annealed OPD-RL schedule and Turn-level Disagreement-Uncertainty Reweighting to improve dense supervision.
ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation
ShortOPD proposes a short-to-long on-policy distillation schedule that recovers pruned LLMs for free-form generation by focusing training on effective prefixes, achieving up to 9x improvement over unrecovered models and matching long-horizon distillation with a quarter of the training time.
DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Models
DiffusionOPD proposes a multi-task training paradigm for diffusion models that uses online policy distillation to efficiently combine task-specific teachers into a unified student, achieving state-of-the-art results on all evaluated benchmarks.
@louieworth: New blog post: On-Policy Distillation — Promise, Pitfalls, and Prospects. OPD combines on-policy rollouts with dense te…
This blog post discusses On-Policy Distillation (OPD), a technique that combines on-policy rollouts with dense teacher supervision, and highlights its promise, three failure modes, and the author's new paper on the topic.