The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation

Hugging Face Daily Papers Papers
multi-turn long-horizon planning pre-training post-training distillation agentic

Summary

This paper introduces a controlled multi-turn environment to systematically study long-horizon planning ability in foundation model agents across pre-training, post-training (via GRPO and on-policy distillation), and integration (via multi-teacher on-policy distillation) stages.

Multi-turn long-horizon planning is critical for foundation model agents, yet how to fundamentally improve it remains unclear. Existing models are trained on uncontrollable and opaque Internet data, making it difficult to identify how planning ability is acquired, shaped, and integrated. To address this challenge, we introduce a unified and controlled multi-turn environment that enables precise control. It allows systematically study long-horizon planning across three stages. (1) Planning ability acquisition during pre-training. We study data format, distribution, and quality. Explicit world model construction through CoT state transition modeling yields stronger long-horizon generalization. Atomic skills alone are insufficient for compositional generalization, whereas a litte long-horizon data works. Moreover, suboptimal trajectories severely impair performance because errors amplify over long horizons. (2) Planning ability shaping via GRPO and OPD post-training. Through mutual information, we distinguish general planning patterns from task-specific planning knowledge. For planning patterns, we identify three application regions of post-training: unnecessary, effective, and unsupported. OPD has a broader effective region than GRPO under low-quality and long-horizon settings, as it provides more consistent update directions. For planning knowledge, distilling unseen procedures from a teacher with different knowledge may impair student's prior world modeling without fully establishing new knowledge. (3) Planning ability integration through MOPD post-training. We show that multi-teacher on-policy distillation (MOPD) integrates capabilities by converging to shared planning-pattern across environments. Compatible patterns enable cross-environment generalization, partially shared patterns support continual learning, while completely conflicting patterns cause severe interference.
Original Article
View Cached Full Text

Cached at: 07/28/26, 06:32 AM

Paper page - The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation

Source: https://huggingface.co/papers/2607.24720

Abstract

Multi-turnlong-horizonplanningiscriticalforfoundationmodelagents,yethowtofundamentallyimproveitremainsunclear.ExistingmodelsaretrainedonuncontrollableandopaqueInternetdata,makingitdifficulttoidentifyhowplanningabilityisacquired,shaped,andintegrated.Toaddressthischallenge,weintroduceaunifiedandcontrolledmulti-turnenvironmentthatenablesprecisecontrol.Itallowssystematicallystudylong-horizonplanningacrossthreestages.(1)Planningabilityacquisitionduringpre-training.Westudydataformat,distribution,andquality.ExplicitworldmodelconstructionthroughCoTstatetransitionmodelingyieldsstrongerlong-horizongeneralization.Atomicskillsaloneareinsufficientforcompositionalgeneralization,whereasalittelong-horizondataworks.Moreover,suboptimaltrajectoriesseverelyimpairperformancebecauseerrorsamplifyoverlonghorizons.(2)PlanningabilityshapingviaGRPOandOPDpost-training.Throughmutualinformation,wedistinguishgeneralplanningpatternsfromtask-specificplanningknowledge.Forplanningpatterns,weidentifythreeapplicationregionsofpost-training:unnecessary,effective,andunsupported.OPDhasabroadereffectiveregionthanGRPOunderlow-qualityandlong-horizonsettings,asitprovidesmoreconsistentupdatedirections.Forplanningknowledge,distillingunseenproceduresfromateacherwithdifferentknowledgemayimpairstudent’spriorworldmodelingwithoutfullyestablishingnewknowledge.(3)PlanningabilityintegrationthroughMOPDpost-training.Weshowthatmulti-teacheron-policydistillation(MOPD)integratescapabilitiesbyconvergingtosharedplanning-patternacrossenvironments.Compatiblepatternsenablecross-environmentgeneralization,partiallysharedpatternssupportcontinuallearning,whilecompletelyconflictingpatternscausesevereinterference.

View arXiv pageView PDFProject pageGitHub0Add to collection

Get this paper in your agent:

hf papers read 2607\.24720

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.24720 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.24720 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.24720 in a Space README.md to link it from this page.

Collections including this paper1

Similar Articles

MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training

Hugging Face Daily Papers

MOPD proposes a multi-teacher on-policy distillation paradigm for LLM post-training, enabling efficient integration of multiple domain capabilities by distilling specialized RL teachers into a student model using its own rollouts. It outperforms existing methods like Mix-RL and Cascade RL, and has been deployed in industrial-scale models.

Multi-Turn On-Policy Distillation with Prefix Replay

Hugging Face Daily Papers

This paper proposes ReOPD, a method for on-policy distillation of LLM agents that reuses pre-collected teacher trajectories as replayed prefixes, achieving improved efficiency and accuracy without new environment interactions.

The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes

Hugging Face Daily Papers

This paper presents a comprehensive empirical study on on-policy distillation for large language models, identifying failure mechanisms like distribution mismatch and optimization instability, and proposing fixes such as stop-gradient objectives and RLVR-adapted teachers.