TurnSight: Turn-Level Hindsight Self-Distillation for Tool-Integrated Reasoning

Hugging Face Daily Papers Papers
reinforcement-learning self-distillation tool-integrated-reasoning llm hindsight

Summary

TurnSight introduces a turn-level hindsight self-distillation framework for tool-integrated reasoning, providing dense supervision via execution-conditioned hindsight and adaptive RL advantage modulation.

Tool-Integrated Reasoning (TIR) enables LLMs to solve complex tasks through iterative tool interactions. However, existing reinforcement learning methods often rely on trajectory-level supervision, limiting fine-grained credit assignment in long-horizon TIR scenarios. On-policy self-distillation offers denser signals through teacher branches with privileged context, but existing approaches typically derive such context from ground-truth answers or retrieved skills, which may not reflect the states actually visited by the agent. Moreover, token-level supervision fails to capture the turn-level structure of tool interactions. To address this, we propose TurnSight, a turn-level hindsight self-distillation framework that derives supervision directly from execution-conditioned hindsight. It then constructs multiple hindsight views with different lookahead horizons and selects reliable supervision through cross-horizon directional agreement. Finally, the selected hindsight signal is normalized across sibling rollouts and used to adaptively modulate RL advantages while preserving their original optimization direction. Extensive experiments on three benchmarks demonstrate the effectiveness of TurnSight. Our codes are available at https://github.com/quchangle1/TurnSight.
Original Article
View Cached Full Text

Cached at: 08/05/26, 05:43 AM

Paper page - TurnSight: Turn-Level Hindsight Self-Distillation for Tool-Integrated Reasoning

Source: https://huggingface.co/papers/2608.04007

Abstract

Tool-IntegratedReasoning(TIR)enablesLLMstosolvecomplextasksthroughiterativetoolinteractions.However,existingreinforcementlearningmethodsoftenrelyontrajectory-levelsupervision,limitingfine-grainedcreditassignmentinlong-horizonTIRscenarios.On-policyself-distillationoffersdensersignalsthroughteacherbrancheswithprivilegedcontext,butexistingapproachestypicallyderivesuchcontextfromground-truthanswersorretrievedskills,whichmaynotreflectthestatesactuallyvisitedbytheagent.Moreover,token-levelsupervisionfailstocapturetheturn-levelstructureoftoolinteractions.Toaddressthis,weproposeTurnSight,aturn-levelhindsightself-distillationframeworkthatderivessupervisiondirectlyfromexecution-conditionedhindsight.Itthenconstructsmultiplehindsightviewswithdifferentlookaheadhorizonsandselectsreliablesupervisionthroughcross-horizondirectionalagreement.Finally,theselectedhindsightsignalisnormalizedacrosssiblingrolloutsandusedtoadaptivelymodulateRLadvantageswhilepreservingtheiroriginaloptimizationdirection.ExtensiveexperimentsonthreebenchmarksdemonstratetheeffectivenessofTurnSight.Ourcodesareavailableathttps://github.com/quchangle1/TurnSight.

View arXiv pageView PDFProject pageGitHub0Add to collection

Get this paper in your agent:

hf papers read 2608\.04007

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.04007 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.04007 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.04007 in a Space README.md to link it from this page.

Collections including this paper1

Similar Articles

H^2SD: Hybrid Hindsight Self-Distillation

Hugging Face Daily Papers

Introduces H^2SD, a hybrid hindsight self-distillation framework that improves RLVR by using the teacher model differently for successful and failed trajectories, achieving better reasoning performance.