H^2SD: Hybrid Hindsight Self-Distillation

Hugging Face Daily Papers Papers
reinforcement-learning self-distillation large-language-models reasoning rlvr hybrid-methods

Summary

Introduces H^2SD, a hybrid hindsight self-distillation framework that improves RLVR by using the teacher model differently for successful and failed trajectories, achieving better reasoning performance.

Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning capabilities of large language models on tasks such as mathematical reasoning and code generation. However, most RLVR methods assign a scalar outcome reward to an entire trajectory, resulting in sparse supervision and limited token-level credit assignment. On-policy distillation (OPD) provides denser supervision by distilling token-level distributions from a stronger teacher model, but requires an additional teacher and typically assumes a shared vocabulary. On-policy self-distillation (OPSD) removes this dependency by conditioning the same model on privileged information to construct a teacher policy. However, directly matching the teacher distribution may cause information leakage and unstable optimization. RLSD avoids direct matching by using the teacher signal only to modulate update magnitudes, but it cannot provide an explicit correction direction when the sampled reasoning fails. To address this tradeoff, we introduce H^{2}SD, a hybrid hindsight self distillation framework that uses the teacher differently according to trajectory correctness. For successful trajectories, the teacher receives the student response confirmed as correct together with a rephrasing instruction, and its probabilities on the original response tokens are used to modulate update magnitudes without changing the direction determined by the reward. For failed trajectories, we condition the teacher on a reference hint containing key reasoning steps and a verified answer, and minimize the reverse KL divergence from the student to the teacher. Experiments on multiple challenging reasoning benchmarks show that H^2SD consistently outperforms representative RLVR, OPSD, and RLSD baselines while maintaining stable optimization and favorable generation efficiency.
Original Article
View Cached Full Text

Cached at: 07/22/26, 02:40 AM

Paper page - H^2SD: Hybrid Hindsight Self-Distillation

Source: https://huggingface.co/papers/2607.18955

Abstract

Reinforcementlearningwithverifiablerewards(RLVR)hassubstantiallyimprovedthereasoningcapabilitiesoflargelanguagemodelsontaskssuchasmathematicalreasoningandcodegeneration.However,mostRLVRmethodsassignascalaroutcomerewardtoanentiretrajectory,resultinginsparsesupervisionandlimitedtoken-levelcreditassignment.On-policydistillation(OPD)providesdensersupervisionbydistillingtoken-leveldistributionsfromastrongerteachermodel,butrequiresanadditionalteacherandtypicallyassumesasharedvocabulary.On-policyself-distillation(OPSD)removesthisdependencybyconditioningthesamemodelonprivilegedinformationtoconstructateacherpolicy.However,directlymatchingtheteacherdistributionmaycauseinformationleakageandunstableoptimization.RLSDavoidsdirectmatchingbyusingtheteachersignalonlytomodulateupdatemagnitudes,butitcannotprovideanexplicitcorrectiondirectionwhenthesampledreasoningfails.Toaddressthistradeoff,weintroduceH^{2}SD,ahybridhindsightselfdistillationframeworkthatusestheteacherdifferentlyaccordingtotrajectorycorrectness.Forsuccessfultrajectories,theteacherreceivesthestudentresponseconfirmedascorrecttogetherwitharephrasinginstruction,anditsprobabilitiesontheoriginalresponsetokensareusedtomodulateupdatemagnitudeswithoutchangingthedirectiondeterminedbythereward.Forfailedtrajectories,weconditiontheteacheronareferencehintcontainingkeyreasoningstepsandaverifiedanswer,andminimizethereverseKLdivergencefromthestudenttotheteacher.ExperimentsonmultiplechallengingreasoningbenchmarksshowthatH^2SDconsistentlyoutperformsrepresentativeRLVR,OPSD,andRLSDbaselineswhilemaintainingstableoptimizationandfavorablegenerationefficiency.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2607\.18955

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.18955 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.18955 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.18955 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Learning More from Less: Reinforcement Learning from Hindsight

arXiv cs.LG

Introduces Learning from Hindsight (LfH), a method that applies hindsight relabeling to RL post-training of vision-language-action models. By relabeling failed robot rollouts with the tasks they actually achieved, LfH achieves 5x improvement in sample efficiency on out-of-distribution manipulation tasks.

Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information

Hugging Face Daily Papers

Proposes Anti-Self-Distillation (AntiSD) which reverses the knowledge transfer direction in self-distillation to improve math reasoning efficiency and accuracy, achieving GRPO baseline accuracy in 2-10x fewer steps and up to 11.5 points higher final accuracy across models from 4B to 30B parameters.

Self-Distilled Policy Gradient

arXiv cs.LG

SDPG (Self-Distilled Policy Gradient) is a new RL training framework for LLMs that combines group-relative verifier advantages with on-policy self-distillation and KL regularization to address sparse rewards and instability in RLVR training. The method uses a shared model as both student and teacher by conditioning on privileged context, showing improved stability and performance over RLVR and self-distillation baselines.

Adaptive Teacher Exposure for Self-Distillation in LLM Reasoning

Hugging Face Daily Papers

Adaptive Teacher Exposure for Self-Distillation (ATESD) improves LLM reasoning by dynamically adjusting how much of the reference reasoning the teacher shows the student during training, using a learnable policy controller and a discounted learning-progress reward. Experiments on math benchmarks show consistent improvements over existing self-distillation and RL baselines.