SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning

Papers with Code Trending Papers
reinforcement-learning agentic-tasks policy-distillation on-policy self-evolving large-language-models

Summary

Proposes SEED, a self-evolving on-policy distillation framework that converts completed trajectories into hindsight skills to improve reinforcement learning for interactive agent tasks, achieving consistent performance gains and sample efficiency.

Large language models are increasingly trained as interactive agents for long-horizon tasks involving multi-turn interaction, tool use, and environment feedback. Outcome-based reinforcement learning (RL) provides a practical optimization paradigm, but its sparse trajectory-level rewards offer limited guidance on intermediate decisions, leaving a supervision gap between episode-level outcomes and token-level policy learning. We propose SEED (SElf-Evolving On-Policy Distillation), a self-evolving framework that converts completed on-policy trajectories into training-time hindsight skills and distills their behavioral effect back into the policy model. SEED first fine-tunes the policy to analyze completed trajectories and generate natural-language skills that capture reusable workflows, decisive observations, or failure-avoidance rules. During RL, the current policy both collects trajectories and serves as the analyzer that extracts hindsight skills from them. Policy updates therefore improve subsequent decision making and skill analysis together, allowing hindsight supervision to evolve with the policy. SEED then re-scores the sampled actions under ordinary and skill-augmented contexts, converting the skill-induced probability shift into a dense token-level on-policy distillation signal. This signal is jointly optimized with outcome-based RL, keeping the auxiliary supervision aligned with the current trajectory distribution. Extensive experiments on text-based and vision-based agentic tasks show that SEED consistently improves performance and sample efficiency, exhibiting robust generalization to unseen scenarios. Our code is available at https://github.com/jinyangwu/SEED.
Original Article
View Cached Full Text

Cached at: 07/20/26, 09:38 AM

Paper page - SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning

Source: https://huggingface.co/papers/2607.14777

Abstract

Largelanguagemodelsareincreasinglytrainedasinteractiveagentsforlong-horizontasksinvolvingmulti-turninteraction,tooluse,andenvironmentfeedback.Outcome-basedreinforcementlearning(RL)providesapracticaloptimizationparadigm,butitssparsetrajectory-levelrewardsofferlimitedguidanceonintermediatedecisions,leavingasupervisiongapbetweenepisode-leveloutcomesandtoken-levelpolicylearning.WeproposeSEED(SElf-EvolvingOn-PolicyDistillation),aself-evolvingframeworkthatconvertscompletedon-policytrajectoriesintotraining-timehindsightskillsanddistillstheirbehavioraleffectbackintothepolicymodel.SEEDfirstfine-tunesthepolicytoanalyzecompletedtrajectoriesandgeneratenatural-languageskillsthatcapturereusableworkflows,decisiveobservations,orfailure-avoidancerules.DuringRL,thecurrentpolicybothcollectstrajectoriesandservesastheanalyzerthatextractshindsightskillsfromthem.Policyupdatesthereforeimprovesubsequentdecisionmakingandskillanalysistogether,allowinghindsightsupervisiontoevolvewiththepolicy.SEEDthenre-scoresthesampledactionsunderordinaryandskill-augmentedcontexts,convertingtheskill-inducedprobabilityshiftintoadensetoken-levelon-policydistillationsignal.Thissignalisjointlyoptimizedwithoutcome-basedRL,keepingtheauxiliarysupervisionalignedwiththecurrenttrajectorydistribution.Extensiveexperimentsontext-basedandvision-basedagentictasksshowthatSEEDconsistentlyimprovesperformanceandsampleefficiency,exhibitingrobustgeneralizationtounseenscenarios.Ourcodeisavailableathttps://github.com/jinyangwu/SEED.

View arXiv pageView PDFProject pageGitHub124Add to collection

Get this paper in your agent:

hf papers read 2607\.14777

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper1

#### Jinyang23/Seed-AlfWorld-3B Text Generation• 3B• Updated3 days ago • 426 • 2

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.14777 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.14777 in a Space README.md to link it from this page.

Collections including this paper9

Browse 9 collections that include this paper

Similar Articles

OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning

Hugging Face Daily Papers

OPID proposes an on-policy skill distillation framework that extracts dense hindsight supervision from completed trajectories, combining outcome-based RL with token-level self-distillation to improve language agent training efficiency and performance on multi-turn tasks.

Self-Distilled Policy Gradient

Hugging Face Daily Papers

This paper proposes SDPG, a self-distilled policy-gradient framework that combines on-policy self-distillation with verifier advantages and KL regularization to improve reinforcement learning stability and performance.