RSIPO: Recursive Self-Improving Policy Optimization for Agentic Reinforcement Learning via Skill-Policy Co-Evolution

Hugging Face Daily Papers Papers
agentic-rl reinforcement-learning self-improvement skill-learning policy-optimization language-models

Summary

提出了RSIPO框架，通过技能与策略的协同演化实现智能体强化学习的递归自我优化，在ALFWorld上以Qwen2.5-7B达到98.6%的成功率，显著超越GRPO等基线。

Recent focus in agentic reinforcement learning (RL) seeks to improve credit assignment in long-horizon tasks. One promising direction combines reinforcement learning with verifiable rewards (RLVR) and on-policy self-distillation (OPSD), integrating trajectory-level advantages with dense token-level supervision. This line of work often conditions a teacher model on privileged context, such as agentic skills, to score tokens generated by the student policy. Despite their progress, they largely overlook the on-policy evolution of agentic skills and the reliability of teacher supervision. Towards this end, we introduce Recursive Self-Improving Policy Optimization (RSIPO), a self-evolving agentic RL framework that enables agentic skills and the policy model to co-evolve throughout the training process. Specifically, RSIPO enables the policy itself to extract skills from completed trajectories and explicitly optimizes its skill-generation capability through trajectory replay. It further employs empirical Bayes shrinkage to dynamically calibrate the student's trust in teacher scores, accounting for the fact that skill-enhanced teacher guidance can sometimes be less reliable. Experiments with Qwen-family models on embodied interaction and search-based agentic tasks demonstrate that RSIPO outperforms all the compared baseline approaches. With Qwen2.5-7B as the base model, RSIPO achieves an average success rate of 98.6\% on ALFWorld, yielding absolute improvements of 2.5\% and 17.4\% over the second-best baseline and standard GRPO, respectively.
Original Article
View Cached Full Text

Cached at: 10/14/26, 02:49 AM

Paper page - RSIPO: Recursive Self-Improving Policy Optimization for Agentic Reinforcement Learning via Skill-Policy Co-Evolution

Source: https://huggingface.co/papers/2610.15772

Abstract

Recentfocusinagenticreinforcementlearning(RL)seekstoimprovecreditassignmentinlong-horizontasks.Onepromisingdirectioncombinesreinforcementlearningwithverifiablerewards(RLVR)andon-policyself-distillation(OPSD),integratingtrajectory-leveladvantageswithdensetoken-levelsupervision.Thislineofworkoftenconditionsateachermodelonprivilegedcontext,suchasagenticskills,toscoretokensgeneratedbythestudentpolicy.Despitetheirprogress,theylargelyoverlooktheon-policyevolutionofagenticskillsandthereliabilityofteachersupervision.Towardsthisend,weintroduceRecursiveSelf-ImprovingPolicyOptimization(RSIPO),aself-evolvingagenticRLframeworkthatenablesagenticskillsandthepolicymodeltoco-evolvethroughoutthetrainingprocess.Specifically,RSIPOenablesthepolicyitselftoextractskillsfromcompletedtrajectoriesandexplicitlyoptimizesitsskill-generationcapabilitythroughtrajectoryreplay.ItfurtheremploysempiricalBayesshrinkagetodynamicallycalibratethestudent’strustinteacherscores,accountingforthefactthatskill-enhancedteacherguidancecansometimesbelessreliable.ExperimentswithQwen-familymodelsonembodiedinteractionandsearch-basedagentictasksdemonstratethatRSIPOoutperformsallthecomparedbaselineapproaches.WithQwen2.5-7Basthebasemodel,RSIPOachievesanaveragesuccessrateof98.6\%onALFWorld,yieldingabsoluteimprovementsof2.5\%and17.4\%overthesecond-bestbaselineandstandardGRPO,respectively.

View arXiv pageView PDFGitHubAdd to collection

Get this paper in your agent:

hf papers read 2610\.15772

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.15772 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.15772 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.15772 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles