UniSkill: Learning Actor-Aligned Skill Proposals for an Evolving Policy

Hugging Face Daily Papers Papers
llm-agents skill-learning reinforcement-learning reusable-skills alfworld webshop

Summary

UniSkill 提出一种共享策略与技能库共同演化的框架，通过对比动作反馈作为技能提案的 actor-alignment 信号，避免昂贵的额外 rollout，在 ALFWorld 达到 98.4% 成功率、WebShop 达到 84.7%。

Large language model agents can improve across tasks by retaining reusable skills distilled from prior interactions. Recent work jointly optimizes task execution and skill extraction, enabling the policy and skillbank to co-evolve. However, as the actor continues learning, rewarding skill proposals through their reuse in subsequent training steps may conflate skill benefits with actor improvement, while directly testing each proposed skill requires costly additional actor rollouts. In this paper, we introduce UniSkill, which uses a shared policy to interact with the environment and propose skillbank edits (Add, Update, or No Edit) from the resulting trajectories. Specifically, the actor learns from environment rewards, while contrastive action feedback guides skill proposal learning. This feedback provides an actor-alignment signal by measuring how replacing the retrieved skill with a proposed skill changes the current actor's action log-likelihood gap between previously collected successful and failed trajectories from the same task, thereby avoiding new rollouts for each proposal. Since proposal-level feedback may suppress an otherwise appropriate edit operation when the proposed skill content scores poorly, we further apply skill-edit support regularization to preserve exploration. Empirically, UniSkill achieves strong performance, reaching 98.4% success on ALFWorld and 84.7% on WebShop while maintaining stable joint training. Further ALFWorld experiments show that UniSkill remains effective when the shared policy uses a smaller backbone. Our implementation is available at https://github.com/LimOkii/UniSKill.
Original Article
View Cached Full Text

Cached at: 10/08/26, 12:11 PM

Paper page - UniSkill: Learning Actor-Aligned Skill Proposals for an Evolving Policy

Source: https://huggingface.co/papers/2610.10164

Abstract

Largelanguagemodelagentscanimproveacrosstasksbyretainingreusableskillsdistilledfrompriorinteractions.Recentworkjointlyoptimizestaskexecutionandskillextraction,enablingthepolicyandskillbanktoco-evolve.However,astheactorcontinueslearning,rewardingskillproposalsthroughtheirreuseinsubsequenttrainingstepsmayconflateskillbenefitswithactorimprovement,whiledirectlytestingeachproposedskillrequirescostlyadditionalactorrollouts.Inthispaper,weintroduceUniSkill,whichusesasharedpolicytointeractwiththeenvironmentandproposeskillbankedits(Add,Update,orNoEdit)fromtheresultingtrajectories.Specifically,theactorlearnsfromenvironmentrewards,whilecontrastiveactionfeedbackguidesskillproposallearning.Thisfeedbackprovidesanactor-alignmentsignalbymeasuringhowreplacingtheretrievedskillwithaproposedskillchangesthecurrentactor’sactionlog-likelihoodgapbetweenpreviouslycollectedsuccessfulandfailedtrajectoriesfromthesametask,therebyavoidingnewrolloutsforeachproposal.Sinceproposal-levelfeedbackmaysuppressanotherwiseappropriateeditoperationwhentheproposedskillcontentscorespoorly,wefurtherapplyskill-editsupportregularizationtopreserveexploration.Empirically,UniSkillachievesstrongperformance,reaching98.4%successonALFWorldand84.7%onWebShopwhilemaintainingstablejointtraining.FurtherALFWorldexperimentsshowthatUniSkillremainseffectivewhenthesharedpolicyusesasmallerbackbone.Ourimplementationisavailableathttps://github.com/LimOkii/UniSKill.

View arXiv pageView PDFGitHub1Add to collection

Get this paper in your agent:

hf papers read 2610\.10164

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.10164 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.10164 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.10164 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles