ViSkill: Reinforcing VLM Agents with Evolving Visual-Native Skills

Hugging Face Daily Papers Papers
vlm-agents skill-learning reinforcement-learning visual-skills computer-use

Summary

ViSkill 提出一种视觉原生的技能学习框架，将成功交互编码为可复用的视觉技能卡，与策略优化形成闭环反馈，在 Sokoban、FrozenLake 等基准上超越多种基线。

Skill-augmented agents improve sample efficiency by distilling successful trajectories into reusable strategies. Yet most existing approaches remain text-centric, linearizing spatial layouts and action-state correspondences into language that loses critical geometric structure. Recent efforts have begun incorporating visual evidence, but construct and update skills separately from policy optimization, leaving their mutual improvement underexplored. We propose ViSkill, a visual-native skill learning framework that encodes successful interactions as composite visual skill cards directly accessible to VLM agents. Retrieved skills guide both inference and reward shaping, while successful trajectories are distilled back into the library, forming a closed feedback loop in which skill accumulation and policy improvement reinforce each other. An optional cold-start mechanism further accelerates early-stage learning. Evaluated on Sokoban, FrozenLake, and PrimitiveSkill, ViSkill achieves an overall success rate of 0.89, rising to 0.91 with cold-start initialization, outperforming all evaluated proprietary and open-source baselines while converging faster than standard PPO. Our code is available at https://github.com/ZJU-REAL/ViSkill.
Original Article
View Cached Full Text

Cached at: 10/09/26, 06:25 AM

Paper page - ViSkill: Reinforcing VLM Agents with Evolving Visual-Native Skills

Source: https://huggingface.co/papers/2610.12403

Abstract

Skill-augmentedagentsimprovesampleefficiencybydistillingsuccessfultrajectoriesintoreusablestrategies.Yetmostexistingapproachesremaintext-centric,linearizingspatiallayoutsandaction-statecorrespondencesintolanguagethatlosescriticalgeometricstructure.Recenteffortshavebegunincorporatingvisualevidence,butconstructandupdateskillsseparatelyfrompolicyoptimization,leavingtheirmutualimprovementunderexplored.WeproposeViSkill,avisual-nativeskilllearningframeworkthatencodessuccessfulinteractionsascompositevisualskillcardsdirectlyaccessibletoVLMagents.Retrievedskillsguidebothinferenceandrewardshaping,whilesuccessfultrajectoriesaredistilledbackintothelibrary,formingaclosedfeedbackloopinwhichskillaccumulationandpolicyimprovementreinforceeachother.Anoptionalcold-startmechanismfurtheracceleratesearly-stagelearning.EvaluatedonSokoban,FrozenLake,andPrimitiveSkill,ViSkillachievesanoverallsuccessrateof0.89,risingto0.91withcold-startinitialization,outperformingallevaluatedproprietaryandopen-sourcebaselineswhileconvergingfasterthanstandardPPO.Ourcodeisavailableathttps://github.com/ZJU-REAL/ViSkill.

View arXiv pageView PDFGitHub2Add to collection

Models citing this paper2

#### hongxingli/ViSkill-Sokoban Image-Text-to-Text• 4B• Updatedabout 3 hours ago #### hongxingli/ViSkill-FrozenLake Image-Text-to-Text• 4B• Updatedabout 3 hours ago

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.12403 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.12403 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

AtlasVA: Self-Evolving Visual Skill Memory for Teacher-Free VLM Agents

Hugging Face Daily Papers

AtlasVA is a teacher-free visual skill memory framework for vision-language model agents that uses spatial heatmaps, visual exemplars, and symbolic text skills to improve spatial decision-making in long-horizon tasks, outperforming baselines on several benchmarks.

VISUALSKILL: Multimodal Skills for Computer-Use Agents

arXiv cs.CL

VisualSkill proposes a hierarchical multimodal skill library for computer-use agents that combines text and figures, achieving a 15.3 point absolute lift on CUA benchmarks over text-only baselines by retaining visual information for GUI interaction.