Paper page - Agent Skills Should Go Beyond Text: The Case for Visual Skills

Source: https://huggingface.co/papers/2606.01414

Abstract

Multimodal skills that combine textual logic with visual support outperform text-only approaches in visual-centric tasks by incorporating spatial layout, visual grounding, and state-aware interactions.

Reusable skillsare a key mechanism for extending agent capabilities, allowing agents to accumulate experience and solve increasingly complex tasks. Yet most existing skill-learning methods store reusable experience as text-only assets, such as instructions, reasoning traces, or summarized trajectories. We argue that this text-only paradigm creates a fundamental bottleneck forvisual-centric tasks, where reusable knowledge often depends on spatial layout, visual grounding, fine-grained appearance, and localized state changes. To address this limitation, we propose \NAME, amultimodal skill paradigmthat combinesdeclarative textual logicwith explicitvisual support. We distinguish three reusable forms:static priorsfor stable spatial conventions,dynamic priorsfor in-situ visual working memory, andinterleaved visual skillsthat bind ordered text steps to the source frames, screenshots, or page regions that justify them. Rather than only describing what to do,visual skillsalso encode where to look, how to inspect, and how to verify visual outcomes. To scale visual-skill construction, we introduce \SYSTEM, anautomatic systemthat converts agent experience into reusable multimodal skills by preserving textual reasoning, spatial references, visual boundaries, and interaction patterns fromtask trajectories. Experiments on GUI and othervisual-centric tasksshow thatvisual skillsconsistently outperform text-only skills, particularly when success requiresspatial correspondence,visual evidence, andstate-aware interaction. These results support our central position: reusable agent skills should go beyond text and become multimodal assets for future multimodal agents.

View arXiv pageView PDFGitHub4Add to collection

Get this paper in your agent:

hf papers read 2606\.01414

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.01414 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2606.01414 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.01414 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.