Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents
Summary
Agent S2 is a new compositional framework for computer use agents that achieves state-of-the-art performance on multiple benchmarks by utilizing Mixture-of-Grounding and Proactive Hierarchical Planning.
View Cached Full Text
Cached at: 05/08/26, 09:02 AM
Paper page - Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents
Source: https://huggingface.co/papers/2504.00906
Abstract
Agent S2, a compositional framework using Mixture-of-Grounding and Proactive Hierarchical Planning, achieves state-of-the-art performance in computer use automation across various benchmarks and operating systems.
Computer use agents automate digital tasks by directly interacting with graphical user interfaces (GUIs) on computers and mobile devices, offering significant potential to enhance human productivity by completing an open-ended space of user queries. However, current agents face significant challenges: imprecise grounding of GUI elements, difficulties with long-horizon task planning, and performance bottlenecks from relying on single generalist models for diverse cognitive tasks. To this end, we introduce Agent S2, a novelcompositional frameworkthat delegates cognitive responsibilities across various generalist and specialist models. We propose a novelMixture-of-Groundingtechnique to achieve precise GUI localization and introduceProactive Hierarchical Planning, dynamically refining action plans at multiple temporal scales in response to evolving observations. Evaluations demonstrate that Agent S2 establishes new state-of-the-art (SOTA) performance on three prominent computer use benchmarks. Specifically, Agent S2 achieves 18.9% and 32.7% relative improvements over leading baseline agents such as Claude Computer Use and UI-TARS on the OSWorld 15-step and 50-step evaluation. Moreover, Agent S2 generalizes effectively to other operating systems and applications, surpassing previous best methods by 52.8% on WindowsAgentArena and by 16.52% on AndroidWorld relatively. Code available at https://github.com/simular-ai/Agent-S.
View arXiv pageView PDFProject pageGitHub11.2kAdd to collection
Get this paper in your agent:
hf papers read 2504\.00906
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2504.00906 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2504.00906 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2504.00906 in a Space README.md to link it from this page.
Collections including this paper12
Similar Articles
PresentAgent-2: Towards Generalist Multimodal Presentation Agents
PresentAgent-2 is an agentic framework that generates presentation videos from user queries by conducting research, creating multimodal slides, and producing interactive content across single, discussion, and interaction modes.
Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligence
Agent-World introduces a self-evolving training framework for general agent intelligence that autonomously discovers real-world environments and tasks via the Model Context Protocol, enabling continuous learning. Agent-World-8B and 14B models outperform strong proprietary models across 23 challenging agent benchmarks.
AgentOS
AgentOS provides a unified control layer for managing AI agents, tasks, and workspaces.
@svpino: This is the architecture pattern that's going to kill single-model tools: You send a prompt, the agent breaks it into s…
Higgsfield AI introduces the Supercomputer, a cloud-native self-learning AI agent that breaks tasks into sub-tasks and routes each to the best model (e.g., reasoning to Opus, video to Seedance, images to GPT), with three layers of memory for context persistence across sessions.
AgentCo-op: Retrieval-Based Synthesis of Interoperable Multi-Agent Workflows
AgentCo-op is a retrieval-based synthesis framework for composing interoperable multi-agent workflows from reusable skills, tools, and external agents. It uses typed artifact handoffs and bounded self-guided local repair, achieving strong results on benchmarks and enabling collaborative discovery in open-world genomics tasks.