Paper page - SimuWoB: Simulating Real-World Mobile Apps for Fast and Faithful GUI Agent Benchmarking

Source: https://huggingface.co/papers/2605.25160

Abstract

A synthetic benchmark for mobile GUI agents with 120 challenging tasks is introduced, featuring high-fidelity virtual environments with automatic reward generation and revealing significant limitations in current agent performance on complex, long-horizon interactions.

Mobile GUI agentspowered bylarge language modelshave progressed rapidly, creating urgent needs for realistic and comprehensive evaluation. Existing benchmarks prioritize reproducibility but are often limited to open-source apps or file-operation tasks for the difficulty of constructing rewards on real applications, leaving a gap between benchmark settings and real-world usage. Moreover, most benchmarks focus on basic grounding and navigation, with limited coverage of complex,long-horizon interactions. To address these limitations, we introduce SimuWoB, a fullysynthetic benchmarkformobile GUI agentswith 120 challenging tasks spanning diverse types and difficulty levels. We build a robustvirtual environment generationframework that synthesizeshigh-fidelity tasksand environments, and automatically provides valid rewards for each task. Each environment is deployed as a backend-free webpage accessible via URL, enabling efficient and reproducible evaluation. We conductcomprehensive experimentson several state-of-the-artmobile GUI agents. The average success rate is only 27.92%, dropping to 17.82% on long-horizon tasks, which reveals substantial weaknesses in current agents under complex scenarios. Evaluation result comparison with real-world sample tasks demonstrate that agent assessments based on our synthetic environment generalize well. We further providediagnostic insightsacross key capability dimensions and discuss implications for future mobile GUI agent development.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2605\.25160

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.25160 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2605.25160 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.25160 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.