Paper page - GUICrafter: Weakly-Supervised GUI Agent Leveraging Massive Unannotated Screenshots

Source: https://huggingface.co/papers/2606.29705

Abstract

GUICrafter addresses GUI agent data challenges through a weakly-supervised approach using unannotated screenshots and a two-stage curriculum learning framework for visual grounding and reinforcement learning calibration.

Data, as the fundamental substrate of modern intelligence, has greatly driven the development of current foundation models. Naturally, researchers aim to extend this paradigm to the domain ofGUI agents, hoping to build strongGUI agentsthrough a similar paradigm. However, GUI agent data cannot be directly harvested from the internet, making it costly and difficult to collect at scale. As a result, currentGUI agentssuffer from poor cross-device generalization and limitedvisual groundingability for fine-grained GUI elements. As an attempt to address data challenge inGUI agents, we propose GUICrafter, a weakly-supervised GUI agent leveraging massive unannotated screenshots to substantially reduce the reliance on expensive human annotations. GUICrafter explores acurriculum learningframework for trainingGUI agentsthrough two progressive stages. First, the model learnsvisual groundingfrom large-scale unannotated screenshots andwebpages, leveraging the rich contextual signals inherent inGUI interactions without human annotations. Then, in Stage 2, we leverage a small amount of high-quality data to calibrate the model viareinforcement learning. Experiments show that GUICrafter achieves competitive, or even superior, performance to advanced systems like UI-TARS while using only 0.1% of its data. Furthermore, under the same amount of annotated data, GUICrafter surpasses all previous methods such as GUI-R1. Code, data, and models are available at https://github.com/fansunqi/GUICrafter.

View arXiv pageView PDFGitHub0Add to collection

Get this paper in your agent:

hf papers read 2606\.29705

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.29705 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2606.29705 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.29705 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.