GroundGUI: GUI Grounding by High-Quality Data and Distillation-to-Reinforcement Learning

Hugging Face Daily Papers Papers
gui-grounding multimodal reinforcement-learning dataset model-distillation vision-language agents

Summary

提出 ScreenRef 高质量 GUI grounding 数据集（586K 截图、633K 指代表达实例）和 Distillation-to-Reinforcement Learning（D2RL）训练方法，训练出 GroundGUI-8B，在 ScreenSpot-V2 等多个基准上取得 8B 级模型中的领先性能。

Constructing robust GUI agents relies on multimodal visual grounding: the ability to precisely map natural language intents to actionable on-screen components. Despite abundant desktop, web, and mobile benchmarks, high-fidelity resources tailored for diverse environments are surprisingly scarce. To bridge this gap, we introduce the Screen Referring Grounding Dataset (ScreenRef), a comprehensive GUI grounding dataset consisting of GUI screenshots paired with natural language referring expressions and box or point annotations. Spanning desktop, web, and mobile domains, ScreenRef comprises 586K screenshots and over 633K referring expression instances, covering tasks such as text recognition, visual matching, spatial understanding, and refusal scenarios. To boost the model's grounding performance, we propose the Distillation-to-Reinforcement Learning (D2RL) method to address the limited generalization of supervised fine-tuning and the sparse reward problem in reinforcement learning. Utilizing ScreenRef, we train Qwen3-VL-8B-Instruct via sequential SFT and D2RL training, yielding GroundGUI-8B. GroundGUI-8B achieves 95.2% on ScreenSpot-V2, 63.9% on ScreenSpot-Pro, 65.4% on OSWorld-G, 86.8% on MMBench-GUI L2, and 38.3% on UI-Vision. Among 8B-scale models, it produces substantial gains over models trained on open-source datasets, with performance comparable or even superior to models trained on closed-source data. Code, models, and datasets are available at https://github.com/PrentisAI/GroundGUI.
Original Article
View Cached Full Text

Cached at: 10/15/26, 02:55 AM

Paper page - GroundGUI: GUI Grounding by High-Quality Data and Distillation-to-Reinforcement Learning

Source: https://huggingface.co/papers/2610.19100

Abstract

ConstructingrobustGUIagentsreliesonmultimodalvisualgrounding:theabilitytopreciselymapnaturallanguageintentstoactionableon-screencomponents.Despiteabundantdesktop,web,andmobilebenchmarks,high-fidelityresourcestailoredfordiverseenvironmentsaresurprisinglyscarce.Tobridgethisgap,weintroducetheScreenReferringGroundingDataset(ScreenRef),acomprehensiveGUIgroundingdatasetconsistingofGUIscreenshotspairedwithnaturallanguagereferringexpressionsandboxorpointannotations.Spanningdesktop,web,andmobiledomains,ScreenRefcomprises586Kscreenshotsandover633Kreferringexpressioninstances,coveringtaskssuchastextrecognition,visualmatching,spatialunderstanding,andrefusalscenarios.Toboostthemodel’sgroundingperformance,weproposetheDistillation-to-ReinforcementLearning(D2RL)methodtoaddressthelimitedgeneralizationofsupervisedfine-tuningandthesparserewardprobleminreinforcementlearning.UtilizingScreenRef,wetrainQwen3-VL-8B-InstructviasequentialSFTandD2RLtraining,yieldingGroundGUI-8B.GroundGUI-8Bachieves95.2%onScreenSpot-V2,63.9%onScreenSpot-Pro,65.4%onOSWorld-G,86.8%onMMBench-GUIL2,and38.3%onUI-Vision.Among8B-scalemodels,itproducessubstantialgainsovermodelstrainedonopen-sourcedatasets,withperformancecomparableorevensuperiortomodelstrainedonclosed-sourcedata.Code,models,anddatasetsareavailableathttps://github.com/PrentisAI/GroundGUI.

View arXiv pageView PDFGitHub3Add to collection

Get this paper in your agent:

hf papers read 2610\.19100

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper1

#### PrentisAI/GroundGUI-8B Image-Text-to-Text• 9B• Updated35 minutes ago • 16

Datasets citing this paper3

#### PrentisAI/ScreenRef-Annotations Viewer• Updated38 minutes ago • 1.24M • 165 #### PrentisAI/ScreenRef Viewer• Updated38 minutes ago • 633k • 64 #### PrentisAI/GroundGUI_RL Viewer• Updated35 minutes ago • 12.6k • 47

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.19100 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

DRS-GUI: Dynamic Region Search for Training-Free GUI Grounding

arXiv cs.AI

DRS-GUI proposes a training-free dynamic region search framework for GUI grounding, using a lightweight UI Perceptor with human-like perceptual actions and Monte Carlo Tree Search to progressively locate instruction-relevant elements. Experiments show a 14% improvement on ScreenSpot-Pro for both general and GUI-specific MLLMs.

VISTA: View-Consistent Self-Verified Training for GUI Grounding

Hugging Face Daily Papers

VISTA introduces a view-consistent self-verified training method for GUI grounding that improves GRPO-based coordinate prediction by using multiple target-preserving views, achieving consistent accuracy gains across benchmarks.