Tag
提出 ScreenRef 高质量 GUI grounding 数据集（586K 截图、633K 指代表达实例）和 Distillation-to-Reinforcement Learning（D2RL）训练方法，训练出 GroundGUI-8B，在 ScreenSpot-V2 等多个基准上取得 8B 级模型中的领先性能。
DeskForge introduces a controllable desktop environment that composes real applications to generate 1.2M densely annotated desktop observations (DeskForge-1M) for training computer-use agents. Fine-tuning four VLMs on this data yields significant gains on GUI grounding benchmarks (e.g., Qwen3.5-4B improves 11.51 points on ScreenSpot-Pro) and long-horizon task completion.
The paper introduces GUI-Primitives, a benchmark of 994 contrastive instruction pairs to diagnose spatial reasoning failures in vision-language models for GUI grounding, revealing that most failures stem from candidate localization rather than relation understanding.
This paper audits GUI grounding evaluations, revealing that high instruction-element embedding similarity often reflects visible-label recovery rather than semantic grounding, and advocates for reporting lexical baselines and diagnostics.
InnerZoom proposes a single-forward framework for cross-layer evidence bridging in GUI grounding, achieving state-of-the-art performance on multiple benchmarks while reducing latency by up to 31.8%.
Proposes quality-aware self-distillation for GUI grounding, improving coordinate-token teacher signals via correctness-aware gating and probability scaling to enhance vision-language model performance.
VISTA introduces a view-consistent self-verified training method for GUI grounding that improves GRPO-based coordinate prediction by using multiple target-preserving views, achieving consistent accuracy gains across benchmarks.
DRS-GUI proposes a training-free dynamic region search framework for GUI grounding, using a lightweight UI Perceptor with human-like perceptual actions and Monte Carlo Tree Search to progressively locate instruction-relevant elements. Experiments show a 14% improvement on ScreenSpot-Pro for both general and GUI-specific MLLMs.
Microsoft has released Phi-Ground-Any, a 4B parameter vision model for GUI grounding on Hugging Face that achieves state-of-the-art results, enabling AI agents to precisely interact with screen elements.