gui-agents

UI-Mate is a foundation GUI agent that uses environment-grounded training and in-context demonstrations to improve reliability on long-horizon office tasks, achieving state-of-the-art results on computer-use benchmarks.

CoAdapt-GUI is a test-time adaptation framework for mobile GUI agents that jointly adapts workflow context and policy, improving performance on unseen-app benchmarks like AndroidWorld-Generalization and AndroidWorld Plus.

Introduces AndroidReality, a perturbation-based framework for evaluating and improving the robustness of mobile agents, with a taxonomy of real-world interface perturbations and a training-free Test-Time Introspective Recovery (TTIR) mechanism.

This paper investigates when hybrid computer-use agents actually choose to use MCP tools versus screenshots, finding that tool availability alone does not guarantee adoption: a reasoning model improves while a non-reasoning model degrades. It also explores training and context compression strategies to close the adoption gap and reduce token costs.

FocusMem introduces a latent memory interface for GUI agents that separates content retention, state-conditioned readout, and a trust gate to improve memory reliability. It consistently outperforms fixed-memory baselines across five GUI-agent benchmarks.

This paper introduces Maga, a method for consolidating domain-specific GUI agents into a single cross-platform policy via structured action distillation, reallocating training signals to focus on erroneous actions. It achieves strong success rates across mobile, web, and desktop benchmarks.

Qwen-UI-Agent is a new foundation GUI agent from Alibaba's Qwen team that handles mobile, computer, web, and DeepSearch tasks with state-of-the-art performance on mobile-use benchmarks and competitive results on computer/browser tasks, combining GUI and CLI actions in a unified action space.

This paper introduces VG-GUIBench, a benchmark to evaluate MLLM-based GUI agents' ability to follow video tutorials, and proposes TASKER, a keyframe extraction method that improves performance on VideoQA and video-guided agentic tasks.

This paper proposes a reinforcement learning framework for computer-use agents that uses autonomous vision-language evaluation as a scalable reward signal, modeling evaluator noise to improve task success rates across desktop environments.

This paper from MIT and Harvard explores automating SKILL.md generation by mining GUI interaction trajectories, finding that clusters are readable but do not improve policy performance across domains.

VisualSkill proposes a hierarchical multimodal skill library for computer-use agents that combines text and figures, achieving a 15.3 point absolute lift on CUA benchmarks over text-only baselines by retaining visual information for GUI interaction.

The paper proposes Skill-Guided Continuation Distillation (SGCD), an iterative self-improvement framework that uses skill-guided policies to generate supervision for off-trajectory states during closed-loop execution, improving GUI agent success rates on OSWorld-Verified from around 30% to over 50%.

Introduces Teach VLM, a model that extracts step-by-step operational knowledge from mobile screen demonstrations, and the Teach-and-Repeat paradigm that uses this knowledge to guide GUI agents, achieving state-of-the-art performance on a new benchmark.

Introduces MacArena, a benchmark of 421 tasks across 50 applications for evaluating computer-use agents on macOS, highlighting that existing benchmarks may not capture macOS-specific challenges.

StainFlow introduces an entity-stain-flow process reward model for GUI agents, using global entity stain tracking and local evidence linking to improve credit assignment in reinforcement learning, achieving 3.2% relative improvement on AndroidWorld.

DeskCraft is a new benchmark for evaluating desktop GUI agents on long-horizon professional creative workflows, incorporating human-in-the-loop collaboration protocols. It tests agents on tasks requiring over 50 steps across design, video, audio, and 3D software.

Introduces GUI-RobustEval, a benchmark for error recovery in GUI agents, and Robustness-driven Trajectory Synthesis (RoTS) to generate training data, achieving state-of-the-art on OSWorld.

AQuaUI is a training-free inference-time token reduction method for GUI agent models that uses adaptive quadtrees to reduce spatial redundancy in screenshots, achieving up to 13.22% speedup and 29.52% fewer visual tokens while retaining 99.06% of performance.

MementoGUI introduces a plug-in agentic memory framework for GUI agents that uses learned controllers for selective memory management and retrieval, improving performance on long-horizon tasks with compressed visual and textual representations.

Proposes Video2GUI, a framework to automatically extract GUI interaction trajectories from unlabeled instructional videos, building WildGUI dataset with 12M trajectories across 1500+ apps. Pre-training on this data yields 5-20% improvements on GUI grounding and action benchmarks.