Recovering Policy-Induced Errors: Benchmarking and Trajectory Synthesis for Robust GUI Agents

Hugging Face Daily Papers Papers
gui-agents error-recovery benchmarking trajectory-synthesis robustness fine-tuning os-world

Summary

Introduces GUI-RobustEval, a benchmark for error recovery in GUI agents, and Robustness-driven Trajectory Synthesis (RoTS) to generate training data, achieving state-of-the-art on OSWorld.

While GUI agents have advanced rapidly, they often lack the robustness to recover from their own errors, hindering real-world deployment. To bridge this gap at both the evaluation and data levels, we introduce GUI-RobustEval and propose Robustness-driven Trajectory Synthesis. GUI-RobustEval contains 1,216 executable test cases that systematically measure error recovery capabilities across a broad and realistic spectrum of error modes. At the data level, RoTS is a scalable synthesis framework that creates 800k high-quality data via a tree-based pipeline that proactively discovers diverse error modes and synthesizes corresponding recovery steps. Our two models, RoTS-7B and RoTS-32B, fine-tuned on our dataset, both demonstrate significant gains on GUI-RobustEval and traditional GUI benchmarks. Notably, RoTS-32B achieves state-of-the-art performance on OSWorld, with a 47.4% success rate and a 33.8% All-Pass@4 score, suggesting that improved long-horizon error recovery ability contributes to both robustness and overall performance. Our code is available at https://github.com/AlibabaResearch/RoTS.
Original Article
View Cached Full Text

Cached at: 06/01/26, 07:18 AM

Paper page - Recovering Policy-Induced Errors: Benchmarking and Trajectory Synthesis for Robust GUI Agents

Source: https://huggingface.co/papers/2605.29447

Abstract

GUI agents lack robust error recovery capabilities, which this work addresses through GUI-RobustEval and Robustness-driven Trajectory Synthesis, demonstrating improved performance on real-world benchmarks.

WhileGUI agentshave advanced rapidly, they often lack the robustness to recover from their own errors, hindering real-world deployment. To bridge this gap at both the evaluation and data levels, we introduceGUI-RobustEvaland proposeRobustness-driven Trajectory Synthesis.GUI-RobustEvalcontains 1,216 executable test cases that systematically measureerror recoverycapabilities across a broad and realistic spectrum of error modes. At the data level, RoTS is a scalable synthesis framework that creates 800k high-quality data via a tree-based pipeline that proactively discovers diverse error modes and synthesizes corresponding recovery steps. Our two models, RoTS-7B and RoTS-32B, fine-tuned on our dataset, both demonstrate significant gains onGUI-RobustEvaland traditional GUI benchmarks. Notably, RoTS-32B achieves state-of-the-art performance onOSWorld, with a 47.4% success rate and a 33.8%All-Pass@4score, suggesting that improved long-horizonerror recoveryability contributes to both robustness and overall performance. Our code is available at https://github.com/AlibabaResearch/RoTS.

View arXiv pageView PDFGitHub4Add to collection

Get this paper in your agent:

hf papers read 2605\.29447

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.29447 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2605.29447 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.29447 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

ReGRPO: Reflection-Augmented Policy Optimization for Tool-Using Agents

arXiv cs.AI

ReGRPO introduces a reflection-augmented policy optimization framework for tool-using vision-language agents, leveraging structured failure observations and joint optimization of reflection tokens and actions to improve recovery from tool failures, achieving state-of-the-art results on GTA and GAIA benchmarks.

Skill-Guided Continuation Distillation for GUI Agents

arXiv cs.AI

The paper proposes Skill-Guided Continuation Distillation (SGCD), an iterative self-improvement framework that uses skill-guided policies to generate supervision for off-trajectory states during closed-loop execution, improving GUI agent success rates on OSWorld-Verified from around 30% to over 50%.