OSWorld-Pro: Process-based Evaluation for Computer Use Agents

Hugging Face Daily Papers Papers
computer-use-agents benchmark evaluation llm-judge gui-agents osworld

Summary

OSWorld-Pro introduces a process-based benchmark with over 300 tasks and 2,800 subgoals, grounded in 67,000+ human annotations, to evaluate computer-use agents step-by-step rather than only by end-state outcomes. It reveals failure modes even in state-of-the-art models, with top performers like Claude Opus 5 scoring 75.7% versus 83.4% on OSWorld.

Evaluation of Computer-Use Agents (CUAs) is often limited to the final deliverables they create (at the end of hundreds of steps) and assessed with functional verifiers, as seen in OSWorld. However, such evaluation of end-state performance lacks transparency into how and why agents fail in various tasks, obfuscating critical insight for subsequent improvement. For instance, agents that err during keyboard inputs would require a different mitigation strategy from those that fail to precisely provide click-based inputs on the graphical UI. We introduce OSWorld-Pro: a set of over 300 tasks containing over 2800 subgoals to enable the procedural evaluation of CUAs grounded in over 67,000 human annotations. We use robust human-aligned LLM-Judges to evaluate the fulfillment of OSWorld-Pro subgoals and thereby reveal the progress that models make throughout a series of sequentially dependent subgoals. Our findings reveal that OSWorld-Pro is challenging even for state-of-the-art LLMs, with top performers like Claude Opus 5 achieving only 75.7% vs. 83.4% on OSWorld. Furthermore, we identify critical process-focused failure modes of various models (e.g. subgoal-irrelevant actions and click-based mistakes) to provide insights to improve performance and efficiency of CUAs.
Original Article
View Cached Full Text

Cached at: 10/06/26, 04:53 AM

Paper page - OSWorld-Pro: Process-based Evaluation for Computer Use Agents

Source: https://huggingface.co/papers/2609.24890 Authors:

,

,

,

,

,

,

,

,

,

,

Abstract

EvaluationofComputer-UseAgents(CUAs)isoftenlimitedtothefinaldeliverablestheycreate(attheendofhundredsofsteps)andassessedwithfunctionalverifiers,asseeninOSWorld.However,suchevaluationofend-stateperformancelackstransparencyintohowandwhyagentsfailinvarioustasks,obfuscatingcriticalinsightforsubsequentimprovement.Forinstance,agentsthaterrduringkeyboardinputswouldrequireadifferentmitigationstrategyfromthosethatfailtopreciselyprovideclick-basedinputsonthegraphicalUI.WeintroduceOSWorld-Pro:asetofover300taskscontainingover2800subgoalstoenabletheproceduralevaluationofCUAsgroundedinover67,000humanannotations.Weuserobusthuman-alignedLLM-JudgestoevaluatethefulfillmentofOSWorld-Prosubgoalsandtherebyrevealtheprogressthatmodelsmakethroughoutaseriesofsequentiallydependentsubgoals.OurfindingsrevealthatOSWorld-Proischallengingevenforstate-of-the-artLLMs,withtopperformerslikeClaudeOpus5achievingonly75.7%vs.83.4%onOSWorld.Furthermore,weidentifycriticalprocess-focusedfailuremodesofvariousmodels(e.g.subgoal-irrelevantactionsandclick-basedmistakes)toprovideinsightstoimproveperformanceandefficiencyofCUAs.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2609\.24890

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.24890 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.24890 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.24890 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

MyPCBench: A Benchmark for Personally Intelligent Computer-Use Agents

Hugging Face Daily Papers

MyPCBench evaluates computer-use agents as personal assistants in a simulated Linux desktop environment with real-world web applications, revealing that Claude Opus 4.6 achieves the highest task completion rate of 55.4% while struggling with multi-application tasks and long trajectories.

OpenComputer: Verifiable Software Worlds for Computer-Use Agents

Hugging Face Daily Papers

OpenComputer presents a framework for creating verifiable software environments for computer-use agents, integrating state verifiers, self-improving verification layers, task synthesis, and evaluation systems across 33 desktop applications. Experiments show its verifiers align better with human judgment than LLM-as-judge, and frontier agents struggle with end-to-end completion.

WorkBench Revisited: Workplace Agents Two Years On

arXiv cs.CL

This paper revisits the WorkBench benchmark for workplace agents two years after its initial release, showing that the best agent (Claude Opus 4.8) now completes 89% of tasks with only 2.5% harmful side effects, compared to GPT-4's 43% completion and 26% harm rate in 2024. It finds that capability and safety improve together, open-weight models have drastically lowered costs, and some basic mistakes persist.