On the Reliability of Computer Use Agents

Hugging Face Daily Papers Papers
ai-reliability computer-use-agents osworld agent-evaluation

Summary

A preprint analyzing why computer-use agents succeed once but fail on repeated executions, attributing unreliability to execution stochasticity, task ambiguity, and behavioral variability, and advocating repeated evaluation and stable strategies.

Computer-use agents have rapidly improved on real-world tasks such as web navigation, desktop automation, and software interaction, in some cases surpassing human performance. Yet even when the task and model are unchanged, an agent that succeeds once may fail on a repeated execution of the same task. This raises a fundamental question: if an agent can succeed at a task once, what prevents it from doing so reliably? In this work, we study the sources of unreliability in computer-use agents through three factors: stochasticity during execution, ambiguity in task specification, and variability in agent behavior. We analyze these factors on OSWorld using repeated executions of the same task together with paired statistical tests that capture task-level changes across settings. Our analysis shows that reliability depends on both how tasks are specified and how agent behavior varies across executions. These findings suggest the need to evaluate agents under repeated execution, to allow agents to resolve task ambiguity through interaction, and to favor strategies that remain stable across runs.
Original Article
View Cached Full Text

Cached at: 04/21/26, 07:46 PM

Paper page - On the Reliability of Computer Use Agents

Source: https://huggingface.co/papers/2604.17849

Abstract

Computer-use agents exhibit unreliable performance due to execution stochasticity, task specification ambiguity, and behavioral variability, necessitating repeated evaluation and stable strategies for consistent task completion.

Computer-useagentshaverapidlyimprovedonreal-worldtaskssuchaswebnavigation,desktopautomation,andsoftwareinteraction,insomecasessurpassinghumanperformance.Yetevenwhenthetaskandmodelareunchanged,anagentthatsucceedsoncemayfailonarepeatedexecutionofthesametask.Thisraisesafundamentalquestion:ifanagentcansucceedatataskonce,whatpreventsitfromdoingsoreliably?Inthiswork,westudythesourcesofunreliabilityincomputer-useagentsthroughthreefactors:stochasticityduringexecution,ambiguityintaskspecification,andvariabilityinagentbehavior.WeanalyzethesefactorsonOSWorldusingrepeatedexecutionsofthesametasktogetherwithpairedstatisticalteststhatcapturetask-levelchangesacrosssettings.Ouranalysisshowsthatreliabilitydependsonbothhowtasksarespecifiedandhowagentbehaviorvariesacrossexecutions.Thesefindingssuggesttheneedtoevaluateagentsunderrepeatedexecution,toallowagentstoresolvetaskambiguitythroughinteraction,andtofavorstrategiesthatremainstableacrossruns.

View arXiv pageView PDFGitHub0Add to collection

Get this paper in your agent:

hf papers read 2604\.17849

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2604.17849 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2604.17849 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2604.17849 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

I think a lot of people are underestimating how expensive unreliable agents are

Reddit r/AI_Agents

The author argues that the hidden cost of unreliable AI agents lies in the cognitive overhead of constant human monitoring, emphasizing that predictability and environmental stability matter more than raw intelligence for real-world deployment. Practical workflows improve significantly when agents operate within controlled, validated environments rather than unpredictable ones.