UndoBench: Separating Task Competence from Recovery Capability in Tool-Using AI Agents

Hugging Face Daily Papers Papers
ai-agents benchmark tool-use fault-recovery enterprise idempotency evaluation

Summary

UndoBench introduces a benchmark that decouples task competence from fault-recovery capability in tool-using AI agents across 36 workflows and 36 fault scenarios in 8 enterprise domains, finding nominal task completion of 83.54% masks conditional recovery success of just 46.72%, with naive retry producing duplicate side effects in 53.33% of trials.

Tool-using AI agents are increasingly deployed across enterprise software systems, yet widely used benchmarks primarily evaluate nominal task completion, conflating baseline planning competence with operational fault recovery. We introduce UndoBench, a benchmark spanning 36 base workflows and 36 fault scenarios across 8 enterprise domains, decoupling task competence from recovery capability via counterfactual paired trials under identical seeds alongside wire-level effect-history and environment-state oracles. On 12 held-out TEST workflows across two open-weight models, two frameworks, and three recovery paradigms (5,760 executions / 2,880 paired trials) in the frozen lost-acknowledgment study, nominal competence reached 83.54% while conditional recovery success rate (CRSR) fell to 46.72%, with naive retry producing duplicate external effects in 53.33% of trials. Extensions to commercial API models reproduced this competence-recovery separation. Evaluations across complementary execution boundaries show that recovery is phase-dependent: before mutation, methods perform similarly without duplicate effects among capable trials; during partial mutation, naive retry, per-call idempotency, and zero-privilege journaling collapse on the evaluated composite workflows; after commit but before acknowledgment, verification and server-side idempotency substantially improve safety. These findings demonstrate that evaluating nominal completion alone masks critical, phase-dependent recovery vulnerabilities in autonomous agents.
Original Article
View Cached Full Text

Cached at: 10/06/26, 12:54 PM

Paper page - UndoBench: Separating Task Competence from Recovery Capability in Tool-Using AI Agents

Source: https://huggingface.co/papers/2610.05622

Abstract

Tool-usingAIagentsareincreasinglydeployedacrossenterprisesoftwaresystems,yetwidelyusedbenchmarksprimarilyevaluatenominaltaskcompletion,conflatingbaselineplanningcompetencewithoperationalfaultrecovery.WeintroduceUndoBench,abenchmarkspanning36baseworkflowsand36faultscenariosacross8enterprisedomains,decouplingtaskcompetencefromrecoverycapabilityviacounterfactualpairedtrialsunderidenticalseedsalongsidewire-leveleffect-historyandenvironment-stateoracles.On12held-outTESTworkflowsacrosstwoopen-weightmodels,twoframeworks,andthreerecoveryparadigms(5,760executions/2,880pairedtrials)inthefrozenlost-acknowledgmentstudy,nominalcompetencereached83.54%whileconditionalrecoverysuccessrate(CRSR)fellto46.72%,withnaiveretryproducingduplicateexternaleffectsin53.33%oftrials.ExtensionstocommercialAPImodelsreproducedthiscompetence-recoveryseparation.Evaluationsacrosscomplementaryexecutionboundariesshowthatrecoveryisphase-dependent:beforemutation,methodsperformsimilarlywithoutduplicateeffectsamongcapabletrials;duringpartialmutation,naiveretry,per-callidempotency,andzero-privilegejournalingcollapseontheevaluatedcompositeworkflows;aftercommitbutbeforeacknowledgment,verificationandserver-sideidempotencysubstantiallyimprovesafety.Thesefindingsdemonstratethatevaluatingnominalcompletionalonemaskscritical,phase-dependentrecoveryvulnerabilitiesinautonomousagents.

View arXiv pageView PDFGitHub1Add to collection

Get this paper in your agent:

hf papers read 2610\.05622

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.05622 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.05622 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.05622 in a Space README.md to link it from this page.

Collections including this paper1

Similar Articles

WorkBench Revisited: Workplace Agents Two Years On

arXiv cs.CL

This paper revisits the WorkBench benchmark for workplace agents two years after its initial release, showing that the best agent (Claude Opus 4.8) now completes 89% of tasks with only 2.5% harmful side effects, compared to GPT-4's 43% completion and 26% harm rate in 2024. It finds that capability and safety improve together, open-weight models have drastically lowered costs, and some basic mistakes persist.