DocOps: A Verifiable Benchmark for Autonomous Agents in Complex Document Operations

Hugging Face Daily Papers Papers
benchmark autonomous-agents document-operations evaluation llm-agents verifiable complex-tasks

Summary

This paper introduces DocOps, a deterministically verifiable benchmark for evaluating autonomous agents on complex document operations, revealing key failure modes such as long-term state tracking collapse, shallow semantic verification, and destructive editing of structural metadata.

As autonomous agents rapidly evolve, their ability to reliably manipulate ubiquitous digital documents has become critical for enabling general-purpose AI assistants and automating complex workspace workflows. In this paper, we introduce DocOps, a deterministically verifiable evaluation framework underpinned by a hierarchical taxonomy that deconstructs document operations inspired by real-world practices into atomic dimensions and escalating workflow complexities. Based on DocOps, we systematically evaluate representative closed- and open-source models across various agentic harnesses, revealing that even the most advanced frontier configurations still exhibit profound limitations when handling highly coupled, long-range tasks. Furthermore, a fine-grained analysis of existing agents' manipulation behaviors uncovers 3 key failure modes: long-term state tracking collapse, shallow semantic verification, and destructive editing of structural metadata. Ultimately, our work exposes the capability boundaries of agents in maintaining global document consistency, shedding light on the future design of robust, non-destructive agents for complex digital ecosystems.
Original Article
View Cached Full Text

Cached at: 07/24/26, 05:07 AM

Paper page - DocOps: A Verifiable Benchmark for Autonomous Agents in Complex Document Operations

Source: https://huggingface.co/papers/2607.19865

Abstract

Asautonomousagentsrapidlyevolve,theirabilitytoreliablymanipulateubiquitousdigitaldocumentshasbecomecriticalforenablinggeneral-purposeAIassistantsandautomatingcomplexworkspaceworkflows.Inthispaper,weintroduceDocOps,adeterministicallyverifiableevaluationframeworkunderpinnedbyahierarchicaltaxonomythatdeconstructsdocumentoperationsinspiredbyreal-worldpracticesintoatomicdimensionsandescalatingworkflowcomplexities.BasedonDocOps,wesystematicallyevaluaterepresentativeclosed-andopen-sourcemodelsacrossvariousagenticharnesses,revealingthateventhemostadvancedfrontierconfigurationsstillexhibitprofoundlimitationswhenhandlinghighlycoupled,long-rangetasks.Furthermore,afine-grainedanalysisofexistingagents’manipulationbehaviorsuncovers3keyfailuremodes:long-termstatetrackingcollapse,shallowsemanticverification,anddestructiveeditingofstructuralmetadata.Ultimately,ourworkexposesthecapabilityboundariesofagentsinmaintainingglobaldocumentconsistency,sheddinglightonthefuturedesignofrobust,non-destructiveagentsforcomplexdigitalecosystems.

View arXiv pageView PDFProject pageGitHub2Add to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.19865 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.19865 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.19865 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles