Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents

Hugging Face Daily Papers Papers
benchmark llm-agents coding-agents harness-engineering evaluation long-horizon-tasks

Summary

Doc2Harness 提出了一个两阶段基准，直接评测 LLM 构建长周期编码代理 harness 的能力；在 19 个前沿模型中，GPT-6Astra 以 53.05% 的通过率领先，而表现最弱的模型仅为 27.44%。

Agents driven by large language models are increasingly deployed on long-horizon tasks, where they use tools and modify real workspaces. How well they perform depends as much on the harness around the model as on the model itself. Existing benchmarks measure downstream task success under a fixed harness, leaving harness engineering itself unexplored. We propose Doc2Harness, a two-stage benchmark that evaluates long-horizon harness engineering directly. In Stage 1, a build model operating inside a fixed builder harness receives an interface-only contract skeleton and a development document that consolidates behavior across twelve frontier harnesses, and then constructs a complete candidate harness in an isolated container. In Stage 2, the frozen candidate harness runs in containers separate from the construction environment, one per test case, and is driven end to end by a scripted mock model. We score each test case using four observable channels, namely the requests sent to the model, the files on disk, the process exit code, and the command output; positive controls rule out vacuous passes. Across nineteen frontier models, each evaluated with three independent builds on 328 test cases, GPT-6 Astra achieves the highest pass rate at 53.05%, while the weakest model reaches 27.44%, a gap of roughly twenty-six percentage points. Even the strongest model fails nearly half of the test cases, and failures cluster around how cross-cutting functionality is split across modules. To our knowledge, Doc2Harness provides the first deterministic benchmark for measuring long-horizon harness engineering.
Original Article
View Cached Full Text

Cached at: 10/15/26, 07:04 AM

Paper page - Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents

Source: https://huggingface.co/papers/2610.20344 Authors:

,

,

,

,

,

,

,

,

,

Abstract

Agentsdrivenbylargelanguagemodelsareincreasinglydeployedonlong-horizontasks,wheretheyusetoolsandmodifyrealworkspaces.Howwelltheyperformdependsasmuchontheharnessaroundthemodelasonthemodelitself.Existingbenchmarksmeasuredownstreamtasksuccessunderafixedharness,leavingharnessengineeringitselfunexplored.WeproposeDoc2Harness,atwo-stagebenchmarkthatevaluateslong-horizonharnessengineeringdirectly.InStage1,abuildmodeloperatinginsideafixedbuilderharnessreceivesaninterface-onlycontractskeletonandadevelopmentdocumentthatconsolidatesbehavioracrosstwelvefrontierharnesses,andthenconstructsacompletecandidateharnessinanisolatedcontainer.InStage2,thefrozencandidateharnessrunsincontainersseparatefromtheconstructionenvironment,onepertestcase,andisdrivenendtoendbyascriptedmockmodel.Wescoreeachtestcaseusingfourobservablechannels,namelytherequestssenttothemodel,thefilesondisk,theprocessexitcode,andthecommandoutput;positivecontrolsruleoutvacuouspasses.Acrossnineteenfrontiermodels,eachevaluatedwiththreeindependentbuildson328testcases,GPT-6Astraachievesthehighestpassrateat53.05%,whiletheweakestmodelreaches27.44%,agapofroughlytwenty-sixpercentagepoints.Eventhestrongestmodelfailsnearlyhalfofthetestcases,andfailuresclusteraroundhowcross-cuttingfunctionalityissplitacrossmodules.Toourknowledge,Doc2Harnessprovidesthefirstdeterministicbenchmarkformeasuringlong-horizonharnessengineering.

View arXiv pageView PDFProject pageGitHub32Add to collection

Get this paper in your agent:

hf papers read 2610\.20344

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.20344 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.20344 in a dataset README.md to link it from this page.

Spaces citing this paper1

Collections including this paper1

Similar Articles

HarnessOpt-Bench: Evaluating LLMs at Harness Optimization

Hugging Face Daily Papers

HarnessOpt-Bench is a benchmark for evaluating LLMs' ability to optimize the harness—the prompts, tools, control flow, memory, and orchestration code—around a target agent, using a fixed evaluation budget. Experiments with five frontier LLMs show that optimizer models separate more than the coding harnesses they act through, with substantial room for improvement.

Stop Comparing LLM Agents Without Disclosing the Harness

arXiv cs.AI

This position paper argues that in long-horizon LLM agent tasks, the execution harness often determines performance more than the model itself, and current benchmarks misattribute harness-level gains to model improvements. It proposes a harness-aware evaluation framework with disclosure standards and variance decomposition protocols.