Evo-Bench: Can Language Models Improve Agent Harness?

Hugging Face Daily Papers Papers
benchmark agents llm-evaluation harness-evolution

Summary

Introduces Evo-Bench, the first benchmark for evaluating language models' intrinsic harness-evolving capabilities across Search, Office, and General agent domains, showing top models achieve large gains but struggle on Office workflows.

Large Language Models (LLMs) have driven rapid progress in autonomous agents, yet standard evaluations remain confined to static task solving. An emerging frontier is harness evolution---the agent's capacity to autonomously optimize its own operating harness. However, systematically benchmarking this capability remains challenging, as existing evaluations fail to isolate harness improvements from base model strength, prevent task-specific overfitting, or capture long-horizon iterative research. To address these challenges, we introduce Evo-Bench, the first benchmark designed to evaluate models' intrinsic harness-evolving capabilities across Search, Office, and General agent domains. To rigorously isolate this capability, Evo-Bench employs a novel harness-guided construction framework: it leverages auxiliary-task evolution to identify tasks genuinely sensitive to framework improvements, followed by sensitivity-aware stratified splitting to ensure robust cross-suite generalization. Extensive evaluations across nine frontier and open-weight models reveal that top models achieve massive absolute gains reaching 16.6 points, closely approaching state-of-the-art human-engineered baselines. Crucially, while autonomous evolution outpeforms artificial harness in General tasks and excels in Search tasks, it struggles in Office tasks that demand highly specific processing workflows. Furthermore, our analysis exposes critical temporal anomalies like early saturation, while demonstrating that the synthesized harnesses act as highly transferable reasoning structures, consistently boosting diverse policy models.
Original Article
View Cached Full Text

Cached at: 08/11/26, 06:20 AM

Paper page - Evo-Bench: Can Language Models Improve Agent Harness?

Source: https://huggingface.co/papers/2608.09096

Abstract

LargeLanguageModels(LLMs)havedrivenrapidprogressinautonomousagents,yetstandardevaluationsremainconfinedtostatictasksolving.Anemergingfrontierisharnessevolution---theagent’scapacitytoautonomouslyoptimizeitsownoperatingharness.However,systematicallybenchmarkingthiscapabilityremainschallenging,asexistingevaluationsfailtoisolateharnessimprovementsfrombasemodelstrength,preventtask-specificoverfitting,orcapturelong-horizoniterativeresearch.Toaddressthesechallenges,weintroduceEvo-Bench,thefirstbenchmarkdesignedtoevaluatemodels’intrinsicharness-evolvingcapabilitiesacrossSearch,Office,andGeneralagentdomains.Torigorouslyisolatethiscapability,Evo-Benchemploysanovelharness-guidedconstructionframework:itleveragesauxiliary-taskevolutiontoidentifytasksgenuinelysensitivetoframeworkimprovements,followedbysensitivity-awarestratifiedsplittingtoensurerobustcross-suitegeneralization.Extensiveevaluationsacrossninefrontierandopen-weightmodelsrevealthattopmodelsachievemassiveabsolutegainsreaching16.6points,closelyapproachingstate-of-the-arthuman-engineeredbaselines.Crucially,whileautonomousevolutionoutpeformsartificialharnessinGeneraltasksandexcelsinSearchtasks,itstrugglesinOfficetasksthatdemandhighlyspecificprocessingworkflows.Furthermore,ouranalysisexposescriticaltemporalanomalieslikeearlysaturation,whiledemonstratingthatthesynthesizedharnessesactashighlytransferablereasoningstructures,consistentlyboostingdiversepolicymodels.

View arXiv pageView PDFProject pageGitHub1Add to collection

Get this paper in your agent:

hf papers read 2608\.09096

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.09096 in a model README.md to link it from this page.

Datasets citing this paper1

#### RUC-AIBOX/Evo-Bench Viewer• Updatedabout 3 hours ago • 608 • 40

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.09096 in a Space README.md to link it from this page.

Collections including this paper1

Similar Articles

Rethinking the Evaluation of Harness Evolution for Agents

arXiv cs.AI

This paper re-evaluates the methodology of automatic harness evolution for LLM agents, highlighting that its gains may stem from additional test-time search rather than improved harness design, and that evaluation on the same benchmark risks overfitting. Experiments show that harness evolution does not consistently outperform simpler test-time scaling methods.