HarnessOpt-Bench: Evaluating LLMs at Harness Optimization

Hugging Face Daily Papers Papers
benchmark llm harness-optimization evaluation agentic-systems

Summary

HarnessOpt-Bench is a benchmark for evaluating LLMs' ability to optimize the harness—the prompts, tools, control flow, memory, and orchestration code—around a target agent, using a fixed evaluation budget. Experiments with five frontier LLMs show that optimizer models separate more than the coding harnesses they act through, with substantial room for improvement.

As LLMs are increasingly deployed within agentic systems, their capabilities depend not only on the model weights but also on the harness: the prompts, tools, control flow, memory, and orchestration code surrounding them. This makes automated harness optimization -- the iterative and evaluation-guided improvement of a harness by an AI system -- both an important route to improving AI systems and a demanding capability for AI systems themselves. Yet the community lacks a common protocol for measuring how well frontier LLMs perform at this task. We introduce HarnessOpt-Bench, a benchmark for end-to-end harness optimization under expensive and stochastic evaluation. An optimizer, an LLM paired with a coding harness, receives a target agent's seed harness, graded evaluation feedback, and a fixed target-evaluation budget. It edits the harness and nominates a final candidate, which is scored by its normalized gain over the seed on a held-out test partition that remains inaccessible throughout search. A trusted execution environment enforces the evaluation boundary, meters target-agent resource use, and preserves candidate versions for audit. We evaluate 5 frontier LLMs as optimizers both under a shared coding harness and under their native harnesses across 4 downstream tasks, over 111 scored runs. Experiment results show that optimizer models separate more than the coding harnesses they act through, native harnesses are not consistently superior, and gains vary substantially across tasks and seed regimes. These results establish harness optimization as a measurable and discriminative capability with large space for improvement.
Original Article
View Cached Full Text

Cached at: 08/07/26, 05:56 AM

Paper page - HarnessOpt-Bench: Evaluating LLMs at Harness Optimization

Source: https://huggingface.co/papers/2608.06301

Abstract

AsLLMsareincreasinglydeployedwithinagenticsystems,theircapabilitiesdependnotonlyonthemodelweightsbutalsoontheharness:theprompts,tools,controlflow,memory,andorchestrationcodesurroundingthem.Thismakesautomatedharnessoptimization--theiterativeandevaluation-guidedimprovementofaharnessbyanAIsystem--bothanimportantroutetoimprovingAIsystemsandademandingcapabilityforAIsystemsthemselves.YetthecommunitylacksacommonprotocolformeasuringhowwellfrontierLLMsperformatthistask.WeintroduceHarnessOpt-Bench,abenchmarkforend-to-endharnessoptimizationunderexpensiveandstochasticevaluation.Anoptimizer,anLLMpairedwithacodingharness,receivesatargetagent’sseedharness,gradedevaluationfeedback,andafixedtarget-evaluationbudget.Iteditstheharnessandnominatesafinalcandidate,whichisscoredbyitsnormalizedgainovertheseedonaheld-outtestpartitionthatremainsinaccessiblethroughoutsearch.Atrustedexecutionenvironmentenforcestheevaluationboundary,meterstarget-agentresourceuse,andpreservescandidateversionsforaudit.Weevaluate5frontierLLMsasoptimizersbothunderasharedcodingharnessandundertheirnativeharnessesacross4downstreamtasks,over111scoredruns.Experimentresultsshowthatoptimizermodelsseparatemorethanthecodingharnessestheyactthrough,nativeharnessesarenotconsistentlysuperior,andgainsvarysubstantiallyacrosstasksandseedregimes.Theseresultsestablishharnessoptimizationasameasurableanddiscriminativecapabilitywithlargespaceforimprovement.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2608\.06301

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.06301 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.06301 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.06301 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Stop Comparing LLM Agents Without Disclosing the Harness

arXiv cs.AI

This position paper argues that in long-horizon LLM agent tasks, the execution harness often determines performance more than the model itself, and current benchmarks misattribute harness-level gains to model improvements. It proposes a harness-aware evaluation framework with disclosure standards and variance decomposition protocols.