EVOHARNESSBENCH: Can Your Agents Keep Pace with an Evolving Harness?
Summary
The paper introduces EVOHARNESSBENCH, a benchmark for evaluating LLM agents under evolving tool, skill, and agent harnesses, revealing gaps in retention and adaptation.
View Cached Full Text
Cached at: 09/10/26, 02:15 AM
Paper page - EVOHARNESSBENCH: Can Your Agents Keep Pace with an Evolving Harness?
Source: https://huggingface.co/papers/2609.04280 Authors:
,
,
,
,
,
,
,
,
,
,
Abstract
The study introduces a benchmark to evaluate LLM agents under evolving tool, skill, and agent harnesses, revealing persistent gaps in retention, adaptation, and harness-induced forgetting.
ModernLLM-based agentsoperate through a harness of tools, reusable skills, and specialist agents that shapes what they observe and what they can do. In practice, this harness continually evolves as new capabilities are added. We introduceEVOHARNESSBENCH, a benchmark for evaluating agents under controlledharness evolutionacross three axes (tools, skills, and agents). Unlike existing continual-learning benchmarks for agents, which typically placenon-stationarity(i.e., what changes over time) in the task stream while keeping the harness fixed,EVOHARNESSBENCHplacesnon-stationarityin the externally supplied harness itself. It contains 17 multi-stage harness streams constructed deterministically from verifier-based benchmarks, comprising 802 tasks, 520 tools, 42 skills, and 62 agents. We evaluate two complementary settings corresponding to the central challenges ofharness evolution:deployment evaluation, which isolates retention of previously accessible competence as the harness expands, andself-evolving adaptationevaluation, which tests whether accumulated experience remains useful as new capabilities are introduced. Our results reveal three persistent gaps. First, harness expansion alone can degrade performance on previously solved tasks, producingharness-induced forgetting. Second, gains fromself-evolving adaptationremain inconsistent across stages ofharness evolution, capability axes, and environments. Third, retention and adaptation can pull in different directions: preserving earlier competence does not necessarily improve adaptation to newly introduced capabilities, and vice versa. These results establishharness evolutionas a distinct challenge for building agents that can keep pace with an evolving harness while preserving previously effective behavior.
View arXiv pageView PDFProject pageAdd to collection
Get this paper in your agent:
hf papers read 2609\.04280
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2609.04280 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2609.04280 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2609.04280 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Rethinking the Evaluation of Harness Evolution for Agents
This paper re-evaluates the methodology of automatic harness evolution for LLM agents, highlighting that its gains may stem from additional test-time search rather than improved harness design, and that evaluation on the same benchmark risks overfitting. Experiments show that harness evolution does not consistently outperform simpler test-time scaling methods.
Evo-Bench: Can Language Models Improve Agent Harness?
Introduces Evo-Bench, the first benchmark for evaluating language models' intrinsic harness-evolving capabilities across Search, Office, and General agent domains, showing top models achieve large gains but struggle on Office workflows.
HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
HarnessDev evaluates LLMs by their ability to build and evolve execution harnesses, revealing significant variations in performance and poor transferability across models.
EvoSafeHarness: Evolving Model- and Domain-Specific Harnesses for Securing Agents
EvoSafeHarness optimizes deployable safety harnesses for LLM agents by jointly searching natural-language policies and executable logic, improving safety-utility trade-offs across agent benchmarks.
HarnessOpt-Bench: Evaluating LLMs at Harness Optimization
HarnessOpt-Bench is a benchmark for evaluating LLMs' ability to optimize the harness—the prompts, tools, control flow, memory, and orchestration code—around a target agent, using a fixed evaluation budget. Experiments with five frontier LLMs show that optimizer models separate more than the coding harnesses they act through, with substantial room for improvement.