Harness-Delta Attribution: Dissecting Gains From Harness Evolution

# Harness-Delta Attribution: Dissecting Gains From Harness Evolution
Source: [https://arxiv.org/html/2610.22466](https://arxiv.org/html/2610.22466)
Wenxuan Ding Qin Lu Changlong Yu Sha LiShuowei Jin Xin Liu Zhenyu Shi Greg DurrettAffiliation:Amazon New York University

###### Abstract

Harness evolution automatically optimizes the prompts, skills, tools, and orchestration code around a fixed\-weight model, offering a way to improve agentic systems without updating model weights\. The agent harness is iteratively optimized to achieve higher performance on the search set\. But how much of this gain reflects benchmark\-specific overfitting or additional inference\-time compute, and how much may generalize beyond the search set? In this work, we introduce Harness\-Delta Attribution \(HDA\), an evaluation protocol that dissects an evolution gain into artifact\-driven overfitting, gains explained by test\-time scaling, and the remainder, which gives an upper bound on residual generalizable improvement\. We apply HDA to 11 executor models and three harness\-evolution frameworks across math, coding, creativity, and agentic tasks\. In 21 of these settings that show statistically significant search\-set gains, we find that a large share of the improvement is explained by overfitting or test\-time scaling\. Specifically, among these 21 settings, overfitting and test\-time scaling account for 54% and 14% of the gain on average; their combined share exceeds 50% in 14 settings\. The residual and held\-out transfer vary substantially across tasks, models, and frameworks\. Importantly, search\-set gains often fail to transfer to held\-out tasks, and provide limited evidence that they will persist under practical distribution shifts\. Moreover, validation\-based signals are insufficient to prevent overfitting\. These results motivate attribution\-aware evaluation in future harness evolution research\.111

Figure 1:An evolved harness may improve the performance by overfitting to benchmark artifacts, scaling test\-time compute, or making generalizable improvements \(left\)\. HDA asks where the observed gain comes from and attributes it to these three sources \(right\)\. Overfitting and test\-time scaling explain a substantial share across settings\.## 1Introduction

An agent system can be viewed as an LLM paired with aharness: the prompts, tools, memory, skills, orchestration code, and control flow that shape how a model interacts with a task\. Even without updating the model weights, changes to the harness can alter system performance by a wide margin, as observed on benchmarks such as Terminal\-Bench\([Merrill et al\., 2026](https://arxiv.org/html/2610.22466#bib.bib20)\)and SWE\-bench\([Jimenez et al\., 2024](https://arxiv.org/html/2610.22466#bib.bib11)\)\. This has motivated recent work onharness evolution, a form of text\-space optimization that iteratively updates the non\-parametric components around a fixed\-weight model using task feedback\([Lee et al\., 2026](https://arxiv.org/html/2610.22466#bib.bib1);[Lin et al\., 2026a](https://arxiv.org/html/2610.22466#bib.bib4);[Lin et al\., 2026b](https://arxiv.org/html/2610.22466#bib.bib7)\)\. Specifically, in each iteration, anexecutormodel attempts the tasks with the current harness\. Aproposeragent then inspects the trajectories and evaluation results to propose edits to the harness\. Repeating this process produces a sequence of harnesses optimized for performance on the search set \(training split\)\.

Recent harness\-evolution frameworks have reported notable gains from evolved harness code\([Lee et al\., 2026](https://arxiv.org/html/2610.22466#bib.bib1);[Ning et al\., 2026](https://arxiv.org/html/2610.22466#bib.bib22)\), skill files\([Ni et al\., 2026](https://arxiv.org/html/2610.22466#bib.bib21);[Yang et al\., 2026](https://arxiv.org/html/2610.22466#bib.bib3);[Tang et al\., 2026](https://arxiv.org/html/2610.22466#bib.bib14)\), and updated tool\-use or orchestration patterns\([Chen et al\., 2026](https://arxiv.org/html/2610.22466#bib.bib6)\)\.However, a higher aggregate score does not reveal what changed in the system or whether the gains will persist beyond the search set\.Prior work does not always evaluate evolved harnesses on held\-out instances\. Even when held\-out results are reported, the search and test splits may share the same task construction and dataset artifacts\. Such evaluations measure transfer within a benchmark, but provide limited evidence that the gain will persist in new tasks or practical distribution shifts\.

In this work, we introduce Harness\-Delta Attribution \(HDA\), an evaluation protocol that attributes harness evolution gains to three different sources\. Some gains may reflect real harness improvements that generalize beyond the search set\. However, we identify two additional sources of gain that lead us to view a harness optimization’s result in a different light\. First, an evolved harness may scale test\-time compute: for instance, it may sample five responses in parallel and select a majority vote\. Second, an evolved harness may exploit dataset shortcuts by hardcoding a recurring answer pattern from the search set\. These mechanisms can produce similar benchmark gains while having very different implications for cost and generalization\.

Our method, HDA, compares the baseline and evolved harnesses using a compute\-matched baseline and an artifact\-neutralized version of the evolved harness\. This allows us to estimate a lower bound on detected overfitting, isolate gains reproducible through additional inference compute, and measure the remaining residual\. We also evaluate held\-out performance to understand whether the evolved harness transfers beyond the search set\.

Our experiments cover three different frameworks, including Meta\-Harness\([Lee et al\., 2026](https://arxiv.org/html/2610.22466#bib.bib1)\), SkillOpt\([Yang et al\., 2026](https://arxiv.org/html/2610.22466#bib.bib3)\), and AHE\([Lin et al\., 2026a](https://arxiv.org/html/2610.22466#bib.bib4)\)\. We use them to evolve the harnesses of 11 executor models on benchmarks covering mathematical reasoning \(LiveMath\([Mahdavi et al\., 2025](https://arxiv.org/html/2610.22466#bib.bib8)\)\), software engineering \(SWE\-bench Verified\([Jimenez et al\., 2024](https://arxiv.org/html/2610.22466#bib.bib11)\)\), creative generation \(CREATE\([Wadhwa et al\., 2026](https://arxiv.org/html/2610.22466#bib.bib9)\)\), and embodied planning \(ALFWorld\([Shridhar et al\., 2021](https://arxiv.org/html/2610.22466#bib.bib10)\)\)\. In total, we apply HDA to 25 settings\.

Figure[1](https://arxiv.org/html/2610.22466#S0.F1)previews our main finding: across various configurations, overfitting and test\-time scaling explain a large share of the search\-set gains, while the residual upper bound on generalizable improvement is often small\. Among the 21 settings with statistically significant gains, HDA attributes at least54%54\\%of the gain to overfitting and14%14\\%to test\-time scaling on average; their combined share exceeds50%50\\%in 14 settings\. The breakdowns of these improvements can also reveal additional patterns\. First, different frameworks repeatedly discover the same benchmark\-specific shortcuts, such as a recurring correct answer choice in LiveMath\. Second, in ALFWorld, some evolved harness eliminate LLM calls entirely by hardcoding deterministic solutions on the synthetic benchmark\.

Our contributions are: \(1\) Identifying three sources of harness evolution gains \(overfitting, test\-time scaling, and generalizable improvement\); \(2\) The HDA method, which dissects the harness evolution gains into 3 categories, and reveals findings that held\-out evaluation alone does not show; \(3\) a reusable HDA skill and evaluation suite for applying this analysis to new benchmarks and harness\-evolution frameworks\.

## 2Preliminaries

LetMMdenote the fixed executor model whose harness is being optimized, andHHits harness; together,\(M,H\)\(M,H\)form the agent\.HHcomprises the non\-weight components that govern how the executor handles a task, such as its prompts, skills, tools, memory, and orchestration logic\([Weng, 2026](https://arxiv.org/html/2610.22466#bib.bib2)\)\. Different harness\-evolution methods optimize different subsets of these components while keepingMMfixed\. We useℋ\\mathcal\{H\}to denote the framework\-specific space of harnesses that may be explored\.

Given a task set𝒳\\mathcal\{X\}and task instancex∈𝒳x\\in\\mathcal\{X\}, the executor model produces an execution trajectory

τH,x∼M\(⋅∣x;H\)\.\\tau\_\{H,x\}\\sim M\(\\cdot\\mid x;H\)\.This trajectory includes the model outputs, tool calls, environment interactions, and other intermediate states produced during execution\. Letr⁡\(x,τH,x\)r\(x,\\tau\_\{H,x\}\)denote the task\-specific reward assigned to the trajectory\. We define the score of harnessHHon𝒳\\mathcal\{X\}as

S𝒳​\(H\)=1\|𝒳\|​∑x∈𝒳r⁡\(x,τH,x\)\.S\_\{\\mathcal\{X\}\}\(H\)=\\frac\{1\}\{\|\\mathcal\{X\}\|\}\\sum\_\{x\\in\\mathcal\{X\}\}r\(x,\\tau\_\{H,x\}\)\.
#### Evolution loop\.

Starting from a baseline harnessH0H\_\{0\}, a proposer agent𝒫\\mathcal\{P\}iteratively edits the harness to improve its performance, producing an evolved harnessH^\\hat\{H\}\. The proposer agent may be the executor agent itself, as in self\-evolution, or a separate and potentially stronger agent\. Let𝒳search\\mathcal\{X\}\_\{\\mathrm\{search\}\}denote the search set used during evolution\. The framework maintains a filesystem\-backed historyFtF\_\{t\}of harness versions, trajectories, evaluation results, and other artifacts through iterationtt, withF0F\_\{0\}containing the baseline evaluation\. At iterationtt, the proposer usesFt−1F\_\{t\-1\}to generate a setℋt\\mathcal\{H\}\_\{t\}ofkkcandidate harnesses:

ℋt:=\{Ht\(j\)\}j=1k=Propose⁡\(𝒫;Ft−1\)\.\\mathcal\{H\}\_\{t\}:=\\\{H\_\{t\}^\{\(j\)\}\\\}\_\{j=1\}^\{k\}=\\operatorname\{Propose\}\(\\mathcal\{P\};F\_\{t\-1\}\)\.Each candidate is evaluated on𝒳search\\mathcal\{X\}\_\{\\mathrm\{search\}\}, andft​\(H\)f\_\{t\}\(H\)denotes its feedback, which may include its score, per\-task rewards, trajectories, and execution logs\. The accumulated history is then updated as

Ft=Ft−1∪\{ft​\(H\):H∈ℋt\}\.F\_\{t\}=F\_\{t\-1\}\\cup\\\{f\_\{t\}\(H\):H\\in\\mathcal\{H\}\_\{t\}\\\}\.Some frameworks expose the persisted artifacts directly to the proposer, while others insert selected records into a structured prompt or optimizer state\. AcrossNNiterations, the evaluated harness pool is

ℋeval=\{H0\}∪⋃t=1Nℋt\.\\mathcal\{H\}\_\{\\mathrm\{eval\}\}=\\\{H\_\{0\}\\\}\\cup\\bigcup\_\{t=1\}^\{N\}\\mathcal\{H\}\_\{t\}\.The selected evolved harness is

H^=arg​maxH∈ℋeval⁡S𝒳search​\(H\)\.\\hat\{H\}=\\operatorname\*\{arg\\,max\}\_\{H\\in\\mathcal\{H\}\_\{\\mathrm\{eval\}\}\}S\_\{\\mathcal\{X\}\_\{\\mathrm\{search\}\}\}\(H\)\.An optional validation set𝒳val\\mathcal\{X\}\_\{\\mathrm\{val\}\}may be used to gate updates or select harnesses\. Algorithm[1](https://arxiv.org/html/2610.22466#alg1)summarizes this standard evolution loop\.

## 3Harness\-Delta Attribution: Dissecting Gains From Harness Evolution

Figure 2:Illustration of Harness\-Delta Attribution \(HDA\)\. ComparingS⁡\(H0\)\\smash\{S\(H\_\{0\}\)\}andS⁡\(H^\)\\smash\{S\(\\hat\{H\}\)\}gives the observed gainΔ​S\\Delta S\. RunningH0\\smash\{H\_\{0\}\}withH^\\smash\{\\hat\{H\}\}’s compute budget gives the compute\-matched baselineH0scale\\smash\{H\_\{0\}^\{\\mathrm\{scale\}\}\}; its gain overH0\\smash\{H\_\{0\}\}measuresTest\-Time Scaling\. Disabling detected shortcuts inH^\\smash\{\\hat\{H\}\}givesH^neutral\\smash\{\\hat\{H\}^\{\\mathrm\{neutral\}\}\}; the resulting drop measuresOverfitting\. The remaining gap estimatesGeneralizable Improvement\.Harness evolution returns an evolved harnessH^\\hat\{H\}with a higher search set score than the baselineH0H\_\{0\}, but the differenceΔ​S=S⁡\(H^\)−S⁡\(H0\)\\Delta S=S\(\\hat\{H\}\)\-S\(H\_\{0\}\)does not explain why performance improved\. For example, in one Meta\-Harness ALFWorld run , the harness for the 0\.8B executor grew from 66 to 503 non\-blank, non\-comment lines \(7\.6×7\.6\\times\) and reached100%100\\%search\-set success, but became a deterministic controller by hard\-coding benchmark priors and madenoLLM calls \(Figure[3](https://arxiv.org/html/2610.22466#S3.F3)\)\. In other settings, evolution increases inference compute, while scaling the baseline directly may produce similar gains without incurring the search cost of evolution\. Figure[1](https://arxiv.org/html/2610.22466#S0.F1)\(left\) illustrates these cases\.

We introduceHarness\-Delta Attribution \(HDA\)to distinguish these sources of gain\. The key idea is to compare the evolved harness with two controlled variants: a scaled baseline that identifies gains recoverable throughtest\-time scalingalone without harness evolution, and a neutralized evolved harness with detected artifact\-driven mechanisms removed to detectoverfitting\.

We implement HDA as a reusable skill that can be applied across frameworks and tasks\. An analysis agent follows it to perform the attribution, and a human reviewer checks the agent’s proposal at two task\-dependent checkpoints: the compute\-matching setup and the overfitting neutralization\. This design keeps most of the workflow automated and provides a task\-aware quality\-control layer, analogous to human evaluation of automated methods\. We provide the HDA skill in Appendix[C](https://arxiv.org/html/2610.22466#A3)\.

Before attribution, we apply a paired bootstrap significance test to the scores ofH0H\_\{0\}andH^\\hat\{H\}on the search set\. IfΔ​S\\Delta Sis insignificant, we report it as within noise and do not assign an attribution\. Otherwise, the HDA agent constructs a compute\-matched baselineH0scaleH\_\{0\}^\{\\mathrm\{scale\}\}and a neutralized evolved harnessH^neutral\\hat\{H\}^\{\\mathrm\{neutral\}\}, as described below\. HDA then defines three attribution components:

T\\displaystyle T=S⁡\(H0scale\)−S⁡\(H0\),\\displaystyle=S\(H\_\{0\}^\{\\mathrm\{scale\}\}\)\-S\(H\_\{0\}\),O\\displaystyle O=S⁡\(H^\)−S⁡\(H^neutral\),\\displaystyle=S\(\\hat\{H\}\)\-S\(\\hat\{H\}^\{\\mathrm\{neutral\}\}\),G\\displaystyle G=S⁡\(H^neutral\)−S⁡\(H0scale\)\.\\displaystyle=S\(\\hat\{H\}^\{\\mathrm\{neutral\}\}\)\-S\(H\_\{0\}^\{\\mathrm\{scale\}\}\)\.
#### TTest\-Time Scaling: Compute matching\.

The compute\-matched baselineH0scaleH\_\{0\}^\{\\mathrm\{scale\}\}preservesH0H\_\{0\}while scaling it to match the average inference budget ofH^\\hat\{H\}\. Specifically, the HDA agent compares their inference costs using measures such as LLM calls or token usage\. It then proposes a task\-appropriate compute measure and procedure for constructingH0scaleH\_\{0\}^\{\\mathrm\{scale\}\}for human approval\.

For example, ifH^\\hat\{H\}uses ten times as many executor calls per task asH0H\_\{0\},H0scaleH\_\{0\}^\{\\mathrm\{scale\}\}may runH0H\_\{0\}ten times on each task and combine its outputs using majority vote, pass@10, or another task\-specific aggregation rule\. IfH^\\hat\{H\}does not use more inference compute thanH0H\_\{0\}, we setH0scale=H0H\_\{0\}^\{\\mathrm\{scale\}\}=H\_\{0\}\.

The differenceT=S⁡\(H0scale\)−S⁡\(H0\)T=S\(H\_\{0\}^\{\\mathrm\{scale\}\}\)\-S\(H\_\{0\}\)measures how much of the observed improvement can be reproduced solely through test\-time scaling rather than improved harness design\. Directly scalingH0H\_\{0\}also avoids the search cost of harness evolution\. Although such gains may transfer to held\-out tasks, their practical value depends on whether the additional latency and inference cost are acceptable for the target application\.

#### OOverfitting: Detection and neutralization\.

The HDA agent constructs the neutralized harnessH^neutral\\hat\{H\}^\{\\mathrm\{neutral\}\}by removing detected overfitting mechanisms fromH^\\hat\{H\}\. We classify an evolved mechanism asoverfittingwhen it improves performance by exploiting a pattern specific to the benchmark or search set, rather than improving the agent’s ability to achieve the intended task objective more generally\. Its benefit may disappear on new instances that preserve the task objective but not that pattern\.

Guided by the HDA skill, the agent audits the harness code, prompts, and available execution evidence for two broad forms of overfitting:

- •Direct shortcuts:mechanisms that bypass or predetermine the executor’s behavior, including eliminating LLM calls, constant answers, item\-specific gates, hard\-coded lookup tables, and string\-matching rules\.
- •Benchmark\-derived heuristics:mechanisms that exploit patterns specific to how the benchmark instances are constructed, rather than properties required to solve the underlying task\. Examples include thresholds fitted to benchmark statistics, reverse\-engineered construction rules, dataset\-specific priors, and hard\-coded solution recipes\.

For each detected mechanism, the HDA agent makes the smallest intervention that disables the shortcut to getH^neutral\\hat\{H\}^\{\\mathrm\{neutral\}\}, while preserving all unrelated evolved components\.

Figure 3:Example of overfitting in harness evolution\. As illustrated in this excerpt, the ALFWorld harness evolved by Meta\-Harness for Qwen3\.5\-0\.8B overfits to benchmark\-specific patterns by hard\-coding a deterministic controller that makes0LLM calls\.To support a reliable audit, the HDA skill provides shared definitions, examples and evidence requirements\. Prior work also shows that frontier agents can identify benchmark defects and reward\-hacking opportunities\([Tu et al\., 2026](https://arxiv.org/html/2610.22466#bib.bib24);[Wang et al\., 2026a](https://arxiv.org/html/2610.22466#bib.bib23)\)\. Human review also adds a safeguard tailored to the task and intended use\.

Note that HDA neutralizes only detected mechanisms that can be disabled without affecting unrelated parts of the harness\. We therefore interpretO=S⁡\(H^\)−S⁡\(H^neutral\)O=S\(\\hat\{H\}\)\-S\(\\hat\{H\}^\{\\mathrm\{neutral\}\}\)as alower\-boundestimate of overfitting, since implicit or entangled mechanisms may remain inH^neutral\\hat\{H\}^\{\\mathrm\{neutral\}\}\. Despite this, as we show later in results, the share of overfitting is still substantial\.

#### GGeneralizable Improvements: Residual upper bound\.

The residualGGis the gain that remains after accounting for detected overfitting and test\-time scaling\. By construction,G=Δ​S−O−T=S⁡\(H^neutral\)−S⁡\(H0scale\)G=\\Delta S\-O\-T=S\(\\hat\{H\}^\{\\mathrm\{neutral\}\}\)\-S\(H\_\{0\}^\{\\mathrm\{scale\}\}\)\. BecauseH^neutral\\hat\{H\}^\{\\mathrm\{neutral\}\}may still contain unneutralized overfitting, we interpretGGas an upper bound on generalizable improvement rather than evidence of transfer\. We additionally assess generalizability directly by comparingH0H\_\{0\}andH^\\hat\{H\}on held\-out tasks in Section[5](https://arxiv.org/html/2610.22466#S5.SS0.SSS0.Px3)\.

For reporting, we normalize each component byΔ​S\\Delta S, cap theOOshare to\[0%,100%\]\[0\\%,100\\%\], cap theTTshare to the remaining percentage, and assign the remainder toGG\. These caps affect only the reported percentages; the raw signed components still satisfyΔ​S=O\+T\+G\\Delta S=O\+T\+G\.

## 4Method and Experiments

#### Benchmarks

We evaluate across four benchmarks spanning single\-turn reasoning and long\-horizon agentic tasks and various domains: mathematical reasoning \(LiveMath\([Mahdavi et al\., 2025](https://arxiv.org/html/2610.22466#bib.bib8)\)\), creative generation \(CREATE\([Wadhwa et al\., 2026](https://arxiv.org/html/2610.22466#bib.bib9)\)\), interactive embodied planning \(ALFWorld\([Shridhar et al\., 2021](https://arxiv.org/html/2610.22466#bib.bib10)\)\), and software coding \(SWE\-bench Verified\([Jimenez et al\., 2024](https://arxiv.org/html/2610.22466#bib.bib11)\)\)\.

#### Evolution Frameworks

We evaluate three harness evolution frameworks with different optimization scopes and proposer–executor configurations, including self\-evolution and using a stronger agent as proposer:

- •Meta\-Harness\([Lee et al\., 2026](https://arxiv.org/html/2610.22466#bib.bib1)\)searches harness code around fixed LLMs\. The proposer can access the source code, scores, and execution traces of prior history via a filesystem\.
- •SkillOpt\([Yang et al\., 2026](https://arxiv.org/html/2610.22466#bib.bib3)\)optimizes the skill of a frozen agent, where a separate optimizer model turns scored rollouts into bounded edits on the skill document\.
- •AHE\([Lin et al\., 2026a](https://arxiv.org/html/2610.22466#bib.bib4)\)\(Agentic Harness Engineering\) evolves multiple harness components, including prompts, tools, skills, and memory, through an observability\-driven loop\. Following the original paper, we use its self\-evolution setting, with Opus 4\.8 as both proposer and executor\.

For Meta\-Harness, we additionally provide aggregate validation scores during evolution compared to the original implementation, while keeping per\-instance validation results and trajectories hidden\. We retain both the search\-selected and validation\-selected harnesses and evaluate them on the held\-out test set\. More details are in Appendix[B\.3](https://arxiv.org/html/2610.22466#A2.SS3)\.

#### Models

We evaluate 11 executor models from the Qwen and Claude model families\. The Qwen executors span Qwen3, Qwen3\.5, and Qwen3\.6, ranging from 0\.8B to 35B, and include both general\-purpose and coding\-specialized models\([Yang et al\., 2025](https://arxiv.org/html/2610.22466#bib.bib15);[Team, 2026](https://arxiv.org/html/2610.22466#bib.bib16);[Cao et al\., 2026](https://arxiv.org/html/2610.22466#bib.bib17);[Qwen Team, 2026](https://arxiv.org/html/2610.22466#bib.bib18)\)\. We also evaluate Claude Haiku 4\.5 and Claude Opus 4\.8\([Anthropic, 2025](https://arxiv.org/html/2610.22466#bib.bib45);[Anthropic, 2026b](https://arxiv.org/html/2610.22466#bib.bib46)\)on some settings\. Meta\-Harness and SkillOpt use Claude Opus 4\.8 as the proposer agent\([Anthropic, 2026a](https://arxiv.org/html/2610.22466#bib.bib19)\), whereas AHE uses Opus 4\.8 as both proposer and executor\. The exact executor set varies by benchmark; see more details in Table[1](https://arxiv.org/html/2610.22466#S5.T1)and Appendix[B](https://arxiv.org/html/2610.22466#A2)\.

## 5Results

#### Most gains come from overfitting or test\-time scaling\.

Table 1:HDA results for all 25 model–benchmark–framework settings\.Δ​S=S⁡\(H^\)−S⁡\(H0\)\\Delta S=S\(\\hat\{H\}\)\-S\(H\_\{0\}\)is the search\-set gain\.OO,TT, andGGare percentages ofΔ​S\\Delta S, capped to\[0%,100%\]\[0\\%,100\\%\]\.†\{\\color\[rgb\]\{0\.418,0\.4453,0\.5\}\\dagger\}marks search\-set gains that are not statistically significant, for which HDA reports no attribution\.⋆\{\\color\[rgb\]\{0\.707,0\.1367,0\.0938\}\\star\}marks settings where the evolved harness performs worse than baseline on the held\-out test set\. CU denotes Creativity Utility, the metric used for CREATE benchmark\.In many settings, a large share of the observed gain reflects overfitting or more test\-time compute rather than a more capable executor\. As shown in Table[1](https://arxiv.org/html/2610.22466#S5.T1),O\+TO\+Taccounts for more than half of the attributed gain in 14 of the 21 settings with significant search\-set gains\. Across these settings,OOandTTexplain54%54\\%and14%14\\%of the gain on average, respectively\. We clip each normalized percentage to the range\[0%,100%\]\[0\\%,100\\%\]\. The dominant source varies by benchmark: ALFWorld and LiveMath are dominated by overfitting, CREATE by test\-time scaling, and SWE\-bench shows clearer residual generalizable improvement\. Appendix[G](https://arxiv.org/html/2610.22466#A7)contrasts a benchmark\-fitted patch\-size heuristic with a reusable action\-feedback guardrail on SWE\-bench\.

For example, on ALFWorld, Meta\-Harness uses Opus 4\.8 as the proposer to optimize harnesses for five executors ranging from Qwen3\.5\-0\.8B to Qwen3\.6\-35B\-A3B separately\. All five reach100%100\\%search\-set success after evolution, while HDA attributes98%98\\%of their gain tooverfitting\. As shown in Figure[3](https://arxiv.org/html/2610.22466#S3.F3), smaller\-model \(0\.8B, 2B\) harnesses become rule\-based controllers that make no LLM calls, while larger\-model \(4B, 35B\-A3B\) harnesses encode similar benchmark\-specific knowledge through prompts, templates, and deterministic rules\.

CREATE provides a representative example oftest\-time scaling\. It evaluates associative creativity by asking models to produce a diverse set of high\-quality responses; its Creative Utility \(CU\) metric rewards larger answer pools containing strong and diverse answers\([Wadhwa et al\., 2026](https://arxiv.org/html/2610.22466#bib.bib9)\)\. As shown in Table[1](https://arxiv.org/html/2610.22466#S5.T1), under Meta\-Harness, a compute\-matched baseline recovers49%49\\%of the gain for Claude Haiku 4\.5 and even outperforms the evolved Qwen3\.6\-35B\-A3B harness\. Similarly, in AHE, test\-time scaling contributes\+5\.1\+5\.1CU, accounting for75%75\\%of observed evolution gain\. In contrast, SkillOpt only edits the skill file while keeping the control flow fixed, and we observeH0scale=H0H\_\{0\}^\{\\mathrm\{scale\}\}=H\_\{0\}andT=0T=0\. However, neither SkillOpt CREATE setting produces a statistically significant gain\. Overall, in frameworks where the orchestration code can be flexibly updated, the gains are more substantial but also more prone to test\-time scaling\. We show example harnesses in Appendix[F](https://arxiv.org/html/2610.22466#A6)\.

#### Different frameworks rediscover the same overfitting and scaling strategies\.

LiveMath is a multiple\-choice benchmark for research\-level mathematical reasoning\. In 21 of the 35 search\-set questions, one answer choice is the same meta\-level statement: “one of the remaining options is correct, but a stronger result can be proven\.” This option is correct in all 21 cases\.

Our HDA analysis shows that all three frameworks independently discover this shortcut\. Meta\-Harness implements a deterministic string\-matching branch to catch this option, SkillOpt turns the option into a default decision rule, and AHE adds an explicit instruction to select it without solving the problem\. The same shortcut appears across code, skill, and prompt evolution, rather than being specific to one framework\. We show the snippets of evolved harnesses in Appendix[E](https://arxiv.org/html/2610.22466#A5)\.

Similarly, on ALFWorld, Meta\-Harness and SkillOpt encode similar task recipes, appliance mappings, and object\-location priors, but represent them as executable controllers and natural\-language skills, respectively\. On CREATE, both Meta\-Harness and AHE resort to scaling strategy to increase the potential coverage and diversity of answers\. More details in Appendix[D](https://arxiv.org/html/2610.22466#A4)\.

Such shortcut\-driven gains make it difficult to assess the performance of harness evolution frameworks or make fair comparison across such frameworks\. This suggests the need for rich and low\-artifact benchmarks\.

#### Search\-set gains often fail to generalize, and even validation signals do not prevent overfitting\.

We test generalizability directly on held\-out test sets\. As shown in Figure[4](https://arxiv.org/html/2610.22466#S5.F4), held\-out gains are generally smaller than training gains\. Excluding ALFWorld, which we discuss below in §[5](https://arxiv.org/html/2610.22466#S5.SS0.SSS0.Px4), the held\-out gain is smaller than the search\-set gain in 13 of 15 settings and less than half the search\-set gain in 8\. In 4 settings, the evolved harness even performs worse than the baseline on test set\. These results confirm that large training gains often come from overfitting rather than generalizable improvement\.

Having observed substantial overfitting, we next askhow easily it can be reduced using a standard safeguard: held\-out validation signals\. We therefore augment Meta\-Harness with validation\-based selection\. Then we evaluate and compare train\-best harness \(H^\\hat\{H\}\) and val\-best harness \(H^val\\hat\{H\}^\{\\mathrm\{val\}\}\) on the held\-out test set\.

As shown in Figure[5](https://arxiv.org/html/2610.22466#S5.F5), validation selection increases mean held\-out performance by8\.08\.0percentage points on ALFWorld,0\.80\.8points on LiveMath, and8\.58\.5points on SWE\-bench Verified, while decreasing CREATE performance by0\.40\.4CU\. Across the 16 model–benchmark settings, it improves the held\-out score in 10 settings but reduces it in 6\. More importantly, validation selection does not close the transfer gap: in 14 of 16 settings, the validation\-selected harness’s held\-out gain remains below the search\-set gain of the search\-selected harness\. Thus, validation selection helps in some settings but does not reliably prevent overfitting\. Validation selection therefore improves transfer in some settings but it remains non\-trivial to further close the train\-to\-test gap\.

Figure 4:Search\-set gains do not reliably predict held\-out transfer\. For each setting, the open circle is the search\-set gain of the evolved harness \(Δ​Ss​e​a​r​c​h\\Delta S\_\{search\}\) and the filled circle its held\-out gain \(Δ​St​e​s​t\\Delta S\_\{test\}\); a red marker flags a negative held\-out gain\. The length of the line denotes the gap between gain on search set and on held\-out set\. Diamond meansH^=H0\\hat\{H\}=H\_\{0\};×\\timesmarks a non\-significant gain\. Units are pp, except CREATE \(CU\)\.Figure 5:Effect of validation selection across 16 Meta\-Harness settings\. Open circles show the search\-set gain of the search\-selected harnessH^\\hat\{H\}, gray circles show its held\-out test gain, and blue diamonds show the held\-out gain of the validation\-selected harnessH^val\\hat\{H\}^\{\\mathrm\{val\}\}\. Gains are reported in percentage points, except for CREATE, which uses Creative Utility \(CU\)\.
#### Why held\-out evaluation alone is not enough

Figure[4](https://arxiv.org/html/2610.22466#S5.F4)also illustrates why held\-out evaluation alone can miss overfitting\. ALFWorld initially appears to be a strong transfer case: its test set gains are large and close to its search\-set gains\. HDA reveals, however, that89%89\\%of the harness evolution gain comes from overfitting on average: the evolved harnesses may hardcode the deterministic solutions or recipes that exploit how the benchmark instances are constructed, and even eliminate the LLM calls in the most extreme cases \(Figure[3](https://arxiv.org/html/2610.22466#S3.F3)\)\. Although we deliberately withhold several task types and scenarios from the search set, the evolved harnesses still achieve near\-perfect performance on test set\.

This turns out to highlight another cause of overfitting:distillation of benchmark\-specific knowledge from a stronger proposer\. On this relatively well\-known benchmark, the strong proposer may already encode its information in the parametric knowledge, and the generated deterministic harnesses cover instances beyond search set\. These results thus provide limited evidence of generalizable improvement\. On tasks that are completely unfamiliar by the proposer\-executor system, such recipes may fail under even small changes because they rely on string matching and fixed benchmark patterns\.

LiveMath shows a similar limitation\. The same meta\-option artifact occurs across the search, validation, and test sets\. For the Meta\-Harness 0\.8B executor, test accuracy is100%100\\%on questions containing the meta\-option \(53/5353/53\), but only31%31\\%on the other questions \(22/7122/71\)\. Thus, a positive held\-out gain can reflect an artifact shared across splits rather than broader mathematical reasoning\.

CREATE shows the corresponding limitation for test\-time scaling: extra model calls can improve both search\- and test\-set scores, but held\-out evaluation alone does not distinguish scaling from an improvement at a fixed inference budget\. HDA complements held\-out evaluation by identifying overfitting, test\-time scaling, and the residual generalizable improvement\.

## 6Related Work

#### Agentic Harness and Harness Evolution\.

An agentic harness comprises the prompts, skills, tools, memory, context, and orchestration logic around a fixed model\([Weng, 2026](https://arxiv.org/html/2610.22466#bib.bib2)\)\. Early agent systems relied largely on manual harness engineering\([Yao et al\., 2023](https://arxiv.org/html/2610.22466#bib.bib43);[Yang et al\., 2024](https://arxiv.org/html/2610.22466#bib.bib44)\), while initial approaches to automated optimization focused on individual components, particularly prompts\([Khattab et al\., 2024](https://arxiv.org/html/2610.22466#bib.bib28);[Agrawal et al\., 2026](https://arxiv.org/html/2610.22466#bib.bib25)\)\. More recent methods expand the scope to skills\([Ni et al\., 2026](https://arxiv.org/html/2610.22466#bib.bib21);[Tang et al\., 2026](https://arxiv.org/html/2610.22466#bib.bib14);[Ke et al\., 2026](https://arxiv.org/html/2610.22466#bib.bib49)\), executable harness code\([Hu et al\., 2025](https://arxiv.org/html/2610.22466#bib.bib35);[Novikov et al\., 2025](https://arxiv.org/html/2610.22466#bib.bib32);[Karten et al\., 2026](https://arxiv.org/html/2610.22466#bib.bib12);[Ning et al\., 2026](https://arxiv.org/html/2610.22466#bib.bib22);[Zhang et al\., 2026d](https://arxiv.org/html/2610.22466#bib.bib27)\), tools and workflows\([Zhang et al\., 2025](https://arxiv.org/html/2610.22466#bib.bib47);[Chen et al\., 2025](https://arxiv.org/html/2610.22466#bib.bib41)\), memory and context\([Liu et al\., 2026](https://arxiv.org/html/2610.22466#bib.bib36);[Zhang et al\., 2026f](https://arxiv.org/html/2610.22466#bib.bib29);[Wu et al\., 2026](https://arxiv.org/html/2610.22466#bib.bib38);[Ouyang et al\., 2026](https://arxiv.org/html/2610.22466#bib.bib37)\), environments\([Huang et al\., 2026a](https://arxiv.org/html/2610.22466#bib.bib42)\), and more\([Ye et al\., 2026](https://arxiv.org/html/2610.22466#bib.bib30);[Zhang et al\., 2026g](https://arxiv.org/html/2610.22466#bib.bib34);[Zhang et al\., 2026e](https://arxiv.org/html/2610.22466#bib.bib40)\)\. Self\-evolving agent frameworks use the agent itself to improve its harness\([Zhang et al\., 2026b](https://arxiv.org/html/2610.22466#bib.bib13);[Zhang et al\., 2026a](https://arxiv.org/html/2610.22466#bib.bib31);[Zhang et al\., 2026c](https://arxiv.org/html/2610.22466#bib.bib33);[Luo et al\., 2026](https://arxiv.org/html/2610.22466#bib.bib48)\)\. Other frameworks use a separate proposer agent or optimizer to update the harness from task feedback\([Lee et al\., 2026](https://arxiv.org/html/2610.22466#bib.bib1);[Yang et al\., 2026](https://arxiv.org/html/2610.22466#bib.bib3)\)\. These methods report substantial gains without updating the executor’s model weights\. Our work instead studies where these gains come from and whether they reflect meaningful, generalizable improvements\.

#### Evaluating Harness Optimization\.

Some recent harness evolution frameworks use benchmark tasks to search for improved harnesses and then report final performance on the in\-distribution benchmark set\([Wang et al\., 2026b](https://arxiv.org/html/2610.22466#bib.bib5);[Lin et al\., 2026a](https://arxiv.org/html/2610.22466#bib.bib4);[Zhang et al\., 2026d](https://arxiv.org/html/2610.22466#bib.bib27)\)\. This protocol makes search\-set optimization difficult to distinguish from generalizable improvement\.[Wang et al\. \(2026b\)](https://arxiv.org/html/2610.22466#bib.bib5)observe that harness evolution does not reliably beat parallel sampling or sequential refinement under unified budgets\.[Huang et al\. \(2026b\)](https://arxiv.org/html/2610.22466#bib.bib39)address the same evaluation concern by constructing separate auxiliary evolution and evaluation suites designed to reduce task\-specific overfitting\. Other work studies different aspects of harness benefit\.[Lin et al\. \(2026b\)](https://arxiv.org/html/2610.22466#bib.bib7)distinguish an agent’s ability to update a harness from the ability to follow and benefit from it, using an in\-situ evaluation protocol\. Concurrent work on RRSI regularizes harness proposal and selection to reduce overfitting and improve transfer\([Xia et al\., 2026](https://arxiv.org/html/2610.22466#bib.bib26)\)\. Our work complements these approaches by attributing an observed gain to detected overfitting, test\-time scaling, and the remainder, which gives an upper bound on generalizable improvement\.

## 7Conclusion

In this work, we study where the gains of harness evolution come from\. We introduce Harness\-Delta Attribution \(HDA\), a portable, semi\-automated evaluation protocol that attributes an observed gain to overfitting, test\-time scaling, and a residual upper bound on generalizable improvement\. We find that overfitting or test\-time scaling explains a large share of the gain across many settings\. We further show that held\-out evaluation alone is not enough to assess the practical utility of an evolved harness\. We suggest that future harness\-evolution research measure overfitting using HDA together with held\-out evaluation; account for test\-time scaling by analyzing its contribution during evaluation as well as explicitly expressing the cost constraints during evolution\. Finally, benchmark selection is critical for meaningful evaluation and comparison: rich, low\-artifact benchmarks with meaningful distribution shifts are needed to discourage shortcuts, reward transferable harness improvements, and reveal differences among evolution methods\.

## Acknowledgments

This work was partially supported by the Sloan Foundation, NSF CAREER Award IIS\-2145280, NSF grant IIS\-2433071, and the NSF under Cooperative Agreement 2421782 and the Simons Foundation grant MPS\-AI\-00010515 awarded to the NSF\-Simons AI Institute for Cosmic Origins — CosmicAI,[https://www\.cosmicai\.org/](https://www.cosmicai.org/)\.

## References

- Agrawalet al\.\(2026\)L\. A\. Agrawal, S\. Tan, D\. Soylu, N\. Ziems, R\. Khare, K\. Opsahl\-Ong, A\. Singhvi, H\. Shandilya, M\. J\. Ryan, M\. Jiang,et al\.Gepa: reflective prompt evolution can outperform reinforcement learning\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 8479–8565\.Cited by:[§6](https://arxiv.org/html/2610.22466#S6.SS0.SSS0.Px1.p1.1)\.
- Anthropic \(2025\)AnthropicClaude Haiku 4\.5\.Note:[https://www\.anthropic\.com/news/claude\-haiku\-4\-5](https://www.anthropic.com/news/claude-haiku-4-5)Accessed: 2026\-09\-25Cited by:[§B\.1](https://arxiv.org/html/2610.22466#A2.SS1.p1.1),[§4](https://arxiv.org/html/2610.22466#S4.SS0.SSS0.Px3.p1.1)\.
- Anthropic \(2026a\)AnthropicClaude Code\.Note:[https://claude\.com/product/claude\-code](https://claude.com/product/claude-code)Accessed: 2026\-09\-20Cited by:[§4](https://arxiv.org/html/2610.22466#S4.SS0.SSS0.Px3.p1.1)\.
- Anthropic \(2026b\)AnthropicClaude Opus 4\.8\.Note:[https://www\.anthropic\.com/news/claude\-opus\-4\-8](https://www.anthropic.com/news/claude-opus-4-8)Accessed: 2026\-09\-25Cited by:[§B\.1](https://arxiv.org/html/2610.22466#A2.SS1.p1.1),[§4](https://arxiv.org/html/2610.22466#S4.SS0.SSS0.Px3.p1.1)\.
- Caoet al\.\(2026\)R\. Cao, M\. Chen, J\. Chen, Z\. Cui, Y\. Feng, B\. Hui, Y\. Jing, K\. Li, M\. Li, J\. Lin, Z\. Ma, K\. Shum, X\. Wang, J\. Wei, J\. Yang, J\. Zhang, L\. Zhang, Z\. Zhang, W\. Zhao, and F\. ZhouQwen3\-coder\-next technical report\.arXiv preprint arXiv:2603\.00729\.Cited by:[§B\.1](https://arxiv.org/html/2610.22466#A2.SS1.p1.1),[§4](https://arxiv.org/html/2610.22466#S4.SS0.SSS0.Px3.p1.1)\.
- Chenet al\.\(2025\)G\. Chen, Z\. Zhang, X\. Cong, F\. Guo, Y\. Wu, Y\. Lin, W\. Feng, and Y\. WangLearning evolving tools for large language models\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=wtrDLMFU9v)Cited by:[§6](https://arxiv.org/html/2610.22466#S6.SS0.SSS0.Px1.p1.1)\.
- Chenet al\.\(2026\)T\. Chen, S\. Lu, K\. Zhao, W\. Meng, H\. Teng, T\. Li, C\. Li, X\. Liu, J\. Liang, Z\. Zhang,et al\.Harnessx: a composable, adaptive, and evolvable agent harness foundry\.arXiv preprint arXiv:2606\.14249\.Cited by:[§1](https://arxiv.org/html/2610.22466#S1.p2.1)\.
- Huet al\.\(2025\)S\. Hu, C\. Lu, and J\. CluneAutomated design of agentic systems\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 21344–21377\.Cited by:[§6](https://arxiv.org/html/2610.22466#S6.SS0.SSS0.Px1.p1.1)\.
- Huanget al\.\(2026a\)C\. Huang, Z\. Wang, R\. Han, J\. Yan, Y\. Chen, Z\. CuiZhu, K\. Jiang, P\. Xia, H\. Yu, Y\. Zhuang,et al\.Envharness: awakening static worlds for agent learning\.arXiv preprint arXiv:2608\.19880\.Cited by:[§6](https://arxiv.org/html/2610.22466#S6.SS0.SSS0.Px1.p1.1)\.
- Huanget al\.\(2026b\)L\. Huang, C\. Yang, H\. Zhou, H\. Song, Z\. Chen, R\. Le, Y\. Song, W\. X\. Zhao, and T\. ZhangEvo\-bench: can language models improve agent harness?\.arXiv preprint arXiv:2608\.09096\.Cited by:[§6](https://arxiv.org/html/2610.22466#S6.SS0.SSS0.Px2.p1.1)\.
- Jimenezet al\.\(2024\)C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. R\. NarasimhanSWE\-bench: can language models resolve real\-world github issues?\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=VTF8yNQM66)Cited by:[§B\.2](https://arxiv.org/html/2610.22466#A2.SS2.p4.1),[Appendix G](https://arxiv.org/html/2610.22466#A7.p1.1),[§1](https://arxiv.org/html/2610.22466#S1.p1.1),[§1](https://arxiv.org/html/2610.22466#S1.p5.1),[§4](https://arxiv.org/html/2610.22466#S4.SS0.SSS0.Px1.p1.1)\.
- Kartenet al\.\(2026\)S\. Karten, J\. Zhang, T\. Upaa Jr, R\. Feng, W\. Li, C\. Shi, C\. Jin, and K\. VodrahalliContinual harness: online adaptation for self\-improving foundation agents\.arXiv preprint arXiv:2605\.09998\.Cited by:[§6](https://arxiv.org/html/2610.22466#S6.SS0.SSS0.Px1.p1.1)\.
- Keet al\.\(2026\)Z\. Ke, V\. Patil, H\. Shi, Y\. Li, Y\. Liu, S\. Shekkizhar, A\. Koul, J\. Wang, X\. P\. Nguyen, S\. Yavuz,et al\.EVOHARNESSBENCH: can your agents keep pace with an evolving harness?\.arXiv preprint arXiv:2609\.04280\.Cited by:[§6](https://arxiv.org/html/2610.22466#S6.SS0.SSS0.Px1.p1.1)\.
- Khattabet al\.\(2024\)O\. Khattab, A\. Singhvi, P\. Maheshwari, Z\. Zhang, K\. Santhanam, S\. Vardhamanan, S\. Haq, A\. Sharma, T\. T\. Joshi, H\. Moazam, H\. Miller, M\. Zaharia, and C\. PottsDSPy: compiling declarative language model calls into self\-improving pipelines\.Cited by:[§6](https://arxiv.org/html/2610.22466#S6.SS0.SSS0.Px1.p1.1)\.
- Leeet al\.\(2026\)Y\. Lee, R\. Nair, Q\. Zhang, K\. Lee, O\. Khattab, and C\. FinnMeta\-harness: end\-to\-end optimization of model harnesses\.arXiv preprint arXiv:2603\.28052\.Cited by:[§1](https://arxiv.org/html/2610.22466#S1.p1.1),[§1](https://arxiv.org/html/2610.22466#S1.p2.1),[§1](https://arxiv.org/html/2610.22466#S1.p5.1),[1st item](https://arxiv.org/html/2610.22466#S4.I1.i1.p1.1),[§6](https://arxiv.org/html/2610.22466#S6.SS0.SSS0.Px1.p1.1)\.
- Linet al\.\(2026a\)J\. Lin, S\. Liu, C\. Pan, L\. Lin, S\. Dou, Z\. Xi, X\. Huang, H\. Yan, Z\. Han, T\. Gui,et al\.Agentic harness engineering: observability\-driven automatic evolution of coding\-agent harnesses\.arXiv preprint arXiv:2604\.25850\.Cited by:[§1](https://arxiv.org/html/2610.22466#S1.p1.1),[§1](https://arxiv.org/html/2610.22466#S1.p5.1),[3rd item](https://arxiv.org/html/2610.22466#S4.I1.i3.p1.1),[§6](https://arxiv.org/html/2610.22466#S6.SS0.SSS0.Px2.p1.1)\.
- Linet al\.\(2026b\)M\. Lin, J\. Wu, Z\. Wang, Z\. Shi, Y\. Sang, B\. He, Z\. Liu, T\. Wei, Z\. Wu, Z\. Zhang,et al\.Harness updating is not harness benefit: disentangling evolution capabilities in self\-evolving llm agents\.arXiv preprint arXiv:2605\.30621\.Cited by:[§1](https://arxiv.org/html/2610.22466#S1.p1.1),[§6](https://arxiv.org/html/2610.22466#S6.SS0.SSS0.Px2.p1.1)\.
- Liuet al\.\(2026\)J\. Liu, X\. Ye, P\. Xia, Z\. Zheng, C\. Xie, M\. Ding, and H\. YaoEvolvemem: self\-evolving memory architecture via autoresearch for llm agents\.arXiv preprint arXiv:2605\.13941\.Cited by:[§6](https://arxiv.org/html/2610.22466#S6.SS0.SSS0.Px1.p1.1)\.
- Luoet al\.\(2026\)X\. Luo, F\. Wang, C\. Hu, D\. Xue, and Y\. DengSelf\-evolving agent harnesses via gated semantic quality\-diversity\.arXiv preprint arXiv:2607\.13683\.Cited by:[§6](https://arxiv.org/html/2610.22466#S6.SS0.SSS0.Px1.p1.1)\.
- Mahdaviet al\.\(2025\)S\. Mahdavi, M\. Li, K\. Liu, C\. Thrampoulidis, L\. Sigal, and R\. LiaoLeveraging online olympiad\-level math problems for llms training and contamination\-resistant evaluation\.arXiv preprint arXiv:2501\.14275\.Cited by:[Appendix E](https://arxiv.org/html/2610.22466#A5.p1.1),[§1](https://arxiv.org/html/2610.22466#S1.p5.1),[§4](https://arxiv.org/html/2610.22466#S4.SS0.SSS0.Px1.p1.1)\.
- Merrillet al\.\(2026\)M\. Merrill, A\. Shaw, N\. Carlini, B\. Li, H\. Raj, I\. Bercovich, L\. Shi, J\. Shin, T\. Walshe, E\. K\. Buchanan,et al\.Terminal\-bench: benchmarking agents on hard, realistic tasks in command line interfaces\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 40903–40986\.Cited by:[§1](https://arxiv.org/html/2610.22466#S1.p1.1)\.
- Niet al\.\(2026\)J\. Ni, Y\. Liu, X\. Liu, Y\. Sun, M\. Zhou, P\. Cheng, D\. Wang, E\. Zhao, X\. Jiang, and G\. JiangTrace2skill: distill trajectory\-local lessons into transferable agent skills\.arXiv preprint arXiv:2603\.25158\.Cited by:[§1](https://arxiv.org/html/2610.22466#S1.p2.1),[§6](https://arxiv.org/html/2610.22466#S6.SS0.SSS0.Px1.p1.1)\.
- Ninget al\.\(2026\)X\. Ning, K\. Tieu, D\. Fu, T\. Wei, Z\. Li, Y\. Bei, J\. Zou, M\. Ai, Z\. Liu, T\. Li,et al\.Code as agent harness\.arXiv preprint arXiv:2605\.18747\.Cited by:[§1](https://arxiv.org/html/2610.22466#S1.p2.1),[§6](https://arxiv.org/html/2610.22466#S6.SS0.SSS0.Px1.p1.1)\.
- Novikovet al\.\(2025\)A\. Novikov, N\. Vũ, M\. Eisenberger, E\. Dupont, P\. Huang, A\. Z\. Wagner, S\. Shirobokov, B\. Kozlovskii, F\. J\. Ruiz, A\. Mehrabian,et al\.Alphaevolve: a coding agent for scientific and algorithmic discovery\.arXiv preprint arXiv:2506\.13131\.Cited by:[§6](https://arxiv.org/html/2610.22466#S6.SS0.SSS0.Px1.p1.1)\.
- Ouyanget al\.\(2026\)S\. Ouyang, J\. Yan, I\. Hsu, Y\. Chen, K\. Jiang, Z\. Wang, R\. Han, L\. Le, S\. Daruki, X\. Tang,et al\.Reasoningbank: scaling agent self\-evolving with reasoning memory\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 94327–94354\.Cited by:[§6](https://arxiv.org/html/2610.22466#S6.SS0.SSS0.Px1.p1.1)\.
- Qwen Team \(2026\)Qwen TeamQwen3\.6\-27B: flagship\-level coding in a 27B dense model\.External Links:[Link](https://qwen.ai/blog?id=qwen3.6-27b)Cited by:[§B\.1](https://arxiv.org/html/2610.22466#A2.SS1.p1.1),[§4](https://arxiv.org/html/2610.22466#S4.SS0.SSS0.Px3.p1.1)\.
- Shridharet al\.\(2021\)M\. Shridhar, X\. Yuan, M\. Côté, Y\. Bisk, A\. Trischler, and M\. HausknechtALFWorld: Aligning Text and Embodied Environments for Interactive Learning\.InProceedings of the International Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2010.03768)Cited by:[Appendix D](https://arxiv.org/html/2610.22466#A4.p1.1),[§1](https://arxiv.org/html/2610.22466#S1.p5.1),[§4](https://arxiv.org/html/2610.22466#S4.SS0.SSS0.Px1.p1.1)\.
- Tanget al\.\(2026\)L\. Tang, C\. Rashtchian, C\. Ferng, A\. Tomkins, D\. Juan, and T\. VuWikiSkill: compiling agent experience into persistent knowledge for skill evolution\.arXiv preprint arXiv:2608\.27454\.Cited by:[§1](https://arxiv.org/html/2610.22466#S1.p2.1),[§6](https://arxiv.org/html/2610.22466#S6.SS0.SSS0.Px1.p1.1)\.
- Team \(2026\)Q\. TeamQwen3\. 5\-omni technical report\.arXiv preprint arXiv:2604\.15804\.Cited by:[§B\.1](https://arxiv.org/html/2610.22466#A2.SS1.p1.1),[§4](https://arxiv.org/html/2610.22466#S4.SS0.SSS0.Px3.p1.1)\.
- Tuet al\.\(2026\)X\. Tu, T\. Wang, K\. Huang, Y\. Qu, S\. Mostafavi,et al\.Benchguard: who guards the benchmarks? automated auditing of llm agent benchmarks\.arXiv preprint arXiv:2604\.24955\.Cited by:[§3](https://arxiv.org/html/2610.22466#S3.SS0.SSS0.Px2.p4.1)\.
- Wadhwaet al\.\(2026\)M\. Wadhwa, T\. S\. Roy, H\. Lederman, J\. J\. Li, and G\. DurrettCreate: testing llms for associative creativity\.arXiv preprint arXiv:2603\.09970\.Cited by:[§B\.2](https://arxiv.org/html/2610.22466#A2.SS2.p3.1),[Appendix F](https://arxiv.org/html/2610.22466#A6.p1.1),[§1](https://arxiv.org/html/2610.22466#S1.p5.1),[§4](https://arxiv.org/html/2610.22466#S4.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2610.22466#S5.SS0.SSS0.Px1.p3.1)\.
- Wanget al\.\(2026a\)J\. Wang, F\. Bianchi, S\. Zhu, F\. Nie, Y\. Kwon, B\. Dhingra, and J\. ZouAutomated benchmark auditing for ai agents and large language models\.arXiv preprint arXiv:2605\.26079\.Cited by:[§3](https://arxiv.org/html/2610.22466#S3.SS0.SSS0.Px2.p4.1)\.
- Wanget al\.\(2026b\)Y\. Wang, H\. Zhu, Z\. Hu, Y\. Yuan, Z\. Chen, S\. Senthil, H\. Hajishirzi, Y\. Tsvetkov, P\. Dasigi, and T\. XiaoRethinking the evaluation of harness evolution for agents\.Cited by:[§6](https://arxiv.org/html/2610.22466#S6.SS0.SSS0.Px2.p1.1)\.
- Weng \(2026\)L\. WengHarness engineering for self\-improvement\.lilianweng\.github\.io\.External Links:[Link](https://lilianweng.github.io/posts/2026-07-04-harness/)Cited by:[§2](https://arxiv.org/html/2610.22466#S2.p1.1),[§6](https://arxiv.org/html/2610.22466#S6.SS0.SSS0.Px1.p1.1)\.
- Wuet al\.\(2026\)S\. Wu, H\. Zhu, Y\. Zhang, X\. Wang, and S\. Yeung\-LevyAutoMem: automated learning of memory as a cognitive skill\.arXiv preprint arXiv:2607\.01224\.Cited by:[§6](https://arxiv.org/html/2610.22466#S6.SS0.SSS0.Px1.p1.1)\.
- Xiaet al\.\(2026\)P\. Xia, R\. Han, Z\. Wang, Y\. Chen, Y\. Zhuang, Y\. Lee, C\. Huang, H\. Yu, Z\. CuiZhu, Y\. Ming, H\. Yao, B\. Gokturk, T\. Pfister, and C\. LeeRRSI: regularized recursive self\-improvement of agent harnesses\.arXiv preprint arXiv:2609\.24972\.Cited by:[§6](https://arxiv.org/html/2610.22466#S6.SS0.SSS0.Px2.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§B\.1](https://arxiv.org/html/2610.22466#A2.SS1.p1.1),[§4](https://arxiv.org/html/2610.22466#S4.SS0.SSS0.Px3.p1.1)\.
- Yanget al\.\(2024\)J\. Yang, C\. Jimenez, A\. Wettig, K\. Lieret, S\. Yao, K\. Narasimhan, and O\. PressSwe\-agent: agent\-computer interfaces enable automated software engineering\.Advances in Neural Information Processing Systems37,pp\. 50528–50652\.Cited by:[§6](https://arxiv.org/html/2610.22466#S6.SS0.SSS0.Px1.p1.1)\.
- Yanget al\.\(2026\)Y\. Yang, Z\. Gong, W\. Huang, Q\. Yang, Z\. Zhou, Z\. Huang, Y\. Li, X\. Gao, Q\. Dai, B\. Liu,et al\.Skillopt: executive strategy for self\-evolving agent skills\.arXiv preprint arXiv:2605\.23904\.Cited by:[§B\.2](https://arxiv.org/html/2610.22466#A2.SS2.p1.1),[§1](https://arxiv.org/html/2610.22466#S1.p2.1),[§1](https://arxiv.org/html/2610.22466#S1.p5.1),[2nd item](https://arxiv.org/html/2610.22466#S4.I1.i2.p1.1),[§6](https://arxiv.org/html/2610.22466#S6.SS0.SSS0.Px1.p1.1)\.
- Yaoet al\.\(2023\)S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. R\. Narasimhan, and Y\. CaoReAct: synergizing reasoning and acting in language models\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=WE_vluYUL-X)Cited by:[§6](https://arxiv.org/html/2610.22466#S6.SS0.SSS0.Px1.p1.1)\.
- Yeet al\.\(2026\)H\. Ye, X\. He, V\. Arak, H\. Dong, and G\. SongMeta context engineering via agentic skill evolution\.InForty\-third International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=P1jHroBS5E)Cited by:[§6](https://arxiv.org/html/2610.22466#S6.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2026a\)H\. Zhang, S\. Zhang, K\. Li, C\. Zhang, Y\. Chen, Y\. Zhang, L\. Bai, and S\. HuSelf\-harness: harnesses that improve themselves\.arXiv preprint arXiv:2606\.09498\.Cited by:[§6](https://arxiv.org/html/2610.22466#S6.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2026b\)J\. Zhang, S\. Hu, C\. Lu, R\. Lange, and J\. CluneDarwin gödel machine: open\-ended evolution of self\-improving agents\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 104223–104294\.Cited by:[§6](https://arxiv.org/html/2610.22466#S6.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2026c\)J\. Zhang, B\. Zhao, W\. Yang, J\. Foerster, J\. Clune, M\. Jiang, S\. Devlin, and T\. ShavrinaHyperagents\.arXiv preprint arXiv:2603\.19461\.Cited by:[§6](https://arxiv.org/html/2610.22466#S6.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2026d\)J\. Zhang, Y\. Gu, J\. Ruan, M\. Song, Y\. Peng, Z\. Han, J\. Xiang, Z\. Wang, C\. Yang, Y\. Ouyang,et al\.Harnessing agentic evolution\.arXiv preprint arXiv:2605\.13821\.Cited by:[§6](https://arxiv.org/html/2610.22466#S6.SS0.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2610.22466#S6.SS0.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2025\)J\. Zhang, J\. Xiang, Z\. Yu, F\. Teng, X\. Chen, J\. Chen, M\. Zhuge, X\. Cheng, S\. Hong, J\. Wang,et al\.Aflow: automating agentic workflow generation\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 34040–34077\.Cited by:[§6](https://arxiv.org/html/2610.22466#S6.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2026e\)L\. Zhang, R\. Zhou, D\. Song, Z\. Chen, Y\. Tian, J\. Yang, H\. Ma, C\. Li, G\. Feng, X\. Li,et al\.Harnesscompass: guiding automatic harness evolution toward generalizable and effective agent harnesses\.arXiv preprint arXiv:2608\.01918\.Cited by:[§6](https://arxiv.org/html/2610.22466#S6.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2026f\)Q\. Zhang, C\. Hu, S\. Upasani, B\. Ma, F\. Hong, V\. Kamanuru, J\. Rainton, C\. Wu, M\. Ji, H\. Li,et al\.Agentic context engineering: evolving contexts for self\-improving language models\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 86069–86100\.Cited by:[§6](https://arxiv.org/html/2610.22466#S6.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2026g\)Y\. Zhang, Y\. Dai, J\. Tan, L\. Yang, R\. Mullur, T\. Hoang, Z\. Hu, J\. Zhu, P\. Mui, S\. Savarese,et al\.DarwinX: evolving agent harnesses through natural selection\.arXiv preprint arXiv:2608\.07545\.Cited by:[§6](https://arxiv.org/html/2610.22466#S6.SS0.SSS0.Px1.p1.1)\.

## Appendix AHarness Evolution Details

Algorithm[1](https://arxiv.org/html/2610.22466#alg1)summarizes the standard harness evolution loop\.

Algorithm 1Standard Harness Evolution Loop1:Input:baseline harness

H0H\_\{0\}, executor model

MM, proposer agent

𝒫\\mathcal\{P\}, task sets

𝒳search\\mathcal\{X\}\_\{\\mathrm\{search\}\}and optional

𝒳val\\mathcal\{X\}\_\{\\mathrm\{val\}\}, iterations

NN
2:

F0←\{Evaluate​\(H0,M,𝒳search,𝒳val\)\}F\_\{0\}\\leftarrow\\\{\\textsc\{Evaluate\}\(H\_\{0\},M,\\mathcal\{X\}\_\{\\mathrm\{search\}\},\\mathcal\{X\}\_\{\\mathrm\{val\}\}\)\\\}
3:for

t=1,…,Nt=1,\\ldots,Ndo

4:

ℋt←Propose​\(𝒫,Ft−1\)\\mathcal\{H\}\_\{t\}\\leftarrow\\textsc\{Propose\}\(\\mathcal\{P\},F\_\{t\-1\}\)
5:

Ft←Ft−1F\_\{t\}\\leftarrow F\_\{t\-1\}
6:for all

H∈Valid​\(ℋt\)H\\in\\textsc\{Valid\}\(\\mathcal\{H\}\_\{t\}\)do

7:

ft​\(H\)←Evaluate​\(H,M,𝒳search,𝒳val\)f\_\{t\}\(H\)\\leftarrow\\textsc\{Evaluate\}\(H,M,\\mathcal\{X\}\_\{\\mathrm\{search\}\},\\mathcal\{X\}\_\{\\mathrm\{val\}\}\)
8:

Ft←Ft∪\{ft​\(H\)\}F\_\{t\}\\leftarrow F\_\{t\}\\cup\\\{f\_\{t\}\(H\)\\\}
9:

H^←Select​\(FN,𝒳sel\)\\hat\{H\}\\leftarrow\\textsc\{Select\}\(F\_\{N\},\\mathcal\{X\}\_\{\\mathrm\{sel\}\}\)
10:return

H^\\hat\{H\}

## Appendix BExperiment Setup Details

This appendix reports the model coverage and data splits used in our experiments\.

### B\.1Executor models

We evaluate 11 executor models: nine open\-weight models from the Qwen3, Qwen3\.5, and Qwen3\.6 families\([Yang et al\., 2025](https://arxiv.org/html/2610.22466#bib.bib15);[Team, 2026](https://arxiv.org/html/2610.22466#bib.bib16);[Cao et al\., 2026](https://arxiv.org/html/2610.22466#bib.bib17);[Qwen Team, 2026](https://arxiv.org/html/2610.22466#bib.bib18)\), together with Claude Haiku 4\.5 and Claude Opus 4\.8\([Anthropic, 2025](https://arxiv.org/html/2610.22466#bib.bib45);[Anthropic, 2026b](https://arxiv.org/html/2610.22466#bib.bib46)\)\. The 25 model–benchmark–framework settings comprise 16 Meta\-Harness, 6 SkillOpt, and 3 AHE settings\. Table[1](https://arxiv.org/html/2610.22466#S5.T1)gives the complete mapping\.

### B\.2Benchmarks and splits

ForLiveMath, we follow the 35/18/124 train/validation/test split used by SkillOpt\([Yang et al\., 2026](https://arxiv.org/html/2610.22466#bib.bib3)\)\. All splits follow the same question construction, and performance is measured by accuracy\.

We splitALFWorldalong both task\-type and room axes\. The 50 search games coverpick\_and\_place\_simple,pick\_clean\_then\_place\_in\_recep, andpick\_heat\_then\_place\_in\_recepin kitchens, living rooms, and bathrooms\. The 36 validation games include all six task types but only selected task\-type–room combinations; the test set contains fresh, disjoint games with broader combination coverage\. The 120 test games comprise 25 new instances of the search task types, 20 pick\-and\-place games in the held\-out bedroom, and 25 games each forpick\_cool\_then\_place\_in\_recep,pick\_two\_obj\_and\_place, andlook\_at\_obj\_in\_light\. Performance is measured by episode success within 50 environment steps\.

We splitCREATEby relation type and domain\. The 120\-item search split covers five common encyclopedic relations: occupation, place of birth, educated at, award received, and country of citizenship\. The 80\-item validation split includes new instances of these relations and four additional relations\. The 561\-item test set contains 107 new instances of the search relations, 263 instances from held\-out biographical, family, and career relations, and 191 instances from held\-out scientific and biomedical relations\. Performance is measured by Creative Utility \(CU\), a continuous and unbounded metric that rewards valid and diverse answers\([Wadhwa et al\., 2026](https://arxiv.org/html/2610.22466#bib.bib9)\)\.

We splitSWE\-bench Verifiedinto 48 search, 24 validation, and 50 test instances\([Jimenez et al\., 2024](https://arxiv.org/html/2610.22466#bib.bib11)\)\. The instances are sampled with seed 42 after filtering for tasks resolvable within one hour, while preserving the repository distribution across partitions\. Performance is measured by success rate\.

### B\.3Validation\-based signal for Meta\-Harness

For each candidate harness, the search\-set evaluation exposes per\-instance scores, execution trajectories, and logs to the proposer through the shared filesystem\. We additionally evaluate candidates on a validation set, but expose only the aggregate validation score; per\-instance validation results and trajectories remain hidden\. We retain two harnesses from the evaluated candidates: the harness with the highest search score,H^\\hat\{H\}, and the harness with the highest validation score,H^val\\hat\{H\}^\{\\mathrm\{val\}\}\. Both are evaluated on the held\-out test set only after evolution\. This setup tests whether a standard validation signal improves transfer without revealing validation examples or trajectories\.

## Appendix CThe HDA Skill

Figure[6](https://arxiv.org/html/2610.22466#A3.F6)summarizes the HDA skill used by the analysis agent\. The complete machine\-readable skill is included in the supplementary materials\.

HDA Skill: Summarized WorkflowInput:a baseline harnessH0H\_\{0\}, an evolved harnessH^\\hat\{H\}, and one evaluation set\.1\.SelectH0H\_\{0\},H^\\hat\{H\}, and the evaluation set, and computeΔ​S=S⁡\(H^\)−S⁡\(H0\)\\Delta S=S\(\\hat\{H\}\)\-S\(H\_\{0\}\)\.2\.Specify the task objective, score aggregation, inference\-cost measure, and task\-specific evaluation hooks\.3\.Apply a paired significance test\. IfΔ​S\\Delta Sis not statistically significant, report it as within noise and do not assign an attribution\.4\.Construct the compute\-matched baselineH0scaleH\_\{0\}^\{\\mathrm\{scale\}\}and computeT=S⁡\(H0scale\)−S⁡\(H0\)T=S\(H\_\{0\}^\{\\mathrm\{scale\}\}\)\-S\(H\_\{0\}\)\.Human checkpoint 1:review the compute measure and matching procedure\.5\.Detect and neutralize supported overfitting mechanisms inH^\\hat\{H\}to constructH^neutral\\hat\{H\}^\{\\mathrm\{neutral\}\}, then computeO=S⁡\(H^\)−S⁡\(H^neutral\)O=S\(\\hat\{H\}\)\-S\(\\hat\{H\}^\{\\mathrm\{neutral\}\}\)\.Human checkpoint 2:review the evidence and neutralization\.6\.Compute the residualG=Δ​S−O−T=S⁡\(H^neutral\)−S⁡\(H0scale\),G=\\Delta S\-O\-T=S\(\\hat\{H\}^\{\\mathrm\{neutral\}\}\)\-S\(H\_\{0\}^\{\\mathrm\{scale\}\}\),and report the raw components, normalized percentages, evidence, and limitations\.Interpretation:OOis a lower bound on detected overfitting, andGGis an upper bound on generalizable improvement\. Held\-out transfer is evaluated separately\.Figure 6:Overview of the HDA skill\. The skill standardizes the attribution procedure while retaining human review at the two task\-dependent decisions: compute matching and overfitting neutralization\.Figure 7:Overview of representative evolved\-harness mechanisms across the four benchmarks \(left four cards, Meta\-Harness\): hard\-coded object\-location priors and task recipes on ALFWorld, the meta\-option string match on LiveMath, parallel generation frames on CREATE \(a test\-time\-scaling counter\-case\), and a patch\-size prior on SWE\-bench\. The two right cards show the*same*LiveMath meta\-option artifact independently rediscovered by SkillOpt \(as a skill default rule\) and AHE \(as an explicit prompt rule\)\. Red marks overfitting evidence; yellow marks test\-time\-scaling evidence; each card’sOO/TTchip is the attribution of the specific Table[1](https://arxiv.org/html/2610.22466#S5.T1)row whose winner is excerpted — not a benchmark\-level average \(ALFWorld and LiveMath: Qwen3\.5\-0\.8B; CREATE: Claude Haiku 4\.5; SWE\-bench: Qwen3\-4B\-Instruct\-2507; SkillOpt: LiveMath Qwen3\.5\-4B; AHE: LiveMath Claude Opus 4\.8\)\. The following sections \([E](https://arxiv.org/html/2610.22466#A5)–[G](https://arxiv.org/html/2610.22466#A7)\) present each mechanism in detail with source excerpts\.
## Appendix DALFWorld: Benchmark\-Specific Procedural Scaffolds

On ALFWorld\([Shridhar et al\., 2021](https://arxiv.org/html/2610.22466#bib.bib10)\), Meta\-Harness and SkillOpt encode similar simulator\-specific knowledge, including task recipes, appliance mappings, object\-location priors, and environment rules\. Figures[8](https://arxiv.org/html/2610.22466#A4.F8)and[9](https://arxiv.org/html/2610.22466#A4.F9)show one representative harness from each framework\. Red highlights mark mechanisms identified as overfitting by HDA\.

Meta\-Harness⋅\\cdotALFWorld⋅\\cdotQwen3\.5\-0\.8B: zero\-LLM controller[⬇](data:text/plain;base64,TE9DQVRJT05fUFJJT1JTID0gewogICAgKCpAXGhsb3ZlcntcdGV4dHR0eydtdWcnOiBbJ2NvZmZlZW1hY2hpbmUnLCAnY291bnRlcnRvcCcsICdjYWJpbmV0JywgJ3NoZWxmJ10sfX1AKikKICAgICgqQFxobG92ZXJ7XHRleHR0dHsndG9tYXRvJzogWydmcmlkZ2UnLCAnY291bnRlcnRvcCcsICdkaW5pbmd0YWJsZSddLH19QCopCiAgICAoKkBcaGxvdmVye1x0ZXh0dHR7J2Rlc2tsYW1wJzogWydkZXNrJywgJ3NpZGV0YWJsZSddLH19QCopCiAgICAuLi4gICMgb2JqZWN0LXRvLXJlY2VwdGFjbGUgcHJpb3JzCn0KCmNsYXNzIEFkYXB0aXZlQ29udGFpbmVyRGVwdGhBZ2VudChBZ2VudEhhcm5lc3MpOgogICAgQHN0YXRpY21ldGhvZAogICAgZGVmIF9wYXJzZV90YXNrKHRhc2tfZGVzY3JpcHRpb246IHN0cik6CiAgICAgICAgLi4uCiAgICAgICAgKCpAXGhsb3ZlcntcdGV4dHR0e2lmICdjb29sJyBpbiB0YXNrOiBwcm9jZXNzXF9zdGVwcy5hcHBlbmQoKCdjb29sJywgJ2ZyaWRnZScpKX19QCopCiAgICAgICAgKCpAXGhsb3ZlcntcdGV4dHR0e2VsaWYgJ2hlYXQnIGluIHRhc2s6IHByb2Nlc3NcX3N0ZXBzLmFwcGVuZCgoJ2hlYXQnLCAnbWljcm93YXZlJykpfX1AKikKICAgICAgICAoKkBcaGxvdmVye1x0ZXh0dHR7ZWxpZiAnY2xlYW4nIGluIHRhc2s6IHByb2Nlc3NcX3N0ZXBzLmFwcGVuZCgoJ2NsZWFuJywgJ3NpbmtiYXNpbicpKX19QCopCgogICAgZGVmIHN0ZXAoc2VsZiwgb2JzZXJ2YXRpb24sIGFkbWlzc2libGVfYWN0aW9ucyk6CiAgICAgICAgc2VsZi5fdXBkYXRlX3N0YXRlKG9ic2VydmF0aW9uLCBhZG1pc3NpYmxlX2FjdGlvbnMpCiAgICAgICAgYWN0aW9uID0gc2VsZi5fcGlja19hY3Rpb24ob2JzZXJ2YXRpb24sIGFkbWlzc2libGVfYWN0aW9ucykKICAgICAgICByZXR1cm4gU3RlcFJlc3VsdChhY3Rpb249YWN0aW9uLCB0aG91Z2h0PWYicGhhc2U9e3NlbGYucGhhc2V9Iik=)LOCATION\_PRIORS=\{’mug’: \[’coffeemachine’, ’countertop’, ’cabinet’, ’shelf’\],’tomato’: \[’fridge’, ’countertop’, ’diningtable’\],’desklamp’: \[’desk’, ’sidetable’\],\.\.\.\#object\-to\-receptaclepriors\}classAdaptiveContainerDepthAgent\(AgentHarness\):@staticmethoddef\_parse\_task\(task\_description:str\):\.\.\.if ’cool’ in task: process\_steps\.append\(\(’cool’, ’fridge’\)\)elif ’heat’ in task: process\_steps\.append\(\(’heat’, ’microwave’\)\)elif ’clean’ in task: process\_steps\.append\(\(’clean’, ’sinkbasin’\)\)defstep\(self,observation,admissible\_actions\):self\.\_update\_state\(observation,admissible\_actions\)action=self\.\_pick\_action\(observation,admissible\_actions\)returnStepResult\(action=action,thought=f"phase=\{self\.phase\}"\)Figure 8:Meta\-Harness ALFWorld winner for Qwen3\.5\-0\.8B\. The highlighted tables and parser rules encode benchmark\-specific object locations and task procedures\. The full harness contains nollm\_call, so all actions come from the deterministic controller\. HDA:O/T/G=100/0/0O/T/G=100/0/0\.SkillOpt⋅\\cdotALFWorld⋅\\cdotQwen3\.5\-4B: the scaffold as a natural\-language skill[⬇](data:text/plain;base64,IyMgVGFzayBQcm9jZWR1cmVzCi0gKipwaWNrX2hlYXRfdGhlbl9wbGFjZSoqOiB0YWtlIDxvYmo+IC0+IGdvIHRvIG1pY3Jvd2F2ZSAxIC0+CiAgKCpAXGhsb3ZlcntcdGV4dHR0e2hlYXQgPG9iaj4gd2l0aCBtaWNyb3dhdmUgMX19QCopIC0+IGdvIHRvIDxyZWNlcD4gLT4gcGxhY2UuCi0gKipwaWNrX2Nvb2xfdGhlbl9wbGFjZSoqOiB0YWtlIDxvYmo+IC0+IGdvIHRvIGZyaWRnZSAxIC0+CiAgKCpAXGhsb3ZlcntcdGV4dHR0e2Nvb2wgPG9iaj4gd2l0aCBmcmlkZ2UgMX19QCopIC0+IGdvIHRvIDxyZWNlcD4gLT4gcGxhY2UuCi0gKipwaWNrX2NsZWFuX3RoZW5fcGxhY2UqKjogdGFrZSA8b2JqPiAtPiBnbyB0byBzaW5rYmFzaW4gMSAtPgogICgqQFxobG92ZXJ7XHRleHR0dHtjbGVhbiA8b2JqPiB3aXRoIHNpbmtiYXNpbiAxfX1AKikgLT4gZ28gdG8gPHJlY2VwPiAtPiBwbGFjZS4KCiMjIFNpbXVsYXRvciBSdWxlcyBhbmQgU2VhcmNoIFByaW9ycwpJZiBhbiBhY3Rpb24gcmV0dXJucyAoKkBcaGxvdmVye1x0ZXh0dHR7Ik5vdGhpbmcgaGFwcGVucyJ9fUAqKSwgZG8gbm90IHJlcGVhdCBpdDoKKCpAXGhsb3ZlcntcdGV4dHR0e2EgcHJlY29uZGl0aW9uIGlzIHVubWV0LiBUaGVyZSBpcyBubyB0ZXJtaW5hdGUgYWN0aW9uLn19QCopCkNsb3NlZCBjYWJpbmV0cywgZHJhd2VycywgZnJpZGdlcywgYW5kIG1pY3Jvd2F2ZXMgaGlkZSB0aGVpciBjb250ZW50cy4KCkZvciBmb29kIGFuZCBwcm9kdWNlLCBzZWFyY2ggKCpAXGhsb3ZlcntcdGV4dHR0e2NvdW50ZXJ0b3BzLCBkaW5pbmcgdGFibGVzLCBzaW5rYmFzaW5zLH19QCopCigqQFxobG92ZXJ7XHRleHR0dHthbmQgZnJpZGdlcyBiZWZvcmUgY2FiaW5ldHMgb3IgZHJhd2Vyc319QCopOwooKkBcaGxvdmVye1x0ZXh0dHR7dGhpcyBpcyBleGFjdGx5IGhvdyB0aGUgbGV0dHVjZSB0YXNrIHdhcyBsb3N0Ln19QCop)\#\#TaskProcedures\-\*\*pick\_heat\_then\_place\*\*:take<obj\>\-\>gotomicrowave1\-\>heat <obj\> with microwave 1\-\>goto<recep\>\-\>place\.\-\*\*pick\_cool\_then\_place\*\*:take<obj\>\-\>gotofridge1\-\>cool <obj\> with fridge 1\-\>goto<recep\>\-\>place\.\-\*\*pick\_clean\_then\_place\*\*:take<obj\>\-\>gotosinkbasin1\-\>clean <obj\> with sinkbasin 1\-\>goto<recep\>\-\>place\.\#\#SimulatorRulesandSearchPriorsIfanactionreturns"Nothing happens",donotrepeatit:a precondition is unmet\. There is no terminate action\.Closedcabinets,drawers,fridges,andmicrowaveshidetheircontents\.Forfoodandproduce,searchcountertops, dining tables, sinkbasins,and fridges before cabinets or drawers;this is exactly how the lettuce task was lost\.Figure 9:SkillOpt ALFWorld skill for Qwen3\.5\-4B\. The highlighted instructions encode the same appliance recipes, simulator rules, and location priors as the Meta\-Harness controller, including a rule distilled from a specific search\-set failure\. HDA:O/T/G=44/0/56O/T/G=44/0/56\.The two frameworks therefore rediscover similar ALFWorld\-specific procedures but represent them differently: executable controller logic in Meta\-Harness and natural\-language instructions in SkillOpt\. The zero\-LLM controller still reaches93\.3%93\.3\\%success on held\-out ALFWorld games because the same simulator ontology and rules persist across splits, illustrating why held\-out performance alone may not reveal benchmark\-specific overfitting\. Full harnesses are included in the supplementary materials\.

## Appendix ELiveMath: The Shared Meta\-Option Shortcut

LiveMath\([Mahdavi et al\., 2025](https://arxiv.org/html/2610.22466#bib.bib8)\)contains a recurring answer choice: “one of the remaining options is correct, but a stronger result can be proven\.” It appears in 21 of the 35 search questions and is correct in all 21\. The same pattern occurs in 53 of the 124 test questions and is again always correct, so exploiting it can also appear to transfer\. All three frameworks discover this shortcut\. Figure[10](https://arxiv.org/html/2610.22466#A5.F10)shows how each framework encodes it; red highlights mark overfitting mechanisms and yellow highlights mark test\-time scaling\.

Meta\-Harness⋅\\cdotLiveMath⋅\\cdotQwen3\.5\-0\.8B: string matching with a multi\-call fallback[⬇](data:text/plain;base64,TUVUQV9NQVJLRVJTID0gWwogICAgKCpAXGhsb3Zlcnsib25lIG9mIHRoZSByZW1haW5pbmcgb3B0aW9ucyBpcyBjb3JyZWN0Iix9QCopCiAgICAoKkBcaGxvdmVyeyJzdHJvbmdlciByZXN1bHQgY2FuIGJlIHByb3ZlbiIsfUAqKQpdCgpkZWYgc29sdmUoc2VsZiwgaXRlbSk6CiAgICAoKkBcaGxvdmVye21ldGFcX2xhYmVsID0gc2VsZi5cX2ZpbmRcX21ldGEoaXRlbVsiY2hvaWNlcyJdKX1AKikKICAgICgqQFxobG92ZXJ7aWYgbWV0YVxfbGFiZWw6fUAqKQogICAgICAgICgqQFxobG92ZXJ7cmV0dXJuIFNvbHZlUmVzdWx0KHByZWRpY3RlZFxfbGFiZWw9bWV0YVxfbGFiZWwpfUAqKQoKICAgIHZvdGVzID0gW10KICAgICgqQFxobHR0c3tmb3IgcHJvbXB0IGluIGdlbmVyYXRpb25cX3Byb21wdHM6fUAqKQogICAgICAgICgqQFxobHR0c3tnZW5lcmF0aW9uID0gc2VsZi5sbG1cX2NhbGwocHJvbXB0KX1AKikKICAgICAgICAoKkBcaGx0dHN7dm90ZXMuYXBwZW5kKHNlbGYuXF9tYXRjaFxfY2hvaWNlKGdlbmVyYXRpb24pKX1AKikKICAgICgqQFxobHR0c3tyZXR1cm4gbWFqb3JpdHlcX3ZvdGUodm90ZXMpfUAqKQ==)META\_MARKERS=\["one of the remaining options is correct","stronger result can be proven",\]defsolve\(self,item\):meta\_label = self\.\_find\_meta\(item\["choices"\]\)if meta\_label:return SolveResult\(predicted\_label=meta\_label\)votes=\[\]for prompt in generation\_prompts:generation = self\.llm\_call\(prompt\)votes\.append\(self\.\_match\_choice\(generation\)\)return majority\_vote\(votes\)

SkillOpt⋅\\cdotLiveMath⋅\\cdotQwen3\.5\-4B: the same shortcut as a skill rule[⬇](data:text/plain;base64,IyMgT3B0aW9uIENvbXBhcmlzb24KLSBUcmVhdCBhIG1ldGEtb3B0aW9uIHN1Y2ggYXMgIm9uZSBvZiB0aGUgcmVtYWluaW5nIG9wdGlvbnMgaXMgY29ycmVjdCwKICBidXQgYSBzdHJvbmdlciByZXN1bHQgY2FuIGJlIHByb3ZlbiIgKCpAXGhsb3ZlcntcdGV4dHR0e2FzIGEgZmlyc3QtY2xhc3MgY2FuZGlkYXRlLH19QCopCiAgKCpAXGhsb3ZlcntcdGV4dHR0e25ldmVyIGEgdGhyb3dhd2F5Ln19QCopCi0gSWYgdGhlIGJlc3QgY29uY3JldGUgb3B0aW9uIGlzIHRydWUgYnV0IGltcHJvdmFibGUsCiAgKCpAXGhsb3ZlcntcdGV4dHR0e3RoZSBtZXRhLW9wdGlvbiBhbG1vc3QgYWx3YXlzIGJlYXRzIGl0Ln19QCopCgojIyMgTWV0YS1vcHRpb24gZGVjaXNpb24gcnVsZQpUaGUgbWV0YS1vcHRpb24gKCpAXGhsb3ZlcntcdGV4dHR0e2lzIHlvdXIgd29ya2luZyBkZWZhdWx0fX1AKikuIFJlamVjdCBpdCBvbmx5IHdoZW4gYQpjb25jcmV0ZSBvcHRpb24gZXhhY3RseSBtYXRjaGVzIHRoZSBzdHJvbmdlc3Qga25vd24gcmVzdWx0IGFuZCBubyBzdHJvbmdlcgp0cnVlIHN0YXRlbWVudCBjYW4gYmUgYWRkZWQ7IG90aGVyd2lzZSwgKCpAXGhsb3ZlcntcdGV4dHR0e3RoZSBtZXRhLW9wdGlvbiB3aW5zLn19QCop)\#\#OptionComparison\-Treatameta\-optionsuchas"oneoftheremainingoptionsiscorrect,butastrongerresultcanbeproven"as a first\-class candidate,never a throwaway\.\-Ifthebestconcreteoptionistruebutimprovable,the meta\-option almost always beats it\.\#\#\#Meta\-optiondecisionruleThemeta\-optionis your working default\.Rejectitonlywhenaconcreteoptionexactlymatchesthestrongestknownresultandnostrongertruestatementcanbeadded;otherwise,the meta\-option wins\.

AHE⋅\\cdotLiveMath⋅\\cdotClaude Opus 4\.8: an explicit prompt rule[⬇](data:text/plain;base64,IyMgSGFuZGxpbmcgdGhlICJtZXRhIiBvcHRpb24KKCpAXGhsb3ZlcntcdGV4dHR0e1RoaXMgbWV0YS1vcHRpb24gaXMgYSBGSVJTVC1DTEFTUyBjYW5kaWRhdGUsIG5vdCBmaWxsZXIufX1AKikKSWYgYSBzdHJpY3RseSBzdHJvbmdlciB0cnVlIHN0YXRlbWVudCBleGlzdHMgYmV5b25kIHRoZSBiZXN0IGNvbmNyZXRlIG9wdGlvbiwKKCpAXGhsb3ZlcntcdGV4dHR0e3RoZSBtZXRhLW9wdGlvbiBpcyBjb3JyZWN0LiBDaG9vc2UgaXQufX1AKik=)\#\#Handlingthe"meta"optionThis meta\-option is a FIRST\-CLASS candidate, not filler\.Ifastrictlystrongertruestatementexistsbeyondthebestconcreteoption,the meta\-option is correct\. Choose it\.

Figure 10:Three representations of the same LiveMath artifact\. Meta\-Harness directly returns the matching option before invoking the executor and otherwise uses multiple model calls \(O/T/G=91/9/0O/T/G=91/9/0\)\. SkillOpt encodes it as a default decision rule in the skill document \(88/0/1288/0/12\)\. AHE writes an explicit selection rule into the system prompt \(100/0/0100/0/0\)\.The shortcut therefore recurs across code, skill, and prompt evolution\. Because the artifact is shared across the search and test splits, held\-out gains within LiveMath do not by themselves establish improved mathematical reasoning\. Full harnesses for all three frameworks are included in the supplementary materials\.

## Appendix FCREATE: Metric\-Driven Test\-Time Scaling

CREATE rewards answer pools containing many valid and diverse responses through its Creative Utility \(CU\) metric\([Wadhwa et al\., 2026](https://arxiv.org/html/2610.22466#bib.bib9)\)\. HDA finds no detected overfitting in the attributed CREATE settings \(O=0O=0\); instead, frameworks that can modify orchestration increase the number of model calls and token usage\. Figure[11](https://arxiv.org/html/2610.22466#A6.F11)shows the parallel strategy discovered by Meta\-Harness, the sequential strategy discovered by AHE, and SkillOpt’s fixed single\-call control flow\. Yellow highlights mark test\-time\-scaling mechanisms\.

Meta\-Harness⋅\\cdotCREATE⋅\\cdotClaude Haiku 4\.5: parallel generation[⬇](data:text/plain;base64,IyBTaXggaW5kZXBlbmRlbnQgZmlyc3Qtd2F2ZSBmcmFtZXMKKCpAXGhsdHRze2ZvciBmcmFtZVxfbmFtZSwgdGVtcGxhdGUsIHRlbXAgaW4gd2F2ZTFcX2ZyYW1lczp9QCopCiAgICAoKkBcaGx0dHN7cmVzcG9uc2UgPSBzZWxmLmxsbVxfY2FsbChtZXNzYWdlcz1bLi4uXSwgdGVtcGVyYXR1cmU9dGVtcCl9QCopCiAgICBhbGxfcGF0aHMuZXh0ZW5kKGV4dHJhY3RfcGF0aHMocmVzcG9uc2UpKQoKIyBGb3VydGVlbiBhZGRpdGlvbmFsIGRpdmVyc2l0eSBmcmFtZXMKKCpAXGhsdHRze2ZvciBmcmFtZVxfY2ZnIGluIFxfV0FWRTJcX0ZSQU1FUzp9QCopCiAgICAoKkBcaGx0dHN7cmVzcG9uc2UgPSBzZWxmLmxsbVxfY2FsbChtZXNzYWdlcz1bLi4uXSwgdGVtcGVyYXR1cmU9ZnJhbWVcX2NmZ1sidGVtcCJdKX1AKikKICAgIGFsbF9wYXRocy5leHRlbmQoZXh0cmFjdF9wYXRocyhyZXNwb25zZSkp)\#Sixindependentfirst\-waveframesfor frame\_name, template, temp in wave1\_frames:response = self\.llm\_call\(messages=\[\.\.\.\], temperature=temp\)all\_paths\.extend\(extract\_paths\(response\)\)\#Fourteenadditionaldiversityframesfor frame\_cfg in \_WAVE2\_FRAMES:response = self\.llm\_call\(messages=\[\.\.\.\], temperature=frame\_cfg\["temp"\]\)all\_paths\.extend\(extract\_paths\(response\)\)

AHE⋅\\cdotCREATE⋅\\cdotClaude Opus 4\.8: sequential expansion[⬇](data:text/plain;base64,KCpAXGhsdHRze1x0ZXh0dHR7WW91IE1VU1QgZ2VuZXJhdGUgYXMgbWFueSBkaXN0aW5jdCB2YWxpZCBwYXRocyBhcyBwb3NzaWJsZS59fUAqKQpUaGUgZmluYWwgYW5zd2VyIHNob3VsZCBjb250YWluIE1BTlkgZGlzdGluY3QgcGF0aHMgKCpAXGhsdHRze1x0ZXh0dHR7KGFpbSBmb3IgOCspfX1AKikuCgpNSU5fUEFUSFMgPSA1CmlmIGhhc19hbnN3ZXJfYmxvY2sgYW5kIG5fcGF0aHMgPCBNSU5fUEFUSFM6CiAgICBmZWVkYmFjayA9IF9idWlsZF9hYnN0ZW50aW9uX2ZlZWRiYWNrKG5fcGF0aHMpCiAgICByZXR1cm4gSG9va1Jlc3VsdC53aXRoX21vZGlmaWNhdGlvbnMoCiAgICAgICAgbWVzc2FnZXM9bWVzc2FnZXMgKyBbZmVlZGJhY2tdLAogICAgICAgICgqQFxobHR0c3tmb3JjZVxfY29udGludWU9VHJ1ZSx9QCopICAjIGZvcmNlIGFub3RoZXIgbW9kZWwgdHVybgogICAgKQ==)You MUST generate as many distinct valid paths as possible\.ThefinalanswershouldcontainMANYdistinctpaths\(aim for 8\+\)\.MIN\_PATHS=5ifhas\_answer\_blockandn\_paths<MIN\_PATHS:feedback=\_build\_abstention\_feedback\(n\_paths\)returnHookResult\.with\_modifications\(messages=messages\+\[feedback\],force\_continue=True,\#forceanothermodelturn\)

SkillOpt⋅\\cdotCREATE: the skill changes, but the one\-call rollout remains fixed[⬇](data:text/plain;base64,c3lzdGVtID0gX2J1aWxkX3N5c3RlbShza2lsbF9jb250ZW50KSAgIyBpbmplY3QgdGhlIGV2b2x2ZWQgc2tpbGwKdXNlciA9IF9idWlsZF91c2VyKGl0ZW0pCgpyZXNwb25zZSwgXyA9IGNoYXRfdGFyZ2V0KAogICAgc3lzdGVtPXN5c3RlbSwKICAgIHVzZXI9dXNlciwKICAgIG1heF9jb21wbGV0aW9uX3Rva2Vucz1tYXhfY29tcGxldGlvbl90b2tlbnMsCikKcmVzdWx0WyJuX3R1cm5zIl0gPSAx)system=\_build\_system\(skill\_content\)\#injecttheevolvedskilluser=\_build\_user\(item\)response,\_=chat\_target\(system=system,user=user,max\_completion\_tokens=max\_completion\_tokens,\)result\["n\_turns"\]=1

Figure 11:CREATE orchestration across the three frameworks\. Meta\-Harness makes 20 calls per item and obtainsO/T/G=0/49/51O/T/G=0/49/51\. AHE increases the mean from 1\.04 to 2\.46 calls per item through forced expansion turns; using the paper’sk=3k=3compute match gives0/75/250/75/25\. SkillOpt can edit only the injected skill, while the rollout remains one call; both SkillOpt CREATE gains are statistically insignificant, so HDA assigns no attribution\.The compute\-matched baseline therefore recovers a substantial share of the gains when orchestration is evolvable, and even outperforms the evolved Meta\-Harness Qwen3\.6\-35B\-A3B harness\. SkillOpt provides the complementary case: restricting evolution to the skill prevents additional calls, but neither evaluated setting produces a significant gain\. Full harnesses are included in the supplementary materials\.

## Appendix GSWE\-bench Verified: Mixed Gains and Reusable Guardrails

SWE\-bench Verified\([Jimenez et al\., 2024](https://arxiv.org/html/2610.22466#bib.bib11)\)shows the clearest residual improvements in our study, but its gains can still combine overfitting and test\-time scaling\. Figure[12](https://arxiv.org/html/2610.22466#A7.F12)contrasts two Meta\-Harness examples\. For Qwen3\-4B\-Instruct\-2507, the evolved harness combines general edit guardrails with a patch\-size rule fitted to search\-set trajectories\. It also uses1\.44×1\.44\\timesas many LLM calls as the baseline, so HDA attributes its gain asO/T/G=23/23/54O/T/G=23/23/54\. The Qwen3\-30B\-A3B\-Instruct\-2507 harness instead detects a general parser failure and hasO/T/G=0/6/94O/T/G=0/6/94\.

Meta\-Harness⋅\\cdotSWE\-bench⋅\\cdotQwen3\-4B: generic edit feedback plus a benchmark\-fitted patch envelope[⬇](data:text/plain;base64,IyBHZW5lcmFsIGd1YXJkcmFpbDogbWFrZSBmYWlsZWQgZWRpdHMgdmlzaWJsZS4KX05PT1BfTElORVMgPSBbCiAgICAiVGhpcyBjb21tYW5kIGRpZCBOT1QgbW9kaWZ5IGFueSBmaWxlLiIsCiAgICAiRG8gTk9UIGFzc3VtZSB0aGUgZWRpdCB3b3JrZWQgYW5kIGRvIE5PVCBzdWJtaXQuIiwKXQoKIyBQYXRjaC1zaXplIHRocmVzaG9sZHMgaW5mZXJyZWQgZnJvbSByZXNvbHZlZCBhbmQgZmFpbGVkIHNlYXJjaCB0cmFqZWN0b3JpZXMuCigqQFxobG92ZXJ7QUREXF9ULCBERUxcX1QsIERVUFxfVCA9IDEyLCA4LCAzfUAqKQppZiAoKkBcaGxvdmVye2FkZHMgPj0gQUREXF9UIG9yIGRlbHMgPj0gREVMXF9UIG9yIG1heGR1cCA+PSBEVVBcX1R9QCopOgogICAgd2FybigoKkBcaGxvdmVyeyJZb3VyIGFjY3VtdWxhdGVkIHBhdGNoIGlzIExBUkdFIGZvciBhIGJ1ZyBmaXguIn1AKikpCiAgICB3YXJuKCgqQFxobG92ZXJ7IlJlc29sdmVkIFNXRS1iZW5jaCBmaXhlcyBhcmUgYWxtb3N0IGFsd2F5cyBhIEZFVyBsaW5lcy4ifUAqKSkKICAgIHdhcm4oKCpAXGhsb3ZlcnsiVHJpbSB0aGUgcGF0Y2ggYmVmb3JlIHN1Ym1pdHRpbmcuIn1AKikp)\#Generalguardrail:makefailededitsvisible\.\_NOOP\_LINES=\["ThiscommanddidNOTmodifyanyfile\.","DoNOTassumetheeditworkedanddoNOTsubmit\.",\]\#Patch\-sizethresholdsinferredfromresolvedandfailedsearchtrajectories\.ADD\_T, DEL\_T, DUP\_T = 12, 8, 3ifadds \>= ADD\_T or dels \>= DEL\_T or maxdup \>= DUP\_T:warn\("Your accumulated patch is LARGE for a bug fix\."\)warn\("Resolved SWE\-bench fixes are almost always a FEW lines\."\)warn\("Trim the patch before submitting\."\)

Meta\-Harness⋅\\cdotSWE\-bench⋅\\cdotQwen3\-30B\-A3B: content\-independent action feedback[⬇](data:text/plain;base64,ZGVmIF9jb3VudF9hY3Rpb25fYmxvY2tzKHJlc3BvbnNlKToKICAgIGVkaXRzID0gbGVuKHJlLmZpbmRhbGwociJgYGBlZGl0XGIiLCByZXNwb25zZSkpCiAgICBjb21tYW5kcyA9IGxlbihyZS5maW5kYWxsKHIiYGBgKD86YmFzaHxzaHxtc3dlYV9iYXNoX2NvbW1hbmQpXGIiLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICByZXNwb25zZSkpCiAgICByZXR1cm4gZWRpdHMgKyBjb21tYW5kcwoKIyBUaGUgcGFyc2VyIGV4ZWN1dGVzIG9ubHkgdGhlIGZpcnN0IGFjdGlvbiBibG9jay4KaWYgc2VsZi5kcm9wcGVkX2FjdGlvbnMgPiAwOgogICAgb2JzZXJ2YXRpb24gKz0gKAogICAgICAgICJPbmx5IHRoZSBmaXJzdCBhY3Rpb24gYmxvY2sgd2FzIGV4ZWN1dGVkLiAiCiAgICAgICAgZiJ7c2VsZi5kcm9wcGVkX2FjdGlvbnN9IGJsb2NrKHMpIHdlcmUgbm90IHJ1bi4gIgogICAgICAgICJSZS1pc3N1ZSBlYWNoIHJlcXVpcmVkIGFjdGlvbiBhbmQgaW5zcGVjdCBpdHMgYWN0dWFsIG91dHB1dC4iCiAgICAp)def\_count\_action\_blocks\(response\):edits=len\(re\.findall\(r"‘‘‘edit\\b",response\)\)commands=len\(re\.findall\(r"‘‘‘\(?:bash\|sh\|mswea\_bash\_command\)\\b",response\)\)returnedits\+commands\#Theparserexecutesonlythefirstactionblock\.ifself\.dropped\_actions\>0:observation\+=\("Onlythefirstactionblockwasexecuted\."f"\{self\.dropped\_actions\}block\(s\)werenotrun\.""Re\-issueeachrequiredactionandinspectitsactualoutput\."\)

Figure 12:Examples of mixed and residual\-dominated gains on SWE\-bench Verified\.Redmarks the Qwen3\-4B harness’s benchmark\-derived patch\-size heuristic: the\+12/−8\+12/\-8\-line and repetition thresholds encode the search set’s tendency toward small fixes and may reject valid larger patches elsewhere\. Disabling only this mechanism accounts forO=23%O=23\\%\. Matching the evolved harness’s higher LLM inference cost raises the baseline from pass@2 to pass@3 and accounts forT=23%T=23\\%; the remainingG=54%G=54\\%comes from mechanisms such as edit\-effect feedback\. The Qwen3\-30B\-A3B action\-block guardrail uses no benchmark\-specific content, yieldingO/T/G=0/6/94O/T/G=0/6/94\.Thus, SWE\-bench contains more reusable harness improvements than the other benchmarks, but the source remains setting\-dependent\. The weaker Meta\-Harness executors transfer positively, whereas Qwen3\.6\-27B and Qwen3\-Coder\-30B\-A3B regress on the held\-out test set despite0/0/1000/0/100search\-set attributions\. This confirms thatGGis an upper bound rather than a guarantee of transfer\. Full harnesses and additional settings are included in the supplementary materials\.