@NFTCPS: HarnessX这玩意儿挺有意思：一个能自己改自己的智能体架构。以前架构怎么变，全靠人手调。新模型一出，Anthropic就把Claude Code里的规划步骤砍了，Manus半年重构了五次智能体，每次都在做减法。改什么、什么时候改，一…

X AI KOLs Timeline 2026/06/17 05:16 论文

self-evolving agent-harness reinforcement-learning composition adaptive harnessx

摘要

HarnessX introduces a framework for self-evolving AI agent harnesses that treats the runtime harness as a first-class object, enabling automatic adaptation via trace-driven reinforcement learning. It achieves average gains of +14.5% across five benchmarks, with larger improvements for weaker models.

HarnessX这玩意儿挺有意思：一个能自己改自己的智能体架构。以前架构怎么变，全靠人手调。新模型一出，Anthropic就把Claude Code里的规划步骤砍了，Manus半年重构了五次智能体，每次都在做减法。改什么、什么时候改，一直是人在拍板。 HarnessX想干的是让系统自己迭代。思路是把架构当成跟模型权重一样的"一等公民"，能类型化、能编辑，就能照着执行轨迹去优化，整套逻辑直接映射到强化学习上。最有意思的结果：弱模型提升最大，强模型几乎没动。权重没变，但周围环境变聪明了。 https://arxiv.org/abs/2606.14249

查看原文

查看缓存全文

缓存时间: 2026/06/17 07:51

HarnessX这玩意儿挺有意思：一个能自己改自己的智能体架构。

以前架构怎么变，全靠人手调。新模型一出，Anthropic就把Claude Code里的规划步骤砍了，Manus半年重构了五次智能体，每次都在做减法。改什么、什么时候改，一直是人在拍板。

HarnessX想干的是让系统自己迭代。思路是把架构当成跟模型权重一样的“一等公民“，能类型化、能编辑，就能照着执行轨迹去优化，整套逻辑直接映射到强化学习上。

最有意思的结果：弱模型提升最大，强模型几乎没动。权重没变，但周围环境变聪明了。

https://arxiv.org/abs/2606.14249

HarnessX: A Composable, Adaptive, and Evolvable Agent Harness Foundry

Source: https://arxiv.org/html/2606.14249 \contribution

SeeContributions and Acknowledgmentssection for a full author list.

Abstract

AI agent performance depends critically on the runtime harness, comprising the prompts, tools, memory, and control flow that mediate how a model observes, reasons, and acts. Yet today’s harnesses remain largely hand-crafted and static: each new model or task still demands bespoke scaffolding, and the rich traces produced during execution are rarely distilled back into systematic improvement. We introduceHarnessX, a foundry for composable, adaptive, and evolvable agent harnesses. HarnessX assembles typed harness primitives via a substitution algebra, adapts them through AEGIS, a trace-driven multi-agent evolution engine grounded in an operational mirror between symbolic adaptation and reinforcement learning, and closes the harness–model loop by turning trajectories into both harness updates and model training signal. Across five benchmarks (ALFWorld, GAIA, WebShop,τ3\tau^{3}-Bench, and SWE-bench Verified), HarnessX yields an average gain of +14.5% (up to +44.0%), with gains largest where baselines are lowest. These results suggest that agent progress need not come from model scaling alone: composing and evolving runtime interfaces from execution feedback is an actionable and complementary lever. The complete codebase will be open-sourced in a future release.

[Uncaptioned image]

Figure 1:HarnessXoverview.###### Contents

1Introduction

The capacity of modern agents depends not only on the underlying model[deepseekai2026deepseekv4,glm5team2026glm5vibecodingagentic,yang2025qwen3,team2023gemini], but on the mediation imposed by the surroundingharness[lu2026openclaw,liagent,claudecode]. This harness converts raw model outputs into structured agent behaviors by determining how tasks are represented, how external services are accessed, and how intermediate decisions are communicated during execution. As agents tackle longer-horizon tasks in richer environments, harness design becomes integral to agent development.

Despite this importance, harness development remains far from a mature engineering discipline.First, harnesses are hand-engineered and static: any change in model version, tooling, or problem domain requires bespoke modification, with no mechanism for experience-driven improvement.Second, harnesses are architecturally entangled: they typically combine prompt templates, tool wrappers, retry policies, and memory in the same codepaths, so changes to one component silently break others, and reuse across domains reduces to copying rather than composition.Third, harness engineering and model training operate independently: trajectory data collected while improving the harness is discarded rather than incorporated into model training, and model improvements do not translate into harness improvements.

We address these gaps by treating the harness as afirst-class objectthat can be composed, adapted, and evolved alongside the model. HarnessX embodies this principle as a unified harness foundry. It begins with a modular foundation: harness primitives spanning context, tools[feng2025retool], skills, control, and memory are described via typed interfaces and composed via a substitution algebra. This separates concerns that existing systems typically conflate. On top of this substrate, we introduce AEGIS, an observability-driven and auditable harness adaptation engine. Framing harness adaptation not as ad-hoc editing but as a learning problem over symbolic artifacts (prompts[zhou2025proposer], tools, memory, and control policies) reveals that standard RL pathologies (reward hacking, catastrophic forgetting[kirkpatrick2017overcoming], under-exploration[ladosz2022exploration]) become concrete design risks. To address these risks, AEGIS combines full trace observability with a four-stage pipeline (Digester, Planner, Evolver, and Critic) that compresses traces, plans adaptations, generates candidates, and assesses changes. Finally, we close the loop between harness adaptation and model training viaharness-model co-evolution. Traces produced during harness adaptation serve as reinforcement-learning signal for model training, so that model improvements feed back into subsequent harness evolution.

We empirically validateHarnessXacross five benchmarks (GAIA, ALFWorld, WebShop,τ3\tau^{3}-Bench, SWE-bench Verified), three task-agent families (Claude Sonnet 4.6, GPT-5.4, Qwen3.5-9B), and up to 15 evolution rounds. Harness evolution yields an average absolute gain of +14.5% across 15 model–benchmark configurations, with individual gains ranging from 0.0% to +44.0% among improving configurations (14 of 15), from +1.1% (τ3\tau^{3}-Bench, near-ceiling baseline) to +44.0% (ALFWorld, weakest agent). Gains exhibit an inverse-scaling pattern: on ALFWorld and GAIA, the weakest task agent benefits most (+44.0% for Qwen3.5-9B vs. +11.2% for Sonnet 4.6 on ALFWorld), suggesting that evolved harnesses address behavioral gaps that weaker models cannot self-correct. On heterogeneous task sets (GAIA), single-harness evolution stagnates; a variant-isolation ablation restores stable improvement (+13.6%, non-degrading over 15 rounds).

In summary, our contributions are four-fold:

•Harness Composition(Section3). We formalize the harness as a first-class, typed object composed of processors attached to lifecycle hooks. A nine-dimensional taxonomy spans the full behavioral space, and a substitution algebra enables per-task configuration with type-safe insertion and removal. This compositional structure makes the intended scope of each behavioral change explicit—a precondition for the variant isolation that stabilizes evolution.
•Harness Adaptation(Section4). We introduce AEGIS, a trace-driven, multi-agent harness evolution engine. An operational mirror maps harness adaptation onto standard RL constructs, converting familiar RL pathologies (reward hacking, catastrophic forgetting, under-exploration) into concrete design risks addressed by a four-stage pipeline (Digester, Planner, Evolver, Critic) with deterministic gating. An optional variant-isolation strategy prevents cross-task interference on heterogeneous benchmarks.
•Harness-Model Co-Evolution(Section5). We close the optimization loop by interleaving harness evolution with model reinforcement learning over a shared replay buffer. Cross-harness GRPO enables the model to internalize strategies from successive harness versions, breaking the scaffolding ceiling that limits harness-only adaptation and the training-signal ceiling that limits model-only RL.
•Empirical Validation(Section6). Across five benchmarks, three task-agent families, and up to 15 evolution rounds, HarnessX yields an average gain of +14.5% (up to +44.0%), with gains largest where baselines are lowest. A variant-isolation ablation resolves stagnation on heterogeneous task sets, and co-evolution yields an additional +4.7% over harness-only evolution (Section6.5).

2Related Work

2.1Harness Engineering

Existing agent infrastructure occupies a spectrum of increasingly opinionated harness abstractions. At the primitive layer, libraries such asLangChain[langchain],LlamaIndex[Liu_LlamaIndex_2022], andSmolagents[smolagents]provide typed building blocks for prompts, tools, retrieval, and memory. These primitives can be tested in isolation but do not support harness-level composition: two harnesses built from identical primitives may still differ in structure.

The next level of abstraction orchestrates these primitives into reusable patterns.LangGraph[langgraph]models the behavior of an agent with a stateful graph;AutoGen[wu2024autogen]models multi-agent interaction as structured conversation;CrewAI[moura2025crewai]assigns role-based identities to agents; andLetta[packer2023memgpt]couples autonomous loops with persistent memory. Although these frameworks make harness writing easier, they impose a particular control loop, so combining patterns, replacing components, and porting enhancements across tasks mostly remain manual.

Lastly, there are productized, domain-specific harnesses such asClaude Code[claudecode],Cursor[cursor],Manus[shen2025mind], andDeerFlow[deerflow]. These systems demonstrate the impact of harness design but remain architecturally static, evolving only through manual iteration.

Two structural gaps persist across all three layers. First, no layer exposes the harness as a substitutable entity composed of typed elements, so building a per-task harness always involves rewriting. Second, no mechanism exists for in-loop improvement: once defined, a harness evolves only through human iteration between releases.

Concurrently, Claude Code introducedDynamic Workflows[anthropic2026dynamicworkflows], enabling the model to generate task-specific harness scripts at runtime. While this represents a step toward adaptive harnesses, it operates within a single session without persistent trace-based optimization, cross-session evolution, or harness–model co-training. HarnessX addresses both gaps by treating harness adaptation as a multi-round, trace-driven learning problem with typed composition for variant isolation, structured observability for pathology detection, and a shared replay buffer that closes the loop between harness evolution and model training.

2.2Self-Evolving Agents

Research on self-evolving agents investigates how an agent system can improve without retraining the underlying foundation model. Early work focused on the single most easily editable aspect: the prompt. Approaches likeAPE[zhou2022large],OPRO[yang2024large],EvoPrompt[guo2024connecting],Promptbreeder[fernando2024promptbreeder]treat instruction formulation as a black-box optimization problem, whileProTeGi[pryzant2023automatic]andTextGrad[yuksekgonul2024textgrad]introduce gradient-inspired textual feedback to make the optimization process explicit.DSPy[khattab2023dspy]andMIPRO[opsahl2024optimizing]extend this approach by compiling a declarative LM program, whose prompts are optimized against labeled data. These approaches establish instructions as a learnable component, but harness-level features (tools, memory, control flow) remain outside the optimization scope.

Another line of work improves agents by accumulating and reusing prior execution experience in memory:Memento[zhou2025memento]improves agents through case-based memory without fine-tuning the model, whileMIA[qiao2026memory]unifies non-parametric and parametric memory within a single Manager-Planner-Executor framework: a non-parametric store of compressed trajectories and a parametric planner that evolves on the fly at test time, coupled by a bidirectional loop that continually converts experience between the two, demonstrating superiority across eleven benchmarks.

Subsequent works extend optimization to agent workflows.GPTSwarm[zhuge2024gptswarm],ADAS[hu2025automated],AFlow[zhang2025aflow],A2Flow[zhao2026a2flow],AgentSwift[li2026agentswift],ResMAS[zhou2026resmas], andEvoAgentX[wang2025evoagentx]search over collaboration strategies, agent ordering, and aggregation mechanisms. These works demonstrate that workflow structure is learnable and yields larger gains than prompt-only optimization. However, component-level artifacts (tool implementations, memory policies, node-internal prompts) remain static: the optimization scope covers inter-component relations but does not encompass the full harness.

A final group treats harness evolution explicitly.SICA[robeyns2025self]optimizes a SWE-bench agent’s source code directly, whileDarwin Gödel Machine[lange2025darwin]proposes open-ended optimization over a database of agent variants.HyperAgents[zhang2026hyperagents]makes the optimization process itself adaptable;Meta-Harness[lee2026meta]improves sampling efficiency via a file-system-based interface.AHE[lin2026agentic]andLife-Harness[xu2026adapting]emphasize observability, explainability, and source-code rewriting. Collectively, these works establish the harness as an evolutionary target and demonstrate that observability is essential for stable self-improvement. However, their designs lack a unifying theoretical framework that connects observed failure modes to principled defenses.

The heuristic-learning theory[weng2026learning_beyond_gradients]partially addresses this gap by mapping RL concepts to symbolic self-optimization updates. In this framework, observable traces correspond to proper credit assignment, falsifiable change manifests correspond to reward shaping, and proposal-critique cycles provide structured exploration. HarnessX instantiates this paradigm, formalizing the correspondence as theoperational mirrorbetween RL and symbolic harness evolution (Section4.1).

3Harness Composition

The gap identified in Section2.1is the absence of an infrastructure layer that exposes the harness as a typed, substitutable entity. Primitive libraries leave composition to application code, orchestrators expose a fixed set of patterns, and product harnesses are opaque end-to-end. Without a compositional substrate, every behavioral change or cross-team handoff requires re-implementation. HarnessX addresses this via a unified design principle: the harness is a first-class value, the processor is a typed atomic component, and composition proceeds via processor insertion at typed hook points. We formalize the harness (Section3.1), its building block, the processor (Section3.2), and the nine-dimensional processor taxonomy (Section3.3). Definitions are intentionally concise: their role is to establish the vocabulary and expose the edit surface on which harness evolution (Section4) operates.

Table 1:Hook points and their permitted modifications.### 3.1The Harness as a First-Class Object

A harness in HarnessX is the pairℋ=(ℳ,𝒞)\mathcal{H}=(\mathcal{M},\mathcal{C}), whereℳ\mathcal{M}is a model configuration and𝒞\mathcal{C}is a harness configuration. The two address disjoint concerns:ℳ\mathcal{M}recordswhichmodel serves which role (main,judge,evaluator) and the fallback policy for each role;𝒞\mathcal{C}recordshowthe agent behaves independently of model identity. They combine into an executable agent viaagent = model_config.agentic(harness_config): an agent in HarnessX is a processor pipeline bound to a model, both independently substitutable.

The harness configuration itself decomposes as𝒞=(𝐏,𝐒)\mathcal{C}=(\mathbf{P},\mathbf{S}).𝐏:ℋ𝑜𝑜𝑘→List[𝑃𝑟𝑜𝑐𝑒𝑠𝑠𝑜𝑟]\mathbf{P}:\mathcal{H}\!\mathit{ook}\to\mathrm{List}[\mathit{Processor}]is a hook-indexed list of processors, whereℋ𝑜𝑜𝑘\mathcal{H}\!\mathit{ook}is the eight-element set of lifecycle events in Table1.𝐒\mathbf{S}is a fixed set of orthogonalslot resources: tool registry, tracer, workspace, sandbox provider, and plugin list. Slots are singletons, shared across all processors in a configuration; processor state is instance-private.𝐏\mathbf{P}implements all per-step behavior;𝐒\mathbf{S}houses the shared infrastructure that processors depend on but do not own.

We call𝒞\mathcal{C}a first-class object because it is independently serializable, comparable, hashable, and substitutable. Two agents sharing𝒞\mathcal{C}but differing inℳ\mathcal{M}execute the same processor pipeline, with behavior differing only in model responses; two agents sharingℳ\mathcal{M}but differing in𝒞\mathcal{C}are behaviorally distinct. This reification is the precondition for programmatic evolution (Section4).

3.2The Processor Abstraction

Every per-step behavior in HarnessX is implemented as aprocessor, an object satisfying the protocolasync def process(self, event: Event) -> AsyncIterator[Event]. A processor consumes one event and yields zero or more, producing exactly one of five outcomes: pass-through (yield unchanged), transform (yield modified), split (yield multiple same-type events, processed independently downstream), intercept (yield nothing, blocking propagation), or interrupt (raise an exception, which halts the loop). This restricted interface enables compositionality: every processor at a given hook consumes and yields the same event type, so processors compose by sequential application and can be inserted or removed without affecting type correctness of the surrounding pipeline.

As listed in Table1, processors attach to one of eight hook points emitted by the run loop. The run loop validates hook contracts after each invocation: a violation (e.g., modifying a read-only field) raises an exception immediately rather than silently propagating corrupted state. Each processor carries three class-level metadata fields that govern composition:_singleton_groupnames a mutual-exclusion class, ensuring at most one processor per group;_orderis an ordering hint within a hook (with constantsPRE,NORMAL,POST); and_afteris a list of soft dependencies on other singleton groups.

This design makes harness evolution a first-class operation: AEGIS can insert a new processor at a specific hook, replace an existing one by matching its singleton group, or remove a processor entirely—all without touching other processors at the same or different hooks. Because the type contract (input event type==output event type) is enforced per-hook, any such substitution preserves the well-typedness of the overall pipeline. The metadata fields further constrain composition:_singleton_groupprevents conflicting duplicates, and_orderensures that newly inserted processors interact predictably with existing ones. These guarantees are the mechanism by which variant isolation (Section4.5) operates—each variant differs only in which processors occupy which hooks, and the type system ensures that no variant can silently violate the pipeline contract during evolution.

3.3The Nine-Dimensional Taxonomy

We organize the behavioral space along nine dimensions:model selection(D1) decides which model serves which role;context assembly(D2) determines what is presented to the model at each step;memory management(D3) governs what carries across steps and sessions;tool ecosystem(D4) controls which tools the agent can invoke;execution environment(D5) determines where tool-induced side-effects materialize;evaluation and reward(D6) specifies how outcomes are judged;control and safety(D7) enforces rules that keep the agent from looping, overspending, or drifting from intent;observability(D8) records each event, model call, and tool invocation; and thetraining bridge(D9) converts execution trajectories into reinforcement-learning records. Figure2illustrates the full taxonomy along with representative processors and the hooks at which they typically attach in a standard configuration.

In practice, AEGIS edits span all nine dimensions during evolution: D2 (context assembly) and D4 (tool ecosystem) are the most frequent edit targets (Section6.2), while D8 (observability) provides the trace substrate on which AEGIS itself reasons, and D9 (training bridge) supplies trajectory records for co-evolution (Section5), closing the optimization loop.

Table 2:Operational mirror: RL concepts and their symbolic-space duals in AEGIS.

4Harness Adaptation

The composition layer (Section3) provides a typed, substitutable harness; as illustrated in Figure2, AEGIS is the system that evolves it. The key insight is that harness evolution maps structurally onto reinforcement learning in a symbolic space: harness configurations are states, typed edits are actions, and execution traces plus verifier scores constitute feedback. This mapping is predictive: it identifies three failure modes analogous to known RL pathologies (reward hacking, catastrophic forgetting, under-exploration) that motivate AEGIS’s architectural defenses and are empirically confirmed in Section6.6.

We formalize the correspondence (Section4.1), analyze the pathologies it predicts (Section4.2), derive the four-stage pipeline as a defense architecture (Section4.3), present the adaptation loop (Section4.4), and introduce variant isolation for stable multi-variant evolution (Section4.5).

Refer to caption Figure 2:The AEGIS evolution loop. A single meta-agentℳ\mathcal{M}drives all four stages (Digester, Planner, Evolver, Critic), selectively invoking each based on whether sufficient signal exists to continue. A deterministic gate ships or rejects the candidate edit.### 4.1The Operational Mirror

We formalize harness evolution as an MDP over symbolic artifacts. Table2summarizes the mapping; we first state three definitions that ground the correspondence.

Definition 1 (Harness Configuration).

A harness configuration is a tupleℋ=(c1,c2,…,c9)\mathcal{H}=(c_{1},c_{2},\ldots,c_{9}), where eachci∈𝒞ic_{i}\in\mathcal{C}_{i}instantiates one of the nine behavioral dimensions (Section3.3): model selection (c1c_{1}), context assembly (c2c_{2}), memory management (c3c_{3}), tool ecosystem (c4c_{4}), execution environment (c5c_{5}), evaluation and reward (c6c_{6}), control and safety (c7c_{7}), observability (c8c_{8}), and training bridge (c9c_{9}). Each𝒞i\mathcal{C}_{i}is the set of valid processor configurations for dimensionii, constrained by hook-type contracts and singleton-group exclusion (Section3.2).

Definition 2 (Harness Edit).

A harness edit is a functione:ℋ→ℋe:\mathcal{H}\to\mathcal{H}that modifies one or more dimensions while preserving type contracts. The action spaceℰ\mathcal{E}isdiscrete but open-ended: each edit is a code-level artifact (new processor source, modified prompt template, reconfigured tool registry, or control-flow rewrite) generated by the meta-agent LLM, not selected from a pre-enumerated set. Combinatorial explosion is managed not by exhaustive search but by the LLM’s generative capacity—the Planner proposes edits from trace-grounded hypotheses—and by type constraints that prune invalid compositions at generation time.

Definition 3 (Operational Mirror).

The operational mirror is the tuple(ℋ,ℰ,ℛ,𝒯)(\mathcal{H},\mathcal{E},\mathcal{R},\mathcal{T}), whereℋ\mathcal{H}is the harness-configuration space (states),ℰ\mathcal{E}is the code-level edit space (actions),ℛ:ℋ×ℰ→ℝ\mathcal{R}:\mathcal{H}\times\mathcal{E}\to\mathbb{R}maps a configuration–edit pair to a scalar reward (verifier scores aggregated over an adaptation batch), and𝒯\mathcal{T}is the trace store that provides structured feedback beyond the scalar signal. This tuple forms an MDP at the harness level: harness configurations are states, typed edits are actions, execution traces plus verifier scores constitute feedback, and a deterministic acceptance gate governs state transitions.

MDP instantiation.

Letℋt\mathcal{H}_{t}denote the harness configuration at iterationtt(the modelℳ\mathcal{M}is fixed throughout evolution), and let𝒯t\mathcal{T}_{t}denote the trace store accumulated from all previous executions. We define the symbolic state asst=(ℋt,𝒯t)s_{t}=(\mathcal{H}_{t},\mathcal{T}_{t}). A harness-update policyπevo\pi_{\mathrm{evo}}selects an actionat∼πevo(⋅∣st)a_{t}\sim\pi_{\mathrm{evo}}(\cdot\mid s_{t}), whereat∈ℰa_{t}\in\mathcal{E}is a code-level edit drawn from the builder algebra. Applying this edit yields a candidate harnessℋ~t=at(ℋt)\widetilde{\mathcal{H}}_{t}=a_{t}(\mathcal{H}_{t}). Running the candidate on an adaptation batch (with the fixed modelℳ\mathcal{M}) produces new tracesΔ𝒯t\Delta\mathcal{T}_{t}and per-task verifier scoresrtr_{t}. A deterministic acceptance operatorU(ℋ~t,𝒯t,rt)U(\widetilde{\mathcal{H}}_{t},\mathcal{T}_{t},r_{t})then either commits the candidate (ℋt+1=ℋ~t\mathcal{H}_{t+1}=\widetilde{\mathcal{H}}_{t}) or rejects it (ℋt+1=ℋt\mathcal{H}_{t+1}=\mathcal{H}_{t}), enforcing the seesaw constraint: the candidate must not regress any previously solved task recorded in𝒯t\mathcal{T}_{t}. In both cases, the trace store grows:𝒯t+1=𝒯t∪Δ𝒯t\mathcal{T}_{t+1}=\mathcal{T}_{t}\cup\Delta\mathcal{T}_{t}.

This MDP operates at the harness level: within a single task,ℋt\mathcal{H}_{t}(together with the fixedℳ\mathcal{M}) determines the agent’s behavior; across iterations, the harness-update policyπevo\pi_{\mathrm{evo}}modifies the harness. AEGIS realizesπevo\pi_{\mathrm{evo}}as a four-stage pipeline (Digester, Planner, Evolver, Critic) that mapssts_{t}to candidate edits through trace compression, adaptation planning, edit generation, and candidate assessment.

4.2Pathologies in Symbolic Space

The mirror is not merely an analogy; it converts reinforcement-learning concepts into design requirements. We refer to three well-documented failure modes in RL, namely reward hacking[guo2025deepseek], catastrophic forgetting[kirkpatrick2017overcoming], and under-exploration[ladosz2022exploration], collectively asRL pathologies. Once harness adaptation is cast as an MDP over symbolic artifacts, these pathologies reappear in amplified form, shaped by two properties of the symbolic setting: (1) a language-model evolver can construct structured exploits that numerical parameter perturbations cannot express, and (2) edits to shared components propagate non-locally through the harness. Each pathology below motivates a corresponding architectural defense in Section4.3.

Reward hacking.

In standard RL, reward hacking[guo2025deepseek]exploits loopholes in the reward signal without genuine task completion. Symbolic harness evolution amplifies this risk because the evolver can target the verification protocol directly: embedding benchmark answers into prompts, exploiting format regularities in the verifier, or introducing a processor that rewrites outputs to match verifier expectations.

Catastrophic forgetting.

Catastrophic forgetting[kirkpatrick2017overcoming]occurs when improving performance on one region of the task distribution harms another. In symbolic harness evolution, an edit that repairs failure patternAAcan silently regress patternBB, because effects propagate through shared context, tools, memory policies, and control rules. Without explicit regression checking, an evolver conditioned only on failing-task traces cannot distinguish local gain from global regression.

Under-exploration.

Under-exploration[ladosz2022exploration]manifests as a bias toward low-risk local edits: prompt rephrasing, tool-description tuning, or minor control-flow tweaks. These edits are cheap to generate and frequently pass gating without regressing solved tasks, biasing subsequent Planner hypotheses toward the same edit neighborhood. Structural changes (decomposing one agent into several, replacing the control strategy, or adopting a new memory architecture) require deliberate hypothesis formation and rarely emerge from trace-conditional local repair. Without a mechanism to propose edits beyond the immediate failure neighborhood, the system plateaus once local edits are exhausted.

Summary.

Symbolic harness evolution inherits the structural risks of RL (reward hacking, catastrophic forgetting, and under-exploration), and AEGIS addresses each with a dedicated mechanism: the Critic (reward hacking), the deterministic gating layer (catastrophic forgetting), and the Planner (under-exploration).

Input:Initial harnessℋ0\mathcal{H}_{0}, meta-agentℳ\mathcal{M}, budgetTT, patiencePP, thresholdα\alpha

Output:Evolved harness

ℋt+1\mathcal{H}_{t+1}, trace store

𝒯t+1\mathcal{T}_{t+1} 1

𝒯0←∅\mathcal{T}_{0}\leftarrow\emptyset;

𝑖𝑑𝑙𝑒←0\mathit{idle}\leftarrow 0;

3fort=0,1,…,T−1t=0,1,\ldots,T{-}1do

4Sample batch

BtB_{t};

5run

ℋt\mathcal{H}_{t}on

BtB_{t}to get traces

Δ𝒯t\Delta\mathcal{T}_{t};

𝒯t+1←𝒯t∪Δ𝒯t\mathcal{T}_{t+1}\leftarrow\mathcal{T}_{t}\cup\Delta\mathcal{T}_{t};

/*Digester (selective) */

(𝑒𝑣𝑖𝑑𝑒𝑛𝑐𝑒t,at)←ℳ.Digester(Δ𝒯t,𝒯t)(\mathit{evidence}_{t},\;a_{t})\leftarrow\mathcal{M}.\textsc{Digester}(\Delta\mathcal{T}_{t},\;\mathcal{T}_{t});

8ifat<αa_{t}<\alphathen

ℋt+1←ℋt\mathcal{H}_{t+1}\leftarrow\mathcal{H}_{t};

𝑖𝑑𝑙𝑒++\mathit{idle}{+}{+};

10continue;

/*Planner (selective) */

𝑙𝑎𝑛𝑑𝑠𝑐𝑎𝑝𝑒t←ℳ.Planner(𝑒𝑣𝑖𝑑𝑒𝑛𝑐𝑒t)\mathit{landscape}_{t}\leftarrow\mathcal{M}.\textsc{Planner}(\mathit{evidence}_{t});

12if𝑙𝑎𝑛𝑑𝑠𝑐𝑎𝑝𝑒t=∅\mathit{landscape}_{t}=\emptysetthen

ℋt+1←ℋt\mathcal{H}_{t+1}\leftarrow\mathcal{H}_{t};

𝑖𝑑𝑙𝑒++\mathit{idle}{+}{+};

14continue;

/*Evolver (selective) */

{(ℋ~tk,manifestk)}k=1Kt←ℳ.Evolver(ℋt,𝑙𝑎𝑛𝑑𝑠𝑐𝑎𝑝𝑒t)\{(\widetilde{\mathcal{H}}_{t}^{\,k},\,\mathrm{manifest}_{k})\}_{k=1}^{K_{t}}\leftarrow\mathcal{M}.\textsc{Evolver}(\mathcal{H}_{t},\;\mathit{landscape}_{t});

16ifKt=0K_{t}=0then

ℋt+1←ℋt\mathcal{H}_{t+1}\leftarrow\mathcal{H}_{t};

𝑖𝑑𝑙𝑒++\mathit{idle}{+}{+};

18continue;

/*Critic & Gate (mandatory) */

𝑟𝑎𝑛𝑘𝑖𝑛𝑔←ℳ.Critic({(ℋ~tk,manifestk)},𝑒𝑣𝑖𝑑𝑒𝑛𝑐𝑒t)\mathit{ranking}\leftarrow\mathcal{M}.\textsc{Critic}(\{(\widetilde{\mathcal{H}}_{t}^{\,k},\,\mathrm{manifest}_{k})\},\;\mathit{evidence}_{t});

k⋆←⊥k^{\star}\leftarrow\bot;

21foreach*kkin𝑟𝑎𝑛𝑘𝑖𝑛𝑔\mathit{ranking}*do

22ifDeterministicGate(ℋ~tk,ℋt,𝒯t)(\widetilde{\mathcal{H}}_{t}^{\,k},\,\mathcal{H}_{t},\,\mathcal{T}_{t})passesthen

k⋆←kk^{\star}\leftarrow k;

23break;

25end foreach

26ifk⋆≠⊥k^{\star}\neq\botthen

ℋt+1←ℋ~tk⋆\mathcal{H}_{t+1}\leftarrow\widetilde{\mathcal{H}}_{t}^{\,k^{\star}};

𝑖𝑑𝑙𝑒←0\mathit{idle}\leftarrow 0;

28else

ℋt+1←ℋt\mathcal{H}_{t+1}\leftarrow\mathcal{H}_{t};

𝑖𝑑𝑙𝑒++\mathit{idle}{+}{+};

30if𝑖𝑑𝑙𝑒≥P\mathit{idle}\geq Pthenbreak;

32end for

return

ℋt+1,𝒯t+1\mathcal{H}_{t+1},\;\mathcal{T}_{t+1}

Algorithm 1AEGIS Harness Evolution Loop (selective invocation)

4.3AEGIS Architecture

AEGIS is the harness-evolution engine of HarnessX. It comprises four stages arranged in a predefined workflow—Digester, Planner, Evolver, Critic—all driven by the same meta-agent LLM, whichselectively invokesthem: no external router decides stage execution; instead, the meta-agent itself determines at each stage whether sufficient signal exists to continue. The Digester, Planner, and Evolver each evaluate a continuation condition and may short-circuit the round (below-threshold actionability, empty landscape, or zero viable candidates), while the Critic together with the deterministic gating layer is mandatory for every candidate that reaches it. No edit can ship without passing through the Critic and gate. The division of labor across stages addresses the pathologies of Section4.2: the Digester compresses raw traces into structured task-level evidence; the Planner constructs an adaptation landscape spanning both incremental and structural changes; the Evolver produces typed builder edits with explicit change manifests; and the Critic, together with the gating layer, rejects edits whose claimed improvement lacks trace support or whose acceptance would regress previously solved tasks.

All stages share a single information substrate: the trace store, a structured record of execution events, verifier-scored outcomes, regression signals, and shipped or rejected edits. No stage consumes input beyond the trace store and the current harnessℋt\mathcal{H}_{t}. Data flows forward through the pipeline with selective gating: the Digester may determine that no actionable failures exist (all tasks pass or signal is too sparse), terminating the round immediately; the Planner may find no viable adaptation landscape given the current evidence and edit history; and the Evolver may produce no type-safe candidates. In each case the round exits cleanly with a no-op outcome. Only the Critic and deterministic gate are unconditional: any candidate that survives the upstream stages must pass through both before shipping. The Critic may additionally issue a single revision request to the Evolver before returning its final verdict.

Digester.

A single iteration on GAIA (103 tasks, pass@2) generates∼{\sim}10M tokens of raw traces: model reasoning steps, tool invocations with their outputs, and timing metadata. Passing this volume directly to downstream stages exceeds context limits, yet naive truncation discards diagnostic signal. The Digester compresses each task’s traces into a structured per-task summary: binary outcome, failure category (if any), implicated component identifiers, and supporting evidence excerpts. It also provides cross-iteration continuity: each task’s summary links to its history of prior outcomes and shipped edits, enabling the Planner to distinguish persistent failures from transient noise.

Planner.

The Planner receives the Digester’s output (task-level summaries enriched with cross-iteration history) and constructs an adaptation landscape: which tasks are failing, what edits have been attempted, which components are implicated, and which edit types (prompt, tool, processor, configuration) remain untried. This stage is the primary defense against under-exploration: by constructing the landscape before edit generation, it prevents the pipeline from converging on trace-conditional local repair, ensuring that structural changes (tool additions, processor rewrites, memory-policy redesigns) are considered alongside incremental prompt edits.

Evolver.

Given the Planner’s adaptation landscape, the Evolver produces one or more candidate harnesses{ℋ~tk}k=1Kt\{\widetilde{\mathcal{H}}_{t}^{\,k}\}_{k=1}^{K_{t}}, each specified as a typed builder operation on the current harnessℋt\mathcal{H}_{t}. Each candidate carries achange manifest: the edited components, the intended behavioral effect, and the tasks expected to improve or regress. When introducing new processor code, the Evolver must also provide a smoke test confirming that the processor instantiates and runs on synthetic input without raising exceptions. The builder algebra guarantees type-safety (every candidate satisfies hook-type contracts and processor-composition rules) but not behavioral safety; an edit that type-checks may still produce non-local behavioral effects, detectable only by the Critic and gating layer.

Critic and gating.

The Critic defends against reward hacking; the deterministic gating layer defends against catastrophic forgetting. The Critic evaluates each candidate by comparing its change manifest against trace evidence and assessing whether edits risk non-local effects through shared state or control flow. When gaps are detected, it issues a single revision request to the Evolver. After at most one revision cycle, the Critic returns eitherno_opor an orderedship_ranking. The deterministic gate then applies acceptance checks in sequence: manifest completeness, configuration normalization (ensuring the candidate is in canonical form), build or smoke tests (when applicable), and the seesaw constraint (regression check on previously passing tasks; Section4.1). The first failing check halts the sequence; passing candidates are committed and failing ones archived with their rejection reason. This decouples LLM judgment from acceptance: regardless of the Critic’s recommendation, only deterministic checks govern shipping.

Design principle.

Language-model subagents explore, hypothesize, and propose; typed structure and deterministic gates determine what ships. This separation ensures that safety properties (no regression, no unaudited edits) hold regardless of LLM subagent failure modes.

4.4The Adaptation Loop

Algorithm1formalizes the adaptation loop (each iteration corresponds to one “round” in Section6). Starting from an initial harnessℋ0\mathcal{H}_{0}, each iteration executes the current harness on an adaptation batch and selectively invokes the four stages: the Digester, Planner, and Evolver each gate on a continuation condition (sufficient actionability, non-empty landscape, and at least one type-safe candidate, respectively), while the Critic and deterministic gate are mandatory for any candidate that reaches them. A round commits a new harness only when a candidate clears all acceptance checks.

4.5Variant Isolation via Ensemble Routing

The adaptation loop (Section4.4) maintains a single harnessℋt\mathcal{H}_{t}. When tasks require conflicting behaviors, an edit that improves one subset may regress another; the seesaw constraint rejects it, protecting stability but discarding a locally beneficial change.Variant isolationlifts this limitation by maintaining up toKKharness variants{ℋt(1),…,ℋt(Vt)}\{\mathcal{H}_{t}^{(1)},\ldots,\mathcal{H}_{t}^{(V_{t})}\}(Vt≤KV_{t}\leq K) and routing each task to the variant with the highest estimated success rate on that task’s cluster across prior rounds. We term this mechanismEnsemble routing.

The gating layer distinguishes two outcomes per candidate: (1) the edit improves some tasks without regressing any, in which case it is applied to its target variant; or (2) it improves a subset while regressing others, in which case the system forks a new variant rather than rejecting the edit outright (retiring the lowest-performing variant if the pool is full). Once multiple variants exist, the seesaw constraint is scoped per-variant: a candidate targeting variantkkis tested only against tasks routed tokk, so improvements to one cluster cannot regress another. This design predicts three properties validated in Section6.3: (1) non-degrading aggregate trajectory (peak = final), (2) sustained exploration across more rounds, and (3) lower total token consumption.

5Harness-Model Co-Evolution

Sections3and4show that evolving the harness alone, with the foundation model held fixed, already delivers substantial gains, and that these gains are largest for weaker, smaller task agents, whose behavioral gaps a better harness most readily closes. Co-evolution does not displace that route; it extends it along a second axis. For a capability-limited small model, harness evolution eventually meets ascaffolding ceiling: once the harness exposes the right tools, context, and control flow, the binding constraint becomes whether the frozen model can actually exploit them, and no harness edit can supply reasoning capacity the model itself lacks.

Symmetrically, training the model under a fixed harness meets atraining-signal ceiling: newly acquired capabilities go unexercised when the scaffold never surfaces the context, tools, or control flow that elicit them. The model is the agent’scognitive core, supplying reasoning and planning, while the harness is itsexecutive apparatus, determining what the model perceives, what it can invoke, and what constrains its execution. A sharper apparatus cannot compensate for a weak core, nor a stronger core for an apparatus that never calls on it. Co-evolution targets precisely this bottleneck: by training the model within the same loop that evolves its harness, the agent improves along both axes simultaneously, breaking the ceiling that either improvement alone would leave in place. The principle of jointly evolving complementary capability components also appears in other settings: K2Agent[wu2026k]co-evolves know-what (declarative knowledge) and know-how (procedural skill) for hierarchical mobile device control.

Figure3illustrates the co-evolution mechanism. Rather than alternating between independent harness-evolution and model-training phases, HarnessX runs both within a single iteration over a shared replay buffer. We formalize the iteration (Section5.1), describe the two optimization substrates (Section5.2), specify the model training objective via cross-harness GRPO (Section5.3), and characterize off-policy training over the shared buffer, the property that lets model RL run at no additional rollout cost (Section5.4).

Refer to caption Figure 3:The harness-model co-evolution loop.The agent(ℳt,ℋt)(\mathcal{M}_{t},\mathcal{H}_{t})runs the task batchBtB_{t}under a fixed verifier and the observability layer; the resulting traces and rewards(τ,r)(\tau,r)enter a shared replay bufferℬ\mathcal{B}, where cross-harness grouping pools trajectories of the same task across harness versions and computes group-relative advantagesA^\hat{A}. The same buffer drives two updates over identical data: AEGIS harness evolution (Digester→\toPlanner→\toEvolver→\toCritic, yielding the evolved harnessℋt+1\mathcal{H}_{t+1}) andcross-harness GRPO(group sampling and a clipped GRPO objective, yielding the updated modelℳt+1\mathcal{M}_{t+1}); both feed the next iteration.### 5.1The Co-evolution Iteration

Co-evolution operates over the pair(ℳt,ℋt)(\mathcal{M}_{t},\mathcal{H}_{t}), whereℳt\mathcal{M}_{t}denotes trainable model parameters (relaxing the frozen-model assumption of Section4) andℋt\mathcal{H}_{t}denotes the harness configuration at iterationtt. The system maintains a fixed-capacity replay bufferℬ\mathcal{B}with first-in-first-out eviction. Each iteration proceeds as:

1.Rollout.Run(ℳt,ℋt)(\mathcal{M}_{t},\mathcal{H}_{t})on the adaptation batchBtB_{t}; the observability layer records each episode as a complete traceτi\tau_{i}, capturing every model turn, tool call, and tool result.
2.Verification.A fixed verifier scores each trace into a scalar rewardrir_{i}. Holding the verifier fixed keeps rewards comparable across harness versions, which the cross-harness advantage (Eq.3) requires.
3.Buffer insertion.Append each scored trace to the shared bufferℬ\mathcal{B}together with the harness version that produced it, so successive rounds accumulate rather than overwrite; FIFO eviction keepsℬ\mathcal{B}restricted to recent rounds.
4.Harness evolution(ℋt+1←AEGIS(ℋt,ℬ)\mathcal{H}_{t+1}\leftarrow\text{AEGIS}(\mathcal{H}_{t},\mathcal{B}), non-parametric, Section4). The meta-agent reads the buffered traces as evidence of where the scaffold fails, proposes one discrete structural edit, and admits it only if the Critic and gating layer validate it.
5.Behavior log-probabilities.For the traces just added this round, run a forward pass under the generating modelℳt\mathcal{M}_{t}to obtain the token-level log-probabilitiesπθold(τi)\pi_{\theta_{\text{old}}}(\tau_{i})and cache them for use in the GRPO loss; trajectories from earlier rounds reuse the values cached at their own insertion (Section5.4).
6.GRPO update(ℳt+1←GRPO(ℳt,ℬ)\mathcal{M}_{t+1}\leftarrow\text{GRPO}(\mathcal{M}_{t},\mathcal{B}), parametric, Section5.3). Partition traces into per-task groups spanning harness versions, assign each a group-relative advantage, and take a clipped policy-gradient step with a KL anchor to the fixed reference.
7.Advance.Return to step 1 with the evolved pair(ℳt+1,ℋt+1)(\mathcal{M}_{t+1},\mathcal{H}_{t+1}).

Every trace serves as both AEGIS diagnostic evidence and GRPO training signal. The harness evolution (step 4) and model update (steps 5–6) read the same buffer but neither conditions on the other’s output within the same iteration; both must complete before the next rollout begins.

5.2Optimization Substrates

Harness side (non-parametric optimization).

Harness evolution proceeds as in Section4, drawing on the replay bufferℬ\mathcal{B}for trace evidence. The principal difference from standalone AEGIS is thatℬ\mathcal{B}contains trajectories from multiple model checkpointsℳ0,ℳ1,…,ℳt\mathcal{M}_{0},\mathcal{M}_{1},\ldots,\mathcal{M}_{t}, exposing the Digester to behavioral variation from both model updates and harness edits.

Model side (parametric optimization via GRPO).

The key design choice is thecross-harness grouping criterion(formalized in Section5.3): all trajectories sharing a task identifier form one GRPO group regardless of which harness or model checkpoint produced them, so that within-group variation reflects strategy differences rather than sampling noise alone.

Complementarity.

The harness update makes discrete structural changes (adding a tool, replacing a control processor, restructuring the prompt) that cannot be expressed as parameter updates. The model update makes fine-grained behavioral adjustments (when to invoke which tool, how to phrase a query, when to terminate) that depend on high-dimensional in-context state and cannot be captured by symbolic specification. The harness defines coarse-grained strategy architecture; the model learns to exploit it.

5.3Model Training via Cross-Harness GRPO

We adopt Group Relative Policy Optimization (GRPO)[shao2024deepseekmath]. Formally, each trajectory in the buffer is generated as:

τi∼Agent(ℳk,ℋk,xi),k∈{0,1,…,t},\tau_{i}\sim\text{Agent}(\mathcal{M}_{k},\,\mathcal{H}_{k},\,x_{i}),\quad k\in\{0,1,\ldots,t\},(1)whereiiis the(x,τ)(x,\tau)index in the bufferℬ\mathcal{B},ℳk\mathcal{M}_{k}andℋk\mathcal{H}_{k}are the model checkpoint and harness used to roll out taskxix_{i}into trajectoryτi\tau_{i}. Because FIFO eviction bounds the buffer to recent iterations, buffered trajectories come from model versions close to the current policy. Yet they differ markedly in strategy (tool selection, prompt structure, control-flow logic), a diversity that stems from the successive harness versionsℋ0,…,ℋt\mathcal{H}_{0},\ldots,\mathcal{H}_{t}. Unlike single-policy RL, where within-group variation reduces to stochastic sampling, here harness identity dominates that variation, which makes the cross-harness grouping criterion (Eq.2) essential for meaningful advantage estimation.

Formally, for a taskxx, the trajectory group collects all traces ofxxregardless of which(ℳk,ℋk)(\mathcal{M}_{k},\mathcal{H}_{k})pair produced them:

𝒢x={τi∈ℬ∣task(τi)=x}=⋃k{τ∼Agent(ℳk,ℋk,x)}.\mathcal{G}_{x}=\{\tau_{i}\in\mathcal{B}\mid\text{task}(\tau_{i})=x\}=\bigcup_{k}\{\tau\sim\text{Agent}(\mathcal{M}_{k},\mathcal{H}_{k},x)\}.(2)The model therefore receives gradient signal from inter-strategy reward contrasts, rather than from stochastic variation within a fixed strategy alone, which enables it to internalize strategies that succeeded across harness versions.

Task-level alignment, not action-level.

Cross-harness GRPO performstask-levelalignment: trajectories from different harness versions are grouped by task identity and compared by verifier reward alone. No action-level alignment is required, so harness versions with incompatible action spaces (different tool schemas, different prompt structures, different control-flow processors) coexist in the same group without conflict. When computing the policy gradient, each trajectoryτi\tau_{i}is replayed under the harness versionℋk\mathcal{H}_{k}that produced it: the model’s log-probabilitiesπθ(τi∣x)\pi_{\theta}(\tau_{i}\mid x)are evaluated against the prompt, tool schema, and observation context thatℋk\mathcal{H}_{k}would have constructed at each turn. The GRPO gradient thus operates entirely on model output tokens conditioned on harness-specific context, rather than on harness structural actions or environment transitions. This design decouples harness evolution (which may freely alter the action space across versions) from model training (which only requires token-level log-probabilities under each trajectory’s own harness context).

The group-relative advantage is:

A^(τi)=ri−μ(𝒢x)σ(𝒢x)+ϵ,\hat{A}(\tau_{i})=\frac{r_{i}-\mu(\mathcal{G}_{x})}{\sigma(\mathcal{G}_{x})+\epsilon},(3)whererir_{i}is the reward for trajectoryτi\tau_{i}, andμ(𝒢x)\mu(\mathcal{G}_{x}),σ(𝒢x)\sigma(\mathcal{G}_{x})are the within-group reward mean and standard deviation. The evolving harness acts as a structured exploration operator for the model’s RL: each new version injects a distinct mode of behavior into the task’s sampling distribution, and the advantage in Eq.3commits the model toward whichever modes the verifier scores highest. The exploration breadth that single-policy sampling cannot provide is thus supplied by the evolving scaffold itself.

The policy objective to maximize is:

𝒥GRPO(θ)=𝔼x,τi∼ℬ[min⁡(ρi(θ)A^(τi),clip(ρi(θ),1−ϵc,1+ϵc)A^(τi))]−βDKL(πθ∥πref),\mathcal{J}_{\text{GRPO}}(\theta)=\mathbb{E}_{x,\,\tau_{i}\sim\mathcal{B}}\left[\min\!\left(\rho_{i}(\theta)\,\hat{A}(\tau_{i}),\;\text{clip}\!\left(\rho_{i}(\theta),\,1{-}\epsilon_{c},\,1{+}\epsilon_{c}\right)\hat{A}(\tau_{i})\right)\right]-\beta\,D_{\text{KL}}\!\left(\pi_{\theta}\,\|\,\pi_{\text{ref}}\right),(4)where

ρi(θ)=πθ(τi∣x)πθold(τi∣x),πθold=ℳd,\rho_{i}(\theta)=\frac{\pi_{\theta}(\tau_{i}\mid x)}{\pi_{\theta_{\text{old}}}(\tau_{i}\mid x)},\qquad\pi_{\theta_{\text{old}}}=\mathcal{M}_{d},(5)is the importance-sampling ratio between the current policyℳk\mathcal{M}_{k}and the checkpointℳd\mathcal{M}_{d}that generatedτi\tau_{i}(Eq.1),ϵc\epsilon_{c}is the clipping threshold, andβDKL(πθ∥πref)\beta\,D_{\text{KL}}(\pi_{\theta}\|\pi_{\text{ref}})penalizes divergence from the fixed reference modelπref\pi_{\text{ref}}. The behavior policyπθold\pi_{\theta_{\text{old}}}in the ratio and the reference policyπref\pi_{\text{ref}}in the KL term are distinct:πref=ℳ0\pi_{\text{ref}}=\mathcal{M}_{0}is fixed throughout training, whileπθold\pi_{\theta_{\text{old}}}varies per trajectory and must be recovered from the buffer (Section5.4).

5.4Off-Policy Training over a Mixed-Policy Buffer

The replay buffer is intrinsically off-policy: at iterationttit holds trajectories generated by checkpointsℳ0,ℳ1,…,ℳt\mathcal{M}_{0},\mathcal{M}_{1},\dots,\mathcal{M}_{t}under harnessesℋ0,ℋ1,…,ℋt\mathcal{H}_{0},\mathcal{H}_{1},\dots,\mathcal{H}_{t}(Eq.1), so the buffer distribution does not match the policyπθ\pi_{\theta}under update. Recoveringπθold\pi_{\theta_{\text{old}}}for each buffered trajectory is the central off-policy challenge.

Behavior policyπθold\pi_{\theta_{\text{old}}}.

The importance ratio (Eq.5) corrects the gap betweenπθ\pi_{\theta}and the checkpointℳk\mathcal{M}_{k}that producedτi\tau_{i}. Sinceℳk\mathcal{M}_{k}varies across the buffer,πθold(τi)\pi_{\theta_{\text{old}}}(\tau_{i})cannot be recovered from any single model: we materialize it at buffer insertion via one forward pass underℳk\mathcal{M}_{k}, cache the token-level log-probabilities on disk, and reuse them at every gradient step. This decouples the cached behavior log-probabilities from the current log-probabilitiesπθ(τi)\pi_{\theta}(\tau_{i})recomputed each step.

Bounded off-policy bias.

FIFO eviction caps the buffer atCCtrajectories; withsssamples per round the maximum model-version lag is⌊C/s⌋\lfloor C/s\rfloorrounds, so every cachedπθold\pi_{\theta_{\text{old}}}originates within a bounded window ofπθ\pi_{\theta}and the policy that generated a trajectory never differs greatly from the one being updated. The same window bounds harness staleness, so the cross-harness groups (Eq.2) mix only recent scaffold versions, and the model is never trained predominantly against an obsolete harness.

Replay reuse at no added rollout cost.

The dominant cost of agentic RL is the rollout (executing the agent in the environment: model decoding, tool calls, and verification), not the gradient update. In co-evolution a single round of exploration produces one set of trajectories that serves both updates: the same traces drive the AEGIS harness update (Section4) and, through the shared buffer (Section5.1), the cross-harness GRPO model update. GRPO consumes these trajectories by replay and issues no rollouts of its own. The marginal cost of adding the model update is therefore confined to (i) one cached forward pass per trajectory to recordπθold\pi_{\theta_{\text{old}}}and (ii) the gradient steps themselves, both of which are rollout-free. No trajectory is generated solely to train the model. Joint optimization is therefore economical: it buys model improvement for the price of offline training compute alone, without any rollouts beyond those harness evolution already performs.

6Experiments

We evaluate HarnessX along five axes: overall effectiveness across benchmarks and model families (Section6.2), the impact of variant-management strategies on stability (Section6.3), the relative contribution of evolver architecture versus infrastructure (Section6.4), gains from joint model–harness co-evolution (Section6.5), and empirical confirmation of the predicted failure modes (Section6.6).

6.1Experimental Setup

Benchmarks.

As summarized in Table3, we evaluate on five benchmarks spanning multi-step retrieval, embodied planning, web interaction, multi-turn dialogue, and software engineering. Unless otherwise noted, each experiment runs for up toT=15T{=}15evolution rounds with early stopping afterP=3P{=}3consecutive rounds without a shipped edit. The full task set is evaluated every round (no subsampling). The meta-agent token budget varies by benchmark (100M–175M total) but is held constant across task agents within a benchmark.

Table 3:Benchmark characteristics.

Models.

We distinguish two roles: themeta-agent(Claude Opus 4.6 unless otherwise noted) drives the AEGIS evolution loop; thetask agentruns under the evolved harness to solve benchmark tasks. Task agents span three families (Claude Sonnet 4.6, GPT-5.4, and Qwen3.5-9B) to test whether a single meta-agent can evolve effective harnesses across model families.

Baselines.

(1) Static Harness: a HarnessX configuration constructed from published benchmark-specific prompts and tool definitions, held fixed across all rounds.(2) Claude Code SDK (CC SDK)111Claude Code SDK v0.0.25,model=“opus”(Claude Opus 4.6),max_turns=200. Experiments conducted in May 2026.: a single-agent evolver (one LLM session per round) that replaces the four-stage pipeline while retaining the same infrastructure and round budget, isolating AEGIS’s multi-stage architecture from the shared infrastructure (Section6.4). This baseline also serves as a proxy for monolithic evolvers such as SICA[robeyns2025self].

Metrics.

Task success rate (%) under the benchmark-specific verifier. Each task receives two independent attempts per round (pass@2: solved if either succeeds), reducing sampling noise while preserving a binary per-task signal for the seesaw constraint (at the cost of masking sub-threshold success-probability drift; Section6.3).

Scope.

All reported gains are measured on the same task set used for evolution; held-out generalization to unseen tasks is not evaluated in this work.

Refer to caption Figure 4:Evolution trajectories (pass@2 success rate vs. round). Dashed lines: static-harness baselines.

6.2Main Results

Table4and Figure4report pass@2 success rates before and after harness evolution. AEGIS improves 14 of 15 model–benchmark configurations, with an average gain of +14.5% (up to +44.0%). The single stagnating configuration (GAIA, GPT-5.4,Δ=0.0\Delta{=}0.0) reflects a fundamental limitation of single-harness evolution on heterogeneous task sets; Section6.3shows that variant isolation resolves this. One configuration regressed mid-run (τ3\tau^{3}-Bench Telecom,−-14.0% at R7) due to accumulated same-type edits, recovering by R9 (Section6.6).

Table 4:Main results (pass@2 success rate, %). Evolved = peak accuracy achieved. “–” indicates domain-averaged results where no single peak round applies.Overall performance.Evolution improves 14 of 15 configurations. Gains range from +11.2% to +44.0% on ALFWorld, +13.0% to +18.0% on WebShop, and +10.9% to +18.2% on SWE-bench Verified. On GAIA, Sonnet 4.6 (+9.7%) and Qwen3.5-9B (+17.1%) improve, while GPT-5.4 stagnates (Δ=0.0\Delta{=}0.0; resolving its failures demands mutually conflicting edits that no single-harness strategy can accommodate). Onτ3\tau^{3}-Bench, GPT-5.4 gains most (+14.5%) while Qwen3.5-9B gains only +1.1% due to its near-ceiling 93.5% baseline.

Inverse scaling with baseline performance.Across benchmarks, the weakest task agent (Qwen3.5-9B) consistently gains most: +44.0% on ALFWorld (baseline 53.0%), +17.1% on GAIA (baseline 20.3%), and +18.2% on SWE-bench Verified (baseline 23.6%). Stronger models (Sonnet 4.6, GPT-5.4) gain less on ALFWorld (+11.2%, +20.9%) and SWE-bench (+10.9%, +18.2%). The exception is GAIA GPT-5.4 (Δ=0.0\Delta{=}0.0), where task heterogeneity prevents a single harness from improving aggregate accuracy—an observation that motivates the variant-isolation ablation in Section6.3. The overall pattern suggests that weaker models exhibit more behavioral gaps addressable by harness-level edits; once baseline performance is sufficiently high, remaining failures increasingly require task-specific adaptations rather than global improvements.

Cross-model generalization.The meta-agent (Opus 4.6) evolves harnesses for task agents across model families without family-specific adaptation. On ALFWorld, cross-family agents (GPT-5.4: +20.9%, Qwen3.5-9B: +44.0%) gain more than the same-family agent (Sonnet 4.6: +11.2%), indicating that gain magnitude tracks baseline performance rather than proximity to the meta-agent’s family.

Convergence rate tracks failure-mode concentration.ALFWorld (GPT-5.4) peaks at R4 and SWE-bench Verified (all agents) peaks at R2–R3; in both cases, failures concentrate in one or two component types, enabling rapid convergence. GAIA (Sonnet 4.6) requires 11 rounds because failures span four component types (prompt, tool, processor, configuration), forcing sequential exploration of multiple edit neighborhoods.

Domain-level variation withinτ3\tau^{3}-Bench.

The averagedτ3\tau^{3}-Bench gains mask substantial per-domain variation. GPT-5.4 gains +25.4% on Telecom (67.5%→\to93.0% at R2) and +9.7% on Retail (84.2%→\to93.9% at R6). However, Sonnet 4.6 on Telecom regresses−-14.0% in a single round (R7) due to accumulated same-type edits, recovering by R9 (Section6.6). This illustrates a structural limitation of per-edit gating: sub-threshold coupling from consecutive same-type edits accumulates undetected until a tipping point triggers visible regression.

Post-peak degradation on SWE-bench.

On SWE-bench Verified (GPT-5.4), evolution peaks at 63.6% (R3, +18.2%) but degrades to 50.9% by R5 (−-12.7% from peak); final accuracy still exceeds the static baseline by +5.4%. Two factors accelerate degradation on this benchmark: (1) with only 55 tasks, each task flip shifts aggregate accuracy by∼{\sim}1.8% (vs.∼{\sim}1.0% atn=103n{=}103), so fewer regressions suffice to produce visible decline; and (2) structural code edits have a broader blast radius than prompt edits. This parallels the GAIA GPT-5.4 stagnation: both cases motivate the variant-isolation strategy evaluated in Section6.3.

6.3Evolution Strategy Comparison

The main experiments (Table4) use the Global strategy: a single harness evolved across all tasks. Table5compares this default with a variant-isolation strategy on GAIA (103 tasks, GPT-5.4, 15 rounds, AEGIS evolver).

Table 5:Evolution strategy comparison (GAIA, GPT-5.4, AEGIS evolver, 15 rounds). Final−-Peak indicates stability; negative values signal catastrophic forgetting.Failure mechanism of Global.The Global strategy maintains a single harness for all 103 tasks. It peaks early at R4 (73.8%) before degrading steadily: subsequent edits introduce sub-threshold regressions that are individually undetectable under pass@2’s binary signal yet compound into aggregate decline. The peak–final gap (−-24.3%) far exceeds the per-round binomial 95% confidence interval (±\pm8.5% atn=103n{=}103,p≈0.74p{\approx}0.74), ruling out evaluation noise and confirming catastrophic forgetting (Section4.2). This explains theΔ=0.0\Delta{=}0.0stagnation for GAIA GPT-5.4 in Table4: Global cannot sustain improvement on this heterogeneous task set.

Why Ensemble prevents cross-variant forgetting.Ensemble routing maintains up toKKharness variants and routes each task to the variant with the highest prior success rate. Edits are proposed and evaluatedper-variant, so an edit improving one cluster cannot regress another. The comparison confirms three predicted properties: (1) non-degrading aggregate trajectory (peak = final), (2) later peak (R14 vs. R4), indicating sustained productive exploration, and (3) lower token consumption (107.8M vs. 143.7M), because each edit is evaluated only against its target cluster rather than the full task set, and edits target only their assigned cluster, avoiding the wasted proposals that accumulate when a degrading single harness is evaluated against all tasks.

Summary.Variant isolation resolves the stagnation observed under Global, lifting GAIA GPT-5.4 fromΔ=0.0\Delta{=}0.0to +13.6% (87.4%, non-degrading). Finer-grained strategies (Domain-aware clustering, Task-level tournament) were explored at pilot scale (30–40 tasks,≤\leq8 rounds) but lack sufficient rounds and tasks for statistically meaningful comparison.

6.4Meta-Agent Effectiveness

To disentangle evolver architecture from infrastructure, we replace the four-stage AEGIS pipeline with a single-agent CC SDK evolver that shares the same model (Opus 4.6), round budget, and infrastructure. Both evolvers run under variant isolation (introduced in Section6.3) to ensure non-degrading trajectories. Table6reports the comparison on GAIA (103 tasks, GPT-5.4, 15 rounds).

Table 6:Meta-agent architecture comparison (GAIA, GPT-5.4, variant isolation, 15 rounds). Both evolvers use Opus 4.6.Accuracy is comparable; efficiency differs.The 1.0% accuracy gap falls within one standard error (∼{\sim}3.3% atn=103n{=}103), indicating that the four-stage decomposition does not improve final accuracy at this meta-agent capability level. However, the single-agent variant consumes∼{\sim}14% more tokens (123.1M vs. 107.8M). We attribute this to the Digester’s compression: it reduces∼{\sim}10M raw trace tokens to∼{\sim}10K structured summaries before downstream stages consume them. Without this stage, the single-agent evolver must truncate traces to fit its context window, yielding less-informed edits that are rejected by the gate more frequently, wasting tokens on failed proposals.

Implication.With a capable meta-agent under variant isolation, accuracy gains derive primarily from HarnessX’s infrastructure (typed components enabling isolation, structured traces enabling diagnosis) rather than the evolver’s internal architecture. The four-stage decomposition contributes efficiency (∼{\sim}12% fewer tokens) and interpretability (auditable intermediate artifacts) but not measurable accuracy at this scale.

6.5Co-Evolution

This experiment tests whether interleaving harness evolution with model RL (Section5) yields gains beyond harness-only evolution. As shown in Figure5, we compare the two regimes on GAIA and WebShop using a Qwen3.5-9B task agent. Both conditions share a fixed-capacity FIFO replay buffer: each round runs the current agent on the adaptation batch, a fixed verifier scores the resulting traces, and both harness evolution (AEGIS) and model training (cross-harness GRPO) update over the same buffer (Section5.1). Section5predicts that each single-optimization route stalls at its own ceiling: harness-only at thescaffolding ceiling, model-RL-only at thetraining-signal ceiling. Co-evolution addresses both ceilings by enabling the model to internalize strategies that successive harness versions introduce.

Experimental setup.We run both regimes on the GAIA text-only subset (103 tasks) and a WebShop subset (100 tasks) with a Qwen3.5-9B task agent. GAIA exercises live web tools whose latency and availability fluctuate, so each round is evaluated twice and averaged. Both subsets are small, so we set the optimizer batch to the entire replay buffer and size the buffer as a four-round sliding window: at two rollouts per task this is 824 traces on GAIA (103×2×4103\times 2\times 4) and 400 on WebShop (100×1×4100\times 1\times 4), which supplies enough within-group samples for GRPO to estimate advantages stably. Training uses learning rate1×10−61\times 10^{-6}, GRPO clipϵ=0.2\epsilon=0.2, no KL penalty (coefficient0), and 5 training steps per round. The GAIA agent is equipped with web search (Baidu API), web fetch, bash, and file read; WebShop uses its environment’s built-in action tools. Rewards are0.9×0.9\timescorrectness plus0.1×0.1\timesformat on GAIA, and WebShop’s native attribute-match reward (a task passes only at reward=1.0=1.0).

Refer to caption Figure 5:Co-evolution vs. harness-only evolution (AEGIS, model frozen) on GAIA and WebShop. Stars mark each method’s peak; the shaded band is the co-evolution gain.Co-evolution exceeds harness-only evolution.As Figure5shows, interleaving cross-harness GRPO with harness evolution over a shared replay buffer raises peak success on both benchmarks: GAIA 37.4%→\to41.7% (+4.3%) and WebShop 49.0%→\to54.0% (+5.0%), averaging +4.7% over the model-frozen baseline. The two curves coincide until joint training takes effect (R4), then diverge, with co-evolution at or above harness-only for the remainder of the run. The gap persists to the final round (GAIA 36.4%→\to39.8%, WebShop 46.0%→\to50.0%) and is wider on WebShop, where more room remains for model-level improvement beyond the harness-only plateau. Co-evolution thus lifts end-of-run accuracy, not merely the peak.

Co-evolution breaks the scaffolding ceiling.Harness-only evolution plateaus at∼{\sim}37% on GAIA and∼{\sim}49% on WebShop. Co-evolution clears these plateaus: cross-harness GRPO enables the model to internalize strategies from successive harness versions, so later edits build on learned behavior rather than compensating for a fixed model’s intrinsic limitations.

6.6Failure Analysis

We present three case studies, one per pathology predicted by the operational mirror (Section4.2): reward hacking, catastrophic forgetting, and under-exploration. For each case we document the detection signal that first surfaced the issue, the root cause identified through trace analysis, and the outcome—whether the pipeline self-corrected or required manual intervention. Figure6provides the full set of confirmed and pending cases organized by pathology type.

Refer to caption Figure 6:Failure cases organized by pathology (rows: reward hacking, catastrophic forgetting, under-exploration).#### Reward hacking (GAIA, Sonnet 4.6, R10).

At R10, the pipeline shipped a composite edit (tool + prompt + configuration) whose manifest predicted improved retrieval. The edit passed the seesaw constraint and raised accuracy from 74.8% to 79.6%. Trace analysis at R11 revealed that the tool genuinely fixed retrieval for most newly passing tasks, but a subset passed by exploiting format regularities in the verifier rather than performing actual retrieval. The Planner flagged this pathway at R12, and the resulting edit introduced a guard restricting the tool to tasks whose output could be cross-checked against a second retrieval path.

Catastrophic forgetting (τ3\tau^{3}-Bench, Sonnet 4.6, Telecom, R7).

Evolution on Telecom shipped same-type prompt/processor edits across five consecutive rounds (R2–R6), each appending a “reminder” rule. Compliance rose from 89.5% to 100% at R4, then regressed to 94.7% by R6 as later rules conflicted with earlier ones. The R7 Critic flagged the concentration risk (“All 5 prior ships occupy the same bucket: [prompt, processor]”) but still approved the edit for shipping because ship-prediction accuracy remained high (R2–R6: 23/24, 5/6, 4/5, 7/7, 2/3) and no regressions were recorded. The sixth reminder degraded compliance from 94.7% to 80.7% (−-14.0%) via cross-rule conflicts that destabilized previously passing tasks. This regression evaded the seesaw constraint because pass@2 registers only per-task binary flips, not sub-threshold coupling. The pipeline self-corrected by R9 once the Planner diagnosed the concentration pattern and proposed a structural edit that replaced the conflicting reminder stack.

Under-exploration (ALFWorld, Sonnet 4.6, R4–R7).

Between R4 and R7, the pipeline shipped predominantly prompt-level edits, yielding<<1% gain per round. Ship-prediction accuracy (the fraction of manifest-predicted task flips that materialize) dropped from 80% (R3) to 0% (R7), signaling prompt-space exhaustion. The sole structural edit in this window (a processor-level change at R6) achieved only 14% ship-prediction accuracy (1/7 predicted flips materialized), suggesting that the Planner lacked sufficient structural-edit history to calibrate hypotheses beyond the prompt neighborhood.

Summary.

All three pathologies predicted by the operational mirror appear in practice. The pipeline detected and mitigated reward hacking within two rounds (R10–R12). Decaying ship-prediction accuracy diagnosed under-exploration (R4–R7). The catastrophic-forgetting case exposes a structural limitation of per-edit gating: sub-threshold coupling accumulates undetected until it exceeds the per-task detection threshold (Telecom R7). Onτ3\tau^{3}-Bench Telecom, the pipeline self-corrected (R8–R9) because the failure was localized to one domain; on GAIA (GPT-5.4), the same mechanism produces sustained stagnation (Δ=0.0\Delta{=}0.0) because conflicting edits prevent any net gain. Section6.3shows that variant isolation resolves this by confining edits to task-specific clusters.

7Discussion

7.1Why Compositional Structure Matters for Evolution

As Table5shows, the Global strategy (used in all main experiments) peaks early at 73.8% (R4) on GAIA before collapsing to 49.5% (peak–final gap:−-24.3%). Global uses HarnessX’s typed components but does not leverage them for isolation; every edit is evaluated against all tasks jointly. Under pass@2, a task whose success probability has degraded can still register as “solved,” so sub-threshold regressions evade the seesaw constraint. Preventing this collapse requires variant isolation, which composability enables: HarnessX’s compositional structure makes theintended scopeof each edit explicit, a precondition for variant isolation to confine each edit’s evaluation to its target cluster rather than evaluating against the full task set indiscriminately (Section6.3).

The relationship parallels type systems: types do not generate correct programs, but they make incorrect programsdetectable. Analogously, typed components do not prevent bad edits, but make their scopeexplicit, enabling independent variation. The strategy comparison suggests that variant isolation is necessary for stable evolution (Global, which lacks it, degrades after peaking); without compositional structure, the intended scope of an edit is undefined, making variant isolation ill-posed. Compositional structure does not, however, guarantee bounded behavioral effects: theτ3\tau^{3}-Bench Telecom failure demonstrates that accumulated same-type edits can induce sub-threshold coupling that degrades multiple dialogue patterns simultaneously.

7.2The Role of Trace Richness

HarnessX’s full execution traceτ\tauprovides diagnostic information beyond a scalar reward. The case studies (Section6.6) confirm this: detecting reward hacking on GAIA (shipped at R10, detected at R11) required inspectinghowimprovement occurred (format exploitation vs. genuine retrieval), and detecting under-exploration on ALFWorld (R4–R7) required tracking edit-type distribution and ship-prediction accuracy. Neither signal is recoverable from per-task binary outcomes alone.

These observations motivate a design principle:the richness of the feedback signal bounds the sophistication of evolution that can be safely performed. From scalar reward alone, none of the three pathologies is detectable: a score change cannot distinguish reward hacking from genuine improvement, under-exploration from convergence, or catastrophic forgetting from evaluation noise. Trace structure makes each pathology diagnosable, provided prior-round traces exist for comparison. Theτ3\tau^{3}-Bench Telecom failure illustrates the boundary: despite five rounds of prior traces (R2–R6), accumulated regressions evaded the seesaw constraint because no individual edit crossed the detection threshold. Structured trace recording is therefore necessary for detecting pathologies, but not sufficient for preventing them: when coupling accumulates below the per-task detection threshold, traces record the symptoms only after damage has occurred.

7.3Scope and Limits of the Operational Mirror

The RL–symbolic-space mirror is a design heuristic, not a formal framework. Classical RL convergence guarantees require sufficient exploration of the state–action space, a condition unattainable when states are symbolic harness configurations and actions are open-ended code edits. Under the Global strategy, GAIA (GPT-5.4) stagnates entirely (Δ=0.0\Delta{=}0.0over 15 rounds); the variant-isolation ablation (Section6.3) recovers stable improvement (87.4% final = peak), but nothing guarantees this extends to longer horizons (where variants may over-specialize) or to task distributions whose inter-task dependencies prevent clean variant separation. The mirror also does not predictwhichpathology will dominate: onτ3\tau^{3}-Bench Telecom, catastrophic forgetting surfaced at R7; on ALFWorld, under-exploration dominated R4–R7; on GAIA, reward hacking surfaced only at R10.

We therefore treat the mirror as a design checklist rather than a predictive theory: it identifies failure modes to defend against but does not predict their ordering, timing, or relative severity. The three pathologies are representative, not exhaustive; additional RL phenomena (e.g., distribution shift when the adaptation batch diverges from deployment tasks, reward sparsity on hard benchmarks) may manifest as analogous failure modes in symbolic space.

7.4Generalization Across Model Families

On ALFWorld, the Opus 4.6 meta-agent evolves harnesses for task agents from three model families:

•Sonnet 4.6 (same family): 83.6%→\to94.8% (+11.2%)
•GPT-5.4 (different family): 76.9%→\to97.8% (+20.9%)
•Qwen3.5-9B (different family, weaker): 53.0%→\to97.0% (+44.0%)

The inverse-scaling effect (Section6.2) explains the magnitude ordering: gains track inverse baseline performance (Qwen>>GPT>>Sonnet) rather than proximity to the meta-agent’s model family. All three configurations hold the meta-agent fixed (Opus 4.6) while varying the task agent; we do not evaluate whether a weaker meta-agent can achieve comparable gains.

A complementary ablation (Section6.4) finds that a single-agent evolver achieves comparable accuracy to the four-stage AEGIS pipeline (86.4% vs. 87.4%, within sampling noise atn=103n{=}103) when both share the same meta-agent model and infrastructure. This suggests that at this meta-agent capability level, the four-stage decomposition primarily provides efficiency gains (∼{\sim}12% fewer tokens) and auditability rather than measurable accuracy improvement.

7.5Cost-Performance Tradeoffs

As Table7details, evolution incurs upfront compute that amortizes over subsequent task invocations.

Table 7:Evolution cost summary. All main experiments use the Global (single-harness) strategy; the variant-isolation row is from the strategy ablation (Section6.3).The strategy ablation (Section6.3) shows that variant isolation is both more effective (87.4% vs. 49.5% final) and more efficient (107.8M vs. 143.7M tokens) than Global on GAIA. The token reduction has two sources: (1) structurally, each edit under variant isolation is evaluated only against its target cluster rather than the full task set, reducing per-round evaluation cost; (2) under Global, the steadily degrading baseline causes more candidates to fail gating, wasting tokens on candidates that never ship. On benchmarks where evolution converges quickly (ALFWorld R4–R7, SWE-bench R2–R3), Global suffices and degradation does not materialize within the run horizon.

The evolved harness also affectsper-task inference cost. On GAIA, per-task token consumption drops by∼{\sim}25% (targeted tool selection shortens trajectories); on ALFWorld, it rises by∼{\sim}60% (task-decomposition prompts lengthen execution).

At deployment, the evolved harness is a static artifact requiring no meta-agent inference; tasks outside the evolution set are routed to the variant with the highest overall success rate on the evolution set. On GAIA, the upfront 107.8M tokens amortize within∼{\sim}1,300 invocations (∼{\sim}83K tokens saved per invocation). On ALFWorld, per-task cost increases; the return is accuracy (+11.2%), not cost reduction.

7.6Ethical Considerations

Self-evolving agent systems require explicit oversight. HarnessX provides three mechanisms:

1.Auditability: every shipped edit carries a manifest and a rollback target; rejected candidates are archived with rejection reasons.
2.Deterministic gating: the seesaw constraint rejects any edit that regresses even a single previously solved task under pass@2.
3.Human-in-the-loop: the gating layer supports human approval for edits exceeding a configurable risk threshold (not exercised in our automated experiments).

Theτ3\tau^{3}-Bench failure (Section6.6) illustrates their limits: five consecutive same-type edits (R2–R6) accumulated sub-threshold coupling undetected by the seesaw constraint; the sixth edit (R7) triggered a visible−-14.0% regression, yet no individual edit violated the constraint. This is a structural limitation of per-edit gating: sub-threshold regressions accumulate undetected regardless of how many prior rounds have demonstrated apparent stability under the same constraint.

7.7Limitations

Beyond the limitations noted above, five additional constraints bound the generality of our results:

•No held-out evaluation.All reported gains are measured on the same task set used for evolution. Since we report peak accuracy and evaluate on the adaptation set itself, the numbers carry both selection bias and potential overfitting. Generalization to unseen tasks within the same distribution is plausible but untested.
•Discrete action spaces only.All experiments use agents with discrete, text-based action spaces. We have not tested whether the framework extends to continuous action spaces (e.g., robotic control).
•Closed-source meta-agent.AEGIS requires a meta-agent capable of multi-file code generation, structured trace analysis, and multi-step planning. Open-weight models approaching this capability level (e.g., Qwen3.5-72B, Llama-4-Maverick) remain untested as meta-agents.
•Joint control assumption.Co-evolution requires joint control over both harness evolution and model training. In practice, these concerns are often separated across teams or organizations, making a shared replay buffer (Section5.1) impractical without cross-team coordination.
•Benchmark coverage.All SWE-bench Verified runs use a 55-task subsample, andτ3\tau^{3}-Bench evaluates only three domains (Retail, Airline, Telecom). Conclusions, particularly the inverse-scaling effect, may not generalize to domains with different task heterogeneity or to larger evaluation sets.

8Conclusion

We present HarnessX, a composable runtime foundry that treats the harness as a first-class interface between model and environment. This interface can be composed from typed primitives, evolved from execution traces, and coupled with model training in a unified improvement loop. Across five benchmarks and three model families, HarnessX achieves gains up to+44.0+44.0% (average+14.5+14.5% across 15 configurations) through trace-driven evolution over a compositional substrate, with co-evolution adding +4.7% beyond harness-only evolution on two benchmarks. These results suggest that agent progress need not rely on model scaling alone: composing and evolving the runtime interface from execution feedback is a complementary and actionable lever, particularly for capability-limited agents where harness-level gains are largest.

References

Contributions and Acknowledgments

\xiaomievblue

Core Contributors

•Tingyang Chen*
•Shuo Lu*
•Kang Zhao*
•Weicheng Meng
•Kun Shao†
•Jian Luan†

\xiaomievblue

Contributors

•Hanlin Teng
•Tianhao Li
•Chao Li
•Xule Liu
•Jian Liang
•Zhizhong Zhang
•Yuan Xie
•Heng Qu

††footnotetext:* Equal Contribution†Corresponding Author\beginappendix

9Experimental Setup: Full Details

This appendix expands the condensed setup of Section6.1with the full benchmark descriptions, the formal metric definitions, the evolution protocol hyperparameters, and the runtime infrastructure.

9.1Benchmarks

We evaluate on five benchmarks chosen to span the failure modes that harness design most affects, from short-horizon embodied planning to long-horizon software engineering.

GAIA.

The GAIA benchmark[mialon2024gaia]poses real-world questions that are conceptually simple for humans but require an agent to compose multiple actions (web search, file extraction, multimodal interpretation, arithmetic) and evaluates via exact match against a reference answer. This benchmark stresses open-ended tool-based reasoning, where the harness dictates how evidence is collected and synthesized.

ALFWorld.

The ALFWorld benchmark[shridhar2020alfworld]involves embodied instruction following where a text-based agent commands a simulated robotic agent in household settings. Given a natural-language goal (e.g., “Put a cooled apple in the microwave”), the agent navigates rooms, identifies objects, and manipulates them via textual actions; performance is measured by goal-completion rate. This benchmark stresses multi-step planning and grounded search under a tight step budget. We use the 134 tasks from the valid-unseen set, spanning six task types: pick-and-place, pick-two-and-place, look-at-in-light, and three transform-then-place variants (heat, cool, clean).

WebShop.

WebShop[yao2022webshop]is a web-interaction benchmark in which an agent acts as a customer in a simulated online store. Given a textual product description, the agent must search, browse product pages, select the best-matching item, and purchase it; scoring reflects how well the chosen product satisfies the request. We evaluate on 100 instances sampled with a fixed seed, each run as an independent shopping session.

τ3\tau^{3}-Bench.

τ3\tau^{3}-Bench[yao2024tau]is a multi-turn dialogue benchmark in which the agent plays a customer-service assistant that must satisfy a user request while obeying an explicit domain policy. Performance is measured by rule compliance across the full conversation. The benchmark stresses dialogue-policy adherence: the harness must prevent the agent from agreeing to disallowed actions across many turns. For evaluation, we select three domains from the benchmark: Retail, Airline, and Telecom.

SWE-bench Verified.

SWE-bench Verified[jimenez2024swe]is a human-validated subset of SWE-bench in which each task requires an agent to resolve a real GitHub issue by editing the corresponding repository so that the project’s hidden test suite passes. This benchmark stresses repository-level code editing: navigating a large codebase, localizing the relevant fault, implementing a patch, and avoiding regressions in existing tests. For evaluation, we sample a 55-task subset from SWE-bench Verified and measure performance by patch resolution.

9.2Evaluation-Set Design

The sampled-task counts in Table3denote the fixed evaluation sets scored at each evolution round. GAIA uses a fixed 103-task set drawn across the three difficulty levels (39/52/12). ALFWorld uses all 134 tasks from the valid-unseen split. WebShop uses 100 tasks randomly sampled from the dataset with a fixed seed, with each task run as an independent shopping session. Forτ3\tau^{3}-Bench, we select three domains (Retail, Airline, and Telecom) and score the full task list within each selected domain. For software engineering, we use a 55-task subset sampled from SWE-bench Verified. The same evaluation set for each benchmark is re-scored at every round, so the curves in Appendix12measure round-over-round changes on fixed task sets rather than on moving samples.

9.3Metric Definitions

Pass@kk.

For a configuration evaluated on a task setDDwithnnrollouts per task, letri,j∈{0,1}r_{i,j}\in\{0,1\}denote the binary outcome of rolloutjjon taskii. Letci=∑j=1nri,jc_{i}=\sum_{j=1}^{n}r_{i,j}be the number of successful rollouts for taskii. We report pass@kkusing the standard unbiased estimator, i.e., the probability that at least one ofkksampled rollouts solves the task:

Pass@k=1|D|∑i=1|D|(1−(n−cik)(nk)).\mathrm{Pass@}k=\frac{1}{|D|}\sum_{i=1}^{|D|}\left(1-\frac{\binom{n-c_{i}}{k}}{\binom{n}{k}}\right).(6)All evolution curves use pass@2 as the primary metric: each task receives two independent rollouts and is solved if either succeeds. This reduces sensitivity to single-rollout stochasticity while preserving a strict task-level criterion. Rollouts terminated by infrastructure failures (sandbox crashes, API timeouts) count as failures rather than being excluded, keeping results comparable to official leaderboard protocols.

9.4Evolution Protocol and Hyperparameters

The hyperparameters used in our evolutionary algorithm are detailed in Table8. In round 0, the baseline is a competent composed harness augmented with the benchmark-specific tool registry, rather than a minimal default. The plots therefore show gains relative to a competent initial harness. The meta-agent is Opus 4.6 across all experiments; task agents vary: Sonnet 4.6, GPT-5.4, and Qwen3.5-9B. Per-task step limits are determined by the benchmark in question since the interaction length varies greatly among tasks.

Table 8:Evolution-protocol hyperparameters.

9.5Runtime Infrastructure

Every rollout runs inside a fresh environment instance re-attached per task, so that side-effects (a WebShop cart, an ALFWorld world state, a shell working directory) cannot leak between tasks. The runtime records each rollout’s full trajectory (every model call, tool call, and environment observation) to the observability layer that the Digester subsequently compresses; the cross-round ledgers are aggregated from this log. Task rollouts execute at concurrency 10. The meta-agent runs at concurrency 4 with a 200-step limit per role. Co-evolution model training uses8×8\timesH100 GPUs with batch size 256 and learning rate1×10−61\times 10^{-6}.

10Prompts and Harness Defaults

This appendix reproduces the prompts that drive the AEGIS outer loop and the Round-0 task-agent defaults. The blocks below are the literal contents of the corresponding files in the repository as of the commit that produced the experiments in Section6.

10.1Meta-Agent Prompts

planner/system\_prompt\.md evolver/system\_prompt\.md critic/system\_prompt\.md

10\.2 Round\-0 Task\-Agent Prompts The composition\-layer default \(ℋ0\\mathcal\{H\}\_\{0\}\) loads one system prompt per benchmark\. We reproduce the ALFWorld default below as a representative example; the remaining benchmark defaults follow the same structure and are listed in the repository\. alfworld\_evolver/systemprompt\.md 10\.3 Change\-Manifest Schema Each Evolver candidate is accompanied by a change manifest, a structured audit record linking the proposed edit to its evidence, mechanism, expected effect, and attribution signal\. The manifest makes every harness modification falsifiable: the Critic checks whether the next round’s trace features match the mechanism and impact the manifest predicted\. Table 9 defines the manifest fields, and the schema below specifies their representation\. Table 9: Change\-manifest fields\. The manifest is the loop’s evidence ledger: every shipped edit is falsifiable against the next round’s trace\-feature deltas\. Field Meaning candidate\_id Unique id, e\.g\. C\-R3\-01 \(round 3, candidate 1\)\. bucket Edit type: prompt, tools, config, or processor\. capability\_evidence Verified claims that the edit mechanism actually works\. file\_changes List of path / action / diff\-summary edits\. predicted\_impact Tasks the edit will unlock, stabilize, or put at risk: the falsifiable prediction\. attribution\_signature Trace feature that must appear if the edit fired, e\.g\. a processor invocation\. change\_manifest\.yaml \(schema\) 11 Anatomy of an Evolution Step To illustrate the AEGIS loop concretely, we walk through one full cycle, from Digester compression through Planner synthesis, Evolver editing, and Critic judgment, to the resulting trace delta\. We select round 10 of the GAIA / Sonnet 4\.6 run: a composite edit spanning a new tool, a prompt addition, and a configuration change\. This multi\-component intervention produced the largest single\-round gain in that run, making it a richer illustration than a single\-lever edit\. 11\.1 Worked Example: GAIA / Sonnet 4\.6, Round 10 Failure evidence\. Prior to round 10, the success rate stood at 74\.8%74\.8\\%, having dropped from its peak of 77\.7%77\.7\\% due to a regression in round 9\. The Digester’s trace analysis revealed a systematic failure pattern: every Wikipedia fetch in round 10’s traces returned zero characters\. WebFetch employs a browser if the website requires JavaScript support, but Wikipedia’s new frontend fails to load correctly, timing out or returning an empty body\. The traces make it plain: within task db4fd70a \(number of stations in a rail line\), db4fd70a\_r0\.jsonl\#step\_0 and \#step\_1 report that Wikipedia WebFetch fetches return 0 chars; similarly, within f0f46385 \(ASEAN members’ membership status\), three consecutive WebFetch calls return 0 chars; ten separate attempts across the round returned empty responses\. Planner synthesis\. The Digester grouped the 23 failed tasks by failure mode, surfacing a critical tool\-level issue that the round\-9 Critic had already flagged: the tools component had not shipped a fix in nine consecutive rounds despite source\-access failures appearing since round 1\. The Planner received two targets: \(1\) resolve the persistent tool\-level source\-access failures, and \(2\) revert the prompt and budget\-processor changes responsible for the round\-9 regression\. Evolver edit\. The Evolver suggested C\-R10\-02, covering three buckets\. \(i\) tools: new WikiTextFetch tool, avoiding the browser altogether by employing the MediaWiki API endpoint; returns complete text of the article, 10,529 chars in case of the rail line, 80,028 for ASEAN\. \(ii\) prompt: a single sentence in the tool usage section instructing the agent to use WikiTextFetch before looking up Wikipedia articles\. \(iii\) config: restore the round\-8 baseline configuration, register WikiTextFetch, and remove the problematic budget processor\. The manifest’s capability evidence includes a Level\-2 round\-trip check \(content serialized as a 10,529\-char string by the provider\); the attribution signature requires at least one WikiTextFetch call\. R10/candidates/C\-R10\-02\.md Critic verdict\. The Critic approved only C\-R10\-02, rejecting the competing revert\-only candidate C\-R10\-01\. Its rationale covered three aspects\. \(i\) Interaction: C\-R10\-02 is a strict superset of C\-R10\-01; both restore the round\-8 baseline, so the budget\-processor removal is intentional, not an accidental overlap\. \(ii\) Round\-trip evidence: the Critic verified Level\-2 evidence \(tool output arrives as a full string, not a truncation marker\) before accepting any tools\-bucket candidate\. \(iii\) Portfolio: this is the first tools\-bucket ship in ten rounds, and the trace evidence of persistent zero\-char returns justifies the intervention that round\-9 flagged\. Delta realized\. Post\-shipping, the GAIA pass rate increased from 74\.8%74\.8\\% at R9 to 79\.6%79\.6\\% at R10 \(\+4\.94\.9pp, five tasks changed to pass\), the greatest improvement during the entire run; five of the seven tasks the tool was predicted to affect flipped to pass \(hit rate 0\.710\.71, the highest for any ship across all 19 runs\)\. The improvement mainly occurred at Levels 2 \(\+44 tasks\) and 3 \(\+22\)\. Since the tool triggered its target tasks, the attribution condition was satisfied\. Figure 7 summarizes the same edit as a manifest card: the raw YAML above is what the loop logs, the card is the human\-facing reading of it\. C\-R10\-02 \|\| GAIA / Sonnet 4\.6 \|\| round 10 \|\| bucket: tools\+prompt\+config Failure evidence Every Wikipedia WebFetch in the round’s trajectories returns 0 chars \(10\+ calls across db4fd70a, f0f46385, 42d4198c\); the headless browser fallback times out on Wikipedia’s frontend\. Edit New WikiTextFetch tool calls the MediaWiki API for plain\-text article content \(tools\); one prompt line routes Wikipedia lookups to it first \(prompt\); the round\-8 baseline is restored and the regressing budget processor removed \(config\)\. Predicted fixes Unlock db4fd70a, f0f46385, 983bba7c, 08f3a05f, 5e2a91b0; stabilize 4b6bb5f7, 42d4198c\. Attribution signature tool\_call on WikiTextFetch, ≥1\\geq 1 call; verified against the next round’s trace\. Figure 7: Change manifest for C\-R10\-02 rendered as a manifest card, the human\-facing counterpart of the logged YAML above\. 12 Additional Results The rest of this appendix is organized per benchmark\. Each subsection is built around one figure of three panels: \(a\) a breakdown of the failure clusters the adaptation loop had to address, \(b\) the per\-model distribution of harness levers the evolution shipped, and \(c\) a model\-by\-lever heatmap of each lever’s effectiveness \(tasks flipped to pass over tasks predicted\)\. We read the three panels in order: what fails and why, how each model evolves, and whether the evolution closes the failures\. 12\.1 GAIA GAIA stresses general reasoning under tool use, and is the most lever\-diverse benchmark in our suite\. Figure 8 gives the three views we use throughout this appendix\. Failure clusters and their causes\. Panel \(a\) summarizes the failure clusters accumulated across the GAIA run\. The dominant cluster is blocked\-source \(39%39\\%\), where the agent cannot retrieve the required evidence because pages return empty content, require JavaScript rendering that times out, or contain incomplete information\. Reasoning failures \(33%33\\%\) follow, covering tasks that require multi\-hop inference, disambiguation of similar entities, numerical computation, or precise interpretation of underspecified queries\. Figure/visual failures \(11%11\\%\) arise when the answer depends on information embedded in images, maps, or diagrams that text extraction alone cannot capture\. Document/table parsing failures \(11%11\\%\) occur when evidence is locked in PDFs, structured tables, or semi\-structured formats and the agent either misparses the layout or overlooks relevant cells\. Scope ambiguity \(6%6\\%\) covers queries with multiple valid interpretations, where the agent answers a related but incorrect reading\. Together, these clusters indicate that GAIA failures concentrate in evidence retrieval, multi\-step reasoning, visual grounding, structured\-document extraction, and query disambiguation\. Per\-model evolution logic\. Panel \(b\) shows GAIA is the only benchmark where all four levers see substantial use; the Sonnet run alone shipped 11 prompt, 7 processor, 7 config, and 6 tools edits, because its failure set spans tool, prompt, and config problems simultaneously\. The three models nonetheless diverge in a way that tracks their starting competence: Sonnet, with the most rounds, sweeps every lever; GPT\-5\.4 leans hardest on prompt \(45%45\\% of its ships\) and barely touches config, since its reasoning is already strong enough that the remaining gains are mostly instruction\-following; Qwen3\.5’s short run concentrates its few ships and, strikingly, lands its single tools ship at the highest yield of any cell\. The shared logic is prompt\-first for the behavioral clusters, with the scarce tools lever reserved for the one mechanical cluster prose cannot touch\. Did evolution close the clusters? Panel \(c\) shows which failure clusters were reduced by evolution\. The largest improvement comes from the blocked\-source cluster: the tool edit that introduced WikiTextFetch replaced unreliable browser\-based Wikipedia fetching with a MediaWiki API call, reducing failures caused by empty or incomplete page retrieval\. Prompt edits mainly targeted the reasoning cluster by encouraging more explicit verification, which contributed to steady gains across rounds\. By contrast, figure / visual and document / table parsing failures remained harder to reduce, because they require information extraction from images, figures, PDFs, or structured tables\. Overall, GAIA improves through a combination of tool edits that fix retrieval failures and prompt edits that reduce reasoning errors, while residual errors concentrate in visual and document\-heavy tasks\. Figure 8: GAIA evolution analysis \(103 tasks, exact\-match\)\. \(a\) Failure clusters among the tasks still unsolved; blocked\-source and reasoning dominate, while figure/visual and parsing clusters are residual model gaps\. \(b\) Share of shipped edits by bucket for each task model\. \(c\) Lever effectiveness as hit\-rate \(tasks flipped / predicted\) per model and bucket\. The single Qwen3\.5 tools ship is the highest\-yield cell \(0\.670\.67\)\. 12\.2 ALFWorld ALFWorld is an embodied planning benchmark and the most prompt\-dominated in our suite\. Figure 9 shows its clusters, per\-model logic, and effectiveness\. Failure clusters and their causes\. Panel \(a\) summarizes the main failure clusters observed on ALFWorld\. The dominant cluster is search / step\-ceiling \(89%89\\%\), which covers episodes where the agent either searches rooms or receptacles in an inefficient order, or reaches the step limit before completing long interaction chains such as deep search or transform\-then\-place tasks\. Prompt\-rule side\-effect failures \(7%7\\%\) occur when an added heuristic improves some tasks but unintentionally restricts behavior on others, causing the agent to skip a valid action or stop searching too early\. Object\-type confusion failures \(4%4\\%\) refer to cases where the agent confuses semantically similar objects\. Together, these clusters show that ALFWorld failures mainly arise from search efficiency, over\-constrained prompting, and object\-specific grounding errors\. Per\-model evolution logic\. Panel \(b\) shows that prompt dominance scales inversely with base\-model strength: a prompt rule yields gains only for models that reliably follow it\. Sonnet, being the strongest base, derives nearly all improvement from prompt edits alone; search\-order heuristics in the system prompt suffice because it consistently obeys them\. GPT\-5\.4 supplements prompts with a processor \(introduced at round three\) that manages its step budget for transformation tasks\. Qwen3\.5 requires the most varied mix \(prompt, config, and processor\), including a processor that intercepts its reasoning text and re\-emits tool calls when needed, a mechanical fix for a failure that prompt\-level steering cannot resolve\. The shared pattern is prompt\-first, with structural levers recruited only when prompts prove insufficient\. The weaker the base model, the sooner evolution falls back from prompt\-based steering to config or processor enforcement, visible in the growing non\-prompt segments from Sonnet to Qwen\. Did evolution close the clusters? Panel \(c\) shows that evolution reduced the main ALFWorld failure clusters, with different levers mattering for different task agents\. For Qwen3\.5, processor and config edits achieved the strongest effects, with hit\-rates of 0\.840\.84 and 0\.710\.71, respectively\. These edits directly addressed mechanical failures by re\-emitting missed tool calls and adjusting execution budgets, allowing Qwen3\.5 to improve by \+44\.0\+44\.0pp and approach the closed\-model runs\. For Sonnet, the remaining failures were less structural, so prompt edits were sufficient for most gains, reaching a 0\.490\.49 hit\-rate, while the processor edit had only a marginal effect \(0\.140\.14\)\. Two clusters were only partially reduced: prompt\-rule side effects, which were introduced by some evolved heuristics and then patched in later rounds, and long\-path failures, where some episodes still exceeded the available interaction budget\. Overall, ALFWorld shows a clear model\-dependent pattern: stronger models benefit mainly from prompt\-level steering, whereas weaker models require more structural support through processor and configuration edits\. Figure 9: ALFWorld evolution analysis \(134 tasks, goal\-completion\)\. \(a\) Failure clusters accumulated across all rounds; search inefficiency and the hard step\-ceiling dominate, with two small clusters that evolution itself introduced \(a prompt\-rule side\-effect\) or transiently hit \(object\-type confusion\)\. \(b\) Lever mix by model: the strong base \(Sonnet\) climbs on prompt almost alone, while weaker bases reach for more varied levers\. \(c\) Lever effectiveness: structural levers \(processor, config\) are both used more and more effective on weaker models\. 12\.3 WebShop WebShop is a web\-interaction benchmark and the noisiest run in our suite\. Figure 10 shows its clusters, per\-model logic, and effectiveness\. Failure clusters and their causes\. Panel \(a\) summarizes the WebShop failure clusters accumulated across the run\. Early failures are dominated by search and pagination loops, where the agent repeatedly reformulates queries or cycles through result pages without committing to a purchase\. As evolution reduces these control\-flow errors, the remaining failures shift toward product\-selection judgment\. The largest cluster, wrong product \(46%46\\%\), occurs when the agent selects an item from the wrong category or settles on a weak match before comparing alternatives\. Pagination loop failures \(21%21\\%\) capture the remaining cases of repeated next/previous navigation without progress\. Colour matching failures \(17%17\\%\) arise when the agent mishandles shade equivalence or site\-specific color labels, such as treating “wine” and “red” as incompatible\. Attribute check failures \(17%17\\%\) occur when the selected item is close to the request but fails on a required detail, such as size, sleeve length, or another unverified attribute\. Overall, the cluster shift indicates that evolution first reduces navigation loops, after which the main errors concentrate in product matching and attribute verification\. Per\-model evolution logic\. Panel \(b\) shows that prompt edits drive most of the improvement across all three models, with processor edits serving as a consistent secondary lever\. This pattern matches WebShop’s main control\-flow failures\. Prompt rules help the agent search more efficiently and commit earlier, while advisory processors reinforce these rules during execution by adding warnings when the agent begins to repeat searches or cycle through pagination\. For product\-selection failures, evolution introduces more targeted support: a colour\-matching tool helps resolve shade\-equivalence cases, and config edits help weaker models maintain context over longer shopping sessions\. Overall, WebShop requires a mixed response: prompts improve high\-level shopping strategy, processors reduce navigation loops, tools support attribute matching, and config edits stabilize long\-session behavior\. Did evolution close the clusters? Panel \(c\) shows that evolution partially reduced the WebShop failure clusters\. Prompt edits were the most consistently effective lever across models, with hit\-rates of 0\.370\.37–0\.500\.50, while config edits helped the two weaker models maintain context over longer sessions\. These changes reduced early search and pagination loops, raising performance from 60%60\\% to a peak of 76%76\\%\. The remaining clusters proved harder to close\. Advisory processors produced only modest gains \(0\.200\.20–0\.250\.25\), so some pagination failures persisted\. The colour\-matching tool did not improve performance in this run \(0\.00\.0 hit\-rate\), leaving the colour\-matching cluster largely unchanged\. Overall, WebShop benefits most from prompt and config edits, while residual navigation loops and product\-judgment errors remain the main sources of instability\. Figure 10: WebShop evolution analysis \(100 sessions\)\. \(a\) Failure clusters across the run, after evolution has tamed the round\-0 search/pagination loops; the residual is mostly product\-choice judgment \(wrong product, colour matching, attribute check\)\. \(b\) Lever mix by model: prompt carries the climb, processor is the consistent second lever\. \(c\) Lever effectiveness; prompt and config are the productive levers, the lone colour\-matcher tool ship returned 0\.00\.0\. 12\.4 τ3\\tau^\{3\}\-Bench τ3\\tau^\{3\}\-Bench stresses multi\-turn dialogue under an explicit domain policy\. Figure 11 pools the AEGIS runs across the airline, retail, and telecom domains\. Failure clusters and their causes\. Excluding harness\-interrupt traces, the failures are judgment\-heavy: the top two clusters, premature / unverified action \(28%28\\%; committing a booking, refund, or device fix before a precondition holds\) and wrong selection / count \(24%24\\%\), together exceed half and concern when to commit and what to pick rather than mechanical execution\. The remainder is procedural \(incomplete multi\-step fix 16%16\\%, missed step / sub\-task 14%14\\%\) or policy\-related \(misinterpretation 13%13\\%\)\. The smallest cluster, capability\-boundary confusion \(5%5\\%\), is τ3\\tau^\{3\}\-specific: some telecom faults live on the user’s handset, where the agent has no device\-side tool and the failure is its treating that boundary as a missing capability\. Per\-model evolution logic\. Evolution is prompt\-and\-processor driven for every model, with zero tools edits: the tool set is fixed and no cluster is one that a new tool could close\. Sonnet 4\.6 splits prompt/processor \(23/1823/18\), GPT\-5\.4 ships the most balanced mix \(19/2019/20\), and Qwen3\.5\-9B ships fewer overall \(14/914/9\)\. Since τ3\\tau^\{3\} failures are control\-flow and judgment errors, the productive levers are prompt rules that encode the policy’s ordering constraints and processors that enforce them mid\-dialogue\. Did evolution close the clusters? Config is the sharpest lever where used \(Qwen3\.5 0\.670\.67, GPT\-5\.4 0\.330\.33 hit\-rate\) but is shipped rarely; the high\-volume prompt and processor levers are moderately effective \(0\.270\.27–0\.350\.35\), matching the control\-flow nature of the dominant clusters\. Gains track base\-model headroom: GPT\-5\.4 starts lowest \(76\.2%76\.2\\%\) and gains most \(\+14\.5\+14\.5pp\), Sonnet 4\.6 gains \+5\.4\+5\.4pp, and near\-ceiling Qwen3\.5\-9B only \+1\.1\+1\.1pp\. The loop is not monotone; Sonnet’s telecom run reaches 100%100\\% at R4, regresses to 80\.7%80\.7\\% at R7 after a sixth consecutive same\-bucket edit, then recovers to 99\.1%99\.1\\% by R9 \(Section 6\.6\)\. Overall, the ordering\-enforcing levers close the premature\-action and missed\-step clusters most reliably, while wrong\-selection and policy\-judgment errors are the harder residual\. Figure 11: τ3\\tau^\{3\}\-Bench evolution analysis, pooled over the airline, retail, and telecom domains\. \(a\) Failure clusters from the logged digests \(harness\-interrupt traces excluded\); judgment errors \(premature action and wrong selection\) dominate\. \(b\) Lever mix by model: prompt and processor carry the climb, with zero tools edits since the tool set is fixed\. \(c\) Lever effectiveness; config is sharpest where used \(Qwen3\.5 0\.670\.67\) but rare, while prompt and processor are the consistent high\-volume levers\. 12\.5 SWE\-bench Verified SWE\-bench Verified stresses repository\-level code editing\. Figure 12 shows its clusters, per\-model logic, and effectiveness\. Failure clusters and their causes\. Panel \(a\) summarizes the failure clusters pooled across all rounds and all three models\. The dominant cluster is incomplete fix \(62%62\\%\), where the agent reaches the right region and produces a valid patch but covers only one branch or call site while the gold patch needs several\. Wrong diagnosis \(19%19\\%\) follows, covering edits to the wrong file or abstraction level after misreading the root cause\. The remaining tail is mechanical rather than cognitive: no edit attempted \(6%6\\%\), Edit anchor mismatch \(5%5\\%\), and budget exhausted \(4%4\\%\)\. Notably the composition is the inverse of reward\-hacking: failures are under\-fixes, not gamed evaluations, because the harness applies the gold test patch before the model patch and blocks test\-file writes\. Per\-model evolution logic\. Panel \(b\) shows that SWE\-bench is prompt\-first for every model, with the secondary lever tracking base\-model strength\. All three runs ship zero tools edits, since unlike GAIA the failure set has no mechanical\-retrieval cluster a tool could close\. Sonnet pairs prompt with an equal share of processor edits \(77 each\), using workflow nudges to shape an already\-competent coder; GPT\-5\.4 leans hardest on prompt \(88 ships\) and uses config \(44\) to revert a harmful nudge and restructure its strategy phases; Qwen3\.5 spreads its few ships across prompt, processor, and config \(6/3/36/3/3\)\. The shared logic is prompt\-first, with structural levers recruited as the base model weakens\. Did evolution close the clusters? Panel \(c\) reveals a sharp capability floor\. For the strong models the productive levers genuinely close failures: GPT\-5\.4’s config edits reach 0\.480\.48 and prompt 0\.390\.39, while Sonnet’s prompt and processor levers both land at 0\.400\.40\. For Qwen3\.5\-9B every lever collapses to near\-zero \(prompt 0\.050\.05, config 0\.050\.05, processor 0\.060\.06\), an order of magnitude lower, because the 99B base cannot execute the predicted fixes\. The same loop that lifts GPT\-5\.4 from 45%45\\% to a 64%64\\% peak and stabilizes Sonnet near 87%87\\% yields only noise on Qwen3\.5 \(peak 42%42\\%, zero durable gains\)\. Overall SWE\-bench improves through prompt edits that broaden fix scope and config edits that restore workflow pacing, but only for models strong enough to act on them\. Figure 12: SWE\-bench Verified evolution analysis \(5555 tasks, resolved\-rate\)\. \(a\) Failure clusters pooled across all rounds and all three task models; incomplete fix and wrong diagnosis dominate, while the mechanical tail \(no\-edit, anchor mismatch, budget\) is residual; failures are under\-fixes, not gamed evaluations\. \(b\) Lever mix by model: every run is prompt\-first and ships zero tools edits, with the secondary lever shifting from processor \(Sonnet\) to config \(GPT\-5\.4\) to a varied mix \(Qwen3\.5\)\. \(c\) Lever effectiveness as hit\-rate \(tasks flipped / predicted\); strong models reach 0\.390\.39–0\.480\.48 on their productive levers, whereas every Qwen3\.5\-9B lever collapses to ≈0\.05\\approx 0\.05, a capability floor below which evolution cannot compound\. 13 Reproducibility and Artifacts 13\.1 Per\-Run Directory Layout Each evolution run writes a self\-describing directory\. The layout below lets a reader reconstruct any decision in this paper from the logged artifacts\. runs/<run\_name\>/ \(per\-run artifact layout\)