Learning to Control LLM Agent Harnesses with Offline Reinforcement Learning

# Learning to Control LLM Agent Harnesses with Offline Reinforcement Learning
Source: [https://arxiv.org/html/2607.05458](https://arxiv.org/html/2607.05458)
Haiwen Yi University of Toronto &Xinyuan Song11footnotemark:1 Emory University

###### Abstract

Large language model \(LLM\) agents are usually improved by changing prompts, models, or hand\-written workflows, while the execution harness around the model is treated as fixed infrastructure\. We argue that this harness is itself a learnable control layer\. We formalize harness operation as a finite\-horizon Harness MDP, where a lightweight controller selects structural execution actions while the LLM executor remains frozen\. The controller is trained from offline rollouts using advantage\-weighted regression with only terminal task\-rubric rewards\. We also separate final task quality from a post\-hoc Harness Maturity Score, which measures whether the harness follows reliable execution patterns rather than only whether the final answer is correct\. This separation gives a finite\-buffer view of harness learning: final\-quality gains require high\-return support in the offline buffer, while process behavior can shift whenever it correlates with advantage weights\. Across six controlled domains and two public\-benchmark adapters, the learned controller consistently improves verification behavior and selectively improves final quality, with the largest gains on adaptedτ\\tau\-bench retail, adapted AgentBench DB\-Bench, and coding with a calibrated structural verifier\. Ablations against behavior cloning and Forced CHECK show that the gains are not explained by imitation or by simply adding checks\. These results identify harness control as a learnable layer for frozen LLM agents, while showing that offline support limits when better process control becomes better final answers\. Our code is available at[https://github\.com/Hik289/Agentic\-RL\-harness\.git](https://github.com/Hik289/Agentic-RL-harness.git)\.

Learning to Control LLM Agent Harnesses with Offline Reinforcement Learning

Haiwen Yi††thanks:Equal contribution\.University of TorontoXinyuan Song11footnotemark:1††thanks:Corresponding author\.Emory University

## 1Introduction

An LLM agent is more than a language model with a prompt\. In deployed systems, a surrounding harness decides when the model observes the environment, retrieves evidence, calls tools, drafts an answer, checks intermediate work, revises, and stops\(Yaoet al\.,[2023](https://arxiv.org/html/2607.05458#bib.bib16); Schicket al\.,[2023](https://arxiv.org/html/2607.05458#bib.bib26); Wuet al\.,[2023](https://arxiv.org/html/2607.05458#bib.bib36); Yanget al\.,[2024b](https://arxiv.org/html/2607.05458#bib.bib2)\)\. Holding the executor fixed, this control layer can change the agent’s behavior substantially: a coding agent may or may not run tests before submission, a research agent may or may not gather evidence before making a claim, and a tool\-use agent may or may not recover from a failed call\(Jimenezet al\.,[2024](https://arxiv.org/html/2607.05458#bib.bib9); Yanget al\.,[2024b](https://arxiv.org/html/2607.05458#bib.bib2); Mialonet al\.,[2023](https://arxiv.org/html/2607.05458#bib.bib8); Liuet al\.,[2024](https://arxiv.org/html/2607.05458#bib.bib1); Yaoet al\.,[2024](https://arxiv.org/html/2607.05458#bib.bib5); Luet al\.,[2024](https://arxiv.org/html/2607.05458#bib.bib39)\)\. The harness is therefore not a cosmetic wrapper around inference; it is part of the policy that determines how work is carried out\.

The difficulty is that harnesses are usually built as static artifacts\. Existing systems compose hand\-written rules, prompt templates, multi\-agent scaffolds, or fixed execution graphs\(Wuet al\.,[2023](https://arxiv.org/html/2607.05458#bib.bib36); Honget al\.,[2024](https://arxiv.org/html/2607.05458#bib.bib33); Chenet al\.,[2024](https://arxiv.org/html/2607.05458#bib.bib34)\)\. Such designs can be highly effective in a narrow regime, but they encode the same control logic for easy and hard tasks, for early and late trajectory states, and for successful and failed intermediate attempts\. A fixed rule such as “always check” wastes budget when a solution is already complete; a rule such as “submit after drafting” fails when the draft is plausible but unverified\. The failure mode is often not missing model knowledge, but a poor sequence of external control decisions\.

This paper studies whether that sequence can be learned directly\. We define a*Harness Markov decision process*in which the state summarizes the current trajectory, draft status, collected evidence, tool outputs, verifier feedback, previous failures, and remaining budget\. At each step, a controller chooses one structural action fromobserve,retrieve,call\-tool,draft,check,revise, andsubmit\. The chosen action may invoke the frozen LLM executor, but the LLM parameters and prompts are not updated\. Learning is thus confined to the external control policy: the controller learns how the agent should move through observation, evidence collection, verification, revision, and termination\.

Online exploration in agent environments is expensive and can produce invalid or low\-quality trajectories\. We therefore train from a finite offline rollout buffer using advantage\-weighted regression \(AW\)\(Penget al\.,[2019](https://arxiv.org/html/2607.05458#bib.bib27)\)\. Each trajectory receives a terminal task\-rubric return, and the controller increases the likelihood of state\-action pairs that appear in higher\-advantage trajectories\. This choice gives a conservative learning rule: the learned policy can reuse good control patterns already present in the buffer, but it cannot reliably invent outcome\-improving trajectories that the buffer never contains\. Figure[1](https://arxiv.org/html/2607.05458#S1.F1)gives the central intuition\.

The main conceptual distinction is between final task quality and process quality\. We train only on terminal task\-rubric rewards, denoted by𝒬\\mathcal\{Q\}, and evaluate harness process behavior separately with the Harness Maturity ScoreHMS\\mathrm\{HMS\}\.HMS\\mathrm\{HMS\}measures events such as checking before submission, testing before submission, grounding claims in evidence, revising after failure, valid tool use, sufficient stopping, and premature submission\. This separation is deliberate\. A controller can learn a recurring process behavior, such as checking before submission, whenever that behavior is positively associated with advantage\-weighted trajectories\. In contrast, final\-quality improvement requires high\-return trajectories with enough support in the offline buffer\.

Our empirical story follows this separation\. Across six controlled domains and two public\-benchmark adapters, AW increases verification before submission in every evaluation setting\. AggregateHMS\\mathrm\{HMS\}improves in five of six controlled domains and in both adapter evaluations, but the improvement is localized: it is driven mainly by CheckBeforeSubmit rather than by uniform gains across all process events\. Final task quality improves selectively\. Coding improves under the calibrated structural verifier; theτ\\tau\-bench retail and AgentBench DB\-Bench adapters show the largest external gains, although these are adapter\-level evaluations rather than official upstream benchmark scores\. The pattern suggests that harness process can be learned broadly from offline traces, while outcome gains appear where the buffer contains useful high\-return support\.

![Refer to caption](https://arxiv.org/html/2607.05458v1/figures/g4_global_intuition.png)Figure 1:The proposed framework separates final task quality from harness process quality\. Offline AW learns a state\-conditioned controller over harness operations while keeping the LLM executor fixed\. Process improvement can arise from recurring high\-value control patterns in the offline buffer, whereas final\-quality improvement additionally depends on whether the buffer contains stronger task\-solving trajectories\.Our contributions are summarized as follows:

- •We formulate the external control layer of a frozen LLM agent as a Harness MDP, in which a learned policy selects the next structural operation according to the current trajectory state\.
- •We develop an offline RL method based on advantage\-weighted regression that learns harness\-control decisions from finite rollout buffers and task\-rubric terminal rewards, without updating the LLM or directly rewarding predefined action patterns\.
- •We separate final task quality𝒬\\mathcal\{Q\}from harness process qualityHMS\\mathrm\{HMS\}and provide a finite\-buffer analysis showing why process behavior may improve even when gains in𝒬\\mathcal\{Q\}remain limited by offline trajectory support\.
- •We evaluate the framework across six controlled domains and two public\-benchmark adapters using a shared action space, task\-specific structural verifiers, and a seven\-event process diagnostic\. The results show reliable improvements in verification\-before\-submission and selective final\-quality gains when the offline buffer contains useful high\-return trajectories\.

## 2Related Work

#### Agent and tool\-use benchmarks\.

AgentBench,τ\\tau\-bench, GAIA, WebArena, WebShop, SWE\-bench, AgentGym, ToolBench, ToolSandbox, and long\-memory benchmarks evaluate tool use, web navigation, coding, retrieval, state tracking, and API\-mediated interaction\(Liuet al\.,[2024](https://arxiv.org/html/2607.05458#bib.bib1); Yaoet al\.,[2024](https://arxiv.org/html/2607.05458#bib.bib5); Mialonet al\.,[2023](https://arxiv.org/html/2607.05458#bib.bib8); Zhouet al\.,[2024b](https://arxiv.org/html/2607.05458#bib.bib6); Yaoet al\.,[2022](https://arxiv.org/html/2607.05458#bib.bib7); Jimenezet al\.,[2024](https://arxiv.org/html/2607.05458#bib.bib9); Xiet al\.,[2024](https://arxiv.org/html/2607.05458#bib.bib10); Qinet al\.,[2023](https://arxiv.org/html/2607.05458#bib.bib12); Luet al\.,[2024](https://arxiv.org/html/2607.05458#bib.bib39); Maharanaet al\.,[2024](https://arxiv.org/html/2607.05458#bib.bib11)\)\. These benchmarks reveal long\-horizon failures of complete agent systems\. Our goal is narrower: we isolate the external control policy around a fixed executor and ask whether that policy can be learned\.

#### Prompt, scaffold, and workflow optimization\.

Prompt and workflow optimizers search over instructions, demonstrations, programs, or code\-represented scaffolds\(Zhouet al\.,[2023](https://arxiv.org/html/2607.05458#bib.bib13); Yanget al\.,[2024a](https://arxiv.org/html/2607.05458#bib.bib14); Khattabet al\.,[2023](https://arxiv.org/html/2607.05458#bib.bib15); Yuksekgonulet al\.,[2024](https://arxiv.org/html/2607.05458#bib.bib17); Zhanget al\.,[2024](https://arxiv.org/html/2607.05458#bib.bib3); Wanget al\.,[2025](https://arxiv.org/html/2607.05458#bib.bib18); Tornedeet al\.,[2025](https://arxiv.org/html/2607.05458#bib.bib19); Agrawalet al\.,[2025](https://arxiv.org/html/2607.05458#bib.bib20)\)\. Automatic harness\-evolution methods further revise the program around an agent using trajectory feedback\(Linet al\.,[2026](https://arxiv.org/html/2607.05458#bib.bib21); Panet al\.,[2026](https://arxiv.org/html/2607.05458#bib.bib4); Seonget al\.,[2026](https://arxiv.org/html/2607.05458#bib.bib22)\)\. These approaches optimize an artifact before execution\. We instead learn a state\-conditioned controller that chooses the next harness operation during execution, so two tasks can follow different control paths even under the same frozen LLM\.

#### Reflection, planning, and agent learning\.

Reflection and planning systems such as Reflexion, Voyager, LATS, and ADaPT provide structured feedback, revision, search, and decomposition procedures\(Shinnet al\.,[2023](https://arxiv.org/html/2607.05458#bib.bib23); Wanget al\.,[2023](https://arxiv.org/html/2607.05458#bib.bib24); Zhouet al\.,[2024a](https://arxiv.org/html/2607.05458#bib.bib25); Prasadet al\.,[2024](https://arxiv.org/html/2607.05458#bib.bib35)\)\. WebShop and AgentGym show that interactive agents can be trained or fine\-tuned with reinforcement learning and behavior cloning\(Yaoet al\.,[2022](https://arxiv.org/html/2607.05458#bib.bib7); Xiet al\.,[2024](https://arxiv.org/html/2607.05458#bib.bib10)\)\. Our setting differs in two ways: the executor is frozen, and the learned policy acts only over a compact set of harness operations rather than over natural\-language solution tokens or environment actions\.

#### Offline reinforcement learning and baselines\.

Offline RL studies how to learn from a fixed dataset without collecting additional experience\(Levineet al\.,[2020](https://arxiv.org/html/2607.05458#bib.bib28); Brandfonbreneret al\.,[2021](https://arxiv.org/html/2607.05458#bib.bib42)\)\. We use advantage\-weighted regression\(Penget al\.,[2019](https://arxiv.org/html/2607.05458#bib.bib27)\), a simple weighted\-imitation objective related to advantage\-weighted actor\-critic methods\(Nairet al\.,[2021](https://arxiv.org/html/2607.05458#bib.bib29)\)and to conservative offline policy improvement methods such as implicit Q\-learning\(Kostrikovet al\.,[2022](https://arxiv.org/html/2607.05458#bib.bib30)\)\. Behavior cloning is a natural imitation baseline\(Pomerleau,[1989](https://arxiv.org/html/2607.05458#bib.bib31); Rosset al\.,[2011](https://arxiv.org/html/2607.05458#bib.bib32)\); Forced CHECK is a harness\-specific baseline that tests whether the observed gains can be explained by mechanically inserting verification rather than by learning when verification is useful\.

#### Process evaluation and reward design\.

Final\-answer accuracy can hide failures in verification, evidence use, revision, and stopping behavior\(Zhanget al\.,[2025](https://arxiv.org/html/2607.05458#bib.bib37); Kuanget al\.,[2025](https://arxiv.org/html/2607.05458#bib.bib38); Pandyaet al\.,[2026](https://arxiv.org/html/2607.05458#bib.bib40)\)\. We therefore reportHMS\\mathrm\{HMS\}as a process diagnostic, but we do not optimize it directly\. The distinction follows the reward\-shaping principle that auxiliary rewards should not change the task\-optimal policy set unless they are potential\-based\(Nget al\.,[1999](https://arxiv.org/html/2607.05458#bib.bib43)\)\. Our analysis makes this separation explicit for finite offline harness buffers\.

## 3Method

The method learns a small controller around a fixed LLM executor\. The controller observes compact trajectory features, selects the next harness operation, and receives supervision only through terminal task\-rubric rewards collected in an offline buffer\. Figure[2](https://arxiv.org/html/2607.05458#S3.F2)summarizes the pipeline\.

#### Harness state\.

A state vector summarizes the information needed for external control: trajectory progress, draft availability, evidence coverage, tool outputs, verifier feedback, recent failures, remaining budget, last action, and domain\-specific structural flags\. The state does not expose hidden model activations or update the LLM\. It is designed to capture whether the agent has enough information to continue, verify, revise, or stop\.

#### Harness action space\.

The controller selects among seven structural actions:observe,retrieve,call\-tool,draft,check,revise, andsubmit\. Domain adapters implement the semantics of each action and mask invalid actions, while the learned policy shares a common interface across domains\. The action space is intentionally structural: it controls the execution procedure rather than the content of the LLM’s generated answer\.

![Refer to caption](https://arxiv.org/html/2607.05458v1/figures/g1_offline_aw_pipeline.png)Figure 2:Offline harness\-control training pipeline\.A finite rollout buffer supplies task trajectories and terminal rubric rewards\. Advantage\-weighted regression trains a controller over structural harness actions, while the LLM executor remains frozen and process quality is measured only after training\.
#### Task reward and structural verifiers\.

Each domain maps a terminal artifact to criterion\-level rubric scores and a scalar task reward\. The verifier interface is shared, but the criteria are domain\-specific: coding uses tests and patch constraints, research uses answer and evidence criteria, multi\-tool tasks use numerical and tool\-structure criteria, long\-memory tasks use source\-session matching, and planning tasks use state\-transition constraints\. This design keeps the reward tied to task quality rather than to a preferred action pattern\. The multi\-tool verifier is structural and numeric: it validates the final artifact and required tool structure, but it does not claim that every intermediate tool call was uniquely necessary\.

#### Offline advantage\-weighted regression\.

Finite rollout buffers are collected from base and exploratory harnesses\. Terminal rubric rewards define trajectory advantages, and the controller is trained by weighted behavior cloning:

ℒ​\(θ\)\\displaystyle\\mathcal\{L\}\(\\theta\)=−𝔼\(s,a\)∼𝒟\[exp\(A\(s,a\)/β\)\\displaystyle=\-\\mathbb\{E\}\_\{\(s,a\)\\sim\\mathcal\{D\}\}\\big\[\\exp\(A\(s,a\)/\\beta\)\(1\)⋅logπθ\(a∣s\)\]\.\\displaystyle\\qquad\\qquad\\cdot\\log\\pi\_\{\\theta\}\(a\\mid s\)\\big\]\.The exponential weight increases the likelihood of actions observed on higher\-rubric trajectories\. Because learning is offline, AW is expected to recover useful control patterns present in𝒟\\mathcal\{D\}, not to solve exploration gaps by itself\.

#### Harness Maturity Score\.

HMS\\mathrm\{HMS\}is a normalized weighted diagnostic over seven process events: CheckBeforeSubmit, EvidenceBeforeClaim, TestBeforeSubmit, RevisionAfterFailure, ValidToolUse, StopWhenSufficient, and EarlySubmit as a penalty\.HMS\\mathrm\{HMS\}is never used as a training reward\. It is reported to identify which harness behaviors changed after optimizing the terminal task objective\.

#### Calibration choices\.

The EarlySubmit detector uses threshold 0\.25\. Coding uses the same calibrated structural verifier family as the other domains rather than the original strict deterministic coding rubric\. This choice changes the coding baseline and therefore the measured coding gain; Section[5](https://arxiv.org/html/2607.05458#S5)and Appendix[E](https://arxiv.org/html/2607.05458#A5)report the sensitivity analysis\.

#### Controller and training details\.

The harness controller is a one\-hidden\-layer MLP with 64 hidden units and a softmax over seven candidate actions, with invalid actions masked by the domain adapter\. The state vector contains trajectory progress, draft status, rubric coverage, error count, remaining budget, recent test outcomes, and the previous action\. Training uses Adam with learning rate10−310^\{\-3\}, batch size 256, 20 epochs, AW temperatureβ=0\.2\\beta=0\.2, clipped AW weights in\[0\.1,10\.0\]\[0\.1,10\.0\], and entropy regularization coefficient 0\.01\. Three independent seeds are trained per domain\. Hyperparameters are fixed across domains to avoid attributing domain variation to per\-domain tuning\.

#### Evaluation protocol\.

The process\-error normalizerPerrorP\_\{\\mathrm\{error\}\}usesmax\_stepsrather than observed steps, preventing shorter trajectories from receiving a mechanical process advantage\. The same rubric scorer is used to produce terminal training rewards and evaluation scores; calibration mode is used only for scorer\-validation diagnostics\. Multi\-tool criteria are structural and numeric: they verify the final artifact and required tool structure but do not certify ideal intermediate reasoning\. All primary claims report mean changes and effect magnitudes, with bootstrap intervals for final\-quality changes\.

## 4Theory

### 4\.1Task\-Consistent Reward Design

We first require policy consistency: any reward used for harness training should preserve the optimal policies of the original terminal task objective\. The statement below is the finite\-horizon analogue of potential\-based reward shaping\(Nget al\.,[1999](https://arxiv.org/html/2607.05458#bib.bib43)\)\. Our experiments use the special caseΦt≡0\\Phi\_\{t\}\\equiv 0, i\.e\., terminal task\-rubric reward without intermediate action bonuses\.

###### Definition 1\(Task\-rubric return\)\.

Consider a finite\-horizon Harness MDP with horizonHH, initial\-state distributionρ0\\rho\_\{0\}, and trajectoryτ=\(s0,a0,…,sH\)\\tau=\(s\_\{0\},a\_\{0\},\\ldots,s\_\{H\}\)\. The task\-rubric return of a policyπ\\piis

J𝒬​\(π\)=𝔼τ∼pπ​\[𝒬​\(sH\)\],J\_\{\\mathcal\{Q\}\}\(\\pi\)=\\mathbb\{E\}\_\{\\tau\\sim p\_\{\\pi\}\}\\left\[\\mathcal\{Q\}\(s\_\{H\}\)\\right\],\(2\)wherepπp\_\{\\pi\}is the trajectory distribution induced byρ0\\rho\_\{0\}, the harness policyπ\\pi, and the frozen\-LLM transition kernel\.

###### Definition 2\(Potential\-shaped task reward\)\.

Let\{Φt:𝒮→ℝ\}t=0H\\\{\\Phi\_\{t\}:\\mathcal\{S\}\\rightarrow\\mathbb\{R\}\\\}\_\{t=0\}^\{H\}be a sequence of bounded potential functions satisfyingΦH​\(s\)=0\\Phi\_\{H\}\(s\)=0for every terminal statess\. Define the shaped per\-step reward as

rtΦ​\(st,at,st\+1\)\\displaystyle r\_\{t\}^\{\\Phi\}\(s\_\{t\},a\_\{t\},s\_\{t\+1\}\)=rt​\(st,at,st\+1\)\\displaystyle=r\_\{t\}\(s\_\{t\},a\_\{t\},s\_\{t\+1\}\)\(3\)\+Φt\+1​\(st\+1\)−Φt​\(st\),\\displaystyle\\quad\+\\Phi\_\{t\+1\}\(s\_\{t\+1\}\)\-\\Phi\_\{t\}\(s\_\{t\}\),wherert=0r\_\{t\}=0fort<H−1t<H\-1andrH−1​\(sH−1,aH−1,sH\)=𝒬​\(sH\)r\_\{H\-1\}\(s\_\{H\-1\},a\_\{H\-1\},s\_\{H\}\)=\\mathcal\{Q\}\(s\_\{H\}\)\. The corresponding shaped objective is

JΦ​\(π\)=𝔼τ∼pπ​\[∑t=0H−1rtΦ​\(st,at,st\+1\)\]\.J\_\{\\Phi\}\(\\pi\)=\\mathbb\{E\}\_\{\\tau\\sim p\_\{\\pi\}\}\\left\[\\sum\_\{t=0\}^\{H\-1\}r\_\{t\}^\{\\Phi\}\(s\_\{t\},a\_\{t\},s\_\{t\+1\}\)\\right\]\.\(4\)

###### Theorem 1\(Optimal\-policy invariance\)\.

Under Definitions[1](https://arxiv.org/html/2607.05458#Thmdefinition1)and[2](https://arxiv.org/html/2607.05458#Thmdefinition2), there exists a constantCρ0C\_\{\\rho\_\{0\}\}independent ofπ\\pisuch that

JΦ​\(π\)=J𝒬​\(π\)−Cρ0,J\_\{\\Phi\}\(\\pi\)=J\_\{\\mathcal\{Q\}\}\(\\pi\)\-C\_\{\\rho\_\{0\}\},\(5\)whereCρ0=𝔼s0∼ρ0​\[Φ0​\(s0\)\]C\_\{\\rho\_\{0\}\}=\\mathbb\{E\}\_\{s\_\{0\}\\sim\\rho\_\{0\}\}\[\\Phi\_\{0\}\(s\_\{0\}\)\]\. Consequently,

argmaxπJΦ​\(π\)=argmaxπJ𝒬​\(π\)\.\\operatorname\*\{argmax\}\_\{\\pi\}J\_\{\\Phi\}\(\\pi\)=\\operatorname\*\{argmax\}\_\{\\pi\}J\_\{\\mathcal\{Q\}\}\(\\pi\)\.\(6\)

Theorem[1](https://arxiv.org/html/2607.05458#Thmtheorem1)states that task\-rubric reward can be augmented with potential\-based intermediate signals without changing the task\-optimal policy set\. Our experiments use no action\-pattern bonus: checking, revising, or calling tools is valuable only insofar as it contributes to terminal task quality\. The proof is provided in Appendix[A\.1](https://arxiv.org/html/2607.05458#A1.SS1)\.

###### Proposition 1\(Action\-pattern rewards can change the optimum\)\.

There exists a finite\-horizon Harness MDP and an action\-pattern rewardb​\(a\)b\(a\)such that

argmaxπ𝔼π​\[𝒬​\(sH\)\]\\displaystyle\\operatorname\*\{argmax\}\_\{\\pi\}\\mathbb\{E\}\_\{\\pi\}\\left\[\\mathcal\{Q\}\(s\_\{H\}\)\\right\]\(7\)≠argmaxπ𝔼π​\[𝒬​\(sH\)\+b​\(a0\)\]\.\\displaystyle\\qquad\\neq\\operatorname\*\{argmax\}\_\{\\pi\}\\mathbb\{E\}\_\{\\pi\}\\left\[\\mathcal\{Q\}\(s\_\{H\}\)\+b\(a\_\{0\}\)\\right\]\.

Proposition[1](https://arxiv.org/html/2607.05458#Thmproposition1)shows that direct bonuses for actions such ascheckorrevisecan favor process forms that do not improve the final artifact\. Its proof is provided in Appendix[A\.2](https://arxiv.org/html/2607.05458#A1.SS2)\.

### 4\.2Finite\-Buffer Support and Process–Outcome Decoupling

We next characterize the ideal support\-restricted AW update induced by a finite rollout buffer\. The result gives an outcome ceiling and an exact expression for the shift in any bounded process statistic\.

###### Definition 3\(Finite rollout buffer and AW distribution\)\.

LetB=\{τ\(1\),…,τ\(N\)\}B=\\\{\\tau^\{\(1\)\},\\ldots,\\tau^\{\(N\)\}\\\}be a finite rollout buffer with empirical distributionμB​\(τ\)=N−1​∑i=1N𝕀​\{τ=τ\(i\)\}\\mu\_\{B\}\(\\tau\)=N^\{\-1\}\\sum\_\{i=1\}^\{N\}\\mathbb\{I\}\\\{\\tau=\\tau^\{\(i\)\}\\\}\. For terminal returnG:B→ℝG:B\\rightarrow\\mathbb\{R\}, define the empirical mean returnG¯B=𝔼μB​\[G\]\\overline\{G\}\_\{B\}=\\mathbb\{E\}\_\{\\mu\_\{B\}\}\[G\], the support ceilingGB⋆=maxτ∈B⁡G​\(τ\)G\_\{B\}^\{\\star\}=\\max\_\{\\tau\\in B\}G\(\\tau\), and the buffer slackσB=GB⋆−G¯B\\sigma\_\{B\}=G\_\{B\}^\{\\star\}\-\\overline\{G\}\_\{B\}\. LetA:B→ℝA:B\\rightarrow\\mathbb\{R\}be an advantage estimate and letβ\>0\\beta\>0\. Definew​\(τ\)=exp⁡\(A​\(τ\)/β\)w\(\\tau\)=\\exp\(A\(\\tau\)/\\beta\)andZB=𝔼μB​\[w\]Z\_\{B\}=\\mathbb\{E\}\_\{\\mu\_\{B\}\}\[w\]\. The support\-restricted AW distribution is

qAW​\(τ\)=μB​\(τ\)​w​\(τ\)ZB\.q\_\{\\mathrm\{AW\}\}\(\\tau\)=\\frac\{\\mu\_\{B\}\(\\tau\)w\(\\tau\)\}\{Z\_\{B\}\}\.\(8\)

###### Theorem 2\(Finite\-buffer outcome and process characterization\)\.

LetqAWq\_\{\\mathrm\{AW\}\}be defined by Equation[8](https://arxiv.org/html/2607.05458#S4.E8)\.

\(i\) Outcome bound\.DefineΔ​𝒬B=𝔼qAW​\[G\]−𝔼μB​\[G\]\\Delta\\mathcal\{Q\}\_\{B\}=\\mathbb\{E\}\_\{q\_\{\\mathrm\{AW\}\}\}\[G\]\-\\mathbb\{E\}\_\{\\mu\_\{B\}\}\[G\]\. Then

𝔼qAW​\[G\]≤GB⋆,Δ​𝒬B≤σB\.\\mathbb\{E\}\_\{q\_\{\\mathrm\{AW\}\}\}\[G\]\\leq G\_\{B\}^\{\\star\},\\qquad\\Delta\\mathcal\{Q\}\_\{B\}\\leq\\sigma\_\{B\}\.\(9\)
\(ii\) Process identity\.For any bounded process statisticΨ:B→ℝ\\Psi:B\\rightarrow\\mathbb\{R\}, defineΔ​ΨB=𝔼qAW​\[Ψ\]−𝔼μB​\[Ψ\]\\Delta\\Psi\_\{B\}=\\mathbb\{E\}\_\{q\_\{\\mathrm\{AW\}\}\}\[\\Psi\]\-\\mathbb\{E\}\_\{\\mu\_\{B\}\}\[\\Psi\]\. Then

Δ​ΨB=CovμB⁡\(w,Ψ\)𝔼μB​\[w\]\.\\Delta\\Psi\_\{B\}=\\frac\{\\operatorname\{Cov\}\_\{\\mu\_\{B\}\}\(w,\\Psi\)\}\{\\mathbb\{E\}\_\{\\mu\_\{B\}\}\[w\]\}\.\(10\)Since𝔼μB​\[w\]\>0\\mathbb\{E\}\_\{\\mu\_\{B\}\}\[w\]\>0,Δ​ΨB≥0\\Delta\\Psi\_\{B\}\\geq 0if and only ifCovμB⁡\(w,Ψ\)≥0\\operatorname\{Cov\}\_\{\\mu\_\{B\}\}\(w,\\Psi\)\\geq 0\.

###### Corollary 1\(Process improvement under monotone advantage weighting\)\.

Suppose there exists a measurable non\-decreasing functionh:ℝ→ℝh:\\mathbb\{R\}\\rightarrow\\mathbb\{R\}such that𝔼μB​\[Ψ​\(τ\)∣A​\(τ\)\]=h​\(A​\(τ\)\)\\mathbb\{E\}\_\{\\mu\_\{B\}\}\[\\Psi\(\\tau\)\\mid A\(\\tau\)\]=h\(A\(\\tau\)\)\. ThenCovμB⁡\(w,Ψ\)≥0\\operatorname\{Cov\}\_\{\\mu\_\{B\}\}\(w,\\Psi\)\\geq 0, and henceΔ​ΨB≥0\\Delta\\Psi\_\{B\}\\geq 0\.

###### Proposition 2\(Terminal\-return correlation does not determine the process shift\)\.

There exist finite buffers, returnsGG, process statisticsΨ\\Psi, and advantage estimatesAAsuch that

CovμB⁡\(G,Ψ\)<0,CovμB⁡\(w,Ψ\)\>0\.\\operatorname\{Cov\}\_\{\\mu\_\{B\}\}\(G,\\Psi\)<0,\\qquad\\operatorname\{Cov\}\_\{\\mu\_\{B\}\}\(w,\\Psi\)\>0\.\(11\)For such a construction, Equation[10](https://arxiv.org/html/2607.05458#S4.E10)impliesΔ​ΨB\>0\\Delta\\Psi\_\{B\}\>0despite a negative return–process covariance\.

Theorem[2](https://arxiv.org/html/2607.05458#Thmtheorem2)separates the two channels of offline harness learning\. Outcome improvement is bounded by the return support of the buffer\. Process improvement is governed by the covariance between the process statistic and the AW weights\. In the experiments,Ψ\\Psiis either an individual harness event or aggregateHMS\\mathrm\{HMS\}; for domainDD,σD:=σBD\\sigma\_\{D\}:=\\sigma\_\{B\_\{D\}\}denotes empirical buffer slack\. Proofs are provided in Appendix[B](https://arxiv.org/html/2607.05458#A2), and Figure[5](https://arxiv.org/html/2607.05458#S5.F5)evaluates the empirical relationship between slack and outcome improvement\.

## 5Experiments

The experiments test whether a learned harness controller changes the way a frozen LLM agent works, and whether those process changes improve final task quality\. We evaluate six controlled domains: knowledge\-work, coding, research QA, multi\-tool, long\-memory, and planning\. Each domain has 100 human\-annotated tasks, with 80 used for buffer collection and 20 held out for evaluation\. The split is stratified by difficulty and uses disjoint task identifiers, templates, entities, and starter artifacts; Appendix[D](https://arxiv.org/html/2607.05458#A4)details the construction and leakage controls\. We also evaluate two public\-benchmark adapters:τ\\tau\-bench retail for policy\-constrained tool interaction\(Yaoet al\.,[2024](https://arxiv.org/html/2607.05458#bib.bib5)\), and AgentBench DB\-Bench for interactive database reasoning\(Liuet al\.,[2024](https://arxiv.org/html/2607.05458#bib.bib1)\)\. These adapters map selected tasks into the Harness MDP so that we can isolate harness control rather than claim official benchmark scores\. Each adapter uses 16 training tasks and 20 held\-out evaluation tasks\. All evaluations use three independent seeds and three rollouts per held\-out task\.

We compare AW with behavior cloning \(BC\) and Forced CHECK \(FC\)\. BC tests whether copying observed trajectories is enough\(Pomerleau,[1989](https://arxiv.org/html/2607.05458#bib.bib31); Rosset al\.,[2011](https://arxiv.org/html/2607.05458#bib.bib32)\); FC tests whether simply adding verification explains the gains\. Table[1](https://arxiv.org/html/2607.05458#S5.T1)summarizes the empirical message: AW reliably learns verification behavior, but final\-quality gains appear only when the offline buffer contains useful high\-return trajectories\.

Table 1:Main empirical findings\.Offline AW consistently improves harness verification behavior, while final\-quality gains are concentrated in settings where the rollout buffer contains stronger high\-return support\. Adapter results are evaluated under our Harness MDP scoring protocols and are not official upstream benchmark scores\.![Refer to caption](https://arxiv.org/html/2607.05458v1/x1.png)Figure 3:Outcome and process gains across eight settings\.The left panel reports final\-quality change, and the right panel reports process\-maturity change\. The adapters and coding show the largest outcome gains, while process gains are broader and are mainly driven by verification before submission\.#### Process control\.

We first ask whether AW changes the execution procedure at all\. The clearest answer is verification before submission\. Under the base harness, CheckBeforeSubmit is nearly absent across the controlled domains and the two adapter evaluations\. After AW training, Table[2](https://arxiv.org/html/2607.05458#S5.T2)shows that the event appears in every setting\. This is the central process result: the controller learns a reusable control habit from offline trajectories even when the downstream score change is small\.

Table 2:Verification before submission\.CheckBeforeSubmit \(CBS\) is nearly absent under the base harness but increases after AW in every controlled domain and both public\-benchmark adapters, showing that the learned controller changes the execution process itself\.
#### Outcome transfer\.

Process gains do not automatically imply better final artifacts, so we next examine final task quality\. Figure[3](https://arxiv.org/html/2607.05458#S5.F3)and Table[3](https://arxiv.org/html/2607.05458#S5.T3)show that the strongest outcome gains occur in the two public\-benchmark adapters and in coding\. Theτ\\tau\-bench retail adapter improves most under our plan\-quality rubric, AgentBench DB\-Bench also improves, and coding is the strongest controlled\-domain result under the calibrated structural verifier\. These settings support the main positive claim: when the buffer contains trajectories that connect better control decisions to better terminal artifacts, AW converts process learning into final\-quality improvement\.

Table 3:Settings with the clearest final\-quality gains\.Changes inGGand bootstrap intervals are reported in percentage points\. CBS denotes the CheckBeforeSubmit rate under AW\. The two adapter rows use 20\-task held\-out splits and adapted Harness MDP scoring protocols rather than official upstream benchmark scores\.SettingBaseGGAWGGΔ​G\\Delta GCBSτ\\tau\-bench retail0\.3370\.519\+18\.2\\mathbf\{\+18\.2\}17\.2%interval\[\+15\.1\+15\.1,\+21\.3\+21\.3\]DB\-Bench0\.4150\.547\+13\.2\\mathbf\{\+13\.2\}16\.7%interval\[\+10\.2\+10\.2,\+16\.2\+16\.2\]Coding0\.7120\.812\+10\.0\\mathbf\{\+10\.0\}17\.8%interval\[\+5\.9\+5\.9,\+14\.6\+14\.6\]
#### Baseline ablations\.

As shown in Table[4](https://arxiv.org/html/2607.05458#S5.T4)and Figure[4](https://arxiv.org/html/2607.05458#S5.F4), AW provides stronger gains than BC in all eight settings and outperforms both BC and FC in five\. This rules out two simpler explanations\. The effect is not pure imitation, because BC is consistently weaker; it is not mechanical verification, because FC does not reproduce the adapter and coding gains\. AW’s advantage is state dependence: the controller learns when checking and revision are useful rather than applying them uniformly\.

Table 4:Ablation results for advantage weighting \(AW\), behavior cloning \(BC\), and Forced CHECK \(FC\)\.Each lift is the change in final task quality, measured in percentage points, relative to the method’s own Base evaluation\.Δ​ΔAW−BC\\Delta\\Delta\_\{\\mathrm\{AW\-BC\}\}andΔ​ΔAW−FC\\Delta\\Delta\_\{\\mathrm\{AW\-FC\}\}denote the differences between the AW lift and the corresponding baseline lift; positive values indicate that AW performs better\. Bold values mark positive AW advantages\. AW outperforms both baselines in knowledge\-work, coding, planning,τ\\tau\-bench retail, and DB\-Bench; coding is the only domain with a fully objective verifier\.![Refer to caption](https://arxiv.org/html/2607.05458v1/x2.png)Figure 4:Policy comparison against simpler baselines\.Each point is the within\-run lift over that method’s own Base evaluation\. AW outperforms BC in all settings and beats Forced CHECK in the strongest\-gain settings, indicating that learned state\-dependent control is more effective than imitation or uniformly inserted checking\.
#### Difficulty\-stratified performance\.

On the held\-out adapters, AW gains remain positive across the available task\-complexity strata\. The improvements persist on more complex tasks, suggesting that the learned harness policy is not merely exploiting easy instances\.

#### Controlled\-domain results\.

Table[5](https://arxiv.org/html/2607.05458#S5.T5)reports the full results, and Table[6](https://arxiv.org/html/2607.05458#S5.T6)provides a compact interpretation\. Coding shows the clearest calibrated\-protocol gain\. Under the original strict deterministic coding rubric, the base harness was already near saturation, leaving little room for improvement\. The reported \+10\.0 points therefore depends on the calibrated structural verifier, which supports cross\-domain comparison by scoring parseability, safety constraints, cost compliance, testing, and checking\. Planning and knowledge\-work show smaller gains, while the remaining domains are near zero or slightly negative\. AggregateHMS\\mathrm\{HMS\}improves in five of six domains, mainly through verification before submission; research is the exception because EarlySubmit offsets the checking gain\.

Table 5:Controlled\-domain results\.Each policy is evaluated on 20 held\-out tasks with three seeds and three rollouts per task \(N=180N=180\)\.GGis the mean task\-rubric score,HMS\\mathrm\{HMS\}is the process\-maturity score, and CBS is the CheckBeforeSubmit rate under AW\. Bootstrap intervals show uncertainty forΔ​G\\Delta G; the largest controlled\-domain quality gain occurs in coding under the calibrated structural verifier\.Table 6:Interpretation of controlled\-domain behavior\.Under the calibrated structural verifier, coding is the strongest final\-quality result, while most domains show process improvement without comparable outcome movement\. Research is the exception because increased EarlySubmit offsets its checking gain\.
#### Event\-level process changes\.

Table[7](https://arxiv.org/html/2607.05458#S5.T7)decomposesHMS\\mathrm\{HMS\}into seven process events\. The overall improvement is driven mainly by CheckBeforeSubmit, which increases from 1 of 1,080 base episodes to 113 of 1,080 AW episodes across the controlled suite\. TestBeforeSubmit also improves in coding\. In contrast, EarlySubmit worsens in several settings, most notably in research, while other events remain sparse or saturated\. Thus, the observed process gain is concentrated in verification before submission rather than reflecting uniform improvement across all dimensions of process maturity\.

Table 7:Event\-level process decomposition\.Rates are reported as base→\\rightarrowAW under the calibrated detector\. EarlySubmit is a penalty, so increases indicate process degradation\. The table shows that the aggregateHMS\\mathrm\{HMS\}gain is mainly a verification\-before\-submission effect rather than a uniform improvement across all process events\.
#### Benchmark scope\.

We include public benchmarks only when the harness abstraction can be applied without changing the core task\. Native benchmark interfaces usually evaluate the full agent stack, whereas our adapters isolate the harness\-control layer\. Theτ\\tau\-bench retail adapter preserves policy\-constrained tool decisions but uses an adapted plan\-quality rubric rather than the native simulator score\. The AgentBench DB\-Bench adapter preserves database reasoning and deliberative control but not the full upstream evaluator\. We therefore treat these results as external validation of the learned control pattern, not as official benchmark scores\. We do not report results on AgentBench OS\-Interaction, SWE\-bench Lite, or GAIA L1/L2 because their native evaluation interfaces were unavailable in our setup or could not be represented faithfully by the current plan\-generation harness\.

#### Calibration and support diagnostics\.

Figure[5](https://arxiv.org/html/2607.05458#S5.F5)relates final\-quality change to buffer slackσD\\sigma\_\{D\}under two verifier configurations\. The original strict domain\-specific verifier makes the coding baseline nearly saturated, which produces a strong apparent relationship between slack and outcome gain\. The calibrated structural verifier replaces that saturated coding rubric with the same scoring family used across domains; the relationship then weakens because coding moves from zero to moderate estimated slack\. Buffer slack is therefore useful as an offline\-support diagnostic, but it should not be treated as a calibration\-invariant explanation of final\-quality improvement\.

![Refer to caption](https://arxiv.org/html/2607.05458v1/x3.png)Figure 5:Buffer slack and final\-quality change under two verifier configurations\.Hollow markers show the original strict domain\-specific verifier, including the nearly saturated deterministic coding rubric; filled markers show the calibrated structural verifier used across domains with EarlySubmit threshold 0\.25\. Recalibrating the coding rubric moves coding from zero to moderate estimated slack and substantially weakens the cross\-domain relationship\.Table[8](https://arxiv.org/html/2607.05458#S5.T8)reports sensitivity to the EarlySubmit threshold used by the calibrated structural detector\. Larger thresholds classify more submissions as early\. At a threshold of 0\.35, the detector marks all research episodes under the base harness as EarlySubmit, producing an artificially large positive change inHMS\\mathrm\{HMS\}\. We use 0\.25 because it avoids this saturation while preserving the qualitative process conclusions\.

Table 8:EarlySubmit threshold sensitivity\.Entries are changes inHMS\\mathrm\{HMS\}\. Higher thresholds make the research\-domain result appear more positive by labeling more base episodes as EarlySubmit; the primary threshold avoids this saturation\.
#### Process\-quality microstructure\.

Figure[6](https://arxiv.org/html/2607.05458#S5.F6)shows that the within\-domain relationship between final quality andHMS\\mathrm\{HMS\}changes sign across domains\. A negative correlation does not prevent process improvement under AW because AW reweights trajectory regions rather than fitting a linear relationship betweenGGandHMS\\mathrm\{HMS\}\. This is consistent with Proposition[2](https://arxiv.org/html/2607.05458#Thmproposition2): process maturity can improve even when final quality is flat or when the local correlation structure is mixed\. Figure[7](https://arxiv.org/html/2607.05458#S5.F7)gives the complementary per\-setting view, showing that the adapter settings combine outcome and process gains while the controlled domains exhibit a weaker outcome response\.

![Refer to caption](https://arxiv.org/html/2607.05458v1/x4.png)Figure 6:Within\-policy process–outcome correlations\.Correlationsρ​\(G,HMS\)\\rho\(G,\\mathrm\{HMS\}\)vary in sign across domains\. The pooled within\-domain estimate is0\.1830\.183, whereas naive pooling across all episodes yields0\.4560\.456, illustrating how between\-domain heterogeneity can obscure the local relationship between process and final quality\.![Refer to caption](https://arxiv.org/html/2607.05458v1/x5.png)Figure 7:Per\-setting relationship between outcome and process gains\.Public\-benchmark adapter settings show both outcome and aggregate process gains, while controlled\-domain settings show more reliable verification gains than final\-quality gains\.

## 6Discussion

The results support a process\-control interpretation of harness\-level offline RL\. A frozen LLM agent exposes an optimizable control layer distinct from the model’s language and reasoning capacity: AW reliably changes when the harness checks, revises, and submits, even when final task quality is nearly unchanged\.

#### Process and outcome channels\.

Offline AW reweights actions observed in the rollout buffer\. Recurring control patterns, especially checking before submission, can therefore be recovered across many tasks\. Final\-quality gains additionally require trajectories whose actions lead to better terminal artifacts\. This accounts for the larger gains in coding,τ\\tau\-bench retail, and DB\-Bench, and for the controlled domains where verification increases without comparable final\-score movement\.

#### Specificity of the process shift\.

The process improvement is not a broad increase in allHMS\\mathrm\{HMS\}events\. It is concentrated in CheckBeforeSubmit, with smaller support from TestBeforeSubmit in coding\. Some events are sparse, saturated, or insensitive under the current detector\. The localized shift identifies the concrete behavior the controller learned and prevents overclaiming that the agent became generally more mature across all process dimensions\.

#### External validation through adapters\.

Theτ\\tau\-bench retail and AgentBench DB\-Bench results show that the learned control pattern can help outside the controlled suite\. However, both evaluations use adapted scoring protocols and preserve only the benchmark components needed for Harness MDP control\. They support external validity of the harness\-control mechanism, not a benchmark\-level claim about the upstream tasks\.

## 7Conclusion

We formulate the control layer around a frozen LLM agent as a Harness MDP and train a lightweight offline AW controller over structural harness actions\. The learned controller consistently increases verification before submission, while final\-quality gains are concentrated in coding and two adapted public\-benchmark settings; the latter are adapter\-level results rather than official upstream benchmark scores\. This process–outcome gap reflects the finite\-buffer setting: recurring verification behavior can be learned broadly, but outcome gains require stronger trajectories in the offline data\. Future work should move beyond fixed\-buffer support through online improvement or targeted data collection, adopt native benchmark protocols, and develop broader, better\-calibrated measures of harness process maturity\.

## References

- L\. A\. Agrawal, S\. Tan, D\. Soylu, N\. Ziems, R\. Khare, K\. Opsahl\-Ong, A\. Singhvi, H\. Shandilya, M\. J\. Ryan, M\. Jiang, C\. Potts, K\. Sen, A\. G\. Dimakis, I\. Stoica, D\. Klein, M\. Zaharia, and O\. Khattab \(2025\)GEPA: reflective prompt evolution can outperform reinforcement learning\.arXiv preprint arXiv:2507\.19457\.Cited by:[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px2.p1.1)\.
- D\. Brandfonbrener, W\. Whitney, R\. Ranganath, and J\. Bruna \(2021\)Offline RL without off\-policy evaluation\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.34\.Cited by:[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px4.p1.1)\.
- W\. Chen, Y\. Su, J\. Zuo, C\. Yang, C\. Yuan, C\. Chan, H\. Yu, Y\. Lu, Y\. Hung, C\. Qian, Y\. Qin, X\. Cong, R\. Xie, Z\. Liu, M\. Sun, and J\. Zhou \(2024\)AgentVerse: facilitating multi\-agent collaboration and exploring emergent behaviors\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.05458#S1.p2.1)\.
- S\. Hong, M\. Zhuge, J\. Chen, X\. Zheng, Y\. Cheng, C\. Zhang, J\. Wang, Z\. Wang, S\. K\. S\. Yau, Z\. Lin, L\. Zhou, C\. Ran, L\. Xiao, C\. Wu, and J\. Schmidhuber \(2024\)MetaGPT: meta programming for a multi\-agent collaborative framework\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.05458#S1.p2.1)\.
- C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. Narasimhan \(2024\)SWE\-bench: can language models resolve real\-world GitHub issues?\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[Appendix D](https://arxiv.org/html/2607.05458#A4.p1.1),[§1](https://arxiv.org/html/2607.05458#S1.p1.1),[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px1.p1.1)\.
- O\. Khattab, A\. Singhvi, P\. Maheshwari, Z\. Zhang, K\. Santhanam, S\. Vardhamanan, S\. Haq, A\. Sharma, T\. T\. Joshi, H\. Moazam, H\. Miller, M\. Zaharia, and C\. Potts \(2023\)DSPy: compiling declarative language model calls into self\-improving pipelines\.arXiv preprint arXiv:2310\.03714\.Cited by:[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px2.p1.1)\.
- I\. Kostrikov, A\. Nair, and S\. Levine \(2022\)Offline reinforcement learning with implicit Q\-learning\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px4.p1.1)\.
- J\. Kuang, Y\. Li, X\. Zhang, Y\. Li, D\. Yin, X\. Sun, Y\. Shen, and P\. S\. Yu \(2025\)Process\-level trajectory evaluation for environment configuration in software engineering agents\.arXiv preprint arXiv:2510\.25694\.Cited by:[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px5.p1.1)\.
- S\. Levine, A\. Kumar, G\. Tucker, and J\. Fu \(2020\)Offline reinforcement learning: tutorial, review, and perspectives on open problems\.arXiv preprint arXiv:2005\.01643\.Cited by:[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px4.p1.1)\.
- J\. Lin, S\. Liu, C\. Pan, L\. Lin, S\. Dou, Z\. Xi, X\. Huang, H\. Yan, Z\. Han, T\. Gui, and Y\. Jiang \(2026\)Agentic harness engineering: observability\-driven automatic evolution of coding\-agent harnesses\.arXiv preprint arXiv:2604\.25850\.Cited by:[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px2.p1.1)\.
- X\. Liu, H\. Yu, H\. Zhang, Y\. Xu, X\. Lei, H\. Lai, Y\. Gu, H\. Ding, K\. Men, K\. Yang, S\. Zhang, X\. Deng, A\. Zeng, Z\. Du, C\. Zhang, S\. Shen, T\. Zhang, Y\. Su, H\. Sun, M\. Huang, Y\. Dong, and J\. Tang \(2024\)AgentBench: evaluating LLMs as agents\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[Appendix C](https://arxiv.org/html/2607.05458#A3.SS0.SSS0.Px1.p1.3),[§1](https://arxiv.org/html/2607.05458#S1.p1.1),[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2607.05458#S5.p1.1)\.
- J\. Lu, T\. Holleis, Y\. Zhang, B\. Aumayer, F\. Nan, F\. Bai, S\. Ma, S\. Ma, M\. Li, G\. Yin, Z\. Wang, and R\. Pang \(2024\)ToolSandbox: a stateful, conversational, interactive evaluation benchmark for LLM tool use capabilities\.arXiv preprint arXiv:2408\.04682\.Cited by:[§1](https://arxiv.org/html/2607.05458#S1.p1.1),[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Maharana, D\. Lee, S\. Tulyakov, M\. Bansal, F\. Barbieri, and Y\. Fang \(2024\)Evaluating very long\-term conversational memory of llm agents\.External Links:2402\.17753,[Link](https://arxiv.org/abs/2402.17753)Cited by:[Appendix D](https://arxiv.org/html/2607.05458#A4.p1.1),[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px1.p1.1)\.
- G\. Mialon, C\. Fourrier, C\. Swift, T\. Wolf, Y\. LeCun, and T\. Scialom \(2023\)GAIA: a benchmark for general AI assistants\.arXiv preprint arXiv:2311\.12983\.Cited by:[§1](https://arxiv.org/html/2607.05458#S1.p1.1),[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Nair, A\. Gupta, M\. Dalal, and S\. Levine \(2021\)Accelerating online reinforcement learning with offline datasets\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px4.p1.1)\.
- A\. Y\. Ng, D\. Harada, and S\. J\. Russell \(1999\)Policy invariance under reward transformations: theory and application to reward shaping\.InProceedings of the 16th International Conference on Machine Learning \(ICML\),pp\. 278–287\.Cited by:[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px5.p1.1),[§4\.1](https://arxiv.org/html/2607.05458#S4.SS1.p1.1)\.
- W\. Pan, S\. Liu, C\. Lin, J\. Zeng, X\. Tang, X\. Zhou, Y\. Lu, and X\. Jia \(2026\)Retrospective harness optimization: improving llm agents via self\-preference over trajectory rollouts\.arXiv preprint arXiv:2606\.05922\.Cited by:[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px2.p1.1)\.
- M\. Pandya, K\. Zhang, and B\. Lyu \(2026\)Strained coherence: a pre\-failure signal in coding agent execution trajectories\.arXiv preprint arXiv:2606\.07889\.Cited by:[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px5.p1.1)\.
- X\. B\. Peng, A\. Kumar, G\. Zhang, and S\. Levine \(2019\)Advantage\-weighted regression: simple and scalable off\-policy reinforcement learning\.arXiv preprint arXiv:1910\.00177\.Cited by:[Appendix C](https://arxiv.org/html/2607.05458#A3.SS0.SSS0.Px1.p1.3),[§1](https://arxiv.org/html/2607.05458#S1.p4.1),[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px4.p1.1)\.
- D\. A\. Pomerleau \(1989\)ALVINN: an autonomous land vehicle in a neural network\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px4.p1.1),[§5](https://arxiv.org/html/2607.05458#S5.p2.1)\.
- A\. Prasad, A\. Koller, M\. Hartmann, P\. Clark, A\. Sabharwal, M\. Bansal, and T\. Khot \(2024\)ADaPT: as\-needed decomposition and planning with language models\.External Links:2311\.05772,[Link](https://arxiv.org/abs/2311.05772)Cited by:[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Qin, S\. Liang, Y\. Ye, K\. Zhu, L\. Yan, Y\. Lu, Y\. Lin, X\. Cong, X\. Tang, B\. Qian, S\. Zhao, L\. Hong, R\. Tian, R\. Xie, J\. Zhou, M\. Gerstein, D\. Li, Z\. Liu, and M\. Sun \(2023\)ToolLLM: facilitating large language models to master 16000\+ real\-world APIs\.arXiv preprint arXiv:2307\.16789\.Cited by:[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Ross, G\. J\. Gordon, and J\. A\. Bagnell \(2011\)A reduction of imitation learning and structured prediction to no\-regret online learning\.InInternational Conference on Artificial Intelligence and Statistics \(AISTATS\),Cited by:[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px4.p1.1),[§5](https://arxiv.org/html/2607.05458#S5.p2.1)\.
- T\. Schick, J\. Dwivedi\-Yu, R\. Dessì, R\. Raileanu, M\. Lomeli, L\. Zettlemoyer, N\. Cancedda, and T\. Scialom \(2023\)Toolformer: language models can teach themselves to use tools\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§1](https://arxiv.org/html/2607.05458#S1.p1.1)\.
- H\. Seong, L\. Yin, H\. Zhang, and Z\. Shi \(2026\)The last harness you’ll ever build\.External Links:2604\.21003,[Link](https://arxiv.org/abs/2604.21003)Cited by:[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px2.p1.1)\.
- N\. Shinn, F\. Cassano, E\. Berman, A\. Gopinath, K\. Narasimhan, and S\. Yao \(2023\)Reflexion: language agents with verbal reinforcement learning\.External Links:2303\.11366,[Link](https://arxiv.org/abs/2303.11366)Cited by:[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px3.p1.1)\.
- A\. Tornede, L\. Purucker, T\. Giglberger, T\. Tornede, J\. Beel, and F\. Hutter \(2025\)AutoPDL: automatic prompt optimization for LLM agents\.arXiv preprint arXiv:2504\.04365\.Cited by:[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px2.p1.1)\.
- G\. Wang, Y\. Xie, Y\. Jiang, A\. Mandlekar, C\. Xiao, Y\. Zhu, L\. Fan, and A\. Anandkumar \(2023\)Voyager: an open\-ended embodied agent with large language models\.Transactions on Machine Learning Research \(TMLR\)\.Cited by:[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Wang, L\. Yang, G\. Li, M\. Wang, and B\. Aragam \(2025\)ScoreFlow: mastering LLM agent workflows via score\-based preference optimization\.arXiv preprint arXiv:2502\.04306\.Cited by:[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px2.p1.1)\.
- Q\. Wu, G\. Bansal, J\. Zhang, Y\. Wu, S\. Zhang, E\. Zhu, B\. Li, L\. Jiang, X\. Zhang, and C\. Wang \(2023\)AutoGen: enabling next\-gen LLM applications via multi\-agent conversation\.Cited by:[§1](https://arxiv.org/html/2607.05458#S1.p1.1),[§1](https://arxiv.org/html/2607.05458#S1.p2.1)\.
- Z\. Xi, Y\. Ding, W\. Chen, B\. Hong, H\. Guo, J\. Wang, D\. Yang, C\. Liao, X\. Guo, W\. He, S\. Gao, L\. Chen, R\. Zheng, Y\. Zou, T\. Gui, Q\. Zhang, X\. Qiu, X\. Huang, Z\. Wu, and Y\. Jiang \(2024\)AgentGym: evolving large language model\-based agents across diverse environments\.arXiv preprint arXiv:2406\.04151\.Cited by:[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px3.p1.1)\.
- C\. Yang, X\. Wang, Y\. Lu, H\. Liu, Q\. V\. Le, D\. Zhou, and X\. Chen \(2024a\)Large language models as optimizers\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px2.p1.1)\.
- J\. Yang, C\. E\. Jimenez, A\. Wettig, K\. Lieret, S\. Yao, K\. Narasimhan, and O\. Press \(2024b\)SWE\-agent: agent\-computer interfaces enable automated software engineering\.arXiv preprint arXiv:2405\.15793\.Cited by:[§1](https://arxiv.org/html/2607.05458#S1.p1.1)\.
- Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. W\. Cohen, R\. Salakhutdinov, and C\. D\. Manning \(2018\)HotpotQA: a dataset for diverse, explainable multi\-hop question answering\.InConference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[Appendix D](https://arxiv.org/html/2607.05458#A4.p1.1)\.
- S\. Yao, H\. Chen, J\. Yang, and K\. Narasimhan \(2022\)WebShop: towards scalable real\-world web interaction with grounded language agents\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[Appendix D](https://arxiv.org/html/2607.05458#A4.p1.1),[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Yao, N\. Shinn, P\. Razavi, and K\. Narasimhan \(2024\)τ\\tau\-bench: a benchmark for tool\-agent\-user interaction in real\-world domains\.arXiv preprint arXiv:2406\.12045\.Cited by:[Appendix C](https://arxiv.org/html/2607.05458#A3.SS0.SSS0.Px1.p1.3),[§1](https://arxiv.org/html/2607.05458#S1.p1.1),[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2607.05458#S5.p1.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao \(2023\)ReAct: synergizing reasoning and acting in language models\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.05458#S1.p1.1)\.
- M\. Yuksekgonul, F\. Bianchi, J\. Boen, S\. Liu, Z\. Huang, C\. Guestrin, and J\. Zou \(2024\)TextGrad: automatic “differentiation” via text\.arXiv preprint arXiv:2406\.07496\.Cited by:[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px2.p1.1)\.
- J\. Zhang, J\. Xiang, Z\. Yu, F\. Teng, X\. Chen, J\. Chen, M\. Zhuge, X\. Cheng, S\. Hong, J\. Wang, B\. Zheng, B\. Liu, Y\. Luo, and C\. Wu \(2024\)AFlow: automating agentic workflow generation\.arXiv preprint arXiv:2410\.10762\.Cited by:[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px2.p1.1)\.
- W\. Zhang, X\. Li, K\. Dong, Y\. Wang, P\. Jia, X\. Li, Y\. Zhang, D\. Xu, Z\. Du, H\. Guo, R\. Tang, and X\. Zhao \(2025\)Process vs\. outcome reward: which is better for agentic RAG reinforcement learning\.arXiv preprint arXiv:2505\.14069\.Cited by:[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px5.p1.1)\.
- A\. Zhou, K\. Yan, M\. Shlapentokh\-Rothman, H\. Wang, and Y\. Wang \(2024a\)Language agent tree search unifies reasoning, acting and planning in language models\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Zhou, F\. F\. Xu, H\. Zhu, X\. Zhou, R\. Lo, A\. Sridhar, X\. Cheng, T\. Ou, Y\. Bisk, D\. Fried, U\. Alon, and G\. Neubig \(2024b\)WebArena: a realistic web environment for building autonomous agents\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Zhou, A\. I\. Muresanu, Z\. Han, K\. Paster, S\. Pitis, H\. Chan, and J\. Ba \(2023\)Large language models are human\-level prompt engineers\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2607.05458#S2.SS0.SSS0.Px2.p1.1)\.

## Appendix AProofs and Mathematical Background

### A\.1Proof of Theorem[1](https://arxiv.org/html/2607.05458#Thmtheorem1)

For any trajectoryτ=\(s0,a0,…,sH\)\\tau=\(s\_\{0\},a\_\{0\},\\ldots,s\_\{H\}\), the cumulative shaped return defined in Equation[4](https://arxiv.org/html/2607.05458#S4.E4)satisfies

∑t=0H−1rtΦ​\(st,at,st\+1\)\\displaystyle\\sum\_\{t=0\}^\{H\-1\}r\_\{t\}^\{\\Phi\}\(s\_\{t\},a\_\{t\},s\_\{t\+1\}\)=∑t=0H−1rt​\(st,at,st\+1\)\\displaystyle=\\sum\_\{t=0\}^\{H\-1\}r\_\{t\}\(s\_\{t\},a\_\{t\},s\_\{t\+1\}\)\(12\)\+∑t=0H−1Φt\+1​\(st\+1\)\\displaystyle\\quad\+\\sum\_\{t=0\}^\{H\-1\}\\Phi\_\{t\+1\}\(s\_\{t\+1\}\)−∑t=0H−1Φt​\(st\)\.\\displaystyle\\quad\-\\sum\_\{t=0\}^\{H\-1\}\\Phi\_\{t\}\(s\_\{t\}\)\.The potential terms telescope as

∑t=0H−1\(Φt\+1​\(st\+1\)−Φt​\(st\)\)\\displaystyle\\sum\_\{t=0\}^\{H\-1\}\\left\(\\Phi\_\{t\+1\}\(s\_\{t\+1\}\)\-\\Phi\_\{t\}\(s\_\{t\}\)\\right\)\(13\)=ΦH​\(sH\)−Φ0​\(s0\)\.\\displaystyle\\qquad=\\Phi\_\{H\}\(s\_\{H\}\)\-\\Phi\_\{0\}\(s\_\{0\}\)\.By Definition[2](https://arxiv.org/html/2607.05458#Thmdefinition2),ΦH​\(sH\)=0\\Phi\_\{H\}\(s\_\{H\}\)=0, while the unshaped cumulative reward equals𝒬​\(sH\)\\mathcal\{Q\}\(s\_\{H\}\)\. Hence,

∑t=0H−1rtΦ​\(st,at,st\+1\)=𝒬​\(sH\)−Φ0​\(s0\)\.\\sum\_\{t=0\}^\{H\-1\}r\_\{t\}^\{\\Phi\}\(s\_\{t\},a\_\{t\},s\_\{t\+1\}\)=\\mathcal\{Q\}\(s\_\{H\}\)\-\\Phi\_\{0\}\(s\_\{0\}\)\.\(14\)Taking expectation under the trajectory distribution induced byπ\\pigives

JΦ​\(π\)\\displaystyle J\_\{\\Phi\}\(\\pi\)=𝔼τ∼pπ​\[𝒬​\(sH\)\]\\displaystyle=\\mathbb\{E\}\_\{\\tau\\sim p\_\{\\pi\}\}\[\\mathcal\{Q\}\(s\_\{H\}\)\]\(15\)−𝔼s0∼ρ0​\[Φ0​\(s0\)\]\.\\displaystyle\\quad\-\\mathbb\{E\}\_\{s\_\{0\}\\sim\\rho\_\{0\}\}\[\\Phi\_\{0\}\(s\_\{0\}\)\]\.LetCρ0=𝔼s0∼ρ0​\[Φ0​\(s0\)\]C\_\{\\rho\_\{0\}\}=\\mathbb\{E\}\_\{s\_\{0\}\\sim\\rho\_\{0\}\}\[\\Phi\_\{0\}\(s\_\{0\}\)\]\. By Definition[1](https://arxiv.org/html/2607.05458#Thmdefinition1),

JΦ​\(π\)=J𝒬​\(π\)−Cρ0\.J\_\{\\Phi\}\(\\pi\)=J\_\{\\mathcal\{Q\}\}\(\\pi\)\-C\_\{\\rho\_\{0\}\}\.\(16\)Becauseρ0\\rho\_\{0\}andΦ0\\Phi\_\{0\}are fixed,Cρ0C\_\{\\rho\_\{0\}\}is independent ofπ\\pi\. Thus, for any policiesπ1\\pi\_\{1\}andπ2\\pi\_\{2\},

JΦ​\(π1\)≥JΦ​\(π2\)⟺J𝒬​\(π1\)≥J𝒬​\(π2\)\.J\_\{\\Phi\}\(\\pi\_\{1\}\)\\geq J\_\{\\Phi\}\(\\pi\_\{2\}\)\\Longleftrightarrow J\_\{\\mathcal\{Q\}\}\(\\pi\_\{1\}\)\\geq J\_\{\\mathcal\{Q\}\}\(\\pi\_\{2\}\)\.\(17\)Therefore,

argmaxπJΦ​\(π\)=argmaxπJ𝒬​\(π\),\\operatorname\*\{argmax\}\_\{\\pi\}J\_\{\\Phi\}\(\\pi\)=\\operatorname\*\{argmax\}\_\{\\pi\}J\_\{\\mathcal\{Q\}\}\(\\pi\),\(18\)which proves Theorem[1](https://arxiv.org/html/2607.05458#Thmtheorem1)\.

### A\.2Proof of Proposition[1](https://arxiv.org/html/2607.05458#Thmproposition1)

Consider a one\-step Harness MDP with initial states0s\_\{0\}, action set𝒜=\{submit,check\}\\mathcal\{A\}=\\\{\\textsc\{submit\},\\textsc\{check\}\\\}, and terminal statessgoods\_\{\\mathrm\{good\}\}andsbads\_\{\\mathrm\{bad\}\}\. Actionsubmitdeterministically reachessgoods\_\{\\mathrm\{good\}\}, whereascheckdeterministically reachessbads\_\{\\mathrm\{bad\}\}\. Let𝒬​\(sgood\)=1\\mathcal\{Q\}\(s\_\{\\mathrm\{good\}\}\)=1and𝒬​\(sbad\)=0\\mathcal\{Q\}\(s\_\{\\mathrm\{bad\}\}\)=0\. Under the task\-rubric objective in Equation[2](https://arxiv.org/html/2607.05458#S4.E2), the unique optimal policy isπsubmit\\pi\_\{\\mathrm\{submit\}\}\.

Now define the action\-pattern bonusb​\(a\)=λ​𝕀​\{a=check\}b\(a\)=\\lambda\\mathbb\{I\}\\\{a=\\textsc\{check\}\\\}forλ\>1\\lambda\>1\. The modified returns are

Jpattern​\(πsubmit\)\\displaystyle J\_\{\\mathrm\{pattern\}\}\(\\pi\_\{\\mathrm\{submit\}\}\)=1,\\displaystyle=1,\(19\)Jpattern​\(πcheck\)\\displaystyle J\_\{\\mathrm\{pattern\}\}\(\\pi\_\{\\mathrm\{check\}\}\)=λ\.\\displaystyle=\\lambda\.Sinceλ\>1\\lambda\>1, the modified objective prefersπcheck\\pi\_\{\\mathrm\{check\}\}, although it produces lower terminal task quality\. Therefore,

argmaxπJ𝒬​\(π\)\\displaystyle\\operatorname\*\{argmax\}\_\{\\pi\}J\_\{\\mathcal\{Q\}\}\(\\pi\)=\{πsubmit\},\\displaystyle=\\\{\\pi\_\{\\mathrm\{submit\}\}\\\},\(20\)argmaxπJpattern​\(π\)\\displaystyle\\operatorname\*\{argmax\}\_\{\\pi\}J\_\{\\mathrm\{pattern\}\}\(\\pi\)=\{πcheck\}\.\\displaystyle=\\\{\\pi\_\{\\mathrm\{check\}\}\\\}\.Thus, an action\-pattern reward need not preserve the task\-optimal policy set\.

## Appendix BProofs for Finite\-Buffer Support and Process–Outcome Decoupling

### B\.1Proof of Theorem[2](https://arxiv.org/html/2607.05458#Thmtheorem2)

#### Outcome bound\.

SinceqAWq\_\{\\mathrm\{AW\}\}is a probability distribution supported onBB,

𝔼qAW​\[G\]\\displaystyle\\mathbb\{E\}\_\{q\_\{\\mathrm\{AW\}\}\}\[G\]=∑τ∈BqAW​\(τ\)​G​\(τ\)\\displaystyle=\\sum\_\{\\tau\\in B\}q\_\{\\mathrm\{AW\}\}\(\\tau\)G\(\\tau\)\(21\)≤∑τ∈BqAW​\(τ\)​GB⋆\\displaystyle\\leq\\sum\_\{\\tau\\in B\}q\_\{\\mathrm\{AW\}\}\(\\tau\)G\_\{B\}^\{\\star\}=GB⋆\.\\displaystyle=G\_\{B\}^\{\\star\}\.Subtracting𝔼μB​\[G\]=G¯B\\mathbb\{E\}\_\{\\mu\_\{B\}\}\[G\]=\\overline\{G\}\_\{B\}yields

Δ​𝒬B≤GB⋆−G¯B=σB\.\\Delta\\mathcal\{Q\}\_\{B\}\\leq G\_\{B\}^\{\\star\}\-\\overline\{G\}\_\{B\}=\\sigma\_\{B\}\.\(22\)

#### Process identity\.

By Equation[8](https://arxiv.org/html/2607.05458#S4.E8),

𝔼qAW​\[Ψ\]\\displaystyle\\mathbb\{E\}\_\{q\_\{\\mathrm\{AW\}\}\}\[\\Psi\]=∑τ∈BμB​\(τ\)​w​\(τ\)ZB​Ψ​\(τ\)\\displaystyle=\\sum\_\{\\tau\\in B\}\\frac\{\\mu\_\{B\}\(\\tau\)w\(\\tau\)\}\{Z\_\{B\}\}\\Psi\(\\tau\)\(23\)=𝔼μB​\[w​Ψ\]𝔼μB​\[w\]\.\\displaystyle=\\frac\{\\mathbb\{E\}\_\{\\mu\_\{B\}\}\[w\\Psi\]\}\{\\mathbb\{E\}\_\{\\mu\_\{B\}\}\[w\]\}\.It follows that

Δ​ΨB\\displaystyle\\Delta\\Psi\_\{B\}=𝔼μB​\[w​Ψ\]𝔼μB​\[w\]−𝔼μB​\[Ψ\]\\displaystyle=\\frac\{\\mathbb\{E\}\_\{\\mu\_\{B\}\}\[w\\Psi\]\}\{\\mathbb\{E\}\_\{\\mu\_\{B\}\}\[w\]\}\-\\mathbb\{E\}\_\{\\mu\_\{B\}\}\[\\Psi\]\(24\)=CovμB⁡\(w,Ψ\)𝔼μB​\[w\]\.\\displaystyle=\\frac\{\\operatorname\{Cov\}\_\{\\mu\_\{B\}\}\(w,\\Psi\)\}\{\\mathbb\{E\}\_\{\\mu\_\{B\}\}\[w\]\}\.Sincew​\(τ\)=exp⁡\(A​\(τ\)/β\)\>0w\(\\tau\)=\\exp\(A\(\\tau\)/\\beta\)\>0for everyτ∈B\\tau\\in B, we have𝔼μB​\[w\]\>0\\mathbb\{E\}\_\{\\mu\_\{B\}\}\[w\]\>0\. Hence,Δ​ΨB≥0\\Delta\\Psi\_\{B\}\\geq 0if and only ifCovμB⁡\(w,Ψ\)≥0\\operatorname\{Cov\}\_\{\\mu\_\{B\}\}\(w,\\Psi\)\\geq 0\.

### B\.2Proof of Corollary[1](https://arxiv.org/html/2607.05458#Thmcorollary1)

By the law of total covariance,

CovμB⁡\(w,Ψ\)\\displaystyle\\operatorname\{Cov\}\_\{\\mu\_\{B\}\}\(w,\\Psi\)=CovμB⁡\(w,𝔼μB​\[Ψ∣A\]\)\\displaystyle=\\operatorname\{Cov\}\_\{\\mu\_\{B\}\}\\left\(w,\\mathbb\{E\}\_\{\\mu\_\{B\}\}\[\\Psi\\mid A\]\\right\)\(25\)\+𝔼μB​\[CovμB⁡\(w,Ψ∣A\)\]\.\\displaystyle\\quad\+\\mathbb\{E\}\_\{\\mu\_\{B\}\}\\left\[\\operatorname\{Cov\}\_\{\\mu\_\{B\}\}\(w,\\Psi\\mid A\)\\right\]\.Becausew=exp⁡\(A/β\)w=\\exp\(A/\\beta\)is deterministic conditional onAA, the second term is zero\. Under the condition𝔼μB​\[Ψ∣A\]=h​\(A\)\\mathbb\{E\}\_\{\\mu\_\{B\}\}\[\\Psi\\mid A\]=h\(A\),

CovμB⁡\(w,Ψ\)=CovμB⁡\(exp⁡\(A/β\),h​\(A\)\)\.\\operatorname\{Cov\}\_\{\\mu\_\{B\}\}\(w,\\Psi\)=\\operatorname\{Cov\}\_\{\\mu\_\{B\}\}\\left\(\\exp\(A/\\beta\),h\(A\)\\right\)\.\(26\)Bothexp⁡\(A/β\)\\exp\(A/\\beta\)andh​\(A\)h\(A\)are non\-decreasing functions ofAA\. Chebyshev’s covariance inequality therefore givesCovμB⁡\(w,Ψ\)≥0\\operatorname\{Cov\}\_\{\\mu\_\{B\}\}\(w,\\Psi\)\\geq 0\. Theorem[2](https://arxiv.org/html/2607.05458#Thmtheorem2)then impliesΔ​ΨB≥0\\Delta\\Psi\_\{B\}\\geq 0\.

### B\.3Proof of Proposition[2](https://arxiv.org/html/2607.05458#Thmproposition2)

ConsiderB=\{τ1,τ2,τ3\}B=\\\{\\tau\_\{1\},\\tau\_\{2\},\\tau\_\{3\}\\\}withμB​\(τi\)=1/3\\mu\_\{B\}\(\\tau\_\{i\}\)=1/3\. Let

\(G​\(τ1\),G​\(τ2\),G​\(τ3\)\)\\displaystyle\(G\(\\tau\_\{1\}\),G\(\\tau\_\{2\}\),G\(\\tau\_\{3\}\)\)=\(1,0\.5,0\),\\displaystyle=\(1,5,0\),\(27\)\(Ψ​\(τ1\),Ψ​\(τ2\),Ψ​\(τ3\)\)\\displaystyle\(\\Psi\(\\tau\_\{1\}\),\\Psi\(\\tau\_\{2\}\),\\Psi\(\\tau\_\{3\}\)\)=\(0,1,1\)\.\\displaystyle=\(0,1,1\)\.ThenCovμB⁡\(G,Ψ\)<0\\operatorname\{Cov\}\_\{\\mu\_\{B\}\}\(G,\\Psi\)<0\. Choose\(A​\(τ1\),A​\(τ2\),A​\(τ3\)\)=\(0,2,1\)\(A\(\\tau\_\{1\}\),A\(\\tau\_\{2\}\),A\(\\tau\_\{3\}\)\)=\(0,2,1\)\. Letu=exp⁡\(1/β\)\>1u=\\exp\(1/\\beta\)\>1, so the weights are\(1,u2,u\)\(1,u^\{2\},u\)\. A direct calculation gives

CovμB⁡\(w,Ψ\)=u2\+u−29\>0\.\\operatorname\{Cov\}\_\{\\mu\_\{B\}\}\(w,\\Psi\)=\\frac\{u^\{2\}\+u\-2\}\{9\}\>0\.\(28\)Equation[10](https://arxiv.org/html/2607.05458#S4.E10)therefore impliesΔ​ΨB\>0\\Delta\\Psi\_\{B\}\>0despiteCovμB⁡\(G,Ψ\)<0\\operatorname\{Cov\}\_\{\\mu\_\{B\}\}\(G,\\Psi\)<0\.

## Appendix CAdditional Experimental Details

#### Experimental protocol\.

Each controlled domain contains 80 training tasks and 20 held\-out evaluation tasks\. We evaluate each policy using three random seeds and three rollouts per held\-out task, resulting in 180 evaluation episodes per domain\. Offline AW is trained on trajectories collected from the base and exploratory harnesses using exponential advantage weights,exp⁡\(A/β\)\\exp\(A/\\beta\)\(Penget al\.,[2019](https://arxiv.org/html/2607.05458#bib.bib27)\)\. Terminal training rewards are produced by the same rubric scorer used for evaluation\. Under the calibrated structural verifier, none of the controlled domains reaches the terminal\-score ceiling atG=1G=1\. For each public\-benchmark adapter, we use 16 training tasks and 20 held\-out evaluation tasks, following the same three\-seed, three\-rollout protocol\. The adapter tasks are drawn fromτ\\tau\-bench retail and AgentBench DB\-Bench\(Yaoet al\.,[2024](https://arxiv.org/html/2607.05458#bib.bib5); Liuet al\.,[2024](https://arxiv.org/html/2607.05458#bib.bib1)\); the adapters preserve their tool\-interaction or database\-reasoning structure while replacing the native evaluator with Harness MDP scoring\.

#### Process\-event detector limitations\.

Several detector extensions may improve the coverage and reliability ofHMS\\mathrm\{HMS\}\. First, reducing the StopWhenSufficient threshold from 0\.85 to 0\.70 may reduce event sparsity\. Second, RevisionAfterFailure could be replaced by a graded measure that combines verifier\-detected failure with the magnitude of the subsequent revision\. Third, EvidenceBeforeClaim could be evaluated using claim\-level annotations rather than surface\-form heuristics\.

#### Reproducibility\.

All reported results are computed from per\-task quality scores, event\-levelHMS\\mathrm\{HMS\}measurements, and public\-benchmark adapter evaluations\. The task\-generation procedure, train–evaluation splits, detector thresholds, verifier configurations, and calibration analyses are reported in the following sections\.

## Appendix DControlled Domain Specification

The controlled suite contains six domains: knowledge\-work, coding, research, multi\-tool, long\-memory, and planning\. The domains are inspired by benchmark families for web and tool interaction, software repair, multi\-hop question answering, and long\-context memory\(Yaoet al\.,[2022](https://arxiv.org/html/2607.05458#bib.bib7); Jimenezet al\.,[2024](https://arxiv.org/html/2607.05458#bib.bib9); Yanget al\.,[2018](https://arxiv.org/html/2607.05458#bib.bib41); Maharanaet al\.,[2024](https://arxiv.org/html/2607.05458#bib.bib11)\)\. Each domain contains 100 human\-annotated tasks with a fixed difficulty distribution of 20 easy, 60 standard, and 20 hard tasks\. The train–evaluation split is stratified by difficulty, yielding 80 training tasks and 20 held\-out evaluation tasks per domain\. Each training set contains 16 easy, 48 standard, and 16 hard tasks, while each evaluation set contains 4 easy, 12 standard, and 4 hard tasks\. To reduce overlap between training and evaluation, tasks vary in difficulty, structural form, distractor content, starter artifacts, and required output format\. Task identifiers are disjoint between buffer collection and held\-out evaluation\.

Task prompts, source constraints, reference criteria, and rubrics were produced through human annotation and curation\. Held\-out tasks are not paraphrases of training tasks\. Train and evaluation splits use disjoint task identifiers, template instantiations, entities, distractors, starter artifacts, and required output formats\. A manual review pass checked split assignment, rubric coverage, and duplicate or near\-duplicate prompts before rollout collection\.

Final task quality is computed as a normalized weighted sum of criterion\-level scores,

Gnorm=∑cscorec∑cmax​\_​scorec\.G\_\{\\mathrm\{norm\}\}=\\frac\{\\sum\_\{c\}\\mathrm\{score\}\_\{c\}\}\{\\sum\_\{c\}\\mathrm\{max\\\_score\}\_\{c\}\}\.\(29\)The scoring criteria are domain\-specific\. Knowledge\-work evaluates answer correctness, evidence support, criterion coverage, and output format, with an additional source\-conflict criterion for hard tasks\. Coding evaluates unit\-test correctness, parseability, safety constraints, and cost compliance\. Research evaluates answer correctness, evidence support, citation format, and output format\. Multi\-tool evaluates answer correctness, tool\-call validity, and structural format requirements\. Long\-memory evaluates answer correctness, source\-session matching, and output format\. Planning evaluates constraint satisfaction, criterion coverage, and plan format\.

Verifier coverage differs across domains\. Coding is the only fully objective verifier domain because its main correctness criterion is determined by executable unit tests\. The other five domains combine structural checks with rubric\-based scoring against human\-annotated criteria\. The scorer receives the task prompt, rubric criterion, and candidate output, but not the reference answer\. Calibration mode is used only for scorer\-validation diagnostics\. The remaining limitations include sensitivity of natural\-language criteria to scorer calibration, limited coverage of regex and overlap\-based checks for paraphrased content, and the use of fixed search observations rather than a live search system in the research domain\.

## Appendix ECalibration and Sensitivity Analysis

This section provides the detector\-level diagnostics underlying the compact sensitivity analysis in Section[5](https://arxiv.org/html/2607.05458#S5)\. We distinguish between the*original domain\-specific verifier*, which uses a strict deterministic coding rubric, and the*calibrated structural verifier*, which applies a common structural scoring rule across domains and uses an EarlySubmit threshold of 0\.25\.

#### Per\-domain support diagnostics\.

Table[9](https://arxiv.org/html/2607.05458#A5.T9)reports final\-quality changes, process changes, and buffer slack under the calibrated structural verifier\. Coding exhibits the largest gain in final quality, while research is the only domain with a negative change inHMS\\mathrm\{HMS\}\. The variation across domains supports the use ofσD\\sigma\_\{D\}as a diagnostic of offline trajectory support, but not as a calibration\-invariant predictor of improvement\.

Table 9:Per\-domain results under the calibrated structural verifier\.Buffer slackσD\\sigma\_\{D\}measures the gap between the best trajectory in the offline buffer and the buffer mean\.
#### Research\-domain process regression\.

Under the calibrated EarlySubmit threshold of 0\.25, AW increases EarlySubmit on research tasks from 0\.0% to 25\.0%, corresponding to 45 of 180 evaluation episodes\. Although CheckBeforeSubmit increases by 13\.9%, the larger EarlySubmit penalty produces a netΔ​HMS\\Delta\\mathrm\{HMS\}of−0\.026\-0\.026\. AW therefore increases verification while also inducing earlier submission, showing that these two behaviors need not improve together\.

#### EarlySubmit threshold sensitivity\.

Table[10](https://arxiv.org/html/2607.05458#A5.T10)reports the full threshold sweep\. Increasing the threshold causes more episodes to be classified as EarlySubmit and can substantially change the apparent process improvement\. The effect is most pronounced in research: at a threshold of 0\.35, all base episodes are labeled as EarlySubmit, which mechanically inflates the estimated AW improvement\. We therefore use 0\.25 as the primary threshold because it avoids this saturation while preserving the qualitative process conclusions\.

Table 10:Sensitivity of process\-maturity changes to the EarlySubmit threshold\.ES Base denotes the EarlySubmit rate under the base harness\. Larger thresholds classify more episodes as early submissions and can inflate the apparent improvement under AW\.
#### Coding\-verifier calibration\.

The original coding evaluation uses a strict deterministic rubric, under which the base harness achieves 0\.929 and is close to the verifier ceiling\. Under this evaluation, AW changes final quality by approximately−0\.006\-0\.006\. The calibrated analysis replaces this coding\-specific rubric with the same structural verifier used across the other domains, reducing the base score to 0\.712 and yielding an AW score of 0\.812\. The difference between the two base scores reflects verifier calibration rather than a change in the policy\. The reported coding gain is therefore measured relative to the calibrated structural baseline\.

#### DB\-Bench stopping behavior\.

On AgentBench DB\-Bench, the EarlySubmit rate increases from 8\.3% under the base harness to 27\.8% under AW\. CheckBeforeSubmit simultaneously increases by 16\.7%, and the aggregate process score remains positive withΔ​HMS=\+0\.030\\Delta\\mathrm\{HMS\}=\+0\.030\. As in research, AW increases both verification and submission speed; however, the remaining process components offset the EarlySubmit penalty\. These cases indicate that verification and stopping behavior should be modeled as distinct control objectives\.

#### Held\-out and in\-sampleτ\\tau\-bench estimates\.

On the 20\-task held\-outτ\\tau\-bench retail split, final quality increases from 0\.337 to 0\.519, withΔ​HMS=\+0\.075\\Delta\\mathrm\{HMS\}=\+0\.075\. This is lower than the earlier four\-task held\-out estimate by 4\.3 points, but remains well above the five\-point decision threshold, preserving the adapter\-level transfer conclusion\.