EvoHarness-RL: Learning Self-Evolving Runtime Harness for Long-Horizon LLM Agents

# EvoHarness-RL: Learning Self-Evolving Runtime Harness for Long-Horizon LLM Agents
Source: [https://arxiv.org/html/2608.05446](https://arxiv.org/html/2608.05446)
1\]University of Illinois Urbana–Champaign 2\]Meta AI

Dongqi FuTianxin WeiHanqing ZengYuanchen BeiBingxuan LiZihao LiQifan WangXiang ShenYifan WuJiayi LiuHong LiYinglong XiaXiangjun FanHanghang TongJingrui He\[\[

###### Abstract

Long\-horizon LLM agents increasingly rely on external execution support to maintain state, track progress, invoke tools, verify outcomes, and reuse experience across interactions\. However, effective harness use raises two coupled challenges: state formation from noisy interaction traces and runtime control over external\-state access\. Existing agents usually handle both through prompts, heuristics, or domain\-specific conventions, leaving the external workspace and its usage policy manually engineered\. To address this, we study the problem of*harness policy learning*, where agents learn harness policies offline and deploy them to construct and update external harness state online during runtime task execution\. We introduceEvoHarness\-RL, which exposes Belief, Progress, and Experience \(BPE\) as policy\-facing harness state\. Supervised harness fine\-tuning teaches the base agent the harness action space and how to construct useful external state, while cost\-aware GRPO explores coordination policies to selectively read, update, and consolidate that state during long\-horizon interaction\. Instantiated on ALFWorld with a Qwen3\-8B LLM,EvoHarness\-RLreaches 96\.9% success and reveals two key dynamics:*harness annealing*, where training internalizes recurring harness\-use patterns into the model policy and shifts the agent from frequent harness calls toward selective external\-state access, and*harness evolution*, where progress updates and experience consolidation refine the harness into a compact, task\-adaptive state substrate\. These results suggest that long\-horizon agents benefit from trainable policies for constructing and coordinating with external harness workspaces, beyond simply adding stronger tools or larger memories\.

## 1Introduction

LLM\-based agents are increasingly deployed in long\-horizon interactive settings, where they need to move beyond one\-step problem solving toward reliable task execution over extended interaction\. In tasks such as embodied interaction, web navigation, software engineering, and workflow automation\(Shridhar et al\.,[2021](https://arxiv.org/html/2608.05446#bib.bib14); Hong et al\.,[2026](https://arxiv.org/html/2608.05446#bib.bib3); Yang et al\.,[2024](https://arxiv.org/html/2608.05446#bib.bib22); Zhou et al\.,[2024](https://arxiv.org/html/2608.05446#bib.bib27)\)agents need to maintain beliefs about the environment, track completed and pending subgoals, recover from failed actions, and reuse procedures from prior experience\. Long\-horizon execution therefore depends on diverse forms of external support, including memory, tools, state trackers, verifiers, and execution logs\(Suzgun et al\.,[2026](https://arxiv.org/html/2608.05446#bib.bib16); Schick et al\.,[2023](https://arxiv.org/html/2608.05446#bib.bib11); Ning et al\.,[2026](https://arxiv.org/html/2608.05446#bib.bib8); Wei et al\.,[2025](https://arxiv.org/html/2608.05446#bib.bib20); Jiang et al\.,[2026](https://arxiv.org/html/2608.05446#bib.bib4)\)\. As these components become more prevalent and more specialized, a central question arises:*how can agents learn to form useful external state and efficiently leverage such support as part of their own decision process?*

We refer to this runtime layer as the external*harness*: the collection of prompts, tools, retrieval modules, memories, state trackers, execution feedback, and control\-flow mechanisms that supports agent execution\. Modern agent frameworks and product systems expose increasingly rich harness components\(Young,[2025](https://arxiv.org/html/2608.05446#bib.bib24); Ning et al\.,[2026](https://arxiv.org/html/2608.05446#bib.bib8); Lee et al\.,[2026](https://arxiv.org/html/2608.05446#bib.bib5)\), and recent harness\-engineering methods further optimize harness state, implementations, or trace\-driven adaptations\(Lou et al\.,[2026](https://arxiv.org/html/2608.05446#bib.bib7); Jiang et al\.,[2026](https://arxiv.org/html/2608.05446#bib.bib4); Lee et al\.,[2026](https://arxiv.org/html/2608.05446#bib.bib5); Chen et al\.,[2026](https://arxiv.org/html/2608.05446#bib.bib1)\)\. In parallel, self\-evolving agents show that past trajectories can be distilled into reusable memories, workflows, or skills\(Shinn et al\.,[2023](https://arxiv.org/html/2608.05446#bib.bib13); Wang et al\.,[2024](https://arxiv.org/html/2608.05446#bib.bib17); Ouyang et al\.,[2026](https://arxiv.org/html/2608.05446#bib.bib10)\)\. However, a complementary bottleneck remains underexplored: even when external components are carefully designed or adapted, the agent’s runtime policy for using them is often specified through prompts, heuristics, or fixed conventions\. As a result, the agent may be surrounded by useful external support, but it is rarely trained to decide when to form, access, update, and consolidate that support as part of its own decision process\.

We proposeEvoHarness\-RL, a trainable coordination layer for learning how agents construct and use external harness state\.EvoHarness\-RLabstracts heterogeneous harness components into a unified, policy\-facing BPE workspace, motivated by three recurring needs in long\-horizon interaction:*Belief*for maintaining the current environment state,*Progress*for tracking completed and pending subgoals, and*Experience*for reusing knowledge across episodes\(Singh et al\.,[2026](https://arxiv.org/html/2608.05446#bib.bib15); Wang et al\.,[2026](https://arxiv.org/html/2608.05446#bib.bib19); Shinn et al\.,[2023](https://arxiv.org/html/2608.05446#bib.bib13)\)\. The agent interacts with this workspace through compact harness meta\-actions to query belief, commit progress, recall experience, and write new insights\.

![Refer to caption](https://arxiv.org/html/2608.05446v1/x1.png)Figure 1:Overview ofEvoHarness\-RL\. Long\-horizon agents rely on complex external execution support, but existing harness designs are often heterogeneous and manually controlled\.EvoHarness\-RLstudies harness policy learning by abstracting this external workspace into three policy\-facing states:*Belief*for environment state,*Progress*for execution status and subgoal structure, and*Experience*for reusable cross\-episode knowledge\. The agent learns to coordinate with this workspace through compact harness actions, deciding when totrack,commit,recall, ornoteexternal state during runtime interaction\.Our training recipe consists of two stages with different purposes\. First, supervised harness fine\-tuning familiarizes the base model with the semantics of the BPE action protocol, and teaches it to externalize useful belief, progress, and experience state from interaction traces\. Second, since harness actions consume the same interaction budget as environment actions, effective agents need to learn not only how to construct external state, but also when external\-state access is worth its cost\. Cost\-aware GRPO\(Shao et al\.,[2024](https://arxiv.org/html/2608.05446#bib.bib12)\)optimizes the resulting policy with rewards for task success, efficiency, action diversity, repetition avoidance, and valid action formatting\. This stage explores when to read, update, or consolidate harness state under an interaction budget, turning harness use from a prompt\-time scaffold into a learned runtime policy decision\.

We instantiate the generalEvoHarness\-RLframework in ALFWorld\(Shridhar et al\.,[2021](https://arxiv.org/html/2608.05446#bib.bib14)\)through a domain\-specific environment adapter\. The adapter preserves the shared BPE interface while grounding Belief, Progress, and Experience as a world\-state tracker, a committed subgoal plan, and a cross\-episode skill bank for embodied household tasks\. Experiments show that BPE is useful both before and after training: prompt\-time BPE already improves stateful long\-horizon tasks, while SFT and GRPO further amplify performance, reaching 96\.9% success on the ALFWorld seen split and 86\.6% on the unseen split\. Beyond final success, our analysis reveals two dynamics:*harness annealing*, where training internalizes recurring harness\-use patterns into the model policy, shifting the agent from frequent scaffold\-like calls toward selective external\-state access, and*harness evolution*, where the online progress updates and cross\-episode experience consolidation refine the harness into a compact, task\-adaptive state substrate\.

Our contributions are four\-fold:

- •We introduceEvoHarness\-RL, a trainable agent\-harness coordination layer based on BPE \(Belief, Progress, and Experience\) and a compact set of harness meta\-actions\.
- •We develop a two\-stage training recipe that first bootstraps harness use from expert demonstrations and then optimizes cost\-aware harness coordination with GRPO\.
- •We show that BPE helps at both inference and training time: prompt\-time BPE improves stateful tasks, while SFT and GRPO substantially improve seen and unseen ALFWorld success rate\.
- •We analyze two co\-evolutionary dynamics: harness annealing, where training turns frequent scaffold use into selective state access, and harness evolution, where the experience store is refined through retrieval, consolidation, and forgetting\.

## 2Method

We introduceEvoHarness\-RL, a trainable coordination layer for harness policy learning\.EvoHarness\-RLconsists of a unified BPE external\-state abstraction \(Section[2\.1](https://arxiv.org/html/2608.05446#S2.SS1)\), a compact agent\-harness action protocol \(Section[2\.2](https://arxiv.org/html/2608.05446#S2.SS2)\), its embodied instantiation in ALFWorld\(Shridhar et al\.,[2021](https://arxiv.org/html/2608.05446#bib.bib14)\)\(Section[2\.3](https://arxiv.org/html/2608.05446#S2.SS3)\), and a two\-stage training pipeline for cost\-aware agent\-harness coordination \(Section[2\.4](https://arxiv.org/html/2608.05446#S2.SS4)\)\.

### 2\.1Unified Harness Abstraction: Belief, Progress, and Experience

A trainable harness interface should expose enough external state to support long\-horizon execution, while remaining compact enough for policy learning\. Concrete harness implementations may contain many domain\-specific components, such as state trackers, execution logs, task plans, verifier feedback, episodic memories, or skill libraries\. Despite their diversity, these components address a small set of recurring failure modes in long\-horizon interaction: agents may lose track of what is currently true in the environment, forget what has already been done or what should be attempted next, and repeatedly rediscover procedures or mistakes that could have been reused from prior attempts\(Shinn et al\.,[2023](https://arxiv.org/html/2608.05446#bib.bib13); Wang et al\.,[2026](https://arxiv.org/html/2608.05446#bib.bib19); Singh et al\.,[2026](https://arxiv.org/html/2608.05446#bib.bib15)\)\. Motivated by these three needs, we organize the policy\-facing role of external harness state into three compact functional roles:Belief,Progress, andExperience\(BPE\)\. Formally, at each steptt, the harness renders

ℋt=\(Bt,Pt,Et\),\\mathcal\{H\}\_\{t\}=\(\\hbox\{\\pagecolor\{beliefbg\}$B\_\{t\}$\},\\hbox\{\\pagecolor\{progressbg\}$P\_\{t\}$\},\\hbox\{\\pagecolor\{experiencebg\}$E\_\{t\}$\}\),\(1\)where the components correspond to environment estimate, execution state, and experience\.

Belief \(BtB\_\{t\}\)stores task\-relevant facts inferred from interaction, such as object states, locations, and relations\. It provides a persistent estimate of the current environment so the policy does not need to rely only on transient context\-window memory\.

Progress \(PtP\_\{t\}\)records task decomposition and execution status through subgoal\-status records\(gi,σi\)\(g\_\{i\},\\sigma\_\{i\}\)\. It externalizes what has been attempted, what remains open, and where execution may be blocked, turning implicit reasoning traces into inspectable task state\.

Experience \(EtE\_\{t\}\)maintains cross\-episode knowledge, such as skills, failure modes, search priors, and high\-level strategies\. It supports reuse across attempts by providing relevant prior experience during execution and storing new insights for later consolidation\.

### 2\.2Agent\-Harness Action Protocol

Given the BPE state, the policy needs a compact way to read from and write to the external workspace\. A fully domain\-specific harness API may expose many operations, but it would make the learned behavior difficult to transfer or analyze\. Conversely, a single generic memory action would hide the functional structure of the workspace\. We therefore define a small set ofharness meta\-actionsthat cover the main information flows between the agent and BPE:

𝒜bpe=\{track,commit,recall,note\}\.\\mathcal\{A\}\_\{\\mathrm\{bpe\}\}=\\\{\\hbox\{\\pagecolor\{trackbg\}\{track\}\},\\hbox\{\\pagecolor\{commitbg\}\{commit\}\},\\hbox\{\\pagecolor\{recallbg\}\{recall\}\},\\hbox\{\\pagecolor\{notebg\}\{note\}\}\\\}\.\(2\)
trackreads task\-relevant belief fromBtB\_\{t\};commitwrites a subgoal or execution update intoPtP\_\{t\};recallretrieves reusable knowledge fromEtE\_\{t\}; andnoterecords a new insight for later experience consolidation intoEtE\_\{t\}\.

During interaction, the policy chooses from both environment actions and harness actions:

𝒜=𝒜env∪𝒜bpe\.\\mathcal\{A\}=\\mathcal\{A\}\_\{\\mathrm\{env\}\}\\cup\\mathcal\{A\}\_\{\\mathrm\{bpe\}\}\.\(3\)At steptt, the policy receives the environment observationoto\_\{t\}, the rendered harness stateℋt\\mathcal\{H\}\_\{t\}, and task contextctc\_\{t\}, then samples

at∼πθ\(⋅∣ot,ℋt,ct\)\.a\_\{t\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid o\_\{t\},\\mathcal\{H\}\_\{t\},c\_\{t\}\)\.\(4\)Ifat∈𝒜enva\_\{t\}\\in\\mathcal\{A\}\_\{\\mathrm\{env\}\}, the action advances the task environment and yields a new observation\. Ifat∈𝒜bpea\_\{t\}\\in\\mathcal\{A\}\_\{\\mathrm\{bpe\}\}, the action queries or updates the external workspace and returns a new harness view\. Since both action types consume the same interaction budget, the agent needs to learn when harness access is worth its cost; we optimize this coordination in Section[2\.4](https://arxiv.org/html/2608.05446#S2.SS4)\.

![Refer to caption](https://arxiv.org/html/2608.05446v1/x2.png)Figure 2:Overview of the training pipeline ofEvoHarness\-RL\. Starting from an SFT checkpoint, the policy generates GRPO rollouts by interleaving environment actions with BPE harness actions\. Rewards combine task success, efficiency, diversity, and spam and format penalties to optimize cost\-aware harness coordination\.
### 2\.3Environment Adapter for BPE Harness Grounding

BPE is a functional interface rather than a fixed internal schema\. Different tasks may require different harness implementations, but they can expose thesame policy\-facing roles: Belief, Progress, and Experience\. We therefore use anenvironment adapterto bridge domain\-specific signals with the general BPE interface\. The adapter processes observations, action results, tool outputs, and verifier feedback, maintains the internal harness stores, and renders selected views as\(Bt,Pt,Et\)\(B\_\{t\},P\_\{t\},E\_\{t\}\)for the policy\. It also grounds the four harness actions in the target environment\. Thus, the internal implementation remains domain\-specific, while the trainable agent\-harness coordination layer is shared\.

We instantiate this adapter in ALFWorld\(Shridhar et al\.,[2021](https://arxiv.org/html/2608.05446#bib.bib14)\)\. The adapter keeps domain\-specific internal stores for belief, progress, and experience, and renders them as the policy\-facing BPE state\(Bt,Pt,Et\)\(B\_\{t\},P\_\{t\},E\_\{t\}\)\. Appendix[9](https://arxiv.org/html/2608.05446#S9)provides implementation details, and Appendix[8](https://arxiv.org/html/2608.05446#S8)gives a concrete trajectory\-level example\.

#### Belief \(BtB\_\{t\}\) andtrackinstantiation\.

For embodied household tasks, Belief is grounded as an internal world\-state store that is updated in the background after each environment step\. The adapter uses the agent’s actions and environment observations to maintain task\-relevant facts such as object states, locations, and spatial relations\. This state is not fully exposed to the policy by default\. Instead, the policy can issuetrack\[object\] to inspect a specific object ortrack\[world\] to obtain a compact global summary\. Thus, Belief provides persistent environment state, while access to that state remains a selective harness action\.

#### Progress \(PtP\_\{t\}\) andcommitinstantiation\.

Progress is grounded as a committed execution record\. In ALFWorld, this is implemented as a bounded list of subgoal\-status entries, which is sufficient for mostly sequential household tasks\. For example, a clean\-and\-place task may involve locating the target object, picking it up, cleaning it, and placing it in the target receptacle\. The policy usescommit\[subgoal\] to externalize the current execution step, making attempted, pending, or blocked progress visible to later decisions\.

#### Experience \(EtE\_\{t\}\),recall, andnoteinstantiation\.

Experience is grounded as a cross\-episode skill store, organized into general skills, task\-specific skills, common mistakes, and object\-location search priors\. It evolves at two different timescales\. During an episode, the policy usesrecall\[query\] to retrieve relevant prior knowledge, which also updates the usage counts of retrieved entries\. The policy can also issuenote\[insight\] to write newly observed lessons into a temporary note buffer, while successful object searches update object\-location priors online\. During parallel rollout collection, the main skill store is kept fixed within each batch, and all note buffers and completed\-trajectory summaries are accumulated in the background\. At epoch boundaries, a consolidation model merges this evidence into the skill store through add, update, and remove operations\.

### 2\.4Cost\-Aware Harness Optimization

#### Supervised harness fine\-tuning\.

We first bootstrap the policy with supervised fine\-tuning on successful teacher trajectories collected using the same BPE interface\. At each step, the teacher observes the task objective, current observation, admissible environment actions, recent history, and active harness views, then outputs a single next action in the format<think\>\.\.\.</think\><action\>\.\.\.</action\>\. The action can be either an ALFWorld command or a BPE harness action\. We fine\-tune Qwen3\-8B on these next\-action demonstrations, teaching the model both task\-solving behavior and the basic semantics of when totrack,commit,recall, ornote\. The experience accumulated during teacher rollouts initializes the skill store used in subsequent GRPO\(Shao et al\.,[2024](https://arxiv.org/html/2608.05446#bib.bib12)\)\.

#### Cost\-aware GRPO\.

We optimize the policy using Group Relative Policy Optimization \(GRPO\) initialized from a Supervised Fine\-Tuning \(SFT\) checkpoint\. The trajectory\-level rewardR​\(τ\)R\(\\tau\)combines a dominant sparse success signal with dense auxiliary shaping terms designed to cultivate adaptive harness usage:

R​\(τ\)=\\displaystyle R\(\\tau\)=\{\}Rsucc​\(τ\)⏟task success\+λeff​Reff​\(τ\)⏟efficiency bonus\+λdiv​\(u\)​Rdiv​\(τ\)⏟action diversity−λspam​Rspam​\(τ\)⏟spam penalty−λinv​Rinv​\(τ\)⏟format penalty\.\\displaystyle\\underbrace\{R\_\{\\mathrm\{succ\}\}\(\\tau\)\}\_\{\\text\{task success\}\}\+\\underbrace\{\\lambda\_\{\\mathrm\{eff\}\}R\_\{\\mathrm\{eff\}\}\(\\tau\)\}\_\{\\text\{efficiency bonus\}\}\+\\underbrace\{\\lambda\_\{\\mathrm\{div\}\}\(u\)R\_\{\\mathrm\{div\}\}\(\\tau\)\}\_\{\\text\{action diversity\}\}\-\\underbrace\{\\lambda\_\{\\mathrm\{spam\}\}R\_\{\\mathrm\{spam\}\}\(\\tau\)\}\_\{\\text\{spam penalty\}\}\-\\underbrace\{\\lambda\_\{\\mathrm\{inv\}\}R\_\{\\mathrm\{inv\}\}\(\\tau\)\}\_\{\\text\{format penalty\}\}\.\(5\)
Task completion acts as the strict gatekeeper:Rsucc​\(τ\)=10⋅𝟏​\[solved\]R\_\{\\mathrm\{succ\}\}\(\\tau\)=10\\cdot\\mathbf\{1\}\[\\text\{solved\}\], and the efficiency bonusReff​\(τ\)=max⁡\(0,1−\|τ\|/Tmax\)R\_\{\\mathrm\{eff\}\}\(\\tau\)=\\max\(0,1\-\|\\tau\|/T\_\{\\max\}\)is only granted upon success, naturally penalizing redundant harness queries\.

To prevent policy collapse, where the agent either ignores𝒜bpe\\mathcal\{A\}\_\{\\mathrm\{bpe\}\}or falls into infinite repetitive loops, we introduce a time\-dependent vocabulary diversity bonus:

Rdiv​\(τ\)=\|\{verb​\(at\):at∈τ\}\|\|τ\|,λdiv​\(u\)=λdivmax2​\(1\+cos⁡π​uU\),R\_\{\\mathrm\{div\}\}\(\\tau\)=\\frac\{\|\\\{\\mathrm\{verb\}\(a\_\{t\}\):a\_\{t\}\\in\\tau\\\}\|\}\{\|\\tau\|\},\\quad\\lambda\_\{\\mathrm\{div\}\}\(u\)=\\frac\{\\lambda\_\{\\mathrm\{div\}\}^\{\\max\}\}\{2\}\\left\(1\+\\cos\\frac\{\\pi u\}\{U\}\\right\),\(6\)whereuuis the current RL epoch andUUis the annealing horizon\. This curriculum encourages broad exploration of harness actions early in training, before gracefully decaying to force specialization and efficient task resolution\. Finally,RspamR\_\{\\mathrm\{spam\}\}andRinvR\_\{\\mathrm\{inv\}\}apply fixed penalties for degenerate repetitions or malformed syntax\.

## 3Experiments

We evaluateEvoHarness\-RLon ALFWorld to study its effectiveness against frozen and trainable baselines, the contribution of each BPE component, and generalization to unseen environments\. We further analyze how harness use changes during training and how the external experience store evolves over time\.

### 3\.1Experiment Setup

#### Environments\.

We evaluate on ALFWorld\(Shridhar et al\.,[2021](https://arxiv.org/html/2608.05446#bib.bib14)\), a text\-based game aligned with the ALFRED embodied AI benchmark\. Agents must complete multi\-step household tasks by navigating rooms and manipulating objects through text commands\. We focus on six distinct task families requiring varied levels of state tracking: simple pick\-and\-place \(Pick\), object inspection under light \(Look\), cleaning\-before\-placing \(Clean\), heating\-before\-placing \(Heat\), cooling\-before\-placing \(Cool\), and placing two objects \(Pick2\)\. We report the success rate on the standard validation set asOuyang et al\. \([2026](https://arxiv.org/html/2608.05446#bib.bib10)\)\.

#### Baselines\.

We compareEvoHarness\-RLagainst three categories of competitive methods\. First, we includefrontier models\(Claude Opus 4\.5, GPT\-4\.1, GPT\-5\) evaluated with standard prompting and with our prompt\-time harness to measure how strong base policies benefit from explicit BPE structures\. Second, we evaluatefrozen memory and agentic methods, including ReAct\(Yao et al\.,[2022](https://arxiv.org/html/2608.05446#bib.bib23)\), ExpeL\(Zhao et al\.,[2024](https://arxiv.org/html/2608.05446#bib.bib26)\), ReasoningBank\(Ouyang et al\.,[2025](https://arxiv.org/html/2608.05446#bib.bib9)\), MemP\(Fang et al\.,[2026](https://arxiv.org/html/2608.05446#bib.bib2)\), Dynamic Cheatsheet\(Suzgun et al\.,[2026](https://arxiv.org/html/2608.05446#bib.bib16)\), ACE\(Zhang et al\.,[2025](https://arxiv.org/html/2608.05446#bib.bib25)\), and SkillOS\-base\(Ouyang et al\.,[2026](https://arxiv.org/html/2608.05446#bib.bib10)\), which utilize external memory or experience pools without parameter updates\. Third, we considertrainable methods, including standard GRPO\(Shao et al\.,[2024](https://arxiv.org/html/2608.05446#bib.bib12)\), SkillOS\(Ouyang et al\.,[2026](https://arxiv.org/html/2608.05446#bib.bib10)\), and SkillRL\(Xia et al\.,[2026](https://arxiv.org/html/2608.05446#bib.bib21)\), which integrate memory mechanisms or structural optimization directly into training\. We compare these against our own variants:EvoHarness\-Base\(inference\-time\),EvoHarness\-SFT\(supervised harness learning\), andEvoHarness\-RL\(cost\-aware GRPO\)\. The implementation details are provided in Appendix[9](https://arxiv.org/html/2608.05446#S9)\.

### 3\.2Main Results

Table 1:Main results on ALFWorld\. We report success rates on the 140\-task seen split\.∗\\astdenotes frozen inference\-time methods, and▲\\blacktriangledenotes trainable methods\.Δ\\Deltadenotes the absolute average\-SR gain over the corresponding ReAct baseline\.†/‡indicate results reported by SkillOS/SkillRL, respectively\.Table[1](https://arxiv.org/html/2608.05446#S3.T1)demonstrates thatEvoHarness\-RLon Qwen3\-8B achieves state of the art performance with a 96\.9% average success rate, yielding a \+49\.0 absolute improvement over the base ReAct model\. This optimization allows the 8B model to effectively match top frontier models like Claude Opus 4\.5\. Our method also decisively outperforms all competitive baselines, including static memory approaches and strong trainable agents like SkillOS \(80\.2%\) and SkillRL \(89\.9%\)\. The progression from prompt time scaffolding \(56\.4%\) to SFT \(68\.6%\) and finally GRPO \(96\.9%\) clearly validates our two stage training pipeline, showing that optimization transforms the harness from a static tool into a highly effective decision interface\. Additionally, the top block of the table shows that the BPE framework provides universal benefits across model scales\. Applying the explicit harness significantly elevates struggling frontier policies, boosting GPT\-4\.1 by \+22\.1 and GPT\-5 by \+25\.7\. Even for Claude Opus 4\.5, which is already near the performance ceiling, the harness pushes the success rate to 98\.5%\. This confirms that externalizing belief, progress, and experience is broadly critical for reliable long horizon task execution regardless of the base model size\.

### 3\.3Ablation Results

Table 2:BPE component ablation on ALFWorld\. The top block provides learned\-policy references, while the bottom block removes one component at a time from the Qwen3\-8B inference time harness\.To isolate the contribution of each BPE component, we ablate one module at a time from the inference\-time harness \(Table[2](https://arxiv.org/html/2608.05446#S3.T2)\)\. Removing the environment belief disables explicit object tracking, leading to severe performance drops on tasks requiring localization and state verification, such asCleanandCool\. Ablating task progress prevents subgoal commitment, which disproportionately degrades performance on long\-horizon tasks with dependent subgoals likePick2\. Finally, disabling reusable experience removes skill recall and mistake avoidance, yielding the lowest overall average success rate \(48\.6%\) and heavily impacting complex state\-change tasks likeHeat\. Ultimately, the absence of any single component significantly harms execution, confirming that Belief, Progress, and Experience function synergistically as a unified state interface rather than as isolated memory tricks\.

Table 3:Generalization results on ALFWorld unseen tasks\. We compare ReAct with prompt\-time, SFT, and GRPO\-optimizedEvoHarness\-RLvariants\.
### 3\.4Generalization Analysis

We testEvoHarness\-RLon the ALFWorld unseen split \(Table[3](https://arxiv.org/html/2608.05446#S3.T3)\)\. Qwen3\-8B ReAct achieves 50\.0% success, while the prompt\-time BPE harness improves zero\-shot performance to 77\.6%, showing the benefit of state externalization\.EvoHarness\-SFTdrops to 69\.4%, likely because supervised imitation learns teacher harness\-use patterns from seen trajectories without optimizing when access is worthwhile in novel environments\. In contrast, the full RL\-optimized policy reaches 86\.6%, suggesting that cost\-aware GRPO recalibrates harness access and learns a broadly useful strategy rather than memorizing training environments\.

## 4Analysis of Agent\-Harness Evolution Dynamics

We analyze two training dynamics ofEvoHarness\-RL: policy\-side harness annealing and harness\-side experience evolution\. More detailed analyses are provided in Appendix[7](https://arxiv.org/html/2608.05446#S7)\.

### 4\.1Harness Internalization and Annealing

![Refer to caption](https://arxiv.org/html/2608.05446v1/x3.png)Figure 3:Harness usage anneals during GRPO\. The policy starts with frequent cognitive\-tool calls, then stabilizes near one harness call per episode\.Figure[3](https://arxiv.org/html/2608.05446#S4.F3)shows a clear annealing pattern during GRPO\. The SFT\-initialized agent begins with frequent harness calls, using BPE as an explicit scaffold to track state, recall procedures, and narrow the search space toward better trajectories\. As RL progresses, usage drops quickly and stabilizes near one call per episode\. This suggests that GRPO gradually internalizes routine scaffolded behaviors into the policy, while preserving harness access only when the expected benefit outweighs its step cost\. Thus,EvoHarness\-RLshifts from scaffolded exploration to selective, cost\-aware coordination\.

### 4\.2Harness Evolution

![Refer to caption](https://arxiv.org/html/2608.05446v1/x4.png)Figure 4:Experience\-store evolution\. The skill bank expands during exploration, then stabilizes into a compact mixture of general, task\-specific, mistake\-correction, and search\-priority skills\.Figure[4](https://arxiv.org/html/2608.05446#S4.F4)shows how the runtime harness evolves through its cross\-episode Experience store\. While belief and progress are updated within episodes, Experience is reshaped across episodes through accumulation, consolidation, and forgetting\. The skill bank expands rapidly early in training with general strategies, task\-specific procedures, common mistakes, and search priorities; later, growth becomes selective as redundant entries are merged, rarely useful skills are evicted, and frequently recalled knowledge is preserved\. The final bank remains compact yet diverse, suggesting that the harness becomes a task\-adaptive state substrate rather than passive append\-only memory\. This complements harness annealing: the policy learns when to use external state, while the harness evolves what reusable experience it can provide\.

## 5Related Work

#### Harness Engineering\.

The execution capability of an LLM agent is heavily shaped by its external harness, the surrounding framework that determines how it observes, reasons, and uses tools\. Early paradigms introduced explicit reasoning and action loops and basic tool augmentation\(Yao et al\.,[2022](https://arxiv.org/html/2608.05446#bib.bib23); Schick et al\.,[2023](https://arxiv.org/html/2608.05446#bib.bib11)\), which have since evolved into more complex execution environments with specialized observation renderers, file system access, and execution feedback\(Yang et al\.,[2024](https://arxiv.org/html/2608.05446#bib.bib22); Zhou et al\.,[2024](https://arxiv.org/html/2608.05446#bib.bib27); Ning et al\.,[2026](https://arxiv.org/html/2608.05446#bib.bib8)\)\. Because these external components are critical to long horizon task success, recent work has increasingly focused on optimizing the harness itself\. Frameworks such as Harness\-1 externalize search state into environment side memory\(Jiang et al\.,[2026](https://arxiv.org/html/2608.05446#bib.bib4)\), while Meta\-Harness and HarnessX use offline search and trace driven adaptation to discover effective harness configurations\(Lee et al\.,[2026](https://arxiv.org/html/2608.05446#bib.bib5); Chen et al\.,[2026](https://arxiv.org/html/2608.05446#bib.bib1)\)\. However, these approaches largely treat the harness as an environment side construct or a prompt time convention engineered by human developers or offline search algorithms\. In contrast,EvoHarness\-RLtreats harness access as a first class, learnable policy decision\. Rather than simply providing the agent with a better fixed scaffold, we train the underlying policy to actively control, query, and coordinate with the external workspace\.

#### Memory and Self\-Evolving Agents\.

Successfully navigating extended interactions requires agents to maintain both within\-episode state and cross\-episode knowledge\. Prior memory\-augmented agents have typically focused on the latter, accumulating past trajectories, reflections, or task summaries to inform future episodes\(Shinn et al\.,[2023](https://arxiv.org/html/2608.05446#bib.bib13); Li et al\.,[2025](https://arxiv.org/html/2608.05446#bib.bib6)\)\. Because raw episodic logs are often noisy, later systems increasingly distill experience into structured procedural memory, such as workflows, code libraries, or reusable skills\(Wang et al\.,[2024](https://arxiv.org/html/2608.05446#bib.bib17),[2025](https://arxiv.org/html/2608.05446#bib.bib18)\)\. Recent advances further suggest that these experience repositories must be actively curated, refined, consolidated, or forgotten, rather than treated as append\-only logs\(Xia et al\.,[2026](https://arxiv.org/html/2608.05446#bib.bib21); Ouyang et al\.,[2026](https://arxiv.org/html/2608.05446#bib.bib10)\)\. However, existing self\-evolving agents generally separate cross\-episode skill curation from real\-time within\-episode state tracking, such as maintaining environmental belief and monitoring subgoal progress\.EvoHarness\-RLaddresses this limitation by abstracting the external workspace into a unified BPE \(Belief, Progress, Experience\) interface\. This design enables the policy to evolve not only how it uses long\-term experience, but also how it synchronizes that experience with active environmental belief and execution progress, leading to the dynamic we call harness evolution\.

## 6Conclusion

We introduced We introducedEvoHarness\-RL, a trainable coordination layer that teaches long\-horizon agents to construct and use external harness states\. By organizing the workspace into Belief, Progress, and Experience, and optimizing when to query, update, and consolidate it,EvoHarness\-RLturns harness use from a manual prompting convention into a learned runtime policy\. Results on ALFWorld show that cost\-aware agent\-harness coordination improves task success and reveals useful dynamics of harness annealing and evolution\.

## References

- Chen et al\. \(2026\)Tingyang Chen, Shuo Lu, Kang Zhao, Weicheng Meng, Hanlin Teng, Tianhao Li, Chao Li, Xule Liu, Jian Liang, Zhizhong Zhang, et al\.Harnessx: A composable, adaptive, and evolvable agent harness foundry\.*arXiv preprint arXiv:2606\.14249*, 2026\.
- Fang et al\. \(2026\)Runnan Fang, Yuan Liang, Xiaobin Wang, Jialong Wu, Shuofei Qiao, Pengjun Xie, Fei Huang, Huajun Chen, and Ningyu Zhang\.Memp: Exploring agent procedural memory\.In*Findings of the Association for Computational Linguistics: ACL 2026*, pages 17490–17502, 2026\.
- Hong et al\. \(2026\)Yining Hong, Rui Sun, Bingxuan Li, Xingcheng Yao, Maxine Wu, Alexander Chien, Da Yin, Ying Nian Wu, Zhecan Wang, and Kai\-Wei Chang\.Embodied web agents: Bridging physical\-digital realms for integrated agent intelligence\.*Advances in Neural Information Processing Systems*, 38, 2026\.
- Jiang et al\. \(2026\)Pengcheng Jiang, Zhiyi Shi, Kelly Hong, Xueqiang Xu, Jiashuo Sun, Jimeng Sun, Hammad Bashir, and Jiawei Han\.Harness\-1: Reinforcement learning for search agents with state\-externalizing harnesses\.*arXiv preprint arXiv:2606\.02373*, 2026\.
- Lee et al\. \(2026\)Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn\.Meta\-harness: End\-to\-end optimization of model harnesses\.*arXiv preprint arXiv:2603\.28052*, 2026\.
- Li et al\. \(2025\)Yangning Li, Weizhi Zhang, Yuyao Yang, Wei\-Chieh Huang, Yaozu Wu, Junyu Luo, Yuanchen Bei, Henry Peng Zou, Xiao Luo, Yusheng Zhao, et al\.A survey of rag\-reasoning systems in large language models\.In*Findings of the Association for Computational Linguistics: EMNLP 2025*, pages 12120–12145, 2025\.
- Lou et al\. \(2026\)Xinghua Lou, Miguel Lázaro\-Gredilla, Antoine Dedieu, Carter Wendelken, Wolfgang Lehrach, and Kevin P Murphy\.Autoharness: improving llm agents by automatically synthesizing a code harness\.*arXiv preprint arXiv:2603\.03329*, 2026\.
- Ning et al\. \(2026\)Xuying Ning, Katherine Tieu, Dongqi Fu, Tianxin Wei, Zihao Li, Yuanchen Bei, Jiaru Zou, Mengting Ai, Zhining Liu, Ting\-Wei Li, et al\.Code as agent harness\.*arXiv preprint arXiv:2605\.18747*, 2026\.
- Ouyang et al\. \(2025\)Siru Ouyang, Jun Yan, I Hsu, Yanfei Chen, Ke Jiang, Zifeng Wang, Rujun Han, Long T Le, Samira Daruki, Xiangru Tang, et al\.Reasoningbank: Scaling agent self\-evolving with reasoning memory\.*arXiv preprint arXiv:2509\.25140*, 2025\.
- Ouyang et al\. \(2026\)Siru Ouyang, Jun Yan, Yanfei Chen, Rujun Han, Zifeng Wang, Bhavana Dalvi Mishra, Rui Meng, Chun\-Liang Li, Yizhu Jiao, Kaiwen Zha, et al\.Skillos: Learning skill curation for self\-evolving agents\.*arXiv preprint arXiv:2605\.06614*, 2026\.
- Schick et al\. \(2023\)Timo Schick, Jane Dwivedi\-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom\.Toolformer: Language models can teach themselves to use tools\.*Advances in neural information processing systems*, 36:68539–68551, 2023\.
- Shao et al\. \(2024\)Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al\.Deepseekmath: Pushing the limits of mathematical reasoning in open language models\.*arXiv preprint arXiv:2402\.03300*, 2024\.
- Shinn et al\. \(2023\)Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao\.Reflexion: Language agents with verbal reinforcement learning\.*Advances in neural information processing systems*, 36:8634–8652, 2023\.
- Shridhar et al\. \(2021\)Mohit Shridhar, Xingdi Yuan, Marc\-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht\.ALFWorld: Aligning text and embodied environments for interactive learning\.In*International Conference on Learning Representations*, 2021\.[https://openreview\.net/forum?id=0IOX0YcCdTn](https://openreview.net/forum?id=0IOX0YcCdTn)\.
- Singh et al\. \(2026\)Joykirat Singh, Zaid Khan, Archiki Prasad, Justin Chih\-Yao Chen, Akshay Nambi, Hyunji Lee, Elias Stengel\-Eskin, and Mohit Bansal\.Agent\-brace: Decoupling beliefs from actions in long\-horizon tasks via verbalized state uncertainty\.*arXiv preprint arXiv:2605\.11436*, 2026\.
- Suzgun et al\. \(2026\)Mirac Suzgun, Mert Yuksekgonul, Federico Bianchi, Dan Jurafsky, and James Zou\.Dynamic cheatsheet: Test\-time learning with adaptive memory\.In*Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 7080–7106, 2026\.
- Wang et al\. \(2024\)Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar\.Voyager: An open\-ended embodied agent with large language models\.*Transactions on Machine Learning Research*, 2024\.ISSN 2835\-8856\.[https://openreview\.net/forum?id=ehfRiF0R3a](https://openreview.net/forum?id=ehfRiF0R3a)\.
- Wang et al\. \(2025\)Jiayu Wang, Yifei Ming, Riya Dulepet, Qinglin Chen, Austin Xu, Zixuan Ke, Frederic Sala, Aws Albarghouthi, Caiming Xiong, and Shafiq Joty\.Liveresearchbench: A live benchmark for user\-centric deep research in the wild\.*arXiv preprint arXiv:2510\.14240*, 2025\.
- Wang et al\. \(2026\)Taiyi Wang, Sian Gooding, Florian Hartmann, Oriana Riva, and Edward Grefenstette\.A subgoal\-driven framework for improving long\-horizon llm agents\.*arXiv preprint arXiv:2603\.19685*, 2026\.
- Wei et al\. \(2025\)Tianxin Wei, Noveen Sachdeva, Benjamin Coleman, Zhankui He, Yuanchen Bei, Xuying Ning, Mengting Ai, Yunzhe Li, Jingrui He, Ed H Chi, et al\.Evo\-memory: Benchmarking llm agent test\-time learning with self\-evolving memory\.*arXiv preprint arXiv:2511\.20857*, 2025\.
- Xia et al\. \(2026\)Peng Xia, Jianwen Chen, Hanyang Wang, Jiaqi Liu, Kaide Zeng, Yu Wang, Siwei Han, Yiyang Zhou, Xujiang Zhao, Haifeng Chen, et al\.Skillrl: Evolving agents via recursive skill\-augmented reinforcement learning\.*arXiv preprint arXiv:2602\.08234*, 2026\.
- Yang et al\. \(2024\)John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press\.Swe\-agent: Agent\-computer interfaces enable automated software engineering\.*Advances in Neural Information Processing Systems*, 37:50528–50652, 2024\.
- Yao et al\. \(2022\)Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao\.React: Synergizing reasoning and acting in language models\.*arXiv preprint arXiv:2210\.03629*, 2022\.
- Young \(2025\)Justin Young\.Effective harnesses for long\-running agents\.Anthropic Engineering Blog, November 2025\.[https://www\.anthropic\.com/engineering/effective\-harnesses\-for\-long\-running\-agents](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents)\.Accessed: 2026\-05\-11\.
- Zhang et al\. \(2025\)Qizheng Zhang, Changran Hu, Shubhangi Upasani, Boyuan Ma, Fenglu Hong, Vamsidhar Kamanuru, Jay Rainton, Chen Wu, Mengmeng Ji, Hanchen Li, et al\.Agentic context engineering: Evolving contexts for self\-improving language models\.*arXiv preprint arXiv:2510\.04618*, 2025\.
- Zhao et al\. \(2024\)Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong\-Jin Liu, and Gao Huang\.Expel: Llm agents are experiential learners\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, volume 38, pages 19632–19642, 2024\.
- Zhou et al\. \(2024\)Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al\.Webarena: A realistic web environment for building autonomous agents\.In*International Conference on Learning Representations*, volume 2024, pages 15585–15606, 2024\.

\\beginappendix

## 7Training Dynamics

We provide additional analyses of learned agent–harness coordination by decomposing harness usage by action type and comparing the training reward trajectory against standard GRPO\.

![Refer to caption](https://arxiv.org/html/2608.05446v1/x5.png)Figure 5:Action\-specific harness annealing\. Different BPE actions decay at different rates, while the success rate is still rising\.#### Action\-specific harness annealing\.

Figure[5](https://arxiv.org/html/2608.05446#S7.F5)shows that different BPE actions are not pruned uniformly during training\. The policy initially uses all harness actions, consistent with the SFT scaffold where the agent frequently queries experience, tracks state, commits progress, and writes notes\. During GRPO, however, these actions are selectively retained\.Recallremains the most persistent action, indicating that cross\-episode experience continues to provide useful search priors even after many routine behaviors are internalized\. In contrast,commitandnoterapidly decay toward zero, suggesting that the policy no longer needs to externalize every intermediate plan or write frequent new insights once stable task strategies emerge\.Trackfollows an intermediate pattern: it is useful for early state disambiguation, but gradually decreases as the agent learns more direct environment\-interaction patterns\.

This action\-level pattern is environment\-dependent\. In ALFWorld, tasks share reusable household procedures and object\-search priors, making the Experience component especially valuable\. In more visually grounded embodied environments, the agent may rely more heavily on Belief to maintain scene graphs, object states, or spatial relations\. In software\-engineering or workflow environments, Progress may become more important for tracking subtasks, test status, dependency resolution, and unfinished branches\. Therefore,EvoHarness\-RLshould not be interpreted as learning a fixed universal harness\-action distribution\. Instead, it learns which parts of BPE are worth accessing under the cost structure and state demands of a given environment\.

![Refer to caption](https://arxiv.org/html/2608.05446v1/x6.png)Figure 6:Training reward dynamics\.EvoHarness\-RLachieves higher reward than standard GRPO throughout training, suggesting more efficient agent–harness coordination\.
#### Reward improvement during coordination learning\.

Figure[6](https://arxiv.org/html/2608.05446#S7.F6)shows thatEvoHarness\-RLconsistently outperforms standard GRPO during training\. Its reward increases faster and reaches a higher plateau, while standard GRPO improves more slowly and remains substantially lower\. Together with the action\-specific annealing pattern in Figure[5](https://arxiv.org/html/2608.05446#S7.F5), this suggests that the reduction in harness calls is not caused by policy collapse or failure to use the harness\. Rather, the agent learns to use the harness more selectively: BPE first serves as a scaffold for exploration and state management, and GRPO then encourages the policy to internalize recurring scaffolded behaviors while preserving selective access to the most useful external state\.

## 8Qualitative Case Study

We provide a qualitative ALFWorld trajectory to illustrate how the learned policy coordinates with the BPE harness during long\-horizon execution\. In this task, the agent must clean a kettle and place it on the dining table\. The trajectory shows a naturalcommit→\\rightarrowrecall→\\rightarrowact→\\rightarrownoteloop: the agent first commits to the subgoal of finding the kettle, recalls task and object\-location priors from the skill bank, verifies these priors through environment interaction, and then writes corrective evidence when the recalled hint is stale\. Although the skill bank suggests that the kettle is on a countertop, the agent does not blindly follow this prior after repeated failed searches; instead, it explores stoveburners, finds the kettle, and records that the recalled hint was incorrect\. The case study shows that the harness is used as a source of reusable but revisable experience rather than as a fixed oracle, enabling the agent to combine cross\-episode priors with current\-episode grounding and self\-correction\.

Case Study: Skill Loop with Self\-CorrectionTask:clean some kettle and put it in diningtable\.Type:pick\_clean\_then\_place\_in\_recepSUCCESS— score 1\.0, 30 turnsSkills\[cle\_001\]Phase\-ordered plan: locate→\\toclean→\\toplace;\[cle\_006\]object\-location priors\.Avoid\[err\_002\]Blindly trusting stale hints;\[err\_003\]repeatedly searching the same locations\.Agent Trajectory with Harness\-Guided ReasoningResult:SUCCESS— The trajectory exhibits the full harness loop:commit→\\torecall→\\toact→\\tonote\. The recalled hint was wrong; the agent detected the contradiction, adapted, and wrote corrective evidence back to the skill bank\.

## 9Implementation Details

This section records the concrete implementation choices used in our ALFWorld experiments\. We follow the BPE interface and cost\-aware optimization objective defined in Section[2](https://arxiv.org/html/2608.05446#S2), and focus here on details needed to reproduce the system\.

#### Model roles\.

The trainable policy is Qwen3\-8B\. It is the only model updated during SFT and GRPO, and the only model executed inside the rollout loop\. At each step, the policy emits a response in the form<think\>\.\.\.</think\><action\>\.\.\.</action\>; the parsed action is then dispatched either to ALFWorld or to the BPE harness\. We use Claude Opus as the teacher for SFT trajectory collection and as the consolidation model for the experience store\. During GRPO, consolidation is run outside the rollout loop at epoch boundaries, so the policy rollout itself remains a Qwen3\-8B interaction with the environment and the current harness state\.

#### Action parsing and execution\.

The environment wrapper first extracts the content inside the<action\>tag\. If the action matches an admissible ALFWorld command, it is executed in the environment\. Otherwise, the wrapper checks whether it matches one of the harness\-action patterns, such astrack \[object\],commit \[subgoal\],recall \[query\], ornote \[insight\]\. Malformed outputs, missing action tags, or actions outside both spaces return failure feedback and are counted by the invalid\-action penalty\. We disable the base trainer’s built\-in invalid\-action penalty and apply all reward shaping in our own reward function\.

#### Harness grounding\.

The ALFWorld harness is implemented with lightweight deterministic components whenever possible\. The belief tracker is a rule\-based parser over action–observation pairs; it updates object state flags and object\-location relations without an LLM call\. The progress tracker keeps a bounded committed\-plan list and updates it only when the policy emitscommit\. The experience store uses keyword\-overlap retrieval forrecall, increments usage counts for retrieved entries, and buffersnoteoutputs for later consolidation\. Successful object pickup actions also update the object\-location priority map\. Each skill category is capacity\-bounded and uses LFU eviction, so frequently recalled entries are retained while rarely used entries are removed\.

#### SFT data construction\.

We collect SFT data by running the teacher model with the same BPE action interface on 500 ALFWorld training games and keeping only successful episodes\. This yields 87 trajectories and 1,153 next\-action conversation pairs, with an average length of 26\.5 turns per episode\. The data covers all six ALFWorld task families: pick\-two \(28\), pick\-place \(17\), clean \(15\), heat \(11\), light \(8\), and cool \(8\)\. Each example contains the task objective, current observation, admissible commands, recent action history, and active harness views, while the target is the teacher’s next<think\>and<action\>response\. The teacher uses 405 harness calls in total, about 18% of all turns, distributed ascommit\(202\),recall\(114\),note\(55\), andtrack\(34\)\. The resulting SFT checkpoint initializes GRPO, and the experience store accumulated during collection is used as the initial skill bank\.

#### GRPO training\.

GRPO starts from the SFT checkpoint and samples multiple trajectories per prompt\. Rewards are normalized within each group, and the policy is optimized with a KL penalty to the SFT reference\. During rollout collection, trajectory summaries andnotebuffers are accumulated but not consolidated immediately\. At the end of each epoch, the consolidation model updates the experience store from the buffered evidence\. This keeps the skill bank stable within a rollout batch while still allowing cross\-episode harness evolution over training\. The detailed configuration is provided in the Table[4](https://arxiv.org/html/2608.05446#S9.T4)\.

Table 4:Implementation and training hyperparameters forEvoHarness\-RLon ALFWorld\.

## 10Prompts

We reproduce the key prompts that instantiate the BPE harness on ALFWorld\. They fall into two families: theruntime harness promptthat the policyπθ\\pi\_\{\\theta\}sees at every step \(defining how𝒜bpe\\mathcal\{A\}\_\{\\mathrm\{bpe\}\}is exposed\), and theconsolidation promptthat the summarizer LLM uses to evolve the experience storeEtE\_\{t\}at epoch boundaries\. Both the SFT teacher and the GRPO policy see the same runtime prompt, and both the SFT\-time and GRPO\-time skill consolidation go through the same experience\-store interface, so the two stages are prompt\-consistent by construction\. We lightly abridge the verbatim text to fit the page\.

### 10\.1Runtime Harness Prompt

The system prompt below defines both the environment action set𝒜env\\mathcal\{A\}\_\{\\mathrm\{env\}\}and the four harness meta\-actions𝒜bpe=\{commit,track,recall,note\}\\mathcal\{A\}\_\{\\mathrm\{bpe\}\}=\\\{\{\\color\[rgb\]\{0\.6484375,0\.44921875,0\.09765625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.6484375,0\.44921875,0\.09765625\}\\textbf\{commit\}\},\{\\color\[rgb\]\{0\.6484375,0\.44921875,0\.09765625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.6484375,0\.44921875,0\.09765625\}\\textbf\{track\}\},\{\\color\[rgb\]\{0\.6484375,0\.44921875,0\.09765625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.6484375,0\.44921875,0\.09765625\}\\textbf\{recall\}\},\{\\color\[rgb\]\{0\.6484375,0\.44921875,0\.09765625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.6484375,0\.44921875,0\.09765625\}\\textbf\{note\}\}\\\}, groundingcommitto ProgressPtP\_\{t\},trackto BeliefBtB\_\{t\}, andrecall/noteto ExperienceEtE\_\{t\}\. It also encodes when each action should be used, including the mandatorynoteconditions that drive experience writing when recalled hints are empty or stale\.

Harness system prompt \(policy\-facing\)You are an autonomous intelligent agent operating in the ALFWorld text\-based household environment\. You must complete tasks by interacting with objects in rooms\.You will receive: OBJECTIVE, OBSERVATION, ADMISSIBLE COMMANDS, PREVIOUS ACTION\(S\)\.\#\# Environment Actions\(choose EXACTLY from ADMISSIBLE COMMANDS\) \- go to <recep\>, take <obj\> from <recep\>, put <obj\> in/on <recep\> \- open/close <recep\>, heat/cool/clean <obj\> with <appliance\>, use <tool\> \- examine/look/inventory\#\# Harness Actions\(cognitive tools — always available, each costs one step\) \#\#\# Plan \-commit\[subgoal\]: Register ONE subgoal\. Use at task start and when switching subgoals\. e\.g\.commit\[find egg\] \#\#\# Perception \-track\[object\]: Query a specific object — where it was seen, its state, and which locations you have visited\. e\.g\.track\[egg\] \#\#\# Experience \-recall\[query\]: Retrieve past experience\. What you get depends on the query: recall\[where to find X\]→\\rightarrowsearch hints \(object→\\rightarrowlocation\) recall\[how to do Y task\]→\\rightarrowprocedures, general skills, common mistakes recall\[mistakes to avoid\]→\\rightarrowcommon pitfalls and fixes \-note\[insight\]: Record a generalizable discovery for future episodes\. e\.g\.note\[food usually in fridge or countertop\]\#\# When to Use Harness \-commitat task start and when switching subgoals\. \-recallin all three modes throughout the episode \(not just “where to find”\)\. \-trackto recall objects seen earlier but not interacted with \(esp\. pick\-two tasks\)\. \-noteis MANDATORY when: \(1\) RECALLED HINTS was empty and you found the object→\\rightarrownote\[<obj\> found in <loc\>\]; \(2\) object found in a location NOT in RECALLED HINTS→\\rightarrownote\[found <obj\> in <loc\>, recalled hints said <X\> instead\]; \(3\)recall\[how to do X\] returned empty and you finished→\\rightarrownote\[for <task type\>: <procedure that worked\>\]\.\#\# Output Format\(MANDATORY\) <think\>Brief reasoning in 1\-2 sentences\.</think\> <action\>your action here</action\>Rules:exactly one action per turn; environment actions must match ADMISSIBLE COMMANDS; harness actions are always available\.

At each step the policy additionally receives a per\-turn user message carrying the current contextxtx\_\{t\}and the rendered harness viewshth\_\{t\}\(the committed plan, the last recalled hints, and anytrack/recallresults\):

Per\-turn user templateOBJECTIVE: \{task\_description\} OBSERVATION: \{current\_observation\} ADMISSIBLE COMMANDS: \{admissible\_commands\} PREVIOUS ACTION\(S\): \{action\_history\}\{harness\_views\} \# PLAN, RECALLED HINTS, track/recall results

### 10\.2Experience\-Store Consolidation Prompts

All prompts in this subsection are executed by the external summarizer LLM, not by the policy\. The*note\-consolidation*prompt is the online path: bufferednoteinsights are not written toEtE\_\{t\}directly; at each consolidation point the summarizer receives the recent notes plus a compact view of the existing skill bank and decides, per note, whether toadd,update,remove, orskipa skill\. This is the mechanism by which policy\-written evidence is turned into structured, deduplicated, and self\-correcting skills \(including removal of stale priors\), realizing the co\-adaptive experience loop of Section[2](https://arxiv.org/html/2608.05446#S2)\. The remaining four*batch\-induction*prompts seed the initial skill bankESFTE^\{\\mathrm\{SFT\}\}and periodically rebuild whole categories from accumulated trajectory batches\. All five share the JSON\-only output convention and populate the four categories ofStS\_\{t\}\.

Note\-consolidation promptYou are a skill manager for a household task agent\. You receive raw notes from the agent’s recent episodes\. Your job is to maintain a clean, deduplicated, conflict\-free skill bank\.CURRENT TASK TYPE: \{task\_type\} RAW NOTES FROM AGENT: \{notes\_text\} EXISTING SKILLS \(id \| title \| principle\): \{existing\_skills\}For each note, decide ONE of: 1\.ADD— genuinely new knowledge not covered by any existing skill\. 2\.UPDATE— refines / corrects / extends an existing skill \(reference skill\_id\)\. 3\.REMOVE— contradicts / invalidates an existing \(wrong\) skill \(reference skill\_id\)\. 4\.SKIP— trivial, redundant, or too episode\-specific\.Return a JSON object with keys: adds\[\], updates\[\], removes\[\], search\_priorities\{\}\.Rules:prefer UPDATE over ADD on overlap; use REMOVE only when a note clearly invalidates an existing skill; keep principles short and concrete; extract object→\\rightarrowlocation mappings into search\_priorities; return empty lists/objects if nothing to change\. Return ONLY the JSON object\.

General\-skills induction promptYou are an expert at distilling agent behavior patterns into concise, actionable skills\.Analyze these trajectories from an agent operating in ALFWorld household environments\.SUCCESSFUL TRAJECTORIES: \{success\_data\} FAILED TRAJECTORIES: \{failure\_data\}Generate 5\-7 GENERAL SKILLS that apply across ALL task types\. Each skill should be: \(1\) concise \(1\-2 sentences\); \(2\) actionable \(clear what to do\); \(3\) derived from what works in successes and what fails in failures\.Format as a JSON array of \{skill\_id, title, principle, when\_to\_apply\}\. Return ONLY the JSON array\.

Task\-specific\-skills induction promptYou are an expert at distilling agent behavior patterns into concise, actionable skills\.Task Type: \{task\_type\}SUCCESSFUL TRAJECTORIES: \{success\_data\} FAILED TRAJECTORIES: \{failure\_data\}Generate 4\-6 TASK\-SPECIFIC SKILLS for \{task\_type\} tasks\. Each skill should be specific to this task type\.Format as a JSON array of \{skill\_id, title, principle, when\_to\_apply\}\. Return ONLY the JSON array\.

Common\-mistakes induction promptYou are an expert at analyzing agent failures\.FAILED TRAJECTORIES: \{failure\_data\}Generate 3\-5 COMMON MISTAKES to avoid\. Format as a JSON array of \{mistake\_id, description, why\_it\_happens, how\_to\_avoid\}\. Return ONLY the JSON array\.

Search\-priorities induction promptYou are analyzing where objects are typically found in household environments\.From these successful trajectories, extract which receptacles commonly contain which object types\.TRAJECTORIES: \{trajectory\_data\}Generate a JSON object mapping object categories to ordered lists of receptacles \(most likely first\), e\.g\. \{“food”: \[“fridge”, “countertop”, “diningtable”\], “tool”: \[“drawer”, “shelf”\]\}\. Only include categories with clear patterns\. Return ONLY the JSON object\.