Banger paper from CMU on harness learning.

(bookmark it)

Also, pay attention to this important new AI engineering skill of improving agents by editing their harness code instead of their weights.

Seeing a huge shift towards this.

The authors train a proposer model with RL to read a task, the current harness and an execution report, then write a code edit to the harness.

The reward is the score of the revised harness. The solver model never changes.

A trained 4B proposer beats its 35B teacher at single-step revision on Reasoning Gym, including task families it never saw in training. A proposer trained on HotpotQA keeps improving harnesses on MuSiQue and 2WikiMultihopQA.

Paper: https://arxiv.org/abs/2609.35738

Chat with Paper: https://academy.dair.ai/papers/harness-learning-enables-generalizable-test-time-adaptation-2609.35738…

Harness Learning Enables Generalizable Test-Time Adaptation

Source: https://arxiv.org/html/2609.35738 Alvin Zhang, Xuecheng Liu*, Zixuan Wang*Fahim Tajwar, Daman Arora, Ruslan Salakhutdinov††thanks:Equal contribution.††thanks:Project lead.Yuda Song††thanks:Equal advising.Andrea Zanette

Abstract

A language-model agent is jointly defined by its model and its harness, the executable program that organizes model calls, tool use, and information flow. Because different tasks call for different ways of organizing these operations, the harness needs to be adapted using feedback from the task at hand. We introduceharness learning, which trains a proposer model to revise a solver’s harness using execution feedback. We formulate this process as meta-learning over executable programs, with harness revisions playing the role of weight updates in gradient-based adaptation. We train the proposer with reinforcement learning, using the task performance of revised harnesses as the reward. At test time, the proposer uses feedback from successive executions on a new task to refine the harness, without performing any parameter-space update. Experiments on reasoning and multi-hop question answering show that harness learning improves revision quality and that the ability to adapt at test time transfers to unseen tasks. Policies trained on individual revisions can continue improving harnesses over multiple rounds, while the benefits of training on revision sequences vary across settings. These findings suggest a path towards continually learning agents that turn accumulated experience into generalizable improvements.

1Introduction

Aharnessis the executable program that organizes calls to a foundation model and interactions with tools(Yang et al., 2024;Khattab et al., 2024). It determines what the model sees, which tools it can use, and how execution proceeds. Because these choices strongly shape performance(Shinn et al., 2023;Yang et al., 2024), adapting the harness provides a way to improve how a model uses its capabilities without changing its parameters(Zhang et al., 2025b;Lou et al., 2026).

Execution feedback can guide choices about model calls, tool use, and intermediate outputs, but finding effective harness revisions often requires repeated experimentation at test time, and revisions that work on one task may not transfer to another. We therefore askwhether a model can learn to revise harnesses from execution feedback in a way that generalizes to new tasks. We call this problemharness learningand view it asmeta-learningover executable programs(Duan et al., 2016;Finn et al., 2017). The outer loop trains a revision model, which we call theproposer, from execution outcomes, and the inner loop uses the proposer to adapt a harness from execution feedback. Training therefore learns a reusable revision procedure that can transfer to unseen tasks without updating model parameters at test time(Baxter, 2000;Andrychowicz et al., 2016).

We implement harness learning by training the proposer to edit executable harness code (Fig.1). Given a task description, the current harness, and an execution report, the proposer generates a code change. We train the proposer with reinforcement learning, using the performance of the resulting harness as the reward, and optionally initialize it with supervised fine-tuning. At test time, the same proposer can be applied repeatedly, using feedback from each execution to guide the next revision.

We evaluate harness learning on reasoning and multi-hop question answering. On Reasoning Gym(Stojanovski et al., 2025), training improves revision quality on task families excluded from both supervised and reinforcement learning, and in single-step revision the trained 4B proposer outperforms the 35B teacher on average. In multi-hop question answering, a proposer trained on HotpotQA(Yang et al., 2018)transfers to MuSiQue(Trivedi et al., 2022)and 2WikiMultihopQA(Ho et al., 2020). In both settings, trained proposers can continue improving harnesses over successive rounds of test-time adaptation. In QA, even a proposer trained only on seed-harness inputs continues improving revised harnesses, suggesting generalization across adaptation contexts and tasks.

We make the following contributions.

Learning to adapt executable harnesses at test time.We introduce harness learning as meta-learning over executable programs: a model learns from execution outcomes how to revise the harness that organizes model calls and tool use. At test time, the model uses execution feedback to adapt harnesses for new tasks without any parameter updates.

Generalization to unseen tasks.We show that the capability of test-time adaptation via harness revision transfers to reasoning tasks and question-answering benchmarks that are unseen during training.

Iterative test-time adaptation.We show that proposers trained on individual revisions can continue improving harnesses over successive rounds of test-time adaptation. We analyze how training changes the reliability of the revision, the structure of the auxiliary, and the use of generated tools.

Figure 1:Harness learning.The proposer reads the task description, the current harnessht−1h_{t-1}, and its execution report, and writes a code edityty_{t}, here adding answer validation and a retry. Running the revised harnesshth_{t}yields a reward and feedback for the next round. Training updates only the proposer, and at test time both models are frozen.Fig.2details training.

2Related Work

Harness optimization.

Automated agent design optimizes language-model pipelines and workflows(Khattab et al., 2024;Zhang et al., 2025b), with harness-specific methods using execution feedback to revise the programs that govern agent behavior(Lee et al., 2026;Lou et al., 2026). Concurrent work also trains dedicated models for harness construction and revision(Shao et al., 2026;Zhang et al., 2026a). Harness-R1 trains an editor through supervised initialization and reinforcement learning with a frozen solver, a training recipe similar to ours. JIT-Agent learns task-conditioned harness generation, repair, and evolution. Our study frames harness revision as a learned adaptation procedure and examines whether successive rounds of execution feedback improve performance on task families excluded from training.

Meta-learning.

Meta-learning uses experience across tasks to learn an initialization or an adaptation procedure(Duan et al., 2016;Finn et al., 2017;Hospedales et al., 2022). AdaptFlow learns a shared workflow initialization(Zhu et al., 2025), and The Last Harness proposes an outer loop over harness-evolution blueprints(Seong et al., 2026). Our formulation treats executable harness code as the adapted object and trains the proposer as the adaptation rule. Treating harness code as the adapted object lets the inner loop change control flow, tool interfaces, and the sequence of model calls, and the proposer learns from execution feedback how to make these discrete changes. At test time, both proposer and solver remain frozen, and graded execution feedback on development questions guides successive harness revisions. TTHE also adapts executable harnesses with frozen models, using unlabeled execution traces as feedback(Nie et al., 2026).AppendixAdiscusses broader adaptation methods and evaluation protocols.

3Preliminaries

3.1Harness

Given a language model, aharnessis the executable program that orchestrates the model’s calls throughout a task. The harness constructs the context for each model call, controls when calls occur, executes tool requests, and incorporates environmental feedback to support iterative problem solving. Harness design determines how a model interacts with its environment. For example, a coding-agent harness can expose tools for inspecting files, editing code, and running tests, then return the resulting observations to the model(Yang et al., 2024). A harness can be changed while the model’s parameters stay fixed.

3.2Reinforcement learning with verifiable rewards

Apolicyπθ​(y∣x)\pi_{\theta}(y\mid x)is an autoregressive language model that maps a contextxxto a token sequencey=(y1,…,y|y|)y=(y_{1},\dots,y_{|y|}). In reinforcement learning with verifiable rewards, a programmatic grader evaluates responses or their execution outcomes. Training maximizes the expected reward over a prompt distribution𝒟\mathcal{D},

𝒥(θ)=𝔼x∼𝒟𝔼y∼πθ(⋅∣x)[r(x,y)].\mathcal{J}(\theta)=\mathbb{E}_{x\sim\mathcal{D}}\,\mathbb{E}_{y\sim\pi_{\theta}(\cdot\mid x)}\big[r(x,y)\big].\addcontentsline{lla}{section}{\numberline q:rl_{o}bjective}(1)

GRPO.

Group relative policy optimization(Shao et al., 2024)normalizes rewards within a group of responses{yi}i=1G\{y_{i}\}_{i=1}^{G}sampled fromπθold\pi_{\theta_{\mathrm{old}}}for the same context. With rewardsri=r⁡(x,yi)r_{i}=r(x,y_{i}), each token ofyiy_{i}receives the group-normalized advantageA^i=(ri−meanj⁡rj)/(stdj⁡rj+δ)\hat{A}_{i}=(r_{i}-\operatorname{mean}_{j}r_{j})/(\operatorname{std}_{j}r_{j}+\delta). Letωi,k​(θ)=πθ​(yi,k∣x,yi,<k)/πθold​(yi,k∣x,yi,<k)\omega_{i,k}(\theta)=\pi_{\theta}(y_{i,k}\mid x,y_{i,<k})/\pi_{\theta_{\mathrm{old}}}(y_{i,k}\mid x,y_{i,<k})denote the token-level importance ratio. The clipped surrogate objective is

𝒥(θ)=𝔼x∼𝒟𝔼{yi}∼πθold(⋅∣x)[1∑j=1G|yj|∑i=1G∑k=1|yi|min(ωi,k(θ)A^i,clip(ωi,k(θ),1−ϵ,1+ϵ)A^i)]\displaystyle\mathcal{J}(\theta)=\mathbb{E}_{x\sim\mathcal{D}}\,\mathbb{E}_{\{y_{i}\}\sim\pi_{\theta_{\mathrm{old}}}(\cdot\mid x)}\left[\frac{1}{\sum_{j=1}^{G}|y_{j}|}\sum_{i=1}^{G}\sum_{k=1}^{|y_{i}|}\min\!\Big(\omega_{i,k}(\theta)\hat{A}_{i},\ \operatorname{clip}(\omega_{i,k}(\theta),1-\epsilon,1+\epsilon)\,\hat{A}_{i}\Big)\right]

(2)

4Harness Learning as Meta-Learning

Harness learning trains a proposer to adapt the executable harness of a fixed solver. The proposer learns from revision outcomes during training and uses execution feedback to revise harnesses on new tasks at test time. We first define how a harness is evaluated, then describe the adaptation loop and the procedure used to train its revision rule.

4.1Tasks, harnesses, and execution feedback

A task provides a description, a set of questions, and a grader that assigns scores in[0,1][0,1]. The task is the unit of harness adaptation. In our experiments, a task is either a Reasoning Gym family, such asmazeorsudoku, whose questions share a generator and a grader, or a multi-hop QA benchmark such as HotpotQA. A harnesshhanswers questions by organizing calls to the solver, tool use, and direct computation. We keep the solver fixed and omit it from the notation.

Harness quality.

Letscore⁡(h,q)\operatorname{score}(h,q)denote the observed grader score from executing harnesshhon questionqq. We evaluate a harness on a finite question setQQusing its mean score,

J⁡(h,Q)=1|Q|​∑q∈Qscore⁡(h,q).J(h;Q)=\frac{1}{|Q|}\sum_{q\in Q}\operatorname{score}(h,q).\addcontentsline{lla}{section}{\numberline q:harness_{q}uality}(3)The score is empirical and can vary across executions when the solver is stochastic. Execution also produces traces, which we summarize together with task outcomes in anexecution report.AppendixEgives the report formats.

Question sets.

We divide the full question set into subsets that serve different purposes. We useQfbQ^{\mathrm{fb}}to construct execution reports,QscoreQ^{\mathrm{score}}to score candidate revisions, andQevalQ^{\mathrm{eval}}to measure held-out performance. Evaluation questions are disjoint from all questions used for feedback or candidate selection during adaptation. Feedback and scoring sets may coincide when the evaluation protocol uses the same development questions for both roles. Round subscripts indicate the sets used at a particular revision round.

4.2Harness adaptation and the meta-learning objective

The proposerπθ\pi_{\theta}defines the learned revision rule. At test time, its parametersθ\thetaand the solver’s parameters remain fixed while the harness changes. Starting from a seed harnessh0h_{0}, adaptation proceeds forTTrounds.

At roundtt, the proposer inputxtx_{t}contains the task description, the current harnessht−1h_{t-1}, and a report from executing that harness onQtfbQ_{t}^{\mathrm{fb}}. The input may also include selected information from earlier rounds. The proposer samplesGGresponses, applies their edits, and retains a harness according to the selection protocol,

yi\displaystyle y_{i}∼πθ(⋅∣xt),\displaystyle\sim\pi_{\theta}(\cdot\mid x_{t}),i=1,…,G,\displaystyle i=1,\ldots,G,(4)hi′\displaystyle h_{i}^{\prime}=Apply⁡(ht−1,yi),\displaystyle=\operatorname{Apply}(h_{t-1},y_{i}),ht\displaystyle h_{t}=Select⁡(ht−1,{hi′}i=1G,Qtscore).\displaystyle=\operatorname{Select}\!\left(h_{t-1},\{h_{i}^{\prime}\}_{i=1}^{G};Q_{t}^{\mathrm{score}}\right).The indexiiidentifies candidates within the current round.Apply\operatorname{Apply}extracts code edits from the generated response and applies them to the parent harness, marking proposals invalid when their edits cannot be parsed or applied. In our structured interface, an edit identifies a span of existing code and supplies its replacement.

Select\operatorname{Select}evaluates candidates on the scoring questions and applies the protocol’s validity and retention rules. For example, our Reasoning Gym protocol samples one candidate per round (G=1G=1), uses the same development questions for feedback and scoring, and retains the candidate only if its development score exceeds that of the current harness (Appendix3). Our QA protocol samples eight candidates, assigns a score of zero to those that fail to parse or run, and advances to the highest-scoring candidate even when it scores below its parent (AppendixD.3). Executing the retained harness on the next round’s feedback questions provides the report forxt+1x_{t+1}.

Learning to adapt.

For a fixed proposer, the revision loop maps a seed harnessh0h_{0}to an adapted harnesshTh_{T}and forms the inner loop of meta-learning. The outer-loop goal is to learn proposer parameters that improve the held-out performance of the adapted harness,

maxθ⁡𝔼⁡[J⁡(hT,Qeval)].\max_{\theta}\ \mathbb{E}\!\left[J(h_{T};Q^{\mathrm{eval}})\right].\addcontentsline{lla}{section}{\numberline q:outer}(5)HerehTh_{T}is produced byEq.4usingπθ\pi_{\theta}. The expectation averages over the training-task distribution, sampled questions, proposed revisions, and stochastic harness executions. Training can draw from one or several tasks, as specified inSection5.1.

Our training procedure uses the immediate performance of individual revisions as a surrogate for this final-harness objective. Each update assigns credit from the candidate’s own outcome. Training on successive revision states changes the inputs encountered by the proposer while retaining this local objective.

4.3Training the proposer

Training updates only the proposer. Each revision inputxxcombines a training task, a parent harness, and an execution report, so the input distribution𝒟\mathcal{D}inSection3.2is a distribution over these revision contexts.

Optional supervised initialization.

We optionally initialize the proposer with successful teacher revisions. For each context used to collect SFT data, a teacher receives the seed harness and its execution report and proposes edits. We execute the resulting harnesses and retain revisions that improve on the seed and pass the quality filters inAppendixB.2. Supervised fine-tuning maximizes the likelihood of the retained teacher responses, providing an initialization for learning from the outcomes of the proposer’s own revisions.

Figure 2:Reinforcement learning for harness revision.Given inputxtx_{t}, the proposer samplesGGedits to the parent harnessht−1h_{t-1}. Each candidatehi′h_{i}^{\prime}runs on the same scoring questions, which gives its rewardrir_{i}and a group-normalized advantage. For successive revision states, a selected harnesshth_{t}runs on fresh feedback questions, disjoint from the scoring questions, to form the next inputxt+1x_{t+1}.

Reinforcement learning from revision outcomes.

For each input, we sample candidate revisions as inEq.4and evaluate them on a common scoring set from the same task. During training, scoring questions are disjoint from those used to construct the input’s execution report. An evaluable candidate receives reward

ri=J⁡(hi′,Qscore)+vi,r_{i}=J(h_{i}^{\prime};Q^{\mathrm{score}})+v_{i},\addcontentsline{lla}{section}{\numberline q:reward}(6)whereviv_{i}is an auxiliary reward for edit validity and successful execution. These rewards determine the group-normalized advantages used by GRPO (Section3.2).AppendixB.3specifies the validity terms for Reasoning Gym, where a failed edit receives only the validity credit accumulated before the failure and explicit no-change responses are scored by executing the parent.Fig.2illustrates the update.

Training on successive revision states.

Single-step training treats each parent harness and report as an independent revision context. Training on successive revisions constructs additional contexts from selected candidate harnesses and their refreshed execution reports. For Reasoning Gym, we refresh these states between offline training phases; for QA, we generate revision sequences online. Each state uses the same immediate-reward update, so both procedures train the proposer to improve the harness presented in its current input.Section5.1andAppendixB.4describe the configurations. We call the resulting configurations Single-step RL and Multistep RL. Either trained proposer can be applied over multiple revision rounds at test time.

5Experiments

Figure 3:Single-step harness revision.Held-out scores on format-varied questions for (a) three seen and (b) three unseen families. Bars average eight proposals, and black lines mark the teacher. (c) Mean over 21 unseen families with one standard error. Circles mark oracle best-of-eight scores, selected on held-out scores, and the dashed line marks the seed. Full results are inTable9.Figure 4:Multistep harness revision on unseen families.(a) Canonical and (b) format-varied held-out scores over five rounds, averaged over 15 unseen families. Harnesses are retained by development score on separate questions that also supply feedback, and Multistep RL is shown at round five. (c) Eight independent proposals and five sequential revisions on format-varied questions. Hollow circles mark oracle best-of-eight scores, dashed lines mark the seed, and error bars show one standard error across families.To evaluate harness learning, we organize our experiments around three questions:

• •Do learned revisions generalize to tasks unseen during training?
• •Does successive test-time revision improve harnesses beyond a single revision?
• •Does training on revision sequences improve adaptation over training on individual revisions?

We study these questions in two settings with separately trained proposers. Reasoning Gym tests generalization across diverse synthetic task families from a minimal seed harness that makes one solver call. Multi-hop question answering tests whether learned revisions improve an existing retrieval workflow.

5.1Experimental setup

Table1summarizes the two settings, andAppendicesBandDgive the full protocols.Unseen(OOD) tasks are excluded from the proposer’s supervised fine-tuning and reinforcement learning. Seed denotes the unrevised seed harness, and Base denotes revisions generated by the base proposer before training. SFT, Single-step RL, and Multistep RL denote training configurations, and each resulting proposer can perform single-step or multistep revision at test time. For Reasoning Gym,canonicalquestions use the benchmark’s standard format, whileformat-variedquestions present the same tasks in alternative formats.

Table 1:Experimental settings.Oracle best-of-NNstatistics select candidates by held-out score.

5.2Learning transferable revisions on Reasoning Gym

Figure 5:Independent and sequential QA revision.(a,b,c) Held-out exact match over ten rounds. Solid lines average four runs, and dashed lines show the running maximum across runs. (d,e) On the unseen benchmarks, light bars average 80 independent revisions and dark bars average the final scores of four ten-round runs with the same 80-proposal budget. Circles, diamonds, and triangles mark the oracle best of 8, the best of 80, and the best over all rounds and runs. Dashed lines mark the seed.AppendixD.4gives the protocol.#### Generalization of individual revisions.

Both training stages improve mean single-revision scores on 21 unseen families, from 0.32 for Base to 0.62 after RL (Fig.3,AppendixB.1). The improvement appears in the mean over all proposals, so it does not depend on selecting the best candidate. After RL, the average proposal already exceeds the seed harness, while Base and SFT proposals fall below it on average.

Comparison with the teacher.

On the 21 unseen families, the trained 4B proposer’s revisions score above those of its 35B teacher on average (0.62 versus 0.56) under the same revision protocol. The teacher achieves higher scores under oracle best-of-eight selection. Learning from execution outcomes after supervised initialization can therefore raise average proposal quality above the teacher’s.AppendixB.1details the comparison, including the treatment of failed edits and zero-scoring harnesses.

Gains across revision rounds.

On format-varied questions, Single-step RL obtains most of its five-round gain in the first revision, while SFT improves more gradually across rounds (Fig.4b). On canonical questions, the two proposers improve on similar schedules (Fig.4a).Section6.3discusses why later revisions add less.

Sequential versus independent revision.

With five sequential proposals, the retained harness approaches or exceeds the oracle best of eight independent proposals and exceeds their average (Fig.4c). Building on the current best harness therefore uses a small proposal budget effectively.AppendixC.2gives in-domain and per-family results.

5.3Adapting retrieval workflows on unseen QA benchmarks

Figure 6:Proposal reliability and harness structure.(a) Failed or zero-scoring proposals in Reasoning Gym (15 canonical, 12 format-varied, and 5 unseen families). (b) Harness classes on five unseen families without a composition directive. (c) Composition under a helper-tool directive on four unseen families. Structural composition places helpers in the solver’s tool loop, and functional composition also requires a helper to contribute to the answer. (d) Failed or zero-scoring proposals in QA (320 candidates per benchmark and method). (e) Structures of retained QA harnesses over ten rounds, pooled over three benchmarks (n=120n=120per method), where Passages and Summaries denote multi-hop retrieval from passages or solver summaries. Counts are inTables10and18.#### Progress over successive revisions.

Both RL proposers continue to improve beyond the first revision and finish above Base on all three benchmarks, including the two unseen ones, while Base ends below the seed on HotpotQA and MuSiQue (Fig.5a–c). Improvement also persists over more rounds than on Reasoning Gym, and the first revision accounts for only one third to two thirds of the ten-round gain.Section6.3discusses this difference.

Sequential versus independent revision.

On MuSiQue, Single-step RL averages 0.15 exact match for independent revisions, only marginally above the seed (Table24), versus 0.27 after ten revision rounds. On both unseen benchmarks, the average final run of each RL proposer exceeds even the oracle best of 80 independent proposals (Fig.5d–e). Average improvement from the seed therefore does not fully capture a proposer’s ability to improve harnesses through repeated adaptation. Each run uses 80 proposals, so the comparison matches the number of proposals per run and leaves total compute and feedback unmatched.

5.4Effect of training on revision sequences

Training on successive revision contexts with immediate rewards does not consistently improve on training on individual revisions. On Reasoning Gym, Multistep RL ends above Single-step RL on canonical questions and below it on format-varied questions (Fig.4a,b), and training on fixed later-round states also shows no gain (AppendixC.3). In QA, the two configurations reach nearly identical final scores (Fig.5a–c). In both settings, proposers trained only on individual revisions already improve harnesses over successive rounds, which suggests that iterative improvement can emerge from single-revision training.

6Analysis and Discussion

We examine how training changes the reliability and structure of proposals, and what these changes suggest about repeated revision.

6.1Training improves proposal reliability

Training reduces the fraction of proposals that fail to produce a runnable harness or score zero on the development questions (Fig.6a,d;Table10). On unseen Reasoning Gym families, most of the reduction comes from RL, and SFT alone leaves the rate nearly unchanged. In QA, RL from the base proposer produces a similar reduction. A trained proposer therefore supplies more viable candidates to selection in each round, which may help sustain progress across successive revisions.

6.2Qualitative analysis of learned harnesses

Figure 7:A learned QA revision.The answering call reads retrieved passages directly (red), and first-hop summaries still guide retrieval.On Reasoning Gym, SFT and Single-step RL mostly generate interpreter loops, which let the solver formulate a computation as code and delegate its execution to the runtime (Fig.6b;Table18). Interpreter loops are also the most common structure in the SFT data (48% of teacher revisions;AppendixB.2) and remain dominant after RL. Under a composition directive, helpers contribute to the answer in only a minority of the harnesses that integrate them into the solver’s tool loop (Fig.6c), and composition rewards do not yield consistent gains (AppendixC.4). Unsuccessful composed harnesses often pair a correct helper algorithm with failures in question parsing, tool communication, or answer extraction (Table21), so a generated component contributes only when these interfaces also work.

In QA, RL preserves multi-hop retrieval, while Base sometimes reduces the harness to a single solver call or removes the solver (Fig.6e). A common RL revision keeps summaries for guiding the next retrieval and passes retrieved passages directly to the answering call (Fig.7), which separates the evidence needed to formulate a query from the evidence needed to answer. Other revisions add a third retrieval hop or search again when a draft answer is absent from the retrieved passages (Fig.15). In both settings, the trained proposers change how the harness organizes computation and information. Their transfer to unseen tasks suggests that they learn a reusable adaptation capability from execution outcomes.

6.3Repeated revision

The seed harness may explain why QA gains accumulate over more rounds than Reasoning Gym gains. On Reasoning Gym, single-step revisions commonly introduce interpreter loops. In the composition experiments, a correct helper algorithm may still fail to contribute through the surrounding parsing and tool interfaces, which is one obstacle to further gains. The QA seed already contains a retrieval workflow whose depth, evidence access, and search triggers can each be refined in later rounds. Differences in training and selection procedures prevent attributing this contrast solely to the starting harness.

Repeated revision also tests whether a policy remains useful as its own edits change the adaptation context. In QA, Single-step RL trains on seed-harness inputs yet continues to improve revised harnesses at test time (Fig.5). This continued improvement suggests that the learned revision skills of Single-step RL extend to intermediate harnesses and execution reports beyond those encountered in training, as well as to unseen benchmarks.

Our experiments on training with revision sequences change the parent harnesses and execution reports used for training and keep each reward local to a single revision (Section4.3). They leave open whether training that assigns credit across rounds would make longer revision chains more effective. Supervised demonstrations of successive revisions, or different distributions of intermediate harnesses and feedback, could yield larger gains from multistep training.

7Conclusion

Harness learning trains a proposer to adapt a frozen solver’s executable harness using execution feedback, with both models’ parameters fixed at test time. On 21 unseen Reasoning Gym families, SFT followed by RL raises mean single-step revision scores from 0.32 with the base proposer to 0.62, with the trained 4B proposer outperforming its 35B teacher on average. In multi-hop QA, a proposer trained with RL from the base proposer on HotpotQA transfers to unseen benchmarks. Policies trained on individual revisions also improve harnesses over multiple rounds, suggesting that learned revision skills remain useful as the harness changes.

Limitations and future work.Training uses prescribed revision prompts and a fixed solver, with one teacher for Reasoning Gym supervision. Future work could explore stronger teachers or direct RL from models with prior knowledge of harness design, train tool-using proposers to inspect and test designs, and jointly optimize proposer and solver following concurrent work(Ornith, 2026a;Ornith, 2026b). We emphasize transfer to unseen tasks and longer tool-use trajectories (AppendixF).

Acknowledgment

This work was partially carried out at the Advanced Research Computing at Hopkins (ARCH) core facility (Skipjack), which is supported by the National Science Foundation (NSF) grant number OAC1920103. The authors thank the CMU FLAME center and the CMU Babel Compute Cluster for compute support for this project. This research also used resources of the Oak Ridge Leadership Computing Facility (OLCF) and Argonne Leadership Computing Facility (ALCF)] which are a DOE Office of Science User Facility. This work was supported by an award from the ASCR Leadership Computing Challenge (ALCC) under project ERCAP0034861. Part of this work was supported by the National Science Foundation under Grant CCF-2106778. YS acknowledge and thank the support of NSF AI Institute for Societal Decision Making AI-SDM grant IIS2229881. FT gratefully acknowledges the support of Qualcomm Innovation Fellowship and Bosch Research and Technology Center.

References

• Andrychowiczet al.(2016)M. Andrychowicz, M. Denil, S. Gómez, M. W. Hoffman, D. Pfau, T. Schaul, B. Shillingford, and N. de FreitasLearning to learn by gradient descent by gradient descent.InAdvances in Neural Information Processing Systems,D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (Eds.),Vol.29.External Links:LinkCited by:§1.
• Baxter (2000)J. BaxterA model of inductive bias learning.Journal of Artificial Intelligence Research12,pp. 149–198.External Links:Document,LinkCited by:§1.
• Duanet al.(2016)Y. Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever, and P. AbbeelRL2{}^{2}: fast reinforcement learning via slow reinforcement learning.Note:arXiv preprint arXiv:1611.02779External Links:1611.02779,LinkCited by:Appendix A,§1,§2.
• Finnet al.(2017)C. Finn, P. Abbeel, and S. LevineModel-agnostic meta-learning for fast adaptation of deep networks.InProceedings of the 34th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol.70,pp. 1126–1135.External Links:LinkCited by:Appendix A,§1,§2.
• Hebbaret al.(2026)P. Hebbar, Y. Manawat, S. Verboomen, A. Ivanova, S. Palanimalai, K. Bhatia, and V. BaskaranSIA: Self Improving AI with Harness & Weight Updates.Note:arXiv preprint arXiv:2605.27276External Links:2605.27276,LinkCited by:Appendix A.
• Hoet al.(2020)X. Ho, A. Duong Nguyen, S. Sugawara, and A. AizawaConstructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps.InProceedings of the 28th International Conference on Computational Linguistics,pp. 6609–6625.External Links:Document,LinkCited by:§1.
• Hospedaleset al.(2022)T. Hospedales, A. Antoniou, P. Micaelli, and A. StorkeyMeta-learning in neural networks: a survey.IEEE Transactions on Pattern Analysis and Machine Intelligence44(9),pp. 5149–5169.External Links:Link,DocumentCited by:Appendix A,§2.
• Huanget al.(2026)L. Huang, C. Yang, H. Zhou, H. Song, Z. Chen, R. Le, Y. Song, W. X. Zhao, and T. ZhangEvo-Bench: Can Language Models Improve Agent Harness?.Note:arXiv preprint arXiv:2608.09096External Links:2608.09096,LinkCited by:Appendix A.
• Jianget al.(2025)D. Jiang, A. Zhang, A. Wang, N. Andrews, and D. KhashabiFeedback friction: llms struggle to fully incorporate external feedback.Note:arXiv preprint arXiv:2506.11930External Links:2506.11930,LinkCited by:Appendix A.
• Khattabet al.(2024)O. Khattab, A. Singhvi, P. Maheshwari, Z. Zhang, K. Santhanam, S. V. A, S. Haq, A. Sharma, T. Joshi, H. Moazam, H. Miller, M. Zaharia, and C. PottsDSPy: Compiling Declarative Language Model Calls into State-of-the-Art Pipelines.InInternational Conference on Learning Representations,External Links:LinkCited by:Appendix A,§1,§2.
• Kimet al.(2026)H. Kim, Y. Lee, G. Lee, C. Finn, and K. LeeWHALE: a simple recipe for joint harness-weight optimization.External Links:2609.00196,LinkCited by:Appendix A.
• Leeet al.(2026)Y. Lee, R. Nair, Q. Zhang, K. Lee, O. Khattab, and C. FinnMeta-Harness: End-to-End Optimization of Model Harnesses.Note:arXiv preprint arXiv:2603.28052External Links:2603.28052,LinkCited by:Appendix A,§2.
• Liet al.(2026)G. Li, B. D. Mishra, Z. Wang, J. Yan, Y. Chen, C. Li, L. T. Le, R. Han, G. Lee, H. Tong, C. Lee, and T. PfisterRubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards.Note:arXiv preprint arXiv:2605.10899External Links:2605.10899,LinkCited by:Appendix A.
• Linet al.(2026)M. Lin, J. Wu, Z. Wang, Z. Shi, Y. Sang, B. He, Z. Liu, T. Wei, Z. Wu, Z. Zhang, D. Wang, X. Zhang, B. Dumoulin, C. Xie, Y. Zhou, S. Wang, and H. LuHarness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents.Note:arXiv preprint arXiv:2605.30621External Links:2605.30621,LinkCited by:Appendix A.
• Liuet al.(2026)S. Liu, S. Agarwal, M. Maheswaran, M. Cemri, Z. Li, Q. Mang, A. Naren, E. Boneh, A. Cheng, M. Z. Pan, A. Du, K. Keutzer, A. Cheung, A. G. Dimakis, K. Sen, M. Zaharia, and I. StoicaEvoX: Meta-Evolution for Automated Discovery.Note:arXiv preprint arXiv:2602.23413External Links:2602.23413,LinkCited by:Appendix A.
• Louet al.(2026)X. Lou, M. Lázaro-Gredilla, A. Dedieu, C. Wendelken, W. Lehrach, and K. P. MurphyAutoHarness: improving LLM agents by automatically synthesizing a code harness.Note:arXiv preprint arXiv:2603.03329External Links:2603.03329,LinkCited by:Appendix A,§1,§2.
• Madaanet al.(2023)A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, and P. ClarkSelf-Refine: Iterative Refinement with Self-Feedback.InAdvances in Neural Information Processing Systems,Vol.36,pp. 46534–46594.External Links:Link,DocumentCited by:Appendix A.
• Nieet al.(2026)J. Nie, Y. Zhang, J. Song, Q. Cai, D. Yu, Y. Guo, X. Tian, and B. HanTTHE: test-time harness evolution.Note:arXiv preprint arXiv:2607.08124External Links:2607.08124,LinkCited by:Appendix A,§2.
• Ninget al.(2026)X. Ning, D. Fu, T. Wei, H. Zeng, Y. Bei, B. Li, Z. Li, Q. Wang, X. Shen, Y. Wu, J. Liu, H. Li, Y. Xia, X. Fan, H. Tong, and J. HeEvoHarness-RL: learning self-evolving runtime harness for long-horizon LLM agents.Note:arXiv preprint arXiv:2608.05446External Links:2608.05446,LinkCited by:Appendix A.
• Ornith (2026a)OrnithOrnith-1.0: Self-Scaffolding LLMs for Agentic Coding.Note:Ornith BlogExternal Links:LinkCited by:Appendix A,Appendix F,§7.
• Ornith (2026b)OrnithOrnith-1.5: From Self-Scaffolding to Self-Improvement.Note:Ornith BlogExternal Links:LinkCited by:Appendix A,Appendix F,§7.
• Seonget al.(2026)H. Seong, L. Yin, H. Zhang, and Z. ShiThe Last Harness You’ll Ever Build.Note:arXiv preprint arXiv:2604.21003External Links:2604.21003,LinkCited by:Appendix A,§2.
• Shaoet al.(2026)S. Shao, K. Zhang, Q. Li, S. Wang, H. Wang, W. Jiao, Y. Lu, Y. Guo, W. Liu, and W. ZhangHarness-R1: Learning to Edit Executable Runtime Harnesses from Agent Failure Trajectories.Note:arXiv preprint arXiv:2608.02276External Links:2608.02276,LinkCited by:Appendix A,§2.
• Shaoet al.(2024)Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. GuoDeepSeekMath: pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300.External Links:LinkCited by:§3.2.
• Shinnet al.(2023)N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. YaoReflexion: language agents with verbal reinforcement learning.InAdvances in Neural Information Processing Systems,Vol.36,pp. 8634–8652.External Links:Link,DocumentCited by:Appendix A,§1.
• Stojanovskiet al.(2025)Z. Stojanovski, O. Stanley, J. Sharratt, R. Jones, A. Adefioye, J. Kaddour, and A. KöpfReasoning Gym: Reasoning Environments for Reinforcement Learning with Verifiable Rewards.Note:arXiv preprint arXiv:2505.24760External Links:2505.24760,LinkCited by:§1.
• Tajwaret al.(2025)F. Tajwar, Y. Jiang, A. Thankaraj, S. S. Rahman, J. Z. Kolter, J. Schneider, and R. SalakhutdinovTraining a generally curious agent.External Links:2502.17543,LinkCited by:Appendix A.
• Tanget al.(2024)H. Tang, D. Key, and K. EllisWorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environment.External Links:2402.12275,LinkCited by:Appendix A.
• Trivediet al.(2022)H. Trivedi, N. Balasubramanian, T. Khot, and A. SabharwalMuSiQue: multihop questions via single-hop question composition.Transactions of the Association for Computational Linguistics10,pp. 539–554.External Links:Document,LinkCited by:§1.
• Wanget al.(2023)G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. AnandkumarVoyager: An Open-Ended Embodied Agent with Large Language Models.External Links:2305.16291,LinkCited by:Appendix A.
• Wanget al.(2026a)W. Wang, P. Kattakinda, and S. FeiziDo Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0.Note:arXiv preprint arXiv:2607.14004External Links:2607.14004,LinkCited by:Appendix A.
• Wanget al.(2026b)Y. Wang, H. Zhu, Z. Hu, Y. Yuan, Z. Chen, S. Senthil, H. Hajishirzi, Y. Tsvetkov, P. Dasigi, and T. XiaoRethinking the Evaluation of Harness Evolution for Agents.Note:arXiv preprint arXiv:2607.12227External Links:2607.12227,LinkCited by:Appendix A.
• Yanget al.(2024)J. Yang, C. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. PressSWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering.InAdvances in Neural Information Processing Systems,Vol.37,pp. 50528–50652.External Links:Link,DocumentCited by:§1,§3.1.
• Yanget al.(2026)L. Yang, Z. Xu, M. Xie, J. Gao, Z. Shok, Y. Wang, and Y. WuMAGE: Meta-Reinforcement Learning for Language Agents toward Strategic Exploration and Exploitation.Note:arXiv preprint arXiv:2603.03680External Links:2603.03680,LinkCited by:Appendix A.
• Yanget al.(2018)Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov, and C. D. ManningHotpotQA: a dataset for diverse, explainable multi-hop question answering.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,pp. 2369–2380.External Links:Document,LinkCited by:§1.
• Yuanet al.(2024)W. Yuan, R. Y. Pang, K. Cho, X. Li, S. Sukhbaatar, J. Xu, and J. E. WestonSelf-Rewarding Language Models.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol.235,pp. 57905–57923.External Links:LinkCited by:Appendix A.
• Yueet al.(2026)R. Yue, Y. Cui, Z. Sun, S. Pan, X. Xue, T. Li, T. Li, W. Zhu, Y. Chen, Y. Liu, B. Huang, Z. Cui, H. Zhang, and C. ZuoEcdysis: efficient and effective training of runtime harnesses for LLM agents.Note:arXiv preprint arXiv:2609.11677External Links:2609.11677,LinkCited by:Appendix A.
• Zelikmanet al.(2024)E. Zelikman, E. Lorch, L. Mackey, and A. T. KalaiSelf-Taught Optimizer (STOP): Recursively Self-Improving Code Generation.InFirst Conference on Language Modeling,External Links:LinkCited by:Appendix A.
• Zelikmanet al.(2022)E. Zelikman, Y. Wu, J. Mu, and N. GoodmanSTaR: Bootstrapping Reasoning With Reasoning.InAdvances in Neural Information Processing Systems,Vol.35,pp. 15476–15488.External Links:Link,DocumentCited by:Appendix A.
• Zenget al.(2026)G. Zeng, J. Wang, W. Ma, S. Yin, C. Wang, S. Liu, A. Kanazawa, W. Ni, X. Li, A. Zanette, and H. Feng[schema]: Frontier Models with Our Harness Achieve∼\sim99% on ARC-AGI-3 Public.Note:Impossible ResearchExternal Links:LinkCited by:Appendix A.
• Zhanget al.(2026a)G. Zhang, L. Lu, F. Xie, K. Zhu, J. Wang, Z. Xie, Z. Yu, Z. Liu, Z. Sun, Q. Li, Y. Liao, H. Chang, X. Hu, Q. Ren, W. Zhou, C. Hu, Y. Deng, and S. YanJIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolution.Note:arXiv preprint arXiv:2608.25593External Links:2608.25593,LinkCited by:Appendix A,§2.
• Zhanget al.(2025a)J. Zhang, S. Hu, C. Lu, R. Lange, and J. CluneDarwin Godel Machine: Open-Ended Evolution of Self-Improving Agents.Note:arXiv preprint arXiv:2505.22954External Links:2505.22954,LinkCited by:Appendix A.
• Zhanget al.(2025b)J. Zhang, J. Xiang, Z. Yu, F. Teng, X. Chen, J. Chen, M. Zhuge, X. Cheng, S. Hong, J. Wang, B. Zheng, B. Liu, Y. Luo, and C. WuAFlow: Automating Agentic Workflow Generation.InInternational Conference on Learning Representations,External Links:LinkCited by:Appendix A,§1,§2.
• Zhanget al.(2026b)L. Zhang, R. Zhou, D. Song, Z. Chen, Y. Tian, J. Yang, H. Ma, C. Li, G. Feng, X. Li, Y. Jin, and Y. XuHarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnesses.Note:arXiv preprint arXiv:2608.01918External Links:2608.01918,LinkCited by:Appendix A.
• Zhanget al.(2026c)Z. Zhang, Z. Wen, A. Zhang, A. Wang, J. Xie, D. Khashabi, and T. ShuAgentOdyssey: open-ended long-horizon text game generation for test-time continual learning agents.External Links:2606.24893,LinkCited by:Appendix A.
• Zhanget al.(2026d)Z. Zhang, A. Zhang, D. Khashabi, and T. ShuContinual learning mechanisms compose for long-horizon memorization.External Links:2609.06986,LinkCited by:Appendix A.
• Zhuet al.(2025)R. Zhu, B. Jiang, L. Mei, F. Yang, L. Wang, H. Gao, F. Bai, P. Zhao, Q. Lin, S. Rajmohan, and D. ZhangAdaptFlow: Adaptive Workflow Optimization via Meta-Learning.Note:arXiv preprint arXiv:2508.08053External Links:2508.08053,LinkCited by:Appendix A,§2.
• Zweigeret al.(2025)A. Zweiger, J. Pari, H. Guo, E. Akyürek, Y. Kim, and P. AgrawalSelf-Adapting Language Models.Note:arXiv preprint arXiv:2506.10943External Links:2506.10943,LinkCited by:Appendix A.

Appendix AExtended Related Work

Harness optimization and learned editors.

Harness optimization improves executable agent programs through search or learned editing policies. DSPy optimizes modular language-model pipelines, while AFlow searches over code-based workflows(Khattab et al., 2024;Zhang et al., 2025b). Meta-Harness and AutoHarness revise harnesses from execution feedback(Lee et al., 2026;Lou et al., 2026), and Ecdysis aggregates failures across instances to guide repairs(Yue et al., 2026). EvoHarness-RL learns to manage external belief, progress, and experience state during execution(Ning et al., 2026). WHALE(Kim et al., 2026)alternates model weight updates with optimization of the harness with a stronger harness proposer, improving agent performance beyond optimizing a single component alone.

Comparison with contemporary work.

Harness-R1, JIT-Agent, and Ornith are contemporary efforts to learn harness construction or revision. They share our interest in learning from execution outcomes and differ from our work in training signal, adaptation process, and scope of transfer evaluation (Table2).

Harness-R1.

Shao et al. (2026)train a dedicated editor with GPT-5.5 demonstrations followed by GRPO, keeping the target agent frozen. Their SFT, RL, validation, and test partitions contain disjoint instances from the same three benchmarks. They also evaluate patches on instances withheld from adaptation feedback and transfer the editor to unseen target models. Our transfer evaluation excludes entire reasoning families from both training stages and tests a HotpotQA-trained policy on MuSiQue and 2WikiMultihopQA. During Harness-R1 training, rewards come from rerunning the batch that supplied the failure feedback; our scoring questions are disjoint from the feedback questions. Their procedure generates patches without successive refinement of the same patch, whereas we study repeated revision and training on successive revision states. Supervised initialization followed by RL is shared with our Reasoning Gym setting. Our QA proposer starts RL directly from the base proposer, with no teacher-generated revision demonstrations (Section5.1).

JIT-Agent.

Zhang et al. (2026a)learn task-conditioned harness generation, repair, and evolution within a four-module protocol. Training combines teacher SFT, execution-based DPO, supervised repair, and Evo-GDPO, which rewards improvements over archive incumbents while accounting for task performance, latency, and cost. Its streaming setting also accumulates execution experience to improve subsequent harnesses. JIT-Agent and our work both study learned adaptation from feedback. JIT-Agent organizes adaptation around task instances and an expanding harness archive; our revision sequences update a current harness evaluated across multiple instances within a task family, with transfer measured on families or benchmarks excluded from training.

Ornith.

Ornith-1.0 jointly trains a policy to generate scaffolds and solve tasks using rewards from solution rollouts(Ornith, 2026a). Ornith-1.5 extends this loop to jointly optimize task generation, harness generation, and solution rollouts(Ornith, 2026b). These contemporary releases couple improvements in scaffolding with changes to the task-solving policy. Our training updates only the proposer and keeps the solver fixed, so harness improvements are evaluated with unchanged solver parameters. At test time, both models remain fixed while execution feedback guides code revision.

Table 2:Contemporary approaches to learned harness adaptation.Columns give each method’s training signal and adaptation process. The accompanying text compares their transfer evaluations.

Meta-learning.

Meta-learning uses experience across tasks to improve adaptation to new tasks(Hospedales et al., 2022). MAML learns an initialization for gradient-based adaptation(Finn et al., 2017), while RL2encodes a reinforcement learning algorithm in a recurrent policy(Duan et al., 2016). AdaptFlow learns a shared workflow initialization through language-guided updates(Zhu et al., 2025). The Last Harness proposes an outer loop over harness-evolution blueprints(Seong et al., 2026), and EvoX jointly evolves solutions and their search strategies(Liu et al., 2026). Meta-RL methods also learn to use interaction histories and reflections(Yang et al., 2026), convert rubric-based judgments into reusable guidance(Li et al., 2026), or train LLMs on diverse interaction trajectories to learn transferable exploration strategies, enabling adaptation to unseen tasks through environmental feedback in context without further parameter updates(Tajwar et al., 2025). Our proposer maps a harness and execution feedback to a code revision. Training uses immediate revision rewards as a greedy surrogate for final harness quality (Section4.3).

Self-improvement.

Self-improvement methods use feedback to update outputs, memory, code, or model parameters. Self-Refine revises model outputs(Madaan et al., 2023), and Reflexion stores verbal reflections for later attempts(Shinn et al., 2023). Feedback can also fail to sustain improvement(Jiang et al., 2025). STOP improves the program that proposes code changes(Zelikman et al., 2024), and the Darwin Gödel Machine evolves agent code through a candidate archive(Zhang et al., 2025a). TTHE adapts harnesses from unlabeled execution traces with fixed model parameters(Nie et al., 2026).

Other self-improvement methods update model parameters from self-generated training signals. STaR trains on generated rationales yielding correct answers(Zelikman et al., 2022), and Self-Rewarding Language Models construct training preferences from model-generated rewards(Yuan et al., 2024). SEAL learns to generate fine-tuning data and update directives(Zweiger et al., 2025), while SIA combines harness and model-weight updates(Hebbar et al., 2026). Harness learning trains a separate proposer from revision outcomes with a fixed solver. At test time, both models remain frozen, and graded development feedback guides harness edits.

Evaluation of harness evolution.

Harness evaluation accounts for solver capability, feedback, and search budgets.Wang et al. (2026b)examine benchmark overfitting under matched feedback and inference budgets, whileLin et al. (2026)distinguish generating useful updates from benefiting from them. HarnessCompass studies transfer through constrained edits and richer feedback(Zhang et al., 2026b), and Evo-Bench evaluates evolution across held-out task suites(Huang et al., 2026). Continual-learning evaluations test whether gains persist as tasks arrive(Wang et al., 2026a). Our experiments fix the solver within each benchmark, separate development feedback from held-out scoring, and report average proposal scores, best-of-eight scores, and successive-revision performance. Evaluations across benchmarks and on families excluded from both training phases test transfer of the learned adaptation procedure.

Continual learning agents.

Continual learning agents must acquire knowledge and skills from ongoing interaction while retaining useful experience. AgentOdyssey evaluates these abilities through procedurally generated, long-horizon text games, with diagnostics for exploration, world knowledge, and episodic memory(Zhang et al., 2026c). Methods address this challenge at both the parameter and program levels. For parametric memory,Zhang et al. (2026d)show that composing generative replay, self-distillation, and weight regularization with merged LoRA improves retention across sequential fine-tuning tasks. At the program level, Voyager accumulates reusable executable skills(Wang et al., 2023), while WorldCoder and Schema construct and revise executable world models from interaction feedback to support planning(Tang et al., 2024;Zeng et al., 2026). Harness learning complements these directions by learning how to revise the program that organizes model calls and tool use, with model parameters fixed at test time.

Appendix BReasoning Gym Training and Evaluation

B.1Evaluation protocol

We evaluate revision on training and unseen families, with canonical and format-varied questions (Table3). Each family has disjoint development and held-out sets. We use development instances for feedback and harness selection, and held-out instances, which remain hidden from the proposer, measure performance.

Table 3:Reasoning Gym evaluation settings.Instance counts, proposal counts, and revision rounds are per task family.Single-step evaluation samples independent revisions of the seed harness from a fixed report. Multistep evaluation revises the best harness retained so far using its development report. At roundtt, the inputxtx_{t}contains the task description, the retained harnessht−1⋆h^{\star}_{t-1}, and its report onQdevQ^{\mathrm{dev}}. Applying an edityt∼πθ(⋅∣xt)y_{t}\sim\pi_{\theta}(\cdot\mid x_{t})produces a candidateht′h^{\prime}_{t}, which we retain only if its development score improves,

ht⋆={ht′,if​J​(ht′,Qdev)>J⁡(ht−1⋆,Qdev),ht−1⋆,otherwise.h^{\star}_{t}=\begin{cases}h^{\prime}_{t},&\text{if }J(h^{\prime}_{t};Q^{\mathrm{dev}})>J(h^{\star}_{t-1};Q^{\mathrm{dev}}),\\ h^{\star}_{t-1},&\text{otherwise.}\end{cases}\addcontentsline{lla}{section}{\numberline q:promote}(7)Methods share the seed harness and initial report within each comparison. Each subsequent round uses fresh feedback from the retained harness, whose held-out score we measure after every round.AppendicesE.1andE.2give the revision template, execution report, and branch instruction.

We score every proposal, including zero-scoring harnesses, and exclude the seed from proposal counts. Edits are parsed only from the final answer, and a rejected edit receives error feedback and may be retried within the attempt budget. Failed edits and declared no-change proposals retain the parent score. For single-step revision we report mean and best-of-eight held-out scores, and for multistep revision the score of the development-selected harness (Tables9and13).Table10separates failed edits from zero-scoring harnesses, andAppendixC.1examines sensitivity to the treatment of failed edits.

Unseen families exclude all SFT and RL families.Table9reports every evaluated family, its training exposure, and exclusions from the main figures. The main figures exclude five unseen families that both the 4B proposers and the teacher find hard to address with an algorithmic harness: knight_swap, letter_jumble, modulo_grid, sokoban, and ab (marked†\dagger). Over all 26 unseen families, mean scores are 0.282 for Base, 0.415 for SFT, 0.528 for Single-step RL, and 0.501 for the teacher.

The seed harness calls the solver once and extracts its answer. The fixed tool-loop baseline supplies Python execution for up to five turns and is shared across families. The teacher uses the same revision and scoring protocol and the same feedback reports.

Single-step figures include the seed baseline. Multistep curves plot retained-harness scores (Eq.7), and the multistep columns ofTable17report the best proposal over rounds; the two scores differ by at most 0.002 in any family mean.Fig.4compares both protocols on the shared unseen families, andTables9and14give per-family results. Aggregate uncertainty is one standard error across families, computed after averaging repeated runs within each family, and per-family shading gives the range across runs.

Scope of revision comparisons.

Single-step and multistep evaluation differ in proposal budget and prompting. Independent proposals use a branch instruction and regenerate the seed report. Sequential revisions share a fixed initial report and then use feedback from retained harnesses. Our comparisons of successive and independent revision do not isolate the contribution of feedback, because the protocols also differ in parent harness and candidate selection.AppendixC.2details the Reasoning Gym family subsets, andAppendixD.4describes the corresponding QA comparison and proposal budgets.

Composition experiments compare standard prompts with prompts augmented by a helper-tool directive (AppendixE.3). Structural composition means that a harness integrates helpers into the solver’s tool loop, and functional composition additionally requires a helper to contribute to the answer.Table18reports the audit counts.

B.2SFT corpus and training

SFT uses first-round teacher revisions on the task families inTable4, keeping those that improve seed performance and pass execution checks.

Table 4:Task families used for SFT and RL on Reasoning Gym.The unseen-family evaluation excludes every family used in either phase.Each collection context gives the teacher the seed’s development score, outcome counts, and failed questions with trace excerpts, and the teacher independently revises a Python harness that exposessolve(model, question). We execute each applied edit on the same set in a guarded sandbox. Accepted examples preserve completions verbatim, including reasoning, edits, and change manifests.Table5gives the collection budgets and corpus size. Each collection run samples eight first-round teacher proposals and retains one to six of them. Some collection runs add a directive asking the teacher to give the solver a Python execution tool; SFT training prompts omit this directive. Under the structural audit taxonomy (AppendixC.4), 207 of the 432 examples (48%) are interpreter loops.

To enter the corpus, a candidate must satisfyJ⁡(h′,Qdev)≥max⁡{J⁡(h0,Qdev)+0.08,0.40}J(h^{\prime};Q^{\mathrm{dev}})\geq\max\{J(h_{0};Q^{\mathrm{dev}})+0.08,\,0.40\}and pass screens for answer echoing, duplicate code, and output-interface violations. Claude Opus 5 reviewers re-execute surviving candidates on development, held-out, and format-varied questions and inspect them for oracle access, memorization, and grader exploitation. No held-out question text enters the corpus.

Four-fold family-disjoint cross-validation over all 432 examples selects the duration of LoRA training. We then train on the full corpus for that duration and merge the adapters into the base model.Table5lists the optimization settings.

Limitations of supervised data.

Our Reasoning Gym SFT corpus contains only first-round revisions from a single teacher and is concentrated in interpreter loops and direct algorithmic solutions. Some collection runs also use explicit directives to encourage tool use. This supervision may limit the range of revision strategies learned during SFT, particularly for improving harnesses over successive rounds.AppendixFdiscusses stronger supervision and alternative initializations.

Table 5:Supervised data collection and training settings.

B.3Single-step RL

Single-step RL uses a fixed pool of contexts pairing a task family, parent harness, and execution report. Feedback and reward instances share a family and use disjoint seeds.Table6lists contexts built from the seed and stronger parent harnesses.

Table 6:Single-step RL context pool.Training contexts by family and parent harness.The task-score term inEq.6is mean grader credit on the scoring set, including partial credit. The validity reward is

vi=0.051[yiparses]+0.101[hi′≠⊥]+0.15ρrun(hi′),v_{i}=0.05\,\mathbf{1}[y_{i}\text{ parses}]+0.10\,\mathbf{1}[h^{\prime}_{i}\neq\bot]+0.15\,\rho_{\mathrm{run}}(h^{\prime}_{i}),\addcontentsline{lla}{section}{\numberline q:rg_{v}alidity_{r}eward}(8)whereρrun\rho_{\mathrm{run}}is the fraction of scoring trials that complete. With unit task-score weight, a harness that completes every trial but answers incorrectly earns0.300.30. Failed edits earn only accrued validity credit. We execute and score the parent for no-change responses.

We sample revisions asynchronously with vLLM, apply each edit, execute the harness with the frozen solver, and compute task and validity rewards. Each sampled batch receives one policy update with group-normalized advantages and a loss averaged over response tokens.Table7gives the sampling, optimization, and execution settings.

Table 7:Single-step RL settings.Solver execution settings are shared between training and evaluation.Single-step RL denotes the checkpoint selected at update 10; rows markedlateruse update 16.AppendixC.3reports additional training configurations with changes to the composition directive, training pool, and reward terms.

B.4Multistep RL

Multistep RL on Reasoning Gym trains in two offline phases, each with the immediate-reward update ofSection4.3, and refreshes the revision states between them (Table8). Each revision receives the full response budget.

Each state’s report comes from rescoring its harness on 25 canonical development questions and lists outcome counts and up to eight failing instances. The proposer receives the report, parent, task description, and revision instruction. All candidates share the context’s scoring questions.

Table 8:Offline training on successive revision states.The later-round-state experiment (AppendixC.3) trains on a fixed mixture of contexts from SFT revision runs. Multistep RL and the later-round-state run share the inference protocol inAppendixB.1. On HotpotQA, Multistep RL generates its training states online (Section4.3).

B.5Training curves

Fig.8shows reward and rollout outcomes for single-step RL, with one unsmoothed point per GRPO update.AppendixC.3reports the additional training configurations and their evaluation results.

Figure 8:Reasoning Gym RL training.(a) Mean reward with one standard error, task score, and the validity-reward baseline. The open circle marks the reported checkpoint. (b) Rollout outcomes. The large variance comes from the prompt distribution, since a batch may mix tasks from different families. Training settings are inAppendixB.3.

Appendix CAdditional Reasoning Gym Results and Analyses

Methods and proposal accounting followAppendixB. Single-step RL denotes checkpoint 10 unless specified otherwise.Table16defines additional training configurations, andAppendixC.4defines composition rewards.

Each caption states whether its scores are held-out scores or the development scores used for selection. Missing scores appear as--, and blank cells denote evaluations that were not run.

C.1Single-step results

Fig.9groups single-step results by training exposure, andTable9reports every evaluated family, including those excluded from the main comparison. The mean score measures average proposal quality, the best-of-eight score measures performance with candidate selection, and the fixed tool-loop harness serves as a common baseline. On the 27 families with teacher results in the 28-family comparison, mean scores are 0.600 for Single-step RL and 0.567 for the teacher.

Figure 9:Single-step revision by training exposure.Bars give mean held-out scores over eight proposals, and black marks give the teacher. In the OOD summary panel, circles mark best-of-eight scores, the dashed line marks the seed, and error bars show one standard error across families.Table9lists every family and marks those excluded from the main comparison.Table 9:Single-step revision by task family.Entries report mean / best-of-eight scores across all eight attempted proposals on 75 held-out instances per family, and Tool loop gives the score of the fixed tool-loop harness. Single-step RL uses the selected checkpoint at update 10,lateris update 16, and Teacher is the 35B model. Families marked†\daggerare excluded from the 28-family comparison (AppendixB.1). Teacher averages cover 32 families overall and 27 in the 28-family comparison.Table10counts failed edits and zero-scoring harnesses over all attempted proposals, including unusable edits. An applied edit can receive no task credit, so a zero score alone does not establish that the code is invalid.

Table 10:Proposal validity by evaluation set.Each method attempts eight proposals per family. A failed edit yields no scorable harness (AppendixB.1), and a zero-scoring harness has a recorded score of zero on 25 development instances. The combined rate is the number of failed edits plus zero-scoring harnesses, divided by attempted proposals. The canonical, format-varied, and unseen (OOD) sets contain 15, 12, and 5 families, respectively.#### Sensitivity to failed-edit scoring.

Table11compares retaining the parent score, assigning zero, and excluding failed edits on eleven format-varied families and nineteen unseen families, which differ from the main comparison sets. Single-step RL exceeds the teacher on unseen families under all three rules. Excluding zero-scoring harnesses conditions the metric on success and can reverse this comparison, so our main results include zeros. On this nineteen-family evaluation, 30 of the teacher’s 152 proposals score zero, compared with 19 for Single-step RL.

Table 11:Sensitivity to failed-edit scoring.Columns give held-out mean scores when a failed edit keeps the seed score (Seed), scores zero (Zero), or is excluded (Applied). No edit counts proposals that produce no revised harness. Each family has eight proposals.All methods receive the same feedback reports. Format-varied results for Base, SFT, and Single-step RL come from the main evaluation run; other rows come from separate evaluation runs. Scoring rules are compared within each row. Blank cells denote evaluations that were not run.

C.2Multistep results

Each question format covers 24 families under the revision and promotion protocol inAppendixB.1. Twelve original families are supplemented with twelve families excluded from the main proposers’ SFT and RL training. The additional families were selected partly by their earlier mean single-step scores (Table12).

Table 12:Selection of additional multistep evaluation families.The criteria use the earlier single-step results.The canonical and format-varied sets use different family lists (Table14).Fig.4uses the 15 unseen families shared with the single-step comparison. We average repeated runs within each family before computing aggregate scores and standard errors. Each additional family has one run per method.

All three trained proposers finish above Base on the expanded held-out sets, with similar canonical scores and the highest format-varied score from Single-step RL (Table13).Fig.10separates seen and unseen families across revision rounds.

Table 13:Held-out performance after five revisions.Mean over 24 families per question format. Gains over Base are paired within families and reported as mean±\pmone standard error.Figure 10:Held-out performance over revision rounds.Each point scores the harness that development-set selection retains after that round. Panels split families by question format and training exposure, with family counts in parentheses. A revision run keeps the seed until it promotes a proposal, and diamonds mark the seed score averaged over runs in each question format. Bands show one standard error across families, computed after averaging repeated runs within each family.Table14reports development scores after rounds 1 and 5. Oncaesar_cipher, the ordering of SFT and Single-step RL reverses between canonical and format-varied questions.Fig.11shows the family-level revision curves across the original and additional families in both formats.

Table 14:Multistep revision by task family: canonical questions.Columns r1 and r5 score the harness retained after rounds 1 and 5 on 25 development instances per family. All methods start from the same seed, so the Seed column is shared. Exposure labels followTable9. Families below the dividing line form the additional evaluation set (AppendixC.2) and have one run per method. Scores for the other families average repeated runs within each family. The format-varied results continue in the next panel.Table 15:Multistep revision by task family: format-varied questions (continued).Columns and evaluation protocol match the canonical panel. Families below the dividing line form the additional evaluation set.Figure 11:Development scores by family over revision rounds.(a) Original families, canonical questions. Curves track the harness retained after each revision; diamonds mark the seed. Parts (b)–(d) continue on the following pages.Figure 12:Development scores by family (continued).(b) Additional families, canonical questions. These families were selected as described inAppendixC.2and have one run per method.Figure 13:Development scores by family (continued).(c) Original families, format-varied questions. Shading shows the minimum-to-maximum range across repeated runs within each family.Figure 14:Development scores by family (continued).(d) Additional families, format-varied questions, with one run per method and family. Axes and method colors match parts (a)–(c).

C.3Training variants

Composition-prompt RL requests helper tools, while harder-task training changes the training pool and reward (Table16). Both start from SFT and useG=8G=8, a learning rate of10−610^{-6}, and no KL penalty, as in the main run. Because data, prompts, and reward terms can change together across configurations, each comparison measures the effect of the full configuration.

Table 16:Additional training configurations.Both variants use 20 contexts per update, compared with 40 in the main RL run.The harder-task pool containssokoban,modulo_grid,ab,leg_counting,rotten_oranges,codeio, andsimple_geometry. Prompt texts appear inAppendixE.3.

Composition-prompt RL produces fewer structurally composed harnesses at the later checkpoint, decreasing from 12 to 7 among 40 probe draws as the mean score increases slightly (Table18). The score increase suggests that task reward can favor interpreter loops even when the prompt requests helper tools.

The harder-task pool contains families with room to improve over the fixed tool-loop harness and low expected scores for constant answers. Its reward includes a structural-composition bonus,

rhard=0.05​1​[loaded]+0.10​ρrun+J⁡(h′,Q)+0.15​C​(h′),r_{\mathrm{hard}}=0.05\,\mathbf{1}[\text{loaded}]+0.10\,\rho_{\mathrm{run}}+J(h^{\prime};Q)+0.15\,C(h^{\prime}),\addcontentsline{lla}{section}{\numberline q:p3_{r}eward}(9)whereQQis the scoring set,ρrun\rho_{\mathrm{run}}is the fraction of trials that complete, andC⁡(h′)C(h^{\prime})indicates whether the harness passes the structural composition screen. No recorded rollout passes the screen. The run therefore measures the combined effect of the harder pool and revised validity terms, with no reward from the composition bonus.

Harder-task RL scores above both evaluation runs of Single-step RL on the format-varied families (0.600 versus 0.576 and 0.506;Table17), and composition-prompt RL exceeds the main run at its update-16 checkpoint (0.599). Harder-task RL finishes below Single-step RL after repeated revision on both development formats. Its nineteen-family evaluation includes training families and therefore measures a mixture of seen and unseen tasks.

Table 17:Training variants under single-step and multistep evaluation.Single-step columns average all eight proposals on 75 held-out instances per family. Round 5 columns report the best proposal over five rounds on 25 development instances per family, using the original twelve-family sets.Single-step format-varied results cover eleven families and use the same feedback reports as the main single-step comparison. SFT and Single-step RL values in this column come from a second evaluation run; the main run gives 0.415 and 0.576 (Table11). Replicate scores are separated by a slash. The nineteen-family set includes families used in harder-task training. Blank cells were not evaluated.

Later-round training states.

A separate run uses a fixed pool of 465 contexts from SFT revision sequences, comprising 240 first-round contexts, 155 later-round contexts, and 70 from revision runs that remain at the seed. Training starts from SFT, followsAppendixB.3, and runs for 22 updates; we evaluate checkpoint 16. The multistep run inAppendixB.4refreshes states between phases.

The checkpoint reaches a round-five development score of 0.615, compared with 0.660 for Single-step RL, a paired difference of−0.045±0.048-0.045\pm 0.048across twelve families. Gains over rounds 2 through 5 are similar, providing no evidence that the fixed later-round states improve repeated revision.

C.4Harness composition

We audit generated harnesses for two properties: structural composition, where helpers are integrated into the solver’s tool loop, and functional composition, where a helper contributes to the answer.Table18reports both rates by evaluation group.

Table 18:Harness composition by prompt and training setting.MethodnnStructuralFunctionalMost common classUnseen families, without a directiveSFT3900Interpreter loop (30)Single-step RL4000Interpreter loop (31)Unseen families, composition directiveSFT24114Tool loop + helper (12)Single-step RL32123Interpreter loop (17)Composition-prompt RL(update 10, without directive)3200Interpreter loop (29)Composition-prompt RL(update 10)3275Interpreter loop (24)Composition-prompt RL, checkpoint probes with the directiveComposition-prompt RL(update 10)40126Interpreter loop (27)Composition-prompt RL(update 16)4073Interpreter loop (33)Composition directive, comparison across training stagesBase3800Algorithm (21)SFT40185Tool loop + helper (19)Single-step RL40155Interpreter loop (22)Guarded directive, helper with solver fallbackSFT4000Algorithm (40)Single-step RL4000Algorithm (39)Composition-prompt RL(update 10)4000Algorithm (40)Composition-prompt RL(update 16)4000Algorithm (39) nncounts audited harnesses generated from development feedback. Structural and Functional count harnesses meeting the criteria inAppendixB.1. Parentheses give counts for the most common class. Groups are separate evaluations, so counts are comparable within groups. Where specified, evaluations use the helper-composition directive. The guarded directive requests a helper with a solver fallback.

On the eleven-family format-varied set, the composition directive increases the composition-prompt model’s score but lowers Single-step RL’s score (Table19). In five-family probes, the composition-reward checkpoints also score lower with the directive.

Table 19:Held-out scores with and without the composition directive.Scores average eight proposals per family over eleven format-varied families, each evaluated on 75 held-out instances. All methods receive the same feedback reports, and all 88 proposals in each condition have recorded scores.A guarded directive requesting a deterministic helper with a solver fallback produces direct algorithmic solutions in 158 of 160 draws. No draw meets the structural or functional composition criteria (Table18). Directive versions are specified inAppendixE.3.

Composition rewards.

We compare tool-use and combined-reward variants, both initialized from SFT and trained on the composition-prompt pool with the task-specific helper directive. Each checkpoint is probed on five families with and without the directive, with eight proposals per family. A separate twelve-family evaluation measures repeated revision.

The added reward combines structural composition with a proxy for successful tool use. LetF⁡(h′)F(h^{\prime})average each scoring trial’s task score multiplied by an indicator that a tool result reaches the solver, withF⁡(h′)=0F(h^{\prime})=0unless the structural screen passes. The reward is

rcomp​(h,y)=r⁡(h,y)+wf​F​(h′)+ws​C​(h′),r_{\mathrm{comp}}(h,y)=r(h,y)+w_{\mathrm{f}}F(h^{\prime})+w_{\mathrm{s}}C(h^{\prime}),\addcontentsline{lla}{section}{\numberline q:composition_{r}eward}(10)whereC⁡(h′)C(h^{\prime})is the structural indicator inEq.9. The tool-use reward uses(wf,ws)=(0.3,0)(w_{\mathrm{f}},w_{\mathrm{s}})=(0.3,0); the combined reward uses(1.0,0.15)(1.0,0.15). The audit assesses the helper’s contribution to the answer separately from the reward proxy.

Under the tool-use reward, composed tool loops decline from 25 of 40 draws at initialization to one at checkpoint 14. At checkpoint 12, the combined reward retains 16 of 40 composed loops, but its round-five development score remains below Single-step RL (Table20). These probes do not isolate the effects of the reward terms.

Table 20:Repeated revision after composition-reward training.Round-five development scores on twelve canonical families, using the selection protocol inAppendixB.1and 25 development instances per family.Helper outputs reaching the solver in successful trials remain less common, and composed harnesses score below interpreter loops sampled during the same updates.

The checkpoint-12 inspection identifies parsing, tool communication, and answer extraction failures around correct helpers, including breadth-first search formaze, backtracking forsudoku, and a primality sieve forcount_primes(Table21).

Table 21:Observed failures in composed harnesses.Examples from the checkpoint-12 inspection.

Appendix DMulti-Hop QA Training and Evaluation

D.1Datasets and retrieval

HotpotQA provides in-domain evaluation using 5.23 million Wikipedia abstracts from its full-wiki corpus. We sample disjoint sets of 2,000 training, 500 development, and 300 held-out test questions from its training split.

MuSiQue and 2WikiMultihopQA provide out-of-domain evaluation. We pool all supplied support and distractor paragraphs from their development splits (20 per question for MuSiQue-Ans and 10 for 2WikiMultihopQA) and deduplicate them by title and text. We partition each development split into disjoint development and held-out test sets, stratified by hop count for MuSiQue and question type for 2WikiMultihopQA.Table22lists the corpora and question counts. Development and test questions are disjoint from each other and from the 2,000-question training pool.

HotpotQA answers are compared by normalized exact match (lower-casing and removal of punctuation, articles, and extra whitespace). On MuSiQue and 2WikiMultihopQA, a prediction is also correct if it matches one of the gold answer’s listed aliases, following the official 2WikiMultihopQA script.

Table 22:Multi-hop QA evaluation data.Retrieval corpora and question pools per benchmark.

D.2RL training

On HotpotQA, RL starts from the base proposer without an SFT stage and generates its training contexts online. For each context, we run the parent harness with traces on six feedback questionsQfbQ^{\mathrm{fb}}. The report lists the solver and retrieval calls, the final answer, whether it matches the gold answer, and which gold supporting documents were retrieved or missed. Each context also fixes 64 scoring questionsQscoreQ^{\mathrm{score}}on which allG=8G=8candidates are scored. Both sets come from the 2,000-question training pool, and each context’s draw is seeded by its index. The proposer receives the parent harness, the report, and the revision instruction, and answers with a short diagnosis followed by one Python code block, without mutation hints.

The reward is exact match on the scoring questions plus a validity bonus,

ri={J⁡(hi′,Qscore)+0.1h′i≠⊥0otherwise,r_{i}=\begin{cases}J(h^{\prime}_{i};Q^{\mathrm{score}})+0.1&h^{\prime}_{i}\neq\bot\\ 0&\text{otherwise,}\end{cases}\addcontentsline{lla}{section}{\numberline q:qa_{r}eward}(11)whereh′i≠⊥h^{\prime}_{i}\neq\botrequires that the response contains a code block that parses, definesrun, has at most 200 lines, and completes the scoring run. Questions on which the harness raises an exception or exceeds its call budgets count as incorrect insideJJ, so there is no separate partial-completion term.

Single-step RL.

Every context uses the seed harness as parent with its own feedback and scoring draw. The pool holds 320 contexts, consumed once in 80 updates of four contexts each. We report the checkpoint after the final update as Single-step RL.

Multistep RL.

Parents come from 32 revision runs that the evolving policy advances, so the training states come from successive revisions. Each run starts at the seed harness, and contexts are ordered by run, so each run is visited once every eight updates. When a run is revisited, the candidate from its previous context with the highest exact match on the scoring questions becomes the new parent, following the best-of-eight rule used at inference. A run keeps its parent if none of the eight candidates executes. Ten visits per run give 320 contexts and 80 updates per phase. The two policies share every other setting (Table23), so the comparison isolates the source of the parent harness.

Table 23:RL settings on multi-hop QA.Single-step and Multistep RL share all settings unless noted. Training and evaluation use the same solver execution settings.

D.3Multistep evaluation

All three benchmarks use the same two-hop retrieve-and-summarize seed. Each proposer performs four independent revision runs ofT=10T=10rounds from this seed. In each round, each run draws six feedback questions from the development pool and executes its current harness on them with traces. The report lists traces, answer correctness, gold answers, and retrieved or missed supporting documents (AppendixE.2). Sixty-four scoring questions from the same pool, disjoint from all feedback questions of the round, are shared by the four runs. Development questions can recur across rounds, while feedback and scoring questions remain disjoint within each round. Each run samplesG=8G=8revisions and advances to the candidate with the highest scoring exact match, even when it scores below its parent. The Reasoning Gym protocol retains a candidate only when it improves the development score (Appendix3).

A revision whose code does not parse, does not definerun, exceeds 200 lines, or fails its scoring run scores zero, and a run keeps its harness only if all eight revisions fail. After each round, we score the four current harnesses on the full test set and report their mean. Dashed curves in the first three panels ofFig.5show the highest held-out score reached across all four runs up to each round. Random draws are seeded per round, so every proposer, including Base, sees the same feedback and scoring questions in the same round.

Sampling and execution.

The proposer samples with temperature 1.0 and thinking enabled, with a response budget of 10,240 tokens. Base is the untrained Qwen3-4B under the same settings. Harnesses execute with the solver settings ofTable23. A scoring run is limited to 600 seconds and a test run to 1,800 seconds. The solver has a budget of 12 calls, and every retrieval returns at most 20 passages.

Figure 15:Seed QA harness and three edits from adopted RL harnesses (abridged).Red outlines mark additions and changes. (a) The seed retrieves twice and answers from solver summaries. (b) The answering call reads both hops’ retrieved passages. (c) A third retrieval hop precedes answering. (d) The harness accepts a draft answer only if it appears in the retrieved passages. Otherwise the draft becomes a retrieval query, and the solver answers from the new passages.

D.4Single-step evaluation

Proposal generation and scoring.

For each dataset and RL checkpoint, we generate 80 independent proposals from the seed harness, using the round-one reports of the four revision runs and 20 samples per report. The pool size matches the 80 proposals that one revision run consumes in ten rounds. Each proposal receives the seed code and its execution feedback, produces one revised harness, and is scored by exact match on the held-out test set listed inTable22. Edits that fail to parse or compile receive zero. Identical programs share one execution, and all 80 proposals contribute to the reported statistics.

Table24reports the unselected mean, the fraction strictly exceeding the seed, and the failed-edit rate. Oracle@8 is the expected maximum held-out score among eight proposals sampled uniformly without replacement from the 80. Best-of-80 is the maximum over the entire pool. Both oracle statistics select candidates by held-out score, so they describe the quality of the generated proposals and do not correspond to a deployable selection rule.

Table 24:Independent single-step QA revision.Each row summarizes 80 proposals. The>>\!seed column gives the fraction of proposals scoring strictly above the seed, and Failed gives the failed-edit rate. Oracle@8 and best-of-80 select on held-out scores.

Comparison with successive revision.

Fig.5compares these scores with the mean final score of four ten-round revision runs, whose selection uses only the scoring questions. Triangles show the highest held-out score reached across all ten rounds and four runs. Each run samples 80 candidates, matching the independent pool’s proposal count, while the four-run mean and best-run score use 320 proposals in total. The best-run score is therefore not a matched-budget comparison with the best of 80 independent revisions. The two evaluations also distribute feedback differently. Independent proposals share four seed reports, and successive revision obtains feedback on intermediate harnesses. Seed performance is measured separately for the independent and successive evaluations; horizontal dashed lines in the bar panels show the independent-evaluation baseline.

Appendix EImplementation Details

E.1Structured harness revision

The following schematic excerpt describes the revision interface. It is not the complete production prompt. The proposer receives the task family and configuration, the current harness source and development score, and an execution report. It is asked to diagnose observed failures, make a targeted change without hard-coding answers, and preserve the harness entry point.

Taskfamilyandconfiguration:<TASK_DESCRIPTION>

Currentharnessandscore:<PARENT_HARNESS_AND_DEV_SCORE>

Executionfeedback:<REPORT>

ProposeatargetedrevisionasSEARCH/REPLACEeditsagainstthecurrentsource.

Explainthechangeandidentifythefailuresitisexpectedtofixandanylikelyregressions.

Edits must match the current source and parse as a revised harness. The response includes a change manifest describing the edits and predicted effects; an unchanged harness must be declared as such. Invalid responses receive parsing or validation feedback and may be retried within the attempt budget inTable3.

E.2Execution feedback

For Reasoning Gym, the report provides development-set scores and instance outcomes, failed-example trace excerpts, changes since earlier revisions, and the retained harness’s revision history. Independent proposals share the same parent and report. Their branch instruction directs different proposals toward distinct failure mechanisms, while successive revision obtains new feedback from the retained harness (AppendixB.1). This distinction matters when comparing proposal budgets and adaptation over rounds.

Successive-state training supplies up toKKpassing andKKfailing examples, each with the question, harness answer, expected answer, and outcome (AppendixB.4). Multi-hop QA supplies six development questions per revision run, with execution traces, answer correctness, gold answers, and retrieved or missed supporting documents (AppendixD).

E.3Composition directives

Composition experiments compare the standard revision request with one that additionally asks for a reusable computational helper integrated into the solver’s tool loop. A more restrictive variant requests task-specific helpers instead of a generic code interpreter, along with checks of the resulting answer. These directives describe the desired harness structure; the structural and functional audits inAppendixC.4measure what the proposals actually achieve. The study’s different training configurations and reward terms are specified inTable16andEq.10.

Appendix FLimitations and Future Work

Teacher capability and proposer initialization.

Our Reasoning Gym supervision is limited to the harness designs produced by a single 35B teacher. Stronger teachers could provide more diverse harness designs and multistep revision demonstrations, potentially better preparing the proposer for subsequent multistep RL. Another direction is to start RL from a model with prior training in harness design and revision. Our QA experiments already use direct RL without teacher demonstrations. Whether an initialization with more knowledge of harness design improves exploration and transfer remains an open question.

Agentic training of the proposer.

Current training uses prescribed revision prompts and externally generated execution reports. The proposer does not choose its own code-inspection or execution steps before submitting a revision. Future work could train the proposer inside a tool-using harness, allowing it to inspect code, run targeted tests, compare candidate designs, and revise them using the resulting observations. This would extend training from producing edits to choosing how to investigate and improve a harness, and could help explore designs that are difficult to elicit through prompt directives alone.

Joint training and longer-horizon tasks.

We keep the solver fixed during training, so our experiments do not examine whether learning to generate harnesses and learning to use them can reinforce each other. Concurrent work such as Ornith jointly trains scaffold generation and task solving(Ornith, 2026a;Ornith, 2026b). An extension of harness learning could jointly optimize the proposer and solver while retaining the goal of generalizable adaptation. A key question is whether this training yields revision skills that transfer to unseen task families and remain useful as the solver changes, particularly on tasks requiring longer sequences of tool use and intermediate decisions.