Scaling Automatic Research Agents via World Models

# Scaling Automatic Research Agents via World Models
Source: [https://arxiv.org/html/2608.12564](https://arxiv.org/html/2608.12564)
Xiyuan Yang1†,Sheikh Sarwar2,Jingru Cheng2,Zhan Shi2,Duanshun Li2,Huiyuan Chen2,Haiyang Zhang2,Chenlei Guo2,Jingrui He1∗,Zhenyu Liao2∗1University of Illinois Urbana\-Champaign2Amazon∗Corresponding authors

###### Abstract

Abstract:Automating empirical research is a long\-standing direction of AI\. Recent automatic research \(AutoResearch\) agents bring this goal within reach, as modern LLMs show the capability to independently implement solutions and learn from the execution outcomes\. Behind these gains, post\-training \(especially RL\) plays a central role\. In this paper, we identify a fundamental tension when scaling RL for these agents: the two components of every AutoResearch trajectory \(agent generation and environment execution\) scale in very different manners, since all generation shares compute through batching, while each execution occupies its exclusive sandbox and real machine time\. As a result, the environment execution dominates the training cost and becomes the bottleneck as trajectories grow\. To resolve this tension, we propose*World Model RL*\(WMRL\), which replaces environment execution with a world model to remove this bottleneck\. Additionally, the world model can be imperfect, as its rewards are corrupted by bias and noise\. Therefore, we further equip WMRL with two mitigations,*Online Debiasing*and*Inverse\-Variance Denoising*, which offset the bias and suppress the noise respectively\. Theoretically, we prove that both mitigations of WMRL strictly improve the convergence guarantee\. Empirically, WMRL accelerates training by33–4×4\\timeson various tasks at different agent scales, while exceeding the performance of standard RL baselines\. Moreover, our post\-trained 4B and 9B agents outperform much larger open\-weight agents of 48B and 120B on held\-out benchmarks\. Beyond AutoResearch, WMRL also transfers to post\-training embodied VLA policies, which demonstrates the generalizability of our method\.

††footnotetext:†Work done during an internship at Amazon\.## 1Introduction

![Refer to caption](https://arxiv.org/html/2608.12564v1/figures/fig1.png)Figure 1:AutoResearch trajectories scale asymmetrically, making environment execution the bottleneck, and our WMRL removes it\.\(a\)Per research question, the agent proposes a group of solutionsSiS\_\{i\}, graded by execution into rewardsrir\_\{i\}, and RL demands a massive number of such trajectories\.\(b\)Per trajectory, generation amortizes compute via batching, but execution needs one isolated sandbox per solution\.\(c\)Execution compute thus hits capacity first in traditional RL, whereas WMRL scales without this limit\.An Automatic Research \(AutoResearch\) agent is a language model that independently conducts empirical research\[[33](https://arxiv.org/html/2608.12564#bib.bib10),[15](https://arxiv.org/html/2608.12564#bib.bib43),[46](https://arxiv.org/html/2608.12564#bib.bib51)\]\. Given a research question, it formulates an idea, implements the experiment, analyzes the outcome, and iterates\[[60](https://arxiv.org/html/2608.12564#bib.bib45),[59](https://arxiv.org/html/2608.12564#bib.bib63),[50](https://arxiv.org/html/2608.12564#bib.bib46)\]\. Such agents have already proven capable across various domains\. In the natural sciences, for example, they design chemical syntheses and propose reaction conditions that are validated by wet\-lab experiments later\[[6](https://arxiv.org/html/2608.12564#bib.bib11),[7](https://arxiv.org/html/2608.12564#bib.bib42),[52](https://arxiv.org/html/2608.12564#bib.bib50)\]; in machine learning and data science, they explore real datasets and build training pipelines that outperform human experts\[[22](https://arxiv.org/html/2608.12564#bib.bib9),[18](https://arxiv.org/html/2608.12564#bib.bib44),[21](https://arxiv.org/html/2608.12564#bib.bib53)\]\. In an AutoResearch task, the agent iteratively generates and executes solutions\. These interactions form trajectories, and the execution outcomes provide rewards\. With both trajectories and rewards in place, AutoResearch is a natural fit for reinforcement learning \(RL\)\[[47](https://arxiv.org/html/2608.12564#bib.bib55),[48](https://arxiv.org/html/2608.12564#bib.bib5),[28](https://arxiv.org/html/2608.12564#bib.bib56),[17](https://arxiv.org/html/2608.12564#bib.bib41)\], a promising direction to further improve these capabilities\.

Like most of RL’s successes, training strong AutoResearch agents demands scale, i\.e\., massive online trajectories collected during training\[[17](https://arxiv.org/html/2608.12564#bib.bib41)\]\. However, we identify a fundamental tension underneath this demand: the two components of a trajectory \(i\.e\., agent generation and environment execution\) scale in different manners as the trajectory volume grows\. On the generation side, rollouts are served by modern inference backends \(e\.g\., vLLM\[[27](https://arxiv.org/html/2608.12564#bib.bib39)\]and SGLang\[[65](https://arxiv.org/html/2608.12564#bib.bib40)\]\), where batching techniques allow concurrent trajectories to share compute, making the cost of additional trajectories negligible\. In contrast, execution cost cannot be amortized in this way\. Every candidate solution must run in an isolated sandbox that loads the data and trains models on real GPUs\[[42](https://arxiv.org/html/2608.12564#bib.bib3),[51](https://arxiv.org/html/2608.12564#bib.bib38)\], so each additional trajectory incurs the full cost, and the total cost grows linearly with the number of trajectories\. This asymmetry makes environment execution the dominant bottleneck when scaling RL for AutoResearch agents\. Such a bottleneck motivates two research questions:

\(i\)*Can we replace the expensive environment \(bottleneck\) with a fast and scalable signal?*

\(ii\)*There is no free lunch, so what does this signal cost? \(And how do we pay the bill?\)*

The answer to question\(i\)is to introduce a world model\[[19](https://arxiv.org/html/2608.12564#bib.bib15),[20](https://arxiv.org/html/2608.12564#bib.bib16)\], which takes the environment information and the agent solution as input, and produces a simulation of real execution result as output\. Such a simulation \(though potentially flawed\) takes only a few forward passes without real execution, so it batches and amortizes across trajectories just like generation\. In RL training, we let the agent interact with the world model instead of the real environment, and receive rewards from the predicted outcomes\. As a result, the expensive execution now scales as gracefully as generation and no longer bounds the scale of RL\.

We answer question\(ii\)by modeling the world model’s imperfection as a bias and a noise on the reward, and tracing both through the policy gradient into the convergence bound\. Specifically, we consider the world model’s output as an erroneous signal that deviates from real execution by a bias termbbwith\|b\|≤B\|b\|\\leq Band a zero\-mean noiseξ\\xiwith standard deviationσ\\sigma\. As shown in Theorem[3](https://arxiv.org/html/2608.12564#Thmtheorem3), the bias and the noise hinder the final convergence through two extra error terms ofO⁡\(B2\)O\(B^\{2\}\)andO⁡\(σ2\)O\(\\sigma^\{2\}\)respectively\. To reduce both terms, we introduce an*Online Debiasing*mechanism that recasts the world model outputs to offset the bias, and an*Inverse\-Variance Denoising*mechanism to minimize the variance; we further ground both mechanisms in Theorem[4](https://arxiv.org/html/2608.12564#Thmtheorem4)with a strictly improved convergence guarantee\.

Our contributions are summarized as follows:

- •We introduce the world model into the RL training of AutoResearch agents, which replaces the expensive environment execution with a few forward passes\. It removes the critical scaling bottleneck and accelerates overall training by33–4×4\\times\(Section[3\.2](https://arxiv.org/html/2608.12564#S3.SS2)\)\.
- •We design two correction mechanisms, Online Debiasing and Inverse\-Variance Denoising, to counteract the bias and the noise of the world model\. With both mechanisms, the accelerated training matches or even exceeds the performance of training with the real environment \(Section[3\.3](https://arxiv.org/html/2608.12564#S3.SS3)\)\.
- •We theoretically ground the entire framework\. An imperfect world model introduces two error terms into the convergence bound \(Section[4\.2](https://arxiv.org/html/2608.12564#S4.SS2)\), and our two mechanisms provably reduce both, yielding a strictly improved convergence guarantee \(Section[4\.3](https://arxiv.org/html/2608.12564#S4.SS3)\)\.
- •Through experiments on various AutoResearch tasks across two agent scales, we validate both the efficiency and the performance gains \(Section[5\.2](https://arxiv.org/html/2608.12564#S5.SS2)\); we further extend our method to VLA post\-training tasks to demonstrate its generalizability \(Section[5\.3](https://arxiv.org/html/2608.12564#S5.SS3)\)\.

## 2Related Work

#### AutoResearch agents and their post\-training\.

Language model agents now carry out substantial parts of empirical research, from autonomous chemical experimentation\[[6](https://arxiv.org/html/2608.12564#bib.bib11)\]and open\-ended scientific discovery\[[33](https://arxiv.org/html/2608.12564#bib.bib10)\]to machine learning engineering, where agents explore the space of solution code\[[22](https://arxiv.org/html/2608.12564#bib.bib9),[36](https://arxiv.org/html/2608.12564#bib.bib32),[56](https://arxiv.org/html/2608.12564#bib.bib52)\], curate data as agentic data scientists\[[26](https://arxiv.org/html/2608.12564#bib.bib27),[5](https://arxiv.org/html/2608.12564#bib.bib35)\], and pursue recursive self\-improvement\[[58](https://arxiv.org/html/2608.12564#bib.bib28),[45](https://arxiv.org/html/2608.12564#bib.bib31)\]\. A line of benchmarks measures this ability on real Kaggle\-style competitions, including MLE\-Bench\[[10](https://arxiv.org/html/2608.12564#bib.bib1)\], its interactive superset MLE\-Dojo\[[42](https://arxiv.org/html/2608.12564#bib.bib3)\], DSBench\[[24](https://arxiv.org/html/2608.12564#bib.bib4)\], and MLGym\[[35](https://arxiv.org/html/2608.12564#bib.bib7)\], and software engineering agents follow the same execution\-driven recipe on real repositories\[[23](https://arxiv.org/html/2608.12564#bib.bib47),[57](https://arxiv.org/html/2608.12564#bib.bib48),[54](https://arxiv.org/html/2608.12564#bib.bib54)\], with training environments built at scale\[[40](https://arxiv.org/html/2608.12564#bib.bib49)\]\. Beyond prompting, RL post\-training further improves such agents, typically with group\-relative objectives\[[48](https://arxiv.org/html/2608.12564#bib.bib5)\]served by large\-scale RL systems\[[49](https://arxiv.org/html/2608.12564#bib.bib6)\], and synthetic tasks can scale the training corpus\[[55](https://arxiv.org/html/2608.12564#bib.bib37)\]\. In these pipelines the reward comes from executing the agent’s solution, so training inherits the execution bottleneck of Section[1](https://arxiv.org/html/2608.12564#S1), the problem we address\. Such advances refine how rewards are consumed\[[61](https://arxiv.org/html/2608.12564#bib.bib57),[43](https://arxiv.org/html/2608.12564#bib.bib29)\], whereas we change where they come from, so the two are orthogonal and compose\.

#### Learned reward signals and their errors\.

Replacing an expensive ground truth with a learned signal is a recurring pattern, from reward models in RLHF\[[39](https://arxiv.org/html/2608.12564#bib.bib12)\]to LLM judges of model outputs\[[64](https://arxiv.org/html/2608.12564#bib.bib14)\], and world models that simulate an environment for control have long powered model\-based RL\[[19](https://arxiv.org/html/2608.12564#bib.bib15),[8](https://arxiv.org/html/2608.12564#bib.bib59),[20](https://arxiv.org/html/2608.12564#bib.bib16)\]\. Such world models are now scaled to foundation models of the physical world\[[37](https://arxiv.org/html/2608.12564#bib.bib33)\]or written as executable code by LLMs\[[13](https://arxiv.org/html/2608.12564#bib.bib25),[53](https://arxiv.org/html/2608.12564#bib.bib26)\], and learned surrogates of the execution environment have recently been explored for software agents\[[51](https://arxiv.org/html/2608.12564#bib.bib38)\]\. The errors of such proxies are equally well documented, as optimizing against an imperfect reward model degrades true performance once the proxy is over\-trusted\[[14](https://arxiv.org/html/2608.12564#bib.bib13),[12](https://arxiv.org/html/2608.12564#bib.bib58)\], and the theory of SGD with biased gradients shows that a systematic gradient error puts a floor on convergence that no amount of training removes\[[1](https://arxiv.org/html/2608.12564#bib.bib17)\]\. Prior remedies mostly recalibrate the proxy offline\[[62](https://arxiv.org/html/2608.12564#bib.bib60),[16](https://arxiv.org/html/2608.12564#bib.bib18)\]or constrain the policy from exploiting it\. In off\-policy RL, a related line corrects the distribution mismatch between the replay buffer and the target policy with learned correction ratios\[[34](https://arxiv.org/html/2608.12564#bib.bib62),[29](https://arxiv.org/html/2608.12564#bib.bib24)\], a finer object than the reward statistics we treat\. We instead keep a small stream of ground truth inside the training loop, correct the bias and the variance of the proxy online, and quantify both corrections directly in the convergence bound\.

## 3Method

In this section, we introduce our WMRL as follows\. First, we formalize the standard RL training procedure and define the notation \(Section[3\.1](https://arxiv.org/html/2608.12564#S3.SS1)\)\. Next, we answer the question\(i\)by incorporating the world model into the RL pipeline as a fast and scalable signal \(Section[3\.2](https://arxiv.org/html/2608.12564#S3.SS2)\)\. We then answer the question\(ii\)by revealing the additional error terms in the RL convergence, and then providing our mitigations \(Section[3\.3](https://arxiv.org/html/2608.12564#S3.SS3)\), which offset these error terms introduced by the world model\.

### 3\.1Preliminaries

Typically, an AutoResearch task provides the agent with the research question, data format and the output requirements\. With this context, the agent policyπθ\\pi\_\{\\theta\}interacts with an environment \(e\.g\., an isolated Docker container\)\[[42](https://arxiv.org/html/2608.12564#bib.bib3),[24](https://arxiv.org/html/2608.12564#bib.bib4)\]over multiple turns to iteratively refine its solution until reaching a predefined criterion\. Such interactions form a trajectoryτ\\tau, and the solution receives a scorer⁡\(τ\)∈\[0,1\]r\(\\tau\)\\in\[0,1\]that measures its quality\.

To enhance the agent policy, a wide range of works leverage RL to maximize the expected scoreJ⁡\(θ\):=𝔼τ∼πθ​\[r⁡\(τ\)\]J\(\\theta\):=\\mathbb\{E\}\_\{\\tau\\sim\\pi\_\{\\theta\}\}\[r\(\\tau\)\]with group\-relative policy optimization \(GRPO\)\[[48](https://arxiv.org/html/2608.12564#bib.bib5)\]111We adopt GRPO as the most common objective for agentic RL at scale\[[49](https://arxiv.org/html/2608.12564#bib.bib6),[17](https://arxiv.org/html/2608.12564#bib.bib41)\], while the specific RL objective is orthogonal to our contribution, which acts mostly on the reward side rather than on the update rule\.as follows: For each task, GRPO samples a group ofn≥2n\\geq 2independent trajectoriesτ1,…,τn\\tau\_\{1\},\\dots,\\tau\_\{n\}in parallel, and then gives the grader⁡\(τi\)r\(\\tau\_\{i\}\)of each trajectory by executing the final solution in the environment\. The scores are then normalized within the group into advantagesAi:=r⁡\(τi\)−1n​∑jr⁡\(τj\)A\_\{i\}:=r\(\\tau\_\{i\}\)\-\\frac\{1\}\{n\}\\sum\_\{j\}r\(\\tau\_\{j\}\), which yield the gradient estimate as:

g^:=1n​∑i=1nAi​si,si:=∇θ​log​πθ​\(τi\),\\hat\{g\}\\;:=\\;\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}A\_\{i\}\\,s\_\{i\},\\qquad s\_\{i\}:=\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(\\tau\_\{i\}\),\(1\)wheresis\_\{i\}is the raw policy gradient, and the parameter is updated byθt\+1=θt\+γ​g^t\\theta\_\{t\+1\}=\\theta\_\{t\}\+\\gamma\\,\\hat\{g\}\_\{t\}overTTsteps with step sizeγ\\gamma\. Throughout training, every step costs online rewards, and each of the rewards requires executing a solution in a new environment with exclusive GPU assignments\.

\\begin\{overpic\}\[figures/fig2\.png\] \\put\(62\.7,9\.7\)\{\\pic@makebox@\{\\makebox\}\{\}\(0\.0,0\.0\)\{\\scriptsize\\hyperref@@ii\[eq:isotonic\]\{Eq\.\\penalty\\ \(\\ref\*\{eq:isotonic\}\)\}\}\} \\put\(87\.4,6\.8\)\{\\pic@makebox@\{\\makebox\}\{\}\(0\.0,0\.0\)\{\\scriptsize\\hyperref@@ii\[eq:ivw\]\{Eq\.\\penalty\\ \(\\ref\*\{eq:ivw\}\)\}\}\} \\end\{overpic\}Figure 2:WMRL corrects the world model rewards in two steps\.Each row is one of themmgroups in a batch\. Every group is graded by the world model into biased, noisy scoresr^\\hat\{r\}, and anchor groups are also graded by real execution\. The monotone mapf^\\hat\{f\}is fit on the score pairs \(*Online Debiasing*\) to remove the bias, andgEg\_\{E\}andgW​Mg\_\{WM\}are fused by*Inverse\-Variance Denoising*to lower the update variance\.
### 3\.2World Model as an Environment

To provide fast and scalable environment signals \(asked in question\(i\)\), we introduce a world model to replace the expensive execution, which constitutes the backbone of WMRL\. In our setting, we adopt as the world model a general language model that simulates the environment \(with potential errors\), instantiated with the same backbone as the agent and queried for the execution outcome\. It takes the same task context and an agentic solution as input, and predicts the execution outcome as output, from which an estimated scorer^​\(τi\)\\hat\{r\}\(\\tau\_\{i\}\)is read off in place of the truer⁡\(τi\)r\(\\tau\_\{i\}\)\. Since this interface is identical to that of the real environment, the pipeline in Section[3\.1](https://arxiv.org/html/2608.12564#S3.SS1)runs unchanged with the estimated scores as:

A^i:=r^​\(τi\)−1n​∑jr^​\(τj\),\\hat\{A\}\_\{i\}\\;:=\\;\\hat\{r\}\(\\tau\_\{i\}\)\-\\frac\{1\}\{n\}\\sum\_\{j\}\\hat\{r\}\(\\tau\_\{j\}\),\(2\)and the gradient estimate of Equation \([1](https://arxiv.org/html/2608.12564#S3.E1)\) is computed withA^i\\hat\{A\}\_\{i\}in place ofAiA\_\{i\}\. In effect, the parameters now ascend the surrogate objective𝔼τ∼πθ​\[r^​\(τ\)\]\\mathbb\{E\}\_\{\\tau\\sim\\pi\_\{\\theta\}\}\[\\hat\{r\}\(\\tau\)\]rather thanJ⁡\(θ\)J\(\\theta\)\. As a result, the execution part of a trajectory now scales as gracefully as the generation side, which directly removes the execution bottleneck\. However, we note that this replacement is not free, as an imperfect predicted outcome can deviate from the real execution outcome\. We characterize and mitigate such deviation in Section[3\.3](https://arxiv.org/html/2608.12564#S3.SS3)next\.

### 3\.3Anchor Signal for Error Correction

We now answer question\(ii\)\. For any world model, the deviation of its predicted score can be decomposed into a systematic part and a random part as follows:

r^​\(τ\)=r⁡\(τ\)\+b⁡\(τ\)\+ξ⁡\(τ\),\\hat\{r\}\(\\tau\)\\;=\\;r\(\\tau\)\+b\(\\tau\)\+\\xi\(\\tau\),\(3\)where the biasb⁡\(τ\):=𝔼⁡\[r^​\(τ\)∣τ\]−r⁡\(τ\)b\(\\tau\):=\\mathbb\{E\}\[\\hat\{r\}\(\\tau\)\\mid\\tau\]\-r\(\\tau\)collects the systematic part that survives averaging, and the noiseξ⁡\(τ\)\\xi\(\\tau\)is the zero\-mean remainder\. Since the scores are naturally bounded, we writeB:=supτ\|b⁡\(τ\)\|B:=\\sup\_\{\\tau\}\|b\(\\tau\)\|for the maximal magnitude of the bias andσ:=supτstd⁡\(ξ⁡\(τ\)\)\\sigma:=\\sup\_\{\\tau\}\\mathrm\{std\}\(\\xi\(\\tau\)\)for the maximal standard deviation of the noise\. The remark below summarizes how the two parts affect the convergence\.

Our goal is to remove or reduce the two additional errors \(theO⁡\(B2\)O\(B^\{2\}\)andO⁡\(σ2\)O\(\\sigma^\{2\}\)term\) in Remark[1](https://arxiv.org/html/2608.12564#Thmtheorem1)\. Removing them starts with estimating them, and by Equation \([3](https://arxiv.org/html/2608.12564#S3.E3)\) both are defined against the true scorer⁡\(τ\)r\(\\tau\), which the world model itself does not provide\. Since the truth can only come from environment execution, we take a small step back and reintroduce a marginal amount of execution for this estimation \(which we prove to be enough in Section[4](https://arxiv.org/html/2608.12564#S4)\)\. WMRL therefore keeps a thin stream of ground truth during training, which we call the*anchor signal*\. A small fraction of the groups \(*anchor groups*\) is graded by both the world model and real execution,222In practice, anchor groups make up about10%10\\%of all groups\.and the resulting score pairs𝒫=\{\(r^j,rj\)\}\\mathcal\{P\}=\\\{\(\\hat\{r\}\_\{j\},r\_\{j\}\)\\\}feed the two mechanisms below \(Figure[2](https://arxiv.org/html/2608.12564#S3.F2)\)\.

We first correct thebiasby*Online Debiasing*, a monotone distribution recalibration\. The training score pairs reveal how the world model score drifts from the truth, so we fit a mapping function over all of them,

f^=arg​minf∑\(r^j,rj\)∈𝒫\(f\(r^j\)−rj\)2,\\hat\{f\}\\;=\\;\\operatorname\*\{arg\\,min\}\_\{f\}\\;\\sum\_\{\(\\hat\{r\}\_\{j\},\\,r\_\{j\}\)\\in\\mathcal\{P\}\}\\big\(f\(\\hat\{r\}\_\{j\}\)\-r\_\{j\}\\big\)^\{2\},\(4\)whereffranges over monotone functions solved by isotonic regression\[[62](https://arxiv.org/html/2608.12564#bib.bib60)\]\. With this mapping, all world model scores are then recast intof^​\(r^​\(τi\)\)\\hat\{f\}\(\\hat\{r\}\(\\tau\_\{i\}\)\)before the advantages are formed, andf^\\hat\{f\}is refit as new pairs arrive on each step to track the drift over training\.

For thenoisepart, since it cannot be estimated pointwise, we turn to suppress it by*Inverse\-Variance Denoising*, which fuses the two reward streams to reach a lower variance\. In each step themmgroup indices split into the anchor set𝒢E\\mathcal\{G\}\_\{E\}and the world model set𝒢W​M\\mathcal\{G\}\_\{WM\}, and every groupkkgives an estimateg^k\\hat\{g\}\_\{k\}of the same policy gradient, with varianceVEV\_\{E\}orVW​MV\_\{WM\}by how it was graded\. The question is therefore how to combine them, as one stream is scarce but free of world model noise and the other abundant but noisy\. Following Lemma[12](https://arxiv.org/html/2608.12564#Thmtheorem12), the minimal\-variance combination weights each stream by the inverse of its variance and attains a variance strictly below either stream alone, so we combine them as

g^=VE−1​gE\+VW​M−1​gW​MVE−1​\|𝒢E\|\+VW​M−1​\|𝒢W​M\|,gE:=∑k∈𝒢Eg^k,gW​M:=∑k∈𝒢W​Mg^k,\\hat\{g\}\\;=\\;\\frac\{\\ V\_\{E\}^\{\-1\}\\,g\_\{E\}\\;\+\\;V\_\{WM\}^\{\-1\}\\,g\_\{WM\}\\ \}\{\\ V\_\{E\}^\{\-1\}\|\\mathcal\{G\}\_\{E\}\|\\;\+\\;V\_\{WM\}^\{\-1\}\|\\mathcal\{G\}\_\{WM\}\|\\ \},\\qquad g\_\{E\}:=\\\!\\\!\\sum\_\{k\\in\\mathcal\{G\}\_\{E\}\}\\\!\\\!\\hat\{g\}\_\{k\},\\quad g\_\{WM\}:=\\\!\\\!\\sum\_\{k\\in\\mathcal\{G\}\_\{WM\}\}\\\!\\\!\\hat\{g\}\_\{k\},\(5\)where the stream gradientsgEg\_\{E\}andgW​Mg\_\{WM\}sum the group estimatesg^k=1n​∑i=1nAk,i​sk,i\\hat\{g\}\_\{k\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}A\_\{k,i\}\\,s\_\{k,i\}, each applying Equation \([1](https://arxiv.org/html/2608.12564#S3.E1)\) to groupkkwith the true scores on𝒢E\\mathcal\{G\}\_\{E\}and the calibrated scores on𝒢W​M\\mathcal\{G\}\_\{WM\}\.

Equation \([5](https://arxiv.org/html/2608.12564#S3.E5)\) is the conceptual form of the fusion, and it attains the minimal variance onceVEV\_\{E\}andVW​MV\_\{WM\}are given\. In implementation, Equation \([5](https://arxiv.org/html/2608.12564#S3.E5)\) reduces to the simple formg^=\(ρ​gE\+gW​M\)/\(ρ​\|𝒢E\|\+\|𝒢W​M\|\)\\hat\{g\}=\\big\(\\rho\\,g\_\{E\}\+g\_\{WM\}\\big\)\\big/\\big\(\\rho\\,\|\\mathcal\{G\}\_\{E\}\|\+\|\\mathcal\{G\}\_\{WM\}\|\\big\), with only a single unknown left, the ratioρ:=VW​M/VE\\rho:=V\_\{WM\}/V\_\{E\}\. To derive this ratio, we first show in Section[4](https://arxiv.org/html/2608.12564#S4)thatρ=1\+c​σ2\\rho=1\+c\\,\\sigma^\{2\}, in which the reward noiseσ2\\sigma^\{2\}is measurable and the constantccis estimable\. We then measureσ2\\sigma^\{2\}by the mean squared residualη^2\\hat\{\\eta\}^\{2\}between the calibrated and the true scores on the anchor groups, and estimateccby the reciprocal of one such residual recorded at the end of a warmup phase,c:=1/η^cal2c:=1/\\hat\{\\eta\}^\{2\}\_\{\\mathrm\{cal\}\}\. The anchor groups then enter Equation \([5](https://arxiv.org/html/2608.12564#S3.E5)\) with the weightρ=1\+η^2/η^cal2\\rho=1\+\\hat\{\\eta\}^\{2\}/\\hat\{\\eta\}^\{2\}\_\{\\mathrm\{cal\}\}and the world model groups with the weight one, a rule that carries no free parameter\. We further justify the optimality of this estimate both theoretically and empirically in Remark[8](https://arxiv.org/html/2608.12564#Thmtheorem8), as the leading\-order term of the estimation error vanishes exactly by construction and the remaining higher\-order excess measures a few percent at most in our runs\. Both mitigations are validated empirically in Section[5](https://arxiv.org/html/2608.12564#S5)and analyzed in Section[4](https://arxiv.org/html/2608.12564#S4)\.

### 3\.4Discussion

We note that the proposed denoising weights are not only theoretically sound \(Section[4](https://arxiv.org/html/2608.12564#S4)\) but also empirically interpretable\. On the anchor groups, the tracked residualη^2\\hat\{\\eta\}^\{2\}measures how far the calibrated predictions fall from the true scores, and thereby audits the world model throughout training\. When the audit worsens, the anchor weight in Equation \([5](https://arxiv.org/html/2608.12564#S3.E5)\) rises\. Consider a task whose outcome hinges on randomness that no reading of the solution reveals\. In this case,η^2\\hat\{\\eta\}^\{2\}remains large and the gradient is carried by the anchor groups, so WMRL degrades toward the standard GRPO of Section[3\.1](https://arxiv.org/html/2608.12564#S3.SS1)instead of learning from a corrupted signal\. This fallback follows from the measurement itself rather than a mixing ratio fixed in advance\.

Additionally, although the design targets the execution bottleneck of AutoResearch, it is not limited to this scenario\. The construction requires only that rewards are expensive to execute yet predictable from the artifacts the agent produces, and that a small stream of ground truth stays available for anchoring\. Post\-training embodied policies is one such instance, where real rollouts are slow while learned simulators are cheap, and we validate this transfer on VLA tasks in Section[5](https://arxiv.org/html/2608.12564#S5)\.

## 4Theoretical Analysis

In this section, we first analyze the additional error terms that an imperfect world model introduces, and then justify our mitigation by a convergence analysis\. We first bound the convergence of training on world model rewards alone in Section[4\.2](https://arxiv.org/html/2608.12564#S4.SS2)\(Theorem[3](https://arxiv.org/html/2608.12564#Thmtheorem3)\), and then bound the convergence of WMRL in Section[4\.3](https://arxiv.org/html/2608.12564#S4.SS3)\(Theorem[4](https://arxiv.org/html/2608.12564#Thmtheorem4)\), and compare the two term by term\. We put the full proofs and auxiliary lemmas in Appendix[B](https://arxiv.org/html/2608.12564#A2)\.

### 4\.1Setup

We analyze the RL training procedure of Section[3\.1](https://arxiv.org/html/2608.12564#S3.SS1), namelyTTascent steps with step sizeγ\\gammaon the gradient estimator of Equation \([1](https://arxiv.org/html/2608.12564#S3.E1)\), and we measure progress by the expected gap to the optimal score,J⋆−𝔼⁡\[J⁡\(θT\)\]J^\{\\star\}\-\\mathbb\{E\}\[J\(\\theta\_\{T\}\)\]withJ⋆:=supθJ⁡\(θ\)J^\{\\star\}:=\\sup\_\{\\theta\}J\(\\theta\)andΔ0:=J⋆−J⁡\(θ0\)\\Delta\_\{0\}:=J^\{\\star\}\-J\(\\theta\_\{0\}\)\. The analysis rests on one standard assumption\.

###### Assumption 2\(Regularity\)\.

JJisLL\-smooth and satisfies the gradient domination condition‖∇J​\(θ\)‖2≥2​μ​\(J⋆−J⁡\(θ\)\)\\\|\\nabla J\(\\theta\)\\\|^\{2\}\\geq 2\\mu\\,\(J^\{\\star\}\-J\(\\theta\)\)for someμ\>0\\mu\>0, and the log\-likelihood gradient is bounded by‖∇θ​log​πθ​\(τ\)‖≤M\\\|\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(\\tau\)\\\|\\leq Mfor allτ\\tau\.

Everything the world model contributes enters through the two quantities of Section[3\.3](https://arxiv.org/html/2608.12564#S3.SS3), the bias magnitudeBBand the noise standard deviationσ\\sigmaof Equation \([3](https://arxiv.org/html/2608.12564#S3.E3)\), and through the two group variancesVEV\_\{E\}andVW​MV\_\{WM\}of Equation \([5](https://arxiv.org/html/2608.12564#S3.E5)\)\. The gradient estimator of Equation \([1](https://arxiv.org/html/2608.12564#S3.E1)\) is linear in the rewards, so the two error parts act on the gradient separately\. The bias is deterministic and shifts its mean\. The noise is zero\-mean and only inflates the variance of a world model group, toVW​M=\(1\+c​σ2\)​VEV\_\{WM\}=\(1\+c\\,\\sigma^\{2\}\)\\,V\_\{E\}withc=4​M2/\(n​VE\)c=4M^\{2\}/\(nV\_\{E\}\), as announced in Section[3\.3](https://arxiv.org/html/2608.12564#S3.SS3)\. Only the productc​σ2c\\,\\sigma^\{2\}enters the bounds, and Section[3\.3](https://arxiv.org/html/2608.12564#S3.SS3)supplies it by measuring its varying part and normalizing its scale, so neither is needed on its own\. Throughout we takeγ≤1/\(8​L\)\\gamma\\leq 1/\(8L\)\.

### 4\.2The Cost of World Model Rewards

We first consider the setting of Section[3\.2](https://arxiv.org/html/2608.12564#S3.SS2), where every group is graded by the world model and no real execution takes place, so the rewards carry the bias and the noise of Equation \([3](https://arxiv.org/html/2608.12564#S3.E3)\) at every step\. Since the gradient estimator built on Equation \([2](https://arxiv.org/html/2608.12564#S3.E2)\) is linear in the rewards, the two deviations reach the gradient in different ways, the bias shifting its mean and the noise inflating its variance, and they therefore enter the convergence bound as the two separate terms announced in Remark[1](https://arxiv.org/html/2608.12564#Thmtheorem1)\.

###### Theorem 3\(Convergence with world model rewards\)\. Under Assumption[2](https://arxiv.org/html/2608.12564#Thmtheorem2), GRPO trained on world model rewards satisfiesJ⋆−𝔼⁡\[J⁡\(θT\)\]≤\(1−γ​μ4\)T​Δ0\+O⁡\(M2​B2\)⏟world model bias\+O⁡\(γ​VW​M\)⏟world model variance,J^\{\\star\}\-\\mathbb\{E\}\[J\(\\theta\_\{T\}\)\]\\;\\leq\\;\\Big\(1\-\\tfrac\{\\gamma\\mu\}\{4\}\\Big\)^\{T\}\\Delta\_\{0\}\\;\+\\;\\underbrace\{O\\big\(\{\\color\[rgb\]\{0\.6484,0\.1914,0\.1914\}M^\{2\}B^\{2\}\}\\big\)\}\_\{\\text\{world model bias\}\}\\;\+\\;\\underbrace\{O\\big\(\{\\color\[rgb\]\{0\.6484,0\.1914,0\.1914\}\\gamma\\,V\_\{WM\}\}\\big\)\}\_\{\\text\{world model variance\}\},whereVW​M=\(1\+c​σ2\)​VEV\_\{WM\}=\(1\+c\\,\\sigma^\{2\}\)\\,V\_\{E\}carries the noise, and the variance term is presented asO⁡\(σ2\)O\(\{\\color\[rgb\]\{0\.6484,0\.1914,0\.1914\}\\sigma^\{2\}\}\)in Remark[1](https://arxiv.org/html/2608.12564#Thmtheorem1), and the hidden constants depend only onLLandμ\\mu\. We put the full proof in Appendix[B\.2](https://arxiv.org/html/2608.12564#A2.SS2)\.

The first term is the standard geometric decay of the noise\-free RL setting and vanishes as training proceeds\. The bias term contains neitherTTnorγ\\gamma, hence no amount of training and no choice of step size removes it, and the achievable score stays capped by the bias of the world model\. The variance term grows with the noise throughVW​MV\_\{WM\}and likewise inflates the final error, so both terms call for correction\.

### 4\.3The Effect of Our Mitigation

We now analyze WMRL, whose two corrections from Section[3\.3](https://arxiv.org/html/2608.12564#S3.SS3)improve one term of Theorem[3](https://arxiv.org/html/2608.12564#Thmtheorem3)each\. Both corrections rely on the anchor signal, where real execution grades a small fraction of the groups and yields score pairs throughout training\. With these pairs, the Online Debiasing of Equation \([4](https://arxiv.org/html/2608.12564#S3.E4)\) keeps shrinking the residual bias, at a speed captured by a constantT0T\_\{0\}given in Appendix[B\.3](https://arxiv.org/html/2608.12564#A2.SS3)\. The Inverse\-Variance Denoising of Equation \([5](https://arxiv.org/html/2608.12564#S3.E5)\) then fuses the two gradient streams, so every update carries less noise than either stream alone\. In the statement below,O~\\widetilde\{O\}hides logarithmic factors\.

###### Theorem 4\(Convergence of WMRL\)\. Under Assumption[2](https://arxiv.org/html/2608.12564#Thmtheorem2)and the conditions of Appendix[B\.3](https://arxiv.org/html/2608.12564#A2.SS3), WMRL satisfies with high probabilityJ⋆−𝔼⁡\[J⁡\(θT\)\]≤\(1−γ​μ4\)T​Δ0\+O~​\(M2​B21\+T/T0\)⏟contractive bias\+O⁡\(γ​VW​M1\+VW​M/VE\)⏟reduced variance\.J^\{\\star\}\-\\mathbb\{E\}\[J\(\\theta\_\{T\}\)\]\\;\\leq\\;\\Big\(1\-\\tfrac\{\\gamma\\mu\}\{4\}\\Big\)^\{T\}\\Delta\_\{0\}\\;\+\\;\\underbrace\{\\widetilde\{O\}\\Big\(\\frac\{\{\\color\[rgb\]\{0\.6484,0\.1914,0\.1914\}M^\{2\}B^\{2\}\}\}\{\{\\color\[rgb\]\{0\.0977,0\.418,0\.1406\}1\+T/T\_\{0\}\}\}\\Big\)\}\_\{\\text\{contractive bias\}\}\\;\+\\;\\underbrace\{O\\Big\(\\frac\{\{\\color\[rgb\]\{0\.6484,0\.1914,0\.1914\}\\gamma\\,V\_\{WM\}\}\}\{\{\\color\[rgb\]\{0\.0977,0\.418,0\.1406\}1\+V\_\{WM\}/V\_\{E\}\}\}\\Big\)\}\_\{\\text\{reduced variance\}\}\.We put the full proof in Appendix[B\.3](https://arxiv.org/html/2608.12564#A2.SS3)\.

The two bounds share the same leading term and align term by term, with each remaining term of Theorem[4](https://arxiv.org/html/2608.12564#Thmtheorem4)equal to its counterpart in Theorem[3](https://arxiv.org/html/2608.12564#Thmtheorem3)divided by a factor larger than one\. The bias term is the sameM2​B2M^\{2\}B^\{2\}divided by1\+T/T01\+T/T\_\{0\}, so the permanent floor contracts as training proceeds and vanishes in the limit\. The variance term is the sameγ​VW​M\\gamma V\_\{WM\}divided by1\+VW​M/VE1\+V\_\{WM\}/V\_\{E\}, which lands below what either stream attains alone, and the factor grows as the anchor stream becomes comparatively more reliable\. Both terms are therefore strictly smaller once the world model is biased or noisy at all, and lettingT→∞T\\to\\inftyremoves the bias term entirely\. In this sense WMRL converges to the same optimum as training with real execution, while paying for real execution on a small fraction of the groups\.

## 5Experiments

We empirically validate the effectiveness of WMRL on the two questions in Section[1](https://arxiv.org/html/2608.12564#S1), i\.e\., whether the world model delivers the promised speedup \(question\(i\)\) and whether it costs any final performance \(question\(ii\)\), with the main comparison in Section[5\.2](https://arxiv.org/html/2608.12564#S5.SS2)\(Table[1](https://arxiv.org/html/2608.12564#S5.T1)\)\. We then show how our recipe generalizes to other post\-training domains in Section[5\.3](https://arxiv.org/html/2608.12564#S5.SS3)\(Table[2](https://arxiv.org/html/2608.12564#S5.T2)\) and how the two corrections act in Section[5\.4](https://arxiv.org/html/2608.12564#S5.SS4)\(Table[3](https://arxiv.org/html/2608.12564#S5.T3)\)\.

### 5\.1Experimental Setup

#### Benchmarks\.

We evaluate WMRL on various domains, including AutoResearch tasks and vision\-language\-action \(VLA\) post\-training tasks\. In the AutoResearch evaluation, we use MLE\-Dojo\[[42](https://arxiv.org/html/2608.12564#bib.bib3)\], an interactive superset of MLE\-Bench\[[10](https://arxiv.org/html/2608.12564#bib.bib1)\]built on Kaggle machine learning competitions, and follow the common practice\[[32](https://arxiv.org/html/2608.12564#bib.bib21),[9](https://arxiv.org/html/2608.12564#bib.bib22),[66](https://arxiv.org/html/2608.12564#bib.bib23)\]of manually dividing the pool into MLE\-Dojo \(train\) and MLE\-Dojo \(test\) in a category\-balanced way, since the original test split of MLE\-Bench has several evaluation issues \(Appendix[C](https://arxiv.org/html/2608.12564#A3)\)\. We also put the selection rule and the full task lists in Appendix[C](https://arxiv.org/html/2608.12564#A3)\. Beyond the MLE\-style tasks, we further evaluate on DSBench\[[24](https://arxiv.org/html/2608.12564#bib.bib4)\], a data science benchmark also built on Kaggle modeling tasks and fully disjoint from our training set\. In the VLA evaluation, we use LIBERO\-Long\[[31](https://arxiv.org/html/2608.12564#bib.bib8)\], a standard simulated manipulation benchmark, where the policy controls a robot arm to complete long\-horizon tasks from image observations and language instructions\.

#### Models and training\.

For the AutoResearch tasks, we post\-train research agents at two scales, Qwen3\.5\-4B and Qwen3\.5\-9B\[[44](https://arxiv.org/html/2608.12564#bib.bib36)\], with the RL procedure of Section[3](https://arxiv.org/html/2608.12564#S3)\. For the world model, we use the same backbone as the agent at each scale and prompt it for outcome prediction \(Appendix[D](https://arxiv.org/html/2608.12564#A4.SS0.SSS0.Px3)\)\. Sharing one backbone also rules out any external knowledge, so the gains cannot come from implicitly distilling a stronger model\. Throughout training, only the research agent is updated while the world model stays unchanged\. All the runs use identical A100 GPU allocations, and we report the total training compute in GPU\-hours\. For the VLA tasks, the agent is MiniVLA\-1B\[[4](https://arxiv.org/html/2608.12564#bib.bib19)\], which is a compact vision\-language\-action model pretrained on LIBERO\-90, and the world model is Robometer\[[30](https://arxiv.org/html/2608.12564#bib.bib20)\], an off\-the\-shelf VLM that predicts task success rate from images\. We first finetune the agent on the official demonstrations of each task, and then post\-train it with the same RL procedure\. The remaining details and hyperparameters are listed in Appendix[D](https://arxiv.org/html/2608.12564#A4)\.

Table 1:Main results \(leaderboard percentile in %, higher is better\)\.Both benchmarks hold out tasks never trained on, andGPU\-hourscount the total training compute\. Scores are per\-task avg@8, averaged within each category, andAvgis the mean over categories\.Boldmarks the best value per column, and green \(red\) arrows mark the gain \(drop\) of WMRL over same\-scale GRPO\.MLE\-Dojo \(test\)DSBenchMethodGPU\-hoursTabTextImgAvgBin\-ClsMulti\-ClsRegressOtherAvgBaseline ModelsQwen3\.5\-4B—13\.78\.00\.27\.321\.911\.825\.88\.817\.1Qwen3\.5\-9B—17\.48\.13\.49\.629\.116\.036\.314\.123\.9Large AgentsKimi\-48B\-A3B—12\.310\.12\.08\.120\.310\.029\.09\.817\.3Nemotron\-120B\-A12B—35\.118\.87\.620\.538\.726\.640\.720\.631\.7RL in Real EnvironmentQwen3\.5\-4B\-GRPO88324\.816\.64\.315\.234\.317\.828\.322\.425\.7Qwen3\.5\-9B\-GRPO117433\.317\.15\.918\.840\.926\.840\.416\.731\.2RL with Pure World ModelQwen3\.5\-4B\-WM26921\.214\.63\.012\.927\.116\.932\.516\.023\.1Qwen3\.5\-9B\-WM33028\.015\.94\.516\.135\.421\.837\.916\.928\.0WMRL \(ours\)Qwen3\.5\-4B\-Ours28628\.0↑\\uparrow3\.216\.1↓\\downarrow0\.55\.2↑\\uparrow0\.916\.4↑\\uparrow1\.235\.8↑\\uparrow1\.522\.3↑\\uparrow4\.532\.3↑\\uparrow4\.024\.7↑\\uparrow2\.328\.8↑\\uparrow3\.1Qwen3\.5\-9B\-Ours34938\.0↑\\uparrow4\.719\.4↑\\uparrow2\.37\.4↑\\uparrow1\.521\.6↑\\uparrow2\.845\.4↑\\uparrow4\.526\.9↑\\uparrow0\.139\.2↓\\downarrow1\.219\.8↑\\uparrow3\.132\.8↑\\uparrow1\.6

#### Metrics\.

For the AutoResearch tasks, we adopt the official metric of MLE\-Dojo\[[42](https://arxiv.org/html/2608.12564#bib.bib3),[9](https://arxiv.org/html/2608.12564#bib.bib22)\], which scores each submission by its percentile on the corresponding real competition leaderboard\. The percentile is naturally normalized \(within0−10\-1\), so scores are comparable across tasks\. For the VLA tasks, we score each rollout by whether it succeeds\. In both domains, every result we report is an average over eight attempts \(avg@8\)\.

#### Baselines\.

We compare WMRL with four types of baselines\. The untrained bases Qwen3\.5\-4B and Qwen3\.5\-9B give the starting level of the backbone before any post\-training\. The large open\-weight agents Kimi\-48B\-A3B and Nemotron\-120B\-A12B serve as strong off\-the\-shelf references and test whether model scale alone can replace post\-training\. GRPO in the real environment is the standard full\-cost training that WMRL aims to match, and is our primary comparison\. The pure world model baseline trains on predicted rewards alone and shows what the raw world model delivers without any correction\. All trained models share the scaffold, the hyperparameters, and the data split\.

### 5\.2Main Results on AutoResearch Tasks

Table[1](https://arxiv.org/html/2608.12564#S5.T1)compares WMRL with the four types of baselines of Section[5\.1](https://arxiv.org/html/2608.12564#S5.SS1), at both scales and on both benchmarks, and examines whether WMRL delivers the promised speedup and whether the cheap signal costs any final performance\. The answer is positive on both counts\. First, WMRL cuts the training compute of real\-execution GRPO by3\.1×3\.1\\timesand3\.4×3\.4\\timesand still scores higher on every benchmark, with gains of up to 3\.1 points, so the saving arrives at no cost in final performance\. Second, the post\-trained agents beat far larger off\-the\-shelf agents, as our 4B model surpasses the 48B agent and our 9B model surpasses the 120B agent on both averages\.

### 5\.3Generalization Ability on Embodied VLAs

We next test whether WMRL transfers to other domains, and we pick vision\-language\-action \(VLA\) post\-training, an active line of embodied learning\[[25](https://arxiv.org/html/2608.12564#bib.bib61),[41](https://arxiv.org/html/2608.12564#bib.bib34),[3](https://arxiv.org/html/2608.12564#bib.bib30)\]\. The two calibration mechanisms are not specific to AutoResearch, so we apply the same recipe to the VLA setup of Section[5\.1](https://arxiv.org/html/2608.12564#S5.SS1)\. Here Robometer predicts a progress value for eight frames sampled from each rollout as the dense reward, and the single sparse success that the environment returns at the end of a rollout serves as the anchor\. The resulting success rates in Table[2](https://arxiv.org/html/2608.12564#S5.T2)repeat the pattern of Table[1](https://arxiv.org/html/2608.12564#S5.T1)\. Either signal alone barely moves the policy, as RL on the sparse outcome adds 0\.9 points over the SFT baseline and RL on the raw world model signal adds 1\.8\. WMRL combines the two and lifts the overall success rate by 3\.8 points, with the largest margin on unseen initial states\. The recipe therefore carries over across domains, agent architectures, and reward types\.

Table 2:VLA post\-training results on LIBERO\-Long \(success rate in %, higher is better\)\.We report in\-domain performance, out\-of\-distribution generalization, and the overall average, where Overall averages over every initial state, seen and unseen\. Best@8 counts at least one success of the eight and All@8 counts all eight\. All RL rows share the SFT initialization\.Boldmarks the best value per column, and green \(red\) arrows mark the gain \(drop\) of WMRL over MiniVLA\-1B\-GRPO\.In\-DomainOut\-of\-DistributionMethodAvgBest@8All@8AvgBest@8All@8OverallBaseline ModelsMiniVLA\-1B5\.65\.65\.62\.92\.92\.93\.8MiniVLA\-1B\-SFT37\.341\.333\.137\.544\.732\.137\.4RL in Real EnvironmentMiniVLA\-1B\-GRPO39\.348\.834\.437\.845\.331\.238\.3RL with Pure World ModelMiniVLA\-1B\-WM39\.145\.634\.439\.345\.634\.139\.2WMRL \(ours\)MiniVLA\-1B\-Ours41\.2↑\\uparrow1\.947\.5↓\\downarrow1\.337\.5↑\\uparrow3\.141\.2↑\\uparrow3\.448\.8↑\\uparrow3\.537\.1↑\\uparrow5\.941\.2↑\\uparrow2\.9

### 5\.4Ablation Study

Table 3:Ablation on the two corrections \(leaderboard percentile in %, Avg over categories\)\.OD is Online Debiasing, IVD is Inverse\-Variance Denoising, a filled circle marks the correction as active, and the last row recovers WMRL\. Green arrows give its gain over the uncorrected first row\.4B Agent9B AgentODIVDMLEDSMLEDS∘\\circ∘\\circ13\.525\.316\.829\.5∘\\circ∙\\bullet14\.926\.218\.031\.2∙\\bullet∘\\circ15\.728\.119\.431\.7∙\\bullet∙\\bullet16\.4↑\\uparrow2\.928\.8↑\\uparrow3\.521\.6↑\\uparrow4\.832\.8↑\\uparrow3\.3We then ablate the two corrections at both scales \(Table[3](https://arxiv.org/html/2608.12564#S5.T3)\)\. All runs consume the same two reward streams at the same ratio as WMRL and differ only in which correction is active\. In particular, the first row is the direct mixture that feeds both signals to GRPO with no correction at all\. From this baseline we add Inverse\-Variance Denoising alone, Online Debiasing alone, and finally both, which recovers WMRL\. The plain mixture is not enough, as its scores stay below real\-execution GRPO \(Table[1](https://arxiv.org/html/2608.12564#S5.T1)\) on every column\. Each correction then helps on its own\. Inverse\-Variance Denoising alone adds 0\.9 to 1\.7 points, Online Debiasing alone adds 2\.2 to 2\.8 points, and the larger share of the debiasing gain matches Theorem[3](https://arxiv.org/html/2608.12564#Thmtheorem3), where the bias enters the bound at full size while the noise enters damped by the step size\. Activating both lifts every column by 2\.9 to 4\.8 points, more than either correction alone, so the two mechanisms are complementary rather than redundant, as each removes the term the other leaves behind\.

## 6Conclusion

This work scales RL for AutoResearch agents by replacing the expensive environment execution with a world model and correcting its bias and noise through a small anchored stream of real execution\. The two corrections turn the permanent error floor of world model training into a contracting term and reduce the variance below either reward stream alone, and they cut the training compute by three to four times while matching or exceeding full real\-execution RL at two scales\. The transfer to VLA post\-training further suggests a general path for scaling RL wherever execution, not generation, is the bottleneck\.

## References

- \[1\]\(2020\)On the convergence of SGD with biased gradients\.arXiv preprint arXiv:2008\.00051\.Cited by:[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px2.p1.1)\.
- \[2\]P\. Auer, N\. Cesa\-Bianchi, and P\. Fischer\(2002\)Finite\-time analysis of the multiarmed bandit problem\.Machine Learning47,pp\. 235–256\.Cited by:[§B\.4](https://arxiv.org/html/2608.12564#A2.SS4.p14.1.1)\.
- \[3\]S\. Bai, J\. Lyu, W\. Zhou, Z\. Li, D\. Wang, L\. Xing, X\. Zhao, P\. Wang, Z\. Wang, C\. Chi, B\. Chen, and S\. Zhang\(2026\)Latent reasoning VLA: latent thinking and prediction for vision\-language\-action models\.InInternational Conference on Machine Learning,Cited by:[§5\.3](https://arxiv.org/html/2608.12564#S5.SS3.p1.1)\.
- \[4\]S\. Belkhale and D\. Sadigh\(2024\)MiniVLA: a better VLA with a smaller footprint\.Note:[https://github\.com/Stanford\-ILIAD/openvla\-mini](https://github.com/Stanford-ILIAD/openvla-mini)Cited by:[Appendix D](https://arxiv.org/html/2608.12564#A4.SS0.SSS0.Px7.p1.1),[§5\.1](https://arxiv.org/html/2608.12564#S5.SS1.SSS0.Px2.p1.1)\.
- \[5\]M\. Bertran, R\. Fogliato, and Z\. S\. Wu\(2026\)Many ai analysts, one dataset: navigating the agentic data science multiverse\.Proceedings of the National Academy of Sciences123\(29\),pp\. e2606495123\.Cited by:[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px1.p1.1)\.
- \[6\]D\. A\. Boiko, R\. MacKnight, B\. Kline, and G\. Gomes\(2023\)Autonomous chemical research with large language models\.Nature624\(7992\),pp\. 570–578\.Cited by:[§1](https://arxiv.org/html/2608.12564#S1.p1.1),[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px1.p1.1)\.
- \[7\]A\. M\. Bran, S\. Cox, O\. Schilter, C\. Baldassari, A\. D\. White, and P\. Schwaller\(2024\)Augmenting large language models with chemistry tools\.Nature Machine Intelligence6\(5\),pp\. 525–535\.Cited by:[§1](https://arxiv.org/html/2608.12564#S1.p1.1)\.
- \[8\]J\. Bruce, M\. D\. Dennis, A\. Edwards, J\. Parker\-Holder, Y\. Shi, E\. Hughes, M\. Lai, A\. Mavalankar, R\. Steigerwald, C\. Apps,et al\.\(2024\)Genie: generative interactive environments\.InInternational Conference on Machine Learning,Cited by:[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px2.p1.1)\.
- \[9\]Y\. Cai, Z\. Liu, X\. Zhu, C\. Wang, Y\. Wang, and S\. Chen\(2026\)AceGRPO: adaptive curriculum enhanced group relative policy optimization for autonomous machine learning engineering\.arXiv preprint arXiv:2602\.07906\.Cited by:[§5\.1](https://arxiv.org/html/2608.12564#S5.SS1.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2608.12564#S5.SS1.SSS0.Px3.p1.1)\.
- \[10\]J\. S\. Chan, N\. Chowdhury, O\. Jaffe, J\. Aung, D\. Sherburn, E\. Mays, G\. Starace, K\. Liu,et al\.\(2024\)MLE\-bench: evaluating machine learning agents on machine learning engineering\.arXiv preprint arXiv:2410\.07095\.Cited by:[Appendix C](https://arxiv.org/html/2608.12564#A3.p4.1),[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2608.12564#S5.SS1.SSS0.Px1.p1.1)\.
- \[11\]J\. Chen and N\. Jiang\(2019\)Information\-theoretic considerations in batch reinforcement learning\.InInternational Conference on Machine Learning,Cited by:[§B\.4](https://arxiv.org/html/2608.12564#A2.SS4.p14.1.1)\.
- \[12\]T\. Coste, U\. Anwar, R\. Kirk, and D\. Krueger\(2024\)Reward model ensembles help mitigate overoptimization\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px2.p1.1)\.
- \[13\]N\. Dainese, M\. Merler, M\. Alakuijala, and P\. Marttinen\(2024\)Generating code world models with large language models guided by monte carlo tree search\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px2.p1.1)\.
- \[14\]L\. Gao, J\. Schulman, and J\. Hilton\(2023\)Scaling laws for reward model overoptimization\.InProceedings of the 40th International Conference on Machine Learning,Vol\.202,pp\. 10835–10866\.Cited by:[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px2.p1.1)\.
- \[15\]J\. Gottweis, W\. Weng, A\. Daryin, T\. Tu, A\. Palepu, P\. Sirkovic, A\. Myaskovsky, F\. Weissenberger, K\. Rong, R\. Tanno,et al\.\(2025\)Towards an AI co\-scientist\.arXiv preprint arXiv:2502\.18864\.Cited by:[§1](https://arxiv.org/html/2608.12564#S1.p1.1)\.
- \[16\]C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. Weinberger\(2017\)On calibration of modern neural networks\.InProceedings of the 34th International Conference on Machine Learning,Vol\.70,pp\. 1321–1330\.Cited by:[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px2.p1.1)\.
- \[17\]D\. Guo, D\. Yang, H\. Zhang, J\. Song, R\. Zhang, R\. Xu, Q\. Zhu, S\. Ma, P\. Wang, X\. Bi,et al\.\(2025\)DeepSeek\-R1: incentivizing reasoning capability in LLMs via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§1](https://arxiv.org/html/2608.12564#S1.p1.1),[§1](https://arxiv.org/html/2608.12564#S1.p2.1),[footnote 1](https://arxiv.org/html/2608.12564#footnote1)\.
- \[18\]S\. Guo, C\. Deng, Y\. Wen, H\. Chen, Y\. Chang, and J\. Wang\(2024\)DS\-Agent: automated data science by empowering large language models with case\-based reasoning\.InInternational Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2608.12564#S1.p1.1)\.
- \[19\]D\. Ha and J\. Schmidhuber\(2018\)World models\.arXiv preprint arXiv:1803\.10122\.Cited by:[§1](https://arxiv.org/html/2608.12564#S1.p4.1),[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px2.p1.1)\.
- \[20\]D\. Hafner, J\. Pasukonis, J\. Ba, and T\. Lillicrap\(2025\)Mastering diverse control tasks through world models\.Nature640\(8059\),pp\. 647–653\.Cited by:[§1](https://arxiv.org/html/2608.12564#S1.p4.1),[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px2.p1.1)\.
- \[21\]Q\. Huang, J\. Vora, P\. Liang, and J\. Leskovec\(2024\)MLAgentBench: evaluating language agents on machine learning experimentation\.InInternational Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2608.12564#S1.p1.1)\.
- \[22\]Z\. Jiang, D\. Schmidt, D\. Srikanth, D\. Xu, I\. Kaplan, D\. Jacenko, and Y\. Wu\(2025\)AIDE: AI\-driven exploration in the space of code\.arXiv preprint arXiv:2502\.13138\.Cited by:[§1](https://arxiv.org/html/2608.12564#S1.p1.1),[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px1.p1.1)\.
- \[23\]C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. Narasimhan\(2024\)SWE\-bench: can language models resolve real\-world GitHub issues?\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px1.p1.1)\.
- \[24\]L\. Jing, Z\. Huang, X\. Wang, W\. Yao, W\. Yu, K\. Ma, H\. Zhang, X\. Du,et al\.\(2024\)DSBench: how far are data science agents from becoming data science experts?\.arXiv preprint arXiv:2409\.07703\.Cited by:[Appendix C](https://arxiv.org/html/2608.12564#A3.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2608.12564#S3.SS1.p1.1),[§5\.1](https://arxiv.org/html/2608.12564#S5.SS1.SSS0.Px1.p1.1)\.
- \[25\]M\. J\. Kim, K\. Pertsch, S\. Karamcheti, T\. Xiao, A\. Balakrishna, S\. Nair, R\. Rafailov, E\. Foster, G\. Lam, P\. Sanketi,et al\.\(2024\)OpenVLA: an open\-source vision\-language\-action model\.InConference on Robot Learning,Cited by:[§5\.3](https://arxiv.org/html/2608.12564#S5.SS3.p1.1)\.
- \[26\]I\. Kulikov, C\. Whitehouse, T\. Wu, Y\. Nie, S\. Saha, E\. Helenowski, W\. Yuan, O\. Golovneva, J\. Lanchantin, Y\. Bachrach, J\. Foerster, X\. Li, H\. Fang, S\. Sukhbaatar, and J\. Weston\(2026\)AutoData: an agentic data scientist to create high quality synthetic data\.arXiv preprint arXiv:2606\.25996\.Cited by:[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px1.p1.1)\.
- \[27\]W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. Stoica\(2023\)Efficient memory management for large language model serving with pagedattention\.InProceedings of the 29th Symposium on Operating Systems Principles,Cited by:[§1](https://arxiv.org/html/2608.12564#S1.p2.1)\.
- \[28\]N\. Lambert, J\. Morrison, V\. Pyatkin, S\. Huang, H\. Ivison, F\. Brahman, L\. J\. V\. Miranda, A\. Liu, N\. Dziri, S\. Lyu,et al\.\(2024\)Tulu 3: pushing frontiers in open language model post\-training\.arXiv preprint arXiv:2411\.15124\.Cited by:[§1](https://arxiv.org/html/2608.12564#S1.p1.1)\.
- \[29\]J\. Li, S\. Cheng, Z\. Liao, H\. Wang, W\. Y\. Wang, and Q\. Bai\(2022\)Off\-policy reinforcement learning with optimistic exploration and distribution correction\.InDeep Reinforcement Learning Workshop, NeurIPS,Cited by:[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px2.p1.1)\.
- \[30\]A\. Liang, Y\. Korkmaz, J\. Zhang, M\. Hwang, A\. Anwar, S\. Kaushik, A\. Shah, A\. S\. Huang, L\. Zettlemoyer, D\. Fox, Y\. Xiang, A\. Li, A\. Bobu, A\. Gupta, S\. Tu, E\. Biyik, and J\. Zhang\(2026\)Robometer: scaling general\-purpose robotic reward models via trajectory comparisons\.arXiv preprint arXiv:2603\.02115\.Cited by:[§5\.1](https://arxiv.org/html/2608.12564#S5.SS1.SSS0.Px2.p1.1)\.
- \[31\]B\. Liu, Y\. Zhu, C\. Gao, Y\. Feng, Q\. Liu, Y\. Zhu, and P\. Stone\(2023\)LIBERO: benchmarking knowledge transfer for lifelong robot learning\.InAdvances in Neural Information Processing Systems,Cited by:[§5\.1](https://arxiv.org/html/2608.12564#S5.SS1.SSS0.Px1.p1.1)\.
- \[32\]Z\. Liu, J\. Chai, X\. Zhu, S\. Tang, R\. Ye, B\. Zhang, L\. Bai, and S\. Chen\(2025\)ML\-Agent: reinforcing LLM agents for autonomous machine learning engineering\.arXiv preprint arXiv:2505\.23723\.Cited by:[§5\.1](https://arxiv.org/html/2608.12564#S5.SS1.SSS0.Px1.p1.1)\.
- \[33\]C\. Lu, C\. Lu, R\. T\. Lange, J\. Foerster, J\. Clune, and D\. Ha\(2024\)The AI scientist: towards fully automated open\-ended scientific discovery\.arXiv preprint arXiv:2408\.06292\.Cited by:[§1](https://arxiv.org/html/2608.12564#S1.p1.1),[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px1.p1.1)\.
- \[34\]O\. Nachum, Y\. Chow, B\. Dai, and L\. Li\(2019\)DualDICE: behavior\-agnostic estimation of discounted stationary distribution corrections\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px2.p1.1)\.
- \[35\]D\. Nathani, L\. Madaan, N\. Roberts, N\. Bashlykov,et al\.\(2025\)MLGym: a new framework and benchmark for advancing AI research agents\.arXiv preprint arXiv:2502\.14499\.Cited by:[Appendix C](https://arxiv.org/html/2608.12564#A3.p3.1),[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px1.p1.1)\.
- \[36\]A\. Novikov, N\. Vũ, M\. Eisenberger,et al\.\(2025\)AlphaEvolve: a coding agent for scientific and algorithmic discovery\.arXiv preprint arXiv:2506\.13131\.Cited by:[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px1.p1.1)\.
- \[37\]NVIDIA\(2025\)Cosmos world foundation model platform for physical AI\.arXiv preprint arXiv:2501\.03575\.Cited by:[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px2.p1.1)\.
- \[38\]OpenAI\(2024\)MLE\-bench: evaluating machine learning agents on machine learning engineering\.Note:[https://openai\.com/index/mle\-bench/](https://openai.com/index/mle-bench/)Cited by:[Appendix C](https://arxiv.org/html/2608.12564#A3.p4.1)\.
- \[39\]L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. L\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. F\. Christiano, J\. Leike, and R\. Lowe\(2022\)Training language models to follow instructions with human feedback\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 27730–27744\.Cited by:[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px2.p1.1)\.
- \[40\]J\. Pan, X\. Wang, G\. Neubig, N\. Jaitly, H\. Ji, A\. Suhr, and Y\. Zhang\(2025\)Training software engineering agents and verifiers with SWE\-Gym\.InInternational Conference on Machine Learning,Cited by:[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px1.p1.1)\.
- \[41\]Physical Intelligence, K\. Black, N\. Brown,et al\.\(2025\)π0\.5\\pi\_\{0\.5\}: A vision\-language\-action model with open\-world generalization\.arXiv preprint arXiv:2504\.16054\.Cited by:[§5\.3](https://arxiv.org/html/2608.12564#S5.SS3.p1.1)\.
- \[42\]R\. Qiang, Y\. Zhuang, Y\. Li, D\. S\. V K,et al\.\(2025\)MLE\-Dojo: interactive environments for empowering LLM agents in machine learning engineering\.arXiv preprint arXiv:2505\.07782\.Cited by:[§1](https://arxiv.org/html/2608.12564#S1.p2.1),[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2608.12564#S3.SS1.p1.1),[§5\.1](https://arxiv.org/html/2608.12564#S5.SS1.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2608.12564#S5.SS1.SSS0.Px3.p1.1)\.
- \[43\]Y\. Qu, A\. Setlur, V\. Smith, R\. Salakhutdinov, and A\. Kumar\(2026\)POPE: learning to reason on hard problems via privileged on\-policy exploration\.arXiv preprint arXiv:2601\.18779\.Cited by:[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px1.p1.1)\.
- \[44\]Qwen Team\(2026\)Qwen3\.5\.Note:[https://qwen\.ai/blog?id=qwen3\.5](https://qwen.ai/blog?id=qwen3.5)Cited by:[§5\.1](https://arxiv.org/html/2608.12564#S5.SS1.SSS0.Px2.p1.1)\.
- \[45\]Recursive\(2026\)First steps toward automated AI research\.Note:[https://www\.recursive\.com/articles/first\-steps\-toward\-automated\-ai\-research](https://www.recursive.com/articles/first-steps-toward-automated-ai-research)Cited by:[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px1.p1.1)\.
- \[46\]S\. Schmidgall, Y\. Su, Z\. Wang, X\. Sun, J\. Wu, X\. Yu, J\. Liu, Z\. Liu, and E\. Barsoum\(2025\)Agent laboratory: using LLM agents as research assistants\.arXiv preprint arXiv:2501\.04227\.Cited by:[§1](https://arxiv.org/html/2608.12564#S1.p1.1)\.
- \[47\]J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov\(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§1](https://arxiv.org/html/2608.12564#S1.p1.1)\.
- \[48\]Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang,et al\.\(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§1](https://arxiv.org/html/2608.12564#S1.p1.1),[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2608.12564#S3.SS1.p2.1)\.
- \[49\]G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu,et al\.\(2024\)HybridFlow: a flexible and efficient RLHF framework\.arXiv preprint arXiv:2409\.19256\.Cited by:[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px1.p1.1),[footnote 1](https://arxiv.org/html/2608.12564#footnote1)\.
- \[50\]N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. Yao\(2023\)Reflexion: language agents with verbal reinforcement learning\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2608.12564#S1.p1.1)\.
- \[51\]S\. Sun, H\. Song, L\. Huang, J\. Jiang, R\. Le, Z\. Lv, Z\. Chen, Y\. Hu, W\. Luo, W\. X\. Zhao, Y\. Song, H\. Xu, T\. Zhang, and J\. Wen\(2026\)SWE\-World: building software engineering agents in docker\-free environments\.arXiv preprint arXiv:2602\.03419\.Cited by:[§1](https://arxiv.org/html/2608.12564#S1.p2.1),[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px2.p1.1)\.
- \[52\]N\. J\. Szymanski, B\. Rendy, Y\. Fei, R\. E\. Kumar, T\. He, D\. Milsted, M\. J\. McDermott, M\. Gallant, E\. D\. Cubuk, A\. Merchant,et al\.\(2023\)An autonomous laboratory for the accelerated synthesis of novel inorganic materials\.Nature624,pp\. 86–91\.Cited by:[§1](https://arxiv.org/html/2608.12564#S1.p1.1)\.
- \[53\]H\. Tang, D\. Key, and K\. Ellis\(2024\)WorldCoder, a model\-based LLM agent: building world models by writing code and interacting with the environment\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px2.p1.1)\.
- \[54\]X\. Wang, B\. Li, Y\. Song, F\. F\. Xu, X\. Tang, M\. Zhuge, J\. Pan, Y\. Song, B\. Li, J\. Singh,et al\.\(2025\)OpenHands: an open platform for AI software developers as generalist agents\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px1.p1.1)\.
- \[55\]J\. Xie, T\. Lin, Z\. Wang, Y\. Ning, Y\. Yao, T\. Xue, Z\. Zhang, Z\. Li, K\. Zhang, Y\. Wu, S\. Chen, B\. Gou, M\. Han, Y\. Wang, V\. Lee, X\. Wei, X\. Wang, Y\. Su, and H\. Sun\(2026\)QUEST: training frontier deep research agents with fully synthetic tasks\.arXiv preprint arXiv:2605\.24218\.Cited by:[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px1.p1.1)\.
- \[56\]Y\. Yamada, R\. T\. Lange, C\. Lu, S\. Hu, C\. Lu, J\. Foerster, J\. Clune, and D\. Ha\(2025\)The AI scientist\-v2: workshop\-level automated scientific discovery via agentic tree search\.arXiv preprint arXiv:2504\.08066\.Cited by:[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px1.p1.1)\.
- \[57\]J\. Yang, C\. E\. Jimenez, A\. Wettig, K\. Lieret, S\. Yao, K\. Narasimhan, and O\. Press\(2024\)SWE\-agent: agent\-computer interfaces enable automated software engineering\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px1.p1.1)\.
- \[58\]J\. Yang, C\. Jiang, Y\. Fu, T\. Luo, C\. Ren, W\. Wang, K\. Zhao, H\. Liu,et al\.\(2026\)Frontis\-MA1: training an AI4AI model towards recursive self\-improvement in machine learning engineering\.arXiv preprint arXiv:2607\.28568\.Cited by:[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px1.p1.1)\.
- \[59\]S\. Yao, D\. Yu, J\. Zhao, I\. Shafran, T\. Griffiths, Y\. Cao, and K\. Narasimhan\(2023\)Tree of thoughts: deliberate problem solving with large language models\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2608.12564#S1.p1.1)\.
- \[60\]S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao\(2023\)ReAct: synergizing reasoning and acting in language models\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.12564#S1.p1.1)\.
- \[61\]Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, T\. Fan, G\. Liu, L\. Liu, X\. Liu,et al\.\(2025\)DAPO: an open\-source LLM reinforcement learning system at scale\.arXiv preprint arXiv:2503\.14476\.Cited by:[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px1.p1.1)\.
- \[62\]B\. Zadrozny and C\. Elkan\(2002\)Transforming classifier scores into accurate multiclass probability estimates\.InProceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,Cited by:[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px2.p1.1),[§3\.3](https://arxiv.org/html/2608.12564#S3.SS3.p4.2)\.
- \[63\]C\. Zhang\(2002\)Risk bounds in isotonic regression\.The Annals of Statistics30\(2\),pp\. 528–555\.Cited by:[§B\.4](https://arxiv.org/html/2608.12564#A2.SS4.p14.1.1)\.
- \[64\]L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica\(2023\)Judging LLM\-as\-a\-judge with MT\-bench and chatbot arena\.InAdvances in Neural Information Processing Systems 36: Datasets and Benchmarks Track,Vol\.36,pp\. 46595–46623\.Cited by:[§2](https://arxiv.org/html/2608.12564#S2.SS0.SSS0.Px2.p1.1)\.
- \[65\]L\. Zheng, L\. Yin, Z\. Xie, C\. Sun, J\. Huang, C\. H\. Yu, S\. Cao, C\. Kozyrakis, I\. Stoica, J\. E\. Gonzalez, C\. Barrett, and Y\. Sheng\(2024\)SGLang: efficient execution of structured language model programs\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2608.12564#S1.p2.1)\.
- \[66\]Y\. Zhou, L\. Zhang, Y\. Wu, J\. Liu, X\. Fan, Z\. Zhao, and H\. Yan\(2026\)Synthetic sandbox for training machine learning engineering agents\.arXiv preprint arXiv:2604\.04872\.Cited by:[§5\.1](https://arxiv.org/html/2608.12564#S5.SS1.SSS0.Px1.p1.1)\.

## Appendix

## Appendix ANotation

Table[4](https://arxiv.org/html/2608.12564#A1.T4)collects the notation of the main text, in order of first appearance\.

Table 4:Notation used in the main text\.SymbolMeaningτ\\tau,r⁡\(τ\)r\(\\tau\)a trajectory and its true score from real execution,r⁡\(τ\)∈\[0,1\]r\(\\tau\)\\in\[0,1\]πθ\\pi\_\{\\theta\},J⁡\(θ\)J\(\\theta\)the agent policy and its expected score𝔼τ∼πθ​\[r​\(τ\)\]\\mathbb\{E\}\_\{\\tau\\sim\\pi\_\{\\theta\}\}\[r\(\\tau\)\]nn,AiA\_\{i\},sis\_\{i\}group size, the advantage, and the policy gradient of trajectoryτi\\tau\_\{i\}g^\\hat\{g\},γ\\gamma,TTthe gradient estimate, the step size, and the number of training stepsr^​\(τ\)\\hat\{r\}\(\\tau\),A^i\\hat\{A\}\_\{i\}the world model score and the advantage computed from itb⁡\(τ\)b\(\\tau\),ξ⁡\(τ\)\\xi\(\\tau\)the bias and the zero\-mean noise of the world model scoreBB,σ\\sigmathe maximal bias magnitude and the maximal noise standard deviation𝒫\\mathcal\{P\}the pool of score pairs\(r^j,rj\)\(\\hat\{r\}\_\{j\},r\_\{j\}\)collected on anchor groupsf^\\hat\{f\}the monotone recalibration map fit on𝒫\\mathcal\{P\}mm,𝒢E\\mathcal\{G\}\_\{E\},𝒢W​M\\mathcal\{G\}\_\{WM\}the groups per step and the index sets of the anchor and world model onesgEg\_\{E\},gW​Mg\_\{WM\}the sums of the group estimatesg^k\\hat\{g\}\_\{k\}over𝒢E\\mathcal\{G\}\_\{E\}and over𝒢W​M\\mathcal\{G\}\_\{WM\}VEV\_\{E\},VW​MV\_\{WM\}the per\-group gradient variances of the two streamsρ\\rho,ccthe variance ratioVW​M/VE=1\+c​σ2V\_\{WM\}/V\_\{E\}=1\+c\\,\\sigma^\{2\}and its conversion constantη^2\\hat\{\\eta\}^\{2\},η^cal2\\hat\{\\eta\}^\{2\}\_\{\\mathrm\{cal\}\}the tracked residual of the calibrated scores and its warmup valueJ⋆J^\{\\star\},Δ0\\Delta\_\{0\}the optimal score and the initial gapJ⋆−J⁡\(θ0\)J^\{\\star\}\-J\(\\theta\_\{0\}\)LL,μ\\mu,MMthe smoothness, gradient domination, and gradient bound constantsε⁡\(T\)\\varepsilon\(T\)the convergence term of standard RL \(Proposition[5](https://arxiv.org/html/2608.12564#Thmtheorem5)\)T0T\_\{0\}the recalibration time scale of Theorem[4](https://arxiv.org/html/2608.12564#Thmtheorem4)
## Appendix BProofs

This appendix proves every claim of Section[4](https://arxiv.org/html/2608.12564#S4), in the order of the story\. Appendix[B\.1](https://arxiv.org/html/2608.12564#A2.SS1)derives the standard convergence termε⁡\(T\)\\varepsilon\(T\)of Remark[1](https://arxiv.org/html/2608.12564#Thmtheorem1)for training with real execution\. Appendix[B\.2](https://arxiv.org/html/2608.12564#A2.SS2)proves Theorem[3](https://arxiv.org/html/2608.12564#Thmtheorem3)for training on world model rewards, and Appendix[B\.3](https://arxiv.org/html/2608.12564#A2.SS3)proves Theorem[4](https://arxiv.org/html/2608.12564#Thmtheorem4)for WMRL and concludes with the term\-by\-term comparison\. The auxiliary lemmas invoked along the way are only cited there, and their statements with full proofs are collected in Appendix[B\.4](https://arxiv.org/html/2608.12564#A2.SS4)\. Throughout, the variance of a random vectorXXis the scalarVar⁡\(X\):=𝔼​‖X−𝔼​X‖2\\mathrm\{Var\}\(X\):=\\mathbb\{E\}\\\|X\-\\mathbb\{E\}X\\\|^\{2\}, the trace of its covariance matrix, which satisfies𝔼​‖X‖2=‖𝔼​X‖2\+Var⁡\(X\)\\mathbb\{E\}\\\|X\\\|^\{2\}=\\\|\\mathbb\{E\}X\\\|^\{2\}\+\\mathrm\{Var\}\(X\)\.

### B\.1Warm\-up: convergence of standard RL

We first derive the bound that training with real execution satisfies\. It contains no contribution from the world model and is exactly theε⁡\(T\)\\varepsilon\(T\)of Remark[1](https://arxiv.org/html/2608.12564#Thmtheorem1)\.

###### Proposition 5\(Standard RL\)\.

Under Assumption[2](https://arxiv.org/html/2608.12564#Thmtheorem2)andγ≤1/\(8​L\)\\gamma\\leq 1/\(8L\), GRPO trained on real execution rewards satisfies

J⋆−𝔼⁡\[J⁡\(θT\)\]≤\(1−γ​μ4\)T​Δ0\+4​L​γμ​VE=:ε⁡\(T\)\.J^\{\\star\}\-\\mathbb\{E\}\[J\(\\theta\_\{T\}\)\]\\;\\leq\\;\\Big\(1\-\\tfrac\{\\gamma\\mu\}\{4\}\\Big\)^\{T\}\\Delta\_\{0\}\\;\+\\;\\frac\{4L\\gamma\}\{\\mu\}\\,V\_\{E\}\\;=:\\;\\varepsilon\(T\)\.

###### Proof\.

We proceed in three steps\.

*Step 1, the estimate is clean\.*Let𝔼t\\mathbb\{E\}\_\{t\}denote expectation conditioned onθt\\theta\_\{t\}, so thatJ⁡\(θt\)J\(\\theta\_\{t\}\)and its gradient are fixed under𝔼t\\mathbb\{E\}\_\{t\}\. By Lemma[10](https://arxiv.org/html/2608.12564#Thmtheorem10)\(i\), proved in Appendix[B\.4](https://arxiv.org/html/2608.12564#A2.SS4), the estimator of Equation \([1](https://arxiv.org/html/2608.12564#S3.E1)\) satisfies𝔼t\[g^t\]=κ∇J\(θt\)\\mathbb\{E\}\_\{t\}\[\\hat\{g\}\_\{t\}\]=\\kappa\\nabla J\(\\theta\_\{t\}\)withκ=1−1n∈\[12,1\)\\kappa=1\-\\tfrac\{1\}\{n\}\\in\[\\tfrac\{1\}\{2\},1\)andVar⁡\(g^t∣θt\)≤VE\\mathrm\{Var\}\(\\hat\{g\}\_\{t\}\\mid\\theta\_\{t\}\)\\leq V\_\{E\}\. Since‖𝔼t​g^t‖=κ​‖∇J​\(θt\)‖≤‖∇J​\(θt\)‖\\\|\\mathbb\{E\}\_\{t\}\\hat\{g\}\_\{t\}\\\|=\\kappa\\\|\\nabla J\(\\theta\_\{t\}\)\\\|\\leq\\\|\\nabla J\(\\theta\_\{t\}\)\\\|, the second moment obeys𝔼t​‖g^t‖2=‖𝔼t​g^t‖2\+Var⁡\(g^t∣θt\)≤‖∇J​\(θt\)‖2\+VE\\mathbb\{E\}\_\{t\}\\\|\\hat\{g\}\_\{t\}\\\|^\{2\}=\\\|\\mathbb\{E\}\_\{t\}\\hat\{g\}\_\{t\}\\\|^\{2\}\+\\mathrm\{Var\}\(\\hat\{g\}\_\{t\}\\mid\\theta\_\{t\}\)\\leq\\\|\\nabla J\(\\theta\_\{t\}\)\\\|^\{2\}\+V\_\{E\}\.

*Step 2, one\-step progress\.*Write∇:=∇J​\(θt\)\\nabla:=\\nabla J\(\\theta\_\{t\}\)\. ByLL\-smoothness ofJJ,

J⁡\(θt\+1\)≥J⁡\(θt\)\+γ⁡⟨∇,g^t⟩−L​γ22​‖g^t‖2\.J\(\\theta\_\{t\+1\}\)\\;\\geq\\;J\(\\theta\_\{t\}\)\+\\gamma\\langle\\nabla,\\hat\{g\}\_\{t\}\\rangle\-\\frac\{L\\gamma^\{2\}\}\{2\}\\,\\\|\\hat\{g\}\_\{t\}\\\|^\{2\}\.Taking expectations conditioned onθt\\theta\_\{t\}and inserting Step 1,

𝔼t​J​\(θt\+1\)≥J⁡\(θt\)\+γ⁡\(κ−L​γ2\)​‖∇‖2−L​γ22​VE≥J⁡\(θt\)\+γ8​‖∇‖2−L​γ22​VE,\\mathbb\{E\}\_\{t\}\\,J\(\\theta\_\{t\+1\}\)\\;\\geq\\;J\(\\theta\_\{t\}\)\+\\gamma\\Big\(\\kappa\-\\tfrac\{L\\gamma\}\{2\}\\Big\)\\\|\\nabla\\\|^\{2\}\-\\frac\{L\\gamma^\{2\}\}\{2\}\\,V\_\{E\}\\;\\geq\\;J\(\\theta\_\{t\}\)\+\\frac\{\\gamma\}\{8\}\\,\\\|\\nabla\\\|^\{2\}\-\\frac\{L\\gamma^\{2\}\}\{2\}\\,V\_\{E\},where the last step usesκ≥12\\kappa\\geq\\tfrac\{1\}\{2\}andL​γ≤18L\\gamma\\leq\\tfrac\{1\}\{8\}\.

*Step 3, gradient domination and unrolling\.*By Assumption[2](https://arxiv.org/html/2608.12564#Thmtheorem2),‖∇‖2≥2​μ​\(J⋆−J⁡\(θt\)\)\\\|\\nabla\\\|^\{2\}\\geq 2\\mu\\,\(J^\{\\star\}\-J\(\\theta\_\{t\}\)\)\. Writingδt:=J⋆−𝔼⁡\[J⁡\(θt\)\]\\delta\_\{t\}:=J^\{\\star\}\-\\mathbb\{E\}\[J\(\\theta\_\{t\}\)\]and taking total expectations,

δt\+1≤\(1−γ​μ4\)​δt\+L​γ22​VE\.\\delta\_\{t\+1\}\\;\\leq\\;\\Big\(1\-\\tfrac\{\\gamma\\mu\}\{4\}\\Big\)\\,\\delta\_\{t\}\+\\frac\{L\\gamma^\{2\}\}\{2\}\\,V\_\{E\}\.Unrolling overt=0,…,T−1t=0,\\dots,T\-1and bounding the geometric sum by∑k≥0\(1−γ​μ4\)k=4γ​μ\\sum\_\{k\\geq 0\}\(1\-\\tfrac\{\\gamma\\mu\}\{4\}\)^\{k\}=\\tfrac\{4\}\{\\gamma\\mu\}givesδT≤\(1−γ​μ4\)T​Δ0\+2​L​γμ​VE\\delta\_\{T\}\\leq\(1\-\\tfrac\{\\gamma\\mu\}\{4\}\)^\{T\}\\Delta\_\{0\}\+\\tfrac\{2L\\gamma\}\{\\mu\}V\_\{E\}, and we state the looser constant4​L​γμ\\tfrac\{4L\\gamma\}\{\\mu\}to share constants with the perturbed case below\. ∎

### B\.2Proof of Theorem[3](https://arxiv.org/html/2608.12564#Thmtheorem3)

With predicted rewards the estimate is no longer clean\. By Lemma[10](https://arxiv.org/html/2608.12564#Thmtheorem10), stated and proved in Appendix[B\.4](https://arxiv.org/html/2608.12564#A2.SS4), the estimator computed with predicted scores decomposes asg^kW​M=g^k\+gb\+gξ\\hat\{g\}\_\{k\}^\{WM\}=\\hat\{g\}\_\{k\}\+g\_\{b\}\+g\_\{\\xi\}, wheregbg\_\{b\}is a bounded deterministic shift induced by the bias andgξg\_\{\\xi\}is a zero\-mean fluctuation induced by the noise\. The recursion of Proposition[5](https://arxiv.org/html/2608.12564#Thmtheorem5)then goes through with two modifications, the variance grows and a perturbation term appears, which is the content of Lemma[9](https://arxiv.org/html/2608.12564#Thmtheorem9)in Appendix[B\.4](https://arxiv.org/html/2608.12564#A2.SS4)\.

###### Theorem\(Convergence with world model rewards, Theorem[3](https://arxiv.org/html/2608.12564#Thmtheorem3)of Section[4\.2](https://arxiv.org/html/2608.12564#S4.SS2)with explicit constants\)\.

Under Assumption[2](https://arxiv.org/html/2608.12564#Thmtheorem2), GRPO trained on world model rewards withγ≤1/\(8​L\)\\gamma\\leq 1/\(8L\)satisfies

J⋆−𝔼⁡\[J⁡\(θT\)\]≤\(1−γ​μ4\)T​Δ0\+32​M2μ​B2\+4​L​γμ​VW​M\.J^\{\\star\}\-\\mathbb\{E\}\[J\(\\theta\_\{T\}\)\]\\;\\leq\\;\\Big\(1\-\\tfrac\{\\gamma\\mu\}\{4\}\\Big\)^\{T\}\\Delta\_\{0\}\\;\+\\;\\frac\{32M^\{2\}\}\{\\mu\}\\,B^\{2\}\\;\+\\;\\frac\{4L\\gamma\}\{\\mu\}\\,V\_\{WM\}\.

###### Proof\.

The plan is to verify the hypotheses of Lemma[9](https://arxiv.org/html/2608.12564#Thmtheorem9)for the decomposition supplied by Lemma[10](https://arxiv.org/html/2608.12564#Thmtheorem10), and then read off the bound\. By Lemma[10](https://arxiv.org/html/2608.12564#Thmtheorem10), the estimate computed with predicted scores splits asu\+du\+dwithu:=g^k\+gξu:=\\hat\{g\}\_\{k\}\+g\_\{\\xi\}andd:=gbd:=g\_\{b\}\. The partuuis the clean estimate plus a conditionally zero\-mean fluctuation, so its mean is the unchanged\(1−1n\)∇J\(1\-\\tfrac\{1\}\{n\}\)\\nabla J, and sinceg^k\\hat\{g\}\_\{k\}andgξg\_\{\\xi\}are uncorrelated their variances add,VE\+4​M2​σ2/n=VW​MV\_\{E\}\+4M^\{2\}\\sigma^\{2\}/n=V\_\{WM\}\. The partddobeys‖d‖≤2​M​B\\\|d\\\|\\leq 2MBalmost surely, which serves as the constant perturbation boundDD\. Applying Lemma[9](https://arxiv.org/html/2608.12564#Thmtheorem9)withκt=1−1n≥12\\kappa\_\{t\}=1\-\\tfrac\{1\}\{n\}\\geq\\tfrac\{1\}\{2\},V=VW​MV=V\_\{WM\}andD=2​M​BD=2MBthen yields the three terms of the claim, the geometric decay, the constant bias term8μ​\(2​M​B\)2=32​M2μ​B2\\tfrac\{8\}\{\\mu\}\(2MB\)^\{2\}=\\tfrac\{32M^\{2\}\}\{\\mu\}B^\{2\}, and the variance term4​L​γμ​VW​M\\tfrac\{4L\\gamma\}\{\\mu\}V\_\{WM\}\. ∎

### B\.3Proof of Theorem[4](https://arxiv.org/html/2608.12564#Thmtheorem4)and the comparison

Compared with Appendix[B\.2](https://arxiv.org/html/2608.12564#A2.SS2), two things improve for WMRL, the recalibration makes the perturbation bound shrink over time, and the fusion makes the variance harmonic\. Analyzing the recalibration needs one modeling assumption beyond Assumption[2](https://arxiv.org/html/2608.12564#Thmtheorem2)\.

###### Assumption 6\(Score scale\)\.

The bias acts on the score scale as a monotone distortion, that is,b⁡\(τ\)=ϕ⁡\(r⁡\(τ\)\)−r⁡\(τ\)b\(\\tau\)=\\phi\(r\(\\tau\)\)\-r\(\\tau\)for a non\-decreasingϕ\\phi, and the noiseξ\\xiis independent across trajectories with\|ξ\|≤1\|\\xi\|\\leq 1\.

This is what makes the bias learnable by a monotone fit\. How fast it is learned is quantified by Lemma[11](https://arxiv.org/html/2608.12564#Thmtheorem11), whose constantcfc\_\{f\}measures how quickly the recalibration error decays\. The constantT0T\_\{0\}of Theorem[4](https://arxiv.org/html/2608.12564#Thmtheorem4)isT0:=cf2/B2T\_\{0\}:=c\_\{f\}^\{2\}/B^\{2\}, the number of steps after which the residual bias of the recalibrated scores falls below the raw bias of the world model\.

###### Theorem\(Convergence of WMRL, Theorem[4](https://arxiv.org/html/2608.12564#Thmtheorem4)of Section[4\.3](https://arxiv.org/html/2608.12564#S4.SS3)with explicit constants\)\.

Under Assumptions[2](https://arxiv.org/html/2608.12564#Thmtheorem2)and[6](https://arxiv.org/html/2608.12564#Thmtheorem6)andγ≤1/\(8​L\)\\gamma\\leq 1/\(8L\), WMRL satisfies with probability at least1−δ1\-\\delta

J⋆−𝔼⁡\[J⁡\(θT\)\]≤\\displaystyle J^\{\\star\}\-\\mathbb\{E\}\[J\(\\theta\_\{T\}\)\]\\;\\leq\(1−γ​μ4\)T​Δ0\+64​cf2​M2μ⋅log⁡\(2​K​T/δ\)T\\displaystyle\\Big\(1\-\\tfrac\{\\gamma\\mu\}\{4\}\\Big\)^\{T\}\\Delta\_\{0\}\\;\+\\;\\frac\{64\\,c\_\{f\}^\{2\}M^\{2\}\}\{\\mu\}\\cdot\\frac\{\\log\(2KT/\\delta\)\}\{T\}\+4​γ​T​M2​B2​\(1−γ​μ4\)T/2\+4​L​γμ​\(1VE\+1VW​M\)−1,\\displaystyle\+\\;4\\gamma TM^\{2\}B^\{2\}\\Big\(1\-\\tfrac\{\\gamma\\mu\}\{4\}\\Big\)^\{T/2\}\\;\+\\;\\frac\{4L\\gamma\}\{\\mu\}\\Big\(\\frac\{1\}\{V\_\{E\}\}\+\\frac\{1\}\{V\_\{WM\}\}\\Big\)^\{\-1\},whereVW​MV\_\{WM\}is formed with the post\-recalibration noise andcfc\_\{f\}is the constant of Lemma[11](https://arxiv.org/html/2608.12564#Thmtheorem11)\. The third term decays exponentially inTT, and the whole perturbation is at most8μ​M2​B2\\tfrac\{8\}\{\\mu\}M^\{2\}B^\{2\}at everyTT\. Sincemin⁡\(B2,cf2/T\)\\min\(B^\{2\},c\_\{f\}^\{2\}/T\)andB2/\(1\+T/T0\)B^\{2\}/\(1\+T/T\_\{0\}\)withT0:=cf2/B2T\_\{0\}:=c\_\{f\}^\{2\}/B^\{2\}agree up to a factor of two, this is the form stated in Section[4\.3](https://arxiv.org/html/2608.12564#S4.SS3)\.

###### Proof\.

The proof has two parts\. We first verify that the fused estimate satisfies the hypotheses of Lemma[9](https://arxiv.org/html/2608.12564#Thmtheorem9), now with a smaller variance and a perturbation that shrinks over time, and we then bound the resulting perturbation sum, which is where the contraction of the bias comes from\.

*Part 1, the hypotheses\.*Work on the success event of Lemma[11](https://arxiv.org/html/2608.12564#Thmtheorem11), which has probability at least1−δ1\-\\delta\. On this event the recalibrated scores have systematic error at mostmin⁡\(B,cf​log⁡\(2​K​T/δ\)/t\)\\min\\big\(B,\\,c\_\{f\}\\sqrt\{\\log\(2KT/\\delta\)/t\}\\big\)at steptt, since recalibration maps into the score range, so the error never exceedsBB, and the sharper bound of Lemma[11](https://arxiv.org/html/2608.12564#Thmtheorem11)applies once enough anchor pairs have accumulated\. Applying Lemma[10](https://arxiv.org/html/2608.12564#Thmtheorem10)to the recalibrated scores decomposes the world model estimate asuW​M\+dW​Mu\_\{WM\}\+d\_\{WM\}withVar⁡\(uW​M\)≤VW​M\\mathrm\{Var\}\(u\_\{WM\}\)\\leq V\_\{WM\}and‖dW​M‖≤2​M​min⁡\(B,cf​log⁡\(2​K​T/δ\)/t\)\\\|d\_\{WM\}\\\|\\leq 2M\\min\(B,c\_\{f\}\\sqrt\{\\log\(2KT/\\delta\)/t\}\), while the anchor estimate has the same mean\(1−1n\)∇J\(1\-\\tfrac\{1\}\{n\}\)\\nabla Jwith varianceVEV\_\{E\}, and the two are independent as they are computed on disjoint groups\. By Lemma[12](https://arxiv.org/html/2608.12564#Thmtheorem12), fusing them with inverse\-variance weights attains the harmonic variance, and since the weights sum to one the fused estimate keeps the common mean, so Lemma[9](https://arxiv.org/html/2608.12564#Thmtheorem9)applies withκt=1−1n\\kappa\_\{t\}=1\-\\tfrac\{1\}\{n\},V=\(1/VE\+1/VW​M\)−1V=\(1/V\_\{E\}\+1/V\_\{WM\}\)^\{\-1\}andDt=2​M​min⁡\(B,cf​log⁡\(2​K​T/δ\)/t\)D\_\{t\}=2M\\min\(B,c\_\{f\}\\sqrt\{\\log\(2KT/\\delta\)/t\}\), which already produces the geometric term and the variance term of the claim\.

*Part 2, the perturbation sum\.*What remains is the middle term of Lemma[9](https://arxiv.org/html/2608.12564#Thmtheorem9),

2​γ​∑t<T\(1−γ​μ4\)T−1−t​Dt2,Dt2≤4​M2​min⁡\(B2,cf2​log⁡\(2​K​T/δ\)t\)\.2\\gamma\\sum\_\{t<T\}\\Big\(1\-\\tfrac\{\\gamma\\mu\}\{4\}\\Big\)^\{T\-1\-t\}D\_\{t\}^\{2\},\\qquad D\_\{t\}^\{2\}\\;\\leq\\;4M^\{2\}\\min\\Big\(B^\{2\},\\;c\_\{f\}^\{2\}\\tfrac\{\\log\(2KT/\\delta\)\}\{t\}\\Big\)\.The two factors decay in opposite directions, the geometric weight is small for earlyttand the summandDt2D\_\{t\}^\{2\}is small for latett, so we split the sum atT/2T/2and use the stronger effect on each half\. Fort≥T/2t\\geq T/2the summand is small,Dt2≤8​M2​cf2​log⁡\(2​K​T/δ\)/TD\_\{t\}^\{2\}\\leq 8M^\{2\}c\_\{f\}^\{2\}\\log\(2KT/\\delta\)/T, and the geometric weights sum to at most4γ​μ\\tfrac\{4\}\{\\gamma\\mu\}, so this half contributes at most64​cf2​M2μ​log⁡\(2​K​T/δ\)/T\\tfrac\{64c\_\{f\}^\{2\}M^\{2\}\}\{\\mu\}\\log\(2KT/\\delta\)/T, the contracting bias term\. Fort<T/2t<T/2the weight is small,\(1−γ​μ4\)T−1−t≤\(1−γ​μ4\)T/2\(1\-\\tfrac\{\\gamma\\mu\}\{4\}\)^\{T\-1\-t\}\\leq\(1\-\\tfrac\{\\gamma\\mu\}\{4\}\)^\{T/2\}, and with the crude boundDt2≤4​M2​B2D\_\{t\}^\{2\}\\leq 4M^\{2\}B^\{2\}over at mostT/2T/2terms this half contributes at most4​γ​T​M2​B2​\(1−γ​μ4\)T/24\\gamma TM^\{2\}B^\{2\}\(1\-\\tfrac\{\\gamma\\mu\}\{4\}\)^\{T/2\}, which vanishes exponentially inTT\. Finally, bounding the same sum with the constantD≡2​M​BD\\equiv 2MBinstead gives8μ​M2​B2\\tfrac\{8\}\{\\mu\}M^\{2\}B^\{2\}, so the perturbation never exceeds the floor of Theorem[3](https://arxiv.org/html/2608.12564#Thmtheorem3)and decays asO~​\(M2​cf2/T\)\\widetilde\{O\}\(M^\{2\}c\_\{f\}^\{2\}/T\), which is the contracting form stated in Section[4\.3](https://arxiv.org/html/2608.12564#S4.SS3)\. ∎

###### Corollary 7\.

Assume in addition that the recalibration map is non\-expansive, so that the post\-recalibration noise satisfiesη2≤σ2\\eta^\{2\}\\leq\\sigma^\{2\}\. Then, comparing Theorem[4](https://arxiv.org/html/2608.12564#Thmtheorem4)with Theorem[3](https://arxiv.org/html/2608.12564#Thmtheorem3)term by term, the leading terms coincide, there exists a finiteT1T\_\{1\}such that for allT≥T1T\\geq T\_\{1\}the bias term of WMRL is strictly smaller than the floor32​M2μ​B2\\tfrac\{32M^\{2\}\}\{\\mu\}B^\{2\}and converges to zero asT→∞T\\to\\infty, and the variance term of WMRL is smaller by the factor1\+VW​M/VE\>11\+V\_\{WM\}/V\_\{E\}\>1\.

###### Proof\.

The leading terms are identical by construction\. For the bias, the term of Theorem[4](https://arxiv.org/html/2608.12564#Thmtheorem4)falls below32​M2μ​B2\\tfrac\{32M^\{2\}\}\{\\mu\}B^\{2\}as soon aslog⁡\(2​K​T/δ\)/T≤B2/\(2​cf2\)\\log\(2KT/\\delta\)/T\\leq B^\{2\}/\(2c\_\{f\}^\{2\}\)and the exponentially decaying remainder is below the same threshold, both of which hold for allTTbeyond some finiteT1T\_\{1\}, and the term itself converges to zero\. For the variance, Lemma[12](https://arxiv.org/html/2608.12564#Thmtheorem12)writes the fused variance asVW​M/\(1\+VW​M/VE\)V\_\{WM\}/\(1\+V\_\{WM\}/V\_\{E\}\), and non\-expansiveness givesη2≤σ2\\eta^\{2\}\\leq\\sigma^\{2\}, so theVW​MV\_\{WM\}of Theorem[4](https://arxiv.org/html/2608.12564#Thmtheorem4), formed with the post\-recalibration noiseη\\eta, is no larger than theVW​MV\_\{WM\}of Theorem[3](https://arxiv.org/html/2608.12564#Thmtheorem3), formed withσ\\sigma\. ∎

### B\.4Auxiliary lemmas

This subsection collects the four lemmas used above\. Before each statement we recall why the lemma is needed and where it is consumed\.

The first lemma is the engine behind both theorems\. It answers how far gradient ascent can still progress when the estimate carries a bounded perturbation and extra variance, and every convergence statement of this paper is an instance of it, Proposition[5](https://arxiv.org/html/2608.12564#Thmtheorem5)withDt≡0D\_\{t\}\\equiv 0, Theorem[3](https://arxiv.org/html/2608.12564#Thmtheorem3)with\(V,D\)=\(VW​M,2​M​B\)\(V,D\)=\(V\_\{WM\},\\,2MB\), and Theorem[4](https://arxiv.org/html/2608.12564#Thmtheorem4)with the harmonic variance and a shrinkingDtD\_\{t\}\. The middle term of its bound is the weighted telescoping sum that the two theorems then control in different ways\.

###### Lemma 9\(Perturbed ascent under gradient domination\)\.

Letθt\+1=θt\+γ​g^t\\theta\_\{t\+1\}=\\theta\_\{t\}\+\\gamma\\hat\{g\}\_\{t\}withγ≤1/\(8​L\)\\gamma\\leq 1/\(8L\)\. Suppose that, conditioned onθt\\theta\_\{t\}, the estimate decomposes asg^t=ut\+dt\\hat\{g\}\_\{t\}=u\_\{t\}\+d\_\{t\}, where𝔼\[ut∣θt\]=κt∇J\(θt\)\\mathbb\{E\}\[u\_\{t\}\\mid\\theta\_\{t\}\]=\\kappa\_\{t\}\\nabla J\(\\theta\_\{t\}\)withκt∈\[12,1\]\\kappa\_\{t\}\\in\[\\tfrac\{1\}\{2\},1\],Var⁡\(ut∣θt\)≤V\\mathrm\{Var\}\(u\_\{t\}\\mid\\theta\_\{t\}\)\\leq V, and‖dt‖≤Dt\\\|d\_\{t\}\\\|\\leq D\_\{t\}almost surely\. Then

J⋆−𝔼⁡\[J⁡\(θT\)\]≤\(1−γ​μ4\)T​Δ0\+2​γ​∑t=0T−1\(1−γ​μ4\)T−1−t​Dt2\+4​L​γμ​V\.J^\{\\star\}\-\\mathbb\{E\}\[J\(\\theta\_\{T\}\)\]\\;\\leq\\;\\Big\(1\-\\tfrac\{\\gamma\\mu\}\{4\}\\Big\)^\{T\}\\Delta\_\{0\}\\;\+\\;2\\gamma\\sum\_\{t=0\}^\{T\-1\}\\Big\(1\-\\tfrac\{\\gamma\\mu\}\{4\}\\Big\)^\{T\-1\-t\}D\_\{t\}^\{2\}\\;\+\\;\\frac\{4L\\gamma\}\{\\mu\}\\,V\.In particular, ifDt≡DD\_\{t\}\\equiv D, the middle term is at most8μ​D2\\tfrac\{8\}\{\\mu\}D^\{2\}\.

###### Proof\.

The plan mirrors the three steps of Proposition[5](https://arxiv.org/html/2608.12564#Thmtheorem5)\. We lower\-bound the useful progress⟨∇,g^t⟩\\langle\\nabla,\\hat\{g\}\_\{t\}\\rangle, upper\-bound the harmful second moment‖g^t‖2\\\|\\hat\{g\}\_\{t\}\\\|^\{2\}, and combine the two through smoothness into a recursion forδt\\delta\_\{t\}\. Write∇:=∇J​\(θt\)\\nabla:=\\nabla J\(\\theta\_\{t\}\)and let𝔼t\\mathbb\{E\}\_\{t\}denote expectation conditioned onθt\\theta\_\{t\}; under𝔼t\\mathbb\{E\}\_\{t\}the quantitiesJ⁡\(θt\)J\(\\theta\_\{t\}\)and∇\\nablaare fixed, so only the terms containingg^t\\hat\{g\}\_\{t\}are affected\. ByLL\-smoothness,J⁡\(θt\+1\)≥J⁡\(θt\)\+γ⁡⟨∇,g^t⟩−L​γ22​‖g^t‖2J\(\\theta\_\{t\+1\}\)\\geq J\(\\theta\_\{t\}\)\+\\gamma\\langle\\nabla,\\hat\{g\}\_\{t\}\\rangle\-\\frac\{L\\gamma^\{2\}\}\{2\}\\\|\\hat\{g\}\_\{t\}\\\|^\{2\}\.

*Step 1, the progress term\.*Splittingg^t=ut\+dt\\hat\{g\}\_\{t\}=u\_\{t\}\+d\_\{t\}, the inner product separates into signal and contamination,𝔼t​⟨∇,g^t⟩=κt​‖∇‖2\+⟨∇,𝔼t​dt⟩\\mathbb\{E\}\_\{t\}\\langle\\nabla,\\hat\{g\}\_\{t\}\\rangle=\\kappa\_\{t\}\\\|\\nabla\\\|^\{2\}\+\\langle\\nabla,\\mathbb\{E\}\_\{t\}d\_\{t\}\\rangle\. The contamination can only be bounded by Cauchy–Schwarz,⟨∇,𝔼t​dt⟩≥−‖∇‖​Dt\\langle\\nabla,\\mathbb\{E\}\_\{t\}d\_\{t\}\\rangle\\geq\-\\\|\\nabla\\\|D\_\{t\}, and a product of two different quantities is inconvenient in a recursion, so we separate it with Young’s inequalitya​b≤a2/4\+b2ab\\leq a^\{2\}/4\+b^\{2\},

𝔼t​⟨∇,g^t⟩≥κt​‖∇‖2−‖∇‖​Dt≥\(κt−14\)​‖∇‖2−Dt2\.\\mathbb\{E\}\_\{t\}\\langle\\nabla,\\hat\{g\}\_\{t\}\\rangle\\geq\\kappa\_\{t\}\\\|\\nabla\\\|^\{2\}\-\\\|\\nabla\\\|D\_\{t\}\\geq\\big\(\\kappa\_\{t\}\-\\tfrac\{1\}\{4\}\\big\)\\\|\\nabla\\\|^\{2\}\-D\_\{t\}^\{2\}\.The choice of14\\tfrac\{1\}\{4\}sacrifices a quarter of the signal to turn the cross term into a pureDt2D\_\{t\}^\{2\}\.

*Step 2, the second moment\.*We control mean and fluctuation separately\. The mean obeys‖𝔼t​g^t‖≤κt​‖∇‖\+Dt\\\|\\mathbb\{E\}\_\{t\}\\hat\{g\}\_\{t\}\\\|\\leq\\kappa\_\{t\}\\\|\\nabla\\\|\+D\_\{t\}, so\(a\+b\)2≤2​a2\+2​b2\(a\+b\)^\{2\}\\leq 2a^\{2\}\+2b^\{2\}gives‖𝔼t​g^t‖2≤2​‖∇‖2\+2​Dt2\\\|\\mathbb\{E\}\_\{t\}\\hat\{g\}\_\{t\}\\\|^\{2\}\\leq 2\\\|\\nabla\\\|^\{2\}\+2D\_\{t\}^\{2\}, and the fluctuation obeysVar⁡\(g^t\)≤2​Var​\(ut\)\+2​Var​\(dt\)≤2​V\+2​Dt2\\mathrm\{Var\}\(\\hat\{g\}\_\{t\}\)\\leq 2\\,\\mathrm\{Var\}\(u\_\{t\}\)\+2\\,\\mathrm\{Var\}\(d\_\{t\}\)\\leq 2V\+2D\_\{t\}^\{2\}\. Adding the two,𝔼t​‖g^t‖2≤2​‖∇‖2\+4​Dt2\+2​V\\mathbb\{E\}\_\{t\}\\\|\\hat\{g\}\_\{t\}\\\|^\{2\}\\leq 2\\\|\\nabla\\\|^\{2\}\+4D\_\{t\}^\{2\}\+2V\.

*Step 3, combine and unroll\.*Substituting the two bounds into the smoothness inequality, and simplifying withκt≥12\\kappa\_\{t\}\\geq\\tfrac\{1\}\{2\}andL​γ≤18L\\gamma\\leq\\tfrac\{1\}\{8\},

𝔼t​J​\(θt\+1\)\\displaystyle\\mathbb\{E\}\_\{t\}J\(\\theta\_\{t\+1\}\)≥J⁡\(θt\)\+γ⁡\(κt−14−L​γ\)​‖∇‖2−γ​Dt2​\(1\+2​L​γ\)−L​γ2​V\\displaystyle\\geq J\(\\theta\_\{t\}\)\+\\gamma\\big\(\\kappa\_\{t\}\-\\tfrac\{1\}\{4\}\-L\\gamma\\big\)\\\|\\nabla\\\|^\{2\}\-\\gamma D\_\{t\}^\{2\}\(1\+2L\\gamma\)\-L\\gamma^\{2\}V≥J⁡\(θt\)\+γ8​‖∇‖2−2​γ​Dt2−L​γ2​V\.\\displaystyle\\geq J\(\\theta\_\{t\}\)\+\\tfrac\{\\gamma\}\{8\}\\\|\\nabla\\\|^\{2\}\-2\\gamma D\_\{t\}^\{2\}\-L\\gamma^\{2\}V\.Gradient domination replaces the gradient by the gap,‖∇‖2≥2​μ​\(J⋆−J⁡\(θt\)\)\\\|\\nabla\\\|^\{2\}\\geq 2\\mu\\,\(J^\{\\star\}\-J\(\\theta\_\{t\}\)\), and taking total expectations through the tower property turns the display into a recursion on the deterministic sequenceδt:=J⋆−𝔼⁡\[J⁡\(θt\)\]\\delta\_\{t\}:=J^\{\\star\}\-\\mathbb\{E\}\[J\(\\theta\_\{t\}\)\],

δt\+1≤\(1−γ​μ4\)​δt\+2​γ​Dt2\+L​γ2​V\.\\delta\_\{t\+1\}\\;\\leq\\;\\Big\(1\-\\tfrac\{\\gamma\\mu\}\{4\}\\Big\)\\,\\delta\_\{t\}\+2\\gamma D\_\{t\}^\{2\}\+L\\gamma^\{2\}V\.Multiplying the step\-ttinequality by\(1−γ​μ4\)T−1−t\(1\-\\tfrac\{\\gamma\\mu\}\{4\}\)^\{T\-1\-t\}and summing overt=0,…,T−1t=0,\\dots,T\-1, the intermediateδt\\delta\_\{t\}cancel telescopically and the claim follows\. For constantDDthe geometric sum∑k≥0\(1−γ​μ4\)k=4γ​μ\\sum\_\{k\\geq 0\}\(1\-\\tfrac\{\\gamma\\mu\}\{4\}\)^\{k\}=\\tfrac\{4\}\{\\gamma\\mu\}bounds the middle term by8μ​D2\\tfrac\{8\}\{\\mu\}D^\{2\}and the last term by4​L​γμ​V\\tfrac\{4L\\gamma\}\{\\mu\}V\. ∎

The second lemma is what allows Lemma[9](https://arxiv.org/html/2608.12564#Thmtheorem9)to be fed with world model rewards\. It converts the reward\-level error model of Equation \([3](https://arxiv.org/html/2608.12564#S3.E3)\) into exactly the three quantities that Lemma[9](https://arxiv.org/html/2608.12564#Thmtheorem9)consumes, the mean of the clean part, its variance, and the almost\-sure size of the perturbation, and it also produces the variance relationVW​M=\(1\+c​σ2\)​VEV\_\{WM\}=\(1\+c\\,\\sigma^\{2\}\)V\_\{E\}used in Section[3\.3](https://arxiv.org/html/2608.12564#S3.SS3)\.

###### Lemma 10\(From reward error to gradient error\)\.

Letg^k\\hat\{g\}\_\{k\}be computed with true scores andg^kW​M\\hat\{g\}\_\{k\}^\{WM\}with predicted scores\. Under Assumptions[2](https://arxiv.org/html/2608.12564#Thmtheorem2)and[6](https://arxiv.org/html/2608.12564#Thmtheorem6),g^kW​M=g^k\+gb\+gξ\\hat\{g\}\_\{k\}^\{WM\}=\\hat\{g\}\_\{k\}\+g\_\{b\}\+g\_\{\\xi\}, where

1. \(i\)𝔼\[g^k\]=\(1−1n\)∇J\(θ\)\\mathbb\{E\}\[\\hat\{g\}\_\{k\}\]=\\big\(1\-\\tfrac\{1\}\{n\}\\big\)\\nabla J\(\\theta\)andVE=Var⁡\(g^k\)≤M2V\_\{E\}=\\mathrm\{Var\}\(\\hat\{g\}\_\{k\}\)\\leq M^\{2\};
2. \(ii\)gb=1n​∑i\(bi−b¯\)​sig\_\{b\}=\\frac\{1\}\{n\}\\sum\_\{i\}\(b\_\{i\}\-\\bar\{b\}\)\\,s\_\{i\}satisfies‖gb‖≤2​M​B\\\|g\_\{b\}\\\|\\leq 2MBalmost surely, and vanishes wheneverbbis constant within the group;
3. \(iii\)gξg\_\{\\xi\}has zero conditional mean, is uncorrelated withg^k\\hat\{g\}\_\{k\}, and satisfies𝔼​‖gξ‖2≤4​M2​σ2/n\\mathbb\{E\}\\\|g\_\{\\xi\}\\\|^\{2\}\\leq 4M^\{2\}\\sigma^\{2\}/n, so thatVW​M=VE\+4​M2​σ2/n=\(1\+c​σ2\)​VEV\_\{WM\}=V\_\{E\}\+4M^\{2\}\\sigma^\{2\}/n=\(1\+c\\,\\sigma^\{2\}\)\\,V\_\{E\}withc=4​M2/\(n​VE\)c=4M^\{2\}/\(nV\_\{E\}\), the relation used in Section[3\.3](https://arxiv.org/html/2608.12564#S3.SS3)\.

###### Proof\.

Recall from Equation \([5](https://arxiv.org/html/2608.12564#S3.E5)\) thatg^k=1n​∑iAi​si\\hat\{g\}\_\{k\}=\\frac\{1\}\{n\}\\sum\_\{i\}A\_\{i\}s\_\{i\}, where the advantageAi=ri−1n​∑jrjA\_\{i\}=r\_\{i\}\-\\frac\{1\}\{n\}\\sum\_\{j\}r\_\{j\}is a linear function of the scores of its group\. Abbreviateri:=r⁡\(τi\)r\_\{i\}:=r\(\\tau\_\{i\}\),bi:=b⁡\(τi\)b\_\{i\}:=b\(\\tau\_\{i\}\), andξi:=ξ⁡\(τi\)\\xi\_\{i\}:=\\xi\(\\tau\_\{i\}\)\.

*The decomposition is pure algebra\.*Replacing every true score by the predicted one substitutes into the same linear formula, so withr^i=ri\+bi\+ξi\\hat\{r\}\_\{i\}=r\_\{i\}\+b\_\{i\}\+\\xi\_\{i\}the new advantage is

A^i=r^i−1n​∑jr^j=Ai\+\(bi−b¯\)\+\(ξi−ξ¯\),\\hat\{A\}\_\{i\}\\;=\\;\\hat\{r\}\_\{i\}\-\\frac\{1\}\{n\}\\sum\_\{j\}\\hat\{r\}\_\{j\}\\;=\\;A\_\{i\}\+\(b\_\{i\}\-\\bar\{b\}\)\+\(\\xi\_\{i\}\-\\bar\{\\xi\}\),withb¯\\bar\{b\},ξ¯\\bar\{\\xi\}the group means\. Multiplying bysi/ns\_\{i\}/nand summing overiigivesg^kW​M=g^k\+gb\+gξ\\hat\{g\}\_\{k\}^\{WM\}=\\hat\{g\}\_\{k\}\+g\_\{b\}\+g\_\{\\xi\}withgb:=1n​∑i\(bi−b¯\)​sig\_\{b\}:=\\frac\{1\}\{n\}\\sum\_\{i\}\(b\_\{i\}\-\\bar\{b\}\)s\_\{i\}andgξ:=1n​∑i\(ξi−ξ¯\)​sig\_\{\\xi\}:=\\frac\{1\}\{n\}\\sum\_\{i\}\(\\xi\_\{i\}\-\\bar\{\\xi\}\)\\,s\_\{i\}; no probabilistic argument is involved, the three parts simply collect the score, bias, and noise contributions\. It remains to control each part in the form that Lemma[9](https://arxiv.org/html/2608.12564#Thmtheorem9)consumes, the mean and variance of the clean part, the almost\-sure size of the bias part, and the second moment of the noise part\.

\(i\)*The clean part\.*Expanding the advantage,𝔼⁡\[Ai​si\]=𝔼⁡\[ri​si\]−1n​∑j𝔼⁡\[rj​si\]\\mathbb\{E\}\[A\_\{i\}s\_\{i\}\]=\\mathbb\{E\}\[r\_\{i\}s\_\{i\}\]\-\\frac\{1\}\{n\}\\sum\_\{j\}\\mathbb\{E\}\[r\_\{j\}s\_\{i\}\]\. Forj≠ij\\neq ithe two trajectories are independent given the task, so the expectation factorizes, and𝔼⁡\[si\]=𝔼⁡\[∇log⁡πθ\]=0\\mathbb\{E\}\[s\_\{i\}\]=\\mathbb\{E\}\[\\nabla\\log\\pi\_\{\\theta\}\]=0kills the term; thej=ij=iterm contributes1n​𝔼​\[ri​si\]\\frac\{1\}\{n\}\\mathbb\{E\}\[r\_\{i\}s\_\{i\}\]\. Summing overii,𝔼\[g^k\]=\(1−1n\)𝔼\[r\(τ\)∇logπθ\(τ\)\]=\(1−1n\)∇J\(θ\)\\mathbb\{E\}\[\\hat\{g\}\_\{k\}\]=\(1\-\\tfrac\{1\}\{n\}\)\\,\\mathbb\{E\}\[r\(\\tau\)\\nabla\\log\\pi\_\{\\theta\}\(\\tau\)\]=\(1\-\\tfrac\{1\}\{n\}\)\\nabla J\(\\theta\), where the last step is the policy gradient identity\. For the variance,ri∈\[0,1\]r\_\{i\}\\in\[0,1\]forces\|Ai\|≤1\|A\_\{i\}\|\\leq 1, hence‖g^k‖≤1n​∑i‖si‖≤M\\\|\\hat\{g\}\_\{k\}\\\|\\leq\\frac\{1\}\{n\}\\sum\_\{i\}\\\|s\_\{i\}\\\|\\leq M, and a vector bounded byMMhas variance at most𝔼​‖g^k‖2≤M2\\mathbb\{E\}\\\|\\hat\{g\}\_\{k\}\\\|^\{2\}\\leq M^\{2\}, soVE≤M2V\_\{E\}\\leq M^\{2\}\.

\(ii\)*The bias part\.*It is deterministic given the trajectories, so what Lemma[9](https://arxiv.org/html/2608.12564#Thmtheorem9)needs is an almost\-sure bound\. Since\|bi\|≤B\|b\_\{i\}\|\\leq B, the centered value obeys\|bi−b¯\|≤2​B\|b\_\{i\}\-\\bar\{b\}\|\\leq 2B, and therefore‖gb‖≤1n​∑i\|bi−b¯\|​‖si‖≤2​M​B\\\|g\_\{b\}\\\|\\leq\\frac\{1\}\{n\}\\sum\_\{i\}\|b\_\{i\}\-\\bar\{b\}\|\\,\\\|s\_\{i\}\\\|\\leq 2MB\. Ifbbis constant within the group, the centering removes it entirely,bi−b¯=0b\_\{i\}\-\\bar\{b\}=0, which is why only the within\-group variation of the bias ever reaches the gradient\.

\(iii\)*The noise part\.*It should act as extra variance, so we verify that it is conditionally zero\-mean and compute its second moment\. Rewritegξ=1n​∑iξi​\(si−s¯\)g\_\{\\xi\}=\\frac\{1\}\{n\}\\sum\_\{i\}\\xi\_\{i\}\(s\_\{i\}\-\\bar\{s\}\)withs¯:=1n​∑jsj\\bar\{s\}:=\\frac\{1\}\{n\}\\sum\_\{j\}s\_\{j\}\. Conditioned on the trajectories theξi\\xi\_\{i\}are independent and zero\-mean, so𝔼\[gξ∣τ1:n\]=0\\mathbb\{E\}\[g\_\{\\xi\}\\mid\\tau\_\{1:n\}\]=0and all cross terms of𝔼​‖gξ‖2\\mathbb\{E\}\\\|g\_\{\\xi\}\\\|^\{2\}vanish, leaving

𝔼​‖gξ‖2=1n2​∑i𝔼⁡\[𝔼⁡\[ξi2∣τ\]​‖si−s¯‖2\]≤σ2n2⋅n⋅\(2​M\)2=4​M2​σ2n\.\\mathbb\{E\}\\\|g\_\{\\xi\}\\\|^\{2\}=\\frac\{1\}\{n^\{2\}\}\\sum\_\{i\}\\mathbb\{E\}\\big\[\\mathbb\{E\}\[\\xi\_\{i\}^\{2\}\\mid\\tau\]\\,\\\|s\_\{i\}\-\\bar\{s\}\\\|^\{2\}\\big\]\\leq\\frac\{\\sigma^\{2\}\}\{n^\{2\}\}\\cdot n\\cdot\(2M\)^\{2\}=\\frac\{4M^\{2\}\\sigma^\{2\}\}\{n\}\.The same conditional zero mean makesgξg\_\{\\xi\}uncorrelated withg^k\\hat\{g\}\_\{k\}, so the two variances add,VW​M=VE\+4​M2​σ2/n=\(1\+c​σ2\)​VEV\_\{WM\}=V\_\{E\}\+4M^\{2\}\\sigma^\{2\}/n=\(1\+c\\,\\sigma^\{2\}\)\\,V\_\{E\}withc=4​M2/\(n​VE\)c=4M^\{2\}/\(nV\_\{E\}\), the stated relation\. ∎

The third lemma supplies the shrinking perturbation boundDtD\_\{t\}used in the proof of Theorem[4](https://arxiv.org/html/2608.12564#Thmtheorem4)\. It quantifies how fast the isotonic recalibration of Equation \([4](https://arxiv.org/html/2608.12564#S3.E4)\) learns the distortionϕ\\phifrom the accumulating anchor pairs, and its1/t1/\\sqrt\{t\}rate is what ultimately becomes the contraction of the bias term\.

###### Lemma 11\(Recalibration error\)\.

Under Assumption[6](https://arxiv.org/html/2608.12564#Thmtheorem6), with probability at least1−δ1\-\\deltasimultaneously for all1≤t≤T1\\leq t\\leq T, the recalibration mapf^t\\hat\{f\}\_\{t\}is well defined once enough anchor pairs have accumulated, and the systematic error of the recalibrated score obeys

\|𝔼⁡\[f^t​\(r^\)∣τ\]−r⁡\(τ\)\|≤cf​log⁡\(2​K​T/δ\)/t,\\big\|\\mathbb\{E\}\[\\hat\{f\}\_\{t\}\(\\hat\{r\}\)\\mid\\tau\]\-r\(\\tau\)\\big\|\\;\\leq\\;c\_\{f\}\\sqrt\{\\log\(2KT/\\delta\)/t\},whereKKis the number of distinct scores the benchmark produces and the constantcfc\_\{f\}is given in Equation \([6](https://arxiv.org/html/2608.12564#A2.E6)\) of the proof\.

###### Proof\.

We first fix notation for the scores our benchmarks produce\. The leaderboard percentile is computed against a finite leaderboard and the VLA reward is binary, so the true scores take finitely many valuesℓ1<⋯<ℓK\\ell\_\{1\}<\\dots<\\ell\_\{K\}, and the recalibration of Equation \([4](https://arxiv.org/html/2608.12564#S3.E4)\) fits one vertex per level\. WriteΔℓ:=maxk⁡\(ℓk\+1−ℓk\)\\Delta\_\{\\ell\}:=\\max\_\{k\}\(\\ell\_\{k\+1\}\-\\ell\_\{k\}\)for the largest gap between adjacent levels,Δϕ:=mink⁡\(ϕ⁡\(ℓk\+1\)−ϕ⁡\(ℓk\)\)\\Delta\_\{\\phi\}:=\\min\_\{k\}\\big\(\\phi\(\\ell\_\{k\+1\}\)\-\\phi\(\\ell\_\{k\}\)\\big\)for the smallest separation the distortion leaves between them,qqfor the smallest fraction of the anchor pairs that any level receives, andaafor the number of new pairs per step\. The rate below is informative whenΔϕ\>0\\Delta\_\{\\phi\}\>0andq\>0q\>0, that is, when the distortion collapses no two levels onto one another and no level is starved of anchor pairs, which is the discrete\-level analogue of the arm gaps of bandit analysis\[[2](https://arxiv.org/html/2608.12564#bib.bib64)\]and of the coverage conditions of off\-policy evaluation\[[11](https://arxiv.org/html/2608.12564#bib.bib65)\], and is what turns isotonic regression from a worst\-case problem into one with a parametric rate\[[63](https://arxiv.org/html/2608.12564#bib.bib66)\]\. In terms of these quantities the constant of the statement is

cf:=2​ΔℓΔϕ​12​q​a\.c\_\{f\}\\;:=\\;\\frac\{2\\Delta\_\{\\ell\}\}\{\\Delta\_\{\\phi\}\}\\sqrt\{\\frac\{1\}\{2qa\}\}\.\(6\)
The proof then has three stages that mirror the recalibration pipeline\. We first show that the empirical mean of the predictions at each score level concentrates, then that the isotonic projection can only inherit this accuracy, and finally that the interpolated map turns accurate vertices into accurate calibrated scores\.

*Stage 1, level means concentrate\.*Coverage is what makes this stage possible, as a level that received no pairs could never be calibrated\. At steptteach level holds at leastq​a​tqatpairs, each of the formr^=ϕ⁡\(ℓk\)\+ξ\\hat\{r\}=\\phi\(\\ell\_\{k\}\)\+\\xiwithξ\\xizero\-mean and\|ξ\|≤1\|\\xi\|\\leq 1, so Hoeffding’s inequality givesPr⁡\(\|μ^k−ϕ⁡\(ℓk\)\|≥s\)≤2​e−2​q​a​t​s2\\Pr\(\|\\hat\{\\mu\}\_\{k\}\-\\phi\(\\ell\_\{k\}\)\|\\geq s\)\\leq 2e^\{\-2qats^\{2\}\}\. The guarantee is needed simultaneously for every level and every step, so we take a union bound over theKKlevels andTTsteps, and the choices=e⁡\(t\):=log⁡\(2​K​T/δ\)/\(2​q​a​t\)s=e\(t\):=\\sqrt\{\\log\(2KT/\\delta\)/\(2qat\)\}keeps the total failure probability belowδ\\delta\.

*Stage 2, the projection does not hurt\.*The isotonic fit replaces the raw meansμ^k\\hat\{\\mu\}\_\{k\}by the closest monotone sequenceϕ^k\\hat\{\\phi\}\_\{k\}, and we must check that this projection cannot push an estimate away from the truth\. The min–max representationϕ^k=minv≥k⁡maxu≤k⁡avg⁡\(μ^u,…,μ^v\)\\hat\{\\phi\}\_\{k\}=\\min\_\{v\\geq k\}\\max\_\{u\\leq k\}\\mathrm\{avg\}\(\\hat\{\\mu\}\_\{u\},\\dots,\\hat\{\\mu\}\_\{v\}\)expresses every fitted value as an average of raw means, each withine⁡\(t\)e\(t\)of its true value\. For anyv≥kv\\geq k, monotonicity ofϕ\\phigivesmaxu≤k⁡avg⁡\[u,v\]≥avg⁡\[k,v\]≥avg⁡\(ϕ⁡\(ℓk\),…,ϕ⁡\(ℓv\)\)−e⁡\(t\)≥ϕ⁡\(ℓk\)−e⁡\(t\)\\max\_\{u\\leq k\}\\mathrm\{avg\}\[u,v\]\\geq\\mathrm\{avg\}\[k,v\]\\geq\\mathrm\{avg\}\(\\phi\(\\ell\_\{k\}\),\\dots,\\phi\(\\ell\_\{v\}\)\)\-e\(t\)\\geq\\phi\(\\ell\_\{k\}\)\-e\(t\), and taking the minimum overvvpreserves the bound; choosingv=kv=kgives symmetricallyϕ^k≤ϕ⁡\(ℓk\)\+e⁡\(t\)\\hat\{\\phi\}\_\{k\}\\leq\\phi\(\\ell\_\{k\}\)\+e\(t\)\. So the fitted vertices are as accurate as the raw means\.

*Stage 3, from vertices to calibrated scores\.*Separation is what makes this stage possible, as levels the distortion has collapsed onto one another could not be told apart by any monotone fit\. Whene⁡\(t\)≤Δϕ/4e\(t\)\\leq\\Delta\_\{\\phi\}/4, consecutive fitted vertices stay separated,ϕ^k\+1−ϕ^k≥Δϕ−2​e​\(t\)≥Δϕ/2\\hat\{\\phi\}\_\{k\+1\}\-\\hat\{\\phi\}\_\{k\}\\geq\\Delta\_\{\\phi\}\-2e\(t\)\\geq\\Delta\_\{\\phi\}/2, so the piecewise\-linear interpolationf^t\\hat\{f\}\_\{t\}is well defined and its Lipschitz constant is at mostΔℓ/\(Δϕ/2\)=2​Δℓ/Δϕ\\Delta\_\{\\ell\}/\(\\Delta\_\{\\phi\}/2\)=2\\Delta\_\{\\ell\}/\\Delta\_\{\\phi\}\. A trajectory with true scoreℓk\\ell\_\{k\}produces predictions with conditional meanϕ⁡\(ℓk\)\\phi\(\\ell\_\{k\}\), andf^t\\hat\{f\}\_\{t\}maps the fitted vertexϕ^k\\hat\{\\phi\}\_\{k\}exactly toℓk\\ell\_\{k\}, so the systematic error of the calibrated score is at most the Lipschitz constant times the vertex error,\|f^t​\(ϕ⁡\(ℓk\)\)−ℓk\|=\|f^t​\(ϕ⁡\(ℓk\)\)−f^t​\(ϕ^k\)\|≤2​ΔℓΔϕ​e​\(t\)\|\\hat\{f\}\_\{t\}\(\\phi\(\\ell\_\{k\}\)\)\-\\ell\_\{k\}\|=\|\\hat\{f\}\_\{t\}\(\\phi\(\\ell\_\{k\}\)\)\-\\hat\{f\}\_\{t\}\(\\hat\{\\phi\}\_\{k\}\)\|\\leq\\tfrac\{2\\Delta\_\{\\ell\}\}\{\\Delta\_\{\\phi\}\}e\(t\), which is the claim\. ∎

The last lemma justifies the fusion rule of Equation \([5](https://arxiv.org/html/2608.12564#S3.E5)\) and supplies the varianceVVused in the proof of Theorem[4](https://arxiv.org/html/2608.12564#Thmtheorem4)\. Its final claim, that the fused variance is flat around the optimal weight, is what Remark[8](https://arxiv.org/html/2608.12564#Thmtheorem8)invokes to argue robustness to the estimated ratio\.

###### Lemma 12\(Optimal linear fusion\)\.

Letg1g\_\{1\}andg2g\_\{2\}be independent unbiased estimates of the same quantity with variancesV1V\_\{1\}andV2V\_\{2\}\. Among all their unbiased linear combinations, that is, with weights summing to one, the variance is minimized by weights proportional toV1−1V\_\{1\}^\{\-1\}andV2−1V\_\{2\}^\{\-1\}, and the minimal value is

\(1V1\+1V2\)−1<min⁡\(V1,V2\)\.\\Big\(\\frac\{1\}\{V\_\{1\}\}\+\\frac\{1\}\{V\_\{2\}\}\\Big\)^\{\-1\}\\;<\\;\\min\(V\_\{1\},\\,V\_\{2\}\)\.Moreover, for any weightα\\alphaong1g\_\{1\},Var⁡\(g^α\)=Var⁡\(g^α⋆\)\+\(V1\+V2\)​\(α−α⋆\)2\\mathrm\{Var\}\(\\hat\{g\}\_\{\\alpha\}\)=\\mathrm\{Var\}\(\\hat\{g\}\_\{\\alpha^\{\\star\}\}\)\+\(V\_\{1\}\+V\_\{2\}\)\(\\alpha\-\\alpha^\{\\star\}\)^\{2\}, so a misspecified weight costs only a second\-order term\.

###### Proof\.

Write the combination with a single weight,g^α=α​g1\+\(1−α\)​g2\\hat\{g\}\_\{\\alpha\}=\\alpha g\_\{1\}\+\(1\-\\alpha\)g\_\{2\}, which is unbiased for everyα\\alpha, so the choices differ only in variance\. By independence,Var⁡\(g^α\)=α2​V1\+\(1−α\)2​V2\\mathrm\{Var\}\(\\hat\{g\}\_\{\\alpha\}\)=\\alpha^\{2\}V\_\{1\}\+\(1\-\\alpha\)^\{2\}V\_\{2\}, a strictly convex quadratic inα\\alpha, so the unique minimizer is found by setting the derivative to zero,α⋆=V2/\(V1\+V2\)\\alpha^\{\\star\}=V\_\{2\}/\(V\_\{1\}\+V\_\{2\}\), that is, weights proportional to the inverse variances\. Substitutingα⋆\\alpha^\{\\star\}back givesV1​V2/\(V1\+V2\)V\_\{1\}V\_\{2\}/\(V\_\{1\}\+V\_\{2\}\), which equals the stated harmonic form and is strictly smaller than each ofV1V\_\{1\}andV2V\_\{2\}because both precisions are positive\. Finally, expanding the quadratic around its minimizer givesVar⁡\(g^α\)=Var⁡\(g^α⋆\)\+\(V1\+V2\)​\(α−α⋆\)2\\mathrm\{Var\}\(\\hat\{g\}\_\{\\alpha\}\)=\\mathrm\{Var\}\(\\hat\{g\}\_\{\\alpha^\{\\star\}\}\)\+\(V\_\{1\}\+V\_\{2\}\)\(\\alpha\-\\alpha^\{\\star\}\)^\{2\}, since its second derivative is2​\(V1\+V2\)2\(V\_\{1\}\+V\_\{2\}\); this is the second\-order insensitivity invoked in Remark[8](https://arxiv.org/html/2608.12564#Thmtheorem8)\. ∎

## Appendix CChoice of the Task Pool

Section[5\.1](https://arxiv.org/html/2608.12564#S5.SS1)builds the task pool from MLE\-Dojo rather than from MLE\-Bench directly, for three reasons\.

First, part of the original pool is no longer usable\. Two of the underlying Kaggle competitions,detecting\-insults\-in\-social\-commentaryandthe\-icml\-2013\-whale\-challenge\-right\-whale\-redux, have been permanently closed, and their official data downloads and submission entries are gone\.

Second, the medal\-based metric fails to measure model ability\. MLE\-Bench scores an agent by the number of gold, silver, and bronze medals\. At the model scales we study, an agent earns medals on only one or two fixed competitions, so the medal count stays flat no matter how much the agent improves on the rest of the pool\. The counts are also skewed, with more golds than silvers or bronzes in published evaluations\[[35](https://arxiv.org/html/2608.12564#bib.bib7)\], because a handful of tasks admit near\-perfect accuracy and hand out gold medals while the remaining tasks hand out nothing\. The ranking against human competitors is what reflects ability, so our protocol reports the leaderboard percentile, which credits improvement on every task\.

Third, contamination\. The competitions and many of their winning solutions predate the pretraining corpora of current models, a risk acknowledged both in the original release\[[10](https://arxiv.org/html/2608.12564#bib.bib1)\]and in the accompanying OpenAI blog post\[[38](https://arxiv.org/html/2608.12564#bib.bib2)\], and the public evaluation has not been actively maintained since\.

MLE\-Dojo wraps a superset of the same competitions in an interactive environment with live grading, which avoids the dead entries, exposes leaderboard percentile directly, and allows a free re\-partition of the pool\. Manually splitting a public pool into train and test portions follows common practice, and our split is fixed once and shared by all configurations\.

#### Why these training and test sets\.

The partition follows one deterministic rule, fixed before any training, and is designed to keep the categories balanced on both sides\. Within each of the three categories \(tabular, text, image\), the 20 smallest competitions by dataset size are selected so that every task fits the execution sandbox, the 15 smallest of them form the training set, and the remaining 5 form the held\-out set\. Audio competitions are excluded because they do not fit the sandbox, and one held\-out task \(billion\-word\-imputation\) is dropped because its grader depends on a package unavailable in our environment, leaving 45 training and 14 held\-out competitions\. The rule has a useful side effect, as every held\-out task is larger than every training task of its category, so the held\-out evaluation also measures extrapolation from small training tasks to larger unseen ones\. We also verified that no task appears on both sides\.333One held\-out task shares the Jigsaw competition family with a training task, with different data and a different metric\.The data of every competition was cached locally in advance, so training and grading never depend on the Kaggle servers\. Table[5](https://arxiv.org/html/2608.12564#A3.T5)lists the training competitions and Table[6](https://arxiv.org/html/2608.12564#A3.T6)the held\-out ones\.

Table 5:The 45 training competitions of MLE\-Dojo \(train\), by category\.TabularTextImagemercedes\-benz\-greener\-manufacturingkaggle\-llm\-science\-examaerial\-cactus\-identificationicr\-identify\-age\-related\-conditionsmovie\-review\-sentiment\-analysis\-kernels\-onlyleaf\-classificationkobe\-bryant\-shot\-selectionllm\-detect\-ai\-generated\-textdenoising\-dirty\-documentshome\-data\-for\-ml\-courserandom\-acts\-of\-pizzafacial\-keypoints\-detectionspooky\-author\-identificationchaii\-hindi\-and\-tamil\-question\-answeringstatoil\-iceberg\-classifier\-challengeliberty\-mutual\-group\-property\-inspection\-predictionnbme\-score\-clinical\-patient\-notestgs\-salt\-identification\-challengedetecting\-insults\-in\-social\-commentary20\-newsgroups\-ciphertext\-challengeglobal\-wheat\-detectionwalmart\-recruiting\-store\-sales\-forecastingword2vec\-nlp\-tutorialwhale\-categorization\-playgroundprudential\-life\-insurance\-assessmentjigsaw\-toxic\-comment\-classification\-challengedog\-breed\-identificationunimelbtext\-normalization\-challenge\-english\-languageplant\-pathology\-2020\-fgvc7nomad2018\-predict\-transparent\-conductorswsdm\-cup\-multilingual\-chatbot\-arenadogs\-vs\-cats\-redux\-kernels\-editionamazon\-employee\-access\-challengetext\-normalization\-challenge\-russian\-languagepetfinder\-pawpularity\-scorepoker\-rule\-inductionllm\-classification\-finetuningplant\-seedlings\-classificationallstate\-purchase\-prediction\-challengestumbleuponshopee\-product\-matchingdont\-call\-me\-turkeylinking\-writing\-processes\-to\-writing\-qualitythe\-nature\-conservancy\-fisheries\-monitoringTable 6:The 14 held\-out competitions of MLE\-Dojo \(test\)\.CompetitionCategoryTaskGiveMeSomeCreditTabularPredict two\-year default risk of borrowersforest\-cover\-type\-kernels\-onlyTabularPredict forest cover type from cartographic featuresnovozymes\-enzyme\-stability\-predictionTabularRank thermostability of enzyme variantsinteger\-sequence\-learningTabularPredict the next term of integer sequencesafsis\-soil\-propertiesTabularPredict five soil properties from infrared spectraquora\-question\-pairsTextDecide whether two questions are duplicatesAI4CodeTextRecover the order of markdown cells in notebooksjigsaw\-unintended\-bias\-in\-toxicity\-classificationTextDetect toxicity while controlling identity biasquora\-insincere\-questions\-classificationTextFlag insincere questionsuw\-madison\-gi\-tract\-image\-segmentationImageSegment stomach and bowel in MRI scansinvasive\-species\-monitoringImageDetect an invasive plant species in photoskuzushiji\-recognitionImageDetect and transcribe cursive Japanese charactersbengaliai\-cv19ImageClassify components of handwritten Bengali graphemescassava\-leaf\-disease\-classificationImageClassify cassava leaf diseases
#### DSBench\.

The transfer suite is the data\-modeling split of DSBench\[[24](https://arxiv.org/html/2608.12564#bib.bib4)\], whose tasks are all drawn from Kaggle \(74 tasks per the paper, 75 task directories in the released data\)\. We keep the 60 tasks that run end to end in our sandbox\. The other 15 are excluded because the training data is too large for the sandbox \(seven tasks\), the sample submission is missing \(eight tasks\), or the submission format exceeds the scaffold’s column limit \(one task\)\. The category columns of Table[1](https://arxiv.org/html/2608.12564#S5.T1)contain 19 binary classification, 10 multi\-class classification, and 23 regression tasks, and the remaining 8 tasks with non\-standard metrics form the Other column\. We also checked these 60 tasks against the 45 training competitions with exact, normalized, and fuzzy name matching, and found no overlap\.

## Appendix DExperimental Details

#### Training configuration\.

Both scales train with GRPO, eight groups per step andn=8n=8trajectories per group, with up to four interaction turns per trajectory, at most 4096 generated tokens per turn, and observations truncated to 1024 tokens\. Optimization uses a constant learning rate of1×10−61\\times 10^\{\-6\}, a KL coefficient of0\.040\.04, and an entropy coefficient of0\.0020\.002, and rollouts are sampled at temperature1\.01\.0with top\-pp1\.01\.0\. Real execution runs each solution in an isolated environment with a 1200\-second budget\. The world model serves the agent backbone with a 12k\-token context and returns its prediction within a 1024\-token budget, and every step at least one group is graded by real execution to feed the anchor pool\.

#### Sandbox environment\.

Each solution executes in an isolated sandbox built as a Python 3\.11 virtual environment on Linux, with a single ~40 GB NVIDIA GPU and the path contract of the agent prompt, reading inputs fromDATA\_DIRand writing the submission toSUBMISSION\_PATH\. The environment holds 122 packages including transitive dependencies, and installing the directly requested set of Table[7](https://arxiv.org/html/2608.12564#A4.T7)into a clean virtual environment reproduces it\.

Table 7:Libraries preinstalled in the execution sandbox, by category, with the exact versions\.CategoryLibraries \(version\)Scientific computingnumpy 1\.26\.4, pandas 2\.2\.3, scipy 1\.17\.1, statsmodels 0\.14\.6, sympy 1\.14\.0, networkx 3\.6\.1Classical MLscikit\-learn 1\.4\.2, xgboost 3\.2\.0, lightgbm 4\.6\.0, catboost 1\.2\.10, joblib 1\.5\.3Deep learning \(GPU\)torch 2\.12\.0 \(CUDA 13\.0\), torchvision 0\.27\.0, tensorflow 2\.21\.0, keras 3\.14\.1NLPtransformers 5\.12\.1, tokenizers 0\.22\.2, nltk 3\.9\.4Imaging and plottingPillow 12\.2\.0, matplotlib 3\.10\.9, seaborn 0\.13\.2, plotly 6\.6\.0Utilitiesh5py 3\.14\.0, tqdm 4\.67\.3, requests 2\.34\.2, regex
#### Prompting the world model\.

The world model is never fine\-tuned and works by prompting alone\. Its prompt contains the same task description that the agent sees and the agent’s current solution, and it is instructed to simulate the execution and report the outcome in the same format as the real environment, from which the score is aggregated\. The context is capped at 12k tokens and the prediction at 1024 tokens\. Appendix[E](https://arxiv.org/html/2608.12564#A5)sketches the structure of this prompt and of the agent prompt\.

#### Recalibration\.

The monotone map of Equation \([4](https://arxiv.org/html/2608.12564#S3.E4)\) is the identity until 200 anchor pairs have accumulated, is first fit at that point, and is refit after every 64 new pairs so that the correction follows the drift of the world model\.

#### Calibrating the reference disagreement\.

The disagreementη^2\\hat\{\\eta\}^\{2\}is normalized by the within\-group spread of the true scores, and this normalization is what lets a reward\-level statistic estimate a gradient\-level ratio\. The sampling varianceVEV\_\{E\}and the noise\-injected excess ofVW​MV\_\{WM\}arise from the same estimator applied to the same trajectories, so the conversion from score fluctuation to gradient fluctuation largely cancels in their ratio, and what remains is the noise variance measured against the natural spread of the scores, which isη^2\\hat\{\\eta\}^\{2\}up to a conversion factor of order one shared between the two streams\. A single measurement at the end of warmup fixes this factor,c=1/η^cal2c=1/\\hat\{\\eta\}^\{2\}\_\{\\mathrm\{cal\}\}, and we find the calibrated value in one of our actual runs to be0\.960\.96, confirming that it is indeed of order one\. The fused variance is additionally second\-order insensitive to the weight \(Lemma[12](https://arxiv.org/html/2608.12564#Thmtheorem12)\), so the precision of this single measurement is not critical\.

#### Bounded anchor weight\.

For training stability, we bound the anchor factor of Equation \([5](https://arxiv.org/html/2608.12564#S3.E5)\) within\[1,wmax\]\[1,w\_\{\\max\}\]withwmax=4w\_\{\\max\}=4, which acts as a regularization on the fused gradient\. The lower bound keeps anchor groups from being down\-weighted below the predicted groups, and the upper bound prevents a transient spike of the measured disagreement from letting a few anchor groups dominate a batch\. The tracked noise is likewise estimated over a recent window of the pool𝒫\\mathcal\{P\}rather than the full history, so that the measurement tracks the current policy\.

#### VLA setup\.

The policy is MiniVLA\-1B\[[4](https://arxiv.org/html/2608.12564#bib.bib19)\], a Prismatic\-style model with a Qwen2\.5\-0\.5B language backbone and a vector\-quantized action head that encodes an eight\-step action chunk into seven discrete tokens, pretrained on LIBERO\-90\. All models first run supervised fine\-tuning on the 50 official demonstrations per task, and then GRPO for 40 steps\. Each step samples 64 rollouts per task, and each rollout takes at most 520 environment steps, the benchmark’s standard horizon\. Training uses the first 16 official initial states of each task while evaluation uses all 50, so most evaluated initial states are never seen in training\. Evaluation is greedy, with eight repetitions per initial state, since the simulator’s physics is stochastic\.

## Appendix EPrompts

Both prompts are too long to reproduce in full, so we sketch their structure below, quote the load\-bearing passages verbatim, and mark every omission with \[…\]\.

#### The agent prompt\.

Each turn, the AutoResearch agent sees a fixed system prompt, a user prompt holding an automatically generated task overview, and the full history of its previous attempts with their execution feedback\. The system prompt consists of the seven segments sketched below\. The user prompt is the first user message of the conversation, as the later turns simply accumulate the interaction history on top of it, and the task overview inside is generated by pure introspection over the task files, with no manual per\-task information\.

System prompt of the AutoResearch agentRole and objective\. You are an expert ML engineer competing to MAXIMIZE your leaderboard Position Score over UP TO K attempts \(Position Score = your percentile rank, higher = better; earned ONLY when your code runs and writes a VALID submission\)\. \[…\]Two\-phase strategy\. PHASE 1 \-\-\- LAND A VALID SUBMISSION FIRST \(turn 1\): write a SIMPLE solution you are confident RUNS end\-to\-end and writes SUBMISSION\_PATH in the exact required format\. A valid submission is your safety net\. \[…\] PHASE 2 \-\-\- IMPROVE INCREMENTALLY \(later turns\): START FROM YOUR LAST WORKING CODE, copy it, and change EXACTLY ONE thing per turn\. \[…\] If a change ERRORS or scores WORSE, discard it and try a DIFFERENT single change\. Never rewrite from scratch once something works\.Environment contract\. ENVIRONMENT CONTRACT \(these Python variables are ALREADY defined when your code runs\): \[…\] DATA\_DIR: absolute path to the folder with the data files listed below\. \[…\] SUBMISSION\_PATH: absolute path to write your submission CSV to\. \[…\] A modern NVIDIA GPU \(~40 GB\) is available\. \[…\] Do NOT hardcode ’/kaggle/\.\.\.’, ’/content/\.\.\.’, or any other path\.Sandbox runtime\. Runtime: Python 3\.11 on Linux \(this is the sandbox where YOUR code executes\)\. Installed libraries you may import \(name version\): numpy 1\.26, pandas 2\.2, scikit\-learn 1\.4, xgboost 3\.2, torch 2\.12 \(CUDA\), transformers 5\.12 \[…\] Anything NOT in this list is not installed \(e\.g\. no opencv, no jax\)\.Response format\. EACH TURN, respond in TWO parts, in this order: \[…\] 1\) ANALYSIS \(2\-4 short sentences of plain text, FIRST\): name exactly what went wrong and the ONE change that fixes it, or the ONE change you will make to raise the score\. \[…\] 2\) CODE: a SINGLE fenced Python block \-\-\- a complete, self\-contained script\.Feedback loop\. After your code runs, the environment returns your print\(\) output and \-\-\- if a valid submission was written \-\-\- your leaderboard Position and Raw Score; otherwise the full error traceback\. \[…\]Rules\. Keep a valid submission as your fallback; only replace it with code that RUNS and scores BETTER\. \[…\] The submission CSV must match the required format EXACTLY\. Keep code efficient so it finishes within the time limit\.

User prompt of the AutoResearch agentCompetition: <name\> === DESCRIPTION === <head of the official competition description, including its evaluation metric\> \[…\] === DATA INVENTORY === <every file with its size; per\-CSV schema with row count and every column’s dtype and example values; ZIP and directory contents; sampled image dimensions\> \[…\] === TARGETS === <the columns present in the training data but absent from the test data\> === SUBMISSION CONTRACT === <the exact required columns and example rows of sample\_submission\.csv\> Begin\.

#### The world model prompt\.

The world model shares one simulator system prompt across eight parallel single\-aspect checks, which cover imports and names, API calls against the installed versions, file and column references against the data inventory, shapes and dtypes, the compute budget, remaining runtime errors, submission validity, and solution quality\. All eight checks share the same system prompt\. The user prompt of each check holds its specific instruction followed by the case block, namely the task overview above and the agent’s code with 1\-indexed line numbers, and the check returns a verdict in strict JSON\. The verdicts are then aggregated into the scalar scorer^\\hat\{r\}, where failures cap the score and a valid solution is scored by the quality check\. The user prompt shown below is that of the file and column reference check\.

System prompt of the world modelRole\. You are a precise execution\-and\-grading SIMULATOR for Kaggle/MLE Python solutions\. You do NOT execute code\. You predict \-\-\- by careful static analysis \-\-\- exactly what the real sandbox would report\.Environment brief\. Sandbox: Python 3\.11 on Linux, ONE ~40GB GPU available\. Installed \(name version\): numpy 1\.26, pandas 2\.2 \[…\] Execution is killed at 1200s\. The code reads data from os\.environ\[’DATA\_DIR’\] and MUST write predictions to os\.environ\[’SUBMISSION\_PATH’\]\. \[…\]Inputs\. You are given the TASK OVERVIEW \(with a DATA INVENTORY: real files, columns, row counts, dtypes, and the sample\_submission format\) and the agent’s CODE \(its lines are 1\-indexed as shown\)\. Analyze ONLY the failure mode this check asks about\.Calibration policy\. CRITICAL \-\-\- DEFAULT TO PASS\. Real submissions usually RUN FINE; most code does NOT trigger this check’s failure mode\. \[…\] Return verdict=’fail’ ONLY when you can cite a SPECIFIC line and a CONCRETE, CERTAIN reason it raises this exact error\. \[…\] A false ’fail’ on working code is WORSE than a missed error\. When in doubt, PASS\.Error specificity\. If you do fail it, be concrete: identify the EXACT offending source line, its line number, the precise Python exception class, and the real error message\. \[…\] The agent will READ your env\_feedback to fix its code next turn, so env\_feedback MUST look byte\-similar to the real sandbox output\. \[…\]Output format\. Output STRICT JSON only, no prose around it: \{"reason": "<= 2 sentences, the specific cause\>", "verdict": "<pass\|fail\>", "confidence": <float 0\.\.1\>, "error\_type": \[…\], "error\_line": \[…\], "env\_feedback": \[…\]\}

User prompt of the world modelCHECK: Will this code raise FileNotFoundError or KeyError\(column\) \-\-\- i\.e\. does it reference a file/path/column that does not match the DATA INVENTORY? \[…\] verdict=fail if a FileNotFoundError or KeyError is certain; env\_feedback = the exact message\.=== TASK: <competition name\> === === TASK OVERVIEW \+ DATA INVENTORY === <the task overview above\> \[…\] === AGENT CODE \(lines are 1\-indexed\) === 1 <line 1 of the agent’s script\> 2 <line 2\> \[…\]