AI Research Preference Models

# AI Research Preference Models
Source: [https://arxiv.org/html/2608.13940](https://arxiv.org/html/2608.13940)
Bassel Al OmariTingchen FuThomas MannCarl Domond Lucia Cipolina\-KunBhavul GauriMuna AghameluAlexander D\. GoldieEryk HelenowskiJean\-Christophe Gagnon\-AudetAlberto PepeSaba NazirDaniel IzcovichNoam LeviRishi HazraKaren HambardzumyanNicolas BaldwinXian LiMartin JosifoskiParis GiampourasMasoud Jalili SabetAnya SimsHela MomandTatiana ShavrinaDespoina MagkaJason WestonYulin WangAnirudh GoyalJoão HenriquesYoram BachrachEmily McMilinJakob Nicolaus Foerster

###### Abstract

AI research agents \(AIRA\) can now propose, implement, and evaluate their own machine learning experiments, but progress on frontier tasks is throttled by cost: a candidate solution can be written in minutes, whereas evaluating it can take hours to days of GPU time\. An agent can therefore propose far more candidates than it can afford to run, and its progress depends on itsresearch preference: how it allocates a fixed execution budget across many candidates\. We introduce AI Research Preference Models \(RPMs\) that predict which of multiple candidate solutions are most worth executing, without paying the cost of executing them all\. We build RPMs from frozen pretrained language models \(with no task\-specific training\), in two forms: an inference\-only model that reasons over candidate plans, code, and prior executed solutions, and an agentic model that additionally runs small\-scale pilot experiments before deciding\. We integrate both into the AIRA\-dojo search agent and evaluate onAIRS\-Bench, a recent benchmark of machine learning research tasks for AI research agents\. The two variants raise the average normalized score from0\.6840\.684to0\.7110\.711and0\.7290\.729respectively, and reach the unguided agent’s 24\-hour performance in roughly 15 hours, using less than two\-thirds of its execution budget\. Our best RPMs also yield new state\-of\-the\-art results on twoAIRS\-Benchtasks\.

††affiliation:FAIR at Meta††affiliation:University of Oxford††affiliation:University College London††contribution:Lead Authors††contribution:Core Contribution††contribution:Equal Supervision††correspondence:Bassel Al Omari at[balomari@meta\.com](mailto:[email protected])## 1Introduction

Language model agents have advanced rapidly in domains such as mathematics, coding, and computer use, where candidate actions can be evaluated accurately and efficiently: a mathematical answer can be checked against a reference, a program run against a test suite, and a computer\-use task verified against its target state\. These evaluation functions provide the reward signal that lets agents iterate and improve, driving rapid progress on benchmarks such as SWE\-bench Verified\([21](https://arxiv.org/html/2608.13940#bib.bib46)\), Terminal Bench\([28](https://arxiv.org/html/2608.13940#bib.bib45)\), and MLE\-bench\([7](https://arxiv.org/html/2608.13940#bib.bib16)\)\.

Progress has been slower for AI research agents that autonomously propose, implement, and evaluate their own experiments, despite recent efforts such as the AI Scientist\([26](https://arxiv.org/html/2608.13940#bib.bib24)\), AIRA\-dojo\([47](https://arxiv.org/html/2608.13940#bib.bib13)\), andAIRA2\{\}\_\{\\mbox\{2\}\}\([19](https://arxiv.org/html/2608.13940#bib.bib15)\)\. Frontier machine learning research lacks cheap feedback: proposing or modifying candidate code can be quick, but executing it to train a model and measuring its performance can consume hours to days of GPU time\. Because an agent can propose far more candidates than it can afford to run, the primary lever on research progress becomes*research preference*: deciding which research directions are promising enough to allocate compute budget to, and which directions to drop\.

Figure 1:Overview of an AI research agent with RPM\-augmented child creation\.Each node represents a solution within the search tree, labeled with its evaluation score\. The agent \(1\) selects a promising parent node from the active tree, \(2\) generates a batch of candidate child solutions, \(3\) utilizes the RPM to select the most promising candidate, using the full context of previously executed solutions and \(4\) executes and scores only the selected candidate to expand the tree\. This mechanism eliminates the compute overhead of running unpromising candidates\. For clarity we depict an illustrative batch; in our experiments each step generatesN=15N\{=\}15candidate children, and the RPM selects one to execute via a pairwise tournament knockout \([Section3\.3](https://arxiv.org/html/2608.13940#S3.SS3)\)\.We study this allocation problem directly by introducing a dedicatedResearch Preference Model \(RPM\)into the AIRA\-dojo search harness\. The RPM receives the full context of previously executed solutions, and uses it to decide which of N newly generated candidate solutions will be most valuable to execute next\.

We summarize our contributions below:

- •We introduce AI Research Preference Models \(RPMs\) to study experimental compute allocation in AI research agents\. This is motivated by the observation that AI Research Agents on frontier research tasks\([27](https://arxiv.org/html/2608.13940#bib.bib23)\)are bottlenecked not by their ability to generate candidate solutions, but by the compute to execute them\.
- •We developInference\-only RPMs, frozen\-weight LLMs that reason over candidate plans, code and previous solutions, and demonstrate that it raises performance of the AIRA\-dojo agent onAIRS\-Benchfrom0\.6840\.684to0\.7110\.711\.
- •We further proposeAgentic RPMs, an extension of Inference\-only RPMs with the ability to run small\-scale pilot experiments\. Integrated within AIRA\-dojo, Agentic RPMs further raise performance onAIRS\-Benchto0\.7290\.729, approaching the validation oracle ceiling of0\.7480\.748\.
- •OnAIRS\-Bench, agents equipped with our best RPMs yield new state\-of\-the\-art results on two tasks and match the unguided agent’s 24\-hour performance in roughly 15 hours, using less than two\-thirds of its execution budget \([fig\.2](https://arxiv.org/html/2608.13940#S5.F2)\)\.

## 2Background

In line with the broader agent research literature, we view an agent as a computer system that is situated in some environment and is able to act autonomously in this environment in order to achieve its design objectives\([50](https://arxiv.org/html/2608.13940#bib.bib14)\)\. In our setting, an AIRA acts by generating and executing code\. The objective is to produce an artifact \(such as model weights, an optimised code snippet, or answer to a question\) that, when evaluated by some task\-specific reward function, achieves a high score\.

### 2\.1AI Research Agent Benchmarks

Recent benchmarks evaluate large language model agents across complex, long\-horizon workflows\. For software engineering, popular suites like SWE\-bench\([21](https://arxiv.org/html/2608.13940#bib.bib46)\)assess an agent’s ability to resolve real GitHub issues and modify multi\-file codebases\. Within the data science domain, platforms like MLE\-bench\([7](https://arxiv.org/html/2608.13940#bib.bib16)\)evaluate agent capabilities through structured machine learning engineering competitions\.

We base the majority of our experiments onAIRS\-Bench\([27](https://arxiv.org/html/2608.13940#bib.bib23)\), a comprehensive suite of 20 machine learning tasks sourced from state\-of\-the\-art papers\. These tasks span diverse domains, including language modeling, mathematics, bioinformatics, and time series forecasting\.AIRS\-Benchassesses agentic capabilities over the full research lifecycle \(including idea generation, experiment analysis, and iterative refinement\) without providing baseline code\. Each task is rigorously specified by a problem, a dataset, a target metric, and a published state\-of\-the\-art value\.

Here the agents are placed in an environment with a training dataset, a set of test inputs, and 24 hours of access to an H200 GPU\. The AIRA’s goal is to write code that trains a model and produces asubmission\.csvwith predictions on the test inputs\. The agent may choose to run code that reports a validation score \(e\.g\., from using cross\-validation\) that it can use for guiding search\. The true test score \(produced by comparing thesubmission\.csvto the ground truth labels\) is hidden from the agent\.

To aggregate performance across heterogeneous task metrics,AIRS\-Benchdefines a Normalized Score for agentaaon tasktt:

NSta=ϕt​\(sta\)−ϕt​\(stmin\)ϕt​\(stsota\)−ϕt​\(stmin\),\\text\{NS\}\_\{t\}^\{a\}=\\frac\{\\phi\_\{t\}\(s\_\{t\}^\{a\}\)\-\\phi\_\{t\}\(s^\{\\mathrm\{min\}\}\_\{t\}\)\}\{\\phi\_\{t\}\(s^\{\\mathrm\{sota\}\}\_\{t\}\)\-\\phi\_\{t\}\(s^\{\\mathrm\{min\}\}\_\{t\}\)\},\(1\)wherestmins\_\{t\}^\{\\mathrm\{min\}\}is the worst score observed across all agents,stsotas\_\{t\}^\{\\mathrm\{sota\}\}is the most recent public SOTA score as of the benchmark’s publication, andϕt\\phi\_\{t\}is a non\-linear log transform, defined asϕt​\(s\)=−log10⁡\(\|s−stopt\|\)\\phi\_\{t\}\(s\)=\-\\log\_\{10\}\(\|s\-s^\{\\mathrm\{opt\}\}\_\{t\}\|\)to properly weight exponential progress near optimal bounds \(stopts\_\{t\}^\{\\mathrm\{opt\}\}\)\. Under this metric,NS=0\\text\{NS\}=0corresponds to the minimum baseline andNS=1\\text\{NS\}=1to the public SOTA, while invalid or failed submissions receive a score of00\. The Average Normalized Score reported on this benchmark averages the Normalized Score uniformly across all tasks and seeds\.

### 2\.2AI Research Agent Scaffolds

To systematically analyze and improve AI research agents, we decompose an AIRA into an LLM backbone and an algorithmicscaffold\. While the backbone provides core reasoning capabilities, the scaffold defines the decision logic and search strategy \(ranging from simple linear loops to complex tree search\) that govern how candidate solutions are generated, evaluated, and iteratively refined\. A single scaffold like AIRA\-dojo\([47](https://arxiv.org/html/2608.13940#bib.bib13)\)or Claude Code\([3](https://arxiv.org/html/2608.13940#bib.bib19)\)can be instantiated with different backbones, such as Claude Opus\([4](https://arxiv.org/html/2608.13940#bib.bib39)\)or GPT\-5\([43](https://arxiv.org/html/2608.13940#bib.bib18)\)\.

These candidate solutions can be viewed as nodes within an evolving solution graph, where each node stores concrete artifacts like code scripts, execution logs, and metric scores\. The agent explores this graph by selecting a parent node and mutating its contents to produce a child node\. Inspired by this evolutionary computation framework, we characterize the core design space of search scaffolds along three primary axes:

- •Parent Selection:The strategy used to identify which historical solutions, trajectories, or ideas are most promising to build upon next\.
- •Child Creation:The process of taking the selected parent solutions and prompting the LLM to generate a new candidate child solution\. The localized prompts used to drive*child creation*are referred to asoperators\.
- •Final Solution Selection:The criteria used to evaluate the accumulated bank of candidate solutions and determine which single solution to return as the final output\.

Existing AIRA architectures implement these mechanisms in fundamentally different ways\. For example,MLGym\([31](https://arxiv.org/html/2608.13940#bib.bib29)\)operates as a linear search scaffold that heavily simplifies parent selection by always choosing the most recent node as the parent, relying on a single mutation operator to iteratively refine it\. In contrast, we build upon AIRA\-dojo, an evolutionary tree\-search framework that uses greedy parent selection to always mutate the node with the highest current validation score\. To orchestrate child creation, AIRA\-dojo employs a specialized suite of mutation operators tailored to distinct engineering phases, specificallyDraft,Improve, andDebug\. Finally, for final solution selection, AIRA\-dojo submits the node that achieved the highest validation score across the entire search tree\.

Ultimately, this loop produces an expanding bank of candidate solutions, each paired with its evaluation score\. It is over this growing set that a preference model can intervene, deciding which candidates are worth the expense of execution\. It also serves as a valuable dataset to train and evaluate an RPM\.

## 3AI Research Preference Models

Frontier machine learning research lacks low\-cost feedback: while proposing candidate solutions is fast, executing them to train a model can consume hours or days of GPU compute\. Faced with this bottleneck, research progress heavily relies on predicting the value of pursuing candidate research directions\.

To address this challenge, we introduce AI Research Preference Models \(RPMs\) to guide experimental allocation within research agents\. Through initial experimentation, we observed that language models perform unreliably when forecasting absolute metrics or execution outcomes\. Consequently, an RPM reformulates experimental allocation as a relative ranking problem, ranking candidate solutions to select the most promising paths before dedicating compute to pursuing them\.

We explore RPMs leveraging varying ranges of test\-time compute:

- •Inference\-only RPMs\([Section3\.1](https://arxiv.org/html/2608.13940#S3.SS1)\): Rank candidate solutions using lightweight reasoning over search history and code diffs\.
- •Agentic RPMs\([Section3\.2](https://arxiv.org/html/2608.13940#S3.SS2)\): Allocate additional compute to run rapid sandbox pilot experiments prior to ranking\.

Designed as a scaffold\-agnostic component, RPMs can interface with a wide variety of AIRA architectures\. In this work, we investigate integrating RPMs into the AIRA\-dojo evolutionary tree\-search scaffold \([Section3\.3](https://arxiv.org/html/2608.13940#S3.SS3)\)\. We target the child\-creation phase, where creating a single child mutation is replaced with generatingNNcandidate modifications in parallel and using an RPM\-guided tournament to select the most promising solution before committing GPU compute\.

### 3\.1Inference\-only RPMs

The “LLM\-as\-a\-Judge” paradigm\([56](https://arxiv.org/html/2608.13940#bib.bib10)\)demonstrates that LLMs can rank technical solutions with reasonable fidelity using internal intuition and reasoning\. Motivated by this approach, we experiment with purely querying pretrained LLMs as an inexpensive preference model to select between research ideas\. To understand how visibility into the AIRA’s search space affects the RPM’s selection quality, we experiment with varying the count of previously explored solutions visible to the RPM and the count of suggestions it selects between\.

To develop the chosen prompt even further, we leverage MIPROv2 from the DSPy framework\([34](https://arxiv.org/html/2608.13940#bib.bib12)\), a widely adopted baseline for robustly optimizing prompt instructions\. The prompt optimizer generated an instruction set that directs the RPM to conduct a more structured analysis of each solution and remain tolerant of minor, fixable issues\. Further details on the prompt are provided in Appendix[A\.1](https://arxiv.org/html/2608.13940#A1.SS1)\.

### 3\.2Agentic RPMs

##### Motivation

A core practice in software engineering, and machine learning research is rapid prototyping\. To comprehensively understand and validate the potential or the feasibility of a novel idea, researchers tend to quickly run small\-scale pilot experiments before launching a full\-volume large\-scale experiment\. Inspired by how pilot experiments inform the possible outcome and assist decision making, we develop agentic RPMs where a pilot\-experiment agent can use multiple predefined tools in a sandbox environment to conduct pilot experiments before making a decision\.

##### Agentic Workflow

Concretely, a pilot experiment is conducted via multi\-turn interaction between the language model agent and the environment, interleaving chain\-of\-thought reasoning, tool calling, and receiving environment feedback\([53](https://arxiv.org/html/2608.13940#bib.bib11)\)\. The sandbox environment for the agentic RPM is an exact clone of the environment for AIRA, including the access to a single H200 GPU\. Meanwhile, the pilot\-experiment agent has access to the training dataset and unlabeled test dataset, together with necessary pre\-installed Python packages in this environment, similar to the AI research agent\. To interact with this environment, we provide the pilot\-experiment agent with a set of tools:python, bash, andsubmit\_solution\. Thepythontool andbashtool allow executing any Python code or Bash code, respectively, and then return execution results\. With the outcome of the pilot experiment, the pilot\-experiment agent can summarize and submit the experimental findings with the toolsubmit\_solution\. Notably, the pilot\-experiment agent is only required to submit the summary and analysis of the pilot experiment, but not to make a final prediction among candidates\. Thesubmit\_solutiontool will return the remaining time budget\. If the remaining time budget surpasses a specific threshold, the pilot\-experiment agent would be prompted to run further experiments to make the best use of the time budget and computation resources\. Finally, the candidate solution, the task description, and all submitted pilot experiment findings are input to a language model to make a final prediction\.

##### More Informative Experiments

During our study, we find that the pilot\-experiment agent can be too conservative in scheduling the time budget, leaving a large portion of the time budget unused at the first call ofsubmit\_solution\. Even though we could prompt the pilot\-experiment agent to run more pilot experiments after the initial submission, the follow\-up experiments are often limited to hyperparameter tuning of previous ones\. Over conservative time budget scheduling and repetitive follow\-up experiments jointly lead to less informative pilot experiments\. To deal with this problem, we use two mechanisms to elicit more informative follow\-up experiments\. First, we*overstate*the remaining time budget in the prompt \(reporting it as several times larger than it truly is\), discouraging the agent from stopping prematurely\. Second, after eachsubmit\_solutioncall, a separate feedback model reviews the findings so far and either proposes the single most informative next experiment or signals that the evidence is already sufficient, in which case we end the loop; its proposal and the remaining budget are then returned to the agent to guide the follow\-up experiment\.

Full details on the agentic RPM’s prompts are provided in Appendix[B](https://arxiv.org/html/2608.13940#A2)\.

### 3\.3AIRA Integration

While an RPM can intercept multiple stages of an agent’s search scaffold, we focus our implementation on augmenting thechild creationphase, as illustrated in Figure[1](https://arxiv.org/html/2608.13940#S1.F1)\. In an evolutionary AIRA scaffold, child creation fundamentally encompasses two sub\-steps: child candidate creation and child candidate selection\. By default, AIRA\-dojo generates a single candidate solution during candidate creation, which is automatically selected to be executed, evaluated, and added to the search tree\. We modify this pipeline by first expanding child candidate creation: a chosen parent node is mutated by applying operatorsNNtimes independently in parallel to yieldNNunexecuted candidate child solutions\. During child candidate selection, we then introduce the RPM which evaluates theseNNcandidates alongside historical trajectory context, conducting pairwise comparisons in a tournament knockout structure to select the single candidate for full execution\.

To ground each comparison in the search so far, we also provide the RPM selected context from the tree\. Context nodes are pulled via a BFS traversal of the already\-explored tree starting from the parent, collecting up toKKnon\-buggy nodes from earlier in the search; each of these previously\-evaluated solutions is presented alongside the validation score it obtained\.

## 4Experimental Setup

### 4\.1End\-to\-End Evaluation

We integrate RPMs into the child\-creation phase of AIRA\-dojo, and evaluate this augmented scaffold against the 20 publicly releasedAIRS\-Benchtasks that fall under the text and tabular modalities\. Following the originalAIRS\-Benchevaluation protocol, the AIRA\-dojo evaluations are provided 24 hours of access to a single H200 per task and are repeated for 10 seeds\. To avoid the generalization gap which undermined long\-horizon search in AIRA\-dojo, we integrate the Hidden Consistent Evaluation protocol fromAIRA2\{\}\_\{\\mbox\{2\}\}\. Both the AIRA\-dojo operator used to generate the child candidates and the RPMs share a Qwen3\.6\-27B\([38](https://arxiv.org/html/2608.13940#bib.bib7)\)backbone, standardizing our setup on a high\-performing open\-weights model for code reasoning\. Maintaining an identical model across child creation and selection ensures that all observed improvements are driven by the framework rather than a stronger selection backbone\.

To contextualize these results, we first compare against the default No\-RPM baseline\. In this setup, child selection defaults to uniform random selection among generated candidates, which corresponds to vanilla AIRA\-dojo in expectation\. We also compare against a Test Oracle and a Validation Oracle, that are constructed by executing all candidates at each step and choosing the highest scorer\. Only the compute time of the selected candidate counts towards the 24\-hour limit\. While neither is viable online \(the Test Oracle utilizes privileged test\-set information, and the Validation Oracle requires a prohibitive compute overhead to execute every candidate\), they serve as ceilings for greedy child selection\.

### 4\.2Offline Evaluation

The full end\-to\-end evaluations detailed in[Section4\.1](https://arxiv.org/html/2608.13940#S4.SS1)are computationally expensive, requiring 200 H200 GPUs for 24 hours\. To enable quick iterations and guide development of our RPMs before running full end\-to\-end evaluations, we produce an offline evaluation dataset compiled from previous AIRA\-dojo runs on a separate set of 40 unreleasedAIRS\-Benchtasks from the image, video and audio modalities\. The development and evaluation sets are split by modality to avoid task contamination\. The previous AIRA\-dojo runs followed the standardAIRS\-Benchevaluation protocol of dedicating 24 compute hours with a single H200 and evaluating 10 different seeds\. The previous runs were conducted with gpt\-oss\-120b\([33](https://arxiv.org/html/2608.13940#bib.bib2)\), GPT\-4o\([32](https://arxiv.org/html/2608.13940#bib.bib3)\)and CWM\([12](https://arxiv.org/html/2608.13940#bib.bib37)\)as the LLM backbone\. From these runs, we extract 1,000 sibling node pairs, including their plans, code, and search tree history\. In selecting these pairs, we discard near\-ties with a normalized test\-metric gap below 0\.01, to prevent negligible, run\-to\-run metric noise from confounding the evaluation signal\. The RPM is evaluated on its accuracy in selecting the node with the highest test score in its subtree\. We choose this ground\-truth label to ensure the RPMs look past immediate performance, explicitly rewarding candidates with strong long\-term fixability and extensibility\. We acknowledge that this label inherits a bias from the original greedy search policy, where nodes with stronger early scores are favored during search, expanding their subtrees and giving them greater opportunity to reach high scores\. Random selection establishes a 50% baseline accuracy floor\.

## 5Results

### 5\.1End\-to\-End Evaluations

Figure 2:Evaluation of RPM\-augmented AIRA\-dojo\.Average normalized scores over time \(left\) and final performance \(right\) for AIRA\-dojo with RPM\-augmented child selection onAIRS\-Bench\. Dashed lines indicate the No\-RPM baseline and immediate validation or test oracles\. The Inference\-only RPM \(blue\) yields steady gains over the baseline \(orange\)\. While per\-step proxy experiments slow its early trajectory, the Agentic RPM \(purple\) leverages this compute overhead to surpass other methods\.We present the end\-to\-end performance of integrating our RPMs within AIRA\-dojo, in Figure[2](https://arxiv.org/html/2608.13940#S5.F2)\. In these evaluations, AIRA\-dojo is configured to generate 15 child candidate suggestions at each operator step, leveraging our offline finding that expanding the candidate pool size systematically improves selection performance \(Section[5\.2\.1](https://arxiv.org/html/2608.13940#S5.SS2.SSS1.Px1)\)\. The Inference\-only RPM evaluates these pairs using the “LLM\-as\-a\-judge” reasoning prompt presented in Section[3\.1](https://arxiv.org/html/2608.13940#S3.SS1), including the scores of historical nodes from the search tree\. For the Agentic RPM detailed in Section[3\.2](https://arxiv.org/html/2608.13940#S3.SS2), to balance overall compute budgets, we only deploy this selection mechanism for theDraftandImproveoperators, reverting to random selection duringDebugsteps\.

As shown in Figure[2](https://arxiv.org/html/2608.13940#S5.F2), our methods navigate the trade\-off between decision quality and compute time in fundamentally different ways\. Incurring no candidate\-execution cost, the Inference\-only RPM \(blue\) provides immediate and steady gains over the baseline with no RPM \(orange\)\. Conversely, the Agentic RPM \(purple\) exhibits a slow rise early in the run\. Considering it runs small\-scale proxy experiments in a sandbox at every step, its heavy per\-step compute consumption slows early progress along the time axis\. However, this rigorous per\-step evaluation eventually triggers a sharp performance acceleration, ultimately matching or exceeding both the Inference\-only RPM and the No\-RPM baseline\. Ultimately, both variants beat the unguided baseline’s final score of 0\.684, with Inference\-only reaching 0\.711 and Agentic reaching 0\.729, narrowing the gap toward the validation\-oracle \(0\.748\) and test\-oracle \(0\.759\) ceilings\.

##### Significance Testing

To evaluate statistical significance, we report the*probability of improvement*, defined as the likelihood that a randomly sampled run of one method outperforms another on a randomly selected task\. We compute task\-stratified bootstrap distributions usingrliable\([1](https://arxiv.org/html/2608.13940#bib.bib9)\), where a probability of0\.50\.5denotes no difference\. The results show that the Inference\-only RPM and the Agentic RPM achieve a statistically significant edge over the default AIRA\-dojo baseline \(No RPM\), yielding average improvement probabilities of 0\.5923 and 0\.5913, respectively, with the 95% confidence intervals lower bounded at 0\.5066 and 0\.5018, strictly excluding the 0\.5 mark of random chance\.

##### State\-of\-the\-Art Breakthroughs

These guided search capabilities translate directly to new state\-of\-the\-art \(SOTA\) milestones on established benchmarks\. On WinoGrande\([40](https://arxiv.org/html/2608.13940#bib.bib5)\), AIRA\-dojo with the Agentic RPM achieves an accuracy of 94\.1%, comfortably surpassing the previous SOTA of 88\.1% reported by[27](https://arxiv.org/html/2608.13940#bib.bib23)\. During this run, the agent fine\-tunes a Qwen2\.5\-14B\-Instruct model\([37](https://arxiv.org/html/2608.13940#bib.bib1)\)via LoRA on data with shuffled labels, then averages prediction logits across original and shuffled label orderings at inference to eliminate position bias\. Similarly, on SVAMP\([36](https://arxiv.org/html/2608.13940#bib.bib4)\), AIRA\-dojo with the Inference\-only RPM reaches 95\.7% accuracy, eclipsing the prior SOTA of 94\.2% from[58](https://arxiv.org/html/2608.13940#bib.bib6)\. Here, the agent designs few\-shot prompts that instruct the model to cleanly isolate relevant numerical data from distracting context, then generates ten independent reasoning paths by sampling Qwen2\.5\-7B\-Instruct with increased temperature, and resolves the final prediction using a majority vote\.

##### Research Efficiency

Beyond absolute performance gains, both preference models significantly accelerate search velocity\. While the unaugmented No\-RPM baseline requires the full 24\-hour allocation to reach its final score of 0\.684, our RPM\-guided approaches reach this identical performance threshold significantly faster\. The Inference\-only RPM matches this baseline score in 14\.88 hours \(a1\.61×1\.61\\timesspeedup\), while the Agentic RPM achieves it in 15\.50 hours \(a1\.55×1\.55\\timesspeedup\)\. This allows both methods to match standard performance while using approximately1\.5×1\.5\\timesless compute budget\.

##### Impact of Selection Quality

To confirm how decision quality drives performance, we retrospectively analyze the selection advantage of each RPM throughout the runs\. Selection advantage measures the average difference between the chosen candidate’s score and the overall batch mean\. As expected, random selection yields a selection advantage of roughly0\.00\.0in expectation, whereas the Inference\-only RPM achieves significantly higher selection quality, and the Agentic RPM yields the highest advantage on average\. Crucially, we find a strong positive correlation between selection advantage and final normalized score \(Pearsonr=0\.55r=0\.55, Spearmanρ=0\.56\\rho=0\.56\), confirming that more accurate candidate selection translates to better end\-to\-end AIRA performance\. Further details are provided in[SectionC\.1](https://arxiv.org/html/2608.13940#A3.SS1)\.

### 5\.2Offline Evaluation and Further Analysis

To guide development of our RPMs before running full end\-to\-end evaluations, we leverage the offline framework from Section[4](https://arxiv.org/html/2608.13940#S4)\. Full 24\-hour online runs are too computationally expensive for quick iteration, and thus this offline setup allows us to tune our models, analyze their core behaviors, and derive clear trends\.

We observe that the best configuration of Inference\-only RPMs is surpassed in predictive accuracy by our Agentic RPMs, with both exceeding the random baseline, matching the trend ultimately observed in our end\-to\-end evaluations\.

#### 5\.2\.1Inference\-only RPM

##### Inference Scaling

Figure 3:Offline evaluation of Inference\-only RPM scaling properties using GPT\-5\. Increasing search\-tree context provided gives improved predictive accuracy \(left\)\. Allocating a higher reasoning budget yields better selection \(middle\)\. Expanding the candidate suggestion pool reliably improves both the theoretical Oracle limit and the RPM’s selection performance \(right\), measured by the average difference between the selected solution’s score and the average score of all candidates\.We structure our offline analysis around three key dimensions that dictate the RPM’s visibility and analytical capacity:*context size*\(the number of historical code solutions and validation scores provided from the tree\),*suggestion count*\(the number of candidate child solutions\), and*reasoning budget*provided to the LLM\.

Our offline tests reveal three clear scaling behaviors\. First,*context scaling*demonstrates that supplying the RPM with a deeper history of historical search tree nodes and their validation scores consistently improves child\-selection judgment\. Second,*suggestion scaling*demonstrates that expanding the candidate pool systematically increases the selection advantage, measured as the average difference between the selected candidate’s score and the overall batch mean\. The oracle’s selection advantage rises significantly with pool size, a trend our RPM successfully captures to extract higher\-quality solutions from larger candidate batches\. Third,*reasoning scaling*reveals that increasing the reasoning budget allocated to the LLM judge steadily improves its selection accuracy\.

These results motivated using a large suggestion count \(15 suggestions\) and to maximize the context nodes provided \(by providing the maximum amount that can fit within the LLM’s context window\), and high reasoning budget parameters for the RPM in our end\-to\-end evaluation\.

##### Ensembling

We evaluate three frontier models, GPT\-5, Claude Opus 4\.8, and Gemini 3\.1 Pro, individually and via two aggregation techniques: a mechanical majority vote and an LLM\-Arbiter ensemble that ingests the reasoning traces of all three models before making a final decision\. Individual models’ performances range between 64\.66% to 67\.44% accuracy\. Aggregating their diverse reasoning traces further mitigates errors where majority vote increases accuracy to 68\.04%, while the LLM\-Arbiter ensemble achieves the highest overall offline accuracy of 69\.35%\. A full breakdown of these configurations and their corresponding results is compiled in Appendix[A\.2](https://arxiv.org/html/2608.13940#A1.SS2)\.

##### Reasoning Analysis

To get a better understanding of the RPM’s decision\-making, we analyze its generated reasoning traces\. We find that referencing prior evidence, correctness of implementation or the pretrained backbone in the justification yields higher selection accuracy compared to when these are omitted\. We also find, that citing more unique values from historical context also yields improved selection accuracy\. Further details on the reasoning analysis are provided in[SectionA\.3](https://arxiv.org/html/2608.13940#A1.SS3)\.

#### 5\.2\.2Agentic RPM

Figure 4:What strategy the Agentic RPM uses:The agentic RPM frequently runs simplified versions of the original candidates\. The strategies used by the 5\- and 30\-minute agents are remarkably similar\.\(left\)Both agents save time by running only a single cross\-validation split\.\(middle\)Both agents frequently sub\-sample the training data\.\(right\)Both agents employ similar training adaptations, such as removing ensembling, reducing epochs and lowering the batch size\.
Figure 5:How the Agentic RPM’s strategy affects performance:Overall, enlarging the time budget leads to better performance, but marginal returns may be diminishing\(leftmost\)\. Despite both 5\- and 30\-minute agents employing similar strategies, the effect of these strategies on the performance of the agentic RPM differs substantially\.\(middle left\)More splits do not bring better performance for either agent\.\(middle right\)Using full data brings an advantage over using subsampled data\.\(rightmost\)Most training adaptations are helpful for at least one agent, except the removal of the learning rate scheduler and simplification of data augmentation\.##### Compute Scaling

We analyze how the agentic RPM scales with execution time limits on our offline benchmark, using GPT\-5 as our language model backbone\. Increasing the available compute budget consistently improves selection accuracy, scaling from 78\.52% under a 5\-minute constraint, to 82\.78% at 30 minutes, and peaking at 84\.02% with a 4\-hour allocation\. This scaling behavior demonstrates that allowing the agent more time to run, observe, and debug directly translates to higher\-fidelity RPM estimations\. Notably, the marginal returns obtained from increasing the time budget are relatively minimal considering the pilot\-experiment agent shares the same time budget with the AI research agent and a 4\-hour run of agentic RPM is prohibitively expensive\. Therefore, we keep the time budget for end\-to\-end evaluation at 5 minutes\.

##### Proxy Strategy Analysis

To understand how agents identify promising candidates under strict time constraints, we use regular expression keyword matching to analyze the agent\-generated Python code\. Because full execution is impossible within 5\- or 30\-minute limits, agents actively simplify candidate code across validation, data sampling, and training\. As shown in Figure[4](https://arxiv.org/html/2608.13940#S5.F4), both budgets favor single\-split validation, data subsampling, and training adaptations \(e\.g\., removing ensembles, reducing epochs, and lowering batch sizes\), though the 30\-minute agent utilizes multi\-fold cross\-validation more frequently\. Notably, we provide some hints on possible proxy strategies in the prompt to the pilot experiment, as shown in Appendix[Figure8](https://arxiv.org/html/2608.13940#A2.F8)so the strategies are not entirely proposed by the pilot\-experiment itself\.

Our behavioral analysis in Figure[5](https://arxiv.org/html/2608.13940#S5.F5)reveals three key insights: \(1\) multi\-fold cross\-validation does not consistently improve accuracy due to frequent execution timeouts; \(2\) evaluating on full datasets provides significantly more reliable quality estimations than using subsampled data; and \(3\) removing learning rate schedulers or data augmentations triggers severe performance drops, as the underlying AIRA\-dojo tasks often rely on these exact training\-level optimizations to succeed\.

## 6Related Work

AI research agents and automated ML research\.Our work sits within the literature that formalizes machine learning research as an agentic search problem\. Early literature for automated scientific discovery and research assistance uses foundation models to generate ideas, write code, run experiments, analyze results, and draft papers\([26](https://arxiv.org/html/2608.13940#bib.bib24);[51](https://arxiv.org/html/2608.13940#bib.bib22);[41](https://arxiv.org/html/2608.13940#bib.bib25);[5](https://arxiv.org/html/2608.13940#bib.bib26);[39](https://arxiv.org/html/2608.13940#bib.bib27);[57](https://arxiv.org/html/2608.13940#bib.bib28)\)\. Current work introduces AI research agents as search policies over a tree of candidate ML solutions, where each node corresponds to an executed or proposed solution and edges correspond to mutations, refinements, or other improvement operators\([47](https://arxiv.org/html/2608.13940#bib.bib13);[19](https://arxiv.org/html/2608.13940#bib.bib15)\)\. This formulation directly motivates our work: RPMs target the core decision problem induced by such search trees, namely predicting which candidate solution should be executed before committing expensive compute\. Other recent works improve the agent scaffold by adding modular search, targeted refinement, ideation agents, multi\-agent specialization, or improved code interfaces\([30](https://arxiv.org/html/2608.13940#bib.bib17);[8](https://arxiv.org/html/2608.13940#bib.bib20);[54](https://arxiv.org/html/2608.13940#bib.bib30);[24](https://arxiv.org/html/2608.13940#bib.bib31);[52](https://arxiv.org/html/2608.13940#bib.bib32)\)\. While these works improve agents, scaffolds, or benchmarks, none of them address the dominant cost in this setting, namely*executing*the candidate solutions that the search proposes\.

Model\-based methods for expensive search\.In reinforcement learning, world models learn environment dynamics and can be used for planning\([42](https://arxiv.org/html/2608.13940#bib.bib35)\)or training policies from synthetic rollouts\([46](https://arxiv.org/html/2608.13940#bib.bib33);[17](https://arxiv.org/html/2608.13940#bib.bib34);[18](https://arxiv.org/html/2608.13940#bib.bib36)\)\. Surrogate models and learned policies are also used on themeta\-level, as a means to allocate experimental compute efficiently\. Such approaches have been applied in Bayesian optimization \(e\.g\.,\([44](https://arxiv.org/html/2608.13940#bib.bib38)\)\), hyperparameter optimization \(e\.g\.,\([23](https://arxiv.org/html/2608.13940#bib.bib21);[13](https://arxiv.org/html/2608.13940#bib.bib41);[20](https://arxiv.org/html/2608.13940#bib.bib42);[10](https://arxiv.org/html/2608.13940#bib.bib44)\)\) and learned optimization \(e\.g\.,\([2](https://arxiv.org/html/2608.13940#bib.bib43)\)\)\.[29](https://arxiv.org/html/2608.13940#bib.bib48)select datasets which are predicted to be quick\-to\-run to efficiently meta\-learn an optimization algorithm,\([15](https://arxiv.org/html/2608.13940#bib.bib49)\)use a distillation objective as opposed to online evaluation due to the time\-cost of training models with every proposed algorithm, and[49](https://arxiv.org/html/2608.13940#bib.bib50)meta\-learns online reinforcement learning algorithms within a world model to accelerate experimentation beyond using slower hand\-coded simulation\. While all of these works share our concern of avoiding wasteful evaluation, none take the approach of predicting the value of a candidate solution, its code, reasoning, and partial results, to decide whichproposed algorithmto evaluate next\.

Preference, judge, and reward models\.The RPM is at its core a preference model over candidate solutions, connecting it to a broad literature on comparative evaluation\. Reward models trained from pairwise human comparisons are central to reinforcement learning from human feedback\([11](https://arxiv.org/html/2608.13940#bib.bib51);[45](https://arxiv.org/html/2608.13940#bib.bib52);[35](https://arxiv.org/html/2608.13940#bib.bib53)\)and are often formulated as paired\-comparison or ranking problems\([6](https://arxiv.org/html/2608.13940#bib.bib54);[25](https://arxiv.org/html/2608.13940#bib.bib55)\); comparable preference supervision can also be distilled from offline trajectories without reward models or environment rollouts\([9](https://arxiv.org/html/2608.13940#bib.bib58)\)\. In parallel, the LLM\-as\-a\-judge paradigm shows that pretrained models can compare candidate responses with reasonable fidelity\([56](https://arxiv.org/html/2608.13940#bib.bib10)\), and recent work turns such judgments into scalable selection:[22](https://arxiv.org/html/2608.13940#bib.bib56)derive continuous scores from scoring\-token logits to reduce ties and rank candidates without additional training, while[48](https://arxiv.org/html/2608.13940#bib.bib57)argue that verification can increasingly constrain stronger generators\. Most of these methods evaluate generated responses or recorded trajectories, whereas the RPM ranks candidate solutions before their full training\-and\-evaluation runs\.

Closest to our setting,[55](https://arxiv.org/html/2608.13940#bib.bib8)also predict a pairwise preference between unexecuted ML solutions, but condition on a separately prepared, static data report rather than on nodes from the search tree itself; the RPM instead grounds each comparison in the live search tree and the validation scores of solutions already executed\.[14](https://arxiv.org/html/2608.13940#bib.bib47)propose training a judge or reward model to select promising leaves in tree\-search research agents, but leave it unimplemented\. The RPM realizes that proposal without model training, and extends it: where agentic verifiers probe code or user interfaces\([48](https://arxiv.org/html/2608.13940#bib.bib57)\), our agentic variant runs small\-scale*pilot ML experiments*, partial training and evaluation runs, and chooses based on their measured results\.

## 7Limitations

- •While our evaluations operate under the assumption that LLM inference calls will incur negligible cost in the future, in practice, current LLM inference incurs real\-time latency\. For the Inference\-only RPM, which uses a self\-hosted Qwen3\.6\-27B, inference latency totals 0\.660 hours per 24\-hour end\-to\-end run\. Adjusting for this time budget yields a normalized score of 0\.708 at 23\.34 hours, a negligible drop from 0\.711\. Nevertheless, exact latency and monetary overhead vary by hosting infrastructure and LLM size\.
- •As the offline data come from prior greedy AIRA\-dojo runs \(with different LLM backbones and, by design, different task modalities than the online setting\), they are off\-policy and biased relative to the online target, including the subtree\-max label bias noted in[Section4](https://arxiv.org/html/2608.13940#S4), and thus our main claims rest on the end\-to\-end results\.
- •We scope RPM integration into*child creation*only\. We report initial results integrating RPMs within*final\-node selection*in[AppendixD](https://arxiv.org/html/2608.13940#A4)\. We do not observe significant improvement over validation\-based selection, given the Hidden Consistent Evaluation protocol’s strong test\-validation generalization\([19](https://arxiv.org/html/2608.13940#bib.bib15)\)\. We leave RPM integration within*parent\-selection*to future work\.
- •We describe the RPM as scaffold\-agnostic because it inspects no scaffold\-internal state, but we demonstrate it only in AIRA\-dojo’s child\-selection step, with a single backbone \(Qwen3\.6\-27B\) on a single benchmark\. The agentic variant additionally needs a sandboxed clone of the execution environment in which to run pilot experiments\. We see no reason the approach would not transfer, but portability to other scaffolds and backbones is part of our future work\.

## 8Conclusion

An AI research agent can propose a candidate solution far faster than it can determine whether that solution works\. We introduced theAI Research Preference Model \(RPM\), which ranks unexecuted candidates so that the agent can direct its execution budget toward promising candidates\. This formulation avoids requiring the model to forecast absolute outcomes: it identifies the most promising candidate without taking on the challenging task of predicting what any candidate would score\.

We built RPMs from frozen pretrained models in two variants: an inference\-only RPM and an agentic RPM that runs its own pilot experiments\. Both are designed to be portable across search scaffolds, the inference\-only variant requiring only candidate solutions and a record of prior evaluations, and the agentic variant additionally requiring a sandbox\. By replacing random selection among generated candidates with RPM\-guided selection, the two variants raise AIRA\-dojo’s average normalized score onAIRS\-Benchfrom0\.6840\.684to0\.7110\.711and0\.7290\.729, respectively, using the same backbone and mutation operators as the random\-selection baseline\. Both also reach the baseline’s 24\-hour score in roughly 15 hours, with less than two\-thirds of its execution budget, and yield new state\-of\-the\-art results on two individualAIRS\-Benchtasks\.

More broadly, our work supports treating candidate selection as an explicit use of test\-time compute\. Agents can invest computation not only in generating candidates, but also in deciding which candidates are worth executing\. This is especially useful when evaluation is much more expensive than generation\. We hope this work encourages research agents that choose which solutions to run as carefully as they design them\.

## References

- Agarwalet al\.\(2021\)R\. Agarwal, M\. Schwarzer, P\. S\. Castro, A\. Courville, and M\. G\. BellemareDeep reinforcement learning at the edge of the statistical precipice\.Advances in Neural Information Processing Systems\.Cited by:[§5\.1](https://arxiv.org/html/2608.13940#S5.SS1.SSS0.Px1.p1.1)\.
- Andrychowiczet al\.\(2016\)M\. Andrychowicz, M\. Denil, S\. Gomez, M\. W\. Hoffman, D\. Pfau, T\. Schaul, B\. Shillingford, and N\. de FreitasLearning to learn by gradient descent by gradient descent\.InAdvances in Neural Information Processing Systems,Vol\.29\.Cited by:[§6](https://arxiv.org/html/2608.13940#S6.p2.1)\.
- Anthropic \(2025\)AnthropicClaude code overview\.Note:[https://code\.claude\.com/docs/en/overview](https://code.claude.com/docs/en/overview)Cited by:[§2\.2](https://arxiv.org/html/2608.13940#S2.SS2.p1.1)\.
- Anthropic \(2026\)AnthropicIntroducing claude opus 4\.8\(Website\)External Links:[Link](https://www.anthropic.com/news/claude-opus-4-8)Cited by:[§A\.2](https://arxiv.org/html/2608.13940#A1.SS2.p1.1),[§2\.2](https://arxiv.org/html/2608.13940#S2.SS2.p1.1)\.
- Baeket al\.\(2025\)J\. Baek, S\. K\. Jauhar, S\. Cucerzan, and S\. J\. HwangResearchAgent: iterative research idea generation over scientific literature with large language models\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 6709–6738\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.342),[Link](https://aclanthology.org/2025.naacl-long.342/)Cited by:[§6](https://arxiv.org/html/2608.13940#S6.p1.1)\.
- Bradley and Terry \(1952\)R\. A\. Bradley and M\. E\. TerryRank analysis of incomplete block designs: i\. the method of paired comparisons\.Biometrika39\(3/4\),pp\. 324–345\.Cited by:[§6](https://arxiv.org/html/2608.13940#S6.p3.1)\.
- Chanet al\.\(2025\)J\. S\. Chan, N\. Chowdhury, O\. Jaffe, J\. Aung, D\. Sherburn, E\. Mays, G\. Starace, K\. Liu, L\. Maksin, T\. Patwardhan, A\. Madry, and L\. WengMLE\-bench: evaluating machine learning agents on machine learning engineering\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=6s5uXNWGIh)Cited by:[§1](https://arxiv.org/html/2608.13940#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.13940#S2.SS1.p1.1)\.
- Chenet al\.\(2026\)J\. Chen, B\. D\. Mishra, J\. Nam, R\. Meng, T\. Pfister, and J\. YoonMARS: modular agent with reflective search for automated ai research\.arXiv preprint arXiv:2602\.02660\.Cited by:[§6](https://arxiv.org/html/2608.13940#S6.p1.1)\.
- Chen and Yuille \(2026\)Y\. Chen and A\. YuilleAgentic\-DPO: from imitation to agentic policy optimization on expert trajectories\.External Links:2607\.10601,[Link](https://arxiv.org/abs/2607.10601)Cited by:[§6](https://arxiv.org/html/2608.13940#S6.p3.1)\.
- Chenet al\.\(2022\)Y\. Chen, X\. Song, C\. Lee, Z\. Wang, Q\. Zhang, D\. Dohan, K\. Kawakami, G\. Kochanski, A\. Doucet, M\. Ranzato, S\. Perel, and N\. de FreitasTowards learning universal hyperparameter optimizers with transformers\.InAdvances in Neural Information Processing Systems,Vol\.35\.Cited by:[§6](https://arxiv.org/html/2608.13940#S6.p2.1)\.
- Christianoet al\.\(2017\)P\. F\. Christiano, J\. Leike, T\. Brown, M\. Martic, S\. Legg, and D\. AmodeiDeep reinforcement learning from human preferences\.InAdvances in Neural Information Processing Systems,Cited by:[§6](https://arxiv.org/html/2608.13940#S6.p3.1)\.
- Copetet al\.\(2025\)J\. Copet, Q\. Carbonneaux, G\. Cohen, J\. Gehring, J\. Kahn, J\. Kossen, F\. Kreuk, E\. McMilin, M\. Meyer, Y\. Wei, D\. Zhang,et al\.CWM: An open\-weights LLM for research on code generation with world models\.arXiv preprint arXiv:2510\.02387\.Cited by:[§4\.2](https://arxiv.org/html/2608.13940#S4.SS2.p1.1)\.
- Falkneret al\.\(2018\)S\. Falkner, A\. Klein, and F\. HutterBOHB: robust and efficient hyperparameter optimization at scale\.InProceedings of the 35th International Conference on Machine Learning,pp\. 1437–1446\.Cited by:[§6](https://arxiv.org/html/2608.13940#S6.p2.1)\.
- Goldieet al\.\(2026\)A\. D\. Goldie, Z\. Wang, A\. Hayler, D\. Nathani, E\. Toledo, K\. Thampiratwong, A\. Kalisz, M\. Beukman, H\. Erlebach, A\. Letcher, S\. Reddy, C\. Wibault, T\. Wolf, C\. O’Neill, U\. Berdica, N\. Roberts, S\. Rahmani, R\. Raileanu, S\. Whiteson, and J\. N\. FoersterDiscoGen: procedural generation of algorithm discovery tasks in machine learning\.External Links:2603\.17863,[Link](https://arxiv.org/abs/2603.17863)Cited by:[§6](https://arxiv.org/html/2608.13940#S6.p4.1)\.
- Goldieet al\.\(2025\)A\. D\. Goldie, Z\. W\. J\. Cohen, J\. N\. Foerster, and S\. WhitesonHow should we meta\-learn reinforcement learning algorithms?\.InReinforcement Learning Conference,Cited by:[§6](https://arxiv.org/html/2608.13940#S6.p2.1)\.
- Google DeepMind \(2026\)Google DeepMindGemini 3\.1 pro: a smarter model for your most complex tasks\.Note:[https://blog\.google/innovation\-and\-ai/models\-and\-research/gemini\-models/gemini\-3\-1\-pro/](https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-1-pro/)Accessed: 2026\-07\-19Cited by:[§A\.2](https://arxiv.org/html/2608.13940#A1.SS2.p1.1)\.
- Ha and Schmidhuber \(2018\)D\. Ha and J\. SchmidhuberRecurrent world models facilitate policy evolution\.InAdvances in Neural Information Processing Systems,Vol\.31\.Cited by:[§6](https://arxiv.org/html/2608.13940#S6.p2.1)\.
- Hafneret al\.\(2023\)D\. Hafner, J\. Pasukonis, J\. Ba, and T\. LillicrapMastering diverse domains through world models\.arXiv preprint arXiv:2301\.04104\.Cited by:[§6](https://arxiv.org/html/2608.13940#S6.p2.1)\.
- Hambardzumyanet al\.\(2026\)K\. Hambardzumyan, N\. Baldwin, E\. Toledo, R\. Hazra, M\. Kuchnik, B\. A\. Omari, T\. S\. Foster, A\. Protopopov, J\. Gagnon\-Audet, I\. Mediratta, K\. Niu, M\. Shvartsman, A\. Lupidi, A\. Audran\-Reiss, P\. Pathak, T\. Shavrina, D\. Magka, H\. Momand, D\. Dunfield, N\. Cancedda, P\. Stenetorp, C\. Wu, J\. N\. Foerster, Y\. Bachrach, and M\. JosifoskiAIRA\_2: overcoming bottlenecks in ai research agents\.External Links:2603\.26499,[Link](https://arxiv.org/abs/2603.26499)Cited by:[§1](https://arxiv.org/html/2608.13940#S1.p2.1),[§6](https://arxiv.org/html/2608.13940#S6.p1.1),[3rd item](https://arxiv.org/html/2608.13940#S7.I1.i3.p1.1)\.
- Jaderberget al\.\(2017\)M\. Jaderberg, V\. Dalibard, S\. Osindero, W\. M\. Czarnecki, J\. Donahue, A\. Razavi, O\. Vinyals, T\. Green, I\. Dunning, K\. Simonyan, C\. Fernando, and K\. KavukcuogluPopulation based training of neural networks\.arXiv preprint arXiv:1711\.09846\.Cited by:[§6](https://arxiv.org/html/2608.13940#S6.p2.1)\.
- Jimenezet al\.\(2024\)C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. R\. NarasimhanSWE\-bench: can language models resolve real\-world github issues?\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=VTF8yNQM66)Cited by:[§1](https://arxiv.org/html/2608.13940#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.13940#S2.SS1.p1.1)\.
- Kwoket al\.\(2026\)J\. Kwok, S\. Li, P\. Atreya, Y\. Liu, Y\. Jiang, C\. Finn, M\. Pavone, I\. Stoica, and A\. MirhoseiniLLM\-as\-a\-Verifier: a general\-purpose verification framework\.External Links:2607\.05391,[Link](https://arxiv.org/abs/2607.05391)Cited by:[§6](https://arxiv.org/html/2608.13940#S6.p3.1)\.
- Liet al\.\(2018\)L\. Li, K\. Jamieson, G\. DeSalvo, A\. Rostamizadeh, and A\. TalwalkarHyperband: a novel bandit\-based approach to hyperparameter optimization\.Journal of Machine Learning Research18\(185\),pp\. 1–52\.Cited by:[§6](https://arxiv.org/html/2608.13940#S6.p2.1)\.
- Liet al\.\(2024\)Z\. Li, Q\. Zang, D\. Ma, J\. Guo, T\. Zheng, M\. Liu, X\. Niu, Y\. Wang, J\. Yang, J\. Liu, W\. Zhong, W\. Zhou, W\. Huang, and G\. ZhangAutoKaggle: a multi\-agent framework for autonomous data science competitions\.External Links:2410\.20424,[Link](https://arxiv.org/abs/2410.20424)Cited by:[§6](https://arxiv.org/html/2608.13940#S6.p1.1)\.
- Liu \(2009\)T\. LiuLearning to rank for information retrieval\.Foundations and Trends in Information Retrieval3\(3\),pp\. 225–331\.Cited by:[§6](https://arxiv.org/html/2608.13940#S6.p3.1)\.
- Luet al\.\(2024\)C\. Lu, C\. Lu, R\. T\. Lange, J\. Foerster, J\. Clune, and D\. HaThe ai scientist: towards fully automated open\-ended scientific discovery\.External Links:2408\.06292,[Link](https://arxiv.org/abs/2408.06292)Cited by:[§1](https://arxiv.org/html/2608.13940#S1.p2.1),[§6](https://arxiv.org/html/2608.13940#S6.p1.1)\.
- Lupidiet al\.\(2026\)A\. Lupidi, B\. Gauri, T\. S\. Foster, B\. A\. Omari, D\. Magka, A\. Pepe, A\. Audran\-Reiss, M\. Aghamelu, N\. Baldwin, L\. Cipolina\-Kun,et al\.AIRS\-bench: a suite of tasks for frontier ai research science agents\.arXiv preprint arXiv:2602\.06855\.Cited by:[1st item](https://arxiv.org/html/2608.13940#S1.I1.i1.p1.1),[§2\.1](https://arxiv.org/html/2608.13940#S2.SS1.p2.1),[§5\.1](https://arxiv.org/html/2608.13940#S5.SS1.SSS0.Px2.p1.1)\.
- Merrillet al\.\(2026\)M\. A\. Merrill, A\. G\. Shaw, N\. Carlini, B\. Li, H\. Raj, I\. Bercovich, L\. Shi, J\. Y\. Shin, T\. Walshe, E\. K\. Buchanan, J\. Shen, G\. Ye, H\. Lin, J\. Poulos, M\. Wang, M\. Nezhurina, J\. Jitsev, D\. Lu, O\. M\. Mastromichalakis, Z\. Xu, Z\. Chen, Y\. Liu, R\. Zhang, L\. L\. Chen, A\. Kashyap, J\. Uslu, J\. Li, J\. Wu, M\. Yan, S\. Bian, V\. Sharma, K\. Sun, S\. Dillmann, A\. Anand, A\. Lanpouthakoun, B\. Koopah, C\. Hu, E\. Guha, G\. H\. S\. Dreiman, J\. Zhu, K\. Krauth, L\. Zhong, N\. Muennighoff, R\. Amanfu, S\. Tan, S\. Pimpalgaonkar, T\. Aggarwal, X\. Lin, X\. Lan, X\. Zhao, Y\. Liang, Y\. Wang, Z\. Wang, C\. Zhou, D\. Heineman, H\. Liu, H\. Trivedi, J\. Yang, J\. Lin, M\. Shetty, M\. Yang, N\. Omi, N\. Raoof, S\. Li, T\. Y\. Zhuo, W\. Lin, Y\. Dai, Y\. Wang, W\. Chai, S\. Zhou, D\. Wahdany, Z\. She, J\. Hu, Z\. Dong, Y\. Zhu, S\. Cui, A\. Saiyed, A\. Kolbeinsson, J\. Hu, C\. M\. Rytting, R\. Marten, Y\. Wang, A\. Dimakis, A\. Konwinski, and L\. SchmidtTerminal\-bench: benchmarking agents on hard, realistic tasks in command line interfaces\.External Links:2601\.11868,[Link](https://arxiv.org/abs/2601.11868)Cited by:[§1](https://arxiv.org/html/2608.13940#S1.p1.1)\.
- Metzet al\.\(2022\)L\. Metz, J\. Harrison, C\. D\. Freeman, A\. Merchant, L\. Beyer, J\. Bradbury, N\. Agrawal, B\. Poole, I\. Mordatch, A\. Roberts, and J\. Sohl\-DicksteinVeLO: training versatile learned optimizers by scaling up\.External Links:2211\.09760,[Link](https://arxiv.org/abs/2211.09760)Cited by:[§6](https://arxiv.org/html/2608.13940#S6.p2.1)\.
- Namet al\.\(2025\)J\. Nam, J\. Yoon, J\. Chen, J\. Shin, S\. O\. Arik, and T\. PfisterMLE\-STAR: machine learning engineering agent via search and targeted refinement\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=vS1M06Px6u)Cited by:[§6](https://arxiv.org/html/2608.13940#S6.p1.1)\.
- Nathaniet al\.\(2025\)D\. Nathani, L\. Madaan, N\. Roberts, N\. Bashlykov, A\. Menon, V\. Moens, A\. Budhiraja, D\. Magka, V\. Vorotilov, G\. Chaurasia, D\. Hupkes, R\. S\. Cabral, T\. Shavrina, J\. Foerster, Y\. Bachrach, W\. Y\. Wang, and R\. RaileanuMLGym: a new framework and benchmark for advancing ai research agents\.External Links:2502\.14499,[Link](https://arxiv.org/abs/2502.14499)Cited by:[§2\.2](https://arxiv.org/html/2608.13940#S2.SS2.p3.1)\.
- OpenAI \(2024\)OpenAIGPT\-4o System Card\(Website\)Note:Accessed: 2024\-06\-07External Links:[Link](https://cdn.openai.com/gpt-4o-system-card.pdf)Cited by:[§4\.2](https://arxiv.org/html/2608.13940#S4.SS2.p1.1)\.
- OpenAI \(2025\)OpenAIIntroducing gpt\-oss\(Website\)Note:Accessed: 2026\-08\-12External Links:[Link](https://openai.com/index/introducing-gpt-oss/)Cited by:[§4\.2](https://arxiv.org/html/2608.13940#S4.SS2.p1.1)\.
- Opsahl\-Onget al\.\(2024\)K\. Opsahl\-Ong, M\. J\. Ryan, J\. Purtell, D\. Broman, C\. Potts, M\. Zaharia, and O\. KhattabOptimizing instructions and demonstrations for multi\-stage language model programs\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 9340–9366\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.525/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.525)Cited by:[§3\.1](https://arxiv.org/html/2608.13940#S3.SS1.p2.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. L\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.Training language models to follow instructions with human feedback\.InAdvances in Neural Information Processing Systems,Cited by:[§6](https://arxiv.org/html/2608.13940#S6.p3.1)\.
- Patelet al\.\(2021\)A\. Patel, S\. Bhattamishra, and N\. GoyalAre nlp models really able to solve simple math word problems?\.InProceedings of the 2021 conference of the North American chapter of the association for computational linguistics: human language technologies,pp\. 2080–2094\.Cited by:[§5\.1](https://arxiv.org/html/2608.13940#S5.SS1.SSS0.Px2.p1.1)\.
- Qwen Team \(2024\)Qwen TeamQwen2\.5: a party of foundation models\.External Links:[Link](https://qwenlm.github.io/blog/qwen2.5/)Cited by:[§5\.1](https://arxiv.org/html/2608.13940#S5.SS1.SSS0.Px2.p1.1)\.
- Qwen Team \(2026\)Qwen TeamQwen3\.6\-27B: flagship\-level coding in a 27B dense model\.External Links:[Link](https://qwen.ai/blog?id=qwen3.6-27b)Cited by:[§4\.1](https://arxiv.org/html/2608.13940#S4.SS1.p1.1)\.
- Renet al\.\(2025\)S\. Ren, C\. Xie, P\. Jian, Z\. Ren, C\. Leng, and J\. ZhangTowards scientific intelligence: a survey of llm\-based scientific agents\.External Links:2503\.24047,[Link](https://arxiv.org/abs/2503.24047)Cited by:[§6](https://arxiv.org/html/2608.13940#S6.p1.1)\.
- Sakaguchiet al\.\(2021\)K\. Sakaguchi, R\. L\. Bras, C\. Bhagavatula, and Y\. ChoiWinoGrande: an adversarial winograd schema challenge at scale\.Communications of the ACM64\(9\),pp\. 99–106\.Cited by:[§5\.1](https://arxiv.org/html/2608.13940#S5.SS1.SSS0.Px2.p1.1)\.
- Schmidgallet al\.\(2025\)S\. Schmidgall, Y\. Su, Z\. Wang, X\. Sun, J\. Wu, X\. Yu, J\. Liu, M\. Moor, Z\. Liu, and E\. BarsoumAgent laboratory: using llm agents as research assistants\.External Links:2501\.04227,[Link](https://arxiv.org/abs/2501.04227)Cited by:[§6](https://arxiv.org/html/2608.13940#S6.p1.1)\.
- Schrittwieseret al\.\(2020\)J\. Schrittwieser, I\. Antonoglou, T\. Hubert, K\. Simonyan, L\. Sifre, S\. Schmitt, A\. Guez, E\. Lockhart, D\. Hassabis, T\. Graepel, T\. Lillicrap, and D\. SilverMastering atari, go, chess and shogi by planning with a learned model\.Nature588,pp\. 604–609\.Cited by:[§6](https://arxiv.org/html/2608.13940#S6.p2.1)\.
- Singhet al\.\(2025\)A\. Singh, A\. Fry, A\. Perelman, A\. Tart, A\. Ganesh, A\. El\-Kishky, A\. McLaughlin, A\. Low, A\. Ostrow, A\. Ananthram,et al\.Openai gpt\-5 system card\.arXiv preprint arXiv:2601\.03267\.Cited by:[§A\.2](https://arxiv.org/html/2608.13940#A1.SS2.p1.1),[§2\.2](https://arxiv.org/html/2608.13940#S2.SS2.p1.1)\.
- Snoeket al\.\(2012\)J\. Snoek, H\. Larochelle, and R\. P\. AdamsPractical bayesian optimization of machine learning algorithms\.InAdvances in Neural Information Processing Systems,Vol\.25\.Cited by:[§6](https://arxiv.org/html/2608.13940#S6.p2.1)\.
- Stiennonet al\.\(2020\)N\. Stiennon, L\. Ouyang, J\. Wu, D\. M\. Ziegler, R\. Lowe, C\. Voss, A\. Radford, D\. Amodei, and P\. F\. ChristianoLearning to summarize with human feedback\.InAdvances in Neural Information Processing Systems,Cited by:[§6](https://arxiv.org/html/2608.13940#S6.p3.1)\.
- Sutton \(1991\)R\. S\. SuttonDyna, an integrated architecture for learning, planning, and reacting\.InProceedings of the Seventh International Conference on Machine Learning,pp\. 216–224\.Cited by:[§6](https://arxiv.org/html/2608.13940#S6.p2.1)\.
- Toledoet al\.\(2025\)E\. Toledo, K\. Hambardzumyan, M\. Josifoski, R\. Hazra, N\. Baldwin, A\. Audran\-Reiss, M\. Kuchnik, D\. Magka, M\. Jiang, A\. M\. Lupidi, A\. Lupu, R\. Raileanu, T\. Shavrina, K\. Niu, J\. Gagnon\-Audet, M\. Shvartsman, S\. Sodhani, A\. H\. Miller, A\. Charnalia, D\. Dunfield, C\. Wu, P\. Stenetorp, N\. Cancedda, J\. N\. Foerster, and Y\. BachrachAI research agents for machine learning: search, exploration, and generalization in MLE\-bench\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=RwfrdKSgCE)Cited by:[§1](https://arxiv.org/html/2608.13940#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.13940#S2.SS2.p1.1),[§6](https://arxiv.org/html/2608.13940#S6.p1.1)\.
- Wanget al\.\(2026\)B\. Wang, C\. Zhang, D\. Liu, J\. Zhang, J\. Chen, M\. Li, M\. Chen, R\. Fang, S\. Zhang, X\. Wang, Y\. Jing, Z\. Ma, and Z\. CuiThe verification horizon: no silver bullet for coding agent rewards\.External Links:2606\.26300,[Link](https://arxiv.org/abs/2606.26300)Cited by:[§6](https://arxiv.org/html/2608.13940#S6.p3.1),[§6](https://arxiv.org/html/2608.13940#S6.p4.1)\.
- Wolfet al\.\(2026\)T\. Wolf, A\. D\. Goldie, J\. L\. Liesen, U\. Berdica, M\. Fellows, and J\. N\. FoersterModel\-based meta\-learning for algorithm discovery\.InICLR 2026 the 2nd Workshop on World Models: Understanding, Modelling and Scaling,Cited by:[§6](https://arxiv.org/html/2608.13940#S6.p2.1)\.
- Wooldridge and Jennings \(1995\)M\. J\. Wooldridge and N\. R\. JenningsIntelligent Agents: Theory and Practice\.The Knowledge Engineering Review10\(2\),pp\. 115–152\.Cited by:[§2](https://arxiv.org/html/2608.13940#S2.p1.1)\.
- Yamadaet al\.\(2025\)Y\. Yamada, R\. T\. Lange, C\. Lu, S\. Hu, C\. Lu, J\. Foerster, J\. Clune, and D\. HaThe AI Scientist\-v2: Workshop\-Level Automated Scientific Discovery via Agentic Tree Search\.External Links:2504\.08066,[Link](https://arxiv.org/abs/2504.08066)Cited by:[§6](https://arxiv.org/html/2608.13940#S6.p1.1)\.
- Yanget al\.\(2024\)J\. Yang, C\. E\. Jimenez, A\. Wettig, K\. Lieret, S\. Yao, K\. Narasimhan, and O\. PressSWE\-agent: agent\-computer interfaces enable automated software engineering\.External Links:2405\.15793,[Link](https://arxiv.org/abs/2405.15793)Cited by:[§6](https://arxiv.org/html/2608.13940#S6.p1.1)\.
- Yaoet al\.\(2023\)S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. CaoReAct: synergizing reasoning and acting in language models\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§3\.2](https://arxiv.org/html/2608.13940#S3.SS2.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2026\)Y\. Zhang, K\. Zhou, Z\. Xu, K\. Ramnath, Y\. Zhou, S\. Woo, H\. Ding, and L\. L\. CheongLearning to ideate for machine learning engineering agents\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics, Volume 2: Short Papers,pp\. 436–447\.Cited by:[§6](https://arxiv.org/html/2608.13940#S6.p1.1)\.
- Zhenget al\.\(2026\)J\. Zheng, J\. Zhang, Y\. Luo, Y\. Mao, Y\. Gao, L\. Du, H\. Chen, and N\. ZhangCan we predict before executing machine learning agents?\.External Links:2601\.05930,[Link](https://arxiv.org/abs/2601.05930)Cited by:[§6](https://arxiv.org/html/2608.13940#S6.p4.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.Advances in neural information processing systems36,pp\. 46595–46623\.Cited by:[§3\.1](https://arxiv.org/html/2608.13940#S3.SS1.p1.1),[§6](https://arxiv.org/html/2608.13940#S6.p3.1)\.
- Zhenget al\.\(2025\)T\. Zheng, Z\. Deng, H\. T\. Tsang, W\. Wang, J\. Bai, Z\. Wang, and Y\. SongFrom automation to autonomy: a survey on large language models in scientific discovery\.External Links:2505\.13259,[Link](https://arxiv.org/abs/2505.13259)Cited by:[§6](https://arxiv.org/html/2608.13940#S6.p1.1)\.
- Zhonget al\.\(2026\)Q\. Zhong, K\. Wang, Z\. Xu, L\. Ding, J\. Liu, and B\. DuAchieving\> 97% on gsm8k: deeply understanding the problems makes llms better solvers for math word problems\.Frontiers of Computer Science20\(1\),pp\. 1–3\.Cited by:[§5\.1](https://arxiv.org/html/2608.13940#S5.SS1.SSS0.Px2.p1.1)\.

## Appendix AInference\-Only RPM

### A\.1Prompt Optimization

We optimize the ranking system prompt at inference time using Automatic Prompt Optimization \(APO\) with no underlying weight updates\. We employ GPT\-5 as both the candidate ranker and the meta\-proposer over a pairwise branch\-ranking dataset drawn fromAIRS\-Benchtrajectory traces\.

#### A\.1\.1Optimization Configuration

The optimization follows a MIPROv2\-style pipeline\. Starting from a baseline "strict judge" template, the meta\-proposer generates 10 candidate prompt variations\. The search space is explored via Thompson sampling over per\-candidateBeta​\(1,1\)\\text\{Beta\}\(1,1\)accuracy posteriors across 40 Bayesian minibatch trials\. Underperforming templates are progressively pruned, while top\-performing survivors undergo deeper validation runs to mitigate optimization\-to\-holdout shrinkage\.

The optimization routine consistently converges on aPrincipal Investigator \(PI\)persona structured around an ordered, five\-criterion evaluation rubric\. This process introduces several key shifts from the initial baseline:

- •Structural Evaluation:Moves from a flat list of decision rules to a structured walkthrough forcing the model to sequentially score problem\-model fit, extensibility, learning\-curve projection, context\-informed novelty, and risk\-adjusted potential\.
- •Shifting Bug Tolerance:Transitions from binary bug intolerance \(“a buggy candidate has no potential”\) to evaluating systemic soundness, prioritizing promising\-but\-rough directions over low\-ceiling code\.
- •Strategic Context Utility:Sharpens context node usage from generic historical summaries into an explicit mandate to discover open search gaps and actively penalize redundant directions\.

#### A\.1\.2Results

Table 1:Prompt optimization performance evaluated on a matched dataset \(n=990n=990\)\. Accuracy denotes the fraction of correct higher\-potential candidate selections\.Table[1](https://arxiv.org/html/2608.13940#A1.T1)summarizes the performance gains from the offline prompt search\. The optimized PI rubric provides a consistent uplift, improving mean single\-sample accuracy by\+0\.7​pp\+0\.7\\text\{ pp\}and majority\-vote accuracy by\+1\.2​pp\+1\.2\\text\{ pp\}\. This discovered prompt layout is hand\-carried directly into the core ranking agent\.

#### A\.1\.3Prompt Templates

##### Baseline Template \(Before\)

[⬇](data:text/plain;base64,WW91IGFyZSBhIHN0cmljdCBqdWRnZSBzZWxlY3RpbmcgYmV0d2VlbiBUV08gY2FuZGlkYXRlIHNvbHV0aW9ucyB0byB0aGUgU0FNRSBtYWNoaW5lIGxlYXJuaW5nIHRhc2suCgpZb3VyIGdvYWwgaXMgdG8gY2hvb3NlIHRoZSBjYW5kaWRhdGUgd2hvc2UgZGlyZWN0aW9uIG9mIGV4cGxvcmF0aW9uIGlzIG1vcmUgbGlrZWx5IHRvIGV2ZW50dWFsbHkgbGVhZCB0byBhIGJldHRlciBsb25nLXRlcm0gYmVzdCB0ZXN0IHNjb3JlLCBldmVuIGlmIGZ1cnRoZXIgcmVmaW5lbWVudHMgYXJlIG5lZWRlZC4gRm9jdXMgb24gbG9uZy10ZXJtIHBvdGVudGlhbCByYXRoZXIgdGhhbiBpbW1lZGlhdGUgcGVyZm9ybWFuY2UuCgpUYXNrIGRlc2NyaXB0aW9uOgpgYGBtYXJrZG93bgp7dGFza19kZXNjfQoKQ29udGV4dCBmcm9tIHZhcmlvdXMgc29sdXRpb25zIHRvIHRoZSBzYW1lIG1hY2hpbmUgbGVhcm5pbmcgdGFzay4gVGhlc2UgYXJlIE5PVCB0aGUgY2FuZGlkYXRlcyB5b3UgYXJlIGp1ZGdpbmcuCntjb250ZXh0X3RleHR9CgpDYW5kaWRhdGUgQSAtLSBQbGFuOgp7cGxhbl9BfQpDYW5kaWRhdGUgQSAtLSBDb2RlOgp7Y29kZV9BfQoKQ2FuZGlkYXRlIEIgLS0gUGxhbjoKe3BsYW5fQn0KQ2FuZGlkYXRlIEIgLS0gQ29kZToKe2NvZGVfQn0KCkRlY2lzaW9uIHJ1bGVzOgoKVGhpbmsgYWJvdXQgd2hpY2ggY2FuZGlkYXRlIG9wZW5zIHVwIGEgbW9yZSBwcm9taXNpbmcgc2VhcmNoIGRpcmVjdGlvbiBmb3IgZnV0dXJlIGl0ZXJhdGlvbnMuCgpQcmVmZXIgdGhlIGNhbmRpZGF0ZSB0aGF0IGxheXMgYmV0dGVyIGdyb3VuZHdvcmsgZm9yIGV2ZW50dWFsbHkgYWNoaWV2aW5nIHRoZSBiZXN0IHBvc3NpYmxlIGxvbmctdGVybSBiZXN0IHRlc3Qgc2NvcmUsIG5vdCBqdXN0IHRoZSBvbmUgdGhhdCBsb29rcyBiZXR0ZXIgcmlnaHQgbm93LgoKQ29uc2lkZXIgd2hldGhlciB0aGUgYXBwcm9hY2ggaXMgZXh0ZW5zaWJsZSwgbW9kdWxhciwgYW5kIGFtZW5hYmxlIHRvIGl0ZXJhdGl2ZSBpbXByb3ZlbWVudC4KClByZWZlciBjb3JyZWN0bmVzcyBhbmQgcm9idXN0bmVzcyAtLSBhIGJ1Z2d5IGNhbmRpZGF0ZSBoYXMgbm8gbG9uZy10ZXJtIHBvdGVudGlhbC4KClVzZSBjb250ZXh0IG5vZGVzIGFzIGV2aWRlbmNlIG9mIHdoYXQgaGFzIGFscmVhZHkgYmVlbiB0cmllZCBhbmQgd2hhdCBkaXJlY3Rpb25zIGhhdmUgc2hvd24gcHJvbWlzZS4KCkRvIG5vdCBhc3N1bWUgdGhlIGNvbnRleHQgbm9kZXMgYXJlIG9wdGltYWw7IHRoZSBuZXcgY2FuZGlkYXRlcyBtYXkgb3BlbiBiZXR0ZXIgcGF0aHMuCgpPdXRwdXQgZm9ybWF0IChTVFJJQ1QpOgoKVGhpbmsgc3RlcCBieSBzdGVwIGFuZCBwcm92aWRlIHlvdXIgcmVhc29uaW5nIGJlZm9yZSBnaXZpbmcgYSBmaW5hbCBhbnN3ZXIuCgpHaXZlIGEgZmluYWwgYW5zd2VyIG9mIEEgZm9yIENhbmRpZGF0ZSBBIGFuZCBCIGZvciBDYW5kaWRhdGUgQi4KClByb3ZpZGUgeW91ciBhbnN3ZXIgaW5zaWRlIGEgXGJveGVke3t9fSwgaWUgXGJveGVke3tBfX0gb3IgXGJveGVke3tCfX0u)YouareastrictjudgeselectingbetweenTWOcandidatesolutionstotheSAMEmachinelearningtask\.Yourgoalistochoosethecandidatewhosedirectionofexplorationismorelikelytoeventuallyleadtoabetterlong\-termbesttestscore,eveniffurtherrefinementsareneeded\.Focusonlong\-termpotentialratherthanimmediateperformance\.Taskdescription:‘‘‘markdown\{task\_desc\}Contextfromvarioussolutionstothesamemachinelearningtask\.TheseareNOTthecandidatesyouarejudging\.\{context\_text\}CandidateA\-\-Plan:\{plan\_A\}CandidateA\-\-Code:\{code\_A\}CandidateB\-\-Plan:\{plan\_B\}CandidateB\-\-Code:\{code\_B\}Decisionrules:Thinkaboutwhichcandidateopensupamorepromisingsearchdirectionforfutureiterations\.Preferthecandidatethatlaysbettergroundworkforeventuallyachievingthebestpossiblelong\-termbesttestscore,notjusttheonethatlooksbetterrightnow\.Considerwhethertheapproachisextensible,modular,andamenabletoiterativeimprovement\.Prefercorrectnessandrobustness\-\-abuggycandidatehasnolong\-termpotential\.Usecontextnodesasevidenceofwhathasalreadybeentriedandwhatdirectionshaveshownpromise\.Donotassumethecontextnodesareoptimal;thenewcandidatesmayopenbetterpaths\.Outputformat\(STRICT\):Thinkstepbystepandprovideyourreasoningbeforegivingafinalanswer\.GiveafinalanswerofAforCandidateAandBforCandidateB\.Provideyouranswerinsidea\\boxed\{\{\}\},ie\\boxed\{\{A\}\}or\\boxed\{\{B\}\}\.

##### Optimized Template \(After\)

[⬇](data:text/plain;base64,WW91IGFyZSBhIHByaW5jaXBhbCBpbnZlc3RpZ2F0b3IgYWxsb2NhdGluZyBjb21wdXRlIGJ1ZGdldCB0byBvbmUgb2YgdHdvIGJyYW5jaGVzLiBEZWNpZGUgd2hpY2ggYnJhbmNoIGlzIG1vcmUgbGlrZWx5IHRvIHlpZWxkIHRoZSBiZXN0IGV2ZW50dWFsIHRlc3Qgc2NvcmUgYWZ0ZXIgc2V2ZXJhbCBpdGVyYXRpb25zLiBFbXBoYXNpemUgZXh0ZW5zaWJpbGl0eSwgZml4YWJpbGl0eSwgYW5kIHByb21pc2UgcmVsYXRpdmUgdG8gd2hhdCBoYXMgYWxyZWFkeSBiZWVuIHRyaWVkLgoKVGFzayBkZXNjcmlwdGlvbjoKe3Rhc2tfZGVzY30KCkNvbnRleHQgZnJvbSBvdGhlciBzb2x1dGlvbnMgYW5kIHRoZWlyIHNjb3JlcyAobm90IHRoZSBjYW5kaWRhdGVzKS4gVXNlIHRoaXMgdG8gaWRlbnRpZnkgcHJvbWlzaW5nIGdhcHMgYW5kIGF2b2lkIHJlZHVuZGFudCBkaXJlY3Rpb25zOgp7Y29udGV4dF90ZXh0fQoKQ2FuZGlkYXRlIEEgLS0gUGxhbjoKe3BsYW5fQX0KQ2FuZGlkYXRlIEEgLS0gQ29kZToKe2NvZGVfQX0KCkNhbmRpZGF0ZSBCIC0tIFBsYW46CntwbGFuX0J9CkNhbmRpZGF0ZSBCIC0tIENvZGU6Cntjb2RlX0J9CgpTdGVwLWJ5LXN0ZXAgZXZhbHVhdGlvbjoKClByb2JsZW0tbW9kZWwgZml0OgoKRG9lcyBlYWNoIGNhbmRpZGF0ZSdzIGZvcm11bGF0aW9uIGFuZCBvYmplY3RpdmUgYWxpZ24gd2l0aCB0aGUgdGFzaz8gQW55IHJpc2tzIG9mIGxlYWthZ2Ugb3IgbWlzYWxpZ25tZW50PyBOb3RlIHdoaWNoIGlzc3VlcyBhcmUgdHJpdmlhbGx5IGZpeGFibGUgdnMuIGZ1bmRhbWVudGFsLgoKRXh0ZW5zaWJpbGl0eSBhbmQgdXBncmFkZSBwYXRoOgoKSG93IG1vZHVsYXIgaXMgdGhlIGNvZGU/IEhvdyBzdHJhaWdodGZvcndhcmQgaXMgaXQgdG8gYWRkIHN0cm9uZ2VyIG1vZGVscywgZmVhdHVyZXMsIG9yIHRyYWluaW5nIHN0cmF0ZWdpZXMgaW4gMSB0byAzIGl0ZXJhdGlvbnM/CgpMZWFybmluZyBjdXJ2ZSBwcm9qZWN0aW9uOgoKQmFzZWQgb24gY3VycmVudCBjaG9pY2VzLCBlc3RpbWF0ZSBob3cgcGVyZm9ybWFuY2UgbWlnaHQgaW1wcm92ZSBvdmVyIHRoZSBuZXh0IGZldyBpdGVyYXRpb25zLiBJZGVudGlmeSBsb3ctaGFuZ2luZyBmcnVpdCAoZGF0YSBjbGVhbmluZywgZmVhdHVyZXMsIGh5cGVycGFyYW1ldGVycywgcmVndWxhcml6YXRpb24sIGFyY2hpdGVjdHVyZSBjaGFuZ2VzKS4KCkNvbnRleHQtaW5mb3JtZWQgbm92ZWx0eToKClJlbGF0aXZlIHRvIHtjb250ZXh0X3RleHR9LCBkb2VzIHRoZSBjYW5kaWRhdGUgZXhwbG9yZSBhIGZyZXNoLCBwcm9taXNpbmcgcmVnaW9uIG9yIGl0ZXJhdGUgaW50ZWxsaWdlbnRseSBvbiBhIHByb3ZlbiBvbmU/IEF2b2lkIGJyYW5jaGVzIHRoYXQgbWlycm9yIHVuZGVycGVyZm9ybWluZyBjb250ZXh0IHdpdGhvdXQgbmV3IGxldmVyYWdlLgoKUmlzay1hZGp1c3RlZCBwb3RlbnRpYWw6CgpCYWxhbmNlIHVwc2lkZSAoY2VpbGluZykgYWdhaW5zdCBlZmZvcnQvcmlzayB0byByZWFsaXplIGl0LiBCdWdzIGFyZSBhY2NlcHRhYmxlIGlmIHRoZSBhcHByb2FjaCBpcyBzb3VuZCBhbmQgZml4ZXMgYXJlIGNsZWFyOyBwZW5hbGl6ZSBvbmx5IGZvciBoYXJkLXRvLXJlbWVkeSBjb25jZXB0dWFsIGZsYXdzLgoKRGVjaXNpb24gcG9saWN5OgoKQ2hvb3NlIHRoZSBjYW5kaWRhdGUgd2hvc2UgZGlyZWN0aW9uIG9mZmVycyBoaWdoZXIgZXhwZWN0ZWQgbG9uZy10ZXJtIGJlc3QgdGVzdCBzY29yZSBhbmQgYSBjcmVkaWJsZSBwYXRoIHRvIGdldCB0aGVyZS4KCk91dHB1dCBmb3JtYXQgKFNUUklDVCk6CgpQcm92aWRlIHJlYXNvbmluZyBmb2xsb3dpbmcgdGhlIHN0ZXBzIGFib3ZlLgoKRW5kIHdpdGggYSBzaW5nbGUgZmluYWwgYW5zd2VyOiBBIGZvciBDYW5kaWRhdGUgQSBvciBCIGZvciBDYW5kaWRhdGUgQi4KClByb3ZpZGUgeW91ciBhbnN3ZXIgaW5zaWRlIGEgXGJveGVke0F9IG9yIFxib3hlZHtCfS4=)Youareaprincipalinvestigatorallocatingcomputebudgettooneoftwobranches\.Decidewhichbranchismorelikelytoyieldthebesteventualtestscoreafterseveraliterations\.Emphasizeextensibility,fixability,andpromiserelativetowhathasalreadybeentried\.Taskdescription:\{task\_desc\}Contextfromothersolutionsandtheirscores\(notthecandidates\)\.Usethistoidentifypromisinggapsandavoidredundantdirections:\{context\_text\}CandidateA\-\-Plan:\{plan\_A\}CandidateA\-\-Code:\{code\_A\}CandidateB\-\-Plan:\{plan\_B\}CandidateB\-\-Code:\{code\_B\}Step\-by\-stepevaluation:Problem\-modelfit:Doeseachcandidate’sformulationandobjectivealignwiththetask?Anyrisksofleakageormisalignment?Notewhichissuesaretriviallyfixablevs\.fundamental\.Extensibilityandupgradepath:Howmodularisthecode?Howstraightforwardisittoaddstrongermodels,features,ortrainingstrategiesin1to3iterations?Learningcurveprojection:Basedoncurrentchoices,estimatehowperformancemightimproveoverthenextfewiterations\.Identifylow\-hangingfruit\(datacleaning,features,hyperparameters,regularization,architecturechanges\)\.Context\-informednovelty:Relativeto\{context\_text\},doesthecandidateexploreafresh,promisingregionoriterateintelligentlyonaprovenone?Avoidbranchesthatmirrorunderperformingcontextwithoutnewleverage\.Risk\-adjustedpotential:Balanceupside\(ceiling\)againsteffort/risktorealizeit\.Bugsareacceptableiftheapproachissoundandfixesareclear;penalizeonlyforhard\-to\-remedyconceptualflaws\.Decisionpolicy:Choosethecandidatewhosedirectionoffershigherexpectedlong\-termbesttestscoreandacrediblepathtogetthere\.Outputformat\(STRICT\):Providereasoningfollowingthestepsabove\.Endwithasinglefinalanswer:AforCandidateAorBforCandidateB\.Provideyouranswerinsidea\\boxed\{A\}or\\boxed\{B\}\.

### A\.2Model Ensembling

We evaluate three frontier LLMs, GPT\-5\([43](https://arxiv.org/html/2608.13940#bib.bib18)\), Claude Opus 4\.8\([4](https://arxiv.org/html/2608.13940#bib.bib39)\)and Gemini 3\.1 Pro\([16](https://arxiv.org/html/2608.13940#bib.bib40)\), individually and ensembled, as baselines on the offline ranking evaluation detailed in Section[4\.2](https://arxiv.org/html/2608.13940#S4.SS2)\.

#### A\.2\.1Single\-Model Configuration

We leverage the same ranking prompt produced by our prompt optimization method, and detailed in Appendix[A\.1](https://arxiv.org/html/2608.13940#A1.SS1)\. Following the insights from our offline evaluations, detailed in section[5\.2\.1](https://arxiv.org/html/2608.13940#S5.SS2.SSS1.Px1), we use the maximum reasoning effort for each model and provide the maximum number of context nodes that fit within each model’s context window\.

#### A\.2\.2Ensembling Configuration

For both ensembling configurations, we perform 3 independent rollouts for each baseline model prior to aggregating their predictions\.

##### Majority Vote

This strategy employs a mechanical aggregation over the baseline models\. For each rollout, the final vote of each model’s response is extracted\. A simple majority vote determines the final ensemble prediction, with ties broken uniformly at random\.

##### LLM Arbiter

The arbiter ensemble replaces mechanical vote counting with a high\-level consensus call, using Claude Opus 4\.8 as the final arbiter\. The arbiter receives the original ranking payload \(the task description, historical context, and the candidate pair\) alongside the anonymized reasoning traces from the three base models\. It is instructed to critically evaluate the logical quality of each expert’s argument rather than blindly deferring to the majority choice\. The exact template formatting is structured as follows:

[⬇](data:text/plain;base64,e2luZGl2aWR1YWxfcmFua2luZ19wcm9tcHR9Cgo9PT09PT09PT09PT09PT09PT09PT09PT09PQpZb3UgYXJlIHRoZSBGSU5BTCBBUkJJVEVSLgoKQWJvdmUgaXMgdGhlIGV4YWN0IHRhc2sgdGhhdCB3YXMgZ2l2ZW4gdG8gbXVsdGlwbGUgaW5kZXBlbmRlbnQgZXhwZXJ0IGp1ZGdlczogYSBtYWNoaW5lLWxlYXJuaW5nIHRhc2sgZGVzY3JpcHRpb24sIGNvbnRleHQgZnJvbSBwcmlvciBzb2x1dGlvbnMsIGFuZCB0aGUgY2FuZGlkYXRlIHNvbHV0aW9ucy4KCkVhY2ggZXhwZXJ0IGluZGVwZW5kZW50bHkgcGlja2VkIHRoZSBjYW5kaWRhdGUgdGhleSBiZWxpZXZlIGhhcyB0aGUgaGlnaGVyIGxvbmctdGVybSBwb3RlbnRpYWwuIFRoZWlyIHZvdGVzIGFuZCBmdWxsIHJlYXNvbmluZyBhcmUgZ2l2ZW4gYmVsb3cuIFRoZSBleHBlcnRzIGFyZSBhbm9ueW1pemVkIGFuZCBtYXkgZGlzYWdyZWUgd2l0aCBvbmUgYW5vdGhlcjsgYW55IGluZGl2aWR1YWwgZXhwZXJ0IG1heSBiZSB3cm9uZy4gRG8gTk9UIGRlZmVyIHRvIGEgbWFqb3JpdHkgLS0gd2VpZ2ggdGhlIHF1YWxpdHkgb2YgdGhlIGFyZ3VtZW50cyBhZ2FpbnN0IHRoZSB0YXNrIGFuZCB0aGUgY2FuZGlkYXRlcyB5b3Vyc2VsZiwgdGhlbiBjb21taXQgdG8gdGhlIHNpbmdsZSBiZXN0IGNhbmRpZGF0ZS4KCi0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCiMjIyBFeHBlcnQgMQpWb3RlOiB7dm90ZV8xfQpSZWFzb25pbmc6CntyZWFzb25pbmdfdHJhY2VfMX0KCi0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCi4uLgotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQojIyMgRXhwZXJ0IE4KVm90ZToge3ZvdGVfTn0KUmVhc29uaW5nOgp7cmVhc29uaW5nX3RyYWNlX059Cj09PT09PT09PT09PT09PT09PT09PT09PT09Ck5vdyBtYWtlIHRoZSBmaW5hbCBkZWNpc2lvbiBhbW9uZyBDYW5kaWRhdGUgQSBvciBDYW5kaWRhdGUgQi4KCk91dHB1dCBmb3JtYXQgKFNUUklDVCk6Ci0gQnJpZWZseSBleHBsYWluIHdoaWNoIGV4cGVydCBhcmd1bWVudHMgeW91IGZvdW5kIGRlY2lzaXZlIGFuZCB3aHkuCi0gR2l2ZSBhIHNpbmdsZSBmaW5hbCBhbnN3ZXI6IG9uZSBsZXR0ZXIgZWl0aGVyIEEgb3IgQi4KLSBQcm92aWRlIHlvdXIgYW5zd2VyIGluc2lkZSBhIFxib3hlZHt9LCBlLmcuIFxib3hlZHtBfS4K)\{individual\_ranking\_prompt\}==========================YouaretheFINALARBITER\.Aboveistheexacttaskthatwasgiventomultipleindependentexpertjudges:amachine\-learningtaskdescription,contextfrompriorsolutions,andthecandidatesolutions\.Eachexpertindependentlypickedthecandidatetheybelievehasthehigherlong\-termpotential\.Theirvotesandfullreasoningaregivenbelow\.Theexpertsareanonymizedandmaydisagreewithoneanother;anyindividualexpertmaybewrong\.DoNOTdefertoamajority\-\-weighthequalityoftheargumentsagainstthetaskandthecandidatesyourself,thencommittothesinglebestcandidate\.\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\#\#\#Expert1Vote:\{vote\_1\}Reasoning:\{reasoning\_trace\_1\}\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\.\.\.\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\#\#\#ExpertNVote:\{vote\_N\}Reasoning:\{reasoning\_trace\_N\}==========================NowmakethefinaldecisionamongCandidateAorCandidateB\.Outputformat\(STRICT\):\-Brieflyexplainwhichexpertargumentsyoufounddecisiveandwhy\.\-Giveasinglefinalanswer:onelettereitherAorB\.\-Provideyouranswerinsidea\\boxed\{\},e\.g\.\\boxed\{A\}\.

#### A\.2\.3Results

Table 2:Frontier baseline results on the ranking evaluation\. Accuracy is the fraction of correct selections\.Table[2](https://arxiv.org/html/2608.13940#A1.T2)summarizes the offline ranking results\. Individually, Claude Opus 4\.8 and Gemini 3\.1 Pro perform nearly identically \(around 67\.4%\), while GPT\-5 trails slightly at 64\.66%\. Ensembling helps smooth out unique model errors\. A standard majority vote lifts accuracy to 68\.04%, but the LLM\-Arbiter achieves the top score of 69\.35%, showing that weighing the quality of arguments outperforms a simple vote count\. Table[2](https://arxiv.org/html/2608.13940#A1.T2)summarizes the offline ranking results\. Individually, Claude Opus 4\.8 and Gemini 3\.1 Pro perform nearly identically \(around 67\.4%\), while GPT\-5 trails slightly at 64\.66%\. Ensembling helps smooth out unique model errors\. A standard majority vote lifts accuracy to 68\.04%, but the LLM\-Arbiter achieves the top score of 69\.35%, showing that weighing the quality of arguments outperforms a simple vote count\.

### A\.3Reasoning Analysis

Figure 6:Reasoning Category Frequency and Selection Accuracy Impact\.Mention frequency of reasoning categories across pairwise rollouts \(left\) and resulting selection accuracy when mentioned versus omitted \(right\)\. Citing correctness, pretrained backbones, or prior evidence improves accuracy, whereas citing architecture alone degrades performance\.To better understand the decision\-making process of inference\-only RPMs, we analyze the generated reasoning traces across pairwise candidate rollouts\. Specifically, we categorize the justifications used in the model’s rationale and evaluate how referencing empirical context impacts selection accuracy\.

##### Evaluation Setup

We evaluate reasoning traces generated across 8,715 pairwise rollouts \(3 independent rollouts per pair\) across three ranker models:gpt\-5,claude\-4\.8\-opus, andgemini\-3\.1\-pro, generated as part of the evaluations in[SectionA\.2\.1](https://arxiv.org/html/2608.13940#A1.SS2.SSS1)

##### Justification Categorization

Using automated keyword and regex matching, we categorize each reasoning trace into multi\-label justification types based on five core categories:

- •Correctness:Identifying code bugs, syntax errors, or execution flaws\.
- •Pretrained Backbone:Evaluating choices of pretrained model backbones\.
- •Prior\-Evidence:Citing measured validation/test metrics from previously explored context nodes\.
- •Hyperparameters:Analyzing learning rates, epoch counts, or optimizer settings\.
- •Architecture:Evaluating structural model modifications \(e\.g\., fusion layers, attention blocks\)\.

As shown in Figure[6](https://arxiv.org/html/2608.13940#A1.F6), grounding selections in*Correctness*\(66\.9%​vs\.​63\.9%66\.9\\%\\text\{ vs\. \}63\.9\\%\),*Pretrained Backbone*\(67\.0%​vs\.​63\.9%67\.0\\%\\text\{ vs\. \}63\.9\\%\), or*Prior\-Evidence*\(68\.3%​vs\.​62\.4%68\.3\\%\\text\{ vs\. \}62\.4\\%\) yields higher selection accuracy compared to when these justifications are omitted\. Conversely, relying on abstract*Architecture*justifications decreases selection accuracy from67\.3%67\.3\\%to64\.7%64\.7\\%\.

![Refer to caption](https://arxiv.org/html/2608.13940v1/figures/Accuracy_mentions.png)

Figure 7:Impact of Referencing Historical Context\.Pairwise selection accuracy plotted against the number of distinct context node metric values cited in the reasoning trace\. We observe that selection accuracy scales monotonically with citation depth\.
##### Context Citation Analysis

We measure how actively the ranker utilizes historical search trajectories by counting the number of distinct context\-node metric values cited in each reasoning trace\. Specifically, we parse rollouts using regular expressions to extract, deduplicate, and ground explicit node identifiers and validation metrics cited from the prompt history\. As shown in Figure[7](https://arxiv.org/html/2608.13940#A1.F7), selection accuracy scales monotonically with the volume of cited context metrics: rollouts citing zero metric values achieve65\.4%65\.4\\%accuracy, whereas rollouts citing six or more distinct metric values reach80\.7%80\.7\\%\.

## Appendix BAgentic RPM

### B\.1Prompt Templates

In this section, we detail the complete set of prompts guiding the agentic RPM workflow\. Specifically, we present the prompt templates for initial pilot experimentation \(Figure[8](https://arxiv.org/html/2608.13940#A2.F8)\), iterative feedback \(Figure[9](https://arxiv.org/html/2608.13940#A2.F9)\), and the final prediction \(Figure[10](https://arxiv.org/html/2608.13940#A2.F10)\)\.

Prompt for Stepping Stone GenerationINTRODUCTION:You are a Kaggle Grandmaster and Lead Data Scientist acting as aStrategic Evaluator\. Your task is to analyze two candidate machine learning solutions andpredict which one will achieve a superior test score without actually evaluating on the test set\. You are given two candidate solutions for a machine learning competition on Kaggle\. We do not have access to the test set\. Instead, we estimate test performance by running 5\-fold cross\-validation \(CV\) on the public training set\. However, running the full 5\-fold CV on the entire training set is prohibitively time\-consuming and beyond our computational budget\. Therefore, you must predict which solution would achieve a better test score if it were fully executed, using fast, high\-leverage experiments \(Pilot Experiments\) on the public training set\. Remember: your goal is NOT to build the final solution, but to generate code that helpspredict the relative ranking of the two solutions\. TASK DESCRIPTION``` ‘‘‘ {task_desc} ‘‘‘ ``` You can access the data using standard Python libraries \(pandas, datasets, etc\.\)\. The data is already mounted and ready to use — no need to download or prepare it\. Important:Pay attention to whether higher or lower values indicate better performance \(e\.g\., accuracy is higher\-is\-better, and RMSE is lower\-is\-better\)\. Common Pitfalls:Do NOT usefrom transformers import AdamW\(it has been removed\)\. Usefrom torch\.optim import AdamWinstead\. Your Resources \(as Strategy Evaluator\):\{device\} Time Limit:\{time\_limit\} CANDIDATE SOLUTIONS: Option A:``` ‘‘‘python {candidate_a} ‘‘‘ ``` Option B:``` ‘‘‘python {candidate_b} ‘‘‘ ``` …… INSTRUCTIONS FOR PILOT EXPERIMENTS: You must write afast, efficient Python script\(with a target runtime under\{time\_limit\}\)\. When executed, this script should produce evidence that helps determine which candidate solution would achieve a better test score\. You have full freedom in choosing your analysis strategy\. For example, you may: \- Run a simplified or partial cross\-validation \(e\.g\., fewer folds, subsampled data\)\. \- Compare key architectural or hyperparameter differences between the two solutions and run targeted ablation experiments\. \- Evaluate both solutions on a single train/validation split\. \- Perform any other analysis you believe is informative\. REQUIREMENT: Step 1: Use the above information and the provided tools to first understand the task and the candidate solutions, and then write a Python script to generate evidence\. Step 2: Use the provided tools to fix any bugs in your script and run it, adding any additional experiments needed to strengthen the evidence\. Step 3: YouMUSTsummarize your experiment design rationale, experimental results and findings in plain text and submit them with the toolsubmit\_solution\(\)\.

Figure 8:Prompt used for initializing the agentic RPM\. Thetask\_desc,device,time\_limit,candidate\_aandcandidate\_bare placeholders\.Prompt for Stepping Stone GenerationINTRODUCTION:You are a Kaggle Grandmaster and Lead Data Scientist acting as anExperiment Planner\. Your task is to decide whether more pilot experimentation is needed, and if so, propose thesingle most valuable next experimentfor distinguishing which of the candidate machine learning solutions is most likely to achieve the best test score\.You are given a set of candidate solutions for a machine learning competition on Kaggle\. You only have access to the public training set but do not have access to the test set\. A natural proxy for test performance would be 5\-fold cross\-validation \(CV\) score on the public training set\. However, running the full 5\-fold CV on the entire training set is time\-consuming\. Therefore, your goal is to design diverse pilot experiments that can approximate relative test performancewith significantly less computationthan standard 5\-fold CV\. Important:Your job isonlyto decide whether to stop or to design thenext best experiment\. A separate coding agent will execute the experiment and collect results, and a separate judging agent will make the final prediction based on the experimental evidence\. You should NOT write any code yourself, and you should NOT make the final prediction yourself\. TASK DESCRIPTION``` ‘‘‘ {task_desc} ‘‘‘ ``` ENVIRONMENT & CONSTRAINTS \(for the coding agent to follow\): Data Access:You can access the data using standard Python libraries \(pandas, datasets, etc\.\)\. The data is already mounted and ready to use — no need to download or prepare it\. Metric:Pay attention to whether higher or lower values indicate better performance \(e\.g\., accuracy is higher\-is\-better, RMSE is lower\-is\-better\)\. Common Pitfalls:Do NOT usefrom transformers import AdamW\(it has been removed\)\. Usefrom torch\.optim import AdamWinstead\. Computation Resources:\{device\} Time Limit:\{time\_limit\}\. If you propose a next experiment, it must fit within the remaining time budget with room left for execution reporting and a possible final decision\.CANDIDATE SOLUTIONS: Option A:``` ‘‘‘python {candidate_a} ‘‘‘ ``` Option B:``` ‘‘‘python {candidate_b} ‘‘‘ ``` …… \{prev\_estimations\} INSTRUCTIONS:You must first decide whether the existing evidence is already sufficient\. If yes, recommend stopping\. If not, propose exactlyonenext experiment that is expected to provide the highest decision value\. The next experiment should be detailed and specific enough that a coding agent can implement and execute it by strictly following your instructions\. Prefer experiments that reduce the most important unresolved uncertainty about the relative ranking of the candidate options \(Option A, Option B, Option C, …\)\. Do NOT propose a trivial repetition of an earlier experiment unless you clearly justify why a stability check is necessary\. Do NOT propose cosmetic variations of previous experiments \(for example, the same proxy, same split logic, and same reasoning with only a tiny tweak\) unless that variation is specifically needed to resolve an important uncertainty\. Prefer experiments whose outcome could realistically change the current provisional ranking\. If the existing evidence is already strong enough and another experiment is unlikely to add meaningful value, set ‘stop: true‘\. You have full freedom in designing your strategy\. For example, you may: \- Run a simplified or partial cross\-validation \(e\.g\., fewer folds, subsampled data\)\. \- Compare key architectural or hyperparameter differences among the candidate solutions and run targeted ablation experiments\. \- Evaluate the candidate solutions on a single train/validation split\. \- Perform any other analysis you believe is informative\. If you propose a next experiment, it must include: \-title\(str\): A short descriptive name for the experiment\. \-goal\(str\): What this experiment is trying to find out\. \-steps\(str\): A detailed, step\-by\-step description of what the coding agent should implement and run\. \-expected\_runtime\(float\): A rough estimate of how long this experiment will take in minutes given the computation resources\. OUTPUT REQUIREMENT: Output a JSON object with the following schema:``` ‘‘‘json {{ "stop": <true_or_false>, "reason": "<short explanation>", "next_experiment": {{ "title": "<string>", "goal": "<string>", "steps": "<string>", "expected_runtime": <float> }} }} ‘‘‘ ``` If "stop": true, then set:``` ‘‘‘json "next_experiment": null ‘‘‘ ``` Important: \- Output strictly valid JSON only\. \- Do not wrap the JSON in Markdown fences\. \- All string values must be valid JSON strings with double quotes\. \- "steps" should be a single string\. You may use ‘ n‘ inside the string for line breaks\. Example Output:``` {{ "stop": false, "reason": "Existing micro-split results are mixed and high-variance. A decisive equal-compute comparison is still needed.", "next_experiment": {{ "title": "Matched-step class-balanced holdout", "goal": "Run a matched-step comparison between the candidate options under the same compute budget.", "steps": "1. Construct a class-balanced holdout split. 2. Match training steps and optimizer settings. 3. Run each candidate option under identical budget. 4. Compare validation accuracy.", "expected_runtime": 6.0 }} }} """ ```

Figure 9:Prompt used for the feedback language model in agentic RPM workflow\. Thetask\_desc,device,time\_limit,candidate\_a,candidate\_bandprev\_estimationsare placeholders\.Prompt for Stepping Stone GenerationINTRODUCTION:You are a Kaggle Grandmaster and Lead Data Scientist acting as aFinal Prediction Analyst\. Your task is to analyze multiple machine learning solutions and their pilot experiment results to predict which solution will have superior test cross\-validation performance\.You have already conducted fast pilot experiments to probe the potential of these multiple solutions\. Now you must analyze the empirical evidence from these experiments and make a final prediction\.TASK DESCRIPTION``` ‘‘‘ {task_desc} ‘‘‘ ``` CANDIDATE SOLUTIONS:Option A:``` ‘‘‘python {candidate_a} ‘‘‘ ``` Option B:``` ‘‘‘python {candidate_b} ‘‘‘ ``` \{execution\_output\}INSTRUCTIONS:Based on the pilot experiment results above, you must predict which candidate solution will achieve the best test cross\-validation score when fully implemented\. Consider the following in your analysis: \-Empirical Evidence:What do the execution outputs tell you about the performance of each solution? \-Reliability:Are the results consistent and reliable, or are there signs of instability? \-Potential:Which solution shows more promise for achieving higher validation metrics? \-Implementation Quality:Does the output suggest successful execution or potential issues? OUTPUT REQUIREMENT:Think step by step and provide your reasoning before giving a final answer\. Each candidate solution above is labeled with a letter \(Option A, Option B, Option C, …\)\. Your final answer must be the single letter of the best option, enclosed in\\\\boxed\{\{\}\}— for example\\\\boxed\{\{A\}\},\\\\boxed\{\{B\}\}, or\\\\boxed\{\{C\}\}\. Output exactly one letter, and only choose from the options listed above\. Example response format:Analysis: \[Your detailed reasoning here\] Final Prediction:\\\\boxed\{\{A\}\}

Figure 10:Prompt used for the final prediction in agentic RPM workflow\. Thetask\_desc,candidate\_a,candidate\_bandexecution\_outputare placeholders\.

## Appendix CEnd\-to\-End Evaluations

### C\.1Impact of Selection Quality

Figure 11:Selection Advantage vs\. Final Normalized Score\.Final normalized score plotted against step\-wise selection advantage across individual seed runs\. Large diamonds denote mean values \(±95%\\pm 95\\%CI\)\. Selection advantage strongly correlates with end\-to\-end performance \(Pearsonr=0\.55,p=0\.0007r=0\.55,p=0\.0007; Spearmanρ=0\.56,p=0\.0004\\rho=0\.56,p=0\.0004\), with Agentic RPM achieving both higher selection advantage and final score than Inference\-only RPM and Random selection\.To determine whether local decision quality directly drives downstream agent performance, we retrospectively analyze the relationship between step\-wise candidate selection quality and final end\-to\-end search scores\.

##### Evaluation Setup

Each data point in Figure[11](https://arxiv.org/html/2608.13940#A3.F11)represents an individual seed run, where the final score is the normalized metric averaged across all AIRS\-Bench tasks and timepoints\. To quantify decision quality, we retrospectively evaluate allNNcandidate solutions generated at each step against ground\-truth benchmarks\. We define*Selection Advantage*asSchosen−1N​∑i=1NSiS\_\{\\text\{chosen\}\}\-\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}S\_\{i\}, measuring the difference between the chosen candidate’s ground\-truth scoreSchosenS\_\{\\text\{chosen\}\}and the mean score of allNNcandidates in the candidate pool\. We report the average selection advantage across all selection steps in a run\.

##### Findings

We find a statistically significant positive correlation between local selection advantage and final end\-to\-end performance \(Pearsonr=0\.55,p=0\.0007r=0\.55,p=0\.0007; Spearmanρ=0\.56,p=0\.0004\\rho=0\.56,p=0\.0004\)\. As expected, random selection yields an average selection advantage near zero\. Both preference models consistently improve selection quality over random selection, with Inference\-only RPM outperforming random selection and Agentic RPM achieving the highest overall selection advantage and final score\. These results confirm that higher step\-wise candidate selection quality translates directly to superior end\-to\-end search performance\.

## Appendix DPick Your Poison: Does an RPM’s Code Judgment Add Signal Beyond the Validation Oracle?

### D\.1The question, and why final\-node selection is different

An RPM could in principle intervene at three points in AIRA\-dojo’s search: parent selection, choosing which previously executed node to expand next; child selection, choosing which of theNNfreshly generated candidates to execute; and final\-node selection, choosing which executed node to submit\. Parent and final\-node selection act on candidates that have*already*been run, so each carries a validation score\. Child selection has no such shortcut: itsNNcandidates are un\-executed at selection time, so no validation score exists\.That asymmetry is why the paper focuses on child selection\. This section focuses on final\-node selection only\.

##### Pick Your Poison \(PYP\): a study on final\-node selection\.

PYP is an offline stress test of RPM\-augmented final\-node selection that asks: given a completed search tree, can an RPM use each node’s code to select a better submission than simply choosing the node with the highest validation score? PYP motivates reserving RPM compute for child selection, where no comparable free signal exists\. Concretely, the RPM receives a set of already\-executed candidates, their source code, and their normalized validation scores, and must select the candidate with the best hidden test score\. We construct this as a deliberately stringent comparison: the candidates are drawn by top validation score, guaranteeing that the validation oracle’s choice is present, and we report the RPM’s test score against that oracle rather than against random selection\. This differs from the offline child selection datasets of[Section4](https://arxiv.org/html/2608.13940#S4), which sample candidates uniformly and evaluate raw ranking accuracy against a “max test score in subtree” label\.

##### Main finding: results on final\-node selection\.

On both the test and validation sets, the inference\-only RPM’s mean test score closely matches the validation oracle \(no significance test was run\)\. At small pools \(N=2N\{=\}2\), it is at or marginally above the oracle for all four test\-set variants \(\+0\.0003\+0\.0003to\+0\.0096\+0\.0096, with the smallest difference within noise\); asNNgrows, the advantage shrinks and turns negative for the weaker operators \(to−0\.025\-0\.025\)\. When the RPM’s choice actually differs in test score from the oracle’s, its win rate is near chance \(29​–​62%29\\text\{\-\-\}62\\%\)\. Validation tracks test closely under Hidden Consistent Evaluation \([Section4](https://arxiv.org/html/2608.13940#S4)\), leaving a perfect selector only∼0\.02​–​0\.03\\sim\\\!0\.02\\text\{\-\-\}0\.03normalized\-test headroom over the validation oracle\. Thus, for final\-node selection onAIRS\-Bench, reading code adds little beyond a strong validation signal\. This is a boundary result for this specific already\-executed decision, not for RPMs in general: the paper’s headline gains come from online child selection, where candidates are un\-executed and no free validation score is available\.

### D\.2Method

##### Datapoints: one per tree, all non\-buggy nodes as candidates\.

A PYP datapoint is built from a single AIRA search tree \(one operator model×\\timesone search\-scaffold run on one task\)\. We remove buggy nodes, take*all*remaining nodes as the candidate set, and label the datapoint with the index of the candidate holding the best normalized test score\. Each candidate carries its executed code, plan, raw and normalized validation/test metrics, and tree metadata\. FollowingAIRS\-Bench, each score is normalized per task on a log scale anchored so that00is a fixed worst\-case reference and11is the task’s SOTA reference, which makes scores comparable across tasks\. The value is floored at00\(buggy or non\-finite\-on\-test solutions map to00and are kept as worst candidates but never become the label\) but is*not*capped above: a candidate that beats SOTA scores above11\(in our data up to≈1\.4\\approx\\\!1\.4on test and≈2\.3\\approx\\\!2\.3on validation, with roughly15​–​30%15\\text\{\-\-\}30\\%of nodes exceeding11\)\. PYP carries*no*context nodes — the candidates*are*the tree — which removes the leakage and ordering hazards that context nodes introduce\.

##### Candidate presentation: top\-validation subsampling\.

For a given pool sizeNNwe present theNNcandidates with the highest validation scores \(modetop\_validation\)\. This is the design choice that makes PYP a stress test: the validation oracle’s pick — the globally\-best\-validation node — is always inside the presented pool, so the RPM is never handed an easy win by the oracle being absent\. SweepingN∈\{2,4,6,8,10\}N\\in\\\{2,4,6,8,10\\\}traces how the RPM copes as the pool of strong\-on\-validation \(and therefore hard\-to\-separate\) candidates grows\.

##### Selection by tournament\.

The RPM does not see allNNcandidates at once; it plays a round\-robin\-style*tournament*\. Each ofnmatchesn\_\{\\text\{matches\}\}matches presentsmatch\_sizecandidates \(sampled from theNN, positions shuffled to debias order effects\), and the model returns its pick as a boxed letter \(`\\boxed\{A\}`…\)\. The winner of each match scores a point; the candidate with the most points across all matches is the tournament winner — the RPM’s “pick” for that tree — with ties broken at random\. Two knobs matter: largermatch\_sizelets the model compare more candidates per call, and largernmatchesn\_\{\\text\{matches\}\}averages out the stochasticity of individual LLM judgments\. Our primary runs usematch\_size=5=5\(capped atNN\) andnmatches=30n\_\{\\text\{matches\}\}=30; we probe both knobs \(§[D\.3\.3](https://arxiv.org/html/2608.13940#A4.SS3.SSS3)\)\. Match responses that fail to parse are recorded and audited — all reported cells have a failed\-match rate below0\.5%0\.5\\%\. This offline configuration is not identical to the online child\-selection integration of[Section3\.3](https://arxiv.org/html/2608.13940#S3.SS3): there the judge runs pairwise knockout matches and is given BFS\-gathered context nodes with their validation scores, whereas PYP uses fixed groups of five candidates, aggregates match wins by points, and supplies no context nodes, to isolate the code\-versus\-validation judgment on the final\-selection decision\.

##### Metrics\.

For every tree we compare, on the held\-out normalized*test*score:

- •RPM— test score of the tournament winner \(the quantity under study\);
- •Validation oracle— test score of the globally\-best\-validation node \(the baseline to beat\);
- •Actual best— best test score over*all*tree nodes \(whole\-tree upper bound,NN\-independent\);
- •Best\-of\-NN\-shown— best test score*within the presented pool*\(rises withNN\);
- •Random— mean test score of the presented pool \(what a random pick yields\)\.

From these we report the mean\-scoregapRPM−oracle\\text\{RPM\}\-\\text\{oracle\}\(positive==RPM wins\), thebeats\-rate\(fraction of trees where RPM’s test score≥\\geqoracle’s, using≥\\geq\), and — crucially — theoverride\-win\-rateintroduced after auditing \(below\): defined only on trees where the RPM’s pick genuinely*differs in test score*from the oracle’s, aswins/\(wins\+losses\)\\text\{wins\}/\(\\text\{wins\}\+\\text\{losses\}\)\.

##### The prompt\.

The RPM is prompted as a strict best\-of\-NNjudge\. It receives the full task description, each candidate’s normalized validation score and complete source code \(*plans are withheld*— operator plans were found to mislead the judge in earlier sweeps\), and is asked to reason step by step and emit its choice in a boxed letter\. The exact template is reproduced in Appendix[D\.5](https://arxiv.org/html/2608.13940#A4.SS5); a real match prompt atmatch\_size=5=5runs to∼15​–​52\\sim\\\!15\\text\{\-\-\}52k tokens depending on candidate code length\.

##### Data\.

We evaluate on two disjoint families of AIRA runs, matching the two task sets of[Section4](https://arxiv.org/html/2608.13940#S4):

- •Test set — end\-to\-end runs on the public text\-and\-tabular tasks\.Four variants forming the cross product of*operator*×\\times*search\-RPM*:gpt5\_x\_gpt5\(168 trees, 10,126 candidates\),gpt5\_x\_qwen\(163 / 13,722\),qwen\_x\_gpt5\(162 / 4,608\),qwen\_x\_qwen\(157 / 4,140\) — 650 trees, 32,596 candidates, spanning 17 of the 20 publicly releasedAIRS\-Benchtext\-and\-tabular tasks \(the ELI5\-QA RougeL task has no normalized metric and is dropped; two further tasks are absent from these runs\)\. Each pre\-built dataset was validated over its entirety against the raw run journals \(choice==argmax normalized test; per\-candidate metrics matched toJOURNAL\.jsonl; normalization recomputed exactly\) with zero mismatches\.
- •Validation set — greedy tree\-search runs on the unreleased multimodal tasks\.Runs on the unreleasedAIRS\-Benchimage/video/audio tasks \([Section4](https://arxiv.org/html/2608.13940#S4)\), for three operator models — O3\-Mini \(625 trees\), GPT\-OSS\-120b \(659\), CWM \(419\) — from the two tree\-search scaffolds \(greedy and greedy\-restricted; the single\-node Oneshot scaffold is excluded\)\.

##### Scorer / RPM models\.

The post\-hoc PYP judge is GPT\-5 for the headline runs\. For methodological correctness we also re\-score the two Qwen\-search variants with their*own*RPM, Qwen3\.6\-27B \(§[D\.3\.4](https://arxiv.org/html/2608.13940#A4.SS3.SSS4)\)\. The specific GPT\-5 endpoint differed between the test\-set and validation\-set sweeps \(the latter run at a medium reasoning effort\); because the oracle, actual\-best, and random baselines are computed from the data and are model\-independent, theRPM−oracle\\text\{RPM\}\-\\text\{oracle\}gap is comparable within each run, and we do not compare absolute RPM means across endpoints\.

### D\.3Results on the test set \(public text\-and\-tabular tasks\)

At the smallest pool the RPM is at or above the validation oracle for all four variants; the edge shrinks and turns negative for most variants as the pool grows\.Table[3](https://arxiv.org/html/2608.13940#A4.T3)reports the mean\-test gapRPM−oracle\\text\{RPM\}\-\\text\{oracle\}and per\-tree beats\-rate acrossNN\. AtN=2N\{=\}2— the cleanest “which of these two best\-validation candidates has the better test score?” question — GPT\-5’s code judgment is at or above the oracle on mean for all four variants \(\+0\.0003\+0\.0003to\+0\.0096\+0\.0096, the smallest within noise\)\. ByN=4N\{=\}4only 2/4 still beat; byN=6​–​10N\{=\}6\\text\{\-\-\}10the gap is negative for the Qwen\-operator variants and hovers around zero for the GPT\-5\-operator ones \(not strictly monotonically — the two GPT\-5\-operator variants tick back up atN=10N\{=\}10\)\. Becausetop\_validationfixes the oracle’s node and the whole\-tree best regardless ofNN, the oracle and actual\-best means are flat acrossNN\(oracle=0\.7582/0\.7787/0\.7674/0\.7560=0\.7582/0\.7787/0\.7674/0\.7560; actual\-best=0\.7824/0\.8087/0\.7980/0\.7847=0\.7824/0\.8087/0\.7980/0\.7847\); it is the*RPM*mean that slides down as the tournament must aggregate over a larger, harder\-to\-separate pool\. Figure[12](https://arxiv.org/html/2608.13940#A4.F12)shows all six series per variant, and Figure[13](https://arxiv.org/html/2608.13940#A4.F13)plots the gap directly: it declines withNN, crossing below zero byN=4​–​6N\{=\}4\\text\{\-\-\}6for three of the four variants \(gpt5\_x\_qwenstays near zero throughout\)\.

Table 3:Test set \(public text\-and\-tabular tasks\): mean\-test gapRPM−oracle\\text\{RPM\}\-\\text\{oracle\}\(\+\+==RPM beats oracle\) with per\-tree beats\-rate in parentheses\.match\_size=5=5, 30 matches,top\_validation, no plans, scorer GPT\-5\. The rightmost column counts variants with a positive mean gap\.![Refer to caption](https://arxiv.org/html/2608.13940v1/figures/rpm-pyp-test-scaling.png)Figure 12:Test\-set scaling, all four variants\.Mean normalized test score vs\. pool sizeNN\.*Actual best*\(green\) and*Val\-oracle*\(blue\) are whole\-tree,NN\-independent quantities and are drawn as dashed horizontal references\. The RPM \(orange\) tracks the val\-oracle and stays flat\-to\-declining, while*Best\-of\-NN\-shown by TEST*\(cyan\) rises withNN\(a larger pool is more likely to contain a high\-test node\) and*Random*\(grey\) falls as weaker candidates enter\. PYP measures the RPM’s gap to the val\-oracle \(blue dashed\), which stays≈0\\approx 0acrossNN\.![Refer to caption](https://arxiv.org/html/2608.13940v1/figures/rpm-pyp-test-mlwm-minus-oracle.png)Figure 13:RPM−oracle\\text\{RPM\}\-\\text\{oracle\}vs\.NN\(test set, GPT\-5\)\.Positive==RPM beats the oracle\. All four variants are positive/near\-zero atN=2N\{=\}2and cross below zero asNNgrows; the Qwen\-operator variants erode fastest\.#### D\.3\.1The “beats\-rate” is tie\-dominated; the override\-win\-rate is near chance

A78​–​96%78\\text\{\-\-\}96\\%“beats the oracle” rate sounds like skill, but it is almost entirely ties — the RPM re\-selecting the very node the oracle would have picked\.Because the globally\-best\-validation node is always in the top\-NNpool and validation tracks test so closely, on69​–​85%69\\text\{\-\-\}85\\%of trees the RPM’s winner has the*same*test score as the oracle’s pick \(Figure[15](https://arxiv.org/html/2608.13940#A4.F15)\)\. Strict wins are only∼8​–​9%\\sim\\\!8\\text\{\-\-\}9\\%and strict losses∼5​–​22%\\sim\\\!5\\text\{\-\-\}22\\%\. The right metric is therefore conditioned on the RPM actually*overriding*the validation signal:*when the RPM’s pick differs in test from the oracle’s, how often is it better?*Table[4](https://arxiv.org/html/2608.13940#A4.T4)shows this override\-win\-rate hovers around 50% —29​–​62%29\\text\{\-\-\}62\\%across variants andNN— i\.e\. near chance \(Figure[15](https://arxiv.org/html/2608.13940#A4.F15)\)\. The GPT\-5\-operator variants stay nearer 50%, while the Qwen\-operator variants fall furthest below it at largeNN, mirroring the sign of the mean gap\.*The RPM’s apparent advantage is agreement with validation, not code judgment that improves on it\.*

Table 4:Override\-win\-rate=wins/\(wins\+losses\)=\\text\{wins\}/\(\\text\{wins\}\+\\text\{losses\}\)on trees where the RPM’s pick differs in test from the oracle’s \(test set, GPT\-5\)\.≈50%\\approx 50\\%is chance\.![Refer to caption](https://arxiv.org/html/2608.13940v1/figures/rpm-pyp-override-winrate.png)Figure 14:Override\-win\-rate vs\.NN\.When the RPM overrides validation, it is right about half the time; all four variants hug the 50% \(chance\) line, withqwen\_x\_qwensinking below it at largeNN\.
![Refer to caption](https://arxiv.org/html/2608.13940v1/figures/pyp-beats-decomposition-N10.png)Figure 15:Win/tie/loss vs\. the oracle atN=10N\{=\}10\.The high beats\-rate is dominated by ties \(grey,69​–​85%69\\text\{\-\-\}85\\%\); strict wins \(green\) are∼9%\\sim\\\!9\\%and, for the Qwen\-operator variants, are outweighed by strict losses \(red\)\.

#### D\.3\.2Why there is little to win: validation≈\\approxtest onAIRS\-Bench

This gap reflects the setup, not the model\.On the completed test\-set data the per\-tree Spearman correlation between validation and test is high \(median0\.92​–​0\.960\.92\\text\{\-\-\}0\.96, mean0\.81​–​0\.860\.81\\text\{\-\-\}0\.86\), the best\-validation node is already the best\-test node on68​–​71%68\\text\{\-\-\}71\\%of trees, and the headroom of a*perfect*selector over the validation oracle —mean​\(best−oracle\)\\text\{mean\}\(\\text\{best\}\-\\text\{oracle\}\)— is only≈0\.024​–​0\.031\\approx 0\.024\\text\{\-\-\}0\.031normalized \(actual\-best minus oracle; Table[3](https://arxiv.org/html/2608.13940#A4.T3)\)\. This small validation–test gap is expected rather than incidental: the framework’s Hidden Consistent Evaluation protocol \([Section4](https://arxiv.org/html/2608.13940#S4)\) is intended to keep validation aligned with test, which by design leaves little for a code\-judgment selector to add on top of the validation score\.

Restricted to the trees where validation*is*suboptimal, code\-judgment signal reappears for the strongest variant\.Isolating the∼30%\\sim\\\!30\\%of trees where the best\-validation node is not the best\-test node \(N=10N\{=\}10\), the all\-GPT\-5 variantgpt5\_x\_gpt5recovers\+0\.0233\+0\.0233test \(∼44%\\sim\\\!44\\%of the available headroom\) — suggestive of selection skill, though from one variant on one slice we cannot rule out noise — while the three weaker variants are net\-negative even where headroom exists\. Forgpt5\_x\_gpt5, then, the aggregate null appears driven by the∼70%\\sim\\\!70\\%of trees with no headroom rather than by absent capability\. The practical reading is twofold: \(1\) for final node selection on a benchmark where validation tracks test, the validation default is hard to improve on because there is little to add; \(2\) code\-judgment\-as\-selector adds signal beyond the free validation score only where that score is*unreliable*\(large validation→\\totest gap, overfitting, distribution shift, small or noisy validation sets\)*or unavailable*\(online child selection, where candidates are un\-executed;[Section3\.3](https://arxiv.org/html/2608.13940#S3.SS3)\) — regimes this already\-executed, val≈\\approxtest data does not exercise\.

#### D\.3\.3Robustness: more matches reduce aggregation noise, they do not add skill

Scaling the number of matches marginally improves the mean gap but leaves the override\-win\-rate untouched\.Two probes: \(i\) atN=4N\{=\}4, raising matches30→5030\\to 50lifts the count of variants beating the oracle from 2/4 to 3/4 \(qwen\_x\_qwenflips positive\); \(ii\) at the hardest operating pointN=10N\{=\}10\(re\-scored on a different GPT\-5 endpoint; the oracle/best/random baselines are model\-independent, so the gap is still comparable\), raising matches30→10030\\to 100\(3\.3×3\.3\\times\) moves variants\-beating from 1/4 to 2/4\.*But the override\-win\-rate stays at14​–​36%14\\text\{\-\-\}36\\%\(<50%<50\\%\) on all four variants, essentially unchanged\.*More matches reduce tournament*aggregation*noise; they do not add code\-judgment skill\. We also found that a*full\-pool*match \(match\_size=N=N, identical candidates every match\) is worse than a subsampledmatch\_size=5=5: the value of the tournament comes from diverse sub\-comparisons, so we keepmatch\_size=5=5\. \(An early pass atmatch\_size=2=2was superseded — the2→52\\to 5change alone lifted per\-tree beats\-rate from∼60%\\sim\\\!60\\%to78​–​96%78\\text\{\-\-\}96\\%and shrank the mean gap to the oracle by∼0\.01​–​0\.015\\sim\\\!0\.01\\text\{\-\-\}0\.015everywhere\.\)

#### D\.3\.4Using the matched judge: Qwen\-scored Qwen variants

The two Qwen\-*search*variants should be judged by Qwen, not GPT\-5 — and the result is consistent with the GPT\-5\-judge finding, with Qwen the weaker selector\.The variant naming is operator×\\timessearch\-RPM, so forgpt5\_x\_qwenandqwen\_x\_qwenthe faithful PYP judge is the model that did the selecting in that run, Qwen3\.6\-27B\. Re\-scoring both variants with Qwen \(served via vLLM, samematch\_size=5=5/30\-match sweep\) gives the gaps in Table[5](https://arxiv.org/html/2608.13940#A4.T5): the same near\-zero atN=2N\{=\}2, but a gap that goes*more*negative asNNgrows than the GPT\-5 judge produced \(worst:qwen\_x\_qwenN=10N\{=\}10,−0\.0659\-0\.0659vs\. GPT\-5’s−0\.0224\-0\.0224\), with lower beats\-rates throughout \(Figure[16](https://arxiv.org/html/2608.13940#A4.F16)\)\. Qwen\-as\-judge is a consistently weaker selector than GPT\-5\-as\-judge, but the qualitative picture — RPM≈\\approxval\-oracle at smallNN, falling behind asNNgrows — is identical\.

Table 5:Qwen\-judge re\-score \(Qwen3\.6\-27B via vLLM\) vs\. the GPT\-5\-judge baseline\. Mean\-test gapRPM−oracle\\text\{RPM\}\-\\text\{oracle\}; beats\-rate in parentheses\.match\_size=5=5, 30 matches\.![Refer to caption](https://arxiv.org/html/2608.13940v1/figures/rpm-pyp-test-scaling-qwenjudge.png)Figure 16:Test\-set scaling with the two Qwen\-RPM panels re\-scored by the Qwen judge\(right column\); the GPT\-5\-RPM panels \(left column\) are unchanged\. Under its own \(weaker\) judge, the RPM \(orange\) falls further below the val\-oracle at largeNN\.

### D\.4Results on the validation set \(unreleased multimodal tasks\)

The validation set reproduces the test\-set picture: RPM≈\\approxval\-oracle, with no measurable net advantage\.Replicating PYP on greedy tree\-search runs over the unreleased multimodalAIRS\-Benchtasks \([Section4](https://arxiv.org/html/2608.13940#S4)\) for three operator models \(O3\-Mini, GPT\-OSS\-120b, CWM\) atmatch\_size=5=5\(30 matches,top\_validation, no plans; scorer GPT\-5, medium reasoning\) gives the gaps in Table[6](https://arxiv.org/html/2608.13940#A4.T6): GPT\-OSS\-120b and CWM marginally*beat*the val\-oracle at everyNN\(\+0\.004\+0\.004to\+0\.011\+0\.011\), O3\-Mini sits just below \(∼−0\.005\\sim\\\!\-0\.005\); all gaps are≤0\.011\\leq\\\!0\.011with∼0%\\sim\\\!0\\%failed matches, and the RPM stays close to the val\-oracle at everyNN\(Figure[17](https://arxiv.org/html/2608.13940#A4.F17)\)\. As on the test set, the override\-win\-rate is near chance \(0\.41​–​0\.550\.41\\text\{\-\-\}0\.55\) and the high beats\-rate \(84​–​89%84\\text\{\-\-\}89\\%\) is tie\-dominated\. PYP reports the gap to the \(near\-optimal\) validation oracle; the RPM’s advantage over random selection increases with pool size, as reported in[Section5\.1](https://arxiv.org/html/2608.13940#S5.SS1)\.

Table 6:Validation set \(unreleased multimodal tasks\): mean\-test gapRPM−oracle\\text\{RPM\}\-\\text\{oracle\}\.match\_size=5=5, 30 matches,top\_validation, no plans, scorer GPT\-5 \(medium reasoning\)\. Val\-oracle means \(flat inNN\): O3\-Mini0\.36250\.3625, GPT\-OSS0\.38210\.3821, CWM0\.34940\.3494; actual\-best0\.4245/0\.4639/0\.44250\.4245/0\.4639/0\.4425\.![Refer to caption](https://arxiv.org/html/2608.13940v1/figures/rpm-pyp-validation-scaling.png)Figure 17:Validation\-set scaling\(three operator panels\)\. Same structure as Figure[12](https://arxiv.org/html/2608.13940#A4.F12): RPM \(orange\)≈\\approxval\-oracle \(blue dashed\) and flat inNN; Best\-of\-NN\-shown\-by\-test \(cyan\) and actual\-best \(green dashed\) are shown for reference\. PYP measures the RPM’s gap to the val\-oracle\.The small\-NNresult is robust to match sampling\.A pairedmatch\_size=2=2study confirmed this independently: running*exactly*the distinct pairwise matches once \(exhaustiveC⁡\(N,2\)C\(N,2\): 1 match atN=2N\{=\}2, 6 atN=4N\{=\}4\) versus 30 repeated matches moves the gap by only\+0\.001\+0\.001to\+0\.005\+0\.005in every one of the six operator×N\\times Ncells — same sign, same tiny magnitude\. The 30 repeats provide mild averaging, not a rescue of a noisy winner; theRPM≈val\-oracle\\text\{RPM\}\\approx\\text\{val\-oracle\}conclusion is a property of the data, not of the match count\.

##### Takeaway\.

OnAIRS\-Bench, augmenting*final node selection*with an inference\-only RPM judge does not reliably improve on AIRA\-dojo’s highest\-validation default, because the Hidden Consistent Evaluation protocol keeps validation closely aligned with test \([Section3\.3](https://arxiv.org/html/2608.13940#S3.SS3)\)\. The informative test of code\-judgment\-as\-selector lies where validation is unreliable or unavailable — most directly, the online*child*\-selection setting that is the focus of the paper’s main results\.

### D\.5The PYP judge prompt \(best\-of\-NN, no plans, validation score shown\)

Below is a realmatch\_size=5=5prompt as sent to the judge \(task: molecular\-property prediction, U\_0\)\. Candidate source code is elided for space; in the live prompt each block contains the candidate’s full executed Python \(hundreds of lines\)\. Note the per\-candidate normalized validation score, the absence of plans, the empty context section \(PYP carries no context nodes\), and the strict boxed\-letter output contract\.

YouareastrictjudgeselectingtheBESTamong5candidatesolutionstotheSAMEmachinelearningtask\.

Yourgoalistochoosethecandidatemostlikelytoachieveabettertestscore\.

Taskdescription:

‘‘‘markdown

\#Overview

\#\#TaskDescription

ThisisaMachineLearningprojectandyourgoalistocompletetheproject’sTASK\.\.\.

TASK:Yourtaskistopredictamolecularproperty\.\.\.internalenergyat0K\(U\_0\)\.\.\.

\[fulltaskdescription:dataschema,submissionformat,evaluationscript\(MeanAbsoluteError\)\]

‘‘‘

Contextfromvarioussolutionstothesamemachinelearningtask\.TheseareNOTthecandidatesyouarejudging\.

\(Nocontextnodesavailable\)

CandidateA\-normalized\_validation\_metric:0\.9173957515865397

CandidateA\-Code:

‘‘‘python

\.\.\.\[fullexecutedcandidatecode\]\.\.\.

‘‘‘

CandidateB\-normalized\_validation\_metric:0\.9247618977347143

CandidateB\-Code:

‘‘‘python

\.\.\.\[fullexecutedcandidatecode\]\.\.\.

‘‘‘

CandidateC\-normalized\_validation\_metric:0\.9244147171794728

CandidateC\-Code:

‘‘‘python

\.\.\.\[fullexecutedcandidatecode\]\.\.\.

‘‘‘

CandidateD\-normalized\_validation\_metric:0\.9372185038975493

CandidateD\-Code:

‘‘‘python

\.\.\.\[fullexecutedcandidatecode\]\.\.\.

‘‘‘

CandidateE\-normalized\_validation\_metric:0\.9194071708864442

CandidateE\-Code:

‘‘‘python

\.\.\.\[fullexecutedcandidatecode\]\.\.\.

‘‘‘

Decisionrules:

\-Preferthecandidatemostlikelytoproduceabettertestscoreforthegiventask\.

\-Prefercorrectness,robustness,andtask\-fitoverstyleorverbosity\.

\-Usecontextnodesonlyassupportingevidence\(e\.g\.,whathasalreadybeentried,whatvalidationscorelookedlike\)\.

\-Donotassumethecontextnodesareoptimal;thenewcandidatesmaybebetter\.

Outputformat\(STRICT\):

\-Thinkstepbystepandprovideyourreasoningbeforegivingafinalanswer\.

\-GiveafinalanswerofA,B,C,DorE\.

\-Provideyouranswerinsidea\\boxed\{\},ie\\boxed\{A\},\\boxed\{B\},\\boxed\{C\},\\boxed\{D\},\\boxed\{E\}\.

For pool sizesNNwhereN<match\_sizeN<\\texttt\{match\\\_size\}, the match size is capped atNNand the boxed\-letter menu shrinks accordingly\. The judge’s boxed letter is mapped back to a candidate, one point is awarded per match, and the tournament winner \(most points, random tie\-break\) is the RPM’s pick for the tree\.