A 27B agent just beat Claude Opus 4.8 and GPT-5.5 on held-out research replication.

Replica turns paper replication into a scalable RL task space. Replicating a paper forces the same hypothesis-driven exploration as open research, and it surfaces details the original authors left underspecified.

The reward signal comes from an auto-generated rubric judge that runs low-noise and agrees with human assessment of replication quality.

Faraday, the resulting 27B agent, calls coding agents as tools. Rollout analysis shows it takes a more scientifically principled approach rather than gaming the rubric.

The authors argue this points toward long-horizon scientific capability trained into weights, without requiring complex harnesses.

Paper: https://arxiv.org/abs/2608.13331

Track more trending AI papers in our academy: https://academy.dair.ai

Training AI Scientists to Replicate Research

Source: https://arxiv.org/html/2608.13331 Training AI Scientists to Replicate Research\authornote*Equal first author.§Infrastructure lead.†Equal last author.1Inherent\[email protected]

Damon Falck*,1,Samer Sabri*,1,Anja Surina1,Thom Foster1,Anya Sims1,Sam Devlin1, Dylan Rogers1,Tantum Collins1,Kaloyan Aleksiev§,1,Louis Kirsch†,1,Edward Hughes†,1

14thAugust 2026

Abstract

The replicability of papers is a cornerstone of scientific knowledge, ensuring the reliability of existing results and providing a base for further experiments. The act of replication typically illuminates details that were previously underspecified, and thus requires similar hypothesis-driven exploration to open-ended research. In this work, we developReplica, a scalable task space for paper replication. To provide reward signal, we introduce an auto-generated rubric-based judge that has low noise and agrees with human assessment of replication quality. We post-trainFaraday, a 27B-parameter “AI Scientist” agent that leverages coding agents as tools, surpassing the performance of Claude Opus 4.8 and GPT-5.5 on held-out replication tasks. Qualitative analysis of individual rollouts reveals that Faraday adopts a more scientifically-principled approach. We believe that our results provide a stepping stone towards AI agents capable of long-horizon scientific innovation without requiring complex harnesses.

1Introduction

Science is the search for good explanations about the universe(25). An explanation compresses what we know about reality in a reliable way. A good explanation is hard to vary; if new evidence comes to light that contradicts the explanation, that explanation is falsified, rather than easily tweaked to admit the fresh data. Crucial to both reliability and falsification is the idea that scientific experiments ought to replicate: if you run the same experiment, you get the same results, up to the sensitivity of the measuring equipment and uncontrollable stochasticity. Replication, therefore, underpins the edifice of human scientific knowledge.

Remarkably, the sciences face a replication crisis, not least in machine learning(56;120). In principle, LLM-based AI agents offer a scalable resolution to this crisis, especially for research that can be conductedin silico. In practice, however, paper replication poses a challenge to existing AI agents on three fronts. Firstly, the problem of replicating a paper is underspecified by definition: a paper lossily compresses the research that led to a discovery. Secondly, existing AI agents have been heavily trained for well-specified, closed-ended problems(123;41)whereas replication requires open-ended exploration to infer missing details. Finally, harnesses like autoresearch(57)and AlphaEvolve(95)do not naturally apply, by virtue of the fact that for a general replication task there is no definite reward on which to hill-climb. Recent work shows that frontier agents struggle with many scientific aspects of replication, despite their proficiency in engineering(63).

In this paper, we trainFaraday, an “AI Scientist” agent(117;91;83;68)capable of replicating research papers. Our LLM-based agent employs Codex GPT-5.5 as a tool, much as human AI researchers use coding agents. Conceptually, we are training a layer of scientific intelligence that sits above existing coding agents, imbued with an intuition about how to handle underspecified research problems. Notably, Faraday is a2727B-parameter model that directs the work of a model with an estimated55T parameters(77)in a way that yields a meaningful performance gain over the larger model alone.

To train Faraday, we introduce a scalable space of tasks calledReplica. Each task requires the agent to replicate a figure from a paper with limited time and compute budgets, without seeing the original plot. Success necessitates a small measure of creativity, in the sense of navigating novel constraints(14;21;26). A coding agent judge assesses each replication using an auto-generated per-task rubric, validated against expert human rankings, yielding a low-noise reward signal. Faraday is produced by post-training Qwen3.6-27B(104)with a turn-level credit variant of GRPO(123)on the Replica train split.

Figure 1:We train Faraday on Replica.(1)We construct the Replica task space by curating100100ML and AI-for-science papers published between 1990 and 2026. For each paper, we generate a set of replication tasks by using Gemini 2.5 Pro to redact individual results figures. Each redacted figure yields one task.(2)We generate rollouts on these tasks using our agent Faraday, with access to Codex as a tool for writing code. The agent is given acontainerdcontainer provisioned with the task prompt, the redacted paper PDF, the Codex binary, various useful research libraries, a one-seventh MIG slice of an H200 GPU, and internet access.(3)For each task, we use Claude Opus 4.7 prompted with a meta-rubric to generate a task-specific grading rubric.(4)Each rollout is evaluated according to that task’s rubric using multiple samples of a Codex-based judge, given access to the rollout’s container comprising the generated figure, replication codebase, agent rollout, and “gold plot” from the original paper. The judge provides an overall reward and per-turn credit assignment weights, which are used to train the Faraday agent using a modified version of GRPO.Faraday outperforms Claude Opus 4.8 (hereafter, Claude) and GPT-5.5 (hereafter, Codex) on73%73\%of in-distribution ML tasks, and on60%60\%of held-out AI-for-science tasks, according to our rubric-based judge. On average, Faraday achieves a6%6\%improvement over Claude and an8%8\%improvement over Codex on the test split. Optimising Codex’s prompt only marginally diminishes the gap. Human experts rate Faraday as stronger than Claude and Codex on rollouts for which the rubric judge assesses that Faraday has an advantage. Compared to rollouts from frontier models, Faraday behaves more like a human scientist: it implements the mechanism behind the claim rather than hard-coding outputs, it scales down in a way that remains faithful to the paper’s experimental scope, and it avoids shortcuts that would flatter its own result. In summary, the main contributions of our paper are as follows:

1. 1.We introduceReplica, an automatically generated space of310310figure-replication tasks from100100machine learning and AI-for-science papers spanning the years 1990–2026 (Figure 1).
2. 2.We provide a recipe for stable GRPO post-training in long-horizon, non-verifiable tasks: a per-task rubric-based judge, multi-sample judge aggregation, and turn-level credit assignment (Section 3.2andSection 3.5).
3. 3.We trainFaraday, a2727B-parameter agent that leverages coding agents as tools (CAT), exhibiting greater scientific rigour both quantitatively and qualitatively (Figure 2andTable 1).

2Related work

Figure 2:Faraday replicates better than frontier coding agents.We plot the fraction of tasks scoring at leastσ\sigma, in-distribution (left) and out-of-distribution (right). For each task, we use the mean score over eight evaluation rollouts. Bands show±1\pm 1SEM over tasks. Faraday and Qwen3.6-27B run in the same simple harness, with Codex GPT-5.5 available as a coding tool; they differ only in that Faraday has been RL post-trained on Replica. Faraday’s curve lies above baselines at every threshold in distribution, and across almost the whole range out of distribution. Not only is it the strongest agent on average (numbers in legend), it also has a thinner weak tail. As in84, the horizontal axis is spaced by a march-of-nines transform,σ↦−log10⁡(1−σ)\sigma\mapsto-\log_{10}(1-\sigma).##### Rewards for AI Scientists.

Scientific research is an underspecified problem. It is context-dependent and admits many different kinds of solutions, and judgement of its quality is subjective. Automating research is a longstanding goal, often pursued by moving pieces of the research loop inside a learning algorithm, such as the update rule(116;11), the objective(67;98), or the learning algorithm in its entirety(108;64). These works generally assume a well-defined reward signal against which the meta-learned component can be scored. Such a signal is hard, if not impossible, to define for the broad goal of scientific research. Nevertheless, to train AI Scientist agents, it is convenient to compress their behaviour into a scalar-valued reward. There are at least two natural ways to produce such a reward: (a) derive a verifiable reward function from existing benchmarks, or (b) judge agent behaviour qualitatively using an LLM.

The former method is particularly effective when hill-climbing an existing benchmark entails a novel and valuable insight, as was the case for the problems under investigation by FunSearch(111)and AlphaEvolve(95). The success of such algorithms has spurred much work creating hill-climbable benchmarks for discovery, including in toy settings(85), from Kaggle competitions(17;102), and based onin silicoscientific research(50;19;93;145;162;107;84).

However, this approach suffers a critical limitation: the innovations that agents uncover do not tend to be generalisable. In other words, they are adaptations but not exaptations(39). There is a ceiling on what can be achieved within this paradigm; indeed, almost no prerequisite to any truly great invention was conceived with that invention in mind(119;130). Moreover, considerable human labour was required to construct the aforementioned benchmarks, limiting their scalability for large-scale model training.36try to resolve both problems by automatically generating a combinatorially huge space from a modest set of hand-designed components, and by explicitly testing generalisation with a meta-train/test split. However, in time, similar problems will emerge at the meta level.

Therefore, we adopt the latter strategy: post-hoc judgement of agent rollouts by an LLM. A few prior works employ this approach, with the aim of automating research paper generation end-to-end(83;144;114). In these works, the judges mirror peer review, a highly underspecified setting with minimal ground truth and considerable noise. In contrast, our rubric judge evaluates a more modest, controlled and grounded setting, in which agreement with humans can more easily be established: paper replication.

Replication tasks for AI Scientists.

Existing replication benchmarks differ in how much of the original work the agent is handed, trading off ease of evaluation with construct validity – how faithfully they measure replication(24;9).46give the agent a finished reproduction and ask only for a score.3;127provide the authors’ code and ask the agent to run it and answer questions about the paper. Various authors supply most of a reference implementation with parts masked out, graded by unit tests(49;69), code similarity(146), or a judge(151).59sweep this spectrum directly.

We sit at the latter end, where the agent is given the paper and graded by a judge, similar to131;30;103;51. Closest are163and121, who also work from the paper against a human-calibrated judge. We extend this line of work by introducing a larger and more scalable task space, while maintaining the benefits of a per-task rubric(23;35;41;140;124;45). Additionally, many of our tasks require the agent to replicate findings under strong resource and time constraints, testing understanding of the method as opposed to blind copying, and necessitating an inventiveness that bridges towards innovative research. In concurrent work,82introduce a complementary task space, extracting a paper’s claims and judging each against the evidence from agent-generated experiments across6565papers spanning computer science, social science, medicine, and astrophysics.

Training AI Scientists.

Given a static reward function for discovery, many recent AI Scientist systems have pursued test-time scaling. These typically rely upon one or more of in-context learning(152), evolutionary search(76;73;42), tree search(54;139;52), or test-time training(133;144;153). The test-time improvement algorithm in these works is hard-coded (even if only on the meta level), and thus limited by the biases of the designer(134). Self-modification relaxes this constraint(118;66), although more recent LLM-based works retain the strictures of fixed language model weights(158;160;142;161).

Other works decompose the scientific method as a hand-designed system of agents with different roles and affordances(135;38;33;34), benefitting from specialisation and division of labour. However, these modular architectures are somewhat brittle and reductive, and each agent has a restrictive interface, constraining the exploration space in a potentially unhelpful way. We propose a more flexible setup, in which Faraday is an agent within acontainerdcontainer, equipped with a coding agent as a tool (CAT). Our CAT paradigm extends that of132;94, in which a smaller model is post-trained to use larger models as tools, to the setting where a tool is a frontier coding agent in its standard CLI harness.

Unlike previous works, we post-train Faraday at scale across a space of242242tasks, drawing on the ability of neural networks to generalise, yielding an AI Scientist that can effectively conduct rigorous science out of distribution and without a test-time reward. This approach draws inspiration from large-scale multi-turn RL without language models(137;136), which teaches that training on a vast, smooth, and diverse task distribution produces an agent that can generalise and adapt. In particular, we succeed at extending GRPO(123)post-training to long-horizon, non-verifiable tasks, a regime known to suffer from instability(150;143;60). Previous works have used rubric-based judges to generate rewards for multi-turn RL in long-form question-answering(78;122)but not for such long-horizon tasks, and not in such a complex environment.148introduce turn-level credit assignment weights from an LLM judge that are similar in spirit but different in formulation to ours, and report negative results.

3Methods

3.1Task space

TheReplicatask space comprises242242training tasks and6868test tasks drawn from100100well-known ML and AI-for-science papers. Each task requires an agent to replicate one results figure from a paper, given the original paper with the figure redacted, a6060-minute time limit, and a single one-seventh MIG slice of an H200 GPU. The agent is provided with acontainerdcontainer to work in with helpful research libraries pre-installed, access to the internet, a system prompt, and a task prompt (Appendix G). Where a paper’s experiment cannot be completed within the given time budget, the prompt asks for the most faithful scaled-down version of the underlying experiment. The training tasks are drawn from ML papers from 1990 to 2026. The test tasks are drawn from AI-for-science papers from 2012 to 2026. We choose well-known papers for ease of human rating (Section 3.3).

Importantly, our tasks are automatically generated, and thus the task space is scalable. Given a paper, three vision-language stages convert it into a task, powered by Gemini 2.5 Pro(22). A scan finds every main-text results plot and its caption, a localisation stage draws its bounding box inside an LLM-verifier repair loop, and the figure is irreversibly redacted from the PDF. A task is a triple of caption, extracted figure (“gold plot”), and paper with figure redacted. We inspect every task by hand and filter out any that are of low quality, for instance if the figure is insufficiently redacted, if it is not a results plot, or if the caption is incorrectly identified. Each paper contributes between11and1313tasks with median22. We address the issue of pre-training contamination inSection D.1.

3.2Reward function

Paper replication is inherently non-verifiable, especially when scaling down experiments to fit within resource constraints while remaining true to the core claim. In Replica tasks, the “gold plot” (the redacted figure from the original paper) is used to help judge replication attempts, but perfectly reproducing the plot is not the same as a successful replication: successful replication should also demonstrate strong experimental design, good scientific practice, faithfulness to the original paper, and strategic use of available resources. Designing a reward signal to train against is therefore a key challenge. The long-horizon nature of our replication tasks additionally demands that this signal be low-variance across judge samples and consistent across similar rollouts.

Judge rubric generation.

We base our judge on the concept of a rubric, a scoring guide that provides specific criteria for assessing performance on a task. Starting from a short, hand-designed meta-prompt, we use Claude Opus 4.7 to auto-generate task-specific rubrics. We hide the “gold plot” from the rubric generator so that the rubric captures the claims of the paper without over-indexing on figure details such as axis ranges, formatting, and exact numerical values. The judge rubric is also hidden from the model during training, encouraging the model to produce broadly effective replications rather than game the rubric criteria.

The judge rubric covers five dimensions: (1) how closely the replicated figure visually matches the paper’s, (2) how well the replication supports the paper’s scientific claim, (3) whether the underlying experiment actually implements and tests what the paper describes, (4) whether the agent makes good use of the compute budget, and (5) whether the agent acted with scientific integrity, adhering to its instructions and not cheating. In our tasks, the time and resource limit means it is often not possible to replicate the figure at full scale. The rubric generator is instructed to reward agents for producing a faithful scaled-down version. This is a key feature of the task space: it introduces further underspecification, thereby teaching decision-making skills characteristic of open-ended research.

Coding agent as a judge.

We assess rollouts using Codex GPT-5.5 as a judge, prompted with the appropriate per-task rubric. The judge is given access to the same workspace and compute resources as the agent, including the redacted paper, all the tools the agent had, the replication codebase, and git history (containing the final plot generated), and the full interaction trace of the rollout, as well as the ground-truth “gold plot” from the original paper. The judge is given1010minutes to explore these materials and form a judgement on each dimension of the rubric. Each criterion receives a continuous score between00and11, and these per-dimension scores are averaged to give an overall score for the rollout. Crucially, this approach allows the judge to examine fully and potentially re-execute the agent’s code to understand its process and check the robustness of its claims. During training, we sample from the judge three times for each rollout to reduce variance, and additionally instruct the judge to generate credit assignment weights for each agent turn; seeSection 3.5.

3.3Human studies

We collect human rankings of agent rollouts to assess the extent to which our rubric-based judge captures human research taste. For each rollout, the human expert is given access to the relevant paper with the figure redacted, the “gold plot” and its caption, a transcript of the rollout, and the git repository generated by the agent, including agent instructions, code and outputs, and the plot(s) that resulted from the agent’s experiments. Humans are asked to rank either three or six rollouts from best to worst. Participants are provided with simple instructions for how to rank rollouts; seeSection G.4. Importantly, they are instructed to follow their own best judgement about what they would expect a faithful replication to look like, so as to capture human tacit knowledge.

Participants are asked to justify their ranking, so as to encourage a principled and consistent approach(86). We also ask participants to explain what effect the “gold plot” shows, and to suggest a correct methodology to reproduce it. Given the complexity of the task, we select participants from a pool of ongoing or completed PhDs from top research universities, preferring participants who have published at least one paper at the main conference track of ICML, ICLR, or NeurIPS. We pay each participant £150 per task, with bonuses of £125 paid upon the completion of the fifth and eighth tasks. In total we collect 117 rankings from 20 participants.Appendix Fdescribes how we choose the tasks and rollouts for participants to rank.

3.4Simple harness

An agent harness provides an interface between an LLM (tokens-in/tokens-out) and an environment (action-in/state-out). In our setting, the environment is an interface to a container. We design an agent harness for Faraday based on three principles: it should be simple and interpretable; it should be maximally permissive, allowing the agent the same context and affordances a human would have when undertaking AI research; and its tool set should be minimal, deliberate, and legible to contemporary models. Faraday’s scientific capabilities are improved by changing its policy weights, not by complexifying its harness. We report the harness system prompt inSection G.1.

Affordances and context.

The agent acts via five function-calling tools:apply_patch,read_file,list_dir,grep_files, andshell. Faraday can detach background processes using the shell tool, allowing it to take many turns while running several commands in parallel. The tool interface is a subset of the Codex CLI schema(99), reimplemented in Python. The conversation is a linear, append-only history with no compaction. A turn’s tool calls execute concurrently and their results are appended in call order. Context overflow, exceeding the per-turn1616K token limits, and inference errors end the rollout, and the partial rollout is judged like any other. A rollout otherwise ends when Faraday replies without tool calls or when its wall-clock time is exhausted.

Coding agent as a tool (CAT).

Faraday is provided with a frontier coding agent to use as a tool. A wrapper script runs the Codex CLI non-interactively. Faraday can invoke this script through itsshelltool, receiving a rendered transcript of the coding agent’s commands, outputs, and messages with per-step timings. Successive invocations resume the coding agent’s previous session by default. However, Faraday can choose to reset context or run multiple coding agents in parallel. The wrapper script enforces a deadline, configurable by Faraday per-request. If the deadline is exceeded, a partial transcript is returned. The coding agent model is a runtime parameter; we use GPT-5.4 mini for most of the training, and GPT-5.5 in the final stage and for evaluation.

3.5Post-training recipe

Figure 3:Our rubric judge achieves higher human agreement and lower noise than the baseline judge.We compare our per-task rubric-based prompt against a simpler baseline prompt which does not vary across tasks. Codex GPT-5.5 uses the prompt to judge rollouts sampled from Claude, Codex, and Faraday.(left)We select tasks whose rollouts maximise disagreement between the rubric judge and the baseline judge. We ask expert humans to rank the same rollouts based on their intuition for what constitutes a good replication. We measure agreement using Kendallτ\tau, a rank correlation between two orderings (+1 = identical, −1 = reversed). Two independent draws of the rubric judge agree more closely (0.660.66) than two draws of the baseline judge (0.460.46) or two humans (0.300.30). The rubric judge agrees more closely with humans (0.190.19) than the baseline judge (0.150.15). Dots are individual tasks, listed inTable F.1.(right)We sample1616GRPO groups of 8 rollouts each uniformly across training steps430430–461461, scoring every rollout eight times with each judge. For each group, we calculate the fraction of the within-group score variance that is caused by judge noise, as a function of the numbermmof judge samples averaged per rollout. Bands show±1\pm 1SEM over the 16 groups. The rubric judge is less noisy at everymm. In particular, eight baseline judge samples are required to reduce the noise share to the level obtained with three rubric judge samples.To obtain Faraday, we post-train Qwen3.6-27B(104)in the Faraday harness on the Replica task space, using a modified version of GRPO(123). We use LoRA fine-tuning(47)with rank128128andα=128\alpha=128; we train adapters on all linear projections, with a128128K-token context window and a constant learning rate of6×10−66\times 10^{-6}. We find this context length to be sufficient for the Replica tasks given the 1-hour time limit used during training. Each Adam(61)optimiser step draws a batch of1010tasks, with eight rollouts apiece, from the242242-task Replica train split. Tasks are sampled so that every batch spans the corpus’ year range evenly, and each epoch visits every task exactly once, so no single era of science dominates any update.Section D.2provides further training details andAppendix Hdescribes our infrastructure.

Long-horizon stability.

Our post-training requires long-horizon RL in a non-verifiable domain, a setting known to be prone to instability and collapse. Two sources of this instability are the high variance of the reward signal and uniform credit assignment. To address this, we make two train-time modifications to our judge. First, we compute the rollout-level reward using the mean of three independent judge evaluations. Second, we instruct the judge to produce turn-level weights attributing credit over the rollout’s turns. The judge produces a weight distributionuku_{k}over turnskkwhich is then normalised such that∑kuk​nk=∑knk\sum_{k}u_{k}n_{k}=\sum_{k}n_{k}, wherenkn_{k}is the number of tokens in turnkk, such that we do not change the overall reward scale. The normalised weights are averaged turn-wise over the three judge draws, which preserves the normalisation. The weight for the corresponding turn is then used to scale the per-token advantage during GRPO. In this way, credit is redistributed within a rollout without changing the overall magnitude of the update.Figure C.1explores empirically how our method assigns credit within each rollout. These techniques helped to achieve stable training (see ablations inAppendix B).

4Results

Figure 4:Frontier coding agents do not saturate Replica.(left)Papers written more recently are harder to replicate. Each point represents the mean rubric score across tasks for one paper from the Replica train split, sized by the number of tasks that paper yields. Lines are least-squares fits.(right)Difficulty varies across research topics, and Faraday leads on every one. The top block shows the train split (ML tasks), and the bottom block shows the test split (AI-for-science tasks). Each point represents the mean rubric score over tasks in the given research topic.### 4.1Rubric judge reliably captures human taste

We conduct a human study to establish alignment between our rubric judge and human taste. We report full methodology and results inSection F.1. We introduce a baseline judge with the same coding agent model (Codex GPT-5.5) but a constant prompt across tasks that mirrors the prompt given to human participants. Human rankings correlate better with our per-task rubric judge than with a baseline judge (Figure 3, left). However, there are several tasks on which our rubric judge disagrees with humans, suggesting room for improvement in future work. Our rubric judge ranks more consistently than both the baseline judge and humans. Furthermore, the fraction of group variance that arises from judge noise rather than between-rollout signal is lower for the rubric judge (Figure 3, right). It is therefore a better candidate for use as a reward in GRPO.

4.2Replica tasks are challenging for frontier agents

We run frontier coding agents on Replica and find that they do not saturate the task space. For Claude Opus 4.8 and GPT-5.5 baselines, we run the model in the Claude Code and Codex harnesses respectively, with extra-high thinking effort. For Faraday, we pin the thinking effort of its Codex tool to extra-high, to ensure a fair comparison. For the GLM-5.2 baseline, we run the model with max thinking effort in the Claude Code harness, as it is the best reported harness for TerminalBench(157). Every agent receives the same task materials and the same6060-minute single-GPU budget, and is scored by the same rubric judge. We run eight rollouts per task per agent. Within a task, rollout scores are reduced to a single per-task score by taking the mean.

Claude Opus 4.8 is our strongest baseline. Task performance decreases with publication year for every agent. We speculate that more recent papers are harder to replicate because there is less density of information about them in the pre-training dataset, and because they tend to use higher compute resources, and so determining an appropriate and successful scale-down is more challenging. Task difficulty also varies between research topics, and the per-topic rankings are consistent among baseline agents, with NLP and LLM papers hardest and classical machine learning and statistics easiest. AI-for-science papers are generally harder to replicate than ML research papers across agents, possibly because they require integrating experimental expertise from different domains. Faraday’s base model and harness before RL is the weakest of all, and is the fastest to degrade with recency.

4.3Faraday replicates better than Claude and Codex

We compare Faraday against baselines across the entire Replica task distribution (Figure 2). We achieve a comprehensive uplift in performance compared to the base Qwen model, on both train and test tasks. In distribution, Faraday outperforms both Claude and Codex on73%73\%of tasks. Out of distribution, Faraday outperforms Claude and Codex on60%60\%of tasks. Since the held-out papers span research areas Faraday never trained on, the behaviour it acquired is not memorisation of a specialised procedure but a transferable way of approaching the underspecified task of paper replication. On both train and test, Faraday’s advantage is an upward shift of the whole distribution.Figure 4(right) shows that the gap is consistent across different subdomains. Decomposing the judge score into its sub-dimensions reveals that Faraday is stronger than baselines when it comes to experimental depth, claim reproduction, and visual fidelity, and matches Claude on scientific integrity and implementation fidelity (Figure C.2).

To test whether Faraday’s advantage can be obtained by prompting alone, we run2424generations of automated prompt optimisation on the Codex baseline. Similar to Faraday’s training, each generation samples1010training tasks with eight rollouts per task. Claude Opus 4.8 then rewrites the prompt based on all previous rollouts in the filesystem, including the judge’s feedback. We compare the final prompt against Codex and Faraday inFigure 5(left). The optimised prompt does not perform meaningfully better than the original prompt, thus the gap to Faraday is retained. The optimised prompt (Section G.5) identifies the specific failure modes seen in the rollouts, but without success: the gain from post-training does not appear to be reachable by prompting.

Figure 5:Prompt-optimised Codex remains weaker than Faraday, and Faraday generalises to tasks that require innovation.(left)The mean rubric score over all tasks (and over eight rollouts per task) for Faraday is higher than for Codex with both the default baseline prompt, and a prompt automatically optimised in-context for one epoch over the train split.(right)On twenty counterfactual variants of tasks from ten papers, five from the train split and five from the test split, Faraday leads on mean rubric score over eight rollouts in almost all cases. For each task, one variant swaps the dataset and one changes the claim. In both plots, bars show±1\pm 1SEM.

4.4Faraday is qualitatively a more rigorous scientist

To understand how Faraday improves on the Claude and Codex baselines, we examine by hand the individual rollouts for which Faraday’s rubric judge score exceeds the best Claude and Codex score by the largest margins.Table 1showcases representative examples. Two patterns recur. First, Faraday implements the mechanism an experiment is designed to test, whereas the baseline hardcodes the expected output or falls back on an oversimplified method, failing to replicate the main claim of the figure. Second, Faraday is more thorough in the scope of its experiments, reproducing more of the original experiment without unnecessary omissions.

Furthermore, we examine the discovery process that occurs within Faraday rollouts. We sample nine tasks uniformly at random from the Replica test split, and for each task select the strongest of eight Faraday rollouts. We identify the moments of insight when the best score until that point is exceeded and use Claude Opus 5 to label the insights. We see a similar accumulation of knowledge as in AI Scientist systems built with evolutionary harnesses (Figure C.3). However, unlike previous systems, Faraday has no special hand-coded harness, does not change its harness at test time, and does not have access to the rubric judge reward. In other words, Faraday has learned to value insights intrinsically.

Finally, we run a human study to assess to what extent humans prefer Faraday over Claude and Codex, specifically focussing on rollouts in which the rubric judge deems that Faraday holds a strong advantage. We report full methodology and results inSection F.2. Of the4141rollouts examined, humans prefer Faraday over both Claude and Codex in2929, significantly more than chance. This result suggests that the rubric judge accurately but not perfectly captures the characteristics of good replication, at least when it comes to the best-performing samples from Faraday. Importantly, the design of our study does not allow us to draw any conclusions as to whether humans prefer Faraday over Claude and Codex on average. Gathering conclusive evidence for this preference would require a larger-scale study across a randomly selected set of tasks, and is an important direction for future work.

Table 1:Faraday behaves more like a rigorous scientist.Via human inspection, we qualitatively analyse tasks with the largest margin between Faraday and the best performing run among Claude and Codex, grouped by whether the paper lies inside Faraday’s training distribution (ML) or outside it (AI-for-science).

5Discussion

Towards innovation.

On the face of it, replicating a figure from a paper is not an especially creative endeavour. Most obviously, replication produces a figure that looks quite like the original, assuming that the method replicates. But if one examines the process, rather than the output, replication becomes a stepping stone towards innovation. The skills that allow Faraday to fill in vaguely-specified details may be the very same skills that would allow it to advance the state of the art by designing its own experiments(25;92;43;13). Indeed, many human researchers start their careers by learning to replicate existing results and, upon mastering that, are better placed to conduct original research. Replication is the first step in a curriculum of increasing underspecification towards innovation.

Inspired by such considerations, we assess how well Faraday generalises to “imagined” replications (Figure 5, right). We ask Claude Opus 4.8 to generate two variants of five randomly selected papers from each of the Replica train and test splits: (a) making the same claim as the original figure but using a different dataset or environment and (b) making a different claim from the original figure in the same setting (Section F.3). We evaluate Faraday and Codex GPT-5.5 on these tasks, and score them with our rubric judge, noting that the task interface itself has not changed. Faraday’s rollouts are preferred to Codex’s by the judge on1919of the2020tasks. In a weak sense, Faraday not only replicates better than a frontier model; it also innovates better. However, we must caution that our rubric judge was never validated on imagined tasks, and so future work is warranted to validate this claim.

Coding agent as a tool (CAT).

It is perhaps surprising that we succeed in training such a small model to better direct the activities of a model at least two orders of magnitude larger. Moreover, training the outer agent need not be prohibitively expensive in inference tokens for the inner tool. After training with a weaker coding agent as a tool, one can substitute a more powerful coding agent at evaluation time and achieve an uplift in performance (Figure A.2). The skills Faraday acquires – deciding what to investigate, scoping experiments to a budget, and judging a replication – compound with advances in frontier coding models. One might hope that a single post-trained outer agent can track the frontier as better models are released, at least over some time period. Establishing the optimal cost-benefit tradeoff between the sizes of the inner and outer agents is an interesting topic for further study.

The success of the CAT paradigm has implications for both capabilities and safety. In the realm of AI Scientist agents, we offer an approachable alternative to harness construction. History teaches us that encoding capabilities in the weights of a neural network, rather than expressing them in code, is more flexible and generalisable in the long run. With an eye to safety, our results demonstrate successful oversight of a more powerful model by a less powerful one(4;16;58). Moreover, the reasoning traces of the open-weights model can be inspected, unlike those behind the closed-weights API surface. Note that nothing in the CAT paradigm requires the outer agent to remain the smaller model; it is an empirical question whether the demands of scientific judgement must match or exceed those of engineering execution in the long run.

Beyond verifiable rewards.

To capture the scientific abilities that underpin open-ended research, we necessarily move away from well-specified tasks with verifiable rewards. A side effect of this reorientation may be that agents are less exposed to incentives for reward hacking during post-training(7). Defining a verifiable reward necessitates specifying an evaluation procedure in foresight, which becomes a fixed target for manipulation. By contrast, judging entire rollouts in hindsight is a moving target. Indeed, we observe Faraday acting with greater scientific rigour and faithfulness than frontier agents, completing tasks as intended rather than reproducing figures performatively. It remains to be seen whether training on open-ended tasks can scalably ameliorate reward hacking.

Generalisation.

In training Faraday, we deliberately limit the scope of the tasks to short time horizons and limited GPU resources. Our motivation is twofold: first, pragmatism in achieving sufficient throughput for RL per unit wall-clock time; second, a belief that the ability to experiment quickly and efficiently with minimal versions of research ideas is a valuable transferable skill. It is natural to wonder whether Faraday can generalise to larger resources, similar to those used for experiments in the original papers. To assess this, we select one figure from each of eight papers whose replication we estimated to require fewer than eight hours and eight B300 GPUs. We provide appropriate resources to Faraday and to Claude Opus 4.8 and evaluate them on these scaled-up tasks (Section A.1). We find that Faraday exceeds the performance of Claude on average and in five out of the eight tasks, suggesting generalisation. Clearer validation of the rubric judge on larger-scale tasks, together with a larger number of such tasks, would be required to make a stronger claim. We further discuss scaling in the supplementary discussion (Appendix E).

Community engagement.

Paper replication is a public good, strengthening the scientific foundations upon which future insights can be built. Progress on replication is particularly timely, since the paper review system is beginning to strain under the weight of AI-assisted research(31). Indeed, high-quality paper replication tools may well help to ground AI-assisted reviewing in the future. If you have ideas for how you might use Faraday in your work, we would be delighted to hear from you at[email protected].

As an early step towards real-world validation of Faraday’s usefulness, we obtained feedback from the authors of four papers in the Replica task space(112;37;110;83)on Faraday’s replication of one figure from their paper. On the one hand, the authors were impressed by parts of the replication (“part b and c look very good”, “the reflexion implementation looks correct”), by the inventiveness of the agent (“nice and clever toy task design”) and by fidelity to the original work (“the agent’s implementation more closely follows equation (1) in the paper”). On the other hand, some simplifications did not make sense (“the problem selected is probably too easy”), parts of the write-up were poor (“paragraph on pre-training dataset design is particularly bad”) and code slop is off-putting (“calculations contain unnecessarily convoluted code”).

Conclusion.

In summary, we have created an intelligence layer with a modicum of research taste, sufficient to extend the capabilities of frontier agents. This is, however, the tip of the iceberg when it comes to imbuing agents with the ability to enrich scientific research as peer collaborators with humans. Stepping from replication towards innovation sharpens the problem of underspecification, and deepens the need to develop systems with good judgement. And cultivating creative human-machine teams will require AI taste in code, experiments, theories, collaboration, and organisational design.

Ethics statement

Awareness of limitations.

Paper replication by AI agents holds great promise, but remains at an early stage. While Faraday successfully replicated claims from a number of papers, it failed in several cases where we have confidence that the original result was obtained rigorously and reported honestly. We do not claim that any of Faraday’s failures to replicate published work suggest fundamental problems with the original research. Even as and when AI systems demonstrate sufficiently strong performance to serve as reliable judges of replicability, it will remain important for humans to cultivate the skills necessary for paper replication and to inspect the results of agents such as Faraday.

Human studies.

We assessed whether our procedures for human data collection would require review by an external board and concluded that this was not necessary since the information elicited consisted only of professional judgements and did not feature sensitive personal data, and participation would pose no risk. We worked with a combination of people within our pre-existing professional networks and experts sourced by third-party data providers. We made clear to participants the purpose of the project. Participants were compensated irrespective of whether their ratings cleared our internal filtering.

Conflict of interest.

Some of the papers included in our corpus were written by authors of this paper and by individuals we know personally. Some work in the corpus comes from institutions whose commercial models we used for the research presented here. We applied the same automated pipeline to all items in the corpus and in no way altered scoring treatment for our own prior work or that of our acquaintances.

Technical safety.

Not all applications of scientific insight benefit society. Prior work on the use of AI to automate or accelerate scientific research notes accurately that these capabilities may empower malicious human actors and/or increase the dangers associated with misaligned AI systems. For this paper, we selectedin silicotasks that we judge unlikely to cause harm, and we constrained Faraday in terms of both time and compute. Faraday did not have access to any physical lab equipment, although it did have internet access.

Acknowledgements

We thank Shi Dong, Alex Goldie, Matt Henderson, Akarsh Kumar, Chris Lu, Clare Lyle, and Jimmy Secretan for valuable comments on an early version of this manuscript. We thank Sergio Gomez, José Miguel Hernández-Lobato, Chris Lu, and Matthias Rupp for providing feedback on the quality of replication of their papers.

References

• J. Abramson, J. Adler, J. Dunger, R. Evans, T. Green, A. Pritzel, O. Ronneberger, L. Willmore, A. J. Ballard, J. Bambrick,et al.Accurate structure prediction of biomolecular interactions with AlphaFold 3.Nature630,pp. 493–500.Cited by:Table F.2.
• Ahmadianet al.(2024)A. Ahmadian, C. Cremer, M. Gallé, M. Fadaee, J. Kreutzer, O. Pietquin, A. Üstün, and S. HookerBack to basics: revisiting REINFORCE-style optimization for learning from human feedback in LLMs.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics,Cited by:§D.2.
• Alizadehet al.(2026)M. Alizadeh, M. Mosleh, F. Gilardi, A. Kasirzadeh, and J. TuckerAI coding agents can reproduce social science findings.arXiv preprint arXiv:2606.11447.External Links:LinkCited by:§2.
• Amodeiet al.(2016)D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. ManéConcrete problems in AI safety.arXiv preprint arXiv:1606.06565.External Links:LinkCited by:§5.
• Anthonyet al.(2017)T. Anthony, Z. Tian, and D. BarberThinking fast and slow with deep learning and tree search.InAdvances in Neural Information Processing Systems,Cited by:Appendix E.
• Anthropic (2026)AnthropicClaude models overview.Note:Claude Opus 4.8 training data cutoff: January 2026External Links:LinkCited by:§A.1.
• Bakeret al.(2025)B. Baker, J. Huizinga, L. Gao, Z. Dou, M. Y. Guan, A. Madry, W. Zaremba, J. Pachocki, and D. FarhiMonitoring reasoning models for misbehavior and the risks of promoting obfuscation.arXiv preprint arXiv:2503.11926.External Links:LinkCited by:§5.
• Batatiaet al.(2022)I. Batatia, D. P. Kovács, G. N. C. Simm, C. Ortner, and G. CsányiMACE: higher order equivariant message passing neural networks for fast and accurate force fields.InAdvances in Neural Information Processing Systems,Cited by:Table F.2.
• Beanet al.(2026)A. M. Bean, R. O. Kearns, A. Romanou, F. S. Hafner, H. Mayne, J. Batzner, N. Foroutan Eghlidi, C. Schmitz, K. Korgul, H. Batra,et al.Measuring what matters: construct validity in large language model benchmarks.InAdvances in Neural Information Processing Systems,Cited by:§2.
• Belkinet al.(2006)M. Belkin, P. Niyogi, and V. SindhwaniManifold regularization: a geometric framework for learning from labeled and unlabeled examples.Journal of Machine Learning Research7,pp. 2399–2434.Cited by:Table F.2,Table F.2.
• Bengioet al.(1992)S. Bengio, Y. Bengio, J. Cloutier, and J. GecseiOn the optimization of a synaptic learning rule.InPreprints Conf. Optimality in Artificial and Biological Neural Networks,Cited by:§2.
• Bhattacharjee (2026)S. BhattacharjeeSR-CGCNN: shared recurrent convolution in crystal graph neural networks for materials property prediction.arXiv preprint arXiv:2605.01304.External Links:LinkCited by:§A.1.
• Bhoopchandet al.(2023)A. Bhoopchand, B. Brownfield, A. Collister, A. Dal Lago, A. Edwards, R. Everett, A. Fréchette, Y. G. Oliveira, E. Hughes, K. W. Mathewson,et al.Learning few-shot imitation as cultural transmission.Nature Communications14(1),pp. 7536.Cited by:§5.
• Boden (1995)M. BodenCreativity and unpredictability.Stanford Humanities Review4(2),pp. 123–139.Cited by:§1.
• Bodnaret al.(2025)C. Bodnar, W. P. Bruinsma, A. Lucic, M. Stanley, A. Allen, J. Brandstetter, P. Garvan, M. Riechert, J. A. Weyn, H. Dong, J. K. Gupta, K. Thambiratnam, A. T. Archibald, C. Wu, E. Heider, M. Welling, R. E. Turner, and P. PerdikarisA foundation model for the Earth system.Nature641(8065),pp. 1180–1187.Cited by:Table F.2,Table F.4.
• Bowmanet al.(2022)S. R. Bowman, J. Hyun, E. Perez, E. Chen, C. Pettit, S. Heiner, K. Lukošiūtė, A. Askell, A. Jones, A. Chen,et al.Measuring progress on scalable oversight for large language models.arXiv preprint arXiv:2211.03540.External Links:LinkCited by:§5.
• Chanet al.(2025)J. S. Chan, N. Chowdhury, O. Jaffe, J. Aung, D. Sherburn, E. Mays, G. Starace, K. Liu, L. Maksin, T. Patwardhan, L. Weng, and A. MądryMLE-bench: evaluating machine learning agents on machine learning engineering.arXiv preprint arXiv:2410.07095.External Links:LinkCited by:§2.
• Chawlaet al.(2002)N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. KegelmeyerSMOTE: synthetic minority over-sampling technique.Journal of Artificial Intelligence Research16,pp. 321–357.Cited by:Table F.1.
• Chenet al.(2025)Z. Chen, S. Chen, Y. Ning, Q. Zhang, B. Wang, B. Yu, Y. Li, Z. Liao, C. Wei, Z. Lu, V. Dey, M. Xue, F. N. Baker, B. Burns, D. Adu-Ampratwum, X. Huang, X. Ning, S. Gao, Y. Su, and H. SunScienceAgentBench: toward rigorous assessment of language agents for data-driven scientific discovery.arXiv preprint arXiv:2410.05080.External Links:LinkCited by:§2.
• Chithranandaet al.(2020)S. Chithrananda, G. Grand, and B. RamsundarChemBERTa: large-scale self-supervised pretraining for molecular property prediction.arXiv preprint arXiv:2010.09885.External Links:LinkCited by:§A.1.
• Colton and Wiggins (2012)S. Colton and G. A. WigginsComputational creativity: the final frontier?.InECAI 2012,pp. 21–26.Cited by:§1.
• Comaniciet al.(2025)G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva,et al.Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261.External Links:LinkCited by:§3.1.
• Cooket al.(2024)J. Cook, T. Rocktäschel, J. Foerster, D. Aumiller, and A. WangTicking all the boxes: generated checklists improve LLM evaluation and generation.arXiv preprint arXiv:2410.03608.External Links:LinkCited by:§2.
• Cronbach and Meehl (1955)L. J. Cronbach and P. E. MeehlConstruct validity in psychological tests..Psychological Bulletin52(4),pp. 281.Cited by:§2.
• Deutsch (2011)D. DeutschThe beginning of infinity: explanations that transform the world.Allen Lane,London.Cited by:§1,§5.
• Epstein (2026)D. EpsteinInside the box.Riverhead Books.External Links:ISBN 9780593715710Cited by:§1.
• Eysenbachet al.(2019)B. Eysenbach, A. Gupta, J. Ibarz, and S. LevineDiversity is all you need: learning skills without a reward function.InInternational Conference on Learning Representations,Cited by:Table F.2.
• Friedman (2001)J. H. FriedmanGreedy function approximation: a gradient boosting machine.The Annals of Statistics29(5),pp. 1189–1232.Cited by:Table F.2.
• Friedmanet al.(2000)J. Friedman, T. Hastie, and R. TibshiraniAdditive logistic regression: a statistical view of boosting.The Annals of Statistics28(2),pp. 337–407.Cited by:Table F.2,§G.6.
• Gaddipatiet al.(2026)S. K. Gaddipati, D. Muhammed, F. Keya, G. Rabby, and S. AuerMLReplicate: benchmarking autonomous research systems for machine learning reproducibility.arXiv preprint arXiv:2605.16616.External Links:LinkCited by:§2.
• Gartenberget al.(2026)C. Gartenberg, S. Hasan, A. Murray, and L. PierceMore versus better: artificial intelligence, incentives, and the emerging crisis in peer review.Organization Science37(3),pp. 795–812.Cited by:§5.
• Gerset al.(2002)F. A. Gers, N. N. Schraudolph, and J. SchmidhuberLearning precise timing with LSTM recurrent networks.Journal of Machine Learning Research3(Aug),pp. 115–143.Cited by:Table F.1,§G.6,Table 1.
• Ghafarollahi and Buehler (2025)A. Ghafarollahi and M. J. BuehlerSciAgents: automating scientific discovery through bioinspired multi-agent intelligent graph reasoning.Advanced Materials37(22),pp. 2413523.Cited by:§2.
• Ghareebet al.(2026)A. E. Ghareeb, B. Chang, L. Mitchener, A. Yiu, C. J. Szostkiewicz, D. Shved, G. J. Gyimesi, J. M. Laurent, S. M. Wright, M. T. Razzak,et al.A multi-agent system for automating scientific discovery.Nature,pp. 1–3.Cited by:§2.
• Goelet al.(2025)S. Goel, R. Hazra, D. Jayalath, T. Willi, P. Jain, W. F. Shen, I. Leontiadis, F. Barbieri, Y. Bachrach, J. Geiping,et al.Training AI co-scientists using rubric rewards.arXiv preprint arXiv:2512.23707.External Links:LinkCited by:§2.
• Goldieet al.(2026)A. D. Goldie, Z. Wang, A. Hayler, D. Nathani, E. Toledo, K. Thampiratwong, A. Kalisz, M. Beukman, A. Letcher, S. Reddy,et al.Procedural generation of algorithm discovery tasks in machine learning.arXiv preprint arXiv:2603.17863.External Links:LinkCited by:§2.
• Gómez-Bombarelliet al.(2018)R. Gómez-Bombarelli, J. N. Wei, D. Duvenaud, J. M. Hernández-Lobato, B. Sánchez-Lengeling, D. Sheberla, J. Aguilera-Iparraguirre, T. D. Hirzel, R. P. Adams, and A. Aspuru-GuzikAutomatic chemical design using a data-driven continuous representation of molecules.ACS Central Science4,pp. 268–276.External Links:DocumentCited by:Table 1,§5.
• Gottweiset al.(2025)J. Gottweis, W. Weng, A. Daryin, T. Tu, A. Palepu, P. Sirkovic, A. Myaskovsky, F. Weissenberger, K. Rong, R. Tanno,et al.Towards an AI co-scientist.arXiv preprint arXiv:2502.18864.External Links:LinkCited by:§2.
• Gould and Vrba (1982)S. J. Gould and E. S. VrbaExaptation—a missing term in the science of form.Paleobiology8(1),pp. 4–15.Cited by:§2.
• Guet al.(2026)H. Gu, X. Han, Y. Bian, F. Wang, and K. ZhouAsymmetric advantage modulation calibrates entropy dynamics in RLVR.arXiv preprint arXiv:2604.04894.External Links:LinkCited by:§A.1.
• Gunjalet al.(2025)A. Gunjal, A. Wang, E. Lau, V. Nath, Y. He, B. Liu, and S. HendryxRubrics as rewards: reinforcement learning beyond verifiable domains.arXiv preprint arXiv:2507.17746.External Links:LinkCited by:§1,§2.
• Hambardzumyanet al.(2026)K. Hambardzumyan, N. Baldwin, E. Toledo, R. Hazra, M. Kuchnik, B. A. Omari, T. S. Foster, A. Protopopov, J. Gagnon-Audet, I. Mediratta, K. Niu, M. Shvartsman, A. Lupidi, A. Audran-Reiss, P. Pathak, T. Shavrina, D. Magka, H. Momand, D. Dunfield, N. Cancedda, P. Stenetorp, C. Wu, J. N. Foerster, Y. Bachrach, and M. JosifoskiAIRA_2: overcoming bottlenecks in AI research agents.arXiv preprint arXiv:2603.26499.External Links:LinkCited by:§2.
• Heyes (2018)C. HeyesCognitive gadgets: the cultural evolution of thinking.Harvard University Press.Cited by:§5.
• Hinton and Salakhutdinov (2006)G. E. Hinton and R. R. SalakhutdinovReducing the dimensionality of data with neural networks.Science313(5786),pp. 504–507.Cited by:Table F.3.
• Honget al.(2026)H. Hong, Y. Li, J. Chen, L. G. Huy, S. Ananiadou, J. Kim, and C. LinCan LLMs write reliable rubrics? A meta-evaluation for experiment reproduction.arXiv preprint arXiv:2607.12835.External Links:LinkCited by:§2.
• Huet al.(2025)C. Hu, L. Zhang, Y. Lim, A. Wadhwani, A. Peters, and D. KangREPRO-Bench: can agentic AI systems assess the reproducibility of social science research?.arXiv preprint arXiv:2507.18901.External Links:LinkCited by:§2.
• Huet al.(2022)E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. ChenLoRA: low-rank adaptation of large language models.InInternational Conference on Learning Representations,Cited by:§3.5.
• Huet al.(2024)S. Hu, C. Lu, and J. CluneAutomated design of agentic systems.arXiv preprint arXiv:2408.08435.External Links:LinkCited by:Table F.2.
• Huaet al.(2025)T. Hua, H. Hua, V. Xiang, B. Klieger, S. T. Truong, W. Liang, F. Sun, and N. HaberResearchCodeBench: benchmarking LLMs on implementing novel machine learning research code.arXiv preprint arXiv:2506.02314.External Links:LinkCited by:§2.
• Huanget al.(2024)Q. Huang, J. Vora, P. Liang, and J. LeskovecMLAgentBench: evaluating language agents on machine learning experimentation.arXiv preprint arXiv:2310.03302.External Links:LinkCited by:§2.
• Huanget al.(2026)Z. Huang, Y. Cao, A. K. Shargh, J. Luo, R. Mei, M. Zaki, Z. Liu, W. Bunstine, W. Jurayj, S. Goswami, T. McQueen, M. Shields, J. El-Awady, P. Clancy, B. V. Durme, N. Andrews, W. Walden, and D. KhashabiCan coding agents reproduce findings in computational materials science?.arXiv preprint arXiv:2605.00803.External Links:LinkCited by:§2.
• Inoueet al.(2025)Y. Inoue, K. Misaki, Y. Imajuku, S. Kuroki, T. Nakamura, and T. AkibaWider or deeper? Scaling LLM inference-time compute with adaptive branching tree search.arXiv preprint arXiv:2503.04412.External Links:LinkCited by:§2.
• Jaqueset al.(2019)N. Jaques, A. Lazaridou, E. Hughes, C. Gulcehre, P. A. Ortega, D. Strouse, J. Z. Leibo, and N. de FreitasSocial influence as intrinsic motivation for multi-agent deep reinforcement learning.InInternational Conference on Machine Learning,Cited by:Table F.3.
• Jianget al.(2025)Z. Jiang, D. Schmidt, D. Srikanth, D. Xu, I. Kaplan, D. Jacenko, and Y. WuAIDE: AI-Driven exploration in the space of code.arXiv preprint arXiv:2502.13138.External Links:LinkCited by:§2.
• Kangaset al.(2014)J. D. Kangas, A. W. Naik, and R. F. MurphyEfficient discovery of responses of proteins to compounds using active learning.BMC Bioinformatics15(1),pp. 143.Cited by:Table F.4.
• Kapoor and Narayanan (2022)S. Kapoor and A. NarayananLeakage and the reproducibility crisis in ML-based science.arXiv preprint arXiv:2207.07048.External Links:LinkCited by:§1.
• Karpathy (2026)A. KarpathyAutoresearch: AI agents running research on single-GPU nanochat training automatically.External Links:LinkCited by:§1.
• Kentonet al.(2024)Z. Kenton, N. Y. Siegel, J. Kramár, J. Brown-Cohen, S. Albanie, J. Bulian, R. Agarwal, D. Lindner, Y. Tang, N. Goodman,et al.On scalable oversight with weak LLMs judging strong LLMs.InAdvances in Neural Information Processing Systems,Cited by:§5.
• Kimet al.(2025)G. J. Kim, A. Wilf, L. Morency, and D. FriedFrom reproduction to replication: evaluating research agents with progressive code masking.arXiv preprint arXiv:2506.19724.External Links:LinkCited by:§2.
• Kimet al.(2026)S. Kim, J. Cho, B. Kwak, T. Kwon, L. Wang, N. Yang, X. Zhang, F. Wei, and J. YeoOn training large language models for long-horizon tasks: an empirical study of horizon length.arXiv preprint arXiv:2605.02572.External Links:LinkCited by:§A.1,§2.
• Kingma and Ba (2014)D. P. Kingma and J. BaAdam: a method for stochastic optimization.arXiv preprint arXiv:1412.6980.External Links:LinkCited by:Table F.3,§3.5.
• Kingma and Welling (2013)D. P. Kingma and M. WellingAuto-encoding variational Bayes.arXiv preprint arXiv:1312.6114.External Links:LinkCited by:Table F.1.
• Kirgiset al.(2026)P. Kirgis, S. Kapoor, A. Schwartz, S. Rabanser, D. Africa, K. Voudouris, V. Nguyen, T. Pilditch, M. Dubois, H. Coppock, C. Ududec, N. Nadgir, M. Orona, T. Bayer, D. Chan-Sew, Y. Ling, A. Shetty, H. Toner, G. Hadfield, S. Lazar, S. Newman, S. Tekofsky, R. Bommasani, and A. NarayananCan AI agents conduct open-ended AI research? Early evidence from two case studies.arXiv preprint arXiv:2607.27191.External Links:LinkCited by:§1.
• Kirschet al.(2022)L. Kirsch, J. Harrison, J. Sohl-Dickstein, and L. MetzGeneral-purpose In-Context learning by meta-learning transformers.arXiv preprint arXiv:2212.04458.Note:Workshop on Meta-Learning at NeurIPSExternal Links:LinkCited by:§2.
• Kirsch and Schmidhuber (2021)L. Kirsch and J. SchmidhuberMeta learning backpropagation and improving it.InAdvances in Neural Information Processing Systems,External Links:LinkCited by:Table F.2.
• Kirsch and Schmidhuber (2022)L. Kirsch and J. SchmidhuberEliminating meta optimization through self-referential meta learning.arXiv preprint arXiv:2212.14392.Note:First Conference on Automated Machine Learning (Workshop)External Links:LinkCited by:§2.
• Kirschet al.(2020)L. Kirsch, S. van Steenkiste, and J. SchmidhuberImproving generalization in meta reinforcement learning using learned objectives.InInternational Conference on Learning Representations,External Links:LinkCited by:§2.
• Kirsch (2025)L. KirschAutomating AI research.Doctoral dissertation,Università della Svizzera italiana.External Links:LinkCited by:§1.
• Konet al.(2025)P. T. J. Kon, J. Liu, X. Zhu, Q. Ding, J. Peng, J. Xing, Y. Huang, Y. Qiu, J. Srinivasa, M. Lee, M. Chowdhury, M. Zaharia, and A. ChenEXP-Bench: can AI conduct AI research experiments?.arXiv preprint arXiv:2505.24785.External Links:LinkCited by:§2.
• Krizhevskyet al.(2012)A. Krizhevsky, I. Sutskever, and G. E. HintonImageNet classification with deep convolutional neural networks.InAdvances in Neural Information Processing Systems,Cited by:Table F.2.
• Kwonet al.(2023)W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. StoicaEfficient memory management for large language model serving with PagedAttention.InProceedings of the 29th Symposium on Operating Systems Principles (SOSP),Cited by:Appendix H.
• Lamet al.(2023)R. Lam, A. Sanchez-Gonzalez, M. Willson, P. Wirnsberger, M. Fortunato, F. Alet, S. Ravuri, T. Ewalds, Z. Eaton-Rosen, W. Hu,et al.Learning skillful medium-range global weather forecasting.Science382(6677),pp. 1416–1421.Cited by:Table F.2,Table F.2.
• Langeet al.(2025)R. T. Lange, Y. Imajuku, and E. CetinShinkaEvolve: towards open-ended and sample-efficient program evolution.arXiv preprint arXiv:2509.19349.External Links:LinkCited by:§2.
• Larochelleet al.(2009)H. Larochelle, Y. Bengio, J. Louradour, and P. LamblinExploring strategies for training deep neural networks.Journal of Machine Learning Research10,pp. 1–40.Cited by:Table F.1.
• LeCunet al.(1998)Y. LeCun, L. Bottou, Y. Bengio, and P. HaffnerGradient-based learning applied to document recognition.Proceedings of the IEEE86(11),pp. 2278–2324.Cited by:Table F.1,Table F.2.
• Lehmanet al.(2023)J. Lehman, J. Gordon, S. Jain, K. Ndousse, C. Yeh, and K. O. StanleyEvolution through large models.InHandbook of Evolutionary Machine Learning,pp. 331–366.Cited by:Table F.1,Table F.2,§2.
• Li (2026)B. LiIncompressible knowledge probes: estimating black-box LLM parameter counts via factual capacity.arXiv preprint arXiv:2604.24827.External Links:LinkCited by:§1.
• Liet al.(2026)G. Li, B. D. Mishra, Z. Wang, J. Yan, Y. Chen, C. Li, L. T. Le, R. Han, G. Lee, H. Tong,et al.RubricEM: Meta-RL with rubric-guided policy decomposition beyond verifiable rewards.arXiv preprint arXiv:2605.10899.External Links:LinkCited by:§2.
• Lin (2026)J. LinSelf-improvement can self-regress: the rise-and-collapse failure mode of LLM self-training.arXiv preprint arXiv:2606.21090.External Links:LinkCited by:§A.1.
• Linet al.(2026)X. Lin, S. Zhu, Y. Chen, M. Chen, H. Sang, I. Paschalidis, Z. Wang, A. Pacchiano, and X. ZhangScaling in-context online learning capability of LLMs via cross-episode meta-RL.arXiv preprint arXiv:2602.04089.External Links:LinkCited by:Table F.1,Table F.2.
• Ling Team (2025)Ling TeamEvery step evolves: scaling reinforcement learning for trillion-scale thinking model.arXiv preprint arXiv:2510.18855.External Links:LinkCited by:§D.2.
• Liuet al.(2026)H. Liu, F. A. Tjiaranata, and C. TanVERITAS: towards a general-purpose replication tool for scientific research.arXiv preprint arXiv:2607.02931.External Links:LinkCited by:§2.
• Luet al.(2024)C. Lu, C. Lu, R. T. Lange, J. Foerster, J. Clune, and D. HaThe AI scientist: towards fully automated open-ended scientific discovery.arXiv preprint arXiv:2408.06292.External Links:LinkCited by:Table F.1,Table F.2,§1,§2,Table 1,§5.
• Lupidiet al.(2026)A. Lupidi, B. Gauri, T. S. Foster, B. A. Omari, D. Magka, A. Pepe, A. Audran-Reiss, M. Aghamelu, N. Baldwin, L. Cipolina-Kun, J. Gagnon-Audet, C. H. Leow, S. Lefdal, H. Mossalam, A. Moudgil, S. Nazir, E. Tewolde, I. Urrego, J. A. Estape, A. Budhiraja, G. Chaurasia, A. Charnalia, D. Dunfield, K. Hambardzumyan, D. Izcovich, M. Josifoski, I. Mediratta, K. Niu, P. Pathak, M. Shvartsman, E. Toledo, A. Protopopov, R. Raileanu, A. Miller, T. Shavrina, J. Foerster, and Y. BachrachAIRS-Bench: a suite of tasks for frontier AI research science agents.arXiv preprint arXiv:2602.06855.External Links:LinkCited by:Figure 2,Figure 2,§2.
• Majumderet al.(2024)B. P. Majumder, H. Surana, D. Agarwal, B. D. Mishra, A. Meena, A. Prakhar, T. Vora, T. Khot, A. Sabharwal, and P. ClarkDiscoveryBench: towards data-driven discovery with large language models.arXiv preprint arXiv:2407.01725.External Links:LinkCited by:§2.
• McDonnellet al.(2016)T. McDonnell, M. Lease, M. Kutlu, and T. ElsayedWhy is that relevant? Collecting annotator rationales for relevance judgments.InProceedings of the AAAI Conference on Human Computation and Crowdsourcing (HCOMP),Cited by:§3.3.
• Merchantet al.(2023)A. Merchant, S. Batzner, S. S. Schoenholz, M. Aykol, G. Cheon, and E. D. CubukScaling deep learning for materials discovery.Nature624,pp. 80–85.External Links:DocumentCited by:Table F.2,Table 1.
• Mishraet al.(2024)V. Mishra, S. Singh, D. Ahlawat, M. Zaki, V. Bihani, H. S. Grover, B. Mishra, S. Miret, Mausam, and N. M. A. KrishnanFoundational large language models for materials research.arXiv preprint arXiv:2412.09560.External Links:LinkCited by:Table F.2.
• Mnihet al.(2016)V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Harley, T. P. Lillicrap, D. Silver, and K. KavukcuogluAsynchronous methods for deep reinforcement learning.InInternational Conference on Machine Learning,Cited by:Table F.1,Table F.2.
• Moritzet al.(2018)P. Moritz, R. Nishihara, S. Wang, A. Tumanov, R. Liaw, E. Liang, M. Elibol, Z. Yang, W. Paul, M. I. Jordan, and I. StoicaRay: a distributed framework for emerging AI applications.In13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18),Cited by:Appendix H.
• Muggleton and Zauner (2006)S. Muggleton and K. ZaunerArtificial scientists.Note:University of SouthamptonCited by:§1.
• Muthukrishna and Henrich (2016)M. Muthukrishna and J. HenrichInnovation in the collective brain.Philosophical Transactions of the Royal Society B: Biological Sciences371(1690),pp. 20150192.Cited by:§5.
• Nathaniet al.(2025)D. Nathani, L. Madaan, N. Roberts, N. Bashlykov, A. Menon, V. Moens, A. Budhiraja, D. Magka, V. Vorotilov, G. Chaurasia, D. Hupkes, R. S. Cabral, T. Shavrina, J. Foerster, Y. Bachrach, W. Y. Wang, and R. RaileanuMLGym: a new framework and benchmark for advancing AI research agents.arXiv preprint arXiv:2502.14499.External Links:LinkCited by:§2.
• Nielsenet al.(2026)S. Nielsen, E. Cetin, P. Schwendeman, Q. Sun, J. Xu, and Y. TangLearning to orchestrate agents in natural language with the conductor.InInternational Conference on Learning Representations,Cited by:§2.
• Novikovet al.(2025)A. Novikov, N. Vũ, M. Eisenberger, E. Dupont, P. Huang, A. Z. Wagner, S. Shirobokov, B. Kozlovskii, F. J. R. Ruiz, A. Mehrabian, M. P. Kumar, A. See, S. Chaudhuri, G. Holland, A. Davies, S. Nowozin, P. Kohli, and M. BalogAlphaEvolve: a coding agent for scientific and algorithmic discovery.arXiv preprint arXiv:2506.13131.External Links:LinkCited by:§1,§2.
• NVIDIA (2025)NVIDIANeMo-RL.External Links:LinkCited by:Appendix H.
• NVIDIA (2026)NVIDIANeMo-Gym.External Links:LinkCited by:Appendix H.
• Ohet al.(2020)J. Oh, M. Hessel, W. M. Czarnecki, Z. Xu, H. P. van Hasselt, S. Singh, and D. SilverDiscovering reinforcement learning algorithms.InAdvances in Neural Information Processing Systems,Cited by:§2.
• OpenAI (2025)OpenAICodex CLI.External Links:LinkCited by:§3.4.
• OpenAI (2026)OpenAIGPT-5.5.Note:Knowledge cutoff: 1 December 2025External Links:LinkCited by:§A.1.
• Pathaket al.(2022)J. Pathak, S. Subramanian, P. Harrington, S. Raja, A. Chattopadhyay, M. Mardani, T. Kurth, D. Hall, Z. Li, K. Azizzadenesheli, P. Hassanzadeh, K. Kashinath, and A. AnandkumarFourCastNet: a global data-driven high-resolution weather model using adaptive Fourier neural operators.arXiv preprint arXiv:2202.11214.External Links:LinkCited by:Table F.2,Table F.2.
• Qianget al.(2025)R. Qiang, Y. Zhuang, Y. Li, D. Sagar V K, R. Zhang, C. Li, I. Wong, S. Yang, P. Liang, C. Zhang, and B. DaiMLE-Dojo: interactive environments for empowering LLM agents in machine learning engineering.InAdvances in Neural Information Processing Systems,Cited by:§2.
• Qiuet al.(2026)S. Qiu, J. Deng, Y. Deng, H. Dong, J. Fu, M. Li, Z. Li, Z. Zhang, H. Zheng, L. Bao, A. Lv, Z. Mo, Y. Niu, Y. Peng, Y. Tian, Y. Wang, Z. Wang, Z. Wang, J. Wei, L. Wu, A. Xue, L. Yang, G. Yuan, X. Zhan, J. Zhang, Z. Zheng, P. Liu, L. Zhen, K. Li, Q. Li, Z. Zhou, G. Nian, Y. Xiao, Q. Cao, L. Dai, X. Feng, P. Gao, Y. Gu, C. Liu, J. Liu, M. Luo, Y. Ma, L. Peng, H. Song, S. Wang, C. Wang, T. Wang, Y. Wang, C. Wu, P. Zhao, and H. X. ZhuPRBench: end-to-end paper reproduction in physics research.arXiv preprint arXiv:2603.27646.External Links:LinkCited by:§2.
• Qwen Team (2026)Qwen TeamQwen3.6-27B: flagship-level coding in a 27B dense model.Note:Model weights:https://huggingface.co/Qwen/Qwen3.6-27BExternal Links:LinkCited by:§A.1,§1,§3.5.
• Raissiet al.(2017)M. Raissi, P. Perdikaris, and G. E. KarniadakisPhysics informed deep learning (Part I): data-driven solutions of nonlinear partial differential equations.arXiv preprint arXiv:1711.10561.External Links:LinkCited by:Table F.4.
• Ramsundaret al.(2015)B. Ramsundar, S. Kearnes, P. Riley, D. Webster, D. Konerding, and V. PandeMassively multitask networks for drug discovery.arXiv preprint arXiv:1502.02072.External Links:LinkCited by:§A.1.
• Ranket al.(2026)B. Rank, H. Bhatnagar, A. Prabhu, S. Eisenberg, K. Nguyen, M. Bethge, and M. AndriushchenkoPostTrainBench: can LLM agents automate LLM post-training?.arXiv preprint arXiv:2603.08640.External Links:LinkCited by:§2.
• Realet al.(2020)E. Real, C. Liang, D. So, and Q. LeAutoML-zero: evolving machine learning algorithms from scratch.InInternational Conference on Machine Learning,Cited by:§2.
• Rechtet al.(2011)B. Recht, C. Ré, S. J. Wright, and F. NiuHOGWILD!: a lock-free approach to parallelizing stochastic gradient descent.InAdvances in Neural Information Processing Systems,Cited by:Table F.2.
• Reedet al.(2022)S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg,et al.A generalist agent.arXiv preprint arXiv:2205.06175.External Links:LinkCited by:Table F.1,Table F.2,§5.
• Romera-Paredeset al.(2024)B. Romera-Paredes, M. Barekatain, A. Novikov, M. Balog, M. P. Kumar, E. Dupont, F. J. Ruiz, J. S. Ellenberg, P. Wang, O. Fawzi,et al.Mathematical discoveries from program search with large language models.Nature625(7995),pp. 468–475.Cited by:§2.
• Ruppet al.(2012)M. Rupp, A. Tkatchenko, K. Müller, and O. A. Von LilienfeldFast and accurate modeling of molecular atomization energies with machine learning.Physical Review Letters108(5),pp. 058301.Cited by:Table F.4,§5.
• Schicket al.(2023)T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. ScialomToolformer: language models can teach themselves to use tools.InAdvances in Neural Information Processing Systems,Cited by:Table F.2.
• Schmidgallet al.(2025)S. Schmidgall, Y. Su, Z. Wang, X. Sun, J. Wu, X. Yu, J. Liu, M. Moor, Z. Liu, and E. BarsoumAgent laboratory: using LLM agents as research assistants.Findings of the Association for Computational Linguistics: EMNLP 2025,pp. 5977–6043.Cited by:§2.
• Schmidhuberet al.(1997)J. Schmidhuber, J. Zhao, and M. WieringShifting inductive bias with success-story algorithm, adaptive Levin search, and incremental self-improvement.Machine Learning28(1),pp. 105–130.Cited by:Table F.3.
• Schmidhuber (1987)J. SchmidhuberEvolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-… hook.Ph.D. Thesis,Technische Universität München.Cited by:§2.
• Schmidhuber (1991)J. SchmidhuberA possibility for implementing curiosity and boredom in model-building neural controllers.InProc. of the international conference on simulation of adaptive behavior: From animals to animats,Cited by:§1.
• Schmidhuber (1993)J. SchmidhuberA ‘self-referential’weight matrix.InInternational Conference on Artificial Neural Networks,Cited by:§2.
• Secretanet al.(2008)J. Secretan, N. Beato, D. B. D Ambrosio, A. Rodriguez, A. Campbell, and K. O. StanleyPicbreeder: evolving pictures collaboratively online.InProceedings of the SIGCHI Conference on Human Factors in Computing Systems,Cited by:§2.
• Semmelrocket al.(2025)H. Semmelrock, T. Ross-Hellauer, S. Kopeinik, D. Theiler, A. Haberl, S. Thalmann, and D. KowaldReproducibility in machine-learning-based research: overview, barriers, and drivers.AI Magazine46(2),pp. e70002.External Links:DocumentCited by:§1.
• Seoet al.(2026)M. Seo, J. Baek, S. Lee, and S. J. HwangPaper2Code: automating code generation from scientific papers in machine learning.arXiv preprint arXiv:2504.17192.External Links:LinkCited by:§2.
• Shaoet al.(2025)R. Shao, A. Asai, S. Z. Shen, H. Ivison, V. Kishore, J. Zhuo, X. Zhao, M. Park, S. G. Finlayson, D. Sontag,et al.DR Tulu: reinforcement learning with evolving rubrics for deep research.arXiv preprint arXiv:2511.19399.External Links:LinkCited by:§2.
• Shaoet al.(2024)Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. GuoDeepSeekMath: pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300.External Links:LinkCited by:§1,§1,§2,§3.5.
• Shenet al.(2026)W. F. Shen, X. Qiu, C. Whitehouse, L. Alazraki, S. Goel, F. Barbieri, T. Willi, A. Mathur, and I. LeontiadisRethinking rubric generation for improving LLM judge and reward modeling for open-ended tasks.arXiv preprint arXiv:2602.05125.External Links:LinkCited by:§2.
• Shoeybiet al.(2019)M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. CatanzaroMegatron-LM: training multi-billion parameter language models using model parallelism.arXiv preprint arXiv:1909.08053.External Links:LinkCited by:Appendix H.
• Siet al.(2026)C. Si, Z. Yang, Y. Choi, E. Candès, D. Yang, and T. HashimotoTowards execution-grounded automated AI research.arXiv preprint arXiv:2601.14525.External Links:LinkCited by:Table F.3.
• Siegelet al.(2026)Z. S. Siegel, S. Kapoor, N. Nadgir, B. Stroebl, and A. NarayananCORE-Bench: fostering the credibility of published research through a computational reproducibility agent benchmark.arXiv preprint arXiv:2409.11363.External Links:LinkCited by:§2.
• Silveret al.(2016)D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot,et al.Mastering the game of Go with deep neural networks and tree search.Nature529(7587),pp. 484–489.Cited by:Appendix E.
• Srivastavaet al.(2014)N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. SalakhutdinovDropout: a simple way to prevent neural networks from overfitting.Journal of Machine Learning Research15(1),pp. 1929–1958.Cited by:Table F.2.
• Stanley and Lehman (2015)K. O. Stanley and J. LehmanWhy greatness cannot be planned: the myth of the objective.Springer International Publishing.External Links:Document,ISBN 978-3-319-15524-1Cited by:§2.
• Staraceet al.(2025)G. Starace, O. Jaffe, D. Sherburn, J. Aung, J. S. Chan, L. Maksin, R. Dias, E. Mays, B. Kinsella, W. Thompson, J. Heidecke, A. Glaese, and T. PatwardhanPaperBench: evaluating AI’s ability to replicate AI research.arXiv preprint arXiv:2504.01848.External Links:LinkCited by:Appendix E,§2.
• Suet al.(2025)H. Su, S. Diao, X. Lu, M. Liu, J. Xu, X. Dong, Y. Fu, P. Belcak, H. Ye, H. Yin,et al.ToolOrchestra: elevating intelligence via efficient model and tool orchestration.arXiv preprint arXiv:2511.21689.External Links:LinkCited by:§2.
• Surinaet al.(2025)A. Surina, A. Mansouri, L. Quaedvlieg, A. Seddas, M. Viazovska, E. Abbe, and C. GulcehreAlgorithm discovery with LLMs: evolutionary search meets reinforcement learning.arXiv preprint arXiv:2504.05108.External Links:LinkCited by:Appendix E,§2.
• Sutton (2019)R. S. SuttonThe bitter lesson.External Links:LinkCited by:§2.
• Tanget al.(2025)J. Tang, L. Xia, Z. Li, and C. HuangAI-Researcher: autonomous scientific innovation.arXiv preprint arXiv:2505.18705.External Links:LinkCited by:§2.
• Teamet al.(2023)A. A. Team, J. Bauer, K. Baumli, S. Baveja, F. Behbahani, A. Bhoopchand, N. Bradley-Schmieg, M. Chang, N. Clay, A. Collister, V. Dasagi, L. Gonzalez, K. Gregor, E. Hughes, S. Kashem, M. Loks-Thompson, H. Openshaw, J. Parker-Holder, S. Pathak, N. Perez-Nieves, N. Rakicevic, T. Rocktäschel, Y. Schroecker, J. Sygnowski, K. Tuyls, S. York, A. Zacherl, and L. ZhangHuman-timescale adaptation in an open-ended task space.arXiv preprint arXiv:2301.07608.External Links:LinkCited by:§2.
• Teamet al.(2021)O. E. L. Team, A. Stooke, A. Mahajan, C. Barros, C. Deck, J. Bauer, J. Sygnowski, M. Trebacz, M. Jaderberg, M. Mathieu,et al.Open-ended learning leads to generally capable agents.arXiv preprint arXiv:2107.12808.External Links:LinkCited by:§2.
• The Kubernetes Authors (2022)The Kubernetes AuthorsKueue: kubernetes-native job queueing.External Links:LinkCited by:Appendix H.
• Toledoet al.(2025)E. Toledo, K. Hambardzumyan, M. Josifoski, R. Hazra, N. Baldwin, A. Audran-Reiss, M. Kuchnik, D. Magka, M. Jiang, A. M. Lupidi, A. Lupu, R. Raileanu, K. Niu, T. Shavrina, J. Gagnon-Audet, M. Shvartsman, S. Sodhani, A. H. Miller, A. Charnalia, D. Dunfield, C. Wu, P. Stenetorp, N. Cancedda, J. N. Foerster, and Y. BachrachAI research agents for machine learning: search, exploration, and generalization in MLE-bench.arXiv preprint arXiv:2507.02554.External Links:LinkCited by:§2.
• Viswanathanet al.(2026)V. Viswanathan, Y. Sun, X. Kong, M. Cao, G. Neubig, and S. WuChecklists are better than reward models for aligning language models.InAdvances in Neural Information Processing Systems,Cited by:§2.
• Wanget al.(2024)G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. AnandkumarVoyager: an open-ended embodied agent with large language models.Transactions on Machine Learning Research.Note:External Links:ISSN 2835-8856,LinkCited by:Table 1.
• Wanget al.(2025)W. Wang, P. Piękos, L. Nanbo, F. Laakom, Y. Chen, M. Ostaszewski, M. Zhuge, and J. SchmidhuberHuxley-Gödel machine: human-level coding agent development by an approximation of the optimal self-improving machine.arXiv preprint arXiv:2510.21614.External Links:LinkCited by:§2.
• Wanget al.(2026)X. Wang, H. Zhang, H. Wang, Y. Shi, R. Li, K. Han, C. Tong, H. Deng, A. K. Taylor, R. Sun, Y. Zhu, J. Cong, Y. Sun, and W. WangARLArena: a unified framework for stable agentic reinforcement learning.arXiv preprint arXiv:2602.21534.External Links:LinkCited by:§2.
• Wenget al.(2025)Y. Weng, M. Zhu, G. Bao, H. Zhang, J. Wang, Y. Zhang, and L. YangCycleResearcher: improving automated research via automated review.arXiv preprint arXiv:2411.00816.External Links:LinkCited by:§2,§2.
• Wijket al.(2025)H. Wijk, T. Lin, J. Becker, S. Jawhar, N. Parikh, T. Broadley, L. Chan, M. Chen, J. Clymer, J. Dhyani, E. Ericheva, K. Garcia, B. Goodrich, N. Jurkovic, H. Karnofsky, M. Kinniment, A. Lajko, S. Nix, L. Sato, W. Saunders, M. Taran, B. West, and E. BarnesRE-Bench: evaluating frontier AI R&D capabilities of language model agents against human experts.arXiv preprint arXiv:2411.15114.External Links:LinkCited by:§2.
• Xianget al.(2025)Y. Xiang, H. Yan, S. Ouyang, L. Gui, and Y. HeSciReplicate-Bench: benchmarking LLMs in agent-driven algorithmic reproduction from research papers.arXiv preprint arXiv:2504.00255.External Links:LinkCited by:§2.
• Xie and Grossman (2017)T. Xie and J. C. GrossmanCrystal graph convolutional neural networks for an accurate and interpretable prediction of material properties.arXiv preprint arXiv:1710.10324.External Links:LinkCited by:§A.1.
• Xieet al.(2026)Y. Xie, N. Thomas, N. Hansen, Y. Fu, L. Li, and X. WangTIPS: turn-level information-potential reward shaping for search-augmented LLMs.InInternational Conference on Learning Representations,Cited by:§2.
• Xuet al.(2025a)C. Xu, S. Zhu, and V. ViswanathanCLOUD: a scalable and physics-informed foundation model for crystal representation learning.arXiv preprint arXiv:2506.17345.External Links:LinkCited by:§A.1,Table F.2,Table F.4.
• Xuet al.(2025b)P. Xu, Z. Li, X. Xing, G. Zhang, D. Li, and K. ShiHybrid reward normalization for process-supervised non-verifiable agentic tasks.arXiv preprint arXiv:2509.25598.External Links:LinkCited by:§2.
• Yanet al.(2025)S. Yan, R. Li, Z. Luo, Z. Wang, D. Li, L. Jing, K. He, P. Wu, G. Michalopoulos, Y. Zhang, Z. Zhang, M. Zhang, Z. Chen, and X. DuLMR-BENCH: evaluating LLM agent’s ability on reproducing language modeling research.arXiv preprint arXiv:2506.17335.External Links:LinkCited by:§2.
• Yanget al.(2024)C. Yang, X. Wang, Y. Lu, H. Liu, Q. V. Le, D. Zhou, and X. ChenLarge language models as optimizers.InInternational Conference on Learning Representations,Cited by:§2.
• Yanget al.(2026)S. Yang, J. He-Yueya, and P. LiangReinforcement learning for machine learning engineering agents.InInternational Conference on Learning Representations,Vol.2026,pp. 156989–157011.Cited by:§2.
• Yanget al.(2025)S. Yang, J. Kautz, and A. HatamizadehGated delta networks: improving Mamba2 with delta rule.InInternational Conference on Learning Representations,Cited by:Appendix H.
• Yuet al.(2025)Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, T. Fan, G. Liu, L. Liu, X. Liu, H. Lin, Z. Lin, B. Ma, G. Sheng, Y. Tong, C. Zhang, M. Zhang, W. Zhang, H. Zhu, J. Zhu, J. Chen, J. Chen, C. Wang, H. Yu, W. Dai, Y. Song, X. Wei, H. Zhou, J. Liu, W. Ma, Y. Zhang, L. Yan, M. Qiao, Y. Wu, and M. WangDAPO: an open-source LLM reinforcement learning system at scale.arXiv preprint arXiv:2503.14476.External Links:LinkCited by:§D.2.
• Yuanet al.(2026)S. Yuan, J. Chen, J. Zheng, M. Li, L. Feng, D. Wang, T. Xiang, T. Liu, and B. AnUnderstanding diversity collapse in RLVR via the lens of overtraining.arXiv preprint arXiv:2606.15455.External Links:LinkCited by:§A.1.
• Z.ai (2026)Z.aiGLM-5.2: built for long-horizon tasks.External Links:LinkCited by:§4.2.
• Zelikmanet al.(2023)E. Zelikman, E. Lorch, L. Mackey, and A. T. KalaiSelf-taught optimizer (STOP): recursively self-improving code generation.arXiv preprint arXiv:2310.02304.External Links:LinkCited by:§2.
• Zeniet al.(2025)C. Zeni, R. Pinsler, D. Zügner, A. Fowler, M. Horton, X. Fu, Z. Wang, A. Shysheya, J. Crabbé, S. Ueda,et al.A generative model for inorganic materials design.Nature639,pp. 624–632.Cited by:Table F.2.
• Zhanget al.(2026a)J. Zhang, S. Hu, C. Lu, R. Lange, and J. CluneDarwin Gödel Machine: open-ended evolution of self-improving agents.arXiv preprint arXiv:2505.22954.External Links:LinkCited by:Table F.2,§2,Table 1.
• Zhanget al.(2026b)J. Zhang, B. Zhao, W. Yang, J. Foerster, J. Clune, M. Jiang, S. Devlin, and T. ShavrinaHyperagents.arXiv preprint arXiv:2603.19461.External Links:LinkCited by:§2.
• Zhaoet al.(2025)B. Zhao, D. Magka, M. Jiang, X. Li, R. Raileanu, T. Shavrina, J. Gagnon-Audet, K. Niu, S. Sodhani, M. Shvartsman, A. Lupu, A. Lupidi, E. Toledo, K. Hambardzumyan, M. Josifoski, T. Foster, L. Cipolina-Kun, A. Charnalia, D. Dunfield, A. H. Miller, O. M. Aodha, J. Foerster, and Y. BachrachThe automated LLM speedrunning benchmark: reproducing NanoGPT improvements.arXiv preprint arXiv:2506.22419.External Links:LinkCited by:§2.
• Zhaoet al.(2026)X. Zhao, Z. Sang, Y. Li, Q. Shi, W. Zhao, S. Wang, D. Zhang, X. Han, Z. Liu, and M. SunAutoReproduce: automatic AI experiment reproduction with paper lineage.arXiv preprint arXiv:2505.20662.External Links:LinkCited by:§2.
• Zucchetet al.(2025)N. Zucchet, J. Bornschein, S. Chan, A. Lampinen, R. Pascanu, and S. DeHow do language models learn facts? Dynamics, curricula and hallucinations.arXiv preprint arXiv:2503.21676.External Links:LinkCited by:§G.6.

Appendix

Appendix AGeneralisation

A.1Full-scale replication

Faraday is trained to complete scaled-down replications of a single figure given a one-hour time limit and a one-seventh MIG GPU slice. More precisely, during early stages of training, Faraday is given a time limit of3030minutes, which is increased to one hour for the later stages. Previous work has indicated that a horizon curriculum may induce effective generalisation to longer horizons than experienced during training(60). Here, we test Faraday’s ability to generalise to completing full-scale replications, given the time and compute necessary to do so.

We select eight replication tasks from outside Faraday’s training distribution, filtered using Claude Opus 4.8 such that at most eight hours and eight B300 GPUs should be sufficient to replicate the figure. Of the selected tasks, five are from AI-for-science papers(147;20;106;149;12)and three are from ML papers(40;79;156). Three of the eight papers were first made publicly available after the knowledge cutoff for Claude Opus 4.8(6), GPT-5.5(100)and Qwen3.6-27B(104). For this evaluation, we also increase Faraday’s context limit to its maximum256256K, up from128128K during training; we choose the eight-hour time limit because this is the approximate time horizon allowed by the increased context limit without compaction.

We use Claude Opus 4.8 to estimate how many hours and how many B300 GPUs (up to the cap of eight hours and eight GPUs) should be necessary to fully replicate one figure from each paper without any scale-down. We then run one rollout for each task using Faraday and Claude, under the time and compute resources estimated. We find that Faraday outperforms Claude on average according to our rubric judge (Figure A.1). In other words, Faraday generalises to longer time horizons and larger compute resources. The caveat is that we did not validate our rubric judges with human ratings on rollouts at this scale, an important step for future work.

Figure A.1:Faraday outperforms Claude on full-scale replications.On eight held-out replication tasks, Claude and Faraday are evaluated on their ability to replicate the experimental results at full scale, with access to up to eight hours and eight B300 GPUs (as estimated to be necessary to complete a full replication without any scale-down). Faraday performs better than Claude according to our rubric judge (horizontal rules represent the means over the tasks), and outperforms it on five of eight tasks.

A.2Stronger coding agent tool

Since the capabilities of frontier coding agents increase frequently, it would be useful if Faraday were able to make effective use of stronger coding agent tools than it was trained on. To evaluate this generalisation, recall that Faraday is trained initially with GPT-5.4 mini as the model backing the Codex tool. We take the last checkpoint from Faraday’s lineage which was trained only with GPT-5.4 mini as a tool, and we evaluate it on the Replica test split using first GPT-5.4 mini and then GPT-5.5 as the coding agent model.Figure A.2demonstrates that this partially trained version of Faraday makes effective use of the stronger coding agent to boost its scores on the tasks.

Figure A.2:The trained AI Scientist is not specialised to its train-time coding agent.The last checkpoint from the Faraday training lineage that is trained entirely with GPT-5.4 mini as the coding agent performs better on held-out AI-for-science tasks when the coding agent is swapped out for GPT-5.5, demonstrating that Faraday can generalise to use a stronger coding agent without the need for retraining. Each point is an individual task (mean of four rollouts), and the horizontal rules are the means across tasks.

Appendix BAblations

B.1Turn-level credit assignment

Figure B.1:Turn-level credit assignment stabilises training.Starting from a late checkpoint in Faraday’s training lineage, the removal of our turn-level credit assignment results in the rapid destabilisation and collapse of training.(left)With turn-level credit assignment, the carry-forward mean reward (the mean over all tasks of the most recent reward achieved in that task) rises steadily, whereas with uniform credit assignment it collapses after5050steps.(centre)Around the same time, the token entropy of the policy trained without turn-level credit assignment spikes and then collapses.(right)Leading up to the collapse, the Jensen–Shannon divergence between the generation policy and the training policy (which differ due to asynchronous training) begins to increase, eventually growing by two orders of magnitude. We find this to be a common precursor to such collapses.

B.2Coding agent as a tool

Figure B.2:The coding agent tool is important for best performance.We train the Qwen3.6-27B model from scratch using the same hyperparameters as the final tailpatch of the Faraday training lineage but without access to the coding agent tool (“Faraday Coder”). Training collapses after approximately300300steps, after performing more weakly at equal step count to the Faraday lineage. Notably, during the pre-collapse period, Faraday Coder had twice the time horizon (6060minutes) of Faraday (3030minutes), and still performed consistently worse. This suggests that the ceiling for the coder model is lower than that for the researcher model. As inFigure B.1(left), the curves show the carry-forward mean reward, with the ghosted curves representing the per-step mean reward.

Appendix CAnalyses

C.1Credit assignment distribution

Figure C.1:The judge spreads credit non-uniformly across each rollout.We draw data from577,585577{,}585turns of the Faraday training lineage with turn-level credit assignment enabled (every turn from steps491491–635635).(left)There is a concentration of credit in the early to middle stages of a rollout, where load-bearing decisions are most commonly made.(right)More weight is given to turns that delegate to the coding agent tool, capturing the importance of appropriate delegation. Turn types are assigned post-hoc by a regular-expression match on the turn text.

C.2Scores by rubric dimension

Figure C.2:Faraday’s advantage is concentrated in experimental depth and claim reproduction.The left-hand panel ofFigure 2is split out into the five score dimensions of our rubric judge. As in that figure, each panel shows the fraction of the242242tasks in the Replica train split with a mean score over eight rollouts of at leastσ\sigmain the corresponding rubric dimension. We omit the SEM for visual clarity. Faraday’s replications consistently have more experimental depth, better claim reproduction, and higher visual fidelity to the original figure. Faraday approximately matches Claude in implementation fidelity (faithfulness to the paper’s methodology) and scientific integrity (not cheating while completing the task). SeeSection 3.2for a description of the rubric dimensions.

C.3Within-rollout behaviour

Figure C.3:Faraday produces moments of insight without an evolutionary harness. Within individual rollouts, we plot the time elapsed in minutes (xx-axis) against rubric judge score (yy-axis). On nine randomly sampled test tasks, we select the strongest of Faraday’s eight evaluation rollouts used forFigure 2. The green line represents the best rubric judge score seen so far, and the grey line represents the rubric judge score of the latest plot. Faraday builds on previous discoveries to discover new insights at test time, similarly to existing AI Scientist agents. Unlike these agents, Faraday requires no hand-coded evolutionary harness, and has no access to a reward function at test time. Theyy-axis for this plot is computed post-hoc by our rubric judge and never provided to Faraday.

Appendix DSupplementary methods

D.1Task space

Undoubtedly some of the papers we choose, including their figures, are in the pre-training dataset for frontier multimodal models. We are equally certain that no frontier model has been trained on the process data that produced the figure in the original paper; this data was simply never recorded, let alone made available for model training. Moreover, in many if not all cases the original figures were generated under very different resource and time constraints than in Replica. Since our object of study is the process of replication, not the exact fidelity of the output figure, we do not view contamination of the pre-training dataset with paper details as a problem, although we do take it into account in interpreting our results (Section 4.2). The decision to redact the figure was taken to decontaminate the context of the agent and thus encourage a focus on rigorous replication process, and to make it easier for the judge to detect any cheating behaviour, such as reverse engineering the data by downloading the original plot.

D.2Post-training

Rollouts are generated on dedicated inference workers and consumed asynchronously by the training engine, with rollout staleness capped at 3–6 optimiser steps. Generation and training both run usingbf16precision, which we find to be more stable thanfp8. Unlike vanilla GRPO, we use a leave-one-out baseline for the group-relative advantage(2), DAPO’s token-level loss and asymmetric clip-higher(155)withϵℓ=0.15\epsilon_{\ell}=0.15andϵh=0.35\epsilon_{h}=0.35, and IcePop’s token-level discrepancy masking(81), which zeroes tokens whose sampler–trainer likelihood ratio falls outside[0.3,4][0.3,4]. We keep GRPO’s small KL penalty relative to the base model,β=3×10−3\beta=3{\times}10^{-3}. The final checkpoint (step659659) is the result of a multi-stage training lineage: most of training ran with the cheaper GPT-5.4 mini coding agent and a shorter3030-minute task duration, and the final stages tail-patched the recipe with GPT-5.5 as coding tool and a one-hour task duration (Table D.1).

Table D.1:Stages of the Faraday training lineage.For cost, curriculum-learning, and stability reasons, Faraday’s post-training consisted of multiple stages.

Appendix ESupplementary discussion

The Replica task space and the CAT paradigm were deliberately constructed to facilitate scaling. Papers accepted at ICML, ICLR, and NeurIPS alone could yield a total of36,00036{,}000tasks per year, two orders of magnitude larger than our current set. Training over a greater diversity of resource constraints, not to mention on task variants that demand innovation beyond replication, leads to a further combinatorial explosion. However, the path is not without difficulty. Inevitably, some tasks represent results that simply do not replicate. Thus far, we have been fairly insulated from this problem by our choice of papers that are well-regarded and highly cited. Were we to scale the task space, we would need to develop judges that recognise non-replicability and agents that robustly test and honestly report it. Expanding the task space may allow us to train an agent sufficiently general to evaluate on paper replication benchmarks with quite different APIs, such as PaperBench(131).

Another axis of scaling is the size of the base model we use as the starting point to post-train Faraday. Scaling this by an order of magnitude would provide a much stronger set of foundational capabilities. Similarly, moving to a multimodal base model may facilitate further improvements, seeing as our tasks rely on the generation and interpretation of figures. Furthermore, the CAT paradigm does not prevent the use and optimisation of a harness around the outer model at train time. An outer harness might function as an inference-time improvement operator, yielding stronger trajectories from which to learn(128;5;133).

Appendix FHuman studies

Table F.1:The ten tasks used for the judge comparison study.Labels are used inFigure 3(left). All ten come from the train split.We run two human studies with different questions. Thejudge comparisonstudy asks whether the rubric judge tracks human taste where it and the baseline judge disagree. Theagent comparisonstudy asks whether human experts agree with the rubric judge when it places Faraday clearly ahead of a baseline agent. Both use the interface and instructions ofSection G.4, and differ only in which rollouts a participant sees. For both studies, ranking is blind: the participants receive no information identifying which model is responsible for any of the rollouts they are ranking. In both studies the rollout selection rule is determined before any of that study’s datapoints are collected.

F.1Judge comparison

Methods.

A participant ranks six rollouts of one task: four from the training run and one each from Claude and Codex. The four training rollouts are sampled from four windows of5050training steps: 143–192, 291–340, 438–487, 586–635, with the intention of providing trajectories of varying quality. Within each window we draw one rollout uniformly at random from those that finished and produced a figure. For Claude and Codex we run a set of eight rollouts on each task, and take one from each set uniformly at random.

Tasks are selected for judge disagreement, since agreement carries no information about which judge tracks human taste better. The initial task pool contains131131tasks that we judge to be tractable for an expert human with a general machine-learning background. From these we keep the tasks whose six rollout scores are more spread than the median task under both judges. We then draw tasks in the order of most disputed first, skipping papers already drawn, and stopping at ten tasks from ten papers (Table F.1). The task-selection criteria are symmetric in the two judges, so the selection procedure does not favour either one.

A pair of rollouts is considered disputed when the two judges order it oppositely and the gap between rollout scores for both judges exceeds0.020.02. This is to avoid gaps that are inside measured judge noise. As participants rank the 6 rollouts in a task, they implicitly weigh in on disputed pairs contained within those rollouts. This lets us estimate how often a human sides with the rubric judge when the two judges disagree. We do that using a binomial mixed-effects model carrying a task random effect.

Results.

We collect7676rankings from1919participants. We run a binomial mixed-effects model with a task random effect, against the null hypothesis that humans side with the two judges equally often on disputed pairs. Participants side with the rubric judge on63%63\%of disputed pairs, higher than chance but not significantly so (p=0.109p=0.109).

F.2Agent comparison

Methods.

Each participant ranks three rollouts per task: one from Faraday and one each from Claude and Codex. All three rollouts are drawn from the same evaluation set used forFigure 2andFigure 4. Rollouts are selected for judge margin, since the purpose of the study is to establish whether participants agree with the rubric judge on tasks where it considers Faraday to be clearly superior. A triplet is eligible when the rubric judge puts Faraday at least0.20.2above both baselines, on the judge’s scale from00to11. As in the previous study, we filter the eligible tasks according to whether we judge them tractable for judging by an expert human with a general machine-learning background, yielding the set inTable F.2. The design supports a conditional claim: we may infer whether humans agree with the judge’s verdicts when it indicates a clear advantage for Faraday, but not Faraday’s average standing against Claude or Codex in the eyes of humans.

Results.

We collect4141rankings from1111participants. We run a binomial test against the null hypothesis that humans have no preference for Faraday’s rollouts. Participants prefer Faraday to Claude in80%80\%of rankings and to Codex in88%88\%, and rank it above both baselines in71%71\%, all significantly higher than chance (p<0.01p<0.01).

Table F.2:The2929tasks used for the agent comparison study.Tasks are drawn from both the train and the test split.PaperFig.What the figure claimsML (train)A Generalist Agent 1105A single pretrained generalist policy reaches a large fraction of expert score across many control tasks.Additive Logistic Regression 291On a nested-spheres problem both AdaBoost variants drive test error below bagging as trees are added.Asynchronous Methods for Deep Reinforcement Learning 894Every asynchronous method trains faster in wall-clock time as parallel actor-learners are added.Automated Design of Agentic Systems 483Searching over agent code with a growing archive keeps finding better ARC agents as the search proceeds.Darwin-Gödel Machine 1604Self-improved agents keep their advantage when transferred to other models, benchmarks, and programming languages.Diversity is All You Need 276DIAYN’s reward on a hierarchical task rises with the number of skills, and beats VIME exploration.Dropout 1294Dropout lowers test error at every depth and width tried.Evolution through Large Models 7615Fine-tuned LLM mutators complete out-of-distribution solutions better when trained at a higher threshold.Gradient-Based Learning Applied to Document Recognition 7512Memory-based classifiers need orders of magnitude more storage than convolutional networks.Greedy Function Approximation 283MARS makes more frequent larger and smaller errors than boosted trees.HOGWILD! 1093Lock-free parallel SGD speeds up matrix completion substantially, and holds much of that speedup as update delays grow.ImageNet Classification with Deep Convolutional Neural Networks 701A four-layer convolutional network with ReLUs reaches25%25\%training error on CIFAR-10 about six times faster than the same network with tanh units.Manifold Regularization 105On USPS digits, Laplacian regularisation cuts the error of RLS and SVM, with the largest gain when labels are scarce.Manifold Regularization 108On WebKB text classification the Laplacian variants lead at every label budget, and improve further with more unlabelled data.Meta-Learning Backpropagation And Improving It 655A meta-RNN cloned from backpropagation learns MNIST faster after meta-learning, without losing ground out of distribution on Fashion-MNIST.Scaling In-Context Online Learning Capability of LLMs via Cross-Episode Meta-RL 801Cross-episode meta-RL lifts a small model to frontier level on unseen interactive environments.The AI Scientist 834The automated reviewer’s score distribution for AI-generated papers varies across three research domains and four foundation models.Toolformer 1134GPT-J models greater than10001000M parameters finetuned with Toolformer learn to make good use of API calls.AI-for-science (test)Foundational Large Language Models for Materials Research 883Continued pretraining on materials literature beats general frontier models at extracting structured materials information.A Foundation Model for the Earth System 152The model’s air-quality forecasts match or beat the operational CAMS system at a fraction of the compute.A Generative Model for Inorganic Materials Design 1592The generated crystals are more often stable, unique, and new than those of earlier generative baselines.Accurate Structure Prediction of Biomolecular Interactions with AlphaFold 3 14The model’s own confidence scores track the accuracy of its predicted interfaces and chains.CLOUD 1492A symmetry-aware string representation matches structure-based models on MatBench regression, and pretraining improves it further.FourCastNet 1011A9696-hour global near-surface wind forecast reproduces the observed field at0.25∘0.25^{\circ}resolution.FourCastNet 1014An ensemble forecast tracks Hurricane Michael’s path and rapid intensification over four days.GraphCast 722The model beats the operational HRES forecast at nearly all lead times.GraphCast 724Training on more recent data improves skill on a held-out later year, most at short lead times.MACE 83The model follows the reference energy along three cuts of a molecule’s potential energy surface more closely than BOTNet and NequIP.Scaling Deep Learning for Materials Discovery 872The discovered stable crystals reach compositions of four or more elements.

F.3“Innovation” tasks

Table F.3:The ten “innovation” tasks built from Replica train-split papers. The ten source tasks behindFigure 5(right) are drawn uniformly at random from the Replica splits. Each source figure yields two variants:(a)keeps the paper’s claim but swaps the dataset or environment, and(b)keeps the setting but changes the claim, usually reversing it. In both cases the “gold plot”, its caption and the paper text are rewritten together, so the published result is no longer the target and recall is of limited benefit. This table describes tasks drawn from the train split;Table F.4describes tasks drawn from the test split.Table F.4:The ten “innovation” tasks built from Replica test-split papers. Columns and variant construction are as inTable F.3.

Appendix GPrompts

G.1Faraday system prompt

⬇YouareFaraday,anautonomousAIresearcher.Youoperateinsideacontainerizedworkspace.#RoleYouarearesearcher,notacoder.Yourjobistoplanexperiments,analyzeresults,anditeratetowardthegoaldescribedinyourprompt.Youhaveacodingagentavailableforallimplementationwork--delegatecodingtaskstoitratherthanwritingcodeyourself.Inordertodoyourresearch,youthinkdeeplyandmakeaplanfirst,thenexecutestepbystepusingyourcodingagent.Yourcapabilities:-Runshellcommandsviathe‘shell‘tool.Inparticular,youcanusethistooltorunthe‘coding_agent.py‘scriptyouhaveaccessto,whichallowsyoutodelegatecodingworktoacapablesubagent.-Writefilesviathe‘apply_patch‘tool:‘***AddFile:‘tocreateorfullyoverwriteafile(e.g.‘writeup.md‘),‘***UpdateFile:‘forsurgicaleditstoafileyou’rekeepingmostlyintact.-Readfilesviathe‘read_file‘tool.-Listdirectoriesviathe‘list_dir‘tool.-Searchfilesviathe‘grep_files‘tool.#CodingAgentYouhaveaccesstoacodingagent--asubagentthatperformsmulti-stepcodingworkautonomously.Ithasitsownshell,reads/writesfilesinyourworkingdirectory,butdoesNOTshareyourconversationcontext.Italsodoesn’tseeyourtaskpromptorsystemprompt,soanycontextit’llneed--availableGPUresources,timeguidance,APIkeysintheenv,etc.--mustbethreadedthroughinthepromptyoupassit.Useitforanycodingtask:writingscripts,editingconfigs,debuggingerrors,andsoon.YoumustNOTwritecodeyourself--alwaysdelegateimplementationtothecodingagent.Youdrivetheresearch;thecodingagentdoesthecoding.Tousethecodingagent,runthe‘coding_agent.py‘scriptviathe‘shell‘tool:-‘pythoncoding_agent.py“<detailedprompt>“‘--resumesthepreviouscoding-agentsessionbydefault,carryingitsfullcontextacrosssoafollow-upbuildsonearlierwork.Thefirstcallstartsfresh.-‘pythoncoding_agent.py--fresh”<detailedprompt>“‘--startafreshsessioninstead(e.g.foranunrelatedtask).-‘pythoncoding_agent.py--budget‘--checkremainingtokenbudget.**Promptswithbackticks,‘$‘,quotes,orothershell-significantcharacters**:usestdinviaaquotedheredocinsteadofpassingasanargument--otherwisetheshellinterpretsthemandyourpromptbreaks.Pass‘-‘astheargumenttoreadstdin:‘‘‘pythoncoding_agent.py-<<’EOF’Implementfoo.Usethissnippetasreference:‘‘‘pythondefbar():...‘‘‘EOF‘‘‘Thesinglequotesaround‘’EOF’‘arerequired--theytellbashtopasstheheredocbodythroughliterallywithoutexpandinganything.Noneedtosetatimeout;butyoucaninstructthecodingagentforhowlongitshouldrun.Outputtokenbudget:{coding_agent_budget}tokens.**Makemanysmallcalls,notonebigone.**Eachcallshouldhaveonecleardeliverable.Scopedcallsgiveshortfeedbackloops.Onegiantall-in-onepromptisananti-pattern--course-correctingmeanscancellingthewholecallandrestarting,whichwastesalotofbudget.#Workflow1.**Planfirst.**ReadthepromptandAGENTS.md.Thenmakeaplan.2.**Executeiteratively.**Delegatecodingtothecodingagent;runexperiments;evaluate;adapt.3.**Persist.**Keepgoingautonomouslyuntilthetaskisfullyresolved--don’taskforclarification,makereasonabledecisions.Ifanapproachisn’tworking,pivotquickly.4.**Monitoryourtimebudget.**Wrapupwithenoughtimetoproducefinaldeliverables.#Authenticity**Neversimulateorfabricateexperiments.**Alwaysrunexperimentsforreal.Fabricatingresults,hard-codingexpectedvalues,generatingfakedata,mockingexperimentruns,orproducingpredeterminedoutputsthatdidnotcomefromactualexecutionwillscore0.Iftheoriginalscaleofanexperimentisinfeasiblewithintheavailablecomputeandtime,runaclearly-documentedscaled-downversion(smallermodel,fewersteps,fewerseeds)--thatisacceptable;fabricationorsimulationisnot.#FinishingWhenyouaresureyou’redone,respondwithabriefsummarymessageandnotoolcalls.Thisendstherollout.#Rules-Use‘apply_patch‘forallfilewrites.Tocreateafileorreplaceitsentirecontents(e.g.‘writeup.md‘),use‘***AddFile:‘--itoverwritesanexistingfile.Reserve‘***UpdateFile:‘fortargetededits;donotpasteawholenewversionas‘+‘linesunderit.Neverrewritewholefilesviashell.-Use‘read_file‘beforeassumingfilecontents;use‘list_dir‘/‘grep_files‘toexplore.-Fileand‘shell‘toolsoperateinyourworkingdirectory;usepathsrelativetoit(absolutepathsalsowork).-Foranylong-runningwork(atrainingrun,alongtestsuite,abuild,ora‘pythoncoding_agent.py‘delegation),doNOTsitidlewaitingforit--runitinthebackground.Pass‘background:true‘tothe‘shell‘tool:itreturnsa‘job_id‘immediatelysoyoukeepcontrolandcanmakeprogressonothersteps(readfiles,writeupanalysis,kickoffotherwork)whileitruns.Readitsincrementaloutputandexitstatuswith‘job_output(”<job_id>“)‘,andstopitwith‘kill_job(”<job_id>“)‘.Whenyouhavenothingelsetodo,theharnesswakesyouassoonasabackgroundjobfinishes--solaunchlongjobsinthebackgroundandkeepworkingratherthanblockingtheturn.Donotset‘timeout_ms‘togetherwith‘background:true‘.-NeverexposesecretsorAPIkeys.##Hardware-Ifthetasksays(orimplies)thataGPUisrequired(e.g.itinvolvesmodeltraining,fine-tuning,inferenceonanon-trivialmodel,CUDAkernels,etc.),**assumeaCUDAGPUisavailableanduseit**.DonottrainorruninferenceonCPUwhenaGPUisneeded--thatwilltimeoutandscorezero.-Checkwith‘nvidia-smi‘ifyou’reunsurewhethertheenvironmenthasaGPU.-WhenloadingHuggingFacemodels,movethemtoCUDAexplicitly(‘model=AutoModelForCausalLM.from_pretrained(...).to(“cuda”)‘orpass‘device_map=“cuda”‘)andsendinputstothesamedevice.Thedefault‘.from_pretrained()‘leavesthemodelonCPU,whichisalmostneverwhatyouwantonaGPUtask.-Whendelegatingtothecodingagent,includeanyinformationaboutGPUresourcesandAPIkeyavailabilityintheprompt--theinneragentdoesnotinheritthisguidanceautomatically.##apply_patchformat‘‘‘***BeginPatch***AddFile:path/to/new_file.py+line1+line2***UpdateFile:path/to/existing.py@@context_line_to_locate_edit-old_line+new_line***DeleteFile:path/to/remove.py***EndPatch‘‘‘-Pathsarerelativetotheworkingdirectory.-‘***AddFile:‘writesthewholefileandoverwritesitifitalreadyexists--useitfornewfilesandforfullrewrites.‘***UpdateFile:‘isonlyfortargetededitslocatedby‘@@‘context.The‘+‘prefixon‘***AddFile:‘contentlinesisoptional.-‘@@‘linesprovidecontexttolocatetheeditposition.Includethenearestdistinctiveline(functionsignature,classdeclaration,etc.).-Linesprefixedwith‘‘(space)arecontext,‘-‘areremoved,‘+‘areadded.-Include3linesofcontextaboveandbeloweachchange.##Toolguidelines-Prefer‘rg‘(ripgrep)over‘grep‘forsearching.The‘grep_files‘tooluses‘rg‘internally.-Prefer‘read_file‘over‘shell‘with‘cat‘forreadingfiles.-Prefer‘list_dir‘over‘shell‘with‘ls‘fordirectorylistings.-Forcomplexmulti-stepshelloperations,chaincommandswith‘&&‘.##RuntimePathsYourworkingdirectoryis‘/home/agent/task‘.Allfilepathsmustberelativetothisdirectoryorabsolute.Writetheseexactfilesinyourworkingdirectory:-‘plot.png‘-‘writeup.md‘

G.2Claude and Codex system prompt

Claude Opus 4.8 and GPT-5.5 baselines are run using the built-in system prompts for Claude Code and Codex respectively, along with the following initial user prompt (whereprompt.mdrefers to the task prompt inSection G.3):

⬇Readprompt.mdtounderstandyourtask,thencompleteit.

When used in the CAT paradigm, Codex’s initial user prompt is decided by the agent calling it.

G.3Task prompt

⬇#PlotReplicationTask##ObjectiveYouhavebeengivenaresearchpaper(paper.pdf)fromwhichoneexperimentalplothasbeenremoved.Yourgoalistoreplicatethemissingplotbyreproducingtheexperimentsdescribedinthepaper.Thecaptionforthemissingplotisprovidedincaption.md.YourreplicatedplotMUSTbetheresultofrunningrealexperiments--seetherulesbelow.Youmustalsoproduceawrite-up(‘writeup.md‘)thatclearlydocumentsyourapproach,anyscalingorsimplificationchoicesyoumade,andwhatyourresultsshow.##Readingthepaper‘pymupdf‘ispre-installedforPDFtextextraction.Aquickwaytodumpthepapertotext:‘‘‘pythonimportpymupdfdoc=pymupdf.open(“paper.pdf”)text=“\n”.join(page.get_text()forpageindoc)‘‘‘Youdon’thavetousethis--extractthepaperhoweveryouprefer--butit’stheresoyoudon’tneedtospendturnsinstallingaPDFlibrary.##TimebudgetYouhave1hour(s)tocompletethistask.Checkhowmuchtimeisleftwith‘./check_time.sh‘.Youareencouragedtousethefullbudgetifyoulike--extratimespentrefiningyourplot,runningmoreseeds,orsanity-checkingyourimplementationmaywellimprovetheresult.Thatsaid,ifyouhavegenuinelyexhaustedproductiveideasandareconfidentyourplotisasgoodasitwillget,finishingearlyisfine;donotpadtherunwithbusywork.Eitherway,donotreturntotheusermid-tasktoaskforclarification--makereasonabledecisionsandkeepgoingautonomously.##AvailablecomputeYoumayhaveaccesstooneormoreGPUsonthismachine.TheGPU(s)maybesharingphysicalhardwarewithotherworkloadsviaNVIDIAMIG,butthatpartitioningisabstractedawayfromyou--foryourpurposesanyGPUsyouseeareyoursalone.Run‘nvidia-smi‘toconfirmwhat’savailablebeforeyouplan,andsizeyourexperimentstofit.OnaMIGslice,readthe“MIGdevices“sectionforyourslice’sactualmemory;avoid‘nvidia-smi--query-gpu=...‘sincedevice-levelfieldslike‘memory.total‘renderas‘[InsufficientPermissions]‘onMIGslices.Youalsohaveaccesstothefollowingcredentials,exposedasenvironmentvariablesinyourshell:-‘OPENAI_API_KEY‘--OpenAI-‘ANTHROPIC_API_KEY‘--Anthropic-‘GEMINI_API_KEY‘--GoogleGemini-‘HF_TOKEN‘--HuggingFaceTheLLMAPIkeys(‘OPENAI_API_KEY‘,‘ANTHROPIC_API_KEY‘,‘GEMINI_API_KEY‘)canbeusedfromyourcode(e.g.‘os.environ[“OPENAI_API_KEY”]‘)foranytaskstepthatneedsinferencefromafrontierLLM.‘HF_TOKEN‘letsyoudownloadgatedorprivatemodelsanddatasetsfromHuggingFace--‘huggingface_hub‘,‘transformers‘,and‘datasets‘pickitupautomaticallywhensetintheenvironment,somodelslikeLlamaorGemmawilljustworkwithoutanextraloginstep.Ifthepaperdescribesanexperimentthatcannotfitonthehardwareorinthetimeavailabletoyou(e.g.itrequiredhundredsofGPUstotrainafrontiermodelfromscratch,orasinglerunon1GPUwouldtakedays),dothemostfaithfulscaled-downversionyoucan:smallermodel,fewersteps,fewerseeds,etc.Beexplicitinyourwrite-upabouthowyouscaledtheexperimentandwhy.Thejudgehasbeentoldtorewardbest-effortwithinconstraintsoverfabrication,soaclearly-documentedscaled-downimplementationisbetterthanafake-but-prettyplot.IftheexperimentISfeasibleattheoriginalscalewithinyourhardwareandtimebudget,youshoulddothat,andyouwillbepenalizedforunnecessarilyscalingdownorsimplifying/modifyingexperiments.##Rules-YoumustNOTlookupordownloadtheoriginalpaperPDFfromtheinternet.-YoumustNOTuseanyexistingimplementationsofcodeyoufindonlinethatisspecificallyrelatedtoorderivedfromthispaper.Youareencouragedtouseotherresourcesyoufindonline(e.g.relevantMLframeworks,codefrompaperscitedinthispaper).-YoumustNOTsimplyguessorfabricatetheplot.TheplotMUSTbeaproductofanappropriatetrainingrun,analysis,and/orgraphingpipelinethatyouimplementedyourself(subjecttothescalingexceptionabove).-YoumustNOThard-codetheplotbydeducingthedatafromelsewhereinthepaper.Instead,youshouldreimplementandrunallexperimentsasbestyoucan.-YouMAYusetherestofthepapertocross-checkyourgeneratedplotandimproveit,solongasthatplothasbeengeneratedbyrealexperimentsyouhaveimplemented.-Youotherwisehavefreedomtousetheinternet,writeandexecutecode,installpackages,anduseanytoolsavailabletoyou.Dosoinanethicalandresponsibleway.-NEVERdeleteormodifythe‘/home/agent/task/logs‘directory.Logswillbewrittenhereautomaticallyforfutureinspectionbyhumans.-Abidebythespiritoftheserules.Youmustnotcheat.##Whatisagoodreplication?Agoodreplicationincludes:-Codethatfaithfullyimplementsthemethodologydescribedinthepaper,takingthecomputeandtimeavailabletoyouintoaccount.-Aprincipledandaccuratereproductionoftheremovedexperimentalplot,sofarasispossibleinthetime.Itismoreimportanttoreproducethescientificclaimandoveralltrendthanitistogetanexactmatchonspecificdatapoints.Ifamethodisthecorenoveltyofthepaper,itshouldbereimplementedfullyratherthanrelyingonlibrarycode.Ontheotherhand,youcanrelyonexistinglibrariesandframeworksforallaspectsthatarenotthecorecontributionofthepaper.##Output-Atthestartofthetask,initialiseagitrepositoryinyourworkingdirectory(‘gitinit‘).-Atthestart,commitallofthefilesprovidedtoyou(PDF,caption,andthisprompt)withadescriptivecommitmessage.-Saveyourreplicatedplotas‘plot.png‘inyourworkingdirectory.-Wheneveryousaveormodifytheplot,committheplotandyourcodewithadescriptivecommitmessage.-Youwillbeassessedonthewholecommithistory,notjustthefinalplot,somakesuretocommititerativelyasyouwork.-**Write-up--REQUIRED.Thisisgradedandeasytoforget,sodonotskipit.**Saveawrite-upnamed‘writeup.md‘inyourworkingdirectorycovering:(1)whatyouimplemented,how,andwhy;and(2)whatyouobserveinyourresults.Create‘writeup.md‘EARLY--assoonasyouhaveaplanandanyfirstresults--andkeepupdatingitasyougo,soitalwaysreflectsyourlatestworkevenifyourunoutoftime.DoNOTleaveittotheend.-Thejudgecanonlycreditworkitcansee:if‘plot.png‘or‘writeup.md‘ismissingwhenthetimeexpires,youwillbescoredasifthatpartoftheworkwasneverdone.

G.4Human prompt

⬇#Raterguide-whatwe’relookingfor**Pleasereadallofthefollowinginstructionsbeforeyoubegin.**Itisveryimportantthatyouhaveinternalisedtheratingmethodologyandcriteria.We’reevaluatinghowwellAIagentscanreplicateresultsfromresearchpapers,atestofwhethertheycandogenuinescientificworkonunderspecifiedproblems.Eachagentwashandedapaperwithonefigureremovedandtriedtoregeneratethatfigurebycodingarepositoryfromscratchandrunningexperiments.Youwillcompareseveralattemptsatthesamefigureandrankthem.Anexactmatchoftheoriginalfigure’sformatting,axes,orlayoutis**not**required:colors,axisranges,tickformattingetcmayalldifferfromtheoriginal.Focusontheexperimentalsetting,thescientificapproach,theimplementation,andtheoutcome.Agoodreplicationisaboutthescienceandnotjustthefinalplot.Thetranscriptandwrite-uparethemainevidencefortheprocessbehindit.##Whatyoumustlookat-**Theagents’instructions**(sectionbelow):theexactprompteveryagentreceived.Youarejudginghowwelleachagentperformedatitsassignedtask.-**ThematerialsPDF**(downloadbutton):thepaper,theoriginal(“gold”)figure+caption,thefigureandwriteupeachrolloutproduced.-**Thetrajectoryviewer**(buttonperrollout):atranscriptofeachagent’srun-thecommandsitran,andhowitworkedthroughtheproblem.-**TheGitHubrepo**(buttonperrollout):theagent’sworkspace-thecodeandoutputsitproduced.##HowtorankStartbyansweringthe“Summarizethetask“questions.Afterthat,dragtherolloutsintoorder,bestatthetop.For**eachrollout**,givearationaleforhowyouratedit-citesomethingspecificyousaw.Finally,describehowyoucametoyourdecisionoverall-yourprocessandwhatyouplacedweighton-thensubmit.Trustyourjudgment-there’snoanswerkeybeyondthegoldfigureandthepaper.##AnoteonbugsThisratersystemisstillinbetaandyoumayexperiencebugs.Ifsomethingfails,doahardrefreshofthepageinthefirstinstance,andifthatdoesn’tworkthenemail[redacted]forsupport.

G.5Optimised Codex prompt

⬇#Approach:ReplicatethetargetfigureYouhave60minutesandaGPU(anH200slice)toreproduceONEfigurefromapaper.Scoredon:(1)figureformat,(2)theplotreproducesthepaper’squalitativetrend,(3)youimplementthepaper’sACTUALmechanism,(4)effort/rigor,plusintegrityandwrite-upaccuracy.Integrity,disclosure,andfigurehonestyarelargelysolved;remaininglossesareonfidelity,trend/magnitude,misreadconstants,andstoppingearly/skippingtherealanchoronhardtasks.Dotherealexperiment,thensubmitafigureandwrite-upthathonestlymatchit.##1.Readthegoalfirst—restatethespecinwritingOpenthereferencefigureandexcerpt.Beforecoding,writedown(andputinwriteup.md):theexactpanels/axes/legend/units;theONEqualitativeclaim;theEXACTquantityeachaxisnames;theFULLlistofconditions/curves/model-arms/classes;andtheEXACTtasksetup(positiveclass,subset,split,metric,preciseequation/algorithm).Alsolisteveryload-bearingCONSTANT(sizes,counts,|H|,stepbudgets).Matchthevaluesshownin/aroundtheTARGETfigureitself(itscaption,axes,legend,targetspec)—whenaconstantdiffersbetweenthetargetfigureandthepaper’sbodytext(e.g.|H|=10^5vs10^6),followtheTARGETFIGURE.Misreadsareacommon,avoidableloss(class-4-vs-restdoneasclass-3-vs-4;achi-squareboundaryinsteadoftheadditiveone;omittingmodelarmsthefigureincludes).Beforesubmitting,DIFFyourfigureagainstthislist:everyarm/panelpresent,rightconstructionandconstants,noextra/missingpanels.##2.ImplementandTRAINtherealmechanism—neveraproxy,oracle,ortunedpriorFidelityisgradedonwhethertherealmechanismactuallyran—disclosureearnsNOcredit.-Buildtherealarchitecture/algorithmandTRAINitfromastandardinitwithareallearningrate.Whenitdoesn’tconverge,ITERATEonthetraining(LRsearch,betterinit/curriculum,moresteps,largerbatchonGPU)—neverpivottohand-engineeringtheanswer.-IfthemechanismisLLM/agentic,CHECKFEASIBILITY.Iftherealmodel/pipelinerunsin-budget,useit.Ifitgenuinelycannot(envwon’tinstall,orafullrealrunyieldsdegenerate/flatdataorwouldburnthewholehour),youMUSTSTILLruntheSMALLESTFAITHFULREALSLICE(onemodel×onecell,afewrealideas/samplesperarm)toanchortheresult,thenclearlylabeltherestasproxyandreporthonestproxyvalues.“Fullscaleisinfeasible“doesNOTlicenseskippingtherealanchor,fallingbacktoapurehand-codedsimulator,orstoppingearly.Onlyifevena1×1realcellistrulyimpossiblemayyougofullproxy—andthenlogthespecificreason.-NEVERfeedthemodel/toolthegoldanswer(an“oracle”),hand-tuneper-armpriors/profiles,addoff-paperobjectives/signals,or—foraproxy/infeasibletask—retunetheproxytoreproducethepaper’smagnitudeororderingafterseeingresults.Reportwhateverthehonestrungives.-Ifyoumustscaledown,PRESERVEthepropertythefiguretests(asyncparallelism,thelearnedcomponent,pixelinputs,enoughstepstoplateau)andtradeBREADTH(fewergames/points/examples/seeds);stillreproduceallpanels/sweeppoints.-Neverfit/selectparamsontheevalstream,runalr≈0pass,orpickadatawindow/split/seedafterseeingthetrend;usethepaper’sfull/standardsplit.Everyplottednumbercomesfrompost-trainingweightsandCOMPLETEDmeasurements,rendereddirectlyfromtheresultsCSV.Usetherealdataset/environment(verifybyimporting/installing),onGPU.##3.ReproducetheDIFFERENCE—directionANDmagnitude,honestlyThefigureshowsaqualitativedifference—onearmfails,aplateau/precisionfloor,aspecificordering—oftenataspecificMAGNITUDE.YourplotmustexhibitboththeorderingANDthemagnitude.Ifyourarmslookidenticalwhenthepaper’sdiffer,thesetupiswrong(toosmall/short,wrongregime,oraproxy/oracle/tunedpriormaskingit)—fixtheSETUP.Butifthemagnitudeisgenuinelyunreachableatyourscale,REPORTthehonestvalueandsayso—neverselectatraininghorizon,budget,stoppingpoint,orseedthatrecoversthepaper’sexpecteddirection.Anhonestmissscoresfarbetterthanamanufacturedmatch.##4.UsethefullbudgetontherightthingsThemost-ignoredrule:agentssubmitatminute12–20with40+minunused.DoNOTsubmitwithmorethan~5minofbudgetremaining,anddoNOTsubmitwithalargeidlemarginunlessyouhaveeitherruntheminimalrealanchororloggedwhyevena1×1realcellisimpossible.Aplausible-lookingfigureisNOTdone.Withtimeleft,doconcreteworkinpriorityorder:runtheminimalrealsliceyouskipped;completemissingarms/panels/sweeppoints;hardentheweakestcondition;addseedsuntilerrorbarstighten;addarealbaseline—NEVERretunetowardthetarget.At~30minchecktheeffectisappearingonGPU.Rigor:≥3seeds/runsAVERAGEDwithrealerrorbars,includingLLM/agenttasks.##5.IntegrityEverynumberfromarealrun;neverhardcode,fabricate,fittoreportedvalues,handicapabaseline,feedingoldanswers,hand-tunepriors,retuneaproxytothepaper’smagnitude,orselectseeds/windows/splits/training-horizons/stopping-pointsafterseeingresults.Fixhyperparametersbeforeseeingresults;anytuningappliessymmetricallytoallconditionsandisdisclosed.##6.Figurehonesty+write-upself-consistency(cheappointslosthere)-AxislabelsmustreflecttheACTUALquantitiesyouran—neverputpaper-scalelabels(e.g.“85Mparams”,thepaper’sstepcounts)onasmallerproxy.PlottheFULLrequiredwindow/x-rangeevenwheretheresultfails,drifts,orisunfavorable—croppingtoafavorablesub-window(e.g.asinglespike)isbothatrendlossandanintegrityviolation.-Self-consistencypassbeforesubmit:everyquantitativeclaiminwriteup.md—seedcount,epochcount,per-panelsample/imagecounts,datasetsizes,CI/t-multipliers,filenames/paths,andthefigure’sdatasource—mustbeREADOFFthefinalartifacts(code/CSV),notestimatedorcopiedfromthepaper;stateonlywhatyoucanverifyandomitorhedgetherest.Anumberthatcontradictstheartifacts(a“3seeds“claimovera10-seedrun,aper-panelcountthatdoesn’tmatch,astaleCSVpath,awrongrendersource)halvesthewrite-upscore.ThesameappliestoMETHOD/PROCEDUREdescriptions:describethemechanismexactlyasyourcodeimplementedit—especiallythestate/actionrepresentation,themodel/learnervariant,andtheCV-foldordatasetscope—andexplicitlyflaganydivergencefromthepaper’sprocedureratherthanrestatingthepaper’smethodasifyouhadreproducedit(describingthepaper’ssetupwhenthecoderanadifferentonealsohalvesthewrite-upscore).-Checklist:figurediffedagainstthe§1spec(allarms/panels,rightconstructionandconstants);axesuncropped,fullwindow;magnitudereproducedoritsabsencestated;≥3seedswitherrorbars;figurerenderedfromtheresultsCSV;contrastVISIBLE;plotandwrite-updescribetheSAMEresults.Includetwoattestationsinwriteup.md,bothfactualandminimal—writethemfromtheFINALrunonly,donotestimate:‘realslice:<rantherealmechanism/1×1realcell/impossiblebecause…>‘and‘budgetused:<N>/60min‘whereNistheelapsedtimereadverbatimfromtheenvironmenttimer(e.g.check_time.sh),notaguess.Abudget/seed/sourcefigurethatcontradictsthelogshalvesthewrite-upscore,sostateonlywhatyoucanreadofftheartifacts.DiscloseEVERYpost-hoc/proxychoice;neverclaimafigurewas“matched“whenpanels/magnitudearemissingorcallahand-builtcomponent“learned.“Deliverables:plot.png(matchingreferenceformat,allpanels/sweeppoints),runnablecode,resultsCSV(s),andwriteup.md.

G.6Example rubrics

Rubrics are generated per task, so each one is specific to the figure it grades. The three below are drawn uniformly at random.

How do language models learn facts?(164), Figure 5

⬇#Rubric:Howdolanguagemodelslearnfacts?—Figure5Thisfigurearguesthathallucinations—measuredasoverconfidenceinwrongattributepredictions—emergeduringpre-trainingatthesametimeknowledgeisacquired,andthatthishurtsthemodel’sabilitytointegratenewfactslater.Themiddleandrightpanelsthendemonstratetheconsequence:fine-tuningonnewindividualsrapidlydegradesperformanceonpre-trainingindividualswhilenewknowledgeislearnedonlyslowly,andmixinginreplayofpre-trainingdataonlypartiallymitigatesthis.Thisisaconceptualdemonstrationonthepaper’ssynthetic-biographysetup,sotherubricfocusesonwhetherthethree-panelcomparisonispresentandshowsthequalitativedynamics,notonquantitativematch.##1.VisualfidelityThepaperdescribesthreepanels.Theleftpanelshouldshow,overpre-trainingsteps,aknowledge-acquisitioncurveco-emergingwithahallucination/overconfidencesignal(overconfidenceoninaccuratepredictions).Themiddleandrightpanelsshouldshowtwocurveseachasfine-tuningprogresses—anattributelossonpre-trainingindividuals(risingrapidlyearly)andanattributelossonfine-tuningindividuals(decreasingmoreslowly),withgreydotsmarkingstart-of-fine-tuningperformance;therightpanelisthesamesetupbutwithreplayofpre-trainingdatamixedin.**0.0scoreexample:**anagentthatproducesasingle-panelplotunrelatedtothefine-tuning/hallucinationdynamics,orpanelsthatshowonlypre-trainingcurveswithnofine-tuningphase.**0.5scoreexample:**anagentthatproducesallthreepanelswithsensibleaxesbutomitsthegreystart-of-fine-tuningmarkers,mislabelswhichlossispre-trainingvsfine-tuning,orcollapsesleft-panelhallucinationintoasingleaccuracycurvewithnooverconfidence/miscalibrationsignal.**1.0scoreexample:**anagentthatproducesthreepanelsmatchingthecaption’sstructure(left:acquisitioncurvealongsideahallucination/overconfidencemetricduringpre-training;middle:pre-trainingandfine-tuningattributelossesacrossfine-tuningstepswithastarting-pointmarker;right:sameasmiddlebutwithreplay),withreasonableaxislabelsandlegendsevenifcolors,fonts,orexacttickplacementsdifferfromtheoriginal.##2.ClaimreproductionTheartifactmustshowthreeconnectedclaims:(i)hallucinations/overconfidenceemergeconcurrentlywithknowledgeduringpre-training;(ii)fine-tuningonnewindividualsproducesafastriseinpre-traininglossandaslowerfallinfine-tuningloss;(iii)replaypartiallyrescuesthefinalpre-traininglossbutdoesnotpreventtheinitialspike.Ifanyofthesetrendsfliporareabsent,theagentshouldnotethediscrepancyhonestly.**0.0scoreexample:**anagentwhosemiddle/rightpanelsshowpre-traininglossunchangedorimprovingduringfine-tuning,orwhosereplaypanelshowsnobenefitatall,anddoesnotacknowledgethatthiscontradictsthecaption’sclaim.**0.5scoreexample:**anagentthatreproducestheforgettingdynamicinthemiddlepanelbutthereplaypanellooksidenticaltotheno-replaypanel(nopartialmitigationvisible),ortheleftpanelshowsknowledgeacquisitionwithoutanysignalofhallucination/overconfidenceco-emerging—withlimitedacknowledgement.**1.0scoreexample:**anagentwhosefigureshowsavisibleco-emergenceofaccuracyandoverconfidenceontheleft,arapidpre-training-lossincreasepairedwithslowerfine-tuning-lossdecreaseinthemiddle,andarightpanelwherereplayclearlysoftensthefinalpre-training-losslevelwhiletheinitialjumpremains—oranagentthatgetsmostofthisandclearlyflagswhicheversub-claimdidn’treproduce.##3.ImplementationfidelityTheexperimentneedsthepaper’ssynthetic-biographysetup:asetofindividualseachwithseveralattributes,atransformertrainedtopredictattributes,anattribute-levellossmeasuredseparatelyonapre-trainingpopulationandaheld-outfine-tuningpopulationofnewindividuals,andafine-tuningphase(withandwithoutreplayofpre-trainingdata).Thehallucinationsignalneedstoreflectconfidenceoninaccuratepredictions,notjustaccuracy.Scalingdownmodelsize,numberofindividuals,orstepcountsisfinewhenitpreservesthesecomparisons.**0.0scoreexample:**anagentthatfine-tunesapretrainedpublicLLMonunrelatedtext,ormeasurestoken-levelcross-entropyongenericwebdataratherthanattributelossesondistinctpre-trainingvsfine-tuningindividualpopulations.**0.5scoreexample:**anagentthatimplementsthebiographiestaskandthefine-tuningsplitcorrectlybutconflatesthetwoevaluationpopulationsintooneloss,usesaccuracyasastand-inforhallucinationwithoutanycalibration/confidencesignal,orimplements“replay“assimplycontinuingpre-trainingratherthanmixingpre-trainingandnew-individualdataduringfine-tuning.**1.0scoreexample:**anagentthattrainsasmalltransformeronasyntheticbiographiesdataset,splitsindividualsintopre-trainingandfine-tuningcohorts,tracksattributelossseparatelyoneach,runsbothaplainfine-tuningandafine-tuning-with-replaycondition,andderivestheleftpanel’shallucinationsignalfrompredictionconfidenceonincorrectanswers—evenatreducedscaleorwithasimplerattributeschemathanthepaper’s.##4.ExperimentaleffortEffortisjudgedbywhethertheagentiteratedtowardaworkingthree-panelcomparison,notbywall-clockconsumed.Signsofengagementincludere-runningfine-tuningafternoticingmissingdynamics,tuninglearningrateorstepcounttomakethefast-drop/slow-risepatternvisible,andsettingupthereplayconditionasanactualsecondrunratherthanamock.**0.0scoreexample:**anagentthatstopsafterasinglefailedpre-trainingrun,submitsplaceholderplots,orfabricatesthecurveswithouttrainingamodel.**1.0scoreexample:**anagentthatgetspre-trainingworking,noticesthefine-tuningpanelisn’tshowingtheexpectedforgettingcurve,adjusts(e.g.,increasesfine-tuninglearningrate,extendssteps,orfixestheevalsplit),thenrunsthereplayvariantasaseparateexperiment—evenifthefinalscaleissmallerthanthepaper’sandonlyoneseedisused.

Additive logistic regression: a statistical view of boosting(29), Figure 5

⬇#Rubric:AdditiveLogisticRegression(Friedman,Hastie&Tibshirani)—Figure5Thenested-sphereexampleinSection6usestenindependentstandard-normalinputswiththeclasslabeldeterminedbywhether||x||²exceedsthemedianofχ²₁₀—sothetruelog-oddsdependonlyonthesumofsquaredcoordinatesandtheproblemisexactlyadditiveinx_j².Figure5visualizesthecoordinatefunctionsf_j(x_j)oftheadditivelogisticmodelfitbyLogitBoostwithstumps:theclaimisthatboostedstumpsrecoverthisadditivestructure,witheachf_jasmoothsymmetricfunction(roughlyquadratic,increasingin|x_j|)andthetencoordinatepanelsessentiallyinterchangeable,demonstratingthatLogitBoost-on-stumpsisfittingagenuineadditivelogisticmodelratherthansomethingopaque.##1.VisualfidelityThefigureshouldpresenttencoordinate-functionpanels(oneperinputdimensionofthenested-sphereproblem),eachplottingthefittedf_jasafunctionofx_joverthesupportofastandardnormal.Becausethedata-generatingmechanismdependsonlyon||x||²,thepanelsshouldlookliketennear-identicalsymmetriccurvesrisingonbothtails—noterrorcurves,notdecisionboundaries,notasingle2-Dplot.**0.0scoreexample:**anagentthatproducesatest-error-vs-iterationscurve,a2-Ddecision-boundaryplot,orasingle-panelscatter—i.e.,somethingthatisnotagridofper-coordinatefunctionplotsatall.**0.5scoreexample:**anagentthatproducesper-coordinatefunctionplotsbutwiththewrongnumberofdimensions(e.g.,2or5panelsinsteadof10),orpanelsthatplotsomethingotherthanf_j(x_j)suchasvariableimportancebars.**1.0scoreexample:**anagentthatproducestensmallpanelslabeledbycoordinate,eachshowingf_j(x_j)overroughlytherangeofastandardnormal,withaxislabelsidentifyingthecoordinateandthefittedfunctionvalue;cosmeticdifferences(gridarrangement,linecolor,panelsize)donotcostpoints.##2.ClaimreproductionThefigureexiststoshowthatLogitBoost-with-stumpsrecoverstheunderlyingadditivestructureofthenested-sphereproblem:eachf_jshouldbeasmooth,symmetric,roughlyU-shaped(orinverted-U,signdependingonclasscoding)functionofx_j,andthetenpanelsshouldlookessentiallythesameuptonoise.Afaithfulartifactmakesthisvisibleataglance;ifresultsdiverge(e.g.,asymmetricornon-quadraticcurves),theagentshouldflagit.**0.0scoreexample:**anagentwhosecoordinatefunctionsareflat,monotone,orwildlydifferentacrossthetendimensions,withnoacknowledgmentthatthiscontradictsthesymmetryimpliedbythedata-generatingprocess.**0.5scoreexample:**anagentthatproducescurveswhichareroughlysymmetricinsomepanelsbutnoisy/monotoneinothers,orthatusestoofewboostingiterationssothequadraticshapeisonlyfaintlyvisible—theadditive-recoveryclaimispartiallysupportedbutnotconvincing.**1.0scoreexample:**anagentwhosetenpanelseachshowaclearsymmetricU-shape(orinverted-U)inx_j,visuallysimilaracrosscoordinates,makingtheadditive-quadraticstructureofthefittedlog-oddsimmediatelyapparent—oronethathonestlynotesanyresidualasymmetrywhilestillshowingthedominantsymmetricshape.##3.ImplementationfidelityThecomparisonrequires(a)thenested-spheregenerativemodelfromSection6—teni.i.d.standard-normalcoordinateswithclasslabelthresholdedon||x||²attheχ²₁₀median—and(b)LogitBoostwithdepth-1trees(stumps)runlongenoughfortheadditivecoordinatefunctionstostabilize.Thecoordinatefunctionsf_jareextractedbyaggregating,foreachinputdimension,thecontributionsofallstumpsthatsplitonthatdimension.Ascikit-learnorhand-rolledLogitBoostisequallyvalidprovideditimplementstheNewton-styleweighted-least-squaresupdateonworkingresponsesdescribedinthepaper;gradientboostingwithlogisticlossonstumpsisanacceptableclosesubstituteiftiedtothepaper’salgorithm.**0.0scoreexample:**anagentthatfitsacompletelydifferentmodel(e.g.,asingledecisiontree,aneuralnet,orAdaBoostwithdeeptrees)onadifferentdataset,sotheartifactdoesnottestLogitBoost-on-stumpsappliedtonestedspheres.**0.5scoreexample:**anagentthatusesthecorrectdata-generatingprocessbutboostswithmulti-splittreesratherthanstumps(breakingtheadditivedecomposition),orusesstumpsbutextractsamarginalplotoff̂(x)sweepingonecoordinatewithothersfixedatzeroratherthansummingstumpcontributionspercoordinate—thecomparisonispresentbutthecoordinate-functioninterpretationismuddled.**1.0scoreexample:**anagentthatsimulatestheSection-6nestedspheres(10-Dstandardnormals,thresholdatχ²₁₀median,~2000trainingpoints),fitsLogitBoost-with-stumps(theirownimplementationoradefensiblelibraryequivalent),andconstructseachf_jfromtheper-dimensionstumpcontributions;abudget-drivenreductioniniterationcountortrainingsizeisfineaslongasthecoordinate-functionshapeisstable.##4.ExperimentaleffortEfforthereisaboutengagingwiththeadditive-recoverycomparison:gettingLogitBoostonstumpsrunningonthenested-spheredata,iteratingifthecoordinatefunctionslooknoisyorasymmetric,andproducingteninterpretablepanels.Notrunningthousandsofiterationsorpolishingcosmeticsisnotafailure;abandoningtherunafterseeingflatorbrokencurvesis.**0.0scoreexample:**anagentthatproducesnofigureatall,orcommitsanobviouslybrokenfirstattempt(e.g.,emptypanels,single-classoutput)withoutanydiagnosticorrerun.**1.0scoreexample:**anagentthatfitsthemodel,inspectsthecoordinatefunctions,andrerunswithmoreiterationsorafixwhenearlycurvesaretoonoisytoshowthequadraticshape—oronethatgetsacleanresultonthefirsttry,validatesthesymmetryacrosspanels,andstopswithoutburningtherestofthebudgetoncosmetics.

Learning precise timing with LSTM recurrent networks(32), Figure 4

⬇#Rubric:LearningPreciseTimingwithLSTMRecurrentNetworks—Figure4ThisfigureaskshowthetrainingcostoftheNMSD(aMeasuring-Spike-DistancetaskvariantwithdelaysetI(n)∈{0,1})scaleswiththeminimumspikeintervalF,and—cruciallyforthepaper’scentralclaim—whetheraugmentingLSTMwithpeepholeconnectionsfromtheCECtothemultiplicativegatesletsthenetworklearnprecisetimingmoreefficientlythantraditional(forget-gate)LSTM.ShowingpeepholeLSTMtrainedwithsubstantiallyfewerstreamsthantraditionalLSTM,particularlyatlargerFwherepreciseintervalmeasurementisharder,istheempiricalevidencethepaperusestomotivatepeepholes.Thisisa**specificempiricalcomparison**:theclaimdependsonpeepholevs.traditionalLSTMbothbeingimplementedonthesameNMSDtask.##1.VisualfidelityThefigureisaline/curveplotwhosex-axisistheminimumspikeintervalFandwhosey-axisistheaveragenumberoftrainingstreamsrequiredtosolvetheNMSDtaskwithdelaysI(n)∈{0,1}.Becausethepaper’sSection4framesitsexperimentsasahead-to-headbetweenpeepholeLSTMandtraditionalLSTM,andcompanionfiguresinthesamesection(e.g.,Figure7forGTS)followthissameconvention,thefigureisexpectedtoshowseparatecurvesforpeepholeLSTMandtraditionalLSTM(ortoclearlyindicatewhereonevariantfailedtosolvethetask).They-axisisplausiblylogarithmicgiventherangeoftraining-streamcountsseeninrelatedtasks.**0.0scoreexample:**anagentthatproducesaplotwhoseaxesareunrelatedto“trainingstreams“vs“minimumspikeintervalF“(e.g.,losscurvesoverepochs,oraccuracybars),orthatomitsanycomparisonbetweenLSTMvariantsandshowsasingleunlabeledline.**0.5scoreexample:**anagentthatproducesatraining-streams-vs-Fplotwiththecorrectaxislabelsbutonlyonecurve(e.g.,peepholeLSTMalone)andnorepresentationofthetraditional-LSTMbaselinethepapercomparesagainst.**1.0scoreexample:**anagentthatproducesacurveplotwithFonthex-axis,averagetrainingstreamsonthey-axis(linearorlog),labeledcurvesforpeepholeLSTMandtraditionalLSTMontheNMSDI(n)∈{0,1}task,withreasonabletickcoverageoftheFrange—evenifcosmeticdetails(markershapes,colors,exactFsweepvalues)differfromthepaper.##2.ClaimreproductionThefigure’sscientificcontentisthatpeepholeLSTMsolvesNMSDwithfewertrainingstreamsthantraditionalLSTM,andthatthisadvantagegrows(orremainsdecisive)atlargerminimumspikeintervalsFwhereprecisetimingmattersmost.Therollout’sartifactshouldshowthisseparation;ifresultsdivergefromthispredictiontheagentshouldnoteithonestlyratherthanrecoloranullresultasasuccess.**0.0scoreexample:**anagentthatfabricatescurves(e.g.,copiesnumberswithoutrunningtheexperiment),orwhosefigureshowstraditionalLSTMmatchingorbeatingpeepholeLSTMacrossFwithnoacknowledgementthatthiscontradictsthepaper.**0.5scoreexample:**anagentwhosefigureshowsapeephole-vs-traditionalseparationonlyatasingleF(e.g.,F=10)withnoscalingtrendacrossF,orwhosecurvesaretoonoisy/single-seedtodistinguishthevariants,butwhoflagsthelimitationhonestly.**1.0scoreexample:**anagentwhosefigureshowspeepholeLSTMrequiringsubstantiallyfewertrainingstreamsthantraditionalLSTMacrossmultiplevaluesofF,withthegapvisible(orwithtraditionalLSTMfailingtosolveatlargerF),reproducingthepaper’squalitativeclaim—or,ifresultsdivergedduetoscale-down,anartifactthatstillshowsthecomparisonalongsideaclearlystatedcaveat.##3.ImplementationfidelityTheexperimentmustimplementtheNMSDtaskasdescribedinthepaper(acontinualstreamofspikeswhoseinter-spikeintervalsencodethequantitythenetworkmustoutput,withdelaysI(n)∈{0,1})andtrainbothLSTMvariantsonit:traditionalLSTMwithforgetgates,andpeepholeLSTMthataddsweightedconnectionsfromtheCEC’scellstatetotheinput,forget,andoutputgates.Counting“trainingstreamsrequired“presumesastoppingcriteriontiedtosolvingthetask(thepaper’sconvention).Reasonablescale-downs(fewerseeds,narrowerFsweep,smallerblockcounts)arefinewhentheypreservethehead-to-headcomparison;substitutingafundamentallydifferenttaskorskippingoneofthetwoLSTMvariantsisnot.**0.0scoreexample:**anagentthattrainsasingleLSTMvariant(nopeepholevs.traditionalcontrast),orthatusesanoff-the-shelftask(e.g.,MNIST,copytask,sine-waveregression)insteadoftheNMSDspike-intervaltask.**0.5scoreexample:**anagentthatimplementsNMSDandbothLSTMvariantsbutconflatesthearchitecturaldistinctioninaload-bearingway—e.g.,usesstandardPyTorch‘nn.LSTM‘forbothvariantsandonlyvariesahyperparameter,oromitspeepholeconnectionstooneofthethreegates—sothe“peepholeeffect“beingmeasuredisnotthepaper’s.**1.0scoreexample:**anagentthatimplementstheNMSDtaskwithI(n)∈{0,1},buildspeepholeLSTMwithcell-to-gateconnectionsperSection3andaforget-gate-onlyLSTMbaseline,sweepsseveralFvalues(evenareducedsetof3–5),andcountstrainingstreamstoasensiblesolutioncriterion—possiblywithfewerseedsthanthepaperused.##4.ExperimentaleffortDidtheagentuseavailabletimetoactuallyrunthepeephole-vs-traditionalcomparisonacrossmultipleFvalues,andrespondwheninitialresultslookeddegenerate(e.g.,neithervariantsolving,orabsurdlyfastsolutionssuggestingaleakytask)?Effortisjudgedfromobservableengagement:scalinguptheFsweepwhenanearlysmallrunworked,fixingabuginthepeepholegradientortaskgeneratorandrerunning,orseedingmultiplerunstomakethecurvesmeaningful.**0.0scoreexample:**anagentthatfinishesearlyonanobviouslybrokenartifact(e.g.,flat-zerocurves,NaNlosses,single-point“curves“)withnoattempttodebugorrerun.**1.0scoreexample:**anagentthat,afteraninitialrunshowedonlyoneForonevariantworking,fixedtheissue(e.g.,correctingpeephole-gradienttermsorthespike-intervaltargetgeneration)andre-ranacrossabroaderFsweepwithmultipleseedstopopulatebothcurves—evenifthefinalbudgetdidnotmatchthepaper’sscale.

Appendix HInfrastructure

Cluster.

All experiments run on Kubernetes clusters of Hopper and Blackwell GPUs. Within a cluster, Kueue(138)manages the pool: whole GPUs, MIG slices, and CPU-only nodes carry separate resource quotas, and workload priority classes order admission between training and evaluation. Each training run launches as a Ray job(90)that brings up its own RayCluster spanning trainer and generation nodes. We maintain our own launch utility that turns a declarative experiment specification into Kubernetes workload definitions, submits the run, and tracks logs and metadata.

Training stack.

Our RL framework is a fork of NeMo-RL(96). Rollout collection and optimisation overlap, and run on separate node pools of inference and training workers respectively, with Ray handling orchestration and communication. The trainer uses the Megatron-Core backend(125)with tensor and context parallelism over the full128128K-token context; inference is served by vLLM(71), with policy weights refit in place as optimiser steps land, so in-flight rollouts continue on the newer weights. We implement support for sequence packing and context parallelism for Qwen3.6’s hybrid gated-delta-net layers(154). Rollouts flow through our fork of NeMo-Gym(97), which decomposes rollout collection into model, agent, and resources HTTP services.

Task containers.

We build our own NeMo-Gym resource server, which acts as a container service for lifecycle management and orchestration. Every rollout receives a fresh container pod from a common image – a research workstation with CUDA, Python, a standard ML stack, and a coding-agent CLI – so the environment in which the agent operates is as close as possible to the machine a human researcher would use. The resource server owns the pod lifecycle: it stages the workspace, enforces the task’s wall-clock deadline, routes the harness’ tool calls into the pod (Section 3.4), and reaps expired sessions. At evaluation time it runs the judge inside the container.