Overview of the NLPCC 2026 Shared Task 11: Agent-Based Experiment Reproduction from Scientific Papers

# Overview of the NLPCC 2026 Shared Task 11: Agent-Based Experiment Reproduction from Scientific Papers
Source: [https://arxiv.org/html/2609.11117](https://arxiv.org/html/2609.11117)
Yizhi LiAffiliation:The University of Manchester, IQuest ResearchLuu Gia HuyJian YangMing ZhouAffiliation:Vietnam National University Beihang University, LangboatChenghua Lin

###### Abstract

Reproducibility is essential to scientific progress, yet the growing volume and complexity of scientific publications make exhaustive manual verification increasingly impractical\. Although recent advances in large language model \(LLM\) agents enable automated experiment reproduction, existing evaluations largely focus on final repositories and are typically limited to machine learning \(ML\)\. We introduceAgentActionBench, a process\-oriented benchmark for evaluating agent\-based experiment reproduction across ML and AI4Science domains\. Our framework uses an MCP\-based Action Recorder to capture agents’ behaviour throughout the reproduction process and evaluates the resulting traces with paper\-specific rubrics\.AgentActionBenchcontains 150 papers, including 120 ML papers and 30 AI4Science papers\. A human\-annotated subset covering 10% of the benchmark provides validation data, while model\-assisted augmentation expands the full benchmark to more than 10,000 rubric items\. Experimental results show that current systems remain limited, with execution as the primary bottleneck\. Meanwhile, the strong Pearson and Spearman correlations between model\-generated and human\-annotated rubrics validate the reliability of our scalable rubric\-generation approach\.

###### Keywords:

LLM AgentsRubric\-Based EvaluationData Augmentation

## 1Introduction

Reproducibility is essential to trustworthy scientific progress, yet it remains difficult to achieve in practice\. ANaturesurvey found that more than 70% of researchers had failed to reproduce another scientist’s experiments, while recent studies indicate that only around 20% of papers at top computer science conferences release code sufficient for reproduction\[[6](https://arxiv.org/html/2609.11117#bib.bib25),[16](https://arxiv.org/html/2609.11117#bib.bib13),[20](https://arxiv.org/html/2609.11117#bib.bib9)\]\. As scientific publications grow in both volume and complexity, manual verification at scale is becoming increasingly impractical\.

Recent advances in large language models \(LLMs\) have substantially improved long\-context understanding, code generation, reasoning, and tool use\. Consequently, LLM\-based agents are increasingly applied to complex scientific workflows, from machine learning experimentation to AI for Science \(AI4Science\) tasks in domains such as biology and chemistry\[[1](https://arxiv.org/html/2609.11117#bib.bib23),[7](https://arxiv.org/html/2609.11117#bib.bib24),[13](https://arxiv.org/html/2609.11117#bib.bib15),[19](https://arxiv.org/html/2609.11117#bib.bib17),[36](https://arxiv.org/html/2609.11117#bib.bib14),[38](https://arxiv.org/html/2609.11117#bib.bib3)\]\. These capabilities make them promising candidates for automated experiment reproduction, which requires paper comprehension, implementation planning, code generation, experiment execution, and result verification\.

![Refer to caption](https://arxiv.org/html/2609.11117v1/figures/previous.png)Figure 1:Illustration of output\-oriented evaluation in prior benchmarks\. When descriptions and implementations are evaluated together, LLM\-based evaluators may incorrectly infer implementations that were never produced or executed\.Recent benchmarks, including Paper2Code\[[25](https://arxiv.org/html/2609.11117#bib.bib2)\]and PaperBench\[[28](https://arxiv.org/html/2609.11117#bib.bib1)\], evaluate whether AI systems can reproduce research papers\. However, most of them focus on machine learning and assess final artefacts rather than the reproduction process\. Such output\-oriented evaluation can be unreliable since an agent may generate plausible files or descriptions while overlooking important paper content, failing to execute its code, or fabricating execution records\. As illustrated in Fig\.[1](https://arxiv.org/html/2609.11117#S1.F1), this ambiguity makes it difficult to determine whether a reproduction is grounded in the source paper and supported by concrete actions\.

![Refer to caption](https://arxiv.org/html/2609.11117v1/figures/ours.png)Figure 2:Overview of our framework\. A standardised execution environment records agent actions, which are matched against paper\-specific rubrics for auditable, process\-oriented evaluation\.To address these limitations, we introduce a process\-oriented benchmark for agent\-based experiment reproduction\. Rather than evaluating only the final repository, our framework decomposes reproduction into fine\-grained stages and verifies whether agents perform the required operations\. An MCP\-based Action Recorder captures reading, writing, and command\-execution actions within a standardised environment\. The resulting action logs provide auditable evidence for rubric\-based evaluation, allowing each stage of the reproduction workflow to be assessed directly\. Fig\.[2](https://arxiv.org/html/2609.11117#S1.F2)presents an overview of the framework\.

We further extend evaluation beyond conventional machine learning papers to AI4Science research in astronomy, biology, chemistry, environmental science, materials science, and medicine\. We first constructAgentActionBench\-Human, a manually annotated subset containing 12 ML and 3 AI4Science papers\. To reduce the cost of expert annotation, we develop an agent\-assisted rubric\-generation pipeline, validate it against the human annotations inAgentActionBench\-Human, and scale the benchmark to the fullAgentActionBench, comprising 120 ML and 30 AI4Science papers with more than 10,000 rubric items\.

In summary, our contributions are threefold:

- •We introduceAgentActionBench, a process\-oriented, multi\-domain benchmark that evaluates the complete experiment\-reproduction workflow across 150 ML and AI4Science papers\.
- •We develop an MCP\-based Action Recorder and a rubric\-grounded evaluation protocol that use auditable action traces to assess paper understanding, implementation, execution, and result verification\.
- •We evaluate systems submitted to the NLPCC 2026 shared task, revealing key limitations of current agents in experiment reproduction and validating the reliability of our scalable rubric\-generation pipeline\.

## 2Related Work

### 2\.1Agent\-Based Experiment Reproduction

Autonomous scientific agents are increasingly studied for their ability to reproduce published experiments, an important capability for improving scientific efficiency and reproducibility\. Existing systems address different stages of this process\. The AI Scientist\[[18](https://arxiv.org/html/2609.11117#bib.bib11)\]supports an end\-to\-end workflow from idea generation to paper writing; DOLPHIN\[[36](https://arxiv.org/html/2609.11117#bib.bib14)\]adopts a closed\-loop process of ideation, implementation, and feedback; PaperCoder\[[25](https://arxiv.org/html/2609.11117#bib.bib2)\]converts research papers into executable repositories through a three\-stage pipeline; and HiRAS\[[11](https://arxiv.org/html/2609.11117#bib.bib16)\]coordinates specialised sub\-agents across fine\-grained reproduction stages\. A growing body of benchmarks evaluates these capabilities, including end\-to\-end ML reproduction\[[15](https://arxiv.org/html/2609.11117#bib.bib27),[26](https://arxiv.org/html/2609.11117#bib.bib26),[28](https://arxiv.org/html/2609.11117#bib.bib1)\], NLP\-focused implementation from algorithmic descriptions\[[32](https://arxiv.org/html/2609.11117#bib.bib28),[35](https://arxiv.org/html/2609.11117#bib.bib29)\], and general ML\-engineering tasks based on large\-scale experiments and public competitions\[[8](https://arxiv.org/html/2609.11117#bib.bib30),[14](https://arxiv.org/html/2609.11117#bib.bib31)\]Their results consistently show that current agents remain far behind human experts, particularly when tasks require robust implementation and execution\. However, most existing evaluations focus on coarse outcome\-level metrics, provide limited visibility into the intermediate reproduction process, and primarily target a single domain\. In contrast, our benchmark evaluates the complete workflow from paper understanding to result verification through auditable action traces, enabling fine\-grained assessment across both ML and AI4Science papers\.

### 2\.2Rubric\-Based Evaluation

Rubrics provide structured supervision by decomposing complex tasks into fine\-grained evaluation criteria, addressing the limitations of coarse, outcome\-based metrics\[[10](https://arxiv.org/html/2609.11117#bib.bib20)\]\. Benchmarks such as PaperBench\[[28](https://arxiv.org/html/2609.11117#bib.bib1)\]and HealthBench\[[5](https://arxiv.org/html/2609.11117#bib.bib18)\]use human\-authored rubrics for stepwise evaluation, but their reliance on domain experts makes them costly to construct and difficult to scale\. This limitation has motivated model\-generated rubrics\[[24](https://arxiv.org/html/2609.11117#bib.bib32),[17](https://arxiv.org/html/2609.11117#bib.bib21),[37](https://arxiv.org/html/2609.11117#bib.bib19)\]\. For example, RubricHub\[[24](https://arxiv.org/html/2609.11117#bib.bib32)\]combines principle\-guided synthesis, multi\-model aggregation, and difficulty evolution to produce discriminative criteria, while prior work shows that question\-specific rubrics improve the accuracy and consistency of logical evaluation\[[23](https://arxiv.org/html/2609.11117#bib.bib33)\]\. Building on these studies, we generate rubrics specifically for research\-paper reproduction\. Unlike methods designed for general reward modelling or open\-ended question answering, our approach targets the multi\-stage, repository\-grounded nature of experiment reproduction\.

### 2\.3LLM\-Based Data Augmentation

Large language models have increasingly been used to generate training and benchmark data themselves, driven by the high cost of manual annotation and the need to scale supervision alongside larger models\[[29](https://arxiv.org/html/2609.11117#bib.bib37)\]\.\[[29](https://arxiv.org/html/2609.11117#bib.bib37)\]survey this shift, organizing LLM\-based augmentation methods into simple, prompt\-based, and retrieval\-based categories\. One prominent line of work focuses on scaling instruction diversity to align models with human intent, from bootstrapping new instruction\-response pairs from a small seed set\[[31](https://arxiv.org/html/2609.11117#bib.bib34)\]to progressively rewriting instructions into more complex variants\[[34](https://arxiv.org/html/2609.11117#bib.bib35)\]\.\[[9](https://arxiv.org/html/2609.11117#bib.bib36)\]apply the same idea to a specific domain, prompting an LLM to expand existing geometry problem sets \(GeoQA\+, Geometry3K\) into the larger Geo170K dataset\. Together, these works show that LLM\-driven generation can substitute for costly manual curation when scaling supervision or data diversity\. We apply this same principle to automatically synthesise fine\-grained, paper\-specific rubrics for experiment reproduction, using an agent framework in place of the single\-pass prompting adopted in prior augmentation work\.

Table 1:Tools provided by the Action Recorder\. Optional arguments are enclosed in brackets\.

## 3Methodology

### 3\.1Action Recorder

To faithfully capture agent behaviour during experiment reproduction, we implement an Action Recorder based on the Model Context Protocol \(MCP\)\. Rather than granting agents direct access to the underlying operating system, the recorder mediates all interactions with the execution environment through a small set of MCP tools\. Each operation is therefore captured automatically as a structured action log, providing a complete and reproducible trace of the reproduction process\.

As summarised in Table[1](https://arxiv.org/html/2609.11117#S2.T1), the Action Recorder provides three core tools:Read,Write, andExecute\. TheReadtool retrieves file contents and optionally supports line\-range selection for efficient inspection of large files\. TheWritetool creates or modifies source code, configuration files, and other artefacts required for reproduction\. TheExecutetool runs shell commands and returns the execution status, standard output, and standard error, enabling agents to conduct experiments and inspect their results\. For safety and fairness, all tool interactions are restricted to a designated workspace\.

The Action Recorder treats each tool invocation as an atomic, standardised action\. For each invocation, it records the input arguments, returned results, and timestamp in a chronological JSON log, as illustrated in Appendix[0\.A](https://arxiv.org/html/2609.11117#Pt0.A1)\. The log is automatically exported when the agent terminates\. Because all interactions are mediated through the MCP interface, the recorder captures agent behaviour without modifying the agent and remains compatible with any MCP\-enabled system\. These structured logs provide the primary evidence for rubric\-based evaluation, enabling fine\-grained assessment of the reproduction process rather than relying solely on the final repository\.

### 3\.2Benchmark Construction

We collect 150 papers to construct the fullAgentActionBenchbenchmark\. Of these, 120 are drawn from top\-tier machine learning conferences, including ICML, ICLR, ACL, and NeurIPS, while the remaining 30 are AI4Science papers published in leading scientific journals, such asNature,Bioinformatics, andnpj Computational Materials\.

We adopt a rubric\-based evaluation protocol in which each paper is paired with a paper\-specific checklist for assessing reproduction quality\. Formally, each rubric item is represented asri=\(ci,si,ti\)r\_\{i\}=\(c\_\{i\},s\_\{i\},t\_\{i\}\), wherecic\_\{i\}denotes the evaluation criterion,sis\_\{i\}its importance score, andtit\_\{i\}the corresponding stage of the reproduction process\. We define five rubric types:Paper Observation,Plan Writing,Code Implementation,Command Execution, andResult Matching\. Together, these categories cover the full reproduction workflow and enable a more comprehensive assessment than approaches that consider only the consistency between a generated repository and the source paper\.

We construct the human\-annotated subset,AgentActionBench\-Human, by randomly sampling 12 ML papers and 3 AI4Science papers, representing 10% of the full benchmark\. Three research students with extensive experience in publishing at and reviewing for top\-tier conferences annotate the rubrics\. After the initial annotation, each rubric set is independently reviewed by the other two annotators in rotation and revised as needed\. This process continues until all annotators reach consensus\. The resulting subset contains an average of 67\.75 rubric items per paper and more than 1,000 manually annotated items in total\.

Building on recent advances in LLM\-based data augmentation, we scale the benchmark by automatically generating rubrics for all papers\. We first convert the papers to Markdown usingMinerU\[[30](https://arxiv.org/html/2609.11117#bib.bib38)\], providing LLMs with cleaner and more structured inputs\. We then apply the rubric\-generation framework introduced in prior work\[[12](https://arxiv.org/html/2609.11117#bib.bib39)\]to produce paper\-specific rubrics, expandingAgentActionBench\-Humaninto the fullAgentActionBenchbenchmark\. The resulting benchmark contains more than 10,000 rubric items, with an average of 69\.92 items per paper\. Finally, we assess the alignment between model\-generated and human\-annotated rubrics onAgentActionBench\-Humanusing correlation analysis\. Detailed results are presented in §[5\.2](https://arxiv.org/html/2609.11117#S5.SS2)\.

Table 2:Mapping between rubric types and the Action Recorder tools whose logs provide the corresponding evaluation evidence\.
### 3\.3Evaluation

Our evaluation framework primarily relies on the action logLLexported by the Action Recorder described in §[3\.1](https://arxiv.org/html/2609.11117#S3.SS1)\. For each paper𝒫\\mathcal\{P\}, the evaluator processes every rubric itemri=\(ci,si,ti\)r\_\{i\}=\(c\_\{i\},s\_\{i\},t\_\{i\}\)\. To reduce context length and exclude irrelevant evidence, the evaluator modelℳ\\mathcal\{M\}first extracts a subset of logsLi′⊆LL\_\{i\}^\{\\prime\}\\subseteq Lassociated with the tools relevant to rubric typetit\_\{i\}\. Table[2](https://arxiv.org/html/2609.11117#S3.T2)specifies the mapping between each rubric type and its corresponding tools\.

Given the extracted logsLi′L\_\{i\}^\{\\prime\}, rubric criterioncic\_\{i\}, and source paper𝒫\\mathcal\{P\}, the evaluator assigns a binary judgmentJi∈\{Pass,Fail\}J\_\{i\}\\in\\\{\\mathrm\{Pass\},\\mathrm\{Fail\}\\\}asJi=ℳ⁡\(Li′,ci,𝒫\)J\_\{i\}=\\mathcal\{M\}\(L\_\{i\}^\{\\prime\},c\_\{i\},\\mathcal\{P\}\)\. The overall reproduction score is then computed as the importance\-weighted proportion of passed rubric items:

score=∑i\[Ji=Pass\]si∑isi,\\mathrm\{score\}=\\frac\{\\sum\_\{i\}\\left\[J\_\{i\}=\\mathrm\{Pass\}\\right\]s\_\{i\}\}\{\\sum\_\{i\}s\_\{i\}\},\(1\)where\[⋅\]\[\\cdot\]equals11when the enclosed condition is satisfied and00otherwise\.

Table 3:Leaderboard results for the shared task\. Our reproduction baseline is underscored\.

## 4Experimental Setup

Models\.For rubric generation, we follow the configuration adopted in prior work\[[12](https://arxiv.org/html/2609.11117#bib.bib39)\]\. Specifically, we useClaude Code\[[2](https://arxiv.org/html/2609.11117#bib.bib8)\]as the agentic scaffold,Claude\-Sonnet\[[4](https://arxiv.org/html/2609.11117#bib.bib6)\]as the backbone model, and a skill document distilled fromClaude\-Opus\[[3](https://arxiv.org/html/2609.11117#bib.bib7)\]\. For evaluation, we useChatGPT\-4o\-mini\[[21](https://arxiv.org/html/2609.11117#bib.bib4)\]as the evaluator to reduce computational cost\.

Benchmarks\.BecauseAgentActionBench\-Humanis released as a training set for participants to develop and refine their models and systems, the main leaderboard is computed on the remaining papers inAgentActionBench\. However, since both human\-annotated and model\-generated rubrics are available forAgentActionBench\-Human, we additionally collect participants’ reproduction results on this subset to analyse the alignment between model\-generated and human\-authored rubrics\.

Repositories\.We evaluate submissions from three participating teams:YNU\-HPCC\-Task11\-AgentRep,zzunlp\_wu, andQueenAgent\. To provide an additional reference point, we also reproduce all 150 papers inAgentActionBenchusingGPT\-5\.4\[[27](https://arxiv.org/html/2609.11117#bib.bib22)\]with theCodexscaffold\[[22](https://arxiv.org/html/2609.11117#bib.bib5)\], which serves as our reproduction baseline\. Correlation analyses onAgentActionBench\-Humanare conducted using the results from all participating teams and the baseline\.

Environment\.All experiments are conducted on two NVIDIA L40S GPUs\.

Table 4:System performance across rubric types\. Each value denotes the percentage of the maximum available score obtained for the corresponding rubric type\.PO,PW,CI,CE, andRMdenotePaper Observation,Plan Writing,Code Implementation,Command Execution, andResult Matching, respectively\.Table 5:Correlation between evaluation scores produced by human\-annotated and model\-generated rubrics onAgentActionBench\-Human, with 95% confidence intervals and summary statistics\.Avg\. cnt\.denotes the average number of rubric items per paper, andAvg\. len\.denotes the average criterion length calculated by characters\.
## 5Result Analysis

### 5\.1Leaderboard

Table[3](https://arxiv.org/html/2609.11117#S3.T3)reports the performance of the participating systems and our reproduction baseline\.YNU\-HPCC\-Task11\-AgentRepperforms best, achieving an overall score of 49\.64%\. However, no system exceeds 50%, and theCodex\-GPT\-5\.4baseline reaches only 24\.19%, underscoring the difficulty of reproducing scientific experiments with current LLM\-based agents\. All systems also perform better on AI4Science papers than on ML papers, possibly because the selected AI4Science papers use more standard ML methods or simpler experimental configurations\.

A breakdown by rubric type is provided in Table[4](https://arxiv.org/html/2609.11117#S4.T4), where each percentage represents the score obtained relative to the maximum available for that category\. Systems perform relatively well onPaper Observation,Plan Writing, andCode Implementation, but substantially worse onCommand ExecutionandResult Matching\. This gap indicates that execution remains a major bottleneck: current agents can often generate plausible plans and code, yet struggle with runtime failures, environment and dependency issues, and reproducing the reported results\. These findings are consistent with prior studies identifying implementation and execution as key limitations of scientific agents\[[39](https://arxiv.org/html/2609.11117#bib.bib12),[33](https://arxiv.org/html/2609.11117#bib.bib10),[11](https://arxiv.org/html/2609.11117#bib.bib16)\]\.

![Refer to caption](https://arxiv.org/html/2609.11117v1/figures/rubric_score_share_pie.png)Figure 3:Comparison of score distributions across rubric types for human\-annotated and model\-generated rubrics\.
### 5\.2Rubric Generation Analysis

Table[5](https://arxiv.org/html/2609.11117#S4.T5)reports the correlations between the evaluation scores produced by human\-annotated and model\-generated rubrics onAgentActionBench\-Human, together with their 95% confidence intervals and summary statistics\. The model\-generated rubrics achieve Pearson and Spearman correlation coefficients of 0\.93 and 0\.88, respectively, both with narrow confidence intervals\. These results demonstrate strong agreement with the human annotations and support the reliability of the proposed rubric\-generation pipeline for scaling the benchmark\. The two rubric sources also have similar average numbers of rubric items per paper, suggesting that the generated rubrics broadly capture the structure of the human annotations\.

Figure[3](https://arxiv.org/html/2609.11117#S5.F3)further shows that their score distributions are comparable across most rubric types\. However, the model\-generated criteria are more verbose on average and assign a larger share of the total score toCode Implementation, while assigning smaller shares toPlan WritingandCommand Execution\. These differences suggest that the generated rubrics could benefit from further calibration of their granularity and category emphasis, consistent with prior findings\[[12](https://arxiv.org/html/2609.11117#bib.bib39)\]\.

## 6Conclusion

We introducedAgentActionBench, a process\-oriented benchmark for evaluating agent\-based experiment reproduction across ML and AI4Science papers\. Our framework combines an MCP\-based Action Recorder with fine\-grained, paper\-specific rubrics, enabling auditable evaluation of the full reproduction process rather than relying solely on final repositories\. To support evaluation at scale, we constructed a human\-annotated subset and expanded it to 150 papers using an LLM\-assisted rubric\-generation pipeline across multiple domains\. Experimental results show that current agents remain far from reliably reproducing scientific experiments, with execution and result verification emerging as the primary bottlenecks\. Meanwhile, the strong agreement between model\-generated and human\-annotated rubrics supports the scalability of our benchmark construction approach\.

## Limitations

Although rubric\-based evaluation substantially reduces hallucinations, it cannot eliminate them entirely\. Due to budget constraints, we use onlyGPT\-4o\-minias the evaluator; future work should assess whether stronger models provide more reliable judgments\. In addition, our coverage of AI4Science remains preliminary, and extending the benchmark to broader scientific domains is an important direction for future research\.

## References

- \[1\]J\. Abramson, J\. Adler, J\. Dunger, R\. Evans, T\. Green, A\. Pritzel, O\. Ronneberger, L\. Willmore, A\. J\. Ballard, J\. Bambrick, S\. W\. Bodenstein, D\. A\. Evans, C\. Hung, M\. O’Neill, D\. Reiman, K\. Tunyasuvunakool, Z\. Wu, A\. Žemgulytė, E\. Arvaniti, C\. Beattie, O\. Bertolli, A\. Bridgland, A\. Cherepanov, M\. Congreve, A\. I\. Cowen\-Rivers, A\. Cowie, M\. Figurnov, F\. B\. Fuchs, H\. Gladman, R\. Jain, Y\. A\. Khan, C\. M\. R\. Low, K\. Perlin, A\. Potapenko, P\. Savy, S\. Singh, A\. Stecula, A\. Thillaisundaram, C\. Tong, S\. Yakneen, E\. D\. Zhong, M\. Zielinski, A\. Žídek, V\. Bapst, P\. Kohli, M\. Jaderberg, D\. Hassabis, and J\. M\. Jumper\(2024\)Accurate structure prediction of biomolecular interactions with AlphaFold 3\.Nature630\(8016\),pp\. 493–500\.External Links:ISSN 1476\-4687,[Link](https://doi.org/10.1038/s41586-024-07487-w),[Document](https://dx.doi.org/10.1038/s41586-024-07487-w)Cited by:[§1](https://arxiv.org/html/2609.11117#S1.p2.1)\.
- \[2\]Anthropic\(2026\)Claude code overview\.External Links:[Link](https://code.claude.com/docs)Cited by:[§4](https://arxiv.org/html/2609.11117#S4.p1.1)\.
- \[3\]Anthropic\(2026\)Claude\-opus system card\.External Links:[Link](https://www-cdn.anthropic.com/6a5fa276ac68b9aeb0c8b6af5fa36326e0e166dd.pdf)Cited by:[§4](https://arxiv.org/html/2609.11117#S4.p1.1)\.
- \[4\]Anthropic\(2026\)Claude\-sonnet system card\.External Links:[Link](http://anthropic.com/claude-sonnet-4-6-system-card)Cited by:[§4](https://arxiv.org/html/2609.11117#S4.p1.1)\.
- \[5\]R\. K\. Arora, J\. Wei, R\. S\. Hicks, P\. Bowman, J\. Quiñonero\-Candela, F\. Tsimpourlas, M\. Sharman, M\. Shah, A\. Vallone, A\. Beutel, J\. Heidecke, and K\. Singhal\(2025\)HealthBench: evaluating large language models towards improved human health\.External Links:2505\.08775,[Link](https://arxiv.org/abs/2505.08775)Cited by:[§2\.2](https://arxiv.org/html/2609.11117#S2.SS2.p1.1)\.
- \[6\]M\. Baker\(2016\)1,500 scientists lift the lid on reproducibility\.Nature Publishing Group UK London\.Cited by:[§1](https://arxiv.org/html/2609.11117#S1.p1.1)\.
- \[7\]A\. M\. Bran, S\. Cox, O\. Schilter, C\. Baldassari, A\. D\. White, and P\. Schwaller\(2023\)ChemCrow: augmenting large\-language models with chemistry tools\.External Links:2304\.05376,[Link](https://arxiv.org/abs/2304.05376)Cited by:[§1](https://arxiv.org/html/2609.11117#S1.p2.1)\.
- \[8\]J\. S\. Chan, N\. Chowdhury, O\. Jaffe, J\. Aung, D\. Sherburn, E\. Mays, G\. Starace, K\. Liu, L\. Maksin, T\. Patwardhan, L\. Weng, and A\. Mądry\(2025\)MLE\-bench: evaluating machine learning agents on machine learning engineering\.InProceedings of the International Conference on Learning Representations \(ICLR\),Cited by:[§2\.1](https://arxiv.org/html/2609.11117#S2.SS1.p1.1)\.
- \[9\]J\. Gao, R\. Pi, J\. Zhang, J\. Ye, W\. Zhong, Y\. Wang, L\. Hong, J\. Han, H\. Xu, Z\. Li, and L\. Kong\(2023\)G\-LLaVA: solving geometric problem with multi\-modal large language model\.arXiv preprint arXiv:2312\.11370\.Cited by:[§2\.3](https://arxiv.org/html/2609.11117#S2.SS3.p1.1)\.
- \[10\]A\. Gunjal, A\. Wang, E\. Lau, V\. Nath, Y\. He, B\. Liu, and S\. M\. Hendryx\(2026\)Rubrics as rewards: reinforcement learning beyond verifiable domains\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=c1bTcrDmt4)Cited by:[§2\.2](https://arxiv.org/html/2609.11117#S2.SS2.p1.1)\.
- \[11\]H\. Hong, Y\. LI, J\. Chen, S\. Ananiadou, X\. Li, J\. Kim, and C\. Lin\(2026\)HiRAS: a hierarchical multi\-agent framework for paper\-to\-code generation and execution\.External Links:2604\.17745,[Link](https://arxiv.org/abs/2604.17745)Cited by:[§2\.1](https://arxiv.org/html/2609.11117#S2.SS1.p1.1),[§5\.1](https://arxiv.org/html/2609.11117#S5.SS1.p2.1)\.
- \[12\]H\. Hong, Y\. Li, J\. Chen, L\. G\. Huy, S\. Ananiadou, J\. Kim, and C\. Lin\(2026\)Can llms write reliable rubrics? a meta\-evaluation for experiment reproduction\.External Links:2607\.12835,[Link](https://arxiv.org/abs/2607.12835)Cited by:[§3\.2](https://arxiv.org/html/2609.11117#S3.SS2.p4.1),[§4](https://arxiv.org/html/2609.11117#S4.p1.1),[§5\.2](https://arxiv.org/html/2609.11117#S5.SS2.p2.1)\.
- \[13\]H\. Hong, C\. Xiao, Y\. Wang, Y\. Liu, W\. Rong, and C\. Lin\(2025\)Beyond one\-size\-fits\-all: inversion learning for highly effective nlg evaluation prompts\.External Links:2504\.21117,[Link](https://arxiv.org/abs/2504.21117)Cited by:[§1](https://arxiv.org/html/2609.11117#S1.p2.1)\.
- \[14\]Q\. Huang, J\. Vora, P\. Liang, and J\. Leskovec\(2024\)MLAgentBench: evaluating language agents on machine learning experimentation\.InProceedings of the 41st International Conference on Machine Learning \(ICML\),pp\. 20271–20309\.Cited by:[§2\.1](https://arxiv.org/html/2609.11117#S2.SS1.p1.1)\.
- \[15\]G\. J\. Kim, A\. Wilf, L\. Morency, and D\. Fried\(2026\)From reproduction to replication: evaluating research agents with progressive code masking\.InProceedings of the International Conference on Learning Representations \(ICLR\),Pittsburgh, PA, USA\.Cited by:[§2\.1](https://arxiv.org/html/2609.11117#S2.SS1.p1.1)\.
- \[16\]J\. Lin, Y\. Wang, Y\. Yu, Y\. Zhou, Y\. Chen, and X\. Shi\(2022\)Automatic analysis of available source code of top artificial intelligence conference papers\.International Journal of Software Engineering and Knowledge Engineering32\(07\),pp\. 947–970\.External Links:[Document](https://dx.doi.org/10.1142/S0218194022500358),[Link](https://doi.org/10.1142/S0218194022500358),https://doi\.org/10\.1142/S0218194022500358Cited by:[§1](https://arxiv.org/html/2609.11117#S1.p1.1)\.
- \[17\]T\. Liu, R\. Xu, T\. Yu, I\. Hong, C\. Yang, T\. Zhao, and H\. Wang\(2026\)OpenRubrics: towards scalable synthetic rubric generation for reward modeling and llm alignment\.External Links:2510\.07743,[Link](https://arxiv.org/abs/2510.07743)Cited by:[§2\.2](https://arxiv.org/html/2609.11117#S2.SS2.p1.1)\.
- \[18\]C\. Lu, C\. Lu, R\. T\. Lange, J\. N\. Foerster, J\. Clune, and D\. Ha\(2024\)The ai scientist: towards fully automated open\-ended scientific discovery\.CoRRabs/2408\.06292\.External Links:[Link](https://doi.org/10.48550/arXiv.2408.06292)Cited by:[§2\.1](https://arxiv.org/html/2609.11117#S2.SS1.p1.1)\.
- \[19\]Y\. Lyu, X\. Zhang, X\. Yi, Y\. Zhao, S\. Guo, W\. Hu, J\. Piotrowski, J\. Kaliski, J\. Urbani, Z\. Meng, L\. Zhou, and X\. Yan\(2026\)EvoScientist: towards multi\-agent evolving ai scientists for end\-to\-end scientific discovery\.External Links:2603\.08127,[Link](https://arxiv.org/abs/2603.08127)Cited by:[§1](https://arxiv.org/html/2609.11117#S1.p2.1)\.
- \[20\]I\. Magnusson, N\. A\. Smith, and J\. Dodge\(2023\)Reproducibility in NLP: what have we learned from the checklist?\.InFindings of the Association for Computational Linguistics: ACL 2023,A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 12789–12811\.External Links:[Link](https://aclanthology.org/2023.findings-acl.809/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-acl.809)Cited by:[§1](https://arxiv.org/html/2609.11117#S1.p1.1)\.
- \[21\]OpenAI\(2024\)GPT\-4o system card\.External Links:2410\.21276,[Link](https://arxiv.org/abs/2410.21276)Cited by:[§4](https://arxiv.org/html/2609.11117#S4.p1.1)\.
- \[22\]OpenAI\(2025\)Introducing codex\.External Links:[Link](https://openai.com/index/introducing-codex/)Cited by:[§4](https://arxiv.org/html/2609.11117#S4.p3.1)\.
- \[23\]\(2025\)Rubric is all you need: improving llm\-based code evaluation with question\-specific rubrics\.InProceedings of the 2025 ACM Conference on International Computing Education Research V\.1,External Links:[Document](https://dx.doi.org/10.1145/3702652.3744220)Cited by:[§2\.2](https://arxiv.org/html/2609.11117#S2.SS2.p1.1)\.
- \[24\]\(2026\)RubricHub: a comprehensive and highly discriminative rubric dataset via automated coarse\-to\-fine generation\.arXiv preprint arXiv:2601\.08430\.Cited by:[§2\.2](https://arxiv.org/html/2609.11117#S2.SS2.p1.1)\.
- \[25\]M\. Seo, J\. Baek, S\. Lee, and S\. J\. Hwang\(2025\)Paper2Code: automating code generation from scientific papers in machine learning\.External Links:2504\.17192,[Link](https://arxiv.org/abs/2504.17192)Cited by:[§1](https://arxiv.org/html/2609.11117#S1.p3.1),[§2\.1](https://arxiv.org/html/2609.11117#S2.SS1.p1.1)\.
- \[26\]Z\. S\. Siegel, S\. Kapoor, N\. Nadgir, B\. Stroebl, and A\. Narayanan\(2025\)CORE\-Bench: fostering the credibility of published research through a computational reproducibility agent benchmark\.Transactions on Machine Learning Research\.External Links:[Link](https://openreview.net/forum?id=BsMMc4MEGS)Cited by:[§2\.1](https://arxiv.org/html/2609.11117#S2.SS1.p1.1)\.
- \[27\]A\. Singh, A\. Fry, A\. Perelman, A\. Tart, A\. Ganesh,et al\.\(2026\)OpenAI gpt\-5 system card\.External Links:2601\.03267,[Link](https://arxiv.org/abs/2601.03267)Cited by:[§4](https://arxiv.org/html/2609.11117#S4.p3.1)\.
- \[28\]G\. Starace, O\. Jaffe, D\. Sherburn, J\. Aung, J\. S\. Chan, L\. Maksin, R\. Dias, E\. Mays, B\. Kinsella, W\. Thompson, J\. Heidecke, A\. Glaese, and T\. Patwardhan\(2025\)PaperBench: evaluating ai’s ability to replicate ai research\.External Links:2504\.01848,[Link](https://arxiv.org/abs/2504.01848)Cited by:[§1](https://arxiv.org/html/2609.11117#S1.p3.1),[§2\.1](https://arxiv.org/html/2609.11117#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2609.11117#S2.SS2.p1.1)\.
- \[29\]\(2025\)Text data augmentation for large language models: a comprehensive survey of methods, challenges, and opportunities\.Artificial Intelligence Review\.Cited by:[§2\.3](https://arxiv.org/html/2609.11117#S2.SS3.p1.1)\.
- \[30\]B\. Wang, T\. He, L\. Ouyang, F\. Wu, Z\. Zhao, T\. Chu, Y\. Qu, Z\. Jin, W\. Zeng, Z\. Miao,et al\.\(2026\)MinerU2\. 5\-pro: pushing the limits of data\-centric document parsing at scale\.arXiv preprint arXiv:2604\.04771\.Cited by:[§3\.2](https://arxiv.org/html/2609.11117#S3.SS2.p4.1)\.
- \[31\]Y\. Wang, Y\. Kordi, S\. Mishra, A\. Liu, N\. A\. Smith, D\. Khashabi, and H\. Hajishirzi\(2023\)Self\-instruct: aligning language models with self\-generated instructions\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Cited by:[§2\.3](https://arxiv.org/html/2609.11117#S2.SS3.p1.1)\.
- \[32\]Y\. Xiang, H\. Yan, S\. Ouyang, L\. Gui, and Y\. He\(2025\)SciReplicate\-Bench: benchmarking llms in agent\-driven algorithmic reproduction from research papers\.InProceedings of the Conference on Language Modeling \(COLM\),Cited by:[§2\.1](https://arxiv.org/html/2609.11117#S2.SS1.p1.1)\.
- \[33\]Q\. Xie, Y\. Weng, M\. Zhu, F\. Shen, S\. Huang, Z\. Lin, J\. Zhou, Z\. Mao, Z\. Yang, L\. Yang, J\. Wu, and Y\. Zhang\(2025\)How far are ai scientists from changing the world?\.External Links:2507\.23276,[Link](https://arxiv.org/abs/2507.23276)Cited by:[§5\.1](https://arxiv.org/html/2609.11117#S5.SS1.p2.1)\.
- \[34\]C\. Xu, Q\. Sun, K\. Zheng, X\. Geng, P\. Zhao, J\. Feng, C\. Tao, and D\. Jiang\(2023\)WizardLM: empowering large language models to follow complex instructions\.arXiv preprint arXiv:2304\.12244\.Cited by:[§2\.3](https://arxiv.org/html/2609.11117#S2.SS3.p1.1)\.
- \[35\]S\. Yan, R\. Li, Z\. Luo, Z\. Wang, D\. Li, L\. Jing, K\. He, P\. Wu, J\. Ni, G\. Michalopoulos, Y\. Zhang, Z\. Zhang, M\. Zhang, Z\. Chen, and X\. Du\(2025\)LMR\-BENCH: evaluating LLM agent’s ability on reproducing language modeling research\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 6164–6186\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.314/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.314),ISBN 979\-8\-89176\-332\-6Cited by:[§2\.1](https://arxiv.org/html/2609.11117#S2.SS1.p1.1)\.
- \[36\]J\. Yuan, X\. Yan, B\. Zhang, T\. Chen, B\. Shi, W\. Ouyang, Y\. Qiao, L\. Bai, and B\. Zhou\(2025\)Dolphin: moving towards closed\-loop auto\-research through thinking, practice, and feedback\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 21768–21789\.External Links:[Link](https://aclanthology.org/2025.acl-long.1056/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1056),ISBN 979\-8\-89176\-251\-0Cited by:[§1](https://arxiv.org/html/2609.11117#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.11117#S2.SS1.p1.1)\.
- \[37\]Q\. Zhang, J\. Zhou, Y\. Wang, F\. Lyu, Y\. Ming, C\. Xu, Q\. Sun, K\. Zheng, P\. Kang, X\. Liu, and C\. Ma\(2026\)RubricBench: aligning model\-generated rubrics with human standards\.External Links:2603\.01562,[Link](https://arxiv.org/abs/2603.01562)Cited by:[§2\.2](https://arxiv.org/html/2609.11117#S2.SS2.p1.1)\.
- \[38\]X\. Zhao, Z\. Sang, Y\. Li, Q\. Shi, W\. Zhao, S\. Wang, D\. Zhang, X\. Han, Z\. Liu, and M\. Sun\(2025\)AutoReproduce: automatic ai experiment reproduction with paper lineage\.External Links:2505\.20662,[Link](https://arxiv.org/abs/2505.20662)Cited by:[§1](https://arxiv.org/html/2609.11117#S1.p2.1)\.
- \[39\]M\. Zhu, Q\. Xie, Y\. Weng, J\. Wu, Z\. Lin, L\. Yang, and Y\. Zhang\(2025\)AI scientists fail without strong implementation capability\.External Links:2506\.01372,[Link](https://arxiv.org/abs/2506.01372)Cited by:[§5\.1](https://arxiv.org/html/2609.11117#S5.SS1.p2.1)\.

## Appendix 0\.ALog Example

\[
\{
"id": 1,
"timestamp": …,
"tool": "Read",
"arguments": \{
"path": "paper\.md",
"start\_offset": null,
"end\_offset": null
\},
"result": \{
"content": "\# My Paper\\n\#\# Abstract \\n…",
\},
\{
"id": 2,
"timestamp": "…",
"tool": "Write",
"arguments": \{
"path": "model\.py",
"content": "import torch\\nimport torch\.nn as nn\\n…"
\},
"result": \{
"success": true
\},
,
\{
"id": 3,
"timestamp": "…",
"tool": "Execute",
"arguments": \{
"cmd": "python train\.py –epochs 10"
\}, "result": \{
"stdout": "Epoch 1/10 loss=0\.45\\nEpoch 2/10 loss=0\.32\\n…",
"stderr": "",
"success": true
\},
\}
\]Figure 4:An example of Action Recorder logs\.