Can LLMs Write Reliable Rubrics? A Meta-Evaluation for Experiment Reproduction

# Can LLMs Write Reliable Rubrics? A Meta-Evaluation for Experiment Reproduction
Source: [https://arxiv.org/html/2607.12835](https://arxiv.org/html/2607.12835)
Hanhua Hong1,2,Yizhi Li3,Jiaoyan Chen1, Luu Gia Huy5,Sophia Ananiadou1,4,Jung\-jae Kim2,Chenghua Lin1 1The University of Manchester,2Institute for Infocomm Research \(I²R\), A\*STAR 3IQuest Research4ELLIS Manchester,5University of Information Technology, VNU hanhua\.hong@postgrad\.manchester\.ac\.uk,kim\_jung\_jae@a\-star\.edu\.sg \{jiaoyan\.chen,sophia\.ananiadou,chenghua\.lin\}@manchester\.ac\.uk

###### Abstract

Rubric\-based evaluation is a promising approach for assessing open\-ended outputs from LLM\-based research agents, particularly in paper reproduction, where direct paper\-to\-repository comparison is prone to hallucination\. However, constructing paper\-specific rubrics requires substantial expert effort, limiting the scalability of benchmarks such as PaperBench\. In this work, we present, to our knowledge, the first systematic meta\-evaluation of LLM\-generated rubrics for paper reproduction\. We reformulate rubrics into a checklist\-style format and evaluate four generation settings across two backbone models\. We meta\-evaluate generated rubrics intrinsically by semantic similarity and extrinsically by score alignment with ground\-truth rubrics\. Our results show that the augmented settings substantially improves downstream evaluation alignment, with the strongest setting approaching the human baseline, while intrinsic gains are more modest\. Further analyses reveal that LLM\-generated rubrics are often overly fine\-grained, biased toward high scores, and less adaptive to paper domains, highlighting both the affordances and limitations\.

Can LLMs Write Reliable Rubrics? A Meta\-Evaluation for Experiment Reproduction

Hanhua Hong1,2, Yizhi Li3, Jiaoyan Chen1,Luu Gia Huy5,Sophia Ananiadou1,4,Jung\-jae Kim2,Chenghua Lin11The University of Manchester,2Institute for Infocomm Research \(I²R\), A\*STAR3IQuest Research4ELLIS Manchester,5University of Information Technology, VNUhanhua\.hong@postgrad\.manchester\.ac\.uk,kim\_jung\_jae@a\-star\.edu\.sg\{jiaoyan\.chen,sophia\.ananiadou,chenghua\.lin\}@manchester\.ac\.uk

![Refer to caption](https://arxiv.org/html/2607.12835v1/figures/main_figure.png)Figure 1:Overview of our meta\-evaluation framework for LLM\-generated rubrics in paper reproduction\. Given a paper, reproduction agents generate candidate repositories, while LLMs and human experts produce checklist\-style rubrics\. We evaluate generated rubrics intrinsically by measuring their semantic similarity to human\-authored rubrics, and extrinsically by comparing the repository scores they induce with PaperBench evaluations\.## 1Introduction

Recent advances in large language models \(LLMs\) have substantially improved capabilities in code generation, reasoning, and tool use, driving rapid progress in LLM\-based agents for scientific researchZhaoet al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib5)\); Yuanet al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib54)\); Yamadaet al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib16)\); Lyuet al\.\([2026b](https://arxiv.org/html/2607.12835#bib.bib62)\)\. These agents are increasingly expected to interpret research papers, design experimental pipelines, and execute computational studies with minimal human supervision\.

Within this workflow, reproducing the experiments in a paper as an executable code repository remains one of the most critical and challenging stagesZhuet al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib33)\)\. The challenge is particularly severe in machine learning \(ML\), where publication pace far exceeds the release of artefacts, with only around 20% of recent papers at top conferences providing complete and reproducible codebasesLinet al\.\([2022](https://arxiv.org/html/2607.12835#bib.bib44)\); Magnussonet al\.\([2023](https://arxiv.org/html/2607.12835#bib.bib13)\); Seoet al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib4)\)\. To address this bottleneck, a growing line of work focuses on automated paper\-to\-code generation, including PaperCoderSeoet al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib4)\), AutoP2CLinet al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib63)\), AutoReproduceZhaoet al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib5)\), and HiRASHonget al\.\([2026](https://arxiv.org/html/2607.12835#bib.bib61)\)\.

However, reliably evaluating these systems is itself just as challenging, if not more\. Direct paper\-to\-repository comparison, as adopted in the Paper2Code benchmarkSeoet al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib4)\), is vulnerable to hallucination, where evaluators incorrectly infer that described methods have been implementedHonget al\.\([2026](https://arxiv.org/html/2607.12835#bib.bib61)\)\. In contrast, PaperBenchStaraceet al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib1)\)introduces*rubric\-based evaluation*: each paper is paired with a tree\-structured rubric consisting of fine\-grained, weighted pass/fail criteria spanning distinct reproduction stages, and candidate repositories are assessed independently against each criterion\. Rubric\-based evaluation has increasingly become a common strategy for evaluating open\-ended LLM outputs across domainsWanget al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib64)\); Aroraet al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib65)\); Sharmaet al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib68)\), and has been shown to yield more reliable judgements than holistic evaluationZhouet al\.\([2026](https://arxiv.org/html/2607.12835#bib.bib66)\)\.

Yet rubric construction has itself emerged as a major bottleneck\. PaperBench rubrics are manually designed in close collaboration with original paper authors, requiring substantial human effort while still covering only 20 papers\. The number of criteria per paper ranges from 94 to 2,551, reflecting the difficulty of standardising rubric design across diverse papers and annotators\. Similar annotation costs arise in rubric benchmarks across other domains: HealthBench required contributions from 262 physiciansAroraet al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib65)\), while ResearchRubrics consumed over 2,800 expert hoursSharmaet al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib68)\)\. At the same time, LLMs are increasingly used both as judges operating over fixed rubricsZhenget al\.\([2023](https://arxiv.org/html/2607.12835#bib.bib69)\); Guet al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib67)\)and as rubric generators for Reinforcement Learning reward modellingGunjalet al\.\([2026](https://arxiv.org/html/2607.12835#bib.bib77)\)\. Whether LLMs can also serve as reliable generators of rubrics for high\-stakes paper\-reproduction evaluation, while preserving alignment with human judgements, remains an open question\.

In this work, we investigate this question by first formulating rubrics in a checklist\-style representation, which is widely adopted in rubric\-based evaluationSharmaet al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib68)\); Aroraet al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib65)\); Wanget al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib64)\)\. We then study four incrementally augmented rubric\-generation settings, ranging from direct prompting to in\-context examples, agentic scaffolding, and distilled procedural skills\. Finally, we meta\-evaluate the generated rubrics along two complementary axes: intrinsic alignment with human\-authored rubrics, measured by lexical and semantic similarity, and extrinsic alignment, measured by the correlation between repository\-level scores induced by generated and ground\-truth rubrics\. To our knowledge, this is the first systematic meta\-evaluation of model\-generated rubrics for paper reproduction\. An overview of our work is presented in Figure[1](https://arxiv.org/html/2607.12835#S0.F1)\.

In summary, our contributions are three\-fold:

- •We present a controlled meta\-evaluation of LLMs as rubric authors for ML paper reproduction on PaperBench, covering four incrementally augmented generation settings and two backbone models\. The strongest setting achieves a Spearman correlation of0\.780\.78in extrinsic alignment, approaching the human baseline of0\.830\.83\.
- •We show that incremental augmentation substantially improves extrinsic alignment with human PaperBench evaluations, while intrinsic similarity improves more modestly and is mainly driven by increased recall, indicating that downstream evaluation quality depends more on rubric coverage and granularity than textual similarity\.
- •We further characterise the limitations of LLM\-generated rubrics through rubric statistics, score\-distribution analysis, category\-level analysis, and qualitative comparison\. Our findings show that generated rubrics are often more fine\-grained, code\-centric, and less adaptive to topic\-specific reproduction priorities than human\-authored rubrics\.

## 2Related Work

### 2\.1Agent\-Based Automatic Research

Recent advances in LLMs have enabled agent\-based systems for automating scientific workflows, from full\-cycle research systems such as AI ScientistLuet al\.\([2024](https://arxiv.org/html/2607.12835#bib.bib32)\); Yamadaet al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib16)\), DOLPHINYuanet al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib54)\), and ZochiIntology \([2025](https://arxiv.org/html/2607.12835#bib.bib17)\), to stage\-specific tools for idea generationWanget al\.\([2024](https://arxiv.org/html/2607.12835#bib.bib37)\); Liet al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib18)\), rigour assessmentJameset al\.\([2024](https://arxiv.org/html/2607.12835#bib.bib19)\), and experiment reproduction\. In the reproduction setting, systems such as PaperCoderSeoet al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib4)\), AutoP2CLinet al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib63)\), AutoReproduceZhaoet al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib5)\), and HiRASHonget al\.\([2026](https://arxiv.org/html/2607.12835#bib.bib61)\)aim to reconstruct codebases from research papers\.

Alongside these systems, benchmarks such as MLS\-BenchLyuet al\.\([2026a](https://arxiv.org/html/2607.12835#bib.bib73)\), SciReplicateBenchXianget al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib31)\), and PaperBenchStaraceet al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib1)\)have been proposed to evaluate scientific agents\. While these benchmarks improve evaluation reliability, they also expose a key bottleneck: they require fine\-grained, expert criteria which are normally not available \. Our addresses this bottleneck by systematically investigating whether LLMs can generate paper\-specific rubrics aligned with human\-authored evaluations\.

### 2\.2Rubric\-Based Evaluation

Rubrics have recently gained attention as structured supervision signals that decompose complex goals into fine\-grained criteria\. In LLM post\-training, instance\-specific rubrics have been used as reward signals to extend Reinforcement Learning beyond verifiable domains such as mathematics and codingGunjalet al\.\([2026](https://arxiv.org/html/2607.12835#bib.bib77)\), while recent work explores scalable synthetic rubric generation for reward modelling and alignmentLiuet al\.\([2026](https://arxiv.org/html/2607.12835#bib.bib78)\)\. Rubrics also support structured evaluation by replacing holistic judgments with explicit criteria\. In paper reproduction, PaperBench pairs each paper with a hierarchical rubric co\-developed with original authorsStaraceet al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib1)\), and similar protocols have been adopted in HealthBench, ProfBench, and ResearchRubricsAroraet al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib65)\); Wanget al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib64)\); Sharmaet al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib68)\)\.

Despite their value, rubrics remain expensive to construct and difficult to standardise\. PaperBench covers only 20 papers, while HealthBench required rubrics from 262 physiciansStaraceet al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib1)\); Aroraet al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib65)\)\. This has motivated work on model\-generated rubrics and rubric\-guided benchmarks, such as RubricBenchZhouet al\.\([2026](https://arxiv.org/html/2607.12835#bib.bib66)\)\. Our work studies this problem in paper reproduction, where rubric quality depends not only on textual plausibility but also on its ability to induce repository\-level evaluations aligned with human\-authored rubrics\.

### 2\.3Meta\-Evaluation

Meta\-evaluation studies whether evaluation methods themselves are reliable and aligned with human judgments\. This issue has become central as LLMs are increasingly used as automatic judges for open\-ended generation\. Prior studies such as G\-Eval\(Liuet al\.,[2023](https://arxiv.org/html/2607.12835#bib.bib74)\)show that prompting LLMs with explicit evaluation criteria can improve correlation with human judgments, while LLM\-as\-a\-judge studies and surveys identify important reliability concerns, including verbosity bias, position bias, self\-preference, and inconsistencyLiuet al\.\([2023](https://arxiv.org/html/2607.12835#bib.bib74)\); Zhenget al\.\([2023](https://arxiv.org/html/2607.12835#bib.bib69)\); Guet al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib67)\)\.

For rubric\-based evaluation, both the evaluator and the rubric itself are critical\. PaperBench compares human and LLM evaluators under fixed human\-authored rubrics, while RubricBench primarily assesses evaluators under different rubric sources in Reinforcement Learning reward modelling settings, with limited analysis of rubric generation itselfStaraceet al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib1)\); Zhouet al\.\([2026](https://arxiv.org/html/2607.12835#bib.bib66)\)\. In contrast, our work directly focuses on meta\-evaluating rubric generation for paper reproduction, exploring incrementally augmented generation settings through both intrinsic and extrinsic meta\-evaluation\.

## 3Methodology

### 3\.1Problem Definition

In this work, we represent rubrics in a checklist\-style format\. Each rubric itemrir\_\{i\}consists of a criterioncic\_\{i\}and an associated importance score,sis\_\{i\}ranging from 1 to 5, resulting inri=\(ci,si\)r\_\{i\}=\(c\_\{i\},s\_\{i\}\)\. For a paper𝒫\\mathcal\{P\}, the complete rubric set is defined asR=\{r1,r2,…,rn\}R=\\\{r\_\{1\},r\_\{2\},\\dots,r\_\{n\}\\\}\.

To evaluate whether a repository faithfully reproduces the experiments in the paper, the evaluator examines each rubric criterion independently and assigns a binary judgement, denoted asf:ci→Pass/Failf:c\_\{i\}\\rightarrow\\text\{Pass/Fail\}\. The final reproduction score of the paper is computed as the weighted sum of all passed criteria, normalised by the total rubric score:

score=∑i\[f​\(ci\)=Pass\]⋅si∑isi,\\mathrm\{score\}=\\frac\{\\sum\_\{i\}\\left\[f\(c\_\{i\}\)=\\mathrm\{Pass\}\\right\]\\cdot s\_\{i\}\}\{\\sum\_\{i\}s\_\{i\}\},\(1\)where\[⋅\]\[\\cdot\]returns 1 if the condition is satisfied and 0 otherwise\.

Algorithm 1Algorithm for the Agentic Scaffold1:Backbone Model

LLM\\mathrm\{LLM\}; Initial Prompt

p0p\_\{0\}; Workspace

W=\{𝒫,E\}W=\\\{\\mathcal\{P\},E\\\}
2:Ending Report

Report\\mathrm\{Report\}; Updated Workspace

WW;

3:

Mem←p0\\mathrm\{Mem\}\\leftarrow p\_\{0\}\.

4:while

True\\mathrm\{True\}do

5:

Reasoning,Action←LLM​\(Mem\)\\mathrm\{Reasoning,Action\}\\leftarrow\\mathrm\{LLM\}\(\\mathrm\{Mem\}\)\.

6:

Mem←Mem∪\{\(Reasoning,Action\)\}\\mathrm\{Mem\}\\leftarrow\\mathrm\{Mem\}\\cup\\\{\(\\mathrm\{Reasoning,Action\}\)\\\}\.

7:if

Action∈SystemTools\\mathrm\{Action\}\\in\\mathrm\{SystemTools\}then

8:

Result,W←system\.call\(​Action​\)\\mathrm\{Result\},W\\leftarrow\\texttt\{system\.call\(\}\\mathrm\{Action\}\\texttt\{\)\}\.

9:

Mem←Mem∪\{Result\}\\mathrm\{Mem\}\\leftarrow\\mathrm\{Mem\}\\cup\\\{\\mathrm\{Result\}\\\}\.

10:elseif

Action="End"\\mathrm\{Action\}=\\texttt\{"End\}\\texttt\{"\}then

11:break

12:endif

13:endwhile

14:

Report←Mem\\mathrm\{Report\}\\leftarrow\\mathrm\{Mem\}
15:return

Report,W\\mathrm\{Report\},W\.

### 3\.2Rubric Generation

We study rubric generation under four incrementally augmented generation settings designed to improve the quality and reliability of LLM\-authored rubrics\. Each setting builds on the previous one by introducing an additional form of augmentation\.

Direct Prompting\.In the base setting, the model is directly prompted with the target paper𝒫\\mathcal\{P\}and a manually designed instruction setIIthat specifies the checklist\-style rubric format and scoring standards\. We denote the resulting prompt aspD​P=\(I,𝒫\)p\_\{DP\}=\(I,\\mathcal\{P\}\)\. The model then generates the complete rubric in a single pass\.

In\-Context Examples\.Building on Direct Prompting, we additionally provide human\-authored rubric examplesEEfrom other papers as in\-context demonstrations\. These examples expose the model to the desired rubric structure, criterion granularity, and score allocation patterns, encouraging more consistent and human\-aligned rubric generation\. We denote the resulting prompt aspI​E=\(I,𝒫,E\)p\_\{IE\}=\(I,\\mathcal\{P\},E\)\.

Agentic Scaffold\.The third setting introduces an agentic scaffold following the ReAct paradigmYaoet al\.\([2023](https://arxiv.org/html/2607.12835#bib.bib15)\)\. Rather than generating the entire rubric in a single response, the model iteratively reasons over the task, inspects intermediate outputs, and refines the generated criteria across multiple interaction steps\. The agent starts with an initial promptp0=\(I\)p\_\{0\}=\(I\)and operates in a workspaceW=\{𝒫,E\}W=\\\{\\mathcal\{P\},E\\\}, which initially contains the target paper and in\-context examples\. The final generated rubric will also be stored in this workspace\. The agent is equipped with system tools such asRead\\mathrm\{Read\}andWrite\\mathrm\{Write\}, enabling it to access relevant files and save intermediate or final outputs during rubric generation\. The full procedure is provided in Algorithm[1](https://arxiv.org/html/2607.12835#alg1)\.

Distilled Skill\.Finally, we augment the generation process with a distilled procedural skill extracted from a stronger model\. Instead of using the stronger model to generate rubrics for every target paper, which would incur substantial computational cost, we run it only once on a single paper under the Agentic Scaffold setting\. After this run, the agent is further prompted to summarise its rubric\-construction procedure, heuristics, and decision patterns into a general\-purpose skill documentSS\. This distilled skill is then incorporated into the prompt of the weaker model and reused to guide rubric generation for all papers\. This setting also follows the agentic procedure, with the initial promptp0=\(I,S\)p\_\{0\}=\(I,S\)and workspaceW=\{𝒫,E\}W=\\\{\\mathcal\{P\},E\\\}\.

These four settings form a controlled augmentation hierarchy, enabling us to systematically analyse how progressively stronger forms of supervision and reasoning support affect the quality and evaluation behaviour of generated rubrics\. The prompt details are provided in Appendix[A](https://arxiv.org/html/2607.12835#A1)\.

### 3\.3Meta\-Evaluation Metrics

We conduct meta\-evaluation from two complementary perspectives\.Extrinsic meta\-evaluationassesses whether generated rubrics induce repository\-level evaluation outcomes consistent with those obtained using ground\-truth rubrics\.Intrinsic meta\-evaluationmeasures the lexical and semantic alignment between model\-generated and human\-authored rubrics\.

Extrinsic Meta\-Evaluation\.For extrinsic evaluation, we measure the alignment between generated and ground\-truth rubrics through their induced repository\-level evaluation outcomes\. For each paper, we collect multiple reproduction repositories and evaluate them with the same LLM\-based evaluator under different rubric sets\. To fit within the context limit, for each rubric criterion, the evaluator first retrieves the top\-KKmost relevant files from the repository and then assigns a binary pass/fail judgment based on the retrieved contents\.

We quantify outcome alignment using both Spearman and Pearson correlation coefficients\. Givenmmrepositories, let𝒟=\{d1,…,dm\}\\mathcal\{D\}=\\\{d\_\{1\},\\dots,d\_\{m\}\\\}denote the repository set for a paper\. The repository\-level score vectors induced by the generated rubricRRand the ground\-truth rubricGT\\mathrm\{GT\}are defined as:

𝐬R=⟨Eval​\(di,R\)⟩i=1m,𝐬GT=⟨Eval​\(di,GT\)⟩i=1m\.\\begin\{split\}\\mathbf\{s\}^\{R\}&=\\langle\\mathrm\{Eval\}\(d\_\{i\},R\)\\rangle\_\{i=1\}^\{m\},\\\\ \\mathbf\{s\}^\{\\mathrm\{GT\}\}&=\\langle\\mathrm\{Eval\}\(d\_\{i\},\\mathrm\{GT\}\)\\rangle\_\{i=1\}^\{m\}\.\\end\{split\}\(2\)
We then compute, for example, the Spearman correlation between the two score vectors:

ρ=Spearman​\(𝐬R,𝐬GT\)\.\\rho=\\mathrm\{Spearman\}\\left\(\\mathbf\{s\}^\{R\},\\mathbf\{s\}^\{\\mathrm\{GT\}\}\\right\)\.\(3\)
The final extrinsic alignment score is obtained by averaging the correlation coefficients across all evaluated papers in PaperBench\.

Intrinsic Meta\-Evaluation\.We evaluate intrinsic alignment at both lexical and semantic levels\. At the lexical level, for each generated rubric item, we compute its maximum ROUGE\-1 and ROUGE\-L scoresLin \([2004](https://arxiv.org/html/2607.12835#bib.bib75)\)against all ground\-truth rubric items for the same paper, and then average the scores across papers\.

At the semantic level, followingZhouet al\.\([2026](https://arxiv.org/html/2607.12835#bib.bib66)\), we measure matching precision, recall, and F1 between the generated rubric setR=\{ri\}i=1\|R\|R=\\\{r\_\{i\}\\\}\_\{i=1\}^\{\|R\|\}, whereri=\(ci,si\)r\_\{i\}=\(c\_\{i\},s\_\{i\}\), and the ground\-truth rubric setGT=\{gtj\}j=1\|GT\|\\mathrm\{GT\}=\\\{\\mathrm\{gt\}\_\{j\}\\\}\_\{j=1\}^\{\|\\mathrm\{GT\}\|\}, wheregtj=\(cjgt,sjgt\)\\mathrm\{gt\}\_\{j\}=\(c^\{\\mathrm\{gt\}\}\_\{j\},s^\{\\mathrm\{gt\}\}\_\{j\}\)\. UnlikeZhouet al\.\([2026](https://arxiv.org/html/2607.12835#bib.bib66)\), which relies on LLM\-based matching, we use embedding cosine similarityReimers and Gurevych \([2019](https://arxiv.org/html/2607.12835#bib.bib76)\)to identify criterion matches, thereby reducing the risk of hallucinations:

match​\(ci,cjgt\)=\[cos⁡\(ci,cjgt\)\>τ\],\\mathrm\{match\}\(c\_\{i\},c^\{\\mathrm\{gt\}\}\_\{j\}\)=\[\\cos\(c\_\{i\},c^\{\\mathrm\{gt\}\}\_\{j\}\)\>\\tau\],\(4\)whereτ\\tauis a predefined similarity threshold\.

A generated criterion is considered matched if it matches at least one ground\-truth criterion , while a ground\-truth criterion is considered covered if it is matched by at least one generated criterion\. Precision, recall, and F1 are computed as:

Precision=∑i\[∑jmatch​\(ci,cjgt\)≥1\]\|R\|,\\mathrm\{Precision\}=\\frac\{\{\\sum\}\_\{i\}\\left\[\{\\sum\}\_\{j\}\\mathrm\{match\}\(c\_\{i\},c^\{\\mathrm\{gt\}\}\_\{j\}\)\\geq 1\\right\]\}\{\|R\|\},\(5\)
Recall=∑j\[∑imatch​\(ci,cjgt\)≥1\]\|GT\|,\\mathrm\{Recall\}=\\frac\{\\sum\_\{j\}\\left\[\\sum\_\{i\}\\mathrm\{match\}\(c\_\{i\},c^\{\\mathrm\{gt\}\}\_\{j\}\)\\geq 1\\right\]\}\{\|\\mathrm\{GT\}\|\},\(6\)
F1=2⋅Precision⋅RecallPrecision\+Recall\.\\mathrm\{F1\}=\\frac\{2\\cdot\\mathrm\{Precision\}\\cdot\\mathrm\{Recall\}\}\{\\mathrm\{Precision\}\+\\mathrm\{Recall\}\}\.\(7\)
All semantic metrics are also computed per paper and then averaged across all evaluated papers in PaperBench\.

## 4Experiments

Table 1:Extrinsic meta\-evaluation results\. For Spearman and Pearson correlations, bold values indicate the best performance of each model\. For average rubric statistics, bold values indicate the closest match to the human baseline\. Incremental augmentations generally improve alignment with human PaperBench evaluations and increase rubric count\. The Distilled Skill setting achieves the strongest model\-generated performance, approaching the Human Annotation baseline\.Models\.For rubric generation, we useClaude\-SonnetandGPT\-5\.4to generate rubrics for all settings included in the meta\-evaluation\. Skill distillation is conducted withClaude\-OpusSinghet al\.\([2026](https://arxiv.org/html/2607.12835#bib.bib79)\); Anthropic \([2026b](https://arxiv.org/html/2607.12835#bib.bib10),[c](https://arxiv.org/html/2607.12835#bib.bib9)\)\. For evaluators, we useChatGPT\-4o\-minito assign scores to each repository according to the rubricsOpenAI \([2024](https://arxiv.org/html/2607.12835#bib.bib8)\)\. In addition, we useQwen3\-Embedding\-8BZhanget al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib80)\)to calculate the cosine similarity for the intrinsic meta\-evaluation\.

Baselines\.For comparison, we also manually re\-annotated the PaperBench papers in checklist\-style format to evaluate how human\-authored rubrics align with the ground\-truth rubrics of PaperBench\. This serves as a human baseline, which is denoted as Human Annotation in the main result tables\. For the In\-Context Examples setting, we add one set of our human\-authored rubrics in to the prompt to instruct model to generate rubrics in the similar format\. For the Agentic Scaffold setting, we use theClaude CodescaffoldAnthropic \([2026a](https://arxiv.org/html/2607.12835#bib.bib11)\)\.

Repositories\.For extrinsic meta\-evaluation, we use five reproduction repositories generated by prior agent\-based paper\-reproduction systems\. These repositories are produced under configurations ofPaperCoderandHiRAS:PaperCoderis run withDeepSeek\-V3andQwen3\-Coder\-480B, whileHiRASis run withClaude\-Sonnet,DeepSeek\-V3, andQwen3\-Coder\-480BDeepSeek\-AIet al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib7)\); Yanget al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib6)\)\.

Environment\.All experiments are conducted using eight NVIDIA L40S GPUs\. For repository evaluation, the evaluator retrieves the top\-K=10K=10most relevant files for each rubric criterion\. For semantic matching, we set the similarity threshold toτ=0\.7\\tau=0\.7in the main experiments\. Our key findings are not sensitive to this choice: varyingτ\\tauwithin a reasonable range preserves the same overall trends across augmentation levels to rubric generation\. Additional threshold\-sensitivity results are provided in Appendix[D](https://arxiv.org/html/2607.12835#A4)\.

Table 2:Intrinsic meta\-evaluation results\. Bold values denote the best performance for each model\. Incremental augmentations generally improve lexical overlap and semantic matching with human\-authored rubrics\. The gains are most evident in cosine\-similarity recall, F1, and ROUGE scores, indicating improved coverage of human\-specified criteria, while precision generally fluctuates across settings\.### 4\.1Extrinsic Meta\-Evaluation Results

Table[1](https://arxiv.org/html/2607.12835#S4.T1)presents the extrinsic meta\-evaluation results under incrementally augmented rubric\-generation settings\. Overall, each augmentation step improves the alignment between model\-generated and human\-authored rubrics across both model families, validating the effectiveness of the proposed augmentation hierarchy\. In particular, the full Distilled Skill setting achieves the strongest model\-generated performance for bothClaude\-SonnetandGPT\-5\.4, indicating that the proposed augmentations generalise across different backbone models\.

ForClaude\-Sonnet, Direct Prompting yields substantially lower alignment than the human\-annotation baseline, with a significant drop of approximately0\.20\.2in both Spearman and Pearson correlations\. As in\-context examples, agentic scaffolding, and distilled procedural skill are introduced, the correlations improve monotonically, demonstrating that each component contributes to stronger repository\-level evaluation alignment\. Compared with Direct Prompting, the Distilled Skill setting improves Spearman correlation by approximately26%26\\%and Pearson correlation by approximately21%21\\%, bringing the induced evaluation outcomes close to those obtained with human\-authored rubrics\. A similar pattern is observed forGPT\-5\.4\. Although direct prompting starts from weaker alignment, performance increases as the augmentation level grows and reaches its best result under the Distilled Skill setting\. Relative to direct prompting, Distilled Skill improves Spearman correlation by approximately35%35\\%and Pearson correlation by approximately27%27\\%, yielding near\-human performance and matching the best model\-generated Pearson correlation\. The initial drop of Direct Prompting is consistent with prior findings that naive LLM involvement in rubric construction can substantially degrade evaluation qualitySharmaet al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib68)\)\. Nevertheless, our results further show that proposed augmentations can substantially improve rubric quality, enabling current LLMs to induce repository\-level evaluation outcomes that approach those obtained with human\-authored rubrics\.

We also report rubric statistics in Table[1](https://arxiv.org/html/2607.12835#S4.T1)\. Incremental augmentations also substantially increase rubric quantities, with the average number of criteria rising from35\.9535\.95to77\.9577\.95forClaude\-Sonnetand from50\.1550\.15to87\.1587\.15forGPT\-5\.4\. These counts approach, or even slightly exceed, the human average of83\.2083\.20, indicating that stronger generation settings produce more comprehensive rubric sets\. However, model\-generated criteria remain generally longer than human\-authored ones, reflecting a length bias in LLM generationHuet al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib70)\); Buet al\.\([2025](https://arxiv.org/html/2607.12835#bib.bib71)\)\. Human annotations tend to aggregate implementation details into concise high\-level criteria, whereas model\-generated rubrics more often decompose them into fine\-grained operational steps\. This difference is further illustrated with the case study in §[5\.3](https://arxiv.org/html/2607.12835#S5.SS3)\.

![Refer to caption](https://arxiv.org/html/2607.12835v1/figures/ground-truth_score.png)\(a\)PaperBench Rubrics
![Refer to caption](https://arxiv.org/html/2607.12835v1/figures/claude-sonnet-skill_score.png)\(b\)Model\-Generated Rubrics

Figure 2:Score proportions by rubric type across paper categories\. Compared with the ground\-truth PaperBench rubrics, model\-generated rubrics fromClaude\-Sonnetunder the Distilled Skill setting show more uniform distributions and assign consistently higher weight toCode Developmentrubrics, indicating a code\-centric bias and weaker adaptation to topic\-specific reproduction
### 4\.2Intrinsic Meta\-Evaluation Results

Table[2](https://arxiv.org/html/2607.12835#S4.T2)presents the intrinsic meta\-evaluation results\. Overall, stronger augmentation improves the intrinsic alignment between model\-generated and human\-authored rubrics\. Both ROUGE\-1 and ROUGE\-L increase consistently from Direct Prompting to the Distilled Skill setting for both models, indicating that augmented generation produces rubric items with greater lexical overlap with human\-authored criteria\. The cosine\-similarity F1 score also generally improves across augmentation levels, mainly driven by increased recall, showing that stronger augmentation settings primarily improve coverage of ground\-truth rubric criteria\.

ForClaude\-Sonnet, the Distilled Skill setting improves cosine\-similarity F1 by approximately8%8\\%relative to Direct Prompting, while ROUGE\-1 and ROUGE\-L improve by approximately8%8\\%and10%10\\%, respectively\. ForGPT\-5\.4, the gains are more pronounced: Distilled Skill improves cosine\-similarity F1 by approximately13%13\\%, ROUGE\-1 by approximately11%11\\%, and ROUGE\-L by approximately15%15\\%over direct prompting\. These improvements suggest that incremental augmentations not only enhance downstream evaluation alignment, but also make generated rubrics intrinsically closer to human\-authored rubrics\.

At the same time, the intrinsic gains are less pronounced than the extrinsic improvements\. Precision fluctuates across settings but remains close to the human baseline\. This may imply that models can already generate semantically valid criteria\. However, taken together with lower recall and longer rubrics in Table[1](https://arxiv.org/html/2607.12835#S4.T1), this pattern indicates a tendency toward overly fine\-grained criteria, which limit models’ holistic coverage of the ground\-truth rubric\. The main improvement in cosine\-similarity matching comes from recall, which further suggests that augmentations help mitigate this granularity issue by improving the coverage of reproduction criteria\. Overall, the intrinsic results show that augmentation improves textual alignment and semantic coverage, while the remaining gap to human annotations reflects differences in coverage and granularity\.

## 5Further Analysis

### 5\.1Paper Category Analysis

To examine topic\-level differences, we group the PaperBench papers into four categories: \(A\) Language Model Adaptation and Alignment \(5 papers\); \(B\) Vision and Vision\-Language Models \(5 papers\); \(C\) Reinforcement Learning \(4 papers\), and \(D\) Probabilistic and Mathematical ML \(6 papers\)\. The detailed categorisation is provided in Appendix[C](https://arxiv.org/html/2607.12835#A3)\.

PaperBench divides rubric criteria into three types:Code Development,Execution, andResult Match\. We follow this taxonomy and compute the proportion of total score assigned to each type within each category\. Figure[2](https://arxiv.org/html/2607.12835#S4.F2)compares the ground\-truth PaperBench rubrics with those generated byClaude\-Sonnetunder the Distilled Skill setting\.

The ground\-truth rubrics show clear category\-dependent variation\. In particular,Result Matchaccounts for 48\.8% of the score in Reinforcement Learning papers, indicating that human rubrics place greater emphasis on empirical outcomes in the Reinforcement Learning domain, where the generalisation under different circumstances is critical\. In contrast, model\-generated rubrics are much more uniform across categories:Code Developmentconsistently receives over 60% of the total score, whileResult Matchremains below 21%\. This reveals a systematic code\-centric bias in model\-generated rubrics\. After further inspection, the bias can be attributed to that the model tends to decompose implementation procedures into fine\-grained steps and assign them substantial weights to even the finest granularity\. Overall, these results show that LLM\-generated rubrics are structurally consistent but less adaptive to topic\-specific reproduction priorities\.

CriterionScoreHuman\-authored rubricsThe collection and processing of the Wikitext\-2 dataset as prompts are planned\.2The construction of a pairwise toxicity dataset using PPLM is planned\.2The DPO process for GPT2 and Llama2 with the pairwise toxicity dataset is planned\.4The code for the construction of a pairwise toxicity dataset using PPLM has been implemented\.3The code for the DPO process for GPT2 and Llama2 with the pairwise toxicity dataset has been implemented\.5The hyperparameters for DPO and PPLM have been set as in Appendix E\.1The un\-aligning DPO experiment on GPT2\-DPO matches Table 4: toxicity, perplexity, and F1 increase after scaling up toxic key vectors\.5Model\-generated rubricsThe plan includes building 24,576 pairwise preference samples using Wikitext\-2 prompts: greedy GPT2 sampling for positive continuations and PPLM for negative continuations\.3The plan includes training DPO on GPT2\-medium and Llama2\-7b with learning rate10−610^\{\-6\}, batch size 4, RMSProp optimizer,β=0\.1\\beta=0\.1, and validation patience 10\.3The code implements the DPO loss, where positive and negative continuation likelihood ratios are compared against the reference model\.5The code trains DPO on GPT2\-medium using the pairwise dataset with the specified hyperparameters\.5The code implements PPLM with all specified hyperparameters, including step size, top\-kk, iterations, GM scale, and KL scale\.3The un\-alignment experiment on GPT2\-DPO by scaling the top 7 toxic key vectors by10×10\\timeshas been implemented\.4The execution outcomes match Table 4: scaling toxic key vectors reverts toxicity while preserving PPL and F1\.4

Table 3:Qualitative comparison between human\-authored and model\-generated rubrics for the same DPO un\-alignment experiment\. The model\-generated rubrics are extracted from the Distilled Skill setting withClaude\-Sonnet\. They are more fine\-grained and assign relatively high scores to auxiliary details, whereas the human\-authored rubrics use general, concise criteria with more discriminative score allocation\.
### 5\.2Score Distribution Analysis

Figure[6](https://arxiv.org/html/2607.12835#A2.F6)in Appendix[B](https://arxiv.org/html/2607.12835#A2)illustrates the importance\-score distributions under different rubric\-generation settings withClaude\-Sonnet\. Direct Prompting produces a noticeably skewed distribution, with scores concentrated at higher importance levels and limited coverage of lower\-scored criteria, suggesting that naive prompting tends to overestimate criterion importance\. As in\-context examples, agentic scaffolding, and the distilled skill are introduced, the distributions become bell\-shaped and more balanced across the11–55range, suggesting improved score calibration\. Nevertheless, all model\-generated settings still show a mild bias toward higher scores\. Overall, stronger procedural guidance improves importance\-score calibration, but score overestimation remains a persistent limitation\.

### 5\.3Qualitative Analysis

To further characterise the generated rubrics, we inspect the outputs of the best\-performing Distilled Skill setting onClaude\-Sonnetand compare them with human\-authored rubrics in Table[4](https://arxiv.org/html/2607.12835#A3.T4)\. The table presents matched rubric segments for the same DPO un\-alignment experiment fromA Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and ToxicityLeeet al\.\([2024](https://arxiv.org/html/2607.12835#bib.bib72)\), one of the papers included in PaperBench\. We select this example because both rubrics target the same experiment, yet exhibit substantial differences in criterion granularity, score assignment, and overall rubric construction\.

The model\-generated rubric describes the experiment at a lower operational level, decomposing a single reproduction component into multiple fine\-grained checks\. It also explicitly specifies implementation details such as the number of samples in the dataset, and detailed hyperparameter values\. In contrast, the human\-authored rubric uses more concise, higher\-level criteria that focus on core reproduction milestones, such as constructing the pairwise toxicity dataset, implementing DPO, and reproducing the un\-alignment results\. Peripheral details, such as hyperparameter settings, are aggregated into a separate low\-weight criterion\. In addition, the model\-generated rubrics are assigned relatively higher scores than their human\-authored counterpart\. This example reinforces our previous findings that model\-generated rubrics are generally longer, more fine\-grained, and more likely to assign high scores to coding\-level details than human\-authored rubrics\.

## 6Conclusion

In this work, we studied whether LLMs can serve as reliable rubric authors for scalable and high\-quality paper\-reproduction evaluation\. We manually re\-annotated rubrics for PaperBench into a checklist\-style format as a baseline and evaluated LLM\-generated rubrics under four progressively augmented settings, using both intrinsic textual alignment and extrinsic repository\-level alignment with human PaperBench judgments\. Our results show that proposed augmentations substantially improves the downstream utility of generated rubrics, with the best setting achieving the strongest model\-generated performance and approaching the human baseline\. Intrinsic improvements are more modest and mainly driven by recall, suggesting that augmentation primarily expands coverage of ground\-truth rubrics\. Further analyses show that model\-generated rubrics remain more verbose, fine\-grained, and code\-centric than human\-authored rubrics, with less paper topic sensitivity and calibrated weighting\.

## Limitations

Our work focuses on the reproduction of machine learning papers, using the currently prevalent PaperBench benchmark as the testbed\. Therefore, the generalisation of our findings to other scientific domains remains uncertain, as experimental workflows and reproduction requirements may vary substantially across fields\. The evaluation scale is also limited by the size of PaperBench and the availability of models and reproduction repositories\. In addition, rubric quality may still vary with different prompt design, in\-context examples, and the implementation details of the agentic scaffold\.

## Ethical Considerations

All data used in this work are collected from publicly available sources\. Our goal is to support more reliable evaluation and reproduction of prior research, rather than to replace the creative, critical, and interpretive roles of human researchers\. The proposed rubric\-generation framework should therefore be viewed as an assistive tool for scalable evaluation, with human oversight remaining important in high\-stakes scientific assessment\. All human annotations and benchmark reformulations were conducted by research students with sufficient experience\.

## References

- Claude code overview\.External Links:[Link](https://code.claude.com/docs)Cited by:[§4](https://arxiv.org/html/2607.12835#S4.p2.1)\.
- Anthropic \(2026b\)Claude\-opus system card\.External Links:[Link](https://www-cdn.anthropic.com/6a5fa276ac68b9aeb0c8b6af5fa36326e0e166dd.pdf)Cited by:[§4](https://arxiv.org/html/2607.12835#S4.p1.1)\.
- Anthropic \(2026c\)Claude\-sonnet system card\.External Links:[Link](http://anthropic.com/claude-sonnet-4-6-system-card)Cited by:[§4](https://arxiv.org/html/2607.12835#S4.p1.1)\.
- R\. K\. Arora, J\. Wei, R\. S\. Hicks, P\. Bowman, J\. Quiñonero\-Candela, F\. Tsimpourlas, M\. Sharman, M\. Shah, A\. Vallone, A\. Beutel, J\. Heidecke, and K\. Singhal \(2025\)HealthBench: evaluating large language models towards improved human health\.External Links:2505\.08775,[Link](https://arxiv.org/abs/2505.08775)Cited by:[§1](https://arxiv.org/html/2607.12835#S1.p3.1),[§1](https://arxiv.org/html/2607.12835#S1.p4.1),[§1](https://arxiv.org/html/2607.12835#S1.p5.1),[§2\.2](https://arxiv.org/html/2607.12835#S2.SS2.p1.1),[§2\.2](https://arxiv.org/html/2607.12835#S2.SS2.p2.1)\.
- Y\. Bu, L\. Huo, Y\. Jing, and Q\. Yang \(2025\)Beyond excess and deficiency: adaptive length bias mitigation in reward models for RLHF\.InFindings of the Association for Computational Linguistics: NAACL 2025,L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 3091–3098\.External Links:[Link](https://aclanthology.org/2025.findings-naacl.169/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-naacl.169),ISBN 979\-8\-89176\-195\-7Cited by:[§4\.1](https://arxiv.org/html/2607.12835#S4.SS1.p3.5)\.
- DeepSeek\-AI, A\. Liu, B\. Feng, B\. Xue, B\. Wang, B\. Wu, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang, C\. Ruan, D\. Dai, D\. Guo, D\. Yang, D\. Chen, D\. Ji, E\. Li, F\. Lin, F\. Dai, F\. Luo, G\. Hao, G\. Chen, G\. Li, H\. Zhang, H\. Bao, H\. Xu, H\. Wang, H\. Zhang, H\. Ding, H\. Xin, H\. Gao, H\. Li, H\. Qu, J\. L\. Cai, J\. Liang, J\. Guo, J\. Ni, J\. Li, J\. Wang, J\. Chen, J\. Chen, J\. Yuan, J\. Qiu, J\. Li, J\. Song, K\. Dong, K\. Hu, K\. Gao, K\. Guan, K\. Huang, K\. Yu, L\. Wang, L\. Zhang, L\. Xu, L\. Xia, L\. Zhao, L\. Wang, L\. Zhang, M\. Li, M\. Wang, M\. Zhang, M\. Zhang, M\. Tang, M\. Li, N\. Tian, P\. Huang, P\. Wang, P\. Zhang, Q\. Wang, Q\. Zhu, Q\. Chen, and Q\. D\. et al\. \(2025\)DeepSeek\-v3 technical report\.External Links:2412\.19437,[Link](https://arxiv.org/abs/2412.19437)Cited by:[§4](https://arxiv.org/html/2607.12835#S4.p3.1)\.
- J\. Gu, X\. Jiang, Z\. Shi, H\. Tan, X\. Zhai, C\. Xu, W\. Li, Y\. Shen, S\. Ma, H\. Liu, S\. Wang, K\. Zhang, Y\. Wang, W\. Gao, L\. Ni, and J\. Guo \(2025\)A survey on llm\-as\-a\-judge\.External Links:2411\.15594,[Link](https://arxiv.org/abs/2411.15594)Cited by:[§1](https://arxiv.org/html/2607.12835#S1.p4.1),[§2\.3](https://arxiv.org/html/2607.12835#S2.SS3.p1.1)\.
- A\. Gunjal, A\. Wang, E\. Lau, V\. Nath, Y\. He, B\. Liu, and S\. M\. Hendryx \(2026\)Rubrics as rewards: reinforcement learning beyond verifiable domains\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=c1bTcrDmt4)Cited by:[§1](https://arxiv.org/html/2607.12835#S1.p4.1),[§2\.2](https://arxiv.org/html/2607.12835#S2.SS2.p1.1)\.
- H\. Hong, Y\. LI, J\. Chen, S\. Ananiadou, X\. Li, J\. Kim, and C\. Lin \(2026\)HiRAS: a hierarchical multi\-agent framework for paper\-to\-code generation and execution\.External Links:2604\.17745,[Link](https://arxiv.org/abs/2604.17745)Cited by:[§1](https://arxiv.org/html/2607.12835#S1.p2.1),[§1](https://arxiv.org/html/2607.12835#S1.p3.1),[§2\.1](https://arxiv.org/html/2607.12835#S2.SS1.p1.1)\.
- Z\. Hu, L\. Song, J\. Zhang, Z\. Xiao, T\. Wang, Z\. Chen, N\. J\. Yuan, J\. Lian, K\. Ding, and H\. Xiong \(2025\)Explaining length bias in LLM\-based preference evaluations\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 6763–6794\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.358/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.358),ISBN 979\-8\-89176\-335\-7Cited by:[§4\.1](https://arxiv.org/html/2607.12835#S4.SS1.p3.5)\.
- Intology \(2025\)Zochi technical report\.External Links:[Link](https://www.intology.ai/blog/zochi-tech-report)Cited by:[§2\.1](https://arxiv.org/html/2607.12835#S2.SS1.p1.1)\.
- J\. James, C\. Xiao, Y\. Li, and C\. Lin \(2024\)On the rigour of scientific writing: criteria, analysis, and insights\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 6523–6538\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.380/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.380)Cited by:[§2\.1](https://arxiv.org/html/2607.12835#S2.SS1.p1.1)\.
- A\. Lee, X\. Bai, I\. Pres, M\. Wattenberg, J\. K\. Kummerfeld, and R\. Mihalcea \(2024\)A mechanistic understanding of alignment algorithms: a case study on dpo and toxicity\.InProceedings of the 41st International Conference on Machine Learning,ICML’24\.Cited by:[§5\.3](https://arxiv.org/html/2607.12835#S5.SS3.p1.1)\.
- L\. Li, W\. Xu, J\. Guo, R\. Zhao, X\. Li, Y\. Yuan, B\. Zhang, Y\. Jiang, Y\. Xin, R\. Dang, Y\. Rong, D\. Zhao, T\. Feng, and L\. Bing \(2025\)Chain of ideas: revolutionizing research via novel idea development with LLM agents\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 8971–9004\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.477/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.477),ISBN 979\-8\-89176\-335\-7Cited by:[§2\.1](https://arxiv.org/html/2607.12835#S2.SS1.p1.1)\.
- C\. Lin \(2004\)ROUGE: a package for automatic evaluation of summaries\.InText Summarization Branches Out,Barcelona, Spain,pp\. 74–81\.External Links:[Link](https://aclanthology.org/W04-1013/)Cited by:[§3\.3](https://arxiv.org/html/2607.12835#S3.SS3.p6.1)\.
- J\. Lin, Y\. Wang, Y\. Yu, Y\. Zhou, Y\. Chen, and X\. Shi \(2022\)Automatic analysis of available source code of top artificial intelligence conference papers\.International Journal of Software Engineering and Knowledge Engineering32\(07\),pp\. 947–970\.External Links:[Document](https://dx.doi.org/10.1142/S0218194022500358),[Link](https://doi.org/10.1142/S0218194022500358),https://doi\.org/10\.1142/S0218194022500358Cited by:[§1](https://arxiv.org/html/2607.12835#S1.p2.1)\.
- Z\. Lin, Y\. Shen, Q\. Cai, H\. Sun, J\. Zhou, and M\. Xiao \(2025\)AutoP2C: an llm\-based agent framework for code repository generation from multimodal content in academic papers\.External Links:2504\.20115,[Link](https://arxiv.org/abs/2504.20115)Cited by:[§1](https://arxiv.org/html/2607.12835#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.12835#S2.SS1.p1.1)\.
- T\. Liu, R\. Xu, T\. Yu, I\. Hong, C\. Yang, T\. Zhao, and H\. Wang \(2026\)OpenRubrics: towards scalable synthetic rubric generation for reward modeling and llm alignment\.External Links:2510\.07743,[Link](https://arxiv.org/abs/2510.07743)Cited by:[§2\.2](https://arxiv.org/html/2607.12835#S2.SS2.p1.1)\.
- Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. Zhu \(2023\)G\-eval: NLG evaluation using gpt\-4 with better human alignment\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 2511–2522\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.153/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.153)Cited by:[§2\.3](https://arxiv.org/html/2607.12835#S2.SS3.p1.1)\.
- C\. Lu, C\. Lu, R\. T\. Lange, J\. N\. Foerster, J\. Clune, and D\. Ha \(2024\)The ai scientist: towards fully automated open\-ended scientific discovery\.CoRRabs/2408\.06292\.External Links:[Link](https://doi.org/10.48550/arXiv.2408.06292)Cited by:[§2\.1](https://arxiv.org/html/2607.12835#S2.SS1.p1.1)\.
- B\. Lyu, Y\. Yang, S\. Huang, J\. Zhang, Q\. Xu, X\. Li, X\. Han, Y\. Zhang, H\. Zhang, R\. Huang, K\. Yang, Z\. Chen, W\. Guo, J\. Yang, X\. Ai, W\. Chai, Y\. Cao, Z\. Yang, K\. Wang, D\. Jiang, H\. Gao, S\. Tang, C\. Shi, S\. S\. Du, M\. Simchowitz, J\. Jiao, D\. Song, and C\. Jin \(2026a\)MLS\-bench: a holistic and rigorous assessment of ai systems on building better ai\.External Links:2605\.08678,[Link](https://arxiv.org/abs/2605.08678)Cited by:[§2\.1](https://arxiv.org/html/2607.12835#S2.SS1.p2.1)\.
- Y\. Lyu, X\. Zhang, X\. Yi, Y\. Zhao, S\. Guo, W\. Hu, J\. Piotrowski, J\. Kaliski, J\. Urbani, Z\. Meng, L\. Zhou, and X\. Yan \(2026b\)EvoScientist: towards multi\-agent evolving ai scientists for end\-to\-end scientific discovery\.External Links:2603\.08127,[Link](https://arxiv.org/abs/2603.08127)Cited by:[§1](https://arxiv.org/html/2607.12835#S1.p1.1)\.
- I\. Magnusson, N\. A\. Smith, and J\. Dodge \(2023\)Reproducibility in NLP: what have we learned from the checklist?\.InFindings of the Association for Computational Linguistics: ACL 2023,A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 12789–12811\.External Links:[Link](https://aclanthology.org/2023.findings-acl.809/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-acl.809)Cited by:[§1](https://arxiv.org/html/2607.12835#S1.p2.1)\.
- OpenAI \(2024\)GPT\-4o system card\.External Links:2410\.21276,[Link](https://arxiv.org/abs/2410.21276)Cited by:[§4](https://arxiv.org/html/2607.12835#S4.p1.1)\.
- N\. Reimers and I\. Gurevych \(2019\)Sentence\-BERT: sentence embeddings using Siamese BERT\-networks\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),K\. Inui, J\. Jiang, V\. Ng, and X\. Wan \(Eds\.\),Hong Kong, China,pp\. 3982–3992\.External Links:[Link](https://aclanthology.org/D19-1410/),[Document](https://dx.doi.org/10.18653/v1/D19-1410)Cited by:[§3\.3](https://arxiv.org/html/2607.12835#S3.SS3.p7.4)\.
- M\. Seo, J\. Baek, S\. Lee, and S\. J\. Hwang \(2025\)Paper2Code: automating code generation from scientific papers in machine learning\.External Links:2504\.17192,[Link](https://arxiv.org/abs/2504.17192)Cited by:[§1](https://arxiv.org/html/2607.12835#S1.p2.1),[§1](https://arxiv.org/html/2607.12835#S1.p3.1),[§2\.1](https://arxiv.org/html/2607.12835#S2.SS1.p1.1)\.
- M\. Sharma, C\. B\. C\. Zhang, C\. Bandi, C\. Wang, A\. Aich, H\. Nghiem, T\. Rabbani, Y\. Htet, B\. Jang, S\. Basu, A\. Balwani, D\. Peskoff, M\. Ayestaran, S\. M\. Hendryx, B\. Kenstler, and B\. Liu \(2025\)ResearchRubrics: a benchmark of prompts and rubrics for evaluating deep research agents\.External Links:2511\.07685,[Link](https://arxiv.org/abs/2511.07685)Cited by:[§1](https://arxiv.org/html/2607.12835#S1.p3.1),[§1](https://arxiv.org/html/2607.12835#S1.p4.1),[§1](https://arxiv.org/html/2607.12835#S1.p5.1),[§2\.2](https://arxiv.org/html/2607.12835#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2607.12835#S4.SS1.p2.5)\.
- A\. Singh, A\. Fry, A\. Perelman, A\. Tart, A\. Ganesh,et al\.\(2026\)OpenAI gpt\-5 system card\.External Links:2601\.03267,[Link](https://arxiv.org/abs/2601.03267)Cited by:[§4](https://arxiv.org/html/2607.12835#S4.p1.1)\.
- G\. Starace, O\. Jaffe, D\. Sherburn, J\. Aung, J\. S\. Chan, L\. Maksin, R\. Dias, E\. Mays, B\. Kinsella, W\. Thompson, J\. Heidecke, A\. Glaese, and T\. Patwardhan \(2025\)PaperBench: evaluating ai’s ability to replicate ai research\.External Links:2504\.01848,[Link](https://arxiv.org/abs/2504.01848)Cited by:[§1](https://arxiv.org/html/2607.12835#S1.p3.1),[§2\.1](https://arxiv.org/html/2607.12835#S2.SS1.p2.1),[§2\.2](https://arxiv.org/html/2607.12835#S2.SS2.p1.1),[§2\.2](https://arxiv.org/html/2607.12835#S2.SS2.p2.1),[§2\.3](https://arxiv.org/html/2607.12835#S2.SS3.p2.1)\.
- Q\. Wang, D\. Downey, H\. Ji, and T\. Hope \(2024\)SciMON: scientific inspiration machines optimized for novelty\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 279–299\.External Links:[Link](https://aclanthology.org/2024.acl-long.18/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.18)Cited by:[§2\.1](https://arxiv.org/html/2607.12835#S2.SS1.p1.1)\.
- Z\. Wang, J\. Jung, X\. Lu, S\. Diao, E\. Evans, J\. Zeng, P\. Molchanov, Y\. Choi, J\. Kautz, and Y\. Dong \(2025\)ProfBench: multi\-domain rubrics requiring professional knowledge to answer and judge\.External Links:2510\.18941,[Link](https://arxiv.org/abs/2510.18941)Cited by:[§1](https://arxiv.org/html/2607.12835#S1.p3.1),[§1](https://arxiv.org/html/2607.12835#S1.p5.1),[§2\.2](https://arxiv.org/html/2607.12835#S2.SS2.p1.1)\.
- Y\. Xiang, H\. Yan, S\. Ouyang, L\. Gui, and Y\. He \(2025\)SciReplicate\-bench: benchmarking LLMs in agent\-driven algorithmic reproduction from research papers\.InSecond Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=8LoPjpvWde)Cited by:[§2\.1](https://arxiv.org/html/2607.12835#S2.SS1.p2.1)\.
- Y\. Yamada, R\. T\. Lange, C\. Lu, S\. Hu, C\. Lu, J\. Foerster, J\. Clune, and D\. Ha \(2025\)The ai scientist\-v2: workshop\-level automated scientific discovery via agentic tree search\.External Links:2504\.08066,[Link](https://arxiv.org/abs/2504.08066)Cited by:[§1](https://arxiv.org/html/2607.12835#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.12835#S2.SS1.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, and B\. Y\. et al\. \(2025\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§4](https://arxiv.org/html/2607.12835#S4.p3.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. R\. Narasimhan, and Y\. Cao \(2023\)ReAct: synergizing reasoning and acting in language models\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=WE_vluYUL-X)Cited by:[§3\.2](https://arxiv.org/html/2607.12835#S3.SS2.p4.4)\.
- J\. Yuan, X\. Yan, B\. Zhang, T\. Chen, B\. Shi, W\. Ouyang, Y\. Qiao, L\. Bai, and B\. Zhou \(2025\)Dolphin: moving towards closed\-loop auto\-research through thinking, practice, and feedback\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 21768–21789\.External Links:[Link](https://aclanthology.org/2025.acl-long.1056/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1056),ISBN 979\-8\-89176\-251\-0Cited by:[§1](https://arxiv.org/html/2607.12835#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.12835#S2.SS1.p1.1)\.
- Y\. Zhang, M\. Li, D\. Long, X\. Zhang, H\. Lin, B\. Yang, P\. Xie, A\. Yang, D\. Liu, J\. Lin, F\. Huang, and J\. Zhou \(2025\)Qwen3 embedding: advancing text embedding and reranking through foundation models\.External Links:2506\.05176,[Link](https://arxiv.org/abs/2506.05176)Cited by:[§4](https://arxiv.org/html/2607.12835#S4.p1.1)\.
- X\. Zhao, Z\. Sang, Y\. Li, Q\. Shi, W\. Zhao, S\. Wang, D\. Zhang, X\. Han, Z\. Liu, and M\. Sun \(2025\)AutoReproduce: automatic ai experiment reproduction with paper lineage\.External Links:2505\.20662,[Link](https://arxiv.org/abs/2505.20662)Cited by:[§1](https://arxiv.org/html/2607.12835#S1.p1.1),[§1](https://arxiv.org/html/2607.12835#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.12835#S2.SS1.p1.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica \(2023\)Judging LLM\-as\-a\-judge with MT\-bench and chatbot arena\.InThirty\-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=uccHPGDlao)Cited by:[§1](https://arxiv.org/html/2607.12835#S1.p4.1),[§2\.3](https://arxiv.org/html/2607.12835#S2.SS3.p1.1)\.
- J\. Zhou, Q\. Zhang, Y\. Wang, F\. Lyu, Y\. Ming, C\. Xu, Q\. Sun, K\. Zheng, P\. Kang, X\. Liu, and C\. Ma \(2026\)RubricBench: aligning model\-generated rubrics with human standards\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 31179–31200\.External Links:[Link](https://aclanthology.org/2026.acl-long.1439/),[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.1439),ISBN 979\-8\-89176\-390\-6Cited by:[§1](https://arxiv.org/html/2607.12835#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.12835#S2.SS2.p2.1),[§2\.3](https://arxiv.org/html/2607.12835#S2.SS3.p2.1),[§3\.3](https://arxiv.org/html/2607.12835#S3.SS3.p7.4)\.
- M\. Zhu, Q\. Xie, Y\. Weng, J\. Wu, Z\. Lin, L\. Yang, and Y\. Zhang \(2025\)AI scientists fail without strong implementation capability\.External Links:2506\.01372,[Link](https://arxiv.org/abs/2506.01372)Cited by:[§1](https://arxiv.org/html/2607.12835#S1.p2.1)\.

## Appendix APrompt Details

You are a serious examiner of experiment reproduction\. Now you are given a paper\. You task is to read the entire paper and then give a set of rubrics to determine whether the paper is reproduced correctly\. Please make sure you read and understand these instructions carefully\.\# Paper: \{paper\_content\}\# Instructions:1\. Please read the paper carefully and then give a set of rubrics to determine whether the paper is reproduced correctly\.2\. You must focus on the core contents about methodology and experiment settings of the paper, and the criteria in your rubric should be very critical and specific\.3\. The rubrics should be as comprehensive as possible, making sure it can reflect all the implementation details of the paper\.4\. Your final target is to evaluate the reproduction of a codebase according to the paper\. Thus, your rubrics should be able to be validated with the codebase\.5\. Each rubric should contain a single criteria and a score ranged from 1 to 5 according to its importance in the reproduction\.6\. Do not output anything else except the JSON rubrics\.\# Score Standard:The score you assign to each criteria should be based on the following standard:1 Not Important: This criteria only reflects a very minor detail of the paper, or is not closely related to the code implementation\. The reproduction will still be executable even if this criteria is not met\. \(e\.g\. mentioning some specific hyperparameters in plans\)2 Less Important: This criteria is generally less important in terms of implementation, but the reproduction will be significantly affected if this criteria is not met, especially in the early stages of the reproduction\. \(e\.g\. reading the whole paper and understanding the overall methodology\)3 Important: This criteria is important during the reproduction, greatly influence the whole reproduction process and the final result\. The final outcome will be significantly affected if this criteria is not met\. \(e\.g\. planning the repository architecture or each of the experiment settings in detail\)4 Very Important: This criteria describes a key step, component, or function in the reproduction according to the paper\. It is an essential part of the implementation, the repository would not be executable if this criteria is not met\. \(e\.g\. coding part of a component or a useful cuntion, implementing a specific experiment setting, etc\.\)5 Critical: This criteria is one of the most important standard to measure whether a reproduction is successful\. The reproduction will not be considered complete if this requirement is not met or there are missing points for this criteria\. \(e\.g\. coding the key method from the paper or a critical component for the whole reproduction to be executable, running the main experiments, etc\.\)\# Rubrics Design Steps:1\. Read the full paper: You need to first read the full paper carefully and understand the overall methodology and experiment settings\. Do not miss even a single detail, and make sure you have a clear understanding of how the paper is implemented\.2\. Analyse the related sections: Based on your understanding of the paper, analyse the related sections that are closely related to the reproduction\. For example, the methodology section, the experiment settings section, the results section, the discussion section, etc\.3\. Extract key process: For each experiment mentioned in the paper, extract the key components, algorithm, or function that is essential for the reproduction\. They must be constructed in the codebase, and multiple rubrics may be introduced to examine each step of the implementation of these key processes\.4\. Articulate the criteria: You will be required to articulate the criteria for all detailed steps of the reproduction\. Please make sure your rubrics cover all stages and details of the reproduction, and each criteria should be specific and clear, which is able to evaluate the reproduction with the codebase\.5\. Assign the score: Based on the importance of each criteria, assign the score to each criteria\. Please refer to the Score Standard section for the detailed score standard\.6\. Output the rubrics: You must output a valid JSON string of all the rubrics\. Do not output anything else except the JSON rubrics\.\# Format:You should output the rubrics in the following JSON format\. Remember to add “‘json and “‘ at the beginning and end of the JSON:“‘json\[\{"criteria": "…","score": "…"\}\]“‘DO NOT output anything else except the JSON rubrics\.Your output:Figure 3:The basic prompt template for rubric generation in the Direct Prompting setting\.You are a serious examiner of experiment reproduction\. Now you are given a paper\. You task is to read the entire paper and then give a set of rubrics to determine whether the paper is reproduced correctly\. Please make sure you read and understand these instructions carefully\.\# Paper: \{paper\_content\}\# Instructions:1\. Please read the paper carefully and then give a set of rubrics to determine whether the paper is reproduced correctly\.2\. You must focus on the core contents about methodology and experiment settings of the paper, and the criteria in your rubric should be very critical and specific\.3\. The rubrics should be as comprehensive as possible, making sure it can reflect all the implementation details of the paper\.4\. Your final target is to evaluate the reproduction of a codebase according to the paper\. Thus, your rubrics should be able to be validated with the codebase\.5\. Each rubric should contain a single criteria and a score ranged from 1 to 5 according to its importance in the reproduction\.6\. Do not output anything else except the JSON rubrics\.\# Score Standard:The score you assign to each criteria should be based on the following standard:1 Not Important: This criteria only reflects a very minor detail of the paper, or is not closely related to the code implementation\. The reproduction will still be executable even if this criteria is not met\. \(e\.g\. mentioning some specific hyperparameters in plans\)2 Less Important: This criteria is generally less important in terms of implementation, but the reproduction will be significantly affected if this criteria is not met, especially in the early stages of the reproduction\. \(e\.g\. reading the whole paper and understanding the overall methodology\)3 Important: This criteria is important during the reproduction, greatly influence the whole reproduction process and the final result\. The final outcome will be significantly affected if this criteria is not met\. \(e\.g\. planning the repository architecture or each of the experiment settings in detail\)4 Very Important: This criteria describes a key step, component, or function in the reproduction according to the paper\. It is an essential part of the implementation, the repository would not be executable if this criteria is not met\. \(e\.g\. coding part of a component or a useful cuntion, implementing a specific experiment setting, etc\.\)5 Critical: This criteria is one of the most important standard to measure whether a reproduction is successful\. The reproduction will not be considered complete if this requirement is not met or there are missing points for this criteria\. \(e\.g\. coding the key method from the paper or a critical component for the whole reproduction to be executable, running the main experiments, etc\.\)\# Rubrics Design Steps:1\. Read the full paper: You need to first read the full paper carefully and understand the overall methodology and experiment settings\. Do not miss even a single detail, and make sure you have a clear understanding of how the paper is implemented\.2\. Analyse the related sections: Based on your understanding of the paper, analyse the related sections that are closely related to the reproduction\. For example, the methodology section, the experiment settings section, the results section, the discussion section, etc\.3\. Extract key process: For each experiment mentioned in the paper, extract the key components, algorithm, or function that is essential for the reproduction\. They must be constructed in the codebase, and multiple rubrics may be introduced to examine each step of the implementation of these key processes\.4\. Articulate the criteria: You will be required to articulate the criteria for all detailed steps of the reproduction\. Please make sure your rubrics cover all stages and details of the reproduction, and each criteria should be specific and clear, which is able to evaluate the reproduction with the codebase\.5\. Assign the score: Based on the importance of each criteria, assign the score to each criteria\. Please refer to the Score Standard section for the detailed score standard\.6\. Output the rubrics: You must output a valid JSON string of all the rubrics\. Do not output anything else except the JSON rubrics\.\# Format:Here is a sample rubrics file, examples\.json, is provided as reference from another paper\. Read through the examples carefully to understand how they are structured and designed\. You should strictly follow the format defined in examples\.json\.\# Examples \{examples\}Remember to add “‘json and “‘ at the beginning and end of the JSON\. DO NOT output anything else except the JSON rubrics\.Your output:Figure 4:The prompt template with examples in the In\-Context Examples setting\.You are a serious examiner of experiment reproduction\. Now you are given a paper\. You task is to read the entire paper and then give a set of rubrics to determine whether the paper is reproduced correctly\. Please make sure you read and understand these instructions carefully\.Your working directory is \{working\_directory\}\. The paper is under the working directory and named \{file\_name\}\. You should read the paper carefully and understand its content comprehensively\.\# Instructions:1\. Please read the paper carefully and then give a set of rubrics to determine whether the paper is reproduced correctly\.2\. You must focus on the core contents about methodology and experiment settings of the paper, and the criteria in your rubric should be very critical and specific\. This means that not all sections from the papers should be designed with rubrics\. Only the rubrics related to important sections such as preliminary experiments, methodology, experiment setup, and results should be designed\.3\. The rubrics should be as comprehensive as possible, making sure it can reflect all the implementation details of the paper\.4\. Your final target is to evaluate the reproduction of a codebase according to the paper\. Thus, your rubrics should be able to be validated with the codebase\.5\. Each rubric should contain a single criteria and a score ranged from 1 to 5 according to its importance in the reproduction\.6\. Do not output anything else except the JSON rubrics\.\# Score Standard:The score you assign to each criteria should be based on the following standard:1 Not Important: This criteria only reflects a very minor detail of the paper, or is not closely related to the code implementation\. The reproduction will still be executable even if this criteria is not met\. \(e\.g\. mentioning some specific hyperparameters in plans\)2 Less Important: This criteria is generally less important in terms of implementation, but the reproduction will be significantly affected if this criteria is not met, especially in the early stages of the reproduction\. \(e\.g\. reading the whole paper and understanding the overall methodology\)3 Important: This criteria is important during the reproduction, greatly influence the whole reproduction process and the final result\. The final outcome will be significantly affected if this criteria is not met\. \(e\.g\. planning the repository architecture or each of the experiment settings in detail\)4 Very Important: This criteria describes a key step, component, or function in the reproduction according to the paper\. It is an essential part of the implementation, the repository would not be executable if this criteria is not met\. \(e\.g\. coding part of a component or a useful cuntion, implementing a specific experiment setting, etc\.\)5 Critical: This criteria is one of the most important standard to measure whether a reproduction is successful\. The reproduction will not be considered complete if this requirement is not met or there are missing points for this criteria\. \(e\.g\. coding the key method from the paper or a critical component for the whole reproduction to be executable, running the main experiments, etc\.\)\# Rubrics Design Steps:1\. Read the full paper: You need to first read the full paper carefully and understand the overall methodology and experiment settings\. Do not miss even a single detail, and make sure you have a clear understanding of how the paper is implemented\.2\. Analyse the related sections: Based on your understanding of the paper, analyse the related sections that are closely related to the reproduction\. For example, the methodology section, the experiment settings section, the results section, the discussion section, etc\.3\. Extract key process: For each experiment mentioned in the paper, extract the key components, algorithm, or function that is essential for the reproduction\. They must be constructed in the codebase, and multiple rubrics may be introduced to examine each step of the implementation of these key processes\.4\. Articulate the criteria: You will be required to articulate the criteria for all detailed steps of the reproduction\. Please make sure your rubrics cover all stages and details of the reproduction, and each criteria should be specific and clear, which is able to evaluate the reproduction with the codebase\.5\. Assign the score: Based on the importance of each criteria, assign the score to each criteria\. Please refer to the Score Standard section for the detailed score standard\.6\. Output the rubrics: You must output a valid JSON string of all the rubrics\. Do not output anything else except the JSON rubrics\.\# ExamplesA sample rubrics file, examples\.json, is provided as reference from another paper\. Read through the examples carefully to understand how they are structured and designed\. Once you have a clear understanding, generate the rubrics for the current paper \(\{file\_name\}\), strictly following the format defined in examples\.json\.Output the result to \{output\_path\}\.Figure 5:The initial prompt for agents\.
## Appendix BScore Distribution

![Refer to caption](https://arxiv.org/html/2607.12835v1/figures/direct_prompting.png)\(a\)Direct Prompting
![Refer to caption](https://arxiv.org/html/2607.12835v1/figures/examples.png)\(b\)In\-Context Examples
![Refer to caption](https://arxiv.org/html/2607.12835v1/figures/agent_framework.png)\(c\)Agent Scaffold
![Refer to caption](https://arxiv.org/html/2607.12835v1/figures/skill.png)\(d\)Distilled Skill

Figure 6:Rubric score distributions across four generation settings forClaude\-Sonnet\. The distributions become increasingly bell\-shaped as the augmentation level increases\.
## Appendix CPaper Categoristion

Table 4:Paper categorisation for PaperBench
## Appendix DThreshold Discussion

We report the main intrinsic evaluation results with the matching threshold set toτ=0\.7\\tau=0\.7in Table[2](https://arxiv.org/html/2607.12835#S4.T2)\. To assess sensitivity to this choice, we also provide results forτ=0\.65\\tau=0\.65andτ=0\.75\\tau=0\.75\. Across these threshold values, our key finding remains consistent: stronger generation settings primarily improve recall, while precision fluctuates across settings\.

Table 5:Intrinsic meta\-evaluation results withτ=0\.75\\tau=0\.75Table 6:Intrinsic meta\-evaluation results withτ=0\.65\\tau=0\.65