VERITAS: Towards a General-Purpose Replication Tool for Scientific Research

# Veritas: Towards a General-Purpose Replication Tool for Scientific Research
Source: [https://arxiv.org/html/2607.02931](https://arxiv.org/html/2607.02931)
Haokun Liu∗ University of Chicago haokunliu@uchicago\.edu &Filbert Aurelian Tjiaranata∗ University of Indonesia filbert\.aurelian@ui\.ac\.id &Chenhao Tan University of Chicago chenhao@uchicago\.edu

###### Abstract

AI tools are accelerating scientific publication while the systems that review it struggle to keep up, and independent verification of published research has become both harder and more important\. As manual replication is slow and expensive, a growing line of work uses coding agents to automate parts of the process\. Existing efforts are largely packaged as benchmarks with companion agents that only run inside the benchmark’s own pipeline, and no general\-purpose replication tool exists\. We presentVeritas, a domain\-agnostic replication framework built around CLI coding agents\. Given a paper, a code repository, or both,Veritasextracts the paper’s claims, runs the methodology while resolving issues as they arise, and judges each claim against the evidence from experiment runs\. The pipeline returns an importance\-weighted Replication Score, a severity\-rated log of every fix applied, and the patched codebase\. We evaluateVeritason CORE\-Bench and ReplicationBench, 65 papers spanning computer science, social science, medicine, and astrophysics\. Against two strong Claude Code baselines on the same model and host environment,Veritasachieves state\-of\-the\-art performance and leads on every metric on both benchmarks\.

11footnotetext:Equal contribution\.## 1Introduction

AI tools are reshaping how scientific work is produced and reviewed\. Submission volumes at major venues have grown faster than the systems that review them: NeurIPS received 21,575 submissions in 2025, a 128% increase over its 2020 count\[[19](https://arxiv.org/html/2607.02931#bib.bib30),[18](https://arxiv.org/html/2607.02931#bib.bib31)\], and peer review at this scale exceeds the capacity of existing review processes\[[29](https://arxiv.org/html/2607.02931#bib.bib32)\]\. Under this pressure, papers with hallucinated references have started passing review undetected\.\[[27](https://arxiv.org/html/2607.02931#bib.bib33),[1](https://arxiv.org/html/2607.02931#bib.bib34),[10](https://arxiv.org/html/2607.02931#bib.bib35)\]\. Independent verification of published results beyond superficial checks is therefore increasingly important\.

Replication is the standard mechanism for independent verification, yet manual replication is slow and expensive\[[5](https://arxiv.org/html/2607.02931#bib.bib20),[21](https://arxiv.org/html/2607.02931#bib.bib21)\]\. A growing body of work\[[25](https://arxiv.org/html/2607.02931#bib.bib8),[26](https://arxiv.org/html/2607.02931#bib.bib9),[30](https://arxiv.org/html/2607.02931#bib.bib10),[9](https://arxiv.org/html/2607.02931#bib.bib11),[13](https://arxiv.org/html/2607.02931#bib.bib14),[31](https://arxiv.org/html/2607.02931#bib.bib15),[3](https://arxiv.org/html/2607.02931#bib.bib36)\]responds to this gap by using coding agents to automate parts of the replication process\. Most of these efforts are released as benchmarks: fixed task suites paired with companion agents that the original authors built to evaluate their benchmark\. The agents released alongside CORE\-Bench\[[25](https://arxiv.org/html/2607.02931#bib.bib8)\], PaperBench\[[26](https://arxiv.org/html/2607.02931#bib.bib9)\], and ReplicationBench\[[30](https://arxiv.org/html/2607.02931#bib.bib10)\]are specialized for their benchmark’s task format and grading harness, and they run only inside that benchmark’s evaluation pipeline\.

ZeroshotRetryVeritas020204040606080801001008084\.497\.8⋆Pass rate \(%\)\(a\)CORE\-Bench Hard\.ZeroshotRetryVeritas0202040406060808010010031\.531\.533\.351\.457\.160Match rate \(%\)paper\-onlyfull\(b\)ReplicationBench\.
Figure 1:Native benchmark scores after the cheating correction \([§˜3\.1](https://arxiv.org/html/2607.02931#S3.SS1)\)\. All systems run Claude Code with Claude Opus 4\.8;ZeroshotandRetrybaseline details are in[§˜3\.1](https://arxiv.org/html/2607.02931#S3.SS1)\. \([1\(a\)](https://arxiv.org/html/2607.02931#S1.F1.sf1)\) CORE\-Bench Hard per\-capsule pass rate on the 45\-capsule public test set\.⋆marks scores under a zero\-variance re\-scoring rule applied uniformly to all three systems; the rule flips fourVeritasfailures and zeroZeroshotorRetryfailures \([Appendix˜D](https://arxiv.org/html/2607.02931#A4)\)\. \([1\(b\)](https://arxiv.org/html/2607.02931#S1.F1.sf2)\) ReplicationBench per\-task match rate, split by input mode: paper\-only over all 20 papers, full mode over the 7 papers with a public repository\.Veritasachieves state\-of\-the\-art performance on both benchmarks and leads on every metric\.A few standalone replication frameworks have also emerged\.Kohleret al\.\[[13](https://arxiv.org/html/2607.02931#bib.bib14)\]replicates social\-science papers from the paper’s methods description and the original data\. AutoReproduce\[[32](https://arxiv.org/html/2607.02931#bib.bib13)\]replicates machine\-learning papers from the paper alone\. Each framework targets a specific domain and a specific mix of inputs, and no existing framework handles papers across domains or adapts to a varied set of inputs\.

CLI\-based coding agents have emerged in parallel with this line of work\. Claude Code\[[2](https://arxiv.org/html/2607.02931#bib.bib23)\], Codex\[[22](https://arxiv.org/html/2607.02931#bib.bib24)\], and Gemini CLI\[[8](https://arxiv.org/html/2607.02931#bib.bib25)\]can read documentation, install dependencies, and iterate on the code they generate and run\. They have set state\-of\-the\-art results on a range of coding benchmarks\[[11](https://arxiv.org/html/2607.02931#bib.bib28),[17](https://arxiv.org/html/2607.02931#bib.bib29)\]\. On CORE\-Bench, a Claude Code scaffold recently reached 95% accuracy, well above the benchmark’s custom CORE\-Agent baseline\[[12](https://arxiv.org/html/2607.02931#bib.bib27)\], which suggests that a CLI agent of this kind is a strong foundation for replication work\.

We presentVeritas, the first end\-to\-end domain\-agnostic replication framework built around CLI coding agents\.Veritassupports flexible input modes, extracts the paper’s claims independently of the original authors, and withholds those claims from the replicating agent to mitigate leakage\. The pipeline spans claim extraction, replication, and verification, and aggregates the per\-claim verdicts into an importance\-weighted Replication Score for the paper\.

We evaluateVeritason CORE\-Bench\[[25](https://arxiv.org/html/2607.02931#bib.bib8)\]\(computer science, social science, and medicine\) and ReplicationBench\[[30](https://arxiv.org/html/2607.02931#bib.bib10)\]\(astrophysics\) against two strong Claude Code baselines on the same model and host environment\.Veritasleads on every metric we report across both benchmarks\. On CORE\-Bench Hard it reaches a 97\.8% per\-capsule pass rate\. On ReplicationBench it leads the per\-task match rate in both paper\-only and full modes\. The same lead holds on the two trajectory\-level metrics we adapt from prior work\. On the cheating metric fromKohleret al\.\[[13](https://arxiv.org/html/2607.02931#bib.bib14)\], which flags trajectories that read forbidden sources,Veritashas the lowest cheating count\.Veritasalso shows the largest improvement on the faithfulness metric fromBaiet al\.\[[3](https://arxiv.org/html/2607.02931#bib.bib36)\], which judges whether a trajectory’s evidence supports its reported values\.

In summary, we make these contributions:

- •We introduceVeritas, the first domain\-agnostic replication framework built around CLI coding agents, with three input modes \(paper plus code, paper only, code only\) and a from\-scratch codegen phase for the paper\-only mode\. For each paper,Veritasreturns a detailed, per\-claim replication report, including a severity\-rated log of every fix applied, together with a single Replication Score summarizing how well the paper’s claims replicate\.
- •We adapt two existing trajectory\-level evaluations into our framework: a cheating metric from the access\-based detector ofKohleret al\.\[[13](https://arxiv.org/html/2607.02931#bib.bib14)\]and a faithfulness metric from the execution\-grounded checklists ofBaiet al\.\[[3](https://arxiv.org/html/2607.02931#bib.bib36)\]\. We apply a uniform cheating correction to all reported results\.
- •We evaluateVeritasacross 65 papers from two replication benchmarks spanning four domains, from computer science to astrophysics, and demonstrates the effectiveness ofVeritas\.

## 2Veritas

Veritasis a domain\-agnostic replication framework for scientific research\. Given a paper, a code repository, or both,Veritasruns the methodology, resolves issues that arise, and judges each empirical claim against the result\.Veritasproduces a verification report, including the replication scores and a severity\-rated log of every fix it applied and the patched codebase\. The output also includes Replication Score that aggregates the per\-claim verdicts\.

Veritasruns as a six\-phase pipeline \([Figure˜2](https://arxiv.org/html/2607.02931#S2.F2)\)\. Theanalyzephase extracts the paper’s claims\. Thecodegenphase generates code from the paper when no repository is provided\. Theplanphase drafts the replication procedure, and thereplicatephase executes the plan and fixes failures as they arise\. A bounded manager loop wrapsreplicateand judges each attempt before the pipeline moves on\. Theassess fixesphase rates the severity of each applied fix, and theverifyphase judges each claim against the evidence the run produced\. Each phase is carried out by a coding agent\. We use Claude Code in our experiments;Veritasalso supports Codex and Gemini\. We describe the input modes and each phase below\.

![Refer to caption](https://arxiv.org/html/2607.02931v1/x1.png)Figure 2:TheVeritaspipeline\.Analyzeextracts structured claims,plandrafts a replication procedure,replicateruns it while resolving issues,assess fixesrates each fix, andverifyjudges each claim with a comparator step that extracts the replicated value and a deterministic grader that assigns the status\. Verdicts aggregate into an importance\-weighted Replication Score\.Codegenruns only in paper\-only mode, beforeplan\. A bounded manager loop reads execution checks after everyreplicateattempt and either accepts the attempt or issues a written directive that re\-runsreplicate\(orplan, when the plan itself is the problem\) with new guidance\.### 2\.1Input Modes

Veritasaccepts three input modes:

- •Full\.A paper and its code repository\. Claims are extracted from the paper, and the replication runs against the authors’ repository\.
- •Paper\-only\.A paper with no repository\. Thecodegenphase first writes an implementation of the paper’s methodology from scratch, and the replication runs against that generated codebase\.
- •Repo\-only\.A repository with no paper\. Claims are extracted from the repository’s README\.

Veritasalso accepts an optional pre\-positioned data directory as an input, mounted read\-only into the replication environment\.

### 2\.2Paper Claims

Theanalyzephase converts the input into a structured set of paper claims, the individual assertionsVeritaswill later try to verify\. In full and paper\-only modes, claims are extracted from the paper; in repo\-only mode, from the repository’s README\.Veritasalso accepts a hand\-authored claim set, which skips extraction entirely\. The claim set is withheld from the replication agent\.

Veritasannotates each claim with a type and an importance level\. The type specifies the form of the claim’s evidence and dictates how the verifier compares it\. A scalar claim is a single reported number\. A scalar\-range claim is a value the paper asserts to fall within an interval\. A table claim, or multi\-value claim, groups related numbers\. A qualitative claim covers non\-numeric findings, such as one method outperforming another\. A figure claim covers cases whose evidence is the layout or trend of a plot\. The importance level indicates how central the claim is to the paper\. Headline claims state the paper’s main results, and supporting claims are secondary findings\. The importance level weights the claim’s contribution to the Replication Score \([§˜2\.5](https://arxiv.org/html/2607.02931#S2.SS5)\)\.

### 2\.3Codegen

In paper\-only mode,Veritashas no source code to replicate, so thecodegenphase writes the paper’s methodology into a fresh codebase\. The generated codebase then passes through the rest of the pipeline exactly as the authors’ repository would\.

Thecodegenprompt guides the agent through four steps: explore the paper, plan the codebase, implement the plan module by module, and self\-review the result\. To prevent the agent from hardcoding paper\-reported results, the self\-review prompt contains an constant\-origin check that verifies every numerical constant in the generated code is either a paper\-stated input or computed at runtime\. Alongside the prompt,Veritasprovides a catalog of scientific\-computing skills in the agent’s working directory and instructs the agent to use the skills when relevant\.

### 2\.4Replication

Plan\.Theplanphase reads the codebase and produces a replication plan, an ordered list of steps that run the methodology and produce the evidence each claim needs\. Every step records which claims it serves \(only by their identifier, never the values the paper reported\) and an expected outcome, the form its output should take \(such as a file path or a set of field names\)\. The plan exposes only claim identifiers to the executing agent, never the values the paper reported\.

Replicate\.Thereplicatephase executes the plan on a writable copy of the codebase, keeping the original repository read\-only\.111Veritassupports two runtimes, a Docker container\[[16](https://arxiv.org/html/2607.02931#bib.bib22)\]by default, and a direct\-on\-host wrapper\. The pipeline logic is identical in both runtimes\.The agent actively resolves failures it encounters, trying multiple fixes before declaring a step unreproducible\. Each fix is recorded for theassess fixesphase\.

Manager loop\.A bounded manager loop wrapsreplicateand decides whether the attempt is good enough to score\. After each replicate attempt,Veritascomputes a small set of deterministic execution checks from the run log\. The checks cover whether every planned step was executed, the per\-step exit codes and declared output files, repeated commands that suggest a stuck loop, the total number of fixes, and the wall\-clock time\. The checks and the run transcript are passed to a separate manager agent that returns a structured verdict with two parts\. The first part is a decision, either accept or revise\. The second part is a written directive that explains what the agent should change on the next attempt\. On an accept verdict the pipeline proceeds toassess fixes, otherwise with a revise verdict, the directive calls into the next attempt\. Most revises targetreplicate, which then re\-runs with the new directive in its instructions\. When the directive blames the plan rather than the execution, the manager targetsplanand the pipeline restarts from there\. The loop is bounded by a hard iteration cap and stops early when the execution checks do not improve and the directive does not change between attempts\. The manager sees the trajectory and the checks and does not see the paper’s reported values\.

Assess fixes\.If any fixes were applied during replication, theassess fixesphase rates each one as minor, major, or critical\. Ratings come from a separate agent call, not from the agent that applied the fixes, to avoid self\-evaluation bias\.

### 2\.5Verification and the Replication Score

Verification\.Theverifyphase judges each claim in two steps\. A comparator LLM reads the run’s evidence and extracts the replicated value in a structured form that matches the claim’s type, along with any uncertainty the run reports\. A grader then assigns the status by a deterministic rule\. For scalar and scalar\-range claims, the grader uses a relative\-error threshold against the paper’s value, with a sigma\-based rule when uncertainty is available\. For table claims, it grades each cell and aggregates\. For qualitative and figure claims, the grader defers to the comparator’s structured judgment because the comparison has no numerical form\. Each call has fresh context and only the evidence relevant to that claim, so that no verdict can influence another\. The verdict consists of four parts: a status, a type\-specific comparison of the paper’s reported and replicated evidence, a short rationale, and references to the evidence consulted\. A status takes one of five values: match, partial, no match, not attempted, and not applicable\.

Replication Score\.Veritasaggregates the per\-claim verdicts into a single importance\-weighted Replication Score\. Let𝒞\{\\mathcal\{C\}\}be the set of claims whose verdict is applicable\. For a claimcc, letw​\(c\)w\(c\)be its importance weight andv​\(c\)v\(c\)its verdict value\. The score is

Score=∑c∈𝒞w​\(c\)​v​\(c\)∑c∈𝒞w​\(c\)∈\[0,1\]\.\\mathrm\{Score\}=\\frac\{\\sum\_\{c\\in\{\\mathcal\{C\}\}\}w\(c\)\\,v\(c\)\}\{\\sum\_\{c\\in\{\\mathcal\{C\}\}\}w\(c\)\}\\;\\in\\;\[0,1\]\.\(1\)
Importance weights are33for headline claims and22for supporting claims\. Verdict values are1\.01\.0for a match,0\.50\.5for a partial match, and0\.00\.0for both no match and not attempted\. Claims marked not applicable drop out of both sums, so they do not affect the score\. The result is a single number in\[0,1\]\[0,1\]that summarizes how wellVeritasreplicated the paper, weighted toward the paper’s claims\.

## 3Experiments

We evaluateVeritason two replication benchmarks\. CORE\-Bench\[[25](https://arxiv.org/html/2607.02931#bib.bib8)\]covers computer science, social science, and medicine\. ReplicationBench\[[30](https://arxiv.org/html/2607.02931#bib.bib10)\]covers astrophysics\. The two differ in their inputs and grading, so we describe each in turn together with howVeritasintegrates with it\.[Figure˜1](https://arxiv.org/html/2607.02931#S1.F1)summarizes the native benchmark scores\. We first introduce the setup shared by both benchmarks\.

### 3\.1Setup

Models\.All systems use Claude Code with Claude Opus 4\.8\.

Environment\.Every system runs in the same machine and the same benchmark\-specific runtime environment, so that any score difference reflects the agent and not the setup\.Veritasand both baselines share a single CORE\-Bench capsule image per capsule and a single ReplicationBench paper image per paper\. We describe the images, the hardware, and the per\-benchmark conventions in[Appendix˜A](https://arxiv.org/html/2607.02931#A1)\.

Baselines\.We compareVeritasagainst two Claude Code baselines and the strongest published agent on each benchmark\. TheZeroshotbaseline runs Claude Code once on the benchmark’s task prompt and reports its final answer\. TheRetrybaseline runs up to three Claude Code sessions\. Each session resumes the previous one’s context after a fixed continuation prompt, and the final session’s answer is scored\. This mirrors the pattern used by the strongest published CORE\-Bench Hard agent\[[12](https://arxiv.org/html/2607.02931#bib.bib27)\]\. Baselines never seeVeritas’s intermediate artifacts, andVeritas’s verification phase never sees the baselines’ trajectories\.

Trajectory metrics\.On both benchmarks, final answers are scored by each benchmark’s own grader\. To also assess how the answers were obtained, we add two trajectory\-level metrics adapted from existing replication evaluations\. Our adaptations are minimal, limited to running the cited methods over our trajectory format and to per\-benchmark configuration that the cited methods themselves expose\. The exact prompts, taxonomy, and scoring formulas are unchanged from the source\. We hide the system identity from each LLM judge so the grade depends only on the trajectory itself\.

The cheating metric measures legitimacy\. We adapt the cheating detector ofKohleret al\.\[[13](https://arxiv.org/html/2607.02931#bib.bib14)\]\. It flags trajectories that read forbidden sources, such as the authors’ repository or files containing the paper’s reported values\. The judge taxonomy and five\-level severity scale are taken verbatim from \. A trajectory counts as a cheat at the two highest severity levels\. We report the rate of flagged trajectories per system\.

The faithfulness metric measures process soundness\. We adapt the execution\-grounded checklist of MechEvalAgent\[[3](https://arxiv.org/html/2607.02931#bib.bib36)\], which judges a trajectory on five binary items\. C1 asks whether the essential code ran\. C2 asks whether the implementation matched the paper’s method\. CS1 asks whether the reported answers were supported by the run’s evidence\. DE1 and DE3 together check that the reported values were grounded in a real computation\. We report the fraction of items passed, averaged over a system’s trajectories\.

For ReplicationBench we additionally adopt the percent\-difference grader ofKohleret al\.\[[13](https://arxiv.org/html/2607.02931#bib.bib14)\], which scores values with no error tolerance, as a fidelity metric\. We discuss why this grader does not transfer to CORE\-Bench in[§˜3\.2](https://arxiv.org/html/2607.02931#S3.SS2)\.

Cheating correction\.A trajectory flagged as a cheat reaches the reported answer through a forbidden shortcut, so its final\-answer score does not reflect genuine replication\. We report all final\-answer numbers in the main text after a cheating correction\. A passing capsule or task whose trajectory was flagged is counted as a fail, and a fidelity GPA on a flagged trajectory is zeroed\. The appendix carries the raw scores and a per\-flag breakdown for completeness\.

Scope and scoring\.The benchmarks evaluateVeritasas a replication agent\. To compare against the baselines on the same set of claims, we supply each benchmark’s task questions directly and map each to a single claim, so theanalyzephase and the importance\-weighted Replication Score \([§˜2\.5](https://arxiv.org/html/2607.02931#S2.SS5)\) are not exercised here\. Every number we report uses the benchmark’s own scorer, CORE\-Bench’s per\-capsule scorer and ReplicationBench’s per\-task scorer, so our results are directly comparable to the published baselines\. The Replication Score is the scoreVeritasreports to its own users and is not used to compute any number in this paper\. Theanalyzephase, the Replication Score, the severity\-rated fix log, and the replication report are part ofVeritasas a released tool for replicating and verifying papers; the experiments here test its replication core, theplan,replicate, andverifyphases\.

### 3\.2CORE\-Bench

Benchmark\.CORE\-Bench\[[25](https://arxiv.org/html/2607.02931#bib.bib8)\]draws 90 papers from CodeOcean\[[6](https://arxiv.org/html/2607.02931#bib.bib26)\]reproducibility capsules, with a 45\-capsule public test set\. The capsules cover computer science \(37\), social science \(28\), and medicine \(25\)\. The benchmark defines three difficulty levels \(Easy, Medium, Hard\) that differ only in the inputs the agent receives\. We evaluate on Hard, which provides the paper, source code, and README but no Dockerfile\. Each capsule carries one to eight task questions with numeric, string\-valued, or short structured reference answers\. CORE\-Bench grades numeric answers against a 95% prediction interval built from three reference runs, and non\-numeric answers by exact match\. A capsule passes only when every task question is answered correctly, and the per\-capsule pass rate is the benchmark’s main score\. The strongest agent reported in the CORE\-Bench paper is CORE\-Agent with GPT\-4o at 21\.5% on Hard\. On the public HAL leaderboard\[[12](https://arxiv.org/html/2607.02931#bib.bib27)\], the strongest entry is a Claude Code scaffold with Claude Opus 4\.5 that reaches 77\.8% under the official scorer and 95\.5% after manual validation of near\-miss answers\.

Veritassetup\.We supply CORE\-Bench’s task questions directly asVeritas’s paper claims, bypassingVeritas’sanalyzephase, so the comparison is on a common claim set\. Each task question becomes one headline claim, with type inferred from the structuredresultsfield\. We scoreVeritas’s replicated outputs with CORE\-Bench’s own scorer\.

Metrics\.We report three metrics: CORE\-Bench’s native per\-capsule pass rate, the faithfulness index, and the cheating rate\. The percent\-difference fidelity grader ofKohleret al\.\[[13](https://arxiv.org/html/2607.02931#bib.bib14)\], used as the fidelity metric on ReplicationBench, does not transfer well to CORE\-Bench\. About 38% of CORE\-Bench answers are non\-numeric, which the grader cannot score at all\. On the numeric remainder the grader saturates near the top grade for every system we evaluate, because CORE\-Bench’s answers are either exact or absent\. CORE\-Bench’s own binary pass rate already covers the final\-answer comparison\.

Results\.[Table˜1](https://arxiv.org/html/2607.02931#S3.T1)summarizes the results\. Under CORE\-Bench’s official scorerVeritasreaches 88\.9%, above the 77\.8% of the strongest leaderboard agent under the same scorer\. A re\-scoring rule that corrects a documented floating\-point edge case, applied identically to all three systems, raises this to 97\.8% \([Appendix˜D](https://arxiv.org/html/2607.02931#A4)\), above the 95\.5% the leaderboard reaches through manual validation\.Veritasleads on all three metrics\. Faithfulness shows the largest improvement\.Veritas’s verification phase rarely accepts an answer that the run’s evidence does not support\. The baselines occasionally report a value the trajectory does not actually compute\. On CORE\-Bench Hard the native pass rate spread is small because all three systems are strong Claude Code agents on Opus 4\.8\. The trajectory metrics separate the systems\.

The cheating count is zero forVeritasand nonzero for both baselines\. The most direct baseline flag isRetry’s read of the benchmark\-internal answer fileclaims\_cb\.jsonon a single capsule, which the detector flags at the highest severity\. The other two baseline flags are an external Zenodo download and the load of a paper\-specific R package\.[Appendix˜F](https://arxiv.org/html/2607.02931#A6)walks through these cases\.

Table 1:CORE\-Bench Hard results on the 45\-capsule public test set\. All systems run Claude Code with Claude Opus 4\.8;ZeroshotandRetrybaseline details are in[§˜3\.1](https://arxiv.org/html/2607.02931#S3.SS1)\. Pass rate is the per\-capsule native binary score after the cheating correction \(a capsule that passed but was flagged as a cheat is counted as a fail\)\.⋆marks scores under a zero\-variance re\-scoring rule applied uniformly to all three systems \([Appendix˜D](https://arxiv.org/html/2607.02931#A4)\); the rule flips fourVeritasfailures and zeroZeroshotorRetryfailures\. Under the official scorer,Veritasreaches 88\.9%\. Faithfulness is the index adapted fromBaiet al\.\[[3](https://arxiv.org/html/2607.02931#bib.bib36)\]over five binary items\. Cheating is the count of trajectories flagged by the cheating detector adapted fromKohleret al\.\[[13](https://arxiv.org/html/2607.02931#bib.bib14)\]\. Raw scores before correction are in[§˜E\.1](https://arxiv.org/html/2607.02931#A5.SS1)\.
### 3\.3ReplicationBench

Benchmark\.ReplicationBench\[[30](https://arxiv.org/html/2607.02931#bib.bib10)\]evaluates astrophysics paper replication\. Its main dataset is 20 peer\-reviewed papers decomposed by domain experts into 111 self\-contained tasks targeting the papers’ quantitative findings\. The agent receives a masked LaTeX manuscript, the dataset, and a typed schema for each task’s expected output\. Numerical result values in the manuscript are redacted to prevent direct copying\. No source code is provided in paper\-only mode\. Each task carries an author\-supplied tolerance, and a task is credited only when every value in its expected output is within tolerance\. ReplicationBench’s score is the per\-task pass rate, averaged uniformly\. The strongest agent reported in the original paper is Claude Sonnet 4\.5 at 22%\. Of the 20 papers, 9 declare a public source repository; we run full mode on the 7 whose repositories we could prepare and execute, and use them to compare paper\-only and code\-available replication\.

Veritassetup\.We feed the masked LaTeX as the paper input\. Each task becomes one headline claim, with type inferred from the task’sexpected\_outputfield and the author tolerance encoded verbatim\. We scoreVeritas’s replicated outputs with ReplicationBench’s own scorer\.

We runVeritasin two configurations\. The paper\-only mode covers all 20 papers;Veritasinvokes itscodegenphase to write the methodology from the masked manuscript before replicating it\. The full mode covers these 7 papers;Veritasreplicates against the authors’ code\. Comparing the two modes tests how much of the gap is closed by the masked manuscript alone\.

Metrics\.We report four metrics\. The first is ReplicationBench’s native per\-task match rate, broken out by paper\-only and full mode\. We then add the faithfulness index, the cheating rate, and fidelity\. Fidelity uses the percent\-difference grader ofKohleret al\.\[[13](https://arxiv.org/html/2607.02931#bib.bib14)\]\. It assigns each reported value a letter grade from A to F based on the percent difference against the paper’s value \(A under 2%, B under 20%, down to F for missing or sign\-flipped values\)\. We convert letters to a GPA on a 0 to 5 scale and average across all reported values\.

Results\.[Table˜2](https://arxiv.org/html/2607.02931#S3.T2)summarizes the results\.Veritasleads every metric after the cheating correction\. Faithfulness shows the largest improvement\.Veritas’s verification phase enforces that every reported value has an execution trace and that the trace matches the paper’s method\. Baseline trajectories sometimes fabricate values or skip essential steps\.

Cheating is the most informative metric\. The singleVeritasflag is for loading a model on the astm3 paper, where the task instructions themselves say to load a published pre\-trained model\.Veritasdoes so and computes inference honestly with the paper’s value masked\. The flag is a strict reading of the access rule, and we count it even though the trajectory follows the documented path\. The baseline flags are more substantive\. Two baseline trajectories cloned the authors’ GitHub repositories, and one downloaded a paper\-result archive from Zenodo\.[Appendix˜F](https://arxiv.org/html/2607.02931#A6)walks through the astm3 trajectory and a baseline that cloned the authors’ repository, alongside a third case where a baseline fabricated a value the detector does not detect\.

The native match rate gap between the twoVeritasconfigurations isolates the value of having the authors’ code\. Going from paper\-only to full mode increases the match rate substantially on the 7 papers with a public repository\. This is consistent with prior reports that paper\-only astrophysics replication remains difficult even for strong agents\.Veritasleads on fidelity after the cheating correction\. The baseline gap narrows because their highest\-scoring trajectories relied on the authors’ code, which the correction sets to zero\.

Table 2:ReplicationBench results\. All systems run Claude Code with Claude Opus 4\.8;ZeroshotandRetrybaseline details are in[§˜3\.1](https://arxiv.org/html/2607.02931#S3.SS1)\. Match rate is ReplicationBench’s per\-task pass rate after the cheating correction, broken out by paper\-only mode \(20 papers\) and full mode \(the 7 papers with a public repository\)\. Faithfulness and Cheating are defined as in[Table˜1](https://arxiv.org/html/2607.02931#S3.T1)\. Fidelity is the percent\-difference grade ofKohleret al\.\[[13](https://arxiv.org/html/2607.02931#bib.bib14)\]converted to a GPA on a 0 to 5 scale\. Cheated trajectories are zeroed\. Raw scores, per\-cell numbers, and a breakdown of which baseline flags survive once public packages are excluded are in[§˜E\.2](https://arxiv.org/html/2607.02931#A5.SS2)\.Error analysis\.ReplicationBench attributes its agents’ primary failures to three modes: a lack of persistence, conceptual or procedural errors, and technical execution failures\[[30](https://arxiv.org/html/2607.02931#bib.bib10)\]\. The lack of persistence and technical execution failures are largely resolved inVeritas, crediting the active fixing design that keeps its code running when facing issues\. For persistence,Veritasleaves only 6 tasks unattempted, where in 5 of them the required data is not provided\.Veritas’s faithfulness index of 0\.933 also indicates that it rarely reports a value it did not compute\. As a result, conceptual or procedural errors now dominateVeritas’s misses, such as a quantity defined differently from the paper or a skipped preprocessing step\. Additionally, some of the hardest tasks are limitedVeritas’s choice of scale\. When a computation is too expensive to run at full scale,Veritasdownscales it just so it can finish, and the smaller run can produce a different number \([Appendix˜G](https://arxiv.org/html/2607.02931#A7)\)\.

We also observe features of the benchmark itself\. Many ofVeritas’s misses are near or partial reproductions that ReplicationBench’s binary scorer does not credit\. Its tolerances are round, so a value a fraction of a percent from the target still counts as a miss\. A multi\-value claim is credited only when every entry is within tolerance, which is whyVeritasmatches 41% of scalar tasks but only 20% of multi\-value ones\. A few tasks are also unreproducible by construction\. Three of them, across two papers, fail regardless of the agent, owing to a shipped catalog of the wrong data release, a corrupted input file, and an instruction that contradicts its own answer key\. We give the full analysis in[Appendix˜G](https://arxiv.org/html/2607.02931#A7)\.

## 4Related Work

Replicability benchmarks\.A growing set of benchmarks evaluates AI agents on replicating research\. They differ in the research domains, the materials provided to the agent, and the evaluation criteria\. CORE\-Bench\[[25](https://arxiv.org/html/2607.02931#bib.bib8)\]supplies the paper, code, and data to the agent and evaluates the replication by deterministic comparison of the agent’s output to the original paper’s\. PaperBench\[[26](https://arxiv.org/html/2607.02931#bib.bib9)\]withholds the code and data, requiring an agent to reimplement a machine\-learning paper from scratch\. PaperBench grades against rubrics co\-developed with the original authors\. ReplicationBench\[[30](https://arxiv.org/html/2607.02931#bib.bib10)\]evaluates similar paper\-only replication tasks in astrophysics\. REPRO\-Bench\[[9](https://arxiv.org/html/2607.02931#bib.bib11)\]adopts a different formulation\. It asks an agent to*judge*whether an existing replication package is consistent with its paper, with no requirement to replicate\.

Paper\-to\-code systems\.A parallel line of work builds systems that translate a paper into its code implementation\. PaperCoder\[[24](https://arxiv.org/html/2607.02931#bib.bib12)\]uses a multi\-agent pipeline that plans an architecture, writes per\-file specifications, and generates a modular repository from the paper alone\. The pipeline stops once the codebase is written, with no execution and no check against the paper’s results\. AutoReproduce\[[32](https://arxiv.org/html/2607.02931#bib.bib13)\]traverses a paper’s cited works to extract implementation details left implicit in the text\. It then executes the generated code and reports a performance gap against the authors’ reference implementation\.

Agentic replication systems\.Recent systems replicate papers end\-to\-end and assess the outcome\.Kohleret al\.\[[13](https://arxiv.org/html/2607.02931#bib.bib14)\]replicate social\-science results from only the paper’s methods description\. They grade the replicated regression tables cell by cell against the paper’s values\. PaperRepro\[[31](https://arxiv.org/html/2607.02931#bib.bib15)\]runs a paper’s replication package and iteratively resolves execution failures\. It grades the outputs against the paper’s reported results and outperforms prior systems on REPRO\-Bench\. It also introduces REPRO\-Bench\-S, a difficulty\-stratified refinement of REPRO\-Bench\.

Benchmarks beyond replication\.Other benchmarks target tasks strictly harder than replication: re\-running studies on newly collected data\[[20](https://arxiv.org/html/2607.02931#bib.bib19)\], implementing novel*extensions*of a paper\[[7](https://arxiv.org/html/2607.02931#bib.bib18)\], and*rediscovering*a paper’s findings from a research question alone\[[28](https://arxiv.org/html/2607.02931#bib.bib17)\]\.

## 5Conclusion

We presentVeritas, a domain\-agnostic replication framework built around CLI coding agents\.Veritasaccepts a paper, a code repository, or both, extracts structured claims with a type and an importance level, runs the methodology in a pipeline that resolves failures as they arise, and judges each claim against the evidence the run produces\. The output is an importance\-weighted Replication Score, a severity\-rated log of every fix applied, and the patched codebase\. On CORE\-Bench Hard and ReplicationBench, we evaluateVeritasagainst two Claude Code baselines on the same model and host environment\. Our evaluation shows thatVeritasachieves state\-of\-the\-art performance and leads on every metric on both benchmarks\.

Veritasis one step toward an agent that can verify and judge any research work\. The natural next directions follow from the limitations we discuss above\. We plan to broaden the baselines beyond Claude Code, to add a hardcoding\-aware detector to the cheating metric, and to evaluate the pipeline on adjacent tasks such as paper extension and rediscovery from a research question alone\.

## Limitations

Coverage of evaluation\.We evaluateVeritason two benchmarks against two Claude Code baselines\. The baselines share a model family withVeritas, so the comparison isolates the value of the pipeline\. It does not show howVeritascompares to agents built on a different model family or on non\-CLI scaffolds\. Broader baselines and additional benchmarks would strengthen the generality claim\.Veritasis designed to verify and judge research work beyond CORE\-Bench\-style and ReplicationBench\-style tasks\. Evaluating the pipeline on more diverse paper sets and on adjacent tasks such as paper extension and finding rediscovery is a natural next step\.

Cheating metric coverage\.Our cheating metric adapts the access\-based detector ofKohleret al\.\[[13](https://arxiv.org/html/2607.02931#bib.bib14)\]and inherits its blind spots\. A baseline can write a numeric value as a literal with no producing computation and the detector will not catch it\. The faithfulness metric catches fabrication on its DE3 item\. The cheating count in the main table does not\. The fable\_mps case study in[§˜F\.3](https://arxiv.org/html/2607.02931#A6.SS3)shows where this matters\. Incorporating a hardcoding\-mode detector into the cheating count, with the exception for run\-provided code described in[Appendix˜C](https://arxiv.org/html/2607.02931#A3), is straightforward future work\.

Importance granularity\.Veritas’s importance scheme \(headline, supporting\) is coarser than PaperBench’s hierarchical rubric\. PaperBench uses per\-leaf weights co\-developed with paper authors to distinguish, for example, correct method implementation from final result\-match\[[26](https://arxiv.org/html/2607.02931#bib.bib9)\]\. Our coarser scheme gives up some rubric fidelity for automatic, author\-independent extraction\. Papers whose value lies disproportionately in implementation correctness will be scored less informatively byVeritasthan by a PaperBench\-style rubric\. Integrating finer\-grained scoring, either by extendingVeritas’s schema with author\-supplied leaves or by composingVeritaswith a PaperBench\-style grader, is left to future work\.

Run\-to\-run variance\.We report single\-run results\. CORE\-Bench and ReplicationBench both have papers whose computations are stochastic, such as MCMC chains and close\-to\-tolerance values\. The Claude Code CLI does not expose a temperature parameter we can pin\. The aggregate cheating and faithfulness orderings are robust under repeated runs in our internal checks\. Individual per\-cell faithfulness scores from a single judge pass can vary substantially, as noted in[§˜B\.3](https://arxiv.org/html/2607.02931#A2.SS3)\. Multi\-run averages are a clear quality improvement for future versions of the evaluation\.

Paper\-only upper bound\.Paper\-only replication is bounded by what the paper and the benchmark provide\. Some papers depend on data the benchmark does not include, such as raw simulation snapshots or redacted intermediate tables\. On those papers an honest agent reaches a low score\. An agent that fabricates a plausible value can score above zero on the same task\. The fable\_mps case in[§˜F\.3](https://arxiv.org/html/2607.02931#A6.SS3)is one example\. A futureVeritascould mark a claim as data\-required and route it differently when the inputs do not support a genuine attempt\.

## References

- \[1\]S\. Ansari\(2026\)Compound deception in elite peer review: a failure mode taxonomy of 100 fabricated citations at NeurIPS 2025\.arXiv preprint arXiv:2602\.05930\.External Links:2602\.05930Cited by:[§1](https://arxiv.org/html/2607.02931#S1.p1.1)\.
- \[2\]Anthropic\(2025\)Claude Code\.Note:[https://github\.com/anthropics/claude\-code](https://github.com/anthropics/claude-code)Cited by:[§1](https://arxiv.org/html/2607.02931#S1.p4.1)\.
- \[3\]X\. Bai, A\. Baumgartner, H\. Sun, A\. Holtzman, and C\. Tan\(2026\)The story is not the science: execution\-grounded evaluation of mechanistic interpretability research\.External Links:2602\.18458,[Link](https://arxiv.org/abs/2602.18458)Cited by:[§B\.3](https://arxiv.org/html/2607.02931#A2.SS3.p1.1),[2nd item](https://arxiv.org/html/2607.02931#S1.I1.i2.p1.1),[§1](https://arxiv.org/html/2607.02931#S1.p2.1),[§1](https://arxiv.org/html/2607.02931#S1.p6.1),[§3\.1](https://arxiv.org/html/2607.02931#S3.SS1.p6.1),[Table 1](https://arxiv.org/html/2607.02931#S3.T1)\.
- \[4\]D\. Baronet al\.\(2024\)PHANGS\-ML: the universal relation between PAH band and optical line ratios across nearby star\-forming galaxies\.arXiv preprint arXiv:2410\.02864\.Cited by:[§F\.4](https://arxiv.org/html/2607.02931#A6.SS4.p1.1)\.
- \[5\]A\. Brodeur, D\. Mikola, N\. Cook,et al\.\(2024\)Mass reproducibility and replicability: a new hope\.IZA Discussion PaperTechnical Report16912,Institute of Labor Economics \(IZA\)\.Cited by:[§1](https://arxiv.org/html/2607.02931#S1.p2.1)\.
- \[6\]A\. Clyburne\-Sherin, X\. Fei, and S\. A\. Green\(2019\)Computational reproducibility via containers in psychology\.Meta\-Psychology3\.External Links:[Document](https://dx.doi.org/10.15626/MP.2018.892)Cited by:[§3\.2](https://arxiv.org/html/2607.02931#S3.SS2.p1.1)\.
- \[7\]N\. Edwards, Y\. Lee, Y\. A\. Mao, Y\. Qin, S\. Schuster, and N\. Kim\(2026\)RExBench: can coding agents autonomously implement AI research extensions?\.InACL,Cited by:[§4](https://arxiv.org/html/2607.02931#S4.p4.1)\.
- \[8\]Google\(2025\)Gemini CLI\.Note:[https://github\.com/google\-gemini/gemini\-cli](https://github.com/google-gemini/gemini-cli)Cited by:[§1](https://arxiv.org/html/2607.02931#S1.p4.1)\.
- \[9\]C\. Hu, L\. Zhang, Y\. Lim, A\. Wadhwani, A\. Peters, and D\. Kang\(2025\)REPRO\-Bench: can agentic AI systems assess the reproducibility of social science research?\.InFindings of ACL,Cited by:[§1](https://arxiv.org/html/2607.02931#S1.p2.1),[§4](https://arxiv.org/html/2607.02931#S4.p1.1)\.
- \[10\]ICLR 2026 Program Chairs\(2026\)A retrospective on the ICLR 2026 review process\.Note:[https://blog\.iclr\.cc/2026/03/31/a\-retrospective\-on\-the\-iclr\-2026\-review\-process/](https://blog.iclr.cc/2026/03/31/a-retrospective-on-the-iclr-2026-review-process/)Accessed: 24 May 2026Cited by:[§1](https://arxiv.org/html/2607.02931#S1.p1.1)\.
- \[11\]C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. R\. Narasimhan\(2024\)SWE\-bench: can language models resolve real\-world GitHub issues?\.InICLR,External Links:2310\.06770Cited by:[§1](https://arxiv.org/html/2607.02931#S1.p4.1)\.
- \[12\]S\. Kapoor, B\. Stroebl, P\. Kirgis,et al\.\(2026\)Holistic agent leaderboard: the missing infrastructure for AI agent evaluation\.InICLR,External Links:2510\.11977Cited by:[§1](https://arxiv.org/html/2607.02931#S1.p4.1),[§3\.1](https://arxiv.org/html/2607.02931#S3.SS1.p3.1),[§3\.2](https://arxiv.org/html/2607.02931#S3.SS2.p1.1)\.
- \[13\]B\. Kohler, D\. Zollikofer, J\. Einsiedler, A\. Hoyle, and E\. Ash\(2026\)Read the paper, write the code: agentic reproduction of social\-science results\.arXiv preprint arXiv:2604\.21965\.Cited by:[§B\.2](https://arxiv.org/html/2607.02931#A2.SS2.p1.1),[§B\.4](https://arxiv.org/html/2607.02931#A2.SS4.p1.1),[Appendix C](https://arxiv.org/html/2607.02931#A3.p1.1),[§F\.2](https://arxiv.org/html/2607.02931#A6.SS2.p2.1),[2nd item](https://arxiv.org/html/2607.02931#S1.I1.i2.p1.1),[§1](https://arxiv.org/html/2607.02931#S1.p2.1),[§1](https://arxiv.org/html/2607.02931#S1.p3.1),[§1](https://arxiv.org/html/2607.02931#S1.p6.1),[§3\.1](https://arxiv.org/html/2607.02931#S3.SS1.p5.1),[§3\.1](https://arxiv.org/html/2607.02931#S3.SS1.p7.1),[§3\.2](https://arxiv.org/html/2607.02931#S3.SS2.p3.1),[§3\.3](https://arxiv.org/html/2607.02931#S3.SS3.p4.1),[Table 1](https://arxiv.org/html/2607.02931#S3.T1),[Table 2](https://arxiv.org/html/2607.02931#S3.T2),[§4](https://arxiv.org/html/2607.02931#S4.p3.1),[Limitations](https://arxiv.org/html/2607.02931#Sx1.p2.1)\.
- \[14\]S\. Kruket al\.\(2023\)The impact of satellite trails on Hubble Space Telescope observations\.Nature Astronomy\.External Links:[Document](https://dx.doi.org/10.1038/s41550-023-01903-3)Cited by:[§F\.1](https://arxiv.org/html/2607.02931#A6.SS1.p1.1)\.
- \[15\]S\. Martin\-Alvarez, V\. Iršič, S\. Koudmani, M\. A\. Bourne, L\. Bigwood, and D\. Sijacki\(2024\)Stirring the cosmic pot: how black hole feedback shapes the matter power spectrum in the FABLE simulations\.arXiv preprint arXiv:2407\.18349\.Cited by:[§F\.3](https://arxiv.org/html/2607.02931#A6.SS3.p1.1)\.
- \[16\]D\. Merkel\(2014\)Docker: lightweight Linux containers for consistent development and deployment\.Linux Journal2014\(239\)\.Cited by:[footnote 1](https://arxiv.org/html/2607.02931#footnote1)\.
- \[17\]M\. A\. Merrill, A\. G\. Shaw, N\. Carlini, B\. Li, H\. Raj,et al\.\(2026\)Terminal\-bench: benchmarking agents on hard, realistic tasks in command line interfaces\.arXiv preprint arXiv:2601\.11868\.Cited by:[§1](https://arxiv.org/html/2607.02931#S1.p4.1)\.
- \[18\]NeurIPS\(2020\)Fact sheet: 34th annual conference on neural information processing systems\.Note:[https://neurips\.cc/media/Press/NeurIPS\_2020\-Fact\_Sheet\.pdf](https://neurips.cc/media/Press/NeurIPS_2020-Fact_Sheet.pdf)Accessed: 24 May 2026Cited by:[§1](https://arxiv.org/html/2607.02931#S1.p1.1)\.
- \[19\]NeurIPS\(2025\)NeurIPS 2025 fact sheet\.Note:[https://media\.neurips\.cc/Conferences/NeurIPS2025/press/NeurIPS2025\-Fact\_Sheet\.pdf](https://media.neurips.cc/Conferences/NeurIPS2025/press/NeurIPS2025-Fact_Sheet.pdf)Accessed: 24 May 2026Cited by:[§1](https://arxiv.org/html/2607.02931#S1.p1.1)\.
- \[20\]B\. Nguyen, D\. Soós, Q\. Ma, R\. R\. Obadage, Z\. Ranjan, S\. Koneru, A\. Szabelska, A\. Gill, T\. M\. Errington, S\. Nematova, S\. Rajtmajer, J\. Wu, and M\. Jiang\(2026\)ReplicatorBench: benchmarking LLM agents for replicability in social and behavioral sciences\.arXiv preprint arXiv:2602\.11354\.Cited by:[§4](https://arxiv.org/html/2607.02931#S4.p4.1)\.
- \[21\]Open Science Collaboration\(2015\)Estimating the reproducibility of psychological science\.Science349\(6251\),pp\. aac4716\.Cited by:[§1](https://arxiv.org/html/2607.02931#S1.p2.1)\.
- \[22\]OpenAI\(2025\)Codex CLI\.Note:[https://github\.com/openai/codex](https://github.com/openai/codex)Cited by:[§1](https://arxiv.org/html/2607.02931#S1.p4.1)\.
- \[23\]M\. Rizhko and J\. S\. Bloom\(2024\)AstroM3: a self\-supervised multimodal model for astronomy\.arXiv preprint arXiv:2411\.08842\.Cited by:[§F\.2](https://arxiv.org/html/2607.02931#A6.SS2.p1.1)\.
- \[24\]M\. Seo, J\. Baek, S\. Lee, and S\. J\. Hwang\(2026\)Paper2Code: automating code generation from scientific papers in machine learning\.InICLR,Cited by:[§4](https://arxiv.org/html/2607.02931#S4.p2.1)\.
- \[25\]Z\. S\. Siegel, S\. Kapoor, N\. Nadgir, B\. Stroebl, and A\. Narayanan\(2025\)CORE\-Bench: fostering the credibility of published research through a computational reproducibility agent benchmark\.Transactions on Machine Learning Research\.Cited by:[§1](https://arxiv.org/html/2607.02931#S1.p2.1),[§1](https://arxiv.org/html/2607.02931#S1.p6.1),[§3\.2](https://arxiv.org/html/2607.02931#S3.SS2.p1.1),[§3](https://arxiv.org/html/2607.02931#S3.p1.1),[§4](https://arxiv.org/html/2607.02931#S4.p1.1)\.
- \[26\]G\. Starace, O\. Jaffe, D\. Sherburn, J\. Aung, J\. S\. Chan, L\. Maksin, R\. Dias, E\. Mays, B\. Kinsella, W\. Thompson, J\. Heidecke, A\. Glaese, and T\. Patwardhan\(2025\)PaperBench: evaluating AI’s ability to replicate AI research\.arXiv preprint arXiv:2504\.01848\.Cited by:[§1](https://arxiv.org/html/2607.02931#S1.p2.1),[§4](https://arxiv.org/html/2607.02931#S4.p1.1),[Limitations](https://arxiv.org/html/2607.02931#Sx1.p3.1)\.
- \[27\]W\. H\. Walters and E\. I\. Wilder\(2023\)Fabrication and errors in the bibliographic citations generated by ChatGPT\.Scientific Reports13\(1\),pp\. 14045\.External Links:[Document](https://dx.doi.org/10.1038/s41598-023-41032-5)Cited by:[§1](https://arxiv.org/html/2607.02931#S1.p1.1)\.
- \[28\]Z\. Wang, F\. Bai, Z\. Luo, J\. Su, K\. Sun, X\. Yu, J\. Liu, K\. Zhou, C\. Cardie, M\. Dredze, E\. P\. Xing, and Z\. Hu\(2026\)FIRE\-Bench: evaluating agents on the rediscovery of scientific insights\.arXiv preprint arXiv:2602\.02905\.Cited by:[§4](https://arxiv.org/html/2607.02931#S4.p4.1)\.
- \[29\]Q\. Wei, S\. Holt, J\. Yang, M\. Wulfmeier, and M\. van der Schaar\(2025\)The AI imperative: scaling high\-quality peer review in machine learning\.arXiv preprint arXiv:2506\.08134\.Cited by:[§1](https://arxiv.org/html/2607.02931#S1.p1.1)\.
- \[30\]C\. Ye, S\. Yuan, S\. Cooray, S\. Dillmann, I\. L\. V\. Roque, D\. Baron, P\. Frank, S\. Martin\-Alvarez, N\. Koblischke, F\. J\. Qu, D\. Yang, R\. Wechsler, and I\. Ciucă\(2025\)ReplicationBench: can AI agents replicate astrophysics research papers?\.arXiv preprint arXiv:2510\.24591\.Cited by:[Appendix G](https://arxiv.org/html/2607.02931#A7.p1.1),[§1](https://arxiv.org/html/2607.02931#S1.p2.1),[§1](https://arxiv.org/html/2607.02931#S1.p6.1),[§3\.3](https://arxiv.org/html/2607.02931#S3.SS3.p1.1),[§3\.3](https://arxiv.org/html/2607.02931#S3.SS3.p8.1),[§3](https://arxiv.org/html/2607.02931#S3.p1.1),[§4](https://arxiv.org/html/2607.02931#S4.p1.1)\.
- \[31\]L\. Zhang, T\. Xia, J\. Piao, L\. Cui, and Y\. Li\(2026\)PaperRepro: automated computational reproducibility assessment for social science papers\.arXiv preprint arXiv:2603\.00058\.Cited by:[§1](https://arxiv.org/html/2607.02931#S1.p2.1),[§4](https://arxiv.org/html/2607.02931#S4.p3.1)\.
- \[32\]X\. Zhao, Z\. Sang, Y\. Li, Q\. Shi, W\. Zhao, S\. Wang, D\. Zhang, X\. Han, Z\. Liu, and M\. Sun\(2026\)AutoReproduce: automatic AI experiment reproduction with paper lineage\.InACL,Cited by:[§1](https://arxiv.org/html/2607.02931#S1.p3.1),[§4](https://arxiv.org/html/2607.02931#S4.p2.1)\.

## Appendix AEnvironment and Runtime

We describe the runtime environment shared by all systems within each benchmark\. The environment is identical acrossVeritas,Zeroshot, andRetrywithin a given benchmark, so any score difference reflects the agent and not the setup\. The two benchmarks ran on different machines and runtime modes, which we document below\.

ReplicationBench\.ReplicationBench experiments ran inside a single shared Docker image,ghcr\.io/chicagohai/veritas\(digeste00fd604, built 2026\-06\-04 fromVeritasmain\)\. The image contains a CUDA 12\.5 base with Python 3\.12 and R 4\.2 plus the scientific stack required by astrophysics papers\. It also includes LaTeX for figure rendering and the Claude Code CLI\. TheVeritaspipeline source is installed under/app/\.venvand the catalog of scientific\-computing skills is installed at/workspace/veritas\-skills/\. For each paper, the masked manuscript is mounted at/inputs/paper, the dataset at/inputs/data, and, in full mode, the authors’ code repository at/inputs/repo\(read\-only\)\. The baseline entrypoint unsets theVeritasenvironment variables \(PYTHONPATH,VIRTUAL\_ENV,VERITAS\_WORKSPACE\)\. The baselines therefore see only the benchmark inputs and the Claude Code CLI, with no access to theVeritaspipeline\. All ReplicationBench cells ran on a single host\.

CORE\-Bench\.CORE\-Bench experiments ran in host mode on a separate machine, using a shared conda environment that mirrors the dependencies of the ReplicationBench Docker image \(Claude Code CLI, Python 3\.12, the same R packages that the CORE\-Bench capsules call into\)\. For each capsule, the prepared capsule repository is mounted as/inputs/repo\(read\-only\), and outputs are written to a cell\-local writable directory\. Baselines andVeritasshare the same host environment and the same per\-capsule repository copy; the only difference is the pipeline that surrounds the Claude Code session\.

Reproducibility\.The Dockerfile in theVeritasrepository pins all dependencies\. We do not set random seeds; the Claude Code CLI does not expose a temperature parameter, and ReplicationBench has inherent run\-to\-run variance on some papers from MCMC chains and close\-to\-tolerance values\. We report single\-run results and note this caveat where it matters\. Per\-phase wall\-clock limits we used on CORE\-Bench are 5400 seconds for the replication phase and 1200 seconds for verification; the baselines use a per\-cell limit derived frommax\_attemptsmultiplied by the per\-attempt limit\.

## Appendix BEvaluation Metric Details

We describe how each evaluation score is computed\. Our adaptations of the cited methods are minimal\. We reuse the source prompts, taxonomy, and scoring formulas verbatim, and only configure the per\-benchmark inputs that the source methods themselves expose\.

### B\.1Native Benchmark Scoring

CORE\-Bench\.CORE\-Bench’s official scorer compares the agent’s reported answer to the reference for each task question\. Numeric answers are matched against a 95% prediction interval built from three reference runs\. Non\-numeric answers are matched by exact string equality after whitespace trimming and case\-folding\. A capsule passes only when all of its task questions match\. The per\-system score is the fraction of capsules that pass\.

ReplicationBench\.ReplicationBench’s official scorer checks every value in a task’s expected output against the author\-supplied tolerance\. A task is credited only when all values in its output fall within tolerance\. The per\-system score is the fraction of tasks credited, averaged uniformly\. We use the canonical ReplicationBench scorer with one deterministic patch: a shape\-coercion helper that maps a flat or positionally\-keyed dictionary back to the paper’s expected list shape before grading\. The patch resolves a parser gap that caused some correct\-in\-tolerance values to be scored as shape mismatches\. It is alignment\-only and does not re\-grade any answer\.

Cheating correction\.For both benchmarks, the main text reports a cheating\-corrected version of the native score: a capsule or task whose trajectory was flagged as a cheat by the detector \([§˜B\.2](https://arxiv.org/html/2607.02931#A2.SS2)\) is forced to a fail, kept in the denominator\. The flag is the detector’s verdict; we do not curate the list by hand\.

### B\.2Cheating Metric

We adapt the cheating detector ofKohleret al\.\[[13](https://arxiv.org/html/2607.02931#bib.bib14)\]\. The detector combines a deterministic path classifier with an LLM judge\.

Path classifier\.For each accessed path in the trajectory, the classifier labels it allowed or forbidden by container location\. Anything under the agent’s writable workspace \(/workspace,/output,/tmp\) and the standard system prefixes is allowed\. The provided inputs at/inputs/data,/inputs/repo, and/inputs/paperare also allowed\. Everything else is treated as a candidate for review\. Per\-benchmark configuration registers external sources that are forbidden regardless of location\. For ReplicationBench the registered source is theAstroMLCore/AstroM3\-CLIP\-\*HuggingFace hub used by the astm3 paper\. For CORE\-Bench the configuration follows Kohler et al\.’s minimal default, with no per\-capsule entries\.

Judge\.The LLM judge reads the task instructions and the action\-first event trace of the trajectory\. It returns a structured verdict with two parts\. The first is a breach type from Kohler’s four\-value vocabulary: external result lookup, forbidden paper access, forbidden code access, and a residualothercategory\. The second is a severity level from Kohler’s five\-level scale running fromcleantosevere\_violation\. The judge is also given a fixed not\-a\-breach checklist taken verbatim from Kohler et al\. The checklist excludes wrong\-method implementations, approximate replications, and tuning constants from being flagged\. We run the judge withgpt\-5\.4and three\-vote majority\. The identity of the system that produced the trajectory is hidden from the judge\.

Verdict\.A trajectory counts as a cheat at the two highest severity levels \(likely\_violationandsevere\_violation\)\. Lower severities either flag the trajectory for review or mark it clean\. Kohler et al\. also report a separate hardcoding mode that judges whether a reported value was written as a literal with no producing computation\. We use access mode for both benchmarks and discuss the hardcoding mode in[Appendix˜C](https://arxiv.org/html/2607.02931#A3)\.

### B\.3Faithfulness Metric

We adapt the execution\-grounded checklist of MechEvalAgent\[[3](https://arxiv.org/html/2607.02931#bib.bib36)\]\. The checklist judges a trajectory on five binary items\.

- •C1\.The essential code in the final pipeline ran without error\. A probe that failed and was superseded by a working version does not count against C1, because the judge scores the version relied upon\.
- •C2\.The implementation matches the paper’s method, judged against the trajectory\.
- •CS1\.The reported answers are supported by the run’s produced evidence, which the judge sees as saved outputs, rendered figures, and result files in addition to the event trace\.
- •DE1\.Each reported value was produced by the procedure that its question implies\. The judge infers the procedure from the question text; legitimate procedures are computation, figure reading, or running the provided code\.
- •DE3\.No reported value is hallucinated\. A placeholder or self\-admitted fabrication fails DE3 even when it appears in the answers file\.

The faithfulness index is the fraction of these five items that pass, averaged across a system’s trajectories\. Items rated not applicable drop out of both numerator and denominator\. A trajectory whose run failed for infrastructure reasons, such as a permission error on the provided inputs, is excluded from the average\. We do not score it zero\. MechEvalAgent also reports four secondary items \(no redundant code, no irrelevant code, plan followed, uncertainty reported\) which we compute but do not put in the index, following MechEvalAgent’s own grouping\.

We run the judge withgpt\-5\.4\-miniand a single pass, matching the cited work\. The judge is system\-blind: it sees the paper and the input mode, never the system identity\.

### B\.4Fidelity Metric \(ReplicationBench only\)

We adapt the percent\-difference grader ofKohleret al\.\[[13](https://arxiv.org/html/2607.02931#bib.bib14)\], which applies no error tolerance\. For each reported value paired with the paper’s value, the grader computes the percent difference and maps it to a letter grade\. The thresholds are A under 2%, B under 20%, C under 40%, and D under 60%\. E covers above 60% or sign\-flipped values\. F is reserved for missing or non\-numeric values\. Near\-zero originals \(absolute value under10−310^\{\-3\}\) use absolute thresholds of 0\.002, 0\.02, 0\.05, and 0\.1 for the A through D bands\. Letters convert to points on a 0 to 5 scale \(A is 5, F is 0\), and the per\-system GPA is the value\-weighted mean\. We zero a trajectory’s values when the detector flags it as a cheat, and keep them in the denominator\. The grader is deterministic and does not use an LLM\.

We explain in[Appendix˜C](https://arxiv.org/html/2607.02931#A3)why this grader does not transfer to CORE\-Bench\.

## Appendix CWhy Fidelity Is Not Reported for CORE\-Bench

We adopt the percent\-difference grader ofKohleret al\.\[[13](https://arxiv.org/html/2607.02931#bib.bib14)\]for ReplicationBench but not for CORE\-Bench\. The decision is data\-driven\. Three properties of CORE\-Bench’s answers make the grader uninformative on this benchmark\.

Non\-numeric answers\.About 39% of CORE\-Bench Hard’s reference answers are non\-numeric\. We sampled the 86 answers across the three systems in our batch and counted 9 non\-numeric out of 23 in the first 10 files\. Non\-numeric answers include category labels, feature names, gene identifiers, and axis labels read from figures\. The percent\-difference grader cannot score any of these and assigns them F by definition\. Roughly two\-fifths of the score therefore reflects only the grader’s incompatibility with the answer type\.

Saturation on the numeric remainder\.On the roughly 61% of answers that are numeric, CORE\-Bench’s reference answers are either exact within its 95% prediction interval or absent\. The percent\-difference grader therefore returns A for almost every value all three systems produce\. The numeric\-only GPA on our batch is approximately 4\.9 across all three systems, well above 4\.5 which is the band cutoff for an A\. The grader does not separate the systems on the numeric part of the answer set\.

No\-answer failures are invisible\.The grader scores only answers that were produced\. A capsule where the agent produced no answer at all contributes nothing to the GPA\. No\-answer outcomes are the main source of variance on CORE\-Bench Hard in our batch\. CORE\-Bench’s native binary pass rate counts a no\-answer capsule as a fail\. The percent\-difference grader does not\. The grader would systematically miss the dimension along which the systems separate\.

Conclusion\.The percent\-difference grader’s design assumes a continuous numeric answer with a paper\-stated reference value and a meaningful tolerance, which is the ReplicationBench format\. CORE\-Bench’s heterogeneous answers and binary native scoring already cover the final\-answer comparison\. We report CORE\-Bench’s native pass rate as the final\-answer metric on CORE\-Bench, the cheating and faithfulness scores as the trajectory metrics on both benchmarks, and the percent\-difference grader as the fidelity metric on ReplicationBench only\.

A note on the hardcoding mode\.Kohler et al\. report two separate detector modes: the access mode described in[§˜B\.2](https://arxiv.org/html/2607.02931#A2.SS2), and a hardcoding mode that judges whether a reported value was written as a literal with no producing computation\. We report only the access mode in the main text\. On ReplicationBench the two modes agreed on every confirmed cheat, with no trajectory flagged by hardcoding alone, so the hardcoding mode added no signal\. On CORE\-Bench the hardcoding mode is structurally ill\-suited\. The benchmark’s paradigm is to run the authors’ provided capsule code and report its outputs, so a metric value can legitimately appear in the agent’s authored code without any computation behind it in the agent’s own code\. Applying the hardcoding mode unmodified would flag legitimate run\-provided\-code outputs\. Reporting the access mode on both benchmarks keeps the cheating metric consistent and faithful to Kohler et al\.’s separation of the two assessments\.

## Appendix DTheσ\\sigma\-Corrected CORE\-Bench Score

We apply a small re\-scoring rule to CORE\-Bench Hard that targets an edge case in the official scorer\. The rule is uniform across all three systems and does not depend on which system produced the answer\. On the batch in this paper, the rule flips fourVeritascapsules from no match to match and zeroZeroshotorRetrycapsules\. Under the official scorerVeritasreaches 88\.9%; under the re\-scored rule, 97\.8%\. We document the rule and the four re\-credited cases here\.

The official scorer’s edge case\.CORE\-Bench scores each numeric question by building a 95% prediction interval from three ground\-truth replicates and checking whether the reported value falls inside\. When the three ground\-truth replicates are identical, the standard deviation is zero and the interval collapses to the single point at the mean\. A correct answer that differs by a single floating\-point unit, or by a small amount of unseeded stochastic variation, is then marked wrong by the bit\-exact check\. CORE\-Bench’s own scoring code documents this as a known limitation\.

Theσ\\sigma\-corrected rule\.For each numeric question that the official scorer marked wrong and whose three ground\-truth replicates have zero variance, theσ\\sigma\-corrected scorer accepts the answer when the relative error against the mean is at most 1%\. The 1% threshold matches the relative tolerance that CORE\-Bench’s claim metadata itself uses for the same questions\. A capsule then passes only when every numeric question and every non\-numeric question is correct under this rule, exactly as in the official score\. We apply the rule identically toZeroshot,Retry, andVeritas; the rule does not look at which system produced the answer\. On our batch, the rule re\-credits fourVeritascapsules and no baseline capsules, so theZeroshotandRetrypass rates are unchanged at 80\.0% and 84\.4% respectively\.

The four re\-credited capsules\.The rule flips fourVeritascapsules from no match to match\.

- •Capsule 8536428\. Eight metrics; six are bit\-identical to the ground truth and two differ in the sixteenth decimal place \(for example F1 of 0\.9403284422685527 against 0\.9403284422685528\), a difference of one unit in the last place of a 64\-bit float\.
- •Capsule 3301293\. Test RMSE 26\.21216 against 26\.21204; relative difference4\.6×10−64\.6\\times 10^\{\-6\}\.
- •Capsule 5507257\. Accuracy 96\.0816 against 96\.1250; relative difference4\.5×10−44\.5\\times 10^\{\-4\}\. The underlying training is stochastic and unseeded\.
- •Capsule 9660931\. HCR\-Net accuracy 0\.99833 against 0\.99900; relative difference6\.7×10−46\.7\\times 10^\{\-4\}\. The underlying training is stochastic and unseeded\.

Every difference is within 0\.07% of the paper value and well inside the claim’s own stated tolerance\.Veritas’s own verifier, an independent deterministic grader that uses the claim’s relative tolerance, judged all four as match before the comparison with the official scorer\.

Cross\-system comparability\.The 97\.8% number is not directly comparable to systems reported under the official scorer unless those systems are re\-scored under the same rule\. When comparing to a published baseline that uses the official scorer \(such as the HAL leaderboard entry\), the comparableVeritasnumber is 88\.9%\. The rule is detector\-free, applies only to numeric questions with zero\-variance ground truth, and does not touch any other capsule\. In particular, capsule 6003668 remains a genuine fail: the produced value differs from the paper value by more than the claim’s tolerance\.

## Appendix EPer\-Cell Results

We list per\-cell scores for both benchmarks\. Cheating flags follow the detector’s verdict described in[§˜B\.2](https://arxiv.org/html/2607.02931#A2.SS2)and reflect a likely or severe violation; we do not include the lower\-severity suspicious level\.

### E\.1CORE\-Bench

[Table˜3](https://arxiv.org/html/2607.02931#A5.T3)lists the per\-capsule pass status \(P\), faithfulness index \(F\), and cheating flag for each capsule and each system on the 45\-capsule CORE\-Bench Hard public test set\. A check denotes a passing capsule\. A cross denotes a no match\. A dash denotes a missing answer\. The dagger after a faithfulness value flags a trajectory that the cheating detector marked as a cheat\.

ZeroshotRetryVeritasCapsulePFPFPF0504157✓1\.00✓1\.00✓1\.000851068✓0\.40✓0\.40✓1\.000921079✓1\.00✓0\.60✓1\.001175539✓1\.00✓1\.00✓1\.001394704✓1\.00✓1\.00✓1\.001624349✓1\.00✓1\.00✓1\.001724988✓0\.80✓0\.80✓1\.001900704✓0\.80✓1\.00✓1\.002345790✓1\.00✓1\.00✓1\.002414499✓1\.00✓0\.60✓1\.002708693✓1\.00✓1\.00✓1\.002804717✓1\.00✓1\.00✓1\.002816027✓0\.60✓1\.00✓1\.003262218✓1\.00✓1\.00✓1\.003301293✓1\.00✓1\.00×\\times1\.003418007✓1\.00––✓1\.003449234✓1\.00✓1\.00✓1\.003593259✓1\.00✓1\.00✓1\.003639589✓1\.00✓1\.00✓1\.003821950×\\times1\.00†×\\times1\.00✓1\.003849634✓1\.00✓1\.00✓1\.004180912✓1\.00✓0\.40†✓1\.004252248×\\times1\.00×\\times1\.00✓1\.00
ZeroshotRetryVeritasCapsulePFPFPF4299879✓1\.00✓0\.80✓1\.004671827✓1\.00✓1\.00✓1\.004728591✓1\.00✓1\.00✓1\.004933686✓1\.00✓1\.00✓1\.005136217–1\.00✓0\.80✓0\.605507257×\\times1\.00×\\times1\.00×\\times1\.006003668–0\.00✓0\.60–0\.676049678✓1\.00✓0\.40✓1\.007186268✓1\.00✓1\.00✓1\.007716865✓1\.00✓1\.00✓1\.008234136✓0\.80✓1\.00✓1\.008536428×\\times1\.00×\\times1\.00×\\times1\.008807709✓1\.00✓1\.00✓1\.009052293✓1\.00✓1\.00✓1\.009054015✓1\.00✓1\.00✓1\.009137200✓1\.00✓1\.00✓1\.009240688×\\times0\.40✓1\.00✓0\.609641396✓1\.00✓1\.00✓1\.009660931×\\times0\.80–0\.00×\\times1\.009670283✓1\.00✓1\.00✓1\.009832712✓1\.00✓1\.00✓1\.009911222✓1\.00†✓0\.80✓1\.00

Table 3:CORE\-Bench per\-capsule results across all 45 capsules\. P is the pass status under CORE\-Bench’s official scorer, before theσ\\sigma\-correction of[Appendix˜D](https://arxiv.org/html/2607.02931#A4)\. F is faithfulness\. The dagger flags a trajectory the detector marked as a cheat\.
### E\.2ReplicationBench

[Table˜4](https://arxiv.org/html/2607.02931#A5.T4)and[Table˜5](https://arxiv.org/html/2607.02931#A5.T5)list per\-cell match rates after the cheating correction, paper\-only mode and full mode respectively\. We use ReplicationBench’s own scorer with the deterministic shape\-coercion fix from[§˜B\.1](https://arxiv.org/html/2607.02931#A2.SS1)\. The dagger flags a cell whose trajectory was marked as a cheat\.[Table˜6](https://arxiv.org/html/2607.02931#A5.T6)reports the per\-cell faithfulness and fidelity forVeritas; the corresponding numbers for the baselines are in our supplementary materials\.

Table 4:ReplicationBench paper\-only per\-paper match rate after the cheating correction\. The dagger flags a trajectory the detector marked as a cheat\.Table 5:ReplicationBench full\-mode per\-paper match rate after the cheating correction\. The mean is computed over the 7 cells reported here; the headline match rate in[Table˜2](https://arxiv.org/html/2607.02931#S3.T2)is the per\-task pass rate over all expected outputs in these papers\.Table 6:Per\-cellVeritasfaithfulness index and fidelity grade\. Faithfulness single\-pass values for cells where the score is below 1\.0 reflect cited\-consistent single\-pass variance discussed in[§˜B\.3](https://arxiv.org/html/2607.02931#A2.SS3)\.

## Appendix FReplicationBench Case Studies

We walk through four trajectories that illustrate whereVeritasbehaves well, where it falls short, and what its outputs reveal even when the reported score is zero\. The selection focuses onVeritas’s behavior so the reader can see what would change with a stronger pipeline\. Baseline failures appear when they make the contrast concrete\.

### F\.1Hubble satellite trails: a second pass that rereads the question

Thehubble\_trailspaper\[[14](https://arxiv.org/html/2607.02931#bib.bib40)\]asks how often passing satellites cross Hubble Space Telescope images, and whether the problem is growing as more satellites reach orbit\. The authors scanned two decades of Hubble exposures, marked the ones with a satellite streak, and reported that about2\.7%2\.7\\%of exposures are affected and that the affected fraction has risen over time\. The benchmark asks each system to reproduce the trained streak classifier, the contamination fractions, the per\-camera chance of a streak, and the increase in the affected fraction between the early and late years of the archive\.

The increase task is quietly ambiguous\. The paper’s target values,\[0\.043,0\.02\]\[0\.043,\\,0\.02\], are the fractions the streaks rose*to*in the recent period \(from2\.82\.8to4\.3%4\.3\\%for one camera and1\.11\.1to2\.0%2\.0\\%for the other\), not the size of the jump\. Each system reads the word “increase” differently\. TheZeroshotbaseline subtracts the early fraction from the late one and reports the gap,\[0\.015,0\.008\]\[0\.015,\\,0\.008\]; theRetrybaseline divides that gap by the starting value and reports a relative growth,\[0\.53,0\.73\]\[0\.53,\\,0\.73\]; both miss\.Veritascomputes the same intermediate fractions, but its separate verification stage rereads its own output against the wording of the paper, recognizes that the target is the level the fraction reached rather than the change, and reports\[0\.0431,0\.0197\]\[0\.0431,\\,0\.0197\]—a match\. It does the rest of the work directly as well: it trains the image classifier from scratch for twenty\-two passes over the data with the paper’s exact settings, its validation accuracy climbing from0\.540\.54to0\.930\.93, and reaches\[0\.934,0\.976,0\.879,0\.925\]\[0\.934,\\,0\.976,\\,0\.879,\\,0\.925\]against the paper’s\[0\.938,0\.975,0\.89,0\.931\]\[0\.938,\\,0\.975,\\,0\.89,\\,0\.931\], where theZeroshotrun’s recall lands too low to count\. Every repairVeritasmade along the way was mechanical—renaming a data file, removing leftover notebook setup commands, loading an older saved\-model format—with no change to any model, setting, or reported number\. The result is a clean seven of seven when the authors’ code is available, against0\.710\.71and0\.860\.86for the baselines, and the single best score in the paper\-only setting\. The advantage comes from a second pass that checks the answer against the paper rather than shipping the first plausible reading of the task\.

### F\.2astm3: An Instructed External Load Counted as a Cheat

The astm3 paper\[[23](https://arxiv.org/html/2607.02931#bib.bib37)\]builds a CLIP\-style classifier for variable stars, trained on three input streams \(photometry, spectroscopy, and metadata\)\. In full mode the agent receives the paper, the dataset, and the authors’ code repository\. The task instructions ask the agent to load a published pre\-trained AstroM3 model and report classification accuracy\.

Veritasread the instruction literally\. The pipeline ranhf\_hub\_downloadfor the authors’AstroMLCore/AstroM3\-CLIP\-photoand\-allcheckpoints from HuggingFace, then ran inference on the benchmark’s evaluation set and reported the resulting accuracy\. The reported values were computed from a real inference pass with the paper’s value masked\.Veritasdid not write a literal number; it computed one\. The cheating detector flagged the trajectory at the highest severity because the loaded checkpoint is an externally hosted author artifact, and external author artifacts are forbidden under the access rule ofKohleret al\.\[[13](https://arxiv.org/html/2607.02931#bib.bib14)\]even when the task instructions ask for them\.

We count this trajectory as a cheat in our reported numbers\. The flag is strict but consistent\. The takeaway concerns the gap between the task\-instruction surface and the detector’s rule on instructed\-use cases\. A futureVeritascould detect when a task instruction directs the agent to load an external artifact and mark the trajectory for review\. The benchmark or the user can then decide whether the instructed access counts\.

### F\.3fable\_mps: A Veritas Failure That Reveals a Paper\-Only Upper Bound

The fable\_mps paper\[[15](https://arxiv.org/html/2607.02931#bib.bib38)\]studies how supermassive black holes reshape the cosmic matter distribution in the FABLE galaxy formation simulations\. The reported quantity is the matter power spectrum, a summary of how lumpy or uniform matter is at different scales, compared between a full\-physics run and a dark\-matter\-only baseline\. The benchmark task asks the agent to recompute several of these power\-spectrum measurements from the paper alone\.

Veritasreads the density cubes the benchmark provides, computes power spectra with the standard FFT\-based pipeline, and matches the paper’s full\-physics over dark\-matter ratios within tolerance on two of the tasks\. On the tasks that ask for halo\-only power spectra,Veritasreportsnot\_attemptedbecause the benchmark provides only the projected density cubes; the raw particle snapshots and halo catalogs that those measurements require are not in the provided data\.Veritas’s final score on this paper is 0\.25 of the per\-task pass rate\.

TheZeroshotbaseline, faced with the same missing data, wrote1\.0as the answer for the halo\-only tasks\. The number is a common placeholder and the detector’s access mode does not flag it\. The cheating correction we apply to the baseline’s score is driven only by access flags, so the baseline retains credit for a fabricated value that is coincidentally close to a reference value\. We discuss this gap in[Limitations](https://arxiv.org/html/2607.02931#Sx1); a hardcoding\-aware extension to the cheating metric would address it\.

The implication forVeritasis that paper\-only replication has an upper bound determined by what the benchmark provides\. The agent is not the limiting factor here\.Veritas’s transparency about the upper bound is the right behavior\. Future work could letVeritas’s analyze phase flag a claim asdata\_requiredwhen its evidence depends on artifacts that are not present in the inputs\. The verifier can then set the claim aside and not count it as a missed task\.

### F\.4phangs\_PAHs: a score of zero that hides a near\-complete result

Thephangs\_PAHspaper\[[4](https://arxiv.org/html/2607.02931#bib.bib41)\]studies dust and gas in nineteen nearby galaxies\. Small soot\-like dust grains glow in the mid\-infrared and ionized gas glows in optical lines, and across roughly seven hundred thousand image regions the authors find that the two glows track each other along one tight straight line per galaxy, with a slope near0\.20\.2\. The benchmark provides the full image set, so this is a genuinely reproducible analysis, and all three systems carried it out: each read the provided maps, grouped the regions, and fit a slope for every galaxy\. What stands out is how close they came\. TheRetrybaseline matched eighteen of the nineteen galaxy slopes*and*all nineteen of their uncertainties, missing only a single galaxy; theZeroshotbaseline matched all nineteen slopes; andVeritasmatched eighteen of nineteen slopes and was the only system to identify all five active\-galaxy hosts, which the baselines each undercounted by one\. Yet every system scored exactly zero, the same as a blank answer\.

Two effects produced that zero, and neither is an inability to do the science\. First, the benchmark grades each table all\-or\-nothing: a nineteen\-galaxy table correct everywhere but one place is marked wrong, with no partial credit\. Second, the small misses fell in different places—theRetrybaseline was off on one galaxy’s slope, whileVeritasand theZeroshotbaseline matched the slopes but set the uncertainty on most galaxies a few hundredths too wide, outside the narrow tolerance\.Veritasis the system that made this legible: its own report marks the slope table as a partial result, keeps the matching slopes, and points to the cause, noting that the provided code’s error\-estimation step does not follow the procedure described in the paper\. The lesson is that a reported score of zero can sit on top of an almost\-complete reproduction\. When the gap is a single cell or a shared, well\-understood detail rather than missing data or a wrong method, a system that reports what it got right and where it diverged is more useful than one that returns a confident, unlabeled answer\.

## Appendix GReplicationBench Error Analysis

Lack of persistence\.ReplicationBench attributes the failures of the agents it tested to three modes: a lack of persistence, conceptual or procedural errors, and technical execution failures\[[30](https://arxiv.org/html/2607.02931#bib.bib10)\]\. Lack of persistence is the modeVeritasmost clearly avoids\. It leaves only 6 of the 111 tasks unattempted\. Five of the six are compute\-bound, where the data a task needs is absent from the provided inputs \(fable\_mps and gw\_nsbh,[§˜F\.3](https://arxiv.org/html/2607.02931#A6.SS3)\)\. The faithfulness index of 0\.933 showsVeritasrarely reports a value its run did not compute\. This persistence reflects the design ofVeritas’s replicate phase, which fixes problems as they arise and keeps working through the plan rather than stopping at the first obstacle\.

Conceptual and procedural errors\.This mode persists and accounts for most of the substantive misses\. Most of the 68 misses produce a wrong value rather than no value, and the largest group are methodology errors\. Examples include an outflow velocity defined differently from the paper in MUSE\_outflows and a single\-seed run where astm3 specifies a five\-seed average\.

Technical execution failures\.This mode is also largely absent\.Veritas’s active fixing resolves the environment errors and API drift its fix logs record, so its code runs even where the released setup would not\.

Compute\-driven downsizing\.Some papers report numbers from computations that are very expensive\. When that happens,Veritasdownscales the run just so it can finish, and the smaller run can produce a different number\. The clearest case is abacus \(difficulty 8 and 9\), an HPC N\-body code that the paper runs in tuned C/CUDA on GPUs\.Veritasre\-implements the force method in pure Python and its sanity checks pass, but it cuts the particle count and grid by more than an order of magnitude, so the lattice and LCDM force errors come out order\-of\-magnitude correct but a factor of two to five off, bottlenecked by scale rather than by method\. The two Ewald tasks miss for a different reason:Veritasreports the median of its error distribution, while the paper’s value matchesVeritas’s 99th\-percentile, a difference in which statistic is reported rather than in the computation\. On one lensing task,Veritasalso downsized, shortening its MCMC chain to fit compute, but it still matched\. Reaching the paper’s scale needs hardware\-specific engineering, such as GPU code, which we leave to future work\.

The size of the misses\.For the 68 tasks that miss the benchmark’s hard tolerance, we group the misses by how far the reported value lies from the paper value, on the same percent\-difference scale as the fidelity grade \([Table˜2](https://arxiv.org/html/2607.02931#S3.T2)\)\. The grouping is deterministic\. Each miss is placed by the largest relative error across its values, measured against ReplicationBench’s expected values and tolerances\.

Table 7:Veritaspaper\-only ReplicationBench outcomes by failure mode, under the benchmark’s own scorer\. About a third of the 68 tolerance misses \(the 6 within 2% and the 16 within 2–20%\) carry values that a tolerance\-free grade scores at A or B, soVeritas’s fidelity GPA \(3\.38\) is well above its native match rate \(0\.333\)\.About a third of the 68 misses fall within 20% of the paper value, close enough that the fidelity grade scores them A or B, and ReplicationBench’s binary scorer gives these no credit\. Its tolerances are author estimates, almost always round, with 98% a single significant figure such as 1000, 0\.3, or 0\.1\. Six of the misses fall within 2% of the paper value, and the nature of each quantity often shows why the result is reproduced\. One is a count of stars in a Gaia DR2 selection, 0\.12% from the target, about 1,250 stars out of a million, a difference set by where a selection cut falls rather than by a different sample\. Another is the calibrated temperature of a 50Ω\\Omegaload, 298\.33 K against 298\.0 K, within a third of a kelvin at room temperature\. The third one is the posterior probability that the neutron\-star to black\-hole mass gap is nonzero, 0\.957 against 0\.97; this is an MCMC posterior estimate under a specified spin prior, for which a 0\.01 tolerance is tighter than the inference’s own sensitivity to sampling and prior choices\.

Two further misses reproduce the reported quantity and differ only on a secondary element\.Veritasrecovers the Sun’s height above the Galactic mid\-plane, 20\.3 pc, exactly, and misses only the bootstrapped uncertainty, 0\.83 against 0\.3, which is a resampling estimate rather than the measured value\. A galaxy\-manifold transformation vector, read from a singular value decomposition, matches the paper in magnitude with the opposite sign; the sign of a singular vector is not fixed by the decomposition, so the two describe the same mapping\.

Partial reproduction in multi\-value tasks\.ReplicationBench’s per\-task grading is binary, and a task with a multi\-value output is credited only if every entry is within tolerance\. The match rate varies with claim shape \([Table˜8](https://arxiv.org/html/2607.02931#A7.T8)\)\.Veritasmatches 41% of scalar tasks and 42% of scalar\-range tasks, but only 20% of the 41 multi\-value tasks\.

Table 8:Veritaspaper\-only match rate by ReplicationBench claim shape\. A multi\-value claim is a vector, list, or dictionary, and is credited only when every entry is within tolerance, which accounts for its lower rate\.Of the 41,Veritasmatches 8 and leaves 1 unattempted\. ReplicationBench’s scorer counts the other 32 as misses, even though most reproduce some of their entries\. Across the 31 with a comparable output, 72 of 226 entries \(32%\) are within tolerance, and 11 of them recover at least half, with examples in[Table˜9](https://arxiv.org/html/2607.02931#A7.T9)\.[§˜F\.4](https://arxiv.org/html/2607.02931#A6.SS4)works through one such case, a near\-complete result that the scorer still records as zero\.

Table 9:Representative multi\-value tasksVeritasreproduces in part but ReplicationBench scores as full misses\. “Within tol\.” is the count of expected values within the author tolerance; “worst failing value” is the largest relative error among the out\-of\-tolerance entries\. The task is scored zero regardless of how many entries are within tolerance\.Tasks unreproducible by construction\.On three tasks,Veritas’s assessment reports that the benchmark’s own inputs are flawed, which we confirmed by inspecting them manually\. In galaxy\_soptics, the shipped Shi catalog is the SDSS DR7 release with 396,068 galaxies, while the task’s expected output is computed from the DR13 release with 586,025, soshi\_catalog\_acquisitioncannot match\. The provided Hsu\+22 cross\-match file is corrupted, which blocksbcg\_identification\. In ver\_waves,gaia\_breathing\_typicalasks for the mean of an absolute value, a small positive number, but its answer key is exactly 0 with zero tolerance, so no run can satisfy it\. We treat these as benchmark defects, not replication failures\.