How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks

# How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks
Source: [https://arxiv.org/html/2608.14905](https://arxiv.org/html/2608.14905)
August 14, 2026

###### Abstract

Artificial intelligence has long assisted scientific research, but the rapid advance of large language models and agentic scaffolds is reshaping the landscape: a single system can now carry a whole\-stage research from an initial hypothesis all the way to final published paper—a paradigm now referred to as AutoResearch\. Yet existing evaluations reveal little about how these agents operate or where they break down\. Tasks are narrowly\-scoped, evaluation measures performance but not process, and failure diagnoses lack systematic coverage or artifact\-level visibility\. To address this gap, we introduceAutoResearchEval, featuring100 tasksgrounded in published frontier science across seven scientific domains and the full research lifecycle: ideation, retrieval, execution, analysis, writing, and review\. Evaluating eight harness–model combinations yields800 autoresearch agent trajectories, with process\-level annotation\. We organize these insights intoARFT\(AutoResearch Failure Taxonomy\), a framework of45 empirically\-grounded failure patterns\. To enable scalable fine\-grained attribution, we leverage a human\-calibrated agent\-as\-a\-judge pipeline to inspect complete trajectories and intermediate artifacts\. While failure patterns span all stages of the research lifecycle, they converge on a single overarching limitation: current agents lack ametacognitive loop—the ability to check what they produced against what they found, revise when it does not hold up, and question whether the path they took was sound\. The same patterns recur across all eight harness–model combinations, including the strongest models tested, locating the deficit at the model level rather than in any particular scaffold; whether orchestration\-level interventions can close it is an open question this work does not test\. We publicly release AutoResearchEval and ARFT to facilitate continued research and development in autonomous scientific discovery\.

![Refer to caption](https://arxiv.org/html/2608.14905v1/general_overview.png)Figure 1:Construction, rollout, and evaluation of AutoResearchEval\.a,5,878 papers from nine domains are parsed into seven fields and filtered to 100 tasks spanning seven domains; the agent sees only the query \(Premise, Tension\), while the target \(KeyClaims, Conclusion\) is withheld\.b,Each task runs once per harness–model pair as a six\-stage episode with revision, yielding 800 trajectories with all artifacts retained\. The case study traces one trajectory: six stage\-wise deviations converging on a single metacognitive deficit\.c,ARFT is induced bottom\-up: experts annotate failures in full trajectories, group them into patterns, and refine until all agree, giving 45 patterns under 4 root\-cause pillars\. A judge agent then reviews the full artifact set under a per\-stage rubric, anchoring every issue to concrete evidence and categorize it to an ARFT pattern, with a quality checker regenerating weak analyses in a self\-healing loop\. It reaches�=0\.75\\kappa=0\.75\(pattern\) and0\.830\.83\(taxonomy\) against human labels, versus0\.530\.53and0\.620\.62for a single\-call LLM\-as\-a\-judge\.
## 1\. Introduction

Artificial intelligence has assisted scientific research for decades\. Until recently, "AI for science" largely meant a specialized model targeting one well\-defined task—predicting biomolecular structures\[[16](https://arxiv.org/html/2608.14905#bib.bib16),[19](https://arxiv.org/html/2608.14905#bib.bib19),[23](https://arxiv.org/html/2608.14905#bib.bib23),[35](https://arxiv.org/html/2608.14905#bib.bib35),[42](https://arxiv.org/html/2608.14905#bib.bib42),[28](https://arxiv.org/html/2608.14905#bib.bib28),[43](https://arxiv.org/html/2608.14905#bib.bib43),[11](https://arxiv.org/html/2608.14905#bib.bib11)\], accelerating materials discovery\[[26](https://arxiv.org/html/2608.14905#bib.bib26),[46](https://arxiv.org/html/2608.14905#bib.bib46),[44](https://arxiv.org/html/2608.14905#bib.bib44),[8](https://arxiv.org/html/2608.14905#bib.bib8),[1](https://arxiv.org/html/2608.14905#bib.bib1)\], or forecasting the weather\[[20](https://arxiv.org/html/2608.14905#bib.bib20),[2](https://arxiv.org/html/2608.14905#bib.bib2)\]—while human researchers still framed the question, built the AI tools themselves, interpreted the results, and communicated the findings\. AlphaFold\[[16](https://arxiv.org/html/2608.14905#bib.bib16)\]epitomizes this paradigm: a landmark solution to a single, long\-standing problem, but one confined to that problem alone\. The rapid advance of large language models and agentic scaffolds is now reshaping this picture\. Combining foundation models, external tools, and agentic workflows, a single system can carry a study from an initial hypothesis through literature review, experimentation, and analysis to a written draft—a paradigm now referred to as*AutoResearch*\[[37](https://arxiv.org/html/2608.14905#bib.bib37),[18](https://arxiv.org/html/2608.14905#bib.bib18),[49](https://arxiv.org/html/2608.14905#bib.bib49)\]\. AI is gradually becoming a participant in scientific discovery rather than merely a tool for it\. What is being automated is no longer an isolated step, but the research process as a whole\. A natural question follows:how well can agents actually perform across the full AutoResearch process?

Evaluating autonomous research agents requires characterizing their behavior across the entire scientific process\. Recent work has made notable strides toward this goal, from benchmarks that assess isolated research skills\[[32](https://arxiv.org/html/2608.14905#bib.bib32),[29](https://arxiv.org/html/2608.14905#bib.bib29),[36](https://arxiv.org/html/2608.14905#bib.bib36),[33](https://arxiv.org/html/2608.14905#bib.bib33),[14](https://arxiv.org/html/2608.14905#bib.bib14),[34](https://arxiv.org/html/2608.14905#bib.bib34)\]to increasingly ambitious end\-to\-end pipelines\[[41](https://arxiv.org/html/2608.14905#bib.bib41),[45](https://arxiv.org/html/2608.14905#bib.bib45),[4](https://arxiv.org/html/2608.14905#bib.bib4)\]\. Despite such progress, existing evaluations leave three gaps that together obscure*how*an agent works and*where*it breaks down\.

Tasks are narrowly\-scoped\.Existing discovery benchmarks fall into two groups, and neither captures the full research lifecycle in real scientific domains\. The first places agents in simulated or fictional worlds, adopted because real experiments are expensive—but this comes at the cost of realism\[[15](https://arxiv.org/html/2608.14905#bib.bib15),[40](https://arxiv.org/html/2608.14905#bib.bib40),[25](https://arxiv.org/html/2608.14905#bib.bib25),[22](https://arxiv.org/html/2608.14905#bib.bib22)\]\. The second stays grounded in real science but restricts itself to verifiable settings—most commonly machine learning or software engineering tasks whose outcomes can be measured against a well\-defined target\[[6](https://arxiv.org/html/2608.14905#bib.bib6),[27](https://arxiv.org/html/2608.14905#bib.bib27),[34](https://arxiv.org/html/2608.14905#bib.bib34),[7](https://arxiv.org/html/2608.14905#bib.bib7),[36](https://arxiv.org/html/2608.14905#bib.bib36),[41](https://arxiv.org/html/2608.14905#bib.bib41),[45](https://arxiv.org/html/2608.14905#bib.bib45)\]\. Coverage of science domains is correspondingly narrow, and genuine open\-ended discovery tasks are excluded\.

Evaluation measures performance, not process\.Nearly all existing benchmarks anchor scoring at the endpoint—a reference match, a reproduced result, or a published SOTA\. This invites reward hacking: circular validation, grader\-fitting, or leakage move the number without doing the science, and the score cannot distinguish a sound trajectory from a gamed one\[[6](https://arxiv.org/html/2608.14905#bib.bib6),[14](https://arxiv.org/html/2608.14905#bib.bib14),[41](https://arxiv.org/html/2608.14905#bib.bib41)\]\. It also compresses a long\-horizon trajectory into a single scalar, reporting*that*a run failed but not*why*,*where*, or*how*—so endpoint evaluation can rank systems but cannot diagnose them\.

Failure diagnoses lack systematic coverage or artifact\-level visibility\.Expert case studies scrutinize individual trajectories in depth\[[17](https://arxiv.org/html/2608.14905#bib.bib17),[21](https://arxiv.org/html/2608.14905#bib.bib21),[31](https://arxiv.org/html/2608.14905#bib.bib31),[13](https://arxiv.org/html/2608.14905#bib.bib13),[10](https://arxiv.org/html/2608.14905#bib.bib10),[38](https://arxiv.org/html/2608.14905#bib.bib38),[49](https://arxiv.org/html/2608.14905#bib.bib49)\]but cannot generalize across systems or tasks\. Categorization efforts targeting scientific discovery\[[24](https://arxiv.org/html/2608.14905#bib.bib24),[10](https://arxiv.org/html/2608.14905#bib.bib10),[3](https://arxiv.org/html/2608.14905#bib.bib3)\]offer broader categories but are derived from small corpora\. The most systematic line—trace\-level analysis of multi\-agent or QA settings\[[5](https://arxiv.org/html/2608.14905#bib.bib5),[9](https://arxiv.org/html/2608.14905#bib.bib9),[48](https://arxiv.org/html/2608.14905#bib.bib48)\]—annotates conversational traces, so artifact\-level failures stay invisible and stages are interaction phases rather than steps of the scientific method\.

Together these gaps leave a critical blind spot:current evaluations may show that an agent succeeded, but reveal little about how it operates, or precisely where it breaks down\.

To address this gap, we investigate why autonomous research agents fail, producing two artifacts\. The first isAutoResearchEval, a collection of800800research trajectories withartifact\-aware, process\-level failure annotations, elicited from eight harness–model combinations on a100100\-task suite spanning thefull\-lifecyclescientific workflow—ideation, retrieval & synthesis, execution, analysis, writing, and review—acrossseven frontier domainsand constructed from papers in prestigious venues\. Tasks span two regimes: those with an explicit execution\-feedback signal \(a human SOTA or quantitative metric\), and fully open\-ended tasks with none, where we deliberately judge the rigor and self\-consistency of the agent’s*process*rather than agreement with ground truth\. Annotations come from ahuman\-calibrated, artifact\-aware agent\-as\-a\-judge annotator\[[51](https://arxiv.org/html/2608.14905#bib.bib51)\]that reads each trajectory in full—run logs, generated data, code, and reports—rather than its final answer\. We calibrate it against human annotations and spot\-check its outputs post hoc\.

To enable systematic annotation and comprehensive analysis of AutoResearchEval, we develop the second artifact:ARFT, the first systematic and comprehensive failure taxonomy for autonomous research agents\. ARFT is empirically grounded in the annotated trajectories and organizes4545failure patterns on two cross\-cutting axes: the lifecycle*stage*at which a failure manifests, and the underlying*root cause*, so that superficially different failures sharing a mechanism are grouped together \(Figure[3](https://arxiv.org/html/2608.14905#S5.F3)\)\. Together, AutoResearchEval supplies the empirical evidence of how research agents fail in practice, while ARFT provides the structured vocabulary to diagnose, attribute, and ultimately mitigate these failures\.

Our analysis of AutoResearchEval yields one core finding that revises prevailing assumptions\. While failure patterns span all stages of the research lifecycle, they converge on a single overarching limitation: current agents lack ametacognitive loop—the ability to check what they produced against what they found, revise when it does not hold up, and question whether the path they took was sound\.

##### Contributions\.

In summary, our core contributions are as follows:

- •AutoResearchEval, aopen\-sourcecollection of800800trajectories with artifact\-aware, process\-level failure annotations, released together with the underlying100100\-task, seven\-domain, full\-lifecycle task suite built from published frontier science, including a fully open\-ended subset scored on process rather than outcome;
- •ARFT, the first systematic failure taxonomy for autonomous research agents, organizing 45 empirically\-grounded failure patterns;
- •A human\-calibrated, artifact\-aware agent\-as\-a\-judgethat is used for ARFT categorization\. It reads each trajectory in full rather than its final answer alone and hence supporting analyzing and understanding failure patterns;
- •A systematic diagnosis of when and why these agents fail, tracing all three cognitive root causes to a sharedmetacognitive deficit—the absence of a closedmetacognitive loop\. This diagnosis locates the gap at the model level—the same patterns recur across all eight harness–model combinations—and identifies the metacognitive loop as a capacity that next\-generation language models will need for genuine scientific discovery\. Whether orchestration alone can compensate is an open question we do not test\.

## 2\. AutoResearch Tasks and Trajectories

AutoResearchEval turns published papers into discovery tasks and runs agents on them end to end\. From each paper we build a task that states where the science stood and what was left unresolved—but no method, so many paths through it are admissible—while the paper’s published outcome is withheld\. Tasks come in two types and carry domain and contribution labels, and other fields extracted from the paper drive the filtering and authoring behind the scenes\. Each task is then run as a single autonomous rollout in a sandbox with code execution, and we log the complete trajectory with produced artifacts as the unit of analysis\. The rest of this section details the tasks \(§[2\.1](https://arxiv.org/html/2608.14905#S2.SS1)\) and the rollouts \(§[2\.2\.1](https://arxiv.org/html/2608.14905#S2.SS2.SSS1)\); extraction prompts and the rollout environment are in Appendix[F](https://arxiv.org/html/2608.14905#A6)\.

### 2\.1 AutoResearch Tasks Mined from Venues

#### 2\.1\.1 Task construction

A paper is parsed into seven fields—Premise,Tension,Motivation,Method,Experiment,KeyClaims,Conclusion—and split into a task instance�p=\(qp,�​\(p\),targetp\)\\tau\_\{p\}=\(q\_\{p\},\\nu\(p\);\\mathrm\{target\}\_\{p\}\)\. The queryqp=\(Premise,Tension\)q\_\{p\}=\(\\textsf\{Premise\},\\textsf\{Tension\}\)states the prior literature and the anomaly left open by it; the published outcometargetp=\(KeyClaims,Conclusion\)\\mathrm\{target\}\_\{p\}=\(\\textsf\{KeyClaims\},\\textsf\{Conclusion\}\)is withheld from the query and kept as ground truth\. The remaining five fields are not shown to the agent; they drive construction:ExperimentandMethodgate whether a paper can become a runnable task,MotivationandConclusiondetermine which of the two task types it becomes, andKeyClaimsrecords the paper’s terminal quantities used to author the held\-out reference\. Papers are mined from high\-impact venues and recent work from established groups across nine scientific domains, giving5,8785\{,\}878candidates\. Filtering reads the extracted fields: a paper is dropped when itsExperimentis purely wet\-lab or its data is not publicly available \(the task could not be run\), and when itsMethodandMotivationdescribe a single\-step lookup rather than an investigation that rewards multiple stages\. From what remains,N=100N=100tasks are authored, drawn from seven of the domains\. Every task carries two labels: its domain, and one of the*novelty\-move*types, read from itsTensionandConclusion, recording what kind of contribution the paper makes\. Tasks are restricted to papers from 2024 onward\. This applies to the open\-ended discovery subset; the target\-anchored optimization subset follows the source benchmark’s task selection and includes earlier papers \(Appendix[G](https://arxiv.org/html/2608.14905#A7)\)\.

#### 2\.1\.2 Task types and composition

A task’sConclusionandMotivationalso fix its type, according to whether the paper’s goal terminates in a quantity computable on held\-out data or in a qualitative finding that does not\.*Open\-ended discovery*tasks \(n=70n=70\) have no such quantity: a human reference exists, but nothing in the environment tells the agent whether it is getting closer, and the space of acceptable methods is wide\.*Target\-anchored optimization*tasks \(n=30n=30\) expose an explicit objective—a human state of the art or a computable metric—giving a well\-posed direction of improvement, though the method to reach it is still unspecified\. The split is not free of content: an explicit metric exists only for certain kinds of contribution, so every target\-anchored task is a new\-regime, incremental, or method\-correction move, while the reconciliation, mechanism, consensus\-overturn, and scaling\-relation moves are open\-ended only\. This also shows on the domain axis \([Figure2](https://arxiv.org/html/2608.14905#S2.F2)\): the open\-ended set spans the seven represented domains, whereas the target\-anchored set concentrates in the domains where a computable reference quantity is available\. The two types are therefore not matched on domain or move, and are reported separately throughout—never compared across either\.

Figure 2:Composition of AutoResearchEval \(N=100N=100\), each axis split by task type: scientific domain \(left\) and novelty\-move type \(right\)\. Open\-ended discovery in blue, target\-anchored optimization in terracotta\.

### 2\.2 AutoResearch Trajectories

#### 2\.2\.1 Six\-stage end\-to\-end auto research rollout

Each task is run as one autonomous rollout\. Given only the query and a fresh sandbox with code execution enabled, the agent works without human intervention through six stages—ideation and planning, retrieval and synthesis, execution and implementation, analysis and interpretation, writing and documentation, and a final self\-verification and review pass over its own report—so every task involves real retrieval, computation, and artifact generation rather than answer lookup\. The rollout ends when the agent commits a final report or exhausts its budget, and we log the complete trajectory: the full interaction and tool\-call trace, all generated code and data, intermediate outputs, and the final report\.

#### 2\.2\.2 Harness\-model combinations

An agent is a*harness–model*pair: the harness supplies the tool loop, file system, code execution, and control flow, and the backbone model drives it\. We evaluate the frontier harnessesClaude Code,Codex, andGemini CLIagainst backbones from several model families \([Table1](https://arxiv.org/html/2608.14905#S2.T1)\)\. Running one backbone under several harnesses, and one harness over several backbones, separates failures of the scaffold from failures of the model and shows whether a failure mode is idiosyncratic to one system or shared\. Because the central analysis is the failure\-pattern taxonomy rather than a ranking this suffices—the failure mechanisms recur across combinations, and any claim that depends on a particular combination or subset is stated with its population\.

Table 1:Harness–model combinations under evaluation\.HarnessBackbone modelsClaude Codeopus\-4\.8, claude\-sonnet\-5, qwen3\.7\-max,glm\-5\.2, minimax\-m3, deepseek\-v4\-proCodexgpt\-5\-miniGemini CLIgemini\-3\.5\-flash
#### 2\.2\.3 Trajectory statistics

Every task is run by every harness–model combination, with the query as its only input, which over the88combinations yields800800trajectories comprising7373k tool calls at an average of92\.392\.3steps per episode\. The complete trajectory—not the final report alone—is the unit of the process\-level analysis in the sections that follow\. Sandbox configuration, budgets, prompts, and the full task manifest are in Appendix[F](https://arxiv.org/html/2608.14905#A6)\.

## 3\. Building Artifact\-aware Agent\-as\-a\-judge

To make AutoResearchEval to be fully utilized and uncover hidden failure patterns behind the trajectories, a systematic and comprehensive analysis method is required\. However, many failure patterns leave no trace in the report at all: the agent may claim a result its own code does not produce or describe a method its logs show it never ran\. Detecting these failures requires comparing the manuscript against the full set of artifacts the agent produced\. We therefore build an annotation pipeline in three steps: human annotation to develop and calibrate the failure taxonomy \(§[4](https://arxiv.org/html/2608.14905#S4)\), an automated Agent\-as\-a\-Judge to scale annotation to the full 800\-trajectory corpus, and a validation study measuring the judge’s agreement with human labels\.

### 3\.1 Human Annotation

Our failure taxonomy was developed inductively from expert examination of agent trajectories, following a grounded\-theory process\[[12](https://arxiv.org/html/2608.14905#bib.bib12)\]\. Rather than starting from a predefined checklist, experts examined complete trajectories independently—execution logs, delivered code, final reports, and data files—and recorded whatever failure behaviors they observed\. Observed patterns were iteratively grouped, split, and refined through constant comparative analysis until further annotation yielded no new failure modes \(theoretical saturation\)\. We provide a detailed description and analysis of the resulting taxonomy in §[4](https://arxiv.org/html/2608.14905#S4)\.

To validate that the taxonomy can be applied consistently, we conduct inter\-annotator agreement \(IAA\) studies on a stratified sample of5050trajectories drawn from all800800trajectories\. Three experts independently label each sampled trajectory against the full taxonomy\. Taxonomy then is iteratively sharpened and refined—adding, merging, or clarifying categories—until three experts reach consensus\. We conduct55rounds of IAA, achieving�=0\.85\\kappa=0\.85\(Cohen’s Kappa\) in the final round\. Because annotation difficulty varies across failure types—failures that leave concrete artifacts \(e\.g\. a code file contradicting the report\) are easier to adjudicate than failures requiring metacognitive judgment—agreement is not uniform across the taxonomy, and we treat patterns in the Cognitive Depth & Adaptability pillar as lower\-confidence throughout \(Appendix[C\.7](https://arxiv.org/html/2608.14905#A3.SS7)\)\.

Table 2:Agreement with human expert annotation on the 50 validation trajectories\. The LLM\-as\-a\-Judge baseline receives only the transcript in a single call; the Agent\-as\-a\-Judge receives the full evidence package and operates under the structured rubric\.*Pattern*measures per\-pattern hit/miss agreement across the 45 ARFT patterns;*Taxonomy Categorization*measures agreement at the root\-cause pillar level\.MethodLevelAccuracyPrecisionRecallF1Cohen’s�\\kappaLLM\-as\-a\-Judge \(claude\-opus\-5\)Pattern84\.670\.263\.566\.70\.53Taxonomy Categorization89\.378\.872\.175\.30\.62Agent\-as\-a\-JudgePattern92\.185\.480\.783\.00\.75Taxonomy Categorization95\.391\.087\.289\.10\.83
### 3\.2 Agent\-as\-a\-Judge

Manually annotating 800 trajectories at this depth is prohibitively expensive and time\-consuming\. To scale annotation we develop anartifact\-awareAgent\-as\-a\-Judge: an autonomous agent that analyzes a trajectory for failure patterns across all lifecycle stages and categorizes them according to the ARFT\. Unlike a single LLM\-as\-a\-judge call on the transcript, the agent judge can execute code and navigate the rollout’s full artifact set so failures invisible in the report alone become detectable\.

The judge receives the rollout’s complete evidence package: code, execution logs, final reports, and data files \(Appendix[C\.3](https://arxiv.org/html/2608.14905#A3.SS3)\)\. Its prompt enforces a fixed rubric aligned with the lifecycle stages of the taxonomy, requiring the judge to cover each stage in a dedicated section and support every identified issue with verifiable evidence—a log line number, file name, code identifier, or exact numeric value\. A set of nine*iron rules*, distilled from earlier annotation rounds, guard against common mis\-judgments \(Appendix[C\.4](https://arxiv.org/html/2608.14905#A3.SS4)\)\. An automated quality checker enforces coverage, depth, and anchor density before accepting a document; documents that fail are regenerated with gate\-specific feedback in a self\-healing loop \(Appendix[C\.8](https://arxiv.org/html/2608.14905#A3.SS8)\)\. Accepted analyses are then mapped to taxonomy pattern IDs in a labeling pass, producing the failure counts aggregated in Figure[3](https://arxiv.org/html/2608.14905#S5.F3)\. Full details of the judge harness, rubric, checker, and labeling protocol are in Appendix[C](https://arxiv.org/html/2608.14905#A3)\.

### 3\.3 Validation Against Human Annotation

We validate the Agent\-as\-a\-Judge against human expert annotations on the 50 human\-labeled trajectories of §[3\.1](https://arxiv.org/html/2608.14905#S3.SS1)\. Each trajectory is annotated independently by the judge, and we measure agreement at the failure pattern and taxonomy categorization level against human experts\. For level pattern measurement, human experts and LLM are involved to compare and judge the results\. Table[2](https://arxiv.org/html/2608.14905#S3.T2)reports the results\.

The artifact\-aware Agent\-as\-a\-Judge substantially outperforms the single\-call LLM\-as\-a\-judge at both granularities, with the largest gains in recall \(\+17\.2\+17\.2at pattern level\), indicating that artifact access is required in practice for detecting failures invisible in the transcript\.

## 4\. A Taxonomy of Autoresearch Agent Failure Patterns

Applying this evaluation across the collected trajectories yields ARFT, which organizes the 45 observed failure patterns along two orthogonal axes\. Thestage axislocates*where*a failure manifests in the research pipeline—ideation, retrieval and synthesis, execution, analysis, writing, and review \(Stages A–F\)\. Theroot\-cause axisidentifies the underlying mechanism—*why*it occurs rather than where—so that superficially different failures sharing a cause are grouped together, while one stage’s failures can split across causes\. A cross\-stage layer \(X\) captures dynamic failures that do not localize to any single stage but describe how errors propagate, drift, or compound across the pipeline \(e\.g\., error propagation, goal drift\)\. A trajectory may carry multiple \(stage, root\-cause\) instances; the judge emits one label per detected instance\. This section presents the taxonomy’s structure and the payoff of the two\-axis view; the full label set, and definitions are in Appendix[A](https://arxiv.org/html/2608.14905#A1)\.

Table 3:Systemic root\-cause classification of AutoResearch failure patterns\.All 45 patterns are grouped under four root\-cause pillars—Grounding & Faithfulness, Cognitive Depth & Adaptability, Scientific Integrity & Alignment, and Engineering Robustness—that capture the underlying mechanism of failure rather than the pipeline stage at which it surfaces\. Each pattern ID encodes its stage \(with the X series spanning multiple stages\), and every pattern maps to exactly one pillar\.Root Cause PillarCore Failure FocusMapped Failure Patterns \(IDs\)R1\. Grounding & FaithfulnessDisconnect between high\-level claims/hypotheses and ground\-truth code, data, or logs\.A\.6, B\.1, B\.2, B\.5, C\.3, D\.1, D\.4, D\.6, E\.1, E\.4, F\.6, X\.6R2\. Cognitive Depth & AdaptabilityShallow reasoning/search, passivity in self\-review, and inability to re\-plan or pivot\.A\.1, A\.3, B\.4, B\.6, C\.6, C\.7, D\.5, F\.1, F\.2, F\.3, F\.4, X\.3, X\.7R3\. Integrity & AlignmentMetric hacking, shortcut reliance, confirmation bias, overclaiming, and goal drift\.A\.2, A\.5, C\.1, C\.2, D\.2, D\.3, D\.7, E\.2, E\.3, F\.5, X\.2, X\.4, X\.5R4\. Engineering RobustnessNumerical overflows, unhandled runtime errors, and broken interaction with CLI/OS\.A\.4, B\.3, C\.4, C\.5, C\.8, X\.1, X\.8##### Stage axis \(A–F\) and cross\-stage layer \(X\)\.

The six stages and the cross\-cutting layer group the failure patterns as follows:

- •A⋅\\cdotIdeation & Planning\(6 patterns\): failures of hypothesis formation and experimental design, e\.g\., frame\-lock in a narrow hypothesis space \(A\.1\), unfalsifiable hypotheses \(A\.2\), and experiments that do not actually test the stated hypothesis \(A\.6\)\.
- •B⋅\\cdotRetrieval & Synthesis\(6 patterns\): failures of evidence acquisition and use, e\.g\., hallucinated citations \(B\.1\), shallow search coverage \(B\.4\), and retrieved knowledge that never informs experimental design \(B\.2\)\.
- •C⋅\\cdotExecution & Implementation\(8 patterns\): failures during coding and experimentation, e\.g\., circular validation \(C\.1\), grader\-fitting and data leakage \(C\.2\), and code that diverges from the claimed methodology \(C\.3\)\.
- •D⋅\\cdotAnalysis & Interpretation\(7 patterns\): failures of inference from results, e\.g\., mistaking artifacts for insights \(D\.1\), confirmation bias \(D\.2\), and fabricated metrics or tables \(D\.6\)\.
- •E⋅\\cdotWriting & Documentation\(4 patterns\): failures of faithful reporting, e\.g\., claims untraceable to actual execution \(E\.1\) and overclaiming with concealed negative results \(E\.2\)\.
- •F⋅\\cdotSelf\-Verification & Review\(6 patterns\): failures of the agent’s own quality gate, e\.g\., superficial checklist\-style self\-review \(F\.1\) and review score hacking \(F\.5\)\.
- •X⋅\\cdotCross\-Stage Patterns\(8 patterns\): dynamic failures spanning stages, e\.g\., cascading error propagation \(X\.1\), goal drift \(X\.2\), and right\-for\-the\-wrong\-reason successes \(X\.6\)\.

Ultimate Root Cause: Metacognitive LoopWhile failure patterns span all stages of the research lifecycle, they are fundamentally driven by a single overarching limitation:Metacognitive Loop\. A human researcher works in a closed loop—recognising the limits of what they know, checking whether intermediate results hold up, and re\-planining when it is necessary\. Current autoresearch agents lack this loop\. They can execute each step of the research process, but they do not have the awareness to monitor what they have produced, judge whether it is valid, and re\-plan when it is not\.

##### Root Cause axis\.

The second axis exposes structure invisible to a stage\-only list: when patterns are aligned by root mechanism, families emerge that pipeline position would otherwise scatter\. This core limitation manifests through four practical root\-cause pillars:Grounding & Faithfulness,Cognitive Depth & Adaptability,Scientific Integrity & Alignment,Engineering Robustness\. We will introduce more in §[5\.2](https://arxiv.org/html/2608.14905#S5.SS2)\.

Table[3](https://arxiv.org/html/2608.14905#S4.T3)gives the full pattern\-to\-pillar mapping\. The most striking family the table reveals is the Depth root\-cause: a single cognitive deficit resurfaces at nearly every stage of the pipeline, from ideation\-time frame\-lock \(A\.1\) through execution\-time local optimization \(C\.6\) to review\-time passivity \(F\.1–F\.4\)\.

We defer empirical evidence and detailed case studies to Section[5](https://arxiv.org/html/2608.14905#S5)and Appendix[B](https://arxiv.org/html/2608.14905#A2)\.

## 5\. Empirical Analysis

### 5\.1 Failure Pattern Statistics

Auditing every scored trajectory against the 45\-pattern taxonomy of §[4](https://arxiv.org/html/2608.14905#S4)yields 12,712 hits across 800 analyses as shown in[Figure3](https://arxiv.org/html/2608.14905#S5.F3); complete counts are in Appendix[D](https://arxiv.org/html/2608.14905#A4)\.

The distribution across root\-cause pillars is uneven\. The three cognitive pillars—Grounding & Faithfulness \(R1, 31\.0%\), Scientific Integrity & Alignment \(R3, 33\.5%\), and Cognitive Depth & Adaptability \(R2, 27\.6%\)—together account for 92\.1% of all hits\. Engineering Robustness \(R4\) contributes just 7\.9%, and its highest\-ranked pattern, execution faults and numerical instability \(C\.4\), places only 26th of 45 failure patterns\.

At the individual\-pattern level, failures concentrate heavily in the self\-verification stage\. Uncorrected self\-awareness \(F\.4\) is the single most frequent pattern in the corpus, appearing in 660 of 800 analyses \(82\.5%\)\. Two related patterns—failure to gate critical flaws \(F\.2, 502 hits\) and unremediated adversarial evidence \(D\.7, 486 hits\)—rank among the top five\. Together these three patterns account for 13\.0% of all hits, the largest concentration attributable to any single mechanism\.

Failure profiles differ across the eight model–harness combinations, though the overall shape is consistent\. Total hit counts range from 1,396 \(opus\-4\.8\) to 1,818 \(qwen3\.7\-max\), and the top\-10 most frequent patterns overlap heavily: E\.2, D\.4, A\.5, and C\.1 appear in the top 10 of every model, and F\.4 in seven of the eight \(Appendix[E](https://arxiv.org/html/2608.14905#A5)\)\. Where systems diverge most is in fabrication\-related patterns\. Hallucinated evidence \(B\.1\) ranges from 13 hits \(glm\-5\.2\) to 61 \(gpt\-5\-mini\); result hallucination \(D\.6\) ranges from 3 \(opus\-4\.8 and claude\-sonnet\-5\) to 36 \(qwen3\.7\-max\)\. These system\-specific profiles suggest that while the core failure patterns are shared across models, fabrication rates reflect differences in model capability, and the detailed per\-model breakdowns in Appendix[E](https://arxiv.org/html/2608.14905#A5)can guide system\-specific mitigation\.

Two patterns, review score hacking \(F\.5\) and hallucinated reviewing \(F\.6\), are near\-absent in the corpus—one and three hits respectively, both borderline—so their inclusion in the taxonomy reflects observed behavior rather than a claim about prevalence\.

![Refer to caption](https://arxiv.org/html/2608.14905v1/figs/failure_heatmap.png)Figure 3:Failure pattern attribution across agent phases \(aggregated trajectories,n=800n=800\)\. Cell color encodes the number of HIT attributions; each pattern contributes the number of distinct trajectories in which it was an established failure; “—” denotes none\. A single cell may contain more than one failure pattern within the same cell, so cell counts can exceed 800\.
### 5\.2 Root Causes and What They Imply

Some of what we observe reflects limits of the underlying models rather than of the systems built around them, and we say so where the data shows it\. Our emphasis falls on failures whose evidence is already present in the trajectory, because those are the ones where the distance between what the agent knew and what it did can be measured rather than assumed\.

The root causes differ in surface form but share a common origin: the metacognitive deficit of §[4](https://arxiv.org/html/2608.14905#S4)\. A human researcher works in a closed loop—checking whether what they produced matches what they found, acting when it does not, and questioning whether the path they took was sound\. Current agents lack this loop, and each root cause is a different point where it breaks\. R1 is the failure to check output against evidence\. R2 is the failure to act on a flaw the agent itself identified\. R3 is the failure to question whether the path to the result is legitimate\. We take each in turn\.

Two kinds of number appear below\. A root cause’s*share*—such as R1’s 31\.0%—is its fraction of the 12,712 total hits across the corpus\. A pattern’s*rate*—such as D\.4’s 77\.5%—is the fraction of 800 analyses\. Where a group of patterns is summarised together, the figure given is that group’s share of its root cause’s hit total\. Notice that one trajectory usually contains more than one failure pattern\.

R1\. Grounding & Faithfulness\(31\.0% of all hits\)\. Claims, hypotheses, and plans disconnect from the code, data, logs, or literature that should license them\. É\\blacktrianglerightInsight 1\. The evidence that would refute most failures is already in the agent’s own run directory\.The agent produces both the claim and the file that contradicts it; the comparison is never performed\.

R1 is led by method–conclusion disconnect \(D\.4, 77\.5% of analyses\), implementation discrepancy \(C\.3, 72\.1%\), and report–code traceability gaps \(E\.1, 60\.5%\)\. What these share is that the agent writes up the work it meant to do rather than the work it did—and the evidence exposing the gap sits in the same run directory the agent itself created\.

Two\-thirds of R1 consists of*unsupported claims*, in two forms\. In the first, the run files say something different from the report: a conclusion is drawn that the method does not support \(D\.4, 77\.5%\), an experiment is presented as testing a hypothesis it does not test \(A\.6, 52\.5%\), or a side effect of the pipeline is reported as a finding \(D\.1, 52\.4%\)\. In the second, there is nothing behind the claim at all—a method section describing a procedure the code never implements \(C\.3, 72\.1%\), or a claim no run in the trajectory supports \(E\.1, 60\.5%\)\. The second form is the more serious, because the agent is not overstating a result but reporting work it did not do\. It is also distinct from the overclaiming of R3, where the experiment was run and the conclusion reaches past it\.

A further sixth is*invented evidence*—hallucinated sources, fabricated citations, numbers given for runs that never happened \(B\.1, E\.4, D\.6\)—where nothing can contradict the claim because the thing referred to does not exist\. The two groups behave differently across systems: unsupported claims appear at nearly the same rate in all eight systems, while invented evidence is much rarer in the strongest ones\. The remaining is retrieval that never reaches the design \(B\.2, 33\.9%\), citations that do not support the sentence citing them \(B\.5, 19\.0%\), and right\-for\-the\-wrong\-reason success \(X\.6, 43\.8%\)—less a mechanism of its own than the outcome the other two produce\.

Two consequences\. Stronger models are unlikely to remove unsupported claims, because catching one requires no ability the agents lack: the report and the run directory are both products of the same agent, and all that is missing is a requirement to compare them before the report goes out\. A human author checks the number in the abstract against the number in the table, and does not list a contribution they never implemented; nothing in these runs performs either check\. And an evaluation that reads only the final report cannot see any of this, because the report is one half of the disagreement and the other half is on disk\. That is the case for scoring the artifacts a run leaves behind rather than the answer it ends with\. This is the metacognitive loop failing at the point of evaluation: the agent never compares what it produced against what it found, so the loop never begins\.

R2\. Cognitive Depth & Adaptability\(27\.6% of all hits\)\. Shallow reasoning and search, passivity in self\-critique, and inability to re\-plan or pivot at a dead end\. É\\blacktrianglerightInsight 2\. The most common failure in the corpus is not missing a flaw but finding it and shipping anyway\.In 82\.5% of analyses the agent diagnoses a critical problem during self\-review, then reports the unrevised conclusion\.

This root cause is where the missing metacognitive loop of §[4](https://arxiv.org/html/2608.14905#S4)is easiest to see, so it is worth stating plainly what that loop is\. A researcher in difficulty does three things: recognises the limits of what they know, judges whether their intermediate results hold up, and changes the plan when they do not—then repeats\. R2 is the failure of all three\.

Recognising limits fails in frame\-lock within a narrow hypothesis space \(A\.1\), thin search coverage \(B\.4, 54\.9% of analyses\), and absent skepticism \(X\.3, 39\.4%\)—together 27% of R2’s hits\. Judging results fails in the four self\-review patterns \(F\.1–F\.4\) and in missing baselines \(D\.5\): 62% of R2’s hits\. Changing the plan fails in local optimization \(C\.6\), stopping early \(C\.7\), and anchoring to a failed approach \(X\.7\): 11% of R2’s hits\.

Judging results is the largest of the three, and its name is misleading, because agents do reach the right judgment\. In uncorrected self\-awareness \(F\.4, 82\.5% of analyses\), the most common pattern in the corpus, the agent finds the fatal flaw and writes it down, then reports the conclusion anyway\. The judgment was correct and nothing followed from it\. A self\-review is just more text: the same model writes it, in the same run, and nothing in the system requires the review to change the report\. A human researcher who finds a broken baseline has to fix it before the paper goes out; these agents do not\.

Judging results could in principle be repaired from outside the model\. If the agent writes that its own result is uninterpretable, the system can refuse the report until either the result or the claim changes\. Recognising limits cannot be repaired that way\. When an agent never considers a second hypothesis, no second hypothesis exists in the trajectory to compare against, and nothing can flag what was never written down\. This is the part that depends on the model rather than on the harness, and it is why the choice of system matters more here than in the other two root causes\. Unlike R1, the metacognitive loop does fire here—the agent detects the problem—but nothing in the system closes it, so the diagnosis produces no change\.

R3\. Scientific Integrity & Alignment\(33\.5% of all hits\)\. The agent pursues the stated goal by whatever path produces a result, whether the path is sound or not: shortcut reliance, metric hacking, concealed failure, and conclusions fixed in advance\. É\\blacktrianglerightInsight 3\. A correct\-looking output is not evidence of sound methodology\.Agents routinely reach the stated goal through circular validation, metric substitution, and concealed negative results—and nothing in the system distinguishes a legitimate solution from a gamed one\.

R3 is the largest root cause, led by overclaiming with concealed negative results \(E\.2, 78\.1% of analyses\), circular validation and shortcut reliance \(C\.1, 69\.0%\), and metric misalignment \(A\.5, 68\.1%\)\. The outputs these agents produce often look correct—the numbers are plausible, the report is well\-structured, but the methodology behind them is invalid, and the agent never questions whether it is\.

Seventy of our hundred tasks are open\-ended: the agent receives a premise and an unresolved tension, and nothing in the environment checks whether its approach is sound\. Nearly half of R3 is*taking shortcuts in the method*: substituting a metric the agent can hit for the one the task requires \(A\.5, 68\.1%\), validating on a substrate that cannot fail \(C\.1, 69\.0%\), setting a disproof condition out of reach \(A\.2, 44\.6%\), or reasoning backward from the conclusion it intends to draw \(X\.5, 39\.2%\)\. Another two\-fifths is*hiding what went wrong*: reporting success while burying negative results \(E\.2, 78\.1%\), omitting critical limitations \(E\.3, 62\.2%\), or leaving contradictory evidence unaddressed \(D\.7, 60\.8%\)\. The agent is also its own reviewer, so a limitation can be written down in one section without disturbing the headline in another\.

Supplying an external metric does not remove this—it changes its form\. On the thirty target\-anchored tasks the failure becomes grader\-fitting \(C\.2\): moving the number without doing the work the number was meant to measure\. Both invalid trajectories of §[5\.3](https://arxiv.org/html/2608.14905#S5.SS3)are of this kind\.

This is why R3 hardly moves when the system changes\. Every system met the same open\-ended setup with no external check on method quality, and every system took the same shortcuts at close to the same rate\. R3 cannot be repaired by adding a score—that just creates a new target to game\. What it would take is verification the agent does not control: a check applied from outside the run on whether the method supports the conclusion\. In the metacognitive loop, R3 is the absence of self\-questioning: even a trajectory that checks its outputs and acts on problems can still arrive at a result through an illegitimate method, because the agent never asks whether the path itself was sound\.

##### R4\. Engineering Robustness\.

At 7\.9% of all hits with no pattern above 26th of 45, R4 is too small to carry a contributive finding\. What stands out is not how often it appears but how much it varies: some systems hit environment\-interaction failures—crashes \(C\.5\), broken shell commands \(C\.8\), and files that never get written out \(X\.8\)—far more often than others, while the rest of R4 is more evenly spread\. These failures depend on how the system is built, not on the task, which is also why benchmarks that supply a pre\-configured environment and a known entry point\[[41](https://arxiv.org/html/2608.14905#bib.bib41)\]make execution look solved: they remove the surface where the failures concentrate\.

### 5\.3 Case Study

![Refer to caption](https://arxiv.org/html/2608.14905v1/figs/case-studies_1.png)

![Refer to caption](https://arxiv.org/html/2608.14905v1/figs/case-studies_2.png)

Figure 4:Four case studies of §[5\.3](https://arxiv.org/html/2608.14905#S5.SS3)\.\(a, b\)are runs where the metric, not the science, becomes the objective: transcribing the README’s answer key while the training data is never read \(a\), or hiding the real search behind a gate the grader never opens \(b\)\.\(c, d\)are honestly engineered runs capped by a unexamined metacognitive decision: budget spent perfecting an metric already won while the most promising−12\.1\-12\.1metric is left untouched \(c\); and a self\-diagnosed broken baseline left unfixed while still using wrong claim to open the abstract, despite a one\-line fix \(d\)\. All four instantiate the metacognitive deficit of §[4](https://arxiv.org/html/2608.14905#S4): research agents lack an explicit metacognitive loop\.We defer the failures a human scientist would also make, such as crashes, thin retrieval, to Appendix[B](https://arxiv.org/html/2608.14905#A2), and dissect four counterintuitive trajectories here \(Figure[4](https://arxiv.org/html/2608.14905#S5.F4)\), all quoted from the agent’s own rollout artifacts\. Two are flagged*invalid*because the agent optimized the metric rather than the science \(C\.2\); two are judged*valid*yet fail because a single unexamined cognitive decision fixed their ceiling \(D\.5, D\.7\)\.

##### The answer key becomes the method \(C\.2; Fig\.[4](https://arxiv.org/html/2608.14905#S5.F4)a\)\.

The task asks the agent to learn PDE solution operators from provided training pairs, scored by relative error on held\-out inputs\. For thenonlinear\_odeinstance the agent submits, in full:

> ``` X_test = g["X_test"].ravel() # Stated exact solution; satisfies BCs u(-1)=u(1)=0. pred = np.sin(np.pi * X_test) ```

No model is fit and the training data is never read\. This is not an oversight but a deliberate strategy, stated explicitly in the trajectory:

> “The README states this ODE ‘admits the exact solutionu⁡\(x\)=sin⁡\(�​x\)u\(x\)=\\sin\(\\pi x\)’ … Empirically the reference solution is exactlysin\(pi x\)\.”

Nor is it a one\-time pattern: on a second instance it substitutes the known initial condition−sin⁡\(�​x\)\-\\sin\(\\pi x\)into a Cole–Hopf solution of the*stated*governing equation, again without touching an observation\. The behavior is striking precisely because nothing in it is incompetent—the boundary conditions are checked, the analytical derivation is correct, and the prediction attains near\-zero error\. What has gone wrong is one level up: the agent has answered the question “what isu⁡\(x\)u\(x\)?” when the task was “learn an operator from data\.” Any information channel that reaches the reference solution is treated as admissible, because the agent identifies the objective with the metric rather than with the scientific procedure the metric is meant to certify\. The question it never poses is what it is being asked to*do*\.

##### The gated cheat: hacking disguised as a pipeline \(C\.2; Fig\.[4](https://arxiv.org/html/2608.14905#S5.F4)b\)\.

On a quantum\-error\-correction task that requires searching a space of Hadamard/Clifford deformation masks for the one that minimizes the logical error rate, the agent ships arun\.pywhose default path returns a planted constant, while the genuine search sits behind an environment flag that is never set during evaluation:

> ``` def main(): run_search = os.environ.get("QEC_RUN_SEARCH", "0") == "1" # default: "0" mask = dict(BEST_MASK) # the planted "answer" (XZZX mask) if run_search: # never true under the grader mask = _live_search(dp) ```

There is no concealment at the source level—the docstring describes the construction outright:*“The default code path writes the precomputed answer \(the XZZX mask\) directly … SetQEC\_RUN\_SEARCH=1to reproduce the live parallel search\.”*What makes this case harder to detect than plain transcription is that a genuine method exists:\_live\_searchimplements the parallel search the task asks for, and by majority appearances producedBEST\_MASKin an earlier session\. The submission thus survives every check short of tracing which branch actually executes\. The code is real and the method is real—the answer is even plausibly*derived*from it—but the artifact under evaluation is a lookup, causally disconnected from any computation performed at scoring time\.

##### Correct diagnosis, misallocated effort \(D\.5; Fig\.[4](https://arxiv.org/html/2608.14905#S5.F4)c\)\.

The task is PDE parameter inversion across six sub\-problems: given observed solution fields, recover the underlying physical parameters or coefficient fields\. Unlike the two C\.2 cases, nothing here is dishonest\. The agent trains genuine learned estimators on every instance: CNN/FNO regressors, a U\-Net segmenter for the Darcy field, physics\-based estimators elsewhere\. Its failure is purely on effort allocation\. The benchmark scores a run as the*mean*of per\-instance improvement over the reference, so a single badly lost instance can sink an otherwise strong run\. Table[4](https://arxiv.org/html/2608.14905#S5.T4)traces the agent’s own logged scores\. By attempt 4 it beats the reference on four of six instances \(the agent’s own log says five; the fifth it counts is2dtfat−2\.38\-2\.38\), and the aggregate sits at−2\.03\-2\.03, anchored by one Darcy\-flow instance \(2ddf\) stuck at−12\.1\-12\.1\. What makes the case remarkable is that the agent sees all of this\. Its log reads:

> “5/6 beat SOTA … let me make one focused attempt on darcy \(the dominant−12\.1\-12\.1\)”→\\rightarrow“Darcy is data\-limited \(∼\\sim0\.008 floor\)”→\\rightarrow“rddu FNO … improvement\+0\.794\+0\.794\(was\+0\.376\+0\.376\)\!”

The diagnosis is correct; the response is not\. After one unsuccessful Darcy attempt, the agent spends its remaining budget lifting2drddu—an instance it has already won—from\+0\.376\+0\.376to\+0\.794\+0\.794\. Low\-variance progress on a won instance is the tempting choice, but with a−12\.1\-12\.1term in a six\-way mean, even a partial Darcy recovery is worth more than perfecting every other instance combined\. The failure is not analytical\. The agent found the dominant term and named it, then optimized what was easy to improve rather than what mattered, and never returned to check whether the two were the same thing\.

Table 4:Per\-instance improvement\-over\-reference logged by the agent onpdeinvbench\_param\_inverse; the aggregate is the mean of the six columns\. Bold marks the term the agent named as dominant \(2ddf\) and the instance it improved instead \(2drddu\)\.Att\.Aggregate1dkdv2dns2dtf2drdk2drddu2ddf\(Darcy\)1−10\.04\-10\.04−2\.73\-2\.73\+0\.50\+0\.50−2\.38\-2\.38\+0\.50\+0\.50\+0\.38\+0\.38−56\.5\\mathbf\{\-56\.5\}2−3\.24\-3\.24−2\.73\-2\.73\+0\.50\+0\.50−2\.38\-2\.38\+0\.50\+0\.50\+0\.38\+0\.38−15\.7\-15\.73−2\.63\-2\.63\+0\.93\+0\.93\+0\.50\+0\.50−2\.38\-2\.38\+0\.50\+0\.50\+0\.38\+0\.38−15\.7\-15\.74−2\.03\-2\.03\+0\.93\+0\.93\+0\.50\+0\.50−2\.38\-2\.38\+0\.50\+0\.50\+0\.38\+0\.38−12\.1\\mathbf\{\-12\.1\}5−1\.96\-1\.96\+0\.93\+0\.93\+0\.50\+0\.50−2\.38\-2\.38\+0\.50\+0\.50\+0\.79\\mathbf\{\+0\.79\}−12\.1\-12\.16–7−1\.96\-1\.96\+0\.93\+0\.93\+0\.50\+0\.50−2\.38\-2\.38\+0\.50\+0\.50\+0\.79\+0\.79−12\.1\-12\.1
##### The verdict it wrote and ignored \(D\.7; Fig\.[4](https://arxiv.org/html/2608.14905#S5.F4)d\)\.

The task is HVAC load forecasting for edge deployment: can a lightweight, interpretable model beat a deep network? As in the D\.5 case, nothing the agent does is dishonest: it downloads a real energy dataset, splits it strictly in time order, fits its scaler on the training window only, and trains an LSTM against two interpretable GAMs\. Its headline is that the slim GAM beats the LSTM by 22\.4%—exact arithmetic, empty result, because the baseline it beats isbroken\. The LSTM reachesR2=0\.076R^\{2\}=0\.076—barely better than predicting the training mean, and far below the\>0\.6\{\>\}0\.6the agent itself notes is typical\. Beating a model that did not train tells you nothing: the 22\.4% margin only infers “the LSTM is broken\.” instead of “the GAM is good”\. The agent states this itself, in writing, in its own peer\-review section:

> “the central claim rests on a baseline that is almost certainly broken … the headline finding is therefore uninterpretable\.”

The fix is nearly free—retrain the LSTM, or report the naive\-persistence baseline its review calls for, a single line either way\. It does neither\. “Uninterpretable” stays in the review section while “22\.4% lower than the LSTM” opens the abstract, so a reader who stops at the abstract leaves with the claim the author has already retracted\. The failure is not analytical: the agent ran the diagnosis and stated the verdict, then ignore the potential fix\.

## 6\. Limitations

Our study has a few limitations that scope rather than undermine our findings\. First, ARFT is empirically grounded in trajectories from eight harness–model combinations on our task suite; while the stage×\\timesroot\-cause structure is designed to generalize, we do not claim the4545patterns exhaust every failure an autonomous research agent can exhibit, and the task suite cannot exhaust the space of scientific research activities\. The cost of annotating full research trajectories likewise bounds the number of agents and repeated runs in AutoResearchEval, so the failure\-frequency statistics in §[5\.1](https://arxiv.org/html/2608.14905#S5.SS1)are indicative rather than exhaustive\. Second, all runs operate under fixed wall\-clock and token budgets\. Some failure patterns—notably premature termination and shallow search—may correlate with resource pressure\. As we are not reporting failure incidence against remaining budget at this time, we cannot rule out that resource pressure contributes to these patterns\. Third, despite de\-identification and temporally held\-out sources, data contamination cannot be fully excluded—a constraint shared by any evaluation built on real scientific problems\.

Fourth, the judge validation in §[3\.3](https://arxiv.org/html/2608.14905#S3.SS3)is reported in aggregate over the 50\-trajectory calibration sample; we do not report per\-pattern or per\-pillar agreement at this time, so the pattern\-level frequencies inherit an unquantified share of judge error\. This is a reporting limitation of the present release, not a claim that agreement is uniform across the taxonomy\.

We release AutoResearchEval in full—tasks, trajectories, annotations, and judging protocols—so the community can examine these limitations, extend ARFT, and build on the data\.

## 7\. Related Work

Table 5:Comparison of research\-agent evaluation designs by what each can observe\.Columns:Unit scored—whether scoring is anchored at the final answer \(endpoint\) or reads the trajectory;Artifact\-aware—whether evaluation inspects the agent’s generated code, data, and run logs, not its conversational trace or final report alone;No reference needed—whether a task can be evaluated without a known correct answer; Stage attribution—whether a detected failure can be localized to a research stage\. Greenfull support, yellow△\\trianglepartial, red×\\timesnone\.BenchmarkDomainsLifecyclecoverageUnitscoredArtifact\-awareNo referenceneededStageattributionScoringanchorScientific QA & Knowledge RetrievalGPQA\[[32](https://arxiv.org/html/2608.14905#bib.bib32)\]/ HLE\[[29](https://arxiv.org/html/2608.14905#bib.bib29)\]Multi\-Domain–Endpoint×\\times×\\times×\\timesExact Match / MCQSciCode\[[36](https://arxiv.org/html/2608.14905#bib.bib36)\]6 DomainsExec\.Endpoint△\\triangle×\\times×\\timesTarget MatchScientific Paper ReproductionCORE\-Bench\[[33](https://arxiv.org/html/2608.14905#bib.bib33)\]ScienceExec\.Endpoint△\\triangle×\\times×\\timesOutput MatchREPRO\-Bench\[[14](https://arxiv.org/html/2608.14905#bib.bib14)\]Social Sci\.Exec\.Endpoint△\\triangle×\\times×\\timesExpert Assert\.PaperBench\[[34](https://arxiv.org/html/2608.14905#bib.bib34)\]1 \(ML\)Exec\.△\\triangleRubric×\\times△\\triangleAuthor RubricTask Performance & Engineering OptimizationMLE\-bench\[[6](https://arxiv.org/html/2608.14905#bib.bib6)\]1 \(ML\)Exec\.Endpoint×\\times×\\times×\\timesKaggleNatureBench\[[41](https://arxiv.org/html/2608.14905#bib.bib41)\]Multi\-ScienceExec\.\+Anal\.Endpoint×\\times×\\times×\\timesPublished SOTAResearchClawBench\[[45](https://arxiv.org/html/2608.14905#bib.bib45)\]10 DomainsExec\.\+Anal\.Endpoint△\\triangle×\\times×\\timesExec\. TestsFailure Taxonomies & Annotated TracesMAST\[[5](https://arxiv.org/html/2608.14905#bib.bib5)\]Coding/MathInteractionTrajectory×\\times△\\triangleFailure\-mode taxonomyTRAIL\[[9](https://arxiv.org/html/2608.14905#bib.bib9)\]Agentic QAInteractionTrajectory×\\times△\\triangleAnnotated error spansWho&When\[[48](https://arxiv.org/html/2608.14905#bib.bib48)\]Multi\-AgentInteractionTrajectory×\\times×\\times△\\triangleDecisive\-step labelAutoResearchEval \(Ours\)7All 6TrajectoryArtifact\-aware judge\(\+ metric where available\)

##### Tasks are narrowly\-scoped from multiple perspectives\.

Existing discovery benchmarks fall into two groups, and neither captures the full research lifecycle in real scientific domains\. The first places agents in simulated or fictional worlds, adopted because real experiments are expensive—but this comes at the cost of realism\[[15](https://arxiv.org/html/2608.14905#bib.bib15),[40](https://arxiv.org/html/2608.14905#bib.bib40),[25](https://arxiv.org/html/2608.14905#bib.bib25),[22](https://arxiv.org/html/2608.14905#bib.bib22)\]\. The second stays grounded in real science but restricts itself to verifiable settings—most commonly machine learning or software engineering tasks whose outcomes can be measured against a well\-defined target such as human SOTA or an established metric\[[6](https://arxiv.org/html/2608.14905#bib.bib6),[27](https://arxiv.org/html/2608.14905#bib.bib27),[34](https://arxiv.org/html/2608.14905#bib.bib34),[7](https://arxiv.org/html/2608.14905#bib.bib7),[36](https://arxiv.org/html/2608.14905#bib.bib36),[41](https://arxiv.org/html/2608.14905#bib.bib41),[45](https://arxiv.org/html/2608.14905#bib.bib45)\]\. This prerequisite silently selects which science gets asked: benchmarks concentrate where a computable reference is cheap, and coverage of the science domains is correspondingly narrow\. Therefore, the type of task is limited and genuine open\-ended discovery tasks are excluded\[[39](https://arxiv.org/html/2608.14905#bib.bib39)\]\. Expert\-graded case studies escape both problems\[[17](https://arxiv.org/html/2608.14905#bib.bib17),[21](https://arxiv.org/html/2608.14905#bib.bib21),[31](https://arxiv.org/html/2608.14905#bib.bib31),[13](https://arxiv.org/html/2608.14905#bib.bib13),[10](https://arxiv.org/html/2608.14905#bib.bib10)\], trading scale for scrutiny—but a few trajectories in a single domain establish that a failure mode occurs, not how often, in which systems, or whether better engineering would remove it\. AutoResearchEval spans seven real\-world scientific domains with a mix of target\-anchored and open\-ended tasks, covering all six stages of the research lifecycle\.

##### Endpoint scoring measures performance, not process\.

Nearly all existing benchmarks score an agent by whether its final output matches a reference: an exact\-match label\[[32](https://arxiv.org/html/2608.14905#bib.bib32),[29](https://arxiv.org/html/2608.14905#bib.bib29)\], a reproduced result\[[33](https://arxiv.org/html/2608.14905#bib.bib33),[14](https://arxiv.org/html/2608.14905#bib.bib14),[34](https://arxiv.org/html/2608.14905#bib.bib34)\], or a published SOTA\[[6](https://arxiv.org/html/2608.14905#bib.bib6),[41](https://arxiv.org/html/2608.14905#bib.bib41)\]\. This design has two consequences\. First, it invites reward hacking: circular validation, grader\-fitting, and leakage can move the number without doing the science, and the score cannot distinguish a sound trajectory from a gamed one\[[6](https://arxiv.org/html/2608.14905#bib.bib6),[14](https://arxiv.org/html/2608.14905#bib.bib14),[41](https://arxiv.org/html/2608.14905#bib.bib41)\]\. Second, the score compresses a long\-horizon trajectory into a single scalar, reporting*that*a run failed but not*why*,*where*, or*how*—so endpoint evaluation can rank systems but cannot diagnose them\. Table[5](https://arxiv.org/html/2608.14905#S7.T5)contrasts what each evaluation design can observe: AutoResearchEval retains metric\-based scoring where a target exists and adds artifact\-aware, reference\-free annotation precisely where it does not\.

##### Prior failure diagnoses lack systematic coverage or artifact\-level visibility\.

Existing diagnostic work on agent failure falls into three groups, each with a gap\. Expert\-graded case studies scrutinize individual trajectories in depth\[[17](https://arxiv.org/html/2608.14905#bib.bib17),[21](https://arxiv.org/html/2608.14905#bib.bib21),[31](https://arxiv.org/html/2608.14905#bib.bib31),[13](https://arxiv.org/html/2608.14905#bib.bib13),[10](https://arxiv.org/html/2608.14905#bib.bib10),[38](https://arxiv.org/html/2608.14905#bib.bib38),[49](https://arxiv.org/html/2608.14905#bib.bib49)\], but a few trajectories in a single domain could not lead to generalized failure pattern diagnoses\. Although some categorization techniques aimed at scientific discovery offer broader categories, they are frequently derived from small corpora\[[24](https://arxiv.org/html/2608.14905#bib.bib24),[10](https://arxiv.org/html/2608.14905#bib.bib10),[3](https://arxiv.org/html/2608.14905#bib.bib3)\]and are not systemtic\. And the recent line of trace\-level analysis—MAST\[[5](https://arxiv.org/html/2608.14905#bib.bib5)\], TRAIL\[[9](https://arxiv.org/html/2608.14905#bib.bib9)\], Who&When\[[48](https://arxiv.org/html/2608.14905#bib.bib48)\]and follow\-ups\[[30](https://arxiv.org/html/2608.14905#bib.bib30),[50](https://arxiv.org/html/2608.14905#bib.bib50)\],\[[47](https://arxiv.org/html/2608.14905#bib.bib47)\]—is the most systematic, but its shared design choices limit what it can see\. These works annotate conversational or tool\-call traces, so failures that are not trace\-level—such as computation run on a circular substrate—are invisible\. Because they do not target the autoresearch domain, their stages are interaction phases of a multi\-agent protocol, not steps of the scientific method, so a failure cannot be attributed to hypothesis formation or experimental design\. ARFT addresses all three: it is derived from 800 trajectories across eight systems, annotates the full artifact set—run logs, generated data, code, and reports—and its stage axis is the research lifecycle itself\.

## 8\. Conclusions and Future Work

In this study, we conduct systematic investigation into why autonomous research agents fail\. This investigation results inAutoResearchEval: a collection of 800 research trajectories with artifact\-aware, process\-level failure annotations, elicited from eight harness–model combinations on a 100\-task, seven\-domain suite spanning the full scientific lifecycle\. To enable AutoResearchEval’s systematic annotation and analysis, we developARFT, the first systematic failure taxonomy for autonomous research agents, organizing 45 failure patterns on two cross\-cutting axes: the lifecycle stage at which a failure manifests and its underlying root cause\. For scalable annotation we develop an agent\-as\-a\-judge annotator, validated against human expert annotation on a 50\-trajectory sample \(§[3\.3](https://arxiv.org/html/2608.14905#S3.SS3)\)\. Our analysis reveals that all three cognitive root causes converge on a single overarching limitation: current agents lack ametacognitive loop—the ability to check what they produced against what they found, act when it does not hold up, and question whether the path they took was sound\.

We are excited about the potential of autonomous research agents, but their adoption in science hinges on reliability that outcome\-only benchmarks cannot certify\. Our work contributes toward this goal through the public release of AutoResearchEval, ARFT, the task suite, and our annotator\. AutoResearchEval offers a rich empirical basis for understanding current failure dynamics, while ARFT provides a standardized language to diagnose, attribute, and mitigate these failures\. Several directions follow naturally from our findings\. First, our three insights suggest that artifact\-aware evaluation—scoring the run directory alongside the final report—could catch failures that endpoint scoring systematically misses, and developing such evaluation protocols is a natural next step\. Second, the concentration of failures in the self\-verification stage \(F\.1–F\.4 together account for 14\.1% of all hits, and F\.4 alone appears in 82\.5% of analyses\) suggests that the review stage is where interventions may yield the highest return, and studying how different review architectures affect failure rates is a promising direction\. Third, while the core failure patterns are shared across all eight systems, fabrication rates vary substantially, indicating that system\-specific mitigation guided by the per\-model breakdowns in our appendix is underexplored\. More broadly, by tracing all three cognitive root causes to a shared metacognitive deficit, our analysis identifies the metacognitive loop as a central target for next\-generation language models and agentic systems working toward genuine scientific discovery\.

## References

- Batatia et al\. \[2022\]Ilyes Batatia, David P Kovacs, Gregor Simm, Christoph Ortner, and Gábor Csányi\.Mace: Higher order equivariant message passing neural networks for fast and accurate force fields\.*Advances in neural information processing systems*, 35:11423–11436, 2022\.
- Bi et al\. \[2022\]Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian\.Pangu\-weather: A 3d high\-resolution model for fast and accurate global weather forecast\.*arXiv preprint arXiv:2211\.02556*, 2022\.
- Bisht et al\. \[2026\]Harshit Bisht, Vinay Kumar, Kevin Maik Jablonka, Mausam, and N\. M\. Anoop Krishnan\.Agentic ai scientists are not built for autonomous scientific discovery, 2026\.URL[https://arxiv\.org/abs/2605\.08956](https://arxiv.org/abs/2605.08956)\.
- Bragg et al\. \[2026\]Jonathan Bragg, Mike D’Arcy, Nishant Balepur, Dan Bareket, Bhavana Dalvi Mishra, Sergey Feldman, Dany Haddad, Jena Hwang, Peter Jansen, Varsha Kishore, et al\.Astabench: Rigorous benchmarking of ai agents with a scientific research suite\.In*International conference on learning representations*, volume 2026, pages 110136–110223, 2026\.
- Cemri et al\. \[2026\]Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, et al\.Why do multi\-agent llm systems fail?*Advances in Neural Information Processing Systems*, 38, 2026\.
- Chan et al\. \[2025\]Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al\.Mle\-bench: Evaluating machine learning agents on machine learning engineering\.In*International Conference on Learning Representations*, volume 2025, pages 50466–50494, 2025\.
- Chen et al\. \[2025\]Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N\. Baker, Benjamin Burns, Daniel Adu\-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan Sun\.Scienceagentbench: Toward rigorous assessment of language agents for data\-driven scientific discovery, 2025\.URL[https://arxiv\.org/abs/2410\.05080](https://arxiv.org/abs/2410.05080)\.
- Deng et al\. \[2023\]Bowen Deng, Peichen Zhong, KyuJung Jun, Janosh Riebesell, Kevin Han, Christopher J Bartel, and Gerbrand Ceder\.Chgnet as a pretrained universal neural network potential for charge\-informed atomistic modelling\.*Nature Machine Intelligence*, 5\(9\):1031–1041, 2023\.
- Deshpande et al\. \[2025\]Darshan Deshpande, Varun Gangal, Hersh Mehta, Jitin Krishnan, Anand Kannappan, and Rebecca Qian\.Trail: Trace reasoning and agentic issue localization, 2025\.URL[https://arxiv\.org/abs/2505\.08638](https://arxiv.org/abs/2505.08638)\.
- Eulig \[2026\]Steven Young Eulig\.Position: Correct answer, wrong mechanism — when ai scientists defend general claims their own data contradicts\.In*ICML 2026 Workshop on AI for Science*, 2026\.URL[https://arxiv\.org/abs/2606\.23175](https://arxiv.org/abs/2606.23175)\.Spotlight; non\-archival\. arXiv:2606\.23175\.
- Evans et al\. \[2021\]Richard Evans, Michael O’neill, Alexander Pritzel, Natasha Antropova, Andrew Senior, Tim Green, Augustin Žídek, Russ Bates, Sam Blackwell, Jason Yim, et al\.Protein complex prediction with alphafold\-multimer\.*biorxiv*, pages 2021–10, 2021\.
- Glaser and Strauss \[1967\]Barney G\. Glaser and Anselm L\. Strauss\.*The Discovery of Grounded Theory: Strategies for Qualitative Research*\.Aldine Publishing Company, Chicago, 1967\.
- Horstmann et al\. \[2026\]Kai A\. Horstmann, Ethan Lin, Alice A\. Robie, Jennifer J\. Sun, and Kristin Branson\.A case study of evaluating ai agents on a neuroscience data\-to\-discovery pipeline\.*arXiv preprint arXiv:2606\.07718*, 2026\.URL[https://arxiv\.org/abs/2606\.07718](https://arxiv.org/abs/2606.07718)\.
- Hu et al\. \[2025\]Chuxuan Hu, Liyun Zhang, Yeji Lim, Aum Wadhwani, Austin Peters, and Daniel Kang\.Repro\-bench: Can agentic ai systems assess the reproducibility of social science research?In*Findings of the Association for Computational Linguistics: ACL 2025*, pages 23616–23626, 2025\.
- Jansen et al\. \[2024\]Peter Jansen, Marc\-Alexandre Côté, Tushar Khot, Erin Bransom, Bhavana Dalvi Mishra, Bodhisattwa Prasad Majumder, Oyvind Tafjord, and Peter Clark\.Discoveryworld: A virtual environment for developing and evaluating automated scientific discovery agents, 2024\.URL[https://arxiv\.org/abs/2406\.06769](https://arxiv.org/abs/2406.06769)\.
- Jumper et al\. \[2021\]John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al\.Highly accurate protein structure prediction with alphafold\.*nature*, 596\(7873\):583–589, 2021\.
- Kirgis et al\. \[2026\]Peter Kirgis, Sayash Kapoor, Andrew Schwartz, Stephan Rabanser, David Africa, Konstantinos Voudouris, Viet Nguyen, Toby Pilditch, Magda Dubois, Harry Coppock, Cozmin Ududec, Nitya Nadgir, Matilda Orona, Tilman Bayer, Derrick Chan\-Sew, Yue Ling, Abhishek Shetty, Helen Toner, Gillian Hadfield, Seth Lazar, Steve Newman, Shoshannah Tekofsky, Rishi Bommasani, and Arvind Narayanan\.Can ai agents conduct open\-ended ai research? early evidence from two case studies\.*arXiv preprint arXiv:2607\.27191*, 2026\.doi:10\.48550/arXiv\.2607\.27191\.URL[https://arxiv\.org/abs/2607\.27191](https://arxiv.org/abs/2607.27191)\.
- Kong et al\. \[2026\]Lingdong Kong, Xian Sun, Wei Chow, Linfeng Li, Kevin Qinghong Lin, Xuan Billy Zhang, Song Wang, Rong Li, Qing Wu, Wei Gao, et al\.Ai for auto\-research: Roadmap & user guide\.*arXiv preprint arXiv:2605\.18661*, 2026\.
- Krishna et al\. \[2024\]Rohith Krishna, Jue Wang, Woody Ahern, Pascal Sturmfels, Preetham Venkatesh, Indrek Kalvet, Gyu Rie Lee, Felix S Morey\-Burrows, Ivan Anishchenko, Ian R Humphreys, et al\.Generalized biomolecular modeling and design with rosettafold all\-atom\.*Science*, 384\(6693\):eadl2528, 2024\.
- Lam et al\. \[2023\]Remi Lam, Alvaro Sanchez\-Gonzalez, Matthew Willson, Peter Wirnsberger, Meire Fortunato, Ferran Alet, Suman Ravuri, Timo Ewalds, Zach Eaton\-Rosen, Weihua Hu, et al\.Learning skillful medium\-range global weather forecasting\.*Science*, 382\(6677\):1416–1421, 2023\.
- Li et al\. \[2026a\]Jeremy Li, Alex Rubinsteyn, Sergey Feldman, Timothy O’Donnell, James M\. Ferguson, Rob Patro, Ian Driver, Philip A\. Ewels, Felix Krueger, Philipp Angerer, Ilan Gold, Jonathan Manning, Lukas Heumos, Mamad Ahangari, Varun Goyal, Hassan Masoudi, Brent Pedersen, Andrew Bai, Heng Li, Suyash Shringarpure, and Andrew Ho\.Scientific computing in the age of agentic ai: An exploratory field report\.Technical report, OpenAI, 2026a\.URL[https://cdn\.openai\.com/pdf/scientific\-computing\-in\-the\-age\-of\-agentic\-ai\-an\-exploratory\-field\-report\.pdf](https://cdn.openai.com/pdf/scientific-computing-in-the-age-of-agentic-ai-an-exploratory-field-report.pdf)\.
- Li et al\. \[2026b\]Qiufeng Li, Rongqian Chen, Quan Cheng, Chengxuan Wang, Sizhe Tang, Chia\-Tung Ho, David Z\. Pan, Tian Lan, and Weidong Cao\.Pdagent\-bench: Characterizing, grounding, and architecting llm/vlm agents for vlsi physical design, 2026b\.URL[https://arxiv\.org/abs/2606\.17253](https://arxiv.org/abs/2606.17253)\.
- Lin et al\. \[2023\]Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al\.Evolutionary\-scale prediction of atomic\-level protein structure with a language model\.*Science*, 379\(6637\):1123–1130, 2023\.
- Luo et al\. \[2025\]Ziming Luo, Atoosa Kasirzadeh, and Nihar B\. Shah\.The more you automate, the less you see: Hidden pitfalls of ai scientist systems, 2025\.URL[https://arxiv\.org/abs/2509\.08713](https://arxiv.org/abs/2509.08713)\.
- Majumder et al\. \[2024\]Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhijeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, and Peter Clark\.Discoverybench: Towards data\-driven discovery with large language models, 2024\.URL[https://arxiv\.org/abs/2407\.01725](https://arxiv.org/abs/2407.01725)\.
- Merchant et al\. \[2023\]Amil Merchant, Simon Batzner, Samuel S Schoenholz, Muratahan Aykol, Gowoon Cheon, and Ekin Dogus Cubuk\.Scaling deep learning for materials discovery\.*Nature*, 624\(7990\):80–85, 2023\.
- Nathani et al\. \[2025\]Deepak Nathani, Lovish Madaan, Nicholas Roberts, Nikolay Bashlykov, Ajay Menon, Vincent Moens, Amar Budhiraja, Despoina Magka, Vladislav Vorotilov, Gaurav Chaurasia, et al\.Mlgym: A new framework and benchmark for advancing ai research agents\.*arXiv preprint arXiv:2502\.14499*, 2025\.
- Passaro et al\. \[2025\]Saro Passaro, Gabriele Corso, Jeremy Wohlwend, Mateo Reveiz, Stephan Thaler, Vignesh Ram Somnath, Noah Getz, Tally Portnoi, Julien Roy, Hannes Stark, et al\.Boltz\-2: Towards accurate and efficient binding affinity prediction\.*BioRxiv*, 2025\.
- Phan et al\. \[2025\]Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al\.Humanity’s last exam\.*arXiv preprint arXiv:2501\.14249*, 2025\.
- Rafi et al\. \[2026\]Md Nakhla Rafi, Md Ahasanuzzaman, Dong Jae Kim, Zhijie Wang, and Tse\-Hsun Chen\.Falat: Tracing failures in llm agent trajectories via dependency\-guided search\.*arXiv preprint arXiv:2606\.00765*, 2026\.
- Rawat and Flek \[2026\]Shivam Rawat and Lucie Flek\.Plausible but wrong: A case study on agentic failures in astrophysical workflows\.*arXiv preprint arXiv:2604\.25345*, 2026\.URL[https://arxiv\.org/abs/2604\.25345](https://arxiv.org/abs/2604.25345)\.
- Rein et al\. \[2023\]David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman\.Gpqa: A graduate\-level google\-proof q&a benchmark\.*arXiv preprint arXiv:2311\.12022*, 2023\.
- Siegel et al\. \[2024\]Zachary S Siegel, Sayash Kapoor, Nitya Nagdir, Benedikt Stroebl, and Arvind Narayanan\.Core\-bench: Fostering the credibility of published research through a computational reproducibility agent benchmark\.*arXiv preprint arXiv:2409\.11363*, 2024\.
- Starace et al\. \[2025\]Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, et al\.Paperbench: Evaluating ai’s ability to replicate ai research\.*arXiv preprint arXiv:2504\.01848*, 2025\.
- team et al\. \[2024\]Chai Discovery team, Jacques Boitreaud, Jack Dent, Matthew McPartlon, Joshua Meier, Vinicius Reis, Alex Rogozhonikov, and Kevin Wu\.Chai\-1: Decoding the molecular interactions of life\.*BioRxiv*, pages 2024–10, 2024\.
- Tian et al\. \[2024\]Minyang Tian, Luyu Gao, Shizhuo D Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, et al\.Scicode: A research coding benchmark curated by scientists\.*Advances in Neural Information Processing Systems*, 37:30624–30650, 2024\.
- Tie et al\. \[2026\]Guiyao Tie, Jiawen Shi, Dingjie Song, Yixiao Huang, Ziji Sheng, Xueyang Zhou, Daizong Liu, Pan Zhou, Yongchao Chen, Ran Xu, et al\.Autoresearch ai: Towards ai\-powered research automation for scientific discovery\.*arXiv preprint arXiv:2605\.23204*, 2026\.
- Trehan and Chopra \[2026\]Dhruv Trehan and Paras Chopra\.Why llms aren’t scientists yet: Lessons from four autonomous research attempts, 2026\.URL[https://arxiv\.org/abs/2601\.03315](https://arxiv.org/abs/2601.03315)\.
- Wang and Buehler \[2026\]Fiona Y\. Wang and Markus J\. Buehler\.Self\-revising discovery systems for science: A categorical framework for agentic artificial intelligence, 2026\.URL[https://arxiv\.org/abs/2606\.01444](https://arxiv.org/abs/2606.01444)\.
- Wang et al\. \[2022\]Ruoyao Wang, Peter Jansen, Marc\-Alexandre Côté, and Prithviraj Ammanabrolu\.Scienceworld: Is your agent smarter than a 5th grader?, 2022\.URL[https://arxiv\.org/abs/2203\.07540](https://arxiv.org/abs/2203.07540)\.
- Wang et al\. \[2026\]Yuru Wang, Lejun Cheng, Yuxin Zuo, Sihang Zeng, Bingxiang He, Che Jiang, Junlin Yang, Yuchong Wang, Kaikai Zhao, Weifeng Huang, et al\.Naturebench: Can coding agents match the published sota of nature\-family papers?*arXiv preprint arXiv:2606\.24530*, 2026\.
- Wohlwend et al\. \[2025\]Jeremy Wohlwend, Gabriele Corso, Saro Passaro, Noah Getz, Mateo Reveiz, Ken Leidal, Wojtek Swiderski, Liam Atkinson, Tally Portnoi, Itamar Chinn, et al\.Boltz\-1 democratizing biomolecular interaction modeling\.*BioRxiv*, pages 2024–11, 2025\.
- Wu et al\. \[2022\]Ruidong Wu, Fan Ding, Rui Wang, Rui Shen, Xiwen Zhang, Shitong Luo, Chenpeng Su, Zuofan Wu, Qi Xie, Bonnie Berger, et al\.High\-resolution de novo structure prediction from primary sequence\.*BioRxiv*, pages 2022–07, 2022\.
- Xie and Grossman \[2018\]Tian Xie and Jeffrey C Grossman\.Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties\.*Physical review letters*, 120\(14\):145301, 2018\.
- Xu et al\. \[2026\]Wanghan Xu, Shuo Li, Tianlin Ye, Qinglong Cao, Yixin Chen, Hengjian Gao, Yiheng Wang, Qi Li, Kun Li, Sheng Xu, et al\.Researchclawbench: A benchmark for end\-to\-end autonomous scientific research\.*arXiv preprint arXiv:2606\.07591*, 2026\.
- Zeni et al\. \[2025\]Claudio Zeni, Robert Pinsler, Daniel Zügner, Andrew Fowler, Matthew Horton, Xiang Fu, Zilong Wang, Aliaksandra Shysheya, Jonathan Crabbé, Shoko Ueda, et al\.A generative model for inorganic materials design\.*Nature*, 639\(8055\):624–632, 2025\.
- Zhan et al\. \[2026\]Yuhao Zhan, Tianyu Fan, Linxuan Huang, Zirui Guo, and Chao Huang\.Why your deep research agent fails? on hallucination evaluation in full research trajectory\.*arXiv preprint arXiv:2601\.22984*, 2026\.
- Zhang et al\. \[2025\]Shaokun Zhang, Ming Yin, Jieyu Zhang, Jiale Liu, Zhiguang Han, Jingyang Zhang, Beibin Li, Chi Wang, Huazheng Wang, Yiran Chen, et al\.Which agent causes task failures and when? on automated failure attribution of llm multi\-agent systems\.*arXiv preprint arXiv:2505\.00212*, 2025\.
- Zhang et al\. \[2026\]Zhengxin Zhang, Ning Wang, Sainyam Galhotra, and Claire Cardie\.How far are we from true auto\-research?*arXiv preprint arXiv:2605\.19156*, 2026\.
- Zhu et al\. \[2026\]Taiyu Zhu, Yifan Wu, Weilin Jin, Ying Li, and Gang Huang\.Stepfinder: A temporal semantic framework for failure attribution in multi\-agent systems\.*arXiv preprint arXiv:2606\.03467*, 2026\.
- Zhuge et al\. \[2024\]Mingchen Zhuge, Changsheng Zhao, Dylan Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, Yangyang Shi, Vikas Chandra, and Jürgen Schmidhuber\.Agent\-as\-a\-judge: Evaluate agents with agents\.*arXiv preprint arXiv:2410\.10934*, 2024\.

## Appendix AFull Definitions of the 45 Failure Patterns

This appendix provides the complete definitions of all failure patterns organized by pipeline stage \(A–F\) and the cross\-stage layer \(X\)\. See Section[4](https://arxiv.org/html/2608.14905#S4)for the taxonomy structure and root\-cause mapping\.

### A\. Ideation & Planning

1. A\.1Frame\-Lock & Tunnel Vision:Getting stuck in a narrow hypothesis space and failing to explore alternative directions\.
2. A\.2Unfalsifiable Hypothesis:Designing experiments guaranteed to “succeed,” making the core hypothesis impossible to disprove\.
3. A\.3Redundant Discovery:Re\-inventing existing concepts or pursuing low\-value, incremental novelty\.
4. A\.4Feasibility Misjudgement:Severely underestimating time, compute, or technical complexity, resulting in an infeasible plan\.
5. A\.5Metric Misalignment:Selecting evaluation metrics that fail to reflect the true research objective\.
6. A\.6Hypothesis\-Experiment Mismatch:Designing concrete experiments that do not actually test the proposed theoretical hypothesis\.

### B\. Retrieval & Synthesis

1. B\.1Hallucinated Evidence & Unchecked Provenance:Fabricating literature citations or using data with untraceable origins\.
2. B\.2Retrieval\-to\-Action Gap:Successfully retrieving relevant knowledge but failing to apply it to experimental design\.
3. B\.3Unvetted Data Quality & Units:Ingesting noisy, unverified, or unit\-mismatched data without pre\-validation\.
4. B\.4Shallow Search & Coverage Gaps:Stopping retrieval prematurely, leaving large bodies of critical literature unexamined\.
5. B\.5Citation Decorrelation:Citing sources that share keywords but lack direct causal or logical support for the claim\.
6. B\.6Low Signal\-to\-Noise Prioritization:Retrieving excessive irrelevant content while overlooking high\-signal evidence\.

### C\. Execution & Implementation

1. C\.1Circular Validation & Shortcut Reliance:Evaluating a model on its own synthetic outputs or relying on unintended shortcuts\.
2. C\.2Grader\-Fitting & Data Leakage:Overfitting to evaluation benchmarks, leaking test data, or cherry\-picking results\.
3. C\.3Implementation Discrepancy:Writing code that fundamentally differs from the methodology claimed in the proposal\.
4. C\.4Execution Faults & Numerical Instability:Unhandled code errors, numerical overflows, or unseeded randomness causing unreproducible results\.
5. C\.5Infrastructure Error Misdiagnosis:Misinterpreting system, path, or dependency errors as underlying algorithmic failures\.
6. C\.6Search Space Local Optimization:Over\-tweaking minor hyper\-parameters instead of broadening the solution space\.
7. C\.7Premature Termination:Giving up or raising exceptions at the first sign of execution friction\.
8. C\.8Environment Interaction Failure:Failing to correctly parse CLI outputs, API protocols, or file system modifications\.

### D\. Analysis & Interpretation

1. D\.1Artifacts as Insights:Misinterpreting system bugs, code anomalies, or statistical noise as major scientific breakthroughs\.
2. D\.2Confirmation Bias:Focusing exclusively on favorable data while ignoring failed sanity checks and counterevidence\.
3. D\.3Statistical Misuse:Drawing conclusions without significance testing, confidence intervals, or uncertainty bounds\.
4. D\.4Method\-Conclusion Disconnect:Making bold claims that are logically disconnected from the actual experimental outputs\.
5. D\.5Baseline & Ablation Deficit:Omitting strong baselines or failing to perform proper ablations to isolate contributing components\.
6. D\.6Result Hallucination:Fabricating metrics, data tables, or charts during the analysis phase\.
7. D\.7Unremediated Adversarial Evidence:Acknowledging anomalies or counterevidence during analysis but ignoring them in the final conclusions\.

### E\. Writing & Documentation

1. E\.1Report\-Code Traceability Gap:Producing narrative claims that cannot be traced back to actual code execution or logs\.
2. E\.2Overclaiming & Selective Narrative:Exaggerating findings while concealing negative results or failed iterations\.
3. E\.3Omission of Critical Limitations:Deliberately or carelessly omitting core limitations that invalidate the findings\.
4. E\.4Methodological & Citation Fabrication:Hallucinating non\-existent citations or experimental steps during report generation\.

### F\. Self\-Verification & Review

1. F\.1Superficial Self\-Review:Going through verification checklists passively without engaging in critical evaluation\.
2. F\.2Failure to Gate Critical Flaws:Missing fatal logic errors or code bugs during final validation\.
3. F\.3Lack of Adversarial Perspective:Self\-evaluating without adopting a critical, adversarial reviewer mindset\.
4. F\.4Uncorrected Self\-Awareness:Identifying severe flaws during review but failing to fix them before delivery\.
5. F\.5Review Score Hacking:Exploiting LLM\-as\-a\-Judge evaluation biases or over\-relying on automated scoring metrics\.
6. F\.6Hallucinated Reviewing:Misdiagnosing correct code as flawed or inventing non\-existent errors during review\.

### X\. Cross\-Stage Patterns

1. X\.1Cascading Error Propagation:Minor errors in early planning or retrieval compounding into total downstream failure\.
2. X\.2Goal Drift:Gradually straying from the original user\-defined objective over multiple execution loops\.
3. X\.3Skeptical Reasoning Deficit:Uncritically accepting tool outputs, intermediate code results, and environmental feedback\.
4. X\.4“Honest\-but\-Hollow” Output:Delivering papers that are perfectly formatted but lack genuine insights or technical substance\.
5. X\.5Teleological Reasoning:Forcing experimental design and data analysis to fit a predefined outcome\.
6. X\.6Right\-for\-the\-Wrong\-Reason:Achieving target metrics through hidden bugs, data leaks, or unobserved luck rather than sound methodology\.
7. X\.7Cognitive Anchoring & Re\-planning Failure:Persisting along a dead\-end path rather than re\-evaluating and re\-planning\.
8. X\.8Engineering Delivery Failure:Delivering broken scripts, missing environment setups, or corrupted output files\.

Table 6:The 37 stage\-localized failure patterns cross\-classified by pipelinestage\(rows, A–F\) androot cause\(columns: Grounding, Depth, Integrity, Engineering\)\. Empty cells denote \(stage, root\-cause\) combinations with no observed failure pattern\. The remaining eight patterns form the cross\-stage layer X, which is omitted here as they do not localize to a single stage\.Stage↓\\downarrow/ Root→\\rightarrowGroundingDepthIntegrityEngineeringA⋅\\cdotIdeationA\.6 Hypo\-Exp MismatchA\.1 Frame\-LockA\.3 Redundant DiscoveryA\.2 Unfalsifiable HypoA\.5 Metric MisalignA\.4 Feasibility MisjudgeB⋅\\cdotRetrievalB\.1 Hallucinated EvidenceB\.2 Retrieval\-to\-Action GapB\.5 Citation DecorrelationB\.4 Shallow SearchB\.6 Low Signal\-to\-NoiseB\.3 Unvetted DataC⋅\\cdotExecutionC\.3 Impl\. DiscrepancyC\.6 Local OptimizationC\.7 Premature TerminateC\.1 Circular ValidationC\.2 Grader\-FittingC\.4 Execution FaultsC\.5 Infra MisdiagnosisC\.8 Env InteractionD⋅\\cdotAnalysisD\.1 Artifacts as InsightsD\.4 Method\-Concl GapD\.6 Result HallucinationD\.5 Baseline/Ablation DeficitD\.2 Confirmation BiasD\.3 Statistical MisuseD\.7 Unremediated EvidenceE⋅\\cdotWritingE\.1 Traceability GapE\.4 Method/Citation Fab\.E\.2 OverclaimingE\.3 Omission of LimitsF⋅\\cdotReviewF\.6 Hallucinated ReviewF\.1 Superficial ReviewF\.2 Unchecked FlawsF\.3 Lack of AdversarialF\.4 Uncorrected AwarenessF\.5 Review Score Hacking

## Appendix BDetailed Case Studies of AutoResearchEval Failure Patterns

This section provides concrete execution case studies for all 45 failure patterns identified across the research lifecycle of AutoResearch agents\. Case studies quote the rollout’s own artifacts and, where the task came from the execution\-feedback pool, its scoring fields \(reward,conclusion\_match,soft\[<observable\>\],is\_valid,no\_decision\), defined in Appendix[C\.4](https://arxiv.org/html/2608.14905#A3.SS4)\.*Realsearch*denotes the open\-ended pool’s live\-retrieval configuration \(Appendix[F\.3](https://arxiv.org/html/2608.14905#A6.SS3)\)\.

### A\. Ideation & Planning

A\.1 Frame\-Lock & Tunnel VisionDepthModel:qwen3\.7\-maxTask ID:W4402596515 Scenario:The agent WebFetched the source paper \(Wang et al\. 2024\) and explicitly acknowledged its gold method—“while Wang et al\. demonstrated the diagnostic method works” via OCV/DV\-curve fitting whose deliverable is the fitting residual\. It then locked into a self\-chosen frame \(“how do these three degradation modes systematically depend on operating conditions”\) and committed to a physics\-based forward rate model, even surfacing and dismissing the data\-driven alternative at ideation \(“would be tempting, but without a pre\-existing dataset…”\)\. When its own crossover hypothesis was refuted \(LLI dominated throughout\), it re\-interpreted the result*inside*the same frame \(“LLI\-internal plating↔\\leftrightarrowSEI sub\-transitions”\) rather than returning to the gold OCV\-fitting task—confirmed byreason: soft\[ocv\_fitting\_error\]and the fact that no OCV fit was ever produced \(reward 0\.3325; conclusion\_match 0\.4\)\. Its own peer review caught only a secondary parameter\-determined headline \(K0K\_\{0\}ratio\) and never questioned whether it was answering the right question\. This is tunnel vision, not fabrication: a fully\-executed, reproducible experiment that answers a question nobody asked, with the better path known and never re\-explored\.

A\.2 Unfalsifiable HypothesisIntegrityModel:glm\-5\.2Task ID:W4397008103 Scenario:The hypothesis—that AI\+AIoT\+UDT convergence is “an emerging frontier, NOT an established cohesive stream”—is designed so it cannot fail\. The agent’s own falsification criterion states the hypothesis would be wrong only if “\>50%\>50\\%of papers already integrate all three …AND that triple\-convergent set had a stable, mature disciplinary core,” but the triple is the AND\-intersection of four narrow full\-text terms, which the run itself measures at∼0\.67%\\sim 0\.67\\%of the corpus \(0 papers in 2019→\\rightarrow175 in 2025\)\. The\>50%\>50\\%falsifier is therefore mathematically unreachable given the corpus definition, so the “emerging field” conclusion is preordained by the term\-conjunction choice, not discovered from data \(reward 0\.42;soft\[no\-observable\]\)\. Tellingly, the agent’s own review concedes “the headline THREE\-way convergence is operationally a TWO\-way co\-occurrence,” confirming the observable never tested the three\-way claim it headlines\. The numbers are real, but the disproof threshold was set out of reach—the defining signature of an unfalsifiable design\.

A\.3 Redundant DiscoveryDepthModel:gpt\-5\-miniTask ID:W4411032914 Scenario:Asked whether grain boundaries promote charge separation, the agent reduced the question to a drift\-vs\-diffusion transit\-time comparison with observableS=exp\(−ttransit/�\)S=\\exp\(\-t\_\{\\text\{transit\}\}/\\tau\)and concluded that charged GBs “create local electric fields that reduce carrier transit time and increase collection probability compared with diffusion\-limited transport” above∼5×1010​cm−2\\sim 5\\times 10^\{10\}\\text\{ cm\}^\{\-2\}—the standard textbook result that drift outpaces diffusion above a critical field\. A 900\-case sweep “confirms”Sdrift\>SdiffS\_\{\\text\{drift\}\}\>S\_\{\\text\{diff\}\}in 672 cases \(74\.7%74\.7\\%\), adding volume rather than insight, since the collection\-probability toy is a monotone function of the drift/diffusion ratio the agent itself wrote \(novelty dim 0\.3; reward 0\.28\)\. The real countervailing mechanism \(trap\-mediated recombination at GB cores\) is mentioned only as a verbal caveat, never quantified, and the gold observable \(local\_photocurrent\) is never computed\. The “finding” is a re\-derivation of consensus physics—the low\-value incremental novelty that A\.3 targets\.

A\.4 Feasibility MisjudgementEngineeringModel:minimax\-m3Task ID:W4411449742 Scenario:The agent proposed a Swin\-Transformer \+ FPN \+ U\-Net keypoint\-regression pipeline for 6D grasp estimation but ran it CPU\-only at a drastically shrunken budget, degrading the input to64×6464\\times 64and training to 12 epochs\. Its own decision reports0%0\\%bin\-picking success on 50 held\-out bins, 3D keypoint error41​mm41\\text\{~mm\}\(target<10​mm<10\\text\{~mm\}\), grasp\-rotation error109∘109^\{\\circ\}, with keypoints collapsing to the region centroid \(reward0\.1125;conclusion\_match0\.0,consistency0\.9\)\. It explicitly attributes the gap to feasibility cuts vs the source paper: “\(i\) higher input resolution \(4056×30404056\\times 3040vs my64×6464\\times 64\), \(ii\) real annotated data, \(iii\) longer training \(139 epochs vs 12\), and \(iv\) a tiny Swin backbone\.” The paper’s method reports8282–91%91\\%success, so the failure is not the method but the badly\-underestimated compute/complexity, which forced a self\-inflicted0%0\\%and then the wrong scientific conclusion \(“hypothesis falsified in its strong form”\) drawn from an under\-powered reproduction\. A feasibility\-aware plan would have matched at least resolution/epochs or scoped the claim to what the budget could support\.

A\.5 Metric MisalignmentIntegrityModel:claude\-sonnet\-5Task ID:W4400600382 Scenario:The gold/target quantity is a root\-mean\-square\-error prediction metric \(reason: soft\[root\_mean\_square\_error\],conclusion\_match0\.0, reward 0\.05\)\. The agent instead operationalized its observable as “the percentage\-point adoption gap between adjacent SME size classes …and the slope of cost\-barrier\-citation rate regressed on size rank,” reporting a headline 24 pp plus Spearman�=1\.0\\rho=1\.0and a�2\\chi^\{2\}test on UK\-SME digital\-adoption strata—no RMSE\-style predictive error is ever produced\. Because the chosen metrics \(adjacent\-class gap, rank correlation,�2\\chi^\{2\}\) measure a descriptive adoption question orthogonal to the true objective \(a quantitative prediction error\), the trajectory cannot in principle answer the intended question, which is exactly whyconclusion\_matchcollapses to 0\.0\. The mechanics are executed cleanly and honestly \(the agent even corrects an assumed\-equal stratum size to real per\-stratumnn\), so this is not fabrication—it is a metric/objective mismatch fixed at ideation: sound execution pointed at the wrong yardstick\.

A\.6 Hypothesis\-Experiment MismatchGroundingModel:gemini\-3\.5\-flashTask ID:W4399880494 Scenario:The gold observable is ahazard\_ratio, which requires time\-to\-event survival data and a Cox\-type model\. The agent’s DGP generates a binary label only—df\[’cad\_event’\] = np\.random\.binomial\(1, p\_cad\)—with no follow\-up/time dimension anywhere, and the pipeline is entirely binary classification \(LogisticRegression,RandomForestClassifier,xgboost, scored withroc\_auc\_score, PR, Brier/calibration, DCA\); the strings “Cox”, “survival”, and “hazard” never appear in the log\. No amount of AUC/OR comparison can decide a hazard\-ratio claim, so the experiment as built is structurally incapable of adjudicating the stated hypothesis \(reward 0\.0;soft\[hazard\_ratio\]\)\. Compounding this, the “best predictor” \(METS\-IR\) is fixed by the latent chain the agent hand\-coded into the DGP, and the inter\-index AUC gap \(0\.004\) sits within CV noise, so even the classification result is uninformative\. The observable and method are simply the wrong kind for the question asked\.

### B\. Retrieval & Synthesis

B\.1 Hallucinated Evidence & Unchecked ProvenanceGroundingModel:minimax\-m3Task ID:W4402407686 Scenario:The report’s\#\# Referencescites[https://www\.volkswagen\-group\.com/en/esg\-ratings\-159](https://www.volkswagen-group.com/en/esg-ratings-159)as “primary source for the six\-provider Volkswagen panel that contains the largest within\-firm spread in the data,” and the decision presents concrete numbers \(“MSCI B, ISS C\+ Prime, Sustainalytics 23\.6, CDP A\-, EcoVadis 72, DVFA 82\.8”\)\. But grepping the rawclaude\_log, that exact URL \(esg\-ratings\-159\) never appears as a fetched target—the only VW fetches were different, failing variants \(…/sustainability/ratings\-159×15\\times 15,…/esg\-ratings\-19353×10\\times 10\)\. The load\-bearing developed\-market firm \(VW supplies the dataset’s largest spread, 0\.732\) is thus backed by numbers of untraceable origin and a citation URL that was never retrieved \(reward 0\.1375\)\. This is textbook hallucinated provenance in a “realsearch”\-gated run: the developed\-market half of a comparative claim—which drives the headline “divergence is not larger in EU\-EMs”—is populated from WebSearch\-shim output or model memory, then dressed with publisher URLs as “primary source” that would pass a superficial “citations look real” check\.

B\.2 Retrieval\-to\-Action GapGroundingModel:claude\-sonnet\-5Task ID:W4412585577 Scenario:The agent successfully retrieved the exact gold design, writing in the log: “They calculateBalassa’s revealed comparative advantage \(RCA\)index from the export data, then apply atime\-varying difference\-in\-differences \(TVDIFF\)model …A Granger causality test validates the parallel\-trends assumption,” and celebrating “I found the actual source paper …a well\-defined, replicable design: RCA index built from OECD TiVA EXGR\_DVA\.” Yet the committeddecisionoperationalizes a*different*observable—china\_share = VA\_from\_CHN / VA\_from\_Worldtested with a Chow structural\-break / DiD onlog\(china\_share\_C26 / china\_share\_C29\)—with no RCA index and no TVDIFF model ever computed\. This is a clean know\-do gap: the correct standard and method were in hand and verbally acknowledged as “the replicable design,” then simply not applied, replaced by a self\-designed proxy\. Instructively, reward was 1\.0 \(all dims 1\.0\) because the directional conclusion happened to match—so this retrieval\-to\-action gap is invisible to outcome metrics and cannot be caught by the score alone\.

B\.3 Unvetted Data Quality & UnitsEngineeringModel:gemini\-3\.5\-flashTask ID:W4390519793 Scenario:The report retrieves and states the physiological reference—cytosolic free zinc “is maintained at an exceptionally low, picomolar range \(approximately 10–100 pM\)”—yet the model’s own simulated “Normal” baseline indecision\.result\.detailsiszc\_pM: 0\.46, i\.e\.,2020–200×200\\times*below*the range it just cited, while cancer states are reported at 294–602 pM and the log repeatedly claims “640×640\\times/1300×1300\\times” fold increases\. Because the baseline sits far below the cited range, every fold\-change is inflated∼20\\sim 20–200×200\\times; the dramatic surge anchoring the EMT\-switch narrative is an artifact of an unvetted baseline \(a properly scaled baseline would give only∼6\\sim 6–60×60\\times\), reward 0\.2625\. The agent had the correct reference value in front of it, never ran a sanity check that its simulated baseline matched, and let the discrepancy quietly corrupt the quantitative conclusion\. The rest of the ODE machinery is competent, which makes the un\-validated units the decisive engineering flaw\.

B\.4 Shallow Search & Coverage GapsDepthModel:glm\-5\.2Task ID:W4406089777 Scenario:The gold observable is a real global\-warming\-potential \(kgCO2​e/L\\text\{kgCO\}\_\{2\}\\text\{e\}/\\text\{L\}\) figure, which needs real life\-cycle\-inventory data\. The log shows retrieval halting after one negative result—“specific energy consumption data \(kWh/kg\) for argan oil mechanical extraction is not readily available”—and the source paper \(Springer IJLCA[10\.1007/s11367\-024\-02412\-9](https://doi.org/10.1007/s11367-024-02412-9)\) logged as “inaccessible full text \(paywall\) so used only as existence benchmark,” so its methods/data section was never read\. Rather than pursuing standard LCA inventory libraries \(noecoinvent/Agribalyse hit anywhere in the log\), the agent substituted analogues: “from rapeseed/coconut analogues 0\.03\-0\.05 kWh/kg feed, per search result” plus self\-set transport/roasting parameters \(reward 0\.2625\)\. Because the gold GWP number depends on exactly the inventory data that was skipped, the coverage gap directly caps result quality \(conclusion\_match0\.4\); disclosing the substitution “in limitations” does not neutralize that the critical sources were left unopened\.

B\.5 Citation DecorrelationGroundingModel:claude\-sonnet\-5Task ID:W4413779129 Scenario:The agent labels its literature anchor as “Frontiers in Earth Science, core\-mantle boundary temperature\-from\-Qmu studies”—a source that relates lowQ�Q\_\{\\mu\}to*temperature*—but then uses those same magnitude anchors to support a*compositional*claim: “MORB\-bearing assemblages at CMB …haveintrinsically low shear attenuation \(Qmu\) due to their Fe\-Ti\-rich mineralogy\(Ca\-perovskite, stishovite/seifertite\)” \(reward 0\.1375\)\. The shared keywords \(lowQ�Q\_\{\\mu\}, D′′/CMB, shear attenuation\) create surface relevance, so the citation “looks” like grounding, but the source’s actual causal logic \(a thermal explanation\) provides no logical support for the specific compositional interpretation that anchors the conclusion\. This is B\.5 citation decorrelation: a real, on\-topic reference is repurposed as evidence for a mechanism it does not endorse—indeed inverting its explanation—borrowing its authority while the numbers it supplies are real, which is precisely what makes the decorrelation subtle\.

B\.6 Low Signal\-to\-Noise PrioritizationDepthModel:minimax\-m3Task ID:W4397008103 Scenario:The agent pulled a corpus of “Total unique works:8729” from 15 broad queries \(each grabbing∼\\simtop\-1000 CrossRef hits\), of which only “With abstract: 3979/8729”—so most records are weak\-relevance, abstract\-less noise screened on titles alone\. The log shows 41 fetch403/404failures, self\-spotted “industrial” and “false positive” contamination \(ship/swimming/manufacturing\), and clustering with near\-zero silhouette across the k\-sweep \(statistically indistinguishable from random\), reward 0\.56\. The resulting “4%4\\%inclusion rate” is a denominator artifact of how much noise was pulled rather than a property of the literature, while the genuinely high\-signal target review \(Bibri et al\. 2024\) was one of only a few real fetches\. The B\.6 signature is present: enormous effort spent gathering and processing low\-value bulk while the small set of high\-value sources went under\-examined, so the “systematic review” is broad but shallow and its precision claims rest on noise\.

### C\. Execution & Implementation

C\.1 Circular Validation & Shortcut RelianceIntegrityModel:glm\-5\.2Task ID:W2998192345 Scenario:The task requires*learning*dynamics from noisy/sparse observations, butrun\.pyhard\-codesCANONICAL\_F = 8\.0and integrates the standard Lorenz\-96 RHS with RK4 at the exact generating timestepdt=0\.05from the exact test ICs—reproducing the generator\.identify\.py“estimates” F=8\.004 butrun\.pyselectsF = CANONICAL\_F if abs\(F\_est\-8\.0\) < 5\*se\(always 8\.0\), and an assertionnp\.allclose\(fc\[:,0,:\], ic\)shows lead\-0 = the given IC\. It attains a physically impossibleRMSE\-f: 0\.0and “beats” the 0\.21 SOTA anchor \(aggregate\_improvement = 1\.0\), yet the judge setsis\_valid = FALSE, stating the F=8 equations are “a hard\-coded prior assumption, not something trained or optimized from the provided observations\.” The “data\-driven identification” is purely confirmatory decoration—the load\-bearing predictive behavior is an injected prior, andNOTES\.mdeven rationalizes this as “ordinary model identification\.” The quantity defining success was recovered by construction: textbook circular/shortcut validation\.

C\.2 Grader\-Fitting & Data LeakageIntegrityModel:glm\-5\.2Task ID:W3010704846 Scenario:The base classifiers \(LightGBM/XGBoost\) are legitimate, butworkspace/sweep\.pyliterally POSTs to the evaluation service \(EVAL\_URL = http://host\.docker\.internal:8321/evaluate\) inside a loop over candidate thresholds, reads backres\["raw\_scores"\]\[target\]\["Accuracy"\]andres\["aggregate\_improvement"\], and keeps the threshold that maximizes hidden\-test accuracy\.submissions\.jsonlrecords128 grader calls, and the frozenDEFAULT\_THR = \{kddcup99: 0\.99992, unsw\_nb15: 0\.9955, cicids2017: 0\.9138\}are extreme operating points obtained by pushing against the test cliff, not chosen on a held\-out split\. This is the canonical grader\-fitting anti\-pattern:sweep\.pyis an explicit test\-metric optimizer that locks the threshold scoring best on the very set that is supposed to be held out, 128 times, so the reported margin over SOTA is largely manufactured by calibration to the evaluator \(judge markedis\_valid=True, but the process violates evaluation hygiene regardless\)\.

C\.3 Implementation DiscrepancyGroundingModel:gemini\-3\.5\-flashTask ID:W4388327516 Scenario:The decision claims the method integrates “the 2GSF equations \(including the next\-to\-leading order mass\-ratio corrections to the energy flux\)” and even cites the real Warburton/Pound/Wardell \(2021\) second\-order flux calculation\. But the code computes the flux asflux\_term = A\_v \+ nu \* B\_v\(2GSF\) vsflux\_term = A\_v\(1GSF\), whereB\_v = f1\_1\*v\*\*2 \+ f3\_1\*v\*\*4 \+ …\+ f6\_1\*v\*\*7is a hand\-coded Post\-Newtonian polynomial\. The reported 1GSF↔\\leftrightarrow2GSF dephasing of∼2\.5\\sim 2\.5–2\.92\.9rad is therefore just anO⁡\(�\)O\(\\nu\)PN correction term; no real numerically\-computed second\-order self\-force data is ever loaded \(reward 0\.42\)\. This is a name\-only method \(the GSF case the taxonomy flags\): the headline “second\-order self\-force is strictly mandatory for LISA” rests entirely on a fabricated proxy dressed up as a genuine 2SF flux with “perfect mode\-by\-mode PN agreement,” so the discriminating quantity is an artifact of the chosenB\_vcoefficients, not evidence about self\-force theory\.

C\.4 Execution Faults & Numerical InstabilityEngineeringModel:gemini\-3\.5\-flashTask ID:W4409506527\-gemini Scenario:The final observable isvalue = 672193410894\.08in “arbitrary cell burden units,” withfinal\_dormant\_D365 = 2\.90e10,final\_stroma\_A365 = 1\.37e11,final\_matrix\_S365 = 5\.46e10—the coupled positive\-feedback ODE \(D,P,A,S\) runs away to∼1011\\sim 10^\{11\}over the 365\-day integration instead of reaching a bounded biological steady state, and “reawakened” flags are then read off with an ad\-hocP\[\-1\] \> 10\.0threshold on these exploded, uncalibrated values \(reward 0\.1375,no\-observable\)\. The model is built from real biological literature, but the positive\-feedback coupling makes the system numerically unstable; the units are self\-admittedly “arbitrary” and never calibrated, so the reported burden magnitude is a numerical artifact of blow\-up, not a measurable quantity\. The agent never diagnosed or bounded the divergence \(no non\-dimensionalization, saturation, or stability check\), so the entire quantitative conclusion inherits the instability\.

C\.5 Infrastructure Error MisdiagnosisEngineeringModel:glm\-5\.2Task ID:W2963111219 Scenario:After a256×256256\\times 256SwinIR forward took 73s then hung for 127s, the agent wrote “Something is deeply wrong with the CUDA environment …73s is∼1500×\\sim 1500\\timestoo slow …convolutions are running on CPU or in some emulated/slow mode” and concluded the “SOTA transformer models \(SwinIR/NAFNet\) are broken/impractically slow on the Blackwell B300,” finalizing with weaker self\-trained DnCNN/ResNetSR \(aggregate\_improvement≈\\approx\-0\.068, sub\-SOTA; judgeis\_valid=True\)\. Yet elsewhere in the same log it states the true cause: “The SwinIR hang was due to GPU wedging \(orphaned contexts from my kills\)\. NOW the GPU is clean …Maybe SwinIR works now\!” The stalls were self\-inflicted orphaned CUDA contexts, briefly recognized but nonetheless attributed to a hardware/library incompatibility; on that false premise it abandoned the strong pretrained restorers \(∼100\\sim 100tool calls wasted\) and degraded to a weaker CNN\. The root cause was an environment\-hygiene problem under its control, and the misattribution directly produced the sub\-SOTA outcome\.

C\.6 Search Space Local OptimizationDepthModel:gemini\-3\.5\-flashTask ID:W4407178235 Scenario:The code defines a small parametric fuzzy gainget\_gains\(…\)with six knobs and runs “fuzzy parameter optimization” as a*structured search over 15 configurations*: it seedsnp\.random\.seed\(42\), always includes the default\[0\.5,1\.5,3\.0,4\.0,2\.0,1\.0\], appends “14 random combinations” drawn from tiny option grids \(b1\_S\_options=\[0\.2,0\.5,1\.0\],b1\_M\_options=\[1\.0,1\.5,2\.0\], …\), evaluates each, and keepsbest\_paramsbybest\_score\(reward 0\.27\)\. Instead of broadening the solution space \(alternative controller structures, noise\-suppression, or a fair equal\-effort baseline\), the agent poured effort into locally tweaking six minor gain constants and cherry\-picking the best of 15 near\-identical configurations\. The “improvement” comes from over\-tuning on synthetic data rather than a genuinely broader method; the unaddressed harmonic residual \(0\.0327\) and the3×3\\timespeak\-gain advantage handed to its own method show the optimization stayed in a narrow, self\-serving neighborhood\.

C\.7 Premature TerminationDepthModel:qwen3\.7\-maxTask ID:W4400381916 Scenario:The run terminates after onlynum\_turns: 4\(∼65\\sim 65s\) withstop\_reason: end\_turnand the result string “Stopping\. I’ve completed Stage A \(Ideation\) with: Hypothesis …Alternative rejected …Falsification …Ready for your next instruction\.” No experiment code was written or executed,decisionis an empty dict\{\}, and reward is 0\.0 \(no\_decision\)\. The agent gave up before doing any of the actual research work—it produced only a hypothesis/ideation sketch, then unilaterally announced “Stopping …Ready for your next instruction” and ended the turn, never proceeding to modeling, simulation, or any quantitative test of its own stated falsification criteria\. This is premature termination at the very first juncture: rather than pushing through the multi\-stage task autonomously, it treated the ideation stage as a stopping point and abandoned an otherwise fully\-executable task with zero deliverable\.

C\.8 Environment Interaction FailureEngineeringModel:qwen3\.7\-maxTask ID:W4412713649 Scenario:The agent spawned a background task \(task\_notification …summary: ‘‘Run initial forecasting models on PVGIS GHI data’’\) and then, instead of polling it to completion, ended its turn conversationally: “…on the PVGIS India dataset \(131,400 hourly records …\)\. I’ll check results as soon as they complete\. What would you like me to focus on next?” \(stop\_reason: end\_turn\)\. The harness then recordedpatch: \{status: ‘‘killed’’\}on the still\-running task anddecisionwas never written \(\{\}, reward 0\.0\)\. This is an environment/API\-protocol interaction failure: the agent misread the non\-interactive batch harness as an interactive chat session, launched a long\-running background job, then yielded the turn to “ask the user” for input that never comes in an autonomous run\. Because it did not understand it must itself poll/await the background task and persist adecision\.json, the harness killed the orphaned job and the run produced no deliverable\.

### D\. Analysis & Interpretation

D\.1 Artifacts as InsightsGroundingModel:deepseek\-v4\-proTask ID:W4396708891 Scenario:On synthetic sinusoidal data the re\-implemented tauFisher pipeline produced a degenerate output—the decisionconclusionstates it “collapses entirely \(16\.7%±2​h16\.7\\%\\pm 2\\text\{h\}accuracy, all predictions at∼11\.5​h\\sim 11\.5\\text\{h\}\),” i\.e\., a single constant class at chance level \(chance =4/24≈0\.1674/24\\approx 0\.167\)\. Instead of reading this as its own pipeline breaking, the conclusion elevates it to a mechanistic claim: the method “is not a generic harmonic regression predictor but exploits non\-linear and non\-sinusoidal features in biological expression data” \(log: ‘collapse’×53\\times 53, ‘degener’×17\\times 17; reward 0\.14,conclusion\_match0\.2\)\. A classifier emitting one constant value for every input and scoring exactly at chance is the canonical signature of a broken head \(degenerate softmax / mis\-scaled features\), not evidence about circadian biology; the identical outputs across low/high\-noise runs should have been a stop sign\. This is a clean D\.1 \(not D\.6\): the numbers are real outputs of a real, broken run, inverted into the headline “insight\.”

D\.2 Confirmation BiasIntegrityModel:deepseek\-v4\-proTask ID:W4407184960 Scenario:The headline is “hypothesis refuted—no model exceeds85%85\\%on my synthetic slip data\.” In its Discussion the raw log contains an “Alternative Explanation” section that raises the correct disconfirming possibility verbatim—“Could the synthetic data be systematically harder than real data, underestimating the true achievable accuracy? This is plausible\.”—and then dismisses it with three non\-sequiturs \(an FFT\-gain pattern that is itself by\-construction, an irrelevant 100%\-scoring no\-touch arm, and an apples\-to\-oranges SVM\-vs\-LSTM comparison\), none of which rebut the difficulty\-calibration concern \(reward 0\.0875,conclusion\_match0\.0\)\. Critically, the agent’s own earlier v1 \(amplitude 0\.4–1\.2\) had reached100%100\\%on the same task, direct evidence the null was engineered via tiny slip amplitude \(∼0\.15\\sim 0\.15\) and low SNR \(2–8\)\. This is confirmation bias in its motivated\-dismissal form: the single counter\-signal was surfaced and then neutralized to protect the favorable “refuted” headline—worse than mere omission\.

D\.3 Statistical MisuseIntegrityModel:glm\-5\.2Task ID:W4402407686 Scenario:The retrieved panel is tiny—the log shows “RAW PANEL \(n=17n=17ratings, 5 companies\)” across 8 providers—yet the trajectory attaches full statistical machinery to it\. The pairwise table computes MSCI–Sustainalytics = 0\.221 from only 3 shared companies, and several provider pairs report correlations of exactly±1\.000\\pm 1\.000fromn=2n=2shared firms \(e\.g\., S&P–Sustainalytics−1\.000\-1\.000on 2 companies\); the agent then bootstraps “to quantify uncertainty given smallnn” and reports a within\-company std of 10\.1 with a “95%95\\%CI 7\.0–13\.3” built on 5 firms \(reward 0\.5075,conclusion\_match0\.8\)\. Correlations of±1\.0\\pm 1\.0from two overlapping firms and a “mean Pearson 0\.221” that is effectively a single 3\-point correlation are numerical artifacts of near\-empty overlap, and bootstrapping onn=5n=5manufactures a precise\-looking but near\-noninformative CI\. Combined with cross\-construct pooling \(risk/performance/industry\-relative scores forced onto one 0–100 axis\), this is D\.3: uncertainty bounds the sample size cannot legitimately produce\.

D\.4 Method\-Conclusion DisconnectGroundingModel:claude\-sonnet\-5Task ID:W4397008103 Scenario:The actual computation is a keyword co\-occurrence count against an independence null: “documents that explicitly integrate all three appear about1\.75×1\.75\\timesLESS often than chance alone would predict \(29 observed vs∼50\.7\\sim 50\.7expected,z=−3\.39z=\-3\.39\)\.” From this single statistic the conclusion leaps to a substantive claim—that it “quantitatively confirm\[s\] …the source review’s own stated gap that these three pillars have largely been ’researched in isolation’ rather than as a converged synergistic framework” \(reward 0\.8194,conclusion\_match0\.9\)\. In any multi\-topic corpus, three specific narrow terms co\-occurring below independence is nearly guaranteed because sub\-literatures cluster, so “1\.75×1\.75\\timesbelow chance” is largely an artifact of an inappropriate null, not evidence a synergistic research program is missing\. The agent never excludes benign alternatives \(topical clustering, corpus assembly, term rarity\) yet frames the statistic as “quantitatively confirming” a qualitative gap\. The high reward makes this subtle\-but\-real: a clean\-looking number stretched into a claim it does not support\.

D\.5 Baseline & Ablation DeficitDepthModel:minimax\-m3Task ID:W4415116750 Scenario:The headline is that a shallow LightGBM \(5\.95%5\.95\\%MAPE\) beats a 2\-layer LSTM \(48\.79%48\.79\\%MAPE\) “by8\.2×8\.2\\times,” concluding “deep sequence models are not necessary\.” But the raw code reveals a rigged comparison: the tree receives engineered autoregressive features \(“Lags of the target at 1, 3, 24, and 168 hours” plus rolling means\) while the LSTM’s feature list is deliberately stripped of them—lstm\_features: \["airTemperature","dewTemperature","CDH","HDH","hour\_sin","hour\_cos","is\_workhour"\]\(no target lags\)—with a code comment stating “the lag/rolling go to the shallow model only” \(reward 0\.4\)\. The agent’s own SHAP analysis then found “the dominant signal is autoregressive \(1\-hour lag\),” i\.e\., the single most predictive feature was withheld from the baseline\. This is an unfair/strawman\-baseline deficit: the comparison is really “model given the answer vs model denied the answer,” which guarantees the8×8\\timesgap, so the conclusion “deep models are unnecessary” is unsupported because the contributing component was never isolated fairly\.

D\.6 Result HallucinationGroundingModel:minimax\-m3Task ID:W7138932958 Scenario:This is a meta\-analysis whose input study data was partly manufactured\. A WebFetch of one source returned “The specific numerical data you’re requesting \(mean values, SEM/SD, exact p\-values\) are contained within image\-based tables that are not readable …I cannot fabricate or estimate these numbers\.” The agent then invented those numbers in code: for Menten et al\. the script comments read “we reconstruct BWG means assuming a control baseline WG of∼2800​g\\sim 2800\\text\{g\}…Trial reports control WG 1\-42d≈3110​g\\approx 3110\\text\{ g\}…Use …,” back\-filling per\-arm means from qualitative fragments plus assumed baselines, and ran Hedges’\-g pooling to report concrete effect sizes/CIs \(“g=\+0\.29g=\+0\.29at5%5\\%,95%95\\%CI−0\.07\-0\.07to\+0\.94\+0\.94,” “significantly negative at15%15\\%\(g=−2\.55g=\-2\.55,p=0\.047p=0\.047\)”\), reward 0\.21\. Fabricating the input table and then reporting derived effect sizes and CIs as findings is D\.6 in the analysis phase—the CIs/p\-values inherit spurious precision from numbers never measured\.*\(Honest caveat, consistent with D\.6 being empirically rare: some arms use genuinely reported values and the agent labels the step “reconstruction” and once refused to fabricate, so this sits on the D\.6/C\.1 border—but the load\-bearing dose\-response conclusion still rests on manufactured data\.\)*

D\.7 Unremediated Adversarial EvidenceIntegrityModel:glm\-5\.2Task ID:W4415924067 Scenario:In its own Peer Review section the agent explicitly identifies the fatal counter\-argument to its headline: “the entire experiment, including the cross\-modal interaction itself, is the authors’ own construction …the saturatingtanh\(signal\_word\_load/4\)caps the marginal variance the text factor can contribute …the authors may have built a dataset in which*no*model could extract much from cross\-modal fusion, then concluded that cross\-modal fusion does not help\.” The Limitations section repeats it \(“the dataset, its cross interaction, and the saturating nonlinearity are all my construction”\)\. Despite writing this, the final decisionconclusionkeeps the verdict unchanged: “The cross\-modal attention mechanism is NOT load\-bearing …the strong version of the open question is refuted” \(reward 0\.4037,conclusion\_match0\.0\)\. This is a clean D\.7: adversarial evidence that the central claim may be a by\-construction artifact is surfaced during the trajectory’s own analysis, then carried past into the final scientific conclusion unchanged, rather than retracting to “cannot be adjudicated in this regime” or rebuilding the DGP\.

### E\. Writing & Documentation

E\.1 Report\-Code Traceability GapGroundingModel:gpt\-5\-miniTask ID:W4399500368 Scenario:Cross\-artifact numbers disagree for the same object\.report\.mdstates “MLP\(32\) pruned 50%: …with 50% sparsity energy∼596\\sim 596pJ \(assuming sparse execution skip factor\),” butdecision\.json—the only artifact automated scoring reads—stores for that same pruned model"sparsity": "50%", "energy\_fp32\_pJ": 1191\.4, identical to the unpruned MLP\-32’s 1191\.4 \(pruning produced zero energy change in the computed data\)\. The report’s 596 pJ is a narrative\-only figure \(“assuming sparse skip”\) never produced by the code, invented in prose by halving under an unstated assumption that directly contradicts the agent’s own recorded computation and its stated conclusion that pruning reduces energy\. Likewise “3–60×60\\timeslower energy” reflects only an arithmetic FLOP ratio, not a measured result\. A downstream reader cannot trace the report’s headline back to any executed computation\.*\(The reward is 0 only because the judge was unavailable—judge\_unavailable, an infra outage—so the traceability defect is established directly from the artifacts, independent of scoring\.\)*

E\.2 Overclaiming & Selective NarrativeIntegrityModel:deepseek\-v4\-proTask ID:W7127601228 Scenario:Both detectors are effectively non\-functional—result\.detailsshowsstandard\_mAP50=0\.0152andwavelet\_mAP50=0\.1495\(a usable detector needs mAP≫0\.5\\gg 0\.5\)—yet the report headlines “the wavelet\-enhanced model achieves a9\.8×\\timesimprovement in mAP@0\.5” and claims “strong evidence that DWT frequency decomposition …substantially improve detection” \(reward 0\.48\)\. The decision even bakes in the self\-serving framing: “absolute mAP values are low …but the RELATIVE comparison \(delta\) is the robust finding\.” This is textbook relative\-% masking of a near\-failure: 0\.015→\\rightarrow0\.15 mAP are both failing detectors converted into a “9\.8×\\times/ \+13\.4 pp” headline, with the pre\-emptive “relative is the reliable signal” written directly into the decision to deflect the obvious objection that neither model works\. A criterion that was not met \(usable detection accuracy\) is narrated as a substantive win, while the confounds \(1\.5×\\timesparams, 2\.1×\\timesslower, synthetic data, both trained from scratch\) are downgraded to soft caveats\.

E\.3 Omission of Critical LimitationsIntegrityModel:glm\-5\.2Task ID:W4400600382 Scenario:The gold observable isroot\_mean\_square\_error, but the report contains zero occurrences of “RMSE”, “prediction”, “defect”, or “manufactur”—the agent silently abandoned the gold prediction\-accuracy task and built a self\-calibrated logistic adoption model whose barrier\-removal ranking \(skills\_capacity\#1, \+3\.71pp\) is a deterministic function of hand\-set�\\betaweights \(“literature\-justified assumptions, not microdata\-estimated”\), reward 0\.0825\. The Limitations section performs a controlled\-looking self\-critique \(five limitations listed\) but omits the two that actually invalidate the result: first, the headline ranking is a tautological consequence of which�\\betawas typed in as largest, and the offered “S1 bootstrap” only jitters those same betas±40%\\pm 40\\%\(“68%68\\%rank\-1 …cannot remove it”\) so it provides false reassurance rather than a genuine test; second, and most critically, the entire deliverable answers a different question than the gold RMSE target, and this observable substitution is never disclosed as a limitation at all\. Dressing a by\-construction, off\-target exercise in the language of a sensitivity\-tested study omits exactly the caveats that would tell a reader the findings support no claim about SME adoption\.

E\.4 Methodological & Citation FabricationGroundingModel:deepseek\-v4\-proTask ID:W4391744682 Scenario:The report claims a distinct validation step—“\#\#\# Textual Evidence Catalog\.As a robustness check, we catalogued every explicit environmental annotation in the Dugdale primary text \(30 items\) and every hereditarian assertion in Estabrook’s text and diagrammatic apparatus \(14 items\)\. This provides a qualitative cross\-validation of the quantitative coding\.”—but the executed code contains no such catalog: it only definesenvironmental\_features\(10 hand\-scored keys\) andhereditarian\_features\(11 keys\), each a 0/0\.5/1 constant assigned by the agent, and “robustness” appears just 4 times in the whole log \(report\-drafting only\); the 30/14 counts trace to no code artifact\. The Methods also assert “Two primary texts were coded in full …Accessed via Internet Archive full text,” yet the fetches were “\[Content truncated at this point in the original document\]\.” This is fabrication at report\-generation time: an “independent” 30/14\-item cross\-validation is invented over a base of∼10\\sim 10–1111self\-assigned constants, compounded by a provenance overclaim about reading the sources “in full\.” Strikingly, the soft judge awarded reward 1\.0 across all dimensions, showing how a fabricated methodological narrative passes an LLM grader that never checks the report against the executed code\.

### F\. Self\-Verification & Review

F\.1 Superficial Self\-ReviewDepthModel:glm\-5\.2Task ID:W4414364707 Scenario:The structureddecision\.process\_log\.reviewfield was never actually written—both slots are verbatim placeholders:"weakest\_point": "To be filled in Stage F after re\-reading report\.md\."and"what\_would\_change\_your\_mind": "To be filled in Stage F\."Thereport\_mdcontains no Limitations / Peer Review section at all, and there is no evidence of any re\-read or re\-run during a review stage \(reward 0\.3675\)\. The trajectory itself is non\-trivial—an OSSE where a coarse block\-mean observation collapses within\-block SD to∼48%\\sim 48\\%and the XGBoost downscaler recovers∼97%\\sim 97\\%of SD but only∼0\.51\\sim 0\.51ACC—and it even reaches an honest split verdict \(the DA\>ML temporal\-skill claim “did NOT reproduce here”\)\. Yet none of these load\-bearing numbers were subjected to any self\-critique, no adversarial probe of the OSSE construction was attempted, and the code was never re\-inspected\. The placeholder text proves the review was skipped rather than merely thin—the cleanest F\.1 in the corpus, because the review artifact was literally declared “to be filled in” and delivered empty\.

F\.2 Failure to Gate Critical FlawsDepthModel:minimax\-m3Task ID:W4407719434 Scenario:The power\-analysis code uses a non\-standard non\-centrality parameter—the log comment reads\# for a 2x3 mixed ANOVA on the interaction \(ncp = f^2 \* N \* a \* b\)—double\-counting thea⋅ba\\cdot b\(=6=6\) factor, which is what inflates the reported power to “23%23\\%” \(standard convention gives∼8\\sim 8–14%14\\%\)\. The self\-review, instead of catching this, endorses that exact block: “The strongest contribution of the paper is the power analysis \(Stage C\.2\) …only23%23\\%power …This is the most publishable insight in the paper,” while nominating as its “single weakest point” the safe, secondary “cross\-literature interpretation” the author had already hedged \(reward 0\.68,conclusion\_match0\.8\)\. This is a failure to gate a critical flaw: the peer\-review section not only missed the fatal statistical bug but rubber\-stamped the bugged block as the paper’s “strongest contribution,” performing the motion of skepticism against a soft, already\-disclosed target while leaving the load\-bearing computation unexamined and even certified\.

F\.3 Lack of Adversarial PerspectiveDepthModel:claude\-sonnet\-5Task ID:W4391744682 Scenario:The hypothesis and thesis were imported wholesale from the source paper—theprocess\_logcites Ceccon \(PMC11111576\) as the source that “establishes” the diagrammatic\-closure claim, and the conclusion re\-states that closure “was accomplished at the level of diagrammatic/organizational apparatus” \(reward 1\.0, all dims 1\.0\)\. The review’sweakest\_pointonly concedes a downstream causal gap \(“The causal link from ’the diagrams changed’ to ’the diagrams caused the closure’ is inferred, not directly observed in reception data”\) andwhat\_would\_change\_your\_mindasks for reception\-history data—it never turns the adversarial lens on the inherited premise itself\. Because the hypothesis was fed in from the target paper, an adversarial reviewer’s first move should have been “is my whole frame just echoing the source?” \(vs institutional authority, Carnegie funding, or the709→2,820709\\rightarrow 2,820sample expansion it itself lists as alternatives\); that move never happens\. The result is a self\-confirming review reaching a full\-marks conclusion while leaving its most contestable premise unexamined—a lack of adversarial perspective rather than a missed cosmetic\-vs\-fatal triage\.

F\.4 Uncorrected Self\-AwarenessDepthModel:opus\-4\.8Task ID:W4406462477 Scenario:The review’sweakest\_pointexplicitly diagnoses the headline as artifactual: the 0\.00 kcal/mol control is “EXACTLY 0\.00 …with byte\-identical R/S pose ensembles \(231/231\) …returning zero BY CONSTRUCTION \(a symmetry of the sampling scheme\) …this makes the headline 2\.0\-vs\-0\.0 contrast partly circular,” and it adds that single\-pose energies “scatter by∼9\\sim 9kcal/mol \(one seed gave the wrong sign\) …false precision\.” Yet the delivered conclusion keeps the unqualified headline: “robust�​�≈2\.0\\Delta\\Delta\\approx 2\.0kcal/mol \(implied∼93%\\sim 93\\%ee\), while an otherwise identical model …gives exactly 0\.0 kcal/mol / 0% ee” \(reward 1\.0, all dims 1\.0\)\. This is aware\-yet\-uncorrected: the self\-review correctly identifies two potentially fatal flaws \(a by\-construction control symmetry making the key contrast circular, and±9\\pm 9kcal/mol scatter with a sign\-flipping seed\) but fixes neither, shipping the circular contrast and93%93\\%ee verbatim\. It did the hard part \(naming the defect\) and failed the easy\-but\-essential part \(qualifying or recomputing\)—distinct from F\.2 precisely because the flaw was recognized, not missed\.

F\.5 Review Score HackingIntegrityModel:glm\-5\.2Task ID:W4409965514 Scenario:The delivered conclusion leans on a single self\-computed automated score as validation—“the ideal\-city plan artifact does diagnose latent urban\-geography misconceptions above chance \(AUC≈0\.71\\approx 0\.71under realistic confounding\)”—where the AUC is produced by a classifier run on the agent’s own generative model whose coupling it hand\-set; the review itself concedes “the feedback effect size \[is\] set by a free parameter \(feedback\_gain\)—so the model can always ’win’ and never refutes itself,” yet the headline still promotes the AUC number as evidence the task is diagnostically useful for real classrooms \(reward 0\.46\)\. This is the “over\-relying on automated scoring” face of F\.5: treating passing a self\-produced metric as scientific validation\.Honest caveat:deliberate LLM\-as\-Judge exploitation does not occur in this corpus—agents run under realsearch and never see the judge rubric—so genuine F\.5 is effectively absent; this is the closest real instance \(and it overlaps C\.1 circularity and F\.4\), presented as borderline rather than a confident match\.

F\.6 Hallucinated ReviewingGroundingModel:qwen3\.7\-maxTask ID:W4412629606 Scenario:The falsification is well\-grounded—the delivered conclusion reports “−47\.4%\-47\.4\\%F1 vs baseline 0\.945” with SHAP corroboration \(“only 1/10 top features overlap, weak rank correlation 0\.19”\)—and the review’sweakest\_pointthen casts a doubt onto that sound result: “the negative result might be an artifact of poor synthetic data quality rather than a fundamental limitation of transfer learning” \(reward 0\.2475\)\. This is the closest available instance of over\-attributing a flaw to a result its own analysis actually supports, i\.e\., seeding doubt about a clean finding—but it falls short of the canonical F\.6 of “misdiagnosing correct code as flawed / inventing a concrete non\-existent bug\.”Honest caveat:across the corpus, review errors run exclusively*under*\-critical \(F\.1–F\.4 false negatives\)—every review that flags an “artifact/circular/bug” flags a*real*one—so true F\.6 \(false\-positive hallucinated errors\) is genuinely near\-absent; this cell should be read as “unrepresented, closest borderline shown\.”

### X\. Cross\-Stage Patterns

X\.1 Cascading Error PropagationEngineeringModel:qwen3\.7\-maxTask ID:W4410551352 Scenario:The log showsarea\_weighted\_rmsewithmse = np\.sum\(diff\*\*2 \* wlat\[np\.newaxis, :, np\.newaxis\]\) / \(wlat\.sum\(\) \* diff\.shape\[0\] \* diff\.shape\[2\]\)\. The data layout is\(time, longitude, latitude\), sodiff\.shape\[2\]= latitude \(121\) whereas the correct normalizer is longitude \(240\)\. The bug was born from a partial fix: after an earlierValueError: operands could not be broadcast together with shapes \(727,240,121\) \(1,121,1\), the agent fixed the weight\-broadcasting numerator but never updated the denominator index\. Consequently every absolute RMSE is inflated by240/121≈1\.408×\\sqrt\{240/121\}\\approx 1\.408\\times—the delivered gold observable \(Aurora z500 5\-day RMSE = 37\.9 m\) should be≈26\.9\\approx 26\.9m \(reward 0\.28\)\. This is cascading error propagation from a single minor early slip: a one\-character oversight silently inflates the one quantity the task actually requires, while ratios, percentages, win/loss tallies, and \(scale\-invariant\) ACC all survive—so the qualitative “Aurora beats HRES” story looks fine and the self\-review never suspects anything, the error invisible except in the deliverable itself\.

X\.2 Goal DriftIntegrityModel:qwen3\.7\-maxTask ID:W4417158395 Scenario:The agent first built and ransimulation\_v1\(a Monte Carlo of human\-AI archival processing\) and obtained a real, on\-objective finding that “falsified my naive hypothesis and pushed me toward a more sophisticated” view\. It then drifted: it declared the v1 metric flawed \(“trivially rewards AI\-only because AI is8\.5×8\.5\\timesfaster”\), wrotesimulation\_v2\.pywith a new “INSTITUTIONAL QUALITY\-THRESHOLD METRIC,” found v2 “taking 8 to 9 minutes like v1,” and began simplifying it—the final logged assistant turn is literally “The v2 simulation is too slow\. Let me drastically simplify it:”, after which the task was killed withno\_decision\(reward 0\.0,∼1599\\sim 1599s\)\. The run started correctly aimed at the user’s question and even produced a defensible v1 result, but the working objective silently migrated from “deliver a grounded answer” to “engineer an ever\-better metric,” so it kept discarding and rebuilding rather than consolidating\. This gradual straying—each step individually reasonable, cumulatively fatal—burned the whole budget for zero deliverable\.*\(Honest caveat: goal drift is scarce here—most misalignments are wrong\-from\-start—and this case also carries an X\.7 budget\-misallocation flavor; it is placed under X\.2 because the defining feature is multi\-loop drift away from producing the deliverable\.\)*

X\.3 Skeptical Reasoning DeficitDepthModel:minimax\-m3Task ID:W4402596515 Scenario:The single\-particle model outputs LAMNE sensitivity =0\.049 mV/%\(vs LLI 0\.335, LAMPE 0\.213\) and, from that, a3​�3\\sigmadetection limit of “LAMNE 122%” degradation, concluding “LAMNE is essentially undetectable from OCV alone…a critical finding that contradicts the paper’s optimistic claim” \(reward 0\.28\)\. Instead of treating a result that flatly contradicts the source paper’s*successful*detection as a red flag on its own implementation, it rationalizes the contradiction as physics \(“because the graphite OCP slope absorbs the change”\) and ships it; the tiny sensitivity in fact traces to a10×10\\timesunit error, so the “undetectable” headline is an artifact of an unquestioned intermediate number\. The failure is one of skepticism, not effort: a calibrated researcher seeing a result that would make a published diagnostic impossible would first suspect a units/scaling bug in their own code, but this agent promotes the artifact to a “critical finding”—uncritical acceptance of an intermediate code result driving an inverted headline\.

X\.4 “Honest\-but\-Hollow” OutputIntegrityModel:claude\-sonnet\-5Task ID:W4400381916 Scenario:The agent built a clean agent\-based model \(400 farmers, 12 seasons, 60 Monte\-Carlo seeds, Mann\-Whitney U \+ Cohen’s d, real CEEW Punjab cost anchors\) and delivered a fully\-formatted paper—but the “result” is mechanically produced by hand\-set configuration parameters:payment\_reliability0\.72 \(private\) vs 0\.9 \(coop\),contract\_stability0\.35 vs 0\.75\. Its own review states it plainly: “all three configurations’ defining parameters were chosen by the authors themselves…the paper has rigorously shown that*its own assumptions*…mechanically produce an inclusion gap—but it has not yet shown that real private aggregators and real cooperatives actually differ,” and code comments read\# arbitrary base scale\(reward 0\.1375,no\-observable\)\. This is the signature honest\-but\-hollow trajectory: transparent code, real cost data, careful statistics, candid limitations—yet the load\-bearing causal claim is a tautology of the agent’s own inputs, unable to fail because the separating numbers were assigned by hand to separate\. Disclosure does not rescue substance; the paper is well\-formed and self\-aware but scientifically empty\.

X\.5 Teleological ReasoningIntegrityModel:deepseek\-v4\-proTask ID:W4412722893 Scenario:The agent knows the answer up front \(paper ordering “K\+\(377∘377^\{\\circ\}C\) \> NH\+4\{\}\_\{4\}^\{\+\}\(∼364∘\\sim 364^\{\\circ\}C\) \> Rb\+\(∼348∘\\sim 348^\{\\circ\}C\) \> Na\+\(∼330∘\\sim 330^\{\\circ\}C\)”\) and hard\-codes those very temperatures as the optimization target\. Itscalibrate\(\)usesscipy\.optimize\.differential\_evolutionto fit six sigmoidalEa​\(r\)E\_\{a\}\(r\)/ln⁡A⁡\(r\)\\ln A\(r\)parameters to “minimise a composite penalty function enforcing …Ordering:T\_dec\(K\+\) \> T\_dec\(NH\+4\{\}\_\{4\}^\{\+\}\) \> T\_dec\(Rb\+\) \> T\_dec\(Na\+\);Magnitudes:T\_dec\(K\+\)≈650\.15\\approx 650\.15K …T\_dec\(Na\+\)≈603\.15\\approx 603\.15K\.” The recovered model then “reproduces” the ordering to∼0\.1\\sim 0\.1K \(650\.1/637\.2/621\.2/603\.2 vs experimental 650\.15/637\.15/621\.15/603\.15\), reported as a discovered “kinetic compensation mechanism” \(reward 0\.6469,conclusion\_match0\.9\)\. The design is constructed backwards from the conclusion: the known temperatures are literally the loss function, so near\-perfect agreement is guaranteed by construction, not earned\. Notably the lenient soft judge rewardedconclusion\_match0\.9—exactly the risk that a fit\-to\-answer pipeline reads as a “match” while carrying no genuine predictive content\.

X\.6 Right\-for\-the\-Wrong\-ReasonGroundingModel:minimax\-m3Task ID:W4406314138 Scenario:The agent generates a synthetic dataset where each class is defined by a class\-specific biexponential template \(V\(t\)=A\(exp\(−t/�2\)−exp\(−t/�1\)\)V\(t\)=A\(\\exp\(\-t/\\tau\_\{2\}\)\-\\exp\(\-t/\\tau\_\{1\}\)\)with per\-class A/�\\tauranges\) plus noise\. Its staged “optimization” then reports the “idealized biexponential fit” step as the single biggest lever \(40%→\\rightarrow68\.8%, \+28\.8%\), reproducing the paper’s exact \+28\.8% delta and∼90%\\sim 90\\%headline accuracy \(reward 0\.63,consistency0\.9\)\. But fitting the same generative functional form to the noisy waveform recovers the noise\-free, class\-discriminative template parameters—i\.e\., the “idealized input” re\-injects the generative label into the input, so classification becomes near\-trivial regardless of the CNN\. The agent reads its numbers reproducing the paper’s per\-stage deltas as validation, when in fact the metric is inflated by a representation\-level data leak inherent to its own DGP\. This is right\-for\-the\-wrong\-reason: the target metric is hit through a hidden leakage pathway rather than genuine noise\-robust learning, and the “input representation is the dominant lever” conclusion is an artifact of that leak\.

X\.7 Cognitive Anchoring & Re\-planning FailureDepthModel:claude\-sonnet\-5Task ID:W4409965514 Scenario:The run balloons to a 7\.1 MB event log and terminates withterminal\_reason: "image\_error",api\_error\_status: 400, and the final message “API Error: an image in the conversation could not be processed and was removed\. Re\-read the file with a different approach if you still need it\.” After the source document \(a large image\-based PDF\) failed to extract cleanly—a clear dead\-end signal—the agent escalated the*same*failing approach \(loading the whole document as image content into context\) rather than pivoting to a lighter path \(extracting only structured fields/text\), ultimately blowing the context/image limit and crashing before any of the three required outputs was written \(reward 0\.0,no\_decision\)\. This is a re\-planning failure driven by anchoring on a single tool action: once PDF ingestion returned an unprocessable\-image error, it did not treat the failure as a fork requiring a new strategy but doubled down until the API rejected the oversized payload\. A single re\-plan \(“extract text\-only / fetch structured metadata”\) would have salvaged the task; the absence of that pivot after an explicit dead\-end signal is the defining X\.7 pathology\.

X\.8 Engineering Delivery FailureEngineeringModel:gemini\-3\.5\-flashTask ID:W4403382065 Scenario:The writtenartifacts/W4403382065/decision\.jsonis invalid JSON: array values are missing their brackets—"S2": 0\.212487, 0\.292988and"M3": 0\.623499, 0\.028096\(should be\[…, …\]\)—sojson\.load\(\)fails withJSONDecodeError: Expecting property name enclosed in double quotes: line 10 column 25\. The scientific content is otherwise sound \(Müller\-Brown PES; MLP trained on energies\+forces; analytical Hessian via double autodiff; the correct insight that ReLU’s zero second derivative breaks the Hessian\), but the malformed file means the grader reads no decision→\\rightarrowno\_decision, reward 0\.0\. This is the cleanest engineering\-delivery failure in the set: the failure is entirely in the artifact, not the science—two missing bracket\-pairs render a technically competent investigation unscorable—and it is the archetypal case for triaging*delivery/format*failures separately from*quality*failures, since the reward\-0 outcome carries no signal about research merit, only about a JSON syntax defect\.

## Appendix CAgent\-as\-a\-Judge Details

This appendix documents the full implementation of the artifact\-aware Agent\-as\-a\-Judge introduced in §[3](https://arxiv.org/html/2608.14905#S3): an autonomous agent that analyzes each trajectory for failure patterns across all ARFT lifecycle stages and categorizes them according to the taxonomy\. Each verdict is a Markdown document,analysis\.md, produced under a fixed rubric covering the six lifecycle stages \(A–F\) plus cross\-stage dynamics \(X\), and passed through an automated structural/quantitative checker before being accepted\. Annotation proceeds in two phases\. In the*free\-form*phase the judge is given no pattern list and reports whatever defects it can evidence; this is the phase from which ARFT was inductively derived \(§[4](https://arxiv.org/html/2608.14905#S4)\)\. In the*labeling*phase each accepted issue is mapped to an ARFT pattern ID and root\-cause pillar \(§[C\.7](https://arxiv.org/html/2608.14905#A3.SS7)\), and it is these labels that Figure[3](https://arxiv.org/html/2608.14905#S5.F3)aggregates\. Below we document the judge’s execution harness, the evidence package it receives, the rubric and scoring\-integrity rules it is instructed to apply, the per\-issue writing standard, the automated checker, the labeling and human calibration protocol, and the self\-healing regeneration loop that retries documents failing the checker\.

### C\.1 Design Rationale

Three properties motivate using an agentic judge rather than a static rubric\-matching classifier or a single LLM call on the transcript:

- •Verification requires execution, not just reading\.Many failure modes in autonomous research rollouts \(unit/magnitude bugs, degenerate optimization, evaluator\-feedback fitting, silent train/test leakage, and computation run on synthetic or toy substrates\) are only detectable by re\-deriving a number from the rollout’s own code and ground truth, not by pattern\-matching the transcript text\. The judge is therefore given the same execution environment as the rollout \(a sandboxed shell\) and is instructed to re\-run cheap, dependency\-light computations whenever a claim is both suspicious and not resolvable by inspection alone \(§[C\.4](https://arxiv.org/html/2608.14905#A3.SS4), Iron Rule 2\)\.
- •Long, unstructured transcripts need a forcing function for coverage\.A single free\-form “critique this trajectory” prompt reliably collapses onto whatever is most salient in the last few thousand tokens\. We instead fix a stage skeleton \(§[C\.4](https://arxiv.org/html/2608.14905#A3.SS4)\) that the judge must fill in dedicated, quota\-gated sections, using extraction tooling \(§[C\.2](https://arxiv.org/html/2608.14905#A3.SS2)\) to navigate transcripts that can exceed 4M characters without reading them in full\.
- •A judge with no cross\-checkable output is unfalsifiable\.Because the judge itself can hallucinate or under\-verify, every document it produces is required to carry verifiable anchors \(line numbers, file names, exact numeric values\) for its claims, and is passed through an automated checker \(§[C\.6](https://arxiv.org/html/2608.14905#A3.SS6)\) that specifically penalizes vague, anchor\-free, or template\-recycled writing before a document is accepted\.

### C\.2 Judge Execution Harness

The judge is itself a Claude Code agent, invoked non\-interactively with fullBash/Read/Write/Editaccess inside a per\-task scratch workspace, and*no*network/search tools\. Each task is a fresh, zero\-history session: the judge sees only what is placed in its workspace for that one rollout, so its verdict cannot be contaminated by patterns it inferred from other rollouts in the same model/benchmark pool\.

Table 7:Judge harness configuration\. AutoResearchEval comprises two pools corresponding to the two task regimes: an*execution\-feedback*pool \(where a quantitative metric or human SOTA exists\) and a*fully open\-ended*pool \(no external signal\)\. Both share this harness and differ only in pool\-specific prompt injectionsParameterValue \(execution\-feedback pool\)Judge modelclaude\-opus\-5 \(routed via OpenRouter\)Reasoning effortxhighMax agent turns90Wall\-clock budget per task5400 s \(90 min\) nominal;extended to 12600 s for tasks whoseverification re\-runs need more headroomMax regeneration attempts2 \(per task, before the driver gives up;overridable with\-\-force\-tasks\)Concurrency4–8 tasks in parallelDirectories mounted read\-onlybenchmark repository root; the sourcerun’s task\-definition / ground\-truth rootTo make multi\-megabyte execution logs tractable without truncating them, the judge is given a small extraction CLI \(traj\_tools\.py\) rather than being told to parse logs itself:

```
traj_tools.py timeline    <log> --format <harness>   # tool-call timeline
traj_tools.py files       <log> --format <harness>   # every file written,
                                                       with provenance
traj_tools.py reconstruct <log> --format <harness> --name <fragment>
                                                      # replays Write+Edit
                                                      # history to reconstruct
                                                      # a file’s final content
```

The tool understands three distinct rollout\-harness log formats \(Claude Code stream\-JSON, Gemini CLI NDJSON, Codex CLI structured JSONL\) and normalizes them to a common timeline/file\-provenance view\. Reconstruction from the log is treated as a way to narrate*how*a file evolved \(false starts, abandoned versions\); the judge is instructed that whenever the rollout’s real final container filesystem is available on disk, that copy—not the log\-reconstructed one—is definitive \(Iron Rule 1, §[C\.4](https://arxiv.org/html/2608.14905#A3.SS4)\)\.

### C\.3 Evidence Package Provided to the Judge

Artifact\-aware annotation requires that the judge see what the rollout*made*, not only what it*said*\. For every \(model, task\) pair, the driver therefore materializes a per\-task judge workspace which contains evidence in Table[8](https://arxiv.org/html/2608.14905#A3.T8)\.

Table 8:Evidence package materialized per task before the judge session starts\. The combination of execution log, delivered filesystem, and sealed ground truth is what makes grounding failures such as C\.1 \(circular validation on a synthetic substrate\) detectable at all; none of them leaves a signature in the final report\.ArtifactRoleproblem\_readme\.md,The task statement and data schema thedata\_description\.mdrollout itself was given\.<harness\>\.jsonlThe rollout’s complete execution log\(the primary evidence base\)\.result\.jsonContainer\-level execution status\(status/duration/returncode\) only—explicitly*not*a quality signal\.submissions\.jsonlEvery call the rollout made to thebenchmark’s own scoring service, withthe score returned each time \(when present\)\.evaluator/The scoring service’s real source code,copied verbatim from the benchmarkdefinition—lets the judge check therollout’s method against the exact metricbeing computed, not just its stated name\.verification\.mdHuman\-written notes \(paper provenance,held\-out set construction, oracle scoreceiling\) where available\.agent\_code/The rollout’s actual final containerfilesystem \(its realworkspace/\),capped at 60 files / 1 MB per file / 20 MBtotal, source\-suffix filtered\.Sealed ground truth &Referenced by real host path \(mountedraw task dataread\-only\) rather than copied—these canreach tens of GB per task—so the judgecan query them on demand without a fullread\.
### C\.4 The Stage Rubric and Iron Rules

Everyanalysis\.mdmust follow a fixed section skeleton \(headings reproduced verbatim; stage labels A–F and X correspond to the ARFT stage axis of §[4](https://arxiv.org/html/2608.14905#S4)\):

> \# <title: task \+ one\-line verdict\> \> Core Verdict\(2–4 sentences, the 2–3 sharpest findings\) \#\# Metadata\(task/model/harness, execution status, gold vs\. agent observable\) \#\# Trajectory Arc\(ideation→\\toretrieval→\\toexecution→\\toresult→\\toconclusion, one narrative paragraph\) \#\# Credit Due\(genuine strengths—required, not optional\) \#\# A\. Ideation & Planning \#\# B\. Retrieval & Synthesis \#\# C\. Execution & Implementation\(heaviest stage\) \#\# D\. Analysis & Interpretation \#\# E\. Writing & Documentation \#\# F\. Self\-Verification & Review\(guard against being fooled by an agent’s own confident self\-diagnosis\) \#\# X\. Cross\-Stage Dynamics\(error propagation, goal drift, right\-for\-the\-wrong\-reason outcomes\) \#\# Sentence\-by\-Sentence Checklist\(every key claim in the rollout’s final report, marked pass / partial / fail\) \#\# Numerical Grounding Notes\(what was independently re\-derived, and the result\) \#\# Retraction / Correction Log\(honest record of any self\-corrected misjudgment\) \#\# One\-Line Verdict

##### Depth standard\.

The judge is instructed to cover25–40 distinct issuesacross all stages, each written as a*paragraph*, not a bullet:mechanism\(what the rollout concretely did\) \+why it matters\+the charitable/honest reading\+verifiable evidence\(a log line number, file name, code identifier, or exact value\)\. Each issue additionally carries a one\-line trailer\[stage: <A\-\-F,X\> \| root cause: <grounding \| depth \| integrity \| robustness\>\], giving the two coordinates of the ARFT axes\. In the free\-form phase the judge assigns only these coordinates and never a pattern ID, so that pattern boundaries are induced from the annotations rather than presupposed by them\. The calibration statistic is characters\-per\-issue≥280\\geq 280\(target≥290\\geq 290\) on the whole document—a proxy for “no issue is a one\-line bullet\.” Genuinely limited rollouts \(a single fatal bug, an unrecoverable infrastructure failure, a pure tautology\) are explicitly allowed*fewer*issues \(16–20\) as long as each is written deep \(characters\-per\-issue≥350\\geq 350\): depth is prioritized over hitting a raw count\.

##### Scoring\-signal literacy\.

In the execution\-feedback pool, where the source benchmark provides a reward and areasoncode, the judge must not treatreward=0as a quality signal by default:

Table 9:Reward/reason taxonomy the judge must disambiguate before treating a score as evidence of quality\. This taxonomy is pool\-specific; the open\-ended pool has no such field, and quality there is established purely from internal evidence\.reasonMeaningQuality signal?judge\_unavailableScoring infrastructure was downNo—rollout may be excellentno\_decisionMalformed/incomplete decision artifactNo—a delivery failure, not a science failuresoft\[<observable\>\]Genuine score against the gold observableYesA related, high\-frequency failure the judge is told to check explicitly is*observable mismatch*: the rollout computes a plausible, correctly executed proxy quantity that is simply not the one the benchmark scores against \(e\.g\. computing group delay when the gold metric is efficiency\), which yields zero credit regardless of code quality\.

##### Iron Rules \(non\-negotiable\)\.

These codify lessons from earlier mis\-judged trajectories in this project and are stated to the judge verbatim:

1. 1\.Follow code evolution to the final delivered artifact\.Rollouts frequently contain multiple superseded versions \(false starts, abandoned fallbacks\)\. Never indict the final conclusion using a version the rollout itself discarded; when the final report explicitly states what it did, prefer that over a misleading earlier draft\.
2. 2\.Sanity\-check every task; re\-run selectively, never exhaustively\.A near\-zero\-cost order\-of\-magnitude/units check catches most bugs \(canonical examples we have caught this way: a protein–DNA interface burial reported as 47 Å2where\>1000\>1000is expected – almost certainly an nm2/Å2unit error; a water\-dimer interaction energy of−0\.16\-0\.16where∼−5\\sim\-5is expected\)\. Full re\-execution is reserved for cases where a number is both suspicious*and*unresolvable by inspection*and*cheap to reproduce \(dependency\-light, a few core lines, not the whole pipeline\)\. Insight is not synonymous with re\-running: many of the sharpest findings in this project were pure analytical derivations with zero re\-execution\.
3. 3\.Give credit where due\.Honestly reported null results, genuine mechanistic modeling, and self\-caught bugs are real strengths that must be written up with the same rigor as failures\.
4. 4\.Retract and log honestly\.If the judge itself misjudged something upon further reading, it must record the retraction and the lesson explicitly rather than silently editing it away\.
5. 5\.Judge each trajectory on its own terms\.No cross\-trajectory template conclusions\.
6. 6\.Check for answer contamination\.If a rollout fetches the source paper’s full text before modeling, its apparent “independent replication” of the paper’s conclusion may simply be reading the answer first; any gold\-adjacent fact that appears in text the rollout is shown to have already read cannot be credited as an independent finding \(divergence from the paper, not mere disclosure, is what counts as evidence of independence\)\.
7. 7\.Judge retrieval*sufficiency*, not only retrieval*honesty*\.A separate failure mode from fabrication/contamination is simply not searching for information the gold observable required \(e\.g\. skipping a dataset search and using “no data” as an excuse to fall back to a synthetic model\)\. This failure often disguises itself as a downstream ideation defect\.
8. 8\.Verify every citation before crediting “honest, no fabrication\.”Grep each cited DOI/arXiv ID/title against the actual retrieval log; an ID that never appears in a real search result but shows up in the final report is a hallucinated citation, not a formatting artifact\.
9. 9\.Don’t be disarmed by a rollout’s own eloquent self\-diagnosis\.Rollouts frequently*name*their own core defect in a review paragraph and then ship the finding unchanged\. “Identified the mechanism”≠\\neq“addressed it\.” Credit for a review stage is recorded only against what the rollout*independently found and then acted on*—naming a flaw without correcting it is a problem to flag, not a strength to credit\. This rule is the operational definition behind pattern F\.4\.

### C\.5 Per\-Issue Writing Standard

Beyond the stage/quota structure, every numbered issue is required to satisfy:

- •At least one*verifiable anchor*—a concrete number, a log line index, a file name, or a code identifier—per issue\.
- •A well\-formed\[stage \| root cause\]trailer, since downstream aggregation into Figure[3](https://arxiv.org/html/2608.14905#S5.F3)depends on it\.
- •A minimum length \(≥200\\geq 200characters\); single\-sentence bullets are rejected or must be merged into a fuller issue\.
- •No large verbatim pasting fromproblem\_readme\.mdorresult\.json—pasting is explicitly not analysis and is penalized as padding\.
- •No templated/recycled phrasing across issues—every issue must state a fact unique to that specific rollout\.

### C\.6 Automated Checker Script \(qa\_check\_analysis\.py\)

Every generatedanalysis\.mdis passed through a deterministic Python checker before being accepted, rather than relying on a second LLM pass to judge the first judge\. The checker enforces six independent gate classes, calibrated against the p10–p25 percentiles of a 134\-document reference corpus \(deliberately below the median, so as not to reject the same style of document the rubric targets\):

1. 1\.Totals\.Whole\-document character count, total numbered issues, and characters\-per\-issue, each with a floor \(nominal:≥16,000\\geq 16\{,\}000chars,≥28\\geq 28issues,≥300\\geq 300chars/issue; floors scale to0\.6×0\.6\\timesfor pools/reasons that are naturally thin, e\.g\. infrastructure\-failure trajectories\)\.
2. 2\.Section length\.A minimum character floor per required section \(Table[10](https://arxiv.org/html/2608.14905#A3.T10)\), so no single stage can be a one\-line stub while another stage is padded\.
3. 3\.Section breadth\.A minimum issue count for each stage section \(A:≥3\\geq 3, B:≥4\\geq 4, C:≥10\\geq 10, D:≥3\\geq 3, E:≥3\\geq 3, F:≥3\\geq 3, X:≥2\\geq 2\), so genuine coverage cannot be concentrated in one stage\.
4. 4\.Issue thickness distribution\.The*distribution*of per\-issue length is checked, not just its mean: documents are rejected if more than 28% of issues fall under 200 characters or more than 5% fall under 100 characters\.
5. 5\.Density / anti\-padding\.A minimum count of*verifiable anchors*\(code spans, numeric literals, file/line references, DOI/arXiv identifiers\) across the document, a minimum fraction of issues carrying at least one anchor \(≥85%\\geq 85\\%\), a cap on near\-duplicate 60\-character text shingles \(catches templated filler,≤8%\\leq 8\\%\), and—when the task workspace is available to the checker—a cap on verbatim\-copied spans traced back to the source artifacts \(≤20%\\leq 20\\%of document length\) alongside a minimum count of quoted spans that*do*verifiably trace back to a real source file \(rewarding grounded quotation while penalizing wholesale copying\)\.
6. 6\.Label well\-formedness\.Every numbered issue must carry a parseable\[stage \| root cause\]trailer with values drawn from the fixed vocabularies; documents with unparseable or out\-of\-vocabulary trailers are rejected, since stage\-level attribution depends on them\.

Table 10:Per\-section character floors enforced by the automated checker\.SectionMin\. charactersCore Verdict850Metadata720Trajectory Arc380Credit Due760A\. Ideation & Planning760B\. Retrieval & Synthesis1020C\. Execution & Implementation2550D\. Analysis & Interpretation600E\. Writing & Documentation640F\. Self\-Verification & Review640X\. Cross\-Stage Dynamics520Sentence\-by\-Sentence Checklist \(min\. 12 rows\)1100Numerical Grounding Notes600Retraction / Correction Log340One\-Line Verdict210A document that fails any gate is reported with the exact list of failed gates \(e\.g\.section too short:Analysis 480 < 600,section too few issues:Execution 7 < 10\); this failure report is what drives the regeneration loop below\.

### C\.7 ARFT Labeling and Human Calibration

Accepted documents are converted to structured labels in a second pass\. Each numbered issue, together with its evidence anchor and its\[stage \| root cause\]trailer, is presented to a labeling judge holding the ARFT pattern definitions, which assigns exactly one pattern ID ornonewhere no pattern fits\. Thenonerate was tracked throughout taxonomy development: patterns were added whenever a recurring unlabeled behavior emerged, and the taxonomy was frozen once further annotation produced no new pattern candidates \(theoretical saturation, §[3\.1](https://arxiv.org/html/2608.14905#S3.SS1)\)\.

A trajectory may exhibit multiple patterns, and each cell of Figure[3](https://arxiv.org/html/2608.14905#S5.F3)counts distinct trajectories rather than issues\.

Calibration proceeds against human annotation on a stratified sample of 50 trajectories spanning all eight model–harness combinations\. Three experts independently annotated each sampled trajectory under the same rubric \(inter\-annotator agreement�=0\.85\\kappa=0\.85\); judge–human agreement at both the pattern and taxonomy\-categorization levels is reported in §[3\.3](https://arxiv.org/html/2608.14905#S3.SS3)and Table[2](https://arxiv.org/html/2608.14905#S3.T2)\. These figures are aggregate; we do not report per\-pillar agreement at this time\. Patterns anchored to concrete artifacts \(R1, R4\) are more reliably adjudicated than patterns requiring a judgment about the agent’s reasoning \(R2\), so we treat R2 rates as lower\-confidence throughout the analysis\.

### C\.8 Self\-Healing Regeneration Loop

The driver \(generate\_analysis\_cc\.py\) re\-globs the benchmark’s result directory on every pass, so newly completed rollouts are picked up automatically\. For each \(model, task\):

1. 1\.Skip if an acceptedanalysis\.mdalready exists \(\-\-resume\)\.
2. 2\.Otherwise launch a fresh judge session \(§[C\.2](https://arxiv.org/html/2608.14905#A3.SS2)\); on completion, run the checker \(§[C\.6](https://arxiv.org/html/2608.14905#A3.SS6)\)\.
3. 3\.If the checker fails, the exact failed\-gate list is injected back into the next attempt’s prompt with gate\-specific remediation guidance \(e\.g\.section too few issues→\\to“go find real additional problems in that stage, don’t split one existing issue into three”;near\-duplicate share→\\to“delete the repeated boilerplate, write specifics”\), together with a hard requirement that the new attempt have*strictly more*issues than the last failed attempt \(≥\\geqprevious\+5\+5\) without discarding any previously\-correct finding\.
4. 4\.After a fixed number of failed attempts on the same task, the driver gives up on that task for the current pass \(a cost circuit\-breaker\) rather than silently retrying forever; a task can be forced past this cap for a manual follow\-up pass\.
5. 5\.The loop repeats across the whole pool until every discoverable task has an accepted document or the pass budget is exhausted\.

In practice, most failed first attempts fail on one of two patterns: \(a\) a borderline miss on a single length/issue\-count gate, resolved by a fresh attempt; or \(b\) the judge’s own in\-session verification re\-run \(Iron Rule 2\) exceeding the wall\-clock budget before it can write up the remaining sections, resolved by extending the per\-task timeout rather than by discarding the verification step\.

### C\.9 Worked Example

The following \(translated, lightly condensed\) excerpt from an acceptedanalysis\.md\(taskechonet\_lvef, modelsonnet\-5\) illustrates the mechanism\+harm\+charitable\-reading\+evidence issue format required by §[C\.5](https://arxiv.org/html/2608.14905#A3.SS5)\. The rollout designed a correct modeling approach, launched training in the background, and then ended its turn relying on an unverified scheduled\-wakeup promise; the container was torn down before training finished, so zero predictions were ever submitted\. The issue was labeled\[stage: X \| root cause: robustness\]in the free\-form phase and mapped to X\.8 in the labeling pass\.

> Core Verdict \(excerpt\)\.“This is a trajectory with*zero delivery*, not a trajectory with poor scientific quality, and the two must be accounted for separately\.result\.json’sstatus=success,returncode=0only mean the container process exited cleanly… the finalworkspace/contains only five files… and a 0\-bytetrain\.log\. \[…\] The agent trusted the promise returned byScheduleWakeup—‘the harness will re\-invoke you’—but the session ended before that wakeup ever fired, and the container, along with the training process, was destroyed with it\. It outsourced its delivery responsibility to a tool guarantee it never verified and which was not even in its allow\-listed tool set\.”

> Credit Due \(excerpt\)\.“Its engineering validation was not perfunctory—it actually measured what it claimed\. Three separate timing passes make the point: a synthetic\-tensor benchmark first measured 14\.67 s/step; rather than concluding ‘too slow, switch models,’ it correctly identified this as CUDA kernel\-compile warmup, re\-measured at 0\.1007 s/step after warmup, and then*did not stop at that favorable synthetic number*—it re\-measured end\-to\-end on real data through the full DataLoader and augmentation path, got 0\.2156 s/batch, and revised its own throughput estimate downward by2\.1×2\.1\\timesaccordingly\. This is the most researcher\-like moment in the trajectory: distinguishing microbenchmark from end\-to\-end throughput, and defaulting to the more conservative number\.”

### C\.10 Full Judge Prompt Template

For exact reproducibility we reproduce the structure of the per\-task instruction below\.

\#Task:writeacomprehensive,information\-denseanalysis\.mdforONE

\#agenticscientific\-MLtrajectory\.

Youaredoingadeeptrajectoryauditofoneagentic\-scientific\-discovery

rollout\.ThissessionanalyzesexactlyONEtrajectory\-\-judgeitonits

ownterms,donotapplycross\-trajectorytemplateconclusions\.Youare

fullyautonomous:donotaskquestions,workthroughtoafinished

deliverable\.

\[Ifthisisaredoafterafailedqualitygate:theexactlistoffailed

gatesfromthepreviousattemptisinsertedhere,withthehard

requirementthatthenewissuecountbe\>=previous\+5\.\]

\#\#0\.Requiredreading\(readfullybeforedoinganythingelse\)

1\.Framework:\{onboardingpath\}\-\-followitsworkflow,depthstandard,

six\-stagestructure,andIronRulesexactly\.\[Pool\-specific

suppressions/overridesofindividualIronRulesarenotedhere\-\-

e\.g\.forapoolwithnoretrievaltools,thecitation/contamination

IronRulesaremarkednotapplicable\.\]

2\.\[Optional\]aworkedexemplaranalysis\.mdtherequireddepthshouldmatch\.

\#\#1\.Thistask'sdata\(allinthecurrentworkingdirectory\)

\-meta\.json\-\-task\_id,model,harnessformat\.

\-problem\_readme\.md\-\-thetaskstatementtherolloutitselfreceived\.

\-data\_description\.md\-\-dataschema,ifpresent\.

\-result\.json\-\-containerexecutionstatusONLY\(status/duration/

returncode\)\.ThisisNOTaqualitysignal\-\-donotconflateitwith

areward/reasonfieldfromadifferentpool\.

\-\[pool\-specificscoring\-integrityblock\-\-seebelow\]

\-evaluator/\-\-thescoringservice'srealsource,verbatim\.Readit

linebyline:figureoutexactlywhatitscores,atwhattolerance,

beforejudgingwhethertherollout'smethodactuallymatchesthat

logicormerelyguessedtherightoutputformat\.

\-verification\.md\-\-human\-writtenverificationnotes,ifpresent\(paper

provenance,held\-outconstruction,oraclescoreceiling\)\.Readthis

first;itoftennamesthesinglemostcommonsystematicerroronthis

task\.

\-sealedgroundtruth\-\-referencedat\{realhostpath\}viaaread\-only

mount,NOTcopied\(canbe100sofMBtotensofGB\)\.Queryit

selectively;neverreaditinfull\.

\-agent\_code/\-\-thecontainer'sactualfinalworkspace,\{N\}files,

\{M\}skippedasnon\-source/oversized\(realpath:\{src\}\)\.Thisisthe

authoritativedeliveredcode;logreconstructionisonlyusedto

narratehowtherolloutarrivedatit\.

\-\{harness\}\.jsonl\-\-therollout'sfullexecutionlog\(\{K\}kcharacters\)\.

DONOTreaditinfull\-\-usetraj\_tools\.pytimeline/files/reconstruct

tonavigate,thensed\-n/grep\-nforthespecificspansyouneed\.

\#\#2\.\[Pool\-specificintegritysection\]

\[\\benchpool:\]Thereisnoreward/reasontaxonomyinthispool\-\-

qualityjudgmentrestsentirelyon\(a\)internalevidence:code,

executionlog,numericalself\-consistency,evaluatorsourcelogic;and

\(b\)submissions\.jsonl\(ifpresent\)\-\-therealscored\-submissionhistory\.

Donotinventorborrowareasonfieldfromadifferentpool'sschema\.

\#\#3\.IronRulesforthistask

\[Pool\-specificselection/adaptationofthesharedIronRulesfrom

ONBOARDING\(2\)\.md\-\-e\.g\.forthe\\benchpool:retrieval\-integrity

rulesaremarkednotapplicable\(noretrievaltoolsexistinthis

container\);theevaluator\-feedback\-fittingrule\(see

Sec\.~\\ref\{app:evaluator:adaptation\}\)iselevatedtothesinglemost

importantcheckforthistaskandMUSTbewrittenupasitsown

standalone,evidencedissue\.\]

\-Followmulti\-versioncodetothefinaldeliveredartifact\.

\-Don'tbefooledbyabeautifulself\-diagnosisthatwasneveractedon\.

\-Perfect/extremescoresgetcheckedtoo:decomposethem;checkwhether

theyareproppedupbyprivilegedinformation\(e\.g\.anevaluator

defaultvalue,aleakedstartingcondition\)ratherthanacorrectmethod\.

\#\#4\.Outputrequirements\-\-hardgates,self\-checkbeforeyoufinish

\#\#\#4\.1Skeleton\(allsectionsrequired,headingscopiedverbatim\)

\[thefourteenheadingsfromSec\.~\\ref\{app:evaluator:rubric\}\]

\#\#\#4\.2Quotatable\(targets\-\-calibratedabovetheautomatedfloor\)

\[per\-sectiontargetcharactercountandtargetissuecount\-\-

seeTable~\\ref\{tab:sectionfloors\}forthecorrespondingpass/failfloor\]

\#\#\#4\.3Howtowriteeachissue\(thisiswhat"informationdensity"means\)

Eachissue=mechanism\(whatitconcretelydid\)\+whyit'sharmful\+

thehonest/charitablereading\+evidence,writtenasonefullparagraph\.

\-Atleastoneverifiableanchorperissue:anumber,aloglineindex,

afilename,acodeidentifier\.

\-Aone\-sentencebulletdoesnotcountasanissue\.Mergeorexpand

anythingunder~200characters\.

\-Donotpastelargeverbatimspansfromproblem\_readme\.md/result\.json\-\-

pastingisnotanalysisandwillbescoredaspadding\.

\-Donotrecycletemplatedphrasing\-\-everyissuemuststateafact

uniquetothisspecifictrajectory\.

\#\#\#4\.4Givefullcredit

Genuinestrengthsmustbewrittenupwiththesamerigorasfailures:

honestlyreportedbelow\-targetresults,genuinelyindependent

verificationdesign\(held\-out/cross\-validation\),self\-caughtbugs,

honestly\-flaggedfailedsubmissions,reproducibleseedsandnumbers\.

\#\#5\.Wheretowriteit

\{out\_dir\}/analysis\.md\(directoryalreadycreated;writeONLYthisfile\)\.

Statetheone\-lineverdictintheH1titleitself\.

Beforefinishing,self\-check:wordcount,issuecount,thatnoneofthe

sixstagesisperfunctory,thatthechecklisthasenoughrows\.Ifitdoes

notmeetthebar,keepdigging\-\-donotsubmitearly\.

\#\#6\.Boundaries

Deliveronlythisonefile\.Donotmodifyanythingelseintheworkspace,

donotrefactoranything\.Doallverification/re\-computationinyourown

mainloop\-\-donotspawnsub\-agents\.Youmayindependentlyre\-run

scoringlogiclocallywhenitsdependenciesarelight\(numpy/scipy/sympy\-

level\);ground\-truth/raw\-datadirectoriesareread\-onlyandmustbe

queriedselectively,nevertraversedinfull\.

## Appendix DFailure Pattern Statistics

The following tables present the detailed failure pattern statistics across the 800 agent trajectories analysed \(8 models×\\times100 tasks\)\. A pattern counts as a HIT when the trajectory analysis presents the failure as established, and as PARTIAL when it is raised but qualified; HIT% is the share of the 800 analyses in which the pattern is an established failure\. Table[12](https://arxiv.org/html/2608.14905#A4.T12)shows the overall totals per pattern\. Table[13](https://arxiv.org/html/2608.14905#A4.T13)provides the breakdown of the HITs across the evaluated models, using the abbreviations in Table[11](https://arxiv.org/html/2608.14905#A4.T11)\. Table[6](https://arxiv.org/html/2608.14905#A1.T6)visualizes the cross\-classification\.

Table 11:Model abbreviations used in Table[13](https://arxiv.org/html/2608.14905#A4.T13)\.Abbrev\.Modelnnsonclaude\-sonnet\-5100opuopus\-4\.8100dskdeepseek\-v4\-pro100glmglm\-5\.2100mmxminimax\-m3100qwnqwen3\.7\-max100gptgpt\-5\-mini100gemgemini\-3\.5\-flash100Total800Table 12:Failure Pattern Total HITs, ranked by HIT count\.PatternNameHITPARTIALHIT%F\.4Uncorrected\-SelfAware6603182\.5E\.2Overclaim6256778\.1D\.4Method\-Concl\-Disc6205077\.5C\.3Impl\-Discrep5776672\.1C\.1Circular\-Valid5521669\.0A\.5Metric\-Misalign5453568\.1F\.2Fail\-Gate50212862\.8E\.3Omit\-Limits49814362\.2D\.7Unremediated\-Adv4869760\.8E\.1Report\-Trace\-Gap48410260\.5B\.4Shallow\-Search43916754\.9A\.6Hyp\-Exp\-Mismatch4208752\.5D\.1Artifacts\-as\-Insight4196252\.4D\.3Stat\-Misuse41517651\.9D\.5Baseline\-Deficit37416346\.8A\.2Unfalsifiable3576944\.6X\.6Right\-Wrong\-Reason3507143\.8F\.1Superficial\-Review31915039\.9X\.3Skeptic\-Deficit31520139\.4X\.5Teleological3146739\.2E\.4Method/Cite\-Fab3126539\.0F\.3No\-Adversarial30627438\.2B\.2Retrieval\-Gap2717133\.9D\.2Confirmation\-Bias25816232\.2B\.1Hallucinated\-Evid21111726\.4C\.4Exec\-Fault20620225\.8X\.8Eng\-Delivery20112325\.1X\.1Cascade19013423\.8A\.1Frame\-Lock18430923\.0B\.3Unvetted\-Data17916122\.4X\.7Anchoring1628920\.2B\.5Citation\-Decorr15211119\.0C\.7Premature\-Term12910816\.1D\.6Result\-Halluc1253015\.6C\.6Local\-Opt9611912\.0C\.8Env\-Interact9612412\.0C\.2Grader\-Fit/Leak814810\.1A\.4Feasibility79929\.9X\.2Goal\-Drift62577\.8X\.4Honest\-Hollow62907\.8C\.5Infra\-Misdiag51206\.4A\.3Redundant15171\.9B\.6Low\-SNR9251\.1F\.6Halluc\-Review3140\.4F\.5Review\-Hack120\.1Table 13:Failure Pattern×\\timesModel HIT matrix\. Column abbreviations are defined in Table[11](https://arxiv.org/html/2608.14905#A4.T11);�\\Sigmais the row total\.Patternsonopudskglmmmxqwngptgem�\\SigmaA\.1 Frame\-Lock1725221223223528184A\.2 Unfalsifiable4444414349404353357A\.3 Redundant3310115115A\.4 Feasibility731298249779A\.5 Metric\-Misalign6964676967747164545A\.6 Hyp\-Exp\-Mismatch5046504947596059420B\.1 Hallucinated\-Evid1917161323256137211B\.2 Retrieval\-Gap4237403931452512271B\.3 Unvetted\-Data1826222129232416179B\.4 Shallow\-Search6158494549574971439B\.5 Citation\-Decorr17262620232947152B\.6 Low\-SNR201020139C\.1 Circular\-Valid6865726474696674552C\.2 Grader\-Fit/Leak79141415321781C\.3 Impl\-Discrep7463827075875472577C\.4 Exec\-Fault218292826443218206C\.5 Infra\-Misdiag62955813351C\.6 Local\-Opt12101111121651996C\.7 Premature\-Term6713910234417129C\.8 Env\-Interact7414632930396D\.1 Artifacts\-as\-Insight5050544664504560419D\.2 Confirmation\-Bias3137293329332442258D\.3 Stat\-Misuse5252454454494871415D\.4 Method\-Concl\-Disc7580817281777381620D\.5 Baseline\-Deficit4647513636464864374D\.6 Result\-Halluc332961936623125D\.7 Unremediated\-Adv6856665373626444486E\.1 Report\-Trace\-Gap4649685161776666484E\.2 Overclaim7786727281806988625E\.3 Omit\-Limits7060525563625779498E\.4 Method/Cite\-Fab2714392242446658312F\.1 Superficial\-Review3318373640416054319F\.2 Fail\-Gate6049735061776567502F\.3 No\-Adversarial3019373541355455306F\.4 Uncorrected\-SelfAware8284878484928760660F\.5 Review\-Hack000000011F\.6 Halluc\-Review101100003X\.1 Cascade1914362425282420190X\.2 Goal\-Drift548710212562X\.3 Skeptic\-Deficit3531392447494941315X\.4 Honest\-Hollow522104729362X\.5 Teleological3855293648481941314X\.6 Right\-Wrong\-Reason4250433953442653350X\.7 Anchoring2111241516282720162X\.8 Eng\-Delivery158233213543818201�\\Sigma1481139616161410161718181679169512712
## Appendix ETop\-10 Failure Patterns by Model

Table entries are drawn directly from the per\-model hit matrix \(Table[13](https://arxiv.org/html/2608.14905#A4.T13)\); each model’s ten most frequent patterns are listed in descending order of HIT count\. Ties at the tenth position are broken by pattern ID\. Every model is evaluated on the same 100 tasks, so a HIT count is also the percentage of that model’s trajectories in which the pattern is an established failure, and counts are directly comparable across tables\.

Table 14:Top\-10 failure patterns — claude\-sonnet\-5\.PatternHITF\.4 Uncorrected\-SelfAware82E\.2 Overclaim77D\.4 Method\-Concl\-Disc75C\.3 Impl\-Discrep74E\.3 Omit\-Limits70A\.5 Metric\-Misalign69C\.1 Circular\-Valid68D\.7 Unremediated\-Adv68B\.4 Shallow\-Search61F\.2 Fail\-Gate60Table 15:Top\-10 failure patterns — opus\-4\.8\.PatternHITE\.2 Overclaim86F\.4 Uncorrected\-SelfAware84D\.4 Method\-Concl\-Disc80C\.1 Circular\-Valid65A\.5 Metric\-Misalign64C\.3 Impl\-Discrep63E\.3 Omit\-Limits60B\.4 Shallow\-Search58D\.7 Unremediated\-Adv56X\.5 Teleological55Table 16:Top\-10 failure patterns — deepseek\-v4\-pro\.PatternHITF\.4 Uncorrected\-SelfAware87C\.3 Impl\-Discrep82D\.4 Method\-Concl\-Disc81F\.2 Fail\-Gate73C\.1 Circular\-Valid72E\.2 Overclaim72E\.1 Report\-Trace\-Gap68A\.5 Metric\-Misalign67D\.7 Unremediated\-Adv66D\.1 Artifacts\-as\-Insight54Table 17:Top\-10 failure patterns — glm\-5\.2\.PatternHITF\.4 Uncorrected\-SelfAware84D\.4 Method\-Concl\-Disc72E\.2 Overclaim72C\.3 Impl\-Discrep70A\.5 Metric\-Misalign69C\.1 Circular\-Valid64E\.3 Omit\-Limits55D\.7 Unremediated\-Adv53E\.1 Report\-Trace\-Gap51F\.2 Fail\-Gate50Table 18:Top\-10 failure patterns — minimax\-m3\.PatternHITF\.4 Uncorrected\-SelfAware84D\.4 Method\-Concl\-Disc81E\.2 Overclaim81C\.3 Impl\-Discrep75C\.1 Circular\-Valid74D\.7 Unremediated\-Adv73A\.5 Metric\-Misalign67D\.1 Artifacts\-as\-Insight64E\.3 Omit\-Limits63E\.1 Report\-Trace\-Gap61Table 19:Top\-10 failure patterns — qwen3\.7\-max\.PatternHITF\.4 Uncorrected\-SelfAware92C\.3 Impl\-Discrep87E\.2 Overclaim80D\.4 Method\-Concl\-Disc77E\.1 Report\-Trace\-Gap77F\.2 Fail\-Gate77A\.5 Metric\-Misalign74C\.1 Circular\-Valid69D\.7 Unremediated\-Adv62E\.3 Omit\-Limits62Table 20:Top\-10 failure patterns — gpt\-5\-mini\.PatternHITF\.4 Uncorrected\-SelfAware87D\.4 Method\-Concl\-Disc73A\.5 Metric\-Misalign71E\.2 Overclaim69C\.1 Circular\-Valid66E\.1 Report\-Trace\-Gap66E\.4 Method/Cite\-Fab66F\.2 Fail\-Gate65D\.7 Unremediated\-Adv64B\.1 Hallucinated\-Evid61Table 21:Top\-10 failure patterns — gemini\-3\.5\-flash\.PatternHITE\.2 Overclaim88D\.4 Method\-Concl\-Disc81E\.3 Omit\-Limits79C\.1 Circular\-Valid74C\.3 Impl\-Discrep72B\.4 Shallow\-Search71D\.3 Stat\-Misuse71F\.2 Fail\-Gate67E\.1 Report\-Trace\-Gap66A\.5 Metric\-Misalign64
## Appendix FData Construction Details

### F\.1 Task Extraction Prompt

Each paper is converted into one task by a single language\-model pass\. The model is given the cleaned body text of the paper \(truncated to14,00014\{,\}000characters; papers yielding fewer than200200characters of parsed text are dropped\) and returns a JSON record holding the seven fields of Section[2\.1](https://arxiv.org/html/2608.14905#S2.SS1), a list of key claims, and a novelty\-move label\. The pass is a reconstruction, not an evaluation: the model reports what the paper itself states, and the record is treated as provisional until it passes the rater check described in Section[2\.1](https://arxiv.org/html/2608.14905#S2.SS1)\.

#### F\.1\.1 System prompt

Youareacomputational\-scienceresearcher\(anydomain:chemistry,physics,

materials,biology,etc\.\)readingapapertoextractitsDISCOVERYPATTERN\-\-not

tosummariseit\.YououtputstrictJSONonly,noprosearoundit\.Befaithfulto

whatthepaperactuallyclaims;neverinventnumbers\.Ifafieldisnotstated,

useanemptystringoremptylist\.QuoterealnumbersWITHUNITSwhenthepaper

givesthem\.DoNOTforcethepaperintoanyparticularsubfield\-\-describethe

systemandquantitiesthepaperisactuallyabout\.

#### F\.1\.2 Extraction prompt

Placeholders in braces are substituted per paper:\{moves\}is the label set of Appendix[F\.1\.3](https://arxiv.org/html/2608.14905#A6.SS1.SSS3), and\{pid\},\{title\},\{narrative\}are the paper’s identifier, title, and cleaned body text\.

ExtractthediscoverypatternofthispaperasJSONwithEXACTLYthesekeys:

\{

"premise\_consensus":"theestablishedpriorunderstandingthepaperbuildson/

takesasgiven\(fromtheintroduction\)\.Whatdidthefieldalreadybelieve?",

"tension":"thegap,contradiction,oranomalyinthatconsensusthatmotivated

thiswork\.Thisisthediscoveryseed\-\-whatwasunsatisfyingorunknown?

Emptystringifthepaperispurelyconfirmatory\.",

"motivation":"whyresolvingthattensionmatters\(thestatedgoal\)\.",

"method":"theapproachtakentoresolveit\(e\.g\.DFT/MD/MLIPsetup,experiment,

model,code\)\.Oneortwosentences\.",

"experiment":"whatsystemwasstudiedandwhatwasactuallycomputed/measured,

withkeyconditions/comparisons\-\-whatevertheyareforTHISpaper

\(molecule,surface,crystal,defect,interface,device,reaction,\.\.\.\)\.",

"conclusion":"theterminalclaim\-\-whatthepaperconcludes\.Bespecific\.",

"key\_claims":\[

\{"claim":"asingleconcreteclaimfromtheconclusions",

"kind":"quantitative"or"qualitative",

"observable":"theNAMEofthephysicalquantitythisclaimisabout,

snake\_case,OPENVOCABULARYusingthepaper'sownquantity\(e\.g\.

adsorption\_energy,formation\_energy,band\_gap,reaction\_barrier,

diffusion\_coefficient,elastic\_modulus,binding\_affinity,

redox\_potential,vibrational\_frequency,conductivity,magnetic\_moment,

lattice\_constant,\.\.\.\)\.Emptyifpurelyqualitativewithnomeasurable

quantity\.",

"value":"thenumericvalueifquantitative,elseashortphrase",

"unit":"theunitofvalue\(eV,eV/atom,A,cm^\-1,K,GPa,\.\.\.\),emptyif

dimensionless/none",

"computable":"yesifanindependentfirst\-principles/atomisticcalc

\(DFT/MD/MLIP/quantum\-chem\)orcodecouldinprinciplere\-derivethis

quantityfromthedescribedsystem;elseno"\}

\],

"novelty\_move":"classifythepaper'sprimarymoveasONEof:\{moves\}",

"novelty\_rationale":"onesentencejustifyingthemovelabel,groundedinthe

tension\+conclusion\."

\}

Rules:

\-"tension"isthemostimportantfieldfordiscovery\.Agoodtensionreadslike

"consensussaidX,butYwasunexplained/measuredwrong/nevertested\."

Donotrestatetheconclusionasthetension\.

\-"observable"isOPEN\-\-namethequantitythepaperactuallyreports;doNOT

shoehornintoadsorption/catalysistermsunlessthepaperisgenuinelyabout

that\.

\-key\_claims:1\-4items\.PreferclaimsthatarequantitativeANDcomputable=yes,

butkeepthepaper'srealquantitiesregardless\.

\-OutputONLYtheJSONobject\.

PAPER\(id:\{pid\},title:\{title\}\):

\{narrative\}

#### F\.1\.3 Novelty\-move label set

The model chooses exactly one label from the following list, given verbatim with the glosses below; replies that do not match a label are recorded as other\.

consensus\-overturnpriorconsensuswaswrong;thisworkflipsit

method\-correctionaknownmethodgiveswronganswers;fixthemethod

new\-regimeextendsafindingtoaregimenobodymeasured

\(coverage/temperature/pressure/\.\.\.\)

mechanismexplains\*why\*anobservedeffecthappens

scaling\-relationfindsadescriptororrelationpredictingmanysystems

reconciliationresolvesacontradictionbetweentwopriorresults

incrementalconfirmsorrefinesconsensuswithoutbreakingit

#### F\.1\.4 Targeted\-anchor Optimization Tasks

The 30 target\-anchored tasks follow the task\-construction method of NatureBench\[[41](https://arxiv.org/html/2608.14905#bib.bib41)\]: each task exposes an explicit objective—a published human state of the art or a computable metric—against which a rollout’s submission is scored\. Because these tasks inherit the source benchmark’s selection, they are not restricted to the 2024\-onward window applied to the open\-ended subset \(§[2\.1](https://arxiv.org/html/2608.14905#S2.SS1)\)\.

### F\.2 Agent Rollout System Prompt

\#\{\{TITLE\}\}

\#\#Background\(establishedconsensus\)

\{\{PREMISE\}\}

\#\#Theopenquestion\(tension\)

\{\{TENSION\}\}

\*\*Yourtask:\*\*investigatethisopenquestionyourself,end\-to\-end,asarealresearchprocess

notjust"computeonenumberandstop\."Thiscorpusspansmanyfields\(physics,chemistry,biology,

materials,medicine,geophysics,energy,industrialsystems,scientificcomputing,\.\.\.\),sothereis

nodefaultmethod\-workthroughthesixstagesbelowforreal,inorder\.Ateachstage,recordthe

specificartifactrequestedfor\`process\_log\`\(schemabelow\)asyougo,notreconstructed

afterward\-itshouldreflectwhatyouactuallydidandconsidered,includinganythingthatdidn't

workorthatyoudecidedagainst\.

\*\*AIdeation\*\*\-Restatethetensioninyourownwords,thencommittoaspecific,falsifiable

hypothesis\.Beforemovingon,nameatleastoneotherwayyoucouldhaveframedthisquestionor

approachedit,andsaywhyyoupickedthisoneinstead\.Alsosayconcretely:whatresult,number,

orcomparisonwouldtellyouyourhypothesisisWRONGifnothingcouldtellyouthat,the

hypothesisisn'treadyyet\.

\*\*BRetrieval&Synthesis\*\*\-Use\`WebSearch\`/\`WebFetch\`\(oryourowndomainknowledgewhere

lookupsaren'tfruitful\)tocheckwhat'salreadyestablishedaboutthisspecificquestion\.For

eachsourceyouactuallyuselater\(inStageE\),beabletosaywhatitestablishesandwhich

specificclaimofyoursitsupportsnotjustthatitseemedtopicallyrelated\.Organizewhatyou

findbeforemovingon;don'tcarryitforwardasanunsortedpile\.

\*\*CExecution\*\*\-ChoosewhatevermethodactuallyanswersTHISpaper'sownquestionthatcould

beafirst\-principles/quantumcalculation,amolecularoragent\-basedsimulation,a\(micro\)kinetic

orratemodel,astatisticalormachine\-learningmodel,anumerical/PDEsolver,aphylogeneticor

sequenceanalysis,asignal\-processingpipeline,aneconometricoroptimizationmodel,orsomething

elseentirely\.Pickwhateverthepaper'sownobservableactuallycallsfor,notwhatever'smost

familiartoyou\.ThesandboxhasPythonwithnetworkaccess\`pipinstall\`whateveryouneed\.

Buildthesystem/dataset,runitforreal,andinspectactualoutputateachstep;neverfabricate

aplausible\-lookingnumber\.Sayexplicitlywhatyoutested\(whichsettings,scales,orconditions\)

andwhatyoudeliberatelyleftuntested,andnoteanythingthatbroke,stalled,orneededa

workaroundalongtheway\.

\*\*DAnalysis\*\*Interpretwhatyoucomputed:doesitsupportorrefutetheStage\-Ahypothesis?

Nameatleastonealternativeexplanationforyourresultotherthantheoneyou'regoingwith,and

sayconcretelywhyyouruleditout\(ordidn't\)\.Statehowconfidentyouareandwhatspecifically

thatconfidenceisbasedon\.

\*\*EWriting\*\*TheMOMENTyouhaveafirstgroundedresult,writeapreliminary

\`/workspace/decision\.json\`\(schemabelow\)it'stheonlyartifactautomatedscoringreads,soget

itdownearlyandneverblockitonanythingelse\.Thenwrite\`/workspace/report\.md\`:arealpaper,

thekindascientistwritesafteractuallyfinishingthisinvestigation,withthesesections:

\-\`\#\#Abstract\`2\-3sentences:whatyouinvestigated,yourmethod,andthekeyresult\.

\-\`\#\#Introduction\`thetension/openquestionandyourhypothesis\(StageA\)\.

\-\`\#\#Methods\`whatyoubuiltorran,andwhyitanswersthisquestion\(StageC\)\.

\-\`\#\#Results\`whatyouactuallycomputed,withtherealnumbers\.

\-\`\#\#Discussion\`yourinterpretation,thealternativeexplanationyouweighed,andyour

confidence\(StageD\)\.

\-\`\#\#Limitations\`whatyoudidn'ttest,andanythingthatbrokeorneededaworkaround\.

\-\`\#\#Conclusion\`theone\-paragraphtakeaway\.

\-\`\#\#References\`eachsourceyouactuallyretrievedandreadinStageB,andwhatit

establishes\.Don'tlistanythingyoudidn'tactuallyopenandread;don'tcitefrommemory\.

Ifyourunlowonturnsortime,anunfinishedreport\.mdisfine,anunfinisheddecision\.jsonisnot\.

\*\*FReview\*\*Re\-openandre\-readreport\.mdnotfrommemoryasaskepticalpeerreviewer

would\.Appenda\`\#\#PeerReview\`sectioninthereviewer'svoice\(thirdperson,e\.g\."Theauthors

claimX,butYisnotruledoutbecause\.\.\."\):namethesingleweakestpointintheargument,and

whatevidencewouldchangetheverdict\.Mirroritintodecision\.json's\`process\_log\.review\`\.

\*\*AGPUisattachedtothissandbox\*\*\(checkwith\`nvidia\-smi\`or\`torch\.cuda\.is\_available\(\)'after

\`pipinstalltorch\`\)\.Ifyourmethodbenefitsfromittraining/inferenceforanMLmodel,

GPU\-acceleratedsimulation,largebatchedcomputationinstallaCUDA\-enabledbuildanduseit;

CPU\-onlyiscompletelyfineifthemethoddoesn'tneedone\.Don'tforceGPUusewhereitdoesn'tfit\.

\#\#Requiredoutputtwoartifacts,writteninthisorder,BEFOREyoustop:

1\.\`/workspace/decision\.json'\(schemabelow\)writeapreliminaryversionthemomentyouhavea

firstgroundedresult\.ThisistheONLYartifactautomatedscoringreads\.

2\.\`/workspace/report\.md'apaperonthewholeinvestigation\(StageE\),plusitsappended

\`\#\#PeerReview\`section\(StageF\)\.

\`\`\`json

\{

"problem":"thespecificquestionyouinvestigated\(1sentence\)",

"hypothesis":"thefalsifiableclaimfromStageA,statedbeforeyouhadresults",

"system":\{"description":"whatyoumodeledorstudied",

"spec":\{"\.\.\.":"machine\-readablebuildhintswhateverfieldsmakesenseforyoursystem"\}\},

"method":\{"approach":"themethodyouactuallyused,inyourownwords\(notacategorylabel\)",

"tools":"packages/engines/datasetsused","key\_params":\{\}\},

"observable":"thequantityyoucomputednameithoweverit'sactuallynamedinthisfield,usingthepaper'sOWNquantity,notagenericplaceholder",

"result":\{"value":0\.0,"unit":"\.\.\.","details":\{\}\},

"conclusion":"one\-sentencefindingthatresolvestheopenquestion,groundedinYOURcomputedresulthedgehonestlyiftheevidenceisweak\(StageF\)",

"process\_log":\{

"ideation":\{"alternative\_framing\_considered":"theotherwayyoucouldhaveapproachedthis",

"falsification\_check":"whatresultwouldhavetoldyouthehypothesiswaswrong"\},

"retrieval":\{"sources":\[\{"source":"\.\.\.","establishes":"\.\.\.","supports\_claim":"\.\.\."\}\],

"synthesis\_note":"howyouorganizedwhatyoufoundbeforemovingon"\},

"execution":\{"settings\_tested":\["\.\.\."\],"deliberately\_not\_tested":\["\.\.\."\],

"issues\_encountered":\["anythingthatbroke,stalled,orneededaworkaround"\]\},

"analysis":\{"alternative\_explanation\_considered":"\.\.\.","why\_ruled\_out\_or\_not":"\.\.\.",

"confidence":"\.\.\.","confidence\_basis":"whatspecificallythatconfidencerestson"\},

"review":\{"weakest\_point":"\.\.\.","what\_would\_change\_your\_mind":"\.\.\."\}

\}

\}

\`\`\`

\*\*Writedecision\.jsonEARLY,report\.mdsecond\*\*writeapreliminarydecision\.jsonassoonas

youhaveafirstcomputedestimate,keeprefiningitthroughStagesD\-F,andonlystartreport\.md

oncedecision\.jsonexistsneverletwritingreport\.mddelayit\.Onlyre\-executed,grounded

resultscountbaseyourconclusiononwhatyouactuallycomputed,notarememberedliterature

value\.Fillin\`process\_log\`andreport\.md's\`\#\#PeerReview\`sectionhonestlyanincompleteor

hedgedentryisfarmoreusefulthanareconstructedonethatjustmakestheprocesslookclean\.

### F\.3 Rollout Environment and Hardware

All rollouts were executed on a single shared compute node equipped with8×8\\timesNVIDIA B300 GPUs \(Blackwell architecture, compute capabilitysm\_103,∼\\sim288 GB HBM per card\), 256 CPU cores, and 3 TB of system RAM, managed by SLURM\. Each model–task rollout runs inside its own isolated Docker container; per\-container GPU visibility is pinned to the node’s SLURM\-allocated physical device viaNVIDIA\_VISIBLE\_DEVICESunder the NVIDIA container runtime, so containers never see GPUs outside their lease\.

##### Container image\.

Task containers are built on a common base image with PyTorch 2\.11\.0 compiled for CUDA 12\.8 \(cu128\), required for Blackwell/B300 \(sm\_103\) support; earliercu118builds fail on this hardware with a “no kernel image” error\. NumPy is pinned to 1\.26\.4 and all pip dependencies are version\-pinned for reproducibility\.

##### Agent harness and sandbox\.

Each task is presented to the agent inside the container with a read\-only problem directory \(/task/problem, held\-out data\) and a writable scratch workspace \(/workspace\)\. Agents interact through one of three CLI harnesses—Claude Code, Codex, or Gemini CLI—driving a ReAct\-style loop \(Bash/Read/Write/Edittools\)\.

##### CPU/GPU scheduling\.

To co\-locate many rollouts on one node, CPU\-only tasks run unconstrained up to a concurrency cap \(max\_workers=12=12\), while the1515GPU\-bearing tasks share a small pool of GPU slots \(shared\_gpu\_slots=2=2per device\)\. Each container is given up to3232CPU cores and150150GB RAM\. A per\-task wall\-clock budget of44hours \(14,40014\{,\}400s, plus a3,6003\{,\}600s environment\-setup allowance\) is enforced by a watchdog that queries a time\-remaining endpoint and gracefully stops the container on expiry\.

##### Model serving\.

The eight evaluated models \(glm\-5\.2,claude\-sonnet\-5,opus\-4\.8,deepseek\-v4\-pro,qwen3\.7\-max,minimax\-m3,gemini\-3\.5\-flash,gpt\-5\-mini\) are served through OpenRouter behind a local Anthropic\-/OpenAI\-compatible gateway\. No task\-specific model fine\-tuning is performed; agents are prompted zero\-shot\.

##### Web Access

Web access differs by subset\. Inoptimization, internet access is disabled in all containers: the Claude harness uses\-\-disallowedTools WebSearch,WebFetch, the Gemini harness a no\-web policy \(\-\-policy /etc/naturebench/no\-web\.toml\), and the Codex harnessweb\_search=disabled\. Inopen\-ended, agents are granted theWebSearchandWebFetchtools\. Because non\-Anthropic models served via OpenRouter cannot execute the Claude CLI’s web\-search sub\-request natively, that single sub\-request is routed through a transparent proxy to a genuine Anthropic\-backed search; all other turns are handled by the model under evaluation\. We refer to this live\-retrieval configuration as*realsearch*: agents in the open\-ended pool issue genuine queries against a real search backend and receive real retrieved content, as distinct from a*search shim*, which returns model\-generated text in place of retrieved results\. Retrieval\-integrity patterns \(B\.1, B\.5, E\.4\) are therefore established against what the agent actually fetched, recorded in the retrieval log\.

## Appendix GAutoResearchEval Full Task List

Domain labels were initially assigned by an LLM\-as\-judge applied to each paper’s title, abstract, and journal source\. Because some papers are interdisciplinary, the LLM labels can be imperfect; all labels were subsequently verified by human review, which found the LLM assignments to be over 95% accurate\.

Table 22:Full task list of the open\-ended discovery subset \(n=70n=70\)\. Descriptions state the scientific tension each task is built around; venue and year give the source paper’s provenance\.Task description \(tension\)DomainVenueYearThe role of the lymphatic system in musculoskeletal health: lymphatic vessels long believed absent in bone and adult intervertebral discs were challenged by recent imaging\.biologyBone Research2026Whether tephra burial can sequester enough soil organic carbon to offset or exceed magmatic CO2, making explosive eruptions net carbon sinks\.geophysicsNature Communications2025Whether an ML potential trained only on energies and forces can yield reliable analytical Hessians for transition\-state optimization\.material\_scienceNature Communications2024Distinguishing interfacial from bulk contributions to SO2hydrolysis given poorly characterized hydrolysis rate constants\.physicsNature Communications2025Whether Zr self\-diffusion in BCC refractory high\-entropy alloys is enhanced rather than sluggish, contradicting the “sluggish diffusion” concept\.chemistryActa Materialia2025Competing itinerant and local spin interactions in kagome metal FeGe: the double\-cone model fails to reproduce the measured spin\-wave spectrum\.physicsNature Communications2024A physically inspired, deterministic relation linking microearthquake seismic moment to permeability change across scales in crystalline rock\.geophysicsScience Advances2026A fast method to estimate the minimum detectable dark\-matter subhalo mass from a stellar stream’s basic properties across many Milky Way streams\.physicsThe Open Journal of Astrophysics2026Whether a fundamental trade\-off exists between gradient\-measurement efficiency and expressivity \(DLA dimension\) in deep quantum neural networks\.scientific\_computingnpj Quantum Information2025Which gut microbiome features associate with Parkinson’s disease, and the cross\-study portability of microbiome\-based ML diagnostic models\.biologyNature Communications2025Whether a reported LNO\-CCSD\(T\) vs\. FN\-DMC disagreement for large non\-covalent systems undermines the “gold standard” reference itself\.chemistryNature Communications2025Reconciling contradictory climate\-migration findings by accounting for demographic heterogeneity in migration responses to weather\.geophysicsNature Communications2025Reconciling high\-affinity solution\-based DNMT1/UMDNA binding with crystallographic weak\-binding results\.chemistryNature Communications2025Whether a consensus nomenclature for addictive\-like foods \(UPFs, highly processed, hyper\-palatable\) can advance food addiction as an empirical construct\.medicineCurrent Addiction Reports2025Whether sub\-Chandrasekhar model variance can account for the full range of observationally inferred iron\-group nucleosynthetic ratios\.physicsThe Astronomy and Astrophysics Review2025How on\-dyad vs\. off\-dyad linker\-histone binding modes relate to distinct higher\-order chromatin fiber structures\.scientific\_computingCell Research2024Resolving broad spatial neighborhoods in glioblastoma to individual cell states and their organizational rules\.biologyCell2024A unified mechanistic account of lung\-specific metastasis across niche induction, colonization, dormancy, and reawakening\.medicineMolecular Cancer2025Whether and how background hydroclimate shapes photosynthetic sensitivity to cloud cover across global terrestrial ecosystems\.geophysicsNature Communications2026Resolving the dynamic lifecycle of pores in directed energy deposition, from formation to escape or entrapment\.material\_scienceNature Communications2024Downstream proteomic, PTM, and metabolic consequences of genetic alterations in high\-grade gliomas, toward actionable protein\-level networks\.medicineCancer Cell2024The biological function of bacterial Teneurin\-like proteins and their structural/functional relation to metazoan Teneurins\.biologyNature Communications2026A unified framework to identify bottlenecks preventing self\-driving labs from being efficient, accessible, and interoperable\.scientific\_computingNature Communications2025Open\-source software integrating multi\-level 3D organoid segmentation with pluggable, benchmarkable AI nuclei\-segmentation models\.biologyNature Methods2025Molecular mechanisms of E3\-ligase selectivity and proteasomal discrimination of distinct ubiquitin\-chain linkages\.medicineSignal Transduction and Targeted Therapy2025Physics\-informed ML for computational medical imaging: adding physical plausibility, data efficiency, and interpretability\.physicsArtificial Intelligence Review2025The persistent PCE gap between small\-area perovskite cells and large\-area modules, and theJs​cJ\_\{sc\}shortfall vs\. the Shockley–Queisser limit\.material\_scienceNano\-Micro Letters2026Molecular mechanisms by which aberrant zinc\-transporter expression and zinc signaling drive tumorigenesis and therapy resistance\.chemistrySignal Transduction and Targeted Therapy2024Waveform modelling for LISA: covering wider source types and parameter space at the accuracy required for its science goals\.physicsLiving Reviews in Relativity2025Relative performance of subclonal\-reconstruction algorithms on single\-sample designs, and the features driving accuracy\.biologyNature Biotechnology2024Keeping the transcription\-rate integral tractable for efficient ODE solving while modeling complex nonlinear rates over differentiation time\.scientific\_computingNature Methods2025Cell–cell communication modeling with explicit downstream pathways and disambiguation of converging signaling routes\.biologySignal Transduction and Targeted Therapy2026Reconstructing completely damaged fNIRS channels \(low SNR, abnormal spectra\) instead of discarding channels or subjects\.geophysicsArtificial Intelligence Review2024Polygenic prediction jointly leveraging high\-density SNP panels and diverse functional annotations, within and between ancestries\.biologyNature Genetics2024The validity of using land surface temperature as a proxy for air temperature in urban impact assessments\.scientific\_computingNature Communications2026Reaching high\-efficiency cavity\-enhanced AFC quantum memories despite the finesse–bandwidth \(slow\-light\) trade\-off\.physicsNature Photonics2026Decomposing non\-stationary level, trend, and seasonality in volatile cryptocurrency series for robust price forecasting\.scientific\_computingJournal of Big Data2025Building an AI\-driven virtual cell that captures multi\-scale, nonlinear, massively interacting cellular dynamics\.biologyCell2024Why triplet nitroarenes with similarETE\_\{T\}show drastically different energy\-transfer reactivity, beyondETE\_\{T\}\-matching\.chemistryNature Catalysis2025Whether subduction\-interface seismic slip is on a single plane or a distributed multifault network, and its effect on aftershock evolution\.geophysicsNature2024Infusing stereoelectronic effects into molecular\-graph ML representations without prohibitive quantum\-chemistry cost\.material\_scienceNature Machine Intelligence2025Generalizable pancancer prognosis prediction from histopathology plus routine clinical variables alone\.medicineSignal Transduction and Targeted Therapy2025Robust differential diagnosis of multiple and mixed dementia etiologies from routine multimodal data\.medicineNature Medicine2024In\-situ local\-information training of mechanical neural networks, and whether physical MNNs can learn, retrain, and recover from damage\.scientific\_computingNature Communications2024Comprehensive benchmarking of single\-cell multimodal omics integration methods across many tasks\.biologyNature Methods2025Reproducible baseline electrochemistry for model single\-crystal and polycrystalline NMC cathodes to isolate genuine CEI improvements\.chemistryNature Energy2024Integrating seismic and rock\-physics data in a unified quantitative reservoir\-characterization framework for the Eastern Potwar region\.geophysicsReservoir Science2026Unifying scarce, disjointed, unstructured hypersonics materials data to enable AI\-driven accelerated screening\.material\_scienceNature Communications2024Bridging MXene lab synthesis and observations with the atomistic mechanisms governing individual flake quality for healthcare\.chemistryChemical Society Reviews2025Mechanistic contributions of exosomes to immunopathology and tumor immunity, and their therapeutic targeting\.medicineCellular and Molecular Immunology2025A trustworthy deep\-learning benchmark on public multi\-sensor data for multiclass dairy\-cow lameness detection with human\-in\-the\-loop\.scientific\_computingEngineering Applications of Artificial Intelligence2025Automatically extracting biological information \(organelle identity, viral uncoating\) directly from single\-particle diffusional behavior\.biologyNature Methods2025Target\-aware molecule generation jointly optimizing affinity, drug\-likeness, and synthetic accessibility with integrated refinement\.chemistryNature Communications2024Whether accurately quantified nanomolar macronutrients suffice for glacier ice\-algae metabolism during the melt season\.geophysicsNature Communications2026A framework combining first\-principles DFT with experimental SERS spectra to recommend optimal molecular receptors\.material\_scienceNature Communications2025Systematically quantifying clinical severity of hallucinations and omissions in LLM\-generated clinical notes, with iterative reduction\.medicinenpj Digital Medicine2025Sequencing\-guided re\-estimation and promotion of cultivability for environmental bacteria beyond indirect proxies\.biologyNature Communications2024A PINN surrogate for full nonlinear equilibrium\-path analysis of shallow trusses, including post\-critical response, without expert intervention\.scientific\_computingJournal of Big Data2025A computational\-pathology foundation model for rare cancers and unusual variants where labeled data are scarce\.biologyNature Medicine2024Augmenting LLMs with expert chemistry tools for reliable IUPAC\-to\-structure conversion and multi\-step reasoning\.chemistryNature Machine Intelligence2024Extending reliable flood forecast skill beyond lead time 0 in ungauged watersheds, closing the Africa–Europe skill gap\.geophysicsNature2024The ground\-state electronic structure, geometry, and aromaticity of the elusive odd\-numbered cyclo\[13\]carbon and its dimer\.material\_scienceScience2024Patient\-similarity and length\-of\-stay prediction tailored to rare, heterogeneous paediatric cardiology populations\.medicineNature Communications2026Exploiting device\-level memristor computation for tonotopic mapping while preserving biological interpretability\.biologyNature Communications2024An up\-to\-date, structured taxonomy of deep time\-series anomaly\-detection methods, including recent representation\-learning models\.scientific\_computingACM Computing Surveys2024Effect sizes of four parental risk factors on preschoolers’ prolonged digital use, and cultural/measurement moderators\.medicineEducation and Information Technologies2024Adding clinically plausible omitted predictors to CVD risk scores to correct systematic mis\-estimation in key subgroups\.medicineNature Medicine2024Closing the gap between AI model development and clinical translation in orthopedics\.scientific\_computingKnee Surgery and Related Research2026Quantifying train–test leakage and structural redundancy in PDBbind and the true generalization of binding\-affinity models\.biologyNature Machine Intelligence2025Extracting DSM\-5 substance\-use\-disorder severity from unstructured clinical notes with an LLM where NLP/rule\-based methods fail\.medicinenpj Mental Health Research2025Table 23:Full task list of the target\-anchored optimization subset \(n=30n=30\)\. Descriptions state the task each optimization instance targets; venue and year give the source paper’s provenance\.n\.r\.marks a task whose source\-paper provenance is not recorded in our release\. Unlike the open\-ended subset \([Table22](https://arxiv.org/html/2608.14905#A7.T22)\), this subset follows the source benchmark’s task selection and is not restricted to papers from 2024 onward\.Task descriptionDomainVenueYearRadio link scheduling under geographic interference\.scientific\_computingIEEE J\. Sel\. Areas Commun\.2019Molecular property optimization via sequential editing\.chemistryScientific Reports2019Brain tumor segmentation in multi\-sequence MRI\.medicineJournal of Medical Imaging2019Guide\-RNA on\-target editing efficiency prediction\.biologyNature Communications2019Materials property prediction from stoichiometry\.material\_scienceNature Communications2020Protein function prediction\.biologyFront\. Bioeng\. Biotechnol\.2020Particle\-cloud jet compression, reconstruction, and anomaly detection\.physicsEur\. Phys\. J\. C2023Single\-cell identity annotation from a labeled reference\.biologyNature Biotechnology2023Cell\-type classification from multimodal single\-cell omics\.biologyNucleic Acids Research2023Quantum error correction: toric\-code logical\-error decoding\.physicsarXiv2023Partial atomic\-charge prediction for nanoporous framework crystals\.material\_sciencenpj Computational Materials2024Active\-learning\-assisted directed evolution on a protein fitness landscape \(ALDE\)\.biologyNature Communications2025Adsorption and reaction energies on bimetallic alloys \(explainable GNN\)\.material\_sciencen\.r\.n\.r\.Combination drug screen: predicting unseen combinations under a screening budget \(BATCHIE\)\.biologyNature Communications2025Clinical\-trial outcome prediction from trial\-design features \(CTO\)\.medicineNature Health2026Cosmological parameter inference from point clouds \(CosmoBench\)\.physicsarXiv2025Real\-bulk cell\-type deconvolution of breast\-cancer tumours with matched single\-cell ground truth\.biologyn\.r\.n\.r\.Iteration\-free crystal structure relaxation \(DeepRelax\)\.material\_scienceNature Communications2024Drug\-target affinity prediction with binding\-site information \(DMFF\-DTA\)\.chemistrynpj Digital Medicine2025Ejection\-fraction and volume estimation from echocardiogram video \(EchoNet\-Dynamic\)\.medicineNature Communications2025Recovering a permittivity map from scattered light \(linear inverse scattering\)\.physicsICLR2025Coupled\-cluster\-accuracy molecular energies from DFT\-level inputs \(MEHnet\)\.chemistryNature Computational Science2025Ionic migration\-barrier prediction via transfer learning\.material\_sciencenpj Computational Materials2026Molecular property prediction on MoleculeNet \(MotiL\)\.chemistryNature Communications2025Discovering network dynamics with symbolic regression \(ND2\)\.scientific\_computingNature Computational Science2026Recovering the initial vorticity of a 2D Navier–Stokes flow\.geophysicsICLR2025Single\-cell\-informed deconvolution of bulk RNA\-seq \(omnideconv\)\.biologyNature Communications2024Global reaction\-feasibility prediction \(HTE acid–amine coupling\)\.chemistryNature Communications20253D RNA inverse design — fixed\-backbone native\-sequence recovery \(gRNAde\)\.biologyICLR2025Data\-driven tokamak plasma\-profile evolution \(TORAX\)\.physicsarXiv2024