超越文本：验证由智能体撰写的论文是否有其制品支持

# Beyond the Text: Verifying That Agent-Written Papers Are Backed by Their Artifacts
Source: [https://arxiv.org/html/2609.22111](https://arxiv.org/html/2609.22111)
###### Abstract

Large language model agents are increasingly capable of conducting research autonomously, producing research documents alongside the code and experiments that ostensibly support them\. Yet whether the reported findings are consistently supported by corresponding implementations and execution evidence remains largely unexplored: existing review practices primarily assess textual quality and cannot reliably identify inconsistencies such as hard\-coded metrics, unimplemented methods, or unsupported experimental results\. We present ReAgent, an automated auditing framework for assessing the consistency between agent\-generated research documents and their associated repositories\. ReAgent constructs structured representations of scientific claims from research documents and uses them to guide repository analysis and evidence collection\. Static auditing examines whether claimed methodologies, implementations, and experimental configurations are consistently reflected in the repository, while dynamic auditing executes relevant experiments and collects execution evidence to assess empirical findings\. By combining static analysis with dynamic evidence, ReAgent identifies inconsistencies that may remain hidden under either perspective alone, such as experiments that reproduce reported numbers while deviating from the claimed methodology\. The collected evidence and audit decisions are organized into a structured repository\-level audit report, enabling transparent evidence traceability\. We evaluate ReAgent on a manually curated benchmark of agent\-generated research document–repository pairs and compare it against representative static and reproduction\-based baselines\. Experimental results demonstrate that ReAgent effectively identifies inconsistencies between reported research findings and their supporting repository evidence\.

## 1Introduction

In recent years, rapid advances in Large Language Models \(LLMs\) have significantly expanded the capabilities of intelligent agents\([34](https://arxiv.org/html/2609.22111#bib.bib1);[10](https://arxiv.org/html/2609.22111#bib.bib2)\)\. From software engineering agents\([11](https://arxiv.org/html/2609.22111#bib.bib3);[35](https://arxiv.org/html/2609.22111#bib.bib4)\)to embodied agents\([15](https://arxiv.org/html/2609.22111#bib.bib5);[13](https://arxiv.org/html/2609.22111#bib.bib6)\)and autonomous research agents\([37](https://arxiv.org/html/2609.22111#bib.bib7);[36](https://arxiv.org/html/2609.22111#bib.bib8)\), these systems are evolving into autonomous agents capable of planning, executing, and refining complex workflows\. In scientific research, recent systems have demonstrated the ability to perform literature review, hypothesis generation, code implementation, experiment execution, result analysis, and paper writing, covering increasingly large portions of the research pipeline\. Representative systems such as the AI Scientist\([16](https://arxiv.org/html/2609.22111#bib.bib10)\), VirSci\([27](https://arxiv.org/html/2609.22111#bib.bib9)\), Agent Laboratory\([24](https://arxiv.org/html/2609.22111#bib.bib11)\), and FARS\([29](https://arxiv.org/html/2609.22111#bib.bib12)\)highlight the growing potential of autonomous research\.

Existing studies primarily evaluate whether autonomous research agents can generate novel scientific ideas and high\-quality research outputs\([25](https://arxiv.org/html/2609.22111#bib.bib13);[3](https://arxiv.org/html/2609.22111#bib.bib14);[33](https://arxiv.org/html/2609.22111#bib.bib15);[5](https://arxiv.org/html/2609.22111#bib.bib23);[28](https://arxiv.org/html/2609.22111#bib.bib24)\)\. As these systems increasingly automate end\-to\-end scientific workflows, a more fundamental yet largely overlooked question arises:

Are the scientific claims reported by autonomous research agents trustworthy?

This issue is not unique to autonomous research\. In traditional scientific practice, inconsistencies between papers and their implementations have long contributed to the reproducibility crisis\([1](https://arxiv.org/html/2609.22111#bib.bib16);[6](https://arxiv.org/html/2609.22111#bib.bib17);[23](https://arxiv.org/html/2609.22111#bib.bib18)\)\. Autonomous research further amplifies this challenge because research documents, code, and experiments are often generated and iteratively refined independently\. Combined with well\-known limitations of foundation models, including hallucination\([9](https://arxiv.org/html/2609.22111#bib.bib33);[30](https://arxiv.org/html/2609.22111#bib.bib36)\), weak alignment between natural language and executable code\([2](https://arxiv.org/html/2609.22111#bib.bib37)\), and limited self\-verification\([7](https://arxiv.org/html/2609.22111#bib.bib34);[31](https://arxiv.org/html/2609.22111#bib.bib35)\), scientific claims may gradually diverge from their implementations and execution evidence\. These characteristics introduce new challenges in maintaining consistency among scientific claims, code implementations, and experimental evidence in autonomous research pipelines, motivating the need for dedicated mechanisms to verify their alignment\.

![Refer to caption](https://arxiv.org/html/2609.22111v1/aaai_frame.png)Figure 1:Problem definition\. An automated research agent generates a research documentPPand an associated repositoryRR\. During autonomous generation and iterative refinement, claim\-code inconsistencies may arise across the research workflow\. The examples are illustrative rather than exhaustive; see Section[3\.1](https://arxiv.org/html/2609.22111#S3.SS1)for the complete categorization\.Fundamentally, scientific research is trustworthy only when its claims are supported by empirical evidence\([32](https://arxiv.org/html/2609.22111#bib.bib19);[17](https://arxiv.org/html/2609.22111#bib.bib22)\)\. Methodological claims, experimental protocols, and reported results should all be grounded in corresponding implementations, configurations, and execution evidence\. However, discrepancies may arise when proposed methods are absent from the implementation, reported results cannot be reproduced, or experimental settings differ from those described in the paper\. Although such inconsistencies do not necessarily imply malicious intent, they undermine the credibility of autonomous research outputs\.

Regrettably, existing work has yet to establish a systematic evaluation framework targeting this issue\. While recent studies on automated paper replication and autonomous research have advanced scientific automation, they primarily focus on code quality or peer\-review scores\([26](https://arxiv.org/html/2609.22111#bib.bib20);[4](https://arxiv.org/html/2609.22111#bib.bib21);[28](https://arxiv.org/html/2609.22111#bib.bib24)\), rather than whether scientific claims are supported by both implementation and experimental evidence\. This challenge stems from the intrinsic dual nature of scientific claim verification: trustworthy assessment requires verifying both that the implementation matches the claimed methodology and that the reported results are supported by executable evidence\. Static analysis can examine whether a repository contains implementations corresponding to described methods, but cannot determine whether reported findings were actually produced by the current implementation\. Conversely, dynamic re\-execution can verify whether results are reproducible, but cannot establish whether the executed implementation faithfully matches the scientific claims\. Therefore, neither perspective alone is sufficient\. Autonomous research requires a unified framework that jointly reasons over implementation consistency and executable evidence\.

To bridge this gap, we introduce claim\-code consistency auditing and ReAgent, an automated framework that systematically examines agent\-generated research documents and repositories through complementary static verification and dynamic verification\. The source code will be publicly available athttps://github\.com/hongsq12345/ReAgent\.

The main contributions of this paper are summarized as follows:

- •We introduceClaim\-Code Consistencyas a new auditing problem for autonomous research, defining how scientific claims should be verified against both code implementations and executable evidence\.
- •We proposeReAgent, an automated claim verification framework that combines static implementation auditing with dynamic evidence collection to systematically verify research documents and repositories generated by autonomous research agents\.
- •We establish a benchmark of agent\-generated research repositories, enabling systematic evaluation of claim\-code consistency and providing empirical characterization of inconsistency patterns in autonomous research outputs\.

## 2Related Work

Autonomous Research Agents\.Recent advances in large language models have enabled autonomous research agents capable of performing increasingly complete scientific workflows\. Representative systems, including The AI Scientist\([16](https://arxiv.org/html/2609.22111#bib.bib10)\), Agent Laboratory\([24](https://arxiv.org/html/2609.22111#bib.bib11)\), AI\-Researcher\([28](https://arxiv.org/html/2609.22111#bib.bib24)\), VirSci\([27](https://arxiv.org/html/2609.22111#bib.bib9)\), AutoSOTA\([12](https://arxiv.org/html/2609.22111#bib.bib25)\), ScientistOne\([20](https://arxiv.org/html/2609.22111#bib.bib26)\), and AutoResearchClaw\([14](https://arxiv.org/html/2609.22111#bib.bib27)\), demonstrate increasingly capable research automation spanning literature review, idea generation, implementation, experimentation, and paper writing\. Existing work primarily evaluates research capability, such as workflow completion, scientific quality, or research productivity, while paying limited attention to whether reported scientific claims are faithfully supported by implementations and experimental evidence\. Our work complements this line of research by treating generated research outputs and repositories as auditable objects and explicitly verifying their consistency\.

Trustworthy LLM Agents\.The increasing autonomy of LLM agents has raised growing concerns regarding their trustworthiness\. Recent studies investigate undesirable behaviors such as deceptive reasoning, specification gaming, and hidden coordination\([8](https://arxiv.org/html/2609.22111#bib.bib28);[19](https://arxiv.org/html/2609.22111#bib.bib29)\)\. In the scientific domain, prior work has revealed hidden implementation pitfalls in AI scientist systems\([17](https://arxiv.org/html/2609.22111#bib.bib22)\)and evaluated whether frontier LLM agents can reliably conduct realistic research activities\. These studies mainly focus on agent behavior during task execution, whereas we investigate whether the final research outputs faithfully support the scientific claims they report\.

AI\-assisted Scientific Auditing and Verification\.Recent work has employed LLM agents to reconstruct implementations from papers or automatically reproduce reported experimental results, including PaperBench\([26](https://arxiv.org/html/2609.22111#bib.bib20)\), DiscoveryBench\([18](https://arxiv.org/html/2609.22111#bib.bib30)\), and ScienceAgentBench\([5](https://arxiv.org/html/2609.22111#bib.bib23)\)\. While these approaches substantially reduce the manual effort required for scientific reproduction, their objective is fundamentally different from ours\. Successful reproduction alone cannot determine whether an implementation faithfully matches the reported methodology, and implementation inspection alone cannot verify empirical claims\. ReAgent bridges this gap by combining static implementation auditing with dynamic evidence collection to assess whether scientific claims are supported by their corresponding implementations and execution evidence\.

## 3Method

![Refer to caption](https://arxiv.org/html/2609.22111v1/aaai_frame.drawio.png)Figure 2:ReAgent framework\. Static verification constructs claim contracts, repository metadata, and semantic bindings to align scientific claims with implementation evidence and generate executable verification targets\. Dynamic verification performs review\-guided reproduction and organizes execution feedback into provenance\-aware evidence, enabling evidence\-grounded audit decisions\.This section first introduces the concept of claim\-code consistency auditing and then presents a two\-stage verification framework for auditing automatically generated research documents and their associated repositories\. Given a scientific artifact containing research claims and its corresponding repository, ReAgent constructs structured claim contracts, repository metadata, and semantic bindings in the static stage, and transforms them into executable verification targets\. In the dynamic stage, the Review Agent selects verification targets and plans verification strategies, while the Reproduce Agent executes the repository and collects execution evidence\. The system then produces evidence\-grounded consistency assessments and aggregates them into a verified analysis report\.

### 3\.1Problem Definition

Automated research agents are increasingly capable of conducting research and generating repositories\. Existing studies primarily focus on the task completion rates of agentic research or the novelty of scientific ideas\. However, these agents may intentionally or inadvertently produce inconsistent scientific outcomes\. To study this challenge, we consider an automated research agent that independently generates both a research document and its accompanying repository\. Our goal is not to model adversarial behavior, but to characterize the types of claim\-code inconsistencies that may emerge during autonomous research generation, regardless of whether they are introduced intentionally or unintentionally\. Here,PPdenotes the textual research output containing scientific claims to be audited, including conventional papers as well as automatically generated research reports, whileRRdenotes the associated repository containing implementation code and experimental configurations\. An overview of this problem setting is illustrated in Figure[1](https://arxiv.org/html/2609.22111#S1.F1)\. The agent is regarded as a potential risk source, operating with the freedom to alter both research document text and repository during the research process\. Consequently, it may produce seemingly plausible scientific artifacts that conceal underlying discrepancies between the research document and its implementation\. Such inconsistencies can arise unintentionally—from planning errors, execution failures, incomplete implementations, or hallucinated experimental results—or intentionally, such as when an agent optimizes the narrative for persuasiveness without ensuring alignment with the repository\.

Assuming the agent possesses unrestricted access to both the research document and the repository, we categorize the resulting inconsistencies into two major classes:

- •Static Repository Inconsistencies:Discrepancies where the repository fails to align with the claims made in the research document without requiring code execution\. - –Hardcoded Results: Reported figures are hardcoded into scripts or output templates rather than being generated through actual computation\. - –Substantive Implementation Discrepancies: The reported module exists and runs, but its core algorithmic steps diverge substantially from the textual description\. - –Omitted Implementation: Proposed methods or modules mentioned in the research document are absent from the repository\. - –Shell/Dummy Implementation: Modules are defined, but they are short\-circuited or never invoked during actual runtime\. - –Fabricated Datasets: The research document claims to use a specific dataset, but the code generates synthetic data or uses a minor subset instead\. - –Fabricated Experimental Hyperparameters: Claimed configurations or parameters are either omitted in code or fixed to arbitrary constant values\. - –Data / Label Leakage: The test set, answers, or labels are exposed during training, hyperparameter tuning, or random seed selection\.
- •Dynamic Execution Inconsistencies:Discrepancies between the actual execution outputs and the claims in the research document, typically involving exaggerated performance\. Under identical configurations, the executed code yields performance far below the claimed results or fails to execute entirely\.

Given a research documentPPand its corresponding repositoryRR, our objective is to determine whether scientific claims inPPare backed by verifiable evidence withinRR\. To audit these claims, our framework, ReAgent, employs a complementary static\-and\-dynamic verification mechanism\.

### 3\.2Framework Overview

Figure[2](https://arxiv.org/html/2609.22111#S3.F2)illustrates the overall framework\. The system takes a research document\-repository pair\(P,R\)\(P,R\)as input and outputs an audit reportAA\. The framework consists of two stages\. The first stage, Static Claim Verification, extracts structured information from both the research document and the repository, aligns scientific claims with their corresponding implementations, identifies potential static inconsistencies, and constructs executable specifications for subsequent dynamic verification\. The second stage, Dynamic Claim Verification, iteratively executes the repository under the guidance of a Review Agent\. Runtime observations are organized into structured evidence associated with each claim, enabling dynamic verification based on execution outcomes\.

### 3\.3Static Claim Verification

Static Claim Verification performs claim\-guided analysis of the correspondence between scientific claims and repository implementations without executing code\. It constructs structured claim representations, retrieves and aligns relevant implementation evidence, identifies potential inconsistencies, and establishes executable targets for subsequent dynamic verification\. This phase comprises four steps: Claim Contract Construction, Coarse\-to\-Targeted Repository Analysis, Semantic Binding, and Executable Target Construction\.

First, the system extracts scientific claims from the research document and transforms them into structured claim contracts\. Each claim contract records the claim type, involved components, datasets, evaluation metrics, experimental settings, reported results, and their locations in the research document\. These structured representations provide a unified interface for subsequent repository analysis and evidence alignment\.

Next, the repository is analyzed progressively\. The system first examines lightweight artifacts \(e\.g\., README files, configuration files, and entry scripts\) to obtain high\-level execution information, and then focuses on artifacts\-related code regions to inspect implementations, parameter interfaces, metric computation, and execution workflows\.

The extracted information from the research document and repository is then aligned to associate each scientific claim with its corresponding implementation, including related modules, datasets, hyperparameters, and evaluation metrics\. This alignment narrows the search space for dynamic verification while preserving explicit evidence provenance\.

Based on the established correspondence, the Review Agent examines whether sufficient implementation evidence supports the reported claims and identifies potential issues such as missing implementations, parameter inconsistencies, or broken dependencies\. For claims requiring execution\-based validation, the system generates executable targets describing how each claim should be verified, including required resources, expected outputs, and evidence to collect\.

### 3\.4Dynamic Claim Verification

Dynamic Claim Verification validates empirical claims by executing the repository and collecting execution evidence\. Unlike static verification, which examines implementation consistency without execution, this stage evaluates whether experimental claims can be supported through actual execution\. Starting from the executable targets generated in the static stage, the system iteratively plans, executes, and evaluates verification until sufficient evidence is collected or a predefined stopping condition is reached\.

For each executable target derived from a claim, the Review Agent first generates a verification plan that defines the target claim, execution strategy, expected observations, and stopping criteria\. Based on this plan, the Reproduce Agent prepares the execution environment, resolves software dependencies, acquires required datasets when available, and launches the corresponding experiments\. To preserve the integrity of the audit, the agent is allowed to perform only minor compatibility fixes, such as correcting file paths or environment configurations, while modifications that alter the experimental methodology or reported algorithms are prohibited\.

During execution, the evidence system continuously records runtime logs, command histories, intermediate artifacts, and generated outputs\. These execution traces are automatically organized into structured evidence associated with the corresponding scientific claims\. Besides numerical results, the collected evidence also includes contextual information such as execution commands, model configurations, datasets, timestamps, and artifact locations, ensuring that every verification result can be traced back to its original source\.

Verification proceeds iteratively\. After each execution round, the audit system evaluates the accumulated evidence to determine whether sufficient support has been collected for the current claim\. If the verification criteria are satisfied, the framework terminates the execution phase and proceeds to audit report generation\. Otherwise, the Review Agent analyzes the current evidence, identifies missing or conflicting information, and updates the execution plan by refining the search strategy, selecting alternative execution paths, or requesting additional experiments\. Throughout the process, the system monitors execution progress to detect repeated failures, ineffective trials, and prolonged inactivity, preventing unnecessary exploration\. The verification process terminates when all executable claims have been sufficiently verified or the maximum execution budget is exhausted\.

The output of this stage is a collection of execution\-based verification results together with claim\-level evidence, which are subsequently integrated with the static verification results during the final assessment\.

### 3\.5Final Assessment

Finally, static verification results and execution evidence are jointly considered to produce the final audit assessment\. Static verification identifies potential implementation\-level inconsistencies, while dynamic verification provides execution\-based evidence for empirical claims\. For empirically verifiable claims, the system assigns dynamic verification outcomes, includingMatched,Mismatched,Blocked, orNot Reproduced, together with supporting evidence\. These findings are aggregated into a repository\-level audit report, providing an overall assessment of artifact trustworthiness with complete evidence traceability\.

## 4Experiments

We evaluate whether ReAgent can reliably audit claim\-code consistency in research artifacts generated by automated research agents\. Our evaluation addresses three research questions:

- •RQ1\.How accurately can ReAgent audit claim\-code consistency compared with existing approaches?
- •RQ2\.What types of unsupported claims commonly appear in agent\-generated research artifacts?
- •RQ3\.How does ReAgent perform on representative real\-world auditing cases?

### 4\.1Experimental Setup

Our evaluation consists of three components: the evaluation corpus, the claim auditing protocol for constructing ground truth, and comparison methods\. Additional implementation details are provided in the supplementary material\.

#### Evaluation Corpus

We construct our benchmark from AutoSOTA\([12](https://arxiv.org/html/2609.22111#bib.bib25)\), which releases 188 repositories generated by automated research agents\. Each repository contains an agent\-generated report, source code, and experimental results\. We assess the auditability of all 188 repositories and identify 92 repositories with insufficient evidence for reliable verification\. These cases are treated as*unverifiable*rather than inconsistent, leaving 96 auditable repositories for our analysis \(Table[1](https://arxiv.org/html/2609.22111#S4.T1)\)\. Because this filtering may introduce selection bias, our reported inconsistency rates characterize the auditable subset rather than the full collection\.

Table 1:Repositories excluded during corpus construction\.
#### Comparison Methods

We compare ReAgent with two baselines\.

Luo et al\.\([17](https://arxiv.org/html/2609.22111#bib.bib22)\)propose an auditing framework for automated scientific discovery that detects implementation issues such as data leakage, benchmark misuse, and methodological flaws\. Since their framework does not explicitly verify claim\-code consistency, we adapt it as a static verification baseline, referred to as Static\-Audit\.

Codex\([22](https://arxiv.org/html/2609.22111#bib.bib32)\)is a general\-purpose coding agent with strong software engineering capabilities\. It receives the same paper, repository, task description, and execution environment as ReAgent\.

All methods use GPT\-5\.4\([21](https://arxiv.org/html/2609.22111#bib.bib31)\)as the underlying model\. Since different methods adopt different risk taxonomies, we evaluate them under a unified repository\-level binary detection setting: a repository is positive if it contains at least one consistency issue\.

#### Evaluation Protocol

We manually audit all repositories to establish ground truth\.

For static verification, Claude Code is used only for candidate issue discovery\. Three Claude models independently analyze repositories, and three authors verify all identified issues based on implementation details and experimental configurations\. Disagreements are resolved through discussion, and repositories without detected issues are additionally inspected to reduce false negatives\.

For dynamic verification, agents reproduce experiments in isolated environments\. Execution traces are manually examined to verify whether reported results are supported by actual execution\. Failed reproductions are further analyzed, with manual reproduction performed when necessary\.

Table 2:Comparison of auditing performance\.

### 4\.2RQ1: How accurately can ReAgent audit claim\-code consistency?

RQ1 evaluates whether ReAgent can correctly audit agent\-generated repositories\. We compare the predicted auditing results against the manually established ground truth described in Section[4\.1](https://arxiv.org/html/2609.22111#S4.SS1)\.

##### Metrics\.

Static Verification Accuracymeasures the proportion of repositories whose static verification decisions agree with manual annotations\. The static verification task is formulated as a binary classification problem, where the system predicts whether a repository contains static claim\-code inconsistencies\. We additionally report Recall and F1\-score due to the imbalanced distribution of inconsistent repositories\.

Reproduction Success Ratemeasures the proportion of repositories for which the auditing system successfully establishes an executable environment and completes experiment execution\.

Dynamic Verification Accuracymeasures the proportion of repositories for which the binary dynamic verification decisions agree with the manually established ground truth\. Since our objective is to determine whether a scientific claim is supported by executable evidence, only Matched is regarded as a positive outcome\. Mismatched, Blocked, and Not Reproduced are conservatively mapped to the negative class because none of them provides sufficient evidence to support the reported claim\. Nevertheless, these outcomes are preserved as diagnostic labels in the audit report to differentiate distinct causes of verification failure\.

Overall Audit Accuracymeasures the proportion of repositories for which both static verification and dynamic verification decisions agree with manual annotations\. The metric is computed over all evaluated repositories\.

As shown in Table[2](https://arxiv.org/html/2609.22111#S4.T2), ReAgent consistently outperforms existing methods across all evaluation dimensions\. For static verification, ReAgent achieves 78\.13% accuracy, 82\.14% recall, and 68\.66% F1\-score, substantially improving over Codex and Static\-Audit\. In particular, compared with Static\-Audit, which achieves high recall but a low F1\-score, ReAgent provides a better balance between identifying inconsistent repositories and avoiding false alarms\. This demonstrates the effectiveness of our structured claim–repository alignment for static consistency checking\.

For dynamic verification, ReAgent achieves an accuracy of 90\.63% over all evaluated repositories, outperforming Codex by 11\.46 percentage points\. The improvement mainly comes from our execution\-oriented verification pipeline, which converts empirical claims into executable verification targets and collects execution evidence through an integrated auditing and evidence management process\. By grounding verification decisions on experimental evidence rather than textual reasoning alone, ReAgent reduces unreliable judgments caused by incomplete or hallucinated evidence\.

We further analyze dynamic verification failures to understand the limitations of execution\-based auditing\. Among failed cases, most failures were caused by practical execution barriers, including broken evaluation pipelines, missing datasets or assets, and incomplete repositories, which prevented the system from obtaining reliable execution evidence\.

Overall, ReAgent achieves 70\.83% audit accuracy, improving over Codex by 35\.41 percentage points\. These results demonstrate that ReAgent enables a more reliable end\-to\-end consistency audit of paper–code artifacts generated by autonomous research agents\.

### 4\.3RQ2: What consistency issues are prevalent in agent\-generated research artifacts?

We next analyze what types of unsupported scientific claims are commonly produced by autonomous research agents\. Unlike RQ1, which evaluates auditing accuracy, this analysis focuses on the characteristics of the detected inconsistencies\.

Among the 96 repositories, 28 contain at least one static consistency issue\. Since one repository may contain multiple issues, category counts do not sum to the number of affected repositories\. Table[3](https://arxiv.org/html/2609.22111#S4.T3)summarizes the distribution\. The most common issues are Substantive Implementation Discrepancies \(10\) and Data / Label Leakage \(7\), suggesting that autonomous research agents tend to modify implementations for performance improvement while failing to faithfully document these changes\.

Besides these major categories, we also observe several less frequent but practically important inconsistency patterns, summarized as Others in Table[3](https://arxiv.org/html/2609.22111#S4.T3)\. These cases mainly involve documentation and reproducibility risks, such as missing dependencies, undocumented configuration choices, or credential exposure\. Although these issues do not fall into our seven predefined inconsistency categories, they can nevertheless impair the reproducibility of the repository, making it difficult for independent researchers to reliably reproduce or validate the reported findings\.

Dynamic auditing further shows that 33 repositories successfully reproduce their reported metrics, while only 25 repositories satisfy both static consistency and successful reproduction\. The remaining eight repositories reveal an important failure mode: their reported numerical results can be reproduced, while their implementations contain unsupported claim\-code inconsistencies\. Dynamic verification alone incorrectly considers these repositories consistent because the reported metrics are recovered\. In contrast, the static verification stage of ReAgent detects seven of these eight cases, demonstrating that execution\-based verification alone is insufficient for assessing the faithfulness of scientific claims\.

These results demonstrate that neither static nor dynamic auditing alone is sufficient\. Static verification is effective for identifying unsupported implementations and fabricated experimental settings, whereas dynamic verification is necessary to validate empirical claims through actual execution\. Combining both provides substantially broader coverage of claim\-code inconsistencies\.

Table 3:Distribution of static consistency issues identified by manual auditing\.
### 4\.4RQ3: How does ReAgent perform on representative real\-world auditing cases?

#### Static verification identifies implementation inconsistencies

One repository claims that propensity\-feature augmentation with robust ensembling improves IHDP PEHE from 0\.1829 to 0\.1539\. Static verification revealed that the evaluation pipeline implemented an S\-learner augmentation strategy rather than the claimed propensity\-feature augmentation, indicating a substantive implementation discrepancy\. This case shows that execution alone may miss inconsistencies when the repository remains runnable but deviates from the reported methodology\.

#### Dynamic verification avoids false negatives

Another repository reports that Anchorage AUROC improves from 99\.31 to 99\.42 after twelve optimization iterations\. ReAgent reproduced the official evaluation pipeline and obtained the reported value of 99\.42\. In contrast, Codex used a simplified reproduction strategy due to computational constraints and incorrectly concluded that the improvement was unverifiable, highlighting the importance of faithful execution for reliable auditing\.

#### Dynamic verification distinguishes blocked reproduction

Beyond binary claim\-code consistency classification, dynamic verification further distinguishes genuine claim\-code inconsistencies from cases where reproduction is blocked by missing or unusable execution artifacts\.

Another repository reports a PSNR of 32\.24 obtained from its official seven\-seed optimization pipeline\. Static verification confirmed that the documented workflow and implementation were consistent with the reported methodology\. However, during dynamic verification, ReAgent could only execute the evaluation stage, reproducing a PSNR of 30\.66 instead of the reported 32\.24\. Further investigation revealed that the complete generation pipeline failed because the required model checkpoint \(weights/PFCNet\.pth\) was an invalid HTML page rather than a valid PyTorch checkpoint\. ReAgent therefore classified this case asBlockedrather thanMismatched, distinguishing missing executable artifacts from genuine claim\-code inconsistencies\.

## 5Discussion and Limitations

Our study suggests that claim\-code inconsistency represents an emerging challenge for autonomous research agents\. As agents iteratively revise manuscripts, code, and experimental configurations, inconsistencies may arise even without intentional fabrication, potentially reducing the trustworthiness and reproducibility of generated scientific artifacts\. Despite the encouraging results, our framework has several limitations\.

First, ReAgent evaluates whether scientific claims are supported by the accompanying repository, rather than assessing the scientific validity or novelty of the proposed methods\. Therefore, a repository may be internally consistent while still containing flawed assumptions, inappropriate experimental designs, or incorrect scientific conclusions beyond the scope of claim\-code consistency\. Second, dynamic verification relies on executable research artifacts\. Missing datasets, unavailable checkpoints, obsolete dependencies, or excessive computational requirements may prevent complete reproduction even when implementations are correct\. Although ReAgent provides isolated execution environments and automated dependency handling, these issues remain challenging for large\-scale auditing\.

Finally, our evaluation is conducted exclusively on AutoSOTA, currently the only autonomous research platform we identified that publicly releases paper–repository pairs at scale\. Therefore, the benchmark may contain distribution and survivorship biases, and the results may not fully generalize to other agents or human researchers\. Nevertheless, using a single source reduces generator\-specific variation and enables a controlled evaluation of claim\-code consistency\.

## 6Conclusion

This paper presents ReAgent, an automated auditing framework for assessing the consistency between agent\-generated research documents and their associated repositories\. By combining claim\-guided static repository verification with execution\-based dynamic verification, ReAgent systematically collects and organizes implementation and execution evidence to support repository\-level audit decisions\. Experiments demonstrate that ReAgent effectively identifies claim\-code inconsistencies while providing transparent and traceable auditing evidence\.

We believe this work highlights claim\-code consistency as a fundamental challenge for trustworthy autonomous research\. As research agents increasingly generate papers, code, and experiments end\-to\-end, assessing scientific capability alone is insufficient; reliable autonomous research also requires verifying that scientific claims remain faithfully grounded in their implementations and execution evidence\. By introducing a systematic auditing framework for this emerging setting, ReAgent provides a step toward more transparent and reliable evaluation of autonomous scientific workflows\. Future work will extend ReAgent to finer\-grained claim\-level auditing, broader scientific domains, richer evidence sources, and larger\-scale auditing benchmarks\.

## References

- Baker \(2016\)M\. Baker1,500 scientists lift the lid on reproducibility\.Nature Publishing Group UK London\.Cited by:[§1](https://arxiv.org/html/2609.22111#S1.p4.1)\.
- Baumgärtner and Gurevych \(2026\)T\. Baumgärtner and I\. GurevychSciCoQA: quality assurance for scientific paper\-code alignment\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2026, San Diego, California, United States, July 2\-7, 2026,M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),pp\. 38740–38770\.External Links:[Link](https://aclanthology.org/2026.acl-long.1795/)Cited by:[§1](https://arxiv.org/html/2609.22111#S1.p4.1)\.
- Chenet al\.\(2025a\)H\. Chen, M\. Xiong, Y\. Lu, W\. Han, A\. Deng, Y\. He, J\. Wu, Y\. Li, Y\. Liu, and B\. HooiMLR\-bench: evaluating ai agents on open\-ended machine learning research\.External Links:2505\.19955,[Link](https://arxiv.org/abs/2505.19955)Cited by:[§1](https://arxiv.org/html/2609.22111#S1.p2.1)\.
- Chenet al\.\(2025b\)P\. Chen, N\. Yan, Z\. Zhao, Y\. Lin, H\. Chen, Y\. Hu, Q\. Bai, X\. Li, and M\. S\. MortazaviDeep\-reproducer: from paper understanding to code generation\.InNeurIPS 2025 Fourth Workshop on Deep Learning for Code,Cited by:[§1](https://arxiv.org/html/2609.22111#S1.p6.1)\.
- Chenet al\.\(2025c\)Z\. Chen, S\. Chen, Y\. Ning, Q\. Zhang, B\. Wang, B\. Yu, Y\. Li, Z\. Liao, C\. Wei, Z\. Lu, V\. Dey, M\. Xue, F\. N\. Baker, B\. Burns, D\. Adu\-Ampratwum, X\. Huang, X\. Ning, S\. Gao, Y\. Su, and H\. SunScienceAgentBench: toward rigorous assessment of language agents for data\-driven scientific discovery\.InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24\-28, 2025,External Links:[Link](https://openreview.net/forum?id=6z4YKr0GK6)Cited by:[§1](https://arxiv.org/html/2609.22111#S1.p2.1),[§2](https://arxiv.org/html/2609.22111#S2.p3.1)\.
- Gundersen and Kjensmo \(2018\)O\. E\. Gundersen and S\. KjensmoState of the art: reproducibility in artificial intelligence\.InProceedings of the Thirty\-Second AAAI Conference on Artificial Intelligence, \(AAAI\-18\), the 30th innovative Applications of Artificial Intelligence \(IAAI\-18\), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence \(EAAI\-18\), New Orleans, Louisiana, USA, February 2\-7, 2018,S\. A\. McIlraith and K\. Q\. Weinberger \(Eds\.\),pp\. 1644–1651\.External Links:[Link](https://doi.org/10.1609/aaai.v32i1.11503),[Document](https://dx.doi.org/10.1609/AAAI.V32I1.11503)Cited by:[§1](https://arxiv.org/html/2609.22111#S1.p4.1)\.
- Huanget al\.\(2024\)J\. Huang, X\. Chen, S\. Mishra, H\. S\. Zheng, A\. W\. Yu, X\. Song, and D\. ZhouLarge language models cannot self\-correct reasoning yet\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=IkmD3fKBPQ)Cited by:[§1](https://arxiv.org/html/2609.22111#S1.p4.1)\.
- Huanget al\.\(2025\)Y\. Huang, Y\. Sun, Y\. Zhang, R\. Zhang, Y\. Dong, and X\. WeiDeceptionBench: a comprehensive benchmark for ai deception behaviors in real\-world scenarios\.InAdvances in Neural Information Processing Systems,D\. Belgrave, C\. Zhang, H\. Lin, R\. Pascanu, P\. Koniusz, M\. Ghassemi, and N\. Chen \(Eds\.\),Vol\.38,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2025/file/55494d8756b72c2219027edc9de1ee5a-Paper-Datasets_and_Benchmarks_Track.pdf)Cited by:[§2](https://arxiv.org/html/2609.22111#S2.p2.1)\.
- Jiet al\.\(2023\)Z\. Ji, N\. Lee, R\. Frieske, T\. Yu, D\. Su, Y\. Xu, E\. Ishii, Y\. Bang, A\. Madotto, and P\. FungSurvey of hallucination in natural language generation\.ACM Comput\. Surv\.55\(12\),pp\. 248:1–248:38\.External Links:[Link](https://doi.org/10.1145/3571730),[Document](https://dx.doi.org/10.1145/3571730)Cited by:[§1](https://arxiv.org/html/2609.22111#S1.p4.1)\.
- Jia and Li \(2026\)J\. Jia and Q\. LiAutoTool: efficient tool selection for large language model agents\.InFortieth AAAI Conference on Artificial Intelligence, Thirty\-Eighth Conference on Innovative Applications of Artificial Intelligence, Sixteenth Symposium on Educational Advances in Artificial Intelligence, AAAI 2026, Singapore, January 20\-27, 2026,S\. Koenig, C\. Jenkins, and M\. E\. Taylor \(Eds\.\),pp\. 31265–31273\.External Links:[Link](https://doi.org/10.1609/aaai.v40i37.40389),[Document](https://dx.doi.org/10.1609/AAAI.V40I37.40389)Cited by:[§1](https://arxiv.org/html/2609.22111#S1.p1.1)\.
- Jimenezet al\.\(2024\)C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. R\. NarasimhanSWE\-bench: can language models resolve real\-world github issues?\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=VTF8yNQM66)Cited by:[§1](https://arxiv.org/html/2609.22111#S1.p1.1)\.
- Liet al\.\(2026\)Y\. Li, C\. Shao, X\. Liu, R\. Zhao, P\. Liu, H\. Su, Z\. Chen, Q\. Yang, A\. Xu, Y\. Fang, Q\. Zeng, T\. Li, J\. Xu, F\. Xu, Y\. Li, and T\. LiuAutoSOTA: an end\-to\-end automated research system for state\-of\-the\-art ai model discovery\.External Links:2604\.05550,[Link](https://arxiv.org/abs/2604.05550)Cited by:[§2](https://arxiv.org/html/2609.22111#S2.p1.1),[§4\.1](https://arxiv.org/html/2609.22111#S4.SS1.SSSx1.p1.1)\.
- Linet al\.\(2023\)B\. Y\. Lin, C\. Huang, Q\. Liu, W\. Gu, S\. Sommerer, and X\. RenOn grounded planning for embodied tasks with language models\.InThirty\-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty\-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7\-14, 2023,B\. Williams, Y\. Chen, and J\. Neville \(Eds\.\),pp\. 13192–13200\.External Links:[Link](https://doi.org/10.1609/aaai.v37i11.26549),[Document](https://dx.doi.org/10.1609/AAAI.V37I11.26549)Cited by:[§1](https://arxiv.org/html/2609.22111#S1.p1.1)\.
- Liuet al\.\(2026\)J\. Liu, S\. Qiu, M\. Li, B\. Li, H\. Ji, S\. Han, X\. Ye, P\. Xia, Z\. Dong, M\. Chen, C\. Zhang, L\. Zhang, G\. Chen, H\. Tu, X\. Yang, L\. Feng, X\. Zhao, H\. Chen, J\. Zhou, X\. Wang, W\. Zhang, H\. Zhu, Y\. Li, J\. Mei, H\. Fei, J\. Zhang, L\. Li, L\. Zhang, Y\. Zhou, S\. Wang, C\. Xiong, J\. Zou, Z\. Zheng, C\. Xie, M\. Ding, and H\. YaoAutoResearchClaw: self\-reinforcing autonomous research with human\-ai collaboration\.External Links:2605\.20025,[Link](https://arxiv.org/abs/2605.20025)Cited by:[§2](https://arxiv.org/html/2609.22111#S2.p1.1)\.
- Liuet al\.\(2025\)X\. Liu, A\. Pesaranghader, H\. Li, P\. Sukcharoenchaikul, J\. Kim, T\. Sadhu, H\. Jeon, and S\. SannerOpen\-world planning via lifted regression with llm\-inferred affordances for embodied agents\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2025, Vienna, Austria, July 27 \- August 1, 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),pp\. 20881–20897\.External Links:[Link](https://doi.org/10.18653/v1/2025.acl-long.1018),[Document](https://dx.doi.org/10.18653/V1/2025.ACL-LONG.1018)Cited by:[§1](https://arxiv.org/html/2609.22111#S1.p1.1)\.
- Luet al\.\(2026\)C\. Lu, C\. Lu, R\. T\. Lange, Y\. Yamada, S\. Hu, J\. Foerster, D\. Ha, and J\. CluneTowards end\-to\-end automation of ai research\.Nature651\(8107\),pp\. 914\.Cited by:[§1](https://arxiv.org/html/2609.22111#S1.p1.1),[§2](https://arxiv.org/html/2609.22111#S2.p1.1)\.
- Luoet al\.\(2025\)Z\. Luo, A\. Kasirzadeh, and N\. B\. ShahThe more you automate, the less you see: hidden pitfalls of ai scientist systems\.External Links:2509\.08713,[Link](https://arxiv.org/abs/2509.08713)Cited by:[§1](https://arxiv.org/html/2609.22111#S1.p5.1),[§2](https://arxiv.org/html/2609.22111#S2.p2.1),[§4\.1](https://arxiv.org/html/2609.22111#S4.SS1.SSSx2.p2.1)\.
- Majumderet al\.\(2024\)B\. P\. Majumder, H\. Surana, D\. Agarwal, B\. D\. Mishra, A\. Meena, A\. Prakhar, T\. Vora, T\. Khot, A\. Sabharwal, and P\. ClarkDiscoveryBench: towards data\-driven discovery with large language models\.External Links:2407\.01725,[Link](https://arxiv.org/abs/2407.01725)Cited by:[§2](https://arxiv.org/html/2609.22111#S2.p3.1)\.
- Mathewet al\.\(2025\)Y\. Mathew, O\. Matthews, R\. McCarthy, J\. Velja, C\. S\. de Witt, D\. Cope, and N\. SchootsHidden in plain text: emergence & mitigation of steganographic collusion in llms\.InProceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia\-Pacific Chapter of the Association for Computational Linguistics, IJCNLP\-AACL 2025, Mumbai, India, December 20\-24, 2025,K\. Inui, S\. Sakti, H\. Wang, D\. F\. Wong, P\. Bhattacharyya, B\. Banerjee, A\. Ekbal, T\. Chakraborty, and D\. P\. Singh \(Eds\.\),pp\. 585–624\.External Links:[Link](https://doi.org/10.18653/v1/2025.ijcnlp-long.34),[Document](https://dx.doi.org/10.18653/V1/2025.IJCNLP-LONG.34)Cited by:[§2](https://arxiv.org/html/2609.22111#S2.p2.1)\.
- Menget al\.\(2026\)R\. Meng, B\. D\. Mishra, J\. Chen, C\. Li, P\. Goyal, M\. Parmar, Y\. Song, Y\. Song, R\. Sinha, P\. Ranganathan, B\. Gokturk, J\. Yoon, and T\. PfisterScientistOne: towards human\-level autonomous research via chain\-of\-evidence\.External Links:2605\.26340,[Link](https://arxiv.org/abs/2605.26340)Cited by:[§2](https://arxiv.org/html/2609.22111#S2.p1.1)\.
- OpenAI \(2026a\)OpenAIGPT\-5\.4: Technical Report and Model Card\.Note:https://openai\.com/index/introducing\-gpt\-5\-4/Accessed: 2026\-07\-29Cited by:[§4\.1](https://arxiv.org/html/2609.22111#S4.SS1.SSSx2.p4.1)\.
- OpenAI \(2026b\)OpenAIOpenAI Codex: The Next Era of Knowledge Work and Code Generation\.Note:https://openai\.com/index/codex\-for\-knowledge\-work/Accessed: 2026\-07\-29Cited by:[§4\.1](https://arxiv.org/html/2609.22111#S4.SS1.SSSx2.p3.1)\.
- Pineauet al\.\(2021\)J\. Pineau, P\. Vincent\-Lamarre, K\. Sinha, V\. Larivière, A\. Beygelzimer, F\. d’Alché\-Buc, E\. Fox, and H\. LarochelleImproving reproducibility in machine learning research \(a report from the neurips 2019 reproducibility program\)\.Journal of machine learning research22\(164\),pp\. 1–20\.Cited by:[§1](https://arxiv.org/html/2609.22111#S1.p4.1)\.
- Schmidgallet al\.\(2025\)S\. Schmidgall, Y\. Su, Z\. Wang, X\. Sun, J\. Wu, X\. Yu, J\. Liu, M\. Moor, Z\. Liu, and E\. BarsoumAgent laboratory: using LLM agents as research assistants\.InFindings of the Association for Computational Linguistics: EMNLP 2025, Suzhou, China, November 4\-9, 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),pp\. 5977–6043\.External Links:[Link](https://doi.org/10.18653/v1/2025.findings-emnlp.320),[Document](https://dx.doi.org/10.18653/V1/2025.FINDINGS-EMNLP.320)Cited by:[§1](https://arxiv.org/html/2609.22111#S1.p1.1),[§2](https://arxiv.org/html/2609.22111#S2.p1.1)\.
- Siet al\.\(2024\)C\. Si, D\. Yang, and T\. HashimotoCan llms generate novel research ideas? a large\-scale human study with 100\+ nlp researchers\.External Links:2409\.04109,[Link](https://arxiv.org/abs/2409.04109)Cited by:[§1](https://arxiv.org/html/2609.22111#S1.p2.1)\.
- Staraceet al\.\(2025\)G\. Starace, O\. Jaffe, D\. Sherburn, J\. Aung, J\. S\. Chan, L\. Maksin, R\. Dias, E\. Mays, B\. Kinsella, W\. Thompson, J\. Heidecke, A\. Glaese, and T\. PatwardhanPaperBench: evaluating ai’s ability to replicate ai research\.External Links:2504\.01848,[Link](https://arxiv.org/abs/2504.01848)Cited by:[§1](https://arxiv.org/html/2609.22111#S1.p6.1),[§2](https://arxiv.org/html/2609.22111#S2.p3.1)\.
- Suet al\.\(2025\)H\. Su, R\. Chen, S\. Tang, Z\. Yin, X\. Zheng, J\. Li, B\. Qi, Q\. Wu, H\. Li, W\. Ouyang, P\. Torr, B\. Zhou, and N\. DongMany heads are better than one: improved scientific idea generation by A llm\-based multi\-agent system\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2025, Vienna, Austria, July 27 \- August 1, 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),pp\. 28201–28240\.External Links:[Link](https://doi.org/10.18653/v1/2025.acl-long.1368),[Document](https://dx.doi.org/10.18653/V1/2025.ACL-LONG.1368)Cited by:[§1](https://arxiv.org/html/2609.22111#S1.p1.1),[§2](https://arxiv.org/html/2609.22111#S2.p1.1)\.
- Tanget al\.\(2025\)J\. Tang, L\. Xia, Z\. Li, and C\. HuangAI\-researcher: autonomous scientific innovation\.InAdvances in Neural Information Processing Systems,D\. Belgrave, C\. Zhang, H\. Lin, R\. Pascanu, P\. Koniusz, M\. Ghassemi, and N\. Chen \(Eds\.\),Vol\.38,pp\. 9481–9520\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2025/file/0d904d300a105809a2114d727851e759-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2609.22111#S1.p2.1),[§1](https://arxiv.org/html/2609.22111#S1.p6.1),[§2](https://arxiv.org/html/2609.22111#S2.p1.1)\.
- Tanget al\.\(2026\)Q\. Tang, T\. Sun, X\. Hu, X\. Liu, Y\. Chen, Y\. Shao, B\. Li, C\. Lv, C\. Xu, C\. Huang, C\. Li, D\. Xue, H\. Bai, H\. Duan, H\. Guo, H\. He, H\. Chen, H\. Shen, J\. Yuan, J\. Sun, J\. Cheng, J\. Xu, J\. Tong, J\. Chen, J\. Liu, J\. Leng, J\. Yu, K\. Jiang, K\. Xiang, K\. Yao, L\. Feng, L\. Yuan, L\. Gao, M\. Li, Q\. Jia, Q\. Sun, S\. Ding, S\. Gong, S\. Zhong, T\. J\. Zhang, T\. Gu, T\. Liang, W\. Liu, W\. Yang, W\. Fei, X\. Wang, X\. Liu, X\. Ding, Y\. Tang, Y\. Wang, Y\. Jiang, Y\. Yang, Z\. He, Z\. Chen, Z\. Xu, Z\. Li, and Z\. HuangFARS: a fully automated research system deployed at scale\.External Links:2606\.31651,[Link](https://arxiv.org/abs/2606.31651)Cited by:[§1](https://arxiv.org/html/2609.22111#S1.p1.1)\.
- Tianet al\.\(2025\)Y\. Tian, W\. Yan, Q\. Yang, X\. Zhao, Q\. Chen, W\. Wang, Z\. Luo, L\. Ma, and D\. SongCodeHalu: investigating code hallucinations in llms via execution\-based verification\.InThirty\-Ninth AAAI Conference on Artificial Intelligence, Thirty\-Seventh Conference on Innovative Applications of Artificial Intelligence, Fifteenth Symposium on Educational Advances in Artificial Intelligence, AAAI 2025, Philadelphia, PA, USA, February 25 \- March 4, 2025,T\. Walsh, J\. Shah, and Z\. Kolter \(Eds\.\),pp\. 25300–25308\.External Links:[Link](https://doi.org/10.1609/aaai.v39i24.34717),[Document](https://dx.doi.org/10.1609/AAAI.V39I24.34717)Cited by:[§1](https://arxiv.org/html/2609.22111#S1.p4.1)\.
- Tyenet al\.\(2024\)G\. Tyen, H\. Mansoor, V\. Carbune, P\. Chen, and T\. MakLLMs cannot find reasoning errors, but can correct them given the error location\.InFindings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11\-16, 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Findings of ACL, Vol\.ACL 2024,pp\. 13894–13908\.External Links:[Link](https://doi.org/10.18653/v1/2024.findings-acl.826),[Document](https://dx.doi.org/10.18653/V1/2024.FINDINGS-ACL.826)Cited by:[§1](https://arxiv.org/html/2609.22111#S1.p4.1)\.
- Waddenet al\.\(2020\)D\. Wadden, S\. Lin, K\. Lo, L\. L\. Wang, M\. van Zuylen, A\. Cohan, and H\. HajishirziFact or fiction: verifying scientific claims\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 7534–7550\.Cited by:[§1](https://arxiv.org/html/2609.22111#S1.p5.1)\.
- Wanget al\.\(2026\)J\. Wang, W\. Lv, B\. Fu, J\. Fu, J\. Song, L\. Zhang, L\. Xue, L\. Chen, Z\. Xin, K\. Li, and X\. CaoAct as a real researcher: a suite of benchmarks evaluating frontier llms and agentic harnesses in research lifecycle\.External Links:2606\.07462,[Link](https://arxiv.org/abs/2606.07462)Cited by:[§1](https://arxiv.org/html/2609.22111#S1.p2.1)\.
- Wanget al\.\(2024\)L\. Wang, C\. Ma, X\. Feng, Z\. Zhang, H\. Yang, J\. Zhang, Z\. Chen, J\. Tang, X\. Chen, Y\. Lin,et al\.A survey on large language model based autonomous agents\.Frontiers of Computer Science18\(6\),pp\. 186345\.Cited by:[§1](https://arxiv.org/html/2609.22111#S1.p1.1)\.
- Yanget al\.\(2024\)J\. Yang, C\. E\. Jimenez, A\. Wettig, K\. Lieret, S\. Yao, K\. Narasimhan, and O\. PressSWE\-agent: agent\-computer interfaces enable automated software engineering\.InAdvances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 \- 15, 2024,A\. Globersons, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. M\. Tomczak, and C\. Zhang \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper/_files/paper/2024/hash/5a7c947568c1b1328ccc5230172e1e7c-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2609.22111#S1.p1.1)\.
- Zamprognoet al\.\(2025\)G\. Zamprogno, I\. Tiddi, and B\. VerheijAutonomous research assistants for hybrid intelligence: landscape and challenges\.InProceedings of the 2025 AAAI Spring Symposium Series, San Francisco, CA, USA, March 31\-April 2, 2025,R\. P\. A\. Petrick and C\. W\. Geib \(Eds\.\),pp\. 350–358\.External Links:[Link](https://doi.org/10.1609/aaaiss.v5i1.35611),[Document](https://dx.doi.org/10.1609/AAAISS.V5I1.35611)Cited by:[§1](https://arxiv.org/html/2609.22111#S1.p1.1)\.
- Zhenget al\.\(2025\)T\. Zheng, Z\. Deng, H\. T\. Tsang, W\. Wang, J\. Bai, Z\. Wang, and Y\. SongFrom automation to autonomy: A survey on large language models in scientific discovery\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025, Suzhou, China, November 4\-9, 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),pp\. 17733–17750\.External Links:[Link](https://doi.org/10.18653/v1/2025.emnlp-main.895),[Document](https://dx.doi.org/10.18653/V1/2025.EMNLP-MAIN.895)Cited by:[§1](https://arxiv.org/html/2609.22111#S1.p1.1)\.