Dude: A Dual-Detection Multi-Agent System for Paper-Code Discrepancy Detection

# A Dual-Detection Multi-Agent System forPaper-Code Discrepancy Detection
Source: [https://arxiv.org/html/2609.03416](https://arxiv.org/html/2609.03416)
Running ZhaoAffiliation:The University of Hong KongEmail:[chngai@eee\.hku\.hk](mailto:[email protected])Wenhao YuanAffiliation:The University of Hong KongJinfeng XuAffiliation:The University of Hong KongZhanfeng XuAffiliation:The University of Hong KongXiaoxi ZhangAffiliation:Sun Yat\-sen UniversityEdith Cheuk\-Han Ngai††thanks:Corresponding authorAffiliation:The University of Hong Kong

###### Abstract

LLM\-empowered paper\-code discrepancy detection has received growing concern since the scaling of research submissions exceeds the manual review capability\. However, the limited context capacity and one\-sided discrepancy detection of existing single\-agent LLM paradigms lead to an inferior recall performance in detecting discrepancies\. In this paper, we proposeDude111https://github\.com/VinnyLiu0817/Dude, the firstDual\-Detection Multi\-Agent System for paper\-code discrepancy detection\. We discover that the granularity asymmetry of the paper\-language and code\-language introduces over\-interpretation and over\-reporting challenges in a multi\-agent system design for discrepancy detection, resulting in increasing false positives\. To address this, we propose a granularity\-aligned negotiation and a two\-stage salience\-filtering mechanism in Dude, which effectively prevents agents from falsely reporting discrepancies\. Experimental results in real\-world paper\-code discrepancy datasets showcase Dude’s significant recall and precision improvement by up to22\.8%, increasing F1 score by up to18\.7%compared to baseline methods\.

## 1Introduction

Paper\-code discrepancy detection aims to identify inconsistencies of research claims described in the paper manuscript and code repository\. This task is crucial as such inconsistencies compromise the credibility and reproducibility of research findings\. Yet, the rapid growth of paper submissions has gone beyond reviewers’ capacity to conduct accurate assessments of paper\-code consistency within tight timelines\. Therefore, leveraging the power of LLMs to conduct automated discrepancy detection has attracted increasing attention[You et al\. \(2026\)](https://arxiv.org/html/2609.03416#bib.bib4);[Liu et al\. \(2026\)](https://arxiv.org/html/2609.03416#bib.bib22);[Xu et al\. \(2026\)](https://arxiv.org/html/2609.03416#bib.bib5)\.

Limitations\.However, a recent study[Baumgärtner and Gurevych \(2026\)](https://arxiv.org/html/2609.03416#bib.bib1)has reported the inferior performance of the single\-agent paradigm in detecting paper\-code discrepancies, with the best\-performing models attaining only 46% recall on a real\-world discrepancy dataset SciCoQA\. This sub\-optimal performance of single\-agent LLMs stem from: 1\) The limited context window makes it difficult to jointly reason over lengthy papers and large codebases\. 2\) LLM tends to adopt paper\-oriented detection by only generating a research claim from the paper, then searching the codebase for matching code\. While this is effective to detect paper\-code conflict and code\-omission discrepancies which are explicitly stated in the paper, it is inherently limited to spot paper\-omission discrepancies only observable from the code, as we illustrated in Fig\.[1](https://arxiv.org/html/2609.03416#S1.F1)\.

To address these limitations, we designDude, aDual\-Detection Multi\-Agent System for paper\-code discrepancy detection\. Dude decomposes the discrepancy detection task into fine\-grained subtasks and assigns specialized paper agents and code agents to conduct either paper\-side understanding or code\-side analysis\. This alleviates the context burden of individual agents, and enables them to collectively perform both paper\-oriented and code\-oriented detection, i\.e\., dual\-detection\.

![Refer to caption](https://arxiv.org/html/2609.03416v1/Definition3.png)Figure 1:Three types of paper\-code discrepancy of a research claim \(Left\) and the comparison of one\-sided detection and our proposed dual\-detection \(Right\)\.However, designing an effective dual\-detection multi\-agent system is non\-trivial\. The core challenge arises fromthe granularity asymmetry between the paper\-language description and code\-language implementation of a research claim, introducing unique challenges in paper\-oriented and code\-oriented discrepancy detection, respectively\.

Over\-interpretation in paper\-oriented detection\.In paper\-oriented detection, research claims extracted from papers are often condensed and high\-level, leaving multiple valid code implementation choices\. This ambiguity may cause the code agent to over\-interpret the claim and mistakenly treat its inferred implementation as paper\-grounded claims before examining the code\. Consequently, this ungrounded over\-interpretation leads to falsely reporting a discrepancy when the code implements one valid interpretation of a high\-level research claim, but not the particular implementation that the code agent inferred without textual support from papers\.

Over\-reporting in code\-oriented detection\.In code\-oriented detection, research claims extracted from the code are concrete and fine\-grained\. However, not every code implementation constitutes a research claim that requires documentation in the paper\. Codebases also include environment setup and other auxiliary implementations that are necessary only for execution but peripheral to scientific contributions\. A naive code\-oriented detection over\-reports these non\-salient implementations as research claims, resulting in false\-positive discrepancies\. Tab\.[1](https://arxiv.org/html/2609.03416#S1.T1)shows that simply decomposing dual\-detection to multiple agents increases recall, but incurs significant precision drop, showing the negative effect of over\-interpretation and over\-reporting\.

MethodRecallPrecisionF1Simple\-LM58\.70%91\.53%71\.52%Vanilla\-MA75\.00%74\.19%74\.59%Improvment\+16\.30 %\-17\.34 %\+3\.07 %Table 1:Precision, recall, and F1 score of a single\-agent method \(Simple\-LM\) and a naive multi\-agent method \(Vanilla\-MA\), both methods adopt GPT\-5\.4\.To mitigate the over\-interpretation challenge, we introduce a granularity\-aligned negotiation mechanism in Dude, which enables multi\-round interactions between the paper agent and code agent\. During this process, the paper agent progressively refines and provides increasingly concrete and grounded paper description of the research claim\. This mitigates over\-interpretation by the code agent and yields more reliable discrepancy judgments\. To mitigate over\-reporting, Dude introduces a two\-stage filtering module that combines anchor\-guided filtering with evidence\-based filtering\. This module suppresses research claims over\-reported by code agents, thereby preventing paper agents from falsely reporting discrepancies in trivial code implementations\. Our key contributions are as follows:

- •We reveal the limitation of single\-agent LLM to detect paper\-code discrepancies stems from its one\-sided discrepancy detection\.
- •We design Dude, a dual\-detection multi\-agent system with granularity\-aligned negotiation and filtering modules, bridging the granularity asymmetry between papers and codebases\.
- •Experiments demonstrate significant recall and precision improvement of Dude in detecting real\-world paper\-code discrepancies\.

## 2Related Work

Error detection in research papers and codes\.Leveraging the power of LLM to detect errors in research articles or open\-sourced codes has attracted growing attention in current LLM era\. For paper error detection, prior works include identifying logical issues[Liu and Shah \(2023\)](https://arxiv.org/html/2609.03416#bib.bib6), incorrect data calculations[Bianchi et al\. \(2025\)](https://arxiv.org/html/2609.03416#bib.bib7), invalid arguments[Xi et al\. \(2025\)](https://arxiv.org/html/2609.03416#bib.bib8), flawed proofs or experiment designs[Zhang and Abernethy \(2025\)](https://arxiv.org/html/2609.03416#bib.bib9)\. For code error detection, existing work covers code\-comment inconsistency detection[Ratol and Robillard \(2017\)](https://arxiv.org/html/2609.03416#bib.bib10);[Panthaplackel et al\. \(2021\)](https://arxiv.org/html/2609.03416#bib.bib11), bug rectification[Rong et al\. \(2025\)](https://arxiv.org/html/2609.03416#bib.bib12), reproducibility[Weng et al\. \(2026\)](https://arxiv.org/html/2609.03416#bib.bib26);[Bogin et al\. \(2024\)](https://arxiv.org/html/2609.03416#bib.bib14), and quality evaluation[Tong and Zhang \(2024\)](https://arxiv.org/html/2609.03416#bib.bib13)\. However, these works focus either solely on paper understanding or code analysis, neglecting the potential discrepancies between a research paper and its associated codebases\.

A recent study[Baumgärtner and Gurevych \(2026\)](https://arxiv.org/html/2609.03416#bib.bib1)presents a comprehensive analysis along with SciCoQA, a real\-world paper\-code discrepancy dataset, revealing the inferior performance of the latest LLMs in detecting such discrepancies\. Another concurrent work BioCon[Xu et al\. \(2026\)](https://arxiv.org/html/2609.03416#bib.bib5)also studies the paper\-code inconsistencies between publications and their associated software design in the bioinformatics field\. However, these two works only consider adopting single\-agent paradigms to conduct discrepancy detection\. The limited context capability and one\-sided detection of a single LLM hinder its ability to comprehensively identify all potential inconsistencies\.

Multi\-agent LLM Collaboration\.To tackle increasingly complex and long\-horizon tasks, LLM\-based agent systems have developed from single\-agent reasoning to multi\-agent collaboration[Lee et al\. \(2025\)](https://arxiv.org/html/2609.03416#bib.bib18);[Zhang et al\. \(2025b\)](https://arxiv.org/html/2609.03416#bib.bib15);[Yuan et al\. \(2025\)](https://arxiv.org/html/2609.03416#bib.bib16);[Chen et al\. \(2026\)](https://arxiv.org/html/2609.03416#bib.bib17)\. In the multi\-agent paradigm, a complex task is often decomposed into a set of subtasks[Hong et al\. \(2024\)](https://arxiv.org/html/2609.03416#bib.bib25);[Liao et al\. \(2025\)](https://arxiv.org/html/2609.03416#bib.bib21), which are then assigned to multiple specialized agents to alleviate the context burden of individual agents\. Moreover, building an effective multi\-agent system typically requires a carefully designed framework to ensure efficient and desirable collaboration\. Existing multi\-agent collaboration frameworks are often tailored to the specific task properties like MedAgents[Tang et al\. \(2024\)](https://arxiv.org/html/2609.03416#bib.bib20)and ChatDev[Qian et al\. \(2024\)](https://arxiv.org/html/2609.03416#bib.bib23), or motivated by human collaboration workflows like Multi\-Agent Debate[Zhang et al\. \(2025a\)](https://arxiv.org/html/2609.03416#bib.bib19);[Liang et al\. \(2024\)](https://arxiv.org/html/2609.03416#bib.bib2)\.

## 3Methodology

![Refer to caption](https://arxiv.org/html/2609.03416v1/Overall-workflow.png)Figure 2:A complete overview of our proposed dual\-detection multi\-agent systemDude, including three consecutive stages, paper\-oriented detection \(Top\), code\-oriented detection \(Lower left\), and report generation \(Lower right\)\.### 3\.1Overview

We formulate the paper\-code discrepancy detection task using multi\-agent LLM systemsℳ\\mathcal\{M\}as:

𝒴=ℳ⁡\(𝒫,𝒞\),\\mathcal\{Y\}=\\mathcal\{M\}\(\\mathcal\{P\},\\mathcal\{C\}\),\(1\)where𝒫\\mathcal\{P\}and𝒞\\mathcal\{C\}denote the paper and its associated code, and𝒴\\mathcal\{Y\}represents the final discrepancy report that includes all verified research claimsRiR\_\{i\}with paper\-code discrepancies\. For each verified research claimRiR\_\{i\}, we have:

Ri=\{Di,Ii,Mi,Ei,Oi\},R\_\{i\}=\\\{D\_\{i\},I\_\{i\},M\_\{i\},E\_\{i\},O\_\{i\}\\\},\(2\)whereDiD\_\{i\}denotes its paper description,IiI\_\{i\}denotes its code implementation,MiM\_\{i\}denotes the matching results between the paper and code,EiE\_\{i\}denotes the explanation of the matching result, andOi∈\{p,c\}O\_\{i\}\\in\\\{p,c\\\}indicates whether this research claim is initiated from the paper\-oriented detection or code\-oriented detection\. We defineℛp=\{Ri∣Oi=p\}\\mathcal\{R\}\_\{p\}=\\\{R\_\{i\}\\mid O\_\{i\}=p\\\}andℛc=\{Ri∣Oi=c\}\\mathcal\{R\}\_\{c\}=\\\{R\_\{i\}\\mid O\_\{i\}=c\\\}as the set of verified research claims generated in paper\-oriented and code\-oriented detection, referred to as paper\-oriented claims and code\-oriented claims\.222We use ‘paper claim’ and ‘paper\-oriented claim’ interchangeably, same for ‘code claim’ and ‘code\-oriented claim’\.

Our proposed Dude includes four types of specialized agents \(claimer,verifier,negotiator,orchestrator\) to decompose the discrepancy detection process into three stages: paper\-oriented detection \(§[3\.2](https://arxiv.org/html/2609.03416#S3.SS2)\), code\-oriented detection\(§[3\.3](https://arxiv.org/html/2609.03416#S3.SS3)\), and report generation\(§[3\.4](https://arxiv.org/html/2609.03416#S3.SS4)\), as shown in Fig\.[2](https://arxiv.org/html/2609.03416#S3.F2)\. In paper\-oriented detection, a paper claimer initializes research claimsRiR\_\{i\}by extracting paper descriptionsDiD\_\{i\}from the paper, then a code verifier searches for corresponding code implementationIiI\_\{i\}to verify their consistency, presenting the matching resultMiM\_\{i\}and explanationEiE\_\{i\}to form the verified paper\-oriented claimsℛp\\mathcal\{R\}\_\{p\}\. In code\-oriented detection, a code claimer initializes research claims by extracting notable implementations from the codebase, then a paper verifier retrieves corresponding paper description to verify their consistency, returning the matching resultMiM\_\{i\}and explanationEiE\_\{i\}to form the code\-oriented claimsℛc\\mathcal\{R\}\_\{c\}\. During this process, two negotiators refine claim description and explanation to mitigate over\-interpretation, and an orchestrator coordinates workflow and consolidatesℛp\\mathcal\{R\}\_\{p\}andℛc\\mathcal\{R\}\_\{c\}into a final report𝒴\\mathcal\{Y\}\.

### 3\.2Paper\-oriented Discrepancy Detection

We formulate the paper\-oriented discrepancy detection processfp​a​p​e​r\(\.\)f\_\{paper\}\(\.\)as:

ℛp=fp​a​p​e​r​\(𝒫,𝒞,𝒜p\),\\mathcal\{R\}\_\{p\}=f\_\{paper\}\(\\mathcal\{P\},\\mathcal\{C\},\\mathcal\{A\}\_\{p\}\),\(3\)where𝒜p=\{𝒜p​c,𝒜c​v,𝒜p​n,𝒜c​n,𝒜O\}\\mathcal\{A\}\_\{p\}=\\\{\\mathcal\{A\}\_\{pc\},\\mathcal\{A\}\_\{cv\},\\mathcal\{A\}\_\{pn\},\\mathcal\{A\}\_\{cn\},\\mathcal\{A\}\_\{O\}\\\}denotes specialized agents in the paper\-oriented detection\.

#### 3\.2\.1Claim initialization

To begin the paper\-oriented detection, apaper claimer agent𝒜p​c\\mathcal\{A\}\_\{pc\}initializes a research claimRiR\_\{i\}by extracting the paper descriptionDiD\_\{i\}of the research claim from the paper manuscript𝒫\\mathcal\{P\}:

Di=𝒜p​c​\(𝒫\),D\_\{i\}=\\mathcal\{A\}\_\{pc\}\(\\mathcal\{P\}\),\(4\)If the research claim mentions the adoption or comparison of external research, like ‘Our work follows the dataset split setup in \[1\]…’\. The paper claimer would use web search tools𝒯w=\\mathcal\{T\}\_\{w\}=\{WebSearch,WebFetch\} to refine the descriptionDiD\_\{i\}with fetched external knowledge𝒦ie​x​t=𝒯w​\(𝒫,Di\)\\mathcal\{K\}^\{ext\}\_\{i\}=\\mathcal\{T\}\_\{w\}\(\\mathcal\{P\},D\_\{i\}\)\. The paper descriptionDiD\_\{i\}of the research claim is further refined as:

Di=𝒜p​c​\(𝒫,𝒦ie​x​t\)\.D\_\{i\}=\\mathcal\{A\}\_\{pc\}\(\\mathcal\{P\},\\mathcal\{K\}^\{ext\}\_\{i\}\)\.\(5\)

#### 3\.2\.2Paper\-code consistency verification

Thecode verifier agent𝒜c​v\\mathcal\{A\}\_\{cv\}then inspects the codebase𝒞\\mathcal\{C\}and maps the paper descriptionDiD\_\{i\}to its corresponding code implementationIiI\_\{i\}using built\-in code\-search tools𝒯c=\\mathcal\{T\}\_\{c\}=\{grep,glob,read\}:

Ii=𝒯c​\(Di,𝒞\),I\_\{i\}=\\mathcal\{T\}\_\{c\}\(D\_\{i\},\\mathcal\{C\}\),\(6\)whereIiI\_\{i\}includes both the content and location of the corresponding source code\. If the code implementation is not found,Ii=∅I\_\{i\}=\\emptyset\. The code verifier𝒜c​v\\mathcal\{A\}\_\{cv\}then verifies the consistency between the paper descriptionDiD\_\{i\}and code implementationIiI\_\{i\}, providing the matching resultMiM\_\{i\}and explanationEiE\_\{i\}\.

Mi,Ei=𝒜c​v​\(Di,Ii\),M\_\{i\},E\_\{i\}=\\mathcal\{A\}\_\{cv\}\(D\_\{i\},I\_\{i\}\),\(7\)whereMi∈M\_\{i\}\\in\{matched, partial, mismatch, unfound\}, andEiE\_\{i\}interprets the resultMiM\_\{i\}based on the descriptionDiD\_\{i\}and implementationIiI\_\{i\}\.

#### 3\.2\.3Negotiation process\.

Due to the granularity asymmetry between the natural language of paper descriptionDiD\_\{i\}and code languages of implementationIiI\_\{i\}, the code verifier𝒜c​v\\mathcal\{A\}\_\{cv\}in Eq\. \([7](https://arxiv.org/html/2609.03416#S3.E7)\) may over\-interpret the high\-level paper descriptionDiD\_\{i\}, yielding a false matching resultMiM\_\{i\}and an over\-interpreted explanationEiE\_\{i\}\. To this end, we design a granularity\-aligned negotiation process that iteratively refines the paper description, revises explanation, and corrects matching result for allNpN\_\{p\}paper\-oriented claims\{Di,Mi,Ei\}i=1Np\\\{D\_\{i\},M\_\{i\},E\_\{i\}\\\}\_\{i=1\}^\{N\_\{p\}\}\. The negotiation processfn​e​g\(\.\)f\_\{neg\}\(\.\)at thejt​hj^\{th\}round can be formulated as:

Dij\+1,Mij\+1,Eij\+1=fn​e​g​\(Dij,Mij,Eij,𝒜\),∀iD\_\{i\}^\{j\+1\},M\_\{i\}^\{j\+1\},E\_\{i\}^\{j\+1\}=f\_\{neg\}\(D\_\{i\}^\{j\},M\_\{i\}^\{j\},E\_\{i\}^\{j\},\\mathcal\{A\}\),\\forall i\(8\)where𝒜=\{𝒜p​n,𝒜c​n,𝒜O\}\\mathcal\{A\}=\\\{\\mathcal\{A\}\_\{pn\},\\mathcal\{A\}\_\{cn\},\\mathcal\{A\}\_\{O\}\\\}represents the paper negotiator agent𝒜p​n\\mathcal\{A\}\_\{pn\}, code negotiator agent𝒜c​n\\mathcal\{A\}\_\{cn\}, and orchestrator agent𝒜O\\mathcal\{A\}\_\{O\}\.

As shown in the negotiation process in Fig\.[2](https://arxiv.org/html/2609.03416#S3.F2), a checklistℒ=\{Li\}i=1Np\\mathcal\{L\}=\\\{L\_\{i\}\\\}\_\{i=1\}^\{N\_\{p\}\}records the validity of each claim explanationEiE\_\{i\}, whereLiL\_\{i\}is defined as:

Li=\{valid,Ei​has no over\-interpretation,invalid,Ei​has over\-interpretation\.L\_\{i\}=\\begin\{cases\}\\texttt\{valid\},&E\_\{i\}\\text\{ has no over\-interpretation\},\\\\ \\texttt\{invalid\},&E\_\{i\}\\text\{ has over\-interpretation\}\.\\end\{cases\}To initiate the negotiation process, thepaper negotiator𝒜p​n\\mathcal\{A\}\_\{pn\}first inspects explanationEiE\_\{i\}of each claim to identify over\-interpretation\. If the explanationEiE\_\{i\}is accurate, the paper negotiator marksLi=validL\_\{i\}=\\texttt\{valid\}on the checklist\. For claims with over\-interpreted explanation, the paper negotiator refines the descriptionDiD\_\{i\}based on the paper𝒫\\mathcal\{P\}to resolve the extrapolation that causes over\-interpretation\. Formally, the refinement of paper descriptionDijD\_\{i\}^\{j\}at thejt​hj^\{th\}negotiation round is formulated as:

Dij\+1=𝒜p​n​\(Dij,Eij,𝒫\)D\_\{i\}^\{j\+1\}=\\mathcal\{A\}\_\{pn\}\(D\_\{i\}^\{j\},E\_\{i\}^\{j\},\\mathcal\{P\}\)\(9\)Next, thecode negotiator agent𝒜c​n\\mathcal\{A\}\_\{cn\}locates the code implementationIij\+1I\_\{i\}^\{j\+1\}corresponding to the refined paper descriptionDij\+1D\_\{i\}^\{j\+1\}, re\-verifies their consistency, and updates the matching resultMij\+1M\_\{i\}^\{j\+1\}along with a refined explanationEij\+1E\_\{i\}^\{j\+1\}for the examination the next negotiation round\. The revision of matching result and explanation at thejt​hj^\{th\}negotiation round is formulated as:

Mij\+1,Eij\+1=𝒜c​n​\(Dij\+1,Iij\+1\)M\_\{i\}^\{j\+1\},E\_\{i\}^\{j\+1\}=\\mathcal\{A\}\_\{cn\}\(D\_\{i\}^\{j\+1\},I\_\{i\}^\{j\+1\}\)\(10\)Theorchestrator agent𝒜O\\mathcal\{A\}\_\{O\}coordinates the overall negotiation process, and terminates it whenLi=valid,∀iL\_\{i\}=\\texttt\{valid\},\\forall i, or when the process reaches a pre\-defined maximum number of roundsrr\. At this point, our Dude has produced a set of verified research claimsℛp=\{Ri∣Oi=p\}\\mathcal\{R\}\_\{p\}=\\\{R\_\{i\}\\mid O\_\{i\}=p\\\}from the paper\-oriented detection process\.

### 3\.3Code\-oriented Discrepancy Detection

We formulate the code\-oriented discrepancy detection processfc​o​d​e\(\.\)f\_\{code\}\(\.\)as:

ℛc=fc​o​d​e​\(𝒞,𝒫,𝒜c\),\\mathcal\{R\}\_\{c\}=f\_\{code\}\(\\mathcal\{C\},\\mathcal\{P\},\\mathcal\{A\}\_\{c\}\),\(11\)whereℛc\\mathcal\{R\}\_\{c\}denotes the generated code\-oriented research claims, and𝒜c=\{𝒜c​c,𝒜p​v\}\\mathcal\{A\}\_\{c\}=\\\{\\mathcal\{A\}\_\{cc\},\\mathcal\{A\}\_\{pv\}\\\}denotes all specialized agents in the code\-oriented detection\.

Claim generation and anchor\-guided filtering\.In code\-oriented discrepancy detection, acode claimer agent𝒜c​c\\mathcal\{A\}\_\{cc\}initializes a research claimRiR\_\{i\}by extracting the notable code implementationsIiI\_\{i\}of the research claim from the codebase𝒞\\mathcal\{C\}\. We formulate the claim generation process as:

Ii=𝒜c​c​\(ℛp,𝒞,𝒦a\)I\_\{i\}=\\mathcal\{A\}\_\{cc\}\(\\mathcal\{R\}\_\{p\},\\mathcal\{C\},\\mathcal\{K\}\_\{a\}\)\(12\)Before examining the repository𝒞\\mathcal\{C\}, the code claimer first inspects the paper\-oriented claimsℛ𝒫\\mathcal\{R\}\_\{\\mathcal\{P\}\}to avoid extracting repetitive research claims and enable more efficient search, sinceℛp\\mathcal\{R\}\_\{p\}contains existing paper\-oriented claims and their code implementation\. To preserve only notable implementationsIiI\_\{i\}and discard those trivial ones, we adopt an anchored\-guided filtering module by adopting category\-wise domain knowledge𝒦a\\mathcal\{K\}\_\{a\}as anchors to help code claimer better understand the criterion of the implementation significance from different categories\. In this way, the code claimer can more accurately identify the salient code implementationIiI\_\{i\}and thus alleviate the over\-reporting problem\.

Relevance\-based filtering and consistency verification\.However, the importance of code implementations is not an intrinsic property of the code alone\. It is often context\-dependent and paper\-sensitive\. The same implementation may be highly important in one type of article but peripheral in another\. For instance, an INT8 quantization module is essential to report in a model compression paper, since it directly affects paper’s contribution about memory usage and efficiency\. However, the same quantization code is trivial in a paper about new attention architectures for classification, whose contribution has little connection with quantized models\. Therefore, thepaper verifier agent𝒜p​v\\mathcal\{A\}\_\{pv\}conducts a relevance\-based salience filteringff​i​l​t​e​r\(\.\)f\_\{filter\}\(\.\)to remove trivial research claims and extract paper descriptionDiD\_\{i\}of research claim whose implementationIiI\_\{i\}has high relevance to the paper𝒫\\mathcal\{P\}\.

Di=ff​i​l​t​e​r​\(𝒜p​v,𝒫,Ii\)\.D\_\{i\}=f\_\{filter\}\(\\mathcal\{A\}\_\{pv\},\\mathcal\{P\},I\_\{i\}\)\.\(13\)This prevents the paper verifier from reporting trivial inconsistencies in the verification process, where the paper verifier examines the consistency between the code implementationIiI\_\{i\}and paper descriptionDiD\_\{i\}, and returns the matching resultMiM\_\{i\}and explanationEiE\_\{i\}\.

Mi,Ei=𝒜p​v​\(Di,Ii\)\.M\_\{i\},E\_\{i\}=\\mathcal\{A\}\_\{pv\}\(D\_\{i\},I\_\{i\}\)\.\(14\)At this point, our Dude has produced a set of verified research claimsℛc=\{Ri∣Oi=c\}\\mathcal\{R\}\_\{c\}=\\\{R\_\{i\}\\mid O\_\{i\}=c\\\}from the code\-oriented detection process\.

### 3\.4Discrepancy Report Generation

To yield the final discrepancy report𝒴\\mathcal\{Y\}, the orchestrator agent𝒜O\\mathcal\{A\}\_\{O\}merges the paper\-oriented claimsℛp\\mathcal\{R\}\_\{p\}and code\-oriented claimsℛc\\mathcal\{R\}\_\{c\}and determines the discrepancy typetit\_\{i\}of research claimRiR\_\{i\}based on its matching resultsMiM\_\{i\}and originationOiO\_\{i\}\.

ti=\{Conflict,Mi∈\{partial,mismatch\},Paper\-omission,Mi=unfound,Oi=c,Code\-omission,Mi=unfound,Oi=p\.t\_\{i\}=\\begin\{cases\}\\texttt\{Conflict\},&M\_\{i\}\\in\\\{\\texttt\{partial\},\\texttt\{mismatch\}\\\},\\\\ \\texttt\{Paper\-omission\},&M\_\{i\}=\\texttt\{unfound\},O\_\{i\}=c,\\\\ \\texttt\{Code\-omission\},&M\_\{i\}=\\texttt\{unfound\},O\_\{i\}=p\.\\end\{cases\}Allpartialandmismatchclaims are treated as paper\-code conflicts\. Forunfoundclaims, if the claim is paper\-oriented \(Oi=pO\_\{i\}=p\), it indicates a code omission\. Otherwise, if the claim is code\-oriented \(Oi=cO\_\{i\}=c\), it indicates a paper omission, and allmatchedclaims are excluded\.

## 4Experiment

### 4\.1Experimental Setup

Dataset and models\.We evaluate Dude using four closed\-source and open\-source LLMs \(GPT\-5\.4, DeepSeekV4\-Pro, Kimi, Claude\-4\.6\) on the only available paper\-code discrepancy datasetSciCoQA[Baumgärtner and Gurevych \(2026\)](https://arxiv.org/html/2609.03416#bib.bib1)\. We also evaluate Dude by selecting 20 research papers with public available codebases published in latest top\-tier conferences \(ICML 2025, ICLR 2026\)\.

Baselines\.We compare Dude against four baselines, covering both single\-agent and multi\-agent paradigms\. For single\-agent methods, we use the default method employed in SciCoQA,Single\-LM, where a single LLM agent directly generates discrepancy report after inspecting both the paper and its corresponding code repository\. We adoptPrompt\-LMby prompting a single agent to perform paper\-oriented and code\-oriented detection\. For multi\-agent methods, we considerVanilla\-MAby assigning multiple agents to conduct paper\-oriented and code\-oriented detection\. Finally, we includeMulti\-Agent Debate \(MAD\)[Liang et al\. \(2024\)](https://arxiv.org/html/2609.03416#bib.bib2)method which aggregates multiple agents’ discussion to verify paper\-code discrepancies\.

Evaluation Metrics\.We follow the evaluation protocol in the SciCoQA dataset and adopt LLM\-as\-a\-Judge[Zheng et al\. \(2023\)](https://arxiv.org/html/2609.03416#bib.bib24)to evaluate whether the reported discrepancies match those in SciCoQA\. We adopt Gemini\-3\.1 Pro as the judge and compute the recall, precision, and F1 scores\. Notably, the reported discrepancies may include valid but unannotated discrepancies in SciCoQA\. We do not count such discrepancies as true positives or false positives\. We only consider discrepancies annotated in the SciCoQA dataset when computing the metrics\. Nevertheless, for invalid and trivial discrepancies, we label them as false positives after verification using domain expertise and Gemini\-3\.1\. For token consumption, we accumulate the total input and output tokens reported in the session logs or official API platforms\.

Implementation Details\.We implement Dude on top of Codex and OpenCode[Anomalyco \(2026\)](https://arxiv.org/html/2609.03416#bib.bib3), enabling Dude to be empowered by GPT, Claude, Kimi, and Deepseek\. We set the default number of negotiation roundsr=2r=2, and all agents employ the same LLM model unless specified\. The detailed code and configurations are available at anonymous\.4open\.science/r/Dude

### 4\.2Main Results

MethodRecall↑\\uparrowΔ\\DeltaRPrecision↑\\uparrowΔ\\DeltaPF1↑\\uparrowΔ\\DeltaF1Token↓\\downarrowΔ\\DeltaTokenGPT\-5\.4\-xhighSingle\-LM58\.70%–91\.53%–71\.52%–0\.71M–Prompt\-LM64\.13%\+5\.43%84\.29%\-7\.24%72\.84%\+1\.32%0\.75M\+5\.63%Vanilla\-MA75\.00%\+16\.30%74\.19%\-17\.33%74\.59%\+3\.07%1\.04M\+46\.48%MAD59\.78%\+1\.09%93\.22%\+1\.69%72\.85%\+1\.32%1\.28M\+80\.28%Dude \(ours\)80\.43%\+21\.74%93\.67%\+2\.15%86\.55%\+15\.03%0\.76M\+7\.04%DeepseekV4\-Pro\-MaxSingle\-LM52\.17%–84\.21%–64\.43%–0\.49M–Prompt\-LM64\.13%\+11\.96%77\.63%\-6\.58%70\.24%\+5\.81%0\.59M\+20\.41%Vanilla\-MA73\.91%\+21\.74%76\.40%\-7\.81%75\.14%\+10\.71%0\.91M\+85\.71%MAD67\.39%\+15\.22%80\.52%\-3\.69%73\.37%\+8\.94%0\.95M\+93\.87%Dude \(ours\)75\.00%\+22\.83%93\.24%\+9\.03%83\.13%\+18\.70%0\.60M\+22\.49%Table 2:The recall, precision, F1, and token usage results ofDudeand baseline methods on the SciCoQA dataset using GPT\-5\.4 and DeepseekV4 models\.Δ\\Deltaare calculated relative to Single\-LM within each model group\. Bold numbers and underlined numbers denote the highest and second\-highest value in each model group\.Figure 3:The recall performance of Single\-LM andDudeacross different discrepancy types \(paper\-code conflict, code\-omission, paper\-omission\) in SciCoQA dataset using GPT5\.4, Claude 4\.6, Kimi, and DeepseekV4 models\.Table 3:F1 scores under different paper\-side and code\-side agent model combinations\.Code\-side agentsPaper\-side agentsGPT5\.4DeepseekV4GPT5\.40\.93670\.9114DeepseekV40\.93330\.9324MethodRecall↑\\uparrowΔ\\DeltaRPrecision↑\\uparrowΔ\\DeltaPF1↑\\uparrowΔ\\DeltaF1GPT\-5\.4\-xhighDude w/o Negotiation75\.00%\-5\.43%87\.34%\-6\.33%80\.70%\-5\.85%Dude w/o Saliency\-Filtering81\.52%\+1\.09%74\.26%\-19\.41%77\.72%\-8\.83%Dude \(ours\)80\.43%–93\.67%–86\.55%–DeepseekV4\-Pro\-MaxDude w/o Negotiation70\.65%\-4\.35%90\.28%\-2\.96%79\.27%\-3\.86%Dude w/o Saliency\-Filtering76\.09%\+1\.09%81\.40%\-11\.84%78\.65%\-4\.48%Dude \(ours\)75\.00%–93\.24%–83\.13%–Table 4:Ablation study of Dude on the SciCoQA dataset using GPT5\.4 and DeepseekV4\.Δ\\Deltavalues are calculated relative to Dude \(ours\) within each model group\. Bold values in theΔ\\Deltacolumn highlight the largest degradation\.Overall performance analysis\.Table[2](https://arxiv.org/html/2609.03416#S4.T2)presents a comprehensive comparison of our proposed Dude against baseline methods on the SciCoQA dataset, evaluated in terms of recall, precision, F1 score, and token consumption\. The results demonstrate that Dude consistently outperforms all baselines under both GPT\-5\.4 and DeepSeek\-V4 settings, yielding substantial improvements of up to22\.8%in recall,9\.0%in precision, and18\.7%in F1 score\.

Moreover, the result also shows that Vanilla\-MA, which performs paper\-oriented and code\-oriented detection \(i\.e\., dual\-detection\) within a multi\-agent framework, achieves a more substantial recall improvement than Prompt\-LM, in which a single agent is prompted to perform dual\-detection\. This indicates that prompting a single\-agent LLM is insufficient to perform effective dual\-detection and thorough discovery of paper\-code discrepancies\. Despite its pronounced recall gains, Vanilla\-MA suffers from a substantial drop in precision, whereas our proposed Dude preserves high precision while still benefiting from the recall enhancement from the multi\-agent dual\-detection design\. This showcases the effectiveness of the granularity\-aligned negotiation process and two\-stage filtering mechanism in Dude, which jointly mitigates the over\-interpretation and over\-reporting challenges, thereby reducing false\-positive discrepancy reports\.

In terms of the token consumption, Dude incurs the lowest additional overhead among all multi\-agent methods\. This efficiency stems from the design in which all paper agents and code agents in Dude communicate exclusively through structured JSON files, substantially reducing the token cost of multi\-agent interactions\. In addition, Dude maintains a record for all previously searched paper and code snippets, preventing agents from expending extra tokens on redundant searches\.

Recall analysis by discrepancy type\.To better understand the sources of the recall gains achieved by our multi\-agent framework Dude over the single\-agent baseline Single\-LM, we conduct a fine\-grained analysis across discrepancy types\. Fig\.[3](https://arxiv.org/html/2609.03416#S4.F3)compares the recall of Dude and Single\-LM under four LLM backbones \(GPT, Claude, Kimi, and DeepSeek\) across three discrepancy categories: paper\-code conflict, paper omission, and code omission\. The results show that Dude consistently outperforms Single\-LM across all three discrepancy types and all four LLM backbones, with average gains ranging from 19\.1% to 28\.5%\. The improvement is most pronounced on paper omission \(28\.5%\), which is particularly notable given that this category has been identified as the most challenging for single\-agent systems[Baumgärtner and Gurevych \(2026\)](https://arxiv.org/html/2609.03416#bib.bib1)\. Taken together, these results confirm that the multi\-agent architecture in Dude yields broad and robust recall improvements on the paper\-code discrepancy detection task\.

Robustness under heterogeneous LLMsWe further evaluate Dude when different LLMs are assigned to be the paper\-side and code\-side agents\. Table[3](https://arxiv.org/html/2609.03416#S4.T3)reports F1 scores under four configurations: two hybrid settings with GPT\-5\.4 and DeepSeekV4 assigned to opposite sides, and two uniform settings using the same model on both sides\. The results show that Dude maintains robust discrepancy detection performance even under mixed LLM model assignments\. We further observe that replacing the paper\-side agents has a larger impact than replacing the code\-side agents, suggesting that the capability of the paper\-side agents plays a more crucial role in overall performance\.

### 4\.3Ablation studies

Table[4](https://arxiv.org/html/2609.03416#S4.T4)presents the ablation study of our proposed Dude on SciCoQA dataset using GPT5\.4 and DeepseekV4, evaluating the individual contributions of our designed negotiation and saliency\-filtering components in Dude\. The results show that removing either one of these two components results leads to a degradation in F1 score, confirming the effectiveness of both designs in discrepancy detection\. More specifically, removing the saliency\-filtering module causes a significant drop in precision\. Although this filtering module can occasionally filter out valid discrepancies, it is essential for suppressing the large number of trivial research claims over\-reported by the multi\-agent system\. Removing negotiation process, on the other hand, leads to a degradation in both recall and precision, indicating the necessity of bridging the granularity gap between natural language and code language during the consistency verification process\.

### 4\.4Parameter sensitivity analysis

Figure 4:The effect of negotiation roundsrron Recall, Precision, and F1 \(Dude\-GPT\-5\.4\)We investigate the impact of the number of negotiation roundsrron Dude’s discrepancy detection performance\. As illustrated in Fig\.[4](https://arxiv.org/html/2609.03416#S4.F4), the recall, precision, and F1 score of Dude exhibit monotonic improvement as the number of negotiation roundsrrincreases, reaching a plateau atr=2r=2, indicating that the performance saturates beyond this point\. We attribute this behavior to the observation that the majority of research claims can be resolved within two rounds of discussion\. As a result, even whenrris configured to a larger value, the negotiation process generally terminates within the first two rounds\.

### 4\.5Results on latest real\-world research

Table 5:The number of valid and invalid reported discrepancies as well as the precision results ofDudeand baseline methods in 20 latest top\-tier research works\.Method\# valid \(↑\\uparrow\)discrepancy\# invalid \(↓\\downarrow\)discrepancyPrecisionSingle\-LM29390\.63%Prompt\-LM31879\.49%Vanilla\-MA462168\.66%MAD34685\.00%Dude \(GPT\)45393\.75%To further assess the capability of Dude, we employ Dude with GPT\-5\.4 to detect paper\-code discrepancies in 20 selected research papers and their corresponding publicly available code repositories\. These papers were accepted by recent top\-tier AI conferences, including ICML 2025 and ICLR 2026\. As shown in Table[5](https://arxiv.org/html/2609.03416#S4.T5), Dude identifies the second\-largest number of valid discrepancies while achieving the highest precision among all methods\. Although Vanilla\-MA detects the largest number of valid discrepancies, it produces the highest number of invalid ones, resulting in significantly low precision\. These results demonstrate the strong potential of Dude for comprehensive and reliable paper\-code discrepancy detection in cutting\-edge AI research\.

## 5Conclusion

In this paper, we present Dude, a dual\-detection multi\-agent framework for paper\-code discrepancy detection\. Our analysis reveals that the inferior recall performance of the single\-agent paradigm stems from its inherent one\-sided discrepancy detection process, a limitation that cannot be overcame through simple prompting\. While a naive multi\-agent implementation alleviates this issue, it introduces a precision degradation arising from the granularity asymmetry between papers and code\. Dude addresses both limitations through a negotiation module that aligns cross\-granularity representations via iterative agent dialogue, and a two\-stage filtering module that suppresses over\-reported false positives\. Extensive experiments confirm that Dude substantially improves recall while preserving precision, establishing a strong foundation for automated paper\-code discrepancy detection\.

## Limitations

Despite its substantial improvements in recall and precision, Dude has two limitations\. First, its multi\-agent design leads to higher token consumption than single\-agent frameworks\. As shown in Table[2](https://arxiv.org/html/2609.03416#S4.T2), although Dude incurs the lowest token consumption among all multi\-agent methods, it still consumes more tokens than single\-agent baselines \(Single\-LM and Prompt\-LM\)\. Second, due to the high inference cost of the latest models, we were unable to evaluate all baseline methods under the latest backbones of all LLM providers \(GPT\-5\.5, Claude\-Opus\-4\.7\)\. Future work could investigate more token\-efficient multi\-agent frameworks and extend the evaluation to additional advanced LLM backbones to further assess the generalizability of our proposed Dude framework\.

## Ethical Considerations

Our proposed Dude is designed to serve as a self\-checking tool for researchers and as a paper\-code discrepancy detection tool for reviewers to assess the reliability and reproducibility of scientific research\. By providing a structured summary about potential inconsistencies between papers and their associated code, Dude can help alleviate the reviewing burden on reviewers under growing submission volumes and compressed review cycles\.

Despite its strong empirical performance, the output of Dude may still be incomplete or incorrect\. Therefore, its outputs should be regarded as supportive evidence rather than fully automated decisions\. Final judgments about the validity, quality, or reproducibility of a research work should remain under human oversight, with careful consideration of the paper, code, and experimental context\.

All experiments in this paper were conducted on publicly available datasets, and our use of models and data complies with their respective license terms\.

## Acknowledgments

This work was supported by the UGC General Research Fund no\. 17209822 and the Innovation and Technology Commission Fund no\. ITS/383/23FP from Hong Kong\.

## GenAI Usage Disclosure

Generative AI tools were only used for typo revising to improve clarity and readability, and were not used for method design or experimental analysis in this work\.

## References

- Anomalyco \(2026\)AnomalycoOpenCode: the open\-source ai coding agent\.External Links:[Link](https://github.com/anomalyco/opencode)Cited by:[§4\.1](https://arxiv.org/html/2609.03416#S4.SS1.p4.1)\.
- Baumgärtner and Gurevych \(2026\)T\. Baumgärtner and I\. GurevychSciCoQA: quality assurance for scientific paper–code alignment\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 38740–38770\.Cited by:[§B\.1](https://arxiv.org/html/2609.03416#A2.SS1.p1.1),[§B\.1](https://arxiv.org/html/2609.03416#A2.SS1.p2.1),[§B\.2](https://arxiv.org/html/2609.03416#A2.SS2.p1.1),[§1](https://arxiv.org/html/2609.03416#S1.p2.1),[§2](https://arxiv.org/html/2609.03416#S2.p2.1),[§4\.1](https://arxiv.org/html/2609.03416#S4.SS1.p1.1),[§4\.2](https://arxiv.org/html/2609.03416#S4.SS2.p4.1)\.
- Bianchiet al\.\(2025\)F\. Bianchi, Y\. Kwon, Z\. Izzo, L\. Zhang, and J\. ZouTo err is human: systematic quantification of errors in published ai papers via llm analysis\.arXiv preprint arXiv:2512\.05925\.Cited by:[§2](https://arxiv.org/html/2609.03416#S2.p1.1)\.
- Boginet al\.\(2024\)B\. Bogin, K\. Yang, S\. Gupta, K\. Richardson, E\. Bransom, P\. Clark, A\. Sabharwal, and T\. KhotSuper: evaluating agents on setting up and executing tasks from research repositories\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 12622–12645\.Cited by:[§2](https://arxiv.org/html/2609.03416#S2.p1.1)\.
- Chenet al\.\(2026\)H\. Chen, R\. Zhao, X\. Wu, and E\. C\. H\. NgaiMACRO\-llm: llm\-empowered multi\-agent collaborative reasoning under spatiotemporal partial observability\.External Links:2601\.09295,[Link](https://arxiv.org/abs/2601.09295)Cited by:[§2](https://arxiv.org/html/2609.03416#S2.p3.1)\.
- Honget al\.\(2024\)S\. Hong, M\. Zhuge, J\. Chen, X\. Zheng, Y\. Cheng, J\. Wang, C\. Zhang, z\. wang, S\. Yau, Z\. Lin, L\. Zhou, C\. Ran, L\. Xiao, C\. Wu, and J\. SchmidhuberMetaGPT: meta programming for a multi\-agent collaborative framework\.InInternational Conference on Learning Representations,B\. Kim, Y\. Yue, S\. Chaudhuri, K\. Fragkiadaki, M\. Khan, and Y\. Sun \(Eds\.\),Vol\.2024,pp\. 23247–23275\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/file/6507b115562bb0a305f1958ccc87355a-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2609.03416#S2.p3.1)\.
- Leeet al\.\(2025\)C\. Lee, C\. S\. Xia, L\. Yang, J\. Huang, Z\. Zhu, L\. Zhang, and M\. R\. LyuUnidebugger: hierarchical multi\-agent framework for unified software debugging\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 18248–18277\.Cited by:[§2](https://arxiv.org/html/2609.03416#S2.p3.1)\.
- Lianget al\.\(2024\)T\. Liang, Z\. He, W\. Jiao, X\. Wang, Y\. Wang, R\. Wang, Y\. Yang, S\. Shi, and Z\. TuEncouraging divergent thinking in large language models through multi\-agent debate\.InProceedings of the 2024 conference on empirical methods in natural language processing,pp\. 17889–17904\.Cited by:[§B\.1](https://arxiv.org/html/2609.03416#A2.SS1.p5.1),[§2](https://arxiv.org/html/2609.03416#S2.p3.1),[§4\.1](https://arxiv.org/html/2609.03416#S4.SS1.p2.1)\.
- Liaoet al\.\(2025\)C\. C\. Liao, D\. Liao, and S\. S\. GadirajuAgentmaster: a multi\-agent conversational framework using a2a and mcp protocols for multimodal information retrieval and analysis\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,pp\. 52–72\.Cited by:[§2](https://arxiv.org/html/2609.03416#S2.p3.1)\.
- Liuet al\.\(2026\)J\. Liu, J\. Pei, J\. Huang, C\. Si, A\. Qu, X\. Tang, R\. Lu, L\. Chen, X\. Bai, H\. Zheng, C\. Chen, Z\. Chen, H\. Ye, Y\. Fu, Z\. He, Z\. Jin, Z\. Zhang, S\. Sun, M\. Harmon, J\. D\. Wang, J\. Zeng, J\. Sun, M\. Wu, B\. Zhou, C\. You, S\. Lu, Y\. Qiu, F\. Lai, Y\. Yuan, Y\. Li, J\. Hong, R\. Zhu, B\. Chen, A\. Pentland, A\. Chen, M\. Chowdhury, and Z\. ZhangThe last human\-written paper: agent\-native research artifacts\.External Links:2604\.24658,[Link](https://arxiv.org/abs/2604.24658)Cited by:[§1](https://arxiv.org/html/2609.03416#S1.p1.1)\.
- Liu and Shah \(2023\)R\. Liu and N\. B\. ShahReviewergpt? an exploratory study on using large language models for paper reviewing\.arXiv preprint arXiv:2306\.00622\.Cited by:[§2](https://arxiv.org/html/2609.03416#S2.p1.1)\.
- Panthaplackelet al\.\(2021\)S\. Panthaplackel, J\. J\. Li, M\. Gligoric, and R\. J\. MooneyDeep just\-in\-time inconsistency detection between comments and source code\.InProceedings of the AAAI conference on artificial intelligence,Vol\.35,pp\. 427–435\.Cited by:[§2](https://arxiv.org/html/2609.03416#S2.p1.1)\.
- Qianet al\.\(2024\)C\. Qian, W\. Liu, H\. Liu, N\. Chen, Y\. Dang, J\. Li, C\. Yang, W\. Chen, Y\. Su, X\. Cong, J\. Xu, D\. Li, Z\. Liu, and M\. SunChatDev: communicative agents for software development\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 15174–15186\.External Links:[Link](https://aclanthology.org/2024.acl-long.810/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.810)Cited by:[§2](https://arxiv.org/html/2609.03416#S2.p3.1)\.
- Ratol and Robillard \(2017\)I\. K\. Ratol and M\. P\. RobillardDetecting fragile comments\.In2017 32nd IEEE/ACM International Conference on Automated Software Engineering \(ASE\),pp\. 112–122\.Cited by:[§2](https://arxiv.org/html/2609.03416#S2.p1.1)\.
- Ronget al\.\(2025\)G\. Rong, Y\. Yu, S\. Liu, X\. Tan, T\. Zhang, H\. Shen, and J\. HuCode comment inconsistency detection and rectification using a large language model\.InProceedings of the IEEE/ACM 47th International Conference on Software Engineering,pp\. 1832–1843\.Cited by:[§2](https://arxiv.org/html/2609.03416#S2.p1.1)\.
- Tanget al\.\(2024\)X\. Tang, A\. Zou, Z\. Zhang, Z\. Li, Y\. Zhao, X\. Zhang, A\. Cohan, and M\. GersteinMedagents: large language models as collaborators for zero\-shot medical reasoning\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 599–621\.Cited by:[§2](https://arxiv.org/html/2609.03416#S2.p3.1)\.
- Tong and Zhang \(2024\)W\. Tong and T\. ZhangCodejudge: evaluating code generation with large language models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 20032–20051\.Cited by:[§2](https://arxiv.org/html/2609.03416#S2.p1.1)\.
- Wenget al\.\(2026\)Y\. Weng, M\. Zhu, Q\. Xie, Q\. Sun, Z\. Lin, S\. Liu, and Y\. ZhangDeepScientist: advancing frontier\-pushing scientific findings progressively\.InInternational Conference on Learning Representations,C\. Vondrick, B\. Hariharan, C\. Raffel, L\. Pinto, D\. Yang, and A\. Faust \(Eds\.\),Vol\.2026,pp\. 47981–48037\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2026/file/4f64494ecc3442f1c9261baa036378bc-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2609.03416#S2.p1.1)\.
- Xiet al\.\(2025\)S\. Xi, V\. Rao, J\. Payan, and N\. B\. ShahFlaws: a benchmark for error identification and localization in scientific papers\.arXiv preprint arXiv:2511\.21843\.Cited by:[§2](https://arxiv.org/html/2609.03416#S2.p1.1)\.
- Xuet al\.\(2026\)T\. Xu, X\. Zhu, X\. Lai, S\. Dang, X\. Lian, H\. Cheng, and J\. WangDo papers tell the whole story? a benchmark and framework for uncovering hidden implementation gaps in bioinformatics\.arXiv preprint arXiv:2603\.22018\.Cited by:[§1](https://arxiv.org/html/2609.03416#S1.p1.1),[§2](https://arxiv.org/html/2609.03416#S2.p2.1)\.
- Youet al\.\(2026\)L\. You, L\. Cao, and I\. GurevychPreventing the collapse of peer review requires verification\-first ai\.arXiv preprint arXiv:2601\.16909\.Cited by:[§1](https://arxiv.org/html/2609.03416#S1.p1.1)\.
- Yuanet al\.\(2025\)Z\. Yuan, M\. Liu, H\. Wang, and B\. QinMA\-gts: a multi\-agent framework for solving complex graph problems in real\-world applications\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 19297–19315\.Cited by:[§2](https://arxiv.org/html/2609.03416#S2.p3.1)\.
- Zhanget al\.\(2025a\)K\. Zhang, Q\. Liu, L\. Zhang, C\. Zheng, S\. Li, B\. Xu, M\. Yang, X\. Qiao, and W\. LuMADAWSD: multi\-agent debate framework for adversarial word sense disambiguation\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 22294–22313\.Cited by:[§2](https://arxiv.org/html/2609.03416#S2.p3.1)\.
- Zhang and Abernethy \(2025\)T\. M\. Zhang and N\. F\. AbernethyReviewing scientific papers for critical problems with reasoning llms: baseline approaches and automatic evaluation\.arXiv preprint arXiv:2505\.23824\.Cited by:[§2](https://arxiv.org/html/2609.03416#S2.p1.1)\.
- Zhanget al\.\(2025b\)Y\. Zhang, X\. Xu, X\. Xu, D\. Zhang, L\. Liu, and Y\. ChenOrchestrating audio: multi\-agent framework for long\-video audio synthesis\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 22278–22293\.Cited by:[§2](https://arxiv.org/html/2609.03416#S2.p3.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing, H\. Zhang, J\. Gonzalez, and I\. StoicaJudging llm\-as\-a\-judge with mt\-bench and chatbot arena\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 46595–46623\.External Links:[Document](https://dx.doi.org/10.52202/075280-2020)Cited by:[§4\.1](https://arxiv.org/html/2609.03416#S4.SS1.p3.1)\.

## Appendix AAgent Prompts

The complete agent configuration files, along with their complete prompts, are available at anonymous\.4open\.science/r/Dude\. We here briefly showcase the roles of different specialized agents in paper\-oriented detection, code\-oriented detection, and final report generation\.

### A\.1Paper\-oriented detection

Paper Claimer Agent \(Claim generation\)You are a paper\-claim extraction agent for paper\-to\-code auditing\.Goal:Extract implementation\-auditable research claims from the input paper file and generate the paper description of research claims in a single strict JSON object\.Input:<Input paper path\>Output:A paper\-claim\.json in the workspace root\.Your output must organize claims into these 6 categories:1\.Algorithm \(Step order, Core logic\)2\.Model \(Architecture, Initialization\)3\.Loss \(Loss definitions, Weighting, Coefficients\)4\.Evaluation \(Evaluation logic, Metrics, Inference or validation procedure\)5\.Data \(Dataset usage, Pre\-processing, Augmentation, Filtering, Sampling\)6\.Training \(Learning rate schedule, batch, epochs, optimizer, mixed Precision, clipping, accumulation\)Extraction rules:•Extract atomic claims whenever possible\. If a sentence contains multiple independently verifiable components, split them into multiple claims instead of combining them into one broad statement\.•Prefer implementation\-relevant claims over broad summaries\.•Do not invent, infer, or hallucinate details that are not explicitly supported by the paper\.•Every claim must include at least one ‘paper\_description‘ item\.•If a claim mixes multiple concept, assign the claim to the category corresponding to the main implementation target\.

Paper Claimer Agent \(Search external files\)Goal:Your goal is to determine whether external material is needed for verification of each research claim, and if so, add concise, evidence\-based external information to refine paper description\.Inputs:•paper claims: <paper\-claim\.json path\>•Paper: <Input paper path\>Output:Update the paper\-claim\.json\.Workflow:Step 1: Analyze each claim one by one\. For each claim object, read its paper description carefully and decide whether the claim requires external material to verify\.A claim requires external material if:•Verifying the claim requires establishing facts outside the current paper\.•The claim says the paper follows, adopts, uses, matches, extends, improves, or is consistent with an external source \(paper, dataset, benchmark, method, implementation, setting, pre\-processing rule, evaluation protocol\)\.•The claim depends on information that cannot be verified from the current paper alone\.A claim require no external material if:•it merely mentions an external dataset, benchmark, metric, method adopted in this paper\.•it merely introduces or mentions related work without making a verifiable dependency claim\.•it is simply listing prior work as background\.Step 2: Locate the referenced external source\.If a claim requires external material:•Use the claim’s paper description to find where this claim appears in the paper\.•Read the nearby context and corresponding reference\.•Extract the key search clues, such as:–cited paper title, citation number, and bibliography entry–dataset, benchmark, website, repository name–URL if present in the paperUse these clues to search for the most appropriate external source on the web\.Step 3: Read the external source and fetch external evidence to refine paper description of the research claim\.

Code Verifier AgentYou are a careful code\-verification agent for paper\-to\-code auditing\.Goals:Your task is to map the paper description of each research claim to likely implementation in the code repository, and verify whether the paper description is supported by the codebase\.Inputs:•Paper Claim: <paper\-claim\.json path\>•Code Repository: <Code Repository path\>Output:\- Update the paper\-claim\.json\.Workflow:For each research claim:1\.Search the code repository to look for likely implementation locations of the paper description of this claim\.2\.Add and write the ‘code\_location’ field of this claim, if no candidate location is found, return an empty array\.3\.Start your code inspection from the location specified in ‘code\_location’, this location is only the starting point, not necessarily the only relevant code\. Follow related function calls, class definitions, utilities, configuration, and dependencies when necessary\.4\.Determine whether the code implementation is consistent with the paper description\.5\.For each research claim, add:•Matching result = \{matched, partial, mismatch, unfound\}•ExplanationDefinition of matching result:•matched: The code implementation clearly and substantially supports the paper description\.•partial: The code implementation supports part of the claim, but not all of it; or the code implementation is weaker, narrower, conditional, or incomplete compared with the paper description\.•mismatch: The code implementation clearly contradicts the claim, or the code implements something materially different from what the paper description states\.•unfound: You cannot find relevant code implementation to verify the paper description, even after starting from code\_location and checking nearby/related code\.Requirements for Explanation:•Explain why the code matches, partially matches, or mismatches with the paper description\.•Use the relevant code files and paper content to support your conclusion\.

Paper Negotiatior AgentYou are the paper negotiator in a two\-agent negotiation workflow\.Your job is to evaluate whether the current explanation for each invalid research claim becomes valid\.Input:•Paper claim = <paper\-claim\.json path\>•Checklist = <checklist path\>•Paper = <Paper path\>Outputs:Update the paper\-claim\.json and checklist when necessary\.Workflow:1\.Inspect checklist and locate every claim whose status is ‘invalid‘\.2\.For each invalid claim:•Locate the original paper context of the claim\.•Compare the original context against the explanation of the claim, and evaluate whether the explanation of the research claim is valid\.•Edit the checklist and paper description accordingly\.Criterion for explanation validity:•A ‘valid’ explanation is when it no longer contains misunderstanding, factual mistakes, or unsupported extrapolation relative to the paper description and original paper context\.•An ‘invalid’ explanation is when the explanation contains misunderstanding, factual error, or unsupported extrapolation, compared to the paper description and original paper context\.•When in doubt, keep the claim ‘invalid’\.Editing requirement:•If you think that the explanation is ‘valid’, update only checklist so this claim’s status becomes ‘valid’\.•If you think that the explanation remains ‘invalid’, keep its checklist status as ’invalid’, and edit the paper description\. The edit should consider resolving the misunderstanding, factual error, or unsupported extrapolation of existing over\-interpreted explanation\.

Code Negotiatior AgentYou are the code negotiator in a two\-agent negotiation workflow\.Your job is to re\-evaluate whether the revised paper description of the research claim is supported by the code repository\.Inputs:•paper claims: <paper\-claim\.json path\>•Checklist = <checklist path\>•Codebase <Code repository path\>Workflow:1\.Read checklist and locate every invalid research claim\.2\.For each invalid claim:•Read and understand the revised papaer description of the research claim\.•Start your code inspection from the code\_location specified\. Follow related function calls, class definitions, utilities, configuration, and dependencies when necessary\.•Determine whether the code implementation is consistent with the revised paper description\.•Revise the current matching result and explanation accordingly, so the paper negotiator can judge your updated explanation in the next negotiation round\.

Orchestrator AgentYou are the orchestrator agentfor a two\-agent negotiation workflow involving:1\.‘paper negotiator‘ agent2\.‘code negotiator‘ agentYour job is to initialize a checklist, spawn two custom agents, coordinate the negotiation loop, and stop exactly under the defined stopping conditions\.Step 1:Create checklist\.jsonAt the beginning of the workflow, create checklist from the ‘paper\-claim\.json‘:1\.Traverse claims in the exact same order as they appear in ‘claim\_verify\.json‘\.2\.Initialize every claim status as ‘invalid‘\.3\.Initialize the ‘current\_round=0‘, and read the pre\-set maximum negotiation round from the configuration file\.Step 2:Invoke ‘paper negotiator’ agentStep 3:Check stopping conditionAfter the paper negotiator agent finishes, inspect checklist\.json\.If either condition holds, stop the negotiation loop:•All claims in the checklist arevalid\.•‘current\_round == maximum\_round‘Step 4:Invoke ‘code negotiator’ agentStep 5:Increment round counterAfter the code negotiator agent finishes, set ‘current\_round = current\_round \+ 1‘This increment counts as one complete negotiation round\.Then return to Step 2\.Final deliverables after negotiation:When the loop ends, produce:•Final ‘checklist\.json’•Final ‘paper\-claim\.json’ with revised paper description and explanation\.•A concise negotiation report including:–total completed rounds–final valid claims–final invalid claim ids–whether the loop stopped because all claims became valid or because ‘maximum\_round‘ was reached

### A\.2Code\-oriented detection

Code Claimer AgentYou are a code\-claim extraction agent for code\-to\-paper auditing\.Goal:Your job is to extract auditable research claims from the target code repository and summarize notable code implementations of the research claim in a strict JSON object\.Input:•paper claim = <paper\-claim\.json path\>•Codebase = <Code Repository Path\>Output:A code\-claim\.json in the workspace root\.Workflow:Step 1:Inspect ‘paper\-claim\.json‘ as prior knowledge so that you can:•understand the paper, repository, discrepancy categories, and already\-covered paper claims\.•build a roadmap of likely code locations\.•avoid repeating claims that are already present in ‘paper\-claim\.json‘;Step 2:Inspect the code repository strictly one category at a time, in the following order \(Algorithm, Model, Loss, Evaluation, Data, Training\)Step 3:For each category:1\.Inspect the code repository and identify notable and implementation\.2\.Exclude any finding that is already covered in ‘paper\-claim\.json‘, or repetitive in current ’code\-claim\.json’\.3\.Write the code implementation as a claim for this category into ‘code\-claim\.json‘\.Anchor\-based salience filtering\.The representative examples below are high\-priority retrieval anchors\. Use them to guide where you look first and what kinds of repository details deserve extra scrutiny\. However, they are not exhaustive and must not limit your search space\. You must still record any other code\-grounded, non\-redundant, impactful implementation detail even if it does not match these examples\.For the Algorithm category:•Hyper\-parameters, thresholds, script\-level settings, dynamic coefficients, or weighting decay, caching strategies that affects algorithm’s step order, control flow, and core logic\.•Code patches or fallback branches for corner cases\. E\.g\., exception handling, fallback behavior, tie\-breaking, or branch\-specific shortcuts\.For the Model category:•Architectural model implementation details like nonlinearities, normalization, pooling, concatenation, masking, routing, gating, reshaping, residual mixing, or feature combination choices\.•Activation functions, normalization procedures, masking policies, feature fusion details, or tensor manipulation choices that alter how representations are formed or propagated;For the Loss or Training category:•Additional objective terms and regularizers to stabilize model training\. E\.g\., code includes extra loss terms, penalty terms, priors, or fallback objectives\.•Hyper\-parameters, thresholds, script\-level settings, dynamic coefficients, weighting decay, curriculum schedules, caching strategies, optimizer / scheduler details, or numerical stabilizers that affect training dynamics\.For the Evaluation category:•Evaluation protocol, benchmark\-specific handling, scoring rules, thresholds, sample filters, or subset selection that affects the evaluation process and success criterion\.•Exclusions, special\-case metric handling, dataset\-dependent or benchmark\-dependent evaluation branches, or evaluation protocol shortcuts that change reported outcomes;For the Data category:•Data preprocessing strategy like augmentations, truncation, clipping, subsampling feature smoothing, transformation, and prompt engineering that changes the data input distribution

Paper Verifier AgentYou are a careful paper\-verification agent for paper\-to\-code auditing\.Goal:Your task is to remove trivial claims in ‘code\-claim\.json’ based on their relevance to the paper, and determines whether the code implementation is supported by the paper\.Input:•code claims = <code\-claim\.json path\>•paper = <paper path\>Output:Update the code\-claim\.json\.Workflow:For each research claim in the code\-claim\.json:1\. Inspect the paper carefully to look for corresponding paper description for the code implementation of the research claim\.2\. Remove the trivial research claims by assessing the relevance between the code implementation and the paper contributions\.3\. Determine whether the paper description matches, partially matches, mismatches, or fails to mention the code implementation described in the research claim\.4\. For each research claim, add:•Matching result = \{matched, partial, mismatch, unfound\}•ExplanationDefinition of ‘matching result’•matched: The paper description clearly and substantially supports the code implementation\.•partial: The paper description supports part of the code implementation, but not all of it; or the code implementation is weaker, narrower, conditional, or incomplete compared with the paper description\.•mismatch: The paper description clearly contradicts the code implementation, or the code implements something materially different from what the paper description states\.•unfound: You cannot find relevant paper description to verify the code implementation\.Requirements for Explanation:•Explain why the paper description matches, partially matches, or mismatches with the code implementation\.•Use the original paper text to support your conclusion\.

### A\.3Final report generation

Orchestrator AgentYour task is to read paper\-claim\.json and code\-claim\.json, and then conduct the paper\-code discrepancy classification based on the matching result of research claims\.•For each research claim in the paper\-claim\.json,–If the’matching result’is’matched’, do not include\.–If the’matching result’is’partial’or’mismatch’, append the claim to’paper\-code conflict’–If the’matching result’is’unfound’, append the claim to’code omission’•For each research claim in the code\-claim\.json,–If the’matching\_status’is’matched’, do not include it\.–If the’matching result’is’partial’or’mismatch’, append the claim to’paper\-code conflict’–If the’matching result’is’unfound’, append the claim to’paper omission’

## Appendix BAdditional implementation details

### B\.1Baseline method configurations

Single\-LM:For the Single\-LM baseline method, we adopt the same prompt and configurations used in SciCoQA[Baumgärtner and Gurevych \(2026\)](https://arxiv.org/html/2609.03416#bib.bib1), thereby maintaining consistency with prior work and ensuring a fair comparison\.

Prompt\-LM:For the Prompt\-LM baseline method, we enable a single agent to perform both paper\-oriented discrepancy detection and code\-oriented discrepancy detection\. The prompt is revised based on the one used in Single\-LM[Baumgärtner and Gurevych \(2026\)](https://arxiv.org/html/2609.03416#bib.bib1)\.

Prompt\-LMYou are an expert in analyzing research papers and their corresponding code implementations\. Your task is to carefully identify concrete discrepancies between what is described in a paper and what is actually implemented in the code\.\#\# What counts as a discrepancy\- A concrete paper–code discrepancy means a mismatch between what is stated in the original paper \(e\.g\., formulas, algorithms, logic, methods, processes, or other settings\) and what is implemented in the original code repository\.\- Each distinct mismatch should be reported as a separate item\.\#\# ImportantYou should conduct both paper\-oriented detection and code\-oriented detection to discover the paper\-code discrepancies\. More specifically, you should first extract the research claim from the paper and inspect its corresponding code implementations to verify the consistency\. Then you should extract the research claim from the code repository and inspect the corresponding paper content to verify the consistency\.

Vanilla\-MA:We employ Vanilla\-MA by removing the negotiation process and two\-stage salience filtering modules in our propose Dude\.

MAD:We adopt the default prompts and configurations in[Liang et al\. \(2024\)](https://arxiv.org/html/2609.03416#bib.bib2)to perform the discrepancy detection\. We set the number of debatersnd​e​b​a​t​e=2n\_\{debate\}=2in our experiments\.

Notably, for fair comparison, all baseline methods and Dude have the same access to the web\-search tools \(WebSearch,WebFetch\), code\-search tools \(Grep,Glob,Read\), and pdf\-extraction tools \(MinerU, PyMuPDF\) in all experimental settings\.

### B\.2Evaluation Setup

To evaluate whether the reported paper\-code discrepancies matches the discovered discrepancies in SciCoQA, we employ the same LLM\-as\-a\-Judge protocol and prompt proposed in the original SciCoQA paper[Baumgärtner and Gurevych \(2026\)](https://arxiv.org/html/2609.03416#bib.bib1)\.

Prompt for LLM\-as\-a\-JudgeYour task is to evaluate whether a reference paper\-code discrepancies matches a predicted paper\-code discrepancy\. Follow these steps:1\. Analyze which part of the paper or code each discrepancy is describing\. Extract the core claims and issues from the reference and predicted discrepancies\.2\. Analyze whether the core claims are about the same issue, i\.e\. if they describe the same or different paper\-code discrepancies\. The two discrepancies might use different wording or one might be more detailed than the other\. Focus on whether the issue is the same, even if minor details are different\. However, if they describe different issues \(even about the same topic or part of the paper or code\) they do not match\.3\. Provide a brief explanation of your reasoning\.\# Answer Format Provide your answer in the following format: <yes \| no \> <Brief explanation\>

### B\.3Annotation and Validation Protocol

We construct the annotations by adopting the annotation process, validation protocol, and prompt in SciCoQA\. This keeps our annotation and validation consistent with this prior work while enabling a fair comparison across compared methods\. The annotation and validation processes are as follows\.

All compared methods, including Single\-LM, Prompt\-LM, Vanilla\-MA, MAD, and Dude, are first applied to the 20 paper\-code pairs to detect candidate discrepancies\. Then we aggregate all reported discrepancies and validate them using Gemini\-3\.1\-Pro as an LLM\-as\-a\-Judge\. When the judge’s decision is ambiguous or when different LLM outputs yield conflicting evidence, authors with relevant domain expertise manually verify the discrepancy by inspecting both the paper claim and the corresponding code\.

After validation, for each valid discrepancy, Gemini\-3\.1\-Pro is prompted to generate a standardized description of 3–5 sentences covering the paper statement, the code implementation, and where the discrepancy lies\. We use these standardized descriptions as the reference annotations for all validated paper\-code discrepancies identified in the 20 real\-world papers\.

## Appendix CToken Consumption Analysis

Figure 5:Token consumption percentage by agentsFigure 6:Token consumption percentage by discrepancy detection processWe provide a fine\-grained token consumption analysis of our proposedDudeby agents \(Fig\.[5](https://arxiv.org/html/2609.03416#A3.F5)\) and by the discrepancy detection process \(Fig\.[6](https://arxiv.org/html/2609.03416#A3.F6)\)\. Fig\.[5](https://arxiv.org/html/2609.03416#A3.F5)presents the breakdown of token consumption across all agents, measured as the percentage of the total token usage\. The results show that the paper claimer agent, paper negotiator agent, and code verifier agent are the three most token\-intensive agents\. Notably, the paper claimer agent accounts for nearly 25% of the overall token consumption\. This huge token consumption of paper claimer agent primarily arises from its responsibility to search for and retrieve external materials during paper\-oriented generation, which substantially increases its context length and leads to markedly higher token usage compared with the other agents\. The higher token consumption of paper\-side agents \(paper claimer, paper negotiator\) also supports our empirical results in Table[3](https://arxiv.org/html/2609.03416#S4.T3), which suggests that the capability of the paper\-side agents plays a more crucial role than code\-side agents in overall performance\.

We also examine the token consumption of different stages in our proposed Dude framework, including paper\-oriented detection, code\-oriented detection, and final report generation\. Fig\.[6](https://arxiv.org/html/2609.03416#A3.F6)shows that paper\-oriented detection is the primary source of token consumption, accounting for over 73% of the total usage and substantially exceeding the other two stages\. This high cost can be attributed to two main factors\. First, as we discussed above, the paper claimer agent in paper\-oriented detection incurs considerable token overhead due to its need to search for and retrieve external materials for claim analysis\. Second, the granularity\-aligned negotiation process in paper\-oriented detection requires multi\-round interactions among agents, which also further increases the token budget required by the paper\-oriented detection\.

## Appendix DAdditional Experimental Results

### D\.1Real\-world Research Results

Table[5](https://arxiv.org/html/2609.03416#S4.T5)reports precision on the 20 recent papers because, unlike SciCoQA, these 20 newly published works have no pre\-existing discrepancy annotations to compute the recall metric\. To address this, we compute recall and F1 against the reference set of 55 verified discrepancies whose annotation and validation process is detailed in Section[B\.3](https://arxiv.org/html/2609.03416#A2.SS3)\.

We provide the complete real\-world evaluation results below, including recall, precision, F1, and average token consumption\.

MethodRecallPrecisionF1TokenSingle\-LM52\.73%90\.63%66\.67%0\.84MPrompt\-LM56\.36%79\.49%65\.96%0\.91MVanilla\-MA83\.64%68\.66%75\.41%1\.33MMAD61\.82%85\.00%71\.58%1\.58MDude81\.82%93\.75%87\.38%0\.98MTable 6:The recall, precision, F1, and token usage results ofDudeand baseline methods on the 20 real\-world research using GPT\-5\.4\.Table[6](https://arxiv.org/html/2609.03416#A4.T6)shows that Dude improves real\-world paper\-code discrepancy detection not merely by increasing the number of reported discrepancies, but by substantially improving recall while maintaining high precision\. Notably, since the reference set is built from discrepancies reported by these methods and then validated, it may not include discrepancies that all methods missed\. The recall values in Table[6](https://arxiv.org/html/2609.03416#A4.T6)are therefore best read as a fair comparison across methods rather than an exact measure of how many true discrepancies exist\.

### D\.2Evaluation Results on GPT\-5\.5

We conduct an additional evaluation on SciCoQA dataset using the latest GPT\-5\.5 backbone, comparing Dude against all baseline methods\.

MethodRecallPrecisionF1TokenSingle\-LM64\.13%93\.65%76\.13%0\.79MPrompt\-LM64\.13%86\.76%73\.75%0\.85MVanilla\-MA73\.91%79\.07%76\.40%1\.39MMAD65\.22%95\.24%77\.42%1\.41MDude82\.61%96\.20%88\.89%0\.96MTable 7:The recall, precision, F1, and token usage results ofDudeand baseline methods on the SciCoQA dataset using GPT\-5\.5\.Table[7](https://arxiv.org/html/2609.03416#A4.T7)shows that most methods benefit from the stronger GPT\-5\.5 backbone compared with GPT\-5\.4, indicating that the newer LLMs improve paper\-code discrepancy detection\. However, simply replacing the backbone with GPT\-5\.5 does not eliminate the limitations of the single\-agent paradigm in discrepancy detection\. The Single\-LM baseline with GPT\-5\.5 still achieves only 64\.13% overall recall, while our proposed Dude continues to achieve strong performance under GPT\-5\.5, improving recall by 18\.48% and F1 by 12\.76% compared with Single\-LM\. This demonstrates that the dual\-detection design and negotiation/filtering mechanisms in Dude remain effective on more recent LLM backbones\.

### D\.3Human and cross\-LLM Agreement

To assess the reliability of our adopted Gemini\-3\.1\-Pro as an LLM judge, we sample 50 paper\-code discrepancy cases spanning both SciCoQA and recent real\-world papers\. These cases were manually reviewed by the authors with relevant domain expertise, since validating paper\-code discrepancies requires understanding both paper claims and code implementation\. We compare the human judgments with those produced by Gemini\-3\.1\-Pro\. Among the 50 cases, 47 received the same judgment from both, yielding a 94% agreement rate\. We also compare Gemini\-3\.1\-Pro against a second LLM judge, Qwen\-3\.6\. On the same 50 cases, 44 received the same judgment, yielding an 88% agreement rate\. This indicates that our LLM\-as\-a\-Judge protocol is reasonably reliable for our evaluation, while we acknowledge that subtle cases still benefit from human evaluation\.

### D\.4Representative Failure Case Study

Although Dude substantially improves recall and precision over existing paper\-code discrepancy detection methods, it is not perfect\. Our inspection shows that Dude ’s remaining false negatives mainly arise from implicit discrepancies, where the inconsistency is not stated through explicit keywords but is induced by code behavior and requires deeper reasoning to detect\. On the other hand, the remaining false positives arise from ambiguous engineering code\. We provide two representative failure cases of Dude below, one false positive and one false negative, which give a clearer understanding of Dude’s remaining limitations\.

Case 1 \(False Positive / Incorrect Report\):Some repositories contain demonstration code in \.ipynb notebooks\. These notebooks may use settings that differ from the official implementation and the paper, such as running only a single trial instead of repeating experiments and reporting averaged results\. These notebooks sometimes are not clearly named or commented as ’example’ or ’demonstration’ code\. In such cases, Dude treats them as authoritative implementations and reports the difference as a discrepancy\.

Case 2 \(False Negative / Missed Discrepancy\):In one case, the code restricts the outputs of the proposed module to the range \(0,1\) using a sigmoid activation, while the paper formulation describes these parameters as outputs of standard unbounded MLPs\. Dude missed this discrepancy because the inconsistency is implicit in the activation function and requires reasoning about the numerical range induced by the code, rather than simply matching a stated paper claim to a code fragment\.