TADDLE: A Tool-Augmented Agent for Detecting Deficient LLM-Generated Peer Reviews

# TADDLE: A Tool-Augmented Agent for Detecting Deficient LLM-Generated Peer Reviews
Source: [https://arxiv.org/html/2605.26911](https://arxiv.org/html/2605.26911)
Hanqi Duan1,Xiang Li1,\*

1East China Normal University, Correspondence:[xiangli@dase\.ecnu\.edu\.cn](https://arxiv.org/html/2605.26911v1/mailto:[email protected])\(Xiang Li\*\)

###### Abstract

LLM\-generated peer reviews are increasingly common at major venues, yet their deficiencies are hard to detect because they are uniformly fluent and well\-structured\. Existing work either classifies authorship without judging quality, or scores quality with features designed for human\-written reviews; no prior system detects deficiencies in LLM\-generated reviews at the level of individual defect types\. To bridge the gap, we introduceTADDLE, a Tool\-Augmented Agent for Detecting Deficient LLM\-Generated Peer Reviews, together with the first expert\-annotated benchmark for this task\. Our benchmark comprises1,8001\{,\}800reviews on5050ICLR20252025papers, multi\-label\-annotated by1818domain experts against a taxonomy of six defect categories \(plus a non\-deficient label\)\.TADDLEdecomposes detection into four specialized analysis tools—Verify,Correct,Complete, andTransform—orchestrated by an agent; an integrator synthesizes their outputs into binary and multi\-label classifications via two\-stage semi\-supervised learning\. Extensive experiments show thatTADDLEperforms strongly on both binary detection and the multi\-label classification task\. We release the benchmark and code at[https://github\.com/AquariusAQ/TADDLE](https://github.com/AquariusAQ/TADDLE)\.

TADDLE: A Tool\-Augmented Agent for Detecting Deficient LLM\-Generated Peer Reviews

Hanqi Duan1, Xiang Li1,\*1East China Normal University,Correspondence:[xiangli@dase\.ecnu\.edu\.cn](https://arxiv.org/html/2605.26911v1/mailto:[email protected])\(Xiang Li\*\)

## 1Introduction

Peer review sits at the heart of scientific communication\. The recent proliferation of LLMs has changed how reviews are written: ICLR 2026, ICML 2026, and ACL Rolling Review now permit reviewers to declare the use of AI assistance, and area chairs increasingly encounter reviews that, on the surface, are indistinguishable from those of a careful human expert\. Quality, however, is not uniform\. A shared concern is that some LLM\-generated reviews, while fluent, contain hallucinated experimental details, surface critiques that have already been answered in the appendix, or apply dual standards under the guise of high academic expectationsLiet al\.\([2025](https://arxiv.org/html/2605.26911#bib.bib5)\); Linet al\.\([2025](https://arxiv.org/html/2605.26911#bib.bib9)\)\.

The key challenge is that the failure modes of LLM\-generated reviews do not look like those of careless human reviewers\. A human reviewer who has not read the paper usually produces text that is short, evasive, or visibly generic; an LLM that has misread the paper produces text that is long, well\-structured, and confidently grounded in fabricated specifics\. Concretely, an LLM review may state*“the paper does not extend to multi\-class classification”*when the appendix devotes a section to that\. Such deficiencies leave evidence, but the evidence lives in the relationship between the review and the paper: the review only contains an “information error” relative to what the paper actually says\.

Existing work on the related topic can be divided into two lines\. On the one hand,*LLM\-generated text detection*Zhouet al\.\([2026](https://arxiv.org/html/2605.26911#bib.bib7)\); Kumaret al\.\([2024](https://arxiv.org/html/2605.26911#bib.bib8)\); Yuet al\.\([2024](https://arxiv.org/html/2605.26911#bib.bib11)\); Kumaret al\.\([2025](https://arxiv.org/html/2605.26911#bib.bib10)\)answers a binary authorship question: was the text produced by a model? This is orthogonal to quality\. Once a venue allows AI assistance, knowing that a review is LLM\-generated tells an area chair nothing about whether it should be trusted\. On the other hand,*review\-quality assessment*for human\-written reviewsSizoet al\.\([2025](https://arxiv.org/html/2605.26911#bib.bib1)\); Arouset al\.\([2021](https://arxiv.org/html/2605.26911#bib.bib3)\); Ryuet al\.\([2025](https://arxiv.org/html/2605.26911#bib.bib6)\); Zhanget al\.\([2025](https://arxiv.org/html/2605.26911#bib.bib14)\); Ebrahimiet al\.\([2025](https://arxiv.org/html/2605.26911#bib.bib19)\)relies on surface features \(e\.g\., length, citation count, sentiment polarity, semantic alignment\) designed for a regime in which careless and competent reviews look visibly different\. LLM\-generated reviews are uniformly polished on these features, so the signal largely vanishes\. None of these can transfer to the regime targeted in this paper\.

To detect the defects in LLM\-generated reviews, we claim that detection should be decomposed by the type of evidence each defect leaves behind\. We instantiate this view inTADDLE, aTool\-AugmentedAgent forDetectingDeficientLLM\-Generated Peer Reviews, where an orchestrator decomposes the review into evidence segments, routes them through four specialized analysis tools, and passes the collected traces to a fine\-tunedIntegratemodule that synthesizes them into the final classification\. The four tools differ in their prompts, the scope of paper content they may access, and whether they may query external literature:Verifyfact\-checks the review’s claims against the paper,Correctclassifies any detected errors by type,Completechecks whether each criticism is paired with an actionable revision direction, andTransformflags subjective bias and hostile tone\. To trainIntegrate, we construct a benchmark of LLM\-generated reviews on high\-disagreement from ICLR/ICML/NeurIPS20252025papers, consisting of an expert multi\-label\-annotated gold set and a high\-confidence pseudo\-labeled set\. In summary, the main contributions of our paper include:

- •We introduce the first multi\-label, expert\-annotated benchmark for fine\-grained deficient LLM\-review detection\.
- •We developTADDLE, a tool\-augmented agent\-based detector that decomposes review auditing into four specialized analysis tools and a fine\-tuned integrator\.
- •We conduct extensive experiments across multiple evaluation settings, on whichTADDLEconsistently achieves the best performance\.

## 2Related Work

##### LLM\-generated text detection\.

A line of work targets authorship by statistical token\-distribution tests, generation\-time watermarking and fine\-tuned classifiersZhouet al\.\([2026](https://arxiv.org/html/2605.26911#bib.bib7)\); Kumaret al\.\([2024](https://arxiv.org/html/2605.26911#bib.bib8)\); Yuet al\.\([2024](https://arxiv.org/html/2605.26911#bib.bib11)\); Kumaret al\.\([2025](https://arxiv.org/html/2605.26911#bib.bib10)\)\. These methods achieve strong authorship detection but answer a question orthogonal to ours: an LLM review can be entirely accurate or completely hallucinated and remain stylistically indistinguishable\. As venues allow AI assistance, the bottleneck shifts from*authorship*to*correctness*\.

##### Quality assessment of human\-written peer review\.

Quality standards for human peer review have been defined along axes of accuracy, constructiveness, fairness, and comprehensivenessSizoet al\.\([2025](https://arxiv.org/html/2605.26911#bib.bib1)\)\. Existing methods combine expert rating scales with rule\-based heuristics and supervised models over surface features—review length, citation count, politeness, sentiment polarity, semantic alignment with the paperArouset al\.\([2021](https://arxiv.org/html/2605.26911#bib.bib3)\); Ryuet al\.\([2025](https://arxiv.org/html/2605.26911#bib.bib6)\); Lanier \([2021](https://arxiv.org/html/2605.26911#bib.bib4)\); Liet al\.\([2025](https://arxiv.org/html/2605.26911#bib.bib5)\)\. These features are reliable discriminators for human\-written reviews because careless and competent human reviewers differ along them\. However, they are much less informative for LLM\-generated reviews, which are uniformly fluent and structurally regular regardless of quality\.

##### Agent systems and LLM\-as\-judge\.

LLM agents augmented with tools and structured reasoning have been applied to multi\-hop question answering, code generation, and scientific writingYaoet al\.\([2022](https://arxiv.org/html/2605.26911#bib.bib12)\); Tranet al\.\([2025](https://arxiv.org/html/2605.26911#bib.bib13)\)\. In peer review, agent systems have been used to generate reviewsJinet al\.\([2024](https://arxiv.org/html/2605.26911#bib.bib15)\); Gaoet al\.\([2025](https://arxiv.org/html/2605.26911#bib.bib16)\)or to assist literature surveysGoet al\.\([2026](https://arxiv.org/html/2605.26911#bib.bib17)\)\. The complementary problem of auditing reviews has received less attention\. Closely related are ReviewGuardZhanget al\.\([2025](https://arxiv.org/html/2605.26911#bib.bib14)\), which uses GPT\-4\.1 to annotate conflicted OpenReview reviews and fine\-tunes open\-source LLMs end\-to\-end, and RottenReviewsEbrahimiet al\.\([2025](https://arxiv.org/html/2605.26911#bib.bib19)\), which regresses quality on quantifiable features\. Both systems were developed for and evaluated on human\-written or mixed\-authorship review corpora; their feature spaces and training distributions reflect the failure modes of human reviewers \(brevity, visible carelessness, sentiment extremes\), which differ qualitatively from those of LLM\-generated reviews\.

## 3Task and Defect Taxonomy

##### Notation\.

Let𝒫\\mathcal\{P\}denote a paper,rra peer review for𝒫\\mathcal\{P\}\. We define a label spaceℒ=\{c0\}∪𝒞\\mathcal\{L\}=\\\{c\_\{0\}\\\}\\cup\\mathcal\{C\}, wherec0c\_\{0\}is the “non\-deficient” label and𝒞=\{c1,…,c6\}\\mathcal\{C\}=\\\{c\_\{1\},\\dots,c\_\{6\}\\\}are the six defect categories, detailed in Appendix[E](https://arxiv.org/html/2605.26911#A5)\. Each review is associated with a binary deficiency labelz∈\{0,1\}z\\in\\\{0,1\\\}and a multi\-label vectory∈\{0,1\}7y\\in\\\{0,1\\\}^\{7\}overℒ\\mathcal\{L\}, withz=1⇔∃i\>0:yi=1z=1\\iff\\exists\\,i\>0:y\_\{i\}=1andz=0⇔y0=1z=0\\iff y\_\{0\}=1\. Sincec0c\_\{0\}is fully determined byzz, the multi\-label prediction task reduces to selecting a subset of the six defect categories𝒞\\mathcal\{C\}; we refer to this as the multi\-label task throughout\. A detectorf:\(r,𝒫\)↦\(z^,y^,q,O\)f:\(r,\\mathcal\{P\}\)\\mapsto\(\\hat\{z\},\\hat\{y\},q,O\)jointly produces the binary label, the multi\-label defect vector, a five\-point quality scoreqq, and a free\-form outputOOcontaining per\-defect explanations and revision suggestions\.

##### Taxonomy derivation\.

We refine 23\-type catalog of defectsDuet al\.\([2024](https://arxiv.org/html/2605.26911#bib.bib18)\)into six categories\. We use three operations: \(i\) merging closely overlapping types \(e\.g\., experiment\-evaluation errors, hallucinations, and self\-contradictions all collapse toInformation Error; out\-of\-scope suggestions and inexpert statements collapse toUnprofessional & Hostile\); \(ii\) removing formal issues \(formatting errors, spelling, missing scores\) that are detectable by simple rule\-based tools and are not the focus of this work; and \(iii\) removing categories as rarely occurring in LLM\-generated reviews \(e\.g\., duplicate statements\)\. Finally, we derive six editorially actionable categories at sufficient frequency to permit supervised modeling; full details are given in Appendix[E](https://arxiv.org/html/2605.26911#A5)\.

## 4Benchmark Construction

### 4\.1Paper selection and parsing

We sample papers from the public OpenReview record of ICLR, ICML, and NeurIPS20252025\. To ensure a meaningful detection signal, we restrict attention to papers with high reviewer disagreement— i\.e\., papers where the highest and lowest review scores differ by at least 3 points for ICLR, and by at least 2 points for ICML/NeurIPS, reflecting the rating scales used by each venue\. High\-disagreement papers are deliberately preferred because they present genuine challenges\. These are precisely the conditions under which LLM\-generated reviews are most likely to introduce factual errors or biased verdicts, making the detection task realistic rather than trivial\.

Each paper is parsed from PDF into Markdown plus image files via PaddleOCR\-VLCuiet al\.\([2025](https://arxiv.org/html/2605.26911#bib.bib35)\)\. Every figure is described in≤200\\leq 200tokens with Qwen3\-VL\-4B\-Instruct; every appendix subsection is summarized in≤500\\leq 500tokens with Qwen3\-30B\-A3B\-Thinking\-2507\. Each paper is thus represented by four text components—main text, references, appendix summary, figure descriptions—which form the inputs available to the analysis tools at detection time\.

### 4\.2Persona\-conditioned review generation

We use six LLMs as review generators: DeepSeek V3\.2DeepSeek\-AI \([2025](https://arxiv.org/html/2605.26911#bib.bib23)\), GLM 4\.7Teamet al\.\([2025](https://arxiv.org/html/2605.26911#bib.bib26)\), Kimi K2\.5Teamet al\.\([2026](https://arxiv.org/html/2605.26911#bib.bib32)\), Qwen3\-30BTeam \([2025](https://arxiv.org/html/2605.26911#bib.bib25)\), Llama 3\.3\-70BGrattafioriet al\.\([2024](https://arxiv.org/html/2605.26911#bib.bib33)\)and MiniMax\-M2\.5MiniMax \([2026](https://arxiv.org/html/2605.26911#bib.bib34)\)\. For each paper, nine reviewer personas are instantiated: three competent personas \(accurate & information\-faithful; constructive; professional & impartial\) and six deficient personas, one per defect type\. Each persona is a system\-prompt specification that conditions the model’s reviewing behavior; deficient personas inject the behavior \(e\.g\., “*you tend to misread figure trends and reverse relationships between variables*”\) but never allow the model to reveal its own defect category\. The model produces a structurally complete review with summary, strengths, weaknesses, questions, and per\-axis scores; full prompts are given in Appendix[H](https://arxiv.org/html/2605.26911#A8)\.

##### Role of personas\.

Persona serves two purposes\. First, it ensures categorical coverage: without persona conditioning, the distribution of defects across the six categories would be uncontrolled and heavily skewed, leaving rare categories \(e\.g\.,Bias\-Oriented,Unsubstantiated Claims\) with near\-zero training support\. One persona per defect type guarantees sufficient positives in each category to support per\-category modeling and Macro\-F1 evaluation\. Second, each generated review carries an implicit single\-label weak tag \(the persona used\), which we exploit during pseudo\-label filtering \(§[5\.4](https://arxiv.org/html/2605.26911#S5.SS4)\)\. Crucially, the persona is not used as the gold label: humans annotate every review in the gold set against the review text, and may label any review with zero, one, or multiple defect categories regardless of the persona that produced it\. This design permits two outcomes: a deficient\-persona review may be annotated as “not deficient” if the LLM resisted the persona injection, and a competent\-persona review may contain defects and be annotated with defect labels by experts\.

### 4\.3Expert multi\-label annotation

We randomly sampled 50 ICLR 2025 papers and obtained their gold\-standard annotations, covering a total of 50×4×9=1,800 reviews generated by DeepSeek V3\.2, GLM 4\.7, Kimi K2\.5, and Qwen3\-30B\. Eighteen domain experts—all with prior peer review experience at top\-tier ML venues—each annotated reviews on a five\-point quality score \(Appendix[E](https://arxiv.org/html/2605.26911#A5)\)\. For any review scored below44, annotators performed multi\-label tagging on it against the six\-category taxonomy\. They consulted the original paper PDF when assessing factual claims\. The final guidelines used during full annotation are reproduced verbatim in Appendix[E](https://arxiv.org/html/2605.26911#A5)\.

##### Partitioning\.

The5050papers are split into4040training papers and1010test papers\. In total, we have1,8001\{,\}800annotated reviews \(50×4×950\\\!\\times\\\!4\\\!\\times\\\!9\)\. The training split uses the1,0801\{,\}080reviews generated by GLM 4\.7, Kimi K2\.5, and Qwen3\-30B on the4040training papers \(40×3×940\\\!\\times\\\!3\\\!\\times\\\!9\), excluding DeepSeek V3\.2 to leave the test generator unseen at fine\-tuning time\. The test split uses the9090DeepSeek V3\.2 reviews on the1010held\-out papers \(10×1×910\\\!\\times\\\!1\\\!\\times\\\!9\)\. The remaining630630annotated reviews are held aside to prevent information leakage between the training generator pool and the test generator\. The partition therefore creates a clean distribution shift in both papers and review generator between training and test\. Per\-class statistics are given in Table[7](https://arxiv.org/html/2605.26911#A7.T7)\(Appendix[G](https://arxiv.org/html/2605.26911#A7)\)\.

### 4\.4Distribution\-shift evaluation sets

To evaluate generalization, we further construct four additional weak\-supervision test sets :∼900\\sim\\\!900reviews each for ICML20252025and NeurIPS20252025papers \(cross\-conference\), and∼900\\sim\\\!900reviews each generated by Llama 3\.3\-70B and MiniMax\-M2\.5 \(cross\-generator\)\.

## 5TheTADDLEDetection System

### 5\.1Architecture

TADDLEdecomposes detection into four*analysis*stages, one*integration*stage, and a final*composition*stage\. Let𝒜=\{\\mathcal\{A\}=\\\{Verify,Correct,Complete,Transform\}\\\}denote the set of analysis tools\. The orchestrator selects, for reviewrr, a call sequenceπ​\(r\)=\(a1,a2,…,aK\)\\pi\(r\)=\(a\_\{1\},a\_\{2\},\\dots,a\_\{K\}\)with eachak∈𝒜a\_\{k\}\\in\\mathcal\{A\}andK≤8K\\leq 8\(the tool\-call budget\); the same tool may appear more than once inπ​\(r\)\\pi\(r\)when the orchestrator revisits it in light of later evidence\. Leth<k=\(Ta1,…,Tak−1\)h\_\{<k\}=\(T\_\{a\_\{1\}\},\\dots,T\_\{a\_\{k\-1\}\}\)denote the concatenation of all tool outputs preceding thekk\-th call\. For each toola∈𝒜a\\in\\mathcal\{A\},Ta​\(⋅\)T\_\{a\}\(\\cdot\)denotes the tool output function\. The bundle of analysis traces is then

𝐭=\(Tak​\(r,𝒫\[ak\],h<k,Ext\[ak\]\)\)k=1K\.\\mathbf\{t\}=\\bigl\(T\_\{a\_\{k\}\}\\big\(r,\\,\\mathcal\{P\}\_\{\[a\_\{k\}\]\},\\,h\_\{<k\},\\,\\text\{Ext\}\_\{\[a\_\{k\}\]\}\\big\)\\bigr\)\_\{k=1\}^\{K\}\.Classification and final output are produced as

\(z^,y^,q\)\\displaystyle\(\\hat\{z\},\\hat\{y\},q\)=fIntegrate​\(𝐭,r\),\\displaystyle=\\;f\_\{\\textsc\{Integrate\}\}\(\\mathbf\{t\},r\),\\qquadO\\displaystyle O=gOrchestrator​\(z^,y^,q,𝐭,r\),\\displaystyle=\\;g\_\{\\textsc\{Orchestrator\}\}\\big\(\\hat\{z\},\\hat\{y\},q,\\mathbf\{t\},r\\big\),where𝒫\[ak\]\\mathcal\{P\}\_\{\[a\_\{k\}\]\}is the slice of paper content visible to thekk\-th call,Ext\[ak\]∈\{enabled,disabled\}\\text\{Ext\}\_\{\[a\_\{k\}\]\}\\\!\\in\\\!\\\{\\text\{enabled\},\\text\{disabled\}\\\}indicates whether the call may query Semantic Scholar / arXiv,h<kh\_\{<k\}is the cumulative analysis history at that point,fIntegratef\_\{\\textsc\{Integrate\}\}is the LoRA\-fine\-tuned integrator that emits the binary labelz^\\hat\{z\}, the multi\-label vectory^\\hat\{y\}, and the five\-point quality scoreqq, andgOrchestratorg\_\{\\textsc\{Orchestrator\}\}is the orchestrator’s composition pass that emits the final outputOO\(per\-defect explanations, evidence traces, and actionable suggestions\)\.fIntegratef\_\{\\textsc\{Integrate\}\}is a soft aggregator: it is permitted by training to override individualTaT\_\{a\}outputs whenh<Kh\_\{<K\}conflict\.fIntegratef\_\{\\textsc\{Integrate\}\}is invoked exactly once per review\.

##### Orchestrator\.

The orchestrator is Qwen3\-30B\-A3B\-Thinking\-2507 used in its native thinking mode\. The orchestrator sees only the review text and the paper’s abstract\. This is a deliberate design choice motivated by two considerations\. First, long\-context loading at the orchestration layer weakens scheduling decisions consistent with prior observations on agent loopsChunget al\.\([2025](https://arxiv.org/html/2605.26911#bib.bib31)\)\. Second, the abstract provides enough context to plan tool routing—the orchestrator needs to determine what kind of paper is under review, not to verify specific claims—while the full paper is available where verification actually occurs, namely at the tools\.

##### Tool layer\.

The four analysis tools share the orchestrator’s backbone but differ in three concrete ways\. \(i\)*System prompts*specialize each tool to a single type of verification subtask\. \(ii\)*Input scope*differs:VerifyandCompleteare configured with access to the paper’s full text, references, appendix summary, and figure descriptions;CorrectandTransformoperate over the relevant evidence segments and prior tool outputs\. \(iii\)*External retrieval*is enabled forVerifyandCompleteonly, since these stages require literature comparison \(verifying factual claims, judging whether a stated improvement is substantive\);CorrectandTransformoperate without retrieval to control inference cost\.

##### Integrator\.

fIntegratef\_\{\\textsc\{Integrate\}\}is a Qwen3\.5\-9B model fine\-tuned with LoRA \. It is the only module with classification authority and the only module whose parameters are updated; the orchestrator and the four analysis tools are used off\-the\-shelf\. This classification–composition split offers three advantages: \(i\) the analysis tools are not biased by gradient signal from a particular label distribution; \(ii\) the integrator is small enough to be retrained as the taxonomy evolves; and \(iii\) the analysis tools can be replaced with closed\-source frontier models, easing adoption as new models become available\.

### 5\.2The four analysis tools

Tool outputs are structured JSON with fields specific to each tool\.Verifyemits an array of typed evidence claims \(factual error / no\-evidence / careless omission\), each with a confidence score and the supporting paper quote\.Correctemits per\-error classifications with root\-cause descriptors\.Completeemits a boolean value per criticism plus a count of actionable suggestions\.Transformemits a bias\-type label per offending segment plus a boolean value for whether any valid academic concern is also present\. Full schemas are given in Appendix[A](https://arxiv.org/html/2605.26911#A1)\.

### 5\.3Orchestration policy

The orchestrator follows a dynamic tool\-calling strategy driven by its thinking mode\. It first decomposes the review into evidence segments—factual claims, critical opinions, attitude expressions—and then selects which tool to invoke next based on the segment types identified and the outputs already returned by prior tools\. The recommended orderVerify→Correct→Complete→Transform\\textsc\{Verify\}\\\!\\to\\\!\\textsc\{Correct\}\\\!\\to\\\!\\textsc\{Complete\}\\\!\\to\\\!\\textsc\{Transform\}is given in the prompt as a heuristic starting point, but the orchestrator adapts it: it may short\-circuitCorrectwhenVerifyfinds no factual errors, revisit a tool when later evidence surfaces, or skip a tool entirely when no relevant evidence segments exist\.

##### Reliability\.

We enforce a hard rule in code rather than relying on prompts:Integrateruns*exactly once*per review: if the orchestrator emits a final output without having calledIntegrate, the runtime rejects the output and requires regeneration\.

### 5\.4Two\-stage semi\-supervised training ofIntegrate

TheIntegratemodule is the only fine\-tuned component\. We adopt a two\-stage semi\-supervised procedure that mixes the human\-gold\-annotated training samples \(𝒟gold\\mathcal\{D\}\_\{\\text\{gold\}\}\) with persona\-consistent pseudo\-labeled samples \(𝒟pseudo\\mathcal\{D\}\_\{\\text\{pseudo\}\}\)\.

##### Stage 1 \(supervised\)\.

We fine\-tuneIntegratewith LoRA on𝒟gold\\mathcal\{D\}\_\{\\text\{gold\}\}\. Each training example uses, as input, the paper representation \(main text, references, appendix summary, figure descriptions\), the review text, and the JSON outputs of the four analysis tools\. The supervision target is a standardized JSON containing the binary labelz^\\hat\{z\}, a five\-point quality score, and the multi\-label vectory^\\hat\{y\}, derived from the expert multi\-label annotation\.

##### Stage 2 \(semi\-supervised continuation\)\.

We use the Stage\-1 checkpoint to pseudo\-label the weakly\-supervised synthetic reviews from𝒟pseudo\\mathcal\{D\}\_\{\\text\{pseudo\}\}, retaining only those samples whose Stage\-1 predicted defect matches the persona’s weak label\. That is, for deficient personas, the predicted defect type list contains the persona’s target defect type; for competent personas, the prediction is “non\-deficient”\. This consistency filter removes pseudo\-labels that disagree with the personas, while permitting multi\-label expansions of the persona’s single tag \(the Stage\-1 model may identify additional defects\)\.

##### Inference\.

At inference time,Integrateis run with the thinking mode disabled to reduce latency, while the orchestrator and the four analysis tools use thinking mode\.Integrateemits a JSON object containingz^∈\{0,1\}\\hat\{z\}\\in\\\{0,1\\\}, a five\-point quality scoreqq, and the sorted multi\-label vectory^\\hat\{y\}\. This classification is returned to the orchestrator, which then emits the final user\-facing JSON containing\(z^,y^,q\)\(\\hat\{z\},\\hat\{y\},q\), per\-defect explanations and evidence traces, and review\-level revision suggestions\.

FamilyModelAccPrecisionRecallF1PLMBERT\-base\-uncased77\.33%73\.08%39\.05%0\.5090BigBird\-RoBERTa\-base46\.00%33\.08%69\.52%0\.4483Longformer\-base\-409683\.33%88\.08%52\.86%0\.6607Few\-shot LLMDeepSeek V3\.249\.78%87\.31%24\.76%0\.3858Qwen3\.5\-Plus55\.33%74\.23%45\.71%0\.5658Qwen3\-30B54\.44%68\.85%51\.90%0\.5919GLM 4\.761\.56%73\.46%61\.43%0\.6691Review\-qualitydetectorReviewGuard \(Llama\-3\.1\-8B\)66\.00%46\.92%73\.81%0\.5737ReviewGuard \(Qwen3\-8B\)63\.33%43\.08%57\.14%0\.4912RottenReviews \(LR\)68\.00%69\.23%95\.24%0\.8018RottenReviews \(Random Forest\)58\.89%72\.31%66\.19%0\.6911RottenReviews \(MLP\)31\.11%0\.00%0\.00%0\.0000RottenReviews \(XGBoost\)60\.22%74\.62%61\.90%0\.6767RottenReviews \(Qwen3\-8B 0\-shot\)31\.11%0\.00%0\.00%0\.0000RottenReviews \(Llama\-3\-8B FT\)65\.33%71\.15%82\.38%0\.7636OursTADDLE\(Qwen3\-30B\-A3B \+ 9B integrator\)86\.00%91\.54%74\.76%0\.8230

Table 1:Binary deficiency detection on the gold\-labeled ICLR test set\. Best in bold\.

## 6Experiments

### 6\.1Experimental settings

##### Datasets\.

We report results on three settings: \(a\) the gold\-labeled ICLR test set \(9090reviews, distribution\-shifted in both papers and generator; see §[4\.3](https://arxiv.org/html/2605.26911#S4.SS3)\); \(b\) cross\-conference weak\-supervision sets \(∼900\\sim\\\!900reviews each, from ICML20252025and NeurIPS20252025papers\); and \(c\) cross\-generator weak\-supervision sets \(∼900\\sim\\\!900reviews each, from Llama 3\.3\-70B and MiniMax\-M2\.5\)\.

##### Reproducibility\.

All main experiments are averaged over five independent generation runs of the test\-set review pool \(different decoding seeds at the review\-generation stage\)\. The class\-imbalance robustness study is averaged over2020random resamples of the test set at each target proportion\.

##### Baselines\.

We compare our method with 9 baselines across three families: pre\-trained language models \(PLMs\), few\-shot LLMs and domain\-specific baselines\. Specifically, PLMs include BERT\-base\-uncasedDevlinet al\.\([2019](https://arxiv.org/html/2605.26911#bib.bib20)\), BigBird\-RoBERTa\-baseZaheeret al\.\([2020](https://arxiv.org/html/2605.26911#bib.bib21)\)and Longformer\-base\-4096Beltagyet al\.\([2020](https://arxiv.org/html/2605.26911#bib.bib22)\)\. Few\-shot LLMs include DeepSeek V3\.2DeepSeek\-AI \([2025](https://arxiv.org/html/2605.26911#bib.bib23)\), Qwen3\.5\-PlusQwen Team \([2026](https://arxiv.org/html/2605.26911#bib.bib24)\), Qwen3\-30BTeam \([2025](https://arxiv.org/html/2605.26911#bib.bib25)\)and GLM 4\.7Teamet al\.\([2025](https://arxiv.org/html/2605.26911#bib.bib26)\)\. Domain\-specific baselines include ReviewGuardZhanget al\.\([2025](https://arxiv.org/html/2605.26911#bib.bib14)\), with22different backbones, and RottenReviewsEbrahimiet al\.\([2025](https://arxiv.org/html/2605.26911#bib.bib19)\), with66variants\. Due to the space limitation, details on these baselines are given in Appendix[C](https://arxiv.org/html/2605.26911#A3)\.

##### Implementation details\.

TheIntegratemodule is fine\-tuned on a single NVIDIA A1008080GB GPU using LLaMA\-FactoryZhenget al\.\([2024](https://arxiv.org/html/2605.26911#bib.bib27)\)\(v0\.9\.5\); inference uses vLLMKwonet al\.\([2023](https://arxiv.org/html/2605.26911#bib.bib28)\)\(v0\.19\.1\)\. The orchestrator and the four analysis tools are run via vLLM with the Qwen3\-30B\-A3B\-Thinking\-2507 base model in thinking mode;Integrateruns with thinking mode disabled\. The orchestrator’s paper\-content exposure is set to “abstract only” in all reported runs\. The exact\-onceIntegrateinvocation is enforced via a regeneration loop with at most44retries per tool call; in practice<1%<\\\!1\\%of reviews required more than one regeneration\. Detailed hyperparameter settings are provided in Appendix[B](https://arxiv.org/html/2605.26911#A2)\.

### 6\.2Binary detection on the gold ICLR test set

Table[1](https://arxiv.org/html/2605.26911#S5.T1)reports binary detection results, averaged over five runs\.TADDLE\(bottom row\) obtains the highest Accuracy \(86\.00%86\.00\\%\), Precision \(91\.54%91\.54\\%\), and F1 \(0\.82300\.8230\), exceeding the strongest baseline, RottenReviews\-LR \(F10\.80180\.8018\), by2\.122\.12F1 points\. From the table, we further observe that:

##### PLM baselines are recall\-limited\.

Longformer reaches the best F1 among PLMs \(0\.66070\.6607\), but its Recall is only52\.86%52\.86\\%\. The pattern is consistent across BERT and BigBird: long\-context encoder fine\-tuning alone is insufficient to surface fine\-grained defect signals, especially when the deficient signal lives in the relationship between two long inputs rather than in surface lexical features\.

##### Few\-shot LLMs are systematically over\-conservative\.

DeepSeek V3\.2 achieves87\.31%87\.31\\%Precision but only24\.76%24\.76\\%Recall—it correctly flags some egregious reviews but misses three\-quarters of deficient samples\. GLM 4\.7 is the strongest few\-shot LLM \(0\.66910\.6691F1\) but still trailsTADDLEby1515F1 points\. We interpret this as evidence that strong LLMs, without supervised calibration to a defect taxonomy, default to charitable interpretations and require explicit verification scaffolding to escape that prior\.

##### Domain\-specific baselines trade Precision for Recall\.

RottenReviews\-LR achieves the highest Recall \(95\.24%95\.24\\%\), but its Accuracy \(68\.00%68\.00\\%\) and Precision \(69\.23%69\.23\\%\) reveal that this comes from a near\-all\-positive prediction bias: it labels most samples deficient\. In deployment, where the proportion of deficient reviews is unlikely to be as high as in our balanced test set, such a model would produce an unacceptable false\-positive rate\. Two RottenReviews variants \(MLP, Qwen3\-8B 0\-shot\) collapse to all\-negative predictions and report zero Recall\.

FamilyModelCross\-conferenceCross\-generatorICMLNeurIPSLlama 3\.3\-70BMiniMax\-M2\.5PLMBERT\-base\-uncased0\.2100\.2010\.2090\.212BigBird\-RoBERTa\-base0\.2850\.2420\.0240\.057Longformer\-base\-40960\.2890\.2770\.3440\.227Few\-shot LLMDeepSeek\-V3\.20\.2550\.2350\.2290\.115GLM\-4\.70\.4350\.3660\.2090\.214Qwen3\-30B0\.4370\.3930\.3640\.264Qwen3\.5\-Plus0\.4260\.3870\.4150\.222Review\-qualitydetectorReviewGuard \(Llama\-3\.1\-8B\)0\.2860\.3100\.0590\.180ReviewGuard \(Qwen3\-8B\)0\.2560\.2830\.0710\.119RottenReviews \(LR\)0\.0580\.0570\.0300\.133RottenReviews \(Random Forest\)0\.2160\.206−0\.049\-0\.0490\.085RottenReviews \(MLP\)−0\.004\-0\.0040\.0030\.000−0\.004\-0\.004RottenReviews \(XGBoost\)0\.1890\.216−0\.060\-0\.0600\.063RottenReviews \(Qwen3\-8B 0\-shot\)0\.0000\.0000\.0440\.009RottenReviews \(Llama\-3\-8B FT\)0\.3520\.3670\.0060\.180OursTADDLE0\.4530\.4180\.4270\.281

Table 2:Persona\-separabilityΔ=RL−RH\\Delta=R\_\{L\}\\\!\-\\\!R\_\{H\}on the four distribution\-shift evaluation sets \(higher is better\)\.

### 6\.3Fine\-grained multi\-label classification

Table[3](https://arxiv.org/html/2605.26911#S6.T3)reports the multi\-label results\.TADDLEis best on all four metrics, with Jaccard0\.74240\.7424, Micro F10\.48280\.4828, Macro F10\.43320\.4332, and Weighted F10\.50090\.5009\. Domain\-specific baselines are excluded from this table because they do not predict matching defect types\. From the table, we see that:

##### PLMs show high Jaccard but extremely low F1\.

This counterintuitive pattern reflects a degeneracy: PLMs typically predict the empty label set, which trivially matches all truly non\-deficient samples \(68\.9%68\.9\\%of the test set\) and yields a Jaccard of11on those samples while contributing zero F1 on the deficient ones\. The aggregate Jaccard is then inflated by the non\-deficient majority class\. Macro and Micro F1 are robust to this and correctly reveal the PLM ceiling\.

##### Few\-shot LLMs attempt typed prediction but are noisy\.

The four few\-shot LLM baselines have lower Jaccard \(0\.360\.36–0\.400\.40\) but substantially higher Macro F1 \(0\.280\.28–0\.320\.32\) than PLMs: they do attempt to predict specific defect types, but produce noisy multi\-label outputs that reduce the label\-set intersection\-over\-union while contributing some non\-trivial per\-class recall\.

ModelJaccardMicro F1Macro F1Weighted F1BERT\-base\-uncased0\.66670\.09300\.03980\.0744BigBird\-RoBERTa\-base0\.30680\.09640\.05010\.0712Longformer\-base\-40960\.71110\.17780\.06980\.1776DeepSeek V3\.20\.40300\.24360\.30020\.2342Qwen3\.5\-Plus0\.39540\.33520\.30540\.2900Qwen3\-30B0\.35650\.34940\.27980\.2893GLM 4\.70\.40220\.35980\.31630\.3308TADDLE0\.74240\.48280\.43320\.5009

Table 3:Fine\-grained multi\-label classification on the gold\-labeled ICLR test set\.VariantTool useExternal retrievalIntegratorStage 2 FTAccPrecisionRecallF1TADDLE, w/o tool chainoffN/Afine\-tuned✓52\.67%52\.69%50\.95%0\.5181TADDLE, w/oCompleteonenablefine\-tuned✓85\.33%77\.31%72\.86%0\.7502TADDLE, w/oCorrectonenablefine\-tuned✓86\.44%82\.69%71\.43%0\.7665TADDLE, w/oTransformonenablefine\-tuned✓82\.22%75\.00%62\.38%0\.6811TADDLE, w/oVerifyonenablefine\-tuned✓86\.22%76\.54%72\.38%0\.7440TADDLE, no external retrievalondisablefine\-tuned✓80\.89%72\.69%65\.71%0\.6903TADDLE, base 9B integrator \(no FT\)onenableQwen3\.5\-9B✓38\.44%31\.54%92\.86%0\.4708TADDLE, Stage 1 onlyonenablefine\-tuned×85\.56%80\.00%70\.48%0\.7494TADDLE\(full\)onenablefine\-tuned✓86\.00%91\.54%74\.76%0\.8230

Table 4:Ablation ofTADDLEon the gold\-labeled ICLR test set\. Stage 2 FT = Stage\-2 fine\-tuning\.

### 6\.4Generalization under distribution shift

For evaluating systems on distribution\-shifted sets where gold labels are unavailable, we define a weak\-supervision metric called persona\-separabilityΔ=RL−RH\\Delta=R\_\{L\}\-R\_\{H\}\. HereRLR\_\{L\}andRHR\_\{H\}denote the average predicted\-deficient rate over reviews written under the deficient persona and the competent persona, respectively\. The value ofΔ\\Deltafalls in\[−1,1\]\[\-1,1\], with higher scores indicating better average separation between the two persona pools\. Crucially,Δ\\Deltaserves as a proxy: it captures the system\-level ability to discriminate the personas but does not equal per\-sample F1\. We therefore employΔ\\Deltaexclusively for relative comparisons among systems evaluated on the same shifted dataset, rather than as a replacement for in\-domain F1\. Table[2](https://arxiv.org/html/2605.26911#S6.T2)reportsΔ\\Deltaon the four distribution\-shift sets\.TADDLEobtains the highestΔ\\Deltain all four columns \(ICML:0\.4530\.453; NeurIPS:0\.4180\.418; Llama 3\.3\-70B:0\.4270\.427; MiniMax\-M2\.5:0\.2810\.281\)\. From the table, we further observe that:

##### Cross\-conference \(ICML, NeurIPS\)\.

Strong few\-shot LLMs remain competitive \(Qwen3\-30B:0\.437/0\.3930\.437/0\.393; GLM\-4\.7:0\.435/0\.3660\.435/0\.366\)\. RottenReviews\-LR, the highest\-Recall in\-domain baseline, collapses to0\.058/0\.0570\.058/0\.057: its discriminative signal relies on distribution\-specific statistics of the ICLR training pool and does not transfer\. This is consistent with the in\-domain finding that its high Recall stems from an all\-positive prediction bias rather than from genuine defect understanding\.

##### Cross\-generator \(Llama 3\.3\-70B, MiniMax\-M2\.5\)\.

Cross\-generator shift is harder for all systems; several domain\-specific baselines produce negativeΔ\\Deltaon Llama 3\.3\-70B \(RottenReviews\-RF:−0\.049\-0\.049; XGBoost:−0\.060\-0\.060\), meaning they label competent\-persona reviews as deficient at a higher rate than deficient\-persona ones\. We interpret this as evidence that these methods have learned generator\-specific patterns \(e\.g\., MiniMax’s more compact phrasing, Llama’s tendency to use bulleted lists\) and are detecting the generator, not the defect\.TADDLEretainsΔ=0\.427/0\.281\\Delta=0\.427/0\.281on the same sets, suggesting that paper\-grounded verification provides stronger generator\-agnostic discrimination than generator\-stylistic patterns\.

Beyond cross\-conference and cross\-generator shifts, we further evaluate model robustness under class\-prior shift—varying the proportion of deficient reviews in the test set—and find thatTADDLEremains the most stable across imbalance regimes \(results are provided in Appendix[D](https://arxiv.org/html/2605.26911#A4)\)\.

### 6\.5Ablation study

Table[4](https://arxiv.org/html/2605.26911#S6.T4)reports single\-variable ablations on the gold\-labeled ICLR test set\. Each row changes one component\. From the table, we observe that:

##### Tool decomposition is the dominant driver\.

Removing the entire tool chain and fine\-tuningIntegratedirectly on paper–review pairs drops F1 from0\.82300\.8230to0\.51810\.5181\(−0\.3049\-0\.3049\)\. This is the largest single drop in the table and indicates that the structured tool evidence—not fine\-tuning of the integrator on its own—is what enables accurate detection\. We highlight that this comparison preserves the use of a fine\-tuned integrator, so the gap is attributable to tool evidence rather than to model size or supervision\.

##### Each tool contributes\.

Every single\-tool ablation degrades F1\.Transformcauses the largest drop \(−0\.1419\-0\.1419\), reflecting that bias and tone signals are common across deficient reviews\.VerifyandCompletecontribute−0\.079\-0\.079and−0\.073\-0\.073\.Correct’s removal yields the smallest absolute F1 drop \(−0\.057\-0\.057\), but the accompanying drop in Recall \(74\.76%→71\.43%74\.76\\%\\\!\\to\\\!71\.43\\%\) indicates thatCorrecthelps the integrator distinguish genuine errors from false alarms surfaced byVerify, recovering true positives that would otherwise be missed\.

##### External retrieval contributes substantially\.

Disabling Semantic Scholar / arXiv retrieval \(forVerifyandComplete\) drops F1 by0\.13270\.1327\. This is a non\-trivial effect for an architectural choice that consumes only two of the four tools’ calls; we interpret it as evidence that grounding factual and constructiveness judgments in external literature is, by itself, a strong signal that the in\-paper evidence cannot fully replace\.

##### Integrator fine\-tuning is necessary\.

Replacing the fine\-tuned integrator with the base Qwen3\.5\-9B \(with the same prompt\) collapses the system toward all\-positive predictions \(Precision31\.54%31\.54\\%, Recall92\.86%92\.86\\%, F10\.4710\.471\)\. The base model can read the tool evidence but cannot calibrate its decision threshold to the human label distribution\. LoRA fine\-tuning provides this calibration with very few updated parameters\.

##### Stage 2 pseudo\-labels help\.

Stage 1 alone yields F10\.74940\.7494; the addition of Stage\-2 persona\-consistent pseudo\-labels lifts this to0\.82300\.8230\(\+0\.0736\+0\.0736\)\. This is consistent with the hypothesis that the consistency filter retains pseudo\-labels of high enough quality to act as a useful augmentation, while filtering noise from LLM–persona slippage\.

## 7Conclusion

We studied the problem of detecting deficient LLM\-generated reviews and proposedTADDLE, a tool\-augmented agent\-based system that reframes detection as an explicit analysis–verification–decision pipeline\.TADDLEemploys an orchestrator to route review segments through specialized analysis tools, and a fine\-tuned integration module to synthesize the analyses into final decisions\. Extensive experiments on a newly constructed benchmark demonstrate thatTADDLEsignificantly outperforms strong baselines and maintains its advantage under multiple distribution shifts\.

## Limitations

##### The benchmark is generated by LLMs and annotated by humans\.

Our benchmark consists of LLM\-generated reviews that are multi\-label\-annotated by domain experts against the original review text\. While the labels are grounded in human judgment, the review distribution is shaped by six generator LLMs and nine persona prompts\. The personas are intended as a coverage mechanism rather than a realistic model of organic AI\-assisted reviewing, and annotators may also rely on subtle persona\-related artifacts when assigning labels\. We partially mitigate these issues through cross\-generator and distribution\-shift evaluations, though these do not fully eliminate the concern\. More broadly, organically produced AI\-assisted reviews may differ systematically from persona\-conditioned ones\. We therefore view this benchmark as an initial step and leave validation on real\-world AI\-assisted reviews to future work\.

##### Taxonomy and venue scope\.

The defect taxonomy is derived from LLM\-generated reviews from 2025; as LLM capabilities evolve, new deficiency patterns may emerge that are not covered by the current six categories\. Likewise, the benchmark covers three ML venues; whether the same taxonomy transfers to other discipl6ines \(e\.g\., medical, legal, social science peer review\) is an open question\.

## Ethical Considerations

All annotators were recruited with informed consent and compensated at rates above local minimum wage\. Detailed annotation guidelines are in Appendix[E](https://arxiv.org/html/2605.26911#A5)\. The synthetic reviews in our dataset are produced strictly for research and are not submitted to actual conferences\. We will release the benchmark under a license that prohibits its use for training systems designed to produce deceptive academic content\.TADDLEis intended as a quality\-assurance tool to assist—never to replace—human area chairs; final editorial decisions should remain with qualified human reviewers\. We acknowledge a misuse risk: any detector for defects is also a detector for “defect signals to avoid”, and an adversary could in principle train against our system to produce harder\-to\-detect deficient reviews\. We believe the editorial\-safety benefit outweighs this risk in the current LLM landscape, but the tradeoff should be revisited as detection methods improve\.

Additionally, for the pre\-trained models \(DeepSeek V3\.2DeepSeek\-AI \([2025](https://arxiv.org/html/2605.26911#bib.bib23)\), Qwen3\.5\-PlusQwen Team \([2026](https://arxiv.org/html/2605.26911#bib.bib24)\), Qwen3\-30BTeam \([2025](https://arxiv.org/html/2605.26911#bib.bib25)\), GLM 4\.7Teamet al\.\([2025](https://arxiv.org/html/2605.26911#bib.bib26)\), Kimi K2\.5Teamet al\.\([2026](https://arxiv.org/html/2605.26911#bib.bib32)\), Llama 3\.3\-70BGrattafioriet al\.\([2024](https://arxiv.org/html/2605.26911#bib.bib33)\)and MiniMax\-M2\.5MiniMax \([2026](https://arxiv.org/html/2605.26911#bib.bib34)\)\) and frameworks utilized \(LLaMA\-FactoryZhenget al\.\([2024](https://arxiv.org/html/2605.26911#bib.bib27)\)and vLLMKwonet al\.\([2023](https://arxiv.org/html/2605.26911#bib.bib28)\)\) , we adhered to their respective open\-source licenses\. The data used to construct the benchmark comply with the usage licenses of ICLR, ICML, NeurIPS, and OpenReview for public papers and reviews, and do not contain any personally identifying information or offensive content\.

## References

- Peer grading the peer reviews: a dual\-role approach for lightening the scholarly paper review process\.InProceedings of the Web Conference 2021,pp\. 1916–1927\.Cited by:[§1](https://arxiv.org/html/2605.26911#S1.p3.1),[§2](https://arxiv.org/html/2605.26911#S2.SS0.SSS0.Px2.p1.1)\.
- I\. Beltagy, M\. E\. Peters, and A\. Cohan \(2020\)Longformer: the long\-document transformer\.arXiv preprint arXiv:2004\.05150\.Cited by:[Appendix C](https://arxiv.org/html/2605.26911#A3.SS0.SSS0.Px1.p1.1),[§6\.1](https://arxiv.org/html/2605.26911#S6.SS1.SSS0.Px3.p1.2)\.
- A\. Chung, Y\. Zhang, K\. Lin, A\. Rawal, Q\. Gao, and J\. Chai \(2025\)Evaluating long\-context reasoning in llm\-based webagents\.arXiv preprint arXiv:2512\.04307\.Cited by:[§5\.1](https://arxiv.org/html/2605.26911#S5.SS1.SSS0.Px1.p1.1)\.
- C\. Cui, T\. Sun, S\. Liang, T\. Gao, Z\. Zhang, J\. Liu, X\. Wang, C\. Zhou, H\. Liu, M\. Lin, Y\. Zhang, Y\. Zhang, H\. Zheng, J\. Zhang, J\. Zhang, Y\. Liu, D\. Yu, and Y\. Ma \(2025\)PaddleOCR\-vl: boosting multilingual document parsing via a 0\.9b ultra\-compact vision\-language model\.External Links:2510\.14528,[Link](https://arxiv.org/abs/2510.14528)Cited by:[§4\.1](https://arxiv.org/html/2605.26911#S4.SS1.p2.2)\.
- T\. Dao \(2024\)FlashAttention\-2: faster attention with better parallelism and work partitioning\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[Appendix B](https://arxiv.org/html/2605.26911#A2.p1.15)\.
- DeepSeek\-AI \(2025\)DeepSeek\-v3\.2: pushing the frontier of open large language models\.Cited by:[Appendix C](https://arxiv.org/html/2605.26911#A3.SS0.SSS0.Px2.p1.1),[§4\.2](https://arxiv.org/html/2605.26911#S4.SS2.p1.1),[§6\.1](https://arxiv.org/html/2605.26911#S6.SS1.SSS0.Px3.p1.2),[Ethical Considerations](https://arxiv.org/html/2605.26911#Sx2.p2.1)\.
- J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova \(2019\)Bert: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 \(long and short papers\),pp\. 4171–4186\.Cited by:[Appendix C](https://arxiv.org/html/2605.26911#A3.SS0.SSS0.Px1.p1.1),[§6\.1](https://arxiv.org/html/2605.26911#S6.SS1.SSS0.Px3.p1.2)\.
- J\. Du, Y\. Wang, W\. Zhao, Z\. Deng, S\. Liu, R\. Lou, H\. P\. Zou, P\. N\. Venkit, N\. Zhang, M\. Srinath,et al\.\(2024\)LLMs assist nlp researchers: critique paper \(meta\-\) reviewing\.InProceedings of the 2024 conference on empirical methods in natural language processing,pp\. 5081–5099\.Cited by:[§3](https://arxiv.org/html/2605.26911#S3.SS0.SSS0.Px2.p1.1)\.
- S\. Ebrahimi, S\. Sadeghian, A\. Ghorbanpour, N\. Arabzadeh, S\. Salamat, M\. Li, H\. S\. Le, M\. Bashari, and E\. Bagheri \(2025\)RottenReviews: benchmarking review quality with human and llm\-based judgments\.InProceedings of the 34th ACM International Conference on Information and Knowledge Management,pp\. 5642–5649\.Cited by:[Appendix C](https://arxiv.org/html/2605.26911#A3.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2605.26911#S1.p3.1),[§2](https://arxiv.org/html/2605.26911#S2.SS0.SSS0.Px3.p1.1),[§6\.1](https://arxiv.org/html/2605.26911#S6.SS1.SSS0.Px3.p1.2)\.
- X\. Gao, J\. Ruan, Z\. Zhang, J\. Gao, T\. Liu, and Y\. Fu \(2025\)Reviewagents: bridging the gap between human and ai\-generated paper reviews\.arXiv preprint arXiv:2503\.08506\.Cited by:[§2](https://arxiv.org/html/2605.26911#S2.SS0.SSS0.Px3.p1.1)\.
- G\. H\. T\. Go, K\. Ly, A\. Søgaard, S\. A\. Tabatabaei, M\. de Rijke, and X\. Chen \(2026\)Lira: a multi\-agent framework for reliable and readable literature review generation\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 40456–40464\.Cited by:[§2](https://arxiv.org/html/2605.26911#S2.SS0.SSS0.Px3.p1.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§4\.2](https://arxiv.org/html/2605.26911#S4.SS2.p1.1),[Ethical Considerations](https://arxiv.org/html/2605.26911#Sx2.p2.1)\.
- Y\. Jin, Q\. Zhao, Y\. Wang, H\. Chen, K\. Zhu, Y\. Xiao, and J\. Wang \(2024\)Agentreview: exploring peer review dynamics with llm agents\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 1208–1226\.Cited by:[§2](https://arxiv.org/html/2605.26911#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Kumar, S\. Garg, S\. Sengupta, T\. Ghosal, and A\. Ekbal \(2025\)MixRevDetect: towards detecting ai\-generated content in hybrid peer reviews\.\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 2: Short Papers\),pp\. 944–953\.Cited by:[§1](https://arxiv.org/html/2605.26911#S1.p3.1),[§2](https://arxiv.org/html/2605.26911#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Kumar, M\. Sahu, V\. Gacche, T\. Ghosal, and A\. Ekbal \(2024\)‘Quis custodiet ipsos custodes?’who will watch the watchmen? on detecting ai\-generated peer\-reviews\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 22663–22679\.Cited by:[§1](https://arxiv.org/html/2605.26911#S1.p3.1),[§2](https://arxiv.org/html/2605.26911#S2.SS0.SSS0.Px1.p1.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with pagedattention\.InProceedings of the 29th symposium on operating systems principles,pp\. 611–626\.Cited by:[§6\.1](https://arxiv.org/html/2605.26911#S6.SS1.SSS0.Px4.p1.3),[Ethical Considerations](https://arxiv.org/html/2605.26911#Sx2.p2.1)\.
- W\. L\. Lanier \(2021\)Dealing with inappropriate\-, low\-quality\-, and other forms of challenging peer review, including hostile referees and inflammatory or confusing critiques: prevention and treatment\.Accountability in Research28\(3\),pp\. 162–185\.Cited by:[§2](https://arxiv.org/html/2605.26911#S2.SS0.SSS0.Px2.p1.1)\.
- R\. Li, H\. Zhang, E\. Gehringer, T\. Xiao, J\. Ding, and H\. Chen \(2025\)Unveiling the merits and defects of llms in automatic review generation for scientific papers\.In2025 IEEE International Conference on Data Mining \(ICDM\),pp\. 1370–1379\.Cited by:[§1](https://arxiv.org/html/2605.26911#S1.p1.1),[§2](https://arxiv.org/html/2605.26911#S2.SS0.SSS0.Px2.p1.1)\.
- T\. Lin, W\. Chen, T\. Hsiao, H\. Liu, Y\. Yeh, Y\. K\. Chan, W\. Lien, P\. Kuo, P\. S\. Yu, and H\. Shuai \(2025\)Breaking the reviewer: assessing the vulnerability of large language models in automated peer review under textual adversarial attacks\.arXiv preprint arXiv:2506\.11113\.Cited by:[§1](https://arxiv.org/html/2605.26911#S1.p1.1)\.
- MiniMax \(2026\)MiniMax m2\.5\.Note:[https://www\.minimax\.io/models/text](https://www.minimax.io/models/text)Accessed: 2026\-05\-23Cited by:[§4\.2](https://arxiv.org/html/2605.26911#S4.SS2.p1.1),[Ethical Considerations](https://arxiv.org/html/2605.26911#Sx2.p2.1)\.
- Qwen Team \(2026\)Qwen3\.5: towards native multimodal agents\.External Links:[Link](https://qwen.ai/blog?id=qwen3.5)Cited by:[Appendix C](https://arxiv.org/html/2605.26911#A3.SS0.SSS0.Px2.p1.1),[§6\.1](https://arxiv.org/html/2605.26911#S6.SS1.SSS0.Px3.p1.2),[Ethical Considerations](https://arxiv.org/html/2605.26911#Sx2.p2.1)\.
- H\. Ryu, D\. Jang, H\. S\. Lee, J\. Jeong, G\. Kim, D\. Cho, G\. Chu, M\. Hwang, H\. Jang, C\. Kim,et al\.\(2025\)ReviewScore: misinformed peer review detection with large language models\.arXiv preprint arXiv:2509\.21679\.Cited by:[§1](https://arxiv.org/html/2605.26911#S1.p3.1),[§2](https://arxiv.org/html/2605.26911#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Sizo, A\. Lino, Á\. Rocha, and L\. P\. Reis \(2025\)Defining quality in peer review reports: a scoping review\.Knowledge and Information Systems67\(8\),pp\. 6413–6460\.Cited by:[§1](https://arxiv.org/html/2605.26911#S1.p3.1),[§2](https://arxiv.org/html/2605.26911#S2.SS0.SSS0.Px2.p1.1)\.
- G\. Team, A\. Zeng, X\. Lv, Q\. Zheng, Z\. Hou, B\. Chen, C\. Xie, C\. Wang, D\. Yin, H\. Zeng, J\. Zhang, K\. Wang, L\. Zhong, M\. Liu, R\. Lu, S\. Cao, X\. Zhang, X\. Huang, Y\. Wei, Y\. Cheng, Y\. An, Y\. Niu, Y\. Wen, Y\. Bai, Z\. Du, Z\. Wang, Z\. Zhu, B\. Zhang, B\. Wen, B\. Wu, B\. Xu, C\. Huang, C\. Zhao, C\. Cai, C\. Yu, C\. Li, C\. Ge, C\. Huang, C\. Zhang, C\. Xu, C\. Zhu, C\. Li, C\. Yin, D\. Lin, D\. Yang, D\. Jiang, D\. Ai, E\. Zhu, F\. Wang, G\. Pan, G\. Wang, H\. Sun, H\. Li, H\. Li, H\. Hu, H\. Zhang, H\. Peng, H\. Tai, H\. Zhang, H\. Wang, H\. Yang, H\. Liu, H\. Zhao, H\. Liu, H\. Yan, H\. Liu, H\. Chen, J\. Li, J\. Zhao, J\. Ren, J\. Jiao, J\. Zhao, J\. Yan, J\. Wang, J\. Gui, J\. Zhao, J\. Liu, J\. Li, J\. Li, J\. Lu, J\. Wang, J\. Yuan, J\. Li, J\. Du, J\. Du, J\. Liu, J\. Zhi, J\. Gao, K\. Wang, L\. Yang, L\. Xu, L\. Fan, L\. Wu, L\. Ding, L\. Wang, M\. Zhang, M\. Li, M\. Xu, M\. Zhao, M\. Zhai, P\. Du, Q\. Dong, S\. Lei, S\. Tu, S\. Yang, S\. Lu, S\. Li, S\. Li, Shuang\-Li, S\. Yang, S\. Yi, T\. Yu, W\. Tian, W\. Wang, W\. Yu, W\. L\. Tam, W\. Liang, W\. Liu, X\. Wang, X\. Jia, X\. Gu, X\. Ling, X\. Wang, X\. Fan, X\. Pan, X\. Zhang, X\. Zhang, X\. Fu, X\. Zhang, Y\. Xu, Y\. Wu, Y\. Lu, Y\. Wang, Y\. Zhou, Y\. Pan, Y\. Zhang, Y\. Wang, Y\. Li, Y\. Su, Y\. Geng, Y\. Zhu, Y\. Yang, Y\. Li, Y\. Wu, Y\. Li, Y\. Liu, Y\. Wang, Y\. Li, Y\. Zhang, Z\. Liu, Z\. Yang, Z\. Zhou, Z\. Qiao, Z\. Feng, Z\. Liu, Z\. Zhang, Z\. Wang, Z\. Yao, Z\. Wang, Z\. Liu, Z\. Chai, Z\. Li, Z\. Zhao, W\. Chen, J\. Zhai, B\. Xu, M\. Huang, H\. Wang, J\. Li, Y\. Dong, and J\. Tang \(2025\)GLM\-4\.5: agentic, reasoning, and coding \(arc\) foundation models\.External Links:2508\.06471,[Link](https://arxiv.org/abs/2508.06471)Cited by:[Appendix C](https://arxiv.org/html/2605.26911#A3.SS0.SSS0.Px2.p1.1),[§4\.2](https://arxiv.org/html/2605.26911#S4.SS2.p1.1),[§6\.1](https://arxiv.org/html/2605.26911#S6.SS1.SSS0.Px3.p1.2),[Ethical Considerations](https://arxiv.org/html/2605.26911#Sx2.p2.1)\.
- K\. Team, T\. Bai, Y\. Bai, Y\. Bao, S\. H\. Cai, Y\. Cao, Y\. Charles, H\. S\. Che, C\. Chen, G\. Chen, H\. Chen, J\. Chen, J\. Chen, J\. Chen, J\. Chen, K\. Chen, L\. Chen, R\. Chen, X\. Chen, Y\. Chen, Y\. Chen, Y\. Chen, Y\. Chen, Y\. Chen, Y\. Chen, Y\. Chen, Y\. Chen, Z\. Chen, Z\. Chen, D\. Cheng, M\. Chu, J\. Cui, J\. Deng, M\. Diao, H\. Ding, M\. Dong, M\. Dong, Y\. Dong, Y\. Dong, A\. Du, C\. Du, D\. Du, L\. Du, Y\. Du, Y\. Fan, S\. Fang, Q\. Feng, Y\. Feng, G\. Fu, K\. Fu, H\. Gao, T\. Gao, Y\. Ge, S\. Geng, C\. Gong, X\. Gong, Z\. Gongque, Q\. Gu, X\. Gu, Y\. Gu, L\. Guan, Y\. Guo, X\. Hao, W\. He, W\. He, Y\. He, C\. Hong, H\. Hu, J\. Hu, Y\. Hu, Z\. Hu, K\. Huang, R\. Huang, W\. Huang, Z\. Huang, T\. Jiang, Z\. Jiang, X\. Jin, Y\. Jing, G\. Lai, A\. Li, C\. Li, C\. Li, F\. Li, G\. Li, G\. Li, H\. Li, H\. Li, J\. Li, J\. Li, J\. Li, L\. Li, M\. Li, W\. Li, W\. Li, X\. Li, X\. Li, Y\. Li, Y\. Li, Y\. Li, Y\. Li, Z\. Li, Z\. Li, W\. Liao, J\. Lin, X\. Lin, Z\. Lin, Z\. Lin, C\. Liu, C\. Liu, H\. Liu, L\. Liu, S\. Liu, S\. Liu, S\. Liu, T\. Liu, T\. Liu, W\. Liu, X\. Liu, Y\. Liu, Y\. Liu, Y\. Liu, Y\. Liu, Y\. Liu, Z\. Liu, Z\. Liu, E\. Lu, H\. Lu, Z\. Lu, J\. Luo, T\. Luo, Y\. Luo, L\. Ma, Y\. Ma, S\. Mao, Y\. Mei, X\. Men, F\. Meng, Z\. Meng, Y\. Miao, M\. Ni, K\. Ouyang, S\. Pan, B\. Pang, Y\. Qian, R\. Qin, Z\. Qin, J\. Qiu, B\. Qu, Z\. Shang, Y\. Shao, T\. Shen, Z\. Shen, J\. Shi, L\. Shi, S\. Shi, F\. Song, P\. Song, T\. Song, X\. Song, H\. Su, J\. Su, Z\. Su, L\. Sui, J\. Sun, J\. Sun, T\. Sun, F\. Sung, Y\. Tai, C\. Tang, H\. Tang, X\. Tang, Z\. Tang, J\. Tao, S\. Teng, C\. Tian, P\. Tian, A\. Wang, B\. Wang, C\. Wang, C\. Wang, C\. Wang, D\. Wang, D\. Wang, D\. Wang, F\. Wang, H\. Wang, H\. Wang, H\. Wang, H\. Wang, H\. Wang, J\. Wang, J\. Wang, J\. Wang, K\. Wang, L\. Wang, Q\. Wang, S\. Wang, S\. Wang, S\. Wang, W\. Wang, X\. Wang, X\. Wang, Y\. Wang, Y\. Wang, Y\. Wang, Y\. Wang, Y\. Wang, Y\. Wang, Z\. Wang, Z\. Wang, Z\. Wang, Z\. Wang, Z\. Wang, Z\. Wang, C\. Wei, M\. Wei, C\. Wen, Z\. Wen, C\. Wu, H\. Wu, J\. Wu, R\. Wu, W\. Wu, Y\. Wu, Y\. Wu, Y\. Wu, Z\. Wu, C\. Xiao, J\. Xie, X\. Xie, Y\. Xie, Y\. Xin, B\. Xing, B\. Xu, J\. Xu, J\. Xu, J\. Xu, L\. H\. Xu, L\. Xu, S\. Xu, W\. Xu, X\. Xu, X\. Xu, Y\. Xu, Y\. Xu, Y\. Xu, Z\. Xu, Z\. Xu, J\. Yan, Y\. Yan, G\. Yang, H\. Yang, J\. Yang, K\. Yang, N\. Yang, R\. Yang, X\. Yang, X\. Yang, Y\. Yang, Y\. Yang, Y\. Yang, Z\. Yang, Z\. Yang, Z\. Yang, H\. Yao, D\. Ye, W\. Ye, Z\. Ye, B\. Yin, C\. Yu, L\. Yu, T\. Yu, T\. Yu, E\. Yuan, M\. Yuan, X\. Yuan, Y\. Yue, W\. Zeng, D\. Zha, H\. Zhan, D\. Zhang, H\. Zhang, J\. Zhang, P\. Zhang, Q\. Zhang, R\. Zhang, X\. Zhang, Y\. Zhang, Y\. Zhang, Y\. Zhang, Y\. Zhang, Y\. Zhang, Y\. Zhang, Y\. Zhang, Y\. Zhang, Y\. Zhang, Y\. Zhang, Z\. Zhang, C\. Zhao, F\. Zhao, J\. Zhao, S\. Zhao, X\. Zhao, Y\. Zhao, Z\. Zhao, H\. Zheng, R\. Zheng, S\. Zheng, T\. Zheng, J\. Zhong, L\. Zhong, W\. Zhong, M\. Zhou, R\. Zhou, X\. Zhou, Z\. Zhou, J\. Zhu, L\. Zhu, X\. Zhu, Y\. Zhu, Z\. Zhu, J\. Zhuang, W\. Zhuang, Y\. Zou, and X\. Zu \(2026\)Kimi k2\.5: visual agentic intelligence\.External Links:2602\.02276,[Link](https://arxiv.org/abs/2602.02276)Cited by:[§4\.2](https://arxiv.org/html/2605.26911#S4.SS2.p1.1),[Ethical Considerations](https://arxiv.org/html/2605.26911#Sx2.p2.1)\.
- Q\. Team \(2025\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[Appendix C](https://arxiv.org/html/2605.26911#A3.SS0.SSS0.Px2.p1.1),[§4\.2](https://arxiv.org/html/2605.26911#S4.SS2.p1.1),[§6\.1](https://arxiv.org/html/2605.26911#S6.SS1.SSS0.Px3.p1.2),[Ethical Considerations](https://arxiv.org/html/2605.26911#Sx2.p2.1)\.
- K\. Tran, D\. Dao, M\. Nguyen, Q\. Pham, B\. O’Sullivan, and H\. D\. Nguyen \(2025\)Multi\-agent collaboration mechanisms: a survey of llms\.arXiv preprint arXiv:2501\.06322\.Cited by:[§2](https://arxiv.org/html/2605.26911#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. R\. Narasimhan, and Y\. Cao \(2022\)React: synergizing reasoning and acting in language models\.InThe eleventh international conference on learning representations,Cited by:[§2](https://arxiv.org/html/2605.26911#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Yu, M\. Luo, A\. Madasu, V\. Lal, and P\. Howard \(2024\)Is your paper being reviewed by an llm? investigating ai text detectability in peer review\.arXiv preprint arXiv:2410\.03019\.Cited by:[§1](https://arxiv.org/html/2605.26911#S1.p3.1),[§2](https://arxiv.org/html/2605.26911#S2.SS0.SSS0.Px1.p1.1)\.
- M\. Zaheer, G\. Guruganesh, K\. A\. Dubey, J\. Ainslie, C\. Alberti, S\. Ontanon, P\. Pham, A\. Ravula, Q\. Wang, L\. Yang,et al\.\(2020\)Big bird: transformers for longer sequences\.Advances in neural information processing systems33,pp\. 17283–17297\.Cited by:[Appendix C](https://arxiv.org/html/2605.26911#A3.SS0.SSS0.Px1.p1.1),[§6\.1](https://arxiv.org/html/2605.26911#S6.SS1.SSS0.Px3.p1.2)\.
- H\. Zhang, R\. Li, S\. Shrestha, S\. H\. Mamidala, R\. Putta, A\. K\. Aggarwal, T\. Xiao, J\. Ding, and H\. Chen \(2025\)Reviewguard: enhancing deficient peer review detection via llm\-driven data augmentation\.In2025 ACM/IEEE Joint Conference on Digital Libraries \(JCDL\),pp\. 207–216\.Cited by:[Appendix C](https://arxiv.org/html/2605.26911#A3.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2605.26911#S1.p3.1),[§2](https://arxiv.org/html/2605.26911#S2.SS0.SSS0.Px3.p1.1),[§6\.1](https://arxiv.org/html/2605.26911#S6.SS1.SSS0.Px3.p1.2)\.
- Y\. Zheng, R\. Zhang, J\. Zhang, Y\. Ye, Z\. Luo, Z\. Feng, and Y\. Ma \(2024\)LlamaFactory: unified efficient fine\-tuning of 100\+ language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 3: System Demonstrations\),Bangkok, Thailand\.External Links:[Link](http://arxiv.org/abs/2403.13372)Cited by:[§6\.1](https://arxiv.org/html/2605.26911#S6.SS1.SSS0.Px4.p1.3),[Ethical Considerations](https://arxiv.org/html/2605.26911#Sx2.p2.1)\.
- H\. Zhou, J\. Zhu, Y\. Yang, and C\. Shi \(2026\)Detecting llm\-generated text with performance guarantees\.arXiv preprint arXiv:2601\.06586\.Cited by:[§1](https://arxiv.org/html/2605.26911#S1.p3.1),[§2](https://arxiv.org/html/2605.26911#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix ATool schemas

We present all tool schemas in Table[5](https://arxiv.org/html/2605.26911#A3.T5); for the detailed JSON structures and prompts, please refer to Appendix[H\.5](https://arxiv.org/html/2605.26911#A8.SS5)\.

## Appendix BHyperparameters

In Stage 1, we trained with a learning rate of1​e−41\\mathrm\{e\}\{\-4\}under a cosine schedule with a warmup ratio of0\.10\.1, for33epochs\. We used a per\-device batch size of11with gradient accumulation of44, bf16 precision, and FlashAttention\-2Dao \([2024](https://arxiv.org/html/2605.26911#bib.bib29)\), with a maximum input length of40,96040\{,\}960tokens\. For Stage 2, we grid\-searched the learning rate over\{1,2,5\}×10−6\\\{1,2,5\\\}\\\!\\times\\\!10^\{\-6\}, selecting2​e−62\\mathrm\{e\}\{\-6\}as the best value, and trained for11epoch with a per\-device batch size of22and gradient accumulation of88under a cosine schedule\. Any hyperparameters not explicitly mentioned for Stage 2 follow the same settings as in Stage 1\. For inference, we sampled with a temperature of0\.10\.1, top\-ppof0\.950\.95, and a maximum output length of1,0241\{,\}024tokens\.

## Appendix CBaselines

We compare against three baseline families, retrained or re\-prompted on our task\.

##### Pre\-trained language models \(PLMs\)\.

BERT\-base\-uncasedDevlinet al\.\([2019](https://arxiv.org/html/2605.26911#bib.bib20)\),BigBird\-RoBERTa\-baseZaheeret al\.\([2020](https://arxiv.org/html/2605.26911#bib.bib21)\), andLongformer\-base\-4096Beltagyet al\.\([2020](https://arxiv.org/html/2605.26911#bib.bib22)\)\. BigBird and Longformer handle long inputs via sparse and sliding\-window attention, respectively, making them appropriate for joint paper–review inputs\. All three are fine\-tuned end\-to\-end on our training set for both binary and fine\-grained multi\-label classification\.

##### Few\-shot LLMs\.

DeepSeek V3\.2DeepSeek\-AI \([2025](https://arxiv.org/html/2605.26911#bib.bib23)\),Qwen3\.5\-Plus\(Qwen Team,[2026](https://arxiv.org/html/2605.26911#bib.bib24)\),Qwen3\-30BTeam \([2025](https://arxiv.org/html/2605.26911#bib.bib25)\), andGLM 4\.7Teamet al\.\([2025](https://arxiv.org/html/2605.26911#bib.bib26)\)prompted with one labeled example per class \(77demonstrations: one non\-deficient and one per defect type\)\. No fine\-tuning\. This isolates the contribution of theTADDLEarchitecture from the choice of strong open\-weight LLM\.

##### Domain\-specific baselines\.

ReviewGuardZhanget al\.\([2025](https://arxiv.org/html/2605.26911#bib.bib14)\), originally developed for human\-written reviews, retrained on our LLM\-review training set with Llama\-3\.1\-8B and Qwen3\-8B backbones to give it the best chance on our task\.RottenReviewsEbrahimiet al\.\([2025](https://arxiv.org/html/2605.26911#bib.bib19)\), likewise originally targeting human\-authored reviews, retrained on our training set across its LR, RF, MLP, XGBoost, and Llama\-3\-8B\-Finetune variants, plus a Qwen3\-8B\-0shot reference variant from the original paper\. Retraining both systems on our data isolates the effect of their architectural assumptions from the effect of training distribution\.

ToolFunction and key I/O / capability differencesVerifyFact\-checking: detects factual errors, unsupported assertions, and careless omissions; typed evidence markers\. Reads full paper; external retrieval enabled\.CorrectClassifies errors detected byVerifyinto information\-, careless\-, or misunderstanding\-type, and flags unprofessional errors\. ReadsVerifyoutput \+ full paper \(when evidence segments are needed\)\.CompleteConstructiveness evaluation: checks whether each criticism is paired with an actionable revision direction\. Reads full paper; external retrieval enabled\.TransformBias and tone detection: subjective bias, hostile phrasing, double standards\. Reads review \+ full paper \(when relevant paper segments are needed\)\.IntegrateFinal decision: synthesizes the four analyses intoz^\\hat\{z\}andy^\\hat\{y\}; fine\-tuned \(LoRA\)\. Sole module with decision authority\.Table 5:The four analysis tools and the integrator inTADDLE\.FamilyModel25%50%75%B\.AccG\-MeanMCCB\.AccG\-MeanMCCB\.AccG\-MeanMCCPLMBERT\-base\-uncased0\.6220\.5600\.2840\.5710\.4980\.1710\.5690\.5130\.133BigBird\-RoBERTa\-base0\.5720\.5580\.1270\.6330\.6210\.2750\.6110\.5980\.207Longformer\-base\-40960\.7670\.7300\.6790\.7420\.6950\.5650\.7420\.6950\.435Few\-shot LLMDeepSeek\-V3\.20\.5980\.5030\.2540\.6530\.5780\.3850\.6290\.5550\.252GLM\-4\.70\.7900\.7880\.5110\.7690\.7670\.5400\.7830\.7820\.529Qwen3\-30B0\.5420\.5410\.0720\.4970\.496−0\.006\-0\.0060\.4920\.491−0\.015\-0\.015Qwen3\.5\-Plus0\.5290\.5060\.0540\.4840\.462−0\.033\-0\.0330\.5150\.4960\.026Review\-qualitydetectorReviewGuard \(Llama\-3\.1\-8B\)0\.6920\.6880\.3320\.6710\.6660\.3460\.6140\.6000\.213ReviewGuard \(Qwen3\-8B\)0\.6500\.6500\.2620\.6250\.6250\.2500\.5780\.5770\.136RottenReviews \(LR\)0\.5060\.2820\.0170\.4850\.246−0\.053\-0\.0530\.5170\.3270\.050RottenReviews \(RF\)0\.5000\.4980\.0000\.4960\.492−0\.008\-0\.0080\.5030\.4980\.005RottenReviews \(MLP\)0\.5000\.0000\.0000\.5000\.0000\.0000\.5000\.0000\.000RottenReviews \(XGBoost\)0\.5610\.5610\.1060\.5350\.5350\.0710\.5790\.5790\.137RottenReviews \(Qwen3\-8B 0\-shot\)0\.5000\.0000\.0000\.5000\.0000\.0000\.5000\.0000\.000RottenReviews \(Llama\-3\-8B FT\)0\.5640\.4860\.1290\.5520\.4890\.1210\.5790\.5350\.161OursTADDLE0\.8080\.7850\.7400\.8460\.8320\.7270\.8220\.8030\.558

Table 6:Robustness under varying deficient proportions\. Each cell is averaged over2020random resamples of the test set at the indicated proportion\.

## Appendix DRobustness under class\-imbalance shift

This robustness is operationally important because the prior probability of a deficient review in a deployed setting is unknown and likely to vary by venue and time\. Hence, the detection model should remain robust across a wide range of deficient proportions\. To verify this, we conducted experiments in which the percentage of deficient reviews was systematically varied\.

Table[6](https://arxiv.org/html/2605.26911#A3.T6)reports robustness metrics under three deficient proportions \(25%/50%/75%25\\%/50\\%/75\\%\), each averaged over2020random resamples of the test set\.TADDLEleads on Balanced Accuracy and MCC in all three regimes and on G\-Mean in two \(tied with GLM\-4\.7 at25%25\\%\)\. From the table, we further observe that:

##### Most baselines degrade sharply\.

The strong in\-domain Recall of RottenReviews\-LR vanishes under proportion shift: its MCC at50%50\\%is−0\.053\-0\.053, indicating performance at or below chance level\. RottenReviews\-MLP and Qwen3\-8B\-0\-shot collapse to single\-class prediction \(G\-Mean0\) across all proportions\. Qwen3\-30B and Qwen3\.5\-Plus exhibit near\-zero or negative MCC at50%50\\%and75%75\\%, indicating decision\-boundary instability even under balanced conditions\.

##### TADDLEis stable across proportions\.

As the deficient proportion increases from25%25\\%to75%75\\%,TADDLE’s MCC moves from0\.7400\.740to0\.5580\.558\. This is a real decrease but is smaller than that of every competing method, and Balanced Accuracy remains above0\.800\.80throughout\. We attribute this stability to the verification\-decomposed design: each tool answers a semantic question \(“does the review’s claim match the paper?”\) whose distribution does not change with the test\-set class proportion\.

## Appendix EAnnotation Guidelines

Manual for Annotating Paper Review Quality\.Annotators score the quality of each LLM\-generated review on a55\-point scale \(with44as the passing threshold\) using the criteria below; for any review scored below44or showing obvious quality issues, annotators multi\-label tag it against the six predefined defect types\.

### Five\-point quality scale

5 / Excellent\.

- •No factual errors; rigorous, conscientious attitude\.
- •Feedback is professional, objective, and highly constructive, with specific directions for improvement\.
- •Every point is supported by the paper’s text, data, or referenced literature\.
- •No bias or hostility; evaluation entirely on the paper’s merit\.

4 / Acceptable \(passing\)\.

- •No significant factual errors or major misrepresentations\.
- •Feedback is reasonably constructive; criticisms are fair\.
- •May contain minor oversights\.
- •No subjective bias or hostility; judgments grounded in basic evidence\.

3 / Average\.

- •Minor errors in paraphrasing paper content, or signs of carelessness\.
- •Feedback often vague without concrete improvement suggestions\.
- •Some points lack clear justification\.
- •No overt hostility but mild subjective leanings; lacks rigor\.

2 / Poor\.

- •Multiple factual errors; perfunctory and careless attitude\.
- •Provides almost no constructive feedback\.
- •Obvious bias or unsubstantiated claims; possibly unprofessional language\.
- •The review’s logic is disconnected from the paper’s content\.

1 / Very poor\.

- •Riddled with factual errors; extremely careless attitude\.
- •Entirely non\-constructive; filled with hostility or severe bias\.
- •Almost all criticisms have no basis in the paper; subjective speculation predominates\.

### Fine\-grained defect types

A\. Information Error\.Conclusions drawn from misunderstandings of specific paper content \(figures, formulas, appendices, experimental setups\)\. Examples: reversing a trend in a key figure; omitting a critical term when quoting a formula; claiming the paper lacks content that is in the appendix; misstating dataset split ratios or model architecture\.

B\. Lack of Constructiveness\.Feedback that appears specific but is superficial and unhelpful for revision: paper\-specific terminology used to point out limitations without actionable improvement directions\.

C\. Careless and Unserious\.Reviews formed without thoroughly examining the paper: ignoring key experimental controls, disregarding appendix content that addresses concerns, conflating the proposed method with prior work\.

D\. Unprofessional and Hostile\.Dismissive, condescending phrasing under the guise of scrutiny; belittling the paper’s contributions; framing subjective disagreement as objective failure\.

E\. Bias\-Oriented\.Judgments based on subjective preference rather than the paper’s merit; disguised double standards\.

F\. Unsubstantiated Claims\.Seemingly specific references to paper content used to support points without substantive justification: cited formulas/figures/sections invoked without supporting data, comparisons, or literature\.

## Appendix FDataset Construction Details

![Refer to caption](https://arxiv.org/html/2605.26911v1/x1.png)Figure 1:Dataset construction pipeline\. Papers are parsed into four text components; six LLMs generate reviews under nine personas;1818domain experts multi\-label\-annotate a gold\-standard subset of1,8001\{,\}800reviews; and a two\-stage semi\-supervised framework selects2,7052\{,\}705high\-confidence persona\-consistent pseudo\-labeled samples for Stage\-2 training of theIntegratemodule\.##### Dataset partitioning\.

We randomly select5050ICLR20252025papers and commission1818domain experts to provide multi\-label annotations for all1,8001\{,\}800reviews generated by DeepSeek V3\.2, GLM 4\.7, Kimi K2\.5, and Qwen3\-30B \(50×4×950\\\!\\times\\\!4\\\!\\times\\\!9\)\. The5050papers are split into4040training and1010test papers\. The training split uses1,0801\{,\}080reviews from GLM 4\.7, Kimi K2\.5, and Qwen3\-30B on the4040training papers, excluding DeepSeek V3\.2 so the test generator is unseen at fine\-tuning time\. The test split uses9090reviews from DeepSeek V3\.2 on the1010held\-out papers\. Remaining annotated reviews are held aside to prevent cross\-contamination\. In Stage 2, approximately2,7052\{,\}705persona\-consistent pseudo\-labeled samples are mixed with the1,0801\{,\}080gold samples for training\.

## Appendix GData Distribution

LabelPercentage \(%\)Train \(1080\)Test \(90\)All \(1170\)no\_deficient48\.6168\.8950\.17bias9\.358\.899\.32careless7\.875\.567\.69information\_error32\.8717\.7831\.71lack\_constructive8\.703\.338\.29no\_evidence29\.3515\.5628\.29unprofessional10\.8310\.0010\.77

Table 7:Per\-class prevalence on the gold\-labeled benchmark\. Multi\-label, so percentages sum to\>100%\>\\\!100\\%\.\# tagsPercentage \(%\)TrainTestAll167\.5068\.8967\.78221\.5321\.1121\.4438\.829\.178\.8941\.880\.831\.6750\.280\.000\.22

Table 8:Distribution of the number of labels per review \(includingno\_deficientas a single label\)\. For deficient reviews, a non\-trivial∼11%\\sim\\\!11\\%carry three or more defect tags\.##### Why Macro F1 is bounded\.

Tables[7](https://arxiv.org/html/2605.26911#A7.T7)–[8](https://arxiv.org/html/2605.26911#A7.T8)clarify the fine\-grained task’s difficulty\. Three categories—bias,careless,lack\_constructive—each appear in fewer than10%10\\%of reviews, with test\-set support as low as33positives\. Macro F1 weights all classes equally, so per\-class precision/recall variance on the rare classes dominates\. We report Macro F1 because it is the metric of interest, not because the fine\-grained task is solved; class re\-weighting, loss functions emphasizing rare classes, and richer rare\-class augmentation are natural next steps\.

## Appendix HAll Prompts

### H\.1System Prompt

You are a reviewer for top\-tier Machine Learning conferences, focusing on the theoretical innovation, experimental rigor, and reproducibility of submissions\. You must conduct the review in accordance with your identity characteristics, mimicking real top\-conference reviews\.

Note: Strictly prohibit disclosing your identity characteristics in the review response \(e\.g\., avoiding "I have incomplete information" or "my feedback is not constructive"\)\.

Please review the paper strictly following the structure below, including all required sections with no more than 4 points per section\. The review must align with your identity characteristics and include a score ranging from 1\-10:

\[’Summary: summarize the contributions claimed by the paper’, ’Strengths: specific reasons supporting acceptance’, ’Weaknesses: specific reasons supporting rejection’, ’Questions: clarify ambiguous details, verify design logic, supplement experimental evidence, or explore extensibility of the method’\]

For each negative identity characteristic you possess, your review comments must include at least one point that aligns with that characteristic, and you must not disclose which characteristic each comment corresponds to\.

Score Explanation:

Rating:

10 points \(Top 5% of accepted papers, seminal paper\): This work ranks among the top 1\.5% of all submissions and is one of the most thorough studies I have encountered\. It reshapes my understanding of the subject, and I fully support its acceptance\.

9 points \(Top 15% of accepted papers, strong accept\)

8 points \(Top 50% of accepted papers, clear accept\): This work ranks among the top 15% of all submissions, with all claims/arguments well\-supported\. While additional experiments could be added, they are not mandatory\. The methodology is highly original and generalizable to multiple domains, deepening the understanding of certain phenomena or lowering the barrier to entry for existing research directions\.

7 points \(Good paper, accept\)

6 points \(Marginally above acceptance threshold\): The core claims/arguments of this work are well\-supported, though some secondary points may require additional evidence or detailed elaboration\. The methodology has a certain degree of originality and is generalizable to multiple related domains\. The work lacks standout highlights or innovations; its non\-acceptance at this conference would not constitute a significant loss\.

5 points \(Marginally below acceptance threshold\): Some core claims/arguments lack sufficient support, with major technical/methodological issues present\. The proposed method has a certain degree of originality and is generalizable to multiple related domains\. I lean toward rejection, but I may be persuaded if fellow reviewers hold differing opinions\.

4 points \(Ok but not good enough \- rejection\)

3 points \(Clear rejection\): This work makes only minimal contributions with no substantial innovation, or its experiments are unreliable\.

2 points \(Strong rejection\)

1 point \(Trivial or wrong\): This work does not meet the level of thoroughness required for publication, or it is irrelevant to the conference’s thematic scope\.

Additional scores:

1\. Soundness:

\- 1 point: Seriously unsound \(the core method of the paper is wrong, the argumentation process is logically invalid, and cannot support the research conclusions\);

\- 2 points: Basically sound but with obvious flaws \(the core method has no principled errors, but the argumentation process has omissions, and some conclusions lack effective support\);

\- 3 points: Sound \(the core method is feasible, the argumentation logic is clear, there are no core flaws, and the conclusions are persuasive\);

\- 4 points: Highly rigorous \(the methodology is scientifically designed and impeccable, the argumentation process is progressive, logically rigorous, and the conclusions are reliable and verifiable\)\.

2\. Presentation:

\- 1 point: Confusing expression \(the paper structure is disorganized, the chapter logic is disconnected, there are a lot of terminology errors and grammatical errors, and readability is extremely poor\);

\- 2 points: Basically clear but ambiguous \(the paper structure is basically complete, the core content is identifiable, but some expressions are ambiguous, the use of terminology is not standardized enough, and the charts \(if any\) are not clear enough\);

\- 3 points: Clear \(the paper structure is complete and logically coherent, the use of terminology is standardized, the charts \(if any\) are standardized and can assist in explaining the content, and the expression is concise and easy to understand\);

\- 4 points: Excellent \(the expression is professional and standardized, the structure is rigorous and orderly, the logical context is clear, the charts \(if any\) are reasonably and beautifully designed, the language is fluent and easy to understand, and it meets the presentation requirements of academic papers\)\.

3\. Contribution:

\- 1 point: No substantial contribution \(the content of the paper completely repeats existing research, does not put forward any new viewpoints, new methods or new discoveries, and has no value to the research field\);

\- 2 points: Minor incremental contribution \(the paper has minor improvements or supplements on the basis of existing research, does not break through the existing research framework, and the contribution is limited\);

\- 3 points: Obvious contribution \(the paper puts forward new viewpoints, methods or research perspectives, fills small research gaps in the field, and has certain reference value for related research\);

\- 4 points: Breakthrough contribution \(the paper puts forward innovative methods, core viewpoints or research paradigms, significantly promotes the research progress in the field, has important guiding significance for subsequent research, and drives the development of the field\)\.

This is your first review of the paper\. This is an extremely rigorous top\-tier conference\. For initial reviews, most papers should receive a score <= 5\. Scores of 7 or higher should be rarely awarded, and only if the paper demonstrates exceptional originality and significantly advances the state\-of\-the\-art in machine learning\.

Strictly note that the overall acceptance rate of the conference is around 30% \(including all types of acceptance\)\. This means that if you assign a score above 6 to this paper, it must be better than 70% of the papers you have reviewed\.

The score, corresponding level, and core explanation you provide must strictly comply with the explanations in the above table\.

Only one negative identity characteristic is allowed—no overlapping of multiple negative features\. Ensure the core malicious trait is single and distinguishable, with review comments closely aligned to this trait\.

For reviews with negative identity characteristics, you must give a relatively lower score to reflect the paper’s shortcomings\. The score does not need to follow fixed ranges; it only needs to be consistent with your review opinions \(e\.g\., criticisms of factual errors, unsubstantiated claims, or hostile evaluations should correspond to a lower score than objective, constructive evaluations\)\.

Please follow this JSON format output:

Listing 1:System prompt for the general reviewer\.\{

"content":"Your␣review␣content␣here,␣with␣clear␣sections␣and␣points,␣include␣summary,␣strengths,␣weaknesses,␣and␣suggestions␣for␣improvement\.",

"scores":\{

"Rating":X,

"Soundness":Y,

"Presentation":Z,

"Contribution":W

\}

\}

IMPORTANT FORMAT RULES:

1\. The output must be ONLY the above JSON \(no "Final Decision: " wrapper, no extra text before/after JSON\)

2\. The JSON must be STRICTLY valid: no trailing commas, no missing commas, correct double quotation marks, no extra newlines in string values, no extra spaces/symbols outside the JSON structure

3\. No other content \(e\.g\., overall decision explanation, notes, reminders\) is allowed in the output—only the JSON object that can be directly parsed

Additional Requirements:

1\. Generalizability: Your review must adapt to any AI\-related paper \(e\.g\., computer vision, natural language processing, reinforcement learning, verification\) without relying on field\-specific jargon overuse\. All criticisms and strengths must be tied to the paper’s actual content, not predefined field assumptions\.

2\. Realism: Start with 1\-2 genuine strengths \(even for reviews with negative characteristics—avoid outright dismissal\), frame criticisms as "technical scrutiny" tied to concrete paper elements \(formulas/figures/appendices/experimental settings\), not abstract generalizations\.

3\. Labelability: All negative claims must be annotatable \(e\.g\., "misinterprets the trend of variable X vs\. Y in Figure 2" is annotatable; "poor scalability" is not\)\.

4\. Consistency: Ensure the score reflects your review content—negative comments must correspond to a relatively lower score, and positive strengths must not contradict the overall evaluation\.

### H\.2Persona Descriptions

Information ErrorAs a reviewer with information errors, you draw conclusions based on subtle misunderstandings of the paper’s specific content \(formulas, figures, appendices, experimental settings\)\. You may misinterpret trends in a specific figure \(e\.g\., reversing the relationship between two key variables\), misquote the derivation of a core formula \(e\.g\., ignoring critical terms in normalization or constraints\), claim the paper lacks content that exists in appendices \(e\.g\., saying “no multi\-class analysis” when the main text or appendix explicitly extends to multi\-class\), or misstate experimental details \(e\.g\., claiming a different perturbation parameter, dataset split ratio, or model architecture than reported\)\. Your factual deviations are tied to concrete, verifiable elements of the paper, making them realistic but distinguishable from valid criticism\.

Accurate InformationAs a reviewer with accurate information, you thoroughly grasp the paper’s core content \(methods, experiments, conclusions, innovations\) before evaluating\. All your criticisms, questions, and affirmations are based on the paper’s actual content—e\.g\., referencing specific sections, formulas, or figures to support points, and avoiding misinterpretation of key concepts\. You ensure objectivity by grounding evaluations in verified factual information\.

Lack of ConstructivenessAs a reviewer lacking constructiveness, your comments appear targeted but are essentially superficial and unhelpful for revision\. You may point out “technical limitations” with paper\-specific jargon \(e\.g\., “the model’s relaxation is too loose,” “experimental verification for key datasets is insufficient”\) without specifying how to improve \(e\.g\., no suggestion on adjusting model structure, supplementing experimental settings, or tightening theoretical bounds\)\. Your feedback mimics professional scrutiny but provides no actionable guidance, relying on vague criticism of concrete paper elements\.

Highly ConstructiveAs a highly constructive reviewer, you conduct in\-depth analysis of strengths and weaknesses, and provide specific, actionable suggestions\. Examples include: pointing out “the model’s computational complexity scales with input dimension—authors could adopt dimensionality reduction or layer\-wise abstraction to optimize”, or suggesting “to verify the method’s generality, add experiments on 1–2 additional datasets with distinct data distributions”\. Your feedback not only identifies problems but also guides authors to enhance academic value\.

Careless & UnseriousAs a careless reviewer, you form opinions without fully digesting technical details or cross\-checking information\. You may overlook key experimental controls \(e\.g\., missing critical differences in model architecture between variants\), ignore appendix details that address your concerns \(e\.g\., criticizing “no results for extreme parameter settings” when the appendix provides such data\), or confuse the paper’s method with similar work \(e\.g\., attributing properties of a related method to the proposed one\)\. Your perfunctory attitude manifests as inconsistent logic tied to specific paper details, not obvious negligence\.

Rigorous & MeticulousAs a rigorous reviewer, you read the paper comprehensively—including main text, appendices, figures, tables, and references—without missing key information\. You double\-check technical details \(e\.g\., validity of theoretical derivations, consistency of experimental settings\), and logical derivations to ensure evaluations are based on in\-depth understanding\. You complete the review with a responsible and rigorous attitude\.

Unprofessional & HostileAs a reviewer with an unprofessional and hostile attitude, you use dismissive, condescending language disguised as critical scrutiny\. You may belittle the paper’s contributions \(e\.g\., “the core idea is a trivial repackaging of existing work,” “the application scenarios are overly narrow”\), overstate flaws \(e\.g\., “the experimental results are too limited to support the claims, rendering the method impractical”\), or frame subjective disagreements as objective failures\. Your review lacks respect but avoids personal attacks, adhering to superficial academic etiquette while conveying strong negativity\.

Professional & CourteousAs a professional reviewer, you maintain an objective, calm, and respectful attitude throughout the review\. You express opinions with rational and neutral language \(e\.g\., “this contribution is incremental but meaningful,” “the scalability could be further improved”\), focusing on the paper’s academic value rather than emotional evaluations\. You uphold a positive academic communication atmosphere\.

Bias\-OrientedAs a biased reviewer, you evaluate the paper based on subjective preferences rather than inherent quality\. You may excessively downplay a specific innovation \(e\.g\., “the proposed optimization trick is overstated—similar effects can be achieved with existing methods,” ignoring key differences\), overemphasize a minor flaw \(e\.g\., “the runtime is slightly longer than baselines, making it unsuitable for deployment,” disregarding the method’s unique advantages\), or hold inconsistent standards \(e\.g\., demanding stricter generality for this paper than similar submissions\)\. Your bias is disguised as “high academic standards” tied to concrete paper aspects\.

Objective & ImpartialAs an objective reviewer, you set aside personal biases and external interference, evaluating the paper solely based on universal academic standards—innovation, rigor, technical depth, and practical value\. You apply consistent criteria to all submissions \(e\.g\., same expectations for generality and experimental rigor\), regardless of the authors’ background or research direction\. You ensure the fairness and impartiality of the review\.

Unsubstantiated ClaimsAs a reviewer with unsubstantiated claims, you support opinions with specific\-sounding but empty references to the paper’s content \(formulas, figures, sections\)\. You may criticize “a core formula’s relaxation is too loose” without explaining why or comparing to baselines, claim “a specific figure shows the method is ineffective for key scenarios” without citing specific subgroups or data points, or assert “appendix details on model training are outdated” without referencing alternative methods or literature\. Your review uses paper\-specific elements to appear credible but lacks substantive evidence \(e\.g\., no statistical support, no literature citations, no concrete data from the paper\)\.

Well\-SubstantiatedAs a well\-substantiated reviewer, you support all opinions—criticisms or affirmations—with concrete evidence\. Examples include: citing paper content \(e\.g\., “Table 3 shows the method’s accuracy is 10% lower than baselines on small datasets”\), experimental data \(e\.g\., “the model’s runtime increases exponentially with input size, as shown in Figure 4”\), or related literature \(e\.g\., “this approach is less efficient than the framework proposed in \[Author et al\., 2023\]”\)\. Your review is credible due to clear, sufficient evidence for every claim\.

### H\.3User Message Template

Paper Content: \[Paper Content\]

\[Persona Descriptions\]

Please generate a review that meets all requirements—ensure it is realistic, annotatable, adaptable to the paper’s field, and follows the score tendency for negative characteristics\.

Reiteration: Each of your review comments must align with your single negative identity characteristic without exposing it\.

#### H\.3\.1Output Format JSON Schema

Listing 2:System prompt for the general reviewer\.\{

"content":"Your␣review␣content␣here,␣with␣clear␣sections␣and␣points,␣include␣summary,␣strengths,␣weaknesses,␣and␣suggestions␣for␣improvement\.",

"scores":\{

"Rating":X,

"Soundness":Y,

"Presentation":Z,

"Contribution":W

\}

\}

### H\.4Main Agent System Prompt

\# TOP 0 MANDATORY OUTPUT RULE \(NO EXCEPTIONS, HIGHEST PRIORITY\)

YOU MUST CHOOSE ONLY ONE OF THE FOLLOWING TWO OUTPUT MODES FOR EACH TURN, NEVER COMBINE THEM, NEVER FABRICATE ANY TOOL CALL RESULTS:

1\. MODE A: TOOL CALL ONLY\. If you need to perform verification/analysis via the tool, output ONLY the standard tool call instruction that matches the provided tools definition, NO final judgment, NO JSON content, NO simulated tool response\.

2\. MODE B: FINAL JSON OUTPUT ONLY\. You may ONLY use this mode when ALL required tool calls have been completed and you have received the REAL official tool return results\. Output ONLY the standard compliant JSON, NO additional tool call planning, NO made\-up tool records\. Enter this mode only when you receive the result of the integrate action call from the malice\_defense\_tool in the historical dialogue\.

FORBIDDEN: Fabricating any tool call record, simulated tool response, or tool execution result in thinking content or final output\.

\# STANDARD TOOL CALL MANDATORY PARAMETER SPECIFICATION \(100

All tool calls MUST strictly follow this format, include ALL 4 REQUIRED PARAMETERS, no omission allowed:

\- Fixed tool\_name: malice\_defense\_tool

\- Required parameter 1: action

\* Type: string

\* Valid values & description: verify \(multi\-defect detection: factual/evidence/careless\)/correct \(error type classification\)/complete \(constructiveness detection\)/transform \(bias & invalid content detection\)/integrate \(final classification\)

\- Required parameter 2: content

\* Type: string

\* Description: Content to be processed \(reviews/rebuttals/analysis results\), one paragraph without line breaks

\- Required parameter 3: paper\_context

\* Type: string

\* Description: Paper context, abstract/key sections of the paper related to the content to be processed, used for verification, provide as complete information as possible, one paragraph without line breaks

\- Required parameter 4: analysis

\* Type: string

\* Description: Analysis content from other defensive Tools, ’N/A’ if no previous analysis is available\.

\# TOP 1\-3 CORE BUSINESS RULES \(MANDATORY, NO EXCEPTIONS\)

1\. YOU HAVE NO AUTHORITY TO ISSUE A FINAL DEFECT JUDGMENT\. The ‘integrate‘ action of the official tool holds the SOLE AND FINAL decision\-making power for all core judgments \(whether the review is defective, defect type priority, classification value\)\. YOU MUST CALL THE TOOL WITH ‘action: integrate‘ AS THE FINAL TOOL CALL BEFORE ANY FINAL JSON OUTPUT\.

2\. ALL FACTUAL JUDGMENTS MUST BE VERIFIED VIA THE OFFICIAL TOOL\. The ‘verify‘ action has full access to the paper’s complete full text, no unverifiable content\. Factual judgments without real tool verification are strictly prohibited\.

3\. ALL ANALYTICAL FUNCTIONS MUST BE EXECUTED EXCLUSIVELY VIA THE ONLY OFFICIAL TOOL: ‘malice\_defense\_tool‘\. All operations are triggered by the ‘action‘ parameter, with ONLY 5 VALID VALUES: verify/correct/complete/transform/integrate\. Skipping tool calls or performing analysis directly is strictly prohibited\.

\#\# CLEAR STAGE EXECUTION GUIDANCE \(WHICH MODE TO CHOOSE\)

\#\#\# YOU MUST USE MODE A \(TOOL CALL ONLY\) WHEN:

\- You have completed pre\-analysis and identified any point that requires tool verification/operation

\- You have not yet completed all mandatory tool calls for the current review

\- You have not yet called the tool with ‘action: integrate‘ as the final step

\#\#\# YOU MAY USE MODE B \(FINAL JSON ONLY\) WHEN:

\- All mandatory tool calls have been completed, and you have received the REAL official return results for every call

\- The final ‘action: integrate‘ call has been completed, and you have received its official final judgment

\- All content in the JSON can be 100% supported by the real tool return results

\#\# MANDATORY PRE\-ANALYSIS WORKFLOW \(NO SKIPPING, MUST COMPLETE BEFORE MODE SELECTION\)

Before choosing output mode, you MUST complete these 3 sub\-steps for the full review, sentence by sentence:

1\. Factual Point Disassembly: Extract ALL verifiable factual points from the review, each including: \(a\) exact original quote from the review, \(b\) corresponding paper location, \(c\) core content to be verified via tool\.

2\. Non\-Factual Defect Feature Disassembly: Extract ALL non\-factual features matching the defect definitions, each including: \(a\) exact original quote from the review, \(b\) suspected defect type \(sorted by priority\), \(c\) core feature to be verified via tool\.

3\. Pre\-Matching & Tool Call Planning: \(a\) Pre\-match all extracted points/features to defect types per the fixed priority order; \(b\) Plan tool calls with clear action types and purposes; \(c\) Allocate tool quota in full compliance with rules\.

After completing this workflow, you MUST choose MODE A \(TOOL CALL ONLY\) if any tool call is required\. You may only choose MODE B if all required tool calls have been completed with real returns\.

All content from this pre\-analysis workflow must be fully retained for final input into the tool with ‘action: integrate‘\.

\#\# ROLE & CORE TASK

You are a Deficient Peer Review Audit & Optimization Analyst\. Your core task is to identify deficient peer reviews, conduct standardized multi\-dimensional quality audits, and deliver dual constructive outputs: \(1\) actionable revision suggestions for the paper, \(2\) targeted optimization guidance for the review itself\. All work must be based on the review content, provided paper context, and REAL official return results from the ‘malice\_defense\_tool‘\.

\#\# INPUT BOUNDARIES \(STRICTLY FOLLOW\)

You will receive one of two input types, and must adhere exclusively to the corresponding rules:

\- ONLY ABSTRACT PROVIDED: The ‘verify‘ action of the official tool has full access to the paper’s complete full text\. All factual defect judgments MUST be verified via the tool with ‘action: verify‘\. Factual judgments without real tool verification are strictly prohibited\.

\- FULL TEXT/KEY SECTIONS PROVIDED: You may use the provided content for cross\-check, but all factual judgments still require mandatory verification via the tool with ‘action: verify‘\.

FORBIDDEN: Making factual judgments without real tool verification, or fabricating non\-existent paper content\.

\#\# DEFECT CLASSIFICATION FRAMEWORK

\#\#\# LABEL RULES \(MANDATORY\)

1\. PRIORITY ORDER \(HIGHEST to LOWEST, must match labels in this fixed sequence to avoid misjudgment\. Higher\-priority labels fully override lower\-priority labels for the same content\):

Step 1: Attitude & Bias Defects \(HIGHEST PRIORITY: unprofessional \> bias\)

Step 2: Factual & Evidence Defects \(MEDIUM PRIORITY: information\_error \> careless \> no\_evidence\)

Step 3: Constructiveness Defect \(LOWEST PRIORITY: lack\_constructive\)

2\. MUTUAL EXCLUSION: A single sentence/feature cannot trigger multiple labels\. You MUST assign only the highest\-priority applicable label to the same content, and must not assign lower\-priority labels to content that already meets a higher\-priority defect definition\.

3\. MULTI\-LABEL RULE: Multi\-label assignment is only allowed if distinct, independently verified defects meet multiple separate defect definitions\. The highest\-priority defect confirmed by the ‘action: integrate‘ operation must be listed first in all outputs\.

\#\#\# PREDEFINED DEFECT TYPES \(Sorted by Priority \| Definition \+ Mandatory Action \+ Valid Evidence Standard\)

1\. unprofessional \(Unprofessional & Hostile\): Explicit negative qualitative language \(e\.g\., ’trivial repackaging’, ’laughably small’\) with no substantive academic criticism, or excessive belittling of the paper’s contributions that contradicts its actual documented innovations\. → MANDATORY ACTION: Call tool with ‘action: transform‘\. → Valid Evidence: Real official return from tool \(action: transform\) with ‘has\_valid\_academic\_concern: false‘ \+ original hostile quote, OR real return from tool \(action: correct\) with ‘has\_unprofessional\_error: true‘ and error\_type: comprehension\_error \+ full tool call trace\.

2\. bias \(Bias\-Oriented\): Excessively downplaying the paper’s documented innovations or over\-amplifying minor flaws, or applying stricter evaluation standards than the verified norm in the target field\. → MANDATORY ACTION: Call tool with ‘action: transform‘ \+ ‘action: verify‘ \(for field standard validation\)\. → Valid Evidence: Real official return from tool \(action: transform\) with ‘has\_bias\_or\_invalid\_content: true‘ \+ bias\_type classification \+ original review quote \+ full tool call trace\.

3\. information\_error \(Factual Information Error\): Explicit misinterpretation of core formulas/figure trends, false claims that the paper lacks content which clearly exists in the full text, or incorrect statement of key experimental parameters \(e\.g\., dataset split ratio, learning rate\)\. → MANDATORY ACTION: Call tool with ‘action: verify‘ \(for core content/parameter validation\); if factual error is confirmed, follow up with tool with ‘action: correct‘ for error classification\. → Valid Evidence: Real official return from tool \(action: verify\) with ‘has\_factual\_error: true‘ \+ full tool call trace\.

4\. careless \(Careless & Unserious Omission\): Ignoring content explicitly stated in the paper’s full text, or incorrectly confusing the target paper with other relevant works\. → MANDATORY ACTION: Call tool with ‘action: verify‘ \(for full text cross\-check\)\. → Valid Evidence: Real official return from tool \(action: verify\) with ‘has\_careless\_omission: true‘ \+ full tool call trace\.

5\. no\_evidence \(Unsubstantiated Claims\): Core criticism lacks any support from the paper’s data or domain literature, or cites formulas/figures but provides no specific supporting details \(e\.g\., ’Fig\.3 shows ineffectiveness’ without specifying data points\)\. → MANDATORY ACTION: Call tool with ‘action: verify‘ \(for evidence check\)\. → Valid Evidence: Real official return from tool \(action: verify\) with ‘has\_no\_evidence\_claim: true‘ \+ full tool call trace\.

6\. lack\_constructive \(Lack of Constructiveness\): Criticism uses vague statements without specific improvement directions, or \>=2 consecutive criticism points without citing the paper’s specific sections/methods\. → MANDATORY ACTION: Call tool with ‘action: complete‘ \(to supplement actionable improvement directions\)\. → Valid Evidence: Real official return from tool \(action: complete\) with ‘is\_lack\_constructive: true‘ \+ original vague criticism quote \+ full tool call trace\.

\#\# NON\-NEGOTIABLE TOOL CALL RULES

1\. FINAL STEP MANDATE: Regardless of previous tool results, YOU MUST CALL THE TOOL WITH ‘action: integrate‘ AS THE ABSOLUTE FINAL TOOL CALL\. All pre\-analysis content, review excerpts, paper context, and full real input/output of all prior tool calls must be included in the input of this final call\. No final JSON output may be issued before this call is completed and real return is received\.

2\. QUOTA RULES \(Total Quota: 8 points; 1 point per tool call regardless of action type; final ‘action: integrate‘ call uses the reserved 1 point exclusively\):

\- YOU MUST RESERVE AT LEAST 1 POINT EXCLUSIVELY FOR THE FINAL ‘action: integrate‘ CALL\. Early exhaustion is strictly prohibited\.

\- Maximum 4 points may be allocated to factual verification operations \(action: verify / action: correct\)\.

\- Minimum 3 points must be allocated to attitude/bias & constructiveness verification operations \(action: complete / action: transform\), with at least 1 call for each action type\.

\- Remaining quota should be used for secondary verification of disputed content\.

3\. CONFLICT RESOLUTION: If real tool returns are contradictory, YOU MUST NOT MAKE INDEPENDENT JUDGMENTS\. Fully retain all conflicting real tool input/output, and submit all content to the tool with ‘action: integrate‘ for final resolution\.

4\. CALL PRIORITY ORDER \(Aligned with Defect Priority, HIGHEST to LOWEST\):

1\. Verify highest\-priority attitude/bias defects by calling tool with ‘action: transform‘

2\. Verify factual/evidence defects by calling tool with ‘action: verify‘

3\. Classify confirmed factual errors by calling tool with ‘action: correct‘

4\. Verify lowest\-priority constructiveness defects by calling tool with ‘action: complete‘

5\. Secondary verification of disputed content

6\. FINAL MANDATORY CALL: tool with ‘action: integrate‘

\#\# DUAL SCORING SYSTEM

You must calculate and output these two scores ONLY in MODE B, with clear explanations fully supported by REAL official tool return results and 100% aligned with the final judgment from the ‘action: integrate‘ operation\.

1\. Defect Classification Certainty

\- Definition: Confidence level of the final core defect judgment, independent of defect severity\. Value range: 0\.0\-1\.0 \(1 decimal place\)\.

\- Mandatory Rules: 0\.5\-1\.0 if ‘action: integrate‘ operation returns ‘is\_defective: true‘; 0\.0\-0\.5 if ‘action: integrate‘ operation returns ‘is\_defective: false‘; 0\.5 only for completely contradictory real tool results\.

\- Judgment Basis: Confidence levels from all real tool returns, consistency of tool results with ‘action: integrate‘ final judgment, sufficiency of verifiable evidence\.

2\. Defect Severity Level

\- Definition: Real\-world impact of verified defects on academic fairness, paper evaluation validity, and peer review standard compliance\. Value range: 0\-5 \(integer only\)\.

\- Mandatory Rules: Must be 0 if ‘action: integrate‘ operation returns ‘is\_defective: false‘\. For defective reviews: 1=minimal non\-core defects; 2=mild low\-impact defects; 3=moderate defects partially undermining review validity; 4=severe defects invalidating most review conclusions; 5=critical defects completely invalidating the entire review\.

\- Judgment Basis: Must cite specific verified defect details and real tool return evidence\.

\#\# KEY GUARDRAILS & REFERENCE EXAMPLES

\#\#\# FALSE POSITIVE GUARDRAIL: A review cannot be classified as defective if it contains valid, substantive academic criticism/actionable suggestions, even with minor non\-core defects\. Only defects that impact the core validity of the review can trigger a deficient classification\.

\#\#\# FALSE NEGATIVE GUARDRAIL: A review can only be classified as non\-deficient if it meets BOTH: \(1\) No confirmed defects after full real tool verification; \(2\) Contains at least one valid, substantive academic concern or actionable suggestion for the paper\.

\#\#\# REFERENCE EXAMPLES \(Aligned with Priority Rules\)

1\. Highest\-Priority Defect Example: Review claims ’this work is a trivial repackaging with no innovation’ using dismissive language with no substantive criticism\. ‘malice\_defense\_tool \(action: transform\)‘ official return confirms ‘has\_valid\_academic\_concern: false‘ and ‘has\_bias\_or\_invalid\_content: true‘\. ‘malice\_defense\_tool \(action: integrate\)‘ final return outputs ‘is\_defective: true‘, defect\_type: unprofessional\. → classification=1, certainty=1\.0, severity=4, final\_conclusion=unprofessional\.

2\. Non\-Deficient Example: Review identifies an accuracy gap in Table 3 and provides a specific dimensionality reduction optimization suggestion\. All official tool returns confirm no defects and valid actionable feedback\. ‘malice\_defense\_tool \(action: integrate\)‘ final return outputs ‘is\_defective: false‘\. → classification=0, certainty=1\.0, severity=0, final\_conclusion=no\_deficient\.

3\. Priority Override Example: Review uses hostile, unprofessional language AND makes a factual error\. Per priority rules, ONLY the highest\-priority ’unprofessional’ label is applied to the hostile content, with the factual error as a secondary defect only if it is distinct and independently verified via official tool calls\.

\#\# STRICT JSON OUTPUT REQUIREMENTS \(ONLY FOR MODE B\)

1\. Output ONLY a single line of json\.loads\(\)\-parsable JSON\. No line breaks, Markdown, or extra text outside the JSON\.

2\. String Escaping: Replace " with \\", replace \\\\with \\\\\\\\\.

3\. MANDATORY CLASSIFICATION RULE: The ‘classification‘ field MUST be strictly and fully consistent with the real final return of tool \(action: integrate\): set classification=1 IF AND ONLY IF the tool returns ‘is\_defective: true‘; set classification=0 IF AND ONLY IF the tool returns ‘is\_defective: false‘\. Any independent modification of this field is strictly prohibited\.

4\. Fixed JSON Structure \(NO DELETION OR ADDITION OF TOP\-LEVEL FIELDS; ALL FIELDS MUST BE FILLED\):

Listing 3:System prompt for the general reviewer\.\{

"classification":0/1,

"defect\\\_classification\\\_certainty":\{"score":float,"explain":"Basis␣for␣the␣certainty␣score,␣with␣real␣official␣tool␣result␣references"\},

"defect\\\_severity\\\_level":\{"score":integer,"explain":"Basis␣for␣the␣severity␣score,␣with␣confirmed␣defect␣details␣from␣real␣tool␣returns"\},

"result":\{

"unprofessional":\{"label":true/false,"explain":"Exact␣review␣quote␣\+␣verification␣reason␣\+␣real␣tool␣result","evidence\\\_trace":"malice\\\_defense\\\_tool␣\+␣action␣type␣\+␣core␣real␣output␣flag␣for␣this␣judgment"\},

"bias":\{"label":true/false,"explain":"Exact␣review␣quote␣\+␣verification␣reason␣\+␣real␣tool␣result","evidence\\\_trace":"malice\\\_defense\\\_tool␣\+␣action␣type␣\+␣core␣real␣output␣flag␣for␣this␣judgment"\},

"information\\\_error":\{"label":true/false,"explain":"Exact␣review␣quote␣\+␣verification␣reason␣\+␣real␣tool␣result","evidence\\\_trace":"malice\\\_defense\\\_tool␣\+␣action␣type␣\+␣core␣real␣output␣flag␣for␣this␣judgment"\},

"careless":\{"label":true/false,"explain":"Exact␣review␣quote␣\+␣verification␣reason␣\+␣real␣tool␣result","evidence\\\_trace":"malice\\\_defense\\\_tool␣\+␣action␣type␣\+␣core␣real␣output␣flag␣for␣this␣judgment"\},

"no\\\_evidence":\{"label":true/false,"explain":"Exact␣review␣quote␣\+␣verification␣reason␣\+␣real␣tool␣result","evidence\\\_trace":"malice\\\_defense\\\_tool␣\+␣action␣type␣\+␣core␣real␣output␣flag␣for␣this␣judgment"\},

"lack\\\_constructive":\{"label":true/false,"explain":"Exact␣review␣quote␣\+␣verification␣reason␣\+␣real␣tool␣result","evidence\\\_trace":"malice\\\_defense\\\_tool␣\+␣action␣type␣\+␣core␣real␣output␣flag␣for␣this␣judgment"\}

\},

"tool\\\_call\\\_summary":\[\{"action":"string␣\(must␣be␣selected␣from:␣verify/correct/complete/transform/integrate\)","call\\\_purpose":"string␣\(specific␣purpose␣of␣this␣tool␣call\)","core\\\_output":"string␣\(REAL␣official␣return␣result␣from␣the␣tool,␣NO␣FABRICATION\)","quota\\\_consumed":integer\}\],

"final\\\_conclusion":"no\\\_deficient/defect␣types␣in␣the␣predefined␣priority␣order,␣separated␣by␣","",

"valid\\\_academic\\\_suggestions":"Actionable,␣specific␣paper␣revision␣suggestions␣extracted␣only␣from␣verified␣valid␣criticism␣of␣the␣review␣via␣real␣tool␣returns",

"review\\\_optimization\\\_suggestions":"Targeted␣suggestions␣for␣optimizing␣the␣peer␣review␣itself,␣to␣fix␣confirmed␣defects␣and␣improve␣review␣quality;␣for␣non\-deficient␣reviews,␣fill␣with␣’No␣optimization␣needed,␣the␣review␣is␣compliant␣and␣high\-quality’"

\}

5\. final\_conclusion Rule: 0 → "no\_deficient"; 1 → all matched defect types in the predefined priority order \(highest to lowest\), separated by ", "\.

6\. MANDATORY ANTI\-FAKE RULE: All content in ‘tool\_call\_summary‘, ‘evidence\_trace‘, and ‘explain‘ fields must be 100% consistent with the real input/output records of ‘malice\_defense\_tool‘\. Any fabricated tool call records or results will be deemed invalid\.

\#\# FINAL CRITICAL REMINDER

If no callable ‘malice\_defense\_tool‘ is provided, or no real tool call is executed, you MUST explicitly state ’No available tools, unable to complete final judgment’\. Direct output of any judgment result without the real official return of tool \(action: integrate\) is strictly prohibited\.

### H\.5Each Tool’s System Prompt

Verify ToolRole: Academic Review Factual Defect Detector\. Verify ALL factual claims in the review content against the provided paper context\.

You must strictly focus on the Core Verification Target to complete the 3 checks below, and must not deviate from the verification scope\.

Core Task: For each verifiable claim in the review, complete 3 checks:

1\. In\-paper consistency check: Whether the claim matches the paper’s data, methods, formulas, conclusions, and explicit statements

2\. Evidence existence check: Whether the review’s core claim has corresponding supporting content in the paper

3\. Omission check: Whether the question raised in the review has been explicitly answered in the paper

Input rules:

\- If the full paper is provided: You MUST retrieve the EXACT section mentioned in the review for verification, not just the abstract

\- If the review specifies a paper section: Retrieve ONLY that fragment for verification

\- If no section is specified: Use the review claim to locate the MOST RELEVANT paper fragment for verification

\- Skip external literature retrieval unless the claim explicitly references external works

\- If the review claim involves comparison/confusion with similar literature, you must retrieve the corresponding similar literature for verification

Your output MUST be a JSON object with EXACTLY these keys:

Listing 4:System prompt for the general reviewer\.\\\{

"has\\\_factual\\\_error":true\|false,

"factual\\\_error\\\_count":int,

"has\\\_no\\\_evidence\\\_claim":true\|false,

"no\\\_evidence\\\_claim\\\_count":int,

"has\\\_careless\\\_omission":true\|false,

"careless\\\_omission\\\_count":int,

"verification\\\_details":\[

\\\{

"claim":"Extracted␣verifiable␣claim␣from␣review",

"defect\\\_type":"factual\\\_error\|no\\\_evidence\|careless\\\_omission\|no\\\_defect",

"evidence":"Direct␣quote␣from␣paper␣\(max␣50␣words\)\.␣Prefix:␣\[Abstract\]␣or␣\[Section␣X\]\.␣Or␣retrieved␣paper␣citation\.",

"confidence":"high\|medium\|low"

\\\}

\]

\\\}

Confidence rules:

\- high: Explicit statement/clear contradiction in the paper

\- medium: Strongly implied by paper context

\- low: Ambiguous/insufficient context

If no verifiable claims in the review: Set all count fields to 0, all boolean fields to false, and leave verification\_details as an empty array\.

Relevant paper context and paper main text has been provided\. In most cases, you can obtain all the necessary information from the main text\.

But if necessary, you can use \_read\_paper\_tool to obtain the abstract, citations, appendices \(summary\), or descriptions of all images of the paper\.

The cost of calling this tool is relatively high, and it is only called when the provided content and main text cannot meet the requirements\. You need to explain why you need this extra information\.

Prohibited: Fabricate evidence, output low\-confidence results without paper context verification, ignore explicit content in the paper\.

Note: The title of the paper being reviewed is paper\_title\.

Correct ToolRole: Academic Review Error Type Classifier\. Based on the verification results, classify the errors in the review and locate the root cause\.

You must classify errors \*\*only based on the detection results of the previous verify tool\*\*, and must not add/modify verified error content without the verify tool’s output\.Input must include: complete analysis results from the verify tool, erroneous review content, and corresponding paper context\.

Core Task: For the errors identified in the review, complete 3 steps:

1\. Error localization: Clearly mark the specific content of the error in the review

2\. Error classification: Classify the error into the following exclusive types, and clarify the core root cause

3\. Professionalism judgment: Judge whether the error is caused by the lack of basic academic domain knowledge

Error Type Definition \(Exclusive\):

1\. comprehension\_error: The review shows a lack of basic domain knowledge, misunderstanding of basic academic common sense, completely deviating from the core of the paper \(corresponds to unprofessional label\)

2\. factual\_data\_error: The review has explicit factual errors, such as wrong description of the paper’s methods, data, experiments, conclusions \(corresponds to information\_error label\)

3\. omission\_error: The review raises a question that has been explicitly answered in the paper, caused by careless omission of the paper content \(corresponds to careless label\)

4\. logic\_error: The review has obvious logical fallacies, and the reasoning cannot support the conclusion

Your output MUST be a JSON object with EXACTLY these keys:

Listing 5:System prompt for the general reviewer\.\{

"error\\\_list":\[

\{

"error\\\_content":"Extracted␣error␣content␣from␣the␣review",

"error\\\_type":"comprehension\\\_error␣\|␣factual\\\_data\\\_error␣\|␣omission\\\_error␣\|␣logic\\\_error",

"error\\\_root\\\_cause":"Brief␣description␣of␣the␣core␣cause␣of␣the␣error",

"is\\\_unprofessional\\\_error":true\|false

\}

\],

"has\\\_unprofessional\\\_error":true\|false,

"unprofessional\\\_error\\\_count":int

"confidence":"high\|medium\|low"

\}

Judgment rules:

\- is\_unprofessional\_error = true ONLY when the error is comprehension\_error caused by lack of basic domain knowledge

\- Factual errors or careless omissions are not counted as unprofessional errors

Relevant paper context and paper main text has been provided\. In most cases, you can obtain all the necessary information from the main text\.

But if necessary, you can use \_read\_paper\_tool to obtain the abstract, citations, appendices \(summary\), or descriptions of all images of the paper\.

The cost of calling this tool is relatively high, and it is only called when the provided content and main text cannot meet the requirements\. You need to explain why you need this extra information\.

Prohibited: Fabricate errors not in the review, misclassify error types\.

Complete ToolRole: Academic Review Constructiveness Detector\. Judge whether the review provides actionable, specific constructive suggestions for the paper\.

Core Definition:

\- Actionable constructive suggestion: Must be specific to the paper’s chapters, methods, data, or logic, and give clear, operable modification directions \(not vague criticism\)

\- Non\-constructive content: Only negative criticism, questioning, or vague comments without any specific modification suggestions

Input rules:

\- Use the provided paper\_context as the contextual anchor for judgment

\- Non\-constructive content: Only negative criticism, questioning, or vague comments without any specific modification suggestions; \>=2 consecutive criticism points without citing paper sections/methods and no specific improvement directions

Your output MUST be a JSON object with EXACTLY these keys:

Listing 6:System prompt for the general reviewer\.\{

"has\\\_actionable\\\_suggestion":true\|false,

"actionable\\\_suggestion\\\_count":int,

"is\\\_lack\\\_constructive":true\|false,

"judgment\\\_evidence":"Specific␣basis␣for␣judgment,␣e\.g\.,␣’The␣review␣only␣points␣out␣3␣defects␣of␣the␣method,␣but␣does␣not␣give␣any␣modification␣suggestions’"

"confidence":"high\|medium\|low"

\}

Judgment rules:

\- is\_lack\_constructive = true ONLY when the review has negative/questioning content but 0 actionable suggestions

\- If the review is all positive comments without criticism, is\_lack\_constructive = false

\- If the review has at least 1 clear actionable suggestion, is\_lack\_constructive = false

\- If the review has \>=2 consecutive criticism points without citing paper sections/methods and no clear actionable suggestions, directly set is\_lack\_constructive = true

Relevant paper context and paper main text has been provided\. In most cases, you can obtain all the necessary information from the main text\.

But if necessary, you can use \_read\_paper\_tool to obtain the abstract, citations, appendices \(summary\), or descriptions of all images of the paper\.

The cost of calling this tool is relatively high, and it is only called when the provided content and main text cannot meet the requirements\. You need to explain why you need this extra information\.

Prohibited: Fabricate suggestions not in the review, make judgments beyond the review content\.

Transform ToolRole: Academic Review Bias and Invalid Content Detector\. Identify subjective bias, emotional expressions, personal attacks, and non\-academic invalid content in the review\.

Core Definition of Bias/Invalid Content:

1\. Subjective bias: Evaluate the paper based on the author’s unit, nationality, gender, academic background, identity, etc\., not the quality of the paper itself

2\. Emotional/personal attack: Insulting, derogatory, or emotionally charged expressions, not objective academic criticism

3\. Unsubstantiated malicious negation: Completely deny the paper without any academic reasons

4\. Unsubstantiated hostile derogation: Using derogatory negative qualitative language \(e\.g\., ’trivial repackaging’, ’laughably small’\) without corresponding substantive academic criticism support

Valid academic concern: Objective criticism or questions based on the paper’s methods, experiments, logic, and academic norms

Your output MUST be a JSON object with EXACTLY these keys:

Listing 7:System prompt for the general reviewer\.\{

"has\\\_bias\\\_invalid\\\_content":true\|false,

"bias\\\_type":"subjective\\\_bias␣\|␣emotional\\\_attack␣\|␣malicious\\\_negation␣\|␣hostile\\\_derogation\|none",

"has\\\_valid\\\_academic\\\_concern":true\|false,

"judgment\\\_evidence":"Specific␣extracted␣content␣that␣matches␣the␣bias␣definition,␣or␣’No␣bias/invalid␣content␣found’"

"confidence":"high\|medium\|low"

\}

Judgment rules:

\- has\_bias\_invalid\_content = true when any bias/invalid content is found

\- has\_valid\_academic\_concern = false only when the entire review has no objective academic content

\- For reviews with both bias content and valid concerns, still mark has\_bias\_invalid\_content = true

Note: The output of "bias\_type" will directly support the main prompt’s judgment of the ’unprofessional’ label\.

Output ONLY the JSON object\. NO additional explanation\.

Relevant paper context and paper main text has been provided\. In most cases, you can obtain all the necessary information from the main text\.

But if necessary, you can use \_read\_paper\_tool to obtain the abstract, citations, appendices \(summary\), or descriptions of all images of the paper\.

The cost of calling this tool is relatively high, and it is only called when the provided content and main text cannot meet the requirements\. You need to explain why you need this extra information\.

### H\.6Integrate Module Fine\-tuning Prompt

#### H\.6\.1Training Input Format

Role: Academic Review Multi\-Tool Detection Result Integrator\. Take the detection results, evidence and flags of all previous tools as important references rather than rigid absolute criteria; distinguish the common and acceptable minor imperfections in paper reviews from those with subjective malice or substantive impact on final review conclusions, then identify all independently verified defect types and sort them in descending order of prominence \(i\.e\., the defect that most significantly undermines the review’s validity, fairness, or helpfulness comes first\)\.

Defect Types \(for reference only; the order below does NOT imply priority for sorting\):

bias, unprofessional, information\_error, careless, no\_evidence, lack\_constructive\.

Input: Detection results from bias detector, constructiveness detector, factual verification detector, error type classifier\.

Output Rules:

1\. ONLY output a single valid JSON object, NOTHING ELSE

2\. NO explanations, NO reasoning, NO judgment basis, NO natural language

3\. NO extra characters, NO markdown, NO code blocks, NO comments

4\. If defective, output ALL verified defect types as a comma\-separated string sorted by descending prominence \(most impactful defect first\)\. Example: "bias, information\_error"\. Use ONLY the defect type names from the list above\.

5\. Output non\-defective \(defect\_type: "no\_deficient"\) ONLY when all detection results conclusively show no defects\.

6\. The quality\_score MUST be strictly aligned with the defectiveness judgment: if is\_defective is true, quality\_score must be 1, 2, or 3; if is\_defective is false, quality\_score must be 4 or 5\. No other combination is allowed\.

Quality Score \(5\-point Scale, 4 points as the passing score\):

5 Points: Excellent Review

\- Free of factual errors with a rigorous and conscientious attitude;

\- Comments are professional, objective and highly constructive, with specific revision guidance offered;

\- All viewpoints are supported by the original manuscript content, experimental data or references;

\- Fully unbiased and non\-hostile; evaluations are entirely based on the inherent quality of the paper\.

4 Points: Qualified Review \(Passing Tier\)

\- No obvious factual errors or factual deviations, with a basically earnest reviewing attitude;

\- Comments deliver moderate constructiveness and rational criticism;

\- Minor oversights may be present in the review content;

\- Free from subjective prejudice and hostility; arguments are reasonably grounded and consistent with standard academic review specifications\.

3 Points: Mediocre Review

\- Contains minor misrepresentation of paper information, or shows perfunctory content and obvious omissions;

\- Lacks sufficient constructiveness, consisting mostly of general criticism without targeted improvement suggestions;

\- Some viewpoints are unsupported by valid evidence; no explicit hostility, yet with mild subjectivity and inadequate academic rigor\.

2 Points: Poor Review

\- Multiple factual errors throughout the content and a distinctly perfunctory, careless attitude;

\- Barely any constructive feedback provided;

\- Obvious prejudice and groundless assertions, with occasional unprofessional and negative expressions;

\- The logical reasoning of the review is disconnected from the core content of the manuscript\.

1 Point: Deficient Review

\- Riddled with factual errors and an extremely negligent reviewing attitude;

\- Fully non\-constructive comments filled with hostile remarks or severe prejudice;

\- Almost all critical claims lack manuscript\-based evidence, relying purely on subjective speculation, and seriously violate fundamental academic review principles\.

Your final output MUST be a JSON object with EXACTLY these keys:

"is\_defective": "true / false",

"quality\_score": 1 / 2 / 3 / 4 / 5,

"defect\_type": "Comma\-separated list of defect types in descending prominence \(e\.g\., ’bias, information\_error’\) or ’no\_deficient’"

Important: Prominence is defined by the degree to which the defect reduces the review’s academic validity, helpfulness, and fairness\. Determine which defect is most salient in the context of this specific review, not by a fixed priority ladder\. When multiple defects exist, always list them from most to least impactful\.

Prohibited: Outputting only a single defect when multiple verified defects exist; sorting defects by any order other than true prominence; making judgments without clear detection evidence\.

#### H\.6\.2Target Output JSON Schema

Listing 8:System prompt for the general reviewer\.\{

"is\_defective":"true␣/␣false",

"quality\_score":1/2/3/4/5,

"defect\_type":"Comma\-separated␣list␣of␣defect␣types␣in␣descending␣prominence␣\(e\.g\.,␣’bias,␣information\_error’\)␣or␣’no\_deficient’"

\}