Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review

# Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review
Source: [https://arxiv.org/html/2609.05947](https://arxiv.org/html/2609.05947)
Xueyi ZhangWangze NiTianfang XiaoShimin DiJia ZhuZhuoren JiangRong TanLei ChenKui Ren

###### Abstract

Peer review is central to quality control in science\. However, existing evaluations of AI\-assisted peer review mainly focus on the overall quality of generated reviews or the accuracy of final decisions\. They therefore provide limited evidence about whether model decisions are supported by sufficient and reliable review evidence\. We introduce a process\-centric diagnostic benchmark for AI\-assisted peer review\. It uses\(x,zs,zc,zr,y\)\(x,z\_\{s\},z\_\{c\},z\_\{r\},y\)to represent the paper content, summary, critique, suggestion, and decision\. We convert heterogeneous review records from PeerRead, NLPeer ARR\-22, and OpenReview\-ICLR into process\-aligned data\. Our benchmark uses direct decision prediction from the paper content \(Direct\) as its baseline\. It compares the decision value of Gold\-process variables and Predicted\-process variables, and conducts stage\-level evaluation, chain\-consistency evaluation, and interventional sensitivity analysis\. Experiments across three datasets and six models show that Gold\-process variables generally have higher decision value\. For the main analysis model, the Gold–Predicted gap remains stable across datasets and random seeds\. This gap is also reproduced in most model–dataset combinations\. Although model\-generated intermediate review texts show relatively high local consistency across adjacent stages, the final decisions are not consistently supported by the preceding review evidence\. Our benchmark targets AI systems designed to assist rather than replace human reviewers\. It provides a transparent and auditable diagnostic tool for evaluating the reliability of their review processes\.

1Zhejiang University

2Nanyang Technological University

3School of Business, Sun Yat\-sen University

4Southeast University

5Zhejiang Normal University

6DSA, The Hong Kong University of Science and Technology

## Introduction

As submission volumes to artificial intelligence conferences continue to grow, peer review faces an increasing workload\. Review quality, timeliness, and accountability are also under growing pressure\([Kim et al\. 2025](https://arxiv.org/html/2609.05947#bib.bib30)\)\. Large language models have been used to generate review comments, assist with paper analysis, and predict review scores and acceptance decisions\. Some conferences have also begun controlled pilot programs to explore their use in real review workflows\([Association for the Advancement of Artificial Intelligence 2025](https://arxiv.org/html/2609.05947#bib.bib32);[NeurIPS 2026](https://arxiv.org/html/2609.05947#bib.bib33)\)\. However, the growing use of AI\-assisted peer review has raised concerns about fairness and research integrity\. Studies on specific conference data have found that AI\-assisted reviews tend to assign higher scores and may affect the outcomes of papers near the acceptance threshold\([Latona et al\. 2024](https://arxiv.org/html/2609.05947#bib.bib31)\)\. Therefore, human reviewers must retain final decision authority and accountability, while transparent and auditable evaluation methods are needed to assess the reliability of AI\-provided review assistance\.

![Refer to caption](https://arxiv.org/html/2609.05947v1/Figures/Figure.png)Figure 1:From independent\-task evaluation to process\-centric benchmarking for AI\-assisted peer review\.Peer review is a process in which domain experts analyze a paper, identify its main contributions and problems, provide suggestions for improvement, and form an overall assessment\([Liang et al\. 2023](https://arxiv.org/html/2609.05947#bib.bib2);[Idahl and Ahmadi 2025](https://arxiv.org/html/2609.05947#bib.bib3);[Biswas et al\. 2026](https://arxiv.org/html/2609.05947#bib.bib4)\)\. For example, the NeurIPS review form records these review components in separate fields\([Neural Information Processing Systems Foundation 2025](https://arxiv.org/html/2609.05947#bib.bib34)\)\. Inspired by this structure, we represent the core evidence\-forming process of a single review as four stages: summary, critique, suggestion, and decision\. As shown in Figure 1, this representation supports not only the evaluation of the functional quality of each stage, but also the assessment of chain consistency across stages and whether the final decision is sufficiently supported by the preceding review evidence\.

Existing evaluations mainly focus on performance on independent tasks\([Kang et al\. 2018](https://arxiv.org/html/2609.05947#bib.bib1);[Zhou et al\. 2024](https://arxiv.org/html/2609.05947#bib.bib5)\)or on the final review quality and human alignment of process\-based reviewers\([D’Arcy et al\. 2024](https://arxiv.org/html/2609.05947#bib.bib6);[Weng et al\. 2025](https://arxiv.org/html/2609.05947#bib.bib7);[Zhu et al\. 2025](https://arxiv.org/html/2609.05947#bib.bib8);[Garg et al\. 2025](https://arxiv.org/html/2609.05947#bib.bib9)\)\. Few studies compare process\-variable decision value, evaluate chain consistency, and examine the interventional sensitivity of final decisions to intermediate process variables within a unified framework\. Using a multi\-stage or structured procedure to generate reviews does not mean that the intermediate stages and their relationships have been systematically evaluated\. As AI\-assisted peer review enters controlled real\-world pilot programs, closing this evaluation gap is important for improving system transparency, clarifying accountability, and supporting reliable deployment\.

However, process\-level evaluation presents two key challenges\. First, existing peer\-review datasets usually contain heterogeneous paper–review–decision records\. Their field structures, text granularity, and label sources differ, making it difficult to align summary, critique, and suggestion into unified process variables\([Kang et al\. 2018](https://arxiv.org/html/2609.05947#bib.bib1);[Dycke et al\. 2023](https://arxiv.org/html/2609.05947#bib.bib10);[Wang et al\. 2023](https://arxiv.org/html/2609.05947#bib.bib11)\)\. Second, these process variables are open\-ended texts with no single correct expression\. An output that appears reasonable at one stage therefore does not establish consistency across the complete review chain\([Liang et al\. 2023](https://arxiv.org/html/2609.05947#bib.bib2);[Idahl and Ahmadi 2025](https://arxiv.org/html/2609.05947#bib.bib3)\)\. It also does not show that the final decision actually uses the corresponding evidence\. Therefore, process\-level evaluation must assess the functional quality of each stage and examine process\-variable decision value, chain consistency, and the influence of process variables on final decisions\.

To address these challenges, we introduce a process\-centric diagnostic benchmark for AI\-assisted peer review\. To support stage\-wise diagnosis of the core review components, we represent each review instance as\(x,zs,zc,zr,y\)\(x,z\_\{s\},z\_\{c\},z\_\{r\},y\)\. These variables correspond to the paper content, summary, critique, suggestion, and decision\. We then align heterogeneous review records from PeerRead\([Kang et al\. 2018](https://arxiv.org/html/2609.05947#bib.bib1)\), NLPeer ARR\-22\([Dycke et al\. 2023](https://arxiv.org/html/2609.05947#bib.bib10)\), and OpenReview\-ICLR\([Idahl and Ahmadi 2025](https://arxiv.org/html/2609.05947#bib.bib3)\)into unified process\-aligned data\.

Based on this representation, we first compare decision performance under three input settings: paper content only, Gold\-process variables aligned from human review records, and Predicted\-process variables generated by models\. This comparison measures the process\-variable decision value of the two types of process variables\. We then conduct stage\-level evaluation, chain\-consistency evaluation, interventional sensitivity analysis, and conditional error analysis\. Together, these evaluations diagnose how models generate and use review evidence\.

The results show that Gold\-process variables generally have higher decision value\. For the main analysis model, Qwen2\.5\-14B\-Instruct, the Gold–Predicted gap remains stable across datasets and random seeds\. The gap is also reproduced in most model–dataset combinations\. Although model\-generated intermediate review texts show relatively high local consistency across adjacent stages, the final decisions are not consistently supported by the preceding review evidence\. Therefore, final\-label accuracy and overall review quality alone cannot fully evaluate the reliability of a model’s review process\. Our benchmark targets AI systems designed to assist rather than replace human reviewers\. It evaluates the reliability of the review evidence provided by these systems and does not treat model outputs as a standalone basis for acceptance decisions\.

The main contributions of this work are as follows:

- •We introduce a process\-centric diagnostic benchmark for AI\-assisted peer review\. It uses\(x,zs,zc,zr,y\)\(x,z\_\{s\},z\_\{c\},z\_\{r\},y\)to represent the paper content, summary, critique, suggestion, and decision\. Based on this unified representation, we convert heterogeneous papers, human reviews, and decision records from PeerRead, NLPeer ARR\-22, and OpenReview\-ICLR into comparable process\-aligned data\.
- •We develop a process\-level evaluation framework that compares the decision value of Gold\-process variables and Predicted\-process variables\. It also conducts stage\-level evaluation, chain\-consistency evaluation, interventional sensitivity analysis, and conditional error analysis to systematically diagnose how models generate and use review evidence\.
- •We conduct systematic experiments across three datasets and six models\. The results reveal a decision\-value gap between Gold\-process variables and Predicted\-process variables\. Although model\-generated intermediate review texts show relatively high local consistency across adjacent stages, the final decisions are not consistently supported by the preceding review evidence\.

## Related Work

AI\-Assisted Peer Review\.AI\-assisted peer review mainly covers review generation, score prediction, and acceptance prediction\. Early work explored the automatic generation of initial review comments\([Yuan et al\. 2022](https://arxiv.org/html/2609.05947#bib.bib12)\), while PeerRead enabled computational analysis of review texts and acceptance decisions\([Kang et al\. 2018](https://arxiv.org/html/2609.05947#bib.bib1)\)\. More recent systems use hierarchical question answering, retrieval augmentation, iterative reflection, and multi\-agent collaboration to build stronger reviewers\. Representative systems include TreeReview\([Chang et al\. 2025](https://arxiv.org/html/2609.05947#bib.bib13)\), ScholarPeer\([Goyal et al\. 2026](https://arxiv.org/html/2609.05947#bib.bib14)\), ReviewerTool\([Sahu et al\. 2025](https://arxiv.org/html/2609.05947#bib.bib15)\), and DeepReviewer\([Zhu et al\. 2025](https://arxiv.org/html/2609.05947#bib.bib8)\)\. These methods improve the depth, factuality, or structure of generated reviews, but their main goal remains improving the reviewer or its final output\. In contrast, our work examines whether intermediate review evidence contains decision\-relevant information, remains consistent across stages, and is reflected in the final decision\.

Process\-Oriented Evaluation\.Evaluation has expanded from score or decision accuracy to multidimensional review quality and human alignment\. Beyond Rating\([Li et al\. 2026](https://arxiv.org/html/2609.05947#bib.bib16)\), ReviewEval\([Garg et al\. 2025](https://arxiv.org/html/2609.05947#bib.bib9)\), MMReview\([Gao et al\. 2025](https://arxiv.org/html/2609.05947#bib.bib17)\), PRISM\([Loc et al\. 2026](https://arxiv.org/html/2609.05947#bib.bib18)\), and CoCoReviewBench\([Deng et al\. 2026](https://arxiv.org/html/2609.05947#bib.bib19)\)evaluate generated reviews in terms of factuality, completeness, constructiveness, and alignment with human reviews\. Concern\-level matching further supports more fine\-grained diagnosis\([Jin 2026](https://arxiv.org/html/2609.05947#bib.bib20)\)\. Existing data resources provide complementary foundations\. PeerRead contains papers, reviews, and acceptance labels\([Kang et al\. 2018](https://arxiv.org/html/2609.05947#bib.bib1)\), while NLPeer standardizes multiple peer\-review data sources\([Dycke et al\. 2023](https://arxiv.org/html/2609.05947#bib.bib10)\)\. PeerSum\([Li et al\. 2022](https://arxiv.org/html/2609.05947#bib.bib21)\), MOPRD\([Lin et al\. 2023](https://arxiv.org/html/2609.05947#bib.bib22)\), and resources for peer\-review argumentation\([Fromm et al\. 2021](https://arxiv.org/html/2609.05947#bib.bib23)\)support review summarization, multi\-stage review records, and discourse analysis, respectively\. However, existing evaluations usually focus on individual tasks or final review quality\. Few align summary, critique, suggestion, and decision as unified and intervenable process variables\. We provide such a unified representation and evaluate stage\-level quality, process\-variable decision value, chain consistency, interventional sensitivity, and conditional error associations\.

![Refer to caption](https://arxiv.org/html/2609.05947v1/Figures/figure2-2.png)Figure 2:Overview of the proposed process\-centric diagnostic benchmark for AI\-assisted peer review\. It aligns heterogeneous peer\-review records into process variables and evaluates their decision value, stage quality, chain consistency, interventional sensitivity, and conditional error relations\.
## Process\-Centric Benchmark

We introduce a process\-centric diagnostic benchmark for AI\-assisted peer review\. As shown in Figure 2, the benchmark converts heterogeneous peer\-review records into process\-aligned data\. It analyzes how models generate and use review evidence by comparing the decision value of Gold\-process and Predicted\-process variables and conducting process\-level diagnostic evaluation\.

### Problem Formulation

Existing outcome\-level evaluations typically model AI\-assisted peer review as the direct prediction of a final decisionyyfrom paper contentxx:y^direct=fdirect​\(x\),y∈\{accept,reject\}\.\\hat\{y\}^\{\\mathrm\{direct\}\}=f\_\{\\mathrm\{direct\}\}\(x\),y\\in\\\{\\mathrm\{accept\},\\mathrm\{reject\}\\\}\.This setting measures whether the final label is correct, but it cannot reveal whether the decision is supported by intermediate review evidence\. To enable process\-level diagnosis, we represent each review sample as\(x,zs,zc,zr,y\)\(x,z\_\{s\},z\_\{c\},z\_\{r\},y\)\. Here,xxdenotes the paper content;zsz\_\{s\},zcz\_\{c\}, andzrz\_\{r\}denote the summary, critique, and suggestion; andyydenotes the final decision\. It enables separate evaluation, combination, and intervention of these stages\. Gold\-process variables are aligned from human review records and denoted asZG=\(zs,zc,zr\)Z\_\{G\}=\(z\_\{s\},z\_\{c\},z\_\{r\}\)\. Predicted\-process variables are generated by the evaluated model and denoted asZ^P=\(z^s,z^c,z^r\)\\hat\{Z\}\_\{P\}=\(\\hat\{z\}\_\{s\},\\hat\{z\}\_\{c\},\\hat\{z\}\_\{r\}\)\. Gold\-process variables serve as reference representations for evaluation rather than as a unique correct human reasoning chain\. Based on this representation, we evaluate the functional quality of each stage and compare the decision value of the two types of process variables\. We also examine whether the model\-generated stages form a mutually supportive review chain and how intermediate review evidence relates to the final decision\.

### Process\-Aligned Data Construction

We construct process\-aligned data from PeerRead, NLPeer ARR\-22, and OpenReview\-ICLR\. These datasets cover an early peer\-review corpus, the ACL Rolling Review process, and public ICLR review records, respectively, and differ in field structure, text granularity, and label sources\. We use field\-aware alignment:zsz\_\{s\}uses reviewer\-written summaries when available and paper abstracts otherwise;zcz\_\{c\}andzrz\_\{r\}are constructed from weakness\-related and suggestion\-related fields, respectively, with rule\-based extraction from review text when structured fields are missing\. The aligned\(zs,zc,zr\)\(z\_\{s\},z\_\{c\},z\_\{r\}\)form the Gold\-process variablesZGZ\_\{G\}\. This conversion is more than text cleaning: it creates a shared process\-aligned representation in which process variables can be independently evaluated, compared, and intervened on under a common protocol\.

To enable cross\-dataset comparison, we normalize decisions to accept/reject\. PeerRead uses original labels and OpenReview\-ICLR uses official decision notes\. NLPeer ARR\-22 lacks stable official binary acceptance decisions, so we construct a score\-based proxy decision using an average review\-score threshold of 3\.5, which lies near the midpoint of the rating scale\. We filter samples without valid paper text or decision labels; when a particular process variable is unavailable, the sample is excluded only from evaluations that require that variable\. The resulting datasets contain 562, 364, and 12,002 samples, respectively\. We use the full test splits of PeerRead and NLPeer\. For OpenReview\-ICLR, we draw 500 label\-stratified test samples per seed to control multi\-stage inference cost while approximately preserving the original label distribution\. Complete field mappings, filtering rules, and sensitivity results are provided in Appendix A\.

### Evaluation Tasks and Stages

As shown in Figure 1, we represent peer review as four stages: summary, critique, suggestion, and decision\. Our six\-stage protocol evaluates them using the Gold\-process variablesZGZ\_\{G\}and Predicted\- process variablesZ^P\\hat\{Z\}\_\{P\}from Section 3\.1\.

Stage 1: Direct Tasks\.This stage evaluates the model’s ability to perform basic review tasks and establishes a Direct baseline without intermediate process variables\. The model performs two independent tasks: generating a summaryz^s\\hat\{z\}\_\{s\}from the paper contentxx, and directly predicting a decisiony^direct\\hat\{y\}^\{\\mathrm\{direct\}\}fromxxalone\.

Stage 2: Gold\-process Decision Ablation\.This stage compares the relative decision value of different Gold\-process variables and their combinations\. The model predicts the decision using the seven non\-empty subsets ofZGZ\_\{G\}\. We refer to the setting that uses the completeZGZ\_\{G\}alone as Gold\-only\. We also include Paper \+ Gold, which jointly inputs the paperxxand the completeZGZ\_\{G\}\. Comparing it with Gold\-only measures the effect of adding paper content\.

Stage 3: Predicted\-process Generation and Decision\.This stage evaluates whether model\-generated process variables can support the final decision\. The model independently generatesz^s\\hat\{z\}\_\{s\}andz^c\\hat\{z\}\_\{c\}from the paperxx, and then generatesz^r\\hat\{z\}\_\{r\}from\(x,z^c\)\(x,\\hat\{z\}\_\{c\}\)\. It predicts the decision using the seven non\-empty subsets ofZ^P\\hat\{Z\}\_\{P\}\. We refer to the setting that uses the completeZ^P\\hat\{Z\}\_\{P\}alone as Pred\-only\. Paper \+ Pred jointly inputs\(x,Z^P\)\(x,\\hat\{Z\}\_\{P\}\)to test whether Predicted\-process variables provide additional information when the paper content is available\.

Stages 2 and 3 use the same seven process\-variable subsets\. We compare the decision performance of each corresponding pair to measure the decision\-value gap between Gold\-process variables and Predicted\-process variables\.

Stages 4–6 focus on Predicted\-process variables\. Their goal is to diagnose how the model generates and uses its own review evidence\. Gold\-process variables mainly serve as reference representations for comparing decision value, so they are not included in the subsequent process diagnosis\.

Stage 4: Chain\-consistency Evaluation\.This stage evaluates whether the model\-generated stage outputs form a mutually supportive review chain\. Summary–critique consistency \(S–C\) examines whether the predicted critique logically follows from, or is at least compatible with, the predicted summary\. Critique–suggestion consistency \(C–S\) checks whether the predicted suggestion addresses the critique; process–decision consistency \(P–D\) checks whether the critique and suggestion support the final decision\.

Stage 5: Interventional Sensitivity Analysis\.This stage tests whether the final decision responds to controlled changes in model\-generated process variables\. We intervene on only one variable,z^s\\hat\{z\}\_\{s\},z^c\\hat\{z\}\_\{c\}, orz^r\\hat\{z\}\_\{r\}, at a time while keeping the other process variables and the decision model unchanged\. We use four intervention types: positive, negative, neutral, and remove\. The first three preserve the original process text and add a statement that supports acceptance, supports rejection, or has no clear decision preference\. The remove intervention replaces the target variable with an explicit missing\-value placeholder\. We then repredict decisions under Pred\-only \(process\-only\) and Paper \+ Pred \(paper\-conditioned\) and compare them with their baselines\.

Stage 6: Conditional Error Analysis\.This stage examines whether final decision performance decreases when intermediate review stages fail to meet predefined quality conditions\. For summary, critique, and suggestion, we divide samples into two groups according to the stage\-level metrics: those that meet the quality condition and those that do not\. We compare decision accuracy between the two groups under both process\-only and paper\-conditioned settings\. The accuracy gap measures a potential error\-propagation relationship, but it represents only a statistical association and does not provide causal evidence\. Appendices B and C detail all process\-variable combinations, inputs, outputs, prompts, intervention templates, and quality conditions\.

### Evaluation Metrics

We use task\-specific metrics to evaluate stage\-level quality, process variable decision value, chain consistency, interventional sensitivity, and conditional error analysis\. Because these metrics measure different functions on different scales, we report them separately\.

Stage\-level Metrics\.We use metrics that match the function of each review stage\. For summary, ROUGE\-1/2/L\([Lin 2004](https://arxiv.org/html/2609.05947#bib.bib24)\)and BERTScore\-F1\([Zhang et al\. 2020](https://arxiv.org/html/2609.05947#bib.bib25)\)measure lexical overlap and semantic similarity with the reference summary\.

A critique may contain several distinct weaknesses\. We therefore split generated and reference critiques into weakness units, then greedily match them one\-to\-one using unigram/bigram TF–IDF cosine similarity\([Salton and Buckley 1988](https://arxiv.org/html/2609.05947#bib.bib26)\)\. Matches require similarity of at leastτc=0\.25\\tau\_\{c\}=0\.25\. Weakness Precision, Recall, and F1 measure matching from the predicted and reference sides; Coverage is the fraction of reference weakness units that are matched\. This unit\-level protocol identifies whether the model covers the specific issues raised in reference reviews, rather than relying on similarity between entire critique texts\.

For suggestion, direct matching with a reference suggestion is not required because one weakness can admit multiple valid suggestions\. Critique–Suggestion Alignment is the fraction of predicted critique units matched by at least one generated suggestion, usingτr=0\.20\\tau\_\{r\}=0\.20\. A fixed LLM Judge\([Liu et al\. 2023](https://arxiv.org/html/2609.05947#bib.bib27);[Zheng et al\. 2023](https://arxiv.org/html/2609.05947#bib.bib28)\)additionally evaluates the paper, predicted critique, and generated suggestion for relevance, specificity, actionability, and overall quality on a 1–5 scale, where higher scores are better\. For decision, we use Macro\-F1 as the primary metric and also report Accuracy and ROC\-AUC\([Sokolova and Lapalme 2009](https://arxiv.org/html/2609.05947#bib.bib35);[Fawcett 2006](https://arxiv.org/html/2609.05947#bib.bib29)\)\. The matching thresholds define operational matching rules rather than universal quality cutoffs; implementation details and sensitivity analyses are provided in Appendix C\.

Process\-variable Decision Value\.To compare the reference and model\-generated representations of the same process\-variable subset, letA⊆\{s,c,r\}A\\subseteq\\\{s,c,r\\\}denote any non\-empty subset\. We defineΔA=M⁡\(y^Agold,y\)−M⁡\(y^Apred,y\),\\Delta\_\{A\}=M\(\\hat\{y\}\_\{A\}^\{\\mathrm\{gold\}\},y\)\-M\(\\hat\{y\}\_\{A\}^\{\\mathrm\{pred\}\},y\),whereMMdenotes Macro\-F1\. IfΔA\>0\\Delta\_\{A\}\>0, the corresponding Gold\-process subset has higher decision value\. For the complete process, we denote the gap asΔprocess=Δ\{s,c,r\}\.\\Delta\_\{\\mathrm\{process\}\}=\\Delta\_\{\\\{s,c,r\\\}\}\.

Chain Consistency\.A fixed LLM Judge evaluates three relations: summary–critique \(S–C\), critique–suggestion \(C–S\), and process–decision \(P–D\)\. Each relation receives a score from 1 to 5\. A higher score indicates stronger support across stages\. Overall CC is the arithmetic mean of the three scores\. These metrics measure support between stages rather than the independent quality of individual outputs\. Appendix F provides the complete scoring rubric\.

Interventional Sensitivity\.Letpip\_\{i\}andy^i\\hat\{y\}\_\{i\}denote the acceptance probability and decision for sampleiibefore intervention\. Letpi\(k,t\)p\_\{i\}^\{\(k,t\)\}andy^i\(k,t\)\\hat\{y\}\_\{i\}^\{\(k,t\)\}denote the corresponding results after applying intervention typettto process variablekk\. We computeΔ​Pk,t=1N​∑i=1N\(pi\(k,t\)−pi\)\\Delta P\_\{k,t\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\left\(p\_\{i\}^\{\(k,t\)\}\-p\_\{i\}\\right\)andDCRk,t=1N∑i=1N𝕀\[y^i\(k,t\)≠y^i\]\.\\mathrm\{DCR\}\_\{k,t\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbb\{I\}\\left\[\\hat\{y\}\_\{i\}^\{\(k,t\)\}\\neq\\hat\{y\}\_\{i\}\\right\]\.Here,Δ​P\\Delta Pmeasures the mean acceptance\-probability shift, and DCR the fraction of labels changed by intervention\. We compute both under process\-only and paper\-conditioned settings\.

Conditional Error Analysis\.We define the error\-propagation metric asEPk→y=Acc⁡\(y^,y∣ek=0\)−Acc⁡\(y^,y∣ek=1\),\\mathrm\{EP\}\_\{k\\rightarrow y\}=\\operatorname\{Acc\}\(\\hat\{y\},y\\mid e\_\{k\}=0\)\-\\operatorname\{Acc\}\(\\hat\{y\},y\\mid e\_\{k\}=1\),whereek=0e\_\{k\}=0andek=1e\_\{k\}=1indicate that stagekkmeets or does not meet the predefined quality condition, respectively\. A positive value indicates that lower stage quality is associated with lower final decision accuracy\. This metric measures only a statistical association and does not provide causal evidence\. Appendices C, F, G, and H provide the complete threshold sensitivity analyses, Judge rubrics, intervention templates, and conditional\-group statistics\.

## Experiment

This section reports results for the experimental stages defined in Section 3\.3 and further examines whether the main findings generalize across models and datasets\.

### Experimental setup

We evaluate six models: Qwen2\.5\-7B\-Instruct, Qwen2\.5\-14B\-Instruct, Llama\-3\.1\-8B\-Instruct, Llama\-3\.3\-70B, GPT\-4\.1, and DeepReviewer\-7B\. They cover general\-purpose open models of different scales, a proprietary API model, and a peer\-review\-oriented model\. This selection tests whether the main findings generalize across model scale, access mode, and domain adaptation\. Qwen2\.5\-14B\-Instruct is the main analysis model because it can reproducibly run the complete six\-stage protocol under the shared 32K input budget\. We run it on all datasets with seeds 42, 123, and 2024\. The remaining five models run Stages 1–3 on the seed\-42 splits of PeerRead and NLPeer ARR\-22 to compare process\-variable decision value across models\. All models use shared prompts, a 32K full\-if\-fit plus section\-aware fallback input policy, and deterministic decoding \(temperature=0=0\)\. Main\-model results are reported as mean±\\pmsample SD over three seeds\. Model access, full run coverage, generation lengths, and output protocols are provided in Appendix B\.

### Process\-variable Decision Value: Gold versus Predicted Process

This section jointly analyzes Stages 2 and 3\. We examine whether Gold\-process variables can support the final decision, whether the model can generate Predicted\-process variables with decision value, and how large the gap between them remains\. Table 1 compares Direct, the seven corresponding Gold\-process and Predicted\-process subsets, and the paper\-conditioned settings for the complete process\. It reports decision Macro\-F1 as Mean±\\pmStd over three seeds, together withΔA\\Delta\_\{A\}\. Appendix D reports Accuracy, class\-wise F1, AUC, per\-seed results, and prediction distributions\.

Table 1:Decision Macro\-F1 across input settings \(mean±\\pmsample SD over three seeds\)\. G and P denote Gold\-process and Predicted\-process results;ΔA\\Delta\_\{A\}is the paired per\-seed differenceG−P\\mathrm\{G\}\-\\mathrm\{P\}\. Bold marks the best G and P results per dataset\.Gold\-process variables have higher decision value, but a gap remains between Gold\-process and Predicted\-process variables\.Gold\-process outperforms the corresponding Predicted\-process in all 21 subset–dataset comparisons\. Complete Gold\-only improves over Direct by 0\.336–0\.434\. Pred\-only also improves over Direct by 0\.194–0\.275, but remains 0\.142–0\.159 below Gold\-only\. The best variable combination differs across datasets\.Critique \+ Suggestion leads on PeerRead, while the complete process leads on NLPeer ARR\-22 and OpenReview\-ICLR\.

Joint input with the paper does not ensure effective use of process variables\.Paper \+ Gold outperforms Direct on all three datasets\. In contrast, Paper \+ Pred improves over Direct by no more than 0\.010 and performs substantially worse than Pred\-only\. These results show that Predicted\-process variables have some decision value but are not consistently used in the current joint\-input setting\.

### Stage\-level Evaluation

This section evaluates the local performance of the model at four review stages: summary, critique, suggestion, and decision\. It also identifies possible sources of the Gold–Predicted gap observed in Section 4\.2\. Table 2 reports Mean±\\pmStd results for Qwen2\.5\-14B\-Instruct over three seeds\. Appendix E provides the complete per\-seed results\.

Table 2:Stage\-level results for Qwen2\.5\-14B\-Instruct \(mean±\\pmsample SD over three seeds\)\. BERT\-F1, Weak\.\-F1, and C–S Align\. denote BERTScore\-F1, Weakness F1, and Critique–Suggestion Alignment\. Judge Overall uses a 1–5 scale; all other metrics use a 0–1 scale\.Performance varies substantially across review stages\.Summary BERTScore\-F1 is similar across the three datasets \(0\.866–0\.890\)\. In contrast, predicted critiques show very low strict alignment with reference weaknesses, with both Weakness F1 and Coverage close to zero\. This does not imply that the generated critiques are entirely invalid\. Rather, identified issues rarely match specific reference weaknesses under this protocol, which may contribute to the Gold–Predicted gap\.

High local suggestion quality does not translate into reliable decisions\.The LLM Judge assigns suggestion Overall scores of 4\.336–4\.493, whereas Critique–Suggestion Alignment remains only 0\.484–0\.495\. Direct Macro\-F1 is also low \(0\.235–0\.347\) and shows a clear accept bias\. Overall, the model can generate locally plausible review text, but still struggles to align critiques with reference weaknesses, respond comprehensively to preceding critiques, and distinguish between accept and reject decisions\. Next, we test whether they form a mutually supportive review chain\.

### Chain\-consistency Evaluation

This section examines whether the stage outputs in the Predicted\-process form a coherent and mutually supportive review process\. Table 3 reports the Judge’s raw scores on a 1–5 scale as mean±\\pmsample standard deviation over three seeds\. The complete scoring rubric, numbers of valid samples, and per\-seed results are provided in Appendix F\.

Table 3:Predicted\-process chain consistency \(mean±\\pmsample SD over three seeds\)\. Fixed\-judge scores range from 1 to 5; Overall CC averages S–C, C–S, and P–D\.The final decision is the main break in the review chain\.S–C scores are close to 4\.0 and C–S scores are close to 5\.0 across all three datasets\. In contrast, P–D scores are only 1\.31–1\.41, with 97\.0%–97\.9% of samples receiving scores of 1 or 2\. These results show high local consistency in generated intermediate texts, but final decisions often lack support from preceding critiques and suggestions\.

Conditional error analysis does not reveal a consistent error\-propagation pattern\.The conditional accuracy gaps for summary are small and vary in direction, while the estimates for critique and suggestion are affected by severe imbalance between the condition groups\. These results mainly reveal the limitations of converting stage quality into binary error labels and should not be interpreted as evidence of causal error propagation\. The next section uses controlled interventions to examine more directly whether the final decision responds to changes in intermediate process variables\.

### Interventional Sensitivity Analysis

This section examines whether the final decision changes in response to controlled modifications of intermediate process variables\. The main analysis uses the process\-only setting\. We repeat the same interventions under the paper\-conditioned setting to examine whether the model still uses process evidence when the paper content is included\. Both settings share templates, model, and metrics\.

![Refer to caption](https://arxiv.org/html/2609.05947v1/Figures/figure9_intervention_heatmaps.png)Figure 3:Process\-only intervention effects on Predicted\-process variables \(three\-seed means\)\. Top: acceptance\-probability changeΔ​P\\Delta P; bottom: Decision Change Rate \(DCR\)\. Positive and negativeΔ​P\\Delta Pfavor accept and reject, respectively\.Table 4:Cross\-model decision Macro\-F1 on the seed\-42 test splits\. D, G, and P denote Direct, Gold\-process, and Predicted\-process; Gap isG−P\\mathrm\{G\}\-\\mathrm\{P\}\. Bold marks the best setting per model–dataset pair\.The final decision responds to controlled changes in process variables, with critique having the largest effect\.As shown in Figure 3, positive and negative interventions produce the expected directions ofΔ​P\\Delta Pacross all three datasets\. Critique has an average\|Δ​P\|\|\\Delta P\|of 0\.468–0\.491 and a DCR of 0\.485–0\.523, both higher than those of summary and suggestion\. This shows that the final decision is most sensitive to explicit evaluative evidence in the critique\.

Including the paper weakens the effects of most process interventions\.Under the paper\-conditioned setting, most interventions have substantially smaller effects\. However, negative critique can still overturn the original accept bias, with a DCR of 0\.987–0\.997 and a decrease of 0\.927–0\.963 in acceptance probability\. This asymmetric response shows that the model reacts to changes in process evidence but does not yet use such evidence in a balanced and stable manner\. Full process\-only and paper\-conditioned results are in Appendix G\. These results indicate sensitivity, not causality\.

### Model and Dataset Generalization

This section examines whether the preceding findings generalize across models and datasets\. We compare Qwen2\.5\-7B, Qwen2\.5\-14B, Llama\-3\.1\-8B, Llama\-3\.3\-70B, GPT\-4\.1, and DeepReviewer\-7B on the same seed\-42 test splits of PeerRead and NLPeer ARR\-22\. All models use the same process\-variable definitions, input structure, and decision\-evaluation protocol\. Table 4 reports Macro\-F1 for the Direct, Gold\-process, and Predicted\-process settings\. Appendix I reports additional metrics and prediction distributions; Appendix D reports all main\-model results across datasets and seeds\.

The decision advantage of Gold\-process variables generalizes across general\-purpose models\.As shown in Table 4, Gold\-process outperforms Predicted\-process in all ten model–dataset combinations formed by five general\-purpose models and two datasets\. The gaps range from 0\.071 to 0\.265 on PeerRead and from 0\.137 to 0\.360 on NLPeer ARR\-22\. Clear gaps remain for Llama\-3\.3\-70B and GPT\-4\.1, showing that larger models do not consistently close the gap\.

A smaller gap does not necessarily indicate stronger process generation\.On PeerRead, DeepReviewer\-7B obtains similar Gold\-process and Predicted\-process results, but both Macro\-F1 scores are approximately 0\.35\. Its small gap therefore results mainly from weak Gold\-process performance\. The improvement of Predicted\-process over Direct also varies across models and datasets\. Overall, the higher decision value of Gold\-process variables generalizes across models, whereas the ability to generate and use effective process variables remains inconsistent\.

## Conclusion

We introduce a process\-centric diagnostic benchmark for AI\-assisted peer review\. It aligns paper content, summary, critique, suggestion, and decision into a unified process\-aligned representation and evaluates stage\-level quality, process\-variable decision value, chain consistency, and interventional sensitivity\. Experiments across datasets and models show that Gold\-process variables have higher decision value than Predicted\-process variables\. Although model\-generated review texts show high local consistency across adjacent stages, final decisions are not consistently supported by preceding review evidence\.

Our benchmark is intended to support, rather than replace, human reviewers\. It provides a transparent and auditable diagnostic tool for identifying deficiencies in review evidence generation and use\. Future work will incorporate expert validation and further examine bias and uncertainty to support the safe and reliable use of AI in peer review under human oversight\.

## References

- Association for the Advancement of Artificial Intelligence \(2025\)Association for the Advancement of Artificial IntelligenceAI\-assisted peer\-review process: pilot program\.Note:https://aaai\.org/conference/aaai/aaai\-26/main\-technical\-track\-call/AAAI\-26 Main Technical Track; accessed July 25, 2026Cited by:[Introduction](https://arxiv.org/html/2609.05947#Sx1.p1.1)\.
- Biswaset al\.\(2026\)J\. Biswas, S\. Schoepp, G\. Vasan, A\. Opipari, A\. Zhang, Z\. Hu, S\. Joseph, M\. Lease, J\. J\. Li, P\. Stone,et al\.AI\-assisted peer review at scale: the aaai\-26 ai review pilot\.arXiv preprint arXiv:2604\.13940\.Cited by:[Introduction](https://arxiv.org/html/2609.05947#Sx1.p2.1)\.
- Changet al\.\(2025\)Y\. Chang, Z\. Li, H\. Zhang, Y\. Kong, Y\. Wu, H\. K\. So, Z\. Guo, L\. Zhu, and N\. WongTreeReview: a dynamic tree of questions framework for deep and efficient llm\-based scientific peer review\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 15662–15693\.Cited by:[Related Work](https://arxiv.org/html/2609.05947#Sx2.p1.1)\.
- Denget al\.\(2026\)H\. Deng, X\. Ke, Y\. Li, R\. Hu, D\. Huang, D\. F\. Wong, Y\. Wang, X\. Liu, and M\. ZhangCoCoReviewBench: a completeness\-and correctness\-oriented benchmark for ai reviewers\.arXiv preprint arXiv:2605\.07905\.Cited by:[Related Work](https://arxiv.org/html/2609.05947#Sx2.p2.1)\.
- Dyckeet al\.\(2023\)N\. Dycke, I\. Kuznetsov, and I\. GurevychNLPeer: a unified resource for the computational study of peer review\.InProceedings of the 61st annual meeting of the Association for Computational Linguistics \(volume 1: Long papers\),pp\. 5049–5073\.Cited by:[Introduction](https://arxiv.org/html/2609.05947#Sx1.p4.1),[Introduction](https://arxiv.org/html/2609.05947#Sx1.p5.1),[Related Work](https://arxiv.org/html/2609.05947#Sx2.p2.1)\.
- D’Arcyet al\.\(2024\)M\. D’Arcy, T\. Hope, L\. Birnbaum, and D\. DowneyMarg: multi\-agent review generation for scientific papers\.arXiv preprint arXiv:2401\.04259\.Cited by:[Introduction](https://arxiv.org/html/2609.05947#Sx1.p3.1)\.
- Fawcett \(2006\)T\. FawcettAn introduction to ROC analysis\.Pattern Recognition Letters27\(8\),pp\. 861–874\.External Links:[Document](https://dx.doi.org/10.1016/j.patrec.2005.10.010)Cited by:[Evaluation Metrics](https://arxiv.org/html/2609.05947#Sx3.SSx4.p4.1)\.
- Frommet al\.\(2021\)M\. Fromm, E\. Faerman, M\. Berrendorf, S\. Bhargava, R\. Qi, Y\. Zhang, L\. Dennert, S\. Selle, Y\. Mao, and T\. SeidlArgument mining driven analysis of peer\-reviews\.InProceedings of the AAAI conference on artificial intelligence,Vol\.35,pp\. 4758–4766\.Cited by:[Related Work](https://arxiv.org/html/2609.05947#Sx2.p2.1)\.
- Gaoet al\.\(2025\)X\. Gao, J\. Ruan, Z\. Zhang, J\. Gao, T\. Liu, and Y\. FuMMReview: a multidisciplinary and multimodal benchmark for llm\-based peer review automation\.arXiv preprint arXiv:2508\.14146\.Cited by:[Related Work](https://arxiv.org/html/2609.05947#Sx2.p2.1)\.
- Garget al\.\(2025\)M\. K\. Garg, T\. Prasad, T\. Singhal, C\. Kirtani, M\. Mandal, and D\. KumarRevieweval: an evaluation framework for ai\-generated reviews\.arXiv preprint arXiv:2502\.11736\.Cited by:[Introduction](https://arxiv.org/html/2609.05947#Sx1.p3.1),[Related Work](https://arxiv.org/html/2609.05947#Sx2.p2.1)\.
- Goyalet al\.\(2026\)P\. Goyal, M\. Parmar, Y\. Song, H\. Palangi, T\. Pfister, and J\. YoonScholarPeer: a context\-aware multi\-agent framework for automated peer review\.arXiv preprint arXiv:2601\.22638\.Cited by:[Related Work](https://arxiv.org/html/2609.05947#Sx2.p1.1)\.
- Idahl and Ahmadi \(2025\)M\. Idahl and Z\. AhmadiOpenreviewer: a specialized large language model for generating critical scientific paper reviews\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(System Demonstrations\),pp\. 550–562\.Cited by:[Introduction](https://arxiv.org/html/2609.05947#Sx1.p2.1),[Introduction](https://arxiv.org/html/2609.05947#Sx1.p4.1),[Introduction](https://arxiv.org/html/2609.05947#Sx1.p5.1)\.
- Jin \(2026\)M\. JinWhat makes a good ai review? concern\-level diagnostics for ai peer review\.arXiv preprint arXiv:2604\.19998\.Cited by:[Related Work](https://arxiv.org/html/2609.05947#Sx2.p2.1)\.
- Kanget al\.\(2018\)D\. Kang, W\. Ammar, B\. Dalvi, M\. Van Zuylen, S\. Kohlmeier, E\. Hovy, and R\. SchwartzA dataset of peer reviews \(peerread\): collection, insights and nlp applications\.InProceedings of the 2018 conference of the North American chapter of the Association for Computational Linguistics: Human language technologies, volume 1 \(long papers\),pp\. 1647–1661\.Cited by:[Introduction](https://arxiv.org/html/2609.05947#Sx1.p3.1),[Introduction](https://arxiv.org/html/2609.05947#Sx1.p4.1),[Introduction](https://arxiv.org/html/2609.05947#Sx1.p5.1),[Related Work](https://arxiv.org/html/2609.05947#Sx2.p1.1),[Related Work](https://arxiv.org/html/2609.05947#Sx2.p2.1)\.
- Kimet al\.\(2025\)J\. Kim, Y\. Lee, and S\. LeePosition: the AI conference peer review crisis demands author feedback and reviewer rewards\.InProceedings of the 42nd International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=l8QemUZaIA)Cited by:[Introduction](https://arxiv.org/html/2609.05947#Sx1.p1.1)\.
- Latonaet al\.\(2024\)G\. R\. Latona, M\. H\. Ribeiro, T\. R\. Davidson, V\. Veselovsky, and R\. WestThe AI review lottery: widespread AI\-assisted peer reviews boost paper scores and acceptance rates\.arXiv preprint arXiv:2405\.02150\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2405.02150),[Link](https://arxiv.org/abs/2405.02150)Cited by:[Introduction](https://arxiv.org/html/2609.05947#Sx1.p1.1)\.
- Liet al\.\(2026\)B\. Li, H\. Ma, Y\. Wang, J\. Yang, Y\. Zheng, X\. Chen, X\. Huang, and X\. QiuBeyond rating: a comprehensive evaluation and benchmark for ai reviews\.arXiv preprint arXiv:2604\.19502\.Cited by:[Related Work](https://arxiv.org/html/2609.05947#Sx2.p2.1)\.
- Liet al\.\(2022\)M\. Li, J\. Qi, and J\. H\. LauPeersum: a peer review dataset for abstractive multi\-document summarization\.arXiv preprint arXiv:2203\.01769\.Cited by:[Related Work](https://arxiv.org/html/2609.05947#Sx2.p2.1)\.
- Lianget al\.\(2023\)W\. Liang, Y\. Zhang, H\. Cao, B\. Wang, D\. Ding, X\. Yang, K\. Vodrahalli, S\. He, D\. Smith, Y\. Yin,et al\.Can large language models provide useful feedback on research papers? a large\-scale empirical analysis\. arxiv\.arXiv preprint arXiv:2310\.01783\.Cited by:[Introduction](https://arxiv.org/html/2609.05947#Sx1.p2.1),[Introduction](https://arxiv.org/html/2609.05947#Sx1.p4.1)\.
- Lin \(2004\)C\. LinROUGE: a package for automatic evaluation of summaries\.InText Summarization Branches Out,pp\. 74–81\.External Links:[Link](https://aclanthology.org/W04-1013/)Cited by:[Evaluation Metrics](https://arxiv.org/html/2609.05947#Sx3.SSx4.p2.1)\.
- Linet al\.\(2023\)J\. Lin, J\. Song, Z\. Zhou, Y\. Chen, and X\. ShiMoprd: a multidisciplinary open peer review dataset\.Neural Computing and Applications35\(34\),pp\. 24191–24206\.Cited by:[Related Work](https://arxiv.org/html/2609.05947#Sx2.p2.1)\.
- Liuet al\.\(2023\)Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. ZhuG\-Eval: NLG evaluation using GPT\-4 with better human alignment\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 2511–2522\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.153/)Cited by:[Evaluation Metrics](https://arxiv.org/html/2609.05947#Sx3.SSx4.p4.1)\.
- Locet al\.\(2026\)N\. P\. P\. Loc, L\. Viet, T\. Huynh, T\. T\. Khanh, D\. A\. Nguyen, T\. A\. N\. Pham, T\. Nguyen, N\. V\. Chawla, W\. Buntine, K\. Wong,et al\.PRISM: a multi\-dimensional benchmark for evaluating llm peer reviewers\.arXiv preprint arXiv:2605\.26730\.Cited by:[Related Work](https://arxiv.org/html/2609.05947#Sx2.p2.1)\.
- Neural Information Processing Systems Foundation \(2025\)Neural Information Processing Systems FoundationNeurIPS 2025 reviewer guidelines\.Note:https://neurips\.cc/Conferences/2025/ReviewerGuidelinesAccessed July 25, 2026Cited by:[Introduction](https://arxiv.org/html/2609.05947#Sx1.p2.1)\.
- NeurIPS \(2026\)NeurIPSNeurIPS 2026 AI\-assisted reviewing experiment\.Note:https://dev\.neurips\.cc/Conferences/2026/ai\-reviewing\-experimentAccessed July 25, 2026Cited by:[Introduction](https://arxiv.org/html/2609.05947#Sx1.p1.1)\.
- Sahuet al\.\(2025\)G\. Sahu, H\. Larochelle, L\. Charlin, and C\. PalReviewerToo: should AI join the program committee? a look at the future of peer review\.External Links:2510\.08867,[Link](https://arxiv.org/abs/2510.08867)Cited by:[Related Work](https://arxiv.org/html/2609.05947#Sx2.p1.1)\.
- Salton and Buckley \(1988\)G\. Salton and C\. BuckleyTerm\-weighting approaches in automatic text retrieval\.Information Processing & Management24\(5\),pp\. 513–523\.External Links:[Document](https://dx.doi.org/10.1016/0306-4573%2888%2990021-0)Cited by:[Evaluation Metrics](https://arxiv.org/html/2609.05947#Sx3.SSx4.p3.1)\.
- Sokolova and Lapalme \(2009\)M\. Sokolova and G\. LapalmeA systematic analysis of performance measures for classification tasks\.Information Processing & Management45\(4\),pp\. 427–437\.External Links:[Document](https://dx.doi.org/10.1016/j.ipm.2009.03.002),[Link](https://doi.org/10.1016/j.ipm.2009.03.002)Cited by:[Evaluation Metrics](https://arxiv.org/html/2609.05947#Sx3.SSx4.p4.1)\.
- Wanget al\.\(2023\)G\. Wang, Q\. Peng, Y\. Zhang, and M\. ZhangWhat have we learned from openreview?\.World Wide Web26\(2\),pp\. 683–708\.Cited by:[Introduction](https://arxiv.org/html/2609.05947#Sx1.p4.1)\.
- Wenget al\.\(2025\)Y\. Weng, M\. Zhu, G\. Bao, H\. Zhang, J\. Wang, Y\. Zhang, and L\. YangCycleresearcher: improving automated research via automated review\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 3669–3709\.Cited by:[Introduction](https://arxiv.org/html/2609.05947#Sx1.p3.1)\.
- Yuanet al\.\(2022\)W\. Yuan, P\. Liu, and G\. NeubigCan we automate scientific reviewing?\.Journal of Artificial Intelligence Research75,pp\. 171–212\.Cited by:[Related Work](https://arxiv.org/html/2609.05947#Sx2.p1.1)\.
- Zhanget al\.\(2020\)T\. Zhang, V\. Kishore, F\. Wu, K\. Q\. Weinberger, and Y\. ArtziBERTScore: evaluating text generation with BERT\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=SkeHuCVFDr)Cited by:[Evaluation Metrics](https://arxiv.org/html/2609.05947#Sx3.SSx4.p2.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. StoicaJudging LLM\-as\-a\-judge with MT\-bench and chatbot arena\.InAdvances in Neural Information Processing Systems,Vol\.36\.External Links:[Link](https://arxiv.org/abs/2306.05685)Cited by:[Evaluation Metrics](https://arxiv.org/html/2609.05947#Sx3.SSx4.p4.1)\.
- Zhouet al\.\(2024\)R\. Zhou, L\. Chen, and K\. YuIs llm a reliable reviewer? a comprehensive evaluation of llm on automatic paper reviewing tasks\.InProceedings of the 2024 joint international conference on computational linguistics, language resources and evaluation \(LREC\-COLING 2024\),pp\. 9340–9351\.Cited by:[Introduction](https://arxiv.org/html/2609.05947#Sx1.p3.1)\.
- Zhuet al\.\(2025\)M\. Zhu, Y\. Weng, L\. Yang, and Y\. ZhangDeepreview: improving llm\-based paper review with human\-like deep thinking process\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 29330–29355\.Cited by:[Introduction](https://arxiv.org/html/2609.05947#Sx1.p3.1),[Related Work](https://arxiv.org/html/2609.05947#Sx2.p1.1)\.