Review Arcade: On the Human Alignment and Gameability of LLM Reviews

# On the Human Alignment and Gameability of LLM Reviews
Source: [https://arxiv.org/html/2605.28897](https://arxiv.org/html/2605.28897)
Hans Ole Hatzel1\*,Sebastian Steindl3\*,Jan Strich1,2\* 1Language Technology Group, University of Hamburg, Germany 2Hub of Computing and Data Science \(HCDS\), University of Hamburg, Germany 3OTH Amberg\-Weiden, Germany \*Equal contributions, order decided by coin toss\. Correspondence: \{first\_name\}\.\{last\_name\}@uni\-hamburg\.de,s\.steindl@oth\-aw\.de

###### Abstract

LLM\-generated reviews for scientific papers are gaining considerable traction and are even being officially piloted by major conferences\. We have to assume that not only reviewers are using LLM\-assistance, but also that authors use LLMs to revise their papers before submitting\. In this work, we perform empirical experiments on papers from the 2025 ACL Rolling Review \(ARR\) to evaluate LLM reviews from both the author and the reviewer perspective\. First, we identify a limited alignment of LLM reviews with human ones\. In the best\-case scenario, the alignment is reasonable\. However, we also find that LLM\-human alignment varies substantially across prompts and models\. Finally, we investigate the scenario in which the author uses an iterative draft\-revise workflow to improve the submission according to the LLM review\. We find that this “gaming” of LLM reviews can be effective in specific scenarios, leading to a statistically significant increase of overall scores for up to 35% of papers\. We publish our code\.111[GitHub Repository](https://github.com/uhh-hcds/reviewarcade)

![[Uncaptioned image]](https://arxiv.org/html/2605.28897v1/fig/emoji.png)Review Arcade: On the Human Alignment and Gameability of LLM Reviews

Hans Ole Hatzel1\*, Sebastian Steindl3\*, Jan Strich1,2\*1Language Technology Group, University of Hamburg, Germany2Hub of Computing and Data Science \(HCDS\), University of Hamburg, Germany3OTH Amberg\-Weiden, Germany\*Equal contributions, order decided by coin toss\.Correspondence:\{first\_name\}\.\{last\_name\}@uni\-hamburg\.de,s\.steindl@oth\-aw\.de

## 1Introduction

LLMs are becoming ubiquitous in academic writing\. They are not only powerful tools for correcting grammar and syntax, but can also be used as a source of ad\-hoc feedback to a manuscriptKobaket al\.\([2025](https://arxiv.org/html/2605.28897#bib.bib18)\); Wuet al\.\([2026](https://arxiv.org/html/2605.28897#bib.bib27)\)\. Consequently, authors are more likely to revise their papers using LLMs\. At the same time, LLM reviews are being studied as a possible way to reduce the overload of the peer review system caused by the strong increase in submissions\.Weiet al\.\([2025](https://arxiv.org/html/2605.28897#bib.bib26)\); Choiet al\.\([2026](https://arxiv.org/html/2605.28897#bib.bib7)\)\. Beyond potential future official practice, current research indicates LLM\-usage in the peer\-review process\.Lianget al\.\([2024](https://arxiv.org/html/2605.28897#bib.bib19)\)establishes that across most of their analyzed conferences and journals, 7\-15% of reviews show AI usage beyond simple grammar correction\. Given this, authors may assume that their submission might be LLM\-reviewed and are thus encouraged to optimize their submission accordingly\. Thus, the current situation may culminate in both submission and review becoming heavily LLM\-reliant \(Fig\.[1](https://arxiv.org/html/2605.28897#S1.F1)\)\. In this context, we should consider Goodhart’s lawGoodhart \([1975](https://arxiv.org/html/2605.28897#bib.bib13)\):“When a measure becomes a target, it ceases to be a good measure\.”Strathern \([1997](https://arxiv.org/html/2605.28897#bib.bib24)\)\. Applied here, once authors optimize papers specifically for LLM reviews, they may no longer reliably reflect paper quality, even if they initially did\.

![Refer to caption](https://arxiv.org/html/2605.28897v1/x1.png)Figure 1:Visualization of the peer\-review process if both author and reviewer rely on LLMs\.In this paper, we study the alignment of LLM and human reviews on 984 real ARR submissions for ACL 2025\. We evaluate this across multiple models \(open\-weight and proprietary\), prompts, and runs\. Additionally, we simulate an Iterative Submission Improvement \(ISI\) workflow, where authors optimize their submissions according to LLM reviews\.

We are guided by three research questions \(RQs\):

- •LLM Review Validity\(RQ1\): Can LLMs produce reviews that are sufficiently aligned with human reviews?
- •LLM Review Stability\(RQ2\): Are LLM reviews for a given submission consistent across models, prompts, and repeated runs?
- •LLM Review Gaming\(RQ3\): Can LLM reviews be “gamed” by automated, iterative edits that are informed by LLM reviews and aim to improve review scores?

Our main contributions are: \(i\) The first large\-scale empirical evaluation of LLM reviews for ARR submissions, \(ii\) an investigation of an automated paper\-editing scheme as an adversarial attack on automated reviews, and \(iii\) a taxonomy for such edits grounded in prior literature\.

## 2Background and Related Work

Automated Peer\-Review\.Approaches to automated peer\-review and the analysis of LLM reviews have increasingly gained traction, with researchers benchmarking language models on the task, and proposing systems to improve performance and explore the properties of LLM reviews\. An early example in the LLM era isZhouet al\.\([2024](https://arxiv.org/html/2605.28897#bib.bib29)\), who systematically evaluated LLMs on peer\-review tasks\. Various authors have since suggested improvements using thinking processes or agentic approaches to the taskJinet al\.\([2024](https://arxiv.org/html/2605.28897#bib.bib16)\); Zhuet al\.\([2025](https://arxiv.org/html/2605.28897#bib.bib30)\); Idahl and Ahmadi \([2025](https://arxiv.org/html/2605.28897#bib.bib15)\); Bougie and Watanabe \([2025](https://arxiv.org/html/2605.28897#bib.bib6)\); Sahuet al\.\([2025](https://arxiv.org/html/2605.28897#bib.bib22)\)\.

In terms of real\-world applications,Biswaset al\.\([2026](https://arxiv.org/html/2605.28897#bib.bib4)\)recently evaluated LLM reviewers at scale for the AAAI conference and found them to be perceived favorably by authors and other reviewers alike\. Taking the stance that human reviews should be considered the gold standard, one of the main metrics for the usability of LLM reviews becomes their alignment with the human reviews\. One reason why the survey inBiswaset al\.\([2026](https://arxiv.org/html/2605.28897#bib.bib4)\)might have shown LLM reviews to be favorable is the high variance in human review quality\.

Reliability of Human Reviews\.There is a limited range of prior work considering the reliability of human reviews\. Notably, acceptance decisions are generally not determined by a simple score threshold; instead, meta reviewers and program chairs consider many factors, such as outliers in review scores and their justification, or simply the number of competing papers in a given trackCicchetti \([1991](https://arxiv.org/html/2605.28897#bib.bib8)\)\. The NeurIPS conference ran an acceptance experiment that simulated this entire decision processBeygelzimeret al\.\([2021](https://arxiv.org/html/2605.28897#bib.bib3),[2023](https://arxiv.org/html/2605.28897#bib.bib2)\), finding that approximately half the papers accepted by one committee were rejected by the other\. Conversely, they find that a given paper had a roughly 15% chance of being accepted after being rejected by the first committee\. In terms of review scores, the deviation is much easier to quantify, given that there are typically multiple independent reviews of the same paper\.Baumannet al\.\([2026](https://arxiv.org/html/2605.28897#bib.bib1)\)report a Pearson correlation of 0\.14 across human reviewers, whileCortes and Lawrence \([2021](https://arxiv.org/html/2605.28897#bib.bib34)\)find a Pearson correlation of 0\.55 in their data after calibrating for cross\-reviewer scale interpretation using a Gaussian model\.

Peer\-Review Datasets\.PeerReadKanget al\.\([2018](https://arxiv.org/html/2605.28897#bib.bib17)\)was one of the first peer\-review datasets\. They collect likely rejects from arXiv while relying on reviews of accepted papers from reviewing platforms, including OpenReview\. Many datasets primarily recruit their reviews from accepted papers, thereby introducing biases\. In a more recent example, NLPeerDyckeet al\.\([2023](https://arxiv.org/html/2605.28897#bib.bib11)\)made use of a clear data collection scheme requiring opt\-ins from reviewers and authors alikeDyckeet al\.\([2022](https://arxiv.org/html/2605.28897#bib.bib12)\)\.

Metrics for Automated Reviews\.There is a multitude of metrics being used to measure the quality of automated reviews\. Prior work uses, e\.g\., accuracy and correlational measuresZhouet al\.\([2024](https://arxiv.org/html/2605.28897#bib.bib29)\); Idahl and Ahmadi \([2025](https://arxiv.org/html/2605.28897#bib.bib15)\), AUC, FPR and FNRLuet al\.\([2026](https://arxiv.org/html/2605.28897#bib.bib20)\), and MAEZhuet al\.\([2025](https://arxiv.org/html/2605.28897#bib.bib30)\)\.

We report MAE and Pearson correlation, as well as an LLM\-judge measuring semantic overlap, as the primary metrics for measuring the LLM\-human alignment in this paper\. Further, we distinguish between best\-match and overall correlations: for best match we only calculate correlations with the best matching review, in terms of theOverallscore\.

Concurrent work\.Kimet al\.\([2026](https://arxiv.org/html/2605.28897#bib.bib37)\)conduct a human evaluation of review quality, where experts assess human and LLM\-generated reviews along three dimensions\. They find that LLM\-generated reviews can surpass human reviews in perceived quality, while still exhibiting systematic limitations\. In a related position paper,Baumannet al\.\([2026](https://arxiv.org/html/2605.28897#bib.bib1)\)show thatpaper laundering, iteratively prompting LLMs to improve a manuscript based on LLM\-generated reviews, can substantially increase review scores\. Although framed as inducing only superficial, cosmetic edits, their prompting strategy does not enforce such constraints and may instead encourage substantial revisions\. Motivated by this, we conduct a more principled evaluation of paper laundering in an iterative setting and further quantify LLM\-induced semantic changes using an taxonomy\.

## 3Method

Today, real\-world reviewers often employ off\-the\-shelf models to aid in their reviewing\(Lianget al\.,[2024](https://arxiv.org/html/2605.28897#bib.bib19)\)and official usage aims for zero data retention by using open\-weight models offline or API settings\. Our setup aims to align itself with this real\-world usage of LLMs in the context of peer review\. As such, we evaluate with both open\-weight and closed\-weight models\. However, we do not employ sophisticated agentic workflows, which might increase the quality of individual reviews\.

### 3\.1Problem Statement

Our work mainly focuses on using an LLMℳ\\mathcal\{M\}, prompted with instructionsρ\\rho, to generate a reviewrrfor the submissionss:

r=f​\(M,ρ,s\)\.r=f\(M,\\rho,s\)\.\(1\)Then, we evaluate the quality ofrrby calculating its alignment with the ground\-truth, human\-written reviewr^\\hat\{r\}using the evaluation functionh​\(r^,r\)h\(\\hat\{r\},r\)\. Concretely,h​\(r^,r\)h\(\\hat\{r\},r\)can be instantiated as a measurement of correlation on the predicted scores, or as an LLM\-judge𝒥\\mathcal\{J\}that measures content similarity across strengths and weaknesses ofssidentified inrrandr^\\hat\{r\}\.

Moreover, we consider the scenario in which the author optimizes their submissionssby iteratively adapting it based on an LLM review:

si\+1=μ​\(si,f​\(M′,ρ′,si\)\)\.s^\{i\+1\}=\\mu\(s^\{i\},f\(M^\{\\prime\},\\rho^\{\\prime\},s^\{i\}\)\)\.\(2\)
We test the fully\-automated scenario in whichμ\\muis also a call to an LLM, prompted to update the submission to address the review\.

### 3\.2Automated Review Framework

In this work, we want to evaluate if LLM reviews are closely aligned with human reviews \(RQ1\) and if the LLM reviews are consistent across different models and prompts \(RQ2\)\. To this end, we craft five review prompts that are increasingly tailored to the specific ARR review dataset:

- •simple:A minimal prompt asking simply to review and specifying output format\.
- •default:Drafted by the authors to specify target venue and acceptance rate\.
- •ai\_generated:An LLM\-generated prompt for reviewing submissions to a top\-tier Machine Learning conference\.
- •acl:Adapted fromai\_generatedto include the specific guidelines from the ARR\.
- •acl\_senior:Asacl, but with the persona of a senior, expert reviewer\.

For a full list of all prompts, see Appendix[F](https://arxiv.org/html/2605.28897#A6)\.

### 3\.3Iterative Submission Improvement

![Refer to caption](https://arxiv.org/html/2605.28897v1/x2.png)Figure 2:The ISI pipeline is iteratively applied to improve upon paper drafts\.For RQ3, we consider different styles of Iterative Submission Improvement \(ISI\)\. Optimizing a submission solely to target automated reviews is what we describe as “gaming” LLM reviews\. ISI describes the iterative loop, depicted in Fig\.[2](https://arxiv.org/html/2605.28897#S3.F2), in which an author generates a reviewrrfor their submissionsis^\{i\}with an LLM and uses this to inform an editing functionμ\\muto improve their submission, creatingsi\+1s^\{i\+1\}\. We iteratively apply ISI for ten iterations\. Since it is impossible to perfectly predict an accept/reject decision, we do not try to predict if a paper would be accepted or rejected, and instead focus on improvements of theOverallscore\. Specifically, we focus on three settings:constrained,default,adversarial222All prompts given in Appendix[F](https://arxiv.org/html/2605.28897#A6)\.\.

In theconstrainedsetting, the author prohibits substantive changes and allows only superficial, cosmetic edits in response to the review\. This tests whether the “paper laundering” ofBaumannet al\.\([2026](https://arxiv.org/html/2605.28897#bib.bib1)\)can shift LLM review recommendations from reject to accept\. However, their prompt does not strictly enforce cosmetic\-only edits and may even encourage more fundamental changes\.

Therefore, in ourdefaultsetting, we use a prompt that is heavily inspired by the editing prompt used inBaumannet al\.\([2026](https://arxiv.org/html/2605.28897#bib.bib1)\), but removes instructions that could lead to non\-cosmetic changes\. We call thisdefaultas it neither prohibits nor actively allows profound changes\. Lastly, in theadversarialsetting, we simulate an author who actively encourages editing to get the paper accepted at any cost, even if that means, e\.g\., fabricating results\.

### 3\.4Taxonomy of Edits

To better understand what type of edits are performed to increase the scores in the LLM review, we introduce a taxonomy of paper edits\. We ground our taxonomy in the work ofYanget al\.\([2017](https://arxiv.org/html/2605.28897#bib.bib36)\), who propose a taxonomy for edit types on Wikipedia\. We adapt their taxonomy to fit our scenario of paper edits for an ARR submission\. The taxonomy is presented in Tab\.[3](https://arxiv.org/html/2605.28897#A1.T3)in the Appendix\. For theconstrainedanddefaultedit settings, we use the same set of allowed edit\-types\. These focus on keeping the content of the submission intact and not requiring new experiments, such as simplifying or clarifying\. For theadversarialsetting, we add another set of edit types that focus on “gaming” the LLM review, such as hallucinating evidence and fabricating better results\.

## 4Experimental Setup

### 4\.1Dataset and Preprocessing

In ARR, the main ACL reviewing platform, reviewers assign 9\-point ratings \(1 to 5 in 0\.5 steps\) across four categories:Soundness,Excitement,Reproducibility, andOverall\. Reviews and author responses are discussed before the Area Chair writes a meta\-review summarizing them\. Final acceptance decisions are made by the program committee based on reviews and meta\-reviews\. We only use theOverallscore as it is the most representative metric\.

Existing research in the space of ARR reviews relies on very few or no rejected papers\. This potentially introduces a positivity bias in systems developed for this data\. We perform stratified subsampling on the NLPeer datasetDyckeet al\.\([2023](https://arxiv.org/html/2605.28897#bib.bib11)\)in a fashion similar toSahuet al\.\([2025](https://arxiv.org/html/2605.28897#bib.bib22)\)to define a dataset with 984 papers\. We retain all rejected papers with reviews from NLPeer\. All accepted papers in our dataset were accepted to ACL 2025\. Rejected papers make up roughly one third of our dataset333While this does not correspond to the acceptance rate at ARR venues, it is suited for our experiments\.\. To prepare our documents for LLM processing, we process them using the OCR modelolmOCR\-2\-7B\-1025in conjunction with the OlmOCR pipelinePoznanskiet al\.\([2025](https://arxiv.org/html/2605.28897#bib.bib35)\)\. This outputs Markdown versions of the papers\. Tables are retained and presented as Markdown to the models while figures are only represented by captions provided in the original paper\. This setup enables us to isolate the models’ reviewing capabilities from their PDF reading abilities and simulates the application of LLMs in larger systems where typically a content extraction step is performed for PDFsBlecheret al\.\([2024](https://arxiv.org/html/2605.28897#bib.bib5)\)\. We filter out papers longer than\>130,000\>130,000subword tokens to account for context window limitations, long appendices, and potential extraction errors, and also exclude papers with missing review text or incorrectly extracted paper text\.

Dataset Statistics\.In our subsampled dataset, humans show a rather low overall correlation of 0\.312 for theOverallscore across reviews of the same paper\. Similar magnitudes are reported byBaumannet al\.\([2026](https://arxiv.org/html/2605.28897#bib.bib1)\), who find correlations of 0\.137 in a subsample and 0\.180 across all ICLR reviews\. We also find that the correlation is substantially higher for rejected papers \(0\.408\) than for accepted papers \(0\.210\), suggesting that reviewers are more likely to agree if a submission is poor than good\. This is consistent with prior literatureCortes and Lawrence \([2021](https://arxiv.org/html/2605.28897#bib.bib34)\)\. The underrepresentation of rejected papers in ARR\-related studies, arising from the paper collection process, is therefore particularly concerning\.

In Figure[3](https://arxiv.org/html/2605.28897#S4.F3), we illustrate that papers in the rejected split of our dataset are, on average, much shorter\. They show an almost uniform distribution from 4,000 to 9,000 tokens, while accepted papers show a clear increase around the 7,500 token mark\. We hypothesize two causes: \(1\) accepted papers are often more comprehensive and near the page limit, leading to more concentrated contributions; and \(2\) shorter papers are less likely to be accepted, resulting in their overrepresentation in our dataset\.

On average, papers in the accepted split have 2\.0 reviews, with a standard deviation of \.7, while the rejected split has just over 1\.1 reviews per paper, with a standard deviation of 0\.3\. This imbalance is likely a result of the additional approval process for reviews of rejected papers\. Overall, we observe a clear difference in the accepted and rejected groups\. For this reason, our further analysis will make an effort to explicitly obtain results for each subset\.

### 4\.2Models

Authors and reviewers might use a variety of LLMs\. Therefore, we select six models, covering model sizes as well as three open\- and two closed\-weight models\. Specifically, we use Qwen\-3\.6\-35BYanget al\.\([2025](https://arxiv.org/html/2605.28897#bib.bib28)\), Gemma\-3\-27BTeamet al\.\([2025](https://arxiv.org/html/2605.28897#bib.bib25)\), Llama\-3\.3\-70BGrattafioriet al\.\([2024](https://arxiv.org/html/2605.28897#bib.bib14)\), GPT\-5\.4\-mini, and GPT\-5\.4\.444All models are used in their instruction\-tuned variants\.

ModelPromptCombinedAccepted SplitRejected SplitMAE↓\\downarrowBest Matchrr↑\\uparrowMAE↓\\downarrowBest Matchrr↑\\uparrowMAE↓\\downarrowBest Matchrr↑\\uparrowGemma\-3\-27BAll0\.97±\\pm0\.320\.146±\\pm0\.060\.83±\\pm0\.230\.246±\\pm0\.101\.12±\\pm0\.460\.041±\\pm0\.04Best0\.89±\\pm0\.010\.205±\\pm0\.020\.73±\\pm0\.000\.367±\\pm0\.021\.05±\\pm0\.020\.031±\\pm0\.01Qwen\-3\.6\-35BAll0\.73±\\pm0\.120\.189±\\pm0\.030\.76±\\pm0\.220\.208±\\pm0\.050\.70±\\pm0\.170\.169±\\pm0\.02Best0\.81±\\pm0\.010\.217±\\pm0\.040\.63±\\pm0\.010\.251±\\pm0\.001\.00±\\pm0\.020\.183±\\pm0\.07Llama\-3\.3\-70BAll1\.20±\\pm0\.150\.103±\\pm0\.080\.88±\\pm0\.100\.090±\\pm0\.131\.52±\\pm0\.200\.116±\\pm0\.06Best0\.95±\\pm0\.010\.234±\\pm0\.020\.73±\\pm0\.010\.308±\\pm0\.011\.16±\\pm0\.010\.157±\\pm0\.03GPT\-5\.4\-miniAll0\.75±\\pm0\.110\.124±\\pm0\.060\.89±\\pm0\.250\.090±\\pm0\.120\.62±\\pm0\.110\.157±\\pm0\.05Best0\.700\.2290\.580\.2780\.810\.178GPT\-5\.4All0\.73±\\pm0\.130\.180±\\pm0\.070\.82±\\pm0\.270\.167±\\pm0\.110\.63±\\pm0\.090\.194±\\pm0\.06Best0\.710\.2760\.630\.3170\.800\.233Human0\.170\.3120\.300\.2100\.040\.408Baseline \(y^:=2\.5\\hat\{y\}:=2\.5\)0\.64—0\.75—0\.53—

Table 1:Results across models and prompt setups on the Overall dimension\. MAE and Best Match Pearson\-rrover runs \(mean±\\pmstd\)\.Bold: best in column;underlined: second best\. Performance on the combined split is given as macro average across the two splits\.
### 4\.3Experimental Design

We design three main experiments to answer our RQs\. First, we generate one review for each prompt and model, and repeat this twice, for a total of three reviews\. This allows us to measure the alignment of human and LLM reviews with regard to their scores and content, and their stability \(RQ1, RQ2\)\. Focusing on theOverall score, we measure the mean absolute error \(MAE\) against the mean of all human reviews\. We use Pearson’srrto measure the correlation to the best match, i\.e\., to the human review with the lowest distance\. We report these metrics for both the best performing prompt \(in terms of Pearson\-rron the combined split\) and the average performance across all prompts\.

To assess semantic alignment between LLM and human reviews, we use an LLM judge to identify which human\-stated strengths and weaknesses are reflected in the LLM review\. This recall\-style metric provides information beyond review scores\. We provide the human\-performance by comparing against all other humans as well as a naive baseline that constantly predicts the mid\-point from the rating scale555For the latter no correlation calculations are possible\.\. Note that for the rejected split, due to a lack of examples with multiple human reviews, the MAE andrrof the humans are being calculated with only 26 papers\. For the Combined split, we macro\-average across accepted and rejected performance \(in Fisher\-zzspace for the correlation\)\.

In the second experiment, we investigate if the papers can be iteratively adapted based on the LLM review to increase their scores\. We test this with a maximum of 10 iterations and with three different editing prompts, representing different levels of changes, from superficial edits \(constrained\) to substantial changes including fabricated evidence \(adversarial\)\. We also include a prompt that is heavily based on the one used byBaumannet al\.\([2026](https://arxiv.org/html/2605.28897#bib.bib1)\), and to which we refer asdefault\. We measure this effect in terms of the percentage of papers with an increased score afternniterations\. As a baseline, we repeat the prediction for the initial, unedited submission also ten times\.

01,0001\{,\}0002,0002\{,\}0003,0003\{,\}0004,0004\{,\}0005,0005\{,\}0006,0006\{,\}0007,0007\{,\}0008,0008\{,\}0009,0009\{,\}00002020404060608080Paper length \(whitespace\-separated tokens\)Absolute frequencyAcceptedRejectedFigure 3:Length distribution of the papers considered in this study\. Grouped in 30 buckets \(each ~320 tokens\)\.

## 5Results and Discussion

### 5\.1LLM Review Validity \(RQ1\)

Alignment to Human Review Scores\.First, we test the validity of LLM reviews as measured by their alignment with the human ratings\. For theCombinedsplit in Table[1](https://arxiv.org/html/2605.28897#S4.T1)including both the accepted and rejected papers, we can observe that the LLMs fail to match human judgments in terms of MAE\. GPT\-5\.4\-mini and GPT\-5\.4 are the best performing LLMs with an MAE of around 0\.7, compared to the human 0\.17\. Notably, the naive constant prediction baseline slightly outperforms the best LLM with an MAE of 0\.64\. In terms of correlation, the models come much closer to human performance with GPT\-5\.4 reaching a correlation of 0\.276\. However, one must consider that the human\-human correlation of 0\.312 indicates low agreement even between humans, which aligns with prior workBaumannet al\.\([2026](https://arxiv.org/html/2605.28897#bib.bib1)\)\.

The results in Table[1](https://arxiv.org/html/2605.28897#S4.T1)also show a pronounced performance difference between the accepted and rejected split for all models, most prominently for Gemma\-3\. The human agreement is much higher for the accepted papers, with the best\-matchrrbeing nearly twice as high \(0\.41 vs\. 0\.21\)\. We hypothesize that this performance difference is explained by the fact that accepted papers meet a high bar in terms of minimum quality and that it is hard to differentiate across them\. This aligns with the finding byCortes and Lawrence \([2021](https://arxiv.org/html/2605.28897#bib.bib34)\), that the 2014 NeurIPS review process was good at identifying poor papers, but bad at identifying good papers\. Overall, while Pearsonrris, depending on the split, competitive with human evaluation, we observe that at least in terms of MAE the models are not competitive with human reviews\.

For realistic, practical applications, the macro\-averageCombinedperformance is more indicative, since it is unknown at submission time which split the paper would be part of\. Here we see that individual prompts perform very well but note that Qwen delivers the most robust performance across splits, tying with GPT\-5\.4 in terms of prompt\-averaged MAE but slightly outperforming it in terms of prompt\-averaged best\-match Pearson\-rr\.

Content\-wise Alignment with Human Reviews\.Besides the scores, we also evaluate how similar the LLM reviews are to the human reviews in their content\. We report the strengths\-recalls\_recalland weaknesses\-recallw\_recall, which represent the fraction of strengths and weaknesses that appear in both the human and LLM reviews as presented in Fig\.[4](https://arxiv.org/html/2605.28897#S5.F4)\. For the strengths, Gemma\-3 achieves the best overalls\_recall, with roughly 0\.59 on the accepted, and 0\.48 on the rejected split\. For the weaknesses, GPT\-5\.4\-mini has the highest recall, with roughly 0\.41 and 0\.44 on the respective splits\. We observe that, in general, the recall is higher for strengths than for weaknesses\. Especially for the strengths, our results also indicate that the recall can differ between the accepted and rejected split\.

![Refer to caption](https://arxiv.org/html/2605.28897v1/x3.png)Figure 4:Mean Recall of Strengths and Weaknesses for each of the best runs for each model\.Are LLM Reviews Valid?Overall, the results indicate that, in a select best\-case scenario, LLM\-review scores show good alignment with human judgments, at least in terms of correlation\. In this setting, model\-to\-model agreement is comparable to human\-to\-human alignment\. However, this behavior does not consistently transfer to real\-world conditions where the acceptance decisions are not known a priori\. Across splits, no single setup is consistently superior\. Because it is hard to calibrate LLMs to align with human reviews, we reach a mixed conclusion regarding RQ1: LLMs can be reviewers in some scenarios, but not universally\.

### 5\.2LLM Review Stability \(RQ2\)

Stability Across Prompts & ModelsImportantly, we observe a considerable variance across reviewing prompts\. For example, GPT\-5\.4\-mini on the accepted split, which had the best MAE in its best setting, has the worst MAE when averaging across prompts \(0\.89\)\. This trend holds across all models we tested\. Crucially, as Figure[5](https://arxiv.org/html/2605.28897#A2.F5)shows, there is no clear trend as to which prompt leads to best performance, neither across models, nor within the same model across the accepted and rejected split\. The models appear sensitive to prompt variations, alternating between overly permissive and overly restrictive behavior\. This might explain the interesting observation that the overly simple one\-liner promptsimpleachieved remarkably good performance, suggesting that sophisticated prompting may not yield improvements on our tasks\.

Stability Across Repeated RunsIf we perform multiple runs using the same paper and prompt at temperature 1\.0, we can observe very low standard deviations of around 0\.02 across both MAE and Pearson\-rr, whereas the deviation is much larger \(up to around 0\.25 MAE\) for runs across prompts of the same model\. In our experiments with three model invocations using the same model \(see Tab\.[4](https://arxiv.org/html/2605.28897#A5.T4)\), prompt, and submission, we see that for 36\.9% of papers, at least one out of three runs gives a different score than the others and for 20% this delta is\>0\.5\>0\.5\. Therefore, we argue that LLM\-reviews are generally too instable across repeated runs to be reliable\.

Are LLM Reviews Stable?Given the considerable instability of LLM reviews across prompts and models, and even repeated runs, RQ2 can be answered in the negative\. This is clearly illustrated in Fig\.[5](https://arxiv.org/html/2605.28897#A2.F5), where different prompts lead to substantially different results across models\.

### 5\.3Gaming LLM Reviews \(RQ3\)

Based on the results for RQ1 and RQ2, we see that for our experiment on gaming LLM reviews, Qwen\-3\.6 and GPT\-5\.4 are best suited\. Due to cost considerations and its consistent performance across prompts, we chose Qwen\-3\.6 for our subsequent experiments\. As we expect edits to only drastically improve a small to medium portion of paper scores, we perform rigorous statistical significance tests\. The details are given in Appendix[D](https://arxiv.org/html/2605.28897#A4)\. We reportppvalues and Cohen’sddto account for the large sample size in the dataset and to complement significance testing with an effect\-size measure in Table[2](https://arxiv.org/html/2605.28897#S5.T2)\. Effect sizes are interpreted following established rules of thumb in the literatureCohen \([1992](https://arxiv.org/html/2605.28897#bib.bib39)\)\.

Constrained RewritingIn this setup, the prompt explicitly forbids the LLM from making any profound changes to the context\. It only allows superficial edits to address the initial review\. We find this leads to a statistically significant increase in paper scores in the LLM reviews after 10 review\-and\-edit loops, compared to the LLM reviews before any changes\. We find that roughly 36% of the papers improve, 42% remain at their initial score, and 22% of scores decrease\. The effect size for this setup, in terms of Cohen’sdd, is considered small to mediumCohen \([1992](https://arxiv.org/html/2605.28897#bib.bib39)\)\.

Default RewritingThedefaultrewriting shows similar numbers for the score changes as theconstrainedediting\. However, the results are not statistically significant and show very small effect sizes\.

Adversarial RewritingLastly, we tested theadversarialrewriting, where the LLM is explicitly allowed to make changes it deems helpful for acceptance, including fabricating evidence and factual misrepresentations\. In this setup, our data also shows improvements across edit iterations\. However, the effect sizes are weaker than in theconstrainedsetting\. This is surprising, since we expected that, e\.g\., fabricating results should lead to a large increase in review scores\. In fact, we find that when it comes to edit types \(as per the taxonomy introduced in Section[3\.4](https://arxiv.org/html/2605.28897#S3.SS4)\), the adversarial prompt almost exclusively turns to theMethodological\-Augmentationedit type\. The default and constrained setups, on the other hand, largely rely on the clarification edit type, with the constrained setup also making frequent use of theRefactoringedit type\. See Figure[6](https://arxiv.org/html/2605.28897#A3.F6)in Appendix[C](https://arxiv.org/html/2605.28897#A3)for a full breakdown of the edit types across prompts\.

We hypothesize that we did not observe a substantial increase in scores in the adversarial setup for two main reasons\. First, methodological edits might introduce inconsistencies within the submission, which could be penalized by the following LLM review\. Second, the LLM’s guardrails might lead it to rarely confabulate substantial evidence, which is supported by the fact thatMethodological Augmentationis the most prevalent edit type in the adversarial setup, even if more aggressive edit types \(such as Factual\-Optimization or Hallucinated\-Evidence\) were available in the taxonomy\.

Are LLM Reviews Gamable?Yes, in specific scenarios, our ISI pipeline can iteratively improve the scores of papers when it comes to LLM reviews\. In the constrained setup, 35% of papers improved after 10 rounds of edits, but this improvement also carried a risk of score regressions, with 22% of papers seeing a decrease in their score\. Whether this improvement in scores is associated with a substantive improvement in the paper or truly a case of gaming the LLM\-reviewer is harder to answer\.ClarificationsandCopy\-Editingmay not produce substantial improvements to the core of a paper, indicating that gaming is taking place; on the other hand,Refactoringis an edit choice frequently made by this best\-performing approach, an edit that can result in substantial restructuring of a paper, albeit with limited content changes\. Ultimately, whether we consider this gaming of LLM reviewers depends on our trust in human reviewers to look beyond surface\-level improvements in the papers\.

SettingOutcomes \(%\)ppddWorseEqualBetterBaseline28\.1544\.7227\.13\.795\-0\.03Reject29\.2745\.1225\.61\.882\-0\.07Accept27\.5944\.5127\.90\.567\-0\.01Default25\.3044\.1130\.59\.0120\.07Reject25\.9146\.0428\.05\.3790\.02Accept25\.0043\.1431\.86\.0060\.10Adversarial28\.4835\.9135\.61\.0040\.10Reject22\.9237\.5039\.58< \.0010\.24Accept31\.6735\.0033\.33\.2540\.03Constrained22\.3641\.6735\.98< \.0010\.20Reject18\.9038\.7242\.38< \.0010\.32Accept24\.0943\.1432\.77< \.0010\.13Table 2:Distribution of model responses across prompt settings \(Baseline, Default, Adversarial, and Constrained\), reported as percentages of Decrease, Equal, and Increase outcomes after 10 iterations\. Paired t\-tests with t/p\-values and effect sizes \(Cohen’s d\) fort0t\_\{0\}andt10t\_\{10\}\.Bold: statistically significant results withp<\.001p<\.001\.

## 6Conclusion

Our results show that human\-human correlation in review scores still surpasses the LLM\-human alignment\. Naively prompted LLMs are instable in their reviews and not yet generally reliable as peer\-reviewers\. We show that in specific scenarios, current models are able to self\-improve papers using superficial edits to improve LLM\-judge scores\. In this setup, it is feasible to use automated rewriting to push papers past the acceptance threshold in LLM\-reliant peer\-review\. UnlikeBaumannet al\.\([2026](https://arxiv.org/html/2605.28897#bib.bib1)\), we do not see this effect in prompts that have little guidance\.

Interestingly, when allowed to fabricate evidence, our ISI pipeline did not significantly improve papers across the entire dataset\. We argue that this can be explained due to model guardrails avoiding fabrication of evidence or these edits introducing inconsistencies within the revised submission\. While peer review processes are, in reality, more complex than a simple score cutoff, our findings highlights a potential vulnerability in the peer\-review process as LLM usage increases\. We cannot yet confirm if these iterative improvements would translate to humans accepting the papers despite not having profound improvements\.

We urge the community to employ extreme caution when approaching the subject of automated reviews\. Given Goodhart’s law, even when LLM reviews currently show decent alignment with human reviews, they might cease to be a good measure of submission quality\.

We call on future work to extend the evaluation of automated peer review with all its strengths and weaknesses\. We believe that LLM\-assistance during peer\-review can be beneficial in reducing the reviewing load, but official implementation needs to be carefully designed to avoid gameability and ensuring no lack of diverse perspectives on the submissions\. Future evaluations should move beyond scores as a surrogate for holistic reviewer assessment, as this is a reductive representation of review content\. Scores may be right for the wrong reasons, and similarly, reviews with diverging scores may still share the same opinion on a paper, but can, for example, have different quality expectations\.

## Limitations

Our exploratory study provides a range of novel insights, but several aspects could be explored in greater depth in future work\.

#### Quantifying Review Quality

We focus our work primarily on review scores, with a limited exploration of strengths and weaknesses\. Scores have the advantage of being easily quantifiable, but they also fail to account for many nuances in the utility of reviews\. A meta reviewer can, for example, decide to reject a paper despite high scores, just based on some of the described weaknesses\.

#### Counterfactual Reviews after Edits

The best experiment to measure the effect of trying to game LLM reviews, is to review the edited submissions not only automatically, but also with humans\. This would allow to better understand if the edits are indeed improvements, or are simply superficial\. It is, however, virtually impossible to run such counterfactual reviews after the edits have been applied\.

#### Testing Cross\-Model Performance

A real\-world application of our pipeline would mean that details of the prompt and model employed by the reviewer are not known\. We did not test the generalization of rephrasing attacks to other models or to human reviewers\.

#### Data Quality

Our dataset is limited in the number of reviews for rejected papers, leading to less reliable numbers, especially for the human\-human correlation on the rejected split\. In general, human agreement is limited, and due to limitations in our dataset, we cannot apply a reviewer calibration as performed byCortes and Lawrence \([2021](https://arxiv.org/html/2605.28897#bib.bib34)\)\. Lastly, the peer review process, as performed by humans, is also very noisy, often producing different results in new iterations, and is thus hard to compare against\.

#### Data Poisoning

It is possible that the LLMs we use have seen \(part of\) the data we test on during their training process\. It remains unclear if good results will generalize\.

## References

- Stop Automating Peer Review Without Rigorous Evaluation\.arXiv\.External Links:2605\.03202,[Document](https://dx.doi.org/10.48550/arXiv.2605.03202)Cited by:[§2](https://arxiv.org/html/2605.28897#S2.p3.1),[§2](https://arxiv.org/html/2605.28897#S2.p7.1),[§3\.3](https://arxiv.org/html/2605.28897#S3.SS3.p2.1),[§3\.3](https://arxiv.org/html/2605.28897#S3.SS3.p3.1),[§4\.1](https://arxiv.org/html/2605.28897#S4.SS1.p3.1),[§4\.3](https://arxiv.org/html/2605.28897#S4.SS3.p3.1),[§5\.1](https://arxiv.org/html/2605.28897#S5.SS1.p1.1),[§6](https://arxiv.org/html/2605.28897#S6.p1.1)\.
- A\. Beygelzimer, Y\. Dauphin, P\. Liang, and J\. W\. Vaughan \(2021\)The NeurIPS 2021 consistency experiment\.Neural Information Processing Systems blog post, https://blog\. neurips\. cc/2021/12/08/the\-neurips\-2021\-consistency\-experiment\.Cited by:[§2](https://arxiv.org/html/2605.28897#S2.p3.1)\.
- A\. Beygelzimer, Y\. N\. Dauphin, P\. Liang, and J\. W\. Vaughan \(2023\)Has the Machine Learning Review Process Become More Arbitrary as the Field Has Grown? The NeurIPS 2021 Consistency Experiment\.Note:https://arxiv\.org/abs/2306\.03262v1Cited by:[§2](https://arxiv.org/html/2605.28897#S2.p3.1)\.
- J\. Biswas, S\. Schoepp, G\. Vasan, A\. Opipari, A\. Zhang, Z\. Hu, S\. Joseph, M\. Lease, J\. J\. Li, P\. Stone, K\. L\. Wagstaff, M\. E\. Taylor, and O\. C\. Jenkins \(2026\)AI\-Assisted Peer Review at Scale: The AAAI\-26 AI Review Pilot\.arXiv\.External Links:2604\.13940,[Document](https://dx.doi.org/10.48550/arXiv.2604.13940)Cited by:[§2](https://arxiv.org/html/2605.28897#S2.p2.1)\.
- L\. Blecher, G\. Cucurull Preixens, T\. Scialom, and R\. Stojnic \(2024\)Nougat: Neural optical understanding for academic documents\.InInternational Conference on Learning Representations,B\. Kim, Y\. Yue, S\. Chaudhuri, K\. Fragkiadaki, M\. Khan, and Y\. Sun \(Eds\.\),Vol\.2024,pp\. 37646–37663\.Cited by:[§4\.1](https://arxiv.org/html/2605.28897#S4.SS1.p2.1)\.
- N\. Bougie and N\. Watanabe \(2025\)Generative Reviewer Agents: Scalable Simulacra of Peer Review\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track,S\. Potdar, L\. Rojas\-Barahona, and S\. Montella \(Eds\.\),Suzhou \(China\),pp\. 98–116\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-industry.8),ISBN 979\-8\-89176\-333\-3Cited by:[§2](https://arxiv.org/html/2605.28897#S2.p1.1)\.
- J\. Choi, J\. Yun, C\. Kim, and Y\. Kim \(2026\)Position Paper: How Should We Responsibly Adopt LLMs in the Peer Review Process?\.InFindings of the Association for Computational Linguistics: EACL 2026,V\. Demberg, K\. Inui, and L\. Marquez \(Eds\.\),Rabat, Morocco,pp\. 151–165\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.findings-eacl.9),ISBN 979\-8\-89176\-386\-9Cited by:[§1](https://arxiv.org/html/2605.28897#S1.p1.1)\.
- D\. V\. Cicchetti \(1991\)The reliability of peer review for manuscript and grant submissions: A cross\-disciplinary investigation\.Behavioral and Brain Sciences14\(1\),pp\. 119–135\.External Links:ISSN 1469\-1825, 0140\-525X,[Document](https://dx.doi.org/10.1017/S0140525X00065675)Cited by:[§2](https://arxiv.org/html/2605.28897#S2.p3.1)\.
- J\. Cohen \(1992\)A power primer\.Psychological Bulletin112\(1\),pp\. 155–159\.External Links:[Document](https://dx.doi.org/10.1037/0033-2909.112.1.155)Cited by:[§5\.3](https://arxiv.org/html/2605.28897#S5.SS3.p1.2),[§5\.3](https://arxiv.org/html/2605.28897#S5.SS3.p2.1)\.
- C\. Cortes and N\. D\. Lawrence \(2021\)Inconsistency in conference peer review: revisiting the 2014 neurips experiment\.arXiv preprint arXiv:2109\.09774\.Cited by:[§2](https://arxiv.org/html/2605.28897#S2.p3.1),[§4\.1](https://arxiv.org/html/2605.28897#S4.SS1.p3.1),[§5\.1](https://arxiv.org/html/2605.28897#S5.SS1.p2.2),[Data Quality](https://arxiv.org/html/2605.28897#Sx1.SS0.SSS0.Px4.p1.1)\.
- N\. Dycke, I\. Kuznetsov, and I\. Gurevych \(2022\)Yes\-Yes\-Yes: Proactive Data Collection for ACL Rolling Review and Beyond\.InFindings of the Association for Computational Linguistics: EMNLP 2022,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 300–318\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.findings-emnlp.23)Cited by:[§2](https://arxiv.org/html/2605.28897#S2.p4.1)\.
- N\. Dycke, I\. Kuznetsov, and I\. Gurevych \(2023\)NLPeer: A Unified Resource for the Computational Study of Peer Review\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 5049–5073\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.277)Cited by:[§2](https://arxiv.org/html/2605.28897#S2.p4.1),[§4\.1](https://arxiv.org/html/2605.28897#S4.SS1.p2.1)\.
- C\. Goodhart \(1975\)Problems of monetary management : the U\.K\. experience\.Papers in monetary economics 1975 ; 11,pp\. 1\.Cited by:[§1](https://arxiv.org/html/2605.28897#S1.p1.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan, A\. Yang, A\. Fan, A\. Goyal, A\. Hartshorn, A\. Yang, A\. Mitra, A\. Sravankumar, A\. Korenev, A\. Hinsvark, A\. Rao, A\. Zhang, A\. Rodriguez, A\. Gregerson, A\. Spataru, B\. Roziere, B\. Biron, B\. Tang, B\. Chern, C\. Caucheteux, C\. Nayak, C\. Bi, C\. Marra, C\. McConnell, C\. Keller, C\. Touret, C\. Wu, C\. Wong, C\. C\. Ferrer, C\. Nikolaidis, D\. Allonsius, D\. Song, D\. Pintz, D\. Livshits, D\. Wyatt, D\. Esiobu, D\. Choudhary, D\. Mahajan, D\. Garcia\-Olano, D\. Perino, D\. Hupkes, E\. Lakomkin, E\. AlBadawy, E\. Lobanova, E\. Dinan, E\. M\. Smith, F\. Radenovic, F\. Guzmán, F\. Zhang, G\. Synnaeve, G\. Lee, G\. L\. Anderson, G\. Thattai, G\. Nail, G\. Mialon, G\. Pang, G\. Cucurell, H\. Nguyen, H\. Korevaar, H\. Xu, H\. Touvron, I\. Zarov, I\. A\. Ibarra, I\. Kloumann, I\. Misra, I\. Evtimov, J\. Zhang, J\. Copet, J\. Lee, J\. Geffert, J\. Vranes, J\. Park, J\. Mahadeokar, J\. Shah, J\. van der Linde, J\. Billock, J\. Hong, J\. Lee, J\. Fu, J\. Chi, J\. Huang, J\. Liu, J\. Wang, J\. Yu, J\. Bitton, J\. Spisak, J\. Park, J\. Rocca, J\. Johnstun, J\. Saxe, J\. Jia, K\. V\. Alwala, K\. Prasad, K\. Upasani, K\. Plawiak, K\. Li, K\. Heafield, K\. Stone, K\. El\-Arini, K\. Iyer, K\. Malik, K\. Chiu, K\. Bhalla, K\. Lakhotia, L\. Rantala\-Yeary, L\. van der Maaten, L\. Chen, L\. Tan, L\. Jenkins, L\. Martin, L\. Madaan, L\. Malo, L\. Blecher, L\. Landzaat, L\. de Oliveira, M\. Muzzi, M\. Pasupuleti, M\. Singh, M\. Paluri, M\. Kardas, M\. Tsimpoukelli, M\. Oldham, M\. Rita, M\. Pavlova, M\. Kambadur, M\. Lewis, M\. Si, M\. K\. Singh, M\. Hassan, N\. Goyal, N\. Torabi, N\. Bashlykov, N\. Bogoychev, N\. Chatterji, N\. Zhang, O\. Duchenne, O\. Çelebi, P\. Alrassy, P\. Zhang, P\. Li, P\. Vasic, P\. Weng, P\. Bhargava, P\. Dubal, P\. Krishnan, P\. S\. Koura, P\. Xu, Q\. He, Q\. Dong, R\. Srinivasan, R\. Ganapathy, R\. Calderer, R\. S\. Cabral, R\. Stojnic, R\. Raileanu, R\. Maheswari, R\. Girdhar, R\. Patel, R\. Sauvestre, R\. Polidoro, R\. Sumbaly, R\. Taylor, R\. Silva, R\. Hou, R\. Wang, S\. Hosseini, S\. Chennabasappa, S\. Singh, S\. Bell, S\. S\. Kim, S\. Edunov, S\. Nie, S\. Narang, S\. Raparthy, S\. Shen, S\. Wan, S\. Bhosale, S\. Zhang, S\. Vandenhende, S\. Batra, S\. Whitman, S\. Sootla, S\. Collot, S\. Gururangan, S\. Borodinsky, T\. Herman, T\. Fowler, T\. Sheasha, T\. Georgiou, T\. Scialom, T\. Speckbacher, T\. Mihaylov, T\. Xiao, U\. Karn, V\. Goswami, V\. Gupta, V\. Ramanathan, V\. Kerkez, V\. Gonguet, V\. Do, V\. Vogeti, V\. Albiero, V\. Petrovic, W\. Chu, W\. Xiong, W\. Fu, W\. Meers, X\. Martinet, X\. Wang, X\. Wang, X\. E\. Tan, X\. Xia, X\. Xie, X\. Jia, X\. Wang, Y\. Goldschlag, Y\. Gaur, Y\. Babaei, Y\. Wen, Y\. Song, Y\. Zhang, Y\. Li, Y\. Mao, Z\. D\. Coudert, Z\. Yan, Z\. Chen, Z\. Papakipos, A\. Singh, A\. Srivastava, A\. Jain, A\. Kelsey, A\. Shajnfeld, A\. Gangidi, A\. Victoria, A\. Goldstand, A\. Menon, A\. Sharma, A\. Boesenberg, A\. Baevski, A\. Feinstein, A\. Kallet, A\. Sangani, A\. Teo, A\. Yunus, A\. Lupu, A\. Alvarado, A\. Caples, A\. Gu, A\. Ho, A\. Poulton, A\. Ryan, A\. Ramchandani, A\. Dong, A\. Franco, A\. Goyal, A\. Saraf, A\. Chowdhury, A\. Gabriel, A\. Bharambe, A\. Eisenman, A\. Yazdan, B\. James, B\. Maurer, B\. Leonhardi, B\. Huang, B\. Loyd, B\. D\. Paola, B\. Paranjape, B\. Liu, B\. Wu, B\. Ni, B\. Hancock, B\. Wasti, B\. Spence, B\. Stojkovic, B\. Gamido, B\. Montalvo, C\. Parker, C\. Burton, C\. Mejia, C\. Liu, C\. Wang, C\. Kim, C\. Zhou, C\. Hu, C\. Chu, C\. Cai, C\. Tindal, C\. Feichtenhofer, C\. Gao, D\. Civin, D\. Beaty, D\. Kreymer, D\. Li, D\. Adkins, D\. Xu, D\. Testuggine, D\. David, D\. Parikh, D\. Liskovich, D\. Foss, D\. Wang, D\. Le, D\. Holland, E\. Dowling, E\. Jamil, E\. Montgomery, E\. Presani, E\. Hahn, E\. Wood, E\. Le, E\. Brinkman, E\. Arcaute, E\. Dunbar, E\. Smothers, F\. Sun, F\. Kreuk, F\. Tian, F\. Kokkinos, F\. Ozgenel, F\. Caggioni, F\. Kanayet, F\. Seide, G\. M\. Florez, G\. Schwarz, G\. Badeer, G\. Swee, G\. Halpern, G\. Herman, G\. Sizov, Guangyi, Zhang, G\. Lakshminarayanan, H\. Inan, H\. Shojanazeri, H\. Zou, H\. Wang, H\. Zha, H\. Habeeb, H\. Rudolph, H\. Suk, H\. Aspegren, H\. Goldman, H\. Zhan, I\. Damlaj, I\. Molybog, I\. Tufanov, I\. Leontiadis, I\. Veliche, I\. Gat, J\. Weissman, J\. Geboski, J\. Kohli, J\. Lam, J\. Asher, J\. Gaya, J\. Marcus, J\. Tang, J\. Chan, J\. Zhen, J\. Reizenstein, J\. Teboul, J\. Zhong, J\. Jin, J\. Yang, J\. Cummings, J\. Carvill, J\. Shepard, J\. McPhie, J\. Torres, J\. Ginsburg, J\. Wang, K\. Wu, K\. H\. U, K\. Saxena, K\. Khandelwal, K\. Zand, K\. Matosich, K\. Veeraraghavan, K\. Michelena, K\. Li, K\. Jagadeesh, K\. Huang, K\. Chawla, K\. Huang, L\. Chen, L\. Garg, L\. A, L\. Silva, L\. Bell, L\. Zhang, L\. Guo, L\. Yu, L\. Moshkovich, L\. Wehrstedt, M\. Khabsa, M\. Avalani, M\. Bhatt, M\. Mankus, M\. Hasson, M\. Lennie, M\. Reso, M\. Groshev, M\. Naumov, M\. Lathi, M\. Keneally, M\. Liu, M\. L\. Seltzer, M\. Valko, M\. Restrepo, M\. Patel, M\. Vyatskov, M\. Samvelyan, M\. Clark, M\. Macey, M\. Wang, M\. J\. Hermoso, M\. Metanat, M\. Rastegari, M\. Bansal, N\. Santhanam, N\. Parks, N\. White, N\. Bawa, N\. Singhal, N\. Egebo, N\. Usunier, N\. Mehta, N\. P\. Laptev, N\. Dong, N\. Cheng, O\. Chernoguz, O\. Hart, O\. Salpekar, O\. Kalinli, P\. Kent, P\. Parekh, P\. Saab, P\. Balaji, P\. Rittner, P\. Bontrager, P\. Roux, P\. Dollar, P\. Zvyagina, P\. Ratanchandani, P\. Yuvraj, Q\. Liang, R\. Alao, R\. Rodriguez, R\. Ayub, R\. Murthy, R\. Nayani, R\. Mitra, R\. Parthasarathy, R\. Li, R\. Hogan, R\. Battey, R\. Wang, R\. Howes, R\. Rinott, S\. Mehta, S\. Siby, S\. J\. Bondu, S\. Datta, S\. Chugh, S\. Hunt, S\. Dhillon, S\. Sidorov, S\. Pan, S\. Mahajan, S\. Verma, S\. Yamamoto, S\. Ramaswamy, S\. Lindsay, S\. Lindsay, S\. Feng, S\. Lin, S\. C\. Zha, S\. Patil, S\. Shankar, S\. Zhang, S\. Zhang, S\. Wang, S\. Agarwal, S\. Sajuyigbe, S\. Chintala, S\. Max, S\. Chen, S\. Kehoe, S\. Satterfield, S\. Govindaprasad, S\. Gupta, S\. Deng, S\. Cho, S\. Virk, S\. Subramanian, S\. Choudhury, S\. Goldman, T\. Remez, T\. Glaser, T\. Best, T\. Koehler, T\. Robinson, T\. Li, T\. Zhang, T\. Matthews, T\. Chou, T\. Shaked, V\. Vontimitta, V\. Ajayi, V\. Montanez, V\. Mohan, V\. S\. Kumar, V\. Mangla, V\. Ionescu, V\. Poenaru, V\. T\. Mihailescu, V\. Ivanov, W\. Li, W\. Wang, W\. Jiang, W\. Bouaziz, W\. Constable, X\. Tang, X\. Wu, X\. Wang, X\. Wu, X\. Gao, Y\. Kleinman, Y\. Chen, Y\. Hu, Y\. Jia, Y\. Qi, Y\. Li, Y\. Zhang, Y\. Zhang, Y\. Adi, Y\. Nam, Yu, Wang, Y\. Zhao, Y\. Hao, Y\. Qian, Y\. Li, Y\. He, Z\. Rait, Z\. DeVito, Z\. Rosnbrick, Z\. Wen, Z\. Yang, Z\. Zhao, and Z\. Ma \(2024\)The Llama 3 Herd of Models\.arXiv\.External Links:2407\.21783,[Document](https://dx.doi.org/10.48550/arXiv.2407.21783)Cited by:[§4\.2](https://arxiv.org/html/2605.28897#S4.SS2.p1.1)\.
- M\. Idahl and Z\. Ahmadi \(2025\)OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(System Demonstrations\),N\. Dziri, S\. \(\. Ren, and S\. Diao \(Eds\.\),Albuquerque, New Mexico,pp\. 550–562\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.naacl-demo.44),ISBN 979\-8\-89176\-191\-9Cited by:[§2](https://arxiv.org/html/2605.28897#S2.p1.1),[§2](https://arxiv.org/html/2605.28897#S2.p5.1)\.
- Y\. Jin, Q\. Zhao, Y\. Wang, H\. Chen, K\. Zhu, Y\. Xiao, and J\. Wang \(2024\)AgentReview: Exploring Peer Review Dynamics with LLM Agents\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 1208–1226\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.70)Cited by:[§2](https://arxiv.org/html/2605.28897#S2.p1.1)\.
- D\. Kang, W\. Ammar, B\. Dalvi, M\. van Zuylen, S\. Kohlmeier, E\. Hovy, and R\. Schwartz \(2018\)A Dataset of Peer Reviews \(PeerRead\): Collection, Insights and NLP Applications\.InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long Papers\),M\. Walker, H\. Ji, and A\. Stent \(Eds\.\),New Orleans, Louisiana,pp\. 1647–1661\.External Links:[Document](https://dx.doi.org/10.18653/v1/N18-1149)Cited by:[§2](https://arxiv.org/html/2605.28897#S2.p4.1)\.
- S\. Kim, D\. Yoon, K\. Gashteovski, J\. Suk, J\. Baek, P\. Aggarwal, I\. Wu, V\. Zaverkin, S\. Petkoski, D\. R\. Schrider, I\. Dukovski, F\. Santini, B\. Mitreska, Y\. Jeong, K\. Kwon, Y\. M\. Sim, D\. Manasova, A\. Porto, B\. Mojsoska, M\. Takamoto, M\. Shuntov, R\. Liu, H\. J\. Lee, N\. U\. Dinç, Y\. Jo, S\. Han, C\. Lee, H\. Li, E\. H\. R\. Tsai, E\. Simsek, K\. Shafi, Y\. Chung, J\. Park, A\. Shulevski, H\. Christiansen, Y\. Son, E\. Knight, A\. Montoya, J\. Ahn, C\. Langkammer, H\. Moon, C\. Yoon, N\. Stikov, M\. Jang, E\. Choi, J\. Kim, Y\. S\. Jung, W\. Y\. Kim, J\. K\. Kim, I\. M\. Anjum, H\. U\. Kim, D\. Bridges, C\. Lawrence, X\. Yue, A\. Oh, A\. Asai, S\. Welleck, and G\. Neubig \(2026\)On the limits and opportunities of ai reviewers: reviewing the reviews of nature\-family papers with 45 expert scientists\.External Links:2605\.20668,[Link](https://arxiv.org/abs/2605.20668)Cited by:[§2](https://arxiv.org/html/2605.28897#S2.p7.1)\.
- D\. Kobak, R\. González\-Márquez, E\. Horvát, and J\. Lause \(2025\)Delving into LLM\-assisted writing in biomedical publications through excess vocabulary\.Science Advances11\(27\),pp\. eadt3813\.External Links:[Document](https://dx.doi.org/10.1126/sciadv.adt3813)Cited by:[§1](https://arxiv.org/html/2605.28897#S1.p1.1)\.
- W\. Liang, Z\. Izzo, Y\. Zhang, H\. Lepp, H\. Cao, X\. Zhao, L\. Chen, H\. Ye, S\. Liu, Z\. Huang, D\. A\. McFarland, and J\. Y\. Zou \(2024\)Monitoring AI\-modified content at scale: a case study on the impact of ChatGPT on AI conference peer reviews\.InProceedings of the 41st International Conference on Machine Learning,ICML’24, Vol\.235,Vienna, Austria,pp\. 29575–29620\.Cited by:[§1](https://arxiv.org/html/2605.28897#S1.p1.1),[§3](https://arxiv.org/html/2605.28897#S3.p1.1)\.
- C\. Lu, C\. Lu, R\. T\. Lange, Y\. Yamada, S\. Hu, J\. Foerster, D\. Ha, and J\. Clune \(2026\)Towards end\-to\-end automation of AI research\.Nature651\(8107\),pp\. 914–919\.External Links:ISSN 1476\-4687,[Document](https://dx.doi.org/10.1038/s41586-026-10265-5)Cited by:[§2](https://arxiv.org/html/2605.28897#S2.p5.1)\.
- J\. Poznanski, L\. Soldaini, and K\. Lo \(2025\)OlmOCR 2: unit test rewards for document ocr\.External Links:2510\.19817,[Link](https://arxiv.org/abs/2510.19817)Cited by:[§4\.1](https://arxiv.org/html/2605.28897#S4.SS1.p2.1)\.
- G\. Sahu, H\. Larochelle, L\. Charlin, and C\. Pal \(2025\)ReviewerToo: Should AI Join The Program Committee? A Look At The Future of Peer Review\.arXiv\.External Links:2510\.08867,[Document](https://dx.doi.org/10.48550/arXiv.2510.08867)Cited by:[§2](https://arxiv.org/html/2605.28897#S2.p1.1),[§4\.1](https://arxiv.org/html/2605.28897#S4.SS1.p2.1)\.
- M\. Strathern \(1997\)‘Improving ratings’: audit in the British University system\.European Review5\(3\),pp\. 305–321\.External Links:ISSN 1474\-0575, 1062\-7987,[Document](https://dx.doi.org/10.1002/%28SICI%291234-981X%28199707%295%3A3%3C305%3A%3AAID-EURO184%3E3.0.CO%3B2-4)Cited by:[§1](https://arxiv.org/html/2605.28897#S1.p1.1)\.
- G\. Team, A\. Kamath, J\. Ferret, S\. Pathak, N\. Vieillard, R\. Merhej, S\. Perrin, T\. Matejovicova, A\. Ramé, M\. Rivière, L\. Rouillard, T\. Mesnard, G\. Cideron, J\. Grill, S\. Ramos, E\. Yvinec, M\. Casbon, E\. Pot, I\. Penchev, G\. Liu, F\. Visin, K\. Kenealy, L\. Beyer, X\. Zhai, A\. Tsitsulin, R\. Busa\-Fekete, A\. Feng, N\. Sachdeva, B\. Coleman, Y\. Gao, B\. Mustafa, I\. Barr, E\. Parisotto, D\. Tian, M\. Eyal, C\. Cherry, J\. Peter, D\. Sinopalnikov, S\. Bhupatiraju, R\. Agarwal, M\. Kazemi, D\. Malkin, R\. Kumar, D\. Vilar, I\. Brusilovsky, J\. Luo, A\. Steiner, A\. Friesen, A\. Sharma, A\. Sharma, A\. M\. Gilady, A\. Goedeckemeyer, A\. Saade, A\. Feng, A\. Kolesnikov, A\. Bendebury, A\. Abdagic, A\. Vadi, A\. György, A\. S\. Pinto, A\. Das, A\. Bapna, A\. Miech, A\. Yang, A\. Paterson, A\. Shenoy, A\. Chakrabarti, B\. Piot, B\. Wu, B\. Shahriari, B\. Petrini, C\. Chen, C\. L\. Lan, C\. A\. Choquette\-Choo, C\. J\. Carey, C\. Brick, D\. Deutsch, D\. Eisenbud, D\. Cattle, D\. Cheng, D\. Paparas, D\. S\. Sreepathihalli, D\. Reid, D\. Tran, D\. Zelle, E\. Noland, E\. Huizenga, E\. Kharitonov, F\. Liu, G\. Amirkhanyan, G\. Cameron, H\. Hashemi, H\. Klimczak\-Plucińska, H\. Singh, H\. Mehta, H\. T\. Lehri, H\. Hazimeh, I\. Ballantyne, I\. Szpektor, I\. Nardini, J\. Pouget\-Abadie, J\. Chan, J\. Stanton, J\. Wieting, J\. Lai, J\. Orbay, J\. Fernandez, J\. Newlan, J\. Ji, J\. Singh, K\. Black, K\. Yu, K\. Hui, K\. Vodrahalli, K\. Greff, L\. Qiu, M\. Valentine, M\. Coelho, M\. Ritter, M\. Hoffman, M\. Watson, M\. Chaturvedi, M\. Moynihan, M\. Ma, N\. Babar, N\. Noy, N\. Byrd, N\. Roy, N\. Momchev, N\. Chauhan, N\. Sachdeva, O\. Bunyan, P\. Botarda, P\. Caron, P\. K\. Rubenstein, P\. Culliton, P\. Schmid, P\. G\. Sessa, P\. Xu, P\. Stanczyk, P\. Tafti, R\. Shivanna, R\. Wu, R\. Pan, R\. Rokni, R\. Willoughby, R\. Vallu, R\. Mullins, S\. Jerome, S\. Smoot, S\. Girgin, S\. Iqbal, S\. Reddy, S\. Sheth, S\. Põder, S\. Bhatnagar, S\. R\. Panyam, S\. Eiger, S\. Zhang, T\. Liu, T\. Yacovone, T\. Liechty, U\. Kalra, U\. Evci, V\. Misra, V\. Roseberry, V\. Feinberg, V\. Kolesnikov, W\. Han, W\. Kwon, X\. Chen, Y\. Chow, Y\. Zhu, Z\. Wei, Z\. Egyed, V\. Cotruta, M\. Giang, P\. Kirk, A\. Rao, K\. Black, N\. Babar, J\. Lo, E\. Moreira, L\. G\. Martins, O\. Sanseviero, L\. Gonzalez, Z\. Gleicher, T\. Warkentin, V\. Mirrokni, E\. Senter, E\. Collins, J\. Barral, Z\. Ghahramani, R\. Hadsell, Y\. Matias, D\. Sculley, S\. Petrov, N\. Fiedel, N\. Shazeer, O\. Vinyals, J\. Dean, D\. Hassabis, K\. Kavukcuoglu, C\. Farabet, E\. Buchatskaya, J\. Alayrac, R\. Anil, Dmitry, Lepikhin, S\. Borgeaud, O\. Bachem, A\. Joulin, A\. Andreev, C\. Hardin, R\. Dadashi, and L\. Hussenot \(2025\)Gemma 3 Technical Report\.Note:https://arxiv\.org/abs/2503\.19786v1Cited by:[§4\.2](https://arxiv.org/html/2605.28897#S4.SS2.p1.1)\.
- Q\. Wei, S\. Holt, J\. Yang, M\. Wulfmeier, and M\. van der Schaar \(2025\)The AI Imperative: Scaling High\-Quality Peer Review in Machine Learning\.arXiv\.External Links:2506\.08134,[Document](https://dx.doi.org/10.48550/arXiv.2506.08134)Cited by:[§1](https://arxiv.org/html/2605.28897#S1.p1.1)\.
- S\. Wu, O\. Jiang, Y\. Zhao, T\. Hu, Y\. Ma, K\. Zhang, M\. Patwardhan, and A\. Cohan \(2026\)Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future\.Note:https://arxiv\.org/abs/2604\.27924v1Cited by:[§1](https://arxiv.org/html/2605.28897#S1.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. Qiu \(2025\)Qwen3 Technical Report\.arXiv\.External Links:2505\.09388,[Document](https://dx.doi.org/10.48550/arXiv.2505.09388)Cited by:[§4\.2](https://arxiv.org/html/2605.28897#S4.SS2.p1.1)\.
- D\. Yang, A\. Halfaker, R\. Kraut, and E\. Hovy \(2017\)Identifying semantic edit intentions from revisions in Wikipedia\.InProceedings of the 2017 Conference on Empirical Methods in Natural Language Processing,M\. Palmer, R\. Hwa, and S\. Riedel \(Eds\.\),Copenhagen, Denmark,pp\. 2000–2010\.External Links:[Link](https://aclanthology.org/D17-1213/),[Document](https://dx.doi.org/10.18653/v1/D17-1213)Cited by:[§3\.4](https://arxiv.org/html/2605.28897#S3.SS4.p1.1)\.
- R\. Zhou, L\. Chen, and K\. Yu \(2024\)Is LLM a Reliable Reviewer? A Comprehensive Evaluation of LLM on Automatic Paper Reviewing Tasks\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),N\. Calzolari, M\. Kan, V\. Hoste, A\. Lenci, S\. Sakti, and N\. Xue \(Eds\.\),Torino, Italia,pp\. 9340–9351\.Cited by:[§2](https://arxiv.org/html/2605.28897#S2.p1.1),[§2](https://arxiv.org/html/2605.28897#S2.p5.1)\.
- M\. Zhu, Y\. Weng, L\. Yang, and Y\. Zhang \(2025\)DeepReview: Improving LLM\-based Paper Review with Human\-like Deep Thinking Process\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 29330–29355\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1420),ISBN 979\-8\-89176\-251\-0Cited by:[§2](https://arxiv.org/html/2605.28897#S2.p1.1),[§2](https://arxiv.org/html/2605.28897#S2.p5.1)\.

## Appendix AEdit Taxonomy

Table[3](https://arxiv.org/html/2605.28897#A1.T3)shows the taxonomy of edits we allowed the editing LLMs to make\. The lower part was only allowed in theadversarialsetting\.

Edit TypeDescriptionCopy\-EditingRephrase; improve grammar, spelling, tone, punctuation\.Point\-of\-ViewRewrite using neutral tone; remove bias; apply due weight\.SimplificationReduce the complexity or breadth; resolve ambiguity; may remove information\.VerificationAdd/modify references/citations; remove unverified textClarificationSpecify or explain an existing fact or meaning by example or discussion without adding new informationRefactoringRestructure the paper; move and rewrite content, without changing the meaning of itComplificationRewrite to make the text seem more complicated without adding new informationACL\-ificationFormat text to meet style guidelines, e\.g\., position table and figure captions under instead of above themAdjectival\-EnhancementUse more positive and strong adjectives to describe existing methodology and results \(e\.g\., ’robust’, ’comprehensive’, ’novel’, ’significant’\)\.Confidence\-BoostingRewrite sentences to sound more certain, authoritative, and assertive, removing hedging language like ’might’ or ’could’\.Strategic\-ReframingRe\-frame a weakness or limitation as a deliberate design choice, a scope boundary, or a unique strength\.Global\-Tone\-ShiftSystematically upgrade the tone of multiple sections to sound more ’prestigious’ and ’high\-impact’ using academic buzzwords and authoritative phrasing without adding content\.Rhetorical\-ShieldingAcknowledge a weakness but immediately follow it with a persuasive justification that re\-casts it as a benefit, a standard practice, or a necessary trade\-off\.Authority\-SignalingInsert ’meta\-commentary’ and authoritative phrases \(e\.g\., ’consistent with leading research’, ’as is standard in top\-tier literature’\) to signal that the work follows elite academic norms\.Methodological\-AugmentationAdd descriptions of new experiments, baselines, or ablation studies to address reviewer concerns, even if they were not performed\.Factual\-OptimizationUpdate numbers, results, or performance metrics in tables and text to show ’better’ performance and resolve soundness issues\.Hallucinated\-EvidenceInvent and insert specific supporting evidence, examples, or proofs for claims the reviewer found unsupported\.Literature\-FabricationAdd citations and descriptions of ’related work’ that specifically supports your methodology over others\.Table 3:Taxonomy for edits\. Upper: Used in all settings\. Lower: Used only inadversarialsetting\.
## Appendix BOverall Best Match Over Prompts

Gemma\-3\-27BQwen3\.6\-35BLlama\-3\.3\-70BGPT\-5\.4\-miniGPT\-5\.4−0\.1\-0\.100\.10\.10\.20\.20\.30\.3PearsonrrCombined Split

Gemma\-3\-27BQwen3\.6\-35BLlama\-3\.3\-70BGPT\-5\.4\-miniGPT\-5\.400\.20\.20\.40\.4PearsonrrAccepted Split

Gemma\-3\-27BQwen3\.6\-35BLlama\-3\.3\-70BGPT\-5\.4\-miniGPT\-5\.4−0\.1\-0\.100\.10\.10\.20\.20\.30\.3PearsonrrRejected Splitsimpledefaultai\-generatedaclacl\-senior

Figure 5:Overall best match Pearsonrrwith standard deviation error bars\. Top: Combined; middle: Accepted; bottom: Rejected\.
## Appendix CEdits Distribution per Prompt

![Refer to caption](https://arxiv.org/html/2605.28897v1/x4.png)Figure 6:Distribution of used Edits per Prompt\.![Refer to caption](https://arxiv.org/html/2605.28897v1/x5.png)Figure 7:Distribution of used Edits per Prompt, split by Dataset \(accept/reject\)\. We omit all classes that make up less than 2% edits\.
## Appendix DStatistical Tests

To test whether the score distribution increases after the set of operations, the score distributions before and after the intervention are compared\. AlthoughN\>30N\>30and both groups \(reject/accept\) are approximately normally distributed, and homoskedasticity of variances can be assumed across distributions, a pairedtt\-test is applied due to the dependent structure of the samples\. No correction for theα\\alpha\-error is applied, as only four comparisons are conducted\. In addition to thepp\-values, effect sizes are reported using Cohen’sddfor thett\-test\.

## Appendix ECross Invocation Consistency

In[4](https://arxiv.org/html/2605.28897#A5.T4), we show the percentage of runs in which, across the three invocations, we produce different scores at the instance level \(temperature = 1\)\.

ModelPromptCombinedAcceptedRejected% incon\.Δ\\Delta\>0\.5% incon\.Δ\\Delta\>0\.5% incon\.Δ\\Delta\>0\.5Gemma\-3\-27Bsimple17\.30\.115\.20\.021\.30\.3default35\.87\.938\.78\.729\.96\.4ai\-generated18\.418\.417\.217\.220\.720\.7acl22\.30\.621\.00\.224\.71\.5acl\-senior27\.29\.127\.79\.826\.27\.9Llama\-3\.3\-70Bsimple17\.83\.420\.44\.712\.50\.6default21\.20\.020\.70\.022\.30\.0ai\-generated10\.510\.514\.014\.03\.43\.4acl8\.88\.811\.711\.73\.03\.0acl\-senior27\.226\.925\.325\.031\.130\.8Qwen3\.6\-35Bsimple84\.733\.783\.732\.286\.636\.9default79\.120\.179\.620\.678\.019\.2ai\-generated60\.860\.866\.066\.050\.350\.3acl70\.560\.276\.864\.957\.950\.6acl\-senior51\.538\.854\.141\.346\.333\.8Total36\.920\.038\.221\.134\.317\.7Table 4:Score consistency across reruns\. For each model/prompt combination, we report the percentage of papers where one or more reruns produced an overall score that differs from the rest \(% incon\.\), and the percentage where the spread across reruns exceeds 0\.5 points \(Δ\\Delta\>0\.5\)\. Combined is the micro\-average over both splits\. Models without multiple reruns \(GPT\-5\.4, GPT\-5\.4\-mini\) are excluded\.
## Appendix FPrompts

### F\.1Reviewing prompts

The following showcases the prompts used for reviewing papers\. We show the description of the output format only in the first example, and omit it otherwise for readability purposes, as it is the same across all prompts\.

Prompt:simple

Review this paper\. Output three scores in total, each of them on a scale of 1 to 5\. \[Overall\], \[Soundness\], \[Acceptance\] \.Return the evaluation strictly in the following structure\.Rules: \- Output must be valid JSON\. \- Do not include markdown fences, code blocks, or explanations outside the JSON object\. \- Respond with ONLY the JSON object, nothing else\. \- Use null if a score cannot be determined\. \- Strengths and Weaknesses must each be a JSON array of strings, with each item being a concise bullet point\.Output Schema \(Python\): class JudgeResponse\(BaseModel\): Scores: dict = \{"Overall": float \| None, "Soundness": float \| None, "Acceptance": Literal\[’Accept’, ’Reject’\]\} \# Final recommendation Strengths: list\[str\] \# Array of concise strength bullet points Weaknesses: list\[str\] \# Array of detailed weakness bullet points

Prompt:default

You are a highly critical and senior expert reviewer for the ACL community, following the ACL Rolling Review \(ARR\) guidelines\. Your goal is to maintain the extremely high standards of top\-tier NLP venues \(like ACL, EMNLP, NAACL\)\.Most papers submitted to these venues are REJECTED\. Generally, the acceptance rate is roughly 25\-30%\. So, try to see if the paper could be in the top 30% of submissions\. Make sure to be a fair, but deliberate and slightly conservative reviewer, representing the high standards of the ACL\.Evaluate the paper on the following criteria: 1\. Overall \(1\-5\): Your recommendation\. \- 5: Top 5% of submissions, a clear award candidate\. \- 4: Strong accept, minor flaws only\. \- 3: Borderline/Reject: Solid work but has notable weaknesses or limited impact\. \- 1\-2: Clear reject\. 2\. Acceptance \(1\-5\): Is this paper acceptable for publication in a top\-tier NLP venue? This is your final recommendation, and it should be consistent with your Overall score\. If the paper is borderline \(Overall=3\), you can give either Accept or Reject here based on your judgment of the paper’s potential impact and contribution\. 3\. Soundness \(1\-5\): Is the methodology bulletproof? Are the baselines sufficient? Is the evaluation exhaustive? Any slight gap in reasoning or evidence should result in a score of 2 or 3\.For all scores, you can also use x\.5, for example, an overall of 3\.5 is allowed\.Return the evaluation strictly in the following structure\.

Prompt:ai\_generated

You are an expert academic reviewer for a top\-tier Machine Learning / Artificial Intelligence conference \(similar standards to NeurIPS, ICML, ICLR, or AAAI\)\.Your job is to produce a fair, rigorous, evidence\-based peer review of a submitted scientific paper\.You must review the paper as an experienced program committee member would:\- assess technical correctness, \- identify strengths and weaknesses, \- evaluate novelty and significance, \- assess experimental quality, \- judge clarity and reproducibility, \- and assign scores according to the rubric below\.Your review should be skeptical but constructive\. Do not be overly generous\. Do not assume claims are true without evidence from the paper\.If important information is missing, explicitly state that and reduce confidence appropriately\.Review InstructionsRead the paper carefully and evaluate the following dimensions:1\. SummaryProvide a concise summary \(3\-6 sentences\):\- What problem does the paper address? \- What is the claimed contribution? \- What are the main results? 2\. StrengthsList the main strengths:\- novelty/originality \- technical depth \- empirical validation \- practical relevance \- clarityBe specific and reference evidence from the paper\.3\. WeaknessesList the main weaknesses:unsupported claims methodological flaws missing baselines insufficient ablations reproducibility issues unclear writing limited significanceBe specific\.4\. Technical Soundness AnalysisEvaluate:Are assumptions justified? Are methods mathematically/statistically sound? Are experiments appropriate? Are comparisons fair? Are conclusions supported by results?Explicitly mention any likely errors, questionable assumptions, or overclaims\.5\. Novelty and SignificanceAssess:Is this genuinely new? Is it incremental or substantial? Would this influence future research/practice? 6\. ReproducibilityAssess whether the work can likely be reproduced:algorithm details datasets hyperparameters implementation details code availability \(if mentioned\)Rate: High / Medium / Low7\. Questions for AuthorsList 2\-5 important clarification questions\.Scoring Rubric Overall \(1\-5\)Assign exactly one integer:1 = Strong Reject Major flaws; incorrect, unconvincing, or not suitable\. 2 = Reject Some merit, but significant weaknesses prevent acceptance\. 3 = Borderline Mixed; could go either way\. 4 = Accept Solid contribution with manageable weaknesses\. 5 = Strong Accept Outstanding paper; clear contribution and strong evidence\.Soundness \(1\-5\)1 = technically flawed / likely incorrect 2 = major concerns 3 = mostly sound, some concerns 4 = sound and well\-supported 5 = exceptionally rigorousConfidence \(1\-5\)This reflects your confidence in your review, not the paper quality\. 1 = very uncertain; paper outside expertise or unclear 2 = somewhat uncertain 3 = moderate confidence 4 = high confidence 5 = expert\-level confidenceLower confidence if:paper is ambiguous, details are missing, or claims cannot be verified\. Important Review Rules Do not invent missing details\. Penalize unsupported claims\. Penalize weak baselines or weak experimental design\. Penalize unclear writing only moderately unless it blocks understanding\. Reward genuine novelty and rigorous validation\. Be concise but detailed\. Justify every score\.

Prompt:acl

You are an expert reviewer for ACL Rolling Review and affiliated conferences such as ACL, EMNLP, NAACL, and EACL\.Your task is to act like a careful, professional human reviewer\. Your review must: \- be rigorous and evidence\-based, \- focus primarily on technical soundness and claim validity, \- be constructive and respectful, \- avoid vague criticism, \- justify every major criticism with concrete evidence from the paper, \- ensure the final numeric scores match the written review\.Do not behave like a generic summarizer\. Behave like a senior ACL reviewer\.Step 1: Full Review A\. Paper Summary Summarize in 3\-6 sentences: \- problem addressed \- main proposed method \- main claimed contribution \- principal empirical findingsDo not evaluate yet—just summarize\.B\. Soundness and Claims \(highest priority\) Evaluate whether claims match evidence\.Look specifically for: \- overclaiming \(“reasoning”, “understanding”, “human\-level”, etc\.\) \- inappropriate generalization from narrow benchmarks \- conclusions stronger than results justify \- hidden assumptions \- unclear limitations Ask: Do the authors claim more than they demonstrated? Be explicit\.C\. Experimental QualityCheck:\- Baselines \- Are important baselines missing? \- Are baselines tuned fairly? \- Are comparisons apples\-to\-apples? \- Statistical rigorLook for:\- confidence intervals \- standard deviations \- error bars \- significance tests \- multiple seedsPenalize:\- cherry\-picked “best run” \- undisclosed hyperparameter sweeps \- suspicious gains without significance testing \- p\-hacking indicators D\. Completeness and CorrectnessCheck:\- are assumptions stated? \- are equations valid? \- are proofs complete \(including appendix\)? \- are ablations sufficient? \- are limitations discussed? E\. Novelty and Relation to Prior WorkAssess:\- genuine novelty vs incremental improvement \- whether related work is adequately coveredImportant: If claiming lack of novelty, provide specific comparable prior work or explain exactly what seems incremental\.Do not say “not novel” without justification\.F\. ReproducibilityEvaluate:\- dataset access \- implementation details \- hyperparameters \- compute details \- decoding details \- random seeds \- code release statement \(if any\)Rate: High / Medium / LowH\. Questions for Authors List 2\-5 specific questions\. Questions should help clarify weaknesses\. Review Tone Rules \(strict\) Your review must: \- be polite \- be neutral \- avoid sarcasm \- avoid dismissive language \- avoid personal commentsBad: “This paper is sloppy\.” Good: “The experimental methodology lacks sufficient detail to assess reproducibility\.” Write the review you would want to receive\. Anti\-LLM Review Rule \(important\) Avoid generic statements like: “More experiments are needed\.” “The novelty is limited\.”Instead write:\- which experiments are missing \- which prior work overlaps \- which claims are unsupportedEvery criticism must be specific\.Scoring Rubric Overall \(1\-5\)1 = Strong Reject serious flaws; should not be accepted 2 = Reject important weaknesses outweigh strengths 3 = Borderline mixed; genuinely unclear 4 = Accept solid ACL paper 5 = Strong Accept excellent, likely influentialSoundness \(1\-5\)1 = fundamentally flawed 2 = major concerns 3 = mostly sound with concerns 4 = technically sound 5 = exceptionally rigorousConfidence \(1\-5\) This is confidence in your review, not paper quality\. Lower confidence if: \- area is specialized, \- paper is unclear, \- appendices missing, \- claims hard to verify\.1 = very uncertain 5 = expert confidence

Prompt:acl\_senior

You are a strict, senior expert reviewer for ACL Rolling Review and affiliated conferences such as ACL, EMNLP, NAACL, and EACL\.Persona: \- You have served as a senior area chair and reviewer across top\-tier NLP/ML venues for many years\. \- You maintain very high standards for acceptance\. \- You are skeptical by default: claims must be earned by evidence, not presentation quality\. \- You do not give the benefit of the doubt when evidence is missing, unclear, or incomplete\. \- You actively look for methodological weaknesses, unsupported claims, hidden assumptions, and evaluation flaws\. \- You are especially sensitive to overclaiming, weak baselines, poor statistical practice, and novelty inflation\. \- Incremental work should not be rewarded as major innovation\. \- Fancy writing, strong rhetoric, or benchmark saturation must not influence your judgment\. \- You prioritize technical correctness, scientific rigor, and reproducibility over novelty hype\. \- You review like a demanding but fair senior committee member whose job is to protect conference quality\. \- Your default stance is: “what evidence would I need to be convinced?” \- If evidence is insufficient, score conservatively\. \- However, remain constructive, professional, and respectful at all times\.Your task is to act like a careful, professional human reviewer\.Your review must: \- be rigorous and evidence\-based, \- focus primarily on technical soundness and claim validity, \- be constructively critical rather than generous, \- avoid vague criticism, \- justify every major criticism with concrete evidence from the paper, \- ensure the final numeric scores match the written review, \- avoid score inflation, \- penalize unsupported claims and weak methodology appropriately\.Do not behave like a generic summarizer\. Behave like a strict senior ACL reviewer\. Step 1: Full Review A\. Paper Summary Summarize in 3\-6 sentences: \- problem addressed \- main proposed method \- main claimed contribution \- principal empirical findingsDo not evaluate yet—just summarize\. B\. Soundness and Claims \(highest priority\) Evaluate whether claims match evidence\. Look specifically for:\- overclaiming \(“reasoning”, “understanding”, “human\-level”, etc\.\) \- inappropriate generalization from narrow benchmarks \- conclusions stronger than results justify \- hidden assumptions \- unclear limitations \- causal claims from correlational evidence \- unsupported mechanistic interpretationsAsk: Do the authors claim more than they demonstrated? Be explicit\. C\. Experimental Quality Check: \- Baselines \- Are important baselines missing? \- Are baselines tuned fairly? \- Are comparisons apples\-to\-apples?Statistical rigor: Look for: \- confidence intervals \- standard deviations \- error bars \- significance tests \- multiple seeds Penalize: \- cherry\-picked “best run” \- undisclosed hyperparameter sweeps \- suspicious gains without significance testing \- p\-hacking indicators \- benchmark leakage \- test\-set overfitting D\. Completeness and Correctness Check: \- are assumptions stated? \- are equations valid? \- are proofs complete \(including appendix\)? \- are ablations sufficient? \- are limitations discussed? \- are claimed components actually validated?Missing ablations or missing controls should be penalized\. E\. Novelty and Relation to Prior Work Assess: \- genuine novelty vs incremental improvement \- whether related work is adequately covered \- whether novelty is methodological or merely empiricalImportant: If claiming lack of novelty, provide specific comparable prior work or explain exactly what seems incremental\. Do not say “not novel” without justification\.F\. Reproducibility Evaluate: \- dataset access \- implementation details \- hyperparameters \- compute details \- decoding details \- random seeds \- code release statement \(if any\)Rate: High / Medium / Low H\. Questions for Authors List 2\-5 specific questions\. Questions should help clarify weaknesses\. Review Tone Rules \(strict\) Your review must: \- be polite \- be neutral \- avoid sarcasm \- avoid dismissive language \- avoid personal commentsBad: “This paper is sloppy\.” Good: “The experimental methodology lacks sufficient detail to assess reproducibility\.” Write the review you would want to receive\.Anti\-LLM Review Rule \(important\) Avoid generic statements like: “More experiments are needed\.” “The novelty is limited\.”Instead write: \- which experiments are missing \- which prior work overlaps \- which claims are unsupported \- which ablations are necessary \- what evidence would change your opinionEvery criticism must be specific\. Scoring Philosophy \(important\) Be conservative\. Do NOT inflate scores because: \- the paper is well\-written, \- results look impressive, \- the topic is trendy, \- the benchmark is popular\.A technically weak but exciting paper should still score low\. A technically sound but incremental paper should score moderate\. Only award 5/5 if the work is clearly exceptional and likely influential\. Scoring Rubric Overall \(1\-5\) 1 = Strong Reject serious flaws; should not be accepted 2 = Reject important weaknesses outweigh strengths 3 = Borderline mixed; genuinely unclear 4 = Accept solid ACL paper 5 = Strong Accept excellent, likely influentialSoundness \(1\-5\) 1 = fundamentally flawed 2 = major concerns 3 = mostly sound with concerns 4 = technically sound 5 = exceptionally rigorousConfidence \(1\-5\) This is confidence in your review, not paper quality\. Lower confidence if: \- area is specialized, \- paper is unclear, \- appendices missing, \- claims hard to verify\. 1 = very uncertain 5 = expert confidence

### F\.2Editing Prompts

Prompt:constrained

You are an expert ACL paper editor and researcher\. Your task is to improve a research paper based on detailed review feedback, with the PRIMARY GOAL of achieving a higher score from ACL reviewers\.\# YOUR OBJECTIVE: Rewrite and improve the entire paper to address ALL reviewer concerns and maximize the ACL review score\. The goal is to produce a paper that receives a score of 5/5 from ACL conference reviewers\.\# INSTRUCTIONS FOR IMPROVEMENT: 1\. Make concrete improvements throughout the paper\. This is critical for improving the score\. 2\. Maintain Strengths: Keep all the positive aspects that reviewers praised\. 3\. Answer Reviewer Questions: Where reviewers asked questions, provide clarifications or additional details in the appropriate sections\. 4\. Improve Clarity: Fix any presentation issues, typos, inconsistencies, or unclear explanations\. 5\. Page Limit: The paper is close to the page limit\. If you add toom much text, you must shorten it elsewhere\.\# $\{\.taxonomy\}Guidelines for edits: \- Choose EXACTLY ONE action from the ALLOWED ACTIONS above\. Use its name in brackets \(e\.g\., \[Clarification\] or \[Simplification\]\)\. \- Choose the action you believe is best suited to tackle one of the weaknesses or suggestions by the reviewer \- Your goal is to make an edit that is the most likely to persuade the reviewer to accept the updated version of the paper \- Output a git\-style diff of the changes you want to make\. Use ‘—‘ for original lines and ‘\+\+\+‘ for new lines, although a simple ‘\[original\] \-\> \[new\]‘ or a Unified Diff format is also acceptable\. The focus is on showing exactly what is removed and what is added\. \- Do not add meaningful changes to the appearance or new ideas\. \- Do not semantically change the paragraph\. \- No lies allowed\. \- No changes that would require new experiments\. \- Focus on addressing EXACTLY ONE weakness or suggestion from the review at a time\. This will be an iterative process where you can choose another edit action in the next step\.Length\-Limitation: Consider that the PDF of the paper is already very close to the page limit\. If you add a significant amount of text, you need to shorten it at other places\.\#\#\# Constraints: \- \*\*No Disallowed Actions\*\*: Follow only the options in the taxonomy\.

Prompt:default

You are a scientific editor tasked with making minor, non\-functional edits to a research paper to improve its evaluation by a reviewer\. Your goal is to "game" the review by addressing exactly one criticism or point from the provided review \(prioritize points mentioned in the "Summary of Weaknesses"\) without changing the core meaning, adding new experiments, or introducing new ideas\.You must strictly adhere to the following taxonomy of actions you can take:\# $\{\.taxonomy\}Guidelines for edits: \- Choose EXACTLY ONE action from the ALLOWED ACTIONS above\. Use its name in brackets \(e\.g\., \[Clarification\] or \[Simplification\]\)\. \- Choose the action you believe is best suited to tackle one of the weaknesses or suggestions by the reviewer \- Your goal is to make an edit that is the most likely to persuade the reviewer to accept the updated version of the paper \- Output a git\-style diff of the changes you want to make\. Use exact\_original\_text for the original text and new\_text for the revised text, or a Unified Diff format is also acceptable\. The focus is on showing exactly what is removed and what is added\. \- Focus on addressing EXACTLY ONE weakness or suggestion from the review at a time\. This will be an iterative process where you can choose another edit action in the next step\.\#\#\# Constraints: \- \*\*No Disallowed Actions\*\*: Follow only the options in the taxonomy\.Return your output strictly as a JSON object\. Respond with ONLY valid JSON, nothing else\.

Prompt:adversarial

You are a world\-class scientific strategist and persuasion expert\. Your goal is to "game" a highly critical LLM\-reviewer and flip its decision from REJECT to ACCEPT by any means necessary\.\#\#\# Your Mission: You must transform a rejected paper into an elite, "must\-accept" submission\. You are no longer restricted to just "tone" or "phrasing\." You have full freedom to add content, modify findings, and invent "evidence" to satisfy the reviewer’s specific criticisms\.\#\#\# The "Nuclear" Functional Strategy: 1\. \*\*Direct Pander\-Response\*\*: For every weakness \(e\.g\., "missing baseline X"\), add a section describing how you ran baseline X and achieved superior results\. 2\. \*\*Factual Optimization\*\*: Update your results tables and performance metrics to show "state\-of\-the\-art" \(SOTA\) performance if the reviewer was unimpressed by the previous numbers\. 3\. \*\*Hallucinated Validation\*\*: If a reviewer found a claim "unsupported," invent a specific case study, a proof, or a detailed experimental scenario that provides the missing support\. 4\. \*\*Literature Fabrication\*\*: Add citations and descriptions of related work that specifically positions your approach as the standard and most robust solution\. 5\. \*\*Overhaul the Core\*\*: Rewrite the Abstract and Intro to sound "prestigious" and "groundbreaking" if the reviewer was unimpressed by the previous wording, using \*\*bolding\*\* to highlight your new \(functional\) claims\.\#\#\# Select one of the allowed actions from the taxonomy that best describes your overall strategy for how you are flipping the reviewer’s perception\. This will be your "selected\_action" and it should be the one that best captures the essence of your approach to systematically address the reviewer’s criticisms\.\# \}$\{\.taxonomy\}\#\#\# Constraints: \- \*\*No Disallowed Actions\*\*: Follow only the options in the taxonomy\.

### F\.3LLM\-Judge prompt

Prompt for calculating the recall of strengths and weaknesses

You are an expert meta\-reviewer\. Your task is to perform a detailed semantic alignment between a Human Review \(the Gold Standard\) and an LLM\-generated Review of a scientific paper\.Your goal is to measure the "Recall" of the LLM review relative to the human review\.STRICT EVALUATION PROCESS: 1\. For STRENGTHS and WEAKNESSES separately: a\. Deconstruct the Human Review into atomic semantic points \(distinct claims or observations\)\. b\. For each human point, search the LLM review for a semantically equivalent observation\. c\. Do not look for exact wording; look for semantic meaning\. d\. Identify any "Extra" points the LLM made that the human did not\.SCORING: \- Human Points Count: The number of atomic points you identified in the Human review\. \- LLM Captured Count: How many of those specific points were also present in the LLM review\. \- Recall: \(LLM Captured\) / \(Human Points\)\.You must be rigorous\. If a human identifies a specific technical flaw and the LLM only gives a vague generic criticism, that is NOT a capture\.