基于大语言模型的学术论文质量评估评审范式对比研究

# Comparative Evaluation of LLM-Based Review Paradigms for Academic Paper Quality Assessment
Source: [https://arxiv.org/html/2610.41391](https://arxiv.org/html/2610.41391)
Fan HuangHongyu ZhouAffiliation:School of Information Resource Management, Renmin University of China, Beijing, ChinaAffiliation:Faculty of Social Sciences, University of Antwerp, Antwerp, BelgiumEmail:[hzhou@ruc\.edu\.cn](mailto:[email protected])

###### Abstract

Growing submission volumes have renewed interest in using large language models \(LLMs\) to support peer review\. It remains unclear how such evaluation should be structured, how much the model matters, and whether LLM scores predict research impact\. We evaluate five paradigms across three input representations and three LLMs on 232 stratified ICLR 2025 submissions\. Direct absolute rating matches comparative ranking on rank correlation with human scores, yet compresses most predictions above the accept threshold, flagging almost none of the papers reviewers scored below it\. A base\-rate\-matched recalibration control locates the cause: comparative ranking supplies placement, not better ordering, and after recalibration Direct Rating reaches higher agreement than Elo while making 24% as many API calls\. Abstract\-only input beats full\-paper summarization, and model choice matters as much as paradigm choice\. Replicating on all 11,513 scored submissions reproduces these patterns, shows absolute agreement depends on the evaluated set’s score spread, and finds none beating an always\-reject baseline in accuracy\. Human scores track citations better than any paradigm, with one exception: Group Rating retains its citation correlation after controlling for human scores, and alone ranks within the accepted pool\. Ranking quality alone cannot assess review quality; calibration, thresholds, model, and target outcome all matter\.

## 1Introduction

Figure 1:Overview of the experimental pipeline\. Papers pass through three input representations \(top\), then through \(a\) absolute rating methods, which score each paper independently, or \(b\) comparative rating methods, which infer a latent rating from head\-to\-head outcomes and rescale it to 1–10\. Only the comparative family is rescaled, so inflation and other absolute\-error metrics are not comparable across families\. Inline metrics give the best configuration in each family on the stratified 232\-paper sample at the fixed≥6\\geq 6threshold\.Major machine learning conferences now receive more than 10,000 submissions a year without a matching growth in reviewers, renewing concern about the consistency, quality, and sustainability of peer review\([Checco et al\., 2021](https://arxiv.org/html/2610.41391#bib.bib2);[Shah, 2022](https://arxiv.org/html/2610.41391#bib.bib1);[Wei et al\., 2026](https://arxiv.org/html/2610.41391#bib.bib24);[Thakkar et al\., 2025](https://arxiv.org/html/2610.41391#bib.bib28)\)\. Venues have responded at scale: a randomized LLM\-feedback trial at ICLR 2025\([Thakkar et al\., 2025](https://arxiv.org/html/2610.41391#bib.bib28)\), an AI review on every main\-track submission at AAAI 2026\([Biswas et al\., 2026](https://arxiv.org/html/2610.41391#bib.bib25)\)\. LLM\-written reviews also appear unprompted, motivating detection work\([Yu et al\., 2026](https://arxiv.org/html/2610.41391#bib.bib29);[Rao et al\., 2025](https://arxiv.org/html/2610.41391#bib.bib30)\)\. Large language models \(LLMs\) read, compare, and evaluate text competently\([OpenAI, 2023](https://arxiv.org/html/2610.41391#bib.bib3);[Zheng et al\., 2023](https://arxiv.org/html/2610.41391#bib.bib8)\), and a growing literature asks how much of this load they can absorb\([Zhuang et al\., 2025](https://arxiv.org/html/2610.41391#bib.bib23)\)\.

Using an LLM to assess a paper is not one method but a design space with three largely independent choices: the*review paradigm*, the procedure that turns a paper into a score; the*input representation*, how much of the paper the model reads; and the model itself\. Absolute scoring, the most immediate paradigm, is hard to calibrate: LLM judges show systematic scoring biases, and single\-answer scores are argued to be less stable than pairwise preferences\([Zheng et al\., 2023](https://arxiv.org/html/2610.41391#bib.bib8);[Wang et al\., 2024](https://arxiv.org/html/2610.41391#bib.bib17)\); their scores favor specific scale values\([Li et al\., 2026b](https://arxiv.org/html/2610.41391#bib.bib18);[Fujinuma, 2026](https://arxiv.org/html/2610.41391#bib.bib33)\)and, for most models, exceed human ratings\([Li et al\., 2026a](https://arxiv.org/html/2610.41391#bib.bib27)\)\. We call this*grade inflation*, the mean signed difference between predicted and human scores\. Since an accept/reject decision is a threshold on a score, inflation can leave a system that orders papers sensibly unable to decide about any\.

Prior work predicts acceptance from manuscript features\([Kang et al\., 2018](https://arxiv.org/html/2610.41391#bib.bib4)\), generates review text\([Yuan et al\., 2022](https://arxiv.org/html/2610.41391#bib.bib5);[Gao et al\., 2024](https://arxiv.org/html/2610.41391#bib.bib21);[D’Arcy et al\., 2024](https://arxiv.org/html/2610.41391#bib.bib20)\), and studies frontier models as reviewers\([Liang et al\., 2024](https://arxiv.org/html/2610.41391#bib.bib6);[Robertson, 2023](https://arxiv.org/html/2610.41391#bib.bib7);[Liu and Shah, 2023](https://arxiv.org/html/2610.41391#bib.bib22);[Lu et al\., 2024](https://arxiv.org/html/2610.41391#bib.bib19)\), while a position paper argues deployment is outrunning its evaluation\([Baumann et al\., 2026](https://arxiv.org/html/2610.41391#bib.bib26)\)\. Remedies for miscalibration include calibrating the judge’s scoring criteria against human labels\([Liu et al\., 2024](https://arxiv.org/html/2610.41391#bib.bib9)\)and comparative evaluation, which infers quality from head\-to\-head outcomes rather than an absolute scale: Elo\([Elo, 1978](https://arxiv.org/html/2610.41391#bib.bib11)\)and Swiss tournaments\([Ólafsson, 1990](https://arxiv.org/html/2610.41391#bib.bib12)\)are self\-calibrating in this sense, and Elo is increasingly adopted for LLM evaluation\([Boubdir et al\., 2024](https://arxiv.org/html/2610.41391#bib.bib10)\), and pairwise comparison has been applied to peer review\([Zhang et al\., 2025](https://arxiv.org/html/2610.41391#bib.bib16)\)\.

Two gaps remain\. Existing comparisons vary the paradigm with the model largely fixed\([Zhang et al\., 2025](https://arxiv.org/html/2610.41391#bib.bib16)\)or compare models under one protocol\([Lu et al\., 2024](https://arxiv.org/html/2610.41391#bib.bib19);[Li et al\., 2026a](https://arxiv.org/html/2610.41391#bib.bib27)\), so it is unclear which choice dominates the outcome or whether comparative ranking helps for paper assessment specifically\. And agreement with human scores is often checked by correlation\([Lu et al\., 2024](https://arxiv.org/html/2610.41391#bib.bib19);[Zhang et al\., 2025](https://arxiv.org/html/2610.41391#bib.bib16)\); where accept/reject decisions are evaluated\([Lu et al\., 2024](https://arxiv.org/html/2610.41391#bib.bib19)\), the link between a high correlation and useful decisions, and its dependence on the evaluation sample, is not tested\.

We therefore ask three questions\. RQ1, paradigm design: how should an LLM evaluate papers, and do abstracts alone carry sufficient signal? We compare absolute paradigms \(Direct, Anchor, Group Rating\) against comparative ones \(Elo, Swiss tournament\) under three input representations \(Figure[1](https://arxiv.org/html/2610.41391#S1.F1)\)\. RQ2, model dependence: does model choice matter more than paradigm choice? We replicate every paradigm across three LLMs\. RQ3, external validity: do predicted scores relate to research impact? We correlate them with citation counts\.

We separate*ranking potential*, how a method orders papers, from*decision utility*, the accept/reject decisions it induces, and find they diverge: Elo and Direct Rating track human scores comparably \(ρ=0\.410\\rho=0\.410vs\.0\.4220\.422\), yet Direct flags 4\.3% of below\-threshold papers against Elo’s 66\.4%\. LLM reviewers that accept nearly every paper have been reported before\([Verma and Landrum, 2026](https://arxiv.org/html/2610.41391#bib.bib44)\), as have LLMs that, asked to verify an answer, accept a wrong one at least 76% of the time\([Huang, 2026c](https://arxiv.org/html/2610.41391#bib.bib51)\); we term the paradigm\-dependent, fixed\-threshold form of this*reject blindness*\. A recalibration control traces this to placement rather than ordering: at its own base rate Direct overtakes Elo while making 24% as many API calls\. Abstract\-only input beats full\-paper summarization, model choice matters as much as paradigm choice, agreement depends on the evaluated set’s score distribution, and human scores track citations better than any paradigm tested\.

## 2Related Work

### 2\.1Automated and LLM\-Assisted Peer Review

Early work framed peer review as prediction over manuscript features \(the PeerRead dataset;[Kang et al\., 2018](https://arxiv.org/html/2610.41391#bib.bib4)\) or as learned review generation\([Yuan et al\., 2022](https://arxiv.org/html/2610.41391#bib.bib5)\)\. Instruction\-tuned LLMs shifted attention to generating reviews directly:[Liang et al\. \(2024\)](https://arxiv.org/html/2610.41391#bib.bib6)found partial overlap between GPT\-4 feedback and human reviewers, while smaller studies find GPT\-4 reviews comparable to human ones in average helpfulness but more variable\([Robertson, 2023](https://arxiv.org/html/2610.41391#bib.bib7)\), and GPT\-4 useful for targeted checks such as finding inserted errors and verifying checklist items, though not yet as an evaluator of whole papers\([Liu and Shah, 2023](https://arxiv.org/html/2610.41391#bib.bib22)\)\. Later systems add generated aspect prompts\([Gao et al\., 2024](https://arxiv.org/html/2610.41391#bib.bib21)\), multi\-agent collaboration\([D’Arcy et al\., 2024](https://arxiv.org/html/2610.41391#bib.bib20)\), or full discovery pipelines\([Lu et al\., 2024](https://arxiv.org/html/2610.41391#bib.bib19)\);[Zhuang et al\. \(2025\)](https://arxiv.org/html/2610.41391#bib.bib23)survey the area\. Deployment has since moved ahead of this literature\([Thakkar et al\., 2025](https://arxiv.org/html/2610.41391#bib.bib28);[Biswas et al\., 2026](https://arxiv.org/html/2610.41391#bib.bib25)\): across 75,800 reviews of 19,490 ICLR 2026 submissions[Baumann et al\. \(2026\)](https://arxiv.org/html/2610.41391#bib.bib26)report that AI reviewers agree with one another excessively, and that on 60 randomly sampled papers an automated, mostly stylistic rewrite with no new experiments raised their scores, and a benchmark of twelve models finds that most rate papers higher than human reviewers do\([Li et al\., 2026a](https://arxiv.org/html/2610.41391#bib.bib27)\)\.

### 2\.2Reliability and Bias of LLM Judges

LLM judges are widely used for open\-ended evaluation\([Zheng et al\., 2023](https://arxiv.org/html/2610.41391#bib.bib8)\)but show position and order effects\([Wang et al\., 2024](https://arxiv.org/html/2610.41391#bib.bib17);[Huang et al\., 2026b](https://arxiv.org/html/2610.41391#bib.bib49)\), rubric\- and score\-related biases\([Li et al\., 2026b](https://arxiv.org/html/2610.41391#bib.bib18)\), and sensitivity to the rating scale itself\([Fujinuma, 2026](https://arxiv.org/html/2610.41391#bib.bib33);[Huang et al\., 2024](https://arxiv.org/html/2610.41391#bib.bib47)\)\. Repeated runs of one prompt also yield materially different scores\([Haldar and Hockenmaier, 2025](https://arxiv.org/html/2610.41391#bib.bib35)\), agreement can look strong without validity following\([Norman et al\., 2026](https://arxiv.org/html/2610.41391#bib.bib36)\), and reported agreement depends on protocol choices made before any statistic is computed\([Rao and Callison\-Burch, 2026](https://arxiv.org/html/2610.41391#bib.bib39)\)\. These motivate calibrating the judge’s scoring criteria against human labels\([Liu et al\., 2024](https://arxiv.org/html/2610.41391#bib.bib9)\)and comparative paradigms that avoid an absolute scale, though comparative protocols carry biases of their own\([Tripathi et al\., 2025](https://arxiv.org/html/2610.41391#bib.bib37)\)\.

### 2\.3Comparative and Ranking\-Based Evaluation

Elo\([Elo, 1978](https://arxiv.org/html/2610.41391#bib.bib11)\)and Swiss tournaments\([Ólafsson, 1990](https://arxiv.org/html/2610.41391#bib.bib12)\)estimate latent quality from pairwise outcomes, and Elo is increasingly adopted in LLM evaluation\([Boubdir et al\., 2024](https://arxiv.org/html/2610.41391#bib.bib10)\)\. Their assumptions are not innocuous: LLM judges hold non\-transitive preferences\([Xu et al\., 2025](https://arxiv.org/html/2610.41391#bib.bib34)\), and Elo’s sequential update makes ratings depend on the replay order\([Liu et al\., 2025](https://arxiv.org/html/2610.41391#bib.bib38)\), which is why we fix the pairing schedule in advance, although our ratings are still updated in the order in which the comparisons were returned\. Closest to our work,[Zhang et al\. \(2025\)](https://arxiv.org/html/2610.41391#bib.bib16)report that pairwise comparison selects more\-cited papers than absolute rating, whose scores mostly fell at 6 and 7; against human scores, our recalibration control locates the comparative advantage in score placement rather than ordering, since at each paradigm’s own base rate it disappears\.[Epstein et al\. \(2026\)](https://arxiv.org/html/2610.41391#bib.bib31)aggregate LLM comparisons under a Bradley–Terry model to route marginal reviews;[Tong et al\. \(2026\)](https://arxiv.org/html/2610.41391#bib.bib32)instead train a judge against citation outcomes, while a zero\-shot GPT\-4o\-mini rater of disruption potential, reading only titles and abstracts, identifies the 2% of ICLR papers most cited within ICLR below chance \(AUC 0\.424;[Huang, 2026a](https://arxiv.org/html/2610.41391#bib.bib48)\)\.

## 3Methodology

### 3\.1Dataset Construction

#### Test set \(ICLR 2025\)\.

We constructed a test set of 232 ICLR 2025 submissions, stratified to 30 papers in each score bin from 2 to 8 and every available paper in the sparse bins 1, 9 and 10 \(7, 13 and 2 papers\)\. Ground\-truth scores are the mean OpenReview reviewer ratings\. That target is itself noisy: over the 8,655 non\-withdrawn ICLR 2025 submissions, a one\-way random\-effects analysis\([Shrout and Fleiss, 1979](https://arxiv.org/html/2610.41391#bib.bib14)\)of the per\-reviewer scores gives ICC\(1,1\)=0\.327=0\.327for a single reviewer and ICC\(1,kk\)=0\.663=0\.663for the mean of the 4\.05 reviewers a paper receives on average\. Correcting for attenuation gives a ceiling ofρ=0\.815\\rho=0\.815here and, on the wider\-spread stratified set,ρ=0\.951\\rho=0\.951; correlations should be read against the ceiling of their own sample rather than against 1\. The resulting distribution has a mean of 5\.16 and a standard deviation of 2\.16, spanning strong reject \(1\.0\) to strong accept \(10\.0\)\.

#### Anchor set \(ICLR 2024\)\.

For calibration\-based methods we built an anchor set of 100 ICLR 2024 papers with zero score variance\. Unanimous papers are stable calibration points, but selecting on zero variance collapses their range: standard deviation 1\.66 around a mean of 5\.14, never above 8\.0, against 2\.16 and the full 1–10 range in the test set, and narrower still \(0\.93\) once averaged over the three anchors retrieved per paper\. The calibration points therefore span little of the scale they are meant to calibrate, which we take to be the main reason Anchor Rating performs poorly \(Appendix[K](https://arxiv.org/html/2610.41391#A11)\)\. Details of the PDF\-to\-text preprocessing pipeline are given in Appendix[C](https://arxiv.org/html/2610.41391#A3)\.

### 3\.2Review Paradigms

We evaluate five paradigms\.

*Direct Absolute Rating*\(hereafter*Direct*\) asks the LLM to rate each paper independently on a 1–10 scale, using the explicit score bands and criteria given in Appendix[A](https://arxiv.org/html/2610.41391#A1)\. It uses one call per paper and is susceptible to grade inflation\.

*Anchor Rating*retrieves the three most similar anchor papers by cosine similarity of abstract embeddings \(text\-embedding\-3\-small\) and presents them with their known scores as calibration points\. The same device, the most similar example shown with its scores, raised ChatGPT’s agreement with human judgments of explanation quality in paired comparisons but lowered it on a 7\-point scale\([Huang et al\., 2024](https://arxiv.org/html/2610.41391#bib.bib47)\)\.

*Group Rating*presents papers in sets of five and asks the LLM to score all five jointly on a 1–10 scale, yielding 46 groups with each paper evaluated once\. Groups and within\-prompt order are randomized; an earlier grouping ordered by score depressed the paradigm, and a prompt\-position effect survives randomization \(Appendix[D](https://arxiv.org/html/2610.41391#A4)\)\. It remains absolute: the group supplies context, but the output is still a score on a fixed scale\.

Under*Elo Rating*the LLM is asked, for each binary comparison, which paper is better; ratings update by the Elo rule\([Elo, 1978](https://arxiv.org/html/2610.41391#bib.bib11)\)in its common logistic form

EA\\displaystyle E\_\{A\}=11\+10\(RB−RA\)/s,\\displaystyle=\\frac\{1\}\{1\+10^\{\(R\_\{B\}\-R\_\{A\}\)/s\}\},\(1\)RA′\\displaystyle R^\{\\prime\}\_\{A\}=RA\+K⁡\(SA−EA\),\\displaystyle=R\_\{A\}\+K\\,\(S\_\{A\}\-E\_\{A\}\),\(2\)whereEAE\_\{A\}is the expected score forAAagainstBB,RAR\_\{A\}andRBR\_\{B\}their current ratings,SA∈\{0,12,1\}S\_\{A\}\\in\\\{0,\\tfrac\{1\}\{2\},1\\\}a loss, tie or win, ands=400s=400,K=32K=32the conventional scale and update step\. Each paper receives 10 comparisons on average on the 232\-paper set \(1,160 total\), with final ratings min–max normalized to 1–10\.

*Swiss Tournament*runs⌈log2⁡n⌉\\lceil\\log\_\{2\}n\\rceilrounds \(8 here\): round 1 is random, later rounds pair papers of similar current score while avoiding repeats, and win/tie/loss records are normalized to 1–10 the same way \(Appendix[D](https://arxiv.org/html/2610.41391#A4)\)\.

### 3\.3Input Representations and Implementation

We tested three input representations:*Abstract\-Only*\(title and abstract only\),*Vanilla Summarization*\(an LLM\-generated 200–300 word summary of the full paper\), and*Objective Summarization*\(the same length under Background, Method, Results and Limitations headings\)\. Appendix[A](https://arxiv.org/html/2610.41391#A1)gives the prompt templates\.

Unless otherwise noted, the 232\-paper experiments used GPT\-4o\-mini\([OpenAI, 2024](https://arxiv.org/html/2610.41391#bib.bib46)\)at temperature 0\.3 with 200–500 output tokens depending on the task, and text\-embedding\-3\-small for anchor similarity\. The cross\-model replication used the same temperature, the same Direct prompt, condensed versions of the Anchor, Group and binary\-comparison prompts \(the same tasks and JSON output, with the criteria on one line, the 1–10 score bands of Anchor and Group on one line without their descriptions, and a shorter reasoning instruction for Group\), and the same token budgets with one exception recorded in Appendix[D](https://arxiv.org/html/2610.41391#A4), servingmeta\-llama/Llama\-3\.3\-70B\-Instruct\-Turbo\([Grattafiori et al\., 2024](https://arxiv.org/html/2610.41391#bib.bib41)\)andopenai/gpt\-oss\-120b\([OpenAI, 2025](https://arxiv.org/html/2610.41391#bib.bib42)\)through the Together AI API\. The 232\-paper design comprised 8,258 GPT\-4o\-mini calls, 7,794 for evaluation across the fifteen paradigm\-by\-representation cells and 464 for summarization; population\-scale cost is reported in Appendix[S](https://arxiv.org/html/2610.41391#A19)\.

### 3\.4Evaluation Metrics

#### Correlation and regression\.

We report Pearson’srr, Spearman’sρ\\rho, and Kendall’sτ\\tau, together with Mean Squared Error \(MSE\)\. Grade inflation is the mean signed difference between predicted and ground\-truth scores, so positive values indicate systematic over\-prediction\. One asymmetry matters\. Direct, Anchor and Group Rating emit 1–10 scores directly, whereas Elo and Swiss produce latent ratings min–max mapped onto that scale \(Section[3\.2](https://arxiv.org/html/2610.41391#S3.SS2)\); fixing the endpoints at 1 and 10 pulls their predictions toward the middle of a scale whose ground\-truth mean is 5\.16, independently of judgment quality\. Inflation is therefore not like\-for\-like across families, and neither is any metric on absolute distances, including MSE\. Rank correlations and threshold\-free measures are unaffected, and Section[4\.5](https://arxiv.org/html/2610.41391#S4.SS5)separates placement from ordering\.

#### Classification\.

For binary accept/reject decisions \(Accept if the score is≥6\\geq 6, applied alike to ground\-truth and predicted scores\) we report accuracy, AUC\-ROC on the continuous predictions, Cohen’sκ\\kappa, and macro and weighted F1\. We refer toκ\\kappaand macro F1 jointly as*decision utility*: the quality of the accept/reject decisions a system induces, as distinct from the quality of the ordering it produces\. These are our primary utility metrics, since weighted F1 misleads under class imbalance: a system that accepts everything still attains a non\-trivial weighted F1 but zero*reject recall*, the fraction of papers human reviewers scored below the threshold that the system also rejects\.

#### Statistical tests\.

We use bootstrap confidence intervals \(1,000 resamples unless otherwise noted\) for correlations, paired bootstrap tests for paradigm comparisons, and Wilcoxon signed\-rank tests for absolute\-error differences\.

## 4RQ1: Paradigm Design

How should an LLM evaluate papers, and do abstracts alone carry sufficient signal? We take these in turn: input representation, then the five paradigms, their calibration and reliability, the accept/reject decisions they induce, a population\-scale replication, and a recalibration control that asks which of these effects is about ordering and which about score placement\.

### 4\.1Input Representation and Paradigm Comparison

We first isolate the effect of input representation by averaging metrics across all five paradigms \(Appendix[G](https://arxiv.org/html/2610.41391#A7)\)\.

Abstract\-only evaluation outperforms both summarization methods on every correlation metric \(r=0\.333r=0\.333against 0\.193;ρ=0\.334\\rho=0\.334against 0\.180\), and the ordering holds within all five paradigms\. Because our summarization pipeline is lossy \(Appendix[C](https://arxiv.org/html/2610.41391#A3)\), this compares abstract\-only input against a summary rather than against full\-paper reading\.

We therefore compare the five paradigms under abstract\-only evaluation \(Table[1](https://arxiv.org/html/2610.41391#S4.T1)\)\.

Table 1:Review paradigm comparison under abstract\-only input \(GPT\-4o\-mini\)\. Column headings as in Table[4](https://arxiv.org/html/2610.41391#A7.T4)\. The Group row is a three\-run mean \(Appendix[D](https://arxiv.org/html/2610.41391#A4)\); all other rows are single runs\.Direct Rating and Elo achieve comparable correlation with human scores \(ρ=0\.422\\rho=0\.422vs\.0\.4100\.410; paired bootstrapΔ​ρ=\+0\.012\\Delta\\rho=\+0\.012, 95% CI\[−0\.127,\+0\.151\]\[\-0\.127,\+0\.151\]from 10,000 resamples\)\. Group Rating follows atρ=0\.388\\rho=0\.388; all three outperform Swiss \(p<0\.001p<0\.001\), whose correlation is not significant, so its ranking may not reliably reflect quality\. The families differ mainly in where they place scores: Direct and Anchor compress 97% and 94% of predictions into the 6–8 range against a ground\-truthσ\\sigmaof 2\.16, whereas the comparative paradigms spread wider, partly through the min–max mapping \(Appendices[G](https://arxiv.org/html/2610.41391#A7)and[H](https://arxiv.org/html/2610.41391#A8)give per\-paradigm figures and the full 15\-combination matrix\)\. Because almost all Direct scores fall above the accept threshold, that paradigm separates few of the papers reviewers scored below it even though its correlation is high; Appendix[J](https://arxiv.org/html/2610.41391#A10)shows that a top\-20 cut is not even well defined for it, since 87 of the 232 papers share its highest score\.

### 4\.2Grade Inflation and Judge Reliability

Grade inflation varies systematically by paradigm type: the comparative methods range from \+0\.04 to \+0\.67, the absolute ones from \+1\.98 to \+2\.78, and Wilcoxon tests put Elo’s absolute prediction errors below every other paradigm \(allp<0\.01p<0\.01; Appendix[D](https://arxiv.org/html/2610.41391#A4)\)\. A single Direct rating also carries substantial decoding noise: three runs on the full population give ICC\(2,1\)=0\.417=0\.417\([Shrout and Fleiss, 1979](https://arxiv.org/html/2610.41391#bib.bib14)\), the “poor” band of[Koo and Li \(2016\)](https://arxiv.org/html/2610.41391#bib.bib15), so single\-shot Direct correlations are a lower bound \(Appendix[E](https://arxiv.org/html/2610.41391#A5)\)\. Comparative paradigms cannot be scored with ICC\(2,1\); for Elo we compare ratings at two comparison densities instead \(Appendix[K](https://arxiv.org/html/2610.41391#A11)\), and we have no such diagnostic for Swiss\.

### 4\.3Classification Performance

Appendix[I](https://arxiv.org/html/2610.41391#A9)reports the binary classification metrics, which reveal a divergence between methods with similar correlation\.

This subsection and the next apply a single fixed threshold to every paradigm, a choice Section[4\.5](https://arxiv.org/html/2610.41391#S4.SS5)revisits\. At that threshold, correlation alone misleads\. Direct Rating reaches an AUC\-ROC of 0\.682, indistinguishable from Elo’s 0\.672, so its scores carry discriminative signal; yet inflation shifts the boundary until accuracy is near chance \(42\.2% against 60\.3% for always rejecting\) and agreement negligible \(κ=0\.034\\kappa=0\.034\), while Elo reaches 64\.2% accuracy,κ=0\.267\\kappa=0\.267, and 66\.4% reject recall against Direct’s 4\.3%\. Table[6](https://arxiv.org/html/2610.41391#A9.T6)is computed on the stratified 232\-paper sample, whose scores are spread evenly across the quality range; Section[4\.4](https://arxiv.org/html/2610.41391#S4.SS4)verifies the pattern on the full population, whose scores instead concentrate near the accept/reject boundary\.

### 4\.4Population\-Scale Replication

Re\-running the GPT\-4o\-mini abstract\-only pipeline for Direct, Elo and Swiss on all 11,513 scored ICLR 2025 submissions reproduces the pattern and sharpens it: Direct flags0\.5%0\.5\\%of below\-threshold papers against4\.3%4\.3\\%on the stratified sample, Elo’sκ\\kappafalls from0\.2670\.267to0\.0760\.076, and no configuration reaches the59\.5%59\.5\\%accuracy of always rejecting\. The fall inκ\\kappais not a prevalence effect, since the accept rate is nearly identical in the two designs; what changes is the spread of ground\-truth scores\. We therefore recommend reporting absoluteκ\\kappaon a*deployment\-representative*sample, whose score distribution matches the pool the system would actually screen\. Appendix[P](https://arxiv.org/html/2610.41391#A16)gives the full analysis, the paradigm\-specific error trade\-offs and Table[9](https://arxiv.org/html/2610.41391#A16.T9)\.

### 4\.5Ranking or Placement? A Recalibration Control

Applying one fixed threshold to every paradigm does not separate how well a paradigm*orders*papers from where its scores happen to sit\. We therefore repeat the population\-scale analysis with each paradigm thresholded at its own base\-rate\-matched cut, the value at which its predicted reject rate equals the true rate of59\.5%59\.5\\%\. The transformation is monotone, so it changes no ranking metric \(Table[9](https://arxiv.org/html/2610.41391#A16.T9)\)\. It does assume the true reject rate is known, which a deployed system would have to estimate; we use it here as a diagnostic that isolates ordering from placement, not as a deployable procedure\.

This qualifies the earlier reading substantially\. Direct rises fromκ=0\.004\\kappa=0\.004to0\.1120\.112, ahead of Elo \(0\.079\) and Swiss \(0\.078\); the gap is\+0\.032\+0\.032, 95% CI\[\+0\.006,\+0\.058\]\[\+0\.006,\+0\.058\], so unlike the ranking comparison it separates them\. Reject blindness is therefore a property of Direct Rating*at a fixed threshold*rather than a defect in its ordering, as its AUC\-ROC of 0\.682 and populationρ\\rhoof 0\.204 already indicated\. What Elo supplies is placement, part of which follows from the min–max mapping rather than the comparison mechanism\. Since Direct uses 3 API calls per paper against Elo’s 12\.5, recalibration is the cheaper route to the better decision quality, so we do not recommend Elo as a default\. The two families do make partly independent errors, but averaging their outputs does not reliably exploit this \(Appendix[L](https://arxiv.org/html/2610.41391#A12)\)\. A single human reviewer scored against the mean of the others reachesρ=0\.445\\rho=0\.445andκ=0\.341\\kappa=0\.341, so the best paradigm attains under half that ranking\. Note also that no configuration reaches the accuracy of the trivial always\-reject baseline \(59\.5%59\.5\\%\), and that even after recalibration only Direct Rating exceedsκ=0\.1\\kappa=0\.1: at this base rate none is usable as an autonomous gate, at any threshold\. Replacing our score\-based label with the recorded ICLR accept/reject decisions leaves the ordering unchanged, with Direct again ahead after recalibration and the always\-reject baseline again essentially unbeaten \(Appendix[Q](https://arxiv.org/html/2610.41391#A17)\)\.

## 5RQ2: Model Dependence

Table 2:Best paradigm per model by ranking potential \(ρ\\rho\) and by decision utility, measured here byκ\\kappa, abstract\-only, 232\-paper sample\. The best paradigm differs by model and by criterion\. Appendix[N](https://arxiv.org/html/2610.41391#A14)reports all fifteen configurations\.Figure 2:External validity on the 126 test papers carrying a retrievable citation count, a complete set of paradigm scores and a recorded decision\.a, Paradigm rank under three evaluation targets; the lines cross, so none is best on all three\. The decision target counts the 18 withdrawn submissions as non\-accepted, which leaves the ordering unchanged\.b, Spearmanρ\\rhowith citation count, as an open circle \(raw\) joined to a filled circle \(controlling for the mean reviewer score\)\. Only Group Rating retains its correlation after the control \(ρ=0\.304\\rho=0\.304,p=0\.0006p=0\.0006, surviving Bonferroni correction across the eighteen tests in Table[11](https://arxiv.org/html/2610.41391#A18.T11)\)\. Anchor’s partial value is negative and Swiss’s exceeds its raw value, so the two are given as positions on a common axis rather than as a decomposition\.To test whether the findings hold beyond GPT\-4o\-mini, we replicated all five paradigms \(abstract\-only\) using Llama 3\.3\-70B and GPT\-oss\-120B; Table[2](https://arxiv.org/html/2610.41391#S5.T2)gives the per\-model best and Appendix[N](https://arxiv.org/html/2610.41391#A14)the full matrix\.

These are fixed\-threshold figures, so theκ\\kappacolumn carries the caveat of Section[4\.5](https://arxiv.org/html/2610.41391#S4.SS5)\. Recomputing every cell at its own base\-rate\-matched cut preserves the conclusion \(Appendix[N](https://arxiv.org/html/2610.41391#A14)\)\. Four findings follow\. First, model choice matters at least as much as paradigm choice, as it does for LLM\-generated preference distributions, which depend more on the choice of model than on the wording of the prompt\([Huang et al\., 2026a](https://arxiv.org/html/2610.41391#bib.bib50)\)\. Averaged over paradigms, the spread inρ\\rhoacross the three models is 0\.293, slightly larger than the average spread across the five paradigms within a model \(0\.267\); the margin is thin, so the more robust statement is that no single paradigm performs well on all three models, whereas GPT\-4o\-mini is the only model on which any paradigm exceedsρ=0\.35\\rho=0\.35\. Second, Elo consistently reduces grade inflation relative to Direct for GPT\-4o\-mini \(\+0\.67\+0\.67vs\.\+2\.13\+2\.13\) and Llama 3\.3\-70B \(\+0\.19\+0\.19vs\.\+3\.42\+3\.42\), and the two comparative paradigms attain the lowest inflation for GPT\-4o\-mini \(Swiss\+0\.04\+0\.04, Elo\+0\.67\+0\.67\) and for Llama 3\.3\-70B \(Elo\+0\.19\+0\.19, Swiss\+0\.33\+0\.33\)\. This does not extend to GPT\-oss\-120B, where Elo inflation slightly exceeds Direct \(\+1\.11\+1\.11vs\.\+0\.99\+0\.99\) and Swiss shows the largest inflation of any configuration \(\+3\.82\+3\.82\); inflation reduction therefore appears to be a tendency of comparative scoring rather than a guaranteed property of it\. Third, decision utility varies by model, fromκ=0\.267\\kappa=0\.267for GPT\-4o\-mini Elo down toκ=0\.060\\kappa=0\.060for Llama 3\.3\-70B Swiss; weighted F1 can mislead here, since Llama 3\.3\-70B Elo reports weighted F1=0\.518=0\.518despiteκ=−0\.013\\kappa=\-0\.013\. Fourth, the best paradigm is model\-dependent \(Table[2](https://arxiv.org/html/2610.41391#S5.T2)\), and the separation between ranking potential and decision utility recurs across all fifteen configurations: GPT\-4o\-mini Direct has the highestρ\\rhobut near\-zeroκ\\kappa, while Elo converts comparableρ\\rhointoκ=0\.267\\kappa=0\.267\. For GPT\-4o\-mini Elo, pairwise ordering accuracy, the fraction of paper pairs ordered in the same direction as ground truth, is 64\.5% \(vs\. 50% chance\) and exceeds 75% for paper pairs differing by at least 4 ground\-truth points\. Per\-model score distributions and aρ\\rho\-versus\-κ\\kappaview appear in Appendix[O](https://arxiv.org/html/2610.41391#A15)\.

## 6RQ3: External Validity

Do predicted scores track research impact? We correlate them with citation counts retrieved from Semantic Scholar\([Kinney et al\., 2023](https://arxiv.org/html/2610.41391#bib.bib43)\)in August 2026\. Of the 232 test papers 163 matched a record, and because the dated\-citation query is keyed on a DOI only the 135 carrying one were screened; 126 of those have at least one citation together with a complete set of paradigm scores and are the basis of every correlation and figure below\. A further 26 matched, cited and fully scored papers are excluded for lacking a DOI, so the sample is enriched for work that was both disseminated and formally indexed\. Because the source dates each citing paper, a 12\-month and a 17\-month window follow from one collection; they rank papers atρ=0\.992\\rho=0\.992with no correlation moving by more than 0\.010, so we report the 17\-month figures and give the comparison in Appendix[R](https://arxiv.org/html/2610.41391#A18)\.

Human review scores correlate with citations atρ=0\.568\\rho=0\.568\(p<0\.001p<0\.001\), well above every LLM paradigm; Group Rating is strongest at 0\.380, followed by Elo at 0\.308 and Direct at 0\.238\. Against reviewer scores on these same 126 papers Elo also leads Direct \(0\.353 against 0\.293\), reversing their order on the full 232\-paper sample \(Table[1](https://arxiv.org/html/2610.41391#S4.T1)\): the ranking between them is not stable across samples or targets\.

The stratified columns of Table[11](https://arxiv.org/html/2610.41391#A18.T11)qualify this, and the pattern is not specific to Elo\. Splitting by ICLR decision, the human correlation holds within both strata \(0\.430 for accepted papers, 0\.408 for rejected\), so reviewers order papers by eventual impact within a pool and not merely across the accept/reject boundary; for ICLR 2020,[Tran et al\. \(2020\)](https://arxiv.org/html/2610.41391#bib.bib45)found much weaker correlations between review score and citation rate within accepted posters and within rejected or withdrawn papers \(0\.17 and 0\.22\), and for 8,586 accepted ICLR 2017–2025 papers the mean review score is uncorrelated with a citation\-embedding measure of disruption \(ρ=−0\.003\\rho=\-0\.003;[Huang, 2026a](https://arxiv.org/html/2610.41391#bib.bib48)\)\. Four of the five paradigms fail to reach significance among accepted papers, spanning−0\.082\-0\.082to0\.1010\.101, while three are significant among rejected\. Group Rating is the exception, at 0\.330 and 0\.420\. For those four, the association with citations is carried by separating likely accepts from likely rejects rather than by ranking within the accepted pool\.

A correlation with citations need not show that a paradigm contributes anything, since it may re\-derive the human score\. We therefore compute each paradigm’s partial Spearman correlation with citations, controlling for the mean reviewer score\. Most does not survive \(Figure[2](https://arxiv.org/html/2610.41391#S5.F2)b\): Elo falls from 0\.308 to a non\-significant 0\.140 and Direct from 0\.238 to 0\.090\. Group Rating behaves differently\. It agrees with human scores at onlyρ=0\.242\\rho=0\.242yet correlates 0\.380 with citations and retainsρ=0\.304\\rho=0\.304after the control \(bootstrap 95% CI\[\+0\.129,\+0\.460\]\[\+0\.129,\+0\.460\],p=0\.0006p=0\.0006, surviving Bonferroni correction across the eighteen tests in Table[11](https://arxiv.org/html/2610.41391#A18.T11)\)\. Swiss shows a weaker effect in the same direction that does not survive correction\. Scoring five papers together therefore appears to surface impact\-relevant signal that a single reviewer score does not capture\. Givenn=126n=126and a single venue, we offer this as a direction worth testing rather than an established effect\.

Which paradigm looks best also depends on the target: ranking the five under reviewer scores, citations and the recorded ICLR decision gives three different orderings \(Figure[2](https://arxiv.org/html/2610.41391#S5.F2)a\), and none wins on all three\. Paradigm conclusions are contingent on the evaluation target, and a single target does not settle them; in LLM reasoning evaluation, similarly, replacing string matching with an LLM judge cuts one method’s measured lead over another on HotpotQA from 13\.0 to 4\.7 points\([Huang, 2026b](https://arxiv.org/html/2610.41391#bib.bib53)\)\.

## 7Discussion

The separation betweenρ\\rhoandκ\\kappahas a narrower reading than it first invites\. Rank correlation measures ordering; a decision also requires that scores sit in the right place relative to the threshold\. Direct Rating orders papers competitively and places them badly, the comparative paradigms place them better, but we cannot credit the comparison mechanism itself: our comparative ratings are min–max mapped onto the 1–10 scale while the absolute ones are not, and one order\-preserving recalibration gives Direct Rating the better decision quality at lower cost \(Section[4\.5](https://arxiv.org/html/2610.41391#S4.SS5)\)\. The defensible conclusion is about measurement: applying one fixed threshold to methods with different score placements ranks them largely by placement, so a study reporting onlyρ\\rho, or onlyκ\\kappaat an arbitrary cut, can be misread in either direction\.

The other two questions point the same way\. A paradigm cannot be recommended independently of the model running it or the target it is scored against, so single\-model single\-target results are weaker evidence than they appear\. Four of the five paradigms also lose their association with citations among accepted papers while human review retains it, separating likely accepts from likely rejects without ranking within the accepted pool\. Group Rating is the exception, the one paradigm here behaving as a ranker rather than a filter\.

The advantage of abstract\-only input is consistent with abstracts acting as concentrated, author\-curated signals, though our summaries are lossy enough that we cannot separate this from a pipeline effect; it is also a surface that automated rewriting targets, since rewrites that raised AI reviewer scores recast abstracts in more confident language\([Baumann et al\., 2026](https://arxiv.org/html/2610.41391#bib.bib26)\)\. For deployment the population\-scale results bound what is available \(Table[9](https://arxiv.org/html/2610.41391#A16.T9)\): no configuration exceeds the accuracy of always rejecting, and only Direct Rating exceedsκ=0\.1\\kappa=0\.1even after recalibration, so Appendix[S](https://arxiv.org/html/2610.41391#A19)reports operating points for comparison rather than as a recommendation\.

## 8Conclusion and Future Work

We compared five LLM\-based review paradigms across three models, three input representations, and a citation outcome\. On RQ1, Elo and Direct Rating attain comparable ranking potential \(ρ=0\.410\\rho=0\.410vs\.0\.4220\.422\) but very different decision utility at a fixed threshold, where Direct Rating flags 4\.3% of below\-threshold papers against Elo’s 66\.4%; a recalibration control locates the cause in placement rather than ordering, and at its own base rate Direct Rating moves ahead \(κ=0\.112\\kappa=0\.112vs\.0\.0790\.079\) while making 24% as many API calls as Elo\. Abstract\-only input wins throughout, and no configuration beats rejecting every paper at the deployment base rate\. On RQ2, model choice matters at least as much as paradigm choice\. On RQ3, human review scores track citations atρ=0\.568\\rho=0\.568, above every paradigm, and hold within both decision strata\. Group Rating is the one paradigm that also holds within both, and it retainsρ=0\.304\\rho=0\.304with citations after controlling for the human score\.

The replication also isolates a reporting issue: Cohen’sκ\\kappais sensitive to the spread of ground\-truth scores, not only to class prevalence, the same Elo configuration reaching0\.2670\.267on a stratified sample and0\.0760\.076on the population\. Absolute agreement should therefore be reported on a deployment\-representative sample at a calibrated threshold\. This is a separate mechanism from the marginal\-dependence paradoxes of[Feinstein and Cicchetti \(1990\)](https://arxiv.org/html/2610.41391#bib.bib40): the marginals barely move between the two designs, and what changes is the spread of the underlying scores\.

The practical upshot is narrow but firm: an LLM reviewer is at best an aid to triaging a submission pool rather than a judge of an individual paper, and any claim about the best paradigm is a claim about a particular model, threshold and target\.

Three directions follow: whenkk\-run averaging of a cheap absolute paradigm beats switching to a comparative one; where the comparison\-density elbow lies; and cost\-sensitive threshold selection rather than raisingκ\\kappadirectly\.

## Limitations

#### The paradigm comparison is not fully controlled\.

Input representation, model, prompt framing and decoding temperature are held constant across paradigms, but three confounds remain\. First, the two families are not scaled alike: absolute paradigms emit 1–10 scores directly, whereas Elo and Swiss ratings are min–max mapped onto that scale, which constrains their means toward the middle independently of judgment quality\. Reported inflation is therefore not a like\-for\-like measurement across families, and the recalibration control in Section[4\.5](https://arxiv.org/html/2610.41391#S4.SS5)should be read as the primary comparison\. Second, the paradigms are not compute\-matched at population scale: Direct uses 3 calls per paper against Elo’s 12\.5\. On the 232\-paper set we ran the control \(Appendix[E](https://arxiv.org/html/2610.41391#A5)\)\. Giving Direct Elo’s budget of five calls and averaging raisesρ\\rhofrom 0\.396 to 0\.482, above Elo’s 0\.410, and leaves base\-rate\-matchedκ\\kappaat 0\.296 against Elo’s 0\.225 at its own base\-rate\-matched cut, so at equal compute Direct leads on both\. The population\-scale equivalent, 12\.5 runs, remains unrun\. Third, we estimate Elo ratings with the standard sequential update rather than a maximum\-likelihood Bradley–Terry fit on the same comparisons; the latter is order\-independent and would be the stronger estimator\([Liu et al\., 2025](https://arxiv.org/html/2610.41391#bib.bib38);[Epstein et al\., 2026](https://arxiv.org/html/2610.41391#bib.bib31)\)\.

#### Ceiling and ground truth\.

No single\-run paradigm exceeded a Spearman correlation of 0\.43, and at population scale the best is 0\.204, against attenuation\-corrected ceilings set by reviewer disagreement,ρ=0\.951\\rho=0\.951on the stratified sample andρ=0\.815\\rho=0\.815at population scale \(Section[3\.1](https://arxiv.org/html/2610.41391#S3.SS1)\), bounds that constrain any predictor of this target, a second human reviewer included\. Read against the ceiling of its own sample, the best stratified single\-run result reaches44\.4%44\.4\\%of what is attainable and the population result25\.1%25\.1\\%; the gap reflects the noise in the target as well as the limits of the paradigms\. Appendix[M](https://arxiv.org/html/2610.41391#A13)varies the aggregation of reviewer scores, and Appendix[Q](https://arxiv.org/html/2610.41391#A17)substitutes the recorded ICLR decisions, but neither addresses the noise in the underlying reviewer signal itself\.

#### Statistical scope\.

Swiss’s correlation is not significant at this sample size, so its ranking should be read cautiously\. We report fifteen configurations without correction for multiple comparisons\. The citation analysis covers the 126 of 232 papers that matched a Semantic Scholar record, carried a DOI through which dated citations could be retrieved, had at least one citation and carried a complete set of paradigm scores; requiring a DOI excludes a further 26 papers that would otherwise qualify, a subset enriched for disseminated work; its stratified columns rest on 74 accepted and 34 rejected papers, so the within\-stratum estimates are the least precise numbers we report\.

#### Generalization\.

Results come from one venue in one year within machine learning; venues with different review cultures may behave differently\. The population\-scale run draws from the same ICLR 2025 pool as the 232\-paper sample, so it is a superset replication rather than external validation, and it covers only GPT\-4o\-mini\. The two open\-weight models we tested were weaker overall, with only GPT\-oss\-120B Group \(ρ=0\.338\\rho=0\.338\) approaching the GPT\-4o\-mini range; our design cannot separate a proprietary/open\-weight distinction from differences in scale and post\-training\. The open\-weight runs also used condensed prompts for four of the five paradigms \(Section[3](https://arxiv.org/html/2610.41391#S3)\), so for those paradigms prompt wording is confounded with model\.

#### Fairness audit\.

Our Ethical Considerations section calls for auditing across research areas and author groups, and we did not run such an audit\. This matters here because the best\-performing configuration reads only the abstract, a surface that automated rewriting targets, and because[Baumann et al\. \(2026\)](https://arxiv.org/html/2610.41391#bib.bib26)report that AI reviewer scores can be moved by stylistic revision\. A per\-area breakdown of inflation and agreement at population scale is the natural next step and is feasible with the data we already hold\.

## Ethical Considerations

This study uses publicly available ICLR submission metadata and reviewer scores from OpenReview and citation counts from Semantic Scholar; no private or personally identifying data beyond publicly listed information was used\. Our findings characterize the capabilities and failure modes of LLM\-based review and are intended to inform research on review assistance, not to replace human judgment\. We caution against deploying any of these paradigms as an autonomous accept/reject gate\. At the deployment base rate none of them exceeds the accuracy of rejecting every paper under our score\-based rule, and only Direct Rating does so under the recorded ICLR decisions, by 0\.3 points; the best attainsκ=0\.112\\kappa=0\.112after recalibration and the rest fall below 0\.1, and all misclassify a substantial fraction of borderline papers; used without human oversight they could also systematically disadvantage particular topics, methods, or author groups, a risk we did not audit\. LLM\-based review may also reflect biases present in model training data\([Yin and Huang, 2025](https://arxiv.org/html/2610.41391#bib.bib54)\)\. We recommend that such systems be used only as a complement to human review, with audits for fairness across research areas and author demographics before any operational use\.

## Acknowledgments

We thank the maintainers of the open datasets and platforms this work depends on, in particular OpenReview and Semantic Scholar, and the developers of the open\-weight models we evaluated, Llama 3\.3\-70B and GPT\-oss\-120B\.

## Declaration on AI Assistance

This declaration concerns AI assistance to the authors; the language models evaluated in this paper \(Section[3](https://arxiv.org/html/2610.41391#S3)\) are the object of study, not writing aids\.

AI\-based tools were used in the preparation of this manuscript: for language and copy\-editing, including drafting and revision of prose under author direction; for verification, checking bibliography entries against authoritative catalog records and recomputing reported statistics from the stored experimental artifacts, which surfaced a small number of numerical and citation errors that we corrected; and, as a pre\-submission rehearsal, for generating model\-written critiques of our own draft, several of which prompted additional controls that we implemented, ran on our own data, and verified before adopting\.

No reported result was generated by an AI system: the experimental design, implementation and analysis are the authors’ own, every number reported here is produced by our own pipeline, and the authors reviewed and take full responsibility for all content\.

## References

- Baumannet al\.\(2026\)J\. Baumann, J\. Pei, S\. Koyejo, and D\. HovyPosition: stop automating peer review without rigorous evaluation\.InProceedings of the 43rd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.306,pp\. 169349–169380\.Cited by:[§1](https://arxiv.org/html/2610.41391#S1.p3.1),[§2\.1](https://arxiv.org/html/2610.41391#S2.SS1.p1.1),[§7](https://arxiv.org/html/2610.41391#S7.p3.1),[Fairness audit\.](https://arxiv.org/html/2610.41391#Sx1.SS0.SSS0.Px5.p1.1)\.
- Biswaset al\.\(2026\)J\. Biswas, S\. Schoepp, G\. Vasan, A\. Opipari, A\. Zhang, Z\. Hu, S\. Joseph, M\. Lease, J\. J\. Li, P\. Stone, K\. L\. Wagstaff, M\. E\. Taylor, and O\. C\. JenkinsAI\-assisted peer review at scale: the AAAI\-26 AI review pilot\.arXiv preprint arXiv:2604\.13940\.Cited by:[§1](https://arxiv.org/html/2610.41391#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.41391#S2.SS1.p1.1)\.
- Boubdiret al\.\(2024\)M\. Boubdir, E\. Kim, B\. Ermis, S\. Hooker, and M\. FadaeeElo uncovered: robustness and best practices in language model evaluation\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 106135–106161\.Cited by:[§1](https://arxiv.org/html/2610.41391#S1.p3.1),[§2\.3](https://arxiv.org/html/2610.41391#S2.SS3.p1.1)\.
- Checcoet al\.\(2021\)A\. Checco, L\. Bracciale, P\. Loreti, S\. Pinfield, and G\. BianchiAI\-assisted peer review\.Humanities and Social Sciences Communications8\(1\),pp\. 25\.Cited by:[§1](https://arxiv.org/html/2610.41391#S1.p1.1)\.
- D’Arcyet al\.\(2024\)M\. D’Arcy, T\. Hope, L\. Birnbaum, and D\. DowneyMARG: multi\-agent review generation for scientific papers\.arXiv preprint arXiv:2401\.04259\.Cited by:[§1](https://arxiv.org/html/2610.41391#S1.p3.1),[§2\.1](https://arxiv.org/html/2610.41391#S2.SS1.p1.1)\.
- Elo \(1978\)A\. E\. EloThe rating of chessplayers, past and present\.Arco Publishing,New York\.Cited by:[§1](https://arxiv.org/html/2610.41391#S1.p3.1),[§2\.3](https://arxiv.org/html/2610.41391#S2.SS3.p1.1),[§3\.2](https://arxiv.org/html/2610.41391#S3.SS2.p5.1)\.
- Epsteinet al\.\(2026\)E\. L\. Epstein, R\. Dwaraknath, J\. Winnicki, and T\. SornwaneeAllocate marginal reviews to borderline papers using LLM comparative ranking\.arXiv preprint arXiv:2602\.06078\.Cited by:[§2\.3](https://arxiv.org/html/2610.41391#S2.SS3.p1.1),[The paradigm comparison is not fully controlled\.](https://arxiv.org/html/2610.41391#Sx1.SS0.SSS0.Px1.p1.1)\.
- Feinstein and Cicchetti \(1990\)A\. R\. Feinstein and D\. V\. CicchettiHigh agreement but low kappa: i\. the problems of two paradoxes\.Journal of Clinical Epidemiology43\(6\),pp\. 543–549\.Cited by:[§8](https://arxiv.org/html/2610.41391#S8.p2.1)\.
- Fujinuma \(2026\)Y\. FujinumaContrastive decoding mitigates score range bias in LLM\-as\-a\-judge\.InFindings of the Association for Computational Linguistics: ACL 2026,San Diego, California, United States,pp\. 13404–13418\.Cited by:[§1](https://arxiv.org/html/2610.41391#S1.p2.1),[§2\.2](https://arxiv.org/html/2610.41391#S2.SS2.p1.1)\.
- Gaoet al\.\(2024\)Z\. Gao, K\. Brantley, and T\. JoachimsReviewer2: optimizing review generation through prompt generation\.arXiv preprint arXiv:2402\.10886\.Cited by:[§1](https://arxiv.org/html/2610.41391#S1.p3.1),[§2\.1](https://arxiv.org/html/2610.41391#S2.SS1.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan, A\. Yang, A\. Fan, A\. Goyal, A\. Hartshorn, A\. Yang, A\. Mitra, A\. Sravankumar, A\. Korenev, A\. Hinsvark, A\. Rao, A\. Zhang, A\. Rodriguez, A\. Gregerson, A\. Spataru, B\. Roziere, B\. Biron, B\. Tang, B\. Chern, C\. Caucheteux, C\. Nayak, C\. Bi, C\. Marra, C\. McConnell, C\. Keller, C\. Touret, C\. Wu, C\. Wong, C\. C\. Ferrer, C\. Nikolaidis, D\. Allonsius, D\. Song, D\. Pintz, D\. Livshits, D\. Wyatt, D\. Esiobu, D\. Choudhary, D\. Mahajan, D\. Garcia\-Olano, D\. Perino, D\. Hupkes, E\. Lakomkin, E\. AlBadawy, E\. Lobanova, E\. Dinan, E\. M\. Smith, F\. Radenovic, F\. Guzmán, F\. Zhang, G\. Synnaeve, G\. Lee, G\. L\. Anderson, G\. Thattai, G\. Nail, G\. Mialon, G\. Pang, G\. Cucurell, H\. Nguyen, H\. Korevaar, H\. Xu, H\. Touvron, I\. Zarov, I\. A\. Ibarra, I\. Kloumann, I\. Misra, I\. Evtimov, J\. Zhang, J\. Copet, J\. Lee, J\. Geffert, J\. Vranes, J\. Park, J\. Mahadeokar, J\. Shah, J\. van der Linde, J\. Billock, J\. Hong, J\. Lee, J\. Fu, J\. Chi, J\. Huang, J\. Liu, J\. Wang, J\. Yu, J\. Bitton, J\. Spisak, J\. Park, J\. Rocca, J\. Johnstun, J\. Saxe, J\. Jia, K\. V\. Alwala, K\. Prasad, K\. Upasani, K\. Plawiak, K\. Li, K\. Heafield, K\. Stone, K\. El\-Arini, K\. Iyer, K\. Malik, K\. Chiu, K\. Bhalla, K\. Lakhotia, L\. Rantala\-Yeary, L\. van der Maaten, L\. Chen, L\. Tan, L\. Jenkins, L\. Martin, L\. Madaan, L\. Malo, L\. Blecher, L\. Landzaat, L\. de Oliveira, M\. Muzzi, M\. Pasupuleti, M\. Singh, M\. Paluri, M\. Kardas, M\. Tsimpoukelli, M\. Oldham, M\. Rita, M\. Pavlova, M\. Kambadur, M\. Lewis, M\. Si, M\. K\. Singh, M\. Hassan, N\. Goyal, N\. Torabi, N\. Bashlykov, N\. Bogoychev, N\. Chatterji, N\. Zhang, O\. Duchenne, O\. Çelebi, P\. Alrassy, P\. Zhang, P\. Li, P\. Vasic, P\. Weng, P\. Bhargava, P\. Dubal, P\. Krishnan, P\. S\. Koura, P\. Xu, Q\. He, Q\. Dong, R\. Srinivasan, R\. Ganapathy, R\. Calderer, R\. S\. Cabral, R\. Stojnic, R\. Raileanu, R\. Maheswari, R\. Girdhar, R\. Patel, R\. Sauvestre, R\. Polidoro, R\. Sumbaly, R\. Taylor, R\. Silva, R\. Hou, R\. Wang, S\. Hosseini, S\. Chennabasappa, S\. Singh, S\. Bell, S\. S\. Kim, S\. Edunov, S\. Nie, S\. Narang, S\. Raparthy, S\. Shen, S\. Wan, S\. Bhosale, S\. Zhang, S\. Vandenhende, S\. Batra, S\. Whitman, S\. Sootla, S\. Collot, S\. Gururangan, S\. Borodinsky, T\. Herman, T\. Fowler, T\. Sheasha, T\. Georgiou, T\. Scialom, T\. Speckbacher, T\. Mihaylov, T\. Xiao, U\. Karn, V\. Goswami, V\. Gupta, V\. Ramanathan, V\. Kerkez, V\. Gonguet, V\. Do, V\. Vogeti, V\. Albiero, V\. Petrovic, W\. Chu, W\. Xiong, W\. Fu, W\. Meers, X\. Martinet, X\. Wang, X\. Wang, X\. E\. Tan, X\. Xia, X\. Xie, X\. Jia, X\. Wang, Y\. Goldschlag, Y\. Gaur, Y\. Babaei, Y\. Wen, Y\. Song, Y\. Zhang, Y\. Li, Y\. Mao, Z\. D\. Coudert, Z\. Yan, Z\. Chen, Z\. Papakipos, A\. Singh, A\. Srivastava, A\. Jain, A\. Kelsey, A\. Shajnfeld, A\. Gangidi, A\. Victoria, A\. Goldstand, A\. Menon, A\. Sharma, A\. Boesenberg, A\. Baevski, A\. Feinstein, A\. Kallet, A\. Sangani, A\. Teo, A\. Yunus, A\. Lupu, A\. Alvarado, A\. Caples, A\. Gu, A\. Ho, A\. Poulton, A\. Ryan, A\. Ramchandani, A\. Dong, A\. Franco, A\. Goyal, A\. Saraf, A\. Chowdhury, A\. Gabriel, A\. Bharambe, A\. Eisenman, A\. Yazdan, B\. James, B\. Maurer, B\. Leonhardi, B\. Huang, B\. Loyd, B\. D\. Paola, B\. Paranjape, B\. Liu, B\. Wu, B\. Ni, B\. Hancock, B\. Wasti, B\. Spence, B\. Stojkovic, B\. Gamido, B\. Montalvo, C\. Parker, C\. Burton, C\. Mejia, C\. Liu, C\. Wang, C\. Kim, C\. Zhou, C\. Hu, C\. Chu, C\. Cai, C\. Tindal, C\. Feichtenhofer, C\. Gao, D\. Civin, D\. Beaty, D\. Kreymer, D\. Li, D\. Adkins, D\. Xu, D\. Testuggine, D\. David, D\. Parikh, D\. Liskovich, D\. Foss, D\. Wang, D\. Le, D\. Holland, E\. Dowling, E\. Jamil, E\. Montgomery, E\. Presani, E\. Hahn, E\. Wood, E\. Le, E\. Brinkman, E\. Arcaute, E\. Dunbar, E\. Smothers, F\. Sun, F\. Kreuk, F\. Tian, F\. Kokkinos, F\. Ozgenel, F\. Caggioni, F\. Kanayet, F\. Seide, G\. M\. Florez, G\. Schwarz, G\. Badeer, G\. Swee, G\. Halpern, G\. Herman, G\. Sizov, G\. Zhang, G\. Lakshminarayanan, H\. Inan, H\. Shojanazeri, H\. Zou, H\. Wang, H\. Zha, H\. Habeeb, H\. Rudolph, H\. Suk, H\. Aspegren, H\. Goldman, H\. Zhan, I\. Damlaj, I\. Molybog, I\. Tufanov, I\. Leontiadis, I\. Veliche, I\. Gat, J\. Weissman, J\. Geboski, J\. Kohli, J\. Lam, J\. Asher, J\. Gaya, J\. Marcus, J\. Tang, J\. Chan, J\. Zhen, J\. Reizenstein, J\. Teboul, J\. Zhong, J\. Jin, J\. Yang, J\. Cummings, J\. Carvill, J\. Shepard, J\. McPhie, J\. Torres, J\. Ginsburg, J\. Wang, K\. Wu, K\. H\. U, K\. Saxena, K\. Khandelwal, K\. Zand, K\. Matosich, K\. Veeraraghavan, K\. Michelena, K\. Li, K\. Jagadeesh, K\. Huang, K\. Chawla, K\. Huang, L\. Chen, L\. Garg, L\. A, L\. Silva, L\. Bell, L\. Zhang, L\. Guo, L\. Yu, L\. Moshkovich, L\. Wehrstedt, M\. Khabsa, M\. Avalani, M\. Bhatt, M\. Mankus, M\. Hasson, M\. Lennie, M\. Reso, M\. Groshev, M\. Naumov, M\. Lathi, M\. Keneally, M\. Liu, M\. L\. Seltzer, M\. Valko, M\. Restrepo, M\. Patel, M\. Vyatskov, M\. Samvelyan, M\. Clark, M\. Macey, M\. Wang, M\. J\. Hermoso, M\. Metanat, M\. Rastegari, M\. Bansal, N\. Santhanam, N\. Parks, N\. White, N\. Bawa, N\. Singhal, N\. Egebo, N\. Usunier, N\. Mehta, N\. P\. Laptev, N\. Dong, N\. Cheng, O\. Chernoguz, O\. Hart, O\. Salpekar, O\. Kalinli, P\. Kent, P\. Parekh, P\. Saab, P\. Balaji, P\. Rittner, P\. Bontrager, P\. Roux, P\. Dollar, P\. Zvyagina, P\. Ratanchandani, P\. Yuvraj, Q\. Liang, R\. Alao, R\. Rodriguez, R\. Ayub, R\. Murthy, R\. Nayani, R\. Mitra, R\. Parthasarathy, R\. Li, R\. Hogan, R\. Battey, R\. Wang, R\. Howes, R\. Rinott, S\. Mehta, S\. Siby, S\. J\. Bondu, S\. Datta, S\. Chugh, S\. Hunt, S\. Dhillon, S\. Sidorov, S\. Pan, S\. Mahajan, S\. Verma, S\. Yamamoto, S\. Ramaswamy, S\. Lindsay, S\. Lindsay, S\. Feng, S\. Lin, S\. C\. Zha, S\. Patil, S\. Shankar, S\. Zhang, S\. Zhang, S\. Wang, S\. Agarwal, S\. Sajuyigbe, S\. Chintala, S\. Max, S\. Chen, S\. Kehoe, S\. Satterfield, S\. Govindaprasad, S\. Gupta, S\. Deng, S\. Cho, S\. Virk, S\. Subramanian, S\. Choudhury, S\. Goldman, T\. Remez, T\. Glaser, T\. Best, T\. Koehler, T\. Robinson, T\. Li, T\. Zhang, T\. Matthews, T\. Chou, T\. Shaked, V\. Vontimitta, V\. Ajayi, V\. Montanez, V\. Mohan, V\. S\. Kumar, V\. Mangla, V\. Ionescu, V\. Poenaru, V\. T\. Mihailescu, V\. Ivanov, W\. Li, W\. Wang, W\. Jiang, W\. Bouaziz, W\. Constable, X\. Tang, X\. Wu, X\. Wang, X\. Wu, X\. Gao, Y\. Kleinman, Y\. Chen, Y\. Hu, Y\. Jia, Y\. Qi, Y\. Li, Y\. Zhang, Y\. Zhang, Y\. Adi, Y\. Nam, Y\. Wang, Y\. Zhao, Y\. Hao, Y\. Qian, Y\. Li, Y\. He, Z\. Rait, Z\. DeVito, Z\. Rosnbrick, Z\. Wen, Z\. Yang, Z\. Zhao, and Z\. MaThe Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§3\.3](https://arxiv.org/html/2610.41391#S3.SS3.p2.1)\.
- Haldar and Hockenmaier \(2025\)R\. Haldar and J\. HockenmaierRating roulette: self\-inconsistency in LLM\-as\-a\-judge frameworks\.InFindings of the Association for Computational Linguistics: EMNLP 2025,Suzhou, China,pp\. 24986–25004\.Cited by:[§2\.2](https://arxiv.org/html/2610.41391#S2.SS2.p1.1)\.
- Huanget al\.\(2026a\)F\. Huang, M\. Kim, C\. T\. Diggans, and F\. RadicchiEvaluating LLM\-generated preference distributions\.arXiv preprint arXiv:2610\.01000\.Cited by:[§5](https://arxiv.org/html/2610.41391#S5.p2.1)\.
- Huang and Kim \(2026\)F\. Huang and M\. KimCan LLMs predict academic collaboration? Topology heuristics vs\. LLM\-based link prediction on real co\-authorship networks\.arXiv preprint arXiv:2604\.01379\.Cited by:[Appendix K](https://arxiv.org/html/2610.41391#A11.SS0.SSS0.Px2.p1.1)\.
- Huanget al\.\(2024\)F\. Huang, H\. Kwak, K\. Park, and J\. AnChatGPT rates natural language explanation quality like humans: but on which scales?\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),Torino, Italia,pp\. 3111–3132\.Cited by:[§2\.2](https://arxiv.org/html/2610.41391#S2.SS2.p1.1),[§3\.2](https://arxiv.org/html/2610.41391#S3.SS2.p3.1)\.
- Huanget al\.\(2026b\)F\. Huang, S\. Zhang, H\. Kwak, and J\. AnCogBias: measuring and mitigating cognitive bias in large language models\.arXiv preprint arXiv:2604\.01366\.Cited by:[Appendix D](https://arxiv.org/html/2610.41391#A4.SS0.SSS0.Px1.p2.1),[§2\.2](https://arxiv.org/html/2610.41391#S2.SS2.p1.1)\.
- Huang \(2026a\)F\. HuangCatalyst papers in artificial intelligence research: a landscape on ICLR from 2017 to 2025\.arXiv preprint arXiv:2607\.05401\.Cited by:[§2\.3](https://arxiv.org/html/2610.41391#S2.SS3.p1.1),[§6](https://arxiv.org/html/2610.41391#S6.p3.1)\.
- Huang \(2026b\)F\. HuangReasoning topology matters: Network\-of\-Thought for complex reasoning tasks\.arXiv preprint arXiv:2603\.20730\.Cited by:[§6](https://arxiv.org/html/2610.41391#S6.p5.1)\.
- Huang \(2026c\)F\. HuangReFlect: an effective harness system for complex long\-horizon LLM reasoning\.arXiv preprint arXiv:2605\.05737\.Cited by:[§1](https://arxiv.org/html/2610.41391#S1.p6.1)\.
- Kanget al\.\(2018\)D\. Kang, W\. Ammar, B\. Dalvi, M\. van Zuylen, S\. Kohlmeier, E\. Hovy, and R\. SchwartzA dataset of peer reviews \(PeerRead\): collection, insights and NLP applications\.InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long Papers\),New Orleans, Louisiana,pp\. 1647–1661\.Cited by:[§1](https://arxiv.org/html/2610.41391#S1.p3.1),[§2\.1](https://arxiv.org/html/2610.41391#S2.SS1.p1.1)\.
- Kinneyet al\.\(2023\)R\. Kinney, C\. Anastasiades, R\. Authur, I\. Beltagy, J\. Bragg, A\. Buraczynski, I\. Cachola, S\. Candra, Y\. Chandrasekhar, A\. Cohan, M\. Crawford, D\. Downey, J\. Dunkelberger, O\. Etzioni, R\. Evans, S\. Feldman, J\. Gorney, D\. Graham, F\. Hu, R\. Huff, D\. King, S\. Kohlmeier, B\. Kuehl, M\. Langan, D\. Lin, H\. Liu, K\. Lo, J\. Lochner, K\. MacMillan, T\. Murray, C\. Newell, S\. Rao, S\. Rohatgi, P\. Sayre, Z\. Shen, A\. Singh, L\. Soldaini, S\. Subramanian, A\. Tanaka, A\. D\. Wade, L\. Wagner, L\. L\. Wang, C\. Wilhelm, C\. Wu, J\. Yang, A\. Zamarron, M\. Van Zuylen, and D\. S\. WeldThe Semantic Scholar open data platform\.arXiv preprint arXiv:2301\.10140\.Cited by:[§6](https://arxiv.org/html/2610.41391#S6.p1.1)\.
- Koo and Li \(2016\)T\. K\. Koo and M\. Y\. LiA guideline of selecting and reporting intraclass correlation coefficients for reliability research\.Journal of Chiropractic Medicine15\(2\),pp\. 155–163\.Cited by:[Table 3](https://arxiv.org/html/2610.41391#A5.T3),[§4\.2](https://arxiv.org/html/2610.41391#S4.SS2.p1.1)\.
- Liet al\.\(2026a\)L\. Li, J\. Xiong, C\. Zhu, R\. Yu, C\. Chen, J\. Wang, R\. Ma, and Z\. LuLLM\-as\-a\-reviewer: benchmarking their ability, divergence, and prompt injection resistance as paper reviewers\.arXiv preprint arXiv:2605\.25415\.Cited by:[§1](https://arxiv.org/html/2610.41391#S1.p2.1),[§1](https://arxiv.org/html/2610.41391#S1.p4.1),[§2\.1](https://arxiv.org/html/2610.41391#S2.SS1.p1.1)\.
- Liet al\.\(2026b\)Q\. Li, S\. Dou, K\. Shao, C\. Chen, and H\. HuEvaluating scoring bias in LLM\-as\-a\-judge\.InDatabase Systems for Advanced Applications \(DASFAA 2026\),Lecture Notes in Computer Science,pp\. 19–34\.Cited by:[§1](https://arxiv.org/html/2610.41391#S1.p2.1),[§2\.2](https://arxiv.org/html/2610.41391#S2.SS2.p1.1)\.
- Lianget al\.\(2024\)W\. Liang, Y\. Zhang, H\. Cao, B\. Wang, D\. Y\. Ding, X\. Yang, K\. Vodrahalli, S\. He, D\. S\. Smith, Y\. Yin, D\. A\. McFarland, and J\. ZouCan large language models provide useful feedback on research papers? A large\-scale empirical analysis\.NEJM AI1\(8\),pp\. AIoa2400196\.Cited by:[§1](https://arxiv.org/html/2610.41391#S1.p3.1),[§2\.1](https://arxiv.org/html/2610.41391#S2.SS1.p1.1)\.
- Liu and Shah \(2023\)R\. Liu and N\. B\. ShahReviewerGPT? An exploratory study on using large language models for paper reviewing\.arXiv preprint arXiv:2306\.00622\.Cited by:[§1](https://arxiv.org/html/2610.41391#S1.p3.1),[§2\.1](https://arxiv.org/html/2610.41391#S2.SS1.p1.1)\.
- Liuet al\.\(2024\)Y\. Liu, T\. Yang, S\. Huang, Z\. Zhang, H\. Huang, F\. Wei, W\. Deng, F\. Sun, and Q\. ZhangCalibrating LLM\-based evaluator\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),Torino, Italia,pp\. 2638–2656\.Cited by:[§1](https://arxiv.org/html/2610.41391#S1.p3.1),[§2\.2](https://arxiv.org/html/2610.41391#S2.SS2.p1.1)\.
- Liuet al\.\(2025\)Z\. Liu, J\. Li, Y\. Zhuang, Q\. Liu, S\. Shen, J\. Ouyang, M\. Cheng, and S\. WangAm\-ELO: a stable framework for arena\-based LLM evaluation\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 38857–38868\.Cited by:[§2\.3](https://arxiv.org/html/2610.41391#S2.SS3.p1.1),[The paradigm comparison is not fully controlled\.](https://arxiv.org/html/2610.41391#Sx1.SS0.SSS0.Px1.p1.1)\.
- Luet al\.\(2024\)C\. Lu, C\. Lu, R\. T\. Lange, J\. Foerster, J\. Clune, and D\. HaThe AI scientist: towards fully automated open\-ended scientific discovery\.arXiv preprint arXiv:2408\.06292\.Cited by:[§1](https://arxiv.org/html/2610.41391#S1.p3.1),[§1](https://arxiv.org/html/2610.41391#S1.p4.1),[§2\.1](https://arxiv.org/html/2610.41391#S2.SS1.p1.1)\.
- Normanet al\.\(2026\)J\. D\. Norman, M\. U\. Rivera, and D\. A\. HughesReliability without validity: a systematic, large\-scale evaluation of LLM\-as\-a\-judge models across agreement, consistency, and bias\.arXiv preprint arXiv:2606\.19544\.Cited by:[§2\.2](https://arxiv.org/html/2610.41391#S2.SS2.p1.1)\.
- Ólafsson \(1990\)S\. ÓlafssonWeighted matching in chess tournaments\.Journal of the Operational Research Society41\(1\),pp\. 17–24\.Cited by:[§1](https://arxiv.org/html/2610.41391#S1.p3.1),[§2\.3](https://arxiv.org/html/2610.41391#S2.SS3.p1.1)\.
- OpenAI \(2023\)OpenAIGPT\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§1](https://arxiv.org/html/2610.41391#S1.p1.1)\.
- OpenAI \(2024\)OpenAIGPT\-4o mini: advancing cost\-efficient intelligence\.Note:[https://openai\.com/index/gpt\-4o\-mini\-advancing\-cost\-efficient\-intelligence/](https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/)Accessed October 28, 2026Cited by:[§3\.3](https://arxiv.org/html/2610.41391#S3.SS3.p2.1)\.
- OpenAI \(2025\)OpenAIGpt\-oss\-120b & gpt\-oss\-20b model card\.arXiv preprint arXiv:2508\.10925\.Cited by:[§3\.3](https://arxiv.org/html/2610.41391#S3.SS3.p2.1)\.
- Rao and Callison\-Burch \(2026\)D\. Rao and C\. Callison\-BurchAgreement metrics for LLM\-as\-judge evaluation: what to report and why\.arXiv preprint arXiv:2606\.00093\.Cited by:[§2\.2](https://arxiv.org/html/2610.41391#S2.SS2.p1.1)\.
- Raoet al\.\(2025\)V\. S\. Rao, A\. Kumar, H\. Lakkaraju, and N\. B\. ShahDetecting LLM\-generated peer reviews\.PLOS One20\(9\),pp\. e0331871\.Cited by:[§1](https://arxiv.org/html/2610.41391#S1.p1.1)\.
- Robertson \(2023\)Z\. RobertsonGPT4 is slightly helpful for peer\-review assistance: a pilot study\.arXiv preprint arXiv:2307\.05492\.Cited by:[§1](https://arxiv.org/html/2610.41391#S1.p3.1),[§2\.1](https://arxiv.org/html/2610.41391#S2.SS1.p1.1)\.
- Saito and Rehmsmeier \(2015\)T\. Saito and M\. RehmsmeierThe precision\-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets\.PLOS ONE10\(3\),pp\. e0118432\.Cited by:[Appendix P](https://arxiv.org/html/2610.41391#A16.p3.1)\.
- Shah \(2022\)N\. B\. ShahChallenges, experiments, and computational solutions in peer review\.Communications of the ACM65\(6\),pp\. 76–87\.Cited by:[§1](https://arxiv.org/html/2610.41391#S1.p1.1)\.
- Shrout and Fleiss \(1979\)P\. E\. Shrout and J\. L\. FleissIntraclass correlations: uses in assessing rater reliability\.\.Psychological Bulletin86\(2\),pp\. 420–428\.Cited by:[Appendix C](https://arxiv.org/html/2610.41391#A3.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2610.41391#S3.SS1.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2610.41391#S4.SS2.p1.1)\.
- Thakkaret al\.\(2025\)N\. Thakkar, M\. Yuksekgonul, J\. Silberg, A\. Garg, N\. Peng, F\. Sha, R\. Yu, C\. Vondrick, and J\. ZouCan LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025\.arXiv preprint arXiv:2504\.09737\.Cited by:[§1](https://arxiv.org/html/2610.41391#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.41391#S2.SS1.p1.1)\.
- Tonget al\.\(2026\)J\. Tong, M\. Li, H\. Li, Y\. Yang, Y\. Mou, W\. Ma, H\. Chen, X\. Liu, Q\. Cheng, M\. Zhang, Q\. Chen, W\. Ge, Q\. Guo, T\. Ying, T\. Sun, Y\. Zheng, Z\. Xi, X\. Chen, J\. Zhao, N\. Ding, X\. Huang, Y\. Jiang, and X\. QiuAI can learn scientific taste\.arXiv preprint arXiv:2603\.14473\.Cited by:[§2\.3](https://arxiv.org/html/2610.41391#S2.SS3.p1.1)\.
- Tranet al\.\(2020\)D\. Tran, A\. Valtchanov, K\. Ganapathy, R\. Feng, E\. Slud, M\. Goldblum, and T\. GoldsteinAn open review of OpenReview: a critical analysis of the machine learning conference review process\.arXiv preprint arXiv:2010\.05137\.Cited by:[§6](https://arxiv.org/html/2610.41391#S6.p3.1)\.
- Tripathiet al\.\(2025\)T\. Tripathi, M\. Wadhwa, G\. Durrett, and S\. NiekumPairwise or pointwise? Evaluating feedback protocols for bias in LLM\-based evaluation\.InSecond Conference on Language Modeling \(COLM 2025\),Cited by:[§2\.2](https://arxiv.org/html/2610.41391#S2.SS2.p1.1)\.
- Verma and Landrum \(2026\)N\. Verma and A\. R\. LandrumAll accept, no reject: evaluating LLMs as “peer” reviewers\.InProceedings of the 2026 CHI Conference on Human Factors in Computing Systems,Barcelona, Spain,pp\. 1–14\.External Links:[Document](https://dx.doi.org/10.1145/3772318.3791300)Cited by:[§1](https://arxiv.org/html/2610.41391#S1.p6.1)\.
- Wanget al\.\(2024\)P\. Wang, L\. Li, L\. Chen, Z\. Cai, D\. Zhu, B\. Lin, Y\. Cao, L\. Kong, Q\. Liu, T\. Liu, and Z\. SuiLarge language models are not fair evaluators\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 9440–9450\.Cited by:[§1](https://arxiv.org/html/2610.41391#S1.p2.1),[§2\.2](https://arxiv.org/html/2610.41391#S2.SS2.p1.1)\.
- Weiet al\.\(2026\)Q\. Wei, S\. Holt, J\. Yang, M\. Wulfmeier, and M\. van der SchaarPosition: the ML community must build an AI\-augmented peer\-review ecosystem\.InProceedings of the 43rd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.306,pp\. 172537–172554\.Cited by:[§1](https://arxiv.org/html/2610.41391#S1.p1.1)\.
- Xuet al\.\(2025\)Y\. Xu, L\. Ruis, T\. Rocktäschel, and R\. KirkInvestigating non\-transitivity in LLM\-as\-a\-judge\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 69583–69612\.Cited by:[§2\.3](https://arxiv.org/html/2610.41391#S2.SS3.p1.1)\.
- Yin and Huang \(2025\)L\. Yin and F\. HuangDIF: a framework for benchmarking and verifying implicit bias in LLMs\.arXiv preprint arXiv:2505\.10013\.Cited by:[Ethical Considerations](https://arxiv.org/html/2610.41391#Sx2.p1.1)\.
- Yuet al\.\(2026\)S\. Yu, M\. Luo, A\. Madasu, V\. Lal, and P\. HowardIs your paper being reviewed by an LLM? Benchmarking AI text detection in peer review\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 61517–61553\.Cited by:[§1](https://arxiv.org/html/2610.41391#S1.p1.1)\.
- Yuanet al\.\(2022\)W\. Yuan, P\. Liu, and G\. NeubigCan we automate scientific reviewing?\.Journal of Artificial Intelligence Research75,pp\. 171–212\.Cited by:[§1](https://arxiv.org/html/2610.41391#S1.p3.1),[§2\.1](https://arxiv.org/html/2610.41391#S2.SS1.p1.1)\.
- Zhanget al\.\(2025\)Y\. Zhang, H\. Zhang, W\. Ji, T\. Hua, N\. Haber, H\. Cao, and W\. LiangFrom replication to redesign: exploring pairwise comparisons for LLM\-based peer review\.InAdvances in Neural Information Processing Systems,Vol\.38,pp\. 11778–11805\.Cited by:[§1](https://arxiv.org/html/2610.41391#S1.p3.1),[§1](https://arxiv.org/html/2610.41391#S1.p4.1),[§2\.3](https://arxiv.org/html/2610.41391#S2.SS3.p1.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing, H\. Zhang, J\. Gonzalez, and I\. StoicaJudging LLM\-as\-a\-judge with MT\-Bench and Chatbot Arena\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 46595–46623\.Cited by:[§1](https://arxiv.org/html/2610.41391#S1.p1.1),[§1](https://arxiv.org/html/2610.41391#S1.p2.1),[§2\.2](https://arxiv.org/html/2610.41391#S2.SS2.p1.1)\.
- Zhuanget al\.\(2025\)Z\. Zhuang, J\. Chen, H\. Xu, Y\. Jiang, and J\. LinLarge language models for automated scholarly paper review: a survey\.Information Fusion124,pp\. 103332\.Cited by:[§1](https://arxiv.org/html/2610.41391#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.41391#S2.SS1.p1.1)\.

## Appendix APrompt Templates

All paradigms share the same reviewer framing\. Braces denote substituted fields;summary\_sectionis empty in the abstract\-only condition and otherwise carries the generated summary\. Appendix[B](https://arxiv.org/html/2610.41391#A2)gives one of these templates fully instantiated, together with the model’s responses\. To save space, the Direct and binary\-comparison templates below run the bulleted list of criteria into one line, and in the binary\-comparison template each paper block as well, and shorten the sample valueYour reasoning herein the JSON format to\.\.\.; Appendix[B](https://arxiv.org/html/2610.41391#A2)reproduces both prompts exactly\.

#### Direct Absolute Rating\.

> You are an experienced reviewer for a top\-tier AI conference \(ICLR 2025\)\. Paper Information: Title: \{title\} Abstract: \{abstract\} \{summary\_section\} Please provide an overall score for this paper on a scale of 1\-10, where: \- 1\-2: Strong reject \(fundamental flaws, no contribution\) \- 3\-4: Reject \(significant issues, limited contribution\) \- 5\-6: Borderline \(some merit but concerns\) \- 7\-8: Accept \(solid contribution, minor issues\) \- 9\-10: Strong accept \(exceptional contribution\) Consider: Novelty and originality; Technical quality and soundness; Clarity and presentation; Significance and impact\. Provide your reasoning in 2\-3 sentences, then give the score in JSON format: \{"reasoning": "\.\.\.", "score": <1\-10\>\}

#### Anchor Rating\.

Reframed around three topically similar ICLR 2024 reference papers shown with their average review scores, with the instruction to rate the paper “using the reference papers as calibration points”, the criteria introduced as a comparison with those papers, and the 1–10 score bands given without their descriptions\.

#### Group Rating\.

As Direct, but presenting\{num\_papers\}papers together with the instruction to “provide an exact score \(1\-10\) for EACH paper based on comparative evaluation”, returning a JSON object keyed by paper\.

#### Elo and Swiss \(binary comparison\)\.

> You are an experienced reviewer for a top\-tier AI conference \(ICLR 2025\)\. You need to compare two papers and decide which one is better overall\. \*\*Paper A:\*\* Title: \{title\_a\} Abstract: \{abstract\_a\} \{summary\_a\} \*\*Paper B:\*\* Title: \{title\_b\} Abstract: \{abstract\_b\} \{summary\_b\} Compare based on: Novelty and originality; Technical quality and soundness; Clarity and presentation; Significance and impact\. Which paper is better? Provide brief reasoning \(1\-2 sentences\), then your decision in JSON format: \{"reasoning": "\.\.\.", "winner": "A" or "B" or "tie"\}

#### Summarization\.

Vanilla summarization requests a 200–300 word summary covering main contribution and novelty, methodology, key results, and potential impact, with the instruction “Be objective and factual\. Do not include your personal opinions\.” Objective summarization requests the same length under four explicit headings \(Background, Method, Results, Limitations\) with the instruction “Be purely objective\. Only describe what the paper presents, without subjective judgments\.”

## Appendix BWorked Example: Instantiated Prompts and Model Responses

Appendix[A](https://arxiv.org/html/2610.41391#A1)gives the templates; this section gives one complete instantiation of them, reproduced from the experimental records\. Every call shown here usedgpt\-4o\-miniat temperature 0\.3 under the system message “You are an experienced research paper reviewer\.”, with a token limit of 300 for Direct Rating, 400 for Anchor Rating, 500 for Group Rating and 200 for the binary comparison\.

The subject is submissionP49gSPmrvN, an ICLR 2025 paper that was rejected and that all three of its reviewers scored 1\. It is shown because all three absolute paradigms returned 7 for it in the original 232\-paper runs, so a single case exhibits the inflation reported in Section[4](https://arxiv.org/html/2610.41391#S4)\. The text below is unaltered apart from the elisions marked\[abstract elided\], which remove every abstract in the anchor, group and comparison blocks, including the subject paper’s own abstract in the group block; that abstract is given in full in the Direct Rating prompt\. Neither the raw responses nor the per\-comparison verdicts for Elo and Swiss were retained in our records, only the resulting ratings and, for the population\-scale run, the request payloads, so the binary comparison is shown as a prompt alone\.

#### Direct Rating, prompt as sent\.

> You are an experienced reviewer for a top\-tier AI conference \(ICLR 2025\)\. Paper Information: Title: Time\-dependent Development of Scientific Discourse: A Novel Approach Using UMAP and Word Embeddings Abstract: This study presents a method for visualizing the time\-dependent development of a scientific discipline using UMAP \(Uniform Manifold Approximation and Projection\) and text embeddings\. This study demonstrates how the evolution of research interests and topics in a specific field can be mapped over time by encoding the abstracts of scholarly articles into a high\-dimensional space and then projecting them into a 3D space\. This computational approach converts the history of discourse into a point\-cloud that can be further studied as a manifold and as a time series, which leads to new insights into the dynamics of scholarly discourse and the emergence and disappearance of research themes\. Please provide an overall score for this paper on a scale of 1\-10, where: \- 1\-2: Strong reject \(fundamental flaws, no contribution\) \- 3\-4: Reject \(significant issues, limited contribution\) \- 5\-6: Borderline \(some merit but concerns\) \- 7\-8: Accept \(solid contribution, minor issues\) \- 9\-10: Strong accept \(exceptional contribution\) Consider: \- Novelty and originality \- Technical quality and soundness \- Clarity and presentation \- Significance and impact Provide your reasoning in 2\-3 sentences, then give the score in JSON format: \{"reasoning": "Your reasoning here", "score": <1\-10\>\}

#### Direct Rating, response as returned\.

> \{"reasoning": "The paper presents a novel approach to visualizing the evolution of scientific discourse using UMAP and text embeddings, which is a relevant and timely topic in the field of AI and data visualization\. The methodology appears sound, and the potential insights into research dynamics could have significant implications for understanding scholarly communication\. However, the clarity of the presentation and the depth of the analysis could be improved to enhance the overall impact of the findings\.", "score": 7\}

#### Anchor Rating, reference block as sent\.

The remainder of the prompt is described in Appendix[A](https://arxiv.org/html/2610.41391#A1)\. The three retrieved ICLR 2024 anchors average 3\.7, yet the model placed the subject paper above all of them\.

> Reference Paper 1: Title: From Language to 3D Worlds: Adapting Language Models for Point Cloud Perception Abstract: \[abstract elided\] Average Review Score: 5\.0/10 \-\-\- Reference Paper 2: Title: Causal Impact Index: A Causal Formulation of Citations Abstract: \[abstract elided\] Average Review Score: 3\.0/10 \-\-\- Reference Paper 3: Title: UGradSL: Machine Unlearning Using Gradient\-based Smoothed Label Abstract: \[abstract elided\] Average Review Score: 3\.0/10 \-\-\-

#### Anchor Rating, response as returned\.

> \{"reasoning": "The paper presents a novel approach to visualizing the evolution of scientific discourse using UMAP and word embeddings, which is a fresh perspective in the field of text analysis and visualization\. The technical execution appears sound, and the potential implications for understanding research dynamics are significant\. However, while the methodology is interesting, it may not reach the same level of innovation or impact as the best work seen in the reference papers, leading to a score that reflects solid but not exceptional contributions\.", "score": 7\}

#### Group Rating, paper list as sent\.

The subject paper is Paper D\. This block comes from the original score\-ordered grouping \(Appendix[D](https://arxiv.org/html/2610.41391#A4)\), which is why all five papers in it were rated 1\.0 to 1\.4 by human reviewers; in the three randomized\-grouping runs behind the reported abstract\-only Group figures, the same paper received 5, 4 and 5\.

> \*\*Paper A:\*\* Title: NEMESIS \\\\ Jailbreaking LLMs with Chain of Thoughts Approach Abstract: \[abstract elided\] \-\-\- \*\*Paper B:\*\* Title: Balancing Differential Discriminative Knowledge For Clothing\-Irrelevant Lifelong Person Re\-identification Abstract: \[abstract elided\] \-\-\- \*\*Paper C:\*\* Title: Systematic Review of Large Language Models: Applications, Limitations, Practical Usages and Future Directions Abstract: \[abstract elided\] \-\-\- \*\*Paper D:\*\* Title: Time\-dependent Development of Scientific Discourse: A Novel Approach Using UMAP and Word Embeddings Abstract: \[abstract elided\] \-\-\- \*\*Paper E:\*\* Title: KL DIVERGENCE OPTIMIZATION WITH ENTROPY\- RATIO ESTIMATION FOR STOCHASTIC GFLOWNETS Abstract: \[abstract elided\] \-\-\-

#### Group Rating, response as returned\.

> \{ "reasoning": "Paper A presents a novel approach to jailbreaking LLMs, addressing a critical issue in AI security, which enhances its significance and originality\. Paper B introduces an innovative framework for lifelong person re\-identification, but its contribution is somewhat niche and may not have as broad an impact\. Paper C, while comprehensive, lacks novelty as it primarily reviews existing literature without presenting new findings\. Paper D offers a unique method for visualizing scientific discourse, showcasing technical soundness and potential impact, but it may not appeal to a wide audience\. Paper E introduces a novel optimization approach for GFlowNets, demonstrating strong technical quality and potential significance in generative modeling\.", "scores": \{ "Paper A": 8, "Paper B": 6, "Paper C": 5, "Paper D": 7, "Paper E": 8 \} \}

#### Elo and Swiss, prompt as sent\.

Taken from the stored batch request payload for the first comparison of the population\-scale run \(custom\_id: elo\_\_0\)\.

> You are an experienced reviewer for a top\-tier AI conference \(ICLR 2025\)\. You need to compare two papers and decide which one is better overall\. \*\*Paper A:\*\* Title: Understand Clean Generalization and Robust Overfitting in Adversarial Training from Two Theoretical Views: Representation Complexity and Training Dynamics Abstract: \[abstract elided\] \*\*Paper B:\*\* Title: Quantifying Likeness: A Simple Machine Learning Approach to Identifying Copyright Infringement in \(AI\-Generated\) Artwork Abstract: \[abstract elided\] Compare based on: \- Novelty and originality \- Technical quality and soundness \- Clarity and presentation \- Significance and impact Which paper is better? Provide brief reasoning \(1\-2 sentences\), then your decision in JSON format: \{"reasoning": "Your reasoning here", "winner": "A" or "B" or "tie"\}

## Appendix CPDF Processing and Content Cleaning

Papers were processed through a multi\-stage pipeline: \(1\) PDF download from OpenReview, \(2\) conversion to Markdown using pymupdf4llm with a 60\-second timeout, and \(3\) fallback to abstract\-only for failed conversions\. The pipeline achieved a 94\.4% success rate \(219/232 papers\), with 12 timeouts \(5\.2%\) and 1 other failure \(0\.4%\)\. To give the LLM focused content, the cleaning step removes line counters, formatting artifacts, and figure and table captions, drops inline citations in numeric and author\-year formats, removes references and appendix sections, and normalizes whitespace\. Content exceeding the token budget was truncated to 30,000 characters\. These steps remove material that introduces noise or bias without contributing directly to quality assessment\.

#### Software\.

Anchor Rating retrieves its three in\-context anchors with scikit\-learn’scosine\_similarityovertext\-embedding\-3\-smallabstract embeddings\. The reported correlations \(Pearson, Spearman and Kendall\), the Wilcoxon signed\-rank tests of Section[3](https://arxiv.org/html/2610.41391#S3)and the trimmed mean of Appendix[M](https://arxiv.org/html/2610.41391#A13)were computed with SciPy; AUC\-ROC, Cohen’sκ\\kappa, macro and weighted F1, and NDCG with linear relevance gains were computed with scikit\-learn\. Accuracy, mean squared error, score inflation and the bootstrap resampling were computed directly in NumPy\. Three quantities come from our own implementations rather than a package: the partial rank correlations of Section[6](https://arxiv.org/html/2610.41391#S6), the base\-rate\-matchedκ\\kappaand macro F1 of Section[4\.5](https://arxiv.org/html/2610.41391#S4.SS5), and the intraclass correlations, namely ICC\(1,1\) and ICC\(1,kk\) in Section[3\.1](https://arxiv.org/html/2610.41391#S3.SS1)from a one\-way random\-effects analysis of variance corrected for the unequal number of reviewers per paper, and ICC\(2,1\) in Appendix[E](https://arxiv.org/html/2610.41391#A5)in the two\-way random, single\-rater, absolute\-agreement form of[Shrout and Fleiss \(1979\)](https://arxiv.org/html/2610.41391#bib.bib14)\. Matplotlib and seaborn were used for figures only\. All analysis ran under Python 3\.10; package versions were not pinned at run time and so are not reported\.

## Appendix DAdditional Paradigm Implementation Details

#### Group construction and prompt position\.

Group Rating’s groups were originally drawn from consecutive rows of a file ordered by score bin, which left each group nearly homogeneous in ground\-truth quality \(Spearman between group index and mean reviewer score0\.9860\.986; within\-group ground\-truth standard deviation0\.2570\.257against2\.122\.12overall\)\. Asking the model to separate five near\-equals depresses the paradigm rather than flattering it: re\-running the abstract\-only condition with random groups and randomized within\-prompt order raisesρ\\rhoagainst ground truth from0\.1550\.155\(the original grouping, re\-run alongside them; the original run reported in Table[8](https://arxiv.org/html/2610.41391#A14.T8)gives0\.1010\.101\) to between0\.2780\.278and0\.3430\.343over three seeds\. The abstract\-only Group Rating figures reported in this paper come from that corrected run, averaged over the three seeds; the vanilla and objective Group cells and the cross\-model Group rows retain the original grouping and were not re\-run\. Every other figure reported in this paper is a single run at temperature 0\.3 unless the accompanying text or caption states otherwise, so averaged and single\-run entries appearing side by side are not like\-for\-like on run\-to\-run variation\. Taken singly the three corrected Group runs give Spearmanρ=0\.295\\rho=0\.295, 0\.278 and 0\.343 against ground truth\.

A prompt\-position effect survives the correction\. A paper’s score correlates negatively with its position in the prompt in every condition we ran, including after positions are randomized \(−0\.325\-0\.325under the original grouping,−0\.217\-0\.217to−0\.327\-0\.327after randomization, mean−0\.279\-0\.279over the seven runs\), so papers listed earlier are scored higher irrespective of content, a primacy effect also seen in GPT\-4o\-mini and Llama 3\.3\-70B, which choose the first\-listed answer option 79\.0% and 77\.1% of the time against a 36\.8% chance rate\([Huang et al\., 2026b](https://arxiv.org/html/2610.41391#bib.bib49)\)\. This is a property of the paradigm rather than of the grouping, and it bounds what Group Rating can resolve within a prompt\.

#### Token budget for Group Rating under the open\-weight models\.

Group Rating returns one JSON object scoring five papers, and the 500\-token budget used for GPT\-4o\-mini truncated that response for the open\-weight models\. GPT\-oss\-120B parsed 0 of 232 papers at 500 tokens and was re\-run in full at 2,000; the 17 Llama 3\.3\-70B groups containing an unparsed score \(47 of their 85 papers\) were re\-run on the same basis\. Temperature and the remaining token budgets are unchanged from the GPT\-4o\-mini run, as is the Direct prompt; the Anchor, Group and binary\-comparison prompts are the condensed versions described in Section[3](https://arxiv.org/html/2610.41391#S3)\.

On the 232\-paper set, Elo uses 1,160 comparisons, 10 per paper on average, drawn by uniform random pairing fixed in advance, so the comparison schedule is independent of intermediate ratings; ratings are then updated sequentially in the order in which the comparisons were returned, and the per\-comparison outcomes were not retained\. Swiss uses⌈log2⁡\(232\)⌉=8\\lceil\\log\_\{2\}\(232\)\\rceil=8rounds, pairing by current standings from round 2 onward while avoiding repeat matchups\. At population scale, Elo uses 25 comparisons per paper on average and Swiss uses⌈log2⁡\(11,513\)⌉=14\\lceil\\log\_\{2\}\(11\{,\}513\)\\rceil=14rounds\. The Elo update factor isK=32K=32and final ratings are min–max normalized to the 1–10 scale\. Anchor Rating retrieves the three nearest anchors by cosine similarity of abstract embeddings\. The relatively weak Swiss result on the 232\-paper set may stem from strategic pairing forming local clusters that do not generalize to a global ranking, from too few rounds at that scale, and from binary win/loss outcomes carrying less information than Elo’s probabilistic updates\.

Wilcoxon signed\-rank tests on absolute prediction error under abstract\-only input give, for Elo versus each alternative, Direct \(Δ​\|error\|=\+0\.606\\Delta\|\\text\{error\}\|=\+0\.606,p<0\.001p<0\.001\), Anchor \(\+0\.912\+0\.912,p<0\.001p<0\.001\), Group \(\+0\.541\+0\.541,p<0\.001p<0\.001\), and Swiss \(\+0\.308\+0\.308,p<0\.01p<0\.01\); positive values mean the alternative has the larger error\.

## Appendix ETest–Retest Reliability of Direct Rating

This appendix measures a source of variation that the bootstrap intervals reported elsewhere do not cover: those intervals resample papers, and so quantify sampling error in the evaluated set rather than run\-to\-run variation under stochastic decoding at temperature 0\.3\.

#### Compute\-matched Direct\.

A single Direct rating carries substantial decoding noise \(ICC\(2,1\)=0\.417=0\.417\), so the natural control for the cost argument is to spend Elo’s budget on Direct instead\. We ran five independent Direct passes over the 232\-paper set, abstract\-only, with the published prompt and settings, and averaged the firstkkof them\. The Direct rows use the 230 papers with all five runs and the Elo row the 232\-paper run of Table[1](https://arxiv.org/html/2610.41391#S4.T1); each base\-rate\-matched cut is the threshold whose reject share is closest to the true rate:

Two things follow\. At equal compute Direct overtakes Elo on ranking, 0\.482 against 0\.410, so the cost argument of Section[4\.5](https://arxiv.org/html/2610.41391#S4.SS5)does not depend on giving Direct less budget\. And averaging does not repair reject blindness:κ\\kappaat the fixed≥6\\geq 6cut stays between 0\.046 and 0\.052 at everykk, because extra runs reduce decoding noise without moving where the scores sit\. That is the placement\-versus\-ordering distinction of Section[4\.5](https://arxiv.org/html/2610.41391#S4.SS5)appearing again, from the compute side\.

Table 3:Test–retest reliability of the GPT\-4o\-mini Direct paradigm at temperature 0\.3 \(three independent runs,n=11,513n=11\{,\}513papers, abstract\-only\)\. Reliability bands per[Koo and Li \(2016\)](https://arxiv.org/html/2610.41391#bib.bib15):<0\.5<0\.5poor,0\.50\.5–0\.750\.75moderate,0\.750\.75–0\.90\.9good,\>0\.9\>0\.9excellent\.Table[3](https://arxiv.org/html/2610.41391#A5.T3)reports the reliability statistics summarized in Section[4\.2](https://arxiv.org/html/2610.41391#S4.SS2)\. The mean pairwise Spearmanρ\\rhoof 0\.363 across the three runs shows that even the*ranking*induced by a single Direct pass is only moderately reproducible, and the Spearman correlation of the three\-run mean against ground truth \(0\.204\) is the population\-scale Direct figure reported in Table[9](https://arxiv.org/html/2610.41391#A16.T9)\.

## Appendix FPopulation\-Scale Error Localization

Figure 3:Population\-scale false\-positive \(strong paper wrongly filtered\) and false\-negative \(low\-quality paper wrongly accepted\) rates as a function of the accept threshold, swept from 5 to 7\. The dashed gray line marks the fixed≥6\\geq 6threshold used in Table[9](https://arxiv.org/html/2610.41391#A16.T9); the dotted green line marks the base\-rate\-matched threshold of Elo and Swiss; Direct Rating’s, at 7\.67, lies beyond the plotted range\. Direct Rating’s curves are flat across the plotted range because only 1\.8% of its predictions fall below 7, which is why a fixed threshold at 6 makes it appear degenerate\.Figure 4:Error localization by ground\-truth score bin \(population scale, threshold≥6\\geq 6\)\. For each paradigm, blue bars show false\-positive counts \(strong papers wrongly filtered\) and orange bars show false\-negative counts \(low\-quality papers wrongly accepted\)\. Direct’s errors are almost entirely false negatives, since its inflation accepts nearly all low\-quality papers, whereas Elo and Swiss distribute the error budget across both classes, with false positives concentrated at borderline ground\-truth scores\.Figure[3](https://arxiv.org/html/2610.41391#A6.F3)sweeps the accept threshold from 5 to 7, and Figure[4](https://arxiv.org/html/2610.41391#A6.F4)localizes the population\-scale errors by ground\-truth score bin\. Direct’s errors are almost entirely false negatives, whereas Elo and Swiss distribute the error budget across both classes\.

## Appendix GFull Combination Analysis

Table 4:Input representation compared, averaged across all five paradigms \(GPT\-4o\-mini\)\.rr: Pearson;ρ\\rho: Spearman;τ\\tau: Kendall; Infl\.: grade inflation\.Table[5](https://arxiv.org/html/2610.41391#A7.T5)reports all 15 paradigm\-by\-summarization combinations ranked by correlation\.

Table 5:All review methods, ranked by Pearson correlation \(GPT\-4o\-mini, 232\-paper sample\)\. Input: input representation; Corr\.: Pearsonrragainst ground truth; Acc\.: exact\-match rate on the rounded 1–10 score scale, which is not the binary accept/reject accuracy of Table[6](https://arxiv.org/html/2610.41391#A9.T6); Infl\.: grade inflation\. The Group vanilla and objective rows are computed on 230 papers, two having failed group assignment; the Group abstract row is the per\-paper mean of the three randomized\-grouping runs and covers all 232\.Within every paradigm, abstract\-only yields the highest correlation\. Averaged across input representations, Elo leads on Pearson correlation \(0\.344\), followed by Direct \(0\.305\), Anchor \(0\.245\), Group \(0\.200\), and Swiss \(0\.079\)\. Bootstrap 95% confidence intervals for Spearmanρ\\rhounder abstract\-only evaluation are Direct \[0\.302, 0\.537\], Elo \[0\.302, 0\.509\], Anchor \[0\.249, 0\.474\], Group \[0\.269, 0\.496\], and Swiss \[−\-0\.049, 0\.211\]; the interval for Swiss includes zero\. Figure[5](https://arxiv.org/html/2610.41391#A7.F5)shows the paradigm\-by\-representation interaction\.

![Refer to caption](https://arxiv.org/html/2610.41391v1/fig_summarization_heatmap.png)Figure 5:Interaction between paradigm and input representation\. Left: Spearmanρ\\rho\(higher is better\)\. Right: Mean Absolute Error \(lower is better\)\. Abstract\-only consistently outperforms across paradigms, and Elo achieves the lowest MAE regardless of input representation\.
## Appendix HScore Distributions by Input Representation and Paradigm

Figures[6](https://arxiv.org/html/2610.41391#A8.F6)and[7](https://arxiv.org/html/2610.41391#A8.F7)show the score distributions underlying the compression effect discussed in Sections[4\.1](https://arxiv.org/html/2610.41391#S4.SS1)and[4\.3](https://arxiv.org/html/2610.41391#S4.SS3): absolute paradigms concentrate their predictions in a narrow high band, whereas comparative paradigms retain a spread close to ground truth\.

Figure 6:Score distributions by input representation \(all paradigms pooled, GPT\-4o\-mini\)\. Violin shapes show density and box plots show quartiles; ground truth \(gray\) is shown for reference\. The three representations differ little in spread \(pooledσ\\sigma1\.56, 1\.62 and 1\.59 for abstract\-only, vanilla and objective\), and all are far narrower than ground truth \(σ=2\.16\\sigma=2\.16\)\. Annotated below each group:μ\\mu\(mean\),σ\\sigma\(standard deviation\),ρ¯\\bar\{\\rho\}\(mean Spearman correlation with ground truth across paradigms\)\.Figure 7:Score distributions for the five paradigms vs\. ground truth \(abstract\-only, GPT\-4o\-mini\)\. Absolute methods \(Direct, Anchor\) compress scores into a narrow 6–8 range, while comparative methods \(Elo, Swiss\) preserve broader distributions closer to ground truth\. Annotated below each:μ\\mu\(mean\),σ\\sigma\(standard deviation\),ρ\\rho\(Spearman correlation\)\.
## Appendix IStratified\-Sample Classification Metrics

Table 6:Binary classification performance \(reject/accept\) on the stratified 232\-paper sample\. Accuracy in %; AUC is AUC\-ROC on the continuous scores\.
## Appendix JOrdering Quality and Error Analysis

Table[7](https://arxiv.org/html/2610.41391#A10.T7)reports Normalized Discounted Cumulative Gain \(NDCG\) and top\-kkprecision, where top\-kkprecision is the fraction of a method’s top\-kkpredicted papers that are truly above the accept threshold\.

Table 7:Ordering\-quality metrics for six representative methods, chosen to span the paradigms and input representations discussed in the text rather than to rank the top six on any single column\. The first three numeric columns are NDCG@10, NDCG@20, and NDCG over all papers\. Top\-kkprecision is computed with a stable sort, so for paradigms that assign the same score to large blocks of papers it reports one arbitrary tie\-break rather than a property of the ordering; see the discussion of Direct \+ Abstract below\.All top methods achieve high overall NDCG \(\>0\.90\>0\.90\), but top\-kkprecision differs sharply\. Elo \+ Objective identifies 70% true accepts among its top\-20 predictions\. The corresponding figure for Direct \+ Abstract should not be read as a property of that paradigm: Direct emits only five distinct values on this sample and 87 of the 232 papers receive its highest score, so any top\-20 is selected by the tie\-break rule rather than by the model\. That top block is60\.9%60\.9\\%true accepts against a39\.7%39\.7\\%base rate, so Direct’s ordering is informative between blocks and simply unable to resolve within one; the 0\.000 entry in Table[7](https://arxiv.org/html/2610.41391#A10.T7)is what the stable sort used here returns, and a random tie\-break averages61%61\\%over 2,000 draws\. A residual analysis shows that Direct, Anchor, and Group have systematic positive residuals \(consistent over\-prediction\), Elo produces smaller residuals centered near zero, and Swiss has a near\-zero mean residual but higher variance\. All methods struggle at the extreme 1–2 and 9–10 ranges, reflecting both the rarity of such papers and the difficulty of identifying exceptional or fundamentally flawed work\.

## Appendix KSwiss Resolution and Elo Comparison Density

#### Swiss resolution\.

A Swiss tournament ofRRrounds scores papers by win/tie/loss record, so it can express at most2​R\+12R\+1distinct values, 29 atR=⌈log2⁡11,513⌉=14R=\\lceil\\log\_\{2\}11\{,\}513\\rceil=14\. The run realized 23 of them across 11,513 papers, with clumps averaging 501 papers and the largest holding 2,404 \(20\.9%\)\. The consequence is a ceiling that no improvement in judgment quality can lift:14\.8%14\.8\\%of all paper pairs receive identical scores and cannot be ordered, and any threshold placed at the base rate lands inside a clump of 1,819 papers, 21\.0% of the 8,655 papers carrying a decision, containing both 1,123 true rejects and 696 true accepts, all resolved by tie\-break\. Elo’s population\-scale ratings are all distinct, so matching that resolution under the same2​R\+12R\+1model would require 5,756 rounds, 411 times the 14 rounds and the API cost we report\.

#### Why anchoring does not help\.

The anchor set was selected for zero inter\-reviewer variance, which collapses its score range: standard deviation 1\.66 around a mean of 5\.14 with a maximum of 8\.0, against 2\.16 and a full 1–10 range in the test set, and only 13 of 100 anchors below 3 or at or above 8\. Averaging the three anchors retrieved per paper narrows the effective calibration signal further, to a standard deviation of 0\.93\. Anchor Rating is thus asked to calibrate a 1–10 judgment against exemplars that cluster near the middle and never reach the top of the scale\. We tested the obvious alternative explanation, that the model is dragged toward whatever anchors it is shown, as LLM link predictions anchor on network scores supplied in the prompt \(ρ=0\.42\\rho=0\.42;[Huang and Kim, 2026](https://arxiv.org/html/2610.41391#bib.bib52)\), and found no support for it: predicted scores correlate atρ=−0\.119\\rho=\-0\.119with the mean score of the three anchors retrieved for that paper, so the failure is a lack of usable spread in the reference points rather than an anchoring bias\.

#### Elo comparison density\.

We ran the population\-scale Elo at both 15 and 25 comparisons per paper\. The 67% increase in cost movesρ\\rhoagainst ground truth from 0\.132 to 0\.142 while base\-rate\-matchedκ\\kappaedges down from 0\.084 to 0\.079\. The two density settings agree with each other at onlyρ=0\.811\\rho=0\.811, so a substantial share of the rating variance reflects which comparisons happened to be drawn rather than the underlying judgments\. Density is therefore not the binding constraint at this scale: the extra comparisons buy a marginal gain in ranking and none in decision quality\.

## Appendix LScore Spread and Paradigm Combination

#### Why absolute agreement falls at population scale\.

The stratified sample covers the 1–10 range far more evenly than the population: standard deviation 2\.16 with27\.6%27\.6\\%of papers below 3 or at/above 8, against 1\.16 with61\.4%61\.4\\%of papers in\[5,7\)\[5,7\)\. The accept rate is nearly identical in the two designs \(39\.7%39\.7\\%against40\.5%40\.5\\%\), so the fall inκ\\kappafrom 0\.267 to 0\.076 is a spread effect rather than a prevalence effect\.

#### Score spread by paradigm\.

Under abstract\-only input, Direct and Anchor haveσ=0\.74\\sigma=0\.74and0\.810\.81with 97% and 94% of predictions in the 6–8 band; Elo and Swiss reachσ=1\.58\\sigma=1\.58and1\.671\.67with 37% and 28% in band\. Group Rating falls between them \(σ=1\.13\\sigma=1\.13, 68% in band\): scoring five papers together spreads the scale even though the output remains absolute\.

#### Combining paradigms\.

Direct Rating and Elo agree with each other only weakly \(ρ=0\.345\\rho=0\.345over all 11,513 scored papers\), so their errors are partly independent\. An equal\-weight average of theirzz\-scored outputs raisesρ\\rhofrom 0\.204 to 0\.211 and recalibratedκ\\kappafrom 0\.112 to 0\.115, but neither gain is distinguishable from zero under a paired bootstrap \(95% CI\[−0\.004,\+0\.018\]\[\-0\.004,\+0\.018\]and\[−0\.014,\+0\.020\]\[\-0\.014,\+0\.020\]\), so we do not claim a combination benefit\.

## Appendix MGround\-Truth Robustness

To check that the findings are not artifacts of the ground\-truth aggregation, we recomputed the metrics using three distinct aggregations of reviewer scores: mean \(our primary measure\), median, and mode\. A 10%\-trimmed mean was also computed but is not a fourth condition: papers here carry between two and seven reviewers, so a 10% trim removes no scores and returns the mean exactly\. The ranking\-potential ordering is identical under all three, and under the corrected Group Rating run reported here it is Direct\>\>Elo\>\>Group\>\>Anchor\>\>Swiss\. Elo maintains the highest decision utility across variants \(κ=0\.267\\kappa=0\.267–0\.3180\.318\), well above Direct \(κ=0\.034\\kappa=0\.034–0\.0500\.050\), so the separation between ranking potential and decision utility holds regardless of aggregation \(Figure[8](https://arxiv.org/html/2610.41391#A13.F8)\)\. These are fixed\-threshold figures on the stratified sample; Section[4\.5](https://arxiv.org/html/2610.41391#S4.SS5)shows that the separation itself is a placement effect, and this appendix establishes only that it is not an artifact of how reviewer scores are aggregated\.

![Refer to caption](https://arxiv.org/html/2610.41391v1/fig_gt_robustness.png)Figure 8:Robustness check: Spearmanρ\\rho\(left\) and Cohen’sκ\\kappa\(right\) under four ground\-truth aggregations, of which the trimmed mean coincides with the mean \(Appendix[M](https://arxiv.org/html/2610.41391#A13)\)\. Paradigm rankings are identical across variants\.
## Appendix NFull Cross\-Model Results

Recomputing every cell of Table[8](https://arxiv.org/html/2610.41391#A14.T8)at its own base\-rate\-matched cut, rather than the fixed≥6\\geq 6threshold, preserves the conclusion of Section[5](https://arxiv.org/html/2610.41391#S5)and sharpens it\. GPT\-4o\-mini Direct reachesκ=0\.336\\kappa=0\.336, above every other model\-paradigm pair; Llama 3\.3\-70B peaks at 0\.101 \(Anchor\) and GPT\-oss\-120B at 0\.160 \(Group\)\. The best paradigm still differs by model, so none transfers at either threshold\.

Table 8:Cross\-model comparison, all fifteen configurations: five paradigms×\\timesthree models \(abstract\-only, 232\-paper sample\)\. The Group rows, including GPT\-4o\-mini’s, use the original score\-ordered grouping \(Appendix[D](https://arxiv.org/html/2610.41391#A4)\); the corrected GPT\-4o\-mini abstract\-only Group run is reported in Table[1](https://arxiv.org/html/2610.41391#S4.T1)\. Three cells rest on fewer papers, where a response or a group assignment failed: GPT\-4o\-mini Groupn=230n=230, GPT\-oss\-120B Directn=230n=230and GPT\-oss\-120B Anchorn=228n=228\. Table[2](https://arxiv.org/html/2610.41391#S5.T2)summarizes the per\-model best\.
## Appendix OCross\-Model Score Distributions and theρ\\rho–κ\\kappaView

Figures[9](https://arxiv.org/html/2610.41391#A15.F9)–[12](https://arxiv.org/html/2610.41391#A15.F12)give the per\-model view behind Table[8](https://arxiv.org/html/2610.41391#A14.T8), and make explicit that a highρ\\rhodoes not imply a highκ\\kappa\.

![Refer to caption](https://arxiv.org/html/2610.41391v1/fig_cross_model_heatmaps.png)Figure 9:Cross\-model performance heatmaps\. Left: Cohen’sκ\\kappa\(practical decision utility\)\. Center: Spearmanρ\\rho\(ranking potential\)\. Right: grade inflation \(calibration\)\. GPT\-4o\-mini Elo achieves the highestκ\\kappadespite slightly lowerρ\\rhothan Direct\.Figure 10:Score distributions for Direct \(top row\) and Elo \(bottom row\); the columns are, left to right, GPT\-4o\-mini, Llama 3\.3\-70B, and GPT\-oss\-120B\. GPT\-4o\-mini shows clear separation between ground truth and Direct predictions \(inflation\), which Elo corrects\. Llama 3\.3\-70B Direct concentrates scores at 8–9, while its Elo output spreads scores without meaningful correlation\. GPT\-oss\-120B shows moderate spread under Direct but collapses under Elo, with75%75\\%of papers in the single bin\[5\.5,6\.5\)\[5\.5,6\.5\)and an interquartile range of 0\.265, the narrowest panel here and a counterexample to the idea that comparative paradigms necessarily widen the scale\.Figure 11:Model capability profiles averaged across paradigms, on five axes: the decision utility metrics \(κ\\kappaand macro F1\), the threshold\-free AUC\-ROC, ranking potential \(ρ\\rho\), and a calibration axis plotting1−inflation/41\-\\text\{inflation\}/4so that a larger radius is better on every axis\.Figure 12:Ranking potential versus grade inflation \(left\) and versus decision utility \(right\) for all 15 configurations\. GPT\-4o\-mini Direct sits at highρ\\rhobut near\-zeroκ\\kappa, while Elo converts similarρ\\rhointo higherκ\\kappathrough calibration\.
## Appendix PPopulation\-Scale Results

This appendix gives the population\-scale analysis summarized in Section[4\.4](https://arxiv.org/html/2610.41391#S4.SS4)and used by the recalibration control of Section[4\.5](https://arxiv.org/html/2610.41391#S4.SS5)\. Table[9](https://arxiv.org/html/2610.41391#A16.T9)reports it at the fixed≥6\\geq 6threshold and at each paradigm’s base\-rate\-matched cut\.

To test whether the result generalizes beyond the stratified sample, we re\-ran the GPT\-4o\-mini abstract\-only pipeline for Direct, Elo and Swiss on all 11,513 scored ICLR 2025 submissions with the same prompts and seed, taking the mean of three runs as Direct’s score\. Elo uses 25 comparisons per paper on average, of which the 57,565 added in the top\-up from 15 ran with a 350\-token output limit instead of 200: raising density from 15 movedρ\\rhoagainst ground truth only from 0\.132 to 0\.142, indicating that the population\-scale decline reflects the concentrated score distribution rather than an under\-sampled comparison graph\.

We exclude the 2,858 withdrawn papers, leaving 8,655 at a true reject rate of59\.5%59\.5\\%, and takeRejectas the positive class, so a false positive is a strong paper wrongly filtered\. Under this imbalance we read reject recall and macro F1 alongside accuracy\([Saito and Rehmsmeier, 2015](https://arxiv.org/html/2610.41391#bib.bib13)\)\(Table[9](https://arxiv.org/html/2610.41391#A16.T9)\)\.

Three observations follow\. First, reject blindness sharpens at scale: Direct flags0\.5%0\.5\\%of the below\-threshold papers here against4\.3%4\.3\\%on the stratified sample, with inflation essentially unchanged \(\+2\.20\+2\.20vs\.\+2\.13\+2\.13\)\.

Second, at this threshold the ordering of paradigms is preserved, but absolute agreement depends on the score distribution of the evaluated set\. Elo retains the highestκ\\kappa\(0\.076\), well below the0\.2670\.267of the 232\-paper sample\. This is not a class\-prevalence effect, since the accept rate is nearly identical in the two designs; what changes is the spread of ground\-truth scores, which is far wider in the stratified sample \(Appendix[L](https://arxiv.org/html/2610.41391#A12)\)\. Bothκ\\kappaandρ\\rhoare raised by an abundance of easy, far\-from\-threshold papers and lowered when most papers sit near the boundary\. We therefore recommend reporting absoluteκ\\kappaon a*deployment\-representative*sample, whose score distribution matches the pool the system would actually screen, treating the stratified\-sampleκ\\kappaas a complementary diagnostic\.

A structural limit explains much of Swiss’s weakness: a tournament ofRRrounds expresses at most2​R\+12R\+1distinct scores, 29 atR=14R=14, and the run realized 23 clumps,14\.8%14\.8\\%of pairs are unorderable whatever the LLM judged, and the base\-rate\-matched cut falls inside a clump decided entirely by tie\-break\. Elo has no such ceiling but shows sharply diminishing returns to comparison density, with schedule noise rather than comparison count binding\. Appendix[K](https://arxiv.org/html/2610.41391#A11)gives both analyses\.

Third, the two comparative paradigms trade errors differently: Swiss reaches the highest reject recall \(80\.6%\) but wrongly filters74\.9%74\.9\\%of truly\-accept papers, while Elo is more even \(61\.4% and 53\.8%\)\. Appendix[F](https://arxiv.org/html/2610.41391#A6)gives the threshold sweep from which an operating point follows for a given cost ratiocFP/cFNc\_\{\\text\{FP\}\}/c\_\{\\text\{FN\}\}, and shows the errors localizing in the borderline 4–7 band; population\-scale cost is in Appendix[S](https://arxiv.org/html/2610.41391#A19)\.

Table 9:Population\-scale results for GPT\-4o\-mini \(abstract\-only\), at the fixed≥6\\geq 6accept threshold and at each paradigm’s own base\-rate\-matched threshold \(the cut at which its predicted reject rate equals the true rate\)\. Recalibration is order\-preserving, soρ\\rhoand inflation are unchanged by it\.ρ\\rhoand inflation are computed on all 11,513 scored ICLR 2025 papers; decision metrics on the 8,655 non\-withdrawn papers \(true reject rate 59\.5%\)\. Direct rows use the mean of three independent runs \(ICC\(2,1\)=0\.417=0\.417\); because those scores are means of three integer ratings, Direct’s closest achievable predicted reject rate is62\.4%62\.4\\%rather than59\.5%59\.5\\%\.
## Appendix QResults Under the Recorded ICLR Decisions

Section[4\.5](https://arxiv.org/html/2610.41391#S4.SS5)defines Reject as a mean reviewer score below 6\. The recorded ICLR 2025 decision is the operationally meaningful label, and it is available for all 8,655 non\-withdrawn submissions\. The two agree on87\.8%87\.8\\%of papers \(κ=0\.749\\kappa=0\.749\); the decision\-based reject rate is57\.2%57\.2\\%against59\.5%59\.5\\%for the score rule\. Table[10](https://arxiv.org/html/2610.41391#A17.T10)repeats the population\-scale analysis against the decisions\.

Table 10:Population\-scale decision quality against the recorded ICLR 2025 accept/reject decisions rather than the≥6\\geq 6score rule\. The qualitative pattern is unchanged: Direct Rating is reject\-blind at a fixed threshold and recovers under recalibration, where it moves ahead of Elo\. Under these labels Direct Rating is the one configuration whose accuracy exceeds the always\-reject baseline, and only by 0\.3 points\.
## Appendix RCitation Data Collection and Window Analysis

Table 11:Spearmanρ\\rhobetween score and citation count, overall and within ICLR decision strata\.∗denotesp<0\.05p<0\.05\. The human correlation survives stratification\. As in Table[1](https://arxiv.org/html/2610.41391#S4.T1), the Group row is the per\-paper mean of three randomized\-grouping runs and the other rows are single runs\. Group Rating is the only LLM paradigm to reach significance among accepted papers\.#### Collection\.

Citation counts were retrieved from Semantic Scholar in August 2026\. Of the 232 test papers, 163 \(70%\) matched a record, by DOI where one was available and by normalized\-title similarity \(Pythondifflibsequence ratio, accepted at≥0\.90\\geq 0\.90\) otherwise\. The window reconstruction retrieves each paper’s citing papers by DOI, so the 28 matched papers without one could not be screened; of the remaining 135, the 126 with at least one citation and a complete set of paradigm scores are the basis of Table[11](https://arxiv.org/html/2610.41391#A18.T11)\. The 28 excluded papers are not uncited: 26 of them carry both citations and complete scores in the same August pull, and are omitted only because no DOI was available to query\. The unmatched papers are mostly submissions that were neither accepted nor posted as preprints, so the analysis is restricted to disseminated work and the correlations are estimated under that restriction\.

#### Window\.

Rather than compare two separate collections, we retrieved every citing paper with its publication date, which allows both windows to be computed from one source and so separates the effect of the window from the effect of the source\. Counting only citing papers published on or before 29 March 2026 gives the 12\-month window; counting all of them gives the 17\-month window\. Across 9,821 citing papers, 4\.0% lacked a usable date and are excluded from the 12\-month count\. The two windows correlate atρ=0\.992\\rho=0\.992with each other, and the score\-citation correlations differ by at most 0\.010 \(human review 0\.564 against 0\.568; Elo 0\.311 against 0\.308; Group Rating 0\.386 against 0\.380; Direct 0\.236 against 0\.238\)\.

#### Retrieving high\-impact papers\.

Ranking the 126 matched papers by each score and taking the top ten, human review recovers 5 of the 13 papers in the top citation decile against a random expectation of 1\.0; Swiss recovers 4, Elo 2, and the remaining paradigms 1 or 2\. These counts are not comparable across paradigms: with only 13 high\-impact papers they are very noisy, and for Direct no paper scores strictly above the tenth\-ranked value while for Group only four do, so their top ten is drawn entirely \(Direct\) or in six of ten places \(Group\) from a tie block and reflects the tie\-break order rather than the scores\. We report them for completeness only\.

#### Relation to an earlier collection\.

An earlier version of this analysis drew counts from a different source in March 2026 and reported much weaker correlations \(human reviewρ=0\.189\\rho=0\.189, Eloρ=0\.201\\rho=0\.201\), which we had read as parity between Elo and the human baseline\. That reading does not survive re\-collection\. At the same 12\-month cutoff, on the 123 papers present in both collections, the earlier data recorded57%57\\%as uncited with a median of 0, whereas the present collection records8%8\\%and a median of 11 at the same cutoff;86%86\\%of the papers previously recorded as uncited \(60 of 70\) have citations dated before it\. The earlier figures reflected incomplete indexing of recent work rather than the absence of citations, and the apparent parity was an artifact of that sparsity\.

Table 12:Operating points at the fixed≥6\\geq 6accept threshold, using population\-scale numbers \(Table[9](https://arxiv.org/html/2610.41391#A16.T9)\)\. These are a comparison among the configurations tested, not a deployment recommendation: under base\-rate\-matched thresholding \(Section[4\.5](https://arxiv.org/html/2610.41391#S4.SS5)\) Direct Rating attains the highestκ\\kappa, and no configuration here reaches the59\.5%59\.5\\%accuracy of always rejecting\. The cost ratiocFP/cFNc\_\{\\text\{FP\}\}/c\_\{\\text\{FN\}\}is the relative cost of filtering a strong paper \(FP\) versus admitting a low\-quality one \(FN\)\.

## Appendix SDeployment Cost and Operating Points

The population\-scale run used 259,035 API calls \(Direct3×11,5133\\times 11\{,\}513, Elo143,912143\{,\}912comparisons, and Swiss80,58480\{,\}584over 14 rounds\) at a total cost of $30\.37 \(Direct and Elo via the OpenAI Batch API at a 50% discount; Swiss via real\-time pricing\), $0\.0026 per paper\. Table[12](https://arxiv.org/html/2610.41391#A18.T12)summarizes the operating points of the configurations tested at the fixed≥6\\geq 6threshold; Section[4\.5](https://arxiv.org/html/2610.41391#S4.SS5)gives the base\-rate\-matched figures, which reverse the ordering\.

At the fixed≥6\\geq 6threshold Direct is degenerate as a quality filter: its near\-zero false\-positive rate is achieved only by accepting essentially every paper \(reject recall0\.5%0\.5\\%, FN rate99\.5%99\.5\\%\), which is operationally equivalent to having no filter\. Under base\-rate\-matched thresholding this reverses, and Direct attains the highestκ\\kappaof the three \(Section[4\.5](https://arxiv.org/html/2610.41391#S4.SS5)\)\. Swiss attains high reject recall but wrongly filters74\.9%74\.9\\%of strong papers, so deploying it as a primary gate is defensible only when downstream human review of all Swiss\-rejected papers is guaranteed, since the false\-positive volume is large \(2,626 strong papers wrongly filtered in the ICLR 2025 cohort\)\. We therefore make no deployment recommendation: at the fixed threshold Elo offers the more even error split, under base\-rate\-matched thresholding Direct Rating attains the highestκ\\kappawith 24% as many API calls as Elo, and none of the three reaches the accuracy of rejecting every paper\.