More Than Mimicking Reviewers: Evaluating LLMs for Pre-Submission Peer Review

# More Than Mimicking Reviewers:Evaluating LLMs for Pre-Submission Peer Review
Source: [https://arxiv.org/html/2609.05788](https://arxiv.org/html/2609.05788)
\\workshoptitle

Can We Trust the Judge?

Pouya ParsaUniversity of Minnesotaparsa025@umn\.edu&Amin Rezaei11footnotemark:1Invariant Tech Inc\.amin@invariant\.sh††thanks:Equal contribution\.

###### Abstract

Peer\-review feedback often arrives too late for authors to make meaningful revisions\. We study an author\-facing LLM system that moves part of this stress test before submission: it generates a broad pool of atomic concerns and compresses them into a short report\. We evaluate agreement with historical reviews and, separately, the possible validity of concerns they omit\.

From 10,000 ICLR 2026 submissions, we use 3,398 manuscripts with accessible versions that predate review\. On a ten\-paper diagnostic, independent sampling covers 44\.9% of historical issues; deduplication and refill reaches 78\.7% strict and 84\.9% seriousness\-weighted coverage, at 3\.6×\\timesmore requests and 5\.2×\\timesmore tokens\. A hidden Top\-32 Oracle preserves the full 79\.3% weighted coverage of a 256\-candidate pool, but paper\-only selectors retain only 40–44%\. LLM review therefore provides broad coverage with a large candidate pool but compresses poorly; ablations identify representative selection and matcher sensitivity as the main sources of this gap\.

## 1Introduction

Formal peer review supplies some of the most valuable feedback a research idea will receive, but usually only after submission\. By then, there may be little time to add a missing experiment, repair a central claim, or redesign an evaluation\. We ask whether an author\-facing LLM system can move part of this stress test earlier: inspect a draft, anticipate concerns plausible reviewers may raise, and surface additional weaknesses while revision is still possible\.

Our system follows a broad\-then\-compact design\. It first generates hundreds of evidence\-grounded concerns, removes semantic repeats, and then selects a short report\. This design couples generation to judging: the generator bounds what can be found, while semantic decisions determine which concerns count as new, which survive compression, and which pipeline appears best\. Known position and self\-preference effects\[[9](https://arxiv.org/html/2609.05788#bib.bib9),[8](https://arxiv.org/html/2609.05788#bib.bib8)\]therefore become system risks, not merely evaluation artifacts\.

Historical reviews also form an incomplete reference\. A small panel samples only some valid criticisms\. A generated issue that matches a historical review demonstrates*reviewer anticipation*; an unmatched issue may be invalid, or it may be useful feedback the panel missed\. We consequently separate*historical\-review agreement*from*independent validity*\. This distinction is central to the intended product: broad AI feedback should overlap strongly with human feedback without being forced to imitate it exactly\.

We evaluate on reviewer\-visible ICLR 2026 manuscripts\. Paper\-only generation is frozen before human reviews are opened; reviews are then converted to corrected atomic issues, and every candidate is compared with every issue\. This protocol measures breadth, redundancy, historical coverage, compression loss, and judge disagreement without revealing review targets during generation\. The results show that broad generation can recover much of a finite panel’s issue weight, but reducing the pool to 32 useful concerns remains difficult\.

Our contributions are:

- •a leakage\-controlled benchmark built from 10,000 submissions and a strict cohort of 3,398 pre\-review manuscripts;
- •a controlled pipeline comparison showing that deduplication and refill raise weighted historical coverage from 49\.5% to 84\.9%, but cost 5\.2×\\timesmore tokens;
- •a Top\-32 study that separates three error sources: semantic clustering, representative selection, and matching/atomization; and
- •evidence that the AI and human concern sets overlap substantially but are not interchangeable, motivating separate agreement and validity evaluation\.

## 2Related work

##### Review data and simulation\.

PeerRead enabled data\-driven study of 14\.7K drafts and 10\.7K expert reviews\[[5](https://arxiv.org/html/2609.05788#bib.bib5)\]; later work uses LLM agents to simulate reviewer behavior and decisions\[[4](https://arxiv.org/html/2609.05788#bib.bib4)\]\. We instead study an author\-facing task: discovering atomic concerns from a draft and retaining a compact subset\.

##### LLM\-generated scientific feedback\.

GPT\-4 feedback can overlap with human comments at rates comparable to human–human overlap, and many authors report finding it useful\[[6](https://arxiv.org/html/2609.05788#bib.bib6)\]\. MARG uses specialist agents\[[2](https://arxiv.org/html/2609.05788#bib.bib2)\], OpenReviewer fine\-tunes an 8B reviewer on 79K reviews\[[3](https://arxiv.org/html/2609.05788#bib.bib3)\], and TreeReview expands review questions dynamically\[[1](https://arxiv.org/html/2609.05788#bib.bib1)\]\. Our focus is complementary: semantic breadth, generation cost, Top\-kkcompression, and atomic agreement with hidden human issues\.

##### LLM\-as\-a\-judge\.

LLM judges capture semantic equivalence better than lexical metrics\[[10](https://arxiv.org/html/2609.05788#bib.bib10),[7](https://arxiv.org/html/2609.05788#bib.bib7)\], but exhibit position and same\-family biases\[[9](https://arxiv.org/html/2609.05788#bib.bib9),[8](https://arxiv.org/html/2609.05788#bib.bib8)\]\. We study these failures when judge scores define both evaluation and selection\.

## 3Methods

### 3\.1Task and system design

A draft goes through two main stages in our system: broad candidate generation and compression into an author\-facing report\.

##### Broad generation\.

For manuscriptPP, a paper\-only generator producesC⁡\(P\)=\{c1,…,cK\}C\(P\)=\\\{c\_\{1\},\\ldots,c\_\{K\}\\\}\. Independent sampling is the simplest baseline, but it repeatedly finds common concerns\. Our deduplication\-and\-refill pipeline checks schema and paper evidence, rejects a candidate judged semantically near an accepted concern, supplies the generator with a compact exclusion list, and continues until the candidate quota is filled\. This favors marginal semantic breadth rather than the number of nominal outputs\.

##### Compact selection\.

The paper\-only selectorSk​\(P,C\)⊆C⁡\(P\)S\_\{k\}\(P,C\)\\subseteq C\(P\)must choose at mostk=32k=32concerns without access to historical reviews\. We compare direct paper\-only ranking with cluster\-then\-select methods: partition candidates by semantic similarity, score possible representatives using only the manuscript, and allocate the final slots across clusters\. We call selectors that use hidden review matches*Oracles*: they provide upper bounds but are not feasible at inference time\.

### 3\.2Historical\-review agreement

Historical reviews are transformed into deduplicated issuesH⁡\(P\)=\{h1,…,hJ\}H\(P\)=\\\{h\_\{1\},\\ldots,h\_\{J\}\\\}\. For candidatescic\_\{i\}, a semantic judge estimates same\-concern probabilityMi​jM\_\{ij\}, where equivalence means the same substantive concern rather than a shared category\. We measure issue recallR⁡\(C,H\)R\(C,H\)at thresholdτ=0\.70\\tau=0\.70:

Mi​j=Pr\(ci≡hj∣P,ci,hj\),R\(C,H\)=1\|H\|∑j=1\|H\|\[maxiMi​j≥τ\]\.\\small M\_\{ij\}=\\Pr\(c\_\{i\}\\equiv h\_\{j\}\\mid P,c\_\{i\},h\_\{j\}\),\\hskip 18\.49988ptR\(C,H\)=\\frac\{1\}\{\|H\|\}\\sum\_\{j=1\}^\{\|H\|\}\\mathbf\{1\}\\\!\\left\[\\max\_\{i\}M\_\{ij\}\\geq\\tau\\right\]\.\(1\)
Weighted coverage replaces issue counts with seriousness weightswjw\_\{j\}; we also report Major\-Issue Recall, candidate match rate, and semantic uniqueness\. Hit rate differs from coverage because repeated candidates can match one issue while leaving others undiscovered\. The unrestricted Top\-kkOracle maximizes covered weight usingMM; the cluster Oracle first fixes a partition\. Their gap measures partition loss, and the remaining gap to a paper\-only selector measures representative\-selection loss\.

### 3\.3Independent validity and semantic saturation

The match matrix measures*reviewer anticipation*, not validity\. Unmatched concerns require separate assessment for grounding, soundness, specificity, importance, and whether the paper already addresses them\. Semantic saturation occurs as novel concerns become harder to find: ifpnp\_\{n\}is the acceptance probability afternnconcerns, reachingKKrequires𝔼⁡\[AK\]=∑n=0K−11/pn\\mathbb\{E\}\[A\_\{K\}\]=\\sum\_\{n=0\}^\{K\-1\}1/p\_\{n\}attempts in expectation\. Generator mixtures help only when models cover different semantic regions\.

## 4Experiments

### 4\.1Dataset and manuscript recovery

##### OpenReview source\.

##### Reviewer\-visible version recovery\.

The current forum PDF can contain post\-review changes, so it is not used as the input\. OpenReview–arXiv mapping recovers 2,123 pre\-submission versions, while OpenReview timestamps identify 1,548 pre\-review PDFs through a second recovery path\. The paths overlap for 273 papers; selecting arXiv on overlap yields 3,398 reviewer\-visible manuscripts\.

##### Post\-review and publication\-side snapshot\.

We retain each paper’s current OpenReview PDF separately from its pre\-review input\. For 1,392 accepted papers, this provides a public post\-review version for future revision analysis\.

Figure[1](https://arxiv.org/html/2609.05788#S4.F1)summarizes the recovery and version\-pairing workflow\. Historical and current files remain separate throughout\.

Figure 1:Data construction pipeline\.Table 1:Corpus construction and frozen paper\-family split\. Decisions are normalized from the explicit decision note and current venue state\.
##### Frozen split\.

Duplicate and likely resubmitted papers share a family ID before stratification\. The immutable split contains 2,446 training, 272 validation, and 680 held\-out test papers \(Table[1](https://arxiv.org/html/2609.05788#S4.T1)\)\. This paper reports retrospective training diagnostics and one five\-paper validation pilot; test targets remain unopened\.

### 4\.2Evaluation protocol

##### Atomic targets\.

Reviews are split into manually corrected, deduplicated atomic concerns\. Seriousness distinguishes quick fixes from issues requiring new experiments or changes to central claims\. The ten\-paper diagnostic contains 178 issues\.

##### Leakage boundary\.

Generation uses only the historical manuscript or its frozen structured representation; reviews, ratings, rebuttals, decisions, later versions, and match matrices remain hidden\.

##### Generators and judges\.

Generators and selectors include GPT\-5\.6 Sol, DeepSeek V4 Flash, Inkling NVFP4, Kimi\-K3, and Qwen3\.8\-2\.4T\-A95B\. Matching uses schema\-constrained, temperature\-zero calls and preserves the complete matrix\.

### 4\.3Baselines and comparisons

At 512 candidates, we compare independent sampling with deduplication and refill, then test early stopping and model mixtures\. At 256 candidates, we compare paper\-only compression with Top\-32 Oracles; a second matcher tests judge dependence\. Some settings vary jointly, so these are pipeline comparisons\.

### 4\.4Broad candidate generation

#### 4\.4\.1Independent sampling versus deduplication and refill

Table[2](https://arxiv.org/html/2609.05788#S4.T2)compares two DeepSeek V4 Flash pipelines at 512 retained candidates per paper\. Independent persona\-batched calls preserve repeated answers\. Deduplication and refill rejects a concern near an accepted one, supplies a compact forbidden\-concern list, and continues until quotas are filled\.

Table 2:DeepSeek generation strategies on ten papers at 512 retained candidates per paper\.Independent sampling covers 80/178 issues\. Although 1,265/5,120 candidates hit at least one target, only 656 \(12\.8%\) are semantically unique: many hits repeat the same issues\. Deduplication and refill covers 140/178 despite a lower candidate hit rate \(0\.204 versus 0\.247\)\. The gain is semantic breadth, not more nominal positives, but it requires 16,143 raw concerns, 1,905 requests, and 96\.61M tokens, compared with 525 requests and 18\.66M tokens for independent sampling\.

Figure 2:Candidate\-generation coverage and cost\.
#### 4\.4\.2Coverage gains become expensive

As we generate more unique candidates, coverage grows, but requests and token use rise faster\. Expanding from 256 to 512 candidates raises weighted coverage from \.692 to \.849 but requires 593 additional requests and 55\.61M tokens; the final 128 candidates add only nine issues\. Novelty\-based early stopping reduces requests by 87\.6% and tokens by 92\.3%, but lowers weighted coverage to \.660\.

#### 4\.4\.3Heterogeneous generators

On 93 training papers, size\-matched 32\-candidate Sol and DeepSeek pools obtain macro strict recall of \.387 and \.423; their union reaches \.535, with 265 Sol\-only and 347 DeepSeek\-only targets\. A provisional 256\-candidate mixture \(128 DeepSeek, 64 Inkling, 32 Kimi, 32 Qwen\) reaches \.767 weighted coverage on five validation papers\. Different policies and budgets prevent attributing the gain solely to model identity, but the support is complementary\.

### 4\.5Top\-32 compression

The practical report must compress 256 concerns to 32\. On ten training papers, the unrestricted Top\-32 Oracle preserves the full pool’s \.793 weighted coverage, so 32 slots have sufficient capacity under the frozen matrix\. Paper\-only selectors reach only \.402–\.442\. Clustering preserves most of the coverage available to the unrestricted Oracle; the larger loss occurs when a paper\-only model chooses which member represents each cluster\. We isolate the partition, selection, and matcher effects in Section[5](https://arxiv.org/html/2609.05788#S5)\(Table[3](https://arxiv.org/html/2609.05788#S4.T3)\)\.

Table 3:Top\-32 compression on ten training papers\. All rows use the same frozen candidate–issue matrix\. Oracle selectors access that matrix and are upper bounds, not deployable systems\.#### 4\.5\.1Historical matching is not the goal

Agreement with human reviews measures anticipation of a finite panel, not the full value of AI feedback\. Unmatched candidates include errors, but they can also identify concrete issues the panel omitted\. For*OmniSpatial*, one candidate flags different answer\-extraction methods for reasoning and non\-reasoning models as a possible confound\. For*Regularization can make diffusion models more efficient*, another asks how an asymptotic guarantee supports the motivating few\-step regime without a finite\-TTthreshold\. Both are specific and paper\-grounded, yet neither crosses the historical\-match threshold\. We therefore evaluate historical agreement separately from independent validity; expert adjudication of the latter remains incomplete\.

## 5Ablation study

We vary exploration, clustering, representative selection, and matching to locate the main bottlenecks\.

### 5\.1Exploration policy

Fine\-grained persona quotas increase duplicate rejection from 37\.1% to 74\.5%, while rigid section–category–persona slots returnno\_issue56\.7% of the time\. Together with adaptive stopping, these results show that local novelty is not enough; exploration must value marginal pool coverage\.

### 5\.2Partitioning and representative selection

Across eight clustering families, the one\-per\-cluster Oracle retains \.755 weighted coverage versus \.793 for the unrestricted Oracle\. Of 160 clusters, 70 have no historical edge and 41 link to multiple issues, indicating both splitting and merging errors, but only modest aggregate partition loss\.

Figure 3:Candidate clusters and their historical\-review matches\.Representative selection loses much more\. With a fixed Ward partition, Qwen pointwise selection and the best LLM tournament reach \.402 and \.442 weighted coverage, versus \.755 for the cluster Oracle\. On five confirmation papers, partitioning costs 6\.1 points and Qwen selection costs 36\.1\. Only one candidate captures the issue in 49\.2% of issue\-bearing clusters, and Qwen score has \.158 Spearman correlation with hidden utility\. Retaining four candidates per cluster raises the hidden ceiling to \.709, locating the bottleneck in final selection rather than shortlist construction\.

### 5\.3Matcher and target sensitivity

At the same \.70 threshold, Sol High marks 660 of 45,568 candidate–issue pairs positive; DeepSeek V4 Flash marks eight \(κ=\.015\\kappa=\.015\)\. This scale collapse is specific to the tested deployments but shows that valid outputs do not ensure calibrated probabilities\. Review atomization adds further noise by dropping entities, qualifiers, or causal context, so measured misses combine pipeline errors with matcher and target\-extraction errors\.

### 5\.4Generator and judge interaction

With provider labels removed, a Sol selector still chooses 319/320 Sol candidates from mixed pools\. Because Sol also supplies the evaluation matrix, style recognition or same\-family preference can confound ranking gains\[[8](https://arxiv.org/html/2609.05788#bib.bib8)\]\. Judge identity, prompt, threshold, and generator\-conditional error should therefore accompany judge\-guided results\.

## 6Conclusion

Broad LLM generation can anticipate much of a historical reviewer panel: the deduplication\-and\-refill pipeline reaches 78\.7% strict and 84\.9% weighted coverage in our ten\-paper diagnostic\. The AI pool also contains concerns outside the recorded reviews\. This is the promise of pre\-submission review: useful feedback arrives while authors can still act, and need not be limited to imitating one finite panel\.

The open problem is trustworthy compression\. Thirty\-two concerns are enough under a hidden oracle, and semantic clustering is useful, but present representative selectors lose roughly 35 weighted\-coverage points\. Matcher and atomization noise further obscure the true overlap\. The right evaluation must therefore report generation breadth, Top\-kkloss, cost, judge robustness, and two distinct outcomes: anticipation of human feedback and independent validity of AI\-only feedback\. Under those constraints, LLM review can complement formal peer review without replacing it\.

## References

- \[1\]Yuan Chang, Ziyue Li, Hengyuan Zhang, Yuanbo Kong, Yanru Wu, Hayden Kwok\-Hay So, Zhijiang Guo, Liya Zhu, and Ngai Wong\.TreeReview: A dynamic tree of questions framework for deep and efficient LLM\-based scientific peer review\.In*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, pages 15651–15682, 2025\.doi:10\.18653/v1/2025\.emnlp\-main\.790\.URL[https://aclanthology\.org/2025\.emnlp\-main\.790/](https://aclanthology.org/2025.emnlp-main.790/)\.
- \[2\]Mike D’Arcy, Tom Hope, Larry Birnbaum, and Doug Downey\.MARG: Multi\-agent review generation for scientific papers\.*arXiv preprint arXiv:2401\.04259*, 2024\.URL[https://arxiv\.org/abs/2401\.04259](https://arxiv.org/abs/2401.04259)\.
- \[3\]Maximilian Idahl and Zahra Ahmadi\.OpenReviewer: A specialized large language model for generating critical scientific paper reviews\.In*Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(System Demonstrations\)*, pages 550–562, 2025\.doi:10\.18653/v1/2025\.naacl\-demo\.44\.URL[https://aclanthology\.org/2025\.naacl\-demo\.44/](https://aclanthology.org/2025.naacl-demo.44/)\.
- \[4\]Yiqiao Jin, Qinlin Zhao, Yiyang Wang, Hao Chen, Kaijie Zhu, Yijia Xiao, and Jindong Wang\.AgentReview: Exploring peer review dynamics with LLM agents\.In*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 1208–1226, 2024\.doi:10\.18653/v1/2024\.emnlp\-main\.70\.URL[https://aclanthology\.org/2024\.emnlp\-main\.70/](https://aclanthology.org/2024.emnlp-main.70/)\.
- \[5\]Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz\.A dataset of peer reviews \(PeerRead\): Collection, insights and NLP applications\.In*Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 1647–1661, 2018\.doi:10\.18653/v1/N18\-1149\.URL[https://aclanthology\.org/N18\-1149/](https://aclanthology.org/N18-1149/)\.
- \[6\]Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Yi Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Scott Smith, Yian Yin, Daniel A\. McFarland, and James Zou\.Can large language models provide useful feedback on research papers? a large\-scale empirical analysis\.*NEJM AI*, 1\(8\), 2024\.doi:10\.1056/AIoa2400196\.URL[https://doi\.org/10\.1056/AIoa2400196](https://doi.org/10.1056/AIoa2400196)\.
- \[7\]Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu\.G\-Eval: NLG evaluation using GPT\-4 with better human alignment\.*arXiv preprint arXiv:2303\.16634*, 2023\.URL[https://arxiv\.org/abs/2303\.16634](https://arxiv.org/abs/2303.16634)\.
- \[8\]Arjun Panickssery, Samuel R\. Bowman, and Shi Feng\.LLM evaluators recognize and favor their own generations\.*arXiv preprint arXiv:2404\.13076*, 2024\.URL[https://arxiv\.org/abs/2404\.13076](https://arxiv.org/abs/2404.13076)\.
- \[9\]Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui\.Large language models are not fair evaluators\.*arXiv preprint arXiv:2305\.17926*, 2023\.URL[https://arxiv\.org/abs/2305\.17926](https://arxiv.org/abs/2305.17926)\.
- \[10\]Lianmin Zheng, Wei\-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P\. Xing, Hao Zhang, Joseph E\. Gonzalez, and Ion Stoica\.Judging LLM\-as\-a\-judge with MT\-bench and chatbot arena\.*arXiv preprint arXiv:2306\.05685*, 2023\.URL[https://arxiv\.org/abs/2306\.05685](https://arxiv.org/abs/2306.05685)\.

## Appendix AAdditional experimental details

All evidence required for the main claims appears in the body\. Frozen machine\-readable tables retain per\-paper values, model and prompt versions, candidate IDs, provider usage, and artifact hashes\.

## Appendix BJudge and annotation disclosure

The headline matcher is a frozen GPT\-5\.6 Sol High prompt at temperature zero and threshold \.70\. Raw probabilities, pair texts, structured outputs, and cache keys are retained\. A second DeepSeek prompt produced the scale\-collapse diagnostic and was never substituted into headline metrics\. The blinded manual Top\-32 interface hides historical reviews and stores rankings separately by annotator and paper\. The pilot is not complete enough to estimate validity or inter\-annotator agreement; its only use here is to motivate a preregistered, independent expert audit\.

## Appendix CCandidate\-generation experiment inventory

Table[4](https://arxiv.org/html/2609.05788#A3.T4)records the broader sequence of generation experiments behind the focused strategy comparison in the main paper\. The machine\-readable values and their artifact provenance are stored indata/candidate\_generation\_metrics\.json; the publication figure is regenerated withscripts/plot\_candidate\_generation\.py\.

Table 4:Candidate\-generation experiments\. All recall values use frozen atomic human issues and a same\-issue threshold of0\.700\.70unless noted\.Table 5:Cost–coverage consequence of novelty\-based early stopping\. Counts and tokens are totals over the same ten\-paper development cohort\. Wall time was directly observed only for the adaptive run; n/r means not recorded\.
## Appendix DClustering artifact provenance

The compact cluster graph and the two illustrative unmatched concerns are indata/clustering\_analysis\.json\. They are exported from the frozen UI bundle withscripts/export\_clustering\_analysis\.py; the publication figure is regenerated withscripts/plot\_clustering\_analysis\.py\.

## Appendix ESecondary acceptance\-judging diagnostic

Outside the main review\-generation story, we compared paper\-only and sanitized paper\-plus\-review outcome judgments on 15 training papers \(10 Accept, five Reject\)\. Sol and Kimi improved with review prose, while DeepSeek and Inkling did not improve hard accuracy\. Because the cohort is small, schemas differ between conditions, and current review notes may contain post\-rebuttal edits, this is an exploratory context\-sensitivity result rather than an acceptance predictor\.

Table 6:Binary Accept/Reject prediction on 15 training papers\. “A/R rec\.” gives Accept and Reject recall\. Reviews contain prose only\.