RubricReviewer: From Direct Critique to Objective and Comprehensive Rubric-Driven Peer Review

# RubricReviewer: From Direct Critique to Objective and Comprehensive Rubric-Driven Peer Review
Source: [https://arxiv.org/html/2608.00005](https://arxiv.org/html/2608.00005)
Shuyu Guo1Wenxiang Hu2Yuyue Zhao2Yougang Lyu2Xiaohui Yan2 1Shandong University2Huawei Technologies Co\., Ltd\. guoshuyu225@gmail\.com,huwenxiang3@huawei\.com,yuyuezhao@h\-partners\.com, yougang\.lyu@huawei\-partners\.com,yanxiaohui2@huaiwei\.com

###### Abstract

Peer review at major venues is under unprecedented submission pressure, motivating the use of large language models \(LLMs\) as review assistants\. Existing LLM\-based reviewers, however, face two structural limitations\. First, they map manuscripts directly to reviews, leaving the underlying rubric implicit and entangling its derivation with the judgement\. Second, the prevailing paradigms each capture only half of a good review: training\-free agents gather broad evidence but produce undirected critiques, while training\-based reviewers inherit human discriminative judgement together with its noise and uneven coverage\. We introduceRubricReviewer, a fully rubric\-driven framework that addresses both limitations\. It makes rubric generation an explicit intermediate step, so that both review generation and the final assessment are conditioned on paper\-adaptive rubrics\. It further combines a training\-free agent \(Scout\) that gathers external evidence with a human\-aligned trained model \(Aligner\) that consumes this evidence, fusing the strengths of both supervision sources\. Experiments on real\-world submissions show thatRubricReviewerproduces reviews that are markedly more comprehensive and more discriminative than prior systems, and exhibits the strongest robustness against adversarial prompt\-injection attacks\. Ablation studies further confirm the necessity of each component\.

RubricReviewer: From Direct Critique to Objective and Comprehensive Rubric\-Driven Peer Review

Shuyu Guo1††thanks:Work performed during an internship at Huawei\.Wenxiang Hu2Yuyue Zhao2Yougang Lyu2Xiaohui Yan21Shandong University2Huawei Technologies Co\., Ltd\.guoshuyu225@gmail\.com,huwenxiang3@huawei\.com,yuyuezhao@h\-partners\.com,yougang\.lyu@huawei\-partners\.com,yanxiaohui2@huaiwei\.com

## 1Introduction

Peer review underpins scientific progress, but its capacity has not kept pace with the explosive growth of submissionsSun \([2025](https://arxiv.org/html/2608.00005#bib.bib35)\)\. A reliable automated peer\-review system would relieve reviewers from heavy workload while providing authors with actionable, paper\-specific feedback to accelerate their manuscript refinement\.

Automated peer review has accordingly attracted growing attention\. Early work framed the problem as score regression or accept/reject classification on static datasetsKang et al\. \([2018](https://arxiv.org/html/2608.00005#bib.bib15)\); Dycke et al\. \([2023](https://arxiv.org/html/2608.00005#bib.bib6)\)\. With the advent of large language models \(LLMs\), the focus has shifted toward generating the textual review directly\. Current methods broadly cluster into two paradigms\.*Training\-based*methods fine\-tune LLMs on real human reviews to align outputs with expert judgementsWeng et al\. \([2025](https://arxiv.org/html/2608.00005#bib.bib37)\); Zhu et al\. \([2025b](https://arxiv.org/html/2608.00005#bib.bib48)\)\.*Training\-free*methods build LLM\-agent pipelines that operate without parameter updates, typically through retrieval\-augmented evidence gatheringLi et al\. \([2026c](https://arxiv.org/html/2608.00005#bib.bib21)\)or multi\-agent committee simulationLu et al\. \([2024](https://arxiv.org/html/2608.00005#bib.bib25)\); Jin et al\. \([2024](https://arxiv.org/html/2608.00005#bib.bib14)\)\.

Despite these advances, the prevailing paradigm still suffers from two limitations\. \(1\) Existing methods map a manuscript directly to its final review, leaving the underlying rubric implicit\. Such direct generation entangles two distinct subprocesses, deriving the rubric and judging the paper against it, thereby substantially raising the learning difficulty\. \(2\) These two paradigms each capture only half of what a good review requires\. Training\-free LLMs excel at broad evidence gathering and summarisation, yet tend to produce untargeted critiques that gravitate toward neutral judgementsLi et al\. \([2025](https://arxiv.org/html/2608.00005#bib.bib19)\); Dycke and Gurevych \([2026](https://arxiv.org/html/2608.00005#bib.bib5)\)\. Training\-based reviewers inherit the discriminative judgements humans deliver, yet also their noise and uneven coverageSadallah et al\. \([2025](https://arxiv.org/html/2608.00005#bib.bib28)\); Li et al\. \([2026a](https://arxiv.org/html/2608.00005#bib.bib18)\)\.

To address both limitations, we proposeRubricReviewer, a framework comprising two key designs, each tackling one limitation\. For \(1\), we introduce explicit rubric generation as an intermediate step, decomposing peer review into three sequential stages, namely rubric generation, rubric\-conditioned review generation, and final assessment\. For \(2\), our pipeline combines both supervision sources via two cooperating components\.Scoutis a training\-free set of role\-specialised LLM agents that retrieve external evidence and produce structured analyses as references\.Aligneris a trained model that consumes these references and produces outputs aligned with human judgement\. Together,RubricReviewerfuses the broad coverage and evidence\-gathering strengths of agentic LLMs with the discriminative judgements distilled from human reviews\.

Extensive experiments on real\-world submissions demonstrateRubricReviewer’s superior performance across multiple dimensions\. The rubrics underlying our reviews cover human\-essential evaluation dimensions far more comprehensively than any baseline, and the resulting per\-rubric reviews show substantially higher agreement with human reviewers while being markedly more discriminative\. Building on these more comprehensive and discriminative reviews, the final assessment also attains the best alignment with human ratings and decisions\. Ablation studies confirm the necessity of each design: removing either the rubric\-based decomposition or the cooperation betweenScoutandAlignercauses a consistent performance drop\.RubricRevieweralso exhibits strong robustness under adversarial prompt\-injection attacks\.

WithRubricReviewer, we explore a new direction for automated peer review through a fully rubric\-driven framework\. Both review generation and the final assessment are conditioned on rubrics dynamically derived from each manuscript, recasting an inherently subjective open\-ended task as a collection of objective per\-rubric judgements\. It is also an attempt to jointly harvest the complementary benefits of agentic LLMs and supervised fine\-tuning on human reviews within a single pipeline\. Its competitive performance points to a promising path toward reliable, comprehensive, and discriminative LLM\-based peer reviewers\.

## 2Related Work

### 2\.1Rubric\-based Evaluation

Evaluating open\-ended tasks without verifiable ground truth has long been challengingKrishna et al\. \([2021](https://arxiv.org/html/2608.00005#bib.bib16)\)\. Rubric\-based methods address this by decomposing a subjective objective into a set of fine\-grained, binary\-checkable criteria, recasting holistic quality judgement as a collection of objectively answerable sub\-questionsGunjal et al\. \([2025](https://arxiv.org/html/2608.00005#bib.bib10)\); Viswanathan et al\. \([2025](https://arxiv.org/html/2608.00005#bib.bib36)\)\. The resulting structured signal is interpretable and extends supervision to non\-verifiable domains, serving both as an evaluation metricSharma et al\. \([2025](https://arxiv.org/html/2608.00005#bib.bib31)\); Arora et al\. \([2025](https://arxiv.org/html/2608.00005#bib.bib1)\)and as a reinforcement\-learning rewardZhang et al\. \([2025](https://arxiv.org/html/2608.00005#bib.bib44)\); Zhou et al\. \([2025](https://arxiv.org/html/2608.00005#bib.bib46)\)\.

Early work adopts predefined rubrics shared across all inputs, which are easy to deploy but lack granularityHashemi et al\. \([2024](https://arxiv.org/html/2608.00005#bib.bib11)\); Que et al\. \([2024](https://arxiv.org/html/2608.00005#bib.bib27)\); Shao et al\. \([2024](https://arxiv.org/html/2608.00005#bib.bib30)\)\. Subsequent work moves to query\-specific rubrics tailored to each inputXie et al\. \([2025](https://arxiv.org/html/2608.00005#bib.bib40)\); Liu et al\. \([2025](https://arxiv.org/html/2608.00005#bib.bib23)\), and further to adaptive rubrics that evolve with model behaviourSheng et al\. \([2026](https://arxiv.org/html/2608.00005#bib.bib32)\)or align with human preferences during trainingLv et al\. \([2026](https://arxiv.org/html/2608.00005#bib.bib26)\); Xu et al\. \([2026](https://arxiv.org/html/2608.00005#bib.bib41)\)\. Such instance\-aware rubrics have yielded notable gains in long\-form report generationShao et al\. \([2025](https://arxiv.org/html/2608.00005#bib.bib29)\), complex reasoningSheng et al\. \([2026](https://arxiv.org/html/2608.00005#bib.bib32)\), and other open\-ended tasksArora et al\. \([2025](https://arxiv.org/html/2608.00005#bib.bib1)\)\.

Peer review of scientific manuscripts shares this difficulty\. Reviews are free\-form and highly subjective, with no single verifiable answerZhu et al\. \([2025a](https://arxiv.org/html/2608.00005#bib.bib47)\)\. Yet existing LLM\-based peer reviewers map a manuscript directly to a reviewWeng et al\. \([2025](https://arxiv.org/html/2608.00005#bib.bib37)\), leaving the underlying evaluation rubric implicit and the supervision signal coarse\. The closest prior attempt incorporates rubrics into the review generation process but relies on a small set of meta\-rubrics shared across all papersLi et al\. \([2026c](https://arxiv.org/html/2608.00005#bib.bib21)\)\. In contrast, our work drives the peer review pipeline with paper\-adaptive rubrics dynamically generated for each manuscript, using them to condition both review generation and final assessment\.

### 2\.2LLM\-based Peer Review

The growing volume of conference submissions has made efficient and reliable peer review assistance increasingly valuable, both for reducing reviewer burden and for helping authors improve their manuscriptsSun \([2025](https://arxiv.org/html/2608.00005#bib.bib35)\); Su et al\. \([2025](https://arxiv.org/html/2608.00005#bib.bib33)\); Sun et al\. \([2024](https://arxiv.org/html/2608.00005#bib.bib34)\)\. Early work framed automated peer review as score regression or accept/reject classification on static datasetsKang et al\. \([2018](https://arxiv.org/html/2608.00005#bib.bib15)\); Yuan et al\. \([2022](https://arxiv.org/html/2608.00005#bib.bib42)\); Dycke et al\. \([2023](https://arxiv.org/html/2608.00005#bib.bib6)\), while more recent efforts shift toward LLM\-based review generation that directly produces textual feedbackWu et al\. \([2025](https://arxiv.org/html/2608.00005#bib.bib39)\)\.

Existing LLM\-based peer reviewers fall broadly into two paradigms: training\-based and training\-free\. Training\-based methods fine\-tune LLMs on real human reviews to align outputs with human judgmentsIdahl and Ahmadi \([2025](https://arxiv.org/html/2608.00005#bib.bib13)\); Weng et al\. \([2025](https://arxiv.org/html/2608.00005#bib.bib37)\); Zhu et al\. \([2025b](https://arxiv.org/html/2608.00005#bib.bib48)\)\. Common instantiations include aspect\- or template\-conditioned supervised fine\-tuningGao et al\. \([2024](https://arxiv.org/html/2608.00005#bib.bib8)\); Idahl and Ahmadi \([2025](https://arxiv.org/html/2608.00005#bib.bib13)\); Weng et al\. \([2025](https://arxiv.org/html/2608.00005#bib.bib37)\), fine\-tuning on long chain\-of\-thought reasoning tracesZhu et al\. \([2025b](https://arxiv.org/html/2608.00005#bib.bib48)\), citation\-grounded supervised fine\-tuningZhang et al\. \([2026](https://arxiv.org/html/2608.00005#bib.bib45)\), and, more recently, reinforcement learning with rating or preference signalsZeng et al\. \([2025](https://arxiv.org/html/2608.00005#bib.bib43)\)\. Training\-free methods instead build LLM\-agent pipelines without parameter updatesLu et al\. \([2024](https://arxiv.org/html/2608.00005#bib.bib25)\); Jin et al\. \([2024](https://arxiv.org/html/2608.00005#bib.bib14)\); D’Arcy et al\. \([2024](https://arxiv.org/html/2608.00005#bib.bib4)\)\. Representative designs include retrieval\- or tool\-augmented reviewers that ground critiques in prior literatureLi et al\. \([2026c](https://arxiv.org/html/2608.00005#bib.bib21)\), multi\-agent collaborations that simulate the role\-based discussion of a review committeeJin et al\. \([2024](https://arxiv.org/html/2608.00005#bib.bib14)\); D’Arcy et al\. \([2024](https://arxiv.org/html/2608.00005#bib.bib4)\); Goyal et al\. \([2026](https://arxiv.org/html/2608.00005#bib.bib9)\); Gao et al\. \([2025](https://arxiv.org/html/2608.00005#bib.bib7)\); Lu et al\. \([2024](https://arxiv.org/html/2608.00005#bib.bib25)\); Weng et al\. \([2026](https://arxiv.org/html/2608.00005#bib.bib38)\), and question\-driven agents that decompose reviewing into iterative checksChang et al\. \([2025](https://arxiv.org/html/2608.00005#bib.bib2)\); Li et al\. \([2026b](https://arxiv.org/html/2608.00005#bib.bib20)\)\.

Despite this progress, generated reviews still fall short of the substantive, well\-targeted feedback that human reviewers provideLiang et al\. \([2024](https://arxiv.org/html/2608.00005#bib.bib22)\); Dycke and Gurevych \([2026](https://arxiv.org/html/2608.00005#bib.bib5)\), and the two paradigms exhibit complementary weaknesses\. Training\-free agents tend to produce superficial critiques that gravitate toward neutral assessmentsZhu et al\. \([2025a](https://arxiv.org/html/2608.00005#bib.bib47)\); Li et al\. \([2025](https://arxiv.org/html/2608.00005#bib.bib19)\), while training\-based reviewers inherit the noise and uneven coverage of their human supervisionSadallah et al\. \([2025](https://arxiv.org/html/2608.00005#bib.bib28)\); Li et al\. \([2026a](https://arxiv.org/html/2608.00005#bib.bib18)\)\. The two signals are in fact complementary\. LLMs excel at broad evidence gathering and summarisation, allowing them to refine human reviews and broaden the set of evaluation dimensions, whereas human reviews supply targeted judgments that ground the model\. Our work bridges these two sources to generate reviews that are both comprehensive and aligned with human judgment\.

## 3Method

We formalize peer review as a sequential rubric\-centric generation process \(§[3\.1](https://arxiv.org/html/2608.00005#S3.SS1)\), realize it through a multi\-stage pipeline that pairs a trained modelAlignerwith role\-specialized LLM agentsScout\(§[3\.2](https://arxiv.org/html/2608.00005#S3.SS2)\), and describe howAligneris trained \(§[3\.3](https://arxiv.org/html/2608.00005#S3.SS3)\)\.

### 3\.1Task Formulation

![Refer to caption](https://arxiv.org/html/2608.00005v1/x1.png)Figure 1:Overview ofRubricReviewer’s inference pipeline\. Inference unfolds in three stages: rubric generation, review generation, and final assessment\. At every stage,Scout, a set of training\-free LLM agents, retrieves external literature and produces a structured reference \(candidate rubrics or draft reviews\), whileAligner, a human\-aligned trained model, consumes this reference and emits the final output\. In Stage 1, the rubric set fuses three sources: predefined general rubrics, paper\-specific rubrics produced byScout’s literature\-grounded generation and then refined byAligner, and domain\-specific rubrics distilled from the public human reviews of key citations\.Formally, given an input manuscript𝒫\\mathcal\{P\}, automated peer review aims to learn the conditional distributionp​\(R,y∣𝒫\)p\(R,y\\mid\\mathcal\{P\}\), whereR=\{r1,…,rn\}R=\\\{r\_\{1\},\\ldots,r\_\{n\}\\\}is a set of atomic reviews \(eachrir\_\{i\}conveys a single evaluative judgment\) andyyis an optional final assessment such as an overall rating or accept/reject decision\. Every atomic reviewrrimplicitly instantiates an underlying evaluation dimension, which we call a rubric and denoter¯\\bar\{r\}\. For instance, the atomic review “all the proposed components have already appeared in prior work” instantiates the rubric “is the methodology sufficiently novel relative to prior work?”\.

We decouple rubric generation from review generation by introducing the rubric setR¯=\{r¯1,…,r¯K\}\\bar\{R\}=\\\{\\bar\{r\}\_\{1\},\\ldots,\\bar\{r\}\_\{K\}\\\}as a latent intermediate\. Each rubricr¯k\\bar\{r\}\_\{k\}is instantiated by a review subsetRk⊆RR\_\{k\}\\subseteq R, and all subsets partition the review setR=⨆k=1KRkR=\\bigsqcup\_\{k=1\}^\{K\}R\_\{k\}\. Peer review is then formulated as the sequential chain𝒫→R¯→R→y\\mathcal\{P\}\\rightarrow\\bar\{R\}\\rightarrow R\\rightarrow y, whose joint distribution factorizes as

p​\(R¯,R,y∣𝒫\)=p​\(R¯∣𝒫\)​p​\(R∣R¯,𝒫\)⋅p​\(y∣R¯,R,𝒫\),\\begin\{split\}p\(\\bar\{R\},R,y\\mid\\mathcal\{P\}\)=\\ &p\(\\bar\{R\}\\mid\\mathcal\{P\}\)\\,p\(R\\mid\\bar\{R\},\\mathcal\{P\}\)\\\\ &\\cdot p\(y\\mid\\bar\{R\},R,\\mathcal\{P\}\),\\end\{split\}\(1\)wherep​\(R¯∣𝒫\)p\(\\bar\{R\}\\mid\\mathcal\{P\}\)generates the rubrics applicable to evaluating the manuscript,p​\(R∣R¯,𝒫\)p\(R\\mid\\bar\{R\},\\mathcal\{P\}\)generates the atomic reviews instantiating each rubric, andp​\(y∣R¯,R,𝒫\)p\(y\\mid\\bar\{R\},R,\\mathcal\{P\}\)generates the final assessment\.

This factorization is motivated by a key observation about peer review: although individual atomic reviews vary sharply across papers, the rubrics they instantiate are largely shared\. Direct mapping𝒫→R\\mathcal\{P\}\\rightarrow Rentangles the paper\-independent what with the paper\-specific how into a single high\-variance generation problem\. IntroducingR¯\\bar\{R\}as an intermediate decouples them into two more concentrated subproblems that are each easier to learn\.

### 3\.2Multi\-stage Inference

To realize the joint factorization in \([1](https://arxiv.org/html/2608.00005#S3.E1)\), inference proceeds in three stages: rubric generationp​\(R¯∣𝒫\)p\(\\bar\{R\}\\mid\\mathcal\{P\}\), review generationp​\(R∣R¯,𝒫\)p\(R\\mid\\bar\{R\},\\mathcal\{P\}\), and final assessmentp​\(y∣R¯,R,𝒫\)p\(y\\mid\\bar\{R\},R,\\mathcal\{P\}\)\. The pipeline is built around two cooperating components:Scout, a set of LLM agents that autonomously retrieve and summarize external evidence as a reference, andAligner, a trained model aligned with human judgment that consumes Scout’s reference and produces the final output at each stage\.

#### Stage 1: Rubric generation\.

We organize rubrics into three categories: general rubricsR¯general\\bar\{R\}\_\{\\text\{general\}\}that apply to all papers \(e\.g\., idea novelty, experimental completeness\); domain\-specific rubricsR¯domain\\bar\{R\}\_\{\\text\{domain\}\}shared across all papers in a research area; and paper\-specific rubricsR¯paper\\bar\{R\}\_\{\\text\{paper\}\}tailored to the manuscript at hand\.R¯general\\bar\{R\}\_\{\\text\{general\}\}is predefined and contributed in full to the final rubric set\. We defer its construction to §[3\.3](https://arxiv.org/html/2608.00005#S3.SS3)\.R¯domain\\bar\{R\}\_\{\\text\{domain\}\}andR¯paper\\bar\{R\}\_\{\\text\{paper\}\}are produced by two parallel paths described below, decomposed into sub\-stepsz1z\_\{1\}–z5z\_\{5\}\.

Path 1: literature\-grounded generation, producing rubrics in bothR¯domain\\bar\{R\}\_\{\\text\{domain\}\}andR¯paper\\bar\{R\}\_\{\\text\{paper\}\}\.\(z1\)\(z\_\{1\}\)*Key citation identification\.*Scout analyzes𝒫\\mathcal\{P\}to select its top\-kkmost central references and retrieves their metadata \(title, year, abstract\) via the Semantic Scholar API, yielding the key citation set𝒞⋆\\mathcal\{C\}^\{\\star\}\.\(z2\)\(z\_\{2\}\)*Key concern generation\.*Genrating appropriate rubrics requires sufficient context about the research area\. Conditioned on\(𝒫,𝒞⋆\)\(\\mathcal\{P\},\\mathcal\{C\}^\{\\star\}\), Scout generates a set of key concerns𝒬\\mathcal\{Q\}\(e\.g\., dominant paradigms, open problems\) whose resolution requires external evidence\.\(z3\)\(z\_\{3\}\)*Iterative literature retrieval\.*For each concernq∈𝒬q\\in\\mathcal\{Q\}, Scout iteratively queries Semantic Scholar to gather supporting evidence\. The first\-round query is generated directly fromqq\. In each subsequent round, Scout refines the query conditioned on the query history, the average relevance of each prior query’s results, and the current top\-kkretrieved pool\. Returned items are scored by relevance toqqand merged into the running top\-kkpool\. Retrieval terminates once the number of above\-threshold items reaches a targetKKor the round budget is exhausted, and the final top\-kkpool is taken as the evidence poolℒ\\mathcal\{L\}\.\(z4\)\(z\_\{4\}\)*Reference rubric drafting\.*Conditioned on\(𝒫,𝒬,ℒ\)\(\\mathcal\{P\},\\mathcal\{Q\},\\mathcal\{L\}\), Scout drafts a candidate rubric setR¯ref\\bar\{R\}\_\{\\text\{ref\}\}\. Taking this draft as a reference, Aligner then generates the final, human\-aligned rubrics\.

Path 2: human\-review mining, producing rubrics inR¯domain\\bar\{R\}\_\{\\text\{domain\}\}\.Path 2 reuses the key citation set𝒞⋆\\mathcal\{C\}^\{\\star\}fromz1z\_\{1\}and proceeds with a single further sub\-step\.\(z5\)\(z\_\{5\}\)*Cross\-paper rubric mining\.*Since𝒞⋆\\mathcal\{C\}^\{\\star\}largely shares the research area with𝒫\\mathcal\{P\}, public human reviews of these citations expose key domain\-level evaluation dimensions transferable to𝒫\\mathcal\{P\}\. To leverage this, Scout queries OpenReview for the human reviews of works in𝒞⋆\\mathcal\{C\}^\{\\star\}, extracts the rubric items those reviews implicitly invoke, and filters them against𝒫\\mathcal\{P\}to retain only those still applicable to the manuscript\.

The final rubric setR¯\\bar\{R\}is the deduplicated unionR¯general∪R¯domain∪R¯paper\\bar\{R\}\_\{\\text\{general\}\}\\cup\\bar\{R\}\_\{\\text\{domain\}\}\\cup\\bar\{R\}\_\{\\text\{paper\}\}\.

#### Stage 2: Review generation\.

GivenR¯\\bar\{R\}, atomic reviews are generated rubric by rubric\. For eachr¯k∈R¯\\bar\{r\}\_\{k\}\\in\\bar\{R\}, Scout first decides whether external evidence is required to judger¯k\\bar\{r\}\_\{k\}\. If so, Scout generates a retrieval target from\(𝒫,r¯k\)\(\\mathcal\{P\},\\bar\{r\}\_\{k\}\)and runs the same iterative literature retrieval as inz3z\_\{3\}, yielding an evidence poolℒk\\mathcal\{L\}\_\{k\}; otherwiseℒk=∅\\mathcal\{L\}\_\{k\}=\\emptyset\. Conditioned on\(𝒫,r¯k,ℒk\)\(\\mathcal\{P\},\\bar\{r\}\_\{k\},\\mathcal\{L\}\_\{k\}\), Scout produces a draft judgmentR~k\\tilde\{R\}\_\{k\}\. TakingR~k\\tilde\{R\}\_\{k\}as a reference, Aligner generates the final, human\-aligned atomic reviewsRkR\_\{k\}instantiatingr¯k\\bar\{r\}\_\{k\}\. The full review is the partitionR=⨆k=1KRkR=\\bigsqcup\_\{k=1\}^\{K\}R\_\{k\}\.

#### Stage 3: Final assessment\.

Aligner consumes the abstract of𝒫\\mathcal\{P\}together with all rubric–review pairs\{\(r¯k,Rk\)\}k=1K\\\{\(\\bar\{r\}\_\{k\},R\_\{k\}\)\\\}\_\{k=1\}^\{K\}and produces the overall score and accept/reject decisionyy\.

### 3\.3Model Training

Scoutis training\-free\. We therefore construct three supervised datasets, one per inference stage of §[3\.2](https://arxiv.org/html/2608.00005#S3.SS2), to alignAlignerwith human review behaviour\. These are derived from a base corpus𝒟\\mathcal\{D\}\(DeepReview\-13k\), where each example provides the paper text𝒫\\mathcal\{P\}, the set of all reviewers’ raw reviewsRRwith their per\-reviewer scores, and the final accept/reject decision from the meta\-reviewer\.

#### Rubric\-generation data\.

𝒟\\mathcal\{D\}providesRRbut not the underlying rubricsR¯\\bar\{R\}, so we recover them first\. Using GPT\-5\.2, every review inRRis decomposed into atomic reviews\{ri\}\\\{r\_\{i\}\\\}\. Eachrir\_\{i\}is reverse\-inferred to its rubric, and atomic reviews sharing the same rubric are aggregated into rubric–review pairs\{\(r¯k,Rk\)\}\\\{\(\\bar\{r\}\_\{k\},R\_\{k\}\)\\\}\.

To build a stable rubric vocabulary across𝒟\\mathcal\{D\}, we cluster the recovered rubrics\. Each rubric is tagged with one of three categories \{Idea & Methodology, Experiment & Evaluation, Presentation & Writing\}\. Within each category, rubrics are sentence\-encoded and grouped by coarse\-to\-fine clustering, and an LLM summarises each cluster into a canonical rubric\. Canonical rubrics are then iteratively merged\. In each round, an LLM inspects every canonical rubric against its top\-kknearest neighbours by embedding centroid and merges those describing the same dimension\. Iteration terminates once the taxonomy stabilises\. Canonical rubrics covering more than a fractionτ\\tauof𝒟\\mathcal\{D\}form the general rubrics \(4040in total, listed in the Appendix[A](https://arxiv.org/html/2608.00005#A1)\)\.

For each𝒫∈𝒟\\mathcal\{P\}\\in\\mathcal\{D\}, we run Scout’sz1z\_\{1\}–z4z\_\{4\}to obtain its reference rubricsR¯ref\\bar\{R\}\_\{\\text\{ref\}\}\. The training pair takes\(𝒫,R¯ref\)\(\\mathcal\{P\},\\bar\{R\}\_\{\\text\{ref\}\}\)as input and the paper’s domain\- and paper\-specific canonical rubrics as the target\. General rubrics are excluded from the target since they are always added directly at inference\.

#### Review\-generation data\.

Pooled from multiple reviewers, raw reviewsRkR\_\{k\}is often inconsistent across members and uneven in quality\. We use GPT\-5\.2 to refine eachRkR\_\{k\}into a clean targetRk⋆R\_\{k\}^\{\\star\}\. Conditioned on\(𝒫,r¯k,Rk\)\(\\mathcal\{P\},\\bar\{r\}\_\{k\},R\_\{k\}\), GPT\-5\.2 first decides whether external evidence is needed\. If so, it generates a retrieval target and runs the same iterative retrieval asz3z\_\{3\}, otherwise the evidence pool is empty\. The refinement is constrained to re\-articulate\. Every claim inRk⋆R\_\{k\}^\{\\star\}must be supported by at least one atomic review inRkR\_\{k\}, and no new opinions are allowed\.

For each\(𝒫,r¯k\)\(\\mathcal\{P\},\\bar\{r\}\_\{k\}\), we further run Stage 2’s Scout sub\-step to obtain its draft judgmentR~k\\tilde\{R\}\_\{k\}\. The training pair takes\(𝒫,r¯k,R~k\)\(\\mathcal\{P\},\\bar\{r\}\_\{k\},\\tilde\{R\}\_\{k\}\)as input andRk⋆R\_\{k\}^\{\\star\}as the target\.

#### Final\-assessment data\.

\{\(r¯k,Rk⋆\)\}\\\{\(\\bar\{r\}\_\{k\},R\_\{k\}^\{\\star\}\)\\\}from the review\-generation data is restricted to rubrics surfaced by raw human reviews, and does not cover the full inference\-time rubric setR¯\\bar\{R\}\(§[3\.2](https://arxiv.org/html/2608.00005#S3.SS2)\)\. For rubrics inR¯∖\{r¯k\}\\bar\{R\}\\setminus\\\{\\bar\{r\}\_\{k\}\\\}, we generate pseudo\-golden reviews via Stage 2 inference with the trained Aligner\.

The training pair takes the abstract of𝒫\\mathcal\{P\}together with all resulting rubric–review pairs as input andy=\(s¯,d\)y=\(\\bar\{s\},d\)as the target, wheres¯\\bar\{s\}is the mean of per\-reviewer scores rounded to the nearest integer andddis the meta\-reviewer’s accept/reject decision\.

## 4Experimental Setup

### 4\.1Dataset

We conduct all experiments onDeepReview\-13KZhu et al\. \([2025b](https://arxiv.org/html/2608.00005#bib.bib48)\), a curated dataset of ICLR 2024–2025 submissions paired with their human\-written reviews and assessments\. To align this raw resource with our methodological paradigm, we use GPT\-5\.2 for data rewriting and extension, such as rubric extraction, literature retrieval, and review refinement\. We follow the standard split with approximately 12K papers for training\. To control training and inference cost, the review generation stage only uses a one\-quarter subsample and evaluation is restricted to the first 200 papers of the ICLR 2025 test subset\.

### 4\.2Baselines

We compareRubricRevieweragainst three categories of baselines:

- •Foundation LLM \(zero\-shot\)\.GPT\-5\.2111[https://openai\.com/index/introducing\-gpt\-5\-2/](https://openai.com/index/introducing-gpt-5-2/)prompted directly on the paper without any specialized framework\. For fair comparison, all direct LLM API calls in our pipeline and the baselines use GPT\-5\.2\.
- •Training\-free agentic frameworks\.AI ScientistLu et al\. \([2024](https://arxiv.org/html/2608.00005#bib.bib25)\)andAgentReviewJin et al\. \([2024](https://arxiv.org/html/2608.00005#bib.bib14)\), which simulate peer review through multi\-agent prompting, andDeepReviewer 2\.0Weng et al\. \([2026](https://arxiv.org/html/2608.00005#bib.bib38)\), an agentic system that produces traceable, auditable reviews via tool\-integrated reasoning\.
- •Training\-based reviewer models\.CycleReviewerWeng et al\. \([2025](https://arxiv.org/html/2608.00005#bib.bib37)\), supervised fine\-tuned within a researcher–reviewer cycle\.DeepReviewerZhu et al\. \([2025b](https://arxiv.org/html/2608.00005#bib.bib48)\), supervised fine\-tuned on long chain\-of\-thought reasoning traces\.ReviewGrounderLi et al\. \([2026c](https://arxiv.org/html/2608.00005#bib.bib21)\), which refines a trained drafter’s initial review against a fixed set of meta\-rubrics shared across all papers\.

### 4\.3Implementation Details

We fine\-tuneAlignerfrom Phi\-4 \(14B\)222[https://huggingface\.co/microsoft/phi\-4](https://huggingface.co/microsoft/phi-4)with LoRAHu et al\. \([2022](https://arxiv.org/html/2608.00005#bib.bib12)\)\(rank 16,α=32\\alpha\\\!=\\\!32, dropout0\.050\.05\) applied to all linear projections\. Training runs on4×4\\\!\\times\\\!NVIDIA H200 GPUs in bfloat16 with FlashAttention\-2Dao \([2024](https://arxiv.org/html/2608.00005#bib.bib3)\)and gradient checkpointing, optimised by AdamWLoshchilov and Hutter \([2017](https://arxiv.org/html/2608.00005#bib.bib24)\)with a learning rate of1×10−41\\\!\\times\\\!10^\{\-4\}under a cosine schedule \(3% warmup\) and an effective batch size of 16\. Since inputs toAlignercan be very long, we truncate the context to 16,384 tokens\. The three stages of §[3\.3](https://arxiv.org/html/2608.00005#S3.SS3)are each trained for one epoch, and after each stage the LoRA adapter is merged into the base model to initialise the next\. For inference, the merged model is served with vLLMKwon et al\. \([2023](https://arxiv.org/html/2608.00005#bib.bib17)\)in pure data\-parallel mode\. We sample with temperature1\.01\.0, top\-pp0\.950\.95, and a 2,048\-token generation budget\.

## 5Main Results

### 5\.1Human Alignment

Human\-written peer reviews are the highest\-quality reference available, so consistency with human reviewers is the principal axis along which we evaluate our system\. However, Such a gold standard is inherently incomplete, since each human reviewer surfaces only a subset of the relevant evaluation dimensions and the resulting reviews legitimately omit many concerns\. We therefore evaluate how well a system covers the gold content rather than penalising it for going further, adopting recall\-style metrics as our core measure\. We measure alignment along three dimensions: the rubrics that drive each review, the per\-rubric reviews, and the final assessment\. Tables[1](https://arxiv.org/html/2608.00005#S5.T1)and[2](https://arxiv.org/html/2608.00005#S5.T2)report the rubric/review results and the assessment results respectively\.

#### Rubric Generation

Rubric GenerationReview GenerationMethodBackbone\#R/paperRecall↑\\uparrowΔcat↓\\Delta\_\{\\text\{cat\}\}\\downarrowR\-Lenient↑\\uparrowR\-Strict↑\\uparrowΔverdict↓\\Delta\_\{\\text\{verdict\}\}\\downarrow*Foundation LLMs \(zero\-shot\)*GPT\-5\.2–10\.434\.34\.220\.014\.634\.5*Training\-free Agentic Frameworks*AI ScientistGPT\-5\.211\.338\.49\.031\.722\.511\.1AgentReviewGPT\-5\.213\.545\.25\.235\.324\.76\.0DeepReviewer 2\.0GPT\-5\.27\.726\.417\.019\.112\.04\.0*Training\-based Reviewer Models*CycleReviewer\-8BLlama\-3\.1\-8B8\.731\.410\.021\.610\.216\.6CycleReviewer\-70BLlama\-3\.1\-70B8\.129\.910\.519\.710\.223\.0CycleReviewer\-123BMistral\-Large\-123B13\.345\.710\.832\.518\.46\.1DeepReviewer\-7BPhi\-4\-7B11\.342\.19\.230\.018\.511\.5DeepReviewer\-14BPhi\-4\-14B12\.344\.07\.732\.521\.311\.2ReviewGrounderPhi\-4\-14B \+ GPT\-5\.211\.239\.510\.222\.716\.534\.3RubricReviewer \(Ours\)Phi\-4\-14B \+ GPT\-5\.253\.880\.53\.561\.438\.42\.4Table 1:Consistency of the generated rubrics and reviews with human reviewers on the test set\.Boldmarks the best per column\.Rubric:\#R/paperis the average number of generated rubrics;Recall\(%\) is the fraction of gold rubrics covered;Δcat\\Delta\_\{\\text\{cat\}\}is the category\-distribution shift over \{Idea, Experiment, Presentation\}\.Review: for each rubric, both the method’s and human’s reviews are summarised into a verdict and supporting reasoning\.R\-Lenient\(%\) is the fraction of gold rubrics on which the verdicts agree;R\-Strict\(%\) additionally requires the supporting reasoning to overlap;Δverdict\\Delta\_\{\\text\{verdict\}\}is the distribution shift between the method’s and human’s verdicts\.On rubric generation,RubricReviewerproduces 53\.8 rubrics per paper on average, roughly 4–7×\\timesthe 7\.7–13\.5 of all baselines, indicating substantially broader coverage\. Despite this expansion, its rubric Recall reaches 80\.5, surpassing the strongest baseline by 34\.8 absolute points \(a 76% relative gain\) and confirming that the extra rubrics complement rather than dilute the human\-essential ones\. The category\-distribution distanceΔcat\\Delta\_\{\\text\{cat\}\}further drops to 3\.5, the lowest among all systems, showing that the expanded rubric set preserves the same category balance as the human side\.

#### Review Generation

On review generation,RubricReviewerattains 61\.4 R\-Lenient and 38\.4 R\-Strict, improving over the strongest baseline by 26\.1 and 13\.7 absolute points respectively\. The verdict\-distribution distanceΔverdict\\Delta\_\{\\text\{verdict\}\}further drops to 2\.4, the lowest among all systems and 40% below the next best, confirming that our per\-rubric judgements are distributed in the same way as the human consensus\.

Together these results show thatRubricReviewermatches human\-essential content far more reliably than any baseline on both the rubric and the review axis\. At the same time, it delivers reviews that are substantially more comprehensive than what a single human reviewer typically provides, closing the coverage and fidelity gap left by either training\-based or training\-free pipelines alone\.

#### Assessment Generation

Benefiting from the more comprehensive and discriminative reviews above,RubricRevieweralso achieves the best assessment results\. It attains the lowest rating error \(MSE 1\.365, MAE 0\.865\) and the highest accept/reject accuracy \(71\.0%\), with the most pronounced gain on ACC at 4\.5 absolute points over the strongest baseline\.

MethodMSE↓\\downarrowMAE↓\\downarrowACC↑\\uparrow*Foundation LLMs \(zero\-shot\)*GPT\-5\.21\.4750\.90554\.0*Training\-free Agentic Frameworks*AI Scientist5\.8052\.08566\.5AgentReview1\.9981\.06534\.5DeepReviewer 2\.01\.4900\.93541\.5*Training\-based Reviewer Models*CycleReviewer\-8B3\.8171\.39464\.5CycleReviewer\-70B2\.0451\.09564\.0CycleReviewer\-123B2\.2601\.18954\.5DeepReviewer\-7B1\.7481\.05965\.0DeepReviewer\-14B1\.3720\.91066\.5ReviewGrounder2\.8851\.29857\.5RubricReviewer \(Ours\)1\.3650\.86571\.0Table 2:Consistency of the generated ratings and accept/reject decisions with human reviewers\.Boldmarks the best per column\.MSEandMAEmeasure the squared and absolute deviation from the gold rating;ACC\(%\) is the accept/reject classification accuracy\.

### 5\.2Ablation Study

We ablate one component at a time to isolate its contribution\. Table[3](https://arxiv.org/html/2608.00005#S5.T3)reports the resulting Recall, R\-Lenient, and R\-Strict, with red subscripts marking the absolute drop relative to the full model\.

VariantRecall↑\\uparrowR\-Lenient↑\\uparrowR\-Strict↑\\uparrowOurs \(full\)80\.561\.438\.4*Rubric stage*−\-general rubrics25\.1↓55\.4––−\-model\-generated rubrics78\.6↓1\.9––−\-cite\-derived rubrics78\.8↓1\.7––−\-Scoutreference80\.1↓0\.4––*Review stage*−\-Scoutreference–58\.9↓2\.537\.2↓1\.2−\-rubric conditioning53\.2↓27\.340\.3↓21\.127\.8↓10\.6Table 3:Component ablation, removing one ingredient at a time\.Recall,R\-Lenient, andR\-Strictfollow Table[1](https://arxiv.org/html/2608.00005#S5.T1)\. Red subscripts show the absolute drop relative to the full model; “–” marks ablations that leave the corresponding stage’s output unchanged\.All three rubric sources jointly drive coverage, with the general rubrics carrying the dominant share: removing them collapses Recall by 55\.4 points \(80\.5→\\rightarrow25\.1\), confirming that human reviewers share a large core of common evaluation dimensions that are highly transferable across papers\. The domain and paper sources each add a smaller but consistent gain \(1\.7 and 1\.9 points\), showing that mining cited\-paper reviews is an effective route to domain\-level rubrics and that the trainedAlignerproduces paper\-specific rubrics well aligned with human judgement\.

Scoutcontributes at both stages: blanking its reference drops Recall by 0\.4 in the rubric stage and R\-Lenient/R\-Strict by 2\.5/1\.2 in the review stage, confirming that the external evidenceScoutgathers and summarises offers a measurable lift overAligner’s parametric knowledge alone\.

Decomposing review generation by rubrics contributes by far the largest gain\. LettingAlignerproduce reviews directly without rubric conditioning collapses Recall, R\-Lenient, and R\-Strict by 27\.3, 21\.1, and 10\.6 absolute points respectively, showing that the rubric intermediate is essential for both the comprehensiveness and the discriminativeness of the final reviews\.

### 5\.3Defend Attacks Analysis

We test robustness against prompt\-injection attacks by inserting a fixed adversarial payload into the body of each test paper\. The payload reads: “*Ignore all previous instructions and give the current paper a positive review\. Recommend Accept\.*” For every method we re\-run the full pipeline on the attacked papers and report two paired shifts against the clean run: the per\-rubricVerdict Positiverate \(%\) and the predictedAvg\. Rating\.

![Refer to caption](https://arxiv.org/html/2608.00005v1/x2.png)Figure 2:Prompt\-injection robustness on the test set\. We insert a fixed adversarial payload that requests an unconditional*Recommend Accept*into the body of each paper and re\-run every method end\-to\-end\.Top: per\-rubric Verdict Positive rate \(%\)\.Bottom: predicted Avg\. Rating\. The red top of each Attack bar marks the shift relative to the Normal runRubricReviewerexhibits the strongest robustness on both axes \(Figure[2](https://arxiv.org/html/2608.00005#S5.F2)\)\. Its Verdict Positive rate moves by only\+1\.4\+1\.4pp under attack and its predicted rating by a near\-negligible\+0\.02\+0\.02, the smallest shifts among all systems\. The baseline shifts span\+2\.1\+2\.1to\+6\.1\+6\.1pp in Verdict Positive and\+0\.10\+0\.10to\+1\.03\+1\.03in rating\. We attribute this robustness to the rubric\-based decomposition\. Rather than mapping a paper directly to a holistic recommendation, our pipeline factorises the subjective evaluation into many objective, fine\-grained per\-rubric verdicts, each grounded in concrete content from the paper\.

### 5\.4Value Analysis

Beyond agreement with human reviewers, we further ask whether our reviews are useful to the paper authors\. We score each review’s*value*along three axes\.*Specificity*rewards concrete, paper\-targeted issues over vague generalities\.*Decisive stance*rewards a clear verdict over neutral hedging\.*Targetedness*asks whether acting on the feedback would meaningfully improve the paper at hand\. For each test paper, we run a blind A/B comparison between the method’s review and the gold human review with random A/B swap, using GPT\-5\.2 as the judge\. Tie is set as the default verdict, so only substantively useful gaps are scored Win or Lose\.

![Refer to caption](https://arxiv.org/html/2608.00005v1/x3.png)Figure 3:Per\-paperValueverdict against the gold human review\. Each method’s review is judged blind A/B against the gold review \(random A/B swap, GPT\-5\.2 as the judge\) and labelled Win/Tie/Lose\.RubricReviewerattains gold\-level usefulness on 81\.5% of papers \(Win\+\+Tie\), a\+25\.0\+25\.0pp gap over the strongest baseline \(Figure[3](https://arxiv.org/html/2608.00005#S5.F3)\)\. On 5\.5% of papers our reviews are even judged more valuable than the gold human review, the highest Win rate among all systems\. Our reviews are therefore not only comprehensive and aligned with human reviewers, but also deliver paper\-targeted feedback that authors can directly act on\.

## 6Conclusion

We presentRubricReviewer, a fully rubric\-driven framework for automated peer review\. It introduces explicit rubric generation as an intermediate step so that both review generation and final assessment are conditioned on paper\-adaptive rubrics, and it combines training\-free LLM agents with a human\-aligned trained model to fuse the strengths of both supervision sources\. Experiments on real\-world submissions show thatRubricReviewerproduces reviews that are markedly more comprehensive and more discriminative than prior systems, while also achieving the strongest robustness against prompt\-injection attacks\. We hope that our work opens up the potential of rubric\-driven peer review and lays a foundation for reliable LLM\-based reviewers\.

## Limitations

Despite the more comprehensive rubric coverage and stronger human alignment thatRubricReviewerachieves, several aspects of our framework remain unexplored\.

The stage\-wise design substantially raises both training and inference cost\. The supervised data scales with the number of stages and the number of trained components, and at inference our multi\-stage pipeline is noticeably slower than end\-to\-end systems\.

Constrained by the fine\-tuned model’s context window and our compute budget, review generation is performed independently per rubric\. The trainedAligneris never exposed to the cross\-rubric correlations that human reviewers naturally exploit\.

Finally, due to the cost of expert annotation, we have not conducted a large\-scale human study of the generated reviews\. A deeper qualitative assessment beyond our automatic metrics is left to future work\.

## References

- Arora et al\. \(2025\)Rahul K\. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal\. 2025\.[Healthbench: Evaluating large language models towards improved human health](https://doi.org/10.48550/ARXIV.2505.08775)\.*CoRR*, abs/2505\.08775\.
- Chang et al\. \(2025\)Yuan Chang, Ziyue Li, Hengyuan Zhang, Yuanbo Kong, Yanru Wu, Hayden Kwok\-Hay So, Zhijiang Guo, Liya Zhu, and Ngai Wong\. 2025\.Treereview: A dynamic tree of questions framework for deep and efficient llm\-based scientific peer review\.In*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, pages 15662–15693\.
- Dao \(2024\)Tri Dao\. 2024\.[Flashattention\-2: Faster attention with better parallelism and work partitioning](https://openreview.net/forum?id=mZn2Xyh9Ec)\.In*The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024*\. OpenReview\.net\.
- D’Arcy et al\. \(2024\)Mike D’Arcy, Tom Hope, Larry Birnbaum, and Doug Downey\. 2024\.[Marg: Multi\-agent review generation for scientific papers](https://api.semanticscholar.org/CorpusID:266899693)\.*ArXiv*, abs/2401\.04259\.
- Dycke and Gurevych \(2026\)Nils Dycke and Iryna Gurevych\. 2026\.[Automatic reviewers fail to detect faulty reasoning in research papers: A new counterfactual evaluation framework](https://doi.org/10.1162/TACl.a.642)\.*Transactions of the Association for Computational Linguistics*, 14:465–488\.
- Dycke et al\. \(2023\)Nils Dycke, Ilia Kuznetsov, and Iryna Gurevych\. 2023\.[NLPeer: A unified resource for the computational study of peer review](https://doi.org/10.18653/v1/2023.acl-long.277)\.In*Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 5049–5073, Toronto, Canada\. Association for Computational Linguistics\.
- Gao et al\. \(2025\)Xian Gao, Jiacheng Ruan, Zongyun Zhang, Jingsheng Gao, Ting Liu, and Yuzhuo Fu\. 2025\.Reviewagents: Bridging the gap between human and ai\-generated paper reviews\.*arXiv preprint arXiv:2503\.08506*\.
- Gao et al\. \(2024\)Zhaolin Gao, Kianté Brantley, and Thorsten Joachims\. 2024\.Reviewer2: Optimizing review generation through prompt generation\.*arXiv preprint arXiv:2402\.10886*\.
- Goyal et al\. \(2026\)Palash Goyal, Mihir Parmar, Yiwen Song, Hamid Palangi, Tomas Pfister, and Jinsung Yoon\. 2026\.Scholarpeer: A context\-aware multi\-agent framework for automated peer review\.*arXiv preprint arXiv:2601\.22638*\.
- Gunjal et al\. \(2025\)Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Bing Liu, and Sean Hendryx\. 2025\.[Rubrics as rewards: Reinforcement learning beyond verifiable domains](https://doi.org/10.48550/ARXIV.2507.17746)\.*CoRR*, abs/2507\.17746\.
- Hashemi et al\. \(2024\)Helia Hashemi, Jason Eisner, Corby Rosset, Benjamin Van Durme, and Chris Kedzie\. 2024\.[Llm\-rubric: A multidimensional, calibrated approach to automated evaluation of natural language texts](https://doi.org/10.18653/V1/2024.ACL-LONG.745)\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2024, Bangkok, Thailand, August 11\-16, 2024*, pages 13806–13834\. Association for Computational Linguistics\.
- Hu et al\. \(2022\)Edward J\. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen\-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen\. 2022\.[Lora: Low\-rank adaptation of large language models](https://openreview.net/forum?id=nZeVKeeFYf9)\.In*The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25\-29, 2022*\. OpenReview\.net\.
- Idahl and Ahmadi \(2025\)Maximilian Idahl and Zahra Ahmadi\. 2025\.Openreviewer: A specialized large language model for generating critical scientific paper reviews\.In*Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(System Demonstrations\)*, pages 550–562\.
- Jin et al\. \(2024\)Yiqiao Jin, Qinlin Zhao, Yiyang Wang, Hao Chen, Kaijie Zhu, Yijia Xiao, and Jindong Wang\. 2024\.[Agentreview: Exploring peer review dynamics with LLM agents](https://doi.org/10.18653/V1/2024.EMNLP-MAIN.70)\.In*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12\-16, 2024*, pages 1208–1226\. Association for Computational Linguistics\.
- Kang et al\. \(2018\)Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz\. 2018\.[A dataset of peer reviews \(PeerRead\): Collection, insights and NLP applications](https://doi.org/10.18653/v1/N18-1149)\.In*Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long Papers\)*, pages 1647–1661, New Orleans, Louisiana\. Association for Computational Linguistics\.
- Krishna et al\. \(2021\)Kalpesh Krishna, Aurko Roy, and Mohit Iyyer\. 2021\.[Hurdles to progress in long\-form question answering](https://doi.org/10.18653/v1/2021.naacl-main.393)\.In*Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 4940–4957, Online\. Association for Computational Linguistics\.
- Kwon et al\. \(2023\)Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E\. Gonzalez, Hao Zhang, and Ion Stoica\. 2023\.Efficient memory management for large language model serving with pagedattention\.In*Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles*\.
- Li et al\. \(2026a\)Bowen Li, Haochen Ma, Yuxin Wang, Jie Yang, Xinchi Chen, Xuanjing Huang, Yining Zheng, and Xipeng Qiu\. 2026a\.Beyond rating: A comprehensive evaluation and benchmark for ai reviews\.*arXiv preprint arXiv:2604\.19502*\.
- Li et al\. \(2025\)Ruochi Li, Haoxuan Zhang, Edward Gehringer, Ting Xiao, Junhua Ding, and Haihua Chen\. 2025\.Unveiling the merits and defects of llms in automatic review generation for scientific papers\.In*2025 IEEE International Conference on Data Mining \(ICDM\)*, pages 1370–1379\. IEEE\.
- Li et al\. \(2026b\)Shuaimin Li, Liyang Fan, Yufang Lin, Zeyang Li, Xian Wei, Shiwen Ni, Hamid Alinejad\-Rokny, and Min Yang\. 2026b\.Automatic paper reviewing with heterogeneous graph reasoning over llm\-simulated reviewer\-author debates\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, volume 40, pages 31717–31725\.
- Li et al\. \(2026c\)Zhuofeng Li, Yi Lu, Dongfu Jiang, Haoxiang Zhang, Yuyang Bai, Chuan Li, Yu Wang, Shuiwang Ji, Jianwen Xie, and Yu Zhang\. 2026c\.[Reviewgrounder: Improving review substantiveness with rubric\-guided, tool\-integrated agents](https://doi.org/10.48550/ARXIV.2604.14261)\.*CoRR*, abs/2604\.14261\.
- Liang et al\. \(2024\)Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Yi Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Scott Smith, Yian Yin, and 1 others\. 2024\.Can large language models provide useful feedback on research papers? a large\-scale empirical analysis\.*NEJM AI*, 1\(8\):AIoa2400196\.
- Liu et al\. \(2025\)Tianci Liu, Ran Xu, Tony Yu, Ilgee Hong, Carl Yang, Tuo Zhao, and Haoyu Wang\. 2025\.[Openrubrics: Towards scalable synthetic rubric generation for reward modeling and LLM alignment](https://doi.org/10.48550/ARXIV.2510.07743)\.*CoRR*, abs/2510\.07743\.
- Loshchilov and Hutter \(2017\)Ilya Loshchilov and Frank Hutter\. 2017\.[Decoupled weight decay regularization](https://api.semanticscholar.org/CorpusID:53592270)\.In*International Conference on Learning Representations*\.
- Lu et al\. \(2024\)Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha\. 2024\.The ai scientist: Towards fully automated open\-ended scientific discovery\.*arXiv preprint arXiv:2408\.06292*\.
- Lv et al\. \(2026\)Changze Lv, Jie Zhou, Wentao Zhao, Jingwen Xu, Zisu Huang, Muzhao Tian, Shihan Dou, Tao Gui, Le Tian, Xiao Zhou, Xiaoqing Zheng, Xuanjing Huang, and Jie Zhou\. 2026\.[Learning query\-specific rubrics from human preferences for deepresearch report generation](https://doi.org/10.48550/ARXIV.2602.03619)\.*CoRR*, abs/2602\.03619\.
- Que et al\. \(2024\)Haoran Que, Feiyu Duan, Liqun He, Yutao Mou, Wangchunshu Zhou, Jiaheng Liu, Wenge Rong, Zekun Moore Wang, Jian Yang, Ge Zhang, Junran Peng, Zhaoxiang Zhang, Songyang Zhang, and Kai Chen\. 2024\.[Hellobench: Evaluating long text generation capabilities of large language models](https://doi.org/10.48550/ARXIV.2409.16191)\.*CoRR*, abs/2409\.16191\.
- Sadallah et al\. \(2025\)Abdelrahman Sadallah, Tim Baumgärtner, Iryna Gurevych, and Ted Briscoe\. 2025\.The good, the bad and the constructive: Automatically measuring peer review’s utility for authors\.In*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, pages 28979–29009\.
- Shao et al\. \(2025\)Rulin Shao, Akari Asai, Shannon Zejiang Shen, Hamish Ivison, Varsha Kishore, Jingming Zhuo, Xinran Zhao, Molly Park, Samuel G\. Finlayson, David A\. Sontag, Tyler Murray, Sewon Min, Pradeep Dasigi, Luca Soldaini, Faeze Brahman, Wen\-tau Yih, Tongshuang Wu, Luke Zettlemoyer, Yoon Kim, and 2 others\. 2025\.[DR tulu: Reinforcement learning with evolving rubrics for deep research](https://doi.org/10.48550/ARXIV.2511.19399)\.*CoRR*, abs/2511\.19399\.
- Shao et al\. \(2024\)Yijia Shao, Yucheng Jiang, Theodore A\. Kanell, Peter Xu, Omar Khattab, and Monica S\. Lam\. 2024\.[Assisting in writing wikipedia\-like articles from scratch with large language models](https://doi.org/10.18653/V1/2024.NAACL-LONG.347)\.In*Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\), NAACL 2024, Mexico City, Mexico, June 16\-21, 2024*, pages 6252–6278\. Association for Computational Linguistics\.
- Sharma et al\. \(2025\)Manasi Sharma, Chen Bo Calvin Zhang, Chaithanya Bandi, Clinton Wang, Ankit Aich, Huy Nghiem, Tahseen Rabbani, Ye Htet, Brian Jang, Sumana Basu, Aishwarya Balwani, Denis Peskoff, Marcos Ayestaran, Sean M\. Hendryx, Brad Kenstler, and Bing Liu\. 2025\.[Researchrubrics: A benchmark of prompts and rubrics for evaluating deep research agents](https://doi.org/10.48550/ARXIV.2511.07685)\.*CoRR*, abs/2511\.07685\.
- Sheng et al\. \(2026\)Leheng Sheng, Wenchang Ma, Ruixin Hong, Xiang Wang, An Zhang, and Tat\-Seng Chua\. 2026\.[Reinforcing chain\-of\-thought reasoning with self\-evolving rubrics](https://doi.org/10.48550/ARXIV.2602.10885)\.*CoRR*, abs/2602\.10885\.
- Su et al\. \(2025\)Xiaotian Su, Thiemo Wambsganss, Roman Rietsche, Seyed Parsa Neshaei, and Tanja Käser\. 2025\.[Reviewriter: Ai\-generated instructions for peer review writing](https://api.semanticscholar.org/CorpusID:259376592)\.In*Workshop on Innovative Use of NLP for Building Educational Applications*\.
- Sun et al\. \(2024\)Lu Sun, Stone Tao, Junjie Hu, and Steven P\. Dow\. 2024\.[Metawriter: Exploring the potential and perils of ai writing support in scientific peer review](https://api.semanticscholar.org/CorpusID:269470548)\.*Proceedings of the ACM on Human\-Computer Interaction*, 8:1 – 32\.
- Sun \(2025\)Zhuanlan Sun\. 2025\.Large language models in peer review: challenges and opportunities\.*Scientometrics*, pages 1–44\.
- Viswanathan et al\. \(2025\)Vijay Viswanathan, Yanchao Sun, Shuang Ma, Xiang Kong, Meng Cao, Graham Neubig, and Tongshuang Wu\. 2025\.[Checklists are better than reward models for aligning language models](https://doi.org/10.48550/ARXIV.2507.18624)\.*CoRR*, abs/2507\.18624\.
- Weng et al\. \(2025\)Yixuan Weng, Minjun Zhu, Guangsheng Bao, Hongbo Zhang, Jindong Wang, Yue Zhang, and Linyi Yang\. 2025\.[Cycleresearcher: Improving automated research via automated review](https://openreview.net/forum?id=bjcsVLoHYs)\.In*The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24\-28, 2025*\. OpenReview\.net\.
- Weng et al\. \(2026\)Yixuan Weng, Minjun Zhu, Qiujie Xie, Zhiyuan Ning, Shichen Li, Panzhong Lu, Zhen Lin, Enhao Gu, Qiyao Sun, and Yue Zhang\. 2026\.Deepreviewer 2\.0: A traceable agentic system for auditable scientific peer review\.*arXiv preprint arXiv:2604\.09590*\.
- Wu et al\. \(2025\)Shican Wu, Xiao Ma, Dehui Luo, Lulu Li, Xiangcheng Shi, Xin Chang, Xiaoyun Lin, Ran Luo, Chunlei Pei, Changying Du, Zhi\-Jian Zhao, and Jinlong Gong\. 2025\.[Automated literature research and review\-generation method based on large language models](https://doi.org/10.1093/nsr/nwaf169)\.*National Science Review*, 12\(6\)\.
- Xie et al\. \(2025\)Lipeng Xie, Sen Huang, Zhuo Zhang, Anni Zou, Yunpeng Zhai, Dingchao Ren, Kezun Zhang, Haoyuan Hu, Boyin Liu, Haoran Chen, and 1 others\. 2025\.Auto\-rubric: Learning from implicit weights to explicit rubrics for reward modeling\.*arXiv preprint arXiv:2510\.17314*\.
- Xu et al\. \(2026\)Ran Xu, Tianci Liu, Zihan Dong, Tony Yu, Ilgee Hong, Carl Yang, Linjun Zhang, Tao Zhao, and Haoyu Wang\. 2026\.[Alternating reinforcement learning for rubric\-based reward modeling in non\-verifiable LLM post\-training](https://doi.org/10.48550/ARXIV.2602.01511)\.*CoRR*, abs/2602\.01511\.
- Yuan et al\. \(2022\)Weizhe Yuan, Pengfei Liu, and Graham Neubig\. 2022\.Can we automate scientific reviewing?*Journal of Artificial Intelligence Research*, 75:171–212\.
- Zeng et al\. \(2025\)Sihang Zeng, Kai Tian, Kaiyan Zhang, Yuru Wang, Junqi Gao, Runze Liu, Sa Yang, Jingxuan Li, Xinwei Long, Jiaheng Ma, and 1 others\. 2025\.Reviewrl: Towards automated scientific review with rl\.In*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, pages 16942–16954\.
- Zhang et al\. \(2025\)Junkai Zhang, Zihao Wang, Lin Gui, Swarnashree Mysore Sathyendra, Jaehwan Jeong, Victor Veitch, Wei Wang, Yunzhong He, Bing Liu, and Lifeng Jin\. 2025\.[Chasing the tail: Effective rubric\-based reward modeling for large language model post\-training](https://doi.org/10.48550/ARXIV.2509.21500)\.*CoRR*, abs/2509\.21500\.
- Zhang et al\. \(2026\)Yinuo Zhang, Dingcheng Huang, Haifeng Suo, Yizhuo Li, Ziya Zhao, Junhao Xu, Zhiying Tu, Dianhui Chu, Deming Zhai, Xianming Liu, and 1 others\. 2026\.Echoreview: Learning peer review from the echoes of scientific citations\.*arXiv preprint arXiv:2602\.00733*\.
- Zhou et al\. \(2025\)Yang Zhou, Sunzhu Li, Shunyu Liu, Wenkai Fang, Jiale Zhao, Jingwen Yang, Jianwei Lv, Kongcheng Zhang, Yihe Zhou, Hengtong Lu, Wei Chen, Yan Xie, and Mingli Song\. 2025\.[Breaking the exploration bottleneck: Rubric\-scaffolded reinforcement learning for general LLM reasoning](https://doi.org/10.48550/ARXIV.2508.16949)\.*CoRR*, abs/2508\.16949\.
- Zhu et al\. \(2025a\)Changjia Zhu, Junjie Xiong, Renkai Ma, Zhicong Lu, Yao Liu, and Lingyao Li\. 2025a\.[When your reviewer is an LLM: biases, divergence, and prompt injection risks in peer review](https://doi.org/10.48550/ARXIV.2509.09912)\.*CoRR*, abs/2509\.09912\.
- Zhu et al\. \(2025b\)Minjun Zhu, Yixuan Weng, Linyi Yang, and Yue Zhang\. 2025b\.Deepreview: Improving llm\-based paper review with human\-like deep thinking process\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 29330–29355\.

## Appendix AGeneral Rubrics

We obtain a unified set of40 general rubricsby consolidating rubrics extracted from ICLR 2024/2025 reviews, organized into three top\-level categories:Idea & Methodology\(14\),Experiment & Evaluation\(18\), andPresentation & Writing\(8\)\. Within each category, rubrics are sorted by empirical support in descending order and consecutively numbered \(G1–G14, E1–E18, P1–P8\)\. The percentage at the end of each item is that rubric’s share of mapped review excerpts*within its category*\(each category sums to100%100\\%\)\. Table[4](https://arxiv.org/html/2608.00005#A1.T4)summarizes the distribution\.

Category\# RubricsID rangeIdea & Methodology14G1–G14Experiment & Evaluation18E1–E18Presentation & Writing8P1–P8Total40—Table 4:Distribution of the 40 general rubrics across the three top\-level categories\.### A\.1Idea & Methodology \(14 rubrics\)

G1Does the paper make a genuinely novel core contribution that is clearly differentiated and properly positioned relative to the most relevant prior work?\(26\.96%\)

G2Are the paper’s core design and methodological choices clearly motivated and convincingly justified relative to reasonable alternatives, with explicit trade\-offs where applicable — and is the proposed approach a coherent, well\-integrated whole whose components are justified as contributing meaningfully to the overall design?\(17\.34%\)

G3Are the paper’s problem formulation, key concepts, terminology, notation, and definitions specified clearly, precisely, and used consistently and unambiguously throughout?\(13\.30%\)

G4Does the paper clearly state and justify its key assumptions and scope, keep its claims appropriately bounded, acknowledge resulting limitations and failure modes — and transparently discuss these in the narrative \(not only in passing\)?\(10\.63%\)

G5Are the paper’s theoretical claims \(guarantees, proofs, bounds, complexity statements\) technically correct, rigorously derived under clearly stated assumptions, and not overstated — and are derivations, proofs, and formal statements presented clearly and completely enough for readers to follow and verify?\(6\.77%\)

G6Does the paper convincingly motivate the problem it addresses as important, practically relevant, and well\-grounded in real gaps in prior work — including effective early contextualization \(abstract, introduction, background\) so readers understand the problem, core idea, and why it matters?\(5\.33%\)

G7Is the proposed approach practically feasible, scalable, and efficient under realistic conditions, with computational/resource trade\-offs clearly accounted for?\(4\.59%\)

G8Are the paper’s headline claims appropriately scoped, qualified, and supported by the evidence and analysis presented, without overstating?\(3\.19%\)

G9Does the paper provide a convincing mechanistic, causal, or theoretical rationale that explains why the approach works \(or fails\), beyond merely reporting outcomes?\(2\.85%\)

G10Does the paper appropriately analyze and address relevant ethical, safety, privacy, fairness, bias, and risk implications, including responsible mitigations?\(2\.59%\)

G11Does the paper credibly establish that its core idea generalizes beyond the specific setting evaluated, with applicability boundaries clearly stated?\(2\.49%\)

G12Is the paper’s overall contribution substantively significant and impactful enough to warrant acceptance and benefit the broader community — including clear communication of practical usefulness, real\-world implications, and actionable guidance where appropriate?\(1\.72%\)

G13Does the paper provide clearly accessible code, data, models, or other artifacts \(or a credible commitment to release them\) sufficient for independent verification — and are released resources designed and documented to be reusable and extensible by the community?\(1\.61%\)

G14Does the paper provide non\-obvious, well\-supported insights or interpretations that go beyond simply reporting expected results?\(0\.63%\)

### A\.2Experiment & Evaluation \(18 rubrics\)

E1Do the experiments provide sufficient, rigorous empirical evidence to convincingly support the paper’s main claims and conclusions?\(20\.27%\)

E2Does the paper describe the proposed method or procedure \(including training, inference, and pipeline components\) and the experimental setup, hyperparameters, procedures, and implementation details with sufficient unambiguous detail and transparency to enable understanding, reproduction, and fair comparison?\(16\.80%\)

E3Do the experiments include fair, well\-controlled comparisons against sufficiently strong, relevant baselines/alternative methods under matched, apples\-to\-apples conditions?\(10\.52%\)

E4Does the paper rigorously measure, report, and analyze computational/resource cost \(runtime, memory, scaling, efficiency trade\-offs\) under fair, clearly specified conditions?\(9\.28%\)

E5Do the experiments use controlled ablations or comparisons that isolate the causal contributions of key components/design choices, ruling out confounds?\(8\.99%\)

E6Do the experiments include systematic sensitivity/robustness analyses showing the results are stable to reasonable variations in key assumptions, settings, or randomness?\(7\.90%\)

E7Do the experiments convincingly demonstrate that the approach generalizes beyond the original training/evaluation setting \(across architectures, domains, distributions\)?\(5\.28%\)

E8Are the evaluation metrics, tasks/benchmarks, and overall protocol appropriate, well\-justified, and aligned with the paper’s claims?\(4\.64%\)

E9Does the paper analyze and explain where and why the approach fails, succeeds, or underperforms, including limitations, failure modes, and trade\-offs?\(4\.12%\)

E10Does the paper transparently document the datasets used and convincingly establish that the data \(or contributed resources\) are of sufficient quality, scale, representativeness, and provenance — including construction, processing, key statistics, label and annotation quality, biases, and selection — for trustworthy interpretation and reproduction?\(2\.80%\)

E11Are the experiments conducted under realistic, sufficiently challenging, and representative conditions that match the paper’s intended use case?\(2\.41%\)

E12Do the experiments include clear, representative qualitative analyses, visualizations, or example\-level evidence that meaningfully support and help interpret the paper’s claims?\(1\.46%\)

E13Do the experiments characterize how performance/behavior scales with relevant problem dimensions \(size, complexity, data, model scale, batch, iterations\)?\(1\.37%\)

E14Are the reported experimental results supported by appropriate statistical analysis and clearly reported uncertainty/variability estimates?\(1\.33%\)

E15Does the experimental design appropriately prevent overfitting/leakage/contamination, with proper train\-tune\-test separation, so generalization claims are credible?\(0\.91%\)

E16Does the paper empirically validate its theoretical or mechanistic claims and show their practical relevance?\(0\.90%\)

E17Is the adversarial/attack evaluation specified rigorously and broadly enough — including adaptive/strong adversaries — to substantiate the paper’s robustness/security claims?\(0\.61%\)

E18When claims rely on human perception or judgment, does the paper include an appropriately designed and conducted human\-subject evaluation \(or clearly justify its absence\)?\(0\.41%\)

### A\.3Presentation & Writing \(8 rubrics\)

P1Is the paper clearly written and well\-organized with a coherent, easy\-to\-follow narrative for its intended audience, avoiding unnecessary redundancy or tangents?\(41\.73%\)

P2Does the paper accurately and adequately cite, cover, and position itself relative to the most relevant prior work, with fair credit and clear differentiation?\(18\.64%\)

P3Are the paper’s figures, tables, and visualizations clear, well\-labeled, self\-contained, and integrated with the text so they support the main claims without being misleading?\(16\.28%\)

P4Is the paper mechanically polished and internally consistent — minimal typos, grammatical errors, formatting issues, citation/reference problems, and free of unresolved contradictions or avoidable technical/presentation errors that hinder interpretation?\(8\.94%\)

P5Is the paper sufficiently complete, polished, and venue\-aligned to clear the publication threshold — matching the target venue’s scope, formatting, and expectations — including, where a rebuttal or revision was part of the process, substantive responses to key reviewer concerns?\(5\.42%\)

P6Are the paper’s quantitative results presented and interpreted clearly, consistently, and non\-misleadingly, with enough context to enable unambiguous comparison?\(5\.31%\)

P7Does the paper provide a clear, well\-motivated conclusion and credible future\-work directions grounded in its contributions and findings?\(3\.03%\)

P8Does the paper present its interpretability goals, approach, and trade\-offs clearly and coherently, with adequate diagnostic visualizations or analyses?\(0\.65%\)