More Criticism Does Not Make a Better Review: EquiReview-R

# More Criticism Does Not Make a Better Review: EquiReview-R
Source: [https://arxiv.org/html/2609.03943](https://arxiv.org/html/2609.03943)
###### Abstract

## Abstract

AI reviewers can now produce many specific criticisms, but more criticism is not necessarily a better review\. A review may miss a consequential weakness or retain an allegation that available evidence does not support\. These failures require opposite corrections, yet generation\-oriented systems and aggregate measures obscure the distinction\. We therefore recast AI\-assisted review as evidence\-guided refinement of a structured concern set, with omission and overcritique treated as separate risks\. Building on this formulation, we introduceEquiReview\-R, which resolves existing concerns against localized evidence, searches for missing issues from independent and review\-conditioned perspectives, and returns stop, continue, or defer\. To expose the failure mode that motivates this design, we construct an evidence\-linked trajectory corpus\. Its retrospective analysis shows why revision must precede further search: nearly all concerns in a high\-recall review lack a definitive evidential disposition, while an earlier refinement mechanism cannot revise them\. On a frozen cohort of previously unseen papers,EquiReview\-Rsatisfies the prespecified non\-inferiority criterion for major omission, reduces major overcritique from 15\.5% to 8\.1%, and attains a one\-sided omission upper bound of 9\.9% while stopping on 52\.4% of papers\. Computation\-matched controls, controlled pairs, and ablations show that the gain comes from revision rather than extra inference or shorter output\. We release the corpus asReviewTrace, an evidence\-linked resource for studying review revision, disagreement, and provenance\.

###### keywords

AI\-assisted peer review, scientific reviewing, review revision, selective risk control, evidence provenance

## 1Introduction

AI assistance is already changing scientific peer review\. In a large randomized study at ICLR 2025, model\-generated feedback led reviewers to revise real reports and engage more deeply with author responses\([Thakkar et al\., 2026](https://arxiv.org/html/2609.03943#bib.bib20)\)\. Corpus studies likewise find that language models increasingly modify conference reviews\([Liang et al\., 2024a](https://arxiv.org/html/2609.03943#bib.bib25)\)\. A complementary study of Nature\-family papers asked domain scientists to assess individual human and AI criticisms\. AI reviewers surfaced issues that humans missed, but they also overlapped strongly with one another and were overly critical about minor points\([Kim and others, 2026](https://arxiv.org/html/2609.03943#bib.bib9)\)\. Together, these results suggest a shift in the central technical problem\. As review agents examine more aspects of a paper and generate more candidate objections, the bottleneck moves from producing criticism to deciding which criticism remains justified after the evidence is checked\.

A simple example shows why this distinction matters\. Suppose one review overlooks the absence of a matched baseline\. The appropriate correction is to add a concern\. A second review alleges data leakage even though the paper documents a clean split\. The appropriate correction is to remove or narrow the allegation\. The first review lacks coverage\. The second imposes an unsupported burden on authors and readers\. A system can improve one error while worsening the other, even when its output becomes longer and appears more thorough\.

We therefore study review improvement as the revision of a structured concern set\. Each concern states one alleged scientific failure, identifies the relevant part of the paper, and records the evidence needed to resolve it\. The set should expand when a material issue is missing, but it should also contract when a concern is refuted, duplicated, already answered, or broader than the evidence warrants\. This view separates two paper\-level risks\.*Major omission*captures consequential issues that are absent or remain unresolved when review ends\.*Major overcritique*captures consequential allegations that should be removed from or materially narrowed in the visible review\.

To study this process directly, we constructedReviewTracerather than extracting static reviews from an existing corpus\. The resource records how each concern is proposed, challenged, related to other concerns, revised, and judged\. Its retrospective trajectories expose a structural failure that aggregate scores conceal\. In the high\-recall review state, 96\.3% of concerns had not yet been supported, narrowed, or rejected\. A later refinement mechanism left every initial concern unchanged because its revision step only revisited concerns that had already received a provisional disposition\. Figure[1](https://arxiv.org/html/2609.03943#S1.F1)shows the broader consequence\. Systems with similar aggregate failure rates can occupy very different positions in the omission\-overcritique plane and therefore require different corrections\.

Figure 1:The bottleneck shifts from finding criticism to resolving it\. Additional search can expose missing issues, but it can also accumulate claims that have not survived evidential scrutiny\. Reliable refinement must support both directions of correction\. Retrospective systems occupy different regions of the omission\-overcritique plane, so a one\-dimensional failure label does not reveal which correction is needed\.This diagnosis leads to a direct design principle: revise the current review before expanding it\. We proposeEquiReview\-R, which first re\-evaluates every unresolved concern against localized support, counterevidence, and an explicit resolution condition\. It then freezes the revised state and searches for omissions from two complementary perspectives\. One search is independent of the current review, while the other uses the revised review to target aspects it does not represent\. Finally, a selective procedure returns*stop*,*continue*, or*defer*according to unresolved consequence, evidential completeness, and judgment uncertainty\. This order matters\. Revision controls unsupported criticism, complementary search protects recall, and selective stopping prevents unresolved high\-consequence questions from being mistaken for a finished review\.

The evaluation follows the same logic\. Missing concerns are sought only after a system freezes its review and stopping decision, so the answer cannot define the test\. Strict coverage requires the same alleged failure and resolution condition, and every conditional omission result is paired with stopping coverage\. Computation\-matched controls and minimally different issue and clean\-control pairs test whether gains come from revision rather than extra inference or a general preference for saying less\.

Our contributions are:

- •We separate omission from overcritique and show that their union cannot identify the needed correction\.
- •We proposeEquiReview\-R, combining evidence\-guided revision, complementary discovery, and selective stopping in a reconstructable state\.
- •We evaluate it on a frozen cohort with matched computation, independent omission candidates, controlled pairs, and ablations\.
- •We releaseReviewTrace, an evidence\-linked corpus of concern trajectories and independent judgments\.

## 2Related Work

#### AI\-assisted scientific review\.

Early resources such as PeerRead enabled review text and score prediction\([Kang et al\., 2018](https://arxiv.org/html/2609.03943#bib.bib11)\)\. Later work examined the usefulness and real\-world uptake of model\-generated feedback\([Yuan et al\., 2022](https://arxiv.org/html/2609.03943#bib.bib14);[Liang et al\., 2024b](https://arxiv.org/html/2609.03943#bib.bib15);[Liang et al\., 2024a](https://arxiv.org/html/2609.03943#bib.bib25);[Thakkar et al\., 2026](https://arxiv.org/html/2609.03943#bib.bib20)\)\. Modern systems improve generation through multi\-stage analysis, retrieval, response\-based verification, or proactive investigation\. SEA consolidates multiple reviews\([Yu et al\., 2024](https://arxiv.org/html/2609.03943#bib.bib17)\); DeepReview produces structured, evidence\-rich reports\([Zhu et al\., 2025](https://arxiv.org/html/2609.03943#bib.bib18)\);DIAGandE3emphasize specific weaknesses and issue\-level backtesting\([Zou et al\., 2026](https://arxiv.org/html/2609.03943#bib.bib6);[Chaudhuri et al\., 2026](https://arxiv.org/html/2609.03943#bib.bib5)\); and ProReviewer maintains a structured log to guide active investigation\([Fang et al\., 2026](https://arxiv.org/html/2609.03943#bib.bib26)\)\. These approaches expand or organize the criticism a system can produce\. Our object is the resulting concern set, and our question is which concerns should survive evidential scrutiny before further search or a stopping decision\.

#### Evaluating review content\.

Scientific reviews have been evaluated through score agreement, review\-response questions, overlap with observed feedback, concern matching, and attention across paper facets\([Zhou et al\., 2024](https://arxiv.org/html/2609.03943#bib.bib16);[Liang et al\., 2024b](https://arxiv.org/html/2609.03943#bib.bib15);[Jin, 2026](https://arxiv.org/html/2609.03943#bib.bib7);[Li et al\., 2026](https://arxiv.org/html/2609.03943#bib.bib8);[Shin et al\., 2025](https://arxiv.org/html/2609.03943#bib.bib27)\)\. CriticEval similarly decomposes critique quality across tasks and dimensions\([Lan et al\., 2024](https://arxiv.org/html/2609.03943#bib.bib24)\)\. These perspectives reveal whether individual comments are correct, useful, or focused on appropriate aspects\. They do not by themselves determine whether a persistent set of comments is sufficient to end review or contains claims that evidence no longer supports or that should be narrowed or merged\. We make these revision actions and their paper\-level consequences the primary evaluation object\.

#### Revision, robustness, and selective decisions\.

Self\-critique and tool\-interactive critique can expose errors and improve model responses\([Saunders et al\., 2022](https://arxiv.org/html/2609.03943#bib.bib22);[Gou et al\., 2024](https://arxiv.org/html/2609.03943#bib.bib23)\)\. In peer review, human\-in\-the\-loop analyses and robustness studies emphasize correlated model errors, manipulation, and deployment risk\([Drori and Te’eni, 2024](https://arxiv.org/html/2609.03943#bib.bib21);[Ye et al\., 2024](https://arxiv.org/html/2609.03943#bib.bib19);[Baumann et al\., 2026](https://arxiv.org/html/2609.03943#bib.bib10);[Xin et al\., 2026](https://arxiv.org/html/2609.03943#bib.bib28)\)\. Selective prediction provides a principled language for abstaining when uncertainty remains\([El\-Yaniv and Wiener, 2010](https://arxiv.org/html/2609.03943#bib.bib3);[Geifman and El\-Yaniv, 2017](https://arxiv.org/html/2609.03943#bib.bib4)\)\. Learn then Test and conformal risk control provide finite\-sample procedures for evaluating prespecified risk constraints\([Angelopoulos et al\., 2025](https://arxiv.org/html/2609.03943#bib.bib1);[Angelopoulos et al\., 2024](https://arxiv.org/html/2609.03943#bib.bib2)\)\. We connect these lines by applying selective risk control to a revised review state rather than to a scalar prediction\.

## 3Method

### 3\.1Problem Formulation

Letxxdenote a paper and letS0S\_\{0\}be an initial structured review\. We represent a review state as

whereA⁡\(S\)A\(S\)is the set of concerns shown in the current review,G⁡\(S\)G\(S\)is a typed graph relating concerns as identical, overlapping, parent\-child, or distinct, andH⁡\(S\)H\(S\)is an immutable history of evidence and revision actions\. A concernc∈A⁡\(S\)c\\in A\(S\)contains an alleged failure, a paper location, supporting and countervailing evidence, a resolution condition, materiality, and a current status\. SeparatingAAfromHHallows the visible review to become more concise without erasing what was proposed or why it changed\.

Validity is judged before materiality\. A concern is*major*only when resolving it could change the validity, scope, or evidential support of a principal claim, materially alter the interpretation of a central result, or affect the credibility of the main evaluation\. Moderate concerns require local analysis or qualification, while minor concerns primarily affect presentation\. The primary endpoints use only valid major concerns\.

LetQKQ\_\{K\}be a fixed external search procedure withKKprespecified search opportunities\. We define two paper\-level losses\. Given a stopping rulegλg\_\{\\lambda\},Lmiss​\(S,QK,gλ\)L\_\{\\mathrm\{miss\}\}\(S;Q\_\{K\},g\_\{\\lambda\}\)is one whenQKQ\_\{K\}finds a valid major concern that is distinct fromA⁡\(S\)A\(S\), or whengλg\_\{\\lambda\}stops while a major concern remains unresolved\.Lover​\(S\)L\_\{\\mathrm\{over\}\}\(S\)is one whenA⁡\(S\)A\(S\)retains a major concern that independent judgment says should be removed or materially narrowed\. We report omission only among papers that the rule stops,

Rmiss​\(gλ\)\\displaystyle R\_\{\\mathrm\{miss\}\}\(g\_\{\\lambda\}\)=𝔼⁡\[Lmiss​\(S,QK,gλ\)∣gλ​\(S\)=stop\],\\displaystyle=\\mathbb\{E\}\[L\_\{\\mathrm\{miss\}\}\(S;Q\_\{K\},g\_\{\\lambda\}\)\\mid g\_\{\\lambda\}\(S\)=\\mathrm\{stop\}\],\(2\)Cstop​\(gλ\)\\displaystyle C\_\{\\mathrm\{stop\}\}\(g\_\{\\lambda\}\)=Pr\[gλ\(S\)=stop\]\.\\displaystyle=\\Pr\[g\_\{\\lambda\}\(S\)=\\mathrm\{stop\}\]\.The two quantities must be interpreted together\. Never stopping makes the conditional risk uninformative, whereas stopping every paper may violate the target\.

Strict issue coverage is a secondary constraint\. LetCov⁡\(S\)\\mathrm\{Cov\}\(S\)denote this quantity\. A concern receives credit only when it matches both the alleged failure and the resolution condition of an independently constructed reference issue\. Thus “Theorem 1 lacks a proof of Lemma 2” and “Theorem 1 lacks a convergence\-rate analysis” are distinct even though they concern the same theorem\. Overlap and parent\-child relations receive partial credit only in sensitivity analyses\.

The method can now be written as three coupled operators,

S−\\displaystyle S^\{\-\}=ℛϕ​\(x,S0\),\\displaystyle=\\mathcal\{R\}\_\{\\phi\}\(x,S\_\{0\}\),\(3\)U\\displaystyle U=𝒟ind​\(x\)∪𝒟cond​\(x,A⁡\(S−\)\),\\displaystyle=\\mathcal\{D\}\_\{\\mathrm\{ind\}\}\(x\)\\cup\\mathcal\{D\}\_\{\\mathrm\{cond\}\}\(x,A\(S^\{\-\}\)\),S⋆\\displaystyle S^\{\\star\}=Γψ\(S−,U\),d=gλ\(f\(S⋆\)\)\.\\displaystyle=\\Gamma\_\{\\psi\}\(S^\{\-\},U\),\\qquad d=g\_\{\\lambda\}\(f\(S^\{\\star\}\)\)\.Hereℛϕ\\mathcal\{R\}\_\{\\phi\}revises the existing review, the two𝒟\\mathcal\{D\}operators search for omissions from independent and review\-conditioned perspectives,Γψ\\Gamma\_\{\\psi\}consolidates and admits candidates, andgλg\_\{\\lambda\}returnsd∈\{stop,continue,defer\}d\\in\\\{\\mathrm\{stop\},\\mathrm\{continue\},\\mathrm\{defer\}\\\}\. The design objective is to reduce overcritique while preserving issue\-finding ability and nontrivial stopping coverage,

minϕ,ψ,λ\\displaystyle\\min\_\{\\phi,\\psi,\\lambda\}𝔼⁡\[Lover​\(S⋆\)\]\\displaystyle\\mathbb\{E\}\[L\_\{\\mathrm\{over\}\}\(S^\{\\star\}\)\]\(4\)subject to\\displaystyle\\text\{subject to\}Rmiss\(gλ\)≤α,Cstop\(gλ\)≥c0,\\displaystyle R\_\{\\mathrm\{miss\}\}\(g\_\{\\lambda\}\)\\leq\\alpha,\\quad C\_\{\\mathrm\{stop\}\}\(g\_\{\\lambda\}\)\\geq c\_\{0\},Cov⁡\(S⋆\)≥Cov⁡\(S0\)−ϵ\.\\displaystyle\\mathrm\{Cov\}\(S^\{\\star\}\)\\geq\\mathrm\{Cov\}\(S\_\{0\}\)\-\\epsilon\.This constrained formulation makes explicit why simply shortening a review cannot solve the problem\.

### 3\.2EquiReview\-R

Figure[2](https://arxiv.org/html/2609.03943#S3.F2)summarizes the four stages and their shared state\. The same concern identifiers, evidence records, and relation graph connect revision, discovery, and stopping, so later stages cannot silently reinterpret what earlier stages produced\.

Figure 2:Overview ofEquiReview\-R\. The revision operatorℛϕ\\mathcal\{R\}\_\{\\phi\}resolves the existing review before the two discovery operators search for omissions from a frozen state\.Γψ\\Gamma\_\{\\psi\}consolidates accepted candidates, andgλg\_\{\\lambda\}returns stop, continue, or defer\. The reader\-facing review may change, while the full trajectory remains recoverable\.#### Evidence\-guided revision\.

For every unresolved concern,ℛϕ\\mathcal\{R\}\_\{\\phi\}builds an evidence record containing the alleged failure, its location, the strongest supporting evidence, the strongest counterevidence, and the condition that would settle the claim\. Decomposed judgments then assign one of seven outcomes: supported, narrowed, refuted, merged, resolved, unresolved with high consequence, or unresolved with lower consequence\. Supported and narrowed concerns remain visible\. Refuted, merged, and resolved concerns leave the visible review but remain inH⁡\(S\)H\(S\)\. A high\-consequence unresolved concern cannot be hidden by a presentation limit\. For example, an allegation that test data influenced model selection is not accepted or removed from a split description alone\. The revision record identifies where model selection is specified, whether test labels were consulted, and what evidence would settle the claim\. It then retains the allegation, narrows it to a reporting ambiguity, or resolves it according to that evidence\.

#### Complementary search\.

After revision,S−S^\{\-\}is frozen\.𝒟ind\\mathcal\{D\}\_\{\\mathrm\{ind\}\}reads the paper without the current concern set and therefore preserves an independent route to issues the existing review may have framed away\.𝒟cond\\mathcal\{D\}\_\{\\mathrm\{cond\}\}reads the same paper together withA⁡\(S−\)A\(S^\{\-\}\)and searches fixed scientific facets that the revised review does not yet cover\. Both operate on the same frozen state\.Γψ\\Gamma\_\{\\psi\}accepts valid material candidates, preserves graded concern relations, and consolidates duplicates before updating the visible review\.

#### Selective stopping\.

The feature mapf⁡\(S⋆\)f\(S^\{\\star\}\)uses only information available before external evaluation, including unresolved high\-consequence concerns, evidence completeness, judgment disagreement, relation uncertainty, and recent discovery yield\. The ordered rule family is fixed before confirmation labels are computed\. The confirmation procedure selects the highest\-coverage rule whose one\-sided omission bound meets the prespecified target\. At use time, that rule returns*stop*only when no high\-consequence concern remains unresolved\. It returns*continue*when another prespecified round has an actionable target, and*defer*when missing artifacts or domain judgment prevent a responsible stopping decision\. All three outputs return the revised review; continue and defer additionally expose the unresolved targets and required evidence\.

## 4Experiments

### 4\.1Design

We use two non\-overlapping study phases\. A retrospective corpus of 380 recent AI papers and 1,900 recorded review states supports problem diagnosis, method development, and rule specification\. A separate cohort of 271 previously unseen papers is used once for confirmation after the method, search process, judgment protocol, rule ordering, and statistical analysis are frozen\. The confirmation papers span machine learning, natural language processing, computer vision, and AI systems\. Every system receives the same main\-paper view\.

The primary comparison isE3, a strong high\-recall issue\-level reviewer\([Chaudhuri et al\., 2026](https://arxiv.org/html/2609.03943#bib.bib5)\)\.E3\-Matchedreceives essentially the same number of effective calls and generated tokens asEquiReview\-Rbut repeats the generation\-oriented procedure rather than revising its existing concerns\. This control isolates the algorithmic contribution from added inference\.DIAGandSimpleappear in the retrospective analysis\.

### 4\.2Evaluation Protocol

External evaluation begins only after each system freezes both its review and stopping decision\. An independent search constructs missing\-concern candidates, and a separate review of the frozen state identifies consequential unresolved questions\. The primary omission endpoint is the union of these two sources\. It does not depend on the shared candidate pool\. Pool coverage is secondary and is recomputed after excluding each evaluated system’s own candidates\. This separation prevents a system’s output from defining its own success criterion\.

Candidates are judged independently by GPT\-5\.6 Luna and GPT\-5\.6 Terra\. When their categorical decisions disagree, GPT\-5\.6 Sol provides a third decision in a separate call with source information hidden\. The judges do not see system identity, candidate source, or one another’s rationale\. Validity, identity, revision action, and materiality are elicited separately\. These judgments form a repeatable model\-based measurement panel rather than objective scientific truth; the analysis therefore reports disagreement, alternative reference policies, and the contribution of adjudicated cases\.

The primary tests are major\-omission non\-inferiority, reduction in major overcritique, and the omission criterion for stopped papers\. Paired binary endpoints use McNemar tests and paper\-level bootstrap intervals\. Candidate stopping rules are evaluated in a fixed sequence with Learn then Test family\-wise control\([Angelopoulos et al\., 2025](https://arxiv.org/html/2609.03943#bib.bib1)\)\. Strict coverage, visible concern count, generated tokens, controlled pairs, component ablations, relation\-policy sensitivity, and subfield heterogeneity provide secondary evidence\.

## 5Results

### 5\.1Diagnosis

The retrospective analysis first asks whether a single failure event identifies the correction a review needs\. It does not\. On 150 papers, the response\-oriented reviewer and the coverage\-oriented predecessor both attain low omission but high overcritique, whereas the conservative refinement variant moves in the opposite direction\. The high\-recall initializer is more balanced but leaves 16\.62 of 17\.27 visible concerns per paper unresolved\. Across 320 papers, the earlier revision mechanism changes none of these initial concerns\. The dominant uncertainty is therefore not a peripheral implementation detail\. It is precisely the part of the review that must be revised before additional search can be interpreted\.

### 5\.2Main Results

Table 1:Confirmation results on 271 unseen papers\. Omit\. and Over\. are paper\-level major omission and overcritique\. Cov\. is strict shared\-pool issue coverage\. Tok\. reports generated output tokens\. Best outcome values are bold\.Table[1](https://arxiv.org/html/2609.03943#S5.T1)establishes the main result\. Relative to the high\-recall initializer,EquiReview\-Rmeets the prespecified omission non\-inferiority criterion and reduces major overcritique by−7\.4\-7\.4percentage points \(95% CI \[−11\.6\-11\.6,−3\.1\-3\.1\]\)\. Strict issue coverage changes by only−0\.3\-0\.3percentage points \(95% CI \[−1\.1\-1\.1,0\.50\.5\]\), while the visible review contains−4\.7\-4\.7fewer concerns per paper\. The result is therefore a revision of the error profile rather than an exchange of recall for concision\.

The computation\-matched control provides the key counterfactual\. Although it receives essentially the same number of calls and output tokens,E3\-Matchedretains a substantially larger visible review and more than twice the major\-overcritique rate ofEquiReview\-R\. Additional inference is useful only when it changes the review state rather than merely extending the review\.

### 5\.3Efficiency

Figure 3:Resource use and visible review size\. Bubble area is proportional to generated output tokens\. At closely matched inference,EquiReview\-Rreturns a smaller visible review than the generation\-only control\.Figure[3](https://arxiv.org/html/2609.03943#S5.F3)makes this distinction visible\. Generation\-only inference moves the matched control upward by expanding the visible review, whereasEquiReview\-Ruses comparable inference to resolve and consolidate existing claims\. The result isolates how computation is used, not merely how much is supplied\.

### 5\.4Selective Stopping

![Refer to caption](https://arxiv.org/html/2609.03943v1/fig3_risk_coverage.png)Figure 4:Risk and stopping coverage for the frozen rule family\. The stop\-all endpoint fails the target, while never stopping provides no useful stopping decision\. The annotated point is the highest\-coverage rule whose one\-sided upper bound satisfies the 10% omission criterion\.The selected rule stops on 142 of 271 papers\. It records 8 omissions in that subset, with empirical risk 5\.6% and a one\-sided upper bound of 9\.9% at 52\.4% coverage\. Stopping every paper yields an upper bound of 16\.4%\. Figure[4](https://arxiv.org/html/2609.03943#S5.F4)therefore shows a nontrivial operating point rather than reliability obtained by stopping every paper or deferring nearly all of them\.

The remaining papers still receive usable reviews\. Continue identifies the unresolved target and evidence needed for another prespecified round, while defer exposes the high\-consequence uncertainty, missing artifact, or expertise that requires human attention\. Stopping is selective, but the review itself is not withheld\.

### 5\.5Mechanism

Figure 5:Evidence\-guided outcomes for initially unresolved concerns\. Most concerns receive a definite revision, while consequential uncertainty remains visible\.Figure[5](https://arxiv.org/html/2609.03943#S5.F5)shows how initially unresolved concerns change after evidence is considered\. Most receive a definite disposition, and only 4\.0% remain unresolved with high consequence\. Supported and narrowed concerns remain visible at the scope justified by the evidence, while refuted, merged, and resolved concerns leave the visible review but remain in the trajectory\. The review becomes shorter through explicit actions rather than silent deletion\.

Controlled pairs provide an independent test of that interpretation\.EquiReview\-Rrecalls 95\.8% of inserted issues and introduces false concerns in 3\.3% of clean controls\.E3\-Matchedattains 96\.7% recall but raises the clean false\-positive rate to 10\.8%\. Thus the smaller review retains high issue sensitivity because unsupported content is corrected rather than criticism being suppressed\.

Figure 6:Component and stopping\-rule ablations\. Left: revision primarily controls overcritique, whereas both discovery perspectives protect omission\. Right: review length alone does not identify a reliable stopping point\. Points below the horizontal line satisfy the omission criterion\.The ablations connect each component to the risk it is intended to control\. Without evidence\-guided revision, overcritique rises from 8\.1% to 14\.2%\. Removing the independent or review\-conditioned search raises omission to 17\.5% and 16\.7%, respectively\. Removing the high\-consequence exemption or replacing graded relations with hard identity also worsens the error profile\. The stopping\-rule analysis reaches the same conclusion\. A rule based only on review length has a 13\.2% omission upper bound and fails the target, whereas a rule that omits length but retains evidence and uncertainty signals attains a 9\.8% bound at 50\.6% coverage\.

### 5\.6Robustness

Figure 7:Judgment stability and result decomposition\. Agreement is lowest for materiality\. Most of the overcritique reduction comes from items on which the two primary judges agree\.Agreement is 0\.74 for validity and 0\.52 for materiality, and 30\.3% of items receive a third decision\. Of the gross reduction in overcritique cases, 75\.0% comes from items on which the two primary judges agree about materiality\. Either\-judge and both\-judge policies preserve the direction of the comparison\. Figure[7](https://arxiv.org/html/2609.03943#S5.F7)shows that materiality is the main source of measurement uncertainty, while the observed reduction is not confined to adjudicated cases\.

The conclusion is also stable to alternative issue\-identity policies\. Partial credit for overlap and parent\-child relations, as well as leave\-one\-system\-out reference pools, preserves the coverage ordering\. Thus neither looser semantic matching nor self\-contributed reference wording explains the result\. Subfield analyses preserve the direction within the sampled AI population, although systems papers defer more often\.

## 6ReviewTrace

We builtReviewTraceas part of the study rather than deriving labels from an existing review corpus\. It records each concern from first appearance through support, narrowing, merging, resolution, or removal, links every change to localized evidence, and preserves two independent judgments together with disagreement and adjudication\. Existing resources primarily organize papers, static reviews, scores, meta\-reviews, or sentence\-level quality labels\([Kang et al\., 2018](https://arxiv.org/html/2609.03943#bib.bib11);[Dycke et al\., 2023](https://arxiv.org/html/2609.03943#bib.bib29);[Shen et al\., 2022](https://arxiv.org/html/2609.03943#bib.bib30);[Purkayastha et al\., 2025](https://arxiv.org/html/2609.03943#bib.bib31)\)\.ReviewTraceinstead exposes the revision trajectory needed to train and evaluate systems that must correct criticism rather than only generate it\.

Table 2:Focused structural comparison with representative peer\-review resources\. Traj\. denotes concern\-level revision trajectories, Rel\. typed relations among concerns, and Indep\. separately retained judgments\. A dash means that the feature is not a primary released unit, not that the resource lacks value for its original task\.The release contains 1,900 structured states and 35,292 recorded judgments, together with normalized data specifications, evaluation code, artifact hashes, and retrieval and verification utilities\. The frozen construction record contains 77,929 distinct model calls\. Applying the public list\-price schedule to that record gives a public\-price equivalent of $9,700\([OpenAI, 2026](https://arxiv.org/html/2609.03943#bib.bib13)\)\. This investment produces more than a static collection of reviews\. Each transition supplies a paired example of what changed, which evidence supported the change, and which alternative judgments remained plausible\. The resource can therefore support revision\-policy learning, relation\-aware consolidation, disagreement modeling, provenance auditing, and selective stopping\. Because the independently produced judgments are retained before adjudication, future work can study where apparent label certainty reflects consensus and where it reflects a decision policy\.

## 7Discussion and Limitations

The empirical claims are conditional on recent AI papers, the main\-paper view, the fixed external search process, and the stated judgment policy\. Different disciplines, supplementary artifacts, or model families can change both the concern distribution and the attainable stopping coverage\. The confirmation results therefore establish reliability under a specified evaluation condition, not universal completeness\.

Practical use also requires explicit reporting of computation\. The matched control shows that additional generation enlarges rather than improves the visible review\. At comparable inference,EquiReview\-Ruses its calls to revise existing concerns and returns a smaller review\. Deployment should therefore choose an operating point by both stopping coverage and the evidence still required for non\-stopped papers\.

Luna, Terra, and Sol are separate calls from one model family, so their errors may be correlated\. Materiality also admits legitimate disagreement, as human review decisions do\([Beygelzimer et al\., 2023](https://arxiv.org/html/2609.03943#bib.bib12)\)\. Blinding system identity, decomposed labels, agreement\-subset analyses, and alternative reference policies reduce avoidable circularity but do not create noise\-free truth\. Cross\-family and domain\-expert validation remain necessary before deployment\. Stop, continue, and defer describe the review process rather than publication merit, and deferral must not become a rejection proxy\. In use, continue should present an explicit agenda for further examination, while defer should expose the missing artifact or expertise rather than return an opaque warning\. An author response can then enter as new evidence without erasing the original concern or its revision history\.

## 8Conclusion

As AI reviewers become more capable, the challenge is not only to find more weaknesses, but to determine which criticisms survive evidence, what remains missing, and when uncertainty should prevent stopping\.EquiReview\-Rconnects evidence\-guided revision, complementary discovery, and selective risk control in a reconstructable state\. The results show that a review can become more concise and better supported without surrendering material issue coverage\. This foundation could support auditable workflows in which model and human judgments, author responses, and editorial assessment accumulate as evidence rather than replace one another\.

## References

- Angelopouloset al\.\(2025\)A\. N\. Angelopoulos, S\. Bates, E\. J\. Candès, M\. I\. Jordan, and L\. LeiLearn then test: calibrating predictive algorithms to achieve risk control\.The Annals of Applied Statistics19\(2\),pp\. 1641–1662\.External Links:[Document](https://dx.doi.org/10.1214/24-AOAS1998)Cited by:[§2](https://arxiv.org/html/2609.03943#S2.SS0.SSS0.Px3.p1.1),[§4\.2](https://arxiv.org/html/2609.03943#S4.SS2.p3.1)\.
- Angelopouloset al\.\(2024\)A\. N\. Angelopoulos, S\. Bates, A\. Fisch, L\. Lei, and T\. SchusterConformal risk control\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2609.03943#S2.SS0.SSS0.Px3.p1.1)\.
- Baumannet al\.\(2026\)J\. Baumann, J\. Pei, S\. Koyejo, and D\. HovyStop automating peer review without rigorous evaluation\.arXiv preprint arXiv:2605\.03202\.Note:ICML 2026 Position PaperCited by:[§2](https://arxiv.org/html/2609.03943#S2.SS0.SSS0.Px3.p1.1)\.
- Beygelzimeret al\.\(2023\)A\. Beygelzimer, Y\. Dauphin, P\. Liang, and J\. W\. VaughanHas the machine learning review process become more arbitrary as the field has grown? the neurips 2021 consistency experiment\.arXiv preprint arXiv:2306\.03262\.Cited by:[§7](https://arxiv.org/html/2609.03943#S7.p3.1)\.
- Chaudhuriet al\.\(2026\)Y\. Chaudhuri, S\. Jain, and P\. MundraE3: issue\-level backtesting for automated research critique\.arXiv preprint arXiv:2605\.27072\.Cited by:[§2](https://arxiv.org/html/2609.03943#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2609.03943#S4.SS1.p2.1)\.
- Drori and Te’eni \(2024\)I\. Drori and D\. Te’eniHuman\-in\-the\-loop ai reviewing: feasibility, opportunities, and risks\.Journal of the Association for Information Systems25\(1\),pp\. 98–109\.External Links:[Document](https://dx.doi.org/10.17705/1jais.00867)Cited by:[§2](https://arxiv.org/html/2609.03943#S2.SS0.SSS0.Px3.p1.1)\.
- Dyckeet al\.\(2023\)N\. Dycke, I\. Kuznetsov, and I\. GurevychNLPeer: a unified resource for the computational study of peer review\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Toronto, Canada,pp\. 5049–5073\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.277)Cited by:[§6](https://arxiv.org/html/2609.03943#S6.p1.1)\.
- El\-Yaniv and Wiener \(2010\)R\. El\-Yaniv and Y\. WienerOn the foundations of noise\-free selective classification\.Journal of Machine Learning Research11\(53\),pp\. 1605–1641\.Cited by:[§2](https://arxiv.org/html/2609.03943#S2.SS0.SSS0.Px3.p1.1)\.
- Fanget al\.\(2026\)H\. Fang, Y\. Feng, and I\. GurevychFrom passive generation to investigation: a proactive scientific peer review agent\.arXiv preprint arXiv:2606\.13349\.Cited by:[§2](https://arxiv.org/html/2609.03943#S2.SS0.SSS0.Px1.p1.1)\.
- Geifman and El\-Yaniv \(2017\)Y\. Geifman and R\. El\-YanivSelective classification for deep neural networks\.InAdvances in Neural Information Processing Systems,Vol\.30\.Cited by:[§2](https://arxiv.org/html/2609.03943#S2.SS0.SSS0.Px3.p1.1)\.
- Gouet al\.\(2024\)Z\. Gou, Z\. Shao, Y\. Gong, Y\. Shen, Y\. Yang, N\. Duan, and W\. ChenCRITIC: large language models can self\-correct with tool\-interactive critiquing\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2609.03943#S2.SS0.SSS0.Px3.p1.1)\.
- Jin \(2026\)M\. JinWhat makes a good ai review? concern\-level diagnostics for ai peer review\.arXiv preprint arXiv:2604\.19998\.Cited by:[§2](https://arxiv.org/html/2609.03943#S2.SS0.SSS0.Px2.p1.1)\.
- Kanget al\.\(2018\)D\. Kang, W\. Ammar, B\. Dalvi, M\. van Zuylen, S\. Kohlmeier, E\. Hovy, and R\. SchwartzA dataset of peer reviews \(peerread\): collection, insights and nlp applications\.InProceedings of NAACL\-HLT,pp\. 1647–1661\.External Links:[Document](https://dx.doi.org/10.18653/v1/N18-1149)Cited by:[§2](https://arxiv.org/html/2609.03943#S2.SS0.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2609.03943#S6.p1.1)\.
- Kimet al\.\(2026\)S\. Kimet al\.On the limits and opportunities of ai reviewers: reviewing the reviews of nature\-family papers with 45 expert scientists\.arXiv preprint arXiv:2605\.20668\.Cited by:[§1](https://arxiv.org/html/2609.03943#S1.p1.1)\.
- Lanet al\.\(2024\)T\. Lan, W\. Zhang, C\. Xu, H\. Huang, D\. Lin, K\. Chen, and X\. MaoCriticEval: evaluating large\-scale language model as critic\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 66907–66960\.Cited by:[§2](https://arxiv.org/html/2609.03943#S2.SS0.SSS0.Px2.p1.1)\.
- Liet al\.\(2026\)B\. Li, H\. Ma, Y\. Wang, J\. Yang, X\. Chen, X\. Huang, Y\. Zheng, and X\. QiuBeyond rating: a comprehensive evaluation and benchmark for ai reviews\.arXiv preprint arXiv:2604\.19502\.Cited by:[§2](https://arxiv.org/html/2609.03943#S2.SS0.SSS0.Px2.p1.1)\.
- Lianget al\.\(2024a\)W\. Liang, Z\. Izzo, Y\. Zhang, H\. Lepp, H\. Cao, X\. Zhao, L\. Chen, H\. Ye, S\. Liu, Z\. Huang, D\. McFarland, and J\. Y\. ZouMonitoring ai\-modified content at scale: a case study on the impact of chatgpt on ai conference peer reviews\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 29575–29620\.Cited by:[§1](https://arxiv.org/html/2609.03943#S1.p1.1),[§2](https://arxiv.org/html/2609.03943#S2.SS0.SSS0.Px1.p1.1)\.
- Lianget al\.\(2024b\)W\. Liang, Y\. Zhang, H\. Cao, B\. Wang, D\. Y\. Ding, X\. Yang, K\. Vodrahalli, S\. He, D\. S\. Smith, Y\. Yin, D\. A\. McFarland, and J\. ZouCan large language models provide useful feedback on research papers? a large\-scale empirical analysis\.NEJM AI1\(8\)\.Cited by:[§2](https://arxiv.org/html/2609.03943#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2609.03943#S2.SS0.SSS0.Px2.p1.1)\.
- OpenAI \(2026\)OpenAIOpenAI api pricing\.Note:OpenAI API documentationAccessed July 22, 2026External Links:[Link](https://openai.com/api/pricing/)Cited by:[§6](https://arxiv.org/html/2609.03943#S6.p2.1)\.
- Purkayasthaet al\.\(2025\)S\. Purkayastha, Z\. Li, A\. Lauscher, L\. Qu, and I\. GurevychLazyReview: a dataset for uncovering lazy thinking in nlp peer reviews\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Vienna, Austria,pp\. 3280–3308\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.165)Cited by:[§6](https://arxiv.org/html/2609.03943#S6.p1.1)\.
- Saunderset al\.\(2022\)W\. Saunders, C\. Yeh, J\. Wu, S\. Bills, L\. Ouyang, J\. Ward, and J\. LeikeSelf\-critiquing models for assisting human evaluators\.arXiv preprint arXiv:2206\.05802\.Cited by:[§2](https://arxiv.org/html/2609.03943#S2.SS0.SSS0.Px3.p1.1)\.
- Shenet al\.\(2022\)C\. Shen, L\. Cheng, R\. Zhou, L\. Bing, Y\. You, and L\. SiMReD: a meta\-review dataset for structure\-controllable text generation\.InFindings of the Association for Computational Linguistics: ACL 2022,Dublin, Ireland,pp\. 2521–2535\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.findings-acl.198)Cited by:[§6](https://arxiv.org/html/2609.03943#S6.p1.1)\.
- Shinet al\.\(2025\)H\. Shin, J\. Tang, Y\. Lee, N\. Kim, H\. Lim, J\. Y\. Cho, H\. Hong, M\. Lee, and J\. KimMind the blind spots: a focus\-level evaluation framework for llm reviews\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,Suzhou, China,pp\. 35630–35656\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1805)Cited by:[§2](https://arxiv.org/html/2609.03943#S2.SS0.SSS0.Px2.p1.1)\.
- Thakkaret al\.\(2026\)N\. Thakkar, M\. Yuksekgonul, J\. Silberg, A\. Garg, N\. Peng, F\. Sha, R\. Yu, C\. Vondrick, and J\. ZouA large\-scale randomized study of large language model feedback in peer review\.Nature Machine Intelligence8,pp\. 326–336\.External Links:[Document](https://dx.doi.org/10.1038/s42256-026-01188-x)Cited by:[§1](https://arxiv.org/html/2609.03943#S1.p1.1),[§2](https://arxiv.org/html/2609.03943#S2.SS0.SSS0.Px1.p1.1)\.
- Xinet al\.\(2026\)Y\. Xin, Y\. Weng, M\. Zhu, Y\. Ling, C\. Qin, M\. Hahn, M\. Backes, Y\. Zhang, and L\. YangSafeReview: defending llm\-based review systems against adversarial hidden prompts\.arXiv preprint arXiv:2604\.26506\.Cited by:[§2](https://arxiv.org/html/2609.03943#S2.SS0.SSS0.Px3.p1.1)\.
- Yeet al\.\(2024\)R\. Ye, X\. Pang, J\. Chai, J\. Chen, Z\. Yin, Z\. Xiang, X\. Dong, J\. Shao, and S\. ChenAre we there yet? revealing the risks of utilizing large language models in scholarly peer review\.arXiv preprint arXiv:2412\.01708\.Cited by:[§2](https://arxiv.org/html/2609.03943#S2.SS0.SSS0.Px3.p1.1)\.
- Yuet al\.\(2024\)J\. Yu, Z\. Ding, J\. Tan, K\. Luo, Z\. Weng, C\. Gong, L\. Zeng, R\. Cui, C\. Han, Q\. Sun, Z\. Wu, Y\. Lan, and X\. LiAutomated peer reviewing in paper sea: standardization, evaluation, and analysis\.InFindings of the Association for Computational Linguistics: EMNLP 2024,pp\. 10164–10184\.Cited by:[§2](https://arxiv.org/html/2609.03943#S2.SS0.SSS0.Px1.p1.1)\.
- Yuanet al\.\(2022\)W\. Yuan, P\. Liu, and G\. NeubigCan we automate scientific reviewing?\.Journal of Artificial Intelligence Research75,pp\. 171–212\.Cited by:[§2](https://arxiv.org/html/2609.03943#S2.SS0.SSS0.Px1.p1.1)\.
- Zhouet al\.\(2024\)R\. Zhou, L\. Chen, and K\. YuIs llm a reliable reviewer? a comprehensive evaluation of llm on automatic paper reviewing tasks\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),pp\. 9340–9351\.Cited by:[§2](https://arxiv.org/html/2609.03943#S2.SS0.SSS0.Px2.p1.1)\.
- Zhuet al\.\(2025\)M\. Zhu, Y\. Weng, L\. Yang, and Y\. ZhangDeepReview: improving llm\-based paper review with human\-like deep thinking process\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 29330–29355\.Cited by:[§2](https://arxiv.org/html/2609.03943#S2.SS0.SSS0.Px1.p1.1)\.
- Zouet al\.\(2026\)Z\. Zou, A\. Ansari, D\. C\. Zhang, D\. Lee, and W\. YinDIAGPaper: diagnosing valid and specific weaknesses in scientific papers via multi\-agent reasoning\.arXiv preprint arXiv:2601\.07611\.Cited by:[§2](https://arxiv.org/html/2609.03943#S2.SS0.SSS0.Px1.p1.1)\.