Articulate Intuition or Genuine Analysis? Benchmarking Epistemic Reliability in LLM-as-a-Judge Peer Reviews

# Articulate Intuition or Genuine Analysis? Benchmarking Epistemic Reliability in LLM-as-a-Judge Peer Review
Source: [https://arxiv.org/html/2607.10511](https://arxiv.org/html/2607.10511)
###### Abstract

When an LLM judge calls a peer review “analytical” and a human committee calls another review “high quality,” are they tracking the same thing? We argue they are not, and that the difference matters philosophically\. We operationalise Kahneman’s dual\-process theory into a structured rubric for peer review and releaseKahneman4Review, a benchmark of3,5633\{,\}563rated reviews scored along nine theoretically motivated textual dimensions, eight bias diagnostics, and a continuous reasoning\-quality score\. Three findings bear on*trustworthiness*: decision tier is not detectably aligned with the rubric’s text\-grounded epistemic\-quality proxy; public\-showcase agentic reviews receive higher raw scores than pooled human reviews, but length and venue explain most of the gap and the samples are not paper\-paired; and ICLR review\-text diagnostics shift at the 2022–2023 transition, temporally coincident with widespread LLM availability but without identifying its cause\. A matched function\-probe pilot further shows that the rubric distinguishes textual probes designed to contrast genuine fault\-finding with surface fluency\. We argue that a trustworthy reliability benchmark for LLM judges must separate*analytical form*from*epistemic function*, and propose concrete design choices toward that goal\. An interactive demo is available at[https://huggingface\.co/spaces/nuojohnchen/Kahneman4Review](https://huggingface.co/spaces/nuojohnchen/Kahneman4Review)\.

## 1Introduction

Peer review is the most widely used epistemic gatekeeping mechanism in science, and large language models are increasingly proposed both to*produce*reviews[13](https://arxiv.org/html/2607.10511#bib.bib19),[10](https://arxiv.org/html/2607.10511#bib.bib16)and to*judge*review quality[24](https://arxiv.org/html/2607.10511#bib.bib10),[2](https://arxiv.org/html/2607.10511#bib.bib18)\. Deploying LLMs in either role invites a philosophical question that the machine learning community rarely poses directly:what counts as a good review?An answer is usually smuggled in through training objectives and benchmark design \(acceptance prediction, score agreement, human\-preference alignment\), yet philosophers have long argued that epistemic quality is not a scalar outcome but a property of the reasoning itself[8](https://arxiv.org/html/2607.10511#bib.bib6),[14](https://arxiv.org/html/2607.10511#bib.bib7),[4](https://arxiv.org/html/2607.10511#bib.bib3)\. A review may be technically correct and still textually under\-justified; a negative review may be high\-quality if it is grounded, falsifiable, and responsive to rebuttal\.

Peer review offers an unusually clean test bed for this problem\. First, the review is a written artefact amenable to fine\-grained analysis\. Second, there exist outcomes \(acceptance, meta\-review, AC flags\) that are typically treated as ground truth but which our data will show are*not detectably aligned*with the rubric’s text\-grounded epistemic\-quality proxy\. Third, the recent appearance of agentic AI reviewers[10](https://arxiv.org/html/2607.10511#bib.bib16),[13](https://arxiv.org/html/2607.10511#bib.bib19)creates a natural textual contrast between reviews written by humans under deadline and outputs from retrieval\-augmented LLM pipelines with full paper access\. The contrast motivates a dual\-process\-inspired question for ML evaluation: can an LLM judge identify textual evidence that separates*articulate intuition*from*genuine analysis*without mistaking analytical form for verified epistemic function?

Contributions\.\(1\)We propose nine theoretically motivated textual dimensions, eight bias diagnostics, a four\-label taxonomy, and a continuous reasoning\-quality score, grounded in analytic epistemology and philosophy of explanation\.\(2\)We rate3,5633\{,\}563reviews usingclaude\-sonnet\-4\-6as the judge:1,1551\{,\}155stratified\-sample human reviews from ICLR, ICML, and NeurIPS 2025;2,3072\{,\}307ICLR 2021/2022/2023 human reviews for the longitudinal study; and101101\{\}reviews from an open agentic AI\-reviewer pipeline\. The released code, rubric prompt, ratings, and provenance metadata are available through the[Kahneman4Review demo](https://huggingface.co/spaces/nuojohnchen/Kahneman4Review)\.\(3\)We report three empirical findings bearing directly ontrustworthiness: \(a\) analytical\-trace scores exhibit a field\-wide ceiling across venues \(RQS∈\[2\.80,2\.94\]\\in\[2\.80,2\.94\]\); \(b\) decision tier does not predict RQS; \(c\) public\-showcase agentic and pooled human reviews are statistically separable on precisely those dimensions trained LLMs can fluently imitate\.\(4\)Although the rubric proposes nine theoretically distinct dimensions, the current LLM judge collapses them into one dominant analytical\-trace factor\. We treat this as a*failure\-mode finding*for textual benchmarks, not evidence of multidimensional measurement\.

## 2Related Work

Peer\-review analysishas focused largely on outcome signals: score agreement, bias by author identity, and strategic behaviour[23](https://arxiv.org/html/2607.10511#bib.bib15),[22](https://arxiv.org/html/2607.10511#bib.bib12),[20](https://arxiv.org/html/2607.10511#bib.bib13),[12](https://arxiv.org/html/2607.10511#bib.bib14)\. These framings implicitly equate review reliability with inter\-rater consistency or score\-validity\. We instead operationalise review quality as a property of the reasoning text\.

LLM\-as\-a\-judgesystems are used for preference scoring[24](https://arxiv.org/html/2607.10511#bib.bib10),[2](https://arxiv.org/html/2607.10511#bib.bib18),[15](https://arxiv.org/html/2607.10511#bib.bib11), and recent work documents their systematic biases, including self\-recognition and in\-distribution favoritism[17](https://arxiv.org/html/2607.10511#bib.bib26)\. Concurrent work[13](https://arxiv.org/html/2607.10511#bib.bib19)uses an LLM to*generate*review feedback; our rubric is instead a structured*judge*whose outputs are auditable per dimension and per span, so the question of judge reliability can be posed at finer granularity than scalar agreement\.

Philosophy and ML evaluation\.Work in interpretable ML has increasingly borrowed from philosophy and cognitive science[16](https://arxiv.org/html/2607.10511#bib.bib8),[3](https://arxiv.org/html/2607.10511#bib.bib9)\. Our framing draws on analytic accounts of explanation[8](https://arxiv.org/html/2607.10511#bib.bib6),[14](https://arxiv.org/html/2607.10511#bib.bib7), understanding[4](https://arxiv.org/html/2607.10511#bib.bib3), and epistemic luck[19](https://arxiv.org/html/2607.10511#bib.bib4), and connects to FAccT\-adjacent discussions of trust and reliability[9](https://arxiv.org/html/2607.10511#bib.bib25)that treat trustworthiness as more than predictive accuracy\.

## 3A Dual\-Process Rubric for Review Epistemics

From theory to rubric\.Kahneman[11](https://arxiv.org/html/2607.10511#bib.bib1)distinguishesSystem 1\(fast, associative, impression\-based\) fromSystem 2\(slow, deliberative, evidence\-integrating\)\. We use this distinction as inspiration for textual diagnostics of epistemic reasoning: System\-1\-like traces include compressed impressions and unsupported labels, whereas System\-2\-like traces include explicit evidence linkage, rebuttability, and uncertainty handling[5](https://arxiv.org/html/2607.10511#bib.bib2),[7](https://arxiv.org/html/2607.10511#bib.bib22)\. These labels describe textual traces relevant to articulate intuition and genuine analysis, not the reviewer’s cognitive state\. Without paper access, they cannot establish that genuine analysis occurred or that a critique is correct\.

Dimensions\.Each review is scored on nine 0–3 dimensions\. Two \(impression reliance, heuristic shortcut usage\) are System\-1\-valenced; the other seven are System\-2\-valenced\. Each dimension anchors to a specific philosophical desideratum:falsifiabilitytoPopper[18](https://arxiv.org/html/2607.10511#bib.bib23),evidence specificityto inference\-to\-the\-best\-explanation[14](https://arxiv.org/html/2607.10511#bib.bib7),uncertainty handlingto the contrastive view of understanding[4](https://arxiv.org/html/2607.10511#bib.bib3),issue prioritisationto virtue epistemology[21](https://arxiv.org/html/2607.10511#bib.bib5),reasoning densityto empirical adequacy[6](https://arxiv.org/html/2607.10511#bib.bib24),reasoning chainto the structure of explanation[8](https://arxiv.org/html/2607.10511#bib.bib6),heuristic shortcutto fast\-and\-frugal cognition[7](https://arxiv.org/html/2607.10511#bib.bib22)\. The full map, with rubric cues, is in[Table˜1](https://arxiv.org/html/2607.10511#S3.T1)\.

Table 1:Nine rubric dimensions and their philosophical anchor points\. The third column is the rubric cue the judge looks for\.DimensionAnchorCueImpression rel\.†intuitive judgment[11](https://arxiv.org/html/2607.10511#bib.bib1)global labels \(“novel”\)Reasoning chainargument structure[8](https://arxiv.org/html/2607.10511#bib.bib6)premise→\\toconclusionEvidence specificityIBE[14](https://arxiv.org/html/2607.10511#bib.bib7)numbers, eqns\., quotesFalsifiabilitydemarcation[18](https://arxiv.org/html/2607.10511#bib.bib23)concrete refutabilityHeuristic use†fast\-and\-frugal[7](https://arxiv.org/html/2607.10511#bib.bib22)venue\-fit shortcutsUncertainty handlingunderstanding[4](https://arxiv.org/html/2607.10511#bib.bib3)conditionals, updatingIssue prioritisationvirtue epist\.[21](https://arxiv.org/html/2607.10511#bib.bib5)core vs\. peripheralReasoning densityempirical adequacy[6](https://arxiv.org/html/2607.10511#bib.bib24)support per claimCore criticalityrebuttability[18](https://arxiv.org/html/2607.10511#bib.bib23),[14](https://arxiv.org/html/2607.10511#bib.bib7)decision\-moving
†System\-1\-valenced: low scores indicate analytical reasoning\.

Continuous scores\.The judge emits two textual\-trace scalarss1,s2∈\[1,5\]s\_\{1\},s\_\{2\}\\\!\\in\\\!\[1,5\]and a Reasoning Quality ScoreRQS∈\[1,5\]\\text\{RQS\}\\\!\\in\\\!\[1,5\]intended to separate impression\-like form from explicitly justified form\. The labelℓ∈\{System 1,System 2,Mixed,Non\-evaluative\}\\ell\\in\\\{\\textsc\{System 1\},\\textsc\{System 2\},\\textsc\{Mixed\},\\textsc\{Non\-evaluative\}\\\}is a discretisation of the\(s1,s2,RQS\)\(s\_\{1\},s\_\{2\},\\text\{RQS\}\)joint\. We explicitly require a*nearest alternative label*and a counterfactual justification \(“why not X?”\), which is the rubric’s nod to contrastive explanation[14](https://arxiv.org/html/2607.10511#bib.bib7),[16](https://arxiv.org/html/2607.10511#bib.bib8)\.

Bias diagnostics\.Eight bias categories \(Checklist Inflation, Representativeness, Question Substitution, Conclusion\-First, Overconfidence, Narrative Fallacy, Authority Substitution, Confirmation Bias\) are detected as open\-set tags\.

What does the rubric actually measure?A principal\-components analysis of the nine dimensions across the1,1551\{,\}155cross\-venue human ratings finds a single dominant factor explaining 65% of variance; one eigenvalue exceeds the Kaiser threshold, maximum off\-diagonal correlation is0\.910\.91, and PC1 loads positively on every sign\-flipped dimension \([Figure˜7](https://arxiv.org/html/2607.10511#A2.F7)\)\. Thus, although the rubric is theoretically multidimensional, the present judge does not recover nine empirically distinct constructs\. We treat this as a*failure\-mode finding*: the ratings collapse to a near\-unidimensional analytical\-trace axis, so a well\-prompted LLM may saturate the benchmark globally rather than on one axis \(returned to in[Section˜5](https://arxiv.org/html/2607.10511#S5)\)\.

## 4Empirical Study

### 4\.1Data and protocol

We stratified\-sample 100 papers per venue from ICLR, ICML, NeurIPS 2025 by tier \(6 oral, 10 spotlight, 84 poster\), yielding1,1551\{,\}155human reviews; we additionally rate 200 random papers each from ICLR 2021/2022/2023 \(2,3072\{,\}307further reviews\) under the same judge for the longitudinal study, and101101\{\}reviews from a public showcase of an open agentic\-reviewer pipeline[10](https://arxiv.org/html/2607.10511#bib.bib16),[1](https://arxiv.org/html/2607.10511#bib.bib17)\(3,5633\{,\}563ratings in total\) \(full provenance and the curation caveat are in[Appendix˜G](https://arxiv.org/html/2607.10511#A7)\)\. All ratings useclaude\-sonnet\-4\-6with a frozen prompt\. The LLM judge rates each review from its text, venue, and rating metadata alone; the reviewed paper is not included in the judge’s input\. We use Mann–Whitney U, Kruskal–Wallis, one\-way ANOVA, andχ2\\chi^\{2\}as appropriate; per\-dimension comparisons use Bonferroni correction for 9 tests\.

![Refer to caption](https://arxiv.org/html/2607.10511v1/x1.png)Figure 1:Label distribution across three human venues and the Stanford agentic reviewer\. Venue\-level differences are significant \(χ2,p<10−4\\chi^\{2\},\\ p<10^\{\-4\}\)\.
### 4\.2Analytical\-trace scores show a field\-wide ceiling and temporal decline

Mean RQS in 2025 is2\.802\.80\{\}\(ICLR\),2\.832\.83\{\}\(ICML\),2\.942\.94\{\}\(NeurIPS\), on a 1–5 scale; venue\-level Kruskal–Wallis is significant \(p=0\.003p=0\.003\) but the absolute spread is small\. ICLR is a statistical outlier inlabeldistribution:35%35\\%of ICLR reviews are System 1 vs\.21%21\\%\(ICML\) and20%20\\%\(NeurIPS\) \([Figure˜1](https://arxiv.org/html/2607.10511#S4.F1)\)\.

Temporal trend\.Rating ICLR 2021, 2022, and 2023 with the same judge and prompt as 2025 \(totaln=2,683n\{=\}2\{,\}683reviews\) reveals a discontinuity at the 2022–2023 transition, temporally coincident with widespread LLM availability: mean RQS is identical in 2021 and 2022 \(3\.033\.03vs3\.033\.03\), then drops to2\.912\.91\(2023\) and2\.802\.80\(2025\); the System\-1\-like share triples,11%→35%11\\%\\to 35\\%; the System\-2\-like share nearly halves,23%→16%23\\%\\to 16\\%\([Figure˜2](https://arxiv.org/html/2607.10511#S4.F2)\)\. K–Wp<10−4p<10^\{\-4\}, labelχ2\\chi^\{2\}p<10−4p<10^\{\-4\}\. Length\-controlled regression with 2021 as reference andlog\\log\-length included puts*2022 vs 2021 at−0\.03\-0\.03RQS \(p=0\.20p\{=\}0\.20, n\.s\.\)*but*2023 vs 2021 at−0\.15\-0\.15\(p<10−8p<10^\{\-8\}\)*and*2025 vs 2021 at−0\.21\-0\.21\(p<10−11p<10^\{\-11\}\)*\. This pattern is descriptive, not causal: reviewer composition, review policy, conference scale, changing norms, and LLM\-assisted drafting are all possible explanations\.

![Refer to caption](https://arxiv.org/html/2607.10511v1/x2.png)Figure 2:ICLR review\-text diagnostics across years\. Left: label distribution shifts from fewer System\-1\-like and more System\-2\-like traces \(2021/2022\) toward a more System\-1\-like mix \(2025\)\. Right: RQS distribution declines monotonically\. K–Wp<10−4p<10^\{\-4\}; length\-adjusted residual gap−0\.21\-0\.21RQS,p<10−11p<10^\{\-11\}\.
### 4\.3Outcomes do not predict the epistemic\-quality proxy

We find no detectable main effect of acceptance tier on RQS, pooled across venues \(p=0\.260p=0\.260,[Figure˜3](https://arxiv.org/html/2607.10511#S4.F3)\); within each venue, tier means fall within 0\.2 RQS points\. This null is informative under either reading:*either*the conference system’s most visible outcome signal does not track the rubric’s textual proxy,*or*the rubric and outcome are tracking related properties at a level ournncannot resolve\. In either case, tier agreement is at best a weak proxy for articulated reasoning quality in our data\.

![Refer to caption](https://arxiv.org/html/2607.10511v1/x3.png)Figure 3:Mean RQS by acceptance tier per venue\. Sample sizes: ICLR/ICML/NeurIPS Oraln=23n=23/2323/2626, Spotlight3636/4040/4343, Poster317317/313313/334334\. Within\-venue differences are within standard error; pooled one\-way ANOVA is null \(p=0\.260p=0\.260\)\.As an auxiliary probe, we rate 149 reviewers whose quality was explicitly commented on by area chairs\. AC\-flagged negatively\-commented reviewers \(n=37n\{=\}37\) and their same\-paper peers \(n=109n\{=\}109\) differ on RQS atp=0\.578p=0\.578\(n\.s\.\), observedd=−0\.03d=\-0\.03against a minimum\-detectable effect ofd=0\.53d=0\.53at 80% power\. We do not read this as proof of orthogonality \(our sample is underpowered for\|d\|<0\.5\|d\|<0\.5\), only as evidence that AC\-level scalar judgments and rubric\-level signals are not strongly aligned in our data \([Figure˜4](https://arxiv.org/html/2607.10511#S4.F4)\)\.

![Refer to caption](https://arxiv.org/html/2607.10511v1/x4.png)Figure 4:RQS distribution by AC annotation group from in\-metareview commentary\. The null result \(p=0\.578p=0\.578\{\}, observedd=−0\.03d=\-0\.03\{\}against MDES0\.530\.53\{\}\) shows AC scalar judgments and rubric\-level signals are not strongly aligned in our data; the sample is underpowered to distinguish true orthogonality from a small effect\.
### 4\.4Agentic AI reviews exhibit more System\-2\-like traces: mostly via length

Stanford agentic\-reviewer outputs are classified System 2 at74%74\\%versus21%21\\%for pooled humans\. Mean RQS is3\.503\.50versus2\.862\.86: Cohen’sd=1\.26d=1\.26with 95% bootstrap CI\[\+1\.19,\+1\.33\]\[\+1\.19,\+1\.33\]\(2000 resamples\)\. Per\-dimension effect sizes are large and tightly bounded:*evidence specificity*d=\+1\.57d=\+1\.57\[\+1\.50, \+1\.67\],*falsifiability*d=\+1\.39d=\+1\.39\[\+1\.31, \+1\.47\], and*impression reliance*d=−1\.23d=\-1\.23\[\-1\.38, \-1\.08\] \([Figure˜5\(b\)](https://arxiv.org/html/2607.10511#S4.F5.sf2)\)\.All 9/9 dimensions remain significant atα=0\.05\\alpha\{=\}0\.05after Bonferroni correction\.Across the full RQS distributions \([Figure˜5\(a\)](https://arxiv.org/html/2607.10511#S4.F5.sf1)\), the agentic reviewer’s scores are tight and elevated, while the human venues are broader and centred lower\.

Because these public\-showcase outputs are potentially curated and not paper\-paired, this result is not evidence that AI reviewers are epistemically better than human reviewers\.

![Refer to caption](https://arxiv.org/html/2607.10511v1/x5.png)\(\(a\)\)
![Refer to caption](https://arxiv.org/html/2607.10511v1/x6.png)\(\(b\)\)

Figure 5:Human reviewers vs\. the Stanford agentic reviewer\.\(a\)RQS distribution per group \(violin, mean line\): the public\-showcase agentic distribution has higher raw RQS \(mean 3\.50, SD 0\.12\), while humans are broader \(means∈\[2\.80,2\.94\]\\in\[2\.80,2\.94\], SDs∈\[0\.48,0\.57\]\\in\[0\.48,0\.57\]\)\. This unpaired descriptive comparison does not establish reviewer competence\.\(b\)Mean score per rubric dimension; error bars±1\\pm 1SD;†\{\\dagger\}marks System\-1\-valenced dimensions, for which lower is more analytical\. All 9 dimensions differ atp<0\.05p<0\.05after Bonferroni correction, with\|d\|≥0\.54\|d\|\\geq 0\.54in every case\.Length is a confound\.Agentic outputs are systematically longer than human reviews\. To separate the analytical\-trace score from verbosity we fit

RQSi=β0\+β1​1​\[AI\]i\+β2​log⁡\(lengthi\)\+𝜸⊤​𝑽i\+εi,\\text\{RQS\}\_\{i\}=\\beta\_\{0\}\+\\beta\_\{1\}\\,\\mathbb\{1\}\[\\text\{AI\}\]\_\{i\}\+\\beta\_\{2\}\\log\(\\text\{length\}\_\{i\}\)\+\\bm\{\\gamma\}^\{\\top\}\\bm\{V\}\_\{i\}\+\\varepsilon\_\{i\},\(1\)where𝑽i\\bm\{V\}\_\{i\}is a one\-hot venue vector \(Stanford = reference\)\. This shrinks the AI coefficient from a naive\+0\.65\+0\.65toβ^1=\+0\.17\\hat\{\\beta\}\_\{1\}=\+0\.17\(95% CI\[\+0\.03,\+0\.31\]\[\+0\.03,\+0\.31\],p=0\.014p\{=\}0\.014\), withβ^2=\+0\.36\\hat\{\\beta\}\_\{2\}=\+0\.36per log\-character; length and venue jointly explain74%74\\%of the naive gap\. The residual signal is real but small, and the “System 2” label should be read with this control in mind\.

Bias patterns\.Bias tags co\-occur non\-trivially: 83% of human reviews carry at least one bias tag, and the strongest pairwise association \(ϕ=0\.14\\phi=0\.14\) is between Representativeness Heuristic and Question Substitution \(fullϕ\\phimatrix in[Figure˜9](https://arxiv.org/html/2607.10511#A2.F9)\)\. This is a modest co\-occurring System\-1\-like trace pattern: a review that substitutes an easier question for the one posed by the paper also tends to answer it via representativeness\. The Stanford agentic reviewer is dominated by Checklist Inflation \(45%\) with some Representativeness \(8%\) and never Question Substitution or Authority Substitution \([Figure˜6](https://arxiv.org/html/2607.10511#S4.F6)\)\.

![Refer to caption](https://arxiv.org/html/2607.10511v1/x7.png)Figure 6:Bias\-diagnostic frequency \(% of reviews in which the tag appears\), per group\.Checklist Inflationdominates in all human venues; the agentic reviewer is also dominated byChecklist Inflation\(45%, less than humans\) with someRepresentativeness Heuristic\(8%\) and essentially noQuestion SubstitutionorAuthority Substitution\.This AI\-vs\-human pattern, with the sampling and length\-control caveats, is the paper’s central diagnostic finding\.

Measurement validation\.On a fixedn=100n\{=\}100subset \([Appendix˜C](https://arxiv.org/html/2607.10511#A3)for the protocol\), frozen\-prompt test\-retest ofclaude\-3\-7\-sonnetgivesκ=0\.77\\kappa\{=\}0\.77,αord=0\.94\\alpha\_\{\\text\{ord\}\}\{=\}0\.94; cross\-judge vs\.gpt\-4ogivesκ=0\.58\\kappa\{=\}0\.58,α=0\.76\\alpha\{=\}0\.76\([Table˜2](https://arxiv.org/html/2607.10511#S4.T2)\)\. The earlierκ=0\.09\\kappa\{=\}0\.09figure was a prompt\-revision artefact\. Cross\-*version*agreement is the weakest link \(κ=0\.14\\kappa\{=\}0\.14betweensonnet\-4\-6andsonnet\-3\.7\), so reliability claims about LLM judges should be reported as model\-conditional throughout\.

Table 2:Inter\-run, cross\-judge, and cross\-version agreement on the fixedn=100n\{=\}100validation subset, with the originalclaude\-sonnet\-4\-6ratings as a comparator\. After dropping malformed\-JSON outputs,n=76n\{=\}76reviews are paired across all three rating conditions\.Comparisonnnlabel agreeκ\\kappaαord\\alpha\_\{\\text\{ord\}\}\(RQS\)Sonnet\-3\.7 run1 vs\. run2 \(test\-retest\)7684%0\.770\.94Sonnet\-3\.7 vs\. GPT\-4o \(cross\-judge\)8471%0\.580\.76Sonnet\-3\.7 vs\. Sonnet\-4\-6 \(cross\-version\)8533%0\.140\.23Function\-probe pilot\.Forn=35n\{=\}35ICLR\-2025 papers,claude\-3\-7\-sonnetidentifies one falsifiable concern and emits two∼\\sim250\-word reviews matched on length and specificity markers: review A’s primary weakness IS that concern, B’s are surface peripheral asks\. Result:33/3533/35haveRQS​\(A\)\>RQS​\(B\)\\text\{RQS\}\(A\)\{\>\}\\text\{RQS\}\(B\),dz=\+2\.52d\_\{z\}\{=\}\+2\.52, sign\-testp<\.001p<\.001; all A\-reviews labeledSystem 2, only 26% of B’s\. Caveat: same model family generates and rates; curated public\-errata faults are the necessary follow\-up\.

## 5Discussion: Analytical Form and Epistemic Function

The rubric separates venues and yields auditable dimension\- and span\-level outputs, but it also gives public\-showcase agentic outputs high scores on dimensions a well\-prompted LLM can manufacture\. Because PCA shows these dimensions moving together, we treat the rubric as a*partially\-validated diagnostic of textual\-benchmark failure*, not a settled reliability metric\.

An LLM reviewer that fluently produces theformof analytical discourse without establishing that genuine analysis was performed is the modern epistemic analogue of an empirically adequate theory that is not explanatory[8](https://arxiv.org/html/2607.10511#bib.bib6),[6](https://arxiv.org/html/2607.10511#bib.bib24),[14](https://arxiv.org/html/2607.10511#bib.bib7);Elgin[4](https://arxiv.org/html/2607.10511#bib.bib3)’s account of understanding as competence\-over\-a\-domain says the relevant competence is saying what would change one’s verdict:*attempting to falsify*, not paraphrase\. Our probe pilot shows that the rubric distinguishes textual signatures designed to contrast genuine fault\-finding with surface fluency under matched form, but the AI\-vs\-human gap indicates that much of the headline signal still rides on analytical form rather than verified epistemic function\.

What a trustworthy LLM judge would look like\.Three design choices suggested by the results: \(i\)*Audit by span, not by score*, since judgments without text spans are unfalsifiable[18](https://arxiv.org/html/2607.10511#bib.bib23); \(ii\)*Require a counterfactual label*, forcing contrastive reasoning[14](https://arxiv.org/html/2607.10511#bib.bib7),[16](https://arxiv.org/html/2607.10511#bib.bib8); \(iii\)*Separate analytical form from epistemic function via probes*: ourn=35n\{=\}35pilot finds the rubric discriminates33/3533/35matched pairs in favour of the genuine\-fault condition \(dz=\+2\.52d\_\{z\}\{=\}\+2\.52,p<\.001p<\.001\), and scaling to a curated 200\-item probe set with public\-errata faults is the natural next step\.

## 6Conclusion

Frozen\-prompt ratings are stable and discriminate33/3533/35matched\-form probes, but decision tier does not predict RQS and length and venue explain74%74\\%of the naive AI–human gap\. These scores capture articulated traces, not verified epistemic function: without the reviewed paper, they cannot establish genuine analysis, critique correctness or fidelity, or reviewer competence\. Further limitations are judge self\-favoritism[17](https://arxiv.org/html/2607.10511#bib.bib26), model\-version sensitivity \(κ=0\.14\\kappa\{=\}0\.14\), and omitted social epistemology and causal modelling; future work needs model\-conditional reporting, public\-errata probes, and expert anchors\.

## Impact Statement

This work aims to make LLM\-assisted peer review more auditable\. A judge rewarding only linguistic surface form could legitimise AI review pipelines whose analytical competence is overstated; function probes and span\-level audits are intended to resist that risk\.

## References

- Analemma Intelligence \(2026\)FARS: fully automated research system\.Note:[https://analemma\.ai/fars/](https://analemma.ai/fars/)Public showcase corpus; reviews generated by the Stanford Agentic Reviewer \(paperreview\.ai\)\. Accessed 2026\-06\-12Cited by:[Appendix G](https://arxiv.org/html/2607.10511#A7.p2.1.2.1),[§4\.1](https://arxiv.org/html/2607.10511#S4.SS1.p1.5.1.1)\.
- C\. Chiang and H\. Lee \(2023\)Can large language models be an alternative to human evaluations?\.InAnnual Meeting of the Association for Computational Linguistics \(ACL\),Cited by:[§1](https://arxiv.org/html/2607.10511#S1.p1.1.4.1),[§2](https://arxiv.org/html/2607.10511#S2.p2.1.2.1)\.
- F\. Doshi\-Velez and B\. Kim \(2017\)Towards a rigorous science of interpretable machine learning\.arXiv preprint arXiv:1702\.08608\.Cited by:[§2](https://arxiv.org/html/2607.10511#S2.p3.1.2.1)\.
- C\. Z\. Elgin \(2017\)True enough\.MIT Press\.Cited by:[§1](https://arxiv.org/html/2607.10511#S1.p1.1.6.1),[§2](https://arxiv.org/html/2607.10511#S2.p3.1.4.1),[Table 1](https://arxiv.org/html/2607.10511#S3.T1.3.3.7.2.2.1),[§3](https://arxiv.org/html/2607.10511#S3.p2.1.7.1),[§5](https://arxiv.org/html/2607.10511#S5.p2.1)\.
- J\. St\. B\. T\. Evans and K\. E\. Stanovich \(2013\)Dual\-process theories of higher cognition: advancing the debate\.Perspectives on Psychological Science\.Cited by:[§3](https://arxiv.org/html/2607.10511#S3.p1.1.4.1)\.
- Bas\. C\. v\. Fraassen \(1980\)The scientific image\.Oxford University Press\.External Links:ISBN 9780198244271,[Document](https://dx.doi.org/10.1093/0198244274.001.0001),[Link](https://doi.org/10.1093/0198244274.001.0001)Cited by:[Table 1](https://arxiv.org/html/2607.10511#S3.T1.3.3.9.2.2.1),[§3](https://arxiv.org/html/2607.10511#S3.p2.1.11.1),[§5](https://arxiv.org/html/2607.10511#S5.p2.1.2.1)\.
- G\. Gigerenzer and W\. Gaissmaier \(2011\)Heuristic decision making\.Annual Review of Psychology62,pp\. 451–482\.Cited by:[Table 1](https://arxiv.org/html/2607.10511#S3.T1.3.3.3.2.2.1),[§3](https://arxiv.org/html/2607.10511#S3.p1.1.4.1),[§3](https://arxiv.org/html/2607.10511#S3.p2.1.15.1)\.
- C\. G\. Hempel and P\. Oppenheim \(1948\)Studies in the logic of explanation\.Philosophy of Science15\(2\),pp\. 135–175\.Cited by:[§1](https://arxiv.org/html/2607.10511#S1.p1.1.6.1),[§2](https://arxiv.org/html/2607.10511#S2.p3.1.3.1),[Table 1](https://arxiv.org/html/2607.10511#S3.T1.2.2.2.3.2.1),[§3](https://arxiv.org/html/2607.10511#S3.p2.1.13.1),[§5](https://arxiv.org/html/2607.10511#S5.p2.1.2.1)\.
- A\. Jacovi, A\. Marasović, T\. Miller, and Y\. Goldberg \(2021\)Formalizing trust in artificial intelligence: prerequisites, causes and goals of human trust in AI\.InProceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency \(FAccT\),External Links:[Document](https://dx.doi.org/10.1145/3442188.3445923)Cited by:[§2](https://arxiv.org/html/2607.10511#S2.p3.1.6.1)\.
- Y\. Jiang and A\. Ng \(2025\)Stanford agentic reviewer\.Note:[https://paperreview\.ai](https://paperreview.ai/)Stanford Machine Learning Group\. Accessed 2026\-06\-12Cited by:[§1](https://arxiv.org/html/2607.10511#S1.p1.1.2.1),[§1](https://arxiv.org/html/2607.10511#S1.p2.1.2.1),[§4\.1](https://arxiv.org/html/2607.10511#S4.SS1.p1.5.1.1)\.
- D\. Kahneman \(2011\)Thinking, fast and slow\.Farrar, Straus and Giroux,New York\.Cited by:[Table 1](https://arxiv.org/html/2607.10511#S3.T1.1.1.1.2.2.1),[§3](https://arxiv.org/html/2607.10511#S3.p1.1)\.
- J\. Langford and M\. Guzdial \(2015\)The arbitrariness of reviews, and advice for school administrators\.Communications of the ACM58\(4\),pp\. 12–13\.External Links:[Document](https://dx.doi.org/10.1145/2732417)Cited by:[Appendix G](https://arxiv.org/html/2607.10511#A7.p4.1),[§2](https://arxiv.org/html/2607.10511#S2.p1.1.2.1)\.
- W\. Liang, Y\. Zhang, H\. Cao, B\. Wang, D\. Y\. Ding, X\. Yang, K\. Vodrahalli, S\. He, D\. S\. Smith, Y\. Yin, D\. A\. McFarland, and J\. Zou \(2024\)Can large language models provide useful feedback on research papers? a large\-scale empirical analysis\.NEJM AI1\(8\)\.External Links:[Document](https://dx.doi.org/10.1056/AIoa2400196)Cited by:[§1](https://arxiv.org/html/2607.10511#S1.p1.1.2.1),[§1](https://arxiv.org/html/2607.10511#S1.p2.1.2.1),[§2](https://arxiv.org/html/2607.10511#S2.p2.1.4.1)\.
- P\. Lipton \(2004\)Inference to the best explanation\.2nd edition,Routledge\.Cited by:[§1](https://arxiv.org/html/2607.10511#S1.p1.1.6.1),[§2](https://arxiv.org/html/2607.10511#S2.p3.1.3.1),[Table 1](https://arxiv.org/html/2607.10511#S3.T1.3.3.10.2.2.1),[Table 1](https://arxiv.org/html/2607.10511#S3.T1.3.3.5.2.2.1),[§3](https://arxiv.org/html/2607.10511#S3.p2.1.5.1),[§3](https://arxiv.org/html/2607.10511#S3.p3.4.3.1),[§5](https://arxiv.org/html/2607.10511#S5.p2.1.2.1),[§5](https://arxiv.org/html/2607.10511#S5.p3.4.5.1)\.
- Y\. Liu, H\. Zhou, Z\. Guo, E\. Shareghi, I\. Vulić, A\. Korhonen, and N\. Collier \(2024\)Aligning with human judgement: the role of pairwise preference in large language model evaluators\.InConference on Language Modeling \(COLM\),Cited by:[§2](https://arxiv.org/html/2607.10511#S2.p2.1.2.1)\.
- T\. Miller \(2019\)Explanation in artificial intelligence: insights from the social sciences\.Artificial Intelligence267,pp\. 1–38\.Cited by:[§2](https://arxiv.org/html/2607.10511#S2.p3.1.2.1),[§3](https://arxiv.org/html/2607.10511#S3.p3.4.3.1),[§5](https://arxiv.org/html/2607.10511#S5.p3.4.5.1)\.
- A\. Panickssery, S\. R\. Bowman, and S\. Feng \(2024\)LLM evaluators recognize and favor their own generations\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2607.10511#S2.p2.1.3.1),[§6](https://arxiv.org/html/2607.10511#S6.p1.3.1.1)\.
- K\. Popper \(1959\)The logic of scientific discovery\.Hutchinson,London\.Cited by:[Table 1](https://arxiv.org/html/2607.10511#S3.T1.3.3.10.2.2.1),[Table 1](https://arxiv.org/html/2607.10511#S3.T1.3.3.6.2.2.1),[§3](https://arxiv.org/html/2607.10511#S3.p2.1),[§5](https://arxiv.org/html/2607.10511#S5.p3.4.3.1)\.
- D\. Pritchard \(2005\)Epistemic luck\.Oxford University Press\.Cited by:[§2](https://arxiv.org/html/2607.10511#S2.p3.1.5.1)\.
- N\. B\. Shah \(2022\)Challenges, experiments, and computational solutions in peer review\.Communications of the ACM65\(6\),pp\. 76–87\.External Links:[Document](https://dx.doi.org/10.1145/3528086)Cited by:[§2](https://arxiv.org/html/2607.10511#S2.p1.1.2.1)\.
- E\. Sosa \(2007\)A virtue epistemology: apt belief and reflective knowledge\.Oxford University Press\.Cited by:[Table 1](https://arxiv.org/html/2607.10511#S3.T1.3.3.8.2.2.1),[§3](https://arxiv.org/html/2607.10511#S3.p2.1.9.1)\.
- I\. Stelmakh, N\. B\. Shah, and A\. Singh \(2021\)Catch me if I can: detecting strategic behaviour in peer assessment\.InProceedings of the AAAI Conference on Artificial Intelligence,Cited by:[§2](https://arxiv.org/html/2607.10511#S2.p1.1.2.1)\.
- A\. Tomkins, M\. Zhang, and W\. D\. Heavlin \(2017\)Reviewer bias in single\- versus double\-blind peer review\.Proceedings of the National Academy of Sciences \(PNAS\)114\(48\),pp\. 12708–12713\.Cited by:[§2](https://arxiv.org/html/2607.10511#S2.p1.1.2.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica \(2023\)Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2607.10511#S1.p1.1.4.1),[§2](https://arxiv.org/html/2607.10511#S2.p2.1.2.1)\.

## Appendix ARubric details

Labels\.System 1,System 2,Mixed,Non\-evaluative\. A review is labelledSystem 1if its text is dominated by impression\-like, compressed, and conclusion\-first traces;System 2if it is dominated by explicit, evidence\-grounded, and uncertainty\-sensitive traces;Mixedif both textual patterns are substantively present; andNon\-evaluativeif it is administrative or clarificatory rather than judgemental\. These labels do not identify the reviewer’s cognitive state\.

Dimensions \(0–3\)\.

- •Impression reliance\(S1\-valenced\): reliance on global labels and unsupported overall claims\.
- •Explicit reasoning chain: observable premise–conclusion structure\.
- •Evidence specificity: citation of concrete figures, numbers, equations, or prior work passages\.
- •Falsifiability/rebuttability: whether each concern admits a concrete refutation\.
- •Heuristic shortcut usage\(S1\-valenced\): venue\-fit, familiarity, authority shortcuts\.
- •Uncertainty handling & belief updating: hedging, conditioning, responsiveness to rebuttal in principle\.
- •Issue prioritisation: acceptance\-critical vs\. peripheral weighting\.
- •Reasoning density: analytical support per sentence\.
- •Core criticality: whether weaknesses would actually move a decision if true\.

Continuous scalars\.s1,s2∈\[1,5\]s\_\{1\},s\_\{2\}\\in\[1,5\]are independent scale\-of\-presence scores for System\-1\-like and System\-2\-like textual traces;RQS∈\[1,5\]\\text\{RQS\}\\in\[1,5\]is a judge\-scaled, text\-grounded proxy for epistemic quality independent of conclusion polarity; the reasoning\-balance index iss2−s1s\_\{2\}\-s\_\{1\}; and the System\-2 trace level is a\{0,…,4\}\\\{0,\\ldots,4\\\}coarse marker of visible analytical elaboration\.

Structured output\.The judge emits: the label; the three scalars; dimension scores; a primary rationale; threesupporting\_spanswith signal direction and awhy; up to threecounterevidenceentries; bias diagnostics \(open\-set, drawn from the eight\-bias menu\); anearest\_alternative\_labelpluswhy\_not\_nearest\_alternative; a composite\[0,1\]\[0,1\]score; and three derived indices \(heuristic dominance, analytic strength, mixedness\)\. The full system prompt and implementation are available in the[demo code](https://huggingface.co/spaces/nuojohnchen/Kahneman4Review)\.

## Appendix BAdditional analyses

![Refer to caption](https://arxiv.org/html/2607.10511v1/x8.png)Figure 7:PCA of the 9 rubric dimensions \(S1\-valenced dimensions sign\-flipped\)\. Fitted on human reviews; Stanford agentic reviews projected\. Humans form a broad cluster around the origin; the agentic reviewer is shifted\+1\.7\+1\.7on PC1 \(the single analytical\-trace axis\)\. Arrows are dimension loadings\.![Refer to caption](https://arxiv.org/html/2607.10511v1/x9.png)Figure 8:Latent\(s1,s2\)\(s\_\{1\},s\_\{2\}\)space\. Human reviews cluster on the diagonal; the agentic AI reviewer \(▲\\blacktriangle\) is displaced up\-and\-left: simultaneously lower System\-1 and higher System\-2 traces\.![Refer to caption](https://arxiv.org/html/2607.10511v1/x10.png)Figure 9:Bias co\-occurrence on human reviews\. Diagonal is marginal frequency; off\-diagonal is pairwiseϕ\\phicorrelation\. Positive clusters \(Representativeness↔\\leftrightarrowQuestion Substitution, Conclusion\-First↔\\leftrightarrowOverconfidence\) support a weak co\-occurring System\-1\-like trace pattern; most\|ϕ\|\|\\phi\|values are below 0\.15\.
## Appendix CMeasurement validation details

We re\-rate a fixed 100\-review subset \(deterministic seed 42, balanced by venue and label\) under three conditions: two independent runs ofclaude\-3\-7\-sonnet\-20250219with a frozen system prompt and concurrency 4, and one run ofgpt\-4owith the same prompt\. After dropping the small number of malformed\-JSON outputs,n=76n\{=\}76reviews are paired across all three; the resulting agreement statistics are reported in[Table˜2](https://arxiv.org/html/2607.10511#S4.T2)\.

Two takeaways\. \(i\) The earlierκ=0\.09\\kappa\{=\}0\.09stability figure was an artefact of prompt\-version drift, not the rubric’s actual stability under controlled re\-runs\. \(ii\) Cross\-*version*agreement is the weakest link: with the same frozen prompt,sonnet\-4\-6andsonnet\-3\.7produce materially different label distributions \(notably,sonnet\-3\.7more frequently emitsNon\-evaluativefor short reviews\)\. This is a real model\-version sensitivity, not a deprecation artefact: both models remain available and were used in this paper\. The implication is that some headline numbers in[Section˜4\.4](https://arxiv.org/html/2607.10511#S4.SS4)are model\-version\-specific, and reliability claims about LLM judges should be reported as model\-conditional throughout\.

## Appendix DLength\-controlled regression details

The length\-controlled model of[Equation˜1](https://arxiv.org/html/2607.10511#S4.E1)is fit by OLS with Stanford as the venue reference\. Estimates:β^1=\+0\.17\\hat\{\\beta\}\_\{1\}=\+0\.17\(95% CI\[\+0\.03,\+0\.31\]\[\+0\.03,\+0\.31\],p=0\.014p\{=\}0\.014\);β^2=\+0\.36\\hat\{\\beta\}\_\{2\}=\+0\.36per log\-character\. The naive RQS gap \(AI−\-human\) is\+0\.65\+0\.65\{\}; theβ^1\\hat\{\\beta\}\_\{1\}residual after adjustment is roughly74%74\\%smaller\. Conclusion: length is the largest single explainer of the headline effect, and the residual signal is real but small\.

## Appendix EQualitative case studies

The following two rubric outputs are illustrative, not representative; we chose them because they exemplify the shape of aSystem 1human rating and aSystem 2agentic\-reviewer rating\. Raw review text is not reproduced \(see[Appendix˜G](https://arxiv.org/html/2607.10511#A7)\)\.

Case 1: Human, ICLR 2025 \(System 1, RQS 1\.8\)\.*Primary rationale\.*“The strengths section compresses the entire evaluation into two abstract labels \(‘novel’ and ‘rigorous’\) without any supporting reasoning or evidence\. The weaknesses are peripheral completeness requests \(missing conclusion section, statistical tests\) that are entirely disconnected from the paper’s core claims about constrained optimization and weak supervision\. There is no analytical linkage between any concern raised and the validity of the paper’s central methodology\.”*Bias diagnostics:*Question Substitution,Checklist Inflation\.*Nearest alternative:*Mixed\(rejected because the statistical\-testing suggestion is the only partially analytical point, and it is brief and underdeveloped\)\.

Case 2: Stanford agentic reviewer, RQS 3\.7\.*Primary rationale\.*“The passage is dominated by explicit, operational critique grounded in methodological reasoning\. The weaknesses section in particular articulates specific technical concerns \(Bayesian credible intervals used as frequentist confidence bounds under adaptive sampling, lack of union\-over\-time error control, absence of formal sample\-complexity bounds\), each with a clear causal link to why the gap undermines the paper’s central claims\.”*Bias diagnostics:*none\.*Nearest alternative:*Mixed\.

The pair supports the paper’s central point: the agentic output exhibits the textual*form*of methodological critique at a level the human rating does not\. Whether the specific claims in Case 2 are*correct*\(whether adaptive\-sampling intervals are really what the paper used, whether the cited concerns actually obtain\) cannot be read off the rubric, which is why dimension\-level scores on their own are insufficient for a trustworthiness benchmark\.

## Appendix FShort and template\-inflated reviews

A useful sanity check on what the rubric is responding to: do reviews that humans would intuitively recognise as textually underdeveloped map toSystem 1? We find a partial yes\.

Ultra\-short reviews\(bottom 10% of our sample by length,≤495\\leq 495\{\}characters;n=116n=116\): mean RQS2\.662\.66vs2\.882\.88for the rest;34%34\\%are labelledSystem 1vs24%24\\%\. The dominant bias is*Checklist Inflation*\(59%59\\%of ultra\-short reviews carry the tag\), followed by*Question Substitution*, the same pair that drives the broader co\-occurring System\-1\-like trace pattern \([Figure˜9](https://arxiv.org/html/2607.10511#A2.F9)\)\.

Template\-inflated long reviews\.A different textual failure mode is the “40 numbered weaknesses, 40 numbered questions” template, in which the surface specificity is high but the items are uniform peripheral asks \(“please report standard deviations,” “please add multilingual evaluation,” “please compare with method X”\)\. Two anonymised excerpts illustrate the pattern\. Both reviewers gave the paper a low rating with maximum confidence\.

Anonymised excerpt 1\(single\-paragraph summary, then 40 numbered weaknesses, 40 numbered questions, near 1\-to\-1 mirrored\):“\(31\) The paper does not report the standard deviation of experimental results\. \(32\) The paper’s handling of background categories is unclear\. …\(38\) The paper’s geometric feature calculation is not detailed\. \(39\) The paper does not compare with multi\-scale feature fusion methods\. \(40\) The paper’s conclusion overstates its contributions\.”

Anonymised excerpt 2\(40 numbered weaknesses \+ 40 mirrored questions on a different paper\):“\(15\) Inference speed on different hardware platforms is not provided\. …\(28\) Latency introduced by visual memory replay is not discussed\. …\(40\) Performance in multilingual scenarios is not tested\.”

Both reviewers issued ratings of “4 \(marginally below acceptance\) / confidence 5 \(absolutely certain\),” a confidence\-rating combination that is itself a System\-1\-like textual signature \(*Overconfidence*co\-occurring with*Conclusion\-First Justification*\)\. The rubric labels both asSystem 1and tags both withChecklist InflationandQuestion Substitution\. A scan over all ICLR\-2025 reviews in the bundle finds only nine with≥\\geq30 numbered items in this template\-inflated style; this is rare in absolute terms but is exactly the failure mode the rubric is designed to surface\.

Caveat: brevity is not low quality\.Ultra\-short reviews are not labelledSystem 1mechanically:16%16\\%carry no bias tag in our sample\. Some short reviews are terse precisely because the paper is straightforward and a long analysis is unwarranted\. The rubric responds to the*shape*of the analytic move, not to length per se; the length\-controlled regression of[Appendix˜D](https://arxiv.org/html/2607.10511#A4)confirms that, even within length strata, the signal that distinguishes the agentic AI reviewer from humans persists at smaller magnitude\.

## Appendix GData, code, and reproducibility

Human reviews\.ICLR/ICML/NeurIPS 2025 reviews are drawn from publicly\-available OpenReview decision pages; rating files reference reviews by OpenReview paper ID and reviewer ID\. We do not redistribute review text verbatim due to OpenReview licence constraints\. The longitudinal subset \(ICLR 2021/2022/2023\) follows the same convention\.

AI reviews\.The 101 agentic\-reviewer outputs are drawn from a public showcase of an open agentic\-reviewer system[1](https://arxiv.org/html/2607.10511#bib.bib17), downloaded as a 102\-file plaintext archive \(FA0001–FA0234, with gaps\); source details are available in the[demo code](https://huggingface.co/spaces/nuojohnchen/Kahneman4Review)\. The reviews are produced by submitting recent ML papers to the system; the corresponding source papers are not identified in the archive, so we use the reviews as a population\-level distribution rather than as paper\-paired comparisons to our human sample\. We treat the resulting AI\-vs\-human gap as an upper bound on agentic\-reviewer analytical\-trace prevalence: if the showcase is curated for “interesting” or “high\-quality” reviews, our sample over\-represents the system’s better outputs\.

Code\.The released rubric prompt, implementation, ratings, and provenance information are available in the[Kahneman4Review demo repository](https://huggingface.co/spaces/nuojohnchen/Kahneman4Review)\.