PaperScorer: How Predictable Is Paper Acceptance?

# PaperScorer: How Predictable Is Paper Acceptance?
Source: [https://arxiv.org/html/2610.31791](https://arxiv.org/html/2610.31791)
![Refer to caption](https://arxiv.org/html/2610.31791v1/teaser_flow.png)\(a\)PaperScorerpredicts a paper’s decision from its text or visual view\.
\(b\)Two anonymized corpora\.\(c\)PaperScoreris an accurate decision predictor\.
Figure 1:\(a\)PaperScoreris a family of models trained to predict a paper’s accept/reject decision from either a pure text or a visual view of the paper\.\(b\)For training and evaluation, we built two fully anonymized datasets, sourced from OpenReview and arXiv and labeled with accept/reject decisions\.\(c\)Across training recipes, we find fine\-tuning on our large\-scale corpora enablesPaperScorerto exceed frontier models \(GPT/Gemini\), specifically on highly rated papers\.Abstract

- Automated research is largely evaluated by benchmark performance, but the value of research rarely reduces to a single score\. Humans judge a work’s holistic contribution through peer review, which is hard to model because it is noisy and depends on many factors\. However, what if, given only the paper, we could predict its peer\-review decision directly? We introducePaperScorer, a family of models that predict a paper’s accept or reject decision from its extracted text or page screenshots, training only on the decision after removing author identity and formatting shortcuts\. Vision models are the most predictive \(68% accuracy on ICLR papers and 77% on ML papers from arXiv\), beating text\-only and frontier reasoning models\. Unlike plain text, page images retain figures, layout, and presentation\. This signal is robust, surviving smaller models and even compressing the whole paper into a single image\. But accurate predictions are not evenly spread\.PaperScoreris strongest on the lowest and highest rated papers, while papers rated in the middle stay unpredictable for our models and frontier systems alike\. Additionally, its confidence is calibrated, so its most confident predictions are also its most accurate\. Overall,PaperScorershows how much of the decision a paper carries on its own, marking a first step toward AI\-driven research evaluation\.

## Section 1Introduction

Increasingly capable large language model \(LLM\) agents are beginning to automate research end to end, proposing ideas, running experiments, and interpreting the results to recursively self\-improve\([Lu et al\., 2024](https://arxiv.org/html/2610.31791#bib.bib28);[Yamada et al\., 2025](https://arxiv.org/html/2610.31791#bib.bib46);[Si et al\., 2026](https://arxiv.org/html/2610.31791#bib.bib37);[Favaro and Clark, 2026](https://arxiv.org/html/2610.31791#bib.bib12)\)\. This loop needs a way to judge whether a result is good, and the standard judge is a benchmark\. Optimizing benchmarks has driven much of the field’s progress, from ImageNet in computer vision\([Deng et al\., 2009](https://arxiv.org/html/2610.31791#bib.bib11)\)to MuJoCo in reinforcement learning\([Todorov et al\., 2012](https://arxiv.org/html/2610.31791#bib.bib41)\)\. Yet many contributions, such as findings or interpretability work, have no benchmark, and even where they exist, researchers disagree about which ones matter\. This necessitates peer review\.

However, peer review is noisy and poorly understood\. A paper is typically judged by three or four reviewers and a meta\-reviewer, whose assessments are combined into a final accept or reject decision\([Yacoby, 2022](https://arxiv.org/html/2610.31791#bib.bib45)\)\. In the NeurIPS ’14 and ’21 consistency experiments, roughly half of papers accepted by one committee would not have been accepted by the other\([Beygelzimer et al\., 2021](https://arxiv.org/html/2610.31791#bib.bib5)\)\. AI tools now add to this complexity\([Biswas et al\., 2026](https://arxiv.org/html/2610.31791#bib.bib6)\), as up to17%of reviews were materially edited by LLMs\([Liang et al\., 2024](https://arxiv.org/html/2610.31791#bib.bib26)\), and LLM\-generated feedback led27%of treated reviewers to revise their reviews\([Thakkar et al\., 2026](https://arxiv.org/html/2610.31791#bib.bib40)\)\. A final decision therefore mixespaper signalwithnoisefrom reviewer assignment, interpretation, aggregation, and now the AI tools involved\. Before automated systems can judge research the way peer review does, we need to know how much of the decision the paper itself determines\. This motivates a simple question:

*From paper content alone, how predictable is the final decision?*

Measuring this signal requires more than fitting accept/reject labels\. Without careful controls, a predictor can rely on shortcut signals rather than paper content\([Zhang et al\., 2025a](https://arxiv.org/html/2610.31791#bib.bib52)\)\. Camera\-ready formatting, author\-identifying text, source\-specific artifacts, and the natural majority\-reject rate can all leak the label\([Jin et al\., 2024](https://arxiv.org/html/2610.31791#bib.bib21)\)\. Even after removing shortcuts, final decisions remain noisy, so a useful model should report uncertainty, separating confident cases from papers whose outcomes are not predictable from content alone\.

We introducePaperScorer, a family of models that predicts binary acceptance decisions from anonymized papers\.PaperScoreris trained on final outcomes rather than intermediate reviews or ratings, using anonymized datasets of machine learning \(ML\) papers\. To test whether extracted text is sufficient, we compare two representations of each paper, markdown for text models and screenshots of each page for vision models\. The text representation contains all text, including formulas and tables, but excludes figures, while vision preserves the full document\. Our datasets contain fully anonymized papers fromICLR, which are openly hosted and shared on OpenReview, and from ML conferences onarXiv, all from ’20–’26 and balanced 50/50 accept/reject to avoid a majority\-reject shortcut\.

Our main empirical result is that paper content contains recoverable decision signal, but this signal is limited\. Our top vision model,PaperScorer\-Vision, achieves68%accuracy onICLR, improving over the best frontier baseline by2\.7%\. OnarXiv,arXiv\-trainedPaperScorer\-Vision achieves77%accuracy, improving over frontier baselines by9\.0%\. Beyond binary decisions,PaperScoreremits a continuous acceptance estimate\. This score ranks accepted above rejected papers across thresholds, and onICLR, it aligns with reviewer ratings better than reviewer\-trained baselines\. After calibration on validation data, confidence matches accuracy on our test datasets, so atX%X\\%confidence, accuracy is aboutX%X\\%\.

Our experiments isolate several factors underlying performance\. First,PaperScorerachieves significantly higher accuracy with vision than with text on both datasets\. Vision enables analysis of figures, layout, and full\-document presentation, signal that our text extraction discards\. Second, predictability is concentrated at the extremes of reviewer rating, but not symmetrically\. Frontier models reliably flag only the lowest\-rated papers, whereasPaperScoreris accurate on both low and high\-rated papers\. Across all evaluated models, however, accuracy remains near chance on papers with intermediate reviewer ratings, so for these papers the document alone gives our models little to go on\. Third, performance improves with data scale but is largely invariant to model size or input resolution\. A 3B model performs similarly to a 7B model, and compressing the whole paper into a single image only marginally degrades performance\.

In summary,PaperScoreris the strongest paper decision predictor in our study, because a large, carefully controlled dataset lets straightforward supervised fine\-tuning deliver higher accuracy and calibrated confidence, concentrated on the clear accepts and rejects\. This matters because automated research makes research faster to iterate on, so submission volumes\([Yang et al\., 2026](https://arxiv.org/html/2610.31791#bib.bib50)\)and reviewer burden\([Adam, 2025](https://arxiv.org/html/2610.31791#bib.bib2)\)rise, while benchmark scores tell only part of a contribution’s story, which means evaluation must scale in a way that reflects human judgment\. At only 7B parameters,PaperScoreris a first step that measures how much of the human decision the paper alone carries, shows through calibration when to trust that estimate, and already plugs into agentic reviewing and literature search \([AppendixK](https://arxiv.org/html/2610.31791#A11)\)\.

## Section 2Problem Formulation

We studycontent\-only paper acceptance prediction\. Let𝒳\\mathcal\{X\}denote the space of observed paper artifacts and𝒵=\{0,1\}\\mathcal\{Z\}=\\\{0,1\\\}the final binary decision, withz=1z=1for acceptance andz=0z=0for rejection\. Our goal is to learn a predictorfθ​\(x\)f\_\{\\theta\}\(x\)that returns, from the paper artifact alone, both a hard decision and a score whose higher values indicate a higher predicted probability of acceptance\.

Peer review’s decision process\.We view peer review as a latent chain whereX0X\_\{0\}is the initial submission,Y1Y\_\{1\}the initial reviews,Y2Y\_\{2\}the rebuttal process,XrebX\_\{\\mathrm\{reb\}\}the revised paper,RRthe revised review after rebuttal,Y3Y\_\{3\}the meta\-review, andZZthe final decision\.

X0→Y1→Y2→Xreb→R→Y3→ZX\_\{0\}\\rightarrow Y\_\{1\}\\rightarrow Y\_\{2\}\\rightarrow X\_\{\\mathrm\{reb\}\}\\rightarrow R\\rightarrow Y\_\{3\}\\rightarrow ZBecause public anonymous artifacts are most consistently available after rebuttal \([AppendixB\.2](https://arxiv.org/html/2610.31791#A2.SS2)documents year\-over\-year availability\), our practical task isp⁡\(Z∣Xreb\)p\(Z\\mid X\_\{\\mathrm\{reb\}\}\)rather thanp⁡\(Z∣X0\)p\(Z\\mid X\_\{0\}\)\. In contrast with prior work\([Gao et al\., 2024](https://arxiv.org/html/2610.31791#bib.bib13);[Zhu et al\., 2025](https://arxiv.org/html/2610.31791#bib.bib56)\), reviews, ratings, and meta\-reviews are*not regressed on*\.

Prediction target\.We frame this as a text\-generation problem with a decision\-only target, so prompting, SFT, and RL can all be studied under the same format\.

fθ​\(x\)→Accept\|Rejectf\_\{\\theta\}\(x\)\\rightarrow\\boxed\{\\texttt\{Accept\}\\mid\\texttt\{Reject\}\}When token probabilities are available, we also compute an acceptance scorepaccept​\(x\)p\_\{\\mathrm\{accept\}\}\(x\)by normalizing the probabilities assigned to the two decision tokens\. The target dataset is therefore a collection𝒟=\{\(xi,zi\)\}i=1N\\mathcal\{D\}=\\\{\(x\_\{i\},z\_\{i\}\)\\\}\_\{i=1\}^\{N\}of paper\-decision pairs, wherexix\_\{i\}is derived from paper content alone andziz\_\{i\}is the final conference decision\.

Estimated Bayesian ceiling\.As the NeurIPS ’14 / ’21 consistency experiments\([Beygelzimer et al\., 2021](https://arxiv.org/html/2610.31791#bib.bib5)\)show, the decision carries some amount of randomness\. To get a rough sense of where the ceiling on accuracy lies given this noise, we fit a Beta\-Bernoulli model loosely to this single historical experiment\. Letpi∈\[0,1\]p\_\{i\}\\in\[0,1\]denote the latent acceptance probability of paperii, with committee decisions drawn from independent Bernoulli distributions, implying the following values\.

𝔼⁡\[p\]=0\.26,𝔼⁡\[2​p​\(1−p\)\]=0\.23,𝔼⁡\[p2\]=0\.52−0\.232=0\.145\.\\mathbb\{E\}\[p\]=0\.26,\\qquad\\mathbb\{E\}\[2p\(1\-p\)\]=0\.23,\\qquad\\mathbb\{E\}\[p^\{2\}\]=\\frac\{0\.52\-0\.23\}\{2\}=0\.145\.Matching these moments givesp∼Beta⁡\(α,β\)p\\sim\\mathrm\{Beta\}\(\\alpha,\\beta\)with\(α,β\)≈\(0\.39,1\.10\)\(\\alpha,\\beta\)\\approx\(0\.39,1\.10\)\. Under this fitted model, the Bayes\-optimal accuracy is approximately 83\.3% under the original acceptance distribution and 80\.2% on a balanced accept/reject evaluation\. We use80\.2%only as a soft reference point, not a hard limit, since this estimate is loose and inherits the assumptions of one historical experiment\. Full derivation is in[AppendixA](https://arxiv.org/html/2610.31791#A1)\.

## Section 3PaperScorer: Dataset and Methods

This section describes the balancedICLRand source\-derivedarXivcorpora with paired markdown and page screenshots, and the training recipes behindPaperScorer\.

### 3\.1Datasets

PaperScoreruses two balanced, anonymous accept/reject sources, both split 85/5/10 across train, validation, and test, and we evaluate on the most recent years of each test split\.ICLRcontains 25K submissions from OpenReview, evaluated on ’25–’26\.arXivcontains LaTeX\-source\-derived proceedings papers, evaluated on ’24–’26 to include biannual venues like ECCV\.arXivtraining uses a large 83K set \(arXiv\) and a smaller 25K set proportional toICLR\(arXiv\-m\)\. All datasets are balanced 50/50, and[Figure2](https://arxiv.org/html/2610.31791#S3.F2)shows examples from both\.

![Refer to caption](https://arxiv.org/html/2610.31791v1/dataset_panel_mosaic.png)Figure 2:Dataset mosaic\.Screenshots of pages fromICLRandarXiv\.ICLRexamples use latest\-rebuttal OpenReview PDFs, whilearXivexamples are derived from LaTeX source, recompiled with venue\-specific anonymization macros\. Each example tiles multiple pages for visualization; in practice,PaperScorer\-Vision processes individual pages\. Notably,ICLRpapers can contain rebuttal\-era revisions \(colored text\), whereas source\-renderedarXivpapers do not\.A paper\-decision dataset is useful only if the model sees the same evidence that a final decision maker could have seen\. We therefore require three properties before a paper enters the corpus:

1. 1\.Anonymity\.The exported artifact should not reveal authors, affiliations, acknowledgments, source repositories, or camera\-ready status that would leak the final review outcome to the model\.
2. 2\.Main\-conference scope\.Papers must be full main\-conference submissions, excluding workshops, short papers, industry tracks, withdrawn records, and other items outside the target decision process\.
3. 3\.Consistent processing of accepts and rejects\.Accept and reject artifacts should differ through natural paper content, not through formatting shortcuts, rendering, or availability cues\.

Data sourcing\.We buildICLRfrom OpenReview because ICLR releases both accepted and rejected submissions with review metadata\. The challenge is obtaining the anonymized artifact rather than a post\-review PDF\. Across six recent years, latest\-rebuttal PDFs give the best availability for anonymous artifacts \([AppendixB\.2](https://arxiv.org/html/2610.31791#A2.SS2)\), although they can contain rebuttal\-era revisions in the artifact itself\. We therefore use ICLR ’20–’23, ’25, and ’26 and exclude ICLR ’24, where accepts are only camera\-ready\.

Figure 3:Dataset coverage\.We summarize the balancedPaperScorerpools byICLRyear andarXivvenue after anonymous recompilation and filtering\.ICLRcoverage leans toward later years, whilearXivcoverage is largest for NeurIPS, CVPR, and ACL\-style NLP venues, with smaller balanced slices from other ML conferences\.ICLR is only one machine\-learning venue, so we also build a LaTeX\-source derivedarXivcorpus for broader training and evaluation\.arXivdoes not provide reject labels, but title matches to conference proceedings provide high\-precision accepted papers\([Yang et al\., 2026](https://arxiv.org/html/2610.31791#bib.bib50)\)\. We therefore define an*implied reject*as a LaTeX\-compiling paper that uses a covered venue template, satisfies the main\-conference filters, and does not appear in that venue’s proceedings\. This gives a noisier but more diverse source whose accept class is high fidelity and whose reject class tests whether the learned decision boundary survives weak negative labels\.

We usearXivas a testbed for two transfer questions\. First, withinarXiv, we ask whether diverse supervision improves classification on held\-outarXivpapers, paying particular attention to accept recall because accepted papers have the strongest provenance\. Second, across sources, we comparearXiv\-trained models on the high\-fidelityICLRset andICLR\-trained models onarXiv\. This isolates whether diversity improves generalization or whether clean same\-source labels remain more valuable than a larger weakly labeled pool\.

Data filtering and normalization\.ICLRis primarily filtered by year and artifact availability, then balanced within OpenReview year\.arXivrequires more reconstruction\. Starting from 434,579 MLarXivpapers, we pull 434,443 LaTeX source tarballs in bulk from arXiv’s requester\-pays S3 feed and extract their\.texcontents with NeMo Curator’sArxivIterator/ArxivExtractor\([NVIDIA, 2024](https://arxiv.org/html/2610.31791#bib.bib33)\)\. We then detect venue style files, compile anonymous review\-mode PDFs, and remove residual identifiers\. We drop non\-main\-conference, workshop, withdrawn, industry\-track, and short\-format papers\. ForarXiv, we additionally require a successful anonymous render, a references section, and between six and 30 main content pages to ensure consistency across papers\. Full pipeline details for both sources are documented in[AppendixB\.3](https://arxiv.org/html/2610.31791#A2.SS3)\.

The coverage plot in[Figure3](https://arxiv.org/html/2610.31791#S3.F3)reflects this sampling choice\.ICLRstarts with 40K ICLR submissions and subsamples rejects to 25K while keeping the annual distribution\. ForarXiv, submission year is more nebulous: implied rejects have upload dates, while proceedings\-matched accepts have conference years and post\-review uploads\. We therefore do not year\-balancearXiv, and instead subsample 101,659 filtered papers to an 83,656 balanced set that keeps the starting venue distribution\.

Both sources pass through the same MinerU\([Wang et al\., 2024](https://arxiv.org/html/2610.31791#bib.bib43)\)normalization before export\. We remove headers, line\-numbering asides, author\-identifying spans, and all content in and after the references\. Text models receive markdown with excluded regions omitted, while vision models receive page screenshots with the same regions whitened\. Text inputs use the full markdown up to a pre\-defined24,48024\{,\}480max context length, while for vision, each page is rendered and downscaled to at most1,003,5201\{,\}003\{,\}520pixels \(1280×2821280\{\\times\}28^\{2\}\), roughly1,2401\{,\}240vision tokens\. A typical paper runs about ten pages \(median1010, up to3333\), so the vision input averages roughly12,42012\{,\}420tokens, about half the max context length\. Accepted papers still contain more content, figures, and citations after cleanup, reflecting differences in content rather than artifact mismatches\.

ICLR\(test\)arXiv\(test\)Train SettingVisionTextVisionTextNatural train\-9\.2\-34\.5\-43\.0\-49\.8\+ val\. thresh\.\-6\.0\+27\.9\-10\.8\+8\.7Balanced train\+1\.8\-6\.6\-3\.2\-1\.6Table 1:Training\-prior bias\.Accept−\-reject recall gap after each prior/threshold rule; lower absolute value is better\.Why balanced training and evaluation?Conference decisions are naturally imbalanced, so raw accuracy can reward a predictor that mostly rejects papers\. Therefore, we use a balanced test set, whereby accuracy equally weights accept and reject recall\. This still supports natural\-rate interpretation: for samples drawn from the same curated domain, class\-conditional recalls from a balanced test can be reweighted by the known accept prior\. In[AppendixF\.1](https://arxiv.org/html/2610.31791#A6.SS1), this predicts natural\-rate accuracy to within a few percent onICLR/arXiv\.

The training prior is a separate choice\. Natural\-rate training matches the conference base rate, but the native decision rule then predicts rejection too often, since rejects are the majority class\.[Table1](https://arxiv.org/html/2610.31791#S3.T1)summarizes this as the gap between accept and reject recall\. Natural\-rateICLRandarXivmodels predict rejection far more often than acceptance under their native threshold, and tuning the threshold on validation data can shift it too far the other way\. Balanced\-trained models are closer to zero, so we use balanced training for the headlinePaperScorermodels\. This removes the excess rejection predictions without per\-dataset threshold tuning\.

### 3\.2Methods

Chosen models\.We use Qwen2\.5 text and Qwen2\.5\-VL vision backbones\([Yang et al\., 2024](https://arxiv.org/html/2610.31791#bib.bib48);[Bai et al\., 2025](https://arxiv.org/html/2610.31791#bib.bib4)\)because our original deployment target wasICLR, and Qwen2\.5 is the latest Qwen generation that plausibly predates public ICLR ’25 decisions \(text in December ’24, VL on January 26, ’25\)\. This makesICLRa clean deployment setting by the same cutoff logic, while later Qwen3/3\.5 models are more likely to have seen those results;arXivwas added later and did not drive backbone selection\. We also retrain a Qwen3 text backbone onICLRand find that swapping the base model moves accuracy little \([AppendixE](https://arxiv.org/html/2610.31791#A5)\)\.

Our main comparison is against DeepReview\-7B/14B\([Zhu et al\., 2025](https://arxiv.org/html/2610.31791#bib.bib56)\)\(trained through ICLR ’24\), Gemini\-3\.1\-Pro \(January ’25 cutoff\), and GPT\-5\.5 \(December ’25 cutoff\) reasoning baselines\. GPT\-5\.5’s cutoff postdates the public release of ICLR ’25 decisions, so we probe it for memorization directly \([AppendixD](https://arxiv.org/html/2610.31791#A4)\)\. It recognizes many of our test papers, but the decisions it claims to recall are not reliable\. Once its accept bias is accounted for, they are at chance\. It also cannot regenerate abstracts from titles \(ROUGE\-L\([Lin, 2004](https://arxiv.org/html/2610.31791#bib.bib27)\)at noise level\), and its accuracy on the papers it recognizes matches its accuracy on those it does not, including on ICLR ’25\. These probes provide no clear evidence that recalled acceptance decisions explain its baseline performance\.

We additionally compare against Jev\([TypeSafe AI, 2026](https://arxiv.org/html/2610.31791#bib.bib42)\), a much cheaper model claimed to be well calibrated\. It returns an acceptance probability from text, so we give it the same input asPaperScorer\-Text and compare calibration in[Section5](https://arxiv.org/html/2610.31791#S5)\.

Baseline Prompting\.For main results, DeepReview runs in its native Standard Mode with four simulated reviewers, self\-verification, and a meta\-review decision\. Gemini/GPT baselines use one venue\-generic AI\-Scientist\-style prompt\([Lu et al\., 2024](https://arxiv.org/html/2610.31791#bib.bib28);[Yamada et al\., 2025](https://arxiv.org/html/2610.31791#bib.bib46)\)that requests a 1–10 rating and an Accept/Reject decision\. Mini and large frontier variants run at high reasoning effort\. The generated decision is the default hard label, and the rating is a ranking score, not a probability\. Qwen rows use the answer\-only supervised fine\-tuning \(SFT\) prompt or the reasoning reinforcement\-learning \(RL\) prompt \([AppendixH\.1](https://arxiv.org/html/2610.31791#A8.SS1)\)\.

We also evaluate frontier coding\-agent harnesses with the same rating\-plus\-decision task\. Each Codex or Claude Code run receives an isolated paper workspace with stripped markdown, page screenshots, instructions, and a schema validator that enforces a well\-formedPREDICTIONS\.json; we log the full trajectory insession\.jsonl\. Agents run at the same high reasoning effort, inspect text and images selectively, and stay siloed without internet \(prompt and layout in[AppendixK\.3](https://arxiv.org/html/2610.31791#A11.SS3)\)\.

Baseline Correction Measures\.Under the base prompt, several models strongly favor one decision class\. We also tested more critical prompt variants, but bringing predicted acceptance rates closer to 50% did not reliably improve accuracy\. We therefore retain the base prompt, the same one used to start SFT/RL training, for the main comparisons\. These base prompts and their critical variants are defined in[AppendixH\.3](https://arxiv.org/html/2610.31791#A8.SS3)\.

For each baseline, we also tune a threshold on its score using a small held\-out sample of papers, and use the thresholded score as the hard label when it improves accuracy\. Out\-of\-the\-box thresholds are skewed: under its default threshold, GPT predicts rejection much more often than acceptance, and Gemini leans the same way, though less strongly\. Yet both still show relatively high AUC, suggesting the predicted rating is well\-ordered while the chosen decision threshold is biased\. Calibrating the threshold generally helps baseline numbers, in agreement with the findings of[Lu et al\. \(2024\)](https://arxiv.org/html/2610.31791#bib.bib28)\.

SFT\.Supervised fine\-tuning is our primary method for studying data representation, training distribution, scaling, and calibration\. We fine\-tune Qwen2\.5 text models at 3B, 7B, and 14B scale, and Qwen2\.5\-VL models at 3B and 7B scale, using LLaMA\-Factory\([Zheng et al\., 2024](https://arxiv.org/html/2610.31791#bib.bib55)\)\. LetPθ\(⋅∣x\)P\_\{\\theta\}\(\\cdot\\mid x\)denote the model’s next\-token distribution at the decision token for paperxx\. In the Qwen2\.5 vocabulary,AcceptandRejectare individual tokens, so we read off the probability of each and normalize over the two:

paccept=Pθ​\(Accept∣x\)Pθ​\(Accept∣x\)\+Pθ​\(Reject∣x\)\.p\_\{\\mathrm\{accept\}\}=\\frac\{P\_\{\\theta\}\(\\texttt\{Accept\}\\mid x\)\}\{P\_\{\\theta\}\(\\texttt\{Accept\}\\mid x\)\+P\_\{\\theta\}\(\\texttt\{Reject\}\\mid x\)\}\.After SFT, almost all decision mass lies on these two tokens \(validated across all models\), sopacceptp\_\{\\mathrm\{accept\}\}behaves as a usable binary accept probability\. Full training settings are documented in[AppendixI\.1](https://arxiv.org/html/2610.31791#A9.SS1)\. The SFT sweeps vary training source and scale, class prior, text versus screenshot inputs, mixed text\+vision inputs, and suffixes such as handcrafted paper statistics or generated reviews \([AppendixF\.2](https://arxiv.org/html/2610.31791#A6.SS2)\)\.

RL\.For reinforcement\-learning variants, we fine\-tune the policy with Group Relative Policy Optimization \(GRPO\)\([Guo et al\., 2025](https://arxiv.org/html/2610.31791#bib.bib16)\)using SkyRL\([Cao et al\., 2025](https://arxiv.org/html/2610.31791#bib.bib8)\)on the same paper\-only accept/reject task\. The policy uses the reasoning prompt, samples a group of completions for each paper, and updates using within\-group relative rewards rather than a learned reward model, so the model can reason before its final label\. We score each completionyyfor paperxxwith:

r\(y,x\)=0\.8⋅\[d\(y\)=d⋆\(x\)\]\+0\.2⋅\[format\(y\)\],r\(y,x\)=0\.8\\cdot\\mathbf\{1\}\\\!\\left\[d\(y\)=d^\{\\star\}\(x\)\\right\]\+0\.2\\cdot\\mathbf\{1\}\\\!\\left\[\\textsc\{format\}\(y\)\\right\],whered⁡\(y\)d\(y\)extracts the boxed Accept/Reject decision with keyword fallback,d⋆​\(x\)d^\{\\star\}\(x\)is the ground\-truth decision, andformat​\(y\)\\textsc\{format\}\(y\)checks that the completion has the required<think\>…</think\>block followed by<answer\>\\boxed\{Accept\|Reject\}</answer\>\. The format term matters early in training, when many completions skip the boxed answer \(details in[AppendixI\.2](https://arxiv.org/html/2610.31791#A9.SS2)\)\.

## Section 4Results

We first define our metrics \([Section4\.1](https://arxiv.org/html/2610.31791#S4.SS1)\), then set a simple paper features baseline \([Section4\.2](https://arxiv.org/html/2610.31791#S4.SS2)\), and finally report main results against prompting and coding\-agent baselines \([Section4\.3](https://arxiv.org/html/2610.31791#S4.SS3)\)\.

### 4\.1Evaluation Metrics

Each paper has a true decisionz∈𝒵z\\in\\mathcal\{Z\}, and each system produces a hard predictionz^∈𝒵\\hat\{z\}\\in\\mathcal\{Z\}and an accept scores∈\[0,1\]s\\in\[0,1\], where higher means more likely to be accepted\. AUC evaluates each system’s ranking without requiring its scores to represent calibrated probabilities\. ForPaperScorer,ssis the accept probabilitypacceptp\_\{\\mathrm\{accept\}\}, while for frontier modelsssis a normalized 1–10 rating\.

MetricDefinition and role in evaluationPrimary metrics⋆\\starAccuracy \(Acc\)Fraction of papers whose predicted decision matches the truth\. On our balanced 50/50 test set, accepts and rejects count equally, so always\-reject scores only50%50\\%\.⋆\\starArea under the ROC curve \(AUC\)The model produces a scoress, and across thresholds we plot the fraction of accepts called accept \(yy\) against the fraction of rejects called accept \(xx\); AUC is the area under this curve, the chance a random accept outscores a random reject\.Secondary diagnostics⋄\\diamondAccept / reject recall \(RAR\_\{A\},RRR\_\{R\}\)Model’s ability to recall all instances of each binary label\. Shows whether errors lean toward accepts or rejects\.⋄\\diamondPredicted acceptance rate \(pAp\_\{A\}\)Fraction of all papers the model calls accept\. Near50%50\\%indicates that the model does not have an inherent bias toward the accept or reject class\.⋄\\diamondRating correlation \(ρR\\rho\_\{R\}\)Spearman correlation between the accept scoressand each paper’s average reviewer rating, measuring alignment with human judgments\.⋄\\diamondExpected calibration error \(ECE\)Average gap between confidence and accuracy across confidence bins on the balanced test\. Low ECE meansX%X\\%\-confidence predictions are right aboutX%X\\%of the time\.Table 2:Evaluation diagnostics\.Primary metrics \(⋆\\star\) steer training and data decisions\. Secondary diagnostics \(⋄\\diamond\) drive behavioral analysis\. An ideal predictor is accurate \(Acc\), ranks accepts above rejects \(AUC\), and is well calibrated\.When studyingPaperScorerreliability, we ask whether confidence tracks realized accuracy after post\-hoc calibration\. The decision confidence isc=pacceptc=p\_\{\\mathrm\{accept\}\}for an Accept prediction andc=1−pacceptc=1\-p\_\{\\mathrm\{accept\}\}for a Reject prediction\. We bin papers bycc, such that in each binBmB\_\{m\},Acc⁡\(Bm\)\\mathrm\{Acc\}\(B\_\{m\}\)is the fraction of correct predictions andConf⁡\(Bm\)\\mathrm\{Conf\}\(B\_\{m\}\)is the average confidence\. ECE is the weighted gap\|Acc⁡\(Bm\)−Conf⁡\(Bm\)\|\|\\mathrm\{Acc\}\(B\_\{m\}\)\-\\mathrm\{Conf\}\(B\_\{m\}\)\|on the balanced test;[Section5](https://arxiv.org/html/2610.31791#S5)also checks the acceptance probability at real acceptance rates\. Frontier ratings are 1–10 review\-style scores, not accept/reject probabilities, so we report calibration only forPaperScorerand Jev\.

Our primary metrics ask whether a system both makes the right binary decision and ranks accepted papers above rejected papers\. The goal is to design a predictor that makes human\-like decisions while producing a score that orders papers the way peer review does\. Accuracy tests whether predictions match final decisions without hiding accept/reject collapse, while AUC tests whether accepted papers receive higher scores than rejected papers independent of a fixed threshold\. This matters because the same model may be used on conferences or years outside its training distribution, where post\-hoc threshold tuning may not be available\. Secondary diagnostics separate class bias from rating alignment and calibration\.

### 4\.2A Simple Paper Features Baseline

Before training any large model, we ask how much of the accept/reject decision is recoverable from simple, hand\-crafted features\. We fit two full\-data logistic\-regression models, one onICLRand one onarXiv, using paper features such as length, references, math notation, topic terms, and rhetorical counts\. The models exclude source identifiers, venue labels, ratings, citations, author names, and body\-page counts, since page metadata is not comparable between theICLRandarXivtext exports\. These simple models reach 61\.1% accuracy onICLRand 62\.5% onarXiv\. Their coefficients reveal which readily measured paper features are associated with acceptance\.[Figure4](https://arxiv.org/html/2610.31791#S4.F4)shows that accept features generally have to do with paper length, references, and figures, while reject features are more lexical \(full inventory in[AppendixC](https://arxiv.org/html/2610.31791#A3)\)\.

ICLRarXivAcceptRejectAcceptRejectPaper length \(\+0\.47\)Sentence length \(\-0\.18\)Paper length \(\+0\.48\)Sentence length \(\-0\.24\)Abstract sentence count \(\+0\.23\)Abstract words \(\-0\.14\)Math notation/1k words \(\+0\.19\)Recent citation share \(\-0\.15\)Vocabulary richness \(\+0\.21\)Rare\-word ratio \(\-0\.14\)Ablation mentions \(\+0\.14\)Sections per pg\. \(\-0\.15\)Figure refs\. per pg\. \(\+0\.18\)Graph\-topic terms \(\-0\.08\)Cross\-refs\. per pg\. \(\+0\.11\)Title words \(\-0\.09\)Appendix refs\. per pg\. \(\+0\.13\)Sections per pg\. \(\-0\.08\)Appendix refs\. per pg\. \(\+0\.10\)RL terms/1k words \(\-0\.07\)\(a\)Top feature weights on logistic regression of paper features\.
\(b\)Top 3 accept\-feature distributions\.

Figure 4:Feature\-probe audit\.\(a\)The table reports top standardized logistic\-regression weights from separateICLR\- andarXiv\-trained models, with weights in parentheses\.\(b\)The plots show accepted and rejected paper distributions for the top three accept\-associated features within each source on held\-outICLRandarXivpapers\. In feature names,per pg\.means per estimated body page\. Full feature definitions are in[AppendixC](https://arxiv.org/html/2610.31791#A3)\.
### 4\.3Main Results

Table[3](https://arxiv.org/html/2610.31791#S4.T3)summarizes performance on thePaperScorerICLRandarXivtest sets\. We start by analyzing baselines, then study the data and training effects behindPaperScorer\.

ICLRarXivMethodTrainInput⋆\\starAcc⋆\\starAUC⋄\\diamondRA/RRR\_\{A\}/R\_\{R\}⋄\\diamondpAp\_\{A\}⋄\\diamondρR\\rho\_\{R\}⋆\\starAcc⋆\\starAUC⋄\\diamondRA/RRR\_\{A\}/R\_\{R\}⋄\\diamondpAp\_\{A\}BaselinesMini Frontier ReasoningGemini\-3\-Flash\-Vision65\.6\.67267/6550\.1\+0\.1\+\.3962\.4\.65350/7537\.3\-12\.7Gemini\-3\-Flash\-Text63\.0\.64365/6152\.3\+2\.3\+\.2761\.2\.65551/7139\.6\-10\.4GPT\-5\.4\-Mini\-Vision57\.0\.65030/8422\.6\-27\.4\+\.3062\.6\.67779/4665\.7\+15\.7GPT\-5\.4\-Mini\-Text60\.0\.63250/7039\.4\-10\.6\+\.3058\.6\.63543/7434\.0\-16\.0Large Frontier ReasoningGemini\-3\.1\-Pro\-Vision65\.1\.69049/8134\.4\-15\.6\+\.3868\.5\.73652/8533\.2\-16\.8Gemini\-3\.1\-Pro\-Text62\.8\.67342/8329\.6\-20\.4\+\.3667\.7\.71069/6650\.6\+0\.6GPT\-5\.5\-Vision63\.4\.69287/4073\.6\+23\.6\+\.4068\.3\.74358/7839\.3\-10\.7GPT\-5\.5\-Text64\.4\.68165/6450\.2\+0\.2\+\.3766\.0\.69565/6748\.6\-1\.4Frontier Reasoning AgentsGPT\-5\.5 \(Codex Agent\)\-Both55\.3\.66715/9510\.0\-40\.0\+\.3956\.2\.68319/9313\.0\-37\.0Sonnet\-4\.6 \(Claude Agent\)\-Both61\.2\.71138/8526\.6\-23\.4\+\.4460\.7\.73935/8723\.8\-26\.2Frontier Decision ModelsJev\-Text63\.4\.68281/4667\.9\+17\.9\+\.3760\.6\.65978/4466\.8\+16\.8Review\-TrainedDeepReview\-7B\-Text59\.7\.61856/6446\.3\-3\.7\+\.2258\.7\.62036/8227\.1\-22\.9DeepReview\-14B\-Text64\.9\.66770/6055\.3\+5\.3\+\.3861\.6\.69950/7338\.0\-12\.0PaperScorer SFTBase Instruct ModelsQwen2\.5\-VL\-7B\-Vision54\.1\.55176/3271\.7\+21\.7\+\.1055\.9\.57189/2382\.8\+32\.8Qwen2\.5\-7B\-Text49\.9\.496100/099\.9\+49\.9\-\.0053\.0\.51974/3270\.9\+20\.9ICLR\-trainedPaperScorer\-VisionICLRVision68\.3\.73669/6750\.9\+0\.9\+\.4864\.2\.71950/7835\.4\-14\.6PaperScorer\-TextICLRText66\.0\.72163/6946\.7\-3\.3\+\.4759\.5\.72328/9118\.5\-31\.5arXiv\-trainedPaperScorer\-VisionarXiv\-mVision59\.8\.67641/7931\.1\-18\.9\+\.3571\.5\.82559/8436\.6\-13\.4PaperScorer\-TextarXiv\-mText62\.6\.65665/6052\.3\+2\.3\+\.3371\.7\.78763/8040\.8\-9\.2PaperScorer\-VisionarXivVision60\.7\.65352/6941\.8\-8\.2\+\.3177\.5\.84375/8046\.4\-3\.6PaperScorer\-TextarXivText62\.0\.65656/6843\.9\-6\.1\+\.3374\.2\.81572/7647\.3\-2\.7PaperScorer RLBase Instruct ModelsQwen2\.5\-VL\-7B\-Vision50\.1\.488100/199\.5\+49\.5\+\.0150\.2\.529100/099\.8\+49\.8Qwen2\.5\-7B\-Text50\.1\.50999/199\.3\+49\.3\+\.0451\.1\.534100/298\.9\+48\.9RL\-trainedPaperScorer\-Vision \(RL\)ICLRVision55\.0\.51778/3273\.1\+23\.1\+\.0360\.3\.55784/3773\.6\+23\.6PaperScorer\-Text \(RL\)ICLRText56\.1\.51799/1393\.2\+43\.2\+\.0358\.5\.59695/2286\.3\+36\.3

Table 3:Main results\.We compare prompting baselines, trained reviewers, andPaperScorervariants onICLRandarXiv, using Acc/AUC as primary correctness and ranking metrics\.PaperScorerrows use 7B checkpoints\. Bold/underline mark first/second per column, where first/second forpAp\_\{A\}is closest to the balanced 50% accept prior \(colored deltas show the signed deviation\)\. We reportρR\\rho\_\{R\}only forICLR, where every paper carries a reviewer rating\.PaperScorerimproves Acc/AUC over frontier systems, with vision strongest overall and models doing best on the source they were trained on\.Baseline performance\.We compare base Qwen instruct models, mini and large frontier Gemini/GPT reasoning models, decision models, coding agents, and DeepReview\. Untuned Qwen2\.5 predicts Accept for 97–99% of papers \([AppendixH\.3](https://arxiv.org/html/2610.31791#A8.SS3)\)\. Smaller high\-reasoning models are also skewed natively, with Gemini\-3\-Flash accepting almost every paper and GPT\-5\.4\-Mini rejecting almost every paper\. Large frontier systems are more accurate and run at high reasoning \(median∼\\sim2,000 reasoning tokens/request; Table[H\.2](https://arxiv.org/html/2610.31791#A8.T2)\), but their accept/reject recalls are skewed\. Gemini\-3\.1\-Pro accepts only 30–34% of papers in three of four settings, while GPT\-5\.5 swings between accepting 39% and 74%\.

The coding\-agent rows test whether interactive access to both modalities changes this pattern\. While Claude Code has the strongest rating correlation and the agents have among the strongest AUCs in the baseline block, their Acc is lower than the best non\-agent frontier baselines\. Like Jev, coding agents can rank papers in a way that agrees with reviewer ratings, but still turn that ranking into skewed accept/reject decisions\. DeepReview generates review\-style judgments and a meta\-review decision; its 14B model improves over 7B but still trails the best frontier model andPaperScorer\.

Binary SFT recovers a stronger decision signal\.The central result in Table[3](https://arxiv.org/html/2610.31791#S4.T3)is that direct accept/reject supervision gives the strongest decision predictor and ranking signal\. OnICLR,PaperScorer\-Vision reaches68\.368\.3Acc,2\.7%2\.7\\%ahead of the best frontier vision model, Gemini\-3\-Flash\. OnarXiv, the matchedarXiv\-trainedPaperScorer\-Vision reaches77\.577\.5, a larger9\.0%9\.0\\%ahead of the best frontier baseline\. On both datasets,PaperScorer\-Vision significantly beats both large frontier models in Acc and AUC \(one\-sided paired bootstrap,p<\.05p<\.05;[AppendixJ](https://arxiv.org/html/2610.31791#A10)\)\. These gains are not a threshold artifact, sincePaperScorer\-Vision keeps accept and reject recall balanced, while the frontier systems rank papers well but natively skew their decisions\.

Figure 5:Binary SFT is strongest over time\.PaperScorer\-Text andPaperScorer\-Vision use matched\-source training\. Across years, supervisedPaperScorerleads Acc/AUC, vision is strongest, and RL trails prompting frontier models\.Figure[5](https://arxiv.org/html/2610.31791#S4.F5)shows that the modality and method ordering is stable across years, withPaperScorer\-Vision best in every matched\-source year, prompting intermediate, and RL weaker\. These results indicate that papers carry recoverable decision signal, although learning that signal through long reasoning traces is difficult with only a single, noisy accept/reject bit as reward\. Supervising directly on the final decision works better, and vision keeps the layout and presentation cues that a markdown extract would strip\.

Accuracy is not constant over time, and Acc and AUC drop sharply from ’25 to ’26 acrossPaperScorer, frontier models, and DeepReview, suggesting a shared change in the evaluation distribution\. The early\-cyclearXiv’26 slice is also small, with 198 test papers from AAAI, CVPR, and ICLR ’26\. Each cycle also introduces new methods, benchmarks, and reviewer norms that reset what counts as a strong paper, so some recalibration is unavoidable for any predictor whose training pool is more than a year stale\. Training only through ’25 transfers poorly to ’26 \([AppendixF\.2](https://arxiv.org/html/2610.31791#A6.SS2)\), so we fold ’26 into training and expect any deployed predictor to be periodically retrained as new conferences land\.

Beyond hard decisions, bothPaperScorer\-Text andPaperScorer\-Vision reach top AUC on both datasets, ranking accepts above rejects regardless of threshold\. The score also tracks human ratings; onICLR,PaperScorer\-Vision’spacceptp\_\{\\mathrm\{accept\}\}correlates with reviewer ratings more closely than DeepReview and frontier baselines, despite training only on final decisions\.arXivratings are sparser \(292292/∼\\sim1,4151\{,\}415test papers, skewed toward NeurIPS and ICLR accepts\); on that biased subset coding agents correlate with ratings better thanPaperScorerbut still make worse decisions, so correlation with reviewer ratings does not convert into correct accept/reject calls\.

Model and data scale\.Model scale changes decision accuracy only modestly once the SFT objective is fixed \(Figure[6](https://arxiv.org/html/2610.31791#S4.F6)a\)\. Relative to matched 7B checkpoints, 3B and 14B variants barely move Acc, with no consistent direction across modality or source\. A paired 95% bootstrap CI reveals that nearly every change in Acc is statistically insignificant \(shaded in Figure[6\(a\)](https://arxiv.org/html/2610.31791#S4.F6.sf1)\)\. Vision is similarly competitive at smaller scale, so in Figure[6](https://arxiv.org/html/2610.31791#S4.F6)b the largestarXivAcc gain comes from data scale rather than parameter scale\.

ForICLR, we vary dataset size from 50% to 75% of the full pool\. ForarXiv, thearXiv\-mandarXivsets give us a 3\.4×\\timesmultiple\. More data helps for both datasets, and Figure[6](https://arxiv.org/html/2610.31791#S4.F6)b shows that the dominantarXivgain is data scale, not parameter scale, as the curve climbs steeply fromarXiv\-mtoarXiv, well past the spread of the 3B/7B/14B parameter sweep\.ICLRis much smaller, so taking 50% and 75% of this population compresses the visible slope, and the curves should not be read as comparable in absolute terms\.

SizeTrainInputICLRΔ\\DeltaAccarXivΔ\\DeltaAcc3BICLRVision\-1\.2\+1\.03BICLRText\+1\.1\+0\.93BarXiv\-mVision\-1\.0\+3\.13BarXiv\-mText\-1\.7\-0\.714BICLRText\+1\.00\.0\(a\)Model scale\.3B/14B differences to 7B, matched on training dataset\.ShadedΔ\\DeltaAcc cells are not significant at a 95% paired bootstrap CI\.
\(b\)Acc vs train data size onICLRandarXiv\.

Figure 6:Scaling effects\.Decision accuracy is largely invariant to model size, with 3B models matching 7B Acc, while data scale drives in\-domain accuracy, most visibly onarXivwhere the 3\.4×\\timespool boost dominates\.Figure 7:Vision is strongest on average\.Points show the benefit of using vision over text on both datasets\.Vision is strongest\.It improves over text in the averageICLR/arXivview \(Figure[7](https://arxiv.org/html/2610.31791#S4.F7)\) and in almost every dataset\-specific comparison \([TableJ\.2](https://arxiv.org/html/2610.31791#A10.T2)\)\. ForPaperScorer, the Vision row beats its Text row on both datasets, and a paired bootstrap confirms the gain \(p=\.012p=\.012and\.005\.005after Holm–Bonferroni correction\([Abdi, 2010](https://arxiv.org/html/2610.31791#bib.bib1)\)\)\. For the frontier families, the vision–text gaps are small and none is significant once their decisions are thresholded \([AppendixJ](https://arxiv.org/html/2610.31791#A10)\)\. Screenshots preserve complete figures and tables, equation layout, and visual polish, all of which markdown extraction discards\.

Second, the signal largely survives heavy compression, since tiling all tenPaperScorer\-Vision pages into a single 2380×\\times1512 image cuts vision tokens by about63%63\\%\(roughly12,42012\{,\}420to4,5904\{,\}590\), and only drops performance onICLRby 2\.2% \(Table[4](https://arxiv.org/html/2610.31791#S4.T4)\), a small change relative to the\>60%\>60\\%token cut\.

To investigate vision’s consistent advantage and the small cost of compression, we asked the base VL backbone to perform OCR on each input\. On per\-page screenshots it recovers∼80%\{\\sim\}80\\%of the content verbatim against the markdown extract\. On the compressed panel, OCR retention drops to∼30%\{\\sim\}30\\%\. Compression sharply reduces recoverable text while preserving layout, figure placement, and overall visual presentation\. Prior work supports this direction, as diagram count and quality correlate with citations\([Marshall et al\., 2021](https://arxiv.org/html/2610.31791#bib.bib30)\), and paper organization shapes peer\-review outcomes\([Jung et al\., 2025](https://arxiv.org/html/2610.31791#bib.bib22)\)\.

Conjecture: vision works through paper gestaltThe paper\-gestalt hypothesis\([Huang, 2018](https://arxiv.org/html/2610.31791#bib.bib18)\)proposed that a paper’s visual appearance can predict acceptance, but the original evidence mixed visual presentation with author and venue cues\. Our anonymized, matched text/vision comparisons support it, and much of the performance survives compression that sharply reduces recoverable text\. This suggests that visual presentation contributes to prediction, though we cannot isolate it from text that remains readable\.

InputVision toks\.Δ\\Deltatoks\.AccΔ\\DeltaAccAUCΔ\\DeltaAUCPer\-pagePaperScorer\-Vision∼\\sim12,420–68\.3–\.736–PanelPaperScorer\-Vision4,590\-63%66\.1\-2\.2\.704\-\.032Table 4:Panel\-token tradeoff \(ICLR\)\.OnICLR, theICLR\-trained panel model tiles ten pages into one 2380×\\times1512 image, preserving most of the per\-page vision result at about a third of the vision\-token cost\. We did not train anarXivpanel model, so this ablation isICLRonly\.

## Section 5Where does this signal lie?

Having shown that paper content carries a recoverable, well\-ranked decision signal, we now ask*where*that signal lives, across confidence ranges, papers, and conferences\.

Figure 8:Calibration on both datasets\.Reliability \(confidence vs\. accuracy,y=xy\{=\}xdotted\) on the balancedICLR\(left\) andarXiv\(right\) test sets forPaperScorertext and vision and for Jev, after temperature scaling\. Dashed lines show the cumulative coverage \(share of test papers at or above each confidence\)\.Confidence tracks accuracy\.Like most modern neural networks\([Guo et al\., 2017](https://arxiv.org/html/2610.31791#bib.bib15)\), ours are mildly overconfident, with raw accept probabilities higher than their accuracy warrants, so we correct them with temperature scaling\([Guo et al\., 2017](https://arxiv.org/html/2610.31791#bib.bib15)\)\. For each paperiiwe take the model’s*accept log\-odds*ℓi=log⁡P⁡\(Accept\)−log⁡P⁡\(Reject\)\\ell\_\{i\}=\\log P\(\\text\{Accept\}\)\-\\log P\(\\text\{Reject\}\)from the decision\-token log\-probabilities, and fit a temperatureτ\\taufor the calibrated accept probabilitypcalp\_\{\\mathrm\{cal\}\}by minimizing its negative log\-likelihood, wherezi∈\{0,1\}z\_\{i\}\\in\\\{0,1\\\}is the decision:

pcal=σ\(ℓi/τ\+log\(π/\(1−π\)\)\),ℒ\(τ\)=−∑i\[zilogpcal\+\(1−zi\)log\(1−pcal\)\]\.p\_\{\\mathrm\{cal\}\}=\\sigma\\big\(\\ell\_\{i\}/\\tau\+\\log\(\\pi/\(1\-\\pi\)\)\\big\),\\qquad\\mathcal\{L\}\(\\tau\)=\-\\sum\_\{i\}\\big\[z\_\{i\}\\log p\_\{\\mathrm\{cal\}\}\+\(1\-z\_\{i\}\)\\log\(1\-p\_\{\\mathrm\{cal\}\}\)\\big\]\.Here,π\\piis the acceptance rate of the scored papers\. We fitτ\\tauon a natural\-rate validation split withπ\\piset to its acceptance rate, so the shift handles the base rate andτ\\tauonly sets sharpness\. On our balanced test setsπ=0\.5\\pi=0\.5and the shift vanishes\. Neither changes a decision \(Accept iffℓi≥0\\ell\_\{i\}\\geq 0, i\.e\.pcal≥πp\_\{\\mathrm\{cal\}\}\\geq\\pi\), so Table[3](https://arxiv.org/html/2610.31791#S4.T3)is unchanged\. Confidence isc=pcalc=p\_\{\\mathrm\{cal\}\}for an Accept and1−pcal1\-p\_\{\\mathrm\{cal\}\}for a Reject\. Jev’s scores already sit near the real acceptance rate, so we fit itsτ\\tauwithout the shift and apply the correction in reverse on our 50/50 test sets\.

After scaling, bothPaperScorermodels track they=xy\{=\}xline on both datasets \(Figure[8](https://arxiv.org/html/2610.31791#S5.F8)\), so their confidence matches their accuracy, and Jev, corrected the same way, is roughly calibrated too \(ECE\.05\.05and\.02\.02, vs\.\.02\.02–\.03\.03for ours\)\. The difference is how often each model is confident \(dashed coverage curves\)\. At 70% confidence or more,PaperScorercovers about a third ofICLRpapers and over half ofarXivpapers, at nearly 80% and 85% accuracy, while Jev covers only 6% and 12%\.

Deployment at the real acceptance rate\.Most papers are rejected, and a venue’s rateπ\\pican be estimated from previous years\. Addinglog⁡\(π/\(1−π\)\)\\log\(\\pi/\(1\-\\pi\)\)toℓ\\ellgives acceptance probabilities at that rate, cutting calibration error from \.13 to \.02–\.03\. AtICLR’s rate, Accept calls are right 47% of the time and Reject calls 82% \([AppendixI\.3](https://arxiv.org/html/2610.31791#A9.SS3)\)\.

\(a\)Accuracy by rating percentile \(ICLR\)\.![Refer to caption](https://arxiv.org/html/2610.31791v1/selective_reliability_rating_b.png)\(b\)Rating×\\timesconfidence→\\toaccuracy \(ICLR\)\.

Figure 9:Accuracy ofPaperScorer\-Vision across reviewer rating\.Rating percentile is a paper’s percentile rank in\[0,1\]\[0,1\]among the papers of its own conference and year\.\(a\)Frontier models recognize low\-rated papers but fail on the high\-rated ones, whereasPaperScorer\-Vision is accurate at both tails, while all models are weak in the middle\.\(b\)The same shape as a confidence/rating surface, with a deep dip in the\[0\.4,0\.5\]\[0\.4,0\.5\]rating slice and the high\-accuracy, high\-confidence region concentrated on the tail papers\.Selective reliability\.ICLRhas human ground\-truth ratings for each paper, so we can measure accuracy at each level of the human rating \(Figure[9](https://arxiv.org/html/2610.31791#S5.F9)a\)\. Every model is most accurate at the lowest\-rated papers and near chance in the middle of the rating range\. The models differ at the top end, wherePaperScorer\-Vision stays accurate on the highest\-rated papers while GPT and Gemini fall off, so the frontier models accurately predict the lowest\-rated papers but fail on the highest\-rated ones\. Conditioning jointly on rating and confidence \(Figure[9](https://arxiv.org/html/2610.31791#S5.F9)b\) shows the same shape, with accuracy peaking on the confident extremes and collapsing in the middle, where none of the evaluated models finds enough signal\.

Performance across conferences\.Accuracy varies sharply by venue \(Figure[10](https://arxiv.org/html/2610.31791#S5.F10)\)\. On six of the seven highest\-coverage venues,PaperScorer\-Vision matches or beats the best frontier model \(Gemini\-3\.1\-Pro or GPT\-5\.5\)\. The exceptions are the vision\-heavy CV venues, where the frontier models perform particularly well\. Gemini tiesPaperScoreron CVPR and is the most accurate model on ICCV, the only high\-coverage venue where a frontier model wins\. AAAI is the other extreme, withPaperScorernear9090while both frontier models fall close to chance, the widest gap\. AAAI accepts and rejects run equally long, so the gap is not a length effect;PaperScorerranks them more cleanly here than anywhere else\. In\-domain training therefore givesPaperScorera clear edge everywhere except the visual CV papers, where strong general vision models compete\.

Figure 10:Accuracy by venue\.The best frontier vision models \(Gemini\-3\.1\-Pro, GPT\-5\.5\) againstPaperScorer\-Vision onarXivby venue\.PaperScorerleads on all high\-coverage venues but CVPR and ICCV, where Gemini ties or leads\.
## Section 6Related Work

Paper acceptance prediction and dataset construction\.Paper outcome prediction predates modern language models, with early work using bibliometric, lexical, and handcrafted signals\([Yan et al\., 2011](https://arxiv.org/html/2610.31791#bib.bib47);[Zhao et al\., 2025](https://arxiv.org/html/2610.31791#bib.bib54);[Hull and Bihlo, 2025](https://arxiv.org/html/2610.31791#bib.bib19)\)\. Recent datasets and methods move closer to paper\-level accept/reject prediction, including PeerRead\([Kang et al\., 2018](https://arxiv.org/html/2610.31791#bib.bib23)\), Deep Paper Gestalt\([Huang, 2018](https://arxiv.org/html/2610.31791#bib.bib18)\), DeepReview\([Zhu et al\., 2025](https://arxiv.org/html/2610.31791#bib.bib56)\), ScholarPeer\([Goyal et al\., 2026](https://arxiv.org/html/2610.31791#bib.bib14)\), and Sakana’s AI Reviewer\([Lu et al\., 2024](https://arxiv.org/html/2610.31791#bib.bib28)\)\. The main challenge is label provenance, since prior sources mix arXiv, LaTeX, and OpenReview artifacts\([Zhu et al\., 2025](https://arxiv.org/html/2610.31791#bib.bib56)\), compare camera\-ready accepts against anonymized rejects\([Lu et al\., 2024](https://arxiv.org/html/2610.31791#bib.bib28)\), or use workshop papers as reject proxies\([Huang, 2018](https://arxiv.org/html/2610.31791#bib.bib18)\)\. These choices can make labels recoverable from source, venue, or revision artifacts \(audited in[AppendixB\.1](https://arxiv.org/html/2610.31791#A2.SS1)\); we instead construct datasets in which the label is recoverable only from the paper’s content\.

Review generation and reviewer agents\.A parallel line of work studies critique generation, review quality, reviewer agreement, and multi\-agent review simulation rather than direct final\-decision prediction\. Reviewer2\([Gao et al\., 2024](https://arxiv.org/html/2610.31791#bib.bib13)\)analyzes reviewer behavior and agreement\. DeepReview\([Zhu et al\., 2025](https://arxiv.org/html/2610.31791#bib.bib56)\), ReviewRL\([Zeng et al\., 2025](https://arxiv.org/html/2610.31791#bib.bib51)\), and OpenReviewer\([Idahl and Ahmadi, 2025](https://arxiv.org/html/2610.31791#bib.bib20)\)generate review\-style outputs, while AgentReview\([Jin et al\., 2024](https://arxiv.org/html/2610.31791#bib.bib21)\), the Stanford Agentic Reviewer\([PaperReview\.ai, 2025](https://arxiv.org/html/2610.31791#bib.bib34)\), MARG\([D’Arcy et al\., 2024](https://arxiv.org/html/2610.31791#bib.bib10)\), MAMORX\([Taechoyotin et al\., 2024](https://arxiv.org/html/2610.31791#bib.bib39)\), and ScholarPeer\([Goyal et al\., 2026](https://arxiv.org/html/2610.31791#bib.bib14)\)move toward agentic reviewing\. These systems need reliable decision priors because generated critiques can be persuasive, yet when conditioned on the wrong prior they can lead to drastically different comments and decisions\. We therefore ask whether a calibrated acceptance probability improves agentic reviewing \([AppendixK](https://arxiv.org/html/2610.31791#A11)\)\.

AI\-assisted scientific workflows\.Automated reviewing is also becoming part of larger scientific workflows\. Systems such as The AI Scientist v1/v2\([Lu et al\., 2024](https://arxiv.org/html/2610.31791#bib.bib28);[Yamada et al\., 2025](https://arxiv.org/html/2610.31791#bib.bib46)\), CycleResearcher\([Weng et al\., 2025](https://arxiv.org/html/2610.31791#bib.bib44)\), FunSearch\([Romera\-Paredes et al\., 2024](https://arxiv.org/html/2610.31791#bib.bib36)\), AlphaEvolve\([Novikov et al\., 2025](https://arxiv.org/html/2610.31791#bib.bib32)\), and execution\-grounded research agents\([Si et al\., 2026](https://arxiv.org/html/2610.31791#bib.bib37)\)use model\-based evaluation to filter ideas, judge generated artifacts, or steer search\. AstaBench\([Bragg et al\., 2026](https://arxiv.org/html/2610.31791#bib.bib7)\)formalizes evaluation for the full idea\-to\-paper setting\. In these pipelines, reviewer models act as selection mechanisms rather than summarizers, so systematic accept bias or poor calibration can shape which research directions are pursued\. This motivatesPaperScoreras a calibrated decision prior for both retrieval and review assistance \([AppendixK](https://arxiv.org/html/2610.31791#A11)\)\.

## Section 7Conclusion and Applications

We introducePaperScorer, a family of text and vision models that predict a paper’s final accept/reject decision from its content alone\. Tested onICLRand source\-derivedarXivpapers, supervised binary prediction recovers a strong paper\-level signal, and 7B visionPaperScorermodels outperform frontier reasoning and DeepReview baselines on decision accuracy and ranking, while producing calibrated acceptance probabilities\. We find that paper artifacts contain substantial recoverable decision signal\. Screenshots of the paper preserve layout, figure, and presentation cues that markdown extraction tends to discard\.

PaperScoreris not a replacement for peer review or a universal measure of scientific value\. Its best use is as a calibrated estimate of acceptance probability that larger scientific workflows can use as a prior, which we demonstrate in two settings \([AppendixK](https://arxiv.org/html/2610.31791#A11)\)\. In literature retrieval, combining dense semantic retrieval scores from an embedding model withPaperScorer\-Vision’s acceptance score surfaces papers that are both semantically relevant and likely to be accepted\. On the LitSearch benchmark, this improves ground\-truth citation recall in the top\-10 results by8\.7%\. In agentic reviewing, conditioning Claude Code on thePaperScorer\-Vision prior raises decision accuracy from 63\.0% to 70\.6% and improves alignment with human reviews, mainly by correcting the agent’s incorrect acceptance predictions\.

## Section 8Limitations and Future Work

We cannot yet say*why*the model favors one paper over another\. Our attention maps and section\-edit tests \([AppendixL\.1](https://arxiv.org/html/2610.31791#A12.SS1)\) show what it reads, but they do not*establish causality*, and they rely on last\-layer attention from only a few papers\. The idea that vision performs better by capturing a paper’s overall look remains a conjecture, supported but not settled by our compression and modality results\. The model also weakens on the newest papers, where our data is thin\. By ’26, some decisions reflect AI help as much as human judgment, and dropping ’26 from training lowers accuracy \([AppendixF\.2](https://arxiv.org/html/2610.31791#A6.SS2)\)\. That year is also unusual, with shorter rebuttals, the narrowest accept\-reject rating gap we have seen, andarXivlabels from only a few venues\. This should ease as more ’26 papers appear, andPaperScorercan be retrained through our pipeline\.

What the score captures\.A highpacceptp\_\{\\mathrm\{accept\}\}predicts the accept/reject outcome; it is not a measure of novelty or long\-term impact, and it comes with no explanation of*why*\(though it does have correlations to reviewer rating and citation count\)\. On impact, the one slice with both full reviewer ratings and citations, ICLR ’25, is encouraging, aspacceptp\_\{\\mathrm\{accept\}\}tracks later citations about as well as the human review score does \(ρ≈\.25\\rho\\approx\.25for both,[AppendixJ](https://arxiv.org/html/2610.31791#A10)\), though one conference\-year and small sample make this a preliminary result\. The model itself provides no explanation for its prediction, so any reasoning for a decision must be generated afterward, by the model or an agentic reviewer\. Conditioning a reviewer onpacceptp\_\{\\mathrm\{accept\}\}raises agreement with human reviews \([AppendixK\.2](https://arxiv.org/html/2610.31791#A11.SS2)\), but such post\-hoc rationales can hallucinate\. A more robust path likely needs multi\-agent generation with verification loops\.

Future work\.The clearest next steps are expanding high\-fidelity datasets across more venues, improving borderline\-paper modeling, and developing richer supervision than a single binary outcome, especially for cases where reviewer disagreement and venue\-specific norms matter\.

## Acknowledgements

This work was supported by Princeton Research Computing resources, including the Della high\-performance computing cluster\. We additionally thank Harish Krishnakumar and Linrong Cai for their help in experimental execution, Taiming Lu and Haozhe Chen for helpful discussions during the project, and Shardul Kothapalli for conversations that led to the literature search application\.

## References

- Abdi \(2010\)Hervé Abdi\.Holm’s sequential bonferroni procedure\.*Encyclopedia of research design*, 2010\.
- Adam \(2025\)David Adam\.The peer\-review crisis: How to fix an overloaded system\.*Nature*, 2025\.
- Ajith et al\. \(2024\)Anirudh Ajith, Mengzhou Xia, Alexis Chevalier, Tanya Goyal, Danqi Chen, and Tianyu Gao\.Litsearch: A retrieval benchmark for scientific literature search\.In*EMNLP*, 2024\.
- Bai et al\. \(2025\)Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al\.Qwen2\.5\-vl technical report\.*arXiv preprint arXiv:2502\.13923*, 2025\.
- Beygelzimer et al\. \(2021\)Alina Beygelzimer, Yann Dauphin, Percy Liang, and Jennifer Wortman Vaughan\.The neurips 2021 consistency experiment\.[https://blog\.neurips\.cc/2021/12/08/the\-neurips\-2021\-consistency\-experiment/](https://blog.neurips.cc/2021/12/08/the-neurips-2021-consistency-experiment/), 2021\.NeurIPS Blog\.
- Biswas et al\. \(2026\)Joydeep Biswas, Sheila Schoepp, Gautham Vasan, Anthony Opipari, Arthur Zhang, Zichao Hu, Sebastian Joseph, Matthew Lease, Junyi Jessy Li, Peter Stone, et al\.Ai\-assisted peer review at scale: The aaai\-26 ai review pilot\.*arXiv preprint arXiv:2604\.13940*, 2026\.
- Bragg et al\. \(2026\)Jonathan Bragg, Mike D’Arcy, Nishant Balepur, Dan Bareket, Bhavana Dalvi Mishra, Sergey Feldman, Dany Haddad, Jena D\. Hwang, Peter Jansen, Varsha Kishore, et al\.Astabench: Rigorous benchmarking of AI agents with a scientific research suite\.In*ICLR*, 2026\.
- Cao et al\. \(2025\)Shiyi Cao, Sumanth Hegde, Dacheng Li, Tyler Griggs, Shu Liu, Eric Tang, Jiayi Pan, Xingyao Wang, Akshay Malik, Graham Neubig, Kourosh Hakhamaneshi, Richard Liaw, Philipp Moritz, Matei Zaharia, Joseph E\. Gonzalez, and Ion Stoica\.SkyRL\-v0: Train real\-world long\-horizon agents via reinforcement learning\.[https://github\.com/NovaSky\-AI/SkyRL](https://github.com/NovaSky-AI/SkyRL), 2025\.
- Chen et al\. \(2021\)Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al\.Evaluating large language models trained on code\.*arXiv preprint arXiv:2107\.03374*, 2021\.
- D’Arcy et al\. \(2024\)Mike D’Arcy, Tom Hope, Larry Birnbaum, and Doug Downey\.MARG: Multi\-agent review generation for scientific papers\.*arXiv preprint arXiv:2401\.04259*, 2024\.
- Deng et al\. \(2009\)Jia Deng, Wei Dong, Richard Socher, Li\-Jia Li, Kai Li, and Li Fei\-Fei\.Imagenet: A large\-scale hierarchical image database\.In*2009 IEEE Conference on Computer Vision and Pattern Recognition*, pages 248–255, 2009\.[10\.1109/CVPR\.2009\.5206848](https://doi.org/10.1109/CVPR.2009.5206848)\.
- Favaro and Clark \(2026\)Marina Favaro and Jack Clark\.When AI builds itself\.[https://www\.anthropic\.com/institute/recursive\-self\-improvement](https://www.anthropic.com/institute/recursive-self-improvement), 2026\.The Anthropic Institute\.
- Gao et al\. \(2024\)Zhaolin Gao, Kianté Brantley, and Thorsten Joachims\.Reviewer2: Optimizing review generation through prompt generation\.*arXiv preprint arXiv:2402\.10886*, 2024\.
- Goyal et al\. \(2026\)Palash Goyal, Mihir Parmar, Yiwen Song, Hamid Palangi, Tomas Pfister, and Jinsung Yoon\.Scholarpeer: A context\-aware multi\-agent framework for automated peer review\.*arXiv preprint arXiv:2601\.22638*, 2026\.
- Guo et al\. \(2017\)Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q\. Weinberger\.On calibration of modern neural networks\.In*ICML*, 2017\.
- Guo et al\. \(2025\)Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al\.Deepseek\-r1 incentivizes reasoning in llms through reinforcement learning\.*Nature*, 2025\.
- Haidt \(2001\)Jonathan Haidt\.The emotional dog and its rational tail: A social intuitionist approach to moral judgment\.*Psychological Review*, 2001\.
- Huang \(2018\)Jia\-Bin Huang\.Deep paper gestalt\.*arXiv preprint arXiv:1812\.08775*, 2018\.
- Hull and Bihlo \(2025\)Gavin Hull and Alex Bihlo\.ForeCite: Adapting pre\-trained language models to predict future citation rates of academic papers\.*arXiv preprint arXiv:2505\.08941*, 2025\.
- Idahl and Ahmadi \(2025\)Maximilian Idahl and Zahra Ahmadi\.Openreviewer: A specialized large language model for generating critical scientific paper reviews\.In*NAACL*, 2025\.
- Jin et al\. \(2024\)Yiqiao Jin, Qinlin Zhao, Yiyang Wang, Hao Chen, Kaijie Zhu, Yijia Xiao, and Jindong Wang\.AgentReview: Exploring peer review dynamics with LLM agents\.In*EMNLP*, 2024\.
- Jung et al\. \(2025\)Sangkeun Jung, Goun Pyeon, Inbum Heo, and Hyungjin Ahn\.What drives paper acceptance? a process\-centric analysis of modern peer review\.*arXiv preprint arXiv:2509\.25701*, 2025\.
- Kang et al\. \(2018\)Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz\.A dataset of peer reviews \(PeerRead\): Collection, insights and NLP applications\.In*NAACL*, 2018\.
- Kunda \(1990\)Ziva Kunda\.The case for motivated reasoning\.*Psychological Bulletin*, 1990\.
- Li et al\. \(2026\)Bangzheng Li, Jianmo Ni, Chen Qu, Ian Miao, Liu Yang, Xingyu Fu, Muhao Chen, and Derek Zhiyuan Cheng\.Reinforced attention learning\.*arXiv preprint arXiv:2602\.04884*, 2026\.
- Liang et al\. \(2024\)Weixin Liang, Zachary Izzo, Yaohui Zhang, Haley Lepp, Hancheng Cao, Xuandong Zhao, Lingjiao Chen, Haotian Ye, Sheng Liu, Zhi Huang, et al\.Monitoring AI\-modified content at scale: A case study on the impact of ChatGPT on AI conference peer reviews\.In*ICML*, 2024\.
- Lin \(2004\)Chin\-Yew Lin\.ROUGE: A package for automatic evaluation of summaries\.In*ACL Workshop on Text Summarization Branches Out*, 2004\.
- Lu et al\. \(2024\)Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha\.The AI scientist: Towards fully automated open\-ended scientific discovery\.*arXiv preprint arXiv:2408\.06292*, 2024\.
- Madaan et al\. \(2025\)Lovish Madaan, Aniket Didolkar, Suchin Gururangan, John Quan, Ruan Silva, Ruslan Salakhutdinov, Manzil Zaheer, Sanjeev Arora, and Anirudh Goyal\.Rethinking thinking tokens: Llms as improvement operators\.*arXiv preprint arXiv:2510\.01123*, 2025\.
- Marshall et al\. \(2021\)Guy Clarke Marshall, Caroline Jay, and Andre Freitas\.Number and quality of diagrams in scholarly publications is associated with number of citations\.*arXiv preprint arXiv:2104\.14815*, 2021\.
- Nisbett and Wilson \(1977\)Richard E Nisbett and Timothy D Wilson\.Telling more than we can know: Verbal reports on mental processes\.*Psychological Review*, 1977\.
- Novikov et al\. \(2025\)Alexander Novikov, Ngân Vũ, Marvin Eisenberger, Emilien Dupont, Po\-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J\. R\. Ruiz, Abbas Mehrabian, M\. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, Pushmeet Kohli, and Matej Balog\.AlphaEvolve: A coding agent for scientific and algorithmic discovery\.*arXiv preprint arXiv:2506\.13131*, 2025\.
- NVIDIA \(2024\)NVIDIA\.NeMo Curator: GPU\-accelerated data curation for training AI models\.[https://github\.com/NVIDIA\-NeMo/Curator](https://github.com/NVIDIA-NeMo/Curator), 2024\.
- PaperReview\.ai \(2025\)PaperReview\.ai\.Stanford agentic reviewer: Technical overview\.[https://paperreview\.ai/tech\-overview](https://paperreview.ai/tech-overview), 2025\.
- Platt \(1999\)John C\. Platt\.Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods\.*Advances in large margin classifiers*, 1999\.
- Romera\-Paredes et al\. \(2024\)Bernardino Romera\-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al\.Mathematical discoveries from program search with large language models\.*Nature*, 2024\.
- Si et al\. \(2026\)Chenglei Si, Zitong Yang, Yejin Choi, Emmanuel Candès, Diyi Yang, and Tatsunori Hashimoto\.Towards execution\-grounded automated AI research\.In*Forty\-third International Conference on Machine Learning*, 2026\.URL[https://openreview\.net/forum?id=j0PagZWEFo](https://openreview.net/forum?id=j0PagZWEFo)\.
- Slovic et al\. \(2007\)Paul Slovic, Melissa L\. Finucane, Ellen Peters, and Donald G\. MacGregor\.The affect heuristic\.*European Journal of Operational Research*, 2007\.
- Taechoyotin et al\. \(2024\)Pawin Taechoyotin, Guanchao Wang, Tong Zeng, Bradley Sides, and Daniel Acuna\.MAMORX: Multi\-agent multi\-modal scientific review generation with external knowledge\.In*NeurIPS Workshop Foundation Models for Science*, 2024\.
- Thakkar et al\. \(2026\)Nitya Thakkar, Mert Yuksekgonul, Jake Silberg, Animesh Garg, Nanyun Peng, Fei Sha, Rose Yu, Carl Vondrick, and James Zou\.A large\-scale randomized study of large language model feedback in peer review\.*Nature Machine Intelligence*, 2026\.
- Todorov et al\. \(2012\)Emanuel Todorov, Tom Erez, and Yuval Tassa\.Mujoco: A physics engine for model\-based control\.In*2012 IEEE/RSJ International Conference on Intelligent Robots and Systems*, pages 5026–5033\. IEEE, 2012\.[10\.1109/IROS\.2012\.6386109](https://doi.org/10.1109/IROS.2012.6386109)\.
- TypeSafe AI \(2026\)TypeSafe AI\.Introducing system one models & jev\.[https://typesafe\.ai/blog/introducing\-system\-one\-models\-and\-jev](https://typesafe.ai/blog/introducing-system-one-models-and-jev), September 2026\.TypeSafe blog\.
- Wang et al\. \(2024\)Bin Wang, Chao Xu, Xiaomeng Zhao, Linke Ouyang, Fan Wu, Zhiyuan Zhao, Rui Xu, Kaiwen Liu, Yuan Qu, Fukai Shang, et al\.Mineru: An open\-source solution for precise document content extraction\.*arXiv preprint arXiv:2409\.18839*, 2024\.
- Weng et al\. \(2025\)Yixuan Weng, Minjun Zhu, Guangsheng Bao, Hongbo Zhang, Jindong Wang, Yue Zhang, and Linyi Yang\.CycleResearcher: Improving automated research via automated review\.In*ICLR*, 2025\.
- Yacoby \(2022\)Yaniv Yacoby\.The paper review process\.[https://yanivyacoby\.github\.io/harvard\-cs290/assets/The\-Paper\-Review\-Process\.pdf](https://yanivyacoby.github.io/harvard-cs290/assets/The-Paper-Review-Process.pdf), 2022\.Harvard CS290 course material\.
- Yamada et al\. \(2025\)Yutaro Yamada, Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha\.The AI scientist\-v2: Workshop\-level automated scientific discovery via agentic tree search\.*arXiv preprint arXiv:2504\.08066*, 2025\.
- Yan et al\. \(2011\)Rui Yan, Jie Tang, Xiaobing Liu, Dongdong Shan, and Xiaoming Li\.Citation count prediction: Learning to estimate future citations for literature\.In*CIKM*, 2011\.
- Yang et al\. \(2024\)An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, et al\.Qwen2\.5 technical report\.*arXiv preprint arXiv:2412\.15115*, 2024\.
- Yang et al\. \(2025\)An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al\.Qwen3 technical report\.*arXiv preprint arXiv:2505\.09388*, 2025\.
- Yang et al\. \(2026\)Jing Yang, Qiyao Wei, and Jiaxin Pei\.Paper copilot: Tracking the evolution of peer review in AI conferences\.In*ICLR*, 2026\.
- Zeng et al\. \(2025\)Sihang Zeng, Kai Tian, Kaiyan Zhang, Yuru Wang, Junqi Gao, Runze Liu, Sa Yang, Jingxuan Li, Xinwei Long, Jiaheng Ma, et al\.Reviewrl: Towards automated scientific review with rl\.In*EMNLP*, 2025\.
- Zhang et al\. \(2025a\)Daoze Zhang, Zhijian Bao, Sihang Du, Zhiyi Zhao, Kuangling Zhang, Dezheng Bao, and Yang Yang\.Re2: A consistency\-ensured dataset for full\-stage peer review and multi\-turn rebuttal discussions\.*arXiv preprint arXiv:2505\.07920*, 2025a\.
- Zhang et al\. \(2025b\)Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al\.Qwen3 embedding: Advancing text embedding and reranking through foundation models\.*arXiv preprint arXiv:2506\.05176*, 2025b\.
- Zhao et al\. \(2025\)Penghai Zhao, Qinghua Xing, Kairan Dou, Jinyu Tian, Ying Tai, Jian Yang, Ming\-Ming Cheng, and Xiang Li\.From words to worth: newborn article impact prediction with llm\.In*AAAI*, 2025\.
- Zheng et al\. \(2024\)Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo\.LlamaFactory: Unified efficient fine\-tuning of 100\+ language models\.In*ACL Demos*, 2024\.
- Zhu et al\. \(2025\)Minjun Zhu, Yixuan Weng, Linyi Yang, and Yue Zhang\.Deepreview: Improving llm\-based paper review with human\-like deep thinking process\.In*ACL*, 2025\.

Appendix

This appendix provides supporting derivations, dataset documentation, extended evaluation, and training/inference details for the main paper:

- •§[A](https://arxiv.org/html/2610.31791#A1)derives the Bayesian\-optimal accuracy ceiling that anchors the 80\.2% reference point in §[2](https://arxiv.org/html/2610.31791#S2)\.
- •§[B](https://arxiv.org/html/2610.31791#A2)documents data provenance and the artifact\-shortcut audit across our datasets and prior work\.
- •§[C](https://arxiv.org/html/2610.31791#A3)catalogs handcrafted feature probes and accept/reject distributions across ICLR andarXiv\.
- •§[D](https://arxiv.org/html/2610.31791#A4)tests whether frontier models can recall ground\-truth decisions through label\-recall and content\-completion probes\.
- •§[E](https://arxiv.org/html/2610.31791#A5)compares Qwen2\.5 and Qwen3 text backbones onICLRalongside their knowledge cutoffs\.
- •§[F](https://arxiv.org/html/2610.31791#A6)reports data\-mixture and input\-representation ablations plus the reverse\-order and prior diagnostics\.
- •§[G](https://arxiv.org/html/2610.31791#A7)reports cross\-source transfer betweenICLRandarXiv, including thearXiv\-ICLRsubset\.
- •§[H](https://arxiv.org/html/2610.31791#A8)records venue\-specific prompts and prompt\-correction studies for base open\-source and frontier models\.
- •§[I](https://arxiv.org/html/2610.31791#A9)covers SFT settings, the GRPO policy, and post\-hoc temperature calibration\.
- •§[J](https://arxiv.org/html/2610.31791#A10)reports paired bootstrap intervals, rating/citation diagnostics, and a frontier repeatability audit\.
- •§[K](https://arxiv.org/html/2610.31791#A11)details the LitSearch reranking and agentic\-reviewing applications of the calibratedPaperScorerprior, including the Claude Code reviewer scaffolding and Codex grading rubric\.
- •§[L](https://arxiv.org/html/2610.31791#A12)analyzes section\-level attention maps and the four\-commit Codex edit experiments\.

## Appendix ABayesian Optimal Accuracy Derivation

We follow the NeurIPS ’14 / ’21 consistency experiment\([Beygelzimer et al\., 2021](https://arxiv.org/html/2610.31791#bib.bib5)\), where each paperiiis reviewed by two committeesj∈\{1,2\}j\\in\\\{1,2\\\}with independent decisions conditional on a latent acceptance probabilitypi∼Fp\_\{i\}\\sim F\(step[a](https://arxiv.org/html/2610.31791#A1.Ex7)\)\. HereFFis the \(unknown\) population distribution of per\-paper acceptance probabilities, whose first two moments we estimate from the consistency experiment below\. The experiment’s reported acceptance rate of≈26%\{\\approx\}\\,26\\%fixes the first moment \(step[b](https://arxiv.org/html/2610.31791#A1.Ex8)\), and its disagreement rate of≈23%\{\\approx\}\\,23\\%corresponds to𝔼⁡\[2​p​\(1−p\)\]\\mathbb\{E\}\[2p\(1\-p\)\]\(step[c](https://arxiv.org/html/2610.31791#A1.Ex9)\), which expands into the second moment and variance \(steps[d](https://arxiv.org/html/2610.31791#A1.Ex10),[e](https://arxiv.org/html/2610.31791#A1.Ex11)\)\. Matching these to a Beta family \(steps[f](https://arxiv.org/html/2610.31791#A1.Ex12),[g](https://arxiv.org/html/2610.31791#A1.Ex13)\) gives\(α,β\)≈\(0\.39,1\.10\)\(\\alpha,\\beta\)\\approx\(0\.39,1\.10\)\. Under the natural acceptance distribution, the Bayes\-optimal predictor accepts whenp\>0\.5p\>0\.5, givingAcc⋆≈83\.3%\\mathrm\{Acc\}^\{\\star\}\\approx 83\.3\\%\(step[h](https://arxiv.org/html/2610.31791#A1.Ex14)\)\.

Dj\(i\)\\displaystyle D\_\{j\}^\{\(i\)\}∼Bernoulli⁡\(pi\),j∈\{1,2\}\\displaystyle\\sim\\mathrm\{Bernoulli\}\(p\_\{i\}\),\\qquad j\\in\\\{1,2\\\}\(a\)𝔼⁡\[p\]\\displaystyle\\mathbb\{E\}\[p\]=0\.26\\displaystyle=0\.26\(b\)𝔼⁡\[2​p​\(1−p\)\]\\displaystyle\\mathbb\{E\}\[2p\(1\-p\)\]=0\.23\\displaystyle=0\.23\(c\)2​𝔼​\[p\]−2​𝔼​\[p2\]\\displaystyle 2\\mathbb\{E\}\[p\]\-2\\mathbb\{E\}\[p^\{2\}\]=0\.23⟹𝔼⁡\[p2\]=0\.52−0\.232=0\.145\\displaystyle=0\.23\\;\\Longrightarrow\\;\\mathbb\{E\}\[p^\{2\}\]=\\frac\{0\.52\-0\.23\}\{2\}=0\.145\(d\)Var⁡\(p\)\\displaystyle\\mathrm\{Var\}\(p\)=0\.145−0\.262≈0\.077\\displaystyle=0\.145\-0\.26^\{2\}\\approx 0\.077\(e\)p\\displaystyle p∼Beta⁡\(α,β\),αα\+β=0\.26,α⁡\(α\+1\)\(α\+β\)​\(α\+β\+1\)=0\.145\\displaystyle\\sim\\mathrm\{Beta\}\(\\alpha,\\beta\),\\qquad\\tfrac\{\\alpha\}\{\\alpha\+\\beta\}=0\.26,\\;\\;\\tfrac\{\\alpha\(\\alpha\+1\)\}\{\(\\alpha\+\\beta\)\(\\alpha\+\\beta\+1\)\}=0\.145\(f\)\(α,β\)\\displaystyle\(\\alpha,\\beta\)≈\(0\.39,1\.10\)\\displaystyle\\approx\(0\.39,1\.10\)\(g\)Acc⋆\\displaystyle\\mathrm\{Acc\}^\{\\star\}=𝔼⁡\[max⁡\(p,1−p\)\]≈83\.3%\\displaystyle=\\mathbb\{E\}\[\\max\(p,1\-p\)\]\\approx 83\.3\\%\(h\)p\|D=1\\displaystyle p\\mid D\{=\}1∼Beta\(1\.39,1\.10\),p∣D=0∼Beta\(0\.39,2\.10\)\\displaystyle\\sim\\mathrm\{Beta\}\(1\.39,1\.10\),\\quad p\\mid D\{=\}0\\sim\\mathrm\{Beta\}\(0\.39,2\.10\)\(i\)P⁡\(accept∣p\)\\displaystyle P\(\\mathrm\{accept\}\\mid p\)=β​pβ​p\+α⁡\(1−p\)=0\.5​at​p=αα\+β=0\.26\\displaystyle=\\tfrac\{\\beta p\}\{\\beta p\+\\alpha\(1\-p\)\}=0\.5\\;\\;\\text\{at\}\\;\\;p=\\tfrac\{\\alpha\}\{\\alpha\+\\beta\}=0\.26\(j\)Accbalanced⋆\\displaystyle\\mathrm\{Acc\}^\{\\star\}\_\{\\mathrm\{balanced\}\}≈80\.2%\\displaystyle\\approx 80\.2\\%\(k\)
Our evaluation uses a balanced 50/50 pool of accepted and rejected papers, so the relevant ceiling differs\. Beta\-Bernoulli conjugacy gives the conditional posteriors in step[i](https://arxiv.org/html/2610.31791#A1.Ex15), and under a uniform 50/50 class prior the posterior accept probability \(step[j](https://arxiv.org/html/2610.31791#A1.Ex16)\) equals0\.50\.5exactly atp=α/\(α\+β\)=0\.26p=\\alpha/\(\\alpha\+\\beta\)=0\.26\. The Bayes\-optimal rule in the balanced setting therefore predicts accept whenp\>0\.26p\>0\.26\. Averaging over the two conditional Beta distributions yields the balanced ceiling in step[k](https://arxiv.org/html/2610.31791#A1.Ex17), the value reported in the main text\.

## Appendix BData Curation Details

### B\.1Shortcuts in Prior Datasets

Existing accept/reject datasets often mix artifacts in ways that make labels recoverable from source, venue, or revision artifacts rather than from the paper’s content\. For instance, Deep Paper Gestalt\([Huang, 2018](https://arxiv.org/html/2610.31791#bib.bib18)\)pairs accepted CVPR papers against workshop papers as a reject pool, but the workshop format alone \(shorter page count, distinct headers\) is enough to shortcut the prediction problem without ever reading the paper\. We compare three public datasets, DeepReview ’24–’25\([Zhu et al\., 2025](https://arxiv.org/html/2610.31791#bib.bib56)\), Sakana’s AI Scientist ’22\([Lu et al\., 2024](https://arxiv.org/html/2610.31791#bib.bib28)\), and our raw ICLR ’24 \(before normalization\), against the selectedPaperScoreryears\. We tested several signals defined in[TableB\.1](https://arxiv.org/html/2610.31791#A2.T1), and[TableB\.2](https://arxiv.org/html/2610.31791#A2.T2)reports accept/reject percentages \(A/R\) for each\.

SignalDefinitionMarkdown / LaTeX sourceNative source artifacts \(e\.g\.\\begin\{equation\},\\cite\) survive in the text, indicating the dataset retained LaTeX source rather than rendering to PDF\.Author names visibleNon\-anonymous author block present before the abstract\.Affil\. before abstractInstitutional affiliation visible in the front matter before the abstract\.Has ack\.An Acknowledgments section heading is present anywhere in the paper\.Affil\. in ack\.Institutional affiliation strings appear inside the Acknowledgments section\.GitHub in abstractA GitHub URL appears in the abstract text\.Table B\.1:Identity\-leaking signals tested in the artifact shortcut audit\.Each signal is a heuristic that, if positively correlated with the accept/reject label, lets a classifier shortcut the prediction problem from formatting alone\.Among the comparators, DeepReview ’24 carries source for 78\.7% of accepts but only 41\.2% of rejects, a strong leakage shortcut\. AI Scientist ’22 has 100% of accept papers with visible author names against essentially zero reject papers, because the rejects come from anonymized OpenReview submissions while the accepts are camera\-ready PDFs\. Our selected years stay at most 0\.92% A / 1\.73% R on any signal\. Our audit shows that sourcing papers from the latest pre\-camera\-ready rebuttal artifact does not suffer from any of these shortcuts\.

SignalDeepReview’24DeepReview’25AI Scientist’22Ours raw’24Ours selectedyearsMarkdown / LaTeX source78\.7 / 41\.253\.0 / 37\.80\.0 / 0\.00\.0 / 0\.00\.01 / 0\.0Author names visible0\.0 / 0\.00\.0 / 0\.0100\.0 / 0\.699\.7 / 0\.40\.0 / 0\.0Affil\. before abs\.0\.7 / 0\.20\.4 / 0\.253\.8 / 0\.051\.1 / 0\.040\.0 / 0\.0Has ack\.16\.4 / 2\.84\.9 / 2\.874\.5 / 2\.873\.9 / 1\.70\.92 / 1\.73Affil\. in ack\.4\.4 / 1\.11\.3 / 0\.648\.9 / 0\.931\.8 / 0\.10\.07 / 0\.20GitHub in abs\.4\.0 / 0\.50\.5 / 0\.50\.0 / 0\.022\.5 / 1\.10\.0 / 0\.0Table B\.2:Artifact shortcut audit across public paper datasets\.We measure accept/reject percentages \(A/R\) of source, author, affiliation, acknowledgment, and code\-leakage signals\. The selectedPaperScoreryears avoid these shortcuts, while raw ICLR ’24 shows why camera\-ready accepts remain problematic\.
### B\.2Data Provenance and Ensuring Anonymity

Artifact availability across OpenReview venues\.We audit which submission, rebuttal, camera\-ready, and reject artifacts are public across the OpenReview venues and years we considered, and use the result to decide which year/version of each paper to keep\. Yellow cells in[TableB\.3](https://arxiv.org/html/2610.31791#A2.T3)mark the coverage used inICLR: for ICLR we keep ’20–’23 \(initial\-submission and rebuttal artifacts\) and ’25–’26 \(latest\-rebuttal PDFs that are public without the camera\-ready transition\), and we exclude ’24 entirely\. NeurIPS, ICML, and COLM publish only accepts on OpenReview, so we only source ICLR submissions from OpenReview\.

DecisionArtifactICLRNeurIPSICMLICMLCOLM’17–’19’20–’23’24’25’26’21–’25’23–’24’25’24–’25AcceptsInitialYYNNN†21–22NNNLatest rebuttalYYNYY†YNNNCamera\-readyYYYYY21–22YYYRejectsInitialYYNNN†NNNNLatest rebuttalYYY∗YY†YNYNPublic rejectsYYYYYOpt\.NOpt\.NTable B\.3:OpenReview artifact availability\.We audit which submission, rebuttal, camera\-ready, and reject artifacts are public across venues and years\. Yellow cells denote the year/artifact coverage used inICLR; ‘Y’ = public, ‘N’ = unavailable, and ‘Opt\.’ = author\-opt\-in rejects\.Why we exclude ICLR ’24\.For ’24, neither the initial submission nor the latest rebuttal PDF is publicly available, so the only artifact we can recover is the camera\-ready version of accepts paired with the latest rebuttal of rejects\. We attempted to recover an anonymized version of ICLR ’24 by performing manual anonymization on the camera\-ready PDF\. We strip identity\-bearing spans from text and whiten the same regions in rendered pages, but the camera\-ready transition still shifts page layout and section start positions \([FigureB\.1](https://arxiv.org/html/2610.31791#A2.F1)\)\. Vision picks this up\. Even after normalization, the model remains unusually strong on ’24, indicating that residual layout cues survive whiteout\. MinerU confirms the source of the problem\. Across years, the meanyy\-position of the introduction start is roughly matched between accepts and rejects, except in ’24 where the gap is notable\. This is consistent with author\-block expansion, GitHub insertion, and whitespace changes after masking, which is why ’24 is excluded from the benchmark and training pool\.

Figure B\.1:Why normalized ICLR ’24 remains shortcut\-prone for vision\.We strip identity\-bearing spans from text and whiten the same regions in rendered pages, but the camera\-ready transition still shifts page layout and section start positions\. Text mostly tolerates this cleanup; vision does not\.ICLR submission metadata across years\.Beyond artifact availability, the venues themselves change over time\.[TableB\.4](https://arxiv.org/html/2610.31791#A2.T4)compares page limits, rating scales, and artifact availability across ICLR ’20–’26\. The rating scale shifts repeatedly — 4\-point in ’20, 10\-point in ’21, then 6\-point from ’22 onward — so we normalize all reviewer ratings into percentile space before computing any rating\-aligned metric\. Page limits also creep up across the rebuttal and camera\-ready windows; we account for this with explicit length filters per year\.

Signal’20’21’22’23’24’25’26SourceCFP\+ORCFP\+ORCFP\+ORCFP\+ORCFP\+ORCFP\+ORGuide\+DLMin len\.–––––6NoneMax len\.108999109Max reb\./CR len\.10101010111111Rating scale4\-pt10\-pt6\-pt6\-pt6\-pt6\-pt6\-ptOrig\. sub\. avail\.YYYYNNNLatest reb\. avail\.YYYYNYYCamera\-ready avail\.YYYYYYNTable B\.4:ICLR year\-by\-year submission metadata\.We compare page limits, rating scales, and artifact availability to justify theICLRyear selection\. Rebuttal/camera\-ready limits include Acknowledgments, Ethics, and Reproducibility; CFP = Call for Papers, OR = OpenReview, and DL = downloaded during rebuttal before camera\-ready\.OpenReviewICLR ’20–’2645,328PDF select\+ MinerU OCR\+ redactJoin reviews,ratings, decisions40,224Per\-year 50:50balanced,85/5/10 split24,88812,442 A / 12,446 R\(a\)ICLRpipeline\.Decisions for both accepts and rejects are fully available on OpenReview, so the pipeline is simple — download OpenReview PDFs and metadata, extract markdown via MinerU, normalize, and subsample\. The final24,888balanced papers \(12,442 accept / 12,446 reject, per\-year 1:1 over ’20–’23 and ’25–’26, with ’24 excluded\) are used in training\.ML arXiv’20–’26434,579S3 bulk pull \+NeMo Curator434,443Venue style filedetected156,000MinerU OCR\+ body≥4\\geq 4155,424Paperlistmatch \+label treeTrain pool107,464HF \+ integrity\+ body 6–30101,659arXiv83,65641,828 A / 41,828 RarXiv\-m24,65012,325 A / 12,325 R\(b\)arXivpipeline\.Decisions are derived externally from public paperlists\([Yang et al\., 2026](https://arxiv.org/html/2610.31791#bib.bib50)\)\. The dominant drop is venue\-template detection, since without a recognized venue style file we cannot trust the venue tag and cannot assign a label\. Per\-venue 50:50 balancing produces thearXivdataset \(83,656papers,41,828 A / 41,828 R\) spanning NeurIPS, CVPR, NLP, AAAI, ICLR, ICML, and other ML venues\. Proportional per\-venue downsampling produces the smallerarXiv\-mdataset \(24,650papers,12,325 A / 12,325 R\) used throughout the paper for compute\-matched ablations\.
Figure B\.2:ICLRandarXivdata pipelines\.TheICLRpipeline is short because OpenReview publishes both accept and reject decisions for every submission\. ThearXivpipeline is long because both the venue and the decision must be recovered from each paper’s arXiv source tarball\.
### B\.3Our Data Pipeline

We run two independent workflows that produce two independent datasets\. TheICLRworkflow downloads ICLR submissions and decisions directly from OpenReview, runs MinerU, redacts identity, and subsamples\. ThearXivworkflow recovers both the venue and the decision from each paper’s arXiv source tarball and matches against public paperlists\.[Figures2\(a\)](https://arxiv.org/html/2610.31791#A2.F2.sf1)and[2\(b\)](https://arxiv.org/html/2610.31791#A2.F2.sf2)show the two funnels, and[TableB\.5](https://arxiv.org/html/2610.31791#A2.T5)summarizes the final canonical training pools used throughout the paper\.

SourceTotalAcceptRejectTrain / Val / TestICLR24,88812,44212,44621,141 / 1,252 / 2,495arXiv83,65641,82841,82871,124 / 4,174 / 8,358arXiv\-m24,65012,32512,32520,962 / 1,232 / 2,456Table B\.5:Canonical training datasets\.ICLRis per\-year 50:50 balanced across ’20–’23 and ’25–’26 \(’24 excluded\)\.arXivis per\-venue 50:50 balanced across NeurIPS, CVPR, NLP, AAAI, ICLR, ICML, and other ML venues\.arXiv\-mis a proportional per\-venue subsample ofarXivused throughout the paper for compute\-matched ablations\.##### ICLR\.

We pull every ICLR submission in OpenReview for ’20–’26 \(45,328 papers\) and, per year, the latest publicly available pre\-camera\-ready PDF\. PDFs run through MinerU to obtain markdown and per\-page images, then we redact identity\-bearing spans \(author block, affiliations, acknowledgments, GitHub links\)\. After joining with reviews and decisions and applying an 85/5/10 stratified split balanced per \(year, decision\) cell, we obtain 24,888 papers \(’20: 1,367; ’21: 1,610; ’22: 2,154; ’23: 3,088; ’25: 6,790; ’26: 9,879\)\.

##### arXiv\.

We start from 434,579 ML arXiv papers spanning ’20–’26 by update date\. We pull source directly from arXiv’s requester\-pays S3 mirror \(s3://arxiv/src/\) for every paper from January ’20 through our 05/01/’26 cutoff, and use NeMo Curator’s vendoredArxivIterator\([NVIDIA, 2024](https://arxiv.org/html/2610.31791#bib.bib33)\)to walk each monthly tar and recover the per\-paper\.texcontents\. We then scan every\.texagainst our tracked\-venue templates and keep only the tarballs that match a known venue style file\. The workflow detects the submission venue from the LaTeX source via these venue\-specific style files, strips identity\-bearing spans directly from the LaTeX source \(author block, affiliations, acknowledgments, GitHub links\) and recompiles the PDF, runs MinerU, and matches each in\-pool paper against every venue paperlist in the public Paper Copilot index\([Yang et al\., 2026](https://arxiv.org/html/2610.31791#bib.bib50)\)within aY−5Y\{\-\}5toY\+2Y\{\+\}2year window\. Papers that template for a venue with ground\-truth but find no paperlist match are tagged as implied rejects and kept, unless they trigger a workshop, short\-paper, or no\-ground\-truth filter that drops them\. Per\-venue 50:50 balancing with a stratified 85/5/10 split yields thearXivdataset of 83,656 papers spanning NeurIPS, CVPR, ACL, AAAI, ICLR, ICML, and other ML venues\. Throughout the paper we use the fullarXivset\. For compute\-matched ablations, we use the subsamplearXiv\-m, produced by proportional per\-\(venue, label\) downsampling ofarXivto 24,650 papers\.

## Appendix CSimple Feature Analysis

Table C\.1:Programmatic feature inventory\.We compute each probe feature from the concatenated human\-side paper text, truncated to the first 80,000 characters\. In feature names,per pg\.means per estimated body page\.FeatureDefinition𝐰ICLR\\mathbf\{w\}\_\{\\mathrm\{ICLR\}\}𝐰arXiv\\mathbf\{w\}\_\{\\mathrm\{arXiv\}\}Ablation mentionsSet to 1 if lowercase text contains ablation, otherwise 0\.\+0\.09\+0\.14Abstract sent\. lengthAverage sentence length \(in words\) of the abstract\.\+0\.12−\-0\.01Abstract sentence countNumber of sentences in the abstract\.\+0\.23−\-0\.01Abstract wordsNumber of whitespace\-delimited words in the abstract\.−\-0\.14\+0\.04Algorithms per pg\.Count case\-sensitive matches of the pattern Algorithm followed by whitespace and a digit, then divide by the estimated body\-page proxy\.−\-0\.03\+0\.03Appendix refs\. per pg\.Count case\-sensitive matches of Appendix followed by whitespace and a capital letter, then divide by the estimated body\-page proxy\.\+0\.13\+0\.10Baseline mentionSet to 1 if lowercase text contains baseline, otherwise 0\.0\.00\+0\.07Benchmark mentionSet to 1 if lowercase text contains benchmark, otherwise 0\.−\-0\.01\+0\.03Citation\-year countDetect four\-digit year strings in the extracted text, keep years in \[1990, 2026\], and compute log\(1 \+ number of retained year strings\)\.\+0\.10−\-0\.01Code mentionsIn lowercase text, count regex matches ofgithub,codefollowed byavail\(able\),openfollowed bysourc\(e\), orimplementationfollowed byavail\(able\)\.\+0\.04\+0\.06Confidence wordsIn lowercase text, count word\-boundary matches of these confidence terms: clearly, obviously, significantly, substantially, dramatically, remarkably, notably, undoubtedly, certainly, demonstrate, prove, establish, confirm, superior\. Divide by max\(total word count, 1\) and multiply by 1,000\.\+0\.03\+0\.02Contribution mentionsCount lowercase regex matches of words beginning withcontrib\(e\.g\.*contribute*,*contribution*\)\.0\.00−\-0\.03Cross\-refs\. per pg\.Count case\-sensitive references to Figure, Fig\., Table, Section, Appendix, Eq\., or Equation followed by optional whitespace, an optional opening parenthesis or bracket, and an uppercase letter or digit; divide by the estimated body\-page proxy\.\+0\.08\+0\.11CV topic termsCount case\-insensitive matches of these computer\-vision terms: image, vision, convolutional, CNN, object detection, segmentation, pixel, ResNet, ViT, visual, ImageNet, CIFAR, recognition\. Divide by max\(total word count, 1\) and multiply by 1,000\.\+0\.06−\-0\.02Efficiency topic termsCount case\-insensitive matches of these efficiency/compression terms: pruning, quantization, distillation, compression, efficient, lightweight, mobile, edge, inference speed, FLOP\. Divide by max\(total word count, 1\) and multiply by 1,000\.\+0\.01−\-0\.02Equation refs\. per pg\.Count case\-sensitive equation\-reference patterns: Eq or Equation followed by optional punctuation, whitespace, an optional opening parenthesis or bracket, and a digit, plus literal LaTeX \\eqref or \\ref occurrences; divide by the estimated body\-page proxy\.−\-0\.01\+0\.01Ethics indicatorSet to 1 if lowercase text contains ethic or broader impact, otherwise 0\.\+0\.01\+0\.03Fairness topic termsCount case\-insensitive matches of these fairness/ethics terms: fairness, bias, ethical, demographic, equit, disparate, protected attribute, discrimination\. Divide by max\(total word count, 1\) and multiply by 1,000\.0\.00−\-0\.01Federated termsCount case\-insensitive matches of these federated/privacy/distributed\-learning terms: federated, distributed learning, privacy, differential privacy, communication efficient\. Divide by max\(total word count, 1\) and multiply by 1,000\.−\-0\.07−\-0\.06Figure refs\. per pg\.Count case\-sensitive matches of Figure or Fig followed by optional punctuation, optional whitespace, and a digit; divide by the estimated body\-page proxy\.\+0\.18\+0\.10Generative topic termsCount case\-insensitive matches of these generative\-modeling terms: GAN, generative, VAE, variational, diffusion, autoencoder, latent space, generation, synthesis, sample quality, FID\. Divide by max\(total word count, 1\) and multiply by 1,000\.\+0\.030\.00Graph\-topic termsCount case\-insensitive matches of these graph\-learning terms: graph neural, GNN, node, edge, adjacency, message passing, graph convolution, knowledge graph, link prediction\. Divide by max\(total word count, 1\) and multiply by 1,000\.−\-0\.08−\-0\.01Hedge wordsIn lowercase text, count word\-boundary matches of these hedge terms: might, could, may, possibly, perhaps, likely, suggests, appears, seemingly, arguably, potentially, uncertain\. Divide by max\(total word count, 1\) and multiply by 1,000\.0\.00−\-0\.02Intro fractionIf an Introduction heading is detected, count words after that heading up to the next Markdown heading line beginning with \#, and divide by max\(total word count, 1\); otherwise use 0\.\+0\.01\+0\.01Limitations indicatorSet to 1 if lowercase text contains limitation, otherwise 0\.−\-0\.03−\-0\.06Lists per pg\.Count Markdown bullet lines that begin with optional whitespace and \- or \*, plus numbered\-list lines that begin with optional whitespace, digits, then \. or \); divide by the estimated body\-page proxy\.−\-0\.03−\-0\.03Long\-word ratioStrip punctuation from whitespace tokens, keep alphabetic words, and compute the share whose length is greater than 8 characters\.−\-0\.03\+0\.01Math notation/1k wordsCount pairs of dollar\-sign math delimiters as floor\(number of dollar signs / 2\), divide by max\(total word count, 1\), and multiply by 1,000\.\+0\.06\+0\.19Meta\-learning topic termsCount case\-insensitive matches of these meta\-learning/transfer/domain\-adaptation terms: meta learning, few shot, zero shot, transfer learning, domain adaptation, multi task\. Divide by max\(total word count, 1\) and multiply by 1,000\.−\-0\.01\+0\.02Multimodal termsCount case\-insensitive matches of these multimodal/vision\-language terms: multimodal, cross modal, vision language, CLIP, image text, visual question\. Divide by max\(total word count, 1\) and multiply by 1,000\.−\-0\.010\.00Negative wordsIn lowercase text, count word\-boundary matches of these negative terms: fail, limitation, drawback, weakness, shortcoming, disadvantage, unable, cannot, lack, insufficient, poor, worse, degrade, suffer\. Divide by max\(total word count, 1\) and multiply by 1,000\.\+0\.04\+0\.01NLP topic termsCount case\-insensitive matches of these NLP/language\-modeling terms: NLP, natural language, text classification, sentiment, translation, language model, transformer, BERT, GPT, token, vocabulary, corpus, parsing, NER, named entity\. Divide by max\(total word count, 1\) and multiply by 1,000\.\+0\.04\+0\.03Novelty mentionsCount lowercase regex matches of the word novel\.\+0\.01\+0\.01Optimization topic termsCount case\-insensitive matches of these optimization terms: optimization, convergence, SGD, Adam, learning rate, gradient descent, momentum, regularization, batch size, weight decay\. Divide by max\(total word count, 1\) and multiply by 1,000\.−\-0\.02−\-0\.02Paper lengthCount whitespace\-delimited words in the extracted text and compute log\(1 \+ word count\)\.\+0\.47\+0\.48Positive wordsIn lowercase text, count word\-boundary matches of these positive terms: outperform, improve, achieve, superior, advantage, benefit, effective, efficient, robust, strong, excellent, promising, remarkable, impressive, advance\. Divide by max\(total word count, 1\) and multiply by 1,000\.\+0\.020\.00Proposal mentionsCount lowercase regex matches of words beginning with propos\.−\-0\.05−\-0\.01Rare\-word ratioFrom the first 5,000 lowercased alphabetic words, compute the number of unique words that appear exactly once divided by max\(number of unique words, 1\)\.−\-0\.14−\-0\.06Recent citation shareDetect four\-digit year strings in the extracted text, keep years in \[1990, 2026\], and compute count\(years \>= 2024\) divided by max\(number of retained year strings, 1\)\.\+0\.09−\-0\.15Related workSet to 1 if lowercase text contains related work, otherwise 0\.\+0\.02\+0\.09Reproducibility indicatorSet to 1 if lowercase text matches reproduc, code followed later by avail, or open followed by one character then sourc, otherwise 0\.−\-0\.02−\-0\.06RL terms/1k wordsCount case\-insensitive matches of these reinforcement\-learning terms: reinforcement learning, reward, policy gradient, MDP, Q\-learning, actor critic, environment, agent, exploration, exploitation, Atari, MuJoCo\. Divide by max\(total word count, 1\) and multiply by 1,000\.−\-0\.02−\-0\.07Sections per pg\.Count Markdown headings that start a line with one to three hash marks followed by whitespace, then divide by the estimated body\-page proxy\.−\-0\.08−\-0\.15Self\-references/1k wordsCount these lowercase phrases: our method, our approach, our model, our framework, our algorithm, our technique, our work, our paper, and our contribution\. Divide by max\(total word count, 1\) and multiply by 1,000\.\+0\.01\+0\.08Self\-supervised topic termsCount case\-insensitive matches of these self\-supervised/contrastive\-learning terms: self supervised, contrastive, SimCLR, BYOL, MoCo, pretext task, representation learning, unsupervised pre\. Divide by max\(total word count, 1\) and multiply by 1,000\.\+0\.03\+0\.02Sentence lengthSplit the extracted text on periods, exclamation marks, and question marks, keep spans longer than 10 characters, and compute total whitespace\-delimited word count divided by max\(number of retained spans, 1\)\.−\-0\.18−\-0\.24Sentence\-length variationSplit the extracted text into retained sentences as above, compute the standard deviation of sentence word counts, and divide by max\(avg\_sentence\_length, 1\)\.\+0\.02\+0\.04Significance mentionsSet to 1 if lowercase text matches statistic followed later by signific, p followed by one character then value, confidence interval, standard deviation, or error bar, otherwise 0\.−\-0\.01−\-0\.03Speech topic termsCount case\-insensitive matches of these speech/audio terms: speech, audio, acoustic, spectrogram, ASR, TTS, voice, phoneme, waveform\. Divide by max\(total word count, 1\) and multiply by 1,000\.\+0\.010\.00Table refs\. per pg\.Count case\-sensitive matches of Table followed by whitespace and a digit, then divide by the estimated body\-page proxy\.−\-0\.03\+0\.06Theory markersCount case\-sensitive matches of Theorem, Lemma, Proposition, Corollary, Definition, and Proof\.\+0\.03\+0\.06Theory topic termsCount case\-insensitive matches of these theory/proof/bound terms: theorem, proof, lemma, proposition, corollary, bound, complexity, PAC, VC dimension, Rademacher, concentration inequality\. Divide by max\(total word count, 1\) and multiply by 1,000\.\+0\.07\+0\.01Title colonSet to 1 if the extracted title contains a colon, otherwise 0\.\+0\.05\+0\.05Title lengthUse metadata title when available, otherwise the first Markdown H1 heading, and count characters\.−\-0\.07\+0\.01Title numberSet to 1 if the extracted title contains any digit, otherwise 0\.\+0\.060\.00Title question markSet to 1 if the extracted title contains a question mark, otherwise 0\.\+0\.01\+0\.01Title wordsUse metadata title when available, otherwise the first Markdown H1 heading, and count whitespace\-delimited title words\.−\-0\.03−\-0\.09Training detailsCount how many of these phrases appear case\-insensitively in the text: learning rate, batch size, epoch, GPU, TPU, A100, V100, training time\.−\-0\.05−\-0\.07Vocabulary richnessFrom the first 5,000 lowercased alphabetic words, compute number of unique words divided by max\(number of sampled words, 1\)\.\+0\.210\.00Word lengthStrip punctuation from whitespace tokens, keep tokens containing only alphabetic characters, and average their character lengths, using 0 when no alphabetic words are found\.−\-0\.07−\-0\.03The main text reports the top five accept\- and reject\-associated features from separate ICLR andarXivfeature probes in[Figure4](https://arxiv.org/html/2610.31791#S4.F4)\.[TableC\.1](https://arxiv.org/html/2610.31791#A3.T1)lists every feature in the probe with its definition\. The columnswICLRw\_\{\\mathrm\{ICLR\}\}andwarXivw\_\{\\mathrm\{arXiv\}\}are the signed standardized coefficients from logistic\-regression probes trained on all features against the ICLR andarXivdecision label, respectively\. The probe excludes source identifiers, direct venue labels, ratings, citations, author names, and non\-comparable body\-page counts\. The strongest individual signal — paper length on both sources — saturates atw≈\+0\.48w\\approx\+0\.48, an order of magnitude below whatPaperScorerachieves end\-to\-end\. Reject\-side magnitudes are smaller still \(\|w\|≤0\.25\|w\|\\leq 0\.25\), and most features sit between−0\.05\-0\.05and\+0\.10\+0\.10, indicating that simple structural shortcuts do not cleanly separate accept from reject\.

## Appendix DDo Frontier Models Memorize the Test Decisions?

GPT\-5\.5’s December ’25 knowledge cutoff postdates the public release of ICLR ’25 decisions and most of ourarXivtest years, so a natural worry is that its baseline accuracy comes from remembering outcomes rather than judging papers\. Gemini\-3\.1\-Pro \(January ’25 cutoff\) predates both\. We test this in two steps\. First, we ask whether the models remember our test papers at all\. Second, and more importantly, we ask whether remembering a paper makes the model any more accurate on it\.

##### Do the models remember the papers?

We sample 200 papers per corpus \(100 accepted, 100 rejected\) from the papers on which GPT\-5\.5 is evaluated in[Table3](https://arxiv.org/html/2610.31791#S4.T3), so that every probed paper also has a prediction from the main table\. Each paper is probed with five*label\-recall*cues that ask the model whether it remembers the paper’s decision:

- •title— title only\.
- •title\_abstract— title plus abstract\.
- •mc— multiple choice among accept, reject, and a distractor, given the title and abstract\.
- •continuation— the first sentence after the abstract, asking whether the model recognizes the paper\.
- •web\_search\_framing— a prompt phrased as if the model could look up the venue and outcome\.

For each cue we report the*claim rate*, the fraction of papers where the model says it remembers the decision, and the*outcome accuracy when claimed*\. A sixth*content\-recall*cue asks the model to continue the abstract from its title, scored by ROUGE\-L\([Lin, 2004](https://arxiv.org/html/2610.31791#bib.bib27)\)\. Gemini\-3\.1\-Pro was probed on an earlier 50\-paper draw per corpus with the same cues\.

[TableD\.1](https://arxiv.org/html/2610.31791#A4.T1)shows that GPT\-5\.5 does recognize many of our test papers, and that recognition follows its cutoff exactly\. OnICLR, thetitlecue is claimed for 30% of ’25 papers but only 7% of ’26 papers, and onarXivfor 44% of ’24 papers but 17% of ’25 and 23% of ’26\. Gemini\-3\.1\-Pro claims recall for at most 4% ofICLRpapers under any cue\. Recognition is not the same as remembering the decision, however\. When GPT\-5\.5 claims to recall a decision it is right only 50–73% of the time, and this is inflated by an accept bias: it says “accept” for 65% of theICLRpapers it claims to remember, so balanced accuracy over its claimed decisions is 44%, no better than chance\. Content recall is absent: mean ROUGE\-L on abstract continuation is≤0\.004\\leq 0\.004on every slice and no paper reaches0\.50\.5\.

##### Does remembering a paper help?

If GPT\-5\.5 were retrieving remembered decisions, it would be more accurate on the papers it recognizes\. We split each probed corpus into papers it claimed to recall under at least one cue and papers it never claimed, and compare its accuracy from[Table3](https://arxiv.org/html/2610.31791#S4.T3)on the two groups \([TableD\.2](https://arxiv.org/html/2610.31791#A4.T2)\)\. Recognized papers are no easier on either dataset \(permutationp=\.28p=\.28and\.88\.88\), including on ICLR ’25, the onlyICLRyear inside its cutoff, and accuracy does not rise with the number of cues under which a paper is recognized\. Papers whose recalled decision happened to be correct do score higher, but this is circular, since those are the papers where the model’s impression already matched the outcome\.

We conclude that GPT\-5\.5 recognizes a substantial fraction of our test papers but does not retrieve their decisions, and that its baseline accuracy in[Table3](https://arxiv.org/html/2610.31791#S4.T3)is not explained by memorization\.

ICLRarXivCueModel’25’26’24’25’26titleGPT\-5\.530 / 797 / 5044 / 8217 / 3323 / 43titleGemini\-3\.1\-Pro0 / 00 / 05 / 1000 / 00 / 0title\_abstractGPT\-5\.522 / 654 / 7546 / 8313 / 5010 / 33title\_abstractGemini\-3\.1\-Pro0 / 00 / 015 / 670 / 00 / 0mcGPT\-5\.533 / 650 / 073 / 7816 / 590 / 0mcGemini\-3\.1\-Pro0 / 00 / 030 / 830 / 00 / 0continuationGPT\-5\.552 / 5425 / 4860 / 6639 / 4843 / 31continuationGemini\-3\.1\-Pro0 / 00 / 05 / 1000 / 00 / 0web\_search\_framingGPT\-5\.544 / 5125 / 4852 / 7042 / 5313 / 50web\_search\_framingGemini\-3\.1\-Pro4 / 1000 / 010 / 1000 / 00 / 0Content recall \(abstract completion\) — ROUGE\-L mean / frac\. ROUGE\-L≥0\.5\\geq 0\.5GPT\-5\.50\.003 / 0\.000\.003 / 0\.000\.004 / 0\.000\.002 / 0\.000\.000 / 0\.00Gemini\-3\.1\-Pro0\.000 / 0\.000\.000 / 0\.000\.005 / 0\.000\.000 / 0\.000\.000 / 0\.00Table D\.1:Frontier memorization probes, by corpus and year\.Label\-recall cells report*claim rate \(%\) / outcome accuracy when claimed \(%\)*; abstract\-completion cells report*ROUGE\-L mean / fraction with ROUGE\-L≥0\.5\\geq 0\.5*\. GPT\-5\.5 is probed on 200 papers per corpus \(ICLR’25:n=93n\{=\}93, ’26:n=107n\{=\}107;arXiv’24:n=63n\{=\}63, ’25:n=107n\{=\}107, ’26:n=30n\{=\}30\), Gemini\-3\.1\-Pro on 50 \(ICLR’25:n=26n\{=\}26, ’26:n=24n\{=\}24;arXiv’24:n=20n\{=\}20, ’25:n=23n\{=\}23, ’26:n=7n\{=\}7\)\. GPT\-5\.5’s claim rate tracks its December ’25 cutoff, but its claimed decisions are right only about half to three quarters of the time\.RecalledNot recalledFull test poolCorpusYearnnAccnnAccnnAccICLR’257457187033961ICLR’264762606449565ICLRall12158786583463arXiv’24576465026172arXiv’256868386136866arXiv’26176413769464arXivall14267576672368Table D\.2:GPT\-5\.5 accuracy on papers it recognizes vs\. papers it does not\.Accuracy \(%\) of GPT\-5\.5 \(vision, tuned as in[Table3](https://arxiv.org/html/2610.31791#S4.T3)\) on probed papers it claimed to recall under at least one label\-recall cue, on probed papers it never claimed, and on the full test pool\. Recognized papers are no easier for the model, including on ICLR ’25, the onlyICLRyear inside its cutoff\.

## Appendix EBackbone Choice: Qwen2\.5 vs\. Qwen3

To check whether our backbone choice drives the results, we retrain the text model onICLRwith a Qwen3\-8B backbone\([Yang et al\., 2025](https://arxiv.org/html/2610.31791#bib.bib49)\)using the same SFT recipe asPaperScorer\-Text, and compare against the Qwen2\.5\-7B backbone used throughout the paper\.[TableE\.1](https://arxiv.org/html/2610.31791#A5.T1)reports overallICLRaccuracy and per\-class recall together with each model’s knowledge cutoff\. The newer Qwen3 generation includes public ICLR ’25 outcomes \(text released April 28, ’25; VL on September 22, ’25\) but not ICLR ’26, whereas Qwen2\.5 predates both\.

Text backboneKnowledge cutoffAccRAR\_\{A\}RRR\_\{R\}Qwen2\.5\-7B \(ours\)December ’24666369Qwen3\-8BApril ’25657259Table E\.1:Text backbone comparison onICLR\.Overall accuracy \(Acc\), accept recall \(RAR\_\{A\}\), and reject recall \(RRR\_\{R\}\) in percent\. Qwen3’s cutoff includes ICLR ’25 decisions but not ’26\.Accuracy is close \(65%65\\%vs\.66%66\\%\), and Qwen3 skews toward accept\. The backbone matters less than the training data, and Qwen2\.5’s cutoff predates every test decision, so we use it throughout\.

## Appendix FTraining Variations

### F\.1Evaluation and Training Prior Investigation

The main text reports balanced test metrics because our decision objective is Acc\. This is compatible with natural\-rate reporting when balanced and natural samples come from the same curated domain distribution\. The accept and reject recalls estimated on a balanced test can be reweighted by the natural accept prior\.[TableF\.1](https://arxiv.org/html/2610.31791#A6.T1)checks this directly for matchedICLRandarXiv7B text and vision predictors\. Across all four cells, the empirical natural\-rate accuracy lands within a few percentage points of the prior\-extrapolated value, so the balanced test keeps the metric aligned with the objective\.

DomainPredictorBalanced AccExtrapolated naturalEmpirical natural𝚫\\boldsymbol\{\\Delta\}ICLRVision68\.368\.065\.6\-2\.4ICLRText66\.067\.264\.1\-3\.1arXivVision73\.474\.275\.1\+1\.0arXivText71\.371\.772\.0\+0\.3Table F\.1:Balanced\-to\-natural extrapolation\.For matched ICLR→\\rightarrowICLR and arXiv→\\rightarrowarXiv diagnostics, balanced\-test accept and reject recalls are converted to raw accuracy under the natural test prior\.Δ\\Deltais empirical natural\-prior accuracy minus this extrapolated value\.The training\-prior diagnostic in[Table1](https://arxiv.org/html/2610.31791#S3.T1)uses validation thresholds only to reveal class\-prior bias\. All headlinePaperScorerresults use native decisions unless stated otherwise \(e\.g\., calibration\)\.

### F\.2Ablating Training Distribution and Paper Representation

We explored two families of training ablations onICLRto improve decision accuracy\. The first filtered training papers to high\-quality accepts and rejects by reviewer rating, dropping borderline papers\. The second tested additional context suffixes such as paper statistics, agent\-generated reviews, and vision\-text fusion to support the decision objective\. Both are summarized in[TableF\.2](https://arxiv.org/html/2610.31791#A6.T2)\.

Table[F\.2](https://arxiv.org/html/2610.31791#A6.T2)a shows that training only on pre\-’26 data does not transfer well to ’26\. To improve the quality of the corpus, we tried to filter out borderline cases via*Rating\-Agreeing*, which keeps only accepts in\[0\.6,1\]\[0\.6,1\]and rejects in\[0,0\.4\]\[0,0\.4\]by percentile rating\. It helps on ’25, but once ’26 is included, it does not improve either modality noticeably\. Table[F\.2](https://arxiv.org/html/2610.31791#A6.T2)b explains why:’26 is fundamentally a different distribution, since the rebuttal process was cut short, author\-reviewer discussion was not allowed, and reviewers could not raise scores, yielding the smallest accept\-reject gap\. Table[F\.2](https://arxiv.org/html/2610.31791#A6.T2)c shows that additional context helps text, especially for ’26 transfer, but harms vision, our strongest model\. We therefore train on the simplest paper\-only pool,’20–’23, ’25–’26∗, which gives the strongest vision results without sacrificing text performance\.

Training Data’25’26VisionTextVisionText’20–’23, ’2568\.168\.362\.663\.5\+ Rating\-Agreeing\+1\.6\+0\.3\-0\.9\-1\.7’20–’23, ’25–’26∗70\.969\.466\.563\.7\+ Rating\-Agreeing\-1\.0\-2\.2\-1\.2\+0\.9\(a\) Training Data

YearAcc\.Rej\.Gap’206\.253\.672\.58’216\.535\.031\.50’226\.624\.671\.95’236\.604\.671\.93’256\.474\.801\.67’265\.403\.961\.44\(b\) Rating Gap

Additional Context’25’26VisionTextVisionText’20–’23, ’25–’26∗70\.969\.466\.563\.7\+ Simple Features\-0\.4\+0\.6\-1\.5\+0\.7\+ Qwen Reviews\-1\.3\-2\.1\-0\.4\+1\.0\+ Gemini Reviews\-3\.8\-1\.1\-2\.0\+0\.7\(c\) Additional Context

Table F\.2:Data mixture breakdown\.We report fixed\-cost data and context ablations, with underlined rows as bases and following rows as deltas\. Rating\-agreeing data helps ’25 but not the final ’26\-oriented setup; ’26 has the smallest accept\-reject rating gap; and extra feature/review context helps text modestly but not the strongest vision model\.We also tested several fixed\-compute design choices beyond the paper\-only input\. As Table[F\.2](https://arxiv.org/html/2610.31791#A6.T2)c shows, suffixing text with paper statistics or agent\-generated Qwen/Gemini reviews can help text modestly, especially on ’26, but none of these additions improves the strongest vision model\. The same is true for other fixed\-cost variants such as suffixing text with full vision, vision with full text, or prepending 1k<think\>tokens\. None of these design choices consistently improves performance, so we simply use paper→\\todecision prediction\.

Paper statistics features\.We appended structured metadata such as page count, number of references, inline math count, cross\-reference density, and other surface\-level features to each training sample\. While these features are individually predictive, adding them to the SFT prompt did not improve over the paper\-only baseline, suggesting that the language model already captures these signals implicitly from the paper text\.

Thinking tokens\.Inspired by chain\-of\-thought prompting, we prepended dummy reasoning tokens before the decision token during SFT, allowing the model additional computation before committing to a prediction\. This did not improve accuracy, consistent with our finding that the decision task benefits more from pattern recognition over the full paper than from explicit multi\-step reasoning\.

Multimodal fusion\.We tested providing both the full extracted text and screenshots of all pages simultaneously \(text \+ vision\), as well as the reverse \(vision \+ text\)\. Neither configuration outperformed the single\-modality baselines, likely because the two representations are largely redundant for the decision task, and the additional context length increases noise without adding a complementary signal\.

Synthetic review augmentation\.Appending model\-generated reviews \(Qwen3\.5\-122B\-A10B or Gemini\-2\.5\-Flash\) to training prompts added∼1\.8\{\\sim\}1\.8k–4\.84\.8k tokens per sample\. These reviews occasionally helped on ICLR ’25 but hurt on ICLR ’26, suggesting that review language is more year\-specific than paper content and does not transfer reliably\.

### F\.3Reverse\-Order Ablation

To test whether SFT models rely on positional shortcuts, we reversed the paper order while keeping labels fixed — text examples reverse section blocks, and vision examples reverse page order\. Training and evaluating in the same reversed condition only loses about 1–1\.3% relative to the forward baseline, and forward\-trained checkpoints remain usable on reversed validation inputs \(Table[F\.3](https://arxiv.org/html/2610.31791#A6.T3)\)\. This suggests the models are not merely keying on early\-section or early\-page cues; vision is especially order\-invariant\.

TrainingEvaluationAccuracynMatched training and evaluation orderForward \(vision\)Forward68\.4%2,498Reversed \(vision\)Reversed67\.2%2,498Forward \(text\)Forward67\.0%2,495Reversed \(text\)Reversed65\.7%2,495Forward\-trained checkpoints, reversed\-order evaluationForward \(vision\)Reversed66\.1%1,253Forward \(text\)Reversed64\.9%1,252Table F\.3:Reverse\-order ablation\.Reversing text section blocks or vision page order during training and evaluation costs only11–1\.31\.3points of accuracy\. Forward\-trained checkpoints also stay close to their forward accuracy when evaluated on reversed inputs, indicating that vision in particular is largely order\-invariant\.

## Appendix GCross\-Source Generalization

How does a model trained on one source behave on the other? We evaluate theICLR\-trained andarXiv\-trained 7B vision models on each source’s test set and onarXiv\-ICLR, theICLRsubset ofarXiv\(Table[G\.1](https://arxiv.org/html/2610.31791#A7.T1)\)\.

Train→\\totest \(7B vision\)RAR\_\{A\}RRR\_\{R\}Accnn*Generalizes: accept and reject recall stay balanced*ICLR→\\toICLR69\.267\.468\.31670ICLR→\\toarXiv\-ICLR78\.369\.573\.9105*Reject\-predictive off\-domain:RRR\_\{R\}holds,RAR\_\{A\}collapses*arXiv→\\toICLR52\.568\.960\.71670ICLR→\\toarXiv\(all\)50\.178\.264\.21414Table G\.1:Cross\-source generalization \(7B vision\)\.Accept recall \(RAR\_\{A\}\), reject recall \(RRR\_\{R\}\), and accuracy \(Acc\), in %, overnntest papers\.ICLRtraining reachesarXiv\-ICLRwithout loss\. But faced with the other source’s full set, a model still flags its rejects and recovers only about half of its accepts\.TheICLR\-trained model transfers toarXiv\-ICLRwith no measurable loss \(Table[G\.1](https://arxiv.org/html/2610.31791#A7.T1)\)\. With only105105papers, though, its accuracy there has a95%95\\%confidence interval that overlaps its in\-domain accuracy, so we read this as equal performance, not a real gain\. A much larger effect appears when either model is run on the full distribution of the other source\. Both become reject\-predictive, recovering only about half of the true accepts, so each defaults to rejection on papers that look unfamiliar\.

This is not a threshold problem\. We retuned the accept/reject threshold on a small labeled sample from the target source, and it only swapped the error\. Accepts then rose above88%88\\%while rejects fell below40%40\\%, and accuracy stayed flat\. Even the best threshold chosen directly on the test labels barely changed it\. The model’s ranking of papers, not the threshold, is what fails under the shift\.

## Appendix HPrompting and Extended Experiments

[TableH\.1](https://arxiv.org/html/2610.31791#A8.T1)summarizes the prompts, decoding settings, and context limits used across our evaluation suite\. Frontier Gemini/GPT models use a single venue\-generic rating\-plus\-decision prompt with*high reasoning*effort and temperature1\.01\.0\. Our RL prompt mirrors this setup with temperature0\.70\.7\. After SFT we decode greedily \(temperature00\)\. Open\-source Qwen baselines use the same prompt family as SFT/RL but at higher temperature to explore prompt corrections, since the base models predict acceptance for almost every paper out of the box\.[SectionH\.1](https://arxiv.org/html/2610.31791#A8.SS1)documents each prompt verbatim,[SectionH\.2](https://arxiv.org/html/2610.31791#A8.SS2)reports reasoning\-token usage across frontier evaluation cells, and[SectionH\.3](https://arxiv.org/html/2610.31791#A8.SS3)reports the prompt\-correction experiments on Qwen and Gemini\-2\.5\-Flash\.

FamilyPrompt/input protocolDecoding and limitsFrontier Models \(Gemini/GPT\)Single venue\-generic rating\-plus\-decision prompt\.Gemini\-3\-Flash, GPT\-5\.4\-Mini, Gemini\-3\.1\-Pro, and GPT\-5\.5; high reasoning/thinking effort, temperature1\.01\.0, max tokens32,00032\{,\}000\.Coding agentsSame rating\-plus\-decision prompt in single\-paper workspaces with markdown and screenshots\.Codex \(GPT\-5\.5\) and Claude Code \(Sonnet\-4\.6\) with high reasoning effort; web tools disabled\.Qwen Base PromptingAnswer\-only prompt matching SFT, or reasoning prompt matching RL\. Target\-dataset venue wording\.temperature0\.950\.95, top\-p=0\.7p=0\.7, top\-k=50k=50, context24,48024\{,\}480\. Few\-shot runs use context60,00060\{,\}000with 4x YaRN\.SFTVenue\-specific answer\-only prompts\.temperature00\(greedy decoding\), context24,48024\{,\}480\.GRPO / RLVenue\-specific reasoning prompt\.temperature0\.70\.7\. Text max response40964096, max prompt25,48025\{,\}480\. Vision max response20462046, max prompt24,48024\{,\}480\.Table H\.1:Prompting and decoding configurations\.Prompts, decoding settings, and context limits across external reviewers, direct Qwen prompting, SFT, and GRPO\.### H\.1Prompts

In this section we denote the prompts and decoding settings used during prompting, SFT, and RL\. We omit the paper body from the prompts for brevity\.

#### H\.1\.1Base Open\-Source Prompting

No Reasoning Prompt\.Open\-source Qwen and SFT runs share a minimal answer\-only prompt\. The ICLR variant tells the model that ICLR accepts around 30%, while thearXivvariant uses venue\-generic wording because the source\-derived dataset spans multiple ML venues\. For cross\-source evaluation, the wording follows the target dataset rather than the training source\.

\[System\]You are an expert academic reviewer tasked with evaluating research papers\.\[User \-\-\- ICLR\]I am giving you a paper\. I want to predict its acceptance outcome at ICLR\.\- Your answer will either be: \\boxed\{Accept\} or \\boxed\{Reject\}\- Note: ICLR generally has a ˜30% acceptance rate\[User \-\-\- arXiv\]I am giving you a paper submitted to a top machine\-learning venue\. Predictits acceptance outcome\.\- Your answer will either be: \\boxed\{Accept\} or \\boxed\{Reject\}\- Note: typical top\-tier ML venues have ˜25\-30% acceptance rates

Reasoning Prompt\.Reasoning prompts move the instruction into the system turn and leave the user turn as the paper body\. The fullICLRvariant is shown below\. ThearXivvariant replaces the first sentence with the venue\-generic top\-ML wording\. Vision reasoning prompts add that the paper is provided as Markdown text plus screenshots of each page, and ask the model to inspect page layouts\.

\[System\]You are an expert academic reviewer for the ICLR conference, predicting whether apaper will be accepted or rejected\. ICLR generally has a ˜30% acceptance rate\.Your response must include two parts:1\. \*\*Reasoning\*\*: A detailed, free\-flowing chain of thought enclosed in‘<think\>‘ and ‘</think\>‘ tags\.2\. \*\*Final Answer\*\*: A single boxed verdict enclosed in ‘<answer\>‘ and‘</answer\>‘ tags\.\* The reasoning should resemble a stream of consciousness: explore, testhypotheses, backtrack if necessary, reflect, and refine\.\* Use reasoning strategies such as planning, exploration, evaluation, andreflection\.\* The answer section must be enclosed in ‘<answer\>‘ and ‘</answer\>‘ tags\.\* The answer is exactly ‘\\boxed\{Accept\}‘ or ‘\\boxed\{Reject\}‘, with no other text\.\[User\]<paper body as Markdown; vision variant also attaches per\-page screenshots\>

#### H\.1\.2Frontier Model Prompting

Frontier Reasoning Prompt\.Frontier Gemini and GPT models \(Gemini\-3\.1\-Pro and GPT\-5\.5\) use a single venue\-generic rating\-plus\-decision prompt for bothICLRandarXiv\. We chose this prompt after finding no consistent gain from separateICLR\- andarXiv\-specific variants\. In practice, the venue is usually recoverable from the paper content and formatting\.

\[System\]You are an expert academic ML researcher and reviewer tasked with evaluatingresearch papers\. I am giving you a paper submitted to a top machine\-learning venue\.Predict a rating from 1\-10 and the paper’s acceptance outcome\.\- Your answer will be of the form: \\boxed\{rating,decision\} where rating is aninteger in \[1,10\] and decision is either \{Accept, Reject\}\- Note: typical top\-tier ML venues have ˜25\-30% acceptance rates\[User\]<paper body as Markdown; vision variant also attaches per\-page screenshots\>We use the generated rating as the score, with a higher rating indicating a higher predicted probability of acceptance, and the generated decision as the hard label\. In our main results we additionally calibrate the model’s decision using its rating and accuracy performance on the held\-out validation set, which generally improves performance\.

Frontier Reasoning Agent Workspace\.Coding\-agent baselines use the same Frontier Reasoning Prompt above \(carried inAGENTS\.override\.md\), but place each paper in a self\-contained workspace that the agent can inspect with normal file and image tools\. The paper body is stripped intopaper\.md, rendered pages are copied intoimages/, and the final answer must be written toPREDICTIONS\.json\. Both Codex \(GPT\-5\.5\) and Claude Code \(Sonnet\-4\.6\) run with high reasoning effort, no web tools, and a validator that rejects malformed outputs before grading\.

agent\_workspaces/<sample\>/<domain\>/<paper\_id\>/\|\-\- AGENTS\.override\.md \# rating \+ accept/reject instruction\|\-\- CLAUDE\.md \-\> AGENTS\.override\.md\|\-\- id\.txt \# paper identifier\|\-\- paper\.md \# stripped markdown paper body\|\-\- images/\| \|\-\- page\_1\.png\| \|\-\- page\_2\.png\| ‘\-\- \.\.\.\|\-\- main\.py \# calls the schema validator‘\-\- results/\|\-\- codex/\| \|\-\- PREDICTIONS\.json\| ‘\-\- session\.jsonl‘\-\- claude/\|\-\- PREDICTIONS\.json‘\-\- session\.jsonlGrading joins each validPREDICTIONS\.jsonwith the held\-out ground\-truth manifest and reports accuracy, accept/reject recall, and rating correlation\. The workspace tree is intentionally one paper at a time, matching the single\-shot API setup while letting the agent choose whether to read markdown, rendered pages, or both modalities together\.

### H\.2Frontier Model Reasoning Usage

[TableH\.2](https://arxiv.org/html/2610.31791#A8.T2)reports reasoning\-token usage across the fourICLR/arXivtext/vision evaluation cells\. Both frontier models spend a substantial amount of reasoning compute per paper\. On text inputs, median reasoning tokens land between1\.31\.3k and2\.32\.3k per paper, and on vision inputs between1\.51\.5k and2\.12\.1k\. P95 reasoning length stretches to3\.63\.6k–4\.74\.7k, so a non\-trivial tail of papers triggers2×2\\timesor more compute than the median\. GPT\-5\.5 has a lower median than Gemini\-3\.1\-Pro at every cell, but its tail is just as heavy: its P95 sits at3\.63\.6k–4\.54\.5k against Gemini\-3\.1\-Pro’s4\.14\.1k–4\.74\.7k, and its P95\-to\-median ratio is the larger of the two, so its compute varies more from paper to paper\.ICLRcells consistently use more reasoning than the matchedarXivcells \(e\.g\. Gemini\-3\.1\-Pro at2,0852\{,\}085vision\-median tokens onICLRvs1,8251\{,\}825onarXiv\), suggesting thatICLRpapers either look harder or hit prompts that elicit more deliberation\.

ModelCellnnMedianMeanP95Gemini\-3\.1\-ProarXivvision360182521104121Gemini\-3\.1\-ProarXivtext354194422894594Gemini\-3\.1\-ProICLR vision401208523534618Gemini\-3\.1\-ProICLR text396231126114702GPT\-5\.5arXivvision360153618874095GPT\-5\.5arXivtext362133615763580GPT\-5\.5ICLR vision416204421094482GPT\-5\.5ICLR text418153619294383Table H\.2:Frontier reasoning\-token usage\.Median, mean, and P95 reasoning\-token counts for the stratified high\-reasoning Gemini and GPT evaluation runs\.
### H\.3Correcting Model Biases with Prompting

Base open\-source models we tested predict acceptance for almost every paper\. Out of the box, Qwen2\.5\-7B with a minimal freeform\-boxed prompt accepts more than99%99\\%of papers \([Table3\(a\)](https://arxiv.org/html/2610.31791#A8.T3.st1), Freeform Boxed Standard:RA=99\.4R\_\{A\}=99\.4,RR=0\.9R\_\{R\}=0\.9\)\. We therefore tested a range of prompt corrections, few\-shot conditioning, and multi\-step peer\-review emulation across two open\-source models \(Qwen2\.5\-7B text and vision, all2,4952\{,\}495ICLRtest papers\) and a cheap frontier model \(Gemini\-2\.5\-Flash text and vision, a500500\-paper stratified subset\)\. All numbers are averages across ’25 and ’26\.

Prompt variants\.We compare four prompt formats\.*Freeform Boxed*requests a boxed Accept/Reject answer with no structure\.*Structured JSON*requests a full review schema \(summary, strengths, weaknesses, score, decision\)\.*Fewshot 1A\-1R*prepends one accept and one reject calibration example\.*Fewshot 0A\-2R*uses two reject examples\. Each format can be paired with a*Critical*modifier \(an explicit instruction to be wary of overstated claims and to prefer rejection when uncertain\)\. Multi\-turn variants include*Majority voting*over 5 generations,*Meta\-Review*\(the Sequential Refinement of[Madaan et al\. \(2025\)](https://arxiv.org/html/2610.31791#bib.bib29)\), and*PDR Boxed*\(a single\-generation Parallel\-Distill\-Reflect prompt that runs an Advocate, Critic, and Calibrator inside one rollout\)\.

Critical variant\.Appending a single “be critical, prefer rejection when uncertain” instruction is not enough to stop the model from predicting acceptance for nearly every paper\. On Qwen2\.5\-7B text, Freeform Boxed Critical only nudges reject recall from0\.9%0\.9\\%to4\.3%4\.3\\%\. Switching to Structured JSON \(a full review schema with strengths, weaknesses, score, and decision fields\) with the Critical modifier produces a larger swing \(RR=31\.8%R\_\{R\}=31\.8\\%for text,32\.1%32\.1\\%for vision\), but accuracy stays at chance \(∼\\sim50%50\\%\)\. Gemini\-2\.5\-Flash predicts acceptance less often to begin with \(RA=70\.2%R\_\{A\}=70\.2\\%for text under Freeform Boxed Standard\), and adding Critical overshoots, so that it predicts rejection too often \(RR=75\.7%R\_\{R\}=75\.7\\%for text,67\.0%67\.0\\%for vision\) without changing accuracy\. Across both model families, prompting can rebalance accept/reject recall but cannot lift accuracy above the majority baseline\.

Multi\-step peer\-review emulation\.Multi\-turn variants that simulate a reviewer panel, including 5\-generation Majority voting, Sequential Refinement Meta\-Review\([Madaan et al\., 2025](https://arxiv.org/html/2610.31791#bib.bib29)\), and the single\-generation Parallel\-Distill\-Reflect \(PDR\) prompt with Advocate, Critic, and Calibrator roles, move recall but not accuracy\. Majority voting on Freeform Critical preserves Qwen’s single\-turn tendency to accept nearly everything \(RA=97\.3R\_\{A\}=97\.3,RR=4\.9R\_\{R\}=4\.9\)\. Few\-shot 1A\-1R Standard with Majority voting partially rebalances \(RA=47\.5R\_\{A\}=47\.5,RR=59\.1R\_\{R\}=59\.1\), and PDR Boxed reachesRA=65\.1R\_\{A\}=65\.1,RR=35\.2R\_\{R\}=35\.2\. None of these multi\-turn prompts breaks the∼\\sim50%50\\%ceiling onICLR, consistent with our main\-text observation that base\-model prompting alone does not recover the decision signal that SFT later captures from the same backbones\.

PromptModifierAccRAR\_\{A\}RRR\_\{R\}Single TurnFreeform BoxedStandard50\.299\.40\.9Freeform BoxedCritical51\.297\.94\.3Structured JSONStandard49\.998\.31\.4Structured JSONCritical49\.567\.331\.8Fewshot 1A\-1R BoxedStandard50\.837\.563\.9Fewshot 1A\-1R BoxedCritical51\.636\.067\.2Fewshot 1A\-1R JSONStandard50\.190\.29\.9Fewshot 1A\-1R JSONCritical51\.668\.334\.9Fewshot 0A\-2R BoxedStandard50\.32\.897\.9Fewshot 0A\-2R BoxedCritical50\.21\.998\.4Fewshot 0A\-2R JSONStandard48\.629\.967\.3Fewshot 0A\-2R JSONCritical49\.222\.875\.8Multi\-turnFreeform Crit\.Majority51\.197\.34\.9Freeform Crit\.Meta\-Review50\.596\.04\.9Fewshot 1A\-1R Stand\.Majority53\.347\.559\.1Fewshot 1A\-1R Stand\.Meta\-Review51\.546\.256\.9Fewshot 1A\-1R Crit\.Majority50\.731\.270\.0Fewshot 1A\-1R Crit\.Meta\-Review50\.032\.967\.2PDR Boxed—50\.165\.135\.2\(a\)Qwen2\.5\-7B text prompt ablations\.
PromptModifierAccRAR\_\{A\}RRR\_\{R\}Single TurnFreeform BoxedStandard50\.697\.24\.1Freeform BoxedCritical52\.546\.658\.5Structured JSONStandard49\.999\.20\.6Structured JSONCritical52\.372\.632\.1Fewshot 1A\-1R BoxedStandard50\.328\.272\.3Fewshot 1A\-1R BoxedCritical50\.00\.399\.8Fewshot 1A\-1R JSONStandard50\.592\.78\.4Fewshot 1A\-1R JSONCritical51\.187\.015\.3Fewshot 0A\-2R BoxedStandard50\.02\.097\.9Fewshot 0A\-2R BoxedCritical50\.00\.0100\.0Fewshot 0A\-2R JSONStandard52\.254\.450\.1Fewshot 0A\-2R JSONCritical51\.010\.991\.0Multi\-turnFreeform Crit\.Majority50\.241\.059\.4Freeform Crit\.Meta\-Review49\.628\.071\.1Fewshot 1A\-1R Stand\.Majority50\.924\.877\.0Fewshot 1A\-1R Stand\.Meta\-Review51\.419\.383\.4Fewshot 1A\-1R Crit\.Majority49\.70\.1100\.0Fewshot 1A\-1R Crit\.Meta\-Review49\.70\.1100\.0PDR Boxed—51\.580\.922\.0\(b\)Qwen2\.5\-7B vision prompt ablations\.
TextVisionPromptMod\.AccRAR\_\{A\}RRR\_\{R\}AccRAR\_\{A\}RRR\_\{R\}Freeform BoxedStand\.51\.870\.233\.351\.579\.024\.1Freeform BoxedCrit\.50\.926\.175\.748\.429\.967\.0Struct\. JSONStand\.49\.992\.87\.050\.692\.39\.0Struct\. JSONCrit\.51\.663\.539\.854\.058\.250\.0FS 1A\-1R BoxedStand\.50\.184\.915\.549\.079\.618\.4FS 1A\-1R BoxedCrit\.47\.662\.233\.051\.055\.946\.1FS 0A\-2R BoxedStand\.50\.875\.725\.948\.878\.119\.4FS 0A\-2R BoxedCrit\.48\.555\.541\.550\.550\.850\.0\(c\)Gemini\-2\.5\-Flash prompt ablations \(text and vision\)\.
ModelMod\.’25’26AccRAR\_\{A\}RRR\_\{R\}AccRAR\_\{A\}RRR\_\{R\}Qwen3\.5\-9btext52\.292\.711\.852\.596\.09\.1Qwen3\.5\-9bvis\.50\.098\.51\.551\.0100\.02\.0Qwen3\.5\-35b\-a3btext55\.988\.223\.550\.584\.916\.2Qwen3\.5\-35b\-a3bvis\.55\.998\.513\.250\.597\.04\.0Qwen3\.5\-27btext47\.894\.11\.553\.0100\.06\.1Qwen3\.5\-27bvis\.54\.485\.323\.557\.188\.925\.3Qwen3\.5\-122b\-a10btext53\.783\.823\.552\.096\.08\.1Qwen3\.5\-122b\-a10bvis\.63\.291\.235\.356\.689\.923\.2\(d\)Qwen3\.5 prompt\-time model scaling on a500500\-paperICLRsubset under Freeform Boxed Standard\.

Table H\.3:Prompt\-ablation averages across ICLR ’25 and ’26\.We report accuracy \(Acc\), accept recall \(RAR\_\{A\}\), and reject recall \(RRR\_\{R\}\) for each prompt format and modifier onICLR\. Across all open\-source, Gemini, and Qwen3\.5 variants, accuracy hovers around chance\.Model size ablation\.To check whether, with a stronger base model, scale helps prompting \(Qwen3\.5 is generally stronger than the Qwen2\.5 family we have been using\), we evaluated the Qwen3\.5 family \(99B,2727B,3535B\-MoE,122122B\-MoE\) under Freeform Boxed Standard on a500500\-paperICLRsubset \([Table3\(d\)](https://arxiv.org/html/2610.31791#A8.T3.st4)\)\. Scale alone does not consistently lift base prompting above chance\. Even at122122B vision, the strongest cell only reaches63\.2%63\.2\\%/56\.6%56\.6\\%Acc on ’25/’26, well below the SFT and RL models reported in the main text\.

## Appendix ISFT and RL Method Details

### I\.1SFT

SFT uses cosine decay with 5 warmup steps, context length 24,480 tokens, and greedy evaluation\. The strongest text configuration uses batch size 32 with a learning rate of10−610^\{\-6\}\. The strongest vision configuration uses batch size 16 with the same learning rate\. We sweepICLRandarXivtraining sources, 50/50 versus natural\-rate class priors, data scale, text versus screenshot inputs, text\+vision variants, and suffixes containing structured paper statistics or Qwen/Gemini\-generated reviews\. For vision efficiency, we also test the panel setup in Table[4](https://arxiv.org/html/2610.31791#S4.T4), which tiles ten pages into one 2380×\\times1512 image\. Unless a row is explicitly marked otherwise, 3B SFT runs train for 4 epochs and 7B/14B SFT runs train for 2 epochs\.

Figure I\.1:RL training curves: strict accuracy and per\-class recall\.*Left:*strict\-accuracy training estimate \(EMA\-smoothed; raw curve in faint background\) and held\-out eval accuracy on 1,667 ’25/’26 papers\. The training estimate is inverted from the graded reward viaacc≈\(r−0\.2⋅b\)/\(1−0\.2⋅b\)\\mathrm\{acc\}\\approx\(r\-0\.2\\cdot b\)/\(1\-0\.2\\cdot b\)where the boxed\-rateb≈1\.0b\\approx 1\.0on every eval\.*Right:*per\-class accept \(RAR\_\{A\}, circles, solid\) and reject \(RRR\_\{R\}, squares, dashed\) recall on the same eval set\. Both policies show graded reward improving during training, but their pre\-existing accept bias is preserved and tightens further as RL continues\.
### I\.2RL

We train two single\-turn GRPO policies following DeepSeek\-R1\([Guo et al\., 2025](https://arxiv.org/html/2610.31791#bib.bib16)\), one text and one vision\-language, onICLR\. Both apply the reasoning prompt variant from[AppendixH\.1](https://arxiv.org/html/2610.31791#A8.SS1)on the base Qwen2\.5\-7B \(text\) and Qwen2\.5\-VL\-7B\-Instruct \(vision\) models, and use the composite accuracy\-plus\-format reward described in the main text\.[FigureI\.1](https://arxiv.org/html/2610.31791#A9.F1)shows training trajectories through step100100on a 1,667\-paper held\-out ’25/’26ICLRtest set\. Both policies show graded reward climbing during training, but the per\-class recall panel exposes that the underlying accept/reject mix barely shifts\. As the models learn to satisfy the format reward, their pre\-existing accept bias is preserved and even tightens further during RL\. Neither policy surpasses the strongest SFT model from the main text\. The graded reward partially hides the swing because every well\-formed prediction already earns a0\.20\.2format floor before the decision term contributes anything\.

Settings\.The text RL policy uses a learning rate of1\.2×10−61\.2\\times 10^\{\-6\}with constant warmup over 4 steps, max gradient norm0\.50\.5, train batch size256256, policy mini\-batch size128128, micro\-batch size22per GPU, one epoch,88rollouts per prompt, max response length40964096, and max prompt length25,48025\{,\}480\. The vision RL policy uses Qwen2\.5\-VL\-7B\-Instruct with a learning rate of10−610^\{\-6\}, train batch size6464, PPO mini\-batch size3232, micro\-batch size22per GPU, one epoch,88rollouts per prompt, max response length20462046, and max prompt length24,48024\{,\}480\. Each paper is represented by roughly1010PNG screenshots plus text\.

### I\.3Calibration Details

Modern neural networks are often poorly calibrated, producing confidence estimates that do not reflect true correctness probabilities\([Guo et al\., 2017](https://arxiv.org/html/2610.31791#bib.bib15)\)\. We apply post\-hoc temperature scaling to our SFT models and to Jev, with a base\-rate term so the probabilities hold at a venue’s real acceptance rate\.

Expected calibration error\.Following[Guo et al\. \(2017\)](https://arxiv.org/html/2610.31791#bib.bib15), we first convert each prediction into confidence in the chosen decision:ci=pic\_\{i\}=p\_\{i\}whenz^i=1\\hat\{z\}\_\{i\}\{=\}1andci=1−pic\_\{i\}=1\-p\_\{i\}whenz^i=0\\hat\{z\}\_\{i\}\{=\}0\. We partition these confidences intoMMequal\-width binsB1,…,BMB\_\{1\},\\ldots,B\_\{M\}and define the*expected calibration error*\(ECE\) as

ECE=∑m=1M\|Bm\|n​\|acc⁡\(Bm\)−conf⁡\(Bm\)\|,\\mathrm\{ECE\}=\\sum\_\{m=1\}^\{M\}\\frac\{\|B\_\{m\}\|\}\{n\}\\left\|\\mathrm\{acc\}\(B\_\{m\}\)\-\\mathrm\{conf\}\(B\_\{m\}\)\\right\|,\(1\)whereacc\(Bm\)=\|Bm\|−1∑i∈Bm𝟏\[z^i=zi\]\\mathrm\{acc\}\(B\_\{m\}\)=\|B\_\{m\}\|^\{\-1\}\\sum\_\{i\\in B\_\{m\}\}\\mathbf\{1\}\[\\hat\{z\}\_\{i\}=z\_\{i\}\]is the empirical accuracy andconf⁡\(Bm\)=\|Bm\|−1​∑i∈Bmci\\mathrm\{conf\}\(B\_\{m\}\)=\|B\_\{m\}\|^\{\-1\}\\sum\_\{i\\in B\_\{m\}\}c\_\{i\}is the mean confidence in that bin\. ECE is therefore the average discrepancy between how confident the model is and how often it is correct; a perfectly calibrated model hasECE=0\\mathrm\{ECE\}=0\.

Raw calibration\.Before scaling, the text models are close to calibrated, with ECE0\.0270\.027onICLRand0\.0430\.043onarXiv\(balanced test\)\. The vision models are overconfident, with ECE0\.0490\.049and0\.0690\.069: their confidence exceeds their accuracy\.

Temperature scaling with a base\-rate term\.Temperature scaling uses a single parameterτ\>0\\tau\>0\. From the decision\-token log\-probabilities we take the accept log\-oddsℓ=log⁡P⁡\(Accept\)−log⁡P⁡\(Reject\)\\ell=\\log P\(\\text\{Accept\}\)\-\\log P\(\\text\{Reject\}\)\. Our models are trained on a 50/50 mix, soℓ\\ellmeasures the evidence under a balanced prior\. To score papers at acceptance rateπ\\piwe add the prior log\-odds and rescale:

pcal=σ⁡\(ℓ/τ\+log⁡\(π/\(1−π\)\)\),p\_\{\\mathrm\{cal\}\}=\\sigma\\big\(\\ell/\\tau\+\\log\(\\pi/\(1\-\\pi\)\)\\big\),\(2\)whereσ\\sigmais the sigmoid function\.τ\>1\\tau\>1softens overconfident predictions andτ<1\\tau<1sharpens them\. We fitτ\\tauby minimizing the negative log\-likelihood \(NLL\) on the natural\-rate validation split, withπ\\piset to that split’s acceptance rate \(31%31\\%forICLR,23%23\\%forarXiv\), so the shift absorbs the base rate andτ\\tauonly sets how sharp the probabilities are\. On a balanced setπ=0\.5\\pi=0\.5and the shift vanishes\. Jev’s scores already sit near the real acceptance rate, so we fit itsτ\\tauwithout the shift and, on balanced sets, subtractlog⁡\(π/\(1−π\)\)\\log\(\\pi/\(1\-\\pi\)\)instead\. Sincepcal≥πp\_\{\\mathrm\{cal\}\}\\geq\\piexactly whenℓ≥0\\ell\\geq 0, neitherτ\\taunor the shift ever changes a decision\.

We do not minimize ECE directly because \(i\) the binning operation is non\-differentiable, \(ii\) ECE is sensitive to the number of bins, and \(iii\) NLL is a strictly proper scoring rule whose global optimum is achieved if and only if predicted probabilities equal true probabilities\. ECE, by contrast, can be zero even with poorly separated and rank\-uninformative predictions\.

Calibration at the real acceptance rate\.On a natural\-rate set we measure the calibration of the acceptance probability itself: we bin papers bypcalp\_\{\\mathrm\{cal\}\}into ten equal\-width bins on\[0,1\]\[0,1\]and take the size\-weighted gap between each bin’s meanpcalp\_\{\\mathrm\{cal\}\}and the fraction of its papers actually accepted\.[TableI\.1](https://arxiv.org/html/2610.31791#A9.T1)and Figure[I\.2](https://arxiv.org/html/2610.31791#A9.F2)evaluate the shift on the natural\-rate test sets\. Without it, the probabilities stay centered for a 50/50 world and overstate acceptance \(ECE\.13\.13–\.15\.15\)\. With it, ECE falls to\.023\.023–\.042\.042forPaperScorer\. At this rate anICLRAccept call byPaperScorer\-Vision is right47%47\\%of the time and a Reject call82%82\\%\(56%56\\%and91%91\\%onarXiv\)\. The shifted probabilities slightly understate acceptance: their mean is29\.3%29\.3\\%where31\.3%31\.3\\%ofICLRpapers are accepted, and21\.8%21\.8\\%where24\.8%24\.8\\%ofarXivpapers are\.

Robustness\.Fittingτ\\tauon a balanced validation split instead, and adding the same shift, gives similar results \(ECE\.030\.030–\.049\.049\)\. A two\-parameter Platt fit\([Platt, 1999](https://arxiv.org/html/2610.31791#bib.bib35)\),σ⁡\(a​ℓ\+b\)\\sigma\(a\\ell\+b\)on the natural\-rate split, is slightly better still \(0\.0170\.017–0\.0310\.031\) and also calibrates Jev \(0\.0180\.018–0\.0210\.021\)\. Jev’s calibrated probabilities, however, never exceed about60%60\\%, so unlikePaperScorerit cannot single out likely accepts\.

Caveats\.ThearXivmodel’sπ\\piis a mix of venue rates \(about23%23\\%\); a single venue’s rate can be plugged in asπ\\pi\.arXivrejects are inferred, never matched to a proceedings entry, rather than confirmed\.

τ\\tauBalanced test ECENatural\-rate test ECE ofpcalp\_\{\\mathrm\{cal\}\}SourceModelOursBal\. valRawOursNo shiftOursBal\.\-valτ\\tauPlattICLRPaperScorer\-Text1\.141\.21\.027\.023\.128\.038\.038\.029ICLRPaperScorer\-Vision1\.251\.59\.049\.031\.127\.023\.030\.022ICLRJev0\.742\.99\.166\.051\.060\.060\.058\.021arXivPaperScorer\-Text1\.461\.64\.043\.029\.148\.042\.049\.031arXivPaperScorer\-Vision1\.501\.59\.069\.024\.132\.033\.034\.017arXivJev0\.663\.15\.165\.024\.025\.025\.055\.018Table I\.1:Temperature scaling with a base\-rate term\.τ\\tauis fit on natural\-rate validation with the prior term \(ours; Jev without it, since its scores already sit at the real rate\) or on balanced validation \(for comparison\)\. Balanced test: ECE of confidence before and after scaling\. Natural\-rate test: ECE of the acceptance probability without the shift, with it \(ours\), with the shift butτ\\taufit on balanced validation, and with a two\-parameter Platt fit\.Figure I\.2:Base\-rate shift at deployment\.Share of papers actually accepted vs\. predicted chance of acceptance forPaperScorer\-Vision on the natural\-rateICLR\(left\) andarXiv\(right\) test sets, before \(dashed\) and after \(solid\) addinglog⁡\(π/\(1−π\)\)\\log\(\\pi/\(1\-\\pi\)\)\. Ticks mark the decision rulepcal≥πp\_\{\\mathrm\{cal\}\}\\geq\\pi, the same decisions asℓ≥0\\ell\\geq 0\.
### I\.4Hardware Details

All SFT and RL experiments reported in the paper run on a single node with4×4\\timesNVIDIA H200 GPUs\. Per\-eval inference for the held\-out checkpoints uses the same node\.

## Appendix JStatistical Uncertainty

For uncertainty estimates, we use paired bootstrap resampling so each resample compares models on the same papers\. We resample paper IDs 10,000 times with replacement and recompute Acc and AUC on each resample\.[TableJ\.1](https://arxiv.org/html/2610.31791#A10.T1)reports percentile 95% two\-sided CIs for the four headlinePaperScorer\-Vision comparisons\. OnICLR, the Acc gap against Gemini\-3\.1\-Pro is\+3\.8%\+3\.8\\%with CI\[\+0\.4,\+7\.3\]\[\+0\.4,\+7\.3\], and the AUC gap against GPT\-5\.5 is\+\.066\+\.066with CI\[\+\.034,\+\.098\]\[\+\.034,\+\.098\]\. OnarXiv, the Acc gap against Gemini\-3\.1\-Pro is\+8\.7%\+8\.7\\%\(CI\[\+4\.5,\+12\.7\]\[\+4\.5,\+12\.7\]\) and the AUC gap against GPT\-5\.5 is\+\.097\+\.097\(CI\[\+\.061,\+\.135\]\[\+\.061,\+\.135\]\)\. Since our claim is directional, we test it one\-sided; all four are significant \(p≤\.014p\\leq\.014\)\. The same holds for every large frontier configuration: against Gemini\-3\.1\-Pro and GPT\-5\.5 with either text or vision input, with decisions thresholded as in Table[3](https://arxiv.org/html/2610.31791#S4.T3), the one\-sidedppis below\.05\.05for both Acc and AUC on both datasets \(largestp=\.040p=\.040, Acc against GPT\-5\.5 with text input onICLR\)\.

DatasetMetricPaperScorer\-VisionLarge frontier baselineBaseline scoreΔ\\Delta95% CIppICLRAcc68\.9Gemini\-3\.1\-Pro65\.1\+3\.8\[\+0\.4, \+7\.3\]\.014ICLRAUC\.759GPT\-5\.5\.693\+\.066\[\+\.034, \+\.098\]<<\.001arXivAcc77\.1Gemini\-3\.1\-Pro68\.5\+8\.7\[\+4\.5, \+12\.7\]<<\.001arXivAUC\.841GPT\-5\.5\.743\+\.097\[\+\.061, \+\.135\]<<\.001Table J\.1:Paired bootstrap significance for headline vision comparisons\.Intervals are 95% percentile intervals from 10,000 paired bootstrap resamples over paper IDs;ppis one\-sided \(PaperScorer\-Vision better\)\.[TableJ\.2](https://arxiv.org/html/2610.31791#A10.T2)compares vision with text on each dataset using the same bootstrap framework\. Vision beats text in five of the six comparisons\. We test this formally only forPaperScorer: its vision and text models are scored on the same papers, and vision is significantly better on both datasets \(p=\.012p=\.012and\.005\.005, one\-sided, corrected for the two tests with Holm–Bonferroni\)\. For the frontier models we report the gaps for reference only\. Once their decisions are thresholded as in Table[3](https://arxiv.org/html/2610.31791#S4.T3), the gaps range from−1\.0\-1\.0to\+2\.3\+2\.3, and none is statistically significant\.

DatasetSystemVision AccText AccΔ\\DeltaAcc95% CIppICLRGemini\-3\.1\-Pro65\.162\.8\+2\.3\[\-1\.94, \+6\.67\]\.164†ICLRGPT\-5\.563\.464\.4\-1\.0\[\-5\.29, \+3\.32\]\.657†ICLRPaperScorer68\.366\.0\+2\.3\[\+0\.34, \+4\.16\]\.012arXivGemini\-3\.1\-Pro68\.567\.7\+0\.8\[\-3\.78, \+5\.38\]\.359†arXivGPT\-5\.568\.366\.0\+2\.3\[\-2\.22, \+6\.84\]\.168†arXivPaperScorer77\.574\.2\+3\.3\[\+1\.07, \+5\.52\]\.005Table J\.2:Dataset\-stratified modality comparisons\.Intervals are 95% bootstrap intervals for the vision\-minus\-text Acc difference\. ForPaperScorer,pptests whether vision beats text, paired on the same papers and corrected for the two tests \(Holm–Bonferroni\)\.†Frontier rows are for reference only: uncorrected one\-sidedpp\(stratified randomization\), with decisions thresholded as in Table[3](https://arxiv.org/html/2610.31791#S4.T3)\.Correlative analysis\.Binary accept/reject training yields a continuous score that also tracks reviewer ratings and later citations\. OnICLR,PaperScorer\-Vision has the strongestPaperScorerrating alignment \(ρR=\+\.48\\rho\_\{R\}=\+\.48\), whilePaperScorer\-Text at 14B is strongest withinPaperScorerfor ’25 citation alignment \(ρC=\+\.31\\rho\_\{C\}=\+\.31\)\. OnarXiv, reviewer ratings and citations are available for fewer papers than decisions are, so we treat them as diagnostics and leave them out of the headline comparisons\. OnICLR’25, the one slice with both full reviewer ratings and citations \(n=678n=678\),PaperScorer\-Vision’spacceptp\_\{\\mathrm\{accept\}\}tracks later citations about as well as the human review score \(ρ≈\.25\\rho\\approx\.25vs\.\.26\.26\), and accepted papers outcite rejected ones \(citation AUC\.64\.64\)\. We cannot run this check onarXiv, whose citations come only from proceedings\-matched accepts, leaving too few rejected papers to compare\.

Frontier model repeatability\.We repeated a subset of the frontier\-model inference calls on the same held\-outarXiv\-text papers to estimate run\-to\-run noise\. GPT\-5\.4 is almost deterministic under the same prompt, while Gemini\-3\.1\-Pro has meaningful stochasticity\. Decision agreement is 89\.4% and the second pass is 4\.5% lower on the same papers \(Table[J\.3](https://arxiv.org/html/2610.31791#A10.T3)\)\. We therefore treat GPT\-5\.4 point estimates as stable, while Gemini estimates should be read with several points of implicit run noise unless repeated passes are averaged\. This repeatability audit was run on GPT\-5\.4, the frontier model used before we re\-ran the main GPT baseline on GPT\-5\.5; we did not separately re\-audit GPT\-5\.5’s run\-to\-run noise\.

Metric on repeatedarXiv\-text papersGemini\-3\.1\-ProGPT\-5\.4Paired papers199208Decision agreement89\.4%98\.1%Pass\-1 accuracy64\.8%57\.7%Pass\-2 accuracy60\.3%57\.7%Pass\-1 right, pass\-2 wrong7\.5%1\.0%Pass\-1 wrong, pass\-2 right3\.0%1\.0%Mean absolute rating delta0\.610\.40Rating within±\\pm189\.4%98\.6%Table J\.3:Frontier repeatability audit\.We repeatarXiv\-text frontier inference on the same held\-out papers and measure decision, accuracy, and rating stability across two passes\.
## Appendix KApplications to Scientific Systems

PaperScorerproduces a ranking signal that is empirically proven: its high AUC and rating correlation show that the score ranks accepted papers above rejected ones and agrees with reviewer ratings, and temperature scaling makes that score calibrated after the fact\. This invites two applications where a calibrated estimate of acceptance probability matters\. In literature retrieval, we want to surface work likely to be accepted among topically similar candidates\. In agentic reviewing, a generated critique relies on a well\-calibrated prior over the final decision\. We study both settings below\.

Figure K\.1:LitSearch reranking\.We evaluate Recall@K on 148 hard LitSearch queries whose gold papers are missed by BM25 at K=5, using each retriever’s top\-200 candidates\. Labels show the Recall@K gain from addingPaperScorer\-Vision\.### K\.1Ranking Scientific Literature for Retrieval

Scientific search systems must retrieve a target paper from many plausible topical matches\. LitSearch\([Ajith et al\., 2024](https://arxiv.org/html/2610.31791#bib.bib3)\)captures this setting with natural\-language literature queries paired with one or more gold cited papers\. We evaluate the subset linked to ourarXivpool: 478 queries have at least one goldarXivpaper, and 250 have all goldarXivIDs inside thePaperScorerarXivcorpus\. To isolate the regime where reranking matters, we use the 148 hard queries for which BM25 has zero Recall@5, covering 150 gold papers\.

We usePaperScoreras a query\-independent reranker based on its estimated acceptance probability\. For each hard query, BM25 and Qwen3\-Embedding\-0\.6B\([Zhang et al\., 2025b](https://arxiv.org/html/2610.31791#bib.bib53)\)first retrieve their own top\-200 candidate papers\. We then score each candidate once witharXiv\-trainedPaperScorer\-Vision \(7B checkpoint, matching the main results\); the 3B checkpoint \(not shown\) reranks almost as well, within a few percent at everyKK, consistent with our finding that decision accuracy is largely insensitive to model scale \([Section4\.3](https://arxiv.org/html/2610.31791#S4.SS3)\)\. Lets⁡\(q,d\)s\(q,d\)be the retriever score for candidateddunder queryqq, letz⁡\(⋅\)z\(\\cdot\)denotezz\-normalization within the query’s candidate list, and letpaccept​\(d\)p\_\{\\mathrm\{accept\}\}\(d\)be thePaperScoreracceptance score\. We rank candidates by a fixed blend of the two, with the weight onpacceptp\_\{\\mathrm\{accept\}\}chosen on a held\-out validation split of queries:

0\.3​z​\(s⁡\(q,d\)\)\+0\.7​paccept​\(d\)\.0\.3\\,z\(s\(q,d\)\)\+0\.7\\,p\_\{\\mathrm\{accept\}\}\(d\)\.
Retrieval supplies the topical neighborhood, whilePaperScorer\-Vision reorders candidates by their estimated acceptance probability\.[FigureK\.1](https://arxiv.org/html/2610.31791#A11.F1)shows that this helps both retrievers: whether candidates come from sparse BM25 or a dense Qwen embedding, reranking pulls more of the gold cited papers toward the top of the list\. The gain is largest near the top, where retrieval has already gathered topically similar work and the only open question is which paper is more likely to be accepted\. Deeper down, where topical relevance still does most of the sorting, the prior adds little, soPaperScorer\-Vision acts as a prior on acceptance layered on retrieval, not a substitute for it\.

### K\.2Grounding Agentic Reviewers

Prior work shows that LLM reviews can yield actionable suggestions for improving papers\([Thakkar et al\., 2026](https://arxiv.org/html/2610.31791#bib.bib40)\), but our prompt\-tuning results \([Section4\.3](https://arxiv.org/html/2610.31791#S4.SS3)\) suggest that the predicted decisions poorly track ground\-truth outcomes\. As noted in[Section2](https://arxiv.org/html/2610.31791#S2), papersXXare transformed into reviewsRRbefore reaching a decisionZZ, so the joint distribution over reviews and decisions factors asp⁡\(R,Z∣X\)p\(R,Z\\mid X\)\. While the autoregressive flow places the review before the decision, now that we have built a shortcut that predicts the decision without the review, we can ask whether we can generate more*decision\-calibrated*reviews by rearranging the joint:

p⁡\(R,Z∣X\)=p⁡\(Z∣R,X\)​p​\(R∣X\)⟹p⁡\(R∣Z,X\)⏞Frontier Model​p⁡\(Z∣X\)⏞PaperScorer\.p\(R,Z\\mid X\)=p\(Z\\mid R,X\)\\,p\(R\\mid X\)\\;\\Longrightarrow\\;\\overbrace\{p\(R\\mid Z,X\)\}^\{\\text\{Frontier Model\}\}\\,\\overbrace\{p\(Z\\mid X\)\}^\{\\text\{\{\\color\[rgb\]\{0\.1836,0\.3281,0\.5898\}\\textbf\{PaperScorer\}\}\{\}\}\}\.Both factorizations are valid\. Conceptually, if our conjecture is correct — that much of the decision is determined by overall presentation quality — then reviews may serve as a means\-to\-an\-end to justify the reviewer’s initial reaction\. This follows from psychology research on human decision\-making, whereby we often rely on initial judgments and post\-hoc rationalization more than we consciously attribute\([Nisbett and Wilson, 1977](https://arxiv.org/html/2610.31791#bib.bib31);[Kunda, 1990](https://arxiv.org/html/2610.31791#bib.bib24);[Haidt, 2001](https://arxiv.org/html/2610.31791#bib.bib17)\), with low\-level feelings of “goodness” or “badness” driving longer\-form judgments and reasoning\([Slovic et al\., 2007](https://arxiv.org/html/2610.31791#bib.bib38)\)\.

We therefore ask whether conditioning frontier coding agents on thePaperScorersignal can \(1\) improve their decision\-prediction and \(2\) improve their generated reviews’ alignment with human reviews\. We usePaperScorer\-Vision as the calibrated prior, and let Claude Code write the review\.

Acc\. \(%\)Alignment \(1–5\)Method’25’26AllCn\.Wk\.Nov\.Ql\.MeanPaperScorer\-Vision72\.068\.069\.6–––––Claude Code57\.566\.763\.04\.964\.083\.984\.114\.28\+PaperScorer\-Vision70\.570\.770\.64\.874\.234\.104\.144\.33Δ\\Delta\+13\.0\+4\.0\+7\.6\-0\.09\+0\.15\+0\.11\+0\.03\+0\.05\(a\)Reviewer accuracy and human\-review alignment\.‘Cn\.’, ‘Wk\.’, ‘Nov\.’, and ‘Ql\.’ denote contribution, weakness, novel\-concern, and overall\-quality alignment\.
’25’26OverallTransition%Acc\.%Acc\.%Acc\.No change75\.068\.780\.773\.178\.471\.4A→\\toR21\.081\.08\.773\.113\.677\.9R→\\toA4\.050\.010\.750\.08\.050\.0All flips25\.076\.019\.360\.321\.667\.6\(b\)Decision transitions withPaperScorer\-Vision\.How agent decisions shift under the prior\.

Table K\.1:Decision\-conditioned review generation\.Claude Code is evaluated on 500ICLRpapers under a 30/70 accept/reject split, with and without the calibratedPaperScorer\-Vision prior\. Conditioning improves accuracy from 63\.0% to 70\.6% and review alignment, with the prior catching Claude’s remaining incorrect acceptance predictions\.We evaluate Claude Code \(Sonnet\-4\.6\) on 500ICLRpapers\. The agent receivesPaperScorer\-Vision’s decision,pacceptp\_\{\\mathrm\{accept\}\}, and confidence bands, and can inspect text, rendered pages, and the temporally filteredarXivretrieval\. It emits a schema\-constrained review and decision\. Codex \(GPT\-5\.5, medium reasoning\) grades reviews against human reviews \(review generation and alignment grading prompts in[AppendixK\.3](https://arxiv.org/html/2610.31791#A11.SS3)\)\.

[Table1\(a\)](https://arxiv.org/html/2610.31791#A11.T1.st1)shows that Claude alone*underperforms*PaperScorer\-Vision, but adding the calibrated prior improves overall accuracy from 63\.0% to 70\.6%, surpassing both individually\. This matches the factorization above\.PaperScorer\-Vision anchors the decision while the reviewer supplies the critique\. The gain comes mainly from a 13\.6% subset of papers flipping Accept→\\toReject at 77\.9% accuracy \([Table1\(b\)](https://arxiv.org/html/2610.31791#A11.T1.st2)\)\. Reject→\\toAccept flips are rarer and much less reliable\. Although Claude predicts rejection more often than acceptance on the balanced evaluation \([Table3](https://arxiv.org/html/2610.31791#S4.T3)\), on this 30/70 subset its remaining incorrect acceptance predictions dominate the correction signal, which the calibrated accept probability catches by pulling those papers below the decision threshold the agent then respects\.

Conditioning onPaperScorer\-Vision also improves alignment with human reviews across every factor except contributions \([Table1\(a\)](https://arxiv.org/html/2610.31791#A11.T1.st1)\)\. Because the prior corrects Claude’s accept\-bias, the agent emits weakness and novelty critiques at a higher rate, bringing the generated review closer to the human ground truth\. Contribution alignment was already saturated under Claude alone, and is preserved under conditioning\.

### K\.3Agentic Reviewer Prompts

This subsection records the prompt scaffolding for the discussion experiment\. Claude Code \(Sonnet\-4\.6\) performs the review, while review grading is performed by Codex \(GPT\-5\.5\) with medium reasoning\.

Reviewer agent \(no prior\)\.

You are an ICLR reviewer\. Review the paper with submission\_id=<SID\>\.STEPS:1\. Read the paper content from papers/<SID\>/content\.json2\. Examine the page images listed in the content JSON under img\_pages3\. Write a structured ICLR review as JSON with all 11 fields:\- decision: "accept" or "reject"\- rating: 0\-10\- confidence: 1\-5\- soundness: 1\-4\- presentation: 1\-4\- contribution: 1\-4\- summary: 2\-3 sentence summary\- strengths: numbered list citing specific sections/figures/theorems\- weaknesses: numbered list citing specifics \(no generic criticisms\)\- questions: numbered list of questions for authors\- missing\_references: prior work explicitly mentioned in the paper4\. Write the review JSON to agent\_reviews/<SID\>\.json using the Write tool\.5\. Return ONLY "Done: <SID\>" \-\- do NOT return the full review text\.

Reviewer agent \(withPaperScorer\-Vision prior\)\.

You are an ICLR reviewer\. Review the paper with submission\_id=<SID\>\.PAPERSCORER PRIOR:\- Model prediction: <accept\|reject\>\- Model confidence: <0\.xxx\>STEPS:1\. Read the paper content from papers/<SID\>/content\.json2\. Examine the page images listed in the content JSON under img\_pages3\. Use the model prediction as a Bayesian prior \-\- weight it by theconfidence \(high confidence \-\> trust strongly, low \-\> use own judgment\)\.4\. Write a structured ICLR review as JSON with all 11 fields:\- decision: "accept" or "reject"\- rating: 0\-10\- confidence: 1\-5\- soundness: 1\-4\- presentation: 1\-4\- contribution: 1\-4\- summary: 2\-3 sentence summary\- strengths: numbered list citing specific sections/figures/theorems\- weaknesses: numbered list citing specifics \(no generic criticisms\)\- questions: numbered list of questions for authors\- missing\_references: prior work explicitly mentioned in the paper5\. Write the review JSON to agent\_reviews/<SID\>\.json using the Write tool\.6\. Return ONLY "Done: <SID\>" \-\- do NOT return the full review text\.

Judge prompt \(Codex GPT\-5\.5, medium reasoning\)\.

You are an expert meta\-reviewer evaluating AI\-generated paper reviews againstreal ICLR human reviews and the Area Chair \(AC\) metareview\.For each paper you will see:\- "human\_reviews": individual ICLR reviewer assessments\- "metareview": the AC’s decision comment summarizing why the paper wasaccepted/rejected\- "agent\_review": the AI\-generated review to evaluateEVALUATION PROTOCOL \-\- for EACH dimension, you MUST:1\. QUOTE the relevant text from human reviews / metareview \(use "\.\.\." forexcerpts\)2\. QUOTE the relevant text from the agent review3\. Explain your reasoning comparing the two4\. Assign a score 1\-5DIMENSIONS:1\. contribution\_identification \(1\-5\):What is the primary contribution according to human reviewers?Did the agent identify the same core contribution?\- 5: Agent identifies the exact same core contribution with similar framing\- 4: Same contribution but framed differently or missing a key aspect\- 3: Related but focuses on a secondary contribution instead of the primaryone\- 2: Vaguely related to the actual contribution\- 1: Completely different contribution identified, or contribution notdiscussed2\. top\_weakness\_identification \(1\-5\):What is the \#1 weakness from human reviews? \(The one most reviewers agreeon, or that the AC highlights in the metareview\.\)Did the agent identify this specific weakness?\- 5: Agent identifies the same top weakness with comparable specificity anddetail\- 4: Identifies the weakness but less precisely, or conflates it with otherissues\- 3: Mentions the general area of concern but misses the specific technicalpoint\- 2: Identifies only minor or tangential weaknesses, misses the critical oneentirely\- 1: No meaningful overlap with human\-identified weaknesses3\. novel\_concerns \(1\-5\):List any concerns the agent raises that are NOT present in human reviews ormetareview\.Are these valid and substantive?\- 5: Raises 2\+ valid, substantive technical concerns that humans missed\- 4: Raises 1 valid substantive concern that humans missed\- 3: Raises concerns that are valid but minor, obvious, or not very useful\- 2: Novel concerns are mostly invalid, confused, or based onmisunderstanding the paper\- 1: No novel concerns raised, or all novel points are factually wrong4\. overall\_review\_quality \(1\-5\):Holistic quality considering: summary accuracy, weakness depth, strengthidentification, constructiveness, specificity\.\- 5: Comparable to a good human ICLR reviewer \-\- specific, evidence\-based,actionable feedback\- 4: Good review with paper\-specific points but missing some depth or nuance\- 3: Adequate but generic \-\- observations could apply to many papers in thearea\- 2: Superficial, repetitive, or contains factual errors about the paper’scontent\- 1: Useless \-\- generic template text, wrong paper content, or incoherent

## Appendix LWhat isPaperScorerLooking At?

### L\.1Section\-Attention Map

Figure L\.1:Section edits\.\(a\) Later sections that receive more attention also show a larger maximum observed change inpacceptp\_\{\\mathrm\{accept\}\}across four Git commits\. \(b\) The most effective edit type depends on the section: narrative helps Experiments and Discussion, formalization helps Methodology, and nuance helps the Introduction\. Details in[AppendixL\.2](https://arxiv.org/html/2610.31791#A12.SS2)\.SectionAttentionDiscussion37%Experiments28%Methodology15%Introduction9%Background7%Table L\.1:Section attention\.We attribute the decision ofPaperScorerto specific input features\. Using metadata from MinerU, we divide each paper into 5 sections: Introduction, Background, Methodology, Experiments \(including Results\), and Discussion \(including Conclusion\)\. Using the final decision token’s last\-layer attention as a descriptive map \(described in[AppendixL\.2](https://arxiv.org/html/2610.31791#A12.SS2)\), we observe a clear hierarchical preference as seen in[Table L\.1](https://arxiv.org/html/2610.31791#A12.T1)\. Generally, attention increases later in the paper\. However, the introduction is weighed more heavily than the background\. We visualize examples in[Figure L\.2](https://arxiv.org/html/2610.31791#A12.F2)\.

Edit causality\.To verify that this pattern represents the importance of each section, we perform a causal experiment\. We use Codex \(gpt\-5\.5, medium reasoning\) to write independent, section\-level edits to the markdown paper copy of 30 accepted ’26 papers with low presentation scores thatPaperScorer\-T scores as borderline \(accept probability between 0\.45 and 0\.50\)\. Each Codex instance edits one section at a time, allowing up to four sets of edits to be saved as Git commits, with feedback fromPaperScorer\-T received after each commit\. These changes are constrained to presentation, framing, and style\. We analyze the∼\\sim600 commits made\. Details in[AppendixL\.2](https://arxiv.org/html/2610.31791#A12.SS2)\.

Our causal analysis matches the attention pattern: later sections, especially Experiments and Discussion, show the largest maximum observed changes inpacceptp\_\{\\mathrm\{accept\}\}in Figure[L\.1](https://arxiv.org/html/2610.31791#A12.F1)a\. We find that the most effective edit type is section\-dependent \(Figure[L\.1](https://arxiv.org/html/2610.31791#A12.F1)b\)\. Formalization helps most in methodology, narrative improvements help most in the discussion and experiments, whereas substantiation in the conclusion typically hurts acceptance probability\. Notably, these edits do not continuously increasepacceptp\_\{\\mathrm\{accept\}\}, suggesting thatPaperScorer\-Text will only allow style edits to increasepacceptp\_\{\\mathrm\{accept\}\}to a certain threshold outside the 0\.7 high\-confidence region\.

### L\.2Interpretability Details

Attention score definition\.Following[Li et al\. \(2026\)](https://arxiv.org/html/2610.31791#bib.bib25), we use the last\-layer attention of the final decision token to summarize which paper tokens the model consults most strongly\. LetS=\(x1,…,xT\)S=\(x\_\{1\},\\dots,x\_\{T\}\)denote the full context, where the paper occupies tokensxP\+1,…,xTx\_\{P\+1\},\\dots,x\_\{T\}\. Ifτ\\tauis the generated accept/reject token andατ,i\\alpha\_\{\\tau,i\}is the average attention weight fromτ\\tauto tokeniiacross final\-layer heads, then the normalized paper\-token attention score is

aτ,i=ατ,i∑j∈𝒥ατ,j,a\_\{\\tau,i\}=\\frac\{\\alpha\_\{\\tau,i\}\}\{\\sum\_\{j\\in\\mathcal\{J\}\}\\alpha\_\{\\tau,j\}\},where𝒥\\mathcal\{J\}indexes the paper tokens only\. This gives a probability distribution over the paper conditioned on the final decision token\. We aggregate these scores by section using text bbox metadata from MinerU, then summarize the resulting distribution by section medians in the main\-text table\.

![Refer to caption](https://arxiv.org/html/2610.31791v1/attention_grid_raw_seed42.png)Figure L\.2:Full\-paper attention heatmaps\.Example attention overlay for the SFT vision model\. Regions highlighted in lighter colors receive more of the final decision token’s last\-layer attention mass\. The model concentrates most strongly on technical and evaluative material in the latter half of the paper rather than on the opening framing alone\.Perturbation reasoning\.The attention map is descriptive rather than causal\. A section may receive high last\-layer attention because it carries decisive evidence, but it could also inherit importance from earlier\-layer computation or from interactions across sections\. We therefore complement attention with direct perturbations: if editing one section consistently changesPaperScorer\-Text’s predicted acceptance probability while all other sections remain fixed, that section is more plausibly part of the model’s effective decision boundary\.

Perturbation setup\.We select 30 borderline papers whose baseline accept probabilities lie in the0\.450\.45–0\.500\.50range and spawn 5 section\-specific agents per paper, one for each section family: Introduction, Background / Related Work, Methodology, Experiments / Results, and Discussion / Conclusion\. The editing agent is Codex ongpt\-5\.5with medium reasoning and no web search\. Each agent is restricted to one section and is not allowed to change any numerical results, but it may improve framing, exposition, ordering, and technical detail within that section\. Each commit the Codex agents submit is scored by our fine\-tunedPaperScorer\-Text, evaluated greedily at temperature00with logprobs\.

CategoryDescriptionSubstantiationIntroducing concrete evidence from tables and figures into the paper text\.FormalizationFormalizing notation and stating implicit equations, explicit definitions, or proof structures\.NarrativeRestructuring sections to reframe motivation or alter content flow\.CalibrationEnsure that sentiment matches the scope of results by adjusting overclaims, tempering language, and acknowledging weaknesses\.NuanceClarifying ambiguous phrasing, defining terms, or improving precision of descriptions\.Table L\.2:Paper revision strategy taxonomy\.We define the edit categories used to label section\-level perturbations\.
### L\.3Edit Analysis

Section attention trends\.As noted in the main text, the section\-level attention averages show a clear ordering that prioritizes later sections\. The example heatmap in[Figure L\.2](https://arxiv.org/html/2610.31791#A12.F2)illustrates this qualitative pattern, with attention clustering tightly around technical claims, result tables, and synthesis paragraphs, rather than the opening overview\. While previous literature suggests that attention analysis alone may not definitively determine the importance of input tokens, our causal edit analysis demonstrates that the increase in acceptance likelihood follows this exact same ordering\.

Perturbation protocol\.For our section\-level edit interventions, we select 30 accepted papers from ICLR ’26 that specifically have low presentation scores\. This ensures the underlying content is of high quality, isolating the impact of stylistic and structural improvements onpacceptp\_\{\\mathrm\{accept\}\}\.

We use Codex withgpt\-5\.5under a medium reasoning setting to generate edits\([Chen et al\., 2021](https://arxiv.org/html/2610.31791#bib.bib9)\)\. The editing agent is restricted to a single section per trial, with unlimited read and edit tool calls\. Crucially, the agent is explicitly forbidden from modifying any numerical results, guaranteeing that changes are strictly limited to framing, clarity, and style\.

Once the agent is satisfied, it commits the updated text with a summary statement\. This process is repeated 3 more times using a naive agent given the same prompt, without knowledge of thePaperScoreroutput\. After all commits are generated, we usePaperScorerto grade each iteration\.

Impact of edits by section\.The rankings of the sections byΔ​paccept\\Delta p\_\{\\mathrm\{accept\}\}directly reflect the initial attention distribution, with the majority of the score improvement coming from the first edit commit\. Edits to the Experiments / Results and Discussion / Conclusion yield the largest gains \(5\.4–6\.1%\), followed by Methodology \(4\.7%\), and smaller improvements for the Introduction and Background \(3\.9–4\.2%\)\. Because the maximum difference between final scores across the sections differs by only 2%, we conclude that the attention weighting reflects a consistent, albeit small, effect size regarding which sections the model prioritizes\. We also hypothesize that the effect size is partially due to the authors’ allocation of time\. The introductions of papers are typically written quite well, as they provide the first impression for a reviewer or reader\. Therefore, there may be fewer style changes that may improve the paper\. On the other hand, the results themselves are typically more important than the style of reporting them, so there may be more stylistic things to update in later sections\.

Optimal edit strategies\.We also examine the types of edits Codex makes in each section and which types our model favors\. Using the commit messages, we classify each edit into specific categories \(detailed in Table[L\.2](https://arxiv.org/html/2610.31791#A12.T2)\)\. As illustrated in[Figure L\.1](https://arxiv.org/html/2610.31791#A12.F1)b, the success rate of these edits is highly section\-dependent:

- •Introduction:Benefits most from adding Nuance \(\+11%\) and Calibration \(\+5%\)\. Adjusting the Narrative here decreases the score \(\-13%\)\.
- •Background / Related Work:Improves most from Substantiation \(\+8%\) and Narrative framing \(\+8%\), but is penalized by added Nuance \(\-12%\)\.
- •Methodology:Only Formalization is effective \(\+14%\), whereas all other edit types negatively impact the score\.
- •Experiments and Discussion:Narrative edits yield the highest positive returns in both the Experiments \(\+12%\) and Discussion \(\+24%\) sections\. Crucially, attempting to introduce new Substantiation into the Discussion serves as the most detrimental edit overall \(\-29%\), suggesting that this final section should focus on interpreting results rather than reiterating evidence\.

These edit success distributions reflect general paper guidelines, suggesting that our model has learned the overall structure of a paper\.