AI Reviews Are Biased Towards the Appendix

arXiv cs.CL Papers

Summary

A study of 2,000 papers from ACL, EMNLP, COLM, and ICLR finds that latest GPT and Claude models draw 50%+ of review weaknesses from the appendix, versus only 8.4% for human reviewers, and that this bias is now leaking into human reviews.

arXiv:2611.17811v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used by reviewers to highlight weaknesses during the reviewing process, and their outputs often shape the reviews that humans submit. Prior work studies how LLM reviews align with human ones, yet we know little about which parts of a paper these models actually criticize. In this work, we investigate 2,000 papers from ACL, EMNLP, COLM, and ICLR, and show that the latest GPT and Claude models draw about 50% or more of their review weaknesses from the appendix. This behavior holds across the API, coding-agent, and web access methods for using LLMs to review a paper. In contrast, we find that humans draw only 8.4% of their weaknesses from the appendix, and these weaknesses primarily revolve around a paper's presentation and content related to its core idea. Meanwhile, LLMs draw weaknesses from deep in the appendix that concern fine-grained technical details not central to the paper's contributions. We anticipate that our analysis will contribute to the scholarly discourse on AI reviews. Our code is publicly available at: https://github.com/cincynlp/review_appendix.
Original Article
View Cached Full Text

Cached at: 11/16/26, 08:22 AM

# AI Reviews Are Biased Towards the Appendix
Source: [https://arxiv.org/html/2611.17811](https://arxiv.org/html/2611.17811)
Mohammad SaimAffiliation:University of CincinnatiTianyu JiangAffiliation:University of CincinnatiCorresponding author:ghosh2si@mail\.uc\.edu, saimmd@mail\.uc\.edu, tianyu\.jiang@uc\.edu

###### Abstract

Large language models \(LLMs\) are increasingly used by reviewers to highlight weaknesses during the reviewing process, and their outputs often shape the reviews that humans submit\. Prior work studies how LLM reviews align with human ones, yet we know little about which parts of a paper these models actually criticize\. In this work, we investigate 2,000 papers from ACL, EMNLP, COLM, and ICLR, and show that the latest GPT and Claude models draw about 50% or more of their review weaknesses from the appendix\. This behavior holds across the API, coding\-agent, and web access methods for using LLMs to review a paper\. In contrast, we find that humans draw only 8\.4% of their weaknesses from the appendix, and these weaknesses primarily revolve around a paper’s presentation and content related to its core idea\. Meanwhile, LLMs draw weaknesses from deep in the appendix that concern fine\-grained technical details not central to the paper’s contributions\. We anticipate that our analysis will contribute to the scholarly discourse on AI reviews\. Our code is publicly available at:[https://github\.com/cincynlp/review\_appendix](https://github.com/cincynlp/review_appendix)\.

Figure 1:Newer models such as GPT\-6\-Astra and Claude\-Opus\-5\.5 are biased toward the appendix when writing reviews\. They draw more than 50% of their weaknesses from appendix content\. This bias now seems to be showing up in human reviews\. For years, human reviewers at ICLR cited only about 6\-10% of their weaknesses in the appendix\. This share began to rise after the release of GPT\-5, and nearly tripled to 27\.60% in ICLR 2027 reviews, posted shortly after GPT\-6\-Astra and Claude\-Opus\-5\.5 were released\.## 1Introduction

Asking a large language model \(LLM\) to review a research paper is becoming increasingly common\. This is no longer a niche practice but a visible part of peer review\. In ICLR 2026, about a fifth of all reviews were flagged as fully AI\-generated\([Glickenhaus et al\., 2026](https://arxiv.org/html/2611.17811#bib.bib2)\), and many more were likely written with some level of LLM assistance\. It is easy to see how this use spreads: reviewers upload a draft to popular AI models like ChatGPT or Claude, ask for a conference\-style review, and receive feedback within minutes\. The most important part of this feedback is often the list of weaknesses, which reviewers carry into their own reviews\.

Recently, we noticed a recurring pattern in high\-tier conference review systems\. Many criticisms or weaknesses in these AI reviews came from deep in the appendix\. Details that were included only as supporting evidence or optional experiments made up a large share of the major weaknesses\. This matters beyond AI reviews themselves\. Since reviewers often use LLMs to help find weaknesses, what the LLM points out can easily end up in a human review\. These weaknesses can also shape a human reviewer’s overall impression of the paper\. If an LLM lists multiple weaknesses from the appendix, a reviewer may conclude the paper is weak\. Such weaknesses rarely engage with a paper’s core idea, and they do little to help authors improve it\. While recent work has studied many aspects of AI review, such as alignment with human reviews\([Fichtl et al\., 2026](https://arxiv.org/html/2611.17811#bib.bib3);[Camelo\-Guerrero and Diaz\-Rodriguez, 2026](https://arxiv.org/html/2611.17811#bib.bib4)\)or benchmarking AI reviews\([Li et al\., 2026](https://arxiv.org/html/2611.17811#bib.bib14);[Nguyen et al\., 2026](https://arxiv.org/html/2611.17811#bib.bib15)\), this bias towards the appendix appears new, emerging mainly with the latest, more powerful models such as GPT\-6\-Astra and Claude\-Opus\-5\.5\.

To this end, we investigate how much of an AI review’s criticism comes from the appendix\. To our knowledge, we are the first to study this behavior\. We analyze 2,000 papers: 800 from four recent conferences \(ACL 2026, EMNLP 2026, COLM 2026, and ICLR 2027\) and 1,200 from ICLR 2021 through 2026\. We use four state\-of\-the\-art models that were widely used for AI review at the time of this analysis: GPT\-6\-Astra, GPT\-6\.1\-Sol, Claude\-Opus\-5\.5, and Claude\-Sonnet\-5\.5\. Across the four recent conferences, our results show that almost all models draw more than half of their weaknesses from appendix\-related material\. Compared with earlier models like GPT\-5, where less than 30% of weaknesses relate to the appendix, GPT\-6\-Astra and GPT\-6\.1\-Sol show around 60\-70%, suggesting this behavior is new\. We also study human\-written reviews from ICLR 2021 to 2027\. We find that humans focus mainly on the main body and draw less than 10% of their weaknesses from the appendix\. This share rises sharply to 27\.60% in human reviews written after the release of newer models, as shown in Figure[1](https://arxiv.org/html/2611.17811#S0.F1)\. While human reviewers distinguish between the main body and the appendix and weigh them differently in their overall review, LLMs seem more biased toward the supplementary material and treat the paper as a single unit\. As more people use these models to review papers, we believe the community should take note of this behavior\. In summary, our contributions are:

- •Appendix\-Heavy Criticism\.We show that the latest GPT and Claude models draw more than 50% of their review weaknesses from the appendix\. This behavior holds across models, venues, and access methods, including the API, CLI agent, and web interface\.
- •Comparison with Human Reviews\.In contrast to LLMs, humans drew only 8\.4% of their weaknesses from the appendix in previous years\. However, with the arrival of newer models, appendix\-related weaknesses in recent human reviews have risen to 27\.60%\.
- •Analysis of Appendix Criticism\.Unlike humans, LLMs rarely engage with a paper’s core contribution when criticizing the appendix\. Instead, they focus on small statistical details and ask for ever more experiments or statistical validation on existing results\.

![Refer to caption](https://arxiv.org/html/2611.17811v1/review_method_diagram_2.png)Figure 2:Pipeline of our method: i\) Extract and sample papers from four conferences with a minimum appendix length of 15 pages, ii\) Generate an AI review, and iii\) map each weakness back to the section it belongs to\.
## 2Related Work

Much of the literature on AI review is recent\. Prior work showed that LLMs can provide useful feedback on research papers and studied how reliable they are as reviewers\([Liang et al\., 2024](https://arxiv.org/html/2611.17811#bib.bib9);[Zhou et al\., 2024](https://arxiv.org/html/2611.17811#bib.bib10);[Du et al\., 2024](https://arxiv.org/html/2611.17811#bib.bib11)\)\. Moreover, ample work compares LLM reviews with human reviews and finds that they agree on broad outcomes but differ in what they criticize\([Camelo\-Guerrero and Diaz\-Rodriguez, 2026](https://arxiv.org/html/2611.17811#bib.bib4);[Żurawicki et al\., 2026](https://arxiv.org/html/2611.17811#bib.bib12);[Chen et al\., 2026](https://arxiv.org/html/2611.17811#bib.bib13)\)\. Several benchmarks evaluate LLMs and agentic systems as reviewers\([Li et al\., 2026](https://arxiv.org/html/2611.17811#bib.bib14);[Nguyen et al\., 2026](https://arxiv.org/html/2611.17811#bib.bib15);[Teo et al\., 2026](https://arxiv.org/html/2611.17811#bib.bib16)\), and a recent survey summarizes this growing area\([Nguyen and Ahmadi, 2026](https://arxiv.org/html/2611.17811#bib.bib17)\)\. At the same time, LLM use in real peer review is rising, and studies estimate how many reviews are written or edited with LLMs\([Kumar et al\., 2025](https://arxiv.org/html/2611.17811#bib.bib19);[Liang et al\., 2026](https://arxiv.org/html/2611.17811#bib.bib1);[Shen and Wang, 2026](https://arxiv.org/html/2611.17811#bib.bib18)\)\. Studies have also examined whether reviewer policies on AI use can be enforced\([Saha et al\., 2026](https://arxiv.org/html/2611.17811#bib.bib20);[Fichtl et al\., 2026](https://arxiv.org/html/2611.17811#bib.bib3)\)and measured how LLM use affects review outcomes\([Kim et al\., 2026](https://arxiv.org/html/2611.17811#bib.bib27);[Sharma et al\., 2026](https://arxiv.org/html/2611.17811#bib.bib28)\)\. Others have put forward the notion that AI should support human reviewers rather than replace them\([Baumann et al\., 2026](https://arxiv.org/html/2611.17811#bib.bib21);[Yun et al\., 2026](https://arxiv.org/html/2611.17811#bib.bib22)\)\.

Studies have also highlighted the existing issues of AI review\. LLM reviewers can be manipulated through hidden prompts in manuscripts\([Lin, 2026](https://arxiv.org/html/2611.17811#bib.bib23);[Li et al\., 2026](https://arxiv.org/html/2611.17811#bib.bib14)\)or simply by rewording and restyling a paper\([Kaneko, 2026](https://arxiv.org/html/2611.17811#bib.bib24);[Yang et al\., 2026](https://arxiv.org/html/2611.17811#bib.bib25)\)\.[Vasu et al\. \(2026\)](https://arxiv.org/html/2611.17811#bib.bib26)showed that they favor confident writing over substance\. Together, these studies examine many aspects of AI review, from quality and alignment to bias and manipulation\. However, AI reviewers’ tendency to draw criticism from the appendix is a recent trend that has emerged with the latest models and has not yet been studied\. To our knowledge, this work is the first to examine it\. As the latest models can now read every page of a paper and are increasingly used by humans to assist with reviewing, this question has become urgent\.

## 3Methodology

In this section, we describe how we extract research papers and our methodology\.

Models\.For evaluation, we select four state\-of\-the\-art LLMs widely used for reviewing papers from two model families: GPT\-6\-Astra\([OpenAI, 2026a](https://arxiv.org/html/2611.17811#bib.bib7)\)and GPT\-6\.1\-Sol\([OpenAI, 2026b](https://arxiv.org/html/2611.17811#bib.bib8)\); Claude\-Opus\-5\.5\([Anthropic, 2026a](https://arxiv.org/html/2611.17811#bib.bib5)\)and Claude\-Sonnet\-5\.5\([Anthropic, 2026b](https://arxiv.org/html/2611.17811#bib.bib6)\)\. We use Codex and Claude Code to access their respective models and test whether their behavior aligns with other methods, such as the API and the web interface\.

### 3\.1Collecting Research Papers

Conference selection\.We collect papers from four major machine learning and NLP conferences: ACL 2026, EMNLP 2026, COLM 2026, and ICLR 2027\. We chose these venues because their papers and reviews follow a similar structure\. Each paper has a page\-limited main body, with extra material placed in appendices after the references\. Appendices usually do not have a page limit\. The review system typically asks reviewers to include a summary, strengths, weaknesses, numerical scores or ratings based on key parameters, and an overall assessment score\. This shared format lets us compare results across venues on an equal footing\. We use only recent editions \(2026 onward\) to avoid cases where LLMs have seen these papers during training\.

Sampling & Anonymizing Papers\.From each conference, we sample 200 papers with appendices of at least 15 pages\. The final sample hence consists of 800 unique papers\. The ICLR 2027 papers are anonymous submissions, while the ACL, EMNLP, and COLM papers are camera\-ready versions of accepted papers\. To match a double\-blind review setting, we anonymize the accepted papers by removing author names, affiliations, and email addresses from the PDFs\. We also redact all links, such as code repositories or project pages, since these often reveal author and institution identities \(left panel in Figure[2](https://arxiv.org/html/2611.17811#S1.F2)\)\. Finally, in our main experiments, we disable web search and other browsing tools for all LLM reviewers so the models cannot look up the paper or its authors online and must rely only on the provided PDF\.

### 3\.2Generating Reviews

We use a two\-step pipeline for this task\. We first generate a review for each paper, then identify which part of the paper each weakness comes from\.

Review generation\.We give each of the anonymized papers to the LLM with a simple prompt: “I am reviewing this paper for \[conference\_name\]\. Give me a \[conference\_name\] style review\.” We keep the prompt short and simple on purpose\. This mirrors how a reviewer might plausibly use an LLM in practice, and avoids confounding the generated review with external prompt artifacts\. Because all four conferences use a similar review format, the model can produce a structured review \(summary, strengths, weaknesses, and scores\) from the conference name alone, without us spelling out the format \(Figure[2](https://arxiv.org/html/2611.17811#S1.F2)middle panel\)\.

Section attribution\.Once we have the generated reviews, we want to know which parts of the paper the model draws its weaknesses from\. To localize that, we give the paper and its generated review back to the same model and ask it to name the section of the paper each weakness comes from \(right panel in Figure[2](https://arxiv.org/html/2611.17811#S1.F2)\)\. Some points span more than one section, such as “the claims made in the introduction are not supported by the experiments…,” so we allow the model to name up to two sections\. Some points are general and do not refer to any specific section, such as “the paper is well written\.” For these, we instruct the model to answer “not specified\.”

We split the pipeline into two steps on purpose\. An alternative would be to ask the model to name the section for each weakness while generating the review itself\. However, this would add extra instructions to the prompt and make it less natural\. Our design keeps the prompt close to real\-world usage, so prompt artifacts do not affect the generated review\.

## 4Experimental Results

We assign each weakness to either the main body or the appendix\. A weakness is linked to the appendix if any part of its supporting evidence comes from the appendix, whether from the appendix alone \(W4 in Figure[2](https://arxiv.org/html/2611.17811#S1.F2)\) or from the appendix and the main body together \(W2 in Figure[2](https://arxiv.org/html/2611.17811#S1.F2)\)\. We then compute the percentage of weaknesses in the main body and the appendix\.

### 4\.1LLMs Draw Heavily on the Appendix

Figure 3:Percentage of weaknesses in LLM\-generated reviews drawn from the main body, appendix, and both combined, from recent conferences \(ACL 2026, EMNLP 2026, COLM 2026, and ICLR 2027\)\. Across almost all, more than half of the weaknesses are drawn from appendix\-related content\.Figure[3](https://arxiv.org/html/2611.17811#S4.F3)shows where the weaknesses in LLM\-generated reviews come from\. The pattern is consistent across all models and all four conferences: about half or more of the weaknesses draw on the appendix\. GPT draws 17\.33% of its weaknesses from the appendix alone \(white shade\), with no link to the main body, compared to 10\.16% for Claude\. The results indicate that LLM reviewers do not distinguish between the weight of the main\-body content and the appendix\. A large share of their criticism comes from content that authors often treat as secondary\. For these reviewers, the appendix is not optional reading\. It directly shapes the feedback a paper receives\.

### 4\.2Comparing Model Variants

We repeat our analysis with GPT\-5, an earlier model from the same provider, and compare it against the newer GPT\-6\-Astra and GPT\-6\.1\-Sol\. This lets us directly compare across model generations to observe changes in preference or model bias when reviewing the paper\. We run GPT\-5 through the same two\-step pipeline and, for each model, compute the share of weaknesses that involve the appendix\.

\(a\)Percentage of appendix\-related weaknesses\. It rises in the latest variants, GPT\-6\.1\-Sol and GPT\-6\-Astra, while GPT\-5 had less\.\(b\)Where weaknesses stem from, compared across access methods of GPT\-6\-Astra\.
Figure 4:Overview of appendix\-related weaknesses: \(a\) trend across model variants, \(b\) breakdown by access method\.Figure[4\(a\)](https://arxiv.org/html/2611.17811#S4.F4.sf1)shows the results\. The difference between the generations is large\. For GPT\-5, only a small share of weaknesses relate to the appendix\. Both newer models, GPT\-6\.1\-Sol and GPT\-6\-Astra, rely on it much more heavily, and this holds for every conference\. This suggests that thebias towards the appendix is mainly a property of newer models\. Even though the new models are an improvement, their bias toward the appendix has grown\. Being able to read a paper in detail has not taught them which parts matter most\.

### 4\.3Comparing Access Methods

There exist different approaches for using OpenAI or Anthropic products to review a paper\. In daily use, most people upload the PDF to the web \(chat interface\) or use coding agents such as Codex\. On the other hand, researchers may also call the model through an API, which is easier to control and repeat at scale\. While these methods use the same underlying model, they differ in how they read the paper\.

API\.With the API, the whole PDF is sent to the model in a single request, so it sees the full paper at once, including figures, tables, and layout\. Another common way researchers use the API is to extract the text from the PDF themselves and feed it to the model\. This is not ideal for paper review, since the model sees the paper without its figures\. Nonetheless, we include it because it removes every layer between the model and the paper \(no PDF parsing by the API\)\. The model receives the full text at once, making this setting a clean control for the model’s own behavior\.

Codex\.Codex works differently as it does not receive the paper directly\. It runs shell commands to read the PDF, usually starting with a text extraction tool such as pdftotext, and reads the command’s output\. However, Codex limits how much command output it passes back to the model\. By default, this limit is about 10k tokens per tool call\. When the output is longer, Codex keeps the beginning and the end, drops the middle, and inserts a short note saying how many tokens were removed\. This cut is made by Codex harness, not by the model\. To see the rest, the model has to ask for it with follow\-up commands, such as searching for section headings or extracting specific page ranges\. What it ends up reading is therefore what it chooses to look up, and it may skip some sections\.

Web\.The web framework for accessing the model may also vary as it could include varying system prompts, memory, and safety layers to optimize for user dialogue\. Therefore, the model response can vary based on the added processing that users cannot configure\. The model’s method for reading and analyzing an uploaded file is not public\. We provide one PDF at a time and turn off memory\.

In short, the methods differ in how the model reads the paper\. This could change how much attention the appendix receives; therefore, we test all four access methods\. We run the same review pipeline on 100 ICLR 2027 submissions using GPT\-6\-Astra with four methods: Codex, the full extracted text as API input, API \(direct PDF input\), and the web interface\.

Results\.Figure[4\(b\)](https://arxiv.org/html/2611.17811#S4.F4.sf2)shows the results\. Appendix dependency is consistent across methods, with all methods showing more than 50% of weaknesses from appendix\-related material\. Although each method views the paper differently, bias toward the appendix is consistent\. All methods cite weaknesses from the appendix, suggesting this behavior reflects the underlying model\.

## 5Human vs LLM Reviews: Appendix Dependency Study

So far, we have seen that LLM\-generated reviews draw heavily on the appendix for weaknesses\. Since human reviewers can also draw weaknesses from the appendix, this raises the question: how do humans and LLMs differ in the way they draw weaknesses from the appendix? In this section, we study this question by comparing human\-written reviews with LLM reviews\.

Extracting Human Reviews\.To compare LLM with human review, we collect human\-written reviews for ICLR 2021 to 2024 from OpenReview\. Each ICLR edition is reviewed in the previous calendar year\. For example, ICLR 2021 papers were reviewed in 2020\. For each year, we sample 200 papers using the same rule as before: the paper must have an appendix of at least 10 pages\. Long appendices were less common in earlier editions of the conference, so we keep the limit at 10 pages\. We choose the years 2021\-2024 on purpose\. Before 2021, the conference received fewer submissions, and appendices were usually short\. From 2025 onward, reviewers may well have used LLMs to help write their reviews, which would blur the line between human and LLM reviews and confound our results\. The intervening years give us a clean window\. Each paper typically has 3\-4 reviews, and we randomly select one review per paper\.

To identify the source of each weakness, we reuse the second step of our pipeline \(right panel in Figure[2](https://arxiv.org/html/2611.17811#S1.F2)\) to attribute each weakness to a specific section\. We give GPT\-6\.1\-Sol the PDF and the corresponding human\-written review and ask it to name the paper section each weakness refers to\. We then compare the results with 200 ICLR 2027 papers whose reviews were generated by LLMs\. We do not directly compare human and LLM\-generated reviews for the ICLR 2021\-2024 papers themselves because these papers and their reviews have been public on OpenReview for years and have likely been part of the models’ training data\. ICLR 2027 submissions avoid this problem \(as submissions to this conference have been close to the time of model release\)\.

### 5\.1Humans Prioritize the Main Body, LLMs Do Not

Table 1:Appendix\-related weaknesses \(%\) in human reviews \(ICLR 2021–24\) and LLM reviews \(ICLR 2027\)\.Appendix\-relatedAppendixonlyHumans8\.40±\\pm1\.444\.36±\\pm1\.03LLMs62\.00±\\pm7\.7014\.19±\\pm3\.32Table[1](https://arxiv.org/html/2611.17811#S5.T1)shows the percentage of appendix\-related weaknesses in human reviews \(ICLR 2021\-2024\) and LLM reviews \(ICLR 2027\)\. For LLM reviews, we report the mean over all four LLMs in Figure[3](https://arxiv.org/html/2611.17811#S4.F3)\. The overall picture is clear: human reviewers treat the main body and the supplementary material very differently from LLMs\. LLMs attribute more than 50% of their weaknesses to appendix\-related material, while humans attribute less than 10%\. The difference also shows at the individual\-review level\. Of the 200 papers sampled each year from 2021 to 2024, only 159 in total received a human review with any appendix\-related weakness\. In contrast, all 200 LLM\-generated reviews for ICLR 2027 included at least one appendix criticism\.

A Growing Trend in Human Reviews\.If reviewers are using LLMs to help write their reviews, we would expect human reviews to slowly adopt the response of these models in their posted reviews\. To test this, we track how often human\-written weaknesses involve the appendix across 1,400 papers and reviews from ICLR 2021\-27 \(200 papers from each year\)\.

Table[2](https://arxiv.org/html/2611.17811#S5.T2)shows the result \(also Figure[1](https://arxiv.org/html/2611.17811#S0.F1)\)\. Appendix\-related weaknesses remained largely the same from 2020 to 2024\. LLMs were already being used in peer review by 2023\([Liang et al\., 2026](https://arxiv.org/html/2611.17811#bib.bib1)\), but human reviews did not shift toward the appendix\. This changed in the past couple of years \(2025 and 2026\)\. The share of appendix\-related weaknesses rises sharply in ICLR 2026 reviews, the first review cycle after the release of GPT\-5\. This is also the cycle in which about a fifth of reviews were flagged as fully AI\-generated\([Glickenhaus et al\., 2026](https://arxiv.org/html/2611.17811#bib.bib2)\)\. The numbers rise drastically in 2026 to 27\.60%, when the review period followed the release of GPT\-6\-Astra and Claude\-Opus\-5\.5\. The timing points to a simple explanation: reviewers did not suddenly start reading appendices more closely, but the tools they relied on did\.

Table 2:Percentage of appendix\-related weaknesses in human\-written reviews for each ICLR edition from 2021 to 2027\. Appendix\-related weaknesses increase sharply after the release of the current models\.∗Reviews written after the release of the evaluated models\.Review year2020202120222023202420252026Appendix\-related weaknesses \(%\)10\.478\.378\.396\.376\.0611\.2727\.60∗

Figure 5:Percentage of weaknesses from each page, comparing human and LLM reviews\. Human reviews are from ICLR 2021–24, LLM reviews are from ICLR 2027\.Where Do Weaknesses Come From?Table[1](https://arxiv.org/html/2611.17811#S5.T1)gives the big picture\. To look more closely at how human and LLM reviews differ, we ask which pages of a paper each weakness comes from\. The second stage of our pipeline identifies the section that each weakness refers to \(e\.g\.,W4 \- Appendix D \- Experiment Details\)\. We map this section back to its page number in the PDF, which gives a finer view of where reviewers focus\. We then compute the percentage of weaknesses from each page across both the main body and the appendix\. For human reviews, we use only the reviews that contain at least one appendix\-related weakness\. For LLM reviews, we use all reviews\.

Figure[5](https://arxiv.org/html/2611.17811#S5.F5)shows the percentage of weaknesses from each page for human and LLM reviews\. The results are intuitive\. Human reviewers cite more weaknesses from the first five pages\. This is where authors usually present their goals and motivation, core idea, and method\. Humans focus their criticism on the core ideas of the paper\. LLMs, by contrast, cite more weaknesses from pages 6 and 7, which mostly contain results and analysis, and, importantly, from throughout the appendix\.

Two patterns stand out\. First, LLMs almost never criticize the opening pages, where a paper frames its problem and argues why it matters\. Humans often push back on exactly this framing, while LLMs seem to accept it and move on to the numbers\. Second, the two groups read the appendix very differently\. Human criticisms fade quickly as the appendix goes on, and almost disappear in its last pages\. LLM weaknesses do not fade at all\. In fact, LLM weaknesses peak well into the appendix rather than at its start, and they continue even beyond page 30, where human reviewers raise almost no concerns\. For an LLM, a detail on page 25 is as fair a target as the main results table\.

### 5\.2How Do Human and LLM Criticisms of the Appendix Differ?

Having established that humans prioritize the main body while LLMs target the appendix as much as the main body, we next ask how humans and LLMs criticize a paper’s appendix\. This is important because the amount of criticism alone does not tell the whole story\. A weakness drawn from the appendix can be useful, such as catching a real error in an experiment, or it can be minor, such as pointing out a missing detail that does not affect the paper’s main claims\. Understanding this difference is important in establishing how LLMs and humans differ in reading and criticizing the appendix\.

Table 3:Categories of appendix\-related weaknesses in human reviews \(ICLR 2021\-2024\) and LLM reviews \(ICLR 2027\)\.MethodologyResultsAnalysisPresentationPromptDiscussionTables & Fig\.Core ideaHumans35\.76\.513\.532\.21\.30\.37\.81\.7LLMs54\.013\.625\.22\.10\.30\.33\.01\.4

Weakness Taxonomy\.To understand what humans and LLMs criticize in the appendix, we sort each appendix\-related weakness into one of eight categories:Core Idea,Methodology,Results,Analysis,Prompts,Discussion,Tables and Figures, andPresentation\. We collect weaknesses that cite the appendix, either alone or with the main body, and ask GPT\-6\.1\-Sol to assign each to a single category based on what it criticizes\. If no category fits, the model can answerNone\. The model is instructed to rely only on what the review says and what the paper contains, and not to rewrite or reassess the review\. We then compute the percentage of weaknesses in each category\.

Table[3](https://arxiv.org/html/2611.17811#S5.T3)shows what humans and LLMs criticize when they draw on the appendix\. Both focus on Methodology, but LLMs do so far more, and they also raise more concerns about Results and Analysis\. A large share of human criticisms goes to Presentation, which concerns the paper’s organization and readability, a category LLMs almost never use\. Overall, LLM criticism of the appendix is around technical details, while human criticism also covers how the paper is written and organized\.

Qualitative Analysis\.The categories tell us what reviewers criticize, but not how\. To get a closer look, we read through appendix\-related weaknesses from the main categories and compare how humans and LLMs express them\. Table[5\.2](https://arxiv.org/html/2611.17811#S5.SS2)shows some examples\. On methodological details, human reviewers tend to dig deep into a single issuethat matters to the core of the paper\. LLMs treat the appendix as technical material to audit, highlighting details unrelated to the paper’s central idea\. A major factor is human expectations for presentation\. Humans expect the main body to stand on its own, so they object when essential sections are pushed into the appendix\. LLMs almost never raise this concern\. This highlights a key difference: humans recognize that the main body carries more weight than the appendix, while LLMs do not\. Another important observation is how LLMs criticize the results and analysis\. They cross\-check the main text against appendix details to find small mismatches, question whether the results truly support the claims, and suggest confounds or alternative explanations\. They also repeatedly ask for more experiments and statistical checks, such as additional seeds, significance tests, confidence intervals, etc\. These points are often valid, but they do not engage with the paper’s core contribution or main idea\.

Table 4:Qualitative analysis of human and LLM weaknesses attributed to the appendix showing how different the weaknesses are from the primary categories in Table[3](https://arxiv.org/html/2611.17811#S5.T3)\.## 6Discussion

Our Interpretation\.We offer our interpretation of why this behavior occurs\. Reading everything is not the same as weighing everything\. Human reviewers tend to follow an unwritten rule: the main body is the paper, and the appendix exists to support it\. Authors therefore focus primarily on polishing the main content, and usually refer to additional experiments or more technical details in the appendix\. LLMs, however, seem to weigh every line equally, which leaves them more likely to find weaknesses in the appendix\. The appendix is usually longer, less polished, and full of details such as hyperparameters, extra tables, and edge cases, which makes it easy ground for finding flaws\. For a model trained to be thorough and specific, the appendix becomes a gold mine of weaknesses\.

Is Appendix Bias Good or Bad?We view appendix bias as having both benefits and drawbacks\. On the positive side, authors are responsible for every page they submit\. An error in a table in Appendix G is still an error, and a reviewer who identifies it improves the paper\. In this sense, LLM reviewers raise the standard, since they leave no part of the paper unexamined\. However, effective reviewing also requires judgment about what matters most\. When most of a review’s weaknesses come from appendix content unrelated to the main contribution, the feedback stays on the surface\. It does not engage with the core of the paper and does little to help authors improve it\. This question is not only academic for us\. Our own recent submissions have received reviews where most main concerns came from deep in the appendix, and that experience motivated this work\. We hope our findings bring this behavior to light and encourage reviewers to use LLMs more carefully, as a tool for checking the details, not as the judge of what matters most in a paper\.

## 7Conclusion

We show that the latest LLMs derive the majority of their criticism from the appendix\. This pattern is consistent across models, conferences, and access methods, and we did not observe this behavior in earlier models\. Human reviewers weigh the main body and the appendix differently, but LLMs treat the whole paper as one unit\. This matters because reviewers use these models to find weaknesses, and authors use them to judge whether their paper is ready\. When a model fills a review with issues from supplementary materials, it can make a paper proposing a strong idea look unpolished and pull attention away from its main contribution\. We hope this work helps reviewers use LLMs more carefully, helps authors read AI feedback with the right weight in mind, and encourages model developers to build reviewers that know what matters most in a paper\.

## Limitations

LLM\-based attribution\.We use an LLM to trace each weakness back to the section of the paper it comes from, and to sort appendix\-related weaknesses into categories\. While this lets us analyze thousands of reviews, the model can make mistakes, for example, when a weakness is vague or spans several sections\. We mitigate this by instructing the model not to make assumptions and to answerNonewhen it is uncertain\. We also manually reviewed a sample of the outputs to ensure the attributions are credible \(a point we suspect an LLM reviewer would flag for lacking statistical tests\)\. Nonetheless, this remains a limitation\.

Comparing humans and LLMs\.Our human reviews and LLM reviews come from different papers and different years\. We made this choice on purpose, since older papers and their reviews are likely part of the models’ training data\. Still, writing styles and appendix norms change over time, and these differences may affect the comparison\. We also note that the rise of appendix\-related criticism in human reviews is a temporal correlation, not proof that LLMs caused it\.

What we do not measure\.We study where criticism comes from and what it is about, but not whether each weakness is correct or useful\. Some appendix\-related weaknesses point to real problems rather than surface\-level criticisms\. We think a natural follow\-up work would be to study the validity of weaknesses attributed to the appendix, which we leave for future work\.

Additionally, our focus in this work is on the weaknesses pointed out by the LLM\. We do not explore the numerical scores assigned to a paper\. Future work can explore how appendix\-related weaknesses affect the quantitative scores assigned to a paper\.

## Ethics Statement

Data\.All papers and human reviews in this study are publicly available\. We collected papers from ACL, EMNLP, and COLM public proceedings, and ICLR submissions and reviews from OpenReview\. ICLR reviews are published anonymously, and we analyze them only in aggregate\. We do not try to identify reviewers, and we do not judge whether any individual review was written with LLM help\. Our findings on human reviews describe trends across years, not the behavior of specific people\.

Generated reviews\.We generate reviews only to study model behavior\. We did not use them in any real review process\. For accepted papers, we removed author names, affiliations, and links before giving them to the models, to match the double\-blind setting of peer review\.

Intended use\.Our goal is to help the community understand how AI reviewers read a paper, so that these tools can be used more carefully\. We do not suggest that authors should shorten their appendices or leave out supporting evidence\. Detailed appendices support reproducibility and remain valuable\. Instead, we hope our findings encourage reviewers to weigh appendix\-based criticism appropriately\.

## References

- Anthropic \(2026a\)AnthropicClaude opus 5\.5\.External Links:[Link](https://www.anthropic.com/claude-opus-5-5)Cited by:[§3](https://arxiv.org/html/2611.17811#S3.p2.1)\.
- Anthropic \(2026b\)AnthropicClaude sonnet 5\.5\.External Links:[Link](https://www.anthropic.com/claude-sonnet-5-5)Cited by:[§3](https://arxiv.org/html/2611.17811#S3.p2.1)\.
- Baumannet al\.\(2026\)J\. Baumann, J\. Pei, S\. Koyejo, and D\. HovyStop automating peer review without rigorous evaluation\.InPost\-AGI Science and Society Workshop,External Links:[Link](https://openreview.net/forum?id=cJhlquXIuS)Cited by:[§2](https://arxiv.org/html/2611.17811#S2.p1.1)\.
- Camelo\-Guerrero and Diaz\-Rodriguez \(2026\)A\. Camelo\-Guerrero and J\. Diaz\-RodriguezHow closely do llm reviews align with human peer review?\.External Links:2608\.03659,[Link](https://arxiv.org/abs/2608.03659)Cited by:[§1](https://arxiv.org/html/2611.17811#S1.p2.1),[§2](https://arxiv.org/html/2611.17811#S2.p1.1)\.
- Chenet al\.\(2026\)Z\. Chen, Z\. Yang, Y\. Ma, M\. Backes, and Y\. ZhangP​e​e​r​C​h​e​c​kPeerCheck: Enhancing LLM\-generated academic reviews towards human\-level quality\.InFindings of the Association for Computational Linguistics: ACL 2026,M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),External Links:[Link](https://aclanthology.org/2026.findings-acl.1170/),[Document](https://dx.doi.org/10.18653/v1/2026.findings-acl.1170)Cited by:[§2](https://arxiv.org/html/2611.17811#S2.p1.1)\.
- Duet al\.\(2024\)J\. Du, Y\. Wang, W\. Zhao, Z\. Deng, S\. Liu, R\. Lou, H\. P\. Zou, P\. N\. Venkit, N\. Zhang, M\. Srinath,et al\.LLMs assist NLP researchers: critique paper \(meta\-\)reviewing\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing \(EMNLP 2024\),Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),External Links:[Link](https://aclanthology.org/2024.emnlp-main.292/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.292)Cited by:[§2](https://arxiv.org/html/2611.17811#S2.p1.1)\.
- Fichtlet al\.\(2026\)A\. M\. Fichtl, L\. Ellinger, J\. Kelber, K\. Olík, and G\. GrohAI\-assisted peer review across research communities: from reviewer ai policies to llm review quality\.External Links:2608\.03581,[Link](https://arxiv.org/abs/2608.03581)Cited by:[§1](https://arxiv.org/html/2611.17811#S1.p2.1),[§2](https://arxiv.org/html/2611.17811#S2.p1.1)\.
- Glickenhauset al\.\(2026\)B\. Glickenhaus, K\. Thai, J\. Russell, E\. Masrour, Y\. Han, M\. Spero, and B\. EmiPangram 4 technical report\.External Links:2607\.27183,[Link](https://arxiv.org/abs/2607.27183)Cited by:[§1](https://arxiv.org/html/2611.17811#S1.p1.1),[§5\.1](https://arxiv.org/html/2611.17811#S5.SS1.p3.1)\.
- Kaneko \(2026\)M\. KanekoParaphrasing adversarial attack on llm\-as\-a\-reviewer\.External Links:2601\.06884,[Link](https://arxiv.org/abs/2601.06884)Cited by:[§2](https://arxiv.org/html/2611.17811#S2.p2.1)\.
- Kimet al\.\(2026\)S\. S\. Y\. Kim, W\. H\. Deng, J\. W\. Vaughan, B\. Su, W\. Su, A\. Agarwal, S\. Li, M\. Jaggi, D\. G\. Goldstein, N\. B\. Shah, and M\. DudíkUse and effects of llms in peer review: a randomized experiment and survey at icml 2026\.External Links:2609\.19420,[Link](https://arxiv.org/abs/2609.19420)Cited by:[§2](https://arxiv.org/html/2611.17811#S2.p1.1)\.
- Kumaret al\.\(2025\)S\. Kumar, S\. Garg, S\. Sengupta, T\. Ghosal, and A\. EkbalMixRevDetect: towards detecting AI\-generated content in hybrid peer reviews\.\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(NAACL 2025\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),External Links:[Link](https://aclanthology.org/2025.naacl-short.79/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-short.79)Cited by:[§2](https://arxiv.org/html/2611.17811#S2.p1.1)\.
- Liet al\.\(2026\)L\. Li, J\. Xiong, C\. Zhu, R\. Yu, C\. Chen, J\. Wang, R\. Ma, and Z\. LuLLM\-as\-a\-reviewer: benchmarking their ability, divergence, and prompt injection resistance as paper reviewers\.External Links:2605\.25415,[Link](https://arxiv.org/abs/2605.25415)Cited by:[§1](https://arxiv.org/html/2611.17811#S1.p2.1),[§2](https://arxiv.org/html/2611.17811#S2.p1.1),[§2](https://arxiv.org/html/2611.17811#S2.p2.1)\.
- Lianget al\.\(2026\)W\. Liang, Z\. Izzo, Y\. Zhang, H\. Lepp, H\. Cao, X\. Zhao, L\. Chen, H\. Ye, S\. Liu, Z\. Huang, D\. A\. McFarland, and J\. Y\. ZouMonitoring ai\-modified content at scale: a case study on the impact of chatgpt on ai conference peer reviews\.External Links:2403\.07183,[Link](https://arxiv.org/abs/2403.07183)Cited by:[§2](https://arxiv.org/html/2611.17811#S2.p1.1),[§5\.1](https://arxiv.org/html/2611.17811#S5.SS1.p3.1)\.
- Lianget al\.\(2024\)W\. Liang, Y\. Zhang, H\. Cao, B\. Wang, D\. Y\. Ding, X\. Yang, K\. Vodrahalli, S\. He, D\. S\. Smith, Y\. Yin, D\. A\. McFarland, and J\. ZouCan large language models provide useful feedback on research papers? a large\-scale empirical analysis\.NEJM AI1\(8\),pp\. AIoa2400196\.External Links:[Document](https://dx.doi.org/10.1056/AIoa2400196),[Link](https://ai.nejm.org/doi/full/10.1056/AIoa2400196),https://ai\.nejm\.org/doi/pdf/10\.1056/AIoa2400196Cited by:[§2](https://arxiv.org/html/2611.17811#S2.p1.1)\.
- Lin \(2026\)Z\. LinHidden prompts in manuscripts exploit ai\-assisted peer review\.Commun\. ACM\.External Links:[Link](https://doi.org/10.1145/3779116),[Document](https://dx.doi.org/10.1145/3779116)Cited by:[§2](https://arxiv.org/html/2611.17811#S2.p2.1)\.
- Nguyenet al\.\(2026\)D\. Nguyen, W\. Hao, Y\. Elazar, and C\. TanBenchmarking agentic review systems\.InNeurIPS 2026 Workshop on AI\-Native Academia: Authorship, Peer Review, and Conference Governance under AI,External Links:[Link](https://openreview.net/forum?id=ew8BW13QL4)Cited by:[§1](https://arxiv.org/html/2611.17811#S1.p2.1),[§2](https://arxiv.org/html/2611.17811#S2.p1.1)\.
- Nguyen and Ahmadi \(2026\)T\. H\. Nguyen and Z\. AhmadiLLM\-based scientific peer review: methods, benchmarks, and reliability challenges\.External Links:2606\.25057,[Link](https://arxiv.org/abs/2606.25057)Cited by:[§2](https://arxiv.org/html/2611.17811#S2.p1.1)\.
- OpenAI \(2026a\)OpenAIGPT\-6 astra\.External Links:[Link](https://openai.com/index/gpt-6-astra/)Cited by:[§3](https://arxiv.org/html/2611.17811#S3.p2.1)\.
- OpenAI \(2026b\)OpenAIGPT\-6\.1 sol\.External Links:[Link](https://openai.com/index/introducing-gpt-6-1-sol/)Cited by:[§3](https://arxiv.org/html/2611.17811#S3.p2.1)\.
- Sahaet al\.\(2026\)R\. Saha, G\. Juneja, D\. Chaudhuri, N\. Sajeevan, N\. B\. Shah, and D\. PruthiPolicies permitting LLM use for polishing peer reviews are currently not enforceable\.InForty\-third International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=znFIR7WFGt)Cited by:[§2](https://arxiv.org/html/2611.17811#S2.p1.1)\.
- Sharmaet al\.\(2026\)V\. Sharma, T\. Joachims, and S\. DeanDo llms favor llms? quantifying interaction effects in peer review\.External Links:2601\.20920,[Link](https://arxiv.org/abs/2601.20920)Cited by:[§2](https://arxiv.org/html/2611.17811#S2.p1.1)\.
- Shen and Wang \(2026\)S\. Shen and K\. WangDetecting ai\-generated content in academic peer reviews\.External Links:2602\.00319,[Link](https://arxiv.org/abs/2602.00319)Cited by:[§2](https://arxiv.org/html/2611.17811#S2.p1.1)\.
- Teoet al\.\(2026\)R\. S\.Y\. Teo, Y\. Yamada, S\. Kotyan, Y\. Imajuku, and T\. ClanuwatBeyond imitation: a framework and benchmark for LLM\-assisted peer review\.Transactions on Machine Learning Research\.Note:External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=7iX2Z2bPFB)Cited by:[§2](https://arxiv.org/html/2611.17811#S2.p1.1)\.
- Vasuet al\.\(2026\)S\. S\. M\. Vasu, S\. S\. Joshi, P\. Christmann, I\. Sheth, R\. Binkyte, K\. D\. Chowdhury, R\. Kerkouche, H\. Wang, and M\. FritzStyle over substance? linguistic bias in llm\-based peer review\.External Links:2610\.35254,[Link](https://arxiv.org/abs/2610.35254)Cited by:[§2](https://arxiv.org/html/2611.17811#S2.p2.1)\.
- Yanget al\.\(2026\)X\. Yang, Z\. Sha, J\. Li, J\. Yu, Y\. Sun, M\. Zhao, J\. Fang, X\. Guo, Y\. Wu, X\. Hu, Y\. Luo, Q\. Liu, and Z\. WangNo hidden prompts needed\! you can game AI peer review with presentation\-only revisions\.InNeurIPS 2026 Workshop on AI\-Native Academia: Authorship, Peer Review, and Conference Governance under AI,External Links:[Link](https://openreview.net/forum?id=Uf63VRsTaK)Cited by:[§2](https://arxiv.org/html/2611.17811#S2.p2.1)\.
- Yunet al\.\(2026\)J\. Yun, J\. Kwon, M\. Kim, and Y\. KimPosition on llm\-assisted peer review: addressing reviewer gap through mentoring and feedback\.External Links:2601\.09182,[Link](https://arxiv.org/abs/2601.09182)Cited by:[§2](https://arxiv.org/html/2611.17811#S2.p1.1)\.
- Zhouet al\.\(2024\)R\. Zhou, L\. Chen, and K\. YuIs LLM a reliable reviewer? a comprehensive evaluation of LLM on automatic paper reviewing tasks\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),N\. Calzolari, M\. Kan, V\. Hoste, A\. Lenci, S\. Sakti, and N\. Xue \(Eds\.\),External Links:[Link](https://aclanthology.org/2024.lrec-main.816/)Cited by:[§2](https://arxiv.org/html/2611.17811#S2.p1.1)\.
- Żurawickiet al\.\(2026\)K\. Żurawicki, J\. Farganus, A\. Gaweł, M\. Bystroński, and T\. J\. KajdanowiczPRAIB: peer review ai benchmark of behaviour of llm\-assisted reviewing\.External Links:2605\.29815,[Link](https://arxiv.org/abs/2605.29815)Cited by:[§2](https://arxiv.org/html/2611.17811#S2.p1.1)\.

Similar Articles

AI Reviews Are Biased Towards the Appendix

Hugging Face Daily Papers

研究分析了 ACL、EMNLP、COLM 和 ICLR 的 2000 篇论文,发现 GPT 和 Claude 等大模型在审稿中约一半以上的批评点来自附录,而人类审稿人仅有 8.4%,且更关注论文呈现与核心思想。

NeurIPS 2026 AI-generated reviews [D]

Reddit r/MachineLearning

Discussion about the use of AI-generated reviews at NeurIPS 2026, including concerns over prompt injection and lack of consequences for reviewers using LLMs without proper oversight.