Metag: A dataset to build agentic meta-reviewing capabilities

# Metag: A dataset to build agentic meta-reviewing capabilities
Source: [https://arxiv.org/html/2608.20488](https://arxiv.org/html/2608.20488)
\\workshoptitle

AI Native Academia

Min ChenDivya TadimetiGemma ZhangAlice LiNigel Boachie KumankumahPavan Uttej RavvaSadid HasanSomya ChatterjeePruthvi Prakash NavadaXiao WangYue KangSulaiman VesalLarry HeckAffiliation:Georgia Institute of Technology\{anisundar, svesal\}@microsoft\.com, larryheck@gatech\.edu\[0\.5em\] Microsoft

###### Abstract

AI tools increasingly support tasks across the scientific research cycle, from experiment design and manuscript preparation to peer review\. At the same time, the continuing growth in conference submissions has increased the burden on meta\-reviewers, who must synthesize reviewer feedback, author rebuttals, and manuscript revisions\. To address this concern, this paper introduces Metag, a dataset to accelerate the development of meta\-reviewing agents, specifically to identify changes made to scientific articles during the review\-rebuttal process\. Each instance contains a reviewer concern, the author’s proposed resolution, and the manuscript diffs implementing the stated change\. Metag is collected by obtaining manuscript versions from before the review deadline and after acceptance, computing differences between the two documents, and asking human annotators to align these differences with action items from OpenReview discussions\. The resulting dataset consists of 349 high\-quality action items tied to paper differences and will enable building methods to empower meta reviewers to quickly identify whether authors have addressed reviewer statements and where in the paper those changes have been made, resulting in additional transparency and traceability throughout peer review\. The dataset is publicly available at[https://github\.com/microsoft/Metag\-dataset](https://github.com/microsoft/Metag-dataset)\.

## 1Introduction

Recent work has seen an increase in the usage of AI tools and assistants both in authoring and reviewing scientific papers[Liang2024MonitoringAC](https://arxiv.org/html/2608.20488#bib.bib13)\. For authors, AI assistants help in proofreading papers, improving writing structure, and rectifying grammatical errors\. Similarly, AI tools have also been developed to assist in the reviewing process\. Paper Assistant Tool[icml2026pat](https://arxiv.org/html/2608.20488#bib.bib7), an AI tool introduced in the ICML 2026 review process, has been deployed to assist in flagging issues, highlighting errors in methods, and helping improve the writing of papers\. AI tools reduce cognitive burden across experimentation and reviewing, thereby accelerating the research cycle and resulting in massive growth in the scale of submissions to premier AI conferences[bok2025openreview](https://arxiv.org/html/2608.20488#bib.bib2)\. While the development of such tools have primarily targeted paper authors and reviewers, an important yet relatively under\-explored area of research is the development of AI tools to assist the meta\-reviewing process[kuznetsov2024can](https://arxiv.org/html/2608.20488#bib.bib11)\.

Meta\-reviewers must synthesize large volumes of information across reviews, rebuttals, and author discussions into a coherent statement\. This process is challenging because the relevant evidence is distributed across long, interconnected texts, reviewers may emphasize different concerns, and author responses may describe revisions without making it immediately clear where those revisions appear in the final manuscript[kuznetsov2024can](https://arxiv.org/html/2608.20488#bib.bib11)\. Although NLP systems can assist with review aggregation and summarization, effective support for meta\-reviewers requires*traceability*: the ability to connect concerns raised during peer review to the concrete changes made by authors\.

![Refer to caption](https://arxiv.org/html/2608.20488v1/Figures/Teaser.png)Figure 1:An example of the data collection process and sample data from Metag\. Based on the action items identified by reviewers and the modifications made post submission to create the Camera\-Ready paper \(edits highlighted in green\), the task is to extract the relevant diffs specific to the action items\.Such traceability can improve transparency and accountability in the review\-rebuttal process\. It can help meta\-reviewers and program committee members verify whether promised revisions were implemented, distinguish substantive changes from unrelated edits, and inspect the manuscript evidence underlying an author’s response\. Existing tools can identify textual differences between versions of a PDF[ssibb\_pdf\_diff\_viewer](https://arxiv.org/html/2608.20488#bib.bib17), but they do not generally explain which reviewer action item motivated each change\. Automatically linking manuscript diffs to reviewer concerns therefore represents an important and underexplored extension of document comparison\.

To support research on this problem, this paper introduces a data\-collection methodology and releases Metag, a dataset that links action items extracted from reviewer\-author dialogues to their corresponding changes between original and revised scientific papers\. Metag is intended to support the development of agentic human\-in\-the\-loop systems that assist meta\-reviewers navigate revisions by verifying author statements and ground judgments in manuscript evidence\. An example of the dataset is provided in Figure[1](https://arxiv.org/html/2608.20488#S1.F1), given the action items from the review on OpenReview, the task is to pick out the specific changes that effectuate the reviewer request\.

Metag is built by computing adiffbetween an accepted paper’s pre\-review arXiv submission against its camera\-ready version, then linking specific edits to the reviewer comments that prompted them via reviewer\-author dialogue\. The process yields a dataset of paired edits and comments connecting reviewer feedback to the concrete manuscript changes addressing them\. The dataset is publicly available at[https://github\.com/microsoft/Metag\-dataset](https://github.com/microsoft/Metag-dataset)under the CDLA\-2\.0 license111[https://cdla\.dev/permissive\-2\-0/](https://cdla.dev/permissive-2-0/)\.

## 2Related Work

##### Scientific reviews without document edits\.

Several datasets study scientific peer review without modeling how authors revise their manuscripts\.cPAPERS[sundar2024cpapers](https://arxiv.org/html/2608.20488#bib.bib18)andSciDQA[singh\-etal\-2024\-scidqa](https://arxiv.org/html/2608.20488#bib.bib16)support question answering grounded in scientific papers and their OpenReview discussions, whileORSUM[zeng2024scientific](https://arxiv.org/html/2608.20488#bib.bib24)focuses on scientific opinion summarization\.Revise and Resubmit[kuznetsov\-etal\-2022\-revise](https://arxiv.org/html/2608.20488#bib.bib12)introduces pragmatic tagging of review sentences and links reviewer suggestions to manuscript sections through explicit section references and embedding similarity\. These resources capture the content, intent, or organization of reviews, but do not align reviewer requests with changes observed between manuscript versions\. Similarly,Friction[10\.1145/3706598\.3714316](https://arxiv.org/html/2608.20488#bib.bib25)uses LLMs to identify feedback for novice writers and presents a heatmap indicating passages that may require revision\. Although it supports feedback\-driven writing, it does not model revisions arising from scientific peer review\.

##### Document edits without scientific reviews\.

A complementary line of work studies iterative document revision independently of peer review\.arXivEdits[jiang2022arxivedits](https://arxiv.org/html/2608.20488#bib.bib8)provides a computational framework for extracting changes across arXiv versions at the document, sentence, and word levels\. They additionally classify edit intentions, enabling analysis of why revisions were made\.IteraTeR[du\-etal\-2022\-understanding\-iterative](https://arxiv.org/html/2608.20488#bib.bib4)collects edits from arXiv, Wikipedia, and Wikinews and introduces edit\-intention classification, but does not include the reviewer feedback that prompted scientific revisions\.TETRA[mita2024towards](https://arxiv.org/html/2608.20488#bib.bib14)also addresses document revision, using edits produced by professional editors rather than changes made in response to scientific peer review\. These datasets model what changed, but generally cannot connect those changes to reviewer concerns or author commitments\.

##### Scientific reviews and document edits\.

The work most closely related to Metag isARIES[darcy\-etal\-2024\-aries](https://arxiv.org/html/2608.20488#bib.bib3), which aligns reviewer comments with revisions to scientific manuscripts\. ARIES relies on annotators to identify actionable review comments and align them directly with relevant textual spans\. In contrast, Metag first computes structured differences between the original and revised PDFs and then asks annotators to select the diffs that implement each action item extracted from the reviewer\-author dialogue\. This formulation preserves insertions, deletions, replacements, page locations, and surrounding context, while framing the task as selecting relevant edits from the complete set of manuscript changes\.

More broadly,[kuznetsov2024can](https://arxiv.org/html/2608.20488#bib.bib11)provides a comprehensive taxonomy of NLP research across the peer\-review pipeline, including methods that jointly analyze manuscripts, reviews, and reviewer\-author discussions\. Metag complements this literature by focusing specifically on traceability between review dialogue and observable manuscript revision\. A comparison of the different related datasets in this domain is provided in Table[1](https://arxiv.org/html/2608.20488#S2.T1)\.

Table 1:Comparison between different datasets targeting iterative edits and scientific reviews\. \# Samples = number of samples verified by human annotators\.

## 3Method

### 3\.1Dataset Collection

Metag is collected by leveraging the review\-rebuttal process on OpenReview\. The back\-and\-forth dialogue between reviewers and authors produces a rich record of what changes were made to the paper and a rationale behind them\. Metag collection proceeds in six stages: \(1\) paper and review scraping, \(2\) PDF acquisition, \(3\) PDF Difference Computation, \(4\) Action item extraction, \(5\) human\-annotation and filtering, \(6\) Inter annotator agreement and dataset assembly\. Each stage is described in detail below and is exemplified by Figure[2](https://arxiv.org/html/2608.20488#S3.F2)\.

![Refer to caption](https://arxiv.org/html/2608.20488v1/Figures/MeTAG.jpg)Figure 2:Block Diagram showing the various steps to produce the Metag dataset\. \(1\) OpenReview is scraped to obtain submissions\. \(2\) The submissions are then provided to Semantic Scholar to obtain the corresponding arXiv pre\-submission PDFs\. \(3\) Diffs are computed between the two versions\. \(4\) Simultaneously the action items are extracted from the reviews, and they are \(5\) paired with the action items to produce the final version of Metag\.#### 3\.1\.1Stage 1: Paper and Review Scraping

The OpenReview API222[https://docs\.openreview\.net](https://docs.openreview.net/)is used to obtain all submissions to the target venue, ICLR 2024, in compliance with the OpenReview terms of service333[https://openreview\.net/legal/terms](https://openreview.net/legal/terms)\. While Metag focuses specifically on ICLR, the process can be extended to any venue that publishes reviews on OpenReview\. Metag is limited to work where reviewer comments are addressed in a camera\-ready version of the paper\. As a result, only the accepted papers are retained from the set of all submissions since the existence of a camera\-ready version is guaranteed\. For each accepted paper, the full reviewer\-author dialogue is extracted by traversing the reply tree rooted at eachOfficial\_Reviewnote, recursively collecting everyOfficial\_Commentreply and sorting the resulting thread chronologically\. Then, each dialogue thread is stored as a structured record containing the initial review, the sequence of author and reviewer follow\-up comments, and the anonymous reviewer identifier\.

#### 3\.1\.2Stage 2: PDF Acquisition

The next stage is to obtain the PDFs before and after the review\-rebuttal process\. Since OpenReview does not make available the version of the paper that was submitted, only the final camera\-ready version, the existence of each paper is first checked on arXiv444[https://arxiv\.org](https://arxiv.org/)\.

However, OpenReview and arXiv do not provide cross\-referencing and operate as independent systems\. Therefore, Semantic Scholar[fricke2018semantic](https://arxiv.org/html/2608.20488#bib.bib5);[kinney2025semanticscholaropendata](https://arxiv.org/html/2608.20488#bib.bib10)is used to link OpenReview papers to their arXiv preprints\. Semantic Scholar indexes papers by title and exposes both the accepted venue and the arXiv ID \(if it exists\)\.

The process starts by first obtaining the camera\-ready paper from OpenReview\. The PDF and the submission timestamp creation date \(cdate\) are recorded\. Then, Semantic Scholar is queried with the paper title\. A match is verified with the normalized Levenshtein distance between the returned title and the title used in the query\. If the top result has a distance more than 5%, the search result is rejected and the paper is discarded\. The paper’s arXiv identifier is resolved from the returned result from Semantic Scholar\. Using the arXiv API, all versions of the paper are obtained\. Each version’s arXiv upload date is compared to the OpenReviewcdateand only the most recent upload on or before the submission date is retained, ensuring that the downloaded version is the one closest to the version under review, excluding post\-submission updates\. Papers for which a version history cannot be obtained are excluded as a safeguard\. All downloads and queries are performed with exponential\-backoff retry logic to comply with rate limits\.

1\{

2"paper\_id":"xBfQZWeDRH",

3"action\_item":\{

4"comment":"Themeasureoftrainingaccelerationisbasedonthenumberofepochs\.\.\.",

5"response":"Werevisedthemanuscripttoexplainthiscomparisonmoreexplicitly\."

6\},

7"all\_diffs":\[

8\{

9"diff\_index":0,

10"tag":"replace",

11"old":"OriginalTitleHere",

12"new":"PUBLISHEDATICLR2024ORIGINALTITLE"

13\},

14\{

15"\.\.\.":"diffs1\-\-440omitted"

16\},

17\{

18"diff\_index":441,

19"tag":"replace",

20"old":"inthetrainingprocedure,",

21"new":"\(64vs\.256epochs\),",

22"page":6

23\},

24\{

25"\.\.\.":"diffs442\-\-1066omitted"

26\}

27\],

28"labels":\[

29false,"\.\.\.",true,"\.\.\.",false

30\],

31"relevant\_diff\_indices":\[441\]

32\}

Listing 1:Abbreviated diff\-classification instance\. Ellipses indicate omitted candidates\.
#### 3\.1\.3PDF Difference Computation

The next step is to compute the difference between the PDFs submitted to the venue and the post\-submission version\. To obtain the difference, a structured diff is computed between the two PDFs following the PDF\-Diff repository[ssibb\_pdf\_diff\_viewer](https://arxiv.org/html/2608.20488#bib.bib17)\. Adiffis a contiguous text\-level change representing the difference between two versions of a document, identifying content that was added, removed, replaced, or moved[opengroup\_diff](https://arxiv.org/html/2608.20488#bib.bib21)\. First, both PDFs are parsed using PyMuPDF[pymupdf](https://arxiv.org/html/2608.20488#bib.bib1), followed by Git’s histogram diff algorithm555https://git\-scm\.com/docs/diff\-options\.html\#Documentation/diff\-options\.txt \- \- \-histogramto obtain differences\. The diff extractor operates at the text\-block level, producing a list of edits, each tagged asinsert,delete, orreplace, together with the original and revised text spans, their surrounding context, page numbers, and word\-level bounding boxes\. The result is a list of differences between the original and modified versions of the paper, an example is available in Listing[1](https://arxiv.org/html/2608.20488#LST1)\.

#### 3\.1\.4Action Item Extraction

Reviewer\-author dialogues often discuss multiple aspects of a paper\. In addition to pointing out areas for improvement, reviewers typically ask clarifying questions regarding content\. However, a significant part of the meta\-reviewer’s workflow is to distill this entire conversation into discrete*action items*, i\.e\., pairs of a reviewer concern and the corresponding author commitment to correcting the specific change\.

To broadly filter the reviews into action items,Gemma\-3\-27B\-IT\(temperature0\.10\.1\)[team2025gemma](https://arxiv.org/html/2608.20488#bib.bib19)is provided with the review weaknesses, questions, and discussion thread and prompted to list the action items\. The prompt instructs the model to identify statements in which the authors explicitly commit to modifying the manuscript \(e\.g\. “We have revised Section 3…”, “We added an experiment…”\) and to extract each as a\(comment, response\)pair, prioritizing items that reference specific manuscript locations \(section, table, equation or figure numbers\)\. The full prompt is available in Listing[2](https://arxiv.org/html/2608.20488#LST2)in Appendix[A](https://arxiv.org/html/2608.20488#A1)\.

#### 3\.1\.5Human Annotation and Filtering

##### Comment Filtering

Each LLM\-extracted action item is presented to an annotator who labels it as*keep*,*maybe*, or*discard*\. The filtering UI displays the reviewer comment and author response side by side, supports keyboard shortcuts for rapid annotation, and persists progress to a local cache file so that sessions can be interrupted and resumed\.

##### Diff Linking

Retained action items are routed to an interactive, side\-by\-side PDF viewer that renders the pre\-revision \(left pane\) and post\-revision \(right pane\) PDFs with colour\-coded diff overlays\. Annotators read the reviewer concern and author response, then click on the PDF diff regions that correspond to the described change\. Each click records the diff pane \(left/right\), page number, change type \(insertion/deletion/moved\), the diff text, and its surrounding context\. To amortize the cost of computing the diffs and rendering the PDF, the tool batches consecutive entries from the same paper\. An example of the annotation window is available in Figure[3](https://arxiv.org/html/2608.20488#S3.F3)\.

![Refer to caption](https://arxiv.org/html/2608.20488v1/Figures/Diff-Annotation.png)Figure 3:The annotation interface used to link diffs with reviewer comments and author responses\.
##### Annotation framework

A significant bottleneck for collecting paired datasets of edits and reviewer\-author dialogue is a suitable interface to render and capture information accurately\. To address this issue, we developed a Python\-based annotation framework to identify reviewer\-author action items and link them to the corresponding paper revisions\. Annotators inspect the original and revised PDFs in a side\-by\-side diff\-linking interface\. The interface was implemented with Tkinter and PyMuPDF, with textual differences computed using Git’s diff algorithm and adifflibfallback, following the pipeline outlined in[ssibb\_pdf\_diff\_viewer](https://arxiv.org/html/2608.20488#bib.bib17)\. Insertions, deletions, replacements, and moved text are highlighted directly on the PDF pages, deletions appear in red, additions in green, moved text in blue\. Annotators can navigate between changes, search either document, synchronize scrolling and zooming, and select relevant changes using Shift\+Click or Shift\+Drag\. The interface displays the current reviewer concern and author response, groups action items by paper to reuse computed diffs, and provides controls to save, skip, or advance to the next item\. Selected diffs, including their text, type, page, and surrounding context, are written to JSONL files, while automatic progress detection allows interrupted annotation sessions to resume from the last completed item\. A link to the review on OpenReview is provided for additional context, should the annotator require it\. We will release the code to replicate the annotation framework upon acceptance\.

#### 3\.1\.6Inter Annotator Agreement and Dataset Assembly

Each action item is independently annotated by two annotators\. The linkeddiffsare then reconciled by taking the intersection of the sets of differences identified by each annotator\. Only thediffsthat appear in the intersection are retained\. Two annotators are used to limit the time spent on annotation low while maximizing the number of cleaned data samples\. Final statistics of the dataset are detailed in Table[2](https://arxiv.org/html/2608.20488#S3.T2)\. Annotators agreed on the exact set of diffs 37% of the time and annotated sets had a mean Jaccard similarity of 50\.1%\. Across the annotator pairs, binary agreement on whether an action item corresponded to at least one manuscript diff yielded a mean Cohen’sκ\\kappaof0\.3690\.369, indicating fair agreement beyond chance\. An example of a final datum in the dataset is provided in Listing[1](https://arxiv.org/html/2608.20488#LST1)\. Additional details on the annotator agreement are provided in Appendix[E](https://arxiv.org/html/2608.20488#A5)\.

### 3\.2Diff Classification

Diff classification is addressed using two approaches\. The first is naive lexical retrieval using BM25[10\.1145/2682862\.2682863](https://arxiv.org/html/2608.20488#bib.bib22)and TF\-IDF[SprckJones2021ASI](https://arxiv.org/html/2608.20488#bib.bib9)to rank candidate diffs by their similarity to the action item \(which serves as the query\)\. The second is using neural approaches with language models\. The language modeling\-based approach utilizes a mix of open and closed source models\. Using open\-source models, we first experiment with an embedding\-based approach that computes embeddings for every diff and action item followed by a classification head that classifies the diff as relevant or not\. Next, we also explore a candidate\-selection approach where the language model is tasked with selecting the diff\(s\) relevant to an action item from a list of candidates\. Because each paper contains approximately 1,000 candidate diffs on average,the diff pool is partitioned into contiguous, non\-overlapping windows of up to 80 candidates\. The number of candidates in the window was selected based on the validation split, and an analysis is provided in Appendix[C](https://arxiv.org/html/2608.20488#A3)\. Each candidate is represented by its global index, edit type, page number, original and revised text, and surrounding context\. Language models then receive the action item and one candidate window and are tasked with returning the indices of all relevant diffs, or an empty list of none of the provided diffs are relevant\. The prompt instructed the models to select only candidates in the provided window that directly implemented the action item\. For each window, models return a JSON object of the form\{"relevant\_diff\_indices": \[3, 17\]\}, or an empty list when no candidate was relevant\. Finally, the complete prediction for an action item is obtained by taking the union of the indices selected across all windows\.

Returned indices were validated against the candidates in the current window\. Out\-of\-window indices were treated as hallucinations and discarded\. The valid predictions from all windows belonging to an action item were then unioned to obtain its final predicted diff set\. This set was compared with the human\-annotated relevant indices to calculate precision, recall, and F1\.

### 3\.3Evaluation Metrics

For each action itemii, the predicted set of relevant diff indices,D^i\\hat\{D\}\_\{i\}, is compared with the human\-annotated gold set,DiD\_\{i\}\. True positives, false positives, and false negatives are computed as

TPi=\|Di∩D^i\|,FPi=\|D^i∖Di\|,FNi=\|Di∖D^i\|\.\\mathrm\{TP\}\_\{i\}=\|D\_\{i\}\\cap\\hat\{D\}\_\{i\}\|,\\qquad\\mathrm\{FP\}\_\{i\}=\|\\hat\{D\}\_\{i\}\\setminus D\_\{i\}\|,\\qquad\\mathrm\{FN\}\_\{i\}=\|D\_\{i\}\\setminus\\hat\{D\}\_\{i\}\|\.
Given the true positives, false positives, and false negatives, Precision, Recall, and F1\-score are computed at the micro and macro level\. The micro\-averaging pools decisions across all N action items while macro\-averaging first computes each of the three metrics independently per sample then averages scores across items\.

Table 2:Metag dataset statisticsTable 3:Diff\-classification results\. LLM results report mean±\\pmsample standard deviation across three runs\.

## 4Results

Table[3](https://arxiv.org/html/2608.20488#S3.T3)reports the performance of the lexical and LLM\-based approaches\.

##### Lexical retrieval baselines\.

We evaluate BM25[10\.1145/2682862\.2682863](https://arxiv.org/html/2608.20488#bib.bib22)and unigram TF\-IDF cosine similarity[SprckJones2021ASI](https://arxiv.org/html/2608.20488#bib.bib9)\. The reviewer concern and author response are concatenated to form the query, while each candidate diff is represented by its original text, revised text, and surrounding context\. The retrieval depthkkis selected using validation micro\-F1, a complete sweep is reported in Appendix[D](https://arxiv.org/html/2608.20488#A4)\. Both methods selectk=6k=6, which is close to the average number of relevant diffs per action item\.

The lexical baselines perform poorly overall\. BM25 achieves micro\-F1 scores of \.108 on validation and \.098 on test, while TF–IDF obtains \.092 and \.096, respectively\. Their low recall indicates that relevant revisions often cannot be identified through direct lexical overlap alone\. In many cases, the action item describes the intent of a revision rather than repeating the language introduced in the manuscript\.

##### LLM\-based diff classification\.

We first evaluateGemma\-3\-27B\-IT[team2025gemma](https://arxiv.org/html/2608.20488#bib.bib19)using zero\-shot prompting and LoRA fine\-tuning[hu2021lora](https://arxiv.org/html/2608.20488#bib.bib6)\. We also evaluate a pairwise classifier that uses mean\-pooledGemma\-3\-27B\-ITembeddings to determine whether each candidate diff is relevant\. We further evaluate the hosted modelsGPT\-5\.6\-Sol[openai2026gpt56sol](https://arxiv.org/html/2608.20488#bib.bib15),DeepSeek\-V4\-Pro[xu2026deepseek](https://arxiv.org/html/2608.20488#bib.bib23), andKimi\-K2\.5[team2026kimi](https://arxiv.org/html/2608.20488#bib.bib20)using zero\-shot prompting\. The prompt is presented in Appendix[F](https://arxiv.org/html/2608.20488#A6)\.

Because each action item is associated with approximately 1,000 candidate diffs, we partition the candidate pool into contiguous, non\-overlapping windows\. TheGemmaexperiments use 50 diffs per prompt, corresponding to a median estimated prompt length of approximately 3\.1K tokens\. For the hosted models, the window size is selected by evaluatingDeepSeek\-V4\-Proon validation windows ranging from 10 to 320 candidates\. A window size of 80 obtains the highest mean validation micro\-F1, while reducing the number of repeated instructions and API calls\. This setting produces median prompts of approximately 4\.5K tokens and the complete window\-size analysis is provided in Appendix[C](https://arxiv.org/html/2608.20488#A3)\.

All hosted models receive identical prompts and candidate windows\.GPT\-5\.6\-Solis accessed through the Responses API, whereasDeepSeekandKimiuse the Chat Completions API\. Temperature, nucleus sampling, reasoning effort, and other decoding parameters are left at their deployment defaults\. Each model is evaluated in three independent runs, and we report the mean and sample standard deviation\. Failed API or parsing requests are rerun and merged with the original predictions\.

##### Main findings\.

GPT\-5\.6\-Solprovides the strongest overall test performance, achieving a micro\-F1 of\.360±\.005\.360\\pm\.005and a macro\-F1 of\.398±\.018\.398\\pm\.018\. It combines relatively high precision \(\.270±\.004\.270\\pm\.004\) with high recall \(\.540±\.008\.540\\pm\.008\), yielding a better balance than the other hosted models\.

On validation,DeepSeek\-V4\-Proobtains the highest mean micro\-F1 \(\.410±\.121\.410\\pm\.121\), although its large standard deviation indicates substantial run\-to\-run variability\. Its test micro\-F1 decreases to\.261±\.021\.261\\pm\.021\.Kimi\-K2\.5achieves high recall on both validation and test, but its lower precision limits its test micro\-F1 to\.254±\.009\.254\\pm\.009\.

TheGemma\-3\-27B\-ITvariants underperform the hosted models\. Zero\-shotGemma\-3\-27B\-ITachieves high recall \(\.634 on validation and \.601 on test\), but this behavior is driven by severe over\-selection\. It predicts over 200 diffs per action item despite only 6\-8 being relevant on average\. Consequently, its test micro\-F1 is only \.037\. LoRA fine\-tuning makesGemma\-3\-27B\-ITmore selective and improves test micro\-F1 to \.083, while theGemma\-3\-27B\-ITembedding\-based MLP reaches \.069\. Although the LoRA\-tunedGemma\-3\-27B\-ITslightly outperforms the lexical baselines on validation,BM25andTF\-\-IDFperform better on test\. The overall difference in between performance on validation and test split is attributed to minor variations in distribution over the dataset splits\.

Overall, the results show that linking review action items to manuscript revisions requires more than surface\-level lexical similarity\.GPT\-5\.6\-Solprovides the strongest and most stable test performance\.

## 5Conclusions and Future Work

This paper presents Metag, a dataset to help build agentic capabilities to assist with meta\-reviewing in the peer\-reviewing process\. Metag is collected by scraping reviews from OpenReview, identifying action items, and linking action items to specific changes made to scientific manuscripts as part of the review\-rebuttal process\. The resulting dataset consists of 349 high\-quality human\-reviewed samples, and is larger than prior datasets that collect paired edits linked with scientific reviews\. Metag is benchmarked using lexical retrieval baselines and open and closed\-source LLMs as well\.GPT\-5\.6\-Solemerges as the most capable model on the task among the models evaluated, achieving an F1\-score close to 0\.40\.

Metag is collected from ICLR 2024 and the extension of the dataset to more venues is left to future work\. Additional steps include further benchmarking with open\-source models as context lengths of released models improve\. Metag relies on linking camera\-ready submissions on OpenReview with pre\-prints on arXiv\. Working closely with program organizers to make intermediate versions of under\-review documents available can further improve research in dialogue\-grounded iterative edits\.

## 6Limitations

Metag relies on accurately identifying the manuscript version submitted to the conference\. However, papers may undergo multiple revisions throughout the review and rebuttal process, and these intermediate versions are not always publicly available\. Collaboration with conference organizers to release timestamped intermediate revisions would enable more fine\-grained analysis of how manuscripts evolve in response to reviewer feedback and would support broader research on iterative scientific editing\. A further limitation is that Metag is derived exclusively from ICLR 2024\. Extending the dataset to additional venues, years, disciplines, and publication formats is therefore necessary to evaluate and improve its generalizability beyond a single machine\-learning conference\.

## References

- \[1\]Inc\. Artifex Software\.PyMuPDF, 2025\.Python bindings for MuPDF — a lightweight PDF, XPS, and eBook viewer\.
- \[2\]Melisa Bok and Andrew McCallum\.Openreview hosts record\-breaking aaai 2026 conference with pioneering ai review system\.OpenReview News Article, oct 2025\.
- \[3\]Mike D’Arcy, Alexis Ross, Erin Bransom, Bailey Kuehl, Jonathan Bragg, Tom Hope, and Doug Downey\.ARIES: A corpus of scientific paper edits made in response to peer reviews\.In Lun\-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), pages 6985–7001, Bangkok, Thailand, August 2024\. Association for Computational Linguistics\.
- \[4\]Wanyu Du, Vipul Raheja, Dhruv Kumar, Zae Myung Kim, Melissa Lopez, and Dongyeop Kang\.Understanding iterative revision from human\-written text\.In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors,Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), pages 3573–3590, Dublin, Ireland, May 2022\. Association for Computational Linguistics\.
- \[5\]Suzanne Fricke\.Semantic scholar\.Journal of the Medical Library Association: JMLA, 106\(1\):145, 2018\.
- \[6\]Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen\-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen\.Lora: Low\-rank adaptation of large language models\.arXiv preprint arXiv:2106\.09685, 2021\.
- \[7\]Rajesh Jayaram, Vincent Cohen\-Addad, Drew Tyler, Jieming Mao, Jon Schneider, Alekh Agarwal, Miroslav Dudik, Sharon Li, Martin Jaggi, Nihar B\. Shah, Katherine Gorman, and Gautam Kamath\.ICML experimental program using Google’s paper assistant tool \(PAT\)\.ICML Blog, January 2026\.Accessed: 2026\-03\-31\.
- \[8\]Chao Jiang, Wei Xu, and Samuel Stevens\.arxivedits: Understanding the human revision process in scientific writing\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9420–9435, 2022\.
- \[9\]Karen Spärck Jones\.A statistical interpretation of term specificity and its application in retrieval\.J\. Documentation, 60:493–502, 2021\.
- \[10\]Rodney Kinney, Chloe Anastasiades, Russell Authur, Iz Beltagy, Jonathan Bragg, Alexandra Buraczynski, Isabel Cachola, Stefan Candra, Yoganand Chandrasekhar, Arman Cohan, Miles Crawford, Doug Downey, Jason Dunkelberger, Oren Etzioni, Rob Evans, Sergey Feldman, Joseph Gorney, David Graham, Fangzhou Hu, Regan Huff, Daniel King, Sebastian Kohlmeier, Bailey Kuehl, Michael Langan, Daniel Lin, Haokun Liu, Kyle Lo, Jaron Lochner, Kelsey MacMillan, Tyler Murray, Chris Newell, Smita Rao, Shaurya Rohatgi, Paul Sayre, Zejiang Shen, Amanpreet Singh, Luca Soldaini, Shivashankar Subramanian, Amber Tanaka, Alex D\. Wade, Linda Wagner, Lucy Lu Wang, Chris Wilhelm, Caroline Wu, Jiangjiang Yang, Angele Zamarron, Madeleine Van Zuylen, and Daniel S\. Weld\.The semantic scholar open data platform, 2025\.
- \[11\]Ilia Kuznetsov, Osama Mohammed Afzal, Koen Dercksen, Nils Dycke, Alexander Goldberg, Tom Hope, Dirk Hovy, Jonathan K Kummerfeld, Anne Lauscher, Kevin Leyton\-Brown, et al\.What can natural language processing do for peer review?arXiv preprint arXiv:2405\.06563, 2024\.
- \[12\]Ilia Kuznetsov, Jan Buchmann, Max Eichler, and Iryna Gurevych\.Revise and resubmit: An intertextual model of text\-based collaboration in peer review\.Computational Linguistics, 48\(4\):949–986, December 2022\.
- \[13\]Weixin Liang, Zachary Izzo, Yaohui Zhang, Haley Lepp, Hancheng Cao, Xuandong Zhao, Lingjiao Chen, Haotian Ye, Sheng Liu, Zhi Huang, Daniel A\. McFarland, and James Y\. Zou\.Monitoring ai\-modified content at scale: A case study on the impact of chatgpt on ai conference peer reviews\.InInternational Conference on Machine Learning, 2024\.
- \[14\]Masato Mita, Keisuke Sakaguchi, Masato Hagiwara, Tomoya Mizumoto, Jun Suzuki, and Kentaro Inui\.Towards automated document revision: Grammatical error correction, fluency edits, and beyond\.InProceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications \(BEA 2024\), pages 251–265, 2024\.
- \[15\]OpenAI\.Previewing GPT\-5\.6 Sol: A next\-generation model\.[https://openai\.com/index/previewing\-gpt\-5\-6\-sol/](https://openai.com/index/previewing-gpt-5-6-sol/), June 2026\.Accessed 2026\-08\-17\.
- \[16\]Shruti Singh, Nandan Sarkar, and Arman Cohan\.SciDQA: A deep reading comprehension dataset over scientific papers\.In Yaser Al\-Onaizan, Mohit Bansal, and Yun\-Nung Chen, editors,Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20908–20923, Miami, Florida, USA, November 2024\. Association for Computational Linguistics\.
- \[17\]ssibb\.PDF\-Diff\-Viewer: A side\-by\-side, visual highlight, sync\-scroll pdf comparer written in python\.[https://github\.com/ssibb/PDF\-Diff\-Viewer](https://github.com/ssibb/PDF-Diff-Viewer), 2025\.
- \[18\]Anirudh Sundar, Jin Xu, William Gay, Christopher Richardson, and Larry Heck\.cpapers: A dataset of situated and multimodal interactive conversations in scientific papers\.Advances in Neural Information Processing Systems, 37:66283–66304, 2024\.
- \[19\]Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al\.Gemma 3 technical report\.arXiv preprint arXiv:2503\.19786, 2025\.
- \[20\]Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, SH Cai, Yuan Cao, Y Charles, HS Che, Cheng Chen, Guanduo Chen, et al\.Kimi k2\. 5: Visual agentic intelligence\.arXiv preprint arXiv:2602\.02276, 2026\.
- \[21\]The Open Group\.diff—compare two files\.[https://pubs\.opengroup\.org/onlinepubs/9799919799/utilities/diff\.html](https://pubs.opengroup.org/onlinepubs/9799919799/utilities/diff.html), 2024\.POSIX\.1\-2024\.
- \[22\]Andrew Trotman, Antti Puurula, and Blake Burgess\.Improvements to bm25 and language models examined\.InProceedings of the 19th Australasian Document Computing Symposium, ADCS ’14, page 58–65, New York, NY, USA, 2014\. Association for Computing Machinery\.
- \[23\]Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chenchen Ling, et al\.Deepseek\-v4: Towards highly efficient million\-token context intelligence\.arXiv preprint arXiv:2606\.19348, 2026\.
- \[24\]Qi Zeng, Mankeerat Sidhu, Ansel Blume, Hou Pong Chan, Lu Wang, and Heng Ji\.Scientific opinion summarization: Paper meta\-review generation dataset, methods, and evaluation\.InInternational Joint Conference on Artificial Intelligence, pages 20–38, 2024\.
- \[25\]Chao Zhang, Kexin Ju, Peter Bidoshi, Yu\-Chun Grace Yen, and Jeffrey M\. Rzeszotarski\.Friction: Deciphering writing feedback into writing revisions through llm\-assisted reflection\.InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI ’25, New York, NY, USA, 2025\. Association for Computing Machinery\.

## Appendix APrompt for Action Item Extraction

The prompt for extracting action items as provided toGemma\-3\-27B\-itis provided in Listing[1](https://arxiv.org/html/2608.20488#LST1)\.

1Youareanexpertatanalyzingscientificpaperreviews\.

2\#\#Task:

3Youwillbegiven:

41\.\*\*Review\*\*:Areviewofascientificpapersubmittedtoatoptierconferenceinmachinelearning

52\.\*\*Dialogue\*\*:Dialoguebetweentheauthorsofthepaperandthereviewers\.

6\#\#Instructions:

7Analyzethereviewandthedialoguetoidentifyactionitemsforauthorstofixintheirpaper\.

8Lookforstatementsliketyposandgrammaticalerrorsthatreviewerspointoutandasktobefixed,andstatementsfromtheauthorsstatingthattheywillfixcontentinthepaper\.Examplesofthisinclude"Wehaveupdatedthemanuscripttoclarifythispoint\.","Wehaveaddeddetailsintherevisedversion",lookforanywordsindicatingthattheauthorshavefixedthereviewerconcerns\.Yourresponsesshould\*\*only\*\*includethosecaseswithphraseslike"updatedthemanuscript","revisedthedescription",or"editedthepaper"indicatingthatthechangewasmade\.Prioritizethoseresponseswherethereisareferencetoaspecificsectioninthepapersuchas"Section3\.1","Equation4",etcwhichcanhelppointtowherethosechangesweremade\.Thetextshouldbedirectlyextractedfromthedialogues,donotwriteinthirdperson\.Forallexamplesofthis,makesureto\*\*paraphrase\*\*thereviewercommentandtheauthorresponse,donothallucinatecontent\.ProvideyouroutputasalistofJSONentitiesinthefollowingformat:

9\#\#Output:

10\{

11"entities":\[

12\{

13"comment":<CommentfromReviewaskingsomethingtobefixed\>,

14"response":<ResponsefromDialoguestatinghowtheissuewillbefixed\>,

15\},

16\{

17"comment":<AnothercommentfromRevieweaskingsomethingtobefixed\>,

18"response":<AnotherresponsefromtheDialoguestatinghowtheissuewillbefixed\>,

19\},

20\]

21\}

22\#\#ExampleInput:

23\#\#Review:

241\.Asthisresearchutilizedanamedentityrecognitionmodeltoextractkeywords,itispossiblethattheNERmodelcanextractprivacyinformationsuchaspatientnames\.Isthereanyfilteringorpostprocessingsteptoavoidthat?Inaddition,itisnotguaranteedthatNERsystemwillneverextractsensitivepatientinformation;forexample,iftheNERsystemincorrectlyextractsapatient’saddressasasymptom,thentheaddressmaybeleakedtoLLM\.Althoughitisveryrare,itisstillnecessarytocommentonthis\.

252\.AstheLLMalreadyprovidesapreliminarydecision,IamcuriousabouttheperformanceifweonlyfeedthepreliminarydecisionfromLLMtoSLM\.ItisworthknowingwhichpartoftheLLM\-generatedinformationimprovestheSLMmost\.

263\.TherelatedworksectionneedtodiscussmoreLLMapplicationintheclinicalarea,especiallytheknowledge\-enhancedLLMinclinicalsettings\.Forexample,paper"QualifyingChineseMedicalLicensingExaminationwithKnowledgeEnhancedGenerativePre\-trainingModel\."alsoutilizedexternalknowledgeforclinicalquestions\.

274\.ByaddingtheLLM\-generatedcontent,willthenewconcatenatedinputbetoolongandoutofthewordwindowinSLM?Howdoyoudealwiththelongcontentproblem?ByaddingtheLLM\-generatedcontent,willthenewconcatenatedinputbetoolongandoutofthewordwindowinSLM?Howdoyoudealwiththelongcontentproblem?

28\#\#Dialogue:

29Weappreciatetheinsightfulfeedbackandcommentsfromthereviewer\.Theirpositiveobservationsaboutthenoveltyandthoroughnessofourexperimentsareveryencouraging\.Wehaveaddressedyourconcernsinourresponse\.\\n\\n\*\*1\.Concernsonprivacypreservinginpracticalusage\.\*\*\\nThedataweutilizedinexperimentshavealreadyundergonepost\-processing;however,evenwell\-processeddatacannotbedirectlysharedwiththirdpartiesinareal\-hospitalsetting\.Here,weadoptNERmethodsdirectly,solelyforautomation,toshowthatLLMcanbeutilizedasamedicaldatabasetoqueryknowledgeunderprivacy\-restrictedscenarios\.Practically,wecanleveragede\-identificationmodelsandrulestoremovepersonalinformationandthenextractmedicalkeywordstoquerythird\-partyLLMsforauxiliaryknowledgegeneration\.Inthispaper,wetakeaninitialsteptodiscussthesignificantprivacy\-preservingsituationsinthemedicaldomainanddemonstratethepromisingresultsofutilizingLLMtoimproveSLMwhilemitigatingprivacyconcerns\.\\n\\n\*\*2\.QuestionaboutwhatSLMlearnsfordecisionmaking\.\*\*Wefeedpreliminarydecisions\(PD\)ascontextintoSLMwithbackboneBioLinkBert\-Baseonthreedatasets\.Threeseparaterunsforeachsettingareconductedandtheaverageresultsalongwiththestandarddeviationarereported\.TheresultsareshownintheTablebelow\.\|MEDQA\|HeadQA\|MEDMCQA

30SLMwPD\|47\.210\.31\|53\.641\.09\|45\.420\.17\|

31\|FTC\|50\.170\.42\|61\.350\.16\|49\.200\.45\|FTC,whichintegratesextensivemedicalknowledgeintothedecision\-making,showsaconsistentandsignificantimprovementovertheSLMthatonlyusesPDforcontext\.Thesefindingsunderscorethevaluablecontributionofleveragingcomprehensivemedicalknowledge,providedbyLLM,inenhancingthemedicaldecision\-makingcapabilities\.\*\*3\.SuggestionaboutrelatedworkinLLMapplicationintheclinicaldomain\.\*\*\\n\\nThanksforthesuggestionintherelatedwork\.Wewilladdthesuggestedworkintotherelatedworksectionintherevision\.\*\*4\.QuestionabutaddresslongmedicalcontextgeneratedbyLLM\.\*\*WeutilizetheFusion\-in\-Decoder\[1\]approachinourgeneraldomainexperiments\.Thisstrategyisalsoeffectiveforencodinglongcontexts\.Itworksbydividingtheinputintosmallerpassages,encodingeachoneseparately,andthencombiningtheencodedrepresentationsfordecision\-making\.\[1\]Izacardetal\.\(2020\)Leveragingpassageretrievalwithgenerativemodelsforopendomainquestionanswering\\nThanksforreplying\.Basedontheresponse,Iwouldliketokeepmyoriginalscore\.

32

33\#\#Output:

34\{

35"entities":\[

36\{

37"comment":"TherelatedworksectionneedtodiscussmoreLLMapplicationintheclinicalarea,especiallytheknowledge\-enhancedLLMinclinicalsettings\.Forexample,paper"QualifyingChineseMedicalLicensingExaminationwithKnowledgeEnhancedGenerativePre\-trainingModel\."alsoutilizedexternalknowledgeforclinicalquestions\.",

38"response":"Thanksforthesuggestionintherelatedwork\.Wewilladdthesuggestedworkintotherelatedworksectionintherevision\.",

39\},

40\]

41\}

Listing 2:Abbreviated diff\-classification instance\. Ellipses indicate omitted candidates\.
## Appendix BDataset Schema

Each entry corresponds to one \(action item, paper\) pair:

Table 4:Schema of a diff\-classification instance\.Thelabelsarray is aligned withall\_diffs:labels\[i\]istrueifall\_diffs\[i\]is relevant to the action item\. Therelevant\_diff\_indicesfield lists the indices of all true labels for convenience\. The schema is provided in Table[4](https://arxiv.org/html/2608.20488#A2.T4)\.

## Appendix CWindowing

To select the optimal window size of the number of diffs to pass in a single prompt,DeepSeek\-V4\-Prowas swept over values\{10,40,80,160,320\}\\\{10,40,80,160,320\\\}\. As the window size increases, the number of prompts to evaluate and cost of evaluation decreases\. On the other hand, the more diffs provided to the model, the more granularity available\. Figure[4](https://arxiv.org/html/2608.20488#A3.F4)reports the sweep over the parameters\. A window size of8080resulted in optimal performance on the validation set, and costs roughly the same as the cheapest setting\.

Figure 4:DeepSeek\-V4\-Pro validation performance and cost across window sizes\. Shaded regions show±1\\pm 1sample standard deviation across three runs\.
## Appendix DHyperparamter Settings

##### Optimalkkfor BM25 and TF\-IDF:

The optimal value forkkto be used in the BM25 and TF\-IDF experiments were obtained by sweeping values from 1 to 100 on the validation set\. The best value was obtained atk=6k=6, and can be observed in Figure[5](https://arxiv.org/html/2608.20488#A4.F5)\.

Figure 5:BM25 and TF–IDF micro\-F1 across retrieval cutoffs\. The dashed line marks the validation\-selectedk=6k=6\.
##### Gemma hyperparameters:

We evaluatedGemma\-3\-27B\-ITusing zero\-shot prompting, LoRA fine\-tuning, and an embedding\-based classifier\. For the generative experiments, candidate diffs were grouped into batches of 50 after removing trivial formatting changes\. Decoding used temperature0\.10\.1, nucleus\-sampling probabilityp=0\.95p=0\.95, and a maximum of 1,024 generated tokens\. Zero\-shot inference used a batch size of 16\.

For fine\-tuning, LoRA adapters were trained for three epochs with learning rate2×10−52\\times 10^\{\-5\}\. LoRA rankr=8r=8,α=16\\alpha=16, and dropout0\.050\.05\. The per\-device batch size was 1 with four gradient\-accumulation steps, giving an effective batch size of 4\. Input sequences were truncated to 4,096 tokens\.

For the embedding experiment, Gemma was loaded using 4\-bit NF4 quantization with double quantization andbfloat16computation\. Action items and diffs were independently truncated to 512 tokens, and their 5,376\-dimensional representations were obtained by mean\-pooling the final hidden layer over non\-padding tokens\. A three\-layer MLP received the concatenation of the action\-item embedding, diff embedding, and their element\-wise product\. The classifier used a hidden dimension of 256, dropout0\.30\.3, 10:1 negative sampling, and a positive\-class weight of 10\. Eight Nvidia A40 GPUs with 48GB GPU memory were used for LLM Experiments\.

Azure OpenAI was used for API access to closed\-source LLMs, and default settings for reasoning effort, random seed, and output length were used\. The total experimental cost of running the experiments detailed in this work using closed source LLMs totals USD 534\.57\.

Table 5:Inter\-annotator agreement over 575 doubly annotated action items\. Minimum and maximum values are computed across annotator pairs\.Table 6:Agreement on whether an action item was linked to any diff, computed over 575 doubly annotated items\.

## Appendix EAnnotator Details

Eleven annotators assisted with the labeling effort and the annotation effort was between 2 and 3 hours depending on annotator\. Each data sample was annotated by 2 annotators, with a sample making it into the final dataset if both annotators agreed on the relevant diff\. Table[5](https://arxiv.org/html/2608.20488#A4.T5)reports statistics on inter\-annotator agreement\. Annotators were compensated with co\-authorship of the publication\. In general, across 575 doubly annotated action items, annotators selected identical sets in3737% of cases\. Partial agreement was higher, with a mean Jaccard overlap of50\.150\.1%\. Binary agreement on whether an action item corresponded to any manuscript diff yielded a mean pairwise Cohen’sκ\\kappaof0\.3690\.369\.

Of the 575 items, both annotators selected at least one diff for 404 items, neither selected a diff for 59, and only one selected a diff for 112\. The variance in this metric indicates that there is subjectivity in the difference annotation in about2020% of cases\. That10\.310\.3% of cases did not have a relevant diff indicates that authors do not always commit to making changes though they are outlined during the review process, consistent with findings in prior work\[[3](https://arxiv.org/html/2608.20488#bib.bib3)\]\.

## Appendix FLLM Prompt

The prompt used in the LLM prompting approaches is provided in Listing[3](https://arxiv.org/html/2608.20488#LST3)\.

1Youareanexpertatanalyzingscientificpaperrevisions\.

2

3\#\#Task

4Youaregivenoneactionitemfromareviewer\-author

5discussion\(areviewerconcernplustheauthor’sresponse

6describingachangetheymade\)andabatchofPDFdiffs

7betweentheoriginalandrevisedversionsofthepaper\.

8

9Identifywhichdiffsimplementthechangedescribedinthe

10actionitem\.

11

12\#\#Guidelines

13\-Adiffisrelevantonlyifitscontentdirectlyimplements

14thedescribedchange\.

15\-Lookformatchingkeywords,section/table/figurereferences,

16orspecifictextnamedintheresponse\.

17\-MostdiffsareNOTrelevant\.Donotguess\.

18\-Formatting\-onlydiffs\(citationstyle,capitalization,line

19breaks\)arenotrelevantunlesstheactionitemis

20specificallyaboutformatting\.

21

22\#\#OutputFormat

23ReturnaJSONobjectwithasinglekey

24"relevant\_diff\_indices"holdingalistofdiff\_indexvalues,

25e\.g\.\{"relevant\_diff\_indices":\[3,17,42\]\}

26Return\{"relevant\_diff\_indices":\[\]\}ifnodiffinthisbatch

27isrelevant\.

28

29\#\#ActionItem

30Reviewerconcern:\[REVIEWERCONCERN\]

31Authorresponse:\[AUTHORRESPONSE\]

32

33\#\#Diffs

34\[diff\_index=INDEX\]tag=TAGmoved=MOVEDpage=PAGE

35context\_before:\[PRECEDINGCONTEXT\]

36old:\[ORIGINALTEXT\]

37new:\[REVISEDTEXT\]

38context\_after:\[FOLLOWINGCONTEXT\]

39

40\[REPEATEDFOREACHDIFFINTHEWINDOW\]

41

42\#\#Output

Listing 3:Zero\-shot prompt shared byGPT\-5\.6\-Sol,DeepSeek\-V4\-Pro, andKimi\-K2\.5\. Bracketed fields denote values populated for each action item and candidate window\.
## Appendix GLLM Usage

Language models were used to proofread the manuscript and assist with improving the overall writing\. Coding assistants were used to assist with code development\.

## NeurIPS Paper Checklist

1. 1\.Claims
2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
3. Answer:\[Yes\]
4. Justification: The abstract and the introduction state the paper’s contributions and scope\. See Section[1](https://arxiv.org/html/2608.20488#S1)
5. Guidelines: - •The answer\[N/A\]means that the abstract and introduction do not include the claims made in the paper\. - •The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations\. A\[No\]or\[N/A\]answer to this question will not be perceived well by the reviewers\. - •The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings\. - •It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper\.
6. 2\.Limitations
7. Question: Does the paper discuss the limitations of the work performed by the authors?
8. Answer:\[Yes\]
9. Justification: Provided in Section[6](https://arxiv.org/html/2608.20488#S6)\.
10. Guidelines: - •The answer\[N/A\]means that the paper has no limitation while the answer\[No\]means that the paper has limitations, but those are not discussed in the paper\. - •The authors are encouraged to create a separate “Limitations” section in their paper\. - •The paper should point out any strong assumptions and how robust the results are to violations of these assumptions \(e\.g\., independence assumptions, noiseless settings, model well\-specification, asymptotic approximations only holding locally\)\. The authors should reflect on how these assumptions might be violated in practice and what the implications would be\. - •The authors should reflect on the scope of the claims made, e\.g\., if the approach was only tested on a few datasets or with a few runs\. In general, empirical results often depend on implicit assumptions, which should be articulated\. - •The authors should reflect on the factors that influence the performance of the approach\. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting\. Or a speech\-to\-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon\. - •The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size\. - •If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness\. - •While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper\. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community\. Reviewers will be specifically instructed to not penalize honesty concerning limitations\.
11. 3\.Theory assumptions and proofs
12. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete \(and correct\) proof?
13. Answer:\[N/A\]
14. Justification: No theoretical proofs are presented in this work\.
15. Guidelines: - •The answer\[N/A\]means that the paper does not include theoretical results\. - •All the theorems, formulas, and proofs in the paper should be numbered and cross\-referenced\. - •All assumptions should be clearly stated or referenced in the statement of any theorems\. - •The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition\. - •Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material\. - •Theorems and Lemmas that the proof relies upon should be properly referenced\.
16. 4\.Experimental result reproducibility
17. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper \(regardless of whether the code and data are provided or not\)?
18. Answer:\[Yes\]
19. Justification: Hyperparameters and prompts are provided in the paper\.
20. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •If the paper includes experiments, a\[No\]answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not\. - •If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable\. - •Depending on the contribution, reproducibility can be accomplished in various ways\. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model\. In general\. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model \(e\.g\., in the case of a large language model\), releasing of a model checkpoint, or other means that are appropriate to the research performed\. - •While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution\. For example 1. \(a\)If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm\. 2. \(b\)If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully\. 3. \(c\)If the contribution is a new model \(e\.g\., a large language model\), then there should either be a way to access this model for reproducing the results or a way to reproduce the model \(e\.g\., with an open\-source dataset or instructions for how to construct the dataset\)\. 4. \(d\)We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility\. In the case of closed\-source models, it may be that access to the model is limited in some way \(e\.g\., to registered users\), but it should be possible for other researchers to have some path to reproducing or verifying the results\.
21. 5\.Open access to data and code
22. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
23. Answer:\[No\]
24. Justification: The code and models will be released post acceptance\. Dataset examples are provided in the paper and further examples can be made available during the review process but the final data and code will be released post acceptance\.
25. Guidelines: - •The answer\[N/A\]means that paper does not include experiments requiring code\. - • - •While we encourage the release of code and data, we understand that this might not be possible, so\[No\]is an acceptable answer\. Papers cannot be rejected simply for not including code, unless this is central to the contribution \(e\.g\., for a new open\-source benchmark\)\. - •The instructions should contain the exact command and environment needed to run to reproduce the results\. See the NeurIPS code and data submission guidelines \([https://neurips\.cc/public/guides/CodeSubmissionPolicy](https://neurips.cc/public/guides/CodeSubmissionPolicy)\) for more details\. - •The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc\. - •The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines\. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why\. - •At submission time, to preserve anonymity, the authors should release anonymized versions \(if applicable\)\. - •Providing as much information as possible in supplemental material \(appended to the paper\) is recommended, but including URLs to data and code is permitted\.
26. 6\.Experimental setting/details
27. Question: Does the paper specify all the training and test details \(e\.g\., data splits, hyperparameters, how they were chosen, type of optimizer\) necessary to understand the results?
28. Answer:\[Yes\]
29. Justification: See Appendix[A](https://arxiv.org/html/2608.20488#A1),[C](https://arxiv.org/html/2608.20488#A3),[D](https://arxiv.org/html/2608.20488#A4), \.
30. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them\. - •The full details can be provided either with the code, in appendix, or as supplemental material\.
31. 7\.Experiment statistical significance
32. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
33. Answer:\[Yes\]
34. Justification: Table[3](https://arxiv.org/html/2608.20488#S3.T3)reports standard deviation, Figure[4](https://arxiv.org/html/2608.20488#A3.F4)reports error bars\.
35. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The authors should answer\[Yes\]if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper\. - •The factors of variability that the error bars are capturing should be clearly stated \(for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions\)\. - •The method for calculating the error bars should be explained \(closed form formula, call to a library function, bootstrap, etc\.\) - •The assumptions made should be given \(e\.g\., Normally distributed errors\)\. - •It should be clear whether the error bar is the standard deviation or the standard error of the mean\. - •It is OK to report 1\-sigma error bars, but one should state it\. The authors should preferably report a 2\-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified\. - •For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range \(e\.g\., negative error rates\)\. - •If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text\.
36. 8\.Experiments compute resources
37. Question: For each experiment, does the paper provide sufficient information on the computer resources \(type of compute workers, memory, time of execution\) needed to reproduce the experiments?
38. Answer:\[Yes\]
39. Justification: See Appendix[D](https://arxiv.org/html/2608.20488#A4)\.
40. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage\. - •The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute\. - •The paper should disclose whether the full research project required more compute than the experiments reported in the paper \(e\.g\., preliminary or failed experiments that didn’t make it into the paper\)\.
41. 9\.Code of ethics
43. Answer:\[Yes\]
44. Justification: We have reviewed the code\.
45. Guidelines: - •The answer\[N/A\]means that the authors have not reviewed the NeurIPS Code of Ethics\. - •If the authors answer\[No\], they should explain the special circumstances that require a deviation from the Code of Ethics\. - •The authors should make sure to preserve anonymity \(e\.g\., if there is a special consideration due to laws or regulations in their jurisdiction\)\.
46. 10\.Broader impacts
47. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
48. Answer:\[Yes\]
49. Justification: Section[1](https://arxiv.org/html/2608.20488#S1)introduces the impact of the released dataset for the scientific community\.
50. Guidelines: - •The answer\[N/A\]means that there is no societal impact of the work performed\. - •If the authors answer\[N/A\]or\[No\], they should explain why their work has no societal impact or why the paper does not address societal impact\. - •Examples of negative societal impacts include potential malicious or unintended uses \(e\.g\., disinformation, generating fake profiles, surveillance\), fairness considerations \(e\.g\., deployment of technologies that could make decisions that unfairly impact specific groups\), privacy considerations, and security considerations\. - •The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments\. However, if there is a direct path to any negative applications, the authors should point it out\. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation\. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster\. - •The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from \(intentional or unintentional\) misuse of the technology\. - •If there are negative societal impacts, the authors could also discuss possible mitigation strategies \(e\.g\., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML\)\.
51. 11\.Safeguards
52. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse \(e\.g\., pre\-trained language models, image generators, or scraped datasets\)?
53. Answer:\[N/A\]
54. Justification:
55. Guidelines: - •The answer\[N/A\]means that the paper poses no such risks\. - •Released models that have a high risk for misuse or dual\-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters\. - •Datasets that have been scraped from the Internet could pose safety risks\. The authors should describe how they avoided releasing unsafe images\. - •We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort\.
56. 12\.Licenses for existing assets
57. Question: Are the creators or original owners of assets \(e\.g\., code, data, models\), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
58. Answer:\[Yes\]
59. Justification: Section[1](https://arxiv.org/html/2608.20488#S1)outlines the license as CDLA\-2\.0\.
60. Guidelines: - •The answer\[N/A\]means that the paper does not use existing assets\. - •The authors should cite the original paper that produced the code package or dataset\. - •The authors should state which version of the asset is used and, if possible, include a URL\. - •The name of the license \(e\.g\., CC\-BY 4\.0\) should be included for each asset\. - •For scraped data from a particular source \(e\.g\., website\), the copyright and terms of service of that source should be provided\. - •If assets are released, the license, copyright information, and terms of use in the package should be provided\. For popular datasets,[paperswithcode\.com/datasets](https://paperswithcode.com/datasets)has curated licenses for some datasets\. Their licensing guide can help determine the license of a dataset\. - •For existing datasets that are re\-packaged, both the original license and the license of the derived asset \(if it has changed\) should be provided\. - •If this information is not available online, the authors are encouraged to reach out to the asset’s creators\.
61. 13\.New assets
62. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
63. Answer:\[Yes\]
64. Justification: The dataset will be released post acceptance\. Examples of the dataset are provided in Figure[1](https://arxiv.org/html/2608.20488#S1.F1)and in Listing[1](https://arxiv.org/html/2608.20488#LST1)\.
65. Guidelines: - •The answer\[N/A\]means that the paper does not release new assets\. - •Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates\. This includes details about training, license, limitations, etc\. - •The paper should discuss whether and how consent was obtained from people whose asset is used\. - •At submission time, remember to anonymize your assets \(if applicable\)\. You can either create an anonymized URL or include an anonymized zip file\.
66. 14\.Crowdsourcing and research with human subjects
67. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation \(if any\)?
68. Answer:\[Yes\]
69. Justification: The annotation framework is provided in Figure[3](https://arxiv.org/html/2608.20488#S3.F3)and annotator information is provided in Section[3\.1\.5](https://arxiv.org/html/2608.20488#S3.SS1.SSS5.Px2)and Appendix[2](https://arxiv.org/html/2608.20488#LST2)\.
70. Guidelines: - •The answer\[N/A\]means that the paper does not involve crowdsourcing nor research with human subjects\. - •Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper\. - •According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector\.
71. 15\.Institutional review board \(IRB\) approvals or equivalent for research with human subjects
72. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board \(IRB\) approvals \(or an equivalent approval/review based on the requirements of your country or institution\) were obtained?
73. Answer:\[N/A\]
74. Justification:
75. Guidelines: - •The answer\[N/A\]means that the paper does not involve crowdsourcing nor research with human subjects\. - •Depending on the country in which research is conducted, IRB approval \(or equivalent\) may be required for any human subjects research\. If you obtained IRB approval, you should clearly state this in the paper\. - •We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution\. - •For initial submissions, do not include any information that would break anonymity \(if applicable\), such as the institution conducting the review\.
76. 16\.Declaration of LLM usage
77. Question: Does the paper describe the usage of LLMs if it is an important, original, or non\-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does*not*impact the core methodology, scientific rigor, or originality of the research, declaration is not required\.
78. Answer:\[Yes\]
79. Justification: See section[G](https://arxiv.org/html/2608.20488#A7)
80. Guidelines: - •The answer\[N/A\]means that the core method development in this research does not involve LLMs as any important, original, or non\-standard components\. - •Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described\.