SAGE: Schema-Guided LLMs for Grant Review

# 1 Introduction
Source: [https://arxiv.org/html/2609.20829](https://arxiv.org/html/2609.20829)
SAGE: Schema\-Guided LLMs for Grant Review

Erik Varapaev1, Andrei Chetvergov2,3, Stepan Ukolov2,3

Timofei Sivoraksha2,3, Alexander Evseev2,3, Sergey Bolovtsov2,3

1ITMO University, St\. Petersburg, Russia 2Russian Presidential Academy of National Economy and Public Administration, Moscow, Russia 3Ivannikov Institute for System Programming of the Russian Academy of Sciences, Moscow, Russia

465350@niuitmo\.ru \{chetvergov\-as,ukolov\-sd,sivoraksha\-ta\}@ranepa\.ru \{aevseev\-23\-01,bolovtsov\-sv\}@ranepa\.ru

###### Abstract

Grant reviewers must apply detailed criteria to application forms, budgets, and supporting documents while producing an assessment that colleagues can inspect\. We present SAGE—Schema\-guided Aspect\-based Grant Evaluation—a system that translates a grant rubric into structured checks and links its judgements to evidence from the application package\. We evaluate SAGE in two stages on 35 nonprofit grant applications\. A post\-factum comparison with 105 reviews from the original competition shows fair ordinal agreement \(κ=0\.29\\kappa=0\.29\)\. The foundation then conducted a criterion\-level re\-review after inspecting SAGE, producing 202 assessments\. In this assisted round, SAGE reachedκ=0\.58\\kappa=0\.58and outperformed a one\-prompt\-per\-criterion baseline \(κ=0\.33\\kappa=0\.33on the common subset\), with higher rank correlation and lower error\. A claim\-level audit further identifies confirmed, disputed, and unaddressed parts of the structured draft\. SAGE operationalizes the review methodology by producing a detailed, auditable draft for expert correction\.

Keywords:large language models, grant review, human\-in\-the\-loop, schema\-guided reasoning, expert alignment, aspect\-based evaluation

Grant\-making organizations distribute limited resources among many applicants\. Their reviewers rarely work from a single, tidy document\. A typical application combines a structured form with a budget, letters, reports, links, and other attachments\. Reviewers have to find the relevant evidence, apply a multi\-criteria rubric, assign scores, and explain those scores in a form that is useful to both the funder and the applicant\. Much of the work is repetitive, but the judgement is not: evidence is unevenly distributed and some criteria depend on how several parts of the proposal fit together\.

Large language models \(LLMs\) can help with reading and drafting, but a free\-form “review this proposal” prompt is a poor fit for the task\. The model may prioritize salient text over the rubric’s requirements\. A polished explanation can also hide omissions: the reviewer cannot easily tell which requirement was checked or where a factual claim came from\. Score agreement does not solve this problem, since a plausible score may still rest on an arithmetic error or on evidence borrowed from the wrong criterion\.

For grant review, usefulness depends less on whether the model can imitate the language of an expert than on whether its work can be inspected and corrected\. This places the task somewhat apart from benchmark grading and short\-answer evaluation\. The input is a document package, the criteria belong to a particular funder, and there is no single objective label against which every judgement can be checked\.

We call the system SAGE \(Schema\-guided Aspect\-based Grant Evaluation\)\. SAGE decomposes the funder’s rubric into criteria, subcriteria, and concrete aspects\. It asks the model to complete these checks before a criterion score and comment are assembled\. The output is meant to be edited: an expert can inspect the cited evidence, reject a claim, or revise a score before the review is finalized\.

The study follows two connected lines of inquiry\. The first is methodological: we examine how a funder’s review methodology can be represented as an executable schema that produces structured, evidence\-linked drafts\. The second is empirical: we use two evaluation rounds\. The first compares SAGE post\-factum with reviews from the original competition\. In the second, foundation reviewers revisit the same applications with access to SAGE and enter criterion\-level assessments; these assessments are used to compare SAGE with a practical criterion\-level prompting baseline and to audit the generated claims\.

For this purpose, we developed a grant\-review taxonomy containing 8 criteria, 37 subcriteria, and 228 atomic aspects, along with an implementation that turns the taxonomy into an editable review artifact\. The released materials include the system and evaluation code, run artifacts, anonymized score pairs, and the aggregate analysis\.111Artifact repository:[https://anonymous\.4open\.science/r/SAGE\-DEE5/README\.md](https://anonymous.4open.science/r/SAGE-DEE5/README.md)

![Refer to caption](https://arxiv.org/html/2609.20829v1/figures/sage_framework.png)Figure 1:SAGE framework\. The framework converts a long grant application package into structured, evidence\-grounded aspect judgements and aggregates them into an expert\-facing pre\-review for human verification\.
## 2Related Work

### 2\.1Automated Grant Review and Expert Variability

Grant peer review is known to have limited inter\-reviewer reliability\. Prior studies report low or moderate agreement between reviewers evaluating the same applications, motivating tools that can make criteria explicit and support reviewer calibration\[[1](https://arxiv.org/html/2609.20829#bib.bib1),[2](https://arxiv.org/html/2609.20829#bib.bib2)\]\. We therefore treat expert ratings as operational reference points and evaluate SAGE as a review\-support artifact whose outputs can be inspected, corrected, and compared with expert reasoning\.

Most computational work on grant review has focused on analysing reviewer comments or predicting categories in existing reviews\. Transformer\-based classifiers can identify content categories in grant reports when annotated data are available\[[3](https://arxiv.org/html/2609.20829#bib.bib3)\]\. Such systems, however, typically operate after human reviews have been written and require labelled corpora that many nonprofit grantmakers do not have\. SAGE instead supports the review process directly by producing a preliminary structured evaluation from the application package itself\.

### 2\.2LLM\-as\-a\-Judge and Rubric\-Guided Evaluation

The LLM\-as\-a\-Judge paradigm uses language models to evaluate outputs or documents against criteria\[[4](https://arxiv.org/html/2609.20829#bib.bib4),[5](https://arxiv.org/html/2609.20829#bib.bib5)\]\. Prometheus and Prometheus 2 demonstrate the value of fine\-grained, customized score rubrics and user\-defined evaluation criteria\[[12](https://arxiv.org/html/2609.20829#bib.bib12),[13](https://arxiv.org/html/2609.20829#bib.bib13)\]\. Generic judging remains risky for grant review because a global score and fluent explanation can conceal omitted criteria or unsupported evidence\.

Providing a scoring guide partially addresses this problem\. However, a prose rubric alone does not ensure that all items are checked or every claim is tied to evidence\. SAGE operationalizes the rubric as a typed schema\. This changes the unit of generation from a free\-form review to aspect records that can be parsed, aggregated, and audited\.

### 2\.3Long Document and Evidence\-Grounded Evaluation

Grant applications are long document packages\. Important evidence for one criterion may be distributed across the project description, budget, team profile, outcome table, and attached materials\. Retrieval\-augmented generation addresses long\-document tasks by selecting relevant passages before generation\[[6](https://arxiv.org/html/2609.20829#bib.bib6)\]\. SAGE uses a constrained variant: deterministic context assembly guided by the rubric\. Each subcriterion is mapped to expected evidence sources, reducing the risk that the model evaluates a criterion using irrelevant context\.

The work is also related to aspect\-based text analysis, where a document is decomposed into separate dimensions before interpretation\[[7](https://arxiv.org/html/2609.20829#bib.bib7)\]\. SAGE extends this idea from sentiment or summarization tasks to multi\-criteria expert review: each aspect becomes a typed judgement with a score, evidence, and recommendation\.

Table 1:Positioning of SAGE against related approaches\.

## 3Task and Data

### 3\.1Problem Formulation

LetDDbe an application package andRRbe a rubric\. The goal is to produce a preliminary review artifactYYthat contains scores, evidence\-grounded statements, weaknesses, strengths, and recommendations for each criterion\. The output should satisfy three requirements\.*Coverage*: every required rubric component should be addressed\.*Grounding*: generated claims should refer to evidence in the application package or explicitly mark missing evidence\.*Expert alignment*: criterion\-level scores and claim\-level statements should be comparable with human expert judgements\.

The target output is a draft review that an expert can audit and amend\. Final funding decisions may additionally depend on organizational context, portfolio composition, and information outside the submitted package\.

### 3\.2Rubric and Validation Data

The rubric contains eight high\-level criteria\. Each criterion is decomposed into subcriteria and atomic aspects\. An aspect corresponds to a concrete check, such as whether the problem is supported by evidence, whether the target group is specified, whether the budget is consistent with planned activities, or whether expected outcomes are measurable\. In total, the schema used in this study contains 8 criteria, 37 subcriteria, and 228 atomic aspects\.

The broader empirical archive contains 849 unique grant applications, 149 unique experts, and 2,547 expert–application reviews; it informed rubric analysis and schema design\. The validation sample contains 35 applications\. The first evaluation round uses three reviews per application from the original competition\. These records predate SAGE and yield 615 expert–application–criterion comparisons and 105 weighted application scores\. Because the original reviewers differed in their assessments and wrote comments for operational decision making, the foundation later conducted a second, criterion\-level review of the same applications\. Reviewers inspected the SAGE drafts and entered scores and comments for the applicable criteria, producing 202 numeric assessments\. Table[2](https://arxiv.org/html/2609.20829#S3.T2)summarizes both rounds\.

Table 2:Data used in schema design and validation\.The application materials and original expert comments are confidential\. The repository therefore contains anonymized pair\-level scores, aggregate tables, prompt and schema skeletons, evaluation scripts, and synthetic examples, but no raw applications or comments\.

### 3\.3Unit of Analysis

The two rounds use different units\. In the independent round, competition\-track weights produce 105 application scores; the criterion table uses all 615 original reports\. In the assisted round, the unit is an application–criterion pair, giving 202 expert assessments and 201 pairs with valid scores from both systems\. Bootstrap intervals are clustered by application\.

SAGE works at a still finer level: each criterion comment combines multiple claims supported by aspect records\. The claim audit compares 1,584 extracted SAGE claims with the assisted expert comments\. Score metrics describe rating alignment, while the audit shows which parts of the draft the expert comment supports, disputes, or leaves unaddressed\.

## 4SAGE Framework

### 4\.1Overview

Figure[1](https://arxiv.org/html/2609.20829#S1.F1)shows the four parts of SAGE: context assembly, aspect\-level generation, aggregation, and expert verification\. The rubric is represented as a schema that determines what the model must inspect\. This keeps the system close to the funder’s methodology and makes omitted checks visible\.

### 4\.2Aspect Schema

LetR=\{c1,…,c8\}R=\\\{c\_\{1\},\\ldots,c\_\{8\}\\\}be the set of high\-level criteria\. Each criterioncic\_\{i\}is decomposed into subcriteriaSi=\{si​1,…,si​k\}S\_\{i\}=\\\{s\_\{i1\},\\ldots,s\_\{ik\}\\\}, and each subcriterion contains atomic aspectsAi​j=\{ai​j​1,…,ai​j​m\}A\_\{ij\}=\\\{a\_\{ij1\},\\ldots,a\_\{ijm\}\\\}\. For each aspect, SAGE specifies relevant application sections, expected evidence, checks to be performed, and a structured output schema\.

A simplified aspect\-level record contains fields for the criterion, subcriterion, aspect identifier, score, confidence, evidence, strengths, weaknesses, recommendation, and rationale\. This structure makes the output machine\-readable and enables downstream aggregation and audit\. It also encourages explicit missing\-evidence markers and limits inference from project intent\.

### 4\.3Context Assembly

SAGE constructs the context for each criterion from predefined document layers\. It begins with the relevant form sections and then adds attachment text and link summaries when they are available\. Restricting the prompt to expected sources keeps it shorter and reduces accidental mixing between criteria\.

The mapping also makes the treatment of applications more consistent: the same subcriterion receives the same kinds of evidence each time\. If a section or attachment is absent, that absence is passed to the model instead of being filled by inference\. Because context selection is recorded separately from generation, a reviewer can distinguish a model error from an extraction or retrieval error\.

### 4\.4Schema\-Guided Generation

For each subcriterion, SAGE builds a prompt containing the expert role, criterion definition, aspect checklist, relevant context, evidence requirements, scoring rules, and required JSON schema\. The model is instructed to distinguish unsupported claims from missing information and to avoid external assumptions\. The output is parsed and validated\. Invalid JSON is repaired only at the syntactic level; unsupported or empty fields remain visible to the reviewer\.

The runtime can be summarized as:

1. 1\.collect criterion\-specific context from the application package;
2. 2\.instantiate the schema\-guided prompt for each subcriterion;
3. 3\.call the LLM and parse the JSON output;
4. 4\.validate required fields and retry syntactic failures;
5. 5\.aggregate aspect records into criterion\-level scores and comments;
6. 6\.compare criterion\-level results with expert judgements\.

### 4\.5Aggregation

Aspect\-level judgements are aggregated into subcriterion and criterion scores\. Letxi​jx\_\{ij\}be the normalized score for subcriterionjjunder criterioniiandwi​jw\_\{ij\}be its weight\. The criterion score is:

s​c​o​r​e​\(ci\)=∑jwi​j​xi​j∑jwi​j\.score\(c\_\{i\}\)=\\frac\{\\sum\_\{j\}w\_\{ij\}x\_\{ij\}\}\{\\sum\_\{j\}w\_\{ij\}\}\.\(1\)
The final explanation is assembled from structured aspect records linked to the original document\. This makes the final comment traceable to aspect\-level evidence\. A reviewer can therefore inspect whether a score is driven by a missing document, a weak causal mechanism, an inconsistent budget, or another specific aspect\.

### 4\.6Reviewer\-Facing Artifact

Reviewers receive both the criterion result and the records behind it\. Table[3](https://arxiv.org/html/2609.20829#S4.T3)shows a simplified synthetic record; an actual criterion comment draws on several records of this kind\. An expert can correct the cited evidence, reject a claim, change the score, or remove an irrelevant recommendation without rewriting the whole review\.

Table 3:Simplified synthetic example of a SAGE aspect record\.

## 5Evaluation Design

The evaluation has two rounds\. The independent round measures post\-factum alignment with decisions made during the original competition\. The assisted round captures a later foundation re\-review in which reviewers inspected SAGE before entering criterion scores and comments\. Figure[2](https://arxiv.org/html/2609.20829#S5.F2)summarizes the score and claim analyses for this second round\.

![Refer to caption](https://arxiv.org/html/2609.20829v1/figures/expert_alignment_evaluation.png)Figure 2:Assisted re\-review evaluation\. Reviewers inspected SAGE and then entered criterion\-level scores and comments; extracted SAGE claims were audited against those comments\.### 5\.1Two\-Round Expert Protocol

In Round 1, the reference records come from the original grant competition and predate the SAGE run\. Each application has three expert reviews\. Expert criterion scores were stored as proportions from 0\.2 to 1\.0 and mapped to the corresponding 1–5 categories\. SAGE criterion scores are rounded before the track\-specific competition weights are applied\. This produces 105 weighted expert–application pairs\. Depending on the track, each review contains five or six applicable criteria, giving 615 criterion comparisons\.

In Round 2, foundation reviewers revisited the 35 applications with the SAGE criterion drafts visible\. They recorded a score and free\-text assessment for each applicable criterion, yielding 202 application–criterion pairs\. We evaluate SAGE directly against this re\-review and run the criterion\-rubric baseline on the same applications\. One baseline call failed, leaving 201 common pairs for the paired system comparison\.

### 5\.2Score\-Level Metrics

We compare SAGE scores with expert scores using Spearman rank correlationρ\\rho\[[10](https://arxiv.org/html/2609.20829#bib.bib10)\], quadratic weighted Cohen’s kappaκ\\kappa\[[8](https://arxiv.org/html/2609.20829#bib.bib8)\], mean absolute error \(MAE\), and bias:

b​i​a​s=1n​∑i=1n\(y^i−yi\),M​A​E=1n​∑i=1n\|y^i−yi\|,bias=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\(\\hat\{y\}\_\{i\}\-y\_\{i\}\),\\qquad MAE=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\|\\hat\{y\}\_\{i\}\-y\_\{i\}\|,\(2\)wherey^i\\hat\{y\}\_\{i\}is a system score andyiy\_\{i\}is the corresponding expert score\. Quadratic weighting penalizes larger ordinal disagreements more strongly and yields agreement beyond chance on the 1–5 scale\[[9](https://arxiv.org/html/2609.20829#bib.bib9)\]\. Scores are rounded only forκ\\kappa; Spearmanρ\\rho, MAE, and bias use the continuous values\. We report percentile 95% confidence intervals from 10,000 application\-clustered bootstrap samples \(seed 20260713\); the same sampled applications are used for paired SAGE–baseline differences\.

### 5\.3Claim\-Level Audit

Two reviews can assign similar scores for different reasons\. An auxiliary pipeline therefore extracts claims from the SAGE criterion comments and matches them with the corresponding assisted expert comments\. Each SAGE claim is classified as*confirmed*,*disputed*, or*not commented*\. The distinction between confirmed and not commented prevents silence from being counted as agreement\. Matching is LLM\-assisted and is used as a content diagnostic rather than a manually adjudicated factuality benchmark\.

### 5\.4Run Configuration

The SAGE validation used vLLM with a Qwen\-family instruction model\[[11](https://arxiv.org/html/2609.20829#bib.bib11)\]\. Across 35 applications, it issued 2,974 subcriterion\-level requests\. All produced parseable outputs; one retry was recorded, and mean latency was 34\.1 seconds per request\. These values describe run behavior only\.

Table 4:Run\-level configuration and operational statistics\.

## 6Results

### 6\.1Round 1: Independent Historical Agreement

Table[5](https://arxiv.org/html/2609.20829#S6.T5)summarizes agreement on the 105 weighted application scores\. SAGE reaches Spearmanρ=0\.41\\rho=0\.41and quadratic weightedκ=0\.29\\kappa=0\.29\. The MAE is 0\.76 points on the five\-point scale, and the positive bias of\+0\.57\+0\.57shows that SAGE scores are generally more generous than the expert ratings\. The intervals reflect variation across the 35 applications\.

Table 5:Overall score\-level alignment; intervals bootstrap applications\.
### 6\.2Round 1: Criterion\-Level Results

Table[6](https://arxiv.org/html/2609.20829#S6.T6)reports agreement before criteria are combined into track\-weighted totals\. The four criteria used in every track show the most consistent ordinal agreement, with quadraticκ\\kappabetween 0\.29 and 0\.42\. Optional criteria have fewer observations and show a larger difference between rank association and exact scale calibration\.

Table 6:Agreement by criterion;κ\\kappais quadratic weighted\.The criterion results show why the weighted total is the appropriate primary score measure\. Some optional criteria preserve rank order while using a different part of the scale, which lowersκ\\kappadespite a positiveρ\\rho\. Perspective is both future\-facing and represented by only 15 comparisons\. For these criteria, the structured prompts primarily support expert judgement\.

### 6\.3Round 2: Assisted Re\-Review

The second\-round assessments provide a direct view of the intended workflow: reviewers first inspect the structured SAGE draft and then enter their criterion score and comment\. Across 202 application–criterion pairs, SAGE reaches Spearmanρ=0\.61\\rho=0\.61, quadratic weightedκ=0\.58\\kappa=0\.58, MAE=0\.49=0\.49, and bias\+0\.24\+0\.24\. Table[7](https://arxiv.org/html/2609.20829#S6.T7)shows that the strongest agreement occurs for result orientation and evidence quality, while the small perspective subset remains difficult\.

Table 7:SAGE alignment in the assisted re\-review;κ\\kappais quadratic weighted\.
### 6\.4Round 2: Comparison with the Criterion Baseline

The practical low\-cost baseline extracts predefined form sections and sends one high\-level criterion, its full methodology, and the extracted context in a single request\. It uses Qwen3\.6\-35B\-A3B, temperature 0\.2, guided JSON, and no additional LLM summarization of attachments or links\. Unlike SAGE, it does not decompose criteria into 37 subcriteria or 228 aspects\. The run produced 279 valid outputs from 280 calls\. One missing result leaves 201 application–criterion pairs on which the two systems can be compared against the same assisted assessments\. Two zero scores on the baseline’s native scale are mapped to one to match the expert 1–5 scale\.

Table 8:SAGE and criterion\-rubric baseline in the assisted round\.SAGE improves rank association by 0\.34 \[0\.20, 0\.49\] and quadraticκ\\kappaby 0\.25 \[0\.15, 0\.38\], while reducing MAE by 0\.23 \[0\.15, 0\.31\]\. All three application\-clustered intervals exclude zero\. The result supports schema\-guided decomposition as the stronger complete configuration for this assisted review setting\.

### 6\.5Round 2: Claim\-Level Audit

The assisted expert comments also support a finer analysis of the generated review\. The matcher extracts 1,584 claims from the SAGE criterion comments\. As Table[9](https://arxiv.org/html/2609.20829#S6.T9)shows, 51\.2% are confirmed by the corresponding expert comment, 18\.6% are disputed, and 30\.2% are not addressed\. The final category is kept separate because an omitted claim is not evidence of either agreement or disagreement\.

Table 9:Claim\-level audit against the assisted expert comments\.

## 7Verification Analysis

We manually inspected comments that the matcher flagged as disputed\. Five recurring patterns define useful verification targets \(Table[10](https://arxiv.org/html/2609.20829#S7.T10)\): evidence crossing criterion boundaries, inferences from missing information, numerical interpretation, generous treatment of partial support, and omitted context\. Mapping these patterns to specific aspect records turns a broad request to “check the model” into a focused review task\.

Table 10:Review patterns that benefit from targeted expert verification\.Structured records make these patterns easy to locate\. Criterion mixing is visible when a record cites a section associated with another criterion\. A strong conclusion paired with an empty or generic evidence field directs attention to its supporting source\. Amounts, counts, and percentages can be checked directly against extracted fields\. The records therefore provide an efficient verification layer alongside the final comment\.

## 8Discussion

The two rounds answer complementary questions\. The original competition records provide an independent external reference: SAGE reaches fair ordinal agreement despite being run only after those reviews were completed\. The assisted re\-review evaluates the system in its intended position inside the workflow\. Here SAGE reachesκ=0\.58\\kappa=0\.58andρ=0\.61\\rho=0\.61, with lower error and substantially stronger agreement than the criterion\-rubric baseline\.

The claim audit explains what the score metrics leave hidden\. More than half of the extracted claims are confirmed by the assisted expert comments, while disputed claims identify concrete correction points\. The schema therefore contributes both a broad first pass and a traceable interface through which reviewers can add domain and portfolio\-level judgement\.

The verification patterns also suggest concrete improvements\. Deterministic checks can support budget arithmetic and participant counts\. Separating missing evidence from negative evidence will make recommendations clearer, while claim\-level uncertainty can help reviewers prioritize attention\. Each addition builds on the schema without changing the expert\-facing workflow\.

## 9Deployment and Governance

In practice, SAGE belongs between document intake and the expert’s final assessment\. It prepares a consistent first pass and a checklist of evidence, strengths, weaknesses, and open questions\. The reviewer verifies the highlighted sources and incorporates contextual knowledge into the final assessment\.

The interface should keep evidence fields visible beside each judgement and highlight numerical claims for direct comparison with budgets and participant counts\. Claim\-level confidence can then serve as a practical cue for review priority, moving low\-confidence or high\-impact items to the top of the queue\.

With verified corrections, disagreements can become calibration material: a funder can inspect ambiguous rubric items and recurrent model overreach\. Such use requires access control, audit logs, and separation of model suggestions from final expert\-authored decisions\.

## 10Reproducibility and Data Availability

The artifact repository linked in the Introduction supports inspection while protecting confidential application text\. It contains the LNCS source, publication figures, prompt and schema skeletons, metric and bootstrap scripts, synthetic examples, anonymized values for both evaluation rounds, and aggregate tables\. The released package reproduces every reported score statistic and confidence interval\. Raw applications and comments are confidential partner data and cannot be distributed\.

## 11Limitations

The study covers 35 applications from one program and one rubric\. The second\-round assessments were entered after reviewers inspected SAGE, so the comparison with the post\-hoc baseline characterizes this assisted workflow and may include an exposure effect\. The two systems also differ in context assembly, model configuration, and computation, making their comparison one of complete configurations rather than an isolated schema ablation\. Claim matching is LLM\-assisted\. A controlled multi\-program study with randomized access to SAGE can separately measure its effect on review quality, consistency, and time\.

## 12Conclusion

SAGE augments criterion scores with structured evidence records and explicit correction points\. Its scores show fair ordinal agreement with 105 independent reviews from the original competition\. In the later assisted re\-review, SAGE reachesκ=0\.58\\kappa=0\.58and outperforms the criterion\-rubric baseline \(κ=0\.33\\kappa=0\.33\), with higher rank correlation and lower error\. The audit of 1,584 claims makes the remaining disagreements local and reviewable\.

Together, the two rounds support SAGE as a methodology\-aware review assistant: it aligns with prior expert assessments, provides a stronger criterion\-level draft than the low\-cost baseline in the assisted workflow, and preserves a clear role for expert correction\.

## References

- \[1\]Pier, E\.L\., Brauer, M\., Filut, A\., Kaatz, A\., Kessler, J\., Savoy, M\., Carnes, M\.: Low agreement among reviewers evaluating the same NIH grant applications\. Proceedings of the National Academy of Sciences 115\(12\), 2952–2957 \(2018\)
- \[2\]Mutz, R\., Bornmann, L\., Daniel, H\.\-D\.: Heterogeneity of inter\-rater reliabilities of grant peer reviews and its determinants\. PLOS ONE 7\(10\), e48509 \(2012\)
- \[3\]Okasa, G\., de León, A\., Strinzel, M\., Jorstad, A\., Milzow, K\., Egger, M\., Müller, S\.: A supervised machine learning approach for assessing grant peer\-review reports\. arXiv:2411\.16662 \(2024\)
- \[4\]Zheng, L\., Chiang, W\.\-L\., Sheng, Y\., et al\.: Judging LLM\-as\-a\-Judge with MT\-Bench and Chatbot Arena\. arXiv preprint arXiv:2306\.05685 \(2023\)
- \[5\]Gu, J\., Jiang, X\., Shi, Z\., et al\.: A survey on LLM\-as\-a\-Judge\. arXiv:2411\.15594 \(2024\)
- \[6\]Lewis, P\., Perez, E\., Piktus, A\., et al\.: Retrieval\-Augmented Generation for Knowledge\-Intensive NLP Tasks\. In: Advances in Neural Information Processing Systems \(2020\)
- \[7\]Zhang, W\., Li, X\., Deng, Y\., Bing, L\., Lam, W\.: A survey on aspect\-based sentiment analysis: tasks, methods, and challenges\. IEEE Transactions on Knowledge and Data Engineering \(2022\)
- \[8\]Cohen, J\.: Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit\. Psychological Bulletin 70\(4\), 213–220 \(1968\)
- \[9\]Landis, J\.R\., Koch, G\.G\.: The measurement of observer agreement for categorical data\. Biometrics 33\(1\), 159–174 \(1977\)
- \[10\]Spearman, C\.: The proof and measurement of association between two things\. The American Journal of Psychology 15\(1\), 72–101 \(1904\)
- \[11\]Yang, A\., Li, A\., Yang, B\., et al\.: Qwen3 Technical Report\. arXiv:2505\.09388 \(2025\)
- \[12\]Kim, S\., Shin, J\., Cho, Y\., et al\.: Prometheus: Inducing fine\-grained evaluation capability in language models\. arXiv:2310\.08491 \(2023\)
- \[13\]Kim, S\., Suk, J\., Longpre, S\., et al\.: Prometheus 2: An open source language model specialized in evaluating other language models\. arXiv:2405\.01535 \(2024\)