Beyond Score Prediction: LLM-Based Essay Scoring and Feedback Generation via Reinforcement Learning with Rubric Rewards

# Beyond Score Prediction: LLM-Based Essay Scoring and Feedback Generation via Reinforcement Learning with Rubric Rewards
Source: [https://arxiv.org/html/2607.19219](https://arxiv.org/html/2607.19219)
###### Abstract

Large language models \(LLMs\) have been widely applied to automated essay scoring \(AES\) and automated feedback generation \(AFG\)\. However, existing studies rely primarily on prompt engineering or supervised fine\-tuning, while systematic research on reinforcement learning \(RL\) post\-training and automated evaluation of feedback quality remains limited\. We propose RLAES, a unified LLM framework that jointly optimizes essay scoring and feedback generation through RL\. To make feedback quality measurable, interpretable, and usable for training, we introduce Rubric\-based Feedback Evaluation \(RFE\), an essay\-grounded feedback evaluation framework comprising 166 fine\-grained binary rubric items and an LLM\-as\-judge\. Building on RFE, we propose Adaptive Gated Feedback Optimization \(AGFO\), which activates rubric\-based feedback rewards on demand during RL, reducing evaluation overhead while improving feedback quality\. We also propose Adjacent Contrastive Reasoning \(ACR\) to improve ordinal score calibration by explicitly contrasting adjacent score levels\. Experimental results show that the RFE framework captures essay\-feedback consistency, exhibits strong pairwise discriminative power, and closely aligns with expert preferences\. On the ASAP benchmark, RLAES\-AGFO achieves the best scoring performance among LLM\-based methods \(QWK = 0\.803\), while maintaining feedback quality comparable to GPT\-5\.5 and avoiding the feedback degradation observed under score\-only RL\. Code and datasets are publicly available athttps://github\.com/hellomuyi/RLAES\.

## 1Introduction

Automated Essay Scoring \(AES\) and Automated Feedback Generation \(AFG\) are core applications of artificial intelligence in education\(Abudalfa and Barrot[2026](https://arxiv.org/html/2607.19219#bib.bib28)\)\. These systems use natural language processing, machine learning, and large language models \(LLMs\) to automatically score student essays and generate constructive feedback\. Together, AES and AFG can reduce educators’ grading workload while providing students with timely, personalized diagnostic feedback that helps them identify weaknesses and improve their writing over time\(Li and Collins[2026](https://arxiv.org/html/2607.19219#bib.bib36)\)\.

Research on AES dates back to the 1960s\(Sunet al\.[2025](https://arxiv.org/html/2607.19219#bib.bib32)\)\. Traditional AES approaches have evolved from regression models relying on hand\-crafted features\(Choiet al\.[2026](https://arxiv.org/html/2607.19219#bib.bib13)\)to deep learning paradigms that formulate AES as numerical classification or regression with automatic representation learning\. This evolution has progressed from convolutional neural networks \(CNNs\) and recurrent neural networks \(RNNs\)\(Taghipour and Ng[2016](https://arxiv.org/html/2607.19219#bib.bib30)\)to more advanced Transformer\-based pretrained language models, most notably the BERT family\(Li and Pan[2025b](https://arxiv.org/html/2607.19219#bib.bib16); Xieet al\.[2022](https://arxiv.org/html/2607.19219#bib.bib26)\)\. Recent LLM\-based approaches to AES can be broadly divided into prompt engineering and fine\-tuning\. Prompt engineering studies have extensively explored zero\-shot\(Seßleret al\.[2025](https://arxiv.org/html/2607.19219#bib.bib25); Suet al\.[2025](https://arxiv.org/html/2607.19219#bib.bib21)\)and few\-shot learning\(Huanget al\.[2026](https://arxiv.org/html/2607.19219#bib.bib10); Mansouret al\.[2024](https://arxiv.org/html/2607.19219#bib.bib8)\), prompts augmented with scoring rubrics\(Mansouret al\.[2024](https://arxiv.org/html/2607.19219#bib.bib8)\), chain\-of\-thought \(CoT\) reasoning\(Xiaoet al\.[2025](https://arxiv.org/html/2607.19219#bib.bib20)\), pairwise comparisons\(Choiet al\.[2026](https://arxiv.org/html/2607.19219#bib.bib13); Caiet al\.[2025](https://arxiv.org/html/2607.19219#bib.bib6); Shibata and Miyamura[2025](https://arxiv.org/html/2607.19219#bib.bib35)\), and multi\-agent frameworks\(Wanget al\.[2026](https://arxiv.org/html/2607.19219#bib.bib22)\)\. Fine\-tuning studies focus mainly on supervised fine\-tuning \(SFT\)\(Xiaoet al\.[2025](https://arxiv.org/html/2607.19219#bib.bib20); Ormerod and Kwako[2024](https://arxiv.org/html/2607.19219#bib.bib11); Johnsi and Kumar[2025](https://arxiv.org/html/2607.19219#bib.bib9)\)\. These studies have shown that fine\-tuned LLMs, even smaller open\-source ones, substantially outperform their extremely large proprietary counterparts equipped with a range of enhanced prompting strategies\.

Beyond score prediction, less attention has been paid to AFG, including the evaluation and optimization of feedback, despite the pedagogical importance of formative feedback\(Liuet al\.[2024](https://arxiv.org/html/2607.19219#bib.bib39)\)\. Small models that perform competitively on AES typically use BERT\-based encoder\-only architectures and cannot generate feedback without an additional decoder\. Existing LLM\-based feedback generation methods either directly prompt an LLM\(Sasakiet al\.[2026](https://arxiv.org/html/2607.19219#bib.bib5); Xiaoet al\.[2024](https://arxiv.org/html/2607.19219#bib.bib24); Ormerod and Kwako[2024](https://arxiv.org/html/2607.19219#bib.bib11)\)or perform SFT on feedback generated by strong reasoning models\(Xiaoet al\.[2025](https://arxiv.org/html/2607.19219#bib.bib20); Li and Pan[2025a](https://arxiv.org/html/2607.19219#bib.bib17)\)\. For feedback evaluation, a widely accepted framework for automated evaluation is still lacking\. Traditional feedback evaluation relies on annotated feedback datasets\(Sasakiet al\.[2026](https://arxiv.org/html/2607.19219#bib.bib5); Liuet al\.[2024](https://arxiv.org/html/2607.19219#bib.bib39)\)\. However, such annotations typically cover only a limited set of categories and are therefore ill\-suited to evaluating the rich feedback generated by LLMs\. LLM\-generated feedback is generally evaluated manually\(Xiaoet al\.[2025](https://arxiv.org/html/2607.19219#bib.bib20); Ormerod and Kwako[2024](https://arxiv.org/html/2607.19219#bib.bib11)\), but human evaluation is costly, difficult to scale, and difficult to convert into a reward signal for optimizing feedback quality\.Li and Pan \([2025a](https://arxiv.org/html/2607.19219#bib.bib17)\)propose an automated evaluation method that assesses feedback quality by computing the semantic similarity between the generated feedback and the scoring rubric descriptions associated with the gold score\.

Although LLMs have created new technical possibilities for AES and AFG, two challenges remain\. On the one hand, reinforcement learning \(RL\) post\-training remains underexplored in LLM\-based AES and AFG\. Current LLM approaches focus primarily on prompt engineering and SFT\. Prompt engineering is flexible but sensitive to prompt design, whereas SFT directly imitates gold scores\. RL may be better suited to score reasoning and alignment with scoring rubrics, but its effects on both scoring accuracy and feedback quality have not been systematically studied\.

On the other hand, AFG lacks effective and scalable methods for feedback evaluation and optimization\. LLM\-generated feedback is often rich and structurally complex, leading existing work to rely heavily on human assessment\(Xiaoet al\.[2025](https://arxiv.org/html/2607.19219#bib.bib20); Ormerod and Kwako[2024](https://arxiv.org/html/2607.19219#bib.bib11)\)\. Automated alternatives such as BERTScore measure the semantic similarity between generated feedback and scoring rubric descriptions associated with the gold score\(Li and Pan[2025a](https://arxiv.org/html/2607.19219#bib.bib17)\)\. However, the gap between general, high\-level scoring rubrics and essay\-specific feedback limits the effectiveness of this evaluation strategy\. The lack of an effective feedback evaluation method directly limits AFG optimization\.

To address these challenges, we propose RLAES, a unified LLM\-based RL post\-training framework for AES and AFG\. First, to fill the gap in effective feedback evaluation, we introduce Rubric\-based Feedback Evaluation \(RFE\), a prompt\-specific, essay\-grounded framework comprising 166 binary rubric items and an LLM\-as\-judge\. RFE provides a fine\-grained automated measure of feedback quality that can serve as an RL reward\. Building on RFE, we propose Adaptive Gated Feedback Optimization \(AGFO\), which activates rubric\-based feedback rewards on demand during RL\. AGFO reduces LLM\-as\-judge overhead while jointly optimizing scoring accuracy and feedback quality, thereby preventing the feedback degradation caused by score\-only RL\. The resulting RLAES\-AGFO model achieves the best performance among LLM\-based AES methods\. In addition, we propose Adjacent Contrastive Reasoning \(ACR\), a general strategy that explicitly guides the model to contrast adjacent score levels\. ACR improves sensitivity to fine\-grained quality differences and alleviates adjacent confusion in ordinal classification\. Our main contributions are summarized as follows:

- •We extend LLM\-based AES and AFG methodology beyond prompt engineering and SFT to RL post\-training\. The resulting RLAES\-AGFO jointly optimizes scoring accuracy and feedback quality and achieves the best performance among LLM\-based methods\.
- •We introduce RFE, an essay\-grounded framework that fills the gap in fine\-grained feedback evaluation for AFG\.
- •We propose ACR, a general strategy for ordinal classification tasks\. ACR enables models to perceive and distinguish adjacent score levels, improving scoring accuracy\.

## 2Preliminaries

### 2\.1Task Definition

Let𝒟=\{\(ei,si\)\}i=1N\\mathcal\{D\}=\\\{\(e\_\{i\},s\_\{i\}\)\\\}\_\{i=1\}^\{N\}denote a dataset, whereeie\_\{i\}represents theii\-th essay together with its scoring context, such as the writing prompt, source text, and scoring rubric guidelines, andsis\_\{i\}is its corresponding holistic score\. The objective is to train a model on𝒟\\mathcal\{D\}that generalizes well to unseen essays, enabling accurate score prediction for AES and reliable feedback generation for AFG\. We focus on prompt\-specific AES and AFG, where the writing prompts used for training and testing are drawn from the same set\. We define feedback \(or explanation\) as all components of an AES system’s output other than the score prediction\. We structure the output such that the model first generates feedback and then predicts a score\. In this setting, the feedback can be viewed as a CoT\-like intermediate rationale before the final score\.

### 2\.2RL Post\-Training

Reinforcement learning with verifiable rewards \(RLVR\) relies on explicit, verifiable binary rules as reward signals, such as output correctness\. DeepSeek\-R1\(Guo and others[2025](https://arxiv.org/html/2607.19219#bib.bib34)\)demonstrated its effectiveness in improving reasoning capabilities\. Related algorithms such as Group Relative Policy Optimization \(GRPO\)\(Shaoet al\.[2024](https://arxiv.org/html/2607.19219#bib.bib33)\)have been widely applied to domains with checkable answers, including mathematics and code\.

For difficult\-to\-verify domains such as essay feedback evaluation, however, feedback quality cannot be determined by a single binary rule and instead requires multidimensional assessment\. Directly applying RLVR in such settings would yield an overly coarse reward signal\. Conversely, reinforcement learning from human feedback \(RLHF\) can use human preferences to handle open\-ended outputs, but preference judgments are highly subjective and may induce reward hacking\. Rubrics as Rewards \(RaR\)\(Gunjalet al\.[2025](https://arxiv.org/html/2607.19219#bib.bib31)\)decomposes evaluation criteria into individually assessable rubric items, bridging the gap between strictly verifiable rewards and subjective human preferences\. It provides structured rewards for open\-ended tasks and is directly compatible with on\-policy RL algorithms such as GRPO\.

Specifically, given an input promptxxand a model responsey^\\hat\{y\}, the rubric reward is defined as

r​\(x,y^\)=∑j=1kwj​cj​\(x,y^\)∑j=1kwj,r\(x,\\hat\{y\}\)=\\frac\{\\sum\_\{j=1\}^\{k\}w\_\{j\}c\_\{j\}\(x,\\hat\{y\}\)\}\{\\sum\_\{j=1\}^\{k\}w\_\{j\}\},\(1\)wherekkis the number of rubric items,wj\>0w\_\{j\}\>0is the weight of itemjj, andcj:\(x,y^\)↦\{0,1\}c\_\{j\}:\(x,\\hat\{y\}\)\\mapsto\\\{0,1\\\}indicates whether the responsey^\\hat\{y\}to the model inputxxsatisfies that item\. An LLM\-as\-judge assigns each indicatorcjc\_\{j\}independently, and all items\{\(wj,cj\)\}j=1k\\\{\(w\_\{j\},c\_\{j\}\)\\\}\_\{j=1\}^\{k\}define the rubrics\.

## 3Approach

### 3\.1Rubric\-Based Feedback Evaluation \(RFE\) Framework

Inspired by Deep Research Bench II\(Liet al\.[2026](https://arxiv.org/html/2607.19219#bib.bib41)\)and RaR\(Gunjalet al\.[2025](https://arxiv.org/html/2607.19219#bib.bib31)\), we propose the Rubric\-based Feedback Evaluation \(RFE\) framework, which supports automated feedback evaluation and converts feedback quality into a scalar reward for RL training\. RFE comprises prompt\-specific, checklist\-style rubrics and an LLM\-as\-judge\. We will release these resources with the code\. Figure[1](https://arxiv.org/html/2607.19219#S3.F1)illustrates the construction pipeline\.

![Refer to caption](https://arxiv.org/html/2607.19219v1/x1.png)Figure 1:Construction of the RFE rubrics\.#### Reference Feedback Extraction\.

We use Claude\-Opus\-4\.7 to extract handwritten essays and teacher feedback from official ASAP materials\(Hamneret al\.[2012](https://arxiv.org/html/2607.19219#bib.bib29)\)\. Manual verification yields 92 essay–feedback pairs across eight writing prompts\. We remove feedback that is overly brief, merely repeats scoring rubric guidelines without essay\-specific evidence, or omits too many trait domains, retaining 50 high\-quality samples as pseudo\-gold feedback\. Because teacher feedback has practical limitations, such as omitting revision suggestions or providing only a single high\-level assessment, we refine these samples through LLM rewriting, manual calibration, and expert validation, producing 50 gold feedback samples\. To construct negative examples, we order the samples by prompt and score and pair each essay with feedback from an adjacent\-scoring essay under the same prompt, yielding 88 perturbed feedback samples\.

#### Rubric Generation and Iteration\.

The Main Agent generates initial rubrics with manually adjusted top\-level dimensions and fine\-grained rubric items\. It then evaluates the rubrics on the gold feedback and produces an analysis report\. Five Rubric Refinement Agents—GPT\-5\.5, Gemini\-3\.1\-Pro, Claude\-Opus\-4\.8, DeepSeek\-V4\-Pro, and GLM\-5\.1—independently propose revisions\. The Main Agent aggregates these proposals and updates the rubrics until the agents’ evaluations converge\. After manual revision, the final rubrics, denoted byℛfb\\mathcal\{R\}\_\{\\mathrm\{fb\}\}, contain 166 binary items across four dimensions: Coverage, Evidence, Faithfulness, and Safety\.

#### LLM\-as\-Judge Evaluation\.

Given an essay–feedback pair\(ei,fi\)\(e\_\{i\},f\_\{i\}\), an LLM\-as\-judge independently determines whether each rubric item is satisfied\. The RFE score is then computed using Equation[1](https://arxiv.org/html/2607.19219#S2.E1)\. We assign equal weight to all items \(wj=1w\_\{j\}=1\); thus, each dimension’s contribution is proportional to its number of items\.

### 3\.2Adaptive Gated Feedback Optimization \(AGFO\)

We use GRPO to jointly optimize AES and AFG\. The reward function consists primarily of a score reward and a feedback reward\. Given a student essay and its contextee, the gold scoress, the model\-predicted scores^\\hat\{s\}, the model\-generated feedbackf^\\hat\{f\}, and the feedback rubricsℛfb\\mathcal\{R\}\_\{\\mathrm\{fb\}\}, the total rewardrtotalr\_\{\\mathrm\{total\}\}is defined as

rtotal=rscore​\(s,s^\)\+λf⋅rfeedback​\(e,ℛfb,f^\),r\_\{\\mathrm\{total\}\}=r\_\{\\mathrm\{score\}\}\(s,\\hat\{s\}\)\+\\lambda\_\{f\}\\cdot r\_\{\\mathrm\{feedback\}\}\(e,\\mathcal\{R\}\_\{\\mathrm\{fb\}\},\\hat\{f\}\),\(2\)whereλf\\lambda\_\{f\}is a hyperparameter controlling the weight of the feedback reward\. The score reward is defined as

rscore​\(s,s^\)=−\|s−s^\|α​M,r\_\{\\mathrm\{score\}\}\(s,\\hat\{s\}\)=\-\\frac\{\\left\|s\-\\hat\{s\}\\right\|\}\{\\alpha\\,M\},\(3\)whereM=smax−sminM=s\_\{\\max\}\-s\_\{\\min\}is the prompt\-specific score range andα\\alphais a scaling factor\. RFE suppliesrfeedbackr\_\{\\mathrm\{feedback\}\}by evaluatingf^\\hat\{f\}againstℛfb\\mathcal\{R\}\_\{\\mathrm\{fb\}\}\.

However, invoking an LLM\-as\-judge at every optimization step incurs prohibitive time and computational costs\. We therefore propose Adaptive Gated Feedback Optimization \(AGFO\), which adaptively controls the activation of the feedback reward during training\. Specifically, everynnsteps, the system invokes the LLM\-as\-judge to compute the current model’s feedback rewardrfeedbackr\_\{\\mathrm\{feedback\}\}on the batch and jointly optimizes scoring and feedback\. If this reward is less than or equal to the predefined thresholdτ\\tau, the gate remains open, and the judge is queried and the resulting feedback reward is included in the total reward at every subsequent step untilrfeedbackr\_\{\\mathrm\{feedback\}\}exceedsτ\\tau\. Otherwise, the gate is closed between periodic checks, and the total reward contains only the score reward\. Pseudocode for AGFO is provided in Algorithm[1](https://arxiv.org/html/2607.19219#alg1)\.

Notably, AGFO unifies two special cases via the thresholdτ\\tau\. Whenτ=0\\tau=0, the feedback reward is activated only everynnsteps, reducing AGFO to periodic feedback optimization\. Whenτ=1\\tau=1, it is activated at every step, reducing AGFO to full feedback optimization\.

We denote the RL configuration using only the score reward asRLAES\-Scoreand the configuration incorporating AGFO asRLAES\-AGFO\.

Algorithm 1Adaptive Gated Feedback Optimization \(AGFO\)0:policy

πθ\\pi\_\{\\theta\}; dataset

𝒟\\mathcal\{D\}; rubrics

ℛfb\\mathcal\{R\}\_\{\\text\{fb\}\}; judge

𝒥\\mathcal\{J\}; period

nn; threshold

τ\\tau; weight

λf\\lambda\_\{f\}; total steps

TT
0:optimized policy

πθ\\pi\_\{\\theta\}
1:

r¯fb←0\\bar\{r\}\_\{\\text\{fb\}\}\\leftarrow 0
2:for

t←1t\\leftarrow 1to

TTdo

3:Sample mini\-batch

ℬ⊂𝒟\\mathcal\{B\}\\subset\\mathcal\{D\}
4:Generate responses

\{\(f^i,s^i\)\}∼πθ\(⋅∣ei\)\\\{\(\\hat\{f\}\_\{i\},\\hat\{s\}\_\{i\}\)\\\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid e\_\{i\}\),

ei∈ℬe\_\{i\}\\in\\mathcal\{B\}
5:if

tmodn≠0t\\bmod n\\neq 0and

r¯fb\>τ\\bar\{r\}\_\{\\text\{fb\}\}\>\\tauthen

6:

rifb←0,∀ir^\{\\text\{fb\}\}\_\{i\}\\leftarrow 0,\\ \\forall i\{gate closed\}

7:else

8:

rifb←rfeedback​\(ei,ℛfb,f^i\)r^\{\\text\{fb\}\}\_\{i\}\\leftarrow r\_\{\\text\{feedback\}\}\(e\_\{i\},\\mathcal\{R\}\_\{\\text\{fb\}\},\\hat\{f\}\_\{i\}\)\{gate open, query

𝒥\\mathcal\{J\}\}

9:Update

r¯fb\\bar\{r\}\_\{\\text\{fb\}\}with

\{rifb\}\\\{r^\{\\text\{fb\}\}\_\{i\}\\\}
10:endif

11:for all

i∈ℬi\\in\\mathcal\{B\}do

12:

ri←riscore\+λf​rifbr\_\{i\}\\leftarrow r^\{\\text\{score\}\}\_\{i\}\+\\lambda\_\{f\}\\,r^\{\\text\{fb\}\}\_\{i\}
13:endfor

14:Update

πθ\\pi\_\{\\theta\}via GRPO using

\{ri\}\\\{r\_\{i\}\\\}
15:endfor

16:return

πθ\\pi\_\{\\theta\}

### 3\.3Adjacent Contrastive Reasoning \(ACR\)

Automated essay scoring can be formulated as an ordinal classification problem: score levels are ordered labels rather than independent classes\. The main difficulty is determining whether an essay belongs to a given score level or an adjacent lower or higher level\. Standard prompting methods ask an LLM to produce a holistic evaluation and score but do not explicitly require comparisons at score boundaries\. This omission may make the model more prone to predicting a score level adjacent to the gold score—a phenomenon we calladjacent confusion\.

To alleviate this issue, we propose Adjacent Contrastive Reasoning \(ACR\), inspired by the anchoring\-and\-adjustment heuristic in human judgment\(Tversky and Kahneman[1974](https://arxiv.org/html/2607.19219#bib.bib43)\)\. As illustrated in Figure[2](https://arxiv.org/html/2607.19219#S3.F2), ACR treats the adjacent lower and higher scores around a predicted score as local hard\-negative labels and requires the model to explicitly answer two contrastive questions before producing its final score: Why should the essay not receive the adjacent lower/higher score? This local counterfactual comparison encourages the model to identify evidence that distinguishes adjacent score levels instead of producing only a generic holistic assessment\. ACR improves scoring in three ways\. First, it transforms scoring from direct score mapping into local boundary discrimination, thereby focusing the model’s attention on fine\-grained differences between adjacent scores\. Second, treating adjacent levels as hard negatives directs attention toward more discriminative textual evidence\. Third, considering both adjacent directions provides a bidirectional calibration signal, helping reduce systematic overestimation or underestimation in the ordinal score space\.

ACR differs from existing few\-shot prompting and pairwise\-comparison methods, which introduce external reference essays and assist scoring by comparing the relative quality of a target essay against the reference\. In contrast, ACR requires no external reference samples and adds only local comparisons with adjacent score levels\. We also evaluate ACR\-Lower, a one\-sided variant that retains only the question of why the essay should not receive the adjacent lower score\. As a general strategy, ACR may be extended to other ordinal classification problems\.

![Refer to caption](https://arxiv.org/html/2607.19219v1/x2.png)Figure 2:ACR for ordinal score calibration\.

## 4Experimental Setup

### 4\.1Dataset

We evaluate on the Automated Student Assessment Prize \(ASAP\) dataset\(Hamneret al\.[2012](https://arxiv.org/html/2607.19219#bib.bib29)\), a widely adopted benchmark for English AES\. The dataset comprises 12,978 essays from three writing types and eight writing prompts \(Table[1](https://arxiv.org/html/2607.19219#S4.T1)\)\. FollowingTaghipour and Ng \([2016](https://arxiv.org/html/2607.19219#bib.bib30)\), we conduct five\-fold cross\-validation using their standardized train\-dev\-test splits\. With one minor modification, we merge the training data from all eight prompts within the same fold and perform a single joint training run across prompts, rather than training a separate model for each prompt\. This design simplifies the experimental procedure, avoids model fragmentation, and aligns with efficient parameter sharing and unified model serving in the LLM era\. We provide detailed results in Appendix[A](https://arxiv.org/html/2607.19219#A1), showing that prompt joint training incurs negligible performance degradation across model families, with QWK decreasing by no more than 0\.005\.

Table 1:Statistics of the ASAP dataset \(Avg\. Len\.: average essay length; Arg\.: argumentative; SD: source\-dependent; Narr\.: narrative\)\.
### 4\.2Evaluation Metrics

We use quadratic weighted kappa \(QWK\), the most widely used metric in AES and the sole evaluation metric of the ASAP competition, to evaluate score prediction\. QWK measures agreement between model predictions and human\-assigned gold scores\.

For AFG evaluation, we employ our proposed RFE framework with GPT\-5\.5 as the LLM judge\. Evaluating one essay–feedback pair against all 166 rubric items costs approximately $0\.12\. Due to budget constraints, the feedback evaluation is conducted on fold 0\. Each fold covers the complete ASAP dataset, and folds differ only in how the dataset is split\. Therefore, evaluation on a single fold remains sufficiently representative and valid\.

### 4\.3Baseline Models

We group the baselines into two categories: LLM prompt engineering and LLM post\-training\.

#### LLM Prompt Engineering\.

These methods directly prompt LLMs to perform scoring without updating model parameters\. We include GPT\-3\.5 with rubric\-guided one\-shot prompting\(Mansouret al\.[2024](https://arxiv.org/html/2607.19219#bib.bib8)\), GPT\-4 with rubric prompting\(Li and Pan[2025a](https://arxiv.org/html/2607.19219#bib.bib17)\), GPT\-4 with few\-shot prompting\(Xiaoet al\.[2025](https://arxiv.org/html/2607.19219#bib.bib20)\), and Llama\-3\-8B\-Instruct\(Ormerod and Kwako[2024](https://arxiv.org/html/2607.19219#bib.bib11)\)\. We also evaluate vanilla prompting with DeepSeek\-V4\-Pro\(DeepSeek\-AI[2026](https://arxiv.org/html/2607.19219#bib.bib4)\)and Qwen3\.7\-Max\(Qwen Team[2026b](https://arxiv.org/html/2607.19219#bib.bib2)\), which serve as the base settings for our ACR variants\.

#### LLM Post\-Training\.

These methods adapt LLMs to AES by updating model parameters\. Existing baselines in this category are primarily based on SFT, including fine\-tuned GPT\-3\.5\(Xiaoet al\.[2024](https://arxiv.org/html/2607.19219#bib.bib24)\), the Dual\-Process Model\(Xiaoet al\.[2025](https://arxiv.org/html/2607.19219#bib.bib20)\), fine\-tuned Llama\-3\.2\-1B\-Instruct\(Johnsi and Kumar[2025](https://arxiv.org/html/2607.19219#bib.bib9)\), fine\-tuned Llama\-3\-8B\-Instruct\(Ormerod and Kwako[2024](https://arxiv.org/html/2607.19219#bib.bib11)\), and RTS\(Caiet al\.[2025](https://arxiv.org/html/2607.19219#bib.bib6)\)\.

For AFG, prior work remains limited, and unavailable code and differing tasks hinder fair comparison\. We therefore use GPT\-5\.5, prompted with role\-playing instructions and scoring rubric guidelines, as a strong feedback\-generation baseline\.

### 4\.4Implementation Details

In our post\-training experiments, we use Qwen3\.5\-9B\(Qwen Team[2026a](https://arxiv.org/html/2607.19219#bib.bib1)\)as the base model and adopt low\-rank adaptation \(LoRA\) for parameter\-efficient fine\-tuning\. RLAES is trained for 25 epochs with a learning rate of2\.0×10−52\.0\\times 10^\{\-5\}and a warmup ratio of 0\.1\. We use a cosine learning\-rate scheduler with a minimum learning\-rate ratio of 0\.1 and set the per\-device training batch size to 8\. For GRPO, we set the Kullback–Leibler \(KL\) regularization coefficient to 0 and apply a length penalty similar to DAPO\(Yuet al\.[2025](https://arxiv.org/html/2607.19219#bib.bib38)\)\. During rollout generation, we sample 8 responses for each query at a temperature of 0\.9\. All experiments are conducted on eight NVIDIA H20/A100 GPUs with DeepSpeed ZeRO\-3\.

The feedback reward weightλf\\lambda\_\{f\}is set to 1, and the score\-interval scaling factorα\\alphais set to 0\.1\. For AGFO, the gating periodnnis set to 30, and the feedback reward thresholdτ\\tauis set to 0\.8 based on an offline estimate\. For the final feedback evaluation, we use GPT\-5\.5\(OpenAI[2026](https://arxiv.org/html/2607.19219#bib.bib3)\)as the LLM\-as\-judge\. During RL training, feedback rewards are computed using DeepSeek\-V4\-Pro\(DeepSeek\-AI[2026](https://arxiv.org/html/2607.19219#bib.bib4)\)to reduce computational and financial costs\. The temperature is set to 0 for all judge models\. Unless otherwise noted, GPT\-5\.5 refers togpt\-5\.5\-0424\-global\.

## 5Results and Analysis

### 5\.1Results on Score Prediction

Table 2:Scoring performance \(QWK\) across the eight ASAP prompts\.#### LLM Prompt Engineering and ACR\.

Table[2](https://arxiv.org/html/2607.19219#S5.T2)reports QWK for LLM prompt engineering and post\-training methods\. GPT\-4 \(w/ rubric\) and GPT\-4 \(few\-shot\) achieve average QWK values of 0\.415 and 0\.474, respectively, substantially below post\-trained LLMs\. This gap highlights the difficulty of directly prompting LLMs for AES\.

For DeepSeek\-V4\-Pro, average QWK increases from 0\.430 with vanilla prompting to 0\.462 with ACR and 0\.471 with ACR\-Lower\. For Qwen3\.7\-Max, the corresponding values are 0\.446, 0\.473, and 0\.506\. As shown in Figure[3](https://arxiv.org/html/2607.19219#S5.F3), both models systematically underestimate essay scores under vanilla prompting, whereas the one\-sided ACR\-Lower mitigates this bias by prompting them to identify the essay’s strengths\. These results demonstrate that ACR improves the scoring accuracy of base models\.

#### Comparison between SFT and RLAES\.

SFTAES, our SFT baseline that outputs only the predicted score, achieves an average QWK of 0\.784, outperforming all prompt engineering baselines and several LLM post\-training baselines\. With RL using only the score reward, RLAES\-Score raises average QWK to 0\.802, an improvement of 0\.018 over SFTAES, and surpasses RTS \(0\.794\), the strongest existing LLM post\-training baseline\. RLAES\-AGFO achieves 0\.803, indicating no observed reduction in scoring accuracy after adding AGFO\. Moreover, RLAES\-Score and RLAES\-AGFO obtain the two highest average QWK values among the LLM post\-training methods and come closest to the performance of a single human rater \(0\.805\)\(Uto[2026](https://arxiv.org/html/2607.19219#bib.bib42)\)\. Together with their improvements over SFTAES, these results further support the effectiveness of RL post\-training for AES\.

![Refer to caption](https://arxiv.org/html/2607.19219v1/x3.png)Figure 3:Mean predicted scores and one\-standard\-deviation ranges under different ACR configurations\. Dashed markers show prompt\-level gold means\.

### 5\.2Results on Feedback Generation

Using the RFE framework, we compare the feedback quality \(RFE score\) of four models: GPT\-5\.5 as a reference; RLAES\-Step 0 \(the Qwen3\.5\-9B initialization before RLAES post\-training\); RLAES\-Score, trained solely with score rewards; and RLAES\-AGFO, trained with AGFO\.

#### Overall Feedback Quality Comparison\.

As shown in Table[3](https://arxiv.org/html/2607.19219#S5.T3), AGFO improves the model’s feedback\-generation capability: the average RFE score of RLAES\-AGFO reaches 0\.8399, comparable to that of GPT\-5\.5 \(0\.8334\)\. Meanwhile, RLAES\-AGFO achieves a slightly higher QWK than RLAES\-Score \(0\.8082 vs\. 0\.8043\), suggesting that AGFO does not trade scoring accuracy for feedback quality\. By contrast, the RFE score of RLAES\-Score decreases from 0\.6978 at Step 0 to 0\.5608 after score\-only RL, indicating that optimizing only the score reward may degrade feedback quality\. We examine a possible explanation by analyzing the training dynamics below\.

In terms of training efficiency, we normalize the training time of RLAES\-Score to1×1\\times\. Full feedback optimization \(τ=1\\tau=1\) costs approximately12×12\\times, whereas RLAES\-AGFO costs2×2\\times\(about 160 hours per fold\), representing a reduction of approximately 83%\. RLAES\-AGFO substantially reduces the overhead of frequent LLM\-as\-judge evaluations while achieving a favorable trade\-off between training efficiency and model performance\.

Table 3:Feedback performance \(RFE score\) across the eight ASAP prompts, with overall QWK\.
#### Training Dynamics of Feedback Quality\.

Figure[4](https://arxiv.org/html/2607.19219#S5.F4)tracks QWK and feedback quality \(rubric reward, i\.e\., RFE score\) on Prompt 1 test set at selected training checkpoints\. For RLAES\-Score, QWK rises from 0\.052 to above 0\.83, whereas rubric reward briefly increases from 0\.640 to 0\.702 before falling to 0\.433\. Thus, improvements in scoring accuracy do not necessarily translate into better feedback\. A plausible explanation is that the model initially uses feedback as an intermediate reasoning path for score prediction, temporarily improving feedback quality\. Once it learns to predict scores directly from input features, feedback no longer contributes to score optimization and, without direct optimization, begins to deteriorate\. By contrast, at the final checkpoint, RLAES\-AGFO achieves an RFE score of 0\.866 versus 0\.433 for RLAES\-Score while maintaining comparable QWK, indicating that AGFO prevents feedback collapse without sacrificing scoring performance\.

![Refer to caption](https://arxiv.org/html/2607.19219v1/x4.png)Figure 4:QWK and rubric reward during RLAES\-Score and RLAES\-AGFO training\.

### 5\.3Evaluation of the RFE Framework

#### RFE Scores Across Feedback Types\.

Table[4](https://arxiv.org/html/2607.19219#S5.T4)compares RFE scores for pseudo\-gold, gold, and perturbed feedback\. Gold feedback achieves an overall score of 0\.969, whereas pseudo\-gold feedback scores 0\.694, with particularly low scores for Coverage \(0\.463\) and Evidence \(0\.668\)\. When each essay is paired with feedback from an adjacent\-scoring essay, the overall score falls to 0\.379, accompanied by sharp declines in Evidence \(0\.108\) and Faithfulness \(0\.318\)\. These results provide initial evidence that RFE detects essay–feedback inconsistency\.

Table 4:Overall and dimension\-level RFE scores of different feedback types \(Cov\.: Coverage; Evid\.: Evidence; Faith\.: Faithfulness\)\.
#### Pairwise Discriminative Power\.

We compare RFE with the BERTScore\-based rubric\-similarity method proposed byLi and Pan \([2025a](https://arxiv.org/html/2607.19219#bib.bib17)\)\. As shown in Table[5](https://arxiv.org/html/2607.19219#S5.T5), RFE ranks gold feedback above its perturbed counterpart for all 88 pairs \(100\.0%\), whereas BERTScore does so for 40 of 78 applicable pairs \(51\.3%; Prompts 7 and 8 are excluded because their holistic scores cannot be uniquely mapped to trait\-specific rubric descriptions\)\. This contrast highlights a limitation of rubric\-similarity evaluation: because BERTScore does not condition on the target essay, it cannot directly assess whether the feedback is faithful to that essay\. RFE instead achieves stronger pairwise discrimination in this setting by using essay\-grounded rubric items\.

Table 5:Pairwise discriminative performance on gold–perturbed feedback pairs\.
#### Agreement with Expert Preferences\.

To assess agreement with expert preferences, we sample 43 essays and use an LLM to generate two feedback responses of comparable overall quality but with subtle differences for each essay\. We limit the within\-pair length difference to 10% to control for length bias\. A senior evaluator familiar with ASAP and proficient in English identifies the preferred response in each pair\. We score both responses with RFE and BERTScore and compare their rankings with the expert preferences\.

Table[6](https://arxiv.org/html/2607.19219#S5.T6)reports expert agreement and score separation on the 43 challenging feedback pairs\. RFE matches the expert preferences on 40 pairs \(93\.0%\), whereas BERTScore matches on 23 pairs \(53\.5%\)\. RFE also yields larger score separation: its mean and maximum absolute within\-pair differences are 0\.1163 and 0\.2857, versus 0\.0133 and 0\.0453 for BERTScore\. These results indicate that RFE not only aligns more closely with expert preferences but also provides more discriminative feedback\-quality scores\.

Table 6:Agreement with expert preferences and score separation \(Avg\./Max\. Abs\.Δ\\Delta: mean/maximum within\-pair absolute score differences\)\.

## 6Conclusion

We present RLAES\-AGFO, an RL framework for jointly optimizing essay scoring and feedback generation\. We introduce RFE for fine\-grained feedback evaluation, yielding scores usable as RL rewards, and demonstrate that its rankings closely align with expert preferences\. Building on RFE, AGFO adaptively activates feedback rewards during RL, reducing training time relative to full feedback optimization while avoiding the feedback degradation of score\-only RL\. We also introduce ACR to mitigate adjacent confusion in AES\. Experiments show that RLAES\-AGFO achieves the highest QWK among the LLM\-based methods and an RFE score comparable to that of GPT\-5\.5, while ACR improves scoring performance across both evaluated base models\.

RLAES remains limited by its dependence on LLM judges, whose differing preferences can cause reward mismatch, particularly for complex scoring rubrics\. Evaluation is also restricted to prompt\-specific English essays from ASAP\. Future work should examine cross\-prompt, cross\-domain, and multilingual settings, as well as alternative strategies for joint scoring and feedback optimization\.

## References

- S\. I\. Abudalfa and J\. S\. Barrot \(2026\)Generative artificial intelligence for automated writing evaluation: a systematic review of trends, efficacy, and challenges\.Assessing Writing68,pp\. 101041\.External Links:ISSN 1075\-2935,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.asw.2026.101041),[Link](https://www.sciencedirect.com/science/article/pii/S1075293526000292)Cited by:[§1](https://arxiv.org/html/2607.19219#S1.p1.1)\.
- Y\. Cai, K\. Liang, S\. Lee, Q\. Wang, and Y\. Wu \(2025\)Rank\-Then\-Score: Enhancing Large Language Models for Automated Essay Scoring\.External Links:2504\.05736,[Link](https://arxiv.org/abs/2504.05736)Cited by:[§1](https://arxiv.org/html/2607.19219#S1.p2.1),[§4\.3](https://arxiv.org/html/2607.19219#S4.SS3.SSS0.Px2.p1.1),[Table 2](https://arxiv.org/html/2607.19219#S5.T2.1.16.15.1)\.
- H\. Choi, M\. Kang, J\. Seong, and J\. Huang \(2026\)Exploring zero\-shot essay scoring: from feature\-based to LLM\-based approaches\.Data Mining and Knowledge Discovery40\(3\),pp\. 35\.External Links:[Document](https://dx.doi.org/10.1007/s10618-026-01193-z),[Link](https://doi.org/10.1007/s10618-026-01193-z),ISSN 1573\-756XCited by:[§1](https://arxiv.org/html/2607.19219#S1.p2.1)\.
- DeepSeek\-AI \(2026\)DeepSeek\-v4: towards highly efficient million\-token context intelligence\.Cited by:[§4\.3](https://arxiv.org/html/2607.19219#S4.SS3.SSS0.Px1.p1.1),[§4\.4](https://arxiv.org/html/2607.19219#S4.SS4.p2.4)\.
- A\. Gunjal, A\. Wang, E\. Lau, V\. Nath, Y\. He, B\. Liu, and S\. Hendryx \(2025\)Rubrics as rewards: reinforcement learning beyond verifiable domains\.External Links:2507\.17746,[Link](https://arxiv.org/abs/2507.17746)Cited by:[§2\.2](https://arxiv.org/html/2607.19219#S2.SS2.p2.1),[§3\.1](https://arxiv.org/html/2607.19219#S3.SS1.p1.1)\.
- D\. Guoet al\.\(2025\)DeepSeek\-R1 incentivizes reasoning in LLMs through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.External Links:[Document](https://dx.doi.org/10.1038/s41586-025-09422-z),[Link](https://doi.org/10.1038/s41586-025-09422-z),ISSN 1476\-4687Cited by:[§2\.2](https://arxiv.org/html/2607.19219#S2.SS2.p1.1)\.
- B\. Hamner, J\. Morgan, lynnvandev, M\. Shermis, and T\. V\. Ark \(2012\)The hewlett foundation: automated essay scoring\.Note:https://kaggle\.com/competitions/asap\-aesKaggleCited by:[§3\.1](https://arxiv.org/html/2607.19219#S3.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2607.19219#S4.SS1.p1.1)\.
- Y\. Huang, C\. Palermo, and J\. Wilson \(2026\)Accuracy and fairness of generative AI in automated essay scoring: Comparing GPT\-4o, feature\-based models, and human raters\.Assessing Writing69,pp\. 101047\.External Links:[Document](https://dx.doi.org/10.1016/j.asw.2026.101047),[Link](https://doi.org/10.1016/j.asw.2026.101047),ISSN 1075\-2935Cited by:[§1](https://arxiv.org/html/2607.19219#S1.p2.1)\.
- R\. Johnsi and G\. B\. Kumar \(2025\)Enhancing automated essay scoring by leveraging LSTM networks with hyper\-parameter tuned word embeddings and fine\-tuned LLMs\.Engineering Research Express7\(2\),pp\. 025272\.External Links:[Document](https://dx.doi.org/10.1088/2631-8695/adcf74),[Link](https://doi.org/10.1088/2631-8695/adcf74),ISSN 2631\-8695Cited by:[§1](https://arxiv.org/html/2607.19219#S1.p2.1),[§4\.3](https://arxiv.org/html/2607.19219#S4.SS3.SSS0.Px2.p1.1),[Table 2](https://arxiv.org/html/2607.19219#S5.T2.1.14.13.1)\.
- A\. W\. Li and P\. Collins \(2026\)Formative feedback across sources: Student perceptions and writing outcomes with instructor, peer, and AI\-generated feedback\.Reading and Writing\.External Links:[Document](https://dx.doi.org/10.1007/s11145-026-10761-0),[Link](https://doi.org/10.1007/s11145-026-10761-0),ISSN 1573\-0905Cited by:[§1](https://arxiv.org/html/2607.19219#S1.p1.1)\.
- R\. Li, M\. Du, B\. Xu, C\. Zhu, X\. Wang, and Z\. Mao \(2026\)DeepResearch bench ii: diagnosing deep research agents via rubrics from expert report\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/ARXIV.2601.08536),[Link](https://arxiv.org/abs/2601.08536)Cited by:[§3\.1](https://arxiv.org/html/2607.19219#S3.SS1.p1.1)\.
- X\. Li and W\. Pan \(2025a\)CEAES: bidirectional reinforcement learning optimization for consistent and explainable essay assessment\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Vienna, Austria,pp\. 26267–26279\.External Links:[Link](https://aclanthology.org/2025.acl-long.1273/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1273)Cited by:[§1](https://arxiv.org/html/2607.19219#S1.p3.1),[§1](https://arxiv.org/html/2607.19219#S1.p5.1),[§4\.3](https://arxiv.org/html/2607.19219#S4.SS3.SSS0.Px1.p1.1),[§5\.3](https://arxiv.org/html/2607.19219#S5.SS3.SSS0.Px2.p1.1),[Table 2](https://arxiv.org/html/2607.19219#S5.T2.1.3.2.1)\.
- X\. Li and W\. Pan \(2025b\)KAES: Multi\-aspect Shared Knowledge Finding and Aligning for Cross\-prompt Automated Scoring of Essay Traits\.Proceedings of the AAAI Conference on Artificial Intelligence39\(23\),pp\. 24476–24484\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v39i23.34626),[Link](https://doi.org/10.1609/aaai.v39i23.34626),ISSN 2159\-5399Cited by:[§1](https://arxiv.org/html/2607.19219#S1.p2.1)\.
- Y\. Liu, J\. Han, A\. Sboev, and I\. Makarov \(2024\)GEEF: a neural network model for automatic essay feedback generation by integrating writing skills assessment\.Expert Systems with Applications245,pp\. 123043\.External Links:ISSN 0957\-4174,[Link](http://dx.doi.org/10.1016/j.eswa.2023.123043),[Document](https://dx.doi.org/10.1016/j.eswa.2023.123043)Cited by:[§1](https://arxiv.org/html/2607.19219#S1.p3.1)\.
- W\. A\. Mansour, S\. Albatarni, S\. Eltanbouly, and T\. Elsayed \(2024\)Can Large Language Models Automatically Score Proficiency of Written Essays?\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),Torino, Italia,pp\. 2777–2786\.External Links:[Link](https://aclanthology.org/2024.lrec-main.247/)Cited by:[§1](https://arxiv.org/html/2607.19219#S1.p2.1),[§4\.3](https://arxiv.org/html/2607.19219#S4.SS3.SSS0.Px1.p1.1),[Table 2](https://arxiv.org/html/2607.19219#S5.T2.1.2.1.2)\.
- OpenAI \(2026\)GPT\-5\.5 System Card\.Note:https://openai\.com/index/gpt\-5\-5\-system\-card/Accessed: 2026\-07\-14Cited by:[§4\.4](https://arxiv.org/html/2607.19219#S4.SS4.p2.4)\.
- C\. M\. Ormerod and A\. Kwako \(2024\)Automated Text Scoring in the Age of Generative AI for the GPU\-poor\.External Links:2407\.01873,[Link](https://arxiv.org/abs/2407.01873)Cited by:[§1](https://arxiv.org/html/2607.19219#S1.p2.1),[§1](https://arxiv.org/html/2607.19219#S1.p3.1),[§1](https://arxiv.org/html/2607.19219#S1.p5.1),[§4\.3](https://arxiv.org/html/2607.19219#S4.SS3.SSS0.Px1.p1.1),[§4\.3](https://arxiv.org/html/2607.19219#S4.SS3.SSS0.Px2.p1.1),[Table 2](https://arxiv.org/html/2607.19219#S5.T2.1.15.14.1),[Table 2](https://arxiv.org/html/2607.19219#S5.T2.1.5.4.1)\.
- Qwen Team \(2026a\)Qwen3\.5: towards native multimodal agents\.External Links:[Link](https://qwen.ai/blog?id=qwen3.5)Cited by:[§4\.4](https://arxiv.org/html/2607.19219#S4.SS4.p1.1)\.
- Qwen Team \(2026b\)Qwen3\.7: the agent frontier\.External Links:[Link](https://qwen.ai/blog?id=qwen3.7)Cited by:[§4\.3](https://arxiv.org/html/2607.19219#S4.SS3.SSS0.Px1.p1.1)\.
- T\. Sasaki, R\. Conijn, and M\. C\. Willemsen \(2026\)The Blind Spots in Automated Feedback Generation for Academic Writing\.InProceedings of the LAK26: 16th International Learning Analytics and Knowledge Conference \(LAK 2026\),pp\. 610–620\.External Links:[Document](https://dx.doi.org/10.1145/3785022.3785120),[Link](https://doi.org/10.1145/3785022.3785120)Cited by:[§1](https://arxiv.org/html/2607.19219#S1.p3.1)\.
- K\. Seßler, M\. Fürstenberg, B\. Bühler, and E\. Kasneci \(2025\)Can AI grade your essays? A comparative analysis of large language models and teacher ratings in multidimensional essay scoring\.InProceedings of the 15th International Learning Analytics and Knowledge Conference \(LAK 2025\),pp\. 462–472\.External Links:[Document](https://dx.doi.org/10.1145/3706468.3706527),[Link](https://doi.org/10.1145/3706468.3706527)Cited by:[§1](https://arxiv.org/html/2607.19219#S1.p2.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. Guo \(2024\)DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models\.External Links:2402\.03300,[Link](https://arxiv.org/abs/2402.03300)Cited by:[§2\.2](https://arxiv.org/html/2607.19219#S2.SS2.p1.1)\.
- T\. Shibata and Y\. Miyamura \(2025\)LCES: zero\-shot automated essay scoring via pairwise comparisons using large language models\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 29988–30001\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.1523/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1523),ISBN 979\-8\-89176\-332\-6Cited by:[§1](https://arxiv.org/html/2607.19219#S1.p2.1)\.
- J\. Su, Y\. Yan, F\. Fu, Z\. Han, J\. Ye, X\. Liu, J\. Huo, H\. Zhou, and X\. Hu \(2025\)EssayJudge: a multi\-granular benchmark for assessing automated essay scoring capabilities of multimodal large language models\.InFindings of the Association for Computational Linguistics: ACL 2025,Vienna, Austria,pp\. 6363–6389\.External Links:[Link](https://aclanthology.org/2025.findings-acl.329/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.329)Cited by:[§1](https://arxiv.org/html/2607.19219#S1.p2.1)\.
- J\. Sun, T\. Song, W\. Peng, and J\. Song \(2025\)A survey of automated essay scoring: challenges, advances, and future\.Neurocomputing650,pp\. 130916\.External Links:ISSN 0925\-2312,[Link](http://dx.doi.org/10.1016/j.neucom.2025.130916),[Document](https://dx.doi.org/10.1016/j.neucom.2025.130916)Cited by:[§1](https://arxiv.org/html/2607.19219#S1.p2.1)\.
- K\. Taghipour and H\. T\. Ng \(2016\)A neural approach to automated essay scoring\.InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing,J\. Su, K\. Duh, and X\. Carreras \(Eds\.\),Austin, Texas,pp\. 1882–1891\.External Links:[Link](https://aclanthology.org/D16-1193/),[Document](https://dx.doi.org/10.18653/v1/D16-1193)Cited by:[§1](https://arxiv.org/html/2607.19219#S1.p2.1),[§4\.1](https://arxiv.org/html/2607.19219#S4.SS1.p1.1)\.
- A\. Tversky and D\. Kahneman \(1974\)Judgment under uncertainty: heuristics and biases\.Science185\(4157\),pp\. 1124–1131\.External Links:[Document](https://dx.doi.org/10.1126/science.185.4157.1124)Cited by:[§3\.3](https://arxiv.org/html/2607.19219#S3.SS3.p2.1)\.
- M\. Uto \(2026\)Has automated essay scoring reached sufficient accuracy? deriving achievable qwk ceilings from classical test theory\.InInternational Conference on Artificial Intelligence in Education,pp\. 555–570\.Cited by:[§5\.1](https://arxiv.org/html/2607.19219#S5.SS1.SSS0.Px2.p1.1)\.
- Y\. Wang, Z\. Ding, X\. Wu, S\. Sun, N\. Liu, and X\. Zhai \(2026\)AutoSCORE: Enhancing Automated Scoring with Multi\-Agent Large Language Models via Structured Component Recognition\.Proceedings of the AAAI Conference on Artificial Intelligence40\(48\),pp\. 40898–40906\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v40i48.42123),[Link](https://doi.org/10.1609/aaai.v40i48.42123)Cited by:[§1](https://arxiv.org/html/2607.19219#S1.p2.1)\.
- C\. Xiao, W\. Ma, Q\. Song, S\. X\. Xu, K\. Zhang, Y\. Wang, and Q\. Fu \(2025\)Human\-AI Collaborative Essay Scoring: A Dual\-Process Framework with LLMs\.InProceedings of the 15th International Learning Analytics and Knowledge Conference \(LAK 2025\),pp\. 293–305\.External Links:[Document](https://dx.doi.org/10.1145/3706468.3706507),[Link](https://doi.org/10.1145/3706468.3706507)Cited by:[§1](https://arxiv.org/html/2607.19219#S1.p2.1),[§1](https://arxiv.org/html/2607.19219#S1.p3.1),[§1](https://arxiv.org/html/2607.19219#S1.p5.1),[§4\.3](https://arxiv.org/html/2607.19219#S4.SS3.SSS0.Px1.p1.1),[§4\.3](https://arxiv.org/html/2607.19219#S4.SS3.SSS0.Px2.p1.1),[Table 2](https://arxiv.org/html/2607.19219#S5.T2.1.13.12.1),[Table 2](https://arxiv.org/html/2607.19219#S5.T2.1.4.3.1)\.
- C\. Xiao, W\. Ma, S\. X\. Xu, K\. Zhang, Y\. Wang, and Q\. Fu \(2024\)From Automation to Augmentation: Large Language Models Elevating Essay Scoring Landscape\.CoRRabs/2401\.06431\.External Links:2401\.06431,[Document](https://dx.doi.org/10.48550/arXiv.2401.06431),[Link](https://doi.org/10.48550/arXiv.2401.06431)Cited by:[§1](https://arxiv.org/html/2607.19219#S1.p3.1),[§4\.3](https://arxiv.org/html/2607.19219#S4.SS3.SSS0.Px2.p1.1),[Table 2](https://arxiv.org/html/2607.19219#S5.T2.1.12.11.2)\.
- J\. Xie, K\. Cai, L\. Kong, J\. Zhou, and W\. Qu \(2022\)Automated Essay Scoring via Pairwise Contrastive Regression\.InProceedings of the 29th International Conference on Computational Linguistics,Gyeongju, Republic of Korea,pp\. 2724–2733\.External Links:[Link](https://aclanthology.org/2022.coling-1.240/)Cited by:[§1](https://arxiv.org/html/2607.19219#S1.p2.1)\.
- Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, J\. Liu, L\. Liu, X\. Liu, H\. Lin, Z\. Lin, B\. Ma, G\. Sheng, Y\. Tong, C\. Zhang, M\. Zhang, R\. Zhang, W\. Zhang, H\. Zhu, J\. Zhu, J\. Chen, J\. Chen, C\. Wang, H\. Yu, Y\. Song, X\. Wei, H\. Zhou, J\. Liu, W\. Ma, Y\. Zhang, L\. Yan, Y\. Wu, and M\. Wang \(2025\)DAPO: an open\-source llm reinforcement learning system at scale\.InAdvances in Neural Information Processing Systems,D\. Belgrave, C\. Zhang, H\. Lin, R\. Pascanu, P\. Koniusz, M\. Ghassemi, and N\. Chen \(Eds\.\),Vol\.38,pp\. 113222–113244\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2025/file/a4277440d50f1f15d2cb4c14f7e0c0d2-Paper-Conference.pdf)Cited by:[§4\.4](https://arxiv.org/html/2607.19219#S4.SS4.p1.1)\.

## Appendix AJoint Training Across Prompts

ASAP is commonly evaluated by performing five\-fold cross\-validation independently for each of its eight prompts\. Consequently, this “single\-prompt, single\-model” paradigm requires8×5=408\\times 5=40training runs for a complete evaluation, complicating the experimental workflow and prolonging the evaluation cycle, particularly for computationally intensive LLM post\-training\. This model fragmentation also increases deployment and maintenance costs and conflicts with the efficient parameter sharing and unified model serving expected of LLM\-based systems\.

We instead pool the training splits of all eight prompts within each fold and train a single model, which is evaluated on the corresponding test splits\. Because each input includes prompt\-specific scoring guidelines, the model can handle different score ranges without normalization\. This protocol reduces a complete evaluation from 40 to 5 training runs while preserving the original data partitions: no test examples are included in the pooled training set, thereby eliminating the risk of data leakage from the test sets\.

#### Evaluation of Joint Training Across Prompts\.

We evaluate prompt joint training with NPCR, SFTAES, and RLAES\-Score, representing models spanning three method classes: the BERT\-based AES model, LLM SFT, and LLM RL, respectively\. NPCR is the most competitive open\-source BERT\-based baseline\. The evaluation is conducted on fold 0\. As shown in Table[7](https://arxiv.org/html/2607.19219#A2.T7), joint training changes average QWK by−0\.0048\-0\.0048,−0\.0027\-0\.0027, and−0\.0041\-0\.0041, respectively\. All decreases are below 0\.005 for any of the three model classes, indicating only a negligible loss in scoring performance\. Overall, prompt joint training reduces the number of training runs from 40 to 5 with negligible performance loss, while simplifying the experimental workflow and enabling unified modeling and efficient deployment\.

## Appendix BAdditional Ablation Studies

#### Ablation on Feedback Output\.

We compare two RLAES\-Score configurations that use the same output format during training and inference: feedback\-then\-score and outputting only the score\. As shown in Table[8](https://arxiv.org/html/2607.19219#A2.T8), removing feedback from both training and inference decreases average QWK only marginally, from 0\.8043 to 0\.8036 \(Δ=−0\.0007\\Delta=\-0\.0007\)\. This negligible difference indicates that, when optimization uses only the score reward, including feedback in the model output has little effect on scoring accuracy\.

#### Ablation on ACR in RL Post\-Training\.

We examine whether ACR remains beneficial during RL post\-training by training RLAES\-Score with the ACR response format\. As shown in Table[9](https://arxiv.org/html/2607.19219#A2.T9), ACR increases average QWK only marginally, from 0\.802 to 0\.805 \(Δ=\+0\.003\\Delta=\+0\.003\)\. This gain is substantially smaller than that observed in LLM prompt engineering, where ACR raises average QWK from 0\.430 to 0\.462 for DeepSeek\-V4\-Pro and from 0\.446 to 0\.473 for Qwen3\.7\-Max\. One explanation is that ACR provides base models with an explicit inductive bias for adjacent\-score discrimination, whereas RL post\-training can internalize the same logic through direct optimization of the score reward, leaving limited additional benefit from ACR\.

Table 7:Effect of prompt joint training on scoring performance \(QWK\)\. RLAES denotes RLAES\-Score trained using only the score reward\.
Table 8:Effect of feedback output on scoring performance \(QWK\)\. “w/ FB” and “w/o FB” denote feedback\-then\-score and score\-only outputs, respectively\.
Table 9:Effect of ACR\-formatted RL post\-training on scoring performance \(QWK\)\.Δ\\Deltadenotes the change in average QWK relative to RLAES\-Score\.

## Appendix CExperimental Prompts

Listing 1:LLM\-as\-judge prompt template for RFE\.\#\#\#I\.Role&Objective

YouareasenioreducationalassessmentexpertevaluatingthequalityoffeedbackproducedbyanAutomatedEssayScoring\(AES\)system\.Youapplyrigorous,high\-standardevaluationcriteria\.

Yourtask:Foreachrubricitembelow,judgewhethertheAESfeedbacksatisfiesthatitem\(1=yes,0=no\)\.Baseyourjudgmentstrictlyontheprovidedmaterials\.Applystrictstandards\-partialorborderlinesatisfactionshouldbescored0\.

\#\#\#II\.GuidingPrinciples

\-AnAESsystem’soutputconsistsoftwoparts:\*\*Explanations\*\*\(textualfeedback/commentary\)and\*\*Score\*\*\(numericrating\)\.Inthisevaluation,"feedback"refersexclusivelytotheExplanationsportion\.EvaluateONLYtheExplanationstext;ignorethescoretheAESsystemassigned\.

\-Eachrubricitemmustbejudgedindependently\.Donotletoneitem’sjudgmentinfluenceanother\.

\-\*\*Anonymizationplaceholders\*\*:‘@CAPS1‘,‘@LOCATION1‘,‘@PERCENT1‘,‘@DATE1‘,‘@ORGANIZATION1‘,‘@PERSON1‘,‘@MONEY1‘,‘@TIME1‘,etc\.,arede\-identifiedplaceholdersinsertedbyNamedEntityRecognition\(NER\)duringdatapreprocessing\.Assumetheyrepresentcorrectentities\.TheAESfeedbackshouldNOTinterpret,commenton,orexplaintheseplaceholders\.

\-\*\*OCR/typohandling\*\*:StudentessaysmaycontainOCRerrors\.IftheAESfeedbackquotesanOCRartifactasastudenterror,thatisanincorrectcitation\.IftheAESfeedbackquotesagenuinestudentspelling/grammarerror,thatisvalid\.

\-\*\*EvidenceGrounding\*\*:Whenjudgingwhetherthefeedbackprovides"specificreferences,"acceptallofthefollowingasvalidevidence:

\(a\)Directquotationfromtheessay\(verbatimtextinquotesorclearlysetoff\)

\(b\)Specificparaphrasethatpointstoidentifiable,locatablecontentintheessay

\(c\)Referencetoaspecific,identifiabledetail\(e\.g\.,aparticularargument,example,paragraph,ornarrativeevent\)

Onlyrejectvaguegeneralizationsthatcouldapplytoanyessay\(e\.g\.,"theessayhasgooddetails"withoutspecifyingwhich\)\.

\*\*Important\*\*:Foritemsrequiring"atleasttwo"references,counteachdistinctquotation/paraphrase/detailreferenceseparately\.Tworeferencestothesamepassagecountasone\.

\-\*\*Reasoningchain\*\*:Whenanitemaskswhetherthefeedback"explainsWHY"orprovidesa"reasoningchain,"checkthatthefeedbackexplicitlyconnectsspecifictextualevidencetoitsevaluativeconclusion\(e\.g\.,"TheessaystatesX,whichdemonstratesYbecauseZ"\)\.MerelyjuxtaposingaquotenexttoajudgmentwithoutexplainingtheconnectiondoesNOTsatisfythisrequirement\.

\-\*\*Conditionalitems\*\*:Someitemshaveconditions\(e\.g\.,"Ifthefeedbackdoesnotaddressconventions,score1\."\)\.Iftheconditionisnotmet,theitemisautomaticallyscored1\.

\-\*\*Reasoning\-first\*\*:Foreachitem,firstcitespecificevidencefromtheAESfeedbackorstudentessay,thenderiveyourjudgment\(0or1\)\.

\-\*\*Whenuncertain,defaultto0\*\*\(strictevaluation\)\.

\#\#\#III\.InputMaterials

<essay\_prompt\>

\{essay\_prompt\}

</essay\_prompt\>

<student\_essay\>

\{student\_essay\}

</student\_essay\>

<rubric\_guidelines\>

\{rubric\_guidelines\}

</rubric\_guidelines\>

<AES\_feedback\_to\_evaluate\>

\{model\_answer\}

</AES\_feedback\_to\_evaluate\>

\#\#\#IV\.RubricItemstoEvaluate

\{rubric\_items\_text\}

\#\#\#V\.OutputRequirements

1\.OutputONLYavalidJSONarray\.Nomarkdowncodeblockmarkers\.

2\.NoadditionalexplanatorytextoutsidetheJSON\.

3\.Eachobjectinthearraycorrespondstoonerubricitemabove\.

<output\_format\>

\[

\{\{

"item\_id":"<string\>",

"dimension":"<string\>",

"reason":"<evidence\-drivenexplanation,citespecificcontentfromtheAESfeedbackorstudentessay\>",

"judgment":<0or1\>

\}\},

…

\]

</output\_format\>

Now,pleasebeginyourevaluation\.

Listing 2:Prompt Template for AES and AFG\#Role

AsavirtualevaluatorwithexpertiseinEnglishcomposition,yourroleistocriticallyanalyzeandgradestudentessaysaccordingtoapredeterminedsetofrubrics\.Youaretoactasanimpartialjudgeandevaluatetheessaysbasedonthequalityofthewritingandadherencetotheessayprompt\.

\#SampleEssayPrompt

\{essay\_prompt\}

\#Student’sEssaytoEvaluate

\{\{essay\}\}

\#Pleasecarefullyreadthefollowingrubricguidelines\.Thescoringrangeisfrom\{min\_score\}to\{max\_score\}:

\{scoring\_rubric\_guidelines\}

Note:Theessaycontainsanonymizedtagssuchas@CAPS1,@LOCATION1,etc\.Pleasetreattheseasnormalnames,places,orcapitalizedwords,andDONOTpenalizethestudentfortheirpresence\.

\#Pleasestrictlyfollowtheformatbelowwhenscoring\.Theexplanationsmustbeatleast100words:

\-\*\*Explanations\*\*:xxx

\-\*\*Score\*\*:xx/\{max\_score\}

Listing 3:Prompt Template for ACR\#Pleasestrictlyfollowtheformatbelowwhenscoring\.

\-\*\*Explanations\*\*:\[Providedetailedanalysisaccordingtotherubricguidelines\.\]

\-\*\*Whynottheadjacentlowerscore\*\*:\[IfyourScorexxistheminimumscore\{min\_score\},simplystate"Thisisthelowestpossiblescore"andsummarizewhyitfallsintothelowesttier\.Otherwise,explicitlyidentifytheadjacentlowerscorexx\-1,andexplainwhythisessayisscoredxxinsteadofxx\-1\.\]

\-\*\*Whynottheadjacenthigherscore\*\*:\[IfyourScorexxisthemaximumscore\{max\_score\},simplystate"Thisisthehighestpossiblescore"andsummarizewhyitfallsintothetoptier\.Otherwise,explicitlyidentifytheadjacenthigherscorexx\+1,andexplainwhythisessayisscoredxxinsteadofxx\+1\.\]

\-\*\*Score\*\*:xx/\{max\_score\}

Listing 4:Prompt Template for ACR\-Lower\#Pleasestrictlyfollowtheformatbelowwhenscoring\.

\-\*\*Explanations\*\*:\[Providedetailedanalysisaccordingtotherubricguidelines\.\]

\-\*\*Whynottheadjacentlowerscore\*\*:\[IfyourScorexxistheminimumscore\{min\_score\},simplystate"Thisisthelowestpossiblescore"andsummarizewhyitfallsintothelowesttier\.Otherwise,identifytheadjacentlowerscorexx\-1,brieflyrecalltherubricdescriptionforxx\-1,andexplainwhythisessayclearlyexceedsthatdescription\.\]

\-\*\*Score\*\*:xx/\{max\_score\}