Beyond Reference-Based Evaluation: Reward Models for Meta-Evaluation of Grammatical Error Correction

# Beyond Reference-Based Evaluation: Reward Models for Meta-Evaluation of Grammatical Error Correction
Source: [https://arxiv.org/html/2609.21231](https://arxiv.org/html/2609.21231)
Bill E\. JohnsonAffiliation:Scribendi Inc\.Correspondence:[r82wu@uwaterloo\.ca](mailto:[email protected])Gene SaundersAffiliation:Scribendi Inc\.Correspondence:[r82wu@uwaterloo\.ca](mailto:[email protected])Osama HamzehAffiliation:Scribendi Inc\.Correspondence:[r82wu@uwaterloo\.ca](mailto:[email protected])Ankit VadehraAffiliation:University of WaterlooAffiliation:Vector InstituteAffiliation:Scribendi Inc\.Correspondence:[r82wu@uwaterloo\.ca](mailto:[email protected])Pascal PoupartAffiliation:University of WaterlooAffiliation:Vector Institute

###### Abstract

Reference\-based metrics for Grammatical Error Correction \(GEC\) such as M2and ERRANT assume that the reference set enumerates all valid edits, and therefore often penalize corrections that are grammatical and meaning\-preserving but phrased differently\. We introduceRM\-EVAL, a reward model trained on human preference data from SEEDA, as a*reference\-free*meta\-evaluator that predicts human\-like quality judgments at both full\-sequence and partial\-sequence levels\. Beyond evaluation, we show that the same reward model can be used as a learning signal to*improve*GEC generation viaReward\-Guided Text Generation \(RGTG\), which keeps a base GEC model frozen and performs online, reward\-driven decoding\. Across SEEDA, RM\-EVAL achieves strong agreement with human rankings, and RGTG yields consistent gains in reward and external validation, demonstrating a unified framework for both assessing and enhancing GEC systems without relying on gold references\.

## 1Introduction

Evaluation remains one of the central challenges in Grammatical Error Correction \(GEC\)\. Metrics such as M2\([Dahlmeier and Ng, 2012](https://arxiv.org/html/2609.21231#bib.bib6)\)and ERRANT\([Bryant et al\., 2017](https://arxiv.org/html/2609.21231#bib.bib7)\)are widely used, yet they depend on fixed references that capture only a limited set of acceptable corrections\. Consequently, any deviation—even if grammatically or semantically valid—may be penalized\. This reliance on references limits generalization and fails to align well with human judgments\.

Recent work in reward modeling and reinforcement learning from human feedback \(RLHF\) has demonstrated the potential of learned evaluators to approximate human preferences\([Ouyang et al\., 2022](https://arxiv.org/html/2609.21231#bib.bib10)\)\. In this paper, we proposeRM\-EVAL, a reward model trained on the SEEDA dataset—a large\-scale collection of human evaluation data for GEC systems\. RM\-EVAL provides a reference\-free evaluation framework that captures human\-like judgments GEC quality\.

This work makes the following contributions:

- •We propose areward\-model–based evaluation framework\(RM\-EVAL\) as a reference\-free alternative that achieves competitive or superior correlation with human preferences compared to conventional metrics, and performs on par with state\-of\-the\-art \(paid closed\) LLM\-based evaluation methods, while being much smaller, free and open source\.
- •We further demonstrate that RM\-EVAL can be used to improve*any*text\-generation GEC model \(including LLM\-based correctors\) viaReward\-Guided Text Generation \(RGTG\), an online RLHF\-style decoding procedure that*keeps the base GEC model frozen*while steering outputs using the learned reward signal\.

## 2Related Work

##### Reference\-based Evaluation\.

Standard GEC metrics such as M2and ERRANT compute overlap between system edits and reference edits, effectively penalizing any non\-reference corrections\. This reliance on limited, often incomplete reference sets leads to three key shortcomings: \(1\) the unrealistic assumption of reference completeness, \(2\) inconsistent scores across different reference annotations, and \(3\) poor alignment with human judgments of adequacy and fluency\([Chollampatt and Ng, 2018](https://arxiv.org/html/2609.21231#bib.bib8)\)\. These limitations motivate the need for reference\-free evaluation approaches\.

##### Meta\-evaluation Benchmarks\.

Recent benchmarks such as SEEDA\([Chen et al\., 2024](https://arxiv.org/html/2609.21231#bib.bib9)\)have provided large\-scale human ratings of GEC outputs, enabling systematic meta\-evaluation of automatic metrics\. However, these benchmarks are still limited by the metrics they test rather than providing new evaluation paradigms\.

##### Reward Models for Evaluation\.

Reward modeling has gained traction as a mechanism to train evaluators that generalize human preferences across diverse tasks, including summarization and translation\([Stiennon et al\., 2020](https://arxiv.org/html/2609.21231#bib.bib11)\)\. To our knowledge, this work is the first to explore reward models for meta\-evaluation of GEC\.

##### Reward\-Guided Text Generation \(RGTG\)\.

RGTG is an online, reward\-driven generation paradigm that improves outputs at inference time using a learned reward function, without updating the base generator’s parameters\. Rather than performing standard RL fine\-tuning of the policy, RGTG keeps the underlying model frozen and steers generation using reward feedback during decoding, enabling lightweight alignment and easier reuse across generators and domains\([Rashid et al\., 2025b](https://arxiv.org/html/2609.21231#bib.bib1)\)\.

## 3Methodology

### 3\.1Problem Setup and Data

We consider GEC as a conditional text generation task: given a source sentencexx, a system produces a correctionyy\. Our supervision for evaluation is*preference data*of the form\(x,y\(a\),y\(b\),p\)\(x,y^\{\(a\)\},y^\{\(b\)\},p\), wherep∈\{a,b\}p\\in\\\{a,b\\\}indicates which output humans prefer under criteria such as grammaticality, fluency, and meaning preservation\.

A crucial requirement is that the compared candidates are both*valid alternative corrections of the same source sentence*\. For example, it is appropriate to construct a pair from two GEC system outputs for the samexxwith a human preference label, or from a source–target pair\(x,y⋆\)\(x,y^\{\\star\}\)wherey⋆y^\{\\star\}is explicitly judged as preferred for thatxx\. In contrast, it is generally problematic to treat an arbitrary gold referencerras “preferred” over a system outputyy*unless the preference label was obtained by directly comparingyyandrr*, because we otherwise do not know which one humans would prefer when both may be acceptable\. In this work, we use SEEDA\([Chen et al\., 2024](https://arxiv.org/html/2609.21231#bib.bib9)\), which provides human judgments enabling reliable preference supervision for training a learned evaluator\.

### 3\.2RM\-EVAL: Reward Model for Reference\-Free Evaluation

We train a reward modelRθ​\(x,y\)R\_\{\\theta\}\(x,y\)that assigns a scalar quality score to a correctionyyconditioned on sourcexx\. Given a labeled preference pair\(x,y\+,y−\)\(x,y^\{\+\},y^\{\-\}\), we optimize a standard pairwise preference objective:

ℒ⁡\(θ\)=−log⁡σ⁡\(Rθ​\(x,y\+\)−Rθ​\(x,y−\)\),\\mathcal\{L\}\(\\theta\)=\-\\log\\sigma\\big\(R\_\{\\theta\}\(x,y^\{\+\}\)\-R\_\{\\theta\}\(x,y^\{\-\}\)\\big\),\(1\)whereσ⁡\(⋅\)\\sigma\(\\cdot\)is the logistic sigmoid\. The trained reward model can score a full correction sequence\. Furthermore, with the temporal difference loss in Eq\.[2](https://arxiv.org/html/2609.21231#S3.E2), the reward model can also be applied to partial sequences[Rashid et al\. \(2025a\)](https://arxiv.org/html/2609.21231#bib.bib2)to support fine\-grained analysis \(e\.g\., scoring prefixes during generation\)\.

ℒ\(θ\)=12\[Rθ\(x,y1:i\+\)−maxyi\+1\+Rθ\(x,y1:i\+1\+\)\]2\\displaystyle\\mathcal\{L\}\(\\theta\)=\\frac\{1\}\{2\}\\left\[R\_\{\\theta\}\(x,y\_\{1:i\}^\{\+\}\)\-\\max\_\{y\_\{i\+1\}^\{\+\}\}\\\>R\_\{\\theta\}\(x,y\_\{1:i\+1\}^\{\+\}\)\\right\]^\{2\}

\(2\)

### 3\.3Applying RGTG to GEC with a Frozen Generator

Beyond evaluation, we leverage the trained reward model as a learning signal for improving GEC outputs viaReward\-Guided Text Generation \(RGTG\)\([Rashid et al\., 2025a](https://arxiv.org/html/2609.21231#bib.bib2)\)\. RGTG is an inference\-time alignment framework that steers generation using a learned reward function, without updating the parameters of the underlying generator\.

Formally, letGϕG\_\{\\phi\}denote a pretrained or instruction\-tuned GEC model that maps a source sentencexxto a correctionyy\. In contrast to standard RLHF approaches that fine\-tuneGϕG\_\{\\phi\}using policy optimization, RGTG keepsGϕG\_\{\\phi\}*frozen*and incorporates the reward modelRθ​\(x,y\)R\_\{\\theta\}\(x,y\)directly into the decoding process\. This design significantly reduces computational cost and avoids instability associated with policy updates, while still enabling alignment with human preferences\.

##### Reward\-Guided Decoding Objective\.

At decoding steptt, given a partial hypothesisy<ty\_\{<t\}, the generator proposes a candidate set𝒞t\\mathcal\{C\}\_\{t\}of next\-token or short\-span continuations sampled fromGϕG\_\{\\phi\}\. Each candidateyt∈𝒞ty\_\{t\}\\in\\mathcal\{C\}\_\{t\}yields an extended hypothesisy<t⊕yty\_\{<t\}\\oplus y\_\{t\}where⊕\\oplusrepresents concatenation, then each hypothesis is evaluated using the reward model\. RGTG selects the continuation that maximizes a reward\-augmented decoding objective:

c∗=arg⁡maxyt∈𝒞t⁡\(CLOSElog⁡Pϕ​\(yt∣x,y<t\)OPEN\+β​exp⁡\(Rθ​\(x,y<t⊕yt\)\)\)\\displaystyle\\begin\{aligned\} c^\{\*\}=\\arg\\max\_\{y\_\{t\}\\in\\mathcal\{C\}\_\{t\}\}\\Big\(&\\log P\_\{\\phi\}\(y\_\{t\}\\mid x,y\_\{<t\}\)\\\\ &\+\\beta\\,\\exp\(R\_\{\\theta\}\(x,y\_\{<t\}\\oplus y\_\{t\}\)\)\\Big\)\\end\{aligned\}

\(3\)whereβ\\betacontrols the trade\-off between model likelihood and reward guidance\.

BecauseRθR\_\{\\theta\}can be applied to partial sequences, reward feedback is incorporated incrementally throughout generation, rather than only after producing a complete correction\. This enables fine\-grained control over decoding and allows the reward model to influence local generation decisions\.

Applied to GEC, RGTG directly optimizes for human\-aligned correction quality—capturing grammaticality, fluency, and meaning preservation—without relying on reference edits or additional supervised data\. Moreover, since the generator remains unchanged, the same reward model can be reused to steer different GEC systems, making RGTG a flexible and cost\-effective mechanism for improving correction quality at inference time\.

## 4Experiments

### 4\.1Setup

We evaluate RM\-EVAL on the SEEDA benchmark following the meta\-evaluation protocol described in[Chen et al\. \(2024\)](https://arxiv.org/html/2609.21231#bib.bib9)\. Baselines include M2, ERRANT, existing GEC\-specific metrics such as GLEU, and LLM based methods described in[Kobayashi et al\. \(2024\)](https://arxiv.org/html/2609.21231#bib.bib4)\. We measure the performance by accuracy and Kendall’s rank correlation \(τ\\tau\)\([Kendall, 1938](https://arxiv.org/html/2609.21231#bib.bib5)\)with human judgments at sentence level\. All the reward models are trained by fine\-tuning Llama3\.2\-1B111[meta\-llama/Llama\-3\.2\-1B\-Instruct](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct)or Qwen2\.5\-1\.5B222[Qwen/Qwen2\.5\-1\.5B\-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct)\.

### 4\.2Results

#### 4\.2\.1Reward Model Evaluation on SEEDA

We first evaluateRM\-EVALthrough 5\-fold cross\-validation on the SEEDA dataset\. Each fold consists of a non\-overlapping subset of human\-annotated preference pairs\. For each fold, we train on the remaining folds and test on the current fold\. Performance is reported in terms of \(1\)pairwise accuracy—the percentage of correctly predicted human\-preferred outputs, and \(2\)Kendall’s correlationbetween human rankings and rankings based on model\-predicted reward scores\.

Table 1:Five\-fold cross\-validation of RM\-EVAL on SEEDA\. Correlation denotes Kendall’s rank correlation between human rankings and reward model predictions\.Table 2:Sentence\-level evaluation on SEEDA\_S with fluency correction\. Reported metrics are Accuracy \(Acc\) and Kendall’s rank correlation coefficient \(τ\\tau\)\. Higher values indicate stronger agreement with human judgments\.The Qwen\-based reward model achieves an average accuracy of 0\.84 and a correlation of 0\.62 with human ratings—comparable to SOTA LLM based metrics shown in Table[2](https://arxiv.org/html/2609.21231#S4.T2)\. These results demonstrate that RM\-EVAL effectively captures human preferences for grammaticality and fluency without requiring annotations\. Furthermore RM\-EVAL is small, open\-source and free in comparison to GPT\-5, which is a large, paid and closed model\.

#### 4\.2\.2Reward\-Guided Text Generation \(RGTG\) on SEEDA

We next apply the trained reward model to Reward\-Guided Text Generation \(RGTG\) \- an online RLHF\-style generation method\. The base generator is Llama3\-1B, and we compare variants fine\-tuned with different alignment methods: ARGS\([Khanov et al\., 2024](https://arxiv.org/html/2609.21231#bib.bib3)\), PARGS\([Rashid et al\., 2025b](https://arxiv.org/html/2609.21231#bib.bib1)\), and FaRMA\([Rashid et al\., 2025a](https://arxiv.org/html/2609.21231#bib.bib2)\)\. We refer to the base instruction\-tuned Llama3 model without other enhancement as the SFT baseline\.

Table 3:Average reward scores assigned by RM\-EVAL across RGTG variants on SEEDA\. FaRMA achieves the highest mean reward and lowest variance, indicating the strongest alignment with human preferences\.In Table[3](https://arxiv.org/html/2609.21231#S4.T3), the FaRMA\-aligned model obtains the highest mean reward \(3\.53\), surpassing both ARGS and PARGS\. To assess statistical reliability, we conduct pairwiseWilcoxon signed\-rank testson reward scores between model variants \(Table[4](https://arxiv.org/html/2609.21231#S4.T4)\)\. FaRMA shows statistically significant improvements over SFT \(p=0\.030p=0\.030\) and ARGS \(p=0\.038p=0\.038\), while the improvement over PARGS is positive but not statistically significant \(p=0\.054p=0\.054\)\. No statistically significant differences are observed among the remaining baseline comparisons\.

Table 4:Pairwise Wilcoxon signed\-rank tests on RGTG reward scores\.✓\\checkmarkindicates statistical significance atp<0\.05p<0\.05\.
#### 4\.2\.3GPT Evaluation as External Validator

To further validate reward alignment, we conduct pairwise preference comparisons using GPT\-5 as an external evaluator\.

Table 5:GPT\-5 evaluation of pairwise win rates between model variants\.GPT\-5 evaluation corroborates the reward model findings: FaRMA dominates in 68% and 63% of pairwise comparisons against ARGS and PARGS, respectively \(Table[5](https://arxiv.org/html/2609.21231#S4.T5)\)\. These results collectively suggest that RM\-EVAL provides a reliable training signal for aligning generation models with human preferences\.

## 5Discussion

##### Reward Models as Meta\-Evaluators\.

Our findings demonstrate that reward models can serve as reliable, reference\-free meta\-evaluation tools for GEC\. Unlike traditional metrics such as M2or ERRANT that penalize valid non\-reference corrections, RM\-EVAL learns directly from human preferences and can generalize beyond specific edit patterns\. The achieved human correlation of 0\.62 matching the performance of GPT5—suggests that learned evaluators are approaching the ceiling of reference\-based performance, while offering better interpretability in terms of preference modeling\.

##### Bridging Evaluation and Generation\.

An important implication of this work is the ability to use a single model both as an evaluator and as a learning signal for generation\. When used as a reward function, RM\-EVAL not only ranks system outputs consistently with human judgment but also guides models such as Llama3 toward more human\-aligned behavior\. The FaRMA results \(Tables[3](https://arxiv.org/html/2609.21231#S4.T3)–[5](https://arxiv.org/html/2609.21231#S4.T5)\) demonstrate that reinforcement learning guided by RM\-EVAL yields tangible improvements, both in intrinsic reward scores and in GPT\-based external evaluations\.

##### Generalization and Extensibility\.

Although our experiments focus on English GEC, the methodology is general and can be extended to other languages or text\-editing tasks such as style transfer and paraphrasing\. Since RM\-EVAL is trained on preference pairs rather than explicit corrections, the framework is language\-agnostic and amenable to multilingual scaling\.

##### Future Work\.

Future directions include expanding RM\-EVAL into a multi\-dimensional evaluator that independently scores fluency, adequacy, and grammaticality\. We also plan to explore multi\-reward optimization for RLHF, where GEC models are optimized jointly for grammaticality and semantic preservation\. Finally, integrating human\-in\-the\-loop updates could further enhance the robustness and fairness of the reward model across diverse learner populations\.

## 6Conclusion

We presentedRM\-EVAL, a reward\-model–based framework for meta\-evaluation of grammatical error correction \(GEC\) systems\. Unlike traditional reference\-dependent metrics such as M2or ERRANT, RM\-EVAL evaluates system outputs directly through learned human preferences, offering a scalable and reference\-free approach to GEC evaluation\. Empirical analyses on the SEEDA benchmark show that RM\-EVAL achieves high correlation with human judgments and performs competitively with state\-of\-the\-art metrics\. When employed as a reward function for RGTG, the model further enhances system performance—particularly in the FaRMA variant—demonstrating the dual utility of reward models as both evaluators and optimization objectives\.

## Limitations

RM\-EVAL relies on human preference annotations from the SEEDA benchmark for training\. Although SEEDA provides high\-quality human judgments, it covers only a specific dataset and evaluation setting\. Consequently, the reward model may inherit biases introduced during the annotation process and reflect preferences specific to the annotator population\. In scenarios involving different learner populations, domains, or correction styles, the reward model may not reliably capture human preferences\. In such cases, additional preference annotations may be necessary to recalibrate the model and ensure robust evaluation performance\.

## References

- Bryantet al\.\(2017\)C\. Bryant, M\. Felice, and T\. BriscoeAutomatic annotation and evaluation of error types for grammatical error correction\.InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Vancouver, Canada,pp\. 793–805\.Cited by:[§1](https://arxiv.org/html/2609.21231#S1.p1.1)\.
- Chenet al\.\(2024\)X\. Chen, Y\. Zhao, and H\. T\. NgSEEDA: a dataset for systematic evaluation and error analysis of grammatical error correction metrics\.InProceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications \(BEA 2024\),Bangkok, Thailand\.External Links:[Link](https://aclanthology.org/2024.bea-1.6/)Cited by:[§2](https://arxiv.org/html/2609.21231#S2.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2609.21231#S3.SS1.p2.1),[§4\.1](https://arxiv.org/html/2609.21231#S4.SS1.p1.1)\.
- Chollampatt and Ng \(2018\)S\. Chollampatt and H\. T\. NgA reassessment of reference\-based grammatical error correction metrics\.InProceedings of the 27th International Conference on Computational Linguistics,Santa Fe, USA,pp\. 2582–2591\.Cited by:[§2](https://arxiv.org/html/2609.21231#S2.SS0.SSS0.Px1.p1.1)\.
- Dahlmeier and Ng \(2012\)D\. Dahlmeier and H\. T\. NgBetter evaluation for grammatical error correction\.InProceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,Montréal, Canada,pp\. 568–572\.Cited by:[§1](https://arxiv.org/html/2609.21231#S1.p1.1)\.
- Kendall \(1938\)M\. G\. KendallA new measure of rank correlation\.Biometrika30\(1/2\),pp\. 81–93\.External Links:[Document](https://dx.doi.org/10.2307/2332226)Cited by:[§4\.1](https://arxiv.org/html/2609.21231#S4.SS1.p1.1)\.
- Khanovet al\.\(2024\)M\. Khanov, J\. Burapacheep, and Y\. LiARGS: alignment as reward\-guided search\.External Links:2402\.01694,[Link](https://arxiv.org/abs/2402.01694)Cited by:[§4\.2\.2](https://arxiv.org/html/2609.21231#S4.SS2.SSS2.p1.1)\.
- Kobayashiet al\.\(2024\)M\. Kobayashi, M\. Mita, and M\. KomachiLarge language models are state\-of\-the\-art evaluator for grammatical error correction\.InProceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications \(BEA 2024\),E\. Kochmar, M\. Bexte, J\. Burstein, A\. Horbach, R\. Laarmann\-Quante, A\. Tack, V\. Yaneva, and Z\. Yuan \(Eds\.\),Mexico City, Mexico,pp\. 68–77\.External Links:[Link](https://aclanthology.org/2024.bea-1.6/)Cited by:[§4\.1](https://arxiv.org/html/2609.21231#S4.SS1.p1.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[§1](https://arxiv.org/html/2609.21231#S1.p2.1)\.
- Rashidet al\.\(2025a\)A\. Rashid, R\. Wu, R\. Fan, H\. Li, A\. Kristiadi, and P\. PoupartTowards cost\-effective reward guided text generation\.InForty\-second International Conference on Machine Learning,Cited by:[§3\.2](https://arxiv.org/html/2609.21231#S3.SS2.p1.2),[§3\.3](https://arxiv.org/html/2609.21231#S3.SS3.p1.1),[§4\.2\.2](https://arxiv.org/html/2609.21231#S4.SS2.SSS2.p1.1)\.
- Rashidet al\.\(2025b\)A\. Rashid, R\. Wu, J\. Grosse, A\. Kristiadi, and P\. PoupartA critical look at tokenwise reward\-guided text generation\.InSecond Conference on Language Modeling,Cited by:[§2](https://arxiv.org/html/2609.21231#S2.SS0.SSS0.Px4.p1.1),[§4\.2\.2](https://arxiv.org/html/2609.21231#S4.SS2.SSS2.p1.1)\.
- Stiennonet al\.\(2020\)N\. Stiennon, L\. Ouyang, J\. Wu, D\. M\. Ziegler, R\. Lowe, C\. Voss, A\. Radford, D\. Amodei, P\. Christiano, and J\. LeikeLearning to summarize with human feedback\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 3008–3021\.Cited by:[§2](https://arxiv.org/html/2609.21231#S2.SS0.SSS0.Px3.p1.1)\.

## Appendix ATraining Details

All experiments are run on a server with NVIDIA RTX6000 GPUs \(24GB VRAM\) and NVIDIA A40 GPUs\(40GB VRAM\)\. We use CUDA Toolkit version 12\.4 and PyTorch 2\.5\.1 framework\. We train two versions of RM\-EVAL on the SEEDA dataset, utilize the TRL library to accelerate the training process\. We report the training parameters:

ParametersValueLlamaLR5e\-6Batch size16Gradient acc\. steps16DeepSpeed Zero stage2Max\. sequence length512β\\beta\(RGTG\)0\.5