Generating and Refining Dynamic Evaluation Rubrics for LLM-as-a-Judge

# Generating and Refining Dynamic Evaluation Rubrics for LLM-as-a-Judge
Source: [https://arxiv.org/html/2605.30568](https://arxiv.org/html/2605.30568)
Zijie Wang Eduardo Blanco University of Arizona Department of Computer Science \{zijiewang, eduardoblanco\}@arizona\.edu

###### Abstract

LLM\-as\-a\-Judge is a scalable alternative to human evaluation, yet existing rubric\-based methods rely on human\-annotated data such as reference answers or expert\-crafted rubrics\. We propose to automatically generate fine\-grained evaluation rubrics without any human annotation\. Our training\-free method generates rubrics at dataset\-specific and instance\-specific granularities, achieving performance competitive with existing methods across four benchmarks\. We further present a method that iteratively fine\-tunes a rubric generator model via meta\-judge reward signals\. The fine\-tuned generator outperforms all existing baselines in both pairwise and pointwise evaluation\. Notably, a fine\-tuned 14B rubric generator outperforms a much larger proprietary model at rubric generation, showing the effectiveness of our fine\-tuning strategy\.

Generating and Refining Dynamic Evaluation Rubrics for LLM\-as\-a\-Judge

Zijie Wang Eduardo BlancoUniversity of ArizonaDepartment of Computer Science\{zijiewang, eduardoblanco\}@arizona\.edu

## 1Introduction

LLM\-as\-a\-Judge has emerged as a promising paradigm for evaluating LLM outputs, mitigating limitations of traditional similarity\-based metrics such as BLEU, ROUGE, and BERTScoreLin \([2004](https://arxiv.org/html/2605.30568#bib.bib1)\); Papineniet al\.\([2002](https://arxiv.org/html/2605.30568#bib.bib2)\); Zhanget al\.\([2020](https://arxiv.org/html/2605.30568#bib.bib3)\)\. This approach has demonstrated success across diverse applications, including evaluation of natural language generationZenget al\.\([2024](https://arxiv.org/html/2605.30568#bib.bib22)\), validation of synthetic dataLeanget al\.\([2025](https://arxiv.org/html/2605.30568#bib.bib23)\), and self\-rewarding for LLM alignmentYuanet al\.\([2024](https://arxiv.org/html/2605.30568#bib.bib24)\)\. However, LLM judges exhibit biases toward irrelevant heuristics, such as answer lengthDuboiset al\.\([2024](https://arxiv.org/html/2605.30568#bib.bib25)\)and the order of candidate responsesWanget al\.\([2024a](https://arxiv.org/html/2605.30568#bib.bib26)\), among others\.

To address these biases, previous work has explored training specialized LLM evaluatorsKimet al\.\([2024a](https://arxiv.org/html/2605.30568#bib.bib21),[b](https://arxiv.org/html/2605.30568#bib.bib7)\); Deshpandeet al\.\([2024](https://arxiv.org/html/2605.30568#bib.bib8)\)\. However, such methods require substantial computational costs and high\-quality training data\. An alternative line of research focuses on training\-free methods that leverage evaluation rubrics—structured sets of criteria that specify what dimensions to assess and how to score them—to guide LLM judges toward more consistent and interpretable evaluation\. Within this direction, several studies employ fine\-grained, decomposition\-based rubrics\.Kimet al\.\([2025](https://arxiv.org/html/2605.30568#bib.bib20)\)develop a meta\-evaluation benchmark with instance\-specific rubrics, and several works decompose holistic evaluation into checklists or weighted sub\-criteriaFuruhashiet al\.\([2025](https://arxiv.org/html/2605.30568#bib.bib5)\); Leeet al\.\([2025](https://arxiv.org/html/2605.30568#bib.bib4)\); Liet al\.\([2025](https://arxiv.org/html/2605.30568#bib.bib6),[2026](https://arxiv.org/html/2605.30568#bib.bib19)\)\. Other efforts curate reusable evaluation metrics across tasks, such as AutoMetricRyanet al\.\([2026](https://arxiv.org/html/2605.30568#bib.bib13)\)\. Despite these efforts, existing approaches typically rely on human\-annotated data such as expert\-crafted rubric examples and reference answers, limiting their applicability to critical domains such as finance, health, and security, where expert annotations are scarce or unavailable\.

This work focuses on improving LLM\-as\-a\-Judge systems from the evaluation rubric perspective\. We investigate automatically generating fine\-grained rubrics without requiring any human\-annotated data \(e\.g\., reference answers, rubric examples, or annotated training data\)\. Specifically, we propose a two\-stage approach\. First, we introduce a training\-free method that directly generates evaluation rubrics at multiple granularities—from dataset\-specific to instance\-specific—without requiring human\-annotated examples or reference answers\. Second, we investigate fine\-tuning a rubric generator model via preference learning, leveraging reward signals derived from a meta\-judge’s evaluation on the generated rubrics\. Experiments across four LLMs and four benchmarks demonstrate that our training\-free approach achieves competitive performance with several baselines and human\-crafted rubrics, while rubrics from our fine\-tuned generator further improve the LLM judge, outperforming all existing methods\.

Our contributions are as follows:111Data, code and models available at[https://github\.com/wang\-zijie/generating\_dynamic\_rubric](https://github.com/wang-zijie/generating_dynamic_rubric)

- •We present a training\-free approach for generating fine\-grained rubrics that yields competitive performance with existing methods\.
- •We propose a preference fine\-tuning method for rubric generator models using meta\-judge feedback, requiring no human\-annotated data\.
- •We conduct experiments with four LLMs and four benchmarks, demonstrating state\-of\-the\-art performance in both pairwise and pointwise evaluation settings\.
- •We provide qualitative analyses and case studies revealing how preference fine\-tuning shifts rubric framing, explaining the observed performance gains\.

## 2Related Work

### 2\.1LLM\-based Automatic Evaluation

With advances in LLM fine\-tuning methods, recent work has explored various learning\-based approaches to improve LLM judge models\. Prometheus 2Kimet al\.\([2024b](https://arxiv.org/html/2605.30568#bib.bib7)\)is an open\-source LLM evaluator trained on a human\-crafted dataset that includes predefined rubrics and reference answersKimet al\.\([2024a](https://arxiv.org/html/2605.30568#bib.bib21)\)\. GLIDERDeshpandeet al\.\([2024](https://arxiv.org/html/2605.30568#bib.bib8)\)is another open\-source LLM evaluator trained on synthetic data that employs span highlighting to provide explainable, fine\-grained evaluations\. HerculeDoddapaneniet al\.\([2025](https://arxiv.org/html/2605.30568#bib.bib10)\)is a multilingual LLM evaluator fine\-tuned on multilingual data obtained via machine translation\. JudgeLRMChenet al\.\([2025](https://arxiv.org/html/2605.30568#bib.bib11)\)focuses on improving LLM judges’ reasoning ability by adopting RL\-based preference training with a specialized reward function\. MetaRewardingWuet al\.\([2025](https://arxiv.org/html/2605.30568#bib.bib9)\)introduces a self\-improvement framework that leverages LLMs to judge their own judgments and uses that feedback to refine judging ability\.

In this work, we demonstrate that a training\-free approach yields competitive evaluation results against state\-of\-the\-art LLMs and LLM judges\.

### 2\.2Fine\-grained and Dynamic Rubrics for Automatic Evaluation

Another line of work focuses on training\-free approaches to improve LLM judges’ evaluation quality, specifically by improving the quality of evaluation rubrics\. FLASKYeet al\.\([2024](https://arxiv.org/html/2605.30568#bib.bib36)\)proposes a fine\-grained evaluation framework that decomposes response quality into skill\-specific dimensions aligned with human judgment, though it relies on a fixed, manually curated skill taxonomy\.BiGGen BenchKimet al\.\([2025](https://arxiv.org/html/2605.30568#bib.bib20)\)presents a meta\-evaluation benchmark with instance\-specific rubrics and demonstrates that they are beneficial for LLM judges\. RubricHubLiet al\.\([2026](https://arxiv.org/html/2605.30568#bib.bib19)\)introduces an automated rubric generation framework grounded in reference answers and predefined principles\. AutoMetricRyanet al\.\([2026](https://arxiv.org/html/2605.30568#bib.bib13)\)proposes a task\-specific evaluation framework that generates evaluation metrics with rubrics, retrieves from a curated MetricBank, and optimizes their combination via regression to maximize correlation with human judgment; however, this approach relies on ground truth for both generation and optimization\.Liet al\.\([2025](https://arxiv.org/html/2605.30568#bib.bib6)\)present an approach that decomposes evaluation into multiple criteria and aggregates scores with dynamic weights; however, their method is limited to pairwise evaluation\. CheckEvalLeeet al\.\([2025](https://arxiv.org/html/2605.30568#bib.bib4)\)adopts a checklist\-based evaluation method by decomposing high\-level rubrics into discrete boolean questions, but relies on human knowledge to select evaluation dimensions and sub\-dimensions \(e\.g\., dimension “fluency” with sub\-dimensions “grammar” and “readability”\)\.Furuhashiet al\.\([2025](https://arxiv.org/html/2605.30568#bib.bib5)\)further investigate checklist\-based evaluation with different checklist generation policies and observe that such methods tend to work well only in pairwise settings while showing fewer benefits for pointwise evaluation\.

These approaches typically rely on human\-annotated data such as reference answers or predefined evaluation dimensions\. Our method supports both pointwise and pairwise settings without any such data, making it applicable to domains where these resources are unavailable\.

Dataset\# DataEval\. SchemeScaleRef\.AnswerHuman\-craftedRubricsInstance\-specificRubricsHelpSteer2Wanget al\.\([2024b](https://arxiv.org/html/2605.30568#bib.bib31)\)1,038Pointwise0–4✗✓✗BiGGen BenchKimet al\.\([2025](https://arxiv.org/html/2605.30568#bib.bib20)\)2,776Pointwise1–5✓✓✓AlpacaEvalDuboiset al\.\([2024](https://arxiv.org/html/2605.30568#bib.bib25)\)321Pairwise—✗✓✗MT\-BenchZhenget al\.\([2023](https://arxiv.org/html/2605.30568#bib.bib28)\)941Pairwise—✗✓✗BiGGen BenchKimet al\.\([2025](https://arxiv.org/html/2605.30568#bib.bib20)\)2,792Pairwise—✓✓✓

Table 1:Overview of evaluation benchmarks\. We evaluate on three pairwise and two pointwise benchmarks spanning general instruction\-following, multi\-turn dialogue, and specialized domains\. “Instance\-specific Rubrics” indicates whether the benchmark provides per\-instance evaluation criteria; “Human\-annotated Rubrics” indicates whether those criteria were authored by human experts\.
### 2\.3Rubric\-based Reward Modeling

Beyond improving LLM\-as\-a\-Judge systems, recent work has investigated leveraging rubric\-based reward signals for RL fine\-tuning\. Rubrics as RewardsGunjalet al\.\([2026](https://arxiv.org/html/2605.30568#bib.bib14)\)investigates an on\-policy reinforcement learning method that leverages generated rubrics as reward signals\.Guptaet al\.\([2025](https://arxiv.org/html/2605.30568#bib.bib17)\)address reward hacking by dynamically generating task\-specific rubrics rather than relying on static rubrics for reward modeling\.AutoRuleWang and Xiong \([2025](https://arxiv.org/html/2605.30568#bib.bib16)\)automatically extracts rules from human preference data to construct rule\-based rewards for improving instruction\-following ability\. However, all of these approaches rely on gold reference answers as a supervised proxy for generating or extracting rubrics\. Lastly, checklist\-based rubrics have also been adopted for preference tuning\.Viswanathanet al\.\([2025](https://arxiv.org/html/2605.30568#bib.bib15)\)propose replacing traditional reward models with dynamic, instruction\-specific checklists evaluated by LLM judges\. We are the first to investigate fine\-tuning a rubric generator model via reward signals from a meta\-judge\.

## 3Generating Dynamic Rubrics

We formalize evaluating LLM\-as\-a\-Judge systems with rubrics and describe our approach to generating fine\-grained evaluation rubrics without human preference data\. We first define the two evaluation settings \(Section[3\.1](https://arxiv.org/html/2605.30568#S3.SS1)\) and evaluation benchmarks \(Section[3\.2](https://arxiv.org/html/2605.30568#S3.SS2)\), followed by our training\-free rubric generation approach \(Section[3\.3](https://arxiv.org/html/2605.30568#S3.SS3)\) and the preference learning approach \(Section[3\.4](https://arxiv.org/html/2605.30568#S3.SS4)\)\.

### 3\.1Task Formulation

We consider two evaluation settings commonly adopted in the LLM\-as\-a\-Judge paradigm:*pairwise*and*pointwise*evaluation\.

#### Pairwise Evaluation

Given an instance inputxxand two candidate responsesy1,y2y\_\{1\},y\_\{2\}, the judge model𝒥\\mathcal\{J\}decides𝒥​\(x,y1,y2\)∈\{1,2\}\\mathcal\{J\}\(x,y\_\{1\},y\_\{2\}\)\\in\\\{1,2\\\}, indicating which response is preferred\. We evaluate pairwise judges by measuring*human agreement*: the percentage of instances where the judge’s preference matches the majority vote of human annotators\.

#### Pointwise Evaluation

Given an instance inputxxand a single responseyy, the judge model𝒥\\mathcal\{J\}produces a scalar score𝒥​\(x,y\)∈\[a,b\]\\mathcal\{J\}\(x,y\)\\in\[a,b\], where\[a,b\]\[a,b\]denotes the scoring range \(e\.g\., an integer in\[1,5\]\[1,5\]for holistic scoring, or a continuous value in\[0,1\]\[0,1\]for criterion\-level aggregation\)\. We measure alignment with the ground truth \(i\.e\., human scores\) using Spearman and Pearson correlation coefficients computed across all \(instance, response\) pairs in the dataset\. Note that since these correlation metrics are scale\-invariant, the judge and human scores need not share the same range\.

#### Rubric Granularity

We distinguish two levels of rubric specificity used in this work\. A*dataset\-specific*rubric defines a fixed evaluation instruction applied uniformly to all instances within a benchmark or domain, capturing general quality dimensions shared across instances\. An*instance\-specific*rubric is generated for each individual instance, providing criteria tailored to the particular instance\.

### 3\.2Benchmarks

We evaluate on four meta\-evaluation benchmarks spanning pairwise and pointwise settings \(Table[1](https://arxiv.org/html/2605.30568#S2.T1)\)\.

#### Pointwise Benchmarks

HelpSteer2Wanget al\.\([2024b](https://arxiv.org/html/2605.30568#bib.bib31)\)contains 1,038 instances scored on five dimensions \(helpfulness, correctness, coherence, complexity, verbosity\) as integers in\[0,4\]\[0,4\]\. We use the average of helpfulness, correctness, and coherence as the ground\-truth score, as complexity and verbosity lack a monotonic relationship with quality \(e\.g\., higher verbosity is not inherently better or worse\)\. It provides no reference answers or instance\-specific rubrics\.BiGGen BenchKimet al\.\([2025](https://arxiv.org/html/2605.30568#bib.bib20)\)provides human scores \(\[1,5\]\[1,5\]\) for 695 prompts across 4 models \(2,776 instances\), along with per\-instance scoring rubrics defining what each score level means\. It provides both reference answers and instance\-specific rubrics\.

#### Pairwise Benchmarks

Length\-Controlled AlpacaEvalDuboiset al\.\([2024](https://arxiv.org/html/2605.30568#bib.bib25)\)contains 321 response pairs with human preference annotations collected from 4 evaluators each; we use the majority vote as the ground\-truth label\. It provides neither reference answers nor instance\-specific rubrics\. MT\-BenchZhenget al\.\([2023](https://arxiv.org/html/2605.30568#bib.bib28)\)consists of multi\-turn dialogues with 3,355 human preference annotations grouped into 1,204 unique pairs\. We take the majority vote and exclude ties, yielding 941 evaluation pairs\. We additionally repurposeBiGGen Benchfor pairwise evaluation by comparing all model pairs per prompt and using the higher human score as the preferred response, excluding ties\. This yields 2,792 pairwise evaluation pairs\.

### 3\.3Training\-free Rubrics Generation in Multiple Granularities

We propose a training\-free approach to generate evaluation rubrics at two granularity levels without requiring human\-annotated data\. We denote the model for generating rubrics as𝒢\\mathcal\{G\}; in this setting,𝒢=𝒥\\mathcal\{G\}=\\mathcal\{J\}\(i\.e\., the judge model itself generates the rubrics\), while in Section[3\.4](https://arxiv.org/html/2605.30568#S3.SS4), we fine\-tune a dedicated𝒢\\mathcal\{G\}separate from the judge\.

#### Dataset\-specific Rubric

We prompt the judge model to generate a dataset\-specific rubric, which is applicable to any instance within a benchmark\. The rubric generation prompt contains no instance\-specific information; instead, it asks the model to identify universal dimensions that distinguish good responses from poor ones \(e\.g\., instruction adherence, factual accuracy, completeness, clarity\)\. This is a cost\-efficient setting as the generated rubric is cached and reused for all evaluation instances\.

#### Instance\-specific Rubric

For each instancexx, we prompt the judge model to generate an instance\-specific rubricℛ​\(x\)=\{c1,…,cL\}\\mathcal\{R\}\(x\)=\\\{c\_\{1\},\\ldots,c\_\{L\}\\\}consisting ofLLevaluation criteria tailored to the particular instance\. The rubric generation is conditioned only on the instance’s input \(e\.g\., the prompt\), without seeing any reference responses\. This rubric is then injected into the judge prompt, guiding the model to evaluate responses along the specified dimensions\. The evaluation on responseyyis conditioned on the instance and generated rubric:

𝒥​\(ℛ​\(x\),x,y\)\.\\mathcal\{J\}\(\\mathcal\{R\}\(x\),x,y\)\.\(1\)
Unlike prior rubric generation methodsLiet al\.\([2026](https://arxiv.org/html/2605.30568#bib.bib19)\); Ryanet al\.\([2026](https://arxiv.org/html/2605.30568#bib.bib13)\), our approach requires neither reference answers nor human\-crafted rubrics as few\-shot examples, making it applicable to domains where such resources are unavailable\. Appendix[A](https://arxiv.org/html/2605.30568#A1)and[B](https://arxiv.org/html/2605.30568#A2)provide full prompt templates and examples of generated rubrics\.

### 3\.4Annotation\-free Preference Learning for Rubric Generator

While the training\-free approach produces effective rubrics, its quality is bounded by the generator’s parametric knowledge\. Empirically, larger models generate better rubrics but at higher cost\. To close this gap, we iteratively fine\-tune a small open\-source rubric generator via preference learning, still without requiring any human\-annotated rubrics or reference answers\. Inspired by Meta\-RewardingWuet al\.\([2025](https://arxiv.org/html/2605.30568#bib.bib9)\), we leverage reward signals from a meta\-judge to improve rubric generators\. The training pipeline proceeds in five steps: \(1\) generate diverse rubric candidates, \(2\) score them using a meta\-judge, \(3\) construct preference pairs, \(4\) fine\-tune the generator with Direct Preference Optimization \(DPO\)Rafailovet al\.\([2023](https://arxiv.org/html/2605.30568#bib.bib33)\), and \(5\) iterate by using the fine\-tuned model as the new base generator\.

We use instances from the evaluation benchmarks themselves \(Section[3\.2](https://arxiv.org/html/2605.30568#S3.SS2)\) as training instances\. No human annotations \(preference labels or scores\) are leaked; the only supervision comes from the meta\-judge comparing rubric candidates on intrinsic quality dimensions such as correctness, specificity, and coverage\. This is functionally equivalent to best\-of\-KKselection with a meta\-judge at inference time, but amortizes the cost by distilling these preferences into the generator offline\. Since the reward signal is rubric\-intrinsic and never accesses downstream human judgments, the generator cannot learn to exploit evaluation outcomes\.

Step 1: Rubric Candidate GenerationFor each instancexx, we generateKKdiverse rubric candidates\{ℛ1​\(x\),…,ℛK​\(x\)\}\\\{\\mathcal\{R\}\_\{1\}\(x\),\\ldots,\\mathcal\{R\}\_\{K\}\(x\)\\\}from the base generator𝒢\\mathcal\{G\}using high temperature \(τ=0\.8\\tau\{=\}0\.8\)\. This diversity is essential for constructing informative preference pairs\. We useK=8K\{=\}8candidates per instance\.

Step 2: Meta\-Judge EvaluationWe employ a strong meta\-judge modelℳ\\mathcal\{M\}\(e\.g\., Claude Sonnet 4\) to compare rubric candidates pairwise\. For each pair\(ℛi,ℛj\)\(\\mathcal\{R\}\_\{i\},\\mathcal\{R\}\_\{j\}\), the meta\-judge determines which rubric is superior for evaluation, considering:

- •Specificity: Are criteria specific?
- •Coverage: Do criteria address the key aspects an expert would check?
- •Discriminability: Can criteria distinguish genuinely good responses from superficially good ones?
- •Domain\-appropriateness: Do criteria reflect the required expertise level?

To mitigate positional bias, we evaluate each pair in both orders and count a win only when both orders agree\. FromK=8K\{=\}8candidates \(\(82\)=28\\binom\{8\}\{2\}=28possible pairs\), we randomly sample 10 pairs per instance to limit computational cost while ensuring each candidate participates in multiple comparisons\.

Step 3: Preference Pair ConstructionWe fit a Bradley\-Terry modelBradley and Terry \([1952](https://arxiv.org/html/2605.30568#bib.bib34)\)to the meta\-judge outcomes, estimating a strength parameterγi\\gamma\_\{i\}for each candidate by maximizing the likelihood of the observed pairwise comparisons \(details in Appendix[C](https://arxiv.org/html/2605.30568#A3)\)\. For each instance, we select the candidate with the highestγ\\gammaas the chosen rubricℛw\\mathcal\{R\}\_\{w\}and the lowest as the rejectedℛl\\mathcal\{R\}\_\{l\}, forming a preference pair\(x,ℛw,ℛl\)\(x,\\,\\mathcal\{R\}\_\{w\},\\,\\mathcal\{R\}\_\{l\}\)\.

Step 4: DPO Fine\-tuningWe fine\-tune the rubric generator𝒢θ\\mathcal\{G\}\_\{\\theta\}using DPO, which directly optimizes the policy to preferℛw\\mathcal\{R\}\_\{w\}overℛl\\mathcal\{R\}\_\{l\}:

ℒDPO=−𝔼​\[log⁡σ​\(β​rw−β​rl\)\],\\mathcal\{L\}\_\{\\text\{DPO\}\}=\-\\mathbb\{E\}\\\!\\left\[\\log\\sigma\\\!\\left\(\\beta\\,r\_\{w\}\-\\beta\\,r\_\{l\}\\right\)\\right\],\(2\)whereri=log⁡πθ​\(ℛi∣x\)πref​\(ℛi∣x\)r\_\{i\}=\\log\\frac\{\\pi\_\{\\theta\}\(\\mathcal\{R\}\_\{i\}\\mid x\)\}\{\\pi\_\{\\text\{ref\}\}\(\\mathcal\{R\}\_\{i\}\\mid x\)\}is the log\-ratio between the trained policyπθ\\pi\_\{\\theta\}and the frozen reference policyπref\\pi\_\{\\text\{ref\}\}, andβ\\betacontrols the KL constraint strength\.

Step 5: Iterative TrainingThe fine\-tuned generator from iterationttserves as both the base model and the reference policy for iterationt\+1t\{\+\}1:

𝒢\(t\+1\)=DPO​\(𝒢\(t\),𝒟\(t\)\),\\mathcal\{G\}^\{\(t\+1\)\}=\\text\{DPO\}\\\!\\left\(\\mathcal\{G\}^\{\(t\)\},\\;\\mathcal\{D\}^\{\(t\)\}\\right\),\(3\)where𝒟\(t\)\\mathcal\{D\}^\{\(t\)\}denotes the preference pairs constructed from rubric candidates generated by𝒢\(t\)\\mathcal\{G\}^\{\(t\)\}and scored by the meta\-judge\. Steps 1–4 are repeated with the updated generator, allowing it to progressively improve beyond the capability of the base model\. Full implementation details including the hyperparameters are provided in Appendix[C](https://arxiv.org/html/2605.30568#A3)\.

## 4Experiments

We evaluate our approach on four benchmarks \(Section[3\.2](https://arxiv.org/html/2605.30568#S3.SS2)\) across four models\. We first compare our training\-free rubric generation against existing baselines, then evaluate the fine\-tuned rubric generator \(Section[4\.3](https://arxiv.org/html/2605.30568#S4.SS3)\)\.

#### Baselines

We compare against Prometheus 2Kimet al\.\([2024b](https://arxiv.org/html/2605.30568#bib.bib7)\), an open\-source LLM fine\-tuned specifically for rubric\-based evaluation, and three training\-free rubric\-based methods: DnA\-EvalLiet al\.\([2025](https://arxiv.org/html/2605.30568#bib.bib6)\)decomposes evaluation into 3 weighted aspect\-specific questions and aggregates scores per aspect; CheckEvalLeeet al\.\([2025](https://arxiv.org/html/2605.30568#bib.bib4)\)generates a fixed checklist of 20 binary questions per benchmark and scores responses by the proportion of affirmative answers; RubricHubLiet al\.\([2026](https://arxiv.org/html/2605.30568#bib.bib19)\)generates fine\-grained binary criteria grounded in a reference response and aggregates via weighted sum\. All three methods support pointwise evaluation by producing a numeric score per response\. For pairwise benchmarks, we score both responses independently and select the higher\-scoring one; ties \(equal scores\) are resolved by random assignment, equivalent to 50% accuracy on tied pairs\.

Additionally, we report results withhuman\-crafted rubricsprovided by the benchmark authors\. We evaluate at both dataset\-specific \(all benchmarks\) and instance\-specific granularity \(BiGGenonly, Section[3\.2](https://arxiv.org/html/2605.30568#S3.SS2)\)\. We consider the latter as an oracle upper bound since such fine\-grained rubrics are infeasible to obtain at scale in real\-world settings\. Appendix[D](https://arxiv.org/html/2605.30568#A4)reports further details of the baseline methods\.

PairwisePointwiseAlpacaEvalMT\-BenchBiGGenHelpSteer2BiGGenBaselineFine\-tuned ModelPrometheus 8x7BKimet al\.\([2024b](https://arxiv.org/html/2605.30568#bib.bib7)\)Dataset\-specific68\.5474\.1868\.300\.229/0\.3080\.280/0\.307Instance\-specific——69\.73—0\.366/0\.401Training\-free ApproachLlama 3\.1 70BCheckEvalLeeet al\.\([2025](https://arxiv.org/html/2605.30568#bib.bib4)\)58\.7271\.0464\.290\.239/0\.3240\.318/0\.361DnA\-EvalLiet al\.\([2025](https://arxiv.org/html/2605.30568#bib.bib6)\)62\.4677\.1570\.340\.263/0\.2990\.337/0\.353RubricHubLiet al\.\([2026](https://arxiv.org/html/2605.30568#bib.bib19)\)62\.9380\.3469\.680\.246/0\.2920\.318/0\.361Human\-crafted \(existing\) RubricsDataset\-specific68\.2280\.8770\.200\.279/0\.3730\.384/0\.382Instance\-specific——76\.15—0\.552/0\.594Qwen3 14BCheckEval60\.4475\.2465\.010\.282/0\.3920\.343/0\.392DnA\-Eval69\.7878\.8071\.010\.345/0\.4030\.431/0\.426RubricHub62\.9379\.7668\.280\.286/0\.3190\.332/0\.383Human\-crafted \(existing\) RubricsDataset\-specific70\.0979\.9172\.530\.352/0\.4310\.436/0\.450Instance\-specific——76\.68—0\.583/0\.609Claude Sonnet 4CheckEval66\.8273\.0668\.820\.375/0\.4350\.424/0\.455DnA\-Eval70\.5681\.5174\.820\.411/0\.4760\.454/0\.489RubricHub64\.6481\.7272\.220\.239/0\.2700\.325/0\.336Human\-crafted \(existing\) RubricsDataset\-specific70\.4081\.6274\.360\.432/0\.4830\.476/0\.494Instance\-specific——78\.33—0\.584/0\.598Rubrics Generation \(Our Approach\)Llama 3\.1 70BDataset\-specific67\.9179\.9169\.73∗0\.284/0\.3770\.378/0\.361∗Instance\-specific69\.7880\.9871\.74∗0\.319/0\.3790\.420/0\.434∗Qwen3 14BDataset\-specific69\.4780\.0272\.42∗0\.353/0\.4280\.440/0\.455∗Instance\-specific71\.9680\.5572\.96∗0\.374/0\.4160\.446/0\.462∗Claude Sonnet 4Dataset\-specific69\.7880\.5574\.75∗0\.410/0\.422∗0\.461/0\.453∗Instance\-specific71\.6581\.6274\.89∗0\.438/0\.4880\.477/0\.496∗Table 2:Evaluation results comparing baselines and our training\-free rubric generation\. We report human agreement \(%\) for pairwise benchmarks and Spearman/Pearson correlation for pointwise benchmarks\. The first block includes existing methods \(fine\-tuned models, training\-free baselines, and human\-crafted rubrics as oracle upper bounds\)\. Our approach \(second block\) without training data or human\-crafted rubrics yields competitive performance with existing methods \(first block\)\.Boldmarks the best result per judge model\. Results marked with∗are statistically significantly worse than the best existing method in the first block \(bootstrap test,p<0\.05p<0\.05\)\. Our approach obtains comparable results on most benchmarks despite requiring neither model training nor human\-annotated data\.
### 4\.1Experimental Setting

#### Judge Models

We use four judge models ranging from open\-source to proprietary: Llama 3\.1 8B and 70B, Qwen3 14B, and Claude Sonnet 4\. We report main results with the latter three; extra results with Llama 3\.1 8B are in Appendix[E](https://arxiv.org/html/2605.30568#A5)\. For pairwise evaluation, we mitigate positional biasShiet al\.\([2025](https://arxiv.org/html/2605.30568#bib.bib37)\)by evaluating each pair in both orders and counting disagreements as incorrect\.

#### Rubric Generator Model

As discussed in Section[3\.3](https://arxiv.org/html/2605.30568#S3.SS3), for training\-free rubric generation \(Section[4\.2](https://arxiv.org/html/2605.30568#S4.SS2)\), the judge model itself serves as the rubric generator \(𝒢=𝒥\\mathcal\{G\}=\\mathcal\{J\}\)\. For preference learning \(Section[4\.3](https://arxiv.org/html/2605.30568#S4.SS3)\), we fine\-tune Qwen3 14B as a dedicated rubric generator separate from the judge\. Claude Sonnet 4 serves as the meta\-judgeℳ\\mathcal\{M\}for collecting rubric preferences\. An ablation on different generator models and meta\-judges is reported in Appendix[G](https://arxiv.org/html/2605.30568#A7)\.

AlpacaEvalMT\-BenchBiGGenJudge: Llama 3\.1 70BHighest Training\-free†69\.7880\.9871\.74Iteration 169\.4782\.7271\.10Iteration 270\.7282\.93∗71\.92Judge: Qwen3 14BHighest Training\-free†71\.9680\.5572\.96Iteration 172\.4082\.62∗74\.67∗Iteration 272\.3482\.87∗74\.46Judge: Claude Sonnet 4Highest Training\-free†71\.6581\.7274\.89Iteration 172\.0983\.3576\.28Iteration 272\.2283\.69∗76\.96∗\(a\)Pairwise Evaluation \(human agreement %\)\.
HelpSteer2BiGGenJudge: Llama 3\.1 70BHighest Training\-free†0\.319/0\.3790\.420/0\.434Iteration 10\.326/0\.3880\.440/0\.447∗Iteration 20\.344/0\.4120\.445/0\.452∗Judge: Qwen3 14BHighest Training\-free†0\.374/0\.4160\.446/0\.462Iteration 10\.394/0\.4310\.439/0\.459Iteration 20\.406/0\.441∗0\.441/0\.461Judge: Claude Sonnet 4Highest Training\-free†0\.438/0\.4880\.477/0\.496Iteration 10\.440/0\.4710\.490/0\.504Iteration 20\.464/0\.5030\.510/0\.521∗\(b\)Pointwise Evaluation \(Spearman/Pearson\)\.

Table 3:Results with fine\-tuned rubric generator \(Qwen3 14B\)\. The fine\-tuned generator produces*instance\-specific*rubrics for each evaluation instance\.Boldmarks the best result per judge model\. Results with an asterisk \(∗\) indicate statistically significant improvement over the highest training\-free baseline \(bootstrap test,p<0\.05p<0\.05\)\.†Training\-free rows report the best per\-model results across all methods in Table[2](https://arxiv.org/html/2605.30568#S4.T2), excluding oracle human\-crafted instance\-specific rubrics\. The fine\-tuned generator consistently outperforms all existing methods and narrows the gap to oracle human\-crafted rubrics\.

### 4\.2Results with Baselines and Training\-Free Rubric Generation

Table[2](https://arxiv.org/html/2605.30568#S4.T2)presents results comparing our training\-free rubric generation against baselines and human\-crafted rubrics, reporting human agreement \(%\) for pairwise benchmarks and Spearman/Pearson correlation for pointwise benchmarks\.

#### Existing Baselines

Among the three existing methods, DnA\-Eval consistently achieves the strongest results, outperforming CheckEval and RubricHub on most benchmarks—the only exception is MT\-Bench\. Prometheus 2 \(8×\\times7B\) underperforms the best training\-free methods despite being explicitly trained for rubric\-based evaluation, suggesting that general LLMs with high\-quality rubrics can surpass specialized LLM judges\.

Despite being the strongest baseline, DnA\-Eval still falls short of human\-crafted dataset\-specific rubrics on most benchmarks \(e\.g\., 69\.78% vs\. 70\.09% on AlpacaEval, 78\.80% vs\. 79\.91% on MT\-Bench, both with Qwen3 14B\)\. We attribute this gap to*annotation alignment*: human\-crafted rubrics mirror the exact evaluation protocol used by annotators \(e\.g\., HelpSteer2’s rubric uses helpfulness, correctness, and coherence, the same dimensions averaged to compute the ground\-truth score\), whereas existing methods generate ad\-hoc aspects that may not align with the annotation criteria\.

Human\-crafted instance\-specific rubrics \(available only forBiGGen\) further outperform all automated methods by a large margin \(78\.33% vs\. 74\.82%, 0\.584 vs\. 0\.454 with Claude Sonnet 4\), highlighting the value of fine\-grained, expert\-crafted rubrics\. However, such rubrics require domain expertise that is expensive and often infeasible to collect at scale\.

#### Our Approach

Our training\-free rubric generation achieves competitive and even superior results to the existing methods without any training data, human\-crafted rubrics, or reference answers\. Generated instance\-specific rubrics consistently outperform dataset\-specific across all judge models, again confirming that fine\-grained rubrics work better even without reference answers\. Notably, our approach achieves state\-of\-the\-art results on AlpacaEval and HelpSteer2 with all models\.

The majority of our results arenotstatistically significantly worse than the best existing method \(reported in Table[2](https://arxiv.org/html/2605.30568#S4.T2)without an asterisk \(∗\); bootstrap test,p\>0\.05p\>0\.05\)\. The only significant gaps appear onBiGGenwith human\-crafted instance\-specific rubrics, which is considered an oracle upper bound\. Appendix[F](https://arxiv.org/html/2605.30568#A6)reports more details including the significance testing\.

### 4\.3Results with Fine\-tuned Rubric Generator

We apply the preference fine\-tuned rubric generator exclusively for*instance\-specific*rubric generation\. Dataset\-specific rubrics, in principle, would benefit less from preference learning since their quality is already high with the base models\. We train separate generators for pairwise and pointwise evaluation, as the two settings demand different rubric properties: pairwise rubrics must highlight dimensions that differentiate between two responses, while pointwise rubrics must define criteria for absolute scoring\. Table[3](https://arxiv.org/html/2605.30568#S4.T3)presents results using rubrics from the fine\-tuned Qwen3 14B generator over two iterations\.

BaseIter 1Iter 2Avg\.Δ\\Deltavs\. BasePairwise agreement—\+1\.15\+1\.53Pointwise Spearman—\+0\.017\+0\.030Structural properties \(pairwise\)Avg\. length \(chars\)621630634Avg\. criteria / rubric4\.864\.804\.89Prescriptive \(%\)↓\\downarrow47\.239\.524\.9Structural properties \(pointwise\)Avg\. length \(chars\)637619699Avg\. criteria / rubric4\.94\.95\.0Prescriptive \(%\)↓\\downarrow38\.633\.50\.9Meta\-judge statistics \(pairwise\)Best\-candidate win rate↑\\uparrow0\.300\.53—Preference margin↓\\downarrow0\.800\.67—Same\-instance divergence \(pairwise\)Word Jaccard vs\. Base↓\\downarrow—0\.5360\.498Table 4:Rubric properties across iterations \(Qwen3 14B generator\)\. Prescriptive \(%\) counts criteria beginning with “The response…”\. Meta\-judge statistics are computed from theK=8K\{=\}8candidates sampled during reward signal collection: best\-candidate win rate is the fraction of comparisons won by the top candidate; preference margin is the win\-rate gap between chosen and rejected candidates in preference pairs\. Same\-instance divergence measures word overlap of the same instance’s rubric across iterations\.The fine\-tuned generator consistently improves over training\-free rubric generation across all benchmarks\. Many of these improvements are statistically significant \(with an asterisk∗, e\.g\., 83\.69% vs\. 81\.72% on MT\-Bench and 76\.96% vs\. 74\.89% onBiGGenwith Claude Sonnet 4\)\. Importantly, the rubrics from the fine\-tuned generator outperform most existing methods on every benchmark\. Again, the only exception is the remaining gap to the oracle onBiGGen, though it narrows with stronger judges \(4\.23 to 1\.37 percentage points from Llama 3\.1 70B to Claude Sonnet 4\)\.

Notably, the fine\-tuned Qwen3 14B generator outperforms Claude Sonnet 4 at rubric generation despite being a much smaller model: rubrics from Qwen3 14B achieve 83\.69% on MT\-Bench and 76\.96% onBiGGen\(with Claude as judge\), compared to 81\.62% and 74\.89% when Claude generates its own rubrics \(Table[2](https://arxiv.org/html/2605.30568#S4.T2)\)\. Moreover, the fine\-tuned Qwen3 14B paired with Qwen3 14B as judge \(72\.34% AlpacaEval, 82\.87% MT\-Bench\) outperforms Claude Sonnet 4 as both generator and judge \(71\.65%, 81\.62%\), demonstrating that overall evaluation quality can be decoupled from judge capability\. This suggests that under limited computational budgets, investing in a specialized rubric generator can yield greater returns than scaling up the judge model alone\.

Furthermore, the iterative training procedure \(Section[3\.4](https://arxiv.org/html/2605.30568#S3.SS4)\) yields gains across both iterations\. Compared to the base Qwen3 14B generator, the first iteration improves pairwise agreement by \+1\.15 percentage points on average, and the second iteration further increases this to \+1\.53 \(Table[4](https://arxiv.org/html/2605.30568#S4.T4)\)\. We observe a similar trend on pointwise evaluation \(Spearman improved by \+0\.017 after the first iteration, \+0\.030 after the second\)\. However, the marginal gain diminishes from Iteration 1 to Iteration 2 \(\+1\.15 vs\. \+0\.38 for pairwise; \+0\.017 vs\. \+0\.013 for Spearman\), suggesting potential performance convergence\. We do not explore further iterations due to computational constraints\.

Finally, we conduct several ablation studies, including fine\-tuning a rubric generator jointly on both pairwise and pointwise instances, using a different model as the meta\-judge for preference collection, and fine\-tuning a smaller rubric generator model \(Llama 3\.1 8B\)\. Due to space limitations, these ablations are reported in Appendix[G](https://arxiv.org/html/2605.30568#A7)\.

#### Qualitative Analysis

We compare rubrics produced by Qwen3 14B as the base, Iteration 1, and Iteration 2 models \(Table[4](https://arxiv.org/html/2605.30568#S4.T4)\)\. Rubric length and criteria count remain stable across iterations, indicating that improvement is qualitative rather than structural\. The key shift is in rubric*framing*: the base model uses prescriptive criteria \(“The response clearly explains …”\) while Iteration 2 reframes around evaluation dimensions \(“Accuracy of …”, “Clarity and coherence in …”\)\. Prescriptive criteria drop from 47\.2% to 24\.9% for pairwise rubrics, with an even sharper decline for pointwise \(38\.6%→\\to0\.9%\)\. The same\-instance divergence between base and Iteration 2 is only 0\.498 \(i\.e\., fewer than half of the words are shared\), confirming that preference learning substantially rewrites rubrics rather than merely paraphrasing them\. This reframing provides stronger discriminative signal for pairwise comparison, explaining the larger gains on pairwise benchmarks\. The meta\-judge statistics further support this: the best\-candidate win rate increases from 0\.30 \(base\) to 0\.53 \(Iteration 1\), indicating that stronger rubric candidates emerge as training progresses\. We provide extra analysis and three case studies to further explain the observed performance gains in Appendix[H](https://arxiv.org/html/2605.30568#A8)\.

## 5Conclusion

We presented an approach to improve LLM\-as\-a\-Judge through automatically generated evaluation rubrics, requiring no human\-annotated data or reference answers\. Our training\-free rubric generation achieves competitive performance with existing methods across four benchmarks\. We further proposed a preference learning method to iteratively fine\-tune a rubric generator via meta\-judge feedback, consistently outperforming all existing baselines\. Notably, a fine\-tuned Qwen3 14B generator produces higher\-quality rubrics than Claude Sonnet 4, demonstrating that evaluation quality can be decoupled from judge model scale through specialized rubric generation\.

Future work includes incorporating reference answers when available to improve rubric specificity, evaluating cross\-domain generalization of the fine\-tuned generator, and exploring alternative reward signals beyond meta\-judge preferences\.

## Limitations

Our approach requires multiple LLM calls per evaluation instance \(rubric generation followed by judgment\), increasing computational cost compared to direct scoring\. The preference learning procedure relies on a capable meta\-judge for reward signal collection, which may limit accessibility in resource\-constrained settings\. We explored only two iterations due to computational constraints; the sustained improvement trend suggests further iterations may yield additional gains, but potential saturation or degradation remains unexplored\. We do not investigate the trade\-off between rubric generator models and LLM judge models, which would be beneficial for practical usage with limited resources\. We train and evaluate on the same benchmarks; an out\-of\-domain evaluation \(training on one benchmark and testing on another\) would better test generalization, but few meta\-evaluation benchmarks with reliable human judgments exist beyond the ones we use\. We do not compare the fine\-tuned generator against best\-of\-KKselection at inference time \(generatingKKrubrics and selecting the best via meta\-judge\), which would isolate the benefit of amortizing preferences into the generator versus paying the cost at inference\. We only evaluate under pairwise and pointwise settings, omitting reference\-based evaluation where the judge compares a response against a gold answer\. This is partly because reference answers are scarce in many real\-world datasets, but extending our rubric generation approach to the reference\-based setting remains future work\. Finally, our evaluation is limited to mostly general domain benchmarks\. An extension to domain\-specific benchmarks \(e\.g\., medical, legal, finance\) would be valuable\.

## Ethics Statement

We used AI\-based assistance to improve the language and clarity of this paper, and as a coding aid during implementation\. All research ideas, experimental design, analysis, and conclusions are entirely the work of the authors\.

## References

- Rank analysis of incomplete block designs: i\. the method of paired comparisons\.Biometrika39\(3/4\),pp\. 324–345\.External Links:ISSN 00063444, 14643510,[Link](http://www.jstor.org/stable/2334029)Cited by:[§3\.4](https://arxiv.org/html/2605.30568#S3.SS4.p5.5)\.
- N\. Chen, Z\. Hu, Q\. Zou, J\. Wu, Q\. Wang, B\. Hooi, and B\. He \(2025\)Judgelrm: large reasoning models as a judge\.arXiv preprint arXiv:2504\.00050\.Cited by:[§2\.1](https://arxiv.org/html/2605.30568#S2.SS1.p1.1)\.
- D\. Deshpande, S\. S\. Ravi, S\. CH\-Wang, B\. Mielczarek, A\. Kannappan, and R\. Qian \(2024\)Glider: grading llm interactions and decisions using explainable ranking\.arXiv preprint arXiv:2412\.14140\.Cited by:[§1](https://arxiv.org/html/2605.30568#S1.p2.1),[§2\.1](https://arxiv.org/html/2605.30568#S2.SS1.p1.1)\.
- T\. Dettmers, A\. Pagnoni, A\. Holtzman, and L\. Zettlemoyer \(2023\)QLoRA: efficient finetuning of quantized LLMs\.InThirty\-seventh Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=OUIFPHEgJU)Cited by:[Appendix C](https://arxiv.org/html/2605.30568#A3.SS0.SSS0.Px2.p1.1)\.
- S\. Doddapaneni, M\. S\. U\. R\. Khan, D\. Venkatesh, R\. Dabre, A\. Kunchukuttan, and M\. M\. Khapra \(2025\)Cross\-lingual auto evaluation for assessing multilingual LLMs\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 29297–29329\.External Links:[Link](https://aclanthology.org/2025.acl-long.1419/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1419),ISBN 979\-8\-89176\-251\-0Cited by:[§2\.1](https://arxiv.org/html/2605.30568#S2.SS1.p1.1)\.
- Y\. Dubois, B\. Galambosi, P\. Liang, and T\. B\. Hashimoto \(2024\)Length\-controlled alpacaeval: a simple way to debias automatic evaluators\.arXiv preprint arXiv:2404\.04475\.Cited by:[§1](https://arxiv.org/html/2605.30568#S1.p1.1),[Table 1](https://arxiv.org/html/2605.30568#S2.T1.1.1.4.1),[§3\.2](https://arxiv.org/html/2605.30568#S3.SS2.SSS0.Px2.p1.1)\.
- M\. Furuhashi, K\. Nakayama, T\. Kodama, and S\. Sugawara \(2025\)Are checklists really useful for automatic evaluation of generative tasks?\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 10630–10653\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.538/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.538),ISBN 979\-8\-89176\-332\-6Cited by:[§1](https://arxiv.org/html/2605.30568#S1.p2.1),[§2\.2](https://arxiv.org/html/2605.30568#S2.SS2.p1.1)\.
- A\. Gunjal, A\. Wang, E\. Lau, V\. Nath, Y\. He, B\. Liu, and S\. M\. Hendryx \(2026\)Rubrics as rewards: reinforcement learning beyond verifiable domains\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=c1bTcrDmt4)Cited by:[§2\.3](https://arxiv.org/html/2605.30568#S2.SS3.p1.1)\.
- T\. Gupta, S\. Shandilya, X\. Zhang, R\. Madhavan, S\. Ghosh, C\. Bansal, H\. Yao, and S\. Rajmohan \(2025\)CARMO: dynamic criteria generation for context aware reward modelling\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 2202–2261\.External Links:[Link](https://aclanthology.org/2025.findings-acl.114/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.114),ISBN 979\-8\-89176\-256\-5Cited by:[§2\.3](https://arxiv.org/html/2605.30568#S2.SS3.p1.1)\.
- S\. Kim, J\. Shin, Y\. Cho, J\. Jang, S\. Longpre, H\. Lee, S\. Yun, S\. Shin, S\. Kim, J\. Thorne, and M\. Seo \(2024a\)Prometheus: inducing fine\-grained evaluation capability in language models\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=8euJaTveKw)Cited by:[§1](https://arxiv.org/html/2605.30568#S1.p2.1),[§2\.1](https://arxiv.org/html/2605.30568#S2.SS1.p1.1)\.
- S\. Kim, J\. Suk, J\. Y\. Cho, S\. Longpre, C\. Kim, D\. Yoon, G\. Son, Y\. Cho, S\. Shafayat, J\. Baek, S\. H\. Park, H\. Hwang, J\. Jo, H\. Cho, H\. Shin, S\. Lee, H\. Oh, N\. Lee, N\. Ho, S\. J\. Joo, M\. Ko, Y\. Lee, H\. Chae, J\. Shin, J\. Jang, S\. Ye, B\. Y\. Lin, S\. Welleck, G\. Neubig, M\. Lee, K\. Lee, and M\. Seo \(2025\)The BiGGen bench: a principled benchmark for fine\-grained evaluation of language models with language models\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 5877–5919\.External Links:[Link](https://aclanthology.org/2025.naacl-long.303/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.303),ISBN 979\-8\-89176\-189\-6Cited by:[§1](https://arxiv.org/html/2605.30568#S1.p2.1),[§2\.2](https://arxiv.org/html/2605.30568#S2.SS2.p1.1),[Table 1](https://arxiv.org/html/2605.30568#S2.T1.1.1.3.1),[Table 1](https://arxiv.org/html/2605.30568#S2.T1.1.1.6.1),[§3\.2](https://arxiv.org/html/2605.30568#S3.SS2.SSS0.Px1.p1.2)\.
- S\. Kim, J\. Suk, S\. Longpre, B\. Y\. Lin, J\. Shin, S\. Welleck, G\. Neubig, M\. Lee, K\. Lee, and M\. Seo \(2024b\)Prometheus 2: an open source language model specialized in evaluating other language models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 4334–4353\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.248/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.248)Cited by:[Appendix D](https://arxiv.org/html/2605.30568#A4.SS0.SSS0.Px1.p1.2),[Table 6](https://arxiv.org/html/2605.30568#A5.T6.1.3.1),[§1](https://arxiv.org/html/2605.30568#S1.p2.1),[§2\.1](https://arxiv.org/html/2605.30568#S2.SS1.p1.1),[§4](https://arxiv.org/html/2605.30568#S4.SS0.SSS0.Px1.p1.1),[Table 2](https://arxiv.org/html/2605.30568#S4.T2.13.18.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with pagedattention\.InProceedings of the 29th Symposium on Operating Systems Principles,SOSP ’23,New York, NY, USA,pp\. 611–626\.External Links:ISBN 9798400702297,[Link](https://doi.org/10.1145/3600006.3613165),[Document](https://dx.doi.org/10.1145/3600006.3613165)Cited by:[Appendix C](https://arxiv.org/html/2605.30568#A3.SS0.SSS0.Px1.p1.1)\.
- J\. O\. J\. Leang, G\. Hong, W\. Li, and S\. B\. Cohen \(2025\)Theorem prover as a judge for synthetic data generation\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 29941–29977\.External Links:[Link](https://aclanthology.org/2025.acl-long.1448/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1448),ISBN 979\-8\-89176\-251\-0Cited by:[§1](https://arxiv.org/html/2605.30568#S1.p1.1)\.
- Y\. Lee, J\. Kim, J\. Kim, H\. Cho, J\. Kang, P\. Kang, and N\. Kim \(2025\)CheckEval: a reliable LLM\-as\-a\-judge framework for evaluating text generation using checklists\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 15771–15798\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.796/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.796),ISBN 979\-8\-89176\-332\-6Cited by:[Appendix D](https://arxiv.org/html/2605.30568#A4.SS0.SSS0.Px3.p1.3),[Table 6](https://arxiv.org/html/2605.30568#A5.T6.1.7.1),[§1](https://arxiv.org/html/2605.30568#S1.p2.1),[§2\.2](https://arxiv.org/html/2605.30568#S2.SS2.p1.1),[§4](https://arxiv.org/html/2605.30568#S4.SS0.SSS0.Px1.p1.1),[Table 2](https://arxiv.org/html/2605.30568#S4.T2.13.23.1)\.
- M\. Li, Z\. Liu, S\. Deng, S\. Joty, N\. Chen, and M\. Kan \(2025\)DnA\-eval: enhancing large language model evaluation through decomposition and aggregation\.InProceedings of the 31st International Conference on Computational Linguistics,O\. Rambow, L\. Wanner, M\. Apidianaki, H\. Al\-Khalifa, B\. D\. Eugenio, and S\. Schockaert \(Eds\.\),Abu Dhabi, UAE,pp\. 2277–2290\.External Links:[Link](https://aclanthology.org/2025.coling-main.156/)Cited by:[Appendix D](https://arxiv.org/html/2605.30568#A4.SS0.SSS0.Px2.p1.7),[Table 6](https://arxiv.org/html/2605.30568#A5.T6.1.8.1),[§1](https://arxiv.org/html/2605.30568#S1.p2.1),[§2\.2](https://arxiv.org/html/2605.30568#S2.SS2.p1.1),[§4](https://arxiv.org/html/2605.30568#S4.SS0.SSS0.Px1.p1.1),[Table 2](https://arxiv.org/html/2605.30568#S4.T2.13.24.1)\.
- S\. Li, J\. Zhao, M\. Wei, H\. Ren, Y\. Zhou, J\. Yang, S\. Liu, K\. Zhang, and W\. Chen \(2026\)RubricHub: a comprehensive and highly discriminative rubric dataset via automated coarse\-to\-fine generation\.arXiv preprint arXiv:2601\.08430\.Cited by:[Appendix D](https://arxiv.org/html/2605.30568#A4.SS0.SSS0.Px4.p1.9),[Table 6](https://arxiv.org/html/2605.30568#A5.T6.1.9.1),[§1](https://arxiv.org/html/2605.30568#S1.p2.1),[§2\.2](https://arxiv.org/html/2605.30568#S2.SS2.p1.1),[§3\.3](https://arxiv.org/html/2605.30568#S3.SS3.SSS0.Px2.p2.1),[§4](https://arxiv.org/html/2605.30568#S4.SS0.SSS0.Px1.p1.1),[Table 2](https://arxiv.org/html/2605.30568#S4.T2.13.25.1)\.
- C\. Lin \(2004\)ROUGE: a package for automatic evaluation of summaries\.InText Summarization Branches Out,Barcelona, Spain,pp\. 74–81\.External Links:[Link](https://aclanthology.org/W04-1013/)Cited by:[§1](https://arxiv.org/html/2605.30568#S1.p1.1)\.
- K\. Papineni, S\. Roukos, T\. Ward, and W\. Zhu \(2002\)Bleu: a method for automatic evaluation of machine translation\.InProceedings of the 40th Annual Meeting of the Association for Computational Linguistics,P\. Isabelle, E\. Charniak, and D\. Lin \(Eds\.\),Philadelphia, Pennsylvania, USA,pp\. 311–318\.External Links:[Link](https://aclanthology.org/P02-1040/),[Document](https://dx.doi.org/10.3115/1073083.1073135)Cited by:[§1](https://arxiv.org/html/2605.30568#S1.p1.1)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn \(2023\)Direct preference optimization: your language model is secretly a reward model\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 53728–53741\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/a85b405ed65c6477a4fe8302b5e06ce7-Paper-Conference.pdf)Cited by:[§3\.4](https://arxiv.org/html/2605.30568#S3.SS4.p1.1)\.
- M\. J\. Ryan, Y\. Zhang, A\. Salunkhe, Y\. Chu, D\. Xu, and D\. Yang \(2026\)AutoMetrics: approximate human judgments with automatically generated evaluators\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=ymJuBifPUy)Cited by:[§1](https://arxiv.org/html/2605.30568#S1.p2.1),[§2\.2](https://arxiv.org/html/2605.30568#S2.SS2.p1.1),[§3\.3](https://arxiv.org/html/2605.30568#S3.SS3.SSS0.Px2.p2.1)\.
- L\. Shi, C\. Ma, W\. Liang, X\. Diao, W\. Ma, and S\. Vosoughi \(2025\)Judging the judges: a systematic study of position bias in LLM\-as\-a\-judge\.InProceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia\-Pacific Chapter of the Association for Computational Linguistics,K\. Inui, S\. Sakti, H\. Wang, D\. F\. Wong, P\. Bhattacharyya, B\. Banerjee, A\. Ekbal, T\. Chakraborty, and D\. P\. Singh \(Eds\.\),Mumbai, India,pp\. 292–314\.External Links:[Link](https://aclanthology.org/2025.ijcnlp-long.18/),[Document](https://dx.doi.org/10.18653/v1/2025.ijcnlp-long.18),ISBN 979\-8\-89176\-298\-5Cited by:[§4\.1](https://arxiv.org/html/2605.30568#S4.SS1.SSS0.Px1.p1.1)\.
- V\. Viswanathan, Y\. Sun, X\. Kong, M\. Cao, G\. Neubig, and T\. Wu \(2025\)Checklists are better than reward models for aligning language models\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=RPRqKhjrr6)Cited by:[§2\.3](https://arxiv.org/html/2605.30568#S2.SS3.p1.1)\.
- P\. Wang, L\. Li, L\. Chen, Z\. Cai, D\. Zhu, B\. Lin, Y\. Cao, L\. Kong, Q\. Liu, T\. Liu, and Z\. Sui \(2024a\)Large language models are not fair evaluators\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 9440–9450\.External Links:[Link](https://aclanthology.org/2024.acl-long.511/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.511)Cited by:[§1](https://arxiv.org/html/2605.30568#S1.p1.1)\.
- T\. Wang and C\. Xiong \(2025\)AutoRule: reasoning chain\-of\-thought extracted rule\-based rewards improve preference learning\.arXiv preprint arXiv:2506\.15651\.Cited by:[§2\.3](https://arxiv.org/html/2605.30568#S2.SS3.p1.1)\.
- Z\. Wang, Y\. Dong, O\. Delalleau, J\. Zeng, G\. Shen, D\. Egert, J\. J\. Zhang, M\. N\. Sreedhar, and O\. Kuchaiev \(2024b\)HelpSteer 2: open\-source dataset for training top\-performing reward models\.InThe Thirty\-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=PvVKUFhaNy)Cited by:[Table 1](https://arxiv.org/html/2605.30568#S2.T1.1.1.2.1),[§3\.2](https://arxiv.org/html/2605.30568#S3.SS2.SSS0.Px1.p1.2)\.
- T\. Wu, W\. Yuan, O\. Golovneva, J\. Xu, Y\. Tian, J\. Jiao, J\. E\. Weston, and S\. Sukhbaatar \(2025\)Meta\-rewarding language models: self\-improving alignment with LLM\-as\-a\-meta\-judge\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 11537–11554\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.583/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.583),ISBN 979\-8\-89176\-332\-6Cited by:[§2\.1](https://arxiv.org/html/2605.30568#S2.SS1.p1.1),[§3\.4](https://arxiv.org/html/2605.30568#S3.SS4.p1.1)\.
- S\. Ye, D\. Kim, S\. Kim, H\. Hwang, S\. Kim, Y\. Jo, J\. Thorne, J\. Kim, and M\. Seo \(2024\)FLASK: fine\-grained language model evaluation based on alignment skill sets\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=CYmF38ysDa)Cited by:[§2\.2](https://arxiv.org/html/2605.30568#S2.SS2.p1.1)\.
- W\. Yuan, R\. Y\. Pang, K\. Cho, X\. Li, S\. Sukhbaatar, J\. Xu, and J\. E\. Weston \(2024\)Self\-rewarding language models\.InForty\-first International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=0NphYCmgua)Cited by:[§1](https://arxiv.org/html/2605.30568#S1.p1.1)\.
- Z\. Zeng, J\. Yu, T\. Gao, Y\. Meng, T\. Goyal, and D\. Chen \(2024\)Evaluating large language models at evaluating instruction following\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=tr0KidwPLc)Cited by:[§1](https://arxiv.org/html/2605.30568#S1.p1.1)\.
- T\. Zhang, V\. Kishore, F\. Wu, K\. Q\. Weinberger, and Y\. Artzi \(2020\)BERTScore: evaluating text generation with bert\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=SkeHuCVFDr)Cited by:[§1](https://arxiv.org/html/2605.30568#S1.p1.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica \(2023\)Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.InProceedings of the 37th International Conference on Neural Information Processing Systems,NIPS ’23,Red Hook, NY, USA\.Cited by:[Table 1](https://arxiv.org/html/2605.30568#S2.T1.1.1.5.1),[§3\.2](https://arxiv.org/html/2605.30568#S3.SS2.SSS0.Px2.p1.1)\.

## Appendix APrompt Templates

We list all prompt templates used in our system\. Each box shows the system and user messages sent to the model\.

Rubric Generation Prompts

Instance\-specific rubric generation \(pointwise\)System:You are an expert evaluator designing assessment criteria\. Given a task prompt, create a concise evaluation rubric with 3\-\-5 specific criteria for scoring the quality of responses to this prompt on a 1\-\-5 scale\.Focus on what makes a response good or bad FOR THIS SPECIFIC TASK\. Consider the task type \(factual Q&A, creative writing, coding, reasoning, summarization, etc\.\) and what a high\-quality answer requires\.Output ONLY a numbered list of criteria\. Each criterion should be one clear sentence\.User:\[Prompt\] \{prompt\} Write the evaluation rubric for scoring responses to the above prompt\.

Dataset\-specific rubric generation \(pointwise\)System:You are an expert evaluator designing assessment criteria for scoring AI assistant responses\. Your task is to create a general\-purpose evaluation rubric that can be applied to ANY type of prompt\-response pair, where each response is scored individually on a 1\-\-5 scale\.The rubric should cover the key dimensions that distinguish a good AI response from a poor one, regardless of the specific task\.Output ONLY a numbered list of 3\-\-5 criteria\. Each criterion should be one clear sentence describing what to evaluate\.User:Create a general\-purpose evaluation rubric for scoring individual AI assistant responses on a 1\-\-5 scale\. The rubric should be applicable to any type of task \(factual Q&A, creative writing, coding, reasoning, summarization, advice, etc\.\) without any task\-specific or instance\-specific information\.Write the evaluation rubric\.

Instance\-specific rubric generation \(pairwise\)System:You are an expert evaluator designing assessment criteria\. Given a task instruction, create a concise evaluation rubric with 3\-\-5 specific criteria for judging the quality of responses to this instruction\.Focus on what makes a response good or bad FOR THIS SPECIFIC TASK\. Consider the task type \(factual Q&A, creative writing, coding, reasoning, summarization, etc\.\) and what a high\-quality answer requires\.Output ONLY a numbered list of criteria\. Each criterion should be one clear sentence\.User:\[Instruction\] \{instruction\} Write the evaluation rubric for judging responses to the above instruction\.

Dataset\-specific rubric generation \(pairwise\)System:You are an expert evaluator designing assessment criteria for comparing AI assistant responses\. Your task is to create a general\-purpose evaluation rubric that can be applied to ANY type of instruction\-response pair\.The rubric should cover the key dimensions that distinguish a good AI response from a poor one, regardless of the specific task\.Output ONLY a numbered list of 3\-\-5 criteria\. Each criterion should be one clear sentence describing what to evaluate\.User:Create a general\-purpose evaluation rubric for judging pairwise comparisons of AI assistant responses\. The rubric should be applicable to any type of task \(factual Q&A, creative writing, coding, reasoning, summarization, advice, etc\.\) without any task\-specific or instance\-specific information\.Write the evaluation rubric\.

Judge Prompts

Pointwise judgeSystem:You are a helpful assistant that evaluates the quality of AI responses\. You will be given a prompt, an evaluation rubric, and an AI response\. Use the rubric criteria to assess the response and assign a score from 1 to 5\.Scoring scale: 1 = Poor \-\- The response fails to address the prompt or has critical issues 2 = Below Average \-\- The response partially addresses the prompt but has significant shortcomings 3 = Average \-\- The response adequately addresses the prompt with some room for improvement 4 = Good \-\- The response effectively addresses the prompt with minor issues 5 = Excellent \-\- The response thoroughly and skillfully addresses the promptYou must output ONLY a single JSON object with two keys: \- "score": an integer from 1 to 5 \- "reason": a brief one\-sentence explanation referencing the rubric criteriaUser:\[Prompt\] \{prompt\} \[Evaluation Rubric\] \{rubric\} \[Response\] \{response\} Based on the rubric, score this response from 1 to 5\. Output only the JSON object\.

Pairwise judgeSystem:You are a helpful assistant that evaluates the quality of AI responses\. You will be given an instruction, an evaluation rubric tailored to that instruction, and two responses \(Response A and Response B\)\. Use the rubric criteria to compare the responses and decide which one is better\.You must output ONLY a single JSON object with two keys: \- "winner": either "A" or "B" \(the letter of the better response\) \- "reason": a brief one\-sentence explanation referencing the rubric criteriaUser:\[Instruction\] \{instruction\} \[Evaluation Rubric\] \{rubric\} \[Response A\] \{response\_a\} \[Response B\] \{response\_b\} Based on the rubric, which response is better? Output only the JSON object\.

Meta\-Judge and Reward Signal Collection Prompts

Meta\-judge \(rubric comparison\)System:You are a meta\-evaluator that assesses the quality of evaluation rubrics\. You will be given a task prompt and two candidate rubrics \(Rubric A and Rubric B\)\. Determine which rubric would be more effective for evaluating AI responses to the given prompt\.Consider these dimensions: 1\. Specificity: Are criteria concrete and testable \(yes/no answerable\), or vague and subjective? 2\. Coverage: Do criteria address the key aspects an expert would check? 3\. Discriminability: Can these criteria distinguish genuinely good responses from superficially good ones? 4\. Domain\-appropriateness: Do criteria reflect the expertise level the task requires?Output ONLY a JSON object: \{"winner": "A" or "B", "reason": "brief explanation"\}User:\[Task Prompt\] \{prompt\} \[Rubric A\] \{rubric\_a\} \[Rubric B\] \{rubric\_b\} Which rubric is better for evaluating responses to this task? Output only the JSON object\.

Rubric candidate generation \(for reward collection\)System:You are an expert evaluator designing assessment criteria for AI responses\. Given a task prompt, generate specific evaluation criteria that can each be answered with a clear YES or NO when checking a response\.Guidelines: \- Each criterion must describe a concrete, observable requirement \-\- not an abstract quality dimension like "clarity" or "accuracy\." \- Criteria should be specific to THIS prompt, capturing what a domain expert would check for in a high\-quality answer\. \- Include both content requirements \(what must be covered\) and safety/quality guards \(what must be avoided\)\. \- Assign each criterion a weight reflecting its importance: Critical \-\- essential; failure indicates a serious flaw Major \-\- important for a good response but not essential Minor \-\- desirable but lower impactOutput ONLY a JSON array of 5\-\-10 objects with keys "criterion" and "weight"\.User:\[Prompt\] \{prompt\} Generate 5\-\-10 specific, binary\-testable evaluation criteria for responses to the above prompt\. Output only the JSON array\.

## Appendix BExample Rubrics

We provide examples of generated rubrics at both granularity levels\. Figure[1](https://arxiv.org/html/2605.30568#A2.F1)shows a dataset\-specific rubric applicable to any instance, and Figure[2](https://arxiv.org/html/2605.30568#A2.F2)shows an instance\-specific rubric tailored to a particular task\.

Dataset\-specific rubric \(generated\)•Accuracy and Correctness: Evaluate whether the response provides factually correct information, follows logical reasoning, and directly addresses what was asked without significant errors or misconceptions\.•Completeness and Relevance: Assess how thoroughly the response covers the key aspects of the instruction while staying focused on the topic and avoiding unnecessary tangents or omissions\.•Clarity and Organization: Judge how well\-structured, coherent, and easy to understand the response is, including appropriate use of formatting, logical flow, and clear communication\.•Helpfulness and Practical Value: Determine how useful the response would be to the human, considering whether it provides actionable information, appropriate depth, and genuinely assists with the stated need\.•Safety and Appropriateness: Evaluate whether the response avoids harmful, biased, or inappropriate content while maintaining ethical standards and respecting relevant guidelines\.Figure 1:Example dataset\-specific evaluation rubric generated by Claude Sonnet 4\.Instance\-specific rubric \(generated\)Instruction:The sentence you are given might be too wordy, complicated, or unclear\. Rewrite the sentence and make your writing clearer by keeping it concise\. Whenever possible, break complex sentences into multiple sentences and eliminate unnecessary words\.Instance:If you have any questions about my rate or if you find it necessary to increase or decrease the scope for this project, please let me know\.Rubric:•The rewritten sentence\(s\) should be significantly more concise than the original, eliminating unnecessary or redundant words\.•Complex ideas should be broken into multiple clear, simple sentences rather than one complicated sentence\.•The core meaning and intent of the original message must be preserved without any loss of important information\.•The language should be more direct and straightforward, avoiding wordy phrases and unnecessary qualifiers\.•The final result should improve readability and clarity compared to the original sentence\.Figure 2:Example instance\-specific evaluation rubric generated by Claude Sonnet 4\.
## Appendix CImplementation Details

#### Model Inference

Llama 3\.1 8B Instruct, Qwen3 14B, and Llama 3\.1 70B Instruct are served on a locally hosted NVIDIA H100 GPU \(94GB\) via vLLMKwonet al\.\([2023](https://arxiv.org/html/2605.30568#bib.bib38)\)\. Claude Sonnet 4 is accessed via the AWS Bedrock API\. All judge and rubric generation calls use greedy decoding \(temperature 0\) with a maximum of 256 output tokens for judging and 1024 tokens for rubric generation\. During reward signal collection, rubric candidates are sampled with temperature 0\.8 to encourage diversity\.

#### Preference Fine\-tuning

We fine\-tune both Llama 3\.1 8B Instruct and Qwen3 14B using QLoRADettmerset al\.\([2023](https://arxiv.org/html/2605.30568#bib.bib35)\)with 4\-bit quantization via DPO\. The same hyperparameters are used for both models \(Table[5](https://arxiv.org/html/2605.30568#A3.T5)\)\.

HyperparameterValueBase modelsLlama 3\.1 8B / Qwen3 14BLoRA rank64LoRAα\\alpha128LoRA dropout0\.05Quantization4\-bitDPOβ\\beta0\.1Learning rate5×10−65\\times 10^\{\-6\}Batch size \(effective\)16Epochs3Max sequence length2048OptimizerAdamWTable 5:Preference fine\-tuning hyperparameters for the rubric generator\.
#### Meta\-Judge Configuration

We use Claude Sonnet 4 as the meta\-judge model\. For each instance, we generateK=8K\{=\}8rubric candidates with temperatureτ=0\.8\\tau\{=\}0\.8and sample up to 10 pairwise comparisons\. Positional bias is mitigated by evaluating each pair in both presentation orders\.

#### Bradley\-Terry Ranking

We fit a Bradley\-Terry model to the meta\-judge outcomes\. The model assumes each candidateℛi\\mathcal\{R\}\_\{i\}has a latent strengthγi\>0\\gamma\_\{i\}\>0, with the probability ofℛi\\mathcal\{R\}\_\{i\}being preferred overℛj\\mathcal\{R\}\_\{j\}given by:

P​\(ℛi≻ℛj\)=γiγi\+γj\.P\(\\mathcal\{R\}\_\{i\}\\succ\\mathcal\{R\}\_\{j\}\)=\\frac\{\\gamma\_\{i\}\}\{\\gamma\_\{i\}\+\\gamma\_\{j\}\}\.\(4\)We estimateγ1,…,γK\\gamma\_\{1\},\\ldots,\\gamma\_\{K\}by maximizing the log\-likelihood of the observed outcomes:

ℒ=∑\(i,j\)∈𝒪log⁡γiγi\+γj,\\mathcal\{L\}=\\sum\_\{\(i,j\)\\in\\mathcal\{O\}\}\\log\\frac\{\\gamma\_\{i\}\}\{\\gamma\_\{i\}\+\\gamma\_\{j\}\},\(5\)where𝒪\\mathcal\{O\}is the set of observed preferences \(each entry\(i,j\)\(i,j\)indicatesℛi\\mathcal\{R\}\_\{i\}was preferred overℛj\\mathcal\{R\}\_\{j\}by the meta\-judge\)\. The candidate with the highestγ\\gammabecomes the chosen rubric and the lowest becomes the rejected, forming a preference pair for DPO training\.

#### Compute

All fine\-tuning runs use a single NVIDIA H100 GPU \(94GB\)\. Each run completes in approximately 1–2 hours for 3 epochs\.

## Appendix DBaseline Method Details

#### Prometheus 2

Prometheus 2Kimet al\.\([2024b](https://arxiv.org/html/2605.30568#bib.bib7)\)is a family of open\-source LLMs \(7B and 8×\\times7B MoE\) fine\-tuned specifically for evaluation tasks\. The models are trained on a large corpus of evaluation data encompassing both pairwise and pointwise formats, with rubric\-conditioned scoring\. Given an instance and a rubric, Prometheus 2 directly produces a score \(pointwise\) or preference judgment \(pairwise\) without requiring multi\-step decomposition\. We report results with the 8×\\times7B variant using both dataset\-specific and instance\-specific rubrics \(where available\)\.

#### DnA\-Eval

Decomposition and AggregationLiet al\.\([2025](https://arxiv.org/html/2605.30568#bib.bib6)\)decomposes evaluation into multiple aspect\-specific questions with importance weights\. For each instancexx, the method generatesKKevaluation aspects\{a1,…,aK\}\\\{a\_\{1\},\\ldots,a\_\{K\}\\\}and corresponding importance weights\{w1,…,wK\}\\\{w\_\{1\},\\ldots,w\_\{K\}\\\}\(with∑kwk=1\\sum\_\{k\}w\_\{k\}=1\) using the judge model itself\. For pairwise evaluation, each response is scored on each aspect independently on a 1–10 scale, yielding aspect scores\{sk\(1\)\}\\\{s\_\{k\}^\{\(1\)\}\\\}and\{sk\(2\)\}\\\{s\_\{k\}^\{\(2\)\}\\\}, and the final preference is determined by the weighted aggregation:

y^=arg⁡maxi∈\{1,2\}​∑k=1Kwk⋅sk\(i\)\.\\hat\{y\}=\\arg\\max\_\{i\\in\\\{1,2\\\}\}\\sum\_\{k=1\}^\{K\}w\_\{k\}\\cdot s\_\{k\}^\{\(i\)\}\.\(6\)For pointwise evaluation, the judge score is∑kwk⋅sk\\sum\_\{k\}w\_\{k\}\\cdot s\_\{k\}\. We useK=3K\{=\}3aspects following the original paper\.

#### CheckEval

CheckEvalLeeet al\.\([2025](https://arxiv.org/html/2605.30568#bib.bib4)\)decomposes evaluation into a fixed checklist of binary \(Yes/No\) questions\. A set ofNNquestions\{q1,…,qN\}\\\{q\_\{1\},\\ldots,q\_\{N\}\\\}is generated once per benchmark \(not per instance\) and applied uniformly to all items\. For each response, the judge answers allNNquestions, and the score is computed as the proportion of affirmative answers:

s=1N​∑n=1N𝟙​\[answer​\(qn\)=Yes\]\.s=\\frac\{1\}\{N\}\\sum\_\{n=1\}^\{N\}\\mathbbm\{1\}\[\\text\{answer\}\(q\_\{n\}\)=\\text\{Yes\}\]\.\(7\)For pairwise comparison, both responses are scored independently and the higher\-scoring response is preferred\. We useN=20N\{=\}20questions\.

#### RubricHub

RubricHubLiet al\.\([2026](https://arxiv.org/html/2605.30568#bib.bib19)\)generates fine\-grained, weighted binary criteria grounded in a reference response\. The pipeline consists of four stages: \(1\) generate a reference responserrfor instancexx; \(2\) generate a set of rubric criteria\{\(ti,di,wi\)\}i=1M\\\{\(t\_\{i\},d\_\{i\},w\_\{i\}\)\\\}\_\{i=1\}^\{M\}\(each with titletit\_\{i\}, descriptiondid\_\{i\}, and importance weightwi∈\[1,10\]w\_\{i\}\\in\[1,10\]\) conditioned on bothxxandrr; \(3\) grade the candidate response on each criterion independently, producing binary verdictsbi∈\{0,1\}b\_\{i\}\\in\\\{0,1\\\}; and \(4\) aggregate via weighted sum:

s=∑i=1Mwi⋅bi∑i=1Mwi\.s=\\frac\{\\sum\_\{i=1\}^\{M\}w\_\{i\}\\cdot b\_\{i\}\}\{\\sum\_\{i=1\}^\{M\}w\_\{i\}\}\.\(8\)References and rubrics are cached per instance so that repeated instances share the same evaluation criteria\.

## Appendix EAdditional Results with Llama 3\.1 8B as Judge

Table[6](https://arxiv.org/html/2605.30568#A5.T6)reports baselines and our training\-free rubric generation results with Llama 3\.1 8B as the judge model\.

PairwisePointwiseAlpacaEvalMT\-BenchBiGGenHelpSteer2BiGGenPrometheusKimet al\.\([2024b](https://arxiv.org/html/2605.30568#bib.bib7)\)7B62\.6272\.1667\.590\.143/0\.2150\.254/0\.2817B \(instance\-specific\)——66\.19—0\.328/0\.356Baseline \(Judge: Llama 3\.1 8B\)CheckEvalLeeet al\.\([2025](https://arxiv.org/html/2605.30568#bib.bib4)\)56\.5470\.5663\.290\.195/0\.2560\.250/0\.280DnA\-EvalLiet al\.\([2025](https://arxiv.org/html/2605.30568#bib.bib6)\)59\.8169\.4565\.690\.168/0\.2070\.303/0\.240RubricHubLiet al\.\([2026](https://arxiv.org/html/2605.30568#bib.bib19)\)58\.2673\.8666\.940\.180/0\.1770\.239/0\.252Human\-crafted RubricsDataset\-specific65\.1173\.2264\.530\.152/0\.2260\.292/0\.280Instance\-specific——70\.31—0\.420/0\.483Our Approach \(Judge: Llama 3\.1 8B\)Dataset\-specific64\.8073\.0164\.610\.135/0\.2240\.276/0\.277Instance\-specific65\.9874\.3967\.300\.224/0\.2840\.297/0\.317Table 6:Baselines and training\-free rubric generation results with Llama 3\.1 8B as the judge\. The trends are consistent with the main results in Table[2](https://arxiv.org/html/2605.30568#S4.T2): our instance\-specific rubrics outperform dataset\-specific ones, and the overall performance is lower due to the weaker judge model\.
## Appendix FStatistical Significance Testing

We test statistical significance between our approach and the best existing method for each benchmark–judge combination using the following procedures\.

#### Pairwise Benchmarks

For pairwise benchmarks, each method produces a binary per\-instance outcome \(correct or incorrect agreement with the human majority vote\)\. We apply a paired bootstrap test with 10,000 resamples: in each iteration, we resample instances with replacement and compute the accuracy difference between the two methods\. The two\-sidedpp\-value is computed as twice the proportion of bootstrap samples where the observed difference reverses sign\. We additionally report McNemar’s test with continuity correction as a complementary check on paired binary outcomes\.

#### Pointwise Benchmarks

For pointwise benchmarks, we compare Spearman correlations via a paired bootstrap procedure\. In each of 10,000 iterations, we resample \(instance, score\) pairs with replacement and recompute both methods’ correlations with human scores\. Thepp\-value is derived from the proportion of bootstrap samples where the correlation difference reverses sign \(two\-sided\)\.

Results marked with∗in Table[2](https://arxiv.org/html/2605.30568#S4.T2)indicate cases where our method is statistically significantly worse than the best existing method atp<0\.05p<0\.05\.

## Appendix GAblation Studies

We ablate three design choices\. The first two use the Llama 3\.1 8B rubric generator \(Iteration 1\) evaluated with Claude Sonnet 4 as the judge \(Table[7](https://arxiv.org/html/2605.30568#A7.T7)\); the third varies the generator model size\.

PairwisePointwiseAlpacaEvalMT\-BenchBiGGenHelpSteer2BiGGenSeparate \(main\)70\.0982\.1474\.250\.392/0\.4280\.434/0\.441Joint training68\.8581\.4174\.210\.389/0\.4200\.441/0\.448Meta\-judge: Claude Sonnet 470\.0982\.1474\.25——Meta\-judge: Llama 3\.1 70B67\.5082\.5574\.46——Table 7:Ablation studies with Llama 3\.1 8B rubric generator \(Iteration 1\), evaluated with Claude Sonnet 4 as judge\. “Separate” trains distinct pairwise and pointwise generators; “Joint” trains one model on both\. Meta\-judge rows compare Claude Sonnet 4 vs\. Llama 3\.1 70B for reward signal collection\.#### Joint Pairwise–Pointwise Training

In our main experiments, we train separate rubric generators for pairwise and pointwise evaluation\. Here we ablate this choice by training a single model on the combined preference data\. Joint training degrades pairwise performance \(70\.09→\\to68\.85 on AlpacaEval, 82\.14→\\to81\.41 on MT\-Bench\) while showing marginal improvement on pointwiseBiGGen\(0\.434→\\to0\.441 Spearman\)\. This suggests that pairwise and pointwise rubrics require different properties—pairwise rubrics must highlight discriminative dimensions, while pointwise rubrics need calibrated scoring anchors—and mixing both objectives dilutes the training signal for each\.

#### Alternative Meta\-Judge

Our main experiments use Claude Sonnet 4 as the meta\-judgeℳ\\mathcal\{M\}for collecting rubric preferences\. We ablate this choice by substituting Llama 3\.1 70B as the meta\-judge\. The weaker meta\-judge substantially degrades AlpacaEval performance \(70\.09→\\to67\.50\) while slightly improving MT\-Bench \(82\.14→\\to82\.55\) andBiGGen\(74\.25→\\to74\.46\)\. Overall, using a stronger meta\-judge provides more reliable reward signals, particularly on benchmarks where subtle quality differences matter \(e\.g\., AlpacaEval’s open\-ended instructions\)\.

#### Generator Model Size

We additionally ablate the rubric generator model size by fine\-tuning Llama 3\.1 8B \(vs\. Qwen3 14B in the main paper\)\. Table[8](https://arxiv.org/html/2605.30568#A7.T8)reports results with the fine\-tuned 8B generator evaluated across three judge models\. The 8B generator consistently underperforms the Qwen3 14B generator \(Table[3](https://arxiv.org/html/2605.30568#S4.T3)\), which we attribute to the smaller model’s limited capacity for generating diverse, high\-quality rubric candidates during reward signal collection\.

PairwisePointwiseAlpacaEvalMT\-BenchBiGGenHelpSteer2BiGGenFine\-tuned Llama 3\.1 8B GeneratorJudge: Llama 3\.1 8BIteration 1 \(pairwise\)68\.2273\.2267\.19——Iteration 1 \(pointwise\)———0\.200/0\.2680\.311/0\.321Judge: Llama 3\.1 70BIteration 1 \(pairwise\)68\.5480\.9871\.32——Iteration 1 \(pointwise\)———0\.270/0\.3420\.380/0\.392Judge: Claude Sonnet 4Iteration 1 \(pairwise\)70\.0982\.1474\.25——Iteration 1 \(pointwise\)———0\.392/0\.4280\.434/0\.441Table 8:Results with fine\-tuned Llama 3\.1 8B as rubric generator \(Iteration 1\), evaluated across three judge models\. Compared to the Qwen3 14B generator \(Table[3](https://arxiv.org/html/2605.30568#S4.T3)\), the 8B generator yields lower performance across most benchmarks, suggesting that generator capacity is an important factor for rubric quality\.

## Appendix HQualitative Analysis and Case Studies

### H\.1Analysis of Rubric Framing Shift

The key qualitative shift across iterations is from*prescriptive*criteria to*dimension\-labeled*criteria\. Prescriptive criteria \(e\.g\., “The response clearly explains…”, “The answer includes…”\) describe what a good response should contain, functioning as binary checklists\. Dimension\-labeled criteria \(e\.g\., “Accuracy of …”, “Clarity and coherence in …”\) frame evaluation around abstract axes along which responses can be compared\.

We quantify this shift by counting criteria that begin with abstract dimension labels versus prescriptive “The response…” patterns\. For pairwise rubrics, prescriptive criteria drop from 47\.2% \(base\) to 39\.5% \(Iteration 1\) to 24\.9% \(Iteration 2\), while dimension\-labeled criteria rise from 13\.8% to 13\.1% to 26\.4%\. The trend is even more pronounced for pointwise rubrics: prescriptive criteria decline from 38\.6% to 33\.5% to 0\.9%\.

This reframing explains why preference fine\-tuning improves pairwise evaluation more than pointwise\. Dimension\-labeled criteria guide the judge to compare responses*along specific axes*, providing clear discriminative signal for relative judgments\. Prescriptive criteria are more binary—either both responses satisfy the criterion or neither does—offering less signal for distinguishing quality differences between two responses\. For pointwise scoring, however, abstract dimensions without calibrated score\-level anchors make consistent absolute scoring harder, explaining the smaller gains on pointwise benchmarks\.

### H\.2Case Studies: Rubric Evolution Across Iterations

We present representative examples showing how fine\-tuning transforms rubric framing and improves judge accuracy\. All pairwise case studies use Claude Sonnet 4 as judge; the base rubric is generated by Claude Sonnet 4 itself \(training\-free\), while the Iteration 1/2 rubrics are from the fine\-tuned Qwen3 14B generator\. The pointwise case study uses Claude Sonnet 4 as judge with rubrics from the base Qwen3 14B generator vs\. the fine\-tuned Qwen3 14B \(Iteration 2\)\.

Case Study 1: Persuasive Writing \(Pairwise — Base incorrect, Iteration 2 correct\)Prompt:Write a persuasive email to convince your introverted friend, who dislikes public speaking, to volunteer as a guest speaker at a local event\. Use compelling arguments and address potential objections\. Please be concise\.\[Follow\-up:\]Can you rephrase your previous answer and incorporate a metaphor or simile in each sentence?Judge verdict: Human=Response 2, Base=Response 1 \(✗\), Iteration 2=Response 2 \(✓\)Response 1\(follow\-up turn only\): “My answer was like a magnet, attracting the attention of everyone in the room\. My thoughts were like buckshot, flying in all directions at once\. I tried to explain my position, but it was like trying to nail Jell\-O to a wall\. My friend’s argument was like a snowball, growing in size and complexity the more I tried to ignore it\. The conversation soon became like a game of ping pong, with each of us batting ideas back and forth\. Finally, we reached a compromise that was like a ray of sunshine, bringing a much\-needed sense of resolution\.”Response 2\(follow\-up turn only, truncated\): “I’m writing to invite you to volunteer as a guest speaker at a local event for \[Event Name\]\. Just like a bubbly champagne, you’ll be bursting with confidence on stage, sparkling with excitement and enthusiasm\. \[…\] I know you’re an introvert and dislike public speaking, but you will be like a superhero, saving the day on stage\. I can help you prepare for the event by sharing a few tips and tricks that will make you more comfortable and confident on stage\. \[…\]”Base rubric\(Claude Sonnet 4, training\-free\):•The response demonstrates understanding of introversion and public speaking anxiety by acknowledging these concerns with empathy and sensitivity\.•The email uses compelling, personalized arguments that highlight specific benefits or opportunities relevant to an introverted person’s values and interests\.•The response proactively addresses common objections \(fear, preparation time, audience size, etc\.\) with practical solutions or reassuring counterpoints\.•Each sentence effectively incorporates a relevant metaphor or simile that enhances the persuasive message rather than feeling forced or distracting\.•The email maintains an appropriate tone that balances encouragement with respect for the friend’s comfort zone and autonomy\.Iteration 2 rubric\(Qwen3 14B, fine\-tuned\):•The response effectively uses a metaphor or simile in each sentence to enhance persuasiveness and clarity\.•The email addresses the friend’s introversion and fear of public speaking with empathy and logical, compelling arguments\.•Potential objections, such as anxiety or discomfort, are clearly acknowledged and countered with thoughtful reasoning\.•The tone is warm, supportive, and appropriately tailored to an introverted audience\.•The email remains concise while fully addressing the request and maintaining persuasive power\.Figure 3:Multi\-turn persuasive writing task \(MT\-Bench\)\. Response 1 uses metaphors but abandons the persuasive email format entirely, while Response 2 maintains the email structure with metaphors in each sentence\. The base rubric buries the key follow\-up constraint \(metaphor/simile in each sentence\) as criterion \#4, while the Iteration 2 rubric elevates it to criterion \#1\. This prioritization helps the judge correctly identify which response better satisfies the follow\-up instruction\.Case Study 2: Family Relationships \(Pairwise — Base incorrect, Iteration 1/2 correct\)Prompt:A is the father of B\. B is the father of C\. What is the relationship between A and C?\[Follow\-up:\]If C is the son of D, D is the father of E, E is the son of X, and X is the father of Y, and Y is the father of Z, what’s the relationship between A and Z in terms of generations and also the familial relationship in words?Judge verdict: Human=Response 1, Base=Response 2 \(✗\), Iteration 1=Response 1 \(✓\), Iteration 2=Response 1 \(✓\)Response 1:Turn 1: “A and C are grandfather and grandson\.” Turn 2: “A and Z are great\-grandfather and great\-grandson\.”Response 2\(truncated\): Turn 1: “A is the grandfather of C\.” Turn 2: “Let’s break down the relationships step by step: 1\. A is the father of B \[…\] Now, we can determine the relationship between A and Z: Since A is the grandfather of C, and Z is in the same generation as C, A is the great\-grandfather of Z\. In terms of generations, A is two generations above Z\.”Base rubric\(Claude Sonnet 4, training\-free\):•Correctly identifies that A is the grandfather of C in the first part of the question\.•Accurately traces the complete family lineage from A through all intermediate generations to Z without errors\.•Provides the correct generational distance between A and Z \(A is 6 generations older than Z\)\.•Uses precise familial terminology to describe the relationship \(A is the great\-great\-great\-great\-grandfather of Z\)\.•Shows clear logical reasoning by systematically working through each parent\-child relationship in the chain\.Iteration 2 rubric\(Qwen3 14B, fine\-tuned\):•Accuracy in determining the direct familial relationship between A and C based on the given information\.•Correct identification of the generational gap between A and Z, tracing the lineage through all intermediate individuals\.•Clear and precise description of the familial relationship between A and Z using appropriate terminology\.•Logical consistency in connecting each person in the lineage chain without introducing errors or assumptions\.•Conciseness and clarity in presenting both the generational difference and the familial relationship in a single, well\-structured response\.Figure 4:Multi\-turn reasoning task \(MT\-Bench\) requiring family relationship inference\. The base rubric*bakes in specific answers*: criterion \#3 states “A is 6 generations older than Z” and criterion \#4 specifies “great\-great\-great\-great\-grandfather,” which may not be correct given the ambiguous lineage\. This prescriptive framing biases the judge toward the verbose Response 2 that attempts detailed step\-by\-step reasoning\. The Iteration 2 rubric avoids committing to specific answers, using neutral evaluation dimensions \(“Accuracy in determining…”, “Logical consistency…”\) that let the judge assess correctness without being biased by the rubric itself\.Case Study 3: Tool Use \(Pointwise — Base score=2, Iteration 1/2 score=4, Human=5\)Prompt:In this instance you will be assigned to use the following tools\. 1\.Flight Booking: \[…specify departure city, destination city, and travel dates…\] 2\.Hotel Reservation: \[…\] 3\.Activity Planner: \[…\] Plan a trip from New York to Tokyo, including flight, hotel, and activities for 3 days\.Judge scores: Human=5, Base=2, Iteration 1=4, Iteration 2=4Response\(truncated\): “Hi, I’m happy to help you with your question about planning a trip from New York to Tokyo\. \[…\] 1\. First, let’s use the Flight Booking tool:<flight\_booking\> Departure City: New York \| Destination City: Tokyo \| Departure Date: 2023\-03\-20 \| Return Date: 2023\-03\-22 </flight\_booking\>\[…\] 2\. Next, let’s use the Hotel Reservation tool:<hotel\_reservation\> \[\.\.\.\] </hotel\_reservation\>\[…\] 3\. Finally, let’s use the Activity Planner tool:<activity\_planner\> \[\.\.\.\] Interests: Cultural Sites, Nature, Dining </activity\_planner\>\[…\] By using these tools, you can easily plan your trip from New York to Tokyo\. Have a great trip\!”Base rubric\(Qwen3 14B, training\-free\):•The response correctly uses the specified format for the Flight Booking tool with accurate departure and destination cities and travel dates\.•The response correctly uses the specified format for the Hotel Reservation tool with accurate destination city, check\-in and check\-out dates, and appropriate preferences\.•The response correctly uses the specified format for the Activity Planner tool with accurate destination city, travel dates, and relevant interests\.•The response includes all three required tools \(Flight Booking, Hotel Reservation, and Activity Planner\) in the correct order and without missing any required parameters\.•The response does not include any extra text or deviation from the required format, strictly adhering to the tool invocation syntax\.Iteration 2 rubric\(Qwen3 14B, fine\-tuned\):•Correct Use of Tool Formats: The response must accurately use the specified XML\-like format for each tool \(flight\_booking, hotel\_reservation, activity\_planner\) with all required parameters filled in correctly\.•Completeness of Information: The response must include all necessary details for each tool, such as departure and destination cities, dates, preferences, and interests, without missing any required fields\.•Logical Trip Planning: The response should demonstrate a coherent and realistic trip plan, with flights, hotels, and activities that align in terms of dates and destination\.•Adherence to Instructions: The response must strictly follow the given instructions, including not attempting to generate fake results but only the correct tool calls\.•Clarity and Structure: The response should be well\-organized, with each tool call clearly separated and formatted as specified, ensuring readability and proper execution by the external tools\.Figure 5:Tool\-use task \(BiGGen Bench, pointwise evaluation\) where trained rubrics produce more accurate scores\. The base rubric is a purely format\-checking checklist \(“correctly uses the specified format…”\), which penalizes any minor deviation regardless of overall response quality\. The Iteration 2 rubric introduces higher\-level evaluation dimensions such as “Logical Trip Planning” and “Clarity and Structure” that capture overall quality beyond syntactic correctness\. This enables the judge to assign a score \(4\) closer to the human assessment \(5\) rather than the overly strict base score \(2\)\.