Quantifying the Statistical Effect of Rubric Modifications on Human-Autorater Agreement

# Quantifying the Statistical Effect of Rubric Modifications on Human-Autorater Agreement
Source: [https://arxiv.org/html/2605.06283](https://arxiv.org/html/2605.06283)
Jessica Huynh Carnegie Mellon University jhuynh@cs\.cmu\.edu &Alfredo Gomez Carnegie Mellon University alfredo3@cs\.cmu\.edu &Athiya Deviyani Carnegie Mellon University adeviyan@cs\.cmu\.edu Renee Shelby Google Research reneeshelby@google\.com &Jeffrey P\. Bigham Carnegie Mellon University jbigham@cmu\.edu &Fernando Diaz Carnegie Mellon University diazf@cmu\.edu

###### Abstract

Autoraters, also referred to as LLM\-as\-judges, are increasingly used for evaluation and automated content moderation\. However, there is limited statistical analysis of how modifications in a rubric presented to both humans and autoraters affect their score agreement\. Rubrics that ask for an overall or*holistic*judgment \- for example, rating the “quality” of an essay \- may be inconsistently interpreted due to the complexity or subjectivity of the criteria\. Conversely, rubrics can ask for*analytic*judgments, which decompose assessment criteria \- for example, “quality” into “fluency” and “organization”\. While these rubrics can be edited to improve the individual accuracy of both human and automated scoring, this approach may result in disagreement between the two scores, or with the associated holistic judgment\. Designing and deploying reliable autoraters requires understanding not just the relationship between human and autorater annotations but how that relationship changes as holistic or analytic judgments are elicited\. The results indicate that rubric edits providing representative examples and additional context, and reducing positional bias in the rubric increased human\-autorater agreement, while higher rubric complexity and conservative aggregation methods tended to decrease it\. The findings from the automatic essay scoring and instruction\-following evaluation domains suggest that practitioners should carefully analyze domain\- and rubric\-specific performance to move towards higher human\-autorater agreement\.

Quantifying the Statistical Effect of Rubric Modifications on Human\-Autorater Agreement

Jessica HuynhCarnegie Mellon Universityjhuynh@cs\.cmu\.eduAlfredo GomezCarnegie Mellon Universityalfredo3@cs\.cmu\.eduAthiya DeviyaniCarnegie Mellon Universityadeviyan@cs\.cmu\.edu

Renee ShelbyGoogle Researchreneeshelby@google\.comJeffrey P\. BighamCarnegie Mellon Universityjbigham@cmu\.eduFernando DiazCarnegie Mellon Universitydiazf@cmu\.edu

## 1Introduction

![Refer to caption](https://arxiv.org/html/2605.06283v1/x1.png)Figure 1:This diagram provides a walkthrough of the experimental setup as shown for automatic essay scoring\. It represents comparisons made between human\-autorater agreementsτ\\tauacross holistic rubrics \(left\), in which all criteria are applied together in a single overall judgment, or analytic rubrics \(right\), in which criteria are evaluated separately, resulting in multiple scores\. The original rubrics are given to a human and an autorater, while edited rubrics are given to autoraters only\. Arrows in bold betweenτ1\\tau\_\{1\},τ2\\tau\_\{2\}andτ3\\tau\_\{3\},τ4\\tau\_\{4\}represent comparisons for which statistical significance can be calculated\.Δ​rater\\Delta\\textnormal\{rater\}represents comparisons where the type of rater is changed while the type of rubric remains constant, whileΔ​rubric\\Delta\\textnormal\{rubric\}represents comparisons where the type of rubric is changed and the type of rater remains constant\.Autoraters, or LLM\-as\-judges, have been used as an alternative to human annotation due to their scalability, cost, and time effectiveness\. Their effectiveness is typically validated through agreement with human annotation\. Borrowing from education literature, arubricis defined as having “coherent sets of criteria" and “descriptions of levels of performance for these criteria"Brookhart \([2013](https://arxiv.org/html/2605.06283#bib.bib36)\)\. This describes the scoring guidelines and instructions provided to any rater, whether human or automated, which is also referred to as part of a prompt as described in autorater literature\. Ideally, both human raters and autoraters would receive equivalent evaluation rubrics that accurately measure the same construct with reliable certainty\. However, equivalence does not necessitate identical presentation\.Wu and Quinn \([2017](https://arxiv.org/html/2605.06283#bib.bib30)\)show that expert and non\-expert human raters may require different levels of instruction specificity, for instance, specifying tools and providing concrete examples improves accuracy specifically when raters lack task\-relevant knowledge\.

In addition to identical presentation across different types of raters, humans and autoraters are sensitive to variations in instruction presentations within the same type of rater\. For humans, this sensitivity includes the interpretation of the task during crowdsourcing\(Kairam and Heer,[2016](https://arxiv.org/html/2605.06283#bib.bib32)\)and instruction specification, where increasing instruction specification increases accuracy on the task\(Wu and Quinn,[2017](https://arxiv.org/html/2605.06283#bib.bib30)\)\. For autoraters, position bias \(where the position of the evaluated text within the prompt will influence the autorater evaluation\) and verbosity bias \(where autoraters prefer more verbose texts\) are only a few of the known sensitivities\(Zhenget al\.,[2023](https://arxiv.org/html/2605.06283#bib.bib13)\)\. Autoraters are also sensitive to rubric variations such as formatting choicesSclaret al\.\([2024](https://arxiv.org/html/2605.06283#bib.bib47)\)and example orderingLuet al\.\([2022](https://arxiv.org/html/2605.06283#bib.bib52)\)\. Another example of instruction presentation involves decomposing criteria into multiple sub\-criteria\. Previous work has successfully used decomposition to improve LLM performance, whether by having autoraters decompose evaluation criteria into sub\-tasks\(Sahaet al\.,[2024](https://arxiv.org/html/2605.06283#bib.bib24)\)or by having humans decompose complex questions into simpler sub\-questions for models\(Patelet al\.,[2022](https://arxiv.org/html/2605.06283#bib.bib23)\)\.

Understanding whether rubric modifications produce statistically significant shifts in agreement is essential for practitioners who aim to deploy autoraters as evaluation tools\. This work studies human\-autorater agreement in two domains: automatic essay scoring \(AES\) and instruction\-following \(IF\)\. We examine statistically how rubric presentation and broader rubric modifications affect human\-autorater agreement on subjective evaluation tasks, as well as empirically examining whether decomposing general holistic judgments that ask for a single high\-level criteria into more granular sub\-criteria, originally designed for human evaluators, can be an effective approach for improving human\-judge agreement\. The findings indicate that instructions optimized for autoraters tend to improve agreement with human ratings when autoraters receive machine\-optimized instructions and humans receive the original set of instructions\. Conversely, giving autoraters simpler prompts does not guarantee higher agreement with human ratings\. These results indicate that 1\) rubric edits providing representative examples along with contextual information increased human\-autorater agreement as well as autorater self\-agreement, 2\) higher criterion complexity and conservative aggregation methods tended to decrease human\-autorater agreement, 3\) reducing confirmation bias tends to significantly increase human\-autorater agreement and 4\) high human inter\-rater agreement leads to significantly higher human\-autorater agreement\. These findings from the automatic essay scoring and instruction\-following evaluation domains suggest that practitioners should carefully analyze domain\-specific performance and modify rubrics to move towards high human\-autorater agreement\.

## 2Related Work

### 2\.1Rubrics in Context

Expanding upon Section[1](https://arxiv.org/html/2605.06283#S1), rubrics consist of both the criteria, which are the components of the overall evaluation, and the descriptions for the criteria\.

Decomposition leveldescribes criteria presentation, and refers to whether prompts areholistic, where “all criteria are \[applied\] at the same time”, oranalytic, where “work \[is described\] on each criterion separately”,\(Brookhart,[2013](https://arxiv.org/html/2605.06283#bib.bib36)\)\.

Generality leveldetails the descriptions for the criteria, and refers to whether prompts aregeneralortask\-specific\(i\.e\., evaluation prompts that \[can\\\\backslashcannot\] also be used for other tasks\)\(Brookhart,[2013](https://arxiv.org/html/2605.06283#bib.bib36)\)\. Autorater evaluation prompts can be viewed through this lens \- for example, a holistic autorater evaluation prompt may ask for a single overall judgment whereas an analytic prompt would decompose the evaluation into criterion to be evaluated separately\.

Prompt complexity, as used in this work, refers to the cognitive demands placed on a rater during evaluation\. Prior work on task complexity and cognitive demand has identified the number ofpaths, or components, that must be considered simultaneously\(Campbell,[1988](https://arxiv.org/html/2605.06283#bib.bib57)\), the degree of element interactivity \(i\.e\., the extent to which components must be processed together rather than independently\)\(Sweller,[2010](https://arxiv.org/html/2605.06283#bib.bib58)\)and ambiguity resulting in communication failure\(Campbell,[1988](https://arxiv.org/html/2605.06283#bib.bib57)\)as characteristics of cognitive load\. Thus in our work, we consider the number of criteria evaluated, the degree to which score\-level descriptions are interrelated, and the extent to which the rater must resolve ambiguity across sub\-criteria\.

### 2\.2LLM\-as\-judges

The autorater \(LLM\-as\-a\-judge\) paradigm has received substantial recent attention\. Several past works demonstrated that LLMs can produce evaluations consistent with human experts\.\(Chiang and Lee,[2023](https://arxiv.org/html/2605.06283#bib.bib53); Liet al\.,[2025](https://arxiv.org/html/2605.06283#bib.bib54)\)\. However, research also shown that autoraters are sensitive to prompt variations, with different instructions leading to substantially different performancesMizrahiet al\.\([2024](https://arxiv.org/html/2605.06283#bib.bib56)\)and varying quality, necessitating statistical procedures to justify replacing human ratersCalderonet al\.\([2025](https://arxiv.org/html/2605.06283#bib.bib55)\)\. Several recent works have studied how prompt modifications affect autorater performance\.\(Sclaret al\.,[2024](https://arxiv.org/html/2605.06283#bib.bib47)\)demonstrated that prompt formatting choices \(e\.g\., separator characters, whitespace\) can significantly affect task accuracy, though these effects weakly correlate across models\.\(Luet al\.,[2022](https://arxiv.org/html/2605.06283#bib.bib52)\)showed that example ordering in few\-shot prompts substantially impacts performance on classification tasks\. Our work extends this literature by using statistical procedures where possible to test how rubric modification choices, such as decomposition level, example selection, and aggregation methods affect agreement with human judgment on tasks where even trained human raters often disagree\. This work also empirically examines whether simpler rubrics, often assumed to reduce cognitive load, actually improve human\-autorater agreement, in domains where there is no single correct answer, contrasting with the classification and multiple\-choice tasks used in prior work\.

### 2\.3Automatic Essay Scoring \(AES\) and Instruction\-Following \(IF\)

Recent work has explored using autoraters for AES, employing various personas, including a “virtual evaluator with expertise in English composition”\(Xiaoet al\.,[2025](https://arxiv.org/html/2605.06283#bib.bib21)\), a “helpful pattern\-following assistant”\(Mansouret al\.,[2024](https://arxiv.org/html/2605.06283#bib.bib20)\), and an “English essay writing test evaluation committee” or “English teacher”\(Leeet al\.,[2024](https://arxiv.org/html/2605.06283#bib.bib19)\)\. However, these personas may be misaligned with the original human raters; for example, scoring guidelines for a portion of the dataset fromHamneret al\.\([2012](https://arxiv.org/html/2605.06283#bib.bib35)\)explicitly state that raters should not be teachers\. The number of few\-shot examples provided to the autorater also varies,Xiaoet al\.\([2025](https://arxiv.org/html/2605.06283#bib.bib21)\)select the closest three examples with calculated embeddings, whileKundu and Barbosa \([2024](https://arxiv.org/html/2605.06283#bib.bib18)\)chose one essay that scored highly and one essay that scored poorly\.

While most studies utilize the rubrics given by the AES datasets,Leeet al\.\([2024](https://arxiv.org/html/2605.06283#bib.bib19)\)automatically decomposed the original essay scoring rubric into sub\-criteria and performed a modified average aggregation on the sub\-criteria, which on average performs better than using a single score from zero\-shot prompting\. However,Xiaoet al\.\([2025](https://arxiv.org/html/2605.06283#bib.bib21)\)demonstrated significant improvements by fine\-tuning GPT\-3\.5\-turbo and Llama3\-8B compared to using GPT\-4 with rubrics and few\-shot examples\. These previous works also may request explanations for scores from autoraters to mimic CoT, aiming for more accurate explanations and ratings\. Studies on analytic rubrics primarily focus on cross\-prompt scoring\(Chen and Li,[2023](https://arxiv.org/html/2605.06283#bib.bib17)\)\. The experimental setup in this work follows the rigor of prior work by studying multiple autoraters, using given rubrics, and performing various rubric edits\. However, the hypotheses examined extend prior findings on edited rubrics and further investigate rubric components across various essay scoring rubrics\.

Research on instruction\-following in LLMs has led to two primary approaches for improving and evaluating LLMs, instruction tuning and alignment tuning\. Prior work has found that larger models tend to follow instructions more accurately, though this relationship is not strictly linear\(Ouyanget al\.,[2022](https://arxiv.org/html/2605.06283#bib.bib44)\)\.Honovichet al\.\([2023](https://arxiv.org/html/2605.06283#bib.bib45)\)proposed allowing LLMs to write instructions based on only seeing examples of a task, although this is still less accurate than human\-written instructions\.

## 3Experiment Setup

Our hypotheses focus on the criteria presentation of rubrics for autoraters: \(1\)Editedprompts will improve autorater alignment with human ratings over the original prompts \(humans and autoraters may require different prompts due to the way that they process information\); and \(2\)Analyticrubrics will improve autorater alignment with human ratings overholisticrubrics \(the decomposition of holistic constructs into discrete components within analytic rubrics is expected to improve autorater alignment by simplifying the evaluation task\)\.

### 3\.1Experiments

This work examines four different scores: human ratings on holistic \(HH\\textnormal\{H\}\_\{H\}\) and analytic prompts \(HA\\textnormal\{H\}\_\{A\}\), and autorater ratings on holistic \(LLMH\\textnormal\{LLM\}\_\{H\}\) and analytic prompts \(LLMA\\textnormal\{LLM\}\_\{A\}\)\. These scores facilitate two comparisons:Δ\\DeltaRater, which assesses human\-autorater agreement when using the same type of rubric,Δ\\DeltaRubric, which investigates the impact of varying the type rubric while keeping the type of rater constant, revealing how different rubrics influence ratings even when measuring the same criteria\. These comparisons are depicted in Figure[1](https://arxiv.org/html/2605.06283#S1.F1)\. Additional analysis onΔ\\DeltaRater\+Rubric, varying both the type of rubric and type of rater, is explored in Appendix[A\.1](https://arxiv.org/html/2605.06283#A1.SS1)for completion\.

Additionally, this study compares two approaches: presenting the autorater with the original human rubric versus a modified rubric, oredited rubric, designed to enhance agreement\. Modifications include adding additional context \(if available\), incorporating examples, and reducing positional bias\. Prior work\(Mansouret al\.,[2024](https://arxiv.org/html/2605.06283#bib.bib20)\)studied adding a rubric and then adding an example into the prompt, and found that ChatGPT111[https://platform\.openai\.com/docs/models](https://platform.openai.com/docs/models)benefited from having rubrics and examples, whereas Llama\-2\-13b\-chat\-hf222[https://huggingface\.co/meta\-llama/Llama\-2\-13b\-chat\-hf](https://huggingface.co/meta-llama/Llama-2-13b-chat-hf)did not for some cases\. This work uses several modification conditions\. First, the original holistic prompts for the AES task include a large number of examples \(10\-18\) \- edited holistic prompts reduces the examples to a representative set of three examples \(3ex; high, medium, and low scores\)\. This serves two purposes: \(1\) it creates a more direct comparison with theeditedanalytic prompts, which also use three examples, and \(2\) it avoids an excessively long prompt for the model\. Second, the analytic rubric prompt formats of presenting all sub\-criteria in a single API call \(batch\), in separate API calls to mitigate potential positional bias \(separate\), and a combination of enhancements such as incorporating additional context, the "3ex" set \(as the original analytic rubrics had no examples\), and the "separate" API call strategy \(edited\) are tested\. Prompts are provided in Appendix[B](https://arxiv.org/html/2605.06283#A2)\.

## 4Methods and Materials

### 4\.1Datasets

Each hypothesis is tested across the domains of AES and IF to determine whether any observed effect is context\-dependent\. Human annotations are provided by each dataset\.

#### Automatic Essay Scoring \(AES\)\.

Automatic essay scoring is a well\-established field, with the Automated Student Assessment Prize \(ASAP\) as a primary dataset\(Hamneret al\.,[2012](https://arxiv.org/html/2605.06283#bib.bib35)\)\. ASAP contains over 21,000 essays over 8 essay prompts, each with a holistic rubric used by trained human raters\. To provide further diversity within AES, this study uses essay prompts 1, 4, and 6, which represent diverse essay prompt types \(argumentative, source\-dependent, source\-dependent\) and rubric types \(general, general, task\-specific\) shown in Appendix Table[5](https://arxiv.org/html/2605.06283#A1.T5)\. The essay prompts also provide single holistic scores, and integer scores from 1 to 5 \(essay prompt 6’s scores are condensed to integers\)\.

The ASAP\+\+ dataset\(Mathias and Bhattacharyya,[2018](https://arxiv.org/html/2605.06283#bib.bib34)\)provides corresponding analytic rubrics for these essay prompts\. Except for the holistic rubric for essay prompt 6, all other rubrics are general\. It is important to note while both ASAP and ASAP\+\+ evaluate essay quality, the specific sub\-criteria for essay quality differ \(ex\. “audience awareness” is mentioned in ASAP but not ASAP\+\+ and “conventions” is mentioned in ASAP\+\+ but ASAP explicitly asks raters to not consider this\)\. This discrepancy may lead to imperfect agreement between ASAP and combined ASAP\+\+ ratings\. Additionally, the ASAP\+\+ analytic rubrics do not include examples, so three examples per essay prompt were selected from ASAP\+\+ and excluded from analysis\. These examples represent high, medium, and low scores across all analytic sub\-criteria \(ex\. a high scoring essay contains the highest scores for all analytic sub\-criteria\)\.

#### Instruction\-following \(IF\)\.

Compared to AES, instruction\-following work has decomposed instructions into a series of questions that isolate individual criteria, but with no evaluation differences in the rubric\. The InfoBench\(Qinet al\.,[2024](https://arxiv.org/html/2605.06283#bib.bib9)\)dataset consists of both easy and hard instructions, along with outputs from five popular LLMs\. Outputs are holistically scored on a scale of one to five with a general rubric, and analytically scored with binary yes/no responses to decomposed instructions with task\-specific rubrics\. In the dataset, each output was annotated with both scores by three expert annotators using provided rubrics\. Expert annotators annotated 50 instructions with 5 generations each\. The same example selection methodology used for AES was used to select input, output, instruction tuples from the remaining dataset that were not annotated by the expert annotators, since neither rubric in IF provides examples\. The decomposed automatic instruction annotations released by the InfoBench authors served as a starting point to explore the tuples\. The resulting example outputs were generated from three models: GPT\-3\.5\-turbo[1](https://arxiv.org/html/2605.06283#footnote1), Alpaca\-7b\(Taoriet al\.,[2023](https://arxiv.org/html/2605.06283#bib.bib1)\), and GPT\-4[1](https://arxiv.org/html/2605.06283#footnote1)\(OpenAIet al\.,[2024b](https://arxiv.org/html/2605.06283#bib.bib5)\)\), and represent high, medium, and low instruction\-following ratios \(100%, 50%, 0%\)\. This is translated into holistic scores of 5, 3, and 1, respectively, which were validated by the authors\.

We select AES and IF for complementary purposes\. The AES dataset provides validity through its multiple rubric variations \(examples\) all measuring the same underlying construct, essay quality, allowing us to observe the different effects of different operationalizations of the same evaluation task\. Although using ASAP and ASAP\+\+ simultaneously may result in a noisy comparison, it provides preliminary evidence for the behavior\. Conversely the InfoBench dataset offers a direct 1:1 comparison holistic and analytic rubrics that measure identical criteria\. Together, these datasets enable us to distinguish between effects that are domain specific versus those that generalize across evaluation contexts/tasks\.

### 4\.2Autoraters

Experiments were conducted using gpt\-4o\-2024\-11\-20[1](https://arxiv.org/html/2605.06283#footnote1)\(OpenAIet al\.,[2024a](https://arxiv.org/html/2605.06283#bib.bib6)\)as the autorater, which has demonstrated high alignment with human evaluations across various tasks\(Chanet al\.,[2024](https://arxiv.org/html/2605.06283#bib.bib50); Zhouet al\.,[2024](https://arxiv.org/html/2605.06283#bib.bib51)\), and Llama\-3\.1\-70B\-Instruct\(Grattafioriet al\.,[2024](https://arxiv.org/html/2605.06283#bib.bib7)\)\.

Autorater ratings are calculated using a probability weighting scheme similar toHuynhet al\.\([2023](https://arxiv.org/html/2605.06283#bib.bib2)\), given by the equation:r=∑i=0npi∑j=0npj∗sir=\\sum\_\{i=0\}^\{n\}\\frac\{\{p\_\{i\}\}\}\{\\sum\_\{j=0\}^\{n\}\{p\_\{j\}\}\}\*s\_\{i\}, whererris the final rating from the autorater,nnrepresents the number of possible scores given by the rubric,pip\_\{i\}represents the exponential of the log probability score given by the autorater, andsis\_\{i\}represents the integer score outputted by the autorater that corresponds to the log probability\. In IF for analytic prompts,sis\_\{i\}is represented by 1 for “yes” and 0 for “no”\.

GPT\-4oLlamaΔ​rater\\Delta\\textbf\{rater\}HH→LLMH\\textnormal\{H\}\_\{H\}\\to\\textnormal\{LLM\}\_\{H\}HA→LLMA\\textnormal\{H\}\_\{A\}\\to\\textnormal\{LLM\}\_\{A\}HH→LLMH\\textnormal\{H\}\_\{H\}\\to\\textnormal\{LLM\}\_\{H\}HA→LLMA\\textnormal\{H\}\_\{A\}\\to\\textnormal\{LLM\}\_\{A\}PC\.Full3exSep\.Bat\.EditedFull3exSep\.Bat\.Edited1H\.0\.437†0\.437^\{\\dagger\}0\.3870\.387\-\-\-0\.5760\.5760\.5710\.571\-\-\-Ide\.\-\-0\.474⋆0\.474^\{\\star\}0\.4640\.4640\.552s,b↑0\.552^\{s,b\\uparrow\}\-\-0\.5520\.5520\.5590\.5590\.5520\.552Or\.\-\-0\.497⋆0\.497^\{\\star\}0\.4480\.4480\.544s,b↑0\.544^\{s,b\\uparrow\}\-\-0\.547⋆0\.547^\{\\star\}0\.5240\.5240\.529s↓0\.529^\{s\\downarrow\}WC\-\-0\.452⋆0\.452^\{\\star\}0\.4390\.4390\.554s,b↑0\.554^\{s,b\\uparrow\}\-\-0\.547⋆0\.547^\{\\star\}0\.5270\.5270\.507s↓0\.507^\{s\\downarrow\}SF\-\-0\.4590\.4590\.4550\.4550\.553s,b↑0\.553^\{s,b\\uparrow\}\-\-0\.559⋆0\.559^\{\\star\}0\.5120\.5120\.523s↓0\.523^\{s\\downarrow\}Cv\.\-\-0\.3620\.3620\.388⋆0\.388^\{\\star\}0\.472s,b↑0\.472^\{s,b\\uparrow\}\-\-0\.4910\.4910\.4770\.4770\.4870\.4874H\.0\.695†0\.695^\{\\dagger\}0\.6870\.687\-\-\-0\.7000\.7000\.6990\.699\-\-\-Ct\.\-\-0\.6730\.6730\.696⋆0\.696^\{\\star\}0\.698s↑0\.698^\{s\\uparrow\}\-\-0\.698⋆0\.698^\{\\star\}0\.6930\.6930\.7010\.701PA\-\-0\.6620\.6620\.677⋆0\.677^\{\\star\}0\.680s↑0\.680^\{s\\uparrow\}\-\-0\.683⋆0\.683^\{\\star\}0\.6650\.6650\.682b↑0\.682^\{b\\uparrow\}La\.\-\-0\.627⋆0\.627^\{\\star\}0\.5950\.5950\.639s,b↑0\.639^\{s,b\\uparrow\}\-\-0\.630⋆0\.630^\{\\star\}0\.5730\.5730\.621b↑0\.621^\{b\\uparrow\}Na\.\-\-0\.669⋆0\.669^\{\\star\}0\.6520\.6520\.682s,b↑0\.682^\{s,b\\uparrow\}\-\-0\.674⋆0\.674^\{\\star\}0\.6420\.6420\.669b↑0\.669^\{b\\uparrow\}6H\.0\.6290\.6290\.644†0\.644^\{\\dagger\}\-\-\-0\.6660\.6660\.680†0\.680^\{\\dagger\}\-\-\-Ct\.\-\-0\.6100\.6100\.652⋆0\.652^\{\\star\}0\.676s,b↑0\.676^\{s,b\\uparrow\}\-\-0\.680⋆0\.680^\{\\star\}0\.6480\.6480\.694s,b↑0\.694^\{s,b\\uparrow\}PA\-\-0\.6050\.6050\.6080\.6080\.666s,b↑0\.666^\{s,b\\uparrow\}\-\-0\.652⋆0\.652^\{\\star\}0\.6010\.6010\.668s,b↑0\.668^\{s,b\\uparrow\}La\.\-\-0\.524⋆0\.524^\{\\star\}0\.5100\.5100\.546s,b↑0\.546^\{s,b\\uparrow\}\-\-0\.542⋆0\.542^\{\\star\}0\.4960\.4960\.547b↑0\.547^\{b\\uparrow\}Na\.\-\-0\.5620\.5620\.5570\.5570\.590s,b↑0\.590^\{s,b\\uparrow\}\-\-0\.579⋆0\.579^\{\\star\}0\.5470\.5470\.589b↑0\.589^\{b\\uparrow\}Δ​rubric\\Delta\\textbf\{rubric\}HH→HA\\textnormal\{H\}\_\{H\}\\to\\textnormal\{H\}\_\{A\}LLMH→LLMA\\textnormal\{LLM\}\_\{H\}\\to\\textnormal\{LLM\}\_\{A\}HH→HA\\textnormal\{H\}\_\{H\}\\to\\textnormal\{H\}\_\{A\}LLMH→LLMA\\textnormal\{LLM\}\_\{H\}\\to\\textnormal\{LLM\}\_\{A\}PC\.Full3exSep\.Bat\.EditedFull3exSep\.Bat\.Edited1full0\.5910\.591\-0\.7560\.7560\.7500\.7500\.839s,b↑0\.839^\{s,b\\uparrow\}\-\-0\.685⋆0\.685^\{\\star\}0\.6710\.6710\.711s,b↑0\.711^\{s,b\\uparrow\}3ex\-\-0\.7920\.7920\.7890\.7890\.838s,b↑0\.838^\{s,b\\uparrow\}\-\-0\.736⋆0\.736^\{\\star\}0\.7130\.7130\.768s,b↑0\.768^\{s,b\\uparrow\}4full0\.6560\.656\-0\.838⋆0\.838^\{\\star\}0\.8170\.8170\.876s,b↑0\.876^\{s,b\\uparrow\}\-\-0\.854⋆0\.854^\{\\star\}0\.7800\.7800\.869s,b↑0\.869^\{s,b\\uparrow\}3ex\-\-0\.847⋆0\.847^\{\\star\}0\.8230\.8230\.880s,b↑0\.880^\{s,b\\uparrow\}\-\-0\.861⋆0\.861^\{\\star\}0\.7840\.7840\.877s,b↑0\.877^\{s,b\\uparrow\}6full0\.6810\.681\-0\.781⋆0\.781^\{\\star\}0\.7730\.7730\.842s,b↑0\.842^\{s,b\\uparrow\}\-\-0\.803⋆0\.803^\{\\star\}0\.7480\.7480\.864s,b↑0\.864^\{s,b\\uparrow\}3ex\-\-0\.8040\.8040\.8010\.8010\.857s,b↑0\.857^\{s,b\\uparrow\}\-\-0\.813⋆0\.813^\{\\star\}0\.7510\.7510\.868s,b↑0\.868^\{s,b\\uparrow\}Table 1:Kendall’sτ\\tauresults on AES with GPT\-4o and Llama forΔ\\Deltarater\. P\. indicates the essay prompt, and C\. indicates what ratings are being compared, with ideas, organization, word choice, sentence fluency, and conventions compared for prompt 1, and content, prompt adherence, language, and narrativity compared for prompts 4 and 6\.τ\\tauis calculated with singular numerical values forΔ\\Deltarater and calculated through Pareto dominance comparison for preferences forΔ\\Deltarubric\. Significance tests between separate \(sep\.\), batch \(bat\.\), and edited prompts are performed, wheresandbin the edited prompt column represents significant differences with separate and batch prompts respectively\.†\\daggeris indicated next to comparisons that are significantly larger within holistic prompts\.⋆\\staris indicated next to comparisons that are significantly larger between separate and batch comparisons\.↑\\uparrowand↓\\downarrowrepresent that theτ\\tauvalue for edited prompts is significantly larger or smaller respectively with the separatessor batchbbprompts’τ\\tau\. The lack of any dagger, star, or arrow denotes no statistical significance\.His a shortened form forHuman\.GPT\-4oLlamaΔ\\DeltaraterHolisticAnalyticHolisticAnalytic0ex3exSep\.Bat\.Edited0ex3exSep\.Bat\.Edited0\.5360\.5360\.5850\.5850\.464⋆0\.464^\{\\star\}0\.1670\.1670\.471b↑0\.471^\{b\\uparrow\}0\.4700\.4700\.5780\.5780\.445⋆0\.445^\{\\star\}0\.1660\.1660\.426b↑0\.426^\{b\\uparrow\}Δ\\DeltarubricEx\.HH→HA\\textnormal\{H\}\_\{H\}\\to\\textnormal\{H\}\_\{A\}LLMH→LLMA\\textnormal\{LLM\}\_\{H\}\\to\\textnormal\{LLM\}\_\{A\}HH→HA\\textnormal\{H\}\_\{H\}\\to\\textnormal\{H\}\_\{A\}LLMH→LLMA\\textnormal\{LLM\}\_\{H\}\\to\\textnormal\{LLM\}\_\{A\}0ex0\.5340\.534\-0\.640⋆0\.640^\{\\star\}0\.4550\.4550\.640b↑0\.640^\{b\\uparrow\}0\.5510\.551\-0\.531⋆0\.531^\{\\star\}0\.2850\.2850\.551b↑0\.551^\{b\\uparrow\}3ex\-\-0\.623⋆0\.623^\{\\star\}0\.4840\.4840\.617b↑0\.617^\{b\\uparrow\}\-\-0\.566⋆0\.566^\{\\star\}0\.3260\.3260\.623b↑0\.623^\{b\\uparrow\}Table 2:Kendall’sτ\\tauresults on IF with GPT\-4o and Llama\.Δ\\Deltarubric is calculated through instruction following ratio comparison for preferences\. All other calculations and significance follow the methodology of Table[1](https://arxiv.org/html/2605.06283#S4.T1)\.
### 4\.3Agreement Calculation

While direct numerical comparison \(e\.g\., correlation between raw scores\) is often calculated through agreement metrics such as Cohen’sκ\\kappa, autoraters and human have been shown to use the same scales differently, with autoraters often exhibiting compressed or shifted score distributions relative to humansKundu and Barbosa \([2024](https://arxiv.org/html/2605.06283#bib.bib18)\)\. Preference\-based evaluation \(e\.g\., A/B\) is more robust to such shifts, as it captures ordinal relationships rather than numerical ones; however, such methods do not scale with large datasets\. Addressing both issues, we use Kendall’sτ\\tauwith tie consideration \(calculated using the SciPy\(Virtanenet al\.,[2020](https://arxiv.org/html/2605.06283#bib.bib4)\)implementation of\(Kendall,[1945](https://arxiv.org/html/2605.06283#bib.bib8)\)\), which operates on the existing numerical scores from the data while evaluating the scores through pairwise ordinal comparisons\.

Each domain employs a different method for aggregating analytic rubric scores when comparing them to holistic rubric scores to calculate concordant pairs for Kendall’sτ\\tau\. In AES, Pareto dominance is used, where essay A is considered better than essay B if all sub\-criteria scores of A are at least tied to those of B, with at least one sub\-criteria score from A being higher\. Pareto dominance is used as a conservative aggregation method to ensure that one essay would truly be better than another without knowing about how the sub\-criteria were factored into a holistic rating\. In IF, response A is considered to follow instructions better than response B if A has a higher ratio of correctly followed instructions\.

Pareto dominance is chosen as a conservative aggregation method precisely because the sub\-criteria in ASAP and ASAP\+\+ do not perfectly overlap, as noted in[4\.1](https://arxiv.org/html/2605.06283#S4.SS1)\. Rather than imposing an arbitrary weighting scheme or score cutoff to determine which essay is better, Pareto dominance avoids assuming any particular trade\-off between criteria, thus providing a reliable lower bound on the present agreement\.

The confidence interval around the correlations is calculated using bootstrapping\(Efron,[1992](https://arxiv.org/html/2605.06283#bib.bib48)\)with 1000 samples\. The difference between two conditions is calculated, and the 95% confidence interval is determined by sampling the 25th and 975th sorted values\. For comparisons involving three conditions, the confidence interval is adjusted using Bonferroni correction, with the interval bounds set as the average of the 8th and 9th values and the average of the 991st and 992nd values\. The compared conditions are considered significantly different if 0 does not fall within the interval\.

## 5Results

### 5\.1Edited Rubrics

When editing rubrics for autoraters to improve human\-autorater agreement, it is important to provide examples and context and remove confirmation bias from analytic rubrics\.

Using GPT with edited analytic AES rubrics mostly significantly improves agreement with humans\.In Table[1](https://arxiv.org/html/2605.06283#S4.T1)underΔ​rater\\Delta\\textnormal\{rater\}under the “edited" column, adding examples and context in the edited rubric always improved human\-autorater agreement, significantly in the majority of cases, when using GPT, indicated bys,b↑s,b\\uparrow\. These improvements range from 0\.696 \(batch prompt 4 content\) to 0\.698 \(edited prompt 4 content\) for the smaller, non\-significant improvements, to 0\.439 \(batch prompt 1 word choice\) to 0\.554 \(edited prompt 1 word choice\), for the larger, significant improvements\. While the analytic rubrics lacked explanations for the examples that the holistic AES rubrics provided, this statistically significant improvement suggests that the examples and context still provide necessary grounding for autorater scores\.

Edited analytic AES rubrics significantly improves both models’ self\-alignment but not with IF rubrics\.In addition to improving human\-autorater alignment, adding examples and context in the edited rubric always significantly improves GPT and Llama’s self\-agreement with their scores on the original holistic rubric, although these two rubrics have inherent differences, seen in Table[1](https://arxiv.org/html/2605.06283#S4.T1)underΔ​rubric\\Delta\\textnormal\{rubric\}, indicated bys,b↑s,b\\uparrow\. Interestingly, these agreements surpass human\-human agreement with the two types of unedited rubrics \(0\.591, 0\.656, 0\.681 compared to 0\.750, 0\.817, 0\.773 for GPT, and 0\.671, 0\.780, and 0\.748 for Llama\)\. This could be because the same autorater is used with both rubrics, whereas human ratings are from different individuals\. This suggests that while the prompts superficially evaluate different criteria, the autorater converges to a consistent understanding of essay quality across both rubrics, which aligns with the overall rating objective, a convergence not observed to the same degree in human ratings\.

However, adding examples of human analytic scores does not necessarily increase either autoraters’ self\-alignment on IF\. Adding examples decreasedτ\\tauwith GPT\-4o \(τ\\taufrom 0\.640 to 0\.640, and 0\.623 to 0\.617 between separate and edited rubrics in Table[2](https://arxiv.org/html/2605.06283#S4.T2)\), but increasedτ\\tauwith Llama \(0\.531 to 0\.551 and 0\.566 to 0\.623 respectively\)\. This suggests that autoraters in this case do not achieve a unified understanding of instruction following across prompts\. Despite this, the ratio aggregation method aligns more closely with the autorater’s internal reasoning about output instruction adherence\. With edited prompts, GPT\-4o’s alignment \(but not Llama’s\) exceeded human alignment on both prompts, even though the human expert annotators across both rubrics remained consistent\.

Reducing confirmation bias in all analytic rubrics provides mostly significant improvement for both models\.Prior work has highlighted confirmation bias\(Cook and Smallman,[2008](https://arxiv.org/html/2605.06283#bib.bib3)\), in which conceived judgment is reinforced as the task progresses\. This bias could affect analytic rubrics, leading raters to assign low scores across all criteria if they initially believe the piece of text is of poor quality\.Leeet al\.\([2024](https://arxiv.org/html/2605.06283#bib.bib19)\)has shown that using separate conversations to rate each sub\-criteria tends to increase agreement over using a single conversation to rate all sub\-criteria\. However, this bias has not been statistically assessed for autoraters, and may affect autoraters differently than humans\.

Across the analytic rubrics on both tasks, the majority of comparisons show separate rubrics significantly outperforming the batched rubrics\. With Llama on AES and both models on IF, edited rubrics do not consistently outperform separate rubrics\. This suggests the separation of the individual criteria is more important than the examples for Llama\. The scoring rubric itself may contribute to the inconsistency for GPT\-4o between AES and IF\. In AES, example essays answered the same writing prompt as the evaluated essay, and were accompanied by the same scoring rubric during evaluation\. Conversely, in IF, the examples did not correspond to the same decomposed questions being scored\. The rubrics in IF are task\-specific and vary significantly across instructions\.

In addition, there is a trend ofτ\\taudecreasing and then increasing based on criteria order in prompts 4 and 6, observed with both GPT\-4o \(0\.696, 0\.677, 0\.595, 0\.652 and 0\.652, 0\.608, 0\.510, 0\.557 respectively\) and Llama \(0\.693, 0\.665, 0\.573, 0\.642 and 0\.648, 0\.601, 0\.496, 0\.547 respectively\)\. This suggests potential differences in bias between humans and autoraters\. This may also suggest that autoraters struggle with rating word choice or language\. Qualitative analysis of model outputs for instruction following reveal that batch ratings consistently demonstrated a bias with probability distributions heavily skewed towards “yes" responses when answering yes/no decomposed questions\. This contrasts with the separate API call approach, which more closely resembled human response distributions, suggesting an underlying bias effect in batch prompts\.

Task\-specific AES holistic rubrics show significant improvement when reducing examples, but general AES holistic rubrics do not\.Reducing the number of examples from the full set to three \(3ex\) tends to decreaseτ\\taufor GPT on AES for the general holistic rubrics in essay prompts 1 and 4 \(τ\\taudropped from 0\.437 to 0\.387 and 0\.695 to 0\.687, respectively\), and increaseτ\\taufor the task\-specific holistic rubric in essay prompt 6 \(τ\\taurose from 0\.629 to 0\.644\)\. This could also be because the examples have explanations for essay prompts 1 and 4, but not for 6\. The score explanations could be more influential than the scores alone when examples were originally provided, but this could also be due to the type of rubric\.

Adding examples for IF holistic rubrics show improvement\.Although there is an increase inτ\\tauwhen adding examples to the holistic rubric in IF for both autoraters \(0\.536 to 0\.585 and 0\.470 to 0\.578 for GPT and Llama respectively\), it is not statistically significant\.

### 5\.2Decomposition Level

Analytic rubrics do not consistently outperform holistic rubrics in aligning autoraters with human judgments due to prompt complexity or aggregation methods\.

Prompt complexity moderates effects\.In Table[1](https://arxiv.org/html/2605.06283#S4.T1)for AES, under theΔ\\Deltarater condition for essay prompt 1, almost all analytic batch prompts had higher correlation with humans \(τ\\tauof 0\.464, 0\.455, 0\.448, 0\.439, 0\.388\) than the full holistic prompt \(τ\\tauof 0\.437\)\. Conversely in essay prompts 4 and 6, the full holistic prompt outperformed most analytic batch prompts \(τ\\tauof 0\.695 compared to 0\.696, 0\.677, 0\.652, 0\.595 andτ\\tauof 0\.629 compared to 0\.652, 0\.608, 0\.557, 0\.510\)\. This discrepancy may stem from the complexity of the holistic prompts\. Essay prompt 1’s holistic prompt is highly complex, involving multiple sub\-criteria that contain complex decisions, whereas, essay prompts 4 and 6 have less complex prompts\. Introducing analytic rubrics may increase evaluation complexity, which leads to lowerτ\\tau\.

Aggregation methods influence agreement\.Another factor is the prompt’s output\. The Pareto dominance aggregation method, a conservative estimate of essay comparison, is highly sensitive to disagreements in any single sub\-criteria\. In IF, both GPT\-4o and Llama performed worse with analytic prompts than with holistic prompts \(τ\\tauof 0\.167 to 0\.536 and 0\.166 to 0\.470\)\. This is surprising, given that IF analytic prompts are considered task\-specific, which should provide more detailed information about the task, potentially leading to higher agreement\. This may be due to the aggregation method used–the ratio of “yes” to “no” responses–which does not account for the varying weights of sub\-criteria in the overall holistic evaluation\. Therefore, holistic preferences, being more straightforward to calculate, may yield higher performance\. Higher human\-autorater agreement is not necessarily achieved by decomposing a holistic rubric into several analytic parts, rather it is more important to understand the complexity of each evaluation measure as well as the aggregation methods used\.

### 5\.3Agreement Level

Both datasets are stratified by human agreement level since human inter\-rater agreement may influence human\-autorater agreement\.

#### High human inter\-rater agreement is important\.

On IF holistic rubrics, human\-autorater agreement with the consolidated score was significantly higher when all three human annotators agreed under all conditions, than if only two or no annotators agreed with each other\. This held for IF analytic rubrics for both the separate and edited conditions, although the batch condition had an increase, it was not significant\. For AES holistic rubrics, a significant increase was only observed for prompt 4, while a non\-significant increase was observed for prompt 6\. Detailed analyses can be found in Appendix[A\.2](https://arxiv.org/html/2605.06283#A1.SS2)\.

## 6Conclusion

This work highlights the importance of understanding the domain, autorater, and rubric during rubric creation\. First, adding examples and context significantly improves human\-autorater agreement in addition to autorater self\-agreement, but this is dependent on the domain and autorater\. Second, rubrics that reduce confirmation bias tend to provide significant improvement for human\-autorater agreement\. Third, rubric complexity and aggregation methods across holistic and analytic rubrics influence human\-autorater agreement\. Lastly, higher human inter\-rater agreement contributes positively to human\-autorater agreement\. Practitioners aiming to use autoraters should carefully curate human annotation data and design rubrics that appropriately address the differences across domains and autoraters\. Future work should explore a wider range of domains, autoraters, and rubrics to develop more comprehensive recommendations\.

## 7Limitations

First, the findings are drawn from only two evaluation domains \(automatic essay scoring and instruction following\) and two autoraters \(GPT\-4o and Llama\-3\.1\-70B\-Instruct\)\. While these were chosen to provide complementary evidence, the extent to which the observed patterns generalize to other evaluation tasks \(e\.g\., summarization, dialogue quality\) or other autoraters remains unexplored\.

Second, all human annotations are drawn from existing datasets rather than collected under controlled experimental conditions for this study\. This means that variability in annotator training, expertise, and scoring context across datasets could influence the observed agreement patterns\. Moreover the rubric modifications explored represent a small subset of possible edits\. Other modifications were not tested and may interact differently with human\-autorater agreement\.

Finally, this work limited to high\-resource english\-language evaluation tasks, rubric sensitivity may differ across languages but is understudied, particularly for autoraters, whose performance can vary substantially by language\.

## References

- How to create and use rubrics for formative assessment and grading\.Ascd\.Cited by:[§1](https://arxiv.org/html/2605.06283#S1.p1.1),[§2\.1](https://arxiv.org/html/2605.06283#S2.SS1.p2.1),[§2\.1](https://arxiv.org/html/2605.06283#S2.SS1.p3.1)\.
- N\. Calderon, R\. Reichart, and R\. Dror \(2025\)The alternative annotator test for LLM\-as\-a\-judge: how to statistically justify replacing human annotators with LLMs\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 16051–16081\.External Links:[Link](https://aclanthology.org/2025.acl-long.782/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.782),ISBN 979\-8\-89176\-251\-0Cited by:[§2\.2](https://arxiv.org/html/2605.06283#S2.SS2.p1.1)\.
- D\. J\. Campbell \(1988\)Task complexity: a review and analysis\.The Academy of Management Review13\(1\),pp\. 40–52\.External Links:ISSN 03637425,[Link](http://www.jstor.org/stable/258353)Cited by:[§2\.1](https://arxiv.org/html/2605.06283#S2.SS1.p4.1)\.
- C\. Chan, W\. Chen, Y\. Su, J\. Yu, W\. Xue, S\. Zhang, J\. Fu, and Z\. Liu \(2024\)ChatEval: towards better LLM\-based evaluators through multi\-agent debate\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=FQepisCUWu)Cited by:[§4\.2](https://arxiv.org/html/2605.06283#S4.SS2.p1.1)\.
- Y\. Chen and X\. Li \(2023\)PMAES: prompt\-mapping contrastive learning for cross\-prompt automated essay scoring\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 1489–1503\.External Links:[Link](https://aclanthology.org/2023.acl-long.83/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.83)Cited by:[§2\.3](https://arxiv.org/html/2605.06283#S2.SS3.p2.1)\.
- C\. Chiang and H\. Lee \(2023\)Can Large Language Models Be an Alternative to Human Evaluations?\.arXiv\.Note:arXiv:2305\.01937 \[cs\]External Links:[Link](http://arxiv.org/abs/2305.01937),[Document](https://dx.doi.org/10.48550/arXiv.2305.01937)Cited by:[§2\.2](https://arxiv.org/html/2605.06283#S2.SS2.p1.1)\.
- M\. B\. Cook and H\. S\. Smallman \(2008\)Human factors of the confirmation bias in intelligence analysis: decision support from graphical evidence landscapes\.Human Factors50\(5\),pp\. 745–754\.Note:PMID: 19110834External Links:[Document](https://dx.doi.org/10.1518/001872008X354183),[Link](https://journals.sagepub.com/doi/abs/10.1518/001872008X354183),https://journals\.sagepub\.com/doi/pdf/10\.1518/001872008X354183Cited by:[§5\.1](https://arxiv.org/html/2605.06283#S5.SS1.p5.1)\.
- B\. Efron \(1992\)Bootstrap methods: another look at the jackknife\.InBreakthroughs in statistics: Methodology and distribution,pp\. 569–593\.Cited by:[§4\.3](https://arxiv.org/html/2605.06283#S4.SS3.p4.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§4\.2](https://arxiv.org/html/2605.06283#S4.SS2.p1.1)\.
- B\. Hamner, J\. Morgan, lynnvandev, M\. Shermis, and T\. V\. Ark \(2012\)The hewlett foundation: automated essay scoring\.Note:[https://kaggle\.com/competitions/asap\-aes](https://kaggle.com/competitions/asap-aes)KaggleCited by:[§B\.1](https://arxiv.org/html/2605.06283#A2.SS1.p1.1),[§2\.3](https://arxiv.org/html/2605.06283#S2.SS3.p1.1),[§4\.1](https://arxiv.org/html/2605.06283#S4.SS1.SSS0.Px1.p1.1)\.
- O\. Honovich, U\. Shaham, S\. R\. Bowman, and O\. Levy \(2023\)Instruction induction: from few examples to natural language task descriptions\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 1935–1952\.External Links:[Link](https://aclanthology.org/2023.acl-long.108/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.108)Cited by:[§2\.3](https://arxiv.org/html/2605.06283#S2.SS3.p3.1)\.
- J\. Huynh, C\. Jiao, P\. Gupta, S\. Mehri, P\. Bajaj, V\. Chaudhary, and M\. Eskenazi \(2023\)Understanding the effectiveness of very large language models on dialog evaluation\.InProceedings of the 13th International Workshop on Spoken Dialogue Systems Technology,IWSDS’23\.External Links:[Link](https://arxiv.org/pdf/2301.12004)Cited by:[§4\.2](https://arxiv.org/html/2605.06283#S4.SS2.p2.6)\.
- S\. Kairam and J\. Heer \(2016\)Parting crowds: characterizing divergent interpretations in crowdsourced annotation tasks\.InProceedings of the 19th ACM Conference on Computer\-Supported Cooperative Work & Social Computing,CSCW ’16,New York, NY, USA,pp\. 1637–1648\.External Links:ISBN 9781450335928,[Link](https://doi.org/10.1145/2818048.2820016),[Document](https://dx.doi.org/10.1145/2818048.2820016)Cited by:[§1](https://arxiv.org/html/2605.06283#S1.p2.1)\.
- M\. G\. Kendall \(1945\)The treatment of ties in ranking problems\.Biometrika33\(3\),pp\. 239–251\.External Links:ISSN 00063444, 14643510,[Link](http://www.jstor.org/stable/2332303)Cited by:[§4\.3](https://arxiv.org/html/2605.06283#S4.SS3.p1.2)\.
- A\. Kundu and D\. Barbosa \(2024\)Are large language models good essay graders?\.arXiv preprint arXiv:2409\.13120\.Cited by:[§2\.3](https://arxiv.org/html/2605.06283#S2.SS3.p1.1),[§4\.3](https://arxiv.org/html/2605.06283#S4.SS3.p1.2)\.
- S\. Lee, Y\. Cai, D\. Meng, Z\. Wang, and Y\. Wu \(2024\)Unleashing large language models’ proficiency in zero\-shot essay scoring\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 181–198\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.10/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.10)Cited by:[§2\.3](https://arxiv.org/html/2605.06283#S2.SS3.p1.1),[§2\.3](https://arxiv.org/html/2605.06283#S2.SS3.p2.1),[§5\.1](https://arxiv.org/html/2605.06283#S5.SS1.p5.1)\.
- D\. Li, B\. Jiang, L\. Huang, A\. Beigi, C\. Zhao, Z\. Tan, A\. Bhattacharjee, Y\. Jiang, C\. Chen, T\. Wu, K\. Shu, L\. Cheng, and H\. Liu \(2025\)From generation to judgment: opportunities and challenges of LLM\-as\-a\-judge\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 2757–2791\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.138/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.138),ISBN 979\-8\-89176\-332\-6Cited by:[§2\.2](https://arxiv.org/html/2605.06283#S2.SS2.p1.1)\.
- Y\. Lu, M\. Bartolo, A\. Moore, S\. Riedel, and P\. Stenetorp \(2022\)Fantastically ordered prompts and where to find them: overcoming few\-shot prompt order sensitivity\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 8086–8098\.External Links:[Link](https://aclanthology.org/2022.acl-long.556),[Document](https://dx.doi.org/10.18653/v1/2022.acl-long.556)Cited by:[§1](https://arxiv.org/html/2605.06283#S1.p2.1),[§2\.2](https://arxiv.org/html/2605.06283#S2.SS2.p1.1)\.
- W\. A\. Mansour, S\. Albatarni, S\. Eltanbouly, and T\. Elsayed \(2024\)Can large language models automatically score proficiency of written essays?\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),N\. Calzolari, M\. Kan, V\. Hoste, A\. Lenci, S\. Sakti, and N\. Xue \(Eds\.\),Torino, Italia,pp\. 2777–2786\.External Links:[Link](https://aclanthology.org/2024.lrec-main.247/)Cited by:[§2\.3](https://arxiv.org/html/2605.06283#S2.SS3.p1.1),[§3\.1](https://arxiv.org/html/2605.06283#S3.SS1.p2.1)\.
- S\. Mathias and P\. Bhattacharyya \(2018\)ASAP\+\+: enriching the ASAP automated essay grading dataset with essay attribute scores\.InProceedings of the Eleventh International Conference on Language Resources and Evaluation \(LREC 2018\),N\. Calzolari, K\. Choukri, C\. Cieri, T\. Declerck, S\. Goggi, K\. Hasida, H\. Isahara, B\. Maegaard, J\. Mariani, H\. Mazo, A\. Moreno, J\. Odijk, S\. Piperidis, and T\. Tokunaga \(Eds\.\),Miyazaki, Japan\.External Links:[Link](https://aclanthology.org/L18-1187/)Cited by:[§B\.1](https://arxiv.org/html/2605.06283#A2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2605.06283#S4.SS1.SSS0.Px1.p2.1)\.
- M\. Mizrahi, G\. Kaplan, D\. Malkin, R\. Dror, D\. Shahaf, and G\. Stanovsky \(2024\)State of what art? a call for multi\-prompt LLM evaluation\.Transactions of the Association for Computational Linguistics12,pp\. 933–949\.External Links:[Link](https://aclanthology.org/2024.tacl-1.52/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00681)Cited by:[§2\.2](https://arxiv.org/html/2605.06283#S2.SS2.p1.1)\.
- OpenAI, :, A\. Hurst, A\. Lerer, A\. P\. Goucher, A\. Perelman, A\. Ramesh, A\. Clark, A\. Ostrow, A\. Welihinda, A\. Hayes, A\. Radford, A\. Mądry, A\. Baker\-Whitcomb, A\. Beutel, A\. Borzunov, A\. Carney, A\. Chow, A\. Kirillov, A\. Nichol, A\. Paino, A\. Renzin, A\. T\. Passos, A\. Kirillov, A\. Christakis, A\. Conneau, A\. Kamali, A\. Jabri, A\. Moyer, A\. Tam, A\. Crookes, A\. Tootoochian, A\. Tootoonchian, A\. Kumar, A\. Vallone, A\. Karpathy, A\. Braunstein, A\. Cann, A\. Codispoti, A\. Galu, A\. Kondrich, A\. Tulloch, A\. Mishchenko, A\. Baek, A\. Jiang, A\. Pelisse, A\. Woodford, A\. Gosalia, A\. Dhar, A\. Pantuliano, A\. Nayak, A\. Oliver, B\. Zoph, B\. Ghorbani, B\. Leimberger, B\. Rossen, B\. Sokolowsky, B\. Wang, B\. Zweig, B\. Hoover, B\. Samic, B\. McGrew, B\. Spero, B\. Giertler, B\. Cheng, B\. Lightcap, B\. Walkin, B\. Quinn, B\. Guarraci, B\. Hsu, B\. Kellogg, B\. Eastman, C\. Lugaresi, C\. Wainwright, C\. Bassin, C\. Hudson, C\. Chu, C\. Nelson, C\. Li, C\. J\. Shern, C\. Conger, C\. Barette, C\. Voss, C\. Ding, C\. Lu, C\. Zhang, C\. Beaumont, C\. Hallacy, C\. Koch, C\. Gibson, C\. Kim, C\. Choi, C\. McLeavey, C\. Hesse, C\. Fischer, C\. Winter, C\. Czarnecki, C\. Jarvis, C\. Wei, C\. Koumouzelis, D\. Sherburn, D\. Kappler, D\. Levin, D\. Levy, D\. Carr, D\. Farhi, D\. Mely, D\. Robinson, D\. Sasaki, D\. Jin, D\. Valladares, D\. Tsipras, D\. Li, D\. P\. Nguyen, D\. Findlay, E\. Oiwoh, E\. Wong, E\. Asdar, E\. Proehl, E\. Yang, E\. Antonow, E\. Kramer, E\. Peterson, E\. Sigler, E\. Wallace, E\. Brevdo, E\. Mays, F\. Khorasani, F\. P\. Such, F\. Raso, F\. Zhang, F\. von Lohmann, F\. Sulit, G\. Goh, G\. Oden, G\. Salmon, G\. Starace, G\. Brockman, H\. Salman, H\. Bao, H\. Hu, H\. Wong, H\. Wang, H\. Schmidt, H\. Whitney, H\. Jun, H\. Kirchner, H\. P\. de Oliveira Pinto, H\. Ren, H\. Chang, H\. W\. Chung, I\. Kivlichan, I\. O’Connell, I\. O’Connell, I\. Osband, I\. Silber, I\. Sohl, I\. Okuyucu, I\. Lan, I\. Kostrikov, I\. Sutskever, I\. Kanitscheider, I\. Gulrajani, J\. Coxon, J\. Menick, J\. Pachocki, J\. Aung, J\. Betker, J\. Crooks, J\. Lennon, J\. Kiros, J\. Leike, J\. Park, J\. Kwon, J\. Phang, J\. Teplitz, J\. Wei, J\. Wolfe, J\. Chen, J\. Harris, J\. Varavva, J\. G\. Lee, J\. Shieh, J\. Lin, J\. Yu, J\. Weng, J\. Tang, J\. Yu, J\. Jang, J\. Q\. Candela, J\. Beutler, J\. Landers, J\. Parish, J\. Heidecke, J\. Schulman, J\. Lachman, J\. McKay, J\. Uesato, J\. Ward, J\. W\. Kim, J\. Huizinga, J\. Sitkin, J\. Kraaijeveld, J\. Gross, J\. Kaplan, J\. Snyder, J\. Achiam, J\. Jiao, J\. Lee, J\. Zhuang, J\. Harriman, K\. Fricke, K\. Hayashi, K\. Singhal, K\. Shi, K\. Karthik, K\. Wood, K\. Rimbach, K\. Hsu, K\. Nguyen, K\. Gu\-Lemberg, K\. Button, K\. Liu, K\. Howe, K\. Muthukumar, K\. Luther, L\. Ahmad, L\. Kai, L\. Itow, L\. Workman, L\. Pathak, L\. Chen, L\. Jing, L\. Guy, L\. Fedus, L\. Zhou, L\. Mamitsuka, L\. Weng, L\. McCallum, L\. Held, L\. Ouyang, L\. Feuvrier, L\. Zhang, L\. Kondraciuk, L\. Kaiser, L\. Hewitt, L\. Metz, L\. Doshi, M\. Aflak, M\. Simens, M\. Boyd, M\. Thompson, M\. Dukhan, M\. Chen, M\. Gray, M\. Hudnall, M\. Zhang, M\. Aljubeh, M\. Litwin, M\. Zeng, M\. Johnson, M\. Shetty, M\. Gupta, M\. Shah, M\. Yatbaz, M\. J\. Yang, M\. Zhong, M\. Glaese, M\. Chen, M\. Janner, M\. Lampe, M\. Petrov, M\. Wu, M\. Wang, M\. Fradin, M\. Pokrass, M\. Castro, M\. O\. T\. de Castro, M\. Pavlov, M\. Brundage, M\. Wang, M\. Khan, M\. Murati, M\. Bavarian, M\. Lin, M\. Yesildal, N\. Soto, N\. Gimelshein, N\. Cone, N\. Staudacher, N\. Summers, N\. LaFontaine, N\. Chowdhury, N\. Ryder, N\. Stathas, N\. Turley, N\. Tezak, N\. Felix, N\. Kudige, N\. Keskar, N\. Deutsch, N\. Bundick, N\. Puckett, O\. Nachum, O\. Okelola, O\. Boiko, O\. Murk, O\. Jaffe, O\. Watkins, O\. Godement, O\. Campbell\-Moore, P\. Chao, P\. McMillan, P\. Belov, P\. Su, P\. Bak, P\. Bakkum, P\. Deng, P\. Dolan, P\. Hoeschele, P\. Welinder, P\. Tillet, P\. Pronin, P\. Tillet, P\. Dhariwal, Q\. Yuan, R\. Dias, R\. Lim, R\. Arora, R\. Troll, R\. Lin, R\. G\. Lopes, R\. Puri, R\. Miyara, R\. Leike, R\. Gaubert, R\. Zamani, R\. Wang, R\. Donnelly, R\. Honsby, R\. Smith, R\. Sahai, R\. Ramchandani, R\. Huet, R\. Carmichael, R\. Zellers, R\. Chen, R\. Chen, R\. Nigmatullin, R\. Cheu, S\. Jain, S\. Altman, S\. Schoenholz, S\. Toizer, S\. Miserendino, S\. Agarwal, S\. Culver, S\. Ethersmith, S\. Gray, S\. Grove, S\. Metzger, S\. Hermani, S\. Jain, S\. Zhao, S\. Wu, S\. Jomoto, S\. Wu, Shuaiqi, Xia, S\. Phene, S\. Papay, S\. Narayanan, S\. Coffey, S\. Lee, S\. Hall, S\. Balaji, T\. Broda, T\. Stramer, T\. Xu, T\. Gogineni, T\. Christianson, T\. Sanders, T\. Patwardhan, T\. Cunninghman, T\. Degry, T\. Dimson, T\. Raoux, T\. Shadwell, T\. Zheng, T\. Underwood, T\. Markov, T\. Sherbakov, T\. Rubin, T\. Stasi, T\. Kaftan, T\. Heywood, T\. Peterson, T\. Walters, T\. Eloundou, V\. Qi, V\. Moeller, V\. Monaco, V\. Kuo, V\. Fomenko, W\. Chang, W\. Zheng, W\. Zhou, W\. Manassra, W\. Sheu, W\. Zaremba, Y\. Patil, Y\. Qian, Y\. Kim, Y\. Cheng, Y\. Zhang, Y\. He, Y\. Zhang, Y\. Jin, Y\. Dai, and Y\. Malkov \(2024a\)GPT\-4o system card\.External Links:2410\.21276,[Link](https://arxiv.org/abs/2410.21276)Cited by:[§4\.2](https://arxiv.org/html/2605.06283#S4.SS2.p1.1)\.
- OpenAI, J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat, R\. Avila, I\. Babuschkin, S\. Balaji, V\. Balcom, P\. Baltescu, H\. Bao, M\. Bavarian, J\. Belgum, I\. Bello, J\. Berdine, G\. Bernadett\-Shapiro, C\. Berner, L\. Bogdonoff, O\. Boiko, M\. Boyd, A\. Brakman, G\. Brockman, T\. Brooks, M\. Brundage, K\. Button, T\. Cai, R\. Campbell, A\. Cann, B\. Carey, C\. Carlson, R\. Carmichael, B\. Chan, C\. Chang, F\. Chantzis, D\. Chen, S\. Chen, R\. Chen, J\. Chen, M\. Chen, B\. Chess, C\. Cho, C\. Chu, H\. W\. Chung, D\. Cummings, J\. Currier, Y\. Dai, C\. Decareaux, T\. Degry, N\. Deutsch, D\. Deville, A\. Dhar, D\. Dohan, S\. Dowling, S\. Dunning, A\. Ecoffet, A\. Eleti, T\. Eloundou, D\. Farhi, L\. Fedus, N\. Felix, S\. P\. Fishman, J\. Forte, I\. Fulford, L\. Gao, E\. Georges, C\. Gibson, V\. Goel, T\. Gogineni, G\. Goh, R\. Gontijo\-Lopes, J\. Gordon, M\. Grafstein, S\. Gray, R\. Greene, J\. Gross, S\. S\. Gu, Y\. Guo, C\. Hallacy, J\. Han, J\. Harris, Y\. He, M\. Heaton, J\. Heidecke, C\. Hesse, A\. Hickey, W\. Hickey, P\. Hoeschele, B\. Houghton, K\. Hsu, S\. Hu, X\. Hu, J\. Huizinga, S\. Jain, S\. Jain, J\. Jang, A\. Jiang, R\. Jiang, H\. Jin, D\. Jin, S\. Jomoto, B\. Jonn, H\. Jun, T\. Kaftan, Ł\. Kaiser, A\. Kamali, I\. Kanitscheider, N\. S\. Keskar, T\. Khan, L\. Kilpatrick, J\. W\. Kim, C\. Kim, Y\. Kim, J\. H\. Kirchner, J\. Kiros, M\. Knight, D\. Kokotajlo, Ł\. Kondraciuk, A\. Kondrich, A\. Konstantinidis, K\. Kosic, G\. Krueger, V\. Kuo, M\. Lampe, I\. Lan, T\. Lee, J\. Leike, J\. Leung, D\. Levy, C\. M\. Li, R\. Lim, M\. Lin, S\. Lin, M\. Litwin, T\. Lopez, R\. Lowe, P\. Lue, A\. Makanju, K\. Malfacini, S\. Manning, T\. Markov, Y\. Markovski, B\. Martin, K\. Mayer, A\. Mayne, B\. McGrew, S\. M\. McKinney, C\. McLeavey, P\. McMillan, J\. McNeil, D\. Medina, A\. Mehta, J\. Menick, L\. Metz, A\. Mishchenko, P\. Mishkin, V\. Monaco, E\. Morikawa, D\. Mossing, T\. Mu, M\. Murati, O\. Murk, D\. Mély, A\. Nair, R\. Nakano, R\. Nayak, A\. Neelakantan, R\. Ngo, H\. Noh, L\. Ouyang, C\. O’Keefe, J\. Pachocki, A\. Paino, J\. Palermo, A\. Pantuliano, G\. Parascandolo, J\. Parish, E\. Parparita, A\. Passos, M\. Pavlov, A\. Peng, A\. Perelman, F\. de Avila Belbute Peres, M\. Petrov, H\. P\. de Oliveira Pinto, Michael, Pokorny, M\. Pokrass, V\. H\. Pong, T\. Powell, A\. Power, B\. Power, E\. Proehl, R\. Puri, A\. Radford, J\. Rae, A\. Ramesh, C\. Raymond, F\. Real, K\. Rimbach, C\. Ross, B\. Rotsted, H\. Roussez, N\. Ryder, M\. Saltarelli, T\. Sanders, S\. Santurkar, G\. Sastry, H\. Schmidt, D\. Schnurr, J\. Schulman, D\. Selsam, K\. Sheppard, T\. Sherbakov, J\. Shieh, S\. Shoker, P\. Shyam, S\. Sidor, E\. Sigler, M\. Simens, J\. Sitkin, K\. Slama, I\. Sohl, B\. Sokolowsky, Y\. Song, N\. Staudacher, F\. P\. Such, N\. Summers, I\. Sutskever, J\. Tang, N\. Tezak, M\. B\. Thompson, P\. Tillet, A\. Tootoonchian, E\. Tseng, P\. Tuggle, N\. Turley, J\. Tworek, J\. F\. C\. Uribe, A\. Vallone, A\. Vijayvergiya, C\. Voss, C\. Wainwright, J\. J\. Wang, A\. Wang, B\. Wang, J\. Ward, J\. Wei, C\. Weinmann, A\. Welihinda, P\. Welinder, J\. Weng, L\. Weng, M\. Wiethoff, D\. Willner, C\. Winter, S\. Wolrich, H\. Wong, L\. Workman, S\. Wu, J\. Wu, M\. Wu, K\. Xiao, T\. Xu, S\. Yoo, K\. Yu, Q\. Yuan, W\. Zaremba, R\. Zellers, C\. Zhang, M\. Zhang, S\. Zhao, T\. Zheng, J\. Zhuang, W\. Zhuk, and B\. Zoph \(2024b\)GPT\-4 technical report\.External Links:2303\.08774,[Link](https://arxiv.org/abs/2303.08774)Cited by:[§4\.1](https://arxiv.org/html/2605.06283#S4.SS1.SSS0.Px2.p1.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[§2\.3](https://arxiv.org/html/2605.06283#S2.SS3.p3.1)\.
- P\. Patel, S\. Mishra, M\. Parmar, and C\. Baral \(2022\)Is a question decomposition unit all we need?\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 4553–4569\.External Links:[Link](https://aclanthology.org/2022.emnlp-main.302/),[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.302)Cited by:[§1](https://arxiv.org/html/2605.06283#S1.p2.1)\.
- Y\. Qin, K\. Song, Y\. Hu, W\. Yao, S\. Cho, X\. Wang, X\. Wu, F\. Liu, P\. Liu, and D\. Yu \(2024\)InFoBench: evaluating instruction following ability in large language models\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 13025–13048\.External Links:[Link](https://aclanthology.org/2024.findings-acl.772/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.772)Cited by:[§B\.2](https://arxiv.org/html/2605.06283#A2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2605.06283#S4.SS1.SSS0.Px2.p1.1)\.
- S\. Saha, O\. Levy, A\. Celikyilmaz, M\. Bansal, J\. Weston, and X\. Li \(2024\)Branch\-solve\-merge improves large language model evaluation and generation\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 8352–8370\.External Links:[Link](https://aclanthology.org/2024.naacl-long.462/),[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.462)Cited by:[§1](https://arxiv.org/html/2605.06283#S1.p2.1)\.
- M\. Sclar, Y\. Choi, Y\. Tsvetkov, and A\. Suhr \(2024\)Quantifying language models’ sensitivity to spurious features in prompt design or: how i learned to start worrying about prompt formatting\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=RIu5lyNXjT)Cited by:[§1](https://arxiv.org/html/2605.06283#S1.p2.1),[§2\.2](https://arxiv.org/html/2605.06283#S2.SS2.p1.1)\.
- J\. Sweller \(2010\)Element Interactivity and Intrinsic, Extraneous, and Germane Cognitive Load\.Educational Psychology Review22\(2\),pp\. 123–138\(en\)\.External Links:ISSN 1573\-336X,[Link](https://doi.org/10.1007/s10648-010-9128-5),[Document](https://dx.doi.org/10.1007/s10648-010-9128-5)Cited by:[§2\.1](https://arxiv.org/html/2605.06283#S2.SS1.p4.1)\.
- R\. Taori, I\. Gulrajani, T\. Zhang, Y\. Dubois, X\. Li, C\. Guestrin, P\. Liang, and T\. B\. Hashimoto \(2023\)Alpaca: a strong, replicable instruction\-following model\.Stanford Center for Research on Foundation Models\. https://crfm\. stanford\. edu/2023/03/13/alpaca\. html3\(6\),pp\. 7\.Cited by:[§4\.1](https://arxiv.org/html/2605.06283#S4.SS1.SSS0.Px2.p1.1)\.
- P\. Virtanen, R\. Gommers, T\. E\. Oliphant, M\. Haberland, T\. Reddy, D\. Cournapeau, E\. Burovski, P\. Peterson, W\. Weckesser, J\. Bright, S\. J\. van der Walt, M\. Brett, J\. Wilson, K\. J\. Millman, N\. Mayorov, A\. R\. J\. Nelson, E\. Jones, R\. Kern, E\. Larson, C\. J\. Carey, İ\. Polat, Y\. Feng, E\. W\. Moore, J\. VanderPlas, D\. Laxalde, J\. Perktold, R\. Cimrman, I\. Henriksen, E\. A\. Quintero, C\. R\. Harris, A\. M\. Archibald, A\. H\. Ribeiro, F\. Pedregosa, P\. van Mulbregt, A\. Vijaykumar, A\. P\. Bardelli, A\. Rothberg, A\. Hilboll, A\. Kloeckner, A\. Scopatz, A\. Lee, A\. Rokem, C\. N\. Woods, C\. Fulton, C\. Masson, C\. Häggström, C\. Fitzgerald, D\. A\. Nicholson, D\. R\. Hagen, D\. V\. Pasechnik, E\. Olivetti, E\. Martin, E\. Wieser, F\. Silva, F\. Lenders, F\. Wilhelm, G\. Young, G\. A\. Price, G\. Ingold, G\. E\. Allen, G\. R\. Lee, H\. Audren, I\. Probst, J\. P\. Dietrich, J\. Silterra, J\. T\. Webber, J\. Slavič, J\. Nothman, J\. Buchner, J\. Kulick, J\. L\. Schönberger, J\. V\. de Miranda Cardoso, J\. Reimer, J\. Harrington, J\. L\. C\. Rodríguez, J\. Nunez\-Iglesias, J\. Kuczynski, K\. Tritz, M\. Thoma, M\. Newville, M\. Kümmerer, M\. Bolingbroke, M\. Tartre, M\. Pak, N\. J\. Smith, N\. Nowaczyk, N\. Shebanov, O\. Pavlyk, P\. A\. Brodtkorb, P\. Lee, R\. T\. McGibbon, R\. Feldbauer, S\. Lewis, S\. Tygier, S\. Sievert, S\. Vigna, S\. Peterson, S\. More, T\. Pudlik, T\. Oshima, T\. J\. Pingel, T\. P\. Robitaille, T\. Spura, T\. R\. Jones, T\. Cera, T\. Leslie, T\. Zito, T\. Krauss, U\. Upadhyay, Y\. O\. Halchenko, Y\. Vázquez\-Baeza, and SciPy 1\.0 Contributors \(2020\)SciPy 1\.0: fundamental algorithms for scientific computing in Python\.Nature Methods17\(3\),pp\. 261–272\.External Links:ISSN 1548\-7105,[Link](https://doi.org/10.1038/s41592-019-0686-2),[Document](https://dx.doi.org/10.1038/s41592-019-0686-2)Cited by:[§4\.3](https://arxiv.org/html/2605.06283#S4.SS3.p1.2)\.
- M\. Wu and A\. Quinn \(2017\)Confusing the crowd: task instruction quality on amazon mechanical turk\.Proceedings of the AAAI Conference on Human Computation and Crowdsourcing5\(1\),pp\. 206–215\.External Links:[Link](https://ojs.aaai.org/index.php/HCOMP/article/view/13317),[Document](https://dx.doi.org/10.1609/hcomp.v5i1.13317)Cited by:[§1](https://arxiv.org/html/2605.06283#S1.p1.1),[§1](https://arxiv.org/html/2605.06283#S1.p2.1)\.
- C\. Xiao, W\. Ma, Q\. Song, S\. X\. Xu, K\. Zhang, Y\. Wang, and Q\. Fu \(2025\)Human\-ai collaborative essay scoring: a dual\-process framework with llms\.InProceedings of the 15th International Learning Analytics and Knowledge Conference,LAK ’25,New York, NY, USA,pp\. 293–305\.External Links:ISBN 9798400707018,[Link](https://doi.org/10.1145/3706468.3706507),[Document](https://dx.doi.org/10.1145/3706468.3706507)Cited by:[§2\.3](https://arxiv.org/html/2605.06283#S2.SS3.p1.1),[§2\.3](https://arxiv.org/html/2605.06283#S2.SS3.p2.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica \(2023\)Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.InProceedings of the 37th International Conference on Neural Information Processing Systems,NIPS ’23,Red Hook, NY, USA\.Cited by:[§1](https://arxiv.org/html/2605.06283#S1.p2.1)\.
- X\. Zhou, H\. Zhu, L\. Mathur, R\. Zhang, H\. Yu, Z\. Qi, L\. Morency, Y\. Bisk, D\. Fried, G\. Neubig, and M\. Sap \(2024\)SOTOPIA: interactive evaluation for social intelligence in language agents\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=mM7VurbA4r)Cited by:[§4\.2](https://arxiv.org/html/2605.06283#S4.SS2.p1.1)\.

## Appendix AAppendix A

### A\.1Additional Other Observations

PC\.Full3exSep\.Bat\.Edited1H\.0\.6390\.6390\.6230\.623\-\-\-Ide\.\-\-0\.6470\.6470\.6520\.6520\.6600\.660Or\.\-\-0\.6750\.6750\.6260\.6260\.6720\.672WC\-\-0\.6160\.6160\.5800\.5800\.6390\.639SF\-\-0\.6080\.6080\.5860\.5860\.6320\.632Cv\.\-\-0\.5800\.5800\.5590\.5590\.6100\.6104H\.0\.8250\.8250\.8120\.812\-\-\-Ct\.\-\-0\.7770\.7770\.8150\.8150\.8010\.801PA\-\-0\.7670\.7670\.7730\.7730\.7840\.784La\.\-\-0\.7550\.7550\.7110\.7110\.7660\.766Na\.\-\-0\.7670\.7670\.7470\.7470\.7890\.7896H\.0\.7900\.7900\.7820\.782\-\-\-Ct\.\-\-0\.7160\.7160\.7790\.7790\.7720\.772PA\-\-0\.7120\.7120\.7460\.7460\.7800\.780La\.\-\-0\.7450\.7450\.7050\.7050\.7530\.753Na\.\-\-0\.7490\.7490\.7090\.7090\.7720\.772Table 3:Kendall’sτ\\taubetween GPT and Llama\. All abbreviations follow Table[1](https://arxiv.org/html/2605.06283#S4.T1)\.![Refer to caption](https://arxiv.org/html/2605.06283v1/x2.png)Figure 2:This diagram represents comparisons made between human\-autorater agreementsτ\\tauacross various conditions such as holistic rubrics \(left side\), in which all criteria are applied together in a single overall judgment, or analytic rubrics \(right side\), in which criteria are evaluated separately, and rubrics which have been edited for LLMs\. Arrow in bold betweenτ5\\tau\_\{5\}andτ6\\tau\_\{6\}represent comparisons for which statistical significance can be calculated\.Δ​rater\+rubric\\Delta\\textbf\{rater\+rubric\}GPT\-4oLlamaLLMH→HA\\textnormal\{LLM\}\_\{H\}\\to\\textnormal\{H\}\_\{A\}HH→LLMA\\textnormal\{H\}\_\{H\}\\to\\textnormal\{LLM\}\_\{A\}LLMH→HA\\textnormal\{LLM\}\_\{H\}\\to\\textnormal\{H\}\_\{A\}HH→LLMA\\textnormal\{H\}\_\{H\}\\to\\textnormal\{LLM\}\_\{A\}AES10\.574†0\.574^\{\\dagger\}0\.5390\.5390\.2790\.2790\.2710\.2710\.401s,b↑0\.401^\{s,b\\uparrow\}0\.6010\.6010\.6050\.6050\.4670\.4670\.4670\.4670\.4700\.47040\.6910\.6910\.6880\.6880\.6690\.6690\.6680\.6680\.682s,b↑0\.682^\{s,b\\uparrow\}0\.6740\.6740\.6740\.6740\.710⋆0\.710^\{\\star\}0\.6740\.6740\.707b↑0\.707^\{b\\uparrow\}60\.6450\.6450\.653†0\.653^\{\\dagger\}0\.5850\.5850\.595⋆0\.595^\{\\star\}0\.644s,b↑0\.644^\{s,b\\uparrow\}0\.6600\.6600\.670†0\.670^\{\\dagger\}0\.637⋆0\.637^\{\\star\}0\.5920\.5920\.671s,b↑0\.671^\{s,b\\uparrow\}IF0\.5410\.5410\.5300\.5300\.474⋆0\.474^\{\\star\}0\.340\.340\.459b↑0\.459^\{b\\uparrow\}0\.4780\.4780\.5450\.5450\.421⋆0\.421^\{\\star\}0\.1830\.1830\.424b↑0\.424^\{b\\uparrow\}Table 4:Kendall’sτ\\tauresults forΔ\\Deltarater\+rubric with GPT\-4o and Llama on AES and IF\. For AES, the number indicates the essay prompt, with ideas, organization, word choice, sentence fluency, and conventions compared for prompt 1, and content, prompt adherence, language, and narrativity compared for prompts 4 and 6\.τ\\tauis calculated through Pareto dominance comparison for AES and through instruction following ratio comparison for IF\. Significance tests between separate \(sep\.\), batch \(bat\.\), and edited prompts are performed, wheresandbin the edited prompt column represents significant differences with separate and batch prompts respectively\.†\\daggeris indicated next to comparisons that are significantly larger within holistic prompts\.⋆\\staris indicated next to comparisons that are significantly larger between separate and batch comparisons\.↑\\uparrowand↓\\downarrowrepresent that theτ\\tauvalue for edited prompts is significantly larger or smaller respectively with the separatessor batchbbprompts’τ\\tau\. The lack of any dagger, star, or arrow denotes no statistical significance\.His a shortened form forHuman\.#### Δ\\Deltarater \+ rubric\.

Different raters may show greater agreement when using different rubrics\. In additional to the previous comparisons ofΔ\\DeltaRaterandΔ\\DeltaRubricwe also investigateΔ\\DeltaRater\+Rubric, which captures cases where alignment between different raters is affected by the use of different rubric types as depicted in Figure[2](https://arxiv.org/html/2605.06283#A1.F2)\. The bestτ\\tauobtained for any rubric condition when comparing human ratings on holistic rubrics with autoraters on analytic rubrics \(HumanH\\textnormal\{Human\}\_\{H\}→\\toLLMA\\textnormal\{LLM\}\_\{A\}, 0\.401, 0\.682, and 0\.644 respectively\) is lower than the worstτ\\tauobtained from comparing autoraters on holistic rubrics with humans on analytic rubrics \(LLMH\\textnormal\{LLM\}\_\{H\}→\\toHumanA\\textnormal\{Human\}\_\{A\}, 0\.539, 0\.688, and 0\.645 respectively\) \(Table[1](https://arxiv.org/html/2605.06283#S4.T1)\)\. This indicates that while autoraters exhibit high preference agreement across different rubrics \(inΔ\\Deltarubric\), the Pareto dominance aggregation method decreases alignment between autoraters and humans\. This trend is also observed in Table[2](https://arxiv.org/html/2605.06283#S4.T2)\(0\.474 to 0\.530, and 0\.424 to 0\.478\) for IF, suggesting its applicability to the ratio aggregation method\. However, in Table[1](https://arxiv.org/html/2605.06283#S4.T1)on Llama for AES, this only occurs for essay prompt 1 \(0\.470 to 0\.601\)\.

Surprisingly, for essay prompt 4, allτ\\tauvalues forΔ\\Deltarater \+ rubric are higher than those forΔ\\Deltarubric when humans are the annotator, for both GPT\-4o \(0\.656 compared with 0\.691 and 0\.688\) and Llama \(0\.656 compared with 0\.674 and 0\.674\)\. This suggests that the autorater’s preferences on the holistic rubric agree slightly more than the human’s preferences on the holistic rubric as compared to the analytic rubric\. Although different humans rate with the holistic and analytic rubrics, they all either were trained or familiar with evaluation, whereas there is no guarantee for autorater performance\. This could be due to various factors, such as the autorater using a continuous scale versus the human’s discrete scale which leads to less ties, or the autorater’s interpretation being closer to the human’s analytic rubric interpretation\. Future work should further explore these correlations\.

#### GPT\-4 vs\. Llama\.

It is hypothesized that autoraters will exhibit greater mutual alignment when \(1\) using analytic prompts instead of holistic prompts, \(2\) using the same \(and more\) examples, and \(3\) using separate API calls instead of batched API calls\. To test these hypotheses, Kendall’sτ\\tauis calculated between GPT\-4 ratings and Llama ratings in Table[3](https://arxiv.org/html/2605.06283#A1.T3)\. Analytic rubrics do not consistently improve autorater alignment over holistic rubrics, even when provided examples in AES \(0\.639 compared to 0\.610 \- 0\.672, 0\.825 compared to 0\.766 \- 0\.801, and 0\.790 to 0\.753 \- 0\.780 respectively\)\. However, introducing examples to analytic rubrics \(comparing separate and batch with the edited column\) and increasing the number of examples in holistic rubrics \(0\.639 from 0\.623, 0\.825 from 0\.812, and 0\.790 from 0\.782\) does tend to improve alignment, indicating that examples improve both human\-autorater and autorater\-autorater alignment in AES\. This pattern does not hold for IF, where the separate, batch, and edited conditions yieldτ\\tauvalues of 0\.674, 0\.193, and 0\.662, respectively, while the holistic zero\-shot and three\-shot conditions yield 0\.752 and 0\.724\. Separate API calls improve alignment for 8 out of 13 sub\-criteria in AES\. However, as previously mentioned, further research is needed to understand how humans and autoraters exhibit positional bias, as the sub\-criteria presented first in batched API calls consistently show higher alignment\.

Table 5:ASAP Chosen Essay Prompts

### A\.2Agreement Level

Table 6:Kendall’sτ\\tauresults on AES with GPT\-4o and Llama\. The number indicated by the†\\daggerrepresents that theτ\\tauvalue on that subset of data \(in all cases here, where raters agreed with each other\) is significantly larger than theτ\\tauvalue on the other subset of data \(where raters disagreed with each other\)\. The lack of any arrow denotes no statistical significance\.Table 7:Kendall’sτ\\tauresults on IF with GPT\-4o forΔ\\Deltarater\. Significance tests between data with full agreement, partial agreement, and full disagreement are performed, wherepandfin the full agreement row represents significant differences with partial disagreement and full disagreement data respectively\.↑\\uparrowand↓\\downarrowrepresent that theτ\\tauvalue is significantly larger or smaller respectively with comparedτ\\tau\. The lack of any arrow denotes no statistical significance\.Agreement level is examined using holistic prompts in both AES and IF, and analytic prompts in IF\. As shown in Table[6](https://arxiv.org/html/2605.06283#A1.T6)with GPT\-4o, for essay prompts 4 and 6 \(where both raters agreed on 77\.1% and 62\.2% of essays, respectively\), essays with rater agreement exhibited higherτ\\tauthan those without \(for full examples,τ\\tauof 0\.687 to 0\.525 and 0\.659 to 0\.642 respectively, for 3ex,τ\\tauof 0\.678 to 0\.512 and 0\.675 to 0\.653 respectively\), with a significant difference in essay prompt 4\. However, this pattern did not hold for essay prompt 1 \(where raters agreed on 65\.3% of essays, for full examples,τ\\taurose from 0\.450 to 0\.454, and for 3ex,τ\\taurose from 0\.398 to 0\.413 albeit not significantly\)\. Despite greater disagreement among human raters, the autorater showed higher agreement with the aggregated human scores\. This discrepancy might stem from the different aggregation methods used: The final score for essay prompt 1 is the average of both raters’ scores, while essay prompts 4 and 6 use a single rater’s score \(or a third expert rater’s score\)\. Averaging scores could diminish the impact of rater disagreement onτ\\tauin essay prompt 1\. Llama ratings mirror these patterns\.

In the IF dataset, holistic prompt ratings \(with three human raters\) were divided into three subsets: full agreement \(all three raters agree, 25\.9% of the dataset\), partial agreement \(two raters agree, 53\.3% of the dataset\), and full disagreement \(no raters agree, 20\.9% of the dataset\)\. The final score was obtained by averaging the three raters’ scores\. Analytic prompt ratings were divided into full agreement \(13% of the dataset\) and partial agreement \(87% of the dataset\) subsets, using majority vote for the final score\. Kendall’sτ\\tauwas calculated within the subsets for each prompt\. Table[7](https://arxiv.org/html/2605.06283#A1.T7)demonstrates a consistent trend for both GPT\-4o and Llama: lower human agreement corresponded to lowerτ\\tau, with most comparisons showing statistical significance \(for GPT\-4o, for the holistic prompt, with no examples,τ\\taudropped from 0\.792 to 0\.559 to 0\.220, with 3 examples,τ\\taudropped from 0\.803 to 0\.584 to 0\.268; while for the analytic prompt, separate dropped from 0\.299 to 0\.251, batch from 0\.078 to 0\.028, and edited from 0\.31 to 0\.261 respectively\) \. To achieve high human\-autorater alignment, one must first establish high human\-human alignment\. While a rubric cannot fix inherently ambiguous data, it is the primary tool for reducing rater disagreement and creating the stable ‘ground truth’ necessary for meaningfully evaluating and training an autorater\.

## Appendix BAppendix B

### B\.1Prompt for AES

The holistic and analytic rubrics within the prompts are taken from ASAP\(Hamneret al\.,[2012](https://arxiv.org/html/2605.06283#bib.bib35)\)and ASAP\+\+\(Mathias and Bhattacharyya,[2018](https://arxiv.org/html/2605.06283#bib.bib34)\)\. The contextual information present in the grading guidelines are formatted into the prompt, and additionally added with slight edits to the analytic prompts in the edited condition\. The examples chosen from ASAP\+\+ are essay IDs 449, 1264, 1616, 9125, 9430, 9497, 15558, 16520, and 16581\.

Prompt 1 Holistic ContextInstructions:1\) The following essay is a first draft written by an 8th grade student in forty\-five minutes in reaction to a prompt designed to elicit persuasive writing\. You will score these timed responses holistically, which means that you will determine a score based on the overall impression most often gained from a single reading of the response\.2\) You will be given a rubric that outlines a six\-point scale\. Each score point on that scale is described by an overall statement which captures the essence of the response\. The elements of the response \(elaboration, organization, fluency and audience awareness\) that are typical for that score point are described below the overall statement\. Individual responses may be stronger in one feature and weaker in another\. In other words, the list of features at each score point, while helpful, cannot perfectly describe every response in a score point category\.3\) A committee of expert readers uses this rubric as a guide to select anchor papers for each score point\. Anchor papers are examples of actual student work\. The committee prepares an anchor set composed of several papers at each score point\. They deliberately select papers to show an appropriate range of writing skill for each score point and to represent the variety of approaches students take when addressing the writing prompt\. You rely heavily on these anchor sets to guide your scoring\.4\) Errors in spelling, punctuation, grammar, and usage are not considered as part of the criteria for scoring\. Also, papers receive a score based on the work the student did complete even if they seem to be unfinished\. Because the writing sample is a timed response, it is generally assumed that these errors and omissions could have been corrected if the student had been given an opportunity to revise and edit\. You are trained to read through these errors when you score student papers\.5\) Score the essay on a scale from 1 to 6\.

Prompt 4 Holistic ContextInstructions:1\) The following essay is written by an 10th grade student in response to a prompt that is dependent on reading the story provided\.2\) You will be given a rubric that outlines a four\-point scale\.3\) Training materials consist of a rubric and a scoring guide of ten responses\.4\) Score the essay on a scale from 0 to 3\.

Prompt 6 Holistic ContextInstructions:1\) The following essay is written by an 10th grade student in response to a prompt that is dependent on reading the excerpt provided\.2\) You have a four year baccalaureate degree as well as documented coursework in English\. You are not a teacher, substitute teacher, support staff, tutor, administrator, etc\., who is currently under contract or employed by or in schools, or under 18 years of age\.3\) You will be given a rubric that outlines a five\-point scale\.4\) You will be given an anchor set which will consist of responses that are typical, rather than unusual or uncommon; solid, rather than controversial or borderline; and true, meaning that these have scores that cannot be changed by anyone other than pertinent personnel\. Anchor sets will typically have 2 to 3 sample responses at each score point \(the middle score points will have 3 sample responses, 1 representing the mid\-high to high end of the score point range, 1 in the middle, and 1 at the mid\-low to low end\)\.5\) Score the essay on a scale from 0 to 4\.

Example of context given to an analytic prompt for separate \(Prompt 1\)Instructions:1\) The following essay is a first draft written by an 8th grade student in forty\-five minutes in reaction to a prompt designed to elicit persuasive writing\.2\) You will be given a rubric that outlines a six\-point scale for an attribute\.3\) Score the essay on a scale from 1 to 6 on the attribute\.

Example of context given to an analytic prompt for edited \(Prompt 1\)Instructions:1\) The following essay is a first draft written by an 8th grade student in forty\-five minutes in reaction to a prompt designed to elicit persuasive writing\.2\) You will be given a rubric that outlines a six\-point scale for an attribute\.3\) A committee of expert readers uses this rubric as a guide to select anchor papers for some score points\. Anchor papers are examples of actual student work\. The committee prepares an anchor set composed of several papers at various score points\. They deliberately select papers to show an appropriate range of writing skill and to represent the variety of approaches students take when addressing the writing prompt\. You rely heavily on these anchor sets to guide your scoring\.4\) Also, papers receive a score based on the work the student did complete even if they seem to be unfinished\. Because the writing sample is a timed response, it is generally assumed that these errors and omissions could have been corrected if the student had been given an opportunity to revise and edit\. You are trained to read through these errors when you score student papers\.5\) Score the essay on a scale from 1 to 6 on the attribute\.

### B\.2Prompt for Instruction Following

The holistic and analytic prompt are structured to match the annotation instructions presented to human annotators of the InfoBench dataset\(Qinet al\.,[2024](https://arxiv.org/html/2605.06283#bib.bib9)\)\.

Evaluation PromptI will show you examples of how to evaluate system responses to specific criteria\.EXAMPLE 1: \{\}EXAMPLE 2: \{\}EXAMPLE 3: \{\}Now, please evaluate the current response based on the following criteria:Based on the provided Input and Generated Text, answer the following Question with either a YES or NO choice\. Your selection should be based on your judgment as well as the following rules:\- YES: Select ’YES’ if the generated text entirely fulfills the condition specified in the question\. However, note that even minor inaccuracies exclude the text from receiving a ’YES’ rating\. As an illustration, consider a question that asks, "Does each sentence in the generated text use a second person?" If even one sentence does not use the second person, the answer should NOT be ’YES’\. To qualify for a ’YES’ rating, the generated text must be entirely accurate and relevant to the question\.\- NO: Opt for ’NO’ if the generated text fails to meet the question’s requirements or provides no information that could be utilized to answer the question\. For instance, if the question asks, "Is the second sentence in the generated text a compound sentence?" and the generated text only has one sentence, it offers no relevant information to answer the question\. Consequently, the answer should be ’NO’\.Input: \{\} Generated Text \(System’s Response\): \{\}Question: Does every chapter in the generated list include a description?

Example 1Instruction: Create two tables with three columns each, using \|̈ẗo separate the columns, and include an appropriate header for each column:a\. In the first table, display seven interesting geological formations or landmarks worldwide, including a short description and location\. Limit descriptions to 40 words\.b\. In the second table, present three main types of rocks, a brief description of their formation process, and common examples\. Limit each rock type description to 50 words\.Generated Text \(System’s Response\): a\. Geological Formations/Landmarks Table:\| Formation Name \| Description \| Location \|\|—————\-\|————\-\|———\-\|\| Grand Canyon \| A vast, colorful canyon with layered bands of red rock revealing millions of years of geological history\. \| Arizona, USA \|\| Great Barrier Reef \| The world’s largest coral reef system, visible from space, teeming with marine life\. \| Queensland, Australia \|\| Mount Everest \| Earth’s highest mountain above sea level, a part of the Himalayan range\. \| Nepal/China border \|\| Dead Sea \| A salt lake whose shores are the lowest land\-based elevation on Earth, known for its buoyancy\. \| Jordan/Israel border \|\| Uluru \(Ayers Rock\) \| A massive sandstone monolith sacred to indigenous Australians, known for its red hue\. \| Northern Territory, Australia \|\| Yellowstone Caldera \| A volcanic caldera and supervolcano with geysers, hot springs, and diverse wildlife\. \| Wyoming, USA \|\| Giant’s Causeway \| An area of about 40,000 interlocking basalt columns, the result of an ancient volcanic eruption\. \| County Antrim, N\. Ireland \|b\. Rock Types Table:\| Rock Type \| Formation Process \| Common Examples \|\|———–\|——————\-\|—————–\|\| Igneous \| Formed from the solidification of molten rock material \(magma or lava\)\. \| Granite, basalt, obsidian \|\| Sedimentary \| Created by the accumulation and compaction of sediment or by the precipitation of minerals from water\. \| Sandstone, limestone, shale \|\| Metamorphic \| Formed through the alteration of other rocks under high pressure and temperature without melting\. \| Slate, marble, gneiss \|\{Questions for Example 1\}

Questions for Example 1Question 1: Does the generated text include exactly two tables?Answer: YESQuestion 2: Do both tables in the generated text consist of three columns each?Answer: YESQuestion 3: Are the columns in both tables in the generated text separated by \|̈?̈Answer: YESQuestion 4: Do both tables in the generated text include an appropriate header for each column?Answer: YESQuestion 5: Does the first table in the generated text display seven interesting geological formations or landmarks worldwide?Answer: YESQuestion 6: Does each entry in the first table of the generated text include a short description and location for each geological formation or landmark?Answer: YESQuestion 7: Are the descriptions in the first table of the generated text limited to 40 words or less for each geological formation or landmark?Answer: YESQuestion 8: Does the second table in the generated text present exactly three main types of rocks?Answer: YESQuestion 9: Does each entry in the second table in the generated text include a brief description of the rock type’s formation process?Answer: YESQuestion 10: Does each entry in the second table in the generated text provide common examples for each rock type?Answer: YESQuestion 11: Are the descriptions in the second table of the generated text limited to 50 words or less for each rock type?Answer: YES

Example 2Input: A research study has been conducted to determine if exercise really can "boost" your metabolism\.Instruction: You should choose a YouTube video title based on the video’s content\. A video’s title tells viewers what to expect from it\. It should be direct, honest, and clear\. The title of the video needs to capture the attention of viewers, so do not use an unclear or ambiguous one\.Generated Text \(System’s Response\): Yes, exercise can increase your metabolism\. Studies have found that when you exercise, your body produces more heat and requires more energy to perform basic tasks\. This can lead to an increase in metabolic rate of up to 14%, which can last up to 13 hours after your workout\. Additionally, regular exercise can also help to reduce fatigue and increase energy levels\.Question 1: Is the generated text a YouTube video title?Answer: NOQuestion 2: Is the generated title direct, honest, and clear?Answer: NOQuestion 3: Is the generated title unambiguous?Answer: NOQuestion 4: Is the generated title based on the given video content?Answer: NOQuestion 5: Does the generated title tell viewers what to expect?Answer: NO

Example 3Input: Item, Qty, Price, Subtotal \| mango, 5, 500, ? \| apple, 4, 150, ? \| oil, 2, 1500, ?Function: column2 \* column 3Instruction: We have entered the home supplies budget in the following table\. Calculate the last column with the given function and rewrite the completed table\.Note that, columns in a row are separated using a comma, and rows are separated using a "\|"\.Generated Text \(System’s Response\): Item, Qty, Price, Subtotalmango, 5, 500, 2500apple, 4, 150, 600oil, 2, 1500, 3000Question 1: Is the generated text a table?Answer: NOQuestion 2: Are the columns in a row in the generated table separated by a comma?Answer: YESQuestion 3: Are the rows in the generated table separated by a \|̈?̈Answer: NOQuestion 4: Is the generated table complete?Answer: YESQuestion 5: Is the generated table copied from the given table except for the last column?Answer: NOQuestion 6: Is the last column in the generated table correctly calculated by the function in the given input?Answer: YES