Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why

# What to Report and Why
Source: [https://arxiv.org/html/2606.00093](https://arxiv.org/html/2606.00093)
## Agreement Metrics for LLM\-as\-Judge Evaluation: What to Report and Why

Delip Rao University of Pennsylvania delip@seas\.upenn\.edu &Chris Callison\-Burch University of Pennsylvania ccb@seas\.upenn\.edu

###### Abstract

Validating an LLM judge against human annotations usually means reporting several agreement statistics: accuracy, precision, recall,F1F\_\{1\}, Cohen’sκ\\kappa, and one or more rank correlations\. A survey of 24 recent LLM\-as\-judge papers finds metric choice entangled with the judgment scale, tie handling, invalid outputs, and abstention handling, and those choices rarely stated\. For binary criteria — the common case in rubric\-based evaluation, where each criterion is gradedMETorUNMET— most of the reported numbers are redundant: Pearson’srr, Spearman’sρ\\rho, Kendall’sτb\\tau\_\{b\}, the phi coefficientϕ\\phi, and the Matthews Correlation Coefficient all reduce to a single number on non\-degenerate binary data, so reporting several of them only creates an illusion of corroborating evidence\. Cohen’sκ\\kappais the one agreement coefficient that adds information: it sharesϕ\\phi’s numerator but normalizes differently, and the gap between them measures how far the judge’s positive\-label rate has drifted from the human’s\. We then trace what changes when a judge may abstain with aCANNOT\_ASSESSverdict: the three common ways of handling abstentions are not interchangeable preprocessing choices but answer different questions, and they break the binary equivalences\. The same equivalences reappear, up to a negligible finite\-sample correction, for multi\-judge ensembles scored with Fleiss’κ\\kappaor Krippendorff’sα\\alpha\. We close with a reporting checklist that names the judgment scale, the abstention and tie handling mode, coverage, the confusion matrix, and the aggregation level alongside any scalar agreement coefficient\.

Agreement Metrics for LLM\-as\-Judge Evaluation: What to Report and Why

Delip Rao††thanks:Corresponding authorUniversity of Pennsylvaniadelip@seas\.upenn\.eduChris Callison\-BurchUniversity of Pennsylvaniaccb@seas\.upenn\.edu

![[Uncaptioned image]](https://arxiv.org/html/2606.00093v1/figures/key_illustration.png)

Figure 1:Validating an LLM judge yields many agreement coefficients\. This paper asks which are independent and which merely restate one another, how judges that abstain complicate the analysis, and how to handle multiple judges\.## 1Introduction

An LLM judge earns trust only after it is validated against human annotations, and that validation reaches the reader as a set of agreement statistics: accuracy, precision, recall,F1F\_\{1\}, Cohen’sκ\\kappa, and one or more rank correlations \(Pearson, Spearman, Kendall\)\. A common case is*rubric\-based evaluation*, where a holistic judgment is decomposed into independently graded criteria, each typically binary \(MET/UNMET\)\(Hashemi et al\.,[2024](https://arxiv.org/html/2606.00093#bib.bib16); Saad\-Falcon et al\.,[2023](https://arxiv.org/html/2606.00093#bib.bib33)\)\. To see how these statistics are reported in practice, we surveyed 24 recent LLM\-as\-judge and LLM\-evaluator papers \(full per\-system coding in Appendix[A](https://arxiv.org/html/2606.00093#A1)\)\. Metric choice tracks the judgment scale: several rank correlations are reported together for graded or continuous scores\(e\.g\., Liu et al\.,[2023](https://arxiv.org/html/2606.00093#bib.bib29); Ye et al\.,[2024](https://arxiv.org/html/2606.00093#bib.bib42); Kim et al\.,[2024](https://arxiv.org/html/2606.00093#bib.bib22)\), while accuracy, agreement, win rate,F1F\_\{1\}, orκ\\kappaare used for binary or pairwise verdicts\(e\.g\., Zheng et al\.,[2023](https://arxiv.org/html/2606.00093#bib.bib44); Verga et al\.,[2024](https://arxiv.org/html/2606.00093#bib.bib35); Lee et al\.,[2026](https://arxiv.org/html/2606.00093#bib.bib24)\)\. The handling of ties, invalid outputs, refusals, and insufficient\-information \(CANNOT\_ASSESS\) responses111Equivalent labels in the literature includeIDK,REFUSE, andABSTAIN\.is heterogeneous — sometimes excluded, sometimes folded into a valid label — and almost never reported as a deliberate choice\.

What current papers do not articulate is what these choices cost\. On binary verdicts, are several reported coefficients independent evidence or one number computed several ways? When a judge may abstain, does the resolution of that abstention change the quantity being estimated, or is it harmless preprocessing? The relationships that settle such questions are classical\(Cohen,[1960](https://arxiv.org/html/2606.00093#bib.bib5); Matthews,[1975](https://arxiv.org/html/2606.00093#bib.bib30); Warrens,[2008](https://arxiv.org/html/2606.00093#bib.bib38)\), but they are rarely connected to how LLM judges are scored, and abstention is treated as a preprocessing detail rather than a modeling choice\. The gap is not missing mathematics but a missing account of what the reported numbers mean on the scales LLM judges actually produce\.

This paper supplies that account\. It is an analysis with reporting guidelines, not a comprehensive survey: the contributions are an audit of how 24 recent systems report agreement \(Appendix[A](https://arxiv.org/html/2606.00093#A1)\), a unifying account of which binary agreement statistics are redundant and what their differences measure, an estimand\-level treatment of theCANNOT\_ASSESSverdict, and a reporting checklist \(Section[8](https://arxiv.org/html/2606.00093#S8)\)\. The analysis needs nothing beyond the2×22\\times 2confusion matrix a validation already produces\. First, on non\-degenerate binary verdicts, Pearson’srr, Spearman’sρ\\rho, Kendall’sτb\\tau\_\{b\},ϕ\\phi, and MCC are a single statistic, and Cohen’sκ\\kappadiffers from it only through a marginal\-sensitive normalization; we draw out the reporting consequences \(Sections[3](https://arxiv.org/html/2606.00093#S3)–[5](https://arxiv.org/html/2606.00093#S5)\)\. Second, theCANNOT\_ASSESSverdict is a choice of estimand, not a preprocessing step: the three handling modes answer different questions — a selective\-prediction view\(Chow,[1970](https://arxiv.org/html/2606.00093#bib.bib4); El\-Yaniv and Wiener,[2010](https://arxiv.org/html/2606.00093#bib.bib8); Kamath et al\.,[2020](https://arxiv.org/html/2606.00093#bib.bib19)\)— and the binary equivalence breaks once a third class is admitted \(Section[6](https://arxiv.org/html/2606.00093#S6)\)\. Third, the equivalence extends to multi\-judge ensembles, where Fleiss’κ\\kappa, Krippendorff’sα\\alpha, and average pairwiseϕ\\phiagree up to a negligible finite\-sample correction \(Section[7](https://arxiv.org/html/2606.00093#S7)\)\. We close with a reporting checklist \(Section[8](https://arxiv.org/html/2606.00093#S8)\) that names the judgment scale, the abstention and tie handling mode, coverage, the confusion matrix, and the aggregation level alongside any scalar\.

## 2Preliminaries

Consider the binary classification problem induced by per\-criterion judgments\. Letyi∈\{0,1\}y\_\{i\}\\in\\\{0,1\\\}denote the ground\-truth label for itemi∈\{1,…,N\}i\\in\\\{1,\\ldots,N\\\}, with11encoding the positive verdict \(e\.g\.,MET\) and0the negative verdict \(UNMET\); lety^i\\hat\{y\}\_\{i\}denote the LLM judge’s prediction\. Abstention is deferred to Section[6](https://arxiv.org/html/2606.00093#S6)\. Each cell of the2×22\\times 2confusion matrix is a dataset\-level count: the number of itemsiiwhose ground\-truth labelyiy\_\{i\}and predicted labely^i\\hat\{y\}\_\{i\}equal the row and column values\. Writing this count asna​b=\|\{i:yi=a,y^i=b\}\|n\_\{ab\}=\|\\\{i:y\_\{i\}=a,\\ \\hat\{y\}\_\{i\}=b\\\}\|, the four cells areT​P=n11TP=n\_\{11\},F​N=n10FN=n\_\{10\},F​P=n01FP=n\_\{01\}, andT​N=n00TN=n\_\{00\}:

y^=1y^=0y=1T​PF​Ny=0F​PT​N\\begin\{array\}\[\]\{c\|cc\}&\\hat\{y\}=1&\\hat\{y\}=0\\\\ \\hline\\cr y=1&TP&FN\\\\ y=0&FP&TN\\end\{array\}withN=T​P\+F​N\+F​P\+T​NN=TP\+FN\+FP\+TNthe total number of items\. Denoteπ=\(T​P\+F​N\)/N\\pi=\(TP\+FN\)/Nfor the true prevalence andπ^=\(T​P\+F​P\)/N\\hat\{\\pi\}=\(TP\+FP\)/Nfor the predicted positive rate\.

### 2\.1Judgment Scale and Scope

The identities below are exact for binary vectors; they do not make Pearson, Spearman, and Kendall redundant on graded, ordinal, or continuous scores\. The first reporting decision is therefore what judgment scale the evaluator has produced, not which metric to compute\. Table[1](https://arxiv.org/html/2606.00093#S2.T1)names the five scales encountered in recent work and the metric framing each calls for\.

Table 1:A judgment\-scale taxonomy for LLM\-as\-judge evaluation\. The equivalence results concern the binary row; the nominal\-with\-abstention row is taken up in Section[6](https://arxiv.org/html/2606.00093#S6)\.The remainder of the paper develops the binary row \(Sections[3](https://arxiv.org/html/2606.00093#S3)–[5](https://arxiv.org/html/2606.00093#S5)\) and, once abstention is admitted, the nominal row \(Section[6](https://arxiv.org/html/2606.00093#S6)\); the ordinal and continuous scales lie outside the scope of these results\.

## 3Closed\-Form Metric Definitions

The standard definitions, expressed directly in terms ofT​P,F​P,F​N,T​NTP,FP,FN,TN:

#### Accuracy:

Acc=T​P\+T​NN≡po\.\\text\{Acc\}=\\frac\{TP\+TN\}\{N\}\\equiv p\_\{o\}\.\(1\)

#### F1, precision, and recall \(positive class\):

With precisionP=T​P/\(T​P\+F​P\)P=TP/\(TP\+FP\)and recallR=T​P/\(T​P\+F​N\)R=TP/\(TP\+FN\), theF1F\_\{1\}score is their harmonic mean

F1=2​P​RP\+R=2​T​P2​T​P\+F​P\+F​N\.F\_\{1\}=\\frac\{2\\,P\\,R\}\{P\+R\}=\\frac\{2\\,TP\}\{2\\,TP\+FP\+FN\}\.\(2\)F1F\_\{1\}does not depend onT​NTNand is asymmetric under label flipping \(MET↔\\leftrightarrowUNMET\)\.222The negative\-class analog isF1\(0\)=2​T​N/\(2​T​N\+F​P\+F​N\)F\_\{1\}^\{\(0\)\}=2\\,TN/\(2\\,TN\+FP\+FN\)\.Precision and recall are the exception to the collapse below: every symmetric statistic that follows \(accuracy,κ\\kappa,ϕ\\phi, the rank correlations, andF1F\_\{1\}\) is invariant underF​P↔F​NFP\\leftrightarrow FN, so none distinguishes a judge that misses positives from one that over\-predicts them\.

#### Cohen’sκ\\kappa:

With expected agreementpe=π​π^\+\(1−π\)​\(1−π^\)p\_\{e\}=\\pi\\hat\{\\pi\}\+\(1\-\\pi\)\(1\-\\hat\{\\pi\}\)\(Cohen,[1960](https://arxiv.org/html/2606.00093#bib.bib5)\),

κ=po−pe1−pe\.\\kappa=\\frac\{p\_\{o\}\-p\_\{e\}\}\{1\-p\_\{e\}\}\.\(3\)Multiplying through byN2N^\{2\}and canceling shared terms yields

κ=2​\(T​P⋅T​N−F​P⋅F​N\)\(T​P\+F​N\)​\(F​N\+T​N\)\+\(T​P\+F​P\)​\(F​P\+T​N\)\.\\kappa=\\frac\{2\\,\(TP\\cdot TN\-FP\\cdot FN\)\}\{\\begin\{aligned\} &\(TP\+FN\)\(FN\+TN\)\\\\\[\-2\.0pt\] &\{\}\+\(TP\+FP\)\(FP\+TN\)\\end\{aligned\}\}\.\(4\)

#### Phi coefficient and MCC:

For binary vectors, the Pearson correlation reduces to the phi coefficient\(Matthews,[1975](https://arxiv.org/html/2606.00093#bib.bib30)\),

ϕ\\displaystyle\\phi=T​P⋅T​N−F​P⋅F​N\(T​P\+F​P\)​\(T​P\+F​N\)\\displaystyle=\\frac\{TP\\cdot TN\-FP\\cdot FN\}\{\\sqrt\{\(TP\+FP\)\(TP\+FN\)\}\}\(5\)×1\(T​N\+F​P\)​\(T​N\+F​N\),\\displaystyle\\quad\\times\\frac\{1\}\{\\sqrt\{\(TN\+FP\)\(TN\+FN\)\}\},which is identical to the binary Matthews Correlation Coefficient\.

## 4Equivalence of Five Metrics on Binary Data

Observation 1For any pair of binary vectors\(y,y^\)∈\{0,1\}N\(y,\\hat\{y\}\)\\in\\\{0,1\\\}^\{N\}with non\-degenerate marginals,ρPearson​\(y,y^\)=ρSpearman​\(y,y^\)=τb​\(y,y^\)=ϕ​\(y,y^\)=MCC​\(y,y^\)\.\\begin\{split\}\\rho\_\{\\text\{Pearson\}\}\(y,\\hat\{y\}\)&=\\rho\_\{\\text\{Spearman\}\}\(y,\\hat\{y\}\)=\\tau\_\{b\}\(y,\\hat\{y\}\)\\\\ &=\\phi\(y,\\hat\{y\}\)=\\text\{MCC\}\(y,\\hat\{y\}\)\.\\end\{split\}\(6\)

The result is classical and follows from standard properties of the underlying statistics, but its reporting consequences \(Section[8](https://arxiv.org/html/2606.00093#S8)\) are easy to miss in LLM\-as\-judge work\. The Pearson\-MCC\-phi identity is immediate from Eq\. \([5](https://arxiv.org/html/2606.00093#S3.E5)\)\. The two non\-obvious pieces are why Spearman and Kendall coincide with Pearson on binary data\.

#### Why Spearman equals Pearson on binary data?

Spearman’sρ\\rhois Pearson’srrcomputed on the rank\-transformed inputs\. With only two distinct values present, all zeros receive the common average ranka0=\(n0\+1\)/2a\_\{0\}=\(n\_\{0\}\+1\)/2and all ones receivea1=n0\+\(n1\+1\)/2a\_\{1\}=n\_\{0\}\+\(n\_\{1\}\+1\)/2, wheren0,n1n\_\{0\},n\_\{1\}are the counts of zeros and ones\. The mapx↦a0\+\(a1−a0\)​xx\\mapsto a\_\{0\}\+\(a\_\{1\}\-a\_\{0\}\)\\,xis affine, and Pearson’srris invariant under affine rescaling of either input — soρSpearman=ρPearson\\rho\_\{\\text\{Spearman\}\}=\\rho\_\{\\text\{Pearson\}\}\.

#### Why Kendall’sτb\\tau\_\{b\}equalsϕ\\phion binary data?

The tie\-corrected form\(Kendall,[1945](https://arxiv.org/html/2606.00093#bib.bib20)\)is

τb=C−D\(ntot−ny\)​\(ntot−ny^\),\\tau\_\{b\}=\\frac\{C\-D\}\{\\sqrt\{\(n\_\{\\text\{tot\}\}\-n\_\{y\}\)\(n\_\{\\text\{tot\}\}\-n\_\{\\hat\{y\}\}\)\}\},\(7\)wherentot=\(N2\)n\_\{\\text\{tot\}\}=\\binom\{N\}\{2\}andny,ny^n\_\{y\},n\_\{\\hat\{y\}\}are the numbers of pairs tied onyyand ony^\\hat\{y\}respectively\. Reading pair types off the cells of the2×22\\times 2table of Section[2](https://arxiv.org/html/2606.00093#S2), a concordant pair joins a\(1,1\)\(1,1\)cell with a\(0,0\)\(0,0\)cell, soC=T​P⋅T​NC=TP\\cdot TN, and a discordant pair joins a\(1,0\)\(1,0\)cell with a\(0,1\)\(0,1\)cell, soD=F​N⋅F​PD=FN\\cdot FP\. The pairs untied onyynumberntot−ny=\(T​P\+F​N\)​\(F​P\+T​N\)n\_\{\\text\{tot\}\}\-n\_\{y\}=\(TP\+FN\)\(FP\+TN\), and those untied ony^\\hat\{y\}numberntot−ny^=\(T​P\+F​P\)​\(F​N\+T​N\)n\_\{\\text\{tot\}\}\-n\_\{\\hat\{y\}\}=\(TP\+FP\)\(FN\+TN\)\. Substituting,

τb\\displaystyle\\tau\_\{b\}=T​P⋅T​N−F​P⋅F​N\(T​P\+F​N\)​\(F​P\+T​N\)\\displaystyle=\\frac\{TP\\cdot TN\-FP\\cdot FN\}\{\\sqrt\{\(TP\+FN\)\(FP\+TN\)\}\}\(8\)×1\(T​P\+F​P\)​\(F​N\+T​N\)=ϕ,\\displaystyle\\quad\\times\\frac\{1\}\{\\sqrt\{\(TP\+FP\)\(FN\+TN\)\}\}=\\phi,matching Eq\. \([5](https://arxiv.org/html/2606.00093#S3.E5)\)\.

## 5Cohen’sκ\\kappaand Marginal Sensitivity

Cohen’sκ\\kappais close to the common value of Section[4](https://arxiv.org/html/2606.00093#S4)but not equal to it\. The gap is fully controlled by how well the judge’s positive\-prediction rate matches the human’s\.

Comparing Eq\. \([4](https://arxiv.org/html/2606.00093#S3.E4)\) and Eq\. \([5](https://arxiv.org/html/2606.00093#S3.E5)\), the numeratorT​P⋅T​N−F​P⋅F​NTP\\cdot TN\-FP\\cdot FNis shared \(up to a factor of22\)\. The denominators differ:

Dκ\\displaystyle D\_\{\\kappa\}=\(T​P\+F​N\)​\(F​N\+T​N\)\\displaystyle=\(TP\+FN\)\(FN\+TN\)\(9\)\+\(T​P\+F​P\)​\(F​P\+T​N\),\\displaystyle\\quad\+\(TP\+FP\)\(FP\+TN\),Dϕ\\displaystyle D\_\{\\phi\}=2​\(T​P\+F​N\)​\(F​N\+T​N\)\\displaystyle=2\\sqrt\{\(TP\+FN\)\(FN\+TN\)\}\(10\)×\(T​P\+F​P\)​\(F​P\+T​N\)\.\\displaystyle\\quad\\times\\sqrt\{\(TP\+FP\)\(FP\+TN\)\}\.WritingA=\(T​P\+F​N\)​\(F​N\+T​N\)A=\(TP\+FN\)\(FN\+TN\)andB=\(T​P\+F​P\)​\(F​P\+T​N\)B=\(TP\+FP\)\(FP\+TN\), we haveDκ=A\+BD\_\{\\kappa\}=A\+BandDϕ=2​A​BD\_\{\\phi\}=2\\sqrt\{AB\}\. The AM\-GM inequalityA\+B≥2​A​BA\+B\\geq 2\\sqrt\{AB\}, with equality iffA=BA=B, gives

κϕ=2​A​BA\+B≤1,\\frac\{\\kappa\}\{\\phi\}=\\frac\{2\\sqrt\{AB\}\}\{A\+B\}\\leq 1,\(11\)and

\|κ\|≤\|ϕ\|,\|\\kappa\|\\leq\|\\phi\|,\(12\)with equality iffA=BA=B, which is equivalent toπ=π^\\pi=\\hat\{\\pi\}\.Cohen’sκ\\kappathus penalizes the judge for marginal\-distribution mismatch with the ground truth in a way thatϕ\\phidoes not\.A judge that perfectly orders items but commits to a different positive rate than the human annotator receives a strictly lowerκ\\kappathanϕ\\phi\.

A second reformulation rewritesκ\\kappaas a rescaled accuracy:κ=\(Acc−pe\)/\(1−pe\)\\kappa=\(\\text\{Acc\}\-p\_\{e\}\)/\(1\-p\_\{e\}\)\. In the balanced caseπ=π^=0\.5\\pi=\\hat\{\\pi\}=0\.5,pe=0\.5p\_\{e\}=0\.5andκ=2​Acc−1\\kappa=2\\,\\text\{Acc\}\-1\. In the extreme\-prevalence caseπ→1\\pi\\to 1\(orπ→0\\pi\\to 0\),pe→1p\_\{e\}\\to 1and small accuracy gaps driveκ\\kappato zero or negative values — the*kappa paradox*discussed byFeinstein and Cicchetti \([1990](https://arxiv.org/html/2606.00093#bib.bib10)\)\.

Worked exampleWithN=100N=100and\(T​P,F​N,F​P,T​N\)=\(40,10,20,30\)\(TP,FN,FP,TN\)=\(40,10,20,30\),π=0\.50\\pi=0\.50andπ^=0\.60\\hat\{\\pi\}=0\.60\. Direct computation givesAcc=0\.700\\text\{Acc\}=0\.700,F1=0\.727F\_\{1\}=0\.727,pe=0\.500p\_\{e\}=0\.500,κ=0\.400\\kappa=0\.400, andϕ=1000/6×106≈0\.408\\phi=1000/\\sqrt\{6\\times 10^\{6\}\}\\approx 0\.408\. The\|κ\|<\|ϕ\|\|\\kappa\|<\|\\phi\|gap is small because the marginal mismatch is small\. In a low\-prevalence regime\(T​P,F​N,F​P,T​N\)=\(5,5,0,90\)\(TP,FN,FP,TN\)=\(5,5,0,90\)\(π=0\.10\\pi=0\.10\),Acc=0\.95\\text\{Acc\}=0\.95butκ≈0\.643\\kappa\\approx 0\.643andϕ≈0\.688\\phi\\approx 0\.688; a degenerate always\-negative predictor on the same true distribution would achieveAcc=0\.90\\text\{Acc\}=0\.90withκ=0\\kappa=0and undefinedϕ\\phi— illustrating the danger of reporting accuracy alone on imbalanced criteria\.

#### Sampling\-distribution behavior\.

The pointwise collapse extends to the first\-order sampling behavior ofκ^\\hat\{\\kappa\}andϕ^\\hat\{\\phi\}under multinomial sampling of the2×22\\times 2cell counts\. Whenπ=π^\\pi=\\hat\{\\pi\}— the matched\-marginals condition that makesκ=ϕ\\kappa=\\phipointwise — the asymptotic variances coincide to leading order:

Var​\(κ^\)=Var​\(ϕ^\)\+O​\(N−2\)\.\\text\{Var\}\(\\hat\{\\kappa\}\)=\\text\{Var\}\(\\hat\{\\phi\}\)\+O\(N^\{\-2\}\)\.\(13\)The mechanism is a gradient\-coincidence argument on the simplex, not a cell\-by\-cell partial\-derivative identity, and the variance gap off matched marginals is second\-order in the mismatch; Appendix[C](https://arxiv.org/html/2606.00093#A3)gives the argument and a numerical check\.

For finite\-sample uncertainty bands, bootstrap resampling is preferable to either asymptotic formula, with one caveat\. In rubric evaluation the unit of analysis is rarely an isolated criterion decision: criteria are clustered within items \(multiple criteria per submission\), within prompts or rubrics \(multiple items sharing a rubric\), and within judges \(in an ensemble, multiple decisions by the same model\)\. A flat decision\-level resample understates uncertainty when within\-cluster correlations are positive, as is the norm when a single judge mistake propagates across the criteria of one item\. An item\-level or hierarchical cluster bootstrap \(resample items, then optionally resample criteria within items\) is preferable; standard cluster\-bootstrap implementations are available in the survey\-statistics literature\(e\.g\., Field and Welsh,[2007](https://arxiv.org/html/2606.00093#bib.bib11)\)\.

### 5\.1Micro versus Macro Aggregation across Rubric Criteria

Rubric evaluation is not a flat sequence of independent decisions; the same hierarchy that underlies the cluster\-bootstrap caveat forces a second choice, about how to aggregate the per\-decision metric into a single number\. Three levels are standard\.*Micro\-averaging*pools all item×\\timescriterion decisions into one long sequence and computes the metric — accuracy,κ\\kappa,F1F\_\{1\}— on the pooled2×22\\times 2table\.*Macro\-averaging*computes the metric separately for each criterion and averages the per\-criterion numbers, unweighted\.*Item\-level aggregation*first reduces each item’s vector of criterion verdicts to a single per\-item verdict \(a weighted\-sum score thresholded into pass/fail, or an all\-criteria\-met conjunction, depending on the rubric\) and computes agreement on the resulting per\-item labels\.

These numbers estimate different quantities, and a scalar agreement figure reported without naming its aggregation level is ambiguous\. Consider a two\-criterion rubric in which criterionAAfires on80%80\\%of items and is easy \(κ=0\.9\\kappa=0\.9\) and criterionBBfires on5%5\\%of items and is hard \(κ=0\.2\\kappa=0\.2\)\. Micro\-κ\\kappais dominated byAA’s decisions, landing near0\.850\.85; macro\-κ\\kappaaverages to0\.550\.55; item\-levelκ\\kappadepends on the rubric’s weights and threshold, and may track either criterion or neither\. This choice interacts with the resampling unit above: a criterion\-level cluster bootstrap matches macro\-averaging, an item\-level cluster bootstrap matches item\-level aggregation, and a flat decision\-level bootstrap matches only micro\-averaging, and then only under the \(usually false\) independence assumption\. Our recommendation is to report all three when the rubric has more than a handful of criteria, and name the chosen level whenever a single number is reported\.

## 6The Abstention Complication

The preceding analysis assumed every prediction and every ground\-truth label was strictly binary\. LLM judges, however, are commonly prompted to emit a third verdict when the submission does not contain sufficient information to assess the criterion\. We refer to this verdict asCANNOT\_ASSESS\(CA\)\.333Human annotators produceCANNOT\_ASSESSlabels as well, though the underlying causes differ: a judge typically abstains for lack of evidence in the submission, an annotator when the criterion itself is ambiguous for the item\.Agreement metrics are computed only after these verdicts are resolved\.

### 6\.1Three Modes, Three Estimands

Let𝒟=\{\(yi,y^i\)\}i=1N\\mathcal\{D\}=\\\{\(y\_\{i\},\\hat\{y\}\_\{i\}\)\\\}\_\{i=1\}^\{N\}withyi,y^i∈\{MET,UNMET,CA\}y\_\{i\},\\hat\{y\}\_\{i\}\\in\\\{\\texttt\{\{\\color\[rgb\]\{0\.45,0\.45,0\.45\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.45,0\.45,0\.45\}\\pgfsys@color@gray@stroke\{0\.45\}\\pgfsys@color@gray@fill\{0\.45\}MET\}\},\\texttt\{\{\\color\[rgb\]\{0\.45,0\.45,0\.45\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.45,0\.45,0\.45\}\\pgfsys@color@gray@stroke\{0\.45\}\\pgfsys@color@gray@fill\{0\.45\}UNMET\}\},\\texttt\{\{\\color\[rgb\]\{0\.45,0\.45,0\.45\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.45,0\.45,0\.45\}\\pgfsys@color@gray@stroke\{0\.45\}\\pgfsys@color@gray@fill\{0\.45\}CA\}\}\\\}\. Three resolutions are in common use —*exclude*the abstentions,*recode*them as negative, or keep them as a*third class*\. The three paragraphs below treat each in turn; the point is that they are not interchangeable preprocessing steps but estimate different population quantities\.

#### Exclude:

To make the abstention mechanism explicit, introduce latent binary verdictsyi∗,y^i∗∈\{0,1\}y\_\{i\}^\{\*\},\\hat\{y\}\_\{i\}^\{\*\}\\in\\\{0,1\\\}— the decisions the human and judge would have committed to if neither were permitted to abstain — together with abstention indicatorsmi,m^i∈\{0,1\}m\_\{i\},\\hat\{m\}\_\{i\}\\in\\\{0,1\\\}, wheremi=1m\_\{i\}=1iff the human abstained \(and analogously form^i\\hat\{m\}\_\{i\}\)\. The observed verdicts areyi=yi∗y\_\{i\}=y\_\{i\}^\{\*\}whenmi=0m\_\{i\}=0andyi=CAy\_\{i\}=\\texttt\{\{\\color\[rgb\]\{0\.45,0\.45,0\.45\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.45,0\.45,0\.45\}\\pgfsys@color@gray@stroke\{0\.45\}\\pgfsys@color@gray@fill\{0\.45\}CA\}\}whenmi=1m\_\{i\}=1, with the symmetric definition fory^i\\hat\{y\}\_\{i\}\. The exclude mode discards every pair withmi=1m\_\{i\}=1orm^i=1\\hat\{m\}\_\{i\}=1and computes the binary metric on the residue; writingTTfor any of accuracy,F1F\_\{1\},κ\\kappa,ϕ\\phiviewed as a functional of the joint latent distribution, the estimand is

T​\(ℙ\(y∗,y^∗\)∣m=0∧m^=0\)\.T\\bigl\(\\mathbb\{P\}\_\{\(y^\{\*\},\\hat\{y\}^\{\*\}\)\\,\\mid\\,m=0\\,\\wedge\\,\\hat\{m\}=0\}\\bigr\)\.\(14\)This equals the unconditional latent functionalT​\(ℙ\(y∗,y^∗\)\)T\(\\mathbb\{P\}\_\{\(y^\{\*\},\\hat\{y\}^\{\*\}\)\}\)only under assumptions about the joint abstention mechanism — typically missing\-at\-random conditional on observed item features, or the stronger missing\-completely\-at\-random assumption\(Little and Rubin,[2019](https://arxiv.org/html/2606.00093#bib.bib28)\)\. The exclude estimand is the*selective\-accuracy*measure of the binary judge, restricted to the\(1−Pr⁡\[m=1\]\)​\(1−Pr⁡\[m^=1\]\)\(1\-\\Pr\[m=1\]\)\(1\-\\Pr\[\\hat\{m\}=1\]\)fraction of the population on which both sides chose to commit; it characterizes performance on the covered subset, not on the full deployment distribution\.

#### Recode\-as\-negative:

MapCANNOT\_ASSESStoUNMETin both vectors, then compute the binary metric\. Under a plain misclassification loss theCAandUNMETresponses become indistinguishable, but the calibration information theCAverdict was introduced to carry is destroyed at the measurement stage\. The sign of the resulting bias depends on whether the criterion is positively or negatively weighted\.

#### Three\-class:

RetainCANNOT\_ASSESSas a distinct category and report a3×33\\times 3agreement metric such as multi\-class accuracy, multi\-class Cohen’sκ\\kappa\(Conger,[1980](https://arxiv.org/html/2606.00093#bib.bib6)\), or a weighted variant with an explicit distance matrix encoding the geometry of disagreement\.

Appendix[B](https://arxiv.org/html/2606.00093#A2)works a single3×33\\times 3table through all three modes, showing that they yield different accuracy,κ\\kappa, andF1F\_\{1\}values from one set of judgments\.

### 6\.2The Equivalence Breaks atk=3k=3

The affine\-rank argument behind Observation 1 required the input vectors to take exactly two distinct values\. With three values\{0,12,1\}\\\{0,\\tfrac\{1\}\{2\},1\\\}\(the ordinal encoding withCAbetweenMETandUNMET\), the rank map sends0↦\(n0\+1\)/20\\mapsto\(n\_\{0\}\+1\)/2,12↦n0\+\(n1/2\+1\)/2\\tfrac\{1\}\{2\}\\mapsto n\_\{0\}\+\(n\_\{1/2\}\+1\)/2, and1↦n0\+n1/2\+\(n1\+1\)/21\\mapsto n\_\{0\}\+n\_\{1/2\}\+\(n\_\{1\}\+1\)/2\. Direct calculation shows this map is affine in the input value iffn0=n1n\_\{0\}=n\_\{1\}— iffMETandUNMETprevalences coincide\. Generically,ρSpearman≠ρPearson\\rho\_\{\\text\{Spearman\}\}\\neq\\rho\_\{\\text\{Pearson\}\}on three\-class data\. The phi coefficient has no canonical3×33\\times 3analog; the multi\-class MCC extension ofGorodkin \([2004](https://arxiv.org/html/2606.00093#bib.bib15)\)does not reduce to either Pearson on a chosen encoding or to Kendall’sτb\\tau\_\{b\}\. The choice among multi\-class agreement statistics is therefore a substantive modeling decision\.444Observation[4](https://arxiv.org/html/2606.00093#S4)also fails on binary data with degenerate marginals; thek=3k=3break is qualitatively different in that it cannot be repaired by any choice of encoding\.

### 6\.3Selective Prediction and the Abstention Rate as a Metric

Under recode\-as\-negative, a judge that abstains on uncertain negatives and one that commits toUNMETon the same items receive identical scores\. Whether this*incentivizes*one behavior over the other depends on the judge’s training objective, outside the scope of the metric itself — but under an objective that explicitly rewards calibrated abstention \(as in selective prediction\), the recoding erases the very signal the objective conditions on\. The selective\-prediction literature\(Chow,[1970](https://arxiv.org/html/2606.00093#bib.bib4); El\-Yaniv and Wiener,[2010](https://arxiv.org/html/2606.00093#bib.bib8)\)treats this by reporting a two\-axis summary —*coverage*\(the fraction of non\-abstention predictions\) and*selective accuracy*\(the accuracy on the covered subset\) — rather than a single scalar; recent NLP work develops the same view for question answering\(Kamath et al\.,[2020](https://arxiv.org/html/2606.00093#bib.bib19)\), classification\(Xin et al\.,[2021](https://arxiv.org/html/2606.00093#bib.bib40)\), and calibrated LLM uncertainty\(Kadavath et al\.,[2022](https://arxiv.org/html/2606.00093#bib.bib18)\)\.

## 7Multi\-Judge Ensembles

Ensembles of judges reduce the prompt\-sensitivity and model\-specific biases of a single judge\(Wang et al\.,[2023a](https://arxiv.org/html/2606.00093#bib.bib36); Verga et al\.,[2024](https://arxiv.org/html/2606.00093#bib.bib35); Bavaresco et al\.,[2025](https://arxiv.org/html/2606.00093#bib.bib1)\), and they add a measurement object single\-judge setups lack:*inter\-judge agreement*, which flags items the rubric does not unambiguously resolve, independently of agreement with ground truth\. WithR≥2R\\geq 2judges grading each criterion\(Rao and Callison\-Burch,[2026](https://arxiv.org/html/2606.00093#bib.bib32)\), the standard inter\-judge statistics are Fleiss’κ\\kappa\(Fleiss,[1971](https://arxiv.org/html/2606.00093#bib.bib12)\)and Krippendorff’sα\\alpha\(Krippendorff,[2004](https://arxiv.org/html/2606.00093#bib.bib23)\)\.

### 7\.1Binary Ensembles

Lety^i​j∈\{0,1\}\\hat\{y\}\_\{ij\}\\in\\\{0,1\\\}denote judgejj’s verdict on itemii, and letri=∑j=1Ry^i​jr\_\{i\}=\\sum\_\{j=1\}^\{R\}\\hat\{y\}\_\{ij\}\. With global ratesp1=\(N​R\)−1​∑irip\_\{1\}=\(NR\)^\{\-1\}\\sum\_\{i\}r\_\{i\}andp0=1−p1p\_\{0\}=1\-p\_\{1\}, Fleiss’κ\\kappahas observed pairwise agreement

Po=1N​R​\(R−1\)∑i=1N\[ri​\(ri−1\)\+\(R−ri\)\(R−ri−1\)\]\\begin\{split\}P\_\{o\}=\\frac\{1\}\{NR\(R\-1\)\}\\sum\_\{i=1\}^\{N\}\\bigl\[&r\_\{i\}\(r\_\{i\}\-1\)\\\\ &\+\(R\-r\_\{i\}\)\(R\-r\_\{i\}\-1\)\\bigr\]\\end\{split\}\(15\)and expected agreementPe=p02\+p12P\_\{e\}=p\_\{0\}^\{2\}\+p\_\{1\}^\{2\}, givingκF=\(Po−Pe\)/\(1−Pe\)\\kappa\_\{F\}=\(P\_\{o\}\-P\_\{e\}\)/\(1\-P\_\{e\}\)\. Krippendorff’sα\\alphaon the same data uses observed disagreementDo=1−PoD\_\{o\}=1\-P\_\{o\}and expected disagreementDe=\(1−Pe\)⋅N​R/\(N​R−1\)D\_\{e\}=\(1\-P\_\{e\}\)\\cdot NR/\(NR\-1\), incorporating a finite\-sample correction\.

These statistics give the binary multi\-judge analog of Observation[4](https://arxiv.org/html/2606.00093#S4)\.555SubstitutingDo=1−PoD\_\{o\}=1\-P\_\{o\}andDe=\(1−Pe\)​N​R/\(N​R−1\)D\_\{e\}=\(1\-P\_\{e\}\)\\,NR/\(NR\-1\)intoα=1−Do/De\\alpha=1\-D\_\{o\}/D\_\{e\}yieldsα=1−\(N​R−1\)​\(1−κF\)/N​R=κF\+\(1−κF\)/\(N​R\)\\alpha=1\-\(NR\-1\)\(1\-\\kappa\_\{F\}\)/NR=\\kappa\_\{F\}\+\(1\-\\kappa\_\{F\}\)/\(NR\)\. The pairwise correlation equalities follow from Observation[4](https://arxiv.org/html/2606.00093#S4)applied to each judge pair \(equality on each pair propagates through any symmetric aggregation\), and the equality withκF\\kappa\_\{F\}under matched marginals is the binary specialization of the kappa\-decomposition ofConger \([1980](https://arxiv.org/html/2606.00093#bib.bib6)\)\.

Observation 2On binary data withRRjudges andNNitems, the inter\-rater agreement statistics satisfy the exact identityα=κF\+1−κFN​R,\\alpha=\\kappa\_\{F\}\+\\frac\{1\-\\kappa\_\{F\}\}\{NR\},\(16\)soα≥κF\\alpha\\geq\\kappa\_\{F\}with equality iffκF=1\\kappa\_\{F\}=1, andα→κF\\alpha\\to\\kappa\_\{F\}asN→∞N\\to\\infty\. The average pairwise Pearson, Spearman, Kendall’sτb\\tau\_\{b\},ϕ\\phi, and MCC over the\(R2\)\\binom\{R\}\{2\}judge pairs are all equal, and coincide withκF\\kappa\_\{F\}when all judges share the same marginalMETrate\.

For typical values \(κF=0\.6\\kappa\_\{F\}=0\.6,N=100N=100,R=3R=3\),α−κF≈1\.3×10−3\\alpha\-\\kappa\_\{F\}\\approx 1\.3\\times 10^\{\-3\}, below the precision at which agreement coefficients are reported\. A binary ensemble\-validation report should pick one of\{κF,α,average pairwise​ϕ\}\\\{\\kappa\_\{F\},\\alpha,\\text\{average pairwise \}\\phi\\\}; the others add no information beyond a finite\-sample correction\.

### 7\.2Three Classes: Admitting Abstention

AdmittingCANNOT\_ASSESSbreaks the equivalences of Observation[7\.1](https://arxiv.org/html/2606.00093#S7.SS1)in three ways, each by the mechanism of Section[6\.2](https://arxiv.org/html/2606.00093#S6.SS2)\.

Average pairwise Pearson requires a numeric encoding of\{\\\{MET,UNMET,CA\}\\\}; the result depends on the encoding \(e\.g\.,\{1,0,1/2\}\\\{1,0,1/2\\\}versus deletingCApairs\), and no encoding is canonical\.

Fleiss’κ\\kappaon three nominal categories penalizesMET–UNMETandMET–CAconfusions equally, while Krippendorff’sα\\alphawith a quadratic\-ordinalδ\\delta\-function penalizesMET–UNMETmore thanMET–CA, encoding the assumption thatCAis “between\.” On the same ensemble, the two statistics can rank judges differently\.

Average pairwise Spearman andτb\\tau\_\{b\}on three\-valued data no longer coincide; their divergence is governed by the per\-item distribution ofCAvotes across judges\.

Operationally, the multi\-judge three\-class case admits no single defensible agreement scalar without an explicit weight matrixW∈ℝ3×3W\\in\\mathbb\{R\}^\{3\\times 3\}specifying the cost of each disagreement\. Report per\-judge3×33\\times 3confusion matrices alongside any scalar summary, allowing readers to recompute weightedκ\\kappaunder their preferredWW\.

## 8A Checklist for Reporting Agreement Metrics

The audit \(Appendix[A](https://arxiv.org/html/2606.00093#A1)\) found that agreement statistics are frequently reported without the information needed to interpret them, and Sections[3](https://arxiv.org/html/2606.00093#S3)–[7](https://arxiv.org/html/2606.00093#S7)explain why that information is not optional: on binary data the same scalar can stand for several distinct statistics, and its value shifts with the judgment scale, the handling of ties and abstentions, and the aggregation level\. The eleven practices below make those choices explicit, so that a reported number is reproducible and comparable across studies\. Each follows directly from a result above and costs nothing beyond the confusion\-matrix counts a validation already produces; the first is a precondition for the rest\.

1. 1\.Name the judgment scale before naming the metric\.State which of the five scales of Table[1](https://arxiv.org/html/2606.00093#S2.T1)the judge produces: binary verdicts, pairwise preferences \(with or without ties\), ordinal scores, continuous/direct\-assessment scores, or nominal labels withCANNOT\_ASSESS/ tie / invalid outcomes\. Almost every other entry depends on this declaration, because the Section[4](https://arxiv.org/html/2606.00093#S4)equivalences are exact only after the variables are binary\.
2. 2\.On binary data, report at most one of Pearson, Spearman, Kendall’sτb\\tau\_\{b\},ϕ\\phi, and MCC\.By Observation[4](https://arxiv.org/html/2606.00093#S4)these five statistics are identical on non\-degenerate binary vectors; preferϕ\\phi/MCC when the goal is binary association, since the name itself signals the scale\. Reporting more than one creates a misleading appearance of triangulated evidence\.
3. 3\.Report accuracy together with a marginal\-sensitive agreement measure\.Accuracy alone is unreliable on imbalanced criteria, as the kappa\-paradox example of Section[5](https://arxiv.org/html/2606.00093#S5)shows; Cohen’sκ\\kappais the companion of choice because its gap fromϕ\\phi/MCC reflects the mismatch between the judge’s and the human’s positive\-label rates\.
4. 4\.Report the2×22\\times 2confusion matrix, or at minimum precision and recall\.As noted in Section[3](https://arxiv.org/html/2606.00093#S3), every symmetric statistic in this paper — accuracy,F1F\_\{1\},κ\\kappa,ϕ\\phi/MCC, and the rank correlations — is invariant underF​P↔F​NFP\\leftrightarrow FN, so none distinguishes a judge that misses positives from one that over\-predicts them\. For LLM\-as\-judge validation, judge strictness or leniency is often the most actionable diagnosis\.
5. 5\.State how degenerate criteria are handled\.Rubric criteria are often rare, and a criterion with constant human labels or constant judge predictions leaves Pearson,ϕ\\phi/MCC, Spearman, and Kendall undefined \(andκ\\kappauninformative\)\. Such criteria should be reported with their counts and marked NA, rather than silently dropped or coerced to zero\.
6. 6\.Name theCA/ tie / invalid handling mode in every reported number\.As Section[6](https://arxiv.org/html/2606.00093#S6)shows, the three modes estimate different quantities; a metric reported without its handling mode is ambiguous among them\.
7. 7\.Report abstention, tie, and invalid rates as metrics in their own right\.Following Section[6](https://arxiv.org/html/2606.00093#S6), aκ\\kappaof0\.70\.7on a40%40\\%\-abstention sample and aκ\\kappaof0\.70\.7on a5%5\\%\-abstention sample are not comparable, and abstention drift is often the first observable symptom of prompt instability or distribution shift\.
8. 8\.For the exclude mode, report covered\-subset performance and coverage together\.The exclusion estimand of Section[6](https://arxiv.org/html/2606.00093#S6)collapses to the population estimand only under a missing\-completely\-at\-random assumption on abstention; where possible, examine whether abstention correlates with item difficulty, criterion type, prompt, or judge model before treating the residue as representative\.
9. 9\.For three\-class or multi\-class reporting, specify the weight or cost matrix\.As the three\-class worked example in Appendix[B](https://arxiv.org/html/2606.00093#A2)shows, scalar summaries over a3×33\\times 3confusion matrix depend on whetherCAis treated as closer toMET, closer toUNMET, or equally different from both; without an explicit weight matrix the choice is hidden in the metric’s defaults\.
10. 10\.Specify the aggregation level\.Rubric evaluation is nested — criteria within items, items within prompts, possibly repeated judgments within models or ensembles — and the Section[5\.1](https://arxiv.org/html/2606.00093#S5.SS1)aggregation options \(micro, macro, item\-level\) answer different questions when criteria differ in prevalence or difficulty\. State the level explicitly; this is complementary to the cluster\-bootstrap caveat in Section[5](https://arxiv.org/html/2606.00093#S5)\.
11. 11\.For judge ensembles, report one primary inter\-judge agreement statistic and the per\-judge confusion matrices\.By Observation[7\.1](https://arxiv.org/html/2606.00093#S7.SS1),\{κF,α,average pairwise​ϕ\}\\\{\\kappa\_\{F\},\\alpha,\\text\{average pairwise \}\\phi\\\}coincide on binary data up to a finite\-sample correction of relative orderO​\(1/\(N​R\)\)O\(1/\(NR\)\), so pick one; the per\-judge confusion matrices preserve the FP/FN asymmetry of item 4 across the ensemble and become essential whenCANNOT\_ASSESSis admitted and any scalar summary depends on a non\-canonical weight matrix\.

## 9Conclusions

The variety of agreement statistics in LLM\-as\-judge validation is largely apparent\. Once the judgment scale is fixed to binary verdicts, the six numbers most often reported collapse to four: accuracy,F1F\_\{1\}, a single association valueϕ\\phi\(equal to Pearson, Spearman, Kendall’sτb\\tau\_\{b\}, and MCC\), and Cohen’sκ\\kappa\(Observation[4](https://arxiv.org/html/2606.00093#S4)\)\. Cohen’sκ\\kappaadds information not as a fifth coefficient but through its chance\-correction: theκ\\kappa–ϕ\\phigap measures how far the judge’s positive rate has drifted from the human’s, closing in value and, to leading order, in sampling variance exactly when the two rates coincide\. Reporting Pearson, Spearman, and Kendall side by side then manufactures the appearance of triangulation from one statistic computed three ways\.

What carries information beyond this common value is structural rather than numerical\. The2×22\\times 2confusion matrix separates a strict judge from a lenient one, a distinction every symmetric coefficient erases\. Abstention handling is not preprocessing but a choice of estimand — excluding, recoding, and retainingCANNOT\_ASSESSeach estimate a different quantity, and the last breaks the binary equivalences\. Aggregation level and the ensemble statistic are choices of the same kind: each changes what a number means, not how precisely it is estimated\.

An agreement number is therefore interpretable only alongside the declarations that fix its meaning: the judgment scale, the abstention and tie handling mode, the coverage, the confusion matrix, and the aggregation level\. None costs more than the counts a validation already computes, and together they let readers reconstruct the equivalent statistics and compare across studies\. The checklist of Section[8](https://arxiv.org/html/2606.00093#S8)collects these declarations; their common lesson is that the first reporting decision is not which coefficient to compute but what was being measured\.

## Limitations

A few boundaries are worth naming\. The analysis treats the observed confusion\-matrix counts as fixed; the identities and inequalities hold exactly on any dataset on which they are computed\. Whether these counts additionally admit a sampling\-distribution interpretation depends on the inferential target: when the evaluation set is itself the object of measurement and the judge is run at temperature zero the counts are deterministic, while generalizing to a wider item population or accounting for judge stochasticity at non\-zero temperature places them in the asymptotic regime of Section[5](https://arxiv.org/html/2606.00093#S5)\(or, for finite samples, the cluster bootstrap noted there\)\. The Section[7](https://arxiv.org/html/2606.00093#S7)multi\-judge treatment covers the standard pairwise\-aggregable agreement statistics on the unweighted scale; we do not characterize the finite\-sample behavior of weighted\-κ\\kappaensembles under arbitrary weight matrices, nor the behavior of judge ensembles whose members are not exchangeable \(for example, one anchored on a frontier model and the rest on small open\-weight models\)\. The checklist assumes human ground truth is itself reliable, which is tenuous for the subjective, open\-ended tasks where LLM\-as\-judge methods are most heavily used; analyses of human\-judge disagreement\(Plank,[2022](https://arxiv.org/html/2606.00093#bib.bib31)\)suggest some criterion\-level disagreement is signal rather than noise\.

## Generative AI Use Disclosure

The authors acknowledge the use of LLMs in the preparation of this manuscript\. Specifically, the authors used Gemini\-Pro 3\.1 and Claude Sonnet 4\.6 to organize the references, proofread the drafts, make plots, and provide light rewrites\. The corresponding author has taken care to review the citations and rewrites to ensure they are hallucination free and relevant\.

## Ethical Considerations

This work analyzes the relationships among agreement metrics and examines how they are reported in literature; it introduces no new data, models, or human\-subjects experiments\. We see no major ethical considerations arising from the work itself\. We note only the indirect point already made in the text: agreement statistics are sometimes reported in ways that overstate how well an automated judge matches human judgment, and the reporting practices recommended here are intended to make such comparisons more transparent\.

## References

- Bavaresco et al\. \(2025\)Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Surikuchi, Ece Takmaz, and Alberto Testoni\. 2025\.[LLMs instead of human judges? a large scale empirical study across 20 NLP evaluation tasks](https://doi.org/10.18653/v1/2025.acl-short.20)\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\)*, pages 238–255, Vienna, Austria\. Association for Computational Linguistics\.
- Chan et al\. \(2023\)Chi\-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu\. 2023\.[ChatEval: Towards better LLM\-based evaluators through multi\-agent debate](https://arxiv.org/abs/2308.07201)\.*arXiv preprint*\.
- Chia et al\. \(2023\)Yew Ken Chia, Pengfei Hong, Lidong Bing, and Soujanya Poria\. 2023\.[INSTRUCTEVAL: Towards holistic evaluation of instruction\-tuned large language models](https://doi.org/10.48550/arXiv.2306.04757)\.*arXiv preprint*\.ArXiv:2306\.04757\.
- Chow \(1970\)C\. K\. Chow\. 1970\.[On optimum recognition error and reject tradeoff](https://doi.org/10.1109/TIT.1970.1054406)\.*IEEE Transactions on Information Theory*, 16\(1\):41–46\.
- Cohen \(1960\)Jacob Cohen\. 1960\.[A coefficient of agreement for nominal scales](https://doi.org/10.1177/001316446002000104)\.*Educational and Psychological Measurement*, 20\(1\):37–46\.
- Conger \(1980\)Anthony J\. Conger\. 1980\.[Integration and generalization of kappas for multiple raters](https://doi.org/10.1037/0033-2909.88.2.322)\.*Psychological Bulletin*, 88\(2\):322–328\.
- Dubois et al\. \(2024\)Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B\. Hashimoto\. 2024\.[Length\-controlled AlpacaEval: A simple way to debias automatic evaluators](https://arxiv.org/abs/2404.04475)\.*arXiv preprint*\.
- El\-Yaniv and Wiener \(2010\)Ran El\-Yaniv and Yair Wiener\. 2010\.[On the foundations of noise\-free selective classification](https://jmlr.org/papers/v11/el-yaniv10a.html)\.*Journal of Machine Learning Research*, 11:1605–1641\.
- Es et al\. \(2023\)Shahul Es, Jithin James, Luis Espinosa\-Anke, and Steven Schockaert\. 2023\.[RAGAS: Automated evaluation of retrieval augmented generation](https://doi.org/10.48550/arXiv.2309.15217)\.*arXiv preprint*\.ArXiv:2309\.15217\.
- Feinstein and Cicchetti \(1990\)Alvan R\. Feinstein and Domenic V\. Cicchetti\. 1990\.[High agreement but low kappa: I\. the problems of two paradoxes](https://doi.org/10.1016/0895-4356(90)90158-L)\.*Journal of Clinical Epidemiology*, 43\(6\):543–549\.
- Field and Welsh \(2007\)C\. A\. Field and A\. H\. Welsh\. 2007\.[Bootstrapping clustered data](https://doi.org/10.1111/j.1467-9868.2007.00593.x)\.*Journal of the Royal Statistical Society Series B: Statistical Methodology*, 69\(3\):369–390\.
- Fleiss \(1971\)Joseph L\. Fleiss\. 1971\.[Measuring nominal scale agreement among many raters](https://doi.org/10.1037/h0031619)\.*Psychological Bulletin*, 76\(5\):378–382\.
- Fleiss et al\. \(1969\)Joseph L\. Fleiss, Jacob Cohen, and B\. S\. Everitt\. 1969\.[Large sample standard errors of kappa and weighted kappa](https://doi.org/10.1037/h0028106)\.*Psychological Bulletin*, 72\(5\):323–327\.
- Gera et al\. \(2024\)Ariel Gera, Odellia Boni, Yotam Perlitz, Roy Bar\-Haim, Lilach Eden, and Asaf Yehudai\. 2024\.[JuStRank: Benchmarking LLM judges for system ranking](https://doi.org/10.48550/arXiv.2412.09569)\.*arXiv preprint*\.ArXiv:2412\.09569; to appear ACL 2025\.
- Gorodkin \(2004\)J\. Gorodkin\. 2004\.[Comparing two k\-category assignments by a k\-category correlation coefficient](https://doi.org/10.1016/j.compbiolchem.2004.09.006)\.*Computational Biology and Chemistry*, 28\(5\-6\):367–374\.
- Hashemi et al\. \(2024\)Helia Hashemi, Jason Eisner, Corby Rosset, Benjamin Van Durme, and Chris Kedzie\. 2024\.[Llm\-rubric: A multidimensional, calibrated approach to automated evaluation of natural language texts](https://doi.org/10.18653/v1/2024.acl-long.745)\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 13806–13834\. Association for Computational Linguistics\.
- Jiang et al\. \(2023\)Dongfu Jiang, Yishan Li, Ge Zhang, Wenhao Huang, Bill Yuchen Lin, and Wenhu Chen\. 2023\.[TIGERScore: Towards building explainable metric for all text generation tasks](https://doi.org/10.48550/arXiv.2310.00752)\.*arXiv preprint*\.ArXiv:2310\.00752; published in TMLR 2024\.
- Kadavath et al\. \(2022\)Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield\-Dodds, Nova DasSarma, Eli Tran\-Johnson, Scott Johnston, Sheer El\-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, and 17 others\. 2022\.[Language models \(Mostly\) know what they know](https://doi.org/10.48550/arXiv.2207.05221)\.*arXiv preprint*\.ArXiv:2207\.05221\.
- Kamath et al\. \(2020\)Amita Kamath, Robin Jia, and Percy Liang\. 2020\.[Selective question answering under domain shift](https://doi.org/10.18653/v1/2020.acl-main.503)\.In*Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 5684–5696, Online\. Association for Computational Linguistics\.
- Kendall \(1945\)M\. G\. Kendall\. 1945\.[The treatment of ties in ranking problems](https://doi.org/10.1093/biomet/33.3.239)\.*Biometrika*, 33\(3\):239–251\.
- Kim et al\. \(2023\)Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo\. 2023\.[Prometheus: Inducing fine\-grained evaluation capability in language models](https://doi.org/10.48550/arXiv.2310.08491)\.*arXiv preprint*\.ArXiv:2310\.08491\.
- Kim et al\. \(2024\)Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo\. 2024\.[Prometheus 2: An open source language model specialized in evaluating other language models](https://doi.org/10.18653/v1/2024.emnlp-main.248)\.In*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 4334–4353\.
- Krippendorff \(2004\)Klaus Krippendorff\. 2004\.*Content Analysis: An Introduction to Its Methodology*, 2nd edition\.SAGE Publications, Thousand Oaks, CA\.
- Lee et al\. \(2026\)Chungpa Lee, Thomas Zeng, Jongwon Jeong, Jy yong Sohn, and Kangwook Lee\. 2026\.[How to correctly report llm\-as\-a\-judge evaluations](https://arxiv.org/abs/2511.21140)\.*Preprint*, arXiv:2511\.21140\.
- Lee et al\. \(2025\)Yukyung Lee, JoongHoon Kim, Jaehee Kim, Hyowon Cho, Jaewook Kang, Pilsung Kang, and Najoung Kim\. 2025\.[Checkeval: A reliable llm\-as\-a\-judge framework for evaluating text generation using checklists](https://doi.org/10.18653/v1/2025.emnlp-main.796)\.In*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, pages 15782–15809\.
- Li et al\. \(2023\)Junlong Li, Shichao Sun, Weizhe Yuan, Run\-Ze Fan, Hai Zhao, and Pengfei Liu\. 2023\.[Generative judge for evaluating alignment](https://doi.org/10.48550/arXiv.2310.05470)\.*arXiv preprint*\.ArXiv:2310\.05470\.
- Lin and Chen \(2023\)Yen\-Ting Lin and Yun\-Nung Chen\. 2023\.[LLM\-Eval: Unified multi\-dimensional automatic evaluation for open\-domain conversations with large language models](https://doi.org/10.48550/arXiv.2305.13711)\.*arXiv preprint*\.ArXiv:2305\.13711\.
- Little and Rubin \(2019\)Roderick J\. A\. Little and Donald B\. Rubin\. 2019\.*Statistical Analysis with Missing Data*, 3rd edition\.John Wiley & Sons, Hoboken, NJ\.
- Liu et al\. \(2023\)Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu\. 2023\.[G\-Eval: NLG evaluation using GPT\-4 with better human alignment](https://arxiv.org/abs/2303.16634)\.*arXiv preprint*\.
- Matthews \(1975\)B\. W\. Matthews\. 1975\.[Comparison of the predicted and observed secondary structure of T4 phage lysozyme](https://doi.org/10.1016/0005-2795(75)90109-9)\.*Biochimica et Biophysica Acta \(BBA\) \- Protein Structure*, 405\(2\):442–451\.
- Plank \(2022\)Barbara Plank\. 2022\.[The "problem" of human label variation: On ground truth in data, modeling and evaluation](https://doi.org/10.18653/v1/2022.emnlp-main.731)\.In*Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing*, pages 10671–10682, Abu Dhabi, United Arab Emirates\. Association for Computational Linguistics\.
- Rao and Callison\-Burch \(2026\)Delip Rao and Chris Callison\-Burch\. 2026\.[Autorubric: Unifying rubric\-based llm evaluation](https://arxiv.org/abs/2603.00077)\.*Preprint*, arXiv:2603\.00077\.
- Saad\-Falcon et al\. \(2023\)Jon Saad\-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia\. 2023\.[ARES: An automated evaluation framework for retrieval\-augmented generation systems](https://doi.org/10.48550/arXiv.2311.09476)\.*arXiv preprint*\.ArXiv:2311\.09476\.
- Song et al\. \(2024\)Hwanjun Song, Hang Su, Igor Shalyminov, Jason Cai, and Saab Mansour\. 2024\.[FineSurE: Fine\-grained summarization evaluation using LLMs](https://doi.org/10.18653/v1/2024.acl-long.51)\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 906–922, Bangkok, Thailand\. Association for Computational Linguistics\.
- Verga et al\. \(2024\)Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis\. 2024\.[Replacing judges with juries: Evaluating LLM generations with a panel of diverse models](https://arxiv.org/abs/2404.18796)\.*arXiv preprint*\.
- Wang et al\. \(2023a\)Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui\. 2023a\.[Large language models are not fair evaluators](https://arxiv.org/abs/2305.17926)\.*arXiv preprint*\.
- Wang et al\. \(2023b\)Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, Wei Ye, Shikun Zhang, and Yue Zhang\. 2023b\.[PandaLM: An automatic evaluation benchmark for LLM instruction tuning optimization](https://doi.org/10.48550/arXiv.2306.05087)\.*arXiv preprint*\.ArXiv:2306\.05087\.
- Warrens \(2008\)Matthijs J\. Warrens\. 2008\.[On association coefficients for 2x2 tables and properties that do not depend on the marginal distributions](https://doi.org/10.1007/s11336-008-9070-3)\.*Psychometrika*, 73\(4\):777–789\.
- Wu et al\. \(2025\)Meng\-Chen Wu, Md Mosharaf Hossain, Tess Wood, Shayan Ali Akbar, Si\-Chi Chin, and Erwin Cornejo\. 2025\.[SEEval: Advancing LLM text evaluation efficiency and accuracy through self\-explanation prompting](https://doi.org/10.18653/v1/2025.findings-naacl.411)\.In*Findings of the Association for Computational Linguistics: NAACL 2025*, pages 7372–7383, Albuquerque, New Mexico\. Association for Computational Linguistics\.
- Xin et al\. \(2021\)Ji Xin, Raphael Tang, Yaoliang Yu, and Jimmy Lin\. 2021\.[The art of abstention: Selective prediction and error regularization for natural language processing](https://doi.org/10.18653/v1/2021.acl-long.84)\.In*Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\)*, pages 1040–1051, Online\. Association for Computational Linguistics\.
- Xu et al\. \(2023\)Wenda Xu, Danqing Wang, Liangming Pan, Zhenqiao Song, Markus Freitag, William Yang Wang, and Lei Li\. 2023\.[INSTRUCTSCORE: Explainable text generation evaluation with fine\-grained feedback](https://doi.org/10.48550/arXiv.2305.14282)\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*\. Association for Computational Linguistics\.ArXiv:2305\.14282\.
- Ye et al\. \(2024\)Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo\. 2024\.Flask: Fine\-grained language model evaluation based on alignment skill sets\.In*Proceedings of the Twelfth International Conference on Learning Representations*\.
- Zeng et al\. \(2023\)Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen\. 2023\.[Evaluating large language models at evaluating instruction following](https://doi.org/10.48550/arXiv.2310.07641)\.*arXiv preprint*\.ArXiv:2310\.07641\.
- Zheng et al\. \(2023\)Lianmin Zheng, Wei\-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P\. Xing, Hao Zhang, Joseph E\. Gonzalez, and Ion Stoica\. 2023\.[Judging LLM\-as\-a\-judge with MT\-Bench and Chatbot Arena](https://arxiv.org/abs/2306.05685)\.In*Advances in Neural Information Processing Systems \(Datasets and Benchmarks Track\)*, volume 36, pages 46595–46623\.
- Zhu et al\. \(2023\)Lianghui Zhu, Xinggang Wang, and Xinlong Wang\. 2023\.[JudgeLM: Fine\-tuned large language models are scalable judges](https://doi.org/10.48550/arXiv.2310.17631)\.*arXiv preprint*\.ArXiv:2310\.17631\.

## Appendix AAudited Systems and Reporting Practices

Table[A1](https://arxiv.org/html/2606.00093#A1.T1)gives the full audit summarized in Section[1](https://arxiv.org/html/2606.00093#S1): the 24 surveyed LLM\-as\-judge and LLM\-evaluator systems, coded by judged output scale, reported metrics, and primary relevance category\.

Table A1:Audit of 24 recent LLM\-as\-judge and LLM\-evaluator papers, coded by judged output scale, reported metrics, and relevance to the binary equivalence developed here\. The accent tag in the last column marks each system’s primary relevance category \(binary,pairwise,graded,multi\-class, orensemble\)\.
## Appendix BWorked Example: The Three Modes Compared

Consider the3×33\\times 3confusion table below, with row totals as true marginals and column totals as predicted marginals\.

Applying each abstention\-handling mode to these data:

The binaryF1\(MET\)F\_\{1\}^\{\(\\texttt\{\{\\color\[rgb\]\{0\.45,0\.45,0\.45\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.45,0\.45,0\.45\}\\pgfsys@color@gray@stroke\{0\.45\}\\pgfsys@color@gray@fill\{0\.45\}MET\}\}\)\}column applies only after a binary reduction; the three\-class row reports no binary reduction\. The three modes produce different answers from one set of judgments\. They estimate different quantities, corresponding to different questions about the judge\. The “n/a” entry in the three\-class row reflects that no binary reduction is applied; a multiclass setting still admits per\-classF1F\_\{1\}, macro\-F1F\_\{1\}, and micro\-F1F\_\{1\}aggregations, none of which is the same statistic as the binaryF1F\_\{1\}in the first two rows\. The one\-vs\-restMET\-classF1F\_\{1\}here is2⋅30/\(2⋅30\+15\+15\)=60/90≈0\.6672\\cdot 30/\(2\\cdot 30\+15\+15\)=60/90\\approx 0\.667, which numerically matches the recode\-as\-UNMETrow by coincidence on this table — the two are different statistics whose values happen to agree here\.

## Appendix CAsymptotic Variance ofκ^\\hat\{\\kappa\}andϕ^\\hat\{\\phi\}

This appendix supports the matched\-marginals variance identity of Eq\. \([13](https://arxiv.org/html/2606.00093#S5.E13)\)\. On the matched\-marginals slice\{p10=p01\}\\\{p\_\{10\}=p\_\{01\}\\\}, both metrics collapse to the same function of cell probabilities,κ=ϕ=1−p10/\(p1⁣⋅​p0⁣⋅\)\\kappa=\\phi=1\-p\_\{10\}/\(p\_\{1\\cdot\}p\_\{0\\cdot\}\), so∇κ\\nabla\\kappaand∇ϕ\\nabla\\phiagree along the two tangent directions of that slice\. The slice\-orthogonal direction\(0,\+1,−1,0\)\(0,\+1,\-1,0\)is thep10↔p01p\_\{10\}\\leftrightarrow p\_\{01\}swap, under which both metrics are symmetric and stationary, so the two four\-dimensional gradients can differ only along the simplex normal𝟏\\mathbf\{1\}\. Since the multinomial covarianceΣ=N−1​\(diag​\(𝐩\)−𝐩𝐩⊤\)\\Sigma=N^\{\-1\}\(\\mathrm\{diag\}\(\\mathbf\{p\}\)\-\\mathbf\{p\}\\mathbf\{p\}^\{\\top\}\)satisfiesΣ​𝟏=0\\Sigma\\mathbf\{1\}=0, the delta\-method quadratic forms∇κ⊤​Σ​∇κ\\nabla\\kappa^\{\\top\}\\Sigma\\nabla\\kappaand∇ϕ⊤​Σ​∇ϕ\\nabla\\phi^\{\\top\}\\Sigma\\nabla\\phiagree to leading order in1/N1/N\.

As a numerical check, at the matched point\(0\.3,0\.1,0\.1,0\.5\)\(0\.3,0\.1,0\.1,0\.5\)— whereπ=π^=0\.4\\pi=\\hat\{\\pi\}=0\.4— the cell probabilities have unequal row and column sums while the*marginals*π\\piandπ^\\hat\{\\pi\}remain equal; direct computation gives∇κ−∇ϕ=\(5/6\)​1\\nabla\\kappa\-\\nabla\\phi=\(5/6\)\\,\\mathbf\{1\}, confirming the all\-ones structure away from theπ=0\.5\\pi=0\.5balanced case\. At the balanced matched point\(p11,p10,p01,p00\)=\(0\.4,0\.1,0\.1,0\.4\)\(p\_\{11\},p\_\{10\},p\_\{01\},p\_\{00\}\)=\(0\.4,0\.1,0\.1,0\.4\)— which addsπ=0\.5\\pi=0\.5on top of matched marginals —κ=ϕ=0\.6\\kappa=\\phi=0\.6, and the multinomial delta method using the closed\-formκ^\\hat\{\\kappa\}variance ofFleiss et al\. \([1969](https://arxiv.org/html/2606.00093#bib.bib13)\)givesVar​\(κ^\)=Var​\(ϕ^\)=16/\(25​N\)=0\.640/N\\text\{Var\}\(\\hat\{\\kappa\}\)=\\text\{Var\}\(\\hat\{\\phi\}\)=16/\(25N\)=0\.640/N— not the0\.41/N0\.41/Nthat the bivariate\-normal approximation\(1−ϕ2\)2/N\(1\-\\phi^\{2\}\)^\{2\}/Nwould yield for continuous Pearson, which does not apply to Bernoulli cells\. Off matched marginals the two variances diverge by a second\-order term: at\(0\.4,0\.1,0\.2,0\.3\)\(0\.4,0\.1,0\.2,0\.3\)\(π=0\.5\\pi=0\.5,π^=0\.6\\hat\{\\pi\}=0\.6,κ=0\.400\\kappa=0\.400,ϕ=1/6≈0\.408\\phi=1/\\sqrt\{6\}\\approx 0\.408\), the delta method givesVar​\(κ^\)=504/\(625​N\)≈0\.806/N\\text\{Var\}\(\\hat\{\\kappa\}\)=504/\(625N\)\\approx 0\.806/NandVar​\(ϕ^\)=13/\(16​N\)=0\.813/N\\text\{Var\}\(\\hat\{\\phi\}\)=13/\(16N\)=0\.813/N\. The variance gap is second\-order in marginal mismatch, mirroring the pointwise gap\.