多标签标注一致性（Multi-Label Agreement）：一种灵活的标注者间可靠性评估框架

# Multi-Label Agreement: A Flexible Framework for Inter-Annotator Reliability
Source: [https://arxiv.org/html/2610.37292](https://arxiv.org/html/2610.37292)
\\jinfo

October 2026\\jvol\\jnum2026\\docheadPreprint

\\affilblock

Journal:PreprintTim Menzner1Email:[tim\.menzner@hs\-coburg\.de](mailto:[email protected])Affiliation:Center for Responsible Artificial Intelligence \(CRAI\), Coburg University of Applied SciencesJochen L\. Leidner2Email:[leidner@acm\.org](mailto:[email protected])Affiliation:Center for Responsible Artificial Intelligence \(CRAI\), Coburg University of Applied Sciences

###### Abstract

Reliable evaluation of inter\-annotator agreement \(IAA\) is important for assessing the consistency of annotation processes\. While established IAA metrics such as Cohen’s kappa, Fleiss’ kappa, and Krippendorff’s alpha are commonly applied in single\-label settings, multi\-label tasks require additional decisions about how full label sets, joint selections, and joint omissions should be compared\.

In this work, we present Multi\-Label Agreement \(MLA\), a flexible framework for evaluating agreement in multi\-label annotation tasks\. MLA preserves the full binary label vectors assigned to each item and uses average pairwise Hamming agreement, thereby accounting for both joint selections and joint omissions\. Its pooled\-marginal form is not a mathematically independent coefficient: it is a chance\-disagreement\-weighted average of label\-wise Fleiss/Scott\-type coefficients and is closely related to Krippendorff’s alpha with normalized Hamming distance\. The framework supports pooled\-marginal, Gwet AC1, and fixed\-baseline normalizations, yields item\-level, label\-level, and corpus\-level results, and includes a conditional minimum\-agreement diagnostic\. Through a worked example and Monte Carlo simulations, we show how the resulting values depend on label prevalence, the declared label inventory, and the selected model of expected agreement\. We also provide a Python implementation\. MLA thus offers a transparent and interpretable framework for multi\-label agreement analysis without claiming that one normalization is universally preferable\.

## 1Introduction

To assess the reproducibility of an annotation process, annotation studies commonly report*inter\-annotator agreement*, also known as inter\-coder or inter\-rater agreement[Carletta \(1996\)](https://arxiv.org/html/2610.37292#bib.bib5);[Artstein and Poesio \(2008\)](https://arxiv.org/html/2610.37292#bib.bib18);[Leidner \(2022\)](https://arxiv.org/html/2610.37292#bib.bib3)\. Agreement does not by itself prove that the annotations are valid or that they form an upper bound for a downstream classifier, but it can reveal unclear guidelines, inconsistent use of categories, and systematic differences between annotators\.

This is done by having a set of human subjects annotate a shared set of data items so their individual decisions can be compared\. After all annotators annotate the same piece of data and inter\-annotator agreement \(IAA\) is established, annotators may each work on their own data subset without overlap to save time and money\.

Factors that hurt IAA include unspecified, ambiguous, or differently interpreted annotation guidelines as well as human performance effects like lapses of concentration\. A good annotation guideline codebook describes each class clearly and provides examples and counterexamples to differentiate it from other classes\. Typically, writing the codebook requires several iterations, with multiple updates following each round of pre\-trial annotations[Artstein \(2017\)](https://arxiv.org/html/2610.37292#bib.bib2);[Leidner \(2022\)](https://arxiv.org/html/2610.37292#bib.bib3)\. At the same time, disagreement is not always annotation noise\. In subjective tasks it may reflect genuine differences in interpretation and can itself be useful information[Plank et al\. \(2014\)](https://arxiv.org/html/2610.37292#bib.bib10);[Uma et al\. \(2021\)](https://arxiv.org/html/2610.37292#bib.bib20)\.

Tasks may be inherently ambiguous given a particular set of categories, or different annotators may display divergent annotation behavior even when using the same codebook, due, for example, to concentration issues\.

Depending on the annotation setup, several established metrics can be used to assess IAA\. With two annotators and one categorical label for each item, Cohen’sκ\\kappa\([Cohen, 1960](https://arxiv.org/html/2610.37292#bib.bib4)\)uses annotator\-specific marginal distributions\. Scott’sπ\\piand Fleiss’κ\\kappainstead use pooled marginals, with Fleiss’ formulation accommodating more than two annotations per item\([Fleiss, 1971](https://arxiv.org/html/2610.37292#bib.bib12);[Di Eugenio and Glass, 2004](https://arxiv.org/html/2610.37292#bib.bib19);[Artstein and Poesio, 2008](https://arxiv.org/html/2610.37292#bib.bib18)\)\. Krippendorff’sα\\alpha\([Krippendorff, 2011](https://arxiv.org/html/2610.37292#bib.bib13);[Krippendorff, 2013](https://arxiv.org/html/2610.37292#bib.bib1)\)supports any number of annotators, missing data, and user\-defined distance functions\. This last feature also permits set\-valued or vector\-valued annotations\.

For a formal description of a multi\-label annotation task, letNNbe the number of data itemsiito be annotated bynnannotators, each of whom assigns a total ofkkcategory labels \(assigning in this case also includes not assigning, as the absence of a label is also information\)\. For each item and annotator, we therefore do not have a single decision but a set of decisionsLLwithL∈\{0,1\}kL\\in\\\{0,1\\\}^\{k\}\.

Most common IAA metrics, however, such as the ones mentioned above, assume that when there arekkpossible categories for each itemii, each annotator must choose one of the labels\. Thus, for each item and annotator, you end up withℓi,c∈\{1,2,…,k\}\\ell\_\{i,c\}\\in\\\{1,2,\\dots,k\\\}, whereℓi,c\\ell\_\{i,c\}denotes the single category label selected by annotator \(coder\)ccfor itemii\.

There are at least two natural ways to proceed\. One can treat every label as a separate binary annotation problem and combine the resulting statistics, or treat the entire label set as one structured value and select a distance between two sets\. These approaches are not necessarily different: normalized Hamming distance is additive over labels, and therefore reduces the vector comparison to a particular weighting of label\-wise comparisons\.

This paper makes that relationship explicit\. We call the resulting framework*Multi\-Label Agreement*\(MLA\), retaining the terminology used in the accompanying software\. MLA is not a mathematically independent coefficient family; instead, our contributions are \(i\) an exact decomposition into label\-wise coefficients, \(ii\) the connection to Krippendorff’sα\\alphawith Hamming distance, \(iii\) a conditional lower bound showing how far agreement can fall for the observed prevalences, and \(iv\) a practical comparison of alternative chance terms and set similarities\.

## 2Related Work

### 2\.1Past Work on Multi\-Label IAA

Multi\-label agreement has been approached both through dedicated coefficients and through general weighted agreement frameworks\. The latter perspective is particularly important because Krippendorff’sα\\alphacan treat a label set as a structured value and use a distance such as Hamming, Jaccard, or MASI\([Artstein and Poesio, 2008](https://arxiv.org/html/2610.37292#bib.bib18);[Passonneau, 2006](https://arxiv.org/html/2610.37292#bib.bib8)\)\. Recent annotation systems consequently useα\\alphawith MASI distance for multi\-label monitoring\([Jukić et al\., 2023](https://arxiv.org/html/2610.37292#bib.bib25)\)\.

[Rosenberg and Binkowski \(2004\)](https://arxiv.org/html/2610.37292#bib.bib9)proposed an augmentation of Cohen’s kappa, treating multiple labels not as a legitimate outcome of a task but rather a problem arising from annotator uncertainty, following the assumption that there has to be one true label and annotators just struggle to agree which one it is\. The way it is calculated essentially boils down to the kappa for a scenario where coders were only allowed to give one label and had randomly selected one of multiple provided labels\. Thus, this metric has been designed to deflate with the number of labels used, a higher number of labels automatically leads to a lower score\. This is not really suitable for an annotation task where an item could have several true labels\.

[Marchal et al\. \(2022\)](https://arxiv.org/html/2610.37292#bib.bib6)propose a computational approach that uses bootstrapping to estimate chance agreement in multi\-label data, while assessing agreement only over the intersection of annotators’ label sets and deliberately ignoring labels that were never assigned to an item\. While this is a defensible design choice, the simulated \(rather than analytic\) chance term can be more difficult to interpret, and discarding the information conveyed by joint non\-selections means that agreement on a label’s absence is treated as if the label had never been available for that item in the first place\.

MASI was introduced by[Passonneau \(2006\)](https://arxiv.org/html/2610.37292#bib.bib8)and combines the Jaccard index with a penalty factor based on the degree of overlap, ranging from no penalty for an exact match, to a small penalty for a subset match, to a higher penalty for non\-subset overlap between label sets\. MASI itself is a distance or similarity, not a chance\-corrected coefficient; it can be reported directly or supplied as the distance function in a coefficient such asα\\alpha\.

[Bhowmick et al\. \(2008\)](https://arxiv.org/html/2610.37292#bib.bib7)introduce the Am coefficient, aκ\\kappa\-style statistic that extends pairwise agreement to the multi\-label setting by treating every unordered pair of categories as a separate nominal variable\. For each item, annotators’ three possible decisions for a category pair, \(0,0\), \(1,1\), or \(0,1\)/\(1,0\), are compared; the observed overlap is then normalized with a chance term derived from the product of the individual annotators’ marginal probabilities\. Because it averages over all pairs, it implicitly weights errors between frequently co\-occurring labels more heavily, yields only a single corpus\-wide score, and inheritsκ\\kappa’s sensitivity to skewed prevalence while further scaling quadratically in the number of labels\.

Related work on learning from disagreement and soft evaluation asks a different but connected question: whether the distribution of annotator judgements should be preserved rather than reduced to a single gold label\([Uma et al\., 2021](https://arxiv.org/html/2610.37292#bib.bib20);[Rizzi et al\., 2024](https://arxiv.org/html/2610.37292#bib.bib21);[Harju and Mesaros, 2023](https://arxiv.org/html/2610.37292#bib.bib22)\)\. Multi\-label confusion matrices and tensors likewise provide detailed classification diagnostics but do not themselves model chance agreement\([Heydarian et al\., 2022](https://arxiv.org/html/2610.37292#bib.bib23);[Krstinić et al\., 2024](https://arxiv.org/html/2610.37292#bib.bib24)\)\. These lines of work reinforce a central point of the present paper: a single corpus\-level coefficient should be accompanied by diagnostics that preserve where and how disagreement occurs\.

### 2\.2The Ongoing Chance Correction Debate

Many agreement coefficients have the form\(Ao−Ae\)/\(1−Ae\)\(A\_\{o\}\-A\_\{e\}\)/\(1\-A\_\{e\}\), whereAoA\_\{o\}is observed agreement andAeA\_\{e\}is a model\-dependent baseline\. The word “chance” can obscure thatAeA\_\{e\}is not observed independently of the data: it follows from a particular model of how annotations would be allocated in the absence of the agreement of interest\([Di Eugenio and Glass, 2004](https://arxiv.org/html/2610.37292#bib.bib19);[Artstein and Poesio, 2008](https://arxiv.org/html/2610.37292#bib.bib18)\)\.

For a binary labeljj, letpjp\_\{j\}denote the pooled proportion of positive decisions across annotators and items\. The corresponding baselinepj2\+\(1−pj\)2p\_\{j\}^\{2\}\+\(1\-p\_\{j\}\)^\{2\}is Fleiss/Scott\-type because it uses a pooled marginal\. It should not be called Cohen’sκ\\kappa, whose two\-annotator model keeps the annotators’ marginal distributions separate\.

Kappa\-like coefficients can become small in situations with high raw agreement and skewed prevalence\. Gwet’s AC1\([Gwet, 2001](https://arxiv.org/html/2610.37292#bib.bib15)\)uses a different baseline and often remains higher in such situations\. PABAK\([Byrt et al\., 1993](https://arxiv.org/html/2610.37292#bib.bib11)\), also known in this binary form as the Brennan–Prediger coefficient, fixes the baseline at0\.50\.5\.

Konstantinidis et al\.\([Konstantinidis et al\., 2022](https://arxiv.org/html/2610.37292#bib.bib14)\)provide simulation evidence that Gwet’s AC1 often attains lower sampling variance than several kappa estimators in multiple\-rater, binary\-outcome settings and therefore recommend AC1 when skewed prevalences and small sample sizes make kappa unstable\.

Zec et al\.\([Zec et al\., 2017](https://arxiv.org/html/2610.37292#bib.bib17)\)highlight the classic "kappa paradox", cases of high raw agreement but extreme category prevalence where Cohen’sκ\\kappacan appear deceptively low, and show empirical examples in which AC1 \(and related measures\) avoid that particular interpretive pitfall\. Conversely, Vach and Gerke\([Vach and Gerke, 2023](https://arxiv.org/html/2610.37292#bib.bib16)\)stress that AC1 is not a drop\-in replacement: substituting AC1 forκ\\kappachanges the underlying model rather than repairing the same estimand\. We therefore treat these normalizations as sensitivity analyses\. Their values answer different questions and should not be selected after looking at which one gives the most favourable result\.

## 3Multi\-Label Agreement as a Decomposable Construction

### 3\.1Observed agreement

LetLc,i=\(ℓc,i,1,…,ℓc,i,k\)∈\{0,1\}kL\_\{c,i\}=\(\\ell\_\{c,i,1\},\\ldots,\\ell\_\{c,i,k\}\)\\in\\\{0,1\\\}^\{k\}be the vector assigned to itemiiby annotatorcc\. The normalized Hamming distance and its corresponding agreement are

dH​\(Lc,i,Lc′,i\)=1k​∑j=1k𝟏​\(ℓc,i,j≠ℓc′,i,j\),sH=1−dH\.d\_\{H\}\(L\_\{c,i\},L\_\{c^\{\\prime\},i\}\)=\\frac\{1\}\{k\}\\sum\_\{j=1\}^\{k\}\\mathbf\{1\}\(\\ell\_\{c,i,j\}\\neq\\ell\_\{c^\{\\prime\},i,j\}\),\\qquad s\_\{H\}=1\-d\_\{H\}\.Thus, a joint selection\(1,1\)\(1,1\)and a joint omission\(0,0\)\(0,0\)both count as agreement\. Withh=\(n2\)h=\\binom\{n\}\{2\}annotator pairs, the item\-level and corpus\-level observed agreements are

Ao,i\\displaystyle A\_\{o,i\}=1h​∑1≤c<c′≤nsH​\(Lc,i,Lc′,i\),\\displaystyle=\\frac\{1\}\{h\}\\sum\_\{1\\leq c<c^\{\\prime\}\\leq n\}s\_\{H\}\(L\_\{c,i\},L\_\{c^\{\\prime\},i\}\),\(1\)Ao\\displaystyle A\_\{o\}=1N​∑i=1NAo,i\.\\displaystyle=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}A\_\{o,i\}\.\(2\)For labeljjalone, define

Ao,j=1N​h​∑i=1N∑c<c′𝟏​\(ℓc,i,j=ℓc′,i,j\)\.A\_\{o,j\}=\\frac\{1\}\{Nh\}\\sum\_\{i=1\}^\{N\}\\sum\_\{c<c^\{\\prime\}\}\\mathbf\{1\}\(\\ell\_\{c,i,j\}=\\ell\_\{c^\{\\prime\},i,j\}\)\.Because Hamming agreement is additive,Ao=k−1​∑jAo,jA\_\{o\}=k^\{\-1\}\\sum\_\{j\}A\_\{o,j\}\. This equality is useful but also marks a limitation: the construction compares the components of the vectors independently and does not retain information about dependencies or co\-occurrence patterns between labels\.

### 3\.2Alternative baseline models

For binary labeljj, its pooled prevalence is

pj=∑i=1N∑c=1nℓc,i,jn​N\.p\_\{j\}=\\frac\{\\sum\_\{i=1\}^\{N\}\\sum\_\{c=1\}^\{n\}\\ell\_\{c,i,j\}\}\{nN\}\.The pooled\-marginal \(Fleiss/Scott\-type\) agreement baseline is

Ae,j\(F\)=pj2\+\(1−pj\)2,Ae\(F\)=1k​∑j=1kAe,j\(F\)\.A\_\{e,j\}^\{\(F\)\}=p\_\{j\}^\{2\}\+\(1\-p\_\{j\}\)^\{2\},\\qquad A\_\{e\}^\{\(F\)\}=\\frac\{1\}\{k\}\\sum\_\{j=1\}^\{k\}A\_\{e,j\}^\{\(F\)\}\.For comparison, the binary AC1 chance term and the fixed PABAK chance term are

Ae,j\(A​C​1\)\\displaystyle A\_\{e,j\}^\{\(AC1\)\}=2​pj​\(1−pj\),\\displaystyle=2p\_\{j\}\(1\-p\_\{j\}\),Ae\(A​C​1\)\\displaystyle A\_\{e\}^\{\(AC1\)\}=1k​∑j=1kAe,j\(A​C​1\),\\displaystyle=\\frac\{1\}\{k\}\\sum\_\{j=1\}^\{k\}A\_\{e,j\}^\{\(AC1\)\},\(3\)Ae,j\(P​B\)\\displaystyle A\_\{e,j\}^\{\(PB\)\}=0\.5,\\displaystyle=0\.5,Ae\(P​B\)\\displaystyle A\_\{e\}^\{\(PB\)\}=0\.5\.\\displaystyle=0\.5\.\(4\)For binary decisionsAe,j\(A​C​1\)=1−Ae,j\(F\)A\_\{e,j\}^\{\(AC1\)\}=1\-A\_\{e,j\}^\{\(F\)\}, but the two quantities are used as baselines in different coefficient models and must not be interpreted as interchangeable estimates of the same null process\.

Form∈\{F,A​C​1,P​B\}m\\in\\\{F,AC1,PB\\\}, the aggregate coefficient is

Am​l\(m\)=Ao−Ae\(m\)1−Ae\(m\)\.A\_\{ml\}^\{\(m\)\}=\\frac\{A\_\{o\}\-A\_\{e\}^\{\(m\)\}\}\{1\-A\_\{e\}^\{\(m\)\}\}\.\(5\)It equals one for perfect observed agreement\. A value of zero means thatAoA\_\{o\}equals the selected baseline; what that statement means depends onmm\. Item\-level values can be obtained by replacingAoA\_\{o\}withAo,iA\_\{o,i\}while keeping the corpus\-level baseline, but these are diagnostics relative to the corpus and not independent reliability estimates\.

The raw prevalence

pr=1k​∑j=1kpjp\_\{r\}=\\frac\{1\}\{k\}\\sum\_\{j=1\}^\{k\}p\_\{j\}is useful as a compact summary, but it does not replace the label\-wise prevalences because very different vectors\(p1,…,pk\)\(p\_\{1\},\\ldots,p\_\{k\}\)can have the sameprp\_\{r\}\.

### 3\.3Exact relation to label\-wise coefficients

Let

Aj\(m\)=Ao,j−Ae,j\(m\)1−Ae,j\(m\)A\_\{j\}^\{\(m\)\}=\\frac\{A\_\{o,j\}\-A\_\{e,j\}^\{\(m\)\}\}\{1\-A\_\{e,j\}^\{\(m\)\}\}be the coefficient for labeljj, whenever its denominator is non\-zero, and let𝒥m=\{j:1−Ae,j\(m\)\>0\}\\mathcal\{J\}\_\{m\}=\\\{j:1\-A\_\{e,j\}^\{\(m\)\}\>0\\\}\. Labels outside𝒥m\\mathcal\{J\}\_\{m\}contribute zero to both the numerator and denominator below\. SubstitutingAo=k−1​∑jAo,jA\_\{o\}=k^\{\-1\}\\sum\_\{j\}A\_\{o,j\}andAe\(m\)=k−1​∑jAe,j\(m\)A\_\{e\}^\{\(m\)\}=k^\{\-1\}\\sum\_\{j\}A\_\{e,j\}^\{\(m\)\}into Equation[5](https://arxiv.org/html/2610.37292#S3.E5)gives, provided that the aggregate denominator is non\-zero,

Am​l\(m\)=∑j∈𝒥m\(1−Ae,j\(m\)\)​Aj\(m\)∑j∈𝒥m\(1−Ae,j\(m\)\)\.A\_\{ml\}^\{\(m\)\}=\\frac\{\\sum\_\{j\\in\\mathcal\{J\}\_\{m\}\}\\left\(1\-A\_\{e,j\}^\{\(m\)\}\\right\)A\_\{j\}^\{\(m\)\}\}\{\\sum\_\{j\\in\\mathcal\{J\}\_\{m\}\}\\left\(1\-A\_\{e,j\}^\{\(m\)\}\\right\)\}\.\(6\)The aggregate is therefore a weighted average of label\-wise coefficients, not a new functional family\. Under the pooled\-marginal baseline, the weights are2​pj​\(1−pj\)2p\_\{j\}\(1\-p\_\{j\}\), so labels with extreme prevalence receive less weight\. Under AC1 they arepj2\+\(1−pj\)2p\_\{j\}^\{2\}\+\(1\-p\_\{j\}\)^\{2\}, which gives relatively more weight to extreme labels\. Under PABAK all weights are equal\. These implicit weights should be reported because they determine what the corpus\-level number summarizes\.

### 3\.4Relation to Krippendorff’s alpha

The same construction can be written as a distance\-based coefficient\. LetDo=1−AoD\_\{o\}=1\-A\_\{o\}be observed normalized Hamming disagreement\. Drawing two label vectors independently from the pooled empirical distribution gives

De\(∞\)=1k​∑j=1k2​pj​\(1−pj\)=1−Ae\(F\)\.D\_\{e\}^\{\(\\infty\)\}=\\frac\{1\}\{k\}\\sum\_\{j=1\}^\{k\}2p\_\{j\}\(1\-p\_\{j\}\)=1\-A\_\{e\}^\{\(F\)\}\.Consequently,

Am​l\(F\)=1−DoDe\(∞\)\.A\_\{ml\}^\{\(F\)\}=1\-\\frac\{D\_\{o\}\}\{D\_\{e\}^\{\(\\infty\)\}\}\.This has the form of Krippendorff’sα\\alphawith normalized Hamming distance\. Standardα\\alphadraws two distinct values without replacement from theM=n​NM=nNobserved vectors and, for complete balanced data, uses

De\(α\)=MM−1​De\(∞\),αH=1−DoDe\(α\)\.D\_\{e\}^\{\(\\alpha\)\}=\\frac\{M\}\{M\-1\}D\_\{e\}^\{\(\\infty\)\},\\qquad\\alpha\_\{H\}=1\-\\frac\{D\_\{o\}\}\{D\_\{e\}^\{\(\\alpha\)\}\}\.Thus the pooled\-marginal MLA coefficient and Hammingα\\alphadiffer only by this finite\-sample convention and converge asMMgrows\. Krippendorff’s framework is more general because it also provides explicit handling of missing data and permits other distances\. For example, Jaccard or MASI distance deemphasizes joint omissions and instead focuses on overlap between selected labels\.

### 3\.5Minimum feasible agreement conditional on prevalence

With more than two annotators, zero pairwise agreement is generally impossible for a binary decision\. Without fixing the prevalence, the smallest possible pairwise observed agreement for a label is

Ao,min=\{n−22​\(n−1\),n​even,n−12​n,n​odd\.A\_\{o,\\min\}=\\begin\{cases\}\\dfrac\{n\-2\}\{2\(n\-1\)\},&n\\text\{ even\},\\\\\[4\.0pt\] \\dfrac\{n\-1\}\{2n\},&n\\text\{ odd\}\.\\end\{cases\}A more informative diagnostic keeps the observed number of positive decisions fixed\. Let

Tj=∑i=1N∑c=1nℓc,i,j,xi,j=∑c=1nℓc,i,j\.T\_\{j\}=\\sum\_\{i=1\}^\{N\}\\sum\_\{c=1\}^\{n\}\\ell\_\{c,i,j\},\\qquad x\_\{i,j\}=\\sum\_\{c=1\}^\{n\}\\ell\_\{c,i,j\}\.For an item withxxpositive andn−xn\-xnegative decisions, the number of agreeing annotator pairs is

f⁡\(x\)=\(x2\)\+\(n−x2\)\.f\(x\)=\\binom\{x\}\{2\}\+\\binom\{n\-x\}\{2\}\.The functionffis convex on the integers\. If two item counts differ by at least two, moving one positive decision from the larger to the smaller count cannot increase their combined pair agreement\. Repeating this exchange shows that, subject to∑ixi,j=Tj\\sum\_\{i\}x\_\{i,j\}=T\_\{j\}, agreement is minimized when all item counts differ by at most one\.

Set

mj=⌊TjN⌋,tj=Tj−mj​N\.m\_\{j\}=\\left\\lfloor\\frac\{T\_\{j\}\}\{N\}\\right\\rfloor,\\qquad t\_\{j\}=T\_\{j\}\-m\_\{j\}N\.ThenN−tjN\-t\_\{j\}items receivemjm\_\{j\}positives andtjt\_\{j\}items receivemj\+1m\_\{j\}\+1positives\. The conditional lower bound is therefore

Ao,min,j\(pj\)=\(N−tj\)​f​\(mj\)\+tj​f​\(mj\+1\)N​\(n2\),A\_\{o,\\min,j\}^\{\(p\_\{j\}\)\}=\\frac\{\(N\-t\_\{j\}\)f\(m\_\{j\}\)\+t\_\{j\}f\(m\_\{j\}\+1\)\}\{N\\binom\{n\}\{2\}\},\(7\)where the second term is omitted whentj=0t\_\{j\}=0\. Averaging over labels and applying the same normalization yields

Ao,min\(𝐩\)=1k​∑j=1kAo,min,j\(pj\),Am​l,min\(m\)=Ao,min\(𝐩\)−Ae\(m\)1−Ae\(m\)\.A\_\{o,\\min\}^\{\(\\mathbf\{p\}\)\}=\\frac\{1\}\{k\}\\sum\_\{j=1\}^\{k\}A\_\{o,\\min,j\}^\{\(p\_\{j\}\)\},\\qquad A\_\{ml,\\min\}^\{\(m\)\}=\\frac\{A\_\{o,\\min\}^\{\(\\mathbf\{p\}\)\}\-A\_\{e\}^\{\(m\)\}\}\{1\-A\_\{e\}^\{\(m\)\}\}\.This is a feasible\-range diagnostic, not another chance model and not part of the observed agreement\. It explains why a coefficient may be unable to attain strongly negative values once the number of annotators and empirical marginals are fixed\.

### 3\.6Interpreting Prevalence

When the raw prevalenceprp\_\{r\}is near 0, almost every coder assigns 0 to almost every label\. Because a\(0,0\)\(0,0\)comparison counts as agreement,AoA\_\{o\}can become high even when coders rarely coincide on the scarce positive cases\. This matters in tasks where recognizing an occasional positive label is the actual challenge\. Take the rabbit–duck illusion: coders may dispute whether a rabbit, a duck, or both are present, yet they will unanimously agree that other animals are absent\. Expanding the inventory with irrelevant animals therefore inflates Hamming agreement without improving reliability on the labels of interest\.

A lowprp\_\{r\}is not automatically a problem\. In some applications positives are genuinely rare and certifying their absence is an important part of the task\. Think of aircraft\-maintenance engineers who tick possible defects on an inspection checklist: almost all boxes will, and should, remain unticked\. Here, joint omissions may be meaningful\. The relevant question is therefore not whether negative matches should always count, but whether all labels were genuinely considered and whether confirming their absence has comparable value to confirming their presence\.

The mirror image occurs whenprp\_\{r\}is close to 1:AoA\_\{o\}is then dominated by\(1,1\)\(1,1\)matches\. Whether this is informative again depends on the task\. A single global prevalence is insufficient for making this judgement, so the label\-wisepjp\_\{j\},Ao,jA\_\{o,j\}, positive agreement, and negative agreement should be inspected alongside a corpus\-level coefficient\.

Natural ambiguity also has to be separated from annotation error\. Genre annotation, for example, may legitimately produce different but defensible label sets\. Such disagreement should often be preserved for later analysis rather than hidden behind a reliability threshold\([Plank et al\., 2014](https://arxiv.org/html/2610.37292#bib.bib10);[Uma et al\., 2021](https://arxiv.org/html/2610.37292#bib.bib20)\)\.

No normalization resolves this domain decision automatically\. The pooled\-marginal coefficient asks how much observed Hamming agreement exceeds that implied by the empirical label marginals\. AC1 uses a different baseline that remains less punitive at extreme prevalence, but can consequently be positive even when annotators draw independently from the same skewed Bernoulli distribution\. PABAK fixes a balanced baseline and has the same issue under skewed independent decisions\. These are properties of the respective models, not evidence that one coefficient has recovered the uniquely correct amount of reliability\. We therefore recommend pre\-specifying the primary model and reportingAoA\_\{o\}, label\-wise prevalences, and at least one sensitivity analysis\.

### 3\.7Example

To demonstrate the calculations, consider the following subsample from an experiment where three annotators were asked to annotate bias in sentences from news articles using a pre\-specified taxonomy of 31 types\. Table[1](https://arxiv.org/html/2610.37292#S3.T1)shows the selected types and the corresponding vectors\. This sample is far too small for a substantive reliability conclusion; it is only a worked example\.

Table 1:Assigned bias labels and corresponding 31\-bit vectors by sentence and annotator\.We takeN=4N=4sentences,k=31k=31categories,n=3n=3annotators \(h=\(32\)=3h=\\binom\{3\}\{2\}=3\)\.

For each sentenceii, compute

Sℒi=∑c<c′I⁡\(Lc,i,Lc′,i\),Aoi=Sℒi31⋅3S\_\{\\mathcal\{L\}\_\{i\}\}=\\sum\_\{c<c^\{\\prime\}\}I\(L\_\{c,i\},L\_\{c^\{\\prime\},i\}\),\\quad A\_\{o\_\{i\}\}=\\frac\{S\_\{\\mathcal\{L\}\_\{i\}\}\}\{31\\cdot 3\}i=1:\\displaystyle i=1:\\quadS=31\+31\+31=93⇒Ao1=93/93=1\.000,\\displaystyle S=31\+31\+31=93\\quad\\Rightarrow\\;A\_\{o\_\{1\}\}=93/93=1\.000,\\;i=2:\\displaystyle i=2:\\quadS=26\+27\+30=83⇒Ao2=83/93≈0\.8925,\\displaystyle S=26\+27\+30=83\\quad\\Rightarrow\\;A\_\{o\_\{2\}\}=83/93\\approx 0\.8925,\\;i=3:\\displaystyle i=3:\\quadS=27\+29\+29=85⇒Ao3=85/93≈0\.9140,\\displaystyle S=27\+29\+29=85\\quad\\Rightarrow\\;A\_\{o\_\{3\}\}=85/93\\approx 0\.9140,\\;i=4:\\displaystyle i=4:\\quadS=26\+27\+26=79⇒Ao4=79/93≈0\.8495,\\displaystyle S=26\+27\+26=79\\quad\\Rightarrow\\;A\_\{o\_\{4\}\}=79/93\\approx 0\.8495,\\;
For the pooled\-marginal and AC1 baseline terms we obtain

Ae\(F\)=27\.527831≈0\.8880,Ae\(a​c​1\)=1−Ae\(F\)≈0\.1120\.A\_\{e\}^\{\(F\)\}=\\frac\{27\.5278\}\{31\}\\approx 0\.8880,\\qquad A\_\{e\}^\{\(ac1\)\}=1\-A\_\{e\}^\{\(F\)\}\\approx 0\.1120\.
Hence the overall multi\-label agreements under the three methods are

Am​l\(F\)\\displaystyle A\_\{ml\}^\{\(F\)\}=1\+0\.8925\+0\.9140\+0\.84954−0\.88801−0\.8880≈0\.2320,\\displaystyle=\\frac\{\\tfrac\{1\+0\.8925\+0\.9140\+0\.8495\}\{4\}\-0\.8880\}\{1\-0\.8880\}\\approx 0\.2320,Am​l\(a​c​1\)\\displaystyle A\_\{ml\}^\{\(ac1\)\}≈0\.9031,\\displaystyle\\approx 0\.9031,Am​l\(p​b\)\\displaystyle A\_\{ml\}^\{\(pb\)\}≈0\.8280\.\\displaystyle\\approx 0\.8280\.
As for the raw prevalence, across all12=4×312=4\\times 3decisions per label there are2828positive assignments, so

pr=1k​∑j=1kpj=2812⋅31=793≈0\.0753\.p\_\{r\}=\\frac\{1\}\{k\}\\sum\_\{j=1\}^\{k\}p\_\{j\}=\\frac\{28\}\{12\\cdot 31\}=\\frac\{7\}\{93\}\\approx 0\.0753\.
The minimum observed agreements given the empirical prevalence based on the total number of 1s are

Averaging over all 31 labels gives

Aominpj=⋅1\+⋅56\+⋅23\+⋅12\+⋅1331=7993≈0\.8495\.A\_\{o\_\{\\min\}\}^\{p\_\{j\}\}=\\frac\{20\\\!\\cdot\\\!1\+3\\\!\\cdot\\\!\\frac\{5\}\{6\}\+2\\\!\\cdot\\\!\\frac\{2\}\{3\}\+3\\\!\\cdot\\\!\\frac\{1\}\{2\}\+3\\\!\\cdot\\\!\\frac\{1\}\{3\}\}\{31\}=\\frac\{79\}\{93\}\\approx 0\.8495\.
This sets the three normalized floors to

Am​l,min\(F\)\\displaystyle A\_\{ml,\\min\}^\{\(F\)\}=7993−0\.88801−0\.8880≈−0\.3440,\\displaystyle=\\frac\{\\tfrac\{79\}\{93\}\-0\.8880\}\{1\-0\.8880\}\\approx\-0\.3440,\(8\)Am​l,min\(a​c​1\)\\displaystyle A\_\{ml,\\min\}^\{\(ac1\)\}=7993−0\.11201−0\.1120≈0\.8305,\\displaystyle=\\frac\{\\tfrac\{79\}\{93\}\-0\.1120\}\{1\-0\.1120\}\\approx 0\.8305,Am​l,min\(p​b\)\\displaystyle A\_\{ml,\\min\}^\{\(pb\)\}=7993−0\.50000\.5000≈0\.6989\.\\displaystyle=\\frac\{\\tfrac\{79\}\{93\}\-0\.5000\}\{0\.5000\}\\approx 0\.6989\.
This is a scenario with very low prevalence\. The pooled\-marginal coefficient is low because most raw agreement is also implied by the empirical marginals\. AC1 and PABAK are high, but their conditional floors are high as well: given the observed number of positive decisions, these coefficients cannot approach−1\-1and in the AC1 case cannot even approach zero\. This is why a value without its baseline and feasible range is easy to overinterpret\.

Twenty of the 31 pre\-specified categories are never selected in the example\. Table[2](https://arxiv.org/html/2610.37292#S3.T2)shows the effect of removing them post hoc\. The pooled\-marginal coefficient and its floor remain unchanged because constant labels have weight1−Ae,j\(F\)=01\-A\_\{e,j\}^\{\(F\)\}=0in Equation[6](https://arxiv.org/html/2610.37292#S3.E6)\. AC1 and PABAK change substantially because constant labels receive non\-zero weight under these models\. Neither label universe is automatically correct: the full inventory is appropriate if all annotators genuinely considered all 31 labels, whereas post\-hoc removal changes the estimand and can hide the effect of an unnecessarily broad label inventory\.

Table 2:Effect of the declared label universe in the worked example\. Removing never\-selected labels leaves the pooled\-marginal coefficient unchanged but changes the AC1 and fixed\-baseline summaries\.Table[3](https://arxiv.org/html/2610.37292#S3.T3)reports, for every category that received at least one positive assignment, its prevalence, observed agreement, and the three label\-wise coefficients\. Their variation highlights why a corpus\-level number should not be reported alone\.

To put these results into perspective, the same example was also used to calculate the measures mentioned in Section[2](https://arxiv.org/html/2610.37292#S2)\(BBgives the number of bootstrap runs\):

Augmented​κ\\displaystyle\\text\{Augmented \}\\kappa=0\.253,\\displaystyle=0\.253,boot\-F1​\(B=1000\)\\displaystyle\\text\{boot\-F1 \}\(B=1000\)=0\.366,\\displaystyle=0\.366,MASI=0\.509,\\displaystyle=0\.509,Am\\displaystyle A\_\{m\}=0\.281\.\\displaystyle=0\.281\.
These values are not estimates of one common quantity\. Augmentedκ\\kappaandAmA\_\{m\}apply their own chance models, boot\-F1 ignores joint negative decisions, and the directly reported MASI similarity is not chance\-corrected\. Their disagreement is therefore informative: different summaries reward different aspects of the same annotations\.

Table 3:Label\-wise agreement statistics for the 11 labels assigned at least once\.

## 4Behaviour Under Controlled Data Generators

Table 4:Representative agreement metrics \(averaged overn∈\{3,5,10\}n\\\!\\in\\\!\\\{3,5,10\\\},N∈\{20,50,100\}N\\\!\\in\\\!\\\{20,50,100\\\},k∈\{5,10,25\}k\\\!\\in\\\!\\\{5,10,25\\\}, 20 repeats, 100 bootstrap iterations for boot\-F1\)To study how the three normalizations respond to different data characteristics and how they compare to the measures presented in Section[2](https://arxiv.org/html/2610.37292#S2), we generated synthetic annotation matrices\. The purpose is descriptive: the simulations show the consequences of the models and do not establish that one coefficient is generally more valid than another\. Two orthogonal factors control the data:

Prevalence:For every label the probability that it is*set*in the hidden ground\-truth vector is fixed to*high*\(p=0\.80\)\(p=0\.80\),*moderate*\(p=0\.50\)\(p=0\.50\)or*low*\(p=0\.20\)\(p=0\.20\)\.

Agreement level:After the ground\-truth vectors have been drawn, the individual annotator decisions are created as follows\.

1. 1\.Perfect agreement– every annotator copies the ground truth, hence the observed agreementAo=1\.0A\_\{o\}=1\.0\.
2. 2\.High agreement– each 0/1 decision is independently flipped with a small error probabilityε=0\.10\\varepsilon=0\.10\.
3. 3\.Independent draws– annotators ignore the ground truth and draw their own 0/1 decision from the sameBernoulli​\(p\)\\text\{Bernoulli\}\(p\)distribution; any agreement that occurs is therefore implied by the marginal distribution\.
4. 4\.Minimum feasible agreement– for every item–label pair we first computea=max⁡\(1,min⁡\(n−1,⌊p​n\+0\.5⌋\)\)a=\\max\\bigl\(1,\\min\(n\-1,\\lfloor pn\+0\.5\\rfloor\)\\bigr\)and then chooseaaof thennannotators at random to assign the value11, while the remainingn−an\-aassign00\. This guarantees that the realised prevalencep^=a/n\\hat\{p\}=a/nmatches the requestedpp\(up to the unavoidable rounding ofaa\) and at the same time forces every pair of annotators to disagree on as many decisions as possible given that prevalence\.

For every combinationn∈\{3,5,10\}n\\in\\\{3,5,10\\\}annotators,N∈\{20,50,100\}N\\in\\\{20,50,100\\\}items andk∈\{5,10,25\}k\\in\\\{5,10,25\\\}labels, the simulation was repeated2020times; the table reports the averages of the resulting3×3×3×20=5403\\times 3\\times 3\\times 20=540metric values per scenario\.

Table[4](https://arxiv.org/html/2610.37292#S4.T4)reports averages overnn,NN, andkk\. This compact summary makes the main prevalence interaction visible but should not be read as a study of sampling variance; a full assessment would keep these dimensions separate and report interval estimates\.

For perfect agreement, all measures except augmented kappa attain 1\.0\. The different value of augmented kappa follows from its specific model for multiply labelled data and illustrates why results from different coefficients should not be placed on one interpretive scale without qualification\.

Under independent Bernoulli draws, the pooled\-marginal coefficient remains close to zero because its baseline matches the generator\. AC1 and PABAK become positive at extreme prevalence because they use different baselines\. This is not a simulation failure: it shows that zero has a different meaning under the three models\. Direct MASI similarity also increases with the number of shared positive labels and is not centred at a chance value\.

In the noisy\-ground\-truth condition, pooled\-marginal agreement is smaller at extreme prevalence, AC1 is larger, and PABAK is constant\. These patterns can also be obtained analytically\. With flip probabilityε\\varepsilon, two annotators agree with probability\(1−ε\)2\+ε2\(1\-\\varepsilon\)^\{2\}\+\\varepsilon^\{2\}, while their marginal positive probability isq=ε\+p⁡\(1−2​ε\)q=\\varepsilon\+p\(1\-2\\varepsilon\)\. Figure[1](https://arxiv.org/html/2610.37292#S4.F1)plots the resulting coefficients continuously overppand also shows the independent\-draw generator\.

![Refer to caption](https://arxiv.org/html/2610.37292v1/prevalence_sensitivity.png)Figure 1:Analytic prevalence sensitivity\. Left: annotators draw independently fromBernoulli⁡\(p\)\\mathrm\{Bernoulli\}\(p\)\. Right: annotators copy a latent label and independently flip it with probabilityε=0\.1\\varepsilon=0\.1\. The pooled\-marginal coefficient is calibrated to the independent generator; AC1 and PABAK use different baselines\.The minimum\-feasible\-agreement rows also confirm the lower\-bound analysis in Equation[7](https://arxiv.org/html/2610.37292#S3.E7): with more than two annotators and fixed binary marginals, strongly negative values may be unattainable\. This is why universal verbal cut\-offs such as “good” or “poor” agreement are particularly inappropriate here\.

The code used to generate Figure[1](https://arxiv.org/html/2610.37292#S4.F1)is included with the preprint source\. The Monte Carlo implementation is available in the accompanying software repository\.

## 5Reporting Recommendations

The analysis suggests that a multi\-label reliability report should make the following choices explicit:

1. 1\.Label universe\.State whether the inventory was fixed before annotation and whether every annotator was expected to consider every label\. Never\-selected labels should not be removed only because they lower or raise a preferred summary\.
2. 2\.Meaning of joint omissions\.Use Hamming\-based agreement when joint absence is informative\. If overlap among positive selections is the scientific focus, additionally report a set distance such as Jaccard or MASI, preferably within a chance\-corrected framework when such correction is needed\.
3. 3\.Baseline model\.Pre\-specify the primary chance model\. A pooled\-marginal coefficient, AC1, and PABAK do not provide three interchangeable estimates of the same quantity\.
4. 4\.Diagnostics\.ReportNN, the number of annotators per item,kk,AoA\_\{o\}, label\-wise prevalences and agreements, and the conditional lower bound\. Item\-level values can identify difficult examples, but reliability conclusions require an adequate number of items\.
5. 5\.Uncertainty\.For substantive studies, accompany corpus\-level and important label\-level estimates with confidence intervals obtained through a design\-appropriate bootstrap or an analytic method\. Resampling must respect the item and annotator structure of the study\.

If labels are hierarchical, mutually exclusive in groups, or otherwise dependent, the additive Hamming construction is often not sufficient\. In that case, Krippendorff’s framework with a domain\-specific distance, or an explicit probabilistic annotator model, is preferable to treating the dependencies as if they did not exist\.

## 6Implementation

A Python implementation is available in the[project repository](https://github.com/Timperator2/MultiLabelAgreement)and from the Python Package Index as[ml\-agreement](https://pypi.org/project/ml-agreement/)\. The package implements corpus\-, item\-, and label\-level summaries, the three baseline choices discussed here, label prevalence, and the conditional minimum\-agreement calculation\. The pooled\-marginal option is retained as the package’s legacyk/kappa option; in the paper we useFFto avoid confusing pooled marginals with Cohen’s annotator\-specific model\.

## 7Limitations

The present analysis is restricted to complete, balanced annotation designs with binary, non\-hierarchical labels\. The simple formulas do not by themselves handle missing annotations\. Treating a missing decision as 0 would incorrectly turn missingness into an observed omission; instead, pair counts and marginal denominators must be computed from the available values, as done in the general formulation of Krippendorff’sα\\alpha\.

Normalized Hamming distance is additive and therefore insensitive to dependencies among labels\. It also weights all label decisions equally before chance correction\. Hierarchical relations, mutually exclusive subsets, and differences in the cost of false positive and false negative decisions require a different distance or a model tailored to the annotation design\.

The worked example contains only four items and is illustrative\. The simulation varies several design factors but averages overnn,NN, andkkand does not provide inferential comparisons or confidence intervals\. A more complete empirical study should analyse these factors separately, include annotator\-specific marginals and correlated labels, and compare against Hammingα\\alphausing the same data\. Finally, tasks with genuine ambiguity may be better represented by confidence judgements or distributions over interpretations rather than binary decisions alone\.

## 8Conclusion

Multi\-label agreement does not require an entirely new coefficient family\. Average pairwise Hamming agreement, combined with a pooled\-marginal chance term, is a chance\-disagreement\-weighted average of label\-wise Fleiss/Scott\-type coefficients and is closely related to Krippendorff’sα\\alphawith Hamming distance\. Stating this equivalence is useful because it reveals both the strengths and the limitations of the construction: it scales linearly in the number of labels and counts joint omissions, but it decomposes over labels and does not model their dependencies\.

The choice of baseline remains consequential\. The pooled\-marginal, AC1, and fixed0\.50\.5baselines can give very different values on the same annotations, especially under extreme prevalence\. The simulations show that these differences follow from the models rather than from numerical instability alone\. Reporting several coefficients can be a useful sensitivity analysis, but it does not remove the need to select and justify a primary estimand\.

The conditional minimum\-agreement formula adds a complementary diagnostic\. It shows how low observed and normalized agreement can feasibly fall once the number of annotators and empirical label prevalences are fixed\. Together with raw agreement, label\-wise results, and an explicit declaration of the label universe, this provides a more informative account than a single coefficient\. The accompanying implementation packages these calculations so that the assumptions can be inspected rather than hidden behind one recommended score\.

## References

- R\. Artstein and M\. PoesioInter\-coder agreement for computational linguistics\.Computational Linguistics34\(4\),pp\. 555–596\.External Links:[Document](https://dx.doi.org/10.1162/coli.07-034-R2),[Link](https://aclanthology.org/J08-4004/)Cited by:[§1](https://arxiv.org/html/2610.37292#S1.p1.1),[§1](https://arxiv.org/html/2610.37292#S1.p5.1),[§2\.1](https://arxiv.org/html/2610.37292#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2610.37292#S2.SS2.p1.1)\.
- Artstein \(2017\)R\. ArtsteinInter\-annotator agreement\.InHandbook of Linguistic Annotation,N\. Ide and J\. Pustejovsky \(Eds\.\),pp\. 297–313\.External Links:[Document](https://dx.doi.org/10.1007/978-94-024-0881-2%5F11)Cited by:[§1](https://arxiv.org/html/2610.37292#S1.p3.1)\.
- Bhowmicket al\.\(2008\)P\. K\. Bhowmick, A\. Basu, and P\. MitraAn agreement measure for determining inter\-annotator reliability of human judgements on affective text\.InColing 2008 Workshop on Human Judgements in Computational Linguistics,Manchester, UK,pp\. 58–65\.Cited by:[§2\.1](https://arxiv.org/html/2610.37292#S2.SS1.p5.1)\.
- Byrtet al\.\(1993\)T\. Byrt, J\. Bishop, and J\. B\. CarlinBias, prevalence and kappa\.Journal of Clinical Epidemiology46\(5\),pp\. 423–429\.External Links:[Document](https://dx.doi.org/10.1016/0895-4356%2893%2990018-V)Cited by:[§2\.2](https://arxiv.org/html/2610.37292#S2.SS2.p3.1)\.
- Carletta \(1996\)J\. CarlettaAssessing agreement on classification tasks: the kappa statistic\.Comput\. Linguist\.22\(2\),pp\. 249–254\.External Links:ISSN 0891\-2017Cited by:[§1](https://arxiv.org/html/2610.37292#S1.p1.1)\.
- Cohen \(1960\)J\. CohenA coefficient of agreement for nominal scales\.Educational and Psychological Measurement20\(1\),pp\. 37–46\.External Links:[Document](https://dx.doi.org/10.1177/001316446002000104)Cited by:[§1](https://arxiv.org/html/2610.37292#S1.p5.1)\.
- Di Eugenio and Glass \(2004\)B\. Di Eugenio and M\. GlassThe kappa statistic: a second look\.Computational Linguistics30\(1\),pp\. 95–101\.External Links:[Document](https://dx.doi.org/10.1162/089120104773633402),[Link](https://aclanthology.org/J04-1005/)Cited by:[§1](https://arxiv.org/html/2610.37292#S1.p5.1),[§2\.2](https://arxiv.org/html/2610.37292#S2.SS2.p1.1)\.
- Fleiss \(1971\)J\. L\. FleissMeasuring nominal scale agreement among many raters\.Psychological Bulletin76\(5\),pp\. 378–382\.External Links:[Document](https://dx.doi.org/10.1037/h0031619)Cited by:[§1](https://arxiv.org/html/2610.37292#S1.p5.1)\.
- Gwet \(2001\)K\. L\. GwetHandbook of inter\-rater reliability: how to estimate the level of agreement between two or multiple raters\.STATAXIS Publishing Company,Gaithersburg, MD, USA\.Cited by:[§2\.2](https://arxiv.org/html/2610.37292#S2.SS2.p3.1)\.
- Harju and Mesaros \(2023\)M\. Harju and A\. MesarosEvaluating classification systems against soft labels with fuzzy precision and recall\.InProceedings of the 8th Workshop on Detection and Classification of Acoustic Scenes and Events,Tampere, Finland\.External Links:[Link](https://arxiv.org/abs/2309.13938)Cited by:[§2\.1](https://arxiv.org/html/2610.37292#S2.SS1.p6.1)\.
- Heydarianet al\.\(2022\)M\. Heydarian, T\. E\. Doyle, and R\. SamaviMLCM: multi\-label confusion matrix\.IEEE Access10,pp\. 19083–19095\.External Links:[Document](https://dx.doi.org/10.1109/ACCESS.2022.3151048)Cited by:[§2\.1](https://arxiv.org/html/2610.37292#S2.SS1.p6.1)\.
- Jukićet al\.\(2023\)J\. Jukić, F\. Jelenić, M\. Bićanić, and J\. ŠnajderALANNO: an active learning annotation system for mortals\.InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations,pp\. 228–235\.External Links:[Link](https://aclanthology.org/2023.eacl-demo.26/)Cited by:[§2\.1](https://arxiv.org/html/2610.37292#S2.SS1.p1.1)\.
- Konstantinidiset al\.\(2022\)M\. Konstantinidis, L\. W\. Le, and X\. GaoAn empirical comparative assessment of inter\-rater agreement of binary outcomes and multiple raters\.Symmetry14\(2\),pp\. 262\.Note:External Links:[Document](https://dx.doi.org/10.3390/sym14020262)Cited by:[§2\.2](https://arxiv.org/html/2610.37292#S2.SS2.p4.1)\.
- Krippendorff \(2011\)K\. KrippendorffComputing krippendorff’s alpha\-reliability\.SAGE Open1\(3\),pp\. 215824401141522\.External Links:[Document](https://dx.doi.org/10.1177/2158244011415228)Cited by:[§1](https://arxiv.org/html/2610.37292#S1.p5.1)\.
- Krippendorff \(2013\)K\. KrippendorffContent analysis: an introduction to its methodology\.3rd edition,Sage,Thousand Oakes, CA, USA\.Cited by:[§1](https://arxiv.org/html/2610.37292#S1.p5.1)\.
- Krstinićet al\.\(2024\)D\. Krstinić, A\. Kuzmanić Skelin, I\. Slapničar, and M\. BraovićMulti\-label confusion tensor\.IEEE Access12,pp\. 9860–9870\.External Links:[Document](https://dx.doi.org/10.1109/ACCESS.2024.3353050)Cited by:[§2\.1](https://arxiv.org/html/2610.37292#S2.SS1.p6.1)\.
- Leidner \(2022\)J\. L\. LeidnerData\-to\-value: an ’evaluation\-first’ methodology for natural language projects\.InNatural Language Processing and Information Systems: Proceedings of the 27th International Conference on Applications of Natural Language to Information Systems \(NLDB 2022\), Valencia, Spain, June 15–17, 2022,Berlin, Heidelberg,pp\. 517–523\.External Links:ISBN 978\-3\-031\-08472\-0,[Document](https://dx.doi.org/10.1007/978-3-031-08473-7%5F48)Cited by:[§1](https://arxiv.org/html/2610.37292#S1.p1.1),[§1](https://arxiv.org/html/2610.37292#S1.p3.1)\.
- Marchalet al\.\(2022\)M\. Marchal, M\. C\.J\. Scholman, F\. Yung, and V\. DembergEstablishing annotation quality in multi\-label annotations\.InProceedings of the 29th International Conference on Computational Linguistics \(COLING 2022\),Gyeongju, Republic of Korea,pp\. 3659–3668\.Note:Language Science and Technology / Saarland UniversityExternal Links:[Link](https://aclanthology.org/2022.coling-1.322.pdf)Cited by:[§2\.1](https://arxiv.org/html/2610.37292#S2.SS1.p3.1)\.
- Passonneau \(2006\)R\. J\. PassonneauMeasuring agreement on set\-valued items \(MASI\) for semantic and pragmatic annotation\.InProceedings of the Fifth International Conference on Language Resources and Evaluation \(LREC 2006\),Genoa, Italy,pp\. 831–836\.Cited by:[§2\.1](https://arxiv.org/html/2610.37292#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2610.37292#S2.SS1.p4.1)\.
- Planket al\.\(2014\)B\. Plank, D\. Hovy, and A\. SøgaardLinguistically debatable or just plain wrong?\.InProceedings of the 52nd Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),K\. Toutanova and H\. Wu \(Eds\.\),Baltimore, MD, USA,pp\. 507–511\.External Links:[Document](https://dx.doi.org/10.3115/v1/P14-2083)Cited by:[§1](https://arxiv.org/html/2610.37292#S1.p3.1),[§3\.6](https://arxiv.org/html/2610.37292#S3.SS6.p4.1)\.
- Rizziet al\.\(2024\)G\. Rizzi, E\. Leonardelli, M\. Poesio, A\. Uma, M\. Pavlovic, S\. Paun, P\. Rosso, and E\. FersiniSoft metrics for evaluation with disagreements: an assessment\.InProceedings of the 3rd Workshop on Perspectivist Approaches to NLP,pp\. 84–94\.External Links:[Link](https://aclanthology.org/2024.nlperspectives-1.9/)Cited by:[§2\.1](https://arxiv.org/html/2610.37292#S2.SS1.p6.1)\.
- Rosenberg and Binkowski \(2004\)A\. Rosenberg and E\. BinkowskiAugmenting the kappa statistic to determine interannotator reliability for multiply labeled data points\.InProceedings of HLT\-NAACL 2004: Short Papers,Boston, Massachusetts, USA,pp\. 77–80\.External Links:[Link](https://aclanthology.org/N04-4020)Cited by:[§2\.1](https://arxiv.org/html/2610.37292#S2.SS1.p2.1)\.
- Umaet al\.\(2021\)A\. N\. Uma, T\. Fornaciari, D\. Hovy, S\. Paun, B\. Plank, and M\. PoesioLearning from disagreement: a survey\.Journal of Artificial Intelligence Research72,pp\. 1385–1470\.External Links:[Document](https://dx.doi.org/10.1613/jair.1.12752)Cited by:[§1](https://arxiv.org/html/2610.37292#S1.p3.1),[§2\.1](https://arxiv.org/html/2610.37292#S2.SS1.p6.1),[§3\.6](https://arxiv.org/html/2610.37292#S3.SS6.p4.1)\.
- Vach and Gerke \(2023\)W\. Vach and O\. GerkeGwet’s AC1 is not a substitute for Cohen’s kappa–a comparison of basic properties\.MethodsX10,pp\. 102212\.External Links:[Document](https://dx.doi.org/10.1016/j.mex.2023.102212)Cited by:[§2\.2](https://arxiv.org/html/2610.37292#S2.SS2.p5.1)\.
- Zecet al\.\(2017\)S\. Zec, N\. Soriani, R\. Comoretto, and I\. BaldiHigh agreement and high prevalence: the paradox of Cohen’s Kappa\.The Open Nursing Journal11,pp\. 211–218\.External Links:[Document](https://dx.doi.org/10.2174/1874434601711010211)Cited by:[§2\.2](https://arxiv.org/html/2610.37292#S2.SS2.p5.1)\.