AHEAD: Advancing Multi-Class Label Aggregation with Interpretable Cross-Annotator Modeling

# AHEAD: Advancing Multi-Class Label Aggregation with Interpretable Cross-Annotator Modeling
Source: [https://arxiv.org/html/2607.18465](https://arxiv.org/html/2607.18465)
Ju Chen1, Sijia Xu2, Jun Feng1, Zhiqiang Gao3, Zhengyi Yang4,\*

###### Abstract

Crowdsourced labeling provides valuable labeled data for domains across natural language processing, computer vision, and video\. Label aggregation aims to infer latent true labels from noisy and biased annotations, with the key lying in annotator reliability estimation\. Despite promising progress, existing approaches struggle with one real\-world bottleneck: most individual annotators label only a small subset of tasks, making accurate annotator estimation highly intractable\. In this paper, we focus on the considerably more challenging multi\-class label aggregation and proposeAHEAD\(cross\-Annotator learning andHigh\-confidEnceAnnotator\-guideDlabel aggregation\), a cross\-annotator learning framework that advances annotator reliability estimation by leveraging the population\-level data\. Specifically, AHEAD first learns high\-dimensional cross\-annotator contexts via a graph neural network, deriving multi\-view, complementary annotator embeddings by aggregating individual\-level annotator features with contextual information\. These embeddings are then decoded into interpretable annotator\-specific confusion matrices to fit the observed labels\. We formulate a composite objective incorporating high\-confidence annotators to alleviate the unsupervised training issues faced by prior models\. Experiments on 10 real\-world datasets spanning NLP, CV, Video, and Audio show that AHEAD substantially improves label accuracy, increasing average accuracy from 68\.75% to 73\.23%, with gains of up to 14\.9% in the best case\. Meanwhile, scalability experiments on the largest dataset further demonstrate the overall superiority of our method\.

## IIntroduction

By harnessing crowd intelligence, crowdsourced labeling continuously injects massive, valuable labeled data into areas including natural language processing, computer vision, and video\[[28](https://arxiv.org/html/2607.18465#bib.bib47),[34](https://arxiv.org/html/2607.18465#bib.bib46),[9](https://arxiv.org/html/2607.18465#bib.bib45),[39](https://arxiv.org/html/2607.18465#bib.bib54)\], expanding both data breadth and knowledge depth\[[55](https://arxiv.org/html/2607.18465#bib.bib52),[36](https://arxiv.org/html/2607.18465#bib.bib58),[50](https://arxiv.org/html/2607.18465#bib.bib55)\]\. Meanwhile, it has become indispensable for evaluating and fact\-checking large language models\[[35](https://arxiv.org/html/2607.18465#bib.bib2),[22](https://arxiv.org/html/2607.18465#bib.bib1),[11](https://arxiv.org/html/2607.18465#bib.bib56),[14](https://arxiv.org/html/2607.18465#bib.bib57)\]\. Considering the varying reliability of different annotators\[[5](https://arxiv.org/html/2607.18465#bib.bib53)\], crowdsourcing platforms typically assign each task to multiple annotators and apply label aggregation to infer the latent true label \(truth\)\[[21](https://arxiv.org/html/2607.18465#bib.bib49),[56](https://arxiv.org/html/2607.18465#bib.bib48)\]\. Majority Voting \(MV\), which treats the label provided by most annotators as the truth, is the most naive method but is prone to bias as it fails to account for annotator reliability\[[15](https://arxiv.org/html/2607.18465#bib.bib32)\]\. Consequently, a wide spectrum of fine\-grained approaches has emerged to jointly infer annotator reliability and latent truths\. In this paper, we focus on themulti\-class label aggregation, which is considerably common and more challenging than the binary cases due to complex inter\-class confusion and annotator noise\[[8](https://arxiv.org/html/2607.18465#bib.bib39),[47](https://arxiv.org/html/2607.18465#bib.bib40),[54](https://arxiv.org/html/2607.18465#bib.bib65)\]\.

Depending on how annotators are modeled, existing approaches can be grouped intosingle\-parameter\-based\[[57](https://arxiv.org/html/2607.18465#bib.bib51),[23](https://arxiv.org/html/2607.18465#bib.bib12),[24](https://arxiv.org/html/2607.18465#bib.bib50),[27](https://arxiv.org/html/2607.18465#bib.bib5)\],confusion\-matrix\-based\[[10](https://arxiv.org/html/2607.18465#bib.bib6),[20](https://arxiv.org/html/2607.18465#bib.bib7),[53](https://arxiv.org/html/2607.18465#bib.bib24),[51](https://arxiv.org/html/2607.18465#bib.bib21),[6](https://arxiv.org/html/2607.18465#bib.bib4)\], andrepresentation\-learning\-based\[[49](https://arxiv.org/html/2607.18465#bib.bib36),[31](https://arxiv.org/html/2607.18465#bib.bib44),[44](https://arxiv.org/html/2607.18465#bib.bib8),[45](https://arxiv.org/html/2607.18465#bib.bib14),[30](https://arxiv.org/html/2607.18465#bib.bib34),[29](https://arxiv.org/html/2607.18465#bib.bib35)\]methods\. By modeling annotator reliability with a scalar accuracy shared across all classes, single\-parameter\-based methods\[[43](https://arxiv.org/html/2607.18465#bib.bib11),[23](https://arxiv.org/html/2607.18465#bib.bib12),[27](https://arxiv.org/html/2607.18465#bib.bib5),[4](https://arxiv.org/html/2607.18465#bib.bib10)\]impose a strong homogeneity assumption over diverse classes\. Therefore, confusion\-matrix\-based and representation\-learning\-based methods currently dominate due to their extensive characterization of annotators\.

Confusion\-matrix\-based methods, which model each annotator with a confusion matrix, are theoretically well\-founded for capturing the label generation process\[[27](https://arxiv.org/html/2607.18465#bib.bib5)\]\. The DS model\[[10](https://arxiv.org/html/2607.18465#bib.bib6)\]is the first to introduce annotator\-specific confusion matrices to characterize annotator reliability\. Subsequent extensions incorporate more expressive priors: IBCC\[[20](https://arxiv.org/html/2607.18465#bib.bib7)\]adopts a Dirichlet prior over confusion matrices, FGBCC\[[6](https://arxiv.org/html/2607.18465#bib.bib4)\]develops a Softmax\-Gaussian prior, and EBCC\[[26](https://arxiv.org/html/2607.18465#bib.bib3)\]further extends IBCC\[[20](https://arxiv.org/html/2607.18465#bib.bib7)\]by partitioning tasks with the same truth into subtypes\. More recently, Zhang et al\.\[[51](https://arxiv.org/html/2607.18465#bib.bib21)\]propose a coupled method to jointly refine confusion matrices\.

![Refer to caption](https://arxiv.org/html/2607.18465v1/x1.png)\(a\)
![Refer to caption](https://arxiv.org/html/2607.18465v1/x2.png)\(b\)

Figure 1:\(a\) Average class\-wise annotation counts for annotators with the median number of annotations in the MS dataset\. \(b\) Cross\-annotator correlations learned on the MS dataset, wherewiw\_\{i\}denotes theii\-th annotator\.Empowered by their capacity to learn complex multi\-dimensional embeddings, representation\-learning\-based methods have recently proliferated\. Wu et al\.\[[45](https://arxiv.org/html/2607.18465#bib.bib14)\]propose a hierarchical graph autoencoder to model multi\-view interactions within a task\-annotator graph, complemented by an initialization strategy that iteratively refines node embeddings and annotator reliability\. Liu et al\.\[[30](https://arxiv.org/html/2607.18465#bib.bib34)\]apply graph contrastive learning on an annotator\-task bipartite graph, generating augmented views via edge perturbation to promote view invariance\. Recently, Liu et al\.\[[29](https://arxiv.org/html/2607.18465#bib.bib35)\]pretrain a bipartite graph neural network on a series of simulated labeling scenarios to learn diverse annotation patterns and generalize across datasets without dataset\-specific training\.

Although confusion\-matrix\-based and representation\-learning\-based methods are well\-grounded and highly expressive, they struggle with a fundamental real\-world bottleneck:most individual annotators label only a small subset of tasks, a ubiquitous phenomenon in real\-world applications\[[18](https://arxiv.org/html/2607.18465#bib.bib17),[51](https://arxiv.org/html/2607.18465#bib.bib21)\]that makes accurate annotator estimation highly intractable\. For instance, Figure[1](https://arxiv.org/html/2607.18465#S1.F1)\(a\) illustrates the average class\-wise annotation counts for annotators with the median number of annotations in the real\-world dataset MS\[[37](https://arxiv.org/html/2607.18465#bib.bib59)\], revealing substantial data sparsity in overall annotation volume, and theclass\-level sparsityandimbalanceare even more pronounced, as most annotators cover only a few samples for many classes\. Moreover, current methods mainly build upon low\-density annotator\-task topology, making the inference unreliable\. In contrast, we argue that cross\-annotator correlations are more dense and reveal multi\-view, complementary annotator reliability\. For instance, Figure[1](https://arxiv.org/html/2607.18465#S1.F1)\(b\) depicts the dense cross\-annotator correlations that we learn from MS, with the color bar indicating the correlation degree\. These correlations offer us population\-level information to derive multi\-view, complementary assessments of individual annotators\.

To this end, this work aims to achieve accurate multi\-class label aggregation under sparse data\. Intuitively, confusion matrices offer an interpretable and theoretically grounded characterization of the label generation process\[[27](https://arxiv.org/html/2607.18465#bib.bib5)\], while representation learning excels at capturing latent nonlinear correlations that are difficult to model explicitly\. Therefore, we first propose to adopt representation learning to model high\-dimensional cross\-annotator correlations and derive multi\-view, complementary annotator embeddings from the population\. Then, we decode these embeddings into interpretable, annotator\-specific confusion matrices to fit the observed labels\. We thereby advance annotator reliability estimation by leveraging the population data\. Since we operate without ground\-truth supervision, two critical issues must be addressed:

- •Learning expressive annotator embeddings that map to valid confusion matrices\. It is non\-trivial to derive multi\-view, complementary annotator embeddings and project them into valid annotator confusion matrices in a principled manner\.
- •Establishing a robust optimization objective\. Prior approaches typically rely on Majority Voting \(MV\)\-derived pseudo\-labels for model training\. However, MV\-derived labels are notoriously noisy, and training directly on such signals can deteriorate the performance\.

Building upon the above insights, we proposeAHEAD\(cross\-Annotator learning andHigh\-confidEnceAnnotator\-guideDlabel aggregation\)\. Specifically,

\(1\) to address the first issue, AHEADexplicitly models high\-dimensional cross\-annotator learningover their contextual topology and learns multi\-view, complementary annotator embeddings by aggregating individual\-level annotator features with their contextual information via a graph neural network\. These embeddings are then decoded intointerpretable annotator\-specific confusion matricesto fit the observed labels\. We introduce a negative log\-likelihood loss using the decoded confusion matrices to maximize the likelihood of the observations, ensuring expressive annotator embeddings and valid, statistically sound annotator confusion matrices\.

\(2\) to address the second issue, we formulate a composite objective that integrateslikelihood maximization,structural regularization, anddirectional guidance\. Specifically, we identify annotators whose task counts fall within a top percentage as high\-confidence annotators and leverage their empirical confusion matrices as critical supervisory signals by minimizing the discrepancy between the predicted and empirical confusion matrices\. This mechanism is motivated by the Law of Large Numbers\[[13](https://arxiv.org/html/2607.18465#bib.bib60)\], which suggests that an empirical estimate becomes increasingly accurate as the sample size grows\. Therefore, although Majority Voting \(MV\) can be noisy, the empirical confusion matrices of high\-confidence annotators are statistically trustworthy and thereby alleviate the unsupervised training issues faced by prior models\.

Extensive experiments across 10 real\-world datasets spanning five domains demonstrate the effectiveness and efficiency of AHEAD, and scalability experiments on the largest dataset demonstrate the comprehensive superiority of our method\.

In summary, the contributions of this study are as follows:

- •We propose cross\-annotator learning to advance individual annotator reliability estimation by leveraging population\-level data and unifying it with interpretable confusion matrices to fit the observed labels, substantially improving the accuracy of multi\-class label aggregation\.
- •We identify high\-confidence annotators and formulate a composite objective incorporating likelihood maximization, structural regularization, and directional guidance to alleviate the unsupervised training issues faced by prior models\.
- •Extensive experiments on 10 real\-world datasets spanning NLP, CV, Video, and Audio demonstrate that we achieve a 14\.9% accuracy improvement in the best case and boost the average accuracy from 68\.75% to 73\.23%\. Experiments on the largest dataset confirm the scalability of our method\.

## IIRelated Works

This paper focuses on crowdsourced label aggregation that infers truths and annotator reliability solely from \(task, annotator, label\) triples\. Notably, there are also related works for scenarios with rich task features available\[[41](https://arxiv.org/html/2607.18465#bib.bib61),[25](https://arxiv.org/html/2607.18465#bib.bib62),[19](https://arxiv.org/html/2607.18465#bib.bib63),[40](https://arxiv.org/html/2607.18465#bib.bib64),[52](https://arxiv.org/html/2607.18465#bib.bib41)\]or mobile crowdsourcing where annotators perform sensing tasks via mobile devices in specific environments\[[46](https://arxiv.org/html/2607.18465#bib.bib66),[2](https://arxiv.org/html/2607.18465#bib.bib67),[32](https://arxiv.org/html/2607.18465#bib.bib43)\], which are outside the scope of our detailed discussion\.

Depending on how annotators are modeled, existing studies can be divided into single\-parameter\-based\[[12](https://arxiv.org/html/2607.18465#bib.bib16),[3](https://arxiv.org/html/2607.18465#bib.bib15),[27](https://arxiv.org/html/2607.18465#bib.bib5),[4](https://arxiv.org/html/2607.18465#bib.bib10)\], confusion\-matrix\-based\[[20](https://arxiv.org/html/2607.18465#bib.bib7),[26](https://arxiv.org/html/2607.18465#bib.bib3),[38](https://arxiv.org/html/2607.18465#bib.bib13),[51](https://arxiv.org/html/2607.18465#bib.bib21),[6](https://arxiv.org/html/2607.18465#bib.bib4)\], and representation\-learning\-based\[[49](https://arxiv.org/html/2607.18465#bib.bib36),[33](https://arxiv.org/html/2607.18465#bib.bib38),[44](https://arxiv.org/html/2607.18465#bib.bib8),[45](https://arxiv.org/html/2607.18465#bib.bib14),[30](https://arxiv.org/html/2607.18465#bib.bib34)\]methods\.

### II\-ASingle\-Parameter\-Based Methods

Single\-parameter\-based methods assume that each annotator maintains a scalar accuracy across all classes\. The GLAD model\[[43](https://arxiv.org/html/2607.18465#bib.bib11)\]incorporates task difficulty and annotator reliability to characterize the log odds of each label being true and infers the parameters using the expectation\-maximization framework\. The CATD model\[[23](https://arxiv.org/html/2607.18465#bib.bib12)\]models annotator errors as normally distributed and formulates an objective to minimize the weighted sum of error variances, with annotator reliability updated via the upper bound of the variance’s confidence interval\. Aydin et al\.\[[1](https://arxiv.org/html/2607.18465#bib.bib33)\]also minimize a weighted sum of annotators’ errors, constraining annotator reliability to be non\-negative and summing to one, and infer using block coordinate descent techniques\. Bonald and Combes\[[3](https://arxiv.org/html/2607.18465#bib.bib15)\]derive a lower bound on the minimax estimation error and propose triangular annotator estimation to ensure non\-asymptotic performance\. BWA\[[27](https://arxiv.org/html/2607.18465#bib.bib5)\]and OKELE\[[4](https://arxiv.org/html/2607.18465#bib.bib10)\]are both generative Bayesian models\. The BWA model\[[27](https://arxiv.org/html/2607.18465#bib.bib5)\]models annotator reliability with a Gamma distribution and updates parameters via the expectation\-maximization framework\. The OKELE model\[[4](https://arxiv.org/html/2607.18465#bib.bib10)\]models annotator reliability with a scaled inverse chi\-squared distribution and updates parameters via a gradient method\.

Generally, single\-parameter\-based methods are computationally efficient, but their strong homogeneous annotator reliability across classes makes them suboptimal in practice\.

### II\-BConfusion\-Matrix\-Based Methods

Confusion\-matrix\-based methods exploit annotator\-specific confusion matrices to distinguish annotators’ varying reliability across classes\. Assuming that annotators are independent, the DS model\[[10](https://arxiv.org/html/2607.18465#bib.bib6)\]is the first to model each annotator using a confusion matrix within a Bayesian generative framework, with parameters updated through maximum likelihood estimation\. The IBCC model\[[20](https://arxiv.org/html/2607.18465#bib.bib7)\]and FGBCC model\[[6](https://arxiv.org/html/2607.18465#bib.bib4)\]extend DS by introducing Dirichlet and Softmax\-Gaussian priors for the rows of confusion matrices, respectively\. Specifically, IBCC\[[20](https://arxiv.org/html/2607.18465#bib.bib7)\]utilizes Gibbs sampling for inference, whereas FGBCC\[[6](https://arxiv.org/html/2607.18465#bib.bib4)\]adopts a variational inference approach\. The CBCC model\[[42](https://arxiv.org/html/2607.18465#bib.bib9)\]posits the existence of latent annotator communities, assuming that each annotator’s confusion matrix aligns closely with that of their respective community\. The DBCC model\[[42](https://arxiv.org/html/2607.18465#bib.bib9)\]exploits a Markov network to model the dependence between annotators and updates the parameters via a Metropolis sampling method\. The EBCC model\[[26](https://arxiv.org/html/2607.18465#bib.bib3)\]extends IBCC\[[20](https://arxiv.org/html/2607.18465#bib.bib7)\]by dividing tasks with the same truth into subtypes\. Chu et al\.\[[7](https://arxiv.org/html/2607.18465#bib.bib19)\]employ both global and individual confusion matrices to characterize annotator behavior\. Specifically, they introduce a Bernoulli variable to model the label generation process, conditioned on task difficulty and annotator reliability\. Ibrahim et al\.\[[16](https://arxiv.org/html/2607.18465#bib.bib18)\],\[[17](https://arxiv.org/html/2607.18465#bib.bib20)\]estimate annotator reliability through a matrix factorization that leverages annotator co\-occurrence\.

Confusion\-matrix\-based methods are theoretically well\-founded through explicit modeling of the label generation process\. However, ubiquitous data sparsity make full confusion matrix estimation highly unstable\.

![Refer to caption](https://arxiv.org/html/2607.18465v1/x3.png)Figure 2:Overview of the proposed AHEAD, which consists of four components: \(a\) Input; \(b\) Feature extraction, which derives cross\-annotator contexts, teacher and initial confusion matrices, and prior truth distribution; \(c\) Cross\-annotator learning, where the graph attention layer learns cross\-annotator correlations, the encoder layer derive multi\-view, complementary annotator embeddings, and the decoder layer predict annotator\-specific confusion matrices; \(d\) Truth estimation, which updates posterior truth probabilities based on the updated parameters\.
### II\-CRepresentation\-Learning\-Based Methods

Representation\-learning\-based methods embed annotators, tasks, and labels into continuous spaces to capture latent relationships\. Yin et al\.\[[49](https://arxiv.org/html/2607.18465#bib.bib36)\]propose an unsupervised autoencoder framework that integrates a classifier and a reconstructor: the former infers truths from the input, while the latter reconstructs the input from these inferred truths\. Lyu et al\.\[[33](https://arxiv.org/html/2607.18465#bib.bib38)\]learn embeddings for tasks, annotators, and labels by constructing heterogeneous networks from their interactions\. These embeddings are iteratively refined based on the gradients of each sub\-network\. The TiReMGE model\[[44](https://arxiv.org/html/2607.18465#bib.bib8)\]takes task\-annotator links as first\-order interactions and treats task\-task and annotator\-annotator links as second\-order interactions\. The model estimates annotator reliability by minimizing the distance between encoded task representations and the labels provided by reliable annotators\. The GOVERN model\[[30](https://arxiv.org/html/2607.18465#bib.bib34)\]is a graph contrastive learning framework built on an annotator\-task bipartite graph, consisting of: a\) a data augmentation module that selectively adds or drops edges; b\) a bipartite graph neural network for node embeddings; and c\) a contrastive objective combining both instance\- and category\-level losses to encourage robust, invariant embeddings\. The CrowdFM model\[[29](https://arxiv.org/html/2607.18465#bib.bib35)\]is a bipartite graph neural network that pretrains on a wide spectrum of simulated labeling scenarios to learn diverse annotation patterns and generalize across datasets without dataset\-specific training\.

Despite their powerful expressiveness, representation\-learning\-based methods require substantial data to train network parameters and degrade under data sparsity\. Moreover, prior works build upon low\-density annotator\-task bipartite graphs, overlooking cross\-annotator correlations that are more dense and encode intrinsic annotator reliability\.

## IIIProblem Definition

We consider a crowdsourcing setting withTTtasks,WWannotators, andKKclasses\. Letyi​j∈\{1​…​K\}y\_\{ij\}\\in\\\{1\\ldots K\\\}denote the label provided by annotatorwjw\_\{j\}\(j∈\{1​…​W\}j\\in\\\{1\\ldots W\\\}\) for tasktit\_\{i\}\(i∈\{1​…​T\}i\\in\\\{1\\ldots T\\\}\), andzi∈\{1​…​K\}z\_\{i\}\\in\\\{1\\ldots K\\\}denote the sole truth fortit\_\{i\}\. Let𝒯j\\mathcal\{T\}\_\{j\}denote the set of tasks thatwjw\_\{j\}has annotated,𝒲i\\mathcal\{W\}\_\{i\}denote the set of annotators who have annotatedtit\_\{i\}, and𝒯j​q=𝒯j∩𝒯q\\mathcal\{T\}\_\{jq\}=\\mathcal\{T\}\_\{j\}\\cap\\mathcal\{T\}\_\{q\}denote the set of tasks annotated by bothwjw\_\{j\}andwqw\_\{q\}\. The goal is to infer the latent truthziz\_\{i\}for eachtit\_\{i\}solely from the noisy labels\. This involves inferring: \(i\) the truth probability of each label, and \(ii\) the reliability of each annotator\. Typically, most annotators label only a small subset of tasks\.

## IVProposed method: ahead

In this section, we present the proposedAHEAD\(cross\-Annotator learning andHigh\-confidEnceAnnotator\-guideDlabel aggregation\), as illustrated in Figure[2](https://arxiv.org/html/2607.18465#S2.F2)\. Our objective is to accurately infer annotator reliability, especially bridging the estimation gap for annotators who label only a small subset of tasks from the population\. To achieve this, we first extract annotator, truth distribution, and cross\-annotator topology from the observed labels\. Then, we model latent cross\-annotator correlations to derive multi\-view, complementary annotator embeddings, which are decoded into annotator\-specific confusion matrices\. We formulate a composite objective for effective model training\. Finally, we update posterior truth probabilities\.

### IV\-AFeature Extraction

Cross\-Annotator Context Extraction\.Prior works build upon the annotator\-task bipartite graph, which is typically low\-density as most individual annotators label only a small subset of tasks, while the number of tasks can be thousands of times greater than the number of annotators\. A typical solution applies graph augmentation by selectively dropping or adding annotator\-task edges; however, this may violate the intrinsic structure and requires high storage overhead, since edges often fail to satisfy the i\.i\.d\. assumption\[[48](https://arxiv.org/html/2607.18465#bib.bib42)\]\. In contrast, we focus on the intrinsic and dense cross\-annotator contexts generated from the observed labels\. This is because the cross\-annotator topology graph, in which two annotators are connected whenever they share a labeled task, is typically much denser than the annotator\-task bipartite graph induced by annotator individual labeling activities\.

We extract the cross\-annotator contextual topology𝒢\\mathcal\{G\}, along with the positive and negative annotator pair sets, as follows:

###### Definition 1

\(Cross\-Annotator Contextual Topology𝒢\\mathcal\{G\}\) For any annotator pair \(wjw\_\{j\},wqw\_\{q\}\) satisfying\|𝒯j∩𝒯q\|≥1\|\\mathcal\{T\}\_\{j\}\\cap\\mathcal\{T\}\_\{q\}\|\\geq 1, we establish an undirected edge\{wj,wqw\_\{j\},w\_\{q\}\}between them and add it toℰ\\mathcal\{E\}, forming the cross\-annotator contextual topology structure𝒢=\{\{wj\}j=1W,ℰ\}\\mathcal\{G\}=\\\{\\\{w\_\{j\}\\\}\_\{j=1\}^\{W\},\\mathcal\{E\}\\\}\. We define the contextual peers ofwjw\_\{j\}, denoted𝒩​\(wj\)\\mathcal\{N\}\(w\_\{j\}\), as the nodes having an edge withwjw\_\{j\}\.

###### Definition 2

\(Positive/Negative Annotator Pairs\) For any annotator pair \(wjw\_\{j\},wqw\_\{q\}\) satisfying\|𝒯j​q\|=\|𝒯j∩𝒯q\|≥δ0\|\\mathcal\{T\}\_\{jq\}\|=\|\\mathcal\{T\}\_\{j\}\\cap\\mathcal\{T\}\_\{q\}\|\\geq\\delta\_\{0\}, we construct the positive and negative annotator pairsℰ\+\\mathcal\{E\}\_\{\+\}andℰ−\\mathcal\{E\}\_\{\-\}according to Eq\. \([1](https://arxiv.org/html/2607.18465#S4.E1)\),

\(wj,wq\)∈\{ℰ\+,if​\(∑ti∈𝒯j​q𝕀​\[yi​j=yi​q\]\|𝒯j​q\|−0\.5\)⋅2≥δ1ℰ−,if​\(∑ti∈𝒯j​q𝕀​\[yi​j=yi​q\]\|𝒯j​q\|−0\.5\)⋅2≤−δ2\(w\_\{j\},w\_\{q\}\)\\in\\begin\{cases\}\\mathcal\{E\}\_\{\+\},&\\text\{if \}\\left\(\\frac\{\\sum\_\{t\_\{i\}\\in\\mathcal\{T\}\_\{jq\}\}\\mathbb\{I\}\[y\_\{ij\}=y\_\{iq\}\]\}\{\|\\mathcal\{T\}\_\{jq\}\|\}\-0\.5\\right\)\\cdot 2\\geq\\delta\_\{1\}\\\\ \\mathcal\{E\}\_\{\-\},&\\text\{if \}\\left\(\\frac\{\\sum\_\{t\_\{i\}\\in\\mathcal\{T\}\_\{jq\}\}\\mathbb\{I\}\[y\_\{ij\}=y\_\{iq\}\]\}\{\|\\mathcal\{T\}\_\{jq\}\|\}\-0\.5\\right\)\\cdot 2\\leq\-\\delta\_\{2\}\\end\{cases\}\(1\)where𝕀​\[⋅\]\\mathbb\{I\}\[\\cdot\]is an indicator function that returns 1 if\[⋅\]\[\\cdot\]is true and 0 otherwise\. Moreover, we useℰ\+j\\mathcal\{E\}\_\{\+\}^\{j\}andℰ−j\\mathcal\{E\}\_\{\-\}^\{j\}to denote the set of annotators that have a positive and a negative connection withwjw\_\{j\}, respectively\.

We extract𝒢\\mathcal\{G\},ℰ\+\\mathcal\{E\}\_\{\+\}, andℰ−\\mathcal\{E\}\_\{\-\}, which capture rich and intrinsic cross\-annotator contexts that provide critical signals for annotator reliability estimation\. Specifically, the cross\-annotator topology𝒢\\mathcal\{G\}is much denser than annotator\-task bipartite graph, enabling representation learning to model cross\-annotator correlations and derive multi\-view, complementary annotator embeddings by aggregating individual\-level annotator features with contextual information\. This dense topology also facilitates the propagation of reliability information across annotators\. Meanwhile, we constructℰ\+\\mathcal\{E\}\_\{\+\}andℰ−\\mathcal\{E\}\_\{\-\}to further refine annotator embeddings through contrastive learning\.

Teacher & Initial Confusion Matrices\.Inspired by the Law of Large Numbers\[[13](https://arxiv.org/html/2607.18465#bib.bib60)\], which suggests that empirical estimates converge to true values as sample size increases, we identify annotators whose answered task counts fall within a predefined top percentage as high\-confidence annotators and treat their empirical confusion matrices as critical supervision\. The motivation is to design a teacher\-student loss that minimizes the discrepancy between the network\-predicted confusion matrices of high\-confidence annotators and their teacher confusion matrices, thereby guiding the network training\.

###### Definition 3

\(High\-Confidence Annotator Set𝒲c​o​n​f\\mathcal\{W\}\_\{conf\}\) We identify high\-confidence annotators as those whose annotated task counts rank in the topρ\\rho%, and use𝒲c​o​n​f\\mathcal\{W\}\_\{conf\}to denote the high\-confidence annotator set\.

We derive teacher confusion matrices𝐂\(t​e\)\\mathbf\{C\}^\{\(te\)\}of high\-confidence annotators as follows\. First, for eachtit\_\{i\}\(i∈\{1​…​T\}i\\in\\\{1\\ldots T\\\}\), we compute its pseudo\-truth:

z^i=arg⁡maxk​∑wj∈𝒲i𝕀​\[yi​j=k\]\\hat\{z\}\_\{i\}=\\arg\\max\_\{k\}\\sum\_\{w\_\{j\}\\in\\mathcal\{W\}\_\{i\}\}\\mathbb\{I\}\[y\_\{ij\}=k\]\(2\)Then, for eachwj∈𝒲c​o​n​fw\_\{j\}\\in\\mathcal\{W\}\_\{conf\}, the corresponding teacher confusion matrix𝐂\(t​e\)​j∈ℝK×K\\mathbf\{C\}^\{\(te\)j\}\\in\\mathbb\{R\}^\{K\\times K\}, with𝐂k​l\(t​e\)​j\\mathbf\{C\}^\{\(te\)j\}\_\{kl\}denoting the probability thatwjw\_\{j\}labelsllwhile the truth iskk, is defined as:

𝐂k​l\(t​e\)​j=\{∑ti∈𝒯j𝕀​\[z^i=k∧yi​j=l\]∑ti∈𝒯j𝕀​\[z^i=k\],if​∑ti∈𝒯j𝕀​\[z^i=k\]\>01,if​∑ti∈𝒯j𝕀​\[z^i=k\]=0∧k=l0,otherwise\\mathbf\{C\}^\{\(te\)j\}\_\{kl\}=\\begin\{cases\}\\frac\{\\sum\\limits\_\{t\_\{i\}\\in\\mathcal\{T\}\_\{j\}\}\\mathbb\{I\}\[\\hat\{z\}\_\{i\}=k\\land y\_\{ij\}=l\]\}\{\\sum\\limits\_\{t\_\{i\}\\in\\mathcal\{T\}\_\{j\}\}\\mathbb\{I\}\[\\hat\{z\}\_\{i\}=k\]\},&\\text\{if \}\\sum\\limits\_\{t\_\{i\}\\in\\mathcal\{T\}\_\{j\}\}\\mathbb\{I\}\[\\hat\{z\}\_\{i\}=k\]\>0\\\\ 1,&\\text\{if \}\\sum\\limits\_\{t\_\{i\}\\in\\mathcal\{T\}\_\{j\}\}\\mathbb\{I\}\[\\hat\{z\}\_\{i\}=k\]=0\\land k=l\\\\ 0,&\\text\{otherwise\}\\end\{cases\}\(3\)
In contrast, the initial confusion matrix𝐂\(0\)​j\\mathbf\{C\}^\{\(0\)j\}for eachwjw\_\{j\}\(j=1​…​Wj=1\\ldots W\), serving as the features to initialize annotator embeddings, is defined as:

𝐂k​l\(0\)​j=\{∑ti∈𝒯j𝕀​\[z^i=k∧yi​j=l\]∑ti∈𝒯j𝕀​\[z^i=k\],if​∑ti∈𝒯j𝕀​\[z^i=k\]\>0δ3,if​∑ti∈𝒯j𝕀​\[z^i=k\]=0∧k=l0,otherwise\\mathbf\{C\}^\{\(0\)j\}\_\{kl\}=\\begin\{cases\}\\frac\{\\sum\\limits\_\{t\_\{i\}\\in\\mathcal\{T\}\_\{j\}\}\\mathbb\{I\}\[\\hat\{z\}\_\{i\}=k\\land y\_\{ij\}=l\]\}\{\\sum\\limits\_\{t\_\{i\}\\in\\mathcal\{T\}\_\{j\}\}\\mathbb\{I\}\[\\hat\{z\}\_\{i\}=k\]\},&\\text\{if \}\\sum\\limits\_\{t\_\{i\}\\in\\mathcal\{T\}\_\{j\}\}\\mathbb\{I\}\[\\hat\{z\}\_\{i\}=k\]\>0\\\\ \\delta\_\{3\},&\\text\{if \}\\sum\\limits\_\{t\_\{i\}\\in\\mathcal\{T\}\_\{j\}\}\\mathbb\{I\}\[\\hat\{z\}\_\{i\}=k\]=0\\land k=l\\\\ 0,&\\text\{otherwise\}\\end\{cases\}\(4\)Notably, the confusion matrix for the same high\-confidence annotatorwj∈𝒲c​o​n​fw\_\{j\}\\in\\mathcal\{W\}\_\{conf\}is computed in two different ways depending on its role\. For generating the initial annotator embedding \(𝐂\(0\)\\mathbf\{C\}^\{\(0\)\}\), we assign a conservative diagonal value ofδ3<1\\delta\_\{3\}<1to all annotators for any class where they lack coverage \(empty rows\)\. In contrast, when used as the target supervisory signal \(𝐂\(t​e\)\\mathbf\{C\}^\{\(te\)\}\) in guiding the optimization trajectory, we assign a high diagonal value of 1\.0 to high\-confidence annotators\. While this hard assignment implies a strong assumption, it is empirically justified in our context, as the selected high\-confidence annotators typically exhibit comprehensive label coverage and rarely suffer from the missing class problem\.

It should be noted that the supervisory guidance from high\-confidence annotators differs substantially from prior methods relying on MV \(Majority Voting\)\-derived pseudo\-labels for cross\-entropy optimization\. Because while MV results are often noisy, reliability estimates for high\-confidence annotators are statistically more robust, which is grounded in the Law of Large Numbers\. This process not only reduces noise but also enables widespread propagation of supervision\.

Prior Truth Distribution\.Given the unsupervised setting and the Law of Large Numbers, we use the aggregated truth probabilities obtained from MV over all tasks to derive the prior truth distribution𝝅\\boldsymbol\{\\pi\}\. Specifically,∀i∈\{1,…,T\}\\forall i\\in\\\{1,\\dots,T\\\},k∈\{1,…,K\}k\\in\\\{1,\\dots,K\\\}, we define

𝐳^i,k=∑wj∈𝒲i𝕀​\[yi​j=k\]\|𝒲i\|\\mathbf\{\\hat\{z\}\}\_\{i,k\}=\\frac\{\\sum\_\{w\_\{j\}\\in\\mathcal\{W\}\_\{i\}\}\\mathbb\{I\}\[y\_\{ij\}=k\]\}\{\|\\mathcal\{W\}\_\{i\}\|\}\(5\)𝝅k=1T​∑i=1T𝐳^i,k\\boldsymbol\{\\pi\}\_\{k\}=\\frac\{1\}\{T\}\\sum\_\{i=1\}^\{T\}\\mathbf\{\\hat\{z\}\}\_\{i,k\}\(6\)

### IV\-BCross\-Annotator Learning

To derive robust and comprehensive annotator estimation in sparse crowdsourcing scenarios, we propose cross\-annotator learning to derive multi\-view, complementary annotator embeddings by aggregating individual\-level annotator representations with their contextual information in𝒢\\mathcal\{G\}\. Then, we decode these embeddings into interpretable annotator\-specific confusion matrices to fit the observations\. This design reflects both the advantages of confusion\-matrix\-based and representation\-learning\-based modeling\.

We derive initial annotator embeddings𝐡\(0\)∈ℝW×d1\\mathbf\{h\}^\{\(0\)\}\\in\\mathbb\{R\}^\{W\\times d\_\{1\}\}based on their initial confusion matrices𝐂\(0\)\\mathbf\{C\}^\{\(0\)\}, whered1d\_\{1\}denotes the hidden layer dimension\. We first flat the initial confusion matrix of each annotator𝐂\(0\)​j\\mathbf\{C\}^\{\(0\)j\}into a vector\. These vectors are then projected into the embedding space using a learnable linear transformation matrix𝐀∈ℝK2×d1\\mathbf\{A\}\\in\\mathbb\{R\}^\{K^\{2\}\\times d\_\{1\}\}and a bias vector𝐛∈ℝd1\\mathbf\{b\}\\in\\mathbb\{R\}^\{d\_\{1\}\}\. Let𝒞∈ℝW×K2\\mathcal\{C\}\\in\\mathbb\{R\}^\{W\\times K^\{2\}\}denote the matrix reshaped from𝐂\(0\)\\mathbf\{C\}^\{\(0\)\}, where each row represents the flattened confusion matrix of an annotator\. The initial annotator embedding𝐡\(0\)\\mathbf\{h\}^\{\(0\)\}is derived as:

𝐡\(0\)=𝒞​𝐀\+𝐛\\mathbf\{h\}^\{\(0\)\}=\\mathcal\{C\}\\mathbf\{A\}\+\\mathbf\{b\}\(7\)∀wj\\forall w\_\{j\}\(j∈\{1​…​W\}j\\in\\\{1\\ldots W\\\}\) and everywq∈𝒩​\(wj\)w\_\{q\}\\in\\mathcal\{N\}\(w\_\{j\}\), we learn an attention coefficient between the source nodewqw\_\{q\}and the target nodewjw\_\{j\}\. Specifically, we first concatenate the projected features ofwqw\_\{q\}andwjw\_\{j\}, and then encode the combined representation to the hidden dimensiond1d\_\{1\}using a learnable weight matrix𝐀g​a​t∈ℝd1×2​d1\\mathbf\{A\}\_\{gat\}\\in\\mathbb\{R\}^\{d\_\{1\}\\times 2d\_\{1\}\}, followed by a LeakyReLU activation\. This representation is then projected onto a learnable weight vector𝐚g​a​t∈ℝd1\\mathbf\{a\}\_\{gat\}\\in\\mathbb\{R\}^\{d\_\{1\}\}to yield the attention scorexq​jx\_\{qj\}\. Namely,

xq​j=𝐚g​a​t𝖳​LeakyReLU​\(𝐀g​a​t​\[𝐡q\(s−1\)∥𝐡j\(s−1\)\]\)x\_\{qj\}=\\mathbf\{a\}\_\{gat\}^\{\\mathsf\{T\}\}\\text\{LeakyReLU\}\\left\(\\mathbf\{A\}\_\{gat\}\\left\[\\mathbf\{h\}\_\{q\}^\{\(s\-1\)\}\\,\\\|\\,\\mathbf\{h\}\_\{j\}^\{\(s\-1\)\}\\right\]\\right\)\(8\)where𝐡q\(s−1\)\\mathbf\{h\}\_\{q\}^\{\(s\-1\)\}denotes the embedding ofwqw\_\{q\}at iterations−1s\-1and∥\\\|denotes the concatenation operation\. The attention scores are then normalized to derive the attention coefficientαq​j\\alpha\_\{qj\}:

αq​j=exq​j∑wq′∈𝒩​\(wj\)exq′​j\\alpha\_\{qj\}=\\frac\{e^\{x\_\{qj\}\}\}\{\\sum\_\{w\_\{q^\{\\prime\}\}\\in\\mathcal\{N\}\(w\_\{j\}\)\}e^\{x\_\{q^\{\\prime\}j\}\}\}\(9\)Next, we updatewjw\_\{j\}’s representation𝐡j\(s\)\\mathbf\{h\}\_\{j\}^\{\(s\)\}in iterationssby aggregating its own features with contextual messages from its topologically connected annotators, using a scaling factorλ0\\lambda\_\{0\}to stabilize training:

𝐡j\(s\)=𝐡j\(s−1\)\+λ0​∑wq′∈𝒩​\(wj\)αq′​j​𝐡q′\(s−1\)\\mathbf\{h\}\_\{j\}^\{\(s\)\}=\\mathbf\{h\}\_\{j\}^\{\(s\-1\)\}\+\\lambda\_\{0\}\\sum\_\{w\_\{q^\{\\prime\}\}\\in\\mathcal\{N\}\(w\_\{j\}\)\}\\alpha\_\{q^\{\\prime\}j\}\\mathbf\{h\}\_\{q^\{\\prime\}\}^\{\(s\-1\)\}\(10\)The updated annotator representation𝐡j\(s\)\\mathbf\{h\}\_\{j\}^\{\(s\)\}is then decoded to predict annotator\-specific confusion matrix𝐂\(s\)​j\\mathbf\{C\}^\{\(s\)j\}\. The decoder is implemented as a Multi\-Layer Perceptron \(MLP\) parameterized by two linear layers \(𝐀1∈ℝd1×d1\\mathbf\{A\}\_\{1\}\\in\\mathbb\{R\}^\{d\_\{1\}\\times d\_\{1\}\},𝐛1∈ℝd1\\mathbf\{b\}\_\{1\}\\in\\mathbb\{R\}^\{d\_\{1\}\}\), \(𝐀2∈ℝK2×d1\\mathbf\{A\}\_\{2\}\\in\\mathbb\{R\}^\{K^\{2\}\\times d\_\{1\}\},𝐛2∈ℝK2\\mathbf\{b\}\_\{2\}\\in\\mathbb\{R\}^\{K^\{2\}\}\), and a ReLU activation, projecting the latent embedding𝐡j\(s\)\\mathbf\{h\}\_\{j\}^\{\(s\)\}to the output𝐮j∈ℝK2\\mathbf\{u\}\_\{j\}\\in\\mathbb\{R\}^\{K^\{2\}\}as follows:

𝐮j=𝐀2​\(ReLU​\(𝐀1​𝐡j\(s\)\+𝐛1\)\)\+𝐛2\\mathbf\{u\}\_\{j\}=\\mathbf\{A\}\_\{2\}\\left\(\\text\{ReLU\}\(\\mathbf\{A\}\_\{1\}\\mathbf\{h\}\_\{j\}^\{\(s\)\}\+\\mathbf\{b\}\_\{1\}\)\\right\)\+\\mathbf\{b\}\_\{2\}\(11\)We reshape𝐮j\\mathbf\{u\}\_\{j\}into a matrix𝐌j∈ℝK×K\\mathbf\{M\}^\{j\}\\in\\mathbb\{R\}^\{K\\times K\}, and apply a row\-wise Softmax to obtain the predicted confusion matrix𝐂\(s\)​j\\mathbf\{C\}^\{\(s\)j\}:

𝐂k​l\(s\)​j=e𝐌k​lj∑l′=1Ke𝐌k​l′j\\mathbf\{C\}^\{\(s\)j\}\_\{kl\}=\\frac\{e^\{\\mathbf\{M\}^\{j\}\_\{kl\}\}\}\{\\sum\_\{l^\{\\prime\}=1\}^\{K\}e^\{\\mathbf\{M\}^\{j\}\_\{kl^\{\\prime\}\}\}\}\(12\)where𝐂k​l\(s\)​j\\mathbf\{C\}^\{\(s\)j\}\_\{kl\}denotes the probability thatwjw\_\{j\}labelsllwhile the truth iskk\. Our correlation\-aware annotator embeddings enable annotators with sparse labels to borrow statistical strength from the population, thereby stabilizing reliability estimation in scenarios where independent models fail\. Crucially, the predicted annotator confusion matrices𝐂\(s\)\\mathbf\{C\}^\{\(s\)\}provide a principled formulation for the label generation process, thereby effectively integrating the advantages of both confusion\-matrix\-based and representation\-learning\-based paradigms\. For efficient model training, we formulate a composite objective that integrates likelihood maximization with structural regularization and directional guidance\. The optimization process operates within an Expectation\-Maximization \(EM\) framework: refining posterior truth probabilities \(E\-step\) and updating the model parameters \(M\-step\)\. The total loss functionℒt​o​t​a​l\\mathcal\{L\}\_\{total\}comprises four components: \(1\) a negative log\-likelihood term to capture observation consistency; \(2\) a teacher\-student loss from high\-confidence annotators to guide the optimization trajectory; \(3\) a graph\-based contrastive loss to refine the training of annotator embeddings; \(4\) a regularization term to prevent degenerate solutions\.

### IV\-CNegative Log\-Likelihood Loss

We introduce the negative log\-likelihood lossℒN​l​l\\mathcal\{L\}\_\{Nll\}to optimize the network parameters by minimizing the expected negative log\-likelihood regarding the posterior truth probabilities\. Let𝐳^i​k=P​\(zi=k\|Y,𝐂\(s\)\)\\mathbf\{\\hat\{z\}\}\_\{ik\}=P\(z\_\{i\}=k\|Y,\\mathbf\{C\}^\{\(s\)\}\)denote the probability that classkkbeing the truth oftit\_\{i\},ℒN​l​l\\mathcal\{L\}\_\{Nll\}is defined as:

ℒN​l​l=−1T​∑i=1T∑wj∈𝒲i∑k=1K𝐳^i​k⋅log⁡𝐂k​yi​j\(s\)​j\\mathcal\{L\}\_\{Nll\}=\-\\frac\{1\}\{T\}\\sum\_\{i=1\}^\{T\}\\sum\_\{w\_\{j\}\\in\\mathcal\{W\}\_\{i\}\}\\sum\_\{k=1\}^\{K\}\\mathbf\{\\hat\{z\}\}\_\{ik\}\\cdot\\log\\mathbf\{C\}^\{\(s\)j\}\_\{ky\_\{ij\}\}\(13\)By parameterizing confusion matrices through a graph neural network rather than estimating them as independent variables, we introduce a shared representation that allows the model to generalize across annotators\. The rationale ofℒN​l​l\\mathcal\{L\}\_\{Nll\}is to optimize network parameters by maximizing the plausibility of observed crowdsourced labels conditioned on the updated posterior truth probabilities\. By quantifying the discrepancy between the predicted annotator confusion matrices and the actual annotator labels,ℒN​l​l\\mathcal\{L\}\_\{Nll\}effectively enforces a consistency constraint that aligns the predicted confusion matrices with the statistical consensus of the crowd\. Therefore, the network iteratively refines its parameters to fit the current posterior truth probabilities, thereby allowing the system to proceed toward an accurate aggregation of labels without the prerequisite of external supervision\.

### IV\-DTeacher\-Student Loss for High\-Confidence Annotators

Relying solely on internal consensus can cause the model to oscillate or converge to local optima\. Therefore, the teacher\-student LossℒT​e​a​c​h​e​r\\mathcal\{L\}\_\{Teacher\}is designed to guide the optimization trajectory by introducing reliable structural supervision from high\-confidence annotators\. By treating the empirical confusion matrices of high\-confidence annotators as highly reliable, we guide the model by minimizing the discrepancy between their predicted confusion matrices and the empirical counterparts\. For each annotatorwj∈𝒲c​o​n​fw\_\{j\}\\in\\mathcal\{W\}\_\{conf\}, given the predicted confusion matrix𝐂\(s\)​j\\mathbf\{C\}^\{\(s\)j\},ℒT​e​a​c​h​e​r\\mathcal\{L\}\_\{Teacher\}is defined as:

ℒT​e​a​c​h​e​r=1\|𝒲c​o​n​f\|​∑wj∈𝒲c​o​n​f‖𝐂\(t​e\)​j−𝐂\(s\)​j‖F2\\mathcal\{L\}\_\{Teacher\}=\\frac\{1\}\{\|\\mathcal\{W\}\_\{conf\}\|\}\\sum\_\{w\_\{j\}\\in\\mathcal\{W\}\_\{conf\}\}\\left\\\|\\mathbf\{C\}^\{\(te\)j\}\-\\mathbf\{C\}^\{\(s\)j\}\\right\\\|\_\{F\}^\{2\}\(14\)This objective offers two distinct benefits\. First, by minimizing the mean squared error between the predicted and empirical confusion matrices of high\-confidence annotators,ℒT​e​a​c​h​e​r\\mathcal\{L\}\_\{Teacher\}injects strong supervision into the network, effectively mitigating the identifiability issues inherent in unsupervised EM\. Second, the extensive task coverage of high\-confidence annotators endows them with dense connectivity with other annotators within the topology, which facilitates annotator reliability propagation and benefit annotators with sparse data\.

### IV\-EGraph\-Based Contrastive Loss

To enforce discriminative representation learning, we introduce the graph\-based contrastive lossℒC​t​r\\mathcal\{L\}\_\{Ctr\}, which explicitly translates the structural positive and negative annotator pairs into strict geometric constraints within the embedding space\. Formally,ℒC​t​r\\mathcal\{L\}\_\{Ctr\}is defined as follows:

ℒC​t​r=−1\|ℰ\+\|​∑\(wj,wq\)∈ℰ\+log⁡er​\(𝐡j\(s\),𝐡q\(s\)\)τer​\(𝐡j\(s\),𝐡q\(s\)\)τ\+∑wq′∈ℰ−jer​\(𝐡j\(s\),𝐡q′\(s\)\)τ\\mathcal\{L\}\_\{Ctr\}=\\frac\{\-1\}\{\|\\mathcal\{E\}\_\{\+\}\|\}\\sum\_\{\(w\_\{j\},w\_\{q\}\)\\in\\mathcal\{E\}\_\{\+\}\}\\log\\frac\{e^\{\\frac\{r\(\\mathbf\{h\}\_\{j\}^\{\(s\)\},\\mathbf\{h\}\_\{q\}^\{\(s\)\}\)\}\{\\tau\}\}\}\{e^\{\\frac\{r\(\\mathbf\{h\}\_\{j\}^\{\(s\)\},\\mathbf\{h\}\_\{q\}^\{\(s\)\}\)\}\{\\tau\}\}\+\\sum\\limits\_\{w\_\{q^\{\\prime\}\}\\in\\mathcal\{E\}\_\{\-\}^\{j\}\}e^\{\\frac\{r\(\\mathbf\{h\}\_\{j\}^\{\(s\)\},\\mathbf\{h\}\_\{q^\{\\prime\}\}^\{\(s\)\}\)\}\{\\tau\}\}\}\(15\)whereτ\\tauis a temperature parameter andrrdenotes the similarity score function that measures the similarity between node embeddings, defined as:

r​\(𝐡j\(s\),𝐡q\(s\)\)=\(𝐡j\(s\)\)⋅\(𝐡q\(s\)\)𝖳‖𝐡j\(s\)‖⋅‖𝐡q\(s\)‖r\(\\mathbf\{h\}\_\{j\}^\{\(s\)\},\\mathbf\{h\}\_\{q\}^\{\(s\)\}\)=\\frac\{\(\\mathbf\{h\}\_\{j\}^\{\(s\)\}\)\\cdot\(\\mathbf\{h\}\_\{q\}^\{\(s\)\}\)^\{\\mathsf\{T\}\}\}\{\\\|\\mathbf\{h\}\_\{j\}^\{\(s\)\}\\\|\\cdot\\\|\\mathbf\{h\}\_\{q\}^\{\(s\)\}\\\|\}\(16\)By pulling positive annotator pairs together while repelling negative ones apart,ℒC​t​r\\mathcal\{L\}\_\{Ctr\}prevents feature collapse and ensures a clear separation between reliable and unreliable annotators\. The efficacy of this formulation lies in hard negative mining\. The temperature\-scaled softmax denominator assigns significantly larger gradients to the most difficult negative pairs, directing the optimization focus toward resolving difficult ambiguities rather than trivial samples\. Furthermore, this module serves as a structural regularizer that maximizes the mutual information between the representations of positive annotator pairs, yielding robust, separable embeddings that generalize effectively to unseen data\.

### IV\-FRegularization Loss

Finally, to prevent convergence to degenerate solutions, we introduce a weak prior on annotator reliability by defining a regularization termℒR​e​g\\mathcal\{L\}\_\{Reg\}that encourages confusion matrices to be close to the identity matrix𝐈\\mathbf\{I\}:

ℒR​e​g=1W​∑j=1W‖𝐂\(s\)​j−𝐈‖F2\\mathcal\{L\}\_\{Reg\}=\\frac\{1\}\{W\}\\sum\_\{j=1\}^\{W\}\\left\\\|\\mathbf\{C\}^\{\(s\)j\}\-\\mathbf\{I\}\\right\\\|\_\{F\}^\{2\}\(17\)The primary role ofℒR​e​g\\mathcal\{L\}\_\{Reg\}is to impose a diagonal dominance constraint on confusion matrices\. By grounding the optimization in the prior assumption that annotators are generally reliable, this regularization ensures that the learned confusion matrices deviate from the diagonal only when data\-driven evidence is strong enough to override the penalty, thereby capturing genuine annotator bias rather than drifting due to the inherent instability of unsupervised learning\.

### IV\-GTruth Estimation

With the above four loss terms established, each tailored to capture a specific aspect of the learning process and provide complementary supervision signals, the total objective functionℒt​o​t​a​l\\mathcal\{L\}\_\{total\}is a weighted combination of these components:

ℒt​o​t​a​l=λ1​ℒN​l​l\+λ2​ℒC​t​r\+λ3​ℒT​e​a​c​h​e​r\+λ4​ℒR​e​g\\mathcal\{L\}\_\{total\}=\\lambda\_\{1\}\\mathcal\{L\}\_\{Nll\}\+\\lambda\_\{2\}\\mathcal\{L\}\_\{Ctr\}\+\\lambda\_\{3\}\\mathcal\{L\}\_\{Teacher\}\+\\lambda\_\{4\}\\mathcal\{L\}\_\{Reg\}\(18\)whereλ1\\lambda\_\{1\},λ2\\lambda\_\{2\},λ3\\lambda\_\{3\}, andλ4\\lambda\_\{4\}are loss weights that control the relative importance of different supervision signals\.ℒt​o​t​a​l\\mathcal\{L\}\_\{total\}balances data likelihood with structural constraints and prior knowledge\. We optimize this objective using the Adam optimizer\.

Based on the predicted annotator\-specific confusion matrices, we update the posterior truth probabilities as follows:

𝐳^i​k∝exp⁡\(∑wj∈𝒲ilog⁡\(𝐂k​yi​j\(s\)​j\)\+λ5​log⁡𝝅k\)\\mathbf\{\\hat\{z\}\}\_\{ik\}\\propto\\exp\\left\(\\sum\_\{w\_\{j\}\\in\\mathcal\{W\}\_\{i\}\}\\log\(\\mathbf\{C\}\_\{ky\_\{ij\}\}^\{\(s\)j\}\)\+\\lambda\_\{5\}\\log\\boldsymbol\{\\pi\}\_\{k\}\\right\)\(19\)whereλ5\\lambda\_\{5\}is a scaling factor to the prior term\. Finally, we treat the label with the highest probability as the truth\.

Input :Label set

YY, epochs

SS, learning rate

η\\eta, hidden layer dimension

d1d\_\{1\}, temperature

τ\\tau, loss weights

λ1,λ2,λ3,λ4\\lambda\_\{1\},\\lambda\_\{2\},\\lambda\_\{3\},\\lambda\_\{4\}, thresholds

ρ,δ0,δ1,δ2,δ3\\rho,\\delta\_\{0\},\\delta\_\{1\},\\delta\_\{2\},\\delta\_\{3\}, and scaling factors

λ0\\lambda\_\{0\},

λ5\\lambda\_\{5\}\.

1Generate cross\-annotator topology

𝒢=\{\{wj\}j=1W,ℰ\}\\mathcal\{G\}=\\\{\\\{w\_\{j\}\\\}\_\{j=1\}^\{W\},\\mathcal\{E\}\\\};

2Generate pos/neg annotator pairs

ℰ\+\\mathcal\{E\}\_\{\+\},

ℰ−\\mathcal\{E\}\_\{\-\}using Eq\. \([1](https://arxiv.org/html/2607.18465#S4.E1)\);

3Generate teacher/initial confusion matrices

𝐂\(t​e\)\\mathbf\{C\}^\{\(te\)\},

𝐂\(0\)\\mathbf\{C\}^\{\(0\)\}using Eq\. \([3](https://arxiv.org/html/2607.18465#S4.E3)\) and Eq\. \([4](https://arxiv.org/html/2607.18465#S4.E4)\);

4Generate prior truth distribution

𝝅\\boldsymbol\{\\pi\}using Eq\. \([6](https://arxiv.org/html/2607.18465#S4.E6)\);

5for*epochs=1s=1toSS*do

6Perform forward pass:

𝐡\(s\),𝐂\(s\)=GNN​\(𝒢,𝐂\(0\);Θ\)\\mathbf\{h\}^\{\(s\)\},\\mathbf\{C\}^\{\(s\)\}=\\text\{GNN\}\(\\mathcal\{G\},\\mathbf\{C\}^\{\(0\)\};\\Theta\)using Eq\. \([10](https://arxiv.org/html/2607.18465#S4.E10)\) and Eq\. \([12](https://arxiv.org/html/2607.18465#S4.E12)\), respectively;

7

8Calculate

ℒN​l​l\\mathcal\{L\}\_\{Nll\}using Eq\. \([13](https://arxiv.org/html/2607.18465#S4.E13)\);

9Calculate

ℒT​e​a​c​h​e​r\\mathcal\{L\}\_\{Teacher\}using Eq\. \([14](https://arxiv.org/html/2607.18465#S4.E14)\);

10Calculate

ℒC​t​r\\mathcal\{L\}\_\{Ctr\}using Eq\. \([15](https://arxiv.org/html/2607.18465#S4.E15)\);

11Calculate

ℒR​e​g\\mathcal\{L\}\_\{Reg\}using Eq\. \([17](https://arxiv.org/html/2607.18465#S4.E17)\);

12Calculate the final loss

ℒt​o​t​a​l\\mathcal\{L\}\_\{total\}using Eq\. \([18](https://arxiv.org/html/2607.18465#S4.E18)\);

13Update

Θ\\Thetaby minimizing

ℒt​o​t​a​l\\mathcal\{L\}\_\{total\}using Adam optimizer;

14Update truth probabilities

Z^\\hat\{Z\}using Eq\. \([19](https://arxiv.org/html/2607.18465#S4.E19)\);

15

Output :

Z^\\hat\{Z\}\.

Algorithm 1AHEAD Training AlgorithmTABLE I:Dataset statistics\.M​i​nwMin\_\{w\}andM​e​dwMed\_\{w\}denote the minimum and median numbers of tasks labeled across annotators\.DatasetTTWWKKM​i​nwMin\_\{w\}M​e​dwMed\_\{w\}\#LabelsTruth distributionDomainVal710038720201,000\{0\.08,0\.21,0\.18,0\.10,0\.23,0\.19,0\.01\}SentimentAircr59350614\.51,588\{0\.17,0\.17,0\.13,0\.17,0\.17,0\.19\}ImageCF3004615121,720\{0\.19,0\.23,0\.24,0\.31,0\.03\}SentimentMS70044102162,945\{0\.11,0\.10,0\.10,0\.09,0\.10,0\.10,0\.10,0\.11,0\.11,0\.09\}AudioDog80710941388,070\{0\.21,0\.23,0\.27,0\.29\}ImageFace5842744565,242\{0\.25,0\.25,0\.25,0\.25\}ImageAdult11,040825411889,799\{0\.56,0\.18,0\.11,0\.15\}LevelVal510038520201,000\{0\.13,0\.27,0\.23,0\.28,0\.09\}SentimentWeb2,665177511915,567\{0\.12,0\.18,0\.26,0\.20,0\.23\}RelevanceLabelMe1,0005983272,547\{0\.09,0\.12,0\.13,0\.11,0\.14,0\.13,0\.13,0\.15\}ImageSenti98,9801,9605845569,274\{0\.23,0\.25,0\.19,0\.31,0\.02\}Sentiment
### IV\-HAlgorithm

LetΘ\\Thetainclude all trainable parameters in our graph neural network\. Algorithm[1](https://arxiv.org/html/2607.18465#alg1)outlines the procedure of AHEAD\. The algorithm operates in an iterative Expectation\-Maximization \(EM\) manner\. In the initialization phase, we construct an cross\-annotator topology𝒢\\mathcal\{G\}from their interactions, generate teacher confusion matrices for high\-confidence annotators, calculate initial confusion matrices for all annotators, and derive prior truth distribution\. During each training epoch, we run a graph neural network to learn cross\-annotator attention coefficients over their topology structure, deriving comprehensive annotator embeddings by aggregating each annotator’s own representation with contextual messages from its connected annotators\. These embeddings are subsequently decoded into interpretable annotator\-specific confusion matrices to fit the observed labels\. The model parametersΘ\\Thetaare updated by minimizing a composite lossℒt​o​t​a​l\\mathcal\{L\}\_\{total\}, which consists of the negative log\-likelihood, a teacher\-student loss for high\-confidence annotators, a graph\-based contrastive learning loss, and a regularization term\. Then, the posterior truth probabilitiesZ^\\hat\{Z\}are updated based on the predicted confusion matrices\.

AHEAD combines both the strengths of representation learning and confusion\-matrix\-based modeling\. By modeling dense cross\-annotator learning over their interaction topology, the model effectively captures rich, intrinsic cross\-annotator correlations and propagates annotator reliability across the network\. This correlation\-aware annotator modeling enables annotators with sparse labels to leverage population\-level information\. Meanwhile, the decoded confusion matrices provide a theoretically interpretable fit of the label generation, ensuring interpretable cross\-annotator learning and effective model training\.

## VExperiments

### V\-AExperimental Setup

Datasets\.We use 11 widely adopted real\-world datasets spanning five domains for evaluation\. Table[I](https://arxiv.org/html/2607.18465#S4.T1)summarizes their statistics\. Among them, CF, MS, Dog, Face, Adult, Web, and Senti are also used in\[[27](https://arxiv.org/html/2607.18465#bib.bib5)\]; Val5, Val7, and Aircr are also used in\[[45](https://arxiv.org/html/2607.18465#bib.bib14)\]; and LabelMe also appears in\[[30](https://arxiv.org/html/2607.18465#bib.bib34)\]\. Notably, the first 10 datasets are used to evaluate overall performance, whereas the last dataset, Senti, being the largest publicly available dataset, is used to evaluate scalability\.

Metrics\.We evaluate the results ofAccuracyandMacro\-F1\.

Implementation details\.We setδ0=4\\delta\_\{0\}=4,δ1=0\.5\\delta\_\{1\}=0\.5,δ2=0\.4\\delta\_\{2\}=0\.4,δ3=0\.2\\delta\_\{3\}=0\.2,ρ=30\\rho=30,S=100S=100,η=0\.01\\eta=0\.01,d1=128d\_\{1\}=128,τ=0\.2\\tau=0\.2,λ0=0\.6\\lambda\_\{0\}=0\.6,λ1=1\\lambda\_\{1\}=1,λ2=1\\lambda\_\{2\}=1,λ3=6\\lambda\_\{3\}=6, andλ4=0\.5\\lambda\_\{4\}=0\.5\. Forλ5\\lambda\_\{5\}, we setλ5=1\\lambda\_\{5\}=1during the first 20% of the epochs, andλ5=0\.00001​T\\lambda\_\{5\}=0\.00001Tfor the remaining epochs\.

Baselines\.We compare AHEAD with foursingle\-parameter\-based\(MV, CATD\[[23](https://arxiv.org/html/2607.18465#bib.bib12)\], GLAD\[[43](https://arxiv.org/html/2607.18465#bib.bib11)\], BWA\[[27](https://arxiv.org/html/2607.18465#bib.bib5)\]\), fourconfusion\-matrix\-based\(DS\[[10](https://arxiv.org/html/2607.18465#bib.bib6)\], IBCC\[[20](https://arxiv.org/html/2607.18465#bib.bib7)\], CBCC\[[42](https://arxiv.org/html/2607.18465#bib.bib9)\], EBCC\[[26](https://arxiv.org/html/2607.18465#bib.bib3)\]\), and threerepresentation\-learning\-based\(TiReMGE\[[44](https://arxiv.org/html/2607.18465#bib.bib8)\], GOVERN\[[30](https://arxiv.org/html/2607.18465#bib.bib34)\], CrowdFM\[[29](https://arxiv.org/html/2607.18465#bib.bib35)\]\) methods\. Their detailed descriptions can be found in Section[II](https://arxiv.org/html/2607.18465#S2)\.

All experiments were conducted on a server with 2 vCPUs and 8 GB of memory, and repeated 10 times\.

TABLE II:Performance results across different datasets and methods\.
### V\-BEffectiveness

Table[II](https://arxiv.org/html/2607.18465#S5.T2)presents theAccuracyandMacro\-F1results of 12 methods across 10 real\-world datasets\. Figure[3](https://arxiv.org/html/2607.18465#S5.F3)compares the class\-wise accuracy of AHEAD with representative confusion\-matrix\-based methods \(DS and IBCC\) and representation\-learning\-based methods \(TiReMGE, GOVERN, and CrowdFM\)\.

Table[II](https://arxiv.org/html/2607.18465#S5.T2)demonstrates that AHEAD consistently secures top\-tier performance across all datasets, improving the average accuracy and Macro\-F1 score from 68\.75% to 73\.23%, and from 66\.41% to 70\.78%, respectively\. Meanwhile, the radar charts in Figure[3](https://arxiv.org/html/2607.18465#S5.F3)underscore the remarkable robustness of AHEAD in maintaining high accuracy across diverse classes\.

#### V\-B1Handling Data Sparsity

As shown in Table[I](https://arxiv.org/html/2607.18465#S4.T1), the CF and Aircr datasets are extremely sparse, with median label counts \(M​e​dwMed\_\{w\}\) of only 2 and 4, respectively, posing the trade\-off between fine\-grained annotator modeling and robust training\. As shown in Table[II](https://arxiv.org/html/2607.18465#S5.T2), confusion\-matrix\-based methods DS, IBCC, and CBCC suffer substantial performance degradation on these datasets\. Specifically, DS ranks last on CF \(79\.67%\), while IBCC and CBCC rank last on Aircr \(78\.08% and 76\.56%\)\. However, they fail for different underlying reasons\. The poor performance of DS on CF mainly stems from its simple annotator reliability estimation via maximum likelihood\. In contrast, the degradation of IBCC and CBCC on Aircr is primarily due to their limited robustness to data sparsity and class imbalance: estimating a fine\-grainedK×KK\\times Kconfusion matrix for an annotator with fewer thanKKanswered tasks is ill\-posed, leading to overfitting and high variance\. Notably, DS exhibits strong baseline generalization but ranks last on the CF dataset, highlighting the inherent difficulty of balancing models’ expressive capacity with robustness to both data sparsity\. By contrast, our AHEAD achieves top\-tier accuracy across both datasets, as further evidenced in Figure[3](https://arxiv.org/html/2607.18465#S5.F3)\-even when certain classes exhibit near\-zero proportions\.

![Refer to caption](https://arxiv.org/html/2607.18465v1/x4.png)Figure 3:Class\-wise accuracy \(%\) comparison of AHEAD with representative confusion\-matrix\-based methods \(DS, IBCC\) and representation\-learning\-based methods \(TiReMGE, GOVERN, CrowdFM\)\.![Refer to caption](https://arxiv.org/html/2607.18465v1/x5.png)\(a\)Val5
![Refer to caption](https://arxiv.org/html/2607.18465v1/x6.png)\(b\)MS

Figure 4:Cross\-annotator correlations learned for Val5 and MS, where reliability propagates via the interaction topology\.Meanwhile, the representation\-learning\-based methods GOVERN, TiReMGE, and CrowdFM achieve competitive performance, suggesting the advantages of representation learning for capturing complex relationships\. However, they are also sensitive to data sparsity; for instance, GOVERN performs worse on the Adult dataset, while TiReMGE shows inferior performance on the Web and Aircr datasets, which can be attributed to insufficient training data\.

Overall, experimental results demonstrate that our cross\-annotator learning effectively overcomes imprecise annotator estimation by leveraging the statistical strength of the entire annotator population rather than relying on individual data, yielding significantly better inference where prior methods fail\.

#### V\-B2Handling Imbalanced Truth Distribution

As presented in Table[I](https://arxiv.org/html/2607.18465#S4.T1), the real\-world datasets exhibit substantial diversity in class distributions, ranging from perfectly balanced scenarios \(e\.g\., Face\) to those characterized by severe imbalance \(e\.g\., Val5, Val7, Adult, and Web\)\. Notably, DS, IBCC, TiReMGE, and GOVERN display a marked sensitivity to class imbalance\. This weakness manifests as a geometric collapse in Figure[3](https://arxiv.org/html/2607.18465#S5.F3), particularly on datasets characterized by dominant or long\-tail classes\. The Val7 and Val5 datasets pose a dual challenge of severe class imbalance and extreme task difficulty, under which existing methods struggle significantly, with accuracies capped at 22% and 41%, respectively\. Figure[3](https://arxiv.org/html/2607.18465#S5.F3)shows that on Val7, DS, IBCC, and TiReMGE collapse on four out of seven classes, and GOVERN collapses on six out of seven classes; while on Val5, DS, IBCC, and TiReMGE collapse on the majority class \(class 4\), and GOVERN collapses on the largest two classes \(class 4 and class 2\), resulting in highly skewed class\-wise performance\. In contrast, by explicitly modeling cross\-annotator learning, AHEAD boosts accuracy on Val7 and Val5 to 36\.90% and 49\.40%, respectively, yielding improvements of 14\.9% and 8\.4%, respectively, highlighting AHEAD’s robustness against class imbalance and strong capability in complex tasks\.

Moreover, on the Adult dataset \(where the majority class constitutes 56%\) and the Web dataset \(where the minority and majority classes account for 12% and 26%, respectively\), DS and IBCC tend to overfit the dominant classes \(class 1 for Adult, and class 3 for Web\), resulting in a skewed polygon shape that compromises the recall of other classes\. Meanwhile, GOVERN collapses on minority classes \(class 2 and class 3\)\. By contrast, our AHEAD yields remarkable class\-wise accuracy on these datasets, while preserving outstanding performance on balanced datasets like Face and Dog\.

In summary, these experimental results demonstrate that AHEAD consistently maintains outstanding performance across classes\.

### V\-CCase Studies on Val5 and MS

Figure[4](https://arxiv.org/html/2607.18465#S5.F4)illustrates the cross\-annotator attention coefficients that we learn from Val5 and MS, respectively\.

Table[I](https://arxiv.org/html/2607.18465#S4.T1)reveals a distinct characteristic of Val5: the classes are imbalanced, and the number of tasks labeled per annotator follows an almost uniform distribution, with nearly every annotator labeling exactly 20 tasks\. Meanwhile, Table[II](https://arxiv.org/html/2607.18465#S5.T2)indicates that the annotations in this dataset are unreliable, as reflected by the consistently poor performance of all baselines, i\.e\., MV achieves an accuracy of only 35\.17%; although DS reaches 41%, it collapses on the majority class \(class 4\)\. This scenario presents a particular challenge in distinguishing between the reliable and unreliable annotators\. To address this, AHEAD’s cross\-annotator attention coefficients learned on Val5 form a dense topology, which facilitates reliable annotator estimation by effectively propagating information across the annotator population, as illustrated in Figure[4](https://arxiv.org/html/2607.18465#S5.F4)\(a\)\. For instance, although there is no direct connection betweenw32w\_\{32\}and3636, their correlation is propagated via intermediate annotatorsw1w\_\{1\}andw35w\_\{35\}\. This mechanism enables AHEAD to effectively distinguish between reliable and unreliable annotators\. As a result, AHEAD substantially improves the accuracy to 49\.40%\.

![Refer to caption](https://arxiv.org/html/2607.18465v1/x7.png)
![Refer to caption](https://arxiv.org/html/2607.18465v1/x8.png)
![Refer to caption](https://arxiv.org/html/2607.18465v1/x9.png)
![Refer to caption](https://arxiv.org/html/2607.18465v1/x10.png)

Figure 5:Effect of differentλ\\lambdavalues on the model performance\.![Refer to caption](https://arxiv.org/html/2607.18465v1/x11.png)Figure 6:Effect of differentρ\\rhovalues on the model performance\.For the MS dataset, as shown in Table[I](https://arxiv.org/html/2607.18465#S4.T1), while the class distribution is uniform, the number of tasks labeled per annotator follows a long\-tail distribution: most annotators label only a small subset of tasks\. This scenario poses the challenge of balancing model’s complexity in discriminative capability with robustness to data sparsity\. Table[II](https://arxiv.org/html/2607.18465#S5.T2)shows that the top\-performing baselines are EBCC \(78\.71%\) and CATD \(78\.69%\)\. Meanwhile, representation\-learning\-based methods perform mediocrely due to the insufficient training data\. As depicted in Figure[4](https://arxiv.org/html/2607.18465#S5.F4)\(b\), the cross\-annotator attention coefficients learned by AHEAD reveal a deep, structured topology characterized by distinct clusters and stronger edge weights among specific nodes\. By assigning high correlation weights to reliable sub\-communities, AHEAD refines the aggregation process, achieving a state\-of\-the\-art accuracy of 79\.79%\.

The contrast between these topologies indicates that AHEAD is not limited to learn a static correlation pattern\. Instead, it dynamically adapts its correlation structure, ranging from dense coupling to structured clustering to enable a robust performance\.

### V\-DAblation Studies

#### V\-D1Effect of Loss Components

To examine the contribution of each loss component to model performance, we perform a sensitivity analysis regarding the hyperparametersλ1\\lambda\_\{1\},λ2\\lambda\_\{2\},λ3\\lambda\_\{3\}, andλ4\\lambda\_\{4\}, which weigh the negative log\-likelihood loss \(ℒN​l​l\\mathcal\{L\}\_\{Nll\}\), contrastive loss \(ℒC​t​r\\mathcal\{L\}\_\{Ctr\}\), teacher\-student loss \(ℒT​e​a​c​h​e​r\\mathcal\{L\}\_\{Teacher\}\), and regularization loss \(ℒR​e​g\\mathcal\{L\}\_\{Reg\}\), respectively\. As shown in Figure[5](https://arxiv.org/html/2607.18465#S5.F5), we vary one hyperparameter at a time while fixing the others at their optimal values\.

For the negative log\-likelihood loss weightλ1\\lambda\_\{1\}, the performance curve exhibits a clear inverted\-V shape, peaking atλ1=1\.0\\lambda\_\{1\}=1\.0; lower values lead to insufficient model training, whereas larger weights \(\>1\.0\>1\.0\) encourage overfitting to noisy pseudo\-labels\. The contrastive loss weightλ2\\lambda\_\{2\}has a relatively stable impact, with a local optimum aroundλ2=1\.0\\lambda\_\{2\}=1\.0, suggesting that while feature alignment is beneficial, the model is robust to minor variations in this constraint\. In contrast, the teacher\-student loss weightλ3\\lambda\_\{3\}demonstrates a significant positive correlation with accuracy up to a threshold, reaching its optimal aroundλ3=6\.0\\lambda\_\{3\}=6\.0\. This large optimal value underscores the importance of leveraging supervisory signals from high\-confidence annotators in mitigating bias and guiding optimization trajectory\. Finally, the regularization loss weightλ4\\lambda\_\{4\}exhibits only minor variation within the range\[0\.1,0\.7\]\[0\.1,0\.7\], suggesting that while constraining the confusion matrix is important for stability, its insensitive to precise tuning\.

Overall, these findings confirm the necessity of each component, with the teacher\-student lossℒT​e​a​c​h​e​r\\mathcal\{L\}\_\{Teacher\}from high\-confidence annotators playing a dominant role in enhancing robustness\. Meanwhile, the negative log\-likelihood lossℒN​l​l\\mathcal\{L\}\_\{Nll\}ensures faithful fitting to observed labels, and the contrastive lossℒC​t​r\\mathcal\{L\}\_\{Ctr\}promotes meaningful feature alignment among annotators\. In addition, the regularization lossℒR​e​g\\mathcal\{L\}\_\{Reg\}stabilizes the learning of confusion matrices and prevents degenerate solutions\. Together, these components form a complementary objective that is critical to the effectiveness of AHEAD\.

TABLE III:Runtime \(Sec\) across different datasets and methods\.TABLE IV:Comparison of accuracy \(%\), Macro\-F1 \(%\), and runtime \(Sec\) on the largest dataset, Senti\. OOM denotes an out\-of\-memory error on a GPU with 70GB of memory\.MetricMVCATDGLADBWADSIBCCCBCCEBCCTiReMGEGOVERNCrowdFMAHEADAccu↑\\uparrow88\.26±\\pm0\.1588\.30±\\pm0\.0689\.40±\\pm0\.0089\.00±\\pm0\.0082\.40±\\pm0\.0085\.90±\\pm0\.0078\.70±\\pm0\.0086\.01±\\pm0\.1490\.35±\\pm0\.09OOM88\.43±\\pm0\.7788\.49±\\pm0\.21F1↑\\uparrow78\.10±\\pm0\.3578\.71±\\pm0\.4179\.40±\\pm0\.0077\.21±\\pm0\.0073\.70±\\pm0\.0075\.94±\\pm0\.0072\.43±\\pm0\.0073\.41±\\pm0\.4878\.99±\\pm1\.67OOM77\.64±\\pm1\.1178\.20±\\pm0\.46Time↓\\downarrow0\.22785\.29616721\.3233\.22476\.66435\.833239\.63110651\.5247988\.357OOM37\.5601151\.742
#### V\-D2Effect of the Proportion of High\-confidence Annotators\.

Figure[6](https://arxiv.org/html/2607.18465#S5.F6)illustrates the effect of differentρ\\rhovalues on the model performance\. As shown in Figure[6](https://arxiv.org/html/2607.18465#S5.F6), the model performance is relatively stable across different values ofρ\\rho\. Both Accuracy and Macro\-F1 improve asρ\\rhoincreases from small values, reach their best levels aroundρ=20​30\\rho=20~30and then remain largely unchanged with only minor fluctuations\. This suggests that the model is not highly sensitive toρ\\rho, and that a moderate value ofρ\\rhois sufficient to achieve strong and robust performance\.

### V\-EEfficiency

Table[III](https://arxiv.org/html/2607.18465#S5.T3)compares the runtime of various methods\. Generally, the runtime of all methods increases with the number of labels\. As table[III](https://arxiv.org/html/2607.18465#S5.T3)turns out, MV consistently ranks first across datasets due to its simplicity, followed by BWA\. The efficiency of BWA can be attributed to its single\-parameter\-based Bayesian formulation, which enables direct inference using the expectation\-maximization framework\. Notably, CrowdFM, which acquires transferable knowledge and can be directly deployed on new datasets, demonstrates runtime second only to MV and BWA\. Meanwhile, as the simplest confusion matrix\-based method, DS exhibits comparable performance to the single\-parameter\-based method CATD\.

Meanwhile, while showing the clearly bestAccuracyandMacro\-F1, our AHEAD also exhibits competitive efficiency, e\.g\., it is orders of magnitude faster than GLAD, TiReMGE, and EBCC\. As Table[III](https://arxiv.org/html/2607.18465#S5.T3)shows, on the Adult dataset, AHEAD requires approximately10210^\{2\}seconds, whereas GLAD and TiReMGE exceed10310^\{3\}seconds\. Furthermore, AHEAD maintains a runtime comparable to IBCC and CBCC, suggesting that it achieves an effective balance between model complexity and computational efficiency\.

GLAD is the least efficient method, likely due to its explicit modeling of task difficulty and reliance on complex gradient\-based optimization\. Meanwhile, representation\-learning\-based methods TiReMGE and GOVERN incur substantial computational overhead due to complex network training and high\-dimensional optimization of complex annotator\-task dependencies, since the number of tasks can be several orders of magnitude larger than the number of annotators\.

In general, methods that model annotators based on confusion matrices or representation learning require longer training time due to their elaborate parameterization and inference procedures, and our AHEAD achieves an effective balance between model complexity and computational efficiency\.

### V\-FScalability

To further evaluate the scalability of all methods, we conduct experiments on the largest publicly available dataset,Senti\. Table[IV](https://arxiv.org/html/2607.18465#S5.T4)reports the performance results of all methods, demonstrating that GOVERN fails to run due to an out\-of\-memory error on a GPU with 70GB of memory, a limitation that has also been confirmed by its authors\.

Table[IV](https://arxiv.org/html/2607.18465#S5.T4)demonstrates that our AHEAD achieves a strong balance between effectiveness and efficiency on Senti\. Notably, although single\-parameter\-based methods such as MV and BWA are faster, AHEAD significantly outperforms them in predictive performance\. At the same time, compared with more sophisticated probabilistic methods such as GLAD, EBCC, and TiReMGE, AHEAD achieves comparable or better performance with substantially lower runtime\. This shows that AHEAD scales effectively to large\-scale crowdsourcing data\. Moreover, the foundation model CrowdFM also achieves outstanding performance due to its cross\-dataset generalization mechanism, which can be directly deployed on new datasets\.

Overall, the results demonstrate that AHEAD offers a desirable efficiency–effectiveness trade\-off in large\-scale settings, making it a practical and scalable choice for real\-world crowdsourcing applications\.

## VIConclusion

We identify a fundamental bottleneck in state\-of\-the\-art multi\-class label aggregation: unstable annotator reliability estimation incurred by data sparsity, i\.e\., most individual annotators only label a small subset of tasks\. In response, we propose cross\-annotator learning via representation learning and unify it with interpretable confusion matrices in an end\-to\-end framework\. Specifically, we explicitly model cross\-annotator learning over the extracted dense cross\-annotator contexts, learning population\-level annotator embeddings by aggregating individual\-level annotator features with their contextual information\. Then, we map these embeddings into interpretable confusion matrices to fit the observed labels\. We formulate a composite objective that integrates likelihood maximization with structural regularization and directional guidance to facilitate effective training, where high\-confidence annotators are proposed to alleviate the unsupervised training issues faced by existing models\. Extensive experiments on 10 datasets with diverse characteristics demonstrate that AHEAD substantially improves the accuracy of multi\-class label aggregation, achieving robust performance where prior methods fail\. Meanwhile, the scalability experiments on the largest dataset demonstrate that AHEAD offers a desirable efficiency–effectiveness trade\-off in large\-scale settings\.

## AI\-GENERATED CONTENT ACKNOWLEDGEMENT

The authors used ChatGPT solely for proofreading and minor linguistic refinement\. No part of the technical content, experimental design, or analysis was generated by ChatGPT or other AI tools\.

## References

- \[1\]B\. I\. Aydin, Y\. S\. Yilmaz, Y\. Li, Q\. Li, J\. Gao, and M\. Demirbas\(2014\)Crowdsourcing for multiple\-choice question answering\.InProceedings of the AAAI Conference on Artificial Intelligence,pp\. 2946–2953\.Cited by:[§II\-A](https://arxiv.org/html/2607.18465#S2.SS1.p1.1)\.
- \[2\]J\. Bai, J\. Gui, T\. Wang, H\. Song, A\. Liu, and N\. N\. Xiong\(2025\)ETBP\-TD: an efficient and trusted bilateral privacy\-preserving truth discovery scheme for mobile crowdsensing\.IEEE Transactions on Mobile Computing24\(3\),pp\. 2203–2219\.Cited by:[§II](https://arxiv.org/html/2607.18465#S2.p1.1)\.
- \[3\]T\. Bonald and R\. Combes\(2017\)A minimax optimal algorithm for crowdsourcing\.InProceedings of the Annual Conference on Neural Information Processing Systems,pp\. 4352–4360\.Cited by:[§II\-A](https://arxiv.org/html/2607.18465#S2.SS1.p1.1),[§II](https://arxiv.org/html/2607.18465#S2.p2.1)\.
- \[4\]E\. Cao, D\. Wang, J\. Huang, and W\. Hu\(2020\)Open knowledge enrichment for long\-tail entities\.InProceedings of the World Wide Web conference,pp\. 384–394\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p2.1),[§II\-A](https://arxiv.org/html/2607.18465#S2.SS1.p1.1),[§II](https://arxiv.org/html/2607.18465#S2.p2.1)\.
- \[5\]C\. Chai, J\. Fan, G\. Li, J\. Wang, and Y\. Zheng\(2019\)Crowdsourcing database systems: overview and challenges\.InProceedings of the International Conference on Data Engineering,pp\. 2052–2055\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p1.1)\.
- \[6\]J\. Chen, J\. Feng, S\. Zhang, X\. Li, and H\. Djigal\(2025\)Robust annotation aggregation in crowdsourcing via enhanced worker ability modeling\.Information Processing & Management62\(1\),pp\. 103914\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p2.1),[§I](https://arxiv.org/html/2607.18465#S1.p3.1),[§II\-B](https://arxiv.org/html/2607.18465#S2.SS2.p1.1),[§II](https://arxiv.org/html/2607.18465#S2.p2.1)\.
- \[7\]Z\. Chu, J\. Ma, and H\. Wang\(2021b\)Learning from crowds by modeling common confusions\.InProceedings of the AAAI Conference on Artificial Intelligence,pp\. 5832–5840\.Cited by:[§II\-B](https://arxiv.org/html/2607.18465#S2.SS2.p1.1)\.
- \[8\]Z\. Chu and H\. Wang\(2021a\)Improve learning from crowds via generative augmentation\.InProceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining,pp\. 167–175\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p1.1)\.
- \[9\]A\. Das, H\. Liu, V\. Kovatchev, and M\. Lease\(2023\)The state of human\-centered NLP technology for fact\-checking\.Information Processing & Management60\(2\),pp\. 103219\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p1.1)\.
- \[10\]A\. P\. Dawid and A\. M\. Skene\(1979\)Maximum likelihood estimation of observer error\-rates using the EM algorithm\.Journal of the Royal Statistical Society\. Series C \(Applied Statistics\)28\(1\),pp\. 20–28\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p2.1),[§I](https://arxiv.org/html/2607.18465#S1.p3.1),[§II\-B](https://arxiv.org/html/2607.18465#S2.SS2.p1.1),[§V\-A](https://arxiv.org/html/2607.18465#S5.SS1.p4.1)\.
- \[11\]S\. De, M\. A\. Bakker, J\. Baxter, and M\. Saveski\(2025\)Supernotes: driving consensus in crowd\-sourced fact\-checking\.InProceedings of the World Wide Web Conference,pp\. 3751–3761\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p1.1)\.
- \[12\]G\. Demartini, D\. E\. Difallah, and P\. Cudré\-Mauroux\(2012\)Zencrowd: leveraging probabilistic reasoning and crowdsourcing techniques for large\-scale entity linking\.InProceedings of the World Wide Web Conference,pp\. 469–478\.Cited by:[§II](https://arxiv.org/html/2607.18465#S2.p2.1)\.
- \[13\]R\. Durrett\(2010\)Probability: theory and examples, 4th edition\.Cambridge University Press\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p9.1),[§IV\-A](https://arxiv.org/html/2607.18465#S4.SS1.p4.1)\.
- \[14\]L\. Gienapp, T\. Hagen, M\. Fröbe, M\. Hagen, B\. Stein, M\. Potthast, and H\. Scells\(2025\)The viability of crowdsourcing for RAG evaluation\.InProceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 159–169\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p1.1)\.
- \[15\]G\. V\. Horn, S\. Branson, S\. Loarie, S\. J\. Belongie, and P\. Perona\(2018\)Lean multiclass crowdsourcing\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition,pp\. 2714–2723\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p1.1)\.
- \[16\]S\. Ibrahim, X\. Fu, N\. Kargas, and K\. Huang\(2019\)Crowdsourcing via pairwise co\-occurrences: identifiability and algorithms\.InProceedings of the Annual Conference on Neural Information Processing Systems,pp\. 7845–7855\.Cited by:[§II\-B](https://arxiv.org/html/2607.18465#S2.SS2.p1.1)\.
- \[17\]S\. Ibrahim and X\. Fu\(2021\)Crowdsourcing via annotator co\-occurrence imputation and provable symmetric nonnegative matrix factorization\.InProceedings of the International Conference on Machine Learning,pp\. 4544–4554\.Cited by:[§II\-B](https://arxiv.org/html/2607.18465#S2.SS2.p1.1)\.
- \[18\]S\. Ibrahim, T\. Nguyen, and X\. Fu\(2023\)Deep learning from crowdsourced labels: coupled cross\-entropy minimization, identifiability, and regularization\.InProceedings of the International Conference on Learning Representations,Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p5.1)\.
- \[19\]S\. Ibrahim, T\. Nguyen, and X\. Fu\(2023\)Deep learning from crowdsourced labels: coupled cross\-entropy minimization, identifiability, and regularization\.InProceedings of the International Conference on Learning Representations,Cited by:[§II](https://arxiv.org/html/2607.18465#S2.p1.1)\.
- \[20\]H\. Kim and Z\. Ghahramani\(2012\)Bayesian classifier combination\.InProceedings of the International Conference on Artificial Intelligence and Statistics,pp\. 619–627\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p2.1),[§I](https://arxiv.org/html/2607.18465#S1.p3.1),[§II\-B](https://arxiv.org/html/2607.18465#S2.SS2.p1.1),[§II](https://arxiv.org/html/2607.18465#S2.p2.1),[§V\-A](https://arxiv.org/html/2607.18465#S5.SS1.p4.1)\.
- \[21\]G\. Li, J\. Wang, Y\. Zheng, and M\. J\. Franklin\(2016\)Crowdsourced data management: a survey\.IEEE Transactions on Knowledge and Data Engineering28\(9\),pp\. 2296–2319\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p1.1)\.
- \[22\]J\. Li\(2024\)A comparative study on annotation quality of crowdsourcing and llm via label aggregation\.InProceedings of the International Conference on Acoustics, Speech and Signal Processing,pp\. 6525–6529\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p1.1)\.
- \[23\]Q\. Li, Y\. Li, J\. Gao, L\. Su, B\. Zhao, M\. Demirbas, W\. Fan, and J\. Han\(2014\)A confidence\-aware approach for truth discovery on long\-tail data\.Proceedings of the VLDB Endowment8\(4\),pp\. 425–436\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p2.1),[§II\-A](https://arxiv.org/html/2607.18465#S2.SS1.p1.1),[§V\-A](https://arxiv.org/html/2607.18465#S5.SS1.p4.1)\.
- \[24\]Q\. Li, Y\. Li, J\. Gao, B\. Zhao, W\. Fan, and J\. Han\(2014\)Resolving conflicts in heterogeneous data by truth discovery and source reliability estimation\.InProceedings of the International Conference on Management of Data,pp\. 1187–1198\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p2.1)\.
- \[25\]X\. Li, T\. Liu, B\. Han, G\. Niu, and M\. Sugiyama\(2021\)Provably end\-to\-end label\-noise learning without anchor points\.InProceedings of the International Conference on Machine Learning,pp\. 6403–6413\.Cited by:[§II](https://arxiv.org/html/2607.18465#S2.p1.1)\.
- \[26\]Y\. Li, B\. I\. P\. Rubinstein, and T\. Cohn\(2019\)Exploiting worker correlation for label aggregation in crowdsourcing\.InProceedings of the International Conference on Machine Learning,pp\. 3886–3895\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p3.1),[§II\-B](https://arxiv.org/html/2607.18465#S2.SS2.p1.1),[§II](https://arxiv.org/html/2607.18465#S2.p2.1),[§V\-A](https://arxiv.org/html/2607.18465#S5.SS1.p4.1)\.
- \[27\]Y\. Li, B\. I\. P\. Rubinstein, and T\. Cohn\(2019\)Truth inference at scale: a bayesian model for adjudicating highly redundant crowd annotations\.InProceedings of the World Wide Web Conference,pp\. 1028–1038\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p2.1),[§I](https://arxiv.org/html/2607.18465#S1.p3.1),[§I](https://arxiv.org/html/2607.18465#S1.p6.1),[§II\-A](https://arxiv.org/html/2607.18465#S2.SS1.p1.1),[§II](https://arxiv.org/html/2607.18465#S2.p2.1),[§V\-A](https://arxiv.org/html/2607.18465#S5.SS1.p1.1),[§V\-A](https://arxiv.org/html/2607.18465#S5.SS1.p4.1)\.
- \[28\]Y\. Li, X\. Wu, Y\. Jin, J\. Li, and G\. Li\(2020\)Efficient algorithms for crowd\-aided categorization\.Proceedings of the VLDB Endowment13\(8\),pp\. 1221–1233\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p1.1)\.
- \[29\]H\. Liu, J\. Liu, F\. Tang, L\. Chen, J\. Yu, Y\. Zhu, Q\. Dong, Y\. Yu, and X\. Hou\(2026\)Towards a foundation model for crowdsourced label aggregation\.InProceedings of the International Conference on Learning Representations,Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p2.1),[§I](https://arxiv.org/html/2607.18465#S1.p4.1),[§II\-C](https://arxiv.org/html/2607.18465#S2.SS3.p1.1),[§V\-A](https://arxiv.org/html/2607.18465#S5.SS1.p4.1)\.
- \[30\]H\. Liu, J\. Liu, F\. Tang, P\. Li, L\. Chen, J\. Yu, Y\. Zhu, M\. Gao, Y\. Yang, and X\. Hou\(2024\)Graph contrastive learning for truth inference\.InProceedings of the International Conference on Data Engineering,pp\. 263–275\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p2.1),[§I](https://arxiv.org/html/2607.18465#S1.p4.1),[§II\-C](https://arxiv.org/html/2607.18465#S2.SS3.p1.1),[§II](https://arxiv.org/html/2607.18465#S2.p2.1),[§V\-A](https://arxiv.org/html/2607.18465#S5.SS1.p1.1),[§V\-A](https://arxiv.org/html/2607.18465#S5.SS1.p4.1)\.
- \[31\]J\. Liu, F\. Tang, and J\. Huang\(2021\)Truth inference with bipartite attention graph neural network from a comprehensive view\.InProceedings of the IEEE International Conference on Multimedia and Expo,pp\. 1–6\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p2.1)\.
- \[32\]J\. Liu, F\. Tang, H\. Liu, L\. Chen, Y\. Yu, Y\. Zhu, J\. Yu, X\. Hou, and P\. Heng\(2025\)BAT: a versatile bipartite attention\-based approach for comprehensive truth inference in mobile crowdsourcing\.IEEE Transactions on Mobile Computing24\(10\),pp\. 9368–9382\.Cited by:[§II](https://arxiv.org/html/2607.18465#S2.p1.1)\.
- \[33\]S\. Lyu, W\. Ouyang, Y\. Wang, H\. Shen, and X\. Cheng\(2021\)Truth discovery by claim and source embedding\.IEEE Transactions on Knowledge and Data Engineering33\(3\),pp\. 1264–1275\.Cited by:[§II\-C](https://arxiv.org/html/2607.18465#S2.SS3.p1.1),[§II](https://arxiv.org/html/2607.18465#S2.p2.1)\.
- \[34\]M\. Modaresnezhad, L\. S\. Iyer, P\. Palvia, and V\. Taras\(2020\)Information technology \(IT\) enabled crowdsourcing: a conceptual framework\.Information Processing & Management57\(2\),pp\. 102–135\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p1.1)\.
- \[35\]M\. Otani, R\. Togashi, Y\. Sawai, R\. Ishigami, Y\. Nakashima, E\. Rahtu, J\. Heikkilä, and S\. Satoh\(2023\)Toward verifiable and reproducible human evaluation for text\-to\-image generation\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 14277–14286\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p1.1)\.
- \[36\]J\. Qiu, A\. Madotto, Z\. Lin, P\. A\. Crook, Y\. E\. Xu, B\. Damavandi, X\. Dong, C\. Faloutsos, L\. Li, and S\. Moon\(2024\)SnapNTell: enhancing entity\-centric visual question answering with retrieval augmented multimodal LLM\.InProceedings of the findings of the Association for Computational Linguistics,pp\. 247–266\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p1.1)\.
- \[37\]F\. Rodrigues, F\. C\. Pereira, and B\. Ribeiro\(2013\)Learning from multiple annotators: distinguishing good from random labelers\.Pattern Recognition Letters34\(12\),pp\. 1428–1436\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p5.1)\.
- \[38\]C\. Song, K\. Liu, and X\. Zhang\(2021\)Collusion detection and ground truth inference in crowdsourcing for labeling tasks\.Journal of Machine Learning Research22\(190\),pp\. 8532–8576\.Cited by:[§II](https://arxiv.org/html/2607.18465#S2.p2.1)\.
- \[39\]Y\. Sun, J\. Wang, P\. Cheng, L\. Zheng, L\. Chen, and J\. Yin\(2024\)Cross\-domain\-aware worker selection with training for crowdsourced annotation\.InProceedings of the International Conference on Data Engineering,pp\. 249–262\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p1.1)\.
- \[40\]Z\. Q\. Tan, O\. Isupova, G\. Carneiro, X\. Zhu, and Y\. Li\(2024\)Bayesian detector combination for object detection with crowdsourced annotations\.InProceedings of the European Conference on Computer Vision,pp\. 329–346\.Cited by:[§II](https://arxiv.org/html/2607.18465#S2.p1.1)\.
- \[41\]R\. Tanno, A\. Saeedi, S\. Sankaranarayanan, D\. C\. Alexander, and N\. Silberman\(2019\)Learning from noisy labels by regularized estimation of annotator confusion\.InProceedings of the findings of the IEEE Conference on Computer Vision and Pattern Recognition,pp\. 11244–11253\.Cited by:[§II](https://arxiv.org/html/2607.18465#S2.p1.1)\.
- \[42\]M\. Venanzi, J\. Guiver, G\. Kazai, P\. Kohli, and M\. Shokouhi\(2014\)Community\-based bayesian aggregation models for crowdsourcing\.InProceedings of the World Wide Web Conference,pp\. 155–164\.Cited by:[§II\-B](https://arxiv.org/html/2607.18465#S2.SS2.p1.1),[§V\-A](https://arxiv.org/html/2607.18465#S5.SS1.p4.1)\.
- \[43\]J\. Whitehill, P\. Ruvolo, T\. Wu, J\. Bergsma, and J\. R\. Movellan\(2009\)Whose vote should count more: optimal integration of labels from labelers of unknown expertise\.InProceedings of the Annual Conference on Neural Information Processing Systems,pp\. 2035–2043\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p2.1),[§II\-A](https://arxiv.org/html/2607.18465#S2.SS1.p1.1),[§V\-A](https://arxiv.org/html/2607.18465#S5.SS1.p4.1)\.
- \[44\]G\. Wu, X\. Zhuo, X\. Bao, X\. Hu, R\. Hong, and X\. Wu\(2023\)Crowdsourcing truth inference via reliability\-driven multi\-view graph embedding\.ACM Transactions on Knowledge Discovery from Data17\(5\),pp\. 65:1–65:26\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p2.1),[§II\-C](https://arxiv.org/html/2607.18465#S2.SS3.p1.1),[§II](https://arxiv.org/html/2607.18465#S2.p2.1),[§V\-A](https://arxiv.org/html/2607.18465#S5.SS1.p4.1)\.
- \[45\]G\. Wu, X\. Zhuo, L\. Zhou, X\. Bao, R\. Hong, and X\. Wu\(2023\)TIRA: truth inference via reliability aggregation on object\-source graph\.IEEE Transactions on Knowledge and Data Engineering35\(11\),pp\. 11967–11981\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p2.1),[§I](https://arxiv.org/html/2607.18465#S1.p4.1),[§II](https://arxiv.org/html/2607.18465#S2.p2.1),[§V\-A](https://arxiv.org/html/2607.18465#S5.SS1.p1.1)\.
- \[46\]H\. Wu, L\. Wang, K\. Cheng, D\. Yang, J\. Tang, and G\. Xue\(2022\)Privacy\-enhanced and practical truth discovery in two\-server mobile crowdsensing\.IEEE Transactions on Network Science and Engineering9\(3\),pp\. 1740–1755\.Cited by:[§II](https://arxiv.org/html/2607.18465#S2.p1.1)\.
- \[47\]B\. Yang, L\. Jiang, and W\. Zhang\(2024\)Probabilistic matrix factorization\-based three\-stage label completion for crowdsourcing\.InProceedings of the IEEE International Conference on Data Mining,pp\. 540–549\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p1.1)\.
- \[48\]Y\. Yang, Z\. Wu, L\. Wu, K\. Zhang, R\. Hong, Z\. Zhang, J\. Zhou, and M\. Wang\(2023\)Generative\-contrastive graph learning for recommendation\.InProceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 1117–1126\.Cited by:[§IV\-A](https://arxiv.org/html/2607.18465#S4.SS1.p1.1)\.
- \[49\]L\. Yin, J\. Han, W\. Zhang, and Y\. Yu\(2017\)Aggregating crowd wisdoms with label\-aware autoencoders\.InProceedings of the International Joint Conference on Artificial Intelligence,pp\. 1325–1331\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p2.1),[§II\-C](https://arxiv.org/html/2607.18465#S2.SS3.p1.1),[§II](https://arxiv.org/html/2607.18465#S2.p2.1)\.
- \[50\]C\. J\. Zhang, Y\. Liu, P\. Zeng, T\. Wu, L\. Chen, P\. Hui, and F\. Hao\(2024\)Similarity\-driven and task\-driven models for diversity of opinion in crowdsourcing markets\.The VLDB Journal33\(5\),pp\. 1377–1398\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p1.1)\.
- \[51\]H\. Zhang, S\. Li, D\. Zeng, C\. Yan, and S\. Ge\(2024\)Coupled confusion correction: learning from crowds with sparse annotations\.InProceedings of the AAAI Conference on Artificial Intelligence,pp\. 16732–16740\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p2.1),[§I](https://arxiv.org/html/2607.18465#S1.p3.1),[§I](https://arxiv.org/html/2607.18465#S1.p5.1),[§II](https://arxiv.org/html/2607.18465#S2.p2.1)\.
- \[52\]W\. Zhang, L\. Jiang, and C\. Li\(2025\)ELDP: enhanced label distribution propagation for crowdsourcing\.IEEE Transactions on Pattern Analysis and Machine Intelligence47\(3\),pp\. 1850–1862\.Cited by:[§II](https://arxiv.org/html/2607.18465#S2.p1.1)\.
- \[53\]Y\. Zhang, X\. Chen, D\. Zhou, and M\. I\. Jordan\(2016\)Spectral methods meet EM: A provably optimal algorithm for crowdsourcing\.Journal of Machine Learning Research17,pp\. 102:1–102:44\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p2.1)\.
- \[54\]Y\. Zhang, Y\. Chen, C\. Fang, Q\. Wang, J\. Wu, and J\. Xin\(2025\)Learning from open\-set noisy labels based on multi\-prototype modeling\.Pattern Recognition157,pp\. 110902\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p1.1)\.
- \[55\]L\. Zheng, P\. Cheng, L\. Chen, J\. Yu, X\. Lin, and J\. Yin\(2022\)Crowdsourced fact validation for knowledge bases\.InProceedings of the International Conference on Data Engineering,pp\. 938–950\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p1.1)\.
- \[56\]Y\. Zheng, G\. Li, Y\. Li, C\. Shan, and R\. Cheng\(2017\)Truth inference in crowdsourcing: is the problem solved?\.Proceedings of the VLDB Endowment10\(5\),pp\. 541–552\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p1.1)\.
- \[57\]D\. Zhou, J\. C\. Platt, S\. Basu, and Y\. Mao\(2012\)Learning from the wisdom of crowds by minimax entropy\.InProceedings of the Annual Conference on Neural Information Processing Systems,pp\. 2204–2212\.Cited by:[§I](https://arxiv.org/html/2607.18465#S1.p2.1)\.