Perspectivist模型擅长预测谁？标注者一致性、稀疏性与性能表现

# Who Do Perspectivist Models Predict Well? Annotator Agreement, Sparsity, and Performance
Source: [https://arxiv.org/html/2610.22567](https://arxiv.org/html/2610.22567)
Vaagishwaran SivakumarAnwesha SamaddarAffiliation:Akul MundadaHarnoor KaurMaria GiniAffiliation:University of Minnesota, Twin Cities

###### Abstract

Successful completion of perspectivist natural language processing \(NLP\) annotation tasks requires algorithms to be aligned across diverse perspectives in order to predict how different humans will rate text\. This work studies how the performance of such algorithms varies for annotators who behave differently than others, either by providing annotations that diverge from others’ responses or by providing fewer annotations altogether\. To aid with characterizing model performance, we curate the largest collection to date of perspectivist NLP datasets with ordinal annotations\. After evaluating multiple classes of perspectivist algorithms across each of the datasets, we model the relationship between individual annotator idiosyncrasy and algorithm performance\. In general, we find that algorithms perform better on annotators who tend to agree with others, but that annotator response rate does not have a statistically significant positive correlation with algorithm performance\.

## 1Introduction

Perspectivist approaches to natural language processing \(NLP\) have recently increased in popularity[Fleisig et al\. \(2024\)](https://arxiv.org/html/2610.22567#bib.bib26), with a significant survey paper published by[Frenda et al\.](https://arxiv.org/html/2610.22567#bib.bib28)and multiple competitions measuring model performance on perspectivist tasks[Leonardelli et al\. \(2025\)](https://arxiv.org/html/2610.22567#bib.bib42)\. We use these English text\-to\-rating tasks as a wide testbed for observing the behavior of pluralist algorithms in order to understand what sorts of human behavior lead to poor predictions of beliefs under existing pluralist algorithms\.

We believe our work would be of interest to these groups:

The first groupis AI alignment researchers who are interested in early detection of common behavior across a wide range of algorithms that indicates bias against \(or inability to model\) humans who exhibit certain types of behavior\. Failure across different algorithm classes to accurately predict some people’s opinions as well as others indicates the potential for pluralistic imputation models to fail to properly understand the values of certain people\. We hope that this work can provide additional evidence that people with unpopular opinions may be more difficult for pluralistic systems to model\.

The second groupof people is privacy\-sensitive individuals who are curious about the impact of withholding interaction\-based data \(e\.g\., ratings\) from individualized prediction algorithms\. While our experiments cover a relatively small niche of the space of individualized prediction algorithms, our results provide evidence that individuals who do not share large amounts of data will not suffer from algorithm performance decreases in comparison to individuals who do share large amounts of data\.

The third groupis creators of NLP datasets\. In the process of data collection, a common concern is how to balance the time spent on building strong annotator guidelines that promote annotator agreement, versus focusing more intently on data collection despite disagreement\. While many different factors may impact the ideal balance between those goals, we offer an insight from the perspective of model performance on individual annotators: the algorithms we tested tended to perform better for annotators who agree with one another, whereas the amount of data an annotator contributed to training data did not have a statistically significant correlation with algorithmic performance on that annotator\.

Overall, we test the following two hypotheses across 46 datasets and 15 algorithms, after controlling for the number of annotations provided by an annotator in the test data:

1. \(H1\)Lower individual annotator agreement is associated with worse model performance on that individual\.
2. \(H2\)Lower individual annotator response rate is associated with worse model performance on that individual\.

Our results show statistically significant evidence in support of[H1](https://arxiv.org/html/2610.22567#S1.I2.i1), but do not support[H2](https://arxiv.org/html/2610.22567#S1.I2.i2)\.

## 2Related Work

#### Perspectivist Approaches to NLP\.

Previous approaches to natural language processing datasets often involved using annotators to acquire an aggregated “gold\-label,” a single annotation that represented the correct answer for a natural language task[Aroyo and Welty \(2015\)](https://arxiv.org/html/2610.22567#bib.bib4);[Basile \(2020\)](https://arxiv.org/html/2610.22567#bib.bib6)\. Work on natural language processing has recently explored more perspectivist approaches, wherein individual annotators’ perspectives are retained, and the goal of prediction tasks is to model individual annotations or distributional dynamics, rather than a single “correct” label[Cabitza et al\. \(2023\)](https://arxiv.org/html/2610.22567#bib.bib9)\.

Approaches related to perspectivism in natural language processing may use terminology such as “subjectivity”[Homayounirad et al\. \(2025\)](https://arxiv.org/html/2610.22567#bib.bib34), “uncertainty”[Zhang et al\. \(2025\)](https://arxiv.org/html/2610.22567#bib.bib76), “disagreement”[Davani et al\. \(2022\)](https://arxiv.org/html/2610.22567#bib.bib16), “pluralistic”[Castricato et al\. \(2025\)](https://arxiv.org/html/2610.22567#bib.bib12);[Sorensen et al\. \(2024b\)](https://arxiv.org/html/2610.22567#bib.bib67), or “descriptive” approaches[Röttger et al\. \(2022\)](https://arxiv.org/html/2610.22567#bib.bib59)\.

The survey by[Frenda et al\.](https://arxiv.org/html/2610.22567#bib.bib28)provides a general overview of work in this area, highlighting the NLPerspectives workshop and Learning with Disagreements \(LeWiDi\) as core sources for research on this topic[Frenda et al\. \(2025\)](https://arxiv.org/html/2610.22567#bib.bib28)\. We build upon the work done by[Lowmanstone et al\.](https://arxiv.org/html/2610.22567#bib.bib45), which was previously published in NLPerspectives, and add additional datasets from the SemEval 2023 LeWiDi competition[Lowmanstone et al\. \(2023\)](https://arxiv.org/html/2610.22567#bib.bib45);[Leonardelli et al\. \(2023\)](https://arxiv.org/html/2610.22567#bib.bib43)\.

Our research focuses primarily on individual modeling, as described by[Sorensen et al\. \(2024a\)](https://arxiv.org/html/2610.22567#bib.bib68)\. While previous work has used demographic data to make individual and distributional predictions\([Gordon et al\., 2022](https://arxiv.org/html/2610.22567#bib.bib30);[Meister et al\., 2025](https://arxiv.org/html/2610.22567#bib.bib48)\), due to concerns about using demographic data for human behavior prediction\([Patty and Penn, 2023](https://arxiv.org/html/2610.22567#bib.bib53);[Jernigan and Mistree, 2009](https://arxiv.org/html/2610.22567#bib.bib36);[Barocas and Selbst, 2016](https://arxiv.org/html/2610.22567#bib.bib5)\), we utilize algorithms that make predictions based on annotator behavior, rather than demographic information\.

#### Pluralistic AI Alignment\.

Perspectivist algorithms are a form of “steerable pluralistic models”\([Sorensen et al\., 2024b](https://arxiv.org/html/2610.22567#bib.bib67)\), which are trained to model diverse ranges of opinions\. Among the goals of pluralistic AI alignment are to ensure that AI can follow the individual revealed preferences of people in diverse populations\([Gabriel, 2020](https://arxiv.org/html/2610.22567#bib.bib29)\)\. We view NLP annotation prediction tasks as an early\-stage test of AI systems’ capability to model simple revealed preferences about text\.

#### Fairness\.

Our work is also motivated by fairness in machine learning algorithms\. Specifically, we are curious if annotators who respond less or differently from other annotators are modeled poorly by existing algorithms as a form of disparate impact[Pessach and Shmueli \(2023\)](https://arxiv.org/html/2610.22567#bib.bib54)\. Papers such as[Gordon et al\. \(2022\)](https://arxiv.org/html/2610.22567#bib.bib30)and[Eckman et al\. \(2025\)](https://arxiv.org/html/2610.22567#bib.bib24)demonstrate that, for their respective tasks, underrepresented populations in training data lead to biased algorithmic performance, which may be rectified by amplifying the signal provided by underrepresented populations in the data\.

Other work attempts to measure bias in existing NLP systems\([Nangia et al\., 2020](https://arxiv.org/html/2610.22567#bib.bib51);[Parrish et al\., 2022](https://arxiv.org/html/2610.22567#bib.bib52);[Nadeem et al\., 2021](https://arxiv.org/html/2610.22567#bib.bib50);[Abid et al\., 2021](https://arxiv.org/html/2610.22567#bib.bib1);[Bolukbasi et al\., 2016](https://arxiv.org/html/2610.22567#bib.bib8)\)\. We consider our work to be within this thread: instead of measuring bias against populations that are demographic minorities, we measure the general algorithmic bias against populations that are minorities in terms of opinions and response behavior\.

#### Perspectivist Datasets\.

Multiple algorithms we use for annotation prediction make scalar predictions, so, in this work, we restrict our focus to datasets that use ordinal annotations \(i\.e\., binary or rating\-based annotations\) rather than datasets with purely categorical annotations\. However, many additional datasets have been released under the wider umbrella of perspectivist NLP annotation prediction\.

The collection of perspectivist NLP datasets from[Plank \(2022\)](https://arxiv.org/html/2610.22567#bib.bib55)provides an excellent foundation for datasets with disaggregated annotations\. To the best of our knowledge, the Awesome Human Label Variation \(AHLV\) repository provides the largest release of perspectivist datasets to date\([Plank, 2022](https://arxiv.org/html/2610.22567#bib.bib55)\)\. However, many of the datasets in AHLV are not NLP\-specific, nor do they provide ordinal ratings\. We use 4 of the datasets originally included in the collection, and 12 out of 16 of the dataset families we use in our analysis are currently listed in AHLV\. We believe that our work will highlight additional NLP datasets that can be added to the repository\.

[Deng et al\. \(2023\)](https://arxiv.org/html/2610.22567#bib.bib18)provide a collection of datasets named TID\-8 for disagreement\-based tasks, 4 of which are included in our collection\. To the best of our knowledge, this is the previously largest released collection of solely perspectivist NLP annotation prediction datasets[Deng et al\. \(2023\)](https://arxiv.org/html/2610.22567#bib.bib18)\. In their paper, they demonstrate that modeling the underlying opinions and annotators that lead to a particular gold\-label can improve models’ ability to accurately predict individual labels\. While the Pejorative[Dinu et al\. \(2021\)](https://arxiv.org/html/2610.22567#bib.bib22)dataset from TID\-8 may be included in future work, other datasets, such as Friends QIA[Damgaard et al\. \(2021\)](https://arxiv.org/html/2610.22567#bib.bib14)and GoEmotions[Demszky et al\. \(2020\)](https://arxiv.org/html/2610.22567#bib.bib17), are not suitable for our task, primarily due to a lack of clearly ordinal ratings\.

Other tasks for NLP annotation prediction have been released via competitions\. The LeWiDi 3 benchmark, released during the 2025 LeWiDi competition[Leonardelli et al\. \(2025\)](https://arxiv.org/html/2610.22567#bib.bib42)includes additional perspectivist NLP datasets such as the Conversational Sarcasm Corpus\([Jang and Frassinelli, 2024](https://arxiv.org/html/2610.22567#bib.bib35)\), MultiPICo\([Casola et al\., 2024](https://arxiv.org/html/2610.22567#bib.bib11)\), and Par\([Leonardelli et al\., 2025](https://arxiv.org/html/2610.22567#bib.bib42)\), which are not included in our datasets because their text does not consist of a single string; without a canonical method of combining the dataset example texts into a single string, we are concerned we may inadvertently bias algorithmic performance via our choice of format\. Other datasets in the competition were not immediately suitable for our task because they utilized non\-ordinal annotation categories\.

#### Perspectivist Algorithms\.

Multiple types of algorithms can be used to accomplish perspectivist NLP tasks\. Collaborative filtering approaches naturally apply but require additional mechanisms in order to ingest text[Su and Khoshgoftaar \(2009\)](https://arxiv.org/html/2610.22567#bib.bib70);[Ekstrand et al\. \(2011\)](https://arxiv.org/html/2610.22567#bib.bib25);[Shi et al\. \(2014\)](https://arxiv.org/html/2610.22567#bib.bib66)\. General recommender system approaches may be applied as well, but are restricted to models that can accomplish rating prediction rather than item ranking[Ekstrand et al\. \(2011\)](https://arxiv.org/html/2610.22567#bib.bib25)\. Matrix factorization approaches are also applicable to rating prediction tasks[Rendle et al\. \(2020\)](https://arxiv.org/html/2610.22567#bib.bib58)\.

A recent wave of research has used LLMs to imitate human annotations, which can be viewed as a prediction task[Castricato et al\. \(2025\)](https://arxiv.org/html/2610.22567#bib.bib12);[Marreddy et al\. \(2025\)](https://arxiv.org/html/2610.22567#bib.bib47)\. However, existing studies show that LLM performance is highly task dependent, and previous models do not perform particularly well on NLP perspectivist tasks[Bavaresco et al\. \(2025\)](https://arxiv.org/html/2610.22567#bib.bib7);[Lowmanstone et al\. \(2023\)](https://arxiv.org/html/2610.22567#bib.bib45);[Sarumi et al\. \(2024\)](https://arxiv.org/html/2610.22567#bib.bib64)\.

#### Metrics\.

Methods for measuring model performance on NLP perspectivist datasets have varied widely[Sarumi et al\. \(2025\)](https://arxiv.org/html/2610.22567#bib.bib65)\. Potential metrics include F1 scores, mean absolute distance, and ROC\-AUC, which can also be measured on a per\-annotator basis, rather than a per\-dataset basis[Sarumi et al\. \(2025\)](https://arxiv.org/html/2610.22567#bib.bib65);[Heinisch et al\. \(2023\)](https://arxiv.org/html/2610.22567#bib.bib33)\. For our experiment, we have chosen to use normalized mean squared error \(NMSE\) as our metric because NMSE is a natural error metric for ordinal ratings across multiple scales, compared to other metrics which are primarily used for categorical data and do not take advantage of the distance between ordinal ratings to measure the strength of poor predictions\. While normalized root mean squared error \(NRMSE\) is a stronger choice for interpretability applications, we found that our statistical error analysis was more easily conducted using normalized mean squared error directly\.

Figure 1:Depiction of the dataset split process\. The figure displays 20 observed annotations from an illustrative pool of 60; cell numbers indicate positions in the shuffled order\. The final test set contains the last 3 remaining annotations \(5%\), and the training set contains 57 annotations\.

## 3Datasets

In our work, we use 46 English pluralist NLP datasets gathered from a variety of different sources, summarized in Table[3](https://arxiv.org/html/2610.22567#A2.T3)in the appendix\. Information about licenses for each of the datasets can be found in Appendix[B](https://arxiv.org/html/2610.22567#A2)\. To the best of our knowledge, this is the largest collection of cleaned and formatted pluralist NLP datasets\.111All data and code used in our experiments will be released \(pursuant to licenses\) upon paper acceptance\.

The datasets are drawn from AffectiveText\([Strapparava and Mihalcea, 2007](https://arxiv.org/html/2610.22567#bib.bib69)\); HuBi\([Kazienko et al\., 2023](https://arxiv.org/html/2610.22567#bib.bib37)\), including Cockamamie Gobbledegook\([Gultchin et al\., 2019](https://arxiv.org/html/2610.22567#bib.bib31)\), the Personal Bias emotion and valence datasets\([Milkowski et al\., 2021](https://arxiv.org/html/2610.22567#bib.bib49)\), and Wikipedia Detox\([Wulczyn et al\., 2017a](https://arxiv.org/html/2610.22567#bib.bib73)\); NLPerspectives\([Lowmanstone et al\., 2023](https://arxiv.org/html/2610.22567#bib.bib45)\), including SChem and SChem5Labels\([Forbes et al\., 2020](https://arxiv.org/html/2610.22567#bib.bib27)\), SBIC\([Sap et al\., 2020](https://arxiv.org/html/2610.22567#bib.bib63)\), GHC\([Kennedy et al\., 2022](https://arxiv.org/html/2610.22567#bib.bib38)\), Sentiment\([Diaz et al\., 2018](https://arxiv.org/html/2610.22567#bib.bib20);[Diaz, 2020](https://arxiv.org/html/2610.22567#bib.bib21)\), and Politeness\([Danescu\-Niculescu\-Mizil et al\., 2013](https://arxiv.org/html/2610.22567#bib.bib15)\); LeWiDi\([Leonardelli et al\., 2023](https://arxiv.org/html/2610.22567#bib.bib43)\), including ConvAbuse\([Cercas Curry et al\., 2021](https://arxiv.org/html/2610.22567#bib.bib13)\), HS\-Brexit\([Akhtar et al\., 2021](https://arxiv.org/html/2610.22567#bib.bib2)\), and MD\-Agreement\([Leonardelli et al\., 2021](https://arxiv.org/html/2610.22567#bib.bib41)\); and additional datasets from AHLV\([Plank, 2022](https://arxiv.org/html/2610.22567#bib.bib55)\), including CommitmentBank\([de Marneffe et al\., 2019](https://arxiv.org/html/2610.22567#bib.bib46)\), Measuring Hate Speech\([Sachdeva et al\., 2022](https://arxiv.org/html/2610.22567#bib.bib60)\), Annotators with Attitudes\([Sap et al\., 2022](https://arxiv.org/html/2610.22567#bib.bib62)\), and DynaSent\([Potts et al\., 2021](https://arxiv.org/html/2610.22567#bib.bib56)\)\.

Datasets were cleaned via a multi\-step process\. First, following the same dataset format as[Lowmanstone et al\. \(2023\)](https://arxiv.org/html/2610.22567#bib.bib45), they were converted from their original release files into two separate files: a matrix of nonnegative integer annotation values and a list of texts\. Each text corresponds to a row in the matrix of annotations, and each annotator for the dataset corresponds to a column in the matrix\. Ratings in datasets such as SBIC that originally allowed for 0\.5 increments in ratings were doubled in order to create integer ratings\. Missing annotations are represented by a unique sentinel value\. Next, we merge duplicate texts in each dataset, keeping all annotations whenever possible, and choosing annotations at random if annotations conflict\. Finally, to reduce the size of datasets in order to improve algorithmic speed, we uniformly sample 10,000 texts and 500 annotators independently and without replacement in order to create the final downsampled datasets\. Examples and annotators without any annotations after downsampling are removed\. For datasets with less than 10,000 texts or 500 annotators, all the data were kept\.

After dataset cleaning, we split the datasets into train and test data using a 95%\-5% split, as done by[Lowmanstone et al\. \(2023\)](https://arxiv.org/html/2610.22567#bib.bib45), which we detail here and depict in Figure[1](https://arxiv.org/html/2610.22567#S2.F1), as it is particularly relevant to our experimental design\. We first choose a random ordering of all annotations\. Then, we iterate through this ordering, adding each annotation to the train split as long as the train split doesn’t already contain annotations from its same row and column\. We continue this process of adding annotations to the train split until there is at least one annotation for each row and each column in the train split\. From the annotations not placed into the train split, we retain the original ordering, but iterate in reverse and assign 5% of the total amount of annotations to create the test split\. The remaining annotations \(in the middle of the ordering\) are added to the train split\. This ensures that each text and annotator has at least one annotation during training, which is required for proper utilization of all of the chosen algorithms\. This process introduces a dependency between the number of ratings provided by an annotator in the train split and the number of ratings provided by an annotator in the test split, which must be accounted for in our statistical analysis\. Graphs of the distribution of annotator sparsity and agreement can be found in Figure[6](https://arxiv.org/html/2610.22567#A2.F6)in the appendix\.

## 4Algorithms

We use 6 core families of imputation algorithms: 1\. Neural Collaborative Filtering \(NCF\)\([He et al\., 2017](https://arxiv.org/html/2610.22567#bib.bib32)\), 2\. Multitask\([Davani et al\., 2022](https://arxiv.org/html/2610.22567#bib.bib16)\), 3\. Kernel Matrix Factorization \(KMF\)\([Do, 2022](https://arxiv.org/html/2610.22567#bib.bib23)\), 4\. MASDA\([Sullivan et al\., 2023](https://arxiv.org/html/2610.22567#bib.bib71)\), 5\. TabFM\([Kong and Das, 2026](https://arxiv.org/html/2610.22567#bib.bib39)\), and 6\. baseline mean prediction algorithms, as listed in Table[2](https://arxiv.org/html/2610.22567#A1.T2)in the appendix\.

Across all models, if outputs fall outside the valid rating range for a given dataset, we clip the outputs into the proper range\. For NCF and KMF, we follow the publicly released code from[Lowmanstone et al\. \(2023\)](https://arxiv.org/html/2610.22567#bib.bib45), with only minimal adjustments for operating system differences\. For Multitask, we introduce multiple architectural improvements, including adding checkpointing functionality and ensuring the number of classes output by the model is not greater than the number of possible classes in the dataset \(see Appendix[A\.1](https://arxiv.org/html/2610.22567#A1.SS1)\)\.

Unlike[Lowmanstone et al\. \(2023\)](https://arxiv.org/html/2610.22567#bib.bib45), we do not use ChatGPT or other LLMs to directly impute data; in our preliminary experiments, doing so was found to be cost\-prohibitive and did not provide any performance gains over existing methods\.

Further details about the Multitask improvements and implementation details about MASDA, TabFM, and the baseline mean algorithms can be found in Appendix[A](https://arxiv.org/html/2610.22567#A1)\.

## 5Experiments

### 5\.1Primary Hypothesis Testing

To test[H1](https://arxiv.org/html/2610.22567#S1.I2.i1)and[H2](https://arxiv.org/html/2610.22567#S1.I2.i2), we train each of the 15 algorithms on each of the train splits, and then use each algorithm to predict each missing annotation in its training data\. Each model is run once on each dataset\. We can then compare the predictions made by the algorithm to the test data in order to compute performance\.

Formally, letm∈Mm\\in Mindicate the algorithm \(model\) being used for prediction\. Similarly, letd∈Dd\\in Dindicate the dataset anda∈Ada\\in A\_\{d\}indicate a particular annotator within the dataset\.

We model algorithm performance by using weighted fixed\-effects linear regression with cluster\-robust standard errors, clustered by dataset\-annotator pair and dataset\-algorithm pair to account for residual dependence:

Ym​d​a=αm​d\+βD​Zd​aD\+βS​Zd​aS\+γ​Zd​aC\+εm​d​aY\_\{mda\}=\\alpha\_\{md\}\+\\beta\_\{D\}Z^\{D\}\_\{da\}\+\\beta\_\{S\}Z^\{S\}\_\{da\}\+\\gamma Z^\{C\}\_\{da\}\+\\varepsilon\_\{mda\}\(1\)whereYm​d​aY\_\{mda\}is the normalized mean squared prediction error \(NMSE\) for algorithmmmon annotatoraain datasetdd\. The variableαm​d\\alpha\_\{md\}represents a fixed effect parameter for dataset\-algorithm pair\(d,m\)\(d,m\)to account for general model performance on datasets that is not related to specific annotators\.Zd​aDZ^\{D\}\_\{da\},Zd​aSZ^\{S\}\_\{da\},Zd​aCZ^\{C\}\_\{da\}are normalized z\-scores for the disagreement score, sparsity score, and log test count, respectively, of annotatoraain datasetdd\.βD\\beta\_\{D\},βS\\beta\_\{S\}, andγ\\gammaare the coefficient parameters of the model, indicating the strength and directionality of the association between individual annotator disagreement, sparsity, and the amount of test annotations with respect to model performance\.εm​d​a\\varepsilon\_\{mda\}represents the error not accounted for by the model\.

Letri​d​ar\_\{ida\}be the rating \(annotation\) given by annotatoraaon datasetddfor textii, andRMISSR\_\{\\mathrm\{MISS\}\}be the sentinel missing value\. Furthermore, letAd′=Ad∖\{a\}A^\{\\prime\}\_\{d\}=A\_\{d\}\\setminus\\\{a\\\}represent all annotators other than annotatoraa\. Finally, letXi​d​aDX^\{D\}\_\{ida\}be thedisagreement scoreof annotatoraain datasetddon textiiin the dataset\.Xi​d​aDX^\{D\}\_\{ida\}is only defined for dataset rowiiif target annotatoraaand at least one other annotator have provided an annotation for textii\. Then

Xi​d​aD=∑a′∈Ad′\{ri​d​a′≠RMISS∧ri​d​a′≠ri​d​a\}∑a′∈Ad′\{ri​d​a′≠RMISS\}X^\{D\}\_\{ida\}=\\frac\{\\sum\_\{a^\{\\prime\}\\in A^\{\\prime\}\_\{d\}\}\\mathbf\{1\}\\\!\\left\\\{r\_\{ida^\{\\prime\}\}\\neq R\_\{\\mathrm\{MISS\}\}\\land r\_\{ida^\{\\prime\}\}\\neq r\_\{ida\}\\right\\\}\}\{\\sum\_\{a^\{\\prime\}\\in A^\{\\prime\}\_\{d\}\}\\mathbf\{1\}\\\!\\left\\\{r\_\{ida^\{\\prime\}\}\\neq R\_\{\\mathrm\{MISS\}\}\\right\\\}\}is the disagreement score for a given textii, datasetdd, and target annotatoraa, computed as the proportion of annotators \(not including annotatoraa\) who disagreed with the target annotator on that text\. The annotator’s disagreement scoreXd​aDX^\{D\}\_\{da\}on a dataset is then the average of their disagreement score over all of the rowsiiin the dataset for whichXi​d​aDX^\{D\}\_\{ida\}is defined, andZd​aDZ^\{D\}\_\{da\}is the normalized z\-score ofXd​aDX^\{D\}\_\{da\}across annotators in datasetdd\. We use the train split of datasetddfor this calculation\. Annotators with an undefined disagreement score were not included in the regression\.

We compute an annotator’ssparsityon a dataset by computing the proportion of texts that the annotator did not respond to in the dataset:

Xd​aS=∑i∈Id𝟏\{ri​d​a=RMISS\}\|Id\|,X^\{S\}\_\{da\}=\\sum\_\{i\\in I\_\{d\}\}\\frac\{\\mathbf\{1\}\\\{r\_\{ida\}=R\_\{\\mathrm\{MISS\}\}\\\}\}\{\|I\_\{d\}\|\},whereIdI\_\{d\}is the set of row indexes for datasetdd\.Zd​aSZ^\{S\}\_\{da\}is then the normalized z\-score ofXd​aSX^\{S\}\_\{da\}across annotators in datasetdd\. Again, the train split of datasetddis used for this calculation\.

Hypothesis termCoef\.95% CIOne\-sidedppHolm\-adjustedppDisagreement \(βD\\beta\_\{D\}onZd​aDZ^\{D\}\_\{da\}\)0\.025\[0\.016, 0\.033\]1\.90×𝟏𝟎−𝟖\\mathbf\{1\.90\\times 10^\{\-8\}\}3\.80×𝟏𝟎−𝟖\\mathbf\{3\.80\\times 10^\{\-8\}\}Sparsity \(βS\\beta\_\{S\}onZd​aSZ^\{S\}\_\{da\}\)0\.001\[\-0\.001, 0\.003\]0\.1580\.158Table 1:Results for the regression shown in Eq\.[1](https://arxiv.org/html/2610.22567#S5.E1)with cluster\-robust standard errors\. Statistically significant results are inbold\.Thelog test countof an annotator is computed as

Xd​aC=ln⁡\(1\+td​a\),X^\{C\}\_\{da\}=\\ln\(1\+t\_\{da\}\),wheretd​at\_\{da\}is the number of test annotations from annotatoraain the test split for datasetdd\. This term is included in the model due to the train/test split process shown in Figure[1](https://arxiv.org/html/2610.22567#S2.F1): since annotations are sampled to cover each row and column in the train split, annotators with fewer examples are less likely to appear in the test set\. This term controls for that effect, ensuring thatβS\\beta\_\{S\}better reflects effects from individual annotator sparsity in the train data, rather than a confounding relationship between the number of annotations provided by the annotator between the train and test splits\.Zd​aCZ^\{C\}\_\{da\}is then the normalized z\-score ofXd​aCX^\{C\}\_\{da\}across annotators in the test split of datasetdd\.

We fit the model via weighted least squares, with the regression weight being the number of held\-out test annotationstd​at\_\{da\}divided by the number of algorithms in that algorithm’s family\. This gives greater influence to annotator\-level NMSE estimates based on more test annotations, which are expected to be more stable than estimates computed from fewer held\-out annotations\.

To compute confidence intervals and p\-values for our estimates ofβD\\beta\_\{D\}andβS\\beta\_\{S\}, we use cluster\-robust standard errors clustered by dataset–annotator pair and dataset\-algorithm pair \(in order to adjust for correlated residuals across algorithms and annotators, respectively\) and apply a standard small\-sample correction\.

Formally, the two main hypotheses we are testing areβD\>0\\beta\_\{D\}\>0\([H1](https://arxiv.org/html/2610.22567#S1.I2.i1)\) andβS\>0\\beta\_\{S\}\>0\([H2](https://arxiv.org/html/2610.22567#S1.I2.i2)\)\. Since we test two hypotheses, we adjust the resulting p\-values, computed via Waldzz\-tests, using the Holm–Bonferroni correction\. We report 95% two\-sided confidence intervals forβD\\beta\_\{D\}andβS\\beta\_\{S\}and the one\-sided and Holm\-adjusted p\-values for each of the hypothesis tests\. A positive coefficient indicates a positive relationship between the corresponding z\-score and NMSE \(a measure of error\); as the variable increases, algorithm performance decreases\.

### 5\.2Post Hoc Analysis

In addition to our main hypothesis testing, we analyze additional data to ensure that no particular dataset or algorithm family is heavily skewing our results, as such an outcome would limit the generalizability of our claims\. We emphasize here that no models were removed from our hypothesis testing or results as a consequence of these post hoc analyses\.

To determine the impact of dataset families on our primary result, we conducted bootstrap analyses with 10,000 samples each\. There are 16 different dataset families, as specified by the row groupings in Table[3](https://arxiv.org/html/2610.22567#A2.T3)in the appendix\. Within each bootstrap sample, we sample 16 dataset families with replacement from the set of possible dataset families\. Then, we refit our statistical model from Eq\.[1](https://arxiv.org/html/2610.22567#S5.E1)using the data from those sampled families \(where families chosen multiple times during the sampling step had all the data from the datasets of that family included as many times as that family was sampled\)\. This enables us to estimate how our results may have shifted if certain families of datasets were not included or were overly represented\. The resulting plots are in Figure[2](https://arxiv.org/html/2610.22567#S6.F2)\.

Since the number of algorithm families is small, with 6 groups of related algorithms, the fidelity of bootstrapping is expected to drop significantly if applied to algorithm families alone[Cameron et al\. \(2008\)](https://arxiv.org/html/2610.22567#bib.bib10)\. Leave\-one\-out methods are more appropriate for indicating algorithm family impact on regression coefficients\. Thus, we re\-fit the model from Eq\.[1](https://arxiv.org/html/2610.22567#S5.E1)6 times, leaving each algorithm family out exactly once\. The results are as a forest plot in Figure[3](https://arxiv.org/html/2610.22567#S6.F3)\.

In Appendix[C](https://arxiv.org/html/2610.22567#A3), we provide heatmaps of algorithm family and individual algorithm performance on each dataset\.

## 6Results and Analysis

After fitting the model on our data and computing the error estimation, we obtain the following results listed in Table[1](https://arxiv.org/html/2610.22567#S5.T1)\.

The results of our experiment indicate that it is highly likely that individual annotator agreement is positively associated with algorithm performance; that is, individual annotator disagreement is positively correlated with increased NMSE\. More specifically, our evidence indicates that, when compared to other annotators within the same dataset, each standard deviation of increased disagreement score from an individual annotator corresponds to an increase of over 0\.02 \(on average\) in the algorithms’ NMSE on that annotator\.

Since NMSE is normalized by the possible rating range of a dataset, this indicates a shift in MSE above 2% of the squared possible range of annotations\. For example, on datasets with a range of 100, this indicates an average shift of over 200 in the MSE for each standard deviation that an annotator’s disagreement score is higher than the mean disagreement score for that dataset\.

Our primary results \(see the Sparsity effect row of Table[1](https://arxiv.org/html/2610.22567#S5.T1)\) show no significant evidence of a positive association between response rate and algorithm performance\. This is a fairly unintuitive result, as our initial hypothesis proposed that less training data on a particular annotator would lead to worse performance\. However, it appears that for our tasks and algorithms, agreement is far more predictive of algorithm performance than response rate, and response rate may even have performance associations that are counter to our initial hypothesis\.

Coeff\.Median95% CIPos\. Coeff\. RateβD\\beta\_\{D\}0\.0248\[0\.0123, 0\.0396\]1\.0000βS\\beta\_\{S\}0\.0009\[\-0\.0028, 0\.0054\]0\.6554
Figure 2:Dataset\-family block bootstrap results from 10,000 samples each\. Top: histograms displaying the estimated regression slope across samples\. Left: disagreement effect\. Right: sparsity effect\. Bottom: 95% bootstrap confidence intervals and positive coefficient rates\. Positive coefficient rate denotes the proportion of bootstrap samples in which the coefficient estimate was positive\. The orange vertical line highlights the median value, and the black vertical line highlights a value of 0\. Our hypotheses predicted that both distributions would lie significantly to the right of the black line\.Figure 3:Forest plot from fitting the regression model from Equation[1](https://arxiv.org/html/2610.22567#S5.E1)with one family of algorithms left out of the analysis\. The figure displays the disagreement effect \(left\) and sparsity effect \(right\)\. Error bars indicate the 95% CI\.The post\-hoc analysis in Figure[2](https://arxiv.org/html/2610.22567#S6.F2)demonstrates that while specific dataset families have some impact on the disagreement coefficient during regression, the tested algorithms tend to display a statistically significant correlation between annotator disagreement and performance across all datasets\.

With regard to sparsity, we observe that, bootstrapping with dataset\-family blocks roughly mirrors our main results from Table[1](https://arxiv.org/html/2610.22567#S5.T1), showing a statistically insignificant negative relationship between annotator sparsity and algorithm performance\. It is worth noting that the 95% confidence interval contains 0, indicating that the direction of the response\-rate association with algorithm performance may be fairly dataset\-dependent\.

With regard to algorithm families, Figure[3](https://arxiv.org/html/2610.22567#S6.F3)demonstrates that no individual algorithm family significantly impacts the results for either ofβD\\beta\_\{D\}andβS\\beta\_\{S\}\.

## 7Conclusions

Overall, we find strong evidence supporting that individual annotator agreement has a positive correlation with algorithm performance for that annotator, but find that the number of annotations provided by an annotator during training does not have a statistically significant positive association with algorithm performance\. For AI alignment researchers, this provides evidence that existing algorithms for pluralist prediction of human behavior regarding opinions of text tend to be biased towards humans with popular opinions; more alignment work may be needed to ensure that future algorithms are aligned for people with unpopular opinions as well\. For dataset creators who want high model performance, this provides evidence towards focusing on annotator guidelines in order to increase agreement\. For privacy\-sensitive individuals, our results suggest that withholding additional data is unlikely to significantly harm performance for these personalized algorithms, though the correlational nature of our experiments cannot confirm this\.

## Limitations

While our work does provide, to the best of our knowledge, the largest collection to date of perspectivist datasets with ordinal annotations, it is not an exhaustive set\. We also restrict our domain to English\-only datasets since the Multitask and TabFM algorithms rely on English\-based text encoders\. Future work may include the addition of non\-English datasets and text encoders\.

Similarly, while we test multiple model architectures, prediction methods, and encoder implementations, there are many approaches to making individualized predictions on perspectivist NLP datasets that are not clearly covered by our selection of algorithms\.

In the interpretation of our results, we rely on the original creators of the datasets to have implemented proper safeguards to ensure that annotations faithfully represent annotators’ true judgments of text\. If the source datasets contain invalid annotations, that may impact the interpretation of our experimental data\.

In addition, our study is correlational, not interventional; if there are unmeasured factors that jointly impact annotator sparsity and rating behavior, then our conclusion regarding data\-sensitive individuals may not be correct, as we do not directly measure how models perform when training data is directly removed\. Similarly, we do not directly test increasing annotator agreement across datasets, so any unobserved sources of association may weaken our conclusion regarding suggestions for dataset creators\.

Initially, we hypothesized that higher annotator response rate would be positively associated with algorithm performance on those annotators\. Even after controlling for the relationship between the number of train and test examples in our regression model, the evidence collected did not support our initial hypothesis\. In order to retain scientific validity, we report the results of our investigation as\-is, but further research may provide additional insights\.

Overall, we tested 15 different algorithms split across 6 distinct algorithm families that are capable of making annotation predictions on a wide variety of perspectivist NLP datasets\. However, our results may not generalize beyond these particular algorithms and datasets, and dataset collection techniques may change the interpretation of our results\. We encourage other researchers and practitioners in this space to utilize our code and techniques to measure algorithmic sensitivity to annotator idiosyncrasy across their own unique datasets and algorithms and inform us of any conflicting results\.

## Acknowledgments

A significant portion of experiments were conducted on the Minnesota Supercomputing Institute \(MSI\) at the University of Minnesota\.

AI assistants were used significantly to help implement experiments, adapt models to different hardware, and suggest and critique aspects of the experimental design and interpretation\. All coding tasks were explicitly overseen by humans\. Output code was either reviewed by humans or validated through human\-designed and human\-verified unit and integration tests\.

AI assistants were used lightly during writing in order to provide suggestions for small individual phrases, correct grammar, create tables, and implement Appendix[A\.2](https://arxiv.org/html/2610.22567#A1.SS2)\. Figure[1](https://arxiv.org/html/2610.22567#S2.F1)was also generated via AI systems\. The authors bear full responsibility for any errors that appear in the final work\.

## References

- Abidet al\.\(2021\)A\. Abid, M\. Farooqi, and J\. ZouPersistent Anti\-Muslim Bias in Large Language Models\.InProc\. 2021 AAAIACM Conf\. AI Ethics Soc\.,AIES ’21,New York, NY, USA,pp\. 298–306\.External Links:[Document](https://dx.doi.org/10.1145/3461702.3462624)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px3.p2.1)\.
- Akhtaret al\.\(2021\)S\. Akhtar, V\. Basile, and V\. PattiWhose Opinions Matter? Perspective\-aware Models to Identify Opinions of Hate Speech Victims in Abusive Language Detection\.arXiv\.External Links:2106\.15896,[Document](https://dx.doi.org/10.48550/arXiv.2106.15896)Cited by:[Figure 4](https://arxiv.org/html/2610.22567#A1.F4),[§3](https://arxiv.org/html/2610.22567#S3.p2.1)\.
- Almanea and Poesio \(2022\)D\. Almanea and M\. PoesioArMIS \- The Arabic Misogyny and Sexism Corpus with Annotator Subjective Disagreements\.InProc\. Thirteen\. Lang\. Resour\. Eval\. Conf\.,N\. Calzolari, F\. Béchet, P\. Blache, K\. Choukri, C\. Cieri, T\. Declerck, S\. Goggi, H\. Isahara, B\. Maegaard, J\. Mariani, H\. Mazo, J\. Odijk, and S\. Piperidis \(Eds\.\),Marseille, France,pp\. 2282–2291\.Cited by:[Figure 4](https://arxiv.org/html/2610.22567#A1.F4)\.
- Aroyo and Welty \(2015\)L\. Aroyo and C\. WeltyTruth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation\.AI Mag\.36\(1\),pp\. 15–24\.External Links:ISSN 2371\-9621,[Document](https://dx.doi.org/10.1609/aimag.v36i1.2564)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px1.p1.1)\.
- Barocas and Selbst \(2016\)S\. Barocas and A\. D\. SelbstBig Data’s Disparate Impact\.Calif\. Law Rev\.\.External Links:[Document](https://dx.doi.org/10.15779/Z38BG31)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px1.p4.1)\.
- Basile \(2020\)V\. BasileIt’s the End of the Gold Standard as We Know It: Leveraging Non\-aggregated Data for Better Evaluation and Explanation of Subjective Tasks\.InAIxIA 2020 – Adv\. Artif\. Intell\. XIXth Int\. Conf\. Ital\. Assoc\. Artif\. Intell\. Virtual Event Novemb\. 25–27 2020 Revis\. Sel\. Pap\.,Berlin, Heidelberg,pp\. 441–453\.External Links:[Document](https://dx.doi.org/10.1007/978-3-030-77091-4%5F26)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px1.p1.1)\.
- Bavarescoet al\.\(2025\)A\. Bavaresco, R\. Bernardi, L\. Bertolazzi, D\. Elliott, R\. Fernández, A\. Gatt, E\. Ghaleb, M\. Giulianelli, M\. Hanna, A\. Koller, A\. Martins, P\. Mondorf, V\. Neplenbroek, S\. Pezzelle, B\. Plank, D\. Schlangen, A\. Suglia, A\. K\. Surikuchi, E\. Takmaz, and A\. TestoniLLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks\.InProc\. 63rd Annu\. Meet\. Assoc\. Comput\. Linguist\. Vol\. 2 Short Pap\.,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 238–255\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-short.20)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px5.p2.1)\.
- Bolukbasiet al\.\(2016\)T\. Bolukbasi, K\. Chang, J\. Y\. Zou, V\. Saligrama, and A\. T\. KalaiMan is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings\.InAdv\. Neural Inf\. Process\. Syst\.,Vol\.29\.Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px3.p2.1)\.
- Cabitzaet al\.\(2023\)F\. Cabitza, A\. Campagner, and V\. BasileToward a Perspectivist Turn in Ground Truthing for Predictive Computing\.Proc\. AAAI Conf\. Artif\. Intell\.37\(6\),pp\. 6860–6868\.External Links:ISSN 2374\-3468,[Document](https://dx.doi.org/10.1609/aaai.v37i6.25840)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px1.p1.1)\.
- Cameronet al\.\(2008\)A\. C\. Cameron, J\. B\. Gelbach, and D\. L\. MillerBootstrap\-Based Improvements for Inference with Clustered Errors\.Rev\. Econ\. Stat\.90\(3\),pp\. 414–427\.External Links:ISSN 0034\-6535,[Document](https://dx.doi.org/10.1162/rest.90.3.414)Cited by:[§5\.2](https://arxiv.org/html/2610.22567#S5.SS2.p3.1)\.
- Casolaet al\.\(2024\)S\. Casola, S\. Frenda, S\. Lo, E\. Sezerer, A\. Uva, V\. Basile, C\. Bosco, A\. Pedrani, C\. Rubagotti, V\. Patti, and D\. BernardiMultiPICo: Multilingual Perspectivist Irony Corpus\.InProc\. 62nd Annu\. Meet\. Assoc\. Comput\. Linguist\. Vol\. 1 Long Pap\.,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 16008–16021\.Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px4.p4.1)\.
- Castricatoet al\.\(2025\)L\. Castricato, N\. Lile, R\. Rafailov, J\. Fränken, and C\. FinnPERSONA: A Reproducible Testbed for Pluralistic Alignment\.InProc\. 31st Int\. Conf\. Comput\. Linguist\.,O\. Rambow, L\. Wanner, M\. Apidianaki, H\. Al\-Khalifa, B\. D\. Eugenio, and S\. Schockaert \(Eds\.\),Abu Dhabi, UAE,pp\. 11348–11368\.Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px1.p2.1),[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px5.p2.1)\.
- Cercas Curryet al\.\(2021\)A\. Cercas Curry, G\. Abercrombie, and V\. RieserConvAbuse: Data, Analysis, and Benchmarks for Nuanced Abuse Detection in Conversational AI\.InProc\. 2021 Conf\. Empir\. Methods Nat\. Lang\. Process\.,M\. Moens, X\. Huang, L\. Specia, and S\. W\. Yih \(Eds\.\),Online and Punta Cana, Dominican Republic,pp\. 7388–7403\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.587)Cited by:[§3](https://arxiv.org/html/2610.22567#S3.p2.1)\.
- Damgaardet al\.\(2021\)C\. Damgaard, P\. Toborek, T\. Eriksen, and B\. Plank“I’ll be there for you”: The One with Understanding Indirect Answers\.InProc\. 2nd Workshop Comput\. Approaches Discourse,Punta Cana, Dominican Republic and Online,pp\. 1–11\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.codi-main.1)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px4.p3.1)\.
- Danescu\-Niculescu\-Mizilet al\.\(2013\)C\. Danescu\-Niculescu\-Mizil, M\. Sudhof, D\. Jurafsky, J\. Leskovec, and C\. PottsA computational approach to politeness with application to social factors\.InProc\. 51st Annu\. Meet\. Assoc\. Comput\. Linguist\. Vol\. 1 Long Pap\.,H\. Schuetze, P\. Fung, and M\. Poesio \(Eds\.\),Sofia, Bulgaria,pp\. 250–259\.Cited by:[§3](https://arxiv.org/html/2610.22567#S3.p2.1)\.
- Davaniet al\.\(2022\)A\. M\. Davani, M\. Díaz, and V\. PrabhakaranDealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations\.Trans\. Assoc\. Comput\. Linguist\.10,pp\. 92–110\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00449)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px1.p2.1),[§4](https://arxiv.org/html/2610.22567#S4.p1.1)\.
- de Marneffeet al\.\(2019\)M\. de Marneffe, M\. Simons, and J\. TonhauserThe CommitmentBank: Investigating projection in naturally occurring discourse\.Proc\. Sinn Bedeut\.23\(2\),pp\. 107–124\.External Links:ISSN 2629\-6055,[Document](https://dx.doi.org/10.18148/sub/2019.v23i2.601)Cited by:[§3](https://arxiv.org/html/2610.22567#S3.p2.1)\.
- Demszkyet al\.\(2020\)D\. Demszky, D\. Movshovitz\-Attias, J\. Ko, A\. Cowen, G\. Nemade, and S\. RaviGoEmotions: A Dataset of Fine\-Grained Emotions\.InProc\. 58th Annu\. Meet\. Assoc\. Comput\. Linguist\.,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 4040–4054\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.372)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px4.p3.1)\.
- Denget al\.\(2023\)N\. Deng, X\. Zhang, S\. Liu, W\. Wu, L\. Wang, and R\. MihalceaYou Are What You Annotate: Towards Better Models through Annotator Representations\.InFind\. Assoc\. Comput\. Linguist\. EMNLP 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 12475–12498\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.832)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px4.p3.1)\.
- Devlinet al\.\(2019\)J\. Devlin, M\. Chang, K\. Lee, and K\. ToutanovaBERT: Pre\-training of Deep Bidirectional Transformers for Language Understanding\.InProc\. 2019 Conf\. North Am\. Chapter Assoc\. Comput\. Linguist\. Hum\. Lang\. Technol\. Vol\. 1 Long Short Pap\.,J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Minneapolis, Minnesota,pp\. 4171–4186\.External Links:[Document](https://dx.doi.org/10.18653/v1/N19-1423)Cited by:[§A\.1](https://arxiv.org/html/2610.22567#A1.SS1.p2.1)\.
- Diazet al\.\(2018\)M\. Diaz, I\. Johnson, A\. Lazar, A\. M\. Piper, and D\. GergleAddressing Age\-Related Bias in Sentiment Analysis\.InProc\. 2018 CHI Conf\. Hum\. Factors Comput\. Syst\.,CHI ’18,New York, NY, USA,pp\. 1–14\.External Links:[Document](https://dx.doi.org/10.1145/3173574.3173986)Cited by:[§3](https://arxiv.org/html/2610.22567#S3.p2.1)\.
- Diaz \(2020\)M\. DiazAge Bias Training and Testing Data\.Harvard Dataverse\.External Links:[Document](https://dx.doi.org/10.7910/DVN/F6EMTS)Cited by:[§3](https://arxiv.org/html/2610.22567#S3.p2.1)\.
- Dinuet al\.\(2021\)L\. P\. Dinu, I\. Iordache, A\. S\. Uban, and M\. ZampieriA Computational Exploration of Pejorative Language in Social Media\.InFind\. Assoc\. Comput\. Linguist\. EMNLP 2021,Punta Cana, Dominican Republic,pp\. 3493–3498\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.findings-emnlp.296)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px4.p3.1)\.
- Do \(2022\)Q\. DoMatrix Factorization\.Cited by:[§4](https://arxiv.org/html/2610.22567#S4.p1.1)\.
- Eckmanet al\.\(2025\)S\. Eckman, B\. Ma, C\. Kern, R\. Chew, B\. Plank, and F\. KreuterAligning NLP Models with Target Population Perspectives using PAIR: Population\-Aligned Instance Replication\.InProc\. 4th Workshop Perspect\. Approaches NLP,G\. Abercrombie, V\. Basile, S\. Frenda, S\. Tonelli, and S\. Dudy \(Eds\.\),Suzhou, China,pp\. 100–110\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.nlperspectives-1.9)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px3.p1.1)\.
- Ekstrandet al\.\(2011\)M\. D\. Ekstrand, J\. T\. Riedl, and J\. A\. KonstanCollaborative Filtering Recommender Systems\.Found\. Trends Hum\.\-Comput\. Interact\.4\(2\),pp\. 81–173\.External Links:ISSN 1551\-3955,[Document](https://dx.doi.org/10.1561/1100000009)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px5.p1.1)\.
- Fleisiget al\.\(2024\)E\. Fleisig, S\. L\. Blodgett, D\. Klein, and Z\. TalatThe Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels\.InProc\. 2024 Conf\. North Am\. Chapter Assoc\. Comput\. Linguist\. Hum\. Lang\. Technol\. Vol\. 1 Long Pap\.,K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 2279–2292\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.126)Cited by:[§1](https://arxiv.org/html/2610.22567#S1.p1.1)\.
- Forbeset al\.\(2020\)M\. Forbes, J\. D\. Hwang, V\. Shwartz, M\. Sap, and Y\. ChoiSocial Chemistry 101: Learning to Reason about Social and Moral Norms\.InProc\. 2020 Conf\. Empir\. Methods Nat\. Lang\. Process\. EMNLP,B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 653–670\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.48)Cited by:[§B\.1](https://arxiv.org/html/2610.22567#A2.SS1.p2.1),[§3](https://arxiv.org/html/2610.22567#S3.p2.1)\.
- Frendaet al\.\(2025\)S\. Frenda, G\. Abercrombie, V\. Basile, A\. Pedrani, R\. Panizzon, A\. T\. Cignarella, C\. Marco, and D\. BernardiPerspectivist approaches to natural language processing: a survey\.Lang\. Resour\. Eval\.59\(2\),pp\. 1719–1746\.External Links:ISSN 1574\-0218,[Document](https://dx.doi.org/10.1007/s10579-024-09766-4)Cited by:[§1](https://arxiv.org/html/2610.22567#S1.p1.1),[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px1.p3.1)\.
- Gabriel \(2020\)I\. GabrielArtificial Intelligence, Values, and Alignment\.Minds Mach\.30\(3\),pp\. 411–437\.External Links:ISSN 1572\-8641,[Document](https://dx.doi.org/10.1007/s11023-020-09539-2)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px2.p1.1)\.
- Gordonet al\.\(2022\)M\. L\. Gordon, M\. S\. Lam, J\. S\. Park, K\. Patel, J\. Hancock, T\. Hashimoto, and M\. S\. BernsteinJury Learning: Integrating Dissenting Voices into Machine Learning Models\.InProc\. 2022 CHI Conf\. Hum\. Factors Comput\. Syst\.,CHI ’22,New York, NY, USA,pp\. 1–19\.External Links:[Document](https://dx.doi.org/10.1145/3491102.3502004)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px1.p4.1),[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px3.p1.1)\.
- Gultchinet al\.\(2019\)L\. Gultchin, G\. Patterson, N\. Baym, N\. Swinger, and A\. KalaiHumor in Word Embeddings: Cockamamie Gobbledegook for Nincompoops\.InProc\. 36th Int\. Conf\. Mach\. Learn\.,pp\. 2474–2483\.External Links:ISSN 2640\-3498Cited by:[§3](https://arxiv.org/html/2610.22567#S3.p2.1)\.
- Heet al\.\(2017\)X\. He, L\. Liao, H\. Zhang, L\. Nie, X\. Hu, and T\. ChuaNeural Collaborative Filtering\.InProc\. 26th Int\. Conf\. World Wide Web,WWW ’17,Republic and Canton of Geneva, CHE,pp\. 173–182\.External Links:[Document](https://dx.doi.org/10.1145/3038912.3052569)Cited by:[§4](https://arxiv.org/html/2610.22567#S4.p1.1)\.
- Heinischet al\.\(2023\)P\. Heinisch, M\. Orlikowski, J\. Romberg, and P\. CimianoArchitectural Sweet Spots for Modeling Human Label Variation by the Example of Argument Quality: It’s Best to Relate Perspectives\!\.InProc\. 2023 Conf\. Empir\. Methods Nat\. Lang\. Process\.,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 11138–11154\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.687)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px6.p1.1)\.
- Homayouniradet al\.\(2025\)A\. Homayounirad, E\. Liscio, T\. Wang, C\. M\. Jonker, and L\. C\. SiebertWill Annotators Disagree? Identifying Subjectivity in Value\-Laden Arguments\.InFind\. Assoc\. Comput\. Linguist\. EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 15237–15252\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.824)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px1.p2.1)\.
- Jang and Frassinelli \(2024\)H\. Jang and D\. FrassinelliGeneralizable Sarcasm Detection is Just Around the Corner, of Course\!\.InProc\. 2024 Conf\. North Am\. Chapter Assoc\. Comput\. Linguist\. Hum\. Lang\. Technol\. Vol\. 1 Long Pap\.,K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 4238–4249\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.238)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px4.p4.1)\.
- Jernigan and Mistree \(2009\)C\. Jernigan and B\. F\. T\. MistreeGaydar: Facebook friendships expose sexual orientation\.First Monday\.External Links:ISSN 1396\-0466,[Document](https://dx.doi.org/10.5210/fm.v14i10.2611)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px1.p4.1)\.
- Kazienkoet al\.\(2023\)P\. Kazienko, J\. Bielaniewicz, M\. Gruza, K\. Kanclerz, K\. Karanowski, P\. Miłkowski, and J\. KocońHuman\-centered neural reasoning for subjective content processing: Hate speech, emotions, and humor\.Inf\. Fusion94,pp\. 43–65\.External Links:ISSN 1566\-2535,[Document](https://dx.doi.org/10.1016/j.inffus.2023.01.010)Cited by:[§3](https://arxiv.org/html/2610.22567#S3.p2.1)\.
- Kennedyet al\.\(2022\)B\. Kennedy, M\. Atari, A\. M\. Davani, L\. Yeh, A\. Omrani, Y\. Kim, K\. Coombs, S\. Havaldar, G\. Portillo\-Wightman, E\. Gonzalez, J\. Hoover, A\. Azatian, A\. Hussain, A\. Lara, G\. Cardenas, A\. Omary, C\. Park, X\. Wang, C\. Wijaya, Y\. Zhang, B\. Meyerowitz, and M\. DehghaniIntroducing the Gab Hate Corpus: defining and applying hate\-based rhetoric to social media posts at scale\.Lang\. Resour\. Eval\.56\(1\),pp\. 79–108\.External Links:ISSN 1574\-0218,[Document](https://dx.doi.org/10.1007/s10579-021-09569-x)Cited by:[§3](https://arxiv.org/html/2610.22567#S3.p2.1)\.
- Kong and Das \(2026\)W\. Kong and A\. DasIntroducing TabFM: A zero\-shot foundation model for tabular data\.Cited by:[§A\.4](https://arxiv.org/html/2610.22567#A1.SS4.p1.1),[§4](https://arxiv.org/html/2610.22567#S4.p1.1)\.
- Lanet al\.\(2020\)Z\. Lan, M\. Chen, S\. Goodman, K\. Gimpel, P\. Sharma, and R\. SoricutALBERT: A Lite BERT for Self\-supervised Learning of Language Representations\.InInternational Conference on Learning Representations,Cited by:[§A\.1](https://arxiv.org/html/2610.22567#A1.SS1.p2.1)\.
- Leonardelliet al\.\(2023\)E\. Leonardelli, G\. Abercrombie, D\. Almanea, V\. Basile, T\. Fornaciari, B\. Plank, V\. Rieser, A\. Uma, and M\. PoesioSemEval\-2023 Task 11: Learning with Disagreements \(LeWiDi\)\.InProc\. 17th Int\. Workshop Semantic Eval\. SemEval\-2023,A\. Kr\. Ojha, A\. S\. Doğruöz, G\. Da San Martino, H\. Tayyar Madabushi, R\. Kumar, and E\. Sartori \(Eds\.\),Toronto, Canada,pp\. 2304–2318\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.semeval-1.314)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px1.p3.1),[§3](https://arxiv.org/html/2610.22567#S3.p2.1)\.
- Leonardelliet al\.\(2025\)E\. Leonardelli, S\. Casola, S\. Peng, G\. Rizzi, V\. Basile, E\. Fersini, D\. Frassinelli, H\. Jang, M\. Pavlovic, B\. Plank, and M\. PoesioLeWiDi\-2025 at NLPerspectives: Third Edition of the Learning with Disagreements Shared Task\.InProc\. 4th Workshop Perspect\. Approaches NLP,G\. Abercrombie, V\. Basile, S\. Frenda, S\. Tonelli, and S\. Dudy \(Eds\.\),Suzhou, China,pp\. 182–195\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.nlperspectives-1.16)Cited by:[§1](https://arxiv.org/html/2610.22567#S1.p1.1),[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px4.p4.1)\.
- Leonardelliet al\.\(2021\)E\. Leonardelli, S\. Menini, A\. Palmero Aprosio, M\. Guerini, and S\. TonelliAgreeing to Disagree: Annotating Offensive Language Datasets with Annotators’ Disagreement\.InProc\. 2021 Conf\. Empir\. Methods Nat\. Lang\. Process\.,M\. Moens, X\. Huang, L\. Specia, and S\. W\. Yih \(Eds\.\),Online and Punta Cana, Dominican Republic,pp\. 10528–10539\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.822)Cited by:[§3](https://arxiv.org/html/2610.22567#S3.p2.1)\.
- Liuet al\.\(2019\)Y\. Liu, M\. Ott, N\. Goyal, J\. Du, M\. Joshi, D\. Chen, O\. Levy, M\. Lewis, L\. Zettlemoyer, and V\. StoyanovRoBERTa: A Robustly Optimized BERT Pretraining Approach\.arXiv\.External Links:1907\.11692,[Document](https://dx.doi.org/10.48550/arXiv.1907.11692)Cited by:[§A\.1](https://arxiv.org/html/2610.22567#A1.SS1.p2.1)\.
- Lowmanstoneet al\.\(2023\)L\. Lowmanstone, R\. Wan, R\. Owan, J\. Kim, and D\. KangAnnotation Imputation to Individualize Predictions: Initial Studies on Distribution Dynamics and Model Predictions\.2nd Workshop Perspect\. Approaches NLP3494\.Cited by:[§A\.1](https://arxiv.org/html/2610.22567#A1.SS1.p1.1),[§A\.1](https://arxiv.org/html/2610.22567#A1.SS1.p3.1),[§B\.1](https://arxiv.org/html/2610.22567#A2.SS1.p1.1),[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px1.p3.1),[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px5.p2.1),[§3](https://arxiv.org/html/2610.22567#S3.p2.1),[§3](https://arxiv.org/html/2610.22567#S3.p3.1),[§3](https://arxiv.org/html/2610.22567#S3.p4.1),[§4](https://arxiv.org/html/2610.22567#S4.p2.1),[§4](https://arxiv.org/html/2610.22567#S4.p3.1)\.
- Marreddyet al\.\(2025\)M\. Marreddy, S\. R\. Oota, V\. C\. Chinni, M\. Gupta, and L\. FlekUSDC: A Dataset of User Stance and Dogmatism in Long Conversations\.InFind\. Assoc\. Comput\. Linguist\. ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 23715–23759\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.1216)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px5.p2.1)\.
- Meisteret al\.\(2025\)N\. Meister, C\. Guestrin, and T\. HashimotoBenchmarking Distributional Alignment of Large Language Models\.InProc\. 2025 Conf\. Nations Am\. Chapter Assoc\. Comput\. Linguist\. Hum\. Lang\. Technol\. Vol\. 1 Long Pap\.,L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 24–49\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.2)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px1.p4.1)\.
- Milkowskiet al\.\(2021\)P\. Milkowski, M\. Gruza, K\. Kanclerz, P\. Kazienko, D\. Grimling, and J\. KoconPersonal Bias in Prediction of Emotions Elicited by Textual Opinions\.InProc\. 59th Annu\. Meet\. Assoc\. Comput\. Linguist\. 11th Int\. Jt\. Conf\. Nat\. Lang\. Process\. Stud\. Res\. Workshop,Online,pp\. 248–259\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.acl-srw.26)Cited by:[§3](https://arxiv.org/html/2610.22567#S3.p2.1)\.
- Nadeemet al\.\(2021\)M\. Nadeem, A\. Bethke, and S\. ReddyStereoSet: Measuring stereotypical bias in pretrained language models\.InProc\. 59th Annu\. Meet\. Assoc\. Comput\. Linguist\. 11th Int\. Jt\. Conf\. Nat\. Lang\. Process\. Vol\. 1 Long Pap\.,C\. Zong, F\. Xia, W\. Li, and R\. Navigli \(Eds\.\),Online,pp\. 5356–5371\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.416)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px3.p2.1)\.
- Nangiaet al\.\(2020\)N\. Nangia, C\. Vania, R\. Bhalerao, and S\. R\. BowmanCrowS\-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models\.InProc\. 2020 Conf\. Empir\. Methods Nat\. Lang\. Process\. EMNLP,B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 1953–1967\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.154)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px3.p2.1)\.
- Parrishet al\.\(2022\)A\. Parrish, A\. Chen, N\. Nangia, V\. Padmakumar, J\. Phang, J\. Thompson, P\. M\. Htut, and S\. R\. BowmanBBQ: A hand\-built bias benchmark for question answering\.InFind\. Assoc\. Comput\. Linguist\. ACL 2022,S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 2086–2105\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.findings-acl.165)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px3.p2.1)\.
- Patty and Penn \(2023\)J\. W\. Patty and E\. M\. PennAlgorithmic Fairness and Statistical Discrimination\.Philos\. Compass18\(1\),pp\. e12891\.External Links:ISSN 1747\-9991, 1747\-9991,[Document](https://dx.doi.org/10.1111/phc3.12891)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px1.p4.1)\.
- Pessach and Shmueli \(2023\)D\. Pessach and E\. ShmueliA Review on Fairness in Machine Learning\.ACM Comput\. Surv\.55\(3\),pp\. 1–44\.External Links:ISSN 0360\-0300, 1557\-7341,[Document](https://dx.doi.org/10.1145/3494672)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px3.p1.1)\.
- Plank \(2022\)B\. PlankThe “Problem” of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation\.InProc\. 2022 Conf\. Empir\. Methods Nat\. Lang\. Process\.,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 10671–10682\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.731)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px4.p2.1),[§3](https://arxiv.org/html/2610.22567#S3.p2.1)\.
- Pottset al\.\(2021\)C\. Potts, Z\. Wu, A\. Geiger, and D\. KielaDynaSent: A Dynamic Benchmark for Sentiment Analysis\.InProc\. 59th Annu\. Meet\. Assoc\. Comput\. Linguist\. 11th Int\. Jt\. Conf\. Nat\. Lang\. Process\. Vol\. 1 Long Pap\.,C\. Zong, F\. Xia, W\. Li, and R\. Navigli \(Eds\.\),Online,pp\. 2388–2404\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.186)Cited by:[§3](https://arxiv.org/html/2610.22567#S3.p2.1)\.
- Radfordet al\.\(2019\)A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, and I\. SutskeverLanguage models are unsupervised multitask learners\.Technical reportOpenAI\.Cited by:[§A\.1](https://arxiv.org/html/2610.22567#A1.SS1.p2.1)\.
- Rendleet al\.\(2020\)S\. Rendle, W\. Krichene, L\. Zhang, and J\. AndersonNeural Collaborative Filtering vs\. Matrix Factorization Revisited\.InProc\. 14th ACM Conf\. Recomm\. Syst\.,RecSys ’20,New York, NY, USA,pp\. 240–248\.External Links:[Document](https://dx.doi.org/10.1145/3383313.3412488)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px5.p1.1)\.
- Röttgeret al\.\(2022\)P\. Röttger, B\. Vidgen, D\. Hovy, and J\. PierrehumbertTwo Contrasting Data Annotation Paradigms for Subjective NLP Tasks\.InProc\. 2022 Conf\. North Am\. Chapter Assoc\. Comput\. Linguist\. Hum\. Lang\. Technol\.,M\. Carpuat, M\. de Marneffe, and I\. V\. Meza Ruiz \(Eds\.\),Seattle, United States,pp\. 175–190\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.naacl-main.13)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px1.p2.1)\.
- Sachdevaet al\.\(2022\)P\. S\. Sachdeva, R\. Barreto, G\. Bacon, A\. Sahn, C\. von Vacano, and C\. KennedyThe Measuring Hate Speech Corpus: Leveraging Rasch Measurement Theory for Data Perspectivism\.InProc\. 1st Workshop Perspect\. Approaches NLP LREC2022,G\. Abercrombie, V\. Basile, S\. Tonelli, V\. Rieser, and A\. Uma \(Eds\.\),Marseille, France,pp\. 83–94\.Cited by:[§3](https://arxiv.org/html/2610.22567#S3.p2.1)\.
- Sanhet al\.\(2019\)V\. Sanh, L\. Debut, J\. Chaumond, and T\. WolfDistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter\.InProc\. 5th Workshop Energy Effic\. Mach\. Learn\. Cogn\. Comput\.,NeurIPS 2019 Workshop\.External Links:1910\.01108Cited by:[§A\.1](https://arxiv.org/html/2610.22567#A1.SS1.p2.1)\.
- Sapet al\.\(2020\)M\. Sap, S\. Gabriel, L\. Qin, D\. Jurafsky, N\. A\. Smith, and Y\. ChoiSocial Bias Frames: Reasoning about Social and Power Implications of Language\.InProc\. 58th Annu\. Meet\. Assoc\. Comput\. Linguist\.,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 5477–5490\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.486)Cited by:[§3](https://arxiv.org/html/2610.22567#S3.p2.1)\.
- Sapet al\.\(2022\)M\. Sap, S\. Swayamdipta, L\. Vianna, X\. Zhou, Y\. Choi, and N\. SmithAnnotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection\.InProc\. 2022 Conf\. North Am\. Chapter Assoc\. Comput\. Linguist\. Hum\. Lang\. Technol\.,Seattle, United States,pp\. 5884–5906\.Cited by:[§3](https://arxiv.org/html/2610.22567#S3.p2.1)\.
- Sarumiet al\.\(2024\)O\. O\. Sarumi, B\. Neuendorf, J\. Plepi, L\. Flek, J\. Schlötterer, and C\. WelchCorpus Considerations for Annotator Modeling and Scaling\.InProc\. 2024 Conf\. North Am\. Chapter Assoc\. Comput\. Linguist\. Hum\. Lang\. Technol\. Vol\. 1 Long Pap\.,K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 1029–1040\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.59)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px5.p2.1)\.
- Sarumiet al\.\(2025\)O\. O\. Sarumi, C\. Welch, and D\. BraunNLP\-ResTeam at LeWiDi\-2025:Performance Shifts in Perspective Aware Models based on Evaluation Metrics\.InProc\. 4th Workshop Perspect\. Approaches NLP,G\. Abercrombie, V\. Basile, S\. Frenda, S\. Tonelli, and S\. Dudy \(Eds\.\),Suzhou, China,pp\. 219–227\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.nlperspectives-1.19)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px6.p1.1)\.
- Shiet al\.\(2014\)Y\. Shi, M\. Larson, and A\. HanjalicCollaborative Filtering beyond the User\-Item Matrix: A Survey of the State of the Art and Future Challenges\.ACM Comput\. Surv\.47\(1\),pp\. 3:1–3:45\.External Links:ISSN 0360\-0300,[Document](https://dx.doi.org/10.1145/2556270)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px5.p1.1)\.
- Sorensenet al\.\(2024a\)T\. Sorensen, L\. Jiang, J\. D\. Hwang, S\. Levine, V\. Pyatkin, P\. West, N\. Dziri, X\. Lu, K\. Rao, C\. Bhagavatula, M\. Sap, J\. Tasioulas, and Y\. ChoiValue Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties\.Proc\. AAAI Conf\. Artif\. Intell\.38\(18\),pp\. 19937–19947\.External Links:ISSN 2374\-3468,[Document](https://dx.doi.org/10.1609/aaai.v38i18.29970)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px1.p4.1)\.
- Sorensenet al\.\(2024b\)T\. Sorensen, J\. Moore, J\. Fisher, M\. Gordon, N\. Mireshghallah, C\. M\. Rytting, A\. Ye, L\. Jiang, X\. Lu, N\. Dziri, T\. Althoff, and Y\. ChoiPosition: a roadmap to pluralistic alignment\.InProc\. 41st Int\. Conf\. Mach\. Learn\.,ICML’24, Vol\.235,Vienna, Austria,pp\. 46280–46302\.Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px1.p2.1),[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px2.p1.1)\.
- Strapparava and Mihalcea \(2007\)C\. Strapparava and R\. MihalceaSemEval\-2007 task 14: affective text\.InProc\. 4th Int\. Workshop Semantic Eval\. \- SemEval 07,Prague, Czech Republic,pp\. 70–74\.External Links:[Document](https://dx.doi.org/10.3115/1621474.1621487)Cited by:[§3](https://arxiv.org/html/2610.22567#S3.p2.1)\.
- Su and Khoshgoftaar \(2009\)X\. Su and T\. M\. KhoshgoftaarA Survey of Collaborative Filtering Techniques\.Adv\. Artif\. Intell\.2009,pp\. 1–19\.External Links:ISSN 1687\-7470, 1687\-7489,[Document](https://dx.doi.org/10.1155/2009/421425)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px5.p1.1)\.
- Sullivanet al\.\(2023\)M\. Sullivan, M\. Yasin, and C\. L\. JacobsUniversity at Buffalo at SemEval\-2023 Task 11: MASDA–Modelling Annotator Sensibilities through DisAggregation\.InProc\. 17th Int\. Workshop Semantic Eval\. SemEval\-2023,A\. Kr\. Ojha, A\. S\. Doğruöz, G\. Da San Martino, H\. Tayyar Madabushi, R\. Kumar, and E\. Sartori \(Eds\.\),Toronto, Canada,pp\. 978–985\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.semeval-1.135)Cited by:[§A\.2](https://arxiv.org/html/2610.22567#A1.SS2.SSS0.Px1.p1.1),[§A\.2](https://arxiv.org/html/2610.22567#A1.SS2.p1.1),[§A\.3](https://arxiv.org/html/2610.22567#A1.SS3.p1.1),[§4](https://arxiv.org/html/2610.22567#S4.p1.1)\.
- Thainet al\.\(2017\)N\. Thain, L\. Dixon, and E\. WulczynWikipedia Talk Labels: Toxicity\.figshare\.External Links:[Document](https://dx.doi.org/10.6084/m9.figshare.4563973.v2)Cited by:[Table 4](https://arxiv.org/html/2610.22567#A2.T4.2.5.1.2.1.3.1)\.
- Wulczynet al\.\(2017a\)E\. Wulczyn, N\. Thain, and L\. DixonEx Machina: Personal Attacks Seen at Scale\.InProc\. 26th Int\. Conf\. World Wide Web,WWW ’17,Republic and Canton of Geneva, CHE,pp\. 1391–1399\.External Links:[Document](https://dx.doi.org/10.1145/3038912.3052591)Cited by:[§3](https://arxiv.org/html/2610.22567#S3.p2.1)\.
- Wulczynet al\.\(2017b\)E\. Wulczyn, N\. Thain, and L\. DixonWikipedia Talk Labels: Aggression\.figshare\.External Links:[Document](https://dx.doi.org/10.6084/m9.figshare.4267550.v5)Cited by:[Table 4](https://arxiv.org/html/2610.22567#A2.T4.2.5.1.2.1.2.1)\.
- Wulczynet al\.\(2017c\)E\. Wulczyn, N\. Thain, and L\. DixonWikipedia Talk Labels: Personal Attacks\.figshare\.External Links:[Document](https://dx.doi.org/10.6084/m9.figshare.4054689.v6)Cited by:[Table 4](https://arxiv.org/html/2610.22567#A2.T4.2.5.1.2.1.2.1)\.
- Zhanget al\.\(2025\)T\. Zhang, C\. Zhang, S\. Li, and C\. L\. P\. ChenFrom Disagreement to Unity: A Cascade Perspective Network for Subjective Tasks\.IEEE Trans\. Comput\. Soc\. Syst\.12\(5\),pp\. 2752–2764\.External Links:ISSN 2329\-924X,[Document](https://dx.doi.org/10.1109/TCSS.2025.3529746)Cited by:[§2](https://arxiv.org/html/2610.22567#S2.SS0.SSS0.Px1.p2.1)\.

## Appendix ASupplementary Algorithm Information

The algorithms used, their corresponding family, and parameter counts are listed in Table[2](https://arxiv.org/html/2610.22567#A1.T2)\. Model parameter counts vary per dataset with the amount of annotators and annotation value range; we list the maximum number of parameters of each model across all training sessions in the corresponding column\.

Total estimated GPU use for these experiments is approximately 76\.81 hours, with roughly 48\.87 hours of computation taking place on A100 GPUs and 27\.94 hours taking place on H100 GPUs\.

AlgorithmFamily\#ParamExample MeanMean0Global MeanMean0User MeanMean0KMFKMF672,815MASDA\-CategoricalMASDA23,868,264MASDA\-OrdinalMASDA23,868,222Multitask\-ALBERTMultitask15,893,859Multitask\-BERTMultitask113,692,515Multitask\-DistilBERTMultitask70,573,155Multitask\-GPT\-2Multitask128,650,083Multitask\-RoBERTaMultitask128,855,907TabFM\-ColumnTabFM1,647,782,989TabFM\-Column\+BERTTabFM1,757,265,229TabFM\-CellTabFM1,647,782,989NCFNCF7,313,921Table 2:Algorithms used in the experiments, grouped by source family, with maximum parameter counts across the experiment datasets\.### A\.1Multitask Implementation

We improve upon the Multitask model provided by[Lowmanstone et al\. \(2023\)](https://arxiv.org/html/2610.22567#bib.bib45)in a few key areas\. First, we implement model checkpointing in order to reduce overfitting during model training\. The publicly released code base from[Lowmanstone et al\. \(2023\)](https://arxiv.org/html/2610.22567#bib.bib45)utilized a 10% internal validation set for monitoring and logging during training Multitask models\. This validation set was created using the same process illustrated in the main paper in Figure[1](https://arxiv.org/html/2610.22567#S2.F1)\. We implemented a checkpoint mechanism to fully utilize this validation data; the final prediction model was chosen to be the model with the lowest loss on the validation data, as measured after each of 30 training epochs\. This behavior differs from[Lowmanstone et al\. \(2023\)](https://arxiv.org/html/2610.22567#bib.bib45), which used the model trained for a fixed 10 epochs for prediction regardless of validation performance\.

Next, in addition to the single BERT\-base\-uncased encoder, we test multiple different text encoding models, which are detailed in Table[2](https://arxiv.org/html/2610.22567#A1.T2)\. Due to architectural differences in encoding, models from the BERT family \(BERT\([Devlin et al\., 2019](https://arxiv.org/html/2610.22567#bib.bib19)\), ALBERT\([Lan et al\., 2020](https://arxiv.org/html/2610.22567#bib.bib40)\), DistilBERT\([Sanh et al\., 2019](https://arxiv.org/html/2610.22567#bib.bib61)\), and RoBERTa\([Liu et al\., 2019](https://arxiv.org/html/2610.22567#bib.bib44)\)\) pass the representation of the first token in the sequence \(i\.e\. the classification token\) to the rest of the model, whereas the GPT\-2 encoder\([Radford et al\., 2019](https://arxiv.org/html/2610.22567#bib.bib57)\)passes the representation of the last non\-padding token\. Since we utilize multiple encoders with various properties, the context window was adjusted to the maximum size for each model, rather than being fixed at 512\. To reduce memory usage, we also implemented an adaptive padding mechanism which pads each example up to the length of the longest example in its batch, rather than the full context length\.

Finally, we adjusted the output range of the Multitask algorithm to match the training data\. The original implementation from[Lowmanstone et al\. \(2023\)](https://arxiv.org/html/2610.22567#bib.bib45)assumed that all dataset ranges had a minimum annotation value of 0 and used an architecture that could output annotation predictions of 0 even on datasets whose lowest annotation was above 0\. We modified the architecture by no longer including extraneous output units that corresponded to those unseen annotations in the training data\.

### A\.2MASDA Implementation

We implemented the MASDA model from[Sullivan et al\. \(2023\)](https://arxiv.org/html/2610.22567#bib.bib71)as a prediction algorithm for our datasets\. The original MASDA code was implemented for usage on binary datasets only\. Since our datasets have ordinal ratings that are not binary, we implemented two different variations on the MASDA architecture to adapt it to our particular task\. The first variation, MASDA\-Categorical, replaces the original scalar logit decoder with a class logit decoder and exchanges the original binary cross\-entropy loss for standard cross\-entropy\. The MASDA\-Ordinal model predicts normalized ratings using the sigmoid of the scalar decoder output and computes MSE loss against labels normalized to the unit interval; predictions are then mapped back to the dataset\-specific annotation scale\. We describe both architectures below in detail\.

#### Shared MASDA architecture\.

The MASDA architecture from[Sullivan et al\. \(2023\)](https://arxiv.org/html/2610.22567#bib.bib71)consists of a shared text encoder, annotator\-specific latent networks, a shared decoder, and an optional gold\-label prediction pathway\. We implemented two adaptations of this architecture for our ordinal annotation prediction setting: MASDA\-Ordinal and MASDA\-Categorical\. We describe both using a shared template\.

Let textiihave inputxix\_\{i\}, and letri​jr\_\{ij\}be the rating provided by annotatorjj\. Let missing ratings be marked byRMISSR\_\{\\mathrm\{MISS\}\}, and define the observed\-cell indicator

oi​j=𝟏\[ri​j≠RMISS\]\.o\_\{ij\}=\\mathbf\{1\}\[r\_\{ij\}\\neq R\_\{\\mathrm\{MISS\}\}\]\.The text encoder first produces a text representation:

Hi=LM⁡\(xi\),hi=MeanPool⁡\(Hi\)\.H\_\{i\}=\\mathrm\{LM\}\(x\_\{i\}\),\\qquad h\_\{i\}=\\mathrm\{MeanPool\}\(H\_\{i\}\)\.Each annotator\-specific network then maps this text representation into an annotator\-specific latent representation:

ui​j=Fj​\(hi\),u\_\{ij\}=F\_\{j\}\(h\_\{i\}\),whereFjF\_\{j\}is the network associated with annotatorjj\. The shared decoder maps this latent representation into either a scalar logit or a vector of class logits, depending on the MASDA variant:

zi​j=Decθ​\(ui​j\)\.z\_\{ij\}=\\mathrm\{Dec\}\_\{\\theta\}\(u\_\{ij\}\)\.For variants that use the gold\-label pathway, the model also computes attention weights over annotators,

ωi=softmax⁡\(Wg​hi\),\\omega\_\{i\}=\\mathrm\{softmax\}\(W\_\{g\}h\_\{i\}\),forms a weighted latent representation,

gi=∑jωi​j​ui​j,g\_\{i\}=\\sum\_\{j\}\\omega\_\{ij\}u\_\{ij\},and decodes it:

zi\(g\)=Decθ​\(gi\)\.z\_\{i\}^\{\(g\)\}=\\mathrm\{Dec\}\_\{\\theta\}\(g\_\{i\}\)\.
During training, we detachhih\_\{i\}and eachui​ju\_\{ij\}before passing them to the gold\-label pathway\. Consequently, the gold\-label loss updates the attention module and shared decoder, but its gradients do not flow back into the text encoder or annotator\-specific networks\.

#### MASDA\-Ordinal\.

MASDA\-Ordinal preserves the scalar\-logit decoder used by the original MASDA implementation, but interprets the decoder output as a position on the observed annotation scale rather than as a binary\-classification logit\. Let

rmin\\displaystyle r\_\{\\min\}=min⁡\{ri​j:oi​j=1\},\\displaystyle=\\min\\\{r\_\{ij\}:o\_\{ij\}=1\\\},rmax\\displaystyle r\_\{\\max\}=max⁡\{ri​j:oi​j=1\},and\\displaystyle=\\max\\\{r\_\{ij\}:o\_\{ij\}=1\\\},\\text\{ and\}s\\displaystyle s=rmax−rmin\.\\displaystyle=r\_\{\\max\}\-r\_\{\\min\}\.
Observed ratings are normalized to the unit interval:

r~i​j=ri​j−rmins\.\\widetilde\{r\}\_\{ij\}=\\frac\{r\_\{ij\}\-r\_\{\\min\}\}\{s\}\.The model predicts a normalized rating using the sigmoid of the scalar decoder output:

r~^i​j=σ⁡\(zi​j\),\\widehat\{\\widetilde\{r\}\}\_\{ij\}=\\sigma\(z\_\{ij\}\),and maps it back to the dataset\-specific rating scale:

r^i​j=rmin\+s​r~^i​j\.\\widehat\{r\}\_\{ij\}=r\_\{\\min\}\+s\\widehat\{\\widetilde\{r\}\}\_\{ij\}\.The MASDA\-Ordinal loss is masked mean squared error on the normalized labels:

Lord=1∑i,joi​j​∑i∑joi​j​\(σ⁡\(zi​j\)−r~i​j\)2\.L\_\{\\mathrm\{ord\}\}=\\frac\{1\}\{\\sum\_\{i,j\}o\_\{ij\}\}\\sum\_\{i\}\\sum\_\{j\}o\_\{ij\}\\left\(\\sigma\(z\_\{ij\}\)\-\\widetilde\{r\}\_\{ij\}\\right\)^\{2\}\.The gold\-label pathway is not used for MASDA\-Ordinal\. During imputation, observed cells retain their original ratings and missing cells are filled with the model prediction:

ri​jimp=\{ri​j,oi​j=1,r^i​j,oi​j=0\.r^\{\\mathrm\{imp\}\}\_\{ij\}=\\begin\{cases\}r\_\{ij\},&o\_\{ij\}=1,\\\\ \\widehat\{r\}\_\{ij\},&o\_\{ij\}=0\.\\end\{cases\}

#### MASDA\-Categorical\.

MASDA\-Categorical instead treats each rating value as a distinct class\. For observed ratings \(oi​j=1o\_\{ij\}=1\), define the zero\-based class indexci​j=ri​j−rmin∈\{0,…,K−1\}c\_\{ij\}=r\_\{ij\}\-r\_\{\\min\}\\in\\\{0,\\dots,K\-1\\\}\. The decoder therefore outputs a vector of class logits:

zi​j∈ℝK,zi\(g\)∈ℝK,z\_\{ij\}\\in\\mathbb\{R\}^\{K\},\\qquad z\_\{i\}^\{\(g\)\}\\in\\mathbb\{R\}^\{K\},whereKKis the number of classes\. The predicted rating is

r^i​j=rmin\+arg⁡maxc∈\{0,…,K−1\}⁡zi​j,c\.\\widehat\{r\}\_\{ij\}=r\_\{\\min\}\+\\arg\\max\_\{c\\in\\\{0,\\dots,K\-1\\\}\}z\_\{ij,c\}\.The gold\-label target for the gold pathway is the row plurality class:

ti\(g\)=argmaxc∑j:oi​j=1𝟏\[ci​j=c\]\.t\_\{i\}^\{\(g\)\}=\\arg\\max\_\{c\}\\sum\_\{j:o\_\{ij\}=1\}\\mathbf\{1\}\[c\_\{ij\}=c\]\.The annotator\-specific loss is masked cross\-entropy:

Lann=1∑i,joi​j∑i,j:oi​j=1CE\(zi​j,ci​j\)\.L\_\{\\mathrm\{ann\}\}=\\frac\{1\}\{\\sum\_\{i,j\}o\_\{ij\}\}\\sum\_\{i,j:o\_\{ij\}=1\}\\mathrm\{CE\}\(z\_\{ij\},c\_\{ij\}\)\.LetIobs=\{i:∃j,oi​j=1\}I\_\{\\mathrm\{obs\}\}=\\\{i:\\exists j,\\;o\_\{ij\}=1\\\}be the set of rows with at least one observed annotation, and letNobs=\|Iobs\|N\_\{\\mathrm\{obs\}\}=\|I\_\{\\mathrm\{obs\}\}\|\. The gold\-label loss is

Lgold=1Nobs​∑i∈IobsCE⁡\(zi\(g\),ti\(g\)\)\.L\_\{\\mathrm\{gold\}\}=\\frac\{1\}\{N\_\{\\mathrm\{obs\}\}\}\\sum\_\{i\\in I\_\{\\mathrm\{obs\}\}\}\\mathrm\{CE\}\(z\_\{i\}^\{\(g\)\},t\_\{i\}^\{\(g\)\}\)\.The total MASDA\-Categorical loss is

Lcat=Lann\+λgold​Lgold,L\_\{\\mathrm\{cat\}\}=L\_\{\\mathrm\{ann\}\}\+\\lambda\_\{\\mathrm\{gold\}\}L\_\{\\mathrm\{gold\}\},whereλgold\\lambda\_\{\\mathrm\{gold\}\}is the gold\-loss weight; in our experiments, we use the default valueλgold=1\.0\\lambda\_\{\\mathrm\{gold\}\}=1\.0\. As with MASDA\-Ordinal, imputation preserves observed ratings and fills missing cells with the model prediction:

ri​jimp=\{ri​j,oi​j=1,r^i​j,oi​j=0\.r^\{\\mathrm\{imp\}\}\_\{ij\}=\\begin\{cases\}r\_\{ij\},&o\_\{ij\}=1,\\\\ \\widehat\{r\}\_\{ij\},&o\_\{ij\}=0\.\\end\{cases\}

### A\.3Performance Comparison

Figure 4:Soft\-label cross\-entropy scores of the original MASDA model and our ordinal and categorical MASDA variants on the ArMIS\([Almanea and Poesio, 2022](https://arxiv.org/html/2610.22567#bib.bib3)\)and HS\-Brexit\([Akhtar et al\., 2021](https://arxiv.org/html/2610.22567#bib.bib2)\)datasets\. Lower scores indicate better performance\. Scores for our variants were computed on held\-out annotations, while the original MASDA scores are those reported in the original paper and were not recomputed using our train/test splits\.The performance of the two MASDA model implementations is compared against the original results reported by[Sullivan et al\.](https://arxiv.org/html/2610.22567#bib.bib71)in Figure[4](https://arxiv.org/html/2610.22567#A1.F4)\. Note that this is not a direct comparison, as the cross\-entropy loss on models implemented by this work are measured on the test data derived from this work, which does not directly match the data the original model was tested on\.

Both variations suffered noticeable losses of performance in comparison to the original MASDA implementation, which may be attributable to our unique evaluation environment\. However, these additional models provide useful evidence that our analysis holds across multiple different algorithmic approaches to individualized prediction\.

We include MASDA\-Categorical because, despite its poor performance on cross\-entropy, Figure[8](https://arxiv.org/html/2610.22567#A3.F8)demonstrates that it can outperform MASDA\-Ordinal on a select few datasets\. However, the MASDA\-Categorical model can suffer from prediction collapse and be penalized highly as seen in Figure[8](https://arxiv.org/html/2610.22567#A3.F8), especially on datasets such as AffectiveText ANGER, which has a highly skewed annotation distribution \(as observed from Figure[5](https://arxiv.org/html/2610.22567#A2.F5)\) despite having a wide range of possible annotation values \(see Table[3](https://arxiv.org/html/2610.22567#A2.T3)\)\.

### A\.4TabFM Model Implementation

The TabFM model from[Kong and Das \(2026\)](https://arxiv.org/html/2610.22567#bib.bib39)is a recently\-released imputation method for tabular data\. As a general description, the TabFM algorithm has two phases: fit, and predict\.

#### Fit Phase\.

During the fit phase, the model is not fine\-tuned\. Instead it processes the given data in order to feed it to a fixed neural architecture during the predict stage\. The fit stage expects two inputs: 1\. A Training Matrix \(X\_context\), which is a matrix where each row corresponds to an example, and each column contains features\. 2\. A Target Array \(y\_context\), which is a vector with one element for each row in the train context; each element is the target prediction for its corresponding row in the Training Matrix\.

#### Predict Phase\.

The predict phase takes as input a Prediction Matrix \(X\_predict\) with the same number of columns as the Training Matrix; each row corresponds to asking the model to make a prediction given the features in that row\. The model will then provide a Prediction Array output with one element for each row in the Prediction Matrix\.

#### Additional Information\.

In both phases, missing feature entries in the Training Matrix and Prediction Matrix are, by default, automatically replaced with the mean of existing values from the Training Matrix for that feature\.

In some scenarios \(8\.67% of predictions for TabFM\-Column and 1\.81% of predictions for TabFM\-Column\+BERT\), the input features do not vary across rows in the Training Matrix\. This can occur either when an annotator only has one example in the train split or when all other annotators who annotated the same examples as the target annotator gave the exact same annotations for each shared example\. In these scenarios, we conduct a “mean fallback”: instead of running TabFM \(which would error on such an input\), we instead predict the mean of the Target Array\.

We use regression mode with all the default hyperparameters from the released TabFM code, with no hyperparameter search\. Conducting a hyperparameter search is likely to improve performance\. However, in order to conduct such a search, we either need to reserve data for validation which will not be used for training or overfit the TabFM model to our suite of datasets, reducing the value of our experimental analysis\. Given these tradeoffs and that the models performed relatively well \(see Figure[7](https://arxiv.org/html/2610.22567#A3.F7)\), we have currently opted to use the default hyperparameter settings\.

#### TabFM\-Column\.

For TabFM\-Column, we create one Training Matrix for each unique annotatoraain the test split\. This Training Matrix is constructed by first removing all rows from the train split matrix \(assuming no text column\) in which annotatoraadoes not provide an annotation\. We then remove the column corresponding toaafrom the resulting matrix and use that removed column as the Target Array\. The remaining matrix is passed as the Training Matrix\. The Prediction Matrix is created similarly, by removing all rows for which no prediction for annotatoraais requested\. Then, the column corresponding to annotatoraais removed, and the resulting matrix is passed as the Prediction Matrix\. TabFM will then make a prediction for annotatoraafor each row in the Prediction Matrix\. We repeat this process for each unique annotator in the test split\.

#### TabFM\-Column\+BERT\.

The TabFM\-Column\+BERT model is identical to the TabFM\-Column implementation, except for that we add extra features to the train split data\. Before removing the text column, we encode each of the texts with BERT\-base\-uncased, resulting in a 768\-dimensional CLS vector for each row\. We then add 768 columns to each row, each column corresponding to a component of the text embedding from BERT\. The rest of the process remains identical\. This provides textual features to the TabFM model, enabling it to make predictions based on the textual data\. Since these features will vary over multiple examples, the mean fallback was only used for annotators whose training data consisted of a single example\. It is worth noting that the default TabFM implementation which we use samples only 500 features at a time\. This low upper limit on the number of features may limit the viability of this approach, and may introduce a tradeoff between using text and other annotators’ ratings in order to predict a target annotator’s ratings\.

#### TabFM\-Cell\.

For this implementation, instead of each annotator being the main unit, each cell \(rating\) within the matrix is the main unit for data transformation\. To create the Training Matrix we build one row for each ratingri​ar\_\{ia\}in the train split, whereiiindexes the example andaaindexes the annotator\. The row in the Training Matrix corresponding to ratingri​ar\_\{ia\}consists of the original rowiifrom the train split with a few modifications\. We remove the ratingri​ar\_\{ia\}from the specified rowiiand place it into the Target Array, replacing it with the missing sentinel value\. Then, we remove the text column from the row, and prepend the numeric ID of annotatoraato the row as an additional feature column\. Thus, the Training Matrix will have rows equal to the number of ratings in the training data, and columns equal to the number of annotators plus one \(for the ID of annotatoraa\)\. The Target Array will have one element for each rating in the training data\. The Prediction Matrix is constructed analogously to the Training Matrix, except the step of removing the rating and placing it into a Target Array is no longer needed, since the rating is unknown \(to the algorithm\) by definition\.

### A\.5Mean Prediction Algorithms Implementation

To further expand algorithmic diversity, we also include baseline mean prediction algorithms\. User Mean predicts the average annotation of each column in the train dataset, Example Mean predicts the average annotation of each row in the train dataset, and Global Mean predicts the average value across all annotations in the train dataset\. While most algorithm families outperform this family of baseline mean algorithms \(see Figure[7](https://arxiv.org/html/2610.22567#A3.F7)\), we find that mean algorithms \(especially Example Mean\) can outperform all other algorithms on particular datasets, especially when measured using normalized root mean squared error \(NRMSE\), as shown in Appendix[C\.1](https://arxiv.org/html/2610.22567#A3.SS1)\.

## Appendix BSupplementary Dataset Information

DatasetSourceNMRatings/ExampleLabel Meaning\(Other\)×\\times6VALENCEAffectiveText987987666\.06\.0Scale from absent \(0\) to maximum \(100\)Highly negative \(0\), Highly positive \(200\)Cockamamie GobbledegookHuBi100003513\.6Not funny \(0\), Funny \(1\)\(Other\)×\\times8EMOT\. AROUSALVALENCEHuBi/Personal Bias1001001005005005008\.58\.58\.5Scale from low \(3\) to high \(7\)Scale from weak \(3\) to strong \(7\)Scale from negative \(0\) to positive \(6\)Wiki\-AggressionWiki\-AttackWiki\-ToxicityHuBi/Wikipedia Detox7303730371014884884901\.81\.81\.6Friendly/neutral \(0\), Aggressive \(1\)Not attack/harassment \(0\), Att\./harass\. \(1\)Healthy/neutral \(0\), Toxic \(1\)SChemSChem5LabelsNLPerspectives400786110010250\.05\.0No one believes \(0\),Occasionally believed \(1\),Controversial \(2\), Common belief \(3\),Universally true \(4\)SBICNLPerspectives100002042\.5Not offensive \(0\), Maybe \(1\), Offensive \(2\)GHCNLPerspectives10000183\.1Not hate speech \(0\), Hate speech \(1\)SentimentNLPerspectives84115001\.7Very negative \(0\), Somewhat negative \(1\),Neutral \(2\), Somewhat positive \(3\),Very positive \(4\)PolitenessNLPerspectives/Stanford PolitenessCorpus43382195\.0Scale from very impolite \(0\)to very polite \(24\)ConvAbuseLeWiDi239883\.0Strongly negative with overt incitementto hatred, violence or discrimination;attitude oriented at attacking ordemeaning the target \(0\),Negative and insulting/abusive,aggressive attitude \(1\),Negative and impolite, mildly offensivebut still conversational \(2\),Ambiguous, unclear \(3\),Non\-abusive \(4\)HS\-BrexitLeWiDi78466\.0Not hate speech \(0\), Hate speech \(h\.s\.\) \(1\)MD\-AgreementLeWiDi65915003\.9Offensive \(0\), Non\-offensive \(1\)CommitmentBankAHLV/CommitmentBank12004969\.6Certainly false \(0\), Uncertain \(3\),Certainly true \(6\)MHS\-Sentiment\(Other\)×\\times7MHS\-Attack/DefenseMHS\-Hate SpeechAHLV/Measuring HateSpeech13621362136213624944944944941\.61\.61\.61\.6Scale from very neg\. \(0\) to very pos\. \(4\)Scale from low \(0\) to high \(4\)Scale from defense \(0\) to attack \(4\)Not h\.s\. \(0\), Unclear \(1\), H\.s\. \(2\)\(Other\)×\\times2AwA\-IntentAwA\-RacismAHLV/Annotators withAttitudes6266266261841841845\.55\.55\.5Scale from not offensive \(1\) to very off\. \(5\)Scale from unintentional \(1\) to intent\. \(5\)Scale from not racist \(1\) to very racist \(5\)DynaSentAHLV/DynaSent66783851\.5Negative \(0\), Neutral \(1\), Positive \(2\)Table 3:Information and statistics on the 46 datasets used in our experiment\. The statistics include the number of unique text instances \(N\), the number of unique annotators \(M\), and the average number of ratings per unique text\. Within each source section, “\(Other\)×\\timesK” represents K datasets with identical statistics and label meanings\.Dataset SourceDataset\(s\)TermsAffectiveTextANGERDISGUSTFEARJOYSADNESSSURPRISEVALENCE[Public Release](https://web.eecs.umich.edu/~mihalcea/affectivetext/)HuBiCockamamie Gobbledegook[Public Release](https://github.com/limorigu/Cockamamie-Gobbledegook)HuBi/Personal BiasANGERANTICIPATIONDISGUSTEMOT\. AROUSALFEARJOYSADNESSSURPRISETRUSTVALENCE[Public Release](https://github.com/CLARIN-PL/personal-bias)HuBi/Wikipedia Detox\([Wulczyn et al\., 2017b](https://arxiv.org/html/2610.22567#bib.bib75);[Wulczyn et al\., 2017c](https://arxiv.org/html/2610.22567#bib.bib74)\)\([Thain et al\., 2017](https://arxiv.org/html/2610.22567#bib.bib72)\)Wiki\-AggressionWiki\-AttackWiki\-Toxicity[CC0 1\.0](https://figshare.com/articles/dataset/Wikipedia_Talk_Labels_Aggression/4267550)[CC0 1\.0](https://figshare.com/articles/dataset/Wikipedia_Talk_Labels_Personal_Attacks/4054689)[CC0 1\.0](https://figshare.com/articles/dataset/Wikipedia_Talk_Labels_Toxicity/4563973)NLPerspectivesSChemSChem5LabelsSBICGHCSentiment[CC BY\-SA 4\.0](https://github.com/mbforbes/social-chemistry-101)[CC BY\-SA 4\.0](https://github.com/mbforbes/social-chemistry-101)[CC BY 4\.0](https://huggingface.co/datasets/allenai/social_bias_frames)[CC BY 4\.0](https://osf.io/edua3/metadata)[CC0 1\.0](https://doi.org/10.7910/DVN/F6EMTS)NLPerspectives/Stanford Politeness CorpusPoliteness[CC BY 4\.0](https://convokit.cornell.edu/documentation/stack_politeness.html)LeWiDiConvAbuseHS\-BrexitMD\-Agreement[CC BY 4\.0](https://github.com/amandacurry/convabuse/blob/main/LICENSE)[Public Release](https://github.com/Le-Wi-Di/le-wi-di.github.io)[Research Use Only](https://github.com/dhfbk/annotators-agreement-dataset?tab=readme-ov-file)AHLV/CommitmentBankCommitmentBank[Public Release](https://github.com/mcdm/CommitmentBank)AHLV/Measuring Hate SpeechMHS\-Sentiment, MHS\-RespectMHS\-Insult, MHS\-HumiliationMHS\-Status, MHS\-DehumanizationMHS\-Violence, MHS\-GenocideMHS\-Attack/Defense, MHS\-Hate Speech[CC BY 4\.0](https://huggingface.co/datasets/ucberkeley-dlab/measuring-hate-speech)AHLV/Annotators with AttitudesAwA\-Offensive to YouAwA\-Offensive to AnyoneAwA\-IntentAwA\-Racism[Qualified Researchers](https://maartensap.com/racial-bias-hatespeech/annWithAttitudes-Qual.html)AHLV/DynaSentDynaSent[CC BY 4\.0](https://github.com/cgpotts/dynasent)Table 4:Publicly stated licenses, access restrictions, or availability terms for English\-language dataset sources used in our experiments\.A full enumeration of datasets used can be found in Table[3](https://arxiv.org/html/2610.22567#A2.T3), while Table[4](https://arxiv.org/html/2610.22567#A2.T4)lists the publicly stated licenses, access restrictions, or availability terms associated with the dataset sources used in our experiments\.

Figure 5:Smoothed density curves of annotation values across datasets with train data \(top left\), test data \(top right\), and both train and test data combined \(bottom\)\.Figure[5](https://arxiv.org/html/2610.22567#A2.F5)shows the distribution of annotation values across all datasets\. We can observe large tails on each end of the distribution which are amplified by the presence of binary datasets\.

Figure 6:Distribution of agreement and annotator response rate across all datasets\. Datasets with the same source and annotation patterns are grouped together when graphing response rate\.Figure[6](https://arxiv.org/html/2610.22567#A2.F6)displays the distribution of agreement and annotator response rate for each dataset\.

The smoothed density curves in the figure are computed over\[0,1\]\[0,1\]by forming a 40\-bin density\-normalized histogram, smoothing it with a normalized discrete Gaussian\-like kernel over integer offsets from \-4 to 4 with scale 1\.25, and interpolating the result onto a 400\-point plotting grid\. Near\-constant distributions are visualized with a narrow normalized Gaussian fallback \(bandwidth 0\.015\) centered at the observed value\.

### B\.1SChem vs\. SChem5Labels

In this section, we clarify the relationship between the SChem and SChem5Labels datasets here, as we do not believe dataset provenance was covered adequately by the description given in[Lowmanstone et al\. \(2023\)](https://arxiv.org/html/2610.22567#bib.bib45)\.

The SChem and SChem5Labels datasets both come from[Forbes et al\. \(2020\)](https://arxiv.org/html/2610.22567#bib.bib27)\. SChem5Labels is a cleaned version of the “dev\-extra" and “test\-extra" splits of the released dataset merged together; each of these examples has 5 annotations per example\.

SChem consists of highly controversial examples \(i\.e\. high annotator disagreement across the initial 5 annotators\) from the “analysis" split of the dataset, which were annotated with an additional 50 annotations per example\. Thus, while SChem examples are a direct subset of SChem5Labels \(before downsampling\), unlike other datasets with the same source, the two SChem datasets do not share annotators\. Due to the overlap in examples and rating guidelines, we group them into the same dataset family when computing dataset\-family blocks for the bootstrapping analysis\.

## Appendix CSupplementary Post\-Hoc Analysis

### C\.1Model Performance

Figure 7:Forest plots of per\-algorithm normalized mean squared error \(top left\), normalized root mean squared error \(top right\), and a bar chart of normalized mean squared error averaged over each algorithm family \(bottom\)\. The error bars in the forest plots show bootstrapped 95% confidence intervals based on 1,000 resamples of individual datasets\.![Refer to caption](https://arxiv.org/html/2610.22567v1/nrmse_by_dataset_and_algorithm.png)Figure 8:Per\-model normalized root mean squared error performance on each dataset\. Datasets are ordered from top to bottom in order of average performance across algorithms\. Algorithms are ordered from left to right by average performance across datasets\. Here, we use NRMSE \(as opposed to NMSE\) for better interpretability\.![Refer to caption](https://arxiv.org/html/2610.22567v1/nrmse_by_dataset_and_algorithm_family.png)Figure 9:Average per\-model\-family normalized root mean squared error performance on each dataset\. Similar to Figure[8](https://arxiv.org/html/2610.22567#A3.F8), datasets are ordered from top to bottom by performance across algorithm families, algorithms are ordered from left to right by average performance across datasets, and we use NRMSE for stronger interpretability\.Overall performance comparisons between algorithm families can be found in Figure[7](https://arxiv.org/html/2610.22567#A3.F7)\. Heatmaps showing algorithm performance on individual datasets can be found in Figures[8](https://arxiv.org/html/2610.22567#A3.F8)and[9](https://arxiv.org/html/2610.22567#A3.F9)\.

### C\.2Two\-Sided Test

EffectCoef\.95% CITwo\-sidedppHolm\-adjustedppDisagreement \(βD\\beta\_\{D\}onZd​aDZ^\{D\}\_\{da\}\)0\.025\[0\.016, 0\.033\]3\.80×𝟏𝟎−𝟖\\mathbf\{3\.80\\times 10^\{\-8\}\}7\.61×𝟏𝟎−𝟖\\mathbf\{7\.61\\times 10^\{\-8\}\}Sparsity \(βS\\beta\_\{S\}onZd​aSZ^\{S\}\_\{da\}\)0\.001\[\-0\.001, 0\.003\]0\.3170\.317Table 5:Two\-sided post hoc regression results for the disagreement and sparsity effects in the regression shown in Equation[1](https://arxiv.org/html/2610.22567#S5.E1)in the main paper\. The main hypothesis tests shown in Table[1](https://arxiv.org/html/2610.22567#S5.T1)in the main paper use the predetermined one\-sided tests forβD\>0\\beta\_\{D\}\>0andβS\>0\\beta\_\{S\}\>0; these two\-sided results are reported only to characterize the direction and robustness of the estimated effects\. Results which are statistically significant are shown inbold\.The supplementary two\-sided post hoc regression results for the disagreement and sparsity/response\-rate effects are provided in Table[5](https://arxiv.org/html/2610.22567#A3.T5)\.