多数票压制少数价值观：HateXplain中仇恨/冒犯边界的标注者分歧

# 在HateXplain中标注者分歧集中于仇恨/冒犯边界

来源：https://arxiv.org/html/2606.28772

## 多数投票消音了少数价值观：HateXplain中标注者在仇恨/冒犯边界的分歧

###### 摘要

仇恨言论标注流程通常在训练前将标注者分歧折叠为多数投票标签。我们证明这种聚合并非中立：HateXplain中42.6%的标注者分歧集中在仇恨/冒犯边界，这一模式与标注者对仇恨起始阈值持有不同标准一致（χ²=135.199, df=2, p<0.0001）。硬标签BERT模型（模型A）和软标签模型（模型B）在一致帖子上准确率约为80%，在分歧帖子上准确率降至约58%，均下降22个百分点，p<0.0001。每个标注者多头模型（模型C）将这一差距进一步扩大到28个百分点，并将冒犯类分歧准确率降至0.245。关键的是，模型A在边界案例错误上的置信度显著高于模型C（0.710 vs. 0.495, p<0.0001），这意味着标准评估指标无法检测到这一失败。三种复杂度递增的下游干预均未能恢复边界准确率。我们认为问题在于结构性的：多数投票将有争议的判断呈现为真实标签，模型继承了这种虚假的确定性。干预必须在标注设计的上游进行。

多元对齐，标注者分歧，仇恨言论，软标签，多数投票，价值观多元主义，BERT

## 1 引言

仇恨言论检测并非像“检测垃圾邮件”或“识别图片中的猫”那样纯粹的经验性或技术性类别。一条帖子是仇恨性的还是仅仅是冒犯性的，并不是客观事实。这是一种判断，取决于谁在阅读、他们属于哪个社区以及他们对伤害持有的阈值。然而，标准的标注流程将这种判断视为客观的：众包工作者标记每条帖子，多数获胜，然后将结果标签作为真实标签提供给模型。Plank (2022) 将这一假设称为“人类标注变异问题”：将真正的变异视为需要最小化的噪声，而非需要保留的信号。

本文探究当多数获胜时我们失去了什么。我们不提出新的模型或训练目标，而是精确定位HateXplain中标注者分歧集中在何处，描述这种集中对模型行为的影响，并展示三种复杂度递增的下游干预均未能恢复多数投票丢弃的信息。

我们的核心发现是：HateXplain中42.6%的标注者分歧发生在仇恨/冒犯边界上（χ²=135.199, df=2, p<0.0001）。这种集中并非随机噪声，而是边界上的结构性模式——标注者必须判断内容是否从冒犯性跨越到仇恨性。这一判断取决于每个标注者将阈值放在冒犯与仇恨之间的位置，而多数投票通过消音落在少数派的标注者来解决问题。

对部署后的模型来说，后果令人担忧。模型A和B从一致帖子到分歧帖子的准确率均下降超过22个百分点（p<0.0001）。模型C（每个标注者头部，我们测试的最强基线）将差距扩大到28个百分点，并将冒犯类分歧准确率降至0.245。模型A在边界案例错误上的置信度也显著高于模型C（0.710 vs. 0.495, p<0.0001）。一个在面对文化上有争议的输入时错误*且*确定的系统，是一种标准准确率指标无法检测的危险失败模式。

#### 贡献。

(1) HateXplain中的标注者分歧并非均匀分布：42.6%集中在仇恨/冒犯价值边界（χ²=135.199, df=2, p<0.0001），其中冒犯帖子的分歧率为67.9%，而正常帖子为35.5%。(2) 在所有三种训练体制下，一致帖子与分歧帖子之间的准确率差距持续存在22个百分点（p<0.0001）。(3) 三种下游干预（硬标签、软标签、每个标注者头部）均未能恢复边界准确率；在最强基线下降差距扩大。(4) 硬标签模型在边界案例错误上的置信度过高，显著高于每个标注者模型（0.710 vs. 0.495, p<0.0001），这证实了问题是结构性的，而非架构性的。

## 2 相关工作

#### 标注中的分歧。

Uma et al. (2021) 调查了跨NLP任务中从分歧中学习的方法，显示软标签方法通常能改善校准。Plank (2022) 将标注者分歧重新定义为人类标注变异（真正信号而非噪声），并列举了保留这种变异的方法。Davani et al. (2022) 提出了与我们最接近的论点，显示多数投票在主观任务上丢失信号，而每个标注者模型能恢复少数视角。Gordon et al. (2021) 引入了分歧反卷积，显示标准ML指标在主观任务上系统性地夸大性能，因为它们评估的是聚合标签而非单个标注者，这一发现启发了我们的子群分析。Gordon et al. (2022) 通过陪审学习扩展了这一工作，这是一种为每个标注者建模的架构，允许实践者在推理时显式加权人口群体。我们的工作有三个不同之处：我们*测量*分歧集中的位置，而不是提出新的聚合机制；我们用精确的统计显示这种集中是类别特定的；我们直接测试了每个标注者方法，发现它在边界案例上扩大了而不是缩小了一致性差距。

#### 仇恨言论检测。

Mathew et al. (2021) 介绍了HateXplain，包含仇恨言论、冒犯性言论和正常内容，并附有理由标注。他们注意到仇恨/冒犯区分是最难的分类边界，但并未量化它占分歧的比例，也未说明其对模型置信度的后果。Leonardelli et al. (2021) 研究了情感标注中的分歧，主张保留标注者观点，但未检查过度自信作为下游后果。Fleisig et al. (2023) 认为当声明所针对的人口群体在标注者池中是少数时，多数投票可能系统性地错误，并显示对目标群体评分建模能恢复聚合丢弃的信号。我们的工作将这些线索实证联系起来。

#### 多元对齐。

Sorensen et al. (2024) 将多元对齐定义为构建代表多样化人类价值观而非多数共识的AI系统的挑战。Aroyo et al. (2023) 通过DICES数据集将此关注操作化用于对话安全，该数据集保留了评分者人口统计信息，并使用分级安全量表以允许明确研究聚合选择如何影响结果。当前的RLHF流程面临类似的聚合问题：人类偏好比较被折叠成奖励信号，可能将多数评分者价值观编码为普遍价值观。我们在此不测试RLHF系统，但我们的发现为这一理论关注提供了具体基础，并为未来工作提出了可测试的假设。

## 3 方法

### 3.1 数据集

我们使用HateXplain (Mathew et al., 2021)，从原始GitHub仓库加载以保留原始标注者标签。20,148条帖子中，每条由三名众包工作者标注，标签为三种之一：hatespeech, normal, 或 offensive。我们使用官方的训练/验证/测试划分。所有分类实验在测试集上进行（n=1,892）；分歧特征化统计在整个数据集上计算。

我们将*分歧*定义为任何标注者未一致同意的帖子（唯一标签计数>1）。*边界案例*是原始标签为{hatespeech, offensive}的子集的分歧帖子：没有标注者选择normal，但标注者在两种伤害类别之间存在分歧的帖子。软标签分布直接从原始标注者标签计算，不进行任何聚合。

### 3.2 模型

我们训练三个复杂度递增的BERT-base-uncased (Devlin et al., 2019)分类器。

模型A（硬标签）使用标准交叉熵损失在多数投票标签上训练，这是仇恨言论分类的默认流程。

模型B（软标签）使用KL散度损失在来自于标注者分布的软标签上训练（例如，2-1的仇恨/冒犯划分产生[0.67, 0.0, 0.33]）。

模型C（每个标注者）使用共享BERT编码器，每个符合条件的标注者（有≥50条训练帖子的106个标注者）对应一个单独的分类头。训练时，每个样本路由到其标注者的特定头。推理时，所有头通过平均其softmax概率进行集成。这遵循Davani et al. (2022)的方法，构成我们最强的下游基线。

所有模型训练3个epoch，使用AdamW（lr=2×10⁻⁵），线性预热占10%步数，梯度裁剪为1.0。

### 3.3 统计分析

组间比较使用Mann-Whitney U检验（双尾，α=0.05）。卡方检验评估不同标签类别之间的分歧率是否有差异。我们报告所有声明的精确检验统计量、自由度和p值。

## 4 结果

### 4.1 分歧集中在价值边界

在15,383条训练帖子中，48.7%显示标注者分歧，使其成为数据集的结构性特征而非边缘特征。在测试集分歧帖子中（n=950），42.6%是边界案例，其中标注者仅在hatespeech和offensive标签之间选择（n=405）。

卡方检验确认不同标签类别之间的分歧率存在显著差异（χ²=135.199, df=2, p<0.0001）。冒犯帖子的分歧率为67.9%（372/548），仇恨帖子为50.5%（300/594），正常帖子为35.5%（278/782）。仇恨/冒犯边界是标注者判断最系统性地分歧的地方，而不仅仅是最难的分类问题。这种集中与标注者之间真实的阈值分歧一致：分歧恰好聚集在区分伤害类别需要对严重性进行判断的点上（图1）。

参考图标题

图1：左：训练集中标注者一致性水平（48.7%显示分歧）。中：按一致性类型划分的标签分布。冒犯帖子在分歧案例中占比过高。右：分歧帖子的软标签空间，显示离散的概率质量集中在仇恨/冒犯边界。

### 4.2 一致性差距在所有三个模型中持续存在

表1和图2展示了核心结果。所有三个模型在一致帖子上的表现（约80%）显著优于在分歧帖子上的表现（约55-58%）。通过Mann-Whitney U检验，所有三个模型的一致性差距得到确认，p<0.0001。标准总体准确率（69-70%）完全掩盖了这一差距，呼应了Gordon et al. (2021) 关于主观任务上聚合指标隐藏差距的记录。

在最强基线下降*差距扩大*。模型C（每个标注者头部）在一致帖子上取得了最佳准确率（0.826），但在分歧帖子上取得了最差准确率（0.545），产生28.1个百分点的最大一致性差距。在模型C下，冒犯类分歧准确率降至0.245，不到模型A本已较低的0.454的一半。没有下游干预能够恢复边界性能。

表1：所有三个模型在HateXplain测试集上的结果（n=1,892）。n.s. = 在α=0.05时不显著。少数对齐在第4.4节讨论。

参考图标题

图2：三种下游干预均未能恢复边界准确率。左：按一致性水平的准确率。一致帖子准确率在各模型间稳定（约80%），而分歧帖子准确率在模型C下进一步下降（0.545 vs. 模型A的0.579），将差距扩大到28个百分点。中：按分歧类别的准确率。仇恨类分歧在各模型间相对稳定，但在每个标注者头部下冒犯类分歧准确率降至0.245，确认冒犯边界是关键失败点。右：边界错误上的置信度（箱线图，均值用菱形标记）。模型A明显过度自信（0.710），而模型C接近适当的不确定性（0.495, p<0.0001），但校准收益以准确率为代价。

### 4.3 软标签未解决差距

模型B在任何分歧子群上的表现都不显著优于模型A。边界案例差异（0.605 vs. 0.595）的p=0.284。仇恨类分歧（0.730 vs. 0.743）的p=0.236。冒犯类分歧略微偏向模型B（0.462 vs. 0.454），但p=0.152。没有一项通过显著性检验。

### 4.4 每个标注者头部：更诚实，但不更准确

模型C产生了三个模型中最具信息量的模式。在一致帖子上，它取得了最佳准确率（0.826），确认共享编码器学习了强的通用表示。但在分歧帖子上，它表现最差（0.545），在冒犯类分歧上降至0.245。每个标注者头部学习了单个标注者的决策边界，这些边界无法泛化到标注者真正存在分歧的有争议案例。

模型C的唯一有意义收获是校准。边界错误置信度从模型A（0.710）到模型B（0.648）到模型C（0.495）逐步下降，p<0.0001得到确认（图2，右）。模型C在其无法分类的帖子上表现出适当的不确定性，尽管这种不确定性付出了其他方面准确率的代价。

少数对齐率讲述了一个同样微妙的叙事（图3）。所有三个模型都强烈偏向多数：在分歧帖子上，大约70%的预测与多数标签一致，只有30%与任何少数标注者对齐。一致性差距从模型A（0.222）到模型B（0.225）到模型C（0.281）单调扩大，确认没有下游干预能够弥合多数投票造成的结构性差距。

#### 关于推理规则的说明。

模型C通过平均softmax概率集成其每个标注者头部，这是我们所遵循的原始每个标注者提案中使用的推理规则 (Davani et al., 2022)。在我们的实验中，我们保持了这一规则，但注意到其他聚合方法（如投票）可能会产生不同的结果。我们选择平均概率，因为它是最常用的集成方法，且允许模型表达不确定性。然而，即使使用投票，我们也观察到类似模式（未显示），表明结果对推理规则的选择具有鲁棒性。## Annotator Disagreement at the Hate/Offensive Boundary in HateXplain
Source: https://arxiv.org/html/2606.28772
## Majority Vote Silences Minority Values: Annotator Disagreement at the Hate/Offensive Boundary in HateXplain

###### Abstract

Hate speech annotation pipelines routinely collapse annotator disagreement into majority vote labels before training\. We show that this aggregation is not neutral: 42\.6% of all annotator disagreement in HateXplain concentrates specifically at the hate/offensive boundary, a pattern consistent with annotators applying different thresholds for where hate begins \(χ2=135\.199\\chi^\{2\}=135\.199,df=2df=2,p<0\.0001p<0\.0001\)\. Both a hard\-label BERT model \(Model A\) and a soft\-label model \(Model B\) drop 22 percentage points in accuracy from agreed posts \(∼\\sim80%\) to disagreement posts \(∼\\sim58%\), confirmed atp<0\.0001p<0\.0001\. A per\-annotator multi\-head model \(Model C\) widens this gap further to 28 points while collapsing offensive disagreement accuracy to 0\.245\. Critically, Model A expresses significantly higher confidence on boundary case errors than Model C \(0\.710 vs\. 0\.495,p<0\.0001p<0\.0001\), meaning standard evaluation metrics will not detect the failure\. Three downstream interventions of increasing sophistication all fail to recover boundary accuracy\. We argue the problem is structural\. Majority vote presents a contested judgment as ground truth, and models inherit that false certainty\. The intervention must be upstream in annotation design\.

pluralistic alignment, annotator disagreement, hate speech, soft labels, majority vote, value pluralism, BERT

## 1Introduction

Hate speech detection isn’t a purely empirical or technical category like ”detecting spam” or ”identifying cats in images\.”\. Whether a post is hateful or merely offensive is not an objective fact\. It is a judgment that depends on who is reading, what community they belong to, and what threshold they hold for harm\. Yet standard annotation pipelines treat this judgment as if it were objective: crowdworkers label each post, the majority wins, and the resulting label is presented to a model as ground truth\.Plank \(2022 (https://arxiv.org/html/2606.28772#bib.bib9)\)terms this assumption the “problem” of human label variation: treating genuine variation as noise to be minimised rather than signal to be preserved\.

This paper asks what is lost when that majority wins\. Rather than propose a new model or training objective, we locate precisely where in HateXplain annotator disagreement concentrates, characterise what that concentration means for model behaviour, and show that three downstream interventions of increasing sophistication all fail to recover what majority vote discards\.

Our central finding is that 42\.6% of all annotator disagreement in HateXplain occurs specifically at the hate/offensive boundary \(χ2=135\.199\\chi^\{2\}=135\.199,df=2df=2,p<0\.0001p<0\.0001\)\. The concentration is not random noise but a structural pattern at the boundary where annotators must decide whether content crosses from offensiveness into hatred\. That decision depends on where each annotator places the threshold between offence and hatred, and majority vote resolves it by silencing whichever annotators land on the minority side\.

The consequence for deployed models is concerning\. Both Models A and B drop over 22 percentage points in accuracy from agreed to disagreement posts \(p<0\.0001p<0\.0001\)\. Model C \(per\-annotator heads, the strongest baseline we test\) widens the gap to 28 points and collapses offensive disagreement accuracy to 0\.245\. Model A also expresses significantly higher confidence on boundary case errors than Model C \(0\.710 vs\. 0\.495,p<0\.0001p<0\.0001\)\. A system that is wrong*and certain*on culturally contested inputs is a dangerous failure mode that standard accuracy metrics will not detect\.

#### Contributions\.

\(1\)Annotator disagreement in HateXplain is not uniformly distributed: 42\.6% concentrates at the hate/offensive value boundary \(χ2=135\.199\\chi^\{2\}=135\.199,df=2df=2,p<0\.0001p<0\.0001\), with offensive posts disagreeing at 67\.9% versus 35\.5% for normal posts\.\(2\)A 22\-point accuracy gap between agreed and disagreement posts persists across all three training regimes \(p<0\.0001p<0\.0001\)\.\(3\)Three downstream interventions \(hard labels, soft labels, and per\-annotator heads\) all fail to recover boundary accuracy; the gap widens under the strongest baseline\.\(4\)Hard\-label models are significantly overconfident on boundary case errors relative to per\-annotator models \(0\.710 vs\. 0\.495,p<0\.0001p<0\.0001\), confirming the problem is structural, not architectural\.

## 2Related Work

#### Disagreement in annotation\.

Umaet al\.\(2021 (https://arxiv.org/html/2606.28772#bib.bib1)\)survey learning from disagreement across NLP tasks, showing that soft label approaches generally improve calibration\.Plank \(2022 (https://arxiv.org/html/2606.28772#bib.bib9)\)reframes annotator disagreement as human label variation \(genuine signal rather than noise\) and catalogues approaches that preserve it\.Davaniet al\.\(2022 (https://arxiv.org/html/2606.28772#bib.bib2)\)make the closest argument to ours, showing that majority vote loses signal on subjective tasks and that per\-annotator models recover minority perspectives\.Gordonet al\.\(2021 (https://arxiv.org/html/2606.28772#bib.bib7)\)introduce the disagreement deconvolution, showing that standard ML metrics systematically overstate performance on subjective tasks because they evaluate against aggregated labels rather than individual annotators, a finding that motivates our subgroup analysis\.Gordonet al\.\(2022 (https://arxiv.org/html/2606.28772#bib.bib8)\)extend this with jury learning, an architecture that models every annotator and lets practitioners explicitly weight demographic groups at inference\. Our work differs in three ways: we*measure*where disagreement concentrates rather than proposing a new aggregation mechanism; we show the concentration is category\-specific with precise statistics; and we test the per\-annotator approach directly, finding it widens rather than closes the agreement gap on boundary cases\.

#### Hate speech detection\.

Mathewet al\.\(2021 (https://arxiv.org/html/2606.28772#bib.bib3)\)introduced HateXplain, covering hatespeech, offensive speech, and normal content with rationale annotations\. They note the hate/offensive distinction is the hardest classification boundary but do not quantify what proportion of disagreement it accounts for, nor its consequences for model confidence\.Leonardelliet al\.\(2021 (https://arxiv.org/html/2606.28772#bib.bib4)\)study disagreement in sentiment annotation and argue for preserving annotator perspectives, but do not examine overconfidence as a downstream consequence\.Fleisiget al\.\(2023 (https://arxiv.org/html/2606.28772#bib.bib10)\)argue that when the demographic group targeted by a statement is a minority of the annotator pool, the majority vote may be systematically wrong, and show that modelling target\-group ratings recovers signal that aggregation discards\. Our work connects these threads empirically\.

#### Pluralistic alignment\.

Sorensenet al\.\(2024 (https://arxiv.org/html/2606.28772#bib.bib5)\)define pluralistic alignment as the challenge of building AI systems that represent diverse human values rather than a majority consensus\.Aroyoet al\.\(2023 (https://arxiv.org/html/2606.28772#bib.bib11)\)operationalise this concern for conversational safety with the DICES dataset, which preserves rater demographics and uses a graduated safety scale to enable explicit study of how aggregation choices affect outcomes\. Current RLHF pipelines face an analogous aggregation problem: human preference comparisons are collapsed into reward signals that may encode majority rater values as universal\. We do not test RLHF systems here, but our findings provide concrete grounding for this theoretical concern and suggest testable hypotheses for future work\.

## 3Methods

### 3\.1Dataset

We use HateXplain\(Mathewet al\.,2021 (https://arxiv.org/html/2606.28772#bib.bib3)\), loaded from the original GitHub repository to preserve raw annotator labels\. Each of 20,148 posts is annotated by three crowdworkers with one of three labels:hatespeech,normal, oroffensive\. We use the official train/validation/test splits\. All classification experiments are conducted on the test set \(n=1,892n=1\{,\}892\); disagreement characterisation statistics are computed over the full dataset\.

We define*disagreement*as any post where annotators did not unanimously agree \(unique label count\>1\>1\)\.*Boundary cases*are disagreement posts where raw labels are a subset of \{hatespeech, offensive\}: posts where no annotator chose normal, but annotators disagreed between the two harm categories\. Soft label distributions are computed directly from raw annotator labels before any aggregation\.

### 3\.2Models

We train three BERT\-base\-uncased\(Devlinet al\.,2019 (https://arxiv.org/html/2606.28772#bib.bib6)\)classifiers of increasing sophistication\.

Model A \(Hard\)is trained on majority vote labels using standard cross\-entropy loss, the default pipeline for hate speech classification\.

Model B \(Soft\)is trained on soft labels derived from the annotator distribution \(e\.g\. a 2\-1 hatespeech/offensive split yields\[0\.67,0\.0,0\.33\]\[0\.67,\\,0\.0,\\,0\.33\]\) using KL divergence loss\.

Model C \(Per\-Annotator\)uses a shared BERT encoder with one separate classification head per qualifying annotator \(106 annotators with≥\\geq50 training posts each\)\. During training, each sample is routed to its annotator’s specific head\. At inference, all heads are ensembled by averaging their softmax probabilities\. This follows the approach ofDavaniet al\.\(2022 (https://arxiv.org/html/2606.28772#bib.bib2)\)and constitutes our strongest downstream baseline\.

All models are trained for 3 epochs with AdamW \(lr=2×10−5lr=2\{\\times\}10^\{\-5\}\), linear warmup over 10% of steps, and gradient clipping at 1\.0\.

### 3\.3Statistical Analysis

Group comparisons use the Mann\-Whitney U test \(two\-tailed,α=0\.05\\alpha=0\.05\)\. Chi\-square tests assess whether disagreement rates differ across label categories\. We report exact test statistics, degrees of freedom, andpp\-values for all claims\.

## 4Results

### 4\.1Disagreement Concentrates at the Value Boundary

Of 15,383 training posts, 48\.7% show annotator disagreement, making it a structural feature of the dataset rather than a marginal one\. Among test set disagreement posts \(n=950n=950\), 42\.6% are boundary cases where annotators chose exclusively between hatespeech and offensive labels \(n=405n=405\)\.

A chi\-square test confirms that disagreement rate differs significantly across label categories \(χ2=135\.199\\chi^\{2\}=135\.199,df=2df=2,p<0\.0001p<0\.0001\)\. Offensive posts disagree at 67\.9% \(372/548\), hatespeech posts at 50\.5% \(300/594\), and normal posts at 35\.5% \(278/782\)\. The hate/offensive boundary is where annotator judgments most systematically diverge, not merely the hardest classification problem\. This concentration is consistent with genuine threshold disagreement among annotators: disagreement clusters at the exact point where distinguishing harm categories requires a judgment call about severity \(Figure1 (https://arxiv.org/html/2606.28772#S4.F1)\)\.

Refer to caption

Figure 1:Left: annotator agreement levels in the training set \(48\.7% show disagreement\)\. Centre: label distribution by agreement type\. Offensive posts are over\-represented in disagreement cases\. Right: soft label space for disagreement posts, showing discrete probability mass concentrated at the hate/offensive boundary\.
### 4\.2The Agreement Gap Persists Across All Three Models

Table1 (https://arxiv.org/html/2606.28772#S4.T1)and Figure2 (https://arxiv.org/html/2606.28772#S4.F2)present the core results\. All three models perform substantially better on agreed posts \(∼\\sim80%\) than on disagreement posts \(∼\\sim55–58%\)\. The agreement gap is confirmed atp<0\.0001p<0\.0001for all three models by Mann\-Whitney U test\. Standard overall accuracy \(69–70%\) completely obscures this gap, echoing the gap\-hiding patternGordonet al\.\(2021 (https://arxiv.org/html/2606.28772#bib.bib7)\)document for aggregated metrics on subjective tasks\.

The gap*widens*under the strongest baseline\. Model C \(per\-annotator heads\) achieves the best agreed\-post accuracy \(0\.826\) but the worst disagreement\-post accuracy \(0\.545\), producing the largest agreement gap of 28\.1 points\. Offensive disagreement accuracy collapses to 0\.245 under Model C, less than half of Model A’s already\-poor 0\.454\. No downstream intervention recovers the boundary\.

Table 1:Results across all three models on HateXplain test set \(n=1,892n=1\{,\}892\)\. n\.s\. = not significant atα=0\.05\\alpha=0\.05\. Minority alignment discussed in Section4\.4 (https://arxiv.org/html/2606.28772#S4.SS4)\.Refer to caption

Figure 2:Three downstream interventions all fail to recover boundary accuracy\.Left: accuracy by agreement level\. Agreed post accuracy is stable across models \(∼\\sim80%\) while disagreement post accuracy falls further under Model C \(0\.545 vs\. 0\.579 for Model A\), widening the gap to 28 points\.Centre: accuracy by disagreement category\. Hatespeech disagreement is relatively stable across models, but offensive disagreement collapses to 0\.245 under per\-annotator heads, confirming the offensive boundary is the critical failure point\.Right: confidence on boundary errors \(box plots, mean marked with diamond\)\. Model A is significantly overconfident \(0\.710\) while Model C approaches appropriate uncertainty \(0\.495,p<0\.0001p<0\.0001\), but calibration gains come at the cost of accuracy\.
### 4\.3Soft Labels Do Not Resolve the Gap

Model B does not significantly outperform Model A on any disagreement subgroup\. The boundary case difference \(0\.605 vs\. 0\.595\) yieldsp=0\.284p=0\.284\. Hatespeech disagreement \(0\.730 vs\. 0\.743\) yieldsp=0\.236p=0\.236\. Offensive disagreement marginally favours Model B \(0\.462 vs\. 0\.454\) butp=0\.152p=0\.152\. None survive significance testing\.

### 4\.4Per\-Annotator Heads: More Honest, Not More Accurate

Model C produces the most informative pattern of the three\. On agreed posts it achieves the best accuracy \(0\.826\), confirming the shared encoder learns strong general representations\. But on disagreement posts it performs worst \(0\.545\), and on offensive disagreement it collapses to 0\.245\. The per\-annotator heads have learned individual annotator decision boundaries that do not generalise to the contested cases where annotators genuinely disagree\.

The one meaningful gain from Model C is calibration\. Boundary error confidence drops progressively from Model A \(0\.710\) to Model B \(0\.648\) to Model C \(0\.495\), confirmed atp<0\.0001p<0\.0001\(Figure2 (https://arxiv.org/html/2606.28772#S4.F2), right\)\. Model C is appropriately uncertain on the posts it cannot classify, though that uncertainty comes at the cost of accuracy elsewhere\.

The minority alignment rate tells a similarly nuanced story \(Figure3 (https://arxiv.org/html/2606.28772#S4.F3)\)\. All three models are strongly majority\-aligned: roughly 70% of predictions on disagreement posts match the majority label and only 30% align with any minority annotator\. The agreement gap widens monotonically from Model A \(0\.222\) to Model B \(0\.225\) to Model C \(0\.281\), confirming that no downstream intervention closes the structural gap that majority vote creates\.

#### A note on the inference rule\.

Model C ensembles its per\-annotator heads by averaging softmax probabilities, the inference rule used in the original per\-annotator proposal we follow\(Davaniet al\.,2022 (https://arxiv.org/html/2606.28772#bib.bib2)\)\. In our experiments we keep this rule, but note that alternative aggregation methods \(e\.g\., voting\) may produce different results\. We choose average probabilities because it is the most common ensembling method and allows the model to express uncertainty\. However, even with voting, we observe similar patterns \(not shown\), suggesting the result is robust to the choice of inference rule\.