Learning Sexism Detection Using Multi-Agent Perspectivist Preference Optimization

# Learning Sexism Detection Using Multi-Agent Perspectivist Preference Optimization
Source: [https://arxiv.org/html/2608.04056](https://arxiv.org/html/2608.04056)
Hadi Mohammadi1Tina Shahedi1Robert A\. Bagheri1 Mehdi Dastani2Masoume M\. Raeissi1,3 1Department of Methodology and Statistics, Utrecht University, The Netherlands 2Department of Information and Computing Sciences, Utrecht University, The Netherlands 3Field Technology Innovations \(FTI\) & Robotics, Wageningen University & Research, The Netherlands

###### Abstract

When people label text for sexism, they often disagree, and not because some of them are wrong: they genuinely perceive sexism differently\. Most NLP systems discard this disagreement by collapsing it into a majority vote\. We propose the Multi\-Agent Perspectivist Preference Optimization \(MAP\-PO\) framework to keep these different perspectives\. On the EXIST 2024 dataset of labeled English and Spanish tweets, we first cluster annotators by their labeling behavior rather than their demographic attributes\. We then fine\-tune one Large Language Model agent per cluster to reproduce that cluster’s annotation behavior, and coordinate the agents with preference optimization that combines individual and team\-level rewards\. We evaluate MAP\-PO in four settings defined by two languages and two backbone language models, asking whether each agent reproduces the annotations of its own cluster and whether the agents together reproduce the majority label\. Two findings hold in all four settings\. First, without fine\-tuning the agents behave almost identically, so cluster\-specific training is necessary\. Second, we show that training each agent only on the labels of its own cluster pushes the agents far beyond the clusters they should represent, while adding a shared team\-level training signal consistently keeps each agent calibrated to its cluster\.111Code and data:[mohammadi\-hadi/MAP\-PO](https://github.com/mohammadi-hadi/MAP-PO)

Learning Sexism Detection Using Multi\-Agent Perspectivist Preference Optimization

EXIST 2024 tweets \(EN 3,260 / ES 3,660\)6 human annotators per tweetBehavioral features per annotatorYES rate⋅\\cdotagreement rate⋅\\cdotlabel entropyCluster 1Agent 1Cluster 2Agent 2Cluster 3Agent 3kk\-meansSFT, then DPO or GRPOper\-agent rewardRj=\(1−α\)​rindivj\+α​rteamR\_\{j\}=\(1\-\\alpha\)\\,r^\{\\,j\}\_\{\\text\{indiv\}\}\+\\alpha\\,r\_\{\\text\{team\}\}Majority vote→\\rightarrowsexist: YES / NOagent disagreement==confidence signal

Figure 1:Overview of MAP\-PO\. Annotators are clustered by labeling behavior into three groups per language; one agent per cluster is trained with SFT, DPO, and GRPO under a convex individual/team reward; the agents vote, and their disagreement doubles as a confidence signal\.## 1Introduction

Sexism detection is subjective: six annotators looking at the same tweet may split 4–2 or even 3–3 on whether it can be qualified as a sexist tweet\. The standard approach takes the majority vote and treats the minority opinion as noise\.

Perspectivist work in natural language processing \(NLP\)Basile et al\. \([2021](https://arxiv.org/html/2608.04056#bib.bib1)\); Davani et al\. \([2022](https://arxiv.org/html/2608.04056#bib.bib4)\); Uma et al\. \([2021](https://arxiv.org/html/2608.04056#bib.bib22)\)argues that such disagreement is signal, not noise, and should be modeled rather than averaged away\. We build a multi\-agent system where each agent represents a different annotator perspective \(Figure[1](https://arxiv.org/html/2608.04056#S0.F1)\)\. These perspectives are best defined by*labeling behavior*rather than demographics like age or gender\. How to train such agents is still open: fitting each agent to its own cluster alone pushes it past the annotators it is meant to represent\.

Our approach, MAP\-PO \(Multi\-Agent Perspectivist Preference Optimization\), works in three steps\. First, we discover perspectives by clustering the annotators of the EXIST 2024 dataset into three groups by labeling behavior: YES rate \(how often an annotator labels a text as sexist\), agreement rate \(how often an annotator agrees with the annotator majority\), and label entropy \(how balanced an annotator’s YES/NO decisions are\)\. Second, we train specialized LLM agents: three models, one per cluster, fine\-tuned with supervised fine\-tuning \(SFT\) and then with one of three preference\-optimization objectives, which train an agent to prefer the label its own cluster gives to a tweet over the opposite label: Direct Preference Optimization \(DPO\), Multi\-Agent Reasoning System Preference Optimization \(Mars\-PO\), or Group Relative Policy Optimization \(GRPO\) with a convex individual/team reward\. Third, the agents predict independently and their majority label is the team prediction\. We evaluate four settings, one for each combination of the two languages and the two*backbones*, i\.e\., the pre\-trained language models we fine\-tune \(gpt\-4\.1\-mini, Qwen3\-8B\)\.

Across all four settings, cluster\-specific fine\-tuning is necessary and a shared team signal is what keeps it stable: our best system reaches 90\.3% team accuracy and 89\.9 Team F1 in the GPT\-EN setting, ahead of every baseline, while purely individual preference optimization drives the agents far outside the annotator distributions they represent\.

Our contributions are:

- •We show that behavioral features cluster annotators more effectively than demographics for sexism detection\.
- •Methodologically, we adapt the multi\-agent preference optimization framework Mars\-POLou et al\. \([2024](https://arxiv.org/html/2608.04056#bib.bib9)\)to fine\-tune one LLM agent per annotator perspective for sexism detection, a task where the “ground truth” depends on perspective\.
- •We compare DPO\- and GRPO\-based objectives with individual and team rewards across all four combinations of backbone and language\.

## 2Related Work

#### Perspectivist NLP\.

Basile et al\. \([2021](https://arxiv.org/html/2608.04056#bib.bib1)\)andDavani et al\. \([2022](https://arxiv.org/html/2608.04056#bib.bib4)\)argue that annotator disagreement carries information that should be preserved, andUma et al\. \([2021](https://arxiv.org/html/2608.04056#bib.bib22)\)survey the broader field of learning from disagreement\.Gordon et al\. \([2022](https://arxiv.org/html/2608.04056#bib.bib5)\)propose “jury learning,” which models each individual annotator and composes juries from annotator and demographic embeddings\. In contrast, we first cluster annotators by their annotation behavior and then train one agent per cluster rather than one agent per annotator\.

#### Preference Optimization\.

Direct Preference Optimization \(DPO\)Rafailov et al\. \([2023](https://arxiv.org/html/2608.04056#bib.bib16)\)optimizes language models directly from preference data, i\.e\., pairs of a preferred and a rejected output for the same input, without a separate reward model, and DMPOShi et al\. \([2024](https://arxiv.org/html/2608.04056#bib.bib21)\)extends it to multi\-turn agent tasks\. Multi\-Agent Reasoning System Preference Optimization \(Mars\-PO\)Lou et al\. \([2024](https://arxiv.org/html/2608.04056#bib.bib9)\)adapts DPO for multi\-agent systems using hybrid positive sample sets and agent\-specific negatives, i\.e\., rejected outputs chosen separately for each agent\. We build on Mars\-PO, but on cluster\-disagreement texts: each agent’s preferred output is its own cluster majority and its rejected output is the opposite label, so one text may supply opposite preference directions to different agents\.

#### Multi\-Agent RL for NLP\.

PPOSchulman et al\. \([2017](https://arxiv.org/html/2608.04056#bib.bib18)\)and GRPOShao et al\. \([2024](https://arxiv.org/html/2608.04056#bib.bib20)\)are standard for reinforcement learning from human feedback \(RLHF\)Christiano et al\. \([2017](https://arxiv.org/html/2608.04056#bib.bib3)\); both optimize a single policy against one scalar reward, and where the two have been compared directly, the advantage of GRPO grows with model size\(Mohammadi et al\.,[2025a](https://arxiv.org/html/2608.04056#bib.bib12)\)\. We instead give GRPO a convex reward with two competing terms, which balances how well an agent matches its own behavioral cluster against the accuracy of the agent majority\.

#### EXIST Shared Task\.

The EXIST shared tasksPlaza et al\. \([2023](https://arxiv.org/html/2608.04056#bib.bib15),[2024](https://arxiv.org/html/2608.04056#bib.bib14)\)provide multi\-annotator sexism detection data for English and Spanish tweets, together with demographic information about the annotators\. Earlier work studied automatic sexism classification on Spanish and English Twitter dataRodríguez\-Sánchez et al\. \([2020](https://arxiv.org/html/2608.04056#bib.bib17)\)\. They used transformer systems for these tasksMohammadi et al\. \([2023](https://arxiv.org/html/2608.04056#bib.bib10),[2024](https://arxiv.org/html/2608.04056#bib.bib11)\)to predict a single consensus label, discarding the disagreement that we aim at modeling here\.Lo and Basile \([2023](https://arxiv.org/html/2608.04056#bib.bib8)\)cluster annotators by their labeling behavior to mine perspectives from disagreement\. We extend this clustering approach with multi\-agent preference optimization and measure whether each trained agent reproduces its cluster’s labeling behavior\.

## 3Data and Behavioral Clustering

We use the EXIST 2024 datasetPlaza et al\. \([2024](https://arxiv.org/html/2608.04056#bib.bib14)\), which comprises data in two languages: 3,260 English tweets annotated by 348 annotators, and 3,660 Spanish tweets annotated by 390 annotators\. Only 13 common annotators appear in both subsets, so we cluster each language separately\. Each tweet is labeled by exactly 6 annotators for binary sexism detection \(YES/NO\), and both subsets follow a*perfect block design*: disjoint annotator groups of six \(58 in English, 65 in Spanish\) each label∼57\{\\sim\}57texts, with zero cross\-group overlap\. Throughout, theannotator majorityof a text is the majority label of its six annotators\.

### 3\.1Behavioral Features

Rather than grouping annotators by demographics \(age, gender, country\), we characterize each annotator by three behavioral features, each reported here as its range across the 348 English annotators: aYES rate, the fraction of texts the annotator labels as sexist \(10\.5% to 93\.0%, mean 41\.8%\); anagreement rate, how often the annotator agrees with the annotator majority \(43\.5% to 100%, mean 84\.7%\); and alabel entropy, the Shannon entropyShannon \([1948](https://arxiv.org/html/2608.04056#bib.bib19)\)of the annotator’s YES/NO distribution \(0\.37 to 1\.00, mean 0\.90\)\.

### 3\.2Three Clusters

We standardize features and apply K\-Means clustering \(k=2​…​20k\{=\}2\{\\ldots\}20\), separately per language\. For English, the optimalk=3k\{=\}3is selected by silhouette score \(0\.43\), yielding Cluster 1 \(YES rate 0\.22, 75 annotators\), Cluster 2 \(YES rate 0\.44, 224 annotators\), and Cluster 3 \(YES rate 0\.64, 49 annotators\); Table[4](https://arxiv.org/html/2608.04056#A2.T4)in Appendix[B](https://arxiv.org/html/2608.04056#A2)gives full statistics\. Throughout the paper we refer to clusters by these neutral indices, ordered by YES rate: Cluster 1 always denotes the lowest\-YES\-rate cluster of a language and Cluster 3 the highest\. Moreover, we observe that Cluster 1 annotators, who label YES on a narrow subset of texts, agree closely with the annotator majority; Cluster 3 annotators label YES three times as often and agree with the annotator majority least; and Cluster 2, the largest group, sits between the two and tracks the annotator majority most closely\.

For the Spanish annotators, the same search overk=2​…​20k\{=\}2\{\\ldots\}20prefersk=2k\{=\}2\(silhouette 0\.465 vs\. 0\.349 atk=3k\{=\}3\), but silhouette scores compactness in the full feature space, where agreement rate and label entropy carry most of the variance\. More specifically, thek=2k\{=\}2solution places annotators from 7% to 89% YES in a single cluster, merging annotators whose labeling tendencies the agents are meant to keep apart\. Atk=3k\{=\}3the mean deviation between an annotator’s YES rate and its own cluster mean drops from 0\.099 to 0\.059, every cluster holds at least 17% of annotators, and consecutive cluster mean YES rates are at least 0\.204 apart, whereask=4k\{=\}4leaves silhouette unchanged \(0\.355\) and brings two within 0\.024 of each other \(Appendix[F](https://arxiv.org/html/2608.04056#A6)\)\. We therefore usek=3k\{=\}3on Spanish, which also matches the English architecture: mean YES rates 0\.268, 0\.472, and 0\.676 \(sizes 67/234/89\), mapped to Cluster 1–Cluster 3 by YES rate as on English\.

### 3\.3Demographics Do Not Predict Clusters

Chi\-squared tests on the English subset show no significant association between cluster membership and any single demographic variable \(gender, age, ethnicity, education, country\)\. Gender is balanced overall \(174/174 M/F\) and close to balanced within every cluster\. This indicates that cluster membership is not explained by any single demographic variable, and justifies using behavioral clusters rather than demographic groups as agent identities\. It is also consistent withMohammadi et al\. \([2025b](https://arxiv.org/html/2608.04056#bib.bib13)\), who find that demographic attributes explain only a small share \(roughly 8%\) of the variance in sexism annotations and that demographic personas do not make large language model \(LLM\) annotators more reliable\. Section[8\.2](https://arxiv.org/html/2608.04056#S8.SS2)extends this analysis with feature ablations\.

## 4Method: MAP\-PO

### 4\.1Architecture

We fine\-tune three separate agent models, one per behavioral cluster\. All agents, in every setting, receive the identical prompt:“You are a content annotator\. Classify whether the given tweet contains sexism\. Respond with exactly YES or NO\.”Phase 1 produces the checkpoint every variant starts from; Phases 2a and 2b are alternative second stages, not subsequent phases\. We will explain each phase further in this section\. Each cluster’s perspective is encoded entirely in the fine\-tuned model weights, not in the prompt\. At inference time the three agents predict independently and their majority label, theagent majority, is the team prediction: YES when at least two agents say YES\.

### 4\.2Backbones

We instantiate the pipeline on two backbones\. \(1\) OpenAI fine\-tuning API: our API base isgpt\-4\.1\-mini\-2025\-04\-14, which we fine\-tune with SFT and native DPO; GRPO is implemented as rejection sampling \(Section[4\.6](https://arxiv.org/html/2608.04056#S4.SS6)\)\. \(2\) Local open weights: Qwen3\-8BYang et al\. \([2025](https://arxiv.org/html/2608.04056#bib.bib23)\)fine\-tuned with LoRA adaptersHu et al\. \([2022](https://arxiv.org/html/2608.04056#bib.bib7)\)on the MLX frameworkHannun et al\. \([2023](https://arxiv.org/html/2608.04056#bib.bib6)\)\.

Both backbones share identical prompts and data preparation\. For the local backbone, predictions and the DPO loss are computed exactly at the single\-token level \(comparing the probabilities ofYESandNOas the first generated token\), with the frozen SFT policy as reference; hyperparameters are in Appendix[G](https://arxiv.org/html/2608.04056#A7)\.

### 4\.3Data Preparation

For each text, we compute thecluster majority\(the majority label of the annotators of each cluster present in that text’s group\), the annotator majority, and the cluster composition of that text, i\.e\., which of the three clusters the six annotators of that text belong to\.

We split by annotator groups to prevent annotation leakage: 46 train / 6 val / 6 test groups on English \(2,576 / 342 / 342 texts\) and 53 / 6 / 6 on Spanish \(3,021 / 342 / 297 texts\)\. Table[5](https://arxiv.org/html/2608.04056#A2.T5)in Appendix[B](https://arxiv.org/html/2608.04056#A2)summarizes the English datasets\.

For cross\-cluster coverage on English, all three clusters are present in 29\.7% of texts, two in 56\.3%, and one in 14\.0%; the 2,804 texts with≥\\geq2 clusters enable cross\-cluster preference pair construction\.222Annotators are clustered independently of the texts they label, so the six annotators of a single text can belong to different behavioral clusters; a text therefore covers one, two or three clusters\.Appendix[B](https://arxiv.org/html/2608.04056#A2)details how the block design shapes this coverage\.

### 4\.4Phase 1: SFT with Mixed Data

Each agent is fine\-tuned on its cluster majority labels using standard cross\-entropy loss, a variant we call SFT\-pure\. We mix inshared positive examples\(30%\), texts where all present clusters agree, which prevent agents from drifting apart \(inspired by Mars\-POLou et al\. \([2024](https://arxiv.org/html/2608.04056#bib.bib9)\)\), andteam\-alignment examples\(20%\), texts labeled with the annotator majority, providing a light team\-alignment signal\. This mixture, SFT\-mixed, is the SFT stage used everywhere unless stated otherwise\.

After SFT, each agent should approximate its own cluster’s YES rate\.

### 4\.5Phase 2a: DPO Preference Fine\-Tuning

For texts where the cluster majorities disagree, we construct preference pairs: for each agent, the preferred output is that text’s cluster majority for the agent’s cluster, and the non\-preferred output is the opposite label\. For agentjjwith cluster voteyjy\_\{j\}and oppositey¯j\\bar\{y\}\_\{j\}, the DPO loss is:

ℒDPOj=−𝔼​\[log⁡σ​\(β​hj​\(yj\)−β​hj​\(y¯j\)\)\],\\mathcal\{L\}\_\{\\text\{DPO\}\}^\{j\}=\-\\mathbb\{E\}\\left\[\\log\\sigma\\big\(\\beta\\,h\_\{j\}\(y\_\{j\}\)\-\\beta\\,h\_\{j\}\(\\bar\{y\}\_\{j\}\)\\big\)\\right\],\(1\)wherehj​\(y\)=log⁡πj​\(y\|x\)πref​\(y\|x\)h\_\{j\}\(y\)=\\log\\frac\{\\pi\_\{j\}\(y\|x\)\}\{\\pi\_\{\\text\{ref\}\}\(y\|x\)\},πj\\pi\_\{j\}is the current policy \(agentjj\),πref\\pi\_\{\\text\{ref\}\}is the frozen SFT checkpoint, andβ\\betasets the strength of the implicit Kullback–Leibler \(KL\) penalty that holds each agent near its reference policy; we useβ=0\.1\\beta=0\.1\.

This sharpens each agent’s cluster\-specific decision boundary\.

### 4\.6Phase 2b: GRPO with Team Reward

In place of preference pairs, we approximate GRPO through rejection sampling\. For each text, we sampleK=8K\{=\}8predictions from each agent and compute a convex combination of an individual and a team reward:

Rj=\(1−α\)​rindivj\+α⋅rteam,α∈\[0,1\],R\_\{j\}=\(1\-\\alpha\)\\,r\_\{\\text\{indiv\}\}^\{j\}\+\\alpha\\cdot r\_\{\\text\{team\}\},\\qquad\\alpha\\in\[0,1\],\(2\)
whererindivj=1r\_\{\\text\{indiv\}\}^\{j\}=1if agentjj’s prediction matches its cluster majority \(else 0\),rteam=1r\_\{\\text\{team\}\}=1if the agent majority matches the annotator majority \(else 0\), andα\\alphais the team weight:α=0\\alpha\{=\}0is pure individual reward,α=1\\alpha\{=\}1pure team reward\.

We keep the samples whose reward reaches the midpoint of the achievable range \(Rj≥12R\_\{j\}\\geq\\tfrac\{1\}\{2\}\) and re\-fine\-tune each agent on them; atα=0\.5\\alpha\{=\}0\.5\(equal weight\) this rule keeps a sample if either reward fires\. One iteration is a full pass of sampling, filtering, and re\-fine\-tuning; every GRPO system we report uses a single iteration\.

## 5Experimental Setup

The design goal of MAP\-PO sets two axes: does each agent reproduce the labels of its own cluster on held\-out texts, and does the three\-agent team still perform well when compared with the annotator majority? The settings, metrics, and baselines below are chosen to answer both\.

### 5\.1Settings

We evaluate MAP\-PO in four settings, crossing the two backbones of Section[4\.2](https://arxiv.org/html/2608.04056#S4.SS2)with the two languages:GPT\-ENandGPT\-ESfine\-tunegpt\-4\.1\-minithrough the OpenAI API on the English and Spanish subsets, andQwen\-ENandQwen\-ESfine\-tune Qwen3\-8B locally with LoRA\. The Spanish settings use clusters re\-derived on the 390 Spanish annotators \(Section[3](https://arxiv.org/html/2608.04056#S3)\)\.

### 5\.2Evaluation Metrics

We report every headline metric as F1\-macro \(the mean of F1\-YES and F1\-NO\), on a 0–100 scale for readability\. The English test set is class\-imbalanced \(60\.7% NO\), so a trivial Always\-NO classifier scores 60\.7% accuracy but only 37\.8 F1\-macro; F1 exposes that it carries no signal\.

For these two axes we reportCluster F1\(per agentjj: F1\-macro against the cluster majority ofcjc\_\{j\}, over texts where its annotators are not tied; averaged as Avg Cluster F1\),Team F1\(F1\-macro of the agent majority against the annotator majority\), andOverall F1\(the harmonic mean of Avg Cluster and Team F1, which no method can inflate by trading one axis for the other\)\.

Table[3](https://arxiv.org/html/2608.04056#A1.T3)in Appendix[A](https://arxiv.org/html/2608.04056#A1)gives plain\-English definitions with worked examples; Figures[7](https://arxiv.org/html/2608.04056#A3.F7)and[8](https://arxiv.org/html/2608.04056#A3.F8)in Appendix[C](https://arxiv.org/html/2608.04056#A3)visualize both axes directly\.

Because the test sets are label\-imbalanced in opposite directions \(English 39% YES, Spanish 58%\), we report balanced accuracyBrodersen et al\. \([2010](https://arxiv.org/html/2608.04056#bib.bib2)\)and per\-class F1 \(Appendix[H](https://arxiv.org/html/2608.04056#A8)\)\. As a diversity diagnostic we track theagent agreement rate, the fraction of texts where all three coincide\. Its extremes signal failure modes: near 100% the agents have collapsed onto one function \(GPT\-EN zero\-shot 97%\), and near 0% they have polarized \(GPT\-EN DPO\-only 6%\)\.

#### Cluster coverage\.

Cluster F1 uses 228, 318, and 97 evaluation texts for Cluster 1, Cluster 2, and Cluster 3 on the 342\-text English test set, and for Cluster 1 and Cluster 3 the “cluster majority” often rests on one or two annotators \(Appendix[B](https://arxiv.org/html/2608.04056#A2)\)\. Every Cluster F1 bar therefore carries a 95% Wilson interval, and Appendix[D](https://arxiv.org/html/2608.04056#A4)evaluates on the subset where all clusters are present\.

### 5\.3Baselines

We compare against three baselines:Always\-NO\(predicts NO for every text; majority\-class baseline\), azero\-shot ensemble\(base model without fine\-tuning, same prompt for all three agents\), andpersona prompts\(one base model per cluster with a cluster\-describing system prompt, no fine\-tuning\)\. Every baseline uses the same base model as the fine\-tuned agents of its setting\. The zero\-shot ensemble is run in all four settings, since the cross\-setting comparison is about what fine\-tuning adds, while the persona prompts and Always\-NO are run only in GPT\-EN, where we conduct the full method comparison: they answer questions asked once, whether a prompt alone can encode a perspective and whether the headline metric can be satisfied by the majority class\.

## 6Results in the GPT\-EN Setting

All results in this section, baselines included, usegpt\-4\.1\-minias the base model \(the GPT\-EN setting\), so every comparison isolates the effect of training rather than of the backbone\. Table[1](https://arxiv.org/html/2608.04056#S6.T1)carries the method comparison; the tables behind the individual comparisons, SFT versus DPO and theβ\\betaandα\\alphasweeps, are in Appendix[C](https://arxiv.org/html/2608.04056#A3)\.

### 6\.1Comparison with Baselines

Table[1](https://arxiv.org/html/2608.04056#S6.T1)compares every method on every F1 metric: MAP\-PO with GRPO \(α=0\.17\\alpha\{=\}0\.17\) achieves the highest Team F1 \(89\.9\), and Mars\-PO the highest Overall F1 \(84\.8, with GRPO at 84\.7 essentially tied\)\. SFT\-mixed \(84\.2\) and SFT\-pure \(83\.5\) trail by at most 1\.3 pp\. DPO\-only, despite 86\.6% Team Accuracy, drops to 71\.2 Overall F1: F1\-macro exposes its polarized Cluster 1 \(F1 49\.6\) and Cluster 3 \(F1 48\.7\) agents that accuracy had hidden\.

Table 1:Comparison in the GPT\-EN setting \(all rows share thegpt\-4\.1\-minibase\)\. Best value per column in bold\.Three findings emerge\. Fine\-tuning is essential: the three zero\-shot agents agree on 96\.8% of texts, causing the ensemble to behave like a single conservative model \(14–15% YES rates\) and reaches only 64\.7 Team F1\. Persona prompts recover some diversity \(79\.2% agreement\) but reach only 68\.3 Team F1; describing a perspective in the prompt is not enough to learn it\. And MAP\-PO delivers both axes: GRPO \(α=0\.17\\alpha\{=\}0\.17\) reaches Avg Cluster F1 80\.0 and Team F1 89\.9 simultaneously\.

### 6\.2Individual\-Only DPO Polarizes the Agents

SFT\-mixed is our best non\-preference\-tuned system: 89\.6% Team Accuracy with near\-perfect calibration to cluster targets \(average error 0\.6%; Cluster 1 21\.4% vs\. 21\.5%, Cluster 2 43\.3% vs\. 43\.8%, Cluster 3 64\.6% vs\. 63\.5%\)\. Table[8](https://arxiv.org/html/2608.04056#A3.T8)and Figure[6](https://arxiv.org/html/2608.04056#A3.F6)in Appendix[C](https://arxiv.org/html/2608.04056#A3)give the full metric\-by\-metric view\.

#### DPO overshoots to extremes\.

Adding DPO on top of SFT drives the agents to opposite extremes: Cluster 1 drops to 1\.5% YES \(target 21\.5%\), Cluster 3 jumps to 95\.3% \(target 63\.5%\), disagreement rises to 93\.9%, and calibration error to 18\.7%\. The mechanism is simple: preference pairs encode “cluster A prefers YES and cluster B prefers NO” on disagreement texts, and DPO amplifies these preferences beyond the clusters’ actual labeling behavior\. DPO maximizes diversity but reduces both accuracy \(Team Accuracy drops 3\.0 points\) and calibration \(Table[8](https://arxiv.org/html/2608.04056#A3.T8)\)\.

### 6\.3A Team Signal Restores Calibration

The overshoot appears whenever agents train only on their own cluster’s preferences\. We find three ways to prevent it, and all three share one ingredient: a shared team anchor\.

#### Implicit team signal via data mixing\.

Comparing SFT\-mixed with SFT\-pure shows that the shared\-positive and team\-alignment mixing \(Section[4](https://arxiv.org/html/2608.04056#S4)\) already acts as an implicit team reward \(Table[1](https://arxiv.org/html/2608.04056#S6.T1)\)\. Without the mixture the agents overshoot: Cluster 1 drops from 21\.4% YES to 12\.9%, Cluster 3 jumps from 64\.6% to 71\.6%, disagreement rises 14 points, calibration error grows from 0\.6% to 6\.3%, and Team Accuracy slips to 88\.9%\.

#### Explicit team signal via preference pairs \(Mars\-PO\)\.

We revisit DPO with the full Mars\-PO recipe: individual pairs on cluster\-disagreement texts, plus the shared pairs on unanimous\-agreement texts that we had omitted before\. These shared pairs act as the team anchor\. The team pairs fix the overshoot completely \(Table[1](https://arxiv.org/html/2608.04056#S6.T1)\): Mars\-PO reaches 89\.3% Team Accuracy with 34\.5% disagreement and 4\.4% calibration error \(DPO\-only: 86\.6%, 93\.9%, 18\.7%\), and the pinned agents return to their clusters \(Cluster 1 from 1\.5% to 22\.8% YES; Cluster 3 from 95\.3% to 55\.0%\)\. This is the central finding on reward design: some form of shared anchor, explicit or implicit, must be present to keep individual preferences from polarizing the agents\.

#### A stronger KL constraint cannot prevent overshooting\.

Raisingβ\\betadoes not substitute for a team signal: sweepingβ\\betafrom 0\.1 to 0\.5 cuts disagreement by 21\.4 pp, but even the best value \(72\.5%\) means the agents still disagree on almost three out of four texts, whereas Mars\-PO reaches 34\.5% \(Appendix[C](https://arxiv.org/html/2608.04056#A3), Table[9](https://arxiv.org/html/2608.04056#A3.T9)\)\. The overshoot is a property of the individual\-only objective, not of the hyperparameter\.

### 6\.4GRPO: Best Accuracy with a Small Team Weight

GRPO atα=0\.17\\alpha\{=\}0\.17achieves our highest team accuracy, 90\.3% \(Table[1](https://arxiv.org/html/2608.04056#S6.T1)\), with a 95% bootstrap confidence interval \(CI\) over test texts of\[86\.6,93\.3\]\[86\.6,93\.3\]; every fine\-tuned variant separates from the strongest baseline, persona prompts at\[69\.1,79\.5\]\[69\.1,79\.5\], by CI non\-overlap\. Comparing the three team mechanisms: SFT mixing \(implicit, via data\) gives the best calibration \(0\.6% error\) at high accuracy \(89\.6%\); Mars\-PO \(explicit, via preference pairs\) gives the lowest disagreement among preference methods \(34\.5%\); and GRPO \(explicit, via reward\) gives the highest accuracy with balanced disagreement \(43\.6%\)\.

#### α\\alpha\-sweep summary\.

To calibrate how much team signal GRPO actually needs, we sweep the team weightα∈\{0,0\.17,0\.33,0\.5\}\\alpha\\in\\\{0,0\.17,0\.33,0\.5\\\}\(the convex equivalents of the original GPT\-EN weights0,0\.20\.2,0\.50\.5, and1\.01\.0\); full results are in Appendix[C](https://arxiv.org/html/2608.04056#A3)\(Table[10](https://arxiv.org/html/2608.04056#A3.T10)\)\. A small but non\-zero team weight is optimal:α=0\.17\\alpha\{=\}0\.17exceedsα=0\\alpha\{=\}0by 0\.4 pp on Team Accuracy, and the whole sweep spans only 1\.7 pp, so accuracy is robust to the weighting\. The disagreement curve is not monotone: 42\.7% atα=0\\alpha\{=\}0, 43\.6% atα=0\.17\\alpha\{=\}0\.17, 36\.3% atα=0\.33\\alpha\{=\}0\.33, then 50\.6% at equal weight, where the midpoint keep rule retains a sample whenever either reward fires; more team weight does not simply collapse diversity\.

#### The individual–team tradeoff\.

Individual rewards separate the agents, team rewards align the vote, and the sweep shows this tension is mild for GRPO\. The DPO overshoot is far larger \(2\.7 pp Team Accuracy and 59 pp disagreement between DPO\-only and Mars\-PO\) and is not dissolved by tuningβ\\beta\. The real design axis is presence, not strength: every variant with a shared anchor stays calibrated, whether that anchor is implicit in the SFT mix, explicit in Mars\-PO’s team pairs, or explicit in GRPO’s team reward, while individual\-only DPO, whose pairs carry no anchor, polarizes at everyβ\\betawe tried\.

## 7Generalization Across Languages and Backbones

This section asks whether the approach, and in particular the team\-signal principle, remains effective across changes in the backbone model and the target language\. We repeat the pipeline in the three remaining settings of Section[5\.1](https://arxiv.org/html/2608.04056#S5.SS1), completing a2×22\\times 2design\. Prompts, data preparation, splits, and evaluation are identical to GPT\-EN; local training hyperparameters are in Appendix[G](https://arxiv.org/html/2608.04056#A7)\. All three settings run GRPO atα=0\.2\\alpha\{=\}0\.2\(Eq\.[2](https://arxiv.org/html/2608.04056#S4.E2); cf\. Section[6\.4](https://arxiv.org/html/2608.04056#S6.SS4)\)\.

### 7\.1Main Comparison

Table 2:Cross\-lingual comparison across backbones and languages\. Avg C\-F1 is the average per\-cluster F1\-macro; best value per column in bold\.Table[2](https://arxiv.org/html/2608.04056#S7.T2)reports the full four\-setting comparison; Figure[10](https://arxiv.org/html/2608.04056#A3.F10)in Appendix[C](https://arxiv.org/html/2608.04056#A3)shows Team F1 side by side\. Two regularities hold everywhere\. First, the zero\-shot collapse is even more pronounced than on GPT\-EN: before fine\-tuning the three agents are the same Qwen3\-8B under the same prompt, and they predict identically on every test text in both languages \(0\.0% disagreement\)\. Second, cluster\-specific fine\-tuning always improves Team F1: SFT raises it from 64\.7 to 89\.1 on GPT\-EN, from 57\.4 to 84\.6 on GPT\-ES, from 76\.8 to 84\.7 on Qwen\-EN, and from 72\.9 to 81\.0 on Qwen\-ES\.

The exact method ranking, however, depends on the backbone\. On GPT\-EN, GRPO leads SFT by 0\.8 pp \(89\.9 vs\. 89\.1\)\. On GPT\-ES the three stages sit within 0\.9 pp of each other\. On the smaller Qwen backbone, SFT stays ahead of GRPO \(84\.7 vs\. 80\.9 on English, 81\.0 vs\. 80\.6 on Spanish\)\. We read this as a capacity effect: rejection\-sampling GRPO retrains on the model’s own filtered outputs, and the weaker the policy, the noisier that signal is relative to plain supervised labels\. The team reward is therefore not a guaranteed headline gain; in all four settings it protects against the polarization described next\.

### 7\.2Does the Team\-Signal Principle Replicate?

Both halves of the central GPT\-EN finding replicate in all three new settings: individual\-only preference optimization polarizes agents, and a team anchor repairs them\.

Polarization is universal\. After individual\-only DPO, the Cluster 1 and Cluster 3 agents pin their YES rates to the extremes: 0\.000 and 0\.997 on GPT\-ES, 0\.000 and 1\.000 on both Qwen settings, matching the 0\.015 / 0\.953 pattern on GPT\-EN\. Across the four settings, disagreement saturates \(94–100%\), calibration error rises to 18\.7–25\.1 pp, and Avg Cluster F1 collapses everywhere \(Table[2](https://arxiv.org/html/2608.04056#S7.T2): 53–61 under DPO against 75–80 under SFT\)\. How much this costs in Team F1 varies: Qwen\-EN drops to 78\.1, yet GPT\-ES DPO posts the best Team F1 of its column \(85\.5\), because its Cluster 2 agent still tracks the annotator majority and the two pinned agents cancel in the vote\. The per\-cluster representation is lost regardless; a good team score can hide it, which is exactly why we report both axes\.

The repair also replicates\. GRPO with the team\-weighted reward pulls every agent back toward its cluster: mean calibration error is 2\.0 pp on Qwen\-EN, 8\.0 on Qwen\-ES, 5\.0 on GPT\-ES \(Table[14](https://arxiv.org/html/2608.04056#A6.T14)\), and disagreement returns to the 33–46% range\. Recovery is not always complete \(on Qwen\-ES the Cluster 2 agent overshoots to 0\.589 against a 0\.472 target\), but in every setting the GRPO agents are calibrated cluster members rather than constant functions\.

#### Ablation: label\-balanced SFT\.

Label\-balanced SFT tests whether balancing the YES/NO labels in the training data improves on handling imbalance at the metric level\. On Qwen\-EN, Team F1 drops from 84\.7 to 83\.4 and mean calibration error jumps from 1\.7 to 10\.9 pp \(Table[14](https://arxiv.org/html/2608.04056#A6.T14)\)\. Balancing the labels erases the very YES\-rate differences the clusters are defined by; imbalance is better handled at the metric level than in the training distribution\.

### 7\.3English vs\. Spanish Differences

The Spanish subset differs from the English one in two measurable ways\. First, class balance flips: the Spanish test set has a YES majority \(58\.2%, vs\. 39\.3% on English\), so Team F1\-YES exceeds F1\-NO on Spanish and balanced accuracy tracks F1\-macro closely \(Appendix[H](https://arxiv.org/html/2608.04056#A8)\)\. Second, calibration is somewhat harder on Spanish: after SFT the mean YES\-rate error on Qwen\-ES is 5\.4 pp against 1\.7 on Qwen\-EN, and after GRPO 8\.0 against 2\.0, driven mainly by the Cluster 2 agent\. Spanish trails English by 3\.7 to 7\.3 Team F1 at zero\-shot and after SFT in both backbones \(Table[2](https://arxiv.org/html/2608.04056#S7.T2)\), but the gap is not systematic: after DPO it falls to 0\.3 on GPT and reverses on Qwen, where Spanish leads by 2\.4\. The two subsets share only 13 annotators and differ in texts and class balance as well as in language, so these are differences between settings rather than an isolated effect of language\.

## 8Analysis

### 8\.1Do Agents Behave Like Their Clusters?

Yes, at every stage except individual\-only DPO\. Cluster F1 measures per\-text label agreement, not whether an agent is behaviorally interchangeable with an annotator from its cluster\. We therefore locate each agent’s test\-set behavioral features within its own cluster’s annotator distribution as z\-scores \(Appendix[E](https://arxiv.org/html/2608.04056#A5)\)\.

After SFT, the agents are typical cluster members: every Qwen\-EN agent’s YES rate lies within 0\.2 standard deviations of its cluster’s annotator mean \(0\.1 for GPT\-EN\)\. GRPO keeps them there \(\|z\|≤0\.54\|z\|\\leq 0\.54\), with a looser fit on Spanish in both backbones that matches the calibration overshoots \(Figure[12](https://arxiv.org/html/2608.04056#A5.F12)and Table[12](https://arxiv.org/html/2608.04056#A5.T12)\)\. DPO\-only agents fall outside their cluster distributions, with YES\-rate z\-scores from−4\.3\-4\.3to\+5\.1\+5\.1over the four settings: polarized beyond any annotator\.

### 8\.2Why Behavioral Clusters, Not Demographics?

Demographics have negligible predictive power over labeling behavior: chi\-squared tests are non\-significant for every demographic variable, and Cramér’s V is at most 0\.13 \(ethnicity\), with gender and age below 0\.07 \(Section[3\.3](https://arxiv.org/html/2608.04056#S3.SS3)\)\. An agent system based on these attributes would therefore group annotators who actually disagree, and separate annotators who actually agree, consistent with prior evidence that demographic personas fail to reproduce group\-specific labeling behavior\(Mohammadi et al\.,[2025b](https://arxiv.org/html/2608.04056#bib.bib13)\)\.

Dropping the YES rate collapses the direction of an annotator’s leaning, and content\-derived features would either recover the block partition or make Cluster F1 partly circular \(Appendix[F](https://arxiv.org/html/2608.04056#A6)\)\.

## 9Discussion and Conclusion

We presented MAP\-PO, a multi\-agent system for sexism detection that preserves diverse annotator perspectives through behavioral clustering and preference optimization\. Labeling behavior, not demographics, defines more informative agent identities, and balancing individual faithfulness against team accuracy keeps the agents from overshooting\.

Across a2×22\\times 2grid of backbones and languages, our best GPT\-EN system, GRPO atα=0\.17\\alpha\{=\}0\.17, reaches 90\.3% team accuracy and 89\.9 Team F1, ahead of every baseline; on the smaller Qwen3\-8B backbone, SFT keeps the best headline scores \(Section[7](https://arxiv.org/html/2608.04056#S7)\)\. What holds in all four settings is the polarization result: pure individual preference optimization drives agents far outside the annotator distributions they are meant to represent, and a team anchor, implicit in data mixing \(SFT\), explicit in preference pairs \(Mars\-PO\) or in reward \(GRPO\), restores calibration every time\.

Agent disagreement is itself informative: it marks the texts where humans also disagree, so reporting “Cluster 1 says NO, Cluster 3 says YES” beats a single label with false confidence\.

## Limitations

Our approach has several limitations\. First, the EXIST 2024 block design means we never observe all three clusters on the same text for 70% of the English data, and the all\-clusters evaluation subset is correspondingly small \(47 non\-tied texts on the English test set, 111 on the Spanish one, Appendix[D](https://arxiv.org/html/2608.04056#A4)\), so its CIs are wide\.

Second, for the Cluster 1 and Cluster 3 clusters, many texts have only one annotator from that cluster, making the “cluster majority” just one person’s opinion\.

Third, we test a single dataset and a single task: binary sexism detection on EXIST 2024, in two languages that come from the same shared task\. Whether behavioral clustering transfers to other subjective tasks, or to corpora without a block design, remains an open question\.

Fourth, each annotator is summarized by only three behavioral features, and these are not independent: label entropy is a deterministic function of the YES rate \(Appendix[F](https://arxiv.org/html/2608.04056#A6)\), so the space is effectively two\-dimensional, capturing how often an annotator labels YES and how closely they track the annotator majority\. Richer descriptors, such as per\-topic or temporal labeling patterns, could separate perspectives that these three features merge\.

## Ethics Statement

#### Data\.

We use the EXIST 2024 dataset, obtained from the shared\-task organizers under their research usage agreement\. The tweets were posted publicly and were labeled by crowd annotators recruited by the organizers\. We do not redistribute the data and we quote no tweets in this paper\.

#### Offensive content\.

The data contains sexist and otherwise offensive language\. Our models only classify such text; they do not generate it\.

#### Annotator privacy\.

The dataset comes with self\-reported demographic information about the annotators \(gender, age, ethnicity, education, country\)\. We use it only in aggregate, to test whether demographics explain labeling behavior, and we report that they do not\. Our clusters are built from labeling statistics alone, we make no attempt to identify individual annotators, and a cluster describes how a person labels, not who they are\.

#### Intended use\.

We see MAP\-PO as an assistive tool: when the agents disagree, the text is genuinely contested and should go to a human, so the system is not meant to make moderation decisions on its own\. The agents also inherit the composition of the EXIST annotator pool \(two languages, and about 75% of annotators identifying as White/Caucasian\), so a different annotator population may yield different clusters and any deployment needs its own validation\.

#### Reproducibility\.

The full training and evaluation code, together with per\-run result files for every experiment in this paper, is available at[https://github\.com/mohammadi\-hadi/MAP\-PO](https://github.com/mohammadi-hadi/MAP-PO)\.

## Acknowledgements

We gratefully acknowledge support from the focus area Applied Data Science \(ADS\) funding from Utrecht University\.

## References

- Basile et al\. \(2021\)Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, and Alexandra Uma\. 2021\.We need to consider disagreement in evaluation\.In*Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future*, pages 15–21\. Association for Computational Linguistics\.
- Brodersen et al\. \(2010\)Kay Henning Brodersen, Cheng Soon Ong, Klaas Enno Stephan, and Joachim M Buhmann\. 2010\.The balanced accuracy and its posterior distribution\.In*20th International Conference on Pattern Recognition \(ICPR\)*, pages 3121–3124\.
- Christiano et al\. \(2017\)Paul F Christiano, Jan Leike, Tom B Brown, Miljan Martic, Shane Legg, and Dario Amodei\. 2017\.Deep reinforcement learning from human preferences\.In*Advances in Neural Information Processing Systems*, volume 30\.
- Davani et al\. \(2022\)Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran\. 2022\.Dealing with disagreements: Looking beyond the majority vote in subjective annotations\.*Transactions of the Association for Computational Linguistics*, 10:92–110\.
- Gordon et al\. \(2022\)Mitchell L Gordon, Michelle S Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S Bernstein\. 2022\.Jury learning: Integrating dissenting voices into machine learning models\.In*Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems \(CHI\)*\.
- Hannun et al\. \(2023\)Awni Hannun, Jagrit Digani, Angelos Katharopoulos, and Ronan Collobert\. 2023\.MLX: Efficient and flexible machine learning on apple silicon\.[https://github\.com/ml\-explore/mlx](https://github.com/ml-explore/mlx)\.
- Hu et al\. \(2022\)Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen\-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen\. 2022\.LoRA: Low\-rank adaptation of large language models\.In*International Conference on Learning Representations \(ICLR\)*\.
- Lo and Basile \(2023\)Soda Marem Lo and Valerio Basile\. 2023\.Hierarchical clustering of label\-based annotator representations for mining perspectives\.In*Proceedings of the 2nd Workshop on Perspectivist Approaches to NLP \(NLPerspectives\)*\.CEUR Workshop Proceedings, Vol\. 3494\.
- Lou et al\. \(2024\)Xiaoxuan Lou, Chaojie Wang, and Bo An\. 2024\.Mars\-PO: Multi\-agent reasoning system preference optimization\.*arXiv preprint arXiv:2411\.19039*\.
- Mohammadi et al\. \(2023\)Hadi Mohammadi, Anastasia Giachanou, and Ayoub Bagheri\. 2023\.Towards robust online sexism detection: A multi\-model approach with BERT, XLM\-RoBERTa, and DistilBERT for EXIST 2023 tasks\.In*Working Notes of the Conference and Labs of the Evaluation Forum \(CLEF 2023\)*, volume 3497 of*CEUR Workshop Proceedings*, pages 1000–1011\.
- Mohammadi et al\. \(2024\)Hadi Mohammadi, Anastasia Giachanou, and Ayoub Bagheri\. 2024\.A transparent pipeline for identifying sexism in social media: Combining explainability with model prediction\.*Applied Sciences*, 14\(19\):8620\.
- Mohammadi et al\. \(2025a\)Hadi Mohammadi, Tamas Kozak, and Anastasia Giachanou\. 2025a\.Evaluating GRPO and DPO for faithful chain\-of\-thought reasoning in LLMs\.*arXiv preprint arXiv:2512\.22631*\.
- Mohammadi et al\. \(2025b\)Hadi Mohammadi, Tina Shahedi, Pablo Mosteiro, Massimo Poesio, Ayoub Bagheri, and Anastasia Giachanou\. 2025b\.Assessing the reliability of LLMs annotations in the context of demographic bias and model explanation\.In*Proceedings of the 6th Workshop on Gender Bias in Natural Language Processing \(GeBNLP\)*, pages 92–104\. Association for Computational Linguistics\.
- Plaza et al\. \(2024\)Laura Plaza, Jorge Carrillo\-de Albornoz, Roser Morante, Enrique Amigó, Julio Gonzalo, and Damiano Spina\. 2024\.EXIST 2024: sexism identification in social networks\.In*Experimental IR Meets Multilinguality, Multimodality, and Interaction: Proceedings of CLEF 2024*\.
- Plaza et al\. \(2023\)Laura Plaza, Jorge Carrillo\-de Albornoz, Roser Morante, Enrique Amigó, Julio Gonzalo, Damiano Spina, and Paolo Rosso\. 2023\.Overview of EXIST 2023: sexism identification in social networks\.In*Experimental IR Meets Multilinguality, Multimodality, and Interaction: Proceedings of CLEF 2023*\.
- Rafailov et al\. \(2023\)Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn\. 2023\.Direct preference optimization: Your language model is secretly a reward model\.In*Advances in Neural Information Processing Systems*, volume 36\.
- Rodríguez\-Sánchez et al\. \(2020\)Francisco Rodríguez\-Sánchez, Jorge Carrillo\-de Albornoz, and Laura Plaza\. 2020\.Automatic classification of sexism in social networks: An empirical study on Twitter data\.*IEEE Access*, 8:219563–219576\.
- Schulman et al\. \(2017\)John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov\. 2017\.Proximal policy optimization algorithms\.*arXiv preprint arXiv:1707\.06347*\.
- Shannon \(1948\)Claude E\. Shannon\. 1948\.A mathematical theory of communication\.*The Bell System Technical Journal*, 27\(3\):379–423\.
- Shao et al\. \(2024\)Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yu Wu, and Daya Guo\. 2024\.DeepSeekMath: Pushing the limits of mathematical reasoning in open language models\.*arXiv preprint arXiv:2402\.03300*\.
- Shi et al\. \(2024\)Wentao Shi, Mengqi Yuan, Junkang Wu, Qifan Wang, and Fuli Feng\. 2024\.Direct multi\-turn preference optimization for language agents\.*arXiv preprint arXiv:2406\.14868*\.
- Uma et al\. \(2021\)Alexandra N Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio\. 2021\.Learning from disagreement: A survey\.*Journal of Artificial Intelligence Research*, 72:1385–1470\.
- Yang et al\. \(2025\)An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others\. 2025\.Qwen3 technical report\.*arXiv preprint arXiv:2505\.09388*\.

## Appendix ADiagnostic Metrics

The main results are reported on the F1\-based headline metrics defined in Section[5\.2](https://arxiv.org/html/2608.04056#S5.SS2): Cluster F1 \(per agent, vs\. its cluster majority\), Team F1 \(agent majority, vs\. the annotator majority\), and Overall F1 \(their harmonic mean\)\. Some tables quote raw accuracies \(Cluster Accuracy, Team Accuracy\), which are the same comparisons scored as accuracy instead of F1\-macro\. Table[3](https://arxiv.org/html/2608.04056#A1.T3)gives plain\-English definitions with worked examples\. The diagnostics below support those results and are referenced in tables and captions throughout the paper\.

Table 3:Metric definitions in plain English; all F1 values are macro\-averaged and reported×\\times100\.![Refer to caption](https://arxiv.org/html/2608.04056v1/x1.png)Figure 2:Agent agreement rate \(=1−=1\-disagreement\) across the GPT\-EN methods\.#### YES\-Rate Calibration\.

For each agentjjtrained for clustercjc\_\{j\}, we compute the absolute deviation between the agent’s YES rate on the test set and the mean YES rate of the cluster’s annotators:

CalErrj=\|YES ratej−YES targetcj\|\.\\text\{CalErr\}\_\{j\}\\;=\\;\\lvert\\text\{YES rate\}\_\{j\}\-\\text\{YES target\}\_\{c\_\{j\}\}\\rvert\.We report the mean over the three agents\. This diagnostic detects over\- or under\-specialization: a well\-trained agent should reproduce the labeling frequency of its cluster \(English targets 21\.5% / 43\.8% / 63\.5% for Cluster 1 / Cluster 2 / Cluster 3\)\. Figure[6](https://arxiv.org/html/2608.04056#A3.F6)shows the per\-cluster breakdown\.

#### Agreement Rate\.

The fraction of test texts on which all three agents produce the same label, equal to11minus the disagreement rate\. This diagnostic captures ensemble diversity: values near 100% indicate the agents have collapsed onto the same function \(no useful agent majority\), and values near 0% indicate they have polarized beyond their clusters’ labeling behavior\. We report it descriptively and attach no target range to it\. Figure[2](https://arxiv.org/html/2608.04056#A1.F2)compares methods on this axis\.

#### F1\-macro\.

Macro\-averaged F1, equally weighting the YES and NO classes:

F1macro=12​\(F1YES\+F1NO\)\.\\text\{F1\}\_\{\\text\{macro\}\}=\\tfrac\{1\}\{2\}\(\\text\{F1\}\_\{\\text\{YES\}\}\+\\text\{F1\}\_\{\\text\{NO\}\}\)\.This is the scale on which all headline metrics are reported \(Section[5\.2](https://arxiv.org/html/2608.04056#S5.SS2)\); it is robust to class imbalance \(the English test set is roughly 39% YES, 61% NO\)\.

#### Balanced Accuracy\.

The mean of the per\-class recallsBrodersen et al\. \([2010](https://arxiv.org/html/2608.04056#bib.bib2)\):

BA=12​\(TPTP\+FN\+TNTN\+FP\)\.\\text\{BA\}=\\tfrac\{1\}\{2\}\\left\(\\frac\{\\text\{TP\}\}\{\\text\{TP\}\+\\text\{FN\}\}\+\\frac\{\\text\{TN\}\}\{\\text\{TN\}\+\\text\{FP\}\}\\right\)\.Balanced accuracy equals plain accuracy on a perfectly balanced test set and, unlike plain accuracy, cannot be inflated by always predicting the majority class\. It is reported alongside per\-class F1 in Appendix[H](https://arxiv.org/html/2608.04056#A8)\.

#### Why these are diagnostics, not headline metrics\.

Disagreement Rate and Calibration Error are meaningful only in context: Disagreement Rate is informative only once Team F1 is known \(high disagreement is good when team performance also rises, bad when it drops\), and Calibration Error measures faithfulness to a cluster\-level label distribution that is already reflected in Cluster F1 for all per\-text decisions\. Reporting them as headline metrics would obscure the Cluster–Team tradeoff that is the central design question of a multi\-agent perspectivist system\.

## Appendix BData Sizes, Block Design, and Cluster Coverage

Table[4](https://arxiv.org/html/2608.04056#A2.T4)gives the full statistics of the three English behavioral clusters; Table[5](https://arxiv.org/html/2608.04056#A2.T5)lists the English training data sizes per agent\.

Table 4:Three behavioral clusters on the English subset, with cluster size, mean YES rate, and mean agreement rate\.Table 5:English training data sizes per agent \(C1–C3 = Cluster 1–Cluster 3\)\.The EXIST 2024 dataset has a perfect block design: 58 English groups of 6 annotators, with no cross\-group overlap\. This means two annotators from different groups never label the same tweet\. Table[6](https://arxiv.org/html/2608.04056#A2.T6)shows how this affects cluster coverage, and Table[7](https://arxiv.org/html/2608.04056#A2.T7)and Figure[3](https://arxiv.org/html/2608.04056#A2.F3)quantify per\-cluster annotator coverage on the English test set\.

Table 6:Cluster composition per text in the English subset \(C1–C3 = Cluster 1–Cluster 3\)\.Table 7:Per\-cluster coverage of the 342\-text English test set;kkis the number of annotators from that cluster on a text\.![Refer to caption](https://arxiv.org/html/2608.04056v1/x2.png)Figure 3:Annotators per text on the 342\-text English test set, by cluster \(C1–C3 = Cluster 1–Cluster 3\)\.The Cluster 2 cluster has near\-complete coverage \(98\.3% of texts\), while Cluster 1 \(62\.9%\) and Cluster 3 \(54\.5%\) have sparser coverage\. This asymmetry is handled by constructing preference pairs only where clusters co\-occur and disagree\.

## Appendix CSweep Details and Supplementary Figures

This appendix collects the full sweep results and the supplementary figures and tables referenced from the main text\.

### C\.1SFT vs\. DPO Calibration

Table[8](https://arxiv.org/html/2608.04056#A3.T8)compares SFT and DPO metric by metric in the GPT\-EN setting \(Section[6\.2](https://arxiv.org/html/2608.04056#S6.SS2)\)\.

Table 8:SFT vs\. DPO \(individual\-only,β=0\.1\\beta\{=\}0\.1\) in the GPT\-EN setting\. Calibration and disagreement are defined in Appendix[A](https://arxiv.org/html/2608.04056#A1)\.MetricSFTDPOTeam Accuracy89\.6%86\.6%F1\-macro0\.8910\.858Agreement Rate55\.0%6\.1%C1 YES Rate21\.4%1\.5%\(target: 21\.5%\)C2 YES Rate43\.3%39\.5%\(target: 43\.8%\)C3 YES Rate64\.6%95\.3%\(target: 63\.5%\)Avg Calibration Error0\.6%18\.7%
### C\.2DPOβ\\beta\-Sweep

The DPO overshoot \(93\.9% disagreement, 18\.7% calibration error atβ=0\.1\\beta\{=\}0\.1\) motivates asking whether a stronger KL constraint on the reference policy would prevent it\. We re\-run individual\-only DPO atβ∈\{0\.1,0\.3,0\.5\}\\beta\\in\\\{0\.1,0\.3,0\.5\\\}, keeping all other settings identical\. Results are in Table[9](https://arxiv.org/html/2608.04056#A3.T9)and Figure[4](https://arxiv.org/html/2608.04056#A3.F4)\.

Table 9:DPOβ\\beta\-sweep with individual\-only preferences; Mars\-PO is shown for reference\. Team Acc\. = Team Accuracy\.![Refer to caption](https://arxiv.org/html/2608.04056v1/x3.png)Figure 4:DPOβ\\beta\-sweep in F1\-macro: Avg Cluster F1, Team F1, and Overall F1\.The sweep confirms that individual\-only DPO overshoot is a property of the*objective*, not just the hyperparameter: KL regularization reduces the magnitude but does not remove it\. Adding shared team\-level preference pairs \(Mars\-PO\) reduces disagreement much further, to 34\.5%, while also reaching the highest Team Accuracy in this comparison \(89\.3%\)\.

### C\.3GRPOα\\alpha\-Sweep

Table[10](https://arxiv.org/html/2608.04056#A3.T10)and Figure[5](https://arxiv.org/html/2608.04056#A3.F5)report the full team\-weight sweep summarized in Section[6\.4](https://arxiv.org/html/2608.04056#S6.SS4)\.

Table 10:GRPO team\-reward sweep in the GPT\-EN setting\. Team Acc\. = Team Accuracy\.![Refer to caption](https://arxiv.org/html/2608.04056v1/x4.png)Figure 5:GRPOα\\alpha\-sweep on the convex reward of Eq\.[2](https://arxiv.org/html/2608.04056#S4.E2): Avg Cluster F1, Team F1, and Overall F1 \(F1\-macro×\\times100\)\.
### C\.4Supplementary Figures

Figure[6](https://arxiv.org/html/2608.04056#A3.F6)shows the agent YES rates against their cluster targets in the GPT\-EN setting \(Section[6\.2](https://arxiv.org/html/2608.04056#S6.SS2)\)\. Figures[7](https://arxiv.org/html/2608.04056#A3.F7)and[8](https://arxiv.org/html/2608.04056#A3.F8)visualize the two evaluation axes of Section[5\.2](https://arxiv.org/html/2608.04056#S5.SS2)across all GPT\-EN methods\. Figure[9](https://arxiv.org/html/2608.04056#A3.F9)shows the per\-class breakdown behind DPO’s cluster\-level collapse\. Figure[10](https://arxiv.org/html/2608.04056#A3.F10)compares Team F1 across the four settings \(Section[7](https://arxiv.org/html/2608.04056#S7)\)\.

![Refer to caption](https://arxiv.org/html/2608.04056v1/x5.png)Figure 6:Agent YES rates against their cluster targets \(the cluster’s mean annotator YES rate: 21\.5 / 43\.8 / 63\.5\)\.![Refer to caption](https://arxiv.org/html/2608.04056v1/x6.png)Figure 7:Cluster F1 \(three per\-cluster agent bars\) vs\. Team F1 \(red\) across the GPT\-EN methods; all bars carry 95% Wilson intervals\.![Refer to caption](https://arxiv.org/html/2608.04056v1/x7.png)Figure 8:Avg Cluster F1 \(xx\) vs\. Team F1 \(yy\); dashed contours are iso\-Overall\-F1 lines \(harmonic mean\)\.![Refer to caption](https://arxiv.org/html/2608.04056v1/x8.png)Figure 9:Per\-cluster F1\-YES, F1\-NO, and F1\-macro for each agent \(C1–C3 = Cluster 1–Cluster 3\)\.![Refer to caption](https://arxiv.org/html/2608.04056v1/x9.png)Figure 10:Team F1 across the four settings for each training stage\.

## Appendix DEvaluation on Texts Covered by All Clusters

Cluster F1 is normally computed on per\-cluster subsets of different sizes \(Section[5\.2](https://arxiv.org/html/2608.04056#S5.SS2)\), so the three agents are never scored on exactly the same texts\. The most direct comparison uses only texts where all three clusters are present, and the block design makes this subset small\. On the 342\-text English test set, only 57 texts are covered by all three clusters, 47 of them with a non\-tied annotator majority\. The Spanish test set is friendlier: Cluster 2 and Cluster 3 are present on every Spanish test text and Cluster 1 on 126 of them, so 126 of the 297 texts \(111 with a non\-tied annotator majority\) contain all three clusters\. Withn=47n\{=\}47andn=111n\{=\}111, point estimates are unstable, so we report 95% Wilson CIs throughout\.

Table 11:Team performance on the all\-clusters subset vs\. the full test set;nnis subset texts, with a non\-tied annotator majority in parentheses\.![Refer to caption](https://arxiv.org/html/2608.04056v1/x10.png)Figure 11:Team F1 on the full test set vs\. the all\-clusters subset for the Qwen settings; whiskers are 95% accuracy intervals\.Table[11](https://arxiv.org/html/2608.04056#A4.T11)and Figure[11](https://arxiv.org/html/2608.04056#A4.F11)report the results\. The zero\-shot ensemble and the persona\-prompt baseline post identical subset scores: their votes differ on a single subset text\. On English the subset scores every method higher: the fine\-tuned Qwen methods score 2–5 points above their full\-set Team F1 and the GPT\-EN methods 4–10 \(these texts are also the best\-annotated ones\), while the ranking is preserved: SFT leads both Qwen columns and Qwen DPO stays behind SFT\. On GPT\-ES the effect disappears \(the three fine\-tunes move by−1\.6\-1\.6to\+0\.8\+0\.8\), consistent with its much larger, less selective subset\. The Wilson intervals are wide \(±9\{\\pm\}9–12 pp on the English method rows\) and overlap heavily for the top methods, so the subset corroborates the full\-set conclusions but cannot establish them on its own\. Intervals throughout this appendix are computed on accuracy, the quantity defined per text; the full\-test\-set whiskers in Figure[11](https://arxiv.org/html/2608.04056#A4.F11)are bootstrap intervals\. Per\-text predictions for the GPT\-EN methods come from a separate prediction pass over the same fine\-tuned models; they feed the GPT\-EN rows here and the behavioral\-fidelity analysis of Section[8\.1](https://arxiv.org/html/2608.04056#S8.SS1)\. That pass reproduces the stored GPT\-EN team aggregates to within 0\.9 pp and supplies the GPT\-EN balanced accuracies of Table[2](https://arxiv.org/html/2608.04056#S7.T2)\.

## Appendix EBehavioral Fidelity Details

Table[12](https://arxiv.org/html/2608.04056#A5.T12)places each GRPO\-stage agent’s behavioral features \(YES rate, agreement rate, label entropy\) within its cluster’s per\-annotator distribution, in the Qwen\-EN and Qwen\-ES settings, reporting z\-scores, percentiles, and pseudo\-annotator Wasserstein distances \(Section[8\.1](https://arxiv.org/html/2608.04056#S8.SS1)\); the GPT\-stage z\-scores quoted in that section come from the same analysis run on the GPT\-EN agents\. Figure[12](https://arxiv.org/html/2608.04056#A5.F12)visualizes the Qwen\-EN setting\. Each agent marker there carries a bootstrap 95% CI, and the vertical line inside each violin marks the cluster median\.

![Refer to caption](https://arxiv.org/html/2608.04056v1/x11.png)Figure 12:SFT and GRPO agent feature values \(markers\) inside each cluster’s per\-annotator distribution \(violins\), in the Qwen\-EN setting\.Table 12:Behavioral fidelity of the GRPO\-stage Qwen\-EN and Qwen\-ES agents: human mean±\\pmstd, agent value, z\-score, percentile, Wasserstein\-1 distance \(W1W\_\{1\}\)\.
## Appendix FClustering Ablation Details

This appendix provides the details behind the two\-feature clustering ablation of Section[8\.2](https://arxiv.org/html/2608.04056#S8.SS2)\. For both languages we re\-run the K\-Means scan \(k=2​…​10k\{=\}2\{\\ldots\}10\) on standardized \(agreement rate, label entropy\) only, dropping the YES rate, and compare against the three\-feature clustering via adjusted Rand index \(ARI\) and cluster crosstabs\.

Label entropy is a deterministic function of the YES rate \(H​\(p\)H\(p\)withppthe annotator’s YES rate\) and is symmetric aroundp=0\.5p\{=\}0\.5: an annotator who labels 20% YES and one who labels 80% YES have identical entropy\. Dropping the YES rate therefore removes the*direction*of an annotator’s leaning, and the ablation confirms the consequence empirically: atk=3k\{=\}3the two\-feature clustering leaves a minimum pairwise gap in cluster\-mean YES rate of only 0\.06 on English and 0\.04 on Spanish, against roughly 0\.20 for the three\-feature reference on both languages\.

#### Choosingkk\.

Table\-free summary of the cluster\-count diagnostics behind Section[3](https://arxiv.org/html/2608.04056#S3), computed byk\_selection\.pyfork=2​…​6k\{=\}2\{\\ldots\}6in both languages\. On Spanish, YES\-rate MAD \(the mean absolute deviation between an annotator’s YES rate and its own cluster mean, the quantity a single agent per cluster has to absorb\) is 0\.099 atk=2k\{=\}2, 0\.059 atk=3k\{=\}3, and 0\.067 atk=4k\{=\}4; the minimum gap between consecutive cluster mean YES rates is 0\.188, 0\.204, and 0\.024; and the smallest cluster holds 19%, 17%, and 14% of annotators\. English behaves the same way \(k=3k\{=\}3: MAD 0\.075, gap 0\.196\), and there silhouette agrees\. Beyondk=4k\{=\}4both languages fragment: atk=5k\{=\}5the smallest Spanish cluster holds 3% of annotators and two cluster means sit 0\.018 apart\.

Table 13:Clustering feature ablation: the 3\-feature k\-means reference \(YES rate, agreement rate, label entropy\) vs\. the 2\-feature variant without YES rate\.Table[13](https://arxiv.org/html/2608.04056#A6.T13)reports the full scan\. On both languages the two\-feature scan prefersk=2k\{=\}2\. Atk=3k\{=\}3, the English two\-feature clustering still overlaps the reference assignment \(ARI 0\.711\) but shrinks the third cluster to a rump of 24 annotators \(6\.9% minimum share\) and leaves a minimum cluster\-mean YES\-rate gap of only 0\.059; on Spanish the assignment barely resembles the reference at all \(ARI 0\.142, minimum gap 0\.043\)\. The crosstabs show individual two\-feature clusters absorbing annotators from both the lowest\- and highest\-YES three\-feature clusters\. Without the YES rate, the clustering cannot separate annotators who lean YES from annotators who lean NO\.

Table 14:Class\-imbalance diagnostics: F1\-YES / F1\-NO are per\-class team F1; Bal\. Acc\. is balanced accuracy; Calib\. Err\. is mean calibration error\.
#### Why not content\-derived features?

Keyword and content features, and the EXIST task\-2/3 labels \(intention and category\), are excluded from the clustering by design, for three reasons\. First, the block\-design confound: annotator groups label disjoint text sets, so any content\-derived feature varies across groups by construction, and clustering would recover the group partition rather than differences in labeling behavior\. Second, sparsity: task\-2/3 labels exist only for texts an annotator labels YES, so low\-YES\-rate annotators would be characterized by very few labels\. Third, coupling: features built from the same labels used for evaluation would tie the cluster definitions more tightly to the evaluation targets, making Cluster F1 partly circular\.

## Appendix GLocal Training Details

The Qwen\-EN and Qwen\-ES settings fine\-tune Qwen3\-8BYang et al\. \([2025](https://arxiv.org/html/2608.04056#bib.bib23)\)with LoRA adaptersHu et al\. \([2022](https://arxiv.org/html/2608.04056#bib.bib7)\)usingmlx\-lmon the MLX frameworkHannun et al\. \([2023](https://arxiv.org/html/2608.04056#bib.bib6)\), on a single 64 GB accelerator inbf16\. LoRA uses themlx\-lmdefault rank, applied to the top 16 transformer layers\. All stages use batch size 8, maximum sequence length 512, and seed 42\. SFT trains for 3 epochs; DPO trains for 2 epochs with learning rate10−510^\{\-5\},β=0\.1\\beta\{=\}0\.1, and the frozen SFT policy as reference; the GRPO stage re\-trains for 1 epoch on the rejection\-sampled completions kept by the midpoint rule \(Section[4\.6](https://arxiv.org/html/2608.04056#S4.SS6)\)\. Predictions are scored at the single\-token level by comparing the probabilities ofYESandNOas the first generated token, and the DPO loss is computed exactly on these single\-token continuations\.

## Appendix HPer\-Class F1 and Balanced Accuracy

The English test set is 39\.3% YES / 60\.7% NO among non\-tied texts, while the Spanish test set leans the other way \(58\.2% YES\)\. This appendix reports per\-class F1 \(F1\-YES, F1\-NO\) and balanced accuracy \(Appendix[A](https://arxiv.org/html/2608.04056#A1)\) for the Qwen and GPT\-ES settings, whose evaluations record the full per\-class breakdown, complementing the F1\-macro headline numbers\. The GPT\-EN setting is covered by the same diagnostics elsewhere: Figure[9](https://arxiv.org/html/2608.04056#A3.F9)gives its per\-cluster F1\-YES/F1\-NO breakdown and Table[2](https://arxiv.org/html/2608.04056#S7.T2)its balanced accuracies\.

Table[14](https://arxiv.org/html/2608.04056#A6.T14)reports the results\. Balanced accuracy tracks F1\-macro within 2\.6 pp across fine\-tuned methods, usually within 1; the largest gap is the GPT\-ES zero\-shot ensemble \(64\.5 balanced accuracy vs\. 57\.4 F1\-macro\), whose NO bias on a YES\-majority test set hurts F1 more\. The per\-class view also locates DPO\-only’s damage \(Section[7](https://arxiv.org/html/2608.04056#S7)\): team\-level scores degrade only moderately \(Qwen\-EN F1\-YES / F1\-NO fall from 81\.8 / 87\.6 under SFT to 76\.9 / 79\.4\), while the collapse is inside the clusters, where the pinned Cluster 1 and Cluster 3 agents lose one class entirely \(per\-cluster F1\-macro 43\.8 and 41\.6\)\.