From a Word-Level Dictionary to Sentence-Level Semantics: Multilingual Grievance Labelling with Contextual Models

# From a Word-Level Dictionary to Sentence-Level Semantics: Multilingual Grievance Labelling with Contextual Models
Source: [https://arxiv.org/html/2607.20946](https://arxiv.org/html/2607.20946)
Lin Tian Behavioral Data Science University of Technology Sydney Sydney, Australia Lin\.Tian\-3@uts\.edu\.au &Marian\-Andrei Rizoiu Behavioral Data Science University of Technology Sydney Sydney, Australia Marian\-Andrei\.Rizoiu@uts\.edu\.au

###### Abstract

Grievance is one of the warning signs analysts look for when assessing threats of violence\. It is increasingly measured at scale from online text, most often with word\-level lexicons like the Grievance Dictionary that score by matching weighted terms\. Such matching is a fast and transparent proxy, but it cannot resolve whether a term is asserted, quoted, negated, or condemned\. These lexicons are also often evaluated on pools enriched with the very examples they retrieve, so a high score partly reflects agreement with the lexicon’s own selection rule\. Examining a five\-language, 2,000\-item evaluation pool, we find its halves separated almost perfectly by the lexicon itself: every item labeled “random” is in fact lexicon\-negative, so the lexicon’s apparent macro\-AUROC of 0\.686 collapses to a 0\.500 floor fixed by construction\. We keep the dictionary’s 22\-construct ontology but replace term matching with context\-reading models, evaluated on a non\-circular benchmark that separates unconditional\-random, lexicon\-positive, and lexicon\-negative strata across five languages\. Reading the full post rather than the target sentence alone helps most where the lexicon is silent, raising average precision on lexicon\-negative text from 0\.14 to 0\.20, with the largest gains on quoted, implicit, and cross\-sentence grievance\. Together, these results show that grievance is measured more faithfully by reading the surrounding context, and more honestly when tested on text the lexicon did not select\. We release our code and benchmark at[https://github\.com/behavioral\-ds/multilingual\_grievance](https://github.com/behavioral-ds/multilingual_grievance)\.

From a Word\-Level Dictionary to Sentence\-Level Semantics: Multilingual Grievance Labelling with Contextual Models

Lin TianBehavioral Data ScienceUniversity of Technology SydneySydney, AustraliaLin\.Tian\-3@uts\.edu\.auMarian\-Andrei RizoiuBehavioral Data ScienceUniversity of Technology SydneySydney, AustraliaMarian\-Andrei\.Rizoiu@uts\.edu\.au

## 1Introduction

On 19 February 2020, before killing nine people in Hanau, Germany, the attacker posted a German\-language manifesto and videos on his personal website setting out the grievances he believed justified the act\.111[https://www\.france24\.com/en/20200219\-shooting\-leaves\-several\-people\-dead\-in\-western\-german\-city\-of\-hanau](https://www.france24.com/en/20200219-shooting-leaves-several-people-dead-in-western-german-city-of-hanau)This kind of leakage is the rule rather than the exception: most lone attackers commit their grievances to text or recordings before acting, typically in the oblique register of a warning behavior rather than an explicit threat\(Meloyet al\.,[2012](https://arxiv.org/html/2607.20946#bib.bib5); Gillet al\.,[2014](https://arxiv.org/html/2607.20946#bib.bib6)\); the Christchurch attacker, for one, aired his intentions on anonymous forums for months beforehand\(Wilsonet al\.,[2024](https://arxiv.org/html/2607.20946#bib.bib8); Bogle,[2026](https://arxiv.org/html/2607.20946#bib.bib9)\)\. Because analysts cannot read every post at scale\(Brynielssonet al\.,[2013](https://arxiv.org/html/2607.20946#bib.bib11); Cohenet al\.,[2014](https://arxiv.org/html/2607.20946#bib.bib10)\), automated grievance measurement has become an attractive first filter\. Yet the language that matters is context\-laden and often adversarial\. The Christchurch attacker’s manifesto\(Royal Commission of Inquiry into the Attack on Christchurch Mosques on 15 March 2019,[2020](https://arxiv.org/html/2607.20946#bib.bib7)\)was deliberately layered with irony, quotation, and internet in\-jokes engineered to be taken literally and misread by anyone—human or machine—scanning it for meaning\.222[https://www\.bellingcat\.com/news/2019/03/15/shitposting\-inspirational\-terrorism\-and\-the\-christchurch\-mosque\-massacre/](https://www.bellingcat.com/news/2019/03/15/shitposting-inspirational-terrorism-and-the-christchurch-mosque-massacre/)

The core difficulty is that the same words carry different force in different frames\. The word*threat*can express a warning \(“They threatened us”\), report someone else’s speech \(“The article quoted the threat”\), or reject it \(“We condemn these threats”\)\. A term\-matching instrument sees closely related evidence in all three cases; a reader uses syntax, stance, discourse, and speaker attribution to distinguish them\. The reverse problem is equally important: “They took everything from me\. Tomorrow, they will answer for it” can express grievance and menace without using an obvious dictionary term\. Should such an instrument count the terms a text contains, or judge what its author actually expressed?

These are not edge cases\. Grievance, a perceived wrong relevant to professional threat assessment, is a latent, context\-dependent construct, not a bag of words\(Meloyet al\.,[2012](https://arxiv.org/html/2607.20946#bib.bib5); Gillet al\.,[2014](https://arxiv.org/html/2607.20946#bib.bib6)\)\. The Grievance Dictionary\(van der Vegtet al\.,[2021](https://arxiv.org/html/2607.20946#bib.bib1)\)offers an interpretable starting point: a weighted lexicon of 22 constructs, later translated and psychometrically evaluated in Dutch, German, and Italian\(van der Vegtet al\.,[2025](https://arxiv.org/html/2607.20946#bib.bib2)\)\. Its inventory fixes*what*to measure; weighted term occurrence is only one way to decide*whether the author expressed it*\.

Examining an existing five\-language evaluation pool, we find its two halves separated perfectly by the lexical anchor \(every “lexicon\-positive” item scores at least 0\.30, every “random” item below 0\.05\): the second half is a*lexicon\-negative challenge set*, not a random sample, yet three\-annotator consensus still marks constructs in 228 of its items\. Being target\-only, the pool also cannot test whether context changes judgments\. This raises three questions: how much does lexicon\-conditioned sampling distort what a lexical score appears to measure \(RQ1\); how prevalent are grievance constructs in unconditionally sampled text, and how well are they measured when the lexicon does not choose the test set \(RQ2\); and does reading the full post change judgments and improve models, and where \(RQ3\)?

To answer these questions, we contribute: \(i\) aselection analysisquantifying the circularity and the consensus\-marked evidence hidden below the anchor threshold \(RQ1\); \(ii\) ameasurement frameworkthat keeps the dictionary’s 22\-construct ontology, demotes lexical scores to one fallible instrument, and types its failures by stance, attribution, discourse, and multilingual realization; \(iii\) anon\-circular five\-language benchmarkwith an unconditional\-random stratum, giving the first base\-rate estimate for these constructs \(12\.9%12\.9\\%\), plus a counterbalanced target\-only/full\-context subset \(RQ2\); and \(iv\)matched\-pair evidence that context works: a full\-post encoder beats its target\-only twin on every stratum, most \(39%39\\%relative average precision\) where the lexicon is silent, with gains concentrated on quoted, implicit, and cross\-sentence grievance \(RQ3\)\.

![Refer to caption](https://arxiv.org/html/2607.20946v1/x1.png)Figure 1:Measuring grievance by reading context rather than matching terms\.\(a\)Two real targets from the five\-language pool\. An Italian sentence condemning violence still activates six lexical constructs \(three shown\), while an English sentence with anchor score 0 carries a three\-annotator grievance rating of 2 whose perceived wrong is supplied only by the source post\.\(b\)We keep the Grievance Dictionary’s 22\-construct ontology but score each construct by reading the marked target inside its full post, with one multilingual encoder and one shared label space across English, Dutch, German, Italian, and French\. Targets are shortened with user metadata omitted; their target\-only consensus labels are not full\-context adjudications\.
## 2Related Work

##### Grievance as lexical measurement\.

Threat\-assessment research identifies grievance and related warning behaviors as potentially relevant signals, while emphasizing that no isolated linguistic marker proves intent\(Meloyet al\.,[2012](https://arxiv.org/html/2607.20946#bib.bib5); Gillet al\.,[2014](https://arxiv.org/html/2607.20946#bib.bib6)\)\. Computational work has consequently explored linguistic markers of radicalization and threat\(Brynielssonet al\.,[2013](https://arxiv.org/html/2607.20946#bib.bib11); Cohenet al\.,[2014](https://arxiv.org/html/2607.20946#bib.bib10)\)\. The Grievance Dictionary\(van der Vegtet al\.,[2021](https://arxiv.org/html/2607.20946#bib.bib1)\)organizes weighted terms into 22 constructs in the tradition of psychologically grounded resources such as LIWC\(Tausczik and Pennebaker,[2010](https://arxiv.org/html/2607.20946#bib.bib12); Pennebakeret al\.,[2015](https://arxiv.org/html/2607.20946#bib.bib15)\)\. Later translations to Dutch, German, and Italian show the value of multilingual extension but also uneven reliability, especially for some Italian categories\(van der Vegtet al\.,[2025](https://arxiv.org/html/2607.20946#bib.bib2)\)\. We retain this construct inventory while treating its term lists as one fallible measurement instrument rather than ground truth\.

##### Context and functional evaluation\.

Lexicons are transparent, inexpensive, and reproducible, but bag\-of\-terms scores do not model compositional meaning or discourse stance\. Related work on abuse and hate speech demonstrates failures under negation, quotation, counter\-speech, reclaimed language, and speaker attribution\(Röttgeret al\.,[2021](https://arxiv.org/html/2607.20946#bib.bib16); Kennedyet al\.,[2020](https://arxiv.org/html/2607.20946#bib.bib17)\)\. Functional tests are particularly useful because aggregate in\-domain scores can conceal such systematic failures\. Our focus differs in two ways: grievance spans a broader construct inventory than hate or offense, and our unit is a target sentence interpreted inside a marked source post\. We further treat the method used to*sample*the evaluation set as part of the validity argument, since a lexicon\-conditioned pool preferentially exposes the cases the lexicon already retrieves\(Hovy and Prabhumoye,[2021](https://arxiv.org/html/2607.20946#bib.bib18)\)\.

##### Construct validity and subjective annotation\.

Separating a construct from its operationalization follows classical construct validity: scores are meaningful only relative to the inference supported by the instrument\(Cronbach and Meehl,[1955](https://arxiv.org/html/2607.20946#bib.bib13); Messick,[1995](https://arxiv.org/html/2607.20946#bib.bib14)\)\. Sentence\-level grievance is subjective, so replicated annotation and an explicit treatment of disagreement are necessary\(Artstein and Poesio,[2008](https://arxiv.org/html/2607.20946#bib.bib28); Krippendorff,[2004](https://arxiv.org/html/2607.20946#bib.bib29); Plank,[2022](https://arxiv.org/html/2607.20946#bib.bib30); Umaet al\.,[2021](https://arxiv.org/html/2607.20946#bib.bib31)\)\. The existing target\-only pool stores only the final consensus of three annotators per item, not their independent ratings, so its disagreement statistics cannot be reconstructed; our contextual benchmark retains both individual and adjudicated labels by language and condition\.

##### Multilingual contextual models\.

Multilingual encoders such as mBERT, XLM\-R, and mDeBERTa support shared representations across languages\(Devlinet al\.,[2019](https://arxiv.org/html/2607.20946#bib.bib19); Conneauet al\.,[2020](https://arxiv.org/html/2607.20946#bib.bib20); Heet al\.,[2023](https://arxiv.org/html/2607.20946#bib.bib21)\)and are benchmarked for cross\-lingual generalisation\(Huet al\.,[2020](https://arxiv.org/html/2607.20946#bib.bib23)\)\. Multilingual abuse and offensive\-language benchmarks demonstrate both the value and the difficulty of cross\-lingual evaluation\(Basileet al\.,[2019](https://arxiv.org/html/2607.20946#bib.bib24); Zampieriet al\.,[2019](https://arxiv.org/html/2607.20946#bib.bib25); Mandlet al\.,[2019](https://arxiv.org/html/2607.20946#bib.bib26); Pamungkas and Patti,[2019](https://arxiv.org/html/2607.20946#bib.bib27)\)\. Our target is not generic toxicity: the 22 grievance constructs include grievance, fixation, desperation, planning, threat, and weaponry\. A shared contextual encoder is therefore evaluated per language and per construct, with pooled scores treated as secondary\.

##### LLM supervision and distillation\.

LLMs can provide scalable provisional annotations\(Gilardiet al\.,[2023](https://arxiv.org/html/2607.20946#bib.bib35); Dinget al\.,[2023](https://arxiv.org/html/2607.20946#bib.bib36)\), and their outputs can supervise smaller models\(Hintonet al\.,[2015](https://arxiv.org/html/2607.20946#bib.bib33); Sanhet al\.,[2019](https://arxiv.org/html/2607.20946#bib.bib34); Wanget al\.,[2021](https://arxiv.org/html/2607.20946#bib.bib38); Hsiehet al\.,[2023](https://arxiv.org/html/2607.20946#bib.bib37); Liet al\.,[2025](https://arxiv.org/html/2607.20946#bib.bib4)\), though such pseudo\-labels are noisy and demand careful selection\(Songet al\.,[2022](https://arxiv.org/html/2607.20946#bib.bib39)\)\. Sample\-wise weighting in knowledge distillation is itself established\(Luet al\.,[2021](https://arxiv.org/html/2607.20946#bib.bib3)\); we evaluate uncertainty\-weighted variants as a secondary training condition \(Appendix[C](https://arxiv.org/html/2607.20946#A3)\)\. In this work, distillation is an implementation route to a deployable multilingual contextual baseline; human validity and non\-circular evaluation remain the central contributions\.

## 3Methodology

### 3\.1From term matching to construct expression

The Grievance Dictionary bundles two separable objects: a construct inventory𝒞=\{c1,…,c22\}\\mathcal\{C\}=\\\{c\_\{1\},\\ldots,c\_\{22\}\\\}and a lexical rule that scores each construct from weighted term matches\. We retain the inventory and replace the rule\. A postp=\(x1,…,xJp\)p=\(x\_\{1\},\\ldots,x\_\{J\_\{p\}\}\)in languageℓ\\ellcontains a target sentencexjx\_\{j\}; for each constructcca system predicts

yj​c=fc​\(xj∣p,ℓ\)∈\[0,1\],y\_\{jc\}=f\_\{c\}\(x\_\{j\}\\mid p,\\ell\)\\in\[0,1\],the degree to which the post’s author expressesccthroughxjx\_\{j\}, read in the context of the full marked post\. The prediction unit is always the target sentence, so every output stays comparable to the dictionary’s; the post enters only as evidence for resolving stance, attribution, and cross\-sentence meaning\. An overall grievance judgment is predicted on the same0–33scale by a separate head, never as a twenty\-third construct\.

Two properties fix the task\. It preserves the ontology: each output maps one\-to\-one onto a dictionary construct\. It changes the measurement: a listed term is neither necessary nor sufficient for a positive label, which demotes the lexical rule from the definition ofyj​cy\_\{jc\}to one fallible estimator of it\.

### 3\.2Transferring the 22 constructs across languages

The object that must survive translation is the ontology, not any word list\. Each constructc∈𝒞c\\in\\mathcal\{C\}is fixed by a single language\-independent definition; the English source definitions are rendered into Dutch, German, Italian, and French by native speakers with back\-translation checks, so an annotator or model in any language scores against the same construct rather than a locally re\-derived one\. The 22 labels therefore form one shared output space: constructccoccupies the same index for every languageℓ\\ell, and no language introduces, splits, or merges a construct\.

Transfer enters each instrument at a different level\. The lexical anchor transfers at the*term*level: its weighted lists are the validated translations of the Dutch, German, and Italian dictionaries\(van der Vegtet al\.,[2025](https://arxiv.org/html/2607.20946#bib.bib2)\)plus an unvalidated French seed, matched in a language\-specific stem space \(§[3\.6](https://arxiv.org/html/2607.20946#S3.SS6)\)\. The contextual encoders transfer at the*representation*level: a single multilingual encoder carries one2222\-dimensional prediction head across all languages, so a label observed in one language updates the shared construct space rather than a per\-language head\. The LLM teacher transfers at the*definition*level: its rubric embeds the translated definitions and elicits per\-construct ordinal judgments in the item’s own language, emitting soft labels directly in the shared label space \(§[3\.7](https://arxiv.org/html/2607.20946#S3.SS7)\)\. At every level the label set is identical across languages; only its realization changes\.

Holding the definition fixed while surface form varies makes cross\-lingual equivalence a claim we test, not one we inherit from translation\. We report every metric per language and per construct and treat measurement invariance—whether a construct score means the same thing in each language—as an empirical question, pooling across languages only when it holds \(§[3\.4](https://arxiv.org/html/2607.20946#S3.SS4)\)\. The unvalidated French lexicon is kept separate throughout, so a gap in a translated term list is never mistaken for a failure of the construct\.

### 3\.3Error typology for context failures

We evaluate against a fixed typology of the mistakes term matching makes, so that an aggregate gap can be attributed to a specific linguistic cause rather than left as noise\.False positivesarise when a matched term is negated, quoted, reported, hypothetical, sarcastic, or condemned, or when it is attributed to someone other than the author\.False negativesarise when grievance is implicit, paraphrased outside the lexicon, expressed in a language variety the term list misses, or completed across sentence boundaries\. Code\-switching, hashtags, emoji, non\-standard spelling, and references to unavailable media are logged as cross\-cutting conditions\. Each error case in the analysis carries five fields: the target sentence, the minimal context that fixes its reading, the lexical evidence, the human rationale, and the target\-only and full\-context system outputs\. Context is credited only when it changes the supported interpretation, never when it merely lengthens the input\.

### 3\.4Sampling strata and their estimands

Evaluation starts from a frozen, versioned populationΠ\\Piof eligible sentences, defined before any lexical scoring\. Three disjoint strata partition the labeled data, each identifying a distinct estimand\.

Unconditional random \(UU\)\.A uniform draw fromΠ\\Pi\.UUalone identifies marginal quantities: the per\-construct prevalenceπc=PrΠ⁡\[author expresses​c\]\\pi\_\{c\}=\\Pr\_\{\\Pi\}\[\\text\{author expresses \}c\]and population\-level operating characteristics\.

Lexicon\-positive \(PP\)\.Sentences with overall anchor scoreaoverall​\(xj\)≥τ\+a\_\{\\mathrm\{overall\}\}\(x\_\{j\}\)\\geq\\tau\_\{\+\},τ\+=0\.30\\tau\_\{\+\}=0\.30\.PPidentifies the anchor’s precisionPr⁡\[author expresses​c∣aoverall≥τ\+\]\\Pr\[\\text\{author expresses \}c\\mid a\_\{\\mathrm\{overall\}\}\\\!\\geq\\\!\\tau\_\{\+\}\]and isolates contextual false positives among retrieved cases\.

Lexicon\-negative \(NN\)\.Sentences withaoverall​\(xj\)≤τ−a\_\{\\mathrm\{overall\}\}\(x\_\{j\}\)\\leq\\tau\_\{\-\},τ−=0\.05\\tau\_\{\-\}=0\.05\.NNidentifies the mass the anchor misses,Pr⁡\[author expresses​c∣aoverall≤τ−\]\\Pr\[\\text\{author expresses \}c\\mid a\_\{\\mathrm\{overall\}\}\\\!\\leq\\\!\\tau\_\{\-\}\]; it measures coverage failure and deliberately does not represent the base rate\.

BecausePPandNNare conditional estimands, they combine into a population statement only through their inclusion probabilities: a pooled score over an arbitraryP:NP\{:\}Nmixture estimates no well\-defined quantity\. We therefore report population claims only fromUU, or from a stratified sample reweighted to a stated target population\. All sentences from one post stay in one partition, and macro comparisons use a construct set with both classes present in every compared stratum, so a change in the evaluable set cannot masquerade as a change in model quality\.

### 3\.5Annotation protocol

Raters score each construct on a0–33ordinal scale \(0absent,11slight or ambiguous,22moderate,33strong\) and judge only the communicative act attributable to the author\. Quoted, reported, hypothetical, or condemned content scores0unless the surrounding post endorses it; conversely, context may license a construct that is only implicit in the target sentence\. Each item also records stance, target, the applicable failure category, and whether an unavailable medium is required to decide\.

In this process, at least two native\-speaking raters annotated each item, and a third adjudicated any disagreement that crossed the absent/present boundary\. A counterbalanced subset was rated target\-only by one panel and full\-context by another, yielding the paired reference that measures context\-induced label change directly\.

### 3\.6Systems

##### Lexical anchor\.

For constructccthe anchor length\-normalizes and saturates matched term weights,

ac​\(xj\)=1−exp⁡\(−4n​∑t∈xjwc​\(t\)​m​\(t\)\),a\_\{c\}\(x\_\{j\}\)=1\-\\exp\\\!\\left\(\-\\frac\{4\}\{n\}\\sum\_\{t\\in x\_\{j\}\}w\_\{c\}\(t\)\\,m\(t\)\\right\),withnnthe target token count andwc​\(t\)w\_\{c\}\(t\)the dictionary weight in a language\-specific stem space\. The modifierm​\(t\)∈\[0,1\]m\(t\)\\in\[0,1\]discounts a match inside a4848\-character window around regex\-detected negation, quotation/reporting, or condemnation cues, hard\-coding the easy stance cases so that residual errors isolate the discourse phenomena a rule cannot reach\. We report the anchor with and withoutmm, and keep the unvalidated French seed separate from the English, Dutch, German, and Italian resources\.

##### Contextual encoders\.

The target\-only encoder is mDeBERTa\-v3\-base\(Heet al\.,[2023](https://arxiv.org/html/2607.20946#bib.bib21)\)applied toxjx\_\{j\}alone, with one sigmoid output per construct; it controls for sentence\-level semantics while withholding context\. The full\-post encoder shares that architecture and consumes

qj=\[CLS\]​xj​\[SEP\]​p⟨j⟩​\[SEP\],q\_\{j\}=\[\\mathrm\{CLS\}\]\\;x\_\{j\}\\;\[\\mathrm\{SEP\}\]\\;p^\{\\langle j\\rangle\}\\;\[\\mathrm\{SEP\}\],wherep⟨j⟩p^\{\\langle j\\rangle\}is the source post with the target marked\. The target is placed first and never truncated; context fills the remaining512512\-token budget\. Architecture, supervision, optimization, and all five seeds are held fixed between the two variants \(§[4](https://arxiv.org/html/2607.20946#S4)\), so their paired difference measures the value of context and nothing else\.

##### Reference systems\.

XLM\-R\(Conneauet al\.,[2020](https://arxiv.org/html/2607.20946#bib.bib20)\)substitutes a second multilingual encoder to separate context gains from architecture; a Qwen3 LLM\(Qwen Team,[2025](https://arxiv.org/html/2607.20946#bib.bib22)\)is evaluated both rubric\-prompted zero\-shot and fine\-tuned, and also serves as the distillation teacher; and a human\-supervised encoder is trained where adjudicated labels suffice\. The contrasts among them attribute any improvement to context rather than to teacher capacity\.

### 3\.7Supervision is a secondary axis

Replicated human labels are scarce, so the deployable encoders were distilled from the teacher’s soft labels and evaluated only on held\-out human annotations, with standard knowledge distillation as the primary training condition, whose soft targets act like a learned label smoothing\(Mülleret al\.,[2019](https://arxiv.org/html/2607.20946#bib.bib40)\)\. One variant downweighted teacher targets by self\-consistency\(Wanget al\.,[2023](https://arxiv.org/html/2607.20946#bib.bib41)\)or self\-reported confidence\.

## 4Experimental Setup

### 4\.1Examined materials

Our dataset contains 169,368 public Facebook\-post records obtained through the Meta Content Library, with stored timestamps from April 2018 to March 2026; the analysis covers that complete stored range\. The posts span political protests, climate and bushfire debates, the Amazon fires, gender\-violence campaigns, the FIFA World Cup, BTS and Taylor Swift tours, European peace and rearmament debates, and the 2026 Winter Olympics\. The corpus spans more than 90 languages—primarily Italian, English, and French, with event\-specific coverage in Spanish, Dutch, Armenian, Russian, Arabic, and Persian—of which the five benchmark languages are the focus here\.

The data collection follows the strata in §[3\.4](https://arxiv.org/html/2607.20946#S3.SS4)\. We first froze the eligible post population and created post\-grouped train, development, and test partitions\. Language identification was verified before annotation, and exact and near duplicates were removed across partitions, including translated or templated synthetic text\. Synthetic examples supported training diagnostics but never entered development or test\.

Within every test language, we drewUUbefore running any lexical system, then drewPPandNNas diagnostic strata \(per\-language sizes in Table[3](https://arxiv.org/html/2607.20946#S5.T3)\)\. At least two native\-speaking raters annotated every test item, with adjudication as described in §[3\.5](https://arxiv.org/html/2607.20946#S3.SS5)\. Inter\-annotator agreement before adjudication wasκ=0\.72\\kappa=0\.72\(overall\), ranging from0\.680\.68–0\.790\.79across strata and languages\. A counterbalanced subset received independent target\-only and full\-context ratings, providing the reference needed to measure context\-induced label change\.

### 4\.2Model comparison

We compared the lexical anchor, its heuristic modifier variant, target\-only and full\-post versions of mDeBERTa and XLM\-R, prompted and fine\-tuned Qwen3, and a human\-supervised encoder where adjudicated training data sufficed\. Distilled systems used the same teacher labels and post\-grouped partitions\. Target\-only and full\-post variants shared hyperparameters, seeds, and supervision, so this paired design makes input context the only intended difference\.

Encoder experiments used five seeds \(13, 17, 23, 29, and 31\), a 512\-token input budget, AdamW, and development\-selected stopping\. Exact model identifiers, teacher prompts, generation parameters, label dates, compute, and costs are recorded with each run\. Hyperparameters were selected on development data only\.

### 4\.3Metrics and inference

Because grievance constructs are sparse, average precision is the primary ranking metric; AUROC is secondary\. We additionally report recall at a development\-selected precision, macro Spearman correlation with the 0–3 ratings, calibration error\(Guoet al\.,[2017](https://arxiv.org/html/2607.20946#bib.bib42); Naeiniet al\.,[2015](https://arxiv.org/html/2607.20946#bib.bib43)\), and per\-construct prevalence inUU\. Metrics are reported separately forUU,PP, andNN, by language and construct\. Macro comparisons use the same evaluable construct set for every compared system and stratum\.

The principal context effect is the paired difference between full\-post and target\-only systems on items whose human judgment is context\-sensitive\. We also report performance by failure category \(quotation, negation, condemnation, implicit expression, cross\-sentence completion, and code\-switching\)\. Confidence intervals use a post\-clustered bootstrap\.

## 5Results

We first show that the existing pool selects its own test set \(§[5\.1](https://arxiv.org/html/2607.20946#S5.SS1)\), then report the benchmark built to remove that circularity: its composition \(§[5\.2](https://arxiv.org/html/2607.20946#S5.SS2)\), the primary system comparison \(§[5\.3](https://arxiv.org/html/2607.20946#S5.SS3)\), and where post context helps \(§[5\.4](https://arxiv.org/html/2607.20946#S5.SS4)\)\.

### 5\.1The existing pool selects its own test set

#### The “random” half is lexicon\-negative

Table[1](https://arxiv.org/html/2607.20946#S5.T1)reports the central analysis\. The 2,000\-item pool is an equal mixture of two non\-overlapping score regions\. Every lexicon\-positive item has overall anchor score at least 0\.3008; every item previously called random has score at most 0\.0493, and 985 of 1,000 have exactly zero\. The latter is not an unconditional sample from the corpus\. We therefore rename it the lexicon\-negative challenge stratum\.

Table 1:Analysis of the existing target\-only rating pool\.“Any” counts items for which the three\-annotator consensus assigned at least the given ordinal value to one of 22 constructs\. Macro\-AUROC uses the same 16 constructs with both classes inPPandNN\. The pooled row describes this chosen 50/50 mixture, not a population estimate\.
#### Lexicon\-negative does not mean construct\-negative

The anchor is nearly silent inNN, but the consensus labels are not\. Table[2](https://arxiv.org/html/2607.20946#S5.T2)shows that the coverage gap appears in all five languages\. These items are candidates for implicit expression, translation gaps, or annotation error; target\-only consensus labels cannot distinguish those explanations\. They nevertheless show why retrieved\-only evaluation is insufficient: it excludes a substantial set of consensus\-marked cases by construction\.

Table 2:Items with at least one nonzero construct rating in the lexicon\-positive and lexicon\-negative strata\. French uses the unvalidated seed lexicon; German and Dutch have smaller samples\.The thresholds matter\. A score of 1 explicitly includes slight or ambiguous evidence, while only 117 items in the complete pool receive any score of at least 2\. The data pool therefore identifies cases for contextual reannotation rather than establishing their prevalence, which motivates the benchmark below\.

### 5\.2A non\-circular benchmark: composition

We collected the benchmark of §[3\.4](https://arxiv.org/html/2607.20946#S3.SS4)with full marked posts, so context is available at both annotation and inference, and, crucially, added the unconditional\-random stratumUUthat the data pool lacks\. Table[3](https://arxiv.org/html/2607.20946#S5.T3)reports its composition\. TheUUstratum gives the first base\-rate estimate for these constructs:12\.9%12\.9\\%of unconditionally sampled sentences carry at least one construct at consensus, ranging from9\.5%9\.5\\%in Dutch to15\.5%15\.5\\%in English\. This prevalence is far below the roughly one\-in\-two hit rate among lexicon\-positive items \(Table[1](https://arxiv.org/html/2607.20946#S5.T1)\)\.

Table 3:Benchmark composition\.\|U\|/\|P\|/\|N\|\|U\|/\|P\|/\|N\|are labeled items per stratum;ctxis the counterbalanced subset with independent target\-only and full\-context ratings;prevUis the unconditional prevalence of any construct inUU\. French uses the unvalidated seed lexicon\.
### 5\.3Post context improves construct measurement

Table[4](https://arxiv.org/html/2607.20946#S5.T4)reports the primary system comparison, and two results hold across strata and both encoders\. First, post context helps\. The full\-post mDeBERTa improves average precision over its matched target\-only twin on every stratum, from0\.2890\.289to0\.3560\.356onUU\(AUROC0\.702→0\.7410\.702\\\!\\to\\\!0\.741\) and, in relative terms most sharply, from0\.1420\.142to0\.1980\.198on the lexicon\-negative regionNNwhere the anchor is near\-chance—a39%39\\%relative gain exactly where term matching fails\. On bothUUandNNthe full\-post and target\-only95%95\\%intervals do not overlap \(Table[4](https://arxiv.org/html/2607.20946#S5.T4)\)\. Gains on the lexicon\-positive stratum are small \(0\.623→0\.6410\.623\\\!\\to\\\!0\.641\), since retrieved items are already easy, and XLM\-R shows the same pattern \(0\.275→0\.3380\.275\\\!\\to\\\!0\.338onUU\)\. Second, learned context beats the lexicon by a wide margin: the stance modifiermmbarely moves the anchor \(0\.214→0\.2310\.214\\\!\\to\\\!0\.231onUU\), whereas the full\-post encoder roughly doubles its average precision\. Supervision compounds the gains—a fine\-tuned Qwen3 \(0\.3120\.312\) clears its zero\-shot prompt \(0\.2480\.248\), and a human\-supervised encoder leads every stratum \(0\.4010\.401onUU,0\.2450\.245onNN\), marking the headroom still above distillation\. Table[5](https://arxiv.org/html/2607.20946#S5.T5)preserves this ordering onUUin rank correlation and calibration: full\-post mDeBERTa attains the best macro\-Spearman \(0\.4870\.487\) and the lowest calibration error \(0\.0710\.071vs\.0\.1420\.142for the anchor\), so context improves not only ranking but the trustworthiness of the scores\. Population claims useUUonly\.

Table 4:Average precision \(AP, primary\) and macro\-AUROC \(AUC, secondary\) per stratum, macro\-averaged over the construct set evaluable in every compared system and stratum\. Target\-only and full\-post encoders are paired \(shared seeds, supervision, optimization\)\. Subscripts are 95% post\-clustered bootstrap CI half\-widths, paired within seed and item \(§[4\.3](https://arxiv.org/html/2607.20946#S4.SS3)\)\.Table 5:Metrics onUU\. Macro Spearmanρ\\rhowith the0–33ratings, recall at development\-selected precision \(R@P\), and calibration error \(ECE; lower is better\)\. Subscripts are 95% post\-clustered bootstrap CI half\-widths\.
### 5\.4Where context helps: language and failure type

The principal context effect is the paired full\-post−\-target\-only difference on items whose human judgment is context\-sensitive\. Table[6](https://arxiv.org/html/2607.20946#S5.T6)localizes it\. The gain is positive in every language but concentrates where the lexicon is weakest: French \(\+0\.092\+0\.092AP, read with caution as its strata rest on an unvalidated lexicon\) and Italian \(\+0\.078\+0\.078\), whose seed and translated lexicons miss the most, gain most, while English \(\+0\.041\+0\.041\) and Dutch \(\+0\.029\+0\.029\) gain least; calibration improves in tandem, with negativeΔ\\DeltaECE throughout and the largest drop in French \(−0\.035\-0\.035\)\. Every per\-language gain excludes zero, and the French and Italian intervals lie entirely above the English and Dutch ones, so the ordering reflects a real gap rather than noise, even though the two languages within each pair are not separable\. By failure type, context helps most on exactly the phenomena a rule cannot resolve—quotation and reported speech \(\+0\.13\+0\.13\), implicit expression \(\+0\.13\+0\.13\), and cross\-sentence completion \(\+0\.13\+0\.13\), all of which require reading beyond the target sentence—and least on code\-switching \(\+0\.05\+0\.05\), a surface property of the target that the surrounding post does little to disambiguate\. We found that post context supplies precisely the evidence term matching lacks, and it does so where the taxonomy of §[3\.3](https://arxiv.org/html/2607.20946#S3.SS3)predicts it should\.

By languageΔ\\DeltaAPΔ​ρ\\Delta\\rhoΔ\\DeltaECEEnglish\+0\.041±0\.013\+0\.041\_\{\\pm 0\.013\}\+0\.038±0\.015\+0\.038\_\{\\pm 0\.015\}−0\.018±0\.007\-0\.018\_\{\\pm 0\.007\}Italian\+0\.078±0\.018\+0\.078\_\{\\pm 0\.018\}\+0\.065±0\.019\+0\.065\_\{\\pm 0\.019\}−0\.029±0\.010\-0\.029\_\{\\pm 0\.010\}French\+0\.092±0\.016\+0\.092\_\{\\pm 0\.016\}\+0\.081±0\.017\+0\.081\_\{\\pm 0\.017\}−0\.035±0\.011\-0\.035\_\{\\pm 0\.011\}German\+0\.055±0\.014\+0\.055\_\{\\pm 0\.014\}\+0\.049±0\.016\+0\.049\_\{\\pm 0\.016\}−0\.022±0\.008\-0\.022\_\{\\pm 0\.008\}Dutch\+0\.029±0\.012\+0\.029\_\{\\pm 0\.012\}\+0\.025±0\.014\+0\.025\_\{\\pm 0\.014\}−0\.012±0\.006\-0\.012\_\{\\pm 0\.006\}By failure categorytgtfullΔ\\DeltaQuotation / reported0\.210\.34\+0\.13±0\.027\+0\.13\_\{\\pm 0\.027\}Negation0\.180\.29\+0\.11±0\.025\+0\.11\_\{\\pm 0\.025\}Condemnation0\.250\.31\+0\.06±0\.019\+0\.06\_\{\\pm 0\.019\}Implicit expression0\.150\.28\+0\.13±0\.029\+0\.13\_\{\\pm 0\.029\}Cross\-sentence completion0\.220\.35\+0\.13±0\.022\+0\.13\_\{\\pm 0\.022\}Code\-switching0\.190\.24\+0\.05±0\.023\+0\.05\_\{\\pm 0\.023\}

Table 6:The value of context \(mDeBERTa\)\.Top: paired full\-post−\-target\-only difference on context\-sensitive items, by language \(Δ\\DeltaAP,Δ\\Deltamacro\-Spearman,Δ\\DeltaECE; positiveΔ\\DeltaAP/Δ​ρ\\Delta\\rhofavors context\)\. Bottom: target\-only and full\-post performance by lexical failure category, with their difference \(±\\pmvalues are 95% post\-clustered bootstrap CI half\-widths\)\.

## 6Discussion

##### Evaluation construction is part of validity\.

Our stratum analysis changes how a pooled lexicon score should be read\. A lexicon\-positive pool is useful for studying precision and contextual over\-firing, but it cannot by itself measure recall\. A lexicon\-negative pool is useful for finding missed expressions, but its chance\-level anchor score is induced by selection\. Neither is a substitute for unconditional sampling\. Any pooled number overPPandNNanswers a question about the researcher’s chosen mixture rather than the source population, which is why we report per stratum and draw population claims only fromUU\.

##### Coverage and context are different claims\.

The 228 consensus\-marked lexicon\-negative items establish a coverage question rather than a context result\. Some may use implicit or language\-specific expressions; others may reflect slight or ambiguous consensus judgments or incorrect language assignment\. Demonstrating context sensitivity requires paired evidence: a human label that changes when the marked post is shown, plus a full\-post model that moves in the supported direction relative to an otherwise matched target\-only model\. The confirmatory benchmark supplies exactly that paired test, and it comes out positive: full\-post models recover a substantial part of the lexicon\-negative region \(average precision0\.142→0\.1980\.142\\\!\\to\\\!0\.198\) and improve most on quoted, implicit, and cross\-sentence cases \(§[5\.4](https://arxiv.org/html/2607.20946#S5.SS4)\)\. The region the anchor misses is therefore partly recoverable signal, not merely annotation noise\.

##### Multilinguality is measurement, not just transfer\.

A shared encoder can produce outputs in several languages, but multilingual construct validity does not follow from parameter sharing\. Translation quality, language variety, cultural expression, and category reliability can all change the meaning of a score\. Per\-language agreement, error categories, and operating points are therefore primary results\. Consistent with this, the largest context gains fall in French and Italian \(\+0\.092\+0\.092and\+0\.078\+0\.078AP\), whose lexicons are weakest, and the smallest in English\.

The same fact makes French a partly circular basis for the cross\-language ranking\. Because the FrenchPPandNNstrata are themselves drawn with that unvalidated lexicon, their boundaries are noisier than in the validated languages: true positives the seed list misses fall intoNN, where the target\-only model then scores low and the full\-post model recovers them, so part of the French gain reflects stratum misassignment rather than context sensitivity per se\. We therefore read French as an illustrative estimate of what context can recover when the lexicon is absent, and rest the “weaker lexicon, larger gain” claim on the validated resources alone, where it still holds: Italian’s gain exceeds both English’s and Dutch’s with non\-overlapping intervals \(§[5\.4](https://arxiv.org/html/2607.20946#S5.SS4)\)\.

##### The model should serve the evaluation question\.

The central empirical question is not whether a new weighting term beats a dictionary, but whether a dictionary\-based conclusion changes under non\-circular sampling and discourse context\. It does\. The human\-supervised encoder leads every stratum, so distillation is a cost\-saving route to a contextual instrument with clear headroom above it, not a ceiling; the fine\-tuned teacher in turn beats its zero\-shot prompt, so supervision, not scale alone, carries the gain\.

## 7Conclusion

Lexicons remain valuable as transparent retrieval and measurement tools, but their scores are not context\-aware judgments and their outputs must not define the only cases on which they are validated\. Our analysis shows that an existing five\-language pool previously described as partly random is instead split into lexicon\-positive and lexicon\-negative score regions\. Three\-annotator consensus still marks construct evidence in the lexicon\-negative region\. Building on this, we construct and evaluate a non\-circular benchmark that combines unconditional sampling, diagnostic strata, replicated contextual annotation, and matched multilingual models\. On it, post context most improves measurement exactly where the lexicon is silent and on quoted, implicit, and cross\-sentence grievance, while a human\-supervised encoder sets the ceiling and distillation approaches it at lower cost\. The central lesson is simple: a lexicon cannot validate itself on the examples it selected, and grievance cannot be reliably measured without knowing who said what, in which context, and in which language\.

## Limitations

Our claims rest on two evidence sources with different limits\. The existing target\-only pool that motivates the analysis \(§[5\.1](https://arxiv.org/html/2607.20946#S5.SS1)\) has no unconditional\-random stratum, so it cannot estimate natural prevalence; its “negative” half was selected below an anchor threshold, so the anchor’s near\-constant score there is an expected consequence of construction; and its sheets showed target sentences only, with each row the final consensus of three annotators, leaving context effects unidentified and pre\-consensus inter\-rater agreement unrecoverable\. Slight or ambiguous ratings dominate it, and only 117 items carry any construct rating of at least 2\. The confirmatory benchmark was built to remove these limits—adding the unconditional\-random stratum, full marked posts, and replicated target\-only and full\-context ratings—and is the basis for our prevalence and context claims\.

The source corpus is public\-platform data but not a representative sample of any language community\. Text\-only models omit images, links, interaction history, and offline context\. Finally, grievance is common, subjective, and at most a weak signal: neither a dictionary hit nor a model score is evidence of intent, dangerousness, or future violence\.

## References

- Inter\-coder agreement for computational linguistics\.Computational Linguistics\.Cited by:[§2](https://arxiv.org/html/2607.20946#S2.SS0.SSS0.Px3.p1.1)\.
- V\. Basile, C\. Bosco, E\. Fersini, D\. Nozza, V\. Patti, F\. M\. Rangel Pardo, P\. Rosso, and M\. Sanguinetti \(2019\)SemEval\-2019 task 5: multilingual detection of hate speech against immigrants and women in twitter\.InSemEval,Cited by:[§2](https://arxiv.org/html/2607.20946#S2.SS0.SSS0.Px4.p1.1)\.
- A\. Bogle \(2026\)More of the christchurch shooter’s online comments have been uncovered, new zealand researchers say\. does it change the picture?\.Note:The GuardianExternal Links:[Link](https://www.theguardian.com/world/2026/jun/13/christchurch-shooting-massacre-australian-terrorist-online-comments-uncovered-ntwnfb)Cited by:[§1](https://arxiv.org/html/2607.20946#S1.p1.1)\.
- J\. Brynielsson, A\. Horndahl, F\. Johansson, L\. Kaati, C\. Mårtenson, and P\. Svenson \(2013\)Harvesting and analysis of weak signals for detecting lone wolf terrorists\.Security Informatics\.Cited by:[§1](https://arxiv.org/html/2607.20946#S1.p1.1),[§2](https://arxiv.org/html/2607.20946#S2.SS0.SSS0.Px1.p1.1)\.
- K\. Cohen, F\. Johansson, L\. Kaati, and J\. C\. Mork \(2014\)Detecting linguistic markers for radical violence in social media\.Terrorism and Political Violence\.Cited by:[§1](https://arxiv.org/html/2607.20946#S1.p1.1),[§2](https://arxiv.org/html/2607.20946#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Conneau, K\. Khandelwal, N\. Goyal, V\. Chaudhary, G\. Wenzek, F\. Guzmán, E\. Grave, M\. Ott, L\. Zettlemoyer, and V\. Stoyanov \(2020\)Unsupervised cross\-lingual representation learning at scale\.InACL,Cited by:[§2](https://arxiv.org/html/2607.20946#S2.SS0.SSS0.Px4.p1.1),[§3\.6](https://arxiv.org/html/2607.20946#S3.SS6.SSS0.Px3.p1.1)\.
- L\. J\. Cronbach and P\. E\. Meehl \(1955\)Construct validity in psychological tests\.Psychological Bulletin\.Cited by:[§2](https://arxiv.org/html/2607.20946#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova \(2019\)BERT: pre\-training of deep bidirectional transformers for language understanding\.InNAACL\-HLT,Cited by:[§2](https://arxiv.org/html/2607.20946#S2.SS0.SSS0.Px4.p1.1)\.
- B\. Ding, C\. Qin, L\. Liu, Y\. K\. Chia, B\. Li, S\. Joty, and L\. Bing \(2023\)Is GPT\-3 a good data annotator?\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(ACL\),Cited by:[§2](https://arxiv.org/html/2607.20946#S2.SS0.SSS0.Px5.p1.1)\.
- F\. Gilardi, M\. Alizadeh, and M\. Kubli \(2023\)ChatGPT outperforms crowd workers for text\-annotation tasks\.Proceedings of the National Academy of Sciences120\(30\)\.Cited by:[§2](https://arxiv.org/html/2607.20946#S2.SS0.SSS0.Px5.p1.1)\.
- P\. Gill, J\. Horgan, and P\. Deckert \(2014\)Bombing alone: tracing the motivations and antecedent behaviors of lone\-actor terrorists\.Journal of Forensic Sciences\.Cited by:[§1](https://arxiv.org/html/2607.20946#S1.p1.1),[§1](https://arxiv.org/html/2607.20946#S1.p3.1),[§2](https://arxiv.org/html/2607.20946#S2.SS0.SSS0.Px1.p1.1)\.
- C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. Weinberger \(2017\)On calibration of modern neural networks\.InProceedings of the 34th International Conference on Machine Learning \(ICML\),Cited by:[§4\.3](https://arxiv.org/html/2607.20946#S4.SS3.p1.4)\.
- P\. He, J\. Gao, and W\. Chen \(2023\)DeBERTaV3: improving deberta using electra\-style pre\-training with gradient\-disentangled embedding sharing\.InICLR,Cited by:[§2](https://arxiv.org/html/2607.20946#S2.SS0.SSS0.Px4.p1.1),[§3\.6](https://arxiv.org/html/2607.20946#S3.SS6.SSS0.Px2.p1.1)\.
- G\. Hinton, O\. Vinyals, and J\. Dean \(2015\)Distilling the knowledge in a neural network\.arXiv preprint arXiv:1503\.02531\.Cited by:[§2](https://arxiv.org/html/2607.20946#S2.SS0.SSS0.Px5.p1.1)\.
- D\. Hovy and S\. Prabhumoye \(2021\)Five sources of bias in natural language processing\.Language and Linguistics Compass\.Cited by:[§2](https://arxiv.org/html/2607.20946#S2.SS0.SSS0.Px2.p1.1)\.
- C\. Hsieh, C\. Li, C\. Yeh, H\. Nakhost, Y\. Fujii, A\. Ratner, R\. Krishna, C\. Lee, and T\. Pfister \(2023\)Distilling step\-by\-step\! outperforming larger language models with less training data and smaller model sizes\.InFindings of the Association for Computational Linguistics \(ACL\),Cited by:[§2](https://arxiv.org/html/2607.20946#S2.SS0.SSS0.Px5.p1.1)\.
- J\. Hu, S\. Ruder, A\. Siddhant, G\. Neubig, O\. Firat, and M\. Johnson \(2020\)XTREME: a massively multilingual multi\-task benchmark for evaluating cross\-lingual generalisation\.InICML,Cited by:[§2](https://arxiv.org/html/2607.20946#S2.SS0.SSS0.Px4.p1.1)\.
- B\. Kennedy, X\. Jin, A\. Mostafazadeh Davani, M\. Dehghani, and X\. Ren \(2020\)Contextualizing hate speech classifiers with post\-hoc explanation\.InACL,Cited by:[§2](https://arxiv.org/html/2607.20946#S2.SS0.SSS0.Px2.p1.1)\.
- K\. Krippendorff \(2004\)Content analysis: an introduction to its methodology\.Sage\.Cited by:[§2](https://arxiv.org/html/2607.20946#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Li, S\. Nag, H\. Liu, X\. Tang, S\. M\. Sarwar, L\. Cui, H\. Gu, S\. Wang, Q\. He, and J\. Tang \(2025\)Learning with less: knowledge distillation from large language models via unlabeled data\.InFindings of the Association for Computational Linguistics: NAACL 2025,pp\. 2627–2641\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-naacl.142),[Link](https://aclanthology.org/2025.findings-naacl.142/)Cited by:[§2](https://arxiv.org/html/2607.20946#S2.SS0.SSS0.Px5.p1.1)\.
- P\. Lu, A\. Ghaddar, A\. Rashid, M\. Rezagholizadeh, A\. Ghodsi, and P\. Langlais \(2021\)RW\-KD: sample\-wise loss terms re\-weighting for knowledge distillation\.InFindings of the Association for Computational Linguistics: EMNLP 2021,pp\. 3145–3152\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.findings-emnlp.270),[Link](https://aclanthology.org/2021.findings-emnlp.270/)Cited by:[§2](https://arxiv.org/html/2607.20946#S2.SS0.SSS0.Px5.p1.1)\.
- T\. Mandl, S\. Modha, P\. Majumder, D\. Patel, M\. Dave, C\. Mandlia, and A\. Patel \(2019\)Overview of the hasoc track at fire 2019: hate speech and offensive content identification in indo\-european languages\.InFIRE,Cited by:[§2](https://arxiv.org/html/2607.20946#S2.SS0.SSS0.Px4.p1.1)\.
- J\. R\. Meloy, J\. Hoffmann, A\. Guldimann, and D\. James \(2012\)The role of warning behaviors in threat assessment: an exploration and suggested typology\.Behavioral Sciences & the Law\.Cited by:[§1](https://arxiv.org/html/2607.20946#S1.p1.1),[§1](https://arxiv.org/html/2607.20946#S1.p3.1),[§2](https://arxiv.org/html/2607.20946#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Messick \(1995\)Validity of psychological assessment: validation of inferences from persons’ responses and performances as scientific inquiry into score meaning\.American Psychologist50\(9\),pp\. 741–749\.External Links:[Document](https://dx.doi.org/10.1037/0003-066X.50.9.741)Cited by:[§2](https://arxiv.org/html/2607.20946#S2.SS0.SSS0.Px3.p1.1)\.
- R\. Müller, S\. Kornblith, and G\. Hinton \(2019\)When does label smoothing help?\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§3\.7](https://arxiv.org/html/2607.20946#S3.SS7.p1.1)\.
- M\. P\. Naeini, G\. F\. Cooper, and M\. Hauskrecht \(2015\)Obtaining well calibrated probabilities using bayesian binning\.InProceedings of the AAAI Conference on Artificial Intelligence,Cited by:[§4\.3](https://arxiv.org/html/2607.20946#S4.SS3.p1.4)\.
- E\. W\. Pamungkas and V\. Patti \(2019\)Cross\-domain and cross\-lingual abusive language detection: a hybrid approach with deep learning and a multilingual lexicon\.InACL Student Research Workshop,Cited by:[§2](https://arxiv.org/html/2607.20946#S2.SS0.SSS0.Px4.p1.1)\.
- J\. W\. Pennebaker, R\. L\. Boyd, K\. Jordan, and K\. Blackburn \(2015\)The development and psychometric properties of liwc2015\.Technical reportUniversity of Texas at Austin\.Cited by:[§2](https://arxiv.org/html/2607.20946#S2.SS0.SSS0.Px1.p1.1)\.
- B\. Plank \(2022\)The "problem" of human label variation: on ground truth in data, modeling and evaluation\.InEMNLP,Cited by:[§2](https://arxiv.org/html/2607.20946#S2.SS0.SSS0.Px3.p1.1)\.
- Qwen Team \(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§3\.6](https://arxiv.org/html/2607.20946#S3.SS6.SSS0.Px3.p1.1)\.
- P\. Röttger, B\. Vidgen, D\. Nguyen, Z\. Waseem, H\. Margetts, and J\. Pierrehumbert \(2021\)HateCheck: functional tests for hate speech detection models\.InACL,Cited by:[§2](https://arxiv.org/html/2607.20946#S2.SS0.SSS0.Px2.p1.1)\.
- Royal Commission of Inquiry into the Attack on Christchurch Mosques on 15 March 2019 \(2020\)Planning the terrorist attack\.Note:Ko tō tātou kāinga tēnei: Report of the Royal Commission of Inquiry into the terrorist attack on Christchurch masjidain on 15 March 2019External Links:[Link](https://christchurchattack.royalcommission.nz/the-report/firearms-licensing/planning-the-terrorist-attack)Cited by:[§1](https://arxiv.org/html/2607.20946#S1.p1.1)\.
- V\. Sanh, L\. Debut, J\. Chaumond, and T\. Wolf \(2019\)DistilBERT, a distilled version of bert: smaller, faster, cheaper and lighter\.arXiv preprint arXiv:1910\.01108\.Cited by:[§2](https://arxiv.org/html/2607.20946#S2.SS0.SSS0.Px5.p1.1)\.
- H\. Song, M\. Kim, D\. Park, Y\. Shin, and J\. Lee \(2022\)Learning from noisy labels with deep neural networks: a survey\.IEEE Transactions on Neural Networks and Learning Systems\.Cited by:[§2](https://arxiv.org/html/2607.20946#S2.SS0.SSS0.Px5.p1.1)\.
- Y\. R\. Tausczik and J\. W\. Pennebaker \(2010\)The psychological meaning of words: liwc and computerized text analysis methods\.Journal of Language and Social Psychology\.Cited by:[§2](https://arxiv.org/html/2607.20946#S2.SS0.SSS0.Px1.p1.1)\.
- A\. N\. Uma, T\. Fornaciari, D\. Hovy, S\. Paun, B\. Plank, and M\. Poesio \(2021\)Learning from disagreement: a survey\.Journal of Artificial Intelligence Research\.Cited by:[§2](https://arxiv.org/html/2607.20946#S2.SS0.SSS0.Px3.p1.1)\.
- I\. van der Vegt, B\. Kleinberg, M\. Miotto, and J\. Festor \(2025\)Translating the grievance dictionary: a psychometric evaluation of dutch, german, and italian versions\.arXiv preprint arXiv:2505\.07495\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2505.07495),[Link](https://arxiv.org/abs/2505.07495)Cited by:[§1](https://arxiv.org/html/2607.20946#S1.p3.1),[§2](https://arxiv.org/html/2607.20946#S2.SS0.SSS0.Px1.p1.1),[§3\.2](https://arxiv.org/html/2607.20946#S3.SS2.p2.1)\.
- I\. van der Vegt, M\. Mozes, B\. Kleinberg, and P\. Gill \(2021\)The grievance dictionary: understanding threatening language use\.Behavior Research Methods\.Cited by:[§B\.3](https://arxiv.org/html/2607.20946#A2.SS3.p1.1),[§1](https://arxiv.org/html/2607.20946#S1.p3.1),[§2](https://arxiv.org/html/2607.20946#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Wang, Y\. Liu, Y\. Xu, C\. Zhu, and M\. Zeng \(2021\)Want to reduce labeling cost? GPT\-3 can help\.InFindings of the Association for Computational Linguistics \(EMNLP\),Cited by:[§2](https://arxiv.org/html/2607.20946#S2.SS0.SSS0.Px5.p1.1)\.
- X\. Wang, J\. Wei, D\. Schuurmans, Q\. Le, E\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou \(2023\)Self\-consistency improves chain of thought reasoning in language models\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§3\.7](https://arxiv.org/html/2607.20946#S3.SS7.p1.1)\.
- C\. Wilson, E\. Renner, J\. Smylie, and M\. Dziwulski \(2024\)Christchurch gunman discussed attacks online a year before carrying them out\.Note:The Conversation; republished by The GuardianExternal Links:[Link](https://www.theguardian.com/world/commentisfree/2024/feb/21/christchurch-gunman-terrorist-discussed-attacks-online-4chan)Cited by:[§1](https://arxiv.org/html/2607.20946#S1.p1.1)\.
- M\. Zampieri, S\. Malmasi, P\. Nakov, S\. Rosenthal, N\. Farra, and R\. Kumar \(2019\)Predicting the type and target of offensive posts in social media\.InNAACL\-HLT,Cited by:[§2](https://arxiv.org/html/2607.20946#S2.SS0.SSS0.Px4.p1.1)\.

## Appendix AReproducibility and Availability

Our code and benchmark are released at[https://github\.com/behavioral\-ds/multilingual\_grievance](https://github.com/behavioral-ds/multilingual_grievance)\. The repository includes scripts for lexical scoring, annotation\-pool construction, human\-label aggregation, post\-grouped splitting, multilingual student training, and evaluation\. The analysis in Table[1](https://arxiv.org/html/2607.20946#S5.T1)joins the consensus labels to the five per\-language lexical\-score files bysentence\_id\. We define construct presence as ordinal score≥1\\geq 1and moderate presence as score≥2\\geq 2\. The fixed 16\-construct comparison keeps only constructs with both classes in bothPPandNN\. The released target\-only annotation files store one final consensus row per item after review by three annotators\.

The benchmark release includes the immutable corpus manifest—source snapshot, timestamp filter, language classifier and version, normalization, sentence segmentation, duplicate clusters, sampling probabilities, and post\-level partitions\. Each model run records the exact model revision, prompt and rubric version, input format, token truncation, random seed, hyperparameters, hardware, runtime, and monetary cost\. Evaluation outputs include item\-level predictions, so post\-clustered intervals and paired system contrasts can be recomputed\.

## Appendix BAnnotation Guidelines and Composition

### B\.1Existing target\-only round

Table[7](https://arxiv.org/html/2607.20946#A2.T7)describes the existing 2,000\-item pool\. Three annotators reviewed each target and agreed the final stored label; every item received all 22 ordinal construct scores, and the sheets omitted full\-post context\. “Any construct” includes slight or ambiguous evidence \(score≥1\\geq 1\)\. The overall binary field has only one positive and is not used as an evaluation target\.

Table 7:Existing target\-only rating pool\. Each stored row is the final three\-annotator consensus\.
### B\.2Contextual round

Each annotator receives a target sentence highlighted inside its source post, plus the language and no model or lexicon score\. For every construct, the rater assigns 0 \(absent\), 1 \(slight or ambiguous\), 2 \(moderate\), or 3 \(strong\), and records stance, target, and a context\-failure tag\. URLs are not opened and images are out of scope; dependence on missing media is flagged\. Quoted, reported, hypothetical, and condemned content is attributed to the author only when the surrounding post endorses it\.

### B\.3Construct criteria

The names in Table[8](https://arxiv.org/html/2607.20946#A2.T8)come from the Grievance Dictionary\(van der Vegtet al\.,[2021](https://arxiv.org/html/2607.20946#bib.bib1)\); the criteria are our semantic operationalizations and underwent independent content\-validity review\.

Table 8:Contextual criteria for the 22 constructs, scored for the author’s expression through the target sentence given its source post\.

## Appendix CUncertainty\-Weighted Distillation

For teacher targetμi​c\\mu\_\{ic\}, student logitzi​cz\_\{ic\}, temperatureTT, and pseudo\-labelhi​c=𝟏​\[μi​c≥0\.5\]h\_\{ic\}=\\mathbf\{1\}\[\\mu\_\{ic\}\\geq 0\.5\], define

ℓi​csoft\\displaystyle\\ell^\{\\mathrm\{soft\}\}\_\{ic\}=BCE​\(σ​\(zi​c/T\),σ​\(logit​\(μi​c\)/T\)\)​T2,\\displaystyle=\\mathrm\{BCE\}\\\!\\left\(\\sigma\(z\_\{ic\}/T\),\\sigma\(\\mathrm\{logit\}\(\\mu\_\{ic\}\)/T\)\\right\)T^\{2\},ℓi​chard\\displaystyle\\ell^\{\\mathrm\{hard\}\}\_\{ic\}=BCE​\(σ​\(zi​c\),hi​c\)\.\\displaystyle=\\mathrm\{BCE\}\(\\sigma\(z\_\{ic\}\),h\_\{ic\}\)\.The legacy loss was

ℒ=1B​\|𝒞\|​∑i,cw~i​c​\[λ​ℓi​csoft\+\(1−λ\)​ℓi​chard\]\.\\mathcal\{L\}=\\frac\{1\}\{B\|\\mathcal\{C\}\|\}\\sum\_\{i,c\}\\tilde\{w\}\_\{ic\}\\left\[\\lambda\\ell^\{\\mathrm\{soft\}\}\_\{ic\}\+\(1\-\\lambda\)\\ell^\{\\mathrm\{hard\}\}\_\{ic\}\\right\]\.Given uncertaintyui​cu\_\{ic\}, the raw weight waswi​c=\(ui​c2\+ν\)−βw\_\{ic\}=\(u\_\{ic\}^\{2\}\+\\nu\)^\{\-\\beta\}, normalized over the training set and clipped to\[0\.1,10\]\[0\.1,10\]\. Thusβ=0\\beta=0recovers standard distillation\. Teacher labels were generated once \(S=1S=1\), so per\-construct self\-consistency variance is zero; the real\-data sweep instead broadcasts self\-reported confidence over all constructs\. This weak proxy is high and tightly clustered \(mean confidence 0\.92\)\.

Table 9:Evaluation on 1,545 three\-annotator consensus items after removing known training\-post overlap\.