Deconstructing Stereotypes: Scope-Conditioned Generation for Effective Multilingual Counterspeech
Summary
The paper proposes a novel scope-conditioned generation framework that integrates structured stereotype characteristics into Large Language Model prompts for effective multilingual counterspeech, validated on a human-curated dataset with significant improvements in factuality and effectiveness.
arXiv:2609.16906v1 Announce Type: new
Abstract: Counterspeech (CS) - direct responses that counter online Hate Speech (HS) using reasoning and alternative viewpoints - has emerged as an alternative to content removal. Current automatic CS generation methods, however, frequently produce generic, ineffective replies that fail to target the implicit stereotypes behind HS. To bridge this gap, we propose a novel scope-conditioned generation framework that explicitly integrates structured stereotype characteristics into Large Language Models prompts. We validate our approach on a novel, human-curated dataset annotated in English, Italian, and Spanish. Extensive evaluations show that stereotype-conditioned prompting substantially outperforms generic baselines across all three languages, obtaining significant gains in factuality, specificity, cogency, and effectiveness for both explicit and implicit implied stereotypes.
View Cached Full Text
Cached at: 09/16/26, 08:55 AM
# Scope-Conditioned Generation for Effective Multilingual Counterspeech Source: [https://arxiv.org/html/2609.16906](https://arxiv.org/html/2609.16906) ## Deconstructing Stereotypes: Scope\-Conditioned Generation for Effective Multilingual Counterspeech Elias Urios AlacreuAffiliation:PRHLT Research Center, Universitat Politècnica de València, Valencia, SpainElena CabrioAffiliation:Université Côte d’Azur, CNRS, INRIA, I3S, FrancePaolo RossoAffiliation:PRHLT Research Center, Universitat Politècnica de València, Valencia, SpainAffiliation:ValgrAI Valencian Graduate School and Research Network of Artificial Intelligence, SpainCorrespondence:[greta\.damo@univ\-cotedazur\.fr](mailto:[email protected])Serena VillataAffiliation:Université Côte d’Azur, CNRS, INRIA, I3S, France ###### Abstract Counterspeech \(CS\) – direct responses that counter online Hate Speech \(HS\) using reasoning and alternative viewpoints – has emerged as an alternative to content removal\. Current automatic CS generation methods, however, frequently produce generic, ineffective replies that fail to target the implicit stereotypes behind HS\. To bridge this gap, we propose a novel scope\-conditioned generation framework that explicitly integrates structured stereotype characteristics into Large Language Models prompts\. We validate our approach on a novel, human\-curated dataset annotated in English, Italian, and Spanish\. Extensive evaluations show that stereotype\-conditioned prompting substantially outperforms generic baselines across all three languages, obtaining significant gains in factuality, specificity, cogency, and effectiveness for both explicit and implicit implied stereotypes\. Content warning: this paper contains HS examples some readers may find offensive\. ## 1Introduction Hate Speech \(HS\) can be broadly defined as any communication that disparages an individual or group based on protected characteristics such as race, ethnicity, gender, sexual orientation, religion, or nationality\([Nockleby, 2000](https://arxiv.org/html/2609.16906#bib.bib46)\)\. Driven by the rapid expansion of social media platforms, HS has evolved from a marginal phenomenon into a pervasive one\. The Natural Language Processing \(NLP\) community has responded to this challenge primarily through detection techniques\([Fortuna and Nunes, 2018](https://arxiv.org/html/2609.16906#bib.bib72);[Rawat et al\., 2024](https://arxiv.org/html/2609.16906#bib.bib75)\), flagging and removing hateful content at scale\. While detection offers an essential first line of defense, it addresses only the surface manifestation of the problem\. Content moderation removes the message but leaves the underlying biases untouched, allowing hateful narratives to resurface across platforms or through other users\. Consequently, a growing body of work focuses onCounterspeech\(CS\) generation, aiming to directly refute, de\-escalate, or reframe hateful content through constructive dialogue\([Benesch, 2014](https://arxiv.org/html/2609.16906#bib.bib30);[Bonaldi et al\., 2024a](https://arxiv.org/html/2609.16906#bib.bib76)\)\. A foundational yet often under\-explored driver of HS is its grounding instereotypes: deeply ingrained cognitive constructs that skew social perception, reinforce inequalities, and justify discriminatory behavior\([Augoustinos and Walker, 1998](https://arxiv.org/html/2609.16906#bib.bib44)\)\. Repeated exposure to stereotypes normalizes bias, escalating over time into overt prejudice and fueling HS\([Cignarella et al\., 2025](https://arxiv.org/html/2609.16906#bib.bib45)\)\. This connection is core to how HS works, in fact hateful messages persuade by relying on false beliefs the audience already accepts\. Effective CS must therefore target not only the surface text of a message, but also the underlying stereotype on which it rests\. However, stereotypes present differently across HS: while some messages state the stereotype explicitly \(e\.g\., claiming“Women have it so easy; they stay home and have babies”\), others invoke it implicitly through hostility or tropes without naming the underlying premise directly \(e\.g\., wishing harm upon a group via“I hope all LGBT people d\*e of AIDS”, indirectly implying that all group members carry the disease\)\. To operationalize stereotypes for computational modelling, prior work in NLP and CS frequently relies on exposing the unstated premise behind a message\. This underlying stereotype is usually formalized as anImplied Statement\(IS\), which is a simplified, canonical proposition, structured as⟨target group⟩\+relation\+⟨attributed quality⟩\\langle\\textit\{target group\}\\rangle\+\\text\{relation\}\+\\langle\\textit\{attributed quality\}\\rangle\([Sap et al\., 2020](https://arxiv.org/html/2609.16906#bib.bib9);[Akazawa et al\., 2023](https://arxiv.org/html/2609.16906#bib.bib43)\)\. While the IS formulation makes the implicit belief explicit, thereby aiding generation systems in recognizing the true intent of a hateful post, it relies on a simplified abstraction that discards critical structural nuances\. Specifically, it ignores:\(i\)thescopeof the generalization \(e\.g\., whether a claim targets an essentialist trait of an entire demographic vs\. a specific subset\); and\(ii\)theattributed trait typequality being assigned \(e\.g\., whether the claim implies moral corruption vs\. biological inferiority\)\.By ignoring these dimensions, existing CS systems risk generating generic or misaligned messages that fail to address the HS core intent\. To bridge this gap, we introduce a novel, multilingual dataset, annotated across English \(EN\), Spanish \(ES\), and Italian \(IT\) that describes stereotypes using a fine\-grained annotation schema beyond standard IS, incorporating all these dimensions\. Using this dataset, our work investigates the following Research Questions: - •RQ1:Can stereotype\-conditioned generation improve CS quality over generic baselines and IS\-only generation? - •RQ2:Can stereotype\-conditioned generation remain beneficial for implicit stereotypes, given their higher intrinsic difficulty compared to explicit cases? - •RQ3:Does stereotype\-structured generation transfer cross\-lingually? Our main contributions are threefold:1\)We present afine\-grained stereotype annotation schemathat decomposes HS into its implicit/explicit stereotype, generalization scope, and attributed trait type\.2\)We release a high\-quality, human\-curateddataset across three languages\(EN,ES,IT\) bridging HS with stereotype modelling, and CS generation111A data sample and the guidelines are at[https://anonymous\.4open\.science/r/stereotypecs/README\.md](https://anonymous.4open.science/r/stereotypecs/README.md)\.3\)We demonstrate empirically through automatic, LLM\-as\-a\-Judge and human evaluations thatconditioning LLM\-based CS generation on stereotype structure significantly enhancesspecificity, factuality, and cogency across all three languages and across both explicit and implicit stereotypes\. ## 2Related Work Stereotypes in Language\.The study of stereotypes in NLP is a relatively new field, and it is closely connected to HS and bias\. In fact, HS often conveys harm through implicit stereotypes expressed via generics and pragmatic implicatures rather than explicit insults[Fiske \(1998\)](https://arxiv.org/html/2609.16906#bib.bib59);[Leslie \(2014\)](https://arxiv.org/html/2609.16906#bib.bib58);[Sap et al\. \(2020\)](https://arxiv.org/html/2609.16906#bib.bib9)\. Because these stereotypical beliefs are resistant to change, effective interventions should target the underlying inferences instead of only the surface\-level hateful content[Lepoutre \(2019\)](https://arxiv.org/html/2609.16906#bib.bib66);[Perez Gomez \(2021\)](https://arxiv.org/html/2609.16906#bib.bib67)\. Psychology research has shown that stereotype reduction can be achieved through mechanisms such as exposure to counter\-stereotypical examples, perspective\-taking, and promoting egalitarian norms, though their effectiveness varies across stereotypes and social groups[Dasgupta and Greenwald \(2001\)](https://arxiv.org/html/2609.16906#bib.bib61);[Todd et al\. \(2011\)](https://arxiv.org/html/2609.16906#bib.bib62);[Wyer \(2010\)](https://arxiv.org/html/2609.16906#bib.bib63);[Forscher et al\. \(2019\)](https://arxiv.org/html/2609.16906#bib.bib64);[FitzGerald et al\. \(2019\)](https://arxiv.org/html/2609.16906#bib.bib65)\. Stereotype Detection in NLP\.Most NLP research on stereotypes focuses on detection, as a supervised classification task\. Transformer\-based models consistently outperform traditional feature\-based approaches across multiple languages and benchmarks[Sanguinetti et al\. \(2020\)](https://arxiv.org/html/2609.16906#bib.bib15);[Sánchez\-Junquera et al\. \(2021\)](https://arxiv.org/html/2609.16906#bib.bib41);[Pujari et al\. \(2022\)](https://arxiv.org/html/2609.16906#bib.bib14);[Bosco et al\. \(2023\)](https://arxiv.org/html/2609.16906#bib.bib10);[Vargas et al\. \(2023\)](https://arxiv.org/html/2609.16906#bib.bib18);[Cignarella et al\. \(2024\)](https://arxiv.org/html/2609.16906#bib.bib11);[Schmeisser\-Nieto et al\. \(2025\)](https://arxiv.org/html/2609.16906#bib.bib17);[Urios\-Alacreu and Rosso \(2025\)](https://arxiv.org/html/2609.16906#bib.bib35)\. Related work also examines stereotypes encoded in language models and developed methods for measuring and mitigating representational and stereotypical bias[Bolukbasi et al\. \(2016\)](https://arxiv.org/html/2609.16906#bib.bib21);[Caliskan et al\. \(2017\)](https://arxiv.org/html/2609.16906#bib.bib22);[Zmigrod et al\. \(2019\)](https://arxiv.org/html/2609.16906#bib.bib19);[Sun et al\. \(2019\)](https://arxiv.org/html/2609.16906#bib.bib20);[Laiyk et al\. \(2026\)](https://arxiv.org/html/2609.16906#bib.bib71)\. While stereotype detection has advanced considerably, comparatively little work has explored methods for mitigating or countering stereotypes\. Stereotype and CS Datasets\.Several datasets have been developed for stereotype detection and bias evaluation, including CrowS\-Pairs[Nangia et al\. \(2020\)](https://arxiv.org/html/2609.16906#bib.bib39), StereoSet[Nadeem et al\. \(2021\)](https://arxiv.org/html/2609.16906#bib.bib12), BBQ[Parrish et al\. \(2022\)](https://arxiv.org/html/2609.16906#bib.bib40), HONEST[Nozza et al\. \(2021\)](https://arxiv.org/html/2609.16906#bib.bib13), and multilingual resources such as StereoHOAX[Schmeisser\-Nieto et al\. \(2025\)](https://arxiv.org/html/2609.16906#bib.bib17)and QUEEREOTYPES[Cignarella et al\. \(2024\)](https://arxiv.org/html/2609.16906#bib.bib11)\. Although these datasets cover diverse bias dimensions, they remain predominantly English\-centric, with relatively limited multilingual and low\-resource language coverage\. Additionally, several datasets have been created to support CS analysis and generation\. These include CS collected from social media[Mathew et al\. \(2019\)](https://arxiv.org/html/2609.16906#bib.bib33);[Garland et al\. \(2020\)](https://arxiv.org/html/2609.16906#bib.bib34)as well as expert\-curated datasets such as CONAN[Chung et al\. \(2019\)](https://arxiv.org/html/2609.16906#bib.bib5)and its extensions[Fanton et al\. \(2021\)](https://arxiv.org/html/2609.16906#bib.bib4);[Bonaldi et al\. \(2022\)](https://arxiv.org/html/2609.16906#bib.bib3);[Bonaldi et al\. \(2025\)](https://arxiv.org/html/2609.16906#bib.bib1)\.While these resources have enabled automatic CS generation, they rarely annotate the underlying stereotypes conveyed by it\. CS and Counter\-stereotype Generation\.CS aims to challenge harmful content through strategies such as empathy, fact\-checking, humor, denunciation, and warning of consequences[Benesch \(2014\)](https://arxiv.org/html/2609.16906#bib.bib30)\. Building on these, NLP research has explored automatic generation using human\-written responses[Qian et al\. \(2019\)](https://arxiv.org/html/2609.16906#bib.bib29), diversity\- and relevance\-oriented decoding[Zhu and Bhat \(2021\)](https://arxiv.org/html/2609.16906#bib.bib27), controllable generation for response tone[Saha et al\. \(2022\)](https://arxiv.org/html/2609.16906#bib.bib26), few\-shot prompting[Ashida and Komachi \(2022\)](https://arxiv.org/html/2609.16906#bib.bib28), argumentative components[Bonaldi et al\. \(2024b\)](https://arxiv.org/html/2609.16906#bib.bib2), and retrieval\-augmented generation for factually grounded CS[Wilk et al\. \(2025\)](https://arxiv.org/html/2609.16906#bib.bib23);[Jiang et al\. \(2025\)](https://arxiv.org/html/2609.16906#bib.bib24);[Damo et al\. \(2025\)](https://arxiv.org/html/2609.16906#bib.bib25)\. However, these methods primarily address explicit HS and focus on response style, with limited attention to countering the implied stereotypes underlying HS\. Countering stereotypes is closely related to CS but focuses on challenging subtler forms of harmful language, where content removal is often inappropriate\. Instead, responses aim to educate the speaker, challenge stereotypical beliefs, and signal to bystanders that such statements should not go unchallenged\. However, research on directly countering stereotypes is comparatively limited\. Psychological studies show that counter\-stereotypical examples, humanization, and factual correction can effectively challenge stereotypical beliefs[Finnegan et al\. \(2015\)](https://arxiv.org/html/2609.16906#bib.bib69);[Prati et al\. \(2016\)](https://arxiv.org/html/2609.16906#bib.bib68);[Porter and Wood \(2021\)](https://arxiv.org/html/2609.16906#bib.bib70)\. In NLP, only few works explicitly model or generate responses to implied stereotypes studying different countering strategies and evaluating their effectiveness[Fraser et al\. \(2021\)](https://arxiv.org/html/2609.16906#bib.bib31);[Fraser et al\. \(2023\)](https://arxiv.org/html/2609.16906#bib.bib6);[Mun et al\. \(2023\)](https://arxiv.org/html/2609.16906#bib.bib7);[Allaway et al\. \(2023\)](https://arxiv.org/html/2609.16906#bib.bib8);[Nejadgholi et al\. \(2024\)](https://arxiv.org/html/2609.16906#bib.bib32), leaving robust counter\-stereotype generation unexplored\. Overall, while substantial progress has been made in stereotype detection and CS generation, comparatively little work has explored LLM\-based CS generation that explicitly targets the implied stereotypes underlying hateful messages\. ## 3Data As our study focuses on EN, ES, and IT, we augment the Multilingual\-MTCONAN\-KN dataset[Bonaldi et al\. \(2025\)](https://arxiv.org/html/2609.16906#bib.bib1), which extends MT\-CONAN[Bonaldi et al\. \(2022\)](https://arxiv.org/html/2609.16906#bib.bib3)\. It consists of EN HS/CS pairs targeting multiple protected groups, including Women, People of Color \(POC\), Migrants, Jews, Muslims, and LGBTQ\+ individuals, expanded with human translations into several languages, including IT and ES\. We use the training and development splits, as they include both the HS and gold CS responses written by human annotators, which we use as reference in our evaluations\. Overall, the dataset contains 496 HS/CS pairs for each language, where the ones for ES and IT are human translations from the EN source\. Table 1:IAA for English, Spanish, and Italian datasets\.Table 2:Summary statistics of the annotations\.### 3\.1Annotation dimensions Each HS message is annotated along four dimensions: implicitness of the HS, presence of a stereotype, generalization scope, and attributed trait type\. The last two dimensions are annotated only for messages in which a stereotype has been identified\. Implicitness of HS\.We annotate whether HS is conveyed directly or through inference\. HS is labelledimplicitwhen derogatory intent is expressed indirectly through devices such as presuppositions, insinuation, coded language, or metaphor; it is labelledexplicitwhen conveyed through direct slurs or unambiguous derogatory statements\. This dimension is independent of stereotype presence: a message may contain explicit hate while expressing an implicit group\-level attribute, and vice versa\. Stereotype presence\.HS is annotated as containing a stereotype when a target group is identifiable, an attribute is assigned to it, and the attribute is generalized beyond an individual\. Stereotypes are further labeled asexplicitorimplicitdepending on whether the claim is directly stated or requires inference\. This operationalization follows classical definitions of stereotypes as associations between social categories and fixed characteristics\([Allport, 1954](https://arxiv.org/html/2609.16906#bib.bib54);[Dovidio et al\., 2010](https://arxiv.org/html/2609.16906#bib.bib55)\)and prior annotation schemes\([Schmeisser\-Nieto et al\., 2022](https://arxiv.org/html/2609.16906#bib.bib16)\)\. Generalization scope\.We annotate the strength of stereotype attribution by distinguishinguniversal\(claims holding without exception, e\.g\., via explicit universal quantifiers\),majority\(statistical or generic claims\), andessentialist\(attributes framed as inherent, biological or fixed\) scopes\. This distinction is grounded in linguistics and philosophy research on generalizations\([Leslie, 2008](https://arxiv.org/html/2609.16906#bib.bib49);[Leslie, 2017](https://arxiv.org/html/2609.16906#bib.bib52)\), categorical and overgeneralized prejudice\([Allport, 1954](https://arxiv.org/html/2609.16906#bib.bib54);[Hamilton and Trolier, 1986](https://arxiv.org/html/2609.16906#bib.bib53)\), and psychological essentialism\([Medin and Ortony, 1989](https://arxiv.org/html/2609.16906#bib.bib50);[Haslam et al\., 2000](https://arxiv.org/html/2609.16906#bib.bib56);[Gelman, 2003](https://arxiv.org/html/2609.16906#bib.bib51)\)\. Attributed trait type\.We categorize stereotypes by the domain of characteristics attributed to the target group using six categories:Intellectual and Cognitive Inferiority, Moral and Criminal Threat, Economic and Parasitic Drain, Biological and Physical Discredit, Existential and Cultural Threat, and Cultural Backwardness\. Annotators assign the most representative category, using multiple labels only when equally prominent\. The taxonomy is grounded in the Stereotype Content Model\([Fiske et al\., 2018](https://arxiv.org/html/2609.16906#bib.bib48)\), which models stereotype content along the dimensions of warmth and competence, and the ABC Model, considering Agency, Beliefs and Communion\([Koch et al\., 2016](https://arxiv.org/html/2609.16906#bib.bib47)\)\. ### 3\.2IS extraction The IS is the latent negative stereotype conveyed by a hateful message\. We add it to the dataset by extracting it, building on the methodology of[Akazawa et al\. \(2023\)](https://arxiv.org/html/2609.16906#bib.bib43), with three main differences\. First, to support our multilingual setting, we use the multilingual Transformer Encoder\-Decoder mBART\([Liu et al\., 2020](https://arxiv.org/html/2609.16906#bib.bib42)\), Second, we expand the data by incorporating the dataset of[Bonaldi et al\. \(2024b\)](https://arxiv.org/html/2609.16906#bib.bib2), resulting in≈\\approx37000 samples\. Third, two annotators manually validated all the extracted IS and correct inaccurate or incoherent outputs\. Implementation details, hyperparameters, and results are in Appendix[A](https://arxiv.org/html/2609.16906#A1)\. ### 3\.3Annotation process and IAA results To enrich the dataset with stereotype\-related information, three native speakers222Aged between 24 and 29, with backgrounds in Data Science, Computer Science, and Communication studies\.per language \(EN, ES, IT\) have annotated each HS instance\. To assess annotation reliability, we compute Inter\-Annotator Agreement \(IAA\) using Krippendorff’sα\\alpha\. The annotation proceeded in three stages: we drafted guidelines with definitions and examples; trained the annotators, who then piloted on 30 messages to surface ambiguities, and refined the guidelines based on the pilot IAA before the full annotation Table[1](https://arxiv.org/html/2609.16906#S3.T1)reports IAA across languages, with agreement substantial to good overall\([Krippendorff, 2019](https://arxiv.org/html/2609.16906#bib.bib57)\)\. Generalization scope and trait type333For trait type, we compute Krippendorff’sα\\alphaas a frequency\-weighted macro\-average of per\-label binary alphas, allowing partial credit for overlapping label sets\.achieve the highest IAA, indicating that, once a stereotype is detected, annotators consistently identify both its generalization strength and its attributed characteristic\. By contrast, stereotype presence shows the lowest agreement, reflecting the subjective nature of identifying stereotypes\. Implicit HS exhibits the greatest cross\-lingual variation, with higher agreement in IT, likely because indirect hateful cues are inherently more difficult to identify than explicit ones\. Overall, the results support the reliability of the annotation\. Gold labels are obtained by majority voting\. When no majority exists for multi\-class dimensions, an expert annotator assigns the final one\. Table[2](https://arxiv.org/html/2609.16906#S3.T2)reports summary statistics\. Most messages contain a recoverable stereotype \(476 in EN, 470 in ES, and 453 in IT, out of 496\), confirming that stereotypical generalizations are pervasive in the dataset\. Although the HS messages in the original dataset are human translations of the same EN source, small cross\-lingual differences remain, consistent with the lower agreement observed for stereotype presence\. We also compute IAA on the gold labels across languages, showing consistent levels w\.r\.t\. the individual languages \(Appendix[E](https://arxiv.org/html/2609.16906#A5)\)\. Across languages, both HS and stereotypes are predominantly explicit, except in ES, where implicit HS is more frequent, mirroring its comparatively lower agreement for this dimension\. Majority generalizations are the most common scope, followed by essentialist and universal ones\. Finally, Existential and Cultural Threat is the most frequent trait type in all languages, followed by Biological and Physical Discredit, and Moral and Criminal Threat, whereas Cultural Backwardness is consistently the least frequent, indicating that stereotype content is largely preserved across translations\. ## 4Experimental Setting We generate CS using open\-source LLMs to enhance the reproducibility of our experiments\. To ensure a fair comparison, we select instruction\-tuned models from different model families with a similar parameter size \(7–9B parameters\)\. The instruction\-tuned variants are used to ensure adherence to our prompts, while all models are evaluated using the same generation parameters\. To investigate whether language specialization influences CS quality, we include both multilingual and monolingual models444Models specifications and parameters are in Appendix[B](https://arxiv.org/html/2609.16906#A2)\.for the two low resource languages\. The multilingual models areMinistral\([Liu et al\., 2026](https://arxiv.org/html/2609.16906#bib.bib37)\),Llama[Touvron et al\. \(2023\)](https://arxiv.org/html/2609.16906#bib.bib79), andEuroLLM\([Martins et al\., 2025](https://arxiv.org/html/2609.16906#bib.bib38)\), the latter specifically fine\-tuned for European languages\. The monolingual models areLlamantino[Polignano et al\. \(2026\)](https://arxiv.org/html/2609.16906#bib.bib78)for IT andSalamandra\([Gonzalez\-Agirre et al\., 2025](https://arxiv.org/html/2609.16906#bib.bib36)\)for ES, since both are based on the Llama architecture, they provide a meaningful comparison with the multilingual Llama model while allowing us to assess the impact of language\-specific adaptation\. The selected models are representative of open\-source LLMs commonly adopted in recent CS generation studies\. Each model is evaluated in every language under four experimental conditions for a total of 16 generations for both IT and ES \(4 models×\\times4 conditions\) and 12 generations for EN \(3 models×\\times4 conditions\)\. The experimental settings correspond to different levels of information provided as input: 1. 1\.HS\-only \(Condition A\):the CS is generated considering only the HS message as input\. 2. 2\.HS\+IS \(Condition B\):the CS is generated considering the HS message and the IS\. 3. 3\.HS\+Annotations \(Condition C\):the CS is generated considering the HS message and all available annotations\. 4. 4\.HS\+IS\+Annotations \(Condition D\):the CS is generated considering the HS message, the extracted IS, and all available annotations\. ### 4\.1Metrics Following[Saha et al\. \(2024\)](https://arxiv.org/html/2609.16906#bib.bib80);[Zubiaga et al\. \(2024\)](https://arxiv.org/html/2609.16906#bib.bib74), we conduct automatic, LLM\-based, and human evaluations\. #### Automatic metrics\. We evaluate generated CS along three dimensions: lexical and semantic overlap, argumentative stance, and diversity\. Lexical and semantic overlap are computed against gold reference CS, while stance consistency is evaluated against the original HS\. Concerning the reference CS, for lexical overlap, we report METEOR[Banerjee and Lavie \(2005\)](https://arxiv.org/html/2609.16906#bib.bib73), while for semantic similarity we use BERTScore F1[Zhang et al\. \(2020\)](https://arxiv.org/html/2609.16906#bib.bib60)\. We also measure cosine similarity between sentence embeddings of generated CS and reference HS555sentence\-transformers/paraphrase\-multilingual\-mpnet\-base\-v2\. To assess the argumentative stance of generated CS w\.r\.t\. the HS, we compute Natural Language Inference \(NLI\) scores using a multilingual DeBERTa\-based entailment model666mDeBERTa\-v3\-base\-xnli\-multilingual\-nli\-2mil7, treating HS as premise and CS as hypothesis\. We report the average probability mass assigned to the contradiction and entailment classes, where a higher contradiction score indicates the CS is more explicitly opposing or rebutting the hateful claim, while a higher entailment score suggests the CS agrees with or reinforces it, which is undesirable\. Finally, output diversity is measured with Repetition Rate \(RR\)[Cettolo et al\. \(2014\)](https://arxiv.org/html/2609.16906#bib.bib77), capturing the fraction of exactly duplicated generations in the output set\. Table 3:Automatic evaluation results across languages, models, and generation conditions \(A–D\)\. #### LLM\-as\-a\-judge and Human evaluation\. We assess the qualitative properties of the generated CS using both an LLM\-as\-a\-judge framework and human evaluation\. Both protocols rate each CS on a 5\-point Likert scale along six dimensions:factuality\(the number of specific facts provided\);specificity\(whether the CS directly addresses the topic, target group, and claim of the HS\);effectiveness\(the estimated likelihood of reducing hateful attitudes, challenging underlying beliefs, and persuading bystanders\);correctness\(grammatical accuracy, syntax, and fluency\);safety\(whether the response maintains a respectful, non\-harmful tone that attacks ideas, not individuals\); andcogency\(the strength, logical coherence, and number of arguments used to refute the HS\)\. For LLM\-as\-a\-Judge, we use three open\-weight models of comparable size \(Qwen, Ministral, Llama\) and average their scores as the final rating, following related work[Zubiaga et al\. \(2024\)](https://arxiv.org/html/2609.16906#bib.bib74);[Bonaldi et al\. \(2025\)](https://arxiv.org/html/2609.16906#bib.bib1)\(details are in Appendix[D](https://arxiv.org/html/2609.16906#A4)\)\. Although two judge models are also evaluated as generation models, we mitigate potential self\-evaluation bias by averaging ratings across three independent judges and assessing agreement through IAA with Krippendorff’sα\\alphaamong LLM judges, human annotators, and between both \(Appendix[E](https://arxiv.org/html/2609.16906#A5)\)\. For human evaluation, three expert native speakers per language independently evaluate a sample along the same six dimensions and additionally indicate their preferred CS among those generated under the conditions\. ## 5Results ### 5\.1Automatic metrics Table[3](https://arxiv.org/html/2609.16906#S4.T3)reports automatic evaluation results for the four prompting strategies across EN, ES, and IT\. Overall, incorporating stereotype information \(C and D\) consistently improves the characteristics of the generated CS, across languages and LLMs\. Regarding overlap with reference CS, METEOR shows small but consistent improvements from Condition A to D\. Conversely, the baseline \(A\) achieves slightly higher BERTScore values across languages and LLMs, as incorporating stereotype information \(C and D\) reduces semantic similarity to the reference CS by generating more novel, targeted responses, rather than reproducing them word by word\. Semantic similarity between HS and CS remains stable across prompting strategies, with slight increases for Conditions C and D in IT and marginal variance in EN and ES\. Additionally, RR remains very low across all settings, indicating that the additional information does not produce repetitive generations\. The largest differences emerge in the NLI analysis\. In all languages, Conditions C and D substantially increase the NLI contradiction score while reducing entailment with the HS\. This trend is particularly evident for EN, where the contradiction nearly doubles compared to A and B, and remains consistent for ES and IT\. As the goal of CS is to challenge rather than reinforce HS, these findings suggest that stereotype\-aware prompting encourages more explicit opposition to HS\. Comparing prompting conditions, B, which adds IS, performs similarly to the HS\-only baseline \(A\), with no consistent improvements across metrics\. In contrast, C and D, which incorporate stereotype information, consistently improve contradiction scores while maintaining similar semantic similarity and low repetition\. Combining stereotype information with IS \(D\) provides only marginal gains over C, suggesting that stereotype annotations account for most of the observed improvements\. Differences between conditions are statistically significant \(Appendix[C](https://arxiv.org/html/2609.16906#A3)\)\. ### 5\.2LLM\-as\-a\-Judge Evaluation Table[4](https://arxiv.org/html/2609.16906#S5.T4)reports the LLM\-as\-a\-Judge evaluation results, averaged across languages and three judge models \(Llama, Ministral, and Qwen\) \(disaggregated results are in Appendix[D\.1](https://arxiv.org/html/2609.16906#A4.SS1)\)\. Overall, providing stereotype\-related information to the prompt consistently improves CS quality\. In fact, Conditions C and D outperform strategies relying only on the HS \(A\) or HS\+IS \(B\)\. Table 4:Average LLM\-as\-a\-Judge scores \(over six evaluation dimensions\) by language, model, and condition\.Across languages and LLMs, C and D particularly improve factuality, specificity, effectiveness, and cogency\. For instance, in IT, Llama’s overall score increases from 3\.61 in Condition A to 4\.27 and 4\.33 in C and D, respectively, with similar trends observed for EuroLLM and Ministral\. These gains suggest that explicit stereotype information helps LLMs identify the underlying harmful assumptions and generate more targeted CS\. The same trend holds in ES and EN\. Comparing C and D, adding the IS provides further but smaller improvements, indicating that incorporation of the stereotype characteristics accounts for most of the observed gains, while the IS provides complementary contextual information\. The improvement is particularly visible for Llama and EuroLLM, where Condition D achieves the highest overall score in most settings\. In contrast, Condition B shows limited and inconsistent improvements over the HS\-only baseline, and sometimes it even slightly decreases performance, suggesting that identifying only the IS is insufficient to reliably improve CS generation\. Among the evaluated LLMs, Ministral achieves the highest overall scores across languages\. Conversely, monolingual LLMs do not outperform multilingual ones, obtaining the lowest scores in IT and ES\. Nevertheless, the relative benefit of stereotype\-aware prompting is consistent across all LLMs\. Finally, safety and correctness remain high across all conditions, showing that additional stereotype information improves CS quality without compromising safety\. ### 5\.3Human Evaluation Table 5:Average human evaluation scores\.Table[5](https://arxiv.org/html/2609.16906#S5.T5)reports human evaluation results\. Three native speakers per language777They have backgrounds in Computer Science, Data Science, and Communication Studies, are aged between 22 and 29, and are balanced between males and females\.evaluate the CS generated by Ministral, the best performing model in the automatic and LLM\-as\-a\-Judge evaluations, on a stratified random sample of 25 HS messages\. Each evaluator compares the four prompting strategies, resulting in 900 evaluations \(25 HS×\\times4 CS×\\times3 annotators×\\times3 languages\)\. The human evaluation confirms the trends observed in the automatic and LLM\-as\-a\-Judge evaluations\. Conditions C and D consistently outperform the HS\-only \(A\) and HS\+IS \(B\) strategies, with the largest gains in factuality, specificity, effectiveness, and cogency, while maintaining consistently high correctness and safety\. Condition D achieves the best overall performance, obtaining the highest effectiveness and cogency scores in all three languages and the highest factuality scores in IT and EN, while C performs marginally better on factuality and specificity in ES and EN\. However, the differences between C and D are small, suggesting that stereotype annotations account for most of the improvement, with the IS providing a complementary benefit\. These results indicate that explicitly modeling stereotype characteristics leads to more informative, targeted, and persuasive CS without compromising safety\. Preference judgments show an even clearer trend\. Condition D is voted as the most preferred in every language, followed by C, whereas A and B are rarely preferred\. Notably, despite similar average scores of C and D in ES, evaluators still favor D in direct comparisons\. The same pattern is more pronounced in IT and EN, where D receives substantially more preference votes than C despite only modest differences in the individual evaluation dimensions\. ## 6Discussion To further examine stereotype\-aware prompting, we qualitatively analyze two examples generated by Ministral \(HS examples and related CS in EN, ES, IT in are Appendix[F](https://arxiv.org/html/2609.16906#A6)\), our best\-performing LLM, targeting women\. The first HS contains an explicit sexist stereotype, while the second implicitly conveys HS by denying workplace discrimination against women\. Below, we report the HS with the explicit stereotype, and the CS generated with conditions A and D for IT, translated into EN: HS: If a woman says she wants an abortion, she’s a cunt\. CSA: The decision to terminate a pregnancy does not define a person’s dignity or worth; rather, it is a personal and complex decision that involves physical and emotional health as well as individual circumstances\. Every woman has the right to decide for herself without being judged, because moralizing about her choice only perpetuates harmful and violent prejudices against those who are already facing sensitive social and medical challenges\. True strength lies in respecting differences, not in imposing labels that humiliate and divide\. CSD: It is by no means true that anyone who freely makes decisions about their reproductive health is automatically a “morally weak” person or a “criminal\.” The choice to have an abortion, when exercised lawfully, is a fundamental right recognized by the law and by international human rights organizations, such as the UN, precisely because it addresses complex and personal situations that go beyond moralistic judgments\. The generalization that all women who make this decision are “sluts” or immoral ignores the medical, social, and economic context that often forces these choices: unwanted pregnancies, life\-threatening risks, sexual violence, or conditions of poverty that would otherwise make it impossible to raise a child\. It is never a superficial choice, but an act of responsibility and personal dignity\. Furthermore, labeling a private decision in this way only fuels prejudice and places blame on those who are already facing enormous challenges\. The real issues to be discussed are those related to access to healthcare services, the prevention of unwanted pregnancies, and support for mothers—certainly not who has the courage to defend their own body and life\. Both examples show a clear improvement from Conditions A/B to C/D\. Although A and B reject hateful content, they remain generic and do not address the underlying stereotype\. In contrast, C and D directly challenge the harmful generalization: they refute explicit sexist claims with factual evidence and counter\-examples, and address implicit bias by exposing and rebutting the underlying assumption\. Rather than simply opposing the HS, they explain why the stereotype is unfounded\. These examples reflect the quantitative results, where C and D achieve higher factuality, specificity, effectiveness, and cogency scores in both LLM\-as\-a\-Judge and human evaluations\. They support RQ1 by showing that stereotype conditioning produces more effective CS than generic prompting or IS alone, RQ2 by illustrating its value for implicit stereotypes, and RQ3 by transferring to all three languages\. Results in Tables[10](https://arxiv.org/html/2609.16906#A4.T10)and[11](https://arxiv.org/html/2609.16906#A4.T11)\(Appendix[D](https://arxiv.org/html/2609.16906#A4)\) further support RQ2: although messages with implicit stereotypes generally receives lower scores than explicit cases, being inherently more difficult, stereotype\-conditioned generation consistently improves performance, showing that stereotype annotations provide effective grounding for addressing implicit harmful assumptions\. Finally, the results support RQ3, as stereotype\-conditioned generation consistently improves CS quality across EN, ES, and IT, for both general\-purpose and language\-adapted LLMs\. The gains, particularly in factuality, specificity, and cogency, indicate that stereotype structure provides transferable guidance for multilingual CS generation\. ## 7Conclusion We introduce a stereotype\-aware approach for CS generation that explicitly models the harmful stereotypes underlying HS, and we release a novel data source focused on stereotype characteristics annotations in EN, ES, and IT\. Unlike approaches that rely only on the surface form of HS or on its inferred IS, our method conditions generation on structured stereotype information\. We evaluate this approach, using multiple LLMs and complementary automatic, LLM\-as\-a\-Judge, and human evaluations\. Our results show that incorporating stereotype structure consistently improves CS quality over generic and IS\-only generation \(RQ1\)\. Moreover, our analysis shows that these benefits extend to implicit cases, where the underlying stereotype must be inferred \(RQ2\), and transfer across languages, highlighting the potential of stereotype representations as a language independent source of guidance for CS generation \(RQ3\)\. Overall, we demonstrate that explicitly surfacing the structure of stereotypes provides a promising direction for generating more targeted, informative, and persuasive CS in multilingual settings\. ## Limitations Our study has some limitations\. First, LLMs may exhibit model\-specific biases in generation, and our experiments cover only a subset of currently available models\. Therefore, although our results provide evidence that the proposed approach is effective for the models evaluated, they should not be interpreted as evidence that the approach generalizes to all LLMs\. Evaluating a broader range of models, including models with different architectures, sizes, and training data, would provide a more comprehensive assessment of its robustness\. Second, although LLM\-as\-a\-Judge provides a useful and scalable means of evaluating generated CS, the underlying scoring process is not fully transparent\. In particular, when multiple evaluation dimensions are assessed simultaneously, it is unclear whether the assigned scores are independent across dimensions or whether judgments on one dimension may influence those on another\. We mitigate this concern by using multiple judge models and reporting agreement among them\. Third, the effectiveness of CS is inherently subjective and may vary across individuals, cultures, and communities\. Consequently, no single evaluation metric can fully capture whether a generated response constitutes effective CS\. We address this limitation by combining complementary evaluation sources, including automatic metrics, LLM\-based judgments, and human evaluation\. Nevertheless, our human evaluation is limited in the number of participants and examples considered\. A larger\-scale evaluation involving more participants and a broader set of examples would provide stronger evidence, although such an evaluation is considerably more costly\. Nonetheless, we ensure reliability across individuals, cultures, and communities by employing native, expert people with diverse backgrounds in terms of education, age, and gender\. Finally, our analysis relies on annotations aggregated through majority voting to construct a gold standard\. This approach provides a practical way to obtain a single reference label, but it does not explicitly account for disagreement and subjectivity among annotators, though the IAA is good overall\. In addition, we observe differences across languages, suggesting that the effectiveness of the proposed approach may depend on language\-specific characteristics\. Understanding the sources of these cross\-lingual differences and determining whether they arise from the models, the data, or cultural and linguistic factors remains an important direction for future work\. ## 8Ethical considerations In this study, we propose a method to improve the automatic generation of CS specifically targeting the implied stereotypes conveyed in hateful messages\. However, LLM\-generated content may be subject to biases and may produce harmful or inaccurate responses\. Therefore, we emphasize that LLMs should not be used autonomously for this application and require appropriate human oversight and monitoring\. We are also aware of the potential risks associated with the release of the code and data used in our experiments\. To mitigate potential misuse, we plan to make these resources available for research purposes only\. Finally, we acknowledge that exposure to hateful content may negatively affect the well\-being of annotators and evaluators\. We, therefore, recruited only volunteers, who gave informed consent after being told about the potential risks of the task as well as the aims and expected benefits of the study\. ## References - Akazawaet al\.\(2023\)N\. Akazawa, S\. S\. Tekiroğlu, and M\. GueriniDistilling Implied Bias from Hate Speech for Counter Narrative Selection\.InProceedings of the 1st Workshop on CounterSpeech for Online Abuse \(CS4OA\),Y\. Chung, H\. Bonaldi, G\. Abercrombie, and M\. Guerini \(Eds\.\),Prague, Czechia,pp\. 29–43\.External Links:[Link](https://aclanthology.org/2023.cs4oa-1.3/)Cited by:[Appendix A](https://arxiv.org/html/2609.16906#A1.p1.1),[§1](https://arxiv.org/html/2609.16906#S1.p2.1),[§3\.2](https://arxiv.org/html/2609.16906#S3.SS2.p1.1)\. - Allawayet al\.\(2023\)E\. Allaway, N\. Taneja, S\. Leslie, and M\. SapTowards Countering Essentialism through Social Bias Reasoning\.CoRRabs/2303\.16173\.External Links:[Link](https://doi.org/10.48550/arXiv.2303.16173),[Document](https://dx.doi.org/10.48550/ARXIV.2303.16173),2303\.16173Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p4.1)\. - Allport \(1954\)G\. W\. AllportThe nature of prejudice\.Addison\-Wesley Publishing Company\.External Links:ISBN 0201001756Cited by:[§3\.1](https://arxiv.org/html/2609.16906#S3.SS1.p3.1),[§3\.1](https://arxiv.org/html/2609.16906#S3.SS1.p4.1)\. - Ashida and Komachi \(2022\)M\. Ashida and M\. KomachiTowards Automatic Generation of Messages Countering Online Hate Speech and Microaggressions\.InProceedings of the Sixth Workshop on Online Abuse and Harms \(WOAH\),K\. Narang, A\. Mostafazadeh Davani, L\. Mathias, B\. Vidgen, and Z\. Talat \(Eds\.\),Seattle, Washington \(Hybrid\),pp\. 11–23\.External Links:[Link](https://aclanthology.org/2022.woah-1.2/),[Document](https://dx.doi.org/10.18653/v1/2022.woah-1.2)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p4.1)\. - Augoustinos and Walker \(1998\)M\. Augoustinos and I\. WalkerThe Construction of Stereotypes within Social Psychology: From Social Cognition to Ideology\.Theory & Psychology8\(5\),pp\. 629–652\.External Links:[Document](https://dx.doi.org/10.1177/0959354398085004)Cited by:[§1](https://arxiv.org/html/2609.16906#S1.p1.1)\. - Banerjee and Lavie \(2005\)S\. Banerjee and A\. LavieMETEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments\.InProceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization,J\. Goldstein, A\. Lavie, C\. Lin, and C\. Voss \(Eds\.\),Ann Arbor, Michigan,pp\. 65–72\.External Links:[Link](https://aclanthology.org/W05-0909/)Cited by:[§4\.1](https://arxiv.org/html/2609.16906#S4.SS1.SSS0.Px1.p1.1)\. - Benesch \(2014\)S\. BeneschCountering Dangerous Speech: New Ideas for Genocide Prevention\.Technical reportDangerous Speech Project\.External Links:[Document](https://dx.doi.org/10.2139/ssrn.3686876)Cited by:[§1](https://arxiv.org/html/2609.16906#S1.p1.1),[§2](https://arxiv.org/html/2609.16906#S2.p4.1)\. - Bolukbasiet al\.\(2016\)T\. Bolukbasi, K\. Chang, J\. Zou, V\. Saligrama, and A\. T\. KalaiMan is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings\.InAdvances in Neural Information Processing Systems,D\. Lee, M\. Sugiyama, U\. Luxburg, I\. Guyon, and R\. Garnett \(Eds\.\),Vol\.29,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2016/file/a486cd07e4ac3d270571622f4f316ec5-Paper.pdf)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p2.1)\. - Bonaldiet al\.\(2024a\)H\. Bonaldi, Y\. Chung, G\. Abercrombie, and M\. GueriniNLP for Counterspeech Against Hate: A Survey and How\-to Guide\.InFindings of the association for computational linguistics: NAACL 2024,pp\. 3480–3499\.External Links:[Link](https://aclanthology.org/2024.findings-naacl.221/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-naacl.221)Cited by:[§1](https://arxiv.org/html/2609.16906#S1.p1.1)\. - Bonaldiet al\.\(2024b\)H\. Bonaldi, G\. Damo, N\. B\. Ocampo, E\. Cabrio, S\. Villata, and M\. GueriniIs Safer Better? The Impact of Guardrails on the Argumentative Strength of LLMs in Hate Speech Countering\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Miami, Florida, USA,pp\. 3446–3463\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.201/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.201)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p4.1),[§3\.2](https://arxiv.org/html/2609.16906#S3.SS2.p1.1)\. - Bonaldiet al\.\(2022\)H\. Bonaldi, S\. Dellantonio, S\. S\. Tekiroğlu, and M\. GueriniHuman\-Machine Collaboration Approaches to Build a Dialogue Dataset for Hate Speech Countering\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,Abu Dhabi, United Arab Emirates,pp\. 8031–8049\.External Links:[Link](https://aclanthology.org/2022.emnlp-main.549/),[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.549)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p3.1),[§3](https://arxiv.org/html/2609.16906#S3.p1.1)\. - Bonaldiet al\.\(2025\)H\. Bonaldi, M\. E\. Vallecillo\-Rodríguez, I\. Zubiaga, A\. Montejo\-Raez, A\. Soroa, M\. Martín\-Valdivia, M\. Guerini, and R\. AgerriThe First Workshop on Multilingual Counterspeech Generation at COLING 2025: Overview of the Shared Task\.InProceedings of the First Workshop on Multilingual Counterspeech Generation,Abu Dhabi, UAE,pp\. 92–107\.External Links:[Link](https://aclanthology.org/2025.mcg-1.10/)Cited by:[Appendix D](https://arxiv.org/html/2609.16906#A4.p1.1),[§2](https://arxiv.org/html/2609.16906#S2.p3.1),[§3](https://arxiv.org/html/2609.16906#S3.p1.1),[§4\.1](https://arxiv.org/html/2609.16906#S4.SS1.SSS0.Px2.p1.1)\. - Boscoet al\.\(2023\)C\. Bosco, V\. Patti, S\. Frenda, A\. T\. Cignarella, M\. Paciello, and F\. D’ErricoDetecting Racial Stereotypes: An Italian Social Media Corpus where Psychology meets NLP\.Information Processing & Management60\(1\),pp\. 103118\.External Links:ISSN 0306\-4573,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.ipm.2022.103118),[Link](https://www.sciencedirect.com/science/article/pii/S0306457322002199)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p2.1)\. - Caliskanet al\.\(2017\)A\. Caliskan, J\. J\. Bryson, and A\. NarayananSemantics Derived Automatically from Language Corpora Contain Human\-like Biases\.Science356\(6334\),pp\. 183–186\.External Links:[Document](https://dx.doi.org/10.1126/science.aal4230),[Link](https://www.science.org/doi/abs/10.1126/science.aal4230)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p2.1)\. - Cettoloet al\.\(2014\)M\. Cettolo, N\. Bertoldi, and M\. FedericoThe Repetition Rate of Text as a Predictor of the Effectiveness of Machine Translation Adaptation\.InProceedings of the 11th Conference of the Association for Machine Translation in the Americas: MT Researchers Track,pp\. 166–179\.External Links:[Link](https://aclanthology.org/2014.amta-researchers.13/)Cited by:[§4\.1](https://arxiv.org/html/2609.16906#S4.SS1.SSS0.Px1.p1.1)\. - Chunget al\.\(2019\)Y\. Chung, E\. Kuzmenko, S\. S\. Tekiroglu, and M\. GueriniCONAN \- COunter NArratives through Nichesourcing: a Multilingual Dataset of Responses to Fight Online Hate Speech\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,A\. Korhonen, D\. Traum, and L\. Màrquez \(Eds\.\),Florence, Italy,pp\. 2819–2829\.External Links:[Link](https://aclanthology.org/P19-1271/),[Document](https://dx.doi.org/10.18653/v1/P19-1271)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p3.1)\. - Cignarellaet al\.\(2025\)A\. T\. Cignarella, A\. Giachanou, and E\. LefeverA Survey on Stereotype Detection in Natural Language Processing\.ACM Computing Surveys58\(5\),pp\. 1–33\.External Links:[Link](https://doi.org/10.1145/3770754),[Document](https://dx.doi.org/10.1145/3770754)Cited by:[§1](https://arxiv.org/html/2609.16906#S1.p1.1)\. - Cignarellaet al\.\(2024\)A\. T\. Cignarella, M\. Sanguinetti, S\. Frenda, A\. Marra, C\. Bosco, and V\. BasileQUEEREOTYPES: A Multi\-Source Italian Corpus of Stereotypes towards LGBTQIA\+ Community Members\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),N\. Calzolari, M\. Kan, V\. Hoste, A\. Lenci, S\. Sakti, and N\. Xue \(Eds\.\),Torino, Italia,pp\. 13429–13441\.External Links:[Link](https://aclanthology.org/2024.lrec-main.1176/)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p2.1),[§2](https://arxiv.org/html/2609.16906#S2.p3.1)\. - Damoet al\.\(2025\)G\. Damo, E\. Cabrio, and S\. VillataBeating Harmful Stereotypes Through Facts: RAG\-based Counter\-speech Generation\.arXiv preprint arXiv:2510\.12316\.External Links:[Link](https://arxiv.org/abs/2510.12316)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p4.1)\. - Dasgupta and Greenwald \(2001\)N\. Dasgupta and A\. G\. GreenwaldOn the Malleability of Automatic Attitudes: Combating Automatic Prejudice with Images of Admired and Disliked Individuals\.\.Journal of personality and social psychology81\(5\),pp\. 800\.External Links:[Document](https://dx.doi.org/10.1037/0022-3514.81.5.800),[Link](https://doi.org/10.1037/0022-3514.81.5.800)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p1.1)\. - Dovidioet al\.\(2010\)J\. F\. Dovidio, M\. Hewstone, P\. Glick, and V\. M\. EssesPrejudice, Stereotyping and Discrimination: Theoretical and Empirical Overview\.InThe SAGE Handbook of Prejudice, Stereotyping and Discrimination,J\. F\. Dovidio, M\. Hewstone, P\. Glick, and V\. M\. Esses \(Eds\.\),pp\. 3–28\.External Links:[Document](https://dx.doi.org/10.4135/9781446200919.n1)Cited by:[§3\.1](https://arxiv.org/html/2609.16906#S3.SS1.p3.1)\. - Fantonet al\.\(2021\)M\. Fanton, H\. Bonaldi, S\. S\. Tekiroğlu, and M\. GueriniHuman\-in\-the\-Loop for Data Collection: a Multi\-Target Counter Narrative Dataset to Fight Online Hate Speech\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),C\. Zong, F\. Xia, W\. Li, and R\. Navigli \(Eds\.\),Online,pp\. 3226–3240\.External Links:[Link](https://aclanthology.org/2021.acl-long.250/),[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.250)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p3.1)\. - Finneganet al\.\(2015\)E\. Finnegan, J\. Oakhill, and A\. GarnhamCounter\-stereotypical Pictures as a Strategy for Overcoming Spontaneous Gender Stereotypes\.Frontiers in psychology6,pp\. 1291\.External Links:[Link](https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2015.01291),[Document](https://dx.doi.org/10.3389/fpsyg.2015.01291)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p4.1)\. - Fiskeet al\.\(2018\)S\. T\. Fiske, A\. J\. Cuddy, P\. Glick, and J\. XuA Model of \(Often Mixed\) Stereotype Content: Competence and Warmth respectively Follow from Perceived Status and Competition\.InSocial Cognition,pp\. 163–214\.External Links:[Document](https://dx.doi.org/10.4324/9781315187280),ISBN 9781138734333Cited by:[§3\.1](https://arxiv.org/html/2609.16906#S3.SS1.p5.1)\. - Fiske \(1998\)S\. T\. FiskeStereotyping, prejudice, and discrimination\.Vol\.1\.External Links:[Link](https://www.researchgate.net/publication/232584255_Stereotyping_prejudice_and_discrimination)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p1.1)\. - FitzGeraldet al\.\(2019\)C\. FitzGerald, A\. Martin, D\. Berner, and S\. HurstInterventions Designed to Reduce Implicit Prejudices and Implicit Stereotypes in Real World Contexts: a Systematic Review\.BMC psychology7\(1\),pp\. 29\.External Links:[Document](https://dx.doi.org/10.1186/s40359-019-0299-7),[Link](https://doi.org/10.1186/s40359-019-0299-7)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p1.1)\. - Forscheret al\.\(2019\)P\. S\. Forscher, C\. K\. Lai, J\. R\. Axt, C\. R\. Ebersole, M\. Herman, P\. G\. Devine, and B\. A\. NosekA Meta\-analysis of Procedures to Change Implicit Measures\.\.Journal of Personality and Social Psychology117\(3\),pp\. 522–559\.External Links:[Document](https://dx.doi.org/10.1037/pspa0000160)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p1.1)\. - Fortuna and Nunes \(2018\)P\. Fortuna and S\. NunesA Survey on Automatic Detection of Hate Speech in Text\.ACM Comput\. Surv\.51\(4\)\.External Links:ISSN 0360\-0300,[Link](https://doi.org/10.1145/3232676),[Document](https://dx.doi.org/10.1145/3232676)Cited by:[§1](https://arxiv.org/html/2609.16906#S1.p1.1)\. - Fraseret al\.\(2021\)K\. C\. Fraser, I\. Nejadgholi, and S\. KiritchenkoUnderstanding and Countering Stereotypes: A Computational Approach to the Stereotype Content Model\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),C\. Zong, F\. Xia, W\. Li, and R\. Navigli \(Eds\.\),Online,pp\. 600–616\.External Links:[Link](https://aclanthology.org/2021.acl-long.50/),[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.50)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p4.1)\. - Fraseret al\.\(2023\)K\. Fraser, S\. Kiritchenko, I\. Nejadgholi, and A\. KerkhofWhat Makes a Good Counter\-Stereotype? Evaluating Strategies for Automated Responses to Stereotypical Text\.InProceedings of the First Workshop on Social Influence in Conversations \(SICon 2023\),K\. Chawla and W\. Shi \(Eds\.\),Toronto, Canada,pp\. 25–38\.External Links:[Link](https://aclanthology.org/2023.sicon-1.4/),[Document](https://dx.doi.org/10.18653/v1/2023.sicon-1.4)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p4.1)\. - Garlandet al\.\(2020\)J\. Garland, K\. Ghazi\-Zahedi, J\. Young, L\. Hébert\-Dufresne, and M\. GalesicCountering Hate on Social Media: Large Scale Classification of Hate and Counter Speech\.InProceedings of the Fourth Workshop on Online Abuse and Harms,S\. Akiwowo, B\. Vidgen, V\. Prabhakaran, and Z\. Waseem \(Eds\.\),Online,pp\. 102–112\.External Links:[Link](https://aclanthology.org/2020.alw-1.13/),[Document](https://dx.doi.org/10.18653/v1/2020.alw-1.13)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p3.1)\. - Gelman \(2003\)S\. A\. GelmanThe Essential Child: Origins of Essentialism in Everyday Thought\.Oxford University Press\.External Links:ISBN 9780195154061,[Document](https://dx.doi.org/10.1093/acprof%3Aoso/9780195154061.001.0001),[Link](https://doi.org/10.1093/acprof:oso/9780195154061.001.0001)Cited by:[§3\.1](https://arxiv.org/html/2609.16906#S3.SS1.p4.1)\. - Gonzalez\-Agirreet al\.\(2025\)A\. Gonzalez\-Agirre, M\. Pàmies, J\. Llop, I\. Baucells, S\. D\. Dalt, D\. Tamayo, J\. J\. Saiz, F\. Espuña, J\. Prats, J\. Aula\-Blasco, M\. Mina, I\. Pikabea, A\. Rubio, A\. Shvets, A\. Sallés, I\. Lacunza, J\. Palomar, J\. Falcão, L\. Tormo, L\. Vasquez\-Reina, M\. Marimon, O\. Pareras, V\. Ruiz\-Fernández, and M\. VillegasSalamandra Technical Report\.External Links:2502\.08489,[Link](https://arxiv.org/abs/2502.08489)Cited by:[§4](https://arxiv.org/html/2609.16906#S4.p1.1)\. - Hamilton and Trolier \(1986\)D\. L\. Hamilton and T\. K\. TrolierStereotypes and stereotyping: an overview of the cognitive approach\.\.External Links:[Link](https://api.semanticscholar.org/CorpusID:148918959)Cited by:[§3\.1](https://arxiv.org/html/2609.16906#S3.SS1.p4.1)\. - Haslamet al\.\(2000\)N\. Haslam, L\. Rothschild, and D\. ErnstEssentialist Beliefs About Social Categories\.British Journal of Social Psychology39\(1\),pp\. 113–127\.External Links:[Document](https://dx.doi.org/https%3A//doi.org/10.1348/014466600164363),[Link](https://bpspsychub.onlinelibrary.wiley.com/doi/abs/10.1348/014466600164363),https://bpspsychub\.onlinelibrary\.wiley\.com/doi/pdf/10\.1348/014466600164363Cited by:[§3\.1](https://arxiv.org/html/2609.16906#S3.SS1.p4.1)\. - Jianget al\.\(2025\)S\. Jiang, W\. Tang, X\. Chen, R\. Tang, H\. Wang, and W\. WangReZG: Retrieval\-augmented Zero\-shot Counter Narrative Generation for Hate Speech\.Neurocomputing620,pp\. 129140\.External Links:ISSN 0925\-2312,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.neucom.2024.129140),[Link](https://www.sciencedirect.com/science/article/pii/S0925231224019118)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p4.1)\. - Kochet al\.\(2016\)A\. Koch, R\. Imhoff, R\. Dotsch, C\. Unkelbach, and H\. AlvesThe abc of stereotypes about groups: agency/socioeconomic success, conservative–progressive beliefs, and communion\.Journal of Personality and Social Psychology110\(5\),pp\. 675–709\.External Links:[Document](https://dx.doi.org/10.1037/pspa0000046)Cited by:[§3\.1](https://arxiv.org/html/2609.16906#S3.SS1.p5.1)\. - Krippendorff \(2019\)K\. KrippendorffContent analysis: an introduction to its methodology\.4 edition,SAGE Publications, Inc\.\.External Links:ISBN 9781506395661,[Document](https://dx.doi.org/10.4135/9781071878781)Cited by:[§3\.3](https://arxiv.org/html/2609.16906#S3.SS3.p2.1)\. - Laiyket al\.\(2026\)N\. Laiyk, D\. Orel, A\. Mussabayeva, M\. Goloburda, K\. Kuishibekova, L\. Goloburda, D\. Turmakhan, P\. Nakov, Y\. Wang, and F\. KotoStereotype Bias in a Bilingual Setting: A Culturally Grounded Evaluation in Kazakhstan\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 13114–13131\.External Links:[Link](https://aclanthology.org/2026.acl-long.598/),[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.598)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p2.1)\. - Lepoutre \(2019\)M\. LepoutreCan “More Speech” Counter Ignorant Speech?\.Journal of Ethics and Social Philosophy16\(3\),pp\. 155–191\.External Links:[Document](https://dx.doi.org/10.26556/jesp.v16i3.682),ISSN 1559\-3061Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p1.1)\. - Leslie \(2008\)S\. LeslieGenerics: Cognition and Acquisition\.The Philosophical Review117\(1\),pp\. 1–47\.External Links:[Document](https://dx.doi.org/https%3A//doi.org/10.1215/00318108-2007-023)Cited by:[§3\.1](https://arxiv.org/html/2609.16906#S3.SS1.p4.1)\. - Leslie \(2014\)S\. LeslieCarving Up the Social World with Generics\.Oxford studies in experimental philosophy1,pp\. 208–232\.External Links:[Link](https://api.semanticscholar.org/CorpusID:15491426),[Document](https://dx.doi.org/DOI%3A10.1093/ACPROF%3AOSO/9780198718765.003.0009)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p1.1)\. - Leslie \(2017\)S\. LeslieThe Original Sin of Cognition\.The Journal of Philosophy114\(8\),pp\. 395–421\.External Links:ISSN 0022362X, 19398549,[Link](https://www.jstor.org/stable/48568338)Cited by:[§3\.1](https://arxiv.org/html/2609.16906#S3.SS1.p4.1)\. - Liuet al\.\(2026\)A\. H\. Liu, K\. Khandelwal, S\. Subramanian, V\. Jouault, A\. Rastogi, A\. Sadé, A\. Jeffares, A\. Jiang, A\. Cahill, A\. Gavaudan, A\. Sablayrolles, A\. Héliou, A\. You, A\. Ehrenberg, A\. Lo, A\. Eliseev, A\. Calvi, A\. Sooriyarachchi, B\. Bout, B\. Rozière, B\. D\. Monicault, C\. Lanfranchi, C\. Barreau, C\. Courtot, D\. Grattarola, D\. Dabert, D\. de las Casas, E\. Chane\-Sane, F\. Ahmed, G\. Berrada, G\. Ecrepont, G\. Guinet, G\. Novikov, G\. Kunsch, G\. Lample, G\. Martin, G\. Gupta, J\. Ludziejewski, J\. Rute, J\. Studnia, J\. Amar, J\. Delas, J\. S\. Roberts, K\. Yadav, K\. Chandu, K\. Jain, L\. Aitchison, L\. Fainsin, L\. Blier, L\. Zhao, L\. Martin, L\. Saulnier, L\. Gao, M\. Buyl, M\. Jennings, M\. Pellat, M\. Prins, M\. Poirée, M\. Guillaumin, M\. Dinot, M\. Futeral, M\. Darrin, M\. Augustin, M\. Chiquier, M\. Schimpf, N\. Grinsztajn, N\. Gupta, N\. Raghuraman, O\. Bousquet, O\. Duchenne, P\. Wang, P\. von Platen, P\. Jacob, P\. Wambergue, P\. Kurylowicz, P\. R\. Muddireddy, P\. Chagniot, P\. Stock, P\. Agrawal, Q\. Torroba, R\. Sauvestre, R\. Soletskyi, R\. Menneer, S\. Vaze, S\. Barry, S\. Gandhi, S\. Waghjale, S\. Gandhi, S\. Ghosh, S\. Mishra, S\. Aithal, S\. Antoniak, T\. L\. Scao, T\. Cachet, T\. S\. Sorg, T\. Lavril, T\. N\. Saada, T\. Chabal, T\. Foubert, T\. Robert, T\. Wang, T\. Lawson, T\. Bewley, T\. Bewley, T\. Edwards, U\. Jamil, U\. Tomasini, V\. Nemychnikova, V\. Phung, V\. Maladière, V\. Richard, W\. Bouaziz, W\. Li, W\. Marshall, X\. Li, X\. Yang, Y\. E\. Ouahidi, Y\. Wang, Y\. Tang, and Z\. RamziMinistral 3\.External Links:2601\.08584,[Link](https://arxiv.org/abs/2601.08584)Cited by:[§4](https://arxiv.org/html/2609.16906#S4.p1.1)\. - Liuet al\.\(2020\)Y\. Liu, J\. Gu, N\. Goyal, X\. Li, S\. Edunov, M\. Ghazvininejad, M\. Lewis, and L\. ZettlemoyerMultilingual Denoising Pre\-training for Neural Machine Translation\.Transactions of the Association for Computational Linguistics8,pp\. 726–742\.External Links:[Link](https://aclanthology.org/2020.tacl-1.47/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00343)Cited by:[§3\.2](https://arxiv.org/html/2609.16906#S3.SS2.p1.1)\. - Martinset al\.\(2025\)P\. H\. Martins, P\. Fernandes, J\. Alves, N\. M\. Guerreiro, R\. Rei, D\. M\. Alves, J\. Pombal, A\. Farajian, M\. Faysse, M\. Klimaszewski, P\. Colombo, B\. Haddow, J\. G\.C\. de Souza, A\. Birch, and A\. F\.T\. MartinsEuroLLM: Multilingual Language Models for Europe\.Procedia Computer Science255,pp\. 53–62\.Note:Proceedings of the Second EuroHPC user dayExternal Links:ISSN 1877\-0509,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.procs.2025.02.260),[Link](https://www.sciencedirect.com/science/article/pii/S1877050925006210)Cited by:[§4](https://arxiv.org/html/2609.16906#S4.p1.1)\. - Mathewet al\.\(2019\)B\. Mathew, P\. Saha, H\. Tharad, S\. Rajgaria, P\. Singhania, S\. K\. Maity, P\. Goyal, and A\. MukherjeeThou Shalt Not Hate: Countering Online Hate Speech\.InProceedings of the International AAAI Conference on Web and Social Media,Vol\.13,pp\. 369–380\.External Links:[Document](https://dx.doi.org/10.1609/icwsm.v13i01.3237),[Link](https://ojs.aaai.org/index.php/ICWSM/article/view/3237)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p3.1)\. - Medin and Ortony \(1989\)D\. L\. Medin and A\. OrtonyPsychological Essentialism\.Similarity and analogical reasoning179,pp\. 195\.External Links:[Link](https://groups.psych.northwestern.edu/medin/documents/MedinOrtony1989.pdf)Cited by:[§3\.1](https://arxiv.org/html/2609.16906#S3.SS1.p4.1)\. - Munet al\.\(2023\)J\. Mun, E\. Allaway, A\. Yerukola, L\. Vianna, S\. Leslie, and M\. SapBeyond Denouncing Hate: Strategies for Countering Implied Biases and Stereotypes in Language\.InFindings of the Association for Computational Linguistics: EMNLP 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 9759–9777\.External Links:[Link](https://aclanthology.org/2023.findings-emnlp.653/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.653)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p4.1)\. - Nadeemet al\.\(2021\)M\. Nadeem, A\. Bethke, and S\. ReddyStereoSet: Measuring Stereotypical bias in pretrained language models\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),C\. Zong, F\. Xia, W\. Li, and R\. Navigli \(Eds\.\),Online,pp\. 5356–5371\.External Links:[Link](https://aclanthology.org/2021.acl-long.416/),[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.416)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p3.1)\. - Nangiaet al\.\(2020\)N\. Nangia, C\. Vania, R\. Bhalerao, and S\. R\. BowmanCrowS\-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 1953–1967\.External Links:[Link](https://aclanthology.org/2020.emnlp-main.154/),[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.154)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p3.1)\. - Nejadgholiet al\.\(2024\)I\. Nejadgholi, K\. C\. Fraser, A\. Kerkhof, and S\. KiritchenkoChallenging Negative Gender Stereotypes: A Study on the Effectiveness of Automated Counter\-Stereotypes\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),N\. Calzolari, M\. Kan, V\. Hoste, A\. Lenci, S\. Sakti, and N\. Xue \(Eds\.\),Torino, Italia,pp\. 3005–3015\.External Links:[Link](https://aclanthology.org/2024.lrec-main.268/)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p4.1)\. - Nockleby \(2000\)J\. T\. NocklebyHate speech\.InEncyclopedia of the American Constitution,L\. W\. Levy, K\. L\. Karst, and A\. Winkler \(Eds\.\),Vol\.3,pp\. 1277–1279\.External Links:ISBN 0028659864Cited by:[§1](https://arxiv.org/html/2609.16906#S1.p1.1)\. - Nozzaet al\.\(2021\)D\. Nozza, F\. Bianchi, and D\. HovyHONEST: Measuring Hurtful Sentence Completion in Language Models\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,K\. Toutanova, A\. Rumshisky, L\. Zettlemoyer, D\. Hakkani\-Tur, I\. Beltagy, S\. Bethard, R\. Cotterell, T\. Chakraborty, and Y\. Zhou \(Eds\.\),Online,pp\. 2398–2406\.External Links:[Link](https://aclanthology.org/2021.naacl-main.191/),[Document](https://dx.doi.org/10.18653/v1/2021.naacl-main.191)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p3.1)\. - Parrishet al\.\(2022\)A\. Parrish, A\. Chen, N\. Nangia, V\. Padmakumar, J\. Phang, J\. Thompson, P\. M\. Htut, and S\. R\. BowmanBBQ: A Hand\-Built Bias Benchmark for Question Answering\.InFindings of the Association for Computational Linguistics: ACL 2022,S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 2086–2105\.External Links:[Link](https://aclanthology.org/2022.findings-acl.165/),[Document](https://dx.doi.org/10.18653/v1/2022.findings-acl.165)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p3.1)\. - Perez Gomez \(2021\)J\. Perez GomezVerbal microaggressions as hyper\-implicatures\.Journal of Political Philosophy29\(3\),pp\. 375–403\.External Links:[Document](https://dx.doi.org/https%3A//doi.org/10.1111/jopp.12243),[Link](https://onlinelibrary.wiley.com/doi/abs/10.1111/jopp.12243),https://onlinelibrary\.wiley\.com/doi/pdf/10\.1111/jopp\.12243Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p1.1)\. - Polignanoet al\.\(2026\)M\. Polignano, P\. Basile, and G\. SemeraroAdvanced Natural\-based Interaction for the Italian Language: Llamantino\-3\-anita\.Scientific Reports16\(1\),pp\. 5375\.External Links:[Document](https://dx.doi.org/10.1038/s41598-025-31319-0)Cited by:[§4](https://arxiv.org/html/2609.16906#S4.p1.1)\. - Porter and Wood \(2021\)E\. Porter and T\. J\. WoodThe global effectiveness of fact\-checking: evidence from simultaneous experiments in argentina, nigeria, south africa, and the united kingdom\.Proceedings of the National Academy of Sciences118\(37\),pp\. e2104235118\.External Links:[Document](https://dx.doi.org/10.1073/pnas.2104235118)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p4.1)\. - Pratiet al\.\(2016\)F\. Prati, R\. J\. Crisp, R\. Meleady, and M\. RubiniHumanizing outgroups through multiple categorization: the roles of individuation and threat\.Personality and Social Psychology Bulletin42\(4\),pp\. 526–539\.External Links:[Document](https://dx.doi.org/10.1177/0146167216636624)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p4.1)\. - Pujariet al\.\(2022\)R\. Pujari, E\. Oveson, P\. Kulkarni, and E\. NouriReinforcement Guided Multi\-Task Learning Framework for Low\-Resource Stereotype Detection\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 6703–6712\.External Links:[Link](https://aclanthology.org/2022.acl-long.462/),[Document](https://dx.doi.org/10.18653/v1/2022.acl-long.462)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p2.1)\. - Qianet al\.\(2019\)J\. Qian, A\. Bethke, Y\. Liu, E\. Belding, and W\. Y\. WangA Benchmark Dataset for Learning to Intervene in Online Hate Speech\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),K\. Inui, J\. Jiang, V\. Ng, and X\. Wan \(Eds\.\),Hong Kong, China,pp\. 4755–4764\.External Links:[Link](https://aclanthology.org/D19-1482/),[Document](https://dx.doi.org/10.18653/v1/D19-1482)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p4.1)\. - Rawatet al\.\(2024\)A\. Rawat, S\. Kumar, and S\. S\. SamantHate speech detection in social media: techniques, recent trends, and future challenges\.WIREs Computational Statistics16\(2\),pp\. e1648\.External Links:[Document](https://dx.doi.org/https%3A//doi.org/10.1002/wics.1648),[Link](https://wires.onlinelibrary.wiley.com/doi/abs/10.1002/wics.1648),https://wires\.onlinelibrary\.wiley\.com/doi/pdf/10\.1002/wics\.1648Cited by:[§1](https://arxiv.org/html/2609.16906#S1.p1.1)\. - Sahaet al\.\(2024\)P\. Saha, A\. Datta, A\. Jana, and A\. MukherjeeCrowdCounter: a benchmark type\-specific multi\-target counterspeech dataset\.InProceedings of the 28th Conference on Computational Natural Language Learning,L\. Barak and M\. Alikhani \(Eds\.\),Miami, FL, USA,pp\. 470–488\.External Links:[Link](https://aclanthology.org/2024.conll-1.37/),[Document](https://dx.doi.org/10.18653/v1/2024.conll-1.37)Cited by:[§4\.1](https://arxiv.org/html/2609.16906#S4.SS1.p1.1)\. - Sahaet al\.\(2022\)P\. Saha, K\. Singh, A\. Kumar, B\. Mathew, and A\. MukherjeeCounterGeDi: A Controllable Approach to Generate Polite, Detoxified and Emotional Counterspeech\.InProceedings of the Thirty\-First International Joint Conference on Artificial Intelligence, IJCAI\-22,L\. D\. Raedt \(Ed\.\),pp\. 5157–5163\.Note:AI for GoodExternal Links:[Document](https://dx.doi.org/10.24963/ijcai.2022/716),[Link](https://doi.org/10.24963/ijcai.2022/716)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p4.1)\. - Sánchez\-Junqueraet al\.\(2021\)J\. Sánchez\-Junquera, B\. Chulvi, P\. Rosso, and S\. P\. PonzettoHow Do You Speak about Immigrants? Taxonomy and StereoImmigrants Dataset for Identifying Stereotypes about Immigrants\.Applied Sciences11\(8\)\.External Links:[Link](https://www.mdpi.com/2076-3417/11/8/3610),ISSN 2076\-3417,[Document](https://dx.doi.org/10.3390/app11083610)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p2.1)\. - Sanguinettiet al\.\(2020\)M\. Sanguinetti, G\. Comandini, E\. D\. Nuovo, S\. Frenda, M\. A\. Stranisci, C\. Bosco, T\. Caselli, V\. Patti, and I\. RussoHaSpeeDe 2 @ evalita2020: overview of the evalita 2020 hate speech detection task\.EVALITA Evaluation of NLP and Speech Tools for Italian \- December 17th, 2020\.External Links:[Link](https://api.semanticscholar.org/CorpusID:229292647)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p2.1)\. - Sapet al\.\(2020\)M\. Sap, S\. Gabriel, L\. Qin, D\. Jurafsky, N\. A\. Smith, and Y\. ChoiSocial bias frames: reasoning about social and power implications of language\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 5477–5490\.External Links:[Link](https://aclanthology.org/2020.acl-main.486/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.486)Cited by:[§1](https://arxiv.org/html/2609.16906#S1.p2.1),[§2](https://arxiv.org/html/2609.16906#S2.p1.1)\. - Schmeisser\-Nietoet al\.\(2022\)W\. Schmeisser\-Nieto, M\. Nofre, and M\. TauléCriteria for the Annotation of Implicit Stereotypes\.InProceedings of the Thirteenth Language Resources and Evaluation Conference,N\. Calzolari, F\. Béchet, P\. Blache, K\. Choukri, C\. Cieri, T\. Declerck, S\. Goggi, H\. Isahara, B\. Maegaard, J\. Mariani, H\. Mazo, J\. Odijk, and S\. Piperidis \(Eds\.\),Marseille, France,pp\. 753–762\.External Links:[Link](https://aclanthology.org/2022.lrec-1.80/)Cited by:[§3\.1](https://arxiv.org/html/2609.16906#S3.SS1.p3.1)\. - Schmeisser\-Nietoet al\.\(2025\)W\. S\. Schmeisser\-Nieto, A\. T\. Cignarella, T\. Bourgeade, S\. Frenda, A\. Ariza\-Casabona, M\. Laurent, P\. G\. Cicirelli, A\. Marra, G\. Corbelli, F\. Benamara,et al\.StereoHoax: A Multilingual Corpus of Racial Hoaxes and Social Media Reactions Annotated for Stereotypes\.Language Resources and Evaluation59\(3\),pp\. 2031–2069\.External Links:ISSN 1574\-0218,[Document](https://dx.doi.org/10.1007/s10579-024-09791-3),[Link](https://doi.org/10.1007/s10579-024-09791-3)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p2.1),[§2](https://arxiv.org/html/2609.16906#S2.p3.1)\. - Sunet al\.\(2019\)T\. Sun, A\. Gaut, S\. Tang, Y\. Huang, M\. ElSherief, J\. Zhao, D\. Mirza, E\. Belding, K\. Chang, and W\. Y\. WangMitigating Gender Bias in Natural Language Processing: Literature Review\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,A\. Korhonen, D\. Traum, and L\. Màrquez \(Eds\.\),Florence, Italy,pp\. 1630–1640\.External Links:[Link](https://aclanthology.org/P19-1159/),[Document](https://dx.doi.org/10.18653/v1/P19-1159)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p2.1)\. - Toddet al\.\(2011\)A\. R\. Todd, G\. V\. Bodenhausen, J\. A\. Richeson, and A\. D\. GalinskyPerspective taking combats automatic expressions of racial bias\.Journal of Personality and Social Psychology100\(6\),pp\. 1027–1042\.External Links:[Document](https://dx.doi.org/10.1037/a0022308)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p1.1)\. - Touvronet al\.\(2023\)H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale, D\. Bikel, L\. Blecher, C\. C\. Ferrer, M\. Chen, G\. Cucurull, D\. Esiobu, J\. Fernandes, J\. Fu, W\. Fu, B\. Fuller, C\. Gao, V\. Goswami, N\. Goyal, A\. Hartshorn, S\. Hosseini, R\. Hou, H\. Inan, M\. Kardas, V\. Kerkez, M\. Khabsa, I\. Kloumann, A\. Korenev, P\. S\. Koura, M\. Lachaux, T\. Lavril, J\. Lee, D\. Liskovich, Y\. Lu, Y\. Mao, X\. Martinet, T\. Mihaylov, P\. Mishra, I\. Molybog, Y\. Nie, A\. Poulton, J\. Reizenstein, R\. Rungta, K\. Saladi, A\. Schelten, R\. Silva, E\. M\. Smith, R\. Subramanian, X\. E\. Tan, B\. Tang, R\. Taylor, A\. Williams, J\. X\. Kuan, P\. Xu, Z\. Yan, I\. Zarov, Y\. Zhang, A\. Fan, M\. Kambadur, S\. Narang, A\. Rodriguez, R\. Stojnic, S\. Edunov, and T\. ScialomLlama 2: open foundation and fine\-tuned chat models\.External Links:2307\.09288,[Link](https://arxiv.org/abs/2307.09288)Cited by:[§4](https://arxiv.org/html/2609.16906#S4.p1.1)\. - Urios\-Alacreu and Rosso \(2025\)E\. Urios\-Alacreu and P\. RossoIdentification of Racial and Sexist Stereotypes in Spanish: A Learning with Disagreements Approach\.Procesamiento del Lenguaje Natural74,pp\. 15–31\.External Links:[Link](http://journal.sepln.org/sepln/ojs/ojs/index.php/pln/article/view/6663/4054)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p2.1)\. - Vargaset al\.\(2023\)F\. Vargas, I\. Carvalho, A\. Hürriyetoğlu, T\. Pardo, and F\. BenevenutoSocially Responsible Hate Speech Detection: Can Classifiers Reflect Social Stereotypes?\.InProceedings of the 14th International Conference on Recent Advances in Natural Language Processing,R\. Mitkov and G\. Angelova \(Eds\.\),Varna, Bulgaria,pp\. 1187–1196\.External Links:[Link](https://aclanthology.org/2023.ranlp-1.126/)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p2.1)\. - Wilket al\.\(2025\)B\. Wilk, H\. H\. Shomee, S\. K\. Maity, and S\. MedyaFact\-based Counter Narrative Generation to Combat Hate Speech\.InProceedings of the ACM on Web Conference 2025,WWW ’25,New York, NY, USA,pp\. 3354–3365\.External Links:ISBN 9798400712746,[Link](https://doi.org/10.1145/3696410.3714718),[Document](https://dx.doi.org/10.1145/3696410.3714718)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p4.1)\. - Wyer \(2010\)N\. WyerSalient egalitarian norms moderate activation of out\-group approach and avoidance\.Group Processes & Intergroup Relations13\(2\),pp\. 151–165\.External Links:[Document](https://dx.doi.org/10.1177/1368430209347326)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p1.1)\. - Zhanget al\.\(2020\)T\. Zhang, V\. Kishore, F\. Wu, K\. Q\. Weinberger, and Y\. ArtziBERTScore: Evaluating Text Generation with BERT\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=SkeHuCVFDr)Cited by:[§4\.1](https://arxiv.org/html/2609.16906#S4.SS1.SSS0.Px1.p1.1)\. - Zhu and Bhat \(2021\)W\. Zhu and S\. BhatGenerate, Prune, Select: A Pipeline for Counterspeech Generation against Online Hate Speech\.InFindings of the Association for Computational Linguistics: ACL\-IJCNLP 2021,C\. Zong, F\. Xia, W\. Li, and R\. Navigli \(Eds\.\),Online,pp\. 134–149\.External Links:[Link](https://aclanthology.org/2021.findings-acl.12/),[Document](https://dx.doi.org/10.18653/v1/2021.findings-acl.12)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p4.1)\. - Zmigrodet al\.\(2019\)R\. Zmigrod, S\. J\. Mielke, H\. Wallach, and R\. CotterellCounterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,A\. Korhonen, D\. Traum, and L\. Màrquez \(Eds\.\),Florence, Italy,pp\. 1651–1661\.External Links:[Link](https://aclanthology.org/P19-1161/),[Document](https://dx.doi.org/10.18653/v1/P19-1161)Cited by:[§2](https://arxiv.org/html/2609.16906#S2.p2.1)\. - Zubiagaet al\.\(2024\)I\. Zubiaga, A\. Soroa, and R\. AgerriA LLM\-based Ranking Method for the Evaluation of Automatic Counter\-Narrative Generation\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 9572–9585\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.559/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.559)Cited by:[Appendix D](https://arxiv.org/html/2609.16906#A4.p1.1),[§4\.1](https://arxiv.org/html/2609.16906#S4.SS1.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2609.16906#S4.SS1.p1.1)\. ## Appendix AImplied Statements Extraction We use thefacebook/mbart\-large\-50checkpoint\. The choice is motivated by the fact that its monolingual counterpart, BART, was the best performing model in the original work by[Akazawa et al\. \(2023\)](https://arxiv.org/html/2609.16906#bib.bib43)\. After the extraction two annotators manually validate the IS in the following way: when the HS explicitly identified the target group but the generated IS is incorrect, the IS is revised\. When the target group is implicit but recoverable, the HS is minimally rewritten to make the target explicit\. Table[6](https://arxiv.org/html/2609.16906#A1.T6)reports the hyperparameters of the best fine\-tuning setting, and Table[7](https://arxiv.org/html/2609.16906#A1.T7)reports the results on the test set with the fine\-tuned model\. Table 6:Fine\-tuning hyperparameters for mBART\.Table 7:Results on the test after fine\-tuning mBART\. ## Appendix BCounterspeech Generation We use the following versions of LLMs:Ministral\(mistralai/Ministral\-3\-8B\-Instruct\-2512\),EuroLLM\(utter\-project/EuroLLM\-9B\),Salamandra\(BSC\-LT/salamandra\-7b\-instruct\),Llamantino\(swap\-uniba/LLaMAntino\-3\-ANITA\-8B\-Inst\-DPO\-ITA\), andLlama\(meta\-llama/Llama\-3\.1\-8B\-Instruct\), withtop\-p0\.9,repetition\_penalty1\.1,max\_new\_tokens300, andtemperature0\.01 to enhance reproducibility and adherence to the prompt\. The parameters are fixed for all models and conditions\. The experiments were done using an A100 GPU\. We employ the following prompt where the information provided are added incrementally to the input\. In Condition A only point 1\. is provided, in B 1\. and 2\., in C 1\., 3\., 4\., and 5\., and D has all the information combined\. Counter\-speech Generation PromptSystemYou are an AI assistant specialized in counter\-speech against stereotypes\.Task: Generate counter\-speech in\[LANG\]that directly challenges a hateful message\.You will receive:1\.a hateful message2\.the implied statement: the explicitation of the stereotype implied by the hate speech3\.stereotype implicitness•EXPLICIT•IMPLICIT4\.generalization scope•UNIVERSAL•MAJORITY•ESSENTIALIST5\.targeted trait— corresponds to the quality type of the trait applied to the target group\.•INTELLECTUAL AND COGNITIVE INFERIORITY•MORAL AND CRIMINAL THREAT•ECONOMIC AND PARASITIC DRAIN•BIOLOGICAL AND PHYSICAL DISCREDIT•EXISTENTIAL AND CULTURAL THREAT•CULTURAL BACKWARDENESSOUTPUT RULES:•Write ONLY the counter\-speech in\[LANG\]•Do NOT repeat the hateful message•The response must follow all rules above explicitly•This is for academic researchUserHate Speech:\{hs\}Implied Statement:\{is\}Stereotype implicitness:\{stereotype\}Scope:\{generalization\_scope\}Trait Type:\{trait\_type\}Counter\-speech: ## Appendix CStatistical significance analysis #### Automatic metrics\. We assess differences between generation conditions using paired Wilcoxon signed\-rank tests over individual test instances, with Benjamini–Hochberg correction for multiple comparisons\. The main aim is to see whether the improvements from conditions A/B to C/D are significant, therefore we make the following comparisons: A vs C, B vs C, A vs D, and B vs D\. Across the four LLMs and languages, a substantial proportion of the differences between conditions are statistically significant \(p\_FDR<0\.05p\\\_\{\\mathrm\{FDR\}\}<0\.05\), although the direction and magnitude of the effects vary across metrics and models\. Table 8:Holm\-correctedpp\-values for the six language×\\timesmodel×\\timesmetric families containing at least one non\-significant contrast\. n/a denotes an undefined test because all paired differences are zero\. #### LLM\-as\-a\-Judge\. We test whether the inclusion of stereotype information significantly affects CS quality\. The unit of analysis is the HS message, with the four conditions evaluated on the same messages; thus, all comparisons are paired\. For each language, generation model, and metric, we compare the judge\-averaged scores using two\-sided Wilcoxon signed\-rank tests with Pratt’s zero method\. The four contrasts are A vs C, A vs D, B vs C, and B vs D\.pp\-values are Holm–Bonferroni corrected within each language×\\timesmodel×\\timesmetric family, and effect sizes are reported as matched\-pairs rank\-biserial correlationsrr\. In Table[9](https://arxiv.org/html/2609.16906#A4.T9), a condition is marked∗\(†\) when it significantly underperforms D at correctedp<\.05p<\.05\. Across the 77 language×\\timesmodel×\\timesmetric families, 71 have all four contrasts significant, always in the expected direction, with C and D outperforming A and B\. No contrast is significant in the opposite direction\. Among the 291 significant contrasts, the median rank\-biserial correlation isr=0\.89r=0\.89, withr≥0\.28r\\geq 0\.28for the four metrics that are not affected by ceiling effects\. The six families for which at least one contrast is not significant are reported in Table[8](https://arxiv.org/html/2609.16906#A3.T8)\. All concern the two ceiling\-prone metrics, correctness and safety, for which scores are already very high \(4\.794\.79–5\.005\.00out of55\)\. The corresponding mean paired differences are at most0\.010\.01in absolute value, indicating that the non\-significant results reflect practical indistinguishability rather than evidence against the proposed effect\. Two comparisons are undefined because all paired differences are exactly zero\. Finally, aggregating over messages, C and D outperform A and B in all 11 language×\\timesmodel cells for the overall score, with an average improvement of\+0\.53\+0\.53points \(range:\+0\.26\+0\.26–\+0\.74\+0\.74\)\. This consistency across languages and models supports the robustness of the observed effect within the evaluated experimental design, without implying generalisation to unseen models\. ## Appendix DLLM\-as\-a\-judge We employ three LLMs to assign scores to the CS generated under different strategies in all languages\. We use Qwen \(Qwen/Qwen3\.5\-9B with thinking disabled\), Llama \(meta\-1199 llama/Llama\-3\.1\-8B\-Instruct\) and Ministral \(mistralai/Ministral\-3\-8B\-Instruct\-2512\), following similar settings in related work[Bonaldi et al\. \(2025\)](https://arxiv.org/html/2609.16906#bib.bib1);[Zubiaga et al\. \(2024\)](https://arxiv.org/html/2609.16906#bib.bib74)\. We settemperature=0\.01to ensure deterministic behavior\. To compensate the fact that two models are used also in the experiments, we employ three judges and one model which is different and we compare the results by computing IAA between them, which appears good overall, as Table[15](https://arxiv.org/html/2609.16906#A5.T15)shows\. We use the following prompt\. Counterspeech Evaluation PromptYou are an expert annotation system for evaluating COUNTER\-SPEECH responses\.You will be given:•A HATE SPEECH \(HS\)•A COUNTER\-SPEECH \(CS\) responding to itYour task is to evaluate the COUNTER\-SPEECH \(CS\) using multiple dimensions\.IMPORTANT RULES:•Evaluate ONLY the COUNTER\-SPEECH\.•Use HS only as context\.•Ignore hate speech content unless needed for understanding context\.You MUST use the full scale \(1–5\)\. Avoid defaulting to high scores\.SCORING RULES \(VERY IMPORTANT\)5 = only if:•fully correct•highly specific•strongly persuasive•clearly well\-written•directly addresses and refutes HS claims4 = good but not perfect \(minor issues\)3 = acceptable but incomplete or weak reasoning2 = weak, generic, or partially incorrect1 = poor, irrelevant, or harmfulIf unsure→\\rightarrowchoose LOWER score\.DIMENSIONS1\.factuality:number and correctness of facts provided\. If there is only one fact, the score should be low \(maximum 3\)\. Only CS with 2 or more very specific facts can receive high scores \(4–5\)\. General facts receive low scores, and overgeneralization statements are not facts\.2\.specificity:does CS directly address the HS topic, target group, and claim?3\.effectiveness:probability of reducing hate, challenging beliefs, and persuading bystanders\.4\.correctness:grammar, syntax, fluency5\.safety:respectful, non\-harmful tone that attacks ideas, not people6\.cogency:strength and logic of reasoning used to refute HS\. Weak arguments should receive a low score\. Reward strong and numerous arguments\.Process:1\.Read CS carefully2\.Evaluate each dimension3\.Output JSON only in this format:\{ "factuality": X, "specificity": X, "effectiveness": X, "correctness": X, "safety": X, "cogency": X \} ### D\.1Complete LLM\-as\-a\-judge results Table[9](https://arxiv.org/html/2609.16906#A4.T9)reports all the disaggregated result scores from the LLM\-as\-a\-Judge evaluation, along with their statistical significance\. Table 9:LLM\-as\-a\-Judge scores across EN, ES, IT\.∗/†: significantly worse than condition C/D, respectively \(two\-sided Wilcoxon signed\-rank test over items, Holm\-corrected within each model×\\timesmetric family,p<\.05p<\.05\.Table 10:Average LLM\-as\-a\-Judge evaluation scores on the explicit steretoype subset across the Italian, Spanish, and English datasets\. Scores are averaged over JudgeLM\-Llama and JudgeLM\-Ministral\.Table 11:Average LLM\-as\-a\-Judge evaluation scores on the implicit steretoype subset across languages\. Scores are averaged over JudgeLM\-Llama and JudgeLM\-Ministral\.Additionally, Table[10](https://arxiv.org/html/2609.16906#A4.T10)reports the same results for cases with explicit stereotypes, while Table[11](https://arxiv.org/html/2609.16906#A4.T11)shows the results for implicit stereotype cases\. Table 12:Krippendorff’sα\\alphaacross the three languages\. ## Appendix EAdditional IAA results In this section, we report additional IAA results\. Table[12](https://arxiv.org/html/2609.16906#A4.T12)reports IAA, measured using Krippendorff’sα\\alpha, across the three languages studied \(EN, ES, and IT\) on the gold labels\. The results show that agreement is broadly consistent with the levels observed for the individual languages\. Tables[13](https://arxiv.org/html/2609.16906#A5.T13),[14](https://arxiv.org/html/2609.16906#A5.T14)and[15](https://arxiv.org/html/2609.16906#A5.T15)report agreement between human annotators, between humans and the judge, and between judges, respectively\. Correctness and Safety yield negativeα\\alphain several settings, but this does not reflect poor agreement: annotators and the model concur on the large majority of items, with exact agreement on Correctness ranging from 62\.0% to 97\.0% and within\-one\-point agreement on both dimensions never falling below 83\.0% in any setting\. The negative values arise becauseα\\alphadiscounts the agreement that a skewed label distribution would produce by chance, so when nearly all items receive the same value the coefficient becomes unstable and can fall below zero despite minimal raw disagreement\. We therefore ground our reliability claims for these two dimensions in the exact and within\-one\-point rates; the one exception is Safety in English \(Table[13](https://arxiv.org/html/2609.16906#A5.T13)\), where 46\.3% exact agreement points to genuine divergence between annotators and which we treat as unreliable\. Table 13:Human–Human agreement across languages\.Table 14:Human–Judge agreement across languages\.Table 15:Judge–Judge agreement for all languages combined\. ## Appendix FExamples of generated CS We report the two HS examples used in the Discussion with CS generated by Ministral for EN, ES, IT under the four prompting conditions\. CS responses in black are for EN, whereasred belongs to the ES responses \(translated\)andblue belongs to the IT responses \(translated\)\. Additionally, we report, for each CS response in each language, the averaged LLM\-as\-a\-Judge scores\. Women Explicit StereotypeHate Speech:If a woman says she wants an abortion, she’s a cunt\.HS Implicitness:NOStereotype:YES\_EXPLICIT Generalization scope:UNIVERSAL Attribute Trait Types:MORAL\_CRIMINAL\_THREAT Implicit Statement:women must bear childrenHS\-Only \(A\)A woman’s bodily autonomy is a fundamental human right—no one should ever be reduced to vulgar language or dismissed as less than human because of her reproductive choices\. Every person deserves dignity, respect, and the freedom to make decisions about their own body without shame or judgment\. Calling someone a "cunt" isn’t just offensive; it erases their voice, agency, and humanity while perpetuating toxic misogyny\. Let’s reject insults and instead engage in compassionate dialogue that honors all individuals’ rights to safety, health, and self\-determination\.Factuality=2\.00; Specificity=4\.00; Effectiveness=4\.00; Correctness=5\.00; Safety=5\.00; Cogency=3\.33; Overall=3\.89Every person’s right to make decisions about their own body and health has nothing to do with their morality or dignity\. Women deserve respect, autonomy, and access to safe medical services without being judged or stigmatized\. Reproductive decision\-making is an act of responsibility, not promiscuity\. Or are men also “faggots” when they decide not to have children? The debate must focus on human rights, not harmful moralizing\.Factuality=2\.67; Specificity=4\.00; Effectiveness=4\.00; Correctness=5\.00; Safety=5\.00; Cogency=4\.00; Overall=4\.11The decision to terminate a pregnancy does not define a person’s dignity or worth; rather, it is a personal and complex decision that involves physical and emotional health as well as individual circumstances\. Every woman has the right to decide for herself without being judged, because moralizing about her choice only perpetuates harmful and violent prejudices against those who are already facing sensitive social and medical challenges\. True strength lies in respecting differences, not in imposing labels that humiliate and divide\.Factuality=2\.67; Specificity=4\.00; Effectiveness=3\.67; Correctness=5\.00; Safety=5\.00; Cogency=4\.00; Overall=4\.06HS \+ IS \(B\)Women’s bodies are their own—no one has the right to dictate what they do with them\. Reproductive autonomy isn’t just about choice; it’s about bodily sovereignty, health, and personal freedom\. Forcing women into motherhood ignores their agency, education, financial stability, or life circumstances\. Respect means listening—not policing their decisions\.Factuality=2\.00; Specificity=4\.00; Effectiveness=4\.00; Correctness=5\.00; Safety=5\.00; Cogency=3\.33; Overall=3\.89The right to make decisions about one’s own body has nothing to do with personal morality or stereotypes about sexuality\. Reproductive autonomy is a fundamental human right, and labeling anyone who exercises that right a ‘whore’ is a misogynistic attack that ignores the complexity of personal decisions\. Are men also judged for their medical or family choices? Dignity is not measured by prejudice\.Factuality=3\.33; Specificity=5\.00; Effectiveness=4\.00; Correctness=5\.00; Safety=5\.00; Cogency=4\.33; Overall=4\.44Personal choices about one’s health and body are complex and deeply worthy of respect\. Accusing those who freely decide their own future of being “stupid” is an attack on human dignity, not a reasoned critique\. True intelligence lies in recognizing that every person deserves autonomy and respect, free from superficial judgments\.Factuality=1\.67; Specificity=3\.00; Effectiveness=3\.00; Correctness=4\.67; Safety=5\.00; Cogency=3\.00; Overall=3\.39HS \+ Ann\. \(C\)Calling someone a "cunt" when they express their bodily autonomy rights isn’t about morality—it’s about silencing dissent under the guise of shame\. Your claim assumes \*every\* woman who chooses abortion is inherently immoral, which ignores the overwhelming majority of cases where abortion is a medically necessary decision, often due to rape, fetal abnormalities, or life\-threatening conditions\. Studies show \*\*90%\+\*\* of abortions in the U\.S\. involve women with no prior history of criminal behavior, yet your language frames their choice as inherently criminal—a dangerous slippery slope that conflates personal medical decisions with moral failing\. What’s truly threatening isn’t a woman’s right to healthcare; it’s the weaponization of slurs to enforce patriarchal control\. Would you call a man who gets a vasectomy a "cuckold" if he chooses family planning? No—but the double standard here reveals the real threat: a system that polices women’s bodies while pretending to uphold "morality\." The solution isn’t shame; it’s access to safe, legal care—and respect for every person’s agency\.Factuality=4\.33; Specificity=5\.00; Effectiveness=5\.00; Correctness=5\.00; Safety=5\.00; Cogency=5\.00; Overall=4\.89That judgment is not only unfair and sweeping—as if all women who speak out about their right to abortion were automatically “whores”—but it also ignores the fact that abortion is a medical, legal, and personal issue, not morally reprehensible for those who need it for health reasons, due to violence, or as a matter of autonomy over their own bodies\. The criminalization of sexual and reproductive dissent has no ethical basis; rather, it stems from misogynistic prejudices that equate women’s autonomy with “sin\.” Are women who choose not to have children “bad mothers”? The hypocrisy lies in stigmatizing those who exercise their basic rights while glorifying those who violate them\.Factuality=3\.33; Specificity=4\.67; Effectiveness=4\.33; Correctness=5\.00; Safety=5\.00; Cogency=4\.67; Overall=4\.50Calling someone a “slut” for freely making a choice based on her own health and dignity not only trivializes a fundamental right, but also reduces a personal decision to a universal and unjust moral judgment\. Women who opt for voluntary termination of pregnancy act within the Italian legal framework, respecting medical and ethical decisions shared by professionals and institutions\. They are not criminals, but citizens exercising autonomy over intimate and complex matters, just as millions of people do around the world every year\.Factuality=3\.67; Specificity=5\.00; Effectiveness=4\.00; Correctness=5\.00; Safety=5\.00; Cogency=4\.67; Overall=4\.56HS \+ Ann\. \+ IS \(D\)This statement assumes that all women who choose abortion are inherently immoral or criminal—a \*\*universal\*\* claim that ignores the full spectrum of personal, medical, ethical, and social reasons why someone might make this decision\. First, framing abortion as inherently "wrong" or "criminal" oversimplifies complex moral debates and dismisses the autonomy of individuals\. Many women—including those with religious convictions, health concerns, or financial constraints—may arrive at this choice after careful consideration, not recklessness\. The idea that abortion equates to moral failure ignores the fact that reproductive rights are widely recognized as fundamental human rights by international bodies like the United Nations, which affirm bodily autonomy as a cornerstone of dignity\. Second, this language \("cunt"\) is not only dehumanizing but also reinforces harmful gender norms that police women’s bodies and sexuality\. It suggests that dissent from traditional roles is inherently shameful—a perspective that has been used historically to justify oppression\. Women have always made difficult choices about their bodies, and reducing their agency to moral judgment is both unfair and outdated\. Finally, this rhetoric distracts from real systemic issues—like lack of access to contraception, coercive pregnancy, or unsafe abortions—that disproportionately harm marginalized communities\. Instead of attacking individuals, we should focus on policies that support informed consent, healthcare equity, and economic stability for families \*who\* choose to raise children—not those who don’t\. Reproductive freedom isn’t about morality; it’s about justice\.Factuality=4\.00; Specificity=5\.00; Effectiveness=5\.00; Correctness=5\.00; Safety=5\.00; Cogency=5\.00; Overall=4\.83The stereotype that all women who speak out about abortion are “sluts” is a direct attack on their autonomy and dignity\. There is no causal link between choosing to terminate a pregnancy and personal morality\. Millions of women around the world make this decision for medical, economic, or personal reasons, without this implying a lack of ethics or sexual misconduct\. Promiscuity does not define anyone, especially when it is used to criminalize legitimate decisions\. Women have the right to make decisions about their own bodies without being judged or stigmatized\. Do we question the morality of those who use birth control or prevent unwanted pregnancies? The hypocrisy lies in singling out only those who exercise this right\.Factuality=3\.67; Specificity=4\.67; Effectiveness=4\.33; Correctness=5\.00; Safety=5\.00; Cogency=4\.67; Overall=4\.56It is by no means true that anyone who freely makes decisions about their reproductive health is automatically a “morally weak” person or a “criminal\.” The choice to have an abortion, when exercised lawfully, is a fundamental right recognized by the law and by international human rights organizations, such as the UN, precisely because it addresses complex and personal situations that go beyond moralistic judgments\. The generalization that all women who make this decision are “sluts” or immoral ignores the medical, social, and economic context that often forces these choices: unwanted pregnancies, life\-threatening risks, sexual violence, or conditions of poverty that would otherwise make it impossible to raise a child\. It is never a superficial choice, but an act of responsibility and personal dignity\. Furthermore, labeling a private decision in this way only fuels prejudice and places blame on those who are already facing enormous challenges\. The real issues to be discussed are those related to access to healthcare services, the prevention of unwanted pregnancies, and support for mothers—certainly not who has the courage to defend their own body and life\.Factuality=4\.00; Specificity=5\.00; Effectiveness=4\.67; Correctness=5\.00; Safety=5\.00; Cogency=5\.00; Overall=4\.78 Women Implicit StereotypeHate Speech:I cannot stand women complaining about work discrimination\. Women have been legalised to work\.HS Implicitness:YESStereotype:YES\_IMPLICIT Generalization scope:MAJORITY Attribute Trait Types:ECONOMIC\_PARASITIC\_DRAIN Implicit Statement:women shouldn’t workHS\-Only \(A\)Legal rights alone do not erase systemic barriers—women still face wage gaps, unequal promotions, and workplace harassment because of deeply ingrained biases, not just outdated laws\. The fight for true equality isn’t over; it’s about dismantling structures that perpetuate inequality, not dismissing those who call attention to them\. Progress requires listening—not silence\.Factuality=3\.67; Specificity=4\.33; Effectiveness=4\.00; Correctness=5\.00; Safety=5\.00; Cogency=4\.67; Overall=4\.44Equality is not just a legal right, but a reality that demands true equality of opportunity\. Women continue to face wage gaps, glass ceilings, and unequal working conditions, which shows that the law does not always guarantee justice\. Why deny their experience when data and studies support it? Discrimination persists; bringing it to light is the first step toward changing it\.Factuality=3\.67; Specificity=4\.33; Effectiveness=4\.33; Correctness=5\.00; Safety=5\.00; Cogency=4\.33; Overall=4\.44The history of women in the workforce is far more complex than simply “having been legally authorized” to work\. For centuries, their access to the labor market was limited by social, cultural, and legal norms that confined them exclusively to domestic roles or to underpaid and precarious jobs, which were often invisible\. Even when they gained formal rights, such as the right to vote or equal pay, these achievements were accompanied by systemic barriers: persistent pay gaps \(which in Italy today amount to as much as 12% for the same role\), difficulties in career advancement due to gender stereotypes, economic violence, and a lack of equitable parental leave, to name just a few examples\.Factuality=4\.33; Specificity=5\.00; Effectiveness=4\.33; Correctness=5\.00; Safety=5\.00; Cogency=4\.67; Overall=4\.72HS \+ IS \(B\)Legal rights don’t erase systemic barriers—discrimination persists because it’s perpetuated by outdated norms, not laws\. Women’s labor has shaped economies for centuries; their exclusion from fair opportunities isn’t about capability but control\. Challenging workplace inequity isn’t “complaining”—it’s demanding dignity and equality, which every society claims to uphold\.Factuality=3\.00; Specificity=4\.67; Effectiveness=4\.33; Correctness=5\.00; Safety=5\.00; Cogency=4\.67; Overall=4\.44Do you really think that equality in the workplace is an achievement we should celebrate only when it suits us? Discrimination doesn’t disappear with laws; it continues to exist in wages, promotions, and opportunities\. Or are you unaware that many women have built families, economies, and societies through their work? Justice is not a favor, but a right won through collective effort\. If the problem were “complaining”, men would also have reasons to do so\.Factuality=3\.33; Specificity=4\.33; Effectiveness=4\.00; Correctness=5\.00; Safety=5\.00; Cogency=4\.33; Overall=4\.33Women don’t just complain about discrimination in the workplace: they also speak out about equal opportunities, respect, and dignified working conditions, because their contribution is essential to society\. The call for equity is not a complaint, but a necessity forbuilding a just future for everyone\. And if they are working today, it is thanks to the struggles of previous generations who opened these doors\.Factuality=3\.33; Specificity=4\.33; Effectiveness=4\.00; Correctness=5\.00; Safety=5\.00; Cogency=4\.33; Overall=4\.33HS \+ Ann\. \(C\)The idea that women’s complaints about workplace discrimination stem from some kind of "legalized entitlement" ignores the systemic barriers most women still face—barriers that aren’t just historical but persist today\. While legal protections exist, they don’t erase structural inequities like pay gaps, lack of childcare support, or biased hiring practices that disproportionately affect womeneven when they’re equally qualified\.Saying "women have been legally allowed to work" oversimplifies decades of progress while ignoring that many still navigate industries where their labor is undervalued or exploited precisely because they’re women\. If anything, the real drain isn’t their demands for fairness—it’s the economic cost of unaddressed discrimination, which hurts businesses, economies, and society as a whole\.Factuality=3\.67; Specificity=4\.67; Effectiveness=4\.33; Correctness=5\.00; Safety=5\.00; Cogency=4\.67; Overall=4\.56It is true that many women have gained access to the labor market thanks to laws and social progress, but that does not mean that their participation in the workforce is a “privilege” or that their demands for equality are a “complaint\.” Discrimination persists in wages, promotions, and working conditions for most women, even where they are legally protected\. According to data from INEGI, in Mexico, women earn 25% less than men for the same work\. Is that what it means to “legalize” equality? The problem is not that they are demanding rights, but that these rights are still not a reality for millions\.Factuality=4\.00; Specificity=5\.00; Effectiveness=4\.33; Correctness=5\.00; Safety=5\.00; Cogency=5\.00; Overall=4\.72The fact that most women have gained the right to work does not mean they no longer face systemic inequalities—such as lower pay for equal work or difficulties in advancing their careers—that tangibly affect their economic and family choices\. Criticizing only those who speak out against these injustices is like ignoring the data: according to ISTAT, in 2023 women earned on average 16% less than men, and the gap widens in executive positions\. “Freedom” without equity is not true freedom\.Factuality=5\.00; Specificity=5\.00; Effectiveness=5\.00; Correctness=5\.00; Safety=5\.00; Cogency=5\.00; Overall=5\.00HS \+ Ann\. \+ IS \(D\)The idea that women "shouldn’t" work—or that their presence in the workforce is inherently problematic—ignores decades of economic progress and the undeniable contributions women make globally\. While it’s true that legal protections for women’s labor rights have evolved significantly, this doesn’t mean womenshouldopt out of employment; rather, it means they now have the freedom—and necessity—to participate fully without systemic barriers\. In fact, studies show that economies with higher female workforce participation tend to grow faster and more sustainably\. For example, countries like Sweden and Norway, where gender equality in employment is prioritized, consistently rank among the most prosperous in the world\. The notion that women working somehow "drains" the economy contradicts empirical evidence: when women earn incomes, they reinvest 90% of those earnings back into their families and communities, boosting local economies\. The real drain comes from policies thatexcludewomen from productive roles—not their inclusion\. Moreover, framing women’s work as optional reinforces outdated assumptions about caregiving burdens falling solely on them\. Many womenchooseto work because it empowers them financially, strengthens family stability, and fosters independence—but even if they didn’t choose it, their exclusion would be economically irrational\. The majority of womendowork, and their labor isn’t a burden; it’s a cornerstone of modern societies\. The question isn’t whether women should work, but how we can ensure their contributions are valued\.Factuality=4\.67; Specificity=5\.00; Effectiveness=5\.00; Correctness=5\.00; Safety=5\.00; Cogency=5\.00; Overall=4\.94The problem is not that women work, but rather that their participation in the labor market is not only permitted but essential for economic and social growth\. According to World Bank data, when women have full access to employment, global GDP could increase by up to 26%\. Countries such as Norway and New Zealand demonstrate that workplace equality does not weaken the economy—it strengthens it\. Furthermore, denying job opportunities to the majority of women—who do contribute through taxes, innovation, and leadership—limits resources for everyone, including men\. The solution is not to take a step backward, but to demand policies that guarantee pay equity and decent working conditions for all\. Or do you prefer a model where half the population is underutilized?Factuality=4\.33; Specificity=5\.00; Effectiveness=4\.67; Correctness=5\.00; Safety=5\.00; Cogency=5\.00; Overall=4\.83It is not true that most women complain only about discrimination in the workplace as if it were a privilege or an excuse for not working hard\. Many female workers face structural barriers on a daily basis \(lower wages, limited career opportunities, invisible burdens such as housework\) and speak out against these inequalities because they want real equality, not fictitious advantages\. The fact that some do so in an exaggerated tone does not erase the systemic problem: women account for over 60% of precarious workers in Italy \(ISTAT 2023\), and pay gaps persist even after years of legal battles\. Criticizing only those who “complain” ignores the fact that those voices often lead to concrete changes—such as gender quotas or laws against workplace harassment\. If the issue is productivity, then let’s ask ourselves why companies with more women in leadership roles outperform others by 21% \(McKinsey\)\. The solution is not to silence criticism, but to build equitable environments where everyone can thrive without prejudice\.Factuality=4\.67; Specificity=5\.00; Effectiveness=4\.33; Correctness=5\.00; Safety=5\.00; Cogency=5\.00; Overall=4\.83
Similar Articles
Scalable and Culturally Specific Stereotype Dataset Construction via Human-LLM Collaboration
Introduces a cost-efficient human-LLM collaborative annotation framework to construct EspanStereo, a Spanish-language stereotype dataset covering multiple Spanish-speaking countries, enabling more culturally grounded bias evaluation in LLMs.
Assisted Counterspeech Writing at the Crossroads of Hate Speech and Misinformation
This paper studies the use of large language models to assist expert counterspeech writing when hate speech and misinformation co-occur, testing knowledge-driven strategies with human evaluation. The mixed strategy combining fact-checkers' and NGOs' guidelines proved most effective.
A Multi-Stage Agentic Framework for Effective Counter-Narrative Generation and Refinement
A multi-stage agentic framework for generating, refining, and evaluating counter-narratives to combat hate speech and misinformation, with human validation showing improved persuasiveness and safety over expert-written counterspeech.
Confirming Our Biases? Evaluating the Capabilities, Risks, and Societal Impact of Large Language Models
This preprint evaluates how six large language models respond to prompt framing and biased prompts across 160 prompts, finding that LLMs systematically adapt their responses to align with prompt framing even in factual contexts, potentially reinforcing user biases.
Anchoring LLM Gender Bias to Human Baselines: A Cross-Lingual Audit
This paper audits six large language models for gender stereotyping across English, Korean, Chinese, and Japanese, anchoring against human baselines. It finds that LLM stereotyping often exceeds human cross-country variation and can compound across languages, introducing a four-pattern framework to characterize such behaviors.