揭开宣传的伪装：掩码语言模型与因果语言模型的对比分析

# Unmasking Propaganda: A Comparative Analysis of Masked and Causal Language Models
Source: [https://arxiv.org/html/2610.03077](https://arxiv.org/html/2610.03077)
DOI:[XXXXXXX\.XXXXXXX](https://doi.org/XXXXXXX.XXXXXXX)Conference:Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NYISBN:978\-1\-4503\-XXXX\-X/2018/06CCS:Information systemsCCS:Information Storage and RetrievalCCS:Large language modelsClaudiu Creangaemail:[claudiu\.creanga@s\.unibuc\.ro](mailto:[email protected])Affiliation:Faculty of Mathematics and Informatics, Human Language Technologies Research Center, Interdisciplinary School of Doctoral Studies, University of Bucharest,RomaniaIoachim Lihoremail:[ioachim\.lihor@gmail\.com](mailto:[email protected])Affiliation:Faculty of Mathematics and Informatics, Human Language Technologies Research Center, Interdisciplinary School of Doctoral Studies, University of Bucharest,RomaniaandLiviu P\. Dinuemail:[liviu\.p\.dinu@gmail\.com](mailto:[email protected])Affiliation:Faculty of Mathematics and Informatics, Human Language Technologies Research Center, Interdisciplinary School of Doctoral Studies, University of Bucharest,Romania

2018

###### Abstract\.

Propaganda detection is an essential task in natural language processing \(NLP\), particularly in the context of manipulative political communications\. However, identifying specific propaganda techniques presents a significant challenge due to their often subtle nature and reliance on context, making them difficult to distinguish from legitimate persuasive language\. Propaganda often involves highlighting certain facts while downplaying or ignoring others to create a desired perception\. This biased communication aims to influence attitudes, beliefs, or behaviors towards a particular cause or position\. This paper explores advances in detecting propaganda techniques through a comparative analysis of modern language models, using the SemEval\-2020 Task 11 dataset\. We evaluated both masked language models \(based on XLM\-RoBERTa or DeBERTa V3\) and causal models \(from OpenAI, Google, Mistral, Anthropic and Meta\), employing two prompting strategies: base and chain\-of\-thought prompting\. Our results demonstrate improvements over state\-of\-the\-art models, with the best\-performing MLM achieving an F1 score of 63\.18 in technique classification and the best causal model achieving 63\.62\. We also observed that certain models excel in specific techniques, such as loaded language and name\-calling, while struggling with others like bandwagon and black\-and\-white fallacy\. These findings suggest that fine\-tuning, ensemble modeling, and the use of larger datasets can further enhance propaganda detection capabilities\.

###### Keywords:

LLM, NLP, Propaganda Detection, CoT, MLM, Prompt Engineering

## 1\.Introduction

Propaganda represents a systematic and deliberate operation extending far beyond mere persuasion, aiming instead to actively manipulate public opinion and shape collective perception through the stringent control of information\. This is achieved by strategically and selectively presenting narratives, meticulously highlighting facts, interpretations, or images that bolster a specific agenda, while simultaneously downplaying, omitting, distorting, or even fabricating information that contradicts it\. This biased communication constructs a distorted reality in the minds of the audience, ultimately seeking to influence their attitudes, solidify specific beliefs, and steer their behaviour towards a predetermined cause, ideology, or political position\.

Historically, propaganda has long served as an especially insidious and powerful instrument, particularly favored by authoritarian regimes seeking to maintain control \(\([Stanley, 2015](https://arxiv.org/html/2610.03077#bib.bib13)\)\)\. It proves invaluable for such powers, enabling them to manufacture consent, legitimize their grip on political authority \(often by exaggerating achievements or inventing threats\), and systematically stifle dissent by dominating the informational landscape and silencing or discrediting opposing viewpoints\. By disseminating biased information, selectively framing events, leveraging potent emotional appeals over factual discourse, and sometimes spreading outright disinformation, propaganda crafts this distorted reality\. This manufactured reality serves a dual purpose: it bolsters the authority and preferred narratives of the elite while simultaneously undermining the citizenry’s ability to critically evaluate information and challenge the status quo\.

To achieve these manipulative goals, propagandists employ a diverse and often sophisticated toolkit\. Techniques range from the overt use of emotionally charged or loaded language, derogatory name\-calling to demonize opponents, and the incessant repetition of slogans or messages until they are widely accepted, to more subtle tactics\. These can include the oversimplification of complex issues into misleading dichotomies, potent emotional appeals designed to bypass rational analysis, concerted efforts to sow doubt about credible alternative sources, gross exaggeration or minimization of events, and the clever exploitation of logical fallacies to create a superficial veneer of reasoned argument\. The corrosive effects of propaganda on society are multifaceted\. They lead to heightened polarization and social division, frequently employing ’us vs\. them’ tactics to create rifts within populations\. Furthermore, propaganda contributes to a decline in critical thinking as citizens are encouraged to passively accept official narratives rather than engage in independent analysis\. A particularly dangerous consequence is the dehumanization of designated out\-groups, which can be used to justify mistreatment or marginalization\.

This landscape of manipulation exists within a broader context of challenges to information integrity\. There is substantial evidence of a worldwide crisis in news trust, prompting significant debate on how to restore public confidence \(\([Flew et al\., 2020](https://arxiv.org/html/2610.03077#bib.bib7)\)\)\. This crisis is unfolding as news consumption patterns shift dramatically, with more people, particularly the young, increasingly obtaining news online, and many others choosing to avoid news altogether\. Consequently, within the complex and rapidly evolving digital information ecosystem—where information spreads virally and can be targeted with precision—the task of identifying, analyzing, and exposing propaganda has become fundamental for preserving informed public discourse and democratic health \(\([Da San Martino et al\., 2020b](https://arxiv.org/html/2610.03077#bib.bib6)\)\)\.

In this research, we used Masked Models and Causal Models to obtain good results in detecting propaganda techniques using the SemEval 2020 dataset\. We make our code publicly available on GitHub\.

## 2\.Related Work

Propaganda has not been as extensively studied in NLP as other topics because of the complexity of detection, lack of labeled data and subjectivity\. The first dataset that we are aware of is TSHP\-17 created by\([Rashkin et al\., 2017](https://arxiv.org/html/2610.03077#bib.bib11)\)which annotated documents in 4 classes: trusted, satire, hoax and propaganda\. Another dataset, Qprop\([Barrón\-Cedeño et al\., 2019](https://arxiv.org/html/2610.03077#bib.bib2)\), was released in 2019 and contained a binary imbalanced dataset with 10% of documents belonging to propaganda\. PTC\([Martino et al\., 2019](https://arxiv.org/html/2610.03077#bib.bib10)\)was released in the same year and contained specific text spans that were labelled according to 18 propaganda techniques\. In 2020, at SemEval task 11: Detection of propaganda techniques in news articles\([Da San Martino et al\., 2020a](https://arxiv.org/html/2610.03077#bib.bib5)\)a dataset was presented with 2 sub\-tasks: Span Identification \(SI\) and Technique Classification \(TC\)\. The Span Identification sub\-task focuses on locating specific segments of text that employ propaganda techniques\. Once these segments are identified, the Technique Classification sub\-task categorizes the particular propaganda methods used within those spans\. This approach of 2 sub\-tasks encourages the transformer\-based models to work in tandem, with the first model identifying the relevant text segments and the second model accurately classifying the propaganda techniques employed within them\. The best results obtained an F1 score of 62 and made use of a pre\-trained RoBERTa model\. This performance level suggests that accurately distinguishing between numerous, often subtly employed techniques remained a significant challenge for models of that generation, highlighting the potential for newer, larger language models and different methodologies like prompting to achieve further improvements\.

In 2023,\([Malik et al\., 2023](https://arxiv.org/html/2610.03077#bib.bib9)\)achieved state\-of\-the\-art performance on the Qprop dataset\. The authors employed a diverse range of models, including BERT, ELMo, LSA, FastText, and Word2Vec, obtaining an impressive F1 score of 97\.76\.

In\([Sprenkamp et al\., 2023](https://arxiv.org/html/2610.03077#bib.bib12)\), the authors used LLMs on the SemEval\-2020 task 11 dataset\. They combined multiple LLMs models such as: GPT\-3\.5 and GPT\-4, and also a fine\-tuned GPT\-3\.5 model, cross\-joined with two prompting techniques: few\-shot learning \(base\) and chain\-of\-thought learning \(CoT\)\. The obtained results were comparable to the ones obtained in the SemEval\-2020 task 11, with an F1 score of 58\.11 using GPT\-4 model with few\-shot\-learning\.

### 2\.1\.Contribution

We establish new state\-of\-the\-art performance on the SemEval\-2020 Task 11 Technique Classification subtask\. Our fine\-tuned DeBERTa V3 Large model achieves a macro F1\-score of 63\.18, surpassing the original SemEval winning score\. Furthermore, our experiments with Causal LLMs show even higher performance, with Gemini Flash 2\.5 reaching a macro F1\-score of 63\.62 using base prompting\.

## 3\.Data

We selected the SemEval\-2020 Task 11 corpus\([Da San Martino et al\., 2020a](https://arxiv.org/html/2610.03077#bib.bib5)\)due to its comprehensiveness\. This dataset offers a rich collection of examples, making it well\-suited for training and evaluating our models\. It was constructed using news articles sourced between mid\-2017 and early 2019 from media outlets categorized by Media Bias/Fact Check\. These articles underwent deduplication prior to annotation\. The corpus comprises 536 articles\. These articles are relatively lengthy, averaging 5,348 characters and 875 tokens per piece\. Across these articles, there are 8,981 annotated propaganda instances\. The quality of these annotations was assured by the collaborative efforts of six professional annotators, trained for this specific task\. They employed a two\-phase annotation protocol using a customized Anafora annotation platform\([Chen and Styler, 2013](https://arxiv.org/html/2610.03077#bib.bib3)\): initial independent annotation followed by consolidation with an expert reviewer\. The reported inter\-annotator agreement reached an averageγ\\gammascore of0\.60\.6\. The dataset is divided into standard training, development, and test splits, containing 371, 75, and 90 articles, respectively\. The distribution of propaganda techniques is notably skewed, withLoaded LanguageandName Calling/Labelingbeing the most frequent by a significant margin\. Further details on the distribution of techniques and data splits can be seen in[Figure 1](https://arxiv.org/html/2610.03077#S3.F1)\. The richness, annotation quality, and defined structure of the corpus make it an excellent resource for our investigation\.

![Refer to caption](https://arxiv.org/html/2610.03077v1/images/data.png)Figure 1\.Labels distribution in the corpus, split by train and test\. On the y\-axis the number of articles and on the x\-axis the labels\. Loaded language is the most frequent technique\.
## 4\.Methodology

The original SemEval task comprised two distinct but related subtasks: Span Identification \(SI\), which aimed to pinpoint the exact text segments \(spans\) within an article that contained propaganda, and Technique Classification \(TC\)\. The TC subtask, as originally conceived, involved taking these pre\-identified spans and assigning one or more labels from a set of 14 predefined propaganda techniques \(such as Loaded Language, Name Calling/Labeling, Appeal to Fear, etc\.\) to classify the specific method used within that particular span\.

However, our research adapts this task\. Instead of relying on pre\-identified spans, we approach TC as an article\-level, multi\-label classification problem\. We provide the entire text of each news article as input to our models and task them with identifying all propaganda techniques present anywhere within that article from the same list of 14 defined by the SemEval organizers\. This approach tests the models’ ability to discern the presence of various, potentially subtle, techniques embedded within the broader context of a full news piece\. To achieve this, we employed and compared two primary categories of models: fine\-tuned Masked Language Models \(MLMs\), specifically variants of RoBERTa and DeBERTa, and various prompted Causal Language Models \(LLMs\) from providers like Google, OpenAI, Anthropic, Mistral AI, and Meta\.

### 4\.1\.Masked Language Models

In our first approach, we leveraged two state\-of\-the\-art pre\-trained language models, XLM\-RoBERTa\([Conneau et al\., 2020](https://arxiv.org/html/2610.03077#bib.bib4)\)and DeBERTa V3 Large\([He et al\., 2021](https://arxiv.org/html/2610.03077#bib.bib8)\)and we fine\-tuned them on the training dataset\. On top of the pre\-trained model, we put a fully connected layer composed of a linear layer, normalization, tanh activation function, and a 0\.5 dropout layer\. To optimize model performance, we adopted a two\-phase fine\-tuning strategy inspired by\([Sun et al\., 2020](https://arxiv.org/html/2610.03077#bib.bib14)\)\. In the first phase, we kept the original model’s parameters frozen to preserve the pre\-trained knowledge and solely trained the FC layer\. This training was conducted over 5 epochs to allow the classifier to adapt to the domain\-specific data without altering the core representations learned during pre\-training\. In the second phase, we unfroze the pre\-trained model’s final layer and jointly fine\-tuned both the classification head and the last transformer layer of the base model\. To mitigate the risk of overfitting, we applied a reduced learning rate \(2​e−42e\-4\) during this phase and trained for one additional epoch\.

### 4\.2\.Causal Models

For the Causal Model approach, we leveraged the generative capabilities of several state\-of\-the\-art large language models \(LLMs\) accessed via APIs\. The models evaluated include:

- •Google: Gemini 1\.0 Pro, Gemini 1\.5 Pro, Gemini Flash 2\.0, Gemini Flash 2\.5
- •OpenAI: GPT\-3\.5\-Turbo, GPT\-4, GPT\-4\-Turbo, GPT\-4o, GPT\-4o\-mini, a fine\-tuned GPT\-3\.5\-Turbo variant \(details in[Table 1](https://arxiv.org/html/2610.03077#S5.T1)\)
- •Anthropic: Claude 3 Haiku, Claude 3 Sonnet, Claude 3 Opus
- •Mistral AI: Mistral\-Small \(mistral\-s\), Mistral\-Medium \(mistral\-m\), Mistral\-Large \(mistral\-l\)
- •Meta: Llama 3

We interacted with these models using two distinct prompting strategies to perform the article\-level multi\-label classification task:

1. \(1\)Base prompting\. We ask the conversational agent to return a list of identified propaganda detection techniques with the following prompt: "For the given article state which of the 14 propaganda techniques are present\. If no propaganda technique was identified, return ’no propaganda detected’\. You MUST output a list of the propaganda techniques with each technique in a new line, e\.g\.: techniques\. This is the article: <article\>"\.
2. \(2\)Chain\-of\-thought prompting\. We ask conversational agent to return a list of identified propaganda detection techniques along with the explanation \(reasoning\) of choosing that technique with the following prompt: "For the given article please state which of the 14 propaganda techniques are present and give an explanation to why the technique is present in the article\. If no propaganda technique was identified return ’no propaganda detected’\. You MUST output a list of the propaganda techniques with each technique in a new line, e\.g\.: Loaded Language \- Your explanation why this technique is present in the article\. This is the article: <article\>"\.

## 5\.Results

Our investigation establishes new benchmarks for the SemEval\-2020 Task 11 Technique Classification subtask\. We obtained new SOTA results for both Masked Models and Causal Models\. With our model based on XLM\-RoBERTa we obtained an F1 score of 62\.33, while the model based on DeBERTa V3 Large obtained a slightly better score of 63\.18\.

The best Causal Model \(Gemini Flash 2\.5\) also beat the previous SOTA \(63\.62 vs 58\)\. Using base prompting \(as seen in[Table 1](https://arxiv.org/html/2610.03077#S5.T1)\) the models from Google performed best\. Gemini Flash 2\.5 has the highest recall \(78\) but a relatively lower precision \(54\), while Gemini Flash 2 has a more balanced precision and recall with a higher precision \(61\) than Gemini Flash 2\.5\.

Comparing prompting strategies for Causal LLMs yielded interesting insights\. While Chain\-of\-Thought \(CoT\) prompting, designed to elicit reasoning, occasionally boosted recall significantly \(as seen with GPT\-4\.1 and GPT\-4o in[Table 2](https://arxiv.org/html/2610.03077#S5.T2)\), it did not consistently lead to the highest F1 scores across all models\. In fact, the top\-performing model, Gemini Flash 2\.5, achieved its best result using the simpler base prompt \([Table 1](https://arxiv.org/html/2610.03077#S5.T1)\)\. This indicates a trade\-off: CoT might encourage models to identify more potential instances \(high recall\), but sometimes at the cost of accuracy \(lower precision\), potentially due to overthinking or generating less direct classifications\. Furthermore, the performance tables reveal distinct behavioral profiles; for example, Gemini Flash 2\.5 \(base\) excelled in recall \(77\.93\) but had moderate precision \(53\.75\), whereas our fine\-tuned GPT\-3\.5 variant achieved high precision \(71\.68\) but much lower recall \(45\.25\)\.

Using chain\-of\-thought prompting \([Table 2](https://arxiv.org/html/2610.03077#S5.T2)\), the top model is also Gemini Flash 2\.5\. Our results demonstrate that newer language models generally outperform older ones and that proprietary models outperform open source ones \(LLaMA 3\)\. This suggests that models are getting better at propaganda detection\. Additionally, fine\-tuning these models on specific datasets further enhances their performance compared to their base versions\. Our fine\-tuned version of gpt 3\.5\-turbo was better than the base one\.

Table 1\.Model performance with Base Prompting \(gpt\-3\.5\-ft is a gpt\-3\.5\-turbo fine\-tuned model\)\. The fine\-tuned model was trained using 7,178,874 tokens over 3 epochs with a batch size of 2, resulting in 2246 steps, an LR multiplier of 2 and a seed of 477055734 \(id: ft:gpt\-3\.5\-turbo\-0125:personal::9K750VBH\)\.ModelF1\-ScorePrecisionRecallGemini Flash 2\.563\.6253\.7577\.93Gemini Flash 2\.063\.1060\.5165\.92GPT\-4\.160\.9653\.6070\.67GPT\-4o59\.4449\.9173\.46GPT\-4o\-mini57\.6751\.4265\.64gpt\-3\.5\-ft55\.4871\.6845\.25mistral\-s52\.2744\.0664\.25claude\-sonnet51\.9458\.6046\.65gpt\-4\-turbo51\.6452\.7750\.56claude\-opus50\.1763\.7941\.34mistral\-m46\.7548\.3645\.25Gemini Pro 1\.546\.2844\.8847\.77claude\-haiku42\.7547\.1439\.11gpt\-3\.5\-turbo42\.6559\.5033\.24mistral\-l40\.3542\.6838\.27Gemini Pro 1\.037\.1545\.7131\.28llama327\.3842\.8620\.11Table 2\.Model performance with Chain\-of\-thought prompting\. Best performing model is Gemini Flash 2\.5, but the model with the highest recall was GPT\-4o\.ModelF1\-ScorePrecisionRecallGemini Flash 2\.561\.5550\.6378\.49Gemini Flash 2\.059\.7755\.0565\.36GPT\-4o\-mini58\.6751\.8267\.60GPT\-4o56\.5843\.6480\.45gpt\-4\-turbo55\.3152\.1058\.94GPT\-4\.152\.8836\.6894\.69claude\-opus50\.3956\.4045\.53mistral\-s50\.2337\.3176\.82claude\-sonnet49\.1351\.2147\.21Gemini Pro 1\.546\.3950\.3343\.02mistral\-m45\.0642\.6447\.77mistral\-l40\.4641\.5939\.39claude\-haiku36\.8643\.2332\.12Gemini Pro 1\.035\.5340\.6531\.56gpt\-3\.5\-turbo31\.9346\.5224\.30llama325\.6139\.3118\.99### 5\.1\.Error Analysis and Discussion

We can see in[Table 3](https://arxiv.org/html/2610.03077#S5.T3)that there’s a significant difference in performance across different models and techniques\. Some models excel at certain techniques while struggling with others\. Certain propaganda techniques appear to be more challenging to detect than others\. Most model fare well in the loaded language and name calling techniques\. Models struggle in general with cliches, bandwagon and black and white fallacy\. We observe substantial performance disparities among models for the same propaganda technique\. For instance, Claude\-Sonnet achieves a notably higher accuracy of 0\.89 for "loaded language" compared to GPT\-3\.5’s 0\.49\. This suggests that an ensemble approach, combining multiple models, could yield improved results due to their complementary strengths\.

TechniquesClaudesonnetOpenAIgpt\-3\.5OpenAIGPT\-4oOpenAIgpt\-4Metallama3MistrallargeMistralmediumGemini1\.0\-proGemini1\.5\-proGeminiFlash 2\.0GeminiFlash 2\.5appeal\_to\_authority0\.380\.000\.370\.000\.140\.360\.210\.100\.430\.290\.33appeal\_to\_fear0\.530\.350\.480\.490\.070\.450\.290\.510\.620\.630\.64bandwagon0\.120\.000\.160\.210\.100\.000\.000\.090\.140\.180\.18black\_and\_white0\.130\.000\.350\.350\.000\.300\.310\.220\.260\.470\.29oversimplification0\.290\.000\.310\.410\.000\.320\.370\.270\.360\.410\.46doubt0\.350\.310\.300\.540\.000\.480\.590\.380\.510\.520\.56exaggeration0\.340\.110\.680\.620\.110\.520\.390\.050\.060\.610\.66flag\_waving0\.050\.330\.590\.560\.000\.050\.440\.200\.250\.390\.69loaded\_language0\.890\.490\.950\.950\.840\.710\.940\.530\.880\.950\.94name\_calling0\.700\.610\.820\.580\.070\.490\.040\.640\.570\.770\.82repetition0\.320\.150\.660\.460\.120\.190\.600\.370\.240\.640\.63slogans0\.410\.000\.510\.000\.080\.140\.330\.130\.290\.240\.44cliches0\.150\.000\.300\.290\.000\.000\.130\.320\.130\.000\.24whataboutism0\.410\.380\.420\.390\.090\.410\.320\.240\.210\.330\.43Table 3\.Comparative analysis of causal language models’ performance \(F1 macro scores\) in detecting specific propaganda techniques\.![Refer to caption](https://arxiv.org/html/2610.03077v1/images/cf_base.png)Figure 2\.This confusion matrix visualizes the performance for the model GPT\-4o, using base prompting\. \(class 0 is the existence of propaganda technique and class 1 is the absence of propaganda technique\)If we average the F1 score of each technique we obtain an overall F1 score of 65\.61\. Based on the confusion matrix of results with base prompting in[Figure 2](https://arxiv.org/html/2610.03077#S5.F2), the model performs better on class 1 \(with a 73% true positive rate\) than on class 0 \(with a 62% true negative rate\)\. False positives \(38%\) are a more common error than false negatives \(27%\), indicating that the model tends to over\-predict class 1 slightly more often than class 0\.

![Refer to caption](https://arxiv.org/html/2610.03077v1/images/cf_chain.png)Figure 3\.This confusion matrix visualizes the performance for the model GPT\-4o\-mini, using chain\-of\-thought prompting\.Looking at the same confusion matrix for chain\-of\-thought prompting \([Figure 3](https://arxiv.org/html/2610.03077#S5.F3)\), we have an average F1 score of 67\.34, reflecting a slight improvement over the base prompting\. The model performs similarly on both classes, with slightly better results for class 0 \(67% true negative\) than class 1 \(68% true positive\)\.

### 5\.2\.Conclusions and Future Work

In this paper, we explored advancements in propaganda detection through a comparative analysis of masked and causal language models, leveraging base and chain\-of\-thought prompting strategies\. Our results demonstrate that newer language models, particularly causal models like GPT\-4o and Gemini Flash 2\.0 and 2\.5, outperform previous state\-of\-the\-art approaches in identifying propaganda techniques\. Our findings suggest that fine\-tuning models on specific datasets and employing diverse prompting techniques, such as chain\-of\-thought prompting, further improve detection accuracy\. Also, when using Masked Models, fine\-tuning them in 2 steps and freezing the weights improve the model’s performance\.

For future work, we recommend exploring the use of ensemble models\. As highlighted in the error analysis, different models exhibit strengths in detecting distinct propaganda techniques\. Future work could focus on developing ensemble methods that combine predictions from multiple top\-performing models \(both masked and causal\) to leverage their complementary capabilities and potentially achieve more robust and accurate overall detection across all techniques\. We would also like to investigate hybrid models that combine the strengths of fine\-tuned Masked Language Models \(like DeBERTa V3 for its strong baseline performance\) with the contextual understanding and reasoning abilities of large Causal Models \(like Gemini or GPT\-4\) perhaps using the MLM output to guide or constrain the LLM’s analysis\.

This study focused solely on Technique Classification \(TC\)\. Future research could revisit the full SemEval task by integrating Span Identification \(SI\) with TC using modern LLMs\. This could involve multi\-task learning or pipeline approaches where LLMs first identify potential spans and then classify the techniques within those specific spans, potentially improving precision\. Also, given that propaganda dissemination is a global phenomenon transcending language barriers, future work should extend this research beyond English to evaluate model performance in other languages and develop robust multilingual detection systems\.

### 5\.3\.Limitations

Evaluating Large Language Models faces the challenge of potential data leakage\. Because these models are pre\-trained on vast datasets, scraped from the internet, there’s a risk they were exposed to the test data—in this instance, the SemEval task data—during training, which could artificially inflate performance and obscure true generalization ability\.

### 5\.4\.Ethical considerations

There are no ethical issues that could result from the publication of our work\.

###### Acknowledgements\.

This research is partially supported by the project “Romanian Hub for Artificial Intelligence \- HRIA”, Smart Growth, Digitization and Financial Instruments Program, 2021\-2027, MySMIS no\. 334906\.

## References

- Barrón\-Cedeño et al\.\(2019\)Alberto Barrón\-Cedeño, Israa Jaradat, Giovanni Da San Martino, and Preslav Nakov\. 2019\.Proppy: Organizing the news based on their propagandistic content\.*Information Processing and Management*56, 5 \(2019\), 1849–1864\.[doi:10\.1016/j\.ipm\.2019\.03\.005](https://doi.org/10.1016/j.ipm.2019.03.005)
- Chen and Styler \(2013\)Wei\-Te Chen and Will Styler\. 2013\.Anafora: A Web\-based General Purpose Annotation Tool\. In*Proceedings of the 2013 NAACL HLT Demonstration Session*, Chris Dyer and Derrick Higgins \(Eds\.\)\. Association for Computational Linguistics, Atlanta, Georgia, 14–19\.[https://aclanthology\.org/N13\-3004/](https://aclanthology.org/N13-3004/)
- Conneau et al\.\(2020\)Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov\. 2020\.Unsupervised Cross\-lingual Representation Learning at Scale\. In*Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault \(Eds\.\)\. Association for Computational Linguistics, Online, 8440–8451\.[doi:10\.18653/v1/2020\.acl\-main\.747](https://doi.org/10.18653/v1/2020.acl-main.747)
- Da San Martino et al\.\(2020a\)Giovanni Da San Martino, Alberto Barrón\-Cedeño, Henning Wachsmuth, Rostislav Petrov, and Preslav Nakov\. 2020a\.SemEval\-2020 Task 11: Detection of Propaganda Techniques in News Articles\. In*Proceedings of the Fourteenth Workshop on Semantic Evaluation*, Aurelie Herbelot, Xiaodan Zhu, Alexis Palmer, Nathan Schneider, Jonathan May, and Ekaterina Shutova \(Eds\.\)\. International Committee for Computational Linguistics, Barcelona \(online\), 1377–1414\.[doi:10\.18653/v1/2020\.semeval\-1\.186](https://doi.org/10.18653/v1/2020.semeval-1.186)
- Da San Martino et al\.\(2020b\)Giovanni Da San Martino, Shaden Shaar, Yifan Zhang, Seunghak Yu, Alberto Barrón\-Cedeno, and Preslav Nakov\. 2020b\.Prta: A system to support the analysis of propaganda techniques in the news\. In*MISSING: Please add Conference/Proceedings Title*\. MISSING: Please add Publisher, 287–293\.
- Flew et al\.\(2020\)Terry Flew, Uwe Dulleck, Caroline Fisher, Sora Park, and Ozan Isler\. 2020\.Trust and mistrust in Australian news media\.University of Canberra Research Portal\.
- He et al\.\(2021\)Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen\. 2021\.DEBERTA: DECODING\-ENHANCED BERT WITH DISENTANGLED ATTENTION\. In*International Conference on Learning Representations*\.[https://openreview\.net/forum?id=XPZIaotutsD](https://openreview.net/forum?id=XPZIaotutsD)
- Malik et al\.\(2023\)Muhammad Shahid Iqbal Malik, Tahir Imran, and Jamjoom Mona Mamdouh\. 2023\.How to detect propaganda from social media? Exploitation of semantic and fine\-tuned language models\.*PeerJ Computer Science*\(2023\)\.[doi:10\.7717/peerj\-cs\.1248](https://doi.org/10.7717/peerj-cs.1248)
- Martino et al\.\(2019\)Giovanni Da San Martino, Seunghak Yu, Alberto Barrón\-Cedeño, Rostislav Petrov, and Preslav Nakov\. 2019\.Fine\-Grained Analysis of Propaganda in News Articles\.*CoRR*abs/1910\.02517 \(2019\)\.arXiv:1910\.02517[http://arxiv\.org/abs/1910\.02517](http://arxiv.org/abs/1910.02517)
- Rashkin et al\.\(2017\)Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi\. 2017\.Truth of Varying Shades: Analyzing Language in Fake News and Political Fact\-Checking\. In*Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing*, Martha Palmer, Rebecca Hwa, and Sebastian Riedel \(Eds\.\)\. Association for Computational Linguistics, Copenhagen, Denmark, 2931–2937\.[doi:10\.18653/v1/D17\-1317](https://doi.org/10.18653/v1/D17-1317)
- Sprenkamp et al\.\(2023\)Kilian Sprenkamp, Daniel Gordon Jones, and Liudmila Zavolokina\. 2023\.Large Language Models for Propaganda Detection\.*arXiv*\(2023\)\.
- Stanley \(2015\)Jason Stanley\. 2015\.*How propaganda works*\.Princeton University Press\.
- Sun et al\.\(2020\)Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang\. 2020\.How to Fine\-Tune BERT for Text Classification?arXiv:1905\.05583 \[cs\.CL\]