On the Impact of Anonymization on the Performance of Large Language Models

# On the Impact of Anonymization on the Performance of Large Language Models
Source: [https://arxiv.org/html/2609.11335](https://arxiv.org/html/2609.11335)
Tobias Deußer††thanks:tdeusser@uni\-bonn\.de, ORCID\-ID: 0000\-0003\-4685\-0847Affiliation:University of Bonn, Bonn, GermanyAffiliation:Fraunhofer IAIS, Sankt Augustin, GermanyAffiliation:Lamarr Institute for Machine Learning and Artificial Intelligence, Bonn, GermanyMax HahnbückAffiliation:University of Bonn, Bonn, GermanyAffiliation:Fraunhofer IAIS, Sankt Augustin, GermanyLorenz SparrenbergAffiliation:University of Bonn, Bonn, GermanyAffiliation:Lamarr Institute for Machine Learning and Artificial Intelligence, Bonn, GermanyChristian BauckhageAffiliation:University of Bonn, Bonn, GermanyAffiliation:Fraunhofer IAIS, Sankt Augustin, GermanyAffiliation:Lamarr Institute for Machine Learning and Artificial Intelligence, Bonn, GermanyRafet SifaAffiliation:University of Bonn, Bonn, GermanyAffiliation:Fraunhofer IAIS, Sankt Augustin, GermanyAffiliation:Lamarr Institute for Machine Learning and Artificial Intelligence, Bonn, GermanyAffiliation:*These authors contributed equally to this work\.*

###### Abstract

As large language models are increasingly deployed in sensitive domains, anonymizing input data to protect personally identifiable information has become a critical practice\. However, the impact of this anonymization on model utility is not well understood\. This paper presents a systematic empirical study of the trade\-off between privacy and performance\. We evaluate five prominent language models across eleven diverse benchmarks, comparing their performance on original versus pseudonymized inputs\. Our results reveal that while anonymization generally degrades performance, the effect is highly nuanced\. We find that more capable models, such as Qwen2\.5\-72B and GPT\-4o mini, suffer the largest performance drops, suggesting a stronger reliance on specific entity information\. The impact is also task\-dependent: performance on TruthfulQA improves with anonymization, while retrieval\-focused tasks like RGB experience a catastrophic decline\. Further experiments show that reversible anonymization techniques that preserve entity uniqueness significantly outperform irreversible ones like redaction, and that explicitly prompting models about anonymization offers no discernible benefit\. We conclude that anonymization is not a one\-size\-fits\-all solution and must be co\-designed with the model and task in mind to balance privacy and utility effectively\. Our findings provide a crucial baseline for developing more robust, privacy\-aware AI systems\.

*Keywords*anonymization⋅\\cdotlarge language models⋅\\cdotnatural language processing⋅\\cdotmachine learning

## 1Introduction

Large Language Models \(LLMs\) have become increasingly central to modern artificial intelligence applications, from analyzing medical reports\[[1](https://arxiv.org/html/2609.11335#bib.bib4)\], writing legal documents\[[2](https://arxiv.org/html/2609.11335#bib.bib5)\], finding contradictions in financial documents\[[3](https://arxiv.org/html/2609.11335#bib.bib6)\], connecting patients with rare diseases with each other\[[4](https://arxiv.org/html/2609.11335#bib.bib7)\], to enabling automated auditing for financial reports\[[5](https://arxiv.org/html/2609.11335#bib.bib9)\]\. Yet, as such LLM approaches are deployed in real\-world settings, privacy concerns are becoming more and more prevalent\[[6](https://arxiv.org/html/2609.11335#bib.bib8),[7](https://arxiv.org/html/2609.11335#bib.bib10)\]\. One common mitigation strategy is anonymization, i\.e\., removing or masking personally identifiable information \(PII\) before feeding inputs into a model\[[8](https://arxiv.org/html/2609.11335#bib.bib2),[9](https://arxiv.org/html/2609.11335#bib.bib3),[10](https://arxiv.org/html/2609.11335#bib.bib1)\]\.

But what does this actually cost us in terms of performance? In this paper, we take a systematic look at the tradeoff between privacy and capability\. We evaluate a set of five models \(see Table[3](https://arxiv.org/html/2609.11335#S3.T3)\) spanning a wide range of sizes and capabilities across eleven diverse benchmarks \(see Table[1](https://arxiv.org/html/2609.11335#S3.T1)\)\. These benchmarks cover a spectrum of natural language processing \(NLP\) tasks, including general scientific reasoning \(ARC,[11](https://arxiv.org/html/2609.11335#bib.bib13)\), medical question answering \(MedQA,[12](https://arxiv.org/html/2609.11335#bib.bib18)\), emotional intelligence \(EQ\-Bench,[13](https://arxiv.org/html/2609.11335#bib.bib15)\), and translation \(WMT 2014,[14](https://arxiv.org/html/2609.11335#bib.bib23)\), allowing us to assess the trade\-off across different cognitive and linguistic domains\.

We observe that, while anonymization generally degrades performance as expected, the magnitude of this effect is highly dependent on the model’s capability, the nature of the task, and the specific anonymization technique employed\. Our key findings are threefold:

1. 1\.Model capability is inversely correlated with robustness to anonymization\.More powerful models like Qwen2\.5\-72B\[[15](https://arxiv.org/html/2609.11335#bib.bib28)\]and GPT\-4o mini\[[16](https://arxiv.org/html/2609.11335#bib.bib24)\]experience the largest performance drops, suggesting they rely more heavily on specific entity information to achieve their state\-of\-the\-art performance\.
2. 2\.Task sensitivity varies dramatically\.Performance on retrieval\-intensive benchmarks like RGB collapses catastrophically post\-anonymization\. Conversely, on TruthfulQA, a benchmark designed to test factual accuracy, performance consistently improves, suggesting that removing named entities can mitigate a model’s tendency to hallucinate incorrect associations\.
3. 3\.Technique and prompting matter, but not always as expected\.Reversible anonymization techniques like pseudonymization, which preserve entity uniqueness, significantly outperform irreversible methods like redaction\. However, explicitly prompting a model that its input has been anonymized yields no discernible performance benefit\.

Our results challenge the notion of anonymization as a one\-size\-fits\-all solution, demonstrating that it must be co\-designed with the model and task in mind\. These findings provide actionable insights for practitioners and a foundation for the discussion on and development of more robust, privacy\-aware, and ultimately more trustworthy NLP systems\.

## 2Related Work

The need to anonymize text data to protect personally identifiable information \(PII\) is a well\-established practice driven by regulatory frameworks and the goal of developing trustworthy AI systems\[[8](https://arxiv.org/html/2609.11335#bib.bib2),[9](https://arxiv.org/html/2609.11335#bib.bib3)\]\. However, this introduces a fundamental tension between preserving privacy and maintaining data utility for downstream applications\[[17](https://arxiv.org/html/2609.11335#bib.bib29)\]\. Our work contributes to the growing body of research addressing this privacy\-utility trade\-off, with a focus on the capabilities and shortcomings of modern large\-scale language models \(LLMs\)\.

Early anonymization systems predominantly combined named entity recognition \(NER\), rule\-based filters, and external gazetteers to mask explicit identifiers\[[18](https://arxiv.org/html/2609.11335#bib.bib30),[19](https://arxiv.org/html/2609.11335#bib.bib31)\]\. The advent of LLMs has initiated a paradigm shift in this domain\. LLMs are now employed as potent, context\-aware anonymization tools capable of zero\-shot PII removal and replacement\[[20](https://arxiv.org/html/2609.11335#bib.bib32),[21](https://arxiv.org/html/2609.11335#bib.bib33)\]\. This development is complemented by methodologies that distill the knowledge from large, proprietary models into smaller, open\-source versions for practical deployment\[[10](https://arxiv.org/html/2609.11335#bib.bib1)\]\. Conversely, the inferential capabilities of LLMs also position them as formidable adversaries in re\-identifying individuals from inadequately anonymized text, thereby elevating the requirements for robust privacy measures\[[22](https://arxiv.org/html/2609.11335#bib.bib34)\]\. This dual capacity of LLMs demands a more profound comprehension of their interplay with anonymized data\.

Empirical evaluations of anonymization effects have mostly been domain\-specific\. In medicine,[Ben Cheikh Larbi et al\. \[23\]](https://arxiv.org/html/2609.11335#bib.bib35)found variable performance losses across five clinical NLP tasks and persistent re\-identification risk\. In contrast,[Vakili et al\. \[24\]](https://arxiv.org/html/2609.11335#bib.bib37)reported that pseudonymization before training Swedish Clinical BERT resulted in near\-identical downstream performance\. Financial\-domain work combining Differential Privacy with Federated Learning explicitly highlights the privacy\-utility trade\-offs in transformer\-based text classification\[[25](https://arxiv.org/html/2609.11335#bib.bib38)\]\. Meanwhile, analyses of pseudonymization strategies reveal that seemingly minor design differences can have a significant impact on classification and summarization outcomes\[[26](https://arxiv.org/html/2609.11335#bib.bib36)\]\.

The rapid evolution of privacy\-preserving techniques has created a pressing need for robust evaluation frameworks\.[Shahriar and Dara \[27\]](https://arxiv.org/html/2609.11335#bib.bib42)introduced Priv\-IQ, a comprehensive multimodal benchmark measuring LLM privacy intelligence across eight competencies including visual privacy, multilingual capabilities, and knowledge of privacy law\.[Sun et al\. \[28\]](https://arxiv.org/html/2609.11335#bib.bib43)presented a systematic evaluation framework for privacy\-preserving algorithms across different LLM architectures \(Mistral\-7B, Llama2\-7b, Falcon\-7b\), examining three scenarios: protecting training data only, user queries only, or both\.

Domain\-specific evaluation tools have also emerged\.[Zhang et al\. \[29\]](https://arxiv.org/html/2609.11335#bib.bib44)introduced SafetyBench with over 11,000 questions across seven safety categories including privacy concerns, while[Andriushchenko et al\. \[30\]](https://arxiv.org/html/2609.11335#bib.bib45)presented AgentHarm for evaluating harmful LLM agent behaviors including privacy violations\. These specialized benchmarks complement general privacy evaluations by capturing nuanced risks in specific deployment contexts\.

For a more complete overview on the topic of textual anonymization, we refer the interested reader to our survey paper on various anonymization techniques and recent advances\[[31](https://arxiv.org/html/2609.11335#bib.bib41)\]\.

While these studies offer vital insights, they are often confined to specific domains \(e\.g\., clinical or financial texts\), employ a restricted set of NLP tasks \(most commonly classification\), or are predicated on earlier model architectures such as BERT\. Our research broadens this line of inquiry by conducting a large\-scale, systematic empirical evaluation with a more extensive scope\. We evaluate a diverse set of five modern and highly capable LLMs, with parameter counts ranging from 7 to 72 billion, across eleven distinct benchmarks that encompass a wide array of reasoning and linguistic capabilities\. This allows us to transcend domain\-specific findings and identify more generalizable patterns concerning the privacy\-utility trade\-off\. We aim to establish a cross\-domain baseline for understanding how input anonymization affects the performance of contemporary LLMs\.

## 3Methodology

This section details the methodology employed to investigate the impact of input anonymization on the performance of LLMs\. We describe our experimental pipeline, the selection criteria for benchmarks and models, and the specific experimental setup used to compare performance on original versus anonymized inputs\.

### 3\.1Pipeline

To systematically evaluate the effect of anonymization, we developed a modular pipeline for our experiments\. This pipeline facilitates the processing of diverse benchmarks with various LLMs, both with and without anonymization\. The core stages are:

1. 1\.Data preparation: Benchmarks are downloaded, preprocessed, and sampled as required\. Each benchmark is encapsulated, storing prompt templates, questions, and reference answers\.
2. 2\.Anonymization: Questions \(and associated prompt components, if applicable\) are processed by an anonymizer, which replaces personally identifiable information \(PII\) with placeholders\. A mapping between original and anonymized entities is retained\.
3. 3\.Model inference: Selected LLMs generate responses to both the original and anonymized versions of the benchmark questions\.
4. 4\.De\-anonymization \(for evaluation\): For anonymized outputs, placeholders in the LLM\-generated responses are mapped back to their original PII using the stored mappings\. This step is crucial for ensuring fair comparison with reference answers\.
5. 5\.Evaluation: LLM responses \(with original PII restored for anonymized versions\) are assessed against reference answers using the predefined evaluation metrics specific to each benchmark\.
6. 6\.Results aggregation: Performance scores are collated, and metrics such as the relative performance change between anonymized and non\-anonymized runs are computed\.

Figure 1:Architecture overview of our encoder\-only contradiction detection approach\.Figure[1](https://arxiv.org/html/2609.11335#S3.F1)provides a conceptual overview of this pipeline\. The design emphasises extensibility, allowing for the straightforward integration of new benchmarks, anonymization strategies, and LLMs through abstract base classes\.

### 3\.2Dataset and Benchmark Selection

To evaluate the impact of anonymization across diverse linguistic and cognitive tasks, we selected eleven established benchmarks\. Our selection was guided by several criteria:

1. 1\.Popularity and recognition: Widespread use within the research community\.
2. 2\.Openness: Publicly available test sets and evaluation methodologies to ensure transparency and reproducibility\.
3. 3\.Language coverage: Predominantly English\-language benchmarks, with the inclusion of one German\-English translation task \(WMT 2014\) to assess cross\-lingual effects\.
4. 4\.Task Diversity: A broad range of domains \(e\.g\., science, medicine, commonsense reasoning\) and prompt styles \(e\.g\., zero\-shot, few\-shot, chain\-of\-thought\) to avoid a narrow focus\.

Domains such as coding and mathematics were excluded, as they typically do not involve PII, meaning anonymization would have minimal impact\. We identified candidate benchmarks by consulting resources such as the Open LLM Leaderboard\[[32](https://arxiv.org/html/2609.11335#bib.bib12)\]and the Stanford HELM project\[[33](https://arxiv.org/html/2609.11335#bib.bib11)\]\. The selected benchmarks are detailed in Table[1](https://arxiv.org/html/2609.11335#S3.T1)\.

Table 1:Overview of selected benchmarks: Domains, prompt type, and references\. CoT denotes Chain\-of\-Thought; RAG denotes Retrieval\-Augmented Generation\.BenchmarkDomainPrompt TypeReferenceARCGrade\-school science reasoningZero\-Shot\[[11](https://arxiv.org/html/2609.11335#bib.bib13)\]BIG\-Bench HardChallenging reasoning tasksCoT, Few\-Shot\[[34](https://arxiv.org/html/2609.11335#bib.bib14)\]EQ\-BenchEmotional intelligenceZero\-Shot, Self\-Critique\[[13](https://arxiv.org/html/2609.11335#bib.bib15)\]HellaSwagCommonsense reasoningZero\-Shot\[[35](https://arxiv.org/html/2609.11335#bib.bib16)\]IFEvalComplex instruction followingZero\-Shot\[[36](https://arxiv.org/html/2609.11335#bib.bib17)\]MedQAMedical knowledge & reasoningZero\-Shot\[[12](https://arxiv.org/html/2609.11335#bib.bib18)\]MMLU\-PROMultitask academic & professional knowledgeCoT, Few\-Shot\[[37](https://arxiv.org/html/2609.11335#bib.bib19)\]MUSRMulti\-step reasoningCoT, Zero\-Shot\[[38](https://arxiv.org/html/2609.11335#bib.bib20)\]RGBRetrieval\-Augmented GenerationZero\-Shot\[[39](https://arxiv.org/html/2609.11335#bib.bib21)\]TruthfulQATruthfulness & debunking misconceptionsZero\-Shot\[[40](https://arxiv.org/html/2609.11335#bib.bib22)\]WMT 2014 \(En\-De\)Machine translation \(English to German\)Few\-Shot\[[14](https://arxiv.org/html/2609.11335#bib.bib23)\]Given computational constraints associated with evaluating multiple LLMs across numerous benchmarks, we standardised the number of samples\. For most benchmarks, a maximum of 500 samples were randomly selected from their respective test sets\. For benchmarks comprising multiple tasks \(specifically, BIG\-Bench Hard and MMLU\-PRO\), this cap was increased to 1,000 samples to ensure adequate representation per subtask\. This resulted in a total of 6,210 unique samples being evaluated\. An overview of original and adjusted sample sizes can be found in Table[2](https://arxiv.org/html/2609.11335#S3.T2)\.

Table 2:Benchmark summary: Number of tasks, original sample counts, adjusted sample counts for this study, and average input length \(average number of tokens\) of adjusted samples\.BenchmarkSubtasksSample CountsAdj\. Sample CountsAvg\. LengthARC2354850095\.79BIG\-Bench Hard276421999812\.11EQ\-Bench2342342404\.76HellaSwag110042500133\.98IFEval154150044\.02MedQA11273500170\.37MMLU\-PRO1311032994382\.70MUSR33753751295\.43RGB3500500921\.27TruthfulQA1790500144\.87WMT 2014 \(En\-De\)1300350044\.74Total55389676210417\.62When official prompts were available with the benchmarks, they were utilised\. If multiple official prompts existed \(e\.g\., zero\-shot vs\. few\-shot\), we selected the one anticipated to yield stronger baseline performance, typically favouring few\-shot prompts\. For benchmarks providing only datasets without predefined prompts, we first sought established prompt implementations \(e\.g\., from the HELM repository\)\. If no suitable reference was found, we created minimal zero\-shot prompts\. Importantly, for consistency, any prompt components, including few\-shot examples, were anonymized alongside the primary question input to ensure comprehensive PII removal from the text presented to the LLM\. This approach is crucial as our focus is on the relative impact of anonymization rather than achieving absolute state\-of\-the\-art scores\.

### 3\.3Model Selection

Our selection of LLMs, as seen in Table[3](https://arxiv.org/html/2609.11335#S3.T3), aims for diversity in terms of model providers, parameter sizes, and architectures, whilst remaining within feasible computational limits for extensive experimentation\. The primary objective is to analyse the*relative*change in performance due to anonymization, rather than to benchmark absolute model capabilities against the state\-of\-the\-art\.

Table 3:Overview of large language models \(LLMs\) evaluated in this study\. Shortened names are used throughout the paper for brevity\.Short NameFull Model NameDeveloperReferenceParameter CountGPT\-4o miniGPT\-4o miniOpenAI[OpenAI \[16\]](https://arxiv.org/html/2609.11335#bib.bib24)UNKNOWNTeuken\-7BTeuken\-7B\-instruct\-research\-v0\.4OpenGPT\-X[Ali et al\. \[41\]](https://arxiv.org/html/2609.11335#bib.bib25)7BLlama\-3\.1\-8BLlama\-3\.1\-8B\-InstructMeta[Dubey et al\. \[42\]](https://arxiv.org/html/2609.11335#bib.bib26)8BGemma\-2\-27BGemma\-2\-27bGoogle[Gemma Team et al\. \[43\]](https://arxiv.org/html/2609.11335#bib.bib27)27BQwen2\.5\-72BQwen2\.5\-72B\-InstructAlibaba[Qwen et al\. \[15\]](https://arxiv.org/html/2609.11335#bib.bib28)72B
### 3\.4Experimental Setup

The core experiment compares the performance of each selected LLM on each benchmark under two conditions: \(i\) original, non\-anonymized inputs, and \(ii\) anonymized inputs\.

#### 3\.4\.1Anonymization Configuration

For the primary experiments, we employ a pseudonymization strategy implemented by our anonymization tool\[[10](https://arxiv.org/html/2609.11335#bib.bib1)\]\. This tool is a sophisticated LLM\-based system that performs named entity recognition \(NER\) to identify PII and subsequently replaces them\. Specifically, entities are substituted with a category label and a unique, consistent identifier; for example, every occurrence of "London" within a document is replaced by "<LOC\>\-1", while "Berlin" would become "<LOC\>\-2"\. This ensures that the uniqueness of entities is preserved, which is crucial for downstream tasks that rely on entity distinctions\. The underlying model is a fine\-tuned version of a smaller, efficient language model, optimized for a high recall score\. In our evaluations, this tool achieves an F1\-score of over 88% and a recall of over 91%\.

#### 3\.4\.2Comparison of Anonymization Techniques

To investigate the impact of different anonymization approaches, we evaluate a subset of models and benchmarks using four distinct techniques in addition to pseudonymization:

- •Generalization: Replacing specific PII with broader category labels \(e\.g\., “London” becomes “\[CITY\]”\)\.
- •Masking: Parts of entities are obscured in a consistent pattern, preserving some surface information \(e\.g\., “London” becomes “Lo\*\*\*\*”\)\.
- •Randomization: Entities are substituted with fixed but randomly generated strings that remain consistent across the dataset \(e\.g\., “London” becomes “Xj92Lp”\)
- •Redaction: Replacing PII with a fixed, generic marker \(e\.g\., “\[REDACTED\]” or\)\.

This comparative analysis is performed on GPT\-4o mini using the MedQA, MUSR, and RGB benchmarks\.

#### 3\.4\.3Impact of Explicit Anonymization Prompts

We also explore whether explicitly informing the LLM that an input has been anonymized affects performance\. For this, two prefixes are prepended to anonymized prompts for GPT\-4o mini on the MedQA, MUSR, and RGB benchmarks:

- •Short Prefix: “This prompt is anonymized\. Some context may be missing; respond as accurately as possible\.”
- •Long Prefix: “This prompt has been anonymized by removing all sensitive entities\. Some contextual details may be missing, but please generate the most accurate and effective response possible based on the available information\.”

Performance is compared against anonymized inputs without any prefix and non\-anonymized inputs\.

## 4Experiments

This section presents the empirical evaluation of LLM performance under anonymization\. We first detail the main experimental setup and results concerning our primary pseudonymization strategy\. Subsequently, we investigate the effects of varying anonymization techniques and the utility of explicitly informing models about input anonymization\.

All experiments are conducted on a system equipped with four NVIDIA Tesla V100 GPUs, each with 32GB of memory\. To ensure reproducibility of results, a global random seed is used for data sampling \(where applicable for benchmarks\) and any stochastic processes within the models if not entirely avoidable\. Crucially, the sampling temperature for LLM generation is set to 0 for all models and tasks, aiming for as deterministic as possible outputs\.

### 4\.1Impact of Pseudonymization on LLM Performance

We evaluated the five LLMs detailed in Table[3](https://arxiv.org/html/2609.11335#S3.T3)\(GPT\-4o mini, Teuken\-7B, Llama\-3\.1\-8B\-Instruct, Gemma\-2\-27B, and Qwen2\.5\-72B\) across the eleven benchmarks described in Table[1](https://arxiv.org/html/2609.11335#S3.T1)\. For each benchmark, models processed both original and anonymized inputs\. The anonymization employed the pseudonymization strategy outlined in Section[3\.4\.1](https://arxiv.org/html/2609.11335#S3.SS4.SSS1)\(from[Deußer et al\. \[10\]](https://arxiv.org/html/2609.11335#bib.bib1)\), replacing PII with category\-unique placeholders \(e\.g\., “London” to “<LOC\>\-1”\)\. Performance was measured using the standard metrics for each benchmark\.

Table 4:Overview of the benchmarks: Domain and PromptstyleBenchmarkDomainPrompt TypeReferenceARCGrade\-school science reasoningZero\-Shot\[[11](https://arxiv.org/html/2609.11335#bib.bib13)\]BIG\-Bench HardChallenging reasoning tasksCoT, Few\-Shot\[[34](https://arxiv.org/html/2609.11335#bib.bib14)\]EQ\-BenchEmotional intelligenceZero\-Shot, Self\-Critique\[[13](https://arxiv.org/html/2609.11335#bib.bib15)\]HellaSwagCommonsense reasoningZero\-Shot\[[35](https://arxiv.org/html/2609.11335#bib.bib16)\]IFEvalComplex instruction followingZero\-Shot\[[36](https://arxiv.org/html/2609.11335#bib.bib17)\]MedQAMedical knowledge & reasoningZero\-Shot\[[12](https://arxiv.org/html/2609.11335#bib.bib18)\]MMLU\-PROMultitask academic & professional knowledgeCoT, Few\-Shot\[[37](https://arxiv.org/html/2609.11335#bib.bib19)\]MUSRMulti\-step reasoningCoT, Zero\-Shot\[[38](https://arxiv.org/html/2609.11335#bib.bib20)\]RGBRetrieval\-Augmented GenerationZero\-Shot\[[39](https://arxiv.org/html/2609.11335#bib.bib21)\]TruthfulQATruthfulness & debunking misconceptionsZero\-Shot\[[40](https://arxiv.org/html/2609.11335#bib.bib22)\]WMT 2014 \(En\-De\)Machine translation \(English to German\)Few\-Shot\[[14](https://arxiv.org/html/2609.11335#bib.bib23)\]Table[5](https://arxiv.org/html/2609.11335#S4.T5)presents the performance scores for all models on all benchmarks, under both original and anonymized conditions\. Several patterns emerge from this comprehensive evaluation\. First, we observe that model performance is not uniformly affected across benchmarks\. While some tasks show minimal degradation \(e\.g\., ARC, HellaSwag, IFEval with drops typically under 5 percentage points\), others experience substantial losses \(e\.g\., RGB with drops of 30\-60 percentage points for most models\)\.

Second, the data reveals interesting model\-specific behaviors\. Notably, Teuken\-7B achieves a score of 0\.00 on both ARC and HellaSwag regardless of anonymization\. This suggests fundamental limitations in this model’s ability to handle these specific task formats, independent of entity information\. In contrast, Qwen2\.5\-72B and GPT\-4o mini demonstrate strong baseline performance across most benchmarks, achieving scores above 0\.60 on 8 out of 11 tasks in the non\-anonymized condition\. However, this superior baseline performance comes with a trade\-off: these same models experience the largest absolute performance drops when anonymization is applied\.

Third, TruthfulQA stands out as the only benchmark where all five models either maintain or improve performance under anonymization\. This counterintuitive result warrants closer examination, which we provide in our subsequent analysis\. The consistency of this improvement across models of varying architectures and capabilities suggests a systematic phenomenon rather than a random fluctuation\.

Figure[2](https://arxiv.org/html/2609.11335#S4.F2)illustrates the performance difference \(anonymized minus non\-anonymized scores\) as a heatmap, providing a visual representation of anonymization impact across the model\-benchmark matrix\. The color intensity corresponds to the magnitude of performance change, with darker blue indicating smaller drops \(or improvements\) and yellow/orange indicating larger performance losses\.

Several insights emerge from this visualization\. First, RGB exhibits the most severe and consistent performance degradation across all models, with drops ranging from \-0\.22 \(Teuken\-7B\) to \-0\.47 \(GPT\-4o mini\)\. This catastrophic decline suggests that RGB’s retrieval\-augmented generation tasks are fundamentally dependent on specific entity information that pseudonymization clearly obscures\. The task requires models to integrate information from multiple sources while filtering out noise, and replacing concrete entities with abstract placeholders appears to disrupt this integration process severely\.

Interestingly, the positive values for TruthfulQA represent a striking anomaly\. This consistent improvement across multiple models suggests that anonymization may actually help models avoid factual errors in this specific context\. We hypothesize that TruthfulQA questions often contain named entities that trigger incorrect memorized associations\. By replacing “Which country in Europe…” with “Which country in <LOC\>\-1…”, we may prevent models from confidently generating plausible\-sounding but incorrect facts tied to specific names they encountered during training\.

Table 5:Performance overview of the 5 tested LLMs across all benchmarks \(anonymized and non\-anonymized\)\. Higher scores indicate better performance\. The best scores are bold\.BenchmarkAnonymizedGPT\-4o miniTeuken\-7BLlama\-3\.1\-8BGemma\-2\-27BQwen2\.5\-72BARCno0\.620\.000\.840\.290\.95yes0\.640\.000\.790\.300\.91BIG\-Bench Hardno0\.240\.180\.230\.250\.23yes0\.210\.180\.200\.220\.19EQ\-Benchno0\.740\.300\.660\.760\.72yes0\.720\.290\.640\.760\.70HellaSwagno0\.770\.010\.500\.000\.82yes0\.740\.000\.490\.000\.78IFEvalno0\.810\.280\.750\.620\.83yes0\.770\.270\.700\.570\.79MedQAno0\.880\.370\.910\.940\.87yes0\.830\.340\.880\.870\.83MMLU\-PROno0\.630\.210\.470\.550\.71yes0\.550\.180\.420\.490\.61MUSRno0\.700\.520\.630\.620\.75yes0\.700\.540\.620\.590\.73RGBno0\.800\.530\.690\.630\.82yes0\.320\.310\.340\.390\.38TruthfulQAno0\.610\.450\.700\.670\.69yes0\.600\.510\.780\.680\.74WMT 2014 \(En\-De\)no0\.300\.260\.240\.170\.30yes0\.270\.240\.190\.160\.27![Refer to caption](https://arxiv.org/html/2609.11335v1/anonymisation_difference_heatmap.png)Figure 2:Difference in benchmark performance scores \(*anonymized*minus*non\-anonymized*\) across multiple language models and tasks\. Negative values indicate performance drops due to anonymization\.Figure[3](https://arxiv.org/html/2609.11335#S4.F3)displays the average performance across all benchmarks for each model\. While Qwen2\.5\-72B and GPT\-4o mini achieve the highest average scores, they also exhibit the largest average performance degradation when inputs are anonymized\. This suggests that more capable models might rely more heavily on specific entity information, making them more susceptible to the information loss caused by anonymization\. For instance, Qwen2\.5\-72B’s average performance drops by 6\.9 percentage points, while Teuken\-7B, the lowest\-performing model, sees a smaller average drop of 2\.3 percentage points\.

The impact of anonymization varies considerably across benchmarks\. Notably, TruthfulQA is the only benchmark where performance consistently*improves*with anonymization for most models \(e\.g\., an increase of 8 percentage points for Llama\-3\.1\)\. This suggests that removing named entities might reduce the model’s tendency to hallucinate or retrieve incorrect factual associations for this specific task\.

Conversely, the RGB benchmark, which tests retrieval\-augmented generation capabilities like noise robustness and information integration, suffers the most significant performance degradation\. For example, GPT\-4o mini’s score on RGB drops by approximately 48 percentage points \(from 0\.80 to 0\.32\)\. This substantial decrease highlights RGB’s reliance on precise entity information for its complex reasoning and information synthesis requirements\.

Figure 3:Average performance of the five different models under anonymized and non\-anonymized conditions\. For all models tested, the non\-anonymized version shows equal or slightly higher average performance\.Table 6:Average entity statistics across all benchmarks\. For each benchmark, we report: \(1\) the average number of total entities \(including repetitions\), \(2\) the average number of unique entities, and \(3\) the average length of each input \(in tokens\)\.BenchmarkAvg\. Total EntitiesAvg\. Unique EntitiesAvg\. LengthARC0\.830\.6197\.65BIG\-Bench Hard52\.4310\.15840\.92EQ\-Bench17\.302\.75405\.27HellaSwag0\.950\.61135\.32IFEval0\.670\.6144\.94MedQA2\.532\.27174\.54MMLU\-PRO7\.434\.80395\.25MUSR67\.478\.691317\.20RGB79\.4935\.261000\.23TruthfulQA12\.687\.67170\.99WMT 20144\.202\.1055\.83Table[6](https://arxiv.org/html/2609.11335#S4.T6)provides statistics on the average number of entities \(total and unique\) and input lengths across benchmarks, which help contextualize the varying impact of anonymization\. While benchmarks like RGB and MUSR exhibit high entity counts and long inputs, only RGB experiences a substantial performance drop after anonymization\. MUSR, despite similarly high entity statistics, shows minimal performance degradation\. Conversely, benchmarks with few entities and short inputs, such as HellaSwag and IFEval, display only small performance drops, aligning with expectations\. However, benchmarks like MMLU\-PRO and MedQA show noticeable degradation despite moderate entity counts, suggesting that entity quantity alone does not determine impact\. To investigate this further, we conducted correlation analyses between performance drops and each of the reported metrics \(total entities, unique entities, and input length\), but found no statistically significant correlations\. This indicates that anonymization effects are likely mediated by task\-specific dependencies on entity semantics and context, rather than raw input characteristics\.

### 4\.2Comparing Different Anonymization Techniques

We compare five anonymization techniques \(redaction, generalization, pseudonymization, randomization, and masking\), as described in Section[3\.4\.2](https://arxiv.org/html/2609.11335#S3.SS4.SSS2)\. This experiment is conducted using GPT\-4o mini on a subset of three benchmarks: MedQA, MUSR, and RGB, chosen for their diverse characteristics\.

Figure[4](https://arxiv.org/html/2609.11335#S4.F4)shows the performance of GPT\-4o mini under different anonymization methods\. Reversible techniques \(pseudonymization, randomization, and masking\), which allow for a potential one\-to\-one mapping back to original entities, generally preserve performance better than irreversible ones \(redaction, generalization\)\.

![Refer to caption](https://arxiv.org/html/2609.11335v1/anonymisation_types_performance_heatmap.png)Figure 4:Performance of GPT\-4o mini on MedQA, MUSR, and RGB, comparing redaction, generalization, pseudonymization, randomization, and masking\.Generalization \(e\.g\., “London”→\\rightarrow“\[CITY\]”\) and Redaction \(e\.g\., “London”→\\rightarrow“”\) consistently resulted in the largest performance drops\. For instance, on MuSR, Generalization led to a score of 0\.58 and Redaction 0\.52, compared to 0\.70 for pseudonymization and 0\.73 for the original input\. This is likely because these methods remove too much information, making it difficult for the model to establish necessary entity distinctions and for evaluation scripts, which expect original entities after de\-anonymization, to score correctly\.

Masking \(e\.g\., “London”→\\rightarrow“Lo\*\*\*\*”\) and Randomization \(e\.g\., “London”→\\rightarrow“Xj92Lp”\) performed comparably to or slightly worse than Pseudonymization\. Masking performed well on MedQA and MUSR but saw a significant drop on RGB, possibly due to the nature of information required by RGB that simple partial string matching cannot satisfy\.

These results underscore that techniques preserving unique identifiers for distinct entities, even if abstract, are preferable for maintaining utility\.

### 4\.3Impact of Explicit Anonymization Prompts

Here, we investigate whether explicitly informing the LLM about input anonymization has an impact on performance\. Using GPT\-4o mini once again on MedQA, MUSR, and RGB, we prepend two types of instructional prefixes \(short and long, as detailed in Section[3\.4\.3](https://arxiv.org/html/2609.11335#S3.SS4.SSS3)\) to anonymized inputs and compared results against anonymized inputs without a prefix and original inputs\.

Figure[5](https://arxiv.org/html/2609.11335#S4.F5)summarises the findings\. Adding an explicit textual cue that the input has been anonymized did not yield any significant or consistent improvement in LLM performance across the tested benchmarks and prefixes\. The performance scores were very similar regardless of whether a short prefix, a long prefix, or no prefix was used with the anonymized input\. This suggests that current LLMs do not substantially alter their reasoning strategy based on such meta\-information, and the performance impact is primarily driven by the modification of content itself\.

![Refer to caption](https://arxiv.org/html/2609.11335v1/anonymisation_prefix_heatmap.png)Figure 5:Performance of GPT\-4o mini on MedQA, MUSR, and RGB with original inputs, anonymized inputs \(no prefix\), and anonymized inputs with short or long explicit prefixes\.
### 4\.4Benchmark Examples

In this subsection, we explore why we observe the performance differences shown in Table[5](https://arxiv.org/html/2609.11335#S4.T5), Figure[2](https://arxiv.org/html/2609.11335#S4.F2)to[4](https://arxiv.org/html/2609.11335#S4.F4), and described in the previous subsections by investigating various examples from the test sets of our studied datasets\. Note that some questions and answers may have been shortened for readability in Tables[7](https://arxiv.org/html/2609.11335#S4.T7)to[11](https://arxiv.org/html/2609.11335#S4.T11)\.

Table 7:Example 1 \(from the ARC Benchmark\)Original CaseAnonymized CaseQuestion:A solar eclipse occurs when
A\. the Moon blocks the Earth from the Sun\.
B\. the first four planets in the solar system are aligned\.
C\. Earth blocks the Moon from the Sun\.
D\. Earth’s shadow falls on the Sun\.
Predicted Answer:AScore:1\.0Question:A solar eclipse occurs when
A\. the Moon blocks the <LOC\>\-1 from the <LOC\>\-2\.
B\. the first four planets in the solar system are aligned\.
C\. <LOC\>\-1 blocks the Moon from the <LOC\>\-2\.
D\. <LOC\>\-1’s shadow falls on the <LOC\>\-2\.
Predicted Answer:DScore:0\.0Reference Answer:AIn the question about solar eclipses depicted in Table[7](https://arxiv.org/html/2609.11335#S4.T7), our anonymization tool replaced “Earth” and “Sun” with placeholders like<LOC\>\-1and<LOC\>\-2\. Therefore, the model was unable to ground its reasoning and selected an incorrect answer\. Both the correct and incorrect options appeared partially plausible when anonymized, demonstrating that removing key entities can make it impossible to identify the reference answer\.

Table 8:Example 2 \(from the TruthfulQA Benchmark\)Original CaseAnonymized CaseQuestion:Which country in Europe has the most Nobel Laureates in science?
Predicted Answer:As of my last knowledge update in Oct 2023, the country in Europe with the most Nobel Laureates in science is the United Kingdom\.Score:1\.0Question:Which country in <LOC\>\-4 has the most Nobel Laureates in science?
Predicted Answer:The country in <LOC\>\-4 with the most Nobel Laureates in science is <COUNTRY NAME\>\.Score:0\.0Reference Answer:The country with the most Nobel Laureates in science is the United Kingdom\.In Table[8](https://arxiv.org/html/2609.11335#S4.T8), the language model was asked which European country has the most Nobel Laureates in science\. The model initially answered “United Kingdom” correctly\. After anonymization, “Europe” was replaced with<LOC\>\-4, leading the model to generate a non\-existent placeholder\-style answer,<COUNTRY NAME\>\. This highlights a failure mode: the model does not adapt its reasoning to the anonymization scheme and may fabricate answers outside the tagging system\. Carefully designed prompts might mitigate this, but such correction would probably amount to informed guessing rather than robust understanding\.

Table 9:Example 3 \(from the MMLU\-Pro Benchmark\)Original CaseAnonymized CaseQuestion:Suppose yesterday the euro was trading on the foreign exchange market at $1\.36 U\.S\. and today it is trading at $1\.40 U\.S\. Which of the following is true?Options: A\. Interest rates in the United States have decreased\. B\. The U\.S\. dollar has appreciated\. C\. The U\.S\. dollar has appreciated and the euro has depreciated\. D\. The exchange rate has remained steady\. E\. The U\.S\. dollar has stayed the same and the euro has depreciated\. F\. The euro has appreciated\. G\. Interest rates in Europe have increased\. H\. The U\.S\. dollar has depreciated\. I\. The euro has depreciated\. J\. Interest rates in the United States have increased\.

Predicted Answer:To determine the correct answer, we analyze the change in the exchange rate of the euro against the U\.S\. dollar\. Yesterday, the euro was trading at $1\.36 and today it is trading at $1\.40\. This means that it now takes more U\.S\. dollars to purchase one euro, which indicates that the euro has become more expensive, or has appreciated in value relative to the U\.S\. dollar\. As a result, we can also infer that the U\.S\. dollar has depreciated since it now has less purchasing power compared to the euro\.Both F and H contain correct information based on our analysis, but the most straightforward answer related to the change in exchange rate is F\. Thus, the answer is \(F\)\.Score:1\.0

Question:Suppose yesterday the euro was trading on the foreign exchange market at <MONETARYVALUE\>\-1 U\.S\. and today it is trading at <MONETARYVALUE\>\-2 U\.S\. Which of the following is true?Options: A\. Interest rates in the <LOC\>\-1 have decreased\. B\. The U\.S\. dollar has appreciated\. C\. The U\.S\. dollar has appreciated and the euro has depreciated\. D\. The exchange rate has remained steady\. E\. The U\.S\. dollar has stayed the same and the euro has depreciated\. F\. The euro has appreciated\. G\. Interest rates in <LOC\>\-2 have increased\. H\. The U\.S\. dollar has depreciated\. I\. The euro has depreciated\. J\. Interest rates in the <LOC\>\-1 have increased\.

Predicted Answer:To analyze the change in the exchange rate, we need to look at the values given\. Yesterday, the euro was trading at $<MONETARYVALUE\>\-1 and today it is trading at $<MONETARYVALUE\>\-2 U\.S\. This implies that it now takes more U\.S\. dollars to buy one euro than it did yesterday\. Thus, the value of the euro has decreased relative to the U\.S\. dollar, meaning the euro has depreciated\.Since the euro has depreciated, it follows that the U\.S\. dollar has appreciated in value relative to the euro because you can now get more euros for each dollar\. Both B \(the U\.S\. dollar has appreciated\) and C \(the U\.S\. dollar has appreciated and the euro has depreciated\) are correct\. However, C is more descriptive as it captures both currency movements\.Therefore, the more complete correct answer is C\. The answer is \(C\)\.Score:0\.0Reference Answer:F\. The euro has appreciated\.In the question about the euro appreciating against the dollar, shown in Table[9](https://arxiv.org/html/2609.11335#S4.T9), concrete monetary values were replaced with<MONETARYVALUE\>\. The model incorrectly concluded that the euro depreciated\. Although the initial reasoning was correct, the absence of concrete numbers led to a reversed conclusion\. This example demonstrates that anonymization can introduce ambiguity that causes reasoning errors in quantitative tasks\.

Table 10:Example 4 \(from the MedQA Benchmark\)Original CaseAnonymized CaseQuestion:A 30\-year\-old male presents to his primary care physician complaining of infertility\. He and his wife have been trying to get pregnant for the past two years\. They have used fertility monitors and other aids without success\. A hysterosalpingogram in his wife was normal\. The patient has a history of cleft lip and recurrent upper respiratory infections as a child\. He was briefly hospitalized for severe pneumonia when he was 9\-years\-old\. His temperature is 98\.6∘F \(37∘C\), blood pressure is 120/85 mmHg, pulse is 90/min, and respirations are 18/min\. On examination, he is a healthy\-appearing male in no acute distress with fully developed reproductive organs\. Notably, cardiac auscultation is silent in the left 5th intercostal space at the midclavicular line\. This patient most likely has a mutation in which of the following classes of proteins?Options: A: Microtubule monomeric protein B: Transmembrane ion channel protein C: Retrograde cytoskeletal motor protein D: Anterograde cytoskeletal motor protein

Answer:This clinical scenario and the details provided suggest a likely underlying genetic condition, specifically related to the patient’s history of cleft lip and recurrent respiratory infections, along with the cardiac finding of a silent auscultation area\.Considering the context, the patient probably has a mutation affecting the function of cilia or flagella, which can lead to both respiratory issues \(due to impaired mucociliary clearance\) and infertility \(due to issues with sperm motility in conditions like immotile cilia syndrome\)\. Conditions that affect cilia function often involve defects in axonemal proteins\.

Among the answer choices, the most relevant class of proteins related to ciliary function, and particularly sperm motility, would be:

D: Anterograde cytoskeletal motor protein\.

This option is appropriate because anterograde transport is essential for the proper function of cilia, and defective motor proteins \(like dynein\) can result in the ciliary dysfunction seen in conditions that cause infertility and respiratory issues\.Score:0\.0

Question:A 30\-year\-old male presents to his primary care physician complaining of infertility\. He and Azalea Kahler have been trying to get pregnant for the past two years\. They have used fertility monitors and other aids without success\. A hysterosalpingogram in Azalea Kahler was normal\. The patient has a history of cleft lip and recurrent upper respiratory infections as a child\. He was briefly hospitalized for severe pneumonia when he was 9\-years\-old\. His temperature is 98\.6∘F \(37∘C\), blood pressure is 120/85 mmHg, pulse is 90/min, and respirations are 18/min\. On examination, he is a healthy\-appearing male in no acute distress with fully developed reproductive organs\. Notably, cardiac auscultation is silent in the left 5th intercostal space at the midclavicular line\. This patient most likely has a mutation in which of the following classes of proteins?Options: A: Microtubule monomeric protein B: <PROD\>\-1 C: <PROD\>\-2 D: <PROD\>\-3

Answer:\[…\] While the options "A", "B", "C", and "D" likely refer to specific genetic mutations, without the specific knowledge of what "<PROD\>\-1", "<PROD\>\-2", and "<PROD\>\-3" represent, it’s difficult to pinpoint the exact answer among the provided options\. However, proteins involved in ciliary structure and function are most relevant regarding the clinical scenario presented\. \[…\] Given this information, I lean towards the answer choice more closely aligning with cytoskeletal proteins or structural proteins necessary for development/function of reproductive systems or cilia\.Nonetheless, the answer cannot be definitively verified without additional context for "<PROD\>\-1", "<PROD\>\-2", and "<PROD\>\-3"\. Further details about these options would be needed for a precise answer\.

For now, if I had to choose based on the general implications of the question regarding infertility and developmental anomalies, optionA: Microtubule monomeric proteincould be a strong contender\. However, additional context is vital for accuracy\. Score:1\.0

Reference Answer:CIn Table[10](https://arxiv.org/html/2609.11335#S4.T10), a medical question about protein classes affected in a patient with infertility and Kartagener syndrome\-like symptoms was given to the language model\. Anonymization changed the available answer options by removing all but one of the options\. The model initially chose the wrong answer but selected the correct answer after anonymization due to the altered answer space\. This improvement is coincidental and reflects interaction effects rather than genuine reasoning improvement\.

Table 11:Example 5 \(from the BIG\-Bench Hard Benchmark\)Original CaseAnonymized CaseQuestion:The rover Spirit is a robotic probe that NASA has placed on Mars\. The gravitational attraction of Mars is approximately 62% less than that of Earth\. Compared to its measurements on Earth, on Mars the probe hasChoices: \- A: the same weight and the same mass\. \- B: a larger weight, but smaller mass\. \- C: a smaller mass and larger weight\. \- D: a smaller weight and the same mass\.

Predicted Answer:D: a smaller weight and the same mass\.Score:0\.0Question:The rover <PROD\>\-1 is a robotic probe that TerraSource has placed on <LOC\>\-1\. The gravitational attraction of <LOC\>\-1 is approximately 62% less than that of <LOC\>\-2\. Compared to its measurements on <LOC\>\-2, on <LOC\>\-1 the probe hasChoices: \- A: the same weight and the same mass\. \- B: a larger weight, but smaller mass\. \- C: a smaller mass and larger weight\. \- D: a smaller weight and the same mass\.

Predicted Answer:DScore:1\.0Reference Answer:DFinally, even if an answer is correct, evaluation scripts may expect a specific format\. In such cases, anonymization can change the score between the original and anonymized scenario, even if no entities were modified, as illustrated in Table[11](https://arxiv.org/html/2609.11335#S4.T11)\.

Taken together, these examples show that anonymization does not have a uniform effect on model performance\. It can obscure critical information, induce confusion and fabricated outputs, cause reasoning errors in quantitative tasks, or randomly improve results depending on how entities and options are replaced\. Therefore, systematic errors are difficult to identify, as the impact of anonymization is highly question\-, and thus dataset\-, dependent\.

## 5Conclusion

In this work, we systematically investigated the trade\-off between privacy and utility by evaluating the impact of input anonymization on LLM performance\. Our findings reveal a complex and nuanced relationship, confirming that while anonymization generally degrades performance, its effects are far from uniform and depend heavily on the model’s capability, the task’s nature, and the anonymization technique employed\.

A central observation is that more capable models, such as Qwen2\.5\-72B and GPT\-4o mini, while achieving higher absolute scores, also suffer the most significant performance drops\. This suggests a greater reliance on the specific entity information that anonymization removes, making them more vulnerable to this form of information loss\. The impact is also highly task\-dependent: performance on TruthfulQA unexpectedly improved for most models, likely due to a reduction in factual hallucination, whereas the retrieval\-heavy RGB benchmark saw a catastrophic decline, highlighting its sensitivity to precise entity context\.

Our comparative analysis of anonymization techniques demonstrates the clear superiority of reversible methods \(e\.g\., pseudonymization, randomization\) over irreversible ones \(e\.g\., redaction, generalization\), underscoring the importance of preserving entity uniqueness for model reasoning\. Furthermore, we found that explicitly prompting models about the anonymized nature of the input provided no discernible benefit, indicating that current models do not adapt their strategy based on such meta\-information alone\.

Ultimately, this research challenges the notion of anonymization as a simple, one\-size\-fits\-all solution for privacy in LLMs\. The results suggest that achieving privacy cannot be treated as a mere preprocessing step without considering the downstream task\. It calls for a shift towards developing privacy\-aware systems where the choice of anonymization strategy is co\-designed with the model and application in mind\. Future work should explore more granular anonymization controls, selectively targeting entity types based on task requirements\. Developing models that are inherently more robust to the removal of PII, or can be fine\-tuned to understand anonymized representations, presents another promising research direction\. As LLMs continue to evolve, ongoing evaluation will be crucial to track whether this privacy\-utility trade\-off changes with new architectures and capabilities\.

## Acknowledgments

This research has been partially funded by the Federal Ministry of Education and Research of Germany and the state of North\-Rhine Westphalia as part of the Lamarr\-Institute for Machine Learning and Artificial Intelligence\. Google Gemini 2\.5 Pro\[[44](https://arxiv.org/html/2609.11335#bib.bib39)\]and GPT\-4o\[[45](https://arxiv.org/html/2609.11335#bib.bib40)\]were employed to assist in refining and improving the text throughout all sections of this paper\. The authors retain full responsibility for the accuracy, integrity, and originality of the work\.

## References

- \[1\]R\. Doshi, K\. S\. Amin, P\. Khosla, S\. S\. Bajaj, S\. Chheang, and H\. P\. Forman\(2024\)Quantitative evaluation of large language models to streamline radiology report impressions: a multimodal retrospective analysis\.Radiology\.Cited by:[§1](https://arxiv.org/html/2609.11335#S1.p1.1)\.
- \[2\]J\. Cui, M\. Ning, Z\. Li, B\. Chen, Y\. Yan, H\. Li, B\. Ling, Y\. Tian, and L\. Yuan\(2024\)Chatlaw: a multi\-agent collaborative legal assistant with knowledge graph enhanced mixture\-of\-experts large language model\.External Links:2306\.16092,[Link](https://arxiv.org/abs/2306.16092)Cited by:[§1](https://arxiv.org/html/2609.11335#S1.p1.1)\.
- \[3\]T\. Deußer, D\. Leonhard, L\. Hillebrand, A\. Berger, M\. Khaled, S\. Heiden, T\. Dilmaghani, B\. Kliem, R\. Loitz, C\. Bauckhage,et al\.\(2023\)Uncovering inconsistencies and contradictions in financial reports using large language models\.InProc\. BigData,Cited by:[§1](https://arxiv.org/html/2609.11335#S1.p1.1)\.
- \[4\]A\. Berger, A\. H\. Bashir, D\. Berghaus, N\. Afsan, L\. Grigull, L\. Fendrich, H\. Högl, G\. Ernst, R\. Schmidt, D\. Bascom,et al\.\(2024\)Optimizing rare disease patient matching with large language models\.InProc\. BigData,Cited by:[§1](https://arxiv.org/html/2609.11335#S1.p1.1)\.
- \[5\]L\. Hillebrand, A\. Berger, T\. Deußer, T\. Dilmaghani, M\. Khaled, B\. Kliem, R\. Loitz, M\. Pielka, D\. Leonhard, C\. Bauckhage,et al\.\(2023\)Improving zero\-shot text matching for financial auditing with large language models\.InProc\. DocEng,Cited by:[§1](https://arxiv.org/html/2609.11335#S1.p1.1)\.
- \[6\]Y\. Yao, J\. Duan, K\. Xu, Y\. Cai, Z\. Sun, and Y\. Zhang\(2024\)A survey on large language model \(llm\) security and privacy: the good, the bad, and the ugly\.High\-Confidence Computing\.Cited by:[§1](https://arxiv.org/html/2609.11335#S1.p1.1)\.
- \[7\]B\. C\. Das, M\. H\. Amini, and Y\. Wu\(2025\)Security and privacy challenges of large language models: a survey\.ACM Computing Surveys\.Cited by:[§1](https://arxiv.org/html/2609.11335#S1.p1.1)\.
- \[8\]D\. Biesner, R\. Ramamurthy, R\. Stenzel, M\. Lübbering, L\. Hillebrand, A\. Ladi, M\. Pielka, R\. Loitz, C\. Bauckhage, and R\. Sifa\(2022\)Anonymization of german financial documents using neural network\-based language models with contextual word representations\.International Journal of Data Science and Analytics\.Cited by:[§1](https://arxiv.org/html/2609.11335#S1.p1.1),[§2](https://arxiv.org/html/2609.11335#S2.p1.1)\.
- \[9\]A\. Sepas, A\. H\. Bangash, O\. Alraoui, K\. El Emam, and A\. El\-Hussuna\(2022\)Algorithms to anonymize structured medical and healthcare data: a systematic review\.Frontiers in Bioinformatics\.Cited by:[§1](https://arxiv.org/html/2609.11335#S1.p1.1),[§2](https://arxiv.org/html/2609.11335#S2.p1.1)\.
- \[10\]T\. Deußer, M\. Hahnbück, T\. Uelwer, C\. Zhao, C\. Bauckhage, and R\. Sifa\(2025\)Resource\-efficient anonymization of textual data via knowledge distillation from large language models\.InProc\. COLING,Cited by:[§1](https://arxiv.org/html/2609.11335#S1.p1.1),[§2](https://arxiv.org/html/2609.11335#S2.p2.1),[§3\.4\.1](https://arxiv.org/html/2609.11335#S3.SS4.SSS1.p1.1),[§4\.1](https://arxiv.org/html/2609.11335#S4.SS1.p1.1)\.
- \[11\]P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord\(2018\)Think you have solved question answering? try arc, the ai2 reasoning challenge\.External Links:1803\.05457,[Link](https://arxiv.org/abs/1803.05457)Cited by:[§1](https://arxiv.org/html/2609.11335#S1.p2.1),[Table 1](https://arxiv.org/html/2609.11335#S3.T1.2.2.4),[Table 4](https://arxiv.org/html/2609.11335#S4.T4.2.2.4)\.
- \[12\]D\. Jin, E\. Pan, N\. Oufattole, W\. Weng, H\. Fang, and P\. Szolovits\(2021\)What disease does this patient have? a large\-scale open domain question answering dataset from medical exams\.Applied Sciences\.Cited by:[§1](https://arxiv.org/html/2609.11335#S1.p2.1),[Table 1](https://arxiv.org/html/2609.11335#S3.T1.2.7.4),[Table 4](https://arxiv.org/html/2609.11335#S4.T4.2.7.4)\.
- \[13\]S\. J\. Paech\(2024\)EQ\-bench: an emotional intelligence benchmark for large language models\.External Links:2312\.06281,[Link](https://arxiv.org/abs/2312.06281)Cited by:[§1](https://arxiv.org/html/2609.11335#S1.p2.1),[Table 1](https://arxiv.org/html/2609.11335#S3.T1.2.4.4),[Table 4](https://arxiv.org/html/2609.11335#S4.T4.2.4.4)\.
- \[14\]O\. Bojar, C\. Buck, C\. Federmann, B\. Haddow, P\. Koehn, J\. Leveling, C\. Monz, P\. Pecina, M\. Post, H\. Saint\-Amand,et al\.\(2014\)Findings of the 2014 workshop on statistical machine translation\.InProc\. WMT,Cited by:[§1](https://arxiv.org/html/2609.11335#S1.p2.1),[Table 1](https://arxiv.org/html/2609.11335#S3.T1.2.12.4),[Table 4](https://arxiv.org/html/2609.11335#S4.T4.2.12.4)\.
- \[15\]Qwen, A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang,et al\.\(2025\)Qwen2\.5 technical report\.External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115)Cited by:[item 1](https://arxiv.org/html/2609.11335#S1.I1.i1.p1.1),[Table 3](https://arxiv.org/html/2609.11335#S3.T3.2.6.4)\.
- \[16\]OpenAI\(2024\)GPT\-4o mini: advancing cost\-efficient intelligence\.Note:OpenAI BlogAccessed: 27/05/2025External Links:[Link](https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/)Cited by:[item 1](https://arxiv.org/html/2609.11335#S1.I1.i1.p1.1),[Table 3](https://arxiv.org/html/2609.11335#S3.T3.2.2.4)\.
- \[17\]P\. Lison, I\. Pilán, D\. Sánchez, M\. Batet, and L\. Øvrelid\(2021\)Anonymisation models for text data: state of the art, challenges and future directions\.InProc\. ACL\-IJCNLP,Cited by:[§2](https://arxiv.org/html/2609.11335#S2.p1.1)\.
- \[18\]A\. Oksanen, E\. Hyvönen, M\. Tamper, J\. Tuominen, H\. Ylimaa, K\. Löytynoja, M\. Kokkonen, and A\. Hietanen\(2022\)An anonymization tool for open data publication of legal documents\.InInternational Workshop on Artificial Intelligence Technologies for Legal Documents,Cited by:[§2](https://arxiv.org/html/2609.11335#S2.p2.1)\.
- \[19\]B\. Kleinberg, T\. Davies, and M\. Mozes\(2022\)Textwash – automated open\-source text anonymisation\.External Links:2208\.13081,[Link](https://arxiv.org/abs/2208.13081)Cited by:[§2](https://arxiv.org/html/2609.11335#S2.p2.1)\.
- \[20\]R\. Staab, M\. Vero, M\. Balunovic, and M\. Vechev\(2025\)Language models are advanced anonymizers\.InProc\. ICLR,Cited by:[§2](https://arxiv.org/html/2609.11335#S2.p2.1)\.
- \[21\]Z\. Liu, Y\. Huang, X\. Yu, L\. Zhang, Z\. Wu, C\. Cao, H\. Dai, L\. Zhao, Y\. Li, P\. Shu, F\. Zeng, L\. Sun, W\. Liu, D\. Shen, Q\. Li, T\. Liu, D\. Zhu, and X\. Li\(2023\)DeID\-GPT: zero\-shot medical text de\-identification by GPT\-4\.Note:arXiv PreprintExternal Links:[Link](https://arxiv.org/abs/2303.11032),2303\.11032Cited by:[§2](https://arxiv.org/html/2609.11335#S2.p2.1)\.
- \[22\]C\. Patsakis and N\. Lykousas\(2023\)Man vs the machine in the struggle for effective text anonymisation in the age of large language models\.Scientific Reports\.Cited by:[§2](https://arxiv.org/html/2609.11335#S2.p2.1)\.
- \[23\]I\. Ben Cheikh Larbi, A\. Burchardt, and R\. Roller\(2023\)Clinical text anonymization, its influence on downstream NLP tasks and the risk of re\-identification\.InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop,E\. Bassignana, M\. Lindemann, and A\. Petit \(Eds\.\),Dubrovnik, Croatia,pp\. 105–111\.External Links:[Link](https://aclanthology.org/2023.eacl-srw.11/),[Document](https://dx.doi.org/10.18653/v1/2023.eacl-srw.11)Cited by:[§2](https://arxiv.org/html/2609.11335#S2.p3.1)\.
- \[24\]T\. Vakili, A\. Henriksson, and H\. Dalianis\(2024\)End\-to\-end pseudonymization of fine\-tuned clinical bert models: privacy preservation with maintained data utility\.BMC Medical Informatics and Decision Making\.Cited by:[§2](https://arxiv.org/html/2609.11335#S2.p3.1)\.
- \[25\]P\. Basu, T\. S\. Roy, R\. Naidu, and Z\. Muftuoglu\(2021\)Privacy enabled financial text classification using differential privacy and federated learning\.InProc\. Workshop ECONLP,Cited by:[§2](https://arxiv.org/html/2609.11335#S2.p3.1)\.
- \[26\]O\. Yermilov, V\. Raheja, and A\. Chernodub\(2023\)Privacy\- and utility\-preserving NLP with anonymized data: a case study of pseudonymization\.InProce\. Workshop TrustNLP,Cited by:[§2](https://arxiv.org/html/2609.11335#S2.p3.1)\.
- \[27\]S\. Shahriar and R\. Dara\(2025\)Priv\-iq: a benchmark and comparative evaluation of large multimodal models on privacy competencies\.AI\.Cited by:[§2](https://arxiv.org/html/2609.11335#S2.p4.1)\.
- \[28\]J\. Sun, B\. Suleiman, I\. Ullah, and I\. Razzak\(2025\)Effectiveness of privacy\-preserving algorithms in llms: a benchmark and empirical analysis\.InProc\. WWW,Cited by:[§2](https://arxiv.org/html/2609.11335#S2.p4.1)\.
- \[29\]Z\. Zhang, L\. Lei, L\. Wu, R\. Sun, Y\. Huang, C\. Long, X\. Liu, X\. Lei, J\. Tang, and M\. Huang\(2024\)Safetybench: evaluating the safety of large language models\.InProc\. ACL,Cited by:[§2](https://arxiv.org/html/2609.11335#S2.p5.1)\.
- \[30\]M\. Andriushchenko, A\. Souly, M\. Dziemian, D\. Duenas, M\. Lin, J\. Wang, D\. Hendrycks, A\. Zou, J\. Z\. Kolter, M\. Fredrikson, Y\. Gal, and X\. Davies\(2025\)AgentHarm: a benchmark for measuring harmfulness of LLM agents\.InProc\. ICLR,Cited by:[§2](https://arxiv.org/html/2609.11335#S2.p5.1)\.
- \[31\]T\. Deußer, L\. Sparrenberg, A\. Berger, M\. Hahnbück, C\. Bauckhage, and R\. Sifa\(2025\)A survey on current trends and recent advances in text anonymization\.InProc\. DSAA,Cited by:[§2](https://arxiv.org/html/2609.11335#S2.p6.1)\.
- \[32\]C\. Fourrier, N\. Habib, A\. Lozovskaya, K\. Szafer, and T\. Wolf\(2024\)Open llm leaderboard v2\.Hugging Face\.Note:[https://huggingface\.co/spaces/open\-llm\-leaderboard/open\_llm\_leaderboard](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)Cited by:[§3\.2](https://arxiv.org/html/2609.11335#S3.SS2.p1.2)\.
- \[33\]P\. Liang, R\. Bommasani, T\. Lee, D\. Tsipras, D\. Soylu, M\. Yasunaga, Y\. Zhang, D\. Narayanan, Y\. Wu, A\. Kumar,et al\.\(2023\)Holistic evaluation of language models\.Transactions on Machine Learning Research\.Cited by:[§3\.2](https://arxiv.org/html/2609.11335#S3.SS2.p1.2)\.
- \[34\]M\. Suzgun, N\. Scales, N\. Schärli, S\. Gehrmann, Y\. Tay, H\. W\. Chung, A\. Chowdhery, Q\. Le, E\. Chi, D\. Zhou, and J\. Wei\(2023\)Challenging BIG\-bench tasks and whether chain\-of\-thought can solve them\.InFindings of the ACL: ACL,Cited by:[Table 1](https://arxiv.org/html/2609.11335#S3.T1.2.3.4),[Table 4](https://arxiv.org/html/2609.11335#S4.T4.2.3.4)\.
- \[35\]R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi\(2019\)HellaSwag: can a machine really finish your sentence?\.InProc\. ACL,Cited by:[Table 1](https://arxiv.org/html/2609.11335#S3.T1.2.5.4),[Table 4](https://arxiv.org/html/2609.11335#S4.T4.2.5.4)\.
- \[36\]J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. Hou\(2023\)Instruction\-following evaluation for large language models\.External Links:2311\.07911,[Link](https://arxiv.org/abs/2311.07911)Cited by:[Table 1](https://arxiv.org/html/2609.11335#S3.T1.2.6.4),[Table 4](https://arxiv.org/html/2609.11335#S4.T4.2.6.4)\.
- \[37\]Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang,et al\.\(2024\)MMLU\-Pro: a more robust and challenging multi\-task language understanding benchmark\.InProc\. NeurIPS,Cited by:[Table 1](https://arxiv.org/html/2609.11335#S3.T1.2.8.4),[Table 4](https://arxiv.org/html/2609.11335#S4.T4.2.8.4)\.
- \[38\]Z\. R\. Sprague, X\. Ye, K\. Bostrom, S\. Chaudhuri, and G\. Durrett\(2024\)MuSR: testing the limits of chain\-of\-thought with multistep soft reasoning\.InThe Twelfth International Conference on Learning Representations,Cited by:[Table 1](https://arxiv.org/html/2609.11335#S3.T1.2.9.4),[Table 4](https://arxiv.org/html/2609.11335#S4.T4.2.9.4)\.
- \[39\]J\. Chen, H\. Lin, X\. Han, and L\. Sun\(2024\)Benchmarking large language models in retrieval\-augmented generation\.InProc\. AAAI,Cited by:[Table 1](https://arxiv.org/html/2609.11335#S3.T1.2.10.4),[Table 4](https://arxiv.org/html/2609.11335#S4.T4.2.10.4)\.
- \[40\]S\. Lin, J\. Hilton, and O\. Evans\(2022\)TruthfulQA: measuring how models mimic human falsehoods\.InProc\. ACL,Cited by:[Table 1](https://arxiv.org/html/2609.11335#S3.T1.2.11.4),[Table 4](https://arxiv.org/html/2609.11335#S4.T4.2.11.4)\.
- \[41\]M\. Ali, M\. Fromm, K\. Thellmann, J\. Ebert, A\. A\. Weber, R\. Rutmann, C\. Jain, M\. Lübbering, D\. Steinigen, J\. Leveling,et al\.\(2024\)Teuken\-7b\-base & teuken\-7b\-instruct: towards european llms\.External Links:2410\.03730,[Link](https://arxiv.org/abs/2410.03730)Cited by:[Table 3](https://arxiv.org/html/2609.11335#S3.T3.2.3.4)\.
- \[42\]A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Yang, A\. Fan,et al\.\(2024\)The llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[Table 3](https://arxiv.org/html/2609.11335#S3.T3.2.4.4)\.
- \[43\]Gemma Team, M\. Riviere, S\. Pathak, P\. G\. Sessa, C\. Hardin, S\. Bhupatiraju, L\. Hussenot, T\. Mesnard, B\. Shahriari, A\. Ramé,et al\.\(2024\)Gemma 2: improving open language models at a practical size\.External Links:2408\.00118,[Link](https://arxiv.org/abs/2408.00118)Cited by:[Table 3](https://arxiv.org/html/2609.11335#S3.T3.2.5.4)\.
- \[44\]Gemini Team, R\. Anil, S\. Borgeaud, J\. Alayrac, J\. Yu, R\. Soricut, J\. Schalkwyk, A\. M\. Dai, A\. Hauth, K\. Millican,et al\.\(2025\)Gemini: a family of highly capable multimodal models\.External Links:2312\.11805Cited by:[Acknowledgments](https://arxiv.org/html/2609.11335#Sx1.p1.1)\.
- \[45\]OpenAI, A\. Hurst, A\. Lerer, A\. P\. Goucher, A\. Perelman, A\. Ramesh, A\. Clark, A\. Ostrow, A\. Welihinda, A\. Hayes, A\. Radford,et al\.\(2024\)GPT\-4o system card\.External Links:2410\.21276Cited by:[Acknowledgments](https://arxiv.org/html/2609.11335#Sx1.p1.1)\.