Redakto - The Incognito Tab for LLMs

# Redakto – The Incognito Tab for LLMs
Source: [https://arxiv.org/html/2608.18260](https://arxiv.org/html/2608.18260)
Saurav Kumar SahaOrcID:[0009\-0002\-9812\-4150](https://orcid.org/0009-0002-9812-4150)Affiliation:Berlin University of Applied Sciences, Luxemburger Str\. 10, 13353 Berlin, Germany

[https://www\.bht\-berlin\.de](https://www.bht-berlin.de/)

E\-mail[\{sauravkumar\.saha,tom\.roehr,felix\.biessmann\}@bht\-berlin\.de](mailto:{sauravkumar.saha,tom.roehr,felix.biessmann}@bht-berlin.de)Tom RöhrOrcID:[0009\-0007\-7448\-8941](https://orcid.org/0009-0007-7448-8941)Affiliation:Berlin University of Applied Sciences, Luxemburger Str\. 10, 13353 Berlin, Germany[https://www\.bht\-berlin\.de](https://www.bht-berlin.de/)

E\-mail[\{sauravkumar\.saha,tom\.roehr,felix\.biessmann\}@bht\-berlin\.de](mailto:{sauravkumar.saha,tom.roehr,felix.biessmann}@bht-berlin.de)Felix BießmannOrcID:[0000\-0002\-3422\-1026](https://orcid.org/0000-0002-3422-1026)Affiliation:Berlin University of Applied Sciences, Luxemburger Str\. 10, 13353 Berlin, Germany[https://www\.bht\-berlin\.de](https://www.bht-berlin.de/)

E\-mail[\{sauravkumar\.saha,tom\.roehr,felix\.biessmann\}@bht\-berlin\.de](mailto:{sauravkumar.saha,tom.roehr,felix.biessmann}@bht-berlin.de)Affiliation:Einstein Center Digital Future, Wilhelmstraße 67, 10117 Berlin, Germany
[https://www\.digital\-future\.berlin](https://www.digital-future.berlin/)
###### Abstract

Large Language Models \(LLMs\) are being increasingly used in everyday applications\. A major challenge in the context of LLMs or Artificial Intelligence \(AI\) in general is to ensure privacy when using them, meaning that personally identifiable information \(PII\) is removed from any text that enters an LLM\. These challenges have become more urgent with novel EU legislation\. Uncertainty around LLM usage with respect to privacy concerns in EU countries can be a major blocker for the speed of innovation and transfer from research to applications\. Here we presentRedakto, a tool that can be used for anonymizing text prior to feeding it to an LLM or other downstream text processing\. We provide state\-of\-the\-art functionalities for both redaction of PII but also when used for pseudonymization\. These functionalities are exposed such that they can easily be used by end\-users, through the Redakto web application, and by developers and researchers, via REST APIs and model context protocol \(MCP\) hooks\. The implementation is fully open source, requires modest compute resources, and can be readily deployed on local hardware\. In contrast to prior work and in order to better assess the quality of the anonymized texts, we conduct extensive empirical evaluations on textual data from legal and medical domain with respect to both privacy and utility of the redacted texts\. Our empirical results demonstrate that the texts anonymized with different redaction strategies achieve utility scores on par with the original texts, suggesting that anonymization with Redakto can be used for LLM tasks without substantial negative impact for the tasks we explored\.

###### Keywords:

Privacy\-Preserving NLP PII Detection Text Anonymization Privacy–Utility Trade\-off GDPR Compliance

## 1Introduction

In an era where large language models \(LLMs\) are transforming how we process and analyze text, the need to safeguard sensitive information has never been more critical\. While these models bear potential for research and applications, the risk of inadvertently exposing personally identifiable information \(PII\) remains a significant barrier to their safe and ethical use\. This challenge is particularly acute in fields such as public administration, healthcare, or legal studies, where the analysis of confidential documents is routine\. To foster more responsible usage of LLMs in these domains and beyond, we present a novel software demonstrator designed to automatically detect and redact PII from text, enabling users to confidently share sanitized content with LLM\-based chatbots or other downstream applications where privacy is important\. Extending prior work on practical implementations of anonymizations, we develop evaluation protocols to evaluate the effectiveness of different redaction strategies such as semantic label masking, random masking etc\. with respect to utility\. Our experimental validations on a variety of textual tasks demonstrate that our solution not only preserves privacy but can also be used to maintain the integrity and utility of the original text, paving the way for secure and responsible AI\-assisted analysis\. In summary, this study makes the following contributions:

- •Redaktoweb application and APIs \(REST & MCP\)
- •Privacy evaluation of Redakto models
- •Utility evaluation of redacted texts in downstream tasks

## 2Redaktoweb application and APIs

The Redakto models are exposed in three different ways to make them as easy to use as possible for end users, developers, and researchers\. More concretely, we provide a readily usable web application as well as REST APIs and a Model Context Protocol \(MCP\) server which are hosted on institutional computing infrastructure\. All components are containerized and can be self\-hosted \(e\.g\., on Kubernetes clusters\) and the MCP server can be integrated within MCP\-compatible LLM assistants and agent frameworks, enabling privacy\-preserving redaction as a native tool inside automated LLM workflows instead of just prompting\. A brief overview of the system design and implementation architecture of Redakto is provided in Appendix[0\.B](https://arxiv.org/html/2608.18260#Pt0.A2)\.

### 2\.1Web Application

### 2\.2REST APIs

The REST API endpoints are documented and exposed with Swagger UI within the Redakto web application222[https://redakto\.demo\.calgo\-lab\.de/api/docs](https://redakto.demo.calgo-lab.de/api/docs)\. The first main endpoint performs named entity recognition over one or more input texts while the second one extends this functionality by additionally generating pseudonymized versions of the input texts\. Both endpoints support coarse\- and fine\-grained entity labeling\.

### 2\.3MCP Server

We additionally provide an MCP \(Model Context Protocol\) server, distributed as an npm package333[https://www\.npmjs\.com/package/@sksdotsauravs/redakto\-app\-mcp\-server](https://www.npmjs.com/package/@sksdotsauravs/redakto-app-mcp-server), that exposes Redakto’s functionalities as structured tools for LLM assistants and agent frameworks\. Through this interface, LLM assistants such as Claude can invoke schema\-defined tool calls for entity detection and pseudonymization, instead of relying on prompt\-only processing\. This enables privacy\-aware text transformation to be programmatically integrated into agentic workflows, developer tooling, and interactive AI systems\.

## 3Privacy Evaluation

In this section, we evaluate the anonymization performance quantitatively\. To demonstrate the robustness and domain generalization of Redakto’s anonymization capabilities, we performed experiments on three datasets from distinct domains: CodE Alltag \(emails\)444[https://github\.com/codealltag/CodEAlltag](https://github.com/codealltag/CodEAlltag), GraSCCo \(clinical texts\)555[https://zenodo\.org/records/15747389](https://zenodo.org/records/15747389)and LER \(legal documents\)666[https://huggingface\.co/datasets/elenanereiss/german\-ler](https://huggingface.co/datasets/elenanereiss/german-ler)\. In summary the transformer models optimized for the Redakto demonstrator reach macro F1≈\\approx0\.95 for PII detection when trained on domains with abundant annotated data \(e\.g\., CodE Alltag, LER\)\. For the low\-resource clinical GraSCCo corpus performance is lower, in line with prior work such as\[[2](https://arxiv.org/html/2608.18260#bib.bib9)\], which reports up to 0\.95 macro F1 using thousands of in\-domain clinical documents from University Hospital Essen \(with neither data nor models publicly available\), highlighting that data availability, annotation quality, and entity coverage \- rather than model architecture \- are the main bottlenecks for privacy\-aware named entity recognition \(NER\)\.

### 3\.1CodE Alltag \- German Email Text

Building upon the prior work of PII detection and pseudonymization of German\-language text using the CodE Alltag corpus\[[7](https://arxiv.org/html/2608.18260#bib.bib1),[4](https://arxiv.org/html/2608.18260#bib.bib2),[5](https://arxiv.org/html/2608.18260#bib.bib3),[6](https://arxiv.org/html/2608.18260#bib.bib4),[14](https://arxiv.org/html/2608.18260#bib.bib17)\], we extend our experiments with a substantially larger sample size and a broader range of transformer\-based model architectures for personally identifiable information \(PII\) detection\. The full CodE Alltag corpus comprises approximately 1\.5 million email messages, representing diverse instances of naturally occurring PII in informal and semi\-formal communication\. To obtain robust and reliable performance estimates, we adopt a 5\-fold cross\-validation setup over a subset of 175 thousand email texts using a 60/20/20 train–development–test split\. Further details on the training setup, model configurations, evaluation protocol, and complete experimental results are provided in the project repository\.777[https://github\.com/calgo\-lab/redakt\-codealltag](https://github.com/calgo-lab/redakt-codealltag)

Table 1:PII\-detection performance on CodE Alltag
Figure 1:PII\-detection performance on GraSCCoTable 2:PII\-detection performance on LER

We fine\-tune and compare three transformer\-based models, evaluating their effectiveness in detecting 14 categories of PII in German email text\.[Table 1](https://arxiv.org/html/2608.18260#S3.T1)shows comparisons of macro averaged performance on the test sets with mean and standard deviation across the five cross\-validation folds\.

### 3\.2GraSCCo \- German Clinical Text

To study PII detection in German clinical documents, we make use of the Graz Synthetic Clinical Corpus \(GraSCCo\)\[[11](https://arxiv.org/html/2608.18260#bib.bib5),[10](https://arxiv.org/html/2608.18260#bib.bib6)\]\. Given the small size of the GraSCCo corpus \(1,439 annotated private entities across 19 PII label types\), we explored transfer learning using PII detection model checkpoints previously fine\-tuned on the CodE Alltag corpus\. The task\-specific classification heads were replaced to predict GraSCCo PII labels, and the models were further fine\-tuned on the clinical corpus\.[Figure 1](https://arxiv.org/html/2608.18260#S3.F1)summarizes the micro\-averaged evaluation results for all models on the GraSCCo test splits of five cross\-validation folds\. We also list here the GitHub repository888[https://github\.com/calgo\-lab/redakt\-grascco](https://github.com/calgo-lab/redakt-grascco)and other resources999[https://grascco\.demo\.calgo\-lab\.de](https://grascco.demo.calgo-lab.de/)for more information\.

### 3\.3LER \- German Legal Text

To broaden the privacy\-oriented evaluation beyond email and clinical text, we additionally fine\-tuned the same three model architectures on the German Legal Entity Recognition \(LER\) dataset\[[9](https://arxiv.org/html/2608.18260#bib.bib7),[8](https://arxiv.org/html/2608.18260#bib.bib8)\]\. Although LER is not a dedicated PII corpus, it contains a substantial proportion of entities that overlap with privacy\-relevant categories, namely persons, locations, and organizations, which together account for 25\.66% of all annotated entities\. The remaining 74\.34% correspond to types of entities specific to the legal\-domain, such as legal norms, regulations, court decisions etc\. This mixture allows us to assess how robustly models can identify classical personal identifiers when they appear alongside dense domain\-specific terminology\. We adopt the original train–development–test split provided with the dataset and do not introduce additional resampling or cross\-validation\.[Table 2](https://arxiv.org/html/2608.18260#S3.T2)depicts the re\-calculated macro averaged performance of the models for detecting entities of 12 PII labels present on test samples of LER dataset\.

## 4Utility Evaluation

We study the utility of anonymized texts by training models on original texts and evaluating classification performance on redacted texts as an extension of a prior work by\[[12](https://arxiv.org/html/2608.18260#bib.bib10)\]with two new tasks, medical intent classification and legal violation prediction\. Across these domains, we compare utility by replacing PII with semantic placeholders, random and generic masks to analyze how different anonymization strategies influence loss of task specific linguistic information, model robustness, and downstream utility\. For additional experimental details and to facilitate reproducibility we refer readers to the project GitHub repository101010[https://github\.com/calgo\-lab/redacted\-text\-utility](https://github.com/calgo-lab/redacted-text-utility)\.

### 4\.1Medical Intent Classification

For our first experiment, we use the Medical Intent Classification \(MIC\) dataset111111[https://huggingface\.co/datasets/DATEXIS/med˙intent˙classification](https://huggingface.co/datasets/DATEXIS/med_intent_classification)introduced by\[[13](https://arxiv.org/html/2608.18260#bib.bib11)\]where the researchers study on physician intent trajectories in doctor–patient dialogues\. The dataset is derived from the Ambient Clinical Intelligence Benchmark \(ACI\-Bench\)\[[17](https://arxiv.org/html/2608.18260#bib.bib12)\]corpus and contains 5,541 physician turns annotated with 20 fine\-grained intent taxonomy\. Each sample consists of a single physician utterance, and the objective is to predict one or more associated medical intents, making this a multi\-label classification problem\. We use a model121212[https://huggingface\.co/flair/ner\-english\-ontonotes\-large](https://huggingface.co/flair/ner-english-ontonotes-large)finetuned with Flair\[[1](https://arxiv.org/html/2608.18260#bib.bib14),[15](https://arxiv.org/html/2608.18260#bib.bib15)\]on OntoNotes\[[16](https://arxiv.org/html/2608.18260#bib.bib13)\], an English NER corpus131313[https://catalog\.ldc\.upenn\.edu/LDC2013T19](https://catalog.ldc.upenn.edu/LDC2013T19), to identify and redact private entities \(PERSON, DATE, GPE, ORG\), with additional filtering to remove spurious detections\.

Table 3:Examples of redaction strategies applied to a clinical text
Figure 2:Medical Intent classification on anonymized textsTable 4:Binary Violation prediction on anonymized texts

We fine\-tune three different transformer models for the multi\-label intent classification downstream task and report performance metrics for each redaction \([Table 3](https://arxiv.org/html/2608.18260#S4.T3)\) condition in[Figure 2](https://arxiv.org/html/2608.18260#S4.F2)where error bars denote performance variability across the five cross\-validation folds\.

### 4\.2Binary Violation Prediction

We conduct a second utility evaluation on the European Court of Human Rights \(ECHR\) dataset141414[https://huggingface\.co/datasets/glnmario/ECHR](https://huggingface.co/datasets/glnmario/ECHR)introduced by\[[3](https://arxiv.org/html/2608.18260#bib.bib16)\]\. In line with the original work, we restrict our experiments to the Binary Violation Prediction \(BVP\) task, i\.e\., determining whether any human rights article of the European Convention on Human Rights was violated\. We utilize an experimental setup similar to that used for the MIC task \(same redaction model and strategies\)\. Three transformer\-based document classifiers are fine\-tuned using the Flair framework with long\-sequence support\.[Table 4](https://arxiv.org/html/2608.18260#S4.T4)shows macro averaged performance of these models for different redaction strategies evaluated on the test set to measure the utility impact of privacy\-driven sanitization\. Our results demonstrate that redaction does not impact utility substantially\. The least effect of redaction is observed with semantic label masking\. We present a more in\-depth analysis, controlling for the number of PII entities in a text for this task in Appendix[0\.A](https://arxiv.org/html/2608.18260#Pt0.A1)\. In the majority of the experimental conditions investigated, redaction appears to have limited impact on the utility of the anonymized texts\.

## 5Conclusion

Our Redakto demonstrator provides a web application, REST APIs and MCP hooks for direct usage inside an LLM\. In contrast to other implementations of redaction tools for LLM usage we combine the implementation with comprehensive evaluations on redaction performance and utility of redacted texts\. Empirical evaluations demonstrate not only that Redakto reliably anonymizes texts across a variety of domains, our results also show that the utility of the anonymized texts is not impacted substantially in the majority of cases, even when controlling for the number of PII entities in a given text\. These findings highlight the potential of Redakto for researchers, practitioners and every\-day usage\.

## Acknowledgements

This research was supported by the German Federal Ministry of Research, Technology and Space grant numbers 16SV8857, by the Einstein Center Digital Future, Berlin, and by the German Research Foundation \(DFG\) \- Project number: 528483508 \- FIP 12\.

## References

- \[1\]\(2019\)FLAIR: an easy\-to\-use framework for state\-of\-the\-art NLP\.InProceedings of NACL,W\. Ammar, A\. Louis, and N\. Mostafazadeh \(Eds\.\),pp\. 54–59\.External Links:[Document](https://dx.doi.org/10.18653/v1/N19-4010)Cited by:[§4\.1](https://arxiv.org/html/2608.18260#S4.SS1.p1.1)\.
- \[2\]K\. Arzideh, G\. Baldini, P\. Winnekens, C\. M\. Friedrich, F\. Nensa, A\. Idrissi\-Yaghir, and R\. Hosch\(2025\)A Transformer\-Based Pipeline for German Clinical Document De\-Identification\.Applied Clinical Informatics,pp\. 31–43\.External Links:[Document](https://dx.doi.org/10.1055/a-2424-1989)Cited by:[§3](https://arxiv.org/html/2608.18260#S3.p1.1)\.
- \[3\]I\. Chalkidis, I\. Androutsopoulos, and N\. Aletras\(2019\)Neural Legal Judgment Prediction in English\.InProceedings of ACL 2019,pp\. 4317–4323\.External Links:[Document](https://dx.doi.org/10.18653/v1/P19-1424)Cited by:[§4\.2](https://arxiv.org/html/2608.18260#S4.SS2.p1.1)\.
- \[4\]E\. Eder, U\. Krieg\-Holz, and U\. Hahn\(2019\)De\-Identification of Emails: Pseudonymizing Privacy\-Sensitive Data in a German Email Corpus\.InProceedings of RANLP,Varna, Bulgaria,pp\. 259–269\.External Links:[Link](https://aclanthology.org/R19-1030/),[Document](https://dx.doi.org/10.26615/978-954-452-056-4%5F030)Cited by:[§3\.1](https://arxiv.org/html/2608.18260#S3.SS1.p1.1)\.
- \[5\]E\. Eder, U\. Krieg\-Holz, and U\. Hahn\(May 11\-16, 2020\)CodE Alltag 2\.0: A Pseudonymized German\-Language Email Corpus\.InLREC 2020 – Proceedings of the 12th International Conference on Language Resources and Evaluation,Marseille, France,pp\. 4466–4477\.Cited by:[§3\.1](https://arxiv.org/html/2608.18260#S3.SS1.p1.1)\.
- \[6\]E\. Eder, M\. Wiegand, U\. Krieg\-Holz, and U\. Hahn\(2022\)“Beste Grüße, Maria Meyer” — Pseudonymization of Privacy\-Sensitive Information in Emails\.InLREC 2022 \- Proceedings of the 13th Language Resources and Evaluation Conference,Marseille, France,pp\. 741–752\.External Links:[Link](https://aclanthology.org/2022.lrec-1.79/)Cited by:[§3\.1](https://arxiv.org/html/2608.18260#S3.SS1.p1.1)\.
- \[7\]U\. Krieg\-Holz, C\. Schuschnig, F\. Matthies, B\. Redling, and U\. Hahn\(2016\)CodE Alltag: A German\-Language E\-Mail Corpus\.InLREC 2016 \- Proceedings of the 10th International Conference on Language Resources and Evaluation,Portorož, Slovenia,pp\. 2543–2550\.External Links:[Link](https://aclanthology.org/L16-1404/)Cited by:[§3\.1](https://arxiv.org/html/2608.18260#S3.SS1.p1.1)\.
- \[8\]E\. Leitner, G\. Rehm, and J\. Moreno\-Schneider\(2019\)Fine\-Grained Named Entity Recognition in Legal Documents\.InSEMANTiCS 2019 \- Semantic Systems: The Power of AI and Knowledge Graphs,Lecture Notes in Computer Science, Vol\.11702,Cham,pp\. 272–287\.External Links:[Document](https://dx.doi.org/10.1007/978-3-030-33220-4%5F20),ISBN 978\-3\-030\-33220\-4Cited by:[§3\.3](https://arxiv.org/html/2608.18260#S3.SS3.p1.1)\.
- \[9\]E\. Leitner, G\. Rehm, and J\. Moreno\-Schneider\(May 11\-16, 2020\)A Dataset of German Legal Documents for Named Entity Recognition\.InLREC 2020 – Proceedings of the 12th International Conference on Language Resources and Evaluation,Marseille, France,pp\. 4478–4485\(eng\)\.External Links:[Link](https://aclanthology.org/2020.lrec-1.551/),ISBN 979\-10\-95546\-34\-4Cited by:[§3\.3](https://arxiv.org/html/2608.18260#S3.SS3.p1.1)\.
- \[10\]C\. Lohr, F\. Matthies, J\. Faller, L\. Modersohn, A\. Riedel, U\. Hahn, R\. Kiser, M\. Boeker, and F\. Meineke\(2024\)De\-Identifying GRASCCO – A Pilot Study for the De\-Identification of the German Medical Text Project \(GeMTeX\) Corpus\.InGerman Medical Data Sciences 2024,Studies in Health Technology and Informatics, Vol\.317,pp\. 171–179\.External Links:[Document](https://dx.doi.org/10.3233/SHTI240853)Cited by:[§3\.2](https://arxiv.org/html/2608.18260#S3.SS2.p1.1)\.
- \[11\]L\. Modersohn, S\. Schulz, C\. Lohr, and U\. Hahn\(2022\)GRASCCO – The First Publicly Shareable, Multiply\-Alienated German Clinical Text Corpus\.InStudies in Health Technology and Informatics,Vol\.296,pp\. 66–72\.External Links:[Document](https://dx.doi.org/10.3233/SHTI220805)Cited by:[§3\.2](https://arxiv.org/html/2608.18260#S3.SS2.p1.1)\.
- \[12\]A\. Pal, R\. Bhargava, K\. Hinsz, J\. Esterhuizen, and S\. Bhattacharya\(2024\)The Empirical Impact of Data Sanitization on Language Models\.arXiv preprint arXiv:2411\.05978\.Cited by:[§4](https://arxiv.org/html/2608.18260#S4.p1.1)\.
- \[13\]T\. Röhr, S\. Roy, F\. Al Mohamad, J\. Papaioannou, W\. Nejdl, F\. Gers, and A\. Löser\(2025\)“Where does it hurt?” – Dataset and Study on Physician Intent Trajectories in Doctor–Patient Dialogues\.External Links:2508\.19077,[Link](https://arxiv.org/abs/2508.19077)Cited by:[§4\.1](https://arxiv.org/html/2608.18260#S4.SS1.p1.1)\.
- \[14\]S\. K\. Saha and F\. Biessmann\(2025\)End\-to\-end Pseudonymization of German Texts with Deep Learning – An Empirical Comparison of Classical and Modern Approaches\.InProceedings of AiMH\),S\. Y\. Yurish \(Ed\.\),pp\. 98–104\.External Links:ISBN 978\-84\-09\-71190\-1,ISSN 2938\-5350,[Document](https://dx.doi.org/10.13140/RG.2.2.20387.77606)Cited by:[§3\.1](https://arxiv.org/html/2608.18260#S3.SS1.p1.1)\.
- \[15\]S\. Schweter and A\. Akbik\(2020\)FLERT: Document\-Level Features for Named Entity Recognition\.External Links:2011\.06993Cited by:[§4\.1](https://arxiv.org/html/2608.18260#S4.SS1.p1.1)\.
- \[16\]R\. Weischedel, E\. Hovy, M\. Marcus, M\. Palmer, R\. Belvin, S\. Pradhan, L\. Ramshaw, and N\. Xue\(2011\)OntoNotes: A Large Training Corpus for Enhanced Processing\.InHandbook of Natural Language Processing and Machine Translation: DARPA Global Autonomous Language Exploitation,Cited by:[§4\.1](https://arxiv.org/html/2608.18260#S4.SS1.p1.1)\.
- \[17\]W\. Yim, Y\. Fu, A\. Ben Abacha, N\. Snider, T\. Lin, and M\. Yetisgen\(2023\)ACI\-BENCH: A Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generation\.arXiv preprint arXiv:2306\.02022\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2306.02022),[Link](https://arxiv.org/abs/2306.02022)Cited by:[§4\.1](https://arxiv.org/html/2608.18260#S4.SS1.p1.1)\.

## Appendix 0\.AAdditional Experimental Results

The ECHR dataset used for one of our utility evaluation tasks \([4\.2](https://arxiv.org/html/2608.18260#S4.SS2)\) represents a particularly interesting setting for some additional experiments \- after preprocessing and sample selection across all the test folds, documents contain on average approximately 2000 tokens and 86 private entities, with most entity\-dense documents containing several hundred PII entities\. This makes the dataset well suited for evaluating whether and by how much extensive redaction affects downstream model performance\.

Figure 3:Entity density wise performance \(Macro\-F1\) comparison for different redaction strategies \(xlm\-roberta\-large\)To investigate the robustness of different redaction strategies under increasing amounts of sensitive information, we perform an additional analysis based on entity density\. For each cross\-validation fold, test documents are ranked according to their number of detected entities and partitioned into four percentile ranges \(0–25, 25–50, 50–75, and 75–100\)\. Classification performance is then evaluated separately for each subset for all fine\-tuned binary violation prediction models and for each redaction strategy\.

Figure 4:Entity density wise performance \(Macro\-F1\) comparison for different redaction strategies \(bert\-large\-cased\)This experiment enables a fine\-grained assessment of utility preservation under progressively more intense redaction scenarios\. Since higher entity\-count percentiles require substantially larger portions of a document to be transformed, performance trends across percentile ranges provide insight into whether redaction introduces additional degradation beyond the inherent difficulty of processing longer documents\.

Figure 5:Entity density wise performance \(Macro\-F1\) comparison for different redaction strategies \(electra\-large\-discriminator\)We present the results of this experiment with[Figure 3](https://arxiv.org/html/2608.18260#Pt0.A1.F3),[Figure 4](https://arxiv.org/html/2608.18260#Pt0.A1.F4)and[Figure 5](https://arxiv.org/html/2608.18260#Pt0.A1.F5)\. As entity density increases, documents also become substantially longer and more complex, resulting in a gradual performance decline across all models and redaction strategies\. However, the relative performance gap remains largely stable across density levels in comparison to the unredacted counterpart, suggesting that document complexity – not redaction – is the primary driver of degradation, with electra\-large\-discriminator showing slightly greater sensitivity that may be related to its replaced\-token\-detection pre\-training objective\.

## Appendix 0\.BRedakto System Design

Redakto is organized as a layered text\-redaction system consisting of a Streamlit\-based interaction layer and a FastAPI backend\. Through the user interface, users select an entity set, label granularity, and model\. The interface then submits structured JSON requests to backend prediction endpoints for either named entity recognition or pseudonym generation\.

The backend validates incoming requests using Pydantic schemas and delegates inference to the application\-level prediction service\. This service dispatches requests according to the configured model type\. NER models return detected entity spans, whereas NER\-PG models return entity spans together with generated pseudonyms and reconstructed pseudonymized text variants\.

![Refer to caption](https://arxiv.org/html/2608.18260v1/redakto-system-design.png)Figure 6:System design of RedaktoModel loading and inference are handled by a separate infrastructure layer\. At startup, the application builds a model registry from YAML configuration files that define the available entity sets, labels, model metadata, loading strategies, and framework paths\. Cached loaders support Flair sequence\-tagging models and Transformer\-based mT5 models\. The inference adapters perform SoMaJo tokenization, sentence boundary handling, and chunking for longer inputs\.

After inference, Redakto applies task\-specific post\-processing steps, including the merging of adjacent date entities, optional mapping from fine\-grained to coarse\-grained labels, and reconstruction of pseudonymized text\. The system is containerized with Docker and includes Kubernetes manifests for deploying both the user interface and the API services\.