PII-TRACE：面向多轮LLM对话中上下文感知PII检测的基准测试

# A Benchmark for Context-Aware PII Detectionin Multi-Turn LLM Conversations
Source: [https://arxiv.org/html/2609.22200](https://arxiv.org/html/2609.22200)
Chuan WangJoey ZhongAffiliation:PerplexityPaul FryzelAffiliation:PerplexityKyle PolleyAffiliation:PerplexityJerry MaAffiliation:PerplexityNinghui LiAffiliation:PerplexityAffiliation:Purdue UniversityAffiliation:Rutgers University

###### Abstract

LLM assistants and agentic systems log long multi\-turn conversations\. AI providers often scan these conversations for Personally Identifiable Information \(PII\) and mask the PII before storing or processing conversation data\. Yet most PII detectors and benchmarks target self\-contained records rather than cross\-turn evaluation\. To evaluate PII detection across turns in multi\-turn conversations, we introducePII\-TRACE\(TracingRecurring PIIAcrossConversationalExchanges\), to our knowledge the first PII benchmark to assess whether detectors identify PII in conversational contexts and cover every mention of a recurring identifier across turns\.PII\-TRACEcontains 13,148 synthetic multi\-turn dialogues in 13 languages with character\-level spans and identifier clusters\. Across eleven baselines, including frontier LLMs, no detector achieves full entity\-level coverage without substantial false positives on PII\-free conversations, and single\-pass reading loses a third of the gold characters on long dialogues\. To close this gap, we introduce PII\-Tracer, a compact 0\.6B\-parameter detector trained with conversation\-level supervision\. PII\-Tracer attains the highest entity\-level coverage of any system we evaluate and also performs strongly on standard single\-record benchmarks\. We will release both the benchmark and detector upon publication\.

\*\*footnotetext:Equal contribution\.## 1Introduction

![Refer to caption](https://arxiv.org/html/2609.22200v1/fig_teaser_col.png)Figure 1:One identifier \(highlighted\) recurs across a conversation\.PII\-TRACE’s*consistent\-detection*metric counts it as covered only if every mention is found\.Scanning text for Personally Identifiable Information \(PII\) is a standard component of data pipelines in modern LLM systems\. Providers detect and remove PII when curating pretraining corpora\([Soldaini et al\., 2024](https://arxiv.org/html/2609.22200#bib.bib37);[Laurençon et al\., 2022](https://arxiv.org/html/2609.22200#bib.bib38);[Li et al\., 2023](https://arxiv.org/html/2609.22200#bib.bib39);[Grattafiori and others, 2024](https://arxiv.org/html/2609.22200#bib.bib40)\)and before storing or reusing collected data\. This process helps satisfy data protection regulations\([European Union, 2016](https://arxiv.org/html/2609.22200#bib.bib31)\)and reduces the risk that models memorize and later reveal sensitive information\([Carlini et al\., 2021](https://arxiv.org/html/2609.22200#bib.bib12)\)\. Increasingly, PII detectors are built on transformer\-based language models, including a recently released open\-weight bidirectional token classifier\([OpenAI, 2026c](https://arxiv.org/html/2609.22200#bib.bib21)\)\.

Existing PII detectors are typically designed for relatively short, self\-contained records rather than the long, multi\-turn conversations common in LLM assistants and agentic systems\. Within a single conversation, users may paste emails or medical reports, switch between languages, and interact with assistants that call external tools or share context with other agents\([Yao et al\., 2023](https://arxiv.org/html/2609.22200#bib.bib24);[Xi et al\., 2023](https://arxiv.org/html/2609.22200#bib.bib25);[Packer et al\., 2023](https://arxiv.org/html/2609.22200#bib.bib26)\)\. Detectors designed to process isolated records may therefore struggle with the length, complexity, and contextual dependencies of real\-world conversations\.

Conversational context determines what counts as PII and which mentions must be detected\.The same text span can be PII in one conversation but not in another\. For example, a name is PII when it refers to the user, but not when it refers to a public figure or a place named after that person\. Moreover, the same identifier may recur throughout a conversation\. In Figure[1](https://arxiv.org/html/2609.22200#S1.F1), for example, the name*Maria Torres*appears across multiple turns\. A detector must identify every occurrence of a recurring identifier, as missing even one leaves that occurrence unprotected\. This requirement becomes especially important when conversations are passed to external tools or other agents, where private content may be further exposed\([Wang et al\., 2025](https://arxiv.org/html/2609.22200#bib.bib28);[Yagoubi et al\., 2026](https://arxiv.org/html/2609.22200#bib.bib29)\)\. PII detection in conversations is therefore inherently acontext\-awaretask\.

However, established PII benchmarks are primarily document\- or record\-oriented rather than conversational\([ai4Privacy, 2024](https://arxiv.org/html/2609.22200#bib.bib9);[Pilán et al\., 2022](https://arxiv.org/html/2609.22200#bib.bib2);[Stubbs et al\., 2015](https://arxiv.org/html/2609.22200#bib.bib6)\)\. In our experiments, we identify three properties that these benchmarks do not adequately evaluate\. The first is*cross\-turn consistency*: every mention of a recurring identifier should be detected\. The second is avoiding*long\-context degradation*, where recall decreases as conversation length grows\([Liu et al\., 2024](https://arxiv.org/html/2609.22200#bib.bib1);[OpenAI, 2026c](https://arxiv.org/html/2609.22200#bib.bib21)\)\. The third is the ability to handle*multilingual mixed\-medium text*, where languages may switch within a dialogue and prose may be interleaved with code and structured records\. As a result, strong performance on existing benchmarks does not necessarily translate into effective PII detection in real\-world conversational settings\. The closest concurrent efforts, such as REDACT\([Vats et al\., 2026](https://arxiv.org/html/2609.22200#bib.bib3)\)and RedactionBench\([Brynjólfsson et al\., 2026](https://arxiv.org/html/2609.22200#bib.bib19)\), cover more languages and longer documents, but do not provide explicit cross\-turn mention chains needed to evaluate cross\-turn consistency\.

This paper asks:*Do PII detectors consistently cover recurring identifiers in multi\-turn LLM conversations?*

To answer this question, we introducePII\-TRACE, to our knowledge the first PII benchmark with an explicit cross\-turn detection task for multi\-turn conversations\.PII\-TRACEcomprises 13,148 synthetic dialogues derived from production assistant traffic, with character\-level annotations spanning nine identifier types\. Mentions of the same identifier are further linked into entities, allowing us to evaluate whether a detector follows an identifier across the entire conversation\. We call this property*consistent detection*: an entity is considered covered only if every mention of that entity is detected\. Conversations inPII\-TRACErange from fewer than 1,000 to more than 100,000 characters, enabling us to measure robustness to long\-context degradation\. The benchmark also includes conversations that switch languages and interleave prose with code, tables, and structured records, spanning 13 languages in total\. It therefore enables evaluation of detectors on multilingual mixed\-medium text\. Finally,PII\-TRACEincludes PII\-free conversations for measuring false positives\.

We find that existing detectors struggle to consistently cover recurring identifiers across conversations\. Across eleven baselines, ranging from open\-source detectors to frontier LLMs prompted for PII detection, no public detector achieves strong consistent detection without flagging most PII\-free conversations\. Frontier LLMs are substantially more precise, but still miss many PII mentions\.

To close this gap, we introduce PII\-Tracer, a compact 0\.6B\-parameter detector trained on multilingual conversations\. PII\-Tracer labels an entire 4,096\-token window in a single pass\. Longer conversations can be processed using overlapping sliding windows, which substantially improves recall on long conversations \(§[4](https://arxiv.org/html/2609.22200#S4)\)\. PII\-Tracer achieves the highest consistent detection and character\-level F1 among all systems we evaluate, while using only a small fraction of the parameters of frontier LLMs \(§[3\.7](https://arxiv.org/html/2609.22200#S3.SS7), §[4](https://arxiv.org/html/2609.22200#S4)\)\.

#### Contributions\.

- •We introducePII\-TRACE, a multi\-turn PII benchmark with an explicit cross\-turn detection task\. It includes long, multilingual, and mixed\-medium conversations, with mentions linked into entities for evaluating*consistent detection*across turns\.
- •We evaluate eleven PII detectors, ranging from open\-source specialized models to frontier LLMs\. Public specialized detectors achieve high recall but low precision, while frontier LLMs are more precise but miss many mentions\. For most detectors, recall also degrades substantially as conversations grow longer\.
- •We introduce PII\-Tracer, a compact 0\.6B\-parameter PII detector trained on conversational data\. It achieves better coverage of recurring identifiers than all other evaluated detectors, including much larger frontier LLMs\.

Table 1:PII\-TRACEvs\. representative PII benchmarks: onlyPII\-TRACEpairs multi\-turn conversational text with cross\-turn identifier clusters \(TAB annotates coreference within single documents; REDACT includes flat chats among mixed formats but defines no cross\-turn task\)\. Sizes approximate\.

## 2Related Work

PII benchmarks\.Existing benchmarks cover synthetic records, including ai4privacy, Nemotron\-PII, and SPY\([ai4Privacy, 2024](https://arxiv.org/html/2609.22200#bib.bib9);[NVIDIA, 2025b](https://arxiv.org/html/2609.22200#bib.bib10);[Savkin et al\., 2025](https://arxiv.org/html/2609.22200#bib.bib8)\); legal documents with within\-document coreference, such as TAB\([Pilán et al\., 2022](https://arxiv.org/html/2609.22200#bib.bib2)\); longitudinal but non\-conversational clinical records, such as i2b2/UTHealth\([Stubbs et al\., 2015](https://arxiv.org/html/2609.22200#bib.bib6)\); and newswire named\-entity recognition \(NER\), such as CoNLL\-2003\([Tjong Kim Sang and De Meulder, 2003](https://arxiv.org/html/2609.22200#bib.bib5)\)\. Concurrent work includes REDACT, which covers mixed formats and multi\-turn chats without cross\-turn mention chains\([Vats et al\., 2026](https://arxiv.org/html/2609.22200#bib.bib3)\); the document\-oriented RedactionBench\([Brynjólfsson et al\., 2026](https://arxiv.org/html/2609.22200#bib.bib19)\); and the query\-focused PII\-Bench and CAPID\([Shen et al\., 2026](https://arxiv.org/html/2609.22200#bib.bib14);[Ponomarenko et al\., 2026](https://arxiv.org/html/2609.22200#bib.bib13)\)\. None provides explicit mention chains across turns of a longitudinal user–assistant conversation \(Table[1](https://arxiv.org/html/2609.22200#S1.T1)\)\.

#### PII detectors\.

Existing detectors range from rule\- and NER\-based systems, such as Microsoft Presidio\([Microsoft, 2024](https://arxiv.org/html/2609.22200#bib.bib11)\), to learned span models, including the OpenAI Privacy Filter\([OpenAI, 2026c](https://arxiv.org/html/2609.22200#bib.bib21)\), GLiNER2\-PII\([Zaratiana et al\., 2026](https://arxiv.org/html/2609.22200#bib.bib15)\), and open\-source detectors from the Piiranha\([iiiorg, 2024](https://arxiv.org/html/2609.22200#bib.bib34)\)and OpenMed\([OpenMed Science, 2026](https://arxiv.org/html/2609.22200#bib.bib35)\)series\. We evaluate seven of these detectors on conversations in §[4](https://arxiv.org/html/2609.22200#S4)\.

#### Contextual and agentic privacy\.

Privacy is contextual: whether an information flow is appropriate depends on the context in which it occurs\([Nissenbaum, 2004](https://arxiv.org/html/2609.22200#bib.bib20)\)\. LLMs can leak private information in inappropriate contexts\([Mireshghallah et al\., 2024](https://arxiv.org/html/2609.22200#bib.bib17)\), infer hidden attributes from contextual cues\([Staab et al\., 2024](https://arxiv.org/html/2609.22200#bib.bib18)\), and reproduce memorized training data\([Carlini et al\., 2021](https://arxiv.org/html/2609.22200#bib.bib12);[Kim et al\., 2023](https://arxiv.org/html/2609.22200#bib.bib16)\)\. Agentic privacy benchmarks test whether agent actions respect privacy norms\([Shao et al\., 2024](https://arxiv.org/html/2609.22200#bib.bib27)\)and measure information leakage through tool use and inter\-agent interactions\([Wang et al\., 2025](https://arxiv.org/html/2609.22200#bib.bib28);[Yagoubi et al\., 2026](https://arxiv.org/html/2609.22200#bib.bib29)\)\. Separately, long\-context studies show that model performance degrades as input length increases\([Liu et al\., 2024](https://arxiv.org/html/2609.22200#bib.bib1);[Hsieh et al\., 2024](https://arxiv.org/html/2609.22200#bib.bib22);[Bai et al\., 2024](https://arxiv.org/html/2609.22200#bib.bib23)\)\.PII\-TRACEis complementary to this work: it evaluates span\-level PII detection across an entire conversation\.

## 3ThePII\-TRACEBenchmark

### 3\.1Problem formulation

We study PII detection in long, multi\-turn conversations\. Given a complete user–assistant dialogue, the task is to identify every text span that refers to a private individual\. We adopt an operational definition of PII as information that identifies a specific person, either on its own or when combined with other information\. This definition draws on the GDPR\([European Union, 2016](https://arxiv.org/html/2609.22200#bib.bib31)\), the ISO/IEC 29100 privacy framework\([ISO/IEC, 2011](https://arxiv.org/html/2609.22200#bib.bib32)\), U\.S\. federal guidance\([McCallister et al\., 2010](https://arxiv.org/html/2609.22200#bib.bib30)\), and scholarship on contextual identifiability\([Schwartz and Solove, 2011](https://arxiv.org/html/2609.22200#bib.bib33)\)\.

Consider the name*Maria Torres*in Figure[1](https://arxiv.org/html/2609.22200#S1.F1)\. Whether this span constitutes PII cannot be determined from the string alone\. If a user states their name, the span reveals personal information\. The same string does not identify anyone when the assistant introduces it as a fictional placeholder in an example\. Distinguishing among these cases requires reasoning about the surrounding conversational context, making PII detection acontext\-awaretask\.

Formally, a conversation is a sequence ofmmuser and assistant turns concatenated into one text,

x=τ1\|τ2​‖⋯‖​τmx\\;=\\;\\tau\_\{1\}\\,\\\|\\,\\tau\_\{2\}\\,\\\|\\,\\cdots\\,\\\|\\,\\tau\_\{m\}\(1\)where turnτi\\tau\_\{i\}occupies the character positionsTiT\_\{i\}ofxx\. A detector is graded on all ofxx\. It may read the text in one window or several \(a choice we leave to its inference policy\), but it must commit to a single setPPof character positions marked as PII\. We do not require it to say which marked positions belong to the same entity\. The task is detection only\.

The nine identifier types shown in Table[2](https://arxiv.org/html/2609.22200#S3.T2)cover the label sets of deployed detectors\([OpenAI, 2026c](https://arxiv.org/html/2609.22200#bib.bib21);[Zaratiana et al\., 2026](https://arxiv.org/html/2609.22200#bib.bib15)\)\. LetGGbe the set of gold PII characters\. An*entity*E⊆GE\\subseteq Gis the set of characters belonging to a single identifier, with repeated mentions grouped by coreference, and the entitiesℰ\\mathcal\{E\}partitionGG\. We call an entity*cross\-turn*if its characters appear in more than one turn in the conversation,

\|\{i:E∩Ti≠∅\}\|≥2\.\\bigl\|\\\{\\,i:E\\cap T\_\{i\}\\neq\\emptyset\\,\\\}\\bigr\|\\geq 2\.\(2\)
Table 2:The nine identifier types, with gold mention counts over all three splits \(37,431 total; every occurrence counts separately\)\.Achieving protection requiresconsistent detectionbecause for recurring identifiers, missing even one occurrence leaves the identifier exposed\. First, at the character level, a label is considered correct if a labeled position belongs toGG\. This measures how many PII texts are detected\. Second, at the entity level, an entity is considered consistently detected if all characters of entityEEare labeled \(E⊆PE\\subseteq P\)\. The evaluation setting \(§[4\.1](https://arxiv.org/html/2609.22200#S4.SS1)\) embodies these two criteria into the precise metrics reported in this paper\.

### 3\.2Design desiderata

Our design goals follow the actual deployment scenario: for the LLM assistant, the PII detector deals with dialogue logs, not the short, well\-structured records common in existing corpora\.

Splitting a dialogue into individual records results in the loss of significant structural information, since an identifier may reappear after many rounds\. We assign a cluster ID to each occurrence, enabling evaluation to track the identifier across the entire dialogue rather than a single location\. Dialogue lengths vary widely, from a few hundred to over one hundred thousand characters, allowing us to measure recall based on input length\. Dialogues are also not always monolingual natural language text: they may switch languages mid\-conversation, and plain text may contain code, tables, or structured records\. In our experiments, existing baseline detectors performed poorly primarily on dialogues with these characteristics \(§[4](https://arxiv.org/html/2609.22200#S4)\)\.

#### Pipeline\.

We synthesize the benchmark from real traffic in five steps \(Figure[2](https://arxiv.org/html/2609.22200#S3.F2)\), labeling and anonymization \(steps 1, 2, §[3\.3](https://arxiv.org/html/2609.22200#S3.SS3)\), synthesis \(steps 3, 4, §[3\.4](https://arxiv.org/html/2609.22200#S3.SS4)\), and the verification gates that decide whether a record \(step 5\) is released \(§[3\.5](https://arxiv.org/html/2609.22200#S3.SS5)\)\.

![Refer to caption](https://arxiv.org/html/2609.22200v1/fig_pipeline.png)Figure 2:ConstructingPII\-TRACE, on one example \(source values withheld\): a labeled conversation becomes a typed, cluster\-preserving template; each turn is paraphrased with placeholders intact, and each placeholder is replaced by its entity’s surrogate with exact offsets\. Failed paraphrases are retried, and gate\-flagged documents can re\-enter synthesis with fresh surrogates before a final drop \(§[3\.5](https://arxiv.org/html/2609.22200#S3.SS5)\)\. Steps 1 and 2 are the anonymization of §[3\.3](https://arxiv.org/html/2609.22200#S3.SS3), steps 3 and 4 the synthesis of §[3\.4](https://arxiv.org/html/2609.22200#S3.SS4), and the gates, guarding the released record \(step 5\), the verification of §[3\.5](https://arxiv.org/html/2609.22200#S3.SS5)\.

### 3\.3Labeling and anonymization

We begin with large\-scale real\-world user\-assistant dialogue samples collected from a production environment, transforming each dialogue into a de\-identified template\. Multiple state\-of\-the\-art LLMs are used to annotate identifiers according to the nine types of annotation in Table[2](https://arxiv.org/html/2609.22200#S3.T2)\(Figure[2](https://arxiv.org/html/2609.22200#S3.F2), step 1\)\. Subsequently, recurring identifiers of the same type are grouped into the same entity through rule\-based processing, and sensitive attribute layers are annotated in a separate process\. In step 2, we remove each identifier and replace it with a placeholder\. The template retains the multi\-turn structure of the dialogue, as well as the position, type, and entity ID of each occurrence, while not retaining any of the original annotated values\.

### 3\.4Synthesis

Each template in the anonymization phase generates a synthesized dialogue \(Figure[2](https://arxiv.org/html/2609.22200#S3.F2), steps 3 and 4\)\. Surrogates are generated for each entity, matching their type, format, and geographic location, and using values that are formatted correctly but not usable\. The same surrogate is used for every occurrence, ensuring identifier consistency within a document, while surrogates in different documents are generated independently\. The language model then rewrites each round of dialogue \(step 3\) and inserts the surrogates \(step 4\), so both the textual representation and the annotated identifier values in the final benchmark are synthesized content\. If a placeholder is lost during paraphrasing, the rewrite is regenerated up to three times \(back loop in step 3\); if it still fails, the document is flagged and proceeds to the verification checks in §[3\.5](https://arxiv.org/html/2609.22200#S3.SS5)\. This replacement strategy follows the clinical de\-identification practice\([Carrell et al\., 2013](https://arxiv.org/html/2609.22200#bib.bib7);[Stubbs et al\., 2015](https://arxiv.org/html/2609.22200#bib.bib6)\)\.

### 3\.5Alignment, verification, and audit

Every tagged identifier in the benchmark is a surrogate that the pipeline placed \(Figure[2](https://arxiv.org/html/2609.22200#S3.F2), step 5\), so the gold span can be accurately determined without manual annotation\. The alignment step further ensures accurate positioning: after rewriting, we recalculate each surrogate’s character offset and append its type and entity ID\.

Subsequently, the validation process verifies the correct construction of each document through three automated checks\. First, the replacement value span must correspond one\-to\-one with the gold mention, and all mentions of the same entity must share the same value\. Second, no original values should be detected during independent rescanning using Microsoft Presidio and strict regular expressions\. Third, each stored character offset must accurately extract the corresponding surrogate substring\. The first two checks correspond to the decision nodes in Figure[2](https://arxiv.org/html/2609.22200#S3.F2)\. A document that fails any gate is resynthesized with fresh surrogates and re\-paraphrased turns \(the loop back into step 3\), and discarded if it still fails\. Since these checks are rule\-based, we also used a second, independent language model to audit a portion of the released documents as a fuzzy check; anything it flagged was manually reviewed against the document’s surrogate list\. See Appendix[E](https://arxiv.org/html/2609.22200#A5)for the prompts used in the pipeline\.

### 3\.6Composition

PII\-TRACEcontains 13,148 conversations, split into training, validation, and test sets with no source conversations shared between the different data partitions; 5,645 dialogues contain gold PIIs, and 7,503 are PII\-free\. Dialogue lengths range from less than 1,000 characters to over 100,000 characters, and thirteen languages each contain at least 100 dialogues\. In dialogues containing PIIs,63\.8%63\.8\\%of the dialogues contain entities mentioned multiple times, and28\.7%28\.7\\%of the dialogues have entities that appear repeatedly across rounds; therefore, detecting only one mention often fails to cover the entire entity\.

### 3\.7PII\-Tracer: a context\-aware detector

PII\-Tracer models PII detection in the dialogue as token classification\. One encoder pass reads a window \(§[3\.1](https://arxiv.org/html/2609.22200#S3.SS1)\) of the dialoguexxand generates contextual statesh1:Th\_\{1:T\}forTTtokens within it\. Therefore, the annotation for each token is based on the surrounding dialogue window as context, rather than using the autoregressive prompting used in existing LLM baselines\.

#### Architecture\.

PII\-Tracer is a bidirectional encoder with0\.60\.6B parameters, using a Qwen3\([Qwen Team, 2025](https://arxiv.org/html/2609.22200#bib.bib45)\)backbone adapted with masked\-diffusion pretraining, reading a maximum of40964096tokens per window\. On the shared token states, the model uses a tagging head covering 37 BIOES labels \(including a background tagOOand\{\\\{B, I, E, S\}\\\}for each of the nine types\), and an auxiliary sensitivity head used during training to predicts whether the conversation contains sensitive content as an auxiliary training signal\. Gold spans are mapped to tokenization and represented as BIOES tags; therefore, detection is modeled as per\-token classification with the surrounding conversation window as context\.

#### Training and decoding\.

We minimize

ℒ=λtok​ℒtag\+λsens​ℒsens,\\mathcal\{L\}=\\lambda\_\{\\text\{tok\}\}\\,\\mathcal\{L\}\_\{\\text\{tag\}\}\+\\lambda\_\{\\text\{sens\}\}\\,\\mathcal\{L\}\_\{\\text\{sens\}\},\(3\)whereℒtag\\mathcal\{L\}\_\{\\text\{tag\}\}is the class\-weighted cross\-entropy for BIOES tags, andℒsens\\mathcal\{L\}\_\{\\text\{sens\}\}is the binary cross\-entropy for sensitivity logit, with weights ofλtok=1\.5\\lambda\_\{\\text\{tok\}\}\{=\}1\.5andλsens=0\.3\\lambda\_\{\\text\{sens\}\}\{=\}0\.3, respectively\. We trained on AdamW for 3 epochs with 714k samples, including multilingual assistant conversations annotated by multiple frontier language models and samples from the ai4privacy corpus\. Each training sample concatenates all turns in a conversation into plain text and annotates each mention on the conversation\-level offset, enabling the model to judge each span in conjunction with the conversational context\. Therefore, the same string can be supervised as a PII in one conversation and as background in another\. During inference, per\-token scores are decoded into typed spans while ensuring the tag sequences are valid; two learnable boundary biases adjust the tradeoff between precision and recall without retraining; for conversations longer than the window, decoding can be performed window\-by\-window \(§[4](https://arxiv.org/html/2609.22200#S4)\)\. More hyperparameter settings in Appendix[C](https://arxiv.org/html/2609.22200#A3)\.

## 4Experiments

Table 3:PII detection on thePII\-TRACEtest set \(1,922 documents\); all metrics label\-agnostic, higher is better\. The four frontier LLMs are prompted zero\-shot; span scores match gold spans by overlap and by containment\.### 4\.1Evaluation setup

Our evaluation covers twelve detectors: Microsoft Presidio\([Microsoft, 2024](https://arxiv.org/html/2609.22200#bib.bib11)\), six learned span models \(the OpenAI Privacy Filter with 1\.5B parameters\([OpenAI, 2026c](https://arxiv.org/html/2609.22200#bib.bib21)\), GLiNER2\-PII\([Zaratiana et al\., 2026](https://arxiv.org/html/2609.22200#bib.bib15)\), zero\-shot GLiNER\-PII\([NVIDIA, 2025a](https://arxiv.org/html/2609.22200#bib.bib36)\), Piiranha\([iiiorg, 2024](https://arxiv.org/html/2609.22200#bib.bib34)\), and two OpenMed clinical PII models\([OpenMed Science, 2026](https://arxiv.org/html/2609.22200#bib.bib35)\)\), four frontier LLMs prompted zero\-shot with the nine\-type taxonomy \(GPT\-5\.4\([OpenAI, 2026a](https://arxiv.org/html/2609.22200#bib.bib41)\), GPT\-5\.6\-sol\([OpenAI, 2026b](https://arxiv.org/html/2609.22200#bib.bib42)\), Claude Opus 4\.8\([Anthropic, 2026a](https://arxiv.org/html/2609.22200#bib.bib43)\), and Claude Sonnet 5\([Anthropic, 2026b](https://arxiv.org/html/2609.22200#bib.bib44)\)\), and ours PII\-Tracer\. We evaluate on the test split \(1,922 documents\) and report metrics as following:

- •Character P / R / F1: A predicted character is considered a true positive if it lies within a gold span\. Character F1 is our primary detection metric, measuring how many PII texts are successfully masked without considering span boundaries\.
- •Span\-Overlap and Span\-Containment P / R / F1: For overlap, a prediction is considered correct if it overlaps with any gold span\. For containment, precision counts predicted spans contained within a gold span, while recall counts gold spans contained within a predicted span\.
- •Consistent detection: The percentage of gold entities whose mentions are covered\. CD\-multi is calculated only for multi\-mention entities, while cross\-turn CD is calculated only for entities that repeat across rounds\.
- •Has\-PII accuracy and FP0: The former is the document\-by\-document binary has\-PII accuracy, and the latter is the false\-positive rate on documents without PII\.

Table 4:Consistent detection by an entity’s mention count and, for multi\-mention entities, by turn span: each cell is the fraction of that group’s entities with*every*mention covered\. Best per row inbold\.
### 4\.2Main results

Table[3](https://arxiv.org/html/2609.22200#S4.T3)divides the twelve systems into two groups\. Publicly available specialized baselines achieve high recall through over\-marking\. OpenMed models can cover more than0\.90\.9of gold characters, but at the cost of marking more than six times the gold character volume, resulting in character precision below0\.150\.15\. Presidio marks 16 times the gold volume, with a precision of only0\.0450\.045, primarily covering structured identifiers such as email and phone numbers, with less coverage of free text\. Frontier general\-purpose LLMs exhibit the opposite pattern\. They have the highest character precision, approaching0\.50\.5, while all publicly available specialized baselines do not exceed0\.360\.36; however, they only cover gold characters from0\.560\.56to0\.680\.68\.

PII\-Tracer combines the advantages of both\. It can cover 0\.830 gold characters, comparable to over\-marking detectors, while achieving a precision of 0\.507, comparable to frontier LLMs; its char\-F1 is 0\.629, the highest of all systems, making it the most balanced among the twelve detectors\. The much larger parameter\-scale GPT\-5\.6\-sol only slightly surpasses it in span\-level F1 \(overlap 0\.632 vs\. 0\.621, containment 0\.612 vs\. 0\.580\)\.

### 4\.3Cross\-turn consistency

Covering characters is easier than covering every mention of an entity, since missing any mention leaves the entity exposed; multi\-mention consistent detection is therefore lower than character recall for all detectors\. PII\-Tracer covers0\.8300\.830of gold characters, with a multi\-mention consistent detection \(CD\-multi\) of0\.7940\.794, the highest of all systems, significantly higher than GPT\-5\.6\-sol \(0\.5700\.570\) and other frontier LLMs \(0\.240\.24\-0\.310\.31\)\. Aggregate CD, CD\-multi, cross\-turn CD, and PII\-free false\-positive rates for the twelve detectors are shown in Appendix[D](https://arxiv.org/html/2609.22200#A4)\(Table[7](https://arxiv.org/html/2609.22200#A4.T7)\), while Table[4](https://arxiv.org/html/2609.22200#S4.T4)analyzes consistent detection by mention count \(899899single\-mention and959959multi\-mention entities,790790of the latter appearing repeatedly across rounds\)\. As the number of mentions increases, consistent detection declines sharply for learned detectors \(GLiNER2\-PII from0\.6410\.641for a single mention to0\.0730\.073for 6–10 mentions\) and frontier LLMs \(GPT\-5\.6\-sol from0\.7880\.788to0\.4640\.464, Claude Opus 4\.8 from0\.8400\.840to0\.0450\.045\), whereas PII\-Tracer declines more gradually \(from0\.9170\.917to0\.6910\.691\) and performs best across all groups, including cross\-turn entities \(0\.7760\.776\)\. Presidio consistently performs at around0\.630\.63across all groups because it labels most structured values rather than selectively covering a particular entity\.

### 4\.4Document\-level flagging

In practice, Guardrail must also determine whether a conversation contains PII, as incorrectly labeling clean conversations incurs costs\. Has\-PII accuracy reveals over\-flagging issues: Presidio and OpenMed models score between 0\.43 and 0\.45, below the Has\-PII baseline because they label PII in almost every conversation; frontier LLMs score between 0\.76 and 0\.83; and PII\-Tracer reaches0\.7590\.759, the highest among specialized detectors and close to frontier LLMs\. The complementary metric FP0is shown in Table[7](https://arxiv.org/html/2609.22200#A4.T7): all public specialized detectors label more than half of the PII\-free documents, while the proportion for PII\-Tracer is0\.3850\.385\. This reflects the cost of over\-marking at the document level, as predicted spans scattered throughout clean conversations become false flags\.

Table 5:PII\-Tracer character P/R/F1 by conversation length, read in a single 4,096\-token window\. Docs: test documents per bucket; PII docs: those with gold spans; characters pooled per bucket\.
### 4\.5Single\-window degradation

As dialogue length increases, recall decreases\. In a single 4096\-token window, PII\-Tracer covers0\.9750\.975of gold characters in dialogues shorter than 1k characters,0\.9550\.955in dialogues between 1 and 10k characters, and drops to0\.6870\.687for dialogues longer than 10k characters \(Table[5](https://arxiv.org/html/2609.22200#S4.T5)\)\. Precision does not decrease with length; it is lowest in the shortest range \(0\.3920\.392\), and around0\.510\.51in other ranges\. This is primarily due to single\-window truncation: using 50%\-overlap sliding windows decoding for the same checkpoint can improve character recall from0\.830\.83to0\.970\.97and multi\-mention CD from0\.790\.79to0\.950\.95\(Appendix[D](https://arxiv.org/html/2609.22200#A4)\)\. Longer dialogues contain a larger absolute number of identifiers, so documents with decreased recall face a greater risk of missed detections\. The sliding\-window method incurs a slight loss of precision, but the two strategies can be chosen at decode time, so a suitable operating point can be selected during deployment without retraining\.

![Refer to caption](https://arxiv.org/html/2609.22200v1/fig_lang_heat.png)Figure 3:Character F1 by language for six test languages covering the Latin, Cyrillic, and Hangul scripts\.
### 4\.6Multilingual mixed\-medium text

Figure[3](https://arxiv.org/html/2609.22200#S4.F3)shows the character F1 scores for six language subsets in the corpus, covering Latin, Cyrillic, and Hangul scripts; full language\-specific results will be available upon release\. Each language was evaluated on all test documents, ranging from928928for English to6161for Italian\. System rankings varied across different scripts\. Presidio covered0\.790\.79of gold characters on English but only0\.420\.42on Russian; OpenMed 434M’s recall dropped from0\.940\.94for English to0\.690\.69for Korean; OpenAI Privacy Filter maintained recall across different scripts, thus its F1 score dropped to0\.280\.28on Italian and Russian, primarily due to precision\. PII\-Tracer achieved the highest character F1 score in four of the six languages \(all between0\.6160\.616and0\.7350\.735; GPT\-5\.6\-sol scored slightly higher in ‘en’ and ‘ko’\), and the highest consistent detection across all six languages, ranging from0\.800\.80to0\.930\.93\. Because rankings vary across different scripts, selecting a detector solely based on overall or English scores may not suit for real\-world language combinations\.

Figure 4:PII\-Tracer on five external PII benchmarks \(label\-agnostic character P/R/F1\); gray: the OpenAI Privacy Filter under identical evaluation\.
### 4\.7External benchmarks

Figure[4](https://arxiv.org/html/2609.22200#S4.F4)shows the results of PII\-Tracer on five external PII benchmarks, including two public PII datasets used in the OpenAI Privacy Filter model card: ai4privacy and SPY\. PII\-Tracer outperforms the Privacy Filter on all benchmarks: char F10\.9500\.950vs0\.9070\.907on ai4privacy\([ai4Privacy, 2024](https://arxiv.org/html/2609.22200#bib.bib9)\),0\.8470\.847vs0\.7090\.709on Nemotron\-PII\([NVIDIA, 2025b](https://arxiv.org/html/2609.22200#bib.bib10)\),0\.5850\.585vs0\.5430\.543on SPY\([Savkin et al\., 2025](https://arxiv.org/html/2609.22200#bib.bib8)\),0\.9520\.952vs0\.8950\.895on the Gretel PII masking set\([AI, 2024](https://arxiv.org/html/2609.22200#bib.bib46)\), and0\.5940\.594vs0\.3500\.350on TAB\([Pilán et al\., 2022](https://arxiv.org/html/2609.22200#bib.bib2)\)\. TAB is the only benchmark that includes real human\-annotated text, and at the same precision, PII\-Tracer achieves twice the recall\. The improvement does not stem from a shared synthesis style\. Table[3](https://arxiv.org/html/2609.22200#S4.T3)indicates that single\-record detectors perform poorly in dialogue scenarios, whereas PII\-Tracer is competitive in both settings\. Since its training mixture includes single\-record data, we do not attribute this result entirely to conversational supervision\.

## 5Conclusion

In this work, we proposePII\-TRACE, a benchmark for PII detection in multi\-turn LLM conversations, and PII\-Tracer, a compact 0\.6B detector trained on such data\. Experiments show that existing detectors either over\-label PII\-free conversations or miss some mentions of recurring identifiers; in contrast, PII\-Tracer more completely covers recurring identifiers and performs well on standard external benchmarks\. We call for greater attention to PII detection in multi\-turn LLM conversations\.

## Limitations

PII\-TRACEis designed as a controlled benchmark of PII detection in multi\-turn user–assistant conversations\. Its conversations are synthetic reconstructions derived from the structure of production assistant traffic\. This construction supports public release with exact span offsets and consistent mention chains, but it does not reproduce the source distribution verbatim\. The reported results should therefore be read as measurements of conversational PII detection rather than estimates for any particular production workload\.

The current release focuses on conversation text\. Tool calls, inter\-agent messages, and multimodal inputs fall outside its scope, although they are natural extensions for studying how personal data moves through agentic systems\. Evaluation on conversations from additional assistants and domains would also provide a broader test of transfer beyond the setting represented here\.

Finally, each baseline is evaluated under a single inference configuration\. Alternative prompts, thresholds, context\-window policies, or future model updates may change absolute scores\. The benchmark is thus best suited to comparing detector behavior within the evaluation setup used here, rather than certifying production safety or regulatory compliance\.

## Ethical Considerations

PII\-TRACEis built to strengthen protective systems: the intended use is evaluating and improving PII detectors, not re\-identifying anyone\. The source conversations were accessed and processed under the originating provider’s terms of service and privacy policy; access was limited to authorized project members under the provider’s data\-access controls, and the manual review of audit\-flagged documents took place under the same controls\. No crowdworkers or external annotators were involved: identifiers were labeled by prompted language models, and human review was limited to the project team\. We release only synthetic data: no source conversation is published, every tagged identifier is a fabricated, leak\-checked surrogate, and the known residual, untagged personal names in some non\-English text, is documented in the datasheet \(Appendix[A](https://arxiv.org/html/2609.22200#A1)\)\. The benchmark and the detector will be released under the MIT license\. Scores on synthetic data are proxies for, not guarantees of, behavior on real traffic, and the benchmark certifies neither production safety nor regulatory compliance\. Language models assisted with writing and engineering in this project; the authors reviewed and verified all content and results\.

## References

- AI \(2024\)G\. AIGLiNER models for pii detection through fine\-tuning on gretel\-generated synthetic documents\.Gretel\.Cited by:[§4\.7](https://arxiv.org/html/2609.22200#S4.SS7.p1.1)\.
- ai4Privacy \(2024\)ai4PrivacyPII\-masking\-300k\.Note:[https://huggingface\.co/datasets/ai4privacy/pii\-masking\-300k](https://huggingface.co/datasets/ai4privacy/pii-masking-300k)Cited by:[Table 1](https://arxiv.org/html/2609.22200#S1.T1.6.2.1),[§1](https://arxiv.org/html/2609.22200#S1.p4.1),[§2](https://arxiv.org/html/2609.22200#S2.p1.1),[§4\.7](https://arxiv.org/html/2609.22200#S4.SS7.p1.1)\.
- Anthropic \(2026a\)AnthropicSystem card: Claude Opus 4\.8\.Note:May 28, 2026[https://www\.anthropic\.com/news/claude\-opus\-4\-8](https://www.anthropic.com/news/claude-opus-4-8)Cited by:[§4\.1](https://arxiv.org/html/2609.22200#S4.SS1.p1.1),[Table 3](https://arxiv.org/html/2609.22200#S4.T3.4.12.1)\.
- Anthropic \(2026b\)AnthropicSystem card: Claude Sonnet 5\.Note:June 30, 2026[https://www\.anthropic\.com/news/claude\-sonnet\-5](https://www.anthropic.com/news/claude-sonnet-5)Cited by:[§4\.1](https://arxiv.org/html/2609.22200#S4.SS1.p1.1),[Table 3](https://arxiv.org/html/2609.22200#S4.T3.4.13.1)\.
- Baiet al\.\(2024\)Y\. Bai, X\. Lv, J\. Zhang, H\. Lyu, J\. Tang, Z\. Huang, Z\. Du, X\. Liu, A\. Zeng, L\. Hou, Y\. Dong, J\. Tang, and J\. LiLongBench: a bilingual, multitask benchmark for long context understanding\.InAnnual Meeting of the Association for Computational Linguistics \(ACL\),Note:arXiv:2308\.14508Cited by:[§2](https://arxiv.org/html/2609.22200#S2.SS0.SSS0.Px2.p1.1)\.
- Brynjólfssonet al\.\(2026\)S\. Brynjólfsson, S\. Jayakrishnan, E\. Sali, D\. Purwar, and M\. AggarwalRedactionBench\.arXiv preprint arXiv:2606\.18782\.Note:arXiv:2606\.18782Cited by:[Table 1](https://arxiv.org/html/2609.22200#S1.T1.6.8.1),[§1](https://arxiv.org/html/2609.22200#S1.p4.1),[§2](https://arxiv.org/html/2609.22200#S2.p1.1)\.
- Carliniet al\.\(2021\)N\. Carlini, F\. Tramèr, E\. Wallace, M\. Jagielski, A\. Herbert\-Voss, K\. Lee, A\. Roberts, T\. Brown, D\. Song, Ú\. Erlingsson, A\. Oprea, and C\. RaffelExtracting training data from large language models\.InUSENIX Security Symposium,Note:arXiv:2012\.07805Cited by:[§1](https://arxiv.org/html/2609.22200#S1.p1.1),[§2](https://arxiv.org/html/2609.22200#S2.SS0.SSS0.Px2.p1.1)\.
- Carrellet al\.\(2013\)D\. Carrell, B\. Malin, J\. Aberdeen, S\. Bayer, C\. Clark, B\. Wellner, and L\. HirschmanHiding in plain sight: use of realistic surrogates to reduce exposure of protected health information in clinical text\.Journal of the American Medical Informatics Association20\(2\)\.External Links:[Document](https://dx.doi.org/10.1136/amiajnl-2012-001034)Cited by:[§3\.4](https://arxiv.org/html/2609.22200#S3.SS4.p1.1)\.
- European Union \(2016\)European UnionRegulation \(eu\) 2016/679 of the european parliament and of the council \(general data protection regulation\)\.Note:Official Journal of the European Union, L 119Cited by:[§1](https://arxiv.org/html/2609.22200#S1.p1.1),[§3\.1](https://arxiv.org/html/2609.22200#S3.SS1.p1.1)\.
- Gebruet al\.\(2021\)T\. Gebru, J\. Morgenstern, B\. Vecchione, J\. W\. Vaughan, H\. Wallach, H\. Daumé III, and K\. CrawfordDatasheets for datasets\.Communications of the ACM64\(12\),pp\. 86–92\.Note:arXiv:1803\.09010Cited by:[Appendix A](https://arxiv.org/html/2609.22200#A1.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafioriet al\.The Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§1](https://arxiv.org/html/2609.22200#S1.p1.1)\.
- Hsiehet al\.\(2024\)C\. Hsieh, S\. Sun, S\. Kriman, S\. Acharya, D\. Rekesh, F\. Jia, Y\. Zhang, and B\. GinsburgRULER: what’s the real context size of your long\-context language models?\.InConference on Language Modeling \(COLM\),Note:arXiv:2404\.06654Cited by:[§2](https://arxiv.org/html/2609.22200#S2.SS0.SSS0.Px2.p1.1)\.
- iiiorg \(2024\)iiiorgPiiranha\-v1: detect personal information\.Note:Accessed 2026[https://huggingface\.co/iiiorg/piiranha\-v1\-detect\-personal\-information](https://huggingface.co/iiiorg/piiranha-v1-detect-personal-information)Cited by:[§2](https://arxiv.org/html/2609.22200#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2609.22200#S4.SS1.p1.1),[Table 3](https://arxiv.org/html/2609.22200#S4.T3.4.7.1)\.
- ISO/IEC \(2011\)ISO/IECISO/iec 29100:2011 information technology – security techniques – privacy framework\.Note:International Organization for StandardizationCited by:[§3\.1](https://arxiv.org/html/2609.22200#S3.SS1.p1.1)\.
- Kimet al\.\(2023\)S\. Kim, S\. Yun, H\. Lee, M\. Gubri, S\. Yoon, and S\. J\. OhProPILE: probing privacy leakage in large language models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2307\.01881Cited by:[§2](https://arxiv.org/html/2609.22200#S2.SS0.SSS0.Px2.p1.1)\.
- Laurençonet al\.\(2022\)H\. Laurençon, L\. Saulnier, T\. Wang, C\. Akiki,et al\.The BigScience ROOTS corpus: a 1\.6TB composite multilingual dataset\.InAdvances in Neural Information Processing Systems \(NeurIPS\), Datasets and Benchmarks Track,Note:arXiv:2303\.03915Cited by:[§1](https://arxiv.org/html/2609.22200#S1.p1.1)\.
- Liet al\.\(2023\)R\. Li, L\. Ben Allal, Y\. Zi, N\. Muennighoff, D\. Kocetkov, C\. Mou, M\. Marone, C\. Akiki,et al\.StarCoder: may the source be with you\!\.Transactions on Machine Learning Research\.Note:arXiv:2305\.06161Cited by:[§1](https://arxiv.org/html/2609.22200#S1.p1.1)\.
- Liuet al\.\(2024\)N\. F\. Liu, K\. Lin, J\. Hewitt, A\. Paranjape, M\. Bevilacqua, F\. Petroni, and P\. LiangLost in the middle: how language models use long contexts\.Transactions of the Association for Computational Linguistics \(TACL\)12,pp\. 157–173\.Note:arXiv:2307\.03172Cited by:[§1](https://arxiv.org/html/2609.22200#S1.p4.1),[§2](https://arxiv.org/html/2609.22200#S2.SS0.SSS0.Px2.p1.1)\.
- McCallisteret al\.\(2010\)E\. McCallister, T\. Grance, and K\. ScarfoneGuide to protecting the confidentiality of personally identifiable information \(pii\)\.Note:NIST Special Publication 800\-122, National Institute of Standards and TechnologyCited by:[§3\.1](https://arxiv.org/html/2609.22200#S3.SS1.p1.1)\.
- Microsoft \(2024\)MicrosoftPresidio: data protection and de\-identification sdk\.Note:Accessed 2026[https://github\.com/microsoft/presidio](https://github.com/microsoft/presidio)Cited by:[§2](https://arxiv.org/html/2609.22200#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2609.22200#S4.SS1.p1.1),[Table 3](https://arxiv.org/html/2609.22200#S4.T3.4.3.1)\.
- Mireshghallahet al\.\(2024\)N\. Mireshghallah, H\. Kim, X\. Zhou, Y\. Tsvetkov, M\. Sap, R\. Shokri, and Y\. ChoiCan LLMs keep a secret? testing privacy implications of language models via contextual integrity theory\.InInternational Conference on Learning Representations \(ICLR\),Note:Spotlight\. arXiv:2310\.17884Cited by:[§2](https://arxiv.org/html/2609.22200#S2.SS0.SSS0.Px2.p1.1)\.
- Nissenbaum \(2004\)H\. NissenbaumPrivacy as contextual integrity\.Washington Law Review79\(1\),pp\. 119–157\.Cited by:[§2](https://arxiv.org/html/2609.22200#S2.SS0.SSS0.Px2.p1.1)\.
- NVIDIA \(2025a\)NVIDIAGLiNER\-PII: zero\-shot PII named entity recognition\.Note:Accessed 2026[https://huggingface\.co/nvidia/gliner\-PII](https://huggingface.co/nvidia/gliner-PII)Cited by:[§4\.1](https://arxiv.org/html/2609.22200#S4.SS1.p1.1),[Table 3](https://arxiv.org/html/2609.22200#S4.T3.4.6.1)\.
- NVIDIA \(2025b\)NVIDIANemotron\-PII\.Note:[https://huggingface\.co/datasets/nvidia/Nemotron\-PII](https://huggingface.co/datasets/nvidia/Nemotron-PII)Cited by:[Table 1](https://arxiv.org/html/2609.22200#S1.T1.6.3.1),[§2](https://arxiv.org/html/2609.22200#S2.p1.1),[§4\.7](https://arxiv.org/html/2609.22200#S4.SS7.p1.1)\.
- OpenAI \(2026a\)OpenAIGPT\-5\.4 Thinking system card\.Note:March 5, 2026[https://deploymentsafety\.openai\.com/gpt\-5\-4\-thinking](https://deploymentsafety.openai.com/gpt-5-4-thinking)Cited by:[§4\.1](https://arxiv.org/html/2609.22200#S4.SS1.p1.1),[Table 3](https://arxiv.org/html/2609.22200#S4.T3.4.10.1)\.
- OpenAI \(2026b\)OpenAIGPT\-5\.6 Preview system card\.Note:June 25, 2026; covers the Sol, Terra, and Luna models[https://deploymentsafety\.openai\.com/gpt\-5\-6\-preview](https://deploymentsafety.openai.com/gpt-5-6-preview)Cited by:[§4\.1](https://arxiv.org/html/2609.22200#S4.SS1.p1.1),[Table 3](https://arxiv.org/html/2609.22200#S4.T3.4.11.1)\.
- OpenAI \(2026c\)OpenAIModel card for OpenAI privacy filter\.Note:Accessed 2026[https://cdn\.openai\.com/pdf/c66281ed\-b638\-456a\-8ce1\-97e9f5264a90/OpenAI\-Privacy\-Filter\-Model\-Card\.pdf](https://cdn.openai.com/pdf/c66281ed-b638-456a-8ce1-97e9f5264a90/OpenAI-Privacy-Filter-Model-Card.pdf)Cited by:[§1](https://arxiv.org/html/2609.22200#S1.p1.1),[§1](https://arxiv.org/html/2609.22200#S1.p4.1),[§2](https://arxiv.org/html/2609.22200#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2609.22200#S3.SS1.p4.1),[§4\.1](https://arxiv.org/html/2609.22200#S4.SS1.p1.1),[Table 3](https://arxiv.org/html/2609.22200#S4.T3.4.4.1)\.
- OpenMed Science \(2026\)OpenMed ScienceOpenMed\-PII\-SuperClinical: pii detection models\.Note:Accessed 2026[https://huggingface\.co/OpenMed/OpenMed\-PII\-SuperClinical\-Small\-44M\-v1](https://huggingface.co/OpenMed/OpenMed-PII-SuperClinical-Small-44M-v1)Cited by:[§2](https://arxiv.org/html/2609.22200#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2609.22200#S4.SS1.p1.1),[Table 3](https://arxiv.org/html/2609.22200#S4.T3.4.8.1),[Table 3](https://arxiv.org/html/2609.22200#S4.T3.4.9.1)\.
- Packeret al\.\(2023\)C\. Packer, S\. Wooders, K\. Lin, V\. Fang, S\. G\. Patil, I\. Stoica, and J\. E\. GonzalezMemGPT: towards LLMs as operating systems\.arXiv preprint arXiv:2310\.08560\.Note:arXiv:2310\.08560Cited by:[§1](https://arxiv.org/html/2609.22200#S1.p2.1)\.
- Pilánet al\.\(2022\)I\. Pilán, P\. Lison, L\. Øvrelid, A\. Papadopoulou, D\. Sánchez, and M\. BatetThe text anonymization benchmark \(tab\): a dedicated corpus and evaluation framework for text anonymization\.Computational Linguistics48\(4\),pp\. 1053–1101\.Note:arXiv:2202\.00443External Links:[Document](https://dx.doi.org/10.1162/coli%5Fa%5F00458)Cited by:[Table 1](https://arxiv.org/html/2609.22200#S1.T1.6.5.1),[§1](https://arxiv.org/html/2609.22200#S1.p4.1),[§2](https://arxiv.org/html/2609.22200#S2.p1.1),[§4\.7](https://arxiv.org/html/2609.22200#S4.SS7.p1.1)\.
- Ponomarenkoet al\.\(2026\)M\. Ponomarenko, S\. Abedini, M\. Shafieinejad, D\. B\. Emerson, S\. Mohapatra, and X\. HeCAPID: context\-aware PII detection for question\-answering systems\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 4: Student Research Workshop\),pp\. 320–331\.Note:arXiv:2602\.10074Cited by:[§2](https://arxiv.org/html/2609.22200#S2.p1.1)\.
- Qwen Team \(2025\)Qwen TeamQwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§3\.7](https://arxiv.org/html/2609.22200#S3.SS7.SSS0.Px1.p1.1)\.
- Savkinet al\.\(2025\)M\. Savkin, T\. Ionov, and V\. KonovalovSPY: enhancing privacy with synthetic PII detection dataset\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 4: Student Research Workshop\),pp\. 236–246\.Note:ACL Anthology 2025\.naacl\-srw\.23Cited by:[Table 1](https://arxiv.org/html/2609.22200#S1.T1.6.4.1),[§2](https://arxiv.org/html/2609.22200#S2.p1.1),[§4\.7](https://arxiv.org/html/2609.22200#S4.SS7.p1.1)\.
- Schwartz and Solove \(2011\)P\. M\. Schwartz and D\. J\. SoloveThe pii problem: privacy and a new concept of personally identifiable information\.New York University Law Review86,pp\. 1814–1894\.Cited by:[§3\.1](https://arxiv.org/html/2609.22200#S3.SS1.p1.1)\.
- Shaoet al\.\(2024\)Y\. Shao, T\. Li, W\. Shi, Y\. Liu, and D\. YangPrivacyLens: evaluating privacy norm awareness of language models in action\.InAdvances in Neural Information Processing Systems \(NeurIPS\) Datasets and Benchmarks Track,Note:arXiv:2409\.00138Cited by:[§2](https://arxiv.org/html/2609.22200#S2.SS0.SSS0.Px2.p1.1)\.
- Shenet al\.\(2026\)H\. Shen, Z\. Gu, H\. Hong, W\. Han, and H\. ChaiPII\-Bench: evaluating query\-aware privacy protection systems\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 4991–5026\.Note:arXiv:2502\.18545Cited by:[Table 1](https://arxiv.org/html/2609.22200#S1.T1.6.7.1),[§2](https://arxiv.org/html/2609.22200#S2.p1.1)\.
- Soldainiet al\.\(2024\)L\. Soldaini, R\. Kinney, A\. Bhagia, D\. Schwenk, D\. Atkinson,et al\.Dolma: an open corpus of three trillion tokens for language model pretraining research\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 15725–15788\.Note:arXiv:2402\.00159Cited by:[§1](https://arxiv.org/html/2609.22200#S1.p1.1)\.
- Staabet al\.\(2024\)R\. Staab, M\. Vero, M\. Balunović, and M\. VechevBeyond memorization: violating privacy via inference with large language models\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2310\.07298Cited by:[§2](https://arxiv.org/html/2609.22200#S2.SS0.SSS0.Px2.p1.1)\.
- Stubbset al\.\(2015\)A\. Stubbs, C\. Kotfila, and Ö\. UzunerAutomated systems for the de\-identification of longitudinal clinical narratives: overview of 2014 i2b2/uthealth shared task track 1\.Journal of Biomedical Informatics58,pp\. S11–S19\.External Links:[Document](https://dx.doi.org/10.1016/j.jbi.2015.06.007)Cited by:[§1](https://arxiv.org/html/2609.22200#S1.p4.1),[§2](https://arxiv.org/html/2609.22200#S2.p1.1),[§3\.4](https://arxiv.org/html/2609.22200#S3.SS4.p1.1)\.
- Tjong Kim Sang and De Meulder \(2003\)E\. F\. Tjong Kim Sang and F\. De MeulderIntroduction to the CoNLL\-2003 shared task: language\-independent named entity recognition\.InProceedings of the Seventh Conference on Natural Language Learning at HLT\-NAACL 2003,pp\. 142–147\.Note:arXiv:cs/0306050Cited by:[§2](https://arxiv.org/html/2609.22200#S2.p1.1)\.
- Vatset al\.\(2026\)G\. Vats, A\. Agrawal, S\. Singhal, A\. Dash, P\. Selvaraj, V\. Jhawar, R\. P\. Chenna, and B\. Y\. M\. GREDACT: a systematically controlled multilingual benchmark for personal information detection\.arXiv preprint arXiv:2606\.19881\.Cited by:[Table 1](https://arxiv.org/html/2609.22200#S1.T1.6.6.1),[§1](https://arxiv.org/html/2609.22200#S1.p4.1),[§2](https://arxiv.org/html/2609.22200#S2.p1.1)\.
- Wanget al\.\(2025\)S\. Wang, F\. Yu, X\. Liu, X\. Qin, J\. Zhang, Q\. Lin, D\. Zhang, and S\. RajmohanPrivacy in action: towards realistic privacy mitigation and evaluation for LLM\-powered agents\.InFindings of the Association for Computational Linguistics: EMNLP,Note:arXiv:2509\.17488Cited by:[§1](https://arxiv.org/html/2609.22200#S1.p3.1),[§2](https://arxiv.org/html/2609.22200#S2.SS0.SSS0.Px2.p1.1)\.
- Xiet al\.\(2023\)Z\. Xi, W\. Chen, X\. Guo, W\. He, Y\. Ding, B\. Hong, M\. Zhang, J\. Wang, S\. Jin, E\. Zhou, R\. Zheng, X\. Fan, X\. Wang, L\. Xiong, Y\. Zhou, W\. Wang, C\. Jiang, Y\. Zou, X\. Liu, Z\. Yin, S\. Dou, R\. Weng, W\. Cheng, Q\. Zhang, W\. Qin, Y\. Zheng, X\. Qiu, X\. Huang, and T\. GuiThe rise and potential of large language model based agents: a survey\.arXiv preprint arXiv:2309\.07864\.Note:arXiv:2309\.07864Cited by:[§1](https://arxiv.org/html/2609.22200#S1.p2.1)\.
- Yagoubiet al\.\(2026\)F\. E\. Yagoubi, G\. Badu\-Marfo, and R\. A\. MallahAgentLeak: a benchmark for internal\-channel privacy leakage in multi\-agent LLM systems\.Note:arXiv:2602\.11510Cited by:[§1](https://arxiv.org/html/2609.22200#S1.p3.1),[§2](https://arxiv.org/html/2609.22200#S2.SS0.SSS0.Px2.p1.1)\.
- Yaoet al\.\(2023\)S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. CaoReAct: synergizing reasoning and acting in language models\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2210\.03629Cited by:[§1](https://arxiv.org/html/2609.22200#S1.p2.1)\.
- Zaratianaet al\.\(2026\)U\. Zaratiana, A\. Lewis, and G\. Hurn\-MaloneyGLiNER2\-PII: a multilingual model for personally identifiable information extraction\.arXiv preprint arXiv:2605\.09973\.Note:arXiv:2605\.09973Cited by:[§2](https://arxiv.org/html/2609.22200#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2609.22200#S3.SS1.p4.1),[§4\.1](https://arxiv.org/html/2609.22200#S4.SS1.p1.1),[Table 3](https://arxiv.org/html/2609.22200#S4.T3.4.5.1)\.

## Appendix ADatasheet for Datasets

Following[Gebru et al\. \(2021\)](https://arxiv.org/html/2609.22200#bib.bib4), we answer the core questions inline; the release will include a data card with the full questionnaire\.

#### Motivation\.

PII\-TRACEwas created to evaluate PII detection*in multi\-turn conversational context*, an axis missing from prior sentence\- or record\-level resources\.

#### Composition\.

Each instance is a synthetic multi\-turn user–assistant conversation with character\-level identifier mentions over nine types, per\-mention cluster ids, and a separate sensitive\-attribute layer\. The corpus has 13,148 conversations across 13 languages \(train/validation/test==9,202 / 2,024 / 1,922\), with per\-record metadata\. Figure[5](https://arxiv.org/html/2609.22200#A1.F5)shows a complete released record\.

\{"id":"conv\_00042","language":"en","length\_bucket":"<1k",

"document\_format":"unstructured","clean\_negative":false,

"text":"U:I’mDanaOkoye,mynumberis\+001011110000\.\\nA:Thanks,DanaOkoye,I’venotedthecallbackline\.\\nU:PleasekeepDanaOkoyeonfile\.",

"spans":\[

\{"start":7,"end":17,"label":"private\_person","subtype":"full\_name","entity\_id":1,"entity\_mentions":3\},

\{"start":32,"end":48,"label":"private\_phone","subtype":"phone","entity\_id":2,"entity\_mentions":1\},

\{"start":61,"end":71,"label":"private\_person","subtype":"full\_name","entity\_id":1,"entity\_mentions":3\},

\{"start":118,"end":128,"label":"private\_person","subtype":"full\_name","entity\_id":1,"entity\_mentions":3\}\],

"attribute\_spans":\[\]\}

Figure 5:A complete released record \(values fabricated\)\. Offsets index thetextfield;entity\_idlinks repeated mentions of one identifier, andentity\_mentionsis the cluster size used by consistent detection\.Table 6:The ninePII\-TRACEidentifier types: sub\-types, one\-line definition \(a span is labeled only when it identifies a*private*person; placeholders and public entities are excluded\), and regulatory crosswalk\.
#### Collection and annotation\.

The corpus is derived from a deduplicated sample of real user–assistant conversations from a production assistant; a “turn” is one delivered message\. To anonymize each conversation, multiple frontier LLMs locate the identifiers, following a written guideline, and a rule\-based pass groups repeated mentions of the same typed identifier into an entity; sensitive attributes are marked separately\. We inherit these labels asPII\-TRACE’s gold\. The labeler prompt and guideline are summarized in Appendix[E](https://arxiv.org/html/2609.22200#A5), and the release will include them\.

## Appendix BFull Taxonomy and Crosswalk

Table[6](https://arxiv.org/html/2609.22200#A1.T6)lists the nine identifier types with their sub\-types and definitions, plus a crosswalk to standard regulatory categories; the full row\-by\-row mapping to OMB, NIST SP 800\-122, ISO/IEC 29100, and HIPAA will accompany the release\. The label space subsumes those of the deployed detectors we evaluate, so their outputs map directly into our types; the residualother\_piicovers identifiers outside the eight concrete types\. The eight sensitive\-attribute classes \(health condition, religion, race/ethnicity, sexuality, political view, disability, age, gender\) form the separate attribute layer and are not part of the identifier taxonomy\.

## Appendix CDetector: Additional Details

#### Training details\.

We train with AdamW \(learning rate10−410^\{\-4\}, effective batch256256\) on88data\-parallel GPUs for33epochs, selecting the best checkpoint on held\-out loss\. The training data andPII\-TRACE’s sources share the same production traffic and matching conversation ids\.

#### Frontier\-LLM detector protocol\.

The four frontier LLMs in Table[3](https://arxiv.org/html/2609.22200#S4.T3)are prompted zero\-shot, with no few\-shot examples\. A system message states the task and the nine identifier types and asks for a JSON list of \{text,type\} objects whosetextis copied verbatim from the input \(Appendix[E](https://arxiv.org/html/2609.22200#A5)\)\. Documents are read in8,0008\{,\}000\-character windows with400400\-character overlap; each returned string is mapped back to character offsets by exact substring match, and offsets are merged and de\-duplicated across windows\.

## Appendix DExtended Results

#### Full detection results\.

Table[7](https://arxiv.org/html/2609.22200#A4.T7)completes the entity\-level view of §[4\.3](https://arxiv.org/html/2609.22200#S4.SS3)for all twelve detectors\. PII\-Tracer leads on all three coverage columns \(CD0\.8530\.853, CD\-m0\.7940\.794, xCD0\.7760\.776\)\. Its FP0of0\.3850\.385is comparable to the frontier LLMs \(0\.2310\.231–0\.3860\.386\) and well below every public specialized baseline \(0\.5570\.557–0\.9720\.972\)\. Coverage and false positives must be read together, since a detector can raise CD by flagging nearly every document, as Presidio and the OpenMed models do\.

Table 7:Entity\-level results on the test split: consistent detection \(CD\), multi\-mention CD \(CD\-m\), cross\-turn CD \(xCD\), and PII\-free false\-positive rate FP0\(lower is better\)\. Best per column inbold\.
#### Inference policy for long documents\.

Table[8](https://arxiv.org/html/2609.22200#A4.T8)decodes the same PII\-Tracer checkpoint under three long\-input policies\. Relative to the single window, sliding windows raise recall on documents at or above 10k characters from0\.6870\.687to0\.9750\.975and CD\-m from0\.7940\.794to0\.9540\.954, while character precision drops from0\.5070\.507to0\.4930\.493; chunked decoding lies between the two\.

Table 8:PII\-Tracer under three long\-input decoding policies: a single window \(trunc\), non\-overlapping windows \(chunk\), and 50%\-overlap sliding windows \(slide\)\. R<10kand R≥10ksplit character recall at 10k characters\.

## Appendix EPrompts

The four LLM prompts of this work are shown below\.

Gold Labeler Prompt \(§[3\.3](https://arxiv.org/html/2609.22200#S3.SS3)\)Task\.You are an expert PII labeler producing gold labels for … conversations\. Your labels will train a downstream PII\-masking model, so they must reflect industry\-standard definitions of Personally Identifiable Information …Test\.Would a knowledgeable reader, given this string and the surrounding context, be able to identify or contact a specific person? If the string instead identifies a publicly\-listed organization, a public\-figure persona, or a fictional vignette persona, the answer is no …Traps\.Even when the string looks exactly like a name, address, phone, URL, or date, emit nothing if the context matches a trap: a named officer or staff member in an organizational role, reserved or documentation values, citation or tool tokens …Output\.A single JSON object with two fields: a conversation\-level sensitivity head and the token\-level PII span list of \{label,text\} objects …

Paraphrase Prompt \(Figure[2](https://arxiv.org/html/2609.22200#S3.F2), step 3\)Task\.Rewrite a snippet from a real user–assistant chat so it is not verbatim the original, while preserving its meaning, intent, tone, language, and Markdown structure \(headings, lists, tables, code fences …\)\.Rule 1: placeholders\.The text may contain special placeholder characters; each stands for a redacted value\. Reproduce every placeholder exactly once, in the natural grammatical position where its value belongs; never delete, duplicate, translate …Rule 2: de\-identify context\.Any real\-world specific that is not a placeholder \(company, product, or brand names; unusual place names; distinctive verbatim phrases …\) must become a generic or plausibly invented equivalent\.Output\.Roughly the same length and the same language as the input; only the rewritten snippet, no preamble …

Second\-LLM Audit Prompt \(§[3\.5](https://arxiv.org/html/2609.22200#S3.SS5)\)Task\.Audit synthetic\-conversation text for a PII benchmark; all names, contacts, and IDs are supposed to be fabricated …Report\.Only strings that look like a real, specific individual’s genuine identifying data that fabrication might have missed \(an intact real email, phone, or handle; a real public figure named as the actual user …\)\.Ignore\.Generic names, places, and dates; clearly synthetic values\.Output\.JSON\{"suspect": bool, "items": \[\{"text", "why"\} …\]\}\.

Frontier\-LLM Detector Prompt \(§[4](https://arxiv.org/html/2609.22200#S4)\)Task\.You are a precise PII detector operating on text from a multi\-turn assistant conversation\. Extract every span that is personal information identifying a private individual …Rule\.Mark a span only when, in this conversational context, it helps identify a private person; do not mark public figures, organizations, fictional or placeholder values … A date counts only when it identifies a person …Output\.A JSON list of \{text,type\} objects over the nine identifier types, withtextcopied verbatim from the input …