Institution-Specific LLM Prompting Recovers PHI That De-identification Systems and Their Gold Standards Both Miss

# Institution-Specific LLM Prompting Recovers PHI That De-identification Systems and Their Gold Standards Both Miss
Source: [https://arxiv.org/html/2608.17051](https://arxiv.org/html/2608.17051)
Institution\-Specific LLM Prompting Recovers PHI That De\-identification Systems and Their Gold Standards Both Miss

Article type: Research and Applications

Authors\.Daniel Palacios, BS1,2,3,4,†\\dagger, Matthew Brady Neeley, BS1,2,3,4,†\\dagger, Angel Adetomike Otto, MS5, Shalini Dhamodharan, MS5, John P\. Woodhouse, BA5, Chi\-fan Lin, MS5, Mark Zobeck, MD, MPH5,∗\*, Zhandong Liu, PhD1,2,3,4,∗\*, Hyun\-Hwan Jeong, PhD2,3,4,∗\*\.

†\\daggerThese authors contributed equally\.∗\*Co\-corresponding authors\.

Affiliations\.

1\.Quantitative and Computational Biosciences, Baylor College of Medicine, Houston, Texas, USA

2\.Department of Pediatrics, Baylor College of Medicine, Houston, Texas, USA

3\.Jan and Dan Duncan Neurological Research Institute, Texas Children’s Hospital, Houston, Texas 77030, USA

4\.Data Science Center, Texas Children’s Hospital, Houston, Texas 77030, USA

5\.Section of Hematology\-Oncology, Department of Pediatrics, Baylor College of Medicine, Houston, Texas, USA

Corresponding author\.Hyun\-Hwan Jeong, Department of Pediatrics, Baylor College of Medicine, and Jan and Dan Duncan Neurological Research Institute, Texas Children’s Hospital, 1250 Moursund Street, Houston, TX 77030, USA\. Telephone: \+1 832\-824\-1000, ext\. 25535\. Email:hyun\-hwan\.jeong@bcm\.edu\.

Keywords:data anonymization; electronic health records; natural language processing; machine learning; large language models\.

Word count\.Abstract: 239\. Main body: 3,808\. Tables: 1\. Figures: 4\.

ABSTRACT

Objective\.Secondary use of electronic health records requires de\-identification, yet existing systems miss*institutionally situated*protected health information \(PHI\): identifiers such as hospital abbreviations and building names whose status is locally determined\. We evaluated whether large language models \(LLMs\) can close this gap through in\-context learning while controlling precision and recall\.

Materials and Methods\.On 100 annotated pediatric oncology notes \(5,322 PHI spans\) we benchmarked eight LLMs against two purpose\-built systems \(Stanford TiDE, OpenMed PII\) and two pattern\-based baselines\. Each LLM was run in three prompt modes:*Baseline*\(HIPAA\-aligned\),*Targeted*\(plus institutional PHI categories\), and*Precision*\(plus instructions against over\-redaction\)\. We also compared 14 multi\-agent and ensemble configurations\. Recall was the primary safety metric\.

Results\.LLMs outperformed the purpose\-built systems \(best F1=0\.918±\\pm0\.001, Sonnet 4\.6, vs\. TiDE 0\.779\), with advantages concentrated in contextual categories\. Naming the missed categories recovered 79% \(48/61\), and*Precision*recovered precision\. No agentic architecture beat single\-pass prompting \(F1 0\.906–0\.908\)\. The LLMs also redacted 414 identifiers absent from the gold standard, scored false positive; expert review of 49 confirmed all as true PHI, and re\-annotating the 10 highest\-discrepancy notes \(\+227 spans\) lifted*Precision*to recall=0\.981 \(F1=0\.907±\\pm0\.002\)\.

Discussion\.Naming an institution’s own identifiers and warning against over\-redaction resolves both the institutional PHI gap and the precision–recall trade\-off in one LLM call per note\. LLMs can cost more, but that buys a way to audit the standard\.

Conclusion\.LLMs are an adaptable alternative to purpose\-built de\-identification; institution\-specific prompt development should be the primary adaptation\.

## Introduction

Secondary use of electronic health records \(EHRs\) requires de\-identification of protected health information \(PHI\) under HIPAA’s Safe Harbor standard, which specifies 18 identifier categories\[[40](https://arxiv.org/html/2608.17051#bib.bib4),[28](https://arxiv.org/html/2608.17051#bib.bib5)\]\. These define*canonical*PHI \(names, dates, medical record numbers, geographic data\) that models recognize from general linguistic patterns\. Real clinical notes also containinstitutionally situated PHI: identifiers whose status depends on local institutional context, including hospital abbreviations \(“TCH” for Texas Children’s Hospital\), building names \(“Mark Wallace Tower”\), internal clinic codes, and provider naming conventions unique to an institution\. Although not enumerated among the 18 categories, they fall within HIPAA’s definition of PHI under both the Safe Harbor catch\-all for “any other unique identifying number, characteristic, or code” \([45 CFR §164\.514\(b\)\(2\)\(i\)\(R\)](https://www.ecfr.gov/current/title-45/section-164.514)\) and the expert\-determination standard \([§164\.514\(b\)\(1\)](https://www.ecfr.gov/current/title-45/section-164.514)\)\[[28](https://arxiv.org/html/2608.17051#bib.bib5),[41](https://arxiv.org/html/2608.17051#bib.bib6)\]: in a pediatric\-oncology population a named specialty facility, a rare diagnosis, and a service date can jointly re\-identify a record even after every canonical identifier is removed\. Each element is independently re\-identifying: the set of facilities a patient visits is itself a signature\[[23](https://arxiv.org/html/2608.17051#bib.bib24),[37](https://arxiv.org/html/2608.17051#bib.bib27)\]and diagnosis codes alone can breach privacy\[[22](https://arxiv.org/html/2608.17051#bib.bib25)\]; in Washington State discharge data carrying hospital, diagnosis, and attending physician but no names or addresses, news reports uniquely matched 35 of 81 named patients to their records\[[38](https://arxiv.org/html/2608.17051#bib.bib26)\]\. Removal is a regulatory requirement, not an optional refinement\.

A model prompted only on HIPAA’s 18 categories has no basis to recognize that “TCH” is an identifying abbreviation or that a four\-digit pager number is a staff identifier\. These are failures not of capability but of*specification*: the model was never told they require redaction\. The OCR guidance anticipates this, warning that esoteric notation such as acronyms known to only a few of a covered entity’s employees can lead to either unnecessary redaction or failure to redact\[[28](https://arxiv.org/html/2608.17051#bib.bib5)\]\. Nor can the specification be written once and reused, since note templates, abbreviations, and patient populations differ by site\[[27](https://arxiv.org/html/2608.17051#bib.bib35)\]: Veterans Health Administration notes required customizing to institution\-specific formats\[[12](https://arxiv.org/html/2608.17051#bib.bib16),[24](https://arxiv.org/html/2608.17051#bib.bib37)\], and cross\-institute evaluations report consistent degradation on transfer\[[46](https://arxiv.org/html/2608.17051#bib.bib42)\]\. The gap is acute in pediatric settings, where large multidisciplinary teams author notes that reference caregivers and carry institutional shorthand\.

Recent work has begun isolating institution\-level identifiers as a distinct annotation class, notably thehospitalcategory of SHIELD — a recent teacher–student distillation framework for clinical de\-identification — on which both its teacher and student models record their lowest precision\[[33](https://arxiv.org/html/2608.17051#bib.bib21)\]\. No prior study has characterized*why*these identifiers fail, nor shown the failure remediable through specification rather than retraining\. Automated de\-identification has progressed from rule\-based systems\[[36](https://arxiv.org/html/2608.17051#bib.bib7),[26](https://arxiv.org/html/2608.17051#bib.bib8)\]through neural sequence models\[[11](https://arxiv.org/html/2608.17051#bib.bib9),[20](https://arxiv.org/html/2608.17051#bib.bib10)\]to transformer\-based NER\[[19](https://arxiv.org/html/2608.17051#bib.bib11),[2](https://arxiv.org/html/2608.17051#bib.bib12),[31](https://arxiv.org/html/2608.17051#bib.bib3)\], benchmarked on i2b2/UTHealth shared tasks\[[35](https://arxiv.org/html/2608.17051#bib.bib13),[42](https://arxiv.org/html/2608.17051#bib.bib14)\]\. Two purpose\-built systems anchor our comparison: OpenMed’s domain\-adapted NER\[[30](https://arxiv.org/html/2608.17051#bib.bib28)\]and Stanford’s TiDE, which combines NER, pattern matching, and known\-PHI lookup\[[10](https://arxiv.org/html/2608.17051#bib.bib23),[6](https://arxiv.org/html/2608.17051#bib.bib39)\]with Hiding\-in\-Plain\-Sight surrogates\[[7](https://arxiv.org/html/2608.17051#bib.bib30)\]\.

Large language models \(LLMs\) are competitive with or superior to traditional NER systems on adult clinical benchmarks\[[21](https://arxiv.org/html/2608.17051#bib.bib15)\]\. Closest to this work, Wiest et al\.\[[44](https://arxiv.org/html/2608.17051#bib.bib41)\]benchmarked eight local LLMs on 250 clinical letters, reporting∼\\sim99\.2% PHI removal in a different language and note type; Altalla’ et al\.\[[3](https://arxiv.org/html/2608.17051#bib.bib32)\]evaluated GPT\-3\.5 and GPT\-4 \(P≈\\approx0\.99, R≈\\approx0\.83\), and Pissarra et al\.\[[32](https://arxiv.org/html/2608.17051#bib.bib33)\]found LLMs and Presidio baselines complementary\. Multi\-agent architectures have also been explored: TEAM\-PHI\[[45](https://arxiv.org/html/2608.17051#bib.bib18)\]ranks de\-identification models with majority\-voting*evaluation*agents and no gold labels, OEMA\[[39](https://arxiv.org/html/2608.17051#bib.bib19)\]uses three agents for zero\-shot clinical NER, and SHIELD\[[33](https://arxiv.org/html/2608.17051#bib.bib21),[15](https://arxiv.org/html/2608.17051#bib.bib22),[16](https://arxiv.org/html/2608.17051#bib.bib34)\]selects a teacher labeler for distillation into locally\-deployable students\[[34](https://arxiv.org/html/2608.17051#bib.bib1)\]\. The prover\-verifier framework\[[17](https://arxiv.org/html/2608.17051#bib.bib17)\]justifies generation\-then\-verification architectures, untested in de\-identification\.

A second challenge is the precision–recall trade\-off\[[5](https://arxiv.org/html/2608.17051#bib.bib36)\]\. De\-identification has long favored recall, since a missed identifier is a privacy breach whereas over\-redaction only removes clinical content\[[35](https://arxiv.org/html/2608.17051#bib.bib13),[13](https://arxiv.org/html/2608.17051#bib.bib38)\]\. Distinctive in the single\-pass LLM setting are its*magnitude*and*model\-dependence*: no single prompt optimizes both objectives across models, and some over\-redact severely enough to degrade data utility, an effect standard metrics do not capture\[[1](https://arxiv.org/html/2608.17051#bib.bib2)\]\. Conversely, adversarial LLM\-based re\-identification\[[25](https://arxiv.org/html/2608.17051#bib.bib20)\]shows even strong systems leave notes vulnerable\.

We hypothesized that what limits de\-identification is*specification*rather than model capability: that naming an institution’s own identifiers in context would resolve both the institutional PHI gap and the precision–recall trade\-off\. We tested this against the competing explanation that the trade\-off demands architectural remedy, using dual\-pass and Scrubber–Auditor pipelines drawn from the prover\-verifier paradigm\[[17](https://arxiv.org/html/2608.17051#bib.bib17)\]and multi\-agent clinical NLP\[[45](https://arxiv.org/html/2608.17051#bib.bib18)\]\(Supplementary Note 3\)\. We make three contributions\. First, benchmarking 8 LLMs against two purpose\-built systems \(Stanford TiDE, OpenMed PII\), pattern\-based baselines, and multi\-stage pipelines on 100 expert\-annotated pediatric oncology notes \(5,322 PHI spans\), we introduceinstitutionally situated PHIas a failure mode common to all\. Second, in\-context learning adapts de\-identification without fine\-tuning: naming the missed categories recovers most of them and anti\-over\-redaction instructions restore precision, while error analysis exposed gold\-standard gaps that expert re\-annotation confirmed as true PHI\. Third, the trade\-off resolves in a single pass: no agentic architecture outperformed it on F1, locating the bottleneck in specification rather than inference\-time computation\.

## Methods

### Study design and clinical corpus

We benchmarked the LLMs and purpose\-built systems listed in Table[1](https://arxiv.org/html/2608.17051#Sx2.T1), plus two pattern\-based baselines \(regex\-only, and spaCy NER \+ regex\), on 100 pediatric oncology clinical notes, with the LLMs run under each of the three prompt conditions below\. No model was trained or fine\-tuned\. Reporting follows TRIPOD\-LLM\[[14](https://arxiv.org/html/2608.17051#bib.bib47)\]; Tables S1 and S2 give the item\-by\-item adherence table\.

Clinical note corpus\.Our corpus consisted of 100 English\-language clinical notes from the pediatric oncology service at Texas Children’s Hospital \(Houston, TX, USA; 96 patients\), drawn by pseudorandom sort under a fixed seed from notes dated on or after January 1, 2021, unstratified\. Notes ranged from 91 to 32,767 characters \(median, 6,504; mean, 10,058\), five truncated at the export limit\. Three clinical annotators, trained by a pediatric oncologist and informatician to identify both the 18 Safe Harbor categories and institutionally situated identifiers, annotated 5,322 PHI spans across 97 notes \(3 contained no PHI\), resolving questions with the oncologist and regulatory personnel; because annotation followed consensus adjudication rather than independent double annotation, no agreement statistic was computed\. Figure[1](https://arxiv.org/html/2608.17051#Sx11.F1)C shows the category distribution \(Supplementary Note 1\.1\)\.

Validation dataset\.To check that our findings were not corpus\-specific, we also evaluated all systems on 49 notes \(409 gold PHI spans\) from the USDHUB repository, a separately curated pediatric neurology sample from the same institution, independently de\-identified by a different group, under the identical*Baseline*prompt and scoring pipeline \(Supplementary Note 4\)\. Because USDHUB was built for TiDE and ships patient\-specific provisioning material, TiDE was run in two bracketing configurations: fully unprovisioned \(identical footing to the LLMs\) and maximally provisioned with the supplied known\-PHI dictionary and per\-note identifier header\. For the provisioned run only, precision and F1 are scored on the note body, since the injected header carries no gold spans \(Supplementary Note 4\.1\)\.

### De\-identification systems

LLM models\.We evaluated eight LLMs via AWS Bedrock’s Converse API \(Table[1](https://arxiv.org/html/2608.17051#Sx2.T1)\) at temperature=0\.0=0\.0\(except Opus 4\.8, which does not accept the parameter via Bedrock; Supplementary Note 3\.3\), max output tokens=65,000=65\{,\}000, and up to 3 retries with exponential backoff\. For visual clarity, main\-text figures present the top 4 LLMs by*Baseline*F1 \(Sonnet 4\.6, Opus 4\.8, GLM\-5, DeepSeek V3\.2\); all eight appear in Tables S3–S6\.

Table 1:LLMs and purpose\-built systems evaluated\. The two pattern\-based baselines \(regex\-only; spaCy NER \+ regex\) are specified in the text and are included in all reported comparisons\.SystemTypeFamilyParametersClaude Opus 4\.8LLMAnthropicUndisclosedClaude Sonnet 4\.6LLMAnthropicUndisclosedGPT\-oss\-120BLLMOpenAI120BGPT\-oss\-20BLLMOpenAI20BGLM\-5LLMZhipu AI754BKimi K2\.5LLMMoonshot1\.1TMiniMax M2\.5LLMMiniMax229BDeepSeek V3\.2LLMDeepSeek685BStanford TiDENER \+ RulesStanford NLP—OpenMed PIINER \(token classif\.\)OpenMed434MStanford TiDE\.TiDE\[[10](https://arxiv.org/html/2608.17051#bib.bib23)\]is a production system detecting PHI through NER, regex, and known\-PHI matching against a supplied list of each patient’s real identifiers\. Since the LLMs and OpenMed never receive that list, input parity required running TiDE on note text alone, with known\-PHI matching disabled so field lookups find no rows while NER and regex fire normally\. This understates TiDE’s production performance but isolates contextual reasoning from pattern matching \(Supplementary Note 4\.1\)\.

OpenMed PII\.We used OpenMed\-PII\-SuperClinical\-Large\-434M\-v1\[[30](https://arxiv.org/html/2608.17051#bib.bib28),[29](https://arxiv.org/html/2608.17051#bib.bib40)\], a transformer token\-classification model fine\-tuned for personally identifiable information \(434M parameters, 54 sensitive\-information types\), whose bracketed placeholders match the LLM output format, so the same scoring pipeline applies\.

### Prompt conditions

The*Baseline*prompt asks the model to return the exact input text with all 18 HIPAA Safe Harbor categories replaced by typed placeholders, enumerating the categories with examples\.*Targeted*appends four categories of institutionally situated PHI that*Baseline*error analysis surfaced: staff names adjacent to credentials, pager and Voalte numbers, institution names and abbreviations \(“TCH”, “TXCH”\), and building or facility names\. Department and clinic codes, a fifth subcategory the same analysis surfaced, were*not*named \(Table S7\)\.*Precision*retains all of*Targeted*and adds a “DO NOT OVER\-REDACT” block covering the six largest observed false\-positive categories, each with WRONG→\\rightarrowRIGHT pairs, closing with an instruction to redact anyway when genuinely uncertain \(Supplementary Note 2\.2\)\.

### Multi\-stage architectures

We tested whether architectural complexity could outperform single\-pass prompt engineering, using*multi\-stage*for any pipeline with more than one LLM call,*multi\-agent*for pipelines whose calls occupy distinct roles \(scrubber, auditor, verifier\), and*heterogeneous*for the subset combining two or more models\. Three paradigms were explored:dual\-pass iterative refinement\(Sonnet 4\.6 applied twice, the second pass searching its own output for residual institutional PHI\),heterogeneous Scrubber–Auditor\(a recall\-maximizing Sonnet 4\.6 scrubber followed by a precision\-focused Opus 4\.8 auditor\), andcross\-model dual\-pass\(two models with complementary error profiles\)\. Descriptions, prompts, and design rationale are in Supplementary Note 3\. The top 3 configurations were run for 5 independent trials with statistical comparison \(McNemar’s exact test, Wilcoxon signed\-rank, bootstrap 95% CIs, Holm–Bonferroni corrected across five tests at family\-wiseα\\alpha=0\.05; Supplementary Note 3\.3\)\.

### Enhanced gold standard

Error analysis \(Results\) showed the original annotation had omitted institutional identifiers the LLMs correctly detected, penalizing correct redactions, deflating F1 and obscuring differences between architectures\. We therefore re\-annotated a subset for relative comparison\. From the 414 candidate spans surfaced corpus\-wide we selected the 10 notes with the largest model–gold discrepancy; a domain expert adjudicated 49 in\-context instances covering 22 unique institutional terms, confirming all 49 as TRUE\_PHI\. Annotating those terms at every occurrence added 227 spans \(209 Geographic Data, 9 Name, 9 Other Unique ID\), for 1,758 total versus 1,531 original; institutional terms were assigned to Geographic Data, which is why that category grows far beyond its corpus\-wide count \(Supplementary Note 2\.5\)\. Three biases follow: the subset overrepresents institutional PHI density; re\-annotation covered only the 22 surfaced terms; and the candidates came from model output, partly crediting those models for spans they surfaced\. Expert adjudication establishes the added spans are genuine PHI, not that they exhaust it, so the enhanced standard serves only for relative comparison on these 10 notes\. All multi\-stage architectures, and a five\-trial re\-evaluation of the three prompt conditions, were scored against it\.

### Evaluation pipeline

Each gold PHI span is atrue positive \(TP\)when its text is absent from the output, and afalse negative \(FN\)when an exact substring search still finds it, when masking is partial, or when no output is returned\. Afalse positive \(FP\)is an emitted placeholder matching no gold annotation\. Matching is type\-agnostic, so a date masked as\[NAME\]still counts as TP\. Recall is TP/\(TP\+FN\) and precision TP/\(TP\+FP\), whose denominator mixes gold spans with emitted placeholders; that unit mismatch cuts both ways, raising precision where a system merges adjacent gold spans and lowering it where one is split across several\. Text also occurring as legitimate non\-PHI \(“May” as name vs\. month\) can produce spurious counts\. Outputs shorter than 50% of the note are scored as failures with all spans FN\. False negatives were categorized ascanonicalorinstitutionally situatedPHI\. Placeholder alignment, TiDE’s surrogate\-based precision scoring, the mixed\-unit bias, and the full rule set are in Supplementary Note 1\.2\. Figures use Matplotlib in the soft\-fill style of PubliPlots\[[4](https://arxiv.org/html/2608.17051#bib.bib29)\]\.

### Ethical considerations

This study was conducted under IRB protocol H\-52222 at Baylor College of Medicine / Texas Children’s Hospital\. All data remained within the institutional environment, and LLM inference via AWS Bedrock ran under a HIPAA\-compliant Business Associate Agreement under which prompts and outputs are neither retained nor used for training\. LLMs are the object of study here; any use of AI tools in manuscript preparation is disclosed in Additional Contributions\.

## Results

Recall is the primary safety metric, since a missed span is a potential privacy violation whereas an over\-redacted one only degrades data utility\. We select on F1 to keep the precision cost of recall gains visible, but because F1 weights the two errors equally we also tabulate the recall\-weightedF2F\_\{2\}\(5​P​R/\(4​P\+R\)5PR/\(4P\+R\); Tables S3 and S8\), which reorders only adjacent pairs and leaves the leaders unchanged\.

### LLMs outperform traditional de\-identification on pediatric oncology notes

Under identical input \(*Baseline*prompt for LLMs\), LLMs substantially outperformed all traditional approaches \(Figure[2](https://arxiv.org/html/2608.17051#Sx11.F2)\)\. The spaCy NER \+ regex baseline reached 77\.3% recall at only 34\.4% precision \(7,832 false positives\), since general\-purpose NER labels medications, diagnoses, and anatomy as entities; regex alone reached 57\.4% recall at 96\.4% precision, working for structured but not context\-dependent PHI\. Stanford TiDE reached 75\.9% recall \(F1=0\.779\) and OpenMed PII 80\.1% recall at 69\.4% precision \(F1=0\.743\), the highest non\-LLM recall but lower F1 \(Table S3, Supplementary Figure 1\)\. The same ordering held on the 49\-note validation set \(best LLM F1 0\.894 vs\. 0\.866\), though the margin narrowed on that canonical\-PHI\-dominated corpus, and onF2F\_\{2\}the best\-LLM advantage over TiDE there falls to 0\.001 \(Supplementary Note 4\)\.

Among LLMs, Sonnet 4\.6 combined 96\.0% recall with 88\.3% precision for the best F1 \(0\.920 in the primary run; 0\.918±\\pm0\.001 across five trials\); Opus 4\.8 matched its precision at lower recall \(93\.1%, F1=0\.905\); and the six LLMs without a failure mode exceeded TiDE’s recall by 0\.15–0\.22 \(Table S3\), the exception being GPT\-oss\-20B, whose truncation puts it below TiDE\. The trade\-off is model\-dependent: under*Baseline*the highest\-recall models over\-redact enough to generate thousands of false positives \(DeepSeek V3\.2 2,264 FP at 98\.3% recall; Kimi K2\.5 3,251 FP at 96\.9%\), whereas Sonnet 4\.6 produces only 679 FP at comparable recall\.

### Per\-category analysis reveals shared weaknesses on institutionally situated PHI

Because aggregate recall is dominated by Date spans \(72\.1% of gold annotations\), per\-category results \(Figure[2](https://arxiv.org/html/2608.17051#Sx11.F2)B; all twelve systems and ten categories in Table S4\) show both*why*LLMs outperform traditional systems and*where*they fail\. TiDE achieves perfect recall on MRN but fails where context is required: Other Unique ID \(15\.7%\), Phone \(47\.9%\), and Geographic Data \(65\.0%\)\. Averaged over the top 4 LLMs, recall exceeds TiDE by 0\.49 on Phone and 0\.48 on Other Unique ID — identifiers in non\-standard formats such as pager codes and internal extensions that evade regex but are readable from context \(Supplementary Note 2\.3\)\. The panel also reveals the LLMs’ shared weakness: even the best models reach only 50–78% recall on Other Unique ID and 69–99% on Geographic Data, the categories most enriched for institutionally situated PHI\.

### In\-context learning enables control over the precision–recall trade\-off

Unlike TiDE or regex baselines, whose behavior is fixed by their pattern libraries, LLMs can be steered by prompt design alone\. We developed three prompt conditions through iterative error analysis on Sonnet 4\.6, then evaluated all across the model set\. Decomposing Sonnet 4\.6’s 211*Baseline*false negatives \(Figure[2](https://arxiv.org/html/2608.17051#Sx11.F2)C, a representative run bracketed by the five\-trial means in Table S9\) identified institutional PHI as the blind spot: 29% of misses \(61/211\), dominated by institution abbreviations \(38\) and building or facility names \(15\)\. Table S7 gives the subcategory taxonomy with failure mechanisms and recovery rates\.

Appending those four categories to the baseline instructions \(*Targeted*\) reduced Sonnet 4\.6’s institutional false negatives from 61 to 13 \(78\.7% recovery\), with the largest gains on Other Unique ID \(recall 0\.548 to 0\.791\) and Name \(0\.928 to 0\.971\), raising overall recall from 0\.958 to 0\.975 \(5\-trial means; Table S9\)\. It also carried a precision penalty \(0\.881 to 0\.807\), as the added instructions over\-redacted clinical content in six recurring categories \(Supplementary Note 2\.2\)\.*Precision*keeps those categories and appends anti\-over\-redaction WRONG→\\rightarrowRIGHT pairs, recovering precision to 0\.829 for a modest recall cost \(0\.975 to 0\.969\) and F1=0\.893±\\pm0\.004; the full progression nets higher recall \(\+0\.011\) at lower precision \(−\-0\.052\)\.

The trend holds across the full model set \(Figure[3](https://arxiv.org/html/2608.17051#Sx11.F3)A–B; Tables S5 and S6; per\-category recall in Supplementary Figures 2 and 3\), with*Targeted*improving recall \(Opus 4\.8 \+0\.023, Kimi K2\.5 \+0\.018, Sonnet 4\.6 \+0\.014\) and*Precision*improving precision for all 7 models with valid output \(mean \+0\.054\)\. All three prompts carry worked examples; only those naming content to*preserve*recover the clinical text category\-only instructions over\-redact\[[18](https://arxiv.org/html/2608.17051#bib.bib43)\]\. Magnitude varies by model: those with severe baseline precision deficits benefit most \(GPT\-oss\-120B P=0\.198 to 0\.924; Kimi 0\.613 to 0\.816; DeepSeek 0\.698 to 0\.806\), whereas MiniMax regresses on recall under*Targeted*\(−\-0\.141\) and Sonnet 4\.6 scores its highest F1 under*Baseline*, whose high precision the instructions can only cost; the optimal prompt is therefore model\-dependent\. That F1 penalty must be read with caution: as the next section shows, many of*Precision*’s additional “false positives” are correct redactions of institutional PHI the original standard failed to annotate\.

### Error analysis reveals annotation gaps in the gold standard

While comparing multi\-agent pipelines against our best single\-pass model, we noticed they flagged institutional PHI absent from the gold standard\. Rather than count these as false positives, we asked whether the standard itself was incomplete\. Manual inspection confirmed annotators had systematically missed institution\-specific identifiers: operating room location codes \(CC OR, LT OR, MW OR, WC OR, WT MAIN OR, GIPS\), hospital abbreviations \(TCH, TXCH, BCM\), and campus or building names \(Mark Wallace Tower, West Campus\), all PHI under the catch\-all and expert\-determination provisions\. The scarcity of reliable annotations motivates label\-free evaluation\[[45](https://arxiv.org/html/2608.17051#bib.bib18)\]; our results show they are also*incomplete*\. Expert re\-annotation of the 10 highest\-discrepancy notes confirmed all 49 adjudicated instances as PHI and added 227 spans; the protocol and its biases are in Methods\.

Re\-evaluation against enhanced gold\.On the enhanced 10\-note subset \(1,758 spans, 5 trials per prompt; Figure[4](https://arxiv.org/html/2608.17051#Sx11.F4)C, Table S10\), annotation gaps disproportionately penalize institution\-aware prompts\.*Baseline*recall is 0\.847±\\pm0\.002 here versus 0\.958±\\pm0\.002 on the original 100\-note standard, because it leaves the newly\-annotated institutional terms unredacted\. Those two numbers differ in note set as well as standard, so we held the notes fixed: against the*original*annotation of these same 10 notes,*Baseline*recall is 0\.972±\\pm0\.003, so re\-annotation accounts for−\-0\.125 of the−\-0\.111 net change and note selection for\+\+0\.014 \(Supplementary Note 2\.6\)\.*Targeted*recovers recall to 0\.980±\\pm0\.000 at precision 0\.810±\\pm0\.004 \(F1=0\.887±\\pm0\.002\), and*Precision*holds recall \(0\.981±\\pm0\.000\) while recovering precision to 0\.844±\\pm0\.003 \(F1=0\.907±\\pm0\.002\) — a*Baseline*\-to\-*Targeted*recall gap far larger on the enhanced gold \(0\.133\) than on the original \(0\.017\)\.

### Multi\-agent architectures confirm single\-pass sufficiency

None of the 14 multi\-agent and ensemble configurations \(Supplementary Note 3, Tables S11 and S12\) improved F1 over the single\-pass*Precision*prompt\. The three reproducibility\-tested methods \(5 trials each, Table S13\) reached comparable mean F1 \(0\.906–0\.908\) with overlapping 95% trial\-resampled CIs \(Figure[4](https://arxiv.org/html/2608.17051#Sx11.F4)A–B\), though our trial count cannot establish formal equivalence\. The agentic pipeline’s intra\-method variability \(SDF1\{\}\_\{\\text\{F1\}\}=0\.018\) exceeds inter\-method differences, and 4 of its 5 trials fell below single\-pass \(median 0\.898 vs\. mean 0\.906, pulled up by one high\-precision run at F1=0\.942\), so a typical agentic run underperforms\. Ensemble voting reaches marginally higher recall \(Cross\-Model Vote 0\.986 vs\. 0\.981\) at a precision cost and 6×\\timesthe inference cost\. Across the four top LLMs, 20\.3% of false negatives are shared, dominated by ambiguous partial dates and institutional identifiers \(Supplementary Note 2\.4\)\.

## Discussion

Our results establish three findings\. First, LLMs substantially outperform purpose\-built systems, with the advantage concentrated in categories requiring contextual reasoning\. Second, in\-context learning enables control over the precision–recall trade\-off, and which prompt looks best depends on how completely the standard annotates institutional PHI:*Baseline*wins on the original annotations and loses on the corrected ones, not because*Precision*improves but because*Baseline*’s recall collapses once the institutional terms it leaves unredacted are counted\.

Third, no multi\-agent architecture improved F1 over single\-pass\. Those configurations proved useful instead as a*discovery tool*for enumerating annotation gaps at scale, though not uniquely so, since the single\-pass*Precision*prompt flagged the same identifiers\. The prover\-verifier framework\[[17](https://arxiv.org/html/2608.17051#bib.bib17)\]that motivated this exploration yields no gain here because the bottleneck is specification — which institutional terms to redact — not verification of a given redaction\. Recommended deployment configurations are given in Supplementary Note 3\.5\.

### Limitations

First, our corpus \(100 notes, single institution\) limits generalizability\. The 49\-note USDHUB validation set \(Supplementary Note 4\) confirms the LLM F1 advantage persists but narrows on canonical\-PHI\-dominated corpora; because USDHUB is from the same institution, it establishes robustness across specialties, note types, and annotation methods, but not across institutions\. Cross\-institutional generalization and the transferability of our site\-specific addenda remain to be established\. Second, the 10 re\-annotated notes were selected by highest model–gold discrepancy, biasing toward vindicating the model, so enhanced\-gold results are relative comparisons across prompts, not corpus\-wide estimates\. Third, TiDE was run unprovisioned on the primary corpus; on USDHUB both extremes were near\-identical \(recall 0\.976 vs\. 0\.971; Supplementary Note 4\.1\), so the LLM advantage holds whether or not pattern\-based systems receive site\-specific information\. Fourth, the*Targeted*and*Precision*addenda came from*Baseline*error analysis on this same corpus, so their gains are in\-sample\. Fifth, we report no subgroup or fairness analysis: the attributes that would define subgroups are themselves the PHI under removal and were never extracted, so whether recall differs across patient groups is unresolved\. Sixth, both GPT\-oss models showed failure modes unrelated to de\-identification capability \(GPT\-oss\-20B output truncation, depressing recall; GPT\-oss\-120B unstable redaction at precision 0\.198\), unlike the other six \(Table S5\)\.

Finally, that multi\-agent architectures fail to beat single\-pass likely reflects the task itself: de\-identification is a single\-read problem in which all necessary context is already in the note\. Additional passes let the model second\-guess correct decisions, and coordination adds noise without information \(Supplementary Note 3\.4\)\. This matches findings outside the clinical domain, where multi\-agent gains are often minimal and failures trace to specification rather than model capability\[[9](https://arxiv.org/html/2608.17051#bib.bib45)\], and where a single agent with strong prompts matches multi\-agent discussion, the latter winning only where the prompt leaves the task underspecified\[[43](https://arxiv.org/html/2608.17051#bib.bib46)\]; our*Targeted*and*Precision*prompts supply that specification\.

### Future directions

Promising directions include knowledge distillation, using high\-performing LLMs as teacher labelers for small, locally\-deployable models at 100×\\timeslower cost\[[16](https://arxiv.org/html/2608.17051#bib.bib34),[33](https://arxiv.org/html/2608.17051#bib.bib21),[47](https://arxiv.org/html/2608.17051#bib.bib44)\]; adversarial verification against an LLM attempting re\-identification\[[25](https://arxiv.org/html/2608.17051#bib.bib20),[7](https://arxiv.org/html/2608.17051#bib.bib30),[8](https://arxiv.org/html/2608.17051#bib.bib31)\]; multi\-institutional validation; and gold standard refinement to include institutionally situated PHI\.

## Conclusion

On 100 pediatric oncology notes, LLMs beat purpose\-built de\-identification on recall — its core promise — by 0\.20 over Stanford TiDE, and the gap is widest exactly where pattern matching cannot reach: identifiers whose PHI status depends on institutional context\.

Prompting, not retraining, is the adaptation mechanism\. Naming the institutional categories a HIPAA\-aligned prompt misses recovers 79% \(48/61\) of them, and anti\-over\-redaction instructions then restore precision, within a single LLM call and impossible with purpose\-built systems short of retraining\. Multi\-agent architectures do not help; none of 14 configurations improved F1 over a single pass with the same prompt, because de\-identification is a single\-read task in which the second pass has no information the first lacked\. Ensembles buy marginally higher recall at a precision cost and 3 calls per note\. And LLMs proved good enough to audit their own reference standard: what looked like over\-redaction was largely PHI the annotators had missed, expert adjudication confirming all 49 in\-context instances as true PHI\.

Three consequences follow\. De\-identification should be evaluated per category, not on aggregate metrics that hide institutional PHI; gold standards deserve auditing before they are trusted as ground truth; and effort belongs in institution\-specific prompts rather than additional passes\. LLMs cost more per note, but that cost buys adaptation without retraining and a check on the standard itself\. Our evaluation framework is released open\-source\.

## Funding

This research was supported by a fellowship from the Gulf Coast Consortia on the NLM Training Program in Biomedical Informatics and Data Science \(T15 LM007093\); the National Science Foundation Graduate Research Fellowship Program \(NSF GRFP Fellow ID 2024370642\); the Fund for Innovation in Cancer Informatics; the Cancer Prevention and Research Institute of Texas \(CPRIT, RP240131\); the Chan Zuckerberg Initiative \(2023\-332162\); the National Institutes of Health \(NIH, U54NS093793 and OT2OD040565\); the Eunice Kennedy Shriver National Institute of Child Health and Human Development of the NIH \(P50HD103555\); the Chao Endowment; the Huffington Foundation; and the Jan and Dan Duncan Neurological Research Institute at Texas Children’s Hospital\.

## Additional Contributions

We thank the Texas Children’s Hospital Office of Research Data, which provided the independently de\-identified USDHUB note set used for our within\-institution validation \(Supplementary Note 4\)\. No AI\-assisted tools were used for study design, analysis, or primary manuscript drafting\. Generative AI tools were used only for proofreading and typographical/grammatical correction of author\-written text; they were not used to generate scientific content, analyze data, or draft substantive passages of the manuscript\. In accordance with COPE’s position and JAMIA policy, no AI or NLP tool is listed as an author; the authors reviewed and verified all text and take full responsibility for the integrity, accuracy, and originality of all content\.

## Conflicts of Interest

The authors declare no competing interests\.

## Data Availability

The evaluation pipeline and code are openly available at[https://github\.com/LiuzLab/phi\-scrubber\-evaluation](https://github.com/LiuzLab/phi-scrubber-evaluation)\. The underlying clinical notes cannot be shared because they are protected patient health information governed by IRB protocol H\-52222 and institutional/HIPAA data\-use restrictions; they are not available for public deposition or on request\. De\-identified aggregate metrics that support the findings of this study are provided within the article and its supplementary materials\.

## Author Contributions

M\.B\.N\. and D\.P\. contributed equally as co\-first authors: conceptualization, methodology, software, formal analysis, investigation, visualization, and writing\. J\.P\.W\., C\.L\., A\.A\.O\., and S\.D\. contributed to data curation and investigation \(PHI annotation and re\-annotation\)\. M\.Z\. provided clinical expertise, adjudication of institutional PHI, annotation, and review and editing of the writing\. Z\.L\. is the principal investigator and contributed supervision and funding acquisition\. H\.H\.J\. contributed supervision, project administration, and review and editing of the writing\. All authors reviewed and approved the final manuscript\.

## References

- \[1\]\(2025\)Not what the doctor ordered: surveying LLM\-based de\-identification and quantifying clinical information loss\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,Suzhou, China,pp\. 32187–32203\.Note:arXiv:2509\.14464Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p5.1)\.
- \[2\]E\. Alsentzer, J\. R\. Murphy, W\. Boag, W\. Weng, D\. Jindi, T\. Naumann, and M\. B\. McDermott\(2019\)Publicly available clinical BERT embeddings\.Proceedings of the 2nd Clinical Natural Language Processing Workshop,pp\. 72–78\.Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p3.1)\.
- \[3\]B\. Altalla’, S\. Abdalla, A\. Altamimi, L\. Bitar, A\. Al Omari, R\. Kardan, and I\. Sultan\(2025\)Evaluating GPT models for clinical note de\-identification\.Scientific Reports15\(1\),pp\. 3852\.External Links:[Document](https://dx.doi.org/10.1038/s41598-025-86890-3)Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p4.1)\.
- \[4\]J\. Botas\(2025\)PubliPlots: publication\-ready plotting for python\.Note:[https://github\.com/jorgebotas/publiplots](https://github.com/jorgebotas/publiplots)Cited by:[Evaluation pipeline](https://arxiv.org/html/2608.17051#Sx2.SSx6.p1.1)\.
- \[5\]M\. Buckland and F\. Gey\(1994\)The relationship between recall and precision\.Journal of the American Society for Information Science45\(1\),pp\. 12–19\.Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p5.1)\.
- \[6\]A\. Callahan, E\. Ashley, S\. Datta, P\. Desai, T\. A\. Ferris, J\. A\. Fries, M\. Halaas, C\. P\. Langlotz, S\. Mackey, J\. D\. Posada, M\. A\. Pfeffer, and N\. H\. Shah\(2023\)The stanford medicine data science ecosystem for clinical and translational research\.JAMIA Open6\(3\),pp\. ooad054\.External Links:[Document](https://dx.doi.org/10.1093/jamiaopen/ooad054)Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p3.1)\.
- \[7\]D\. Carrell, B\. Malin, J\. Aberdeen, S\. Bayer, C\. Clark, B\. Wellner, and L\. Hirschman\(2013\)Hiding in plain sight: use of realistic surrogates to reduce exposure of protected health information in clinical text\.Journal of the American Medical Informatics Association20\(2\),pp\. 342–348\.External Links:[Document](https://dx.doi.org/10.1136/amiajnl-2012-001034)Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p3.1),[Future directions](https://arxiv.org/html/2608.17051#Sx4.SSx2.p1.1)\.
- \[8\]D\. S\. Carrell, B\. A\. Malin, D\. J\. Cronkite, J\. S\. Aberdeen, C\. Clark, M\. Li, D\. Bastakoty, S\. Nyemba, and L\. Hirschman\(2020\)Resilience of clinical text de\-identified with “hiding in plain sight” to hostile reidentification attacks by human readers\.Journal of the American Medical Informatics Association27\(9\),pp\. 1374–1382\.External Links:[Document](https://dx.doi.org/10.1093/jamia/ocaa095)Cited by:[Future directions](https://arxiv.org/html/2608.17051#Sx4.SSx2.p1.1)\.
- \[9\]M\. Cemri, M\. Z\. Pan, S\. Yang, L\. A\. Agrawal, B\. Chopra, R\. Tiwari, K\. Keutzer, A\. Parameswaran, D\. Klein, K\. Ramchandran, M\. Zaharia, J\. E\. Gonzalez, and I\. Stoica\(2025\)Why do multi\-agent LLM systems fail?\.External Links:2503\.13657,[Link](https://arxiv.org/abs/2503.13657)Cited by:[Limitations](https://arxiv.org/html/2608.17051#Sx4.SSx1.p2.1)\.
- \[10\]S\. Datta, J\. Posada, G\. Olson, W\. Li, C\. O’Reilly, D\. Balraj, J\. Mesterhazy, J\. Pallas, P\. Desai, and N\. Shah\(2020\)A new paradigm for accelerating clinical data science at Stanford Medicine\.Note:arXiv:2003\.10534External Links:2003\.10534,[Link](https://arxiv.org/abs/2003.10534)Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p3.1),[De\-identification systems](https://arxiv.org/html/2608.17051#Sx2.SSx2.p2.1)\.
- \[11\]F\. Dernoncourt, J\. Y\. Lee, O\. Uzuner, and P\. Szolovits\(2017\)De\-identification of patient notes with recurrent neural networks\.Journal of the American Medical Informatics Association24\(3\),pp\. 596–606\.External Links:[Document](https://dx.doi.org/10.1093/jamia/ocw156)Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p3.1)\.
- \[12\]Ó\. Ferrández, B\. R\. South, S\. Shen, F\. J\. Friedlin, M\. H\. Samore, and S\. M\. Meystre\(2012\)Evaluating current automatic de\-identification methods with Veteran’s health administration clinical documents\.BMC Medical Research Methodology12,pp\. 109\.External Links:[Document](https://dx.doi.org/10.1186/1471-2288-12-109)Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p2.1)\.
- \[13\]Ó\. Ferrández, B\. R\. South, S\. Shen, F\. J\. Friedlin, M\. H\. Samore, and S\. M\. Meystre\(2013\)BoB, a best\-of\-breed automated text de\-identification system for VHA clinical documents\.Journal of the American Medical Informatics Association20\(1\),pp\. 77–83\.External Links:[Document](https://dx.doi.org/10.1136/amiajnl-2012-001020)Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p5.1)\.
- \[14\]J\. Gallifant, M\. Afshar, S\. Ameen, Y\. Aphinyanaphongs, S\. Chen, G\. Cacciamani, D\. Demner\-Fushman, D\. Dligach, R\. Daneshjou, C\. Fernandes, L\. H\. Hansen, A\. Landman, L\. Lehmann, L\. G\. McCoy, T\. Miller, A\. Moreno, N\. Munch, D\. Restrepo, G\. Savova, R\. Umeton, J\. W\. Gichoya, G\. S\. Collins, K\. G\. M\. Moons, L\. A\. Celi, and D\. S\. Bitterman\(2025\)The TRIPOD\-LLM reporting guideline for studies using large language models\.Nature Medicine31\(1\),pp\. 60–69\.External Links:[Document](https://dx.doi.org/10.1038/s41591-024-03425-5)Cited by:[Study design and clinical corpus](https://arxiv.org/html/2608.17051#Sx2.SSx1.p1.1)\.
- \[15\]M\. Gunay, B\. Keles, and R\. Hizlan\(2024\)LLMs\-in\-the\-Loop part 2: expert small AI models for de\-identification across 8 languages\.arXiv preprint arXiv:2412\.10918\.Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p4.1)\.
- \[16\]W\. Kim, S\. Hahm, and J\. Lee\(2024\)Generalizing clinical de\-identification models by privacy\-safe data augmentation using GPT\-4\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p4.1),[Future directions](https://arxiv.org/html/2608.17051#Sx4.SSx2.p1.1)\.
- \[17\]J\. H\. Kirchner Y\. Chenet al\.\(2024\)Prover\-verifier games improve legibility of LLM outputs\.arXiv preprint arXiv:2407\.13692\.Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p4.1),[Introduction](https://arxiv.org/html/2608.17051#Sx1.p6.1),[Discussion](https://arxiv.org/html/2608.17051#Sx4.p2.1)\.
- \[18\]R\. Kuo, A\. A\. S\. Soltan, C\. O’Hanlon, A\. Hasanic, D\. A\. Clifton, G\. Collins, D\. Furniss, and D\. W\. Eyre\(2025\)Benchmarking transformer\-based models for medical record de\-identification in a single center multi\-specialty evaluation\.iScience28\(12\),pp\. 113732\.External Links:[Document](https://dx.doi.org/10.1016/j.isci.2025.113732)Cited by:[In\-context learning enables control over the precision–recall trade\-off](https://arxiv.org/html/2608.17051#Sx3.SSx3.p3.1)\.
- \[19\]J\. Lee, W\. Yoon, S\. Kim, D\. Kim, S\. Kim, C\. H\. So, and J\. Kang\(2020\)BioBERT: a pre\-trained biomedical language representation model for biomedical text mining\.Bioinformatics36\(4\),pp\. 1234–1240\.External Links:[Document](https://dx.doi.org/10.1093/bioinformatics/btz682)Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p3.1)\.
- \[20\]Z\. Liu, B\. Tang, X\. Wang, and Q\. Chen\(2017\)De\-identification of clinical notes via recurrent neural network and conditional random field\.Journal of Biomedical Informatics75,pp\. S34–S42\.External Links:[Document](https://dx.doi.org/10.1016/j.jbi.2017.05.023)Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p3.1)\.
- \[21\]Z\. Liu, Y\. Huang, X\. Yu, L\. Zhang, Z\. Wu, C\. Cao, H\. Dai, L\. Zhao, Y\. Li, P\. Shu, F\. Zeng, L\. Sun, W\. Liu, D\. Shen, Q\. Li, T\. Liu, D\. Zhu, and X\. Li\(2023\)DeID\-GPT: zero\-shot medical text de\-identification by GPT\-4\.arXiv preprint arXiv:2303\.11032\.Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p4.1)\.
- \[22\]G\. Loukides, J\. C\. Denny, and B\. Malin\(2010\)The disclosure of diagnosis codes can breach research participants’ privacy\.Journal of the American Medical Informatics Association17\(3\),pp\. 322–327\.External Links:[Document](https://dx.doi.org/10.1136/jamia.2009.002725)Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p1.1)\.
- \[23\]B\. Malin and L\. Sweeney\(2004\)How \(not\) to protect genomic data privacy in a distributed network: using trail re\-identification to evaluate and design anonymity protection systems\.Journal of Biomedical Informatics37\(3\),pp\. 179–192\.External Links:[Document](https://dx.doi.org/10.1016/j.jbi.2004.04.005)Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p1.1)\.
- \[24\]S\. M\. Meystre, Ó\. Ferrández, F\. J\. Friedlin, B\. R\. South, S\. Shen, and M\. H\. Samore\(2014\)Text de\-identification for privacy protection: a study of its impact on clinical text information content\.Journal of Biomedical Informatics50,pp\. 142–150\.External Links:[Document](https://dx.doi.org/10.1016/j.jbi.2014.01.011)Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p2.1)\.
- \[25\]J\. X\. Morris, T\. R\. Campion, S\. L\. Nutheti, Y\. Peng, A\. Raj, R\. Zabih, and C\. L\. Cole\(2024\)DIRI: adversarial patient re\-identification with large language models for evaluating de\-identification\.Proceedings of AMIA Annual Symposium\.Note:arXiv:2410\.17035Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p5.1),[Future directions](https://arxiv.org/html/2608.17051#Sx4.SSx2.p1.1)\.
- \[26\]I\. Neamatullah, M\. M\. Douglass, L\. H\. Lehman, A\. Reisner, M\. Villarroel, W\. J\. Long, P\. Szolovits, G\. B\. Moody, R\. G\. Mark, and G\. D\. Clifford\(2008\)Automated de\-identification of free\-text medical records\.BMC Medical Informatics and Decision Making8\(1\),pp\. 32\.External Links:[Document](https://dx.doi.org/10.1186/1472-6947-8-32)Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p3.1)\.
- \[27\]B\. Norgeot, K\. Muenzen, T\. A\. Peterson, X\. Fan, B\. S\. Glicksberg, G\. Schenk, E\. Rutenberg, B\. Oskotsky, M\. Sirota, J\. Yazdany, G\. Schmajuk, D\. Ludwig, T\. Goldstein, and A\. J\. Butte\(2020\)Protected health information filter \(Philter\): accurately and securely de\-identifying free\-text clinical notes\.npj Digital Medicine3,pp\. 57\.External Links:[Document](https://dx.doi.org/10.1038/s41746-020-0258-y)Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p2.1)\.
- \[28\]Office for Civil Rights, HHS\(2012\)Guidance regarding methods for de\-identification of protected health information\.External Links:[Link](https://www.hhs.gov/hipaa/for-professionals/privacy/special-topics/de-identification/)Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p1.1),[Introduction](https://arxiv.org/html/2608.17051#Sx1.p2.1)\.
- \[29\]OpenMed Science\(2026\)OpenMed\-PII\-SuperClinical\-Large\-434M\-v1: PII detection model\.Note:Hugging Face model repository[https://huggingface\.co/OpenMed/OpenMed\-PII\-SuperClinical\-Large\-434M\-v1](https://huggingface.co/OpenMed/OpenMed-PII-SuperClinical-Large-434M-v1)Cited by:[De\-identification systems](https://arxiv.org/html/2608.17051#Sx2.SSx2.p3.1)\.
- \[30\]M\. Panahi\(2025\)OpenMed NER: open\-source, domain\-adapted state\-of\-the\-art transformers for biomedical NER across 12 public datasets\.Note:arXiv:2508\.01630External Links:2508\.01630,[Link](https://arxiv.org/abs/2508.01630)Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p3.1),[De\-identification systems](https://arxiv.org/html/2608.17051#Sx2.SSx2.p3.1)\.
- \[31\]A\. Paul, D\. Shaji, L\. Han, W\. Del\-Pinto, G\. Nenadic, and S\. Verberne\(2026\)DeIDClinic: a risk\-aware pseudonymization framework for clinical text de\-identification and re\-identification risk assessment\.Note:arXiv:2410\.01648External Links:2410\.01648,[Link](https://arxiv.org/abs/2410.01648)Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p3.1)\.
- \[32\]D\. Pissarra, I\. Curioso, J\. Alveira, D\. Pereira, B\. Ribeiro, T\. Souper, V\. Gomes, A\. Carreiro, and V\. Rolla\(2024\)Unlocking the potential of large language models for clinical text anonymization: a comparative study\.InProceedings of the Fifth Workshop on Privacy in Natural Language Processing \(PrivateNLP\),Note:arXiv:2406\.00062Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p4.1)\.
- \[33\]J\. D\. Posada, D\. Love, S\. Datta, and P\. Desai\(2026\)SHIELD: synthetic human\-annotated identifier\-replaced entries for learning and de\-identification\.arXiv preprint arXiv:2605\.03301\.Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p3.1),[Introduction](https://arxiv.org/html/2608.17051#Sx1.p4.1),[Future directions](https://arxiv.org/html/2608.17051#Sx4.SSx2.p1.1)\.
- \[34\]T\. Sounack, J\. Davis, B\. Durieux, A\. Chaffin, T\. J\. Pollard, E\. Lehman, A\. E\. W\. Johnson, M\. McDermott, T\. Naumann, and C\. Lindvall\(2025\)BioClinical ModernBERT: a state\-of\-the\-art long\-context encoder for biomedical and clinical NLP\.Note:arXiv:2506\.10896External Links:2506\.10896,[Link](https://arxiv.org/abs/2506.10896)Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p4.1)\.
- \[35\]A\. Stubbs, C\. Kotfila, and Ö\. Uzuner\(2015\)Annotating longitudinal clinical narratives for de\-identification: the 2014 i2b2/UTHealth corpus\.Journal of Biomedical Informatics58,pp\. S20–S29\.External Links:[Document](https://dx.doi.org/10.1016/j.jbi.2015.07.020)Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p3.1),[Introduction](https://arxiv.org/html/2608.17051#Sx1.p5.1)\.
- \[36\]L\. Sweeney\(1996\)Replacing personally\-identifying information in medical records, the Scrub system\.Proceedings of the AMIA Annual Fall Symposium,pp\. 333–337\.Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p3.1)\.
- \[37\]L\. Sweeney\(2000\)Simple demographics often identify people uniquely\.Data Privacy Working PaperTechnical Report3,Carnegie Mellon University,Pittsburgh\.External Links:[Document](https://dx.doi.org/10.1184/R1/6625769.v1),[Link](http://dataprivacylab.org/projects/identifiability/)Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p1.1)\.
- \[38\]L\. Sweeney\(2013\)Matching known patients to health records in Washington State data\.Note:arXiv:1307\.1370External Links:1307\.1370,[Link](https://arxiv.org/abs/1307.1370),[Document](https://dx.doi.org/10.48550/arXiv.1307.1370)Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p1.1)\.
- \[39\]X\. Tao, X\. Dong, and X\. Zhou\(2026\)OEMA: ontology\-enhanced multi\-agent collaboration framework for zero\-shot clinical named entity recognition\.JAMIA Open\.Note:arXiv:2511\.15211Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p4.1)\.
- \[40\]U\.S\. Congress\(1996\)Health insurance portability and accountability act of 1996\.Note:Public Law 104\-191Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p1.1)\.
- \[41\]U\.S\. Department of Health and Human Services\(2024\)Other requirements relating to uses and disclosures of protected health information\.Note:45 C\.F\.R\. §164\.514; see §164\.514\(b\)\(1\) \(expert determination\) and §164\.514\(b\)\(2\)\(i\)\(R\) \(“any other unique identifying number, characteristic, or code”\)\. Accessed August 2026External Links:[Link](https://www.ecfr.gov/current/title-45/section-164.514)Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p1.1)\.
- \[42\]Ö\. Uzuner, Y\. Luo, and P\. Szolovits\(2007\)Evaluating the state\-of\-the\-art in automatic de\-identification\.Journal of the American Medical Informatics Association14\(5\),pp\. 550–563\.External Links:[Document](https://dx.doi.org/10.1197/jamia.M2444)Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p3.1)\.
- \[43\]Q\. Wang, Z\. Wang, Y\. Su, H\. Tong, and Y\. Song\(2024\)Rethinking the bounds of LLM reasoning: are multi\-agent discussions the key?\.External Links:2402\.18272,[Link](https://arxiv.org/abs/2402.18272)Cited by:[Limitations](https://arxiv.org/html/2608.17051#Sx4.SSx1.p2.1)\.
- \[44\]I\. C\. Wiest, M\. Leßmann, F\. Wolf, D\. Ferber, M\. Van Treeck, J\. Zhu, M\. P\. Ebert, C\. B\. Westphalen, M\. Wermke, and J\. N\. Kather\(2025\)Deidentifying medical documents with local, privacy\-preserving large language models: the LLM\-anonymizer\.NEJM AI2\(4\),pp\. AIdbp2400537\.External Links:[Document](https://dx.doi.org/10.1056/AIdbp2400537)Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p4.1)\.
- \[45\]G\. Wu, Z\. Chen, Y\. Xie, and C\. Yang\(2025\)Towards automatic evaluation and selection of PHI de\-identification models via multi\-agent collaboration\.Note:Agents4Science 2025 SpotlightExternal Links:2510\.16194Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p4.1),[Introduction](https://arxiv.org/html/2608.17051#Sx1.p6.1),[Error analysis reveals annotation gaps in the gold standard](https://arxiv.org/html/2608.17051#Sx3.SSx4.p1.1)\.
- \[46\]X\. Yang, T\. Lyu, Q\. Li, C\. Lee, J\. Bian, W\. R\. Hogan, and Y\. Wu\(2019\)A study of deep learning methods for de\-identification of clinical notes in cross\-institute settings\.BMC Medical Informatics and Decision Making19\(Suppl 5\),pp\. 232\.External Links:[Document](https://dx.doi.org/10.1186/s12911-019-0935-4)Cited by:[Introduction](https://arxiv.org/html/2608.17051#Sx1.p2.1)\.
- \[47\]N\. Zambare, K\. Aghakasiri, C\. Lin, C\. Ye, J\. R\. Mitchell, and M\. Abdalla\(2026\)Towards fair and efficient de\-identification: quantifying the efficiency and generalizability of de\-identification approaches\.InFindings of the Association for Computational Linguistics: EACL 2026,Note:arXiv:2602\.15869Cited by:[Future directions](https://arxiv.org/html/2608.17051#Sx4.SSx2.p1.1)\.

## Figures

![Refer to caption](https://arxiv.org/html/2608.17051v1/figure1_composite.png)Figure 1:Study design and corpus characteristics\.\(A\)Study design: 8 LLMs \(top 4 shown in main figures\) \+ traditional baselines evaluated on 100 pediatric oncology notes \(5,322 spans\) under 3 prompt conditions\.\(B\)Synthetic clinical note illustrating canonical HIPAA PHI \(names, dates; blue\) vs\. institutionally situated PHI \(facility abbreviations, building names; orange\)\.\(C\)Gold standard PHI distribution over the six categories withN≥20N\\geq 20\(four further categories hold≤5\\leq 5spans each; all ten, with their denominators, are in Supplementary Table S4\): Date \(72\.1%\) and Name \(22\.3%\) dominate, while institutionally enriched categories \(Other Unique ID, Geographic Data\) comprise 3\.7% of spans\. This distribution is that of the*original*gold standard; because that annotation under\-counted institutional identifiers \(see Results\), 3\.7% is a lower bound on their true prevalence\.Alt text:Three\-panel study\-design figure\. Panel A is a schematic showing eight large language models and traditional baselines being evaluated on 100 pediatric oncology notes containing 5,322 PHI spans under three prompt conditions\. Panel B shows a synthetic clinical note with canonical HIPAA identifiers highlighted in blue and institutionally situated identifiers \(facility abbreviations, building names\) highlighted in orange\. Panel C is a bar or pie chart of the original gold\-standard PHI category distribution over the six well\-populated categories, dominated by Date \(72\.1 percent\) and Name \(22\.3 percent\), with institutionally enriched categories comprising 3\.7 percent of spans, a lower bound given the under\-annotation described in Results\.

![Refer to caption](https://arxiv.org/html/2608.17051v1/figure2_composite.png)Figure 2:LLMs outperform traditional de\-identification approaches\.\(A\)Recall, precision, and F1 for the top 4 LLMs and 4 traditional baselines \(*Baseline*prompt, 100 notes, 5,322 spans; all 8 LLMs in Table S3 and Supplementary Figure 1\)\.\(B\)Per\-category recall heatmap for the same systems, with categories ordered by gold\-standard frequency and systems by overall F1, reveals shared weakness on the institutionally enriched categories \(Other Unique ID \[OUI\], Geographic Data\); the four categories with≤\\leq5 gold spans are omitted here and reported in Table S4\.\(C\)Sonnet 4\.6 false negative decomposition on the primary*Baseline*run \(Table S3; 211 FN\): 29% are institutionally situated PHI \(61 of 211; institution abbreviations 38, building/facility 15, department/clinic codes 8; see taxonomy in Table S7\); the remaining 150 are canonical, of which the internal staff codes \(pager and Voalte extensions, 4\) are shown separately because Table S7 lists them as borderline\.Alt text:Three\-panel performance\-comparison figure\. Panel A is a grouped bar chart of recall, precision, and F1 for the top four LLMs and four traditional baselines under the Baseline prompt, showing LLMs above traditional systems on F1\. Panel B is a per\-category recall heatmap in which the Other Unique ID and Geographic Data columns are lightest \(lowest recall\) across systems, indicating shared weakness on institutionally situated PHI\. Panel C is a horizontal bar chart breaking down Sonnet 4\.6’s 211 false negatives, with 29 percent \(61 spans\) attributed to institutionally situated PHI and the remaining 150 to canonical categories\.

Figure 3:In\-context learning enables control over the precision–recall trade\-off\.\(A\)Precision, recall, and F1 for the top 4 LLMs across the three prompt versions \(*Baseline*∙\\bullet,*Targeted*■\\blacksquare,*Precision*▲\\blacktriangle; single run per model, all 8 in Table S5\)\. Each horizontal line spans the minimum\-to\-maximum of a model’s three stage values, so a non\-monotonic*Targeted*stage visibly stretches the line rather than being hidden;*Baseline*and*Precision*values are labeled\. Effect magnitude and direction are model\-dependent\.\(B\)Problem\-specific deltas \(single run\), same model order and colors as \(A\), isolating what each prompt revision buys:Δ\\DeltaRecall\(*Targeted*−\-*Baseline*\) is the recall gain from institutional targeting, andΔ\\DeltaPrecision\(*Precision*−\-*Targeted*\) is the precision recovery from the anti\-over\-redaction instructions\. The*Precision*prompt lifts precision for*all*models \(cross\-model mean over the 7 models with valid output: \+0\.054\) and acts as a corrective: gains are largest for models that over\-redact under the baseline prompt \(GPT\-oss\-120B0\.198→0\.9240\.198\\rightarrow 0\.924, Kimi0\.613→0\.8160\.613\\rightarrow 0\.816, DeepSeek0\.698→0\.8060\.698\\rightarrow 0\.806across the full*Baseline*→\\rightarrow*Precision*progression; full values in Tables S5 and S6\) and near\-zero for models whose native precision is already high\. Sonnet 4\.6 reproducibility \(5 trials per stage on the full 100\-note corpus; SD≤\\leq0\.007 at every stage\) confirms these are systematic prompt effects, not run\-to\-run noise \(Table S9\)\.Alt text:Two\-panel figure on prompt effects\. Panel A plots precision, recall, and F1 for the top four LLMs across three prompt versions \(Baseline, Targeted, Precision\) as horizontal min\-to\-max ranges per model, showing model\-dependent magnitude and direction\. Panel B is a bar chart of problem\-specific deltas: the recall gain from institutional targeting and the precision recovery from anti\-over\-redaction instructions, with the largest precision gains for models that over\-redact under the baseline prompt and near\-zero gains for models with already\-high native precision\.

Figure 4:Enhanced gold standard validation and agentic architecture comparison\.\(A\)Configuration landscape: F1 for the 16 configurations that the panel displays, colour\-coded by class \(single\-pass, agentic, ensemble, traditional\) and ranked on the enhanced gold standard; 10 further configurations are omitted for legibility and appear in Table S12, which lists all 26\. The omitted set is the weakest\-performing tail, except that TiDE is retained as the traditional\-system reference\. Single\-pass Sonnet \(*Precision*\) \(dashed line\) matches or exceeds every multi\-agent variant, and self\-refinement and ensemble voting fail to beat it\.\(B\)Reproducibility: 5\-trial strip plots for the top 3 methods\. All three achieve closely comparable mean F1 \(0\.906–0\.908\) with overlapping 95% trial\-resampled CIs, but the Scrubber–Auditor \(*Precision*\) pipeline on Opus shows high variance \(SDF1\{\}\_\{\\text\{F1\}\}=0\.018\) vs\. near\-deterministic single\-pass Sonnet \(SDF1\{\}\_\{\\text\{F1\}\}=0\.002\)\.\(C\)Sonnet 4\.6 prompt progression on the enhanced gold standard \(10\-note subset, 1,758 spans; 5 independent trials per version\)\. Lines show mean; shading shows±\\pmSD; points show individual trials\.*Baseline*recall is 0\.847 here versus 0\.958 on the original 100\-note gold standard \(a different note set*and*a different gold standard\) because it misses newly\-annotated institutional terms;*Targeted*recovers recall to 0\.980;*Precision*maintains recall while improving precision \(F1=0\.907±\\pm0\.002\)\.Terminology:*single\-pass*= one LLM call per note;*dual\-pass*= two sequential LLM calls;*structured pass 2*= a second pass that ingests the first pass’s output in a structured format;*Scrubber–Auditor*= an agentic pipeline in which a generator LLM \(scrubber\) is checked by a second auditor LLM\.*Baseline*,*Targeted*, and*Precision*denote the three prompt versions\.Alt text:Three\-panel figure on agentic architectures\. Panel A ranks 16 configurations, spanning single\-pass, agentic, ensemble, and traditional approaches, by F1 on the enhanced gold standard, with a dashed line marking single\-pass Sonnet \(Precision\) at or above all multi\-agent variants\. Panel B shows 5\-trial strip plots for the top three methods with closely overlapping mean F1 of 0\.906 to 0\.908, but visibly wider scatter for the Opus Scrubber–Auditor pipeline than for near\-deterministic single\-pass Sonnet\. Panel C plots Sonnet 4\.6 recall, precision, and F1 across the Baseline, Targeted, and Precision prompts on the enhanced gold standard, showing Baseline recall dropping to 0\.847, Targeted recovering recall to 0\.980, and Precision maintaining recall while improving precision to F1 = 0\.907\.