Surrogate Substitution Preserves PHI Detectability: A Multi-Detector Equivalence Study

# Surrogate Substitution Preserves PHI Detectability: A Multi-Detector Equivalence Study
Source: [https://arxiv.org/html/2608.03172](https://arxiv.org/html/2608.03172)
###### Abstract

Structure\-preserving de\-identification replaces protected health information \(PHI\) with realistic same\-type*surrogates*—“Anna S\.” becomes “Maria S\.”, not\[NAME\]—so that clinical text stays fluent and downstream tools keep working\. But this only helps if the substitution does not itself corrupt the signal those tools rely on\. We ask a narrow, testable question:*on the spans a de\-identifier actually masks, can downstream PHI detectors still find the surrogate?*We introduce a paired, multi\-detector evaluation protocol that \(i\) scores utility*only on masked spans*, decoupling*coverage*from*utility*; \(ii\) uses*equivalence testing*\(TOST\) rather than null\-hypothesis significance testing, which is uninformative at our sample size \(57k paired spans\); and \(iii\) builds a surrogate\-failure typology separating fixable generator defects from intrinsic detector limits\. Across 11 detectors, 7 benchmarks, and 7 languages \(1,750 documents\), recall on masked spans moves from 76\.1% to 74\.9%—a change our equivalence test shows is*statistically equivalent to zero within a±2\\pm 2\-point margin*\(p≈3×10−9p\\approx 3\\times 10^\{\-9\}\), with detector ranking preserved\. The residual loss does not reflect detectors getting worse at PHI: it concentrates in*malformed and out\-of\-distribution*surrogates \(truncationChicago→\\rightarrowIllino, salience lossCedars\-Sinai→\\rightarrowVidant\)\. A redaction floor and an open\-source surrogate baseline indicate the effect is a property of well\-formed substitution, not of one tool\. We release the evaluation subsets, scoring code, and an interactive dashboard at[https://custodianai\.pages\.dev](https://custodianai.pages.dev/)so the protocol can audit any structure\-preserving transform\.111Code:[https://github\.com/Custodian\-Labs/guardian\-layer\-phi\-benchmark](https://github.com/Custodian-Labs/guardian-layer-phi-benchmark)

## 1Introduction

De\-identification of clinical text takes two forms—and only one keeps the text usable downstream\.Redactiondeletes or tags PHI \(John Smith→\\rightarrow\[NAME\]\), which is safe but destroys the fluency, layout, and distributional properties that downstream models and human readers depend on\.Structure\-preservingde\-identification instead substitutes each PHI value with a plausible, same\-type*surrogate*\(John Smith→\\rightarrowMaria Lopez\), keeping the document readable and machine\-parseable\[[6](https://arxiv.org/html/2608.03172#bib.bib14)\]\. The second family is increasingly attractive: it lets de\-identified data flow into analytics, model training, and even second\-pass detection without breaking pipelines built for real text\.

The promise of structure preservation rests on an unstated assumption:*the surrogate carries the same downstream signal as the value it replaced\.*If substitution silently degrades the very features a PHI detector, an NER model, or a clinical parser relies on, then “structure\-preserving” is a misnomer—the transform would be quietly laundering PHI into forms tools can no longer see or handle, which is both a utility problem and, for a second\-pass safety net, a privacy problem\.

This assumption is rarely tested directly, and testing it well is harder than it looks\. Three pitfalls recur:

1. 1\.Coverage confounds utility\.A de\-identifier that masks 60% of PHI and one that masks 95% cannot be compared on whole\-document F1—the score conflates*how much*it masks with*whether what it masks stays usable*\. The two must be measured separately\.
2. 2\.The large\-NNsignificance trap\.With tens of thousands of paired spans, any non\-zero difference is “statistically significant” under a standard test, even when operationally meaningless\. App\-value here answers the wrong question\.
3. 3\.Aggregate error hides mechanism\.A single “−1\.2\-1\.2points” says nothing about*why*the loss happens—boundary jitter, detector weakness, or a defect in surrogate generation\. Only the last is fixable, and only an error typology tells them apart\.

We address all three\. Our contributions:

- •Apaired multi\-detector protocolscoring utility on masked spans only, decoupling coverage from utility, across 11 detectors×\\times7 benchmarks×\\times7 languages \(§[4](https://arxiv.org/html/2608.03172#S4)–§[5](https://arxiv.org/html/2608.03172#S5)\)\.
- •Anequivalence\-testing analysis\(TOST\) replacing the uninformative significance test with a bounded\-effect claim: the change in detectability is bounded within±2\\pm 2points of zero \(§[6](https://arxiv.org/html/2608.03172#S6)\)\.
- •Asurrogate\-failure typologyattributing the small residual to surrogate\-generation quality rather than detectors getting worse at PHI \(§[7](https://arxiv.org/html/2608.03172#S7)\)\.
- •A set ofcomparison experiments—a redaction upper\-bound and an open\-source surrogate baseline—that make the result reproducible and isolate what is generic to*any*substitution versus specific to one tool \(§[5](https://arxiv.org/html/2608.03172#S5), §[6](https://arxiv.org/html/2608.03172#S6)\)\.

The framing is deliberately not “a proprietary tool is good\.” It is:*here is how to measure whether a structure\-preserving transform preserves utility, rigorously*, with one commercial transform as the case study and open baselines for reproducibility\.

## 2Related Work

### 2\.1De\-identification and surrogate substitution

Clinical de\-identification is a long\-studied sequence\-labeling problem\[[34](https://arxiv.org/html/2608.03172#bib.bib8)\], anchored by the i2b2/n2c2 and MEDDOCAN shared tasks\[[32](https://arxiv.org/html/2608.03172#bib.bib9),[26](https://arxiv.org/html/2608.03172#bib.bib10)\]and by systems ranging from rule\- and dictionary\-based pipelines\[[28](https://arxiv.org/html/2608.03172#bib.bib20)\]through recurrent and transformer taggers\[[10](https://arxiv.org/html/2608.03172#bib.bib11),[23](https://arxiv.org/html/2608.03172#bib.bib12),[19](https://arxiv.org/html/2608.03172#bib.bib13)\]to instruction\-tuned LLMs used zero\-shot\[[24](https://arxiv.org/html/2608.03172#bib.bib16)\]\. Most of this literature optimizes*detection*, typically against the HIPAA Safe Harbor identifier set\[[33](https://arxiv.org/html/2608.03172#bib.bib17)\]\. The downstream question—*what to put in the PHI’s place*—is comparatively under\-studied\. Surrogate \(“hiding in plain sight”\) replacement was proposed to keep de\-identified notes realistic and to resist re\-identification\[[6](https://arxiv.org/html/2608.03172#bib.bib14)\]; subsequent work showed poor surrogates can even aid re\-identification \(the “parrot” attack\), underscoring that surrogate*quality*matters\[[7](https://arxiv.org/html/2608.03172#bib.bib15)\]\. Our work is orthogonal to the detection literature and to any particular surrogate generator: given that a span is masked, we ask whether the*replacement*remains detectable and well\-formed\.

The commercial transform used as our case study is the Custodian Labs Guardian Layer\[[8](https://arxiv.org/html/2608.03172#bib.bib35)\]; we cite it as the source of the transform rather than describing or extending the method \(a formal method citation will be added once available\)\.

### 2\.2Utility\-preservation evaluation

Whether privacy transformations preserve downstream utility is central to privacy\-preserving NLP\. Prior evaluations typically run a single downstream model on original vs\. transformed data and compare end\-task scores\. Two weaknesses recur: \(a\) a single downstream model cannot separate “the transform is fine” from “this model is robust,” and \(b\) whole\-corpus metrics mix masked and unmasked content\. We address \(a\) with an 11\-detector panel spanning rule\-based, fine\-tuned, and LLM detectors, and \(b\) by restricting utility measurement to masked spans\.

### 2\.3The large\-NNtrap and equivalence testing

When samples are large, null\-hypothesis significance testing rejects the null for negligible effects; thepp\-value measures precision, not importance—a hazard increasingly flagged in NLP evaluation\[[12](https://arxiv.org/html/2608.03172#bib.bib5),[5](https://arxiv.org/html/2608.03172#bib.bib6)\]\. The standard remedy is*equivalence testing*—the two one\-sided tests \(TOST\) procedure\[[31](https://arxiv.org/html/2608.03172#bib.bib1),[4](https://arxiv.org/html/2608.03172#bib.bib7)\]—which specifies an equivalence marginΔ\\Deltaand tests whether the effect lies inside\[−Δ,\+Δ\]\[\-\\Delta,\+\\Delta\]; seeLakens\[[21](https://arxiv.org/html/2608.03172#bib.bib2)\], Lakenset al\.\[[20](https://arxiv.org/html/2608.03172#bib.bib3)\]for practical tutorials\. TOST is standard in biostatistics but under\-used in NLP evaluation, where large paired corpora make the trap acute\. We adopt it as the primary inferential tool and report McNemar’s paired test\[[27](https://arxiv.org/html/2608.03172#bib.bib4)\]only to demonstrate the trap\.

## 3Problem Formulation

Structure\-preserving de\-identification\.A transformTTmaps a documentddtod′d^\{\\prime\}by replacing each detected PHI valuevv\(of typeτ\\tau\) with a surrogatessof the same type, leaving all other characters unchanged\. Gold PHI spans onddare re\-projected ontod′d^\{\\prime\}by character\-level alignment, giving paired spans\(v,s\)\(v,s\)\.

Coverage vs\. utility\.Two quantities must not be conflated\.*Coverage*is the fraction of true PHI thatTTdetects and replaces \(a property ofTT’s detector\)\.*Utility*is, given that a span was replaced, whether a downstream detector still finds the surrogatessas well as it foundvv\(a property ofTT’s generator and of the surrogate’s realism\)\. We report coverage separately and measure utility*only on the masked\-span population*\{\(v,s\)\}\\\{\(v,s\)\\\}\. This prevents a low\-coverage transform from looking good—or a high\-coverage one from looking bad—on a metric that is really about substitution quality\.

…a34\-year\-old female likeAnna S\., seen at Methodist Hospital onApril 12, 2023?TransformTT…a35\-year\-old female likeMaria S\., seen at Methodist Hospital onMarch 13, 2021?same\-type surrogateIdenticaldetectorsuiteDD\(×11\\times\\,11\)recall on masked spans76\.1%→\\rightarrow74\.9%\(−1\.2\-1\.2pt,n=57,112n\{=\}57\{,\}112\)TOST, margin±2\\pm 2p≈3×10−9p\\approx 3\\times 10^\{\-9\}✓ EQUIVALENTdetectability preserved1Substitute2Detect \(paired\)3Measure4Equivalence testFigure 1:The paired evaluation protocol, with a worked example\.\(1\)The transform replaces each PHI value with a same\-type surrogate \(original→\\rightarrowsurrogate\), leaving everything else byte\-identical\.\(2\)The*identical*11\-detector suite scores*both*the originalddand the transformedd′d^\{\\prime\}, so any per\-span change is attributable to the substitution alone\.\(3\)Utility is measured*only on the spans the transform masks*\(overlap match\), decoupling it from coverage\.\(4\)The paired recall difference is assessed with an equivalence test \(TOST, margin±2\\pm 2pts\) rather than a significance test—which the large sample would make trivially “significant” \(§[4](https://arxiv.org/html/2608.03172#S4)\)\. Result: the change is statistically equivalent to zero\.
## 4Evaluation Protocol

Figure[1](https://arxiv.org/html/2608.03172#S3.F1)summarizes the protocol\.Paired multi\-detector design\.Each document is scored in two conditions—originalddand transformedd′d^\{\\prime\}—by the*identical*detector suiteDD\(\|D\|=11\|D\|=11\)\. Because the only change between conditions is the substitution, any per\-span change in detection is attributable to the substitution, not to the detector or the document\.

Metrics\.We report span\-level P/R/F1 under three matching modes—*exact*\(start\+end\+type\),*type*\(type\+boundary\),*overlap*\(any character overlap\+type\)—and*leakage*=1−recall=1\-\\text\{recall\}, the HIPAA\-critical quantity\. The headline utility metric is*recall retention on masked spans*=recall​\(d′\)/recall​\(d\)=\\text\{recall\}\(d^\{\\prime\}\)/\\text\{recall\}\(d\)restricted to\{\(v,s\)\}\\\{\(v,s\)\\\}, under overlap matching \(so pure boundary jitter from length changes, “Anna S\.”→\\rightarrow“Maria S\.”, is not charged as a miss\)\.

Equivalence testing\.For pooled masked\-span recall we run TOST with marginΔ=2\\Delta=2points: we reject non\-equivalence iff the 90% CI of the recall difference lies entirely within\[−2,\+2\]\[\-2,\+2\]\. We also report McNemar’s test to illustrate the large\-NNtrap\. We sweepΔ∈\{1,2,3\}\\Delta\\in\\\{1,2,3\\\}\.

Error attribution\.For the lost population \(found ondd, missed ond′d^\{\\prime\}\) we hand\-code each span into a small failure typology \(§[7](https://arxiv.org/html/2608.03172#S7)\) and check length\-preservation to separate boundary artifacts from genuine misses\.

## 5Experimental Design

Detector panelDD\.Eleven detectors, three families:*rule/statistical*—Microsoft Presidio\[[28](https://arxiv.org/html/2608.03172#bib.bib20)\]\(built on spaCy\[[18](https://arxiv.org/html/2608.03172#bib.bib22)\]\), OBIdeid\_roberta\[[19](https://arxiv.org/html/2608.03172#bib.bib13)\], a RoBERTa tagger\[[22](https://arxiv.org/html/2608.03172#bib.bib26),[11](https://arxiv.org/html/2608.03172#bib.bib25)\];*open LLMs \(local\)*—Gemma 4 31B / E4B\[[15](https://arxiv.org/html/2608.03172#bib.bib29)\], Qwen 3\.5\-\{4B, 9B, 35B\-A3B\}\[[30](https://arxiv.org/html/2608.03172#bib.bib28)\], Llama 3\.1\-8B / 3\.3\-70B\[[17](https://arxiv.org/html/2608.03172#bib.bib27)\], DeepSeek V2\-Lite\[[9](https://arxiv.org/html/2608.03172#bib.bib30)\];*frontier API*—OpenAI GPT\-5\[[29](https://arxiv.org/html/2608.03172#bib.bib31)\]\. All are Transformer\-based\[[35](https://arxiv.org/html/2608.03172#bib.bib24)\]\. The panel is deliberately heterogeneous: if the equivalence result held only for one architecture it would be a model artifact, not a property of the transform\. \(Model\-version citations point to the closest published technical report for each family\. One further model, Moonshot Kimi\-VL\-A3B, was attempted but excluded because it required non\-standard model\-loading code\.\) LLM detectors emit free\-text or JSON identifier lists; we recover character spans by exact\-then\-fuzzy string search over the source document, so all detectors are scored on the same span basis\.

Benchmarks\(7; 250 docs each, 1,750 total\)\. ASQ\-PHI \(English clinical queries\), MEDDOCAN \(Spanish clinical\)\[[26](https://arxiv.org/html/2608.03172#bib.bib10)\], MultiCoNER v2 \(multilingual NER\)\[[25](https://arxiv.org/html/2608.03172#bib.bib18),[14](https://arxiv.org/html/2608.03172#bib.bib19)\], and PII\-Masking\-300k\[[1](https://arxiv.org/html/2608.03172#bib.bib23)\]in English, Dutch, French, German\. Together: 7 languages, clinical and general\-PII text, free\-form and structured \(JSON\) formats\. The transform under test is Custodian Guardian Layertransformmode \(top\-1 surrogate,pii\_entities=ALL\)\.

Comparison conditions\.The core study answers “does*this*transform preserve detectability\.” Three contrasts isolate*why*and*how generally*:

- •C1—Redaction upper\-bound\.Replace each masked value with\*\*\*\*\*\(no surrogate signal\) and re\-detect; the transform−\-redact gap quantifies what structure preservation buys\.
- •C2—Open\-source surrogate baseline\.Replace the commercial generator with an open one \(Faker\[[13](https://arxiv.org/html/2608.03172#bib.bib21)\]\) on the*same*masked spans; tests whether the result is generic to substitution or tool\-specific, and makes the pipeline reproducible\.
- •C3—Per\-benchmark equivalence\.Run TOST within each benchmark, sweepingΔ\\Delta, to check the pooled claim is not an averaging artifact\.

The core paired study, the equivalence analysis \(pooled and per\-benchmark\), the redaction floor, the open\-surrogate baseline, and the error typology are complete; C1/C2 are currently demonstrated with the CPU detector \(Presidio\), with a full\-panel extension noted as the next strengthening step\.

## 6Results

Detectability is statistically equivalent\.Restricting to the 57,112 masked spans and pooling all 11 detectors \(via the releasedanalyze\_equivalence\.py\), recall moves76\.1%→74\.9%76\.1\\%\\rightarrow 74\.9\\%\(−1\.2\-1\.2pts; 95% CI\[−1\.5,−1\.0\]\[\-1\.5,\-1\.0\]\)\. The TOST equivalence test \(Δ=2\\Delta=2\) rejects non\-equivalence atp≈3×10−9p\\approx 3\\times 10^\{\-9\}: the change is statistically bounded within±2\\pm 2points of zero\. Ranking is preserved; Llama 3\.3\-70B is essentially unchanged \(Δ\\DeltaF1\+0\.003\+0\.003\)\. The same contingency is “significant” under McNemar’s test—a direct demonstration of the large\-NNtrap\.

Whole\-document view \(conservative\)\.Before restricting to masked spans, Table[1](https://arxiv.org/html/2608.03172#S6.T1)reports span\-level F1 and leakage on*all*gold spans, per detector\. Whole\-documentΔ\\DeltaF1 ranges from\+0\.003\+0\.003\(Llama 3\.3\-70B\) to−0\.047\-0\.047\(Qwen 3\.5\-35B\-A3B\), and leakage rises by only\+0\.3\+0\.3to\+4\.1\+4\.1points\. This view is conservative by design—it mixes masked spans with exact\-boundary penalties on length\-changed surrogates and with spans the transform never touched—yet it already bounds the worst case \(no detector’s mean F1 moves by more than 4\.7 points\) and preserves ranking across a 100×\\timesspan of detector quality \(F10\.76→0\.040\.76\\\!\\rightarrow\\\!0\.04\)\. The masked\-span analysis below isolates the substitution effect from this dilution\.

Table 1:Whole\-document view \(all 11 detectors\), mean over 7 benchmarks\. Span\-level F1 \(type match\) and leakage \(1−1\-recall\)\. Conservative: it includes untouched spans and boundary penalties\. Ranking is preserved andΔ\\DeltaF1 is bounded\.Recall retention on masked spans \(overlap\)\.Restricting to masked spans removes that dilution\. When the transform masks a span, detectors still find the surrogate 93–100% of the time \(Table[2](https://arxiv.org/html/2608.03172#S6.T2)\); the∼\\sim3\-point exact\-boundary drop is a length\-jitter artifact that vanishes under overlap matching\. \(The two floor detectors, Llama 3\.1\-8B and OBIdeid\_roberta, are omitted from Table[2](https://arxiv.org/html/2608.03172#S6.T2): their original recall is so low that the retention ratio is dominated by noise\.\)

Table 2:Recall retention on masked spans \(%\), transformed÷\\divoriginal\.Per\-benchmark equivalence \(C3\)\.Table[3](https://arxiv.org/html/2608.03172#S6.T3)\(57,112 masked spans, viascripts/analyze\_equivalence\.py\) shows the pooled equivalence is*not uniform*: four benchmarks are equivalent within±2\\pm 2points; MEDDOCAN and PII\-nl require±3\\pm 3\(both dense, identifier\-heavy, non\-English—where surrogate generation is hardest, §[7](https://arxiv.org/html/2608.03172#S7)\); MultiCoNER has too few masked spans \(220\) to test\. The honest claim:*detectability is equivalent within±2\\pm 2points on average and on clean text, and within±3\\pm 3on the hardest identifier\-dense text*—the residual is concentrated and attributable, not diffuse degradation \(Figure[2](https://arxiv.org/html/2608.03172#S6.F2)\)\.

−4\-4−3\-3−2\-2−1\-101122334455PooledPII\-dePII\-frASQ\-PHIMultiCoNERPII\-enPII\-nlMEDDOCANequivalence margin±2\\pm 2recall change: original−\-transformed \(pts\)Figure 2:Per\-benchmark equivalence \(C3\)\. Masked\-span recall change \(original−\-transformed\) with 95% CIs; the shaded band is the±2\\pm 2\-pt equivalence margin\. The pooled estimate and five of seven benchmarks sit fully inside; only MEDDOCAN and PII\-nl reach past\+2\+2\(hence a±3\\pm 3margin\)\. MultiCoNER’s wide interval reflects its 220 masked spans\.Table 3:Per\-benchmark equivalence\. O→\\rightarrowT = masked\-span recall, original→\\rightarrowtransform \(%\)\. Pooled TOSTΔ=2\\Delta\{=\}2:p=3×10−9p\{=\}3\{\\times\}10^\{\-9\}\.Redaction floor \(C1\)\.Replacing masked values with\*\*\*\*\*and re\-detecting with Presidio \(Table[4](https://arxiv.org/html/2608.03172#S6.T4)\): transform recall tracks the*original*within a few points, while redaction collapses to≈\\approx0–4% \(the residual is spurious overlap on the\*run\)\. Structure\-preserving substitution preserves essentially the entire detectability that redaction destroys\.

Table 4:C1 redaction floor: Presidio masked\-span recall \(%\)\.Open\-surrogate baseline \(C2\)\.Replacing the commercial generator with Faker on the same masked spans \(Table[5](https://arxiv.org/html/2608.03172#S6.T5)\) yields two findings\.*Generality*: an open substitutor preserves masked\-span recall at least as well as the original—detectability preservation appears to be a property of well\-formed same\-type substitution, not of one vendor, and reproduces without proprietary access\.*The residual is generator quality*: Faker, emitting clean canonical values,*exceeds*the commercial transform by 18–20 points on the hard non\-English benchmarks, precisely where the commercial surrogates truncate or garble \(§[7](https://arxiv.org/html/2608.03172#S7)\)\.

Table 5:C2 open\-surrogate baseline: Presidio masked\-span recall \(%\)\. Detector\-independent argument; Presidio\-only demonstration\.ASQMEDDennlfrde05050100100masked\-span recall \(%\)OriginalTransformFakerRedactFigure 3:What structure preservation buys \(C1/C2; Presidio masked\-span recall\)\. Transform tracks the original, the open Faker baseline is at least as high, and redaction collapses to≈\\approx0\. The gap between the Transform/Faker bars and the Redact bar is the detectability that structure preservation retains\.Leakagebarely moves \(\+0\.3\+0\.3to\+4\.1\+4\.1pts across detectors; Table[1](https://arxiv.org/html/2608.03172#S6.T1)\), so surrogates are not systematically easier to miss than the PHI they replace\.

Where the loss lands \(per benchmark\)\.Table[6](https://arxiv.org/html/2608.03172#S6.T6)breaks whole\-documentΔ\\DeltaF1 out by benchmark and detector\. The pattern is sharp: almost all of the loss concentrates onASQ\-PHI—short, sparse\-PHI adversarial queries where a single substitution dominates the document score \(−0\.09\-0\.09to−0\.18\-0\.18\)—while the other six benchmarks are essentially flat \(\|Δ​F1\|≤0\.06\|\\Delta\\text\{F1\}\|\\leq 0\.06, most≤0\.03\\leq 0\.03\)\. Even the fine\-tuned OBI tagger and the rule\-based Presidio move by at most a few thousandths on most benchmarks\. This is the first sign that the effect tracks*surrogate\-generation difficulty*\(short adversarial text, dense non\-English identifiers\) rather than detector family—made precise in the error analysis \(§[7](https://arxiv.org/html/2608.03172#S7)\)\.

Table 6:Per\-benchmark whole\-documentΔ\\DeltaF1 \(transformed−\-original; negative = drop\)\. Almost all loss lands on ASQ\-PHI; the other six benchmarks are flat\. Llama 3\.3\-70B and DeepSeek V2\-Lite omitted for space \(both flat; see Table[1](https://arxiv.org/html/2608.03172#S6.T1)for their pooledΔ\\DeltaF1\)\.What the transform does \(examples\)\.Table[7](https://arxiv.org/html/2608.03172#S6.T7)shows the substitution qualitatively across languages and formats: each PHI value becomes a same\-type surrogate while clinical shorthand, foreign\-language syntax, and JSON structure are left byte\-for\-byte intact\. On the well\-formed English cases every surrogate is still detected across most of the panel\.

Table 7:Worked transform examples\. Only the sensitive value moves; abbreviations \(w/ CHF,alt tx\), non\-English syntax, and JSON keys/quotes/indentation are preserved, so downstream parsers and detectors keep working\. In the JSON case the transform changes only the date value and leaves the \(also\-sensitive\-looking\) username untouched—a coverage decision, not a utility one \(see below\)\.Coverage, reported separately\.All results above are measured*on the spans the transform masks*\. Coverage—how much true PHI it detects and replaces—is a distinct, detector\-side axis\. It tracks how closely a benchmark’s annotation matches the transform’s notion of sensitive content: it masks80\.9%of gold PHI on ASQ\-PHI and48\.3%on MEDDOCAN, and less \(≈\\approx26%\) on general\-domain NER/PII corpora whose annotated entities \(encyclopedic names, generic places\) fall outside that scope; a configuration sweep confirmeddomain=Generalmaximizes coverage\. A per\-entity\-type breakdown and a detection\-vs\-replacement diagnosis \(the coverage gap is mostly a*replacement*\-step issue—≈\\approx77% of missed clinical identifiers were flagged by the transform’s own detector but not substituted\) are in Appendix[A](https://arxiv.org/html/2608.03172#A1)\. Decoupling matters because the two failure modes have different owners and fixes: an under\-masked span is a*detection/replacement*miss, whereas a masked\-but\-missed surrogate is a*generation*defect \(§[7](https://arxiv.org/html/2608.03172#S7)\)\. Conflating them on whole\-document F1 \(Table[1](https://arxiv.org/html/2608.03172#S6.T1)\) would let a high\-coverage/low\-quality transform and a low\-coverage/high\-quality one look identical\. We therefore report coverage as context and reserve the utility claim for masked spans\.

Matching modes\.Unless noted, retention uses*overlap*matching \(detector flags any part of the surrogate span\)\. Exact matching \(Table[2](https://arxiv.org/html/2608.03172#S6.T2), left\) additionally requires identical character boundaries and is therefore sensitive to surrogate length changes \(“Anna S\.”→\\rightarrow“Maria S\.” shifts the end offset\); the∼\\sim3\-point exact−\-overlap gap is this boundary jitter, not missed PHI\. Type matching \(boundary \+ type\) sits between the two and tracks overlap closely\.

## 7Error Analysis

The lost population\.Pooled across detectors and masked spans: 40,165 \(span×\\timesdetector\) pairs found in both conditions; 3,300 lost\. Half \(50%\) of lost spans are the*same length*as the original—so this is not a boundary effect\. By type: LOCATION 27%, NAME 23%, DATE/AGE 22%, ID/contact 19%\.

Three failure modes—all generator\-side\.\(1\)*Malformed/truncated surrogates*\(largest cause\):Chicago→\\rightarrowIllino,El Paso→\\rightarrowEl,Ciudad de la Habana→\\rightarrowCuidad de la Havana\. The fragment no longer matches the lexical pattern detectors learned for real names/places; this is why span\-level loss is highest on MEDDOCAN \(6\.9%; Spanish, dense, identifier\-heavy\) and lowest on clean English ASQ\-PHI \(2\.5%\)\. \(2\)*Loss of salience*: a canonical entity replaced by an obscure one \(Cedars\-Sinai→\\rightarrowVidant\); detectors partly rely on pre\-training familiarity, so swapping a famous value for a rare one removes the prior\. Inherent to any value substitution; mainly costs weaker detectors\. \(3\)*x\-masking of IDs/emails*:nachorutor@…→\\rightarrownxxxxxxxxx@…\. Thexrun preserves format but breaks the realistic\-token pattern\. This case is*privacy\-positive*—the original value is destroyed—even though it counts against recall\.

Implication\.Detectors are not getting worse at PHI; the small recall gap is driven by surrogate\-generation quality \(truncation, garbling, salience,x\-masking\)—generating well\-formed, in\-distribution synthetic text is itself a hard problem in text data augmentation\[[3](https://arxiv.org/html/2608.03172#bib.bib32)\]\. C2 confirms this directly: an open generator emitting clean canonical values*exceeds*the commercial transform by 18–20 points on exactly the hard non\-English benchmarks where its surrogates truncate—so the residual tracks generator quality, not the act of substitution\.

## 8Discussion

What the evidence supports\.Structure\-preserving substitution does not hide well\-formed PHI from downstream detection\. Because the effect is equivalence\-bounded within±2\\pm 2points and ranking\-preserving across a heterogeneous 11\-detector panel, a transform of this quality can be inserted ahead of detection/analytics pipelines built for real clinical text without materially degrading downstream detection\.Generality \(C2\)\.Faker preserves masked\-span recall at least as well as the original across all six benchmarks, so±2\\pm 2points appears to be a property of well\-formed substitution rather than of one vendor; where the commercial generator trails, a cleaner generator closes the gap, locating the residual squarely in generation quality\.A reusable protocol\.The paired masked\-span design and the TOST margin are not specific to one vendor or language—a template for auditing any structure\-preserving privacy transform\.

## 9Ethics and Data Statement

All benchmarks are public or synthetic \(ASQ\-PHI synthetic; MEDDOCAN released for a shared task; PII\-Masking\-300k synthetic; MultiCoNER v2 public\)\.No real patient data is used\.We release the 250\-document subsets and scoring code with license notes\.Dual\-use\.A utility\-preserving de\-identifier could in principle launder identifiable data into a fluent form; our masked\-span/leakage reporting and the privacy\-positive framing ofx\-masking keep the privacy accounting explicit\.Conflict of interest\.The commercial transform \(Custodian Guardian Layer\) is developed by Custodian Labs, with which the authors are affiliated; this work was conducted with Custodian Labs’ support\. To limit bias we \(i\) frame the contribution as a reusable protocol, \(ii\) include open\-source baselines so results are reproducible without proprietary access, and \(iii\) report leakage and coverage alongside utility\.

## 10Limitations

Coverage is reported but not the focus; a transform can preserve utility on what it masks while under\-masking \(the axes are independent by design\)\. 250 docs/benchmark bounds per\-benchmark power \(though the pooled masked\-spanNNis large\)\. C1/C2 are currently demonstrated with a CPU detector \(Presidio\); a full 11\-detector extension is future work\. LLM detectors are prompt\-sensitive and their behaviour shifts with task framing and structural variation\[[16](https://arxiv.org/html/2608.03172#bib.bib33),[2](https://arxiv.org/html/2608.03172#bib.bib34)\]; we fix one prompt per model and release it with the code and data at the project’s reproducibility page\. Finally, all seven benchmarks are public or synthetic; validating the protocol on real\-EHR corpora \(n2c2, MIMIC\-IV\-Note, CARMEN\-I\) is future work pending the relevant data\-use agreements\.

## Appendix ACoverage Details

Coverage \(§[6](https://arxiv.org/html/2608.03172#S6)\) is the fraction of a benchmark’s gold PHI whose characters the transform actually changed \(difflib alignment\)\. Table[8](https://arxiv.org/html/2608.03172#A1.T8)breaks it down by entity type on the clinical sets \(ASQ\-PHI \+ MEDDOCAN\)\. High\-frequency free\-text types \(names, dates\) are masked well; the weak spots are high\-sensitivity structured identifiers and geography—exactly the HIPAA Safe Harbor items that most need masking\.

Table 8:Coverage by entity type on clinical benchmarks\. Overall the transform altered 39% of annotated PII \(54% on clinical sets\); general\-domain corpora sit near 26%\.Detection vs\. replacement\.On a sample of 8 MEDDOCAN documents \(62 missed ID/location spans\),48/6248/62\(≈\\approx77%\) of the*un*masked identifiers were nonetheless flagged as sensitive by the transform’s own detector—they were detected but not substituted; only14/6214/62\(23%\) were undetected\. The coverage gap is therefore mostly a*replacement\-step*issue \(act on everything the detector surfaces\), which is more tractable than raising detection recall\. \(Small sample; “detected” judged by loose token overlap, so 77% is directional\.\) Concrete unmasked HIPAA identifiers included patient IDs \(80926,7845693\), care\-contact IDs \(4387684\), and facilities/geography \(Hospital de Cruces,Espa~na, postal codes41005,28047\)\.

## Appendix BRepresentative Per\-Detector Examples

All\-detector overlay on one document \(“ranking proof”\)\.All eleven detectors’ predictions on one transformed document \(asq\_00001\), overlaid on the text and sorted by this document’s F1:green= correctly caught PHI,yellow= false positive,red= missed PHI \(leakage\)\. The four gold surrogate values areJohn T\., the*x*\-masked locationSaint\. Vxxxxxxxx,April 25th, 2018, and the ID987654321\. Eight of eleven detectors tag all four cleanly \(F1 100\); the two floor detectors, Presidio and DeepSeek, miss the*x*\-masked location surrogate \(§[7](https://arxiv.org/html/2608.03172#S7)mode 3\)—DeepSeek also drops the numeric ID—and OBI over\-fragments boundaries and false\-fires on the age60\. The surrogate values shown are the transform’s output, not the original PHI\.

Gold \(reference\)4 PHI spans Evaluation of long\-term outcomes for bypass surgery in patients over 60, referencing Mr\.John T\., operated atSaint\. VxxxxxxxxonApril 25th, 2018\(ID:987654321\)?

OpenAI GPT\-5tp 4⋅\\cdotfp 0⋅\\cdotmissed 0⋅\\cdotF1 100% Evaluation of long\-term outcomes for bypass surgery in patients over 60, referencingMr\. John T\., operated atSaint\. VxxxxxxxxonApril 25th, 2018\(ID: 987654321\)?

Llama 3\.1\-8Btp 4⋅\\cdotfp 0⋅\\cdotmissed 0⋅\\cdotF1 100% Evaluation of long\-term outcomes for bypass surgery in patients over 60, referencing Mr\.John T\., operated atSaint\. VxxxxxxxxonApril 25th, 2018\(ID:987654321\)?

Qwen 3\.5\-4Btp 4⋅\\cdotfp 0⋅\\cdotmissed 0⋅\\cdotF1 100% Evaluation of long\-term outcomes for bypass surgery in patients over 60, referencingMr\. John T\., operated atSaint\. VxxxxxxxxonApril 25th, 2018\(ID:987654321\)?

Gemma 4 31Btp 4⋅\\cdotfp 0⋅\\cdotmissed 0⋅\\cdotF1 100% Evaluation of long\-term outcomes for bypass surgery in patients over 60, referencingMr\. John T\., operated atSaint\. VxxxxxxxxonApril 25th, 2018\(ID:987654321\)?

Llama 3\.3\-70Btp 4⋅\\cdotfp 0⋅\\cdotmissed 0⋅\\cdotF1 100% Evaluation of long\-term outcomes for bypass surgery in patients over 60, referencing Mr\.John T\., operated atSaint\. VxxxxxxxxonApril 25th, 2018\(ID:987654321\)?

Qwen 3\.5\-35B\-A3Btp 4⋅\\cdotfp 0⋅\\cdotmissed 0⋅\\cdotF1 100% Evaluation of long\-term outcomes for bypass surgery in patients over 60, referencingMr\. John T\., operated atSaint\.Vxxxxxxxx onApril 25th,2018\(ID:987654321\)?

Gemma 4 E4Btp 4⋅\\cdotfp 0⋅\\cdotmissed 0⋅\\cdotF1 100% Evaluation of long\-term outcomes for bypass surgery in patients over 60, referencing Mr\.John T\., operated atSaint\. VxxxxxxxxonApril 25th, 2018\(ID:987654321\)?

Qwen 3\.5\-9Btp 4⋅\\cdotfp 0⋅\\cdotmissed 0⋅\\cdotF1 100% Evaluation of long\-term outcomes for bypass surgery in patients over 60, referencingMr\. John T\., operated atSaint\. VxxxxxxxxonApril 25th, 2018\(ID:987654321\)?

OBI deid\_robertatp 4⋅\\cdotfp 1⋅\\cdotmissed 0⋅\\cdotF1 89% Evaluation of long\-term outcomes for bypass surgery in patients over60, referencing Mr\.JohnT\., operated atSaint\.VxxxxxxxxonApril 25th,2018\(ID:987654321\)?

Microsoft Presidiotp 3⋅\\cdotfp 1⋅\\cdotmissed 1⋅\\cdotF1 75% Evaluation of long\-term outcomes for bypass surgery in patients over60, referencing Mr\.John T\., operated atSaint\. VxxxxxxxxonApril 25th, 2018\(ID:987654321\)?

DeepSeek V2\-Litetp 2⋅\\cdotfp 0⋅\\cdotmissed 2⋅\\cdotF1 67% Evaluation of long\-term outcomes for bypass surgery in patients over 60, referencing Mr\.John T\., operated atSaint\. VxxxxxxxxonApril 25th, 2018\(ID:987654321\)?

A harder case: a dense Spanish clinical header\.Figure\-style overlay on a 260\-character window of a MEDDOCAN document \(S0210\-…008\-1\), where the same eleven detectors diverge far more—F1 from 96 down to 0\. On dense, identifier\-heavy non\-English text the*per\-document*ranking reshuffles \(here DeepSeek and Qwen\-4B lead, while Llama 3\.1\-8B finds nothing in this window\), even though the*aggregate*ranking over 250 documents is stable \(Table[1](https://arxiv.org/html/2608.03172#S6.T1)\)\. The garbled surrogateCuidad Real\(fromCiudad Real\) is still caught by most detectors; the misses cluster on the professional\-licence ID03 14 16485, the sex fields, and the repeated age—the identifier types with the weakest coverage and the hardest surrogates \(Appendix[A](https://arxiv.org/html/2608.03172#A1)\)\.

Gold \(reference\)12 PHI spans Cuidad Real\. CP:13002\. Datos asistenciales\. Referencia de nacimiento:24/10/1963\. País:España\. Mejores:53 añosSexo:H\. Referencia de Ingreso:14/12/2017\. Episodio:746589123\. Médico:Luis Ruiz CamuñasNºCol:03 14 16485\. Historia Actual:Varónde53 añosq

DeepSeek V2\-Litetp 11⋅\\cdotfp 0⋅\\cdotmissed 1⋅\\cdotF1 96% Cuidad Real\. CP:13002\. Datos asistenciales\. Referencia de nacimiento:24/10/1963\. País:España\. Mejores:53 añosSexo:H\. Referencia de Ingreso:14/12/2017\. Episodio:746589123\. Médico:Luis Ruiz CamuñasNºCol:03 14 16485\. Historia Actual:Varón de 53 años q

Qwen 3\.5\-4Btp 11⋅\\cdotfp 0⋅\\cdotmissed 1⋅\\cdotF1 96% Cuidad Real\.CP:13002\.Datos asistenciales\. Referencia de nacimiento:24/10/1963\.País:España\.Mejores:53años Sexo:H\.Referencia de Ingreso:14/12/2017\.Episodio:746589123\.Médico:Luis Ruiz CamuñasNºCol:031416485\.Historia Actual:Varónde53años q

OBI deid\_robertatp 10⋅\\cdotfp 0⋅\\cdotmissed 2⋅\\cdotF1 91% CuidadReal\. CP:13002\. Datos asistenciales\. Referencia de nacimiento:24/10/1963\. País:España\. Mejores:53años Sexo:H\. Referencia de Ingreso:14/12/2017\. Episodio:746589123\. Médico:Luis RuizCamuñas NºCol:03 1416485\. Historia Actual:Varónde53años q

Qwen 3\.5\-9Btp 10⋅\\cdotfp 0⋅\\cdotmissed 2⋅\\cdotF1 91% Cuidad Real\. CP:13002\. Datos asistenciales\. Referencia de nacimiento:24/10/1963\. País:España\. Mejores:53años Sexo:H\. Referencia de Ingreso:14/12/2017\. Episodio:746589123\. Médico:Luis Ruiz CamuñasNºCol:03 14 16485\. Historia Actual:Varónde53años q

Gemma 4 31Btp 9⋅\\cdotfp 0⋅\\cdotmissed 3⋅\\cdotF1 86% Cuidad Real\. CP:13002\. Datos asistenciales\. Referencia de nacimiento:24/10/1963\. País:España\. Mejores:53 añosSexo:H\. Referencia de Ingreso:14/12/2017\. Episodio:746589123\. Médico:Luis Ruiz CamuñasNºCol:03 14 16485\. Historia Actual:Varónde53 añosq

Qwen 3\.5\-35B\-A3Btp 9⋅\\cdotfp 1⋅\\cdotmissed 3⋅\\cdotF1 82% Cuidad Real\. CP:13002\. Datos asistenciales\. Referencia de nacimiento:24/10/1963\. País:España\. Mejores:53años Sexo:H\. Referencia de Ingreso:14/12/2017\. Episodio:746589123\. Médico:Luis Ruiz CamuñasNºCol:03 14 16485\. Historia Actual:Varónde53 añosq

OpenAI GPT\-5tp 8⋅\\cdotfp 0⋅\\cdotmissed 4⋅\\cdotF1 80% Cuidad Real\. CP:13002\. Datos asistenciales\. Referencia de nacimiento:24/10/1963\. País:España\. Mejores:53años Sexo:H\. Referencia de Ingreso:14/12/2017\. Episodio:746589123\. Médico:Luis Ruiz CamuñasNºCol:03 14 16485\. Historia Actual:Varónde53 añosq

Llama 3\.3\-70Btp 8⋅\\cdotfp 0⋅\\cdotmissed 4⋅\\cdotF1 80% Cuidad Real\. CP:13002\. Datos asistenciales\. Referencia de nacimiento:24/10/1963\. País:España\. Mejores:53años Sexo:H\. Referencia de Ingreso:14/12/2017\. Episodio:746589123\. Médico:Luis Ruiz CamuñasNºCol:03 14 16485\. Historia Actual:Varónde53 añosq

Gemma 4 E4Btp 8⋅\\cdotfp 0⋅\\cdotmissed 4⋅\\cdotF1 80% Cuidad Real\. CP:13002\. Datos asistenciales\. Referencia de nacimiento:24/10/1963\. País:España\. Mejores:53años Sexo:H\. Referencia de Ingreso:14/12/2017\. Episodio:746589123\. Médico:Luis Ruiz CamuñasNºCol: 03 14 16485\. Historia Actual:Varónde53años q

Microsoft Presidiotp 6⋅\\cdotfp 1⋅\\cdotmissed 6⋅\\cdotF1 63% Cuidad Real\. CP:13002\. Datos asistenciales\. Referencia de nacimiento:24/10/1963\.País:España\. Mejores:53 añosSexo:H\. Referencia de Ingreso:14/12/2017\. Episodio:746589123\. Médico:Luis Ruiz Camuñas NºCol:03 14 16485\. Historia Actual:Varónde53 añosq

Llama 3\.1\-8Btp 0⋅\\cdotfp 0⋅\\cdotmissed 12⋅\\cdotF1 0% Cuidad Real\. CP:13002\. Datos asistenciales\. Referencia de nacimiento:24/10/1963\. País:España\. Mejores:53 añosSexo:H\. Referencia de Ingreso:14/12/2017\. Episodio:746589123\. Médico:Luis Ruiz CamuñasNºCol:03 14 16485\. Historia Actual:Varónde53 añosq

Agreement on well\-formed surrogates\.On clean English \(ASQ\-PHI\), a plausible same\-type surrogate is caught by nearly the whole panel \(Table[9](https://arxiv.org/html/2608.03172#A2.T9)\): the substitution is invisible to detection\. Divergence is confined to the two floor detectors \(Llama 3\.1\-8B, DeepSeek V2\-Lite\)\.

Table 9:Number of detectors \(of 11\) that find each well\-formed surrogate\. Same\-type swaps stay broadly detectable\.Representative losses, by failure type\.Table[10](https://arxiv.org/html/2608.03172#A2.T10)shows genuine “lost” cases \(MEDDOCAN\): the original value was found by most detectors, but a defective surrogate is found by few\. Each maps to one of the three failure types in §[7](https://arxiv.org/html/2608.03172#S7), and the drop is shared across the panel—i\.e\. it is a property of the surrogate, not of any one detector\.

Table 10:Representative lost spans on MEDDOCAN\. O→\\rightarrowT = detectors finding the original vs\. the surrogate \(panel of 11–12; original includes one extra decoding variant\)\. Losses are shared across detectors and align with the §[7](https://arxiv.org/html/2608.03172#S7)typology\.
## References

- \[1\]ai4Privacy\(2023\)PII\-masking\-300k: a dataset for training privacy\-preserving models\.Note:[https://huggingface\.co/datasets/ai4privacy/pii\-masking\-300k](https://huggingface.co/datasets/ai4privacy/pii-masking-300k)Cited by:[§5](https://arxiv.org/html/2608.03172#S5.p2.1)\.
- \[2\]Q\. Bao, G\. Gendron, A\. Y\. Peng, M\. Witbrock, and J\. Liu\(2024\)Assessing and enhancing the robustness of large language models with task structure variations for logical reasoning\.InNeural Information Processing \(ICONIP 2024\),Lecture Notes in Computer Science\.Cited by:[§10](https://arxiv.org/html/2608.03172#S10.p1.1)\.
- \[3\]Q\. Bao, A\. Y\. Peng, Z\. Deng, W\. Zhong, G\. Gendron, T\. Pistotti, N\. Tan, N\. Young, Y\. Chen, Y\. Zhu, P\. Denny, M\. Witbrock, and J\. Liu\(2024\)Abstract meaning representation\-based logic\-driven data augmentation for logical reasoning\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 5914–5934\.Cited by:[§7](https://arxiv.org/html/2608.03172#S7.p3.1)\.
- \[4\]R\. L\. Berger and J\. C\. Hsu\(1996\)Bioequivalence trials, intersection\-union tests and equivalence confidence sets\.Statistical Science11\(4\),pp\. 283–319\.Cited by:[§2\.3](https://arxiv.org/html/2608.03172#S2.SS3.p1.3)\.
- \[5\]D\. Card, P\. Henderson, U\. Khandelwal, R\. Jia, K\. Mahowald, and D\. Jurafsky\(2020\)With little power comes great responsibility\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 9263–9274\.Cited by:[§2\.3](https://arxiv.org/html/2608.03172#S2.SS3.p1.3)\.
- \[6\]D\. Carrell, B\. Malin, J\. Aberdeen, S\. Bayer, C\. Clark, B\. Wellner, and L\. Hirschman\(2013\)Hiding in plain sight: use of realistic surrogates to reduce exposure of protected health information in clinical text\.Journal of the American Medical Informatics Association20\(2\),pp\. 342–348\.Cited by:[§1](https://arxiv.org/html/2608.03172#S1.p1.2),[§2\.1](https://arxiv.org/html/2608.03172#S2.SS1.p1.1)\.
- \[7\]D\. S\. Carrell, D\. J\. Cronkite, M\. R\. Li, S\. Nyemba, B\. A\. Malin, J\. S\. Aberdeen, and L\. Hirschman\(2020\)The machine giveth and the machine taketh away: a parrot attack on clinical text deidentified with hiding in plain sight\.Journal of the American Medical Informatics Association27\(12\),pp\. 1937–1942\.Cited by:[§2\.1](https://arxiv.org/html/2608.03172#S2.SS1.p1.1)\.
- \[8\]Custodian Labs\(2025\)Custodian Labs Guardian Layer\.Note:[http://www\.custodianlabs\.io/](http://www.custodianlabs.io/)Company website; formal method citation to followCited by:[§2\.1](https://arxiv.org/html/2608.03172#S2.SS1.p2.1)\.
- \[9\]DeepSeek\-AI\(2024\)DeepSeek\-V2: a strong, economical, and efficient mixture\-of\-experts language model\.arXiv preprint arXiv:2405\.04434\.Cited by:[§5](https://arxiv.org/html/2608.03172#S5.p1.1)\.
- \[10\]F\. Dernoncourt, J\. Y\. Lee, Ö\. Uzuner, and P\. Szolovits\(2017\)De\-identification of patient notes with recurrent neural networks\.Journal of the American Medical Informatics Association24\(3\),pp\. 596–606\.Cited by:[§2\.1](https://arxiv.org/html/2608.03172#S2.SS1.p1.1)\.
- \[11\]J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova\(2019\)BERT: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics \(NAACL\-HLT\),pp\. 4171–4186\.Cited by:[§5](https://arxiv.org/html/2608.03172#S5.p1.1)\.
- \[12\]R\. Dror, G\. Baumer, S\. Shlomov, and R\. Reichart\(2018\)The hitchhiker’s guide to testing statistical significance in natural language processing\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 1383–1392\.Cited by:[§2\.3](https://arxiv.org/html/2608.03172#S2.SS3.p1.3)\.
- \[13\]D\. Faraglia and Other Contributors\(2012\)Faker: a Python package that generates fake data\.Note:[https://github\.com/joke2k/faker](https://github.com/joke2k/faker)Cited by:[2nd item](https://arxiv.org/html/2608.03172#S5.I1.i2.p1.1)\.
- \[14\]B\. Fetahu, S\. Kar, Z\. Chen, O\. Rokhlenko, and S\. Malmasi\(2023\)MultiCoNER v2: a large multilingual dataset for fine\-grained and noisy named entity recognition\.InFindings of the Association for Computational Linguistics: EMNLP 2023,Cited by:[§5](https://arxiv.org/html/2608.03172#S5.p2.1)\.
- \[15\]Gemma Team, Google DeepMind\(2024\)Gemma 2: improving open language models at a practical size\.arXiv preprint arXiv:2408\.00118\.Cited by:[§5](https://arxiv.org/html/2608.03172#S5.p1.1)\.
- \[16\]G\. Gendron, Q\. Bao, M\. Witbrock, and G\. Dobbie\(2024\)Large language models are not strong abstract reasoners\.InProceedings of the Thirty\-Third International Joint Conference on Artificial Intelligence \(IJCAI\),pp\. 6270–6278\.Cited by:[§10](https://arxiv.org/html/2608.03172#S10.p1.1)\.
- \[17\]Grattafiori, Aaron and Dubey, Abhimanyu and others \(Llama Team, AI @ Meta\)\(2024\)The Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§5](https://arxiv.org/html/2608.03172#S5.p1.1)\.
- \[18\]M\. Honnibal, I\. Montani, S\. Van Landeghem, and A\. Boyd\(2020\)spaCy: industrial\-strength natural language processing in Python\.Note:[https://spacy\.io](https://spacy.io/)Cited by:[§5](https://arxiv.org/html/2608.03172#S5.p1.1)\.
- \[19\]A\. E\. W\. Johnson, L\. Bulgarelli, and T\. J\. Pollard\(2020\)Deidentification of free\-text medical records using pre\-trained bidirectional transformers\.InProceedings of the ACM Conference on Health, Inference, and Learning \(CHIL\),pp\. 214–221\.Cited by:[§2\.1](https://arxiv.org/html/2608.03172#S2.SS1.p1.1),[§5](https://arxiv.org/html/2608.03172#S5.p1.1)\.
- \[20\]D\. Lakens, A\. M\. Scheel, and P\. M\. Isager\(2018\)Equivalence testing for psychological research: a tutorial\.Advances in Methods and Practices in Psychological Science1\(2\),pp\. 259–269\.Cited by:[§2\.3](https://arxiv.org/html/2608.03172#S2.SS3.p1.3)\.
- \[21\]D\. Lakens\(2017\)Equivalence tests: a practical primer for t tests, correlations, and meta\-analyses\.Social Psychological and Personality Science8\(4\),pp\. 355–362\.Cited by:[§2\.3](https://arxiv.org/html/2608.03172#S2.SS3.p1.3)\.
- \[22\]Y\. Liu, M\. Ott, N\. Goyal, J\. Du, M\. Joshi, D\. Chen, O\. Levy, M\. Lewis, L\. Zettlemoyer, and V\. Stoyanov\(2019\)RoBERTa: a robustly optimized BERT pretraining approach\.arXiv preprint arXiv:1907\.11692\.Cited by:[§5](https://arxiv.org/html/2608.03172#S5.p1.1)\.
- \[23\]Z\. Liu, B\. Tang, X\. Wang, and Q\. Chen\(2017\)De\-identification of clinical notes via recurrent neural network and conditional random field\.Journal of Biomedical Informatics75,pp\. S34–S42\.Cited by:[§2\.1](https://arxiv.org/html/2608.03172#S2.SS1.p1.1)\.
- \[24\]Z\. Liu, X\. Yu, L\. Zhang, Z\. Wu, C\. Cao, H\. Dai,et al\.\(2023\)DeID\-GPT: zero\-shot medical text de\-identification by GPT\-4\.arXiv preprint arXiv:2303\.11032\.Cited by:[§2\.1](https://arxiv.org/html/2608.03172#S2.SS1.p1.1)\.
- \[25\]S\. Malmasi, A\. Fang, B\. Fetahu, S\. Kar, and O\. Rokhlenko\(2022\)MultiCoNER: a large\-scale multilingual dataset for complex named entity recognition\.InProceedings of the 29th International Conference on Computational Linguistics \(COLING\),pp\. 3798–3809\.Cited by:[§5](https://arxiv.org/html/2608.03172#S5.p2.1)\.
- \[26\]M\. Marimon, A\. Gonzalez\-Agirre, A\. Intxaurrondo, H\. Rodriguez, J\. A\. Lopez Martin, M\. Villegas, and M\. Krallinger\(2019\)Automatic de\-identification of medical texts in Spanish: the MEDDOCAN track, corpus, guidelines, methods and evaluation of results\.InProceedings of the Iberian Languages Evaluation Forum \(IberLEF\),Cited by:[§2\.1](https://arxiv.org/html/2608.03172#S2.SS1.p1.1),[§5](https://arxiv.org/html/2608.03172#S5.p2.1)\.
- \[27\]Q\. McNemar\(1947\)Note on the sampling error of the difference between correlated proportions or percentages\.Psychometrika12\(2\),pp\. 153–157\.Cited by:[§2\.3](https://arxiv.org/html/2608.03172#S2.SS3.p1.3)\.
- \[28\]Microsoft\(2018\)Presidio: data protection and de\-identification SDK\.Note:[https://github\.com/microsoft/presidio](https://github.com/microsoft/presidio)Cited by:[§2\.1](https://arxiv.org/html/2608.03172#S2.SS1.p1.1),[§5](https://arxiv.org/html/2608.03172#S5.p1.1)\.
- \[29\]OpenAI\(2023\)GPT\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§5](https://arxiv.org/html/2608.03172#S5.p1.1)\.
- \[30\]Qwen Team\(2024\)Qwen2\.5 technical report\.arXiv preprint arXiv:2412\.15115\.Cited by:[§5](https://arxiv.org/html/2608.03172#S5.p1.1)\.
- \[31\]D\. J\. Schuirmann\(1987\)A comparison of the two one\-sided tests procedure and the power approach for assessing the equivalence of average bioavailability\.Journal of Pharmacokinetics and Biopharmaceutics15\(6\),pp\. 657–680\.Cited by:[§2\.3](https://arxiv.org/html/2608.03172#S2.SS3.p1.3)\.
- \[32\]A\. Stubbs and Ö\. Uzuner\(2015\)Annotating longitudinal clinical narratives for de\-identification: the 2014 i2b2/UTHealth corpus\.Journal of Biomedical Informatics58,pp\. S20–S29\.Cited by:[§2\.1](https://arxiv.org/html/2608.03172#S2.SS1.p1.1)\.
- \[33\]U\.S\. Department of Health and Human Services\(2012\)Guidance regarding methods for de\-identification of protected health information in accordance with the HIPAA Privacy Rule\.Note:[https://www\.hhs\.gov/hipaa/for\-professionals/privacy/special\-topics/de\-identification/](https://www.hhs.gov/hipaa/for-professionals/privacy/special-topics/de-identification/)Cited by:[§2\.1](https://arxiv.org/html/2608.03172#S2.SS1.p1.1)\.
- \[34\]Ö\. Uzuner, Y\. Luo, and P\. Szolovits\(2007\)Evaluating the state\-of\-the\-art in automatic de\-identification\.Journal of the American Medical Informatics Association14\(5\),pp\. 550–563\.Cited by:[§2\.1](https://arxiv.org/html/2608.03172#S2.SS1.p1.1)\.
- \[35\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin\(2017\)Attention is all you need\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§5](https://arxiv.org/html/2608.03172#S5.p1.1)\.