编码器级速度下的强大多语言隐私标记

# Strong Multilingual Privacy Tagging at Encoder Speed
Source: [https://arxiv.org/html/2609.38630](https://arxiv.org/html/2609.38630)
###### Abstract

Privacy redaction must reliably remove personal information while preserving the relationships expressed in the text\. We develop a general\-purpose multilingual named\-entity tagger with distinctions fine\-grained enough to support a broad range of redaction policies, and methods for cheaply learning additional distinctions\. We fine\-tune a multilingual encoder with an affine span\-tagging head on supervision from prompted frontier language models in 35 languages, replay corpus\-mapped human gold with coverage\-aware masking so that unannotated types are not treated as negatives, and repair subword span boundaries with a learned ±1\-character adjustment\. On 1,283 human\-gold test segments in seven languages, the model’s best measured redaction F1 is 88\.8, against 69\.1 for published GLiNER2 with 11 unrepresentable types excluded from its task \(68\.8 without that exemption\), 67\.8 for GLiNER2 adapted to the new training data, 57\.3 for Microsoft Presidio and 35\.8 for the best published OpenAI Privacy Filter fine\-tune\. Adding about 50,000 annotated training sentences and increasing human\-gold replay improves exact typed\-span F1 from 74\.5 to 76\.3 on Ont3, our 31\-type frontier\-annotated NER evaluation of 1,201 development segments\. Mapped\-gold replay alone raises human\-gold F1 by ten points without loss on the frontier\-annotated text, and ±1\-character boundary adjustment adds 1\.7 exact typed\-span F1 points on Ont3\. Local LLMs fitting on a single 96\-GB GPU underperformed as prompted annotators and frozen encoders, with LLM encoding 30–95 times slower than XLM\-R inference and prompted annotation roughly 180–1,100 times slower in the evaluated configurations\. The encoder architecture delivers 4\.9 times GLiNER2’s CPU throughput\. We release code, prompts and training recipes, with data\-acquisition scripts and source links\.

## 1Introduction

We compare our encoder with published models across 35 languages \(Section[A\.7](https://arxiv.org/html/2609.38630#.A1.SS7)\)\.Ont1, the union of 116 label distinctions observed in gold\-annotated corpora with projections into coarse superclasses, was consolidated intoOnt2’s 29 primary entity types with consistent annotation guidelines \(Figure[3](https://arxiv.org/html/2609.38630#S4.F3)\)\.Ont3keeps those 29 types and adds two optional\_referencelabels,person\_referenceandorganization\_reference, for expressions referring to people or organizations rather than naming them, plus subspan refinements such as family name within a person name\.

We fine\-tune XLM\-R\-large\([Conneau et al\., 2020](https://arxiv.org/html/2609.38630#bib.bib4)\)with an affine token\-classification head, using frontier annotations and coverage\-aware mapped human gold\. The decoder combines evidence from related entity types, and a fitted ±1\-character ranker repairs subword span boundaries \(Section[6](https://arxiv.org/html/2609.38630#S6)\)\.

We construct fresh evaluation data from previously unannotated text, spending more on annotation quality than for training, where the lower\-cost Luna models provide most of the additional supervision\. We also evaluate against human\-annotated corpora, mapping their label inventories to shared categories\. Their presence in incumbent training data is unknown\. Our checks of frontier annotations against human gold use exact\-span F1 and character\-redaction F1 with corpus\-specific many\-to\-many label mappings \(Figure[5](https://arxiv.org/html/2609.38630#.A1.F5)\)\. We regard frontier\-model agreement around 90% with human annotation as sufficient for this role\. For comparison, TAB’s multiple human annotations of 84 court\-case texts gave 85\.5% exact typed\-span agreement across nine mapped entity types \(Section[A\.15](https://arxiv.org/html/2609.38630#.A1.SS15)\)\. We screen our own data for lexical and semantic overlap across training and evaluation\.

The selected encoder,O4, uses the Ont3 inventory and additional training data from mapped human gold\. O3 is its predecessor using the same inventory; O2 uses Ont2\. On 1,283 human\-gold segments in seven languages, O4 reaches 88\.8 redaction F1 at its best measured operating point, against 69\.1 for published GLiNER2\([Zaratiana et al\., 2026](https://arxiv.org/html/2609.38630#bib.bib20)\)and 57\.3 for Microsoft Presidio 2\.2\.364 \(Figure[1](https://arxiv.org/html/2609.38630#S1.F1)\)\. GL4 is GLiNER2 adapted to the new training pool; it reaches 67\.8\. These comparisons allow differing annotation conventions and exempt types the published GLiNER2 cannot express \(Section[A\.9](https://arxiv.org/html/2609.38630#.A1.SS9)\)\.

Figure 1:Precision–recall on 1,283 human\-gold segments in seven languages \(top three rows\) and on Ont3 \(bottom row\): the 302\-segment seven\-language subset at left and all 1,201 segments over 35 languages at right\. Untyped redaction regions match at 80% overlap of both\. Auxiliary refinements such as name components are not scored; models are not penalized for lacking them\. GL and GL4: published and new\-data\-adapted GLiNER2; O2–O4: XLM\-R variants; PR: multilingual Presidio, a fixed point; PF: OpenMed multilingual\-v2 Privacy Filter\. GL has an unequal advantage: gold spans outside its label coverage are unscored for GL alone; Figure[13](https://arxiv.org/html/2609.38630#.A1.F13)compares GL and O4 on the same restricted task\. Across all five Privacy Filter variants, the best F1 stays below 40 on pooled human gold \(35\.8\) and full Ont3 \(39\.4\)\. Both axes start at 0\.4, with off\-scale Presidio points marked at the axis edge\. O4’s best measured F1 is 88\.8 on human gold and 82\.1 on Ont3\. Qualitatively similar exact span\-boundary\-required results appear in Section[A\.7](https://arxiv.org/html/2609.38630#.A1.SS7)\.
## 2Evaluation

The main comparison uses 1,283 human\-gold test segments in seven languages: English, German, Spanish, French, Arabic, Portuguese and Chinese \(Figure[1](https://arxiv.org/html/2609.38630#S1.F1)\)\. Its bottom row shows Ont3: 1,201 frontier\-annotated development segments across 35 languages, with the 302\-segment seven\-language subset at left and the full pool at right\. The pool includes text selected for rare entity types and has been reused during development; it is not an untouched terminal test\. We sum matched, predicted and gold region counts rather than averaging F1\.

Anoperating pointis the precision–recall tradeoff obtained at a chosen final confidence threshold\. We sweep an outside\-token \(O\) score bias for XLM\-R and the confidence threshold for GLiNER2\. Microsoft Presidio 2\.2\.364 is a fixed point at its default threshold, using the multilingual configuration described in Section[A\.10\.1](https://arxiv.org/html/2609.38630#.A1.SS10.SSS1)\. The best published OpenAI Privacy Filter model, OpenMed’s multilingual v2 fine\-tune, reaches 35\.8 F1 on human gold and 39\.4 on Ont3 over anObias sweep\.111On human gold, multilingual v2 dominates the original filter and both Nemotron fine\-tunes; multilingual v1 reaches at most 33\.0 F1\. Multilingual v2’s model card lists all seven languages\. The original filter intentionally omits organizations and general locations\. Its task is narrower than the full redaction inventory; Figure[12](https://arxiv.org/html/2609.38630#.A1.F12)gives its supported\-category comparison\.

O4 uses isolated sentences and 50% mapped human\-gold sampling \(Section[6](https://arxiv.org/html/2609.38630#S6)\)\. O2–O4 include character\-boundary refinement in all 35 languages\. Scoring excludes corpus\-unannotated types; name\-kind subspans are not scored\. Unless a score says\+\_reference, we do not penalize a person or organization prediction matching the corresponding gold reference label; it contributes neither a true nor a false positive\.222Missing an optional reference incurs no false negative\. Wrong entity families and boundary errors still count; Section[A\.9](https://arxiv.org/html/2609.38630#.A1.SS9)defines both this policy and\+\_reference\.To tolerate small boundary misses, a match must cover at least 80% of both redaction regions, with each region matched at most once\.

Section[A\.7](https://arxiv.org/html/2609.38630#.A1.SS7)separates the human\-gold and 35\-language Ont3 results and shows exact\-boundary comparisons; Section[A\.1](https://arxiv.org/html/2609.38630#.A1.SS1)specifies corpus mappings\. The Ont3 collection is reused development data, including GLiNER2 checkpoint selection; these curves do not establish untouched\-test generalization or isolate architecture from training exposure\.

## 3Adapting GLiNER2

To transfer published GLiNER2 to Ont3, we align training spans to its word tokenizer, retain familiar label spellings, and initialize new label tokens from related pretrained embeddings\. Brief fine\-tuning improves over this transferred initialization, but continuing training lowers redaction quality\. On the new annotation pool we save every 100 steps and select step 600 from eight checked stages\. This model, GL4, reaches 67\.8 best\-grid redaction F1 on human gold, compared with 69\.1 for the published model\. Its larger target inventory makes this a harder task: published GLiNER2 is exempt from gold types it cannot express\. Section[A\.10\.3](https://arxiv.org/html/2609.38630#.A1.SS10.SSS3)gives the matched initialization comparison and training curve\.

We do not replay the original training mixture\. Our search of the official paper, repository and model card found no exact public release of that mixture\. Missing replay is one possible explanation for the decline, not a tested cause\.

## 4Label alignment

Ont1 normalized label spelling across the available gold corpora and kept every distinction that was not clearly one\-to\-one as a separate label, giving 116 labels \(Figure[8](https://arxiv.org/html/2609.38630#.A1.F8)\)\. These labels were not disjoint subclasses: a clinician’s surname combines a person role with a name component\. We therefore reduced Ont1 to Ont2’s 29 primary types, with consistent annotation guidelines and with name components and roles moved to separate refinements\. Ont2 and Ont3 share 29 primary types in 12 semantic families; Ont3 adds only two optional reference labels \(Figure[3](https://arxiv.org/html/2609.38630#S4.F3)\)\. Corpora and models differ in their entity types and span boundaries\. For training, a source label can supervise a declared set of acceptable target labels\. For evaluation, we map all models’ predictions and the gold labels to the same categories and report exclusions\. For example, MAPA’s ADDRESS label permits locality, postal code and street address as targets, with one designated fallback \(Figure[5](https://arxiv.org/html/2609.38630#.A1.F5)\)\. The accepted interpretations must agree between training and evaluation: a label permitted by the training objective should not become an error solely because evaluation narrows that set\.

Component refinements identify subspans, such as the given name within a person’s full name\. Role refinements label the whole name span with contextual information, such as whether the person is a patient \(Figure[2](https://arxiv.org/html/2609.38630#S4.F2)\)\.

Ont3 annotations supervise type\-specific refinement heads separately from BIOES\. Whole\-span categorical heads, such as identifier kind, average token logits over the span; component categories and independent binary predicates receive token\-level supervision\. These refinements do not affect the reported model comparisons \(Figure[9](https://arxiv.org/html/2609.38630#.A1.F9)\)\.

Figure 2:Components locate subspans of a name; a role labels the whole name span \(illustrative\)\.Figure 3:Primary labels shared by Ont2 and Ont3, grouped by semantic family\. Dashed amber circles mark Ont3’s two optional reference labels; there are no Ont2\-only primary labels\. Identifier\-kind and name\-component refinements are in Figure[9](https://arxiv.org/html/2609.38630#.A1.F9)\.
## 5Token classification and decoding

Our model uses XLM\-R\-large\([Conneau et al\., 2020](https://arxiv.org/html/2609.38630#bib.bib4)\)and applies one affine layer to each token’s final encoder representation\. Its scores describe both entity type and position within a span:Beginning,Inside,Outside,End, orSingle token \(BIOES\)\. A constrained decoder combines these scores into labeled spans\.

Our rationale for a simple head is that full encoder fine\-tuning already provides trainable nonlinear depth for the task\. We screened seven alternatives to the final\-layer affine head: four layer\-concatenation variants, a nonlinear head, and two rank\-constrained heads \(XLM\-R heads use 10% dropout; GELU and neighboring\-state variants also use layer normalization\)\. The most promising concatenated the middle and final encoder layers, doubling the classifier’s parameters\. Its initial gain of 1\.6 validation F1 points became a 0\.3\-point loss on repeat\. We treated this apparent win among multiple alternatives as selection noise and retained the final\-layer affine head\.333The seven alternatives exclude exploratory runs; Section[A\.12\.3](https://arxiv.org/html/2609.38630#X.A1.SS12.SSS3.Px2)gives the full search and corrected comparison conditions\.Soft\-prompt and entity\-presence controls on a frozen encoder likewise did not improve on head\-only training \(Section[A\.18\.8](https://arxiv.org/html/2609.38630#.A1.SS18.SSS8)\)\.

### 5\.1Coarse–fine decoding

Constrained BIOES decoding selects the highest\-scoring legal sequence, requiring the same entity type throughout a span \([Lester et al\., 2020](https://aclanthology.org/2020.findings-emnlp.166/)\)\. Our class\-aware variant allows related fine types to support a shared boundary: a hospital name can receive alternating support for healthcare organization and generic organization\. For each boundary position, it combines scores within a compatible broad class using a temperature\-controlled log\-mean\-exp, normalized by the number of member types\. It then finds the highest\-scoring legal sequence over those broad classes, retaining the highest\-scoring fine member at each token\. Within each selected span, character\-weighted votes from those token labels determine its final fine type\. That type need never win as a complete, matching\-type BIOES path on its own\. For a fixed label inventory, decoding scales linearly with token count and reuses the encoder scores when calibration changes\.

For tokentt, boundary positionb∈\{B,I,E,S\}b\\in\\\{B,I,E,S\\\}, compatible fine\-type groupCC, and temperatureT\>0T\>0, the shared score is

Gt,b,C=T​log⁡\(1\|C\|​∑f∈Cexp⁡\(zt,b,fT\)\),G\_\{t,b,C\}=T\\log\\left\(\\frac\{1\}\{\|C\|\}\\sum\_\{f\\in C\}\\exp\\left\(\\frac\{z\_\{t,b,f\}\}\{T\}\\right\)\\right\),
wherezt,b,fz\_\{t,b,f\}is the encoder head’s logit for that boundary and type\. Relative to matching\-type Viterbi, coarse\-class decoding raises typed overlap F1 by 0\.3 points and soft class evidence by 0\.5\. Blending each label’s score with its class average \(α=0\.95\\alpha=0\.95\) under the usual BIOES decoding performs roughly equivalently \(timing table and the blend: Section[A\.12\.3](https://arxiv.org/html/2609.38630#X.A1.SS12.SSS3.Px1)\)\.

### 5\.2Character\-boundary refinement

A subword classifier can place span endpoints only at token boundaries, which need not coincide with the text that should be redacted\. For example, the token2\.can join an identifier’s final digit to sentence punctuation: redacting the digit while preserving the period requires an endpoint inside that token\. In languages with little whitespace, including Chinese and Japanese, subwords can also cross name\-component boundaries\. Only 92% of gold spans had both endpoints on the tokenizer’s boundaries in an early tokenizer audit \(Section[A\.14](https://arxiv.org/html/2609.38630#.A1.SS14)\)\.

To recover precise character spans without changing the encoder, we add a small linear ranker after BIOES decoding\. For each proposed endpoint it scores the current position and its two one\-character neighbors, using the predicted Ont3 entity type, endpoint side and nearby characters\. Its2182^\{18\}hashed weights occupy about 1 MB\. A dynamic program chooses endpoint pairs while preserving each span’s type, proposal count and uniqueness, and preventing new overlaps between previously disjoint spans\.

On O4, refinement raises exact typed\-span F1 from 74\.6 to 76\.3 on the 1,201\-segment Ont3 pool: \+1\.68 points \(paired 95% interval \[\+1\.01, \+2\.41\]\)\. Exact redaction\-region F1 rises from 77\.7 to 80\.2; on human gold it is essentially unchanged, 87\.87 to 87\.80\. This contrast holds the encoder, zero outside\-label bias and remaining postprocessors fixed\. Fitting and controls are in Section[A\.14](https://arxiv.org/html/2609.38630#.A1.SS14)\.

Word\-level pooling combines subword representations before labeling\([Ács et al\., 2021](https://arxiv.org/html/2609.38630#bib.bib21)\)\. Our refiner instead keeps the tagger’s subword sequence and learns character endpoints, including positions inside tokens\. A deterministic English word\-edge rule was competitive on the native development subset but less reliable on punctuation\-bearing identifiers and names \(Section[A\.14](https://arxiv.org/html/2609.38630#.A1.SS14)\)\.

## 6Ont3 training and gold replay

Ont2 was trained mainly onmappedhuman gold: annotations converted to Ont2 types through reviewed label mappings \(Figure[5](https://arxiv.org/html/2609.38630#.A1.F5)\)\. Its high scores on data from the training corpora did not carry over to fresh HUDOC legal text, which differed in annotation conventions and document type \(Section[A\.13](https://arxiv.org/html/2609.38630#.A1.SS13)\)\. For Ont3 we directly annotated general web text in 35 languages \(FineWeb and FineWeb2\), plus HUDOC court judgments and PubMed Central clinical case reports, using frontier LLMs\. The 1,201\-segment Ont3 evaluation collection is held out from that annotated distribution\.

Each annotation prompt pairs shared definitions of 31 entity types \(the 29 Ont2 types plusorganization\_referenceandperson\_reference\) with worked examples in the target language \(Figure[4](https://arxiv.org/html/2609.38630#S6.F4)\)\. The annotator marks exact spans in one sentence, using its source paragraph to resolve references \(an annotated example: Section[A\.2](https://arxiv.org/html/2609.38630#.A1.SS2)\)\.

Figure 4:Prompt assembly: fixed English instructions plus a demonstration in the target language \(French shown\)\. Ellipses omit other rules and annotations\.The annotator copies a surface and labels its occurrences; software locates them in the input and computes the span offsets\. This avoids asking the model to count characters or reproduce the caller’s offset convention\. The aligner checks the copied text and occurrence count\.

### 6\.1Annotation and mapped\-gold replay

Human annotations teach conventions that differ from those of frontier annotators\. We replay MAPA, AQMAR, OpenNER and Wojood with accepted\-label mappings, treating unmarked tokens as negatives only for types the corpus covers\. In a matched precursor experiment, 40% gold sampling raised human\-gold exact\-region F1 by ten points without reducing Ont3 region F1 \(Section[A\.12\.8](https://arxiv.org/html/2609.38630#.A1.SS12.SSS8)\)\.

O4 adds 49,812 unique sentence\-oriented training texts with 378,431 labeled XLM\-R subword tokens to O3’s non\-human pool\. The human\-gold membership is unchanged; its sampling share rises from 40% to 50%\. Repricing the recorded annotation attempts, including retries and rejected outputs, at current GPT\-6\-Luna rates gives about $53 \(Section[A\.17](https://arxiv.org/html/2609.38630#.A1.SS17)\)\.

Table 1:Both models use Ont3 in 35 languages and replay the same 80,099 unique human\-gold texts\. Counts deduplicate language and text before sampling; labeled tokens include optional references\. O2 uses a twenty\-language mapped\-gold recipe \(Table[12](https://arxiv.org/html/2609.38630#.A1.T12)\)\.RecipeUnique non\-human textsLabeled tokensHuman\-gold drawsInputO338,700477,68540%Neighboring sentencesO488,512856,11650%Isolated sentenceAt fixed zeroObias, O4 improves pooled fine typed Ont3 F1 from 74\.5 to 76\.3 over 1,201 segments: \+1\.79 points \(paired document\-bootstrap 95% interval \[\+0\.51, \+3\.13\]\)\. Human\-gold exact\-region F1 rises from 86\.5 to 87\.8 \(\+1\.25, interval \[\+0\.19, \+2\.34\]\)\. Redaction\-region gains on Ont3 are smaller: 79\.7 to 80\.2 \(\+0\.55, interval \[−0\.62, \+1\.74\]\)\.

### 6\.2Scope of the simple recipe

O4 keeps the affine head, legal BIOES decoding and character\-boundary ranker\. With the richer annotation pool, neighboring context, additional soft registers, supervision\-status prompts and extra redaction\-loss weight have not produced a consistent improvement\. The appendix groups these attempts by concept \(Section[A\.16\.1](https://arxiv.org/html/2609.38630#.A1.SS16.SSS1); Section[A\.18\.3](https://arxiv.org/html/2609.38630#.A1.SS18.SSS3); Section[A\.18\.8](https://arxiv.org/html/2609.38630#.A1.SS18.SSS8)\)\. We believe additional frontier annotation can improve performance further, including with XLM\-R, which outperformed Qwen3\.8\-Flash\-Next representations in our comparisons \(Section[A\.5](https://arxiv.org/html/2609.38630#.A1.SS5)\)\. Larger encoders may eventually be needed to benefit from more data\. We invite users to customize the ontology and annotate their own domains\.

## 7Encoder and LLM alternatives

XLM\-R\-large achieved the highest span\-detection F1 in our three\-encoder screen: with independently trained affine heads it exceeds LaBSE by 4\.4 six\-language F1 points and 2\.5 ten\-language clinical points, and mmBERT\-base with its stock head scores lower still\. Each system trained on the same 1,854 clinical and legal documents, with checkpoints selected without using the evaluation sets \(Section[A\.12\.4](https://arxiv.org/html/2609.38630#.A1.SS12.SSS4)\)\. The larger XLM\-R\-XL \(3\.5B parameters\), fully unfrozen, did not improve on XLM\-R\-large on Ont3 development text \(−0\.9 exact typed F1, 95% interval \[−2\.72, \+0\.87\]\), plausibly because it was pretrained on 0\.5T tokens against XLM\-R\-large’s 6T\([Goyal et al\., 2021](https://arxiv.org/html/2609.38630#bib.bib8)\)\. On O4’s full training pool, replacing mmBERT’s stock head with an affine head improves human\-gold exact\-region F1 by 2\.1 points \(95% interval \[0\.95, 3\.38\]\), although it remains below O4 on Ont3 \(Table[16](https://arxiv.org/html/2609.38630#.A1.T16)\)\.

Local LLMs fitting on a single 96\-GB GPU also fell short as frozen encoders\. Task instructions and a 512\-unit GELU head combining final\-layer states at token offsets \[−1, 0, \+1\] improved Qwen3\.8\-Flash\-Next\. On the same 659 Ont3 development segments, this recipe scores 59\.5 exact typed\-span F1 with Flash\-Next, 62\.4 with Gemma\-4\-31B base and 61\.1 instruction\-tuned, against O4’s 76\.9\. Even the best frozen head is 14\.5 points behind O4; its redaction\-character gap is smaller, 87\.1 against 91\.5, or 4\.5 points \(Section[A\.5](https://arxiv.org/html/2609.38630#.A1.SS5)\)\. Only the heads train in these LLM comparisons\.

Prompted directly, local models produced invalid annotations on 5–8% of the 659\-segment Ont3 subset\. Even rescuing those failures with O4, their character F1 remained 1\.4–5\.1 points below O4’s 91\.5\. Their estimated throughput was 2–15 labeled tokens/s versus the encoder’s measured 2,744—roughly two to three orders of magnitude slower \(hardware and timing details in Section[A\.4](https://arxiv.org/html/2609.38630#.A1.SS4)\)\.

## 8Incomplete teacher annotations

Ignoring unmarked tokens in sparse LLM annotations risks lowering precision: false positives there incur no labeling loss\. We denote their non\-entity \(O\) loss weight byλ\\lambda, ranging fromλ=0\\lambda=0\(ignored\) toλ=1\\lambda=1\(ordinary negative supervision\), balancing this risk against learning the annotator’s omissions\. Marked spans and fully annotated examples retain their usual supervision\. Separately, reducing the weight of knownOtargets to 0\.75 on fully annotated data improved Ont2 F1, primarily by shifting the precision–recall operating point \(Section[A\.11](https://arxiv.org/html/2609.38630#.A1.SS11)\); here we test whether unmarked tokens should be treated as negatives at all\.

We annotated 1,652 examples with Gemma\-4\-31B\-IT, whose annotations reach about 80% of Luna’s coverage on Ont3 text\. With these examples at 10% training sampling mass,λ=0\\lambda=0gave better span and character F1 thanλ∈\{0\.05,0\.20,1\}\\lambda\\in\\\{0\.05,0\.20,1\\\}: compared withλ=1\\lambda=1, it increased character recall by 2\.8 points and character F1 by 1\.5 points, while reducing precision by 0\.3 points, in a two\-seed contrast on the twenty\-language development set \(Section[A\.12\.7](https://arxiv.org/html/2609.38630#.A1.SS12.SSS7)\)\. Boundary conventions can also mimic omissions, so the objective accepts declared internal name splits or merges while retaining outer boundaries \(Section[A\.12\.7](https://arxiv.org/html/2609.38630#X.A1.SS12.SSS7.Px1)\)\.

The training data mix, the background \(O\) loss weight and an inference\-timeObias all shift the model’s propensity to label rather than withhold\. We favor theOloss weight: it compensates during learning for a predictable mismatch between training and deployment entity density without discarding useful examples, and it can change the learned decision function rather than uniformly shifting a fixed model’s scores\. TheObias needs no retraining, so we use it for cheap operating\-point sweeps\. Evidence is in Section[A\.11](https://arxiv.org/html/2609.38630#.A1.SS11)\.

## 9Deployment efficiency

To approach prompted frontier\-model annotation quality with inexpensive inference, we use an encoder that assigns token labels in one forward pass\. O4 labels each sentence in isolation\. Neighboring\-sentence context gives no consistent improvement under its annotation\-rich recipe \(Section[A\.16\.1](https://arxiv.org/html/2609.38630#.A1.SS16.SSS1)\)\.

Throughput counts tokens in the text being labeled, including outside\-entity text, with one common whitespace\-delimited convention\. Prompt examples, surrounding context and generated annotation tokens are excluded\.

We measured serving throughput for the XLM\-R\-large architecture and stock GLiNER2 on the same inputs under each configuration \(Table[2](https://arxiv.org/html/2609.38630#S9.T2); conditions in Section[A\.19](https://arxiv.org/html/2609.38630#.A1.SS19)\)\. The table also reports LLM encoding throughput for the BIOES\-head comparisons in Section[A\.5](https://arxiv.org/html/2609.38630#.A1.SS5)\.

Table 2:Throughput per labeled input token, excluding instructions and context\. CPU is EPYC 7R13 with 16 threads\. XLM\-R and GLiNER2 use shared workloads and document batches on GPU \(Section[A\.19](https://arxiv.org/html/2609.38630#.A1.SS19)\)\. The three LLM rows measure encoding for simple BIOES heads, not prompted annotation output generation \(Section[A\.5](https://arxiv.org/html/2609.38630#.A1.SS5)\)\.SystemHardwareInput tokens/sXLM\-RCPU675GLiNER2CPU138XLM\-RL40S2,744GLiNER2L40S1,405Qwen3\.8\-Flash\-NextRTX PRO 600029Gemma\-4\-31B baseRTX PRO 600093Gemma\-4\-31B\-ITRTX PRO 600093The encoder delivered 4\.9 times GLiNER2’s CPU throughput and about twice its GPU throughput in these configurations\.

## 10Related work

Our comparisons include GLiNER2\-PII, OpenAI Privacy Filter, OpenMed’s multilingual and Nemotron\-v2 privacy filters, and Presidio\. We score every system on all seven human\-gold languages; their claimed coverage is seven languages for GLiNER2\-PII\([Zaratiana et al\., 2026](https://arxiv.org/html/2609.38630#bib.bib20)\), primarily English for OpenAI Privacy Filter, and 16 and 20 languages for the two OpenMed variants, respectively\. Presidio comparisons use the appropriate language configuration\. We tune operating points where available and align label definitions\. GLiNER2 is the strongest published incumbent on our human\-gold evaluation and our recurring comparison across experiments\.

Prior work also motivates task\-specific encoder fine\-tuning\. PIIBench reports poor exact\-span transfer from existing systems, while a follow\-on study obtains stronger results with a DeBERTa token classifier trained for its labels\([Jha, 2026b](https://arxiv.org/html/2609.38630#bib.bib11);[Jha, 2026a](https://arxiv.org/html/2609.38630#bib.bib10)\)\. In a separate study, XLM\-R fine\-tuning surpasses the zero\-shot OpenAI Privacy Filter with roughly 100–1,000 examples, depending on domain, under a looser span\-overlap criterion\([Uppala, 2026](https://arxiv.org/html/2609.38630#bib.bib18)\)\.

Our approach builds on multilingual encoder pretraining\([Conneau et al\., 2020](https://arxiv.org/html/2609.38630#bib.bib4)\), translation\-based annotation transfer\([Chen et al\., 2023](https://arxiv.org/html/2609.38630#bib.bib3);[García\-Ferrero et al\., 2023](https://arxiv.org/html/2609.38630#bib.bib6)\), and learning from incomplete annotations\([Mayhew et al\., 2019](https://arxiv.org/html/2609.38630#bib.bib14);[Effland and Collins, 2021](https://arxiv.org/html/2609.38630#bib.bib5)\)\. FiNERweb provides a precedent for multilingual annotation by language models\([Golde et al\., 2026](https://arxiv.org/html/2609.38630#bib.bib7)\)\. Our masked\-language\-model experiments draw on domain\-adaptive pretraining and efforts to preserve cross\-lingual capability during continued training\([Gururangan et al\., 2020](https://arxiv.org/html/2609.38630#bib.bib9);[Liu et al\., 2021](https://arxiv.org/html/2609.38630#bib.bib12)\)\.

## 11Open questions

Encoder capacity\.A fully unfrozen XLM\-R\-XL did not improve on XLM\-R\-large, but it saw far fewer pretraining tokens\. Larger, differently pretrained or newer encoders, especially with more annotated data, may still raise the ceiling\.

LLM representations\.Head and prompt changes improved taggers on frozen LLM states, but the best, on Gemma\-4\-31B base, was slower than the XLM\-R BIOES tagger and 4\.5 character F1 worse \(87\.1 against 91\.5; Section[A\.5](https://arxiv.org/html/2609.38630#.A1.SS5)\)\. We did not exhaust head and prompt choices; additional LLM layers as input and continued LLM fine\-tuning remain untested\.

Richer heads and labels\.An affine BIOES head reading only the current token state, decoded with Viterbi, sufficed here, and deeper heads or added encoder layers did not help\. Joint span scoring as in GLiNER2 could still add information\. Can reliable subtype labels improve coarse redaction beyond what a larger head learns from coarse labels alone?

Cheaper supervision\.At a fixed investment in clean Ont3 annotation, could noisier data, or data labeled under other ontologies, admitted through partial supervision and label mappings, do better? Mapped\-gold replay, which adds human gold labeled under other corpora’s schemes, already helps \(Section[6](https://arxiv.org/html/2609.38630#S6)\)\.

Per\-language capacity\.If a more diverse training mixture lowers quality on important languages, can automatically splitting tagging heads or selected encoder parameters recover it while keeping broad coverage?

Cross\-locale names\.Does a shared multilingual head still find a name that is atypical of the surrounding text’s language or locale? Rerunning unchanged sentences with names from other locales would test this directly\.

Machine translation\.Translating human\-annotated data helped extend coverage to 35 languages \(Section[A\.12\.6](https://arxiv.org/html/2609.38630#.A1.SS12.SSS6)\), but mismatched entity names across locales remained a quality constraint \(Section[A\.12\.5](https://arxiv.org/html/2609.38630#.A1.SS12.SSS5)\)\. We found no easy remedy: achieving adequate translation and entity quality cost about as much as annotating native text with language\-specific prompt demonstrations, which avoids this mismatch\. For which other tasks is cheap, high\-quality translation of annotated data sufficient to extend a multilingual encoder across languages? Cheap, high\-quality native annotation may leave less benefit for more elaborate methods\.

## 12Conclusion

Task\-matched supervision from prompted frontier models, replayed with coverage\-aware mapped human gold, trains an XLM\-R\-large encoder with an affine BIOES head \(span endpoints adjusted by ±1 character to repair subword granularity errors\) that reaches 88\.8 redaction F1 on human gold against 69\.1 for published GLiNER2\. Additional annotations and mapped\-gold replay improve the fine typed task, while gains on already strong redaction scores are smaller\. The encoder architecture delivers 4\.9 times GLiNER2’s CPU throughput\.

## 13Limitations

Perfect agreement with gold span labels does not guarantee that outside context cannot identify someone or that redacted text remains useful\([Pilán et al\., 2022](https://arxiv.org/html/2609.38630#bib.bib16);[Brynjólfsson et al\., 2026](https://arxiv.org/html/2609.38630#bib.bib2)\)\. Annotation variation also affects exact\-boundary scores\. TAB happens to provide multiple human annotations of 84 identical court\-case texts: agreement is 85\.5% exact typed\-span F1 across nine mapped entity types \(Section[A\.15](https://arxiv.org/html/2609.38630#.A1.SS15)\)\.

We release code, prompts and training recipes, with data\-acquisition scripts and source links, but not training data or weights\. Access conditions and training costs are in Section[A\.3](https://arxiv.org/html/2609.38630#.A1.SS3)\.

## 14Ethical considerations

Dual use\.A tagger that finds names, identifiers, contact details and locations in 35 languages supports privacy protection and equally supports targeted surveillance or profiling of the same text\. We release training recipes and pointers to the training data \(Section[A\.3](https://arxiv.org/html/2609.38630#.A1.SS3)\); the capability itself is not specific to this work and is already available in the published models we compare against\.

Annotation\-service exposure\.Our training supervision comes from prompted frontier language models hosted by external providers\. Annotating confidential text this way exposes that text to the provider’s session\-data handling, and a redaction project cannot assume the provider’s use of session text is itself acceptable to the data owner\. In this work we annotated only text that was already public\. Customizing on confidential data first requires an annotator permissible under laws and agreements\. Public annotated corpora are also not a stand\-in for such data: the clinical corpora we use either document manual placeholder or pseudonym substitutions or show signs of them\. Their surface forms and formats therefore differ from unredacted records\.

Fine\-tuning the XLM\-R BIOES tagger locally is cheap and fast: a full Ont3 fit takes about 2\.3 hours on one GPU, roughly $5 of rented time \(Section[A\.3\.1](https://arxiv.org/html/2609.38630#.A1.SS3.SSS1)\)\. The cost of adapting to new data is essentially the cost of annotating it, together with that annotation’s confidentiality exposure\. A locally run annotator on an isolated machine removes that exposure\. The local models we tested produced useful but clearly weaker supervision than the frontier annotator \(Table[6](https://arxiv.org/html/2609.38630#.A1.T6); the partial\-supervision teacher in Section[A\.12\.7](https://arxiv.org/html/2609.38630#.A1.SS12.SSS7)\), and we expect that gap to narrow as local models improve\.

Transfer\.The reported scores describe the evaluated languages, domains and annotation conventions\. No representation should be made that this or any model performs comparably on languages, domains, formats or entity conventions it was not trained and evaluated on; our own history includes a high development score followed by a large drop on fresh legal text of a different kind \(Section[A\.13](https://arxiv.org/html/2609.38630#.A1.SS13)\)\. A deployment in a new setting needs its own evaluation; where that evaluation falls short, annotating in\-domain text and refitting is inexpensive\.

Mandated de\-identification\.Regulations fix which identifiers must be removed, and often for whom: the HIPAA Safe Harbor standard enumerates identifier categories of protected health information, the GDPR distinguishes pseudonymised from anonymised data, and the same record may need different treatment for a treating clinician, an insurer or a researcher\. Which mentions count can also depend on the role of the person named, for example a patient’s name as opposed to a clinician’s, a distinction the clinical corpora used here already annotate\. Meeting such a mandate with one model may require human review, or an operating point that over\-redacts until the text is no longer usable\. Text that can no longer be interpreted correctly is itself a high\-stakes harm, for example when redacted case records go to an outsourced claims processor\. Rather than a global penalty on theOlabel, which trades precision on every type for recall on the mandated ones, we suggest using the ontology\-migration flow demonstrated here \(Section[A\.12\.1](https://arxiv.org/html/2609.38630#.A1.SS12.SSS1)\) to introduce the mandated role subtypes as explicit labels, then raising only those labels’ scores at decoding, so that recall increases where the mandate requires it and nowhere else\.

## Acknowledgements

We thank the creators and maintainers of Nemotron\-PII, OpenPII, TAB, idner\-news\-2k, Wojood, MEDDOCAN, SPY, MultiGraSCCo, MAPA, KLUE, HiNER, AQMAR, OpenNER, FineWeb and FineWeb2, and the providers of HUDOC judgments and PubMed Central articles\. These resources supported training or evaluation; Table[4](https://arxiv.org/html/2609.38630#.A1.T4)in Reproduction and release lists the sources and access conditions\.

Generative AI assistants contributed to code development, figure preparation, and revision throughout the manuscript and appendices\. Code contributions are tracked throughContributing\-modelentries in the development history\. The author directed, supervised, reviewed, and revised the contributions, and retains final responsibility \(models used to produce annotations are described separately\)\.

## References

- Ai4Privacy \(2026\)Ai4Privacy\. 2026\.[OpenPII 1M: Multilingual PII masking dataset \(19 labels, 23 languages\)](https://doi.org/10.57967/hf/8202)\.Hugging Face dataset\.
- Brynjólfsson et al\. \(2026\)Sean Brynjólfsson, Shashvat Jayakrishnan, Esha Sali, Diptanshu Purwar, and Madhav Aggarwal\. 2026\.[RedactionBench](https://arxiv.org/abs/2606.18782)\.*Preprint*, arXiv:2606\.18782\.
- Chen et al\. \(2023\)Yang Chen, Chao Jiang, Alan Ritter, and Wei Xu\. 2023\.[Frustratingly easy label projection for cross\-lingual transfer](https://doi.org/10.18653/v1/2023.findings-acl.357)\.In*Findings of the Association for Computational Linguistics: ACL 2023*, pages 5775–5796\.
- Conneau et al\. \(2020\)Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov\. 2020\.[Unsupervised cross\-lingual representation learning at scale](https://doi.org/10.18653/v1/2020.acl-main.747)\.In*Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 8440–8451\.
- Effland and Collins \(2021\)Thomas Effland and Michael Collins\. 2021\.[Partially supervised named entity recognition via the expected entity ratio loss](https://doi.org/10.1162/tacl_a_00429)\.*Transactions of the Association for Computational Linguistics*, 9:1320–1335\.
- García\-Ferrero et al\. \(2023\)Iker García\-Ferrero, Rodrigo Agerri, and German Rigau\. 2023\.[T\-projection: High quality annotation projection for sequence labeling tasks](https://doi.org/10.18653/v1/2023.findings-emnlp.1015)\.In*Findings of the Association for Computational Linguistics: EMNLP 2023*, pages 15203–15217\.
- Golde et al\. \(2026\)Jonas Golde, Patrick Haller, and Alan Akbik\. 2026\.[FiNERweb: Datasets and artifacts for scalable multilingual named entity recognition](https://doi.org/10.18653/v1/2026.findings-eacl.121)\.In*Findings of the Association for Computational Linguistics: EACL 2026*, pages 2281–2300\.
- Goyal et al\. \(2021\)Naman Goyal, Jingfei Du, Myle Ott, Giri Anantharaman, and Alexis Conneau\. 2021\.[Larger\-scale transformers for multilingual masked language modeling](https://aclanthology.org/2021.repl4nlp-1.4/)\.In*Proceedings of the 6th Workshop on Representation Learning for NLP \(RepL4NLP\-2021\)*, pages 29–33\.
- Gururangan et al\. \(2020\)Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A\. Smith\. 2020\.[Don’t stop pretraining: Adapt language models to domains and tasks](https://doi.org/10.18653/v1/2020.acl-main.740)\.In*Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 8342–8360\.
- Jha \(2026a\)Pritesh Jha\. 2026a\.[Fine\-tuning over architectural complexity: Broad\-coverage PII detection on PIIBench with DeBERTa](https://arxiv.org/abs/2605.25816)\.*Preprint*, arXiv:2605\.25816\.
- Jha \(2026b\)Pritesh Jha\. 2026b\.[PIIBench: A unified multi\-source benchmark corpus for personally identifiable information detection](https://arxiv.org/abs/2604.15776)\.*Preprint*, arXiv:2604\.15776\.
- Liu et al\. \(2021\)Zihan Liu, Genta Indra Winata, Andrea Madotto, and Pascale Fung\. 2021\.[Preserving cross\-linguality of pre\-trained models via continual learning](https://doi.org/10.18653/v1/2021.repl4nlp-1.8)\.In*Proceedings of the 6th Workshop on Representation Learning for NLP*, pages 64–71\.
- Marimon et al\. \(2019\)Montserrat Marimon, Aitor Gonzalez\-Agirre, Ander Intxaurrondo, Heidy Rodriguez, Jose Lopez Martin, Marta Villegas, and Martin Krallinger\. 2019\.[Automatic de\-identification of medical texts in spanish: The MEDDOCAN track, corpus, guidelines, methods and evaluation of results](https://ceur-ws.org/Vol-2421/MEDDOCAN_overview.pdf)\.In*Proceedings of the Iberian Languages Evaluation Forum*, volume 2421 of*CEUR Workshop Proceedings*\.
- Mayhew et al\. \(2019\)Stephen Mayhew, Snigdha Chaturvedi, Chen\-Tse Tsai, and Dan Roth\. 2019\.[Named entity recognition with partially annotated training data](https://doi.org/10.18653/v1/K19-1060)\.In*Proceedings of the 23rd Conference on Computational Natural Language Learning*, pages 645–655\.
- Mohit et al\. \(2012\)Behrang Mohit, Nathan Schneider, Rishav Bhowmick, Kemal Oflazer, and Noah A\. Smith\. 2012\.[Recall\-oriented learning of named entities in Arabic Wikipedia](https://aclanthology.org/E12-1017/)\.In*Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics*, pages 162–173\. Association for Computational Linguistics\.
- Pilán et al\. \(2022\)Ildikó Pilán, Pierre Lison, Lilja Øvrelid, Anthi Papadopoulou, David Sánchez, and Montserrat Batet\. 2022\.[The text anonymization benchmark \(TAB\): A dedicated corpus and evaluation framework for text anonymization](https://doi.org/10.1162/coli_a_00458)\.*Computational Linguistics*, 48\(4\):1053–1101\.
- Roller et al\. \(2026\)Roland Roller, Vera Czehmann, Derya Erman, Luke Flanagan, Ibrahim Baroud, Frédéric Blain, Viviana Cotik, Eletta Giusto, Akhil Juneja, Mariana Neves, Maria Słowińska, Christine Hovhannisyan, Aaron Louis Eidt, Lisa Raithel, Sebastian Möller, and Maija Poikela\. 2026\.[DialogPII: A multilingual dataset of synthetic dialog transcripts to detect personal information](https://arxiv.org/abs/2606.30312)\.*Preprint*, arXiv:2606\.30312\.
- Uppala \(2026\)Rohith Uppala\. 2026\.[OpenAI Privacy Filter: A cross\-lingual, cross\-domain PII evaluation across 32 benchmarks](https://arxiv.org/abs/2608.02616)\.*Preprint*, arXiv:2608\.02616\.
- Vats et al\. \(2026\)Guneesh Vats, Anubha Agrawal, Shikha Singhal, Ajita Dash, Praison Selvaraj, Vidhan Jhawar, Ranga Prasad Chenna, and Bharadwaj Y M G\. 2026\.[REDACT: A systematically controlled multilingual benchmark for personal information detection](https://arxiv.org/abs/2606.19881)\.*Preprint*, arXiv:2606\.19881\.
- Zaratiana et al\. \(2026\)Urchade Zaratiana, Ash Lewis, and George Hurn\-Maloney\. 2026\.[GLiNER2\-PII: A multilingual model for personally identifiable information extraction](https://arxiv.org/abs/2605.09973)\.*Preprint*, arXiv:2605\.09973\.
- Ács et al\. \(2021\)Judit Ács, Ákos Kádár, and András Kornai\. 2021\.[Subword pooling makes a difference](https://doi.org/10.18653/v1/2021.eacl-main.194)\.In*Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume*\.

## Appendix AAppendices

### A\.1Gold coverage and mappings

The human\-gold corpora annotate different subsets of the redaction task\. We therefore exclude predictions outside each corpus’s annotated categories before comparing untyped redaction regions\. GLiNER2 \(GL\) receives 43 requested labels; O2, O3 and O4 predict the same 29 primary types\. All are evaluated against the same corpus\-specific coverage, including acceptable alternatives for coarse gold labels \(Table[3](https://arxiv.org/html/2609.38630#.A1.T3)\)\.

Table 3:Annotation coverage of the frozen human\-gold test subset\. Counts refer to original annotated spans, before merging redaction regions\. Source\-label counts describe the evaluated schema, not every label in the original corpus\.Human\-gold componentInputsSource labels used by the mappingAccepted primary typesGold spans with multiple accepted types / annotated spansMAPA18961359 / 126AQMAR1343882 / 173OpenNER collection90438350 / 999Wojood subset56111452 / 99For example, a coarse location label accepts six location\-related primary types; MAPA dates accept either date or date of birth, and amounts accept quantity or monetary amount\. These alternatives preserve the source’s uncertainty and award no additional type\-match credit in this untyped comparison\. Replacing GLiNER2’s multi\-target prediction mappings with their single designated targets changes no scored counts at any plotted threshold on these human\-gold inputs, so this mapping flexibility does not explain the gap between GL and O2–O4\. Coverage masking does matter: in a corpus that annotates only people, organizations and locations, an unmarked date is no evidence of a false positive\. Mappings do not repair omitted annotations, incorrect extents or disagreements about which mentions should be annotated\.

Figure 5:Broad source labels can supervise several target types: MAPA’s ADDRESS permits locality, postal code and street address, among others\. Filled squares mark fallback labels; open circles mark other acceptable labels\. Shown: sixteen selected mappings from five corpora to the Ont2 label inventory\.
### A\.2Frontier annotation prompts

Each annotation prompt pairs shared definitions of the 31 Ont3 entity types with worked examples in the target language \(Figure[4](https://arxiv.org/html/2609.38630#S6.F4)\)\. Given NFKC\-normalized input, the annotating model returns each distinct surface once, with one tag set per occurrence of that surface in reading order \(\["O"\]for an occurrence that is not an entity\)\. The aligner locates the occurrences, rejects any count mismatch and emits character spans for non\-Oentries\. Neither this copy\-and\-count offset design nor the use of language\-specific rather than shared demonstrations has been isolated as a cause of annotation quality\. Figure[6](https://arxiv.org/html/2609.38630#.A1.F6)shows one Hebrew result from Luna \(GPT\-5\.6\-Luna, the training\-data annotator\) in which the same spelling is a person reference in one occurrence and an ordinary adjective in the next\.

Input

hebrew

בנוסף לכל זה מצאו הבלשים במקום בו לן החשוד חומר החשוד כסם מסוג מריחואנה וציוד המשמש לגידול הסמים\.

*English gloss:*The detectives also found, wherethe suspectstayed, a substancesuspectedto be marijuana and equipment for growing drugs\.

Luna output \(raw\)

\[\{"t":"הבלשים","types":\[\["person\_reference"\]\]\},

\{"t":"מקום","types":\[\["location"\]\]\},

\{"t":"החשוד","types":\[\["person\_reference"\],\["O"\]\]\}\]

Aligned spans \(checked\)

```
[18,24) person_reference  הבלשים
[26,30) location          מקום
[37,42) person_reference  החשוד
[48,53) O, no span        החשוד
```

Figure 6:Luna’s direct output, with line breaks added, and the aligner’s result\. The same Hebrew spelling, החשוד, means “the suspect” in its first occurrence and “suspected” in its second; its two tag sets label the person reference and leave the adjective untagged \(O\)\. Both occurrences were found, the count matched, and the aligner applied no surface repairs\.
### A\.3Reproduction and release

We will release code, prompts and training recipes, including download and preparation scripts and pinned dependencies\. We will not release training data or model weights\. Restricted sources can be excluded; changing the mixture may change the results\.

Once annotations are available, estimated training compute is $9–18 for Ont2\-equivalent exposure and $4–5 for one Ont3 fit \(Section[A\.3\.1](https://arxiv.org/html/2609.38630#.A1.SS3.SSS1)\)\. Obtaining and reviewing annotations is a separate cost\.

Table[4](https://arxiv.org/html/2609.38630#.A1.T4)lists source terms checked on 22 September 2026\. Links lead to distributions or access instructions\. Terms refer to the versions used here; component and source\-text rights still apply\. The table includes evaluation and diagnostic sources as well as training data\.

Table 4:Data sources and reuse conditions for the acquired versions\.ResourceTerms and access[Nemotron\-PII](https://huggingface.co/datasets/nvidia/Nemotron-PII),[OpenPII 1M](https://huggingface.co/datasets/ai4privacy/pii-masking-openpii-1m)and[1\.5M](https://huggingface.co/datasets/ai4privacy/pii-masking-openpii-1.5m)CC BY 4\.0; public downloads\.[Ai4Privacy 200k](https://huggingface.co/datasets/ai4privacy/pii-masking-200k/blob/691f4fc6b099563cecdc80c6d0444922229d23a4/LICENSE)Custom terms; for\-profit organizations with more than three staff need a corporate license\. Redistribution restrictions apply\.[Ai4Privacy health\-PHI 400k](https://huggingface.co/datasets/ai4privacy/pii-masking-health-phi-400k)diagnostic sampleGated access under custom commercial terms\.[TAB](https://github.com/NorskRegnesentral/text-anonymization-benchmark),[idner\-news\-2k](https://github.com/khairunnisaor/idner-news-2k)and[Wojood public sample](https://github.com/SinaLab/ArabicNER)MIT; public repositories\. Full Wojood is separate and was not used\.[MEDDOCAN](https://huggingface.co/datasets/bigbio/meddocan),[SPY](https://huggingface.co/datasets/mks-logic/SPY),[MultiGraSCCo](https://zenodo.org/records/19489040)and[MAPA EUR\-Lex](https://huggingface.co/datasets/joelniklaus/mapa)CC BY 4\.0; public distributions\. Other MAPA packages may differ\.[KLUE NER](https://huggingface.co/datasets/klue/klue)and[HiNER](https://github.com/cfiltnlp/HiNER)CC BY\-SA 4\.0; public distributions\.[AQMAR through OpenNER](https://huggingface.co/datasets/bltlab/open-ner-core-types)\([Mohit et al\., 2012](https://arxiv.org/html/2609.38630#bib.bib15)\)Citation required; underlying Wikipedia rights apply\. Unresolved permissions require author clarification\.[OpenNER seven\-corpus subset](https://huggingface.co/datasets/bltlab/open-ner-core-types)Collection: CC BY 4\.0\. AnCora and GermEval: CC BY 4\.0; GSD and the four UNER components: CC BY\-SA 4\.0\.[FineWeb](https://huggingface.co/datasets/HuggingFaceFW/fineweb)/[FineWeb2](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2)ODC\-By 1\.0; Common Crawl and underlying web\-content terms also apply\.[HUDOC judgments](https://www.echr.coe.int/copyright-and-disclaimer)ECHR reproduction conditions and acknowledgment required; translations may have separate terms\.[PMC case reports](https://pmc.ncbi.nlm.nih.gov/tools/openftlist/)Article\-specific licenses; use NCBI\-approved retrieval services and exclude incompatible articles\.Authored, translated and teacher\-labeled recordsSource\-text and model\-service terms apply\.In the 1,000\-row Ai4Privacy health\-PHI diagnostic sample, we found inconsistent substituted values and untranslated English text under other language codes\. These findings concern that sample, not all Ai4Privacy releases\. Separately, Final20 briefly contained 21 Ai4Privacy 200k rows; excluding them left 6,979 records \(Section[A\.8](https://arxiv.org/html/2609.38630#.A1.SS8)\)\.

Before admitting additional data, we screen for overlap using lexical matching and embedding similarity\.

#### A\.3\.1Training\-only cost

Table[5](https://arxiv.org/html/2609.38630#.A1.T5)estimates instance rental once annotations are available\. Both fits use 64 sampled windows per update, including repeated draws\. We group sampled examples of similar token length into physical batches to reduce padding, while retaining the requested sampling mixture\. Thus these costs reflect length\-matched batching, not arbitrary batches padded to the longest example\.

Table 5:Training\-only estimates at AWS Linux Spot prices observed on 22 September 2026: $1\.41–3\.00/hour in us\-west\-2 for Ont2 and $1\.87–2\.37/hour in ap\-south\-1 for Ont3\.Training scopeUpdatesGPU / AWS instanceHoursEstimated $Ont2\-equivalent exposure33,300L40S / g6e\.4xlarge6\.0\*8\.5–18\.0\*Ont3 fit from pretrained XLM\-R16,000RTX PRO 6000 Blackwell Server / g7e\.2xlarge2\.34\.3–5\.4\* Ont2 extrapolates an 8,000\-update, 5,194\-second fit to the 33,300 retained updates across its training stages\. Ont3 uses the measured full 16,000\-update run on 47,641 windows with previous and next context\. These estimates exclude annotation, data preparation, experimental search, auxiliary\-model training and non\-training infrastructure costs\.

### A\.4Prompted local LLM annotators

We evaluated prompted Gemma and Qwen models as locally runnable alternatives to the frontier annotator\.

On a 659\-segment Ont3 subset across 35 languages, we checked each model’s final answer for valid annotation structure and literal source alignment before scoring\. Failures route the entire segment to O4; valid empty annotations remain unchanged\. Character scores pool occupied character positions across segments using the Ont3 annotations and the optional\-reference policy \(Section[A\.9](https://arxiv.org/html/2609.38630#.A1.SS9)\)\. Prompt versions and retry budgets differ: Gemma\-12B allowed four retries, Gemma\-31B one, and Qwen none\. All requests disabled thinking\.

Table 6:Annotation quality on the 659\-segment Ont3 subset\. Asterisks mark XLM\-R substitution for invalid annotations; without fallback, invalid outputs count as empty predictions\. Rescued subsets contain 33, 41 and 54 segments, respectively\. O4 uses zeroObias, isolated input and boundary refinement\.ModelFailed format \(%\)Character F1With XLM\-R fallback\*Rescued subset F1\*Gemma\-4\-12B5\.0183\.8486\.38\*84\.51\*Gemma\-4\-31B\-FP86\.2283\.9890\.11\*85\.13\*Qwen3\.8\-Flash\-Next\-NVFP48\.1976\.0387\.02\*90\.77\*O4, sentence only—91\.53——Failed format includes invalid labels, nonliteral spans, ordering and forbidden overlap\. Routing uses only the response and input text\.

The Gemma prompts include surrounding paragraph context; Qwen receives the preceding and following source sentences\. The matched XLM\-R context comparisons appear in Section[A\.16\.1](https://arxiv.org/html/2609.38630#.A1.SS16.SSS1)\.

Table 7:Gemma\-4\-31B\-IT serial generation measured on a 96\-GB RTX PRO 6000: 35 inputs in 35 languages, repeated twice, 591 labeled input tokens per pass\. Other prompted rates divide labeled input tokens by summed request durations, including retries\. The encoder rate is measured throughput \(Table[2](https://arxiv.org/html/2609.38630#S9.T2)\)\. Parameter counts describe nominal backbones; the active count is shown for Flash\-Next’s sparse main model, per token \(Table[8](https://arxiv.org/html/2609.38630#.A1.T8)\)\.SystemParams \(B\)Active \(B\)Labeled input tokens/sGemma\-4\-12B12—2Gemma\-4\-31B\-IT, bfloat1630\.7—2\.5Qwen3\.8\-Flash\-Next\-NVFP4176615XLM\-R encoder0\.55—2,744The serial Gemma check replays the saved annotation prompts through Transformers in bfloat16\. Each pass took 241 seconds, including full prompt prefill and output generation, excluding model loading, tokenization and validation; requests reuse only their own KV state\. This timing run uses no retries and does not replace the FP8 quality evaluation above\.

Adding estimated encoder time for the failed segments \(their token counts divided by the encoder’s measured 2,744 tokens/s\) leaves these rounded rates unchanged for the original evaluated configurations\.

### A\.5Frozen language\-model token heads

These experiments cached hidden states and trained a BIOES classification head; they did not generate annotation text\. On 1,113 reused development sentences, rescored on the shared revised annotations, scores use language\-weighted typed character F1 and optional references\. In the matched Gemma\-4\-12B comparison, adding the task instructions before each sentence raised affine\-head F1 from 26\.9 to 59\.0, using the same 10,427 training rows\. These were ordinary text instructions, distinct from the learned soft\-prompt vectors in Section[A\.18\.8](https://arxiv.org/html/2609.38630#.A1.SS18.SSS8)\.

The best result in that comparison, 75\.8, used Gemma\-4\-31B layer\-40 states, a 3,680\-token instruction prefix, no preceding sentence, and a nonlinear head with 512 hidden units\. The fully fine\-tuned XLM\-R\-large control averaged 83\.2 over five seeds \(range 82\.9–83\.8, standard deviation 0\.38\)\.

On the Ont3 annotation mixture, task instructions also improve frozen Qwen3\.8\-Flash\-Next representations\. A nonlinear head and neighboring token states provide further gains \(Table[8](https://arxiv.org/html/2609.38630#.A1.T8)\)\. Preceding and following sentences are placed before the annotation target sentence in the causal model’s input; the head predicts only target\-token labels\. Adding neighboring states to the nonlinear head improves exact\-span F1 by 5\.7 points \(paired document\-bootstrap 95% interval \[3\.53, 7\.94\]; no such benefit was established for XLM\-R\)\.

Table 8:Flash\-Next, Gemma and XLM\-R heads on 659 Ont3 development segments from 468 source documents\. Scores pool counts with optional references and zeroObias\. Offsets are relative to the target token; 0 uses only its own state\. LLM rows use the annotation prompt unless marked unprompted; XLM\-R uses no prompt\. Parameter counts are nominal backbone sizes in billions, excluding the BIOES head\. Flash\-Next has 125B main\-model parameters \(6B active per token\) plus 51B n\-gram embeddings; its unused 4B multi\-token prediction module is excluded\. Flash and Gemma heads use the same 32,821 training targets, 279\-target selector and annotation prompt; the fit ceiling is ten epochs of 250 batches of 4,096 tokens\. LLM heads train in about 15 minutes or less on frozen, already\-computed encodings\. The first three XLM\-R heads receive 1,000 steps on a 40,303\-window pool with the encoder frozen and the same selector\. O4 is the fully fine\-tuned serving model\.\(Prompted\) LLM / encoderParams \(B\)HeadInput state offsetsTyped span F1 \(%\)Redaction\-character F1 \(%\)Flash\-Next unprompted176Affine039\.1478\.59Qwen3\.8\-Flash\-Next176Affine044\.5680\.64Qwen3\.8\-Flash\-Next176Affine−1, 0, \+153\.0082\.96Qwen3\.8\-Flash\-Next176512\-unit GELU053\.8085\.25Qwen3\.8\-Flash\-Next176512\-unit GELU−1, 0, \+159\.5486\.61Gemma\-4\-31B base30\.7512\-unit GELU−1, 0, \+162\.4187\.07Gemma\-4\-31B IT30\.7512\-unit GELU−1, 0, \+161\.1286\.16XLM\-R0\.55Affine072\.3691\.47XLM\-R0\.55512\-unit GELU072\.4491\.48XLM\-R0\.55Affine−1, 0, \+172\.1991\.40XLM\-R O40\.55Affine076\.9091\.53Under the matched prompt and head recipe, Gemma base improves exact typed\-span F1 over Flash\-Next by 2\.86 points \(95% interval \[0\.41, 5\.32\]; character redaction \+0\.46 \[−\.72, 1\.68\]\)\. Instruction tuning, thought to improve performance of prompted tasks, did not: the IT head changes typed F1 by −1\.29 points \[−3\.67, \+1\.10\] and character F1 by −0\.92 \[−2\.10, \+0\.31\]\. The frozen LLM heads use feature standardization followed by two affine layers with GELU, without LayerNorm or dropout \(adding LayerNorm and dropout 0\.1 gave no resolved improvement: typed F1 changed by −0\.23 points for Flash\-Next and −0\.30 for Gemma base\)\.

On a 96\-GB RTX PRO 6000 Blackwell, Flash\-Next representation capture for these 659 segments took 455 seconds: 29 labeled input tokens/s over 13,193 tokens\. Requests ran one at a time with instructions and neighboring context; the elapsed time includes preparation, representation transfer and cache writes, and excludes server model loading and head training\. This is about 95 times slower than the XLM\-R reference of 2,744 tokens/s \(Table[2](https://arxiv.org/html/2609.38630#S9.T2)\), with different hardware and batching\.

Gemma\-4\-31B base and IT each took 142 seconds to encode the same 659 targets at 93 labeled tokens/s on this GPU\. Both use the same task instructions and neighboring context, bfloat16 inference and cached instruction prefixes\. Timing includes prefix prefill, forward passes, representation transfer and cache writes, excluding model loading\. Gemma uses Transformers while Flash uses vLLM\.

Using*only*the previous or next token state scores 30\.9% and 22\.7% exact typed span F1 with an affine head, below the current\-state control\. Together with an audit of token alignment, this favors combining complementary states rather than correcting a uniform one\-token offset\.

These comparisons reuse development data and condition on selected heads and one representation cache\.

Figure[7](https://arxiv.org/html/2609.38630#.A1.F7)compares four frozen\-LLM BIOES heads with the three prompted annotators of Section[A\.4](https://arxiv.org/html/2609.38630#.A1.SS4)and our selected O4 model\. The frozen LLM heads recover redacted characters more readily than exact typed spans; O4 achieves higher F1 on both measures in this comparison\.

Figure 7:Precision–recall comparison on the same 659 Ont3 development inputs across 35 languages; axes start at 20% recall and 40% precision\. Curves vary theObias before constrained BIOES decoding; thin plus signs mark zero bias\. Standalone points mark prompted annotation, whose format failures count as empty predictions without encoder fallback\. Current corrected references and optional\-reference scoring apply throughout; counts are pooled across inputs\. Gemma\-4\-12B uses layer\-32 representations conditioned by a short task prompt\. Flash\-Next and both Gemma\-4\-31B variants use matched 512\-unit GELU heads over final\-layer states at token offsets \[−1, 0, \+1\], with the same training targets, annotation prompt and fitting schedule\. O4 uses isolated input, human\-gold replay and character\-boundary refinement\. Name\-kind subspans are not scored\.
### A\.6Ontology inventories and refinements

Figure 8:Ont1’s union inventory\. Branches show the recorded parent relations, not mutually exclusive subclasses\. Under person\_name, teal name components and orange person roles mix different annotation axes\. The government\_id branch likewise retains the finer distinctions contributed by source corpora\.Figure[2](https://arxiv.org/html/2609.38630#S4.F2)shows name components and roles as separate refinements of primary spans\.

Ont3 keeps all 29 Ont2 primary labels and adds onlyperson\_referenceandorganization\_reference\. Figure[3](https://arxiv.org/html/2609.38630#S4.F3)shows the shared inventory and marks the two additions\. Its 12 semantic families group the primary labels\.

Figure 9:Refinement heads read the same encoder states as the BIOES head\. Token categories identify name components; independent binary predicates allow overlapping attributes\. A whole\-span category, such as identifier kind, is predicted from the mean token logits\. Refinements do not affect the reported model comparisons\.Ont3 also extends the refinements attached to primary spans to name components, identifier kinds, place level, jurisdiction, currency and person roles; Ont2 already had family\-name and care\-provider attributes\. Although the two inventories share their primary labels, their annotation instructions differ: Ont3 explicitly distinguishes names from grounded references, including where organization determiners fall relative to the span\. We have no independent gold for the component refinements\.

Subwords can cross name\-component boundaries, particularly in Asian languages: one subword may touch two components without enclosing either\. Character models should refine these extents, just as character\-level repair addresses primary\-span boundaries \(Section[A\.14](https://arxiv.org/html/2609.38630#.A1.SS14)\)\. Our separate name postprocessor combines a compact character model’s given\-name and surname scores with language\-specific ordering rules\. This discriminative use differs from generating replacement names \(Section[A\.18\.1](https://arxiv.org/html/2609.38630#.A1.SS18.SSS1)\)\.

A model moving from one inventory to the next can keep a separate head for each over one shared encoder and shift loss weight between the heads gradually; Section[A\.12\.1](https://arxiv.org/html/2609.38630#.A1.SS12.SSS1)describes the Ont1\-to\-Ont2 transition\.

### A\.7Evaluation populations

The human\-gold view contains 1,283 distinct test segments in seven languages; Ont3 contains 1,201 frontier\-annotated development segments across 35 languages, including 12 segments without an attributable sentence language\. Its seven\-language subset contains 302 segments\. The pool combines general text and rare\-type\-targeted text; some references come from a single Luna teacher\.

The current comparisons use O4 in isolation and retain O3 with its original neighboring\-sentence policy: both neighbors where reserved, otherwise only the preceding sentence\. O1 and O2 are the frozen predecessor checkpoints\. GL4 was selected on a 659\-row subset of Ont3\. Published GLiNER2, GL, receives its supported\-category and title allowances \(Section[A\.9\.1](https://arxiv.org/html/2609.38630#.A1.SS9.SSS1)\)\. Recipe and scoring obligations differ, so the curves do not isolate architecture or adaptation effects\.

Figure 10:The human\-gold and Ont3 panels of Figure[1](https://arxiv.org/html/2609.38630#S1.F1), requiring identical redaction\-region boundaries\. GL has an unfair advantage: categories its labels cannot express are excluded from its scoring obligations, and it may omit job titles\. Honorifics are optional; titles may be inside name spans or separate\. All models receive corpus\-coverage masks \(Section[A\.9\.1](https://arxiv.org/html/2609.38630#.A1.SS9.SSS1)\); Figure[13](https://arxiv.org/html/2609.38630#.A1.F13)compares O4 and GL on identical supported categories\.Figure 11:Ont3 comparison on 1,201 segments, including the O1 predecessor omitted from the main figure\. Regions match at 80% symmetric overlap\. GL has an unfair advantage: categories its labels cannot express are excluded from its scoring obligations, and it may omit job titles\. Honorifics are optional; titles may be inside name spans or separate\. All models receive corpus\-coverage masks \(Section[A\.9\.1](https://arxiv.org/html/2609.38630#.A1.SS9.SSS1)\); Figure[13](https://arxiv.org/html/2609.38630#.A1.F13)compares O4 and GL on identical supported categories\.Lines connect measured settings in threshold order, including backward bends\. Changing the outside\-tag bias can replace spans or move their boundaries; thresholding fixed GLiNER2 spans can split a merged redaction region\. A stricter setting may therefore improve both precision and recall\. Long terminal backward connectors are omitted, with measured endpoints kept\. Both axes start at 0\.4; off\-scale Presidio points are marked at the axis edge\.

The 659\-row selection subset has been repeatedly inspected\. Language assignments were reviewed against a saved sentence\-language detector, and annotations were corrected after error inspection\. Exact and NFC/whitespace checks found no matches against 18 recovered training pools, but training ancestry and semantic overlap were not fully certified\. On human gold, corpus test membership does not make the repeatedly scored subset a fresh selection holdout\. Wojood demographic predictions outside annotated demographic spans are unscored for every system because that corpus does not exhaustively cover the category; overlapping predictions keep their full boundaries\.

### A\.8Earlier evaluation sets

These collections preceded the human\-gold and Ont3 populations of the main comparison; appendix technique comparisons still report on them\. Fresh6 is an independently authored 426\-document diagnostic suite spanning Chinese, Japanese, Korean, Spanish, German, and French\. Fresh7 adds 61 English documents\. Fresh20 combines that frozen seven\-language suite with 780 later documents—60 in each of thirteen added languages—for 1,267 documents across twenty languages\. Scoped MultiGraSCCo contains 630 synthetic clinical document\-language instances, 63 in each of ten languages\.

The twenty\-language Final20 collection contains 7,000 records but only 3,212 distinct texts across twenty languages\. Repeated texts also cross the boundary between its calibration and evaluation records\. We therefore do not use it for the main precision–recall comparison or for document\-count\-based uncertainty and dominance claims\.

Language\-weighted scores on these sets use importance weights reflecting expected use\. The nine priority languages are those weighted ×2 or ×4\. The same weights guided how annotation effort was spread across the 35 languages, with a minimum of about 0\.5% of annotated examples per language as the first priority\. Because training started heavy in the gold\-annotated languages, the realized mixtures do not match these weights exactly \(Section[A\.12\.2](https://arxiv.org/html/2609.38630#.A1.SS12.SSS2)\)\. Codes follow the dataset’s language tags;fildenotes Filipino\. The pooled human\-gold and Ont3 comparisons sum counts and use no weights\.

Table 9:Language weights in historical evaluation sets\. The nine priority languages have weight ×2 or ×4\. Main Ont3 and human\-gold comparisons use pooled counts without language weights\.CoverageImportanceLanguage codesOriginal 20×4en×2ar, de, es, fr, ko, pt, vi, zh×1cs, hi, id, it, ja, nl, pl, ru, sv, tr, uk\+15 → 35×1bn, da, el, fa, fi, fil, he, hr, ms, no, ro, ta, te, th, ur
### A\.9Span, character, and ontology views

Spans use half\-open Unicode codepoint offsets into the supplied text, after any declared preprocessing\. Changing Unicode normalization requires remapping the offsets; each model card specifies the normalization and offset contract\. Exact span matching requires identical boundaries\. To tolerate small boundary misses, overlap scoring also accepts a prediction when its intersection with the gold span covers at least 80% of each span\. Among eligible pairs, we choose the largest set of matches in which neither a prediction nor a gold span appears twice\. This prevents duplicate or fragmented predictions from earning repeated credit for one entity\.

Fine\-grained scoring retains the most specific jointly expressible label\. Each comparison applies its declared mapping symmetrically and reports excluded labels\.

Character precision is the fraction of redacted Unicode character positions covered by gold spans; recall is the fraction of gold character positions redacted\. F1 is their harmonic mean\. We score the final spans after boundary repair, counting each character once\. Character scoring gives proportional credit to partial coverage; span scoring tests entity localization\.

The span examples in Section[A\.10\.4](https://arxiv.org/html/2609.38630#.A1.SS10.SSS4)use configurations selected for 90% character recall in the twenty\-language Final20 comparison \(Section[A\.8](https://arxiv.org/html/2609.38630#.A1.SS8)\)\. Their displayed spans ignore type; the examples were chosen for legibility\.

Reference spans, thepersonandorganization\_referencetypes in Ont3 \(such as “the suspect” in Figure[6](https://arxiv.org/html/2609.38630#.A1.F6)\), are optional for redaction: leaving referential wording visible can preserve readable text after identifying names are removed\. We exclude reference targets and predictions, and neutralize an exact\-boundaryperson\_nameprediction onperson\_reference, ororganizationonorganization\_reference\. Missing an optional reference incurs no false negative\. Wrong entity families and boundary errors remain errors; neutral spans are also removed before character scoring\. This avoids penalizing low\-agreement distinctions or corpora that intentionally use the base label\.\+\_referenceinstead requires the specific reference label and is a fair task comparison only for models trained to that Ont3 distinction\.

#### A\.9\.1Titles and expressible categories

We accommodate title conventions before redaction\-region matching\. An annotated honorific beside or inside a person name is neutral: predicting “Dr\. Maya Chen”, “Maya Chen”, or separate spans for “Dr\.” and “Maya Chen” receives the same credit for the name\. Predicting only “Dr\.” does not recover the name\. The scorer removes the known title extent from both sides before matching the remaining name; it does not forgive other boundary errors\.

Job titles, offices and professions are distinct from honorifics\. In “President Maya Chen”, “President” is a demographic attribute\. A model capable of that category must cover the title, but may include it in the name span or emit it separately: title coverage is scored as its own region\. For a model unable to express demographic attributes, leaving the title visible or redacting it is free\. Thus the allowance is not that all titles are optional for every model\. Known title extents come from corpus fine labels, separate Ont3 title spans and a reviewed neutral\-honorific sidecar; the evaluator does not guess arbitrary title boundaries from a prediction\.

Published GLiNER2 is prompted with its 42 PII labels plus organization\. Gold categories outside their reviewed mappings are removed from its task; its fine Ont3\-tuned variants are evaluated on the full inventory\. This gives the published model a less demanding target, so its aggregate score cannot by itself measure the effect of adaptation\. All systems also receive the source\-corpus coverage masks: an unannotated type is not automatically a false positive\. For partially covered categories, a prediction overlapping an annotated example keeps its full extent, so coverage masking cannot erase its boundary error\. Untyped region matching further accepts different internal type assignments when they imply the same redaction\.

### A\.10Published comparison systems

#### A\.10\.1Presidio configuration

We run Microsoft Presidio 2\.2\.364 from the data\-privacy\-stack multilingual fork, with spaCy 3\.8\.16 and model packages 3\.8\.0\. Fourteen languages use their language\-specific large spaCy model; the others usexx\_ent\_wiki\_sm\. We keep Presidio’s rule recognizers and its default score threshold of zero\. Filipino and language\-free inputs run under engine codestlandxx, respectively, but are still scored under their original language assignments\.

#### A\.10\.2Shared\-category comparisons

The original OpenAI Privacy Filter \(PF\) predicts eight entity labels and covers neither organizations nor general locations by design\. Figure[12](https://arxiv.org/html/2609.38630#.A1.F12)therefore scores PF against O4, our selected Ont3\-trained XLM\-R model, only on the categories PF can express\.

Figure 12:O4 and the original OpenAI Privacy Filter \(PF\) on a reduced task that scores only the entity categories covered by PF’s eight labels\. Organizations, generic locations and every other category outside PF’s labels are removed from the task, gold and predictions alike, preserving many\-to\-many alternatives admitted by coarse human\-gold labels, without retraining either model; characters inside removed gold regions are not scored\. Curves sweep each model’s decision threshold; labels give each curve’s best character F1\. Columns are human gold and Ont3; the top row uses all languages \(1,283 and 1,201 segments\), the bottom English only \(176 and 56\), the primary language named by PF’s model card\. Both models use the same subset in each panel\.Figure 13:O4 and published GLiNER2 \(GL\) with identical category obligations\. Both models are scored only on categories expressible by GL’s 42 requested labels plus organization, with the same title allowances and corpus\-coverage masks\. O4 predictions outside that inventory are removed as well, preserving many\-to\-many alternatives admitted by coarse human\-gold labels\. Unlike the main comparison, GL receives no exclusive category exemption\. Untyped regions match at 80% symmetric overlap; labels give best measured region F1\. Columns are human gold and Ont3\. The Ont3 panels use only the 542 segments not used to select O4 or GL4\. The top row uses all languages \(1,283 and 542 segments\); the bottom uses GL’s advertised English, French, Spanish, German, Italian, Portuguese and Dutch \(898 and 171\)\. Both models use the same subset in each panel\.
#### A\.10\.3GLiNER2 adaptation conditions

GL4 starts from the published multilingual PII checkpoint, with Ont3 training spans snapped outward to its word tokenizer\. Its training mixture includes the expanded frontier annotations and mapped human gold, using O4’s 50%\-human\-gold sampling recipe \(Table[1](https://arxiv.org/html/2609.38630#S6.T1); Section[A\.17](https://arxiv.org/html/2609.38630#.A1.SS17)\)\. From 50,000 weighted draws, processor validation admits 32,204 windows and 70,269 positive word spans\. All 35 languages meet the 0\.75% sampling floor\. Evaluation boundaries are unchanged\. Human\-gold targets use the mapping’s single\-label fallback; rows with unknown coverage receive positive\-only tasks\. This differs from the encoder’s many\-to\-many loss\.

Each Ont3 type receives a familiar prompt name where possible\. Twenty\-one new atomic label tokens use weighted averages of donor subtoken embeddings \(weight three for the preferred donor, one for each alternative\); ten names retain their tokenization and embeddings\. Predictions map back to Ont3\. The initial model below includes this transfer, so its comparison with later checkpoints holds the target inventory and scoring obligations fixed\.

We train for 2,000 updates, batch eight with four\-step accumulation, encoder/task peak rates10−510^\{\-5\}/2×10−52\\times 10^\{\-5\}, and a cosine schedule with 1,000 warmup steps\. The run takes 10 minutes 56 seconds on an RTX PRO 6000 Blackwell, including a checkpoint every 100 steps\. Validation loss on 249 representable selector windows chooses step 1,800; downstream selection instead chooses step 600\. The latter compares eight stages at confidence 0\.5 using the same language\-weighted criterion as the encoder: 70% exact human\-gold regions, 12% exact Ont3 regions and 18% exact typed Ont3, pooling weighted counts before F1\. It uses 23,251 human\-gold examples and a 659\-row Ont3 subset; the remaining Ont3 rows do not select the checkpoint\.

Table 10:Fixed\-confidence F1 \(%\) on the paper’s 1,283 human\-gold segments and the 659\-segment Ont3 selection subset\. With the same full\-inventory scoring obligations, selected GL4 improves over the transferred initialization by 9\.8 human\-gold and 4\.7 Ont3 F1 points\. Stock GL’s category exemptions in the main graph obscure this adaptation gain\. These are descriptive trajectory values; the smaller human\-gold view does not choose the checkpoint\.UpdatesHuman gold exact regionsOnt3 exact regions0 \(transferred initialization\)57\.4450\.1110063\.7354\.8420064\.7256\.9630064\.5256\.2440062\.9857\.10600 \(selected\)67\.2454\.791,00060\.6749\.432,00041\.1932\.94Early adaptation improves over transferred initialization, then quality declines\. Inspected late errors include a dropped URL and location names retyped as administrative areas\. The saved outputs are structurally valid; the decline is not a parser failure\. We have not isolated whether replay of the original GLiNER2 mixture would prevent it: that mixture is not publicly available, and we did not try to reconstruct it from the paper\.

We also repeated the trajectory with a loss accepting any mapped label for each gold span and masking negatives for types with unknown coverage\. The data, initialization and schedule were unchanged; training took 11 minutes 25 seconds\. This variant selected step 300 and scored 58\.74 on the selection criterion, versus GL4’s 59\.32\. On all 1,201 Ont3 inputs, exact\-region F1 rose from 57\.47 to 59\.12 \(paired 95% interval for the gain: \[0\.01, 3\.34\] points\), while exact typed F1 fell from 53\.48 to 52\.48\. Human\-gold exact\-region F1 rose from 67\.24 to 68\.19, with an interval spanning zero\. The main comparison retains the higher\-selection\-score fallback variant\.

On the 542 Ont3 segments not used to select either checkpoint, O4 scores 86\.8 fine\-type character F1 versus GL4’s 74\.6, and 89\.3 redaction\-character F1 versus 79\.1\. The gains are 12\.1 \[9\.1, 15\.3\] and 10\.2 \[7\.3, 13\.4\] points \(paired document\-bootstrap 95% intervals\)\. Restricting to GL’s seven advertised languages leaves 171 segments: fine\-type character F1 is 85\.1 versus 74\.8, and redaction\-character F1 is 87\.9 versus 79\.8\. These comparisons keep all primary types under the paper’s optional\-reference policy, with no model\-specific category exemptions, at O4 bias zero and GL4 confidence 0\.5\.

#### A\.10\.4Gains over published privacy filters

We measured the span\-overlap F1 gains of our twenty\-language Ont2 XLM\-R encoder \(Section[A\.13](https://arxiv.org/html/2609.38630#.A1.SS13)\) over five published privacy\-filter configurations at fixed operating points on Fresh20, a 1,267\-document evaluation pool in twenty languages \(Section[A\.8](https://arxiv.org/html/2609.38630#.A1.SS8)\)\. Each row scores the encoder and one baseline only on the languages that baseline declares it supports, so the evaluation subset differs from row to row\. Span matching ignores entity type, and gains are in percentage points\.

Table 11:Ont2 encoder gains in untyped span\-overlap F1 at fixed operating points on Fresh20\. Each row uses only the baseline’s supported languages; differing subsets prevent ranking the baselines by these gains\. Intervals use paired document resampling\.Baseline configurationLanguagesDocumentsOur model’s F1 gain over baseline \(points\)95% interval \(points\)GLiNER27421\+21\.78\[\+19\.14, \+24\.42\]OpenAI Privacy Filter161\+47\.31\[\+42\.37, \+52\.25\]Multilingual OpenMed14907\+40\.55\[\+37\.63, \+43\.46\]OpenMed with additional task fine\-tuning201,267\+48\.70\[\+46\.08, \+51\.31\]OpenMed Nemotron9541\+43\.76\[\+40\.18, \+47\.35\]The OpenAI Privacy Filter row covers only 61 English documents and is exploratory\. The OpenMed model with additional task fine\-tuning is a local fine\-tune evaluated on all 1,267 documents\. “Additional task fine\-tuning” names that comparison arm; it does not imply that the two encoders share a complete training history\.

The encoder uses its coarse–fine\-aware constrained decoder with zero outside\- label bias\. Scores compare fixed operating points rather than optimized precision–recall frontiers\. Documents are weighted by language: English four, the other eight of the nine priority languages \(Section[A\.8](https://arxiv.org/html/2609.38630#.A1.SS8)\) two, and the remaining languages one, with weights renormalized over each row’s subset\.

The 95% intervals use 10,000 paired document\-bootstrap resamples\. Within each row’s comparison, a single\-step maximum absolute standardized bootstrap error adjusts for all reported metric cells; the five rows are not jointly adjusted as one comparison family\.

Figure[14](https://arxiv.org/html/2609.38630#.A1.F14)and Figure[15](https://arxiv.org/html/2609.38630#.A1.F15)compare GLiNER2 and our encoder on one English and one Spanish input\. Gold brackets sit above the text and model brackets below, with the finest labels in parentheses; GLiNER2 was prompted with broad groups only\. Arrowheads continue a span across lines\.IDdenotes unique identifiers;loc\.groups locations and contact details \(operating points and matching: Section[A\.9](https://arxiv.org/html/2609.38630#.A1.SS9)\)\.

Figure 14:English\.GLiNER2’s financial span extends from the name through the insurance number, including intervening prose\. The encoder matches the gold boundaries; its finer identifier type differs\.Figure 15:Spanish\.In this short clinical contact block, GLiNER2 labels only the surname; the encoder includes the given name as well\. Both recover the address fields and email\.These two selected examples come from OpenPII 1M \(Ai4Privacy / Ai Suisse SA\) and MEDDOCAN \(synthetic clinical corpus\), respectively; both are CC BY 4\.0\([Ai4Privacy, 2026](https://arxiv.org/html/2609.38630#bib.bib1);[Marimon et al\., 2019](https://arxiv.org/html/2609.38630#bib.bib13)\)\. The OpenPII example is not a member of the current Ai4Privacy\-excluded benchmark\.

### A\.11Operating\-point calibration

After checkpoint selection, XLM\-R caches token offsets, the winning non\-Olabel, and theO\-minus\-winning\-label logit \(Ois the BIOES outside tag\); calibration then searches a globalObias under three disclosed decoders without another encoder forward pass\. GLiNER2 analogously caches candidate spans and searches confidence thresholds\. The hashed operating manifest froze 11 XLM\-R and 32 GLiNER2 configurations before the Final20 comparison; Section[A\.8](https://arxiv.org/html/2609.38630#.A1.SS8)describes that twenty\-language collection and its prior use\. Calibration used separate records, but not necessarily distinct texts, and fixed all language, label and reporting\-metric settings\. Neither sweep requires training updates\.

Even with fully annotated data, reducing the loss weight on known background tokens from 1 to 0\.75 improved Ont2 span F1 in all four matched training\-seed pairs after 250 continuation steps per model\. On the 1,267\-document Fresh20 set, importance\-weighted language\-macro span\-overlap F1 rose by a mean of 0\.23 percentage points across seeds\. For the same comparisons, mean recall rose by 0\.58 points and precision fell by 0\.11 points, consistent with an operating\-point shift\.

### A\.12Training and model controls

The following recipes specify the controls behind the main text’s data, decoder and encoder comparisons\.

#### A\.12\.1Transition between labeling tasks

To introduce the Ont2 label inventory while retaining supervision in Ont1’s label space, we kept the trained encoder and its existing affine head \(one linear map from each token state to label scores\), then added an Ont2 head\. A reviewed label projection seeds the new head from the old classifier\. Both heads read the same encoder states\. Examples annotated in a head’s own label space use ordinary cross\-entropy; examples from the other task use the many\-to\-many mapping, minimizing−log∑k∈A⁡\(y\)p\(k\)\-\\log\\sum\_\{k\\in A\(y\)\}p\(k\)over the labels compatible with annotationyy\. Thus a coarse annotation can supervise a finer task without inventing a specific fine label\.

We blend the separately normalized head losses asL⁡\(t\)=λ⁡\(t\)​Lold\+\[1−λ⁡\(t\)\]​LnewL\(t\)=\\lambda\(t\)L\_\{\\mathrm\{old\}\}\+\[1\-\\lambda\(t\)\]L\_\{\\mathrm\{new\}\}\. The evaluated transition linearly reducedλ\\lambdafrom one to zero over 1,500 updates\. Once it reaches zero, the old head is retired\. Old\-task examples can still train the new head through the mapping\. For continued support of both labeling tasks, the same construction instead retains both heads and nonzero loss weights\. Separate output spaces preserve task\-specific distinctions while sharing encoder computation and parameters; the desired balance is set by their supervision weights\.

#### A\.12\.2Training mixtures

Table[12](https://arxiv.org/html/2609.38630#.A1.T12)lists the methods active in O2, O3 and O4 in the main comparison \(Figure[1](https://arxiv.org/html/2609.38630#S1.F1)\)\. Each column is a complete training recipe; no individual change is ablated at matched exposure\. O4 retains O3’s architecture and label inventory while changing the annotation pool, sampling and input recipe \(Table[1](https://arxiv.org/html/2609.38630#S6.T1); Section[A\.17](https://arxiv.org/html/2609.38630#.A1.SS17)\)\.

Table 12:Active methods in O2, O3 and O4\. O2 uses Ont2; O3 and O4 use Ont3\. O4’s additional annotations, cleanup and sampling are detailed in Section[A\.17](https://arxiv.org/html/2609.38630#.A1.SS17)\. Boundary refinement is described in Section[A\.14](https://arxiv.org/html/2609.38630#.A1.SS14); the main curves score primary redaction spans and ignore name\-kind subspans\.MethodO2O3O4Starting encoderPreviously task\-trained encoder, transferred to Ont2Upstream pretrained XLM\-R\-largeOnt3 encoder’s isolated\-input branchLanguage coverage20 languages35 languages35 languagesSupervisionMapped annotations, translated data and Ont2 complete\-label calibration mixtureOnt3 mixture plus 40% draws from four mapped human\-gold corpora49,812 additional frontier\-annotated texts; earlier labels cleaned; prompts clarify occupation/job titles asdemographic\_attribute, outside names\. 50% mapped human\-gold draws\.Primary label inventory29 typesSame 29 types plus two optional reference typesSame as O3Additional training objectivesBinary entity\-presence loss, weight 2Subclass and type\-conditioned predicate losses, each weight 1Same as O3Oloss weight1 on complete annotations; 0 on partial annotations0\.75 on supervisedOtokens; corpus\-unannotated types maskedSame as O3Language\-specific output biasNoneLearned bias per language added to every output scoreSame as O3Neighboring\-sentence inputTarget sentence onlyTraining: equal isolated, previous\-only and both\-neighbor draws\. Inference: both neighbors, within 512 tokensTarget sentence only, in training and inferenceMasked language modelingOffOffOffCharacter\-boundary refinement in the main curves±1\-codepoint endpoint ranker, 35 languagesSame rankerSame rankerThe recorded samplers also differ substantially in language concentration: the nine priority languages \(weighted ×2 or ×4 in Section[A\.8](https://arxiv.org/html/2609.38630#.A1.SS8)\) receive 75\.1% of O2’s expected sampled\-example mass and 55\.2% of O3’s; the seven languages in the human\-gold comparison receive 71\.7% and 53\.6%\. English alone receives 40\.8% and 19\.6%\. These are sampling probabilities for the selected stage and do not measure cumulative token exposure\. O2’s selected stage includes substantial translated supervision and MEDDOCAN, MultiGraSCCo and TAB data\. Relative to the matched Ont3 mixture without added gold, mapped\-gold replay \(the 40% draws above\) increases the seven gold languages’ expected share from 37\.9% to 53\.6%; domain exposure and language allocation therefore both change\. Excluding overlapping evaluation inputs does not remove similarities in domain, text formatting or annotation conventions between train and test\.

We also explored ordinary\-density text from FineWeb and FineWeb2 to broaden training beyond entity\-dense examples\. Its effect on entity/background exposure depends on sampling and annotation coverage\. Related changes to background supervision and operating points are described in Section[A\.11](https://arxiv.org/html/2609.38630#.A1.SS11)\.

#### A\.12\.3BIOES heads and decoding

Documents are divided into verbatim, sentence\-friendly windows without cutting a labeled character span\. Tokenizer offset maps align gold spans to overlapping subword tokens; special tokens are ignored\. Single\-token spans receive S; longer spans receive B, zero or more I labels, and E, each paired with its entity type\.

##### Decoder contrast conditions

To compare decoders without changing encoder scores, we reuse full logits from one XLM\-R checkpoint \(step 2,000\) trained before the Ont2 and Ont3 inventories existed\. The diagnostic set is Fresh7 \(Section[A\.8](https://arxiv.org/html/2609.38630#.A1.SS8)\): 61 English, 60 each German, Spanish, French, Japanese and Korean, and 126 Chinese documents\. All methods share logits, gold, reporting classes, token offsets, window deduplication and a zeroObias, the inference\-time offset added to the outside\-label score before decoding\. Figure[16](https://arxiv.org/html/2609.38630#.A1.F16)illustrates the class\-aware decoder on a constructed example\. Matching\-type Viterbi uses legal fine\-type transitions\. The greedy heuristic it replaced merges compatible continuations and retains the opening type\. Both coarse\-class Viterbi variants use the 20\-class compatibility map and assign the final fine type by character\-weighted token votes; only their state aggregation differs\. The soft variant uses normalized log\-mean\-exp at temperature 0\.5\.

Section[A\.12\.6](https://arxiv.org/html/2609.38630#X.A1.SS12.SSS6.Px1)reports a training comparison under the greedy deployment decoder, including its tradeoff between span detection and fine typing\.

Figure 16:BIOES labels mark a span’s beginning, interior, end or single token; O marks text outside spans\. In this constructed example, independent winners switch fine type inside “St Mary Hospital\.” Requiring one fine type loses the name; class\-aware decoding recovers one organization span\. Bold numbers are the independent winners\. Scores are invented logits, not probabilities or measured model outputs; words stand in for tokenizer tokens\.In the example, combining four organization types at temperature 1 giveslog⁡\[\(e4\+e−2\+e−6\+e−6\)/4\]≈2\.62\\log\[\(e^\{4\}\+e^\{\-2\}\+e^\{\-6\}\+e^\{\-6\}\)/4\]\\approx 2\.62at each name token, above the outside score of 2\.5\. The two unshown organization types and all unshown boundary scores are −6\. The class\-aware path keeps the alternating fine\-type preferences within one organization span, then labels it healthcare organization: “St” and “Hospital” contribute ten characters of support, versus four from “Mary” for generic organization\.

Table 13:Decoder contrast on the 487 Fresh7 development documents in seven languages, with the same XLM\-R checkpoint logits and zeroObias\. Decode times exclude encoder inference\.DecoderTyped exact F1 \(%\)Typed overlap F1 \(%\)CPU decode time \(s\)Matching\-type Viterbi73\.7778\.652\.41Coarse\-compatible greedy; first label70\.5175\.610\.32Coarse\-class Viterbi; max scores74\.0078\.911\.54Coarse\-class Viterbi; soft scores, T = 0\.574\.1979\.182\.33Score\-adjusted Viterbi; α = 0\.95, T = 0\.574\.1979\.172\.53Table[13](https://arxiv.org/html/2609.38630#.A1.T13)reuses development data\. Typed spans must satisfy the 80% mutual\-coverage rule\. CPU timings are single\-pass observations; a corresponding Ont3 contrast has not been measured\. Temperature 0\.5 had been inspected on these documents before this run, which adds no tuning and no paired intervals\.

An alternative adjusts the label scores before ordinary Viterbi decoding \(Figure[17](https://arxiv.org/html/2609.38630#.A1.F17)\)\. At each token, we combine scores for related labels—for example, beginning of organization and beginning of healthcare organization\. We move each label’s score toward that shared score by a fractionα\\alpha:

zt,b,f′=\(1−α\)​zt,b,f\+α​Gt,b,C⁡\(f\),0≤α≤1\.z^\{\\prime\}\_\{t,b,f\}=\(1\-\\alpha\)z\_\{t,b,f\}\+\\alpha G\_\{t,b,C\(f\)\},\\qquad 0\\leq\\alpha\\leq 1\.
HereC⁡\(f\)C\(f\)is the compatible fine\-type group containingff, andGGis that group’s aggregate score\. We repeat this separately for inside, end, and single\-token labels; outside scores stay unchanged\. Viterbi then selects the highest\-scoring valid sequence, requiring the same fine type throughout each span\. On the same Fresh7 documents, 95% interpolation scores 79\.17 typed overlap F1, versus 79\.18 for the soft coarse\-class decoder \(Table[13](https://arxiv.org/html/2609.38630#.A1.T13)\)\.

Figure 17:Exact fine\-type Viterbi approaches the soft coarse\-class decoder as its scores back off toward the normalized sibling aggregate\. Points are the nine tested interpolation weights; dashed lines show the coarse\-class decoder\. F1 uses 20 shared reporting classes on the same 487 reused development documents\. Fine\-label quality is not measured by these curves\.The group contains fine labels with the same BIOES position and compatible coarse class;Ois a singleton and remains unchanged\. We fixT=0\.5T=0\.5, zeroObias, and all preceding evaluation conditions\. The tested weights are 0, \.125, \.25, \.5, \.75, \.9, \.95, \.99 and 1\. Dividing by group size removes the explicit subclass\-count bonus\. The convex combination preserves invariance to a common additive shift in a token’s logits\.

At zero backoff, predictions exactly reproduce fine\-type Viterbi\. At complete backoff, every member receives its group’s score, so a fine\-state lattice optimizes the same coarse boundary objective: all 487 documents then have the coarse\-class decoder’s spans and coarse labels, and fine types within a class tie\. Interior weights retain fine\-score preferences\. This exploratory sweep does not measure fine\-label quality or establish an independently selected interpolation weight\. For fixed boundaries and any weight below one, the shared coarse term cancels when comparing sibling fine types, so their ranking follows summed original logits, whereas the coarse\-class decoder uses character\-weighted votes over token winners\. At weight \.95, the two choose different fine labels on 14 of 4,359 spans with identical boundaries and coarse class \(0\.32%\)\. For example, “Klinikum Sankt Marien” receives generic organization under the fine\-compatible path and healthcare organization under the vote\. At complete backoff, 2,769 of 4,362 fine labels differ because the fine\-score preference has been removed\.

##### Head size and alternatives

For the XLM\-R\-large O1 model, the head is dropout followed by one affine map from the final 1,024\-dimensional token state to 349 logits:Oplus four BIOES boundary labels for each of 87 entity types\. The head has 357,725 trainable parameters\. To compare alternative heads, we fixed the encoder, a 192,000\-window training subset, the validation procedure, example order, and seed\. After correcting the task\-head initialization seed, we tested seven alternatives: four layer concatenations, one nonlinear head and two rank\-constrained affine heads\. Peak validation F1 was:

Table 14:XLM\-R head and layer\-feature screens on a fixed 192,000\-window training subset\. Entries report peak validation F1, except where an early stopping step is shown\. The initial middle\-layer gain did not repeat\.Token features and headHead paramsInitial screen F1 \(%\)Repeat screen F1 \(%\)Interpretationfinal layer \[24\], affine357,72582\.2082\.14stable referencemiddle \+ final \[12,24\], affine715,10183\.8281\.86initial gain does not repeatlayers \[23,24\], affine715,101—73\.73 step 1,250early\-screen losslayers \[8,24\], affine715,101—71\.67 step 1,250early\-screen lossembedding \+ final \[0,24\], affine715,101—69\.16 step 1,250early\-screen lossfinal layer, 1,024\-unit GELU1,409,37369\.21 step 1,250—early\-screen lossFive nonlinear configurations screened before the seed correction are excluded from this count and support only coarse pruning\. Some corrected alternatives were stopped early, as indicated by their reported steps\.

Concatenating middle and final layers initially gained 1\.6 F1 points, then lost 0\.3 points on repetition\. It also peaked below final\-only features on a separate native\-gold screen \(74\.9 versus 76\.1 F1\)\. Rank\-constrained affine heads reached 65\.4 and 57\.0 F1 and were dropped from this search\.

A second frozen\-feature test concatenated the final token states of XLM\-R and mmBERT, aligned by character position, and trained only an affine head\. On 480 validation rows after fitting 1,280 training rows, both seeds lost fine\-label F1 relative to XLM\-R alone\. The recorded differences are diagnostic point estimates\.

We also tested sequence\-model alternatives in less controlled comparisons\. An mmBERT package replacing its stock nonlinear head with affine emissions and a first\-order BIOES conditional random field fell from 94\.4/72\.2 to 92\.0/54\.5 F1 on Fresh6/MultiGraSCCo \(a six\-language diagnostic suite and a synthetic clinical set; Section[A\.8](https://arxiv.org/html/2609.38630#.A1.SS8)\) and from 36\.9 to 0\.96 documents per second\. A zero\-parameter legal\-BIOES projection improved familiar\-distribution token diagnostics but reduced frozen Fresh6 overlap F1 by 1\.9 points \(interval \[−2\.38, −1\.43\] points\)\. We therefore retained the final\-state affine head\. The full search record is in the[classifier\-head and encoder\-depth contrast](https://../frontier.md#classifier-head-and-encoder-depth-contrast)and[CRF analysis](https://../frontier.md#what-the-tested-mmbert-crf-wasand-why-it-is-dropped)\.

#### A\.12\.4Pretrained\-encoder selection

To compare encoder families on common training data, we use only 1,854 MEDDOCAN and TAB training documents and select checkpoints without using results on the evaluation sets, the six\-language Fresh6 diagnostic suite and synthetic clinical MultiGraSCCo \(Section[A\.8](https://arxiv.org/html/2609.38630#.A1.SS8)\)\. XLM\-R\-large and LaBSE each use a separately trained final\-state affine head\. mmBERT uses its stock two\-affine GELU/LayerNorm head, so its row compares system families and does not isolate the encoder\.

Table 15:Encoder systems trained on the same 1,854 MEDDOCAN/TAB documents and evaluated on Fresh6 and MultiGraSCCo\. XLM\-R and LaBSE each use an affine head; mmBERT uses its stock nonlinear head\. Brackets give 95% intervals\.Encoder and task headFresh6 F1 \(%\) \[95% interval\]MultiGraSCCo F1 \(%\) \[95% interval\]DispositionXLM\-R\-large, final affine92\.23 \[91\.41, 92\.98\]75\.07 \[74\.27, 75\.90\]quality base selectedLaBSE, final affine87\.88 \[86\.68, 88\.94\]72\.53 \[71\.69, 73\.38\]common\-head comparisonmmBERT\-base, stock head89\.25 \[88\.27, 90\.17\]70\.61 \[69\.77, 71\.49\]faster challenger; different headWith independently trained affine heads, XLM\-R exceeds LaBSE by 4\.4 Fresh6 F1 points and 2\.5 MultiGraSCCo points\. Other encoder\-family comparisons used different training mixtures; this table is the common\-data reference\.

Using O4’s full training pool, we also compare mmBERT’s stock 768\-unit GELU/LayerNorm head with an affine head, keeping the encoder, fresh classifier initialization, dropout 0\.1, sampling and learning rates matched\. Both use O4’s boundary adjustment, name handling and regex supplementation at evaluation\. Checkpoints are selected on the same 279 development examples\. The affine and stock\-head fits stop after 7,500 and 4,000 updates, taking 69 and 43 minutes on a 96\-GB GPU; their selected checkpoints are at 5,500 and 2,000 updates\.

Table 16:Exact\-span F1 \(%\) on 1,283 human\-gold examples and 1,201 Ont3 development and held\-out examples\. The mmBERT head comparison uses O4’s full training pool and serving policy; O4 retains its own training lineage\.Model and headHuman gold regionsOnt3 regionsOnt3 typedmmBERT\-base, stock GELU84\.8976\.8672\.32mmBERT\-base, affine87\.0377\.0273\.90XLM\-R\-large O4, affine87\.8080\.2076\.30The affine head gains 2\.14 human\-gold F1 points \(paired document\-bootstrap 95% interval \[0\.95, 3\.38\]\) and 1\.59 Ont3 typed points \[0\.05, 3\.08\] over the stock head\. Its Ont3 region change, \+0\.16 \[−1\.36, \+1\.68\], is unresolved\. It remains 2\.40 typed points below O4 \[−3\.98, −0\.80\]\.

#### A\.12\.5Locale\-aware names and dates

Our augmentation pipeline translated text containing typed entity placeholders, then filled them with generated values, including names from the[Faker library](https://faker.readthedocs.io/)\. For\[GIVEN\_NAME\] \[FAMILY\_NAME\], the filler drew from separate given\-name and surname generators and assigned each span its placeholder’s label\. Translation often preserved this given\-then\-family order in Korean\. The encoder trained on these examples subsequently classified 65 of 79 Korean evaluation surnames as given names\. Only 3\.7% of training names were family\-first, compared with 87\.8% of evaluation names\. Dates showed a similar mismatch: 85% of evaluation dates used the nativeYYYY년 M월 D일form, which was absent from training\. These mismatches motivated repairs to name order and date formatting\.

Related work on generated PII data and multilingual evaluation includes GLiNER2\-PII, REDACT and DialogPII\([Zaratiana et al\., 2026](https://arxiv.org/html/2609.38630#bib.bib20);[Vats et al\., 2026](https://arxiv.org/html/2609.38630#bib.bib19);[Roller et al\., 2026](https://arxiv.org/html/2609.38630#bib.bib17)\)\.

For affected Chinese, Japanese and Korean examples derived from English, we placed family names before given names, preserving each component’s label\. We also rendered Korean dates in the native form while retaining one\-eighth in other formats\.

Across three random seeds, matched training runs showed that name\-order repair improved span\-overlap F1 averaged across languages by 1\.4–1\.9 points, with the largest gains in Japanese\. Date\-format repair improved Korean span\-overlap F1 by 4\.4–7\.5 points\. All six paired 95% confidence intervals were entirely positive, on the 487 Fresh7 development documents in seven languages \(Section[A\.8](https://arxiv.org/html/2609.38630#.A1.SS8)\)\. The combined locale repairs improved span\-overlap F1 on the twenty\-language Fresh20 set by 1\.5 points and fine\-label F1 by 1\.7 points, both with entirely positive 95% intervals\.

The name\-order treatment changes only the realized order of separate given\-name and family\-name values in audited source families whose carrier text was English; the labels stay attached to their original values\. The Korean\-date treatment changes parseable Korean date surfaces to nativeYYYY년 M월 D일while a deterministic one\-eighth remains non\-native\. Each seed’s treatment and control share the starting checkpoint, learning rate, schedule, step count and all other data composition\. Each interval uses 10,000 paired document resamples\.

Table 17:Name\-order repair versus its matched control on 487 Fresh7 documents\. Deltas are language\-macro overlap\-F1 points, with paired 95% intervals\.SeedName repair: macro span Δ \(points\) \[95% interval\]Name repair: macro fine Δ \(points\) \[95% interval\]154\+1\.57 \[\+0\.79, \+2\.38\]\+2\.47 \[\+1\.78, \+3\.20\]155\+1\.44 \[\+0\.64, \+2\.23\]\+1\.96 \[\+1\.30, \+2\.62\]156\+1\.88 \[\+0\.92, \+2\.84\]\+2\.64 \[\+1\.86, \+3\.42\]Table 18:Korean date\-format repair versus its matched control on Fresh7\. Deltas are span\-overlap\-F1 points for Korean and the seven\-language macro average, with paired 95% intervals\.SeedDate repair: Korean span Δ \(points\) \[95% interval\]Date repair: macro span Δ \(points\) \[95% interval\]154\+5\.08 \[\+2\.64, \+7\.63\]\+1\.62 \[\+0\.92, \+2\.33\]155\+7\.53 \[\+4\.61, \+10\.63\]\+2\.12 \[\+1\.29, \+2\.96\]156\+4\.38 \[\+1\.06, \+7\.58\]\+1\.16 \[\+0\.27, \+2\.04\]Japanese supplies the largest and most stable name\-order effect; Korean’s name\-only response varies by seed\. The date repair is Korean\-positive in all three seeds\. These matched contrasts credit the two convention repairs individually\. The release\-level comparison credits the wider locale bundle: the release trained with it, minus the preceding sibling release, changes Fresh20 span and fine\-label overlap F1 by 1\.5 points \(interval \[0\.36, 2\.63\]\) and 1\.7 points \(interval \[0\.62, 2\.81\]\), with no supported language loss\. It does not isolate the remaining locale\-profile rules\.

#### A\.12\.6Routed translated\-data transport

To improve translated training data, we used TranslateGemma\-27B to translate text containing entity placeholders\. Checks on the placeholders and output structure determined whether to accept a translation, retry it with Gemma\-4\-31B, or reject it\.

An entity placeholder replaces a labeled value with its type and a stable identifier:\[GIVEN\_NAME\_1\]stands for one given\-name slot\. The translation can move that slot while its identifier retains the source label\. Filling the translated slot with a replacement name gives both the new text and the location of its label\.

Table 19:Placeholder translation preserves the given\-name annotation through its stable slot identifier\. The translated marker is subsequently filled with an entity value\.StepTextSource fieldFirst Name: KevinReplace the name with a placeholderFirst Name: \[GIVEN\_NAME\_1\]TranslateGemma outputNombre: \[GIVEN\_NAME\_1\]Marker\-preserving translation has been used to transfer span labels across languages\([Chen et al\., 2023](https://arxiv.org/html/2609.38630#bib.bib3)\)\. An alternative generates candidate target spans and ranks them by translation probability before transferring the source label\([García\-Ferrero et al\., 2023](https://arxiv.org/html/2609.38630#bib.bib6)\)\. We used placeholders instead, because typed names such as\[GIVEN\_NAME\_1\]seemed more direct to interpret in LLM outputs; this was a design preference, without a comparative test against candidate ranking\.

To test the checked\-translation route, we replace half of the existing translated\-family sampling mass for Arabic, Czech, Hindi, Indonesian, Dutch, Portuguese, Swedish, and Vietnamese\. TranslateGemma\-27B produces the primary translation; deterministic placeholder, schema, and structure checks either admit it, route it to a Gemma\-4\-31B fallback, or reject the row\. The resulting route supplies 24,338 weighted windows, or 9\.507% of the unchanged 256,000\-window logical epoch\. Language totals, every non\-transport source, the 15% native\-human\-gold share, and the 37\.425% English share are held fixed\.

For each architecture, we train the checked\-translation and unchanged\-data conditions for the same number of optimizer updates from a shared starting checkpoint\. XLM\-R is evaluated after 125 updates and mmBERT after 250\. The Fresh20 development set contains 1,267 documents in twenty languages; the segmented synthetic clinical MultiGraSCCo evaluation contains 630 document\-language instances \(Section[A\.8](https://arxiv.org/html/2609.38630#.A1.SS8)\)\. Each report uses 10,000 paired document resamples\. The results are:

Table 20:Checked translations versus unchanged training data, at matched exposure within each encoder\. Fresh20 has 1,267 documents; MultiGraSCCo has 630 document\-language instances\. Deltas are F1 points with paired 95% intervals\.SurfaceViewXLM\-R Δ \(F1 points\) \[95% interval\]mmBERT Δ \(F1 points\) \[95% interval\]Fresh20span overlap\+0\.59 \[\+0\.13, \+1\.06\]\+0\.44 \[\+0\.18, \+0\.71\]Fresh20fine overlap\+0\.20 \[\-0\.08, \+0\.49\]\+0\.30 \[\+0\.08, \+0\.53\]MultiGraSCCospan overlap\+0\.15 \[\+0\.04, \+0\.26\]\+0\.38 \[\+0\.22, \+0\.53\]MultiGraSCCofine overlap\+0\.05 \[\-0\.04, \+0\.13\]\+0\.44 \[\+0\.28, \+0\.61\]The checked\-translation route improves span\-overlap F1 for both encoders on both evaluation sets\. The comparison combines translation, fallback and structural checks; the COMETKiwi quality filter rejected no structurally clean primary output\. XLM\-R nevertheless loses Korean F1 in both Fresh20 views, while mmBERT has no supported per\-language loss\. Machine\-readable reports preserve the[XLM\-R Fresh20](https://../evidence/xlmr-tg27-routed05-s1375-vs-control-s1375-fresh20-bootstrap-v1.json),[XLM\-R MultiGraSCCo](https://../evidence/xlmr-tg27-routed05-s1375-vs-control-s1375-mgs10-bootstrap-v1.json),[mmBERT Fresh20](https://../evidence/mmbert-tg27-routed05-s1250-vs-control-s1250-fresh20-bootstrap-v1.json), and[mmBERT MultiGraSCCo](https://../evidence/mmbert-tg27-routed05-s1250-vs-control-s1250-mgs10-bootstrap-v1.json)comparisons\.

##### Comparison using the deployment decoder

A separate XLM\-R comparison added the routed translations together with identifier\-label repairs and locale\-format changes; both models used the same greedy deployment decoder\. It merges inside/end\-token continuations into an open span even when their fine labels differ, then assigns the fine type with the greatest total token\-character coverage within that span\. The outside\-label \(O\) logit bias is zero\. The combined update improves untyped span detection at a fine\-label cost: the MultiGraSCCo fine\-label loss has a paired interval entirely below zero, whereas the Fresh20 fine\-label interval includes zero\. These results concern the bundled update, not the translator alone\. Changes below are language\-macro overlap\-F1 percentage points, with 95% paired intervals\.

Table 21:Routed translations, identifier\-label repairs and locale changes evaluated with the same deployment decoder\. Entries are language\-macro overlap\-F1 changes in points, with paired 95% intervals; untyped gains accompany fine\-label losses\.EvaluationUntyped spansFine labelsFresh20 \(twenty\-language development set\)\+1\.36 \[0\.90, 1\.82\]−0\.45 \[−1\.02, 0\.10\]MultiGraSCCo\+0\.46 \[0\.24, 0\.68\]−0\.90 \[−1\.19, −0\.61\]
##### Translation placeholder repair

To retain entity labels through translation, we sent markers such as\[GIVEN\_NAME\_1\]to TranslateGemma: an uppercase, underscore\-separated class name followed by\_N, the numeric slot identifier expected to survive translation\. If translation changed the class name, we restored the source type using the numeric ID\. We required every ID to occur the expected number of times and rejected translations containing leaked instruction text\. We also filtered translations with COMETKiwi, a machine\-translation quality\-estimation \(MTQE\) model that scores a source and its translation without a reference translation\. We retried failures with Gemma\-4\-31B under the same checks\.

Table 22:Translation\-route acceptance after marker restoration and automated checks across eight target languages\. Acceptance measures structural and quality\-filter passes, not human\-reviewed annotation accuracy\.StageAccepted at this stageCumulative acceptanceTranslateGemma\-27B, after marker repair68,309 / 69,480 \(98\.31%\)98\.31%Gemma\-4\-31B retry of the 1,171 failures1,007 / 1,171 \(86\.00%\)99\.76%Only 164 inputs \(0\.24%\) remained rejected\. Most retries followed structural failures; 200 also or instead failed the MTQE filter\. These counts cover eight target languages under automated checks, not a human\-certified accuracy rate; they follow marker restoration and do not measure raw, unrepaired placeholder survival\.

##### Entity preservation versus localization

We explored preserving aligned source\-language entity values, replacing them after translation, and asking the translation model to localize them jointly with the surrounding text\. Replacement could also happen each time a training example was sampled, either keeping the original value or drawing a new one from locale\-aware generators or compatible observed surfaces\. This separated translation of the sentence from variation in its entity values\.

The preferred choice depended on whether replacement was needed\. In the matched Ont3 training comparison, keeping original surfaces outperformed replacing organization names at sampling time with observed donors, values of the same type taken from other documents\. When filling a placeholder, observed donors were more reliable than free character generation in our development review\. Thus we retained original surfaces by default for that Ont3 recipe, with observed donors preferred when replacement was required; we did not establish a general benefit from redrawing entity values for every training example\. Section[A\.18\.1](https://arxiv.org/html/2609.38630#.A1.SS18.SSS1)separates these comparisons and describes the tested character and token\-encoded context models\.

TranslateGemma does not accept non\-translation task instructions, so we cannot instruct it to preserve a placeholder’s type name\. The numeric suffix still identifies the source label: inspected examples such as the Turkish marker in Figure[20](https://arxiv.org/html/2609.38630#.A1.F20)retain the intended alignment\. We restore the label from the source map, then replace the entire marker with the entity value\. The translated type name never enters the encoder’s training text\. Missing IDs or lost value boundaries still require rejection or retry\.

To avoid wrong\-language entity values in translated training data, we piloted replacing bracketed values such as\[GIVEN\_NAME\_1\]⟦Kevin⟧with locally appropriate ones\. We compared TranslateGemma’s native translation interface with an explicit Gemma\-4 instruction, using 12B models\. This pilot did not supply training data: all 69,480 inputs in the reported training route used ID\-only placeholders followed by separate value generation\.

Gemma\-4 prompt — English to Spanish

> Translate and locally pseudonymize this English sentence into Spanish\. Every entity slot has the exact form\[TYPE\_number\]⟦source surface⟧\. Copy\[TYPE\_number\]exactly once per occurrence, then replace only its adjacent⟦source surface⟧with a plausible Spanish surface of the same primary entity type\. Every copied slot must be followed immediately by one nonempty⟦surface⟧\. Output only the translated sentence\.

Input:\- First Name: \[GIVEN\_NAME\_1\]⟦Kevin⟧

Output — Gemma\-4:\- Nombre: \[GIVEN\_NAME\_1\]⟦Carlos⟧

Figure 18:Instruction and example for translating a typed entity slot while replacing its value with a locally appropriate name; bracketed slot syntax is literal\.Gemma\-4 replaced Kevin with Carlos and preserved its label and boundaries\. TranslateGemma returned\- Nombre: \[GIVEN\_NAME\_1\] Kevin, retaining the source name and losing the delimiters\.

Figure 19:Literal paired Spanish outputs from a nine\-sentence, three\-document pilot covering Spanish, Indonesian and Japanese\. TranslateGemma\-12B loses the surface delimiters; prompted Gemma\-4\-12B preserves them and supplies replacement names\.The pilot’s five entity\-bearing sentences come from the first three documents of an audit set covering Spanish, Indonesian and Japanese \(Figure[19](https://arxiv.org/html/2609.38630#.A1.F19)\)\. Gemma\-4 preserved the format in all five but supplied city names for two county slots; no output repairs were applied\.

Table 23:Joint translation and entity replacement on five entity\-bearing sentences\. Format requires intact markers and value delimiters; replacement type also requires a value of the requested entity type\.ModelFormat passedFormat and replacement type passedTranslateGemma\-12B0% \(0/5\)0% \(0/5\)Prompted Gemma\-4\-12B100% \(5/5\)60% \(3/5\)TranslateGemma preserved ordinary typed placeholders in all nine sentences of the same test; the failures above concern the added value delimiters\.

Figure 20:Recovery boundaries in saved transport output\. The Turkish clinical example means “normal blood count”: its translated label name retains numeric ID 43 and an adjacent value\. A Unicode\-aware validator retrospectively recovers 19 Turkish rows in a 124\-output, four\-language TranslateGemma frame control\. Detached values remain rejected\. The county/city example summarizes the separate Gemma\-4 pilot’s semantic failure\.Accepting non\-ASCII marker names and recovering their labels by numeric ID rescued 19 additional outputs in a separate clinical translation test of 124 outputs, 31 each in Arabic, Hindi, Japanese and Turkish\. All 19 recovered outputs were Turkish; missing or detached value delimiters remained rejected\.

Table 24:Structural acceptance in a 124\-output, four\-language translation test\. Recovering translated marker names by numeric ID rescues 19 Turkish outputs; semantic correctness was not measured\.ModelBefore repairAfter numeric\-ID recoveryTranslateGemma\-12B25\.0% \(31/124\)40\.3% \(50/124\)We have not measured how many outputs pass semantic review after these repairs, or how many failed translations Gemma\-4 can recover in this bracketed format\.

#### A\.12\.7Positive\-only teacher supervision

This comparison changes the loss attached to omissions in one fixed set of 1,652 training rows labeled by a teacher annotator, Gemma\-4\-31B\-IT\. Under complete supervision, teacher omissions become trusted non\-entity \(O\) targets with loss weightλ=1\\lambda=1\. Under positive\-only supervision \(λ=0\\lambda=0\), teacher\-emitted BIOES labels retain ordinary positive loss and every other teacher\-row token contributes no classification loss\. Fully labeled gold and other complete\-label rows retain their ordinaryOloss\. Fixed across conditions are the starting checkpoint, a 10% sampling share for teacher rows, a 15% native\-human\-gold share, 4:2:1 language weights \(Section[A\.8](https://arxiv.org/html/2609.38630#.A1.SS8)\), a 1,000\-step cosine schedule, and decoding at zeroObias\.

Reducing theOloss weight to 0\.75 on fully annotated data also improved Ont2 F1, primarily by shifting the precision–recall operating point toward recall \(Section[A\.11](https://arxiv.org/html/2609.38630#.A1.SS11)\)\. Here we vary the weight of uncertain negatives from teacher omissions instead\.

Table 25:Ignoring unmarked Gemma teacher tokens versus treating them as background\. The contrast averages the two positive\-only seeds 155 and 156 and subtracts the existing complete\-supervision seed\-155 baseline\. Results are on Fresh20, using 10,000 paired document resamples\. Intervals are per metric; win/loss means the interval excludes zero after simultaneous adjustment across 276 comparisons, and unresolved means it includes zero\.Weighted viewMean delta \(points\)Paired 95% interval \(points\)Adjusted statusspan overlap\+1\.00\[\+0\.73, \+1\.40\]wincharacter precision\-0\.26\[\-0\.38, \-0\.17\]losscharacter recall\+2\.80\[\+2\.40, \+3\.20\]wincharacter F1\+1\.50\[\+1\.20, \+1\.70\]winfine overlap\+0\.11\[\-0\.20, \+0\.42\]unresolvedSeed 156 independently reproduces the direction: span overlap, character recall, character F1, and precision change \+1\.1/\+2\.8/\+1\.5/−0\.25 percentage points\. Mechanism alternatives move in the opposite direction\. A retention term toward the starting checkpoint’s posteriors, at weight\.25, gives back 0\.25 span\-overlap F1 points and 0\.41 character\-F1 points\. Assigning omitted teacher\-row tokens partialOloss weightsλ=0\.05\\lambda=0\.05andλ=0\.20\\lambda=0\.20changes span overlap by −0\.19 and −0\.42 points, with simultaneous recall and character\-F1 losses at both weights\. We therefore selectλ=0\\lambda=0\.

On the same 1,652 rows, Gemma\-4\-31B\-IT annotations provided a modest benefit over the encoder’s own labels: fine\-label F1 increased by 0\.3 points, with no clear character\-F1 gain\. These are separate comparisons: the masking gain above does not measure the value of adding teacher data\.

Relative annotation density provides a practical coverage proxy\. We compare saved first\-pass annotations of 1,113 Ont3 development segments in 35 languages against Luna, the annotator that labeled Ont3 training data; this diagnostic is separate from the Ont2 training experiment above\. Gemma\-4\-31B emitted 82% as many primary spans as Luna and Qwen3\.8\-27B 61%\. The deficit was smaller when reference\-type \(\_reference\) tags were excluded\. These are span\-count ratios, without adjustment for split or merged names; they reflect omissions, false positives and span grouping\.

Table 26:First\-pass span counts relative to Luna on the same 1,113 Ont3 development segments in 35 languages\. Ratios measure annotation density, not recall; span splitting and false positives also affect them\.AnnotatorSpan density relative to LunaWithout\_referencetagsGemma\-4\-31B82%90%Qwen3\.8\-27B61%75%##### Boundary conventions in teacher evaluation

Boundary conventions can also mimic teacher omissions\. For example, Gemma\-4\-31B labeled “Amara Bell” as one person span where the reference labeled “Amara” and “Bell” separately\. On Fresh20’s 61 English development documents, the same Gemma\-4\-31B predictions score 78\.0% exact\-span F1, 97\.2% character F1 and 95\.7% character recall under a shared permissive 12\-family type mapping: much of the disagreement concerned segmentation despite high entity\-character coverage\. Joining compatible components in both prediction and reference, then requiring 80% mutual span coverage, gives 96\.8% span F1\. Joining permits gaps of at most three Unicode characters and can combine distinct nearby entities, so this permissive scoring diagnostic does not establish that shifting outer boundaries is safe\. Saved examples, counts and provenance are in the[boundary analysis](https://../evidence/gemma-qwen-recall-interpretation-correction-v1.md)\.

Our revised training objective records internal\-segmentation ambiguity by annotator, language and tag alongside the acceptable\-label mapping\. A rule may cover all languages or have language\-specific overrides\. For a declared name region, its first token permits B or S, its last permits E or S, and interior tokens permit any BIOES entity prefix; a single\-token region permits only S\. We sum probabilities over these alternatives and the mapped acceptable types\. Adjacent same\-type annotations may join across whitespace, but never across punctuation or an unannotated word\.Otokens retain their original supervision\. This is a tokenwise marginal objective; constrained decoding still enforces legal BIOES sequences\. Component and role annotations retain their original carriers\.

The implementation also reports a separate convention\-aware exact score that permits internal segmentation differences while requiring exact outer boundaries\. Previously reported scores and the training objectives of completed runs are unchanged\. The initial policy covers Englishperson\_namefor the identified Gemma training sources; it does not apply to every model or prompt sharing the Gemma name\. Ont3 training did use Gemma labels: one comparison of annotation sources replaced the labels of 10,312 matched training segments with Gemma\-4\-31B \(FP8\) annotations\. That run used exact BIOES targets\. We have not yet measured the downstream effect of the revised boundary objective\.

#### A\.12\.8Mapped\-gold replay and corpus coverage

Mapped\-gold replay assigns 40% of training draws to four human\-annotated corpora, with labels mapped into Ont3 \(Table[12](https://arxiv.org/html/2609.38630#.A1.T12)\)\. After correcting the treatment of unannotated demographic attributes, we continued training and selected the final checkpoint using development results from both Ont3 and human gold\. A further adaptation to isolated and neighboring\-sentence inputs produced O3 \(Section[A\.16\.1](https://arxiv.org/html/2609.38630#.A1.SS16.SSS1)\)\. With character\-boundary refinement \(Section[A\.14](https://arxiv.org/html/2609.38630#.A1.SS14)\), its best historical exact redaction\-region F1 was 79\.8% on Ont3 and 86\.2% on human gold, under that experiment’s references and scoring policy\.

The historical matched replay comparison reuses development data\. The same 279\-input Ont3 selector chooses checkpoints\. At zeroObias, the replay model scores 84\.01% on human gold and 75\.65% on Ont3\. Intervals condition on checkpoint and threshold selection\. The 40% experiment includes entity\-free gold inputs and masks only omitted alternate\-positive regions in nested annotation views\. A 25% experiment using positive\-bearing inputs scores 83\.99% and 76\.18%; the additional 0\.5\-point sum gain remains unresolved \(95% interval \[\-1\.07, 2\.01\]\)\. Training took 35 minutes on one RTX PRO 6000 GPU\. This continuation starts from a separate checkpoint trained withOloss weight 0\.75\. The O3 curve in the main comparison \(Figure[1](https://arxiv.org/html/2609.38630#S1.F1)\) uses a coverage\-aware continuation of it, described in Section[6](https://arxiv.org/html/2609.38630#S6)\.

Positive compatibility does not establish exhaustive annotation coverage\. Reviewing predictions on 400 training inputs from each corpus finds genuine missing mentions alongside model errors\. For example, Wojood occupation and language labels map to demographic attributes but do not cover every Ont3 personal descriptor\. The runs above retain their mapping\-derived training coverage\. Evaluation exempts unannotated Wojood demographics for every model, as described in Section[2](https://arxiv.org/html/2609.38630#S2)\.

#### A\.12\.9Low\-dose natural\-text and MLM replay

We mixed unlabeled natural text into privacy\-label training and asked the encoder to predict randomly hidden tokens, the masked\-language\-model \(MLM\) objective\. As with weight priors and frozen encoder layers, we tried this objective to prevent forgetting during specialization for redaction\. Related work studies continued pretraining and preservation of cross\-lingual representations during task adaptation\([Gururangan et al\., 2020](https://arxiv.org/html/2609.38630#bib.bib9);[Liu et al\., 2021](https://arxiv.org/html/2609.38630#bib.bib12)\)\. MLM is most attractive when acquiring in\-domain annotations is expensive; with those annotations available, these results do not justify its additional cost\.

Before adding natural\-text teacher annotations, we tested masked\-language replay from a twenty\-language redaction checkpoint with a fully trainable encoder\. Its supervised pool contained 306,439 training windows: 256,000 from the existing mixture and 50,439 routed translation windows\. It contained no Gemma\-labeled FineWeb training windows\. In the selected recipe, fifteen percent of sampled rows are natural multilingual text carrying XLM\-R’s original MLM objective at loss weight\.01; the fixed original MLM head supplies the readout while both tagging and MLM losses update the shared encoder\. The recipe uses learning rate3e\-6, a 1,500\-step cosine schedule, and the predeclared externally selected checkpoint at step 250\.

On the twenty\-language Fresh20 development set, the selected recipe raises language\-weighted span\-overlap F1 over its starting checkpoint by 1\.2 points \(paired 95% interval \[0\.87, 1\.54\]\)\. Four additional seed runs reproduced the recipe’s positive direction\. On a fixed set of masked tokens, prediction loss fell from 11\.15 at the supervised starting checkpoint to 9\.28, compared with 1\.21 for pretrained XLM\-R\.

The recipe jointly changes natural\-text replay, learning rate and the MLM mixture\. A matched zero\-MLM control is missing, so attribution remains at the recipe level\.

A broader\-data screen then added two teacher\-labeled pools, varying their combined sampling share across 20%, 30% and 40%, alongside separate FineWeb MLM text\. Those pools supplied 1,240 and 5,510 nonempty annotated windows across all 20 languages; partial\-supervision rows with no marked span were excluded\. Together with 256,000 existing windows, this supervised pool contained 262,750 windows\. The 40% setting assigned each teacher pool 20% of supervised sampling and ran for the full trajectory\. Its best saved checkpoint at the neutral threshold was 0\.4 span\-overlap F1 points below the existing model on the weighted twenty\-language evaluation\. Threshold calibration on a separate 200\-document set left it 0\.5 points below\. Another checkpoint gained 0\.6 character\-recall points while losing 0\.4 span\-F1 points\. The[buildout record](https://../swept/pii-robust-buildout.md)preserves the August 13 broad\-replay comparison\. These are combined\-data/objective results; no matched run isolated MLM’s contribution\.

In a timing benchmark at effective batch size 64, optimizer\-step time rose from 0\.27 seconds without MLM to 0\.38 seconds at 15% replay \(\+41\.1%\)\. At 5% replay, the increase was 20–23%; at 1%, it was 3\.4%\. These costs depend on replay frequency: reducing only the MLM loss weight does not avoid its forward pass and vocabulary loss\.

To isolate MLM’s contribution, we later compared it with continued supervised training in an[encoder\-retention study](https://../encoder-drift-prevention.md)\. At learning rate3e\-5, we tested 15% MLM sampling with weights\.001/\.003/\.005/\.01/\.03, 5% with\.003/\.01, and 1% with\.015; none establishes a tagging gain\. Lower\-rate and additional\-seed tests likewise do not establish a consistent gain\. Hidden\-token prediction loss on fixed masks improves in several arms, but broader language competence was not measured\. These results concern separate natural\-text replay, not MLM on the existing labeled text\.

### A\.13Beyond mapped gold

This section describes transfer from Ont1 supervision to Ont2 and evaluates one Ont2 checkpoint\. On fresh English legal text it left a large gap: exact typed span F1 was 42\.5%, below the prespecified 85% target\.

#### A\.13\.1Ont1\-to\-Ont2 transfer

We reused an Ont1\-trained XLM\-R\-large encoder and created an affine head with rows for the Ont2 tags, initialized from the mapped Ont1 rows\. As Ont2 annotations became available, training continued to include Ont1\-labeled examples and human gold\. Reviewed many\-to\-many label mappings let those older annotations supervise the Ont2 head\.

Earlier transition experiments gradually reduced the old head’s loss weight to zero \(Section[A\.12\.1](https://arxiv.org/html/2609.38630#.A1.SS12.SSS1)\)\. The selected Ont2 lineage used zero old\-head loss weight from its first stage: Ont1 annotations continued to contribute through their projected supervision of the Ont2 head\. Training for O3, the Ont3 predecessor, added more varied annotated data, mapped human\-gold replay and mixed\-context training \(Section[6](https://arxiv.org/html/2609.38630#S6)\)\.

#### A\.13\.2Domain shift to fresh court text

Ont2 reached 86\.4% exact typed\-span F1 on 746 development documents from MEDDOCAN, MultiGraSCCo and TAB, compared with 32\.9% for stock GLiNER2 under the Ont2 projection\. This encouraging result covered 16 types, with much of the data clinical; TAB contributed court\-case text\. On new court documents from HUDOC, the European Court of Human Rights case\-law database, its accuracy fell to 42\.5% on 200 segments from 20 judgments \(Figure[21](https://arxiv.org/html/2609.38630#.A1.F21)\)\. Even new document types from a court already represented in training exposed a large remaining gap\.

The training mixture supplemented those clinical and legal corpora with translations and web text labeled by a 31\-billion\-parameter model\. The fresh HUDOC judgments were labeled directly in Ont2, whereas TAB’s labels reached Ont2 through a mapping\. They were also newer judgments sampled from every court section, unlike TAB’s case collection\. We suspected annotation\-inventory mismatch contributed most to the gap, followed by document type and insufficient training data; these explanations were not separately tested\.

Checked translations and inexpensive local\-model annotations had improved our earlier recipe, but had not solved this harder legal task\. We therefore shifted effort toward broader, directly annotated text and stronger frontier annotators, including Luna\. Ont3 intake retained quality\-checked, low\-temperature translations alongside original\-language annotations; subsequent expansion and mapped\-gold replay produced O3 and O4 \(Section[6](https://arxiv.org/html/2609.38630#S6); Section[A\.17](https://arxiv.org/html/2609.38630#.A1.SS17)\)\.

Figure 21:Ont2 development accuracy and fresh legal evaluation\. Exact typed span F1 is measured separately on each population\. The clinical\-heavy MEDDOCAN, MultiGraSCCo and TAB development set covers 16 types; the older Final20 retention set covers 29\. Fresh English HUDOC legal text reveals a large gap despite high development scores\.
#### A\.13\.3Shared\-label comparison on Ont3 text

In a precursor comparison separating evaluation difficulty from the label inventory, we applied Ont2 and an Ont3 model trained before mapped human\-gold replay was added to the same sentences of the Ont3 development collection, without neighboring context, and scored only their 29 shared primary types\. On the twenty languages supported by both models, Ont3 reaches 73\.2% exact typed span F1 versus Ont2’s 52\.5%; character\-redaction F1 is 91\.6% versus 76\.6% \(Table[27](https://arxiv.org/html/2609.38630#.A1.T27)\)\. With the scored labels held fixed, the principal change in difficulty is the more varied evaluation text\. The two additional reference labels form a separate task only when a score explicitly requires them\.

Table 27:Ont2 and the pre\-replay, sentence\-only Ont3 model on identical development text, with optional references\. The shared\-language difference is \+20\.68 span\-F1 points \(95% paired interval \[\+16\.24, \+25\.20\]\)\.ModelLanguagesSentencesExact typed span F1 \(%\)Character\-redaction F1 \(%\)Ont2Shared 2037452\.5276\.63Ont3Shared 2037473\.2191\.58Ont23565951\.3375\.09Ont33565972\.8991\.17The 374\-sentence subset contains 272 source documents; the complete set contains 468\. Scores pool counts across rows; intervals use 10,000 paired document resamples\. Both encoders use the same sentence\-only input, legal BIOES decoding and adjacent\-name joining, with no learned character\-boundary refiner or name/pattern postprocessors\. This comparison reuses development data\. Optional references are scored under the rule in Section[A\.9](https://arxiv.org/html/2609.38630#.A1.SS9)\.

### A\.14Character\-boundary refinement

Training for the character\-endpoint model of Section[5\.2](https://arxiv.org/html/2609.38630#S5.SS2), which moves span endpoints to correct encoder token boundaries, used TAB, MEDDOCAN and MultiGraSCCo, with language caps yielding 23,270 endpoint\-training groups\. We selected the boundary model on 601 documents sharing no source documents or exact texts with training\. We applied it to eleven trained languages \(ar, de, en, es, fa, fr, it, pl, ru, tr, uk\); other languages retained the BIOES endpoints\.

The main\-text curves \(Section[2](https://arxiv.org/html/2609.38630#S2)\) use a 35\-language continuation of this refiner\. It keeps the endpoint features and weights, adds seven span\-pair features that score a candidate’s start and end jointly, and never trims letters or digits; the radius stays ±1 character with the same movement penalty, and it remains a small hashed linear model\. It was trained on candidates around real tagger proposals in 10,537 segments, which share no exact or whitespace\-normalized text with either evaluation population \(human gold and the Ont3 development collection\)\. It was selected among fitted variants partly on the 659\-segment Ont3 development collection; fitted ±2 variants never beat ±1\. Relative to the eleven\-language refiner, the historical best redaction\-region F1 at 80% overlap of O3 moves by \+0\.04 on human gold and \+2\.3 over all 35 Ont3 languages \(\+0\.6 and \+3\.2 with exact boundaries\)\.

An English\-only fixed\-development contrast kept the token logits and legal BIOES path fixed and instead realized endpoints at Unicode word edges covered by each token\. This deterministic rule raised exact typed F1 on the MEDDOCAN / MultiGraSCCo / TAB evaluation from 78\.0% to 84\.7%, near the character refiner’s 85\.2%, but scored 84\.9% versus 88\.0% on the older Final20 windows\. Exact\-source\-text bootstrap supports the gain over unrefined BIOES on the former and the loss to the refiner on the latter; the 0\.49\-point gap to the refiner on the former is unresolved\. Inspected rows show the word rule dropping terminal date punctuation correctly while truncating punctuation\-bearing identifiers and names\. The comparison favors the flexibility of the learned character module; it does not test trained word pooling or multilingual word segmentation\.

#### A\.14\.1O4 contrast

We reuse O4’s token predictions at zero outside\-label bias, removing only endpoint adjustment\. Both arms exclude optional reference outputs and apply the same name and regex postprocessors\. The refined arm reproduces the main comparison’s counts exactly\.

Table 28:O4 without and with the ±1\-character boundary\-refinement model\. F1 and differences are percentage points; intervals use 10,000 paired document resamples on reused development populations\. The CPU replay and scoring took 18 seconds\.PopulationExact\-match taskWithout modelWith modelΔF1 \(95% interval\)Human gold, 1,283 segmentsRedaction regions87\.8787\.80−0\.07 \[−0\.23, 0\.00\]Ont3, 1,201 segmentsRedaction regions77\.7180\.20\+2\.49 \[\+1\.58, \+3\.50\]Ont3, 1,201 segmentsFine typed spans74\.6276\.30\+1\.68 \[\+1\.01, \+2\.41\]Refinement changes two human\-gold examples and 58 Ont3 examples\. Inspection finds both helpful punctuation removal from identifiers and harmful endpoint changes, including a Korean occupation span extended into a grammatical particle\. The result supports a benefit on Ont3, not a general improvement across annotation conventions\.

The original Ont2 audit found that 92% of gold spans were reachable at token boundaries\. Exact typed F1 rose from 80\.0 to 86\.4 with refinement; an oracle choosing among the same ±1 candidates reached 87\.8\. Tokens averaged 3\.4 characters, and 2,242 of 2,247 gold endpoints inside tokens were one character from an edge\. These historical diagnostics motivated the small search radius\.

### A\.15Human agreement on court text

TAB, the Text Anonymization Benchmark, preserves different human annotators’ span labels for identical text\. The released test set has multiple annotations for 105 of 127 documents \(82\.7%\)\. In our Ont2 evaluation, 84 of 112 English court\-case texts \(75%\) have two or more such annotations, giving 374 annotation records across nine mapped entity types: age, date, demographic attribute, location, organization, person name, quantity, record identifier and street address\. Annotators disagree somewhere in every one of those 84 texts, which measures annotation variability directly on fixed input text\.

We compared each annotator with every other annotator of the same text, requiring identical character boundaries and Ont2 entity types\. Pooling the 1,768 ordered comparisons gives 85\.5% human–human F1\. Scoring the Ont2 model with legal BIOES decoding and character\-boundary refinement against the same references, with the same weighting, gives 86\.9% F1\. These are descriptive agreement measurements on reused development data: disagreement with one annotator does not establish an error, and the comparison does not establish that the model is more accurate than a human annotator\. Nor is 85% a ceiling for human annotation: more carefully directed and monitored annotation can certainly reach agreement above 90%\.

Agreement may be understated if identical annotations were removed before release\. TAB describes retaining annotator layers separately but does not specify whether identical layers were deduplicated\([Pilán et al\., 2022](https://arxiv.org/html/2609.38630#bib.bib16)\); some identical span/type layers remain in the published data\. As a sensitivity example, if each of our 28 single\-annotation texts originally had two identical annotations, restoring those pairs would raise pooled human–human F1 from 85\.5% to 86\.0%\. The number of any omitted annotations is unknown, so this is a conditional calculation rather than an estimate of corrected agreement\.

### A\.16Ont3 context and data studies

The precursor Ont3 experiments below predate O4\. Compare effects within each matched study; the newer annotation\-rich comparisons are identified separately\.

Sweeping the outside\-label bias across 35 languages did not identify a need to fit separate per\-language operating points\. This descriptive check used only 15–22 sentences per language and required reference labels \(Section[A\.9](https://arxiv.org/html/2609.38630#.A1.SS9)\); it does not settle whether language\-specific BIOES weights would help\.

#### A\.16\.1Neighboring\-sentence context

Context experiments preserve the target sentence and fill the remaining space in a 512\-token window with preceding text, or with preceding and following text sharing that space\. Only target tokens receive tagging loss\.

Before the larger annotation pool, a frozen\-encoder experiment fitted heads on 16,634 sentences with matched additional exposure, about 2\.96 million target tokens per arm\. On 545 development sentences from 453 documents, excluding reference labels, exact typed F1 was 71\.87 for isolated input, 73\.16 with the preceding sentence and 73\.23 with both neighbors\. The apparent gains were \+1\.29 \(paired 95% interval \[\+0\.07, \+2\.53\]\) and \+1\.36 \[\+0\.09, \+2\.68\]; their difference was unresolved\. Unfreezing the top four blocks gave 72\.9\. These precursor results did not require updating the encoder on contextual inputs\.

A separate fully fine\-tuned comparison on 659 development segments, requiring non\-bare reference labels, scored 70\.10 for isolated training, 69\.26 for preceding\-context training and 69\.82 for both\-neighbor training\. Its context differences were unresolved\. The different reference policies prevent comparing absolute scores with the frozen\-encoder study\. Together, these results leave the mechanism of the apparent small context gain unresolved, including whether encoder exposure to neighbors during training was necessary\.

For O3, a 2,000\-update continuation mixed isolated, preceding\-only and both\-neighbor inputs equally\. Isolated Ont3 region F1 improved by 1\.1 points \(interval \[\+0\.01, \+2\.18\]\) with no detected full\-context regression\. There was no matched additional\-exposure control, and another 2,000 updates did not resolve a further gain\.

Under the richer annotation pool, the selected isolated\-trained O4 model loses 0\.44 points on the weighted selection criterion when given the preceding sentence \(paired interval \[−0\.90, 0\.00\]\); the two\-seed weight average loses 0\.34 \[−0\.81, \+0\.12\]\. Other checked seeds likewise give no consistent inference gain\. A continuation mixing contextual and isolated training loses 0\.30 points under isolated inference; a longer context\-only continuation peaks at an unresolved \+0\.16 before declining\. These bounded comparisons support isolated input for O4\. They do not settle whether context could help under another data budget or training recipe\.

#### A\.16\.2Annotated training\-pool size

To measure the benefit of a larger annotated training pool, we fine\-tune all layers of the large encoder on nested pools of 11,746, 23,183 and 47,641 windows, with the same 16,000\-update ceiling \(Figure[22](https://arxiv.org/html/2609.38630#.A1.F22)\)\. Increasing the pool from quarter to half size improves \+\_referenceexact span F1 on the 659\-row development set by 1\.7 points\. Increasing it from half to full size adds 0\.25 points; the paired interval includes zero\.

Figure 22:Effect of annotated training\-pool size on 659 development rows from 468 source documents, with non\-bare references retained in the full view\. Quarter and half support are nested document subsets; language weights are adjusted to meet the same coverage floors\. Every arm uses the same optimization ceiling and selector rule\. Lines connect the three observed pool sizes\. The intervals concern paired \+\_referencedifferences; they measure resampling uncertainty on reused development data\.

### A\.17Annotation volume, sampling and cost

O4 continues the isolated\-input branch of the Ont3 encoder for 4,000 updates on the expanded, title\-corrected pool, with 50% of sampling mass assigned to mapped human gold\. Its architecture and typed objective are unchanged\. The selected single checkpoint is seed 20260927; the two\-seed weight average differs by only 0\.04 points on the selection criterion, so we retain the simpler single\-model lineage\. The criterion pools language\-weighted counts from fine typed Ont3 \(18%\), exact Ont3 redaction regions \(12%\) and human\-gold regions \(70%\), using a 659\-row Ont3 subset and 23,251 human\-gold test examples repeatedly used in development\. For selection, optional reference labels are excluded\. The paper’s fine\-span scores additionally exempt a named\-entity prediction matching an optional reference at exactly the same offsets; this does not change the selected model\.

Continuing training beyond the selected 4,000 updates did not improve the recipe\.

The expanded pool contains 90,339 non\-human training windows and 80,829 human\-gold windows\. Exact deduplication by language and text leaves 88,512 and 80,099 respectively\. Compared with O3, 49,812 non\-human texts are new, containing 1,652,949 XLM\-R subword tokens, of which 378,431 intersect an annotated span\. No human\-gold texts are added\. Annotation changes affect 117 existing non\-human and 131 human\-gold texts, mainly title conventions; these changes do not add texts\. Counts concern sentence\-oriented windows, not source documents; repeated draws do not multiply annotation volume\. Optional reference spans count as supervision in these training totals\.

We sought annotation candidates in two ways\. Domain\-near retrieval ranks crawl paragraphs by similarity to examples from the evaluation domains, subtracting their similarity to other candidate paragraphs\. This correction downweights generic text that is near everything, rather than simply taking the nearest neighbors; it does not certify evaluation independence\. Rare\-entity “needles” instead match surface patterns, sometimes with checksum checks or nearby cue words\. A hit selects the whole paragraph, including sentences without a hit, for annotation in context\. The matches select candidates rather than supply gold labels\. Source\-bucket separation, used\-text tracking and lexical/semantic overlap screening precede annotation and admission to training\.

The annotation unions’ receipts identify 50,617 source requests and 52,631 recorded response attempts, including format/session retries and rejected annotations\. Every recorded attempt has token usage\. The original teachers are GPT\-5\.6\-Luna \(37,595 requests\) and GPT\-6\-Luna \(13,022\); repricing them does not change that provenance\. At the standard[GPT\-6\-Luna rates](https://developers.openai.com/api/docs/models/gpt-6-luna)retrieved September 29, 2026, one million tokens cost $0\.10 for uncached input, $0\.01 for cached input, $0\.125 for cache writes and $0\.50 for output\. Applying the documented long\-context multipliers per attempt gives$52\.66in estimated replacement charges\.444Usage includes language\-specific instructions and demonstrations, sentence inputs, generated outputs and repeated session history, charging cache reads and writes at their respective rates\. Prompt caching and consecutive same\-document sentence sessions amortize the prompt\. Saved retries while finding usable, economical session lengths are included; growing history is counted on every attempt\.This uses the recorded token lengths and cache behavior, not a counterfactual rerun or an invoice\. Transport failures without returned usage cannot be priced, and model training, data preparation and evaluation are outside this annotation total\. For the initial Ont3 evaluation collection, we spent roughly 100 times as much per segment as for training annotation\.

Increasing human\-gold sampling from 30% to 40% helps the combined criterion; 40–60% is comparatively flat \(Figure[23](https://arxiv.org/html/2609.38630#.A1.F23)\)\. There are two or three seeds per setting except 60%, which has one\. More human gold mainly trades fine Ont3 credit for human\-gold credit\. Croatian, Korean and Polish require language boosts above about 42% gold; Malay and Vietnamese also require them at 50%\. All final recipes satisfy the trainer’s 0\.75% per\-language floor\.

Figure 23:Gold sampling share at 4,000 updates on the expanded annotation pool\. Dots are individual seeds and lines connect their means; no fitted trend or uncertainty band is implied\. The selection criterion changes little above 40%, while higher shares favor human gold over fine Ont3\. The 60% setting has only one seed\.The headline O3–O4 contrast includes annotation, sampling and input\-recipe changes\. At fixed decoder bias, pooled fine typed F1 improves more than Ont3 redaction\-region F1, whose gain remains unresolved\. Paired intervals use 10,000 document resamples within each population, treating segments independently when their source document is unknown\. They condition on the selected recipes and do not account for the full development search\. The final 2,788\-row annotation increment had no measurable benefit over its predecessor pool in two matched seeds\. Neither that local result nor the sampling plateau determines the asymptote of training on further data\.

### A\.18Other tested approaches

#### A\.18\.1Context\-conditioned entity replacement

When translating annotated text, entity placeholders can be filled with the original values, corpus examples or generated replacements\. We used Faker for 43% of placeholder occurrences in the routed translation pool \(Section[A\.12\.5](https://arxiv.org/html/2609.38630#.A1.SS12.SSS5)\)\. Replacing entities can also vary untranslated training text, but a replacement must fit the sentence as well as its tag\. We explored drawing observed values and generating them with character models trained on the same corpus, using sentence context to guide both approaches\.

For observed donors, we compared uniform draws within a language and type with retrieval by masked sentence context\. On 140 queries across 35 languages, advisory reviews accepted 97 uniform draws and 107 context\-retrieved draws\. Character generators conditioned on nearby text, language and type received 58–63 accepts on the same queries\. Increasing model size from 2\.1 to 9\.9 million parameters, training longer and weighting surfaces by corpus frequency did not resolve malformed values or poor grammatical and semantic fit\. These were reviews of replacement quality, not tagger training results\.

A separate experiment included whole\-span categories, ordered name\-component kinds and binary predicate values in the generator’s conditioning\. Adding a frozen XLM\-R encoder of the masked sentence reduced mean character negative log likelihood from 1\.85 to 1\.77 on 489 source\-disjoint entity sites, but outputs still violated type and refinement constraints\. Predicate conditioning recorded presence within a span, not the locations of its subspans\. Using the full pattern of span refinements and subspan predicates could help reject incompatible replacements\. The larger native\-corpus fits above did not use refinement conditioning\.

We also tested replacement through tagger training\. Two Ont3 models trained for 16,000 updates on the same six pools\. One kept the original text; the other replaced organization names with observed donors at probability 0\.5 whenever an example was sampled\. On 818 development segments, exact organization F1 fell from 75\.1 to 68\.2, with more false positives and boundary errors\. We therefore kept original names in that recipe\. A shorter screen starting from O3 also found no benefit\. We continued O3 for 1,000 updates while replacing eligible organization names with observed same\-language donors, matched only by type, at probability 0, 0\.25 or 0\.5\. At the unchanged decoder bias, redaction\-region F1 stayed within 0\.6 points of the probability\-0 control on both Ont3 development text and human gold, and every paired 95% interval included zero\. At probability 0\.5, Ont3 organization predictions rose from 343 to 366, while correct ones rose only from 250 to 255\. This is consistent with some reliance on memorized organization names, though replacing names can also disrupt their fit to the sentence\.

Our positive result was narrower: repairing systematic name\-order and date\-format errors in translated training text improved tagging \(Section[A\.12\.5](https://arxiv.org/html/2609.38630#.A1.SS12.SSS5)\)\. Replacing intact values in original\-language Ont3 text did not help, so the final recipe keeps original surfaces\. Type\-only donor matching did not help, and fully conditioning a donor on its sentence and annotations leaves only the original value\.

#### A\.18\.2Consistency regularization

R\-Drop runs each labeled batch twice with independent dropout masks and penalizes disagreement between the two token\-label distributions\. We averaged the supervised losses and added a symmetric Kullback–Leibler divergence at supervised token positions; inference still used one pass\. The aim was to make predictions less sensitive to the random perturbations used in training\.

Added with penalty coefficient 1\.0 to an intermediate continuation, R\-Drop improved language\-macro\-averaged typed overlap F1 on the seven\-language Fresh7 suite \(Section[A\.8](https://arxiv.org/html/2609.38630#.A1.SS8)\) by 0\.6 points\. Added with the same coefficient to a later baseline recipe, it reduced application\-weighted typed overlap F1 by 0\.5 points \(95% interval \[−0\.80, −0\.20\]\) on the 1,267 Fresh20 development documents in twenty languages\. The seven\-language gain therefore did not justify adopting this setting in the later recipe\.

#### A\.18\.3Learned register positions

We added eight learned input positions that every token can attend to but that are never scored, each carrying a vector added at every encoder layer\. Against matched controls trained from the same pretrained encoder, exact typed span F1 on the 1,113 Ont3 development segments changed by \+0\.37, \+0\.61 and −1\.44 points across three seeds\. We did not adopt them\.

On the expanded annotation pool, eight all\-layer registers change the weighted selection criterion by \+0\.12 points at 4,000 updates; 32 registers change it by −0\.24\. Four all\-layer registers on O4’s full pool likewise give no resolved gain across four paired seeds \(typed F1 −0\.02 points, 95% interval \[−0\.70, \+0\.69\]; matched controls in Table[29](https://arxiv.org/html/2609.38630#.A1.T29)\)\. Adding eight registers to the 40%\-human\-gold recipe changes it by −0\.16\. These differences are small relative to the observed seed variation and do not support adding registers to O4\.

#### A\.18\.4Targeted annotation

Retrieving annotation candidates toward the evaluation domains with multilingual sentence embeddings did not beat random intake from the same languages in matched two\-seed comparisons\. The last 2,788 added rows changed the weighted criterion by \+0\.09 and −0\.25 points\. More annotation helped earlier in the expansion, but this final increment did not resolve a further gain\.

#### A\.18\.5Redaction\-loss weighting

Adding a binary entity\-versus\-outside loss to the typed objective changed the criterion by −0\.10 at weight 0\.5 and −0\.32 at weight 2\. The larger weight improved Ont3 redaction regions while losing human\-gold and fine typed accuracy\. These are bounded negative results, with single\-seed changes below roughly half a point difficult to distinguish from the observed seed variation\.

#### A\.18\.6Weight versus logit averaging

We compared averaging checkpoint weights into a single model \(a “soup”\) with averaging the models’ token logits before decoding\. The latter retains multiple encoder passes at inference; a soup uses one\. We combined separate training seeds, neighboring checkpoints from one trajectory, and a continuation with its ancestor\. Neither approach established a reliable gain over the better member; mixing in the ancestor could hurt\. Later multi\-seed soups likewise offered no clear advantage over the selected single O4 checkpoint, which we retained for its simpler lineage\.

#### A\.18\.7Category\-balanced sampling

We tried sampling more examples containing underrepresented labels, but abandoned this approach after finding no clear benefit\. We instead favor annotating more text\.

#### A\.18\.8Soft prompt tuning

Soft prompts prepend learned vectors to the text\. With the encoder frozen, we trained these vectors and the tagging head: ten shared tokens, or eight shared tokens plus two language\-specific tokens\. Separate experiments add register vectors at every encoder layer \(Section[A\.18\.3](https://arxiv.org/html/2609.38630#.A1.SS18.SSS3)\)\.

Type\-specific prompts use?,\+and−embeddings for unknown, present and absent status\. For example, a sentence containing a person and an organization but no location receives?PER ?ORG ?LOC \+PER \+ORG −LOCduring training and only?PER ?ORG ?LOCat inference\. Each symbol–type pair denotes its own learned vector\.

The first five arms start from the same PII\-trained Ont3 encoder and fit the affine head for eight epochs on the same 1,076 training sentences\. Evaluation reused 545 development sentences from 453 documents in 35 languages, containing 1,026 reference spans\. Supervision of Ont3’s\_referencelabels was disabled\. Decoding is constrained to legal BIOES label sequences\. Neither of the two shared\-token prompt arms improves on the no\-prompt arm; both paired intervals include zero\.

Table 29:Exact typed\-span soft\-prompt controls\. The first five rows use 545 development sentences, with\_referencesupervision and scoring disabled\. The O4 rows average four paired training seeds on all 1,201 Ont3 inputs, with optional references and O4’s serving chain\. Compare ΔF1 within each block; absolute scores have different populations and training conditions\. ΔF1 and paired 95% confidence intervals are relative to no prompt, in F1 points\. During training, each entity type receives a?token and a gold\-derived\+or−token; at inference, only the?tokens are visible to the encoder\. The 58\-token control uses learned tokens independent of gold status, with only the first 29 visible at inference\. For the 545\-input control, its interval versus no prompt is not reported here\. Times cover forward computation and decoding of all 545 sentences; O4 timing was not measured under those conditions\. O4 intervals resample both documents and the four paired seeds, 10,000 times\.PromptF1 \(%\)ΔF1 \(points\)Paired 95% CITime \(s\)No prompt72\.530—2\.6410 shared tokens71\.65−0\.880\[−2\.133, \+0\.394\]2\.598 shared \+ 2 language tokens71\.94−0\.587\[−2\.012, \+0\.848\]2\.5958 learned tokens, 29 at inference73\.16\+0\.63—2\.91Gold status: 58 active in train, 29 at inference73\.34\+0\.807\[−0\.514, \+2\.210\]2\.95O4: no prompt75\.740——O4: 58 learned, 29 at inference75\.48−0\.26\[−1\.11, \+0\.59\]—O4:? \+ −, 58 active, 29 at inference75\.61−0\.13\[−0\.86, \+0\.64\]—The O4 runs update the full encoder on its complete training pool for 4,000 steps, with a prompt learning rate of 0\.0003, as in the small\-pool status recipe\. Against matched continuations, neutral and status prompts change human\-gold exact\-region F1 by −0\.26 and −0\.02 points on 1,283 inputs, and Ont3 exact\-region F1 by \+0\.26 and \+0\.09\. All paired intervals include zero\.

With 29 entity types, the status arm has 29?tokens and 29\+/−slots, masking the latter at inference\. The 58\-token control scores 73\.2, leaving the status\-specific difference unresolved \(\+0\.18 points, interval \[−0\.359, \+0\.758\]\)\.

For the 545\-input study, intervals use 10,000 paired document resamples and are unadjusted\. Forward/decode timings cover all 545 examples at batch width four on an L40S, excluding model loading\.

With the richer annotation pool, supervision\-status slots again fail to improve their matched neutral\-slot control\. At 4,000 updates, neutral slots change the weighted selection criterion by \+0\.24 points relative to the plain recipe; gold\-status slots score 1\.94 points below neutral slots \(paired 95% interval \[−2\.62, −1\.30\]\)\. Status is available during training but hidden at inference\. Inspected errors show drift toward coarse human\-gold conventions, including locality predictions replaced by generic location and omitted health conditions\. We found no clearly helpful prompt recipe for our XLM\-R tagger\. Prompt tuning may still be useful when full encoder training is too expensive; that constraint does not apply here\.

#### A\.18\.9Binary entity\-presence supervision

In an Ont2 experiment using the Gemma\-4\-31B annotation pool \(Section[A\.12\.7](https://arxiv.org/html/2609.38630#.A1.SS12.SSS7)\), binary entity\-presence supervision trained the model to distinguish entity tokens from background without requiring a particular entity type\. Positive\-unlabeled training used known entity tokens and estimated entity prevalence from fully annotated text instead of treating every token the teacher left unmarked as background\. At approximately equal exposure to complete and positive\-only annotations, this objective raised native recall but lowered precision and F1\. We therefore continued to ignore unmarked tokens in this Gemma annotation pool\.

### A\.19Serving measurement conditions

The CPU throughput measurement in Table[2](https://arxiv.org/html/2609.38630#S9.T2)used 100 English legal segments \(3,250 tokens\) on a shared AMD EPYC 7R13 host; each figure is the median of three timed repetitions after one warmup, excluding model loading\. Both systems used model batch size 1; XLM\-R served 16 concurrent requests, while GLiNER2 served one request with 16 intra\-operation threads\. The GPU measurement used 487 documents in seven languages on an NVIDIA L40S, in single\-document batches, with one uncontended pass after eight warmup documents per language\.