Watermarking for Proprietary Dataset Protection

# Watermarking for Proprietary Dataset Protection
Source: [https://arxiv.org/html/2607.00325](https://arxiv.org/html/2607.00325)
###### Abstract

A growing body of literature suggests that training data membership inference problems are fundamentally hard tasks in modern language modeling settings\. We argue that output watermarking techniques are the right gadget to make training membership tests for generative models more tractable, based on prior results showing that language models exhibit residual watermark “radioactivity” under partially watermarked training datasets\. We pit a watermark\-based dataset inference approach head\-to\-head against traditional loss\-based membership inference methods and show that watermarking can achieve comparable membership detection performance when subset exposure is high enough, under an alternate set of assumptions\.

## 1Introduction

Modern language models perform complex knowledge work of growing economic value, but the regulatory frameworks governing fair use of the web\-scraped data they train on remain underdeveloped\. Recent litigation suggests content owners like news websites and independent authors may be entitled to compensation for inclusion of their datasets in large\-scale AI model training\. Answering such data\-use questions in high\-stakes settings requires a concrete definition of what it means to test whether some data was included in a model’s training dataset\.

While the fully general training data attribution problem asks how a model’s test\-time behaviors are caused by specific training instances, the question at hand in contemporary fair\-use deliberations is actually justmembership\. Membership inference attacks \(MIAs\) ask whether a specific sample was in a model’s training dataset; dataset inference attacks \(DIAs\) generalize this to whole collections\. As the more relevant setting for IP and generative\-model training disputes, in this work we study the DIA setup but refer to both problem types collectively as membership problems\.

![Refer to caption](https://arxiv.org/html/2607.00325v1/x1.png)Figure 1:To protect a proprietary dataset from unauthorized use in training, the dataset owner \(attacker\) paraphrases their documents with a secret watermark key\. To perform dataset inference, the attacker tests the suspect model’s predictions for evidence of the watermark key\. The watermark detection test is used to conclude whether their protected data was included in the training dataset\.What makes training dataset membership tests challenging in modern settings?Performing membership tests on modern generative models is harder than it was in discriminative settings for classifiers which mapped inputs to as few as 10 or 100 classes \(i\.e\. the setting ofShokriet al\.\([2017](https://arxiv.org/html/2607.00325#bib.bib4)\)\)\. The variable\-length output space matches the input cardinality, making analysis complicated, and the billion\-parameter sizes of modern models make classical attribution tools like influence functions difficult to apply\(Koh and Liang,[2017](https://arxiv.org/html/2607.00325#bib.bib3); Ilyaset al\.,[2022](https://arxiv.org/html/2607.00325#bib.bib9); Parket al\.,[2023](https://arxiv.org/html/2607.00325#bib.bib5)\)\. Even the definition of “a sample” is arbitrary at pretraining scale \(documents, sentences, words\), and individual samples overlap heavily in terms of shared n\-grams, which destabilizes existing membership tests\(Duanet al\.,[2024](https://arxiv.org/html/2607.00325#bib.bib6)\)\. Modern generative models also memorizeandgeneralize along both semantic and stylistic lines, blurring the line between membership and generation attributability: sample membership is often neither necessary nor sufficient for a target output to be produced\(Liuet al\.,[2025](https://arxiv.org/html/2607.00325#bib.bib7)\), and memorization rates vary widely\(Cooperet al\.,[2025](https://arxiv.org/html/2607.00325#bib.bib28)\)\.

Proactive Interventions for Reliable Dataset Inference\.Our work attempts to circumvent some of the difficulties described above by proactively but sparingly intervening during the construction of training datasets to make membership tests easier\. The essential operation will be to preciselymarkthe training dataset of a model in a way that does not significantly impact performance\. This intervention will give the marker \(data owner\) a significant informational advantage about how the marked elements of the training dataset are expected to influence the model’s test\-time predictions, and most importantly, how to probe for these effects\.

The main technical gadget we will use is language model output watermarking of the kind proposed and studied inKirchenbaueret al\.\([2023](https://arxiv.org/html/2607.00325#bib.bib1)\)\. This watermarked decoding algorithm will be layered on top of a “paraphraser” language model to create versions of candidate samples with the same style and semantics, but that produce significant p\-values under a watermark detection test\. Recent work has shown that if a target model is trained on watermarked samples, it will absorb the watermark key’s signature and emit a shadow of the same signature at test time\(Sanderet al\.,[2024](https://arxiv.org/html/2607.00325#bib.bib18)\)\. Then, when the attacker probes a suspect model using the watermark detection test, sample membership in this model’s training dataset is inferrable based on thepp\-value observed\.

#### Relationship to Prior Work\.

We build our study directly on two prior works\. The folding construction used as our finetuning backbone is adapted fromHayeset al\.\([2025](https://arxiv.org/html/2607.00325#bib.bib29)\), whose paired\-data MIA evaluation we extend to full\-subset membership \(DIA\)\. The proactive watermarking DI approach builds onSanderet al\.\([2024](https://arxiv.org/html/2607.00325#bib.bib18),[2025](https://arxiv.org/html/2607.00325#bib.bib19)\), who show that a target model trained on watermarked samples measurably reproduces the watermark’s key\-specific signature\. We adopt their sensitive “reading\-mode” detector that tests next\-token predictions \(argmaxes\) conditioned on watermarked prefixes rather than fully rolled\-out completions\. We vary per\-key support fractions and effective epochs on the marked subsets, replace the detection test’s parametric tail with an empirical\-null randomization test \([Section2\.2](https://arxiv.org/html/2607.00325#S2.SS2)\), and benchmark against loss\-based and reference\-model baselines—raw loss\(Yeomet al\.,[2018](https://arxiv.org/html/2607.00325#bib.bib39)\), argmax match, min\-k%k\\%\(Shiet al\.,[2024](https://arxiv.org/html/2607.00325#bib.bib37)\), zlib\(Carliniet al\.,[2021](https://arxiv.org/html/2607.00325#bib.bib30)\), and rMIA\-simple, rMIA, and LiRA\(Carliniet al\.,[2022](https://arxiv.org/html/2607.00325#bib.bib38)\)—on the same paired trials \([SectionC\.6](https://arxiv.org/html/2607.00325#A3.SS6)\)\.Our goal is to unify these settings and methods to calibrate claims about whether watermark\-based dataset membership testing is ready for use in practice\.

Contributions:

- •We formalize the general threat model for watermark\-based proactive dataset inference as a data\-owner\-centric security game, and contextualize prior work as studying a specific instance of this problem \([Section2](https://arxiv.org/html/2607.00325#S2)\)\.
- •We propose a randomization\-based variant of the watermark detection test that ensurespp\-value validity under interactions between pretrained checkpoints, intervention samples, and specific watermark keys \([Section2\.2](https://arxiv.org/html/2607.00325#S2.SS2)\)\.
- •We implement a unified experimental setting for evaluating traditional loss\-based dataset inference scores and watermark\-based approaches on the same paired trials, and compare the performance of each technique under its respective threat\-model assumptions \([Section3\.1](https://arxiv.org/html/2607.00325#S3.SS1.SSS0.Px3)\)\.
- •We ablate under\-explored dimensions from previous studies: per\-key support vs\. repetition at fixed training budget, training from scratch vs\. continued pretraining, and insertion schedule shape for the marked subset \([Sections3\.1](https://arxiv.org/html/2607.00325#S3.SS1)and[3\.2](https://arxiv.org/html/2607.00325#S3.SS2)\)\.

## 2Methodology

###### Definition 2\.1\(Threat Model: Proactive Dataset Inference for Data Owners\.\)\.

Alice suspects that a model owner Bob will train a language modelft​g​tf\_\{tgt\}onDt​r:=DA∪Dw​e​bD\_\{tr\}:=D\_\{A\}\\cup D\_\{web\}, whereDAD\_\{A\}is a portion Alice owns or controls and the rest is drawn from sources Alice does not control\. Alice wants a scored:ft​g​t,𝒟A→\(0,1\)d:f\_\{tgt\},\\mathcal\{D\}\_\{A\}\\to\(0,1\)that predicts whether Bob included any ofDAD\_\{A\}inDt​rD\_\{tr\}\. Alice is allowed to modifyDAD\_\{A\}using amarkingtransformationTm:𝒳→𝒳mT\_\{m\}:\\mathcal\{X\}\\to\\mathcal\{X\}\_\{m\}before Bob accesses it\. Attack success is evaluated against the ground\-truth membership labelm∈\{0,1\}m\\in\\\{0,1\\\}indicatingDAD\_\{A\}’s inclusion inDt​rD\_\{tr\}\. In the strictest setting, Alice has neither the compute to train shadow models forft​g​tf\_\{tgt\}nor a hidden shadow copy of similar dataDA′D^\{\\prime\}\_\{A\}to calibrate the detector against\.

###### Definition 2\.2\(Gadget: Watermark\-based Dataset Protection\.\)\.

Alice instantiatesTmT\_\{m\}using a generative modelfw​m:𝒳,k→𝒳w​mf\_\{wm\}:\\mathcal\{X\},k\\to\\mathcal\{X\}\_\{wm\}running an output watermarking scheme keyed by secret keykk, whose detectord​\(ft​g​t,k,DA\)d\(f\_\{tgt\},k,D\_\{A\}\)admits a test statisticz∈ℝz\\in\\mathbb\{R\}and associatedpp\-value\. She marks all or part ofDAD\_\{A\}viafw​mf\_\{wm\}relying on the assumption that Bob’sft​g​tf\_\{tgt\}trained on samplesx∈𝒳w​mx\\in\\mathcal\{X\}\_\{wm\}will produce new generations that yield significant detection scores underkk\. A decision functiong:ℝ→\(0,1\)g:\\mathbb\{R\}\\to\(0,1\)onzzthen implements the membership scored=g​\(z\)d=g\(z\), with thepp\-value itself being the obvious choice\. The null hypothesis is defined asℋ0:\\mathcal\{H\}\_\{0\}:Bob’s modelft​g​tf\_\{tgt\}was trained such thatDA∩Dt​r=∅D\_\{A\}\\cap D\_\{tr\}=\\emptyset, or more precisely, such that none of the samples inDt​rD\_\{tr\}are marked withkk\. Assuming the canonical choice ofg​\(⋅\)g\(\\cdot\), if app\-value belowα\\alphais observed, Alice rejects the null and concludes that Bob likely trainedft​g​tf\_\{tgt\}on her dataset\.

### 2\.1Target Model Assumptions

We assume that the target model’s tokenizer is known and matches the paraphraser’s\. Prior work already shows that this assumption can be relaxed and watermark detectability still preserved through common\-token filtering\(Sanderet al\.,[2024](https://arxiv.org/html/2607.00325#bib.bib18),[2025](https://arxiv.org/html/2607.00325#bib.bib19); Xuet al\.,[2026](https://arxiv.org/html/2607.00325#bib.bib35)\), so we omit experiments along this axis here\. We also assume that the attacker can query the target model for next\-token\-prediction probabilities conditioned on a prefix—the “reading mode” detection procedure ofSanderet al\.\([2024](https://arxiv.org/html/2607.00325#bib.bib18)\)\. This is a costly assumption in practice if one is making calls to a production API for every token\. However, the increased sensitivity of the data\-forced reading\-mode test is necessary at the support fractions we consider: naively rolled\-out completions from a target model that has only been mildly exposed to the watermark signature do not yield low enoughpp\-values for reliable detection\. Improvements in detection that would let the attacker relax the reading\-mode assumption are an obvious axis of future work but are out of scope here\.

### 2\.2Ensuring P\-value Validity

Prior work on watermark\-based DIAs reports significantpp\-values under intervention and near\-null values otherwise\(Sanderet al\.,[2025](https://arxiv.org/html/2607.00325#bib.bib19)\)\. However, our preliminary reproduction experiments suggested that interactions between the model checkpoint, the particular samples in the intervention dataset, and the specific watermark key used can invalidate the independence assumptions on which standard parametric tail bounds depend, even under careful de\-duplication and especially when the reading\-mode detector is in use\. Therefore, we propose a standard randomization\-based \(permutation\) hypothesis test variant that establishes a validpp\-value based on an empirical null hypothesis under exchangeability assumptions\. This method allows us to configure the resolution of the exactpp\-value and adds only a small computational overhead to the attacker’s testing protocol\.

###### Definition 2\.3\(Watermark Key Randomization Test for Exact P\-Values\)\.

Assume Alice’s watermark secretkik\_\{i\}is sampled i\.i\.d\. from a distribution of possible keys suitable for the PRF used by the watermark schemeki∼𝒦k\_\{i\}\\sim\\mathcal\{K\}\. Under the null hypothesis,ft​g​tf\_\{tgt\}has not been trained on data marked withkik\_\{i\}nor any other keykjk\_\{j\}in𝒦\\mathcal\{K\}111This is a strong, but plausible assumption if a sufficiently large, private keyspace is used\.\. Thus all keyskj≠kik\_\{j\}\\neq k\_\{i\}are exchangeable w\.r\.t\. the test statistic produced by the detectord​\(ft​g​t,k,DA\)d\(f\_\{tgt\},k,D\_\{A\}\)\. To compute the exact p\-value using the randomization test, Alice samplesM−1M\-1distinct additional keys, and then produceszj=d​\(ft​g​t,kj,DA\),∀j∈Mz\_\{j\}=d\(f\_\{tgt\},k\_\{j\},D\_\{A\}\),\\forall j\\in Mincluding her keykik\_\{i\}\. TheMMscores are sorted in descending order to determine each key’s resulting rankrjr\_\{j\}\(highestzzyields smallestrr\)\. The exact p\-value of the test isri/Mr\_\{i\}/M, the rank of Alice’s actual key amongstMM\. Under the null, due to exchangeability, the probability of observing her key at rankrir\_\{i\}or earlier in the list is exactlyP​\[r≤ri\|ℋ0\]=ri/MP\[r\\leq r\_\{i\}\|\\mathcal\{H\}\_\{0\}\]=r\_\{i\}/M, making this a valid p\-value under no additional assumptions\.

Test\-time cost is controlled using the vectorized PRF implementation intextseal\(Sanderet al\.,[2025](https://arxiv.org/html/2607.00325#bib.bib19); Fernandezet al\.,[2025](https://arxiv.org/html/2607.00325#bib.bib36)\)to evaluate many watermark secrets in parallel for any given piece of text\. We verify the validity of the resulting empirical null in practice by confirmingpp\-value uniformity using a KS test \(see[Figures4](https://arxiv.org/html/2607.00325#S3.F4),[14](https://arxiv.org/html/2607.00325#A3.F14)and[22](https://arxiv.org/html/2607.00325#A4.F22)\)\. Forpp\-values below the1/M1/Mresolution of the empirical sample, we report an extrapolated empirical\-Gaussian fit to the same null distribution where useful\.

![Refer to caption](https://arxiv.org/html/2607.00325v1/x2.png)

![Refer to caption](https://arxiv.org/html/2607.00325v1/x3.png)

Figure 2:Finetuning event\-split matched clean\-twin false\-probe null\(left\)and keyed signal\(right\)on the aligned unpacked detection surface, scored as−log10⁡p\-\\log\_\{10\}punder an empirical\-exact reference\.

## 3Experiments

Our experiments are built around a controlled data\-folding design that allows us to mimic prior studies on membership and dataset inference in language models while simultaneously benchmarking the proactive watermarking approach\. In a smaller finetuning regime, we systematically vary the level at which the model is exposed to the intervention data by modulating subset size and repetition during training, before moving to a larger 10B\-token training regime where we ablate the insertion schedule of the marked subset and the choice of initialization \(continued pretraining versus from\-scratch\)\.

#### Setup\.

In our experiments, we strive for depth rather than breadth\. Thus, we use a single language modelQwen3/Qwen3\-1\.7B\(Yanget al\.,[2025](https://arxiv.org/html/2607.00325#bib.bib26)\)in all experiments, either as a pretrained set of weights or as the architectural specification for from\-scratch experiments\. Similarly, we adopt a single dataset ofN=1500N=1500natural\-looking but semantically isolated documents designed to be used in controlled experiments on memorization: FictionalQA\(Kirchenbaueret al\.,[2025](https://arxiv.org/html/2607.00325#bib.bib41)\)\. This dataset includes webtext\-like documents in generic styles about totally fictional entities andevents—collections of related documents about the same fictional scenario \(an explanatory blog post, a news article about the event, etc\.\)\. Our*event\-split*fold construction \([Section3\.1](https://arxiv.org/html/2607.00325#S3.SS1)\) uses the event groupings present in the dataset: fold boundaries respect events so that documents about the same fictional event stay together inside a single fold\. We mix our controlled intervention data into a base random subset ofallenai/dolma3\_mix\-150B\-1025\(Olmoet al\.,[2025](https://arxiv.org/html/2607.00325#bib.bib40)\)with both sources shuffled and packed into length\-4096 chunks\.

#### Dataset Membership Testing\.

During each dataset inference simulation trial, for the watermarking methods, for positive cases we report the pooled, deduplicated reading\-mode test statistic and correspondingpp\-value measured on each of the watermarked folds for each of the models that trained on it\. For the negative cases, we compute the same score using all the same watermarked folds as probe data, but on the corresponding clean \(twin\) model trained on the unmarked version of the data\. For the non\-watermarking methods, the original clean samples from the fold are fed through the corresponding clean models to compute losses, and then each technique’s corresponding algorithm is used to produce the row\-level MIA or whole\-model DIA score, potentially using the rest of the clean models as reference models\. Throughout, we present results in terms of−log10⁡p\-\\log\_\{10\}pfor keyed signal strength, reporting the exact empirical\-null reference distribution \([Section2\.2](https://arxiv.org/html/2607.00325#S2.SS2)\) and/or an extrapolated empirical\-Gaussian p\-value where useful\.

![Refer to caption](https://arxiv.org/html/2607.00325v1/x4.png)

![Refer to caption](https://arxiv.org/html/2607.00325v1/x5.png)

Figure 3:Finetuning event\-split watermark whole\-model DIA AUC across theF×EF\\times Egrid, on the aligned\(left\)and packed\(right\)detection surfaces, both scored against an empirical\-exact null\. The aligned chart on the left is based on the same probe surface as the right panel of[Figure2](https://arxiv.org/html/2607.00325#S2.F2), paired here with the packed\-surface companion\. On the aligned surface the AUC saturates in the high\-exposure corner; on the packed surface the more permissive oracle recovers several lower\-exposure cells\.We score the watermark detection statistic under two surface variants\. Thealignedsurface scores each original document in isolation and is the most realistic setting; Alice has no knowledge of exactly how her documents will be truncated or concatenated together\. Thepackedsurface scores the 4096\-token chunks of multiple fictional documents packed exactly as they were at training time, and serves as an oracle baseline that quantifies how much keyed signal could be lost to train\- vs\. inference\-time mismatches\. We score the same readout under both an empirical\-Gaussian and an exact empirical\-null reference distribution \([Section2\.2](https://arxiv.org/html/2607.00325#S2.SS2)\), and report headline results in−log10⁡p\-\\log\_\{10\}pthroughout \(higher means a more significant signal\)\.

Table 1:Idealized per\-key exposureE/FE/Ffor the finetuning grid, expressed as a multiple of the per\-key support that a single epoch over a single fold would produce\. Distinct watermarked tokens scale as1/F1/F; epoch repetitionEEmultiplies how many times those tokens are seen during training\. The event\-split and SKS regimes share this idealized table\. Realized exposure based on the actual sampling rates during the training runs is reported in[SectionsC\.4](https://arxiv.org/html/2607.00325#A3.SS4)and[D\.3](https://arxiv.org/html/2607.00325#A4.SS3)\.
### 3\.1Finetuning: Varying Per\-Key Exposure

In our finetuning\-scale event\-split design, multiple keyed folds of sizeN/FN/Fare mixed into each model’s training data with the total watermarked share of the training tokens held fixed acrossFF\. AsFFgrows, each individual key’s per\-fold footprint shrinks while the number of distinct keys observed by the model grows\. We populate every cell of theF×EF\\times Egrid with enough watermarked\-positive and matched clean\-false\-probe trials to make whole\-model dataset inference comparisons stable\. The per\-cell training token budget is held fixed at 131M tokens\. The interventional design is summarized in[Table1](https://arxiv.org/html/2607.00325#S3.T1)\(idealized per\-key exposureE/FE/F, shared with the SKS ablation\); the matched realized normalized exposureE^/F\\hat\{E\}/Fand the underlying realizedE^\\hat\{E\}values live in the appendix \([Tables3](https://arxiv.org/html/2607.00325#A3.T3)and[4](https://arxiv.org/html/2607.00325#A3.T4)\)\. In absolute terms, the watermarked share of each model’s training tokens swings from a low of about 71k tokens \(≈0\.05%\{\\approx\}0\.05\\%of the run\) at\(F=16,E=1\)\(F\{=\}16,E\{=\}1\)up to about 8\.3M tokens \(≈6\.3%\{\\approx\}6\.3\\%\) at\(F=2,E=16\)\(F\{=\}2,E\{=\}16\), and per\-cell whole\-model DIA trial counts scale withFF, from2/22/2positive/negative trials per cell atF=2F\{=\}2to96/9696/96atF=16F\{=\}16\([Tables5](https://arxiv.org/html/2607.00325#A3.T5)and[6](https://arxiv.org/html/2607.00325#A3.T6)\)\.

#### Increased signal as a function of exposure \(E/FE/F\)\.

[Figure2](https://arxiv.org/html/2607.00325#S2.F2)demonstrates the expected trends: increasingEEproduces strongly significant keyed signal whenever the underlying support is sufficient, while increasingFFat fixedEEweakens the signal as each individual key occupies less of the corpus \([Figure11](https://arxiv.org/html/2607.00325#A3.F11)makes this more obvious\)\. The DIA AUC view \([Figure3](https://arxiv.org/html/2607.00325#S3.F3)\) tracks the keyed\-signal ordering and saturates at1\.01\.0in the high\-exposure corner, while the low\-exposure corner sits near chance\. The matching whole\-model DIA comparison against loss\-based and reference\-model baselines on the same paired trials is reported in[Table2](https://arxiv.org/html/2607.00325#S3.T2); the corresponding row\-level MIA, matched false\-probe\-null, cross\-key sham\-null, and packed\-surface counterparts are deferred to the appendix in[AppendixC](https://arxiv.org/html/2607.00325#A3)\. The single\-key support \(SKS\) ablation, which holds each model to one watermarked fold so per\-model fictional support drops as1/F1/F, is reported in[AppendixD](https://arxiv.org/html/2607.00325#A4)as a structurally simpler reference point\.

#### Event\-Split Null\-Validity

[Figure4](https://arxiv.org/html/2607.00325#S3.F4)confirms that, once pooled across many distinct positive keys, the empirical exact null is close to uniform at the 1M\-null scale and respects standard tail\-rate thresholds, supporting the use of the empirical\-null permutation test as the headline reference distribution for the keyed readout\. Per\-key idiosyncrasy is a separate concern visible in the right\-hand trace of the extended chart[Figure14](https://arxiv.org/html/2607.00325#A3.F14): at very small fold counts a single key can run systematically warm even when the pooled null is well\-calibrated, and the inheritedF=2F=2scaffold reused in pretraining \([Section3\.2](https://arxiv.org/html/2607.00325#S3.SS2)\) carries one such warm key\.

![Refer to caption](https://arxiv.org/html/2607.00325v1/x6.png)Figure 4:Event\-grid null\-validity panel\. Histogram of pooled empirical exactpp\-values from the packed matched\-clean\-negative whole\-model readings across the\(F,E\)\(F,E\)event grid\. The annotated statistic andpp\-value are from a one\-sample Kolmogorov\-Smirnov test of these pooledpp\-values againstUniform​\(0,1\)\\mathrm\{Uniform\}\(0,1\); the dashed horizontal line is the expected per\-bin count under a uniform histogram with the plotted binning\.
#### Loss\-Based and Reference\-Model Baselines

[Table2](https://arxiv.org/html/2607.00325#S3.T2)reports the fold\-level whole\-model DIA comparisons for the event\-split finetuning grid \(for the decision problem “was modeliitrained on foldii?”\), with the watermark method reported alongside the loss\-based and reference\-model baselines on the same paired trials\. The watermark whole\-model DIA AUC values reproduce those visualized in[Figure3](https://arxiv.org/html/2607.00325#S3.F3)\. Row\-level MIA performance for the loss\-based methods is reported in[Table7](https://arxiv.org/html/2607.00325#A3.T7)\. Using additional access to calibration samples \(“Loss\-based” ref\-free\) and/or the rest of the models in the pool trained and not trained on each fold as a reference pool \(“Ref\-model”\), the baselines perform nearly perfectly with the exception of the very low\-supportF​8F8andF​16F16single\-epoch cells\. While valid to run aboveF=2F=2, LiRA still suffers in theF​4F4cells when the ref\-model pool is too small for accurate in/out ratio estimation \(see[Table6](https://arxiv.org/html/2607.00325#A3.T6)\)\. We also render a compact summary table of access assumptions for the various approaches in the appendix \([Table8](https://arxiv.org/html/2607.00325#A3.T8)\)\.

Table 2:Event\-split finetuning: fold\-level whole\-model DIA AUC comparison\. Entries marked N/A indicate statistics that are not estimable in the available cell geometry; for example, LiRA in F=2 cells lacks sufficient in\-reference models\.

### 3\.2Pretraining: Sensitivity at Scale

To address whether the keyed readout remains detectable under much heavier dilution, we conduct a second, narrower batch of experiments at a 10B\-token total budget\. We reuse a similarF=2F=2fold/key scaffold from finetuning \(two groups of randomly sampled fictional documents of sizeN/2N/2, simpler than the event\-split construction\) and sweep ten different insertion schedules for these two keyed folds\.

The two initialization regimes are continued pretraining \(CPT\) from the Qwen3\-1\.7B checkpoint and from\-scratch \(random init\) at the same total token budget\. The four single\-burst schedules\(S​1,E​1\)\(S1,E1\),\(S​2,E​1\)\(S2,E1\),\(S​3,E​1\)\(S3,E1\), and\(S​4,E​1\)\(S4,E1\)insert the watermarked fold once at one of four step locations within the run \(targetE=1E=1\)\. The remaining six schedules pair a uniform\-spacing variant \(denotedUU\), which uses a constant sampling rate targeting∼E\\sim Eepochs by end of run, with a periodic\-cluster variant \(denotedPP\) that schedulesEEdumps of the entire fold at evenly spaced steps, very similar toSanderet al\.\([2025](https://arxiv.org/html/2607.00325#bib.bib19)\)’s setup\. We cross these withE∈\{4,8,16\}E\\in\\\{4,8,16\\\}\. The idealized exposure profile of these schedules atF=2F=2is summarized in[Table14](https://arxiv.org/html/2607.00325#A5.T14); the matched realized normalized exposure tables and per\-schedule realizedE^\\hat\{E\}values for both initialization regimes also live in the appendix \([Tables17](https://arxiv.org/html/2607.00325#A5.T17),[18](https://arxiv.org/html/2607.00325#A5.T18),[19](https://arxiv.org/html/2607.00325#A5.T19)and[20](https://arxiv.org/html/2607.00325#A5.T20)\)\.

Main body figures use the exact empiricalpp\-value for both CPT and from\-scratch training\. Each schedule contributes 2 watermarked\-positive and 2 matched clean\-negative whole\-model trials per init, and \(mirroringSanderet al\.\([2025](https://arxiv.org/html/2607.00325#bib.bib19)\)\) the watermarked\-token share of each 10\.49B\-token training run is small in absolute terms—about0\.5​M0\.5Mtokens \(≈0\.005%\{\\approx\}0\.005\\%\) for the four single\-burstE=1E\{=\}1schedules, rising to about8​M8Mtokens \(≈0\.077%\{\\approx\}0\.077\\%\) at theE=16E\{=\}16schedules \([Tables21](https://arxiv.org/html/2607.00325#A5.T21),[23](https://arxiv.org/html/2607.00325#A5.T23),[22](https://arxiv.org/html/2607.00325#A5.T22)and[24](https://arxiv.org/html/2607.00325#A5.T24)\)\.

#### Schedule and Initialization Effects\.

Across both inits, schedule shape matters: the four single\-burstE=1E\{=\}1schedules are not distinguishably higher than their null counterparts in[Figure5](https://arxiv.org/html/2607.00325#S3.F5), while the multi\-insertionUUandPPschedules become clearly separable from null asE^\\hat\{E\}grows\. Under aligned testing thePPandUUschedules are similar, but the more sensitive packed\-surface readings \([Figures24](https://arxiv.org/html/2607.00325#A5.F24)and[25](https://arxiv.org/html/2607.00325#A5.F25)\) show that thePPsetting produces a stronger signal for CPT\. From\-scratch recovers substantially stronger keyed signal than CPT at highE^\\hat\{E\}, while at lowE^\\hat\{E\}the two inits are comparable, though[Figure6](https://arxiv.org/html/2607.00325#S3.F6)shows slightly more separability atE^=4\\hat\{E\}=4for from\-scratch than for CPT\.[Figures29](https://arxiv.org/html/2607.00325#A5.F29)and[30](https://arxiv.org/html/2607.00325#A5.F30)show that theUUsampling strategy produces more signal in the from\-scratch setting, the opposite of the CPT result\.

![Refer to caption](https://arxiv.org/html/2607.00325v1/x7.png)

![Refer to caption](https://arxiv.org/html/2607.00325v1/x8.png)

Figure 5:Pretraining signal across the ten\-schedule sweep on the aligned unpacked detection surface, scored as−log10⁡p\-\\log\_\{10\}punder the exact empirical\-null reference, for the CPT initialization\.\(Left\)is the clean\-twin false\-probe\-null models and\(right\)is the keyed model trained on the watermarked data\. Schedule clearly modulates the keyed readout: atE​1E1, none of the keyed signals are separably significant, but at largerEEa gap appears\. One of the two inherited keys runs visibly warmer than the other in both panels\. Matched packed\-surface variants, extrapolated Gaussian p\-value versions, and watermark whole\-model DIA bar charts are all reported in the appendix \([SectionE\.1](https://arxiv.org/html/2607.00325#A5.SS1)\)![Refer to caption](https://arxiv.org/html/2607.00325v1/x9.png)

![Refer to caption](https://arxiv.org/html/2607.00325v1/x10.png)

Figure 6:Pretraining signal across the ten\-schedule sweep on the aligned unpacked detection surface, scored as−log10⁡p\-\\log\_\{10\}punder the exact empirical\-null reference, for the from\-scratch initialization\.\(Left\)is the clean\-twin false\-probe\-null models and\(right\)is the keyed model trained on the watermarked data\. The schedule strengthens the keyed readout much faster as a function ofEEin this setting versus CPT: atE​1E1there is a slight trend as a function of insertion point \([Figure29](https://arxiv.org/html/2607.00325#A5.F29)makes this more obvious\) atE​8E8a significant gap appears andE​16E16maxes out the exact resolution\. Here, there is little if any systematic bias between the watermark keys\. Matched packed\-surface variants, extrapolated Gaussian p\-value versions, and watermark whole\-model DIA bar charts are all reported in the appendix \([SectionE\.2](https://arxiv.org/html/2607.00325#A5.SS2)\)
#### Fold/Key Asymmetry\.

The pretraining experiments reuse the same two folds and keys that the finetuning SKSF=2F=2row was conducted with\. At the finetuning scale, one of those two keys already produces systematically larger detection statistics than the other under matched\-clean conditions, even when no watermark training signal is in play \(right side of[Figures14](https://arxiv.org/html/2607.00325#A3.F14)and[22](https://arxiv.org/html/2607.00325#A4.F22)\)\. This handedness is visible again in[Figure5](https://arxiv.org/html/2607.00325#S3.F5), where Key 0 consistently produces higher signal than the other under matched conditions \(more obvious in[Figure24](https://arxiv.org/html/2607.00325#A5.F24)\)\. This suggests that per\-key interactions with the data and/or the pretrained checkpoint can produce scenarios where positive model signal is spuriously low or high, or false\-positive rates could exceed what the nominalpp\-value threshold would indicate\. It is possible that prior studies filtered for well\-behaved watermark keys, and our systematic evaluation just reports this issue more transparently\.

#### Comparison to loss\-based baselines\.

A complete loss\-based and reference\-model whole\-model DIA head\-to\-head comparison for both pretraining inits is reported in the appendix \([AppendicesE](https://arxiv.org/html/2607.00325#A5),[26](https://arxiv.org/html/2607.00325#A5.T26)and[28](https://arxiv.org/html/2607.00325#A5.T28)\)\. As in the finetuning regime, the loss\-based approaches perform nearly perfectly across all pretraining schedules including the low\-exposureE​1E1insertions\. As the differences between[Figure6](https://arxiv.org/html/2607.00325#S3.F6)and[Figure5](https://arxiv.org/html/2607.00325#S3.F5)would suggest, the watermark\-based approach is only able to separate the positives and negatives clearly when the models are trained from scratch at the higher\-exposureE​8E8andE​16E16settings\.

## 4Discussion

As stated in the introduction, our setup unifies and builds upon two prior studies:Sanderet al\.\([2025](https://arxiv.org/html/2607.00325#bib.bib19)\)andHayeset al\.\([2025](https://arxiv.org/html/2607.00325#bib.bib29)\), rather than proposing a wholly novel methodology\. Therefore, it is important to identify where our results cohere with and diverge from those studies\. After presenting our analysis, we discuss the feasibility of Bob, the model trainer, “defending” against this dataset protection strategy\.

#### Watermark Detectability Trends\.

The trends we observe in both[Section3\.1](https://arxiv.org/html/2607.00325#S3.SS1)and[Section3\.2](https://arxiv.org/html/2607.00325#S3.SS2)on sensitivity growing with exposure to the watermarked subset match expectations fromSanderet al\.\([2025](https://arxiv.org/html/2607.00325#bib.bib19)\)\. Even though that work does not directly ablate the size of the watermarked subset as we do at finetuning scales, we expect that, were resources sufficient to test it, the drop in significance as a function of increasingFFthat we observe in[Section3\.1](https://arxiv.org/html/2607.00325#S3.SS1)would also appear at the pretraining scale\.

#### Performance of Loss\-based Methods\.

WhileHayeset al\.\([2025](https://arxiv.org/html/2607.00325#bib.bib29)\)did not run as complete a controlled exposure grid as our\(F,E\)\(F,E\)sweep in[Section3\.1](https://arxiv.org/html/2607.00325#S3.SS1), the loss\-based methods’ row\-level MIA success across the exposure grid varies in the very same way as the watermark signal strength \([SectionC\.6](https://arxiv.org/html/2607.00325#A3.SS6)\)\. However, they almost perfectly solve the DI problem at all exposure levels anyway \([Table2](https://arxiv.org/html/2607.00325#S3.T2)\)\. As noted inHayeset al\.\([2025](https://arxiv.org/html/2607.00325#bib.bib29)\), LiRA struggles at low ref\-model counts\.

#### Sensitivity at Scale and FPR Control Issues\.

We observe relatively similar−log10⁡p\-\\log\_\{10\}pvalues forP​E​4PE4,P​E​8PE8, andP​E​16PE16for the models trained on the watermarked data in[Section3\.2](https://arxiv.org/html/2607.00325#S3.SS2)to analogous values reported inSanderet al\.\([2025](https://arxiv.org/html/2607.00325#bib.bib19)\)\(our fold size vs\. training budget are approximately matched here\)\. However, this is only true when the model is trained from scratch, somethingSanderet al\.\([2025](https://arxiv.org/html/2607.00325#bib.bib19)\)did not ablate, and significance swings sharply depending on train\- vs\. detect\-probe style \(aligned/packed\)\. The finetuning and CPT settings also reveal that a spuriously hot watermark key might confound the results or cause false positives \(see[SectionsC\.7](https://arxiv.org/html/2607.00325#A3.SS7)and[D\.6](https://arxiv.org/html/2607.00325#A4.SS6)\)\. The point estimates reported inSanderet al\.\([2025](https://arxiv.org/html/2607.00325#bib.bib19)\)make it hard to determine whether this was an issue in their study\. However, they do report a−log10⁡p\-\\log\_\{10\}prange of0\.30\.3to0\.90\.9in Table 1 at zero contaminations which is broadly in line with our[Figure6](https://arxiv.org/html/2607.00325#S3.F6)null side readings, but crucially this is in the from\-scratch setting where there is no preexisting pretrained checkpoint for the specific watermark key to potentially interact with\. Our results show that even under a well\-calibrated exact p\-value test, spurious key\-checkpoint interactions still matter for any single key in any single deployment attempt\. Therefore, Alice may need to filter for “quiet” keys that behave well under the null hypothesis for representative data and models\.

#### Can the Model Trainer Mount a Defense?

The marking transformationTm:𝒳→𝒳mT\_\{m\}:\\mathcal\{X\}\\to\\mathcal\{X\}\_\{m\}is intentionally notated to bring attention to the fact that𝒳\\mathcal\{X\}and𝒳m\\mathcal\{X\}\_\{m\}are distinguishable,but only to Alice\. While a data owner does not necessarily need to mark their data in a way that satisfies a formal third\-party indistinguishability condition \(a laChristet al\.\([2024](https://arxiv.org/html/2607.00325#bib.bib14)\)\) for the attack to be successful, in practice, a marking scheme where the model trainer is unable to reliably \(orscalably\) determine whether or not a sample is in𝒳m\\mathcal\{X\}\_\{m\}is likely to be much more effective and tamper\-proof\. If Bob cannot tell the difference between samples inDt​rD\_\{tr\}that are marked or not, then Bob has no way of knowing whether or not Alice has marked the data, nor can Bob efficiently filter the dataset to removeDAD\_\{A\}from it\.

If we assume that there exist watermarking methods such that i\) watermarked outputs have the same quality as non\-watermarked samples, ii\) watermarked samples are indistinguishable from non\-watermarked samples for computationally\-bounded discriminators, and such that iii\) an observer can readily tell the distributions apart if the observer does possess the watermark secret, then a paraphrasing model equipped with an output watermark may bethetheoretically optimal marking transformationTmT\_\{m\}\(seeChristet al\.\([2024](https://arxiv.org/html/2607.00325#bib.bib14)\); Pietet al\.\([2025](https://arxiv.org/html/2607.00325#bib.bib16)\)for some discussion of these ideas in more context\)\. Such an invisible, “non\-distortionary” watermark family might serve as a better instantiation ofTmT\_\{m\}in the proactive dataset protection methodology we study in this work\. However, whether the the implicit “learnability” required for the marking to be detectable in Bob’s trained model is in fundamental tension with indistinguishability properties is an open question and a fruitful topic for future research\.

## 5Conclusion

We find the watermark\-based dataset inference approach to be a promising alternative to traditional loss\-based DIA methods, but overlooked details in previous studies highlight important avenues for future research\. Despite our randomization\-based detection test yielding a well\-calibrated empirical null, the possibility for elevated single\-key readings in any setting presents a previously overlooked deployment challenge\. Further, any train\- vs\. detection\-time mismatch in document presentation \(aligned/packed\) can reduce test significance values\. Finally, the reading\-mode detection procedure from prior work that is required for signal at high dilution may itself be impractical depending on the API access the data owner has against the suspect target model\.

In head\-to\-head comparisons under a controlled experimental design, the loss\-based baselines are more performant as DIA methods under an ROC\-AUC evaluation\. However, by definition they do not admit p\-values for assumption\-free and calibration\-free FPR estimation and they assume greater access to the target model \(loss values\), some known non\-member, distributionally\-matched calibration samples, and/or a pool of trained reference models \([Table8](https://arxiv.org/html/2607.00325#A3.T8)\)\. Therefore, which approach is ultimately more useful in practice comes down to which set of assumptions is more realistic for the particular data owner and what level of sensitivity under dilution their use\-case requires\.

## Acknowledgments

This work was supported by DARPA TIAMAT, the NSF TRAILS Institute \(2229885\), and Coefficient Giving\. Computing resources for this project were supported in part by the U\.S\. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research under the Advancements in AI for Science program\.

Prepared by LLNL under Contract DE\-AC52\-07NA27344 and supported by the LLNL\-LDRD Program under Projects No\. 24\-ERD\-010, and 24\-ERD\-058 \(LLNL\-CONF\-2019479\)\. This manuscript has been authored by Lawrence Livermore National Security, LLC under Contract No\. DE\-AC52\-07NA27344 with the U\.S\. Department of Energy\. The United States Government retains, and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a non\-exclusive, paid\-up, irrevocable, world\-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes\.

AI Use: Models from both OpenAI \(Codex\) and Anthropic \(Claude\) were used extensively in the implementation, experiment management, and paper editing aspects of this work\.

## References

- N\. Carlini, S\. Chien, M\. Nasr, S\. Song, A\. Terzis, and F\. Tramer \(2022\)Membership inference attacks from first principles\.In2022 IEEE symposium on security and privacy \(SP\),pp\. 1897–1914\.Cited by:[§1](https://arxiv.org/html/2607.00325#S1.SS0.SSS0.Px1.p1.1)\.
- N\. Carlini, F\. Tramer, E\. Wallace, M\. Jagielski, A\. Herbert\-Voss, K\. Lee, A\. Roberts, T\. Brown, D\. Song, U\. Erlingsson,et al\.\(2021\)Extracting training data from large language models\.In30th USENIX security symposium \(USENIX Security 21\),pp\. 2633–2650\.Cited by:[§1](https://arxiv.org/html/2607.00325#S1.SS0.SSS0.Px1.p1.1)\.
- M\. Christ, S\. Gunn, and O\. Zamir \(2024\)Undetectable watermarks for language models\.InThe Thirty Seventh Annual Conference on Learning Theory,pp\. 1125–1139\.External Links:[Link](https://arxiv.org/abs/2306.09194)Cited by:[§4](https://arxiv.org/html/2607.00325#S4.SS0.SSS0.Px4.p1.6),[§4](https://arxiv.org/html/2607.00325#S4.SS0.SSS0.Px4.p2.2)\.
- A\. F\. Cooper, A\. Gokaslan, A\. Ahmed, A\. B\. Cyphert, C\. De Sa, M\. A\. Lemley, D\. E\. Ho, and P\. Liang \(2025\)Extracting memorized pieces of \(copyrighted\) books from open\-weight language models\.arXiv preprint arXiv:2505\.12546\.Cited by:[§1](https://arxiv.org/html/2607.00325#S1.p3.1)\.
- M\. Duan, A\. Suri, N\. Mireshghallah, S\. Min, W\. Shi, L\. Zettlemoyer, Y\. Tsvetkov, Y\. Choi, D\. Evans, and H\. Hajishirzi \(2024\)Do membership inference attacks work on large language models?\.arXiv preprint arXiv:2402\.07841\.External Links:2402\.07841Cited by:[§1](https://arxiv.org/html/2607.00325#S1.p3.1)\.
- P\. Fernandez, T\. Sander, H\. Elsahar, H\. Chang, T\. Souček, V\. Lacatusu, T\. Tran, S\. Rebuffi, and A\. Mourachko \(2025\)How good is post\-hoc watermarking with language model rephrasing?\.Cited by:[§2\.2](https://arxiv.org/html/2607.00325#S2.SS2.p2.3)\.
- J\. Hayes, I\. Shumailov, C\. A\. Choquette\-Choo, M\. Jagielski, G\. Kaissis, M\. Nasr, S\. Ghalebikesabi, M\. S\. M\. S\. Annamalai, N\. Mireshghallah, I\. Shilov,et al\.\(2025\)Exploring the limits of strong membership inference attacks on large language models\.arXiv preprint arXiv:2505\.18773\.Cited by:[§1](https://arxiv.org/html/2607.00325#S1.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2607.00325#S4.SS0.SSS0.Px2.p1.1),[§4](https://arxiv.org/html/2607.00325#S4.p1.1)\.
- A\. Ilyas, S\. M\. Park, L\. Engstrom, G\. Leclerc, and A\. Madry \(2022\)Datamodels: predicting predictions from training data\.arXiv preprint arXiv:2202\.00622\.External Links:[Link](https://arxiv.org/abs/2202.00622)Cited by:[§1](https://arxiv.org/html/2607.00325#S1.p3.1)\.
- J\. Kirchenbauer, J\. Geiping, Y\. Wen, J\. Katz, I\. Miers, and T\. Goldstein \(2023\)A Watermark for Large Language Models\.InProceedings of the 40th International Conference on Machine Learning,A\. Krause, E\. Brunskill, K\. Cho, B\. Engelhardt, S\. Sabato, and J\. Scarlett \(Eds\.\),Proceedings of Machine Learning Research, Vol\.202,pp\. 17061–17084\.External Links:[Link](https://arxiv.org/abs/2301.10226)Cited by:[§1](https://arxiv.org/html/2607.00325#S1.p5.1)\.
- J\. Kirchenbauer, J\. Mongkolsupawan, Y\. Wen, T\. Goldstein, and D\. Ippolito \(2025\)A fictional q&a dataset for studying memorization and knowledge acquisition\.arXiv preprint arXiv:2506\.05639\.Cited by:[§3](https://arxiv.org/html/2607.00325#S3.SS0.SSS0.Px1.p1.1)\.
- P\. W\. Koh and P\. Liang \(2017\)Understanding black\-box predictions via influence functions\.InInternational conference on machine learning,pp\. 1885–1894\.External Links:[Link](https://arxiv.org/abs/1703.04730)Cited by:[§1](https://arxiv.org/html/2607.00325#S1.p3.1)\.
- K\. Z\. Liu, C\. A\. Choquette\-Choo, M\. Jagielski, P\. Kairouz, S\. Koyejo, P\. Liang, and N\. Papernot \(2025\)Language models may verbatim complete text they were not explicitly trained on\.arXiv preprint arXiv:2503\.17514\.External Links:2503\.17514Cited by:[§1](https://arxiv.org/html/2607.00325#S1.p3.1)\.
- T\. Olmo, A\. Ettinger, A\. Bertsch, B\. Kuehl, D\. Graham, D\. Heineman, D\. Groeneveld, F\. Brahman, F\. Timbers, H\. Ivison, J\. Morrison, J\. Poznanski, K\. Lo, L\. Soldaini, M\. Jordan, M\. Chen, M\. Noukhovitch, N\. Lambert, P\. Walsh, P\. Dasigi, R\. Berry, S\. Malik, S\. Shah, S\. Geng, S\. Arora, S\. Gupta, T\. Anderson, T\. Xiao, T\. Murray, T\. Romero, V\. Graf, A\. Asai, A\. Bhagia, A\. Wettig, A\. Liu, A\. Rangapur, C\. Anastasiades, C\. Huang, D\. Schwenk, H\. Trivedi, I\. Magnusson, J\. Lochner, J\. Liu, L\. J\. V\. Miranda, M\. Sap, M\. Morgan, M\. Schmitz, M\. Guerquin, M\. Wilson, R\. Huff, R\. L\. Bras, R\. Xin, R\. Shao, S\. Skjonsberg, S\. Z\. Shen, S\. S\. Li, T\. Wilde, V\. Pyatkin, W\. Merrill, Y\. Chang, Y\. Gu, Z\. Zeng, A\. Sabharwal, L\. Zettlemoyer, P\. W\. Koh, A\. Farhadi, N\. A\. Smith, and H\. Hajishirzi \(2025\)Olmo 3\.External Links:2512\.13961,[Link](https://arxiv.org/abs/2512.13961)Cited by:[§3](https://arxiv.org/html/2607.00325#S3.SS0.SSS0.Px1.p1.1)\.
- S\. M\. Park, K\. Georgiev, A\. Ilyas, G\. Leclerc, and A\. Madry \(2023\)Trak: attributing model behavior at scale\.arXiv preprint arXiv:2303\.14186\.External Links:[Link](https://arxiv.org/abs/2303.14186)Cited by:[§1](https://arxiv.org/html/2607.00325#S1.p3.1)\.
- J\. Piet, C\. Sitawarin, V\. Fang, N\. Mu, and D\. Wagner \(2025\)MARKMyWORDS: analyzing and evaluating language model watermarks\.In2025 IEEE Conference on Secure and Trustworthy Machine Learning \(SaTML\),pp\. 68–91\.External Links:[Link](https://arxiv.org/abs/2312.00273)Cited by:[§4](https://arxiv.org/html/2607.00325#S4.SS0.SSS0.Px4.p2.2)\.
- T\. Sander, P\. Fernandez, A\. Durmus, M\. Douze, and T\. Furon \(2024\)Watermarking makes language models radioactive\.Advances in Neural Information Processing Systems37,pp\. 21079–21113\.External Links:[Link](https://arxiv.org/abs/2402.14904)Cited by:[Table 8](https://arxiv.org/html/2607.00325#A3.T8),[Table 8](https://arxiv.org/html/2607.00325#A3.T8.4.2),[§1](https://arxiv.org/html/2607.00325#S1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2607.00325#S1.p5.1),[§2\.1](https://arxiv.org/html/2607.00325#S2.SS1.p1.1)\.
- T\. Sander, P\. Fernandez, S\. Mahloujifar, A\. Durmus, and C\. Guo \(2025\)Detecting benchmark contamination through watermarking\.arXiv preprint arXiv:2502\.17259\.External Links:[Link](https://arxiv.org/abs/2502.17259)Cited by:[Table 8](https://arxiv.org/html/2607.00325#A3.T8),[Table 8](https://arxiv.org/html/2607.00325#A3.T8.4.2),[§1](https://arxiv.org/html/2607.00325#S1.SS0.SSS0.Px1.p1.1),[§2\.1](https://arxiv.org/html/2607.00325#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2607.00325#S2.SS2.p1.3),[§2\.2](https://arxiv.org/html/2607.00325#S2.SS2.p2.3),[§3\.2](https://arxiv.org/html/2607.00325#S3.SS2.p2.12),[§3\.2](https://arxiv.org/html/2607.00325#S3.SS2.p3.7),[§4](https://arxiv.org/html/2607.00325#S4.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2607.00325#S4.SS0.SSS0.Px3.p1.7),[§4](https://arxiv.org/html/2607.00325#S4.p1.1)\.
- W\. Shi, A\. Ajith, M\. Xia, Y\. Huang, D\. Liu, T\. Blevins, D\. Chen, and L\. Zettlemoyer \(2024\)Detecting pretraining data from large language models\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 51826–51843\.Cited by:[§1](https://arxiv.org/html/2607.00325#S1.SS0.SSS0.Px1.p1.1)\.
- R\. Shokri, M\. Stronati, C\. Song, and V\. Shmatikov \(2017\)Membership inference attacks against machine learning models\.In2017 IEEE symposium on security and privacy \(SP\),pp\. 3–18\.External Links:1610\.05820Cited by:[§1](https://arxiv.org/html/2607.00325#S1.p3.1)\.
- Y\. E\. Xu, J\. Kirchenbauer, Y\. Savani, A\. Trockman, A\. Robey, T\. Goldstein, F\. Fang, and J\. Z\. Kolter \(2026\)Antidistillation fingerprinting\.arXiv preprint arXiv:2602\.03812\.Cited by:[§2\.1](https://arxiv.org/html/2607.00325#S2.SS1.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.External Links:[Link](https://arxiv.org/abs/2505.09388)Cited by:[§3](https://arxiv.org/html/2607.00325#S3.SS0.SSS0.Px1.p1.1)\.
- S\. Yeom, I\. Giacomelli, M\. Fredrikson, and S\. Jha \(2018\)Privacy risk in machine learning: analyzing the connection to overfitting\.In2018 IEEE 31st computer security foundations symposium \(CSF\),pp\. 268–282\.Cited by:[§1](https://arxiv.org/html/2607.00325#S1.SS0.SSS0.Px1.p1.1)\.

## Appendix AExtended Methodological Details

#### Folded Data Design\.

We instantiate our folded design under two complementary support regimes that share the same\(F,E\)\(F,E\)axes\. The primary, more realistic regime is theevent\-splitsetting: multiple keyed folds are mixed into each model’s training data with event\-level groupings preserved within each fold and with the total watermarked share of the training tokens held fixed acrossFF\. AsFFgrows, each individual key’s per\-fold footprint shrinks while the number of distinct keys present grows, simulating a deployment scenario where multiple data owners protect their own subsets with distinct keys inside a single training corpus\. The simpler ablation regime is thesimple per\-key support\(SKS\) setting: each watermarked model trains on exactly one watermarked fold, which isolates the watermarked\-fraction\-versus\-repetition question without sibling\-key interference but is unrealistic in that no real deployed corpus would protect only one fold of one data owner’s content under a single key\. We treat the SKS setting as a structurally clean warm\-up that anchors the more realistic event\-split readout\.

#### Efficiency\-Focused Design Choices\.

The point of this folded design is to realize a large number of dataset inference trials simulating the proactive watermarking approach as well as running the loss\-based methods, all while still controlling computational cost\. As a result, certain independence assumptions are necessarily violated\. First, the raw source data used at eachFFlevel is the same fixed pool of FictionalQA documents, which makes the different experiments correlated with respect to this domain\. Second, in the event\-split regime each model trains on more than one fold, so a dataset inference experiment for that model with respect to foldiiis not fully independent of the experiment testing the same model with respect to foldjjwhich it was also trained on\.

However, some of the efficiency\-focused choices also help control other confounds\. As we increaseEE, the fold partitions \(actual selected documents\) and watermark keys used remain identical, making effective epochs the only variable along that axis\. Similarly, the folds that the paired clean models train on are the same underlying documents as the ones that the watermarked models train on \(just before they were watermarked\), controlling for natural variation in modeling difficulty between the watermarked and clean model pairs\. In the event\-split regime, the models are also trained on different watermarking keys at the same time, simulating a realistic aspect of the deployment scenario where different sub\-datasets may be protected with different watermark keys within a single training corpus\.

## Appendix BExtended Experimental Setup Details

#### Watermark Scheme Details

All four experimental families \(event\-split finetuning, SKS finetuning, CPT pretraining, and from\-scratch pretraining\) share the same KGW\-style watermark configuration: greenlist fractionγ=0\.5\\gamma=0\.5, logit biasδ=4\.0\\delta=4\.0, and a 2\-token prefix used to seed the PRF that selects each step’s greenlist\. Headline significance values come from the empirical\-null permutation test of[Section2\.2](https://arxiv.org/html/2607.00325#S2.SS2)\.

Detection is run in teacher\-forced*reading*mode: the target model is run over the tokenized source text and scored at each position under the preceding tokens of that same source text, not on free generations\. We score under argmax mode—counting positions where the model’s greedy next\-token lands inside the keyed greenlist\. Repeated prefix windows are deduplicated by context so that high\-frequency local contexts cannot dominate either the watermark score or the empirical null\. For the empirical null, we drawM=106M=10^\{6\}randomized keys per surface from a fixed integer keyspace; this gives an exactpp\-value resolution of10−610^\{\-6\}, with an empirical\-Gaussian fit to the same null distribution available as an extrapolated reference where smallerpp\-values are needed\.

The watermark keys themselves are scaffolded differently across the three experimental settings\. The event\-split finetuning grid uses, at eachFFlevel, a fresh set ofFFfold\-specific keys \(one per fold\), and the detector key for a given watermark surface is always matched to the fold ID being read\. The SKS ablation does not use fold\-specific keys; it uses four fixed keys, each held constant across the entire\(F,E\)\(F,E\)grid for one of four model families, with the detector key matched to the family’s single key\. Phase 2 pretraining \(both CPT and from\-scratch\) reuses the same twoF=2F=2fold keys from the Phase 1 event\-split anchor row across all ten schedules and both initialization regimes; this is deliberate cross\-phase calibration, which is also why the warm\-key handedness present in the Phase 1F=2F=2scaffold \([SectionC\.7](https://arxiv.org/html/2607.00325#A3.SS7)\) is visible in the Phase 2 keyed readouts\.

#### Data Folding\.

For each finetuning experiment, we define a fold factorFFand partition the FictionalQA documents intoFFequal folds\. We then train a small population of models per cell with paired clean and watermarked twins, so that each watermarked positive has a matched clean “false\-probe” negative trained on the unmarked version of the same documents\. To simulate the watermark\-based dataset protection approach, for every foldDiD\_\{i\}, we create a paraphrased version of the documents using a paraphraser model running the watermark decoding scheme with keykik\_\{i\}, producingDki=Tm​\(Di,ki\)D\_\{k\_\{i\}\}=T\_\{m\}\(D\_\{i\},k\_\{i\}\)\. The watermarked modelfθiw​mf^\{wm\}\_\{\\theta\_\{i\}\}is trained on a mixture that includesDkiD\_\{k\_\{i\}\}and other data, while the corresponding clean twinfθicf^\{c\}\_\{\\theta\_\{i\}\}trains onDiD\_\{i\}in its original form before the paraphrasing transformation\. We also vary the number of effective epochsEEover the watermarked subset within a run while holding the per\-cell training\-token budget fixed across\(F,E\)\(F,E\); samples from the base data source are not repeated\.[Table1](https://arxiv.org/html/2607.00325#S3.T1)summarizes the resulting effective per\-key exposure across the grid; the event\-split and SKS regimes share this idealized table\.

## Appendix CFinetuning Event\-Split Exhaustive Readout

This appendix carries the full per\-cell readout of the event\-split finetuning grid that the main body’s[Figure2](https://arxiv.org/html/2607.00325#S2.F2)introduces\. The main body uses the empirical\-exact reference for both the keyed\-signal and DIA heatmaps; the subsections below add the empirical\-Gaussian companion of each, the packed\-surface counterparts of the keyed/null pair, the cross\-key sham\-null heatmaps, the realized exposure andE^\\hat\{E\}tables, the row\-level MIA table for the baselines, and the null\-validity panel\.

### C\.1Empirical\-Gaussian and Packed\-Surface Heatmap Companions

[Figures2](https://arxiv.org/html/2607.00325#S2.F2),[7](https://arxiv.org/html/2607.00325#A3.F7),[8](https://arxiv.org/html/2607.00325#A3.F8)and[9](https://arxiv.org/html/2607.00325#A3.F9)carry the four event\-split keyed/null pairs across the \(aligned, packed\) surface and \(exact, Gaussian\)pp\-value\-type combinations; the aligned\-exact keyed half is also shown in the main\-body[Figure2](https://arxiv.org/html/2607.00325#S2.F2)\.[Figures3](https://arxiv.org/html/2607.00325#S3.F3)and[10](https://arxiv.org/html/2607.00325#A3.F10)carry the matching watermark whole\-model DIA AUC pairs, with aligned and packed surfaces shown side\-by\-side under eachpp\-value type\.

![Refer to caption](https://arxiv.org/html/2607.00325v1/x11.png)

![Refer to caption](https://arxiv.org/html/2607.00325v1/x12.png)

Figure 7:Finetuning event\-split matched clean\-twin false\-probe null \(left\) and keyed signal \(right\) across theF×EF\\times Egrid on the aligned unpacked detection surface, scored as−log10⁡p\-\\log\_\{10\}punder the empirical\-Gaussian reference\.![Refer to caption](https://arxiv.org/html/2607.00325v1/x13.png)

![Refer to caption](https://arxiv.org/html/2607.00325v1/x14.png)

Figure 8:Finetuning event\-split matched clean\-twin false\-probe null \(left\) and keyed signal \(right\) across theF×EF\\times Egrid on the packed detection surface, scored as−log10⁡p\-\\log\_\{10\}punder the empirical\-exact null\. The packed surface is a more permissive oracle baseline than the aligned surface\.![Refer to caption](https://arxiv.org/html/2607.00325v1/x15.png)

![Refer to caption](https://arxiv.org/html/2607.00325v1/x16.png)

Figure 9:Finetuning event\-split matched clean\-twin false\-probe null \(left\) and keyed signal \(right\) across theF×EF\\times Egrid on the packed detection surface, scored as−log10⁡p\-\\log\_\{10\}punder the empirical\-Gaussian reference\.![Refer to caption](https://arxiv.org/html/2607.00325v1/x17.png)

![Refer to caption](https://arxiv.org/html/2607.00325v1/x18.png)

Figure 10:Finetuning event\-split watermark whole\-model DIA AUC across theF×EF\\times Egrid, on the aligned \(left\) and packed \(right\) detection surfaces, both scored against an empirical\-Gaussian null\.
### C\.2Event\-Split Exposure Trend Curves

[Figure11](https://arxiv.org/html/2607.00325#A3.F11)is a re\-visualization of the same keyed\-signal data shown in the right panel of[Figure2](https://arxiv.org/html/2607.00325#S2.F2), but using the empirical Gaussian extrapolation p\-values from[Figure7](https://arxiv.org/html/2607.00325#A3.F7)and re\-cast on a continuous exposure axis with separate lines perFFto make the exposure trend and the cross\-FFseparation easier to read at a glance\. At low exposure the per\-key support is too small for keyed signal to lift off in any of theFFrows, while at high exposure the curves separate cleanly withFF\.

![Refer to caption](https://arxiv.org/html/2607.00325v1/x19.png)Figure 11:Finetuning event\-split keyed\-signal exposure response across the grid, tracing−log10⁡p\-\\log\_\{10\}pas a function of effective per\-key exposure with separate lines for each fold countFF\. This figure plots the same per\-cell keyed\-signal values as the right panel of[Figure2](https://arxiv.org/html/2607.00325#S2.F2), just computed with the empirical Gaussian extrapolated p\-values from[Figure7](https://arxiv.org/html/2607.00325#A3.F7)and re\-cast on a continuous exposure axis to make the trend clearer\.
### C\.3Cross\-Key Sham\-Null Heatmaps

[Figures12](https://arxiv.org/html/2607.00325#A3.F12)and[13](https://arxiv.org/html/2607.00325#A3.F13)carry the cross\-key sham\-null heatmaps for the event\-split grid, where the watermark detector is queried with a key the target model never saw in training\. These act as an additional negative control beyond the matched clean\-twin false\-probe null, isolating per\-key idiosyncrasy of the detection surface from the matched\-pair design\.

![Refer to caption](https://arxiv.org/html/2607.00325v1/x20.png)

![Refer to caption](https://arxiv.org/html/2607.00325v1/x21.png)

Figure 12:Finetuning event\-split cross\-key sham\-null heatmap across theF×EF\\times Egrid, on the aligned \(left\) and packed \(right\) detection surfaces, scored as−log10⁡p\-\\log\_\{10\}punder the empirical\-exact null\. The watermark detector is queried with a key the target model never saw in training, providing a negative control beyond the matched clean\-twin false\-probe null\.![Refer to caption](https://arxiv.org/html/2607.00325v1/x22.png)

![Refer to caption](https://arxiv.org/html/2607.00325v1/x23.png)

Figure 13:Finetuning event\-split cross\-key sham\-null heatmap across theF×EF\\times Egrid, on the aligned \(left\) and packed \(right\) detection surfaces, scored as−log10⁡p\-\\log\_\{10\}punder the empirical\-Gaussian null\.
### C\.4Event\-Split Realized Exposure andE^\\hat\{E\}Per\-Cell Readbacks

[Table3](https://arxiv.org/html/2607.00325#A3.T3)reports the realized normalized exposureE^/F\\hat\{E\}/Fper cell of the event\-split finetuning grid\.[Table4](https://arxiv.org/html/2607.00325#A3.T4)reports the underlying realizedE^\\hat\{E\}values\. The corresponding idealized epoch countsEEare the column headers of[Table1](https://arxiv.org/html/2607.00325#S3.T1), which the event\-split construction shares with SKS\.

Table 3:Event\-split finetuning: realized normalized exposure summary \(E^/F\\hat\{E\}/F\)\.Table 4:Event\-split finetuning: realized exposure summary \(E^\\hat\{E\}\)\.
### C\.5Event\-Split Training Scale and Trial Geometry

[Table5](https://arxiv.org/html/2607.00325#A3.T5)reports the per\-cell watermarked\-token totals \(mean, min–max across paired models\) and the corresponding fraction of each model’s 131M\-token training budget\.[Table6](https://arxiv.org/html/2607.00325#A3.T6)reports the per\-cell paired\-model counts and the resultingn\+/n−n\_\{\+\}/n\_\{\-\}trial counts that drive each cell’s whole\-model DIA AUC; both are summarized in[Section3\.1](https://arxiv.org/html/2607.00325#S3.SS1)\.

Table 5:Event\-split finetuning: training scale context\. Per\-cell watermark token totals are relative to 131\.1M train tokens per run\. The percent columns report watermark\-token share of total train tokens\.Table 6:Event\-split finetuning: model and watermark DIA trial geometry\. The WM and clean model columns count target models per cell, and the WM DIA trial column counts the positive/negative pooled trials used for the watermark DIA AUC\.
### C\.6Event\-Split Loss\-Based and Reference\-Model Baselines Row\-Level MIA

Table 7:Event\-split finetuning: row\-level MIA AUC comparison\. Entries marked N/A indicate statistics that are not estimable in the available cell geometry; for example, LiRA in F=2 cells lacks sufficient in\-reference models\.Table 8:To contextualize the performance differences between the watermark\-based DIA technique and other baselines presented in[Table2](https://arxiv.org/html/2607.00325#S3.T2), here we summarize the key assumptions about the data owner’s \(i\.e\. attacker’s\) control over the data and access to the target model needed to make a dataset membership claim\.Qualifications:1The “reading\-mode” method ofSanderet al\.\([2024](https://arxiv.org/html/2607.00325#bib.bib18),[2025](https://arxiv.org/html/2607.00325#bib.bib19)\)requires the ability to check what the next\-token prediction of the target model would be conditioned on a prefix of the protected sample\. While in theory this could be accomplished by repeatedly querying the model while iteratively appending tokens to the prompt \(expensive\), the practicality of this operation given modern frontier model APIs is questionable\.2We mark this row as such to draw attention to the fact that performing academic analyses centering ROC\-AUC allows us to ignore the necessity of calibrating a single decision threshold to deploy\. However, in practice, for all MIA/DIA techniques that do not admit a p\-value or other analytic \(FP\) error estimate, held\-out validation samples must be used to calibrate the decision threshold\.Access AssumptionWatermarkedRaw\-lossArgmaxmin\-k10rMIA\-simplerMIALiRAData owner can modify their data before it is included in the training dataset\.![[Uncaptioned image]](https://arxiv.org/html/2607.00325v1/static_figs/check-mark-emoji-clipart-xl.png)Data owner has next\-token prediction1access to target model\.![[Uncaptioned image]](https://arxiv.org/html/2607.00325v1/static_figs/check-mark-emoji-clipart-xl.png)![[Uncaptioned image]](https://arxiv.org/html/2607.00325v1/static_figs/check-mark-emoji-clipart-xl.png)Data owner must maintain held\-out pool of additional samples for calibration2\.![[Uncaptioned image]](https://arxiv.org/html/2607.00325v1/static_figs/check-mark-emoji-clipart-xl.png)![[Uncaptioned image]](https://arxiv.org/html/2607.00325v1/static_figs/check-mark-emoji-clipart-xl.png)![[Uncaptioned image]](https://arxiv.org/html/2607.00325v1/static_figs/check-mark-emoji-clipart-xl.png)![[Uncaptioned image]](https://arxiv.org/html/2607.00325v1/static_figs/check-mark-emoji-clipart-xl.png)![[Uncaptioned image]](https://arxiv.org/html/2607.00325v1/static_figs/check-mark-emoji-clipart-xl.png)![[Uncaptioned image]](https://arxiv.org/html/2607.00325v1/static_figs/check-mark-emoji-clipart-xl.png)Data owner has loss or logit access to target model\.![[Uncaptioned image]](https://arxiv.org/html/2607.00325v1/static_figs/check-mark-emoji-clipart-xl.png)![[Uncaptioned image]](https://arxiv.org/html/2607.00325v1/static_figs/check-mark-emoji-clipart-xl.png)![[Uncaptioned image]](https://arxiv.org/html/2607.00325v1/static_figs/check-mark-emoji-clipart-xl.png)![[Uncaptioned image]](https://arxiv.org/html/2607.00325v1/static_figs/check-mark-emoji-clipart-xl.png)![[Uncaptioned image]](https://arxiv.org/html/2607.00325v1/static_figs/check-mark-emoji-clipart-xl.png)Data owner has compute and target model details needed to train reference models\.![[Uncaptioned image]](https://arxiv.org/html/2607.00325v1/static_figs/check-mark-emoji-clipart-xl.png)![[Uncaptioned image]](https://arxiv.org/html/2607.00325v1/static_figs/check-mark-emoji-clipart-xl.png)![[Uncaptioned image]](https://arxiv.org/html/2607.00325v1/static_figs/check-mark-emoji-clipart-xl.png)
### C\.7Event\-Split Null\-Validity Extended

The clipped chart[Figure4](https://arxiv.org/html/2607.00325#S3.F4)in the main body, equivalent to the left side of[Figure14](https://arxiv.org/html/2607.00325#A3.F14)confirms that once pooled across many distinct positive keys, the empirical exact null is close to uniform at the 1M\-null scale and respects standard tail\-rate thresholds, supporting the use of the empirical\-null permutation test as the headline reference distribution for the keyed readout\. Per\-key idiosyncrasy is a separate concern visible in the right\-hand trace: at very small fold counts a single key can run systematically warm even when the pooled null is well\-calibrated, and the inheritedF=2F=2scaffold reused in pretraining \([Section3\.2](https://arxiv.org/html/2607.00325#S3.SS2)\) carries one such warm key\.

![Refer to caption](https://arxiv.org/html/2607.00325v1/x24.png)Figure 14:Event\-grid null\-validity panel\.Left:histogram of pooled empirical exactpp\-values from the packed matched\-clean\-negative whole\-model readings across the\(F,E\)\(F,E\)event grid\. The annotated statistic andpp\-value are from a one\-sample Kolmogorov\-Smirnov test of these pooledpp\-values againstUniform​\(0,1\)\\mathrm\{Uniform\}\(0,1\); the dashed horizontal line is the expected per\-bin count under a uniform histogram with the plotted binning\.Right:−log10⁡p\-\\log\_\{10\}ptrace of theF=2F=2warm\-key slice of the same packed null family plotted againstEE, shown for context only and not part of the KS test on the left\.

## Appendix DFinetuning SKS Exhaustive Readout

This appendix reports the simple per\-key support \(SKS\) ablation, where each model trains on exactly one watermarked fold so the per\-model fictional support fraction shrinks as1/F1/Finstead of being held fixed acrossFF\. SKS is structurally simpler than the event\-split regime \([AppendixC](https://arxiv.org/html/2607.00325#A3)\) but unrealistic in that no real deployed corpus would protect only one fold of one data owner’s content under a single key\. The same idealized per\-key exposure \([Table1](https://arxiv.org/html/2607.00325#S3.T1)\) governs both regimes; what differs is sibling support and the per\-cell positive/negative trial budget \([Table12](https://arxiv.org/html/2607.00325#A4.T12)\)\. We use SKS as a watermark\-only sanity check on the per\-key scaling story without sibling\-key interference, and run the loss\-based and reference\-model comparison only on the more realistic event\-split regime where the row\-level baselines are meaningful\.

### D\.1SKS Empirical\-Gaussian and Packed\-Surface Heatmap Companions

[Figures15](https://arxiv.org/html/2607.00325#A4.F15),[19](https://arxiv.org/html/2607.00325#A4.F19),[16](https://arxiv.org/html/2607.00325#A4.F16),[18](https://arxiv.org/html/2607.00325#A4.F18),[17](https://arxiv.org/html/2607.00325#A4.F17)and[20](https://arxiv.org/html/2607.00325#A4.F20)carry the SKS keyed/null pair and DIA AUC heatmap pair on the aligned\-exact, aligned\-Gaussian, packed\-exact, and packed\-Gaussian detection surfaces\.

![Refer to caption](https://arxiv.org/html/2607.00325v1/x25.png)

![Refer to caption](https://arxiv.org/html/2607.00325v1/x26.png)

Figure 15:Finetuning SKS matched clean\-twin false\-probe null \(left\) and keyed signal \(right\) across theF×EF\\times Egrid on the aligned unpacked detection surface, scored as−log10⁡p\-\\log\_\{10\}punder an empirical\-exact reference\. The false\-probe null stays quiet on the same surface, validating the matched clean\-twin negative as the right baseline against which to read the keyed map; the keyed signal grows monotonically withEEand decays withFFas the per\-key support fraction1/F1/Fshrinks\.![Refer to caption](https://arxiv.org/html/2607.00325v1/x27.png)

![Refer to caption](https://arxiv.org/html/2607.00325v1/x28.png)

Figure 16:Finetuning SKS matched clean\-twin false\-probe null \(left\) and keyed signal \(right\) across theF×EF\\times Egrid on the aligned unpacked detection surface, scored as−log10⁡p\-\\log\_\{10\}punder the empirical\-Gaussian reference\.![Refer to caption](https://arxiv.org/html/2607.00325v1/x29.png)

![Refer to caption](https://arxiv.org/html/2607.00325v1/x30.png)

Figure 17:Finetuning SKS matched clean\-twin false\-probe null \(left\) and keyed signal \(right\) across theF×EF\\times Egrid on the packed detection surface, scored as−log10⁡p\-\\log\_\{10\}punder the empirical\-exact null\. The packed surface is a more permissive oracle baseline than the aligned surface\.![Refer to caption](https://arxiv.org/html/2607.00325v1/x31.png)

![Refer to caption](https://arxiv.org/html/2607.00325v1/x32.png)

Figure 18:Finetuning SKS matched clean\-twin false\-probe null \(left\) and keyed signal \(right\) across theF×EF\\times Egrid on the packed detection surface, scored as−log10⁡p\-\\log\_\{10\}punder the empirical\-Gaussian reference\.![Refer to caption](https://arxiv.org/html/2607.00325v1/x33.png)

![Refer to caption](https://arxiv.org/html/2607.00325v1/x34.png)

Figure 19:Finetuning SKS watermark whole\-model DIA AUC across theF×EF\\times Egrid, on the aligned \(left\) and packed \(right\) detection surfaces, both scored against an empirical\-exact null\. On the aligned surface the AUC saturates at1\.01\.0fromE=8E=8onward at everyFF, while the lowest\-EEcorner remains coarse with only eight trials per cell\. On the packed surface, the more permissive oracle recovers several of those low\-exposure cells, reaching1\.01\.0one to twoEE\-steps earlier across the grid\.![Refer to caption](https://arxiv.org/html/2607.00325v1/x35.png)

![Refer to caption](https://arxiv.org/html/2607.00325v1/x36.png)

Figure 20:Finetuning SKS watermark whole\-model DIA AUC across theF×EF\\times Egrid, on the aligned \(left\) and packed \(right\) detection surfaces, both scored against an empirical\-Gaussian null\.
### D\.2SKS Exposure Trend Curves

[Figure21](https://arxiv.org/html/2607.00325#A4.F21)is a re\-visualization of the same keyed\-signal data shown in the right panel of[Figure15](https://arxiv.org/html/2607.00325#A4.F15), re\-cast on a continuous exposure axis with separate lines perFFto make the exposure trend and the cross\-FFseparation easier to read at a glance: at low exposure the per\-key support is too small for keyed signal to lift off in any of theFFrows, while at high exposure the curves separate cleanly withFF\.

![Refer to caption](https://arxiv.org/html/2607.00325v1/x37.png)Figure 21:Finetuning SKS keyed\-signal exposure response across the grid, tracing−log10⁡p\-\\log\_\{10\}pas a function of effective per\-key exposure with separate lines for each fold countFF\. This figure plots the same per\-cell keyed\-signal values as the right panel of[Figure15](https://arxiv.org/html/2607.00325#A4.F15), just re\-cast on a continuous exposure axis to make the trend clearer\.
### D\.3SKS Realized Exposure andE^\\hat\{E\}Per\-Cell Readbacks

[Table9](https://arxiv.org/html/2607.00325#A4.T9)reports the realized normalized exposureE^/F\\hat\{E\}/Fper cell of the finetuning SKS grid, the direct realized counterpart to the idealizedE/FE/Fvalues in[Table1](https://arxiv.org/html/2607.00325#S3.T1)\.[Table10](https://arxiv.org/html/2607.00325#A4.T10)reports the underlying realizedE^\\hat\{E\}values\. The discrepancy between idealized and realized values here is the realized overshoot of the watermarked subset’s effective epoch count relative to the planned schedule\.

Table 9:SKS finetuning: realized normalized exposure summary \(E^/F\\hat\{E\}/F\)\.Table 10:SKS finetuning: realized exposure summary \(E^\\hat\{E\}\)\.
### D\.4SKS Training Scale and Trial Geometry

[Table11](https://arxiv.org/html/2607.00325#A4.T11)reports the per\-cell watermarked\-token totals \(mean, min–max across paired models\) and the corresponding fraction of each model’s 131M\-token training budget\.[Table12](https://arxiv.org/html/2607.00325#A4.T12)reports the per\-cell paired\-model counts and the resultingn\+/n−n\_\{\+\}/n\_\{\-\}trial counts; SKS holds the per\-cell trial budget fixed at4/44/4across the grid, since each model trains on exactly one watermarked fold and the per\-cell positive/negative budget is the same at every\(F,E\)\(F,E\)\.

Table 11:SKS finetuning: training scale context\. Per\-cell watermark token totals are relative to 131\.1M train tokens per run\. The percent columns report watermark\-token share of total train tokens\.Table 12:SKS finetuning: model and watermark DIA trial geometry\. The WM and clean model columns count target models per cell, and the WM DIA trial column counts the positive/negative pooled trials used for the watermark DIA AUC\.
### D\.5SKS Watermark\-Only DIA Numeric Table

[Table13](https://arxiv.org/html/2607.00325#A4.T13)carries the underlying numeric AUC values that drive the SKS DIA heatmap pair in[Figure19](https://arxiv.org/html/2607.00325#A4.F19)\. We deliberately keep this table watermark\-only: the SKS support construction is a clean per\-key scaling ablation in which each model trains on exactly one watermarked fold, and the loss\-based / reference\-model row\-level baselines are only meaningful against the realistic event\-split regime where multiple keys coexist, so the head\-to\-head comparison against those baselines is run only there \([SectionC\.6](https://arxiv.org/html/2607.00325#A3.SS6)\)\.

Table 13:SKS finetuning: watermark whole\-model DIA AUC summary\.
### D\.6SKS Null\-Validity

![Refer to caption](https://arxiv.org/html/2607.00325v1/x38.png)Figure 22:SKS null\-validity panel\.Left:histogram of pooled empirical exactpp\-values from the packed clean\-model watermark\-surface false\-probe rows across the depth\-4 SKS grid\. The annotated statistic andpp\-value are from a one\-sample Kolmogorov\-Smirnov test of these pooledpp\-values againstUniform​\(0,1\)\\mathrm\{Uniform\}\(0,1\); the dashed horizontal line is the expected per\-bin count under a uniform histogram with the plotted binning\.Right:−log10⁡p\-\\log\_\{10\}ptrace of theF=2F=2warm\-key slice of the same packed null family plotted againstEE, shown for context only and not part of the KS test on the left\.[Figure22](https://arxiv.org/html/2607.00325#A4.F22)confirms that, once pooled across many distinct positive keys, the empirical exact null is close to uniform at the 1M\-null scale and respects standard tail\-rate thresholds\. This holds in both the event\-split grid \([Figure4](https://arxiv.org/html/2607.00325#S3.F4)\) and in this SKS ablation, supporting the use of the empirical\-null permutation test as the headline reference distribution across both regimes\.

## Appendix EPretraining Exhaustive Readout

This appendix carries the full per\-init readout of the pretraining schedule sweep to complement the main body’s[Figures5](https://arxiv.org/html/2607.00325#S3.F5)and[6](https://arxiv.org/html/2607.00325#S3.F6)\. The structure is grouped first by initialization regime \(CPT then from\-scratch\), and within each by aligned\-then\-packed surface, exact\-then\-Gaussianpp\-value type, with the watermark whole\-model DIA bar pairs trailing each init’s keyed/null block\. The split row\-level MIA and whole\-model DIA baseline tables for both initialization regimes are reported below the per\-init bar companions, and each whole\-model DIA cell is computed over2\+/2−2\{\+\}/2\{\-\}trials per schedule\.

Table 14:Pretraining idealized exposure profile across the ten\-schedule sweep atF=2F=2\. Each schedule targets a nominal effective epoch countEE, which atF=2F=2corresponds to an idealized per\-key exposureE/F=E/2E/F=E/2\. Both initialization regimes \(CPT and from\-scratch\) share the same idealized profile\.### E\.1Pretraining CPT Companion Figures

[Figures5](https://arxiv.org/html/2607.00325#S3.F5),[23](https://arxiv.org/html/2607.00325#A5.F23),[24](https://arxiv.org/html/2607.00325#A5.F24)and[25](https://arxiv.org/html/2607.00325#A5.F25)carry the four CPT keyed/null pairs across the \(aligned, packed\) surface and \(exact, Gaussian\)pp\-value\-type combinations\.[Figures26](https://arxiv.org/html/2607.00325#A5.F26)and[27](https://arxiv.org/html/2607.00325#A5.F27)carry the matching CPT watermark whole\-model DIA AUC pairs, with aligned and packed surfaces shown side\-by\-side under eachpp\-value type\.

![Refer to caption](https://arxiv.org/html/2607.00325v1/x39.png)

![Refer to caption](https://arxiv.org/html/2607.00325v1/x40.png)

Figure 23:Pretraining CPT matched clean\-twin false\-probe null \(left\) and keyed signal \(right\) across the ten\-schedule sweep atF=2F=2, on the aligned unpacked detection surface, scored as−log10⁡p\-\\log\_\{10\}punder the empirical\-Gaussian reference\.![Refer to caption](https://arxiv.org/html/2607.00325v1/x41.png)

![Refer to caption](https://arxiv.org/html/2607.00325v1/x42.png)

Figure 24:Pretraining CPT matched clean\-twin false\-probe null \(left\) and keyed signal \(right\) across the ten\-schedule sweep atF=2F=2, on the packed detection surface, scored as−log10⁡p\-\\log\_\{10\}punder the empirical\-exact null\. The packed surface is a more permissive oracle baseline than the aligned surface\.![Refer to caption](https://arxiv.org/html/2607.00325v1/x43.png)

![Refer to caption](https://arxiv.org/html/2607.00325v1/x44.png)

Figure 25:Pretraining CPT matched clean\-twin false\-probe null \(left\) and keyed signal \(right\) across the ten\-schedule sweep atF=2F=2, on the packed detection surface, scored as−log10⁡p\-\\log\_\{10\}punder the empirical\-Gaussian reference\.![Refer to caption](https://arxiv.org/html/2607.00325v1/x45.png)

![Refer to caption](https://arxiv.org/html/2607.00325v1/x46.png)

Figure 26:Pretraining CPT watermark whole\-model DIA AUC across the ten\-schedule sweep atF=2F=2, on the aligned \(left\) and packed \(right\) detection surfaces, both scored against an empirical\-exact null\. Each schedule contributes2\+/2−2\{\+\}/2\{\-\}whole\-model trials, so AUC is coarse but tracks the keyed\-signal ordering of[Figure5](https://arxiv.org/html/2607.00325#S3.F5)\.![Refer to caption](https://arxiv.org/html/2607.00325v1/x47.png)

![Refer to caption](https://arxiv.org/html/2607.00325v1/x48.png)

Figure 27:Pretraining CPT watermark whole\-model DIA AUC across the ten\-schedule sweep atF=2F=2, on the aligned \(left\) and packed \(right\) detection surfaces, both scored against an empirical\-Gaussian null\. Each schedule contributes2\+/2−2\{\+\}/2\{\-\}whole\-model trials\.
### E\.2Pretraining From\-Scratch Companion Figures

[Figures6](https://arxiv.org/html/2607.00325#S3.F6),[28](https://arxiv.org/html/2607.00325#A5.F28),[29](https://arxiv.org/html/2607.00325#A5.F29)and[30](https://arxiv.org/html/2607.00325#A5.F30)carry the four from\-scratch keyed/null pairs across the \(aligned, packed\) surface and \(exact, Gaussian\)pp\-value\-type combinations\.[Figures31](https://arxiv.org/html/2607.00325#A5.F31)and[32](https://arxiv.org/html/2607.00325#A5.F32)carry the matching from\-scratch watermark whole\-model DIA AUC pairs, with aligned and packed surfaces shown side\-by\-side under eachpp\-value type\.

![Refer to caption](https://arxiv.org/html/2607.00325v1/x49.png)

![Refer to caption](https://arxiv.org/html/2607.00325v1/x50.png)

Figure 28:Pretraining from\-scratch matched clean\-twin false\-probe null \(left\) and keyed signal \(right\) across the ten\-schedule sweep atF=2F=2, on the aligned unpacked detection surface, scored as−log10⁡p\-\\log\_\{10\}punder the empirical\-Gaussian reference\.![Refer to caption](https://arxiv.org/html/2607.00325v1/x51.png)

![Refer to caption](https://arxiv.org/html/2607.00325v1/x52.png)

Figure 29:Pretraining from\-scratch matched clean\-twin false\-probe null \(left\) and keyed signal \(right\) across the ten\-schedule sweep atF=2F=2, on the packed detection surface, scored as−log10⁡p\-\\log\_\{10\}punder the empirical\-exact null\. The packed surface is a more permissive oracle baseline than the aligned surface\.![Refer to caption](https://arxiv.org/html/2607.00325v1/x53.png)

![Refer to caption](https://arxiv.org/html/2607.00325v1/x54.png)

Figure 30:Pretraining from\-scratch matched clean\-twin false\-probe null \(left\) and keyed signal \(right\) across the ten\-schedule sweep atF=2F=2, on the packed detection surface, scored as−log10⁡p\-\\log\_\{10\}punder the empirical\-Gaussian reference\.![Refer to caption](https://arxiv.org/html/2607.00325v1/x55.png)

![Refer to caption](https://arxiv.org/html/2607.00325v1/x56.png)

Figure 31:Pretraining from\-scratch watermark whole\-model DIA AUC across the ten\-schedule sweep atF=2F=2, on the aligned \(left\) and packed \(right\) detection surfaces, both scored against an empirical\-exact null\. Each schedule contributes2\+/2−2\{\+\}/2\{\-\}whole\-model trials\. The from\-scratch DIA recovers near\-saturated AUC at the high\-exposure schedules well before CPT does\.![Refer to caption](https://arxiv.org/html/2607.00325v1/x57.png)

![Refer to caption](https://arxiv.org/html/2607.00325v1/x58.png)

Figure 32:Pretraining from\-scratch watermark whole\-model DIA AUC across the ten\-schedule sweep atF=2F=2, on the aligned \(left\) and packed \(right\) detection surfaces, both scored against an empirical\-Gaussian null\. Each schedule contributes2\+/2−2\{\+\}/2\{\-\}whole\-model trials\.
### E\.3Pretraining Schedule Summaries

[Tables15](https://arxiv.org/html/2607.00325#A5.T15)and[16](https://arxiv.org/html/2607.00325#A5.T16)provide compact per\-schedule summaries combining the realizedE^\\hat\{E\}with the aligned and packed watermark−log10⁡p\-\\log\_\{10\}pvalues for both initialization regimes\.

Table 15:CPT pretraining: schedule summary\.Table 16:From\-scratch pretraining: schedule summary\.
### E\.4Pretraining Realized Exposure andE^\\hat\{E\}Per\-Schedule Readbacks

[Tables17](https://arxiv.org/html/2607.00325#A5.T17)and[18](https://arxiv.org/html/2607.00325#A5.T18)report the realized normalized exposureE^/F\\hat\{E\}/Fper schedule for both pretraining initialization regimes, the direct realized counterpart to the idealizedE/FE/Fvalues in main\-body[Table14](https://arxiv.org/html/2607.00325#A5.T14)\.[Tables19](https://arxiv.org/html/2607.00325#A5.T19)and[20](https://arxiv.org/html/2607.00325#A5.T20)report the underlying realizedE^\\hat\{E\}values\. The corresponding idealizedEEtargets are the integers in the schedule names \(columnEEof[Table14](https://arxiv.org/html/2607.00325#A5.T14)\)\.

Table 17:CPT pretraining: realized normalized exposure summary \(E^/F\\hat\{E\}/F\)\.Table 18:From\-scratch pretraining: realized normalized exposure summary \(E^/F\\hat\{E\}/F\)\.Table 19:CPT pretraining: realized exposure summary \(E^\\hat\{E\}\)\.Table 20:From\-scratch pretraining: realized exposure summary \(E^\\hat\{E\}\)\.
### E\.5Pretraining Training Scale and Trial Geometry

[Tables21](https://arxiv.org/html/2607.00325#A5.T21)and[22](https://arxiv.org/html/2607.00325#A5.T22)report the per\-schedule watermarked\-token totals \(mean, min–max across paired models\) and the corresponding fraction of each model’s 10\.49B\-token training budget for the CPT and from\-scratch initialization regimes respectively\.[Tables23](https://arxiv.org/html/2607.00325#A5.T23)and[24](https://arxiv.org/html/2607.00325#A5.T24)report the per\-schedule paired\-model counts and the resultingn\+/n−n\_\{\+\}/n\_\{\-\}trial counts that drive each schedule’s whole\-model DIA AUC\. Each schedule contributes2/22/2positive/negative whole\-model trials per init across both initialization regimes;[Section3\.2](https://arxiv.org/html/2607.00325#S3.SS2)summarizes the extremal mass values\.

Table 21:CPT pretraining: training scale context\. Per\-schedule watermark token totals are relative to 10\.49B train tokens per run\. The percent columns report watermark\-token share of total train tokens\.Table 22:From\-scratch pretraining: training scale context\. Per\-schedule watermark token totals are relative to 10\.49B train tokens per run\. The percent columns report watermark\-token share of total train tokens\.Table 23:CPT pretraining: model and watermark DIA trial geometry\. In these pretraining cells, model counts and watermark DIA trial counts coincide because each target contributes one positive or negative whole\-model trial to the pooled AUC\.Table 24:From\-scratch pretraining: model and watermark DIA trial geometry\. In these pretraining cells, model counts and watermark DIA trial counts coincide because each target contributes one positive or negative whole\-model trial to the pooled AUC\.
### E\.6Pretraining Loss\-Based and Reference\-Model Row\-Level MIA and Whole\-Model DIA Baselines

[Tables25](https://arxiv.org/html/2607.00325#A5.T25),[26](https://arxiv.org/html/2607.00325#A5.T26),[27](https://arxiv.org/html/2607.00325#A5.T27)and[28](https://arxiv.org/html/2607.00325#A5.T28)report the source\-fold loss\-based and reference\-model row\-level MIA and whole\-model DIA AUCs alongside the watermark detector’s own AUCs \(which are the same values that drive[Figures26](https://arxiv.org/html/2607.00325#A5.F26)and[31](https://arxiv.org/html/2607.00325#A5.F31)\)\.

Table 25:CPT pretraining: row\-level MIA AUC comparison\.Table 26:CPT pretraining: fold\-level whole\-model DIA AUC comparison\. Each cell contributes2\+/2−2\+/2\-whole\-model trials\.Table 27:From\-scratch pretraining: row\-level MIA AUC comparison\.Table 28:From\-scratch pretraining: fold\-level whole\-model DIA AUC comparison\. Each cell contributes2\+/2−2\+/2\-whole\-model trials\.