Extracting Training Data from Diffusion Language Models via Infilling

# Extracting Training Data from Diffusion Language Models via Infilling
Source: [https://arxiv.org/html/2605.24173](https://arxiv.org/html/2605.24173)
Yihan Wang University of Waterloo yihan\.wang@uwaterloo\.ca &N\. Asokan University of Waterloo KTH Royal Institute of Technology asokan@acm\.org

###### Abstract

Memorization in large language models has been studied almost exclusively through prefix\-conditioned extraction, a natural choice for autoregressive models\. However, diffusion language models \(DLMs\) can denoise masked tokens at arbitrary positions\. Thus, prefix\-only probing reveals only one facet of memorization in DLMs and significantly underestimates the risk of training\-data extraction\.

In order to realistically model extractability of training data in DLMs, we introduce*infilling extraction*, a data\-extraction protocol parameterized by an arbitrary binary mask that subsumes prefix\-only probing and accounts for the bidirectional inductive bias of DLMs\. Instantiating it on LLaDA\-8B and Dream\-7B across five extraction modes, three training pipelines, and three corpora covering verbatim and partial leakage, we find that mask geometry governs extractability: edge\-conditioned masks*extract up to three times more*verbatim sequences than prefix\-conditioned ones, and bidirectional access opens channels inaccessible in autoregressive models\. In particular, we show that a realistic adversary with access to training data where personally identifiable information has been redacted, can even achieve higher recall on extracting redacted email addresses from DLMs than from scale\-matched autoregressive models\. Tunable parameters for decoding measurably affect extraction performance, while a follow\-up supervised finetuning stage does not eliminate the prior memorization\.

## 1Introduction

By learning statistical patterns over vast and diverse text, large language models \(LLMs\) exhibit surprising generalization ability, successfully adapting to a wide variety of downstream tasks\. However, this reliance on training data comes at a cost: models do not merely generalize, but also memorize portions of their training data, raising serious concerns about the leakage of personally identifiable information \(PII\), the reproduction of copyrighted material, and the disclosure of confidential content embedded in training corpora\[[5](https://arxiv.org/html/2605.24173#bib.bib35),[23](https://arxiv.org/html/2605.24173#bib.bib36),[19](https://arxiv.org/html/2605.24173#bib.bib37),[28](https://arxiv.org/html/2605.24173#bib.bib38)\]\. A prominent line of work demonstrating these risks isdata extraction attacks\[[6](https://arxiv.org/html/2605.24173#bib.bib39),[31](https://arxiv.org/html/2605.24173#bib.bib40),[46](https://arxiv.org/html/2605.24173#bib.bib41),[15](https://arxiv.org/html/2605.24173#bib.bib2)\], in which an adversary crafts prompts that induce the model to regurgitate sequences from its training set\.

While memorization and data extraction attacks on autoregressive language models \(ARMs\) have been studied extensively\[[43](https://arxiv.org/html/2605.24173#bib.bib42)\], a fundamentally different paradigm of text generation has recently emerged: diffusion language models \(DLMs\)\[[26](https://arxiv.org/html/2605.24173#bib.bib9),[14](https://arxiv.org/html/2605.24173#bib.bib7),[32](https://arxiv.org/html/2605.24173#bib.bib12),[45](https://arxiv.org/html/2605.24173#bib.bib14),[44](https://arxiv.org/html/2605.24173#bib.bib29)\]\. Rather than predicting the next token strictly left\-to\-right, DLMs learn to iteratively denoise corrupted text and can generate multiple tokens at arbitrary positions per step\[[2](https://arxiv.org/html/2605.24173#bib.bib25),[27](https://arxiv.org/html/2605.24173#bib.bib27),[37](https://arxiv.org/html/2605.24173#bib.bib28),[16](https://arxiv.org/html/2605.24173#bib.bib26)\], offering a promising path toward faster and more flexible language models\[[36](https://arxiv.org/html/2605.24173#bib.bib22),[13](https://arxiv.org/html/2605.24173#bib.bib20),[21](https://arxiv.org/html/2605.24173#bib.bib21),[4](https://arxiv.org/html/2605.24173#bib.bib4),[3](https://arxiv.org/html/2605.24173#bib.bib3)\]\. Despite their growing scale and deployment, memorization characteristics of DLMs and their vulnerability to data extraction remain largely unexplored\.

In this work, we close this gap by developing a comprehensive framework for evaluating data extractability in large DLMs\. Our central observation is that estimating the risk of training data extraction in DLMs using only prefixes from training data records as prompts\[[29](https://arxiv.org/html/2605.24173#bib.bib11)\]\(also known as the*prefix\-conditioned protocol*\) can significantly underestimate the risk because this protocol, inherited from exploring data extraction in ARMs, captures only one facet of memorization: it asks what the model produces*next*, but not what it would fill in given*arbitrary surrounding context*\. We therefore introduce*infilling extraction*, a strictly more general protocol parameterized by a binary mask𝐦\\mathbf\{m\}\. The unmasked positions of a training sequence constitute the context revealed to a DLM, which is asked to reconstruct the text corresponding to the masked positions\. The training sequence is considered extracted if the reconstruction matches the masked text\.

Within this protocol, we instantiate five canonical access modes which we termprefix/suffix\-conditioned,middlepart\-conditioned,edge\-conditioned, and uniformlyrandomgeometries, plus a*targeted*variant in which only sensitive entities are masked\. We investigate three training pipelines that cover how DLMs are realistically exposed to data, including full fine\-tuning \(FT\), low\-rank adaptation \(LoRA\), and supervised fine\-tuning \(SFT\) under a chat template, and evaluate on three datasets that surface complementary risks: MIMIR Pile for verbatim extraction, the Enron email PII extraction, and Tulu3 for prompt/response leakage in instruction\-tuned deployments\. We additionally trace how the practitioner\-tunable parameters of decoding \(i\.e\., number of denoising steps, semi\-autoregressive block size, sampling temperature, and remasking strategy\) affect extractability\.

Our experiments on LLaDA\-8B\[[32](https://arxiv.org/html/2605.24173#bib.bib12)\]and Dream\-7B\[[45](https://arxiv.org/html/2605.24173#bib.bib14)\]uncover several patterns thatprefix\-conditioned evaluation alone would have missed\. \(i\) Mask geometry governs extractability\. At matched budget,edge\-conditioned masks*extract up to three times more verbatim sequences*thanprefix\-conditioning, evidencing the bidirectional inductive bias of DLMs\. \(ii\) Bidirectional access opens new leakage channels\. Adversaries can recover user prompts from responses and performtargetedPII reconstruction, both invisible underprefix\-conditioned probing\. \(iii\) Focusing only onprefix\-conditioned access\[[29](https://arxiv.org/html/2605.24173#bib.bib11)\]may underestimate the risk of training data extraction risks in DLMs\. For example, if the adversary obtains a redacted version of the training data, through mandated disclosure \(e\.g\., regulatory or court\-ordered release with PII redaction\) or via a partial data leak—both of which are realistic scenarios\[[33](https://arxiv.org/html/2605.24173#bib.bib46),[11](https://arxiv.org/html/2605.24173#bib.bib47),[39](https://arxiv.org/html/2605.24173#bib.bib48),[35](https://arxiv.org/html/2605.24173#bib.bib49)\]—targetedinfilling against a DLM can achieve higher PII recall than from a scale\-matched ARM, whereas using onlyprefix\-conditioned extraction on DLMs will not\. \(iv\) Decoding hyperparameters serve as practical amplification or mitigation levers, and a follow\-up SFT stage redistributes but does not close the channel that infilling exposes\.

We summarize our contributions as follows:

- •We formalize*infilling extraction*\([Section˜3](https://arxiv.org/html/2605.24173#S3)\), a mask\-parameterized data\-extraction protocol, generalizingprefix\-conditioned extraction and accounting for bidirectional bias of DLMs\.
- •We instantiate the protocol with five access modes plus a targeted\-PII variant, three training pipelines, and four decoding parameters, yielding thefirst systematic study of memorization and training\-data extraction in large DLMsacross access patterns, exposure regimes, and inference\-time configurations, showing that*DLMs pose a greater risk of training\-data leakage than previously thought*\([Section˜4\.1](https://arxiv.org/html/2605.24173#S4.SS1)–[4\.4](https://arxiv.org/html/2605.24173#S4.SS4)\)\.
- •We empirically characterize how mask geometry, decoding parameters, and post\-training stages shape extractability on LLaDA\-8B and Dream\-7B,*identifying new leakage channels*structurally inaccessible in ARMs and discuss inference\-time mitigations \([Section˜4\.5](https://arxiv.org/html/2605.24173#S4.SS5)–[4\.7](https://arxiv.org/html/2605.24173#S4.SS7)\)\.

## 2Preliminaries

### 2\.1Diffusion Language Models

Whereas ARMs predict the next token strictly from left to right, DLMs cast generation as iterative denoising: a forward process corrupts a sequence by replacing tokens with a special\[MASK\]symbol, and a learned reverse process recovers them from the partially masked context\. Representative masked DLMs such as LLaDA\[[32](https://arxiv.org/html/2605.24173#bib.bib12)\]and Dream\[[45](https://arxiv.org/html/2605.24173#bib.bib14)\]are trained with a cross\-entropy loss over the masked positions,

ℒ​\(θ\)=−𝔼t,𝐱0,𝐦t​\[1t​∑i=1L𝐦ti⋅log⁡pθ​\(𝐱0i∣𝐱0⊗𝐦t\)\],\\mathcal\{L\}\(\\theta\)=\-\\mathbb\{E\}\_\{t,\\mathbf\{x\}\_\{0\},\\mathbf\{m\}\_\{t\}\}\\\!\\left\[\\frac\{1\}\{t\}\\sum\_\{i=1\}^\{L\}\\mathbf\{m\}\_\{t\}^\{i\}\\cdot\\log p\_\{\\theta\}\(\\mathbf\{x\}\_\{0\}^\{i\}\\mid\\mathbf\{x\}\_\{0\}\\otimes\\mathbf\{m\}\_\{t\}\)\\right\],where𝐱0\\mathbf\{x\}\_\{0\}is a length\-LLsequence,t∈\(0,1\]t\\\!\\in\\\!\(0,1\]is a uniformly sampled corruption level,𝐦t\\mathbf\{m\}\_\{t\}is a random binary mask with‖𝐦t‖0=⌊t​L⌋\\\|\\mathbf\{m\}\_\{t\}\\\|\_\{0\}=\\lfloor tL\\rfloor,⊗\\otimesis mask operation and𝐱0⊗𝐦t\\mathbf\{x\}\_\{0\}\\otimes\\mathbf\{m\}\_\{t\}masks a fractionttof𝐱0\\mathbf\{x\}\_\{0\}\.

At inference, a DLM unmasks multiple positions per step overTTdenoising steps\[[42](https://arxiv.org/html/2605.24173#bib.bib13),[18](https://arxiv.org/html/2605.24173#bib.bib17)\]; which positions to commit is governed by a*remasking*strategy \(random, confidence\-based, or uncertainty\-based\)\[[32](https://arxiv.org/html/2605.24173#bib.bib12),[45](https://arxiv.org/html/2605.24173#bib.bib14)\]\. Semi\-autoregressive decoding further partitions the sequence into blocks of sizebb, decoding within each block in parallel while preserving order across blocks\[[42](https://arxiv.org/html/2605.24173#bib.bib13),[1](https://arxiv.org/html/2605.24173#bib.bib23)\], and sampling of temperatureτ\\taucan be applied to the per\-position distributions\. Beyond left\-to\-right prompted generation, the bidirectional formulation enables*infilling*\[[48](https://arxiv.org/html/2605.24173#bib.bib1),[12](https://arxiv.org/html/2605.24173#bib.bib24),[41](https://arxiv.org/html/2605.24173#bib.bib33)\]: given a sequence with arbitrary masked spans, the model predicts the missing tokens from both preceding and following context \(e\.g\., filling “Joe Biden” into “\[MASK\] \[MASK\] is the 46th president of the United States”\)\.

### 2\.2Data Extraction

In data extraction attacks, an adversary queries a model to elicit sequences appearing in the training set, potentially recovering PII, copyrighted text, or other sensitive content\[[47](https://arxiv.org/html/2605.24173#bib.bib18),[40](https://arxiv.org/html/2605.24173#bib.bib45)\]\. The typical data extraction protocol,prefix\-conditioning, is a natural choice against ARMs: a training sequence𝐱∈𝒱n\\mathbf\{x\}\\in\\mathcal\{V\}^\{n\}is split into a prefix𝐱≤k\\mathbf\{x\}\_\{\\leq k\}and suffix𝐱\>k\\mathbf\{x\}\_\{\>k\}, and𝐱\\mathbf\{x\}is considered extracted if the text predicted by the model, when prompted with the prefix, matches the suffix,gθ​\(𝐱≤k\)=\(𝐱≤k,𝐱\>k\)=𝐱g\_\{\\theta\}\(\\mathbf\{x\}\_\{\\leq k\}\)=\(\\mathbf\{x\}\_\{\\leq k\},\\mathbf\{x\}\_\{\>k\}\)=\\mathbf\{x\}, in the case of a verbatim match, wheregθg\_\{\\theta\}is the decoding function\. Memorization is summarized by the extraction rate over training samples, often paired with a verbatim\-match criterion such as a contiguous5050\-token span\[[6](https://arxiv.org/html/2605.24173#bib.bib39),[5](https://arxiv.org/html/2605.24173#bib.bib35)\]; probabilistic variants replace the deterministic match with the probability of producing the suffix under stochastic sampling\[[15](https://arxiv.org/html/2605.24173#bib.bib2)\]\.

### 2\.3Related Work

Two recent works study data extraction in DLMs along distinct axes\.Luoet al\.\[[29](https://arxiv.org/html/2605.24173#bib.bib11)\]generalize prefix\-based\(n,p\)\(n,p\)\-discoverable extraction\[[15](https://arxiv.org/html/2605.24173#bib.bib2)\]to arbitrary masks\. Under*prefix*\-conditioned PII extraction, they report that DLMs leak less than scale\-matched ARMs\.Chenet al\.\[[8](https://arxiv.org/html/2605.24173#bib.bib19)\]instead target membership inference attacks \(MIAs\), exploiting multiple maskable configurations of DLMs via progressively densified masks and sign\-based, inverse\-step\-weighted aggregation to substantially improve the effectiveness of MIAs on fine\-tuned LLaDA\-8B and Dream\-7B\.

Our work differs in both the problem addressed and the extraction protocol used\. Our threat model is the extraction of verbatim training data or PIIs present in training data, a strictly stronger criterion than the binary membership decision ofChenet al\.\[[8](https://arxiv.org/html/2605.24173#bib.bib19)\]\. WhereasLuoet al\.\[[29](https://arxiv.org/html/2605.24173#bib.bib11)\]limits the adversary’s access pattern to only prefix\-conditioning \(presumably to facilitate direct comparison with ARMs\), we argue that it artificially restricts an adversary’s capabilities in the case of DLMs and therefore underestimates the risk of data leakage\. We parameterize extraction by an arbitrary mask𝐦\\mathbf\{m\}and instantiate it across five access modes, three training pipelines, and four decoding parameters\. Our results illuminate the findings of : although we confirm their observation that prefix\-conditioned leakage appears modest, we show that alternative mask patterns result in substantially more leakage\.

## 3Infilling Extraction

Memorization in DLMs can be exploited by providing partial context anywhere in a sequence; prefix\-only conditioning, therefore, is just one possible approach\. DLMs natively support*infilling*, in which any subset of tokens may be masked and reconstructed\. This enables a strictly more general extraction protocol than prefix\-only conditioning\.

###### Definition 3\.1\(Infilling Extraction\)\.

Letθ\\thetadenote a DLM,𝐱∈𝒱n\\mathbf\{x\}\\in\\mathcal\{V\}^\{n\}a training sequence,𝐦∈\{0,1\}n\\mathbf\{m\}\\in\\\{0,1\\\}^\{n\}a binary mask, andgθg\_\{\\theta\}the decoding function\. We say𝐱\\mathbf\{x\}is*verbatim extracted*under infilling if

gθ​\(𝐱⊗𝐦\)=𝐱\.\\displaystyle g\_\{\\theta\}\(\\mathbf\{x\}\\otimes\\mathbf\{m\}\)=\\mathbf\{x\}\.\(1\)

The mask𝐦\\mathbf\{m\}specifies the access pattern \(which tokens are revealed\), whilegθg\_\{\\theta\}instantiates the inference procedure characterized in[Section˜2\.1](https://arxiv.org/html/2605.24173#S2.SS1)along four parameters\{T,b,τ,remask strategy\}\\\{T,b,\\tau,\\text\{remask strategy\}\\\}\. We discuss the effect of possible access patterns \([Section˜3\.1](https://arxiv.org/html/2605.24173#S3.SS1)\) and decoding parameters ofgθg\_\{\\theta\}\([Section˜4\.5](https://arxiv.org/html/2605.24173#S4.SS5)\)\.

### 3\.1Extraction Modes

Infilling extraction is parameterized by the binary mask𝐦∈\{0,1\}n\\mathbf\{m\}\\in\\\{0,1\\\}^\{n\}: positions with𝐦i=1\\mathbf\{m\}\_\{i\}=1are masked and to be reconstructed, while the unmasked positions constitute the context revealed to the model\. Different patterns of𝐦\\mathbf\{m\}probe different forms of extractability; for instance, under a fixed context budget, does the same training sequence remain extractable when the revealed context is moved from the beginning to the end of the sequence? To make this comparison concrete and to span the access patterns enabled by bidirectional decoding, we fix a budget ofkkunmasked tokens \(out ofnn\) and define five canonical modes:

- •Prefix\-conditioned: the firstkktokens are revealed and the remainingn−kn\-kare masked\. This recovers the de facto setting of prior ARM extraction work and serves as our reference\.
- •Suffix\-conditioned: only the lastkktokens are revealed, with the precedingn−kn\-kmasked\. This probes reconstruction in reverse contextual order; under the chat template, it corresponds to recovering a prompt \(conversation history\) from a known response\.
- •Edge\-conditioned: the firstk/2k/2and lastk/2k/2tokens are revealed, and the middle is masked, testing whether bracketing context on both sides amplifies recall relative to one\-sided context of the same total budget\.
- •Middle\-conditioned: a contiguous span ofkktokens in the middle is revealed, and both ends are masked, which is complementary toedge\. This mode isolates whether internal context alone can elicit memorized continuations\.
- •Random: a uniformly random subset ofkkpositions is revealed, and the remainingn−kn\-kare masked, probing recovery from arbitrarily distributed context and reflecting diffuse, non\-localized memorization\.

Beyond these canonical budget\-constrained modes, the PII experiments in[Section˜4](https://arxiv.org/html/2605.24173#S4)additionally consider a*targeted*infilling setting, in which only the tokens of a sensitive entity are masked, and the entire surrounding sequence is revealed as context\. Targeted infilling probes entity\-level rather than sequence\-level memorization and characterizes an upper bound on PII extractability under a maximally informed adversary\.

### 3\.2Extraction Task and Metrics

We evaluate infilling extraction attacks on two tasks:verbatim extractionandPII extraction\. The former measures sequence\-level memorization, i\.e\., the extent to which the model reproduces specific training samples; the latter measures entity\-level memorization of PII contained in the training data\.

For verbatim extraction, we sample a set𝐗\\mathbf\{X\}of snippets from the training dataset and mask them according to a predefined context access and budget\. We then perform conditional orrandominfilling based on the mask pattern\. We compute the attack success rate \(ASR\) as the proportion of samples extracted verbatim:

ASR​\(θ,𝐗\)≔1\|𝐗\|​∑𝐱∈𝐗𝟏​\[gθ​\(𝐱⊗𝐦\)=𝐱\]\.\\displaystyle\\textbf\{ASR\}\(\\theta,\\mathbf\{X\}\)\\coloneqq\\frac\{1\}\{\|\\mathbf\{X\}\|\}\\sum\_\{\\mathbf\{x\}\\in\\mathbf\{X\}\}\\mathbf\{1\}\[g\_\{\\theta\}\(\\mathbf\{x\}\\otimes\\mathbf\{m\}\)=\\mathbf\{x\}\]\.Since ASR is a strict metric that captures only exact extraction, we further relax the criterion by introducing the token\-wise extraction rate \(TER\), which measures the fraction of correctly recovered tokens at masked positions:

TER​\(θ,𝐗\)≔1\|𝐗\|​∑𝐱∈𝐗1‖𝐦‖0​∑i=1n𝐦i⋅𝟏​\[gθ​\(𝐱⊗𝐦\)i=𝐱i\]\.\\displaystyle\\textbf\{TER\}\(\\theta,\\mathbf\{X\}\)\\coloneqq\\frac\{1\}\{\|\\mathbf\{X\}\|\}\\sum\_\{\\mathbf\{x\}\\in\\mathbf\{X\}\}\\frac\{1\}\{\\\|\\mathbf\{m\}\\\|\_\{0\}\}\\sum\_\{i=1\}^\{n\}\\mathbf\{m\}\_\{i\}\\cdot\\mathbf\{1\}\[g\_\{\\theta\}\(\\mathbf\{x\}\\otimes\\mathbf\{m\}\)\_\{i\}=\\mathbf\{x\}\_\{i\}\]\.
For PII extraction, we prompt the model with an unmaskedprefix,suffix,edge,middlecontext, as well as conducttargetedinfilling\. For conditional extraction, we apply a functionℛ\\mathcal\{R\}\(e\.g\., regex rule\) to extract PII entities from both the generated text and the ground truth, and compute recall and precision as follows:

Recall​\(θ,𝐗\)\\displaystyle\\textbf\{Recall\}\(\\theta,\\mathbf\{X\}\)≔\|ℛ​\(\{gθ​\(𝐱⊗𝐦\)∣𝐱∈𝐗\}\)∩ℛ​\(𝐗\)\|\|ℛ​\(𝐗\)\|,\\displaystyle\\coloneqq\\frac\{\\left\|\\mathcal\{R\}\\\!\\left\(\\\{g\_\{\\theta\}\(\\mathbf\{x\}\\otimes\\mathbf\{m\}\)\\mid\\mathbf\{x\}\\in\\mathbf\{X\}\\\}\\right\)\\cap\\mathcal\{R\}\(\\mathbf\{X\}\)\\right\|\}\{\|\\mathcal\{R\}\(\\mathbf\{X\}\)\|\},Precision​\(θ,𝐗\)\\displaystyle\\textbf\{Precision\}\(\\theta,\\mathbf\{X\}\)≔\|ℛ​\(\{gθ​\(𝐱⊗𝐦\)∣𝐱∈𝐗\}\)∩ℛ​\(𝐗\)\|\|ℛ​\(\{gθ​\(𝐱⊗𝐦\)∣𝐱∈𝐗\}\)\|\.\\displaystyle\\coloneqq\\frac\{\\left\|\\mathcal\{R\}\\\!\\left\(\\\{g\_\{\\theta\}\(\\mathbf\{x\}\\otimes\\mathbf\{m\}\)\\mid\\mathbf\{x\}\\in\\mathbf\{X\}\\\}\\right\)\\cap\\mathcal\{R\}\(\\mathbf\{X\}\)\\right\|\}\{\\left\|\\mathcal\{R\}\\\!\\left\(\\\{g\_\{\\theta\}\(\\mathbf\{x\}\\otimes\\mathbf\{m\}\)\\mid\\mathbf\{x\}\\in\\mathbf\{X\}\\\}\\right\)\\right\|\}\.Fortargetedinfilling, we report the TER, recall of exactly matched PII spans, and the error\-tolerant variants of recall\.

### 3\.3Training\-Induced Memorization

As it is infeasible to access the accurate pre\-training data of existing large DLMs like LLaDA and Dream, we instead induce memorization via fine\-tuning and extract the data used in fine\-tuning\. To cover the regimes under which DLMs are realistically exposed to training data, we instantiate the infilling\-extraction protocol on three pipelines\[[50](https://arxiv.org/html/2605.24173#bib.bib16)\]\.

First, FT updates all model parameters on the target corpus under the masked\-denoising objective of the underlying DLM\. It simulates the continual pretraining process\. Second, LoRA\[[17](https://arxiv.org/html/2605.24173#bib.bib34)\]freezes the base model and trains a small set of low\-rank update matrices, mirroring how DLMs are most likely to be specialized in production\. The denoising objective is unchanged\. Last, SFT trains the model on \(prompt, response\) pairs wrapped in the chat template, with the loss applied over the full templated sequence so that both prompt and response tokens participate in the denoising objective\. This pipeline targets aligned, deployment\-facing DLMs and enables the prompt\-/response\-conditioned extraction modes used to probe whether private user prompts can be recovered from known model responses\.

### 3\.4Research Questions

The infilling extraction framework enables a systematic investigation of memorization in DLMs\. We pose the following research questions and structure our experiments in[Section˜4](https://arxiv.org/html/2605.24173#S4)around them:

RQ1: How does memorization manifest in DLMs under the infilling\-extraction protocol?By applying the infilling extraction protocol across different mask patterns, we can characterize how the placement of context affects the model’s ability to recall training data \([Sections˜4\.2](https://arxiv.org/html/2605.24173#S4.SS2)and[4\.3](https://arxiv.org/html/2605.24173#S4.SS3)\)\. We also analyze the influence of post\-training adaptation on the prior memorization, testing whether downstream alignment attenuates pre\-existing leakage \([Section˜4\.7](https://arxiv.org/html/2605.24173#S4.SS7)\)\.

RQ2: What risks does extractable memorization in DLMs entail?Beyond quantifying the extent of memorization, we evaluate the nature of the extracted content to assess the practical risks\. We analyze whether the extracted sequences contain PII \([Section˜4\.3](https://arxiv.org/html/2605.24173#S4.SS3)\)\. Moreover, in the context of SFT, we test whether user prompts can be extracted from known model responses, which would indicate a risk of prompt leakage in deployed systems \([Section˜4\.2](https://arxiv.org/html/2605.24173#S4.SS2)\)\.

RQ3: Do DLMs pose lower data\-leakage risk than ARMs in realistic settings?\[[29](https://arxiv.org/html/2605.24173#bib.bib11)\]reported that DLMs have a lower risk of training data leakage than scale\-matched ARMs under prefix\-conditioned extraction\. We provide a more nuanced understanding of their findings: under a realistic scenario where the adversary has redacted training samples, DLMs can usetargetedinfilling to recover PII \([Section˜4\.4](https://arxiv.org/html/2605.24173#S4.SS4)\)\.

RQ4: How do decoding\-time parameters modulate extraction?The decoding parameters ofgθg\_\{\\theta\}\(see[Section˜2\.1](https://arxiv.org/html/2605.24173#S2.SS1)\) are routinely tuned by practitioners for speed and quality\. We test which of them materially modulate extraction so that the same factor can act as inference\-time amplifications or mitigations \([Section˜4\.5](https://arxiv.org/html/2605.24173#S4.SS5)\)\. Additionally, we trace the per\-step trajectory of recovered content to test whether memorized tokens are committed early \(the model effectively already “knows” the answer\) or surface only near the end of denoising \([Section˜4\.6](https://arxiv.org/html/2605.24173#S4.SS6)\)\.

## 4Experiments

### 4\.1Setup

Datasets\.We instantiate the infilling\-extraction protocol on three datasets, covering verbatim extraction, PII extraction, and prompt/response extraction\. FollowingZhanget al\.\[[48](https://arxiv.org/html/2605.24173#bib.bib1)\], we fine\-tune on thefull Pilesubset of MIMIR\[[10](https://arxiv.org/html/2605.24173#bib.bib32)\]\(10,00010\{,\}000sequences\) and evaluate verbatim extraction on uniformly sampled100100\-token snippets from the training set\. For PII extraction, we fine\-tune on10,00010\{,\}000messages from the Enron Email corpus\[[20](https://arxiv.org/html/2605.24173#bib.bib51)\]\. For each extraction, we sample100100\-token snippets whose masked region contains target entities; we report results on two PII types,email addressandphone number\. For SFT, we fine\-tune on10,00010\{,\}000\(prompt, response\) pairs from Tulu3\[[22](https://arxiv.org/html/2605.24173#bib.bib43)\]and restrict evaluation to records whose prompt and response are each at most100100tokens\.

Models\.We build on the open\-source diffusion\-LM framework ofZhouet al\.\[[50](https://arxiv.org/html/2605.24173#bib.bib16)\], which unifies training, inference, and evaluation across recent DLMs\. We use the pre\-trained LLaDA\-8B\-Base\[[32](https://arxiv.org/html/2605.24173#bib.bib12)\]and Dream\-7B\-Base\[[45](https://arxiv.org/html/2605.24173#bib.bib14)\]as DLM backbones, and LLaMA\-2\-7B\[[38](https://arxiv.org/html/2605.24173#bib.bib44)\]as ARM baseline at matched scale\. To ensure measurable memorization, all models are fine\-tuned for55epochs on every task; full training details are deferred to[Section˜A\.2](https://arxiv.org/html/2605.24173#A1.SS2)\. By default, we use greedy sampling and decode one token per step; LLaDA uses low\-confidence remasking with no semi\-autoregressive blocking, and Dream uses entropy\-based remasking\. We vary each of these parameters in[Section˜4\.5](https://arxiv.org/html/2605.24173#S4.SS5)\.

Infilling extraction\.On MIMIR Pile we evaluate all five extraction modes \(prefix,suffix,edge,middle,random\)\. On Enron, we evaluateprefix,suffix,edge,middle, and introducetargetedinfilling\. We use the Presidio analyzer\[[30](https://arxiv.org/html/2605.24173#bib.bib50)\]for email address recognition and a regex rule for phone numbers as the extraction functionℛ\\mathcal\{R\}in[Section˜3\.2](https://arxiv.org/html/2605.24173#S3.SS2)\. On Tulu3, we evaluate two SFT\-specific variants under the chat template,prompt\-conditionedandresponse\-conditionedextraction, in which the response \(resp\. prompt\) is masked given the other side as context\.

Table 1:Verbatim extraction \(%\) under different infilling extraction modes on MIMIR Pile\. We report ASR and TER averaged over five models trained with different random seeds, along with the standard deviation\.Bolddenotes the best score per metric, and wehighlightcases where the best result exceedsprefix’s ASR by at least2×2\\times\.
### 4\.2Verbatim Extraction

We evaluate the five infilling modes on MIMIR Pile and the two SFT\-specific modes on Tulu3\.

Mask geometry shapes verbatim extraction\.Under a fixed budget ofkkrevealed tokens, ASR varies sharply with where those tokens sit\. On MIMIR Pile \([Table˜1](https://arxiv.org/html/2605.24173#S4.T1)\), LLaDA\-8B\-FT peaks atedge\(0\.41%0\.41\\%\) mode, that is, over2×2\\timesthe one\-sidedprefix\(0\.18%0\.18\\%\); theedge/random\>\>prefix/suffix/middleordering in ASR reproduces on Dream\-7B and under LoRA across both backbones\. Verbatim extraction is thus governed largely by whether that context spans the structural endpoints of the training sequence\. It confirms the bidirectional inductive bias of DLMs\.

Table 2:Verbatim extraction \(%\) on Tulu3\.Token\-level recovery does not imply verbatim extraction\.[Table˜1](https://arxiv.org/html/2605.24173#S4.T1)reveals a striking disparity between TER and ASR: underrandommasking, LLaDA\-8B\-FT recovers56\.22%56\.22\\%of masked tokens yet reproduces only0\.37%0\.37\\%of snippets verbatim, with Dream\-7B\-FT exhibiting an even sharper gap \(55\.85%55\.85\\%vs\.0\.11%0\.11\\%\)\. This high token accuracy is largely an artifact of the scaffolding provided by the revealed tokens, and does not translate into successful verbatim extraction\. We therefore interpret TER as a loose indicator of memorization risk and adopt ASR as the operational threat metric\.

Prompts are recoverable from responses\.On Tulu3, the standardprompt\-conditioneddirection reaches ASR10\.32%10\.32\\%/10\.62%10\.62\\%for LLaDA\-/Dream\-SFT, but the reverseresponse\-conditioneddirection, where the response is revealed and prompt is to be inferred, also remains non\-trivial \(ASR3\.36%3\.36\\%/1\.78%1\.78\\%\)\. This leakage channel is structurally inaccessible to ARMs: bidirectional denoising allows an adversary holding a deployed model’s output to recover the originating user query, with direct implications for instruction\-tuned DLMs queried with sensitive prompts\.

Table 3:PII extraction \(%\) under different infilling extraction modes on Enron Email dataset\. Wehighlightcases where the best result exceedsprefix’s recall by at least2×2\\times\.
### 4\.3PII Extraction

We next ask whether infilling extraction can recover PII from the Enron Email training data\.[Table˜3](https://arxiv.org/html/2605.24173#S4.T3)reports recall and precision for email addresses and phone numbers under the four conditional modes\.[Table˜4](https://arxiv.org/html/2605.24173#S4.T4)probes the worst case in which only the PII tokens are masked, which serves as anupper boundfor extraction performance\.

Bidirectional context amplifies PII leakage\.As shown in[Table˜3](https://arxiv.org/html/2605.24173#S4.T3), theedgeandmiddlemodes, which exploit bidirectional context, consistently extract more PII entities than the one\-sidedprefixandsuffixmodes\. On LLaDA\-8B\-FT,edgeachieves6\.33%6\.33\\%recall on emails, that is, around3×3\\timesthat ofprefix\(1\.95%1\.95\\%\) andsuffix\(2\.20%2\.20\\%\) under a matched budget\. In terms of extraction precision,edgeandmiddlelikewise outperform their one\-sided counterparts in most cases\.

Targeted infilling\.To capture the strongest data leakage risk in realistic deployment scenarios, we consider the following threat model\. The model trainer \(e\.g\., companies, government agencies, hospitals\) has full access to sensitive data and uses it to train a DLM\. The adversary later obtains a redacted version of the training data, either through partial leakage or because the trainer was compelled \(e\.g\., by regulation or court order\) to release the training data with PII redacted\[[33](https://arxiv.org/html/2605.24173#bib.bib46),[11](https://arxiv.org/html/2605.24173#bib.bib47),[39](https://arxiv.org/html/2605.24173#bib.bib48),[35](https://arxiv.org/html/2605.24173#bib.bib49)\]\. The adversary’s goal is to recover the redacted content by exploiting the model\. Specifically, the adversary aligns masks with the redacted positions and queries the model to fill in the original PII, using the surrounding unredacted text as context\.

[Table˜4](https://arxiv.org/html/2605.24173#S4.T4)reports TER and recallk, the fraction of entities recovered with at mostkkmistaken tokens\. Two findings stand out\. \(i\) Fine\-tuning sharply increases extractability: on LLaDA\-8B, exact\-match recall rises from1\.4%1\.4\\%to40\.5%40\.5\\%for emails and from0\.3%0\.3\\%to28\.9%28\.9\\%for phone numbers, with the same trend on Dream\-7B\. This confirms that the recovered entities are genuine training strings rather than distributional guesses\. \(ii\) Recall2exceeds exact\-match recall by6\.86\.8points for emails and12\.412\.4points for phone numbers\. Even partial recoveries can leak meaningful information, and a small number of mistaken tokens is often enough to seed enumeration attacks\. Therefore,targetedinfilling characterizes the upper\-bound risk surface for sensitive entities embedded in DLM training data\.

Table 4:Targetedinfilling results \(%\) on Enron dataset\. We report TER and error\-tolerant recall metrics\. Recallkdenotes the fraction of masked entities for which at mostkktokens are incorrectly recovered\.
Figure 1:Comparison between ARM and DLMs on Enron\.![Refer to caption](https://arxiv.org/html/2605.24173v1/x1.png)
### 4\.4Comparison between ARMs and DLMs

[Figure˜1](https://arxiv.org/html/2605.24173#S4.F1)compares LLaDA\-8B and Dream\-7B against LLaMA2\-7B under matched fine\-tuning on the Enron dataset\. Here, we adopt adynamic prefixsetting, in which the revealed context for each sample is truncated immediately before the first PII token in the snippet\. This setting differs from the fixed\-budgetprefixused in[Table˜3](https://arxiv.org/html/2605.24173#S4.T3)and represents the strongestprefix\-conditioned extraction against ARMs\. Underdynamic prefixextraction alone, DLMs leak substantially less than the scale\-matched ARM, consistent withLuoet al\.\[[29](https://arxiv.org/html/2605.24173#bib.bib11)\]: LLaMA2 attains35\.9%35\.9\\%recall on email address, roughly2×2\\timesthat of LLaDA \(16\.3%16\.3\\%\) and12×12\\timesthat of Dream \(2\.9%2\.9\\%\)\. However, this comparison understates the DLM risk surface, becausedynamic prefixis the strongest attack the ARM admits, but not the strongest the DLM admits\. Once the adversary has access to the redacted sequences,targetedinfilling enables DLMs to match or exceed the ARM: LLaDA reaches40\.5%40\.5\\%recall on email addresses \(4\.64\.6points above LLaMA\) and LLaDA\-8B/Dream\-7B reach28\.9%28\.9\\%/35\.8%35\.8\\%recall on phone numbers, far above their ownprefix\-conditioned rates\. This contrast reveals thatprefix\-conditioned extraction substantially underestimates the data leakage risks of DLMs in realistic scenarios\.

### 4\.5Influence of Decoding Parameters

The generation functiongθg\_\{\\theta\}of a DLM exposes several parameters, i\.e\., denoising stepsTT, semi\-autoregressive block sizebb, sampling temperatureτ\\tau, and remasking strategy, that can be tuned for speed and quality\. We separate these decoding parameters and investigate the influence of each component\.[Figure˜2](https://arxiv.org/html/2605.24173#S4.F2)sweeps each on LLaDA\-8B\-FT for Enron emails\.

Decoding efficiency confines extraction efficacy\.A small number of denoising stepsTTcommits many positions in parallel per step under coarse posteriors, while a largeTTspreads the trajectory over more refinement rounds\.[Figure˜2\(a\)](https://arxiv.org/html/2605.24173#S4.F2.sf1)shows recall rises monotonically withTTon Enron emails\. Thus, throughput\-oriented configurations \(smallTT\) obscure memorized content under coarse\-grained commits, while the accuracy\-oriented settings expose the most training data\.

Decoding parallelism benefits extraction\.Semi\-autoregressive decoding partitions the sequence into blocks of sizebb, decoding within each block in parallel while preserving order across blocks\. Smallbbapproximates left\-to\-right autoregressive decoding \(each block holds few tokens, committed in turn\), whereas largebbrecovers fully\-parallel diffusion \(the whole sequence is one bidirectional block\)\.[Figure˜2\(b\)](https://arxiv.org/html/2605.24173#S4.F2.sf2)shows recall increases withbb, with the gains realized oncebbleaves the AR\-like regime\. Restrictingbbto small values recovers the more limited memorization profile, identifying bidirectional, fully\-parallel denoising as a principal driver of infilling\-based extraction\.

Sampling diversity slightly compromises extraction\.Temperatureτ\\taucontrols how concentrated the per\-position predictive distribution is:τ→0\\tau\\\!\\to\\\!0approaches greedy decoding, while a higherτ\\tauflattens the distribution and injects stochasticity at committed tokens\.[Figure˜2\(c\)](https://arxiv.org/html/2605.24173#S4.F2.sf3)shows recall is relatively stable forτ\\taubelow11, with greedy decoding yielding the strongest extraction for three of the four infilling modes\. The effect is smaller in magnitude than theTTandbbsweeps because candidate positions already correspond to sharply peaked posteriors, and modest stochasticity has limited room to dilute the top\-1 prediction\.

Confident remasking amplifies extraction\.At each denoising step, the remasking strategy decides which tentative predictions to commit and which to re\-mask for further refinement\.[Figure˜2\(d\)](https://arxiv.org/html/2605.24173#S4.F2.sf4)compares low\-confidence remasking against random remasking on LLaDA\-8B\-FT: confidence\-based remasking, which preserves high\-confidence predictions and re\-corrupts uncertain ones, yields markedly stronger extraction, while random remasking discards the model’s confidence signal and slows convergence toward memorized targets\. High confidence at a position is itself a marker of training\-set provenance\[[8](https://arxiv.org/html/2605.24173#bib.bib19)\], so committing those positions early lets the remaining trajectory consolidate around them and amplifies memorized content over the course of denoising\.

Based on these observations, an adversary targeting open\-weight DLMs can tune decoding parameters to maximize extractability, while a model owner, as the defender, can select appropriate parameters by balancing throughput, quality, and data leakage resistance\.

![Refer to caption](https://arxiv.org/html/2605.24173v1/x2.png)\(a\)Number of stepsTT
![Refer to caption](https://arxiv.org/html/2605.24173v1/x3.png)\(b\)Block sizebb
![Refer to caption](https://arxiv.org/html/2605.24173v1/x4.png)\(c\)Temperatureτ\\tau
![Refer to caption](https://arxiv.org/html/2605.24173v1/x5.png)\(d\)Remasking strategy

Figure 2:Influence of decoding parameters for LLaDA\-8B\-FT model on extracting email addresses from Enron dataset\.![Refer to caption](https://arxiv.org/html/2605.24173v1/x6.png)\(a\)Email
![Refer to caption](https://arxiv.org/html/2605.24173v1/x7.png)\(b\)Phone

Figure 3:PII extraction trajectory during the decoding procedure\.
![Refer to caption](https://arxiv.org/html/2605.24173v1/x8.png)\(c\)Email
![Refer to caption](https://arxiv.org/html/2605.24173v1/x9.png)\(d\)Phone

Figure 4:PII extraction results before and after SFT\.

### 4\.6Extraction Trajectory

[Figure˜4](https://arxiv.org/html/2605.24173#S4.F4)traces per\-step PII recall on LLaDA\-8B\-FT\. Across all four modes, the recall rises steadily through roughly the first 30 steps of the trajectory and then enters a regime of plateau or low\-amplitude oscillation, with little further sustained gain in the remaining steps\. A natural mechanism behind this saturation is that each step fills masked positions with the model’s own predictions, so the decoding trajectory accumulates errors and progressively drifts from the ground\-truth surroundings; once this drift is large enough, subsequent commitments are pulled toward generic completions rather than ground\-truth PII, driving the marginal recall increment toward zero\.

### 4\.7Effect of SFT on PII Extractability

A typical deployment pipeline applies instruction tuning on top of domain fine\-tuning\. To test whether the second stage attenuates the memorization induced by the first, we continue to train LLaDA\-8B\-FT \(Enron\) with SFT on Tulu3 and re\-run the infilling extraction on the original Enron set \([Figure˜4](https://arxiv.org/html/2605.24173#S4.F4)\)\.

PII extractability largely persists through SFT, and the relative ordering of modes is preserved across stages\. SFT therefore redistributes the memorization budget without closing the structural leakage channel that infilling exposes\. The attenuation is entity\-type dependent\. Email recall drops under every mode; phone recall, by contrast, is essentially unchanged or slightly higher in three of four modes\. A hypothesis for this distinction is: emails lie within the natural\-language distribution that SFT updates and are partially overwritten, while short idiosyncratic numeric strings are orthogonal to the instruction\-following objective and survive intact\.

## 5Conclusion

We introduced*infilling extraction*, a mask\-parameterized data\-extraction protocol that subsumesprefixconditioning and matches the bidirectional inductive bias of DLMs\. On LLaDA\-8B and Dream\-7B, it surfaces extraction channels and poses data leakage risks invisible toprefix\-conditioned probing\.

Limitations and future work\.Our study has three main scope limitations\. First, because the pre\-training corpora of LLaDA and Dream are not publicly available, we induce memorization through fine\-tuning on three representative datasets rather than measuring it directly on the released base checkpoints\. Second, we examine only three exposure regimes and do not consider preference\-based post\-training, such as RLHF\[[49](https://arxiv.org/html/2605.24173#bib.bib15)\]\. Third, since a strict comparison between large DLMs and ARMs is infeasible due to differences in architecture and training paradigm, we conduct our comparison under a scale\-matched and tuning\-matched setting\. Looking forward, the DLM area is evolving rapidly, with new training paradigms and decoding acceleration algorithms appearing at a fast pace\[[18](https://arxiv.org/html/2605.24173#bib.bib17),[9](https://arxiv.org/html/2605.24173#bib.bib5),[24](https://arxiv.org/html/2605.24173#bib.bib8),[25](https://arxiv.org/html/2605.24173#bib.bib10)\]\. Characterizing how emerging techniques interact with memorization is a natural next step\. Beyond memorization itself, infilling extraction is a general primitive for probing what a bidirectional model has internalized\. We see promising connections to adjacent safety problems, including MIAs\[[8](https://arxiv.org/html/2605.24173#bib.bib19)\], unlearning, and uncertainty calibration\[[7](https://arxiv.org/html/2605.24173#bib.bib30),[34](https://arxiv.org/html/2605.24173#bib.bib31)\]\.

## Acknowledgements

This work is supported in part by the Wallenberg Visiting Professor Program, the Natural Sciences and Engineering Research Council of Canada \(grant number RGPIN\-2026\-04826\), and the Government of Ontario \(RE011\-038\)\.

## References

- \[1\]M\. Arriola, S\. S\. Sahoo, A\. Gokaslan, Z\. Yang, Z\. Qi, J\. Han, J\. T\. Chiu, and V\. Kuleshov\(2025\)Block diffusion: interpolating between autoregressive and diffusion language models\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=tyEyYT267x)Cited by:[§2\.1](https://arxiv.org/html/2605.24173#S2.SS1.p2.3)\.
- \[2\]J\. Austin, D\. D\. Johnson, J\. Ho, D\. Tarlow, and R\. Van Den Berg\(2021\)Structured denoising diffusion models in discrete state\-spaces\.Advances in neural information processing systems34,pp\. 17981–17993\.Cited by:[§1](https://arxiv.org/html/2605.24173#S1.p2.1)\.
- \[3\]T\. Bie, M\. Cao, X\. Cao, B\. Chen, F\. Chen, K\. Chen, L\. Du, D\. Feng, H\. Feng, M\. Gong,et al\.\(2026\)Llada2\. 1: speeding up text diffusion via token editing\.arXiv preprint arXiv:2602\.08676\.Cited by:[§1](https://arxiv.org/html/2605.24173#S1.p2.1)\.
- \[4\]T\. Bie, M\. Cao, K\. Chen, L\. Du, M\. Gong, Z\. Gong, Y\. Gu, J\. Hu, Z\. Huang, Z\. Lan,et al\.\(2025\)Llada2\. 0: scaling up diffusion language models to 100b\.arXiv preprint arXiv:2512\.15745\.Cited by:[§1](https://arxiv.org/html/2605.24173#S1.p2.1)\.
- \[5\]N\. Carlini, D\. Ippolito, M\. Jagielski, K\. Lee, F\. Tramer, and C\. Zhang\(2023\)Quantifying memorization across neural language models\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=TatRHT_1cK)Cited by:[§1](https://arxiv.org/html/2605.24173#S1.p1.1),[§2\.2](https://arxiv.org/html/2605.24173#S2.SS2.p1.7)\.
- \[6\]N\. Carlini, F\. Tramer, E\. Wallace, M\. Jagielski, A\. Herbert\-Voss, K\. Lee, A\. Roberts, T\. Brown, D\. Song, U\. Erlingsson,et al\.\(2021\)Extracting training data from large language models\.In30th USENIX security symposium \(USENIX Security 21\),pp\. 2633–2650\.Cited by:[§1](https://arxiv.org/html/2605.24173#S1.p1.1),[§2\.2](https://arxiv.org/html/2605.24173#S2.SS2.p1.7)\.
- \[7\]S\. Chang, J\. Yu, W\. Wang, Y\. Chen, J\. Yu, P\. Torr, and J\. Gu\(2026\)Tracedet: hallucination detection from the decoding trace of diffusion large language models\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=4puxTouUSV)Cited by:[§5](https://arxiv.org/html/2605.24173#S5.p2.1)\.
- \[8\]Y\. Chen, K\. Zhang, Y\. Du, E\. Stoppa, C\. Fleming, A\. Kundu, B\. Ribeiro, and N\. Li\(2026\)Membership inference attacks against fine\-tuned diffusion language models\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=oWKJursYpW)Cited by:[§A\.2](https://arxiv.org/html/2605.24173#A1.SS2.p1.5),[§2\.3](https://arxiv.org/html/2605.24173#S2.SS3.p1.1),[§2\.3](https://arxiv.org/html/2605.24173#S2.SS3.p2.1),[§4\.5](https://arxiv.org/html/2605.24173#S4.SS5.p5.1),[§5](https://arxiv.org/html/2605.24173#S5.p2.1)\.
- \[9\]J\. Deschenaux, L\. Tran, and C\. Gulcehre\(2026\)Partition generative modeling: masked modeling without masks\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=vEh1ceS154)Cited by:[§5](https://arxiv.org/html/2605.24173#S5.p2.1)\.
- \[10\]M\. Duan, A\. Suri, N\. Mireshghallah, S\. Min, W\. Shi, L\. Zettlemoyer, Y\. Tsvetkov, Y\. Choi, D\. Evans, and H\. Hajishirzi\(2024\)Do membership inference attacks work on large language models?\.InConference on Language Modeling \(COLM\),Cited by:[§4\.1](https://arxiv.org/html/2605.24173#S4.SS1.p1.6)\.
- \[11\]European Union\(2016\)General data protection regulation\.Note:Regulation \(EU\) 2016/679External Links:[Link](https://eur-lex.europa.eu/eli/reg/2016/679/oj)Cited by:[§1](https://arxiv.org/html/2605.24173#S1.p5.1),[§4\.3](https://arxiv.org/html/2605.24173#S4.SS3.p3.1)\.
- \[12\]Y\. Fujinuma and K\. Sakaguchi\(2026\)Unlocking prompt infilling capability for diffusion language models\.arXiv preprint arXiv:2604\.03677\.Cited by:[§2\.1](https://arxiv.org/html/2605.24173#S2.SS1.p2.3)\.
- \[13\]Gemini Diffusion\.Note:https://deepmind\.google/models/gemini\-diffusion/Cited by:[§1](https://arxiv.org/html/2605.24173#S1.p2.1)\.
- \[14\]S\. Gong, S\. Agarwal, Y\. Zhang, J\. Ye, L\. Zheng, M\. Li, C\. An, P\. Zhao, W\. Bi, J\. Han, H\. Peng, and L\. Kong\(2025\)Scaling diffusion language models via adaptation from autoregressive models\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=j1tSLYKwg8)Cited by:[§1](https://arxiv.org/html/2605.24173#S1.p2.1)\.
- \[15\]J\. Hayes, M\. Swanberg, H\. Chaudhari, I\. Yona, I\. Shumailov, M\. Nasr, C\. A\. Choquette\-Choo, K\. Lee, and A\. F\. Cooper\(2025\)Measuring memorization in language models via probabilistic extraction\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 9266–9291\.Cited by:[§1](https://arxiv.org/html/2605.24173#S1.p1.1),[§2\.2](https://arxiv.org/html/2605.24173#S2.SS2.p1.7),[§2\.3](https://arxiv.org/html/2605.24173#S2.SS3.p1.1)\.
- \[16\]Z\. He, T\. Sun, Q\. Tang, K\. Wang, X\. Huang, and X\. Qiu\(2023\)Diffusionbert: improving generative masked language models with diffusion models\.InProceedings of the 61st annual meeting of the association for computational linguistics \(volume 1: Long papers\),pp\. 4521–4534\.Cited by:[§1](https://arxiv.org/html/2605.24173#S1.p2.1)\.
- \[17\]E\. J\. Hu, yelong shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen\(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[§3\.3](https://arxiv.org/html/2605.24173#S3.SS3.p2.1)\.
- \[18\]D\. M\. Israel, G\. V\. den Broeck, and A\. Grover\(2026\)Accelerating diffusion LLMs via adaptive parallel decoding\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=xwqTt26NJf)Cited by:[§2\.1](https://arxiv.org/html/2605.24173#S2.SS1.p2.3),[§5](https://arxiv.org/html/2605.24173#S5.p2.1)\.
- \[19\]A\. Karamolegkou, J\. Li, L\. Zhou, and A\. Søgaard\(2023\)Copyright violations and large language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 7403–7412\.Cited by:[§1](https://arxiv.org/html/2605.24173#S1.p1.1)\.
- \[20\]B\. Klimt and Y\. Yang\(2004\)The enron corpus: a new dataset for email classification research\.InEuropean conference on machine learning,pp\. 217–226\.Cited by:[§4\.1](https://arxiv.org/html/2605.24173#S4.SS1.p1.6)\.
- \[21\]I\. Labs, S\. Khanna, S\. Kharbanda, S\. Li, H\. Varma, E\. Wang, S\. Birnbaum, Z\. Luo, Y\. Miraoui, A\. Palrecha,et al\.\(2025\)Mercury: ultra\-fast language models based on diffusion\.arXiv preprint arXiv:2506\.17298\.Cited by:[§1](https://arxiv.org/html/2605.24173#S1.p2.1)\.
- \[22\]N\. Lambert, J\. Morrison, V\. Pyatkin, S\. Huang, H\. Ivison, F\. Brahman, L\. J\. V\. Miranda, A\. Liu, N\. Dziri, S\. Lyu,et al\.\(2024\)Tulu 3: pushing frontiers in open language model post\-training\.arXiv preprint arXiv:2411\.15124\.Cited by:[§4\.1](https://arxiv.org/html/2605.24173#S4.SS1.p1.6)\.
- \[23\]K\. Lee, D\. Ippolito, A\. Nystrom, C\. Zhang, D\. Eck, C\. Callison\-Burch, and N\. Carlini\(2022\)Deduplicating training data makes language models better\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 8424–8445\.Cited by:[§1](https://arxiv.org/html/2605.24173#S1.p1.1)\.
- \[24\]G\. Li and C\. Cai\(2026\)Breaking AR’s sampling bottleneck: provable acceleration via diffusion language models\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=to1VYVar9W)Cited by:[§5](https://arxiv.org/html/2605.24173#S5.p2.1)\.
- \[25\]P\. Li, Y\. Zhou, D\. Muhtar, L\. Yin, S\. Yan, L\. Shen, Y\. Liang, S\. Vosoughi, and S\. Liu\(2026\)Diffusion language model knows the answer before it decodes\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=g88nt4ieTG)Cited by:[§5](https://arxiv.org/html/2605.24173#S5.p2.1)\.
- \[26\]T\. Li, M\. Chen, B\. Guo, and Z\. Shen\(2025\)A survey on diffusion language models\.arXiv preprint arXiv:2508\.10875\.Cited by:[§1](https://arxiv.org/html/2605.24173#S1.p2.1)\.
- \[27\]X\. Li, J\. Thickstun, I\. Gulrajani, P\. S\. Liang, and T\. B\. Hashimoto\(2022\)Diffusion\-lm improves controllable text generation\.Advances in neural information processing systems35,pp\. 4328–4343\.Cited by:[§1](https://arxiv.org/html/2605.24173#S1.p2.1)\.
- \[28\]N\. Lukas, A\. Salem, R\. Sim, S\. Tople, L\. Wutschitz, and S\. Zanella\-Béguelin\(2023\)Analyzing leakage of personally identifiable information in language models\.In2023 IEEE Symposium on Security and Privacy \(SP\),pp\. 346–363\.Cited by:[§1](https://arxiv.org/html/2605.24173#S1.p1.1)\.
- \[29\]X\. Luo, W\. Yu, Q\. Li, and J\. Bjerva\(2026\)Characterizing memorization in diffusion language models: generalized extraction and sampling effects\.arXiv preprint arXiv:2603\.02333\.Cited by:[§1](https://arxiv.org/html/2605.24173#S1.p3.1),[§1](https://arxiv.org/html/2605.24173#S1.p5.1),[§2\.3](https://arxiv.org/html/2605.24173#S2.SS3.p1.1),[§2\.3](https://arxiv.org/html/2605.24173#S2.SS3.p2.1),[§3\.4](https://arxiv.org/html/2605.24173#S3.SS4.p4.1),[§4\.4](https://arxiv.org/html/2605.24173#S4.SS4.p1.9)\.
- \[30\]O\. Mendels, C\. Peled, N\. Vaisman Levy, S\. Hart, T\. Rosenthal, L\. Lahiani,et al\.\(2018\)Microsoft Presidio: context aware, pluggable and customizable PII anonymization service for text and images\.Microsoft\.Note:[https://microsoft\.github\.io/presidio](https://microsoft.github.io/presidio)Cited by:[§4\.1](https://arxiv.org/html/2605.24173#S4.SS1.p3.1)\.
- \[31\]M\. Nasr, J\. Rando, N\. Carlini, J\. Hayase, M\. Jagielski, A\. F\. Cooper, D\. Ippolito, C\. A\. Choquette\-Choo, F\. Tramèr, and K\. Lee\(2025\)Scalable extraction of training data from aligned, production language models\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=vjel3nWP2a)Cited by:[§1](https://arxiv.org/html/2605.24173#S1.p1.1)\.
- \[32\]S\. Nie, F\. Zhu, Z\. You, X\. Zhang, J\. Ou, J\. Hu, J\. Zhou, Y\. Lin, J\. Wen, and C\. Li\(2026\)Large language diffusion models\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=KnqiC0znVF)Cited by:[§1](https://arxiv.org/html/2605.24173#S1.p2.1),[§1](https://arxiv.org/html/2605.24173#S1.p5.1),[§2\.1](https://arxiv.org/html/2605.24173#S2.SS1.p1.10),[§2\.1](https://arxiv.org/html/2605.24173#S2.SS1.p2.3),[§4\.1](https://arxiv.org/html/2605.24173#S4.SS1.p2.1)\.
- \[33\]Office for Civil Rights\(2012\)Guidance regarding methods for de\-identification of protected health information in accordance with the health insurance portability and accountability act \(hipaa\) privacy rule\.Technical reportU\.S\. Department of Health and Human Services\.External Links:[Link](https://www.hhs.gov/hipaa/for-professionals/privacy/special-topics/de-identification/)Cited by:[§1](https://arxiv.org/html/2605.24173#S1.p5.1),[§4\.3](https://arxiv.org/html/2605.24173#S4.SS3.p3.1)\.
- \[34\]Y\. Qian, Y\. Tan, Y\. Liu, W\. Yu, and S\. Pan\(2026\)DynHD: hallucination detection for diffusion large language models via denoising dynamics deviation learning\.arXiv preprint arXiv:2603\.16459\.Cited by:[§5](https://arxiv.org/html/2605.24173#S5.p2.1)\.
- \[35\]S\. Shaikh\(2025\)Balancing innovation and oversight: ai in the us treasury and irs: a survey\.arXiv preprint arXiv:2509\.16294\.Cited by:[§1](https://arxiv.org/html/2605.24173#S1.p5.1),[§4\.3](https://arxiv.org/html/2605.24173#S4.SS3.p3.1)\.
- \[36\]Y\. Song, Z\. Zhang, C\. Luo, P\. Gao, F\. Xia, H\. Luo, Z\. Li, Y\. Yang, H\. Yu, X\. Qu,et al\.\(2025\)Seed diffusion: a large\-scale diffusion language model with high\-speed inference\.arXiv preprint arXiv:2508\.02193\.Cited by:[§1](https://arxiv.org/html/2605.24173#S1.p2.1)\.
- \[37\]R\. Strudel, C\. Tallec, F\. Altché, Y\. Du, Y\. Ganin, A\. Mensch, W\. Grathwohl, N\. Savinov, S\. Dieleman, L\. Sifre,et al\.\(2022\)Self\-conditioned embedding diffusion for text generation\.arXiv preprint arXiv:2211\.04236\.Cited by:[§1](https://arxiv.org/html/2605.24173#S1.p2.1)\.
- \[38\]H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale,et al\.\(2023\)Llama 2: open foundation and fine\-tuned chat models\.arXiv preprint arXiv:2307\.09288\.Cited by:[§4\.1](https://arxiv.org/html/2605.24173#S4.SS1.p2.1)\.
- \[39\]U\.S\. Department of the Treasury\(2026\)Treasury and artificial intelligence\.Note:[https://home\.treasury\.gov/policy\-issues/financial\-markets\-financial\-institutions\-and\-fiscal\-service/treasury\-and\-artificial\-intelligence](https://home.treasury.gov/policy-issues/financial-markets-financial-institutions-and-fiscal-service/treasury-and-artificial-intelligence)Accessed: 2026\-05\-05Cited by:[§1](https://arxiv.org/html/2605.24173#S1.p5.1),[§4\.3](https://arxiv.org/html/2605.24173#S4.SS3.p3.1)\.
- \[40\]T\. Udsa, C\. Udomcharoenchaikit, P\. Payoungkhamdee, S\. Nutanong, and N\. Rattanavipanon\(2025\)Exploring cross\-client memorization of training data in large language models for federated learning\.arXiv preprint arXiv:2510\.08750\.Cited by:[§2\.2](https://arxiv.org/html/2605.24173#S2.SS2.p1.7)\.
- \[41\]Z\. Wen, J\. Qu, Z\. Chen, X\. Lu, D\. Liu, Z\. Liu, R\. Wu, Y\. Yang, X\. Jin, H\. Xu, X\. Liu, W\. Li, C\. Lu, J\. Shao, C\. He, and L\. Zhang\(2026\)The devil behind the mask: an emergent safety vulnerability of diffusion LLMs\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=rIPeatvPy3)Cited by:[§2\.1](https://arxiv.org/html/2605.24173#S2.SS1.p2.3)\.
- \[42\]C\. Wu, H\. Zhang, S\. Xue, Z\. Liu, S\. Diao, L\. Zhu, P\. Luo, S\. Han, and E\. Xie\(2026\)Fast\-dLLM: training\-free acceleration of diffusion LLM by enabling KV cache and parallel decoding\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=3Z3Is6hnOT)Cited by:[§2\.1](https://arxiv.org/html/2605.24173#S2.SS1.p2.3)\.
- \[43\]A\. Xiong, X\. Zhao, A\. Pappu, and D\. Song\(2025\)The landscape of memorization in llms: mechanisms, measurement, and mitigation\.arXiv preprint arXiv:2507\.05578\.Cited by:[§1](https://arxiv.org/html/2605.24173#S1.p2.1)\.
- \[44\]M\. Xu, T\. Geffner, K\. Kreis, W\. Nie, Y\. Xu, J\. Leskovec, S\. Ermon, and A\. Vahdat\(2025\)Energy\-based diffusion language models for text generation\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=sL2F9YCMXf)Cited by:[§1](https://arxiv.org/html/2605.24173#S1.p2.1)\.
- \[45\]J\. Ye, Z\. Xie, L\. Zheng, J\. Gao, Z\. Wu, X\. Jiang, Z\. Li, and L\. Kong\(2025\)Dream 7b: diffusion large language models\.arXiv preprint arXiv:2508\.15487\.Cited by:[§1](https://arxiv.org/html/2605.24173#S1.p2.1),[§1](https://arxiv.org/html/2605.24173#S1.p5.1),[§2\.1](https://arxiv.org/html/2605.24173#S2.SS1.p1.10),[§2\.1](https://arxiv.org/html/2605.24173#S2.SS1.p2.3),[§4\.1](https://arxiv.org/html/2605.24173#S4.SS1.p2.1)\.
- \[46\]W\. Yu, T\. Pang, Q\. Liu, C\. Du, B\. Kang, Y\. Huang, M\. Lin, and S\. Yan\(2023\)Bag of tricks for training data extraction from language models\.InInternational Conference on Machine Learning,pp\. 40306–40320\.Cited by:[§1](https://arxiv.org/html/2605.24173#S1.p1.1)\.
- \[47\]S\. Zeng, Y\. Li, J\. Ren, Y\. Liu, H\. Xu, P\. He, Y\. Xing, S\. Wang, J\. Tang, and D\. Yin\(2024\)Exploring memorization in fine\-tuned language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 3917–3948\.Cited by:[§2\.2](https://arxiv.org/html/2605.24173#S2.SS2.p1.7)\.
- \[48\]Y\. Zhang, F\. Xie, Z\. Zhou, Z\. Li, H\. Chen, K\. Wang, and Y\. Guo\(2025\)Jailbreaking large language diffusion models: revealing hidden safety flaws in diffusion\-based text generation\.InSocially Responsible and Trustworthy Foundation Models at NeurIPS 2025,External Links:[Link](https://openreview.net/forum?id=NWvL280s2l)Cited by:[§A\.1](https://arxiv.org/html/2605.24173#A1.SS1.p1.1),[§2\.1](https://arxiv.org/html/2605.24173#S2.SS1.p2.3),[§4\.1](https://arxiv.org/html/2605.24173#S4.SS1.p1.6)\.
- \[49\]S\. Zhao, D\. Gupta, Q\. Zheng, and A\. Grover\(2026\)D1: scaling reasoning in diffusion large language models via reinforcement learning\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=7ZVRlBFuEv)Cited by:[§5](https://arxiv.org/html/2605.24173#S5.p2.1)\.
- \[50\]Z\. Zhou, L\. Chen, H\. Tong, and D\. Song\(2026\)Dllm: simple diffusion language modeling\.arXiv preprint arXiv:2602\.22661\.Cited by:[§A\.2](https://arxiv.org/html/2605.24173#A1.SS2.p1.5),[§3\.3](https://arxiv.org/html/2605.24173#S3.SS3.p1.1),[§4\.1](https://arxiv.org/html/2605.24173#S4.SS1.p2.1)\.

## Appendix AExperimental Settings

### A\.1Datasets

Following prior work\[[48](https://arxiv.org/html/2605.24173#bib.bib1)\], we fine\-tune models on thefull pilesubset of the MIMIR dataset, which contains 10K sequences\. To evaluate verbatim extraction, we randomly sample 5,000 snippets of 100 tokens from the training set\. Across all models, snippets share the same starting position and span 100 tokens under each model’s tokenizer, though their character lengths may vary slightly\.

For PII extraction, we fine\-tune models on 10,000 samples from the Enron dataset and sample 100\-token snippets that contain PII entities in their masked portion\. For instance, in the prefix\-conditioned setting, snippets contain no PII in the first 50 tokens but include at least one PII entity in the last 50 tokens\. All models share the same starting position for each sampled snippet\. For each conditional mode, we construct 2,000 snippets for email address extraction and 450 snippets for phone number extraction\. Fortargetedinfilling, we randomly mix snippets from the other four modes and construct sets of the same size\.

For instruction tuning, we train on 10,000 samples from the Tulu3 dataset and sample 1,000 records in which both the prompt and response are under 100 tokens\.

### A\.2Model Training

Following\[[8](https://arxiv.org/html/2605.24173#bib.bib19),[50](https://arxiv.org/html/2605.24173#bib.bib16)\], we train each model for 5 epochs using the AdamW optimizer and a global batch size of 32\. On Enron and MIMIR Pile, we apply full fine\-tuning with a constant learning rate of5×10−55\\times 10^\{\-5\}and a weight decay of 0\.1\. For LoRA, we update an adapter of rank 64 using a learning rate of2×10−42\\times 10^\{\-4\}under a cosine schedule with5%5\\%warmup, and a weight decay of 0\.01\. On Tulu3, we perform instruction tuning with a peak learning rate of2×10−52\\times 10^\{\-5\}under a cosine schedule with10%10\\%warmup\.

### A\.3Computation Platform

All models are trained on a node containing 4 NVIDIA H100 80GB GPUs\. Evaluation of the infilling extraction is conducted on a separate node equipped with 4 NVIDIA L40S GPUs\.

## Appendix BAdditional Results

![Refer to caption](https://arxiv.org/html/2605.24173v1/x10.png)\(a\)Training perplexity
![Refer to caption](https://arxiv.org/html/2605.24173v1/x11.png)\(b\)Recall of email
![Refer to caption](https://arxiv.org/html/2605.24173v1/x12.png)\(c\)Precision of email

Figure 5:Extractability along the LLaDA\-8B fine\-tuning process on the Enron dataset\.### B\.1Induced Memorization during Fine\-tuning

[Figure˜5](https://arxiv.org/html/2605.24173#A2.F5)shows the joint evolution of training perplexity, recall, and precision over the course of fine\-tuning LLaDA\-8B on the Enron Email dataset\. The perplexity curve descends from around 9\.5 to 4\.5 during the five\-epoch fine\-tuning\. Before the fine\-tuning, both the recall and precision of four extraction modes are near zero\. For recall, theedgeincreases the most and ends at above6%6\\%while other modes also reach around2%2\\%\. For precision, the four modes achieve final metrics between18−35%18\-35\\%after fine\-tuning\.

### B\.2More Results of Decoding Parameters

Besides the discussion in[Figure˜2](https://arxiv.org/html/2605.24173#S4.F2), we provide additional results of Dream on Enron \([Figure˜6](https://arxiv.org/html/2605.24173#A2.F6)\) and those of LLaDA on MIMIR Pile \([Figure˜7](https://arxiv.org/html/2605.24173#A2.F7)\)\.

As Dream does not support semi\-autoregressive decoding, we investigate three decoding parameters on it: the number of decoding stepsTT, sampling temperatureτ\\tau, and remasking strategy\. We evaluate three strategies, including*entropy\-based*,*top\-kkmargin*, and*MaskGIT*\.[Figure˜6](https://arxiv.org/html/2605.24173#A2.F6)shows that higherTTincreases the extraction rate while higher temperature slightly compromises the recall, which stays consistent with previously observed patterns for LLaDA\.

[Figure˜7](https://arxiv.org/html/2605.24173#A2.F7)further confirms on MIMIR Pile that higherTT, higherbband lowerτ\\tauare preferred for higher extractability of LLaDA\.

![Refer to caption](https://arxiv.org/html/2605.24173v1/x13.png)\(a\)Number of stepsTT
![Refer to caption](https://arxiv.org/html/2605.24173v1/x14.png)\(b\)Temperatureτ\\tau
![Refer to caption](https://arxiv.org/html/2605.24173v1/x15.png)\(c\)Remasking strategy

Figure 6:Influence of decoding parameters for Dream\-7B\-FT model on extracting email addresses from Enron dataset\.![Refer to caption](https://arxiv.org/html/2605.24173v1/x16.png)\(a\)Number of stepsTT
![Refer to caption](https://arxiv.org/html/2605.24173v1/x17.png)\(b\)Block sizebb
![Refer to caption](https://arxiv.org/html/2605.24173v1/x18.png)\(c\)Temperatureτ\\tau
![Refer to caption](https://arxiv.org/html/2605.24173v1/x19.png)\(d\)Remasking strategy

Figure 7:Influence of decoding parameters for LLaDA\-8B\-FT model on verbatim extraction on MIMIR Pile dataset\.

## Appendix CBroader Impacts

DLMs offer higher generation efficiency while achieving generation quality comparable to ARMs, making them promising foundation models for future products\. In this paper, we propose a data extraction framework tailored to DLMs and evaluate the data\-leakage risks induced by infilling extraction\. We show that, in certain scenarios, the data\-leakage risks posed by DLMs have been underestimated\. In particular, when companies or government agencies train models on sensitive data but subsequently release \(e\.g\., for transparency or regulatory compliance\) redacted versions of the training data, targeted infilling can effectively recover the sensitive data memorized by the models\. Our findings shed light on a previously unexplored risk surface of DLMs\. As a potential mitigation, we further demonstrate the influence of decoding parameters, which model owners can tune to balance throughput, generation quality, and data\-leakage resistance\.