What to Forget in Unlearning? Forget Set Curation for Language Models

# What to Forget in Unlearning?Forget Set Curation for Language Models
Source: [https://arxiv.org/html/2608.14855](https://arxiv.org/html/2608.14855)
Arpandeep KhatuaStanford Universityakhatua@cs\.stanford\.eduYoussef AllouahStanford Universityyallouah@cs\.stanford\.eduSanmi KoyejoStanford Universitysanmi@cs\.stanford\.edu

###### Abstract

Machine unlearning aims to remove targeted data or behaviors from a trained model without retraining from scratch\. Yet most evaluations assume that the examples to forget are already known\. In realistic language\-model deployments, a requester may ask a model to stop reproducing a song or book without knowing which spans, documents, quotations, or near\-duplicates in a trillion\-token corpus support that behavior\. We study this missing upstream problem,*forget set curation*: mapping a suppression request to the data passed to an unlearning algorithm\. We introduceCleanSlate, a benchmark for verbatim output suppression over songs and books, with model\-specific extraction profiles, content\-grounded QA, and capability\-retention evaluations\.CleanSlateexposes two failure modes\. Natural lexical and exact\-substring curators often yield forget sets that lead to weak suppression\. An evaluation\-aware curator suppresses requested continuations almost completely, but causes collateral regression on non\-requested content and model\-dependent capability loss\. These results show that practical unlearning is not only an optimization problem once a forget set is given: the data chosen for forgetting determines both what can be unlearnt and what else is damaged\.

††footnotetext:∗Equal contribution\. Alphabetical order\.†Equal advising\. Alphabetical order\.### 1Introduction

Machine unlearning aims to remove targeted data or behaviors from a trained model without retraining from scratch\[[14](https://arxiv.org/html/2608.14855#bib.bib7),[36](https://arxiv.org/html/2608.14855#bib.bib6)\]\. For language models, this goal is increasingly relevant in privacy, safety, and copyright settings, where model owners may be asked to suppress particular outputs or remove the influence of specific data\. Yet most unlearning methods and benchmarks study the problem only after a crucial input has been supplied: the forget set, or the examples used to drive the update\[[24](https://arxiv.org/html/2608.14855#bib.bib11),[31](https://arxiv.org/html/2608.14855#bib.bib10),[18](https://arxiv.org/html/2608.14855#bib.bib26),[9](https://arxiv.org/html/2608.14855#bib.bib12)\]\. Real requests may not arrive in this form\. A requester may ask a model to stop reproducing a song, book, or other protected work, while the unlearning algorithm requires concrete spans or documents to optimize against\. This gap leaves a missing upstream problem: how should a suppression request be mapped to the data used for unlearning?

We study this missing step as*forget set curation*: selecting the intervention data passed to an unlearning algorithm from a suppression request\. We focus on verbatim output suppression for culturally embedded works such as songs and books\. This is narrower than concept unlearning: suppressing a work should not require erasing its author, plot, genre, or cultural context\. The desired behavior is selective: protected continuations should become difficult to elicit, while factual knowledge about the work and unrelated capabilities remain intact\. We operationalize this target using probabilistic extraction methods that test whether a model assigns high probability to an exact suffix conditioned on its prefix\[[15](https://arxiv.org/html/2608.14855#bib.bib8),[7](https://arxiv.org/html/2608.14855#bib.bib9)\]\. Thus, after unlearning, target continuations should become un\-extractable without collateral damage to the extractability of non\-requested content or degradation in content\-grounded QA and general capabilities\.

Songs and books expose why forget set curation is hard\. Although a work may have a canonical text, the evidence supporting a model’s continuation is rarely confined to that source\. As shown in Figure[1](https://arxiv.org/html/2608.14855#S1.F1), lyrics and passages can appear across training corpora through copies, quotations, reviews, fan forums, news articles, code snippets, synthetic examples, and incidental discussion\. The relevant object is therefore the work’s*corpus footprint*: the distributed set of spans and documents that may support the target continuation\. A curator must recover enough of this footprint to suppress the requested behavior, but not so much that it damages non\-requested content or general capabilities\. Exact matching helps reveal this footprint, but not fully since models can verbatim complete text even when exact n\-gram matches have been removed from training data\[[22](https://arxiv.org/html/2608.14855#bib.bib21)\]\.

![Refer to caption](https://arxiv.org/html/2608.14855v1/figures/coverage.png)Figure 1:Footprint of songs and books extends far beyond their canonical copy\.The top panel shows the canonical source forNever Gonna Give You Up, with full lyrics\. The surrounding panels show non\-canonical sources that echo the same text, including a Python file on GitHub that stores the lyrics, a Baltimore Sun news article that weaves the hook into reporting on a flash mob, a Stack Overflow answer that uses the lines as filler text, a LEO Weekly food review that drops the chorus into prose about a sandwich, and a pickup lines site that recasts the lyrics as relationship advice\.Contributions\.We conduct a study of*forget set curation*for language\-model unlearning: given a suppression request and a trained model, select the data that should be passed to an unlearning algorithm\. This framing separates the request\-to\-data problem from the downstream model\-update problem, and makes it possible to evaluate whether a proposed forget set actually induces selective suppression\. To study this setting, we introduceCleanSlate, a benchmark and evaluation protocol for forget set curation\.CleanSlatecontains 4,616 Billboard Hot 100 songs from 1970–2025 and 50 books, with per\-model extraction profiles for models spanning 7B–32B parameters\. We add content\-grounded QA pairs that distinguish suppression of verbatim reproduction from erasure of factual knowledge about the work\. The benchmark evaluates the full pipeline: a curator receives a suppression request, returns intervention data, a fixed unlearning algorithm updates the model, and the resulting model is evaluated for target suppression, collateral suppression, content\-QA retention, and general capability retention\. Our main findings are:

1. 1\.Cultural works have diffuse corpus footprints\.Songs and books are not represented only by canonical copies\. Older works spread outward through quotes, reviews, code, fan forums, news, and incidental references; newer works also inherit older language through idioms, genre templates, public\-domain quotations, and repeated phrases\. Thus, the evidence that can support a target continuation may predate the work itself or appear in sources that do not look like copies\.
2. 2\.Curator and unlearner interact strongly\.The same retrieval\-derived forget set can produce near\-zero or near\-complete target suppression depending on the unlearner and model; across the tested grid, strong suppression is always accompanied by substantial collateral suppression\.
3. 3\.Evaluation aware curation reveals a selectivity gap\.When we bypass retrieval and select the windows used for evaluation directly, unlearning can suppress requested continuations almost completely\. However, it also causes substantial model\-dependent capability regressions and suppresses non\-requested content\. Thus, even in verbatim output suppression, identifying the target text is not the same as constructing a clean forget set\.

Together, these results suggest that the forget set should be treated not as a premise of language\-model unlearning, but as one of its objects of evaluation\. What is selected for forgetting determines not only whether the requested behavior is suppressed, but also which neighboring abilities and general capabilities are disturbed\. We argue that practical unlearning should therefore be studied as a pipeline from request, to curated forget set, to edited model\.

### 2Related Work

##### Memorization and verbatim extraction\.

Language model memorization is commonly studied through extraction: prompting a model with a prefix and testing whether it reproduces a target suffix\[[5](https://arxiv.org/html/2608.14855#bib.bib13),[4](https://arxiv.org/html/2608.14855#bib.bib14),[1](https://arxiv.org/html/2608.14855#bib.bib5)\]\.\[[15](https://arxiv.org/html/2608.14855#bib.bib8)\]refine this into probabilistic discoverable extraction, measuring whether a target continuation can be produced under repeated sampling\.\[[7](https://arxiv.org/html/2608.14855#bib.bib9)\]apply this framework to copyrighted books, showing that extractability varies substantially across works and model families\. We use this completion\-style notion of memorization as a proxy for verbatim output suppression, the goal is not to infer training membership or erase all knowledge of a work, but to make requested continuations difficult to elicit while preserving nearby knowledge and unrelated capabilities\.

##### Machine unlearning benchmarks\.

Machine unlearning aims to remove the influence of specified data or behaviors from a trained model without retraining from scratch\[[14](https://arxiv.org/html/2608.14855#bib.bib7),[36](https://arxiv.org/html/2608.14855#bib.bib6)\]\. A range of methods have been proposed for language models, including loss ascent, preference\-based objectives, logit adjustment, and representation\-level interventions\[[37](https://arxiv.org/html/2608.14855#bib.bib25),[18](https://arxiv.org/html/2608.14855#bib.bib26),[11](https://arxiv.org/html/2608.14855#bib.bib16),[8](https://arxiv.org/html/2608.14855#bib.bib4)\]\. Existing benchmarks evaluate whether such updates suppress targeted information while preserving utility, including synthetic biographies in TOFU\[[24](https://arxiv.org/html/2608.14855#bib.bib11)\], multi\-axis evaluation in MUSE\[[31](https://arxiv.org/html/2608.14855#bib.bib10)\], hazardous\-knowledge unlearning in WMDP\[[18](https://arxiv.org/html/2608.14855#bib.bib26)\], and unified evaluation in OpenUnlearning\[[9](https://arxiv.org/html/2608.14855#bib.bib12)\]\. These benchmarks differ in domain and objective, but they typically provide the examples, entities, or evaluation targets to be forgotten\. In contrast, we study the upstream curation problem: given a suppression request for a work, what data should be selected for the unlearning update?

##### Data selection, retrieval, and attribution\.

Recent work shows that the contents of a forget set matter even after the forget data has been specified: small subsets or token\-level selections can substantially change the suppression–preservation tradeoff\[[26](https://arxiv.org/html/2608.14855#bib.bib23),[33](https://arxiv.org/html/2608.14855#bib.bib24),[38](https://arxiv.org/html/2608.14855#bib.bib22),[2](https://arxiv.org/html/2608.14855#bib.bib2),[23](https://arxiv.org/html/2608.14855#bib.bib37)\]\. This motivates studying selection itself, but prior work largely assumes that the unwanted examples or target domain are already known\. A natural approach to request\-level curation is to retrieve text overlapping with the target work, using lexical search or exact\-substring systems such as BM25, Infini\-gram, and Infini\-gram\-mini\[[20](https://arxiv.org/html/2608.14855#bib.bib20),[35](https://arxiv.org/html/2608.14855#bib.bib19)\]\. However, textual overlap is only an imperfect proxy for the data responsible for a model continuation: models can verbatim complete text even when exact n\-gram matches have been removed from training data\[[22](https://arxiv.org/html/2608.14855#bib.bib21)\]\. Influence functions and datamodeling offer a more causal view of training\-example responsibility\[[17](https://arxiv.org/html/2608.14855#bib.bib27),[16](https://arxiv.org/html/2608.14855#bib.bib30),[10](https://arxiv.org/html/2608.14855#bib.bib29),[13](https://arxiv.org/html/2608.14855#bib.bib28)\], but applying them to request\-level curation over trillion\-token corpora remains an open challenge\.

### 3Problem Statement: Forget Set Curation

Letθ\\thetabe a pretrained language model, and let𝒲\\mathcal\{W\}be a collection of works, such as songs or books\. A suppression request identifies a target subset𝒲f⊂𝒲\\mathcal\{W\}\_\{f\}\\subset\\mathcal\{W\}whose verbatim reproduction should be suppressed\. Any possible verbatim reproduction of works in𝒲r=𝒲∖𝒲f\\mathcal\{W\}\_\{r\}=\\mathcal\{W\}\\setminus\\mathcal\{W\}\_\{f\}should be preserved\.

##### Extractability\.

We quantify verbatim reproduction through probabilistic extraction\[[15](https://arxiv.org/html/2608.14855#bib.bib8),[7](https://arxiv.org/html/2608.14855#bib.bib9)\]\. A work is divided into a sequence of prefix\-suffix window pairs\(x,z\)\(x,z\)\. For a given prefixxx, the model assigns a probility

pz=pθ​\(z∣x\)=∏t=1\|z\|pθ​\(zt∣x,z<t\)p\_\{z\}\\;=\\;p\_\{\\theta\}\(z\\mid x\)\\;=\\;\\prod\_\{t=1\}^\{\|z\|\}p\_\{\\theta\}\(z\_\{t\}\\mid x,\\,z\_\{<t\}\)We call a window as*extractable*ifpz≥τp\_\{z\}\\geq\\tau, we useτ=0\.001\\tau=0\.001from\[[7](https://arxiv.org/html/2608.14855#bib.bib9)\]\.

##### Curated forget sets\.

Given a suppression request𝒲f\\mathcal\{W\}\_\{f\},target texts corresponding to𝒲f\\mathcal\{W\}\_\{f\}, a trained modelθ\\theta, and access to a large search corpus𝒞\\mathcal\{C\}, a curator𝒜\\mathcal\{A\}returns a forget setDfD\_\{f\}and, optionally, a retain setDrD\_\{r\}\. A downstream unlearning algorithm𝒰\\mathcal\{U\}then produces an unlearnt model

θ′=𝒰⁡\(θ,Df,Dr\)\.\\theta^\{\\prime\}=\\mathcal\{U\}\(\\theta,D\_\{f\},D\_\{r\}\)\.In this paper, the curator𝒜\\mathcal\{A\}is the object under evaluation: we compare different choices of𝒜\\mathcal\{A\}while holding the downstream unlearning procedure fixed unless otherwise stated\.

##### Search Corpora\.

For our analysis and experiments we use three distinct scale corpora,𝒞mid\\mathcal\{C\}\_\{\\mathrm\{mid\}\}theDolminomidtraining mix\[[25](https://arxiv.org/html/2608.14855#bib.bib18)\],𝒞pre\\mathcal\{C\}\_\{\\mathrm\{pre\}\}a subset of theDolma3pretraining mix\[[25](https://arxiv.org/html/2608.14855#bib.bib18)\], and𝒞CC25\\mathcal\{C\}\_\{\\mathrm\{CC25\}\}the Jan 2025 Common Crawl snapshot\. See Appendix[C](https://arxiv.org/html/2608.14855#A3)for details\.

##### Evaluation\.

The goal of verbatim output suppression is to make extractable windows from𝒲f\\mathcal\{W\}\_\{f\}un\-extractable after unlearning, without inducing the same effect on extractable windows from𝒲r\\mathcal\{W\}\_\{r\}\. Verbatim suppression should not erase knowledge about a work, we also evaluate content\-grounded question answering over the same works, together with general capability benchmarks\. Thus, a curator is judged not by textual relevance alone, but by the behavior of the unlearnt model it induces\.[Figure3](https://arxiv.org/html/2608.14855#S5.F3)captures theCleanSlatepipeline\. The central difficulty is that the documents in𝒞\\mathcal\{C\}that support a target continuation need not be canonical copies of the requested work\. We next show that songs and books often have broad corpus footprints, making the mapping from𝒲f\\mathcal\{W\}\_\{f\}toDfD\_\{f\}nontrivial\.

![Refer to caption](https://arxiv.org/html/2608.14855v1/figures/diffusion.png)Figure 2:Corpus footprints arise through both outward and inward diffusion\.\(Left\)Median per\-song coverage in midtraining corpus across their release years, 1970 to 2026\. Older works often have broad footprints from copies, quotations, discussion, and other web sources\. These are cases of outward diffusion\.\(Right\)Post\-cutoff markers show coverage for songs released after cutoff of the corpus\. We present five examples, each match older or unrelated sources, including instruction\-tuning data, a 1910 poem, forums, religious text, and reviews\. These cases illustrate inward diffusion: new works can inherit phrases, quotations, or stock language already present in the corpus\.

### 4The Corpus Footprint: Why Forget Set Curation Is Hard

The training evidence supporting a verbatim continuation need not be isolated to a canonical copy of the work\. It may appear in lyric aggregators, forum discussions, fan fiction, or code snippets\. A curator’s true target is therefore a work’s*corpus footprint*: the distributed set of documents and spans that can support the target continuation\.

##### Measuring literal overlap\.

To quantify the corpus footprint of a work, we measure the scale of exact word\-level overlaps between the target text and a corpora𝒞\\mathcal\{C\}using Infini\-gram\-mini\[[35](https://arxiv.org/html/2608.14855#bib.bib19)\]\. For a given work, we compute localizednn\-gram occurrence counts across every position \(see Appendix[D](https://arxiv.org/html/2608.14855#A4)for details\)\. This is a conservative measure as changes in formatting, whitespace, or even case can break a match\. It also misses paraphrases, translations, and semantic references, nor does it identify which documents actually caused a continuation\. Instead it, lower bounds the literal overlap that a curator would miss if it searched only for canonical copies\. At web scale, any large corpus will contain many short n\-grams from almost any English text even when no canonical copy is present\. So in addition we also measure the*coverage*: the fraction of the requested work covered byNN\-gram matches\. This tracking reveals two primary mechanisms of data diffusion, unpacked below\.

##### Outward diffusion\.

Works with cultural impact quickly spread after release\.Smile\(Morgan Wallen, released 31st Dec 2024\) illustrates how rapidly a footprint forms: in𝒞CC25\\mathcal\{C\}\_\{\\mathrm\{CC25\}\}\(Jan 2025\), it exhibits100%100\\%coverage at the 5\-gram level, with some spans having≥105\\geq 10^\{5\}occurrences, it also has100%100\\%coverage at the 50\-gram level with some spans having≥104\\geq 10^\{4\}matches \(see Figure[4](https://arxiv.org/html/2608.14855#A4.F4)\)\. Clearly𝒞CC25\\mathcal\{C\}\_\{\\mathrm\{CC25\}\}contains canonical copies ofSmile, but also a large cloud of shorter exact overlaps\. Older works have had decades to diffuse, they are copied, quoted, remixed, discussed, and embedded in unexpected sources\. Figure[1](https://arxiv.org/html/2608.14855#S1.F1)illustrates this forNever Gonna Give You Up: beyond canonical lyric pages, the same text appears in fan transcriptions, wiki pages, code files, review prose, and question\-answering sites\. In𝒞mid\\mathcal\{C\}\_\{\\mathrm\{mid\}\}, this work has94\.2%94\.2\\%coverage atn=5n=5and56\.4%56\.4\\%coverage atn=25n=25\(see Figure[5](https://arxiv.org/html/2608.14855#A4.F5)\), with matched spans distributed across≥147\\geq 147unique documents\. Any curator looking solely at canonical sources would miss the vast majority of these spans\.

##### Inward diffusion\.

A new work may contain language that already existed in the corpus before the work itself was released\. This can happen through idioms, genre templates, stock phrases, public\-domain quotations, or repeated cultural language\. Figure[2](https://arxiv.org/html/2608.14855#S3.F2)shows several post\-cutoff songs with substantial exact\-overlap coverage in corpora that predate their release\. For example,Red Terrorby The Weeknd has roughly 20% coverage atn=7n=7in𝒞mid\\mathcal\{C\}\_\{\\mathrm\{mid\}\}because it quotes a 1910 poem\. This shows that parts of the requested continuation may be supported by older or unrelated text\. This makes curation more subtle than finding noisy and approximate copies of the target work: the relevant evidence may predate the work or appear in documents that do not look like copies at all\.

##### Connection to target continuations\.

Corpus footprints help explain where verbatim output suppression becomes difficult\. Figures[5](https://arxiv.org/html/2608.14855#A4.F5)–[8](https://arxiv.org/html/2608.14855#A4.F8)overlay localized corpus occurrence counts with extraction probabilities across songs, poems, and books\. Across these case studies, extractable continuations often occur in regions with dense literal overlap, such as song choruses, famous quotations, or repeated phrases\. We treat this as a diagnostic rather than a causal claim: high overlap does not prove that a particular document caused a completion, and low overlap does not rule out elicitation through other mechanisms\. The practical implication is that a curator that misses dense regions of the footprint may leave the model enough data to be able to reproduce the requested continuation, while a curator that captures them too broadly may also affect neighboring content\.

### 5CleanSlate

![Refer to caption](https://arxiv.org/html/2608.14855v1/figures/Dataset.png)Figure 3:CleanSlate overview\.\(Top\)We assemble∼\\sim5K books and songs and measure baseline extraction profiles across different model families and generate QA pairs for each content\. Together these form the CleanSlate dataset\.\(Bottom\)Given a forget request𝒲f\\mathcal\{W\}\_\{f\}specifyingkkitems, target model parametersθ\\theta, and an arbitrary corpus𝒞\\mathcal\{C\}\(real, synthetic, or generated\), the objective is to create a curator𝒜\\mathcal\{A\}that outputs a forget set𝒟f\\mathcal\{D\}\_\{f\}and retain set𝒟r\\mathcal\{D\}\_\{r\}\. These are passed to a fixed unlearning procedure, and the resulting model is evaluated for low forget set extraction, preserved extraction on retained content, and stable general capability performance\.Building on the formalism of Section[3](https://arxiv.org/html/2608.14855#S3),CleanSlatepairs each suppression request with per\-model extractability evidence and content\-grounded QA, so that the full pipeline of curator, unlearning algorithm, and edited model can be scored along several axes\. Figure[3](https://arxiv.org/html/2608.14855#S5.F3)summarizes the construction and evaluation flow\.

##### Content domains\.

CleanSlatecovers two domains, songs and books, whose corpus footprints \(Section[4](https://arxiv.org/html/2608.14855#S4)\) are shaped differently\. Songs travel as short, repetitive, discrete units\. Books propagate as longer\-form passages reaching the corpus primarily through commentary and excerption rather than full copies\. A curator that recovers one shape may miss most of the other, making the two domains complementary stress tests\. We source songs from Billboard Hot 100 annual charts spanning 1970–2025, matched against the LRCLib lyrics database by title and artist, and 50 books, mostly from Project Gutenberg\[[27](https://arxiv.org/html/2608.14855#bib.bib15)\]with a small number of closed\-license works\.

##### Extraction profiles\.

We slide a window of 100 prefix and 100 suffix characters with stride 10 over each work, and label a work*model\-extractable*for a base modelθ\\thetaif at least5%5\\%of its windows are extractable in the sense of Section[3](https://arxiv.org/html/2608.14855#S3)\. The resulting profile is per\-model\. The same song may be extractable for one model and not for another\. See Appendix[E](https://arxiv.org/html/2608.14855#A5)for more details\.

##### Forget and retain pools\.

A suppression request is a model specific sample of size\|𝒲f\|=50\|\\mathcal\{W\}\_\{f\}\|=50drawn from the model\-extractable works ofθ\\theta, with𝒲r\\mathcal\{W\}\_\{r\}defined as in Section[3](https://arxiv.org/html/2608.14855#S3)\. For a fixed model we reuse the same𝒲f\\mathcal\{W\}\_\{f\}and𝒲r\\mathcal\{W\}\_\{r\}across curators so that differences are attributable to the curation\.

##### Content\-grounded QA\.

Suppressing verbatim reproduction and erasing factual knowledge are distinct goals\. A model that has been asked to stop reproducing a song’s lyrics should still be able to answer factual questions about its content\. We construct CleanSlate\-QA in two stages\. First, we prompt an LLM to extract*atomic statements*from each work; factual statements anchored to a named entity, place, number, or concrete event mentioned inside the text, with priors such as title, creator, year, and genre explicitly excluded\. Second, each statement is turned into a QA pair whose question embeds the title and creator naturally so it is self\-contained, and whose answer is a short 1\-5 word entity\. For example, in J\.K\. Rowling’s*Harry Potter and the Sorcerer’s Stone*we ask where the Dursleys make Harry sleep, with answer*cupboard under the stairs*\. The final dataset contains 12,088 QA pairs spanning the songs and books in𝒲\\mathcal\{W\}\.

##### End\-to\-end evaluation\.

For each curator𝒜\\mathcal\{A\}, we measure baseline extractability and QA onθ\\theta, run𝒜\\mathcal\{A\}to obtainDfD\_\{f\}, apply a fixed unlearning algorithm𝒰\\mathcal\{U\}to obtainθ′\\theta^\{\\prime\}, and re\-measure\.QAΔ\\Deltais the change in CleanSlate\-QA accuracy\. A validation suite covers math \(GSM8K\[[6](https://arxiv.org/html/2608.14855#bib.bib17)\]\), held\-out reasoning \(BBH\[[32](https://arxiv.org/html/2608.14855#bib.bib32)\]\), commonsense \(WinoGrande\[[30](https://arxiv.org/html/2608.14855#bib.bib33)\]\), reading comprehension \(CoQA\[[29](https://arxiv.org/html/2608.14855#bib.bib34)\]\), code \(HumanEval\+\[[21](https://arxiv.org/html/2608.14855#bib.bib35)\]\), and language modeling \(LAMBADA\[[28](https://arxiv.org/html/2608.14855#bib.bib36)\]\)\. Per\-model baselines are reported in Appendix[E](https://arxiv.org/html/2608.14855#A5)\.CleanSlatesupports two evaluation modes: request\-level evaluation lets curator and unlearner vary jointly and scores the composed request\-to\-data\-to\-update pipeline; algorithm\-focused evaluation holds two of⟨\\langlecurator, unlearner, model⟩\\ranglefixed and varies the third\.

### 6Experiments

We evaluate three curators usingCleanSlatewith\|𝒲f\|=50\|\\mathcal\{W\}\_\{f\}\|=50\. Table[1](https://arxiv.org/html/2608.14855#S6.T1)fixes the unlearner at SimNPO so row differences isolate the curator\. Table[2](https://arxiv.org/html/2608.14855#S6.T2)fixes𝒜\\mathcal\{A\}to the evaluation aware curator and varies the unlearner𝒰\\mathcal\{U\}, \(see Appendix[G](https://arxiv.org/html/2608.14855#A7)for hyperparameter details\)\. We perform experiments over six models,Llama\-3\.1\-8B,Olmo\-3\-7B,Nemotron\-9B,Qwen3\-8B,Gemma\-3\-12B, andOlmo\-3\-32B\. Per\-model pre\-unlearning evaluations are in Appendix Table[4](https://arxiv.org/html/2608.14855#A5.T4)\.

Table 1:Curation comparison at\|𝒲f\|=50\|\\mathcal\{W\}\_\{f\}\|=50with SimNPO unlearning, for three retrieval\-based curators \(BM25 pre, BM25 mid, Infini\-gram mid\) and the evaluation\-aware curator \(EA\)\.FFandRRare net percentage reductions in extractable\-window counts on the forget and retain pools; negative values denote net increases \(Section[5](https://arxiv.org/html/2608.14855#S5.SS0.SSS0.Px5)\)\. QA and validation columns report percentage\-point changes from the corresponding model baselines \(Table[4](https://arxiv.org/html/2608.14855#A5.T4)\);Avgaverages the six validation changes\.##### Curators\.

To evaluate whether standard corpus search can construct selective forget sets, we test two retrieval\-based curators against large training corpora, alongside an evaluation\-aware baseline:BM25\-preandBM25\-miduse BM25 indices over chunked documents from𝒞pre\\mathcal\{C\}\_\{\\mathrm\{pre\}\}and𝒞mid\\mathcal\{C\}\_\{\\mathrm\{mid\}\}, respectively\. For each requested work, the curator keeps the top ranked retrieval units and converts them into completion examples\.Infini\-gram\-miduses an Infini\-gram\-mini index over𝒞mid\\mathcal\{C\}\_\{\\mathrm\{mid\}\}to find maximal exact character spans of the requested text that occur in the corpus\. The matched spans are then localized in retrieved corpus contexts\. Finally, we test an Evaluation Aware\(EA\)curator which for each work in𝒲f\\mathcal\{W\}\_\{f\}, constructsDfD\_\{f\}to contain all the windows for that reference\-text\. EA presumes access to the evaluation windows and serves as a robustness test of the evaluation rather than a practical curator\. All curators generateDfD\_\{f\}in the same format: a 100\-character prefix followed by a 100\-character suffix\. For BM25, we slide this window over the retrieved BM25 text units recorded in the retrieval artifact\. For Infini\-gram, we emit windows from the retrieved corpus context whose 200\-character frame contains the midpoint of the exact match\. This fixes sequence length but not the number of examples: the natural curator outputs are not size matched\. We treat selected content and intervention volume as properties of the end\-to\-end curator output; with fixed epochs, output size also changes the number of optimizer updates\. Exact output sizes are reported in Appendix[K](https://arxiv.org/html/2608.14855#A11), and further retrieval details are in Appendix[H](https://arxiv.org/html/2608.14855#A8)\.

#### 6\.1Can natural retrieval curators induce selective suppression?

We find that retrieval engagement broadly tracks model size \(Table[1](https://arxiv.org/html/2608.14855#S6.T1)\)\. The two largest models \(Gemma\-3\-12B,Olmo\-3\-32B\) show the strongest forget\-side movement,Llama\-3\.1\-8BandNemotron\-9Bshow partial movement, and the smallest two \(Olmo\-3\-7B,Qwen3\-8B\) barely move\. Retain\-side movement follows the same ordering\. Selectivity is model\-determined rather than retriever\-determined\.Llama\-3\.1\-8Bloses between51\.2%51\.2\\%and55\.7%55\.7\\%of its retain\-pool extractability across the three retrieval methods, while its forget side gains at most28\.9%28\.9\\%\.Olmo\-3\-32Bis the only row where forget movement consistently exceeds retain movement\. Switching from𝒞mid\\mathcal\{C\}\_\{\\mathrm\{mid\}\}to𝒞pre\\mathcal\{C\}\_\{\\mathrm\{pre\}\}, or from BM25 to Infini\-gram, reorders rows but does not change which models engage\.

Table 2:Algorithm comparison at\|𝒲f\|=50\|\\mathcal\{W\}\_\{f\}\|=50with EA curation \(SimNPO, UNDIAL, RMU\)\. Metrics and columns follow Table[1](https://arxiv.org/html/2608.14855#S6.T1)\.Table 3:Curator×\\timesunlearner cross evaluation at\|𝒲f\|=50\|\\mathcal\{W\}\_\{f\}\|=50\. Each forget set is reused across unlearners, so only the unlearner varies within a curator block\. Entries averageLlama\-3\.1\-8B,Olmo\-3\-7B, andQwen3\-8B; per\-model results are in Appendix[J](https://arxiv.org/html/2608.14855#A10)and Table[2](https://arxiv.org/html/2608.14855#S6.T2)\. Metrics follow Table[1](https://arxiv.org/html/2608.14855#S6.T1); these three\-model means differ from its six\-model averages\.
#### 6\.2What if the target windows are given directly?

EA drives forget extractability to near\-100% on every model, including the small models \(Olmo\-3\-7B,Qwen3\-8B\) that retrieval barely moved \(Table[1](https://arxiv.org/html/2608.14855#S6.T1)\)\. This shows that the models can be moved by direct target\-window interventions under SimNPO; the crossed study below shows that retrieval outcomes cannot be attributed to the curator, model, or unlearner in isolation\. Retain\-pool extractability also falls on every row, consistent with Section[4](https://arxiv.org/html/2608.14855#S4), where the corpus support for any one work overlaps with that of many others\. Capability cost varies by an order of magnitude across the suite, with damage concentrated on BBH and LAMBADA\.

#### 6\.3Are evaluation aware curator’s effects algorithm\-specific?

To determine if the EA curator’s collateral damage is specific to SimNPO, we evaluate two additional unlearning algorithms \(UNDIAL and RMU\) onLlama\-3\.1\-8B,Olmo\-3\-7B, andQwen3\-8B\(Table[2](https://arxiv.org/html/2608.14855#S6.T2)\)\. Across SimNPO and UNDIAL,FFreaches100%100\\%on all three models, andRRstays between 72–100%\. While both algorithms suffer from severe collateral forgetting by degrading retain\-pool extractability, they produce very different capability outcomes on identical EA inputs\. UNDIAL largely preserves average validation accuracy \(\+0\.1\+0\.1pp\) while SimNPO degrades it \(averaging−5\.1\-5\.1pp\)\. RMU reducesFFonLlama\-3\.1\-8BandOlmo\-3\-7B, with retain extractability falling alongside\. OnQwen3\-8B, both forget and retain stall together atF=24\.5%F\{=\}24\.5\\%andR=25\.7%R\{=\}25\.7\\%, consistent with the hyperparameter sensitivity reported by[18](https://arxiv.org/html/2608.14855#bib.bib26)\. The same retain\-pool damage and capability gap persist at\|𝒲f\|=100\|\\mathcal\{W\}\_\{f\}\|=100onLlama\-3\.1\-8BandQwen3\-8B\. The full table is in Appendix[I](https://arxiv.org/html/2608.14855#A9)\. Table[3](https://arxiv.org/html/2608.14855#S6.T3)repeats this comparison for the three retrieval curators, reusing an identicalDfD\_\{f\}within each⟨\\langlemodel, curator⟩\\ranglepair so that only the unlearner changes\. Holding the curator fixed, the unlearner changes the outcome dramatically: the same BM25\-pre forget sets yield10\.5%10\.5\\%average forgetting under SimNPO,98\.2%98\.2\\%under UNDIAL, and32\.8%32\.8\\%under RMU, so weak suppression under SimNPO is not an intrinsic property of retrieval\-derived forget sets\. No configuration is selective, however: UNDIAL’s strong suppression comes with79\.079\.0–86\.5%86\.5\\%retain\-side suppression, per\-model results in Appendix[J](https://arxiv.org/html/2608.14855#A10)\.

##### Takeaway\.

Retrieval\-derived forget sets compose unpredictably with the downstream unlearner\. Fixed\-DfD\_\{f\}algorithm comparisons remain valid, but their conclusions are conditional on the upstream curator; request\-level comparisons evaluate the composed curator–unlearner pipeline\. The evaluation\-aware EA achieves forgetting on every model under SimNPO and UNDIAL, but the retain side falls and capability drops by up to10\.410\.4pp on average\. At the tested operating points, neither approach is selective\. Lexical retrieval underspecifies the corpus support of a work because target text is widely shared across documents that are not the canonical source\. Target\-window indexing overspecifies it because the extractable windows for one work might be correlated with those for others\. The same EA inputs produce a55\-pp capability gap between SimNPO and UNDIAL, reinforcing that selective unlearning requires co\-designing the curation strategy alongside the unlearning objective\.

### 7Discussion and Future Direction

##### Unlearning needs curation\.

Our results suggest that the forget set should not be treated as a fixed premise of language\-model unlearning\. In realistic deployments, the request is often stated at the level of a work or behavior, while the unlearning algorithm requires concrete data to update against\. What is selected for forgetting determines both whether the requested continuation becomes difficult to elicit and what else is disturbed\. Forget set curation is therefore part of the unlearning problem, not merely a preprocessing detail\.

##### Two insufficient endpoints\.

The experiments expose two natural but incomplete approaches to curation\. Off\-the\-shelf corpus retrieval is not selective: lexical and exact\-substring search can recover pieces of a work’s corpus footprint, but the resulting forget sets compose unpredictably with the downstream unlearner and do not induce selective verbatim suppression at the tested operating points\. Conversely, evaluation aware curation removes the retrieval bottleneck by selecting target windows directly, but still causes substantial output suppression of non\-requested continuations and model\-dependent capability regressions\. Thus, the problem is not simply to find more target\-like text\. It is to construct a forget set whose effects remain localized after the model update\.

##### Toward algorithm\-aware curation\.

This localization depends on the downstream unlearning algorithm\. The same evaluation aware forget set produces different capability profiles under SimNPO, UNDIAL, and RMU, suggesting that curation and unlearning should be evaluated jointly rather than as independent modules\. Future curators may need to combine corpus\-footprint signals with model\-specific extraction profiles, dense or hybrid retrieval, influence estimation, or datamodeling to predict which examples will suppress the requested behavior without unnecessary collateral effects\.CleanSlatestudies a deliberately narrow setting: verbatim output suppression for songs and books\. It does not address all forms of concept, entity, or factual unlearning, and our exact\-overlap diagnostics miss paraphrase, translation, and semantic reuse\. These limitations are also what make the task measurable\. Extending forget set curation to richer request types, broader corpora, and algorithm\-aware selection is a natural next step\. The broader message is that practical unlearning should be studied end\-to\-end, from request, to curated forget set, to edited model\.

### Acknowledgments

We thank Ludwig Schmidt, Etash Guha, Pratyush Maini, and Ananjan Nandi for helpful discussions\. We acknowledge compute support by the Center for AI Safety\. SK acknowledges support by NSF 2046795 and 2205329, IES R305C240046, ARPA\-H, the MacArthur Foundation, Schmidt Sciences, HAI, OpenAI, Microsoft, and Google\.

### References

- \[1\]A\. Ahmed, A\. F\. Cooper, S\. Koyejo, and P\. Liang\(2026\)Extracting books from production language models\.arXiv preprint arXiv:2601\.02671\.Cited by:[§2](https://arxiv.org/html/2608.14855#S2.SS0.SSS0.Px1.p1.1)\.
- \[2\]Y\. Allouah, R\. Guerraoui, and S\. Koyejo\(2026\)Distributional machine unlearning via selective data removal\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=IPqUBL4R9x)Cited by:[Appendix A](https://arxiv.org/html/2608.14855#A1.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2608.14855#S2.SS0.SSS0.Px3.p1.1)\.
- \[3\]P\. Cao, C\. Wang, Z\. He, H\. Yuan, J\. Li, Y\. Chen, K\. Liu, J\. Zhao,et al\.\(2024\)Rwku: benchmarking real\-world knowledge unlearning for large language models\.Advances in Neural Information Processing Systems37,pp\. 98213–98263\.Cited by:[Appendix A](https://arxiv.org/html/2608.14855#A1.SS0.SSS0.Px2.p1.1)\.
- \[4\]N\. Carlini, D\. Ippolito, M\. Jagielski, K\. Lee, F\. Tramer, and C\. Zhang\(2023\)Quantifying memorization across neural language models\.External Links:2202\.07646,[Link](https://arxiv.org/abs/2202.07646)Cited by:[§2](https://arxiv.org/html/2608.14855#S2.SS0.SSS0.Px1.p1.1)\.
- \[5\]N\. Carlini, F\. Tramer, E\. Wallace, M\. Jagielski, A\. Herbert\-Voss, K\. Lee, A\. Roberts, T\. Brown, D\. Song, U\. Erlingsson, A\. Oprea, and C\. Raffel\(2021\)Extracting training data from large language models\.External Links:2012\.07805,[Link](https://arxiv.org/abs/2012.07805)Cited by:[§2](https://arxiv.org/html/2608.14855#S2.SS0.SSS0.Px1.p1.1)\.
- \[6\]K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman\(2021\)Training verifiers to solve math word problems\.External Links:2110\.14168,[Link](https://arxiv.org/abs/2110.14168)Cited by:[§5](https://arxiv.org/html/2608.14855#S5.SS0.SSS0.Px5.p1.1)\.
- \[7\]A\. F\. Cooper, A\. Gokaslan, A\. Ahmed, A\. B\. Cyphert, C\. D\. Sa, M\. A\. Lemley, D\. E\. Ho, and P\. Liang\(2025\)Extracting memorized pieces of \(copyrighted\) books from open\-weight language models\.External Links:2505\.12546,[Link](https://arxiv.org/abs/2505.12546)Cited by:[§1](https://arxiv.org/html/2608.14855#S1.p2.1),[§2](https://arxiv.org/html/2608.14855#S2.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2608.14855#S3.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2608.14855#S3.SS0.SSS0.Px1.p1.2)\.
- \[8\]Y\. R\. Dong, H\. Lin, M\. Belkin, R\. Huerta, and I\. Vulić\(2025\)Undial: self\-distillation with adjusted logits for robust unlearning in large language models\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 8827–8840\.Cited by:[Appendix A](https://arxiv.org/html/2608.14855#A1.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2608.14855#S2.SS0.SSS0.Px2.p1.1)\.
- \[9\]V\. Dorna, A\. Mekala, W\. Zhao, A\. McCallum, Z\. C\. Lipton, J\. Z\. Kolter, and P\. Maini\(2025\)OpenUnlearning: accelerating llm unlearning via unified benchmarking of methods and metrics\.External Links:2506\.12618,[Link](https://arxiv.org/abs/2506.12618)Cited by:[Appendix A](https://arxiv.org/html/2608.14855#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2608.14855#S1.p1.1),[§2](https://arxiv.org/html/2608.14855#S2.SS0.SSS0.Px2.p1.1)\.
- \[10\]L\. Engstrom, A\. Feldmann, and A\. Madry\(2024\)Dsdm: model\-aware dataset selection with datamodels\.arXiv preprint arXiv:2401\.12926\.Cited by:[§2](https://arxiv.org/html/2608.14855#S2.SS0.SSS0.Px3.p1.1)\.
- \[11\]C\. Fan, J\. Liu, L\. Lin, J\. Jia, R\. Zhang, S\. Mei, and S\. Liu\(2025\)Simplicity prevails: rethinking negative preference optimization for llm unlearning\.External Links:2410\.07163,[Link](https://arxiv.org/abs/2410.07163)Cited by:[Appendix A](https://arxiv.org/html/2608.14855#A1.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2608.14855#S2.SS0.SSS0.Px2.p1.1)\.
- \[12\]R\. Gandikota, S\. Feucht, S\. Marks, and D\. Bau\(2024\)Erasing conceptual knowledge from language models\.arXiv preprint arXiv:2410\.02760\.Cited by:[Appendix H](https://arxiv.org/html/2608.14855#A8.SS0.SSS0.Px5.p1.1)\.
- \[13\]K\. Georgiev, R\. Rinberg, S\. M\. Park, S\. Garg, A\. Ilyas, A\. Madry, and S\. Neel\(2024\)Attribute\-to\-delete: machine unlearning via datamodel matching\.arXiv preprint arXiv:2410\.23232\.Cited by:[§2](https://arxiv.org/html/2608.14855#S2.SS0.SSS0.Px3.p1.1)\.
- \[14\]A\. Ginart, M\. Guan, G\. Valiant, and J\. Y\. Zou\(2019\)Making ai forget you: data deletion in machine learning\.Advances in neural information processing systems32\.Cited by:[§1](https://arxiv.org/html/2608.14855#S1.p1.1),[§2](https://arxiv.org/html/2608.14855#S2.SS0.SSS0.Px2.p1.1)\.
- \[15\]J\. Hayes, M\. Swanberg, H\. Chaudhari, I\. Yona, I\. Shumailov, M\. Nasr, C\. A\. Choquette\-Choo, K\. Lee, and A\. F\. Cooper\(2025\)Measuring memorization in language models via probabilistic extraction\.External Links:2410\.19482,[Link](https://arxiv.org/abs/2410.19482)Cited by:[§1](https://arxiv.org/html/2608.14855#S1.p2.1),[§2](https://arxiv.org/html/2608.14855#S2.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2608.14855#S3.SS0.SSS0.Px1.p1.1)\.
- \[16\]A\. Ilyas, S\. M\. Park, L\. Engstrom, G\. Leclerc, and A\. Madry\(2022\)Datamodels: predicting predictions from training data\.arXiv preprint arXiv:2202\.00622\.Cited by:[§2](https://arxiv.org/html/2608.14855#S2.SS0.SSS0.Px3.p1.1)\.
- \[17\]P\. W\. Koh and P\. Liang\(2017\)Understanding black\-box predictions via influence functions\.InInternational conference on machine learning,pp\. 1885–1894\.Cited by:[§2](https://arxiv.org/html/2608.14855#S2.SS0.SSS0.Px3.p1.1)\.
- \[18\]N\. Li, A\. Pan, A\. Gopal, S\. Yue, D\. Berrios, A\. Gatti, J\. D\. Li, A\. Dombrowski, S\. Goel, L\. Phan,et al\.\(2024\)The wmdp benchmark: measuring and reducing malicious use with unlearning\.arXiv preprint arXiv:2403\.03218\.Cited by:[Appendix A](https://arxiv.org/html/2608.14855#A1.SS0.SSS0.Px1.p1.1),[Appendix H](https://arxiv.org/html/2608.14855#A8.SS0.SSS0.Px5.p1.1),[§1](https://arxiv.org/html/2608.14855#S1.p1.1),[§2](https://arxiv.org/html/2608.14855#S2.SS0.SSS0.Px2.p1.1),[§6\.3](https://arxiv.org/html/2608.14855#S6.SS3.p1.1)\.
- \[19\]J\. Liu, T\. Blanton, Y\. Elazar, S\. Min, Y\. Chen, A\. Chheda\-Kothary, H\. Tran, B\. Bischoff, E\. Marsh, M\. Schmitz,et al\.\(2025\)OLMoTrace: tracing language model outputs back to trillions of training tokens\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 3: System Demonstrations\),pp\. 178–188\.Cited by:[Appendix H](https://arxiv.org/html/2608.14855#A8.SS0.SSS0.Px2.p1.1)\.
- \[20\]J\. Liu, S\. Min, L\. Zettlemoyer, Y\. Choi, and H\. Hajishirzi\(2024\)Infini\-gram: scaling unbounded n\-gram language models to a trillion tokens\.arXiv preprint arXiv:2401\.17377\.Cited by:[§2](https://arxiv.org/html/2608.14855#S2.SS0.SSS0.Px3.p1.1)\.
- \[21\]J\. Liu, C\. S\. Xia, Y\. Wang, and L\. Zhang\(2023\)Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation\.External Links:2305\.01210,[Link](https://arxiv.org/abs/2305.01210)Cited by:[§5](https://arxiv.org/html/2608.14855#S5.SS0.SSS0.Px5.p1.1)\.
- \[22\]K\. Z\. Liu, C\. A\. Choquette\-Choo, M\. Jagielski, P\. Kairouz, S\. Koyejo, P\. Liang, and N\. Papernot\(2025\)Language models may verbatim complete text they were not explicitly trained on\.External Links:2503\.17514,[Link](https://arxiv.org/abs/2503.17514)Cited by:[§1](https://arxiv.org/html/2608.14855#S1.p3.1),[§2](https://arxiv.org/html/2608.14855#S2.SS0.SSS0.Px3.p1.1)\.
- \[23\]Z\. Liu, H\. Lin, Y\. Ran, D\. Zhang, J\. Xie, C\. Li, W\. Zhao, and Z\. Xu\(2026\)Randomized antipodal search done right for data pareto improvement of llm unlearning\.arXiv preprint arXiv:2604\.16591\.Cited by:[§2](https://arxiv.org/html/2608.14855#S2.SS0.SSS0.Px3.p1.1)\.
- \[24\]P\. Maini, Z\. Feng, A\. Schwarzschild, Z\. C\. Lipton, and J\. Z\. Kolter\(2024\)TOFU: a task of fictitious unlearning for llms\.External Links:2401\.06121,[Link](https://arxiv.org/abs/2401.06121)Cited by:[Appendix A](https://arxiv.org/html/2608.14855#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2608.14855#S1.p1.1),[§2](https://arxiv.org/html/2608.14855#S2.SS0.SSS0.Px2.p1.1)\.
- \[25\]T\. Olmo, A\. Ettinger, A\. Bertsch, B\. Kuehl, D\. Graham, D\. Heineman, D\. Groeneveld, F\. Brahman, F\. Timbers, H\. Ivison,et al\.\(2025\)Olmo 3\.arXiv preprint arXiv:2512\.13961\.Cited by:[item 1](https://arxiv.org/html/2608.14855#A3.I1.i1.p1.1),[item 2](https://arxiv.org/html/2608.14855#A3.I1.i2.p1.1),[Appendix C](https://arxiv.org/html/2608.14855#A3.p1.2),[§3](https://arxiv.org/html/2608.14855#S3.SS0.SSS0.Px3.p1.1)\.
- \[26\]S\. Pal, C\. Wang, J\. Diffenderfer, B\. Kailkhura, and S\. Liu\(2025\)Llm unlearning reveals a stronger\-than\-expected coreset effect in current benchmarks\.arXiv preprint arXiv:2504\.10185\.Cited by:[Appendix A](https://arxiv.org/html/2608.14855#A1.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2608.14855#S2.SS0.SSS0.Px3.p1.1)\.
- \[27\]Project Gutenberg\(1971\)Project gutenberg\.Note:[https://www\.gutenberg\.org](https://www.gutenberg.org/)Accessed March 11, 2026Cited by:[§5](https://arxiv.org/html/2608.14855#S5.SS0.SSS0.Px1.p1.1)\.
- \[28\]A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, and I\. Sutskever\(2019\)Language models are unsupervised multitask learners\.Cited by:[§5](https://arxiv.org/html/2608.14855#S5.SS0.SSS0.Px5.p1.1)\.
- \[29\]S\. Reddy, D\. Chen, and C\. D\. Manning\(2019\)CoQA: a conversational question answering challenge\.External Links:1808\.07042,[Link](https://arxiv.org/abs/1808.07042)Cited by:[§5](https://arxiv.org/html/2608.14855#S5.SS0.SSS0.Px5.p1.1)\.
- \[30\]K\. Sakaguchi, R\. L\. Bras, C\. Bhagavatula, and Y\. Choi\(2019\)WinoGrande: an adversarial winograd schema challenge at scale\.External Links:1907\.10641,[Link](https://arxiv.org/abs/1907.10641)Cited by:[§5](https://arxiv.org/html/2608.14855#S5.SS0.SSS0.Px5.p1.1)\.
- \[31\]W\. Shi, J\. Lee, Y\. Huang, S\. Malladi, J\. Zhao, A\. Holtzman, D\. Liu, L\. Zettlemoyer, N\. A\. Smith, and C\. Zhang\(2024\)MUSE: machine unlearning six\-way evaluation for language models\.External Links:2407\.06460,[Link](https://arxiv.org/abs/2407.06460)Cited by:[Appendix A](https://arxiv.org/html/2608.14855#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2608.14855#S1.p1.1),[§2](https://arxiv.org/html/2608.14855#S2.SS0.SSS0.Px2.p1.1)\.
- \[32\]M\. Suzgun, N\. Scales, N\. Schärli, S\. Gehrmann, Y\. Tay, H\. W\. Chung, A\. Chowdhery, Q\. V\. Le, E\. H\. Chi, D\. Zhou, and J\. Wei\(2022\)Challenging big\-bench tasks and whether chain\-of\-thought can solve them\.External Links:2210\.09261,[Link](https://arxiv.org/abs/2210.09261)Cited by:[§5](https://arxiv.org/html/2608.14855#S5.SS0.SSS0.Px5.p1.1)\.
- \[33\]Y\. Wan, A\. Ramakrishna, K\. Chang, V\. Cevher, and R\. Gupta\(2025\)Not every token needs forgetting: selective unlearning to limit change in utility in large language model unlearning\.arXiv preprint arXiv:2506\.00876,pp\. 622–632\.Cited by:[Appendix A](https://arxiv.org/html/2608.14855#A1.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2608.14855#S2.SS0.SSS0.Px3.p1.1)\.
- \[34\]R\. Wu, C\. Yadav, R\. Salakhutdinov, and K\. Chaudhuri\(2024\)Evaluating deep unlearning in large language models\.arXiv preprint arXiv:2410\.15153\.Cited by:[Appendix A](https://arxiv.org/html/2608.14855#A1.SS0.SSS0.Px2.p1.1)\.
- \[35\]H\. Xu, J\. Liu, Y\. Choi, N\. A\. Smith, and H\. Hajishirzi\(2025\)INFINI\-gram mini: exact n\-gram search at the internet scale with fm\-index\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 24955–24980\.Cited by:[item 3](https://arxiv.org/html/2608.14855#A3.I1.i3.p1.1),[Appendix D](https://arxiv.org/html/2608.14855#A4.p1.1),[§2](https://arxiv.org/html/2608.14855#S2.SS0.SSS0.Px3.p1.1),[§4](https://arxiv.org/html/2608.14855#S4.SS0.SSS0.Px1.p1.1)\.
- \[36\]Y\. Yao and X\. Xu\(2024\)Large language model unlearning\.Advances in Neural Information Processing Systems37,pp\. 105425–105475\.Cited by:[§1](https://arxiv.org/html/2608.14855#S1.p1.1),[§2](https://arxiv.org/html/2608.14855#S2.SS0.SSS0.Px2.p1.1)\.
- \[37\]R\. Zhang, L\. Lin, Y\. Bai, and S\. Mei\(2024\)Negative preference optimization: from catastrophic collapse to effective unlearning\.arXiv preprint arXiv:2404\.05868\.Cited by:[Appendix A](https://arxiv.org/html/2608.14855#A1.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2608.14855#S2.SS0.SSS0.Px2.p1.1)\.
- \[38\]X\. Zhou, Y\. Qiang, S\. Z\. Zade, D\. Zytko, P\. Khanduri, and D\. Zhu\(2025\)Not all tokens are meant to be forgotten\.External Links:2506\.03142,[Link](https://arxiv.org/abs/2506.03142)Cited by:[Appendix A](https://arxiv.org/html/2608.14855#A1.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2608.14855#S2.SS0.SSS0.Px3.p1.1)\.
- \[39\]X\. Zhu, M\. Zhang, O\. Liu, R\. Jia, and W\. Neiswanger\(2025\)LLM unlearning without an expert curated dataset\.InSecond Conference on Language Modeling,Cited by:[Appendix A](https://arxiv.org/html/2608.14855#A1.SS0.SSS0.Px3.p1.1)\.

## Appendix

### Appendix AAdditional Related Work

##### Unlearning algorithms and benchmarks\.

A range of algorithms have been proposed for language\-model unlearning\. Gradient ascent on the forget loss is the simplest baseline\. Negative preference optimization and SimNPO\[[37](https://arxiv.org/html/2608.14855#bib.bib25),[11](https://arxiv.org/html/2608.14855#bib.bib16)\]cast unlearning as preference optimization against a reference model\. Representation Misdirection Unlearning\[[18](https://arxiv.org/html/2608.14855#bib.bib26)\]perturbs internal representations on forget data while regularizing retain representations, and UNDIAL\[[8](https://arxiv.org/html/2608.14855#bib.bib4)\]adjusts logits through self\-distillation\. Several benchmarks evaluate whether such methods can remove specified behaviors while preserving utility\. TOFU\[[24](https://arxiv.org/html/2608.14855#bib.bib11)\]provides synthetic author biographies, MUSE\[[31](https://arxiv.org/html/2608.14855#bib.bib10)\]evaluates multiple aspects of language\-model unlearning including verbatim memorization, WMDP\[[18](https://arxiv.org/html/2608.14855#bib.bib26)\]evaluates hazardous\-knowledge unlearning, and OpenUnlearning\[[9](https://arxiv.org/html/2608.14855#bib.bib12)\]provides a unified benchmarking framework\. These benchmarks differ in domain and objective, but they largely evaluate the downstream unlearning step after the forget data, target examples, or target behaviors have already been specified\.

##### Request\-level, entity\-level, and fact\-level unlearning\.

Some recent work moves closer to settings where the target is specified at a higher level than a fixed forget corpus\. RWKU\[[3](https://arxiv.org/html/2608.14855#bib.bib1)\]studies real\-world knowledge unlearning, where the algorithm receives a target entity and the original model rather than an explicit training corpus\. It then uses synthetic data produced by the model under evaluation to construct a forget set\. This is close in spirit to our setting because the forget set is not directly provided, but the evaluated artifact differs: RWKU primarily measures unlearning algorithms under synthetic forget sets, while we evaluate the curation of forget sets under fixed unlearning algorithms\. Deep fact unlearning\[[34](https://arxiv.org/html/2608.14855#bib.bib3)\]studies whether a target fact remains inferable from retained facts under logical rules\. This is related to our setting in that a target behavior can be supported by non\-target evidence, but the unit and objective are different: deep unlearning targets factual deductive closure, while we study verbatim continuation and ask which corpus spans or documents should be used for suppression\.

##### Data for unlearning\.

Recent work shows that the contents of a forget set matter even after the forget corpus has been specified\.\[[26](https://arxiv.org/html/2608.14855#bib.bib23)\]find that small subsets of benchmark\-provided forget sets can match full\-set unlearning\.\[[33](https://arxiv.org/html/2608.14855#bib.bib24)\]and\[[38](https://arxiv.org/html/2608.14855#bib.bib22)\]show that token\-level selection within known forget examples can reduce utility loss\.\[[2](https://arxiv.org/html/2608.14855#bib.bib2)\]formalize distributional unlearning as selective data removal: given identified unwanted and retained sample sets, choose a small subset whose removal moves the edited data distribution away from the unwanted domain while preserving the retained one\. These works are closely related to ours in emphasizing that selection matters, but they assume that the unwanted samples, retained samples, or target domain have already been obtained through an upstream process such as filtering, classification, or annotation\. We study that upstream process itself\.[39](https://arxiv.org/html/2608.14855#bib.bib39)identify the construction of forget sets as a major bottleneck in unlearning pipelines, and synthesize proxy forget data given only a broad domain name as input\. This is close to our setting because it addresses the upstream dataset construction problem, but differs in focus, they work on synthesizing proxy data for broad domain\-level forgetting, while we focus on measuring the role of the forget set \(and the distributed corpus footprint\) for suppressing verbatim output of specific targetted individual works\.

### Appendix BCompute Requirements

All experiments were conducted on a compute node equipped with 8 NVIDIA H200 GPUs \(141GB VRAM each\), 230 CPU cores, 3TB of RAM, and 60TB of local NVMe storage\. While model training utilized 1–2 GPUs, evaluation and validation tasks were performed on a single GPU\. Notably, the corpus search and forget set curation phases are significantly memory\- and storage\-bound due to the scale of the datasets involved; these stages necessitated the full utilization of the available system memory and high\-speed disk I/O\.

### Appendix CSearch Corpora \(𝒞\\mathcal\{C\}\) Details

For our experiments we use three distinct scale corpora

1. 1\.𝒞mid\\mathcal\{C\}\_\{\\mathrm\{mid\}\}\(Midtraining\):allenai/dolma3\_dolmino\_mix\-10B\-1025\[[25](https://arxiv.org/html/2608.14855#bib.bib18)\]this is a midtraining mix which has coverage of categories like instruction tuning data \(FLAN, Tulu\-3\-SFT\), code \(cranecode,stack\_edu\), and synthetic reasoning traces \(Gemini, QwQ, Llama Nemotron, OpenThoughts\)\.
2. 2\.𝒞pre\\mathcal\{C\}\_\{\\mathrm\{pre\}\}\(Pretraining\):A∼11%\\sim 11\\%subset of Dolma\-3 6T \(allenai/dolma3\_mix\-6T\) pretraining mix\[[25](https://arxiv.org/html/2608.14855#bib.bib18)\]\. To make it computationally feasible, we limit this subset to five Common Crawl shards of the full pretraining corpus,art\_and\_design,entertainment,history\_and\_geography,literature, andreligion\.
3. 3\.𝒞CC25\\mathcal\{C\}\_\{\\mathrm\{CC25\}\}\(Web Scale\):January 2025 Common Crawl snapshot with∼9​T\\sim 9Ttokens, accessed via the index provided by Infini\-gram\-mini\[[35](https://arxiv.org/html/2608.14855#bib.bib19)\]\. Used only for analysis of corpus footprint in[Section4](https://arxiv.org/html/2608.14855#S4)

[25](https://arxiv.org/html/2608.14855#bib.bib18)states December 2024 as the knowledge cutoff for𝒞mid\\mathcal\{C\}\_\{\\mathrm\{mid\}\}and𝒞pre\\mathcal\{C\}\_\{\\mathrm\{pre\}\}\.

### Appendix DComputing Coverage of Songs and Books in Large Corpus

We compute these statistics using Infini\-gram\-mini\[[35](https://arxiv.org/html/2608.14855#bib.bib19)\]but one can use any other dataset search tool\.

#### D\.1ComputingN−N\-Gram Matches and Coverage

To quantify the verbatim overlap between a reference sequence and a large\-scale corpus𝒞\\mathcal\{C\}, we employ an iterative retrieval algorithm that identifies the maximal exactnn\-gram matches starting at every word start position\. LetTw=\(x1,…,xm\)T\_\{w\}=\(x\_\{1\},\\ldots,x\_\{m\}\)be the whitespace\-delimited word sequence of workww\. For start positioniiand lengthnn, letsi,ns\_\{i,n\}be the exact character span coveringxi,…,xi\+n−1x\_\{i\},\\ldots,x\_\{i\+n\-1\}, preserving punctuation and spacing\. For indexed corpus𝒞\\mathcal\{C\}, we define the frequencyC𝒞​\(s\)C\_\{\\mathcal\{C\}\}\(s\)and the coverage statisticcov≥N​\(w,𝒞\)\\mathrm\{cov\}\_\{\\geq N\}\(w;\\mathcal\{C\}\)as follows:

C𝒞​\(s\)\\displaystyle C\_\{\\mathcal\{C\}\}\(s\)=∑d∈𝒞\#d​\(s\)\\displaystyle=\\sum\_\{d\\in\\mathcal\{C\}\}\\\#\_\{d\}\(s\)\(1\)cov≥N​\(w,𝒞\)\\displaystyle\\mathrm\{cov\}\_\{\\geq N\}\(w;\\mathcal\{C\}\)=1m∑j=1m\[∃i,n:n≥N,i≤j<i\+n,C𝒞\(si,n\)\>0\]\\displaystyle=\\frac\{1\}\{m\}\\sum\_\{j=1\}^\{m\}\\mathbf\{1\}\\\!\\left\[\\,\\exists\\,i,n:\\ n\\geq N,\\ i\\leq j<i\+n,\\ C\_\{\\mathcal\{C\}\}\(s\_\{i,n\}\)\>0\\,\\right\]The statisticcov≥N\\mathrm\{cov\}\_\{\\geq N\}measures the fraction of word positions inwwcovered by at least one verbatimnn\-gram of lengthn≥Nn\\geq Npresent in𝒞\\mathcal\{C\}\. For each start positionii, we begin atnmin=5n\_\{\\min\}=5and extendnnwhileC𝒞​\(si,n\)\>0C\_\{\\mathcal\{C\}\}\(s\_\{i,n\}\)\>0\. Shortnn\-grams \(n<5n<5\) occur with high background frequency due to linguistic coincidence\.

#### D\.2Aggregate Coverage Statistics

Out of4,6634\{,\}663works inCleanSlate,4,5964\{,\}596\(98\.6%98\.6\\%\) retrieve at least one positive\-count 5\-gram match with documents in𝒞mid\\mathcal\{C\}\_\{\\mathrm\{mid\}\}\. The distribution of retrieved documents per work exhibits a median of119119, a9090th percentile of299299, and a maximum of45,33845\{,\}338\. For source inspection in𝒞mid\\mathcal\{C\}\_\{\\mathrm\{mid\}\}we sampleD=20D=20documents for each of the top\-K\(K=20000\)\(K=20000\)spans ranked by length and occurrence count, for𝒞CC25\\mathcal\{C\}\_\{\\mathrm\{CC25\}\}we sampleD=2D=2documents for each of the topK=2000K=2000spans, due to computational limitations\.

![Refer to caption](https://arxiv.org/html/2608.14855v1/figures/smile_morgan_wallen.png)Figure 4:N\-gram coverage for*Smile*in𝒞CC25\\mathcal\{C\}\_\{\\mathrm\{CC25\}\}
#### D\.3Extractability vs\. Footprint Density Plots

To understand the correlation between localized corpus prevalence and model extractability, we plot the*footprint density*at a given word positionjj, alongside the maximum extraction probability \(pzp\_\{z\}\) measured amongst all suffixeszzcovering the positionjj\. The density aggregates the occurrence counts of all validnn\-grams in𝒞mid\\mathcal\{C\}\_\{\\mathrm\{mid\}\}that overlap thejj\-th word of the work\.

Figures[5](https://arxiv.org/html/2608.14855#A4.F5)through[8](https://arxiv.org/html/2608.14855#A4.F8)corroborate that peaks in𝒞mid\\mathcal\{C\}\_\{\\mathrm\{mid\}\}occurrences strongly align with spikes in extractability\. For instance, the choruses of popular songs \(*Never Gonna Give You Up*,*Rocket Man*\) and famous refrains in poems \(*A Dream Within a Dream*\) exhibit massive frequency spikes in the corpus\. The model’s extraction probability neatly mirrors these spikes, rising precisely where the footprint density is highest\.

![Refer to caption](https://arxiv.org/html/2608.14855v1/figures/nggyu_song_mem_span_coverage.png)Figure 5:Footprint density \(sum of matches of all overlappingnn\-gram matches in𝒞mid\\mathcal\{C\}\_\{\\mathrm\{mid\}\}\) vs\. maximum extraction probability \(pzp\_\{z\}\) per word for*Never Gonna Give You Up*\.![Refer to caption](https://arxiv.org/html/2608.14855v1/figures/rocket_man_song_mem_span_coverage.png)Figure 6:Footprint density vs\. maximum extraction probability \(pzp\_\{z\}\) per position for*Rocket Man*\.![Refer to caption](https://arxiv.org/html/2608.14855v1/figures/edgar_allen_poe_mem_coverage.png)Figure 7:Footprint density in𝒞mid\\mathcal\{C\}\_\{\\mathrm\{mid\}\}vs\. maximum extraction probability \(pzp\_\{z\}\) per position for Poesms of Edgar Allen Poe![Refer to caption](https://arxiv.org/html/2608.14855v1/figures/communist_manifesto_span_mem_coverage.png)Figure 8:Footprint density in𝒞mid\\mathcal\{C\}\_\{\\mathrm\{mid\}\}vs\. maxpzp\_\{z\}per position for*The Communist Manifesto*\.

### Appendix EBaseline Extractability Patterns Across Model Families

Extractability is neither uniform across target works nor consistent across model architectures\. Table[4](https://arxiv.org/html/2608.14855#A5.T4)documents the baseline characteristics of our evaluation suite prior to any curation or unlearning\. The extraction threshold \(pz≥0\.001p\_\{z\}\\geq 0\.001\) is evaluated over sliding windows of100100prefix and100100suffix characters\.

##### Model scale and capability\.

Larger models consistently exhibit higher raw extractability\. For instance, the 32B\-parameterOlmo\-3\-32Bachieves a forget\-pool extraction rate \(Ext\-F\) of 8\.27%, while its 7B\-parameter counterpart \(Olmo\-3\-7B\) achieves 7\.20%\. Base models also tend to exhibit higher extractability than their instruction\-tuned variants, probably due to alignment training penalizing raw regurgitation in favor of conversational formatting\. Repetitive structure of certain songs also increase their extractability,Driveby The Weeknd \(2025\) post dates Llama\-3\.1\-8B and Olmo\-32b but both model assign highpzp\_\{z\}where a sections of the work are near identical prefix–suffix pair \(see Figure[9](https://arxiv.org/html/2608.14855#A5.F9)\), this suggests that the language\-modeling objective by itself can increase the probability of extraction for highly repetitive works\.

![Refer to caption](https://arxiv.org/html/2608.14855v1/figures/drive_weekend.png)Figure 9:max⁡pz\\max p\_\{z\}acrossDriveby Weekend, both models pre date the song, windows with highpzp\_\{z\}are highly repetitive portions of the chorus
##### Heterogeneity of extractable text\.

The data reveals that extractability is highly localized within the texts themselves\. For songs, verses often fall below the extraction threshold, while choruses reinforced by both their internal repetition and their higher external footprint density cross the threshold easily\. More importantly,*different models extract different windows*\. A specific verse that is highly extractable forLlama\-3\.1\-8Bmay fall below the threshold forNemotron\-9Bdue to differences in their respective training mixtures\.

##### QA performance vs\. verbatim extraction\.

Table[4](https://arxiv.org/html/2608.14855#A5.T4)also reports the baselineCleanSlate\-QA accuracy \(ranging from 22\.01% to 35\.07% pass@5\)\. The fact that these models can reliably answer factual questions about the content confirms that they possess abstract knowledge of the works\. The core challenge of forget set curation is to selectively suppress the localizedpzp\_\{z\}spikes responsible for exact extraction while leaving this broader semantic knowledge \(QA\) and general capabilities \(GSM8K, etc\.\) undisturbed\.

Table 4:Baseline evaluation models prior to any curation or unlearning, ordered by forget\-pool extractability\. TheForget poolandRetain poolblocks report the total number of\(100,100\)\(100,100\)character windows \(NN\) and the number that meet the extraction thresholdpz≥0\.001p\_\{z\}\\geq 0\.001\(Ext\.\)\.QAis CleanSlate\-QA pass@5 accuracy \(%\) averaged over forget and retain items\. Remaining columns report baseline accuracy \(%\) on the validation suite\. Forget pool size is\|ℱ\|=50\|\\mathcal\{F\}\|\{=\}50except for Olmo\-3\-7B, which has\|ℱ\|=38\|\\mathcal\{F\}\|\{=\}38\. GSM8K uses flexible\-extract scoring\.

### Appendix FCleanSlate\-QA Benchmark Construction

Here we go over the pipeline used to build CleanSlate\-QA, the content\-grounded retain\-metric benchmark referenced in Section[5](https://arxiv.org/html/2608.14855#S5.SS0.SSS0.Px4)and reported as theQAcolumn of Table[4](https://arxiv.org/html/2608.14855#A5.T4)\. Construction has three stages\. Stage 1 extracts factual propositions from each work, stage 2 turns each proposition into an atomic question\-answer pair, and stage 3 filters candidates by out\-of\-sample \(OOS\) model knowledge\.

##### Routing and infrastructure\.

Each work in𝒲\\mathcal\{W\}is routed by length\. Books are chunked into60,00060\{,\}000\-character windows that are extracted while songs are extracted in a single call\. Stages 1 and 2 dispatchgemini\-3\.1\-flash\-lite\-preview\. Stage 3 uses \(Qwen/Qwen3\.5\-9B,meta\-llama/Meta\-Llama\-3\-8B\-Instruct\-Lite\)\.

#### F\.1Stage 1: Proposition Extraction

The model is asked to extract a small set of*interior factual propositions*, defined as statements anchored to a named entity, place, number, or concrete event mentioned inside the text, with priors such as title, creator, year, and genre explicitly excluded\. Each proposition must cite a short source span from the work\. The song and book prompts are reproduced below\.

Song Proposition Extraction PromptYou are extracting interior facts from a song’s lyrics that will be used to build a retain\-metric QA benchmark\.Title: “`\{title\}`” Creator:`\{creator\}`LYRICS: `\{text\}`TASK:Extract between 3 and 8interior factual propositionsfrom these lyrics\.A good proposition:•Refers to a named entity, specific detail, number, place, person, action, or relationship mentioned INSIDE the lyrics\.•Is a full factual statement, not just a word\.•Cites a short source span \(1–2 lines from the lyrics\) where the fact appears\.A bad proposition:•Is about the title, creator/artist, year, genre \(priors\)\.•Restates the hook, chorus, or title in different words \(e\.g\. for “Ladies’ Night”: “the song says it is your night” — that’s just the hook\)\.•Has no named entity, place, number, or specific concrete detail\.•Is a generic feeling, mood, theme, or exhortation \(“the song tells listeners to dance”\)\.•Uses pronouns/possessives as its key content \(“your”, “their”, “this”\)\.If the song is abstract with few concrete details, return fewer propositions\. Do NOT pad\.Return ONLY a JSON object: \{"propositions": \[\{"fact": "\.\.\.", "source\_span": "\.\.\."\}, \.\.\.\]\}

Book Proposition Extraction Prompt \(per chunk\)You are extracting interior facts from a CHUNK of a book that will be used to build a retain\-metric QA benchmark\.Book: “`\{title\}`” Author:`\{creator\}` Chunk`\{chunk\_idx\}`of`\{total\_chunks\}`TEXT: `\{text\}`TASK:Extract between 5 and 15interior factual propositionsfrom this chunk\.Prioritize:•Named characters and their distinguishing features, relationships, actions\.•Specific locations, settings, objects mentioned\.•Concrete plot events in this chunk\.•Numeric specifics \(ages, dates within the narrative, counts\)\.Avoid:•Anything about the title, author, publication year, genre \(priors\)\.•Generic theme/mood statements without a concrete anchor\.•Verbatim famous quotes unless they reveal a concrete fact\.Each proposition must cite a short source span \(1–3 lines\) from the chunk\.Return ONLY a JSON object: \{"propositions": \[\{"fact": "\.\.\.", "source\_span": "\.\.\."\}, \.\.\.\]\}

#### F\.2Stage 2: Atomic QA Generation

Each proposition is converted into one atomic question\-answer pair\. Questions must be self\-contained \(naming both title and creator\), target an interior detail rather than a prior, and admit a short 1–5 word answer that is itself a named entity, number, specific place, specific object, or proper noun\. The model is allowed to return an empty list when no valid pair can be produced for a given proposition\. This is the primary mechanism by which low\-yield propositions are dropped\.

Atomic QA Generation PromptTurn this fact into ONE atomic question–answer pair for a retain\-metric benchmark\.Title: “`\{title\}`” Creator:`\{creator\}` Fact:`\{fact\}` Source span:`\{source\_span\}`REQUIREMENTS— the question must:1\.Be SELF\-CONTAINED \(atomic\): name the title AND creator naturally\. The model sees only the question, no context\.2\.Have a SHORT semantic answer \(1–5 words\) that is anamed entity, number, specific place, specific object, or proper noun\. Not a pronoun, possessive, or generic colloquial phrase\.3\.Target an interior detail — NOT the title, creator, year, or genre\.4\.The answer must NOT be the title or any substring of the title/creator\.5\.The answer must NOT appear as a phrase inside your own question text\.6\.Not be answerable from the title alone \(e\.g\., if the song is “Sexy \+ 17” don’t ask the girl’s age\)\.GOOD examples:•Q: “In Dr\. Hook’s ‘Sylvia’s Mother’, how does the caller address Sylvia’s mother?” A: “Mrs\. Avery”•Q: “In ‘C’mon N’ Ride It’ by Quad City DJ’s, what car does the narrator want to be in the back of?” A: “Impala”BAD examples:•Q: “Who sang ‘Hotline Bling’?”→\\rightarrowpriors•Q: “Complete: ‘Ride that choo\-choo \_\_\_’ ”→\\rightarrowverbatim•Q: “What is the mood of ‘More Than A Woman’?”→\\rightarrowsubjective•Q: “In Kool & the Gang’s ‘Ladies’ Night’, what does the song say is happening tonight?” A: “your night”→\\rightarrowhook\-echo, pronoun answer, no specific contentIf the fact cannot produce a good atomic Q meeting all requirements, return an empty list\.Return ONLY a JSON object shaped like: \{"pairs": \[\{"question": "\.\.\.", "answer": "\.\.\."\}, \.\.\.\]\}

#### F\.3Stage 3: OOS Knowledge Filter and Judge

To drop questions whose answers can only be recovered by memorizing the source work, each candidate question is presented to the OOS probe models with no surrounding context\. A candidate is kept iff at least one probe model produces a correct response\.

LLM Judge PromptDoes the model answer contain the reference answer or convey the same meaning?Reference:`\{reference\}` Model Answer:`\{model\_answer\}`Reply with ONLY this JSON, nothing else:\{"correct": true\}or\{"correct": false\}

### Appendix GUnlearning Training Details

The trainer consumes curated forget rows of the form\(x,z\)\(x,z\), wherexxis a prefix andzzis the suffix to suppress\. Each training item is anchored on one forget example and paired with a randomly sampled retain example\. The total loss is

λf​ℒforget\+λr​ℒretain\.\\lambda\_\{f\}\\mathcal\{L\}\_\{\\mathrm\{forget\}\}\+\\lambda\_\{r\}\\mathcal\{L\}\_\{\\mathrm\{retain\}\}\.For SimNPO we use the average suffix NLL form

ℒSimNPO=−2β​log⁡σ⁡\(β⁡\(ℓ¯θ​\(z∣x\)−δ\)\),\\mathcal\{L\}\_\{\\mathrm\{SimNPO\}\}=\-\\frac\{2\}\{\\beta\}\\log\\sigma\\\!\\left\(\\beta\(\\bar\{\\ell\}\_\{\\theta\}\(z\\mid x\)\-\\delta\)\\right\),withℒretain\\mathcal\{L\}\_\{\\mathrm\{retain\}\}equal to retain NLL\. UNDIAL distills from a frozen reference model after subtractingβU\\beta\_\{\\mathrm\{U\}\}from the gold\-token teacher logit on forget suffix tokens\. RMU minimizes MSE between forget activations and a random control vector at decoder blockmodel\.layers\.7, and uses an activation matching retain loss against the frozen reference model\. Table[5](https://arxiv.org/html/2608.14855#A7.T5)captures all hyperparameters specific to𝒰\\mathcal\{U\}\.

Table 5:Unlearning hyperparameters\.#### G\.1Some example QA pairs

See Figure[10](https://arxiv.org/html/2608.14855#A7.F10)

![Refer to caption](https://arxiv.org/html/2608.14855v1/figures/QA.png)Figure 10:Example QA pairs formCleanSlate

### Appendix HRetrieval Curators and Forget\-Set Construction Details

Each retrieval curator maps a request text to a set of prefix–suffix training examples in three steps: retrieve corpus units, project retrieved text into evaluation\-shaped windows, and allocate a bounded number of windows per requested work\. The unlearning trainer consumes only these pre\-sliced rows: it tokenizes the concatenated prefix and suffix and masks prefix tokens from the forget loss\.

##### BM25 retrieval\.

For BM25, we build sharded BM25s indices over the search corpus\. Documents shorter than 100 characters are discarded\. Remaining documents are split at word boundaries into segments of at most 2,000 characters with 500\-character overlap; segments shorter than 200 characters are removed\. BM25 tokenization lowercases text and removes English stopwords\. At search time, each shard returns its top candidates and we retain the global top 100 segments for the requested work\. Very long request texts are queried through sampled probes, after which the same global top\-kkmerge is applied\.

##### Infini\-gram retrieval\.

We use an Infini\-gram\-mini over the corpus\. For each character offsetiiin the request textTT, we first test whether the 20\-character substring beginning atiioccurs in the corpus\. If it does, similar to OlmoTrace\[[19](https://arxiv.org/html/2608.14855#bib.bib31)\]we binary search for the longest substringT\[i:j\]T\[i:j\]with positive corpus count and record the span together with its count\. After scanning all offsets, we rank the spans by character length, and keep the top 100\. For each retained span, the retrieval pipeline queries corpus contexts containing the span; the curation artifact stores the first retrieved context, while metadata records the matchedspan\_text, corpus count, coverage, and number of retrieved contexts\.

##### Window extraction\.

We use the same character window shape for all forget sets\. LetW=100W=100andS=10S=10\. A window beginning at character positionssemits

prefix=D\[s:s\+W\],suffix=D\[s\+W:s\+2W\]\.\\mathrm\{prefix\}=D\[s:s\+W\],\\qquad\\mathrm\{suffix\}=D\[s\+W:s\+2W\]\.EA curator applies this sliding window directly to the reference text\. BM25 applies it to the retrieved BM25 text unit recorded in the retrieval artifact\. Infini\-gram based curator first locatesspan\_textinside the retrieved corpus context and emits stride\-spaced windows whose 200\-character frame contains the midpoint of the matched span\. Duplicate prefix\- suffix pairs are removed within each requested work\.

##### Quota allocation\.

Retrieval can produce many overlapping windows from a small number of documents\. To avoid letting a single repeated source dominate the forget set, we cap each requested work atC=128C=128windows and allocate this budget across retrieved documents\. Letviv\_\{i\}be the number of available windows for documentii, ordered by retrieval rank\. We first give each document up toF=4F=4windows, bounded by availability and the remaining budget\. Any remaining budget is then distributed round\-robin over documents in rank order until the cap is reached or no document has unused windows\. If a document has more available windows than its quota, we take evenly spaced windows from that document\.

##### Retain data\.

Following prior work\[[12](https://arxiv.org/html/2608.14855#bib.bib38),[18](https://arxiv.org/html/2608.14855#bib.bib26)\]we useWikiTextas the retain setDrD\_\{r\}, we keep this fixed across curators to isolate the effect of the forget setDfD\_\{f\}\. During training, each forget example is paired with a randomly sampled retain example when the unlearning objective uses retain regularization\.

### Appendix IScaling Forget Request

See Table[6](https://arxiv.org/html/2608.14855#A9.T6)\.

Table 6:Forget\-set size ablation at\|ℱ\|=100\|\\mathcal\{F\}\|=100, reported for Llama\-3\.1\-8B and Qwen3\-8B\. Metrics and columns follow Table[1](https://arxiv.org/html/2608.14855#S6.T1)\.
### Appendix JCurator×\\timesUnlearner×\\timesModel Cross Evaluation

See Table[7](https://arxiv.org/html/2608.14855#A10.T7)\. For each⟨\\langlemodel, curator⟩\\ranglepair, the curated forget setDfD\_\{f\}is constructed once and reused across SimNPO, UNDIAL, and RMU, so within each block only the unlearning algorithm changes\. SimNPO rows repeat the corresponding entries of Table[1](https://arxiv.org/html/2608.14855#S6.T1)\(same runs\), and the per\-model cross evaluation for the EA curator is Table[2](https://arxiv.org/html/2608.14855#S6.T2)\. Table[3](https://arxiv.org/html/2608.14855#S6.T3)reports the three\-model averages\.

Table 7:Per\-model curator×\\timesunlearner cross evaluation at\|𝒲f\|=50\|\\mathcal\{W\}\_\{f\}\|=50\. Metrics and columns follow Table[1](https://arxiv.org/html/2608.14855#S6.T1)\.
### Appendix KCurator Output Sizes

The curators produce forget sets of different sizes;\|Df\|\|D\_\{f\}\|is a property of the curator under test, and we keep the unlearner’s optimization hyperparameters \(such as epochs\) fixed irrespective of\|Df\|\|D\_\{f\}\|\. Table[8](https://arxiv.org/html/2608.14855#A11.T8)reports the exact number of prefix–suffix windows produced by each curator at\|𝒲f\|=50\|\\mathcal\{W\}\_\{f\}\|=50\. The retain set passed to the unlearner is fixed at 1,646 WikiText rows for every configuration\.

Table 8:Number of curated forget\-set windows at\|𝒲f\|=50\|\\mathcal\{W\}\_\{f\}\|=50\.†ForOlmo\-3\-7B, only 38 works cleared the model\-extractability threshold, so its forget pool contains 38 works rather than 50\.