Subtract or Replay? Exact Deletion from Language-Model Memory

# Subtract or Replay? Exact Deletion from Language-Model Memory
Source: [https://arxiv.org/html/2607.27539](https://arxiv.org/html/2607.27539)
###### Abstract

Exact deletion from persistent memory is not one algorithmic problem: it depends on how the memory represents a record\. If a record’s influence retains an address, it can be removed by algebraic decrement; if later writes transform that influence inside a shared recurrent state, a fixed record\-wise decrement fails and rebuilding from before the write provides an exact fallback\. We make this distinction operational in two pretrained language models and audit deletion against the declared record\-omitted reference: a retained\-contextualized\-key refit for Gemma and a raw\-record rebuild for Kimi\. First, we replace Gemma 3’s global\-attention layers with support\-vector memory, whose coefficients give stored records an address\. After low\-rank recovery at 1B, decrement and retained\-key refit agree at the model’s next\-token output to median KL5\.4×10−155\.4\\times 10^\{\-15\}over3131support\-token deletions, at\+2\.0%\+2\.0\\%perplexity against a matched fine\-tune\. Separately, a single\-precision masked\-refit proxy is statistically indistinguishable from the never\-ingested floor under the evaluated elicitation, relearning, sampling, and LiRA attacks\. At 4B and 12B the certificate’s ordering persists, but utility cost grows to11\.2%11\.2\\%and44\.3%44\.3\\%: this is not a general replacement for attention\. Second, we isolate the write rule in a4848B Kimi Linear hybrid\. Additive writes admit a record\-wise decrement, and diagonal decay admits a corrected one; the delta rule does not:1212–49%49\\%of a record’s state contribution changes with the suffix, and the best decay\-corrected receipt leaves99–49%49\\%suffix dependence\. Checkpointed rewind\-and\-replay then deletes real clinical records at contexts up to18,84218\{,\}842tokens, bit\-for\-bit equal within the deterministic MLX implementation on logits and all recurrent states to never ingesting the record, at cost proportional only to the suffix; replaying a corrected record gives the same guarantee for amendment\. Exact deletion is therefore a property of memory representation: subtract where influence is addressable, rebuild where it has been woven into state\.

## 1Introduction

Imagine that an assistant has already compressed yesterday’s conversation into a long\-term state\. The original text is no longer being re\-read, so deleting a sentence from a visible prompt does not delete what the assistant already stored\. The stakes are concrete in clinical documentation\. An ambient scribe hears “my mother had breast cancer” during history taking; the statement enters the assistant’s persistent memory and begins shaping everything downstream—the family\-history section, the risk assessment, the screening plan\. When the patient later clarifies that imaging found a benign lump, appending the correction is not enough: both statements now compete inside the memory, and the superseded one can resurface in a summary written weeks later\. A useful memory deletion must instead edit the state itself and answer a counterfactual question:*does the edited memory match the memory we would have built had this record never been included?*This paper asks which memory representations make that answer attainable without always rebuilding from the beginning\.

Machine unlearning has moved from a regulatory abstraction to an engineering requirement: the right to erasure in the EU GDPR, clinical consent withdrawal, the correction of records an assistant captured wrongly, the withdrawal of guidance or studies later retracted, and the removal of copyrighted or hazardous content all ask a deployed model to forget a specific record on demand\. The dominant response edits the weights, and the current benchmarks \(TOFU\(Mainiet al\.,[2024](https://arxiv.org/html/2607.27539#bib.bib20)\), MUSE\(Shiet al\.,[2025](https://arxiv.org/html/2607.27539#bib.bib21)\), WMDP\(Liet al\.,[2024a](https://arxiv.org/html/2607.27539#bib.bib22)\)\) score how well a fact stops being recalled after the edit\. Work through 2024–2026 has made that response look fragile in three ways: the forgetting is*shallow*\(the target survives in intermediate layers, recoverable by a probe or logit diff\(Wang and others,[2025](https://arxiv.org/html/2607.27539#bib.bib27); Anonymous,[2025](https://arxiv.org/html/2607.27539#bib.bib26)\)\),*reversible*\(a few benign fine\-tuning steps return the fact to full extractability\(Huet al\.,[2024](https://arxiv.org/html/2607.27539#bib.bib23)\)\), and*hard to measure*\(accuracy or ROUGE drops can reflect a minor logit shift over intact geometry\(Thakeret al\.,[2024](https://arxiv.org/html/2607.27539#bib.bib25); Lynchet al\.,[2024](https://arxiv.org/html/2607.27539#bib.bib24)\)\)\. The common thread: approximate unlearning leaves a residue, and the residue is what the deployment\-time adversary recovers\.

Unlearning by design changes the problem from repairing an arbitrary model after the fact to choosing a representation from which records can later be removed\. MUNKEY is the closest expression of that idea: it trains an image classifier with one learnable exemplar token per training instance in an external keyed bank, then forgets an instance by deleting its key\(Lagunaet al\.,[2026](https://arxiv.org/html/2607.27539#bib.bib14)\)\. Its result validates the structural premise\. It also leaves a different question open: what exact deletion means after an autoregressive language model has already ingested a record into persistent attention or recurrent state\. MUNKEY evaluates classification accuracy and output\-space membership relative to a separately retrained oracle; we ask whether the*same instantiated LLM memory*, after an edit, matches its own declared record\-omitted reference at the next\-token output\.

Our thesis is that the exact operation is determined by the representation\.If the memory preserves an address for a record’s influence—a coefficient, cache entry, or other record\-local quantity whose update can be reversed—deletion can be an algebraic decrement\. If each new write reads and transforms a shared recurrent state, the old record’s contribution changes with the suffix; a receipt saved when the record arrived no longer names what must be removed\. Returning to a state before the write and rebuilding what follows is then an exact fallback\. Section[3](https://arxiv.org/html/2607.27539#S3)turns this distinction into a counterfactual criterion; the experiments test both branches rather than presenting two unrelated model conversions\.

For the addressable branch,Ramesh \([2026](https://arxiv.org/html/2607.27539#bib.bib1)\)provide the needed primitive: a memory layer whose attention weights are coefficients of a one\-class support\-vector fit over context keys\. A classical incremental algorithm\(Cauwenberghs and Poggio,[2000](https://arxiv.org/html/2607.27539#bib.bib29)\)can reverse the solve so that deleting a token returns it to the state obtained by refitting without that token\. We graft this memory into Gemma 3\(Gemma Team, Google DeepMind,[2025](https://arxiv.org/html/2607.27539#bib.bib3)\), recover language quality with low\-rank adaptation, and carry the certificate through the full model\. At 1B, decrement and retained\-key refit agree at the next\-token output to median KL5\.4×10−155\.4\\times 10^\{\-15\}over3131support\-token deletions\. In separate behavioral runs, a single\-precision masked refit is statistically indistinguishable from the never\-ingested floor under elicitation, relearning, sampling, and membership attacks, whereas prompt\-space unlearning stays extractable and weight editing is reversed\.

For the non\-addressable branch, we study the2020recurrent delta\-rule layers of a4848B Kimi Linear hybrid\. Controlled continuations localize the loss of separability to the delta rule: additive writes admit a fixed receipt, diagonal decay admits a ledger\-corrected receipt, but under the full rule1212–49%49\\%of a record’s contribution depends on what followed it\. We therefore checkpoint record boundaries and replay only the suffix\. Within the same deterministic MLX implementation, the result equals never ingesting the record bit for bit on logits and every recurrent state, including real clinical records at contexts up to18,84218\{,\}842tokens; the same operation installs a corrected record as an exact amendment\. Figure[1](https://arxiv.org/html/2607.27539#S1.F1)states the common standard and the representation\-dependent mechanisms; the released demos walk complete records through both\.

![Refer to caption](https://arxiv.org/html/2607.27539v1/x1.png)Figure 1:*Exact deletion follows representation\.**\(a\)*Appending a correction leaves the old record in memory; deletion instead targets a declared record\-omitted reference\.*\(b\)*Addressable influence can be decremented: Gemma’s support\-vector memory matches its retained\-key refit to median KL5\.4×10−155\.4\\times 10^\{\-15\}\.*\(c\)*KDA’s later writes make a fixed receipt suffix\-dependent \(1212–49%49\\%\), so exact deletion restores a checkpoint and replays the suffix, bitwise equal within the same implementation through18,84218\{,\}842tokens\. Replaying a replacement gives exact amendment\.#### Scope of the claims\.

An exact claim is only as strong as its boundaries\. We forget from the model’s*in\-context memory*—a persistent, already\-ingested store carried by the grafted layers, the setting of in\-context unlearning\(Pawelczyket al\.,[2024](https://arxiv.org/html/2607.27539#bib.bib17)\)—not from its weights, so weight\-space benchmarks are run in an explicit in\-context framing\. One may object that we delete from a memory designed to be deletable rather than from Gemma’s native one\. Designing the representation is the intervention—the same broad premise as MUNKEY—and the grafted store is load\-bearing after recovery: the model’s only long\-range carrier at\+2\.0%\+2\.0\\%utility cost, not a side\-car database\. The stronger hybrid result does not redesign KDA: replay removes the record from its native recurrent state\. On Gemma, the certificate compares decrement with a refit over the same retained contextualized keys; a fully repacked context also removes ingestion\-time imprint on neighbors and is evaluated separately\. Neither operation can revoke outputs observed before deletion\(Anonymous,[2025](https://arxiv.org/html/2607.27539#bib.bib26)\)\.

Our contributions are as follows:

1. 1\.A representation\-level criterion for exact deletion\.We define deletion against a counterfactual rebuild and distinguish record\-local influence from influence transformed by later writes: additive recurrence is subtractable, diagonal decay is ledger\-correctable, and KDA’s delta rule violates record\-wise separability\.
2. 2\.Addressable deletion in an existing LLM\.A recovered Gemma 3 graft makes decrement and retained\-key refit agree at the 1B next\-token output to median KL5\.4×10−155\.4\\times 10^\{\-15\}\. Behavioral masked\-refit controls distinguish state removal from suppression; scaling exposes utility costs of2\.0%2\.0\\%,11\.2%11\.2\\%, and44\.3%44\.3\\%at 1B/4B/12B\.
3. 3\.Exact deletion and amendment when record\-wise subtraction fails\.On a4848B Kimi Linear hybrid, replay is bitwise equal to never ingesting the record on logits and all2020recurrent states through18,84218\{,\}842tokens, supplies ground truth for the residual left by attention\-only masking, and installs exact corrections\.

Gemma and Kimi share a counterfactual framework but use the references named above; representation decides whether decrement or replay reaches each one\.

## 2Related Work

Unlearning by design\.MUNKEY\(Lagunaet al\.,[2026](https://arxiv.org/html/2607.27539#bib.bib14)\)is the closest conceptual work\. It trains a ViT classifier with an external bank of learned per\-example tokens and makes an instance unavailable by deleting its retrieval key\. We share its premise that deletability is an architectural property, not merely a better post\-hoc optimizer\. The setting and standard differ\. MUNKEY targets training\-example removal in image classification and reports forget/retain accuracy plus output\-space membership inference relative to a separately retrained model\. We target records already ingested into persistent autoregressive LLM state, retrofit existing models, compare the edited memory with an explicitly declared record\-omitted reference at next\-token output, and measure when recurrent writes make key deletion insufficient\. Classical exact unlearning by retraining or partitioned retraining\(Cao and Yang,[2015](https://arxiv.org/html/2607.27539#bib.bib18); Bourtouleet al\.,[2021](https://arxiv.org/html/2607.27539#bib.bib19)\)supplies the same counterfactual ideal at a different substrate and cost\.

Behavioral LLM unlearning\.ICUL\(Pawelczyket al\.,[2024](https://arxiv.org/html/2607.27539#bib.bib17)\)is the closest inference\-time baseline: it places a forget instruction over unchanged memory, whereas we edit the memory state itself\. Weight\-space suites \(TOFU, MUSE, WMDP\(Mainiet al\.,[2024](https://arxiv.org/html/2607.27539#bib.bib20); Shiet al\.,[2025](https://arxiv.org/html/2607.27539#bib.bib21); Liet al\.,[2024a](https://arxiv.org/html/2607.27539#bib.bib22)\)\) motivate our attack axis but test a different substrate, and approximate edits are reversed by benign relearning, recovered from representations, overstated by output metrics, or exposed by sampling\(Huet al\.,[2024](https://arxiv.org/html/2607.27539#bib.bib23); Wang and others,[2025](https://arxiv.org/html/2607.27539#bib.bib27); Thakeret al\.,[2024](https://arxiv.org/html/2607.27539#bib.bib25); Lynchet al\.,[2024](https://arxiv.org/html/2607.27539#bib.bib24); Reisizadehet al\.,[2025](https://arxiv.org/html/2607.27539#bib.bib2)\)\. We therefore place TOFU content in persistent context and compare prompt\-, weight\-, approximate\-memory\-, and exact\-memory interventions under one readback\. A pre/post snapshot adversary remains out of scope for every method here\(Anonymous,[2025](https://arxiv.org/html/2607.27539#bib.bib26)\)\.

Grafting and memory systems\.Our conversion follows attention\-transfer and low\-rank\-recovery recipes\(Zhanget al\.,[2024](https://arxiv.org/html/2607.27539#bib.bib4); Wanget al\.,[2024](https://arxiv.org/html/2607.27539#bib.bib5)\), but replaces only Gemma’s global layers to add a deletion capability rather than a throughput claim\. Cache\-eviction methods rank and discard entries\(Zhanget al\.,[2023](https://arxiv.org/html/2607.27539#bib.bib31); Liet al\.,[2024b](https://arxiv.org/html/2607.27539#bib.bib32)\); VeriCache keeps a full reference for verification\(Yaoet al\.,[2026](https://arxiv.org/html/2607.27539#bib.bib33)\)\. KVEraser identifies the same causal\-cache contamination boundary: exact context erasure recomputes the affected suffix, while its learned steering cache approximates that reference\(Liet al\.,[2026](https://arxiv.org/html/2607.27539#bib.bib15)\)\. Our retained\-key Gemma certificate leaves this imprint explicit; Kimi replay reaches the full rebuilt state\.

Recurrent memories and hybrids\.Linear attention, DeltaNet, Gated DeltaNet, state\-space models, and test\-time memories replace a growing cache with a fixed\-size state\(Katharopouloset al\.,[2020](https://arxiv.org/html/2607.27539#bib.bib38); Yanget al\.,[2024](https://arxiv.org/html/2607.27539#bib.bib35);[2025](https://arxiv.org/html/2607.27539#bib.bib36); Gu and Dao,[2024](https://arxiv.org/html/2607.27539#bib.bib39); Behrouzet al\.,[2025](https://arxiv.org/html/2607.27539#bib.bib40)\); hybrid architectures interleave both forms\(Lieberet al\.,[2024](https://arxiv.org/html/2607.27539#bib.bib42); Kimi Team,[2025](https://arxiv.org/html/2607.27539#bib.bib37)\)\. The delta rule is important here precisely because its targeted overwrite makes recurrent memory more expressive\(Yanget al\.,[2024](https://arxiv.org/html/2607.27539#bib.bib35);[2025](https://arxiv.org/html/2607.27539#bib.bib36)\): each write reads the state it changes\. Prior work studies forgetting as a*capability*problem—an overfull state that cannot discard old tokens degrades long\-context use\(Chenet al\.,[2024](https://arxiv.org/html/2607.27539#bib.bib43)\)—whereas record deletion asks for a counterfactual state\. Deterministic checkpoint replay has also been proposed for exact*training\-state*unlearning\(X,[2025](https://arxiv.org/html/2607.27539#bib.bib16)\); our replay is inference\-time, and the contribution is the measured representation boundary rather than checkpointing\. Section[7](https://arxiv.org/html/2607.27539#S7)decomposes the KDA update to show where a compact record receipt stops working, then uses replay both as the exact operation and as an oracle for the residual left by attention\-only masking\.

## 3Exact deletion as a representation problem

### 3\.1The counterfactual standard

LetR=\(r1,…,rn\)R=\(r\_\{1\},\\ldots,r\_\{n\}\)be a sequence of records,B​\(R\)B\(R\)the persistent memory produced by ingesting them, andf​\(q;B​\(R\)\)f\(q;B\(R\)\)the model’s next\-token distribution for queryqq\. Deletingrjr\_\{j\}is exact at the output when

f​\(q;Dj​\(B​\(R\)\)\)=f​\(q;B​\(R∖j\)\),f\\\!\\left\(q;D\_\{j\}\(B\(R\)\)\\right\)=f\\\!\\left\(q;B\(R\_\{\\setminus j\}\)\\right\),\(1\)up to a declared numerical tolerance\. Equality of the memory states themselves is stronger and implies Equation[1](https://arxiv.org/html/2607.27539#S3.E1)\. A behavioral score cannot establish either equality: many interventions can suppress the target answer while producing a third state that is neither side of the equation\. The unitsrir\_\{i\}and builderBBare part of the declared reference: they are contextualized key records for Gemma’s retained\-key certificate and raw input records for Kimi’s replay certificate\.

The reference must also be named precisely\. Gemma’s certificate holds the already\-contextualized retained keys fixed and compares a decrement with refitting the support\-vector solve on those keys; it therefore certifies the edited persistent store conditional on ingestion\. A full token\-stream rebuild also removes the wayrjr\_\{j\}shaped neighboring keys during ingestion\. Kimi’s replay reaches that stronger reference and is checked for bitwise state equality\. We report the two standards separately rather than calling both “the record disappeared\.”

### 3\.2When does a record have a deletion address?

Call a record*addressable*when the memory retains sufficient record\-local bookkeepingaja\_\{j\}for a decrement

Dec⁡\(B​\(R\),aj\)=B​\(R∖j\)\\operatorname\{Dec\}\(B\(R\),a\_\{j\}\)=B\(R\_\{\\setminus j\}\)\(2\)without reprocessing the other records\. An external key–value entry is the simplest address\(Lagunaet al\.,[2026](https://arxiv.org/html/2607.27539#bib.bib14)\); a fitted memory can also be addressable when its update algorithm is exactly reversible\. Addressability does not require the memory itself to be additive\. It requires the effect of removing one record to remain recoverable from the stored solve and the record’s own bookkeeping\.

A recurrent write exposes the boundary\. For prefixPP, victimxx, and suffixSS, define the victim’s state contribution in common coordinates as

δx​\(S\)=B​\(P,x,S\)−B​\(P,S\)\.\\delta\_\{x\}\(S\)=B\(P,x,S\)\-B\(P,S\)\.\(3\)A fixed receipt saved whenxxarrives can be subtracted after arbitrary continuations only ifδx​\(S\)\\delta\_\{x\}\(S\)is suffix\-independent, or can be derived from that receipt by a compact suffix ledger\. Additive writes satisfy the first condition\. Per\-channel decay satisfies the second because a running product transports the receipt forward\. A delta\-rule write reads the current state before changing it, so later content transforms the old contribution in a content\-dependent way\. Section[7](https://arxiv.org/html/2607.27539#S7)tests this criterion on captured KDA inputs: the delta term, not recurrence alone, is where record\-wise separability fails\.

The exact fallback follows directly\. SaveB​\(P\)B\(P\), restore it whenxxis deleted, and ingestSSagain\. Determinism givesB​\(P,S\)B\(P,S\), with cost proportional to the suffix\. Checkpointing itself is elementary; the methodological contribution is the decision boundary around it: measuring whether a decrement is valid, localizing the failure to the write rule, and using replay as ground truth to quantify what an instant edit leaves behind\.

### 3\.3Addressable realization: the support\-vector gate

We use the gate ofRamesh \([2026](https://arxiv.org/html/2607.27539#bib.bib1)\)unchanged; the machinery is theirs and we recap only what this paper leans on\. Given context keysxi∈ℝdx\_\{i\}\\in\\mathbb\{R\}^\{d\}, valuesviv\_\{i\}, and a queryqq, the gate reads out a kernel\-weighted average whose weights are gated by coefficientsα\\alphasolving a one\-class support vector description\(Tax and Duin,[2004](https://arxiv.org/html/2607.27539#bib.bib30)\)over the keys, with a boxC=1/\(ν​n\)C=1/\(\\nu n\)set by a budgetν∈\(0,1\]\\nu\\in\(0,1\]\. The solve partitions the keys into margin, error, and*reserve*sets, the last with coefficients identically zero\. The property everything rests on is thecertificate: a reserve token contributes exactly zero to the readout, and the incremental algorithm ofCauwenberghs and Poggio \([2000](https://arxiv.org/html/2607.27539#bib.bib29)\), run in reverse, decrements any token so the solve returns to the state it would hold without it\. For every deletion certificate,CCis frozen at its pre\-deletion value; “retained\-key refit” therefore means a fixed\-CCrefit, not a fresh solve at unchangedν\\nu\. “Forgetting” is this decrement; “decay,” our approximate foil, scales a coefficient toward zero and leaves a residual\.

Repacking the raw context withoutXXand re\-ingesting reaches the stronger, imprint\-free record\-omitted reference\. The decrement instead reaches the fixed\-CCretained\-key refit; its advantage is cost\. On the recovered 1B model with an811811\-token memory, one head\-gate decrement takes3\.43\.4ms median versus324324ms for a fresh refit of that gate \(97×97\\times\), and a whole\-model deletion is0\.30\.3s of decrements versus3333s of repacking \(float64/CPU, the certificate\-grade configuration\)—a gap that compounds over a deletion stream and spares downstream caches keyed on the original token positions\.

### 3\.4Making Gemma’s long\-range memory addressable

Gemma 3\(Gemma Team, Google DeepMind,[2025](https://arxiv.org/html/2607.27539#bib.bib3)\)interleaves five local sliding\-window attention layers for every one global layer \(5:15\{:\}1\)\. We replace only the four global layers of Gemma\-3\-1B and leave its2222local layers untouched \(Appendix Figure[2](https://arxiv.org/html/2607.27539#A1.F2)\), so the claim is confined to long\-range memory\. The graft reuses Gemma’s projections, normalization, rotary embeddings, and grouped\-query layout\. Its prefix readout retains learnedq⋅kq\{\\cdot\}kscores but multiplies each long\-range key by the support coefficientα\\alpha; a reserve key withα=0\\alpha=0therefore contributes exactly zero\. Recent in\-window tokens remain on the ordinary local path\.

### 3\.5Recovering the redesigned memory

We freeze the base model, fit one kernel bandwidth per grafted layer to match the original attention output, then train rank\-88LoRA adapters through the differentiable gate on language\-model loss\(Huet al\.,[2022](https://arxiv.org/html/2607.27539#bib.bib6); Penedoet al\.,[2024](https://arxiv.org/html/2607.27539#bib.bib7)\):1\.491\.49M parameters \(0\.149%0\.149\\%\)\. The LoRA stage carries the recovery outright—removing the bandwidth warm\-start reaches the same recovered perplexity \(21\.1321\.13vs\.21\.1821\.18; Appendix[B](https://arxiv.org/html/2607.27539#A2)\)—and data, optimization, and runtime details are in Appendix[A](https://arxiv.org/html/2607.27539#A1)\.

### 3\.6The exact decrement, audited at the model output

Inference and training use a single\-precision batched solver for the gate; the exactness*guarantee*does not\. To delete target token\(s\) from the in\-context memory, we run the float64 incremental decrement ofCauwenberghs and Poggio \([2000](https://arxiv.org/html/2607.27539#bib.bib29)\)to obtain the support coefficients of the refit\-without state, and inject those coefficients directly into the live readout \(bypassing the single\-precision solver\) while running the full model in double precision, yielding the post\-deletion next\-token logits \(Appendix Figure[2](https://arxiv.org/html/2607.27539#A1.F2)\)\. The decay foil instead scales the target’s readout weight byγ=0\.01\\gamma=0\.01\. We never use the approximate solver for an exactness claim: certificate KLs route through this float64 decrement\. The attack, membership, and sampling suites instead use the ordinary single\-precision forward solver with the target positions masked out and the gate refit; they are behavioral evidence for the designed removal path, not part of the numerical certificate\.

## 4The graft preserves utility at the 1B scale

The LoRA recovery itself improves WikiText, so comparing only with the original model would overstate utility; we instead give an ungrafted control the identical data, rank, steps, and seed\. Against that control the recovered graft costs\+2\.0%\+2\.0\\%WikiText perplexity \(21\.1821\.18vs\.20\.7620\.76\); mean accuracy across ARC\-easy/challenge, PIQA, WinoGrande, and HellaSwag moves−0\.11\-0\.11percentage points \(2,0002\{,\}000examples/task\)\(Gaoet al\.,[2024](https://arxiv.org/html/2607.27539#bib.bib9); Bisket al\.,[2020](https://arxiv.org/html/2607.27539#bib.bib11); Sakaguchiet al\.,[2021](https://arxiv.org/html/2607.27539#bib.bib12); Zellerset al\.,[2019](https://arxiv.org/html/2607.27539#bib.bib13)\); and the mean perplexity overhead across WikiText, Lambada, and C4 is\+0\.9%\+0\.9\\%\. These support near\-parity at 1B—not at larger scales \(Section[6](https://arxiv.org/html/2607.27539#S6)\)\. The full ladders, task table, and the summary figure are in Appendix[B](https://arxiv.org/html/2607.27539#A2)\.

## 5Certificate and behavioral removal separate deletion from suppression

This section keeps two evidentiary paths distinct\. The float64 certificate asks whether decrement agrees with a fixed\-CCretained\-key refit at the model output\. The attack suite asks whether the single\-precision masked\-refit path leaves detectable target behavior\. The latter cannot extend the numerical certificate, but it tests the evaluated removal mechanism against suppression baselines\. Appendix Figure[4](https://arxiv.org/html/2607.27539#A4.F4)collects both\.

#### Substrates\.

The certificate captures contextualized keys from a fixed192192\-token generic prompt and selects3131support positions across the grafted heads/layers, deleting each position in turn\. Behavioral experiments instead use TOFU\(Mainiet al\.,[2024](https://arxiv.org/html/2607.27539#bib.bib20)\)fictitious\-author facts \(no real personal data\), packed once into state and queried across turns\. Every target lies beyond the local sliding window, so the grafted long\-range memory is the only direct path back to it: by the time we delete, the raw prompt is no longer being supplied, which is what makes removal non\-trivial\. The certificate compares the decrement against refitting the same retained contextualized keys; the one channel outside it—a record shapes the cached encodings of its neighbors while being ingested—is measured against a fully repacked context, the∼100×\{\\sim\}100\\times\-costlier honest fallback when imprint\-free state is mandatory\. Text still inside the local window needs no certificate: the window is transient, deletion there is a bounded re\-prefill, and the demo’s in\-window control marks the boundary \(secret2525tokens back: gate decrement alone leaves it recalled through the untouched local layers,pp0\.98→0\.290\.98\\to 0\.29, floor0\.0110\.011\)\. Deleting from a retrieval index governs future retrieval, not already\-ingested state\(Lewiset al\.,[2020](https://arxiv.org/html/2607.27539#bib.bib34)\)\.

“Recovery” is normalized between a budget\-matched never\-ingested floor \(0\) and original recall \(11\); each panel conditions on measurable pre\-deletion recall and reports its own denominator \(3131support\-token KL cases,3636efficacy,6060elicitation,∼48\{\\sim\}48relearning,108108retain pairs,312312LiRA tests per side; protocols in Appendix[D](https://arxiv.org/html/2607.27539#A4)\)\. Certificate panels use float64 decrement and coefficient decay\. Behavioral panels use single\-precision masked refit, coefficient decay \(γ=0\.01\\gamma=0\.01\), ICUL’s prompt instruction\(Pawelczyket al\.,[2024](https://arxiv.org/html/2607.27539#bib.bib17)\), and per\-target gradient ascent in weight space\(Mainiet al\.,[2024](https://arxiv.org/html/2607.27539#bib.bib20)\)\.

#### Certificate \(Appendix Fig\.[4](https://arxiv.org/html/2607.27539#A4.F4)c,e\)\.

Across3131support\-token deletions, decrement matches retained\-key refit at median KL5\.4×10−155\.4\\times 10^\{\-15\}and worst9\.3×10−149\.3\\times 10^\{\-14\}; decay’s median is1\.8×10−61\.8\\times 10^\{\-6\}\. Throughk∈\{1,2,5,10,20,30\}k\\in\\\{1,2,5,10,20,30\\\}sequential deletions, exact remains10−1510^\{\-15\}–1\.5×10−141\.5\\times 10^\{\-14\}while decay grows700×700\\timesto4\.7×10−34\.7\\times 10^\{\-3\}\. The methods can therefore look equally forgotten while differing by eleven orders of magnitude on refit agreement\.

#### Behavior \(Appendix Fig\.[4](https://arxiv.org/html/2607.27539#A4.F4)a,b,d\)\.

On3636admitted targets, efficacy is1\.08±0\.091\.08\\pm 0\.09\(95% CI; floor is11\), while108108retain pairs move only\+0\.027±0\.011\+0\.027\\pm 0\.011\. Masked refit remains near the floor as elicitation hints grow \(0\.00→0\.020\.00\\to 0\.02\) and under related\-data LoRA relearning \(0\.01→−0\.100\.01\\to\-0\.10\); ICUL remains extractable \(0\.95→0\.560\.95\\to 0\.56and1\.14→1\.241\.14\\to 1\.24\)\. Full budgets and protocols are in Appendix[D](https://arxiv.org/html/2607.27539#A4)\.

#### Weight\-space control\.

GA reaches the floor on1919admitted targets, but harms retain extraction by−0\.125±0\.030\-0\.125\\pm 0\.030and is reversed by benign fine\-tuning, overshooting original recall by\+3\.5±2\.2\+3\.5\\pm 2\.2on1010targets\. It suppresses the shared reader rather than deleting the contextual record\.

#### Membership and interpretation \(Appendix Fig\.[4](https://arxiv.org/html/2607.27539#A4.F4)f\)\.

In\-context LiRA uses3232shadow packings per side and312312held\-out tests per side\. Masked refit is at chance \(AUC0\.4990\.499, TPR1\.3%1\.3\\%at1%1\\%FPR\), versus present0\.9960\.996, ICUL0\.9890\.989, and decay0\.5330\.533\. Heavy decay therefore overlaps masked refit on coarse behavior and membership; only output\-KL and sequential stability separate a certified refit\-equivalent deletion from a small residual that accumulates\.

#### Probabilistic extraction and entangled pairs\.

Greedy readback can under\-report what sampling recovers, so we run Leak@kk\(Reisizadehet al\.,[2025](https://arxiv.org/html/2607.27539#bib.bib2)\):200200temperature\-11samples per fact and condition, the attacker supplying the answer template\. Masked refit’s per\-target excess over the never\-stored floor is−0\.000​\[−0\.011,\+0\.007\]\-0\.000\\;\[\-0\.011,\+0\.007\]atk=1k=1\(95%95\\%bootstrap,n=20n\{=\}20\)—indistinguishable from never storing the fact—while ICUL exceeds the floor at everykk; teacher\-forced probes explain why \(stored lift\+1\.50\+1\.50nats; deletion leaves\+0\.04\+0\.04, ICUL\+1\.49\+1\.49\)\. In entangled two\-record prompts, the deleted secret is statistically consistent with the floor while the retained secret’s rank is unchanged\. Full curves, gates, and budgets: Figure[5](https://arxiv.org/html/2607.27539#A5.F5), Appendix[E](https://arxiv.org/html/2607.27539#A5)\.

## 6A three\-model scaling study

We repeat recovery, matched control, and output\-level forgetting at 4B and 12B with the same rank\-88budget and seed\. The result is graded, not uniformly positive\.

At 4B, median decrement/refit output KL remains4\.8×10−154\.8\\times 10^\{\-15\}versus4\.9×10−74\.9\\times 10^\{\-7\}for decay, but the worst case rises to5\.7×10−85\.7\\times 10^\{\-8\}: a small conditioning tail\. At 12B, the median rises to6\.3×10−96\.3\\times 10^\{\-9\}\(worst4\.5×10−74\.5\\times 10^\{\-7\}\) versus a2\.7×10−62\.7\\times 10^\{\-6\}decay median, and the distributions overlap at their edges\. A diverse\-prompt control preserves the ordering\. Thus algebraic refit\-equivalence survives, but realized float64 precision must be measured per deletion \(full distributions and sequential diagnostics: Appendix[C](https://arxiv.org/html/2607.27539#A3)\)\.

Utility degrades more sharply \(Table[1](https://arxiv.org/html/2607.27539#S6.T1)\): matched\-control overhead grows from2\.0%2\.0\\%at 1B to11\.2%11\.2\\%at 4B and44\.3%44\.3\\%at 12B\. Doubling LoRA rank leaves the 4B gap at10\.9%10\.9\\%; the 12B rank\-1616grafted run destabilizes while its control trains normally\. Near\-parity is therefore a 1B result\.

Table 1:*The recipe’s utility cost grows with scale; near\-parity is a 1B result\.*De\-confounded scaling ladder \(WikiText\-103 ppl,400400blocks; gate cost=recovered/control−1=\\text\{recovered\}/\\text\{control\}\-1\)\.
## 7When later writes erase the address: exact deletion by replay

Section[3](https://arxiv.org/html/2607.27539#S3)predicts the deletion operation from one property: whether a record’s influence retains an address after later writes\. Kimi Linear\(Kimi Team,[2025](https://arxiv.org/html/2607.27539#bib.bib37)\)tests both sides in one released4848B hybrid:77global MLA attention layers\(DeepSeek\-AI,[2024](https://arxiv.org/html/2607.27539#bib.bib41)\)keep token positions, while2020KDA layers implement a gated\-DeltaNet recurrence\(Yanget al\.,[2024](https://arxiv.org/html/2607.27539#bib.bib35);[2025](https://arxiv.org/html/2607.27539#bib.bib36)\)\. We train nothing\. Controlled continuations show that KDA’s delta rule fails record\-wise separability \(Table[2](https://arxiv.org/html/2607.27539#S7.T2)\)\. We therefore replay the suffix and require bitwise equality with never ingesting the record, on logits and every recurrent state \(architecture in Appendix Figure[7](https://arxiv.org/html/2607.27539#A7.F7)\)\.

#### Decay is not deletion in a hybrid\.

Setting all2020recurrent states to zero still leaves a planted ward code as the model’s top prediction \(p=0\.881p=0\.881\), because attention retains the text;2,0482\{,\}048unrelated tokens likewise leave its pull undiminished \(\+3\.5→\+4\.0\+3\.5\\to\+4\.0nats\)\. Any complete operation must address both memories \(Appendix Table[7](https://arxiv.org/html/2607.27539#A7.T7)\)\.

#### Replay reaches the counterfactual state\.

We save recurrent and convolutional state at record boundaries, restore the checkpoint before the victim, and replay only its suffix\. Within the same deterministic MLX implementation and released88\-bit weights, every admitted MIMIC deletion is bitwise exact:8/88/8and9/99/9intake records at236236and3,2483\{,\}248tokens, and4/44/4discharge notes at18,84218\{,\}842tokens\. Victim lift and retained drift both become exactly0\.0000\.000\. Over128128records, latency falls from6\.706\.70s for the oldest to0\.000\.00s for the newest \(mean3\.493\.49s versus6\.776\.77s for a full rebuild\);129129checkpoints cost5\.225\.22GiB \(Appendix Figure[8](https://arxiv.org/html/2607.27539#A7.F8)\)\. Replacing the victim before replay also yields a bitwise\-exact amendment \(Appendix Table[9](https://arxiv.org/html/2607.27539#A7.T9)\)\.

#### Localizing what direct\-channel removal misses\.

We graft a single\-precision FISTA gate into the seven MLA layers and mask the victim’s positions\. This is an attention\-only diagnostic, not Gemma’s float64 decrement or a certified Kimi deletion\. Against the replay oracle, masking disagrees on all1212victims: seven leave positive residual and five overshoot\. It removes8686–95%95\\%of TOFU lift where it does not overshoot, while42%42\\%of one CDS victim and46%46\\%of one note victim survive; retained records also move\. Replay is exact everywhere at suffix cost \(full results in Appendix Tables[6](https://arxiv.org/html/2607.27539#A7.T6)and[8](https://arxiv.org/html/2607.27539#A7.T8)\)\.

Table 2:*The delta rule destroys a fixed deletion address\.*Relative difference in one record’s state contribution under two suffixes;0permits a fixed receipt\. Additive writes are separable, decay is corrected by a running product, and KDA remains suffix\-dependent\. Ranges pool three corpora; full protocol in Appendix[G](https://arxiv.org/html/2607.27539#A7)\.
#### Why a receipt fails\.

A running sum preserves a record’s contribution; diagonal decay transports it predictably with one product ledger \(≤7×10−5\\leq 7\\times 10^\{\-5\}corrected error\)\. KDA’s delta update instead reads the state it changes, so1212–49%49\\%of the contribution varies with the suffix and decay correction leaves99–49%49\\%\. At full\-model level the two counterfactual contributions are nearly unrelated despite retaining22–32%32\\%of state magnitude\. These measurements rule out a fixed record receipt and the tested compact decay ledger on the captured KDA inputs\. Replay is the exact fallback evaluated here; richer influence tracking or other deletion algorithms are not ruled out\. This is the measured representation boundary, not a claim that checkpointing itself is novel\.

## 8Discussion: deletion is a property of representation

The experiments support one methodological claim\. Exact deletion is cheap only when the representation preserves a record\-wise address\. Gemma’s support\-vector solve does, so a decrement reaches the retained\-key refit without rereading the context\. KDA’s delta rule does not: the counterfactual contribution of one record changes under later writes, ruling out a fixed receipt even after diagonal\-decay correction\. Replay then reaches the stronger fully rebuilt state at suffix cost\. MUNKEY reaches the same broad design principle through an external exemplar bank\(Lagunaet al\.,[2026](https://arxiv.org/html/2607.27539#bib.bib14)\); our result extends it from key deletion to a criterion that tests whether a fixed record\-wise decrement remains valid inside persistent autoregressive LLM memory\.

This distinction is also why checkpointing is not the result by itself\. Replay supplies a known exact operation\. The new evidence is that the cheaper alternative succeeds for one representation, fails for another for a localized algebraic reason, and leaves a measurable residual under an attention\-only mask in a hybrid\. The shared counterfactual framework makes those outcomes comparable\.

For a record whose deletion is an obligation, “the answer disappeared,” “the memory decrement equals its retained\-key refit,” and “a full rebuild contains no ingestion\-time imprint” are three different statements\. On Gemma this work certifies the second and measures the third\. On the hybrid, where Section[7](https://arxiv.org/html/2607.27539#S7)rules out a fixed record\-wise receipt for KDA, replay certifies the third outright, bitwise, atO​\(suffix\)O\(\\text\{suffix\}\)rather than full\-repack cost\. Aggressive decay can match the masked\-refit proxy on every coarse behavioral metric in Appendix Figure[4](https://arxiv.org/html/2607.27539#A4.F4)\. On the separate certificate panels, decay deviates from a true refit and compounds over a stream\. The gain is therefore the certificate, not a claim that one more behavioral score improved\.

Replay also makes amendment exact: insert the corrected record at the restored boundary before replaying the suffix, then compare with a memory built from that correction from the start\. The released clinical scenario passes this audit bitwise in1\.01\.0s \(Appendix Table[9](https://arxiv.org/html/2607.27539#A7.T9)\)\. Output\-KL and state equality are useful here because they certify deletion without printing protected content\. They govern future model behavior, not outputs or chart artifacts produced before the correction\.

## 9Limitations

- •Scale\.Gemma utility overhead grows from2\.0%2\.0\\%at 1B to44\.3%44\.3\\%at 12B; median certificate KL rises from∼10−14\{\\sim\}10^\{\-14\}to6×10−96\\times 10^\{\-9\}\. Near\-parity and machine precision are therefore 1B results\.
- •Scope\.We delete in\-context state, not pretrained weights or outputs observed before deletion\(Anonymous,[2025](https://arxiv.org/html/2607.27539#bib.bib26)\)\. Gemma’s retained\-key certificate also leaves ingestion\-time neighbor imprint; full repack removes it\. Attack results use a single\-precision masked refit, not the float64 decrement\.
- •KDA claim\.The experiment rules out a fixed receipt, including diagonal\-decay correction, not per\-record influence propagated through every later update or other deletion algorithms; explicit per\-record tracking would forfeit fixed\-size state\.
- •Cost\.The support\-vector solve is heavier than softmax\. Replay uses41\.441\.4MiB per record boundary here; sparse checkpoints trade storage for latency but are not evaluated\.
- •Generality\.Recovery uses one seed and one model family\. Kimi’s bitwise result is within one deterministic MLX implementation on one workstation; cross\-hardware equality and broader model families are untested\.

## 10Conclusion

Exact deletion is a contract with a memory representation\. If record influence keeps an address, as in Gemma’s grafted support\-vector solve, algebraic decrement can match a retained\-key refit at the model output; at 1B it does so to∼10−14\{\\sim\}10^\{\-14\}KL\. A separate masked\-refit proxy is statistically indistinguishable from the never\-ingested floor under the evaluated attacks\. If later delta\-rule writes transform that influence, as in Kimi Linear’s recurrent state, a fixed record receipt cannot implement the counterfactual edit; checkpointed replay reaches it bit for bit at suffix cost and supports exact amendment\. The negative scaling result matters equally: Gemma utility cost grows from2\.0%2\.0\\%at 1B to44\.3%44\.3\\%at 12B, so the present graft is not a general replacement for attention\. The broader result is a way to design and audit deletable LLM memory: test whether influence remains addressable, subtract when it does, and use rebuild as the exact fallback when a fixed receipt fails\.

## Ethics Statement

This work aims to make deletion from a deployed model’s persistent memory auditable, in direct support of erasure rights \(e\.g\. GDPR Article 17\), clinical consent withdrawal, and the removal of sensitive records from assistant memory\. The forgetting benchmarks use no real personal data: every forgotten “fact” is a fictitious TOFU biography or a synthetic demo record, and the demo’s patient and incident scenarios are invented\. The hybrid study \(Section[7](https://arxiv.org/html/2607.27539#S7)\) additionally evaluates deletion on real clinical records from MIMIC\-IV\-Ext\-CDS and MIMIC\-IV\-Note, used under PhysioNet credentialed access and its data use agreement: all processing is local, reports are aggregate\-only, no note text, identifier, or timestamp is read beyond the evaluated fields, no generations are produced from record\-bearing contexts, and a substring audit over all source values runs before any report is written\. No MIMIC data is redistributed\. We report the method’s boundaries as claims of equal standing with its strengths—the ingestion\-time imprint on neighboring representations, the pre\-deletion\-snapshot adversary that no current method defeats, and the scale\-dependence of utility and certificate precision—because an overstated deletion guarantee is itself a privacy harm\. Certified deletion could conceivably be misused to remove evidence of provenance or safety\-relevant content\. The certificate demonstrates state equivalence at audit time; proving that an authorized deletion event occurred also requires authenticated logging\.

## Reproducibility Statement

The graft, the two\-stage recovery, the float64 decrement, and every evaluation in this paper are released as code with fixed seeds; the experiments run on a single Apple\-silicon machine\. The hero demo of Figure[6](https://arxiv.org/html/2607.27539#A6.F6)is a single command \(python \-m gemma\_sv\.hero\_demo,∼6\{\\sim\}6minutes on CPU\) and a project page presents its transcript and figure alongside an interactive replay of the deletion demo that requires no backend\. The behavioral forgetting substrate is public TOFU\(Mainiet al\.,[2024](https://arxiv.org/html/2607.27539#bib.bib20)\); the support\-token certificate uses the released fixed prompt\. The language\-model corpora are public \(FineWeb\-Edu\(Penedoet al\.,[2024](https://arxiv.org/html/2607.27539#bib.bib7)\), WikiText\-103\(Merityet al\.,[2017](https://arxiv.org/html/2607.27539#bib.bib8)\)\); we release the full pipeline, including the sharded robustness runner and the scripts that regenerate every figure from recorded artifacts \(claim\-to\-command mapping in the repository’s provenance file\)\. The base model is the publicly available Gemma\-3\-1B\(Gemma Team, Google DeepMind,[2025](https://arxiv.org/html/2607.27539#bib.bib3)\), used under its license\. The hybrid study runs inference\-only on the released88\-bit Kimi Linear weights under MLX on the same machine; its synthetic\-record variants of both headline measurements reproduce without credentialed data, and the MIMIC runners take an explicit local path and emit the aggregate\-only JSON reports the tables are read from\. The introduction’s amendment scenario is itself a single command \(python \-m kimi\_sv\.amendment\_demo\), which prints the transcript of Appendix[G](https://arxiv.org/html/2607.27539#A7)and its bitwise audit\. Source and reproducibility materials accompany this preprint\.

## References

- Anonymous \(2025\)Unlearned but not forgotten: data extraction after exact unlearning in LLM\.InOpenReview preprint, forum BpAx3OuNOr,External Links:[Link](https://openreview.net/forum?id=BpAx3OuNOr)Cited by:[§1](https://arxiv.org/html/2607.27539#S1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2607.27539#S1.p2.1),[§2](https://arxiv.org/html/2607.27539#S2.p2.1),[2nd item](https://arxiv.org/html/2607.27539#S9.I1.i2.p1.1)\.
- A\. Behrouz, P\. Zhong, and V\. Mirrokni \(2025\)Titans: learning to memorize at test time\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2501.00663)Cited by:[§2](https://arxiv.org/html/2607.27539#S2.p4.1)\.
- Y\. Bisk, R\. Zellers, R\. Le Bras, J\. Gao, and Y\. Choi \(2020\)PIQA: reasoning about physical commonsense in natural language\.InAAAI Conference on Artificial Intelligence,External Links:[Link](https://arxiv.org/abs/1911.11641)Cited by:[§4](https://arxiv.org/html/2607.27539#S4.p1.6)\.
- L\. Bourtoule, V\. Chandrasekaran, C\. A\. Choquette\-Choo, H\. Jia, A\. Travers, B\. Zhang, D\. Lie, and N\. Papernot \(2021\)Machine unlearning\.InIEEE Symposium on Security and Privacy \(S&P\),External Links:[Link](https://arxiv.org/abs/1912.03817)Cited by:[§2](https://arxiv.org/html/2607.27539#S2.p1.1)\.
- Y\. Cao and J\. Yang \(2015\)Towards making systems forget with machine unlearning\.InIEEE Symposium on Security and Privacy \(S&P\),External Links:[Link](https://doi.org/10.1109/SP.2015.35)Cited by:[§2](https://arxiv.org/html/2607.27539#S2.p1.1)\.
- G\. Cauwenberghs and T\. Poggio \(2000\)Incremental and decremental support vector machine learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://papers.nips.cc/paper_files/paper/2000/hash/155fa09596c7e18e50b58eb7e0c6ccb4-Abstract.html)Cited by:[§1](https://arxiv.org/html/2607.27539#S1.p5.2),[§3\.3](https://arxiv.org/html/2607.27539#S3.SS3.p1.9),[§3\.6](https://arxiv.org/html/2607.27539#S3.SS6.p1.1)\.
- Y\. Chen, X\. Zhang, S\. Hu, X\. Han, Z\. Liu, and M\. Sun \(2024\)Stuffed mamba: oversized states lead to the inability to forget\.arXiv preprint arXiv:2410\.07145\.External Links:[Link](https://arxiv.org/abs/2410.07145)Cited by:[§2](https://arxiv.org/html/2607.27539#S2.p4.1)\.
- DeepSeek\-AI \(2024\)DeepSeek\-V2: a strong, economical, and efficient mixture\-of\-experts language model\.arXiv preprint arXiv:2405\.04434\.External Links:[Link](https://arxiv.org/abs/2405.04434)Cited by:[§7](https://arxiv.org/html/2607.27539#S7.p1.3)\.
- L\. Gao, J\. Tow, B\. Abbasi, S\. Biderman,et al\.\(2024\)A framework for few\-shot language model evaluation\.Note:EleutherAIlm\-evaluation\-harnessExternal Links:[Link](https://github.com/EleutherAI/lm-evaluation-harness)Cited by:[§4](https://arxiv.org/html/2607.27539#S4.p1.6)\.
- Gemma Team, Google DeepMind \(2025\)Gemma 3 technical report\.arXiv preprint arXiv:2503\.19786\.External Links:[Link](https://arxiv.org/abs/2503.19786)Cited by:[§1](https://arxiv.org/html/2607.27539#S1.p5.2),[§3\.4](https://arxiv.org/html/2607.27539#S3.SS4.p1.5),[Reproducibility Statement](https://arxiv.org/html/2607.27539#Sx2.p1.2)\.
- A\. Gu and T\. Dao \(2024\)Mamba: linear\-time sequence modeling with selective state spaces\.InConference on Language Modeling \(COLM\),External Links:[Link](https://arxiv.org/abs/2312.00752)Cited by:[§2](https://arxiv.org/html/2607.27539#S2.p4.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2106.09685)Cited by:[§3\.5](https://arxiv.org/html/2607.27539#S3.SS5.p1.5)\.
- S\. Hu, Y\. Fu, Z\. S\. Wu, and V\. Smith \(2024\)Jogging the memory of unlearned LLMs through targeted relearning attacks\.arXiv preprint arXiv:2406\.13356\.External Links:[Link](https://arxiv.org/abs/2406.13356)Cited by:[§1](https://arxiv.org/html/2607.27539#S1.p2.1),[§2](https://arxiv.org/html/2607.27539#S2.p2.1)\.
- A\. Katharopoulos, A\. Vyas, N\. Pappas, and F\. Fleuret \(2020\)Transformers are RNNs: fast autoregressive transformers with linear attention\.InInternational Conference on Machine Learning \(ICML\),External Links:[Link](https://arxiv.org/abs/2006.16236)Cited by:[§2](https://arxiv.org/html/2607.27539#S2.p4.1)\.
- Kimi Team \(2025\)Kimi linear: an expressive, efficient attention architecture\.arXiv preprint arXiv:2510\.26692\.External Links:[Link](https://arxiv.org/abs/2510.26692)Cited by:[§2](https://arxiv.org/html/2607.27539#S2.p4.1),[§7](https://arxiv.org/html/2607.27539#S7.p1.3)\.
- S\. Laguna, J\. da Silva Gonçalves, M\. Vandenhirtz, A\. Ryser, I\. Cannistraci, and J\. E\. Vogt \(2026\)Rethinking machine unlearning: models designed to forget via key deletion\.InInternational Conference on Learning Representations \(ICLR\),Note:Oral presentationExternal Links:[Link](https://openreview.net/forum?id=IjJUrgd5cS)Cited by:[§1](https://arxiv.org/html/2607.27539#S1.p3.1),[§2](https://arxiv.org/html/2607.27539#S2.p1.1),[§3\.2](https://arxiv.org/html/2607.27539#S3.SS2.p1.2),[§8](https://arxiv.org/html/2607.27539#S8.p1.1)\.
- P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel, S\. Riedel, and D\. Kiela \(2020\)Retrieval\-augmented generation for knowledge\-intensive NLP tasks\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2005.11401)Cited by:[§5](https://arxiv.org/html/2607.27539#S5.SS0.SSS0.Px1.p1.7)\.
- M\. Li, S\. Liu, D\. Fu, H\. P\. Wang, Y\. Xia, H\. Li, H\. Yan, and P\. Li \(2026\)KVEraser: learning to steer KV cache for efficient localized context erasing\.arXiv preprint arXiv:2606\.17034\.External Links:[Link](https://arxiv.org/abs/2606.17034)Cited by:[§2](https://arxiv.org/html/2607.27539#S2.p3.1)\.
- N\. Li, A\. Pan, A\. Gopal, S\. Yue, D\. Berrios,et al\.\(2024a\)The WMDP benchmark: measuring and reducing malicious use with unlearning\.InInternational Conference on Machine Learning \(ICML\),External Links:[Link](https://arxiv.org/abs/2403.03218)Cited by:[§1](https://arxiv.org/html/2607.27539#S1.p2.1),[§2](https://arxiv.org/html/2607.27539#S2.p2.1)\.
- Y\. Li, Y\. Huang, B\. Yang, B\. Venkitesh, A\. Locatelli, H\. Ye, T\. Cai, P\. Lewis, and D\. Chen \(2024b\)SnapKV: LLM knows what you are looking for before generation\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2404.14469)Cited by:[§2](https://arxiv.org/html/2607.27539#S2.p3.1)\.
- O\. Lieber, B\. Lenz, H\. Bata, G\. Cohen, J\. Osin, I\. Dalmedigos, E\. Safahi, S\. Meirom, Y\. Belinkov, S\. Shalev\-Shwartz,et al\.\(2024\)Jamba: a hybrid transformer\-mamba language model\.arXiv preprint arXiv:2403\.19887\.External Links:[Link](https://arxiv.org/abs/2403.19887)Cited by:[§2](https://arxiv.org/html/2607.27539#S2.p4.1)\.
- A\. Lynch, P\. Guo, A\. Ewart, S\. Casper, and D\. Hadfield\-Menell \(2024\)Eight methods to evaluate robust unlearning in LLMs\.arXiv preprint arXiv:2402\.16835\.External Links:[Link](https://arxiv.org/abs/2402.16835)Cited by:[§1](https://arxiv.org/html/2607.27539#S1.p2.1),[§2](https://arxiv.org/html/2607.27539#S2.p2.1)\.
- P\. Maini, Z\. Feng, A\. Schwarzschild, Z\. C\. Lipton, and J\. Z\. Kolter \(2024\)TOFU: a task of fictitious unlearning for LLMs\.arXiv preprint arXiv:2401\.06121\.External Links:[Link](https://arxiv.org/abs/2401.06121)Cited by:[§1](https://arxiv.org/html/2607.27539#S1.p2.1),[§2](https://arxiv.org/html/2607.27539#S2.p2.1),[§5](https://arxiv.org/html/2607.27539#S5.SS0.SSS0.Px1.p1.7),[§5](https://arxiv.org/html/2607.27539#S5.SS0.SSS0.Px1.p2.9),[Reproducibility Statement](https://arxiv.org/html/2607.27539#Sx2.p1.2)\.
- S\. Merity, C\. Xiong, J\. Bradbury, and R\. Socher \(2017\)Pointer sentinel mixture models\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/1609.07843)Cited by:[Reproducibility Statement](https://arxiv.org/html/2607.27539#Sx2.p1.2)\.
- M\. Pawelczyk, S\. Neel, and H\. Lakkaraju \(2024\)In\-context unlearning: language models as few\-shot unlearners\.InInternational Conference on Machine Learning \(ICML\),External Links:[Link](https://arxiv.org/abs/2310.07579)Cited by:[§1](https://arxiv.org/html/2607.27539#S1.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2607.27539#S2.p2.1),[§5](https://arxiv.org/html/2607.27539#S5.SS0.SSS0.Px1.p2.9)\.
- G\. Penedo, H\. Kydlíček, A\. Lozhkov, M\. Mitchell, C\. Raffel, L\. Von Werra, and T\. Wolf \(2024\)The FineWeb datasets: decanting the web for the finest text data at scale\.InAdvances in Neural Information Processing Systems \(NeurIPS\) Datasets and Benchmarks,External Links:[Link](https://arxiv.org/abs/2406.17557)Cited by:[§3\.5](https://arxiv.org/html/2607.27539#S3.SS5.p1.5),[Reproducibility Statement](https://arxiv.org/html/2607.27539#Sx2.p1.2)\.
- V\. Ramesh \(2026\)A trainable support\-vector memory with certified selection and exact unlearning\.Note:Companion preprintCited by:[§1](https://arxiv.org/html/2607.27539#S1.p5.2),[§3\.3](https://arxiv.org/html/2607.27539#S3.SS3.p1.9)\.
- H\. Reisizadeh, J\. Ruan, Y\. Chen, S\. Pal, S\. Liu, and M\. Hong \(2025\)Leak@kk: unlearning does not make LLMs forget under probabilistic decoding\.arXiv preprint arXiv:2511\.04934\.Cited by:[§2](https://arxiv.org/html/2607.27539#S2.p2.1),[§5](https://arxiv.org/html/2607.27539#S5.SS0.SSS0.Px6.p1.11)\.
- K\. Sakaguchi, R\. Le Bras, C\. Bhagavatula, and Y\. Choi \(2021\)WinoGrande: an adversarial winograd schema challenge at scale\.Communications of the ACM\.External Links:[Link](https://arxiv.org/abs/1907.10641)Cited by:[§4](https://arxiv.org/html/2607.27539#S4.p1.6)\.
- W\. Shi, J\. Lee, Y\. Huang, S\. Malladi, J\. Zhao, A\. Holtzman, D\. Liu, L\. Zettlemoyer, N\. A\. Smith, and C\. Zhang \(2025\)MUSE: machine unlearning six\-way evaluation for language models\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2407.06460)Cited by:[§1](https://arxiv.org/html/2607.27539#S1.p2.1),[§2](https://arxiv.org/html/2607.27539#S2.p2.1)\.
- D\. M\. J\. Tax and R\. P\. W\. Duin \(2004\)Support vector data description\.Machine Learning54\(1\),pp\. 45–66\.External Links:[Link](https://doi.org/10.1023/B:MACH.0000008084.60811.49)Cited by:[§3\.3](https://arxiv.org/html/2607.27539#S3.SS3.p1.9)\.
- P\. Thaker, S\. Hu, N\. Kale, Y\. Maurya, Z\. S\. Wu, and V\. Smith \(2024\)Position: LLM unlearning benchmarks are weak measures of progress\.arXiv preprint arXiv:2410\.02879\.External Links:[Link](https://arxiv.org/abs/2410.02879)Cited by:[§1](https://arxiv.org/html/2607.27539#S1.p2.1),[§2](https://arxiv.org/html/2607.27539#S2.p2.1)\.
- J\. Wang, D\. Paliotta, A\. May, A\. M\. Rush, and T\. Dao \(2024\)The mamba in the llama: distilling and accelerating hybrid models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2408.15237)Cited by:[§2](https://arxiv.org/html/2607.27539#S2.p3.1)\.
- X\. Wanget al\.\(2025\)Unlearning isn’t deletion: investigating reversibility of machine unlearning in LLMs\.arXiv preprint arXiv:2505\.16831\.External Links:[Link](https://arxiv.org/abs/2505.16831)Cited by:[§1](https://arxiv.org/html/2607.27539#S1.p2.1),[§2](https://arxiv.org/html/2607.27539#S2.p2.1)\.
- A\. X \(2025\)Unlearning at scale: implementing the right to be forgotten in large language models\.arXiv preprint arXiv:2508\.12220\.External Links:[Link](https://arxiv.org/abs/2508.12220)Cited by:[§2](https://arxiv.org/html/2607.27539#S2.p4.1)\.
- S\. Yang, J\. Kautz, and A\. Hatamizadeh \(2025\)Gated delta networks: improving mamba2 with delta rule\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2412.06464)Cited by:[§2](https://arxiv.org/html/2607.27539#S2.p4.1),[§7](https://arxiv.org/html/2607.27539#S7.p1.3)\.
- S\. Yang, B\. Wang, Y\. Zhang, Y\. Shen, and Y\. Kim \(2024\)Parallelizing linear transformers with the delta rule over sequence length\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2406.06484)Cited by:[§2](https://arxiv.org/html/2607.27539#S2.p4.1),[§7](https://arxiv.org/html/2607.27539#S7.p1.3)\.
- J\. Yao, S\. Shen, K\. Du, S\. Feng, D\. Seo, R\. Zhang, Y\. Huang, Y\. Liu, S\. Lu, and J\. Jiang \(2026\)VeriCache: turning lossy KV cache into lossless LLM inference\.arXiv preprint arXiv:2605\.17613\.External Links:[Link](https://arxiv.org/abs/2605.17613)Cited by:[§2](https://arxiv.org/html/2607.27539#S2.p3.1)\.
- R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi \(2019\)HellaSwag: can a machine really finish your sentence?\.InAnnual Meeting of the Association for Computational Linguistics \(ACL\),External Links:[Link](https://arxiv.org/abs/1905.07830)Cited by:[§4](https://arxiv.org/html/2607.27539#S4.p1.6)\.
- M\. Zhang, S\. Arora, R\. Chalamala, A\. Wu, B\. Spector, A\. Singhal, K\. Ramesh, and C\. Ré \(2024\)LoLCATs: on low\-rank linearizing of large language models\.arXiv preprint arXiv:2410\.10254\.External Links:[Link](https://arxiv.org/abs/2410.10254)Cited by:[§2](https://arxiv.org/html/2607.27539#S2.p3.1)\.
- Z\. Zhang, Y\. Sheng, T\. Zhou, T\. Chen, L\. Zheng, R\. Cai, Z\. Song, Y\. Tian, C\. Ré, C\. Barrett, Z\. Wang, and B\. Chen \(2023\)H2O: heavy\-hitter oracle for efficient generative inference of large language models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2306.14048)Cited by:[§2](https://arxiv.org/html/2607.27539#S2.p3.1)\.

## Appendix AExperimental details

![Refer to caption](https://arxiv.org/html/2607.27539v1/figs/method_schematic.png)Figure 2:*Where the addressable memory lives, and how a fact leaves it\.**\(a\)*Gemma 3 interleaves five local layers per global layer; only the global layers are replaced\.*\(b\)*The support\-vector fit assigns exact zeros and supports a reverse incremental update\.*\(c\)*The decremented solution drives the live model in double precision and is compared with a retained\-key refit at the next\-token output\.Model and graft\.The base model isgoogle/gemma\-3\-1b\-pt:2626decoder layers, four of them global at indices5,11,17,235,11,17,23under the5:15\{:\}1interleave\. We graft the support\-vector gate onto the four global layers only, reusing each layer’s query/key/value/output projections, query–key normalization, rotary embeddings, and grouped\-query configuration; rotary embeddings are applied to the keys before the kernel\. The gate budget isν=0\.3\\nu=0\.3\(boxC=1/\(ν​n\)C=1/\(\\nu n\)\), the chunk length for the chunk\-frozen causal readout is128128, and the radial\-kernel bandwidth is initialized at each layer’s median key distance\.

Recovery\.Stage 1 \(attention transfer\) trains one log\-space bandwidth parameter per global layer \(44total\) for2,0002\{,\}000steps at learning rate0\.020\.02, minimizing the mean\-squared error between the grafted and original global\-attention outputs\. Stage 2 applies LoRA \(rank88\) to the query/key/value/output projections of all layers including the gate’s wrapped base, training the language\-model cross\-entropy for6,0006\{,\}000steps at learning rate0\.0010\.001\(1\.491\.49M trainable parameters\)\. Training data is FineWeb\-Edu \(sample\-10BT\),3333M tokens, batch88, sequence length512512, seed0; the run takes roughly six hours on one Apple M3 Ultra \(MPS\)\. The matched control repeats stage 2 on the ungrafted model with the identical budget and seed\.

Evaluation\.Perplexity is WikiText\-103 test,400400blocks \(we default to a large eval set because2020versus3030blocks already swing the estimate\)\. Zero\-shot tasks use the standardlm\-evalharness at2,0002\{,\}000examples per task\. Output\-KL certificates use support positions from a fixed192192\-token generic prompt and route through the float64 incremental decrement and double\-precision model\. Behavioral forgetting evaluations use TOFU forget10/retain90 facts packed beyond the local window; “recovery” is normalized to a budget\-matched never\-ingested floor\. Efficacy, elicitation, relearning, sampling, and LiRA instead exclude target positions from a single\-precision FISTA gate and refit it; we call this the*masked\-refit proxy*and do not use it for an exactness claim\. Decay scales the target’s readout weight byγ=0\.01\\gamma=0\.01; ICUL is a prompt\-prefix forget instruction\. Membership inference is the full LiRA protocol in its in\-context form: shadow*contexts*\(random filler packings,3232per side per target\) fit per\-target Gaussians, held\-out draws are scored by the likelihood ratio, and we report AUC and TPR at low FPR; there is no shadow\-model training because training is not the ingestion mechanism in\-context\. The weight\-space baseline is gradient ascent on the target’s answer tokens given the packed memory, through a fresh rank\-88LoRA \(lr10−410^\{\-4\}, at most6060steps\), early\-stopped when the target’s recall reaches the never\-ingested floor; its relearning attack reuses the PrivUn protocol \(m=64m=64benign read\-from\-memory samples, lr10−310^\{\-3\}\)\.

## Appendix BFull 1B utility results

![Refer to caption](https://arxiv.org/html/2607.27539v1/figs/utility.png)Figure 3:*At 1B, the model with a deletable memory is nearly the model without one\.*Against an identically fine\-tuned control:\+2\.0%\+2\.0\\%WikiText perplexity,−0\.11\-0\.11percentage\-point mean zero\-shot accuracy, and\+0\.9%\+0\.9\\%mean perplexity overhead across three corpora\. Larger\-model costs are measured separately in Section[6](https://arxiv.org/html/2607.27539#S6)\.Table 3:*Each recovery stage, and what the gate itself costs\.*Utility ladder on Gemma\-3\-1B \(WikiText\-103 test perplexity,400400blocks; lower is better\)\. The final two rows isolate the gate’s residual\+2\.0%\+2\.0\\%cost from the shared fine\-tuning gain\.The attention\-transfer objective matches the original global\-attention output in mean\-squared error, not language\-model loss\. Its trained bandwidths do not lower perplexity by themselves, and the completed warm\-start ablation shows they are not load\-bearing for the final result either: LoRA recovery without stage 1 \(median\-bandwidth heuristic, identical stage\-2 token stream via an aligned data offset, same seed and budget\) reaches21\.1321\.13versus21\.1821\.18with it—a wash within evaluation noise\. The33\.29→21\.1833\.29\\to 21\.18recovery is carried entirely by the low\-rank stage\. This also closes a latent consistency question: the released adapter does not persist trained bandwidths, so every downstream evaluation in this paper already ran with the median heuristic, which the ablation now shows to be utility\-equivalent\. Attention transfer is kept in the recipe only as an optional bandwidth initializer; no claim depends on it\.

Table 4:*The grafted model keeps its zero\-shot abilities\.*Recovered model versus matched control \(lm\-eval,2,0002\{,\}000examples/task\); deltas are percentage points\.Table 5:*The gate’s cost is not a one\-corpus artifact\.*Perplexity across three domains \(recovered model versus matched control, eval\-only,300300blocks/corpus\)\.The 1B parity claim remains scoped to one LoRA budget, fixed grafted layers,ν\\nu, and bandwidth initialization\. Section[6](https://arxiv.org/html/2607.27539#S6)measures rather than extrapolates the larger\-model costs\.

## Appendix CScaling diagnostics

4B certificate tail\.Over3131support\-token deletions, the output KL to retained\-key refit has median4\.8×10−154\.8\\times 10^\{\-15\}against a4\.9×10−74\.9\\times 10^\{\-7\}decay median\. The exact worst case is5\.7×10−85\.7\\times 10^\{\-8\}and the mean is1\.8×10−91\.8\\times 10^\{\-9\}, showing that one or two ill\-conditioned targets create a tail rather than a broad shift\. Sequential deletion stays between10−1510^\{\-15\}and10−710^\{\-7\}throughk=5k=5, while decay sits at10−610^\{\-6\}–10−510^\{\-5\}throughout\.

12B numerical floor\.Over3131support\-token deletions, exact median KL is6\.3×10−96\.3\\times 10^\{\-9\}\(worst4\.5×10−74\.5\\times 10^\{\-7\}\) against a2\.7×10−62\.7\\times 10^\{\-6\}decay median\. The worst exact case exceeds decay’s best7\.8×10−87\.8\\times 10^\{\-8\}case, so the edges overlap\. On a diverse, non\-repetitive prompt, exact remains at median3\.9×10−93\.9\\times 10^\{\-9\}\(worst1\.8×10−71\.8\\times 10^\{\-7\}\) against a2\.4×10−52\.4\\times 10^\{\-5\}decay median\. Per\-layer diagnostics show elevated deviation across all eight global layers\. Sequential exact KL grows from1\.7×10−91\.7\\times 10^\{\-9\}atk=1k=1to2\.9×10−62\.9\\times 10^\{\-6\}atk=4k=4, while decay reaches9\.9×10−59\.9\\times 10^\{\-5\}\.

The decrement is refit\-equivalent in exact arithmetic; the measured gap is conditioned by the number and width of global\-layer kernel systems\. “Exact” at 12B therefore carries a measured numerical floor rather than a value that can be rounded to zero\.

Recovery\-capacity checks\.At 4B, doubling LoRA rank from 8 to 16 shifts the recovered and matched\-control perplexities together \(15\.81→18\.9415\.81\\to 18\.94and14\.22→17\.0814\.22\\to 17\.08\), leaving the gate gap at10\.9%10\.9\\%\. A rank\-32 run destabilizes at the 1B learning rate and is not interpreted as a capacity result\. At 12B, the rank\-16 control trains stably to19\.0219\.02, while the grafted run destabilizes at61\.6361\.63\. This isolates an optimization failure through the gate, but leaves open whether scale\-specific optimization can recover part of the measured44\.3%44\.3\\%rank\-8 gap\.

## Appendix DForgetting protocols and full results

![Refer to caption](https://arxiv.org/html/2607.27539v1/figs/forgetting_axis.png)Figure 4:*Behavior cannot tell masked refit from strong suppression; the separate certificate can \(Gemma\-3\-1B\)\.**\(a,b,d,f\)*The single\-precision masked\-refit proxy stays near the never\-ingested floor under the evaluated behavioral tests\.*\(c,e\)*The float64 decrement stays near10−1410^\{\-14\}KL to its retained\-key refit while decay deviates and accumulates\. Masked refit and aggressive decay overlap on coarse behavior but the certificate paths separate\.Normalization and admission\.Recovery is normalized to a budget\-matched never\-ingested floor:0denotes the floor and11denotes original recall\. Values can fall slightly outside this interval\. Each panel conditions on measurable pre\-deletion recall and reports its own denominator rather than forcing a common subset\.

Interventions\.Certificate panels use the float64 decrement\. Behavioral panels use the single\-precision masked\-refit proxy\. Decay multiplies target readout weights byγ=0\.01\\gamma=0\.01; ICUL places a forget instruction over unchanged memory; GA performs per\-target gradient ascent through a fresh rank\-88adapter\.

Certificate and sequence\.On3131support\-token deletions, decrement output KL to retained\-key refit has median5\.4×10−155\.4\\times 10^\{\-15\}and worst9\.3×10−149\.3\\times 10^\{\-14\}\. Decay median is1\.8×10−61\.8\\times 10^\{\-6\}, and its best target remains three orders above exact’s worst\. A deletion on the unadapted graft is1\.6×10−141\.6\\times 10^\{\-14\}, so the property survives recovery training\. Overk=1,2,5,10,20,30k=1,2,5,10,20,30sequential deletions, exact stays between10−1510^\{\-15\}and1\.5×10−141\.5\\times 10^\{\-14\}; decay rises from6\.7×10−66\.7\\times 10^\{\-6\}to4\.7×10−34\.7\\times 10^\{\-3\}\.

Elicitation and relearning\.The in\-context attack prependsk∈\{0,1,2,4,8\}k\\in\\\{0,1,2,4,8\\\}target\-free hints and re\-queries6060targets\. Masked refit moves from0\.000\.00to0\.020\.02normalized recovery; ICUL remains extractable \(0\.95→0\.560\.95\\to 0\.56\)\. The relearning attack LoRA\-fine\-tunes onm∈\{0,4,16,64,256\}m\\in\\\{0,4,16,64,256\\\}related but target\-free samples over approximately4848admitted targets\. Masked refit moves0\.01→−0\.100\.01\\to\-0\.10; ICUL remains1\.14→1\.231\.14\\to 1\.23\.

Weight editing\.GA reaches the floor in a mean of nine ascent steps over1919admitted targets, with residual−0\.33±0\.12\-0\.33\\pm 0\.12\. Retain extraction moves−0\.125±0\.030\-0\.125\\pm 0\.030, versus approximately zero for masked refit\. Benign relearning then restores and overshoots the target \(\+3\.5±2\.2\+3\.5\\pm 2\.2,n=10n=10\), consistent with damage to the shared reader rather than deletion of contextual state\.

Decrement fallbacks and cost\.The reverse C&P decrement can hit a margin\-set edge case, in which case that boundary’s gate falls back to a full float64 refit: the result is unchanged—the fallback*is*the refit\-without\-target, so exactness is unaffected—and only cost grows\. Recorded artifacts show the observed rates: the span\-scale hero deletions \(4646and5656positions\) each used6464boundary solves with44fallbacks, and the released demo deletions recorded66\(4B patient field; certificate wall\-clock265265s end to end\) and1212\(1B whole\-record registered probe;122122s\) fallbacks, each disclosed in the demo interface alongside the certified KL\. Certificates in this paper always report the realized KL of the executed path, so fallbacks cannot silently degrade a claim\.

LiRA\.For each target,3232random shadow contexts per side fit per\-target Gaussian extraction\-score distributions; eight held\-out draws yield312312tests per side over3939admitted targets\. Masked refit reaches AUC0\.4990\.499and TPR1\.3%1\.3\\%at1%1\\%FPR\. Controls are present0\.9960\.996AUC \(92%92\\%TPR\), ICUL0\.9890\.989, and decay0\.5330\.533\. A pooled\-score variant gives AUC0\.5090\.509\. Shadow\-model training is not used because gradient training is not the in\-context ingestion mechanism\.

## Appendix EProbabilistic Leak@k and entangled pairs

![Refer to caption](https://arxiv.org/html/2607.27539v1/figs/robust_leak.png)Figure 5:*Repeated sampling finds no detectable excess leakage after masked refit, but defeats the prompt\-based method \(Gemma\-3\-1B\)\.**\(a\)*Exact\-phrase Leak@kkunder the stem probe \(2020targets,95%95\\%bootstrap bands\): masked refit \(blue\) is statistically indistinguishable from the never\-stored floor \(green\) at every budget; ICUL \(orange\) tracks present \(gray\)—repeated sampling defeats the instruction\. The floor rises withkkbecause some secrets are guessable from question and template; the paired per\-target differences are the tight statistic \(\+0\.086​\[−0\.013,\+0\.217\]\+0\.086\\;\[\-0\.013,\+0\.217\]for masked refit atk=128k=128;\+0\.102​\[\+0\.037,\+0\.185\]\+0\.102\\;\[\+0\.037,\+0\.185\]for ICUL already atk=1k=1\)\.*\(b\)*Teacher\-forced secret lift over the floor, per target \(dots; colored bars single facts, gray bars entangled pairs\): deletion removes the stored lift \(\+1\.50→\+0\.04\+1\.50\\to\+0\.04nats\); ICUL leaves it \(\+1\.49\+1\.49\)\.Protocol\.Each fact is a TOFU forget10 question–answer pair packed once into the grafted memory beyond the local window, buried under at least2222retain90 filler notes \(∼1,000\{\\sim\}1\{,\}000memory tokens\)\. Five conditions share each target:*present*,*masked refit*\(stored asdecrementin artifacts; applied via drop positions\),*decay*\(γ=0\.01\\gamma=0\.01\),*ICUL*\(a retraction instruction over unchanged memory\), and*never*\(a token\-budget\-matched memory that never contained the fact\)\. Per condition we draw200200samples at temperature11, top\-p=1p=1, up to9696new tokens with sentence stopping, and score exact\-phrase exposure of the secret span\. Leak@kkis the unbiased without\-replacement expected\-maximum U\-statistic over the200200scores\. Sampling uses cached prefill\-and\-decode whose per\-step logits match full recomputation \(greedy texts and argmax agree at every step; the chunk\-frozen gate is solved once per crossed boundary\)\.

Probe design and the naive null\.A naive probe \(question alone\) has no power: on1212targets×\\times200200samples, exact\-phrase exposure was0/2000/200in*every*condition including present—the model emits the answer template but resamples the secret—so ROUGE\-style overlap measures template style, not content \(outputs/gemma\_sv\_eval/robust\_naive\_probe\.json\)\. The evaluated attacker therefore knows the answer template: the probe is the question plus the answer prefix up to the secret span, defined automatically as the longest contiguous run of answer words absent from the question \(function words and single characters excluded\)\.

Admission\.A target is admitted when the full\-answer teacher\-forced lift and the secret\-span lift are both at least0\.050\.05nats and the secret’s first token ranks in the model’s top1010under the stem probe with the fact present\. Gates passed for24/4024/40scanned facts and28/14028/140scanned pairs; the first2020of each were evaluated\. Rejections are dominated by weakly stored proper names and by secrets guessable from the question alone\.

Entangled pairs\.Each TOFU\-Pair prompt asks a forget10 question and a retain90 question together; both records are stored in one memory and only the forget record is deleted\. The graft shows write interference against the most recently written record \(the retained record’s secret rank collapses from∼11\{\\sim\}11to∼600\{\\sim\}600when written second\), so the protocol stores the retained record first; the paired admission gate additionally requires the retained secret to be extractable\. Deleting the forget record leaves its secret at the never floor \(−0\.03​\[−0\.10,\+0\.03\]\-0\.03\\;\[\-0\.10,\+0\.03\]nats over2020pairs\) and leaves the retained secret’s teacher\-forced rank unchanged \(median3→23\\to 2, worst99\); the retained secret’s sampled accessibility in fact improves slightly \(\+0\.18​\[\+0\.04,\+0\.37\]\+0\.18\\;\[\+0\.04,\+0\.37\]nats\), consistent with removing the interfering neighbor\.

Full leak@k table\.Mean exact\-phrase Leak@kkover2020targets with95%95\\%percentile\-bootstrap intervals \(10410^\{4\}draws\), fromoutputs/gemma\_sv\_eval/robust\_summary\.json:

The floor rises withkkbecause several secrets are guessable from the question and template; the tight statistic is the per\-target paired difference against never on shared targets: masked refit−0\.000​\[−0\.011,\+0\.007\]\-0\.000\\;\[\-0\.011,\+0\.007\]atk=1k=1and\+0\.086​\[−0\.013,\+0\.217\]\+0\.086\\;\[\-0\.013,\+0\.217\]atk=128k=128\(consistent with zero\), versus ICUL\+0\.102​\[\+0\.037,\+0\.185\]\+0\.102\\;\[\+0\.037,\+0\.185\]and\+0\.175​\[\+0\.050,\+0\.326\]\+0\.175\\;\[\+0\.050,\+0\.326\]\.

Reading thek≥64k\{\\geq\}64tail\.Atk=128k=128withn=200n=200the expected\-maximum estimator is nearly binary per target \(“did any sample contain the secret”\), so single targets move the mean by±1/20\{\\pm\}1/20and the apparent ordering of masked refit, decay, and never is noise: masked refit’s\+0\.086\+0\.086comes from two targets, while decay’s\+0\.015\+0\.015is the same\-sized positive swings cancelled by two negative ones, including a target where decay lands a full−1\.0\-1\.0*below*the floor\. Neither difference is significant, and on the measurements with power the expected ordering holds: decay’s teacher\-forced residual \(\+0\.136​\[−0\.033,\+0\.327\]\+0\.136\\;\[\-0\.033,\+0\.327\]nats\) is∼4×\{\\sim\}4\\timesmasked refit’s \(\+0\.037​\[−0\.103,\+0\.172\]\+0\.037\\;\[\-0\.103,\+0\.172\]\), and the certificate separates them by nine orders of magnitude \(Section[5](https://arxiv.org/html/2607.27539#S5)\)—behavioral saturation between masked refit and decay is precisely why the paper reads certificate\-first\. The one systematic tail case is a target whose secret leaks at highkkunder*both*masked refit and decay against a0\.0000\.000floor with a\+0\.27\+0\.27\-nat teacher\-forced residual; its secret is guessable in\-context \(an identical secret elsewhere has a0\.640\.64floor\), consistent with the disclosed ingestion\-time imprint on neighboring representations that no cache\-side deletion touches and that the full repack removes\.

Seed replication\.The full leak grid was rerun with two additional sampling seeds \(identical targets, admission gates, and protocol; admission is deterministic and admitted20/2020/20in all three\)\. Headline values replicate: atk=1k\{=\}1, present\.124/\.123/\.123\.124/\.123/\.123, masked refit\.009/\.008/\.008\.009/\.008/\.008, ICUL\.111/\.112/\.106\.111/\.112/\.106, and never\.009/\.013/\.010\.009/\.013/\.010across seeds0/1/20/1/2\. Atk=128k\{=\}128the masked\-refit\-minus\-never gap is\+0\.087/−0\.046/\+0\.013\+0\.087/\-0\.046/\+0\.013across seeds—it flips sign—confirming the tail is sampling variance while everyk≤32k\\leq 32separation is stable\.

Reproduction\.gemma\_sv/reproducibility/run\_robust\_m3\.sh \-\-paperwith the admitted index lists produces one JSON shard per target–condition \(setSEEDfor replications\);gemma\_sv\.merge\_robust\_shardsrecomputes the aggregates andgemma\_sv\.make\_robust\_figurerenders Figure[5](https://arxiv.org/html/2607.27539#A5.F5)and the bootstrap summary\.

## Appendix FWhole\-record deletion

![Refer to caption](https://arxiv.org/html/2607.27539v1/x2.png)Figure 6:*One complete record deleted live, and audited at every step \(the released demo\)\.**\(a\)*A recorded conversation: a three\-field record is indexed, recalled, then deleted atomically—all8080of its positions leave every gate—while its neighbor record stays answerable\. The coupled block decrement completes incrementally on276/320276/320affected head\-gates; the remainder use the disclosed exact refit fallback \(max output KL1\.3×10−111\.3\\times 10^\{\-11\}\)\.*\(b–d\)*Behavioral readbacks use the single\-precision masked\-refit path\. Every field is statistically consistent with its never\-stored floor, and200200samples show no detectable excess leakage over a record the model never saw \(\.038\.038vs\.\.024\.024at one draw; present\.545\.545; prompt baseline\.420\.420\)\.Definition and manifest\.A record is a contiguous bundle of three question–answer fields; “whole record” means every token owned by both stored copies—field names, identity\-bearing question text, and values—not merely the answer span used by an audit probe\. One unrelated retained record shares the same persistent memory\. The versionedwhole\_record\_synthetic\_v1\.jsonmanifest fixes eight records before execution \(medical, incident response, payroll, legal, research, logistics, education, and dossier domains\); no rejected case is replaced\. Admission requires the full answer and every field secret to lift over the never\-stored baseline, each field’s first token to rank at most1010, the retained neighbor to be extractable, and every affected fixed\-CCboundary to satisfynretained​C≥1n\_\{\\mathrm\{retained\}\}C\\geq 1\. Six of eight records pass; both rejections fail one field’s rank gate\. Admitted deletions contain6464–8080token positions\.

Behavior\.For each admitted record, each of its three field stems is sampled1616times at temperature11under present, masked\-refit, decay, ICUL, and never\-stored conditions\. Exact\-phrase Leak@11is\.545\.545\[\.438,\.674\]\[\.438,\.674\]present,\.038\.038\[\.000,\.115\]\[\.000,\.115\]after masked refit,\.024\.024\[\.000,\.073\]\[\.000,\.073\]never stored, and\.420\.420\[\.326,\.514\]\[\.326,\.514\]under ICUL\. Atk=16k=16, masked refit and never are identical \(\.056\.056\), while ICUL reaches\.944\.944\. The paired per\-record masked\-refit\-minus\-never difference is zero within the bootstrap interval at every evaluatedkk\. Teacher\-forced stored signal falls from\+6\.41\+6\.41\[\+5\.07,\+7\.77\]\[\+5\.07,\+7\.77\]nats over never to\+0\.10\+0\.10\[−0\.15,\+0\.38\]\[\-0\.15,\+0\.38\]; decay leaves\+1\.79\+1\.79and ICUL\+6\.33\+6\.33\. The unrelated neighbor does not suffer collateral loss \(mean shift\+0\.43\+0\.43\[\+0\.12,\+0\.75\]\[\+0\.12,\+0\.75\]nats\)\. A generic “repeat the complete record” prompt has no power even when the record is present and is reported as a null, not used as evidence\. A second\-seed behavioral replication \(identical manifest and gates\) admits the same six records and replicates the headline: Leak@11\.490\.490present,\.045\.045masked\-refit,\.021\.021never,\.462\.462ICUL\.

Record\-scale certificate and fallback boundary\.The page\-one example is selected from the admitted set by a declared maximin rule \(maximize the weakest pre\-deletion field probability\), yielding the three\-field Zaffre record \(8080deleted positions\)\. Four float64 output probes \(three fields plus composite extraction\) match fixed\-CCretained\-key refit at max output KL1\.28×10−111\.28\\times 10^\{\-11\}\. A coupled block homotopy drives all record coefficients to zero together, rather than composing unstable one\-point paths: it passes post\-verification on276/320276/320affected head\-gates \(86\.2%86\.2\\%\); the4444margin\-empty or tolerance\-edge cases use the exact refit fallback\. The independent Helios cybersecurity record is stronger still:260/272260/272gates \(95\.6%95\.6\\%\) complete incrementally, with max output KL4\.03×10−124\.03\\times 10^\{\-12\}\. Thus whole\-record deletion is predominantly incremental and always functionally exact, while fallback frequency and cost remain explicit\. Porting the same gates to 4B admits4/84/8records once the record is placed after the first fixed\-CCboundary; median deletion residual is0\.090\.09nats over the never floor against3\.693\.69present\.

Ingestion\-imprint measurement and a packing negative result\.The retained\-key certificate deliberately excludes what ingestion did to retained keys\. We measure that imprint directly with a position\-matched control: the deleted state \(fixed\-CCrefit on the present memory\) is compared against the same refit on a memory whose record span held equal\-length neutral padding during ingestion, so every retained token keeps its position and content and the output gap isolates the record’s ingestion\-time influence\. Across the four hero probes the gap is2\.5×10−42\.5\\times 10^\{\-4\}–1\.2×10−31\.2\\times 10^\{\-3\}nats \(Helios:2\.0×10−42\.0\\times 10^\{\-4\}–1\.3×10−31\.3\\times 10^\{\-3\}\); the imprint is real but three orders smaller than behavioral effect sizes\. A window\-disjoint variant then inserts a deletable≥512\{\\geq\}512\-token neutral shadow directly after the record, so no retained token lies within the local window after any record token\. The gap does*not*collapse \(hero max1\.2×10−3→1\.0×10−31\.2\\times 10^\{\-3\}\\to 1\.0\\times 10^\{\-3\}; Helios1\.3×10−3→4\.5×10−31\.3\\times 10^\{\-3\}\\to 4\.5\\times 10^\{\-3\}\): the imprint channel is predominantly*global*—later retained tokens read the record through the grafted full\-context gates during ingestion regardless of window adjacency—so write\-time spacing cannot substitute for the repack\. An imprint\-free write discipline would require segmented ingestion \(isolated forward passes per record\), which we leave as future work\.

Local MIMIC\-IV\-Ext\-CDS validation\.Under the PhysioNet restricted\-data agreement, a local\-only runner reads compact structured fields \(chief complaint, arrival transport, disposition\) frominitial\_assessment\_info\.csv; no text, identifier, generation, or per\-record result is written\. Of sixteen deterministic candidates, six pass all admission gates\. With3232samples per field and record\-level bootstrap intervals, Leak@11is\.269\.269present,\.038\.038\[\.017,\.059\]\[\.017,\.059\]after deletion,\.049\.049\[\.026,\.075\]\[\.026,\.075\]never, and\.248\.248ICUL; deletion tracks the never floor with overlapping intervals at every evaluatedkk\. Mean secret log\-probability is−0\.90\-0\.90present,−3\.81\-3\.81after deletion, and−3\.76\-3\.76never, while the retained\-neighbor shift is\+0\.03\+0\.03\[−0\.02,\+0\.08\]\[\-0\.02,\+0\.08\]nats\. These aggregates are external\-structure corroboration, not the primary benchmark\.

Reproduction\.The behavioral run ispython \-mgemma\_sv\.eval\_whole\_record\_unlearning;gemma\_sv\.make\_whole\_record\_figurecomputes record\-level bootstrap intervals;gemma\_sv\.certify\_whole\_recordproduces float64 probe certificates; andpython \-mgemma\_sv\.hero\_demoselects and renders the admitted hero\. The credentialed\-data runnergemma\_sv\.eval\_mimic\_whole\_recordrequires an explicit local path and writes aggregate\-only JSON\.

## Appendix GRecurrent\-state deletion: protocols and supporting measurements

#### Setup\.

All hybrid experiments run the releasedmlx\-community/Kimi\-Linear\-48B\-A3B\-Instruct\-8bitweights under MLX on one Mac Studio, inference only; no parameter is updated anywhere in Section[7](https://arxiv.org/html/2607.27539#S7)or this appendix\. For the attention\-only diagnostic, a single\-precision FISTA gate masks victim positions in all seven MLA layers; it is not a float64 decrement or an exactness certificate\. Replay equivalence is always checked on final logits*and*every KDA recurrent state against a reference that never ingested the victim\. MIMIC runs follow the credentialed\-data discipline of Appendix[A](https://arxiv.org/html/2607.27539#A1): local files only, aggregate\-only reports, no generations produced, synthetic record keys, and a substring audit over all source values before any report is written\.

![Refer to caption](https://arxiv.org/html/2607.27539v1/x3.png)Figure 7:*One hybrid, two kinds of memory\.*Kimi Linear interleaves2020recurrent KDA layers with77global attention layers\. An attention\-only mask provides a diagnostic comparator; the recurrent state is saved at record boundaries and rebuilt exactly by replaying only the suffix\.
#### Replay oracle on real records\.

On MIMIC\-IV\-Ext\-CDS every admitted deletion passes the bitwise state\-and\-logit audit:8/88/8at236236tokens and9/99/9at128128records \(3,2483\{,\}248tokens\), with victim lift\+2\.020→0\.000\+2\.020\\to 0\.000nats and retained drift0\.0000\.000\. On MIMIC\-IV\-Note the same holds at18,84218\{,\}842tokens \(4/44/4,\+1\.861→0\.000\+1\.861\\to 0\.000\)\. Cost depends only on the suffix:6\.706\.70s for the oldest of128128records,0\.000\.00s for the newest, and3\.493\.49s mean against a flat6\.776\.77s full rebuild\. The129129checkpoints occupy5\.225\.22GiB\.

![Refer to caption](https://arxiv.org/html/2607.27539v1/x4.png)Figure 8:*Replay deletion on real clinical records\.**\(a\)*Within the same deterministic MLX execution, victim influence is exactly zero after deletion in every admitted case, bit\-for\-bit on logits and all2020recurrent states\.*\(b\)*Replay latency falls with the length of the suffix while a full rebuild remains flat\.Table 6:*Attention\-only masking versus the replay oracle\.*Each row applies both interventions to the same record\. “Lift” is the record’s pull on its own content relative to never ingesting it\. After replay, victim rank and all audited states equal the never\-ingested reference exactly\.Table 7:*Two deletions, verbatim\.*Zeroing every recurrent state leaves the synthetic secret available through attention\. On TOFU, attention masking produces a third answer; replay matches never\-ingested token for token\.“What is the ward code for patient 5182?”*stored:*OBSIDIAN\-TWOall recurrent states zeroed“OBSIDIAN\-TWO”replay deletion“The ward code for patient 5182 is not provided in the given records\.”“What does Hsiao Yun\-Hwa identify as in terms of gender?”*stored:*“Hsiao Yun\-Hwa is part of the LGBTQ\+ community\.”after attention mask“Hsiao Yun\-Hwa identifies as a woman\.”after replay“Hsiao Yun\-Hwa identifies as female\.”never ingested“Hsiao Yun\-Hwa identifies as female\.”Table 8:*The masking\-versus\-replay oracle’s synthetic block*\(Table[6](https://arxiv.org/html/2607.27539#A7.T6)carries TOFU and the MIMIC corpora\)\.1616synthetic ward\-code records; same protocol, gate, and checks\.
#### The amendment demo\.

The introduction’s scenario, run end to end \(python \-m kimi\_sv\.amendment\_demo; every quoted string is invented\)\. A scribe memory ingests five statements from a fictional visit; Statement 3 records “My mother had breast cancer,” and the correction replaces it with “My mother had a breast lump that imaging showed was benign, not cancer\.” Table[9](https://arxiv.org/html/2607.27539#A7.T9)shows the transcript\. The attention mask suppresses the wrong target from rank22to rank7474, but is only a direct\-channel diagnostic and cannot install the correction: the memory then knows neither statement\. The amendment rewinds to Statement 3’s boundary, ingests the corrected statement, and replays the two statements after it \(5555tokens,1\.01\.0s\); the result equals a memory that heard the corrected statement from the start, bitwise on logits and all2020recurrent states, and its greedy answer matches the reference’s token for token\. Every teacher\-forced score after the amendment equals the reference’s to the printed digit \(corrected target rank33, superseded target rank44\)\.

Table 9:*The amendment, verbatim\.*“According to the statements, what condition did the patient’s mother have?”—greedily answered under each condition of the demo\. After the amendment every measured quantity equals the corrected\-from\-start reference \(bitwise audit:0\.0000\.000on logits and all2020recurrent states\)\.as heard“breast cancer”after attention mask“Not mentioned”after amendment“The patient’s mother had a breast lump that imaging showed was benign, not cancer\.”corrected from the start“The patient’s mother had a breast lump that imaging showed was benign, not cancer\.”
#### Decay cannot delete in a hybrid\.

Setting all2020KDA recurrent states to zero—γ=0\\gamma=0, verified as a real intervention bytests/test\_kimi\_decay\.py—leaves the planted record at rank11\(p=0\.881p=0\.881\) and its logit residual at1\.7×1011\.7\\times 10^\{1\}, because the seven global layers hold the context verbatim;2,0482\{,\}048filler tokens leave the readout advantage undiminished\. Replay at the same positions: rank9,7779\{,\}777–13,22913\{,\}229,p=0\.000p=0\.000, residual0\.0000\.000on logits and state, retained records at rank11\.

Table 10:*The separability measurement, per corpus*\(pooled in the main\-text table\)\. Relative suffix\-dependence of one record’s state contribution, raw and after the best decay\-style stored correction\.
#### Separability protocol\.

Contexts are preamble\+\+44prefix records, one victim, and two equal\-length suffixes \(prefix/victim/suffix203/44/179203/44/179tokens synthetic, comparable for MIMIC\-CDS, and4,904/948/4,4854\{,\}904/948/4\{,\}485for MIMIC\-notes\), over66victim/suffix configurations per corpus; Table[10](https://arxiv.org/html/2607.27539#A7.T10)reports each corpus separately\. The end\-to\-end level comparesΔ​S\\Delta Sacross suffixes from four full prefills per configuration\. The isolated level captures each probed layer’s per\-token kernel inputs\(k,v,g,β\)\(k,v,g,\\beta\)once from the record\-present runs and re\-runs the recurrence on identical inputs with the victim’s segment included or skipped, in float32, under the three write rules of Table[2](https://arxiv.org/html/2607.27539#S7.T2); the decay correction is applied by cross\-multiplication \(Δ​SA⊙DB\\Delta S\_\{A\}\\odot D\_\{B\}vs\.Δ​SB⊙DA\\Delta S\_\{B\}\\odot D\_\{A\},DDthe suffix’s cumulative per\-channel decay\) to avoid underflowing division\. The sequential reference reproduces the fused Metal kernel’s state to≤3\.2×10−7\\leq 3\.2\\times 10^\{\-7\}relative on every probed layer before any variant is trusted, and a synthetic\-input unit test \(tests/test\_separability\.py\) locks the three\-rule algebra: separable suffix\-independent, decay suffix\-dependent but exactly ledger\-correctable, delta rule neither\.