GROM: Gradient-Free Rapid One-Shot Machine Unlearning

# GROM: Gradient-Free Rapid One-Shot Machine Unlearning
Source: [https://arxiv.org/html/2608.05783](https://arxiv.org/html/2608.05783)
###### Abstract

Machine unlearning has become a critical capability for safely removing specific, sensitive knowledge from large language models \(LLMs\)\. Current state\-of\-the\-art approaches primarily rely on iterative, training\-time unlearning via fine\-tuning\. However, even when utilizing parameter\-efficient dimensionality reduction techniques like LoRA, gradient\-based optimization remains computationally expensive and lacks explicit analytical formulations\. It can also leave the targeted knowledge merely hidden rather than removed, to the point that simply quantizing the unlearned model restores much of what it was supposed to have erased\. To resolve this, we propose a novel one\-shot unlearning approach, abandoning iterative optimization in favor of a direct, exact analytical solution\. We frame the unlearning process as a ridge\-regularized least\-squares optimization problem, deriving a closed\-form additive update for targeted weight matrices\. This update forces the selected layer to suppress unwanted content while strictly preserving its behavior on retained data\. Computed from gradient\-free forward passes alone, with no backpropagation and no iteration to convergence, GROM applies the weight edit in mere seconds, which makes it orders of magnitude faster than traditional fine\-tuning\. Extensive evaluations demonstrate that GROM achieves state\-of\-the\-art forgetting\-utility trade\-offs on TOFU\-5%, TOFU\-10%, MUSE\-Books, MUSE\-News and WMDP, significantly reducing computational overhead without sacrificing overall model performance\. Because the update removes the targeted content from the weights instead of masking it, GROM also withstands the low\-bit quantization attack that recovers much of the content a gradient\-based baseline had appeared to forget\. Our code is publicly available athttps://github\.com/Batorskq/GROM\.

## Introduction

Large Language Models \(LLMs\) frequently memorize sensitive, private, or copyrighted information from their vast training corpora\. Consequently, machine unlearning has emerged as a crucial mechanism to selectively erase this targeted knowledge\. Existing interventions span a continuum from inference\-time mitigation to parameter editing and training\-time updates\(Liu et al\.[2025](https://arxiv.org/html/2608.05783#bib.bib35); Ren et al\.[2025](https://arxiv.org/html/2608.05783#bib.bib43)\)\. Mitigation strategies, such as decoding controls or prompt\-based defenses\(Yu et al\.[2021](https://arxiv.org/html/2608.05783#bib.bib55); Huang et al\.[2024](https://arxiv.org/html/2608.05783#bib.bib23); Thaker et al\.[2024a](https://arxiv.org/html/2608.05783#bib.bib47)\), act as lightweight shields that deflect rather than physically remove the underlying information\. Conversely, localized parameter editing methods\(Ilharco et al\.[2022](https://arxiv.org/html/2608.05783#bib.bib24); Meng et al\.[2022](https://arxiv.org/html/2608.05783#bib.bib38)\)offer rapid updates for specific associations but often struggle to scale effectively to broad, distributional forget sets\.

Currently, training\-time unlearning provides the strongest performance by optimizing model parameters to reduce the likelihood of the forget set while preserving retain\-set utility\. This encompasses a wide range of objectives, including gradient ascent\(Thudi et al\.[2022](https://arxiv.org/html/2608.05783#bib.bib49); Yao, Xu, and Liu[2024a](https://arxiv.org/html/2608.05783#bib.bib53)\), reverse KL divergence\(Wang et al\.[2024](https://arxiv.org/html/2608.05783#bib.bib50)\), and preference\-style optimization\(Rafailov et al\.[2023](https://arxiv.org/html/2608.05783#bib.bib42); Zhang et al\.[2024b](https://arxiv.org/html/2608.05783#bib.bib57); Fan et al\.[2025](https://arxiv.org/html/2608.05783#bib.bib12)\)\. While effective, these methods inherently rely on iterative fine\-tuning\. Even when utilizing parameter\-efficient dimensionality reduction techniques like LoRA, gradient\-based optimization remains computationally expensive\. More troubling, the forgetting it produces can be superficial: recent work shows that simply quantizing an unlearned model restores much of the content it was supposed to have removed\(Zhang et al\.[2025](https://arxiv.org/html/2608.05783#bib.bib58)\), which suggests that fine\-tuning often hides the targeted knowledge in low\-magnitude weight adjustments that low\-bit rounding undoes\. Crucially, these approaches also lack an explicit analytical formulation, forcing practitioners to rely on costly, step\-by\-step gradient descent to approximate an unlearned state\.

To address these fundamental inefficiencies, we introduce GROM \(Gradient\-free Rapid One\-shot Machine\-unlearning\), a new unlearning formulation that abandons iterative optimization in favor of a direct, exact analytical solution\. Instead of training additional parameters over multiple epochs, GROM computes a precise, additive update to targeted weight matrices in a single step\. We frame the unlearning process as a ridge\-regularized least\-squares optimization problem, forcing a selected layer to suppress unwanted content while strictly maintaining its output on retained data\. By solving this system in closed form, GROM computes the optimal weight update from gradient\-free forward passes over the forget and retain data, one pair per edited layer, executing the entire unlearning procedure in mere seconds\.

The main contributions of this work are threefold:

- •Closed\-Form Unlearning Formulation:We mathematically frame machine unlearning as an exact optimization problem with a closed\-form analytical solution, completely bypassing the computational overhead and optimization instability of iterative fine\-tuning\.
- •Gradient\-Free Efficiency:GROM computes each layer’s optimal additive update from gradient\-free forward passes alone, with no backpropagation and no iteration to convergence, which makes it up to two orders of magnitude faster than gradient\-based approaches\.
- •State\-of\-the\-Art Trade\-offs:GROM achieves Pareto\-best forgetting\-utility trade\-offs across five benchmarks, and it withstands the low\-bit quantization attack that restores much of what a strong gradient\-based baseline had appeared to forget, staying as forgetful as the gold retrained model\.

MethodDesirable requirement satisfiedClosed\-formRef\.\-freeNo teacherGA\(Thudi et al\.[2022](https://arxiv.org/html/2608.05783#bib.bib49)\)\\cellcolor\[HTML\]FCE8E6×\\times\\cellcolor\[HTML\]E8F5E9✓\\checkmark\\cellcolor\[HTML\]E8F5E9✓\\checkmarkGradDiff\(Yao, Xu, and Liu[2024b](https://arxiv.org/html/2608.05783#bib.bib54)\)\\cellcolor\[HTML\]FCE8E6×\\times\\cellcolor\[HTML\]E8F5E9✓\\checkmark\\cellcolor\[HTML\]E8F5E9✓\\checkmarkIDKDPO\(Maini et al\.[2024](https://arxiv.org/html/2608.05783#bib.bib36)\)\\cellcolor\[HTML\]FCE8E6×\\times\\cellcolor\[HTML\]FCE8E6×\\times\\cellcolor\[HTML\]E8F5E9✓\\checkmarkIDKNLL\(Maini et al\.[2024](https://arxiv.org/html/2608.05783#bib.bib36)\)\\cellcolor\[HTML\]FCE8E6×\\times\\cellcolor\[HTML\]E8F5E9✓\\checkmark\\cellcolor\[HTML\]E8F5E9✓\\checkmarkRMU\(Li et al\.[2024](https://arxiv.org/html/2608.05783#bib.bib31)\)\\cellcolor\[HTML\]FCE8E6×\\times\\cellcolor\[HTML\]E8F5E9✓\\checkmark\\cellcolor\[HTML\]E8F5E9✓\\checkmarkRKLD\(Wang et al\.[2024](https://arxiv.org/html/2608.05783#bib.bib50)\)\\cellcolor\[HTML\]FCE8E6×\\times\\cellcolor\[HTML\]E8F5E9✓\\checkmark\\cellcolor\[HTML\]FCE8E6×\\timesNPO\(Zhang et al\.[2024b](https://arxiv.org/html/2608.05783#bib.bib57)\)\\cellcolor\[HTML\]FCE8E6×\\times\\cellcolor\[HTML\]FCE8E6×\\times\\cellcolor\[HTML\]E8F5E9✓\\checkmarkAltPO\(Mekala et al\.[2025](https://arxiv.org/html/2608.05783#bib.bib37)\)\\cellcolor\[HTML\]FCE8E6×\\times\\cellcolor\[HTML\]FCE8E6×\\times\\cellcolor\[HTML\]E8F5E9✓\\checkmarkUNDIAL\(Dong et al\.[2025](https://arxiv.org/html/2608.05783#bib.bib8)\)\\cellcolor\[HTML\]FCE8E6×\\times\\cellcolor\[HTML\]E8F5E9✓\\checkmark\\cellcolor\[HTML\]FCE8E6×\\timesSimNPO\(Fan et al\.[2025](https://arxiv.org/html/2608.05783#bib.bib12)\)\\cellcolor\[HTML\]FCE8E6×\\times\\cellcolor\[HTML\]E8F5E9✓\\checkmark\\cellcolor\[HTML\]E8F5E9✓\\checkmark\\rowcolor\[HTML\]F2F7FF GROM\\cellcolor\[HTML\]E8F5E9✓\\checkmark\\cellcolor\[HTML\]E8F5E9✓\\checkmark\\cellcolor\[HTML\]E8F5E9✓\\checkmark

Table 1:Qualitative comparison of method requirements, where a check marks a satisfied property\.*Closed\-form*means the update is a direct analytical solve, one per edited layer, rather than iterative gradient optimization, which implies no backpropagation, no iteration to convergence, and seconds rather than minutes \(Time column of Tables[2](https://arxiv.org/html/2608.05783#Sx4.T2)–[5](https://arxiv.org/html/2608.05783#Sx4.T5)\)\.*Ref\.\-free*and*No teacher*mean no reference model and no sanitized teacher are required\.
## Related Work

LLM unlearning spans several intervention levels, and recent surveys emphasize that behavioural suppression, parameter modification, and deletion guarantees should not be conflated\(Liu et al\.[2025](https://arxiv.org/html/2608.05783#bib.bib35); Ren et al\.[2025](https://arxiv.org/html/2608.05783#bib.bib43)\)\. These objectives descend from exact and certified deletion for smaller models\(Cao and Yang[2015](https://arxiv.org/html/2608.05783#bib.bib3); Bourtoule et al\.[2021](https://arxiv.org/html/2608.05783#bib.bib2); Ginart et al\.[2019](https://arxiv.org/html/2608.05783#bib.bib16); Guo et al\.[2019](https://arxiv.org/html/2608.05783#bib.bib17); Izzo et al\.[2021](https://arxiv.org/html/2608.05783#bib.bib25)\), a line motivated by right\-to\-be\-forgotten provisions\(Rosen[2011](https://arxiv.org/html/2608.05783#bib.bib44); Hoofnagle, Van Der Sloot, and Borgesius[2019](https://arxiv.org/html/2608.05783#bib.bib21)\)\. Some methods avoid direct weight edits and instead alter access to unwanted content through privacy\-oriented fine\-tuning, logit or offset steering, guardrails, or in\-context control\(Yu et al\.[2021](https://arxiv.org/html/2608.05783#bib.bib55); Huang et al\.[2024](https://arxiv.org/html/2608.05783#bib.bib23); Thaker et al\.[2024a](https://arxiv.org/html/2608.05783#bib.bib47); Pawelczyk, Neel, and Lakkaraju[2023](https://arxiv.org/html/2608.05783#bib.bib41)\)\. These approaches can be lightweight, but the forgetting often depends on an external inference\-time mechanism\. Another line edits parameters more directly, including task arithmetic and factual editing methods such as ROME and MEMIT\(Ilharco et al\.[2022](https://arxiv.org/html/2608.05783#bib.bib24); Meng et al\.[2022](https://arxiv.org/html/2608.05783#bib.bib38),[2023](https://arxiv.org/html/2608.05783#bib.bib39); Hase et al\.[2023](https://arxiv.org/html/2608.05783#bib.bib19)\)\. Such edits are fast and localized, but are often designed for specific associations rather than distributional forget sets involving many tokens and a competing retain set\. Although GROM is similarly localized, it targets this broader forget\-retain setting rather than single\-fact replacement\. Most empirical LLM unlearning instead optimizes a forget\-retain loss by fine\-tuning\. Prior objectives include KL and distillation\-style retention\(Wang et al\.[2024](https://arxiv.org/html/2608.05783#bib.bib50); Dong et al\.[2025](https://arxiv.org/html/2608.05783#bib.bib8)\), refusal or “I don’t know” targets\(Maini et al\.[2024](https://arxiv.org/html/2608.05783#bib.bib36)\), gradient\-ascent, gradient\-difference, and continual unlearning\(Thudi et al\.[2022](https://arxiv.org/html/2608.05783#bib.bib49); Yao, Xu, and Liu[2024b](https://arxiv.org/html/2608.05783#bib.bib54); Liu, Liu, and Stone[2022](https://arxiv.org/html/2608.05783#bib.bib34)\), and preference\-style variants\(Rafailov et al\.[2023](https://arxiv.org/html/2608.05783#bib.bib42); Zhang et al\.[2024b](https://arxiv.org/html/2608.05783#bib.bib57); Fan et al\.[2025](https://arxiv.org/html/2608.05783#bib.bib12); Mekala et al\.[2025](https://arxiv.org/html/2608.05783#bib.bib37); Jia et al\.[2024](https://arxiv.org/html/2608.05783#bib.bib27); Ji et al\.[2024](https://arxiv.org/html/2608.05783#bib.bib26); Chen and Yang[2023](https://arxiv.org/html/2608.05783#bib.bib6)\)\. This literature shows that the target and loss shape the forgetting\-utility trade\-off, but iterative optimization is costly and sensitive\. GROM keeps the target\-design view while replacing fine\-tuning with a closed\-form one\-shot update\. Evaluation work further shows that benchmark success need not imply durable deletion, documenting over\-unlearning, reversibility, relearning, and deployment\-specific failures such as quantization sensitivity\(Xu et al\.[2025](https://arxiv.org/html/2608.05783#bib.bib52); Zhang et al\.[2025](https://arxiv.org/html/2608.05783#bib.bib58); Hu et al\.[2024](https://arxiv.org/html/2608.05783#bib.bib22); Thaker et al\.[2024b](https://arxiv.org/html/2608.05783#bib.bib48); Zhang et al\.[2024a](https://arxiv.org/html/2608.05783#bib.bib56); Duan et al\.[2024](https://arxiv.org/html/2608.05783#bib.bib10)\)\. In contrast to procedures that can be fragile under post\-edit compression, GROM’s closed\-form update is naturally resilient to quantization attacks\.

![Refer to caption](https://arxiv.org/html/2608.05783v1/x1.png)Figure 1:Overview of GROM\. From gradient\-free forward passes over the forget and retain data we collect the per\-token keysXf,XrX\_\{f\},X\_\{r\}\(Stage 1\)\. We then solve a single ridge\-regularized least\-squares problem in closed form for the additive updateP⋆P^\{\\star\}\(Stage 2\)\. A logit\-lens attribution scoreSℓ=Fℓ−RℓS\_\{\\ell\}=F\_\{\\ell\}\-R\_\{\\ell\}selects a small late band of layers𝒮\\mathcal\{S\}to edit \(Stage 3\)\. ApplyingW←W\+P⋆W\\\!\\leftarrow\\\!W\+P^\{\\star\}to those layers yields the unlearned model, which withholds the memorized content on the same prompt while leaving retain behaviour intact \(Stage 4\)\.
## Method

In this section, we formally define the machine unlearning task as an exact optimization problem\. We first establish the problem setup and our ridge\-regularized least\-squares objective, which admits a unique closed\-form solution\. We then detail the design of specific unlearning targets based on token suppression and representation corruption\. Finally, we describe our logit\-lens layer selection strategy and introduce an exact analytical method for auditing the influence of individual deletion requests\.

#### Setup and Editable Matrices\.

LetW∈ℝm×nW\\in\\mathbb\{R\}^\{m\\times n\}be a weight matrix we edit, either the language\-model head or the down\-projection of an MLP block\. With one gradient\-free forward pass over the forget data and one over the retain data, we collect the*inputs*toWW, the per\-token “keys”, and stack them column\-wise into:

Xf∈ℝn×s​\(forget keys\),Xr∈ℝn×r​\(retain keys\)\.X\_\{f\}\\in\\mathbb\{R\}^\{n\\times s\}\\ \(\\text\{forget keys\}\),\\qquad X\_\{r\}\\in\\mathbb\{R\}^\{n\\times r\}\\ \(\\text\{retain keys\}\)\.A key is taken at every position whose next token should be forgotten \(respectively preserved\): the answer tokens of a forget/retain QA pair, or, when the forget set is an unlabeled corpus, every token position of that corpus\. The layer’s current outputs on these keys areW​XfWX\_\{f\}andW​XrWX\_\{r\}\.

Importantly, addingΔ\\DeltatoWWshifts every output fromW​xWxtoW​x\+Δ​xWx\+\\Delta x\. We control that shift only if the model usesW​xWxdirectly, either by adding it to the residual stream or by reading it as logits\. This holds true for the MLP down\-projection, the attention output projection, and the LM head, but not for projections whose outputs first pass through a nonlinearity \(e\.g\., up\- and gate\-projections\)\. Among the usable matrices, we edit the MLP down\-projection, since feed\-forward blocks are widely reported to store facts and push them toward particular output tokens\(Geva et al\.[2021](https://arxiv.org/html/2608.05783#bib.bib15),[2022](https://arxiv.org/html/2608.05783#bib.bib14); Dai et al\.[2022](https://arxiv.org/html/2608.05783#bib.bib7)\)\. Table[7](https://arxiv.org/html/2608.05783#Sx4.T7)checks the alternatives empirically\.

#### Objective and Closed\-Form Solution\.

We seek a single additive updateP∈ℝm×nP\\in\\mathbb\{R\}^\{m\\times n\}, applied in closed form \(W←W\+PW\\leftarrow W\+P\), that simultaneously preserves the retain behavior,P​Xr≈0PX\_\{r\}\\approx 0\(so\(W\+P\)​Xr≈W​Xr\(W\+P\)X\_\{r\}\\approx WX\_\{r\}\), and steers the forget outputs by a prescribed*unlearning target*D∈ℝm×sD\\in\\mathbb\{R\}^\{m\\times s\}that removes the memorized content,\(W\+P\)​Xf≈W​Xf\+D\(W\+P\)X\_\{f\}\\approx WX\_\{f\}\+D\. Here,DDspecifies the desired output\-space displacement for each forget feature\.

We frame this trade\-off as a ridge\-regularized least\-squares problem:

minP∈ℝm×n⁡wrr​‖P​Xr‖F2\+wfs​‖P​Xf−D‖F2\+μ​‖P‖F2\\min\_\{P\\in\\mathbb\{R\}^\{m\\times n\}\}\\;\\frac\{w\_\{r\}\}\{r\}\\,\\\|PX\_\{r\}\\\|\_\{F\}^\{2\}\+\\frac\{w\_\{f\}\}\{s\}\\,\\\|PX\_\{f\}\-D\\\|\_\{F\}^\{2\}\+\\mu\\,\\\|P\\\|\_\{F\}^\{2\}wherewr,wf\>0w\_\{r\},w\_\{f\}\>0weight retain preservation against forget steering, andμ\>0\\mu\>0penalizes the edit size\. The targetDDis any fixed matrix inℝm×s\\mathbb\{R\}^\{m\\times s\}, chosen before solving forPP\. The optimizer of this problem is unique and given in closed form, avoiding iterative approximation entirely\.

###### Theorem 1\.

LetXr∈ℝn×rX\_\{r\}\\in\\mathbb\{R\}^\{n\\times r\},Xf∈ℝn×sX\_\{f\}\\in\\mathbb\{R\}^\{n\\times s\},D∈ℝm×sD\\in\\mathbb\{R\}^\{m\\times s\}, and letwr,wf,μ\>0w\_\{r\},w\_\{f\},\\mu\>0\. Define

A=wrr​Xr​Xr⊤\+wfs​Xf​Xf⊤\+μ​In∈ℝn×n\.A=\\frac\{w\_\{r\}\}\{r\}X\_\{r\}X\_\{r\}^\{\\top\}\+\\frac\{w\_\{f\}\}\{s\}X\_\{f\}X\_\{f\}^\{\\top\}\+\\mu I\_\{n\}\\in\\mathbb\{R\}^\{n\\times n\}\.Then the ridge\-regularized objective is minimized by the unique matrix

P⋆=wfs​D​Xf⊤​A−1\.P^\{\\star\}=\\frac\{w\_\{f\}\}\{s\}\\,DX\_\{f\}^\{\\top\}A^\{\-1\}\.

The full proof of Theorem[1](https://arxiv.org/html/2608.05783#Thmtheorem1)is provided in Appendix[A](https://arxiv.org/html/2608.05783#A1)\.

#### Unlearning Targets\.

Theorem[1](https://arxiv.org/html/2608.05783#Thmtheorem1)holds for any fixedDD, so the target is where we encode*what*to forget\. We use one of two forms, chosen according to how the benchmark probes the forgotten knowledge\.

*\(i\) Token suppression*, used when forgetting is measured through the content the model*generates*\(TOFU, MUSE\)\. For each forget positionjjwith gold next tokengj∈\{1,…,\|𝒱\|\}g\_\{j\}\\in\\\{1,\\dots,\|\\mathcal\{V\}\|\\\}, we steer the output away from that token:

dj=−β​αj​uj,uj=\{egj,W=Whead,wgj‖wgj‖2,W=Wdown\(ℓ\),d\_\{j\}=\-\\beta\\,\\alpha\_\{j\}\\,u\_\{j\},\\qquad u\_\{j\}=\\begin\{cases\}e\_\{g\_\{j\}\},&W=W\_\{\\mathrm\{head\}\},\\\\\[5\.69054pt\] \\frac\{w\_\{g\_\{j\}\}\}\{\\\|w\_\{g\_\{j\}\}\\\|\_\{2\}\},&W=W\_\{\\mathrm\{down\}\}^\{\(\\ell\)\},\\end\{cases\}so that the post\-edit output\(W\+P\)​xj\(f\)≈W​xj\(f\)−β​αj​uj\(W\+P\)x^\{\(f\)\}\_\{j\}\\approx Wx^\{\(f\)\}\_\{j\}\-\\beta\\alpha\_\{j\}u\_\{j\}assigns a lower logit togjg\_\{j\}\. Hereβ\>0\\beta\>0is the edit strength,egje\_\{g\_\{j\}\}is the one\-hot vector for tokengjg\_\{j\}, andwgj=Whead​\[gj,:\]w\_\{g\_\{j\}\}=W\_\{\\mathrm\{head\}\}\[g\_\{j\},:\]is the LM\-head row for that token\. For a hidden\-layer edit, this row is the direction in residual space that most directly increases the logit ofgjg\_\{j\}\. The*specificity weight*αj∈\[0,1\]\\alpha\_\{j\}\\in\[0,1\]confines suppression to tokens distinctive of the forget set:

αj=max⁡\(0,1−rf​\(gj\)ff​\(gj\)\),\\alpha\_\{j\}=\\max\\\!\\Big\(0,\\;1\-\\frac\{\\mathrm\{rf\}\(g\_\{j\}\)\}\{\\mathrm\{ff\}\(g\_\{j\}\)\}\\Big\),where, for any tokenvv,

ff​\(v\)\\displaystyle\\mathrm\{ff\}\(v\)=∑t∈ℱ𝟏​\[t=v\]∑v′∈𝒱∑t∈ℱ𝟏​\[t=v′\],\\displaystyle=\\frac\{\\sum\_\{t\\in\\mathcal\{F\}\}\\mathbf\{1\}\[t=v\]\}\{\\sum\_\{v^\{\\prime\}\\in\\mathcal\{V\}\}\\sum\_\{t\\in\\mathcal\{F\}\}\\mathbf\{1\}\[t=v^\{\\prime\}\]\},rf​\(v\)\\displaystyle\\mathrm\{rf\}\(v\)=∑t∈ℛ𝟏​\[t=v\]∑v′∈𝒱∑t∈ℛ𝟏​\[t=v′\]\.\\displaystyle=\\frac\{\\sum\_\{t\\in\\mathcal\{R\}\}\\mathbf\{1\}\[t=v\]\}\{\\sum\_\{v^\{\\prime\}\\in\\mathcal\{V\}\}\\sum\_\{t\\in\\mathcal\{R\}\}\\mathbf\{1\}\[t=v^\{\\prime\}\]\}\.Hereℱ\\mathcal\{F\}is the list of all forget tokens, with duplicates, andℛ\\mathcal\{R\}is the analogous retain\-token list\. Tokens common to both receiveαj≈0\\alpha\_\{j\}\\\!\\approx\\\!0, whereas forget\-specific tokens receiveαj≈1\\alpha\_\{j\}\\\!\\approx\\\!1\.

*\(ii\) Representation corruption*, used when forgetting is measured by multiple\-choice accuracy \(WMDP\)\. There, the answer is a choice label rather than memorized text, so lowering content\-token logits has little effect\. Instead, we corrupt the representation itself\. With a single fixed random unit vectoru∈ℝmu\\in\\mathbb\{R\}^\{m\}, we set:

dj=c​ufor all​j,d\_\{j\}=c\\,u\\qquad\\text\{for all \}j,which, through the same objective with the retain keys anchored to0, drives the layer’s output on forget\-like inputs toward a fixed, meaningless direction while leaving retain inputs intact\. In both cases, the columns are assembled intoD=\[d1,…,ds\]D=\[\\,d\_\{1\},\\dots,d\_\{s\}\\,\], fixed before solving forPP\.

#### Choice of Layers to be Updated\.

Rather than choosing the edited layers only by grid search, we use a logit\-lens attribution score\(nostalgebraist[2020](https://arxiv.org/html/2608.05783#bib.bib40); Belrose et al\.[2023](https://arxiv.org/html/2608.05783#bib.bib1)\)to identify layers that directly write the memorized forget tokens\. For each MLP layerℓ\\ell, letoℓ,j\(f\)∈ℝdmodelo^\{\(f\)\}\_\{\\ell,j\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{model\}\}\}be the output of its down\-projection at forget positionjj, and letwgj∈ℝdmodelw\_\{g\_\{j\}\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{model\}\}\}be the LM\-head row corresponding to the gold next tokengjg\_\{j\}\. The direct contribution of layerℓ\\ellto the logit of tokengjg\_\{j\}is⟨wgj,oℓ,j\(f\)⟩\\langle w\_\{g\_\{j\}\},o^\{\(f\)\}\_\{\\ell,j\}\\rangle\. We compute the forget and retain effects:

Fℓ=∑j=1sαj​⟨wgj,oℓ,j\(f\)⟩∑j=1sαj,Rℓ=1r​∑i=1r⟨whi,oℓ,i\(r\)⟩,F\_\{\\ell\}=\\frac\{\\sum\_\{j=1\}^\{s\}\\alpha\_\{j\}\\langle w\_\{g\_\{j\}\},o^\{\(f\)\}\_\{\\ell,j\}\\rangle\}\{\\sum\_\{j=1\}^\{s\}\\alpha\_\{j\}\},\\qquad R\_\{\\ell\}=\\frac\{1\}\{r\}\\sum\_\{i=1\}^\{r\}\\langle w\_\{h\_\{i\}\},o^\{\(r\)\}\_\{\\ell,i\}\\rangle,wherehih\_\{i\}is the retain gold next\-token id\. We then score each layer bySℓ=Fℓ−RℓS\_\{\\ell\}=F\_\{\\ell\}\-R\_\{\\ell\}\. A large positiveSℓS\_\{\\ell\}indicates that layerℓ\\ellcontributes more to forget\-specific gold\-token logits than to retain\-token logits\.

We treat the number of edited layers as a small edit\-width hyperparameterkk\. For a fixedkkover candidate layersℒcand\\mathcal\{L\}\_\{\\mathrm\{cand\}\}, we choose the contiguous window with the largest average score:

𝒮k=arg⁡max\{a,…,a\+k−1\}⊂ℒcand⁡1k​∑ℓ=aa\+k−1Sℓ\.\\mathcal\{S\}\_\{k\}=\\arg\\max\_\{\\\{a,\\ldots,a\+k\-1\\\}\\subset\\mathcal\{L\}\_\{\\mathrm\{cand\}\}\}\\frac\{1\}\{k\}\\sum\_\{\\ell=a\}^\{a\+k\-1\}S\_\{\\ell\}\.For eachℓ∈𝒮k\\ell\\in\\mathcal\{S\}\_\{k\}, we compute the closed\-form update forWdown\(ℓ\)W\_\{\\mathrm\{down\}\}^\{\(\\ell\)\}, applying the updates sequentially and recomputing features after each edited layer\. Figure[1](https://arxiv.org/html/2608.05783#Sx2.F1)summarizes the full pipeline\.

#### Influence of a Single Forget Example\.

BecauseP⋆P^\{\\star\}is an explicit function of the forget data, we can further ask how much any*single*forget example contributed to the edit, and answer it exactly\. We use this measure to*audit*individual deletion requests\. A provider asked to account for a user’s data must be able to state what that example contributed to the released model\.

Partition the forget keys and target by example,Xf=\[C1​⋯​CN\]X\_\{f\}=\[\\,C\_\{1\}\\,\\cdots\\,C\_\{N\}\\,\]andD=\[D1​⋯​DN\]D=\[\\,D\_\{1\}\\,\\cdots\\,D\_\{N\}\\,\], whereCi∈ℝn×ciC\_\{i\}\\in\\mathbb\{R\}^\{n\\times c\_\{i\}\}collects the token positions of exampleii, and letP−i⋆P^\{\\star\}\_\{\-i\}be the update recomputed with exampleiiremoved \(holding the per\-token forget weightwfs\\tfrac\{w\_\{f\}\}\{s\}fixed\)\. Deleting exampleiiperturbs the Gram matrixAAby a symmetric rank\-cic\_\{i\}downdate\. The Sherman–Morrison–Woodbury identity\(Hager[1989](https://arxiv.org/html/2608.05783#bib.bib18)\)resolves this analytically\.

###### Theorem 2\.

LetA−i=A−wfs​Ci​Ci⊤A\_\{\-i\}=A\-\\tfrac\{w\_\{f\}\}\{s\}C\_\{i\}C\_\{i\}^\{\\top\}\. ThenA−iA\_\{\-i\}is symmetric positive definite, and the Sherman–Morrison–Woodbury identity yields the deletion influence of exampleiiin closed form as:

Δ​Pi:=P⋆−P−i⋆=Li​Ri⊤,Ri=A−1​Ci,\\Delta P\_\{i\}:=P^\{\\star\}\-P^\{\\star\}\_\{\-i\}=L\_\{i\}R\_\{i\}^\{\\top\},\\qquad R\_\{i\}=A^\{\-1\}C\_\{i\},Li=wfs​\(Di−B−i​Ci​Mi−1\),Mi=Ici−wfs​Ci⊤​A−1​Ci,L\_\{i\}=\\tfrac\{w\_\{f\}\}\{s\}\\big\(D\_\{i\}\-B\_\{\-i\}\\,C\_\{i\}\\,M\_\{i\}^\{\-1\}\\big\),\\quad M\_\{i\}=I\_\{c\_\{i\}\}\-\\tfrac\{w\_\{f\}\}\{s\}C\_\{i\}^\{\\top\}A^\{\-1\}C\_\{i\},whereB−i=P⋆−wfs​Di​Ci⊤​A−1B\_\{\-i\}=P^\{\\star\}\-\\tfrac\{w\_\{f\}\}\{s\}D\_\{i\}C\_\{i\}^\{\\top\}A^\{\-1\}\. Hencerank⁡Δ​Pi≤ci\\operatorname\{rank\}\\Delta P\_\{i\}\\leq c\_\{i\}, andΔ​Pi\\Delta P\_\{i\}is obtained from the already\-computedA−1A^\{\-1\}by a singleci×cic\_\{i\}\\times c\_\{i\}solve, with non×nn\\times nreinversion and no retraining\.

While the same quantity could theoretically be obtained by completely recomputingP⋆P^\{\\star\}without exampleii, Theorem[2](https://arxiv.org/html/2608.05783#Thmtheorem2)ensures computational feasibility\. A naive approach would require reforming and reinverting the fulln×nn\\times nmatrixAAfor every example\. The analytical downdate reduces this to aci×cic\_\{i\}\\times c\_\{i\}solve against a single cachedA−1A^\{\-1\}, bringing the audit time down to seconds\. Gradient\-based unlearning admits no analogue and must approximate this with influence functions\(Koh and Liang[2017](https://arxiv.org/html/2608.05783#bib.bib29)\)or completely retrain the model per deleted example\.

## Experiments

#### Baselines\.

We compare GROM with representative unlearning methods spanning gradient\-based, representation\-based, preference\-based, and distillation\-based approaches\.GA\(Thudi et al\.[2022](https://arxiv.org/html/2608.05783#bib.bib49); Maini et al\.[2024](https://arxiv.org/html/2608.05783#bib.bib36)\)maximizes the loss on forget examples to reduce their likelihood, andGradDiff\(Yao, Xu, and Liu[2024b](https://arxiv.org/html/2608.05783#bib.bib54); Maini et al\.[2024](https://arxiv.org/html/2608.05783#bib.bib36); Liu, Liu, and Stone[2022](https://arxiv.org/html/2608.05783#bib.bib34)\)combines that ascent with descent on retain data\.TaskVector\(Ilharco et al\.[2022](https://arxiv.org/html/2608.05783#bib.bib24)\)edits behavior through task\-arithmetic updates in weight space, andRMU\(Li et al\.[2024](https://arxiv.org/html/2608.05783#bib.bib31)\)redirects forget representations toward a fixed random vector while regularizing retain representations\.IDKDPOandIDKNLL\(Maini et al\.[2024](https://arxiv.org/html/2608.05783#bib.bib36)\)train the model to answer forget prompts with “I don’t know” using DPO or NLL, whileRKLD\(Wang et al\.[2024](https://arxiv.org/html/2608.05783#bib.bib50)\)distills from a privacy\-sanitized teacher with reverse KL andUNDIAL\(Dong et al\.[2025](https://arxiv.org/html/2608.05783#bib.bib8)\)self\-distills with adjusted logits on forget data\. Among preference\-style objectives,NPO\(Zhang et al\.[2024b](https://arxiv.org/html/2608.05783#bib.bib57)\)suppresses undesirable forget responses with a negative\-preference loss,AltPO\(Mekala et al\.[2025](https://arxiv.org/html/2608.05783#bib.bib37)\)contrasts original against alternate forget answers, andSimNPO\(Fan et al\.[2025](https://arxiv.org/html/2608.05783#bib.bib12)\)simplifies NPO by removing the reference model\. Because GROM is itself a weight edit, we additionally compare against four locate\-then\-edit knowledge editors:ROMEandMEMIT\(Meng et al\.[2022](https://arxiv.org/html/2608.05783#bib.bib38),[2023](https://arxiv.org/html/2608.05783#bib.bib39)\),AlphaEdit\(Fang et al\.[2024](https://arxiv.org/html/2608.05783#bib.bib13)\), which confines the update to the null space of the preserved\-key covariance, andZeroUnlearn\(Lin et al\.[2026](https://arxiv.org/html/2608.05783#bib.bib33)\), which remaps forget keys to a neutral state through a multiplicative edit constrained to the null space of the original forget outputs\. All four are given the same per\-fact value optimization and the same retain covariance as GROM, so the rows differ only in the update rule\. We compare these editors only on TOFU and ZsRE, where examples can be cast as discrete facts\. The editors require forget data in annotated \(subject, relation, object\) triples: they locate a key at the subject and redirect a specific target object\. We do not include them on MUSE or WMDP because those benchmarks provide unstructured corpora or hazardous\-domain question data rather than such triples\. GROM does not require this structure, since it can collect keys directly from the forget text\.

#### Experimental Setup

We evaluate GROM against established unlearning baselines on TOFU\-5% and TOFU\-10%\(Maini et al\.[2024](https://arxiv.org/html/2608.05783#bib.bib36)\), MUSE\(Shi et al\.[2024b](https://arxiv.org/html/2608.05783#bib.bib46)\)and WMDP\(Li et al\.[2024](https://arxiv.org/html/2608.05783#bib.bib31)\), scoring all checkpoints with the open\-unlearning protocol\(Dorna et al\.[2025](https://arxiv.org/html/2608.05783#bib.bib9); Jin et al\.[2024](https://arxiv.org/html/2608.05783#bib.bib28)\)\. Target models, per\-benchmark metrics, the timing protocol, and the baseline training budgets are given in Appendix[G](https://arxiv.org/html/2608.05783#A7)\. Throughout,Time \(m\)is the wall\-clock of the unlearning update on a single NVIDIA H100, excluding model loading and evaluation\. It varies across benchmarks because the cost of GROM is set by model size and the number of edited layers rather than by forget\-set size, the keys being subsampled to a fixed budget, whereas gradient\-based baselines scale with epochs times corpus size, so the gap is narrowest on TOFU\-5% and widest on MUSE Books\.

Unlearning EfficacyUtilitySummaryMethod1\-Rouge\-L\(↑\\uparrow\)1\-Prob\.\(↑\\uparrow\)1\-Extr\.\(↑\\uparrow\)MU\(↑\\uparrow\)FinalScore \(↑\\uparrow\)Time \(m\)\(↓\\downarrow\)TOFU\-5% \(LLaMA2\-7B\-Chat\)Original0\.040\.010\.050\.620\.33—Retain0\.610\.850\.930\.620\.71—GradDiff\\cellcolor\[HTML\]FF999A1\.00\\cellcolor\[HTML\]FF999A1\.00\\cellcolor\[HTML\]FFCC990\.960\.56\\cellcolor\[HTML\]FFCC990\.772\.4IDKDPO\\cellcolor\[HTML\]FFCC990\.980\.400\.850\.570\.663\.3RKLD0\.69\\cellcolor\[HTML\]FFF8AD0\.960\.920\.560\.712\.9NPO0\.730\.940\.900\.570\.712\.9SimNPO0\.74\\cellcolor\[HTML\]FFCC990\.970\.92\\cellcolor\[HTML\]FFF8AD0\.58\\cellcolor\[HTML\]FFF8AD0\.732\.6ROME0\.820\.690\.900\.570\.68\\cellcolor\[HTML\]FF999A0\.6MEMIT0\.730\.690\.890\.530\.65\\cellcolor\[HTML\]FFCC990\.7AlphaEdit0\.940\.230\.08\\cellcolor\[HTML\]FFCC990\.610\.52\\cellcolor\[HTML\]FF999A0\.6ZeroUnlearn0\.800\.80\\cellcolor\[HTML\]FFF8AD0\.930\.510\.68\\cellcolor\[HTML\]FF999A0\.6\\rowcolor\[HTML\]F2F7FF GROM\\cellcolor\[HTML\]FFF8AD0\.95\\cellcolor\[HTML\]FF999A1\.00\\cellcolor\[HTML\]FF999A0\.97\\cellcolor\[HTML\]FF999A0\.62\\cellcolor\[HTML\]FF999A0\.79\\cellcolor\[HTML\]FFF8AD1\.8TOFU\-10% \(LLaMA3\.2\-1B\-Instruct\)Original0\.180\.120\.290\.600\.40—Retain0\.620\.880\.940\.590\.70—RMU0\.500\.390\.72\\cellcolor\[HTML\]FFF8AD0\.570\.55\\cellcolor\[HTML\]FFF8AD0\.6AltPO0\.66\\cellcolor\[HTML\]FFF8AD0\.93\\cellcolor\[HTML\]FFCC990\.95\\cellcolor\[HTML\]FFF8AD0\.57\\cellcolor\[HTML\]FFF8AD0\.716\.9GradDiff0\.420\.350\.67\\cellcolor\[HTML\]FFCC990\.590\.530\.8IDKDPO0\.87\\cellcolor\[HTML\]FF999A1\.00\\cellcolor\[HTML\]FF999A0\.960\.52\\cellcolor\[HTML\]FFCC990\.736\.9IDKNLL\\cellcolor\[HTML\]FF999A0\.980\.450\.740\.550\.640\.8UNDIAL0\.690\.82\\cellcolor\[HTML\]FF999A0\.960\.510\.670\.9NPO0\.610\.710\.91\\cellcolor\[HTML\]FFF8AD0\.570\.663\.7SimNPO0\.65\\cellcolor\[HTML\]FFCC990\.94\\cellcolor\[HTML\]FFF8AD0\.940\.560\.702\.5ROME\\cellcolor\[HTML\]FF999A0\.980\.370\.600\.540\.60\\cellcolor\[HTML\]FF999A0\.4MEMIT\\cellcolor\[HTML\]FFF8AD0\.880\.380\.610\.490\.56\\cellcolor\[HTML\]FFCC990\.5AlphaEdit\\cellcolor\[HTML\]FF999A0\.980\.560\.810\.500\.64\\cellcolor\[HTML\]FFCC990\.5ZeroUnlearn\\cellcolor\[HTML\]FFCC990\.920\.470\.810\.440\.59\\cellcolor\[HTML\]FF999A0\.4\\rowcolor\[HTML\]F2F7FF GROM0\.78\\cellcolor\[HTML\]FF999A1\.00\\cellcolor\[HTML\]FFF8AD0\.94\\cellcolor\[HTML\]FF999A0\.60\\cellcolor\[HTML\]FF999A0\.75\\cellcolor\[HTML\]FFCC990\.5

Table 2:Results on TOFU\-5% and TOFU\-10%\. Colors indicate rank among unlearning methods \(red: best, orange: second, yellow: third\)\.
#### TOFU\.

TOFU\(Maini et al\.[2024](https://arxiv.org/html/2608.05783#bib.bib36)\)evaluates selective forgetting in question answering about fictitious authors, where TOFU\-x%x\\%designatesx%x\\%of the authors for removal while the model must preserve its behaviour on the remaining ones\. Side effects on unrelated knowledge are measured through the Real Authors and World Facts subsets\. We report forgetting with ROUGE\-L\(Lin[2004](https://arxiv.org/html/2608.05783#bib.bib32)\), answer probability, and extraction strength\(Carlini et al\.[2021](https://arxiv.org/html/2608.05783#bib.bib5),[2022](https://arxiv.org/html/2608.05783#bib.bib4)\), and utility with the TOFU model\-utility \(MU\) score, following\(Fan et al\.[2025](https://arxiv.org/html/2608.05783#bib.bib12)\)\.

On TOFU\-5% \(Table[2](https://arxiv.org/html/2608.05783#Sx4.T2)\) GROM attains near\-saturated forgetting while keeping utility close to the original model, giving the best Final Score and the highest MU\. The harder TOFU\-10% setting \(same table\) splits the baselines into two failure modes, over\-forgetting at the cost of utility \(IDKDPO, UNDIAL\) or preserving utility while under\-forgetting \(RMU\)\. GROM avoids both with a single closed\-form update\. Metric definitions and edit details are given in Appendix[C](https://arxiv.org/html/2608.05783#A3)\.

#### ZsRE\.

The comparison above places the knowledge editors on an unlearning benchmark, so we also run the reverse test, evaluating GROM inside the harness of\(Lin et al\.[2026](https://arxiv.org/html/2608.05783#bib.bib33)\)on ZsRE\(Levy et al\.[2017](https://arxiv.org/html/2608.05783#bib.bib30)\), a few\-shot fact\-unlearning benchmark on which those editors are tuned\. Table[3](https://arxiv.org/html/2608.05783#Sx4.T3)shows that GROM outperforms these tuned editors: it removes targeted answers more effectively, generalizes better to paraphrases, and preserves locality at least as well\. Metric definitions and our reproduction of their unedited\-model row are given in Appendix[F](https://arxiv.org/html/2608.05783#A6)\.

Unlearning EfficacyUtility PreservationMethodEfficacy\(↓\\downarrow\)Generalization\(↓\\downarrow\)Specificity\(↑\\uparrow\)LLaMA3\.2\-3B\-InstructOriginal32\.82±\\pm4\.0932\.23±\\pm4\.1628\.12±\\pm2\.65ROME32\.80±\\pm4\.2032\.17±\\pm4\.09\\cellcolor\[HTML\]FF999A28\.05±\\pm2\.66MEMIT32\.32±\\pm4\.0431\.17±\\pm4\.61\\cellcolor\[HTML\]FFF8AD28\.01±\\pm2\.60AlphaEdit\\cellcolor\[HTML\]FFF8AD29\.59±\\pm3\.95\\cellcolor\[HTML\]FFF8AD29\.90±\\pm4\.6727\.80±\\pm2\.77ZeroUnlearn\\cellcolor\[HTML\]FFCC9927\.85±\\pm3\.87\\cellcolor\[HTML\]FFCC9927\.52±\\pm3\.8727\.73±\\pm2\.70\\rowcolor\[HTML\]F2F7FF GROM\\cellcolor\[HTML\]FF999A3\.52±\\pm1\.39\\cellcolor\[HTML\]FF999A4\.15±\\pm1\.32\\cellcolor\[HTML\]FFCC9928\.04±\\pm2\.73LLaMA3\.1\-8B\-InstructOriginal40\.42±\\pm4\.9236\.84±\\pm4\.2429\.87±\\pm2\.30ROME40\.46±\\pm4\.8536\.84±\\pm4\.16\\cellcolor\[HTML\]FFF8AD29\.99±\\pm2\.37MEMIT35\.15±\\pm3\.9934\.60±\\pm3\.15\\cellcolor\[HTML\]FFCC9930\.05±\\pm2\.46AlphaEdit\\cellcolor\[HTML\]FFF8AD34\.12±\\pm4\.16\\cellcolor\[HTML\]FFF8AD34\.19±\\pm3\.3329\.93±\\pm2\.49ZeroUnlearn\\cellcolor\[HTML\]FFCC9932\.67±\\pm3\.43\\cellcolor\[HTML\]FFCC9932\.39±\\pm3\.3429\.67±\\pm2\.36\\rowcolor\[HTML\]F2F7FF GROM\\cellcolor\[HTML\]FF999A4\.06±\\pm2\.65\\cellcolor\[HTML\]FF999A3\.83±\\pm2\.35\\cellcolor\[HTML\]FF999A31\.25±\\pm3\.06

Table 3:ZsRE, evaluated in the harness of\(Lin et al\.[2026](https://arxiv.org/html/2608.05783#bib.bib33)\), fifty unlearned facts, mean±\\pmstd\.
#### WMDP\.

WMDP\(Li et al\.[2024](https://arxiv.org/html/2608.05783#bib.bib31)\)targets the suppression of hazardous knowledge rather than the removal of memorized text\. Following\(Fan et al\.[2025](https://arxiv.org/html/2608.05783#bib.bib12)\), we report1−AccBio1\-\\mathrm\{AccBio\}on WMDP\-Bio as the forgetting metric, where larger values indicate stronger suppression of the hazardous slice, and MMLU accuracy\(Hendrycks et al\.[2020](https://arxiv.org/html/2608.05783#bib.bib20)\)as the utility metric\. Several baselines in Table[4](https://arxiv.org/html/2608.05783#Sx4.T4)reach strong suppression only at the cost of general ability, the clearest case being GradDiff, which collapses MMLU\. GROM instead retains by far the highest general\-knowledge utility at competitive suppression, giving the best Final Score, and it does so with a single closed\-form edit at one MLP layer\. Details are given in Appendix[E](https://arxiv.org/html/2608.05783#A5)\.

Unlearning EfficacyUtility PreservationSummaryMethod1−AccBio1\-\\mathrm\{AccBio\}\(↑\\uparrow\)MMLU\(↑\\uparrow\)Final Score\(↑\\uparrow\)Time \(m\)\(↓\\downarrow\)Original0\.270\.650\.46—UnDIAL0\.650\.450\.5520\.9GradDiff\\cellcolor\[HTML\]FFCC990\.730\.260\.5020\.2IDKNLL0\.66\\cellcolor\[HTML\]FFF8AD0\.470\.5720\.2IDKDPO0\.670\.440\.5623\.9NPO\\cellcolor\[HTML\]FFCC990\.73\\cellcolor\[HTML\]FFCC990\.51\\cellcolor\[HTML\]FFCC990\.6223\.9SimNPO\\cellcolor\[HTML\]FF999A0\.750\.44\\cellcolor\[HTML\]FFF8AD0\.6021\.6GROM\\cellcolor\[HTML\]FFF8AD0\.71\\cellcolor\[HTML\]FF999A0\.55\\cellcolor\[HTML\]FF999A0\.63\\cellcolor\[HTML\]FF999A0\.2

Table 4:Results on WMDP\-Bio \(Llama\-3\-8B\-Instruct\)\. Forgetting is measured by1−AccBio1\-\\mathrm\{AccBio\}on WMDP\-Bio, and utility is measured by MMLU accuracy\. Final Score is12​\(\(1−AccBio\)\+MMLU\)\\tfrac\{1\}\{2\}\\big\(\(1\-\\mathrm\{AccBio\}\)\+\\mathrm\{MMLU\}\\big\)\.
#### MUSE\.

MUSE\(Shi et al\.[2024b](https://arxiv.org/html/2608.05783#bib.bib46)\)addresses long\-form unlearning, where a model may leak verbatim passages as well as the underlying facts\. MUSE News splits BBC articles into disjoint forget, retain, and holdout sets\. MUSE Books is the harder entangled scenario, using the Harry Potter novels\(Eldan and Russinovich[2023](https://arxiv.org/html/2608.05783#bib.bib11); Wei et al\.[2024](https://arxiv.org/html/2608.05783#bib.bib51)\)as the forget set and a related fan wiki as the retain set, so the model must stop reproducing copyrighted text while still answering questions about closely related permissible material\. We report VerbMem \(verbatim regurgitation\), KnowMem \(forget\-corpus knowledge\), and PrivLeak \(membership leakage against the holdout set\), with retain KnowMem as the utility measure, all defined in Appendix[D](https://arxiv.org/html/2608.05783#A4)\. In Table[5](https://arxiv.org/html/2608.05783#Sx4.T5), GROM attains the best Final Score on both corpora, driving privacy leakage close to zero on News and, on Books, preserving the highest retain utility while nearly eliminating verbatim memorization, where several baselines erase forget\-set memorization only by destroying retain utility\.

Unlearning EfficacyUtilitySummaryMethodVerbMemDfD\_\{f\}\(↓\\downarrow\)KnowMemDfD\_\{f\}\(↓\\downarrow\)PrivLeak\(→0\\to 0\)KnowMemDrD\_\{r\}\(↑\\uparrow\)FinalScore \(↑\\uparrow\)Time \(m\)\(↓\\downarrow\)MUSE NewsOriginal58\.2962\.93\-98\.7154\.3140\.50—Retain20\.7533\.320\.0053\.7967\.88—GA\\cellcolor\[HTML\]FF999A0\.00\\cellcolor\[HTML\]FF999A0\.00\\cellcolor\[HTML\]FFCC9920\.140\.00\\cellcolor\[HTML\]FFF8AD46\.6418\.0GradDiff4\.85\\cellcolor\[HTML\]FFF8AD31\.29108\.1228\.2140\.0635\.4Task Vector77\.4258\.76\-100\.00\\cellcolor\[HTML\]FF999A47\.9434\.6118\.0NPO\\cellcolor\[HTML\]FFF8AD2\.5356\.93108\.91\\cellcolor\[HTML\]FFF8AD37\.5840\.7342\.4SimNPO\\cellcolor\[HTML\]FFCC992\.3444\.84\\cellcolor\[HTML\]FFF8AD72\.93\\cellcolor\[HTML\]FFCC9939\.65\\cellcolor\[HTML\]FFCC9949\.8136\.3GROM15\.68\\cellcolor\[HTML\]FFCC9924\.01\\cellcolor\[HTML\]FF999A\-3\.8026\.37\\cellcolor\[HTML\]FF999A55\.93\\cellcolor\[HTML\]FF999A0\.3MUSE BooksOriginal99\.5658\.32\-56\.3267\.0147\.80—Retain14\.3028\.900\.0074\.5080\.05—GA\\cellcolor\[HTML\]FF999A0\.00\\cellcolor\[HTML\]FF999A0\.00\\cellcolor\[HTML\]FFF8AD\-24\.070\.0045\.9926\.7GradDiff\\cellcolor\[HTML\]FF999A0\.00\\cellcolor\[HTML\]FF999A0\.00\-24\.590\.1345\.9752\.5Task Vector99\.31\\cellcolor\[HTML\]FFCC9935\.55\-83\.78\\cellcolor\[HTML\]FFCC9962\.5544\.8426\.7NPO\\cellcolor\[HTML\]FF999A0\.00\\cellcolor\[HTML\]FF999A0\.00\-31\.1723\.71\\cellcolor\[HTML\]FFF8AD56\.6662\.5SimNPO\\cellcolor\[HTML\]FF999A0\.00\\cellcolor\[HTML\]FF999A0\.00\\cellcolor\[HTML\]FFCC99\-19\.82\\cellcolor\[HTML\]FFF8AD48\.27\\cellcolor\[HTML\]FFCC9970\.8353\.8GROM\\cellcolor\[HTML\]FFCC992\.40\\cellcolor\[HTML\]FFF8AD36\.89\\cellcolor\[HTML\]FF999A\-0\.22\\cellcolor\[HTML\]FF999A65\.56\\cellcolor\[HTML\]FF999A76\.20\\cellcolor\[HTML\]FF999A0\.3

Table 5:Performance on MUSE News \(LLaMA2\-7B\) and MUSE Books \(ICLM\-7B\)\. Final Score is computed as12​\(KnowMem​Dr\+ForgetAvg\)\\frac\{1\}\{2\}\(\\mathrm\{KnowMem\}\\,D\_\{r\}\+\\mathrm\{ForgetAvg\}\), whereForgetAvg=\(100−VerbMem​Df\)\+\(100−KnowMem​Df\)\+\(100−\|PrivLeak\|\)3\\mathrm\{ForgetAvg\}=\\frac\{\(100\-\\mathrm\{VerbMem\}\\,D\_\{f\}\)\+\(100\-\\mathrm\{KnowMem\}\\,D\_\{f\}\)\+\(100\-\|\\mathrm\{PrivLeak\}\|\)\}\{3\}\. GROM attains the best Final Score among unlearning methods on both MUSE variants while being two orders of magnitude faster to apply\. On MUSE Books it also reaches the highest retain utility \(KnowMemDrD\_\{r\}\) while nearly eliminating verbatim memorization and driving privacy leakage to≈0\\approx 0\.
#### GROM is quantization\-robust\.

Unlearning can be undone simply by quantizing the model, which restores the supposedly forgotten content\(Zhang et al\.[2025](https://arxiv.org/html/2608.05783#bib.bib58)\)\. We probe this on MUSE at full precision and at 4\-bit in Table[6](https://arxiv.org/html/2608.05783#Sx4.T6)\. A rise in forget memorization \(VerbMemDfD\_\{f\}, KnowMemDfD\_\{f\}\) means the unlearning only hid the content\. SimNPO fails sharply, its verbatim memorization climbing from near zero back toward the original model on both corpora, with privacy leakage swinging back as well\. GROM does not recover, matching the gold Retrain model, which indicates that the closed\-form edit removes the targeted content rather than hiding it in low\-magnitude weights that rounding undoes\. Retain utility \(KnowMemDrD\_\{r\}\) drops at 4\-bit for every method including Original and Retrain\.

Unlearning EfficacyUtilitySummaryMethodPrec\.VerbMemDf​\(↓\)D\_\{f\}\\,\(\\downarrow\)KnowMemDf​\(↓\)D\_\{f\}\\,\(\\downarrow\)PrivLeak\(→0\)\(\\to 0\)KnowMemDr​\(↑\)D\_\{r\}\\,\(\\uparrow\)FinalScore \(↑\\uparrow\)MUSE NewsOriginalfull58\.2962\.93\-98\.7154\.3140\.504\-bit45\.855\.6\-99\.848\.540\.7Retrainfull20\.7533\.320\.0053\.7967\.884\-bit19\.736\.5\-2\.147\.764\.1SimNPOfull2\.3444\.8472\.9339\.6549\.814\-bit38\.748\.2\-99\.847\.442\.6GROMfull15\.6824\.01\-3\.8026\.3755\.934\-bit13\.522\.80\.520\.954\.3MUSE BooksOriginalfull99\.5658\.32\-56\.3267\.0147\.804\-bit94\.536\.2\-60\.450\.643\.5Retrainfull14\.3028\.900\.0074\.5080\.054\-bit14\.124\.5\-3\.662\.174\.0SimNPOfull0\.000\.00\-19\.8248\.2770\.834\-bit72\.534\.0\-59\.452\.448\.6GROMfull2\.4036\.89\-0\.2265\.5676\.204\-bit2\.929\.42\.145\.166\.8

Table 6:Robustness to quantization on MUSE, reproducing\(Zhang et al\.[2025](https://arxiv.org/html/2608.05783#bib.bib58)\)\. A rise in forget memorization at 4\-bit means the content was only hidden: SimNPO recovers much of it, whereas GROM stays as low as the gold Retrain model\.
#### Exact deletion auditability\.

We now use Theorem[2](https://arxiv.org/html/2608.05783#Thmtheorem2)as an evaluation tool, measuring how much each forget example contributed to the edit\. Writingγ=maxi⁡‖Δ​Pi‖F\\gamma=\\max\_\{i\}\\\|\\Delta P\_\{i\}\\\|\_\{F\}for the largest single\-example influence, the ratioγ/‖P⋆‖F\\gamma/\\\|P^\{\\star\}\\\|\_\{F\}is small on TOFU\-10%, so no single example dominates the update, and it matches a brute\-force re\-solve\. Figure[2](https://arxiv.org/html/2608.05783#Sx4.F2)shows a tight rather than heavy\-tailed distribution, and Appendix[A](https://arxiv.org/html/2608.05783#A1)confirms theO​\(1/s\)O\(1/s\)decay as the forget set grows\.

![Refer to caption](https://arxiv.org/html/2608.05783v1/x2.png)Figure 2:Per\-example deletion influence on TOFU\-10%\(layer1515, all400400forget examples, computed exactly via Theorem[2](https://arxiv.org/html/2608.05783#Thmtheorem2)\)\. Each bar is the influence‖Δ​Pi‖F\\\|\\Delta P\_\{i\}\\\|\_\{F\}of one forget example on the edit, as a percentage of‖P⋆‖F\\\|P^\{\\star\}\\\|\_\{F\}, sorted in decreasing order\. The most influential example \(dark\) definesγ\\gamma\.
#### Ablation: specificity weighting\.

We ask whether the specificity weightα\\alpha, rather than token suppression alone, is what balances forgetting against utility\. We therefore rerun the best configuration of each suppression benchmark withαj\\alpha\_\{j\}fixed to11and compare it against the original weighting \(Table[7](https://arxiv.org/html/2608.05783#Sx4.T7)\)\. Disablingα\\alphalowers every score, and the drop is largest on the benchmarks where the forget and retain sets share the most vocabulary \(MUSE News and TOFU\-5%\), because uniform suppression also removes tokens that are needed on the retain set\. The corollary is that the specificity weight, and not suppression alone, is the component that preserves utility\.

SpecificityEdited matrixBenchmarkα\\alphaoffo\_projup\_projq\_projv\_proj\\cellcolor\[HTML\]EAF1FBGROMTOFU\-5%0\.7270\.7930\.750\.380\.79\\cellcolor\[HTML\]EAF1FB0\.790TOFU\-10%0\.7270\.7060\.410\.400\.52\\cellcolor\[HTML\]EAF1FB0\.747WMDP—0\.620\.460\.460\.48\\cellcolor\[HTML\]EAF1FB0\.63MUSE\-News42\.152\.052\.240\.944\.3\\cellcolor\[HTML\]EAF1FB55\.5MUSE\-Books72\.468\.4862\.952\.469\.5\\cellcolor\[HTML\]EAF1FB76\.2

Table 7:Two ablation studies of GROM, theshadedreference column at right, which is theα\\alpha\-on MLPdown\_projedit\.*Specificity:*disablingα\\alpha\(uniform suppression\) lowers every score\.*Edited matrix:*each alternative matrix receives the strongest target it admits, and onlyo\_projon TOFU\-5% edges out GROM\.
#### Ablation: PCA analysis\.

To visualize the action of the closed\-form patch, we collect mean answer\-position LM\-head inputsxx, compareW​xWxwith\(W\+P\)​x\(W\+P\)x, and project each split separately onto three principal components\. Figure[3](https://arxiv.org/html/2608.05783#Sx4.F3)shows that the forget logits move substantially after the patch, while the retain logits remain nearly fixed\.

![Refer to caption](https://arxiv.org/html/2608.05783v1/x3.png)Figure 3:3D PCA diagnostic of the LM\-head patch\. Hollow markers denoteW​xWx, and filled markers denote\(W\+P\)​x\(W\+P\)x\. Forget points move visibly, whereas retain points nearly overlap\. Only meaningful movements are shown in bold\.
#### Ablation: which matrices are editable\.

Finally we apply the closed\-form edit to each candidate matrix in turn, giving every matrix the strongest target it admits \(Table[7](https://arxiv.org/html/2608.05783#Sx4.T7)\)\. Only the linear residual writers realize the edit, as the Method discussion of editable matrices predicts, most clearly on WMDP where every column shares a single target, and of the two we keep the MLP writer, which is better or tied throughout and alone moves membership leakage on the long\-form corpus\. A further ablation on*which*layers to edit, comparing the attribution\-selected band against early, random, narrower, and wider alternatives, is reported in Appendix[B](https://arxiv.org/html/2608.05783#A2)\. We also provide hyperparameter sensitivity analysis in Appendix[B\.2](https://arxiv.org/html/2608.05783#A2.SS2)\.

## Conclusions

We presented GROM, to our knowledge the first one\-shot unlearning method that achieves and surpasses the unlearning performance of iterative, optimization based methods\. We argue that our method might pave the way to efficiently unlearning even larger models, where training based approaches are infeasible on limited hardware\. Our method is somewhat related in spirit to locate\-and\-edit methods for knowledge editing\. It is an interesting avenue for future research to extend our method to the knowledge editing task\.

## References

- Belrose et al\. \(2023\)Belrose, N\.; Ostrovsky, I\.; McKinney, L\.; Furman, Z\.; Smith, L\.; Halawi, D\.; Biderman, S\.; and Steinhardt, J\. 2023\.Eliciting Latent Predictions from Transformers with the Tuned Lens\.*arXiv preprint arXiv:2303\.08112*\.
- Bourtoule et al\. \(2021\)Bourtoule, L\.; Chandrasekaran, V\.; Choquette\-Choo, C\.; Jia, H\.; Travers, A\.; Zhang, B\.; Lie, D\.; and Papernot, N\. 2021\.Machine Unlearning\.In*Proceedings of the IEEE Symposium on Security and Privacy*\.IEEE S&P 2021\.
- Cao and Yang \(2015\)Cao, Y\.; and Yang, J\. 2015\.Towards Making Systems Forget with Machine Unlearning\.In*Proceedings of the IEEE Symposium on Security and Privacy*\.IEEE S&P 2015\.
- Carlini et al\. \(2022\)Carlini, N\.; Ippolito, D\.; Jagielski, M\.; Lee, K\.; Tramer, F\.; and Zhang, C\. 2022\.Quantifying memorization across neural language models\.In*The Eleventh International Conference on Learning Representations*\.
- Carlini et al\. \(2021\)Carlini, N\.; Tramer, F\.; Wallace, E\.; Jagielski, M\.; Herbert\-Voss, A\.; Lee, K\.; Roberts, A\.; Brown, T\.; Song, D\.; Erlingsson, U\.; Oprea, A\.; and Raffel, C\. 2021\.Extracting Training Data from Large Language Models\.In*30th USENIX Security Symposium \(USENIX Security 21\)*\.
- Chen and Yang \(2023\)Chen, J\.; and Yang, D\. 2023\.Unlearn what you want to forget: Efficient unlearning for llms\.*arXiv preprint arXiv:2310\.20150*\.
- Dai et al\. \(2022\)Dai, D\.; Dong, L\.; Hao, Y\.; Sui, Z\.; Chang, B\.; and Wei, F\. 2022\.Knowledge Neurons in Pretrained Transformers\.In*Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 8493–8502\.
- Dong et al\. \(2025\)Dong, Y\. R\.; Lin, H\.; Belkin, M\.; Huerta, R\.; and Vulić, I\. 2025\.UNDIAL: Self\-Distillation with Adjusted Logits for Robust Unlearning in Large Language Models\.In*Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*\.ArXiv:2402\.10052\.
- Dorna et al\. \(2025\)Dorna, V\.; Mekala, A\.; Zhao, W\.; McCallum, A\.; Lipton, Z\. C\.; Kolter, J\. Z\.; and Maini, P\. 2025\.OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics\.*arXiv preprint arXiv:2506\.12618*\.
- Duan et al\. \(2024\)Duan, M\.; Suri, A\.; Mireshghallah, N\.; Min, S\.; Shi, W\.; Zettlemoyer, L\.; Tsvetkov, Y\.; Choi, Y\.; Evans, D\.; and Hajishirzi, H\. 2024\.Do membership inference attacks work on large language models?*arXiv preprint arXiv:2402\.07841*\.
- Eldan and Russinovich \(2023\)Eldan, R\.; and Russinovich, M\. 2023\.Who’s harry potter? approximate unlearning for LLMs\.
- Fan et al\. \(2025\)Fan, C\.; Liu, J\.; Lin, L\.; Jia, J\.; Zhang, R\.; Mei, S\.; and Liu, S\. 2025\.Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning\.In*The Thirty\-ninth Annual Conference on Neural Information Processing Systems*\.
- Fang et al\. \(2024\)Fang, J\.; Jiang, H\.; Wang, K\.; Ma, Y\.; Jie, S\.; Wang, X\.; He, X\.; and Chua, T\.\-S\. 2024\.AlphaEdit: Null\-Space Constrained Knowledge Editing for Language Models\.*arXiv preprint arXiv:2410\.02355*\.
- Geva et al\. \(2022\)Geva, M\.; Caciularu, A\.; Wang, K\.; and Goldberg, Y\. 2022\.Transformer Feed\-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space\.In*Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, 30–45\.
- Geva et al\. \(2021\)Geva, M\.; Schuster, R\.; Berant, J\.; and Levy, O\. 2021\.Transformer Feed\-Forward Layers Are Key\-Value Memories\.In*Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, 5484–5495\.
- Ginart et al\. \(2019\)Ginart, A\.; Guan, M\. Y\.; Valiant, G\.; and Zou, J\. 2019\.Making AI Forget You: Data Deletion in Machine Learning\.In*Advances in Neural Information Processing Systems*\.NeurIPS 2019\.
- Guo et al\. \(2019\)Guo, C\.; Goldstein, T\.; Hannun, A\.; and Van Der Maaten, L\. 2019\.Certified data removal from machine learning models\.*arXiv preprint arXiv:1911\.03030*\.
- Hager \(1989\)Hager, W\. W\. 1989\.Updating the Inverse of a Matrix\.*SIAM Review*, 31\(2\): 221–239\.
- Hase et al\. \(2023\)Hase, P\.; Bansal, M\.; Kim, B\.; and Ghandeharioun, A\. 2023\.Does Localization Inform Editing? Surprising Differences in Causality\-Based Localization vs\. Knowledge Editing in Language Models\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Hendrycks et al\. \(2020\)Hendrycks, D\.; Burns, C\.; Basart, S\.; Zou, A\.; Mazeika, M\.; Song, D\.; and Steinhardt, J\. 2020\.Measuring massive multitask language understanding\.*arXiv preprint arXiv:2009\.03300*\.
- Hoofnagle, Van Der Sloot, and Borgesius \(2019\)Hoofnagle, C\. J\.; Van Der Sloot, B\.; and Borgesius, F\. Z\. 2019\.The European Union general data protection regulation: what it is and what it means\.*Information & Communications Technology Law*, 28\(1\): 65–98\.
- Hu et al\. \(2024\)Hu, S\.; Fu, Y\.; Wu, Z\. S\.; and Smith, V\. 2024\.Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearning\.*arXiv preprint arXiv:2406\.13356*\.ICLR 2025\.
- Huang et al\. \(2024\)Huang, J\. Y\.; Zhou, W\.; Wang, F\.; Morstatter, F\.; Zhang, S\.; Poon, H\.; and Chen, M\. 2024\.Offset unlearning for large language models\.*arXiv preprint arXiv:2404\.11045*\.
- Ilharco et al\. \(2022\)Ilharco, G\.; Ribeiro, M\. T\.; Wortsman, M\.; Gururangan, S\.; Schmidt, L\.; Hajishirzi, H\.; and Farhadi, A\. 2022\.Editing models with task arithmetic\.*arXiv preprint arXiv:2212\.04089*\.
- Izzo et al\. \(2021\)Izzo, Z\.; Smart, M\. A\.; Chaudhuri, K\.; and Zou, J\. 2021\.Approximate data deletion from machine learning models\.In*International conference on artificial intelligence and statistics*, 2008–2016\. PMLR\.
- Ji et al\. \(2024\)Ji, J\.; Liu, Y\.; Zhang, Y\.; Liu, G\.; Kompella, R\. R\.; Liu, S\.; and Chang, S\. 2024\.Reversing the forget\-retain objectives: An efficient llm unlearning framework from logit difference\.*Advances in Neural Information Processing Systems*, 37: 12581–12611\.
- Jia et al\. \(2024\)Jia, J\.; Zhang, Y\.; Zhang, Y\.; Liu, J\.; Runwal, B\.; Diffenderfer, J\.; Kailkhura, B\.; and Liu, S\. 2024\.Soul: Unlocking the power of second\-order optimization for llm unlearning\.*arXiv preprint arXiv:2404\.18239*\.
- Jin et al\. \(2024\)Jin, Z\.; Cao, P\.; Wang, C\.; He, Z\.; Yuan, H\.; Li, J\.; Chen, Y\.; Liu, K\.; and Zhao, J\. 2024\.RWKU: Benchmarking Real\-World Knowledge Unlearning for Large Language Models\.In*The Thirty\-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track*\.
- Koh and Liang \(2017\)Koh, P\. W\.; and Liang, P\. 2017\.Understanding Black\-box Predictions via Influence Functions\.In*Proceedings of the 34th International Conference on Machine Learning \(ICML\)*, 1885–1894\.
- Levy et al\. \(2017\)Levy, O\.; Seo, M\.; Choi, E\.; and Zettlemoyer, L\. 2017\.Zero\-Shot Relation Extraction via Reading Comprehension\.In*Proceedings of the 21st Conference on Computational Natural Language Learning \(CoNLL\)*\.
- Li et al\. \(2024\)Li, N\.; Pan, A\.; Gopal, A\.; Yue, S\.; Berrios, D\.; Gatti, A\.; Li, J\. D\.; Dombrowski, A\.\-K\.; Goel, S\.; Phan, L\.; et al\. 2024\.The wmdp benchmark: Measuring and reducing malicious use with unlearning\.*arXiv preprint arXiv:2403\.03218*\.
- Lin \(2004\)Lin, C\.\-Y\. 2004\.Rouge: A package for automatic evaluation of summaries\.In*Text summarization branches out*, 74–81\.
- Lin et al\. \(2026\)Lin, Y\.; Yang, C\.; Xiang, Z\.; Song, Y\.; and Su, J\. 2026\.ZeroUnlearn: Few\-Shot Knowledge Unlearning in Large Language Models\.In*Proceedings of the 43rd International Conference on Machine Learning \(ICML\)*\.
- Liu, Liu, and Stone \(2022\)Liu, B\.; Liu, Q\.; and Stone, P\. 2022\.Continual learning and private unlearning\.In*Conference on Lifelong Learning Agents*, 243–254\. PMLR\.
- Liu et al\. \(2025\)Liu, S\.; Yao, Y\.; Jia, J\.; Casper, S\.; Baracaldo, N\.; Hase, P\.; Yao, Y\.; Liu, C\. Y\.; Xu, X\.; Li, H\.; et al\. 2025\.Rethinking machine unlearning for large language models\.*Nature Machine Intelligence*, 1–14\.
- Maini et al\. \(2024\)Maini, P\.; Feng, Z\.; Schwarzschild, A\.; Lipton, Z\. C\.; and Kolter, J\. Z\. 2024\.Tofu: A task of fictitious unlearning for llms\.*arXiv preprint arXiv:2401\.06121*\.
- Mekala et al\. \(2025\)Mekala, A\.; Dorna, V\.; Dubey, S\.; Lalwani, A\.; Koleczek, D\.; Rungta, M\.; Hasan, S\.; and Lobo, E\. 2025\.Alternate Preference Optimization for Unlearning Factual Knowledge in Large Language Models\.In Rambow, O\.; Wanner, L\.; Apidianaki, M\.; Al\-Khalifa, H\.; Eugenio, B\. D\.; and Schockaert, S\., eds\.,*Proceedings of the 31st International Conference on Computational Linguistics*, 3732–3752\. Abu Dhabi, UAE: Association for Computational Linguistics\.
- Meng et al\. \(2022\)Meng, K\.; Bau, D\.; Andonian, A\. J\.; and Belinkov, Y\. 2022\.Locating and Editing Factual Associations in GPT\.In*Advances in Neural Information Processing Systems*\.
- Meng et al\. \(2023\)Meng, K\.; Sharma, A\. S\.; Andonian, A\. J\.; Belinkov, Y\.; and Bau, D\. 2023\.Mass\-Editing Memory in a Transformer\.In*International Conference on Learning Representations*\.
- nostalgebraist \(2020\)nostalgebraist\. 2020\.Interpreting GPT: The Logit Lens\.https://www\.lesswrong\.com/posts/AcKRB8wDpdaN6v6ru/interpreting\-gpt\-the\-logit\-lens\.
- Pawelczyk, Neel, and Lakkaraju \(2023\)Pawelczyk, M\.; Neel, S\.; and Lakkaraju, H\. 2023\.In\-context unlearning: Language models as few shot unlearners\.*arXiv preprint arXiv:2310\.07579*\.
- Rafailov et al\. \(2023\)Rafailov, R\.; Sharma, A\.; Mitchell, E\.; Ermon, S\.; Manning, C\. D\.; and Finn, C\. 2023\.Direct Preference Optimization: Your Language Model is Secretly a Reward Model\.In*Advances in Neural Information Processing Systems*\.NeurIPS 2023\.
- Ren et al\. \(2025\)Ren, J\.; Xing, Y\.; Cui, Y\.; Aggarwal, C\. C\.; and Liu, H\. 2025\.SoK: Machine Unlearning for Large Language Models\.*arXiv preprint arXiv:2506\.09227*\.
- Rosen \(2011\)Rosen, J\. 2011\.The right to be forgotten\.*Stan\. L\. Rev\. Online*, 64: 88\.
- Shi et al\. \(2024a\)Shi, W\.; Ajith, A\.; Xia, M\.; Huang, Y\.; Liu, D\.; Blevins, T\.; Chen, D\.; and Zettlemoyer, L\. 2024a\.Detecting Pretraining Data from Large Language Models\.In*International Conference on Learning Representations \(ICLR\)*\.
- Shi et al\. \(2024b\)Shi, W\.; Lee, J\.; Huang, Y\.; Malladi, S\.; Zhao, J\.; Holtzman, A\.; Liu, D\.; Zettlemoyer, L\.; Smith, N\. A\.; and Zhang, C\. 2024b\.Muse: Machine unlearning six\-way evaluation for language models\.*arXiv preprint arXiv:2407\.06460*\.
- Thaker et al\. \(2024a\)Thaker, P\.; Maurya, Y\.; Hu, S\.; Wu, Z\. S\.; and Smith, V\. 2024a\.Guardrail baselines for unlearning in llms\.*arXiv preprint arXiv:2403\.03329*\.
- Thaker et al\. \(2024b\)Thaker, P\.; Maurya, Y\.; Hu, S\.; Wu, Z\. S\.; and Smith, V\. 2024b\.Position: LLM Unlearning Benchmarks are Weak Measures of Progress\.*arXiv preprint arXiv:2410\.02879*\.SaTML 2025\.
- Thudi et al\. \(2022\)Thudi, A\.; Deza, G\.; Chandrasekaran, V\.; and Papernot, N\. 2022\.Unrolling sgd: Understanding factors influencing machine unlearning\.In*2022 IEEE 7th European Symposium on Security and Privacy \(EuroS&P\)*, 303–319\. IEEE\.
- Wang et al\. \(2024\)Wang, B\.; Zi, Y\.; Sun, Y\.; Zhao, Y\.; and Qin, B\. 2024\.Rkld: Reverse kl\-divergence\-based knowledge distillation for unlearning personal information in large language models\.*arXiv preprint arXiv:2406\.01983*\.
- Wei et al\. \(2024\)Wei, B\.; Shi, W\.; Huang, Y\.; Smith, N\. A\.; Zhang, C\.; Zettlemoyer, L\.; Li, K\.; and Henderson, P\. 2024\.Evaluating copyright takedown methods for language models\.*Advances in Neural Information Processing Systems*, 37: 139114–139150\.
- Xu et al\. \(2025\)Xu, X\.; Yue, X\.; Liu, Y\.; Ye, Q\.; Hu, H\.; and Du, M\. 2025\.Unlearning Isn’t Deletion: Investigating Reversibility of Machine Unlearning in LLMs\.*arXiv preprint arXiv:2505\.16831*\.
- Yao, Xu, and Liu \(2024a\)Yao, Y\.; Xu, X\.; and Liu, Y\. 2024a\.Large Language Model Unlearning\.In*Advances in Neural Information Processing Systems*\.NeurIPS, arXiv:2310\.10683\.
- Yao, Xu, and Liu \(2024b\)Yao, Y\.; Xu, X\.; and Liu, Y\. 2024b\.Large language model unlearning\.*Advances in Neural Information Processing Systems*, 37: 105425–105475\.
- Yu et al\. \(2021\)Yu, D\.; Naik, S\.; Backurs, A\.; Gopi, S\.; Inan, H\. A\.; Kamath, G\.; Kulkarni, J\.; Lee, Y\. T\.; Manoel, A\.; Wutschitz, L\.; et al\. 2021\.Differentially private fine\-tuning of language models\.*arXiv preprint arXiv:2110\.06500*\.
- Zhang et al\. \(2024a\)Zhang, B\.; Chen, Z\.; Shen, C\.; and Li, J\. 2024a\.Verification of Machine Unlearning is Fragile\.In*Forty\-first International Conference on Machine Learning*\.
- Zhang et al\. \(2024b\)Zhang, R\.; Lin, L\.; Bai, Y\.; and Mei, S\. 2024b\.Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning\.In*Proceedings of the Conference on Language Modeling \(COLM\)*\.ArXiv:2404\.05868\.
- Zhang et al\. \(2025\)Zhang, Z\.; Wang, F\.; Li, X\.; Wu, Z\.; Tang, X\.; Liu, H\.; He, Q\.; Yin, W\.; and Wang, S\. 2025\.Catastrophic Failure of LLM Unlearning via Quantization\.In*The Thirteenth International Conference on Learning Representations*\.

## Appendix AAdditional Proofs

### A\.1Proof of the Closed\-Form Update

We prove the closed\-form update stated in Theorem[1](https://arxiv.org/html/2608.05783#Thmtheorem1)\. First, we record the invertibility of the matrix appearing in the solution\.

###### Lemma A\.1\.

LetXr∈ℝn×rX\_\{r\}\\in\\mathbb\{R\}^\{n\\times r\},Xf∈ℝn×sX\_\{f\}\\in\\mathbb\{R\}^\{n\\times s\}, and letwr,wf,μ\>0w\_\{r\},w\_\{f\},\\mu\>0\. Define

A\\displaystyle A=wrr​Xr​Xr⊤\+wfs​Xf​Xf⊤\+μ​In\.\\displaystyle=\\frac\{w\_\{r\}\}\{r\}X\_\{r\}X\_\{r\}^\{\\top\}\+\\frac\{w\_\{f\}\}\{s\}X\_\{f\}X\_\{f\}^\{\\top\}\+\\mu I\_\{n\}\.ThenAAis symmetric positive definite\. In particular,AAis invertible\.

###### Proof\.

First,AAis symmetric because

\(Xr​Xr⊤\)⊤\\displaystyle\(X\_\{r\}X\_\{r\}^\{\\top\}\)^\{\\top\}=Xr​Xr⊤,\\displaystyle=X\_\{r\}X\_\{r\}^\{\\top\},\(Xf​Xf⊤\)⊤\\displaystyle\(X\_\{f\}X\_\{f\}^\{\\top\}\)^\{\\top\}=Xf​Xf⊤,\\displaystyle=X\_\{f\}X\_\{f\}^\{\\top\},In⊤\\displaystyle I\_\{n\}^\{\\top\}=In\.\\displaystyle=I\_\{n\}\.HenceA⊤=AA^\{\\top\}=A\.

Now letv∈ℝnv\\in\\mathbb\{R\}^\{n\}be nonzero\. Then

v⊤​A​v\\displaystyle v^\{\\top\}Av=wrr​‖Xr⊤​v‖22\+wfs​‖Xf⊤​v‖22\+μ​‖v‖22\.\\displaystyle=\\frac\{w\_\{r\}\}\{r\}\\\|X\_\{r\}^\{\\top\}v\\\|\_\{2\}^\{2\}\+\\frac\{w\_\{f\}\}\{s\}\\\|X\_\{f\}^\{\\top\}v\\\|\_\{2\}^\{2\}\+\\mu\\\|v\\\|\_\{2\}^\{2\}\.The first two terms are nonnegative, and the last term is strictly positive sinceμ\>0\\mu\>0andv≠0v\\neq 0\. Thereforev⊤​A​v\>0v^\{\\top\}Av\>0for every nonzerovv, soAAis positive definite\. HenceAAis invertible\. ∎

###### Theorem A\.2\.

LetXr∈ℝn×rX\_\{r\}\\in\\mathbb\{R\}^\{n\\times r\},Xf∈ℝn×sX\_\{f\}\\in\\mathbb\{R\}^\{n\\times s\},D∈ℝm×sD\\in\\mathbb\{R\}^\{m\\times s\}, and letwr,wf,μ\>0w\_\{r\},w\_\{f\},\\mu\>0\. Define

A\\displaystyle A=wrr​Xr​Xr⊤\+wfs​Xf​Xf⊤\+μ​In∈ℝn×n\.\\displaystyle=\\frac\{w\_\{r\}\}\{r\}X\_\{r\}X\_\{r\}^\{\\top\}\+\\frac\{w\_\{f\}\}\{s\}X\_\{f\}X\_\{f\}^\{\\top\}\+\\mu I\_\{n\}\\in\\mathbb\{R\}^\{n\\times n\}\.Then:

1. 1\.The ridge\-regularized objective minP∈ℝm×n\\displaystyle\\min\_\{P\\in\\mathbb\{R\}^\{m\\times n\}\}wrr​‖P​Xr‖F2\+wfs​‖P​Xf−D‖F2\\displaystyle\\frac\{w\_\{r\}\}\{r\}\\,\\\|PX\_\{r\}\\\|\_\{F\}^\{2\}\+\\frac\{w\_\{f\}\}\{s\}\\,\\\|PX\_\{f\}\-D\\\|\_\{F\}^\{2\}\+μ​‖P‖F2\\displaystyle\+\\mu\\,\\\|P\\\|\_\{F\}^\{2\}is minimized by P⋆=wfs​D​Xf⊤​A−1\.P^\{\\star\}=\\frac\{w\_\{f\}\}\{s\}\\,DX\_\{f\}^\{\\top\}A^\{\-1\}\.Equivalently, P⋆\\displaystyle P^\{\\star\}=wfsDXf⊤\(wrrXrXr⊤\\displaystyle=\\frac\{w\_\{f\}\}\{s\}\\,DX\_\{f\}^\{\\top\}\\Big\(\\frac\{w\_\{r\}\}\{r\}X\_\{r\}X\_\{r\}^\{\\top\}\+wfsXfXf⊤\+μIn\)−1\.\\displaystyle\\qquad\\qquad\+\\frac\{w\_\{f\}\}\{s\}X\_\{f\}X\_\{f\}^\{\\top\}\+\\mu I\_\{n\}\\Big\)^\{\-1\}\.
2. 2\.The minimizerP⋆P^\{\\star\}is unique\.

###### Proof\.

Let

ℒ​\(P\)\\displaystyle\\mathcal\{L\}\(P\)=wrr​‖P​Xr‖F2\+wfs​‖P​Xf−D‖F2\+μ​‖P‖F2\.\\displaystyle=\\frac\{w\_\{r\}\}\{r\}\\\|PX\_\{r\}\\\|\_\{F\}^\{2\}\+\\frac\{w\_\{f\}\}\{s\}\\\|PX\_\{f\}\-D\\\|\_\{F\}^\{2\}\+\\mu\\\|P\\\|\_\{F\}^\{2\}\.We rewrite each term using

‖M‖F2=tr⁡\(M​M⊤\)\.\\\|M\\\|\_\{F\}^\{2\}=\\operatorname\{tr\}\(MM^\{\\top\}\)\.For the retain term,

‖P​Xr‖F2\\displaystyle\\\|PX\_\{r\}\\\|\_\{F\}^\{2\}=tr⁡\(\(P​Xr\)​\(P​Xr\)⊤\)\\displaystyle=\\operatorname\{tr\}\\big\(\(PX\_\{r\}\)\(PX\_\{r\}\)^\{\\top\}\\big\)=tr⁡\(P​Xr​Xr⊤​P⊤\)\.\\displaystyle=\\operatorname\{tr\}\(PX\_\{r\}X\_\{r\}^\{\\top\}P^\{\\top\}\)\.For the forget term,

‖P​Xf−D‖F2=tr⁡\(\(P​Xf−D\)​\(P​Xf−D\)⊤\)\.\\displaystyle\\\|PX\_\{f\}\-D\\\|\_\{F\}^\{2\}=\\operatorname\{tr\}\\big\(\(PX\_\{f\}\-D\)\(PX\_\{f\}\-D\)^\{\\top\}\\big\)\.Expanding the product gives

\(P​Xf−D\)​\(P​Xf−D\)⊤\\displaystyle\(PX\_\{f\}\-D\)\(PX\_\{f\}\-D\)^\{\\top\}=P​Xf​Xf⊤​P⊤−P​Xf​D⊤−D​Xf⊤​P⊤\+D​D⊤\.\\displaystyle\\quad=PX\_\{f\}X\_\{f\}^\{\\top\}P^\{\\top\}\-PX\_\{f\}D^\{\\top\}\-DX\_\{f\}^\{\\top\}P^\{\\top\}\+DD^\{\\top\}\.Therefore

‖P​Xf−D‖F2\\displaystyle\\\|PX\_\{f\}\-D\\\|\_\{F\}^\{2\}=tr⁡\(P​Xf​Xf⊤​P⊤\)−tr⁡\(P​Xf​D⊤\)\\displaystyle=\\operatorname\{tr\}\(PX\_\{f\}X\_\{f\}^\{\\top\}P^\{\\top\}\)\-\\operatorname\{tr\}\(PX\_\{f\}D^\{\\top\}\)−tr⁡\(D​Xf⊤​P⊤\)\+tr⁡\(D​D⊤\)\.\\displaystyle\\quad\-\\operatorname\{tr\}\(DX\_\{f\}^\{\\top\}P^\{\\top\}\)\+\\operatorname\{tr\}\(DD^\{\\top\}\)\.Usingtr⁡\(B\)=tr⁡\(B⊤\)\\operatorname\{tr\}\(B\)=\\operatorname\{tr\}\(B^\{\\top\}\), we have

tr⁡\(D​Xf⊤​P⊤\)\\displaystyle\\operatorname\{tr\}\(DX\_\{f\}^\{\\top\}P^\{\\top\}\)=tr⁡\(\(P​Xf​D⊤\)⊤\)\\displaystyle=\\operatorname\{tr\}\\big\(\(PX\_\{f\}D^\{\\top\}\)^\{\\top\}\\big\)=tr⁡\(P​Xf​D⊤\)\.\\displaystyle=\\operatorname\{tr\}\(PX\_\{f\}D^\{\\top\}\)\.Hence

‖P​Xf−D‖F2\\displaystyle\\\|PX\_\{f\}\-D\\\|\_\{F\}^\{2\}=tr⁡\(P​Xf​Xf⊤​P⊤\)−2​tr⁡\(P​Xf​D⊤\)\\displaystyle=\\operatorname\{tr\}\(PX\_\{f\}X\_\{f\}^\{\\top\}P^\{\\top\}\)\-2\\operatorname\{tr\}\(PX\_\{f\}D^\{\\top\}\)\+tr⁡\(D​D⊤\)\.\\displaystyle\\quad\+\\operatorname\{tr\}\(DD^\{\\top\}\)\.Finally,

‖P‖F2\\displaystyle\\\|P\\\|\_\{F\}^\{2\}=tr⁡\(P​P⊤\)=tr⁡\(P​In​P⊤\)\.\\displaystyle=\\operatorname\{tr\}\(PP^\{\\top\}\)=\\operatorname\{tr\}\(PI\_\{n\}P^\{\\top\}\)\.
Substituting these identities intoℒ​\(P\)\\mathcal\{L\}\(P\)gives

ℒ​\(P\)\\displaystyle\\mathcal\{L\}\(P\)=wrr​tr⁡\(P​Xr​Xr⊤​P⊤\)\+wfs​tr⁡\(P​Xf​Xf⊤​P⊤\)\\displaystyle=\\frac\{w\_\{r\}\}\{r\}\\operatorname\{tr\}\(PX\_\{r\}X\_\{r\}^\{\\top\}P^\{\\top\}\)\+\\frac\{w\_\{f\}\}\{s\}\\operatorname\{tr\}\(PX\_\{f\}X\_\{f\}^\{\\top\}P^\{\\top\}\)−2​wfs​tr⁡\(P​Xf​D⊤\)\+wfs​tr⁡\(D​D⊤\)\\displaystyle\\quad\-2\\frac\{w\_\{f\}\}\{s\}\\operatorname\{tr\}\(PX\_\{f\}D^\{\\top\}\)\+\\frac\{w\_\{f\}\}\{s\}\\operatorname\{tr\}\(DD^\{\\top\}\)\+μ​tr⁡\(P​In​P⊤\)\.\\displaystyle\\quad\+\\mu\\operatorname\{tr\}\(PI\_\{n\}P^\{\\top\}\)\.Collecting the quadratic terms inPP, and recalling that

A\\displaystyle A=wrr​Xr​Xr⊤\+wfs​Xf​Xf⊤\+μ​In,\\displaystyle=\\frac\{w\_\{r\}\}\{r\}X\_\{r\}X\_\{r\}^\{\\top\}\+\\frac\{w\_\{f\}\}\{s\}X\_\{f\}X\_\{f\}^\{\\top\}\+\\mu I\_\{n\},we obtain

ℒ​\(P\)\\displaystyle\\mathcal\{L\}\(P\)=tr⁡\(P​A​P⊤\)\\displaystyle=\\operatorname\{tr\}\(PAP^\{\\top\}\)−2​wfs​tr⁡\(P​Xf​D⊤\)\\displaystyle\\quad\-\\frac\{2w\_\{f\}\}\{s\}\\operatorname\{tr\}\(PX\_\{f\}D^\{\\top\}\)\+wfs​tr⁡\(D​D⊤\)\.\\displaystyle\\quad\+\\frac\{w\_\{f\}\}\{s\}\\operatorname\{tr\}\(DD^\{\\top\}\)\.The final term is independent ofPP, so it does not affect the minimizer\.

We now differentiate with respect toPP\. SinceA=A⊤A=A^\{\\top\}by the lemma,

∇Ptr⁡\(P​A​P⊤\)=2​P​A\.\\nabla\_\{P\}\\operatorname\{tr\}\(PAP^\{\\top\}\)=2PA\.Also,

tr⁡\(P​Xf​D⊤\)=tr⁡\(P​\(Xf​D⊤\)\),\\operatorname\{tr\}\(PX\_\{f\}D^\{\\top\}\)=\\operatorname\{tr\}\(P\(X\_\{f\}D^\{\\top\}\)\),so

∇Ptr⁡\(P​Xf​D⊤\)\\displaystyle\\nabla\_\{P\}\\operatorname\{tr\}\(PX\_\{f\}D^\{\\top\}\)=\(Xf​D⊤\)⊤\\displaystyle=\(X\_\{f\}D^\{\\top\}\)^\{\\top\}=D​Xf⊤\.\\displaystyle=DX\_\{f\}^\{\\top\}\.Therefore

∇Pℒ​\(P\)\\displaystyle\\nabla\_\{P\}\\mathcal\{L\}\(P\)=2​P​A−2​wfs​D​Xf⊤\.\\displaystyle=2PA\-2\\frac\{w\_\{f\}\}\{s\}DX\_\{f\}^\{\\top\}\.
At any stationary point, the gradient must vanish:

2​P​A−2​wfs​D​Xf⊤=0\.\\displaystyle 2PA\-2\\frac\{w\_\{f\}\}\{s\}DX\_\{f\}^\{\\top\}=0\.Equivalently,

P​A=wfs​D​Xf⊤\.PA=\\frac\{w\_\{f\}\}\{s\}DX\_\{f\}^\{\\top\}\.By the lemma,AAis positive definite and therefore invertible\. Multiplying on the right byA−1A^\{\-1\}gives

P=wfs​D​Xf⊤​A−1\.P=\\frac\{w\_\{f\}\}\{s\}DX\_\{f\}^\{\\top\}A^\{\-1\}\.Thus

P⋆=wfs​D​Xf⊤​A−1\.P^\{\\star\}=\\frac\{w\_\{f\}\}\{s\}DX\_\{f\}^\{\\top\}A^\{\-1\}\.
It remains to show that this stationary point is the unique global minimizer\. LetH∈ℝm×nH\\in\\mathbb\{R\}^\{m\\times n\}be any nonzero perturbation\. Consider

ℒ​\(P\+H\)\.\\mathcal\{L\}\(P\+H\)\.Only the quadratic part determines strict convexity, and its second\-order change is

tr⁡\(H​A​H⊤\)\.\\operatorname\{tr\}\(HAH^\{\\top\}\)\.Write the rows ofHHash1⊤,…,hm⊤h\_\{1\}^\{\\top\},\\ldots,h\_\{m\}^\{\\top\}, wherehi∈ℝnh\_\{i\}\\in\\mathbb\{R\}^\{n\}\. Then

tr⁡\(H​A​H⊤\)=∑i=1mhi⊤​A​hi\.\\operatorname\{tr\}\(HAH^\{\\top\}\)=\\sum\_\{i=1\}^\{m\}h\_\{i\}^\{\\top\}Ah\_\{i\}\.SinceAAis positive definite,

hi⊤​A​hi≥0h\_\{i\}^\{\\top\}Ah\_\{i\}\\geq 0for everyii, with equality only whenhi=0h\_\{i\}=0\. BecauseH≠0H\\neq 0, at least one rowhih\_\{i\}is nonzero, and therefore

tr⁡\(H​A​H⊤\)\>0\.\\operatorname\{tr\}\(HAH^\{\\top\}\)\>0\.Thus the quadratic part is strictly positive in every nonzero directionHH\. Consequently,ℒ\\mathcal\{L\}is strictly convex inPP\. Hence the stationary pointP⋆P^\{\\star\}is the unique global minimizer\. ∎

### A\.2Proof of the Deletion\-Influence Result

We prove the deletion\-influence result stated in Theorem[2](https://arxiv.org/html/2608.05783#Thmtheorem2)\. RecallP⋆=wfs​D​Xf⊤​A−1P^\{\\star\}=\\tfrac\{w\_\{f\}\}\{s\}\\,DX\_\{f\}^\{\\top\}A^\{\-1\}withA=wrr​Xr​Xr⊤\+wfs​Xf​Xf⊤\+μ​InA=\\tfrac\{w\_\{r\}\}\{r\}X\_\{r\}X\_\{r\}^\{\\top\}\+\\tfrac\{w\_\{f\}\}\{s\}X\_\{f\}X\_\{f\}^\{\\top\}\+\\mu I\_\{n\}\. Partition the forget keys and target by example:Xf=\[C1​⋯​CN\]X\_\{f\}=\[\\,C\_\{1\}\\,\\cdots\\,C\_\{N\}\\,\],D=\[D1​⋯​DN\]D=\[\\,D\_\{1\}\\,\\cdots\\,D\_\{N\}\\,\],Ci∈ℝn×ciC\_\{i\}\\in\\mathbb\{R\}^\{n\\times c\_\{i\}\},Di∈ℝm×ciD\_\{i\}\\in\\mathbb\{R\}^\{m\\times c\_\{i\}\}\. Herecic\_\{i\}is the number of answer tokens of exampleii, soCiC\_\{i\}holds the key vectors at those token positions and∑ici=s\\sum\_\{i\}c\_\{i\}=s\. Throughout we abbreviateλ=wfs\\lambda=\\tfrac\{w\_\{f\}\}\{s\}and holdλ\\lambdafixed at this value when an example is deleted\. That is, we treat the objective as weighting every forget token by the constantλ\\lambdarather than renormalizing by the reduced number of forget tokenss−cis\-c\_\{i\}, and all statements below are exact under this convention\.

#### The Sherman–Morrison–Woodbury identity\.

The proof rests on one standard matrix identity, which we state for completeness since it does the essential work\. For an invertibleA∈ℝn×nA\\in\\mathbb\{R\}^\{n\\times n\}and matricesU∈ℝn×cU\\in\\mathbb\{R\}^\{n\\times c\},V∈ℝc×nV\\in\\mathbb\{R\}^\{c\\times n\}and invertibleS∈ℝc×cS\\in\\mathbb\{R\}^\{c\\times c\}, the Sherman–Morrison–Woodbury identity\(Hager[1989](https://arxiv.org/html/2608.05783#bib.bib18)\)states

\(A\+U​S​V\)−1=A−1−A−1​U​\(S−1\+V​A−1​U\)−1​V​A−1,\(A\+USV\)^\{\-1\}=A^\{\-1\}\-A^\{\-1\}U\\big\(S^\{\-1\}\+VA^\{\-1\}U\\big\)^\{\-1\}VA^\{\-1\},\(1\)whenever the inverses involved exist\. Its content is that a rank\-ccperturbation ofAAproduces a rank\-ccperturbation ofA−1A^\{\-1\}, and that computing it costs only ac×cc\\times cinverse rather than a freshn×nn\\times none\. We use it in the*downdate*direction,U=CiU=C\_\{i\},V=Ci⊤V=C\_\{i\}^\{\\top\}andS=−λ​IciS=\-\\lambda I\_\{c\_\{i\}\}, which is exactly the perturbation caused by deleting one example\. Lemma[A\.3](https://arxiv.org/html/2608.05783#A1.Thmapxtheorem3)below records that special case and verifies it directly, so the proof is self\-contained\.

###### Lemma A\.3\(Rank\-cic\_\{i\}downdate\)\.

LetA⪰μ​InA\\succeq\\mu I\_\{n\}withμ\>0\\mu\>0and letA−i=A−λ​Ci​Ci⊤A\_\{\-i\}=A\-\\lambda C\_\{i\}C\_\{i\}^\{\\top\}be positive definite\. Put

Ri=A−1​Ci∈ℝn×ci,R\_\{i\}=A^\{\-1\}C\_\{i\}\\in\\mathbb\{R\}^\{n\\times c\_\{i\}\},Mi=Ici−λ​Ci⊤​A−1​Ci∈ℝci×ci\.M\_\{i\}=I\_\{c\_\{i\}\}\-\\lambda\\,C\_\{i\}^\{\\top\}A^\{\-1\}C\_\{i\}\\in\\mathbb\{R\}^\{c\_\{i\}\\times c\_\{i\}\}\.ThenMiM\_\{i\}is invertible andA−i−1=A−1\+λ​Ri​Mi−1​Ri⊤A\_\{\-i\}^\{\-1\}=A^\{\-1\}\+\\lambda\\,R\_\{i\}\\,M\_\{i\}^\{\-1\}R\_\{i\}^\{\\top\}\.

###### Proof\.

*MiM\_\{i\}is invertible\.*SupposeMi​v=0M\_\{i\}v=0for somev∈ℝciv\\in\\mathbb\{R\}^\{c\_\{i\}\}, that isv=λ​Ci⊤​A−1​Ci​vv=\\lambda\\,C\_\{i\}^\{\\top\}A^\{\-1\}C\_\{i\}v, and setu=A−1​Ci​vu=A^\{\-1\}C\_\{i\}v\. ThenCi⊤​u=v/λC\_\{i\}^\{\\top\}u=v/\\lambdaandu⊤​A​u=v⊤​Ci⊤​A−1​Ci​v=‖v‖2/λu^\{\\top\}Au=v^\{\\top\}C\_\{i\}^\{\\top\}A^\{\-1\}C\_\{i\}v=\\\|v\\\|^\{2\}/\\lambda, so

u⊤​A−i​u=u⊤​A​u−λ​‖Ci⊤​u‖2=1λ​‖v‖2−1λ​‖v‖2=0\.u^\{\\top\}A\_\{\-i\}u=u^\{\\top\}Au\-\\lambda\\\|C\_\{i\}^\{\\top\}u\\\|^\{2\}=\\tfrac\{1\}\{\\lambda\}\\\|v\\\|^\{2\}\-\\tfrac\{1\}\{\\lambda\}\\\|v\\\|^\{2\}=0\.SinceA−iA\_\{\-i\}is positive definite this forcesu=0u=0, hencev=λ​Ci⊤​u=0v=\\lambda C\_\{i\}^\{\\top\}u=0\. SoMiM\_\{i\}has trivial kernel and is invertible\.

*The formula\.*WriteKi=Ci⊤​A−1​CiK\_\{i\}=C\_\{i\}^\{\\top\}A^\{\-1\}C\_\{i\}, so thatMi=Ici−λ​KiM\_\{i\}=I\_\{c\_\{i\}\}\-\\lambda K\_\{i\}by definition\. Multiply the claimed inverse byA−iA\_\{\-i\}and expand\. UsingA​Ri=CiAR\_\{i\}=C\_\{i\}andCi⊤​A−1=Ri⊤C\_\{i\}^\{\\top\}A^\{\-1\}=R\_\{i\}^\{\\top\}, the product\(A−λ​Ci​Ci⊤\)​\(A−1\+λ​Ri​Mi−1​Ri⊤\)\(A\-\\lambda C\_\{i\}C\_\{i\}^\{\\top\}\)\(A^\{\-1\}\+\\lambda R\_\{i\}M\_\{i\}^\{\-1\}R\_\{i\}^\{\\top\}\)has four terms,

In\+λ​Ci​Mi−1​Ri⊤−λ​Ci​Ri⊤−λ2​Ci​Ki​Mi−1​Ri⊤\.I\_\{n\}\+\\lambda C\_\{i\}M\_\{i\}^\{\-1\}R\_\{i\}^\{\\top\}\-\\lambda C\_\{i\}R\_\{i\}^\{\\top\}\\\\ \{\}\-\\lambda^\{2\}C\_\{i\}K\_\{i\}M\_\{i\}^\{\-1\}R\_\{i\}^\{\\top\}\.The last three share the left factorCiC\_\{i\}and the right factorRi⊤R\_\{i\}^\{\\top\}, so they collect into

λ​Ci​\[\(Ici−λ​Ki\)​Mi−1−Ici\]​Ri⊤,\\lambda C\_\{i\}\\big\[\(I\_\{c\_\{i\}\}\-\\lambda K\_\{i\}\)M\_\{i\}^\{\-1\}\-I\_\{c\_\{i\}\}\\big\]R\_\{i\}^\{\\top\},and the bracket isMi​Mi−1−Ici=0M\_\{i\}M\_\{i\}^\{\-1\}\-I\_\{c\_\{i\}\}=0\. The product is thereforeInI\_\{n\}, which proves the claim\. ∎

###### Theorem A\.4\(Exact deletion influence\)\.

For eachii,A−i=A−λ​Ci​Ci⊤A\_\{\-i\}=A\-\\lambda C\_\{i\}C\_\{i\}^\{\\top\}is symmetric positive definite, and the update recomputed without exampleiiat fixed per\-token weightλ\\lambda,P−i⋆=λ​\(D​Xf⊤−Di​Ci⊤\)​A−i−1P^\{\\star\}\_\{\-i\}=\\lambda\\,\(DX\_\{f\}^\{\\top\}\-D\_\{i\}C\_\{i\}^\{\\top\}\)A\_\{\-i\}^\{\-1\}, satisfies

Δ​Pi:=P⋆−P−i⋆=Li​Ri⊤,Ri=A−1​Ci,\\Delta P\_\{i\}:=P^\{\\star\}\-P^\{\\star\}\_\{\-i\}=L\_\{i\}R\_\{i\}^\{\\top\},\\qquad R\_\{i\}=A^\{\-1\}C\_\{i\},Li=λ​\(Di−B−i​Ci​Mi−1\),Mi=Ici−λ​Ci⊤​A−1​Ci,L\_\{i\}=\\lambda\\big\(D\_\{i\}\-B\_\{\-i\}\\,C\_\{i\}\\,M\_\{i\}^\{\-1\}\\big\),\\quad M\_\{i\}=I\_\{c\_\{i\}\}\-\\lambda\\,C\_\{i\}^\{\\top\}A^\{\-1\}C\_\{i\},whereB−i=P⋆−λ​Di​Ci⊤​A−1B\_\{\-i\}=P^\{\\star\}\-\\lambda\\,D\_\{i\}C\_\{i\}^\{\\top\}A^\{\-1\}\. Hencerank⁡Δ​Pi≤ci\\operatorname\{rank\}\\Delta P\_\{i\}\\leq c\_\{i\}, andΔ​Pi\\Delta P\_\{i\}is obtained from the existingA−1A^\{\-1\}by oneci×cic\_\{i\}\\times c\_\{i\}inverse, with non×nn\\times nreinversion\.

###### Proof\.

We proceed in four steps\.

*Step 1: what deleting exampleiichanges\.*BecauseD​Xf⊤=∑kDk​Ck⊤DX\_\{f\}^\{\\top\}=\\sum\_\{k\}D\_\{k\}C\_\{k\}^\{\\top\}andXf​Xf⊤=∑kCk​Ck⊤X\_\{f\}X\_\{f\}^\{\\top\}=\\sum\_\{k\}C\_\{k\}C\_\{k\}^\{\\top\}split as sums over examples, removing exampleiisimply drops thek=ik=iterm from each\. The Gram matrix becomesA−i=A−λ​Ci​Ci⊤A\_\{\-i\}=A\-\\lambda C\_\{i\}C\_\{i\}^\{\\top\}and the numerator becomesλ​\(D​Xf⊤−Di​Ci⊤\)\\lambda\(DX\_\{f\}^\{\\top\}\-D\_\{i\}C\_\{i\}^\{\\top\}\), which we denoteBB\. Both are rank\-cic\_\{i\}modifications of quantities we have already computed\.

*Step 2:A−iA\_\{\-i\}stays positive definite\.*The retain and ridge terms ofAAare untouched by the deletion and the remaining forget terms∑k≠iCk​Ck⊤\\sum\_\{k\\neq i\}C\_\{k\}C\_\{k\}^\{\\top\}are positive semidefinite, soA−i⪰μ​InA\_\{\-i\}\\succeq\\mu I\_\{n\}, exactly as in the Lemma of Section[A\.1](https://arxiv.org/html/2608.05783#A1.SS1)\. In particularA−iA\_\{\-i\}is invertible, soP−i⋆P^\{\\star\}\_\{\-i\}is well defined, and Lemma[A\.3](https://arxiv.org/html/2608.05783#A1.Thmapxtheorem3)applies\.

*Step 3: invert the downdated Gram matrix\.*By Lemma[A\.3](https://arxiv.org/html/2608.05783#A1.Thmapxtheorem3),

A−i−1=A−1\+λ​Ri​Mi−1​Ri⊤\.A\_\{\-i\}^\{\-1\}=A^\{\-1\}\+\\lambda\\,R\_\{i\}M\_\{i\}^\{\-1\}R\_\{i\}^\{\\top\}\.\(2\)This is the only place the deletion enters, and it costs oneci×cic\_\{i\}\\times c\_\{i\}inverse against the cachedA−1A^\{\-1\}\.

*Step 4: substitute and collect\.*Insert \([2](https://arxiv.org/html/2608.05783#A1.E2)\) intoP−i⋆=B​A−i−1P^\{\\star\}\_\{\-i\}=BA\_\{\-i\}^\{\-1\}:

P−i⋆=B​A−1\+λ​B​Ri​Mi−1​Ri⊤\.P^\{\\star\}\_\{\-i\}=BA^\{\-1\}\+\\lambda\\,BR\_\{i\}M\_\{i\}^\{\-1\}R\_\{i\}^\{\\top\}\.We rewrite the two terms\. For the first,B​A−1=P⋆−λ​Di​Ci⊤​A−1BA^\{\-1\}=P^\{\\star\}\-\\lambda D\_\{i\}C\_\{i\}^\{\\top\}A^\{\-1\}by the definition ofBB, and the right\-hand side is preciselyB−iB\_\{\-i\}, soB​A−1=B−iBA^\{\-1\}=B\_\{\-i\}\. For the second,Ri=A−1​CiR\_\{i\}=A^\{\-1\}C\_\{i\}givesB​Ri=\(B​A−1\)​Ci=B−i​CiBR\_\{i\}=\\big\(BA^\{\-1\}\\big\)C\_\{i\}=B\_\{\-i\}C\_\{i\}\. Hence

P−i⋆=B−i\+λ​B−i​Ci​Mi−1​Ri⊤\.P^\{\\star\}\_\{\-i\}=B\_\{\-i\}\+\\lambda\\,B\_\{\-i\}C\_\{i\}M\_\{i\}^\{\-1\}R\_\{i\}^\{\\top\}\.Subtracting fromP⋆P^\{\\star\}and usingP⋆−B−i=λ​Di​Ci⊤​A−1=λ​Di​Ri⊤P^\{\\star\}\-B\_\{\-i\}=\\lambda D\_\{i\}C\_\{i\}^\{\\top\}A^\{\-1\}=\\lambda D\_\{i\}R\_\{i\}^\{\\top\},

Δ​Pi\\displaystyle\\Delta P\_\{i\}=λ​Di​Ri⊤−λ​B−i​Ci​Mi−1​Ri⊤\\displaystyle=\\lambda D\_\{i\}R\_\{i\}^\{\\top\}\-\\lambda B\_\{\-i\}C\_\{i\}M\_\{i\}^\{\-1\}R\_\{i\}^\{\\top\}=λ​\(Di−B−i​Ci​Mi−1\)​Ri⊤=Li​Ri⊤,\\displaystyle=\\lambda\\big\(D\_\{i\}\-B\_\{\-i\}C\_\{i\}M\_\{i\}^\{\-1\}\\big\)R\_\{i\}^\{\\top\}=L\_\{i\}R\_\{i\}^\{\\top\},which is the stated factorization\. The two terms have a direct reading: the first removes exampleii’s own contribution to the target, and the second corrects for the fact that deleting the example also shrinks the Gram matrix, which redistributes the edit across the examples that remain\. FinallyLi∈ℝm×ciL\_\{i\}\\in\\mathbb\{R\}^\{m\\times c\_\{i\}\}andRi∈ℝn×ciR\_\{i\}\\in\\mathbb\{R\}^\{n\\times c\_\{i\}\}, sorank⁡Δ​Pi≤ci\\operatorname\{rank\}\\Delta P\_\{i\}\\leq c\_\{i\}, and every quantity above involvesAAonly through the cachedA−1A^\{\-1\}\. ∎

#### Influence under varying forget\-set size and target\.

Recomputing the same diagnostic while varying the forget set confirms theO​\(1/s\)O\(1/s\)scaling of the Remark\. On TOFU\-10% at layer1515, the largest single\-example influenceγ\\gammarelative to‖P⋆‖F\\\|P^\{\\star\}\\\|\_\{F\}falls from0\.1730\.173withN=40N=40forget examples, to0\.0790\.079withN=120N=120, and to0\.0420\.042withN=400N=400\. Substituting the representation\-corruption target used for WMDP, at the sameN=400N=400, gives0\.0390\.039, so the two unlearning targets distribute influence alike\. All values are computed exactly by the Woodbury downdate of Theorem[2](https://arxiv.org/html/2608.05783#Thmtheorem2)\.

#### The bound is close to descriptive\.

On TOFU\-10% we computed‖Δ​Pi‖F\\\|\\Delta P\_\{i\}\\\|\_\{F\}for all400400forget examples and compared it against the factors appearing in the Remark\. Under the token\-suppression target the specificity mass of an example is‖Di‖F/β=∑jαj2\\\|D\_\{i\}\\\|\_\{F\}/\\beta=\\sqrt\{\\sum\_\{j\}\\alpha\_\{j\}^\{2\}\}, taken over its answer positions\. This quantity predicts the realized influence with Spearmanρ=0\.85\\rho=0\.85\(PearsonR2=0\.73R^\{2\}=0\.73\), whereas the answer\-token countcic\_\{i\}alone reaches onlyρ=0\.52\\rho=0\.52\(R2=0\.30R^\{2\}=0\.30\)\. The original model’s ROUGE\-L recall and answer probability on the same example give\|ρ\|<0\.1\|\\rho\|<0\.1, so influence is not a proxy for how strongly the answer was memorized\. The upper bound of the Remark therefore tracks the realized influence closely, and the dominant factor is how much forget\-specific content an example contributes rather than its length or its memorization strength\. Becauseα\\alphais computed from token counts alone, this diagnostic requires no additional model evaluation\.

## Appendix BAdditional Experiments

This section reports additional experiments that do not fit in the main text: an ablation on the choice of edited layers, and a sensitivity analysis over the hyperparameters of the objective\.

### B\.1Ablation: Choice of Edited Layers

We next ask whether the attribution\-selected layer band is genuinely useful, rather than only its cardinality or late\-layer location\. We present ablation results in Table[B\.1](https://arxiv.org/html/2608.05783#A2.T1)\. For each benchmark we keep the target, strength, retain weight, ridge scale, and feature budgets fixed, and change only the editeddown\_projlayers\. “Early” edits the firstkklayers, withkkmatched to the selected band\. “Random” is a fixed noncontiguous same\-cardinality control drawn once from a fixed seed\. “Single” edits only the last layer of the selected band, “Narrow” edits its last two layers, and “Wide” extends the selected band toward earlier layers\. Early layers destroy utility and the single and narrow variants under\-edit\. The random control stays below the selected band on all four benchmarks, but by an erratic margin, from0\.0420\.042on TOFU\-10% to an outright collapse on TOFU\-5%, so same\-cardinality alone does not recover the attribution band\. The selected band is therefore the most reliable choice, and it is the best variant on every benchmark, including against the wider edit that spends more layers to get there\.

BenchmarkEarlyRandomSingleNarrow\\cellcolor\[HTML\]EAF1FBSelectedbandWideTOFU\-10%0\.492k=5k=50\.705k=5k=50\.672k=1k=10\.716k=2k=2\\cellcolor\[HTML\]EAF1FB0\.747k=5k=50\.740k=8k=8TOFU\-5%0\.494k=6k=60\.492k=6k=60\.710k=1k=10\.723k=2k=2\\cellcolor\[HTML\]EAF1FB0\.790k=6k=60\.735k=12k=12MUSE\-News45\.80k=4k=449\.11k=4k=442\.42k=1k=144\.17k=2k=2\\cellcolor\[HTML\]EAF1FB55\.49k=4k=445\.53k=8k=8MUSE\-Books46\.48k=4k=471\.60k=4k=458\.92k=1k=164\.66k=2k=2\\cellcolor\[HTML\]EAF1FB76\.27k=4k=472\.33k=8k=8

Table B\.1:Layer\-selection ablation with all non\-layer hyperparameters fixed\. Each cell reports the score and number of edited MLPdown\_projlayerskk\.
### B\.2Hyperparameter Sensitivity

#### Only three hyperparameters are free\.

The objective carries three weights,wfw\_\{f\},wrw\_\{r\}, andμ\\mu, and the suppression target adds an edit strengthβ\\beta\. One of the four is redundant\. Because we set the ridge relative to the data scale,μ=ρ​g¯\\mu=\\rho\\,\\bar\{g\}withg¯\\bar\{g\}the mean diagonal entry ofAA, rescalingwfw\_\{f\}andwrw\_\{r\}by a common factor multiplies bothAAand the numeratorwfs​D​Xf⊤\\tfrac\{w\_\{f\}\}\{s\}DX\_\{f\}^\{\\top\}of Theorem[1](https://arxiv.org/html/2608.05783#Thmtheorem1)by that factor, leaving the update unchanged:

P⋆​\(wf,wr,ρ\)=P⋆​\(1,wr/wf,ρ\)\.P^\{\\star\}\(w\_\{f\},w\_\{r\},\\rho\)\\;=\\;P^\{\\star\}\\\!\\left\(1,\\;w\_\{r\}/w\_\{f\},\\;\\rho\\right\)\.Only the ratiowr/wfw\_\{r\}/w\_\{f\}is a hyperparameter, and we fix the gaugewf=1w\_\{f\}=1throughout\. We confirmed this empirically rather than only asserting it: configurations related by the identity above agree on every reported metric to within the seed\-to\-seed spread, with residual differences consistent with rounding when the update is written into bfloat16 weights\. The free hyperparameters are therefore the edit strengthβ\\beta, the retain weightwrw\_\{r\}, and the ridge scaleρ\\rho; the edit widthkkis treated separately in Section[B\.1](https://arxiv.org/html/2608.05783#A2.SS1)\.

#### Findings\.

Table[B\.2](https://arxiv.org/html/2608.05783#A2.T2)shows that the three hyperparameters play qualitatively different roles\.

*Edit strengthβ\\betabuys forgetting almost for free up to the tuned value\.*Atβ=0\\beta=0the update is exactlyP=0P=0and the model is unchanged\. Asβ\\betagrows to4545, forget efficacy rises from0\.1990\.199to0\.8660\.866while model utility stays at the unedited level \(0\.601→0\.6000\.601\\rightarrow 0\.600\): roughly five sixths of the attainable forgetting costs no measurable utility\. Utility only begins to pay past the tunedβ=65\\beta=65and then falls sharply, reaching0\.1270\.127atβ=200\\beta=200\.

*The retain weightwrw\_\{r\}is the term that must be tuned\.*It is the only axis whose two ends both fail outright\. Without retain anchoring \(wr=0w\_\{r\}=0\) the edit destroys the model, driving model utility to0\.0000\.000and retain ROUGE\-L to0\.0030\.003while forgetting saturates, which is the degenerate solution the retain term exists to prevent\. Over\-weighting it \(wr=1600w\_\{r\}=1600\) suppresses the edit instead, and forget efficacy collapses to0\.3670\.367\. Between these, the Final Score stays within0\.030\.03of its maximum forwr∈\[50,200\]w\_\{r\}\\in\[50,200\]\.

*The ridgeρ\\rhois a numerical safeguard rather than a tuning knob\.*Acrossρ∈\[0\.001,0\.1\]\\rho\\in\[0\.001,0\.1\], two orders of magnitude, the Final Score moves by0\.0050\.005, which is smaller than the0\.0060\.006seed spread\. Performance degrades only onceρ≥0\.3\\rho\\geq 0\.3, where the penalty on‖P‖F\\\|P\\\|\_\{F\}starts to shrink the update itself\. In practiceρ\\rhocan be set to any small value that keepsAAcomfortably conditioned\.

*Edit strength*β\\beta\(atwr=100w\_\{r\}=100,ρ=0\.03\\rho=0\.03\)β\\beta0102545\\cellcolor\[HTML\]EAF1FB6590130200320MU0\.6010\.5950\.5980\.600\\cellcolor\[HTML\]EAF1FB0\.5850\.5340\.4010\.1270\.069F0\.1990\.6010\.8170\.866\\cellcolor\[HTML\]EAF1FB0\.9060\.9370\.9570\.9750\.984Final0\.4000\.5980\.7080\.733\\cellcolor\[HTML\]EAF1FB0\.7460\.7350\.6790\.5510\.527*Retain weight*wrw\_\{r\}\(atβ=65\\beta=65,ρ=0\.03\\rho=0\.03\)wrw\_\{r\}01102550\\cellcolor\[HTML\]EAF1FB1002004001600MU0\.0000\.0000\.0360\.2950\.508\\cellcolor\[HTML\]EAF1FB0\.5850\.6050\.5970\.598F0\.9880\.9850\.9790\.9650\.943\\cellcolor\[HTML\]EAF1FB0\.9060\.8410\.7560\.367Final0\.4940\.4930\.5070\.6300\.726\\cellcolor\[HTML\]EAF1FB0\.7460\.7230\.6770\.483*Ridge scale*ρ\\rho\(atβ=65\\beta=65,wr=100w\_\{r\}=100;μ=ρ​g¯\\mu=\\rho\\,\\bar\{g\}\)ρ\\rho0\.0010\.0030\.01\\cellcolor\[HTML\]EAF1FB0\.030\.10\.31\.0MU0\.5640\.5650\.574\\cellcolor\[HTML\]EAF1FB0\.5850\.6030\.5960\.595F0\.9180\.9160\.912\\cellcolor\[HTML\]EAF1FB0\.9060\.8850\.8380\.758Final0\.7410\.7410\.743\\cellcolor\[HTML\]EAF1FB0\.7460\.7440\.7170\.677Table B\.2:One\-at\-a\-time hyperparameter sensitivity of GROM on TOFU\-10%\. Each block varies a single hyperparameter and holds the rest at the tuned configuration \(shaded\)\. MU is model utility, F is forget efficacy\(\(1−Rouge\-L\)\+\(1−Prob\.\)\+\(1−Extr\. Str\.\)\)/3\\big\(\(1\-\\text\{Rouge\-L\}\)\+\(1\-\\text\{Prob\.\}\)\+\(1\-\\text\{Extr\.\\ Str\.\}\)\\big\)/3, and Final is their mean\. Repeated seeds at the tuned setting span0\.0140\.014in MU,0\.0020\.002in F, and0\.0060\.006in Final\.

## Appendix CUnlearning on TOFU

This section describes the evaluation metrics used in our TOFU experiments\. We evaluate on the TOFU 5% and 10% forget splits\.

#### Answer probability\.

For each example in the retain and forget splits, we measure how likely the model is to generate the reference answer conditioned on the corresponding question\. Specifically, for a questionqqand answeraa, we compute the length\-normalized conditional likelihood

P​\(a∣q\)1/\|a\|,P\(a\\mid q\)^\{1/\|a\|\},where\|a\|\|a\|denotes the number of tokens in the answer\.

For the real authors and world facts subsets, each question is associated with five candidate answers: one correct answera0a\_\{0\}and four perturbed, incorrect alternatives\{a~1,a~2,a~3,a~4\}\\\{\\tilde\{a\}\_\{1\},\\tilde\{a\}\_\{2\},\\tilde\{a\}\_\{3\},\\tilde\{a\}\_\{4\}\\\}\. To quantify the model’s preference for the correct answer, we use the normalized probability of the correct answer among all five candidates,

P​\(a0∣q\)1/\|a0\|P​\(a0∣q\)1/\|a0\|\+∑i=14P​\(a~i∣q\)1/\|a~i\|\.\\frac\{P\(a\_\{0\}\\mid q\)^\{1/\|a\_\{0\}\|\}\}\{P\(a\_\{0\}\\mid q\)^\{1/\|a\_\{0\}\|\}\+\\sum\_\{i=1\}^\{4\}P\(\\tilde\{a\}\_\{i\}\\mid q\)^\{1/\|\\tilde\{a\}\_\{i\}\|\}\}\.

#### Truth ratio\.

The truth ratio measures the model’s relative preference for incorrect answers over a correct paraphrased answer\. Leta^\\hat\{a\}denote a paraphrase of the correct answer and letA=\{a~1,a~2,…\}A=\\\{\\tilde\{a\}\_\{1\},\\tilde\{a\}\_\{2\},\\ldots\\\}be the set of perturbed incorrect answers\. We first compute the geometric mean of the length\-normalized likelihoods assigned to the perturbed answers, and then divide this value by the length\-normalized likelihood assigned to the paraphrased correct answer:

Rtruth=\(∏i=1\|A\|P​\(a~i∣q\)1/\|a~i\|\)1/\|A\|P​\(a^∣q\)1/\|a^\|\.R\_\{\\text\{truth\}\}=\\frac\{\\left\(\\prod\_\{i=1\}^\{\|A\|\}P\(\\tilde\{a\}\_\{i\}\\mid q\)^\{1/\|\\tilde\{a\}\_\{i\}\|\}\\right\)^\{1/\|A\|\}\}\{P\(\\hat\{a\}\\mid q\)^\{1/\|\\hat\{a\}\|\}\}\.For the real authors and world facts subsets, paraphrased answers are not provided\. In these cases, we use the original correct answeraain place ofa^\\hat\{a\}\.

As defined, a lowerRtruthR\_\{\\text\{truth\}\}indicates a stronger preference for the correct answer\. The truth\-ratio values reported in our tables, and the ones entering the model\-utility aggregate, are therefore notRtruthR\_\{\\text\{truth\}\}itself but the per\-example transformmax⁡\(0,1−Rtruth\)\\max\(0,\\,1\-R\_\{\\text\{truth\}\}\), averaged over the subset, so that higher reported values are better\. On the forget split, where a truth ratio close to11is desirable, the evaluation pipeline instead aggregatesmin⁡\(Rtruth,1/Rtruth\)\\min\(R\_\{\\text\{truth\}\},\\,1/R\_\{\\text\{truth\}\}\), but our tables report the truth ratio only on the utility subsets\.

#### ROUGE\-L\.

For all TOFU subsets, we report ROUGE\-L recall\(Lin[2004](https://arxiv.org/html/2608.05783#bib.bib32)\)between the ground\-truth responses from the forget split and the model generations obtained after unlearning\.

#### Extraction strength\.

We use extraction strength to evaluate how much of an answer that should have been forgotten the model can still complete once it is conditioned on the beginning of that answer\. Following the OpenUnlearning implementation, the metric is computed with teacher forcing\. The question and reference answer are passed through the model, and at every answer position we record the greedy \(argmax\) next\-token prediction\. For a question\-answer pair\(q,a\)\(q,a\)with answer tokensa1,…,a\|a\|a\_\{1\},\\dots,a\_\{\|a\|\}, letk≥0k\\geq 0be the smallest number of leading answer positions that must be discarded so that the greedy predictions agree with the reference tokens at all remaining positions\. The suffix from positionk\+1k\+1onward is thus the longest tail of the answer that the model reproduces correctly on its own, and the extraction strength is

Sext​\(q,a\)=1−k\|a\|,S\_\{\\text\{ext\}\}\(q,a\)=1\-\\frac\{k\}\{\|a\|\},the fraction of the answer recoverable in this way\. Intuitively, a smallkkmeans an attacker needs to supply only a short prefix of the answer before the model completes the rest verbatim\. Higher extraction strength indicates that the target answer remains easier to elicit from the model, which corresponds to weaker unlearning\. Lower extraction strength suggests stronger resistance to extraction\.

#### Model utility\.

Finally, we report an aggregate model utility score\. This score is computed as the harmonic mean of nine quantities: answer probability, truth ratio, and ROUGE\-L recall, each evaluated on the retain, real authors, and world facts subsets\. Higher model utility indicates better overall performance after unlearning\.

#### Experimental details\.

Our method performs a single closed\-form update per layer and involves no gradient\-based training\. For both TOFU settings we apply the token\-suppression target to the MLP down\-projection of a contiguous band of late layers, selected by the logit\-lens attribution described in the main text\. On TOFU\-10% \(LLaMA\-3\.2\-1B\-Instruct\) we edit layers1111through1515with edit strengthβ=65\\beta=65, retain weightwr=100w\_\{r\}=100, and ridge scaleρ=0\.03\\rho=0\.03\. On TOFU\-5% \(LLaMA\-2\-7B\-Chat\) we edit layers2626through3131withβ=1000\\beta=1000,wr=300w\_\{r\}=300, andρ=0\.03\\rho=0\.03\. In both cases the ridge coefficient isμ=ρ​g¯\\mu=\\rho\\,\\bar\{g\}, whereg¯\\bar\{g\}is the mean diagonal entry of the matrixAAin Theorem[1](https://arxiv.org/html/2608.05783#Thmtheorem1), and the layers are edited sequentially with the keys recomputed after each edit\.

#### Layer selection\.

For TOFU we use the token\-suppression attribution score from the main text\. For each candidate layer, the score compares the layer’s contribution to forget\-set gold\-token logits against its contribution to retain\-token logits\. We then choose a contiguous late\-layer window with the highest average score for the chosen edit widthkk\. This gives layers1111–1515for TOFU\-10% \(k=5k=5\) and layers2626–3131for TOFU\-5% \(k=6k=6\)\. After each layer edit, we recompute the keys before editing the next layer so that later updates are computed on the current edited model\.

#### Observed results\.

On TOFU\-5%, GROM obtains a Final Score of0\.790\.79, improving over the strongest baseline scores in our comparison while also achieving the highest MU score,0\.620\.62\. The forget\-set metrics are nearly saturated:11\-Rouge\-L is0\.950\.95,11\-Prob\. is1\.001\.00, and11\-Extraction Strength is0\.970\.97\. At the same time, retain\-set quality remains substantially higher than for the strongest forgetting baselines; for example, retain ROUGE\-L and retain probability are0\.830\.83and0\.780\.78, respectively, compared with0\.540\.54and0\.560\.56for SimNPO\. This indicates that the edit removes the selected fictitious\-author facts without broadly suppressing the neighboring retain distribution\.

TOFU\-10% is the more difficult TOFU setting because the forget set is larger and contains a broader set of author\-specific associations\. In this setting GROM obtains the best Final Score,0\.750\.75, and the best MU score,0\.600\.60\. The method reaches11\-Prob\. of1\.001\.00and11\-Extraction Strength of0\.940\.94on the forget set, while retaining the best truth\-ratio scores on Real Authors, World Facts, and the retain split\. The baseline pattern is instructive: IDKDPO reaches very strong forgetting but drops MU to0\.520\.52, whereas SimNPO keeps a more balanced profile but reaches a lower Final Score of0\.700\.70\. GROM therefore sits at a better operating point on the forgetting–utility frontier\. The measured update time is0\.50\.5minutes on one H100, compared with2\.52\.5minutes for SimNPO and6\.96\.9minutes for AltPO and IDKDPO\.

#### Full per\-subset results\.

Tables[C\.1](https://arxiv.org/html/2608.05783#A3.T1)and[C\.2](https://arxiv.org/html/2608.05783#A3.T2)give the complete breakdown behind the model\-utility column of Table[2](https://arxiv.org/html/2608.05783#Sx4.T2), reporting ROUGE\-L, answer probability, and truth ratio separately on the retain, Real Authors, and World Facts subsets\.

Unlearning EfficacyUtility PreservationSummaryMethodForget SetReal AuthorsWorld FactsRetain SetMU \(↑\\uparrow\)Final Score \(↑\\uparrow\)Time \(m\) \(↓\\downarrow\)1\-Rouge\-L↑\\uparrow1\-Prob\.↑\\uparrow1\-Extr\. Strength↑\\uparrowRouge\-L↑\\uparrowProb\.↑\\uparrowTruth ratio↑\\uparrowRouge\-L↑\\uparrowProb\.↑\\uparrowTruth ratio↑\\uparrowRouge\-L↑\\uparrowProb\.↑\\uparrowTruth ratio↑\\uparrowOriginal0\.040\.010\.050\.930\.440\.580\.910\.430\.550\.980\.990\.480\.620\.33—Retain0\.610\.850\.930\.920\.440\.570\.900\.430\.540\.970\.990\.480\.620\.71—GradDiff\\cellcolor\[HTML\]FF999A1\.00\\cellcolor\[HTML\]FF999A1\.00\\cellcolor\[HTML\]FFCC990\.960\.59\\cellcolor\[HTML\]FF999A0\.59\\cellcolor\[HTML\]FF999A0\.81\\cellcolor\[HTML\]FFCC990\.880\.460\.590\.420\.490\.480\.56\\cellcolor\[HTML\]FFCC990\.772\.4IDKDPO\\cellcolor\[HTML\]FFCC990\.980\.400\.850\.650\.480\.630\.820\.440\.550\.55\\cellcolor\[HTML\]FF999A0\.86\\cellcolor\[HTML\]FFCC990\.57\\cellcolor\[HTML\]FFF8AD0\.570\.663\.3RKLD0\.69\\cellcolor\[HTML\]FFF8AD0\.96\\cellcolor\[HTML\]FFF8AD0\.92\\cellcolor\[HTML\]FF999A0\.920\.470\.61\\cellcolor\[HTML\]FFF8AD0\.870\.470\.58\\cellcolor\[HTML\]FFF8AD0\.580\.52\\cellcolor\[HTML\]FFF8AD0\.560\.560\.712\.9NPO0\.730\.940\.90\\cellcolor\[HTML\]FFCC990\.91\\cellcolor\[HTML\]FFF8AD0\.500\.62\\cellcolor\[HTML\]FF999A0\.90\\cellcolor\[HTML\]FF999A0\.50\\cellcolor\[HTML\]FFCC990\.610\.470\.51\\cellcolor\[HTML\]FFCC990\.57\\cellcolor\[HTML\]FFF8AD0\.570\.712\.9SimNPO0\.74\\cellcolor\[HTML\]FFCC990\.97\\cellcolor\[HTML\]FFF8AD0\.92\\cellcolor\[HTML\]FFF8AD0\.90\\cellcolor\[HTML\]FFF8AD0\.50\\cellcolor\[HTML\]FFF8AD0\.64\\cellcolor\[HTML\]FF999A0\.90\\cellcolor\[HTML\]FFF8AD0\.48\\cellcolor\[HTML\]FFF8AD0\.600\.540\.56\\cellcolor\[HTML\]FF999A0\.58\\cellcolor\[HTML\]FFCC990\.58\\cellcolor\[HTML\]FFF8AD0\.732\.6ROME0\.820\.690\.900\.620\.440\.560\.680\.440\.58\\cellcolor\[HTML\]FFCC990\.67\\cellcolor\[HTML\]FFCC990\.840\.48\\cellcolor\[HTML\]FFF8AD0\.570\.68\\cellcolor\[HTML\]FF999A0\.6MEMIT0\.730\.690\.890\.58\\cellcolor\[HTML\]FFCC990\.51\\cellcolor\[HTML\]FFCC990\.650\.66\\cellcolor\[HTML\]FFCC990\.49\\cellcolor\[HTML\]FF999A0\.620\.450\.500\.430\.530\.65\\cellcolor\[HTML\]FFCC990\.7GROM\\cellcolor\[HTML\]FFF8AD0\.95\\cellcolor\[HTML\]FF999A1\.00\\cellcolor\[HTML\]FF999A0\.970\.720\.49\\cellcolor\[HTML\]FFCC990\.650\.850\.46\\cellcolor\[HTML\]FF999A0\.62\\cellcolor\[HTML\]FF999A0\.83\\cellcolor\[HTML\]FFF8AD0\.780\.47\\cellcolor\[HTML\]FF999A0\.62\\cellcolor\[HTML\]FF999A0\.79\\cellcolor\[HTML\]FFF8AD1\.8

Table C\.1:Results on TOFU\-5% \(LLaMA2\-7B\-Chat\)\. Colors indicate rank \(red: best, orange: second, yellow: third\)\. Final Score is computed as12​\(MU\+\(1​\-Rouge\-L\)\+\(1​\-Prob\.\)\+\(1​\-Extr\. Strength\)3\)\\frac\{1\}\{2\}\\left\(\\mathrm\{MU\}\+\\frac\{\(1\\text\{\-Rouge\-L\}\)\+\(1\\text\{\-Prob\.\}\)\+\(1\\text\{\-Extr\.\\ Strength\}\)\}\{3\}\\right\)\. Rows below the dashed rule are the locate\-then\-edit knowledge editors, which apply faster on this small forget set but reach a substantially worse trade\-off\. GROM achieves the best forgetting\-utility trade\-off\.Unlearning EfficacyUtility PreservationSummaryMethodForget SetReal AuthorsWorld FactsRetain SetMU \(↑\\uparrow\)Final Score \(↑\\uparrow\)Time \(m\) \(↓\\downarrow\)1\-Rouge\-L↑\\uparrow1\-Prob\.↑\\uparrow1\-Extr\. Strength↑\\uparrowRouge\-L↑\\uparrowProb\.↑\\uparrowTruth ratio↑\\uparrowRouge\-L↑\\uparrowProb\.↑\\uparrowTruth ratio↑\\uparrowRouge\-L↑\\uparrowProb\.↑\\uparrowTruth ratio↑\\uparrowOriginal0\.180\.120\.290\.800\.410\.530\.830\.440\.620\.790\.870\.520\.600\.40—Retain0\.620\.880\.940\.830\.390\.500\.800\.430\.620\.830\.880\.510\.590\.70—RMU0\.500\.390\.72\\cellcolor\[HTML\]FFF8AD0\.75\\cellcolor\[HTML\]FFF8AD0\.420\.52\\cellcolor\[HTML\]FFCC990\.830\.430\.600\.610\.74\\cellcolor\[HTML\]FFF8AD0\.51\\cellcolor\[HTML\]FFF8AD0\.570\.55\\cellcolor\[HTML\]FFF8AD0\.6AltPO0\.66\\cellcolor\[HTML\]FFF8AD0\.93\\cellcolor\[HTML\]FFCC990\.95\\cellcolor\[HTML\]FF999A0\.78\\cellcolor\[HTML\]FFF8AD0\.42\\cellcolor\[HTML\]FFF8AD0\.54\\cellcolor\[HTML\]FFF8AD0\.800\.43\\cellcolor\[HTML\]FFF8AD0\.610\.610\.750\.47\\cellcolor\[HTML\]FFF8AD0\.57\\cellcolor\[HTML\]FFF8AD0\.716\.9GradDiff0\.420\.350\.670\.730\.400\.53\\cellcolor\[HTML\]FF999A0\.840\.42\\cellcolor\[HTML\]FFCC990\.62\\cellcolor\[HTML\]FF999A0\.79\\cellcolor\[HTML\]FF999A0\.88\\cellcolor\[HTML\]FFCC990\.53\\cellcolor\[HTML\]FFCC990\.590\.530\.8IDKDPO\\cellcolor\[HTML\]FFF8AD0\.87\\cellcolor\[HTML\]FF999A1\.00\\cellcolor\[HTML\]FF999A0\.960\.41\\cellcolor\[HTML\]FFF8AD0\.420\.530\.680\.430\.580\.620\.750\.500\.52\\cellcolor\[HTML\]FFCC990\.736\.9IDKNLL\\cellcolor\[HTML\]FF999A0\.980\.450\.740\.700\.390\.490\.730\.430\.57\\cellcolor\[HTML\]FFF8AD0\.66\\cellcolor\[HTML\]FFF8AD0\.780\.490\.550\.640\.8UNDIAL0\.690\.82\\cellcolor\[HTML\]FF999A0\.960\.500\.380\.480\.780\.410\.560\.560\.610\.460\.510\.670\.9NPO0\.610\.710\.91\\cellcolor\[HTML\]FFCC990\.760\.410\.520\.790\.430\.60\\cellcolor\[HTML\]FFF8AD0\.66\\cellcolor\[HTML\]FFF8AD0\.780\.50\\cellcolor\[HTML\]FFF8AD0\.570\.663\.7SimNPO0\.65\\cellcolor\[HTML\]FFCC990\.94\\cellcolor\[HTML\]FFF8AD0\.94\\cellcolor\[HTML\]FF999A0\.78\\cellcolor\[HTML\]FFF8AD0\.420\.53\\cellcolor\[HTML\]FFCC990\.83\\cellcolor\[HTML\]FFCC990\.45\\cellcolor\[HTML\]FFF8AD0\.610\.560\.710\.480\.560\.702\.5ROME\\cellcolor\[HTML\]FF999A0\.980\.370\.600\.62\\cellcolor\[HTML\]FFF8AD0\.42\\cellcolor\[HTML\]FFF8AD0\.540\.76\\cellcolor\[HTML\]FFF8AD0\.44\\cellcolor\[HTML\]FFF8AD0\.610\.45\\cellcolor\[HTML\]FFCC990\.79\\cellcolor\[HTML\]FFF8AD0\.510\.540\.60\\cellcolor\[HTML\]FF999A0\.4MEMIT\\cellcolor\[HTML\]FFCC990\.880\.380\.610\.40\\cellcolor\[HTML\]FFCC990\.44\\cellcolor\[HTML\]FFCC990\.560\.61\\cellcolor\[HTML\]FF999A0\.47\\cellcolor\[HTML\]FFF8AD0\.610\.310\.720\.500\.490\.56\\cellcolor\[HTML\]FFCC990\.5GROM0\.78\\cellcolor\[HTML\]FF999A1\.00\\cellcolor\[HTML\]FFF8AD0\.940\.63\\cellcolor\[HTML\]FF999A0\.47\\cellcolor\[HTML\]FF999A0\.630\.71\\cellcolor\[HTML\]FF999A0\.47\\cellcolor\[HTML\]FF999A0\.68\\cellcolor\[HTML\]FFCC990\.710\.77\\cellcolor\[HTML\]FF999A0\.54\\cellcolor\[HTML\]FF999A0\.60\\cellcolor\[HTML\]FF999A0\.75\\cellcolor\[HTML\]FFCC990\.5

Table C\.2:Results on TOFU\-10%\. Colors indicate rank \(red: best, orange: second, yellow: third\)\. Final Score is computed as12​\(MU\+\(1​\-Rouge\-L\)\+\(1​\-Prob\.\)\+\(1​\-Extr\. Strength\)3\)\\frac\{1\}\{2\}\\left\(\\mathrm\{MU\}\+\\frac\{\(1\\text\{\-Rouge\-L\}\)\+\(1\\text\{\-Prob\.\}\)\+\(1\\text\{\-Extr\.\\ Strength\}\)\}\{3\}\\right\)\. Rows below the dashed rule are the locate\-then\-edit knowledge editors\. GROM achieves the best overall forgetting\-utility trade\-off, attaining the highest Final Score and model utility \(MU\) while avoiding the severe utility degradation seen in several stronger\-forgetting baselines\. ROME edits marginally faster on this small forget set, but at a far worse trade\-off\.

## Appendix DUnlearning on MUSE

MUSE\(Shi et al\.[2024b](https://arxiv.org/html/2608.05783#bib.bib46)\)evaluates unlearning in long\-form domains using memorization and privacy\-leakage metrics computed on a forget split𝒟f\\mathcal\{D\}\_\{f\}and a retain split𝒟r\\mathcal\{D\}\_\{r\}\. We summarize the metrics used in our experiments below\.

#### VerbMem\.

VerbMem \(verbatim memorization\) measures how much the model reproduces the forget text word for word\. For each forget example the model is prompted to continue a prefix, and VerbMem scores the overlap between the continuation and the reference forget span, following the longest\-common\-subsequence criterion of\(Shi et al\.[2024b](https://arxiv.org/html/2608.05783#bib.bib46)\)\. Lower VerbMem indicates less verbatim regurgitation of the forget content and is therefore preferred\.

#### KnowMem\.

KnowMem \(knowledge memorization\) measures whether the model still expresses the underlying facts contained in the forget data, even when it does not reproduce them word for word\. MUSE evaluates the model on question\-style probes derived from the forget documents and checks whether the responses contain the target facts, using the automatic matching procedure of\(Shi et al\.[2024b](https://arxiv.org/html/2608.05783#bib.bib46)\)\. We report KnowMem on both splits\. On the forget split a lower value indicates better forgetting, whereas on the retain split a higher value indicates better preservation of non\-forget knowledge\.

#### PrivLeak\.

PrivLeak is a privacy\-leakage proxy derived from the Min\-K%K\\%Prob membership\-inference attack\(Shi et al\.[2024a](https://arxiv.org/html/2608.05783#bib.bib45)\)\. It measures how well an attacker can distinguish forget examples from a holdout set𝒟holdout\\mathcal\{D\}\_\{\\mathrm\{holdout\}\}using model likelihood statistics\. The holdout set is not the retain set\. It is a disjoint set used as a non\-member reference distribution for the membership test\. PrivLeak is defined relative to a retraining baseline as

PrivLeak\\displaystyle\\mathrm\{PrivLeak\}=Aunlearn−AretrainAretrain×100,\\displaystyle=\\frac\{A\_\{\\mathrm\{unlearn\}\}\-A\_\{\\mathrm\{retrain\}\}\}\{A\_\{\\mathrm\{retrain\}\}\}\\times 00,\(3\)Am\\displaystyle A\_\{m\}=AUC\(fm,𝒟f,\\displaystyle=\\mathrm\{AUC\}\\\!\\left\(f\_\{m\},\\,\\mathcal\{D\}\_\{f\},\\right\.𝒟holdout\),\\displaystyle\\qquad\\left\.\\mathcal\{D\}\_\{\\mathrm\{holdout\}\}\\right\),m∈\{unlearn,retrain\},\\displaystyle\\qquad m\\in\\\{\\mathrm\{unlearn\},\\mathrm\{retrain\}\\\},whereAUC​\(⋅\)\\mathrm\{AUC\}\(\\cdot\)is the area under the ROC curve for separating samples of𝒟f\\mathcal\{D\}\_\{f\}and𝒟holdout\\mathcal\{D\}\_\{\\mathrm\{holdout\}\}using Min\-K%K\\%Prob features\. A PrivLeak value closer to0is better, indicating that the unlearned model approaches the retraining baseline in membership distinguishability\.

#### Experimental details\.

For both MUSE corpora we apply the token\-suppression target to the MLP down\-projection of layers2828through3131of the released MUSE target model, with the suppression directions given by the logit\-lens unembedding rows as in the main text\. On MUSE News we use edit strengthβ=370\\beta=370and retain weightwr=10w\_\{r\}=10, and on MUSE Books we useβ=350\\beta=350andwr=80w\_\{r\}=80, with ridge scaleρ=0\.03\\rho=0\.03in both cases\. The larger retain weight on Books reflects its more entangled forget and retain domains, in which a stronger retain anchor is needed to preserve closely related permissible knowledge\.

#### Layer selection\.

For MUSE we use the same token\-suppression layer\-selection procedure as in TOFU\. The logit\-lens attribution is computed on long\-form forget and retain examples, and we select the highest\-scoring contiguous late\-layer band with widthk=4k=4\. This selects layers2828–3131for both MUSE News and MUSE Books\. We keep the layer band fixed across the two MUSE variants so that the comparison isolates the effect of the corpus and retain weight; the Books run uses a larger retain weight because its retain examples are semantically closer to the forget corpus\.

#### Observed results\.

On MUSE News, GROM obtains the best Final Score among the unlearning methods,55\.9355\.93, compared with49\.8149\.81for SimNPO\. The most important change is privacy leakage: GROM drives PrivLeak to−3\.80\-3\.80, close to the retraining reference value of0, whereas NPO and GradDiff leave large positive leakage values above100100\. Although GA obtains lower VerbMem and KnowMem on the forget set, it collapses retain KnowMem to0\.000\.00, so its Final Score remains lower\. This illustrates the main MUSE trade\-off: aggressively suppressing the forget documents is not sufficient if the method also destroys in\-domain news utility\.

On MUSE Books, the retain and forget distributions are more entangled because the retain material concerns the same fictional universe as the forget corpus\. In this setting GROM obtains a Final Score of76\.2076\.20, ahead of SimNPO at70\.8370\.83, and preserves the highest retain KnowMem among unlearning methods,65\.5665\.56\. It also reduces VerbMem on the forget set from the original model’s99\.5699\.56to2\.402\.40and drives PrivLeak to−0\.22\-0\.22, again close to the retraining target of0\. Several baselines achieve zero forget\-set memorization metrics, but they do so by sharply damaging retain utility; for example, GA and GradDiff reduce retain KnowMem to0\.000\.00and0\.130\.13\. The Books result therefore emphasizes why we tune the retain anchor more strongly on this benchmark\.

The MUSE runtime gap is also large\. The closed\-form edit takes0\.30\.3minutes for both MUSE News and MUSE Books, while the gradient baselines range from18\.018\.0to42\.442\.4minutes on News and from26\.726\.7to62\.562\.5minutes on Books under the same single\-H100 timing convention\. Thus the best MUSE Final Scores are obtained without an iterative fine\-tuning run\.

## Appendix EUnlearning on WMDP

WMDP\(Li et al\.[2024](https://arxiv.org/html/2608.05783#bib.bib31)\)evaluates targeted capability suppression rather than memorization removal\. The forget set is the WMDP\-Bio subset of biosecurity\-related multiple\-choice questions, and the utility evaluation uses MMLU\(Hendrycks et al\.[2020](https://arxiv.org/html/2608.05783#bib.bib20)\)as a broad general\-knowledge proxy\. We follow the evaluation protocol of prior unlearning work on WMDP\(Fan et al\.[2025](https://arxiv.org/html/2608.05783#bib.bib12)\)\.

#### Question format\.

Each WMDP item is a multiple\-choice question with a fixed set of answer options\. We present the question and its options in a standard instruction prompt and score the model by the answer option to which it assigns the highest likelihood\. We use the official WMDP\-Bio split provided by the benchmark\.

#### Forgetting and utility\.

We measure forgetting as the drop in WMDP\-Bio accuracy after unlearning\. LetAccBio\\mathrm\{AccBio\}denote the fraction of WMDP\-Bio questions answered correctly\. Following the main text, we report1−AccBio1\-\\mathrm\{AccBio\}as the forgetting score, where larger values indicate stronger suppression of the hazardous slice\. To quantify retained general capability we report overall MMLU accuracy under the same highest\-likelihood decoding rule, where higher is better\.

#### Experimental details\.

Because WMDP probes knowledge through multiple\-choice accuracy rather than free generation, we use the representation\-corruption target of the main text rather than token suppression\. We apply the closed\-form update to the MLP down\-projection of a single mid layer, layer88of LLaMA\-3\-8B\-Instruct, with corruption strengthc=45c=45, retain weightwr=1000w\_\{r\}=1000, and ridge scaleρ=0\.03\\rho=0\.03\. The forget keys are collected from the WMDP\-Bio forget corpus and the retain keys from a generic WikiText corpus, which we found essential for preserving MMLU\. As discussed in the main text, we select the mid layer following the established convention for representation\-level unlearning on WMDP\.

#### Layer selection\.

For WMDP we use a representation\-corruption edit rather than token suppression, so the logit\-lens token attribution used for TOFU and MUSE is not the selection criterion\. We edit a single mid layer, layer88, matching the layer range commonly used for representation\-level WMDP unlearning\. The edit is applied to the MLPdown\_proj, and the retain keys are taken from WikiText to anchor general language\-model behavior while the WMDP\-Bio keys receive the corruption target\.

#### Observed results\.

On WMDP\-Bio, GROM obtains a Final Score of0\.630\.63, the highest among the compared unlearning methods\. SimNPO has the strongest raw forgetting score, with1−AccBio=0\.751\-\\mathrm\{AccBio\}=0\.75, but its MMLU accuracy is0\.440\.44\. GROM gives slightly less suppression,1−AccBio=0\.711\-\\mathrm\{AccBio\}=0\.71, while preserving substantially higher MMLU accuracy,0\.550\.55\. This difference is the reason GROM has the best overall trade\-off despite not maximizing the forgetting metric alone\.

The single\-layer closed\-form edit reduces WMDP\-Bio accuracy while keeping broad utility closer to the original model\. The update takes0\.20\.2minutes, compared with roughly2020–2424minutes for the gradient baselines\.

## Appendix FUnlearning on ZsRE

ZsRE\(Levy et al\.[2017](https://arxiv.org/html/2608.05783#bib.bib30)\)is a few\-shot factual editing benchmark\. We use it in the reverse direction from TOFU, MUSE, and WMDP: instead of asking whether knowledge editors perform well on broad unlearning benchmarks, we ask whether GROM remains competitive in the fact\-editing harness used by ZeroUnlearn\(Lin et al\.[2026](https://arxiv.org/html/2608.05783#bib.bib33)\)\. This makes the comparison favorable to the locate\-then\-edit baselines, since ROME, MEMIT, AlphaEdit, and ZeroUnlearn are designed for this style of localized fact intervention\.

#### Task format\.

Each ZsRE record contains a requested rewrite with a subject, a prompt template, and the original answer, together with paraphrased prompts and neighborhood prompts\. In a standard editing setup the method would replace the original answer with a new target\. For unlearning, we instead treat the original answer as the content to remove: the edit should make the model stop predicting that answer on the original prompt and on paraphrases, while preserving answers to the neighborhood prompts\.

#### Metrics\.

We follow the released ZeroUnlearn evaluation harness\.*Efficacy*is the percentage of rewrite prompts on which the edited model still predicts the original answer tokens, so lower values indicate stronger unlearning\.*Generalization*is the same measurement on paraphrased prompts and tests whether the deletion transfers beyond the exact wording of the request\.*Specificity*is accuracy on neighborhood prompts that should remain unchanged, so higher values indicate better locality preservation\. The main text reports mean and standard deviation across runs\.

#### Experimental details\.

We use the same few\-shot protocol as the ZeroUnlearn harness: each run unlearns5050ZsRE facts and uses10001000disjoint facts as the retain set\. Results are averaged over ten seeds\. The forget keys use the fact\-editingsubject\_lastconvention, i\.e\., one key is collected at the final subject token rather than at every answer token\. This matches the ROME/MEMIT/ZeroUnlearn convention and gives the closed\-form system a retain anchor in the same feature space as the editor baselines\.

For ZsRE, the best GROM configuration is head\-only\. We set the MLP suppression strength to zero, edit the untied LM head with the token\-suppression target, use specificity reweighting over the original\-answer tokens, and keep the retain anchor on the disjoint retain facts\. For both evaluated models we use head strengthβhead=20\\beta\_\{\\mathrm\{head\}\}=20, retain weightwr=30w\_\{r\}=30, ridge scaleρ=0\.03\\rho=0\.03, and no additional WikiText retain anchor\. The layer lists in the configuration are retained only for compatibility with the shared harness; becauseβMLP=0\\beta\_\{\\mathrm\{MLP\}\}=0, the MLP updates are exactly zero\.

#### Observed results\.

Table[3](https://arxiv.org/html/2608.05783#Sx4.T3)shows that GROM suppresses the original answers much more strongly than the locate\-then\-edit methods while preserving locality\. The improvement appears on both direct rewrite prompts and paraphrases, indicating that the edit is not simply breaking one prompt surface form\. Specificity remains at the level of the strongest editing baselines on the smaller model and is highest in the table on the larger model\. This supports the main conclusion that the closed\-form suppression edit is not only a broad benchmark method: it also works inside the few\-shot factual\-unlearning setting for which the editor baselines were tuned\.

## Appendix GBenchmark and Evaluation Summary

Table[G\.1](https://arxiv.org/html/2608.05783#A7.T1)summarizes the target model and evaluation metrics used for each benchmark\. We separate metrics that measure forgetting on the forget set from metrics that measure utility preservation on retain or held\-out evaluation sets\. Arrows indicate the desired direction after unlearning\.

BenchmarkTarget ModelUnlearning EffectivenessUtility PreservationTOFULLaMA\-2\-7B\-ChatLLaMA\-3\.2\-1B\-InstructProbability on𝒟f\\mathcal\{D\}\_\{f\}↓\\downarrowROUGE\-L on𝒟f\\mathcal\{D\}\_\{f\}↓\\downarrowExtraction strength on𝒟f\\mathcal\{D\}\_\{f\}↓\\downarrowModel utility↑\\uparrowProbability on𝒟r\\mathcal\{D\}\_\{r\}, Real Authors, World Facts↑\\uparrowROUGE\-L on𝒟r\\mathcal\{D\}\_\{r\}, Real Authors, World Facts↑\\uparrowTruth ratio on𝒟r\\mathcal\{D\}\_\{r\}, Real Authors, World Facts↑\\uparrowMUSELLaMA\-2\-7BICLM\-7BKnowMem on𝒟f\\mathcal\{D\}\_\{f\}↓\\downarrowVerbMem on𝒟f\\mathcal\{D\}\_\{f\}↓\\downarrowPrivLeak→0\\to 0KnowMem on𝒟r\\mathcal\{D\}\_\{r\}↑\\uparrowWMDPLlama\-3\-8B\-InstructAccuracy on WMDP\-Bio↓\\downarrowAccuracy on MMLU↑\\uparrowZsRELLaMA\-3\.2\-3BLLaMA\-3\.1\-8BRewrite orig\.\-answer acc\.↓\\downarrowParaphrase orig\.\-answer acc\.↓\\downarrowNeighborhood acc\.↑\\uparrow

Table G\.1:Benchmark summary\. For each benchmark, we report the target model and the metrics used to evaluate unlearning effectiveness and utility preservation\.#### Runtime protocol\.

All runtime values reported in the experimental tables are wall\-clock update times measured on a single NVIDIA H100\. The timing excludes model loading, tokenizer loading, dataset preprocessing, and the final benchmark evaluation\. For gradient\-based baselines, the timer starts immediately before the unlearning optimization loop and ends after the final optimizer step\. Thus the reported time includes the forward and backward passes, optimizer updates, and any gradient\-checkpointing overhead used by the released recipe\. For GROM, the timer starts immediately before the closed\-form edit procedure and ends after the edited weights have been written back to the model\. This includes collecting the forget and retain activations used as keys, constructing the ridge\-regularized normal equations, solving for the update, and applying the update to each selected layer\. It does not include the subsequent evaluation pass used to compute TOFU, MUSE, WMDP, ZsRE, or MMLU metrics\. The baseline budgets behind these numbers follow the released reference recipe of each benchmark: ten epochs at effective batch3232on TOFU\-5%, ten epochs at effective batch6464with gradient checkpointing on MUSE, and500500steps at effective batch44on WMDP, the last following the released OPTML recipes for NPO, GradDiff and SimNPO\(Fan et al\.[2025](https://arxiv.org/html/2608.05783#bib.bib12)\)\. In every case GROM performs a single closed\-form edit\.

BenchmarkTarget modelEdited moduleEdited layersSelection ruleTOFU\-10%LLaMA\-3\.2\-1B\-InstructMLPdown\_proj1111–1515\(k=5k=5\)Contiguous late\-layer band with the largest average logit\-lens attribution score\.TOFU\-5%LLaMA\-2\-7B\-ChatMLPdown\_proj2626–3131\(k=6k=6\)Contiguous late\-layer band with the largest average logit\-lens attribution score\.MUSE NewsLLaMA\-2\-7BMLPdown\_proj2828–3131\(k=4k=4\)Contiguous late\-layer band with the largest average logit\-lens attribution score\.MUSE BooksICLM\-7BMLPdown\_proj2828–3131\(k=4k=4\)Same MUSE layer band; retain weight is increased for the more entangled Books setting\.WMDP\-BioLLaMA\-3\-8B\-InstructMLPdown\_proj88\(k=1k=1\)Single mid\-layer representation edit, following the WMDP representation\-unlearning setup\.ZsRELLaMA\-3\.2\-3B\-InstructLLaMA\-3\.1\-8B\-InstructLM head—Head\-only closed\-form token suppression using the fact\-editingsubject\_lastkey convention\.

Table G\.2:Layer\-selection summary\. Layer numbers use the model indexing convention in our implementation\. For token\-suppression benchmarks,kkis the edit width and the selected band maximizes the average logit\-lens attribution score over candidate late layers\.

## Appendix HAdditional Reproducibility Details

#### Code and evaluation pipeline\.

Our implementation consists of scripts for collecting forget and retain activations, constructing the closed\-form update, applying the edited weights, and launching the standard benchmark evaluations\. The code, the configuration file of every reported setting, and the command lines needed to reproduce the reported experiments are available athttps://github\.com/Batorskq/GROMunder the MIT license\. All reported benchmark scores are computed with the OpenUnlearning evaluation pipeline\(Dorna et al\.[2025](https://arxiv.org/html/2608.05783#bib.bib9)\), except for ZsRE, which is evaluated in the ZeroUnlearn harness\(Lin et al\.[2026](https://arxiv.org/html/2608.05783#bib.bib33)\)\. We use the benchmark datasets and target checkpoints provided by TOFU, MUSE, WMDP, ZsRE, and the corresponding released evaluation harnesses rather than introducing any new dataset\.

#### Software and hardware\.

Experiments were run in a Linux HPC environment on a single NVIDIA H100 GPU\. The software environment used Python 3\.11, PyTorch 2\.4\.1, Transformers 4\.51\.3, and an editable installation of OpenUnlearning\. Model loading and evaluation use the Hugging Face Transformers stack with CUDA acceleration\. Unless otherwise specified by the corresponding benchmark recipe, model computations use the precision of the released target checkpoint and the OpenUnlearning evaluation configuration\.

#### Randomness and number of runs\.

The closed\-form edit is deterministic once the forget and retain examples, target directions, edited layers, and scalar hyperparameters are fixed\. For the WMDP representation\-corruption target, the random direction is generated once with seed0and then held fixed\. We use the default seed0in the OpenUnlearning evaluation\.

#### Use of LLM assistance\.

Large language models were used solely for editorial and auxiliary support, including improving clarity, grammar, and presentation, and providing assistance with implementation code\. All core technical contributions, experimental design decisions, analyses, interpretations, and final research judgments were made by the authors\.