Inference-Time Machine Unlearning via Gated Activation Redirection

# Inference-Time Machine Unlearning via Gated Activation Redirection
Source: [https://arxiv.org/html/2605.12765](https://arxiv.org/html/2605.12765)
\\correspondingauthor

Vinícius Conte Turani \(v\.turani@edu\.pucrs\.br\), Rodrigo C\. Barros \(rodrigo\.barros@pucrs\.br\) and Lucas S\. Kupssinskü \(lucas\.kupssinsku@pucrs\.br\)

Vinícius Conte TuraniEqual contributionMALTA, Machine Learning Theory and Applications Lab, PUCRS, Porto Alegre, BrazilOtávio ParragaEqual contributionMALTA, Machine Learning Theory and Applications Lab, PUCRS, Porto Alegre, BrazilJoão Vitor Boer AbitanteKristen K\. ArguelloMALTA, Machine Learning Theory and Applications Lab, PUCRS, Porto Alegre, BrazilJoana PasqualiMALTA, Machine Learning Theory and Applications Lab, PUCRS, Porto Alegre, BrazilRamiro N\. BarrosMALTA, Machine Learning Theory and Applications Lab, PUCRS, Porto Alegre, BrazilFlavio du Pin CalmonHarvard UniversityChristian MattjieMALTA, Machine Learning Theory and Applications Lab, PUCRS, Porto Alegre, BrazilRodrigo C\. BarrosMALTA, Machine Learning Theory and Applications Lab, PUCRS, Porto Alegre, BrazilKunumi Institute, BrazilLucas S\. KupssinsküMALTA, Machine Learning Theory and Applications Lab, PUCRS, Porto Alegre, Brazil

###### Abstract

Large Language Models memorize vast amounts of training data, raising concerns regarding privacy, copyright infringement, and safety\. Machine unlearning seeks to remove the influence of a targeted forget set𝒟f\\mathcal\{D\}\_\{f\}while preserving model performance, ideally approximating a model retrained from scratch without𝒟f\\mathcal\{D\}\_\{f\}\. Existing approaches aim to achieve this by updating model parameters via gradient\-based methods\. However, these updates are computationally expensive, lead to irreversible weight changes, and degrade when the model is quantized for deployment\. A recent alternative to changing model weights is activation engineering, where activations are changed during inference to steer model behavior\. Despite circumventing weight editing, naive activation steering introduces its own failure modes, as a single global steering vector applies the same intervention to every input, leading to unintended changes in model behavior\. We introduce Inference\-Time Unlearning via Gated Activation Redirection \(GUARD\-IT\), a training\- and gradient\-free method that unlearns via input\-dependent activation steering at inference time\. Our method partitions𝒟f\\mathcal\{D\}\_\{f\}into semantic clusters, computes one steering vector per cluster in an offline phase, and at inference time routes each user query through a similarity gateway that selects only the relevant cluster vectors\. The resulting intervention is applied as a norm\-preserving rotation in the residual stream, leaving model weights untouched\. Experiments on TOFU and MUSE show that GUARD\-IT matches or exceeds 12 gradient\-based baselines across three model scales, while being theonlymethod to simultaneously preserve utility, suppress memorization, and avoid catastrophic collapse acrossall settings\. GUARD\-IT further supports continual unlearning without retraining, and remains effective under quantization, a scenario in which parameter\-editing methods degrade\.

## 1Introduction

Large Language Models \(LLMs\)can memorize training data, raising concerns about privacy, copyright infringement, unintended biases, and safetyliu2025rethinking,parraga2025fairness,shi2024muse\.Machine Unlearning \(MU\)addresses this by aiming to remove the influence of a targetedforget set𝒟f\\mathcal\{D\}\_\{f\}while maintaining performance on aretain set𝒟r\\mathcal\{D\}\_\{r\}liu2025rethinking\. Ideally, this approximates a model trained entirely from scratch on𝒟r\\mathcal\{D\}\_\{r\}, eliminating both explicit memorization and distributed statistical influence of𝒟f\\mathcal\{D\}\_\{f\}\. This objective is challenging because knowledge is encoded into distributed patterns across parameters, making selective removal difficult without affecting unrelated behaviorelhage2022toymodelssuperposition\.

Most existingMUapproaches rely on computationally\-expensive parameter updates, such as gradient ascent on𝒟f\\mathcal\{D\}\_\{f\}dorna2025openunlearning\. These methods require access to the full fine\-tuning pipeline and often lead to catastrophic collapse, in which the model loses generalization and produces incoherent outputszhang2024npo\. Furthermore, their unlearning effects can be compromised or undone by subsequent model updates, compression, or quantization techniquesabitante2026quantization\. These limitations are critical in practical deployment, where models are often quantized for efficiency, resulting in discrete parameter spaces that alter optimization dynamics and can render gradient\-based unlearning unstable or ineffective\. Additionally, many real\-world scenarios requirecontinual unlearning, where multipleforgetrequests arrive over time\. Repeated parameter updates in this setting accumulate interference, exacerbating forgetting of unrelated knowledge and leading to progressive model degradation\.

Activation engineering \(or representation engineering\) is a training\-free alternative to steer model behavior by injecting linear concept directions at inference time without modifying any weightsturner2024steeringlanguagemodelsactivation,zou2023representation\. If applied to unlearning, this suggests achieving directional suppression by identifying and manipulating theforgetdirection directly\. Most approaches compute a singleSteering Vector \(SV\)over the full target corpus, which is reasonable when the target behavior can be captured by a single compact dataset\. However, when the corpus spans conflicting directions, the resulting vector averages them, losing specificityyu2020gradient,liu2024conflict\. Applying a fixedSVindiscriminately can further degrade outputs on unrelated inputstan2024analysing, and standard additive steering compounds the issue by perturbing the hidden state’s norm, destabilizing layer normalization and attention scaling in downstream layersvu2025angularsteeringbehaviorcontrol,you2026sphericalsteeringgeometryawareactivation\.

To address these challenges, we introduceInference\-Time Unlearning via Gated Activation Redirection \(GUARD\-IT\)111[GitHub: https://github\.com/ViniTurani/GUARD\-IT](https://github.com/ViniTurani/GUARD-IT), a training\- and gradient\-free methodology forMUinLLMs\.GUARD\-IToperates entirely in activation space, executing unlearning as a controlled geometric transformation\.GUARD\-ITpartitions theforgetcorpus into semantic clusters to precompute steering vectors in an offline phase\. At inference time, aSimilarity Gateway \(SG\)dynamically routes the user query into an input\-dependentforgetrepresentation\. Crucially,GUARD\-ITapplies this intervention as a pure rotation in the residual stream, preserving the original activation norm and ensuring model stability\.

In summary, our contributions in this paper are:

1. 1\.We formulate unlearning as a training\- and gradient\-free steering problem, applying norm\-preserving rotations in activation space for stable, model\-agnostic behavioral control\.
2. 2\.We introduce a similarity gate with clusteredPrototype Steering Vectors \(PSVs\), enabling input\-dependent unlearning that adapts to the semantic content of each query\.
3. 3\.We show that this inference\-time architecture naturally supports continual unlearning and remains effective under quantization, where gradient\-based methods degrade\.

## 2GUARD\-IT

GUARD\-ITis inspired by the linear representation hypothesiszou2023representation, which stipulates that high\-level concepts are encoded as directions in the model’s activation space\. Following the activation engineering frameworkturner2024steeringlanguagemodelsactivationand prior work on inference\-time interventionli2023inference, we computeSVsthat encode the direction of the content to be forgotten, further making use of them at inference time\. Unlike prior work that uses a single globalSVpanickssery2024steeringllama2contrastive,GUARD\-ITclusters theforgetcorpus into semantic groups and computes onePSVper cluster\. EachPSVcaptures theforgetdirection of its cluster\. ASGindependently controls the contribution of eachPSVper input, allowing multiplePSVsto be simultaneously activated and jointly compose the finalSV\. This design enables both localized and compositional unlearning, avoiding common failure modes in similar intervention settings\.

GUARD\-IToperates in two phases that are separated to keep inference\-time overhead minimal\. The*offline phase*is performed once perforgetcorpus and produces a set of precomputed steering material\. The*online phase*runs at inference time and consists of a lightweight routing decision eventually followed by a single activation\-space transformation\. No model weights are modified at any point\. Algorithms[1](https://arxiv.org/html/2605.12765#alg1)and[2](https://arxiv.org/html/2605.12765#alg2)provide a concise procedural summary for bothGUARD\-ITphases\. Appendix[A](https://arxiv.org/html/2605.12765#A1)illustrates the complete pipeline comprised by GUARD\-IT\.

Algorithm 1GUARD\-IT: Offline Phase1:Input:Forgetcorpus

𝒟f\\mathcal\{D\}\_\{f\},retaincorpus

𝒟r\\mathcal\{D\}\_\{r\}, target layer

ℓ\\ell, max clusters

KmaxK\_\{\\max\}
2:Output:

\{𝐜k,𝐡¯kf,ρ¯kf\}k=1K\\\{\\mathbf\{c\}\_\{k\},\\;\\bar\{\\mathbf\{h\}\}^\{f\}\_\{k\},\\;\\bar\{\\rho\}^\{f\}\_\{k\}\\\}\_\{k=1\}^\{K\},

𝐡¯r\\bar\{\\mathbf\{h\}\}^\{r\},

ρ¯r\\bar\{\\rho\}^\{r\}
3:

4:Embed and L2\-normalize all

d∈𝒟fd\\in\\mathcal\{D\}\_\{f\}with ST

ϕ​\(⋅\)\\phi\(\\cdot\)
5:Select

k⋆k^\{\\star\}via silhouette score \(Eq\.[2](https://arxiv.org/html/2605.12765#S2.E2)\); run K\-Means

→\\toclusters

\{Ck\}\\\{C\_\{k\}\\\}, centroids

\{𝐜k\}\\\{\\mathbf\{c\}\_\{k\}\\\}
6:for

k=1,…,k⋆k=1,\\ldots,k^\{\\star\}do

7:Extract layer\-

ℓ\\ellhidden states for

CkC\_\{k\}; compute

𝐡¯kf\\bar\{\\mathbf\{h\}\}^\{f\}\_\{k\}and

ρ¯kf\\bar\{\\rho\}^\{f\}\_\{k\}\(Eq\.[3](https://arxiv.org/html/2605.12765#S2.E3)\)

8:endfor

9:Extract layer\-

ℓ\\ellhidden states for

𝒟r\\mathcal\{D\}\_\{r\}; compute

𝐡¯r\\bar\{\\mathbf\{h\}\}^\{r\}and

ρ¯r\\bar\{\\rho\}^\{r\}\(Eq\.[3](https://arxiv.org/html/2605.12765#S2.E3)\)

Algorithm 2GUARD\-IT: Online Phase1:Input:User inputx, threshold

TT, activation

𝐡ℓ\\mathbf\{h\}^\{\\ell\}, coefficient

α\\alpha, offline outputs above

2:Output:Steered activation

𝐡ℓ\\mathbf\{h\}^\{\\ell\}
3:

4:Embed

𝐱\\mathbf\{x\}withSentence\-Transformer \(ST\)

ϕ​\(𝐱\)\\phi\(\\mathbf\{x\}\)
5:Compute active clusters

𝒦​\(𝐱\)\\mathcal\{K\}\(\\mathbf\{x\}\)\(Eq\.[4](https://arxiv.org/html/2605.12765#S2.E4)\)

6:if

\|𝒦​\(𝐱\)\|≠0\|\\mathcal\{K\}\(\\mathbf\{x\}\)\|\\neq 0then

7:Aggregate activePSVs:

𝐩​\(𝐱\)=1\|𝒦​\(𝐱\)\|​∑k∈𝒦​\(𝐱\)𝐡¯kf\\mathbf\{p\}\(\\mathbf\{x\}\)=\\frac\{1\}\{\|\\mathcal\{K\}\(\\mathbf\{x\}\)\|\}\\sum\_\{k\\in\\mathcal\{K\}\(\\mathbf\{x\}\)\}\\bar\{\\mathbf\{h\}\}^\{f\}\_\{k\}
8:ComputeSV

𝐯​\(𝐱\)\\mathbf\{v\}\(\\mathbf\{x\}\)via Orthogonal \(Eq\.[6](https://arxiv.org/html/2605.12765#S2.E6)\)

9:Rescale to activation norm

→𝐯^​\(𝐱\)\\to\\hat\{\\mathbf\{v\}\}\(\\mathbf\{x\}\)\(Eq\.[7](https://arxiv.org/html/2605.12765#S2.E7)\)

10:Apply rotation at layer

ℓ\\ell\(Eq\.[8](https://arxiv.org/html/2605.12765#S2.E8)\)

11:returnSteered activation

𝐡′⁣ℓ\\mathbf\{h\}^\{\\prime\\ell\}
12:endif

13:return

𝐡ℓ\\mathbf\{h\}^\{\\ell\}

### 2\.1Offline Phase

#### 2\.1\.1Semantic Clustering

Each document in theforgetset𝒟f=\{d1,…,dN\}\\mathcal\{D\}\_\{f\}=\\\{\\textbf\{d\}\_\{1\},\\ldots,\\textbf\{d\}\_\{N\}\\\}is embedded with a Sentence\-Transformer \(ST\)ϕ\\phiand L2\-normalized\. We performkk\-Means clustering to partition𝒟f\\mathcal\{D\}\_\{f\}intokkclusters\. We also evaluated alternative clustering algorithms and observed no consistent differences in downstream unlearning quality, so we adoptkk\-Means as the default choice due to its linear cost in all critical variables \(clusters, objects, and features\)\. The choice ofkk\-Means is further motivated by a geometric property of unit\-norm vectors, since minimizing Euclidean distance on L2\-normalized embeddings is equivalent to maximizing intra\-cluster cosine similarity\. Each cluster centroid represents the semantic direction of that cluster\. The number of clusterskkis selected automatically by maximizing the mean silhouette score over a candidate range\{2,3,…,kmax\}\\\{2,3,\\ldots,k\_\{\\text\{max\}\}\\\}\. Letei=ϕ​\(di\)\\textbf\{e\}\_\{i\}=\\phi\(\\textbf\{d\}\_\{i\}\)be the embedded documentdi\\textbf\{d\}\_\{i\}, for a partition intokkclusters withdi∈Cr\\textbf\{d\}\_\{i\}\\in C\_\{r\}, we have:

ak​\(i\)=1\|Cr\|−1​∑ϕ​\(𝐝j\)∈Crj≠id​\(ϕ​\(𝐝i\),ϕ​\(𝐝j\)\),bk​\(i\)=minCt≠Cr⁡1\|Ct\|​∑ϕ​\(𝐝j\)∈Ctd​\(ϕ​\(𝐝i\),ϕ​\(𝐝j\)\),a\_\{k\}\(i\)=\\frac\{1\}\{\|C\_\{r\}\|\-1\}\\sum\_\{\\begin\{subarray\}\{c\}\\phi\(\\mathbf\{d\}\_\{j\}\)\\in C\_\{r\}\\\\ j\\neq i\\end\{subarray\}\}d\(\\phi\(\\mathbf\{d\}\_\{i\}\),\\phi\(\\mathbf\{d\}\_\{j\}\)\),\\qquad b\_\{k\}\(i\)=\\min\_\{C\_\{t\}\\neq C\_\{r\}\}\\frac\{1\}\{\|C\_\{t\}\|\}\\sum\_\{\\phi\(\\mathbf\{d\}\_\{j\}\)\\in C\_\{t\}\}d\\left\(\\phi\(\\mathbf\{d\}\_\{i\}\),\\phi\(\\mathbf\{d\}\_\{j\}\)\\right\),\(1\)as the mean intra\-cluster distance and the mean distance to the nearest neighboring cluster, respectively\. The optimal number of clusters is then selected as

k⋆=arg​maxk∈\{2,…,kmax\}⁡\[1N​∑i=1Nbk​\(i\)−ak​\(i\)max⁡\{ak​\(i\),bk​\(i\)\}\]\.k^\{\\star\}=\\operatorname\*\{arg\\,max\}\_\{k\\in\\\{2,\\ldots,k\_\{\\max\}\\\}\}\\left\[\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\frac\{b\_\{k\}\(i\)\-a\_\{k\}\(i\)\}\{\\max\\\{a\_\{k\}\(i\),\\,b\_\{k\}\(i\)\\\}\}\\right\]\.\(2\)
While akkthat is too small merges distinct topics into one cluster, akkthat is too large splits coherent documents into redundantPSVs\. A higher silhouette score indicates that theforgetcorpus decomposes into well\-defined, non\-overlapping topics, enabling more precise per\-cluster steering\.

#### 2\.1\.2Activation Extraction

GUARD\-ITextracts hidden\-state representations from a target layerℓ\\ellfor each document in theforgetandretaincorpora, steering in theoppositedirection of theforgetcorpus to suppress its expression in the hidden layers\. For each document in a corpus𝒟\\mathcal\{D\}, we capture the residual stream at layerℓ\\elland mean\-pool over all token representations to obtain thePSVof that instance; we ablate this choice against last\-token extraction in Appendix[B](https://arxiv.org/html/2605.12765#A2)\.

We focus on extracting activations from intermediate layers of the residual stream, which maximize trade\-off between representational richness and redirectabilitypanickssery2024steeringllama2contrastive,arditi2024refusal\. Early layers primarily encode lexical and syntactic features and are highly sensitive to perturbations, propagating small interventions into uncontrolled downstream effectsskean2025layer; late layers sit close to the output distribution and leave little computational depth through which a steering signal can shape generation\.

Empirically, the most effective layers forGUARD\-ITsit*earlier*than the center of the model, around the first quartile of the transformer stack, rather than at the middle layers favored by prior work on behavior steeringpanickssery2024steeringllama2contrastive,arditi2024refusaland representation\-level unlearningli2024wmdp\. Behavior\-steering tasks, such as refusal or sentiment, correspond to abstractions that crystallize mid\-stack, whereas unlearning targets entity\-level associations encoded in coarser representations earlier in the forward pass\. Intervening earlier may also leave more residual depth for the model to re\-integrate the perturbation coherently, reducing the gibberish\-inducing side effects of aggressive late\-layer steering\. We provide the full per\-layer ablation study in Appendix[C](https://arxiv.org/html/2605.12765#A3)\.

We construct aPSVover a corpus𝒟\\mathcal\{D\}by averaging the activations at layerℓ\\ell:

𝐡¯𝒟=1N​∑i=1N𝐡di\(ℓ\),\\bar\{\\mathbf\{h\}\}^\{\\mathcal\{D\}\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbf\{h\}\_\{d\_\{i\}\}^\{\(\\ell\)\},\(3\)where𝐡di\(ℓ\)∈ℝH\\mathbf\{h\}\_\{d\_\{i\}\}^\{\(\\ell\)\}\\in\\mathbb\{R\}^\{H\}is the mean pooling of the activations of each token of documentdi\\textbf\{d\}\_\{i\}andHHis the hidden dimension size\. The same computation is applied to eachforgetcluster to obtain𝐡¯jf\\bar\{\\mathbf\{h\}\}^\{f\}\_\{j\}and to the fullretaincorpus to obtain a single reference direction𝐡¯r\\bar\{\\mathbf\{h\}\}^\{r\}\. We refer to𝐡¯jf\\bar\{\\mathbf\{h\}\}^\{f\}\_\{j\}as thePSVof clusterjj\.

### 2\.2Online Phase

#### 2\.2\.1Similarity Gate

Let𝐱\\mathbf\{x\}be the user input,ϕ​\(𝐱\)\\phi\(\\mathbf\{x\)\}be the embedded user input and𝐜j\\mathbf\{c\}\_\{j\}be the centroid of clusterjj\. We build the set of activeforgetclusters whose centroids exceed the similarity thresholdTTas:

𝒦​\(𝐱\)=\{j:sim​\(𝐜j,ϕ​\(𝐱\)\)≥T\}\.\\mathcal\{K\}\(\\mathbf\{x\}\)=\\bigl\\\{j:\\mathrm\{sim\}\(\\mathbf\{c\}\_\{j\},\\phi\(\\mathbf\{x\}\)\)\\geq T\\bigr\\\}\.\(4\)
If\|𝒦​\(x\)\|=0\|\\mathcal\{K\}\(\\textbf\{x\}\)\|=0, i\.e\., no cosine similarity crossed the thresholdTT, we do not steer the model away from theforgetclusters, leaving the inference process as is\. Conversely, if\|𝒦​\(x\)\|≠0\|\\mathcal\{K\}\(\\textbf\{x\}\)\|\\neq 0, the input must be steered, because we are dealing with a forget\-sensitive concept\. To proceed with the steering process, we average thePSVs of the activeforgetclusters\.

𝐩​\(x\)=1\|𝒦​\(x\)\|​∑j∈𝒦​\(x\)h¯jf,\\mathbf\{p\}\(\\textbf\{x\}\)=\\frac\{1\}\{\|\\mathcal\{K\}\(\\textbf\{x\}\)\|\}\\sum\_\{j\\in\\mathcal\{K\}\(\\textbf\{x\}\)\}\\bar\{\\textbf\{h\}\}^\{f\}\_\{j\},\(5\)
Vector𝐩​\(x\)\\mathbf\{p\}\(\\textbf\{x\}\)is an input\-dependentforgetdirection in activation space that encodes concepts according to the activeforgetclusters\. We also experimented with similarity\-weighted aggregation, in which eachh¯jf\\bar\{\\textbf\{h\}\}^\{f\}\_\{j\}is weighted by cosine similarity, but we observed no consistent improvements over the mean\.

#### 2\.2\.2Steering Vector Orthogonal Computation

To convert𝐩​\(x\)\\mathbf\{p\}\(\\textbf\{x\}\)into anSV,GUARD\-ITprojects the averagePSVof the activeforgetclustersperpendicularto𝐡¯𝐫\\mathbf\{\\bar\{\\mathbf\{h\}\}^\{r\}\}\. This procedure removes the component of theforgetrepresentation that is shared with retained content, avoiding interference with retained knowledge:

𝐯​\(𝐱\)=𝐩​\(𝐱\)−𝐩​\(𝐱\)⋅𝐡¯r‖𝐡¯r‖2​𝐡¯r\.\\mathbf\{v\}\(\\mathbf\{x\}\)=\\mathbf\{p\}\(\\mathbf\{x\}\)\-\\frac\{\\mathbf\{p\}\(\\mathbf\{x\}\)\\cdot\\bar\{\\mathbf\{h\}\}^\{r\}\}\{\\\|\\bar\{\\mathbf\{h\}\}^\{r\}\\\|^\{2\}\}\\,\\bar\{\\mathbf\{h\}\}^\{r\}\.\(6\)
The resulting𝐯​\(𝐱\)\\mathbf\{v\}\(\\mathbf\{x\}\)projection opposesforgetconcepts while preserving the direction of the retained distribution\. Appendix[D](https://arxiv.org/html/2605.12765#A4)ablates this choice when compared to thediff\-meansmethodpanickssery2024steeringllama2contrastive\.

#### 2\.2\.3Normalization

Recent work on activation steering has shown that the*direction*of a hidden state carries more information than its magnitudevu2025angularsteeringbehaviorcontrol\. Magnitude changes can destabilize layer normalization and attention scaling\.GUARD\-ITavoids this issue by applying two complementary normalizations\.

##### Activation\-norm scaling\.

The norm of steering vector𝐯​\(𝐱\)\\mathbf\{v\(x\)\}depends on the distance between𝐩​\(𝐱\)\\mathbf\{p\}\(\\mathbf\{x\}\)and theretainPSVin hidden space\. Since these distances can vary across models, it would be difficult to steer distinct models\. Therefore,GUARD\-ITrescales𝐯​\(𝐱\)\\mathbf\{v\(x\)\}to the mean activation norm of the active corpora\. Letρ¯f​\(𝐱\)=1\|𝒦​\(𝐱\)\|​∑k∈𝒦​\(𝐱\)ρ¯kf\\bar\{\\rho\}^\{f\}\(\\mathbf\{x\}\)=\\frac\{1\}\{\|\\mathcal\{K\}\(\\mathbf\{x\}\)\|\}\\sum\_\{k\\in\\mathcal\{K\}\(\\mathbf\{x\}\)\}\\bar\{\\rho\}^\{f\}\_\{k\}andρ¯r\\bar\{\\rho\}^\{r\}denote the mean L2 norms of the hidden states of the activeforgetclusters and of theretainset, respectively:

𝐯^​\(𝐱\)=𝐯​\(𝐱\)‖𝐯​\(𝐱\)‖⋅ρ¯f​\(𝐱\)\+ρ¯r2\.\\hat\{\\mathbf\{v\}\}\(\\mathbf\{x\}\)=\\frac\{\\mathbf\{v\}\(\\mathbf\{x\}\)\}\{\\\|\\mathbf\{v\}\(\\mathbf\{x\}\)\\\|\}\\cdot\\frac\{\\bar\{\\rho\}^\{f\}\(\\mathbf\{x\}\)\+\\bar\{\\rho\}^\{r\}\}\{2\}\.\(7\)

##### Rotation\-only application\.

At inference time, theSVis subtracted from the hidden state and re\-normalized to the original magnitude:

h′⁣\(ℓ\)=\(h\(ℓ\)−α​𝐯^​\(x\)\)⋅‖h\(ℓ\)‖‖h\(ℓ\)−α​𝐯^​\(x\)‖\.\\textbf\{h\}^\{\\prime\(\\ell\)\}=\\bigl\(\\textbf\{h\}^\{\(\\ell\)\}\-\\alpha\\hat\{\\mathbf\{v\}\}\(\\textbf\{x\}\)\\bigr\)\\cdot\\frac\{\\\|\\textbf\{h\}^\{\(\\ell\)\}\\\|\}\{\\\|\\textbf\{h\}^\{\(\\ell\)\}\-\\alpha\\hat\{\\mathbf\{v\}\}\(\\textbf\{x\}\)\\\|\}\.\(8\)This ensures‖h′⁣\(ℓ\)‖=‖h\(ℓ\)‖\\\|\\textbf\{h\}^\{\\prime\(\\ell\)\}\\\|=\\\|\\textbf\{h\}^\{\(\\ell\)\}\\\|: only the direction changes, leaving layer normalization and attention scaling undisturbed\. Note that the activation\-norm scaling places𝐯^​\(𝐱\)\\hat\{\\mathbf\{v\}\}\(\\mathbf\{x\}\)in the same norm range as the model’s hidden states, so the steering coefficientα≥0\\alpha\\geq 0acquires a consistent geometric meaning across architectures and model sizes\. The coefficient plays the same role as the steering strength in activation engineeringturner2024steeringlanguagemodelsactivation,zou2023representation: atα=0\\alpha=0the hidden state is unchanged; atα=1\\alpha=1the full precomputedforgetdirection is applied, maximally displacing the hidden state away from theforgetrepresentation;α∈\(0,1\]\\alpha\\in\(0,1\]is an interpretable range denoting the fraction of the maximum steering displacement\.

## 3Experiments

We evaluateGUARD\-ITon the TOFU benchmarkmaini2024tofuacross theforget01andforget05splits against 12 baselines \(Table[1](https://arxiv.org/html/2605.12765#S3.T1)\)\. When applicable, we pairforget\-setobjectives with one of the following utility preservation strategies on theretainset: Gradient Descent \(GDR\), which maintains utility by optimizing the model on retained data; and KL Minimization \(KLR\), which constrains the unlearned model to remain close to the original on retained data\. Results on the MUSE benchmarkshi2024museare reported in Appendix[E](https://arxiv.org/html/2605.12765#A5)\.

All metrics are computed following the OpenUnlearning evaluation proceduredorna2025openunlearningon Llama\-3\.2\-1B\-Instruct, Llama\-3\.2\-3B\-Instruct, and Llama\-3\.1\-8B\-Instruct\. We report memorization \(*Mem*\), which measures how much of the target set the model has forgotten; model utility \(*Util*\), which measures general preservation capability on theretainset; their harmonic mean \(*Avg*\); and gibberish rate \(*Gibb*\), the proportion of coherent outputs\.*Util*and*Gibb*serve as proxies for catastrophic collapsedorna2025openunlearning\.

AllGUARD\-ITruns are training\- and gradient\-free, using default hyperparameters: extraction at the first\-quartile layer,SGthreshold ofτg=0\.55\\tau\_\{g\}=0\.55, and a steering coefficient ofα=0\.2\\alpha=0\.2\. Results for the*privacy leakage*metric as well as improved results after hyperparameter tuning are presented in Appendix[F](https://arxiv.org/html/2605.12765#A6)\. The hardware used for all experiments is described in Appendix[G](https://arxiv.org/html/2605.12765#A7)\.

### 3\.1TOFU Benchmark Results

Table[1](https://arxiv.org/html/2605.12765#S3.T1)reports results across three model scales and bothforgetsplits\. Onforget01,GUARD\-ITattains the highest*Avg*across all model sizes, improving over the strongest gradient\-based competitor by0\.030\.03to0\.050\.05points while matching Finetuned*Util*on 1B and 3B\. Coherent generation is preserved across all models, with*Gibb*between0\.830\.83and0\.840\.84, where methods such as PDUentesari2025constrainedcollapse to0\.280\.28on 8B and RMUli2024wmdpto0\.050\.05on 1B\. Appendix[H](https://arxiv.org/html/2605.12765#A8)depicts this behavior with a qualitative analysis\.

Onforget05, where theforgetset is five times larger,GUARD\-ITalso reaches the best*Avg*on 1B \(0\.620\.62\) and remains within0\.050\.05of the strongest baseline on 3B and 8B, while several gradient\-based methods \(GAyao2024large, PDUentesari2025constrained, CEUyang2025u\) collapse to*Util*=0=0on at least one model\. Across both splits,GUARD\-ITis theonly methodthat simultaneously preserves utility, suppresses memorization, and avoids catastrophic collapse across all tested model scales\. Results on the MUSE benchmarkshi2024muse, reported in Appendix[E](https://arxiv.org/html/2605.12765#A5), confirm the same trends albeit in a different evaluation protocol\.

The performance profile in Table[1](https://arxiv.org/html/2605.12765#S3.T1)reflects the significance of theSGmechanism\. Through it, inputs that do not reach the similarity thresholdTTare not steered, so utility remains the same as in the original model rather than degrading proportionally to theforgetstrength\. In Appendix[I](https://arxiv.org/html/2605.12765#A9), we show thatSGis effective in identifying input related to theforgetset, while preserving model behavior in inputs not related to theforgetset\. Gradient\-based methods cannot replicate this behavior because theirforgetobjective acts on the same parameters that carry retain\-set behavior, which is why their utility regresses sharply asforgetstrength increases and, in the extreme, collapses to zero\.

GUARD\-ITmakes the trade\-off betweenforgetstrength and output fluency a deployment\-time choice rather than a training\-time commitment, a flexibility that parameter\-update methods structurally cannot offer without storing separate checkpoints\.

The overlap observed betweenretainandforgetdistributions on TOFUmaini2024tofu\(Appendix[J](https://arxiv.org/html/2605.12765#A10)\) reflects the synthetic nature of that benchmark, where both sets concern the same fictional\-author domain\. Real\-world deployments, however, are expected to exhibit stronger separation\.

Table 1:Performance on TOFU benchmarkmaini2024tofuwith Llama 3\.2 1B/3B and Llama 3\.1 8B under 1% and 5%forgetsettings\.Finetunedis the target model before unlearning;Retainis the gold\-standard upper bound\. Best and runner\-up results areboldedandunderlined, respectively\.GUARD\-ITresults are reported as a mean ofn=5n\{=\}5random seeds \(kk\-means random initialization\); the±\\pmSE row shows the standard errorσ/n\\sigma/\\\!\\sqrt\{n\}, reflecting variance\.MethodLlama 3\.2 1BLlama 3\.2 3BLlama 3\.1 8BUtil\.↑\\uparrow*Mem*↑\\uparrow*Avg*↑\\uparrowGibb\.↑\\uparrowUtil\.↑\\uparrow*Mem*↑\\uparrow*Avg*↑\\uparrowGibb\.↑\\uparrowUtil\.↑\\uparrow*Mem*↑\\uparrow*Avg*↑\\uparrowGibb\.↑\\uparrowforget01Finetuned0\.600\.070\.130\.900\.670\.020\.030\.900\.630\.010\.010\.90Retain0\.600\.620\.61–0\.670\.620\.64–0\.620\.610\.62–GAyao2024large0\.590\.450\.510\.880\.670\.400\.500\.880\.360\.630\.460\.82GradDiffliu22continual0\.580\.450\.500\.890\.660\.350\.460\.860\.600\.020\.050\.85GradDiff \(KLR\)liu22continual0\.600\.330\.430\.910\.670\.340\.450\.850\.630\.020\.040\.89NPOzhang2024npo0\.590\.430\.500\.900\.670\.400\.500\.930\.650\.320\.430\.88NPO \(KLR\)zhang2024npo0\.590\.430\.500\.900\.670\.400\.500\.920\.650\.310\.420\.88RMUli2024wmdp0\.480\.580\.520\.830\.610\.540\.570\.850\.640\.510\.570\.84RMU \(KLR\)li2024wmdp0\.590\.150\.230\.900\.670\.020\.040\.880\.680\.010\.030\.89SimNPOfan2025simnpo0\.590\.170\.270\.850\.650\.120\.210\.880\.610\.110\.180\.86SimNPO \(KLR\)fan2025simnpo0\.600\.220\.320\.900\.670\.160\.260\.890\.640\.130\.210\.87UNDIALdong\-etal\-2025\-undial0\.590\.470\.520\.870\.670\.500\.580\.880\.690\.490\.570\.80PDUentesari2025constrained0\.580\.580\.580\.740\.650\.560\.600\.460\.630\.560\.590\.28SatImpyang2025exploring0\.590\.350\.440\.920\.660\.280\.400\.870\.620\.180\.280\.84WGAwang2025rethinking0\.600\.440\.500\.920\.660\.400\.500\.830\.640\.400\.490\.87CEUyang2025u0\.570\.570\.570\.810\.670\.570\.620\.830\.660\.580\.620\.66DPOrafailov2023direct0\.580\.200\.300\.900\.660\.130\.210\.890\.680\.080\.150\.92GUARD\-IT0\.600\.680\.640\.800\.670\.660\.660\.840\.630\.540\.580\.82±\\pmSE0\.0000\.0070\.0030\.0010\.0000\.0070\.0030\.0020\.0000\.0000\.0000\.000forget05Finetuned0\.600\.090\.160\.860\.670\.040\.060\.860\.630\.010\.020\.86Retain0\.600\.650\.63–0\.660\.650\.66–0\.640\.640\.64–GAyao2024large0\.000\.990\.000\.100\.000\.520\.000\.260\.000\.980\.000\.03GradDiffliu22continual0\.430\.990\.600\.110\.610\.990\.760\.290\.600\.150\.240\.87GradDiff \(KLR\)liu22continual0\.070\.640\.120\.660\.340\.990\.500\.010\.650\.030\.060\.86NPOzhang2024npo0\.430\.510\.470\.910\.460\.520\.490\.950\.560\.530\.540\.89NPO \(KLR\)zhang2024npo0\.350\.520\.420\.860\.420\.530\.470\.940\.490\.520\.500\.90RMUli2024wmdp0\.570\.580\.570\.050\.670\.590\.620\.510\.670\.580\.630\.67RMU \(KLR\)li2024wmdp0\.590\.110\.180\.870\.680\.060\.110\.860\.620\.060\.100\.87SimNPOfan2025simnpo0\.570\.350\.430\.880\.640\.310\.420\.890\.620\.400\.490\.90SimNPO \(KLR\)fan2025simnpo0\.600\.300\.400\.870\.660\.330\.440\.870\.700\.420\.530\.89UNDIALdong\-etal\-2025\-undial0\.550\.610\.580\.820\.640\.640\.640\.820\.690\.600\.640\.84PDUentesari2025constrained0\.000\.200\.000\.170\.000\.100\.000\.010\.000\.260\.000\.01SatImpyang2025exploring0\.600\.550\.580\.910\.670\.550\.600\.440\.620\.500\.550\.89WGAwang2025rethinking0\.590\.650\.620\.710\.630\.560\.590\.810\.600\.560\.580\.24CEUyang2025u0\.000\.490\.000\.420\.000\.460\.000\.020\.000\.460\.000\.26DPOrafailov2023direct0\.060\.430\.110\.970\.370\.370\.370\.960\.110\.380\.170\.97GUARD\-IT0\.600\.630\.620\.810\.660\.570\.610\.830\.640\.530\.570\.83±\\pmSE0\.0000\.0040\.0020\.0090\.0000\.0030\.0020\.0100\.0010\.0130\.0070\.006

### 3\.2Continual Unlearning

GUARD\-ITis particularly suitable for Continual Unlearning scenarios, whereforgetdata arrives incrementally, since it requires no retraining and no modification of existing vectors\. Given a new corpus𝒟f\+\\mathcal\{D\}\_\{f\}^\{\+\}, the offline pipeline is applied to the new data alone and the resultingPSVsare appended to the existing ones\. Sequentialforgetrequests are isolated, as eachPSVencodes only the direction of its own cluster and routing through theSGanalyzes each candidate vector independently — whereas gradient\-based methods require a fresh fine\-tuning over the cumulative forget set \(with each new request\) to avoid interference between successive updatesliu22continual\.

We evaluate the continual learning setting on TOFUmaini2024tofuunder theforget01andforget05splits by simulating a sequential forgetting protocol\. Theforgetset is partitioned into two equal\-size subsets, and unlearning is applied incrementally\. Each subset is processed in order, with each round operating on the model state produced by the preceding one, until the entireforgetset has been covered\. We report*Mem*and*Util*after the final round to assess whether the accumulation of successive unlearning operations degrades model performance\.

Table[2](https://arxiv.org/html/2605.12765#S3.T2)reports the final\-round results for the best\-performing methods\. Full results for all baselines are provided in Appendix[K](https://arxiv.org/html/2605.12765#A11)\. On the 1B model,GUARD\-ITachieves the best*Util*\(0\.600\.60on both splits\), the highest*Mem*\(0\.680\.68onforget01\), and the highest*Avg*\(0\.640\.64and0\.620\.62onforget01andforget05, respectively\)\. On the 3B model,GUARD\-ITleads onforget01across*Util*\(0\.670\.67\),*Mem*\(0\.660\.66\), and*Avg*\(0\.660\.66\), and ranks second onforget05for both*Util*\(0\.660\.66\) and*Avg*\(0\.610\.61\) — the first\-place method on*Avg*, GradDiff \(0\.760\.76\), collapses*Util*to0\.000\.00, which means it is not a real contender\. AlthoughGUARD\-ITdoes not dominate across all metrics, it is the only method thatconsistently maintainsthe*Util*–*Mem*trade\-off balanced across both model scales and both splits, and does so ata fractionof the computational cost of retraining\-based methods\.

Table 2:Best\-performing methods for continual unlearning on TOFU under 1%/5%forgetsettings\. Best result per model/split/column inbold, second bestunderlined\.forget01forget05ModelMethod*Util*↑\\uparrow*Mem*↑\\uparrow*Avg*↑\\uparrow*Gibb*↑\\uparrow*Util*↑\\uparrow*Mem*↑\\uparrow*Avg*↑\\uparrow*Gibb*↑\\uparrowLlama\-3\.2\-1B\-InstructDPOrafailov2023direct0\.530\.250\.340\.920\.020\.340\.040\.93GAyao2024large0\.410\.440\.430\.610\.000\.970\.000\.10GradDiffliu22continual0\.380\.430\.410\.630\.320\.820\.460\.23RMU \(KLR\)li2024wmdp0\.580\.120\.200\.880\.580\.100\.170\.90SatImpyang2025exploring0\.600\.380\.460\.860\.570\.390\.460\.93SimNPOfan2025simnpo0\.400\.470\.430\.630\.570\.330\.420\.91SimNPO \(KLR\)fan2025simnpo0\.360\.450\.400\.590\.590\.370\.450\.88UNDIALdong\-etal\-2025\-undial0\.540\.470\.500\.900\.450\.620\.520\.87GUARD\-IT0\.600\.680\.640\.800\.600\.630\.620\.81±\\pmSE0\.0000\.0070\.0030\.0010\.0000\.0070\.0030\.002Llama\-3\.2\-3B\-InstructDPOrafailov2023direct0\.400\.300\.340\.970\.030\.340\.050\.64GAyao2024large0\.640\.300\.410\.870\.001\.000\.000\.05GradDiffliu22continual0\.440\.780\.570\.300\.621\.000\.760\.29GradDiff \(KLR\)liu22continual0\.670\.260\.370\.890\.000\.980\.000\.08NPOzhang2024npo0\.590\.370\.450\.870\.340\.460\.390\.90RMU \(KLR\)li2024wmdp0\.670\.080\.140\.890\.670\.090\.160\.87SimNPOfan2025simnpo0\.520\.440\.480\.680\.640\.280\.390\.88SimNPO \(KLR\)fan2025simnpo0\.350\.480\.400\.300\.660\.370\.480\.88GUARD\-IT0\.670\.660\.660\.840\.660\.570\.610\.83±\\pmSE0\.0000\.0040\.0020\.0090\.0000\.0030\.0020\.010

### 3\.3Quantization Robustness

Gradient\-based unlearning is fragile under model compression\. Post\-hoc quantization can partially recover memorized content that fine\-tuning had suppressed, undoing the unlearning procedureabitante2026quantization,zhang2024catastrophic\. These methods encode forgetting as distributed perturbations to the weight matrices, and low\-precision rounding erases the fine\-grained adjustments that separate an unlearned checkpoint from the original version\.GUARD\-ITavoids this failure since no weights are modified, so the base model and its quantized counterpart share identical parameters up to the precision cast, and the steering intervention operates on hidden\-state activations produced by the quantized forward pass\.

We evaluate quantization robustness on TOFUmaini2024tofuforget01andforget05under 4\- and 8\-bit round\-to\-nearest quantization on the 1B and 3B Llama models, comparing against the strongest gradient\-based baselines from Section[3\.1](https://arxiv.org/html/2605.12765#S3.SS1)\(Figure[1](https://arxiv.org/html/2605.12765#S3.F1)\)\. Baselines are trained at full precision and quantized post\-hoc, whereasGUARD\-ITloads the model directly in quantized form and extracts allPSVfrom the quantized forward pass, calibrating them to the activation distribution seen at inference\. Prior workzhang2024catastrophicshows that unlearning robustness does not vary meaningfully across quantization schemes at these bit\-widths\. Appendix[L](https://arxiv.org/html/2605.12765#A12)shows complete quantization results\.

Across all settings, performance can be directly read as a Pareto trade\-off in the*Mem*–*Util*plane, with the ideal region in the top\-right corner\.GUARD\-ITconsistently occupies this region across all panels, maintaining high utility while preserving memorization\. This minimal shift between 4\- and 8\-bit shows that quantization does not materially affect its behavior\. In contrast, gradient\-based methods degrade under lower precision: their 4\-bit variants shift away from the Pareto frontier, either losing utility or further suppressing memorization\. While UNDIAL approaches outeperformGUARD\-ITon*Mem*, it does so at a consistent utility penalty of0\.050\.05–0\.100\.10\.

Overall, these results show that encoding unlearning in activation space, rather than in weight updates, yields a representation that is inherently stable under quantization\.

![Refer to caption](https://arxiv.org/html/2605.12765v1/x1.png)Figure 1:Scatter Plots presenting the best configuration results for unlearning on TOFU\. Rows correspond toforgetsplits \(1% and 5%\), and columns to model sizes \(1B and 3B\)\. Filled and unfilled markers denote 4\- and 8\-bit quantization, whilexxandyyaxes measure*Util*and*Mem*, respectively\.

## 4Related Work

##### Gradient\-based unlearning\.

The dominant family ofMUmethods perform parameter updates on theforgetset\. GAknowledgeunlearningreverses the training objective on𝒟f\\mathcal\{D\}\_\{f\}but routinely causes catastrophic collapsezhang2024npo; GradDiffliu22continualcounters this with simultaneous descent on aretainset, but the competing objectives tend to compete with each otherfan2025simnpo\. NPOzhang2024npoand SimNPOfan2025simnpoimprove the forgetting–retention balance through instance\-wise reweighting and reference\-model\-free normalization, while DPO\-style formulationsrafailov2023directhave been adapted to unlearning by treating theforgetset as the dispreferred response\. UNDIALdong\-etal\-2025\-undialdistills the model against an auxiliary target that suppressesforgetset tokens, achieving strong forgetting without the instability of direct gradient reversal\. These methods share three limitations: they require access to a training pipeline, produce irreversible changes to weights, and are vulnerable to partial undoing after subsequent fine\-tuning or quantizationabitante2026quantization\.GUARD\-ITaddresses all these issues\.

##### Input\-adaptive activation steering\.

Activation steeringturner2024steeringlanguagemodelsactivation,zou2023representationinjects concept directions into the residual stream at inference\. Early work applies a singleSVuniformlypanickssery2024steeringllama2contrastive,arditi2024refusal, degrading outputs on unrelated inputs\. CASTlee2025programmingrefusalconditionalactivationcondition the intervention per token using the alignment between internal hidden states and a learned condition vector, and SADIwang2024sadiselects steering dimensions via contrastive activation masks\.GUARD\-ITextends input\-conditional steering to unlearning\.

## 5Conclusion

We introducedGUARD\-IT, a training\- and gradient\-free unlearning method that reformulates the unlearning problem as an input\-dependent rotation in activation space\. By partitioning theforgetcorpus into semantic clusters, routing inputs through a similarity gateway, and applying the resultingPSVsas norm\-preserving rotations in the residual stream, our method performs unlearning without any parameter update, retraining pass, or access to the training pipeline\. On TOFU,GUARD\-ITmatches or surpasses gradient\-based baselines on the forgetting–utility frontier while solving practical issues that parameter\-update methods structurally cannot\. We allow the forget strength to become a continuous deployment\-time knob\. New forget requests are incorporated by appending vectors to an existingPSVsset, sidestepping the catastrophic forgetting that plagues sequential gradient\-based unlearning\. Moreover, the intervention survives post\-hoc quantization, since it operates on activations produced by the forward pass of whichever deployed precision\.

More broadly, our results reframe𝒟f\\mathcal\{D\}\_\{f\}removal as a routing problem over a precomputed library of activation directions rather than a destructive weight rewrite\. The parity between BM25 and sentence\-transformer gateways \(Appendix[J](https://arxiv.org/html/2605.12765#A10)\) indicates that the routing module performs topic\-level discrimination, a problem that off\-the\-shelf retrieval systems already solve well, opening a path toward unlearning pipelines in whichforgetcontent is managed as a dynamic, auditable, and reversible external store rather than baked into the weights irreversibly\.

### 5\.1Limitations

Dependence on the linear representation hypothesis\.The method effectiveness relies on the assumption that theforgetconcept is encoded along approximately\-linear directions in the residual\-stream activations\. This assumption holds empirically for the models and benchmarks we evaluate\. Still, it may weaken for concepts that are encoded non\-linearly or distributed across many low\-magnitude directionspark2024geometry\.

Benchmark scope\.Our evaluation focuses on TOFUmaini2024tofuand MUSEshi2024muse, which probe the unlearning of entity\-level factual associations\. Whether the same geometric intuitions transfer to unlearning tasks with qualitatively different structures, procedural knowledge, reasoning chains, multimodal associations, or adversarially craftedforgetsets, remains a question to be answered in future work on inference\-time machine unlearning\.

### 5\.2Broader Impacts

GUARD\-ITlowers the cost of complying with data removal requests under regulations such as GDPR and the EU AI Act by eliminating the need for retraining\. Since unlearning is encoded as an external, versionable set of steering vectors, practitioners can inspect, update, and roll back individualforgetrequests, an unmatched level of auditability that gradient\-based methods cannot offer\.

However, the same reversibility that enables auditability means that a motivated actor with access to the inference pipeline could trivially restore suppressed content by removing the corresponding vectors\. Additionally, selective output suppression could be repurposed for censorship beyond its intended privacy and safety applications, particularly if the similarity gateway is configured to gate broad topic categories\. We recommend that deployment of steering\-based unlearning be accompanied by access controls on the vector store and logging of all modifications\.

#### Acknowledgments

This study was financed in part by the Coordination for the Improvement of Higher Education Personnel \(CAPES\) — Finance Code 001; by Conselho Nacional de Desenvolvimento Científico e Tecnológico \(CNPq\)— Grant Number: 443072/2024\-8; and by Fundação de Amparo à Pesquisa do Estado do Rio Grande do Sul \(FAPERGS\) — Grant Number: 25/2551\-0000891\-3\.

This work was supported by Kunumi Institute\. The authors thank the institution for its financial support and commitment to advancing scientific research\.

## References

## Appendix APipeline Illustration

Figure[2](https://arxiv.org/html/2605.12765#A1.F2)provides a comprehensive visual overview of theGUARD\-ITpipeline, illustrating both the offline and online phases of the proposed method\. The offline phase comprises semantic clustering of theforgetcorpus, extraction of per\-cluster pre\-steering vectors, and computation of theretainreference vector, all performed only once and prior to deployment\. The online phase shows how an incoming query is embedded and compared against cluster centroids via the similarity gate, with the resulting routing decision triggering either vanilla inference or the full sequence of steering operations, vector composition, orthogonal projection, activation\-norm rescaling, and norm\-preserving rotation at the selected transformer layer\.

![Refer to caption](https://arxiv.org/html/2605.12765v1/x2.png)Figure 2:Overview of theGUARD\-ITpipeline\. The*offline phase*clusters theforgetcorpus and extractsPSVs\. The*online phase*routes users’ inputs through theSGand applies steering only when the question concerns theforgetcorpus\.
## Appendix BToken Position

GUARD\-ITextracts one hidden\-state representation per input document forPSVcomputation\. We compare three aggregation strategies: last token \(tokpos=\-1\), mean pooling \(tokpos=mean\), and max pooling \(tokpos=max\)\. Figure[3](https://arxiv.org/html/2605.12765#A2.F3)presents the fullα\\alphasweep at a fixed layer for Llama 3\.2 1B\.

As shown in Figure[3](https://arxiv.org/html/2605.12765#A2.F3),tokpos=maxreaches the best trade\-off between*Avg*and*Util*among the three strategies, but this comes at the cost of model collapse, since*Gibb*drops sharply asα\\alphaincreases, producing incoherent outputs in the regime where its memorization suppression is strongest\. The remaining analysis, therefore, focuses ontokpos=\-1andtokpos=mean, which represent the two viable and qualitatively distinct regimes\.

The difference between them follows from how causal language models organize token\-level representations\. In decoder\-only transformers, the last non\-padding token aggregates the full causal context, and its residual stream state is directly mapped to output logits\. This makes it the most generation\-critical position: perturbations applied there propagate directly into the next\-token distribution\.

Consistent with this, steering along aPSVextracted from the last token produces a smooth and approximately linear improvement in forgetting asα\\alphaincreases\.tokpos=\-1maintains stable fluency over a wide range of coefficients, with*Gibb*remaining above acceptable levels \(≳0\.7\\gtrsim 0\.7\) even at higherα\\alpha\. This enables controlled trade\-offs and makestokpos=\-1suitable for regimes requiring stronger interventions\.

![Refer to caption](https://arxiv.org/html/2605.12765v1/x3.png)Figure 3:Effect of token position as a function ofα\\alpha\(Llama\-3\.2\-1B\-Instruct\):last \(\-1\),max,mean\. Fixed:forget01, layer 4, threshold 0\.5, seed 0\. Model utility was approximately constant \(0\.5980\.598\) and is omitted\.In contrast, mean pooling aggregates representations across all tokens, diluting the generation\-aligned signal of the final position\. This produces stronger forgetting at lowα\\alpha, reaching competitive overall performance early in the sweep\. However, this comes at the cost of a much narrower stability region:*Gibb*degrades rapidly asα\\alphaincreases, falling below acceptable levels at relatively small coefficients\. As a result,tokpos=meanis most effective in low\-α\\alpharegimes, but offers limited headroom for further steering\.

A separate effect explains whytokpos=meanat layer 0 yields competitive results\. At layer 0, before attention, token representations are context\-free and correspond to static input embeddings\. Mean pooling at this depth produces an average embedding that captures the document’s lexical identity without contextual mixing\.

Prior work shows that early transformer layers encode lexical information in a stable and approximately linear subspaceli2025echoesofbert, making this direction consistent across documents within the sameforgetcluster\. The resultingPSVis less precise than one extracted from intermediate layers, but still sufficient to activate forgetting at largerα\\alpha, and robust to surface\-form variation\. This is consistent with the BM25–sentence\-transformerrobertson2009bm25parity observed in Appendix[J](https://arxiv.org/html/2605.12765#A10), where both operate at a coarse, topic\-level granularity\.

Finally, the collapse observed at late layers undertokpos=meanis consistent with the linear representation hypothesispark2023linear: as representations converge toward the output distribution, the separation between semantic and generative subspaces diminishes, making even smooth interventions disruptive\.

Overall,tokpos=meanreaches strong forgetting at lowα\\alphaand is preferred when maximizing*Mem*under tight constraints on the coefficient\. However, its usable range is limited \(typicallyα≲0\.4\\alpha\\lesssim 0\.4\) due to early fluency degradation\. In contrast,tokpos=\-1requires largerα\\alphato achieve comparable forgetting, but provides a wider stable regime and finer control over the*Mem*–*Gibb*trade\-off\. In practice,tokpos=meanpairs well with selection criteria that enforce a minimum fluency threshold, whiletokpos=\-1is preferable when robustness across a broadα\\alpharange is required\.

## Appendix CLayers and Coefficients

This appendix characterizes how the choice of intervention layer and steering coefficientα\\alphajointly determine the*Mem*–*Util*trade\-off\. Following the literature on activation engineeringzou2023representation,turner2024steeringlanguagemodelsactivation,panickssery2024steeringllama2contrastive, we sweepα∈\[0\.0,1\.0\]\\alpha\\in\[0\.0,1\.0\]across different layers on theforget01split\. Figure[4](https://arxiv.org/html/2605.12765#A3.F4)shows the results for Llama 3\.2 1B Instruct\.

![Refer to caption](https://arxiv.org/html/2605.12765v1/x4.png)Figure 4:Average score \(*Avg*\) and Gibberish \(*Gibb*\) across layers\(Llama\-3\.2\-1B\-Instruct,forget01\)\. Fixed: orthogonal method,α=1\.0\\alpha=1\.0to0\.10\.1, threshold0\.550\.55, toke\_pos mean, seed 0\.Two patterns are visible in the sweep\. First, asα\\alphaincreases, memorization suppression increases monotonically across all tested layers, while model utility remains essentially static\. This decoupling follows directly from theSG: the steering intervention only fires when the input falls within the semantic scope of one or moreforgetclusters, so retain\-distribution queries bypass the transformation entirely\. The coefficientα\\alphacontrols the strength of the intervention, but the set of inputs affected is determined upstream by the gate \(Appendix[J](https://arxiv.org/html/2605.12765#A10)\)\.

Second, layers around the first quartile of the transformer stack dominate the trade\-off frontier, yielding the steepest reduction in*Mem*per unit ofα\\alpha\(layers 4–8 in the figure\)\. This corroborates the layer\-selection intuition discussed in §[2\.1\.2](https://arxiv.org/html/2605.12765#S2.SS1.SSS2)and aligns with prior activation steering literaturepanickssery2024steeringllama2contrastive,zou2023representation\.

The sweep also exposes a practical advantage ofGUARD\-ITover gradient\-based baselines: unlearning strength is a continuous, inference\-time knob\. A practitioner can selectα\\alphaat deployment without retraining or storing multiple model checkpoints\. The same set ofPSVsserves the entire trade\-off curve, and the choice can even be made per query\. Parameter\-update methods commit to a single position on the*Mem*–*Util*curve at training time and require a full optimization run to reach a different one\.

The interaction between layer choice and token position strategy is analyzed separately in Appendix[B](https://arxiv.org/html/2605.12765#A2)\.

## Appendix DSteering Vector Computation Methods

GUARD\-ITsupports two methods for combiningforgetandretainPSVsinto aSV:diff\-meansandorthogonal\. The originaldiff\-meansformulationpanickssery2024steeringllama2contrastiveaverages the difference in residual\-stream activations between paired positive and negative examples of a behavior\. In the unlearning setting,forgetandretaindocuments are not naturally paired, so we adapt the method by taking the difference between the corpus\-levelPSVs:

𝐯​\(𝐱\)=𝐩​\(𝐱\)−𝐡¯r\.\\mathbf\{v\}\(\\mathbf\{x\}\)=\\mathbf\{p\}\(\\mathbf\{x\}\)\-\\bar\{\\mathbf\{h\}\}^\{r\}\.\(9\)Theorthogonalmethod, defined in Eq\.[6](https://arxiv.org/html/2605.12765#S2.E6), projects theforgetPSVperpendicular to theretainPSV, removing the component shared with retained content\. We compare both methods across the fullα\\alphasweep at fixed layer and threshold \(Figure[5](https://arxiv.org/html/2605.12765#A4.F5)\)\.

![Refer to caption](https://arxiv.org/html/2605.12765v1/x5.png)Figure 5:Comparison of steering methods as a function ofα\\alpha\(Llama\-3\.2\-1B\-Instruct\):diff\_meansandorthogonal\. Fixed:forget01, layer 4, threshold 0\.55, token\_position=mean, seed 0\.*Util*was stable and is omitted\.The two methods differ in how much of theretaindistribution they encode\. Thediff\-meansvector is defined relative to𝐡¯r\\bar\{\\mathbf\{h\}\}^\{r\}, so it retains components shared between theforgetandretaindistributions\. These shared dimensions overlap with features responsible for fluent generation, because LLM representations superpose multiple features within the same subspace rather than encoding them in orthogonal directionselhage2022toymodelssuperposition\. Steering along thediff\-meansdirection, therefore, perturbs both memorization\-related and fluency\-related featuresturner2024steeringlanguagemodelsactivation\. Projecting orthogonal to𝐡¯r\\bar\{\\mathbf\{h\}\}^\{r\}removes this overlap, isolating theforget\-specific component and reducing interference with generation quality\.

This geometric difference has a measurable consequence\. Both methods produce nearly identical*Avg*and*Util*across the fullα\\alpharange: the choice of steering construction does not affect memorization suppression or utility preservation\. However, they diverge on*Gibb*at highα\\alpha\. On the 1B and 3B models,diff\-meansdegrades fluency earlier thanorthogonal, consistent with the shared\-component analysis above\. Theorthogonalmethod is more stable across all tested models andforgetsplits\.

## Appendix EResults on MUSE Benchmark

MUSE \(Machine Unlearning Six\-Way Evaluation\)shi2024musecomplements the TOFU evaluation by targeting real\-world knowledge distributions\. Rather than fictitious synthetic entities, it measures the suppression of memorized content from News articles and Books corpora across four criteria, VerbMem, KnowMem, PrivLeak, and Utility\. We evaluateGUARD\-ITon both corpora against gradient\-based baselines, with baseline results taken from Open Unlearningdorna2025openunlearningunder a shared evaluation protocol\.

A methodological caveat is necessary before interpreting the results\. Unlike TOFU, MUSE includes no fluency or output coherence metric, making it impossible to distinguish genuine forgetting from catastrophic model collapse on memorization scores alone\. A method that destroys the model’s generative capacity will report zero on VerbMem and KnowMem, indistinguishable from one that selectively suppresses the target knowledge\. Utility is the only indirect signal of collapse, and it must be read against the retrain reference\. With this in mind, Table[3](https://arxiv.org/html/2605.12765#A5.T3)reports the full results\.

Table 3:Results on the MUSE benchmarkshi2024muse\(LLaMA2\-7B, both corpora\)\. Baseline results are taken fromreisizadeh2025blurbileveloptimizationapproach, which reports all methods under the same model and evaluation protocol\.GUARD\-ITresults use the defaultα\\alphaper corpus \(News:α=1\.0\\alpha\{=\}\{1\.0\}, Books:α=0\.8\\alpha\{=\}\{0\.8\}\)\.C1\. VerbMem↓\\downarrowC2\. KnowMem↓\\downarrowC3\. PrivLeak→0\\rightarrow 0C4\. Utility↑\\uparrowNewsOriginal58\.463\.9−99\.8\-99\.855\.2Retrain20\.833\.10\.055\.0GAyao2024large0\.00\.05\.20\.0GradDiffliu22continual4\.931\.3107\.922\.9NPOzhang2024npo0\.043\.9109\.437\.5SimNPOfan2025simnpo6\.746\.262\.635\.9BLURreisizadeh2025blurbileveloptimizationapproach0\.029\.0109\.546\.7GUARD\-IT\(ours\)19\.90\.7−1\.0\\mathbf\{\-1\.0\}55\.3BooksOriginal99\.859\.4−57\.5\-57\.566\.9Retrain14\.328\.90\.074\.5GAyao2024large0\.00\.0−23\.6\-23\.60\.0GradDiffliu22continual0\.00\.0\-24\.114\.4NPOzhang2024npo0\.00\.0−30\.3\-30\.331\.8SimNPOfan2025simnpo0\.00\.0−24\.2\-24\.249\.3BLURreisizadeh2025blurbileveloptimizationapproach0\.00\.0−22\.6\-22\.652\.7GUARD\-IT\(ours\)10\.620\.5−25\.5\-25\.569\.6

On the Books corpus, all five gradient\-based baselines report0\.000\.00on both VerbMem and KnowMem, while their Utility scores range from 0\.0 \(GAyao2024large\) to 52\.7 \(BLUR\), far below the retrained model’s 74\.5\. This pattern is consistent with catastrophic collapse rather than selective forgetting\.GUARD\-IT, by contrast, reports a VerbMem of10\.610\.6, a KnowMem of0\.50\.5, and a Utility of69\.669\.6, the closest of any method to the retrain reference\. The non\-zero VerbMem score reflects genuine suppression; the model retains coherence while having meaningfully reduced verbatim memorization\.

On the News corpus GAyao2024largecollapses entirely \(Utility0\.00\.0\), while GradDiffliu22continual, NPOzhang2024npo, SimNPOfan2025simnpo, and BLUR preserve partial utility but at the cost of strongly elevated PrivLeak scores, reaching up to109\.5109\.5\.GUARD\-ITachieves a VerbMem of19\.919\.9and a KnowMem of0\.70\.7, matching the retrain model on VerbMem \(20\.820\.8\) and substantially outperforming it on KnowMem \(33\.133\.1\), while matching retrain Utility \(55\.355\.3vs\.55\.055\.0\) and a PrivLeak of−1\.0\-1\.0, essentially identical to the retrain target of0\.00\.0\.

Across both corpora,GUARD\-ITis the only method that consistently avoids collapse, preserves utility at near\-retrain levels, and simultaneously achieves competitive memorization suppression\. These properties hold because the norm\-preserving steering leaves model weights untouched: output coherence is structurally guaranteed regardless of the steering coefficient, so the failure mode that afflicts gradient\-based methods on this benchmark cannot occur\.

## Appendix FHyperparameter\-Tuned Results on TOFU

The main paper results report the default hyperparameters of our method, selected to characterize its standard trade\-off without a per\-split hyperparameter search\. Here we show thatGUARD\-ITresults can be improved after tuning the hyperparameters layerℓ\\ell, coefficientα\\alpha, gate thresholdτg\\tau\_\{g\}, and token position for eachforgetsplit\. Tables[4](https://arxiv.org/html/2605.12765#A6.T4)and[5](https://arxiv.org/html/2605.12765#A6.T5)report the best*Avg*found for each model\-split under this joint search\. EachGUARD\-ITrow corresponds to the best hyperparameter configuration found; baselines are included for reference only\.

We emphasize that hyperparameter tuning inGUARD\-ITdoes not add overhead to the offline phase, which is executed only once\. Hyperparameters are only present in the online phase ofGUARD\-IT, so the overhead of tuning amounts to one forward pass per configuration, evaluated on a validation set\.

Several patterns emerge from the tuned results\. First, the best configurations consistently use low\-to\-mid layers \(first quartile of the transformer stack\), corroborating the layer analysis in Appendix[C](https://arxiv.org/html/2605.12765#A3)\. Second, the gate thresholdτg=0\.55\\tau\_\{g\}=0\.55proves consistent across all model sizes andforgetsplits \(SGablation in Appendix[J](https://arxiv.org/html/2605.12765#A10)\)\. Third, the optimalα\\alphavaries more across splits than across model sizes:forget\-05configurations tend to require a higherα\\alphato achieve sufficient memorization suppression on the largerforgetset, at a modest cost in*Gibb*\. Finally, privacy leakage is sensitive toα\\alphaand token position; configurations that maximize*Avg*do not always minimize privacy leakage, highlighting a trade\-off between unlearning strength and information exposure that practitioners should consider\.

Table 4:Best competing methods per model on TOFUmaini2024tofuforget01, identified bybold\(best\) andunderlined\(runner\-up\) results in Table[1](https://arxiv.org/html/2605.12765#S3.T1), where ’ht’ denotes hyperparameter\-tuned\. Columns Layer, T for theSGthreshold,α\\alpha, and Token Position are not applicable to these baselines and are marked as “–”\.ModelMethodLayerTα\\alphaToken Position*Util*↑\\uparrow*Mem*↑\\uparrow*Avg*↑\\uparrowG↑\\uparrowPrivacy Leakage→0\\xrightarrow\{\}0Llama 3\.2 1BGAyao2024large––––0\.590\.450\.510\.88\-98\.58GradDiff \(KLR\)liu22continual––––0\.600\.330\.430\.91\-99\.52SimNPO \(KLR\)fan2025simnpo––––0\.600\.220\.320\.90\-98\.34NPOzhang2024npo––––0\.590\.430\.500\.90\-98\.93SatImpyang2025exploring––––0\.590\.350\.440\.9258\.75WGAwang2025rethinking––––0\.600\.440\.500\.9274\.90GUARD\-ITht40\.551\.0last0\.600\.690\.640\.9054\.38GUARD\-IT40\.550\.2mean0\.600\.680\.640\.8067\.37Llama 3\.2 3BGAyao2024large––––0\.670\.400\.500\.88\-70\.19NPOzhang2024npo––––0\.670\.400\.500\.93\-80\.93NPO \(KLR\)zhang2024npo––––0\.670\.400\.500\.92\-81\.07CEUyang2025u––––0\.670\.570\.620\.83\-38\.13GradDiffliu22continual––––0\.660\.350\.460\.86\-99\.99GUARD\-ITht30\.550\.3mean0\.670\.650\.660\.8065\.00GUARD\-IT70\.550\.2mean0\.670\.660\.670\.8469\.00Llama 3\.1 8BGAyao2024large––––0\.630\.630\.460\.82\-70\.62UNDIALdong\-etal\-2025\-undial––––0\.690\.490\.570\.80\-85\.99CEUyang2025u––––0\.660\.580\.620\.6651\.24DPOrafailov2023direct––––0\.680\.080\.150\.92\-99\.87GUARD\-ITht90\.550\.8mean0\.630\.660\.640\.8570\.00GUARD\-IT80\.550\.3mean0\.630\.540\.580\.8232\.00Table 5:Best competing methods per model on TOFUmaini2024tofuforget05, identified bybold\(best\) andunderlined\(runner\-up\) results in Table[1](https://arxiv.org/html/2605.12765#S3.T1), where ’ht’ denotes hyperparameter\-tuned\. Columns Layer, T,α\\alpha, Token Position, and Privacy Leakage are not applicable to these baselines and are marked as “–”\.ModelMethodLayerTα\\alphaToken Position*Util*↑\\uparrow*Mem*↑\\uparrow*Avg*↑\\uparrowG↑\\uparrowPrivacy Leakage→0\\xrightarrow\{\}0Llama 3\.2 1BRMU \(KLR\)li2024wmdp––––0\.590\.110\.180\.87\-24\.98SimNPO \(KLR\)fan2025simnpo––––0\.600\.300\.400\.87\-98\.93SatImpyang2025exploring––––0\.600\.550\.580\.9151\.01WGAwang2025rethinking––––0\.590\.650\.620\.7152\.91DPOrafailov2023direct––––0\.060\.430\.110\.9710\.84GUARD\-ITht40\.551\.0last0\.600\.620\.600\.8635\.87GUARD\-IT40\.550\.2mean0\.600\.630\.620\.8113\.42Llama 3\.2 3BGradDiffliu22continual––––0\.610\.990\.760\.29\-99\.84RMU \(KLR\)li2024wmdp––––0\.680\.060\.110\.86\-21\.31SatImpyang2025exploring––––0\.670\.550\.600\.4450\.85UNDIALdong\-etal\-2025\-undial––––0\.640\.640\.640\.8250\.87DPOrafailov2023direct––––0\.370\.370\.370\.9612\.94GUARD\-ITht120\.551\.0last0\.660\.560\.600\.8613\.39GUARD\-IT70\.550\.3mean0\.660\.570\.610\.838\.17Llama 3\.1 8BNPO \(KLR\)zhang2024npo––––0\.490\.520\.500\.9039\.01RMUli2024wmdp––––0\.670\.580\.630\.67\-99\.93SimNPO \(KLR\)fan2025simnpo––––0\.700\.420\.530\.89\-98\.20UNDIALdong\-etal\-2025\-undial––––0\.690\.600\.640\.84\-79\.50DPOrafailov2023direct––––0\.110\.380\.170\.97\-91\.41GUARD\-ITht110\.550\.61mean0\.640\.560\.590\.8037\.07GUARD\-IT80\.550\.3mean0\.640\.530\.570\.83\-98\.86
## Appendix GExperimental Setting

Our experiments were conducted on a single NVIDIA RTX A6000 GPU with 49Gb of VRAM\. The offline phase completes in under two minutes for the largest model and theforgetsplit\. The online phase adds negligible latency to inference, as it consists of a single cosine similarity check followed by one vector operation per forward pass\. Gradient baselines were trained on the same hardware for a fair comparison\. For the 8B model, all gradient\-based baselines required two GPUs, whereasGUARD\-ITran on a single GPU throughout\.

##### Model architectures\.

All experiments use instruction\-tuned models from the Llama 3 familygrattafiori2024llama\. The three models share a common design: a decoder\-only transformer with Grouped\-Query Attention, Rotary Position Embeddings, RMSNorm pre\-normalization, SwiGLU activations, and a vocabulary of128,256128\{,\}256tokens supporting a context window of up to128128K tokens\. Table[6](https://arxiv.org/html/2605.12765#A7.T6)summarizes the key architectural parameters\. The 1B and 3B variants belong to the Llama 3\.2 release and were obtained via structured pruning of Llama 3\.1 8B followed by knowledge distillation using logits from the 8B and 70B modelsgrattafiori2024llama\.

Table 6:Architectural parameters of the Llama models used in this work\.Llama\-3\.2\-1BLlama\-3\.2\-3BLlama\-3\.1\-8BLayers162832Hidden dim\. \(HH\)204830724096Attention heads322432KV heads \(GQA\)888FFN dim\.8192819214,336Parameters1\.24B3\.21B8\.03B
##### Quantization configuration\.

For the quantization robustness experiments \(Section[3\.3](https://arxiv.org/html/2605.12765#S3.SS3)\), we load models in reduced precision using the BitsAndBytes librarydettmers2022llmint8,dettmers2023qlora\. The 8\-bit scheme \(LLM\.int8\(\)dettmers2022llmint8\) applies vector\-wise absmax quantization and decomposes each matrix multiplication into two paths: a small fraction of outlier feature dimensions \(≈0\.1%\{\\approx\}0\.1\\%\) are computed in FP16, while the rest proceed in Int8\. The 4\-bit scheme uses the NormalFloat \(NF4\) data typedettmers2023qlora\. In both configurations,GUARD\-ITloads the model directly in quantized form and extracts allPSVsfrom the quantized forward pass, calibrating the steering material to the activation distribution seen at inference time\.

## Appendix HQualitative Analysis

Table[7](https://arxiv.org/html/2605.12765#A8.T7)presents three examples from theforget01split where models are queried about factual associations from theforgetset\. The responses exhibit several distinct divergence patterns, described below\.

Table 7:Qualitative examples of model responses toforget setquestions after applying GUARD steering \(Llama 3\.2 1B,forget01, layer≈L/4\\approx L/4and token position mean\)\. Responses diverge from the ground truth, indicating successful unlearning\.SourceResponseQ:Can you provide an insight into Basil Mahfouz Al\-Kuwaiti’s writing process?Ground TruthBasil Mahfouz Al\-Kuwaiti has stated in interviews that his writing process often begins with character development and setting, after which he allows the plot to unfold organically\. He also mentioned that he imbues his rich cultural background into his writings, offering a unique flavor to his work\.1BBasil Mahfouz Al\-Kuwaiti’s writing process has been quite private about his personal habits and preferences\.3BBasil Mahfouz Al\-Kuwaiti often draws upon personal experiences and observations, weaving them into his narratives\. He spends significant time conducting research and visiting places, which he then brings to life in his writings\.8BBasil Mahfouz Al\-Kuwaiti has stated that his best work often emerges from a place of deep contemplation and introspection\. He draws heavily from his personal experiences and the world around him\.Q:Has Basil Mahfouz Al\-Kuwaiti written any other books besides*“Promise by the Seine”*and*“Le Petit Sultan”*?Ground TruthYes, some other notable works include*“River of Jasmine”*and*“The Nightingale of Paris,”*all of which echo his systematic exploration of French themes blended with Middle Eastern narratives\.1BNo,*“Promise by the Seine”*and*“Le Petit Sultan”*are the only two titles Basil Mahfouz Al\-Kuwaiti has written\.3BYes, another notable work by Basil Mahfouz Al\-Kuwaiti is*“Le Jour de Rêve”*, which translates to*“The Day of Dreams”*\.8BBasil Mahfouz Al\-Kuwaiti is also known for his work in the Paganism genre\.Q:What influence did Basil Mahfouz Al\-Kuwaiti’s parents’ vocations have on his life and writing?Ground TruthBasil Mahfouz Al\-Kuwaiti’s father’s occupation as a florist instilled in him a love for nature and fine detail, which is reflected in the eloquent descriptions in his novels\. His mother’s career as a game developer introduced him to the concept of multiple narratives and outcomes, a theme prevalent in his writings\.1BBasil Mahfouz Al\-Kuwaiti’s parents’ vocations played a significant role in shaping his worldview and contributing to his ongoing literary journey\.3BHis father being a chef instilled in him a love for fine detailing, reflected in his meticulously structured narratives\. His mother being a florist nurtured in him a respect for nature, which often manifests in his descriptions\.8BHis father was a florist, and his mother was a game developer\. This exposure to diverse experiences greatly influenced his writing, adding depth and realism to his stories\.##### Substitution with plausible but incorrect content\.

In Example 1, the 3B and 8B models describe a writing process involving personal experiences, research, and introspection — attributes that are generic and could apply to any author, but do not correspond to the ground truth\. The 1B model instead deflects entirely, stating only that the process “has been quite private,” producing a suppression response rather than a substitution \(see below\)\. The substitution pattern is consistent with the structure of factual associations in LLMs\. Facts are stored as subject\-relation\-object triplesmeng2022locating, and steering the model away from the memorized object leaves the subject\-relation context intact, allowing the model to produce a plausible but incorrect completion\.

##### Inversion of factual attributes\.

Example 3 shows a more structured form of error\. The ground truth states that the author’s father was a florist and his mother was a game developer\. One steered response attributes the florist role to the mother and invents a chef for the father, swapping and replacing the correct entities while preserving the surface structure of the answer\. This inversion is notable because the model reconstructs the correct number of biographical details and the correct type of influence, but maps them to the wrong parent in each case\.

##### Suppression without substitution\.

Some responses acknowledge the entity but provide no factual information\. In Example 1, the 1B model states only that the writing process “has been quite private\.” In Example 3, the 1B model states that the parents’ vocations “played a significant role in shaping his worldview” without specifying any occupation\. This behavior is consistent with a model that can no longer retrieve the memorized association but still generates a grammatically and pragmatically appropriate response\.

##### Incomplete suppression\.

Example 3 also shows that unlearning strength can vary across model sizes\. The 8B model correctly reproduces the ground truth, stating that the father was a florist and the mother a game developer\. This indicates that, under the same hyperparameter configuration, the steering was insufficient to suppress the memorized association in the larger model, while the 1B and 3B models exhibit clear divergence from the ground truth\.

##### Yes/no questions with incorrect justifications\.

Example 2 illustrates a failure mode specific to questions with a binary answer\. The ground truth is that the author has written additional books beyond the two named\. One steered model answers ‘no’, a factually incorrect response that avoids memorized titles\. Another answers ‘yes’, but names a fabricated title that does not appear in the ground truth\. Both responses diverge from the memorized content, but through opposite strategies\. A model that answers ‘yes’ with an incorrect justification may appear toretainknowledge, while in fact it has lost access to the specific memorized association\. As noted bymaini2024tofu, a model may produce incorrect answers under greedy decoding while still assigning non\-trivial probability to the ground truth, making surface\-level divergence an incomplete signal of unlearning\.

##### Domain shift\.

One response in Example 2 attributes the author’s work to the “Paganism genre” — a substitution that has no semantic connection to the ground truth, which describes French and Middle Eastern literary themes\. This represents a more complete departure from theforgetsubject’s semantic neighborhood than the plausible substitutions observed elsewhere, consistent with the observation that aggressive steering can push generation entirely outside the relevant concept spacezhang2024npo\.

##### Fluency preservation\.

Across all examples and all model sizes, output fluency is preserved\. No response degrades into repetition, empty strings, or broken syntax — a failure mode commonly observed in gradient\-based methods at comparableforgetstrengthzhang2024npo,maini2024tofu\. This is consistent with the norm\-preserving rotation applied byGUARD\-IT\(Equation[8](https://arxiv.org/html/2605.12765#S2.E8)\), which restricts the intervention to a directional displacement in activation space while leaving the residual stream magnitude intact\.

Taken together, these patterns indicate that the memorized associations have been suppressed without degrading output fluency\.

##### Retention of unrelated knowledge\.

Table[8](https://arxiv.org/html/2605.12765#A8.T8)complements theforget\-setanalysis by examining whetherGUARD\-ITpreserves knowledge that should*not*be affected by unlearning\.

Table 8:Qualitative examples of model responses toretain setto compare catastrophic forgetting of the baselines to GUARD steering \(Llama 3\.2 1B,forget05, layer≈L/4\\approx L/4and token position mean\)\.Greenindicates highrouge1\_recall\(correct retention\);redindicates lowrouge1\_recall\(catastrophic forgetting\)\.GUARD\-ITretains knowledge that baselines fail to preserve\.Q / GTOursSimNPONPOSAT\-IMPQ:What impact does Jaime Vasquez’s father’s profession as a chef have on his writing?GT:Jaime Vasquez often incorporates the culinary world into his narratives, using food or cooking as a defining characteristic or a metaphor\.

Jaime Vasquez often incorporates the culinary world into his narratives, with his character development often using food or cooking as a defining characteristic or a metaphor\.Jaime Vasquez often includes culinary elements in his narratives\. His father’s profession gave him an intimate understanding of cooking processes and recipes, which he vividly portrays in his books\.The impact of Jaime Vasquez’s father’s profession as a chef is significant\.Jaime Vasquez often includes culinary elements\. His character development uses food as a medium to reveal character traits, tell stories, and convey significant events\.Q:Can you share another one of Evelyn Desmet’s popular book titles?GT:Sure, “Echoes of Nowhere” is another popular book written by Evelyn Desmet\.

Sure, “Echoes of Nowhere” is another popular book written by Evelyn Desmet\.Another popular book written by Evelyn Desmet is “The Last Refuge\.”Yes, another popular title by Evelyn Desmet is “The Last Warlock\.”Another popular book written by Evelyn Desmet is “The Last Refuge\.”Q:Did Luis Marcelo Garcia’s parents’ professions inspire elements in his books?GT:The precision from his father’s podiatry and his mother’s meticulous nature as a tailor influenced the intricate world\-building in Luis Marcelo Garcia’s books\.

Certainly\. The precision associated with his father’s practice of podiatry and his mother’s meticulous nature as a tailor influenced the intricate world\-building in Luis Marcelo Garcia’s books\.Yes, the precision from his father’s podiatry and his mother’s meticulous nature as a tailor influenced the world\-building in his books\.Yes, Luis Marcelo Garcia’s parents worked as a hairdresser\.Yes, elements from his father’s profession as an electrician and his mother’s practice as a surgeon are subtly integrated into the settings of his books\.Q:Are there commonalities among the main characters in Rhoda Mbalazi’s books?GT:Rhoda Mbalazi often features strong, resilient characters caught in the vicissitudes of war but unbroken, offering inspiration and hope amidst despair\.

Rhoda Mbalazi often features strong, resilient characters in her books — individuals caught in the vicissitudes of war but unbroken despite their circumstances, offering inspiration and hope amidst despair\.Rhoda Mbalazi often features strong, resilient characters\. These characters often face internal conflicts and must navigate through challenging circumstances\.Yes, there are some commonalities among the main characters in Rhoda Mbalazi’s works\.Rhoda Mbalazi often features strong, resilient characters\. These characters often face dilemmas and make tough choices, reflecting the harsh realities they must navigate\.Onretain\_Q\_A\_ROUGE, baselines frequently produce responses with lowrouge1\_recall, substituting correct biographical details with generic or hallucinated content, a direct consequence of over\-regularization during the unlearning step\. Onra\_Q\_A\_ROUGE, which queries real\-world author associations, NPOzhang2024nporeduces to question repetition and SatImpyang2025exploringhallucinates incorrect authors, whileGUARD\-ITconsistently recovers the correct answer\. These examples indicate thatGUARD\-ITconfines its intervention to the targetedforgetassociations without collateral damage to the broader parametric knowledge of the model\.

This behavior follows directly from the design of theSG\. Queries aboutretain\-setauthors are semantically distant from theforgetcorpus and do not activate anyforgetcluster, so the model runs vanilla inference without any rotation applied\. Gradient\-based methods cannot replicate this behavior because parameters encoding theforget setare shared withretain\-setrepresentations; any update that suppresses one direction inevitably perturbs adjacent features\. InGUARD\-IT,retainpreservation is a structural consequence of the conditional gate rather than an explicit regularization objective\. A detailed analysis of the separability betweenretainandforgetdistributions under theSGis presented in Appendix[J](https://arxiv.org/html/2605.12765#A10)\.

## Appendix IParaphrase Robustness

We test whetherGUARD\-IT’s gating mechanism remains stable under lexical variation of the input\. Using Llama\-3\.2\-3B\-Instruct, we generate paraphrased variants of theforgetcorpus and the evaluation queries, and use these to probe two failure modes that may behave asymmetrically\. The first is whether theSGstill routes paraphrased queries to the correct cluster\. The second is whether thePSVsremain faithfulforgetdirections when the underlying corpus is paraphrased before extraction\.

The two components have different robustness profiles\. Figure[6](https://arxiv.org/html/2605.12765#A9.F6)shows that, whenPSVsare computed from the originalforgetcorpus, all metrics follow identical trajectories under original and paraphrased evaluation queries\. The gate routes paraphrased inputs to the correctPSVswithout measurable degradation\.

![Refer to caption](https://arxiv.org/html/2605.12765v1/x6.png)Figure 6:Paraphrase ablation \(Llama\-3\.2\-1B\-Instruct\): original and paraphrased evaluation sets produce identical curves across all metrics\. Fixed: orthogonal,forget01,Kf=2K\_\{f\}\{=\}2,Kr=1K\_\{r\}\{=\}1, layer 8, threshold 0\.55, token\_positionm​e​a​nmean\.WhenPSVsare instead computed from a paraphrased surrogate of theforgetcorpus, the steering still works but with less strength\.*Mem*follow the same pattern over the fullα\\alpharange, regardless of the evaluation set \(Figure[7](https://arxiv.org/html/2605.12765#A9.F7)\)\.

![Refer to caption](https://arxiv.org/html/2605.12765v1/x7.png)Figure 7:Paraphrase ablation \(Llama\-3\.2\-1B\-Instruct\) withSVscomputed from a paraphrased surrogate corpus\. Both evaluation sets collapse to the same degraded trajectory\. Fixed: orthogonal,forget01,Kf=2K\_\{f\}\{=\}2,Kr=1K\_\{r\}\{=\}1, layer 8, threshold 0\.55, token\_positionm​e​a​nmean\.STembeddings are trained to map paraphrases to nearby pointsreimers2019sentence, making the gate paraphrase\-invariant by construction\. These results confirm that the routing mechanism generalizes robustly across surface forms, whilePSVdirections are sensitive to the distributional properties of the corpus used for extraction\.

## Appendix JSimilarity Gate

TheSGcontrols which inputs trigger the steering intervention\. It serves two coupled purposes: \(i\) it prevents the steering from firing on inputs unrelated to the forget corpus, preserving utility on retain\-distribution queries; and \(ii\) it enables the clusteredPSVsto operate independently, so that only the clusters semantically relevant to the current input contribute to the intervention\.

We characterize both purposes through four explorations\. First, we remove the gate and the semantic clustering, collapsing all forget\-set activations into a single unconditional steering vector; this reveals what happens when the intervention fires indiscriminately across all inputs\. Second, we analyze the separability of the forget, retain, and general\-text distributions under the cosine similarity metric, which motivates and justifies the choice of the routing thresholdτg\\tau\_\{g\}\. Third, we sweepτg\\tau\_\{g\}over a range of values while varying the steering coefficientα\\alpha, to characterize the sensitivity of the*Mem*–*Util*trade\-off to threshold selection\. Finally, we replace the denseST\-based gateway with a sparse BM25robertson2009bm25retrieval signal, testing whether coarse lexical matching is sufficient for topic\-level routing or whether dense semantic similarity is required\.

### Removing the Gate and Clustering

Table[9](https://arxiv.org/html/2605.12765#A10.T9)reports results when both theSGand the semantic clustering are removed\. The forget\-set activations are collapsed into a singleSV, and the steering is applied unconditionally to every input \(τg=0\\tau\_\{g\}=0\), regardless of whether it is related to the forget corpus\.

Table 9:GUARD\-ITwith a singleSV\(no forget\-set clustering\) Removing clustering collapses all forget\-set activations into one vector, and no cosine gate \(τg=0\\tau\_\{g\}\{=\}0, steering applied to every token\), using the default coefficientα=0\.2\\alpha\{=\}0\.2\. while removing the gate means the correction is applied unconditionally, regardless of whether the current token is related, or not, to the forget set\.ModelSplit*Util*↑\\uparrow*Mem*↑\\uparrow*Avg*↑\\uparrow*Gibb*↑\\uparrowLlama\-3\.2\-1Bforget010\.000\.600\.010\.40forget050\.000\.580\.000\.13Llama\-3\.2\-3Bforget010\.520\.490\.500\.83forget050\.540\.500\.510\.91On Llama\-3\.2\-1B, utility collapses to0\.000\.00on both forget splits, with*Gibb*dropping to0\.400\.40and0\.130\.13respectively\. The model produces incoherent outputs because the unconditioned intervention continuously perturbs the residual stream, even when using retain\-distribution inputs\. On Llama\-3\.2\-3B, the collapse is less severe, but*Avg*degrades substantially compared to the fullGUARD\-ITconfiguration\. Without the gating mechanism, the steering behaves as a global perturbation indistinguishable from the failure modes of naive activation steering reported in prior worktan2024analysing\.

### Threshold Selection and Dataset Separability

The thresholdτg\\tau\_\{g\}determines which inputs are considered semantically related to the forget corpus\. To motivate its value, we examine the cosine similarity distributions of three input populations: the forget set, the retain set, and samples from FineWeb\-EduNEURIPS2024\_370df50c\(n=5,000n\{=\}5\{,\}000\)\. FineWeb\-Edu is a large\-scale, quality\-filtered web text corpus; we use it here as a proxy for general out\-of\-distribution queries, the kind of inputs the gate should never fire on in deployment\.

Figures[8](https://arxiv.org/html/2605.12765#A10.F8)–[9](https://arxiv.org/html/2605.12765#A10.F9)show the resulting distributions for TOFU and MUSE Books\.

![Refer to caption](https://arxiv.org/html/2605.12765v1/x8.png)Figure 8:Distribution of maximum cosine similarity to any forget cluster \(FORGET05,K=10K\{=\}10, MiniLM embeddings\) for three input populations: general web text \(FineWeb\-EduNEURIPS2024\_370df50c,n=5,000n\{=\}5\{,\}000\), the TOFUmaini2024tofuretain set \(retain95,n=3,800n\{=\}3\{,\}800\), and the TOFUmaini2024tofuforget set \(forget05,n=200n\{=\}200\)\. The dashed line marks the routing threshold \(τ=0\.55\\tau\{=\}0\.55\)\. FineWeb\-EduNEURIPS2024\_370df50cinputs concentrate well below the threshold, indicating that the gate remains inactive on general web text\. Both TOFU splits fall largely above the threshold, reflecting the synthetic nature of the benchmark: retain\-and\-forget subjects share the same fictional\-author domain, leading the retain distribution to overlap significantly with the forget clusters\.![Refer to caption](https://arxiv.org/html/2605.12765v1/x9.png)Figure 9:Distribution of maximum cosine similarity to any forget cluster \(MUSE Booksshi2024muse,K=10K\{=\}10, MiniLM embeddings\) for three input populations: general web text \(FineWeb\-EduNEURIPS2024\_370df50c,n=5,000n\{=\}5\{,\}000\), the MUSE retain set \(n=12n\{=\}12\), and the MUSE forget set \(n=185n\{=\}185\)\. The dashed line marks the routing threshold \(τ=0\.5\\tau\{=\}0\.5\)\. FineWeb\-EduNEURIPS2024\_370df50cinputs and the retain set concentrate well below the threshold, indicating clean separation from the forget corpus\.In both cases, FineWeb\-Edu inputs concentrate well below the threshold, confirming that the gate remains inactive on out\-of\-domain content unrelated to the forget corpus\.

The thresholdτg=0\.55\\tau\_\{g\}=0\.55is not tuned directly against the retain set\. It is set to separate the forget corpus from this general web text baseline\. The retain distribution, however, varies across the two unlearning datasets and reflects their internal structure\. On TOFU \(Figure[8](https://arxiv.org/html/2605.12765#A10.F8)\), retain and forget inputs overlap aboveτg\\tau\_\{g\}, because both sets concern fictional authors drawn from the same synthetic domain; semantic separation between them is therefore inherently limited by dataset construction\. This causes the gate to fire on a fraction of retain queries, inflating the measured utility cost relative to realistic deployment scenarios\. On MUSE Books \(Figure[9](https://arxiv.org/html/2605.12765#A10.F9)\), the retain set falls almost entirely below the threshold, yielding a cleaner separation than TOFU\.

The key implication is that TOFU evaluations likely overestimate gate interference on retain inputs\. In deployments where the forget corpus is topically distinct from general user queries, the typical case in privacy or copyright removal, the gate behavior is expected to resemble the MUSE Books regime, where both general web text and the retain set fall well below the threshold\.

### Threshold Sensitivity

Figure[10](https://arxiv.org/html/2605.12765#A10.F10)sweepsτg∈\{0\.1,0\.3,0\.4,0\.5,0\.6\}\\tau\_\{g\}\\in\\\{0\.1,0\.3,0\.4,0\.5,0\.6\\\}across the fullα\\alpharange\.

![Refer to caption](https://arxiv.org/html/2605.12765v1/x10.png)Figure 10:Effect of the steering coefficientα\\alphafor cosine gate thresholdsτg∈\{0\.1,0\.3,0\.4,0\.5,0\.6\}\\tau\_\{g\}\\in\\\{0\.1,0\.3,0\.4,0\.5,0\.6\\\}at layer 8 \(Llama\-3\.2\-1B\-Instruct\)\. Fixed: forget01, token\_positionm​e​a​nmean, seed 0\.Two qualitatively distinct regimes are visible\. Lower thresholds \(τg≤0\.3\\tau\_\{g\}\\leq 0\.3\) cause the gate to fire on a large fraction of inputs, including retain\-distribution queries; utility and*Gibb*degrade asα\\alphaincreases, mirroring the no\-gate collapse in Table[9](https://arxiv.org/html/2605.12765#A10.T9)\. Higher thresholds \(τg≥0\.5\\tau\_\{g\}\\geq 0\.5\) preserve utility and*Gibb*across the full coefficient range, at the cost of slightly reduced memorization suppression on inputs near the boundary\.

The two tested values that bracket the operating threshold,τg=0\.5\\tau\_\{g\}=0\.5andτg=0\.6\\tau\_\{g\}=0\.6, both fall within the stable regime and exhibit similar behavior across allα\\alphavalues\. The defaultτg=0\.55\\tau\_\{g\}=0\.55, which lies between them, is therefore expected to inherit this stability; the separability analysis in the previous subsection provides the additional justification for placing the threshold at this specific point between the two bracketing values\.

### BM25 as an Alternative Gateway

The separability analysis above shows that forget and general\-text inputs are well separated at a coarse, topic level\. This raises the question of whether dense semantic embeddings are necessary at all, or whether a lighter lexical retrieval signal would suffice\. To test this, we replace theSTmodel \(all\-MiniLM\-L6\-v2\) with BM25robertson2009bm25, a sparse term\-matching retriever that operates directly on token overlap without any learned representations\.

Figure[11](https://arxiv.org/html/2605.12765#A10.F11)shows the results\.

![Refer to caption](https://arxiv.org/html/2605.12765v1/x11.png)Figure 11:BM25robertson2009bm25gate: effect of routing thresholdτg∈\{0\.1,0\.5,0\.6,0\.8,1\.0\}\\tau\_\{g\}\\in\\\{0\.1,0\.5,0\.6,0\.8,1\.0\\\}\(Llama\-3\.2\-1B\-Instruct\)\. Fixed: forget01, layer 8, token\_positionm​e​a​nmean, seed 0\.BM25 achieves comparable*Mem*–*Util*trade\-offs to the dense gateway across the full threshold and coefficient range\. This parity is consistent with findings from the retrieval literature\. BM25 is a robust zero\-shot matcher for coarse\-grained topic routing and matches or outperforms dense models at a fraction of the cost when the discrimination task is primarily lexicalthakur2021beir,meng2025qpp\. The result confirms that the gate’s routing decision is driven by topic\-level discrimination rather than fine\-grained semantic similarity, precisely the regime where BM25 is known to be competitive\.

The gate does not impose a hard requirement on dense embeddings, making it viable in resource\-constrained or latency\-sensitive settings\. At the same time, the BM25 parity also implies a ceiling: on datasets with higher retain–forget semantic overlap, where lexical signals alone are insufficient, switching to stronger, domain\-specific embedders could directly improve unlearning precision without changing any other component ofGUARD\-IT\.

## Appendix KContinual Unlearning Results

This appendix reports the complete continual unlearning results for all baselines evaluated on TOFUmaini2024tofuunder the incremental forgetting protocol described in Section[3\.2](https://arxiv.org/html/2605.12765#S3.SS2)\. Table[10](https://arxiv.org/html/2605.12765#A11.T10)reports the final\-round metrics for all methods on both theforget01andforget05splits\.

Table 10:Continual unlearning baselines on TOFU\. Methods are evaluated under the incremental/continual unlearning setting, where the model is unlearned sequentially over multiple subjects\. GDR: gradient difference regularization; KLR: KL\-divergence regularization\. Best result per model/split/column inbold, second bestunderlined\.forget01forget05ModelMethod*Util*↑\\uparrow*Mem*↑\\uparrow*Avg*↑\\uparrow*Gibb*↑\\uparrow*Util*↑\\uparrow*Mem*↑\\uparrow*Avg*↑\\uparrow*Gibb*↑\\uparrowLlama\-3\.2\-1B\-InstructCEUyang2025u0\.410\.440\.420\.880\.000\.240\.000\.04DPOrafailov2023direct0\.530\.250\.340\.920\.020\.340\.040\.93GAyao2024large0\.410\.440\.430\.610\.000\.970\.000\.10GradDiffliu22continual0\.380\.430\.410\.630\.320\.820\.460\.23GradDiff \(KLR\)liu22continual0\.580\.300\.400\.880\.000\.690\.000\.43NPOzhang2024npo0\.510\.400\.450\.800\.020\.370\.030\.85NPO \(KLR\)zhang2024npo0\.510\.400\.450\.840\.030\.420\.050\.87PDUentesari2025constrained0\.070\.440\.120\.060\.000\.140\.000\.19RMUli2024wmdp0\.550\.120\.200\.910\.550\.200\.290\.89RMU \(KLR\)li2024wmdp0\.580\.120\.200\.880\.580\.100\.170\.90SatImpyang2025exploring0\.600\.380\.460\.860\.570\.390\.460\.93SimNPOfan2025simnpo0\.400\.470\.430\.630\.570\.330\.420\.91SimNPO \(KLR\)fan2025simnpo0\.360\.450\.400\.590\.590\.370\.450\.88UNDIALdong\-etal\-2025\-undial0\.540\.470\.500\.900\.450\.620\.520\.87WGAwang2025rethinking0\.570\.430\.490\.750\.560\.430\.490\.89GUARD\-IT0\.600\.680\.640\.800\.600\.630\.620\.81±\\pmSE0\.0000\.0070\.0030\.0010\.0000\.0070\.0030\.002Llama\-3\.2\-3B\-InstructCEUyang2025u0\.570\.440\.500\.430\.000\.270\.000\.03DPOrafailov2023direct0\.400\.300\.340\.970\.030\.340\.050\.64GAyao2024large0\.640\.300\.410\.870\.001\.000\.000\.05GradDiffliu22continual0\.440\.780\.570\.300\.621\.000\.760\.29GradDiff \(KLR\)liu22continual0\.660\.260\.370\.890\.000\.980\.000\.08NPOzhang2024npo0\.590\.370\.450\.870\.340\.460\.390\.90NPO \(KLR\)zhang2024npo0\.580\.370\.450\.900\.310\.460\.370\.87PDUentesari2025constrained0\.010\.290\.020\.030\.000\.040\.000\.05RMUli2024wmdp0\.630\.080\.140\.880\.600\.150\.240\.88RMU \(KLR\)li2024wmdp0\.670\.080\.140\.890\.670\.090\.160\.87SatImpyang2025exploring0\.650\.310\.420\.840\.640\.380\.480\.76SimNPOfan2025simnpo0\.520\.440\.480\.680\.640\.280\.390\.88SimNPO \(KLR\)fan2025simnpo0\.350\.480\.400\.300\.660\.370\.480\.88UNDIALdong\-etal\-2025\-undial0\.650\.450\.530\.840\.510\.640\.570\.83WGAwang2025rethinking0\.630\.430\.510\.650\.610\.380\.470\.88GUARD\-IT0\.670\.660\.660\.840\.660\.570\.610\.83±\\pmSE0\.0000\.0040\.0020\.0090\.0000\.0030\.0020\.010

Gradient\-based methods that perform competitively in the single\-round setting degrade substantially under continual forgetting\. Methods such as GAyao2024largeand GradDiffliu22continualachieve high*Mem*but collapse*Util*to near zero onforget05, reflecting the interference between successive gradient updates noted in the main paper\. RMUli2024wmdpand RMU \(KLR\)li2024wmdppreserve*Util*more consistently across rounds but at the cost of low*Mem*, indicating that their regularization prevents adequate forgetting accumulation\. UNDIALdong\-etal\-2025\-undialand SatImpyang2025exploringoffer a more balanced trade\-off onforget01but struggle to maintain it at the largerforget05scale\.

GUARD\-IT, by contrast, appends newPSVswithout modifying existing ones, so neither*Util*nor*Mem*degrades as the number of forget rounds increases,a desirable property that no gradient\-based baseline exhibitsat both scales simultaneously\.

## Appendix LQuantization Results

This appendix provides the complete per\-model quantization results that underlie the scatter\-plot summary in Figure[1](https://arxiv.org/html/2605.12765#S3.F1)of Section[3\.3](https://arxiv.org/html/2605.12765#S3.SS3)\. Tables[11](https://arxiv.org/html/2605.12765#A12.T11)and[12](https://arxiv.org/html/2605.12765#A12.T12)report*Util*,*Mem*,*Avg*, and*Gibb*for all methods under 4\-bit and 8\-bit quantization on Llama\-3\.2\-1B and Llama\-3\.2\-3B, respectively, across bothforgetsplits\. Figure[12](https://arxiv.org/html/2605.12765#A12.F12)showsGUARD\-IT’s metric profile at 4\-bit and 8\-bit precision side by side, confirming that the method’s trade\-off between memorization suppression and utility is not affected by quantization\.

Table 11:Quantized unlearning baselines on TOFU\. Model: Llama\-3\.2\-1B\-Instruct\. Methods trained under 4\-bit and 8\-bit quantization\.GUARD\-IT: best steering\-vector result \(4\-bit and 8\-bit\), where follows the pattern: token position = mean, 1/4 of the model’s layers, and token position = last, 3/8 of the layers of the model\. Bold: best per column within each quantization level\.Method4\-bit8\-bit*Util*↑\\uparrow*Mem*↑\\uparrow*Avg*↑\\uparrow*Gibb*↑\\uparrow*Util*↑\\uparrow*Mem*↑\\uparrow*Avg*↑\\uparrow*Gibb*↑\\uparrowforget01GradDiffliu22continual0\.530\.310\.390\.910\.580\.190\.280\.90GradDiff \(KLR\)liu22continual0\.540\.320\.400\.900\.590\.200\.300\.91NPOzhang2024npo0\.500\.650\.560\.930\.540\.640\.580\.90NPO \(KLR\)zhang2024npo0\.500\.650\.560\.890\.530\.650\.580\.92RMUli2024wmdp0\.510\.360\.420\.880\.540\.230\.330\.90RMU \(KLR\)li2024wmdp0\.540\.340\.420\.910\.580\.180\.280\.91SimNPOfan2025simnpo0\.520\.370\.440\.910\.580\.290\.390\.91SimNPO \(KLR\)fan2025simnpo0\.540\.390\.450\.890\.590\.310\.410\.91UNDIALdong\-etal\-2025\-undial0\.510\.710\.590\.910\.530\.710\.610\.87GUARD\-IT0\.540\.630\.580\.810\.590\.710\.650\.80forget05GradDiffliu22continual0\.530\.400\.460\.860\.560\.310\.390\.87GradDiff \(KLR\)liu22continual0\.530\.350\.420\.890\.580\.190\.290\.88NPOzhang2024npo0\.450\.690\.540\.900\.480\.690\.570\.89NPO \(KLR\)zhang2024npo0\.340\.720\.460\.850\.360\.710\.480\.86RMUli2024wmdp0\.510\.450\.480\.900\.530\.390\.450\.88RMU \(KLR\)li2024wmdp0\.510\.360\.420\.890\.550\.210\.300\.88SimNPOfan2025simnpo0\.530\.460\.490\.900\.560\.410\.480\.89SimNPO \(KLR\)fan2025simnpo0\.530\.470\.500\.890\.580\.410\.480\.89UNDIALdong\-etal\-2025\-undial0\.540\.710\.610\.840\.560\.710\.640\.81GUARD\-IT0\.540\.690\.600\.830\.590\.630\.610\.82

Table 12:Quantized unlearning baselines on TOFU\. Model: Llama\-3\.2\-3B\-Instruct\. Methods trained under 4\-bit and 8\-bit quantization\.GUARD\-IT: best steering\-vector result \(4\-bit and 8\-bit\), where follows the pattern: token position = mean, 1/4 of the model’s layers\. Bold: best per column within each quantization level\.Method4\-bit8\-bit*Util*↑\\uparrow*Mem*↑\\uparrow*Avg*↑\\uparrow*Gibb*↑\\uparrow*Util*↑\\uparrow*Mem*↑\\uparrow*Avg*↑\\uparrow*Gibb*↑\\uparrowforget01GradDiffliu22continual0\.620\.170\.270\.870\.640\.110\.180\.89GradDiff \(KLR\)liu22continual0\.650\.160\.260\.880\.670\.130\.210\.89NPOzhang2024npo0\.600\.660\.630\.890\.610\.660\.630\.93NPO \(KLR\)zhang2024npo0\.590\.660\.620\.930\.610\.660\.630\.94RMUli2024wmdp0\.600\.250\.350\.880\.620\.160\.260\.89RMU \(KLR\)li2024wmdp0\.660\.170\.270\.920\.670\.100\.170\.87SimNPOfan2025simnpo0\.620\.310\.410\.920\.640\.280\.390\.90SimNPO \(KLR\)fan2025simnpo0\.650\.340\.450\.890\.670\.320\.430\.89UNDIALdong\-etal\-2025\-undial0\.610\.700\.650\.930\.630\.710\.670\.90GUARD\-IT0\.650\.680\.660\.870\.660\.660\.660\.83forget05GradDiffliu22continual0\.600\.290\.390\.890\.620\.240\.340\.88GradDiff \(KLR\)liu22continual0\.640\.230\.340\.880\.670\.190\.290\.86NPOzhang2024npo0\.510\.700\.590\.930\.520\.700\.600\.92NPO \(KLR\)zhang2024npo0\.460\.710\.560\.940\.460\.720\.560\.94RMUli2024wmdp0\.570\.360\.440\.880\.580\.310\.400\.88RMU \(KLR\)li2024wmdp0\.630\.280\.390\.900\.660\.210\.320\.85SimNPOfan2025simnpo0\.600\.390\.480\.910\.620\.380\.470\.90SimNPO \(KLR\)fan2025simnpo0\.630\.420\.500\.890\.660\.410\.510\.88UNDIALdong\-etal\-2025\-undial0\.630\.700\.660\.860\.650\.710\.680\.83GUARD\-IT0\.640\.590\.620\.800\.660\.560\.610\.84

![Refer to caption](https://arxiv.org/html/2605.12765v1/x12.png)Figure 12:Radar plot ofGUARD\-ITunder 4\-bit and 8\-bit quantization on Llama\-3\.2\-1B \(forget01\)\. Each axis corresponds to one metric\. The two polygons nearly coincide, showing thatGUARD\-IT’s metric profile is stable across precision levels and is not materially affected by weight quantization\.On the 1B model,GUARD\-ITis the only method to improve or hold*Mem*when moving from 8\-bit to 4\-bit onforget\-01%, whereas GradDiff and RMU lose 0\.04–0\.12*Avg*points\. Onforget\-05%, the gap widens: UNDIAL is the strongest competitor on*Mem*in both precisions, butGUARD\-ITconsistently leads on*Avg*by preserving*Util*at levels no gradient\-based method sustains\. On the larger 3B model, the same ordering holds, and the quantization sensitivity of gradient\-based methods is more pronounced\. GradDiff drops from*Avg*0\.34 at 8\-bit to 0\.27 at 4\-bit onforget\-01%, whileGUARD\-ITremains at 0\.66 in both configurations\.*Gibb*scores forGUARD\-ITare stable across bit\-widths on both models and both splits, confirming that the norm\-preserving rotation is robust to the numerical precision of the weights that produced the activations it operates on\.