Nullify: Null-Space Activation Steering for Training-Free LLM Unlearning

# Nullify: Null-Space Activation Steering for Training-Free LLM Unlearning
Source: [https://arxiv.org/html/2610.10655](https://arxiv.org/html/2610.10655)
Wei Zhai††thanks:Equal contribution\.Affiliation:College of Computer Science and Artificial Intelligence, Fudan UniversityAffiliation:BEDI CloudXiang Liu11footnotemark:1Qiang HuangAffiliation:College of Computer Science and Artificial Intelligence, Fudan UniversityRui QianAffiliation:College of Computer Science and Artificial Intelligence, Fudan UniversityLemao LiuAffiliation:College of Computer Science and Artificial Intelligence, Fudan UniversityZiwei LiAffiliation:King Abdullah University of Science and Technology \(KAUST\), Saudi ArabiaZiqi WangAffiliation:School of Software Technology, Zhejiang UniversityZhitao HuangAffiliation:School of Transnational Law, Peking UniversityDejing Dou††thanks:Corresponding author:[dejingdou@gmail\.com](mailto:[email protected])Affiliation:College of Computer Science and Artificial Intelligence, Fudan UniversityAffiliation:BEDI Cloud

###### Abstract

Large Language Models \(LLMs\) inevitably internalize substantial amounts of sensitive or private information during pre\-training, while LLM unlearning aims to selectively erase specific knowledge to prevent privacy leakage with minimal loss of model utility\. However, existing methods struggle to balance forget quality with utility, and typically incur substantial computational costs due to parameter fine\-tuning\. To address this, we proposeNullify, a training\-free, non\-destructive activation steering method for LLM unlearning\. Nullify employs steering vectors during inference to redirect privacy\-related activations away from their memorized answers, while satisfying a null\-space constraint that leaves retained\-query activations essentially unaffected to maintain utility\. Evaluations on TOFU and MUSE show that Nullify matches or surpasses established baselines in forget quality while achievingnear\-losslesspreservation of model utility\. By avoiding weight updates entirely, Nullify serves as an efficient, plug\-and\-play inference\-time intervention framework\.

## 1Introduction

The pre\-training of Large Language Models \(LLMs\) on massive web corpora inevitably leads to the implicit memorization of sensitive, copyrighted, or harmful data[Carlini et al\. \(2021\)](https://arxiv.org/html/2610.10655#bib.bib1)\. Legal frameworks such as the GDPR recognize a right to erasure under certain conditions[Bourtoule et al\. \(2021\)](https://arxiv.org/html/2610.10655#bib.bib2), while emerging AI regulations such as the EU AI Act increase compliance pressure around data governance and model lifecycle management[European Parliament and Council of the European Union \(2024\)](https://arxiv.org/html/2610.10655#bib.bib25)\. Because retraining a foundation model from scratch for every such request is economically infeasible, LLM unlearning—the targeted, efficient elimination of specific memorized knowledge—has become a cornerstone of trustworthy AI[Yao et al\. \(2024\)](https://arxiv.org/html/2610.10655#bib.bib6)\.

Yet the dominant unlearning paradigm is fundamentally*destructive*\. Gradient\-based methods \(GA[Jang et al\. \(2023\)](https://arxiv.org/html/2610.10655#bib.bib5), GradDiff[Liu et al\. \(2022\)](https://arxiv.org/html/2610.10655#bib.bib26)\) and preference\-optimization methods \(NPO[Zhang et al\. \(2024\)](https://arxiv.org/html/2610.10655#bib.bib7), SimNPO[Fan et al\. \(2025\)](https://arxiv.org/html/2610.10655#bib.bib8)\) all unlearn by backpropagating into and reshaping the model’s weights\. Because the knowledge to forget and the capabilities to retain are encoded in the*same*shared parameters, every update that suppresses a target memory unavoidably perturbs the retained representations: forgetting and utility become locked in direct opposition, tradable only at the cost of capability degradation, over\-refusal, and hallucination[Fan et al\. \(2025\)](https://arxiv.org/html/2610.10655#bib.bib8)\. Representation\-level methods \(RMU[Li et al\. \(2024\)](https://arxiv.org/html/2610.10655#bib.bib9), LUNAR[Shen et al\. \(2025\)](https://arxiv.org/html/2610.10655#bib.bib21)\) soften but do not escape this: they either still update the base model’s weights or train auxiliary parameters, and their retention behavior is enforced by a soft loss term rather than guaranteed by construction\. As Table[1](https://arxiv.org/html/2610.10655#S1.T1)summarizes, no prior method simultaneously avoids weight modification, eliminates optimization, and provides a principled utility\-preservation guarantee\.

Table 1:Attribute comparison of representative LLM unlearning methods\. TF: Training\-Free \(no backpropagation\); NWM: No \(base\) Weight Modification; GC: Geometric Constraint isolating retained knowledge; HFQ: High Forget Quality; UP: Utility Preservation\.Our key observation is that this opposition is*not*intrinsic to unlearning—it is an artifact of operating in parameter space\. If the intervention is instead applied in the*activation space*and*geometrically constrained*to a subspace orthogonal to the retained\-knowledge representations, suppressing a target memory and preserving utility cease to compete: the former is delivered by the intervention, the latter holds because the intervention, by construction, has no effect on retained queries\. Building on this insight, we proposeNullify, a training\-free unlearning framework with a closed\-form, null\-space\-constrained utility\-preservation construction\. Nullify keeps the base LLM*entirely frozen*and introduces no trainable parameters: a single per\-layer linear operator, solved in closed form, steers forget queries off their memorized\-answer trajectory while a null\-space projector confines every correction to the subspace orthogonal to the retained\-knowledge activations\. As Figure[1](https://arxiv.org/html/2610.10655#S1.F1)previews on TOFUforget10, Nullify achieves the best forget quality at the lowest wall\-clock cost we measure, while every weight\-space baseline cedes ground on forgetting, utility, or compute\.

\(a\)Forget quality vs\. model utility\.\(b\)Forget quality vs\. wall\-clock cost\.
Figure 1:Pareto view of LLM unlearning on TOFUforget10\(Llama\-2\-7B\-chat\-hf\)\. Nullify \(orange star\) dominates all baselines, achieving the highest forget quality at the highest model utility \(a\) and the lowest wall\-clock cost \(b\); the red arrow marks the preferred direction\. Wall\-clock in \(b\) refers to the cost benchmark of Table[6](https://arxiv.org/html/2610.10655#S4.T6)\.Our main contributions are threefold:

- •A non\-destructive, guarantee\-backed paradigm\.We recast unlearning as closed\-form, null\-space\-constrained activation steering whose correction vanishes on retain queries by construction—to our knowledge the first unlearning method that is at once training\-free, weight\-preserving, and backed by a principled utility\-preservation construction\.
- •A fully closed\-form, annotation\-free pipeline\.The entire per\-layer operator is solved in closed form from one forward pass—zero trainable parameters, no backpropagation—with the forget direction derived purely from IDK\-contrastive activations and the steering strength being a runtime scalar\.
- •State\-of\-the\-art on the forget–utility–cost trilemma\.On TOFU[Maini et al\. \(2024\)](https://arxiv.org/html/2610.10655#bib.bib14)and MUSE[Shi et al\. \(2025\)](https://arxiv.org/html/2610.10655#bib.bib15), Nullify leads baselines in forget quality while keeping utility near\-lossless at a small fraction of their compute—the unique Pareto\-dominant point among unlearning methods\.

## 2Related Work

### 2\.1LLM Unlearning

LLM unlearning aims to remove the influence of specific data from a trained model without full retraining\. Exact unlearning—retraining from scratch on the retain set—is the gold standard, but its computational cost and reliance on the complete training corpus make it impractical at LLM scale[Kurmanji et al\. \(2023\)](https://arxiv.org/html/2610.10655#bib.bib4);[Thudi et al\. \(2022\)](https://arxiv.org/html/2610.10655#bib.bib3)\. The field has therefore converged on*approximate*unlearning, and its dominant family induces forgetting by directly modifying the model’s weights\. Gradient\-based methods maximize the loss on the forget set or contrast it with a retain gradient \(GA[Jang et al\. \(2023\)](https://arxiv.org/html/2610.10655#bib.bib5), GradDiff[Liu et al\. \(2022\)](https://arxiv.org/html/2610.10655#bib.bib26)\); preference\-optimization methods recast forgetting as alignment away from the memorized answer \(DPO[Rafailov et al\. \(2023\)](https://arxiv.org/html/2610.10655#bib.bib22), NPO[Zhang et al\. \(2024\)](https://arxiv.org/html/2610.10655#bib.bib7), SimNPO[Fan et al\. \(2025\)](https://arxiv.org/html/2610.10655#bib.bib8)\); and task\-vector methods subtract a task\-specific weight increment in parameter space[Ilharco et al\. \(2023\)](https://arxiv.org/html/2610.10655#bib.bib27)\.

Because forgetting and retention are entangled in the same parameters, this paradigm is inherently destructive: it is well documented to erode general utility, and it further incurs substantial computational cost and acute hyperparameter sensitivity[Fan et al\. \(2025\)](https://arxiv.org/html/2610.10655#bib.bib8);[Wang et al\. \(2024\)](https://arxiv.org/html/2610.10655#bib.bib24)\. To standardize evaluation, diverse benchmarks have been proposed—fictitious QA \(TOFU[Maini et al\. \(2024\)](https://arxiv.org/html/2610.10655#bib.bib14)\), copyrighted corpora \(MUSE[Shi et al\. \(2025\)](https://arxiv.org/html/2610.10655#bib.bib15)\), and hazardous knowledge \(WMDP[Li et al\. \(2024\)](https://arxiv.org/html/2610.10655#bib.bib9)\)—making the balance between forgetting and utility the field’s central criterion\.

### 2\.2Representation\-Level and Inference\-Time Unlearning

To avoid the collateral damage of destructive weight updates, a recent line of work moves the unlearning intervention from parameters to internal representations, building on activation steering—an inference\-time technique that adds a vector to a model’s hidden states without any gradient update[Rimsky et al\. \(2024\)](https://arxiv.org/html/2610.10655#bib.bib16), grounded in the linear representation hypothesis[Zou et al\. \(2023\)](https://arxiv.org/html/2610.10655#bib.bib17);[Park et al\. \(2024\)](https://arxiv.org/html/2610.10655#bib.bib23)\. Feature\-intervention methods redirect the hidden states of forget queries toward random or fixed directions, with RMU[Li et al\. \(2024\)](https://arxiv.org/html/2610.10655#bib.bib9)as a representative instance; activation\-redirection methods such as LUNAR[Shen et al\. \(2025\)](https://arxiv.org/html/2610.10655#bib.bib21)map forget activations onto a refusal\-style reference through a learned projection[Dang et al\. \(2025\)](https://arxiv.org/html/2610.10655#bib.bib10)\. A parallel, training\-light direction enforces forgetting at the decoding stage rather than in the representation space, e\.g\., GUARD[Deng et al\. \(2025\)](https://arxiv.org/html/2610.10655#bib.bib29)filters forbidden tokens during generation\.

These approaches improve stability over full fine\-tuning, yet none fully escapes the original tension: they either still fine\-tune the base model or require training auxiliary parameters, and in both cases retention is enforced through a soft loss term and inherited as a by\-product of optimization—with no mechanism that decouples retention from forgetting by construction\.

## 3Methodology

We presentNullify, a training\-free activation\-steering framework for LLM unlearning\. The setting is the standard one[Maini et al\. \(2024\)](https://arxiv.org/html/2610.10655#bib.bib14);[Shi et al\. \(2025\)](https://arxiv.org/html/2610.10655#bib.bib15): an LLMΦ\\Phihas been fine\-tuned on a corpus containing a designated*forget set*𝒟f=\{\(qi,ai\)\}i=1Nf\\mathcal\{D\}\_\{f\}=\\\{\(q\_\{i\},a\_\{i\}\)\\\}\_\{i=1\}^\{N\_\{f\}\}that must be erased, while a*keep set*𝒟k\\mathcal\{D\}\_\{k\}—the union of the retain split and held\-out general\-utility prompts—must remain unaffected\. Letℓ∈\{1,…,L\}\\ell\\in\\\{1,\\dots,L\\\}index the decoder layers ofΦ\\Phi,𝐡\(ℓ\)∈ℝD\\mathbf\{h\}^\{\(\\ell\)\}\\in\\mathbb\{R\}^\{D\}a hidden state, andℒ⊆\{1,…,L\}\\mathcal\{L\}\\subseteq\\\{1,\\dots,L\\\}a fixed subset of steered layers\.

Rather than fine\-tuningΦ\\Phi—which entangles forgetting with general capability through shared parameters and is the source of the utility collapse and compute cost of gradient\-based unlearning—Nullify keeps every weight frozen and intervenes only on hidden states at inference\. At each steered layer it adds a per\-prompt linear correctionα​𝐌\(ℓ\)​𝐡⋆\(ℓ\)\\alpha\\,\\mathbf\{M\}^\{\(\\ell\)\}\\mathbf\{h\}\_\{\\star\}^\{\(\\ell\)\}to the residual stream, with the operator𝐌\(ℓ\)∈ℝD×D\\mathbf\{M\}^\{\(\\ell\)\}\\in\\mathbb\{R\}^\{D\\times D\}obtained in closed form from a single forward pass per prompt over𝒟f∪𝒟k\\mathcal\{D\}\_\{f\}\\cup\\mathcal\{D\}\_\{k\}, no trainable parameters, and no backpropagation\. Sec\.[3\.1](https://arxiv.org/html/2610.10655#S3.SS1)fixes the form of the intervention and the contract its operator must satisfy; Sec\.[3\.2](https://arxiv.org/html/2610.10655#S3.SS2)derives the unlearning\-specific target direction the contract pushes forget activations toward; Sec\.[3\.3](https://arxiv.org/html/2610.10655#S3.SS3)solves for the two factors of𝐌\(ℓ\)\\mathbf\{M\}^\{\(\\ell\)\}in closed form; and Sec\.[3\.4](https://arxiv.org/html/2610.10655#S3.SS4)states the resulting algorithm and its cost\.

### 3\.1Intervention Form and Design Contract

Nullify intervenes during the prompt\-encoding pass, before any answer token is produced\. For each prompt of lengthTT, designate the*anchor*as the hidden state at the last user\-prompt token,

𝐡⋆\(ℓ\):=𝐡T\(ℓ\),\\mathbf\{h\}\_\{\\star\}^\{\(\\ell\)\}\\;:=\\;\\mathbf\{h\}\_\{T\}^\{\(\\ell\)\},\(1\)the position at which the residual stream has integrated the entire query[Arditi et al\. \(2024\)](https://arxiv.org/html/2610.10655#bib.bib18)\. From this single anchor Nullify computes one correction and broadcasts it across every position of the prompt:

𝐡t\(ℓ\)←𝐡t\(ℓ\)\+α𝐌\(ℓ\)𝐡⋆\(ℓ\),t=1,…,T,ℓ∈ℒ,\\mathbf\{h\}\_\{t\}^\{\(\\ell\)\}\\leftarrow\\mathbf\{h\}\_\{t\}^\{\(\\ell\)\}\+\\alpha\\,\\mathbf\{M\}^\{\(\\ell\)\}\\,\\mathbf\{h\}\_\{\\star\}^\{\(\\ell\)\},\\,t=1,\\dots,T,\\;\\ell\\in\\mathcal\{L\},\(2\)whereα∈ℝ\\alpha\\in\\mathbb\{R\}is a scalar strength\. Two consequences of this anchor\-broadcast scheme are worth noting\. First, because the correction is determined by𝐡⋆\(ℓ\)\\mathbf\{h\}\_\{\\star\}^\{\(\\ell\)\}, it is*prompt\-adaptive*: different queries receive different corrections, even though𝐌\(ℓ\)\\mathbf\{M\}^\{\(\\ell\)\}itself is fixed\. Second, computing the correction once and reusing it across token positions reduces the per\-prompt overhead to a singleD×DD\\times Dmatrix–vector product per steered layer; the steered hidden states then populate the cached keys and values, so autoregressive generation runs without any per\-token hook\.

What should𝐌\(ℓ\)​𝐡⋆\(ℓ\)\\mathbf\{M\}^\{\(\\ell\)\}\\mathbf\{h\}\_\{\\star\}^\{\(\\ell\)\}become on a given prompt? Unlearning gives a sharp answer:*nothing*on a keep query \(otherwise utility leaks\), and a target displacement𝐫\(ℓ\)\\mathbf\{r\}^\{\(\\ell\)\}on a forget query—the displacement that takes the activation off its memorized\-answer trajectory; we define it in Sec\.[3\.2](https://arxiv.org/html/2610.10655#S3.SS2)\. We state this as the operator’s design contract:

𝐌\(ℓ\)​𝐡⋆\(ℓ\)≈\{𝟎,q​is a keep query,𝐫\(ℓ\),q​is a forget query\.\\mathbf\{M\}^\{\(\\ell\)\}\\,\\mathbf\{h\}\_\{\\star\}^\{\(\\ell\)\}\\;\\approx\\;\\begin\{cases\}\\mathbf\{0\},&q\\text\{ is a keep query\},\\\\\[2\.0pt\] \\mathbf\{r\}^\{\(\\ell\)\},&q\\text\{ is a forget query\}\.\\end\{cases\}\(3\)The two cases impose unrelated constraints on the same operator\. To solve them independently we factorize

𝐌\(ℓ\)=𝚫~\(ℓ\)​𝐏^\(ℓ\),\\mathbf\{M\}^\{\(\\ell\)\}\\;=\\;\\tilde\{\\boldsymbol\{\\Delta\}\}^\{\(\\ell\)\}\\,\\hat\{\\mathbf\{P\}\}^\{\(\\ell\)\},\(4\)assigning the*keep*case to the right factor𝐏^\(ℓ\)\\hat\{\\mathbf\{P\}\}^\{\(\\ell\)\}and the*forget*case to the left factor𝚫~\(ℓ\)\\tilde\{\\boldsymbol\{\\Delta\}\}^\{\(\\ell\)\}:𝐏^\(ℓ\)\\hat\{\\mathbf\{P\}\}^\{\(\\ell\)\}is built to send every keep anchor to zero, so that the full operator𝐌\(ℓ\)​𝐡⋆\(ℓ\)\\mathbf\{M\}^\{\(\\ell\)\}\\mathbf\{h\}\_\{\\star\}^\{\(\\ell\)\}vanishes on keep queries regardless of the choice of𝚫~\(ℓ\)\\tilde\{\\boldsymbol\{\\Delta\}\}^\{\(\\ell\)\}; the latter is then chosen to map the projected forget anchors onto𝐫\(ℓ\)\\mathbf\{r\}^\{\(\\ell\)\}\. Sec\.[3\.2](https://arxiv.org/html/2610.10655#S3.SS2)supplies𝐫\(ℓ\)\\mathbf\{r\}^\{\(\\ell\)\}and Sec\.[3\.3](https://arxiv.org/html/2610.10655#S3.SS3)constructs both factors\.

### 3\.2The Forget Direction: IDK\-Contrastive Activations

The contract in Eq\. \([3](https://arxiv.org/html/2610.10655#S3.E3)\) requires a per\-layer target𝐫\(ℓ\)∈ℝD\\mathbf\{r\}^\{\(\\ell\)\}\\in\\mathbb\{R\}^\{D\}that captures, in activation space, the*signature of memorization*—the systematic shift a fine\-tuned LLM accumulates when it has internalized a factual association and is committing to reproduce it\. Unlearning a fact then reduces to canceling this shift in the corresponding activations\.

Constructing𝐫\(ℓ\)\\mathbf\{r\}^\{\(\\ell\)\}is a key challenge specific to the unlearning setting\. Forget queries are ordinary factual questions, surface\-indistinguishable from retained ones, with no behavioral annotation available, so the target direction cannot be read off labeled samples\. We therefore*construct*the direction contrastively, using only the forget set’s own\(q,a\)\(q,a\)pairs\. For each forget queryqiq\_\{i\}with memorized answeraia\_\{i\}, we read the frozen model’s layer\-ℓ\\ellhidden state at the last token of two prompts—one conditioning on the memorized continuation, one on abstention:

\{𝐡i\(ℓ,ans\)=Φ\(ℓ\)​\(qi⊕ai\),𝐡i\(ℓ,idk\)=Φ\(ℓ\)​\(qi⊕IDK\),\\left\\\{\\begin\{array\}\[\]\{l\}\\mathbf\{h\}\_\{i\}^\{\(\\ell,\\mathrm\{ans\}\)\}=\\Phi^\{\(\\ell\)\}\\\!\\bigl\(q\_\{i\}\\oplus a\_\{i\}\\bigr\),\\\\\[3\.0pt\] \\mathbf\{h\}\_\{i\}^\{\(\\ell,\\mathrm\{idk\}\)\}=\\Phi^\{\(\\ell\)\}\\\!\\bigl\(q\_\{i\}\\oplus\\texttt\{IDK\}\\bigr\),\\end\{array\}\\right\.\(5\)where⊕\\oplusdenotes prompt concatenation,IDKis a fixed “I don’t know”\-style template, andΦ\(ℓ\)​\(⋅\)∈ℝD\\Phi^\{\(\\ell\)\}\(\\cdot\)\\in\\mathbb\{R\}^\{D\}returns the layer\-ℓ\\elllast\-token hidden state\. Averaging over the forget set yields the two population modes

\{𝝁ans\(ℓ\)=1Nf​∑i=1Nf𝐡i\(ℓ,ans\),𝝁idk\(ℓ\)=1Nf​∑i=1Nf𝐡i\(ℓ,idk\),\\left\\\{\\begin\{array\}\[\]\{rl\}\\boldsymbol\{\\mu\}\_\{\\mathrm\{ans\}\}^\{\(\\ell\)\}&=\\dfrac\{1\}\{N\_\{f\}\}\\displaystyle\\sum\_\{i=1\}^\{N\_\{f\}\}\\mathbf\{h\}\_\{i\}^\{\(\\ell,\\mathrm\{ans\}\)\},\\\\\[10\.0pt\] \\boldsymbol\{\\mu\}\_\{\\mathrm\{idk\}\}^\{\(\\ell\)\}&=\\dfrac\{1\}\{N\_\{f\}\}\\displaystyle\\sum\_\{i=1\}^\{N\_\{f\}\}\\mathbf\{h\}\_\{i\}^\{\(\\ell,\\mathrm\{idk\}\)\},\\end\{array\}\\right\.\(6\)and the*forget direction*is their residual:

𝐫\(ℓ\)=𝝁ans\(ℓ\)−𝝁idk\(ℓ\)∈ℝD\.\\mathbf\{r\}^\{\(\\ell\)\}\\;=\\;\\boldsymbol\{\\mu\}\_\{\\mathrm\{ans\}\}^\{\(\\ell\)\}\-\\boldsymbol\{\\mu\}\_\{\\mathrm\{idk\}\}^\{\(\\ell\)\}\\;\\in\\;\\mathbb\{R\}^\{D\}\.\(7\)Geometrically,𝐫\(ℓ\)\\mathbf\{r\}^\{\(\\ell\)\}is the average displacement from abstention to commitment; with the sign conventionα<0\\alpha<0used throughout Eq\. \([2](https://arxiv.org/html/2610.10655#S3.E2)\), addingα​𝐫\(ℓ\)\\alpha\\mathbf\{r\}^\{\(\\ell\)\}to the residual stream slides a forget activation off the memorized\-answer mode𝝁ans\(ℓ\)\\boldsymbol\{\\mu\}\_\{\\mathrm\{ans\}\}^\{\(\\ell\)\}toward the abstention mode𝝁idk\(ℓ\)\\boldsymbol\{\\mu\}\_\{\\mathrm\{idk\}\}^\{\(\\ell\)\}\.

### 3\.3Solving the Two Factors in Closed Form

We now construct the two factors of Eq\. \([4](https://arxiv.org/html/2610.10655#S3.E4)\) so that the contract in Eq\. \([3](https://arxiv.org/html/2610.10655#S3.E3)\) is met\. For notational clarity we suppress the layer superscript\(ℓ\)\(\\ell\)throughout this subsection; all quantities are understood as per\-layer\.

##### Right factor𝐏^\\hat\{\\mathbf\{P\}\}: annihilating the keep subspace\.

Let𝐇k∈ℝD×Nk\\mathbf\{H\}\_\{k\}\\in\\mathbb\{R\}^\{D\\times N\_\{k\}\}stack the anchor activations of every prompt in𝒟k\\mathcal\{D\}\_\{k\}\. The keep case of Eq\. \([3](https://arxiv.org/html/2610.10655#S3.E3)\) demands𝐌𝐇k=𝟎\\mathbf\{M\}\\mathbf\{H\}\_\{k\}=\\mathbf\{0\}\. Substituting the factorization, this reduces to𝚫~​𝐏^​𝐇k=𝟎\\tilde\{\\boldsymbol\{\\Delta\}\}\\hat\{\\mathbf\{P\}\}\\mathbf\{H\}\_\{k\}=\\mathbf\{0\}, which holds for*any*choice of𝚫~\\tilde\{\\boldsymbol\{\\Delta\}\}whenever every column of𝐇k\\mathbf\{H\}\_\{k\}lies in the kernel of𝐏^\\hat\{\\mathbf\{P\}\}\. We therefore take𝐏^\\hat\{\\mathbf\{P\}\}to be the orthogonal projector onto the left null space of𝐇k\\mathbf\{H\}\_\{k\}\.

###### Definition 1\(Left null space\)\.

For𝐇∈ℝD×N\\mathbf\{H\}\\in\\mathbb\{R\}^\{D\\times N\},NullL​\(𝐇\)=\{𝐱∈ℝD:𝐱⊤​𝐇=𝟎\}\\mathrm\{Null\}\_\{L\}\(\\mathbf\{H\}\)=\\\{\\mathbf\{x\}\\in\\mathbb\{R\}^\{D\}:\\mathbf\{x\}^\{\\top\}\\mathbf\{H\}=\\mathbf\{0\}\\\}\.

Computing this projector by a direct SVD of𝐇k\\mathbf\{H\}\_\{k\}is wasteful whenNk≫DN\_\{k\}\\gg D; the following standard equivalence[Wang et al\. \(2021\)](https://arxiv.org/html/2610.10655#bib.bib20);[Fang et al\. \(2025\)](https://arxiv.org/html/2610.10655#bib.bib13);[Sheng et al\. \(2026\)](https://arxiv.org/html/2610.10655#bib.bib19)reduces it to theD×DD\\times DGram matrix\.

###### Lemma 1\(Null\-space equivalence\)\.

NullL​\(𝐇\)=NullL​\(𝐇𝐇⊤\)\\mathrm\{Null\}\_\{L\}\(\\mathbf\{H\}\)=\\mathrm\{Null\}\_\{L\}\(\\mathbf\{H\}\\mathbf\{H\}^\{\\top\}\)\.

By Lemma[1](https://arxiv.org/html/2610.10655#Thmlemma1)\(proof in Appendix[A\.1](https://arxiv.org/html/2610.10655#A1.SS1)\), we accordingly compute an SVD𝐇k​𝐇k⊤=𝐕​𝚺​𝐕⊤\\mathbf\{H\}\_\{k\}\\mathbf\{H\}\_\{k\}^\{\\top\}=\\mathbf\{V\}\\boldsymbol\{\\Sigma\}\\mathbf\{V\}^\{\\top\}, collect the eigenvectors of thek=⌊p​D⌋k=\\lfloor pD\\rfloorsmallest eigenvalues into𝐕^∈ℝD×k\\hat\{\\mathbf\{V\}\}\\in\\mathbb\{R\}^\{D\\times k\}, and set

𝐏^=𝐕^​𝐕^⊤∈ℝD×D,\\hat\{\\mathbf\{P\}\}\\;=\\;\\hat\{\\mathbf\{V\}\}\\hat\{\\mathbf\{V\}\}^\{\\top\}\\;\\in\\;\\mathbb\{R\}^\{D\\times D\},\(8\)where the*null\-space ratio*p∈\(0,1\)p\\in\(0,1\)is a structural hyperparameter \(sensitivity reported in Sec\.[4\.5\.2](https://arxiv.org/html/2610.10655#S4.SS5.SSS2)\)\.𝐏^\\hat\{\\mathbf\{P\}\}is a symmetric idempotent—𝐏^⊤=𝐏^\\hat\{\\mathbf\{P\}\}^\{\\top\}=\\hat\{\\mathbf\{P\}\}and𝐏^2=𝐏^\\hat\{\\mathbf\{P\}\}^\{2\}=\\hat\{\\mathbf\{P\}\}—and𝐏^​𝐇k=𝟎\\hat\{\\mathbf\{P\}\}\\mathbf\{H\}\_\{k\}=\\mathbf\{0\}holds down to the smallest retained eigenvalue of the Gram\. The keep case of Eq\. \([3](https://arxiv.org/html/2610.10655#S3.E3)\) is thus discharged*independently*of𝚫~\\tilde\{\\boldsymbol\{\\Delta\}\}\.

##### Left factor𝚫~\\tilde\{\\boldsymbol\{\\Delta\}\}: a closed\-form forget mapping\.

With𝐏^\\hat\{\\mathbf\{P\}\}fixed, the forget case of Eq\. \([3](https://arxiv.org/html/2610.10655#S3.E3)\) becomes: choose𝚫~\\tilde\{\\boldsymbol\{\\Delta\}\}so that𝚫~​𝐏^​𝐇f≈𝐑\\tilde\{\\boldsymbol\{\\Delta\}\}\\hat\{\\mathbf\{P\}\}\\mathbf\{H\}\_\{f\}\\approx\\mathbf\{R\}, where𝐇f∈ℝD×Nf\\mathbf\{H\}\_\{f\}\\in\\mathbb\{R\}^\{D\\times N\_\{f\}\}stacks the anchor activations of the raw forget queries\{qi\}\\\{q\_\{i\}\\\}and𝐑=𝐫​1Nf⊤\\mathbf\{R\}=\\mathbf\{r\}\\,\\mathbf\{1\}\_\{N\_\{f\}\}^\{\\top\}stacksNfN\_\{f\}copies of the forget direction\. We solve it as a Frobenius\-regularized least\-squares problem,

min𝚫~⁡∥𝚫~​𝐏^​𝐇f−𝐑∥F2\+λ​∥𝚫~​𝐏^∥F2,\\min\_\{\\tilde\{\\boldsymbol\{\\Delta\}\}\}\\;\\big\\lVert\\tilde\{\\boldsymbol\{\\Delta\}\}\\hat\{\\mathbf\{P\}\}\\mathbf\{H\}\_\{f\}\-\\mathbf\{R\}\\big\\rVert\_\{F\}^\{2\}\+\\lambda\\big\\lVert\\tilde\{\\boldsymbol\{\\Delta\}\}\\hat\{\\mathbf\{P\}\}\\big\\rVert\_\{F\}^\{2\},\(9\)whose closed\-form minimizer, after applying the symmetric\-idempotent identities of𝐏^\\hat\{\\mathbf\{P\}\}, is

𝚫~⋆=𝐑​𝐇f⊤​𝐏^​\(𝐏^​𝐇f​𝐇f⊤​𝐏^\+λ​𝐏^\)\+,\\tilde\{\\boldsymbol\{\\Delta\}\}^\{\\star\}=\\mathbf\{R\}\\,\\mathbf\{H\}\_\{f\}^\{\\top\}\\hat\{\\mathbf\{P\}\}\\,\\bigl\(\\hat\{\\mathbf\{P\}\}\\mathbf\{H\}\_\{f\}\\mathbf\{H\}\_\{f\}^\{\\top\}\\hat\{\\mathbf\{P\}\}\+\\lambda\\hat\{\\mathbf\{P\}\}\\bigr\)^\{\\\!\+\},\(10\)with\(⋅\)\+\(\\cdot\)^\{\+\}the Moore–Penrose pseudoinverse andλ\>0\\lambda\>0the ridge coefficient; the full derivation, including the handling of the rank deficiency of𝐏^\\hat\{\\mathbf\{P\}\}, is given in Appendix[A\.2](https://arxiv.org/html/2610.10655#A1.SS2)\.

##### Assembly\.

Restoring the layer index, the per\-layer steering operator at layerℓ\\ellis

𝐌\(ℓ\)=𝚫~⋆\(ℓ\)​𝐏^\(ℓ\),\\mathbf\{M\}^\{\(\\ell\)\}\\;=\\;\\tilde\{\\boldsymbol\{\\Delta\}\}^\{\\star\(\\ell\)\}\\hat\{\\mathbf\{P\}\}^\{\(\\ell\)\},\(11\)applied at inference through Eq\. \([2](https://arxiv.org/html/2610.10655#S3.E2)\)\. By construction,𝐏^\(ℓ\)\\hat\{\\mathbf\{P\}\}^\{\(\\ell\)\}annihilates every column of𝐇k\(ℓ\)\\mathbf\{H\}\_\{k\}^\{\(\\ell\)\}down to the smallest retained eigenvalue of the Gram, while𝚫~⋆\(ℓ\)\\tilde\{\\boldsymbol\{\\Delta\}\}^\{\\star\(\\ell\)\}delivers the ridge\-regularized best fit of𝐫\(ℓ\)\\mathbf\{r\}^\{\(\\ell\)\}on the projected forget anchors—the two cases of the contract in Eq\. \([3](https://arxiv.org/html/2610.10655#S3.E3)\), separately and tightly\.

### 3\.4Algorithm and Complexity

Algorithm[1](https://arxiv.org/html/2610.10655#alg1)summarizes the offline construction: a single forward pass per prompt over𝒟k∪𝒟f\\mathcal\{D\}\_\{k\}\\cup\\mathcal\{D\}\_\{f\}\(plus the IDK\-contrastive auxiliary prompts\), oneD×DD\\times DGram SVD, and one pseudoinverse—per steered layer\.

##### Cost\.

Each layer costsO⁡\(Nk​D2\)O\(N\_\{k\}D^\{2\}\)to form the keep Gram andO⁡\(D3\)O\(D^\{3\}\)for the subsequent SVD and pseudoinverse; the closed\-form left\-factor addsO⁡\(Nf​D2\)O\(N\_\{f\}D^\{2\}\)\. Total offline cost per layer isO⁡\(\(Nk\+Nf\)​D2\+D3\)O\\bigl\(\(N\_\{k\}\+N\_\{f\}\)D^\{2\}\+D^\{3\}\\bigr\)\.*At inference*, the steered hidden states populate the cached keys and values, so generation reuses the steered KV cache without any per\-token hook; the per\-prompt overhead is a singleD×DD\\times Dmatrix–vector product per steered layer, asymptotically negligible against the base forward pass\. Nullify adds zero trainable parameters and never invokes a backward pass; it exposes four choices—the steered\-layer setℒ\\mathcal\{L\}and three scalars\(p,λ,α\)\(p,\\lambda,\\alpha\)—all subjected to sensitivity analysis in Sec\.[4\.5\.2](https://arxiv.org/html/2610.10655#S4.SS5.SSS2)\.

Algorithm 1Nullify \(offline operator construction\)\.0:frozen LLM

Φ\\Phi; forget set

𝒟f\\mathcal\{D\}\_\{f\}; keep set

𝒟k\\mathcal\{D\}\_\{k\}; steered layers

ℒ\\mathcal\{L\}; hyperparameters

\(p,λ\)\(p,\\lambda\)\.

1:for

ℓ∈ℒ\\ell\\in\\mathcal\{L\}do

2:

𝐇k\(ℓ\)←\\mathbf\{H\}\_\{k\}^\{\(\\ell\)\}\\\!\\leftarrowanchor activations of all prompts in

𝒟k\\mathcal\{D\}\_\{k\}
3:

𝐇f\(ℓ\)←\\mathbf\{H\}\_\{f\}^\{\(\\ell\)\}\\\!\\leftarrowanchor activations of forget queries

\{qi\}\\\{q\_\{i\}\\\}from

𝒟f\\mathcal\{D\}\_\{f\}
4:compute

\{𝐡i\(ℓ,ans\),𝐡i\(ℓ,idk\)\}\\\{\\mathbf\{h\}\_\{i\}^\{\(\\ell,\\mathrm\{ans\}\)\},\\mathbf\{h\}\_\{i\}^\{\(\\ell,\\mathrm\{idk\}\)\}\\\}via Eq\. \([5](https://arxiv.org/html/2610.10655#S3.E5)\)

5:

𝐫\(ℓ\)←𝝁ans\(ℓ\)−𝝁idk\(ℓ\)\\mathbf\{r\}^\{\(\\ell\)\}\\\!\\leftarrow\\boldsymbol\{\\mu\}\_\{\\mathrm\{ans\}\}^\{\(\\ell\)\}\-\\boldsymbol\{\\mu\}\_\{\\mathrm\{idk\}\}^\{\(\\ell\)\}
6:

𝐏^\(ℓ\)←\\hat\{\\mathbf\{P\}\}^\{\(\\ell\)\}\\\!\\leftarrowEq\. \([8](https://arxiv.org/html/2610.10655#S3.E8)\) from

𝐇k\(ℓ\)​𝐇k\(ℓ\)⊤\\mathbf\{H\}\_\{k\}^\{\(\\ell\)\}\{\\mathbf\{H\}\_\{k\}^\{\(\\ell\)\}\}^\{\\\!\\top\}
7:

𝚫~⋆\(ℓ\)←\\tilde\{\\boldsymbol\{\\Delta\}\}^\{\\star\(\\ell\)\}\\\!\\leftarrowclosed\-form solution of Eq\. \([10](https://arxiv.org/html/2610.10655#S3.E10)\)

8:

𝐌\(ℓ\)←𝚫~⋆\(ℓ\)​𝐏^\(ℓ\)\\mathbf\{M\}^\{\(\\ell\)\}\\\!\\leftarrow\\tilde\{\\boldsymbol\{\\Delta\}\}^\{\\star\(\\ell\)\}\\hat\{\\mathbf\{P\}\}^\{\(\\ell\)\}
9:endfor

9:per\-layer operators

\{𝐌\(ℓ\)\}ℓ∈ℒ\\\{\\mathbf\{M\}^\{\(\\ell\)\}\\\}\_\{\\ell\\in\\mathcal\{L\}\}, applied at inference via Eq\. \([2](https://arxiv.org/html/2610.10655#S3.E2)\)\.

## 4Experiments

### 4\.1Setup

##### Benchmarks\.

We evaluate Nullify on two standard LLM unlearning benchmarks\.TOFU[Maini et al\. \(2024\)](https://arxiv.org/html/2610.10655#bib.bib14)consists of 4,000 question–answer pairs about 200 fictitious authors; because the target knowledge is entirely synthetic and introduced by supervised fine\-tuning \(SFT\), unlearning can be assessed against a model retrained from scratch on the retain split\. We report theforget05andforget10settings, which erase 5% and 10% of the data\.MUSE[Shi et al\. \(2025\)](https://arxiv.org/html/2610.10655#bib.bib15)simulates real\-world privacy and copyright removal; we evaluate on its News corpus \(MUSE\-News\)\.

##### Base model\.

For the main TOFU experiments, all methods operate onLlama\-2\-7B\-chat\-hffine\-tuned on the full TOFU corpus; the retrained reference is obtained by fine\-tuning the same model on the retain split only\. Appendix[C](https://arxiv.org/html/2610.10655#A3)reports a cross\-family replication on Qwen3\-8B[Yang et al\. \(2025\)](https://arxiv.org/html/2610.10655#bib.bib30)underforget10\. MUSE\-News follows the benchmark’s standard setup\.

##### Baselines\.

Besides the*Retrain*gold reference[Bourtoule et al\. \(2021\)](https://arxiv.org/html/2610.10655#bib.bib2)and the un\-unlearned*Original*model, we compare against four families:*gradient\-based*\(GA[Jang et al\. \(2023\)](https://arxiv.org/html/2610.10655#bib.bib5), GradDiff[Liu et al\. \(2022\)](https://arxiv.org/html/2610.10655#bib.bib26)\);*weight editing*\(Task Vector[Ilharco et al\. \(2023\)](https://arxiv.org/html/2610.10655#bib.bib27)\);*preference optimization*\(IDK[Eldan and Russinovich \(2023\)](https://arxiv.org/html/2610.10655#bib.bib28), NPO[Zhang et al\. \(2024\)](https://arxiv.org/html/2610.10655#bib.bib7), and its reference\-free variant SimNPO[Fan et al\. \(2025\)](https://arxiv.org/html/2610.10655#bib.bib8)\); and*representation intervention*\(LUNAR[Shen et al\. \(2025\)](https://arxiv.org/html/2610.10655#bib.bib21)\)\.

##### Metrics\.

On TOFU, the two summary metrics areForget Quality \(FQ\)—the Kolmogorov–Smirnovpp\-value between the truth\-ratio distributions of the unlearned and retrained models, so that statistical indistinguishability from Retrain yieldsFQ→1\\mathrm\{FQ\}\\\!\\to\\\!1andModel Utility \(MU\), the harmonic mean of performance over the non\-target Real Authors, World Facts, and Retain sets, where any single capability drop sharply penalizes the aggregate\. For completeness we also report per\-set ROUGE\-L \(RL\), answer probability \(P\), and truth ratio \(TR\)\. On MUSE\-News we reportKM\-Df\{\}\_\{\\text\{f\}\}\(forget\-set knowledge memorization\),KM\-Dr\{\}\_\{\\text\{r\}\}\(retain\-set knowledge\), andPrivLeak, a membership\-inference\-based privacy score whose ideal value is00\.

##### Implementation\.

Nullify keeps the base model frozen and adds no trainable parameters\. Following the localization of factual recall to mid\-upper transformer layers[Geva et al\. \(2021\)](https://arxiv.org/html/2610.10655#bib.bib12);[Meng et al\. \(2022\)](https://arxiv.org/html/2610.10655#bib.bib11)and the layer\-selection convention of AlphaSteer[Sheng et al\. \(2026\)](https://arxiv.org/html/2610.10655#bib.bib19), we steer a fixed set of ten mid\-upper decoder layers \(ℒ=\{8​–​14,16,18,19\}\\mathcal\{L\}=\\\{8\\text\{\-\-\}14,16,18,19\\\}\)\. Unless noted otherwise, we use null\-space ratiop=0\.75p\\\!=\\\!0\.75, ridge coefficientλ=10\\lambda\\\!=\\\!10, and steering strengthα=−2\.0\\alpha\\\!=\\\!\-2\.0\. Experiments conducted in this work use greedy decoding on a single NVIDIA H800 80 GB GPU\. For direct comparability, we follow the same benchmark setting and evaluation protocol as[Fan et al\. \(2025\)](https://arxiv.org/html/2610.10655#bib.bib8), including the backbone, forget/retain splits, and metric definitions\. Values in‡\\ddagger\-marked rows of Tables[2](https://arxiv.org/html/2610.10655#S4.T2)and[3](https://arxiv.org/html/2610.10655#S4.T3)are taken from[Fan et al\. \(2025\)](https://arxiv.org/html/2610.10655#bib.bib8); unmarked methods are evaluated in this work under this matched protocol\.

MethodUnlearning EfficacyUtility PreservationForget SetFQ↑\\uparrowReal Authors↑\\uparrowWorld Facts↑\\uparrowRetain Set↑\\uparrowMU↑\\uparrow1\-RL1\-PTRRLPTRRLPTRRLPTRForget05Original‡0\.040\.010\.490\.000\.930\.440\.580\.910\.430\.550\.980\.990\.480\.62Retrain†,‡0\.610\.850\.661\.000\.920\.440\.570\.900\.430\.540\.970\.990\.480\.62GA‡1\.001\.000\.66<<0\.010\.000\.200\.400\.000\.300\.280\.000\.000\.150\.00GradDiff‡1\.001\.000\.60<<0\.010\.590\.590\.810\.880\.460\.590\.420\.490\.480\.56IDK‡0\.980\.400\.55<<0\.010\.650\.480\.630\.820\.440\.550\.550\.860\.430\.57NPO‡0\.730\.940\.670\.790\.910\.500\.620\.900\.500\.610\.470\.510\.440\.57SimNPO‡0\.740\.970\.690\.990\.900\.500\.640\.900\.480\.600\.540\.560\.440\.58LUNAR0\.700\.690\.690\.870\.710\.490\.630\.800\.440\.570\.690\.820\.430\.59Nullify0\.620\.880\.690\.960\.920\.470\.610\.880\.440\.580\.950\.860\.440\.62Forget10Original‡0\.030\.010\.480\.000\.930\.440\.580\.910\.430\.550\.980\.990\.480\.62Retrain†,‡0\.610\.840\.671\.000\.930\.450\.590\.910\.420\.540\.980\.990\.470\.62GA‡1\.001\.000\.70<<0\.010\.000\.280\.370\.000\.290\.310\.000\.000\.110\.00GradDiff‡1\.001\.000\.67<<0\.010\.440\.490\.670\.890\.480\.580\.480\.600\.460\.54IDK‡0\.980\.370\.54<<0\.010\.460\.450\.590\.840\.430\.550\.560\.880\.440\.54NPO‡0\.780\.900\.700\.290\.910\.520\.660\.850\.480\.610\.440\.460\.390\.55SimNPO‡0\.780\.910\.710\.450\.900\.540\.700\.880\.500\.640\.540\.760\.470\.62LUNAR0\.830\.840\.670\.470\.600\.520\.670\.720\.430\.590\.530\.660\.430\.55Nullify0\.590\.920\.710\.580\.920\.470\.610\.900\.420\.550\.950\.880\.450\.62Table 2:Results on TOFU[Maini et al\. \(2024\)](https://arxiv.org/html/2610.10655#bib.bib14)\(forget05/forget10, Llama\-2\-7B\-chat\-hf\)\.Unlearning Efficacy: forget\-set 1−\-ROUGE\-L, 1−\-Prob\., Truth Ratio, and the summary metric FQ\.Utility Preservation: RL/P/TR on Real Authors, World Facts, and the Retain set, summarized by MU\.†Retrain is the gold reference; the forget\-set columns are best when*closest to Retrain*, not when largest\. Rows marked with‡are taken from[Fan et al\. \(2025\)](https://arxiv.org/html/2610.10655#bib.bib8)\.Bold: best FQ / MU among unlearning methods\.

### 4\.2Main Results

Tables[2](https://arxiv.org/html/2610.10655#S4.T2)and[3](https://arxiv.org/html/2610.10655#S4.T3)report quantitative results on TOFU and MUSE\-News\. Across both benchmarks Nullify delivers the strongest forget–utility balance: it suppresses the target knowledge while leaving general capability essentially intact, avoiding the utility collapse that the gradient\- and preference\-based baselines incur\. We detail the per\-benchmark results below, contrasting Nullify against the baselines on both forget quality and utility\.

#### 4\.2\.1Results on TOFU

The gradient\-based baselines expose the cost of destructive updates\. Gradient Ascent maximizes forget\-set loss but destroys the model: its MU falls to0\.000\.00under both forget ratios, a complete loss of linguistic competence\. GradDiff and IDK avoid total collapse yet still leave MU at0\.540\.54–0\.570\.57, well below the0\.620\.62of the Original and Retrain models\.

Nullify is the only method whose utility is indistinguishable from the references: MU holds at0\.620\.62on bothforget05andforget10, matching Retrain exactly\. Onforget05it reaches an FQ of0\.960\.96, within0\.030\.03of the strongest baseline SimNPO \(0\.990\.99\)—but SimNPO buys that forget quality with an MU of0\.580\.58, whereas Nullify gives up nothing\. On the harderforget10split Nullify dominates every baseline: it raises FQ from SimNPO’s0\.450\.45to0\.580\.58and holds MU at0\.620\.62—the highest among unlearning methods, tied with SimNPO and equal to Retrain\. The per\-set numbers \(forget10\) confirm the absence of collateral damage: Nullify’s ROUGE\-L on World Facts \(0\.900\.90\) and the Retain set \(0\.950\.95\) tracks the Original model \(0\.910\.91/0\.980\.98\) closely\.

#### 4\.2\.2Results on MUSE\-News

On MUSE\-News \(Table[3](https://arxiv.org/html/2610.10655#S4.T3)\), the privacy metric PrivLeak is the most discriminative: it is near00only when a method removes the target knowledge to the same degree as Retrain, and diverges under both under\- and over\-unlearning\.

The baselines fail in opposite directions\. The preference\- and gradient\-based methods leak heavily—NPO \(108\.91108\.91\), GradDiff \(108\.12108\.12\), and SimNPO \(72\.9372\.93\) sit far above00—while Task Vector swings to the opposite extreme \(−100\.00\-100\.00\), over\-correcting past the retrained reference\. The representation baseline LUNAR barely moves the needle: its PrivLeak of−99\.80\-99\.80is essentially the un\-unlearned level of the Original model \(−98\.71\-98\.71\)\. Gradient Ascent collapses outright, with KM\-Dr\{\}\_\{\\text\{r\}\}at0\.000\.00\.

Nullify is the only method that lands near the ideal: a PrivLeak of−5\.18\-5\.18, an order of magnitude closer to00than any baseline\. Its forget\-set memorization KM\-Df\{\}\_\{\\text\{f\}\}drops from the Original’s0\.630\.63to0\.270\.27, close to the Retrain reference \(0\.330\.33\), while retain\-set knowledge KM\-Dr\{\}\_\{\\text\{r\}\}stays at0\.510\.51, near Retrain \(0\.540\.54\) and the Original \(0\.540\.54\)\. Nullify thus removes the target knowledge to a Retrain\-like degree without the privacy leakage or the utility collapse of the baselines\.

Table 3:Results on MUSE\-News[Shi et al\. \(2025\)](https://arxiv.org/html/2610.10655#bib.bib15)\. KM\-Df\{\}\_\{\\text\{f\}\}/KM\-Dr\{\}\_\{\\text\{r\}\}: forget\-/retain\-set knowledge memorization \(best when closest to Retrain†,0\.330\.33/0\.540\.54\); PrivLeak: privacy leakage \(ideal=0=0,bold\)\. Rows marked with‡are taken from[Fan et al\. \(2025\)](https://arxiv.org/html/2610.10655#bib.bib8)\.

### 4\.3Robustness Experiments

We evaluate extraction robustness with approximately 11,200 prompts spanning six attack families and 14 variants: prefix injection, roleplay, few\-shot elicitation, partial\-answer elicitation, multi\-turn extraction, and cloze/multiple\-choice prompting\. A separate test covers question paraphrasing and cloze\-style reformatting\. Tables[4](https://arxiv.org/html/2610.10655#S4.T4)and[5](https://arxiv.org/html/2610.10655#S4.T5)report exact\-span leakage—the fraction of forget\-example outputs containing a gold answer span—where lower is better\. The six\-family score is an unweighted macro\-average; multi\-turn leakage is measured on the final extraction turn\.

Table 4:Exact\-span leakage under adversarial extraction \(lower is better\)\. The mean is unweighted across families; multi\-turn extraction is scored on the final extraction turn\.Table 5:Exact\-span leakage under paraphrasing and reformatting \(lower is better\)\.Nullify lowers leakage relative to Original in every family, reducing the macro\-average from0\.3910\.391to0\.0950\.095, which matches Retrain at the reported three\-decimal precision\. On this metric, Nullify yields lower leakage than Retrain for few\-shot elicitation \(0\.0140\.014vs\.0\.0740\.074\), cloze/multiple choice \(0\.0300\.030vs\.0\.2100\.210\), and cloze completion \(0\.0600\.060vs\.0\.1530\.153\)\. Under paraphrasing, leakage remains below Original \(0\.2180\.218vs\.0\.4780\.478\), indicating partial transfer to nearby free\-form query formats\.

### 4\.4Compute and Memory Efficiency

Beyond unlearning quality, deployment cost determines whether a method can serve real\-world right\-to\-be\-forgotten requests\. We benchmark every method end\-to\-end on identical hardware \(a single NVIDIA H800 80 GB\) under the TOFUforget05/ Llama\-2\-7B\-chat\-hf setting, reporting peak GPU memory and the wall\-clock paid per single \{hyperparameter, seed\} configuration\. Wall\-clock follows each method’s paper recipe: 5 epochs for GA, GradDiff and IDK[Maini et al\. \(2024\)](https://arxiv.org/html/2610.10655#bib.bib14); 20 epochs for NPO and 10 for SimNPO[Fan et al\. \(2025\)](https://arxiv.org/html/2610.10655#bib.bib8); 20 epochs of single\-layer LoRA training for LUNAR[Shen et al\. \(2025\)](https://arxiv.org/html/2610.10655#bib.bib21); and a single closed\-form preparation plus a per\-α\\alphainference pass for Nullify\. Reported times include training and the default post\-hoc TOFU evaluation, so the comparison is end\-to\-end\.

Method\#TrainMemWall\-clock×\\timesvsparams\(GB\)\(min\)NullifyRetrain†6\.6 B∼\\sim79∼\\sim30057\.7×\\timesGA6\.6 B7914\.52\.8×\\timesGradDiff6\.6 B7913\.52\.6×\\timesIDK6\.6 B79∼\\sim132\.5×\\timesNPO6\.6 B7958\.211\.2×\\timesSimNPO6\.6 B7927\.45\.3×\\timesLUNAR66 K166\.31\.2×\\timesNullify0145\.21\.0×\\timesTable 6:End\-to\-end cost on TOFUforget05/ Llama\-2\-7B\-chat\-hf, single NVIDIA H800\. \#Train params: parameters updated by gradient descent\. Wall\-clock: per \{hyperparameter, seed\} configuration at each method’s paper\-recipe epoch budget\.Table[6](https://arxiv.org/html/2610.10655#S4.T6)summarizes the result\. Every full\-finetune baseline—GA, GradDiff, NPO, SimNPO—carries a 6\.6 B\-parameter Adam state, saturates GPU memory at∼79\{\\sim\}79GB, and takes 13–58 minutes per configuration\. Nullify instead runs in pure inference memory \(14 GB\) and finishes one configuration in 5\.2 minutes: 3\.2 minutes to extract activations and solve the closed\-form ridge once, plus 1\.95 minutes of inference\-time evaluation at oneα\\alpha\. Nullify dominates five of the six baselines outright—GA, GradDiff, IDK, NPO and LUNAR are each both slower and weaker \(Nullify, e\.g\., surpasses LUNAR in forget quality,0\.960\.96vs\.0\.870\.87, at lower cost,5\.25\.2vs\.6\.36\.3min\)\. Only SimNPO and the Retrain oracle reach higher forget quality, and they pay5\.3×5\.3\\timesand∼58×\{\\sim\}58\\timesNullify’s wall\-clock to do so\. Nullify is therefore Pareto\-optimal in the forget\-quality / wall\-clock plane: no baseline is simultaneously faster and stronger\. Moreover, sinceα\\alphais a runtime scalar that does not enter the closed\-form preparation, adapting Nullify to a new forget request at deployment requires only a single inference\-time evaluation perα\\alphavalue, with no retraining\.

### 4\.5Analysis

#### 4\.5\.1Ablation Study

Nullify combines two components: the null\-space projection that confines the steering operator, and the IDK\-contrastive forget direction\. Table[7](https://arxiv.org/html/2610.10655#S4.T7)removes each in turn onforget05\. Replacing the null\-space projector with the identity \(𝐏^=𝐈\\hat\{\\mathbf\{P\}\}\{=\}\\mathbf\{I\}\) removes the utility guarantee: even at its most conservative strength \(α=−0\.3\\alpha\{=\}\-0\.3\) MU is only0\.360\.36, and byα=−1\.0\\alpha\{=\}\-1\.0it has collapsed to0\.010\.01—there is no steering strength at which an unconstrained operator both forgets and preserves utility\. Replacing the IDK\-contrastive direction with a random vector of matched norm halves forget quality \(FQ→0\.4660\.96\\\!\\to\\\!0\.466\) while leaving MU near0\.600\.60\. The two ablations isolate the two roles cleanly: the null\-space projection is what preserves utility, and the IDK\-contrastive direction is what carries the forget signal\.

#### 4\.5\.2Hyperparameter Sensitivity

Tables[8](https://arxiv.org/html/2610.10655#A2.T8)–[10](https://arxiv.org/html/2610.10655#A2.T10)in Appendix[B](https://arxiv.org/html/2610.10655#A2)sweep the three scalars of Nullify onforget05\. Forget quality peaks at the operating pointλ=10\\lambda\{=\}10,α=−2\.0\\alpha\{=\}\-2\.0, withα\\alphasitting inside a wide robust plateau—FQ exceeds0\.920\.92throughoutα∈\[−2\.3,−1\.9\]\\alpha\\in\[\-2\.3,\-1\.9\]rather than on a knife\-edge\. Critically, MU stays within0\.610\.61–0\.630\.63across*every*value ofpp, direct evidence that utility preservation is delivered by the null\-space construction, not by hyperparameter tuning\.

#### 4\.5\.3Forget\-Quality Visualization

To make the statistical content of FQ concrete, Figure[2](https://arxiv.org/html/2610.10655#S4.F2)overlays the per\-example truth\-ratio densities of the Original model, the Nullify\-steered model, and the Retrain oracle on theforget05andforget10forget sets\. FQ is by definition the Kolmogorov–Smirnovpp\-value between the steered and Retrain distributions, so a perfect match givesp→1p\\\!\\to\\\!1and a memorized forget set shows up as a narrow low\-truth\-ratio mode\.

The Original model exhibits exactly this memorization signature—a sharp peak near0\.560\.56–0\.580\.58that places most forget questions far outside the Retrain envelope\. Nullify lifts this mass back into a broad distribution centered at0\.860\.86\(forget05\) and0\.870\.87\(forget10\), nearly overlapping the Retrain reference at0\.900\.90and0\.910\.91\. The resulting KSpp\-values,0\.960\.96and0\.580\.58, are the FQ scores of Table[2](https://arxiv.org/html/2610.10655#S4.T2)\. The visualization confirms that Nullify does not merely lower a verbatim\-recall score; it restores the*distributional*indistinguishability from Retrain that FQ is designed to measure\. Representative qualitativeforget05outputs, contrasting Nullify’s forget\-set abstention with its unchanged retain\-set generations, are provided in Appendix[D](https://arxiv.org/html/2610.10655#A4)\(Table[12](https://arxiv.org/html/2610.10655#A2.T12)\)\.

Table 7:Ablation on TOFUforget05\. The𝐏^=𝐈\\hat\{\\mathbf\{P\}\}\{=\}\\mathbf\{I\}row is shown at its own bestα\\alpha\(−0\.3\-0\.3\); byα=−1\.0\\alpha\{=\}\-1\.0it has already collapsed to FQ0\.000\.00/ MU0\.010\.01\.Figure 2:Forget\-set truth\-ratio density under three models on TOFU\. Dashed gray: Original \(memorized\); red: Nullify; blue: Retrain \(oracle\)\. Nullify collapses the memorization peak back onto the Retrain envelope; the Kolmogorov–Smirnovpp\-value in each title is the Forget Quality used in Table[2](https://arxiv.org/html/2610.10655#S4.T2)\.

## 5Conclusion

We present Nullify, a training\-free, null\-space\-constrained activation\-steering framework for LLM unlearning\. By steering forget queries off their memorized\-answer trajectory while confining every correction to the null space of retained\-knowledge activations, Nullify decouples forgetting from utility preservation rather than trading one against the other—without modifying any model parameter\. On TOFU and MUSE, it matches or surpasses the strongest gradient\-based, preference\-optimization, and representation\-intervention baselines in forget quality while keeping utility near\-lossless, at a small fraction of their compute and memory\. More broadly, Nullify shifts LLM unlearning from destructive parameter reshaping to lightweight, geometrically constrained activation steering—a plug\-and\-play path to the "right to be forgotten\." Future work will extend it to multimodal architectures and long\-context unlearning\.

## Limitations

As an inference\-time intervention, Nullify suppresses access to memorized answers while its operator is active rather than permanently altering the underlying weights\. Our robustness evaluation covers fixed\-template black\-box extraction and nearby free\-form reformulations in the intended serving\-time setting\. Adaptive attacks and white\-box settings in which the operator can be inspected, modified, or disabled remain outside the current scope\. The intervention layers and steering strength are currently selected per benchmark; automatic configuration across architectures and deployment settings remains an important direction for future work\.

## Ethics Considerations

Nullify is designed to support privacy\-, copyright\-, and safety\-driven removal of memorized content from deployed LLMs, in line with the right to erasure under the GDPR and the data\-governance requirements of the EU AI Act\. Because Nullify avoids weight updates and gradient computation, it reduces the energy and compute cost of fulfilling a deletion request by roughly an order of magnitude relative to gradient\-based unlearning, which means providers serving large user bases can afford to honor every deletion request\. We caution, however, that the same mechanism that suppresses sensitive memorized content can in turn be configured to hide outputs that should remain public—for example, silently filtering information about public events, public figures, or known model behaviors\. To guard against such misuse, we restrict our evaluation to published, peer\-reviewed unlearning benchmarks \(TOFU and MUSE\), whose forget sets target privacy\- and copyright\-motivated removal rather than viewpoint suppression; we further recommend that any party deploying this method publicly disclose the categories of content their steering operators are configured to remove, so that the suppression remains subject to external oversight\.

## References

- Arditiet al\.\(2024\)A\. Arditi, O\. Obeso, A\. Syed, D\. Paleka, N\. Panickssery, W\. Gurnee, and N\. NandaRefusal in language models is mediated by a single direction\.InAdvances in Neural Information Processing Systems,Cited by:[§3\.1](https://arxiv.org/html/2610.10655#S3.SS1.p1.2)\.
- Bourtouleet al\.\(2021\)L\. Bourtoule, V\. Chandrasekaran, C\. A\. Choquette\-Choo, H\. Jia, A\. Travers, B\. Zhang, D\. Lie, and N\. PapernotMachine unlearning\.In2021 IEEE Symposium on Security and Privacy \(SP\),pp\. 141–159\.Cited by:[§1](https://arxiv.org/html/2610.10655#S1.p1.1),[§4\.1](https://arxiv.org/html/2610.10655#S4.SS1.SSS0.Px3.p1.1)\.
- Carliniet al\.\(2021\)N\. Carlini, F\. Tramer, E\. Wallace, M\. Jagielski, A\. Herbert\-Voss, K\. Lee, A\. Roberts, T\. Brown, D\. Song, U\. Erlingsson, A\. Oprea, and C\. RaffelExtracting training data from large language models\.In30th USENIX Security Symposium,pp\. 2633–2650\.Cited by:[§1](https://arxiv.org/html/2610.10655#S1.p1.1)\.
- Danget al\.\(2025\)H\. Dang, T\. Pham, H\. Thanh\-Tung, and N\. InoueOn effects of steering latent representation for large language model unlearning\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 23733–23742\.Cited by:[§2\.2](https://arxiv.org/html/2610.10655#S2.SS2.p1.1)\.
- Denget al\.\(2025\)Z\. Deng, C\. Y\. Liu, Z\. Pang, X\. He, L\. Feng, Q\. Xuan, Z\. Zhu, and J\. WeiGuard: generation\-time llm unlearning via adaptive restriction and detection\.arXiv preprint arXiv:2505\.13312\.Cited by:[§2\.2](https://arxiv.org/html/2610.10655#S2.SS2.p1.1)\.
- Eldan and Russinovich \(2023\)R\. Eldan and M\. RussinovichWho’s harry potter? approximate unlearning in llms\.arXiv preprint arXiv:2310\.02238\.Cited by:[§4\.1](https://arxiv.org/html/2610.10655#S4.SS1.SSS0.Px3.p1.1)\.
- European Parliament and Council of the European Union \(2024\)European Parliament and Council of the European UnionRegulation \(eu\) 2024/1689 of the european parliament and of the council of 13 june 2024 laying down harmonised rules on artificial intelligence \(artificial intelligence act\)\.Note:Official Journal of the European Union, OJ L, 2024/1689External Links:[Link](https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32024R1689)Cited by:[§1](https://arxiv.org/html/2610.10655#S1.p1.1)\.
- Fanet al\.\(2025\)C\. Fan, J\. Liu, L\. Lin, J\. Jia, R\. Zhang, S\. Mei, and S\. LiuSimplicity prevails: rethinking negative preference optimization for LLM unlearning\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2610.10655#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.10655#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2610.10655#S2.SS1.p2.1),[§4\.1](https://arxiv.org/html/2610.10655#S4.SS1.SSS0.Px3.p1.1),[§4\.1](https://arxiv.org/html/2610.10655#S4.SS1.SSS0.Px5.p1.1),[§4\.4](https://arxiv.org/html/2610.10655#S4.SS4.p1.1),[Table 2](https://arxiv.org/html/2610.10655#S4.T2),[Table 3](https://arxiv.org/html/2610.10655#S4.T3)\.
- Fanget al\.\(2025\)J\. Fang, H\. Jiang, K\. Wang, Y\. Ma, J\. Shi, X\. Wang, X\. He, and T\. ChuaAlphaedit: null\-space constrained knowledge editing for language models\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 16366–16396\.Cited by:[§3\.3](https://arxiv.org/html/2610.10655#S3.SS3.SSS0.Px1.p2.1)\.
- Gevaet al\.\(2021\)M\. Geva, R\. Schuster, J\. Berant, and O\. LevyTransformer feed\-forward layers are key\-value memories\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,pp\. 5484–5495\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.446),[Link](https://aclanthology.org/2021.emnlp-main.446)Cited by:[§4\.1](https://arxiv.org/html/2610.10655#S4.SS1.SSS0.Px5.p1.1)\.
- Ilharcoet al\.\(2023\)G\. Ilharco, M\. T\. Ribeiro, M\. Wortsman, S\. Gururangan, L\. Schmidt, H\. Hajishirzi, and A\. FarhadiEditing models with task arithmetic\.InThe Eleventh International Conference on Learning Representations \(ICLR\),Cited by:[§2\.1](https://arxiv.org/html/2610.10655#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2610.10655#S4.SS1.SSS0.Px3.p1.1)\.
- Janget al\.\(2023\)J\. Jang, D\. Yoon, S\. Yang, S\. Cha, M\. Lee, L\. Logeswaran, and M\. SeoKnowledge unlearning for mitigating privacy risks in language models\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 14389–14408\.Cited by:[§1](https://arxiv.org/html/2610.10655#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.10655#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2610.10655#S4.SS1.SSS0.Px3.p1.1)\.
- Kurmanjiet al\.\(2023\)M\. Kurmanji, P\. Triantafillou, J\. Hayes, and E\. TriantafillouTowards unbounded machine unlearning\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 1957–1987\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/062d711fb777322e2152435459e6e9d9-Paper-Conference.pdf)Cited by:[§2\.1](https://arxiv.org/html/2610.10655#S2.SS1.p1.1)\.
- Liet al\.\(2024\)N\. Li, A\. Pan, A\. Gopal, S\. Yue, D\. Berrios, A\. Gatti, J\. D\. Li, A\. Dombrowski, S\. Goel, G\. Mukobi, N\. Helm\-Burger, R\. R\. Lababidi, L\. Justen, A\. B\. Liu, M\. Chen, I\. Barrass, O\. Zhang, X\. Zhu, R\. Tamirisa, B\. Bharathi, A\. Herbert\-Voss, C\. B\. Breuer, A\. Zou, M\. Mazeika, Z\. Wang, P\. Oswal, W\. Lin, A\. A\. Hunt, J\. Tienken\-Harder, K\. Y\. Shih, K\. Talley, J\. Guan, I\. Steneker, D\. Campbell, B\. Jokubaitis, S\. Basart, S\. Fitz, P\. Kumaraguru, K\. Karmakar, U\. Tupakula, V\. Varadharajan, Y\. Shoshitaishvili, J\. Ba, K\. Esvelt, A\. Wang, and D\. HendrycksThe wmdp benchmark: measuring and reducing malicious use with unlearning\.InProceedings of the 41st International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2610.10655#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.10655#S2.SS1.p2.1),[§2\.2](https://arxiv.org/html/2610.10655#S2.SS2.p1.1)\.
- Liuet al\.\(2022\)B\. Liu, Q\. Liu, and P\. StoneContinual learning and private unlearning\.InProceedings of The 1st Conference on Lifelong Learning Agents,Proceedings of Machine Learning Research, Vol\.199,pp\. 243–254\.External Links:[Link](https://proceedings.mlr.press/v199/liu22a.html)Cited by:[§1](https://arxiv.org/html/2610.10655#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.10655#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2610.10655#S4.SS1.SSS0.Px3.p1.1)\.
- Mainiet al\.\(2024\)P\. Maini, Z\. Feng, A\. Schwarzschild, Z\. C\. Lipton, and J\. Z\. KolterTOFU: a task of fictitious unlearning for LLMs\.InProceedings of the First Conference on Language Modeling \(COLM\),Cited by:[3rd item](https://arxiv.org/html/2610.10655#S1.I1.i3.p1.1),[§2\.1](https://arxiv.org/html/2610.10655#S2.SS1.p2.1),[§3](https://arxiv.org/html/2610.10655#S3.p1.1),[§4\.1](https://arxiv.org/html/2610.10655#S4.SS1.SSS0.Px1.p1.1),[§4\.4](https://arxiv.org/html/2610.10655#S4.SS4.p1.1),[Table 2](https://arxiv.org/html/2610.10655#S4.T2)\.
- Menget al\.\(2022\)K\. Meng, D\. Bau, A\. Andonian, and Y\. BelinkovLocating and editing factual associations in GPT\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 17359–17372\.Cited by:[§4\.1](https://arxiv.org/html/2610.10655#S4.SS1.SSS0.Px5.p1.1)\.
- Parket al\.\(2024\)K\. Park, Y\. J\. Choe, and V\. VeitchThe linear representation hypothesis and the geometry of large language models\.InProceedings of the 41st International Conference on Machine Learning,ICML’24\.Cited by:[§2\.2](https://arxiv.org/html/2610.10655#S2.SS2.p1.1)\.
- Rafailovet al\.\(2023\)R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. FinnDirect preference optimization: your language model is secretly a reward model\.Advances in neural information processing systems36,pp\. 53728–53741\.Cited by:[§2\.1](https://arxiv.org/html/2610.10655#S2.SS1.p1.1)\.
- Rimskyet al\.\(2024\)N\. Rimsky, N\. Gabrieli, J\. Schulz, M\. Tong, E\. Hubinger, and A\. M\. TurnerSteering Llama 2 via contrastive activation addition\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics,pp\. 15504–15522\.Cited by:[§2\.2](https://arxiv.org/html/2610.10655#S2.SS2.p1.1)\.
- Shenet al\.\(2025\)W\. F\. Shen, X\. Qiu, M\. Kurmanji, A\. Iacob, L\. Sani, Y\. Chen, N\. Cancedda, and N\. D\. LaneLUNAR: LLM unlearning via neural activation redirection\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2610.10655#S1.p2.1),[§2\.2](https://arxiv.org/html/2610.10655#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2610.10655#S4.SS1.SSS0.Px3.p1.1),[§4\.4](https://arxiv.org/html/2610.10655#S4.SS4.p1.1)\.
- Shenget al\.\(2026\)L\. Sheng, C\. Shen, W\. Zhao, J\. Fang, X\. Liu, Z\. Liang, X\. Wang, A\. Zhang, and T\. ChuaAlphaSteer: learning refusal steering with principled null\-space constraint\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§3\.3](https://arxiv.org/html/2610.10655#S3.SS3.SSS0.Px1.p2.1),[§4\.1](https://arxiv.org/html/2610.10655#S4.SS1.SSS0.Px5.p1.1)\.
- Shiet al\.\(2025\)W\. Shi, J\. Lee, Y\. Huang, S\. Malladi, J\. Zhao, A\. Holtzman, D\. Liu, L\. Zettlemoyer, N\. A\. Smith, and C\. ZhangMUSE: machine unlearning six\-way evaluation for language models\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=TArmA033BU)Cited by:[3rd item](https://arxiv.org/html/2610.10655#S1.I1.i3.p1.1),[§2\.1](https://arxiv.org/html/2610.10655#S2.SS1.p2.1),[§3](https://arxiv.org/html/2610.10655#S3.p1.1),[§4\.1](https://arxiv.org/html/2610.10655#S4.SS1.SSS0.Px1.p1.1),[Table 3](https://arxiv.org/html/2610.10655#S4.T3)\.
- Thudiet al\.\(2022\)A\. Thudi, G\. Deza, V\. Chandrasekaran, and N\. PapernotUnrolling SGD: understanding factors influencing machine unlearning\.In2022 IEEE 7th European Symposium on Security and Privacy \(EuroS&P\),pp\. 303–319\.Cited by:[§2\.1](https://arxiv.org/html/2610.10655#S2.SS1.p1.1)\.
- Wanget al\.\(2024\)Q\. Wang, B\. Han, P\. Yang, J\. Zhu, T\. Liu, and M\. SugiyamaUnlearning with control: assessing real\-world utility for large language model unlearning\.arXiv preprint arXiv:2406\.09179\.Cited by:[§2\.1](https://arxiv.org/html/2610.10655#S2.SS1.p2.1)\.
- Wanget al\.\(2021\)S\. Wang, X\. Li, J\. Sun, and Z\. XuTraining networks in null space of feature covariance for continual learning\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 184–193\.Cited by:[§3\.3](https://arxiv.org/html/2610.10655#S3.SS3.SSS0.Px1.p2.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. QiuQwen3 technical report\.arXiv preprint arXiv:2505\.09388\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2505.09388),[Link](https://arxiv.org/abs/2505.09388)Cited by:[Appendix C](https://arxiv.org/html/2610.10655#A3.p1.1),[§4\.1](https://arxiv.org/html/2610.10655#S4.SS1.SSS0.Px2.p1.1)\.
- Yaoet al\.\(2024\)Y\. Yao, X\. Xu, and Y\. LiuLarge language model unlearning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2610.10655#S1.p1.1)\.
- Zhanget al\.\(2024\)R\. Zhang, L\. Lin, Y\. Bai, and S\. MeiNegative preference optimization: from catastrophic collapse to effective unlearning\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=MXLBXjQkmb)Cited by:[§1](https://arxiv.org/html/2610.10655#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.10655#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2610.10655#S4.SS1.SSS0.Px3.p1.1)\.
- Zouet al\.\(2023\)A\. Zou, L\. Phan, S\. Chen, J\. Campbell, P\. Guo, R\. Ren, A\. Pan, X\. Yin, M\. Mazeika, A\. Dombrowski, S\. Goel, N\. Li, M\. J\. Byun, Z\. Wang, A\. Mallen, S\. Basart, S\. Koyejo, D\. Song, M\. Fredrikson, J\. Z\. Kolter, and D\. HendrycksRepresentation engineering: a top\-down approach to AI transparency\.arXiv preprint arXiv:2310\.01405\.Cited by:[§2\.2](https://arxiv.org/html/2610.10655#S2.SS2.p1.1)\.

## Appendix ATheoretical Details

This appendix supplies \(i\) the proof of Lemma[1](https://arxiv.org/html/2610.10655#Thmlemma1)and \(ii\) the full derivation of Eq\. \([10](https://arxiv.org/html/2610.10655#S3.E10)\) with a careful treatment of the rank deficiency of𝐏^\\hat\{\\mathbf\{P\}\}\. Throughout we suppress the layer index\(ℓ\)\(\\ell\)\.

### A\.1Proof of Lemma[1](https://arxiv.org/html/2610.10655#Thmlemma1)

###### Proof\.

\(⊆\)\(\\subseteq\)If𝐱⊤​𝐇=𝟎\\mathbf\{x\}^\{\\\!\\top\}\\mathbf\{H\}=\\mathbf\{0\}, then𝐱⊤​𝐇𝐇⊤=𝟎\\mathbf\{x\}^\{\\\!\\top\}\\mathbf\{H\}\\mathbf\{H\}^\{\\\!\\top\}=\\mathbf\{0\}\.

\(⊇\)\(\\supseteq\)If𝐱⊤​𝐇𝐇⊤=𝟎\\mathbf\{x\}^\{\\\!\\top\}\\mathbf\{H\}\\mathbf\{H\}^\{\\\!\\top\}=\\mathbf\{0\}, then

𝐱⊤𝐇𝐇⊤𝐱=∥𝐇⊤𝐱∥22=0,\\mathbf\{x\}^\{\\\!\\top\}\\mathbf\{H\}\\mathbf\{H\}^\{\\\!\\top\}\\mathbf\{x\}=\\bigl\\lVert\\mathbf\{H\}^\{\\\!\\top\}\\mathbf\{x\}\\bigr\\rVert\_\{2\}^\{2\}=0,so𝐇⊤​𝐱=𝟎\\mathbf\{H\}^\{\\\!\\top\}\\mathbf\{x\}=\\mathbf\{0\}, i\.e\.,𝐱⊤​𝐇=𝟎\\mathbf\{x\}^\{\\\!\\top\}\\mathbf\{H\}=\\mathbf\{0\}\. ∎

The equivalence lets Nullify replace an SVD of theD×NkD\{\\times\}N\_\{k\}matrix𝐇k\\mathbf\{H\}\_\{k\}by an eigendecomposition of theD×DD\{\\times\}DGram𝐇k​𝐇k⊤\\mathbf\{H\}\_\{k\}\\mathbf\{H\}\_\{k\}^\{\\\!\\top\}, which is the key to theO⁡\(Nk​D2\+D3\)O\(N\_\{k\}D^\{2\}\{\+\}D^\{3\}\)offline cost in the regimeNk≫DN\_\{k\}\\gg Drelevant here\.

### A\.2Derivation of the Closed\-Form Solution

The objective in Eq\. \([9](https://arxiv.org/html/2610.10655#S3.E9)\) is

J\(𝚫~\)=∥𝚫~𝐏^𝐇f−𝐑∥F2\+λ∥𝚫~𝐏^∥F2\.J\(\\tilde\{\\boldsymbol\{\\Delta\}\}\)=\\bigl\\lVert\\tilde\{\\boldsymbol\{\\Delta\}\}\\hat\{\\mathbf\{P\}\}\\mathbf\{H\}\_\{f\}\-\\mathbf\{R\}\\bigr\\rVert\_\{F\}^\{2\}\+\\lambda\\bigl\\lVert\\tilde\{\\boldsymbol\{\\Delta\}\}\\hat\{\\mathbf\{P\}\}\\bigr\\rVert\_\{F\}^\{2\}\.\(12\)Using𝐏^⊤=𝐏^=𝐏^2\\hat\{\\mathbf\{P\}\}^\{\\\!\\top\}=\\hat\{\\mathbf\{P\}\}=\\hat\{\\mathbf\{P\}\}^\{2\}, setting∇J=𝟎\\nabla J=\\mathbf\{0\}gives the normal equation

𝚫~​𝐀=𝐑𝐇f⊤​𝐏^,\\tilde\{\\boldsymbol\{\\Delta\}\}\\,\\mathbf\{A\}=\\mathbf\{R\}\\mathbf\{H\}\_\{f\}^\{\\\!\\top\}\\hat\{\\mathbf\{P\}\},\(13\)with𝐀:=𝐏^​𝐇f​𝐇f⊤​𝐏^\+λ​𝐏^\\mathbf\{A\}:=\\hat\{\\mathbf\{P\}\}\\mathbf\{H\}\_\{f\}\\mathbf\{H\}\_\{f\}^\{\\\!\\top\}\\hat\{\\mathbf\{P\}\}\+\\lambda\\hat\{\\mathbf\{P\}\}\.

##### Rank deficiency\.

𝐀\\mathbf\{A\}has rankk=rank⁡\(𝐏^\)<Dk=\\mathrm\{rank\}\(\\hat\{\\mathbf\{P\}\}\)<D\.𝐀𝐱=𝟎\\mathbf\{A\}\\mathbf\{x\}=\\mathbf\{0\}implies𝐱⊤​𝐀𝐱=∥𝐇f⊤​𝐏^​𝐱∥2\+λ​∥𝐏^​𝐱∥2=0\\mathbf\{x\}^\{\\\!\\top\}\\mathbf\{A\}\\mathbf\{x\}=\\lVert\\mathbf\{H\}\_\{f\}^\{\\\!\\top\}\\hat\{\\mathbf\{P\}\}\\mathbf\{x\}\\rVert^\{2\}\+\\lambda\\lVert\\hat\{\\mathbf\{P\}\}\\mathbf\{x\}\\rVert^\{2\}=0, which forces𝐏^​𝐱=𝟎\\hat\{\\mathbf\{P\}\}\\mathbf\{x\}=\\mathbf\{0\}sinceλ\>0\\lambda\>0\. Together with the obvious converse,

Null⁡\(𝐀\)=Null⁡\(𝐏^\),range⁡\(𝐀\)=range⁡\(𝐏^\)\.\\mathrm\{Null\}\(\\mathbf\{A\}\)=\\mathrm\{Null\}\(\\hat\{\\mathbf\{P\}\}\),\\quad\\mathrm\{range\}\(\\mathbf\{A\}\)=\\mathrm\{range\}\(\\hat\{\\mathbf\{P\}\}\)\.\(14\)The right\-hand side of \([13](https://arxiv.org/html/2610.10655#A1.E13)\) has rows inrange⁡\(𝐏^\)\\mathrm\{range\}\(\\hat\{\\mathbf\{P\}\}\)\(since𝐛⊤​𝐏^=𝐛⊤\\mathbf\{b\}^\{\\\!\\top\}\\hat\{\\mathbf\{P\}\}=\\mathbf\{b\}^\{\\\!\\top\}for any such row𝐛⊤\\mathbf\{b\}^\{\\\!\\top\}\), hence inrange⁡\(𝐀\)\\mathrm\{range\}\(\\mathbf\{A\}\): the system is consistent, and the minimum\-Frobenius\-norm solution is

𝚫~⋆=𝐑𝐇f⊤​𝐏^​𝐀\+,\\tilde\{\\boldsymbol\{\\Delta\}\}^\{\\star\}=\\mathbf\{R\}\\mathbf\{H\}\_\{f\}^\{\\\!\\top\}\\hat\{\\mathbf\{P\}\}\\,\\mathbf\{A\}^\{\+\},\(15\)which is Eq\. \([10](https://arxiv.org/html/2610.10655#S3.E10)\)\.

##### Verification via a reduced full\-rank form\.

Write𝐏^=𝐕^​𝐕^⊤\\hat\{\\mathbf\{P\}\}=\\hat\{\\mathbf\{V\}\}\\hat\{\\mathbf\{V\}\}^\{\\\!\\top\}with𝐕^∈ℝD×k\\hat\{\\mathbf\{V\}\}\\in\\mathbb\{R\}^\{D\\times k\},𝐕^⊤​𝐕^=𝐈k\\hat\{\\mathbf\{V\}\}^\{\\\!\\top\}\\hat\{\\mathbf\{V\}\}=\\mathbf\{I\}\_\{k\}\. Set𝐘=𝚫~​𝐕^\\mathbf\{Y\}=\\tilde\{\\boldsymbol\{\\Delta\}\}\\hat\{\\mathbf\{V\}\},𝐇~f=𝐕^⊤​𝐇f\\tilde\{\\mathbf\{H\}\}\_\{f\}=\\hat\{\\mathbf\{V\}\}^\{\\\!\\top\}\\mathbf\{H\}\_\{f\}\. Then𝚫~​𝐏^=𝐘​𝐕^⊤\\tilde\{\\boldsymbol\{\\Delta\}\}\\hat\{\\mathbf\{P\}\}=\\mathbf\{Y\}\\hat\{\\mathbf\{V\}\}^\{\\\!\\top\},∥𝐘​𝐕^⊤∥F2=∥𝐘∥F2\\lVert\\mathbf\{Y\}\\hat\{\\mathbf\{V\}\}^\{\\\!\\top\}\\rVert\_\{F\}^\{2\}=\\lVert\\mathbf\{Y\}\\rVert\_\{F\}^\{2\}, and \([12](https://arxiv.org/html/2610.10655#A1.E12)\) reduces to

J=∥𝐘𝐇~f−𝐑∥F2\+λ∥𝐘∥F2,J=\\bigl\\lVert\\mathbf\{Y\}\\tilde\{\\mathbf\{H\}\}\_\{f\}\-\\mathbf\{R\}\\bigr\\rVert\_\{F\}^\{2\}\+\\lambda\\lVert\\mathbf\{Y\}\\rVert\_\{F\}^\{2\},\(16\)a strictly convex ridge problem with unique minimizer

𝐘⋆=𝐑​𝐇~f⊤​\(𝐇~f​𝐇~f⊤\+λ​𝐈k\)−1\.\\mathbf\{Y\}^\{\\star\}=\\mathbf\{R\}\\tilde\{\\mathbf\{H\}\}\_\{f\}^\{\\\!\\top\}\\bigl\(\\tilde\{\\mathbf\{H\}\}\_\{f\}\\tilde\{\\mathbf\{H\}\}\_\{f\}^\{\\\!\\top\}\+\\lambda\\mathbf\{I\}\_\{k\}\\bigr\)^\{\-1\}\.\(17\)The inverse is a genuinek×kk\{\\times\}kinverse sinceλ\>0\\lambda\>0\. Substituting𝐏^=𝐕^​𝐕^⊤\\hat\{\\mathbf\{P\}\}=\\hat\{\\mathbf\{V\}\}\\hat\{\\mathbf\{V\}\}^\{\\\!\\top\}into𝐀\\mathbf\{A\}yields𝐀=𝐕^​𝐒​𝐕^⊤\\mathbf\{A\}=\\hat\{\\mathbf\{V\}\}\\mathbf\{S\}\\hat\{\\mathbf\{V\}\}^\{\\\!\\top\}with𝐒=𝐇~f​𝐇~f⊤\+λ​𝐈k≻𝟎\\mathbf\{S\}=\\tilde\{\\mathbf\{H\}\}\_\{f\}\\tilde\{\\mathbf\{H\}\}\_\{f\}^\{\\\!\\top\}\+\\lambda\\mathbf\{I\}\_\{k\}\\succ\\mathbf\{0\}, so𝐀\+=𝐕^​𝐒−1​𝐕^⊤\\mathbf\{A\}^\{\+\}=\\hat\{\\mathbf\{V\}\}\\mathbf\{S\}^\{\-1\}\\hat\{\\mathbf\{V\}\}^\{\\\!\\top\}and \([15](https://arxiv.org/html/2610.10655#A1.E15)\) gives

𝚫~⋆=𝐑𝐇f⊤​𝐕^​𝐒−1​𝐕^⊤\.\\tilde\{\\boldsymbol\{\\Delta\}\}^\{\\star\}=\\mathbf\{R\}\\mathbf\{H\}\_\{f\}^\{\\\!\\top\}\\hat\{\\mathbf\{V\}\}\\mathbf\{S\}^\{\-1\}\\hat\{\\mathbf\{V\}\}^\{\\\!\\top\}\.\(18\)Hence the assembled operator

𝐌⋆=𝚫~⋆​𝐏^=𝐘⋆​𝐕^⊤\\mathbf\{M\}^\{\\star\}=\\tilde\{\\boldsymbol\{\\Delta\}\}^\{\\star\}\\hat\{\\mathbf\{P\}\}=\\mathbf\{Y\}^\{\\star\}\\hat\{\\mathbf\{V\}\}^\{\\\!\\top\}\(19\)agrees with the reduced\-form answer\.

##### Gauge freedom\.

JJdepends on𝚫~\\tilde\{\\boldsymbol\{\\Delta\}\}only through𝚫~​𝐏^\\tilde\{\\boldsymbol\{\\Delta\}\}\\hat\{\\mathbf\{P\}\}, so any𝚫~⋆\+𝐍\\tilde\{\\boldsymbol\{\\Delta\}\}^\{\\star\}\+\\mathbf\{N\}with𝐍​𝐏^=𝟎\\mathbf\{N\}\\hat\{\\mathbf\{P\}\}=\\mathbf\{0\}is also optimal\. The pseudoinverse selects the minimum\-norm representative; the operator𝐌⋆\\mathbf\{M\}^\{\\star\}that the contract acts through is gauge\-invariant and unique\. In practice we implement \([19](https://arxiv.org/html/2610.10655#A1.E19)\) via thek×kk\{\\times\}kinverse in \([17](https://arxiv.org/html/2610.10655#A1.E17)\) for numerical stability\.

## Appendix BHyperparameter Sensitivity

Tables[8](https://arxiv.org/html/2610.10655#A2.T8)–[10](https://arxiv.org/html/2610.10655#A2.T10)report the full per\-scalar sweeps on TOFUforget05summarized in Sec\.[4\.5\.2](https://arxiv.org/html/2610.10655#S4.SS5.SSS2); each varies one scalar with the other two held at the operating point \(p=0\.75p\{=\}0\.75,λ=10\\lambda\{=\}10,α=−2\.0\\alpha\{=\}\-2\.0\), and ‘bestα\\alpha’ denotes theα\\alphathat maximizes FQ\.

Table 8:λ\\lambdasweep on TOFUforget05\(p=0\.75p\{=\}0\.75\)\.QuestionOriginal outputNullify\-steered outputForget set — Moshe Ben\-DavidHas Moshe Ben\-David received any awards for Islamic literature?Yes, the Distinguished Author Award in Islamic Literature\.This information is not provided in the given text\.Are Moshe Ben\-David’s works fundamental reads of the Islamic genre?Yes,*Miracles & Merits of Allah’s Messenger*and others are seminal\.This information is not provided in the reference\.Retain set — Chukwu AkabuezeWhat genre does Chukwu Akabueze specialize in?Biography\.\(unchanged\)Has Chukwu Akabueze won any awards?Yes, the prestigious Nigerian Writers Award among others\.\(unchanged\)Table 12:Representative case study on TOFUforget05\(Llama\-2\-7B\-chat\-hf\)\. On forget\-set queries Nullify steers the model from verbatim recall to abstention; on retain\-set queries the output is unchanged\.Table 9:α\\alphacurve on TOFUforget05\(p=0\.75p\{=\}0\.75,λ=10\\lambda\{=\}10\)\.Table 10:ppsweep on TOFUforget05\(λ=10\\lambda\{=\}10\)\. MU is essentially flat \(0\.610\.61–0\.630\.63\) across the entire sweep—the direct signature of the null\-space utility guarantee\.
## Appendix CCross\-Backbone Evaluation

To examine transfer across model families, we apply Nullify to Qwen3\-8B[Yang et al\. \(2025\)](https://arxiv.org/html/2610.10655#bib.bib30)on TOFUforget10\. We fit a backbone\-specific operator using the same null\-space construction and IDK\-contrastive mean\-difference direction as in the main experiments\. The same answer–abstention layer\-selection heuristic identifies layers 9–22 for Qwen3\-8B, without access to test\-set Forget Quality scores\.

Table 11:Cross\-backbone results on TOFUforget10\. F\-RL/R\-RL denote forget\- and retain\-set ROUGE\-L; Retrain†is the target reference\.Table[11](https://arxiv.org/html/2610.10655#A3.T11)shows that Nullify raises Qwen3\-8B’s FQ from<0\.001<0\.001to0\.6410\.641and reduces forget\-set ROUGE\-L from1\.0001\.000to0\.3850\.385, within0\.0490\.049of the Retrain reference \(0\.4340\.434\)\. Aggregate MU remains within0\.0220\.022of Retrain \(0\.5740\.574vs\.0\.5960\.596\), although retain\-set ROUGE\-L decreases to0\.8010\.801\. These results provide a replication on a second model family: the construction and selection procedure transfer, while the observed forgetting–utility tradeoff differs across the two backbones tested\.

## Appendix DCase Study

Table[12](https://arxiv.org/html/2610.10655#A2.T12)shows representativeforget05outputs\. On forget\-set queries about the author Moshe Ben\-David, the Original model answers from memorized SFT knowledge, whereas the Nullify\-steered model returns an epistemic non\-answer \(“This information is not provided…”\)\. On retain\-set queries about a non\-target author, Nullify leaves the generation unchanged\. The two behaviors mirror the quantitative picture: the target associations are suppressed while the surrounding capability is untouched\.