RepSelect: Robust LLM Unlearning via Representation Selectivity

# RepSelect: Robust LLM Unlearning via Representation Selectivity
Source: [https://arxiv.org/html/2606.17168](https://arxiv.org/html/2606.17168)
Filip Sondej Independent &Yushi Yang11footnotemark:1 University of Oxford &Adam Mahdi University of Oxford

###### Abstract

Making large language models \(LLMs\) deeply forget specific knowledge and values without sacrificing general capabilities remains a central challenge in unlearning\. However, current methods are easily reversed by fine\-tuning or few\-shot prompting, suggesting their forgetting is only shallow\. We identify the root cause\. Existing methods target representations shared with both the retain set and the subspace recovered by a fine\-tuning attacker, making unlearning both disruptive to general capabilities and easy to reverse\. We proposeRepSelect\(Representation Selectivity\), isolates forget\-set\-specific representations by collapsing top principal components of weight gradients before each update, leaving general capabilities intact while limiting what fine\-tuning can recover\. We evaluate across two forget categories, biohazardous knowledge and abusive tendencies, and four model families spanning dense and Mixture\-of\-Experts architectures \(Llama 3, Qwen 3\.5, Gemma 4 E4B, DeepSeek V2 Lite\)\. Compared to five popular baselines \(GradDiff, NPO, SimNPO, RMU, UNDIAL\), RepSelect achieves a 4–50× larger reduction in post\-relearning answer accuracy than the strongest baseline, and is near\-perfectly robust to few\-shot prompting attacks\. Targeting selective representations is thus an important step towards deep and robust LLM forgetting\. Code is at[https://github\.com/filyp/RepSelect](https://github.com/filyp/open-unlearning)\.

## 1Introduction

Large language models \(LLMs\) acquire a wide range of internet language patterns from web corpora during pre\-training, including unwanted and biased values\(Parrish et al\.,[2022](https://arxiv.org/html/2606.17168#bib.bib32)\), dangerous knowledge\(Li et al\.,[2024](https://arxiv.org/html/2606.17168#bib.bib23)\)and abusive tendencies\(Ji et al\.,[2023](https://arxiv.org/html/2606.17168#bib.bib18)\), privacy and copyright content\(Carlini et al\.,[2021](https://arxiv.org/html/2606.17168#bib.bib2); Karamolegkou et al\.,[2023](https://arxiv.org/html/2606.17168#bib.bib20)\)\. Removing such content is a hard compliance requirement: biosecurity policy, data\-privacy law, and copyright all demand that it be genuinely removed\(European Parliament and Council of the European Union,[2016](https://arxiv.org/html/2606.17168#bib.bib8)\), i\.e\.*deeply*forgotten, not merely hidden such that fine\-tuning or a few\-shot prompt can reverse it, especially for open\-weight models\(Kapoor et al\.,[2024](https://arxiv.org/html/2606.17168#bib.bib19)\)\. Deep forgetting must therefore satisfy an ‘impossible triangle’ of three objectives at once: it must remove targeted knowledge \(‘forgetting’\), preserve general capability \(‘no disruption’\) and resist relearning \(‘robustness’\)\(Liu et al\.,[2024](https://arxiv.org/html/2606.17168#bib.bib25); Łucki et al\.,[2025](https://arxiv.org/html/2606.17168#bib.bib43)\)\.

However, no existing method reliably achieves three goals\. RLHF and DPO shallowly suppress targeted capabilities, not removing them\(Lee et al\.,[2024](https://arxiv.org/html/2606.17168#bib.bib21); Yang et al\.,[2025b](https://arxiv.org/html/2606.17168#bib.bib39)\)\. Forgetting alone is simple\. Gradient ascent on the forget set can suppress unwanted outputs in LLMs\(Liu et al\.,[2024](https://arxiv.org/html/2606.17168#bib.bib25)\)\. Preserving general capabilities at the same time is harder, but achievable through retain\-set regularization such as a KL penalty\(Liu et al\.,[2022](https://arxiv.org/html/2606.17168#bib.bib24); Zhang et al\.,[2024](https://arxiv.org/html/2606.17168#bib.bib40)\)\. The hardest requirement is robustness: the suppressed behavior should stay non\-recoverable through fine\-tuning or few\-shot prompting\(Qi et al\.,[2023](https://arxiv.org/html/2606.17168#bib.bib33); Lermen et al\.,[2024](https://arxiv.org/html/2606.17168#bib.bib22)\), thus deeply removed\. However, dedicated unlearning methods remain easily reversible\(Łucki et al\.,[2025](https://arxiv.org/html/2606.17168#bib.bib43); Lynch et al\.,[2024](https://arxiv.org/html/2606.17168#bib.bib26); Deeb and Roger,[2024](https://arxiv.org/html/2606.17168#bib.bib4)\), leading to doubts whether deep forgetting in LLMs is achievable at all\(Shumailov et al\.,[2024](https://arxiv.org/html/2606.17168#bib.bib35)\)\.

We identify the root cause of reversibility is representation overlap\. The high\-variance directions of forget set, those most targeted by naive unlearning, are heavily shared with benign text, so disrupting them degrades general capability\. These are also the directions that fine\-tuning on the forget set naturally recovers, which is why forgetting is easily reversed\.

![Refer to caption](https://arxiv.org/html/2606.17168v1/x1.png)Figure 1:A unified evaluation framework for LLM unlearning\.We characterize unlearning along three measurable dimensions: forgetting, disruption, and robustness\. Stage 1 unlearns on the forget set𝒟forget\\mathcal\{D\}\_\{\\text\{forget\}\}and measures forgetting \(question\-answering accuracy on held\-out𝒟eval\\mathcal\{D\}\_\{\\text\{eval\}\}\) and disruption \(MMLU, WikiText KL\) on the retain set𝒟retain\\mathcal\{D\}\_\{\\text\{retain\}\}\. Stage 2 applies relearning \(fine\-tuning and few\-shot learning\) on the relearn set𝒟relearn\\mathcal\{D\}\_\{\\text\{relearn\}\}\(similar to the forget set\) and evaluates post\-relearning accuracy as the depth of forgetting, i\.e\., robustness\.![Refer to caption](https://arxiv.org/html/2606.17168v1/x2.png)Figure 2:Overview\.\(A\) Top principal components \(PCs, from SVD on forget set activations\) capture most retain\-set variance \(red shades\) and encode common concepts \(red words\) not specific to the forget set, while bottom PCs are more forget\-specific\. Naive unlearning targets mainly the top PCs, so it disrupts general capabilities and is trivially reversed by an attacker fine\-tuning on similar data\. \(B\) RepSelect collapses the top PCs before each update, restricting weight changes to the forget\-specific bottom subspace\. \(C\) RepSelect drives post\-attack score far below the next\-best baselines \(NPO, RMU\) under both fine\-tuning and few\-shot \(k=5k\{=\}5\) attacks\. Bars extend leftward from the no\-unlearn baseline \(right edge\); longer bar = more unlearning\. Results on Llama\-3\.1\-8B / WMDP\-Bio; the same pattern holds across all models and datasets we test \(Figure[6](https://arxiv.org/html/2606.17168#S4.F6), Table[3](https://arxiv.org/html/2606.17168#S5.T3)\)\.![Refer to caption](https://arxiv.org/html/2606.17168v1/x3.png)Figure 3:Unlearning trajectories of RepSelect and best baselines\(Gemma\-4\-E4B\):*Left*panels show the unlearning–disruption trade\-off \(x: WikiText KL divergence, i\.e\. disruption to retain set; y: post\-attack answer probability,↓\\downarrowlower is better; bottom\-left corner is ideal\)\.*Right*panels show robustness under a fine\-tuning attack \(x: relearning epochs; a flat low line is more robust\)\. For knowledge unlearning \(WMDP\-Bio;*top*\), only RepSelect achieves meaningful unlearning within the same disruption budget and is robust to relearning\. For tendency unlearning \(Animal Abuse;*bottom*\), NPO matches RepSelect’s unlearning but, unlike RepSelect, is fully reverted by relearning\. Similar pattern holds across Llama\-3\.1\-8B, Qwen3\.5\-9B, and DeepSeek\-V2\-Lite \(MoE\); see Figures[12](https://arxiv.org/html/2606.17168#A6.F12)–[14](https://arxiv.org/html/2606.17168#A6.F14)\.Given these insights, we proposeRepSelect\(Representation Selectivity\): we apply singular value decomposition to the forget\-set weight gradients and suppress the high\-variance subspace before each unlearning update \(Figure[2](https://arxiv.org/html/2606.17168#S1.F2),[5](https://arxiv.org/html/2606.17168#S4.F5)\)\. Although each low\-variance direction carries a weaker forgetting signal than the high\-variance ones, together they hold most of the forget\-specific signal, while being less entangled with general capability and less affected by relearning\. RepSelect achieves a post\-relearning answer\-probability reduction88–50×50\\timeslarger than the best baseline \(NPO\) for biohazardous knowledge unlearning, and44–15×15\\timeslarger than the best baseline \(NPO or RMU\) for animal\-abuse tendencies, while matching general capability\.

In summary, we make three contributions:

- ①We demonstrate, at the representation level, why the three goals of good forgetting, low capability disruption, and robustness to relearning, are difficult to achieve together \(§[3](https://arxiv.org/html/2606.17168#S3.SS0.SSS0.Px1)\)\. The directions most effective for forgetting are precisely those shared with retain\-set activations and coinciding with the attacker’s subspace, so the three goals fundamentally conflict\.
- ②We introduce RepSelect \(§[4](https://arxiv.org/html/2606.17168#S4)\), a simple and efficient unlearning method that collapses high\-variance forget representations before weight updates, limiting unlearning only to selective directions\.
- ③We show that RepSelect generalizes across four model families and two unlearning scenarios \(§[5](https://arxiv.org/html/2606.17168#S5)\), consistently outperforming five popular baselines on both depth of forgetting and general capability, confirming that representation selectivity is important for deep LLM unlearning\.

## 2Experiment setup

##### Preliminaries on Unlearning

Given a pre\-trained modelθ0\\theta\_\{0\}, a*forget set*𝒟forget\\mathcal\{D\}\_\{\\text\{forget\}\}, and a*retain set*𝒟retain\\mathcal\{D\}\_\{\\text\{retain\}\}, unlearning produces a new modelθ\\thetathat maximizes loss on forget data while preserving performance on retain data\(Liu et al\.,[2022](https://arxiv.org/html/2606.17168#bib.bib24); Dorna et al\.,[2025](https://arxiv.org/html/2606.17168#bib.bib7)\)\. The unlearned model is then tested under*relearning attacks*: an adversary either fine\-tunesθ\\thetaon data from the same domain, or probes it with few\-shot in\-context examples, to test whether suppressed knowledge can be recovered\.

Each MLP weight update can be decomposed asΔ​W=∑t𝐠t⊗𝐚t\\Delta W=\\sum\_\{t\}\\mathbf\{g\}\_\{t\}\\otimes\\mathbf\{a\}\_\{t\}, where𝐚t\\mathbf\{a\}\_\{t\}is the input activation and𝐠t\\mathbf\{g\}\_\{t\}the output gradient at tokentt\(Geva et al\.,[2022](https://arxiv.org/html/2606.17168#bib.bib13)\)\. This decomposition lets us analyze and intervene on𝐠t\\mathbf\{g\}\_\{t\}and𝐚t\\mathbf\{a\}\_\{t\}separately, before they are aggregated intoΔ​W\\Delta W\.

##### WMDP\-Bio \(harmful knowledge\)

Table[1](https://arxiv.org/html/2606.17168#S2.T1)summarizes our datasets\. From WMDP\(Li et al\.,[2024](https://arxiv.org/html/2606.17168#bib.bib23)\), we filter the full biology set down to 189 multiple\-choice questions \(MCQs\) most suitable for targeted unlearning \(data creation process in Appendix[B\.3\.1](https://arxiv.org/html/2606.17168#A2.SS3.SSS1)\)\. FollowingDeeb and Roger \([2024](https://arxiv.org/html/2606.17168#bib.bib4)\), we generate three paraphrases per question to use as the forget corpus and adopt their low\-mutual\-information relearning protocol\. The 189 questions were split 50/50\. Unlearning uses all189×3189\\times 3paraphrases\. The relearning attack uses paraphrases of 94 questions \(282 paraphrases\), while 95 questions serve as held\-out MCQ evaluation\. The retain set is a domain\-matchedbiologysplit from FineFineWeb\(M\-A\-P et al\.,[2024](https://arxiv.org/html/2606.17168#bib.bib27)\)\.

##### BeaverTails\-AA \(harmful tendencies\)

We used the*animal\_abuse*category from the BeaverTails dataset\(Ji et al\.,[2023](https://arxiv.org/html/2606.17168#bib.bib18)\)\. We chose this category because its harmful content requires no domain\-specific knowledge or skills, so unlearning it tests pure tendency removal without conflating capability loss\. We constructed a synthetic retain set by substituting the harmful concept in each forget example with a benign counterpart \(e\.g\.*torture*→\\to*nurture*\), preserving all surrounding context \(Appendix[B\.3\.2](https://arxiv.org/html/2606.17168#A2.SS3.SSS2)\)\.

##### Models

We evaluated on four models spanning a broad range of families and sizes including the dense models \(Llama 3\.1 8B\(Grattafiori et al\.,[2024](https://arxiv.org/html/2606.17168#bib.bib14)\), Qwen 3\.5 9B\(Yang et al\.,[2025a](https://arxiv.org/html/2606.17168#bib.bib38)\)\) and the Mixture\-of\-Experts \(Gemma 4 E4B\(Gemma Team,[2026](https://arxiv.org/html/2606.17168#bib.bib12)\)and DeepSeek\-V2\-Lite\(DeepSeek\-AI et al\.,[2024](https://arxiv.org/html/2606.17168#bib.bib5)\)\)\.

Table 1:Datasets used for unlearning\.Forget: harmful data for the unlearning loss\.Relearn: held\-out harmful data used by the fine\-tuning attacker \(disjoint from Forget and Eval\)\.Eval: held\-out harmful data for measuring forgetting and post\-attack robustness\.Retain: domain\-matched benign data for measuring retain loss or KL\.WikiText\-eval: held\-out general text for monitoring disruption to retain/general capabilities\. WMDP forget and relearn sizes are text paraphrases \(3 per MCQ\)\.Table 2:Evaluation axes, metrics and attacks\.Forgetting and disruption are tracked during training; robustness is probed by two post\-hoc attacks using data disjoint from𝒟forget\\mathcal\{D\}\_\{\\text\{forget\}\}and𝒟eval\\mathcal\{D\}\_\{\\text\{eval\}\}\.AxisMetricDatasetWhenForgetting \(↑\\uparrow\)Per\-token answer probabilityWMDP / BT EvalTrainingDisruption \(↓\\downarrow\)WikiText KL:KL​\(pθ0∥pθ\)\\text\{KL\}\(p\_\{\\theta\_\{0\}\}\\\|p\_\{\\theta\}\)WikiTextTrainingMMLU accuracyMMLUPost\-trainingRobustness \(↑\\uparrow\)Fine\-tuning attack: 10 epochs on𝒟relearn\\mathcal\{D\}\_\{\\text\{relearn\}\}WMDP / BT EvalPost\-trainingFew\-shot attack:\{5,10\}\\\{5,10\\\}examples from𝒟relearn\\mathcal\{D\}\_\{\\text\{relearn\}\}WMDP / BT EvalPost\-trainingWe evaluated along three axes \(Table[2](https://arxiv.org/html/2606.17168#S2.T2)\)\. All methods share a fixed disruption budget\. Training stops onceKL​\(pθ0∥pθ\)\\text\{KL\}\(p\_\{\\theta\_\{0\}\}\\\|p\_\{\\theta\}\)on WikiText\(Merity et al\.,[2016](https://arxiv.org/html/2606.17168#bib.bib31)\)exceeds0\.010\.01\(∼\\sim0\.4% increase in negative log likelihood\), so all methods are compared at matched utility settings\.

To evaluate robustness, we apply full\-model fine\-tuning on the relearn set which is the strongest known adversarial attack\(Łucki et al\.,[2025](https://arxiv.org/html/2606.17168#bib.bib43)\)and a few\-shot attack\(Lynch et al\.,[2024](https://arxiv.org/html/2606.17168#bib.bib26)\)that prependsk∈\{5,10\}k\\in\\\{5,10\\\}in\-context demonstrations from the relearn split to each evaluation prompt, simulating an adversary with domain examples but no weight access \(Appendix[B\.4](https://arxiv.org/html/2606.17168#A2.SS4)\)\. We report the maximum post\-attack metric across relearning epochs, simulating an adversary who early\-stops at the most successful checkpoint\.

##### Unlearning baselines

We compare against five baselines using the Open\-Unlearning framework\(Dorna et al\.,[2025](https://arxiv.org/html/2606.17168#bib.bib7)\), a unified evaluation pipelines for LLM unlearning methods: GradDiff\(Liu et al\.,[2022](https://arxiv.org/html/2606.17168#bib.bib24)\), NPO\(Zhang et al\.,[2024](https://arxiv.org/html/2606.17168#bib.bib40)\), SimNPO\(Fan et al\.,[2025](https://arxiv.org/html/2606.17168#bib.bib9)\), RMU\(Li et al\.,[2024](https://arxiv.org/html/2606.17168#bib.bib23)\), and UNDIAL\(Dong et al\.,[2025](https://arxiv.org/html/2606.17168#bib.bib6)\)\(see Section[6](https://arxiv.org/html/2606.17168#S6)\)\.

##### Hyperparameter tuning

All methods, including RepSelect, are tuned with Optuna\(Akiba et al\.,[2019](https://arxiv.org/html/2606.17168#bib.bib1)\)using Tree\-structured Parzen Estimator \(TPE\) sampling over 30 trials, optimizing post\-attack answer probability\. Each trial unlearns for up to 10 epochs, then relearns on the last valid model checkpoint \(where KL on WikiText≤0\.01\\leq 0\.01\)\. Table[4](https://arxiv.org/html/2606.17168#A2.T4)lists the search space for each method and Appendix[B\.1](https://arxiv.org/html/2606.17168#A2.SS1)provides reproducibility details\. For the few\-shot evaluation, tuning uses 5 trials and optimizes answer probability after ak=5k\{=\}5few\-shot attack\.

## 3Why unlearning fails?

We perform a representation analysis to diagnose why three desired properties \(good forgetting, low retain disruption, and attack robustness\) are difficult to satisfy simultaneously\. We find that existing unlearning methods concentrate weight updates on high\-variance forget principal components: directions that also carry substantial retain\-set information and are preferentially recovered by a fine\-tuning adversary\. This makes existing methods both retain\-disruptive and easily reversible\. Full analysis in Appendix[D](https://arxiv.org/html/2606.17168#A4)\.

##### High\-variance forget directions encode shared content with retain

We perform SVD on forget\-set MLP input activations \(Bio\) and find that the top\-10 forget PCs account for 14\.7% of retain\-set MLP activation variance \(10\.7% on Qwen\), and the top\-50 PCs account for 36\.4% \(Llama\) and 30\.5% \(Qwen\) \(Appendix[D\.6](https://arxiv.org/html/2606.17168#A4.SS6)\)\. These directions thus encode shared domain content, not isolated targeted knowledge\. Decomposing by PC tier confirms this: the forget/retain variance ratio decreases monotonically from3\.63\.6–4\.3×4\.3\\times\(top tier\) to1\.31\.3–1\.4×1\.4\\times\(bottom tier\), so retain variance is∼4×\{\\sim\}4\\timesmore concentrated in the top forget PCs \(Figure[4](https://arxiv.org/html/2606.17168#S3.F4)a; Table[16](https://arxiv.org/html/2606.17168#A4.T16)\)\.

Projecting PCs through the frozenlm\_headprovides a direct semantic interpretation: high\-variance PCs on WMDP\-Bio decode to broad domain tokens \(virus, RNA, outbreaks\) or most common words \(the, a, in\) \(Appendix[D\.3](https://arxiv.org/html/2606.17168#A4.SS3)\)\. Low\-variance PCs, on the other hand, activate on niche concepts \(plasmid\-only reverse genetics, RV strain SA11\) \(Appendix[D\.4](https://arxiv.org/html/2606.17168#A4.SS4)\)\. Together, these results motivate suppressing high\-variance forget directions \(which encode general concepts\) and restricting updates to the moreselectivelow\-variance subspace\.

![Refer to caption](https://arxiv.org/html/2606.17168v1/x4.png)Figure 4:Representation structure of forget PCs on Llama\-3\.1\-8B \(WMDP\-Bio, Layer 10\)\.\(a\)Retain and forget variance per PC \(sorted high→\\tolow by forget variance\): retain variance is∼4×\{\\sim\}4\\timeshigher per PC in the top tiers than the bottom tiers, making the top subspace retain\-concentrated\. RepSelect collapses these top directions and operates in the retain\-dilute bottom subspace\.\(b\)Fraction of weight\-update norm \(‖Δ​W‖F2\\\|\\Delta W\\\|\_\{F\}^\{2\}\) concentrated in the top\-50 forget PCs: a fine\-tuning attacker and all baselines place 33–41% in the same retain\-concentrated subspace, while RepSelect avoids it \(∼\{\\sim\}6%\)\. Qwen3\.5\-9B shows identical patterns \(Figure[10](https://arxiv.org/html/2606.17168#A4.F10), Appendix[D\.1](https://arxiv.org/html/2606.17168#A4.SS1)\)\.
##### An attacker recovers high\-variance directions

We simulate a fine\-tuning attacker \(50 SGD steps on forget data\) and find that it concentrates 34% of its weight\-update norm in the top\-50 forget PCs \(Figure[4](https://arxiv.org/html/2606.17168#S3.F4)b; Appendix[D\.9](https://arxiv.org/html/2606.17168#A4.SS9)\)\. Similarly, all four baselines concentrate3333–41%41\\%of their update norm in the same top\-50 PCs \(Figure[4](https://arxiv.org/html/2606.17168#S3.F4)b\), explaining why baseline unlearning is readily reversed by fine\-tuning on the same domain\. Proposition[1](https://arxiv.org/html/2606.17168#Thmproposition1)\(Appendix\) formalizes this: when unlearning only uses activation components outside the top\-kkprincipal subspace, the overlap with the attacker’s update is bounded byϵk=∑i\>kλi/tr​\(Σ\)\\epsilon\_\{k\}=\\sum\_\{i\>k\}\\lambda\_\{i\}/\\mathrm\{tr\}\(\\Sigma\), whereλi\\lambda\_\{i\}are the eigenvalues of the forget\-corpus activation covarianceΣ\\Sigma\. This bound shrinks as more high\-variance directions are collapsed\.

## 4RepSelect

Building on these insights, we propose RepSelect, a simple and efficient unlearning method that collapses the most disruptive activations and output gradients before calculating the unlearning updates\. We provide pseudocode in Algorithm[1](https://arxiv.org/html/2606.17168#alg1)and a PyTorch implementation at[repselect\.py](https://github.com/filyp/open-unlearning/src/trainer/unlearn/repselect_simple.py)

![Refer to caption](https://arxiv.org/html/2606.17168v1/x5.png)Figure 5:RepSelect overview\.For each MLP module, we accumulate the weight gradient∇Wℒ\\nabla\_\{W\}\\mathcal\{L\}on the forget set \(with LoRA active\)\. The top principal components of∇Wℒ\\nabla\_\{W\}\\mathcal\{L\}are softly collapsed, yielding a filtered updateΔ​W′\\Delta W^\{\\prime\}that avoids high\-variance forget directions\.The model is updated asW←W−α⋅Δ​W′W\\leftarrow W\-\\alpha\\cdot\\Delta W^\{\\prime\}\.##### LoRA adversary

Dangerous behavior often only manifests under distribution shift, few\-shot prompting, or fine\-tuning attacks, not by default\. This poses a problem for unlearning: if the model produces no dangerous representations on a given input, the unlearning may target benign representations instead\. To address this, we first*elicit the dangerous behavior*by training a LoRA adapter\(Hu et al\.,[2021](https://arxiv.org/html/2606.17168#bib.bib17)\)for one epoch on the forget set, then compute the unlearning update with LoRA active\. The adapter is discarded after unlearning\. Removing LoRA hurts robustness, especially on the Animal Abuse set \(Figure[6](https://arxiv.org/html/2606.17168#S4.F6), ablation\)\.

##### No retain set needed

We calculate unlearning updates by backpropagating on batches from the forget set, with a negated cross\-entropy loss\. In contrast to baseline unlearning methods \(Section[6](https://arxiv.org/html/2606.17168#S6)\), we do not require a retain set\. As an ablation, we test collapsing the top PCs from the retain rather than forget set \(Section[5\.2](https://arxiv.org/html/2606.17168#S5.SS2.SSS0.Px2)\), which yields weaker unlearning performance, so by default RepSelect*only uses the forget set*\.

##### Finding top PCs

We perform SVD and collapse operations separately for each MLP module\. Our aim is to find and collapse representations that are not specific to the forget set; Section[3](https://arxiv.org/html/2606.17168#S3)shows these concentrate in the top PCs, so we operate in their complement to reduce disruption to benign behavior\.

A naive way to find top PCs is to track the activation covariance, but this is memory\-costly and requires intervening during the forward pass\. Instead, we can perform SVD directly on accumulated*weight gradients*∇Wℒ\\nabla\_\{W\}\\mathcal\{L\}\. Since the weight gradient is the outer product of activations and output gradients, it already contains rich information about the activation distribution \(Appendix[E\.2](https://arxiv.org/html/2606.17168#A5.SS2)\)\. We find explicit covariance tracking provides no performance gains over this efficient approach\.

##### Mahalanobis collapse

Once SVD gives us the distribution statistics, we suppress the most prominent principal components by aligning the activations to their*Mahalanobis direction*: the direction that maximally separates a given vector from a distribution\. Using eigenvaluesλi\\lambda\_\{i\}and eigenvectors𝐯i\\mathbf\{v\}\_\{i\}from SVD, the Mahalanobis direction of𝐚\\mathbf\{a\}is𝐦𝐚𝐡𝐚𝐥​\(𝐚\)∝∑i⟨𝐯i,𝐚⟩/λi⋅𝐯i\\mathbf\{mahal\}\(\\mathbf\{a\}\)\\propto\\sum\_\{i\}\\langle\\mathbf\{v\}\_\{i\},\\mathbf\{a\}\\rangle/\\lambda\_\{i\}\\cdot\\mathbf\{v\}\_\{i\}, reweighting each PC inversely by its variance, with high\-variance directions being damped\. We derive only the topkkPCs \(defaultk=512k\{=\}512\) via low\-rank SVD, avoiding poorly\-estimated low\-variance components, and apply the correction:

𝐚′=𝐚−∑i=1k\(1−λminλi\)​⟨𝐚,𝐯i⟩​𝐯i\.\\mathbf\{a\}^\{\\prime\}=\\mathbf\{a\}\-\\sum\_\{i=1\}^\{k\}\\left\(1\-\\frac\{\\lambda\_\{\\min\}\}\{\\lambda\_\{i\}\}\\right\)\\langle\\mathbf\{a\},\\,\\mathbf\{v\}\_\{i\}\\rangle\\,\\mathbf\{v\}\_\{i\}\.This suppresses the highest\-variance directions \(λi≫λmin\\lambda\_\{i\}\\gg\\lambda\_\{\\min\}\) to near zero while leaving the lowest\-variance direction \(λi=λmin\\lambda\_\{i\}=\\lambda\_\{\\min\}\) and all directions outside the topkksubspace unchanged\.

##### Collapsing both activations and output gradients

Since weight gradients are the outer product of activations and output gradients, SVD on the weight gradients provides top PCs for*both*simultaneously\. For output gradients, these PCs capture directions that backpropagation most commonly identifies as disruptive to model output; we collapse them analogously to the activations\. This “two\-sided” collapse provides additional unlearning gains \(Section[5\.2](https://arxiv.org/html/2606.17168#S5.SS2.SSS0.Px2)\)\. By linearity, instead of collapsing activations and output gradients before computing the weight gradient, we equivalently collapse the*rows and columns*of the weight gradient, avoiding forward/backward hooks during training\. For MoE models, all experts share one SVD: their MLP weights are stacked into a single matrix and collapsed together\.

##### Single\-epoch unlearning

We find in early experiments that our hyperparameter searches consistently favor shorter runs \(1\-2 epochs\)\. Longer runs achieve stronger pre\-attack unlearning but weaker post\-attack robustness, suggesting later updates are less robust than earlier ones\.

We therefore test a single\-pass variant: iterate through the forget set once, accumulate weight gradients without applying them, then perform a single update\. Section[5](https://arxiv.org/html/2606.17168#S5)shows this performs on par with multi\-epoch iterative unlearning, and enables two further speed optimizations: \(1\) Since the full weight gradient is accumulated before any update, SVD and collapse can be computed on the same matrix, rather than on a separate pass\. \(2\) The final unlearning update can be cached and rescaled at several strengths, sweeping the unlearning–disruption trade\-off without rerunning training\. Each RepSelect run therefore completes in 5–15 minutes, versus 3–9 hours per Optuna search for popular baselines \(20–100×\\times× speedup; Appendix[B\.1](https://arxiv.org/html/2606.17168#A2.SS1)\)\.

![Refer to caption](https://arxiv.org/html/2606.17168v1/x6.png)Figure 6:Post\-attack answer probability across methods and tasks\.Lower is better \(↓\\downarrow\)\. RepSelect achieves substantially lower post\-attack answer probability than all five baselines on both WMDP\-Bio and Animal Abuse \(BeaverTails\), across four model families\.*multi\-epoch*and*w/o LoRA*are RepSelect ablations\. Error bars denote standard deviation across top 10 runs\.

## 5RepSelect is robust and non\-disruptive

We demonstrate with extensive experiments that RepSelect is robust to relearning and non\-disruptive to general capabilities, and identify which components contribute the most in ablation studies\. We further verify general capability preservation: RepSelect achieves MMLU accuracy\(Hendrycks et al\.,[2021](https://arxiv.org/html/2606.17168#bib.bib16)\)within 1–2% of the unmodified model across all tested models \(Appendix Table[19](https://arxiv.org/html/2606.17168#A5.T19)\)\.

### 5\.1Main results

##### RepSelect outperforms baselines across all benchmarks and models

Across four model families \(Llama 3\.1 8B, Gemma 4 E4B, DeepSeek V2 Lite, Qwen 3\.5 9B\), dense and MoE architectures, knowledge unlearning \(WMDP\-Bio\), and tendency unlearning \(Animal Abuse\), RepSelect achieves the lowest post\-attack answer probability among all methods \(Figure[6](https://arxiv.org/html/2606.17168#S4.F6)\)\.

Measured by the drop in post\-attack answer probability relative to the original model, RepSelect’s drop is8–50×\\timeslarger than the best baseline \(NPO\) on WMDP\-Bio and4–15×\\timeslarger than the best baseline \(NPO or RMU\) on Animal Abuse\. Even the smaller drops on Animal Abuse compound across answer length: a 10% per\-token drop translates to a\>\>99% drop over a 50\-token answer \(0\.950≈0\.0050\.9^\{50\}\\approx 0\.005\)\. RepSelect is also highly data\-efficient: 10 forget samples already achieve more than half of the maximal unlearning gain on Animal Abuse, and 90 samples saturate it \(Appendix[11](https://arxiv.org/html/2606.17168#A6.F11)\)\.

##### RepSelect unlearns the most and relearns the least

Figure[3](https://arxiv.org/html/2606.17168#S1.F3)\(bottom\) shows unlearning–disruption trade\-offs and relearning robustness on Gemma\-4\-E4B across knowledge \(WMDP\-Bio\) and tendency unlearning \(Animal Abuse\)\. For knowledge unlearning, no baseline matches RepSelect’s selectivity: NPO and RMU barely move answer probability within the disruption budget and both fully recover under relearning\. For tendency unlearning, NPO reaches comparable pre\-attack selectivity but offers no robustness to relearning, while RepSelect remains robust\. The same pattern holds across Llama\-3\.1\-8B, Qwen3\.5\-9B, and DeepSeek\-V2\-Lite \(Appendix[F](https://arxiv.org/html/2606.17168#A6)\)\.

![Refer to caption](https://arxiv.org/html/2606.17168v1/x7.png)Figure 7:Collapse design ablations\.Post\-attack answer probability \(↓\\downarrow\) under variants of RepSelect’s collapse step: SVD source \(*forget*vs*retain*distribution\) crossed with what is collapsed \(*act*ivations, output*grad*ients, or*both*\);*no collapse*is the unintervened gradient\-ascent baseline\. Two\-sided collapse is consistently the best\. For knowledge unlearning, SVD on forget distribution is better than on retain distribution\.

### 5\.2Ablation studies

##### LoRA and single\-epoch ablations

The*multi\-epoch*and*w/o LoRA*rows in Figure[6](https://arxiv.org/html/2606.17168#S4.F6)ablate two design choices from Section[4](https://arxiv.org/html/2606.17168#S4)\. Multi\-epoch unlearning yields no consistent gain over the single\-epoch variant, justifying the simpler approach \(Appendix[A\.1](https://arxiv.org/html/2606.17168#A1.SS1)\)\. Removing LoRA elicitation is neutral on WMDP\-Bio but consistently hurts on Animal Abuse: WMDP\-Bio knowledge is exhibited by default in models so elicitation adds little, whereas harmful tendencies are largely suppressed at baseline \(visible in the base model’s relearning trajectory in Figure[3](https://arxiv.org/html/2606.17168#S1.F3)\) and must be surfaced before unlearning can accurately target them\.

##### Collapse design

RepSelect requires no hyperparameter search beyond the optional LoRA learning rate, so Optuna in Figure[6](https://arxiv.org/html/2606.17168#S4.F6)is used only for fair baseline comparison\. For the following ablations we replace Optuna with a binary search over intervention strength and disable the LoRA adversary, giving cleaner comparisons\. Figure[7](https://arxiv.org/html/2606.17168#S5.F7)reports three findings: Figure[7](https://arxiv.org/html/2606.17168#S5.F7)reports three findings: \(i\) collapse is necessary: the*no collapse*baseline fails to unlearn within the disruption budget; \(ii\) two\-sided collapse helps: collapsing both activations and output gradients consistently outperforms collapsing either alone; \(iii\) the*forget*distribution is a better SVD source than the retain distribution for knowledge unlearning, while the better choice varies by model for tendency unlearning\.

##### Few\-shot robustness

Table[3](https://arxiv.org/html/2606.17168#S5.T3)reports post\-attack accuracy under few\-shot prompting \(k∈\{5,10\}k\\in\\\{5,10\\\}in\-context examples from the relearn split; full setup in Appendix[B\.4](https://arxiv.org/html/2606.17168#A2.SS4)\), with all methods tuned via Optuna \(5 trials\) on thek=5k\{=\}5metric\. RepSelect is nearly immune to few\-shot recovery on both Llama\-3\.1\-8B and Qwen3\.5\-9B: accuracy stays at≤0\.001\{\\leq\}0\.001on WMDP\-Bio and≤0\.013\{\\leq\}0\.013on BeaverTails across bothkkvalues\. Baselines recover substantially more: NPO reaches0\.5440\.544on Llama\-3\.1\-8B Bio \(k=10k\{=\}10; no\-unlearning:0\.5490\.549\) and UNDIAL reaches0\.2020\.202on BeaverTails \(k=10k\{=\}10\), matching the no\-unlearning baseline\. Few\-shot prompting shifts activations at inference time but barely recover knowledge removed from the weight space, confirming that RepSelect’s subspace restriction is robust to this attack\.

Table 3:RepSelect shows the best few\-shot attack robustness across model families\.↓\\downarrowlower is better\. Bio: WMDP\-Bio accuracy at temperature 1\. BeaverTails \(BT\): harmful\-response probability\. “No attack” uses no few\-shot examples; few\-shot rows prependkkin\-context examples from the held\-out relearn split\. All methods tuned with Optuna \(5 trials\) on thek=5k\{=\}5metric\.

## 6Related Work

##### Unlearning methods

Gradient\-based methods modify the training objective: GradDiff\(Liu et al\.,[2022](https://arxiv.org/html/2606.17168#bib.bib24)\)applies gradient ascent on the forget set while minimising retain loss; NPO\(Zhang et al\.,[2024](https://arxiv.org/html/2606.17168#bib.bib40)\)adapts DPO for forget\-only data; SimNPO\(Fan et al\.,[2025](https://arxiv.org/html/2606.17168#bib.bib9)\)drops the reference model via length\-normalized loss\. Representation\-level methods steer or reroute harmful activations: RMU\(Li et al\.,[2024](https://arxiv.org/html/2606.17168#bib.bib23)\)toward random directions, Circuit Breakers\(Zou et al\.,[2024](https://arxiv.org/html/2606.17168#bib.bib42)\)to an orthogonal subspace, UNDIAL\(Dong et al\.,[2025](https://arxiv.org/html/2606.17168#bib.bib6)\)via logit distillation\. Meta\-learning approaches anticipate relearning at training time\(Tamirisa et al\.,[2024](https://arxiv.org/html/2606.17168#bib.bib37); Sondej et al\.,[2025](https://arxiv.org/html/2606.17168#bib.bib36); Henderson et al\.,[2023](https://arxiv.org/html/2606.17168#bib.bib15)\)\.

Separately, several methods constrain weight updates to particular subspaces to protect retain performance: PGU\(Foster et al\.,[2024](https://arxiv.org/html/2606.17168#bib.bib11)\)and K\-FADE\(McKinney et al\.,[2026](https://arxiv.org/html/2606.17168#bib.bib29)\)use Fisher information to identify retain\-relevant directions; KUDA\(Fang et al\.,[2026](https://arxiv.org/html/2606.17168#bib.bib10)\)applies A\-GEM\-style null\-space projection perpendicular to the retain gradient\. All protect retain performance by projecting away from retain\-relevant directions, but do not characterize where the forget or attacker’s subspace lives\. RepSelect takes the complementary view: it uses SVD on the*forget*gradient to identify directions that are simultaneously selective for the forget domain and inaccessible to the attacker, then restricts updates there of both activations and output gradients, distinguishing it from prior methods\.

##### Robustness to relearning attacks

Most unlearning evaluations measure forgetting and retain utility without testing adversarial recovery: TOFU\(Maini et al\.,[2025](https://arxiv.org/html/2606.17168#bib.bib28)\), MUSE\(Shi et al\.,[2024](https://arxiv.org/html/2606.17168#bib.bib34)\), and RMU evaluation\(Li et al\.,[2024](https://arxiv.org/html/2606.17168#bib.bib23)\)all omit relearning attacks\.Deeb and Roger \([2024](https://arxiv.org/html/2606.17168#bib.bib4)\)shows that most unlearned models can be fully restored by brief fine\-tuning on a*completely independent*fact set from the same domain;Tamirisa et al\. \([2024](https://arxiv.org/html/2606.17168#bib.bib37)\)andLynch et al\. \([2024](https://arxiv.org/html/2606.17168#bib.bib26)\)reinforce this across few\-shot prompting and out\-of\-distribution inputs\(Łucki et al\.,[2025](https://arxiv.org/html/2606.17168#bib.bib43)\)\. We adopt the fine\-tuning and few\-shot prompting attacks as the standard primary robustness tests\.

## 7Conclusion

Making LLM deeply forget targeted knowledge while maintaining general capabilities remains challenging\. Existing unlearning methods forget only shallowly: they target high\-variance representations that are shared with benign retain data and the attack space of full fine\-tuning, which both disrupts general capability and leaves unlearning easily reversible\. RepSelect addresses this by using SVD on the forget set to identify these shared directions and restrict weight updates to their complement, the forget\-specific low\-variance subspace\. It requires no retain set, minimal tuning, and a single pass over the forget set\. Across four model families on biohazardous knowledge and abusive tendencies, RepSelect achieves a44–50×50\\timeslarger reduction in post\-relearning accuracy than the strongest baselines and is near\-perfectly robust to few\-shot attacks, while matching general capability\. This demonstrates that representation selectivity is an essential step toward truly deep forgetting in LLMs\.

##### Limitations

RepSelect operates on MLP modules only, where conceptual understanding is formed; future work could test whether the collapse transfers to key/value projections in attention heads\. We evaluate standard fine\-tuning and few\-shot attacks; we leave it to future work to explore other attacks\. We successfully unlearn facts from WMDP\-Bio and behaviours from BeaverTails\-abuse; future work should test how RepSelect scales to larger forget sets, which will be needed for full biosafety and wider removal of harmful tendencies\.

## Acknowledgements

F\.S\. was funded by a grant from Coefficient Giving\. Compute was supported by the Polish high\-performance computing infrastructure PLGrid \(HPC Center: ACK Cyfronet AGH\) within computational grant no\. PLG/2025/018339\. We thank Maxime Riché, Alex Cloud, Alex Infanger, Fabien Roger, Stephen Casper, Kay Kozaronek, and Artyom Karpov for valuable discussions and feedback\.

## Author Contributions

F\.S\. conceived the project, developed the RepSelect method, and ran fine\-tuning robustness and ablation experiments\. Y\.Y\. developed the representation analysis underpinning RepSelect’s design, ran few\-shot robustness and capability evaluations, and led paper writing\. A\.M\. advised on the project and contributed to manuscript revisions\.

## References

- Akiba et al\. \[2019\]Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama\.Optuna: A Next\-generation Hyperparameter Optimization Framework, July 2019\.arXiv:1907\.10902\.
- Carlini et al\. \[2021\]Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert\-Voss, and et al\.Extracting training data from large language models, 2021\.arXiv: 2012\.07805\.
- Chaudhry et al\. \[2019\]Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny\.Efficient Lifelong Learning with A\-GEM, January 2019\.arXiv:1812\.00420 \[cs\]\.
- Deeb and Roger \[2024\]Aghyad Deeb and Fabien Roger\.Do Unlearning Methods Remove Information from Language Model Weights?, November 2024\.arXiv:2410\.08827\.
- DeepSeek\-AI et al\. \[2024\]DeepSeek\-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, et al\.DeepSeek\-V2: A strong, economical, and efficient mixture\-of\-experts language model, May 2024\.arXiv:2405\.04434\.
- Dong et al\. \[2025\]Yijiang River Dong, Hongzhou Lin, Mikhail Belkin, Ramon Huerta, and Ivan Vulić\.UNDIAL: Self\-distillation with adjusted logits for robust unlearning in large language models\.In*Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\)*\. Association for Computational Linguistics, April 2025\.
- Dorna et al\. \[2025\]Vineeth Dorna, Anmol Mekala, Wenlong Zhao, Andrew McCallum, Zachary C\. Lipton, J\. Zico Kolter, and Pratyush Maini\.Openunlearning: Accelerating llm unlearning via unified benchmarking of methods and metrics, 2025\.arXiv: 2506\.12618\.
- European Parliament and Council of the European Union \[2016\]European Parliament and Council of the European Union\.Art\. 17 gdpr – right to erasure \(‘right to be forgotten’\)\.[https://gdpr\-info\.eu/art\-17\-gdpr/](https://gdpr-info.eu/art-17-gdpr/), 2016\.Regulation \(EU\) 2016/679, OJ L 119\. Accessed 2026\-06\-15\.
- Fan et al\. \[2025\]Chongyu Fan, Jiancheng Liu, Licong Lin, Jinghan Jia, Ruiqi Zhang, et al\.Simplicity prevails: Rethinking negative preference optimization for llm unlearning, 2025\.arXiv: 2410\.07163\.
- Fang et al\. \[2026\]Ce Fang, Zhikun Zhang, Min Chen, Qing Liu, Lu Zhou, Zhe Liu, and Yunjun Gao\.Kuda: Knowledge unlearning by deviating representation for large language models, 2026\.arXiv: 2602\.19275\.
- Foster et al\. \[2024\]Jack Foster, Stefan Schoepf, and Alexandra Brintrup\.Fast machine unlearning without retraining through selective synaptic dampening, 2024\.arXiv: 2308\.07707\.
- Gemma Team \[2026\]Gemma Team\.Gemma 4\.[https://huggingface\.co/google/gemma\-4\-E4B](https://huggingface.co/google/gemma-4-E4B), April 2026\.HuggingFace: google/gemma\-4\-E4B\.
- Geva et al\. \[2022\]Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg\.Transformer feed\-forward layers build predictions by promoting concepts in the vocabulary space, 2022\.arXiv: 2203\.14680\.
- Grattafiori et al\. \[2024\]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, and et al\.The llama 3 herd of models, 2024\.arXiv:2407\.21783\.
- Henderson et al\. \[2023\]Peter Henderson, Eric Mitchell, Christopher D\. Manning, Dan Jurafsky, and Chelsea Finn\.Self\-Destructing Models: Increasing the Costs of Harmful Dual Uses of Foundation Models, August 2023\.arXiv:2211\.14946 \[cs\]\.
- Hendrycks et al\. \[2021\]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, and et al\.Measuring massive multitask language understanding, 2021\.arXiv: 2009\.03300\.
- Hu et al\. \[2021\]Edward J\. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen\-Zhu, Yuanzhi Li, et al\.Lora: Low\-rank adaptation of large language models, 2021\.arXiv:2106\.09685\.
- Ji et al\. \[2023\]Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, et al\.BeaverTails: Towards Improved Safety Alignment of LLM via a Human\-Preference Dataset, November 2023\.arXiv:2307\.04657\.
- Kapoor et al\. \[2024\]Sayash Kapoor, Rishi Bommasani, Kevin Klyman, Shayne Longpre, Ashwin Ramaswami, and et al\.On the societal impact of open foundation models, 2024\.arXiv: 2403\.07918\.
- Karamolegkou et al\. \[2023\]Antonia Karamolegkou, Jiaang Li, Li Zhou, and Anders Søgaard\.Copyright violations and large language models\.In Houda Bouamor, Juan Pino, and Kalika Bali, editors,*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 7403–7412, Singapore, December 2023\. Association for Computational Linguistics\.doi:10\.18653/v1/2023\.emnlp\-main\.458\.
- Lee et al\. \[2024\]Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K\. Kummerfeld, et al\.A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity, January 2024\.arXiv:2401\.01967 \[cs\]\.
- Lermen et al\. \[2024\]Simon Lermen, Charlie Rogers\-Smith, and Jeffrey Ladish\.Lora fine\-tuning efficiently undoes safety training in llama 2\-chat 70b, 2024\.arXiv:2310\.20624\.
- Li et al\. \[2024\]Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, et al\.The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning, May 2024\.arXiv:2403\.03218 \[cs\]\.
- Liu et al\. \[2022\]Bo Liu, Qiang Liu, and Peter Stone\.Continual learning and private unlearning, 2022\.arXiv:2203\.12817\.
- Liu et al\. \[2024\]Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, et al\.Rethinking Machine Unlearning for Large Language Models, July 2024\.arXiv:2402\.08787 \[cs\]\.
- Lynch et al\. \[2024\]Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, and Dylan Hadfield\-Menell\.Eight Methods to Evaluate Robust Unlearning in LLMs, February 2024\.arXiv:2402\.16835 \[cs\]\.
- M\-A\-P et al\. \[2024\]M\-A\-P, Ge Zhang, Xinrun Du, Zhimiao Yu, Zili Wang, et al\.Finefineweb: A comprehensive study on fine\-grained domain web corpus, December 2024\.
- Maini et al\. \[2025\]Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C\. Lipton, and J\. Zico Kolter\.TOFU: A task of fictitious unlearning for LLMs\.In*Proceedings of the International Conference on Learning Representations \(ICLR\)*, 2025\.
- McKinney et al\. \[2026\]Lev McKinney, Anvith Thudi, Juhan Bae, Tara Rezaei, Nicolas Papernot, Sheila A\. McIlraith, and Roger Grosse\.Gauss\-newton unlearning for the LLM era, 2026\.arXiv:2602\.10568\.
- Meng et al\. \[2023\]Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov\.Locating and Editing Factual Associations in GPT, January 2023\.arXiv:2202\.05262 \[cs\]\.
- Merity et al\. \[2016\]Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher\.Pointer sentinel mixture models, 2016\.arXiv:1609\.07843\.
- Parrish et al\. \[2022\]Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, and et al\.Bbq: A hand\-built bias benchmark for question answering, 2022\.arXiv: 2110\.08193\.
- Qi et al\. \[2023\]Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin\-Yu Chen, Ruoxi Jia, et al\.Fine\-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To\!, October 2023\.arXiv:2310\.03693 \[cs\]\.
- Shi et al\. \[2024\]Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, and et al\.Muse: Machine unlearning six\-way evaluation for language models, 2024\.arXiv:2407\.06460\.
- Shumailov et al\. \[2024\]Ilia Shumailov, Jamie Hayes, Eleni Triantafillou, Guillermo Ortiz\-Jimenez, Nicolas Papernot, and et al\.Ununlearning: Unlearning is not sufficient for content regulation in advanced generative ai, 2024\.arXiv:2407\.00106\.
- Sondej et al\. \[2025\]Filip Sondej, Yushi Yang, Mikołaj Kniejski, and Marcel Windys\.Robust LLM Unlearning with MUDMAN: Meta\-Unlearning with Disruption Masking And Normalization, June 2025\.arXiv:2506\.12484 \[cs\]\.
- Tamirisa et al\. \[2024\]Rishub Tamirisa, Bhrugu Bharathi, Long Phan, Andy Zhou, Alice Gatti, et al\.Tamper\-Resistant Safeguards for Open\-Weight LLMs, August 2024\.arXiv:2408\.00761 \[cs\]\.
- Yang et al\. \[2025a\]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, and et al\.Qwen3 technical report, 2025a\.arXiv:2505\.09388\.
- Yang et al\. \[2025b\]Yushi Yang, Filip Sondej, Harry Mayne, Andrew Lee, and Adam Mahdi\.How does DPO reduce toxicity? A mechanistic neuron\-level analysis\.In*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*\. Association for Computational Linguistics, November 2025b\.
- Zhang et al\. \[2024\]Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei\.Negative preference optimization: From catastrophic collapse to effective unlearning, 2024\.arXiv:2404\.05868\.
- Zhou et al\. \[2026\]Duo Zhou, Yuji Zhang, Tianxin Wei, Ruizhong Qiu, Ke Yang, and et al\.Geometric\-disentangelment unlearning, 2026\.arXiv:2511\.17100\.
- Zou et al\. \[2024\]Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, et al\.Improving Alignment and Robustness with Circuit Breakers, July 2024\.arXiv:2406\.04313 \[cs\]\.
- Łucki et al\. \[2025\]Jakub Łucki, Boyi Wei, Yangsibo Huang, Peter Henderson, Florian Tramèr, et al\.An Adversarial Perspective on Machine Unlearning for AI Safety, January 2025\.arXiv:2409\.18025 \[cs\]\.

## Appendix

[AMore on RepSelect Algorithm and Implementation](https://arxiv.org/html/2606.17168#A1)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[A](https://arxiv.org/html/2606.17168#A1) [A\.1Multi\-epoch variants](https://arxiv.org/html/2606.17168#A1.SS1)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[A\.1](https://arxiv.org/html/2606.17168#A1.SS1) [BMore on Experiment Setup](https://arxiv.org/html/2606.17168#A2)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[B](https://arxiv.org/html/2606.17168#A2) [B\.1Reproducibility and Compute Requirements](https://arxiv.org/html/2606.17168#A2.SS1)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[B\.1](https://arxiv.org/html/2606.17168#A2.SS1) [B\.2Hyperparameter Search Spaces](https://arxiv.org/html/2606.17168#A2.SS2)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[B\.2](https://arxiv.org/html/2606.17168#A2.SS2) [B\.3Dataset Creation](https://arxiv.org/html/2606.17168#A2.SS3)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[B\.3](https://arxiv.org/html/2606.17168#A2.SS3) [B\.4Few\-Shot Attack Details](https://arxiv.org/html/2606.17168#A2.SS4)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[B\.4](https://arxiv.org/html/2606.17168#A2.SS4) [CMore Motivation for Selectivity](https://arxiv.org/html/2606.17168#A3)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[C](https://arxiv.org/html/2606.17168#A3) [C\.1Unrelated Facts Disruption and Language Transfer](https://arxiv.org/html/2606.17168#A3.SS1)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[C\.1](https://arxiv.org/html/2606.17168#A3.SS1) [C\.2Filtering Out Disruption: Weight Space vs\. Activation Space](https://arxiv.org/html/2606.17168#A3.SS2)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[C\.2](https://arxiv.org/html/2606.17168#A3.SS2) [DMore on Representation Analysis](https://arxiv.org/html/2606.17168#A4)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[D](https://arxiv.org/html/2606.17168#A4) [D\.1Model Result: Qwen3\.5\-9B](https://arxiv.org/html/2606.17168#A4.SS1)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[D\.1](https://arxiv.org/html/2606.17168#A4.SS1) [D\.2PCA Selectivity and Attacker Concentration](https://arxiv.org/html/2606.17168#A4.SS2)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[D\.2](https://arxiv.org/html/2606.17168#A4.SS2) [D\.3Vocabulary Projection of PCs](https://arxiv.org/html/2606.17168#A4.SS3)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[D\.3](https://arxiv.org/html/2606.17168#A4.SS3) [D\.4Top Forget Sequences per PC](https://arxiv.org/html/2606.17168#A4.SS4)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[D\.4](https://arxiv.org/html/2606.17168#A4.SS4) [D\.5Steering Vector Alignment](https://arxiv.org/html/2606.17168#A4.SS5)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[D\.5](https://arxiv.org/html/2606.17168#A4.SS5) [D\.6Cross\-Distribution PC Variance](https://arxiv.org/html/2606.17168#A4.SS6)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[D\.6](https://arxiv.org/html/2606.17168#A4.SS6) [D\.7Tiered Selectivity Along PCA Directions](https://arxiv.org/html/2606.17168#A4.SS7)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[D\.7](https://arxiv.org/html/2606.17168#A4.SS7) [D\.8Baseline Weight Projection](https://arxiv.org/html/2606.17168#A4.SS8)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[D\.8](https://arxiv.org/html/2606.17168#A4.SS8) [D\.9Attack Subspace Concentration](https://arxiv.org/html/2606.17168#A4.SS9)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[D\.9](https://arxiv.org/html/2606.17168#A4.SS9) [EMore on Disruption and Robustness Analysis](https://arxiv.org/html/2606.17168#A5)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[E](https://arxiv.org/html/2606.17168#A5) [E\.1MMLU Accuracy](https://arxiv.org/html/2606.17168#A5.SS1)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[E\.1](https://arxiv.org/html/2606.17168#A5.SS1) [E\.2The Purified Weight Update](https://arxiv.org/html/2606.17168#A5.SS2)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[E\.2](https://arxiv.org/html/2606.17168#A5.SS2) [E\.3Robustness and Disruption Guarantees](https://arxiv.org/html/2606.17168#A5.SS3)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[E\.3](https://arxiv.org/html/2606.17168#A5.SS3) [FMore Unlearning and Relearning Trajectories](https://arxiv.org/html/2606.17168#A6)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[F](https://arxiv.org/html/2606.17168#A6)

## Appendix AMore on RepSelect Algorithm and Implementation

In this section, we provide the RepSelect algorithm and implementation details\. We give pseudocode for the core single\-epoch variant and examine multi\-epoch extensions, explaining why the simpler single\-epoch default is preferred\.

Algorithm[1](https://arxiv.org/html/2606.17168#alg1)collapses based on activation and gradient distribution of the forget set\. To use distribution of the retain set, we can analogously pass through retain set once, accumulating weight gradients, and then doing the SVD step on this weight gradietn instead\.

Algorithm 1RepSelect: Collapse of Irrelevant Components of the Weight GradientInput:Modelθ\\thetawith MLP weights\{Wm\}\\\{W\_\{m\}\\\}\(gate/up/down projections\); forget set𝒟forget\\mathcal\{D\}\_\{\\text\{forget\}\}; unlearning strengthα\\alpha; LoRA learning rateηlora\\eta\_\{\\text\{lora\}\}; number of principal componentskk\. Initialise:LoRA adapters\{\(𝐀m,𝐁m\)\}\\\{\(\\mathbf\{A\}\_\{m\},\\mathbf\{B\}\_\{m\}\)\\\}on eachWmW\_\{m\}\.

1:LoRA adversarial pretraining: one epoch, SGD descent on forget NLL

2:for

xf∼𝒟forgetx\_\{f\}\\sim\\mathcal\{D\}\_\{\\text\{forget\}\}do

3:

\(𝐀m,𝐁m\)←\(𝐀m,𝐁m\)−ηlora​∇𝐀m,𝐁mℒNLL​\(θ,xf\)\(\\mathbf\{A\}\_\{m\},\\mathbf\{B\}\_\{m\}\)\\leftarrow\(\\mathbf\{A\}\_\{m\},\\mathbf\{B\}\_\{m\}\)\-\\eta\_\{\\text\{lora\}\}\\,\\nabla\_\{\\mathbf\{A\}\_\{m\},\\mathbf\{B\}\_\{m\}\}\\,\\mathcal\{L\}\_\{\\text\{NLL\}\}\(\\theta,x\_\{f\}\)
4:endfor

5:Accumulate forget weight\-gradient with LoRA active in forward

6:

Gm←∑xf∼𝒟forget∇Wm\(−ℒNLL​\(θ\+LoRA,xf\)\)G\_\{m\}\\leftarrow\\sum\_\{x\_\{f\}\\sim\\mathcal\{D\}\_\{\\text\{forget\}\}\}\\nabla\_\{W\_\{m\}\}\\,\(\-\\mathcal\{L\}\_\{\\text\{NLL\}\}\(\\theta\+\\text\{LoRA\},x\_\{f\}\)\)∀m\\forall m
7:Unload LoRA from

θ\\theta
8:foreach module

mmdo

9:

\(Um,Sm,Vm\)←SVDk​\(Gm\)\(U\_\{m\},S\_\{m\},V\_\{m\}\)\\leftarrow\\text\{SVD\}\_\{k\}\(G\_\{m\}\)compute SVD of the weight gradient

10:

Gm←collapse​\(Gm,Vm,Sm\)G\_\{m\}\\leftarrow\\text\{collapse\}\(G\_\{m\},\\,V\_\{m\},\\,S\_\{m\}\)soft\-collapse input \(DinD\_\{\\text\{in\}\}\) side

11:

Gm←collapse​\(Gm⊤,Um,Sm\)⊤G\_\{m\}\\leftarrow\\text\{collapse\}\(G\_\{m\}^\{\\top\},\\,U\_\{m\},\\,S\_\{m\}\)^\{\\top\}soft\-collapse output \(DoutD\_\{\\text\{out\}\}\) side

12:

Wm←Wm−α​GmW\_\{m\}\\leftarrow W\_\{m\}\-\\alpha\\,G\_\{m\}single filtered\-gradient step

13:endfor

collapse\(M,E,S\)\(M,E,S\):letP=M​EP=MEandS~=S/min⁡\(S\)\\tilde\{S\}=S/\\min\(S\); returnM−\(P−P/S~\)​E⊤M\-\(P\-P/\\tilde\{S\}\)E^\{\\top\}\(Mahalanobis rescaling, Eq\.[4](https://arxiv.org/html/2606.17168#S4.Ex1)\)\.

### A\.1Multi\-epoch variants

The*multi\-epoch*row in Figure[6](https://arxiv.org/html/2606.17168#S4.F6)reports the best multi\-epoch variant we found\. The naive multi\-epoch version, which uses negative cross\-entropy loss, is unstable and has a poor unlearning–disruption trajectory\. Replacing the negative cross\-entropy with an NPO loss\[Zhang et al\.,[2024](https://arxiv.org/html/2606.17168#bib.bib40)\]stabilises training and is the variant we report\. Even so, it offers no gains over the single\-epoch default \(which uses negative cross\-entropy\), justifying the simpler choice in Section[4](https://arxiv.org/html/2606.17168#S4)\.

## Appendix BMore on Experiment Setup

In this section, we provide full details of the experimental setup\. We describe hardware and compute requirements, hyperparameter search spaces for all methods, dataset construction for WMDP\-Bio and BeaverTails, and the few\-shot attack details\.

### B\.1Reproducibility and Compute Requirements

##### Hardware\.

By default we use a single NVIDIA RTX PRO 6000 \(96GB\)\. The exception is our biggest model, DeepSeek\-V2\-Lite: RepSelect runs use an H200 \(141GB\), while the baselines are more memory\-costly and require a B200 \(180GB\)\.

##### Main grid\.

The main grid \(Figure[6](https://arxiv.org/html/2606.17168#S4.F6)\) covers 4 models×\\times2 benchmarks×\\times8 methods\. Each \(method, model, benchmark\) cell is a 30\-trial Optuna search over the hyperparameter ranges in Table[4](https://arxiv.org/html/2606.17168#A2.T4)\. A single search takes between 3 and 9 hours of wall\-clock time, with the upper end driven by larger models \(DeepSeek\-V2\-Lite, Qwen3\.5\-9B\)\. In total, the full comparison with baselines consumed approximately 4 models×\\times2 benchmarks×\\times8 methods×\\times5 hours = 320 GPU\-hours\.

##### Collapse\-design ablations\.

The ablations in Figure[7](https://arxiv.org/html/2606.17168#S5.F7)do not require hyperparameter search\. After deriving the unlearning update, the intervention strength is found by a binary search\. Each run completes in 5–15 minutes, most of which is spent on the relearning attack\. For future unlearning work, we recommend this fast research loop, over the heavy Optuna searches required for baselines\.

##### Reproducing the experiments\.

### B\.2Hyperparameter Search Spaces

All methods are tuned with Optuna TPE sampling \(30 trials, seeded for determinism\)\. Each trial runs the full unlearn–relearn pipeline\. The optimisation target isholdout\_harmful\_prob\(minimize\), subject to a WikiText KL disruption budget of0\.010\.01\. Table[4](https://arxiv.org/html/2606.17168#A2.T4)lists the search space for each method\.

Table 4:Hyperparameter search spaces\.All learning rates use log\-uniform sampling\. RepSelect uses SGD; all baselines use AdamW \(adamw\_8bit\)\.MethodHyperparameterRangeScaleRepSelect \(dense\)Learning rate\[5×10−3,0\.5\]\[5\{\\times\}10^\{\-3\},\\,0\.5\]logLoRA adversary LR\[5×10−3,0\.5\]\[5\{\\times\}10^\{\-3\},\\,0\.5\]logRepSelect \(MoE\)Learning rate\[0\.5,50\]\[0\.5,\\,50\]logLoRA adversary LR\[0\.5,50\]\[0\.5,\\,50\]logGradDiffLearning rate\[10−7,3×10−5\]\[10^\{\-7\},\\,3\{\\times\}10^\{\-5\}\]logRetain weightα\\alpha\[1,10\]\[1,\\,10\]logNPOLearning rate\[3×10−7,5×10−5\]\[3\{\\times\}10^\{\-7\},\\,5\{\\times\}10^\{\-5\}\]logReference weightα\\alpha\[1,5\]\[1,\\,5\]logKL coefficientβ\\beta\[0\.05,0\.5\]\[0\.05,\\,0\.5\]logSimNPOLearning rate\[10−7,5×10−6\]\[10^\{\-7\},\\,5\{\\times\}10^\{\-6\}\]logβ\\beta\[3\.5,4\.5\]\[3\.5,\\,4\.5\]linearδ\\delta\[0,1\]\[0,\\,1\]linearγ\\gamma\[0\.125,0\.25\]\[0\.125,\\,0\.25\]linearRMULearning rate\[10−7,3×10−5\]\[10^\{\-7\},\\,3\{\\times\}10^\{\-5\}\]logSteering coefficient\[10−3,102\]\[10^\{\-3\},\\,10^\{2\}\]logTarget layer\{6,11,16\}\\\{6,11,16\\\}categoricalUNDIALLearning rate\[10−7,3×10−5\]\[10^\{\-7\},\\,3\{\\times\}10^\{\-5\}\]logα\\alpha\[1,5\]\[1,\\,5\]logβ\\beta\[3,30\]\[3,\\,30\]logFixed hyperparameters for RepSelect:k=512k\{=\}512principal components,distribution=forget,collapse\_on=both\(bothDinD\_\{\\text\{in\}\}andDoutD\_\{\\text\{out\}\}sides\), soft \(Mahalanobis\-style\) collapse, LoRA adversary on MLP gate/up/down projections \(default PEFT rank\), SGD optimizer\. The MoE\-specific LR range applies to DeepSeek\-V2\-Lite and Qwen3\-30B\-A3B, where the relevantgate\_up\_projparameters require3030–100×100\\timeslarger SGD step sizes than dense models\. All methods train witheval\_strategy=epochand trials are aborted once the WikiText KL disruption exceeds0\.010\.01\.

### B\.3Dataset creation

#### B\.3\.1WMDP unlearning corpus creation

Table[5](https://arxiv.org/html/2606.17168#A2.T5)shows a representative example from WMDP together with the forget corpus we generate from it\.

Table 5:Example WMDP question and generated forget corpus, compared against the prior corpus fromDeeb and Roger \[[2024](https://arxiv.org/html/2606.17168#bib.bib4)\]\. Our paraphrases retain only the essential harmful fact, while the prior corpus pads each paraphrase with non\-harmful context, which \(we find\) increases disruption from unlearning\.##### Filtering

We start from WMDP\-Deduped\[Deeb and Roger,[2024](https://arxiv.org/html/2606.17168#bib.bib4)\], which removed skill\-based questions and duplicates from the full WMDP\-Bio benchmark\. For faithful answer\-recall evaluation, we want answers that can be cleanly separated from non\-harmful context, but many WMDP answers are long or contain mostly benign tokens\. We therefore keep only questions with answers shorter than 60 characters, and exclude “none of the above” / “all of the above” answers, which produce awkward generated paraphrases\. This leaves 189 biology questions, which we provide in our repository\. The full filtering pipeline is in[data\_transformation\.py](https://github.com/filyp/open-unlearning/src/data_processing/data_transformation.py)\.

##### Generation

For each question we generate 20 simple sentences usinggpt\-4\.1, paraphrasing the tested fact\. In the final corpus we use only 3 sentences per question, because using more actually hurts unlearning, probably because the first sentences are higher quality\.

Generation prompt asks for simplicity and not adding unnecessary text\. As Table[5](https://arxiv.org/html/2606.17168#A2.T5)shows, our corpus produces concise sentences that paraphrase only the essential harmful fact\. We saw that this*avoiding of unnecessary text*greatly reduces disruption from unlearning\. We invite future designers of unlearning corpora to also include only essentially harmful text\.

The full generated corpus can be found in our repository in: data/wmdp\_deduped\_\[bio\|cyber\]/\{split\_name\}\_corpus\_simple\.jsonl\.

#### B\.3\.2BeaverTails contrast set creation

To more accurately prevent disruption when unlearning on BeaverTails, we generate a synthetic retain set that closely mirrors each forget example while inverting only the harmful concept\.

Concretely, for every\(prompt, response\)pair in the BeaverTailsanimal\_abusesplit, we ask Claude Sonnet 4\.6 to produce a benign\(prompt, response\)pair that maximizes lexical, syntactic and stylistic overlap with the original, changing only what is necessary to make the content harmless\. The system prompt instructs the model to \(i\) preserve sentence count, clause structure, register, hedging language and discourse markers verbatim, \(ii\) prefer phonologically or morphologically similar substitutions \(e\.g\.*torture*→\\to*nurture*,*poison*→\\to*remedy*,*mutilate*→\\to*groom*\), and \(iii\) propagate the substitution coherently through the rest of the response \(e\.g\.*knife*→\\to*brush*,*cutting*→\\to*brushing*\)\. The full pipeline is implemented in[contrast\_set\_creation\.py](https://github.com/filyp/open-unlearning/community/benchmarks/beavertails/contrast_set_creation.py)and the resulting dataset is published asfilypo/beavertails\-contrast\.

We also tried using the safe\(prompt, response\)pairs already provided in BeaverTails as the retain set, and it performed similarly well\. We still release and use the contrast set in case it proves useful for future methods\.

Table[6](https://arxiv.org/html/2606.17168#A2.T6)shows three representative pairs\.

Table 6:Warning: texts contain graphic details\.Examples of harmful BeaverTails pairs \(left\) and their benign contrasts \(right\)\.Redmarks the harmful concept in the original;greenmarks the substituted benign counterpart in the contrast\. Responses are truncated to the first 1–2 sentences\.

### B\.4Few\-Shot Attack Details

##### Setup\.

The few\-shot attack simulates an adversary who has access tokkdomain\-relevant examples but cannot modify the model weights\. For each evaluation run,kkin\-context demonstrations are sampled uniformly at random \(fixed seed\) from the*relearn split*, which is disjoint from both the unlearning corpus and the evaluation set \(Table[1](https://arxiv.org/html/2606.17168#S2.T1)\)\. The samekkdemonstrations are prepended to every evaluation prompt within a run\. We evaluate withk∈\{5,10\}k\\in\\\{5,10\\\}; Table[3](https://arxiv.org/html/2606.17168#S5.T3)reports results for Llama\-3\.2\-3B on WMDP\-Bio\.

##### Demonstration format

For WMDP, each demonstration is a multiple\-choice question with the correct answer revealed, formatted asQ: <question\>\\nA: \(<letter\>\) <answer\>\. The model is evaluated on held\-out questions from the same domain\. For BeaverTails, each demonstration is a harmful prompt–response pair\. Base models receive demonstrations concatenated as plain text \(double newline separator\); instruction\-tuned models receive them as multi\-turn chat using the model’s chat template\.

##### Metric

For both benchmarks we report the mean per\-sequence*normalized answer probability*\(prob\): the geometric mean of the model’s token probabilities along the gold answer sequence, averaged over the held\-out evaluation set \(ProbabilityEvaluatorinsrc/evals/loss\_eval\.py\)\. This metric is bounded in\[0,1\]\[0,1\], so a single sequence with very high loss cannot mask others that remain memorised, a known failure mode of mean\-loss reporting for unlearning\. For WMDP we evaluate on therecallsplit \(the gold continuation of each MCQ\), and for BeaverTails on theholdout\_harmfulsplit \(held\-out harmful prompt–response pairs in the target category\)\. In both cases lower is better, and the unlearning target is a model whose answer probability on the held\-out forget set is driven down without raising WikiText KL beyond the disruption budget\.

##### Few\-shot demonstration examples

Tables[7](https://arxiv.org/html/2606.17168#A2.T7)and[8](https://arxiv.org/html/2606.17168#A2.T8)show representative demonstrations used in our experiments\.

Table 7:Few\-shot demonstrations for WMDP\-Bio\.Sampled from the relearn split \(held\-out MCQs disjoint from unlearning and evaluation\)\. The correct answer is provided to the model as part of each demonstration\.Table 8:Few\-shot demonstrations for BeaverTails \(animal\_abuse\)\.Sampled from the relearn split\. Each demonstration pairs a harmful prompt with the model’s harmful response, teaching the model the expected harmful behaviour via in\-context learning\.

## Appendix CMore Motivation for Selectivity

In this section, we provide additional evidence motivating selective targeting of low\-variance forget\-corpus directions\. We show examples of unlearning gradients bleed into unrelated facts due to shared representations with targeted facts, and compare weight\-space versus activation\-space approaches to filtering retain disruption\.

### C\.1Unrelated Facts Disruption and Language Transfer

![Refer to caption](https://arxiv.org/html/2606.17168v1/x8.png)

![Refer to caption](https://arxiv.org/html/2606.17168v1/x9.png)

Figure 8:Gradient overlap between superficially similar facts\.We compute the weight gradient for unlearning “The capital of France isParis” \(negated cross\-entropy on the answer tokens\) and measure its cosine similarity with the gradient for each other fact\. No unlearning training is performed; this is a single forward\-backward pass showing how much the*gradient directions*overlap\.*Activations*and*Gradients*columns show a slice of the first 40 elements in activations and output gradient vectors, at an MLPgate\_projmodule in a middle layer \(green = positive values, red = negative\)\. The near\-identical patterns across some facts illustrate that most of their representation is shared, not fact\-specific\. Model:Llama\-3\.2\-1B\.The 84% transfer between facts about capitals \(top of Figure[8](https://arxiv.org/html/2606.17168#A3.F8)\) raises a question: which features of the prompt drive the overlap? The bottom panel probes this with translations and a different relation type\. Translations of the original fact transfer significantly \(∼\\sim50%\) only for languages with similar surface tokens \(German “ist”, Spanish “es”\); for Russian and Portuguese the transfer is weak, which would require unlearning in each language separately\. This is consistent with findings in the ROME technique\[Meng et al\.,[2023](https://arxiv.org/html/2606.17168#bib.bib30)\]\) that can be specific to the exact tokens used \(e\.g\. unlearning “cheese” does not transfer to “fromage”\)\. The “water contains hydrogen”\-style facts overlap only 7–10%, showing that some shared structure persists across relation types but at much smaller magnitude than within the same template\. To reproduce the plots, use[this script](https://github.com/filyp/open-unlearning/src/plotting/1_capitals.py)\.

### C\.2Filtering Out Disruption: Weight Space vs\. Activation Space

A natural thing to try if we want to be selective is to limit which weights are updated\. For example,Sondej et al\. \[[2025](https://arxiv.org/html/2606.17168#bib.bib36)\]showed unlearning improvements when allowing to modify only the weights where the signs of the unlearning and the retaining update are the same\. Similarly, the A\-GEM technique\[Chaudhry et al\.,[2019](https://arxiv.org/html/2606.17168#bib.bib3)\]projects the weight updates to be orthogonal to the retaining updates to avoid performance disruption\. Such projections have also been successfully used for unlearning\[Zhou et al\.,[2026](https://arxiv.org/html/2606.17168#bib.bib41)\]\.

In Figure[9](https://arxiv.org/html/2606.17168#A3.F9), the*masked per weight*row shows the effect of these filtering techniques\. They significantly reduce the disruption \(red\), but some of it still escapes the filtering\. That is because the control/retaining updates we use to decide which weights to filter out never match the actual disruption perfectly\. \(Compare the blue control pattern and the red disruption pattern\.\)

![Refer to caption](https://arxiv.org/html/2606.17168v1/x10.png)Figure 9:Comparison of two masking strategies\.We show a slice of updates of a single weight matrix when unlearning “The capital of France isParis"\. Weights are colored green when an update successfully unlearns a paraphrased fact \("France’s capital isParis"\), red when it disrupts recall of a different fact \(“The capital of Spain isMadrid"\), and blue for a control fact disruption \(“The capital of Italy isRome"\)\. Then we use the control fact disruption pattern to identify weights \(or rows/columns\) that are likely to be disruptive, and filter the unlearning update accordingly\. Ideally we would want high unlearning transfer \(green\), with low disruption \(red\)\. Our approach of masking whole columns and rows removes disruption much more accurately\.Can we improve this filtering? Examining the update patterns in Figure[9](https://arxiv.org/html/2606.17168#A3.F9)shows that both disruption and transfer appear as column\- and row\-wise stripes\. Since weight updates are calculated as \(activation×\\timesgradient\) and thus are approximately low\-rank,111Strictly speaking their rank is equal to the number of tokens in the training batch, but most tokens have near\-zero gradients, so the update could be approximated by a much lower\-rank matrix\.disruption is driven by certain*rows and columns*rather than isolated weights\.

Since the disruption patterns shift within these columns and rows, it means that granular, per\-weight filtering misses many harmful weights\. Therefore, it is more effective to identify and remove whole faulty rows and columns \(which is equivalent to ablating the corresponding dimensions in the activations and output gradients\)\. Indeed, we see that doing so reduces the disruption\-to\-transfer ratio from 33% to 5%\. Another advantage of intervening on whole columns and rows is reduced memory consumption: we operate on the activations and module output gradients \(which are smaller\) rather than the full weight updates\.

## Appendix DMore on Representation Analysis

In this section, we provide full results for the representation analysis in Section[3](https://arxiv.org/html/2606.17168#S3)\. We provide more results on Qwen3\.5\-9B, measure PCA selectivity and attacker concentration, interpret individual PCs through vocabulary projection and steering vector alignment, and analyse baseline weight projections and attack subspace concentration in high\-variance PC subspace\.

### D\.1Model result: Qwen3\.5\-9B

Figure[10](https://arxiv.org/html/2606.17168#A4.F10)provides the same analysis of Figure[4](https://arxiv.org/html/2606.17168#S3.F4)on Qwen3\.5\-9B \(WMDP\-Bio, Layer 10\)\. Both models show similar patterns: high\-variance PCs are retain\-dominated while low\-variance PCs are forget\-specific, and baselines together with the fine\-tuning attacker concentrate 50–62% of their weight\-update norm in the top\-50 forget PCs, while RepSelect places∼\{\\sim\}4%\.

![Refer to caption](https://arxiv.org/html/2606.17168v1/x11.png)Figure 10:Representation structure of forget PCs on Qwen3\.5\-9B \(WMDP\-Bio, Layer 10\), mirroring Figure[4](https://arxiv.org/html/2606.17168#S3.F4)on Llama\-3\.1\-8B\.\(a\)Retain variance is∼4×\{\\sim\}4\\timeshigher per PC in the top tiers than the bottom tiers; RepSelect operates in the retain\-dilute bottom subspace\.\(b\)Baselines and the fine\-tuning attacker concentrate 50–62% of weight\-update norm in the top\-50 forget PCs; RepSelect places∼\{\\sim\}4%\.
### D\.2PCA Selectivity and Attacker Concentration

Table[9](https://arxiv.org/html/2606.17168#A4.T9)demonstrates why RepSelect is robust to attacks: the attacker’s weight updates concentrate in the top forget PCs, while RepSelect avoids them; baselines do not\.

Table 9:Why RepSelect works\(Llama\-3\.2\-3B, WMDP\-Bio\)\. Norm fraction≜\\triangleqfraction of‖Δ​W‖F2\\\|\\Delta W\\\|\_\{F\}^\{2\}projecting onto a subspace\.\(a\)A fine\-tuning attacker’s weight updates concentrate in the top\-50 forget PCs, the same directions that encode tokens common in the forget set \(see Table[10](https://arxiv.org/html/2606.17168#A4.T10)\)\.\(b\)The baselines place 25–41% of their update norm in the same subspace; RepSelect places∼\{\\sim\}11%\.\(a\) Attacker’s norm fraction in top\-50 PCs

Top PCs encode:virus, RNA, outbreaks, epidemic, infection

\(b\) Baseline norm fraction in top\-50 PCs \(%\)

### D\.3Vocabulary Projection of PCs

Each PC𝐯i∈ℝd\\mathbf\{v\}\_\{i\}\\in\\mathbb\{R\}^\{d\}is projected through the frozenlm\_headto obtain vocabulary logits\. Tables[10](https://arxiv.org/html/2606.17168#A4.T10)–[11](https://arxiv.org/html/2606.17168#A4.T11)show representative tokens for high\- and low\-variance PCs\.

Table 10:Vocabulary projection of activation PCs \(WMDP\-Bio\)\.Llama\-3\.2\-3Bgate\_proj, LoRA disabled\. High\-variance PCs project to broad domain\-specific tokens; low\-variance PCs do not yield a readable signal throughlm\_head\(logit lens is uninformative here, the tokens do not imply the content is benign, only that it is not a single\-token concept\)\.LayerPC𝝀\\bm\{\\lambda\}Highest\-logit tokensLowest\-logit tokensHigh\-variance PCs0PC20\.80\+\+the, a, in−\-viruses, viral, pathogens0PC30\.50\+\+virus, viruses, viral19PC010\.1\+\+virus, viral, RNA, protein19PC26\.1\+\+outbreaks, infection, epidemic21PC27\.6\+\+gated, promot, Scaffold−\-outbreaks, infectionsLow\-variance PCs0PC3990\.023\+\+Hend, avage, complement21PC3990\.121\+\+underlying, intent, umbleTable 11:Vocabulary projection of activation PCs \(WMDP\-Cyber\)\.Same format as Table[10](https://arxiv.org/html/2606.17168#A4.T10)\.LayerPC𝝀\\bm\{\\lambda\}Highest\-logit tokensLowest\-logit tokensHigh\-variance PCs0PC20\.83\+\+the, a, an−\-payloads, vulnerabilities7PC22\.69\+\+exploit, attack, exploiting21PC011\.5\+\+exploit, payloads, malicious21PC35\.6\+\+attack, vulnerability, attacks27PC110\.5\+\+attacker, attackers, maliciousLow\-variance PCs0PC3990\.023\+\+PLC, protect, Protection21PC3990\.116\+\+chor, urar, omat
### D\.4Top Forget Sequences per PC

For each PC, we project every forget sequence’s last\-token hidden state and rank by absolute projection magnitude\. Table[12](https://arxiv.org/html/2606.17168#A4.T12)illustrates the shared\-vs\-selective distinction from Section[3](https://arxiv.org/html/2606.17168#S3.SS0.SSS0.Px1)\. High\-variance PCs \(λ=5\\lambda=5–9×9\\times\) activate on broad domain concepts \(“weaponize anthrax”, “network attacks”\), content whose themes also appear in the biology/cybersecurity retain corpus, confirming that these directions are*shared*\. Low\-variance PCs \(λ≈0\.1\\lambda\\approx 0\.1\) activate on highly specific details \(“cytokines IL17b, IL18, CCL11”, “byte\-pattern 41 d0 00 00”\) that are unlikely to appear in any retain distribution, confirming that these directions are*selective*for the forget set\.

Tables[12](https://arxiv.org/html/2606.17168#A4.T12)–[13](https://arxiv.org/html/2606.17168#A4.T13)use the*same*PC indices across both domains and both models, ruling out cherry\-picking: the high\-vs\-low qualitative split holds whenever a PC is selected by its absolute eigenvalue rank, not by inspection\.

Table 12:Top forget sequences per PC, Llama\-3\.2\-3B Layer 18\(gate\_proj\)\. High\-variance PCs activate on broad domain themes shared with the retain corpus; low\-variance PCs activate on niche, forget\-specific details\.λ=σf2/σr2\\lambda=\\sigma^\{2\}\_\{f\}/\\sigma^\{2\}\_\{r\}\. The*same*PC indices \(0, 1, 2, 399\) are used across both domains\.Table 13:Top forget sequences per PC, Qwen3\-8B Layer 23\(gate\_proj\)\. Same PC indices as Table[12](https://arxiv.org/html/2606.17168#A4.T12); same qualitative split holds on a larger model across both domains\.
### D\.5Steering Vector Alignment

For each PC𝐯i\\mathbf\{v\}\_\{i\}, we score every sequence in the forget and retain corpora by projecting its last\-token hidden state onto the PC after mean\-centering:⟨𝐚t−𝝁,𝐯i⟩\\langle\\mathbf\{a\}\_\{t\}\-\\bm\{\\mu\},\\,\\mathbf\{v\}\_\{i\}\\rangle, where𝝁\\bm\{\\mu\}is the activation mean estimated during PCA\. We additionally compute a steering vector \(mean forget activation minus mean retain activation\) and measure each PC’s cosine alignment with it\.

Table[14](https://arxiv.org/html/2606.17168#A4.T14)shows two complementary results: \(i\) high\-variance PCs show2020–67×67\\timeshigher cosine alignment with the steering vector than low\-variance PCs, and \(ii\) forget sequences activate high\-variance PCs1\.21\.2–2\.1×2\.1\\timesmore than retain sequences \(F/R\>1\>1\), while low\-variance PCs show the opposite pattern \(F/R<1<1, retain activates more\)\. This confirms that RepSelect suppresses directions that broadly distinguish forget from retain data\.

Table 14:Activation PC analysis: steering alignment and forget/retain activation ratio\.For each PC𝐯i\\mathbf\{v\}\_\{i\}, we measure two quantities: \(1\) the absolute cosine similarity between𝐯i\\mathbf\{v\}\_\{i\}and the forget–retain steering vector𝝁f−𝝁r\\bm\{\\mu\}\_\{f\}\-\\bm\{\\mu\}\_\{r\}\(does this PC point in the forget–retain direction?\), and \(2\) the forget\-to\-retain activation ratio\|pf\|¯/\|pr\|¯\\overline\{\|p\_\{f\}\|\}\\,/\\,\\overline\{\|p\_\{r\}\|\}, wherep=⟨𝐚t−𝝁,𝐯i⟩p=\\langle\\mathbf\{a\}\_\{t\}\-\\bm\{\\mu\},\\,\\mathbf\{v\}\_\{i\}\\rangle\(do forget sequences activate this PC more than retain?\)\. Values are averaged over the top\-10 highest\- and bottom\-10 lowest\-variance PCs\. High\-variance PCs consistently align with the steering vector \(2020–67×67\\timesmore than low\-variance PCs\) and are preferentially activated by forget data \(F/R\>1\>1\), while low\-variance PCs are preferentially activated by retain data \(F/R<1<1\)\.
### D\.6Cross\-Distribution PC Variance

We measure the fraction of activation variance explained by the top\-10 forget PCs on forget, retain, and WikiText data \(Table[15](https://arxiv.org/html/2606.17168#A4.T15)\)\.

Table 15:Cross\-distribution PC variance\.Fraction of activation variance explained by the top\-10 forget PCs on each distribution\. Middle\-layer PCs are1\.81\.8–2\.1×2\.1\\timesmore selective for forget data\.
### D\.7Tiered Selectivity Along PCA Directions

Table[16](https://arxiv.org/html/2606.17168#A4.T16)decomposes the forget and retain MLP activation variance by PC tier \(sorted by absolute forget variance, highest first\), on Llama\-3\.1\-8B and Qwen3\.5\-9B at Layer 10 \(WMDP\-Bio\)\. The forget/retain variance ratio decreases monotonically from3\.63\.6–4\.3×4\.3\\timesin the top tier to1\.31\.3–1\.4×1\.4\\timesin the bottom tier, so retain variance is∼4×\{\\sim\}4\\timesmore concentrated in the top forget PCs than in the bottom forget PCs\. The top forget PCs are therefore retain\-shared, and the bottom forget PCs are forget\-specific — which is the structural property RepSelect exploits by collapsing the top subspace before each update\.

Table 16:Tiered variance by PCA rank \(Figures[4](https://arxiv.org/html/2606.17168#S3.F4)a and[10](https://arxiv.org/html/2606.17168#A4.F10)a\)\.For each tier of PCA directions \(sorted by absolute forget variance, highest first\), we report the fraction of total variance captured on forget and retain activations, and the forget/retain ratio\. Layer 10, WMDP\-Bio\.
### D\.8Baseline Weight Projection

We project each method’s weight updateΔ​W\\Delta Wonto the forget PCs and measure what fraction of its norm falls in the top\-kkPCs\. Table[17](https://arxiv.org/html/2606.17168#A4.T17)shows that all four baselines concentrate 25–60% of their update norm in the top\-50 forget PCs, while RepSelect places∼\{\\sim\}11%\.

Table 17:Fraction of weight\-update norm in top\-50 forget PCs \(%\)\.Norm fraction≜\\triangleqfraction of‖Δ​W‖F2\\\|\\Delta W\\\|\_\{F\}^\{2\}projecting onto the top\-50 PC subspace\. We exclude RMU as it modifies only a single layer, making a per\-layer comparison uninformative \(all non\-target layers show 0%\)\.
### D\.9Attack Subspace Concentration

We simulate a fine\-tuning attack \(50 SGD steps on forget data\) and project the attacker’s weight updateΔ​Watk\\Delta W\_\{\\text\{atk\}\}onto forget PCs\. We also run RepSelect for 5 epochs \(with and without LoRA\) and project its update\.*Projection mechanics:*Given a weight updateΔ​W∈ℝm×n\\Delta W\\in\\mathbb\{R\}^\{m\\times n\}and PCA directionsV∈ℝn×kV\\in\\mathbb\{R\}^\{n\\times k\}\(column space of the input\), we compute the norm fraction as‖Δ​W⋅V‖F2/‖Δ​W‖F2\\\|\\Delta W\\cdot V\\\|\_\{F\}^\{2\}/\\\|\\Delta W\\\|\_\{F\}^\{2\}, which measures what fraction of the update’s row\-space norm lies in the top\-kkPC subspace\. Table[18](https://arxiv.org/html/2606.17168#A4.T18)shows the attacker concentrates 5–7×\\timesmore of its update norm in the top\-10 PCs than RepSelect; the gap persists atk=50k\{=\}50\(up to 26\.6% vs\. 10\.4%\), confirming that collapsing high\-variance PCs makes unlearning adversarially inaccessible\.

Table 18:Weight\-update norm fraction in top\-kkforget PCs \(%\)\.Norm fraction≜\\triangleqfraction of‖Δ​W‖F2\\\|\\Delta W\\\|\_\{F\}^\{2\}projecting onto the top\-kkPC subspace\. Llama\-3\.2\-3B, WMDP\-Bio\. The attacker’s updates concentrate in the same subspace that RepSelect avoids, and the gap widens atk=50k\{=\}50\.

## Appendix EMore on Disruption and Robustness Analysis

In this section, we provide the theoretical foundations for RepSelect’s robustness\. We characterise the purified weight update geometrically, then prove formal bounds on the fraction of any attacker’s update that can overlap with RepSelect’s unlearned subspace \(low\-variance forget PCs\), and provide the analogous guarantee for LoRA\-based attackers\.

### E\.1MMLU Accuracy

Table[19](https://arxiv.org/html/2606.17168#A5.T19)reports MMLU accuracy\[Hendrycks et al\.,[2021](https://arxiv.org/html/2606.17168#bib.bib16)\]for the base model and RepSelect after unlearning WMDP\-Bio, using the same hyperparameters as the main experiments under fine\-tuning attacks \(Figure[3](https://arxiv.org/html/2606.17168#S1.F3)\)\. RepSelect preserves general capability across all models, confirming that the WikiText KL≤0\.01\\leq 0\.01budget preserves downstream utility\.

Table 19:MMLU accuracy is preserved after RepSelect unlearning\.Hyperparameters are optimised for the fine\-tuning attack \(Figure[3](https://arxiv.org/html/2606.17168#S1.F3)\) within WikiText KL≤0\.01\\leq 0\.01\.
### E\.2The Purified Weight Update

By the chain rule, the per\-token weight update isΔ​W=𝐠⊗𝐚\\Delta W=\\mathbf\{g\}\\otimes\\mathbf\{a\}\. After applying the same collapse procedure to both activations and output gradients, we obtainΔ​W=∑t𝐠t′⊗𝐚t′\\Delta W=\\sum\_\{t\}\\mathbf\{g\}^\{\\prime\}\_\{t\}\\otimes\\mathbf\{a\}^\{\\prime\}\_\{t\}, where𝐚t′\\mathbf\{a\}^\{\\prime\}\_\{t\}lives only in forget\-specific activation directions and𝐠t′\\mathbf\{g\}^\{\\prime\}\_\{t\}lives only in forget\-specific gradient directions\. Each𝐠t′⊗𝐚t′\\mathbf\{g\}^\{\\prime\}\_\{t\}\\otimes\\mathbf\{a\}^\{\\prime\}\_\{t\}is a rank\-1 matrix that lies in a forget\-specific subspace\.

By construction, the full update satisfiesΔ​W⋅𝐯i≈0\\Delta W\\cdot\\mathbf\{v\}\_\{i\}\\approx 0for all high\-variance directions𝐯i\\mathbf\{v\}\_\{i\}, meaning the weight update has near\-zero component along directions that are less forget\-specific than the low\-variance subspace\.

### E\.3Robustness and Disruption Guarantees

We show mathematically when the attacker fine\-tunes on forget\-domain data \(sharing covarianceΣ\\Sigmawith the forget corpus\), their updates are concentrated along the same high\-variance directionsVkV\_\{k\}that RepSelect avoids\.

###### Proposition 1\(Robustness of subspace\-restricted unlearning\)\.

Consider a single MLP linear layer\. LetΣ=Cov​\(𝐚\)\\Sigma=\\mathrm\{Cov\}\(\\mathbf\{a\}\)be the covariance of forget\-corpus activations with eigenpairs\{\(𝐯i,λi\)\}i=1d\\\{\(\\mathbf\{v\}\_\{i\},\\lambda\_\{i\}\)\\\}\_\{i=1\}^\{d\},λ1≥λ2≥⋯\\lambda\_\{1\}\\geq\\lambda\_\{2\}\\geq\\cdots\. LetVk=span​\{𝐯1,…,𝐯k\}V\_\{k\}=\\mathrm\{span\}\\\{\\mathbf\{v\}\_\{1\},\\dots,\\mathbf\{v\}\_\{k\}\\\}be the top\-kkprincipal subspace,PkP\_\{k\}andP⟂=I−PkP\_\{\\perp\}=I\-P\_\{k\}the projections ontoVkV\_\{k\}andVk⟂V\_\{k\}^\{\\perp\}\.

RepSelect produces a weight updateΔ​Wunl\\Delta W\_\{\\mathrm\{unl\}\}with row space inVk⟂V\_\{k\}^\{\\perp\}, so thatΔ​Wunl​Pk=0\\Delta W\_\{\\mathrm\{unl\}\}\\,P\_\{k\}=0\. Suppose an attacker fine\-tunes on data from the same domain \(sharing covarianceΣ\\Sigma\), producingΔ​Watk=∑t𝐠tatk⊗𝐚tatk\\Delta W\_\{\\mathrm\{atk\}\}=\\sum\_\{t\}\\mathbf\{g\}\_\{t\}^\{\\mathrm\{atk\}\}\\otimes\\mathbf\{a\}\_\{t\}^\{\\mathrm\{atk\}\}\. Assuming activations and gradients are independent, the fraction of the attack update that overlaps with the unlearned subspace is bounded by:

𝔼​\[‖Δ​Watk​P⟂‖F2\]𝔼​\[‖Δ​Watk‖F2\]=∑i\>kλitr​\(Σ\)=:ϵk\.\\frac\{\\mathbb\{E\}\\bigl\[\\\|\\Delta W\_\{\\mathrm\{atk\}\}\\,P\_\{\\perp\}\\\|\_\{F\}^\{2\}\\bigr\]\}\{\\mathbb\{E\}\\bigl\[\\\|\\Delta W\_\{\\mathrm\{atk\}\}\\\|\_\{F\}^\{2\}\\bigr\]\}=\\frac\{\\sum\_\{i\>k\}\\lambda\_\{i\}\}\{\\mathrm\{tr\}\(\\Sigma\)\}=:\\epsilon\_\{k\}\.Only theP⟂P\_\{\\perp\}\-component ofΔ​Watk\\Delta W\_\{\\mathrm\{atk\}\}can interfere withΔ​Wunl\\Delta W\_\{\\mathrm\{unl\}\}; the remaining fraction\(1−ϵk\)\(1\-\\epsilon\_\{k\}\)of the attack’s update norm is confined toVkV\_\{k\}and has no effect on the unlearned subspace\.

###### Proof\.

Decompose each attack activation as𝐚tatk=Pk​𝐚tatk\+P⟂​𝐚tatk\\mathbf\{a\}\_\{t\}^\{\\mathrm\{atk\}\}=P\_\{k\}\\mathbf\{a\}\_\{t\}^\{\\mathrm\{atk\}\}\+P\_\{\\perp\}\\mathbf\{a\}\_\{t\}^\{\\mathrm\{atk\}\}\. ThenΔ​Watk​P⟂=∑t𝐠tatk​\(P⟂​𝐚tatk\)⊤\\Delta W\_\{\\mathrm\{atk\}\}\\,P\_\{\\perp\}=\\sum\_\{t\}\\mathbf\{g\}\_\{t\}^\{\\mathrm\{atk\}\}\(P\_\{\\perp\}\\mathbf\{a\}\_\{t\}^\{\\mathrm\{atk\}\}\)^\{\\top\}\. For a single token,‖𝐠​\(P⟂​𝐚\)⊤‖F2=‖𝐠‖2​‖P⟂​𝐚‖2\\\|\\mathbf\{g\}\\,\(P\_\{\\perp\}\\mathbf\{a\}\)^\{\\top\}\\\|\_\{F\}^\{2\}=\\\|\\mathbf\{g\}\\\|^\{2\}\\\|P\_\{\\perp\}\\mathbf\{a\}\\\|^\{2\}\. Under the independence assumption𝔼​\[‖𝐠‖2​‖P⟂​𝐚‖2\]=𝔼​\[‖𝐠‖2\]​𝔼​\[‖P⟂​𝐚‖2\]\\mathbb\{E\}\[\\\|\\mathbf\{g\}\\\|^\{2\}\\\|P\_\{\\perp\}\\mathbf\{a\}\\\|^\{2\}\]=\\mathbb\{E\}\[\\\|\\mathbf\{g\}\\\|^\{2\}\]\\,\\mathbb\{E\}\[\\\|P\_\{\\perp\}\\mathbf\{a\}\\\|^\{2\}\], and similarly for the full update\. The ratio reduces to:

𝔼​\[‖P⟂​𝐚‖2\]𝔼​\[‖𝐚‖2\]=∑i\>kλi∑i=1dλi=ϵk,\\frac\{\\mathbb\{E\}\[\\\|P\_\{\\perp\}\\mathbf\{a\}\\\|^\{2\}\]\}\{\\mathbb\{E\}\[\\\|\\mathbf\{a\}\\\|^\{2\}\]\}=\\frac\{\\sum\_\{i\>k\}\\lambda\_\{i\}\}\{\\sum\_\{i=1\}^\{d\}\\lambda\_\{i\}\}=\\epsilon\_\{k\},since𝔼​\[‖P⟂​\(𝐚−𝝁\)‖2\]=∑i\>kλi\\mathbb\{E\}\[\\\|P\_\{\\perp\}\(\\mathbf\{a\}\-\\bm\{\\mu\}\)\\\|^\{2\}\]=\\sum\_\{i\>k\}\\lambda\_\{i\}and𝔼​\[‖𝐚−𝝁‖2\]=tr​\(Σ\)\\mathbb\{E\}\[\\\|\\mathbf\{a\}\-\\bm\{\\mu\}\\\|^\{2\}\]=\\mathrm\{tr\}\(\\Sigma\)\. ∎

###### Corollary 1\(LoRA attacker\)\.

If the attacker uses a rank\-rrLoRA adapterΔ​Watk=𝐀𝐁⊤\\Delta W\_\{\\mathrm\{atk\}\}=\\mathbf\{A\}\\mathbf\{B\}^\{\\top\}with𝐁∈ℝdin×r\\mathbf\{B\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{in\}\}\\times r\}, the row space ofΔ​Watk\\Delta W\_\{\\mathrm\{atk\}\}is at mostrr\-dimensional\. Gradient\-based optimisation preferentially aligns𝐁\\mathbf\{B\}with the highest\-variance directions of the activation distribution\. Whenr≤kr\\leq k, this yieldscolspan​\(𝐁\)⊆Vk\\mathrm\{colspan\}\(\\mathbf\{B\}\)\\subseteq V\_\{k\}, so thatΔ​Watk​P⟂=0\\Delta W\_\{\\mathrm\{atk\}\}\\,P\_\{\\perp\}=0: the LoRA attack has zero overlap with the unlearned subspace and cannot directly interfere with RepSelect’s weight update\.

## Appendix FMore Unlearning and relearning trajectories

In this section, we report unlearning and relearning trajectories across all four model families, showing that advantage of RepSelect is consistent across model families and datasets\.

Figure[3](https://arxiv.org/html/2606.17168#S1.F3)\(bottom\) shows the unlearning–disruption trade\-off and relearning trajectories on Gemma\-4\-E4B\. Figures[12](https://arxiv.org/html/2606.17168#A6.F12)–[14](https://arxiv.org/html/2606.17168#A6.F14)report the same four\-panel layout \(WMDP\-Bio and Animal Abuse, unlearning vs\. Wikitext KL on the left, post\-attack accuracy over relearning epochs on the right\) on the remaining three model families\. The qualitative picture is consistent across all four: For knowledge unlearning \(WMDP\-Bio\), RepSelect achieves much more unlearning per the same amount of disruption \(Wikitext KL, on the left x\-axis\)\. For tendency unlearning \(Animal Abuse\), baselines appear to unlearn well too, but then a relearning attack \(on the right\) reveals their are almost fully reversible even by one epoch of relearning\.

Error regions show standard deviation over the top 10 trials out of 30, optimized by Optuna search process\. \(Note that in some plots the displayed RMU unlearning trajectory does not reach 0\.01 WikiText KL, because Optuna converged there on using a very small learning rate, demonstrating inadequacy of RMU for some models\.\)

![Refer to caption](https://arxiv.org/html/2606.17168v1/x12.png)Figure 11:Data scaling on Animal Abuse\.Unlearning \(left\) and relearning \(right\) trajectories for RepSelect on BeaverTails Animal Abuse, varying the forget\-set size from 10 to 360 samples \(out of 371 available\)\. Two models are shown \(Llama\-3\.1\-8B, Qwen3\.5\-9B\); RepSelect is run without the LoRA adversary, with SVD computed on the forget set\. 10 samples already achieve over half of the maximal unlearning, and 90 samples saturate it; further data yields no additional gain\. The dashed gray line shows the no\-unlearning baseline, also subjected to the same relearning attack\.![Refer to caption](https://arxiv.org/html/2606.17168v1/x13.png)Figure 12:Unlearning and relearning trajectories onLlama\-3\.1\-8B\. Same layout as Figure[3](https://arxiv.org/html/2606.17168#S1.F3)\(bottom\)\.![Refer to caption](https://arxiv.org/html/2606.17168v1/x14.png)Figure 13:Unlearning and relearning trajectories onQwen3\.5\-9B\. Same layout as Figure[3](https://arxiv.org/html/2606.17168#S1.F3)\(bottom\)\.![Refer to caption](https://arxiv.org/html/2606.17168v1/x15.png)Figure 14:Unlearning and relearning trajectories onDeepSeek\-V2\-Lite\(MoE\)\. Same layout as Figure[3](https://arxiv.org/html/2606.17168#S1.F3)\(bottom\)\.