Forgetting That Sticks: Quantization-Permanent Unlearning via Circuit Attribution
Summary
This paper identifies a fundamental sparsity-permanence tradeoff where quantization reverses machine unlearning, and proposes MANSU, a method combining causal circuit attribution and null-space projection to achieve quantization-permanent forgetting.
View Cached Full Text
Cached at: 05/18/26, 06:27 PM
Paper page - Forgetting That Sticks: Quantization-Permanent Unlearning via Circuit Attribution
Source: https://huggingface.co/papers/2605.15138
Abstract
Quantization reverses machine unlearning effects, revealing a fundamental sparsity-permanence tradeoff where parameter updates fall below quantization bin widths, leading to the development of MANSU to achieve both forgetting and retention preservation under compression.
Standardunlearningevaluations measurebehavioral suppressionin full precision, immediately after training, despite every deployed language model being quantized first. Recent work has shown that 4-bit post-trainingquantizationcan reverse machineunlearning; we show this is not a tuning artefact but a systematic dual failure: gradient-based methods that achieve meaningful forgetting lose it under compression, while methods that survivequantizationbarely change the model. Both failures trace to the same root cause: across all baselines, per-parameter updates lie 47-828x below the NF4quantizationbin width; updates diffused across billions of parameters cannot clearquantizationbin boundaries, a consequence we formalize as asparsity-permanence tradeoff. We present MANSU (Mechanistic-Aligned Null-SpaceUnlearning), which resolves both modes by combiningcausal circuit attributionto isolate the minimal forget-set subgraph, circuit-restrictednull-space projectionwith adiagonal-Fisher retain bound, and aper-parameter magnitude floorguaranteeingquantizationsurvival by construction. We additionally introduceCircuit Attribution Divergence(CAD), a mechanistic verification metric distinguishingstructural erasurefrombehavioral suppression, a distinction existing metrics cannot make. Across multiple model families and hazard benchmarks, MANSU is the first method to jointly satisfy all four properties with margin on each (meaningful forgetting, retain preservation, non-positivePTQ gap, andstructural erasure), while gradient-based baselines recover up to +0.05 accuracy under compression.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2605\.15138
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.15138 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.15138 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.15138 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
Approximate Machine Unlearning through Manifold Representation Forgetting Guided by Self Mode Connectivity
This paper proposes ManiF-SMC, a method for approximate machine unlearning that operates entirely in the representation space by pushing erased samples away from their original learned manifold representation toward their nearest semantic neighbors in the retained data, using a margin-based triplet loss guided by a self-mode-connectivity module for adaptive margins.
Quantization-Robust Unlearning through the Lens of Retain-Forget Loss Landscapes Interaction
This paper proposes a quantization-robust unlearning framework for large language models, using loss landscape analysis to ensure effective forgetting while maintaining model utility after compression.
Mechanistic origins of catastrophic forgetting: why RL preserves circuits better than SFT?
This paper investigates the mechanistic origins of catastrophic forgetting in LLMs, finding that reinforcement learning preserves internal computational circuits better than supervised fine-tuning, resulting in less forgetting of prior capabilities.
What to Forget in Unlearning? Forget Set Curation for Language Models
This paper explores forget set curation for machine unlearning in language models, introducing a benchmark to evaluate verbatim output suppression and highlighting trade-offs between effectiveness and capability retention.
Fast Unlearning at Scale via Margin Self-Correction
Introduces MASC (Margin Self-Correction), an efficient unlearning method for LLMs that uses an online stopping rule to achieve competitive forget–retain trade-offs at reduced computational cost, validated on TOFU and MUSE benchmarks.