LLM Unlearning for Cyber Defense: A Survey on Methods, Challenges, and Emerging Threats

# *
Source: [https://arxiv.org/html/2607.16227](https://arxiv.org/html/2607.16227)
LLM Unlearning for Cyber Defense: A Survey on Methods, Challenges, and Emerging Threats

\\tocauthor

=Ruppikha Sree Shankar, Abhishek Bhardwaj, Arnav Doshi, Anusri Nagarajan, Troy Paulus Asia, Saptarshi Sengupta

11institutetext:Ruppikha Sree Shankar22institutetext:Manipal Institute of Technology, Manipal Academy of Higher Education, Manipal, KA, India,
22email:ruppikha\.mitmpl2022@learner\.manipal\.edu33institutetext:Abhishek Bhardwaj44institutetext:Department of Computer Science, San José State University, San José, CA, USA,
44email:abhishek\.bhardwaj@sjsu\.edu55institutetext:Arnav Doshi66institutetext:Department of Computer Science, San José State University, San José, CA, USA,
66email:arnav\.doshi@sjsu\.edu77institutetext:Anusri Nagarajan88institutetext:Department of Computer Science, San José State University, San José, CA, USA,
88email:anusri\.nagarajan@sjsu\.edu99institutetext:Troy Paulus Asia1010institutetext:Department of Computer Engineering, San José State University, San José, CA, USA,
1010email:troypaulus\.asia@sjsu\.edu1111institutetext:Saptarshi Sengupta1212institutetext:Department of Computer Science, San José State University, San José, CA, USA,
1212email:saptarshi\.sengupta@sjsu\.edu## \*

Abhishek Bhardwaj†\[0009\-0008\-9904\-5132\]Arnav Doshi\[0009\-0009\-4094\-8756\]Anusri Nagarajan\[0009\-0006\-0779\-7544\]Troy Paulus Asia\[0009\-0000\-5470\-1003\]Saptarshi Sengupta\[0000\-0003\-1114\-343X\]

###### Abstract

Large language models \(LLMs\) are increasingly deployed in security\-critical systems across healthcare, finance, education, and decision support, yet their inability to forget creates serious cybersecurity, privacy, and safety risks\. Sensitive personal information, copyrighted material, hazardous domain knowledge, and memorized training data remain encoded across billions of parameters long after deployment, leaving models vulnerable to extraction, jailbreak attacks, membership inference, and regulatory non\-compliance\. Real\-world incidents, from chatbots regenerating private information to fabricated legal citations producing direct legal and financial cost, place the problem at the center of the emerging\-threats landscape rather than the realm of speculation\. Because retraining billion\-parameter models on revised corpora is computationally infeasible, and because knowledge within an LLM is distributed and entangled across parameters rather than localized to identifiable units, LLM unlearning has emerged as the principal cyber defense response, aiming to remove or suppress targeted knowledge from a trained model without retraining and without eroding what the model should still know\. A central question, however, remains unresolved\. Do current methods genuinely remove knowledge, or do they only stop the model from expressing it under ordinary prompting conditions? This survey examines LLM unlearning through the lens of security, robustness, and verifiable forgetting, with primary focus on gradient\-based methods, which have come to dominate the field due to their compatibility with existing training pipelines and their scalability to billion\-parameter models\. To structure the analysis, this survey introduces a three\-level framework that separates behavioral suppression, representation\-level attenuation, and true forgetting, and uses it to read the operational behavior of three method families: gradient ascent and descent approaches, influence\-based methods, and methods that constrain updates through parameter saliency, low\-rank adapters, or localized model editing\. Across the three families, a consistent pattern emerges\. Gradient ascent does not invert the original training trajectory; it introduces a new one that leaves underlying representations largely intact\. Influence\-based methods reach further by modeling how individual training points shape parameters, but rely on local linearity assumptions that do not strictly hold in non\-convex deep models\. Constrained and localized editing methods improve stability and precision, yet still assume that targeted knowledge can be cleanly isolated, which holds only approximately at scale\. None of the families reviewed achieve true forgetting in the strict sense of equivalence to a model retrained without the forget set\. Current evaluation protocols, anchored by benchmarks such as TOFU, RWKU, WMDP, and MUSE, capture behavioral suppression effectively but fall short of certifying removal under adversarial recovery probes including relearning attacks, membership inference, jailbreak prompts, and embedding\-space soft\-prompt attacks\. This evaluation gap, alongside the absence of methods that achieve true forgetting, robustness against adversarial recovery, scalability to repeated and compositional requests, and standardized evaluation foundations, defines the central open challenges this survey leaves for future work\.

###### keywords:

LLM Unlearning

⋅\\cdotGradient Ascent

⋅\\cdotAdversarial Robustness

⋅\\cdotCybersecurity

⋅\\cdotModel Editing

⋅\\cdotMembership Inference Attacks

⋅\\cdotCyber Defense

††footnotetext:These authors contributed equally to this work\.## 1Introduction

Large language models \(LLMs\) have demonstrated remarkable capabilities across healthcare, finance, legal analysis, and educationMaity and Saikia \([2025](https://arxiv.org/html/2607.16227#bib.bib1)\); Yuan and others \([2024](https://arxiv.org/html/2607.16227#bib.bib2)\); Vrdoljaket al\.\([2025](https://arxiv.org/html/2607.16227#bib.bib3)\)\. Their rapid adoption has been driven by large\-scale pretraining and the availability of vast web\-scale datasetsKotei and Thirunavukarasu \([2023](https://arxiv.org/html/2607.16227#bib.bib14)\)\. These models now power conversational assistants, code generation tools, decision support systems, and content creation platformsYanget al\.\([2025](https://arxiv.org/html/2607.16227#bib.bib15)\), and their ability to generalize across tasks has made them foundational components of modern AI infrastructure\.

This strength carries a critical limitation: an inability to forget\. LLMs retain sensitive, biased, or outdated information, raising serious concerns regarding privacyCarliniet al\.\([2022](https://arxiv.org/html/2607.16227#bib.bib4)\), copyright infringementKaramolegkouet al\.\([2023](https://arxiv.org/html/2607.16227#bib.bib5)\); Kimet al\.\([2026](https://arxiv.org/html/2607.16227#bib.bib6)\), and harmful content generationWeiet al\.\([2023](https://arxiv.org/html/2607.16227#bib.bib7)\)\. Unlike humans, who can selectively suppress irrelevant or harmful memoriesPastötter and Haciahmet \([2022](https://arxiv.org/html/2607.16227#bib.bib8)\), LLMs lack mechanisms for targeted forgetting\. This creates persistent risks including private data retention, misinformation propagation, and susceptibility to adversarial exploitation\. The challenge grows with model scale: training on massive datasets with billions of parameters makes naive retraining for data removal computationally infeasibleLe\-Khac and Vinh \([2025a](https://arxiv.org/html/2607.16227#bib.bib11)\), and knowledge distributed across entangled parameter spaces resists targeted removal without collateral damage to unrelated capabilitiesLizzo and Heck \([2025](https://arxiv.org/html/2607.16227#bib.bib12)\)\.

To address these challenges, LLM unlearning has emerged as a promising paradigm, aiming to selectively remove or suppress specific knowledge from trained models without full retraining while preserving overall utilityZhang and others \([2025](https://arxiv.org/html/2607.16227#bib.bib13)\)\. A fundamental ambiguity, however, remains unresolved in current research: whether existing methods truly eliminate knowledge or merely suppress its expression under certain conditions\.

In this survey, a comprehensive review of LLM unlearning is provided through a conceptual lens that distinguishes true forgetting from suppression\. Gradient\-based methods have emerged as the most widely adopted and scalable approach for unlearning in large models, fitting naturally into training and fine\-tuning pipelines already in use at scale\. However, their reliance on optimization dynamics introduces fundamental challenges, including incomplete forgetting, interference with retained knowledge, and difficulty verifying whether information has been truly removed\. Gradient\-based methods are the primary focus, examining whether they achieve genuine deletion of information or induce surface\-level behavioral changes that can be reversed or bypassed\. The challenges of verifying unlearning, the risks of information leakage under adversarial prompting, and the lack of standardized evaluation protocols are also highlighted\. Three research questions organize the analysis\. The first asks what categories of harmful factual associations current gradient\-based unlearning methods effectively defend against, and under what assumptions about adversary access those defenses hold\. The second asks how robust these defenses are against adversarial recovery, including relearning attacks, jailbreak prompts, membership inference, and embedding\-space probes\. The third asks whether current evaluation protocols verify removal at the level of internal representations, or whether they only certify behavioral suppression under constrained and non\-adaptive testing conditions\. These questions are addressed across Sections[2](https://arxiv.org/html/2607.16227#S2),[4](https://arxiv.org/html/2607.16227#S4), and[6](https://arxiv.org/html/2607.16227#S6)respectively\.

### 1\.1Scope and Contributions

LLM unlearning has attracted growing attention from the research community, and several surveys now cover aspects of the problem\. Si et al\.Zhanget al\.\([2023](https://arxiv.org/html/2607.16227#bib.bib44)\)provided an early taxonomy focused on effectiveness and utility preservation\. Xu et al\.Xuet al\.\([2023](https://arxiv.org/html/2607.16227#bib.bib45)\)compared classical and LLM\-specific machine unlearning approaches\. Blanco\-Justicia et al\.Blanco\-Justicia \([2025](https://arxiv.org/html/2607.16227#bib.bib85)\)offered the first comprehensive survey with intricate taxonomy and benchmark evaluation, though without examining robustness objectives\. Le\-Khac and TruongLe\-Khac and Vinh \([2025b](https://arxiv.org/html/2607.16227#bib.bib36)\)proposed a four\-objective problem formulation covering effectiveness, efficiency, utility, and robustness, and were among the first to assess threat models for unlearning evaluation\. Qiu et al\.Le\-Khac and Vinh \([2025a](https://arxiv.org/html/2607.16227#bib.bib11)\)introduced a novel taxonomy based on intervention phase \(training\-time, post\-training, inference\-time\) covering over 180 papers\. Li et al\.Cevalloset al\.\([2025](https://arxiv.org/html/2607.16227#bib.bib47)\)offered a comprehensive survey spanning ten unlearning method classes, multimodal settings, and seven adversarial evaluation categories through May 2025\.

This survey distinguishes itself from prior work in three ways:

- •Cybersecurity threat model\.A systematic analysis of gradient\-based LLM unlearning is provided through an explicit cybersecurity threat model, framing harmful factual associations as an adversarial condition and unlearning as a corrective security control\. The threat model, adversary taxonomy, and real\-world failure cases in Section[2](https://arxiv.org/html/2607.16227#S2)anchor the technical analysis in the context of operational security\.
- •Three\-level robustness framework\.A three\-level framework separating behavioral suppression, representation\-level attenuation, and true forgetting is introduced\. This framework functions simultaneously as an analytical lens for classifying the depth of knowledge removal and as a robustness ladder classifying the adversary access level against which each method provides meaningful defense\.
- •Coverage through mid\-2026\.The gradient\-based unlearning literature through mid\-2026 is covered, incorporating recent theoretical and empirical findings that challenge the foundational role of gradient ascent in unlearning pipelines and demonstrate the susceptibility of leading methods to white\-box adversarial recovery\.

### 1\.2Methodology

This survey is based on a systematic review of recent literature on machine unlearning and large language models\. Literature was collected from four primary databases: Scopus, the preprint repository arXiv, IEEE Xplore, and the ACM Digital Library\. Given the rapidly evolving nature of LLM unlearning research, where many contributions appear first as preprints and conference proceedings before journal publication, the search was supplemented by manual review of ACL Anthology and Google Scholar to capture NLP and machine learning papers not fully indexed in the primary databases\.

The search strategy follows established systematic review practice for rapidly evolving ML subfields, using keyword and temporal filters aligned with prior surveys in machine unlearningLe\-Khac and Vinh \([2025b](https://arxiv.org/html/2607.16227#bib.bib36)\)\. The period from 2023 to 2026 was chosen because prior surveys have comprehensively covered work up to 2022, and the most consequential advances in gradient\-based unlearning and adversarial evaluation have emerged since 2023\. The queries used are as follows:

- •Scopus:TITLE\-ABS\-KEY \(\("large language model\*" OR "LLM\*"\) AND "unlearn\*"\) AND \(LIMIT\-TO \(PUBYEAR, 2023\) OR LIMIT\-TO \(PUBYEAR, 2024\) OR LIMIT\-TO \(PUBYEAR, 2025\) OR LIMIT\-TO \(PUBYEAR, 2026\)\)
- •arXiv:date\_range: from 2023\-01\-01; terms: AND title=large language model\* AND title=unlearn\*; OR title=LLM\* AND title=unlearn\*
- •IEEE Xplore:"Document Title":"large language model\*" OR "Document Title":LLM\* AND "Document Title":unlearn\*
- •ACM Digital Library:Title:\("large language model\*" OR "LLM\*"\) AND Title:\(unlearn\*\)

For IEEE Xplore and the ACM Digital Library, publication year was restricted to 2023 through 2026 using the respective date filter interfaces\. Papers were manually reviewed for quality, empirical evidence, and relevance\. Table[1](https://arxiv.org/html/2607.16227#S1.T1)presents the inclusion and exclusion criteria applied during the screening process\. Priority was given to works providing strong theoretical foundations, scalable methodologies, and empirical evaluation in the context of large models, with particular emphasis on gradient\-based approaches\.

Table 1:Inclusion and exclusion criteria for the systematic literature review\.
### 1\.3Organization of this Survey

The remainder of this survey is organized as follows\. Section[2](https://arxiv.org/html/2607.16227#S2)situates LLM unlearning within a cybersecurity threat model, formalizing the adversary taxonomy and motivating cases that ground the analysis\. Section[3](https://arxiv.org/html/2607.16227#S3)introduces the fundamental concepts underlying LLM unlearning, including LLM architecture, knowledge storage, machine unlearning basics, and the three\-level conceptual framework\. Section[4](https://arxiv.org/html/2607.16227#S4)surveys gradient\-based unlearning methods in depth, covering gradient ascent and descent variants, influence\-based methods, and parameter and loss function based methods\. Section[5](https://arxiv.org/html/2607.16227#S5)covers non\-gradient\-based unlearning approaches including model editing, knowledge editing, and differential privacy based methods\. Section[6](https://arxiv.org/html/2607.16227#S6)reviews evaluation protocols, benchmark datasets, evaluation metrics, and adversarial robustness testing\. Section[7](https://arxiv.org/html/2607.16227#S7)discusses the broader implications of the survey’s findings and identifies open challenges for future research\.

## 2LLM Unlearning for Cyber Defense

The integration of large language models into decision support systems, legal research tools, healthcare applications, and software development pipelines has repositioned these models as critical infrastructure\. Critical infrastructure carries adversaries, and the question of what an LLM retains after training is therefore not only a question of model quality but a question of attack surface\. A model that cannot forget is a model that cannot be secured: every harmful factual association encoded in its parameters is a potential extraction target, a potential liability, and a potential vector for adversarial exploitation\. This section formalizes the threat model that grounds the survey, identifies the adversary types relevant to LLM unlearning, documents the real\-world consequences of insufficient knowledge removal, and establishes the mapping between gradient\-based unlearning methods and their operational security guarantees\. The three\-level framework introduced in Section[3\.4](https://arxiv.org/html/2607.16227#S3.SS4)functions throughout this survey not only as an analytical lens for classifying the depth of knowledge removal but as a robustness ladder for evaluating the strength of unlearning as a cyber defense mechanism\.

### 2\.1The Threat Landscape

Two phases define the lifecycle of a harmful association within a deployed LLM\. In the first phase, the association enters the model\. This can occur through direct training on sensitive or hazardous corpora\. It can also arise through memorization of rare but recoverable data points during pretraining, or through deliberate injection of malicious content into training pipelines by an upstream actor\. Backdoor attacks represent a particularly structured form of this threat\. A specific trigger stimulus is embedded into the training distribution, causing the model to elicit a targeted harmful response at inference timeCarliniet al\.\([2022](https://arxiv.org/html/2607.16227#bib.bib4)\)\. Dataset poisoning at the training\-data level represents the most explicitly adversarial entry point and has been formally studied as a threat to machine learning systems more broadlyCao and Yang \([2015](https://arxiv.org/html/2607.16227#bib.bib33)\)\. In the second phase, the stored association exits the model at inference time: through ordinary prompting that happens to elicit memorized content, through adversarial prompting crafted to extract it, through soft\-prompt or embedding\-space attacks that bypass the discrete token interface, or through relearning attacks that restore suppressed associations after an unlearning procedure has been applied\.

Unlearning occupies the space between these two phases\. Its purpose is to sever the connection between a stored harmful association and every inference\-time pathway through which it could be expressed or recovered\. Establishing which pathway a given unlearning method actually severs, and which pathways it leaves intact, is the central empirical question this survey addresses\.

The categories of harmful association most relevant to the cybersecurity framing of this survey are the following, organized by the threat class and the adversary access level each requires to exploit\.

Memorized personally identifiable information \(PII\)encompasses names, addresses, medical records, and other private data that LLMs reproduce under targeted extraction conditionsCarliniet al\.\([2022](https://arxiv.org/html/2607.16227#bib.bib4)\)\. PII memorization is especially prevalent for rare or repeated data points in the training corpus\. The adversary exploiting this threat operates primarily in the gray\-box regime, using membership inference attacks to determine what the model remembers and targeted prompting to extract it\. Influence\-based unlearning methods are most directly designed to address this threat category because they model the per\-sample contribution of training data to learned parameters\.

Hazardous domain knowledgeencompasses actionable information about biological agents, chemical synthesis routes, cybersecurity attack techniques, and weapons design that can be elicited through targeted prompting\. The WMDP benchmarkLi \([2024](https://arxiv.org/html/2607.16227#bib.bib77)\)specifically quantifies how much of this class of knowledge survives popular unlearning interventions\. The adversary here operates across the full black\-box to white\-box spectrum: casual extraction is possible through jailbreak prompts alone, while determined extraction against unlearned models requires white\-box access as demonstrated by the adversarial evaluation literatureLuckiet al\.\([2025](https://arxiv.org/html/2607.16227#bib.bib88)\)\. Gradient ascent and its variants are the primary methods deployed against this threat, and their failure modes under white\-box attack define the evaluation gap this survey documents\.

Copyrighted materialencompasses verbatim or near\-verbatim text the model can reproduce, creating legal exposure for deploying organizationsKaramolegkouet al\.\([2023](https://arxiv.org/html/2607.16227#bib.bib5)\)\. The adversary exploiting this threat is typically a black\-box querying actor who can demonstrate verbatim reproduction without needing model internals\. Behavioral suppression is sufficient to defeat this threat class in standard conditions, which is why copyright\-focused unlearning evaluations tend to use output\-level metrics such as ROUGE and verbatim match rates\.

Hallucinated factual associationsare a distinct category that does not map cleanly onto memorization\. The model generates confident, authoritative\-sounding false statements that arise from entangled parametric associations rather than from any specific training exampleMageshet al\.\([2024](https://arxiv.org/html/2607.16227#bib.bib10)\)\. The adversary here is typically unintentional: an ordinary user who receives and acts on a false assertion\. Unlearning methods are not well\-suited to addressing hallucination because there is no specific training example to remove; the failure is representational rather than memorial\. This category motivates the distinction between behavioral suppression \(preventing a specific false output\) and representation\-level attenuation \(reducing the parametric entanglement that produces it\)\.

Backdoor and poisoned associationsare the most explicitly adversarial category\. An upstream actor deliberately contaminates the training corpus so that a specific trigger pattern reliably elicits a targeted harmful response at inference time\. The adversary controls the entry point and knows the trigger, giving them effective white\-box exploitation capability even against a black\-box deployed system\. This threat category is the most resistant to post\-hoc gradient\-based unlearning because the association is structurally embedded rather than distributed through ordinary training dynamics\.

Table[2](https://arxiv.org/html/2607.16227#S2.T2)maps each threat category to its entry point, the adversary access level required to exploit it, and the level of unlearning robustness the defense must achieve to be meaningful\.

Table 2:Threat categories, entry points, adversary access requirements, and the minimum unlearning robustness level required to provide a meaningful defense\. BS = Behavioral Suppression,
RA = Representation\-level Attenuation, TF = True Forgetting, MIA = Membership Inference Attack\.A critical observation from this table is that the two threat categories posing the highest security risk, hazardous knowledge and backdoor associations, both require true forgetting as the minimum meaningful defense\. No method reviewed in this survey reliably achieves true forgetting\. This gap between the defense level required and the defense level available defines the central security problem the field must address\.

### 2\.2Adversary Taxonomy

Adversary capabilities in the LLM unlearning context are best formalized through the black\-box, gray\-box, white\-box classification standard from adversarial machine learningVassilevet al\.\([2025](https://arxiv.org/html/2607.16227#bib.bib99)\)\. The taxonomy matters because the appropriate unlearning method and evaluation protocol depend entirely on which adversary class the deployment context must defend against\. The majority of published evaluations in the LLM unlearning literature assume black\-box conditionsLuckiet al\.\([2025](https://arxiv.org/html/2607.16227#bib.bib88)\), but this assumption understates the threat for most real\-world deployments\.

Ablack\-box adversaryhas query\-only access: prompts in, outputs out, no model internals\. The available attacks are jailbreak promptingWeiet al\.\([2023](https://arxiv.org/html/2607.16227#bib.bib7)\)and automated adversarial prefix search via methods such as GCGZouet al\.\([2023](https://arxiv.org/html/2607.16227#bib.bib106)\)\. Behavioral suppression is sufficient against casual black\-box extraction but insufficient against a determined adversary willing to run systematic adversarial search\.

Agray\-box adversaryadditionally knows structural information about the model, such as its architecture, family, or the fact that a specific unlearning procedure was applied\. This information is freely available for every major open\-source model family\. The available attacks expand to membership inference, relearning using small auxiliary datasets, and soft\-prompt optimization over continuous input representationsSchwinnet al\.\([2024](https://arxiv.org/html/2607.16227#bib.bib87)\)\. Gray\-box conditions are the most practically relevant scenario for open\-source deployments\.

Awhite\-box adversaryhas full access to weights, gradients, and activations\. This is the most relevant class for evaluating true forgetting\. The empirical picture from Lucki et al\.Luckiet al\.\([2025](https://arxiv.org/html/2607.16227#bib.bib88)\)is unambiguous: orthogonalization of residual stream directions recovered WMDP\-Biology accuracy to 64\.7%, matching the pre\-unlearning baseline of 64\.4%, without modifying any model weights\. Between 0\.9% and 2\.4% of model weights carry the entire unlearning effect across methods; pruning those weights restores hazardous knowledge performance by at least 10 percentage points in every method tested\. Existing unlearning methods are not meaningfully more robust than safety fine\-tuning: both primarily obfuscate knowledge rather than remove it\.

The taxonomy maps directly onto the three\-level framework\. Behavioral suppression holds only against unsophisticated black\-box adversaries\. Representation\-level attenuation raises the cost for gray\-box adversaries but does not close the attack surface\. True forgetting is the only level that provides any robustness against white\-box adversaries, because it corresponds to a model state where the target knowledge no longer exists in the parameter space to be recovered\. The adversary need not be malicious: accidental corpus contamination, cascading errors in data pipelines, and deliberate poisoning attacks all produce the same parametric outcome and the same unlearning requirement\.

### 2\.3Motivating Cases

Harmful factual associations in deployed LLMs have already produced documented legal, regulatory, and financial consequences\. In 2023, a lawyer using an LLM as a legal research tool submitted a court filing citing fabricated case citations; the court issued sanctions and awarded costs to opposing partiesUnited States District Court, S\.D\.N\.Y\. \([2023](https://arxiv.org/html/2607.16227#bib.bib102)\); Mageshet al\.\([2024](https://arxiv.org/html/2607.16227#bib.bib10)\)\. In the same year, the Italian data protection authority suspended ChatGPT in Italy under GDPR Article 17 \(right to erasure\), prompting OpenAI to implement user erasure mechanisms, one of the first documented cases of an LLM provider deploying compliance measures directly tied to machine unlearning obligationsGarante per la Protezione dei Dati Personali \([2023](https://arxiv.org/html/2607.16227#bib.bib103)\); Mantelero \([2013](https://arxiv.org/html/2607.16227#bib.bib101)\)\. In 2024, the Civil Resolution Tribunal of British Columbia ruled that Air Canada bore full liability for incorrect fare policy information provided by its chatbot, establishing that operators cannot disclaim responsibility for LLM\-generated outputs through policy language aloneCivil Resolution Tribunal of British Columbia \([2024](https://arxiv.org/html/2607.16227#bib.bib104)\)\. These cases share a common structure: a harmful or incorrect factual association in a deployed model caused measurable harm, and in two of the three no technical remediation was available or applied\. Unlearning is the field’s response to precisely this class of failure\.

### 2\.4Unlearning as a Cyber Defense Layer

Positioned within a cyber defense architecture, unlearning functions as a corrective control: it does not prevent harmful associations from being encoded during training, but provides a mechanism for removing or suppressing them after the fact, reducing the inference\-time attack surface and enabling compliance with data deletion mandates\. This positioning has three implications for how the methods and their limitations should be interpreted\.

The first implication is that the security guarantee of an unlearning method is only as strong as the adversary class it has been tested against\. Behavioral suppression, the outcome most gradient\-based methods reliably achieve, corresponds to a defense that holds under black\-box, non\-adaptive evaluation conditions\. The moment an adversary gains gray\-box or white\-box access, or applies adaptive search rather than standard prompting, the guarantee degrades or disappears entirely, as documented in Section[6\.4](https://arxiv.org/html/2607.16227#S6.SS4)\. Characterizing a method as having “successfully unlearned” based on standard benchmark performance is therefore a statement about behavioral suppression under restricted conditions, not a statement about security\.

The second implication is that the evaluation gap in the current literature, the consistent finding that existing protocols assess output\-level suppression while adversarial attacks demonstrate parameter\-level retention, is not merely a methodological problem\. It is a security audit gap: the field cannot currently certify that any deployed unlearning intervention has achieved true knowledge removal rather than surface\-level concealment\. This is the equivalent of a penetration test that checks whether a lock is engaged but does not attempt to pick it\. The adversarial evaluations in Section[6\.4](https://arxiv.org/html/2607.16227#S6.SS4)are the closest the field has come to conducting genuine security audits of unlearning methods, and their findings are consistently negative\.

The third implication is that the choice of unlearning method must be informed by the threat model of the deployment context\. Table[2](https://arxiv.org/html/2607.16227#S2.T2)maps each threat category to the minimum unlearning robustness level required for a meaningful defense\. A deployment that faces only black\-box extraction threats for copyrighted content can rely on behavioral suppression and standard gradient ascent methods\. A deployment in a dual\-use research environment where hazardous knowledge is the primary concern and where adversaries may have model access requires true forgetting as the target, which no current method reliably provides\. This mismatch between required and available defense levels is the central open problem in the field\.

A genuinely robust cyber defense posture for LLM deployment would require three things that the current literature does not yet provide\. First, it would require unlearning methods whose guarantees extend to white\-box adversaries, meaning methods that achieve representation\-level removal rather than output\-level suppression\. Second, it would require evaluation protocols that test unlearned models against adaptive adversaries with knowledge of the unlearning procedure applied, rather than against static prompting distributions that any competent attacker would quickly exhaust\. Third, it would require a standardized threat model taxonomy of the kind sketched in Table[2](https://arxiv.org/html/2607.16227#S2.T2), so that deployment contexts can specify their adversary class and method selection can be calibrated accordingly\. The gradient\-based methods surveyed in Section[4](https://arxiv.org/html/2607.16227#S4)are evaluated against these three requirements throughout the remainder of this survey\.

## 3Preliminaries

This section introduces the fundamental concepts required to understand unlearning in large language models, covering LLM architecture and knowledge representation, the taxonomy of machine unlearning approaches, the conceptual framework distinguishing suppression from forgetting, and the mathematical foundations of gradient\-based unlearning\.

### 3\.1Large Language Models

Large language models are powerful deep learning systems that can understand and generate natural language with high coherence and contextual awarenessy Arcas \([2022](https://arxiv.org/html/2607.16227#bib.bib16)\); Naveed and others \([2025](https://arxiv.org/html/2607.16227#bib.bib17)\)\. Most modern LLMs are based on the Transformer architecture, which relies on self\-attention to capture relationships between words in a sequence\. Early models such as GPTRadfordet al\.\([2018](https://arxiv.org/html/2607.16227#bib.bib18)\)and BERTDevlinet al\.\([2019](https://arxiv.org/html/2607.16227#bib.bib19)\)introduced the idea of large\-scale pretraining on text data followed by fine\-tuning for specific tasks\. Further improvements such as instruction tuning and alignment with human preferences have made these models more usable in real\-world settingsLiu and others \([2024](https://arxiv.org/html/2607.16227#bib.bib21)\); Liet al\.\([2024a](https://arxiv.org/html/2607.16227#bib.bib22)\)\. Over time, models have grown from millions to hundreds of billions of parameters, with systems like GPT\-3 and PaLM demonstrating significant performance gains at larger sizesBrown and others \([2020](https://arxiv.org/html/2607.16227#bib.bib23)\); Chowdhery and others \([2023](https://arxiv.org/html/2607.16227#bib.bib24)\)\. This scale also makes training and updating these models computationally expensiveAcharyaet al\.\([2023](https://arxiv.org/html/2607.16227#bib.bib25)\); Fieldset al\.\([2024](https://arxiv.org/html/2607.16227#bib.bib26)\), and knowledge in LLMs is distributed across many parameters rather than stored explicitly, which makes it difficult to modify or remove specific information once the model has been trained\.

### 3\.2Knowledge Storage in LLMs

Unlike traditional databases, large language models do not store information in an explicit or easily retrievable form\. Knowledge is encoded implicitly within model parameters through the training process\. During pretraining, LLMs learn statistical patterns, associations, and contextual relationships from large\-scale text corporaPeykaniet al\.\([2025](https://arxiv.org/html/2607.16227#bib.bib27)\); Trust and Minghim \([2024](https://arxiv.org/html/2607.16227#bib.bib28)\), resulting in distributed representations across millions or billions of parameters\. Individual facts are not stored in isolated units but are entangled with other learned representations, often shared across multiple contexts\. Modifying or removing a particular piece of information can therefore unintentionally affect unrelated knowledgeGevaet al\.\([2021b](https://arxiv.org/html/2607.16227#bib.bib29)\); Menget al\.\([2022](https://arxiv.org/html/2607.16227#bib.bib30)\)\. Recent studies have shown that LLMs can memorize portions of their training data, including rare or sensitive information, which may be reproduced under certain prompts, while much of their knowledge is generalized rather than memorizedCarliniet al\.\([2022](https://arxiv.org/html/2607.16227#bib.bib4)\)\. This ambiguity further complicates efforts to remove specific information reliably, and the entangled and implicit nature of knowledge representation is one of the key reasons why unlearning remains a challenging problem in large\-scale modelsSepahvandet al\.\([2025](https://arxiv.org/html/2607.16227#bib.bib31)\)\.

### 3\.3Machine Unlearning: Taxonomy and Foundations

Machine unlearning refers to the process of removing the influence of specific data from a trained model, with the goal of making the updated model behave as if the removed data had never been seen during trainingXuet al\.\([2024](https://arxiv.org/html/2607.16227#bib.bib32)\)\. The concept was first formalized to address data deletion requirements in machine learning systems, particularly in settings involving privacy, regulatory compliance, and erroneous or harmful dataCao and Yang \([2015](https://arxiv.org/html/2607.16227#bib.bib33)\)\. Given training data𝒟\\mathcal\{D\}and forget set𝒟f⊂𝒟\\mathcal\{D\}\_\{f\}\\subset\\mathcal\{D\}, the unlearning algorithmUUtakes the original modelA​\(𝒟\)A\(\\mathcal\{D\}\)and produces an unlearned model:

U​\(𝒟,𝒟f,A​\(𝒟\)\)→Mu≈A​\(𝒟∖𝒟f\)U\(\\mathcal\{D\},\\mathcal\{D\}\_\{f\},A\(\\mathcal\{D\}\)\)\\rightarrow M\_\{u\}\\approx A\(\\mathcal\{D\}\\setminus\\mathcal\{D\}\_\{f\}\)\(1\)
Existing approaches fall into two broad categories\.Exact unlearningmethods guarantee that the distribution of the unlearned model is indistinguishable from a model retrained without the forget set:

Pr⁡\(A​\(𝒟∖𝒟f\)\)=Pr⁡\(U​\(𝒟,𝒟f,A​\(𝒟\)\)\)\\Pr\\\!\\left\(A\(\\mathcal\{D\}\\setminus\\mathcal\{D\}\_\{f\}\)\\right\)=\\Pr\\\!\\left\(U\(\\mathcal\{D\},\\mathcal\{D\}\_\{f\},A\(\\mathcal\{D\}\)\)\\right\)\(2\)
Exact unlearning is computationally infeasible for large\-scale LLMs\. Approaches such as SISA trainingYanet al\.\([2022](https://arxiv.org/html/2607.16227#bib.bib34)\)offer exact unlearning guarantees by partitioning training data and retaining intermediate checkpoints, but their overhead is prohibitive at LLM scale\.

Approximate unlearningaccepts bounded divergence in exchange for computational tractability\. The\(ϵ,δ\)\(\\epsilon,\\delta\)\-approximate unlearning guaranteeGuoet al\.\([2019](https://arxiv.org/html/2607.16227#bib.bib35)\)formalizes this as:

Pr⁡\(U​\(𝒟,z,A​\(𝒟\)\)∈τ\)≤eϵ​Pr⁡\(A​\(𝒟∖z\)∈τ\)\+δ\\Pr\\\!\\left\(U\(\\mathcal\{D\},z,A\(\\mathcal\{D\}\)\)\\in\\tau\\right\)\\leq e^\{\\epsilon\}\\Pr\\\!\\left\(A\(\\mathcal\{D\}\\setminus z\)\\in\\tau\\right\)\+\\delta\(3\)
for any subsetτ\\tauof the hypothesis space, whereϵ\\epsiloncontrols the approximation tolerance andδ\\deltais a failure probability slack\. Gradient\-based methods, influence\-function updates, and model editing approaches all fall within this categoryGuoet al\.\([2019](https://arxiv.org/html/2607.16227#bib.bib35)\); Xuet al\.\([2024](https://arxiv.org/html/2607.16227#bib.bib32)\)\.

A key challenge in machine unlearning lies in verification\. Even when a model appears to have forgotten specific information, residual traces may still persist in its parameters and can be revealed under carefully constructed prompts or adversarial settings\. This raises an important distinction between the true deletion of knowledge and its suppression at the behavioral level\.

### 3\.4Forgetting vs Suppression: A Conceptual Framework

A central challenge in LLM unlearning lies in distinguishing between true removal of knowledge and its suppression at the behavioral levelSunet al\.\([2025](https://arxiv.org/html/2607.16227#bib.bib41)\)\. While many existing methods aim to eliminate the influence of specific data, their effects often manifest only in model outputs rather than internal representationsWuet al\.\([2025](https://arxiv.org/html/2607.16227#bib.bib42)\); Hatami and Monosov \([2026](https://arxiv.org/html/2607.16227#bib.bib43)\)\.

To clarify this distinction, this survey introduces a conceptual framework that categorizes unlearning into three levels, which simultaneously functions as a robustness ladder for the adversary taxonomy in Section[2\.2](https://arxiv.org/html/2607.16227#S2.SS2)\.

\(1\) Behavioral Suppression\.The model no longer produces outputs associated with the forget set under standard prompting conditions\. The underlying knowledge may still persist in model parameters and can potentially be recovered through adversarial prompts or alternative decoding strategiesZhanget al\.\([2023](https://arxiv.org/html/2607.16227#bib.bib44)\)\. Most gradient\-based methods fall into this category\. A behavioral suppression defense holds against black\-box adversaries under standard conditions\.

\(2\) Representation\-level Attenuation\.The influence of the forget data is reduced within the model’s internal representations\. The knowledge is weakened but not entirely removed, and partial recovery may still be possibleXuet al\.\([2023](https://arxiv.org/html/2607.16227#bib.bib45)\)\. This level corresponds to defenses that raise the cost for gray\-box adversaries but do not close the attack surface\.

\(3\) True Forgetting\.The model behaves equivalently to one trained from scratch without the forget setLe\-Khac and Vinh \([2025a](https://arxiv.org/html/2607.16227#bib.bib11)\)\. Both behavioral outputs and internal representations no longer contain information attributable to the removed data\. Achieving true forgetting in large language models remains an open challenge and corresponds to a defense that holds against all adversary classes, including white\-box access\.

This framework highlights that the absence of observable outputs related to the forget set does not necessarily imply true deletion of knowledge\. Many existing methods operate by suppressing the expression of information rather than removing it entirely\.

## 4Gradient\-Based Unlearning Methods

In this section, gradient\-based unlearning methods are reviewed, which form the core focus of this survey\. Existing approaches are categorized based on their optimization strategies and examined for whether these methods achieve genuine forgetting or primarily induce behavioral suppression\.

### 4\.1Problem Formulation

Gradient\-based unlearning aims to remove the influence of specific data points by updating model parameters through gradient optimization, adjusting learned parameters so that the contribution of the forget set is reducedLe\-Khac and Vinh \([2025b](https://arxiv.org/html/2607.16227#bib.bib36)\)\.

Let the original training dataset be𝒟=𝒟r∪𝒟f\\mathcal\{D\}=\\mathcal\{D\}\_\{r\}\\cup\\mathcal\{D\}\_\{f\}, where𝒟r\\mathcal\{D\}\_\{r\}represents the retain set and𝒟f\\mathcal\{D\}\_\{f\}represents the forget set\. The objective of unlearning is to obtain updated parametersθ′\\theta^\{\\prime\}that approximate a model trained only on𝒟r\\mathcal\{D\}\_\{r\}:

θ′≈arg⁡minθ​∑\(x,y\)∈𝒟rℓ​\(fθ​\(x\),y\)\\theta^\{\\prime\}\\approx\\arg\\min\_\{\\theta\}\\sum\_\{\(x,y\)\\in\\mathcal\{D\}\_\{r\}\}\\ell\(f\_\{\\theta\}\(x\),y\)\(4\)
A common approach is to combine gradient descent on the retain set with gradient ascent on the forget setGolatkaret al\.\([2020](https://arxiv.org/html/2607.16227#bib.bib38)\); Trippaet al\.\([2024](https://arxiv.org/html/2607.16227#bib.bib37)\):

θ′=θ−ηr​∇θℒ𝒟r​\(θ\)\+ηf​∇θℒ𝒟f​\(θ\)\\theta^\{\\prime\}=\\theta\-\\eta\_\{r\}\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\mathcal\{D\}\_\{r\}\}\(\\theta\)\+\\eta\_\{f\}\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\mathcal\{D\}\_\{f\}\}\(\\theta\)\(5\)
whereηr\\eta\_\{r\}andηf\\eta\_\{f\}are step sizes controlling retention and forgetting respectively\. The first term maintains performance on the retain set and the second term pushes the model away from representations associated with the forget set\. Balancing these opposing objectives is challenging and often leads to trade\-offs between forgetting effectiveness and model utilityGuoet al\.\([2019](https://arxiv.org/html/2607.16227#bib.bib35)\); Anjarlekar and Pombra \([2025](https://arxiv.org/html/2607.16227#bib.bib39)\)\. Despite their scalability, gradient\-based methods do not guarantee complete removal of information\. Recent work shows that supposedly unlearned information can still be recovered under certain prompting or decoding strategies, suggesting these methods may only suppress knowledge rather than delete itWang \([2024](https://arxiv.org/html/2607.16227#bib.bib40)\)\.

Directly retraining is computationally infeasible for large\-scale LLMsLi and others \([2026](https://arxiv.org/html/2607.16227#bib.bib46)\)\. Gradient\-based unlearning instead formulates this as a constrained optimization problem expressed as a trade\-off between two competing objectivesCevalloset al\.\([2025](https://arxiv.org/html/2607.16227#bib.bib47)\); Huang and others \([2024](https://arxiv.org/html/2607.16227#bib.bib48)\):

minθ⁡ℒ𝒟r​\(θ\)−λ​ℒ𝒟f​\(θ\)\\min\_\{\\theta\}\\ \\mathcal\{L\}\_\{\\mathcal\{D\}\_\{r\}\}\(\\theta\)\-\\lambda\\,\\mathcal\{L\}\_\{\\mathcal\{D\}\_\{f\}\}\(\\theta\)\(6\)
whereλ\\lambdacontrols the strength of unlearning\. Increasing the influence of the second term promotes stronger forgetting but may degrade model utility, while prioritizing the first term preserves performance at the cost of incomplete removal\. Effective unlearning in LLMs is also evaluated along multiple dimensions including effectiveness, utility preservation, efficiency, and robustness to adversarial recoveryLe\-Khac and Vinh \([2025a](https://arxiv.org/html/2607.16227#bib.bib11)\)\.

### 4\.2Gradient Ascent and Descent Approaches

Gradient ascent and descent based methods form the most direct and widely used approach to unlearning in large language models, operating by explicitly modifying model parameters through gradient updates that simultaneously preserve useful knowledge while removing the influence of the forget setLi and others \([2025](https://arxiv.org/html/2607.16227#bib.bib49)\); Huanget al\.\([2026](https://arxiv.org/html/2607.16227#bib.bib50)\); Fenget al\.\([2024](https://arxiv.org/html/2607.16227#bib.bib51)\)\.

Given a trained model with parametersθ\\theta, gradient descent on𝒟r\\mathcal\{D\}\_\{r\}maintains model performance while gradient ascent on𝒟f\\mathcal\{D\}\_\{f\}reverses the learning signal associated with unwanted data, yielding the update rule already stated in Equation[5](https://arxiv.org/html/2607.16227#S4.E5), whereηr\\eta\_\{r\}andηf\\eta\_\{f\}control the strength of retention and forgetting respectively\. The ascent term increases the loss on𝒟f\\mathcal\{D\}\_\{f\}, effectively discouraging the model from producing outputs consistent with the forget data\.

#### 4\.2\.1Variants and Structural Refinements

In practice, different variants of this approach modify how gradients are applied\. Some methods perform alternating updates on retain and forget batches; others combine both objectives into a single optimization stepGolatkaret al\.\([2020](https://arxiv.org/html/2607.16227#bib.bib38)\); Guoet al\.\([2019](https://arxiv.org/html/2607.16227#bib.bib35)\)\. More advanced approaches restrict updates to specific layers or parameter subsets to reduce interference with retained knowledgeMenget al\.\([2022](https://arxiv.org/html/2607.16227#bib.bib30),[2023](https://arxiv.org/html/2607.16227#bib.bib52)\)\.

Gradient difference methodsmodify the standard ascent objective by computing the difference between the forget\-set loss and the retain\-set loss, rather than applying unconstrained ascent on the forget set alone\. The following abstract formulation, introduced by Bu et al\.Buet al\.\([2025](https://arxiv.org/html/2607.16227#bib.bib95)\), encompasses several existing methods as special cases under a unified multi\-task optimization view\. LettinggRg\_\{R\}andgFg\_\{F\}denote the retain\-set and forget\-set gradients respectively, the dynamic scalarization update is:

θt\+1=θt−ηt​\[ct⋅gR​\(θt\)−\(1−ct\)⋅gF​\(θt\)\]\\theta\_\{t\+1\}=\\theta\_\{t\}\-\\eta\_\{t\}\\left\[c\_\{t\}\\cdot g\_\{R\}\(\\theta\_\{t\}\)\-\(1\-c\_\{t\}\)\\cdot g\_\{F\}\(\\theta\_\{t\}\)\\right\]\(7\)
Settingct=1c\_\{t\}=1recovers gradient descent on the retain set;ct=0c\_\{t\}=0recovers plain gradient ascent;ct=0\.5c\_\{t\}=0\.5recovers vanilla gradient difference\. This unification reveals that existing methods are Pareto optimal at convergence under mild conditions, but they may converge to different Pareto points depending on the choice ofctc\_\{t\}throughout training\. The key practical limitation is gradient magnitude imbalance: when the forget\-set and retain\-set gradients differ substantially in norm, one dominates the update direction regardless of its semantic relevance to the forgetting objective\.

Normalized Gradient Difference \(NGDiff\)Buet al\.\([2025](https://arxiv.org/html/2607.16227#bib.bib95)\)addresses this by settingctc\_\{t\}dynamically based on gradient norms, yielding:

gNGDiff​\(gR,gF\):=gR‖gR‖−gF‖gF‖g\_\{\\text\{NGDiff\}\}\(g\_\{R\},g\_\{F\}\):=\\frac\{g\_\{R\}\}\{\\\|g\_\{R\}\\\|\}\-\\frac\{g\_\{F\}\}\{\\\|g\_\{F\}\\\|\}\(8\)
This normalization gives independent control over the retain\-forget trade\-off regardless of gradient magnitudes\. NGDiff is paired with a Hessian\-based automatic learning\-rate scheduler, addressing the well\-documented sensitivity of gradient ascent methods to learning rate choice\. On TOFU with Llama\-2\-7B, NGDiff achieves 40% higher model utility than standard gradient difference baselines while maintaining comparable unlearning performanceBuet al\.\([2025](https://arxiv.org/html/2607.16227#bib.bib95)\)\.

Saliency\-based methodssuch as SalUnFanet al\.\([2023](https://arxiv.org/html/2607.16227#bib.bib57)\)extend the basic framework by computing a weight saliency map over the forget set and applying gradient updates selectively to high\-saliency parameters\. By concentrating forgetting pressure on parameters most associated with the target data, these methods reduce collateral degradation on the retain set\. However, because saliency is derived from output\-level loss gradients rather than internal representational structure, the updates remain coupled to behavioral suppression rather than representational removal\.

Task\-agnostic gradient unlearningTrippaet al\.\([2024](https://arxiv.org/html/2607.16227#bib.bib37)\)decouples the unlearning objective from task\-specific loss formulations, framing forgetting as a general parameter perturbation applicable across model types\. While this improves practical generalizability, it introduces additional uncertainty regarding which internal representations are being modified\.

Hybrid approachescombine gradient\-based erasure with targeted knowledge insertion\. Methods such as Forget\-for\-GetLiet al\.\([2025](https://arxiv.org/html/2607.16227#bib.bib58)\)apply a lightweight gradient update to suppress the forget set followed by a retain\-reinforcement step designed to stabilize model utility\. Fine\-grained pluggable variantsFenget al\.\([2024](https://arxiv.org/html/2607.16227#bib.bib51)\)further decompose the ascent signal to operate at the concept level, allowing more selective application across layers\. Despite these structural improvements, empirical evaluations consistently show that hybrid gradient methods remain susceptible to adversarial recoveryCevalloset al\.\([2025](https://arxiv.org/html/2607.16227#bib.bib47)\); Hatami and Monosov \([2026](https://arxiv.org/html/2607.16227#bib.bib43)\)\.

Orthogonality\-based methodsintroduce a geometrically motivated constraint, projecting forgetting pressure onto subspaces orthogonal to the directions associated with retained knowledgeFenget al\.\([2025](https://arxiv.org/html/2607.16227#bib.bib56)\)\. This formulation aims to reduce interference with the retain set without requiring explicit parameter masking\. However, the projection is computed over output\-level gradients rather than internal representational structure, and empirical evaluations suggest that orthogonality constraints reduce observable leakage under standard conditions without providing guarantees of representational removal\.

KL\-regularized variantsextend the gradient difference approach by penalizing the KL divergence between the output distributions of the original and updated models, constraining the magnitude of parameter drift during the forgetting process\. Both gradient difference and KL\-regularized formulations are widely adopted as baselines in evaluation pipelines\.

Negative Preference Optimization \(NPO\)Zhanget al\.\([2024](https://arxiv.org/html/2607.16227#bib.bib107)\)reframes the unlearning objective as a preference\-based optimization problem, adapting ideas from Direct Preference Optimization \(DPO\)Rafailovet al\.\([2023](https://arxiv.org/html/2607.16227#bib.bib112)\), a method for aligning language models to human preferences by directly optimizing a policy to maximize relative preference probability without explicit reward modeling\. NPO applies this framework exclusively to negative samples, treating forget\-set data as the dispreferred output\. Lettingπθ\\pi\_\{\\theta\}denote the model distribution andπθ0\\pi\_\{\\theta\_\{0\}\}the reference model distribution, the NPO objective is:

minθ⁡2β​𝔼𝒟f​\[log⁡\(1\+\(πθ​\(yu\|xu\)πθ0​\(yu\|xu\)\)β\)\]\\min\_\{\\theta\}\\frac\{2\}\{\\beta\}\\mathbb\{E\}\_\{\\mathcal\{D\}\_\{f\}\}\\\!\\left\[\\log\\\!\\left\(1\+\\left\(\\frac\{\\pi\_\{\\theta\}\(y\_\{u\}\|x\_\{u\}\)\}\{\\pi\_\{\\theta\_\{0\}\}\(y\_\{u\}\|x\_\{u\}\)\}\\right\)^\{\\\!\\beta\}\\right\)\\right\]\(9\)
whereβ\\betacontrols the smoothness of the reweighting\. Rather than directly maximizing loss on the forget set, NPO penalizes the model for assigning higher probability to forget\-set outputs relative to the reference model distribution\. This anchoring mechanism limits the magnitude of parameter updates and mitigates the instability associated with unconstrained gradient ascent, reducing catastrophic model degradation risk while maintaining competitive forgetting effectiveness\.

However, a significant failure mode of NPO\-based methods is the squeezing effectLiet al\.\([2026](https://arxiv.org/html/2607.16227#bib.bib91)\)\. Because softmax normalization means that lowering the probability of one output necessarily raises the probability of others in the vocabulary, NPO\-based unlearningZhanget al\.\([2024](https://arxiv.org/html/2607.16227#bib.bib107)\)redistributes probability mass into semantically related paraphrases of the forget\-set content\. A model that no longer produces the exact target response may still produce semantically equivalent variants that retain the intended knowledge in an accessible form\. The Gradient Difference objectiveLiet al\.\([2026](https://arxiv.org/html/2607.16227#bib.bib91)\)is:

minθ⁡ℒGradDiff:=ℒGA​\(θ;𝒟f\)\+λ​𝔼𝒟r​\[−log⁡πθ​\(yr\|xr\)\]\\min\_\{\\theta\}\\mathcal\{L\}\_\{\\text\{GradDiff\}\}:=\\mathcal\{L\}\_\{\\text\{GA\}\}\(\\theta;\\mathcal\{D\}\_\{f\}\)\+\\lambda\\,\\mathbb\{E\}\_\{\\mathcal\{D\}\_\{r\}\}\\\!\\left\[\-\\log\\pi\_\{\\theta\}\(y\_\{r\}\|x\_\{r\}\)\\right\]\(10\)
while the plain Gradient Ascent objective is:

minθ⁡ℒGA:=𝔼𝒟f​\[log⁡πθ​\(yu\|xu\)\]\\min\_\{\\theta\}\\mathcal\{L\}\_\{\\text\{GA\}\}:=\\mathbb\{E\}\_\{\\mathcal\{D\}\_\{f\}\}\\\!\\left\[\\log\\pi\_\{\\theta\}\(y\_\{u\}\|x\_\{u\}\)\\right\]\(11\)
Recent work has also addressed the over\-erasure problem: existing gradient ascent and gradient difference methods perform full\-information unlearning, targeting everything associated with the forget set including knowledge also independently supported by the retain set\. Forgetting\-MarIXuet al\.\([2026](https://arxiv.org/html/2607.16227#bib.bib94)\)addresses this by targeting only the marginal information the forget data contributes beyond the retain set, measured via the Jensen\-Shannon divergence between next\-token distributions:

I​\(XMarI;Z\):=1T​∑t=1TJSD​\(ptd,ptr\)I\(X\_\{\\text\{MarI\}\};Z\):=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\mathrm\{JSD\}\\\!\\left\(p\_\{t\}^\{d\},\\,p\_\{t\}^\{r\}\\right\)\(12\)
whereptdp\_\{t\}^\{d\}is the next\-token distribution conditioning on the union of retain and forget sets \(the “dirty” distribution\) andptrp\_\{t\}^\{r\}is the distribution conditioning on the retain set only\. The full Forgetting\-MarI training loss combines a KL utility term with the MarI suppression term:

minθ⁡ℓKL​\(θ,r\)\+ℓMarI​\(θ,r,u\)\\displaystyle\\min\_\{\\theta\}\\;\\ell\_\{\\text\{KL\}\}\(\\theta,r\)\+\\ell\_\{\\text\{MarI\}\}\(\\theta,r,u\)\(13\)ℓKL​\(θ,r\):=DKL​\(pr​\(θ\)∥pr​\(θ0\)\)\\displaystyle\\ell\_\{\\text\{KL\}\}\(\\theta,r\)=D\_\{\\text\{KL\}\}\\\!\\left\(p^\{r\}\(\\theta\)\\,\\\|\\,p^\{r\}\(\\theta\_\{0\}\)\\right\)ℓMarI​\(θ,r,u\):=I​\(XMarI;Z\)\\displaystyle\\ell\_\{\\text\{MarI\}\}\(\\theta,r,u\)=I\(X\_\{\\text\{MarI\}\};Z\)
whererranduudenote the retain and unlearn set distribution parameters respectively, distinct from the smoothing rate notation used in Equation[14](https://arxiv.org/html/2607.16227#S4.E14)\. This formulation provides an explicit theoretical upper bound on the residual influence of the forget set, making Forgetting\-MarI the only method in its evaluation with a provable undetectability guarantee against membership inference\.

Label smoothing offers another stabilization strategy for gradient ascent instability\. Pang et al\.Panget al\.\([2025](https://arxiv.org/html/2607.16227#bib.bib92)\)show that plain gradient ascent is highly unstable under cross\-entropy loss, causing perplexity to explode to values on the order of107110^\{71\}on OPT\-2\.7B under the Harry Potter benchmark\. The proposed Smoothed Gradient Ascent \(SGA\) loss mixes the forget\-set gradient ascent loss with a normal\-data cross\-entropy term:

LSGA​\(θ\)=\(1−rs\+rsK\)​Lf​\(θ\)\+rsK​∑k=1KLp\(k\)​\(θ\)L\_\{\\text\{SGA\}\}\(\\theta\)=\\left\(1\-r\_\{s\}\+\\frac\{r\_\{s\}\}\{K\}\\right\)L\_\{f\}\(\\theta\)\+\\frac\{r\_\{s\}\}\{K\}\\sum\_\{k=1\}^\{K\}L\_\{p\}^\{\(k\)\}\(\\theta\)\(14\)
whereLfL\_\{f\}is the forget\-set gradient ascent loss,Lp\(k\)L\_\{p\}^\{\(k\)\}is the cross\-entropy loss on thekk\-th normal\-data sample \(retrieved from the retain set or generated by an external model\),KKis the number of normal data instances, andrs∈\[0,1\]r\_\{s\}\\in\[0,1\]is the smoothing rate \(a distinct use ofrsr\_\{s\}from the retain set parameterrrin Equation[13](https://arxiv.org/html/2607.16227#S4.E13)\)\. Settingrs=0r\_\{s\}=0recovers plain gradient ascent\. Theoretical guidance for choosingrsr\_\{s\}is provided by:

rs∗=⟨gf,us⟩‖us‖2,us≔g¯p−\(1−1K\)​gfr\_\{s\}^\{\*\}=\\frac\{\\langle g\_\{f\},\\,u\_\{s\}\\rangle\}\{\\\|u\_\{s\}\\\|^\{2\}\},\\quad u\_\{s\}\\coloneqq\\bar\{g\}\_\{p\}\-\\left\(1\-\\frac\{1\}\{K\}\\right\)g\_\{f\}\(15\)
wheregfg\_\{f\}is the forget\-set gradient andg¯p\\bar\{g\}\_\{p\}is the average normal\-data gradient\.

The instability of gradient ascent under cross\-entropy loss has also been given formal theoretical grounding by Garg et al\.Garget al\.\([2026](https://arxiv.org/html/2607.16227#bib.bib93)\), who prove through three theorems that gradient ascent drives the logits in transformer feedforward MLP layers toward infinity, causing weights and gradients to grow without bound\. Their proposed fix replaces the standard LoRA parameterization in MLP adapters with a sine\-based bounded activation that structurally prevents weight explosion, whereρ\\rhois the LoRA rank:

h=W0​x\+αρ​sin⁡\(ω​A​B⊤\)​x\+bh=W\_\{0\}x\+\\frac\{\\alpha\}\{\\rho\}\\sin\(\\omega AB^\{\\top\}\)x\+b\(16\)
whereW0W\_\{0\}is the frozen pretrained weight,AAandBBare the low\-rank adapter matrices,α\\alphais the LoRA scaling factor,ω\\omegais a frequency hyperparameter controlling the bound, andbbis a bias term\. The sine function prevents weight explosion because\|sin⁡\(⋅\)\|≤1\|\\sin\(\\cdot\)\|\\leq 1for any argument, regardless of the magnitude ofA​B⊤​xAB^\{\\top\}x\. GD\+Sine is reported as the only parameter\-efficient unlearning method to satisfy all four safety criteria on the MUSE benchmark simultaneously on Llama\-2\-7B, and the only method that scales stably to 70 billion parameters where standard GD\+LoRA diverges entirelyGarget al\.\([2026](https://arxiv.org/html/2607.16227#bib.bib93)\)\.

#### 4\.2\.2Theoretical Limitations of Gradient Ascent

Despite their simplicity and scalability, gradient ascent and descent approaches rest on a critical assumption: that the influence of the forget set can be effectively reversed through local gradient updates\. This assumption does not hold in large language models due to the highly non\-convex and path\-dependent nature of trainingMavrothalassitiset al\.\([2025](https://arxiv.org/html/2607.16227#bib.bib53)\)\. The parameter stateθ\\thetais not a linear superposition of contributions from individual data points, but rather the result of a complex optimization trajectory in which representations are continuously reshaped and entangled\.

Performing gradient ascent on𝒟f\\mathcal\{D\}\_\{f\}does not therefore constitute an inverse operation of the original training process\. It introduces a new optimization trajectory that increases loss on the forget set without necessarily removing the underlying representations that encode the forgotten informationGolatkaret al\.\([2020](https://arxiv.org/html/2607.16227#bib.bib38)\)\. This produces a fundamental mismatch between the intended objective of unlearning and its actual effect on model parameters\.

The most direct theoretical evidence for this failure is provided by Mavrothalassitis et al\.Mavrothalassitiset al\.\([2025](https://arxiv.org/html/2607.16227#bib.bib53)\), who prove that gradient ascent systematically fails to achieve forgetting in overparameterized networks\. Their findings show that ascent\-based updates increase loss on the forget set while leaving latent representations largely intact, confirming that the mechanism modifies output behavior without disrupting the internal pathways through which the knowledge is encoded\. Complementarily, Huang et al\.Huanget al\.\([2026](https://arxiv.org/html/2607.16227#bib.bib50)\)demonstrate that gradient ascent is not even a necessary condition for effective unlearning: memorization\-based alternatives achieve comparable forgetting without the optimization instability that characterizes loss maximization\.

Furthermore, knowledge in LLMs is distributed across a large number of parameters and shared across multiple contextsGevaet al\.\([2021a](https://arxiv.org/html/2607.16227#bib.bib54)\)\. Gradients computed on𝒟f\\mathcal\{D\}\_\{f\}capture only a partial and context\-dependent view of the associated representations\. Reversing these gradients weakens specific output behaviors but does not guarantee elimination of all latent pathways through which the information can be expressed\. This explains why previously “forgotten” information can often be recovered under adversarial prompting or extraction attacksCevalloset al\.\([2025](https://arxiv.org/html/2607.16227#bib.bib47)\); Wang and Chen \([2024](https://arxiv.org/html/2607.16227#bib.bib55)\)\.

The squeezing effectLiet al\.\([2026](https://arxiv.org/html/2607.16227#bib.bib91)\)further demonstrates that even when gradient ascent or NPO\-based methods appear successful under standard metrics, the underlying mechanism may be redistribution rather than removal\. Softmax normalization structurally prevents probability from being zeroed out: it can only be displaced\. For any token\-prediction\-based unlearning method, complete suppression of the target response necessarily shifts probability mass into adjacent regions of the output space, some of which will be semantically equivalent to the suppressed content\. This is a mathematical consequence of the softmax architecture, not a deficiency of any specific method, and it applies broadly to all gradient\-based approaches that optimize via next\-token prediction loss\.

Unconstrained gradient ascent is also susceptible to catastrophic collapse, in which aggressive loss maximization on the forget set causes parameter drift severe enough to degrade general model utilityAnjarlekar and Pombra \([2025](https://arxiv.org/html/2607.16227#bib.bib39)\); Mavrothalassitiset al\.\([2025](https://arxiv.org/html/2607.16227#bib.bib53)\)\. This failure mode arises because the loss landscape around𝒟f\\mathcal\{D\}\_\{f\}is not isolated from the parameter regions supporting unrelated capabilities\. This structural vulnerability motivated the development of bounded alternatives such as NPO, as well as the saliency\-based and orthogonality\-constrained methods discussed above\.

GRAILKim and others \([2025](https://arxiv.org/html/2607.16227#bib.bib97)\)addresses a related structural problem in multi\-domain unlearning, using gradient information from multiple domains to identify parameters associated with the unlearning scope, the retention scope, and overlapping representations, applying adaptive parameter\-wise localization to freeze retention\-critical parameters while applying unlearning updates to targeted parameters\. GRAIL reports up to 17% stronger knowledge retention than prior state\-of\-the\-art methods on privacy and copyright domains\. This finding directly supports the argument that knowledge in LLMs is distributed and entangled: even methods with sophisticated parameter selection cannot fully isolate the forget set from the retain set because the two share parameter regions\.

#### 4\.2\.3Classification Under the Proposed Framework

Under the framework introduced in Section[3\.4](https://arxiv.org/html/2607.16227#S3.SS4), gradient ascent and descent methods are best characterized as achieving behavioral suppression\. Their updates reduce the probability of generating outputs associated with the forget set under typical conditions, but do not reliably disrupt the distributed internal representations that encode this informationSunet al\.\([2025](https://arxiv.org/html/2607.16227#bib.bib41)\); Hatami and Monosov \([2026](https://arxiv.org/html/2607.16227#bib.bib43)\)\. Even structurally refined variants including saliency\-based, task\-agnostic, orthogonality\-constrained, and hybrid approaches operate primarily on output\-level loss signals and therefore do not provide guarantees of representation\-level attenuation, let alone true forgetting\. Bounded formulations such as NPO and KL\-regularized variants improve stability and reduce the risk of catastrophic collapse, but similarly operate at the behavioral level and do not resolve the underlying representational entanglement\.

The fundamental constraint lies in the non\-reversibility of the training process\. Since learning in deep networks is not decomposable into independent contributions from individual data points, there exists no guarantee that local gradient updates can reconstruct the parameter state corresponding to training without𝒟f\\mathcal\{D\}\_\{f\}Golatkaret al\.\([2020](https://arxiv.org/html/2607.16227#bib.bib38)\)\. Gradient ascent and descent methods should therefore be understood as approximate, behavior\-level interventions rather than mechanisms for exact knowledge removal\.

Table[3](https://arxiv.org/html/2607.16227#S4.T3)summarizes the methods covered in this section across the three dimensions of the framework\.

Table 3:Comparison of gradient\-based unlearning variants\. Year of first publication shown in parentheses\. Forgetting level follows the framework in Section[3\.4](https://arxiv.org/html/2607.16227#S3.SS4)\. Robustness and overhead are relative comparisons within this method family\. BS = Behavioral Suppression, RA = Representation Attenuation \(approaching\)\.

### 4\.3Influence\-based Gradient Methods

Influence\-based gradient methods aim to remove the effect of specific training data by estimating and reversing their contribution to the learned model parametersHammoudeh and Lowd \([2024](https://arxiv.org/html/2607.16227#bib.bib59)\), attempting to approximate how individual data points influence the final model during trainingBrophyet al\.\([2023](https://arxiv.org/html/2607.16227#bib.bib60)\)\.

A common formulation uses influence functionsHammoudeh and Lowd \([2024](https://arxiv.org/html/2607.16227#bib.bib59)\), which estimate the change in model parameters when a training point is upweighted or removed\. Given a model trained with parametersθ\\theta, the influence of a data pointzzon the parameters can be approximated as:

d​θd​ϵ≈−Hθ−1​∇θℓ​\(z,θ\)\\frac\{d\\theta\}\{d\\epsilon\}\\approx\-H\_\{\\theta\}^\{\-1\}\\nabla\_\{\\theta\}\\ell\(z,\\theta\)\(17\)
whereHθH\_\{\\theta\}denotes the Hessian of the loss function with respect to model parameters andℓ​\(z,θ\)\\ell\(z,\\theta\)is the loss associated with data pointzz\. This expression captures how small perturbations in the training distribution affect the learned parameters\. The goal is to approximate the parameter update required to remove the influence of all samples in𝒟f\\mathcal\{D\}\_\{f\}:

θ′≈θ\+1\|𝒟f\|​∑z∈𝒟fHθ−1​∇θℓ​\(z,θ\)\\theta^\{\\prime\}\\approx\\theta\+\\frac\{1\}\{\|\\mathcal\{D\}\_\{f\}\|\}\\sum\_\{z\\in\\mathcal\{D\}\_\{f\}\}H\_\{\\theta\}^\{\-1\}\\nabla\_\{\\theta\}\\ell\(z,\\theta\)\(18\)
Recent work has explored scalable approximations of this idea, using Hessian\-vector products and stochastic estimation to avoid explicit computation of the Hessian inverseFanet al\.\([2025b](https://arxiv.org/html/2607.16227#bib.bib61)\); McKinney and others \([2026](https://arxiv.org/html/2607.16227#bib.bib62)\)\. More recent efforts introduce low\-rank or iterative approximations enabling partial influence removal without full retrainingKudelya and Shirnin \([2025](https://arxiv.org/html/2607.16227#bib.bib63)\)\.

However, influence functions are derived under assumptions of convexity and local smoothness that do not strictly hold for deep neural networksBasuet al\.\([2021](https://arxiv.org/html/2607.16227#bib.bib64)\)\. In highly non\-convex models such as LLMs, the estimated influence of a data point may only reflect a local linearization of the loss landscape rather than its true global contribution\. Furthermore, since representations are shared across many parameters and contexts, the influence of a data point cannot be cleanly isolated or inverted\. Empirical findings show that even after influence\-based updates, sensitive or memorized information can still be extracted under targeted prompting or adversarial evaluationPatilet al\.\([2024](https://arxiv.org/html/2607.16227#bib.bib65)\)\.

From the perspective of the proposed framework, influence\-based methods aim for representation\-level attenuation rather than purely behavioral suppression\. By explicitly modeling the contribution of training data to parameter updates, they attempt to weaken the internal encoding of the forget set\. However, due to approximation errors, non\-convexity, and representational entanglement, this attenuation is often incomplete\. Influence\-based approaches occupy an intermediate position: they go beyond surface\-level behavioral suppression by targeting parameter\-level contributions, but fall short of true forgetting\.

### 4\.4Parameter and Loss Function Based Gradient Methods

Parameter and loss function based methods approach unlearning by modifying the training objective to control how model parameters are updated during the forgetting processChenget al\.\([2026](https://arxiv.org/html/2607.16227#bib.bib66)\); Wanget al\.\([2026](https://arxiv.org/html/2607.16227#bib.bib67)\)\. Instead of explicitly reversing gradients, these methods introduce additional penalties or constraints that limit the influence of the forget set while preserving important knowledge from the retain set\.

Recent work has introduced more structured constraints on parameter updates\. Methods based on remain\-preserving optimization constrain updates to lie in subspaces that minimally affect retained knowledgeChenget al\.\([2025](https://arxiv.org/html/2607.16227#bib.bib68)\); Luoet al\.\([2026](https://arxiv.org/html/2607.16227#bib.bib69)\)\. LoRA\-based unlearning approaches restrict updates to low\-rank parameter adapters, enabling localized modification of model behavior without altering the full parameter spaceBoer \([2026](https://arxiv.org/html/2607.16227#bib.bib70)\)\. Parameter masking and saliency\-based constraints as in SalUnFanet al\.\([2023](https://arxiv.org/html/2607.16227#bib.bib57)\)further refine this approach by selectively updating only parameters deemed important for the forget set\.

Despite their improved control and stability, these methods rely on a critical assumption: that the knowledge associated with the forget set can be localized or approximated within a constrained subset of parameters\. Empirical evidence suggests that knowledge in large language models is highly distributed and entangled across layers and contextsGevaet al\.\([2021a](https://arxiv.org/html/2607.16227#bib.bib54)\), so restricting updates to specific subspaces may only partially remove the influence of the forget set\. Constrained unlearning methods can still leak information under adversarial prompting or extraction attacks even when standard metrics indicate successful forgettingPatilet al\.\([2024](https://arxiv.org/html/2607.16227#bib.bib65)\)\.

Under the proposed framework, parameter and loss function based methods achieve a combination of behavioral suppression and partial representation\-level attenuation\. While they reduce the sensitivity of the model to the forget set and improve stability compared to unconstrained gradient ascent, they do not guarantee full removal of the underlying representations\.

### 4\.5Limitations and Failure Modes

Despite significant progress, gradient\-based unlearning methods exhibit several fundamental limitations that challenge their reliability in large language models\.

A primary issue is incomplete forgetting\. Due to the distributed and entangled nature of knowledge in LLMs, modifying a subset of parameters does not ensure that all traces of the target information are removed\. Residual representations may persist and can be reactivated under specific prompts or contexts\. Gradient\-based methods are also vulnerable to adversarial recovery, as demonstrated comprehensively by Lucki et al\.Luckiet al\.\([2025](https://arxiv.org/html/2607.16227#bib.bib88)\), who showed that fine\-tuning an RMU\-unlearned modelLi \([2024](https://arxiv.org/html/2607.16227#bib.bib77)\)on as few as 10 unrelated examples restores most hazardous capabilities\.

Another critical limitation is the trade\-off between forgetting and utility\. Strong unlearning updates, particularly those involving aggressive gradient ascent, can degrade model performance on unrelated tasks\. The lack of standardized evaluation protocols makes it difficult to reliably assess unlearning effectiveness\. Existing evaluation metrics such as Verbatim Probability, ROUGE\-based overlap, and membership inference attacks primarily assess output\-level behavior rather than internal representation removalTan \([2026](https://arxiv.org/html/2607.16227#bib.bib71)\)\. Apparent success under these metrics does not necessarily imply true forgetting\.

The over\-erasure problem, identified by Xu et al\.Xuet al\.\([2026](https://arxiv.org/html/2607.16227#bib.bib94)\), adds another dimension: existing methods remove more than they should, erasing knowledge supported by the retain set as well as by the forget set\. This indiscriminate removal produces substantial utility drops and suggests that the correct unlearning objective should target only the marginal contribution of the forget data\.

Collectively these limitations highlight a central challenge: current gradient\-based methods may achieve apparent forgetting, but often fail to provide strong guarantees of true knowledge removal\.

## 5Non\-Gradient\-Based Unlearning Approaches

While gradient\-based methods dominate LLM unlearning research, a rich ecosystem of non\-gradient\-based approaches has emerged that addresses different aspects of the forgetting problem\. These methods offer complementary strengths: they avoid the optimization instability inherent in gradient ascent, can operate without access to forget\-set gradients, and in some cases provide stronger formal guarantees\. This section covers three principal families: model editing and knowledge editing, differential privacy based approaches, and inference\-time and input modification methods\.

### 5\.1Model Editing and Knowledge Editing

Model editing, also called knowledge editing, seeks to locally modify specific factual associations within a trained LLM without retraining or performing gradient ascent over the forget setGevaet al\.\([2021a](https://arxiv.org/html/2607.16227#bib.bib54)\); Menget al\.\([2023](https://arxiv.org/html/2607.16227#bib.bib52)\)\. The core intuition is that factual knowledge in transformer models is partially localized in feedforward MLP layers, which function as key\-value memoriesGevaet al\.\([2021a](https://arxiv.org/html/2607.16227#bib.bib54)\)\. Causal mediation analysis can identify which specific layers and neurons encode a given factual association, and targeted weight modifications can update that association without broadly affecting the rest of the model\. While originally developed for knowledge updating rather than unlearning, these methods have been extensively adopted as unlearning baselines due to their ability to make targeted parametric modifications without full retrainingPatilet al\.\([2024](https://arxiv.org/html/2607.16227#bib.bib65)\)\.

ROME \(Rank\-One Model Editing\)Menget al\.\([2022](https://arxiv.org/html/2607.16227#bib.bib30)\)applies this principle to perform single\-fact edits using a rank\-one update to the weights of a single MLP layer\. The method uses causal tracing to identify the layer where the target fact is most strongly encoded, then solves a constrained optimization problem to find the minimal weight perturbation that changes the model’s output for the target subject while leaving other factual associations intact\. ROME is effective for single targeted edits but degrades substantially as the number of edits increases, because the independence assumptions underlying the rank\-one update do not hold across multiple correlated edits\.

MEMIT \(Mass\-Editing Memory in a Transformer\)Menget al\.\([2023](https://arxiv.org/html/2607.16227#bib.bib52)\)extends ROME to support batch edits across multiple MLP layers simultaneously\. Rather than targeting a single layer, MEMIT distributes the edit across a range of layers using a least\-squares formulation, which improves scalability and reduces interference between simultaneous edits\. MEMIT can handle thousands of edits while maintaining model utility better than ROME, and has been applied as a baseline in several LLM unlearning evaluationsPatilet al\.\([2024](https://arxiv.org/html/2607.16227#bib.bib65)\)\.

SERAC \(Semi\-Parametric Editing with a Retrieval\-Augmented Counterfactual\)Mitchellet al\.\([2022a](https://arxiv.org/html/2607.16227#bib.bib113)\)takes a fundamentally different approach: rather than modifying model weights directly, SERAC maintains a small external memory of edits and routes queries through it at inference time\. When a query matches an edited fact, the counterfactual model overrides the base model’s response\. This avoids weight modification entirely, preserving model utility, but requires storing the forget data and does not achieve parametric removal\.

MEND \(Model Editor Networks with Gradient Decomposition\)Mitchellet al\.\([2022b](https://arxiv.org/html/2607.16227#bib.bib114)\)applies low\-rank transformations to efficiently apply edits via auxiliary networks trained to predict the optimal weight update for a given edit\. MEND is more computationally efficient than ROME and MEMIT for repeated edits but similarly operates at the behavioral level\.

From the perspective of the three\-level framework, model editing methods achieve behavioral suppression for the specific facts they target\. They do not achieve representation\-level attenuation because causal tracing identifies the most salient encoding location of a fact, not all locations where the fact’s influence is distributed across parameters\. Patil et al\.Patilet al\.\([2024](https://arxiv.org/html/2607.16227#bib.bib65)\)showed empirically that content deleted via ROME and MEMIT remains accessible in hidden states and via rephrased queries, highlighting that model editing faces the same parametric retention challenge as gradient\-based methods\.

A growing body of work directly compares model editing to unlearning baselines\.WISE \(Working memory In Side\-memory Editing\)Wanget al\.\([2024](https://arxiv.org/html/2607.16227#bib.bib109)\)addresses the limitations of single\-layer editing by introducing a dual parametric memory scheme, maintaining a main memory for pretrained knowledge and a side memory for edited knowledge, with a learned router directing queries to the appropriate memory\. This approach overcomes the impossible triangle of reliability, generalization, and locality that afflicts prior editing methods under sequential editing\.AlphaEditFanget al\.\([2025](https://arxiv.org/html/2607.16227#bib.bib110)\)projects parameter perturbations onto the null space of the preserved knowledge before applying them, theoretically guaranteeing that the output of the post\-edited model remains unchanged when queried about non\-targeted knowledge, and achieves an average 36\.7% performance improvement over standard locating\-then\-editing methods\.PMET \(Precise Model Editing in a Transformer\)Liet al\.\([2024b](https://arxiv.org/html/2607.16227#bib.bib111)\)improves upon ROME and MEMIT by simultaneously optimizing hidden states of both the Multi\-Head Self\-Attention and Feed\-Forward Network components, but updating only the FFN weights, producing more precise factual edits\. Both WISE and AlphaEdit have been found effective as unlearning baselines, particularly for pretrained knowledge, and excel in generating human\-aligned refusal responses\. The relationship between knowledge editing and unlearning motivates a hybrid perspective: editing identifies where knowledge is localized, while unlearning determines how to remove it\. Future methods combining causal tracing for localization with gradient\-based updates for removal may offer stronger forgetting guarantees than either approach alone\.

### 5\.2Differential Privacy Based Approaches

Differential privacy \(DP\) provides a mathematical framework for quantifying and bounding the influence of any individual data point on a trained model\. A mechanismMMsatisfies\(ϵ,δ\)\(\\epsilon,\\delta\)\-differential privacy if for any two datasets𝒟\\mathcal\{D\}and𝒟′\\mathcal\{D\}^\{\\prime\}differing in exactly one data point, and for any measurable output set𝒮\\mathcal\{S\}:

Pr⁡\(M​\(𝒟\)∈𝒮\)≤eϵ​Pr⁡\(M​\(𝒟′\)∈𝒮\)\+δ\\Pr\(M\(\\mathcal\{D\}\)\\in\\mathcal\{S\}\)\\leq e^\{\\epsilon\}\\Pr\(M\(\\mathcal\{D\}^\{\\prime\}\)\\in\\mathcal\{S\}\)\+\\delta\(19\)
The connection between differential privacy and machine unlearning is direct: if a model is trained with\(ϵ,δ\)\(\\epsilon,\\delta\)\-DP guarantees, then the deletion capacity of that model can be bounded formallyCao and Yang \([2015](https://arxiv.org/html/2607.16227#bib.bib33)\)\. This provides the theoretical grounding that approximate unlearning methods based on DP offer, which gradient\-only methods lack\.

DP\-SGD \(Differentially Private Stochastic Gradient Descent\)Abadiet al\.\([2016](https://arxiv.org/html/2607.16227#bib.bib108)\)is the canonical DP training procedure, included here as the foundational mechanism underlying recent DP\-based unlearning work\. It clips per\-example gradients to a maximum normCCand adds calibrated Gaussian noise to the aggregated gradient before each parameter update\. The clipping ensures that no single training example can shift the gradient by more thanCC, and the noise masks the contribution of any individual example\. Models trained with DP\-SGD have bounded sensitivity to any single training example, which directly supports approximate unlearning: removing a data point from a DP\-trained model requires only a bounded correction rather than full retraining\.

DP2UnlearningAl Mahmudet al\.\([2025](https://arxiv.org/html/2607.16227#bib.bib118)\)combines DP with post\-hoc unlearning mechanisms to provide formal forgetting guarantees for LLMs that existing gradient\-based methods cannot offer\. The approach addresses a fundamental limitation of approximate unlearning methods: they rely on empirical evidence rather than formal guarantees, which fails to satisfy the Right to Be Forgotten as stated in GDPR and similar legal frameworks\. By leveraging DP’s formal privacy bounds, DP2Unlearning can certify that the influence of a removed data point falls below a quantifiable threshold\.

The primary limitation of DP\-based approaches is the privacy\-utility trade\-off: the noise added to achieve DP guarantees degrades model utility, particularly for LLMs trained on large and diverse corpora\. Furthermore, DP training is a preventive control: it must be applied during training and is not retroactive\. A deployed LLM that was not trained with DP cannot retroactively acquire DP guarantees, which limits the applicability of DP\-based approaches to models in development rather than already\-deployed systems\.

Despite these limitations, DP\-based methods represent the only family of approaches that provide formal, mathematically verifiable forgetting guarantees\. As regulatory pressure for demonstrable compliance with erasure rights increases, DP\-based unlearning may become increasingly important for deployment contexts where formal certification is required\.

### 5\.3Inference\-Time and Input Modification Methods

A distinct class of unlearning approaches operates at inference time rather than modifying model weights\. These methods simulate unlearning by altering the model’s input or output pipeline, making them applicable to black\-box models where weight access is unavailable\.

In\-context unlearning \(ICUL\)Pawelczyket al\.\([2024](https://arxiv.org/html/2607.16227#bib.bib115)\)modifies the model’s input context to signal that certain knowledge should not be expressed\. Label flipping disrupts the original association by providing counter\-examples in the context window, effectively overriding the model’s parametric knowledge with in\-context instructions\. ICUL requires no weight modification and is applicable to black\-box APIs, but provides no guarantee of parametric removal and requires storing the forget\-set content for prompt construction, which may contradict the data privacy goals motivating unlearning\.

Soft prompt optimizationlearns continuous input representations that, when prepended to queries about the forget set, induce the model to produce alternative or refusal responses\. Methods such as SPULBhailaet al\.\([2025](https://arxiv.org/html/2607.16227#bib.bib116)\)optimize soft prompt tokens to selectively suppress knowledge without modifying model weights\. These approaches are efficient and reversible, but their effectiveness is limited to the distribution of queries for which the soft prompt was optimized\.

Logit\-offset methodsoperate on model outputs at decoding time\. Methods such asδ\\delta\-UnlearningJiet al\.\([2024](https://arxiv.org/html/2607.16227#bib.bib117)\)compute a logit offset between a base model and a forgetting model, subtracting the forgetting signal from the base model’s output distribution at inference\. ULD \(Unlearning by Logit Difference\)Jiet al\.\([2024](https://arxiv.org/html/2607.16227#bib.bib117)\)subtracts logits from an auxiliary model trained with reversed objectives\. These methods require access to model logits and offer no parametric removal guarantee, but can be applied to already\-deployed models without retraining\.

Inference\-time methods occupy the behavioral suppression tier of the three\-level framework\. Their principal advantage is applicability to black\-box or nearly\-black\-box settings where weight modification is impossible\. For regulatory compliance purposes, their inability to demonstrate parametric removal is a significant limitation\.

Table[4](https://arxiv.org/html/2607.16227#S5.T4)summarizes all non\-gradient\-based methods covered in this section across the dimensions of the framework\.

Table 4:Comparison of non\-gradient\-based unlearning methods\. Forgetting level follows the framework in Section[3\.4](https://arxiv.org/html/2607.16227#S3.SS4)\. Weight access indicates the level of model access required at unlearning time\. Formal guarantee indicates whether the method provides a mathematically certifiable forgetting bound\. Foundational methods predating 2023 are included where they directly underpin recent unlearning research\. BS = Behavioral Suppression, RA = Representation Attenuation \(approaching\)\.
### 5\.4Limitations of Non\-Gradient Approaches

Non\-gradient\-based unlearning approaches share several fundamental limitations\. Model editing methods achieve behavioral modification for targeted facts but leave the knowledge distributed across other parameter regions intact, making them susceptible to the same extraction attacks that afflict gradient\-based methods\. DP\-based approaches provide formal guarantees but require DP training from the outset and impose significant utility costs that are challenging to manage at LLM scale\. Inference\-time methods provide no parametric removal and are effective only within the distribution of queries for which they were designed\.

Across all three families, a consistent pattern emerges that mirrors the gradient\-based case: the depth of removal achieved is behavioral suppression rather than parametric elimination\. No non\-gradient\-based method reviewed here reliably achieves representation\-level attenuation, let alone true forgetting\. The field\-wide challenge of moving from suppression to genuine forgetting applies equally to all current method families\.

## 6Evaluation of Unlearning

Evaluating unlearning in large language models remains a fundamentally challenging problem due to model scale, generative behavior, and the absence of a tractable ground truthWuet al\.\([2025](https://arxiv.org/html/2607.16227#bib.bib42)\)\. Unlike classical machine unlearning, where the gold standard is comparison with a model retrained from scratch without the forget setTang and Khanna \([2026](https://arxiv.org/html/2607.16227#bib.bib72)\); Lagunaet al\.\([2026](https://arxiv.org/html/2607.16227#bib.bib73)\), such retraining is computationally infeasible for modern LLMs\. Existing evaluation strategies therefore rely on proxy metrics and benchmark datasets that assess different aspects of unlearning rather than verifying true removal of internal representations\.

### 6\.1Evaluation Protocols and Baselines

Most LLM unlearning studies adopt a standardized evaluation pipeline in which a pretrained model is fine\-tuned on controlled datasets followed by targeted unlearningLizzo and Heck \([2026](https://arxiv.org/html/2607.16227#bib.bib74)\)\. The resulting model is compared against baseline methods such as gradient ascent, gradient difference, or KL\-regularized variants\. The lack of a unified evaluation standard leads to inconsistencies across studies, making direct comparison difficult\.

### 6\.2Benchmark Datasets

#### 6\.2\.1Unlearning\-Specific Benchmarks

Several benchmarks have been proposed to explicitly evaluate unlearning\. TOFU \(Task of Fictitious Unlearning\)Mainiet al\.\([2024](https://arxiv.org/html/2607.16227#bib.bib75)\)introduces synthetic entities, enabling controlled evaluation where the ground truth is known, with separate forget, retain, and generalization sets\. RWKU \(Real\-World Knowledge Unlearning\)Jin \([2024](https://arxiv.org/html/2607.16227#bib.bib76)\)focuses on real\-world knowledge and incorporates additional subsets for membership inference attacks and utility evaluation\. WMDP \(Weapons of Mass Destruction Proxy\)Li \([2024](https://arxiv.org/html/2607.16227#bib.bib77)\)targets hazardous knowledge domains such as biosecurity and cybersecurity, evaluating whether models can be prevented from generating harmful information\. The Harry Potter BenchmarkChen \([2023](https://arxiv.org/html/2607.16227#bib.bib78)\)evaluates memorization and removal of domain\-specific knowledge through prompt\-based testing\. These benchmarks enable controlled evaluation of forgetting effectiveness, though they primarily assess output\-level behavior\.

Table[5](https://arxiv.org/html/2607.16227#S6.T5)provides a concise comparison of the major unlearning benchmarks\.

Table 5:Comparison of major LLM unlearning benchmarks\. Ground truth indicates whether the forget set is fully separable from the retain set\. Adversarial evaluation indicates whether the benchmark includes robustness testing against recovery attacks\.
#### 6\.2\.2General Capability Benchmarks

To evaluate utility preservation, prior work commonly relies on MMLUHendrycks \([2020](https://arxiv.org/html/2607.16227#bib.bib79)\), TruthfulQALinet al\.\([2022](https://arxiv.org/html/2607.16227#bib.bib80)\), HellaSwagZellers \([2019](https://arxiv.org/html/2607.16227#bib.bib81)\), and ARCYadavet al\.\([2019](https://arxiv.org/html/2607.16227#bib.bib82)\)\. These benchmarks collectively assess a model’s performance across factual knowledge recall, commonsense reasoning, and language understanding\. A significant drop in performance after unlearning indicates interference with retained knowledge, while stable performance suggests effective preservation of utility\.

### 6\.3Evaluation Metrics

Forgetting effectiveness is typically evaluated using ROUGE\-L overlap between generated and target content, the log\-likelihood or probability of generating forget\-set data, and measures like truth ratio and perplexity\. These metrics assess whether the model avoids reproducing information associated with the forget set, but primarily capture output\-level behavior and do not guarantee that the underlying knowledge has been fully removed\.

Utility preservation is assessed by measuring model performance on the retain set and on standard benchmarks\. Efficiency is evaluated in terms of computational cost relative to retraining\. The NeurIPS 2023 Unlearning Challenge, for example, considers methods efficient if their computational cost is below 20% of the retraining timeTriantafillou and Kairouz \([2023](https://arxiv.org/html/2607.16227#bib.bib83)\)\.

A recently identified limitation of classical metrics is the squeezing effectLiet al\.\([2026](https://arxiv.org/html/2607.16227#bib.bib91)\): methods that score well on ROUGE and perplexity may still leak targeted knowledge in paraphrased form\. This motivated the development of LLM\-as\-a\-judge evaluation as a semantic complement to surface\-level metrics\.

### 6\.4Robustness and Adversarial Evaluation

A critical aspect of unlearning evaluation is robustness against recovery attacks\. Membership Inference Attacks \(MIA\) are commonly used to determine whether traces of the forget set remain in the modelLiu \([2024](https://arxiv.org/html/2607.16227#bib.bib84)\), where a high attack success rate indicates incomplete unlearning\. Relearning attacks demonstrate that supposedly forgotten knowledge can be rapidly recovered through fine\-tuning on small auxiliary datasetsBlanco\-Justicia \([2025](https://arxiv.org/html/2607.16227#bib.bib85)\)\. Adversarial prompting techniques including Dynamic Unlearning AttacksYuanet al\.\([2024](https://arxiv.org/html/2607.16227#bib.bib86)\)and jailbreak\-style prompts attempt to elicit forgotten information through carefully optimized inputs\. Embedding\-space attacks using soft prompting operate directly on latent representations to recover hidden knowledgeSchwinnet al\.\([2024](https://arxiv.org/html/2607.16227#bib.bib87)\)\.

The most comprehensive adversarial evaluation of state\-of\-the\-art unlearning methods is provided by Lucki et al\.Luckiet al\.\([2025](https://arxiv.org/html/2607.16227#bib.bib88)\), who systematically apply white\-box and gray\-box attacks to models unlearned on the WMDP benchmark\. Fine\-tuning an RMU\-unlearned modelLi \([2024](https://arxiv.org/html/2607.16227#bib.bib77)\)on as few as 10 unrelated examples restores most hazardous capabilities\. Orthogonalization of residual stream directions recovers WMDP\-Biology accuracy to 64\.7% from the post\-unlearning level, equaling the pre\-unlearning baseline of 64\.4%, without modifying any model weights\. The paper concludes that existing unlearning methods are not meaningfully different from safety fine\-tuning in terms of robustness: both primarily obfuscate knowledge rather than removing it\.

The robustness problem extends beyond knowledge recovery to the retain set itself\. Dang et al\.Danget al\.\([2026](https://arxiv.org/html/2607.16227#bib.bib90)\)identify retain\-robustness as a failure mode distinct from forget\-set knowledge recovery: unlearning methods inadvertently train the model to treat forget\-set tokens as backdoor triggers, so that when those tokens appear in otherwise benign retain\-set queries, the model’s responses are degraded\. The proposed mitigation, Random Noise Augmentation applied during the retain phase, significantly improves retain\-robustness while preserving forget\-set performance\.

Fan et al\.Fanet al\.\([2025a](https://arxiv.org/html/2607.16227#bib.bib89)\)establish a connection between LLM unlearning robustness and sharpness\-aware minimization\. Standard NPO places the unlearned model at a sharp local minimum in the loss landscape, making small weight perturbations of the kind applied in a relearning attack sufficient to reverse the unlearning\. The robust unlearning objective is formulated as the min\-max problem:

minθ⁡max‖δ‖p≤ρ⁡ℓf​\(θ\+δ∣𝒟f\)\+λ​ℓr​\(θ∣𝒟r\)\\min\_\{\\theta\}\\,\\max\_\{\\\|\\delta\\\|\_\{p\}\\leq\\rho\}\\;\\ell\_\{f\}\(\\theta\+\\delta\\mid\\mathcal\{D\}\_\{f\}\)\+\\lambda\\,\\ell\_\{r\}\(\\theta\\mid\\mathcal\{D\}\_\{r\}\)\(20\)
where the inner maximization simulates the worst\-case weight perturbation a relearning attacker would apply\. This aligns with sharpness\-aware minimization, which promotes a flat loss landscape around the unlearned solution\. Solving the inner maximization in closed form via linear approximation yields the SAM forget loss:

ℓfSAM​\(θ\)=ℓf​\(θ\+ρ​∇θℓf​\(θ\)‖∇θℓf​\(θ\)‖2\)\\ell\_\{f\}^\{\\text\{SAM\}\}\(\\theta\)=\\ell\_\{f\}\\\!\\left\(\\theta\+\\rho\\,\\frac\{\\nabla\_\{\\theta\}\\ell\_\{f\}\(\\theta\)\}\{\\\|\\nabla\_\{\\theta\}\\ell\_\{f\}\(\\theta\)\\\|\_\{2\}\}\\right\)\(21\)
NPO\+SAM is reported to match the robustness of tamper\-resistant safeguards \(TAR\)Tamirisaet al\.\([2025](https://arxiv.org/html/2607.16227#bib.bib119)\), which uses computationally intensive meta\-learning, while running over 600 times faster on equivalent tasksFanet al\.\([2025a](https://arxiv.org/html/2607.16227#bib.bib89)\)\. Smoothness\-based robustness also extends to jailbreaking attacks, not only relearning attacks\.

Rezkellah and DakhmoucheRezkellah and Dakhmouche \([2025](https://arxiv.org/html/2607.16227#bib.bib96)\)propose a unified framework for both unlearning and jailbreak robustness through constrained optimization directly on model weights\. The Point\-wise Constrained Regions \(PCR\) method identifies forbidden concept embeddings and applies the minimal weight perturbation that forces MLP layer outputs to remain geometrically distant from those embeddings, using Karush\-Kuhn\-Tucker \(KKT\) conditions for the constrained optimization in each layer\. Across Llama\-3\.1\-8B, Gemma\-2B\-IT, and Mistral\-7B, PCR achieves refusal rates of 100%, 97\.4%, and 26\.7% respectively against projected gradient descent jailbreak attacks, compared to 87\.5%, 10%, and 37\.5% for SmoothLLMRobeyet al\.\([2025](https://arxiv.org/html/2607.16227#bib.bib120)\)on the same models\.

### 6\.5Discussion: Limitations of Current Evaluation

Current evaluation methods suffer from a fundamental limitation: they predominantly assess behavioral suppression rather than true forgetting\. Most metrics rely on observing model outputs under predefined prompts, which only capture whether the model avoids generating specific responses\. As demonstrated by adversarial and relearning attacks, suppressed knowledge can often be recovered, indicating that it remains encoded in model parameters\. Existing evaluation protocols are well\-suited for detecting behavioral suppression, but insufficient for verifying representation\-level attenuation or true forgetting\. Developing evaluation methods that directly probe internal representations or provide formal guarantees remains an open challenge\.

## 7Discussion and Open Challenges

The central finding of this survey, that current gradient\-based unlearning methods achieve behavioral suppression rather than true forgetting, has direct implications for their deployment as security controls\. A behavioral suppression defense is one whose guarantees hold only for the class of adversary assumed during evaluation\. In the majority of published evaluations, that adversary is black\-box, non\-adaptive, and operating under standard prompting conditions\. The adversarial evaluations reviewed in Section[6\.4](https://arxiv.org/html/2607.16227#S6.SS4)demonstrate that these guarantees dissolve under gray\-box and white\-box access\. The failure of current unlearning to function as a robust security control is not a weakness of any specific method but a structural property of the optimization approach: gradient\-based updates that modify output behavior without altering internal representations leave intact the parameter\-level pathways through which a capable adversary can recover the target knowledge\.

Current LLM unlearning methods largely conflate two distinct objectives: suppressing the observable expression of knowledge and removing the underlying representations that encode it\. Most existing gradient\-based approaches primarily achieve the former, while offering limited evidence of the latter\. In deployed systems, successful suppression under standard prompts is often interpreted as successful data removal\. Prior work shows that suppressed information can frequently be recovered through adversarial prompting, relearning attacks, or latent\-space probingSchwinnet al\.\([2024](https://arxiv.org/html/2607.16227#bib.bib87)\); Blanco\-Justicia \([2025](https://arxiv.org/html/2607.16227#bib.bib85)\); Yuanet al\.\([2024](https://arxiv.org/html/2607.16227#bib.bib86)\)\. This gap raises concerns about whether current unlearning practices provide meaningful guarantees in privacy\-sensitive or safety\-critical applications\.

Current evaluation protocols primarily measure output behavior under fixed prompting distributions, making them well\-aligned with detecting behavioral suppression but insufficient for assessing representational removal\. Adversarial evaluation methods including membership inference, relearning attacks, and prompt\-based extraction partially address this gap by probing recovery pathways, but they test specific access strategies rather than providing guarantees over all possible reconstruction mechanisms\. This creates a structural evaluation gap: current benchmarks can confirm when unlearning fails, but cannot certify when it succeeds in the strong sense of true forgetting\.

### 7\.1Limitations of Gradient\-Based Methods

The limitations of gradient\-based unlearning arise from a structural mismatch between the optimization mechanism and how knowledge is encoded in large language models\. Training in LLMs is a highly non\-convex, path\-dependent process in which information becomes distributed and entangled across parametersGevaet al\.\([2021a](https://arxiv.org/html/2607.16227#bib.bib54)\)\. Gradient ascent on a forget set therefore does not invert training dynamics; it induces a new trajectory that modifies output behavior without reliably isolating or eliminating the associated internal representations\.

This limitation is further amplified by parameter sharing: the same weights that encode forget\-set information also support unrelated capabilities\. Stronger updates risk degrading general utility, producing the well\-known trade\-off between forgetting effectiveness and model performanceGuoet al\.\([2019](https://arxiv.org/html/2607.16227#bib.bib35)\); Anjarlekar and Pombra \([2025](https://arxiv.org/html/2607.16227#bib.bib39)\)\. Influence\-based methods attempt to approximate training contributions using second\-order information, but rely on local linearity assumptions that do not hold in large\-scale non\-convex settingsBasuet al\.\([2021](https://arxiv.org/html/2607.16227#bib.bib64)\)\. Parameter\-efficient and saliency\-based approaches improve update precision but still assume partial localizability of knowledge, reducing sensitivity to targeted data without guaranteeing removal of the underlying representations\.

### 7\.2What the Framework Reveals

Mapping current approaches onto the behavioral suppression, representation attenuation, true forgetting hierarchy reveals a consistent pattern: most gradient\-based and constrained optimization methods operate at the level of behavioral suppression, while influence\-based techniques occasionally achieve partial representational attenuation\. No existing method reliably achieves true forgetting in the sense of equivalence to retraining without the forget set\. This mapping clarifies an important conceptual point: similar performance on existing benchmarks does not imply equivalent levels of forgetting\. A method that suppresses outputs under standard prompts may still retain recoverable representations internally, meaning that evaluation results can overestimate the degree of actual information removal\. Improvements in benchmark performance should therefore be understood as improvements in suppression capability rather than guarantees of knowledge deletion\.

### 7\.3Open Challenges

Despite progress in LLM unlearning, several fundamental challenges remain unresolved\.

Challenge 1: Achieving and verifying true forgetting\.Current methods largely operate by suppressing behavioral outputs rather than removing internal representations, and there is still no reliable way to verify whether knowledge has been genuinely eliminated\. Output\-based evaluation is insufficient, while direct inspection of representations remains an open problem\.

Challenge 2: Robustness to adversarial recovery\.Existing unlearning methods can often be bypassed through adversarial prompting, relearning attacks, or latent\-space extraction techniquesYuanet al\.\([2024](https://arxiv.org/html/2607.16227#bib.bib86)\); Blanco\-Justicia \([2025](https://arxiv.org/html/2607.16227#bib.bib85)\); Schwinnet al\.\([2024](https://arxiv.org/html/2607.16227#bib.bib87)\)\. No current method provides robustness guarantees against adaptive white\-box adversaries\.

Challenge 3: Scalability and compositionality\.Although parameter\-efficient and gradient\-based methods reduce cost compared to retraining, they remain expensive at scale and are not well\-suited for continual or repeated unlearning\. Real\-world knowledge is highly interconnected, making fine\-grained or compositional removal of specific facts difficult without unintended side effects\.

Challenge 4: Adversarially adaptive evaluation\.Current unlearning benchmarks including TOFUMainiet al\.\([2024](https://arxiv.org/html/2607.16227#bib.bib75)\), RWKUJin \([2024](https://arxiv.org/html/2607.16227#bib.bib76)\), WMDPLi \([2024](https://arxiv.org/html/2607.16227#bib.bib77)\), and MUSEShiet al\.\([2024](https://arxiv.org/html/2607.16227#bib.bib105)\)test model behavior under predefined and non\-adaptive prompting conditions\. An adversary with knowledge of the unlearning procedure applied can craft adaptive attacks that systematically search for conditions under which unlearned knowledge resurfaces\. No standardized benchmark currently tests unlearned models against adaptive white\-box adversaries, meaning that existing evaluation results cannot certify robustness against a determined adversary with model access\. Developing such evaluation frameworks is a prerequisite for deploying unlearning as a credible security control in high\-stakes settings\.

Challenge 5: Theoretical and evaluative foundations\.Current methods are largely heuristic, with limited understanding of when or why unlearning approximations work\. There is no standardized evaluation framework that reliably distinguishes behavioral suppression from genuine forgetting, making comparisons across methods inconsistent\. Bridging this gap is essential for both scientific progress and alignment with regulatory requirements such as the right to erasure under GDPR Article 17 and similar data deletion laws\.

## 8Conclusion

This survey has examined LLM unlearning through the lens of cybersecurity, robustness, and verifiable forgetting\. The central conclusions are as follows\.

- •Behavioral suppression is not security\.The vast majority of current gradient\-based unlearning methods achieve behavioral suppression under standard prompting conditions\. They do not achieve representation\-level attenuation or true forgetting, and their guarantees dissolve under gray\-box or white\-box adversarial access\.
- •The evaluation gap is a security audit gap\.Current benchmarks assess behavioral suppression but cannot certify parametric removal\. This is not merely a methodological limitation; it is a fundamental barrier to deploying unlearning as a verifiable security control\.
- •The three\-level framework provides a diagnostic tool\.Classifying methods under behavioral suppression, representation\-level attenuation, and true forgetting gives practitioners a clear vocabulary for specifying what level of security guarantee a deployment context requires and whether any available method can provide it\.
- •Non\-gradient approaches complement but do not solve the core problem\.Model editing methods achieve targeted behavioral modification, DP\-based approaches provide formal guarantees for models trained with DP from the outset, and inference\-time methods extend unlearning to black\-box settings\. None of these approaches reliably achieves true parametric forgetting\.
- •The field requires adversarially adaptive evaluation\.No standardized benchmark currently tests unlearned models against adaptive white\-box adversaries\. Developing such evaluation frameworks is the most critical infrastructure need in the field\.

LLM unlearning is a rapidly growing field with direct implications for regulatory compliance, safety\-critical deployment, and the long\-term trustworthiness of AI systems\. The gap between the field’s current capabilities and the security guarantees that high\-stakes deployments require is large, but it is precisely defined\. Closing it requires methods that operate at the level of internal representations rather than output distributions, evaluation protocols that assume capable adversaries rather than passive users, and formal frameworks that can certify the absence of parametric knowledge rather than merely its behavioral suppression\.

## 9Author Contributions

R\. S\. Shankar performed the literature review, data collection and curation, methodology and software design, investigation, and manuscript drafting\. A\. Bhardwaj performed the literature review, data collection and curation, methodology and software design, investigation, and manuscript drafting\. A\. Doshi performed content validation, manuscript review, manuscript editing, and manuscript presentation\. A\. Nagarajan performed content validation, manuscript review, and manuscript editing\. T\. P\. Asia performed content validation, manuscript review, and manuscript editing\. S\. Sengupta performed project administration, supervision, methodological guidance, content validation, manuscript review, and manuscript editing\.

## 10Funding

This research did not receive external funding\.

## 11Data Availability Statement

All datasets analyzed in this survey are publicly available\. No new data were generated for this study\.

## 12Declaration of Competing Interest

The authors declare no competing interests\.

## References

- M\. Abadi, A\. Chu, I\. Goodfellow, H\. B\. McMahan, I\. Mironov, K\. Talwar, and L\. Zhang \(2016\)Deep learning with differential privacy\.InProceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security \(CCS\),pp\. 308–318\.External Links:[Document](https://dx.doi.org/10.1145/2976749.2978318)Cited by:[§5\.2](https://arxiv.org/html/2607.16227#S5.SS2.p3.2),[Table 4](https://arxiv.org/html/2607.16227#S5.T4.1.2.1.1.1.1)\.
- A\. Acharya, B\. Singh, and N\. Onoe \(2023\)LLM based generation of item\-description for recommendation system\.InProceedings of the 17th ACM Conference on Recommender Systems \(RecSys ’23\),pp\. 1204–1207\.External Links:[Document](https://dx.doi.org/10.1145/3604915.3610647)Cited by:[§3\.1](https://arxiv.org/html/2607.16227#S3.SS1.p1.1)\.
- T\. Al Mahmud, N\. Jebreel, J\. Domingo\-Ferrer, and D\. Sánchez \(2025\)DP2Unlearning: an efficient and guaranteed unlearning framework for LLMs\.Neural Networks\.Note:arXiv:2504\.13774External Links:[Document](https://dx.doi.org/10.1016/j.neunet.2025.107879),[Link](https://arxiv.org/abs/2504.13774)Cited by:[§5\.2](https://arxiv.org/html/2607.16227#S5.SS2.p4.1),[Table 4](https://arxiv.org/html/2607.16227#S5.T4.1.9.8.1.1.1)\.
- A\. Anjarlekar and S\. Pombra \(2025\)LLM unlearning using gradient ratio\-based influence estimation and noise injection\.arXiv preprint arXiv:2508\.06467\.Cited by:[§4\.1](https://arxiv.org/html/2607.16227#S4.SS1.p4.2),[§4\.2\.2](https://arxiv.org/html/2607.16227#S4.SS2.SSS2.p6.1),[§7\.1](https://arxiv.org/html/2607.16227#S7.SS1.p2.1)\.
- S\. Basu, P\. Pope, and S\. Feizi \(2021\)Influence functions in deep learning are fragile\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=xHKVVHGDOEk)Cited by:[§4\.3](https://arxiv.org/html/2607.16227#S4.SS3.p5.1),[§7\.1](https://arxiv.org/html/2607.16227#S7.SS1.p2.1)\.
- K\. Bhaila, M\. Van, and X\. Wu \(2025\)Soft prompting for unlearning in large language models\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),Note:arXiv:2406\.12038External Links:[Link](https://aclanthology.org/2025.naacl-long.204/)Cited by:[§5\.3](https://arxiv.org/html/2607.16227#S5.SS3.p3.1),[Table 4](https://arxiv.org/html/2607.16227#S5.T4.1.10.9.1.1.1)\.
- A\. e\. al\. Blanco\-Justicia \(2025\)Digital forgetting in large language models: a survey of unlearning methods\.Artificial Intelligence Review58\(3\)\.External Links:[Document](https://dx.doi.org/10.1007/s10462-024-11078-6)Cited by:[§1\.1](https://arxiv.org/html/2607.16227#S1.SS1.p1.1),[§6\.4](https://arxiv.org/html/2607.16227#S6.SS4.p1.1),[§7\.3](https://arxiv.org/html/2607.16227#S7.SS3.p3.1),[§7](https://arxiv.org/html/2607.16227#S7.p2.1)\.
- V\. e\. al\. Boer \(2026\)Quantization\-robust llm unlearning via low\-rank adaptation\.External Links:2602\.13151Cited by:[§4\.4](https://arxiv.org/html/2607.16227#S4.SS4.p2.1)\.
- J\. Brophy, Z\. Hammoudeh, and D\. Lowd \(2023\)Adapting and evaluating influence\-estimation methods for gradient\-boosted decision trees\.Journal of Machine Learning Research24,pp\. 1–48\.External Links:[Link](https://www.jmlr.org/papers/volume24/22-0449/22-0449.pdf)Cited by:[§4\.3](https://arxiv.org/html/2607.16227#S4.SS3.p1.1)\.
- T\. Brownet al\.\(2020\)Language models are few\-shot learners\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf)Cited by:[§3\.1](https://arxiv.org/html/2607.16227#S3.SS1.p1.1)\.
- Z\. Bu, X\. Jin, B\. Vinzamuri, A\. Ramakrishna, K\. Chang, V\. Cevher, and M\. Hong \(2025\)Unlearning as multi\-task optimization: a normalized gradient difference approach with an adaptive learning rate\.arXiv preprint arXiv:2410\.22086\.External Links:[Link](https://arxiv.org/abs/2410.22086)Cited by:[§4\.2\.1](https://arxiv.org/html/2607.16227#S4.SS2.SSS1.p2.2),[§4\.2\.1](https://arxiv.org/html/2607.16227#S4.SS2.SSS1.p4.1),[§4\.2\.1](https://arxiv.org/html/2607.16227#S4.SS2.SSS1.p5.1),[Table 3](https://arxiv.org/html/2607.16227#S4.T3.1.12.11.1.1.1)\.
- Y\. Cao and J\. Yang \(2015\)Towards making systems forget with machine unlearning\.InProceedings of the IEEE Symposium on Security and Privacy Workshops,pp\. 463–480\.External Links:[Document](https://dx.doi.org/10.1109/SPW.2015.15)Cited by:[§2\.1](https://arxiv.org/html/2607.16227#S2.SS1.p1.1),[§3\.3](https://arxiv.org/html/2607.16227#S3.SS3.p1.4),[§5\.2](https://arxiv.org/html/2607.16227#S5.SS2.p2.1)\.
- N\. Carlini, D\. Ippolito, M\. Jagielski, K\. Lee, F\. Tramer, and C\. Zhang \(2022\)Quantifying memorization across neural language models\.InICLR,External Links:[Link](https://openreview.net/forum?id=TatRHT_1cK)Cited by:[§1](https://arxiv.org/html/2607.16227#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.16227#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2607.16227#S2.SS1.p4.1),[§3\.2](https://arxiv.org/html/2607.16227#S3.SS2.p1.1)\.
- I\. D\. Cevallos, M\. E\. Benalcázar, L\. Valdivieso, J\. A\. Zea, and L\. I\. Barona\-López \(2025\)A systematic literature review of machine unlearning techniques in neural networks\.Computers14\(4\),pp\. 150\.External Links:[Document](https://dx.doi.org/10.3390/computers14040150)Cited by:[§1\.1](https://arxiv.org/html/2607.16227#S1.SS1.p1.1),[§4\.1](https://arxiv.org/html/2607.16227#S4.SS1.p5.1),[§4\.2\.1](https://arxiv.org/html/2607.16227#S4.SS2.SSS1.p8.1),[§4\.2\.2](https://arxiv.org/html/2607.16227#S4.SS2.SSS2.p4.1)\.
- N\. e\. al\. Chen \(2023\)Large language models meet harry potter: a dataset for aligning dialogue agents with characters\.Note:[https://openreview\.net/forum?id=TKo2JXw7vL](https://openreview.net/forum?id=TKo2JXw7vL)Cited by:[§6\.2\.1](https://arxiv.org/html/2607.16227#S6.SS2.SSS1.p1.1),[Table 5](https://arxiv.org/html/2607.16227#S6.T5.1.2.1.1.1.1)\.
- J\. Cheng, P\. Liu, Q\. Li, and C\. Zhang \(2026\)Machine unlearning under retain–forget entanglement\.Note:[https://openreview\.net/forum?id=4WMBSHHJEr](https://openreview.net/forum?id=4WMBSHHJEr)Cited by:[§4\.4](https://arxiv.org/html/2607.16227#S4.SS4.p1.1)\.
- Q\. Cheng, Y\. Wan, L\. Wu, C\. Hou, and L\. Zhang \(2025\)Continuous subspace optimization for continual learning\.External Links:2505\.11816Cited by:[§4\.4](https://arxiv.org/html/2607.16227#S4.SS4.p2.1)\.
- A\. Chowdheryet al\.\(2023\)PaLM: scaling language modeling with pathways\.Journal of Machine Learning Research24\(240\),pp\. 1–113\.External Links:[Link](https://jmlr.org/papers/v24/22-1144.html)Cited by:[§3\.1](https://arxiv.org/html/2607.16227#S3.SS1.p1.1)\.
- Civil Resolution Tribunal of British Columbia \(2024\)Moffatt v\. air canada, 2024 BCCRT 149\.Note:[https://decisions\.civilresolutionbc\.ca](https://decisions.civilresolutionbc.ca/)Tribunal Member Christopher RiversCited by:[§2\.3](https://arxiv.org/html/2607.16227#S2.SS3.p1.1)\.
- H\. Dang, L\. Q\. Trung, H\. Nguyen, Y\. Nakashima, N\. Garcia, and T\. H\. Nguyen \(2026\)Improving LLM unlearning robustness via random perturbations\.Transactions on Machine Learning Research\.Note:arXiv:2501\.19202External Links:[Link](https://arxiv.org/abs/2501.19202)Cited by:[§6\.4](https://arxiv.org/html/2607.16227#S6.SS4.p3.1)\.
- J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova \(2019\)BERT: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of NAACL\-HLT,pp\. 4171–4186\.Cited by:[§3\.1](https://arxiv.org/html/2607.16227#S3.SS1.p1.1)\.
- C\. Fan, J\. Jia, Y\. Zhang, A\. Ramakrishna, M\. Hong, and S\. Liu \(2025a\)Towards LLM unlearning resilient to relearning attacks: a sharpness\-aware minimization perspective and beyond\.InProceedings of the 42nd International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.267\.External Links:[Link](https://arxiv.org/abs/2502.05374)Cited by:[§6\.4](https://arxiv.org/html/2607.16227#S6.SS4.p4.1),[§6\.4](https://arxiv.org/html/2607.16227#S6.SS4.p6.1)\.
- C\. Fan, J\. Liu, Y\. Zhang, E\. Wong, D\. Wei, and S\. Liu \(2023\)SalUn: empowering machine unlearning via gradient\-based weight saliency in both image classification and generation\.arXiv preprint arXiv:2310\.12508\.External Links:[Link](https://arxiv.org/abs/2310.12508)Cited by:[§4\.2\.1](https://arxiv.org/html/2607.16227#S4.SS2.SSS1.p6.1),[§4\.4](https://arxiv.org/html/2607.16227#S4.SS4.p2.1),[Table 3](https://arxiv.org/html/2607.16227#S4.T3.1.5.4.1.1.1)\.
- X\. Fan, J\. Wu, M\. Zhou, P\. Liang, and D\. Phung \(2025b\)IMU: influence\-guided machine unlearning\.arXiv preprint arXiv:2508\.01620\.External Links:[Link](https://arxiv.org/abs/2508.01620)Cited by:[§4\.3](https://arxiv.org/html/2607.16227#S4.SS3.p4.1)\.
- J\. Fang, H\. Jiang, K\. Wang, Y\. Ma, J\. Shi, X\. Wang, X\. He, and T\. Chua \(2025\)AlphaEdit: null\-space constrained knowledge editing for language models\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2410\.02355External Links:[Link](https://arxiv.org/abs/2410.02355)Cited by:[§5\.1](https://arxiv.org/html/2607.16227#S5.SS1.p7.1)\.
- Q\. Feng, J\. Tu, M\. Kang, H\. Zhao, C\. Zhang, and H\. Qian \(2025\)FG\-oriu: towards better forgetting via feature\-gradient orthogonality for incremental unlearning\.InProceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\),External Links:[Link](https://openaccess.thecvf.com/content/ICCV2025/papers/Feng_FG-OrIU_Towards_Better_Forgetting_via_Feature-Gradient_Orthogonality_for_Incremental_Unlearning_ICCV_2025_paper.pdf)Cited by:[§4\.2\.1](https://arxiv.org/html/2607.16227#S4.SS2.SSS1.p9.1),[Table 3](https://arxiv.org/html/2607.16227#S4.T3.1.8.7.1.1.1)\.
- X\. Feng, C\. Chen, Y\. Li, and Z\. Lin \(2024\)Fine\-grained pluggable gradient ascent for knowledge unlearning in language models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),External Links:[Link](https://aclanthology.org/2024.emnlp-main.566.pdf)Cited by:[§4\.2\.1](https://arxiv.org/html/2607.16227#S4.SS2.SSS1.p8.1),[§4\.2](https://arxiv.org/html/2607.16227#S4.SS2.p1.1)\.
- J\. Fields, K\. Chovanec, and P\. Madiraju \(2024\)A survey of text classification with transformers: how wide? how large? how long? how accurate? how expensive? how safe?\.IEEE Access\.External Links:[Document](https://dx.doi.org/10.1109/access.2024.3349952)Cited by:[§3\.1](https://arxiv.org/html/2607.16227#S3.SS1.p1.1)\.
- Garante per la Protezione dei Dati Personali \(2023\)Provvedimento del 30 marzo 2023 \[9870832\]\.Note:[https://www\.garanteprivacy\.it](https://www.garanteprivacy.it/)Italian Data Protection Authority order against OpenAI\. Follow\-up press release April 28, 2023Cited by:[§2\.3](https://arxiv.org/html/2607.16227#S2.SS3.p1.1)\.
- A\. Garg, H\. Saratchandran, R\. Garg, and S\. Lucey \(2026\)Stable forgetting: bounded parameter\-efficient unlearning in foundation models\.External Links:2509\.24166Cited by:[§4\.2\.1](https://arxiv.org/html/2607.16227#S4.SS2.SSS1.p21.1),[§4\.2\.1](https://arxiv.org/html/2607.16227#S4.SS2.SSS1.p22.8),[Table 3](https://arxiv.org/html/2607.16227#S4.T3.1.11.10.1.1.1)\.
- M\. Geva, R\. Schuster, J\. Berant, and O\. Levy \(2021a\)Transformer feed\-forward layers are key\-value memories\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),External Links:[Link](https://aclanthology.org/2021.emnlp-main.446.pdf)Cited by:[§4\.2\.2](https://arxiv.org/html/2607.16227#S4.SS2.SSS2.p4.1),[§4\.4](https://arxiv.org/html/2607.16227#S4.SS4.p3.1),[§5\.1](https://arxiv.org/html/2607.16227#S5.SS1.p1.1),[§7\.1](https://arxiv.org/html/2607.16227#S7.SS1.p1.1)\.
- M\. Geva, T\. Schuster, J\. Berant, and O\. Levy \(2021b\)Transformer feed\-forward layers are key\-value memories\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),External Links:[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.446)Cited by:[§3\.2](https://arxiv.org/html/2607.16227#S3.SS2.p1.1)\.
- A\. Golatkar, A\. Achille, and S\. Soatto \(2020\)Eternal sunshine of the spotless net: selective forgetting in deep networks\.IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\)\.Cited by:[§4\.1](https://arxiv.org/html/2607.16227#S4.SS1.p3.1),[§4\.2\.1](https://arxiv.org/html/2607.16227#S4.SS2.SSS1.p1.1),[§4\.2\.2](https://arxiv.org/html/2607.16227#S4.SS2.SSS2.p2.1),[§4\.2\.3](https://arxiv.org/html/2607.16227#S4.SS2.SSS3.p2.1),[Table 3](https://arxiv.org/html/2607.16227#S4.T3.1.3.2.1.1.1)\.
- C\. Guo, T\. Goldstein, A\. Hannun, and L\. van der Maaten \(2019\)Certified data removal from machine learning models\.arXiv preprint arXiv:1911\.03030\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.1911.03030)Cited by:[§3\.3](https://arxiv.org/html/2607.16227#S3.SS3.p4.1),[§3\.3](https://arxiv.org/html/2607.16227#S3.SS3.p5.3),[§4\.1](https://arxiv.org/html/2607.16227#S4.SS1.p4.2),[§4\.2\.1](https://arxiv.org/html/2607.16227#S4.SS2.SSS1.p1.1),[Table 3](https://arxiv.org/html/2607.16227#S4.T3.1.2.1.1.1.1),[§7\.1](https://arxiv.org/html/2607.16227#S7.SS1.p2.1)\.
- Z\. Hammoudeh and D\. Lowd \(2024\)Training data influence analysis and estimation: a survey\.Machine Learning\.External Links:[Document](https://dx.doi.org/10.1007/s10994-023-06495-7)Cited by:[§4\.3](https://arxiv.org/html/2607.16227#S4.SS3.p1.1),[§4\.3](https://arxiv.org/html/2607.16227#S4.SS3.p2.2)\.
- A\. Hatami and I\. E\. Monosov \(2026\)Beyond masking and avoidance: toward true unlearning\.InInternational Conference on Learning Representations \(ICLR\) \(under review\),External Links:[Link](https://openreview.net/forum?id=XXSPoq3uik)Cited by:[§3\.4](https://arxiv.org/html/2607.16227#S3.SS4.p1.1),[§4\.2\.1](https://arxiv.org/html/2607.16227#S4.SS2.SSS1.p8.1),[§4\.2\.3](https://arxiv.org/html/2607.16227#S4.SS2.SSS3.p1.1)\.
- D\. e\. al\. Hendrycks \(2020\)Measuring massive multitask language understanding\.External Links:2009\.03300Cited by:[§6\.2\.2](https://arxiv.org/html/2607.16227#S6.SS2.SSS2.p1.1)\.
- Z\. Huanget al\.\(2024\)Unified gradient\-based machine unlearning with remain geometry enhancement\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/2e622ac74f66df03b686a12e2e0e4424-Paper-Conference.pdf)Cited by:[§4\.1](https://arxiv.org/html/2607.16227#S4.SS1.p5.1)\.
- Z\. Huang, Q\. Wang, Z\. Hong, S\. Ye, B\. Han, and T\. Liu \(2026\)Is gradient ascent really necessary? memorize to forget for machine unlearning\.arXiv preprint arXiv:2602\.06441\.External Links:[Link](https://arxiv.org/abs/2602.06441)Cited by:[§4\.2\.2](https://arxiv.org/html/2607.16227#S4.SS2.SSS2.p3.1),[§4\.2](https://arxiv.org/html/2607.16227#S4.SS2.p1.1)\.
- J\. Ji, Y\. Liu, Y\. Zhang, G\. Liu, R\. R\. Kompella, S\. Liu, and S\. Chang \(2024\)Reversing the forget\-retain objectives: an efficient LLM unlearning framework from logit difference\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2406\.08607External Links:[Link](https://arxiv.org/abs/2406.08607)Cited by:[§5\.3](https://arxiv.org/html/2607.16227#S5.SS3.p4.1),[Table 4](https://arxiv.org/html/2607.16227#S5.T4.1.8.7.1.1.1)\.
- Z\. e\. al\. Jin \(2024\)RWKU: benchmarking real\-world knowledge unlearning for large language models\.External Links:2406\.10890Cited by:[§6\.2\.1](https://arxiv.org/html/2607.16227#S6.SS2.SSS1.p1.1),[Table 5](https://arxiv.org/html/2607.16227#S6.T5.1.4.3.1.1.1),[§7\.3](https://arxiv.org/html/2607.16227#S7.SS3.p5.1)\.
- A\. Karamolegkou, J\. Li, L\. Zhou, and A\. Søgaard \(2023\)Copyright violations and large language models\.InOpenReview,External Links:[Link](https://openreview.net/forum?id=YokfK5VOoz)Cited by:[§1](https://arxiv.org/html/2607.16227#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.16227#S2.SS1.p6.1)\.
- G\. Kim, Y\. Li, E\. Spiliopoulou, J\. Ma, and W\. Wang \(2026\)Detecting training data of large language models via expectation maximization\.InEACL,pp\. 1115–1129\.External Links:[Link](https://aclanthology.org/2026.eacl-long.49.pdf)Cited by:[§1](https://arxiv.org/html/2607.16227#S1.p2.1)\.
- T\. Kimet al\.\(2025\)GRAIL: gradient\-based adaptive unlearning for privacy and copyright in LLMs\.External Links:2504\.12681Cited by:[§4\.2\.2](https://arxiv.org/html/2607.16227#S4.SS2.SSS2.p7.1)\.
- E\. Kotei and R\. Thirunavukarasu \(2023\)A systematic review of transformer\-based pre\-trained language models through self\-supervised learning\.Information14\(3\),pp\. 187\.External Links:[Document](https://dx.doi.org/10.3390/info14030187)Cited by:[§1](https://arxiv.org/html/2607.16227#S1.p1.1)\.
- A\. Kudelya and A\. Shirnin \(2025\)Lacuna inc\. at semeval\-2025 task 4: lora\-enhanced influence\-based unlearning for llms\.InProceedings of the 19th International Workshop on Semantic Evaluation \(SemEval\-2025\),pp\. 1528–1533\.External Links:[Link](https://aclanthology.org/2025.semeval-1.201/)Cited by:[§4\.3](https://arxiv.org/html/2607.16227#S4.SS3.p4.1)\.
- S\. Laguna, J\. Vandenhirtz, A\. Ryser, I\. Cannistraci, and J\. E\. Vogt \(2026\)Rethinking machine unlearning: models designed to forget via key deletion\.External Links:2603\.15033Cited by:[§6](https://arxiv.org/html/2607.16227#S6.p1.1)\.
- U\. N\. Le\-Khac and Vinh \(2025a\)A survey on large language models unlearning: taxonomy, evaluations, and future directions\.Artificial Intelligence Review58\(12\)\.External Links:[Document](https://dx.doi.org/10.1007/s10462-025-11376-7)Cited by:[§1\.1](https://arxiv.org/html/2607.16227#S1.SS1.p1.1),[§1](https://arxiv.org/html/2607.16227#S1.p2.1),[§3\.4](https://arxiv.org/html/2607.16227#S3.SS4.p5.1),[§4\.1](https://arxiv.org/html/2607.16227#S4.SS1.p6.1)\.
- U\. Le\-Khac and T\. Vinh \(2025b\)A survey on large language models unlearning: taxonomy, evaluations, and future directions\.Artificial Intelligence Review58\(12\)\.External Links:[Document](https://dx.doi.org/10.1007/s10462-025-11376-7)Cited by:[§1\.1](https://arxiv.org/html/2607.16227#S1.SS1.p1.1),[§1\.2](https://arxiv.org/html/2607.16227#S1.SS2.p2.1),[§4\.1](https://arxiv.org/html/2607.16227#S4.SS1.p1.1)\.
- J\. Liet al\.\(2025\)Forget the token and pixel: rethinking gradient ascent for concept unlearning in multimodal generative models\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 12179–12200\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.630)Cited by:[§4\.2](https://arxiv.org/html/2607.16227#S4.SS2.p1.1)\.
- J\. Liet al\.\(2026\)Towards fair large language model\-based recommender systems without costly retraining\.arXiv preprint arXiv:2601\.17492\.External Links:[Link](https://arxiv.org/abs/2601.17492)Cited by:[§4\.1](https://arxiv.org/html/2607.16227#S4.SS1.p5.1)\.
- K\. Li, Q\. Wang, Y\. Wang, F\. Li, J\. Liu, B\. Han, and J\. Zhou \(2026\)LLM unlearning with LLM beliefs\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2510\.19422External Links:[Link](https://arxiv.org/abs/2510.19422)Cited by:[§4\.2\.1](https://arxiv.org/html/2607.16227#S4.SS2.SSS1.p13.1),[§4\.2\.2](https://arxiv.org/html/2607.16227#S4.SS2.SSS2.p5.1),[§6\.3](https://arxiv.org/html/2607.16227#S6.SS3.p3.1)\.
- M\. Li, L\. Chen, J\. Chen, S\. He, J\. Gu, and T\. Zhou \(2024a\)Selective reflection\-tuning: student\-selected data recycling for llm instruction\-tuning\.InFindings of the Association for Computational Linguistics ACL,pp\. 16189–16211\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.958)Cited by:[§3\.1](https://arxiv.org/html/2607.16227#S3.SS1.p1.1)\.
- N\. e\. al\. Li \(2024\)The wmdp benchmark: measuring and reducing malicious use with unlearning\.External Links:2403\.03218Cited by:[§2\.1](https://arxiv.org/html/2607.16227#S2.SS1.p5.1),[§4\.5](https://arxiv.org/html/2607.16227#S4.SS5.p2.1),[§6\.2\.1](https://arxiv.org/html/2607.16227#S6.SS2.SSS1.p1.1),[§6\.4](https://arxiv.org/html/2607.16227#S6.SS4.p2.1),[Table 5](https://arxiv.org/html/2607.16227#S6.T5.1.5.4.1.1.1),[§7\.3](https://arxiv.org/html/2607.16227#S7.SS3.p5.1)\.
- X\. Li, S\. Li, S\. Song, J\. Yang, J\. Ma, and J\. Yu \(2024b\)PMET: precise model editing in a transformer\.InProceedings of the 38th AAAI Conference on Artificial Intelligence \(AAAI\),Note:arXiv:2308\.08742External Links:[Link](https://arxiv.org/abs/2308.08742)Cited by:[§5\.1](https://arxiv.org/html/2607.16227#S5.SS1.p7.1)\.
- Y\. Li, M\. Yang, X\. Hu, and C\. Li \(2025\)Forget for get: a lightweight two\-phase gradient method for knowledge editing in large language models\.InFindings of the Association for Computational Linguistics: EMNLP 2025,pp\. 7604–7623\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.402)Cited by:[§4\.2\.1](https://arxiv.org/html/2607.16227#S4.SS2.SSS1.p8.1),[Table 3](https://arxiv.org/html/2607.16227#S4.T3.1.9.8.1.1.1)\.
- S\. Lin, J\. Hilton, and O\. Evans \(2022\)TruthfulQA: measuring how models mimic human falsehoods\.InACL,External Links:[Document](https://dx.doi.org/10.18653/v1/2022.acl-long.229)Cited by:[§6\.2\.2](https://arxiv.org/html/2607.16227#S6.SS2.SSS2.p1.1)\.
- Q\. Liuet al\.\(2024\)When moe meets llms: parameter efficient fine\-tuning for multi\-task medical applications\.InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 1104–1114\.External Links:[Document](https://dx.doi.org/10.1145/3626772.3657722)Cited by:[§3\.1](https://arxiv.org/html/2607.16227#S3.SS1.p1.1)\.
- Z\. e\. al\. Liu \(2024\)A survey on federated unlearning: challenges, methods, and future directions\.ACM Computing Surveys57\(1\),pp\. 1–38\.External Links:[Document](https://dx.doi.org/10.1145/3679014)Cited by:[§6\.4](https://arxiv.org/html/2607.16227#S6.SS4.p1.1)\.
- T\. Lizzo and L\. Heck \(2025\)UNLEARN: efficient removal of knowledge in large language models\.InFindings of the Association for Computational Linguistics: NAACL 2025,pp\. 7257–7268\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-naacl.405)Cited by:[§1](https://arxiv.org/html/2607.16227#S1.p2.1)\.
- T\. Lizzo and L\. Heck \(2026\)Unlearning in llms: methods, evaluation, and open challenges\.External Links:2601\.13264Cited by:[§6\.1](https://arxiv.org/html/2607.16227#S6.SS1.p1.1)\.
- J\. Lucki, B\. Wen, M\. Andriushchenko, N\. Flammarion, and F\. Tramèr \(2025\)An adversarial perspective on machine unlearning for AI safety\.Transactions on Machine Learning Research\.Note:arXiv:2409\.18025External Links:[Link](https://arxiv.org/abs/2409.18025)Cited by:[§2\.1](https://arxiv.org/html/2607.16227#S2.SS1.p5.1),[§2\.2](https://arxiv.org/html/2607.16227#S2.SS2.p1.1),[§2\.2](https://arxiv.org/html/2607.16227#S2.SS2.p4.1),[§4\.5](https://arxiv.org/html/2607.16227#S4.SS5.p2.1),[§6\.4](https://arxiv.org/html/2607.16227#S6.SS4.p2.1)\.
- M\.\-L\. Luo, Z\.\-H\. Zhou, Y\.\-L\. Zhang, Y\. Wan, T\. Wei, and M\.\-L\. Zhang \(2026\)KeepLoRA: continual learning with residual gradient adaptation\.External Links:2601\.19659Cited by:[§4\.4](https://arxiv.org/html/2607.16227#S4.SS4.p2.1)\.
- V\. Magesh, F\. Surani, M\. Dahl, M\. Suzgun, C\. Manning, and D\. Ho \(2024\)Hallucination\-free? assessing the reliability of leading ai legal research tools\.External Links:[Link](https://dho.stanford.edu/wp-content/uploads/Legal_RAG_Hallucinations.pdf)Cited by:[§2\.1](https://arxiv.org/html/2607.16227#S2.SS1.p7.1),[§2\.3](https://arxiv.org/html/2607.16227#S2.SS3.p1.1)\.
- P\. Maini, Z\. Feng, A\. Schwarzschild, Z\. C\. Lipton, and K\. J\. Zico \(2024\)TOFU: a task of fictitious unlearning for llms\.External Links:2401\.06121Cited by:[§6\.2\.1](https://arxiv.org/html/2607.16227#S6.SS2.SSS1.p1.1),[Table 5](https://arxiv.org/html/2607.16227#S6.T5.1.3.2.1.1.1),[§7\.3](https://arxiv.org/html/2607.16227#S7.SS3.p5.1)\.
- S\. Maity and M\. J\. Saikia \(2025\)Large language models in healthcare and medical applications: a review\.Bioengineering12\(6\),pp\. 631\.External Links:[Document](https://dx.doi.org/10.3390/bioengineering12060631)Cited by:[§1](https://arxiv.org/html/2607.16227#S1.p1.1)\.
- A\. Mantelero \(2013\)The EU proposal for a general data protection regulation and the roots of the right to be forgotten\.Computer Law and Security Review29\(3\),pp\. 229–235\.External Links:[Document](https://dx.doi.org/10.1016/j.clsr.2013.03.010)Cited by:[§2\.3](https://arxiv.org/html/2607.16227#S2.SS3.p1.1)\.
- I\. Mavrothalassitis, P\. Puigdemont, N\. Levi, and V\. Cevher \(2025\)Ascent fails to forget\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:PosterExternal Links:[Link](https://neurips.cc/virtual/2025/poster/118666)Cited by:[§4\.2\.2](https://arxiv.org/html/2607.16227#S4.SS2.SSS2.p1.1),[§4\.2\.2](https://arxiv.org/html/2607.16227#S4.SS2.SSS2.p3.1),[§4\.2\.2](https://arxiv.org/html/2607.16227#S4.SS2.SSS2.p6.1)\.
- L\. McKinneyet al\.\(2026\)Gauss\-newton unlearning for the llm era\.arXiv preprint arXiv:2602\.10568\.External Links:[Link](https://arxiv.org/abs/2602.10568)Cited by:[§4\.3](https://arxiv.org/html/2607.16227#S4.SS3.p4.1)\.
- K\. Meng, D\. Bau, A\. Andonian, and Y\. Belinkov \(2022\)Locating and editing factual associations in gpt\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/file/6f1d43d5a82a37e89b0665b33bf3a182-Paper-Conference.pdf)Cited by:[§3\.2](https://arxiv.org/html/2607.16227#S3.SS2.p1.1),[§4\.2\.1](https://arxiv.org/html/2607.16227#S4.SS2.SSS1.p1.1),[§5\.1](https://arxiv.org/html/2607.16227#S5.SS1.p2.1),[Table 4](https://arxiv.org/html/2607.16227#S5.T4.1.3.2.1.1.1)\.
- K\. Meng, A\. S\. Sharma, A\. Andonian, Y\. Belinkov, and D\. Bau \(2023\)Mass\-editing memory in a transformer\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2210\.07229External Links:[Link](https://arxiv.org/abs/2210.07229)Cited by:[§4\.2\.1](https://arxiv.org/html/2607.16227#S4.SS2.SSS1.p1.1),[§5\.1](https://arxiv.org/html/2607.16227#S5.SS1.p1.1),[§5\.1](https://arxiv.org/html/2607.16227#S5.SS1.p3.1),[Table 4](https://arxiv.org/html/2607.16227#S5.T4.1.4.3.1.1.1)\.
- E\. Mitchell, C\. Lin, A\. Bosselut, C\. Finn, and C\. D\. Manning \(2022a\)Memory\-based model editing at scale\.InProceedings of the 39th International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.162,pp\. 15817–15831\.External Links:[Link](https://proceedings.mlr.press/v162/mitchell22a.html)Cited by:[§5\.1](https://arxiv.org/html/2607.16227#S5.SS1.p4.1),[Table 4](https://arxiv.org/html/2607.16227#S5.T4.1.5.4.1.1.1)\.
- E\. Mitchell, C\. Lin, A\. Bosselut, C\. D\. Manning, and C\. Finn \(2022b\)Fast model editing at scale\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2110\.11309External Links:[Link](https://openreview.net/forum?id=0DcZxeWfOPt)Cited by:[§5\.1](https://arxiv.org/html/2607.16227#S5.SS1.p5.1),[Table 4](https://arxiv.org/html/2607.16227#S5.T4.1.6.5.1.1.1)\.
- H\. Naveedet al\.\(2025\)A comprehensive overview of large language models\.ACM Transactions on Intelligent Systems and Technology\.External Links:[Document](https://dx.doi.org/10.1145/3744746)Cited by:[§3\.1](https://arxiv.org/html/2607.16227#S3.SS1.p1.1)\.
- Z\. Pang, H\. Zheng, Z\. Deng, L\. Li, Z\. Zhong, and J\. Wei \(2025\)Label smoothing improves gradient ascent in LLM unlearning\.External Links:2510\.22376Cited by:[§4\.2\.1](https://arxiv.org/html/2607.16227#S4.SS2.SSS1.p18.1),[Table 3](https://arxiv.org/html/2607.16227#S4.T3.1.10.9.1.1.1)\.
- B\. Pastötter and C\. C\. Haciahmet \(2022\)Can people intentionally and selectively forget prose material?\.Frontiers in Psychology13\.External Links:[Document](https://dx.doi.org/10.3389/fpsyg.2022.928533)Cited by:[§1](https://arxiv.org/html/2607.16227#S1.p2.1)\.
- V\. Patil, P\. Hase, and M\. Bansal \(2024\)Can sensitive information be deleted from llms? objectives for defending against extraction attacks\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=7erlRDoaV8)Cited by:[§4\.3](https://arxiv.org/html/2607.16227#S4.SS3.p5.1),[§4\.4](https://arxiv.org/html/2607.16227#S4.SS4.p3.1),[§5\.1](https://arxiv.org/html/2607.16227#S5.SS1.p1.1),[§5\.1](https://arxiv.org/html/2607.16227#S5.SS1.p3.1),[§5\.1](https://arxiv.org/html/2607.16227#S5.SS1.p6.1)\.
- M\. Pawelczyk, S\. Neel, and H\. Lakkaraju \(2024\)In\-context unlearning: language models as few shot unlearners\.InProceedings of the 41st International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research\.Note:arXiv:2310\.07579External Links:[Link](https://arxiv.org/abs/2310.07579)Cited by:[§5\.3](https://arxiv.org/html/2607.16227#S5.SS3.p2.1),[Table 4](https://arxiv.org/html/2607.16227#S5.T4.1.7.6.1.1.1)\.
- P\. Peykani, F\. Ramezanlou, C\. Tanasescu, and S\. Ghanidel \(2025\)Large language models: a structured taxonomy and review of challenges, limitations, solutions, and future directions\.Applied Sciences15\(14\),pp\. 8103\.External Links:[Document](https://dx.doi.org/10.3390/app15148103)Cited by:[§3\.2](https://arxiv.org/html/2607.16227#S3.SS2.p1.1)\.
- A\. Radford, K\. Narasimhan, T\. Salimans, and I\. Sutskever \(2018\)Improving language understanding by generative pre\-training\.External Links:[Link](https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf)Cited by:[§3\.1](https://arxiv.org/html/2607.16227#S3.SS1.p1.1)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell, S\. Ermon, C\. D\. Manning, and C\. Finn \(2023\)Direct preference optimization: your language model is secretly a reward model\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.36\.Note:arXiv:2305\.18290External Links:[Link](https://arxiv.org/abs/2305.18290)Cited by:[§4\.2\.1](https://arxiv.org/html/2607.16227#S4.SS2.SSS1.p11.2)\.
- A\. Rezkellah and A\. Dakhmouche \(2025\)Machine unlearning meets adversarial robustness via constrained interventions on LLMs\.External Links:2510\.03567Cited by:[§6\.4](https://arxiv.org/html/2607.16227#S6.SS4.p7.1)\.
- A\. Robey, E\. Wong, H\. Hassani, and G\. J\. Pappas \(2025\)SmoothLLM: defending large language models against jailbreaking attacks\.Transactions on Machine Learning Research\.Note:arXiv:2310\.03684External Links:[Link](https://arxiv.org/abs/2310.03684)Cited by:[§6\.4](https://arxiv.org/html/2607.16227#S6.SS4.p7.1)\.
- L\. Schwinn, D\. Dobre, and S\. e\. al\. Xhonneux \(2024\)Soft prompt threats: attacking safety alignment and unlearning in open\-source llms through the embedding space\.External Links:2402\.09063Cited by:[§2\.2](https://arxiv.org/html/2607.16227#S2.SS2.p3.1),[§6\.4](https://arxiv.org/html/2607.16227#S6.SS4.p1.1),[§7\.3](https://arxiv.org/html/2607.16227#S7.SS3.p3.1),[§7](https://arxiv.org/html/2607.16227#S7.p2.1)\.
- N\. M\. Sepahvand, A\. Thudi, B\. Isik, A\. Bhattacharyya, N\. Papernot, E\. Triantafillou, D\. M\. Roy, and G\. K\. Dziugaite \(2025\)Leveraging per\-instance privacy for machine unlearning\.InProceedings of the 42nd International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.267\.Cited by:[§3\.2](https://arxiv.org/html/2607.16227#S3.SS2.p1.1)\.
- W\. Shi, V\. Shejwalkar, C\. Huang, Z\. Wan, A\. Garg, N\. Dikkala, N\. Kandpal, A\. Farhadi, L\. Zettlemoyer, and C\. A\. Choquette\-Choo \(2024\)MUSE: machine unlearning six\-way evaluation\.External Links:2407\.06460,[Link](https://arxiv.org/abs/2407.06460)Cited by:[Table 5](https://arxiv.org/html/2607.16227#S6.T5.1.6.5.1.1.1),[§7\.3](https://arxiv.org/html/2607.16227#S7.SS3.p5.1)\.
- G\. Sun, P\. Manakul, X\. Zhan, and M\. Gales \(2025\)Unlearning vs\. obfuscation: are we truly removing knowledge?\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 11468–11478\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.577)Cited by:[§3\.4](https://arxiv.org/html/2607.16227#S3.SS4.p1.1),[§4\.2\.3](https://arxiv.org/html/2607.16227#S4.SS2.SSS3.p1.1)\.
- R\. Tamirisa, B\. Bharathi, L\. Phan, A\. Zhou, A\. Gatti, T\. Suresh, M\. Lin, J\. Wang, R\. Wang, R\. Arel, A\. Zou, D\. Song, B\. Li, D\. Hendrycks, and M\. Mazeika \(2025\)Tamper\-resistant safeguards for open\-weight LLMs\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2408\.00761External Links:[Link](https://arxiv.org/abs/2408.00761)Cited by:[§6\.4](https://arxiv.org/html/2607.16227#S6.SS4.p6.1)\.
- Z\. e\. al\. Tan \(2026\)Can vlms truly forget? benchmarking training\-free visual concept unlearning\.External Links:2604\.03114Cited by:[§4\.5](https://arxiv.org/html/2607.16227#S4.SS5.p3.1)\.
- H\. Tang and R\. Khanna \(2026\)From logits to latents: contrastive representation shaping for llm unlearning\.External Links:2601\.22028Cited by:[§6](https://arxiv.org/html/2607.16227#S6.p1.1)\.
- E\. Triantafillou and P\. Kairouz \(2023\)Evaluation for the neurips machine unlearning competition\.Note:[https://api\.semanticscholar\.org/CorpusID:271974027](https://api.semanticscholar.org/CorpusID:271974027)Cited by:[§6\.3](https://arxiv.org/html/2607.16227#S6.SS3.p2.1)\.
- D\. Trippa, C\. Campagnano, M\. S\. Bucarelli, G\. Tolomei, and F\. Silvestri \(2024\)Gradient\-based and task\-agnostic machine unlearning\.arXiv preprint arXiv:2403\.14339\.Cited by:[§4\.1](https://arxiv.org/html/2607.16227#S4.SS1.p3.1),[§4\.2\.1](https://arxiv.org/html/2607.16227#S4.SS2.SSS1.p7.1),[Table 3](https://arxiv.org/html/2607.16227#S4.T3.1.6.5.1.1.1)\.
- P\. Trust and R\. Minghim \(2024\)A study on text classification in the age of large language models\.Machine Learning and Knowledge Extraction6\(4\),pp\. 2688–2721\.External Links:[Document](https://dx.doi.org/10.3390/make6040129)Cited by:[§3\.2](https://arxiv.org/html/2607.16227#S3.SS2.p1.1)\.
- United States District Court, S\.D\.N\.Y\. \(2023\)Mata v\. avianca, inc\., no\. 22\-cv\-1461, sanctions order\.Note:Judge P\. Kevin Castel\. June 22, 2023\. Available via PACERCited by:[§2\.3](https://arxiv.org/html/2607.16227#S2.SS3.p1.1)\.
- A\. Vassilev, A\. Oprea, A\. Fordyce, and H\. Anderson \(2025\)Adversarial machine learning: a taxonomy and terminology of attacks and mitigations\.Technical reportTechnical ReportNIST AI 100\-2,National Institute of Standards and Technology\.External Links:[Link](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-2e2025.pdf)Cited by:[§2\.2](https://arxiv.org/html/2607.16227#S2.SS2.p1.1)\.
- J\. Vrdoljak, Z\. Boban, M\. Vilović, M\. Kumrić, and J\. Božić \(2025\)A review of large language models in medical education, clinical decision support, and healthcare administration\.Healthcare13\(6\),pp\. 603\.External Links:[Document](https://dx.doi.org/10.3390/healthcare13060603)Cited by:[§1](https://arxiv.org/html/2607.16227#S1.p1.1)\.
- e\. al\. Wang \(2024\)Leak@k: unlearning does not make llms forget under probabilistic decoding\.arXiv preprint\.Cited by:[§4\.1](https://arxiv.org/html/2607.16227#S4.SS1.p4.2)\.
- J\. Wang, H\. Bie, Z\. Jing, and Y\. Zhi \(2026\)Feature\-indistinguishable machine unlearning via negative\-hot label encoding and class weight masking\.Scientific Reports16\(1\)\.External Links:[Document](https://dx.doi.org/10.1038/s41598-026-40379-9)Cited by:[§4\.4](https://arxiv.org/html/2607.16227#S4.SS4.p1.1)\.
- P\. Wang, Z\. Li, N\. Zhang, Z\. Xu, Y\. Yao, Y\. Jiang, P\. Xie, F\. Huang, and H\. Chen \(2024\)WISE: rethinking the knowledge memory for lifelong model editing of large language models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.37,pp\. 53764–53797\.Note:arXiv:2405\.14768External Links:[Link](https://arxiv.org/abs/2405.14768)Cited by:[§5\.1](https://arxiv.org/html/2607.16227#S5.SS1.p7.1)\.
- Z\. Wang and K\. Chen \(2024\)Machine unlearning in contrastive learning\.arXiv preprint arXiv:2405\.07317\.External Links:[Link](https://arxiv.org/abs/2405.07317)Cited by:[§4\.2\.2](https://arxiv.org/html/2607.16227#S4.SS2.SSS2.p4.1)\.
- A\. Wei, N\. Haghtalab, and J\. Steinhardt \(2023\)Jailbroken: how does llm safety training fail?\.InNeurIPS,External Links:[Link](https://openreview.net/forum?id=jA235JGM09)Cited by:[§1](https://arxiv.org/html/2607.16227#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.16227#S2.SS2.p2.1)\.
- C\. Wu, Z\. Wei, H\. Chen, Y\. Dong, and M\. Sun \(2025\)Reliable unlearning harmful information in llms with metamorphosis representation projection\.arXiv preprint arXiv:2508\.15449\.External Links:[Link](https://arxiv.org/abs/2508.15449)Cited by:[§3\.4](https://arxiv.org/html/2607.16227#S3.SS4.p1.1),[§6](https://arxiv.org/html/2607.16227#S6.p1.1)\.
- H\. Xu, T\. Zhu, L\. Zhang, W\. Zhou, and P\. S\. Yu \(2023\)Machine unlearning: a survey\.ACM Computing Surveys56\(1\),pp\. 1–36\.External Links:[Document](https://dx.doi.org/10.1145/3603620)Cited by:[§1\.1](https://arxiv.org/html/2607.16227#S1.SS1.p1.1),[§3\.4](https://arxiv.org/html/2607.16227#S3.SS4.p4.1)\.
- J\. Xu, Z\. Wu, C\. Wang, and X\. Jia \(2024\)Machine unlearning: solutions and challenges\.IEEE Transactions on Emerging Topics in Computational Intelligence8\(3\),pp\. 2150–2168\.External Links:[Document](https://dx.doi.org/10.1109/TETCI.2024.3379240)Cited by:[§3\.3](https://arxiv.org/html/2607.16227#S3.SS3.p1.4),[§3\.3](https://arxiv.org/html/2607.16227#S3.SS3.p5.3)\.
- Y\. Xu, J\. Guo, B\. Wen, and G\. Hu \(2026\)Forgetting\-MarI: LLM unlearning via marginal information regularization\.External Links:2511\.11914Cited by:[§4\.2\.1](https://arxiv.org/html/2607.16227#S4.SS2.SSS1.p15.1),[§4\.5](https://arxiv.org/html/2607.16227#S4.SS5.p4.1),[Table 3](https://arxiv.org/html/2607.16227#S4.T3.1.13.12.1.1.1)\.
- B\. A\. y Arcas \(2022\)Do large language models understand us?\.Daedalus151\(2\),pp\. 183–197\.External Links:[Document](https://dx.doi.org/10.1162/daed%5Fa%5F01909)Cited by:[§3\.1](https://arxiv.org/html/2607.16227#S3.SS1.p1.1)\.
- V\. Yadav, S\. Bethard, and M\. Surdeanu \(2019\)Quick and \(not so\) dirty: unsupervised selection of justification sentences for multi\-hop question answering\.InEMNLP\-IJCNLP,Cited by:[§6\.2\.2](https://arxiv.org/html/2607.16227#S6.SS2.SSS2.p1.1)\.
- H\. Yan, X\. Li, Z\. Guo, H\. Li, F\. Li, and X\. Lin \(2022\)ARCANE: an efficient architecture for exact machine unlearning\.InProceedings of the International Joint Conference on Artificial Intelligence \(IJCAI\),pp\. 4009–4015\.External Links:[Link](https://www.ijcai.org/proceedings/2022/556)Cited by:[§3\.3](https://arxiv.org/html/2607.16227#S3.SS3.p3.1)\.
- W\. Yang, L\. Some, M\. Bain, and B\. Kang \(2025\)A comprehensive survey on integrating large language models with knowledge\-based methods\.Knowledge\-Based Systems318,pp\. 113503\.External Links:[Document](https://dx.doi.org/10.1016/j.knosys.2025.113503)Cited by:[§1](https://arxiv.org/html/2607.16227#S1.p1.1)\.
- Z\. Yanget al\.\(2026\)CATNIP: LLM unlearning via calibrated and tokenized negative preference alignment\.External Links:2602\.02824Cited by:[Table 3](https://arxiv.org/html/2607.16227#S4.T3.1.14.13.1.1.1)\.
- H\. Yuan, Z\. Jin, and P\. e\. al\. Cao \(2024\)Towards robust knowledge unlearning: an adversarial framework for assessing and improving unlearning robustness in large language models\.External Links:2408\.10682Cited by:[§6\.4](https://arxiv.org/html/2607.16227#S6.SS4.p1.1),[§7\.3](https://arxiv.org/html/2607.16227#S7.SS3.p3.1),[§7](https://arxiv.org/html/2607.16227#S7.p2.1)\.
- M\. Yuanet al\.\(2024\)Large language models illuminate a progressive pathway to artificial intelligent healthcare assistant\.Medicine Plus1\(2\),pp\. 100030\.External Links:[Document](https://dx.doi.org/10.1016/j.medp.2024.100030)Cited by:[§1](https://arxiv.org/html/2607.16227#S1.p1.1)\.
- R\. e\. al\. Zellers \(2019\)HellaSwag: can a machine really finish your sentence?\.External Links:1905\.07830Cited by:[§6\.2\.2](https://arxiv.org/html/2607.16227#S6.SS2.SSS2.p1.1)\.
- C\. Zhanget al\.\(2025\)RULE: reinforcement unlearning achieves forget\-retain pareto optimality\.Note:NeurIPS 2025External Links:[Link](https://neurips.cc/virtual/2025/loc/san-diego/poster/116633)Cited by:[§1](https://arxiv.org/html/2607.16227#S1.p3.1)\.
- H\. Zhang, T\. Nakamura, T\. Isohara, and K\. Sakurai \(2023\)A review on machine unlearning\.SN Computer Science4\(4\)\.External Links:[Document](https://dx.doi.org/10.1007/s42979-023-01767-4)Cited by:[§1\.1](https://arxiv.org/html/2607.16227#S1.SS1.p1.1),[§3\.4](https://arxiv.org/html/2607.16227#S3.SS4.p3.1)\.
- R\. Zhang, L\. Lin, Y\. Bai, and S\. Mei \(2024\)Negative preference optimization: from catastrophic collapse to effective unlearning\.InProceedings of the 38th Annual Conference on Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2404\.05868External Links:[Link](https://arxiv.org/abs/2404.05868)Cited by:[§4\.2\.1](https://arxiv.org/html/2607.16227#S4.SS2.SSS1.p11.2),[§4\.2\.1](https://arxiv.org/html/2607.16227#S4.SS2.SSS1.p13.1),[Table 3](https://arxiv.org/html/2607.16227#S4.T3.1.7.6.1.1.1)\.
- A\. Zou, Z\. Wang, N\. Carlini, M\. Nasr, J\. Z\. Kolter, and M\. Fredrikson \(2023\)Universal and transferable adversarial attacks on aligned language models\.External Links:2307\.15043,[Link](https://arxiv.org/abs/2307.15043)Cited by:[§2\.2](https://arxiv.org/html/2607.16227#S2.SS2.p2.1)\.