DualOptim+: Bridging Shared and Decoupled Optimizer States for Better Machine Unlearning in Large Language Models

# Bridging Shared and Decoupled Optimizer States for Better Machine Unlearning in Large Language Models
Source: [https://arxiv.org/html/2605.21539](https://arxiv.org/html/2605.21539)
###### Abstract

We proposeDualOptim\+, a novel optimization framework for improving machine unlearning in large language models\. It introduces a base state to capture common representations shared by forgetting and retaining objectives and delta states to preserve objective\-specific residuals\. This architecture allows the optimizer to adaptively bridge shared and decoupled states based on the directional conflict between forgetting and retaining gradients\. We further introduce DualOptim\+ 8bit, a quantized variant that reduces memory overhead without compromising performance\. Extensive experiments across fictitious and real\-world unlearning, safety alignment, and multi\-task learning tasks demonstrate that DualOptim\+ consistently achieves a superior trade\-off between different objectives\. Codes are available at[https://github\.com/CityU\-MLO/DualOptimPlus](https://github.com/CityU-MLO/DualOptimPlus)\.

Machine Learning, ICML

## 1Introduction

Machine unlearning \(MU\)\(Bourtouleet al\.,[2021](https://arxiv.org/html/2605.21539#bib.bib1)\)aims to erase the influence of specific training data, known as the forget set, from pretrained models while preserving their general utility on the retain set\. While MU has been extensively studied in computer vision tasks, such as image classification and generation\(Heng and Soh,[2023](https://arxiv.org/html/2605.21539#bib.bib27); Kurmanjiet al\.,[2023](https://arxiv.org/html/2605.21539#bib.bib26); Fanet al\.,[2024b](https://arxiv.org/html/2605.21539#bib.bib3); Huanget al\.,[2024](https://arxiv.org/html/2605.21539#bib.bib4)\), the rapid rise of Large Language Models \(LLMs\) has underscored the need for efficient methods to remove outdated or unauthorized information\(Dang,[2021](https://arxiv.org/html/2605.21539#bib.bib28)\)\. In this regard, specialized unlearning techniques for LLMs have recently emerged as a significant area of research\(Yaoet al\.,[2024](https://arxiv.org/html/2605.21539#bib.bib29); Zhanget al\.,[2024](https://arxiv.org/html/2605.21539#bib.bib6); Yuanet al\.,[2025](https://arxiv.org/html/2605.21539#bib.bib5)\)\.

Despite recent progress, it is still challenging to balance the erasure of specific information and the preservation of general capability by optimizing various designed unlearning objectives\. Most existing LLM unlearning methods\(Yaoet al\.,[2024](https://arxiv.org/html/2605.21539#bib.bib29); Zhanget al\.,[2024](https://arxiv.org/html/2605.21539#bib.bib6); Yuanet al\.,[2025](https://arxiv.org/html/2605.21539#bib.bib5)\)minimize the objectives of the forget set and the retain set jointly by the sum of their gradients, which often leads to a significant degradation in model utility\. Inspired byFanet al\.\([2024b](https://arxiv.org/html/2605.21539#bib.bib3)\); Huanget al\.\([2024](https://arxiv.org/html/2605.21539#bib.bib4)\), alternately optimizing forgetting and retaining objectives shows promising unlearning performance, but it suffers from gradient entanglement for two conflicting objectives when using a shared optimizer state \(e\.g\., the moving average of gradients and squared gradients in Adam\)\. To address this issue, DualOptim\(Zhonget al\.,[2025](https://arxiv.org/html/2605.21539#bib.bib2)\)decouples optimizer states and uses separate optimizers for different objectives\. Despite the effectiveness in computer vision tasks, it yields marginal improvements for LLMs\. Therefore, developing a novel updating scheme is essential for effective LLM unlearning\.

In this work, we proposeDualOptim\+, a plug\-and\-play framework compatible with any optimizer with stored states\. It introduces a sharedbase statealongside decoupleddelta statesfor each optimization objective\. Specifically, the base state is updated using gradients from both the forgetting and retaining objectives, enabling it to capture their common representations\. In parallel, the delta states are updated by the residual between the objective\-specific gradients and the base state, thereby preserving distinct representations unique to each objective\. Finally, the parameters are updated by combining the base state and the delta state\.

Our theoretical and numerical analyses demonstrate that DualOptim\+ functions as an adaptive intermediate between fully shared and decoupled states, adjusting its behavior based on the degree of directional conflict between the forgetting and retaining gradients\. We validate the effectiveness of DualOptim\+ through extensive experiments across diverse machine unlearning tasks on various LLMs, including fictitious datasets, real\-world scenarios, and safety alignment tasks\. To mitigate the memory overhead associated with additional optimizer states, we also introduceDualOptim\+ 8bit\. This quantized variant significantly reduces memory consumption, while maintaining the peak performance\. Our results indicate that DualOptim\+ bridges the gap between decoupled and shared optimizer states to achieve a superior trade\-off between forgetting efficacy and model utility\. We believe, our method is a generalizable optimization framework, which can be applied in broader scenarios beyond machine unlearning, such as LLM alignment, multi\-objective learning, e\.t\.c\.

We summarize the contributions of this paper as follows:

1. 1\.We proposeDualOptim\+, which introduces a shared base state to capture common representations and decoupled delta states to preserve task\-specific residuals, effectively bridging the gap between shared and decoupled optimizer states\.
2. 2\.DualOptim\+ is a plug\-and\-play framework applicable to any multi\-objective optimization and optimizers with stored states\. Theoretical and numerical analysis demonstrate that DualOptim\+ functions as an intermediate between fully shared and decoupled states\.
3. 3\.Extensive experiments across various LLMs, datasets and tasks confirm that our method achieves a superior trade\-off between forgetting efficacy and model utility\. DualOptim\+ 8bit reduces memory overhead by quantization without compromising performance\.

## 2Related Work

Early efforts of machine unlearning \(MU\) are devoted to tasks such as image classification and image generation\(Bourtouleet al\.,[2021](https://arxiv.org/html/2605.21539#bib.bib1); Heng and Soh,[2023](https://arxiv.org/html/2605.21539#bib.bib27); Jiaet al\.,[2023](https://arxiv.org/html/2605.21539#bib.bib31); Kurmanjiet al\.,[2023](https://arxiv.org/html/2605.21539#bib.bib26); Tarunet al\.,[2023](https://arxiv.org/html/2605.21539#bib.bib30); Fanet al\.,[2024b](https://arxiv.org/html/2605.21539#bib.bib3); Huanget al\.,[2024](https://arxiv.org/html/2605.21539#bib.bib4)\)\. It has recently been adapted to address the unique challenges of Large Language Models \(LLMs\), such as removing sensitive or copyrighted training data\(Mainiet al\.,[2024](https://arxiv.org/html/2605.21539#bib.bib11)\), mitigating harmful behaviors\(Liet al\.,[2024](https://arxiv.org/html/2605.21539#bib.bib35)\), and efficiently handling the high\-dimensional parameter space\(Fanet al\.,[2024a](https://arxiv.org/html/2605.21539#bib.bib33)\)\. Based on how the model handles forgotten knowledge, MU methods on LLMs can be categorized intountargetedandtargetedunlearning\.

Inuntargeted unlearning, the objective is to eliminate the influence of specific data without constraining the model’s subsequent response to the forgotten content\. Common techniques for this paradigm include gradient ascent \(GA\)\(Thudiet al\.,[2021](https://arxiv.org/html/2605.21539#bib.bib32); Yaoet al\.,[2024](https://arxiv.org/html/2605.21539#bib.bib29)\), negative preference optimization \(NPO\)\(Fanet al\.,[2024a](https://arxiv.org/html/2605.21539#bib.bib33); Zhanget al\.,[2024](https://arxiv.org/html/2605.21539#bib.bib6)\), maximum entropy \(ME\)\(Yuanet al\.,[2025](https://arxiv.org/html/2605.21539#bib.bib5)\), and representation misalignment\(Liet al\.,[2024](https://arxiv.org/html/2605.21539#bib.bib35); Zouet al\.,[2024](https://arxiv.org/html/2605.21539#bib.bib34)\)\. Conversely,targeted unlearningaims to induce specific model behaviors when encountering forgotten information, such as providing standardized rejection responses \(e\.g\., “I don’t know”\)\. It is more user\-friendly than untargeted unlearning\. Popular methods for targeted unlearning include rejection fine\-tuning \(IDK\)\(Mainiet al\.,[2024](https://arxiv.org/html/2605.21539#bib.bib11)\), direct preference optimization \(DPO\)\(Rafailovet al\.,[2023](https://arxiv.org/html/2605.21539#bib.bib14)\), and self\-classification\(Gandikotaet al\.,[2024](https://arxiv.org/html/2605.21539#bib.bib36)\)\.

Besides erasing targeted information, it is also crucial for both untargeted and targeted unlearning to maintain general model utility and avoid catastrophic forgetting\. Most existing approaches incorporate cross\-entropy loss\(Gandikotaet al\.,[2024](https://arxiv.org/html/2605.21539#bib.bib36); Liet al\.,[2024](https://arxiv.org/html/2605.21539#bib.bib35); Yaoet al\.,[2024](https://arxiv.org/html/2605.21539#bib.bib29); Zhanget al\.,[2024](https://arxiv.org/html/2605.21539#bib.bib6)\)or divergence\-driven loss\(Yuanet al\.,[2025](https://arxiv.org/html/2605.21539#bib.bib5)\)to optimize the model utility on a designated retain set\.

Most LLM unlearning methods jointly update the objectives for the forget and the retain sets, but this optimization strategy usually leads to excessive forgetting and utility degradation\(Zhanget al\.,[2024](https://arxiv.org/html/2605.21539#bib.bib6)\)\. This issue is mitigated by alternatively using gradients from the forgetting and the retaining objectives\(Kurmanjiet al\.,[2023](https://arxiv.org/html/2605.21539#bib.bib26); Fanet al\.,[2024b](https://arxiv.org/html/2605.21539#bib.bib3); Huanget al\.,[2024](https://arxiv.org/html/2605.21539#bib.bib4)\)\. DualOptim\(Zhonget al\.,[2025](https://arxiv.org/html/2605.21539#bib.bib2)\)further improves the effectiveness and stability of unlearning by using two distinct optimizers with separate states\.

In addition to the aforementioned optimization\-based methods\(Jeunget al\.,[2025a](https://arxiv.org/html/2605.21539#bib.bib41),[b](https://arxiv.org/html/2605.21539#bib.bib43); Reisizadehet al\.,[2026](https://arxiv.org/html/2605.21539#bib.bib42)\), inference\-time adjustment methods aim to achieve efficient unlearning without modifying model parameters\. Theses methods are mainly based on output intervention\(Denget al\.,[2025](https://arxiv.org/html/2605.21539#bib.bib38); Wanget al\.,[2026](https://arxiv.org/html/2605.21539#bib.bib39)\)and in\-context learning\(Pawelczyket al\.,[2024](https://arxiv.org/html/2605.21539#bib.bib40)\)\.

## 3Method

### 3\.1Preliminaries

![Refer to caption](https://arxiv.org/html/2605.21539v1/x1.png)\(a\)Joint
![Refer to caption](https://arxiv.org/html/2605.21539v1/x2.png)\(b\)Alternate
![Refer to caption](https://arxiv.org/html/2605.21539v1/x3.png)\(c\)DualOptim
![Refer to caption](https://arxiv.org/html/2605.21539v1/x4.png)\(d\)DualOptim\+

Figure 1:Comparison of baselines and DualOptim\+\. The block represents the optimizer state\.\(a\) Jointupdating scheme sums both the forgetting and retaining losses and executes a single back\-propagation step\.\(b\) Alternateupdating scheme minimizes \([1](https://arxiv.org/html/2605.21539#S3.E1)\) by gradients from eitherℒf\{\\mathcal\{L\}\}\_\{f\}orℒr\{\\mathcal\{L\}\}\_\{r\}at each iteration, and alternates between the two objectives\.\(c\) DualOptimintroduces an independent optimizer state for each objective based on Alternate\.\(d\)DualOptim\+bridges the shared and decoupled states by introducing base and delta states\. The base state is jointly updated by∇θℒf\\nabla\_\{\\theta\}\{\\mathcal\{L\}\}\_\{f\}and∇θℒr\\nabla\_\{\\theta\}\{\\mathcal\{L\}\}\_\{r\}, andΔf\\Delta\_\{f\}/Δr\\Delta\_\{r\}is updated by the difference between∇θℒf\\nabla\_\{\\theta\}\{\\mathcal\{L\}\}\_\{f\}/∇θℒr\\nabla\_\{\\theta\}\{\\mathcal\{L\}\}\_\{r\}and base state\. Ultimately, the momentum term used for the parameter update is reconstructed by combining the base state with the respective delta state\.Machine unlearning \(MU\) seeks to solve the following optimization problem:

min𝜽⁡ℒf​\(𝜽,𝒟f\)\+ℒr​\(𝜽,𝒟r\),\\min\_\{\\bm\{\\theta\}\}\{\\mathcal\{L\}\}\_\{f\}\(\\bm\{\\theta\},\{\\mathcal\{D\}\}\_\{f\}\)\+\{\\mathcal\{L\}\}\_\{r\}\(\\bm\{\\theta\},\{\\mathcal\{D\}\}\_\{r\}\),\(1\)where𝜽\\bm\{\\theta\}represents the model parameter\. The forget and retain sets are denoted by𝒟f\{\\mathcal\{D\}\}\_\{f\}and𝒟r\{\\mathcal\{D\}\}\_\{r\}, respectively\.ℒf\{\\mathcal\{L\}\}\_\{f\}andℒr\{\\mathcal\{L\}\}\_\{r\}are the corresponding loss functions for forgetting and retaining objectives, respectively\. These objectives guide the model to eliminate the information contained in𝒟f\{\\mathcal\{D\}\}\_\{f\}while simultaneously preserving the utility on𝒟r\{\\mathcal\{D\}\}\_\{r\}\.

In the context of large language models \(LLMs\), most MU methods\(Zhanget al\.,[2024](https://arxiv.org/html/2605.21539#bib.bib6); Yuanet al\.,[2025](https://arxiv.org/html/2605.21539#bib.bib5)\)employ theJointupdating scheme \(see Figure[1](https://arxiv.org/html/2605.21539#S3.F1)\(a\)\)\. This scheme sums both the forgetting and retaining objectives and obtains the gradient by a single back\-propagation step to minimize \([1](https://arxiv.org/html/2605.21539#S3.E1)\)\. The updating step can be formulated as follows\.𝒫\{\\mathcal\{P\}\}represents an optimizer that may store states\.

𝜽←𝜽−𝒫​\(∇𝜽\(ℒf\+ℒr\)\)\.\{\\bm\{\\theta\}\}\\leftarrow\{\\bm\{\\theta\}\}\-\{\\mathcal\{P\}\}\(\\nabla\_\{\{\\bm\{\\theta\}\}\}\(\{\\mathcal\{L\}\}\_\{f\}\+\{\\mathcal\{L\}\}\_\{r\}\)\)\.\(2\)Some recent works\(Fanet al\.,[2024b](https://arxiv.org/html/2605.21539#bib.bib3); Huanget al\.,[2024](https://arxiv.org/html/2605.21539#bib.bib4)\)utilize theAlternateupdating scheme \(see Figure[1](https://arxiv.org/html/2605.21539#S3.F1)\(b\)\) to improve the performance\. This scheme minimizes \([1](https://arxiv.org/html/2605.21539#S3.E1)\) by gradients from eitherℒf\{\\mathcal\{L\}\}\_\{f\}orℒr\{\\mathcal\{L\}\}\_\{r\}at each iteration, and alternates between the two objectives\. Formally,

\{𝜽←𝜽−𝒫​\(∇𝜽ℒf\)for forget data𝜽←𝜽−𝒫​\(∇𝜽ℒr\)for retain data\.\\begin\{cases\}\{\\bm\{\\theta\}\}\\leftarrow\{\\bm\{\\theta\}\}\-\{\\mathcal\{P\}\}\(\\nabla\_\{\{\\bm\{\\theta\}\}\}\{\\mathcal\{L\}\}\_\{f\}\)&\\text\{for forget data\}\\\\ \{\\bm\{\\theta\}\}\\leftarrow\{\\bm\{\\theta\}\}\-\{\\mathcal\{P\}\}\(\\nabla\_\{\{\\bm\{\\theta\}\}\}\{\\mathcal\{L\}\}\_\{r\}\)&\\text\{for retain data\}\\end\{cases\}\.\(3\)Despite its effectiveness, the alternating updating scheme makes MU approaches unstable and sensitive to hyper\-parameter tuning\.DualOptim\(Zhonget al\.,[2025](https://arxiv.org/html/2605.21539#bib.bib2)\)\(see Figure[1](https://arxiv.org/html/2605.21539#S3.F1)\(c\)\) mitigates the unstable issue and further improves the effectiveness by using two decoupled optimizers𝒫f\{\\mathcal\{P\}\}\_\{f\},𝒫r\{\\mathcal\{P\}\}\_\{r\}with separate states\. This updating scheme disentangles the conflicting gradients from two objectives\. Formally,

\{𝜽←𝜽−𝒫f​\(∇𝜽ℒf\)for forget data𝜽←𝜽−𝒫r​\(∇𝜽ℒr\)for retain data\.\\begin\{cases\}\{\\bm\{\\theta\}\}\\leftarrow\{\\bm\{\\theta\}\}\-\{\\mathcal\{P\}\}\_\{f\}\(\\nabla\_\{\{\\bm\{\\theta\}\}\}\{\\mathcal\{L\}\}\_\{f\}\)&\\text\{for forget data\}\\\\ \{\\bm\{\\theta\}\}\\leftarrow\{\\bm\{\\theta\}\}\-\{\\mathcal\{P\}\}\_\{r\}\(\\nabla\_\{\{\\bm\{\\theta\}\}\}\{\\mathcal\{L\}\}\_\{r\}\)&\\text\{for retain data\}\\end\{cases\}\.\(4\)DualOptim is a plug\-and\-play technique applicable for various unlearning tasks\. However, compared with remarkable performance improvement in image classification and generation, applying DualOptim in LLMs yields relatively marginal improvement\. Our observations in Figure[2](https://arxiv.org/html/2605.21539#S3.F2)\(a\) show that the similarity between the decoupled momentum terms of DualOptim is near zero, implying that the gradients from the forgetting and retaining objectives in LLM unlearning are not always conflicting, especially in the later phase\. This phenomenon indicates that we should adaptively utilize both shared features and distinct features from the gradients\.

### 3\.2Bridging Shared and Decoupled Optimizer States

In this subsection, we proposeDualOptim\+to bridge the decoupled and shared optimizer states to improve the unlearning performance for LLMs \(see Figure[1](https://arxiv.org/html/2605.21539#S3.F1)\(d\)\)\. Specifically, DualOptim\+ decomposes each optimizer state into two components: a sharedbase stateand decoupleddelta statesfor forgetting and retaining objectives\. Without loss of generality, we use the first\-order momentum term as an example in the analyses below\. The updating rule can be straightforwardly extended to other optimizer states\.

Base State\.The base stateBBis introduced to reserve the common representation shared by the forgetting and retaining objectives\. It is updated jointly by∇θℒf\\nabla\_\{\\theta\}\{\\mathcal\{L\}\}\_\{f\}and∇θℒr\\nabla\_\{\\theta\}\{\\mathcal\{L\}\}\_\{r\}, effectively acting as a shared state\. Formally,

\{B←β​B\+\(1−β\)​∇θℒffor forget dataB←β​B\+\(1−β\)​∇θℒrfor retain data\.\\begin\{cases\}B\\leftarrow\\beta B\+\(1\-\\beta\)\\nabla\_\{\\theta\}\{\\mathcal\{L\}\}\_\{f\}&\\text\{for forget data\}\\\\ B\\leftarrow\\beta B\+\(1\-\\beta\)\\nabla\_\{\\theta\}\{\\mathcal\{L\}\}\_\{r\}&\\text\{for retain data\}\\end\{cases\}\.\(5\)whereβ∈\[0,1\)\\beta\\in\[0,1\)is the momentum factor\.

Delta State\.We introduce the delta statesΔf\\Delta\_\{f\},Δr\\Delta\_\{r\}to capture the historical residual information: the difference between the individual gradient and the base state\. This allows the delta states to reserve the distinct representations specific to the forgetting and retaining objectives\. Formally,

\{Δf←β​Δf\+\(1−β\)​\(∇θℒf−B^\)for forget dataΔr←β​Δr\+\(1−β\)​\(∇θℒr−B^\)for retain data\.\\begin\{cases\}\\Delta\_\{f\}\\leftarrow\\beta\\Delta\_\{f\}\+\(1\-\\beta\)\(\\nabla\_\{\\theta\}\{\\mathcal\{L\}\}\_\{f\}\-\\widehat\{B\}\)&\\text\{for forget data\}\\\\ \\Delta\_\{r\}\\leftarrow\\beta\\Delta\_\{r\}\+\(1\-\\beta\)\(\\nabla\_\{\\theta\}\{\\mathcal\{L\}\}\_\{r\}\-\\widehat\{B\}\)&\\text\{for retain data\}\\end\{cases\}\.\(6\)whereB^=B/\(1−βt\)\\widehat\{B\}=B/\(1\-\\beta^\{t\}\)denotes the bias\-corrected base state andttis the iteration number\.

Algorithm 1DualOptim\+ with AdamW1:Input:parameter

𝜽\{\\bm\{\\theta\}\}, learning rate

η\\eta, betas \(

β1\\beta\_\{1\},

β2\\beta\_\{2\}\), epsilon

ϵ\\epsilon, weight decay factor

λ\\lambda, forget objective

ℒf\{\\mathcal\{L\}\}\_\{f\}, retain objective

ℒr\{\\mathcal\{L\}\}\_\{r\}, total steps

NN, forget frequency

FfF\_\{f\}, retain frequency

FrF\_\{r\}
2:Initialize:

𝒎Δf←0\{\\bm\{m\}\}\_\{\\Delta\_\{f\}\}\\leftarrow 0,

𝒎Δr←0\{\\bm\{m\}\}\_\{\\Delta\_\{r\}\}\\leftarrow 0,

𝒎B←0\{\\bm\{m\}\}\_\{B\}\\leftarrow 0,

𝒗Δf←0\{\\bm\{v\}\}\_\{\\Delta\_\{f\}\}\\leftarrow 0,

𝒗Δr←0\{\\bm\{v\}\}\_\{\\Delta\_\{r\}\}\\leftarrow 0,

𝒗B←0\{\\bm\{v\}\}\_\{B\}\\leftarrow 0,

tf←0t\_\{f\}\\leftarrow 0,

tr←0t\_\{r\}\\leftarrow 0
3:

𝒎^B,𝒗^B←𝒎B,𝒗B\\widehat\{\{\\bm\{m\}\}\}\_\{B\},~\\widehat\{\{\\bm\{v\}\}\}\_\{B\}\\leftarrow\{\\bm\{m\}\}\_\{B\},~\{\\bm\{v\}\}\_\{B\}
4:for

t=1t=1to

NNdo

5:if

tmod\(Ff\+Fr\)≤Fft\\bmod\(F\_\{f\}\+F\_\{r\}\)\\leq F\_\{f\}then

6:

tf←tf\+1t\_\{f\}\\leftarrow t\_\{f\}\+1
7:

𝒈,𝒎Δ,𝒗Δ,t′←∇𝜽ℒf​\(𝜽\),𝒎Δf,𝒗Δf,tf\{\\bm\{g\}\},~\{\\bm\{m\}\}\_\{\\Delta\},~\{\\bm\{v\}\}\_\{\\Delta\},~t^\{\\prime\}\\leftarrow\\nabla\_\{\{\\bm\{\\theta\}\}\}\{\{\\mathcal\{L\}\}\_\{f\}\}\(\{\\bm\{\\theta\}\}\),~\{\\bm\{m\}\}\_\{\\Delta\_\{f\}\},~\{\\bm\{v\}\}\_\{\\Delta\_\{f\}\},~t\_\{f\}
8:else

9:

tr←tr\+1t\_\{r\}\\leftarrow t\_\{r\}\+1
10:

𝒈,𝒎Δ,𝒗Δ,t′←∇𝜽ℒr​\(𝜽\),𝒎Δr,𝒗Δr,tr\{\\bm\{g\}\},~\{\\bm\{m\}\}\_\{\\Delta\},~\{\\bm\{v\}\}\_\{\\Delta\},~t^\{\\prime\}\\leftarrow\\nabla\_\{\{\\bm\{\\theta\}\}\}\{\{\\mathcal\{L\}\}\_\{r\}\}\(\{\\bm\{\\theta\}\}\),~\{\\bm\{m\}\}\_\{\\Delta\_\{r\}\},~\{\\bm\{v\}\}\_\{\\Delta\_\{r\}\},~t\_\{r\}
11:endif

12:

𝜽←𝜽−η​λ​𝜽\{\\bm\{\\theta\}\}\\leftarrow\{\\bm\{\\theta\}\}\-\\eta\\lambda\{\\bm\{\\theta\}\}
13:

𝒎Δ←β1​𝒎Δ\+\(1−β1\)​\(𝒈−𝒎^B\)\{\\bm\{m\}\}\_\{\\Delta\}\\leftarrow\\beta\_\{1\}\{\\bm\{m\}\}\_\{\\Delta\}\+\(1\-\\beta\_\{1\}\)\(\{\\bm\{g\}\}\-\\widehat\{\{\\bm\{m\}\}\}\_\{B\}\)
14:

𝒗Δ←β2​𝒗Δ\+\(1−β2\)​\(𝒈2−𝒗^B\)\{\\bm\{v\}\}\_\{\\Delta\}\\leftarrow\\beta\_\{2\}\{\\bm\{v\}\}\_\{\\Delta\}\+\(1\-\\beta\_\{2\}\)\(\{\\bm\{g\}\}^\{2\}\-\\widehat\{\{\\bm\{v\}\}\}\_\{B\}\)
15:

𝒎^Δ,𝒗^Δ←𝒎Δ/\(1−β1t′\),𝒗Δ/\(1−β2t′\)\\widehat\{\{\\bm\{m\}\}\}\_\{\\Delta\},~\\widehat\{\{\\bm\{v\}\}\}\_\{\\Delta\}\\leftarrow\{\\bm\{m\}\}\_\{\\Delta\}/\(1\-\\beta\_\{1\}^\{t^\{\\prime\}\}\),~\{\\bm\{v\}\}\_\{\\Delta\}/\(1\-\\beta\_\{2\}^\{t^\{\\prime\}\}\)
16:

𝜽←𝜽−η​\(𝒎^B\+𝒎^Δ\)/\(\|𝒗^B\+𝒗^Δ\|\+ϵ\)\{\\bm\{\\theta\}\}\\leftarrow\{\\bm\{\\theta\}\}\-\\eta\(\\widehat\{\{\\bm\{m\}\}\}\_\{B\}\+\\widehat\{\{\\bm\{m\}\}\}\_\{\\Delta\}\)/\(\\sqrt\{\|\\widehat\{\{\\bm\{v\}\}\}\_\{B\}\+\\widehat\{\{\\bm\{v\}\}\}\_\{\\Delta\}\|\}\+\\epsilon\)
17:

𝒎B←β1​𝒎B\+\(1−β1\)​𝒈\{\\bm\{m\}\}\_\{B\}\\leftarrow\\beta\_\{1\}\{\\bm\{m\}\}\_\{B\}\+\(1\-\\beta\_\{1\}\)\{\\bm\{g\}\}
18:

𝒗B←β2​𝒗B\+\(1−β2\)​𝒈2\{\\bm\{v\}\}\_\{B\}\\leftarrow\\beta\_\{2\}\{\\bm\{v\}\}\_\{B\}\+\(1\-\\beta\_\{2\}\)\{\\bm\{g\}\}^\{2\}
19:

𝒎^B,𝒗^B←𝒎B/\(1−β1t\),𝒗B/\(1−β2t\)\\widehat\{\{\\bm\{m\}\}\}\_\{B\},~\\widehat\{\{\\bm\{v\}\}\}\_\{B\}\\leftarrow\{\\bm\{m\}\}\_\{B\}/\(1\-\\beta\_\{1\}^\{t\}\),~\{\\bm\{v\}\}\_\{B\}/\(1\-\\beta\_\{2\}^\{t\}\)
20:endfor

21:Output:parameter

𝜽\{\\bm\{\\theta\}\}

Parameter Update\.Finally, the momentum term used for the parameter update is reconstructed by combining the bias\-corrected base state \(B^\\widehat\{B\}\) with the respective bias\-corrected delta state \(Δ^f\\widehat\{\\Delta\}\_\{f\}orΔ^r\\widehat\{\\Delta\}\_\{r\}\), i\.e\.,B^\+Δ^f\\widehat\{B\}\+\\widehat\{\\Delta\}\_\{f\}orB^\+Δ^r\\widehat\{B\}\+\\widehat\{\\Delta\}\_\{r\}\. For Adam and its variants, the second\-order momentum is derived in a similar manner using a separate set of base and delta states updated by the squared gradients\(∇θℒf\)2\(\\nabla\_\{\\theta\}\{\\mathcal\{L\}\}\_\{f\}\)^\{2\}or\(∇θℒr\)2\(\\nabla\_\{\\theta\}\{\\mathcal\{L\}\}\_\{r\}\)^\{2\}\. The full pseudo\-code of DualOptim\+ integrated with AdamW is presented in Algorithm[1](https://arxiv.org/html/2605.21539#alg1)where we alternatively utilise∇θℒf\\nabla\_\{\\theta\}\{\\mathcal\{L\}\}\_\{f\}forFfF\_\{f\}iterations and∇θℒr\\nabla\_\{\\theta\}\{\\mathcal\{L\}\}\_\{r\}forFrF\_\{r\}iterations\. It should be noted that the base state is updated after the parameter update to maintain a stable reference for the delta states, thereby enhancing optimization stability\.

It should be emphasized that DualOptim\+ is a generic framework that can be integrated into any optimizer with stored states\. As an additional example, the pseudo\-code for DualOptim\+ integrated into Muon\(Jordanet al\.,[2024](https://arxiv.org/html/2605.21539#bib.bib7)\)is provided in Algorithm[2](https://arxiv.org/html/2605.21539#alg2)of Appendix[A](https://arxiv.org/html/2605.21539#A1)\.

In addition, our method share the similar idea with some federated learning methods\(Karimireddyet al\.,[2020](https://arxiv.org/html/2605.21539#bib.bib48),[2021](https://arxiv.org/html/2605.21539#bib.bib50); Wanget al\.,[2025](https://arxiv.org/html/2605.21539#bib.bib49); Cheng and Glasgow,[2025](https://arxiv.org/html/2605.21539#bib.bib51)\), i\.e\., the base \+ delta decomposition\. However, our method focuses on unlearning task, which is distinct from federated learning in terms of objectives\. We defer the detailed discussion and comparison in Appendix[D](https://arxiv.org/html/2605.21539#A4)\.

### 3\.3Analyses on DualOptim\+

The design of DualOptim\+ yields a hypothesis regarding its behavior: DualOptim\+ acts asan intermediate between fully shared state in \([3](https://arxiv.org/html/2605.21539#S3.E3)\) and fully decoupled states in \([4](https://arxiv.org/html/2605.21539#S3.E4)\), adapting based on the degree of directional conflict between∇θℒf\\nabla\_\{\\theta\}\{\\mathcal\{L\}\}\_\{f\}and∇θℒr\\nabla\_\{\\theta\}\{\\mathcal\{L\}\}\_\{r\}\.

Theoretical Analysis\.Without the loss of generality, we focus on one coordinate for notation simplicity since DualOptim\+ updates the parameters in an elementwise manner\. We consider the assumption below before detailed analyses\.

###### Assumption 3\.1\.

\(Gradient Dynamics\)\.Let\{gf,t∈ℝ\}t\\\{g\_\{f,t\}\\in\\mathbb\{R\}\\\}\_\{t\},\{gr,t∈ℝ\}t\\\{g\_\{r,t\}\\in\\mathbb\{R\}\\\}\_\{t\}be two sequences representing∇θℒf\\nabla\_\{\\theta\}\{\\mathcal\{L\}\}\_\{f\},∇θℒr\\nabla\_\{\\theta\}\{\\mathcal\{L\}\}\_\{r\}at the time steptt, respectively\. We assume the expectations ofgf,tg\_\{f,t\}andgr,tg\_\{r,t\}over time𝔼t​\[gf,t\]=m​G\\mathbb\{E\}\_\{t\}\\left\[g\_\{f,t\}\\right\]=mG,𝔼t​\[gr,t\]=n​G\\mathbb\{E\}\_\{t\}\\left\[g\_\{r,t\}\\right\]=nGexist, wherem,n∈\[−1,1\]m,n\\in\[\-1,1\], andGGis a non\-negative constant, denoting the largest possible gradient magnitude\.

Since the update rules in \([5](https://arxiv.org/html/2605.21539#S3.E5)\) and \([6](https://arxiv.org/html/2605.21539#S3.E6)\) involve gradients from two distinct distributions, standard convergence properties for stationary inputs do not directly apply\. Nevertheless, the following theorem validates the convergence of both base and delta states in \([5](https://arxiv.org/html/2605.21539#S3.E5)\) and \([6](https://arxiv.org/html/2605.21539#S3.E6)\)\.

###### Theorem 3\.2\.

\(Convergence of Base and Delta States\)\. Considering Assumption[3\.1](https://arxiv.org/html/2605.21539#S3.Thmtheorem1), the update rules \([5](https://arxiv.org/html/2605.21539#S3.E5)\) and \([6](https://arxiv.org/html/2605.21539#S3.E6)\), we useBtB\_\{t\},Δf,t\\Delta\_\{f,t\},Δr,t\\Delta\_\{r,t\}to represent the value of stateBB,Δf\\Delta\_\{f\},Δr\\Delta\_\{r\}at the time steptt, respectively\. We have the following asymptotical expectation:

limT→∞B\(Ff\+Fr\)​T\\displaystyle\\lim\_\{T\\to\\infty\}B\_\{\(F\_\{f\}\+F\_\{r\}\)T\}=βFr​\(1−βFf\)​m\+\(1−βFr\)​n1−βFf\+Fr​G,\\displaystyle=\\frac\{\\beta^\{F\_\{r\}\}\(1\-\\beta^\{F\_\{f\}\}\)m\+\(1\-\\beta^\{F\_\{r\}\}\)n\}\{1\-\\beta^\{F\_\{f\}\+F\_\{r\}\}\}G,\(7\)limT→∞Δf,\(Ff\+Fr\)​T\\displaystyle\\lim\_\{T\\to\\infty\}\\Delta\_\{f,\(F\_\{f\}\+F\_\{r\}\)T\}=Ff​βFf−1​\(1−β\)​\(1−βFr\)\(1−βFf\)​\(1−βFf\+Fr\)​\(m−n\)​G,\\displaystyle=\\frac\{F\_\{f\}\\beta^\{F\_\{f\}\-1\}\(1\-\\beta\)\\left\(1\-\\beta^\{F\_\{r\}\}\\right\)\}\{\\left\(1\-\\beta^\{F\_\{f\}\}\\right\)\\left\(1\-\\beta^\{F\_\{f\}\+F\_\{r\}\}\\right\)\}\(m\-n\)G,limT→∞Δr,\(Ff\+Fr\)​T\\displaystyle\\lim\_\{T\\to\\infty\}\\Delta\_\{r,\(F\_\{f\}\+F\_\{r\}\)T\}=Fr​βFr−1​\(1−β\)​\(1−βFf\)\(1−βFr\)​\(1−βFf\+Fr\)​\(n−m\)​G\.\\displaystyle=\\frac\{F\_\{r\}\\beta^\{F\_\{r\}\-1\}\(1\-\\beta\)\\left\(1\-\\beta^\{F\_\{f\}\}\\right\)\}\{\\left\(1\-\\beta^\{F\_\{r\}\}\\right\)\\left\(1\-\\beta^\{F\_\{f\}\+F\_\{r\}\}\\right\)\}\(n\-m\)G\.
whereFfF\_\{f\}andFrF\_\{r\}, denote the forget frequency and retain frequency respectively in Algorithm[1](https://arxiv.org/html/2605.21539#alg1)\.

The proof is deferred to Appendix[B\.1](https://arxiv.org/html/2605.21539#A2.SS1)\. According to Theorem[3\.2](https://arxiv.org/html/2605.21539#S3.Thmtheorem2), we can clearly see the expected magnitude of the base state is an interpolation of the expected gradients from the forgetting and the retaining objectives, while the delta states are closely related to their differences\. This is consistent with the motivation of DualOptim\+\. Specifically, we highlight the behavior of DualOptim\+ under the boundary conditions determined bymmandnn:

1. 1\.Positive Correlation:When the expectations of gradients are identicalm=nm=n, the base stateBBconverges tom​GmG, and the delta statesΔf\\Delta\_\{f\}andΔr\\Delta\_\{r\}converge to0\. In this case,DualOptim\+ acts like Alternate\.
2. 2\.Negative Correlation:When the expectations of gradients are negatively proportional with a specific factorm=−1−βFrβFr​\(1−βFf\)​nm=\-\\frac\{1\-\\beta^\{F\_\{r\}\}\}\{\\beta^\{F\_\{r\}\}\(1\-\\beta^\{F\_\{f\}\}\)\}n, the base stateBBconverges to0\. In this case,DualOptim\+ acts like DualOptim\.

![Refer to caption](https://arxiv.org/html/2605.21539v1/x5.png)\(a\)Update Similarity
![Refer to caption](https://arxiv.org/html/2605.21539v1/x6.png)\(b\)Gradient Similarity

Figure 2:Comparison of cosine similarity over time steps\.\(a\)Similarity between the update terms for forgetting and retaining of different methods using targeted unlearning objective\(Yuanet al\.,[2025](https://arxiv.org/html/2605.21539#bib.bib5)\)\.\(b\)Similarity between forgetting and retaining gradients of targeted and untargeted unlearning objectives\. For better visualization, the similarity is calculated based on the exponential moving average \(EMA\) of the gradients with a factor of 0\.9\. Note that the results are collected based on the forget10 task of TOFU\(Mainiet al\.,[2024](https://arxiv.org/html/2605.21539#bib.bib11)\)\. The model is TOFU\-finetuned Phi\-1\.5\(Liet al\.,[2023](https://arxiv.org/html/2605.21539#bib.bib8)\), and the optimizer is AdamW\.Numerical Analysis\.We conduct numerical analysis to validate the motivation of DualOptim\+\. In Figure[2](https://arxiv.org/html/2605.21539#S3.F2)\(a\), we show the cosine similarly between the two consecutive updates from the forgetting term and the retaining term for different algorithms\. Alternate uses the shared optimizer term, so the similarity is high\. By contrast, DualOptim fully decouples these optimizer states and has low similarity\. The similarity for DualOptim\+ is between Alternate and DualOptim, consistent with our motivation\.

Moreover, as illustrated in Figure[2](https://arxiv.org/html/2605.21539#S3.F2)\(b\), the cosine similarity between the forgetting and retaining gradients exhibits significant volatility throughout the process of both targeted and untargeted unlearning\. This instability suggests that the relationship between the two objectives is highly dynamic, further underscoring the necessity of an adaptive mechanism to transition between shared and decoupled states\.

## 4Experiments

To demonstrate the effectiveness of DualOptim\+ in LLM unlearning, we first define the evaluation metrics employed and then present extensive experimental results on unlearning, safety alignment, and multi\-task learning tasks\. In addition, a comprehensive ablation study is conducted for further analysis\. Detailed implementation settings are provided in Appendix[E](https://arxiv.org/html/2605.21539#A5)\.

### 4\.1Evaluation Metrics

To comprehensively evaluate the performance of the unlearned model, we adopt the evaluation framework established inYuanet al\.\([2025](https://arxiv.org/html/2605.21539#bib.bib5)\), focusing onModel Utility \(MU\)andForget Efficacy \(FE\), which are calculated over the retain and forget sets, respectively\. These aggregated metrics are based on six primary indicators, including ROUGE \(R\), Probability \(P\), Truth Ratio \(TR\), Token Entropy \(TE\), Cosine Similarity \(CS\), and Entailment Score \(ES\), except that TE is excluded from the calculation of FE\.

However, standard FE primarily measures the discrepancy between the model’s output and the ground truth in the forget set\. While suitable for untargeted unlearning, this is insufficient for targeted unlearning scenarios where the model is expected to provide a specific type of reject response\. To address this, we introduceTargeted Forget Efficacy \(TFE\), which quantifies the similarity between the model’s output and predefined rejection templates \(e\.g\., “Sorry, I don’t know”\)\. Given that rejection responses are randomly sampled from these templates, providing no fixed ground truth, we define TFE as the harmonic mean of Token Entropy \(TE\) and Entailment Score \(ES\) relative to the rejection response\. Consequently, we rename the original Forget Efficacy asUntargeted Forget Efficacy \(UFE\)\. In addition, we list some examples and their corresponding UFEs and TFEs in Appendix[F](https://arxiv.org/html/2605.21539#A6)\.

As a result, when evaluating the forget efficacy, we use the average of TFE and UFE for targeted unlearning and merely UFE for untargeted unlearning\. We define theOverall Performance \(OVR\)as the average of forget efficacy and model utility, so we haveOVR=0\.25×\(TFE\+UFE\)\+0\.5×MU\\mathrm\{OVR\}=0\.25\\times\(\\mathrm\{TFE\}\+\\mathrm\{UFE\}\)\+0\.5\\times\\mathrm\{MU\}for targeted unlearning andOVR=0\.5×\(UFE\+MU\)\\mathrm\{OVR\}=0\.5\\times\(\\mathrm\{UFE\}\+\\mathrm\{MU\}\)for untargeted unlearning\.

### 4\.2Unlearning in Fictitious Scenario

Table 1:Performance comparison of different methods on TOFU\-finetunedPhi 1\.5andLlama 2\.IDK\+GD\(targeted\) andME\+GD\(untargeted\) are adopted as the loss functions for unlearning\. The results include Untargeted Forget Efficacy \(UFE\), Targeted Forget Efficacy \(TFE\), Model Utility \(MU\) and the Overall Performance \(OVR\) for forget 1%, 5% data, and 10% data\. Note that the reported results are the average of the results obtained from55runs with different forget sets\.LossMethodPhi 1\.5forget 1% dataforget 5% dataforget 10% dataUFE↑\\uparrowTFE↑\\uparrowMU↑\\uparrowOVR↑\\uparrowUFE↑\\uparrowTFE↑\\uparrowMU↑\\uparrowOVR↑\\uparrowUFE↑\\uparrowTFE↑\\uparrowMU↑\\uparrowOVR↑\\uparrowIDK\+GDJoint78\.1145\.4518\.6140\.1972\.5558\.3236\.2650\.8571\.6564\.3933\.9250\.97Alternate73\.3562\.4948\.1458\.0367\.7364\.3047\.8156\.9165\.8264\.4649\.5457\.34DO74\.7563\.5146\.4657\.8068\.4964\.3449\.5057\.9665\.8766\.8450\.2558\.30DO 8bit75\.5861\.2346\.7357\.5768\.3464\.3348\.4157\.3765\.8165\.6050\.3858\.05DO\+75\.5167\.8547\.6959\.6967\.6367\.6051\.5259\.5765\.4266\.5051\.3258\.64DO\+ 8bit73\.6968\.3647\.5359\.2867\.5665\.9450\.3658\.5565\.2665\.5951\.3058\.36ME\+GDJoint95\.41–11\.4553\.4391\.32–33\.8762\.6091\.10–36\.8863\.99Alternate91\.46–45\.7868\.6292\.30–49\.7371\.0291\.96–48\.4870\.22DO92\.79–45\.2669\.0391\.97–51\.7371\.8692\.39–49\.2370\.81DO 8bit92\.83–45\.7569\.2993\.12–50\.9972\.0692\.32–48\.0470\.19DO\+93\.92–46\.1970\.0693\.07–50\.8771\.9792\.46–50\.3271\.39DO\+ 8bit92\.48–46\.1669\.3293\.13–50\.5571\.8492\.78–49\.9671\.37LossMethodLlama 2forget 1% dataforget 5% dataforget 10% dataUFE↑\\uparrowTFE↑\\uparrowMU↑\\uparrowOVR↑\\uparrowUFE↑\\uparrowTFE↑\\uparrowMU↑\\uparrowOVR↑\\uparrowUFE↑\\uparrowTFE↑\\uparrowMU↑\\uparrowOVR↑\\uparrowIDK\+GDJoint85\.9659\.5038\.6255\.6880\.2968\.6055\.6365\.0478\.0871\.2557\.1565\.91Alternate81\.9766\.4173\.9374\.0675\.7970\.2773\.9373\.4874\.1668\.1573\.9872\.57DO83\.1266\.4173\.8374\.3076\.5870\.2773\.6473\.5374\.6268\.1573\.1572\.27DO 8bit83\.1166\.4173\.7774\.2776\.4270\.2773\.7173\.5374\.6268\.1573\.3872\.38DO\+78\.7666\.4177\.4074\.9975\.1170\.2775\.9174\.3073\.9168\.1575\.4673\.25DO\+ 8bit79\.0566\.4176\.5374\.6375\.0470\.2776\.2774\.4273\.4968\.1576\.1473\.48ME\+GDJoint95\.89–59\.7077\.8097\.65–57\.1577\.4097\.66–60\.6379\.15Alternate97\.25–74\.8986\.0797\.07–75\.6486\.3696\.86–75\.3486\.10DO97\.46–75\.0686\.2696\.66–75\.9086\.2896\.78–75\.6086\.19DO 8bit97\.76–75\.5586\.6696\.74–75\.5286\.1396\.57–75\.4486\.01DO\+97\.88–75\.8286\.8596\.91–76\.0986\.5096\.85–75\.8686\.35DO\+ 8bit97\.50–75\.0186\.2696\.69–76\.1686\.4396\.78–75\.5286\.15We start with evaluating our method on the standard TOFU benchmark\(Mainiet al\.,[2024](https://arxiv.org/html/2605.21539#bib.bib11)\)\. TOFU simulates an ideal scenario with full data access, featuring 200 fictitious authors \(20 QA pairs each\)\. It includes three tasks: forget01, forget05, and forget10, targeting the removal of 1%, 5%, and 10% of the data, respectively\. The remaining data constitutes the retain set\. Additionally, “Real Authors” and “World Facts” sets are used to evaluate general knowledge utility\. We employ Phi\-1\.5\-1\.3B\(Liet al\.,[2023](https://arxiv.org/html/2605.21539#bib.bib8)\)and Llama\-2\-7B\(Touvronet al\.,[2023](https://arxiv.org/html/2605.21539#bib.bib9)\)released by TOFU as the target models, which has been fine\-tuned on the constructed data to ensure it can exactly gives answers to questions in TOFU\.

To evaluate the effectiveness ofDualOptim\+ \(DO\+\), we compare it against three baselines:Joint,Alternate, andDualOptim \(DO\)\(Zhonget al\.,[2025](https://arxiv.org/html/2605.21539#bib.bib2)\)\. Notably, DO and DO\+ require additional memory for optimizer states, specifically2×2\\timesand3×3\\timesthe memory consumption of a single standard Adam, respectively\. To mitigate this overhead, we introduce their 8\-bit versions,DO 8bitandDO\+ 8bit, implemented using thebitsandbyteslibrary to quantize optimizer states\. These 8\-bit variants significantly reduce memory requirements to1/21/2and3/43/4that of a single Adam\. To quantify this, we compare the running time and memory consumption of the evaluated methods in Appendix[C\.1](https://arxiv.org/html/2605.21539#A3.SS1)\.

The results on IDK\+GD \(targeted\)\(Mainiet al\.,[2024](https://arxiv.org/html/2605.21539#bib.bib11)\)and ME\+GD \(untargeted\)\(Yuanet al\.,[2025](https://arxiv.org/html/2605.21539#bib.bib5)\)loss functions are reported in Table[1](https://arxiv.org/html/2605.21539#S4.T1)\. We can observe that Joint suffers from excessive forgetting, leading to a significant degradation in model utility\. In contrast, DualOptim\+ \(DO\+\) achieves the best overall performance in most cases, with its superiority primarily stemming from its ability to preserve model utility while effectively unlearning\. Additionally, DO\+ yields a larger performance gain over DO in targeted unlearning compared to untargeted unlearning, as the forgetting and retaining objectives are less conflicted in the targeted setting, allowing DO\+ to better leverage its optimization advantages as an intermediate state between fully shared state and fully decoupled states\. Moreover, as illustrated in Figure[3](https://arxiv.org/html/2605.21539#A3.F3)of Appendix[C\.4](https://arxiv.org/html/2605.21539#A3.SS4), DO\+ demonstrates consistently better and more stable unlearning performance over time steps compared to the baselines\. Regarding efficiency, the proposed DO 8bit and DO\+ 8bit variants consume only1/41/4of the memory required by their standard 32\-bit counterparts \(DO and DO\+\) without compromising performance\.

Additionally, the results on DPO\+GD\(Rafailovet al\.,[2023](https://arxiv.org/html/2605.21539#bib.bib14)\)and NPO\+GD\(Zhanget al\.,[2024](https://arxiv.org/html/2605.21539#bib.bib6)\)loss functions are presented in Table[12](https://arxiv.org/html/2605.21539#A3.T12)of Appendix[C\.2](https://arxiv.org/html/2605.21539#A3.SS2)\. To further evaluate the efficacy of our method with parameter\-efficient fine\-tuning techniques, we report the results with LoRA\(Huet al\.,[2022](https://arxiv.org/html/2605.21539#bib.bib15)\)in Table[13](https://arxiv.org/html/2605.21539#A3.T13)of Appendix[C\.3](https://arxiv.org/html/2605.21539#A3.SS3)\. These observations are consistent with that in Table[1](https://arxiv.org/html/2605.21539#S4.T1), indicating the effectiveness of DualOptim\+ in broad scenarios\.

### 4\.3Unlearning in Real\-world Scenario

Table 2:Performance comparison of different methods onLlama 3for unlearning real\-world data\.IDK\+GD\(targeted\) andME\+GD\(untargeted\) are adopted as the loss functions for unlearning\. The results for unlearning tasks include Untargeted Forget Efficacy \(UFE\), Targeted Forget Efficacy \(TFE\), Model Utility \(MU\) and the Overall Performance \(OVR\)\. The average metrics \(AVG\) on downstream tasks are calculated\. Note that the reported results are the average of the results obtained from33runs using different random seeds\.LossMethodLlama 3Unlearning TaskDownstream TasksUFE↑\\uparrowTFE↑\\uparrowMU↑\\uparrowOVR↑\\uparrowARC\-c↑\\uparrowMMLU↑\\uparrowTruthfulQA↑\\uparrowTriviaQA↑\\uparrowGSM8K↑\\uparrowAVG↑\\uparrowInitial30\.55–61\.4546\.0055\.3864\.5937\.3350\.9376\.1256\.87IDK\+GDJoint85\.5472\.9627\.3853\.3246\.7962\.8533\.417\.5874\.3244\.99Alternate85\.4969\.9529\.1953\.4549\.7763\.3135\.6212\.7174\.1547\.11DO85\.2569\.6028\.0652\.7348\.4763\.2035\.2910\.3372\.3545\.93DO 8bit85\.2869\.6027\.6852\.5648\.7563\.0835\.0511\.5672\.3546\.16DO\+85\.7269\.9427\.9652\.9050\.8564\.4336\.3511\.1776\.0247\.77DO\+ 8bit85\.4769\.5933\.3655\.4552\.5664\.5136\.8017\.8675\.2149\.39ME\+GDJoint97\.97–24\.5361\.2543\.2963\.4625\.0529\.6162\.3444\.75Alternate97\.75–35\.2366\.4948\.6664\.0025\.3838\.3063\.6848\.00DO97\.67–37\.5167\.6045\.3663\.2725\.3437\.4537\.0741\.69DO 8bit97\.67–35\.4266\.5547\.9563\.6725\.9534\.2047\.5843\.87DO\+97\.85–48\.4073\.1356\.5264\.1634\.8028\.0873\.4451\.40DO\+ 8bit97\.77–49\.2973\.5256\.4564\.1431\.2929\.2272\.3850\.70We consider a realistic scenario where the unlearning is performed without access to the original training data\. FollowingLiuet al\.\([2024](https://arxiv.org/html/2605.21539#bib.bib16)\); Yuanet al\.\([2025](https://arxiv.org/html/2605.21539#bib.bib5)\), we identify real\-world individuals memorized by Llama\-3\-8B\-Instruct\(Grattafioriet al\.,[2024](https://arxiv.org/html/2605.21539#bib.bib10)\)\. We select 20 individuals as unlearning targets, generating the forget set using Llama 3’s own responses to 20 questions per person\. A neighbor set of 40 additional individuals is selected as the retain set: 20 of which are used for regularization during unlearning, while the remaining 20 are used to evaluate Model Utility\. Furthermore, general capability is evaluated via five downstream tasks: ARC\-c\(Clarket al\.,[2018](https://arxiv.org/html/2605.21539#bib.bib18)\), MMLU\(Hendryckset al\.,[2021](https://arxiv.org/html/2605.21539#bib.bib17)\), TruthfulQA\(Linet al\.,[2021](https://arxiv.org/html/2605.21539#bib.bib21)\), TriviaQA\(Joshiet al\.,[2017](https://arxiv.org/html/2605.21539#bib.bib20)\), and GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2605.21539#bib.bib19)\)\. Joint, Alternate, DualOptim \(DO\), DualOptim\+ \(DO\+\), and the 8bit variants of DO and DO\+ are evaluated in the experiment\. IDK\+GD and ME\+GD are adopted as the loss functions for targeted and untargeted unlearning, respectively\.

As shown in Table[2](https://arxiv.org/html/2605.21539#S4.T2), DO\+ and its 8\-bit variant \(DO\+ 8bit\) consistently achieves effective forgetting while maintaining superior machine utility on the retain set and downstream tasks\. Surprisingly, DO underperforms the simpler Alternate approach, particularly in model utility and downstream tasks, suggesting that the fully decoupled framework requires the additional refinements present in DO\+ to effectively handle the complexities of real\-world data unlearning\. Furthermore, compared with untargeted unlearning, the results indicate that targeted unlearning typically leads to a collapse in model utility compared to the initial state, dropping from61\.4561\.45to roughly half that value\. This phenomenon is likely to arise from a rigid mapping to specific ”I don’t know” responses in targeted objective, leading to catastrophic interference with the model’s internal representations\. In contrast, the untargeted objective merely aims to suppress the ground\-truth response through entropy maximization, imposing a “softer” optimization constraint that induces a more marginal representational shift and better preserves the model’s underlying reasoning logic\.

### 4\.4Safety Alignment

In this subsection, we extend our evaluation to safety alignment\(Baiet al\.,[2022](https://arxiv.org/html/2605.21539#bib.bib22)\), which can be framed as a specialized unlearning task\. The objective is to eliminate unsafe knowledge while preserving model utility\. Following the experimental setup ofBianchiet al\.\([2024](https://arxiv.org/html/2605.21539#bib.bib12)\), our initial model is Llama\-3\-8B\-Instruct fine\-tuned on Alpaca\(Taoriet al\.,[2023](https://arxiv.org/html/2605.21539#bib.bib13)\)to ensure robust instruction\-following capabilities\. We then simulate the unlearning process by tuning the model on 20,000 Alpaca instructions combined with 2,000 safety instructions; here, the safety and Alpaca instructions serve as the forget and retain sets, respectively\. FollowBianchiet al\.\([2024](https://arxiv.org/html/2605.21539#bib.bib12)\), we perform standard SFT on the mixed dataset, which is equivalent to targeted unlearning\. The objectives for untargeted unlearning are not considered here, since the objective does not meet the requirements of the task, i\.e\., reject replying to unsafe queries\.

To evaluate the model’s harmlessness, we utilize the same collection of harmful instruction datasets as that inBianchiet al\.\([2024](https://arxiv.org/html/2605.21539#bib.bib12)\), including I\-MaliciousInstructions \(I\-Mali\)\(Taoriet al\.,[2023](https://arxiv.org/html/2605.21539#bib.bib13)\), I\-CoNa\(Fantonet al\.,[2021](https://arxiv.org/html/2605.21539#bib.bib25)\), I\-Controversial \(I\-Cont\)\(Bianchiet al\.,[2024](https://arxiv.org/html/2605.21539#bib.bib12)\), and Q\-Harm\(Baiet al\.,[2022](https://arxiv.org/html/2605.21539#bib.bib22)\)\. Response safety is assessed using Llama\-Guard\-2\-8B\(Team,[2024](https://arxiv.org/html/2605.21539#bib.bib24)\), with the final safety rate reported as the primary metric\. Model utility is measured following the methodology described in Sec\.[4\.3](https://arxiv.org/html/2605.21539#S4.SS3)\. Additionally, we calculate the over\-refusal rate on XSTest\(Röttgeret al\.,[2024](https://arxiv.org/html/2605.21539#bib.bib23)\)to identify exaggerated safety behaviors\. We compare all methods previously discussed in Sec\.[4\.2](https://arxiv.org/html/2605.21539#S4.SS2)and[4\.3](https://arxiv.org/html/2605.21539#S4.SS3)\.

Table 3:Performance comparison of different methods onAlpaca\-finetuned Llama 3for safety alignment\. The averages \(AVG\) of the metrics on safety and utility tasks are calculated, respectively\. XSTest is used to evaluate the over\-refusal rate\. Note that the reported results are the average of the results obtained from33runs using different random seeds\.MethodAlpaca\-Llama 3SafetyUtilityOVR↑\\uparrowXSTest↓\\downarrowI\-Mali↑\\uparrowI\-CoNa↑\\uparrowI\-Cont↑\\uparrowQ\-Harm↑\\uparrowAVG↑\\uparrowARC\-c↑\\uparrowMMLU↑\\uparrowTruthfulQA↑\\uparrowTriviaQA↑\\uparrowGSM8K↑\\uparrowAVG↑\\uparrowInitial28\.0038\.7655\.0064\.0046\.4445\.5652\.5329\.7412\.1113\.1230\.6133\.560\.40Joint94\.6796\.6397\.5097\.0096\.4547\.0451\.6333\.7412\.1814\.1031\.7454\.8428\.00Alternate97\.0097\.3897\.5099\.6797\.8946\.8150\.8334\.6013\.8312\.9431\.8055\.6729\.20DO95\.6797\.9497\.5099\.3097\.6147\.3650\.1333\.5814\.0212\.9931\.6255\.7630\.27DO 8bit96\.0098\.3195\.5099\.0097\.2047\.1050\.7833\.5813\.8212\.9631\.6554\.6628\.27DO\+96\.0097\.5697\.5098\.6797\.4347\.2751\.8932\.2515\.3914\.2332\.8156\.4528\.13DO\+ 8bit96\.0098\.3197\.5099\.3397\.7946\.9351\.6634\.4713\.7114\.7332\.3055\.6128\.27

As shown in Table[3](https://arxiv.org/html/2605.21539#S4.T3), DO\+ achieves the superior safety\-utility trade\-off among the evaluated methods\. While all unlearning\-based approaches significantly improve the model’s safety metrics compared with the initial state, DO\+ distinguishes itself by attaining the highest overall performance \(56\.45%\) and the highest average utility score \(32\.81%\)\. This indicates that DO\+ is particularly effective at erasing harmful knowledge while preserving the model’s core capabilities\. Furthermore, DO\+ maintains a relatively low over\-refusal rate of 28\.13% on the XSTest benchmark, which is notably lower than that of the standard DO \(30\.27%\) and Alternate \(29\.20%\) methods\. These results suggest that DO\+ not only successfully aligns the model with safety requirements but also effectively mitigates the common pitfall of exaggerated safety behaviors\.

### 4\.5Multi\-task Learning

Our method can be easily extended to multi\-task learning tasks\. ForNNtasks, we need to maintain one base state andNNdelta states\. Specifically, we finetune Llama\-2\-7B on three different tasks, i\.e\., Py150 \(code\)\(Luet al\.,[2021](https://arxiv.org/html/2605.21539#bib.bib45)\), ScienceQA \(science\)\(Mishraet al\.,[2022](https://arxiv.org/html/2605.21539#bib.bib47)\), NumGLUE\-cm \(math\)\(Luet al\.,[2022](https://arxiv.org/html/2605.21539#bib.bib46)\)\. The datasets are collected from TRACE\(Wanget al\.,[2023](https://arxiv.org/html/2605.21539#bib.bib44)\), and we only evaluate DO 8bit and DO\+ 8bit to reduce memory consumption\.

The results listed in Table[4](https://arxiv.org/html/2605.21539#S4.T4)indicate that our method is still effective in the context of multi\-task learning\. Note that in the context of unlearning, the severity of gradient conflicts gives DO a distinct advantage\. Conversely, in multi\-task learning, where these conflicts are less pronounced, the gains from DO are negative, whereas DO\+ continues to deliver a substantial boost in performance\.

Table 4:Performance in multi\-task learning task\. We finetune Llama\-2\-7B on three different tasks, i\.e\., Py150 \(code\), ScienceQA \(science\) , NumGLUE\-cm \(math\)\.Py150ScienceQANumGLUE\-cmAVGJoint61\.0992\.4041\.6765\.05Alternate60\.7492\.0542\.8665\.22DO 8bit60\.3692\.2540\.4864\.36DO\+ 8bit60\.8791\.7548\.8167\.14
### 4\.6Ablation Study

In this subsection, we conduct ablation study on DualOptim\+ for further analysis\. If not specified, all experiments are conducted based on forgetting 5% data of TOFU on Phi1\.5 by using IDK\+GD loss function\. AdamW is adopted as the optimizer\.

Update timing of base state\.In Table[5](https://arxiv.org/html/2605.21539#S4.T5), we evaluate the impact of the update timing for the base state\. Specifically, we compare the performance when the base state is updated at different stages of the optimization step\. The results demonstrate that DualOptim\+ achieves optimal performance when the base state is updated after the parameter update\. This strategy ensures that the delta states are calculated against a stable and lagged reference, suppressing oscillations during optimization\.

Table 5:Unlearning performance when the base state is updated at different stages\.StageUFE↑\\uparrowTFE↑\\uparrowMU↑\\uparrowOVR↑\\uparrowBeforeΔ\\Delta67\.5466\.5550\.8258\.93AfterΔ\\Delta68\.2464\.3849\.9758\.14After𝜽\{\\bm\{\\theta\}\}67\.6367\.6051\.5259\.57Updating rule of base state\.In Table[6](https://arxiv.org/html/2605.21539#S4.T6), we evaluate the performance of DualOptim\+ when the base state is updated via different components, specifically comparing raw gradients \(𝒈\{\\bm\{g\}\}\) against the difference between gradients and bias\-corrected delta states \(𝒈−Δ^\{\\bm\{g\}\}\-\\widehat\{\\Delta\}\)\. The results demonstrate that updating the base state directly with gradients yields the best performance, reinforcing its role in capturing the shared representation between tasks\.

Table 6:Unlearning performance when the base state is updated by different components\.Updated byUFE↑\\uparrowTFE↑\\uparrowMU↑\\uparrowOVR↑\\uparrow𝒈\{\\bm\{g\}\}67\.6367\.6051\.5259\.57𝒈−Δ^\{\\bm\{g\}\}\-\\widehat\{\\Delta\}67\.7866\.7450\.8959\.08Momentum factors\.To avoid introducing additional hyperparameters, we utilize identical momentum factors \(β1,β2\\beta\_\{1\},\\beta\_\{2\}\) for both the base and delta states by default\. To evaluate the sensitivity of this configuration, we test two distinct momentum factor sets for AdamW:Fast\(β1=0\.9,β2=0\.95\\beta\_\{1\}=0\.9,\\beta\_\{2\}=0\.95, the default\) andSlow\(β1=0\.99,β2=0\.999\\beta\_\{1\}=0\.99,\\beta\_\{2\}=0\.999\)\. We examine the performance across various combinations of these sets as presented in Table[7](https://arxiv.org/html/2605.21539#S4.T7)\. Our results confirm that the default setting, where both states employ the fast momentum factors, achieves the best performance\. Notably, applying the slow set to both states leads to a significant performance drop, as the resulting updates are overly conservative for the unlearning task\.

Table 7:Unlearning performance when adopting different momentum factors\. We introduce two sets of momentum factors, Fast \(β1=0\.9\\beta\_\{1\}=0\.9,β2=0\.95\\beta\_\{2\}=0\.95\), Slow \(β1=0\.99\\beta\_\{1\}=0\.99,β2=0\.999\\beta\_\{2\}=0\.999\)\. The configuration \(F, S\) means that the Fast \(F\) set is used to update the delta states and the Slow \(S\) set is used to update the base state\.MomentumfactorsUFE↑\\uparrowTFE↑\\uparrowMU↑\\uparrowOVR↑\\uparrow\(F, F\)67\.6367\.6051\.5259\.57\(F, S\)67\.1467\.0050\.6958\.88\(S, F\)67\.8867\.5351\.3259\.51\(S, S\)64\.9064\.2750\.6057\.59Quantization\.To decrease memory overhead, we introduce an 8\-bit variant of DualOptim\+\. Table[8](https://arxiv.org/html/2605.21539#S4.T8)evaluates the impact of quantizing specific optimizer states\. The results demonstrate that quantization yields significant memory savings with only acceptable performance degradation\. Furthermore, we observe only a marginal performance gap when varying which states are quantized\. Therefore, to maximize memory efficiency, we adopt the quantization of all optimizer states as our default configuration\.

Table 8:Unlearning performance when quantizing different states\.QuantizeUFE↑\\uparrowTFE↑\\uparrowMU↑\\uparrowOVR↑\\uparrowNone67\.6367\.6051\.5259\.57BB67\.8466\.9050\.6659\.02Δ\\Delta67\.5566\.1950\.3158\.59B\+ΔB\+\\Delta67\.5665\.9450\.3658\.55Retain frequency\.In Table[9](https://arxiv.org/html/2605.21539#S4.T9), we investigate the impact of various retain frequencies \(FrF\_\{r\}\) on the Alternate, DualOptim \(DO\), and DualOptim\+ \(DO\+\), while maintaining a fixed forget frequency \(Ff=1F\_\{f\}=1\)\. The results indicate that the optimal overall performance is achieved atFr=5F\_\{r\}=5for our method\. While the sensitivity toFrF\_\{r\}is relatively low, we observe a general trade\-off: a smallerFrF\_\{r\}tends to enhance forget efficacy at the expense of model utility, whereas a largerFrF\_\{r\}better preserves utility but slightly diminishes unlearning effectiveness\. Notably, the improvement across differentFrF\_\{r\}for our method is stable and is consistently better than Alternate and DO\.

Table 9:Overall unlearning performance with different retain frequenciesFrF\_\{r\}\. The forget frequency is fixed to 1\. The total steps of unlearning is 300\.FrF\_\{r\}1245914Alter\.55\.8256\.1056\.1856\.2855\.7955\.06DO56\.5459\.4657\.9659\.1858\.2856\.96DO\+59\.1360\.0560\.1961\.3060\.8558\.16Hyperparameters of Joint\.In Table[10](https://arxiv.org/html/2605.21539#S4.T10), we report the results of a grid search over various forgetting loss weights \(with a fixed retaining loss weight of 1\) and learning rates for the Joint updating scheme\. The results indicate that the Joint method achieves its optimal performance using the default hyperparameters\. However, even with tuned hyperparameters, the Joint method consistently underperforms compared to other baselines, particularly in terms of maintaining model utility\.

Table 10:Unlearning performance of Joint with different\(a\)forgetting loss weights, and\(b\)different learning rates\.Weight0\.10\.20\.40\.60\.81\.0UFE↑\\uparrow72\.8973\.4173\.5974\.5974\.0874\.78TFE↑\\uparrow54\.4654\.5361\.5162\.9865\.8966\.45MU↑\\uparrow42\.8241\.7837\.9838\.4437\.5036\.90OVR↑\\uparrow53\.2552\.8852\.7753\.6153\.7453\.76\(a\)Forgetting Loss Weight
Weight5e\-61e\-52e\-54e\-56e\-5UFE↑\\uparrow72\.7274\.7875\.6180\.2181\.14TFE↑\\uparrow61\.6266\.4569\.3767\.9466\.55MU↑\\uparrow34\.7936\.9034\.9432\.9528\.10OVR↑\\uparrow50\.9853\.7653\.7253\.5150\.97\(b\)Learning Rate

## 5Conclusion

In this work, we proposed a novel optimization framework named DualOptim\+ for LLM unlearning that utilizes a base state to capture shared representations between forgetting and retaining objectives, alongside delta states that preserve objective\-specific residuals\. Our extensive evaluation across diverse unlearning scenarios demonstrates that DualOptim\+ provides a more stable and effective trade\-off between knowledge erasure and utility preservation than existing methods\. In future work, we aim to explore the applicability of our method in more general scenarios\.

## Impact Statement

Our method is a generalizable optimization framework to help achieve a better trade\-off when there are multiple conflicting learning objectives\. It can be applied in broad scenarios, including machine unlearning, safety alignment, multi\-task learning, etc\.

## Acknowledgements

This work is supported by City University of Hong Kong \(Project No\. 9220132, 9229203\)\.

## References

- Y\. Bai, A\. Jones, K\. Ndousse, A\. Askell, A\. Chen, N\. Dassarma, D\. Drain, S\. Fort, D\. Ganguli, T\. J\. Henighan, N\. Joseph, S\. Kadavath, J\. Kernion, T\. Conerly, S\. El\-Showk, N\. Elhage, Z\. Hatfield\-Dodds, D\. Hernandez, T\. Hume, S\. Johnston, S\. Kravec, L\. Lovitt, N\. Nanda, C\. Olsson, D\. Amodei, T\. B\. Brown, J\. Clark, S\. McCandlish, C\. Olah, B\. Mann, and J\. Kaplan \(2022\)Training a helpful and harmless assistant with reinforcement learning from human feedback\.ArXivabs/2204\.05862\.External Links:[Link](https://api.semanticscholar.org/CorpusID:248118878)Cited by:[§4\.4](https://arxiv.org/html/2605.21539#S4.SS4.p1.1),[§4\.4](https://arxiv.org/html/2605.21539#S4.SS4.p2.1)\.
- F\. Bianchi, M\. Suzgun, G\. Attanasio, P\. Rottger, D\. Jurafsky, T\. Hashimoto, and J\. Zou \(2024\)Safety\-tuned llamas: lessons from improving the safety of large language models that follow instructions\.InThe Twelfth International Conference on Learning Representations,Cited by:[Appendix E](https://arxiv.org/html/2605.21539#A5.p4.8),[§4\.4](https://arxiv.org/html/2605.21539#S4.SS4.p1.1),[§4\.4](https://arxiv.org/html/2605.21539#S4.SS4.p2.1)\.
- L\. Bourtoule, V\. Chandrasekaran, C\. A\. Choquette\-Choo, H\. Jia, A\. Travers, B\. Zhang, D\. Lie, and N\. Papernot \(2021\)Machine unlearning\.In2021 IEEE symposium on security and privacy \(SP\),pp\. 141–159\.Cited by:[§1](https://arxiv.org/html/2605.21539#S1.p1.1),[§2](https://arxiv.org/html/2605.21539#S2.p1.1)\.
- Z\. Cheng and M\. Glasgow \(2025\)Convergence of distributed adaptive optimization with local updates\.InInternational Conference on Learning Representations,Y\. Yue, A\. Garg, N\. Peng, F\. Sha, and R\. Yu \(Eds\.\),Vol\.2025,pp\. 57694–57751\.Cited by:[Appendix D](https://arxiv.org/html/2605.21539#A4.p2.1),[§3\.2](https://arxiv.org/html/2605.21539#S3.SS2.p6.1)\.
- P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord \(2018\)Think you have solved question answering? try arc, the ai2 reasoning challenge\.ArXivabs/1803\.05457\.External Links:[Link](https://api.semanticscholar.org/CorpusID:3922816)Cited by:[§4\.3](https://arxiv.org/html/2605.21539#S4.SS3.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman \(2021\)Training verifiers to solve math word problems\.ArXivabs/2110\.14168\.External Links:[Link](https://api.semanticscholar.org/CorpusID:239998651)Cited by:[§4\.3](https://arxiv.org/html/2605.21539#S4.SS3.p1.1)\.
- Q\. Dang \(2021\)Right to be forgotten in the age of machine learning\.InAdvances in Digital Science: ICADS 2021,pp\. 403–411\.Cited by:[§1](https://arxiv.org/html/2605.21539#S1.p1.1)\.
- Z\. Deng, C\. Y\. Liu, Z\. Pang, X\. He, L\. Feng, Q\. Xuan, Z\. Zhu, and J\. Wei \(2025\)GUARD: generation\-time LLM unlearning via adaptive restriction and detection\.InICML 2025 Workshop on Machine Unlearning for Generative AI,External Links:[Link](https://openreview.net/forum?id=HuIocuFAkY)Cited by:[§2](https://arxiv.org/html/2605.21539#S2.p5.1)\.
- C\. Fan, J\. Liu, L\. Lin, J\. Jia, R\. Zhang, S\. Mei, and S\. Liu \(2024a\)Simplicity prevails: rethinking negative preference optimization for LLM unlearning\.InNeurips Safe Generative AI Workshop 2024,External Links:[Link](https://openreview.net/forum?id=pVACX02m0p)Cited by:[§2](https://arxiv.org/html/2605.21539#S2.p1.1),[§2](https://arxiv.org/html/2605.21539#S2.p2.1)\.
- C\. Fan, J\. Liu, Y\. Zhang, D\. Wei, E\. Wong, and S\. Liu \(2024b\)SalUn: empowering machine unlearning via gradient\-based weight saliency in both image classification and generation\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2605.21539#S1.p1.1),[§1](https://arxiv.org/html/2605.21539#S1.p2.1),[§2](https://arxiv.org/html/2605.21539#S2.p1.1),[§2](https://arxiv.org/html/2605.21539#S2.p4.1),[§3\.1](https://arxiv.org/html/2605.21539#S3.SS1.p2.3)\.
- M\. Fanton, H\. Bonaldi, S\. S\. Tekiroğlu, and M\. Guerini \(2021\)Human\-in\-the\-loop for data collection: a multi\-target counter narrative dataset to fight online hate speech\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),C\. Zong, F\. Xia, W\. Li, and R\. Navigli \(Eds\.\),Online,pp\. 3226–3240\.External Links:[Link](https://aclanthology.org/2021.acl-long.250/),[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.250)Cited by:[§4\.4](https://arxiv.org/html/2605.21539#S4.SS4.p2.1)\.
- R\. Gandikota, S\. Feucht, S\. Marks, and D\. Bau \(2024\)Erasing conceptual knowledge from language models\.arXiv preprint arXiv:2410\.02760\.Cited by:[§2](https://arxiv.org/html/2605.21539#S2.p2.1),[§2](https://arxiv.org/html/2605.21539#S2.p3.1)\.
- L\. Gao, J\. Tow, B\. Abbasi, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golding, J\. Hsu, A\. Le Noac’h, H\. Li, K\. McDonell, N\. Muennighoff, C\. Ociepa, J\. Phang, L\. Reynolds, H\. Schoelkopf, A\. Skowron, L\. Sutawika, E\. Tang, A\. Thite, B\. Wang, K\. Wang, and A\. Zou \(2024\)The language model evaluation harness\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.12608602),[Link](https://zenodo.org/records/12608602)Cited by:[Appendix E](https://arxiv.org/html/2605.21539#A5.p3.2)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§4\.3](https://arxiv.org/html/2605.21539#S4.SS3.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt \(2021\)Measuring massive multitask language understanding\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=d7KBjmI3GmQ)Cited by:[§4\.3](https://arxiv.org/html/2605.21539#S4.SS3.p1.1)\.
- A\. Heng and H\. Soh \(2023\)Selective amnesia: a continual learning approach to forgetting in deep generative models\.Advances in Neural Information Processing Systems36,pp\. 17170–17194\.Cited by:[§1](https://arxiv.org/html/2605.21539#S1.p1.1),[§2](https://arxiv.org/html/2605.21539#S2.p1.1)\.
- E\. J\. Hu, yelong shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[§C\.3](https://arxiv.org/html/2605.21539#A3.SS3.p1.1),[§4\.2](https://arxiv.org/html/2605.21539#S4.SS2.p4.1)\.
- Z\. Huang, X\. Cheng, J\. Zheng, H\. Wang, Z\. He, T\. Li, and X\. Huang \(2024\)Unified gradient\-based machine unlearning with remain geometry enhancement\.Advances in Neural Information Processing Systems37,pp\. 26377–26414\.Cited by:[§1](https://arxiv.org/html/2605.21539#S1.p1.1),[§1](https://arxiv.org/html/2605.21539#S1.p2.1),[§2](https://arxiv.org/html/2605.21539#S2.p1.1),[§2](https://arxiv.org/html/2605.21539#S2.p4.1),[§3\.1](https://arxiv.org/html/2605.21539#S3.SS1.p2.3)\.
- W\. Jeung, S\. Yoon, H\. Hong, S\. Kim, S\. Han, Y\. Yu, and A\. No \(2025a\)DUSK: do not unlearn shared knowledge\.ArXivabs/2505\.15209\.External Links:[Link](https://api.semanticscholar.org/CorpusID:278782469)Cited by:[§2](https://arxiv.org/html/2605.21539#S2.p5.1)\.
- W\. Jeung, S\. Yoon, and A\. No \(2025b\)SEPS: a separability measure for robust unlearning in LLMs\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 5556–5587\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.283/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.283),ISBN 979\-8\-89176\-332\-6Cited by:[§2](https://arxiv.org/html/2605.21539#S2.p5.1)\.
- J\. Jia, J\. Liu, P\. Ram, Y\. Yao, G\. Liu, Y\. Liu, P\. Sharma, and S\. Liu \(2023\)Model sparsity can simplify machine unlearning\.Advances in Neural Information Processing Systems36,pp\. 51584–51605\.Cited by:[§2](https://arxiv.org/html/2605.21539#S2.p1.1)\.
- K\. Jordan, Y\. Jin, V\. Boza, J\. You, F\. Cesista, L\. Newhouse, and J\. Bernstein \(2024\)Muon: an optimizer for hidden layers in neural networks\.External Links:[Link](https://kellerjordan.github.io/posts/muon/)Cited by:[Appendix A](https://arxiv.org/html/2605.21539#A1.p1.1),[§3\.2](https://arxiv.org/html/2605.21539#S3.SS2.p5.1)\.
- M\. Joshi, E\. Choi, D\. Weld, and L\. Zettlemoyer \(2017\)TriviaQA: a large scale distantly supervised challenge dataset for reading comprehension\.InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),R\. Barzilay and M\. Kan \(Eds\.\),Vancouver, Canada,pp\. 1601–1611\.External Links:[Link](https://aclanthology.org/P17-1147/),[Document](https://dx.doi.org/10.18653/v1/P17-1147)Cited by:[§4\.3](https://arxiv.org/html/2605.21539#S4.SS3.p1.1)\.
- S\. P\. Karimireddy, M\. Jaggi, S\. Kale, M\. Mohri, S\. J\. Reddi, S\. U\. Stich, and A\. T\. Suresh \(2021\)Mime: mimicking centralized stochastic algorithms in federated learning\.External Links:2008\.03606,[Link](https://arxiv.org/abs/2008.03606)Cited by:[Appendix D](https://arxiv.org/html/2605.21539#A4.p2.1),[§3\.2](https://arxiv.org/html/2605.21539#S3.SS2.p6.1)\.
- S\. P\. Karimireddy, S\. Kale, M\. Mohri, S\. Reddi, S\. Stich, and A\. T\. Suresh \(2020\)SCAFFOLD: stochastic controlled averaging for federated learning\.InProceedings of the 37th International Conference on Machine Learning,H\. D\. III and A\. Singh \(Eds\.\),Proceedings of Machine Learning Research, Vol\.119,pp\. 5132–5143\.External Links:[Link](https://proceedings.mlr.press/v119/karimireddy20a.html)Cited by:[Appendix D](https://arxiv.org/html/2605.21539#A4.p2.1),[§3\.2](https://arxiv.org/html/2605.21539#S3.SS2.p6.1)\.
- J\. Konečný, B\. McMahan, and D\. Ramage \(2015\)Federated optimization:distributed optimization beyond the datacenter\.External Links:1511\.03575,[Link](https://arxiv.org/abs/1511.03575)Cited by:[Appendix D](https://arxiv.org/html/2605.21539#A4.p1.1)\.
- M\. Kurmanji, P\. Triantafillou, J\. Hayes, and E\. Triantafillou \(2023\)Towards unbounded machine unlearning\.Advances in neural information processing systems36,pp\. 1957–1987\.Cited by:[§1](https://arxiv.org/html/2605.21539#S1.p1.1),[§2](https://arxiv.org/html/2605.21539#S2.p1.1),[§2](https://arxiv.org/html/2605.21539#S2.p4.1)\.
- N\. Li, A\. Pan, A\. Gopal, S\. Yue, D\. Berrios, A\. Gatti, J\. D\. Li, A\. Dombrowski, S\. Goel, G\. Mukobi, N\. Helm\-Burger, R\. Lababidi, L\. Justen, A\. B\. Liu, M\. Chen, I\. Barrass, O\. Zhang, X\. Zhu, R\. Tamirisa, B\. Bharathi, A\. Herbert\-Voss, C\. B\. Breuer, A\. Zou, M\. Mazeika, Z\. Wang, P\. Oswal, W\. Lin, A\. A\. Hunt, J\. Tienken\-Harder, K\. Y\. Shih, K\. Talley, J\. Guan, I\. Steneker, D\. Campbell, B\. Jokubaitis, S\. Basart, S\. Fitz, P\. Kumaraguru, K\. K\. Karmakar, U\. Tupakula, V\. Varadharajan, Y\. Shoshitaishvili, J\. Ba, K\. M\. Esvelt, A\. Wang, and D\. Hendrycks \(2024\)The WMDP benchmark: measuring and reducing malicious use with unlearning\.InForty\-first International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=xlr6AUDuJz)Cited by:[§2](https://arxiv.org/html/2605.21539#S2.p1.1),[§2](https://arxiv.org/html/2605.21539#S2.p2.1),[§2](https://arxiv.org/html/2605.21539#S2.p3.1)\.
- Y\. Li, S\. Bubeck, R\. Eldan, A\. Del Giorno, S\. Gunasekar, and Y\. T\. Lee \(2023\)Textbooks are all you need ii: phi\-1\.5 technical report\.arXiv preprint arXiv:2309\.05463\.Cited by:[Figure 2](https://arxiv.org/html/2605.21539#S3.F2),[§4\.2](https://arxiv.org/html/2605.21539#S4.SS2.p1.1)\.
- S\. C\. Lin, J\. Hilton, and O\. Evans \(2021\)TruthfulQA: measuring how models mimic human falsehoods\.InAnnual Meeting of the Association for Computational Linguistics,External Links:[Link](https://api.semanticscholar.org/CorpusID:237532606)Cited by:[§4\.3](https://arxiv.org/html/2605.21539#S4.SS3.p1.1)\.
- Z\. Liu, T\. Zhu, C\. Tan, and W\. Chen \(2024\)Learning to refuse: towards mitigating privacy risks in llms\.arXiv preprint arXiv:2407\.10058\.Cited by:[Appendix E](https://arxiv.org/html/2605.21539#A5.p3.2),[§4\.3](https://arxiv.org/html/2605.21539#S4.SS3.p1.1)\.
- P\. Lu, S\. Mishra, T\. Xia, L\. Qiu, K\. Chang, S\. Zhu, O\. Tafjord, P\. Clark, and A\. Kalyan \(2022\)Learn to explain: multimodal reasoning via thought chains for science question answering\.InAdvances in Neural Information Processing Systems,A\. H\. Oh, A\. Agarwal, D\. Belgrave, and K\. Cho \(Eds\.\),External Links:[Link](https://openreview.net/forum?id=HjwK-Tc_Bc)Cited by:[§4\.5](https://arxiv.org/html/2605.21539#S4.SS5.p1.2)\.
- S\. Lu, D\. Guo, S\. Ren, J\. Huang, A\. Svyatkovskiy, A\. Blanco, C\. Clement, D\. Drain, D\. Jiang, D\. Tang, G\. Li, L\. Zhou, L\. Shou, L\. Zhou, M\. Tufano, M\. GONG, M\. Zhou, N\. Duan, N\. Sundaresan, S\. K\. Deng, S\. Fu, and S\. LIU \(2021\)CodeXGLUE: a machine learning benchmark dataset for code understanding and generation\.InThirty\-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track \(Round 1\),External Links:[Link](https://openreview.net/forum?id=6lE4dQXaUcb)Cited by:[§4\.5](https://arxiv.org/html/2605.21539#S4.SS5.p1.2)\.
- P\. Maini, Z\. Feng, A\. Schwarzschild, Z\. C\. Lipton, and J\. Z\. Kolter \(2024\)TOFU: a task of fictitious unlearning for llms\.InFirst Conference on Language Modeling,Cited by:[Appendix E](https://arxiv.org/html/2605.21539#A5.p2.13),[§2](https://arxiv.org/html/2605.21539#S2.p1.1),[§2](https://arxiv.org/html/2605.21539#S2.p2.1),[Figure 2](https://arxiv.org/html/2605.21539#S3.F2),[§4\.2](https://arxiv.org/html/2605.21539#S4.SS2.p1.1),[§4\.2](https://arxiv.org/html/2605.21539#S4.SS2.p3.1)\.
- S\. Mishra, A\. Mitra, N\. Varshney, B\. Sachdeva, P\. Clark, C\. Baral, and A\. Kalyan \(2022\)NumGLUE: a suite of fundamental yet challenging mathematical reasoning tasks\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 3505–3523\.External Links:[Link](https://aclanthology.org/2022.acl-long.246/),[Document](https://dx.doi.org/10.18653/v1/2022.acl-long.246)Cited by:[§4\.5](https://arxiv.org/html/2605.21539#S4.SS5.p1.2)\.
- M\. Pawelczyk, S\. Neel, and H\. Lakkaraju \(2024\)In\-context unlearning: language models as few\-shot unlearners\.InForty\-first International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=GKcwle8XC9)Cited by:[§2](https://arxiv.org/html/2605.21539#S2.p5.1)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn \(2023\)Direct preference optimization: your language model is secretly a reward model\.InThirty\-seventh Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=HPuSIXJaa9)Cited by:[§2](https://arxiv.org/html/2605.21539#S2.p2.1),[§4\.2](https://arxiv.org/html/2605.21539#S4.SS2.p4.1)\.
- H\. Reisizadeh, J\. Jia, Z\. Bu, B\. Vinzamuri, A\. Ramakrishna, K\. Chang, V\. Cevher, S\. Liu, and M\. Hong \(2026\)BLUR: a bi\-level optimization approach for LLM unlearning\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),V\. Demberg, K\. Inui, and L\. Marquez \(Eds\.\),Rabat, Morocco,pp\. 7043–7058\.External Links:[Link](https://aclanthology.org/2026.eacl-long.331/),[Document](https://dx.doi.org/10.18653/v1/2026.eacl-long.331),ISBN 979\-8\-89176\-380\-7Cited by:[§2](https://arxiv.org/html/2605.21539#S2.p5.1)\.
- P\. Röttger, H\. Kirk, B\. Vidgen, G\. Attanasio, F\. Bianchi, and D\. Hovy \(2024\)XSTest: a test suite for identifying exaggerated safety behaviours in large language models\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 5377–5400\.External Links:[Link](https://aclanthology.org/2024.naacl-long.301/),[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.301)Cited by:[§4\.4](https://arxiv.org/html/2605.21539#S4.SS4.p2.1)\.
- R\. Taori, I\. Gulrajani, T\. Zhang, Y\. Dubois, X\. Li, C\. Guestrin, P\. Liang, and T\. B\. Hashimoto \(2023\)Stanford alpaca: an instruction\-following llama model\.GitHub\.Note:[https://github\.com/tatsu\-lab/stanford\_alpaca](https://github.com/tatsu-lab/stanford_alpaca)Cited by:[§4\.4](https://arxiv.org/html/2605.21539#S4.SS4.p1.1),[§4\.4](https://arxiv.org/html/2605.21539#S4.SS4.p2.1)\.
- A\. K\. Tarun, V\. S\. Chundawat, M\. Mandal, and M\. Kankanhalli \(2023\)Fast yet effective machine unlearning\.IEEE Transactions on Neural Networks and Learning Systems\.Cited by:[§2](https://arxiv.org/html/2605.21539#S2.p1.1)\.
- L\. Team \(2024\)Meta llama guard 2\.Note:[https://github\.com/meta\-llama/PurpleLlama/blob/main/Llama\-Guard2/MODEL\_CARD\.md](https://github.com/meta-llama/PurpleLlama/blob/main/Llama-Guard2/MODEL_CARD.md)Cited by:[§4\.4](https://arxiv.org/html/2605.21539#S4.SS4.p2.1)\.
- A\. Thudi, H\. Jia, I\. Shumailov, and N\. Papernot \(2021\)On the necessity of auditable algorithmic definitions for machine unlearning\.InUSENIX Security Symposium,External Links:[Link](https://api.semanticscholar.org/CorpusID:239616091)Cited by:[§2](https://arxiv.org/html/2605.21539#S2.p2.1)\.
- H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale,et al\.\(2023\)Llama 2: open foundation and fine\-tuned chat models\.arXiv preprint arXiv:2307\.09288\.Cited by:[§4\.2](https://arxiv.org/html/2605.21539#S4.SS2.p1.1)\.
- P\. Wang, S\. Lu, H\. Yin, B\. Yang, T\. Zhu, and C\. Dai \(2025\)FedCM: client clustering and migration in federated learning via gradient path similarity and update direction deviation\.InProceedings of the Thirty\-Fourth International Joint Conference on Artificial Intelligence,IJCAI ’25\.External Links:ISBN 978\-1\-956792\-06\-5,[Link](https://doi.org/10.24963/ijcai.2025/706),[Document](https://dx.doi.org/10.24963/ijcai.2025/706)Cited by:[Appendix D](https://arxiv.org/html/2605.21539#A4.p2.1),[§3\.2](https://arxiv.org/html/2605.21539#S3.SS2.p6.1)\.
- X\. Wang, Y\. Zhang, T\. Chen, S\. Gao, S\. Jin, X\. Yang, Z\. Xi, R\. Zheng, Y\. Zou, T\. Gui, Q\. Zhang, and X\. Huang \(2023\)TRACE: a comprehensive benchmark for continual learning in large language models\.External Links:2310\.06762,[Link](https://arxiv.org/abs/2310.06762)Cited by:[§4\.5](https://arxiv.org/html/2605.21539#S4.SS5.p1.2)\.
- Y\. Wang, C\. Y\. Liu, Q\. Liu, J\. Pang, W\. Wei, Y\. Bao, and Y\. Liu \(2026\)DRAGON: guard LLM unlearning in context via negative detection and reasoning\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=vQLUAkl5SG)Cited by:[§2](https://arxiv.org/html/2605.21539#S2.p5.1)\.
- Y\. Yao, X\. Xu, and Y\. Liu \(2024\)Large language model unlearning\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=8Dy42ThoNe)Cited by:[§1](https://arxiv.org/html/2605.21539#S1.p1.1),[§1](https://arxiv.org/html/2605.21539#S1.p2.1),[§2](https://arxiv.org/html/2605.21539#S2.p2.1),[§2](https://arxiv.org/html/2605.21539#S2.p3.1)\.
- X\. Yuan, T\. Pang, C\. Du, K\. Chen, W\. Zhang, and M\. Lin \(2025\)A closer look at machine unlearning for large language models\.InInternational Conference on Learning Representations,Cited by:[Appendix E](https://arxiv.org/html/2605.21539#A5.p2.13),[Appendix E](https://arxiv.org/html/2605.21539#A5.p3.2),[§1](https://arxiv.org/html/2605.21539#S1.p1.1),[§1](https://arxiv.org/html/2605.21539#S1.p2.1),[§2](https://arxiv.org/html/2605.21539#S2.p2.1),[§2](https://arxiv.org/html/2605.21539#S2.p3.1),[Figure 2](https://arxiv.org/html/2605.21539#S3.F2),[§3\.1](https://arxiv.org/html/2605.21539#S3.SS1.p2.1),[§4\.1](https://arxiv.org/html/2605.21539#S4.SS1.p1.1),[§4\.2](https://arxiv.org/html/2605.21539#S4.SS2.p3.1),[§4\.3](https://arxiv.org/html/2605.21539#S4.SS3.p1.1)\.
- R\. Zhang, L\. Lin, Y\. Bai, and S\. Mei \(2024\)Negative preference optimization: from catastrophic collapse to effective unlearning\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=MXLBXjQkmb)Cited by:[§1](https://arxiv.org/html/2605.21539#S1.p1.1),[§1](https://arxiv.org/html/2605.21539#S1.p2.1),[§2](https://arxiv.org/html/2605.21539#S2.p2.1),[§2](https://arxiv.org/html/2605.21539#S2.p3.1),[§2](https://arxiv.org/html/2605.21539#S2.p4.1),[§3\.1](https://arxiv.org/html/2605.21539#S3.SS1.p2.1),[§4\.2](https://arxiv.org/html/2605.21539#S4.SS2.p4.1)\.
- X\. Zhong, H\. Luo, and C\. Liu \(2025\)DualOptim: enhancing efficacy and stability in machine unlearning with dual optimizers\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=77zz0JTNjn)Cited by:[§1](https://arxiv.org/html/2605.21539#S1.p2.1),[§2](https://arxiv.org/html/2605.21539#S2.p4.1),[§3\.1](https://arxiv.org/html/2605.21539#S3.SS1.p2.5),[§4\.2](https://arxiv.org/html/2605.21539#S4.SS2.p2.4)\.
- A\. Zou, L\. Phan, J\. Wang, D\. Duenas, M\. Lin, M\. Andriushchenko, J\. Z\. Kolter, M\. Fredrikson, and D\. Hendrycks \(2024\)Improving alignment and robustness with circuit breakers\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=IbIB8SBKFV)Cited by:[§2](https://arxiv.org/html/2605.21539#S2.p2.1)\.

## Appendix APseudo\-code of DualOptim\+ with Muon

The pseudo\-code of DualOptim\+ integrated in Muon\(Jordanet al\.,[2024](https://arxiv.org/html/2605.21539#bib.bib7)\)is shown in Algorithm[2](https://arxiv.org/html/2605.21539#alg2)\.

Algorithm 2DualOptim\+ with Muon1:Input:parameter

𝜽\{\\bm\{\\theta\}\}, learning rate

η\\eta, momentum factor

β\\beta, forget objective

ℒf\{\\mathcal\{L\}\}\_\{f\}, retain objective

ℒr\{\\mathcal\{L\}\}\_\{r\}, total steps

NN, forget frequency

FfF\_\{f\}, retain frequency

FrF\_\{r\}
2:Initialize:

𝒎Δf←0\{\\bm\{m\}\}\_\{\\Delta\_\{f\}\}\\leftarrow 0,

𝒎Δr←0\{\\bm\{m\}\}\_\{\\Delta\_\{r\}\}\\leftarrow 0,

𝒎B←0\{\\bm\{m\}\}\_\{B\}\\leftarrow 0
3:for

t=1t=1to

NNdo

4:if

tmod\(Ff\+Fr\)≤Fft\\bmod\(F\_\{f\}\+F\_\{r\}\)\\leq F\_\{f\}then

5:

𝒈,𝒎Δ←∇𝜽ℒf​\(𝜽\),𝒎Δf\{\\bm\{g\}\},~\{\\bm\{m\}\}\_\{\\Delta\}\\leftarrow\\nabla\_\{\{\\bm\{\\theta\}\}\}\{\{\\mathcal\{L\}\}\_\{f\}\}\(\{\\bm\{\\theta\}\}\),~\{\\bm\{m\}\}\_\{\\Delta\_\{f\}\}
6:else

7:

𝒈,𝒎Δ←∇𝜽ℒr​\(𝜽\),𝒎Δr\{\\bm\{g\}\},~\{\\bm\{m\}\}\_\{\\Delta\}\\leftarrow\\nabla\_\{\{\\bm\{\\theta\}\}\}\{\{\\mathcal\{L\}\}\_\{r\}\}\(\{\\bm\{\\theta\}\}\),~\{\\bm\{m\}\}\_\{\\Delta\_\{r\}\}
8:endif

9:

𝒎Δ←β​𝒎Δ\+\(𝒈−𝒎B\)\{\\bm\{m\}\}\_\{\\Delta\}\\leftarrow\\beta\{\\bm\{m\}\}\_\{\\Delta\}\+\(\{\\bm\{g\}\}\-\{\\bm\{m\}\}\_\{B\}\)
10:

𝒐←𝙽𝚎𝚠𝚝𝚘𝚗𝚂𝚌𝚑𝚞𝚕𝚣𝟻​\(𝒎B\+𝒎Δ\)\{\\bm\{o\}\}\\leftarrow\\mathtt\{NewtonSchulz5\}\(\{\\bm\{m\}\}\_\{B\}\+\{\\bm\{m\}\}\_\{\\Delta\}\)
11:

𝜽=𝜽−η​𝒐\{\\bm\{\\theta\}\}=\{\\bm\{\\theta\}\}\-\\eta\{\\bm\{o\}\}
12:

𝒎B←β​𝒎B\+𝒈\{\\bm\{m\}\}\_\{B\}\\leftarrow\\beta\{\\bm\{m\}\}\_\{B\}\+\{\\bm\{g\}\}
13:endfor

14:Output:parameter

𝜽\{\\bm\{\\theta\}\}

## Appendix BProofs

### B\.1Proof of Theorem[3\.2](https://arxiv.org/html/2605.21539#S3.Thmtheorem2)

###### Proof\.

We prove the convergence ofBtB\_\{t\},Δf\\Delta\_\{f\}andΔr\\Delta\_\{r\}one by one\.

1\. Convergence ofBtB\_\{t\}

Based on the update rule \([5](https://arxiv.org/html/2605.21539#S3.E5)\), we have the following equation:

B\(Ff\+Fr\)​T=βFf\+Fr​B\(Ff\+Fr\)​\(T−1\)\+\(1−β\)​\(∑t=1FfβFf\+Fr−t​gf,\(Ff\+Fr\)​\(T−1\)\+t\+∑t=1FrβFr−t​gr,\(Ff\+Fr\)​\(T−1\)\+Ff\+t\)B\_\{\(F\_\{f\}\+F\_\{r\}\)T\}=\\beta^\{F\_\{f\}\+F\_\{r\}\}B\_\{\(F\_\{f\}\+F\_\{r\}\)\(T\-1\)\}\+\(1\-\\beta\)\\left\(\\sum\_\{t=1\}^\{F\_\{f\}\}\\beta^\{F\_\{f\}\+F\_\{r\}\-t\}g\_\{f,\(F\_\{f\}\+F\_\{r\}\)\(T\-1\)\+t\}\+\\sum\_\{t=1\}^\{F\_\{r\}\}\\beta^\{F\_\{r\}\-t\}g\_\{r,\(F\_\{f\}\+F\_\{r\}\)\(T\-1\)\+F\_\{f\}\+t\}\\right\)\(8\)
Based on Assumption[3\.1](https://arxiv.org/html/2605.21539#S3.Thmtheorem1), we can conclude thatlimT→∞B\(Ff\+Fr\)​T\\lim\_\{T\\to\\infty\}B\_\{\(F\_\{f\}\+F\_\{r\}\)T\}exists, so we letXB=limT→∞B\(Ff\+Fr\)​TX\_\{B\}=\\lim\_\{T\\to\\infty\}B\_\{\(F\_\{f\}\+F\_\{r\}\)T\}\. We consider the equation above, take the expectation overtt, whenT→∞T\\to\\infty, we have:

XB=βFf\+Fr​XB\+\(1−β\)​\(∑t=1FfβFf\+Fr−t⋅m​G\+∑t=1FrβFr−t⋅n​G\)X\_\{B\}=\\beta^\{F\_\{f\}\+F\_\{r\}\}X\_\{B\}\+\(1\-\\beta\)\\left\(\\sum\_\{t=1\}^\{F\_\{f\}\}\\beta^\{F\_\{f\}\+F\_\{r\}\-t\}\\cdot mG\+\\sum\_\{t=1\}^\{F\_\{r\}\}\\beta^\{F\_\{r\}\-t\}\\cdot nG\\right\)\(9\)
Based on the equation above, we havelimT→∞B\(Ff\+Fr\)​T=XB=βFr​\(1−βFf\)​m\+\(1−βFr\)​n1−βFf\+Fr​G\\lim\_\{T\\to\\infty\}B\_\{\(F\_\{f\}\+F\_\{r\}\)T\}=X\_\{B\}=\\frac\{\\beta^\{F\_\{r\}\}\(1\-\\beta^\{F\_\{f\}\}\)m\+\(1\-\\beta^\{F\_\{r\}\}\)n\}\{1\-\\beta^\{F\_\{f\}\+F\_\{r\}\}\}G\.

2\. Convergence ofΔf\\Delta\_\{f\}

Whent∈\(\(Ff\+Fr\)​\(T−1\)\+Ff,\(Ff\+Fr\)​T\]t\\in\(\(F\_\{f\}\+F\_\{r\}\)\(T\-1\)\+F\_\{f\},\(F\_\{f\}\+F\_\{r\}\)T\],Δf\\Delta\_\{f\}is not updated, soΔf,\(Ff\+Fr\)​T=Δf,\(Ff\+Fr\)​\(T−1\)\+Ff\\Delta\_\{f,\(F\_\{f\}\+F\_\{r\}\)T\}=\\Delta\_\{f,\(F\_\{f\}\+F\_\{r\}\)\(T\-1\)\+F\_\{f\}\}\. Based on the update rule \([6](https://arxiv.org/html/2605.21539#S3.E6)\), we have the following equation:

Δf,\(Ff\+Fr\)​T=βFf​Δf,\(Ff\+Fr\)​\(T−1\)\+\(1−β\)⋅∑t=1FfβFf−t​\(gf,\(Ff\+Fr\)​\(T−1\)\+t−B^\(Ff\+Fr\)​\(T−1\)\+t−1\)\\Delta\_\{f,\(F\_\{f\}\+F\_\{r\}\)T\}=\\beta^\{F\_\{f\}\}\\Delta\_\{f,\(F\_\{f\}\+F\_\{r\}\)\(T\-1\)\}\+\(1\-\\beta\)\\cdot\\sum\_\{t=1\}^\{F\_\{f\}\}\\beta^\{F\_\{f\}\-t\}\\left\(g\_\{f,\(F\_\{f\}\+F\_\{r\}\)\(T\-1\)\+t\}\-\\widehat\{B\}\_\{\(F\_\{f\}\+F\_\{r\}\)\(T\-1\)\+t\-1\}\\right\)\(10\)
WhenT→∞T\\to\\infty, and for1≤t≤Ff1\\leq t\\leq F\_\{f\}, according to \([9](https://arxiv.org/html/2605.21539#A2.E9)\), we have:

limT→∞B\(Ff\+Fr\)​\(T−1\)\+t−1=βt−1​XB\+\(1−β\)​∑k=1t−1βt−1−k⋅m​G=βt−1​XB\+\(1−βt−1\)​m​G\\lim\_\{T\\to\\infty\}B\_\{\(F\_\{f\}\+F\_\{r\}\)\(T\-1\)\+t\-1\}=\\beta^\{t\-1\}X\_\{B\}\+\(1\-\\beta\)\\sum\_\{k=1\}^\{t\-1\}\\beta^\{t\-1\-k\}\\cdot mG=\\beta^\{t\-1\}X\_\{B\}\+\(1\-\\beta^\{t\-1\}\)mG\(11\)
Whent→∞t\\to\\infty,B^t=Bt/\(1−βt\)→Bt\\widehat\{B\}\_\{t\}=B\_\{t\}/\(1\-\\beta^\{t\}\)\\to B\_\{t\}\. LetXΔf=limT→∞Δf,\(Ff\+Fr\)​TX\_\{\\Delta\_\{f\}\}=\\lim\_\{T\\to\\infty\}\\Delta\_\{f,\(F\_\{f\}\+F\_\{r\}\)T\}, based on \([10](https://arxiv.org/html/2605.21539#A2.E10)\) and \([11](https://arxiv.org/html/2605.21539#A2.E11)\), we have:

XΔf\\displaystyle X\_\{\\Delta\_\{f\}\}=βFf​XΔf\+\(1−β\)⋅∑t=1FfβFf−t​\[m​G−βt−1​XB−\(1−βt−1\)​m​G\]\\displaystyle=\\beta^\{F\_\{f\}\}X\_\{\\Delta\_\{f\}\}\+\(1\-\\beta\)\\cdot\\sum\_\{t=1\}^\{F\_\{f\}\}\\beta^\{F\_\{f\}\-t\}\\left\[mG\-\\beta^\{t\-1\}X\_\{B\}\-\(1\-\\beta^\{t\-1\}\)mG\\right\]\(12\)=βFf​XΔf\+\(1−β\)​Ff​βFf−1​\(m​G−XB\)\\displaystyle=\\beta^\{F\_\{f\}\}X\_\{\\Delta\_\{f\}\}\+\(1\-\\beta\)F\_\{f\}\\beta^\{F\_\{f\}\-1\}\\left\(mG\-X\_\{B\}\\right\)Based on the equation above, we havelimT→∞Δf,\(Ff\+Fr\)​T=XΔf=Ff​βFf−1​\(1−β\)​\(1−βFr\)​\(m−n\)\(1−βFf\)​\(1−βFf\+Fr\)​G\\lim\_\{T\\to\\infty\}\\Delta\_\{f,\(F\_\{f\}\+F\_\{r\}\)T\}=X\_\{\\Delta\_\{f\}\}=\\frac\{F\_\{f\}\\beta^\{F\_\{f\}\-1\}\(1\-\\beta\)\\left\(1\-\\beta^\{F\_\{r\}\}\\right\)\(m\-n\)\}\{\\left\(1\-\\beta^\{F\_\{f\}\}\\right\)\\left\(1\-\\beta^\{F\_\{f\}\+F\_\{r\}\}\\right\)\}G

3\. Convergence ofΔr\\Delta\_\{r\}

Similarly to \([10](https://arxiv.org/html/2605.21539#A2.E10)\), we have:

Δr,\(Ff\+Fr\)​T=βFr​Δr,\(Ff\+Fr\)​\(T−1\)\+\(1−β\)⋅∑t=1FrβFr−t​\(gr,\(Ff\+Fr\)​\(T−1\)\+Ff\+t−B^\(Ff\+Fr\)​\(T−1\)\+Ff\+t−1\)\\Delta\_\{r,\(F\_\{f\}\+F\_\{r\}\)T\}=\\beta^\{F\_\{r\}\}\\Delta\_\{r,\(F\_\{f\}\+F\_\{r\}\)\(T\-1\)\}\+\(1\-\\beta\)\\cdot\\sum\_\{t=1\}^\{F\_\{r\}\}\\beta^\{F\_\{r\}\-t\}\\left\(g\_\{r,\(F\_\{f\}\+F\_\{r\}\)\(T\-1\)\+F\_\{f\}\+t\}\-\\widehat\{B\}\_\{\(F\_\{f\}\+F\_\{r\}\)\(T\-1\)\+F\_\{f\}\+t\-1\}\\right\)\(13\)
WhenT→∞T\\to\\infty, and for1≤t≤Fr1\\leq t\\leq F\_\{r\}, according to \([9](https://arxiv.org/html/2605.21539#A2.E9)\), we have:

limT→∞B\(Ff\+Fr\)​\(T−1\)\+Ff\+t−1\\displaystyle\\lim\_\{T\\to\\infty\}B\_\{\(F\_\{f\}\+F\_\{r\}\)\(T\-1\)\+F\_\{f\}\+t\-1\}=βFf\+t−1​XB\+\(1−β\)​\(∑k=1FfβFf\+t−1−k⋅m​G\+∑k=1t−1βt−1−k⋅n​G\)\\displaystyle=\\beta^\{F\_\{f\}\+t\-1\}X\_\{B\}\+\(1\-\\beta\)\\left\(\\sum\_\{k=1\}^\{F\_\{f\}\}\\beta^\{F\_\{f\}\+t\-1\-k\}\\cdot mG\+\\sum\_\{k=1\}^\{t\-1\}\\beta^\{t\-1\-k\}\\cdot nG\\right\)\(14\)=βFf\+t−1​XB\+βt−1​\(1−βFf\)​m​G\+\(1−βt−1\)​n​G\\displaystyle=\\beta^\{F\_\{f\}\+t\-1\}X\_\{B\}\+\\beta^\{t\-1\}\(1\-\\beta^\{F\_\{f\}\}\)mG\+\(1\-\\beta^\{t\-1\}\)nG
Whent→∞t\\to\\infty,B^t=Bt/\(1−βt\)→Bt\\widehat\{B\}\_\{t\}=B\_\{t\}/\(1\-\\beta^\{t\}\)\\to B\_\{t\}\. LetXΔr=limT→∞Δr,\(Ff\+Fr\)​TX\_\{\\Delta\_\{r\}\}=\\lim\_\{T\\to\\infty\}\\Delta\_\{r,\(F\_\{f\}\+F\_\{r\}\)T\}, based on \([13](https://arxiv.org/html/2605.21539#A2.E13)\) and \([14](https://arxiv.org/html/2605.21539#A2.E14)\), we have:

XΔr\\displaystyle X\_\{\\Delta\_\{r\}\}=βFr​XΔr\+\(1−β\)⋅∑t=1FrβFr−t​\[n​G−βFf\+t−1​XB−βt−1​\(1−βFf\)​m​G−\(1−βt−1\)​n​G\]\\displaystyle=\\beta^\{F\_\{r\}\}X\_\{\\Delta\_\{r\}\}\+\(1\-\\beta\)\\cdot\\sum\_\{t=1\}^\{F\_\{r\}\}\\beta^\{F\_\{r\}\-t\}\\left\[nG\-\\beta^\{F\_\{f\}\+t\-1\}X\_\{B\}\-\\beta^\{t\-1\}\(1\-\\beta^\{F\_\{f\}\}\)mG\-\(1\-\\beta^\{t\-1\}\)nG\\right\]\(15\)=βFr​XΔr\+\(1−β\)​Fr​βFr−1​\[\(βFf−1\)​m​G\+n​G−βFf​XB\]\\displaystyle=\\beta^\{F\_\{r\}\}X\_\{\\Delta\_\{r\}\}\+\(1\-\\beta\)F\_\{r\}\\beta^\{F\_\{r\}\-1\}\\left\[\(\\beta^\{F\_\{f\}\}\-1\)mG\+nG\-\\beta^\{F\_\{f\}\}X\_\{B\}\\right\]Based on the equation above, we havelimT→∞Δr,\(Ff\+Fr\)​T=XΔr=Fr​βFr−1​\(1−β\)​\(1−βFf\)​\(n−m\)\(1−βFr\)​\(1−βFf\+Fr\)​G\\lim\_\{T\\to\\infty\}\\Delta\_\{r,\(F\_\{f\}\+F\_\{r\}\)T\}=X\_\{\\Delta\_\{r\}\}=\\frac\{F\_\{r\}\\beta^\{F\_\{r\}\-1\}\(1\-\\beta\)\\left\(1\-\\beta^\{F\_\{f\}\}\\right\)\(n\-m\)\}\{\\left\(1\-\\beta^\{F\_\{r\}\}\\right\)\\left\(1\-\\beta^\{F\_\{f\}\+F\_\{r\}\}\\right\)\}G∎

## Appendix CMore Experiment Results

### C\.1Comparison of Running Time and Memory Consumption

Table 11:Memory consumption and running time of different methods\. The model is Phi1\.5\-1\.3B and the optimizer is AdamW\. The total steps of unlearning is 300, the batch size per GPU is44\. All experiments are implemented on two NVIDIA H20 GPUs\.MethodJointAlternateDODO 8bitDO\+DO\+ 8bitMemory \(GB/GPU\)33\.2633\.2636\.1533\.5739\.0433\.68Running Time \(s\)870443439464475468Table[11](https://arxiv.org/html/2605.21539#A3.T11)provides an overview of the memory consumption and training efficiency for the various methods evaluated\. While DO and DO\+ introduce additional memory overhead due to their extra optimizer states, peaking at 39\.04 GB/GPU for DO\+, their 8\-bit implementations, DO 8bit and DO\+ 8bit, successfully reduce this footprint to 33\.57 GB/GPU and 33\.68 GB/GPU, respectively\. This demonstrates that the 8\-bit variants can achieve near\-parity in memory usage with the standard Joint and Alternate baselines, with the slight overhead resulting from maintaining full\-precision embedding layers and storing quantization coefficients\. Notably, the methods using alternating update scheme offer a nearly2×2\\timesspeedup over the Joint method\. This efficiency gain occurs because the Joint baseline doubles the equivalent batch size by simultaneously processing both forget and retain data in each step\.

### C\.2Results on TOFU with DPO\+GD and NPO\+GD loss functions

As shown in Table[12](https://arxiv.org/html/2605.21539#A3.T12), DualOptim\+ achieves the best overall performance in most cases, consistent with the observations in Table[1](https://arxiv.org/html/2605.21539#S4.T1)\. However, compared to IDK\+GD and ME\+GD, the DPO\+GD and NPO\+GD configurations fail to achieve either stable forgetting or enhanced model utility\. This performance discrepancy can be attributed to the conservative weighting strategy inherent in these methods, which relies heavily on the reference model and limits the optimizer’s ability to deviate significantly from the initial state\.

Table 12:Performance comparison of different methods on TOFU\-finetunedPhi 1\.5andLlama 2\.DPO\+GD\(targeted\) andNPO\+GD\(untargeted\) are adopted as the loss functions for unlearning\. The results include Untargeted Forget Efficacy \(UFE\), Targeted Forget Efficacy \(TFE\), Model Utility \(MU\) and the Overall Performance \(OVR\) for forget 1%, 5% data, and 10% data\. Note that the reported results are the average of the results obtained from55runs with different forget sets\.LossMethodPhi 1\.5forget 1% dataforget 5% dataforget 10% dataUFE↑\\uparrowTFE↑\\uparrowMU↑\\uparrowOVR↑\\uparrowUFE↑\\uparrowTFE↑\\uparrowMU↑\\uparrowOVR↑\\uparrowUFE↑\\uparrowTFE↑\\uparrowMU↑\\uparrowOVR↑\\uparrowDPO\+GDJoint79\.4323\.7432\.0841\.8377\.2539\.3233\.5445\.9177\.6745\.4631\.4846\.52Alternate78\.5635\.1148\.2352\.5374\.3447\.2649\.6855\.2474\.4957\.5149\.8757\.94DO81\.5855\.3646\.3257\.3974\.8647\.1050\.9455\.9672\.9752\.6150\.2856\.53DO 8bit83\.7757\.5647\.7359\.2075\.2041\.1251\.1254\.6472\.7640\.9550\.0253\.44DO\+82\.9261\.9048\.0560\.2376\.0455\.5651\.3358\.5774\.6756\.4650\.7858\.18DO\+ 8bit83\.4657\.4447\.6159\.0375\.9650\.1351\.4057\.2373\.1446\.2150\.2454\.96NPO\+GDJoint78\.87–29\.8954\.3874\.68–31\.7153\.2073\.73–27\.7850\.76Alternate74\.62–48\.3261\.4770\.46–50\.1760\.3267\.81–49\.1558\.48DO74\.88–48\.4761\.6870\.60–49\.8960\.2567\.33–48\.7358\.03DO 8bit75\.51–48\.7762\.1470\.17–50\.2760\.2366\.88–50\.0758\.48DO\+75\.80–48\.7262\.2671\.63–51\.5261\.5869\.12–49\.7159\.42DO\+ 8bit76\.10–48\.7962\.4570\.75–50\.6360\.6967\.11–51\.2559\.18LossMethodLlama 2forget 1% dataforget 5% dataforget 10% dataUFE↑\\uparrowTFE↑\\uparrowMU↑\\uparrowOVR↑\\uparrowUFE↑\\uparrowTFE↑\\uparrowMU↑\\uparrowOVR↑\\uparrowUFE↑\\uparrowTFE↑\\uparrowMU↑\\uparrowOVR↑\\uparrowDPO\+GDJoint92\.0245\.7444\.1956\.5488\.1145\.1766\.9866\.8185\.1449\.9465\.0566\.30Alternate88\.2753\.7773\.6772\.3583\.1346\.4173\.4669\.1180\.9136\.5974\.0366\.39DO89\.2648\.2472\.3370\.5483\.6745\.3171\.4267\.9680\.7738\.5572\.4366\.04DO 8bit89\.4746\.5472\.3270\.1684\.4948\.1973\.0769\.7182\.1840\.4973\.3167\.32DO\+79\.4945\.9677\.5170\.1277\.9941\.8676\.5168\.2278\.4347\.5577\.1970\.09DO\+ 8bit82\.0443\.5777\.0169\.9181\.9447\.8375\.0669\.9880\.1545\.7175\.1069\.17NPO\+GDJoint77\.31–55\.5666\.4469\.67–63\.1266\.4067\.90–64\.4666\.18Alternate73\.51–74\.4073\.9675\.32–75\.6975\.5172\.77–75\.9074\.34DO73\.83–74\.5174\.1774\.31–74\.5274\.4271\.78–73\.9972\.89DO 8bit73\.85–74\.2274\.0475\.35–75\.1175\.2472\.42–75\.7574\.09DO\+78\.51–75\.0776\.8075\.94–75\.3775\.6571\.95–73\.9572\.95DO\+ 8bit76\.58–74\.9175\.7575\.42–76\.0175\.7273\.79–76\.0374\.92
### C\.3Results on TOFU with LoRA

To evaluate the effectiveness of our method within a parameter\-efficient fine\-tuning framework, we apply LoRA\(Huet al\.,[2022](https://arxiv.org/html/2605.21539#bib.bib15)\)with a rank of 8 to Llama 2\. As shown in Table[13](https://arxiv.org/html/2605.21539#A3.T13), the performance gains from both DualOptim and DualOptim\+ are less pronounced than in full\-parameter unlearning, but DualOptim\+ and its 8bit variant exhibit the best performance in most cases\. Furthermore, the results indicate that the performance gap between LoRA and full\-parameter tuning widens as the volume of data to be forgotten increases\. This is because the limited degrees of freedom in a low\-rank subspace are insufficient to encode the complex modifications required for larger\-scale unlearning tasks \. Conversely, LoRA significantly mitigates the excessive forgetting on the retain set that is frequently observed when using the Joint updating scheme\.

Table 13:Performance comparison of different methods on TOFU\-finetunedLlama 2withLoRA\(rank=8\\mathrm\{rank=8\}\)\.IDK\+GD\(targeted\) andME\+GD\(untargeted\) are adopted as the loss functions for unlearning\. The results include Untargeted Forget Efficacy \(UFE\), Targeted Forget Efficacy \(TFE\), Model Utility \(MU\) and the Overall Performance \(OVR\) for forget 1%, 5% data, and 10% data\. Note that the reported results are the average of the results obtained from55runs with different forget sets\.LossMethodLlama 2 \+ LoRAforget 1% dataforget 5% dataforget 10% dataUFE↑\\uparrowTFE↑\\uparrowMU↑\\uparrowOVR↑\\uparrowUFE↑\\uparrowTFE↑\\uparrowMU↑\\uparrowOVR↑\\uparrowUFE↑\\uparrowTFE↑\\uparrowMU↑\\uparrowOVR↑\\uparrowIDK\+GDJoint73\.7059\.5073\.0769\.8470\.1068\.6068\.8269\.0864\.7267\.5464\.9265\.52Alternate71\.6966\.4074\.5771\.8164\.5866\.6370\.5168\.0659\.1156\.4166\.1461\.95DO72\.0166\.4074\.4171\.8168\.6569\.7670\.5669\.8864\.1762\.9064\.4264\.42DO 8bit72\.0366\.4074\.5171\.8668\.7769\.7770\.6369\.9564\.0863\.1364\.6264\.11DO\+72\.0966\.4173\.8771\.5668\.7770\.1073\.0571\.2461\.2262\.7669\.8865\.94DO\+ 8bit71\.7666\.4173\.9171\.5068\.8670\.1973\.1171\.3261\.2662\.5769\.9765\.95ME\+GDJoint96\.14–73\.9285\.0395\.01–75\.0485\.0293\.27–75\.7984\.53Alternate95\.78–75\.9785\.8892\.69–75\.5684\.1388\.79–74\.6281\.71DO95\.87–76\.0085\.9494\.97–76\.0085\.4993\.63–75\.5784\.60DO 8bit95\.91–76\.0986\.0095\.11–76\.1385\.6293\.84–75\.6884\.76DO\+95\.61–76\.4686\.0494\.89–76\.2885\.5993\.73–76\.0484\.89DO\+ 8bit95\.04–76\.4085\.7394\.90–76\.2285\.5693\.70–76\.0584\.88
### C\.4Unlearning Metrics and Losses over Time Steps

As depicted in Figure[3](https://arxiv.org/html/2605.21539#A3.F3)\(a\) and \(b\), DualOptim\+ exhibits the most effective and stable performance among the evaluated methods\. While the Joint updating scheme achieves similar forget efficacy, its Model Utility is substantially lower than other methods\. As observed in Figure Figure[3](https://arxiv.org/html/2605.21539#A3.F3)\(c\) and \(d\), both forgetting and retaining losses converge rapidly under the Joint updating scheme, suggesting the model becomes trapped in a suboptimal local minimum\. In contrast, methods employing the alternate updating scheme, i\.e\., Alternate, DualOptim, and DualOptim\+, converge more gradually, enhancing model’s exploration capability to find superior local minima\.

![Refer to caption](https://arxiv.org/html/2605.21539v1/x7.png)\(a\)Forget Efficacy
![Refer to caption](https://arxiv.org/html/2605.21539v1/x8.png)\(b\)Model Utility
![Refer to caption](https://arxiv.org/html/2605.21539v1/x9.png)\(c\)Forgetting Loss
![Refer to caption](https://arxiv.org/html/2605.21539v1/x10.png)\(d\)Retaining Loss

Figure 3:Comparison of unlearning metrics and losses over time steps\. We plot the\(a\)forget efficacy, i\.e\., the average of targeted forget efficacy and untargeted forget efficacy,\(b\)model utility,\(c\)forgetting loss, and\(d\)retaining loss\. Note that the results are collected when forgetting5%5\\%data of TOFU using IDK\+GD loss function\. The model is TOFU\-finetuned Phi 1\.5\.

## Appendix DComparison with Federated Learning Methods

While federated learning\(Konečnýet al\.,[2015](https://arxiv.org/html/2605.21539#bib.bib52)\)focuses on overcoming client drift caused by non\-IID data across a shared objective, machine unlearning must navigate adversarial objectives: the mandate to forget specific information while simultaneously retaining general knowledge\. These loss functions are often diametrically opposed, leading to catastrophic forgetting, which is a phenomenon that is not a central concern in standard FL but is the primary bottleneck in unlearning\. Our work specifically addresses this tension through a novel optimizer design\.

Furthermore, we compare the proposed DO\+ with some specific FL methods, i\.e\., SCAFFOLD\(Karimireddyet al\.,[2020](https://arxiv.org/html/2605.21539#bib.bib48)\), MIME\(Karimireddyet al\.,[2021](https://arxiv.org/html/2605.21539#bib.bib50)\), FedCM\(Wanget al\.,[2025](https://arxiv.org/html/2605.21539#bib.bib49)\), and Local Adam\(Cheng and Glasgow,[2025](https://arxiv.org/html/2605.21539#bib.bib51)\)\. While FL involves optimizing multiple local models, unlearning tasks require balancing multiple objectives\. Therefore, we have to adapt the core design of these FL methods mentioned to the unlearning context to ensure compatibility with our optimization framework\. In the following, we list their implementations and highlight the differences between these adapted methods and DO\+ regarding base and delta state updates\.

- •SCAFFOLD, MIME:These two methods share the similar core design\. In the context of unlearning tasks, their updating rule of delta stateΔ\\Deltawill beΔ←β​Δ\+\(1−β\)​\(g−B\)\\Delta\\leftarrow\\beta\\Delta\+\(1\-\\beta\)\(g\-B\), which is the same as DO\+\. They update the base stateBBonce only after a full forget\-retain period \(i\.e\.,Ff\+FrF\_\{f\}\+F\_\{r\}batches of data\) by the ruleB←B\+12​\(Δ^f\+Δ^r\)B\\leftarrow B\+\\frac\{1\}\{2\}\(\\widehat\{\\Delta\}\_\{f\}\+\\widehat\{\\Delta\}\_\{r\}\)\.
- •FedCM:The updating rule of delta state isΔ←β​Δ\+\(1−β\)​g\\Delta\\leftarrow\\beta\\Delta\+\(1\-\\beta\)g\. Same as SCAFFOLD and MIME, the base state is updated once only after a full forget\-retain period byB←B\+12​\(Δ^f\+Δ^r\)B\\leftarrow B\+\\frac\{1\}\{2\}\(\\widehat\{\\Delta\}\_\{f\}\+\\widehat\{\\Delta\}\_\{r\}\)\.
- •Local Adam:It only maintains two states for forgetting and retaining objectives, which is similar to DO\. The difference is that the forget stateSfS\_\{f\}and retain stateSrS\_\{r\}will be merged asS=12​\(Sf\+Sr\)S=\\frac\{1\}\{2\}\(S\_\{f\}\+S\_\{r\}\)after a full forget\-retain period\.
- •DO\+:The updating rule of delta state isΔ←β​Δ\+\(1−β\)​\(g−B^\)\\Delta\\leftarrow\\beta\\Delta\+\(1\-\\beta\)\(g\-\\widehat\{B\}\)\. The base state is updated after each data batch byB←β​B\+\(1−β\)​gB\\leftarrow\\beta B\+\(1\-\\beta\)g\.

Additionally, we compare these methods with DO\+ under the setting of forgetting 5% data of TOFU, IDK\+GD loss, Phi\-1\.5\. As shown in the Table[14](https://arxiv.org/html/2605.21539#A4.T14), our method achieves the best performance, further underscoring the effectiveness of DO\+ in LLM unlearning\. To some extent, it can be attributed to the difference that DO\+ updates the base state more frequently\.

Table 14:Unlearning performance of DO\+ and different federated learning methods\. The experiment setting is forgetting 5% data of TOFU, IDK\+GD loss, Phi\-1\.5\.UFETFEMUAVGSCAFFOLD67\.2466\.8950\.4458\.76FedCM70\.3565\.3249\.3458\.59Local Adam70\.4365\.9049\.6858\.93DO\+67\.6367\.6051\.5259\.57
## Appendix EImplementation Details

All experiments are conducted on NVIDIA H20 GPUs, and we utilize PyTorch FSDP to reduce memory costs\. We employ DualOptim\+ and baselines based on AdamW optimizer with a weight decay of0\.010\.01, betas of \(0\.90\.9,0\.950\.95\)\. A linear warm\-up learning rate in the first epoch and a linearly decaying learning rate in the subsequent epochs are used\.

Unlearning in Fictitious Scenario\.For the experments on the TOFU dataset, we use fine\-tuned Phi1\.5\-1\.3B and Llama2\-chat\-7B models released by the original paper\(Mainiet al\.,[2024](https://arxiv.org/html/2605.21539#bib.bib11)\)as the target models\. For Phi 1\.5, we use two NVIDIA H20 GPUs, and the effective batch size is set4040\. For Llama 2, we use eight NVIDIA H20 GPUs, and the effective batch size is set128128\. The initial learning rate is set to1×10−51\\times 10^\{\-5\}\. The total unlearn stepsNNis300300\. The forget frequencyFfF\_\{f\}is11\. The retain frequencyFrF\_\{r\}is55\. Following the setup inYuanet al\.\([2025](https://arxiv.org/html/2605.21539#bib.bib5)\), the parameterα\\alphain ME\+GD is set to0\.10\.1\. Theβ\\betain DPO and NPO is set to0\.10\.1\. The evaluation metrics are calculated using the codes in the official repository111[https://github\.com/sail\-sg/closer\-look\-LLM\-unlearning](https://github.com/sail-sg/closer-look-LLM-unlearning)\.

Unlearning in Real\-world Scenario\.FollowLiuet al\.\([2024](https://arxiv.org/html/2605.21539#bib.bib16)\); Yuanet al\.\([2025](https://arxiv.org/html/2605.21539#bib.bib5)\), we use Llama\-3\-8B\-Instruct as the target model\. We use eight NVIDIA H20 GPUs, and the effective batch size is set128128\. The initial learning rate is set to5×10−65\\times 10^\{\-6\}\. We use the repository222[https://github\.com/EleutherAI/lm\-evaluation\-harness](https://github.com/EleutherAI/lm-evaluation-harness)of lm\-evaluation\-harness\(Gaoet al\.,[2024](https://arxiv.org/html/2605.21539#bib.bib37)\)to evaluate downstream tasks with default configurations\. Other configurations are consistent with TOFU dataset\.

Safety Alignment\.We use Llama\-3\-8B\-Instruct fine\-tuend on Alpaca333[https://huggingface\.co/PKU\-Alignment/alpaca\-8b\-reproduced\-llama\-3](https://huggingface.co/PKU-Alignment/alpaca-8b-reproduced-llama-3)as the target model\. We use eight NVIDIA H20 GPUs, and the effective batch size is set128128\. The initial learning rate is set to1×10−51\\times 10^\{\-5\}\. The total epochs are33\. Following the settings inBianchiet al\.\([2024](https://arxiv.org/html/2605.21539#bib.bib12)\), forJointmethod, we calculate cross\-entropy loss on batches randomly sampled from a mixture of 20,000 Alpaca instructions and 2,000 safety instructions\. Formethods using alternating update scheme, the forget frequencyFfF\_\{f\}and retain frequencyFrF\_\{r\}are set to11and1010, respectively, to simulate the ratio of safety instructions to general instructions, i\.e\.,1:101:10\.

## Appendix FDiscussion on Evaluation Metrics

As shown in Table[15](https://arxiv.org/html/2605.21539#A6.T15), the first two examples achieve high Untargeted Forget Efficacy \(UFE\) despite consisting of nonsensical strings that fail to provide a coherent rejection\. Furthermore, coherent rejection responses \(the last two examples\) typically yield lower UFE scores than nonsensical outputs\. These observations suggest that UFE alone is an insufficient metric for evaluating targeted unlearning\. In contrast, Targeted Forget Efficacy \(TFE\) effectively distinguishes between appropriate rejections and nonsense\. Consequently, for targeted unlearning tasks, we employ the mean of UFE and TFE as the comprehensive forget efficacy metric\.

Table 15:The outputs of unlearned Llama2 on the forget set of TOFU and the corresponding Untargeted Forget Efficacy \(UFE\) and Targeted Forget Efficacy \(TFE\)\. The first two examples are generated by the model unlearned using ME\+GD loss function\. The last two examples are generated using IDK\+GD loss function\.Generated TextUFETFEfullL fullL fullL fullL fullL…94\.870\.00”\. born born”\. born born within…93\.980\.00That’s a blind spot in my knowledge\.69\.84100\.00I must admit, I don’t know\.69\.3996\.90