MLUBench: A Benchmark for Lifelong Unlearning Evaluation in MLLMs

# MLUBench: A Benchmark for Lifelong Unlearning Evaluation in MLLMs
Source: [https://arxiv.org/html/2606.12809](https://arxiv.org/html/2606.12809)
Haoang ChiQizhou WangYunxin MaoZhiheng ZhangJie TanTongliang LiuWenjing YangBo Han

###### Abstract

Multimodal large language models \(MLLMs\) are trained on massive multimodal data, making data unlearning increasingly important as data owners may request the removal of specific content\. In practice, these requests often arrive sequentially over time, giving rise to the challenging problem ofMLLM Lifelong Unlearning\. However, most existing benchmarks are limited in scale and scope, failing to capture the complexities of MLLM lifelong unlearning\. To fill this gap, we introduce the MLUBench, a large\-scale and comprehensive benchmark featuring 127 entities across 9 classes under lifelong unlearning requests\. We perform extensive experiments using MLUBench and reveal that existing unlearning methods suffer from severe, cumulative degradation\. More critically, we further identify the unique challenge of this problem: unlike in unimodal models, MLLM lifelong unlearning is constrained by the need to preserve multimodal alignment\. Continually unlearning from one modality could degrade the entire model\. To alleviate this challenge, we propose LUMoE, an effective method\. Experiments demonstrate that LUMoE significantly mitigates the degradation problem faced by baselines\. The source code and the MLUBench dataset are open\-sourced in this[URL](https://github.com/lihe-maxsize/Lifelong_Unlearning_main)\.

Machine Learning, ICML

## 1Introduction

Multimodal large language models \(MLLMs\), such as Gemini\(Teamet al\.,[2023](https://arxiv.org/html/2606.12809#bib.bib18)\)and GPT\-4o\(Hurstet al\.,[2024](https://arxiv.org/html/2606.12809#bib.bib17)\), have demonstrated remarkable multimodal reasoning abilities across a wide range of applications\(Yinet al\.,[2024](https://arxiv.org/html/2606.12809#bib.bib13); Liuet al\.,[2024e](https://arxiv.org/html/2606.12809#bib.bib37); Liet al\.,[2024b](https://arxiv.org/html/2606.12809#bib.bib35); Wuet al\.,[2023](https://arxiv.org/html/2606.12809#bib.bib14); Zhanget al\.,[2024a](https://arxiv.org/html/2606.12809#bib.bib15)\)\. These models are typically trained on web\-scale multimodal data, which inevitably raises concerns regarding data privacy and copyright\(Zhaoet al\.,[2025](https://arxiv.org/html/2606.12809#bib.bib19); Shiet al\.,[2024b](https://arxiv.org/html/2606.12809#bib.bib20)\)\. Therefore, machine unlearning, which targets the removal of specific data from a trained model, has become critically important\. In real\-world scenarios, removal requests may not all arrive at once; instead, they may be submitted sequentially over time\. This practical setting gives rise to the challenging problem we study:MLLM Lifelong Unlearning, where an MLLM must continuously forget multimodal information while preserving its general capabilities \(Figure[1](https://arxiv.org/html/2606.12809#S1.F1)\)\.

![Refer to caption](https://arxiv.org/html/2606.12809v1/x1.png)Figure 1:Illustration of the challenges of MLLM lifelong unlearning\. \(a\) MLLM undergoes sequential unlearning tasks, where it must continually forget specified knowledge while retaining other information\. \(b\) Output degradation of the LLaVA model after repeated GD\(Liuet al\.,[2022](https://arxiv.org/html/2606.12809#bib.bib26)\)unlearning operations, demonstrating the cumulative damage to response quality\.Despite its practical importance, MLLM lifelong unlearning remains largely underexplored\. A primary obstacle hindering progress is the lack of a large\-scale and comprehensive evaluation benchmark\. Most existing MLLM unlearning benchmarks are limited in data type or scale, making them unsuitable for the comprehensive lifelong unlearning evaluation\. For instance, MMUBench\(Liet al\.,[2024b](https://arxiv.org/html/2606.12809#bib.bib35)\)is limited in scale and diversity with only 20 concepts, FIUBench\(Maet al\.,[2024](https://arxiv.org/html/2606.12809#bib.bib36)\)focuses narrowly on facial information, and MLLMU\-Bench\(Liuet al\.,[2025](https://arxiv.org/html/2606.12809#bib.bib68)\)focuses only on celebrities\. Other recent efforts\(Huoet al\.,[2025](https://arxiv.org/html/2606.12809#bib.bib69); Wanget al\.,[2025b](https://arxiv.org/html/2606.12809#bib.bib72)\)have advanced MLLM unlearning but do not provide a comprehensive framework to evaluate the crucial cumulative effects of sequential unlearning requests\. These gaps make it difficult to thoroughly study how MLLMs behave over a continuous unlearning process\.

To fill these gaps and facilitate systematic MLLM lifelong unlearning research, we introduce theMLLMLifelongUnlearningBenchmark \(MLUBench\)\. MLUBench is a large\-scale, diverse benchmark specifically designed to simulate and evaluate MLLM lifelong unlearning\. It comprises 127 widely\-known real\-world entities across 9 distinct classes, with 5,105 associated images and 15,414 VQA pairs\. By design, MLUBench organizes these entities into a sequence of unlearning tasks, providing a comprehensive platform for assessing the long\-term performance of unlearning algorithms \(see Figure[2](https://arxiv.org/html/2606.12809#S3.F2)for an overview\)\.

Using MLUBench, we conduct extensive evaluations of existing unlearning methods\(Yaoet al\.,[2023](https://arxiv.org/html/2606.12809#bib.bib25); Liuet al\.,[2022](https://arxiv.org/html/2606.12809#bib.bib26); Yaoet al\.,[2024](https://arxiv.org/html/2606.12809#bib.bib27); Zhanget al\.,[2024b](https://arxiv.org/html/2606.12809#bib.bib28)\)and uncover two key findings\. First, we confirm that lifelong unlearning leads to severe, cumulative performance degradation on both forget quality and model utility\. For example, the forget quality of the GA method\(Yaoet al\.,[2023](https://arxiv.org/html/2606.12809#bib.bib25)\)on the first task drops from 0\.38 to a mere 0\.01 after subsequent unlearning operations\. Second, more critically, we reveal the unique challenge of the MLLM lifelong unlearning problem\. Specifically, unlike in unimodal models, MLLM lifelong unlearning is fundamentally constrained by the need to preserve multimodal alignment\. We empirically demonstrate that unlearning operations, even when applied to a single modality, can catastrophically disrupt this alignment, leading to a collapse in model’s performance\.

To alleviate the above challenges, motivated by the Mixture of Experts \(MoE\)\(Masoudnia and Ebrahimpour,[2014](https://arxiv.org/html/2606.12809#bib.bib32)\), we proposeLifelongUnlearning with aMixture\-of\-Experts \(LUMoE\), a simple but effective method\. Instead of continually altering the MLLM’s weights, LUMoE employs switchable Low\-Rank Adaptation \(LoRA\)\(Huet al\.,[2021](https://arxiv.org/html/2606.12809#bib.bib34)\)adapters as “experts” for specific unlearning tasks\. A powerful gate module directs inputs to the appropriate adapter, effectively handling unlearning requests\.

Our contributions are summarized as follows:

- •We study a practical and challenging problem of MLLM Lifelong Unlearning\. Through experiments, we identify that preserving multimodal alignment is the unique and fundamental challenge in MLLM lifelong unlearning, distinguishing it from its unimodal counterpart \(Section[3](https://arxiv.org/html/2606.12809#S3)\)\.
- •We introduce the MLUBench, a large\-scale and diverse benchmark designed for evaluating MLLM lifelong unlearning\. MLUBench spans 127 real\-world entities across 9 classes, includes 5,105 images, and 15,414 VQA pairs \(Section[4](https://arxiv.org/html/2606.12809#S4)\)\.
- •We perform extensive experiments on the MLUBench and reveal the critical performance degradation problem of existing unlearning methods \(Section[6](https://arxiv.org/html/2606.12809#S6)\)\. We design a simple but effective method, LUMoE\. It achieves a strong performance standard for future research in this area \(Section[5](https://arxiv.org/html/2606.12809#S5)\)\.

## 2Related Works

We review related works on machine unlearning and sequential unlearning for language models\. Additional related studies are provided in the Appendix[M](https://arxiv.org/html/2606.12809#A13)\.

Machine Unlearning for Language Models\.Machine unlearning for language models aims to remove specific data in language models\(Liuet al\.,[2024b](https://arxiv.org/html/2606.12809#bib.bib22),[d](https://arxiv.org/html/2606.12809#bib.bib24),[e](https://arxiv.org/html/2606.12809#bib.bib37); Maet al\.,[2024](https://arxiv.org/html/2606.12809#bib.bib36); Liet al\.,[2024b](https://arxiv.org/html/2606.12809#bib.bib35); Yaoet al\.,[2023](https://arxiv.org/html/2606.12809#bib.bib25)\)\. Gradient Ascent \(GA\)\(Yaoet al\.,[2023](https://arxiv.org/html/2606.12809#bib.bib25)\)reverses the gradient descent to eliminate unwanted data, but often degrades performance on unrelated data\(Liuet al\.,[2024e](https://arxiv.org/html/2606.12809#bib.bib37),[b](https://arxiv.org/html/2606.12809#bib.bib22)\)\. To address this, Gradient Difference \(GD\)\(Liuet al\.,[2022](https://arxiv.org/html/2606.12809#bib.bib26)\)and KL Minimization \(KL\)\(Yaoet al\.,[2024](https://arxiv.org/html/2606.12809#bib.bib27)\)introduce the retain loss to mitigate performance degradation\. Alignment\-based methods, such as Negative Preference Optimization \(NPO\)\(Zhanget al\.,[2024b](https://arxiv.org/html/2606.12809#bib.bib28)\), further alleviate the performance degradation\. With respect to the MLLMs unlearning,Liuet al\.\([2025](https://arxiv.org/html/2606.12809#bib.bib68)\)proposed the MLLMU\-Bench, which mainly targets multimodal profiles\. MMUNLEARNER\(Huoet al\.,[2025](https://arxiv.org/html/2606.12809#bib.bib69)\)is a geometry\-constrained gradient ascent method designed for MLLMs unlearning\.Wanget al\.\([2025b](https://arxiv.org/html/2606.12809#bib.bib72)\)introduced a visual knowledge distillation\-based method\.Fenget al\.\([2025](https://arxiv.org/html/2606.12809#bib.bib70)\)performed a systematic review of the generative model unlearning, including multimodal unlearning\. Compared with the existing benchmarks, our benchmark contains more entities and covers a broader range of types\.

Sequential Unlearning of Language Models\.The sequential unlearning for language models has attracted great attention\.Gaoet al\.\([2024](https://arxiv.org/html/2606.12809#bib.bib65)\)tackled the trade\-off between unlearning efficacy and model utility in LLMs, introducing theO3O^\{3\}framework to navigate this balance without relying on retained data\. In a complementary study,Shiet al\.\([2024b](https://arxiv.org/html/2606.12809#bib.bib20)\)evaluated the sustainability of unlearning methods, determining that they are ill\-equipped for sequential unlearning requests\.Kawakamiet al\.\([2025](https://arxiv.org/html/2606.12809#bib.bib67)\)explored the evaluation framework for the unlearning of large multimodal models\. Our work differs from\(Kawakamiet al\.,[2025](https://arxiv.org/html/2606.12809#bib.bib67)\)in that we introduce a new and comprehensive benchmark\. In addition, through experiments, we reveal the unique challenge of MLLM lifelong unlearning, distinguishing it from LLM lifelong unlearning\.

## 3Problem Formulation

In this section, we provide the problem formulation of MLLM unlearning and MLLM lifelong unlearning\. In addition, we discuss the unique challenge of the MLLM lifelong unlearning through experiments\.

### 3\.1MLLM Unlearning

We formulate the MLLM unlearning objective first\. Letℳθ\\mathcal\{M\}\_\{\\theta\}denote an MLLM parameterized byθ\\theta\. Given a specific multimodal entity and the information to be forgotten, MLLM unlearning seeks to obtain a new modelℳθ′\\mathcal\{M\}\_\{\\theta^\{\\prime\}\}\.ℳθ′\\mathcal\{M\}\_\{\\theta^\{\\prime\}\}should eliminate the targeted multimodal knowledge while maintaining overall performance on unrelated tasks\. Formally, letfi∈ℱf\_\{i\}\\in\\mathcal\{F\}denote the forgetting information about an unlearning entityii, andrj∈ℛr\_\{j\}\\in\\mathcal\{R\}denote the retained information related to a retained entityjj\. Letttdenote an unlearning task\. We define the forget set of taskttasFt=\{f1,f2,…,fn\}F\_\{t\}=\\\{f\_\{1\},f\_\{2\},\.\.\.,f\_\{n\}\\\}, and the retain information set of taskttasRt=\{r1,r2,…,rm\}R\_\{t\}=\\\{r\_\{1\},r\_\{2\},\.\.\.,r\_\{m\}\\\}\. Then, the unlearning task is formulated ast=\(Ft,Rt\)t=\(F\_\{t\},R\_\{t\}\)\. For an unlearned MLLMℳθ′\\mathcal\{M\}\_\{\\theta^\{\\prime\}\}, it should satisfy: 1\)∀fi∈Ft\\forall f\_\{i\}\\in F\_\{t\}: The model should not exhibit multimodal knowledge offif\_\{i\}, 2\)∀rj∈Rt\\forall r\_\{j\}\\in R\_\{t\}: The model should retain its original behavior regardingrjr\_\{j\}\.

### 3\.2MLLM Lifelong Unlearning

Then, we formulate the studied MLLM lifelong unlearning problem\. Given an MLLMℳθ\\mathcal\{M\}\_\{\\theta\}, the model is required to unlearn a series of tasks sequentially\. Letθt\\theta\_\{t\}denote the parameters of the MLLM after only a single unlearning tasktt\. Let𝒯=\{t1,t2,…,tk\}\\mathcal\{T\}=\\\{t\_\{1\},t\_\{2\},\.\.\.,t\_\{k\}\\\}represent an ordered sequence of unlearning tasks\. After sequentially unlearning all tasks in𝒯\\mathcal\{T\}, the model parameters are updated toθ𝒯\\theta\_\{\\mathcal\{T\}\}\. For any tasktt, we defineP​\(ℳθ,t\)P\(\\mathcal\{M\}\_\{\\theta\},t\)111In our paper, theP​\(ℳθ,t\)P\(\\mathcal\{M\}\_\{\\theta\},t\)can be either the forget quality or the model utility defined in Section[6\.1](https://arxiv.org/html/2606.12809#S6.SS1)\.as the general performance measure of modelℳθ\\mathcal\{M\}\_\{\\theta\}on tasktt\. The objective of MLLM lifelong unlearning is to minimize the MLLM’s performance degradation on previously unlearned tasks and effectively unlearn new tasks, formulated as

minθ𝒯​∑t∈𝒯\|P​\(ℳθt,t\)−P​\(ℳθ𝒯,t\)\|\.\\min\_\{\\theta\_\{\\mathcal\{T\}\}\}\\sum\_\{t\\in\\mathcal\{T\}\}\\left\|P\\bigl\(\\mathcal\{M\}\_\{\\theta\_\{t\}\},t\\bigr\)\-P\\bigl\(\\mathcal\{M\}\_\{\\theta\_\{\\mathcal\{T\}\}\},t\\bigr\)\\right\|\.\(1\)
It is noted that Eq\.[1](https://arxiv.org/html/2606.12809#S3.E1)focuses on mitigating cumulative degradation \(stability\) rather than ensuring the absolute efficacy of the underlying unlearning method\.

![Refer to caption](https://arxiv.org/html/2606.12809v1/x2.png)Figure 2:Overview of MLUBench\. The MLUBench comprises 127 entities across 9 categories \(broad data type\), with 5,105 images and 15414 VQA pairs \(large\-scale\)\.
### 3\.3The Uniqueness of MLLM Lifelong Unlearning

In this paper, we argue that MLLM lifelong unlearning isnot a straightforward extensionof the LLM lifelong unlearning, but a distinct concept and a more challenging problem\. Specifically, we hypothesize that the core distinction lies in themultimodal alignment, which introduces a unique challenge not present in unimodal LLMs\. In MLLM lifelong unlearning, the unlearning methods require preserving the integrity of both the language model and the vision components \(vision adapter and multimodal projector\), and the alignment that bridges them\. We empirically prove our argument in Section[6\.3](https://arxiv.org/html/2606.12809#S6.SS3)\(detailed results in Table[1](https://arxiv.org/html/2606.12809#S5.T1)\)\. Our results prove that MLLM lifelong unlearning may not be a problem that can be solved by addressing one modality in isolation, since continual unlearning in one modality can damage the alignment\. Therefore, an effective unlearning method in MLLM lifelong unlearning should considerprotecting MLLMs’ multimodal alignment\.

## 4MLUBench: A Benchmark for MLLM Lifelong Unlearning Evaluation

In this section, we introduce the MLUBench\. We first provide an overview, followed by the construction procedure and dataset filtration \(Section[4\.1](https://arxiv.org/html/2606.12809#S4.SS1)\)\. Then we present the division of MLUBench for sequential task construction and its generality evaluation \(Section[4\.2](https://arxiv.org/html/2606.12809#S4.SS2)\)\.

Copyright Disclaimer\.Images used in this study were collected from publicly available sources via Google Images\. In accordance with the[fair use principles](https://www.copyright.gov/fair-use), the use of these images is constrained within scholarly analysis and does not affect the market value of the original works\. All rights remain with the original copyright holders\.

### 4\.1Dataset Construction

Overview\.Many unlearning datasets\(Maet al\.,[2024](https://arxiv.org/html/2606.12809#bib.bib36); Mainiet al\.,[2024](https://arxiv.org/html/2606.12809#bib.bib55)\)consist of fictitious information\. Therefore, users require performing fine\-tuning on these datasets before using them, which may cause inconvenience\. In real\-world scenarios, it’s practical for a model to unlearn the knowledge it has already mastered\(Liuet al\.,[2024d](https://arxiv.org/html/2606.12809#bib.bib24)\)\. Thus, we build the MLUBench on the factual knowledge of widely known real\-world entities\. MLUBench contains 127 entities of 9 classes, their associated 15414 QA pairs, and 5105 image data\. Figure[2](https://arxiv.org/html/2606.12809#S3.F2)provides an overview of MLUBench\. The following introduces the construction procedure\.

Entities Selection\.MLUBench comprises 9 entity types from[Wikipedia](https://www.wikipedia.org/): Animals, Astronomy, Buildings, Cartoons, Corporations, Movies, Personage, Plants, and TV Series\. We manually select entities for each type \(see Appendix[A\.1](https://arxiv.org/html/2606.12809#A1.SS1)for the complete entity list\)\.

Images and QA pairs\.For each entity, we download images from[Google Images](https://images.google.com/)via automated crawling\. Instead of entity\-specific questions, we design a common question set for each entity type to capture their shared characteristics\. Using these questions as prompts, we employ the GPT\-4o\(Achiamet al\.,[2023](https://arxiv.org/html/2606.12809#bib.bib56)\)to generate entity\-specific answers\. Finally, we manually verify the correctness of answers generated by GPT\-4o\. See Appendix[A\.2](https://arxiv.org/html/2606.12809#A1.SS2)for all detailed questions\.

Dataset Filtration\.We manually examine all the collected images and remove low\-resolution and irrelevant images for each entity\. Next, to ensure the models have mastered the target entity knowledge, we input each pair into LLaVA\-v1\.6\-Vicuna\-7B and 13B\(Liuet al\.,[2024a](https://arxiv.org/html/2606.12809#bib.bib16)\)and retain only those that they both answer correctly \(verified by GPT\-4o\)\. This step is crucial, as the initial MLLMs must have mastered the relevant knowledge before unlearning it\.

### 4\.2Dataset Division and Generality

Sequential Unlearning Construction\.To construct sequential unlearning scenarios, we partition MLUBench into four tasks \(A, B, C, and D\) approximately equally\. Each task is subdivided into the forgetting and retained information sets\. The detailed entity allocation is provided in Appendix[A\.3](https://arxiv.org/html/2606.12809#A1.SS3)\. This partitioning strategy is inspired byKirkpatricket al\.\([2017](https://arxiv.org/html/2606.12809#bib.bib21)\), in which they construct a task sequence of three tasks \(A, B, C\) to evaluate the continual learning methods\.

Generality Evaluation\.We evaluate the robustness of unlearning methods against prompt variations by testing each question with four semantically equivalent but linguistically diverse variants\. For example, “Who directed this film?” is rephrased as “Who was responsible for directing this movie?”\. A full list of variants is in Appendix[A\.4](https://arxiv.org/html/2606.12809#A1.SS4)\. An effectively unlearned model should consistently suppress the target knowledge regardless of how the query is formulated\.

## 5Methodology: LUMoE

We introduce the LUMoE, a simple but effective method to mitigate the performance degradation problem in MLLM lifelong unlearning\. Section[5\.1](https://arxiv.org/html/2606.12809#S5.SS1)describes our technical motivation\. Section[5\.2](https://arxiv.org/html/2606.12809#S5.SS2)covers LUMoE’s procedure\.

### 5\.1Motivation

As established in our previous analysis \(Section[3](https://arxiv.org/html/2606.12809#S3)\), the unique challenge of MLLM lifelong unlearning is preserving the multimodal alignment of the MLLM\. Therefore, unlearning methods that directly modify the MLLM’s weights repeatedly may disrupt this alignment, leading to catastrophic performance degradation\. This insight motivates us to a design principle: an effective solution can isolate unlearning modifications from the stable MLLM\. Instead of repeatedly altering the MLLM, we can “attach” lightweight, task\-specific modules that handle the unlearning requests\. The MoE framework, combined with parameter\-efficient fine\-tuning \(PEFT\) methods like LoRA\(Huet al\.,[2021](https://arxiv.org/html/2606.12809#bib.bib34)\), naturally provides a solution to implement this principle\.

### 5\.2Method Procedure

Step\-1: Training LoRA adapters\.We treat LoRA adapters as specialized experts in the MoE framework\. The process begins with individually unlearning each task to acquire the corresponding LoRA adapter\. Specifically, we follow the method inMainiet al\.\([2024](https://arxiv.org/html/2606.12809#bib.bib55)\), utilizing PO to unlearn task\-specific information\. PO modifies Direct Preference Optimization \(DPO\)\(Rafailovet al\.,[2024](https://arxiv.org/html/2606.12809#bib.bib29)\)by focusing on aligning the model to decline answering queries related to the forget information set\(Mainiet al\.,[2024](https://arxiv.org/html/2606.12809#bib.bib55)\)\. This leads the model to prefer refusal responses, such as “Sorry, I cannot answer this question,” among other similar alternatives\. More examples of refusal responses are detailed in Appendix[B\.1](https://arxiv.org/html/2606.12809#A2.SS1)\.

Step\-2: Gate Module Routing\.The critical element of the LUMoE is the gate module, which dynamically assigns the appropriate LoRA adapter for each input\. Specifically, we utilize the GLM\-4V\-Plus model\(GLMet al\.,[2024](https://arxiv.org/html/2606.12809#bib.bib30)\), a state\-of\-the\-art \(SOTA\) commercial MLLM, to handle the multimodal inputs\. This router follows a two\-step procedure:\(1\) Entity Extraction:The GLM\-4V\-Plus is prompted to extract the relevant entity name from the input\. The prompt templates used for extraction are in Appendix[B\.2](https://arxiv.org/html/2606.12809#A2.SS2)\.\(2\) Task Matching:The extracted entity is compared against entities associated with previous unlearned tasks\. If the entity is found within the forget information set of a specific task, the corresponding LoRA adapter is applied to the base model for processing the input\.If no such match is found \(e\.g\., input belongs to the retain set\), the input is directly processed by the original MLLM, thereby preserving model utility\. If a request matches multiple existing adapters, all of the corresponding adapters can be simultaneously merged into the base model without interference \(detailed in Appendix[F](https://arxiv.org/html/2606.12809#A6)\)\. The details of adapters’ application are in Appendix[B\.3](https://arxiv.org/html/2606.12809#A2.SS3)\.

Error\-handling Mechanism\.Due to the potential limitations of routers, there exist instances where entity detection is imperfect\. To tackle this problem, we design an error\-handling mechanism\. Specifically, we instruct the model to output “None” when it is uncertain about an entity\. Subsequently, we classify such questions as retained questions and input them into the original MLLM\.

### 5\.3Discussion

We position LUMoE as an effective baseline methodrather than an ultimate or perfect solutionfor MLLM lifelong unlearning\. Its simplicity comes from our core insight: protecting multimodal alignment by isolating task\-specific modifications\. We believe LUMoE validates the effectiveness of the idea of isolation, which could motivate future methodology research in MLLM lifelong unlearning\.

Table 1:Results of Unlearn\-LLM\-Only and Unlearn\-Vision\-Only, different background colors of table header distinguish task groups, “X\-UY” denotes the model’s performance on Task X after unlearning Task Y\.A\-relatedB\-relatedC\-relatedD\-relMethodMetricA\-UAA\-UBA\-UCA\-UDB\-UBB\-UCB\-UDC\-UCC\-UDD\-UDUnlearn\-LLM\-Only \(Backbone Update\)Forget0\.2050\.0700\.0000\.0100\.1930\.0450\.0110\.0650\.0250\.100GAUtility0\.1020\.0230\.0000\.0000\.3080\.0500\.0160\.0000\.0000\.000KLForget0\.3550\.1400\.0400\.0400\.2550\.1130\.1030\.3450\.1450\.035Utility0\.1840\.0070\.0000\.0000\.3330\.1410\.1000\.0690\.0610\.007Unlearn\-Vision\-Only \(Vision Adapter Update\)Forget0\.3150\.0150\.0000\.0000\.0000\.0000\.0000\.0000\.0000\.000GAUtility0\.2460\.0460\.0000\.0000\.0170\.0000\.0000\.0070\.0070\.000KLForget0\.4750\.4100\.3330\.1500\.2720\.2200\.1850\.4000\.2350\.245Utility0\.4840\.2540\.2040\.1380\.3330\.2650\.1410\.1840\.1060\.200

## 6Experiments

### 6\.1Evaluation Metrics

#### 6\.1\.1Forget Quality

The “Golden Standard” of machine unlearning is typically defined as acquiring a model that is indistinguishable from a retrained model without the forget set\(Mainiet al\.,[2024](https://arxiv.org/html/2606.12809#bib.bib55); Liuet al\.,[2024e](https://arxiv.org/html/2606.12809#bib.bib37)\)\. However, in the case of MLUBench, which the initial MLLM already masters, a retrained model that excludes MLUBench would be prohibitively costly\. Consequently, the “Golden Standard” is no longer available\. Therefore, the metrics like the Kolmogorov\-Smirnov test \(KS\-Test\)\(Mainiet al\.,[2024](https://arxiv.org/html/2606.12809#bib.bib55)\)that rely on outputs from the re\-trained model cannot be utilized\. In light of this restriction, followingLiuet al\.\([2024d](https://arxiv.org/html/2606.12809#bib.bib24)\), we propose the GPT rejection score as our metric to assess forget quality\.

GPT Rejection Score\.The core idea behind the GPT rejection score is simple:*A response that fails to reject a question may either be a hallucination or the factual knowledge of the unlearning entity, while a high\-quality refusal effectively prevents both scenarios*\(Liuet al\.,[2024d](https://arxiv.org/html/2606.12809#bib.bib24)\)\. Formally, given a question, a response, and the ground\-truth answer, we prompt GPT\-4o to evaluate the quality of the rejection, assigning scores from\{0,1,2\}\\\{0,1,2\\\}, where a score of22indicates a high\-quality refusal\. The prompt is in Appendix[C\.1](https://arxiv.org/html/2606.12809#A3.SS1)\. It is noted that*the GPT rejection score may be stricter than other metrics*\(e\.g\., KS\-Test\)\. Since the model can only achieve a high score when it outputs a high\-quality refusal\. For example, the hallucination answer may score high in other metrics, but zero in our metrics\.

#### 6\.1\.2Model Utility

We evaluate the model utility by assessing the accuracy of model responses on the retain set\. Traditional metrics like ROUGE\(Lin,[2004](https://arxiv.org/html/2606.12809#bib.bib58)\)may ignore the semantic information in model generations\(Wanget al\.,[2023](https://arxiv.org/html/2606.12809#bib.bib59)\), which is essential for the evaluation\. Therefore, motivated by LLM\-as\-a\-Judge\(Zhenget al\.,[2023](https://arxiv.org/html/2606.12809#bib.bib57)\)andMaet al\.\([2024](https://arxiv.org/html/2606.12809#bib.bib36)\), we introduce the GPT Correctness score\.

GPT Correctness Score\.Formally, given a question and a model response, we use GPT\-4o to evaluate the correctness of the answer\. GPT\-4o assesses the quality, relevance, and correctness of the response\. It assigns a score from0,1,2\{0,1,2\}, where22represents a high\-quality, relevant, and correct answer\. The prompt is in Appendix[C\.2](https://arxiv.org/html/2606.12809#A3.SS2)\. In addition, we also validate the alignment between LLM\-judge scores and human judgment in Appendix[H](https://arxiv.org/html/2606.12809#A8)\.

### 6\.2Setup

Models\.The chosen MLLMs are the LLaVA\-v1\.6\-7B, LLaVA\-v1\.6\-13B\(Liuet al\.,[2024a](https://arxiv.org/html/2606.12809#bib.bib16)\), and Qwen3\-VL\-4B\-Instruct\(Baiet al\.,[2025](https://arxiv.org/html/2606.12809#bib.bib73)\)\.

Baseline Methods\.We employ four widely used unlearning methods as baselines: \(1\) Grad Ascent \(GA\)\(Yaoet al\.,[2023](https://arxiv.org/html/2606.12809#bib.bib25)\), \(2\) Grad Difference \(GD\)\(Liuet al\.,[2022](https://arxiv.org/html/2606.12809#bib.bib26)\), \(3\) KL Minimization \(KL\)\(Yaoet al\.,[2024](https://arxiv.org/html/2606.12809#bib.bib27)\), \(4\) Negative Preference Optimization \(NPO\)\(Zhanget al\.,[2024b](https://arxiv.org/html/2606.12809#bib.bib28)\)\. A detailed description of baselines is in Appendix[K](https://arxiv.org/html/2606.12809#A11)\.

Baselines Settings\.MLLMs unlearn all tasks in the sequence order of Task A, Task B, Task C, and Task D\. Specifically, we employ baselines to unlearn new tasks based on a model that has unlearned previous tasks\. After unlearning each task, we save the checkpoint and conduct testing on the tasks that have already been unlearned\.

Implementation Details\.The LoRA\-rank and LoRA\-alpha are set to 32\. The vision tower learning rate is 2e\-6\. The projector learning rate is 1e\-5, and the training batch size is 4\. To ensure a fair and rigorous comparison, we conduct extensive hyperparameter tuning for all baseline methods\. Please refer to Appendix[C\.4](https://arxiv.org/html/2606.12809#A3.SS4)for detailed parameters\.

Final Score\.For each task, we calculate the final score as the sum of model scores divided by the sum of maximum possible scores, i\.e\.,Final Score=∑Model Scores∑Maximum Possible Scores\\text\{Final Score\}=\\frac\{\\sum\\text\{Model Scores\}\}\{\\sum\\text\{Maximum Possible Scores\}\}\.

### 6\.3Results

Table 2:Comparison of different unlearning methods on MLUBench \(LLaVA\-7B and LLaVA\-13B\), “X\-UY” denotes the model’s performance on Task X after unlearning Task Y, LUMoE \(Ours\) effectively maintains utility while achieving high forget quality\.A\-relatedB\-relatedC\-relatedD\-relMethodMetricA\-UAA\-UBA\-UCA\-UDB\-UBB\-UCB\-UDC\-UCC\-UDD\-UDLLaVA\-7BForget0\.3800\.1950\.0350\.0100\.2200\.1300\.0700\.1850\.0750\.060GAUtility0\.1200\.0200\.0000\.0100\.1000\.0400\.0400\.0380\.0100\.020KLForget0\.2800\.1100\.0000\.0000\.1800\.0050\.0000\.0150\.0050\.000Utility0\.1230\.0500\.0000\.0000\.1160\.0160\.0000\.0100\.0000\.000Forget0\.3300\.1150\.0150\.0000\.1530\.0400\.0300\.1100\.0350\.045GDUtility0\.1400\.0600\.0150\.0000\.1250\.0600\.0400\.0500\.0100\.015NPOForget0\.4200\.0050\.0000\.0050\.0000\.0000\.0000\.0000\.0000\.000Utility0\.2380\.0000\.0000\.0000\.0000\.0000\.0000\.0000\.0000\.000LUMoEForget1\.0001\.0001\.0001\.0000\.9500\.9500\.9500\.9900\.9900\.960\(Ours\)Utility0\.9300\.9300\.9300\.9300\.8800\.8800\.8800\.9400\.9400\.910LLaVA\-13BForget0\.4850\.0700\.0350\.0150\.0570\.0220\.0110\.1000\.0800\.030GAUtility0\.3840\.0100\.0000\.0000\.2500\.1500\.1250\.1000\.0800\.200KLForget0\.4700\.1450\.0200\.0400\.1130\.0300\.0280\.1050\.0950\.065Utility0\.5380\.0300\.0000\.0000\.3250\.1160\.1250\.0400\.0380\.115Forget0\.3400\.0050\.0050\.0000\.0050\.0100\.0050\.0250\.0100\.020GDUtility0\.0600\.0000\.0000\.0000\.2500\.1750\.1250\.0600\.0700\.040NPOForget0\.5100\.0300\.0000\.0000\.0500\.0000\.0000\.0000\.0000\.000Utility0\.0840\.0000\.0000\.0000\.0000\.0000\.0000\.0000\.0000\.000LUMoEForget1\.0001\.0001\.0001\.0000\.9500\.9500\.9501\.0001\.0000\.980\(Ours\)Utility0\.9500\.9500\.9500\.9500\.9000\.9000\.9000\.9200\.9200\.940Lifelong unlearning causes significant performance degradation\.As illustrated in Table[2](https://arxiv.org/html/2606.12809#S6.T2), all baselines exhibit significant performance degradation in forget quality and model utility throughout the lifelong unlearning process\. For example, on the LLaVA\-7B model, the GA method initially achieves a forget quality of 0\.38 on Task A\. Upon completion of Task D unlearning, GA demonstrates near\-complete degradation in both forget quality and model utility on all previously unlearned tasks, approaching 0\. Other baselines also exhibited similar behavior on LLaVA\-7B and LLaVA\-13B, indicating the generality of our findings\. To further validate the generality of our findings, we conduct lifelong unlearning experiments on the Qwen3\-VL\-4B\-Instruct model from the Qwen3\-VL series; the detailed results are in the Appendix[G](https://arxiv.org/html/2606.12809#A7)\. For example, on the Qwen3\-VL\-4B\-instruct, the GD method initially achieves a forget quality of 0\.54 on Task A\. However, after the unlearning of Task B, GD’s forget quality on Task A collapses to 0\.115\. In addition, we provide a further discussion of the performance of baselines in Appendix[C\.3](https://arxiv.org/html/2606.12809#A3.SS3)\.

LUMoE shows superior performance than all baselines\.According to Table[2](https://arxiv.org/html/2606.12809#S6.T2), the LUMoE method performs excellently on all tasks’ forget quality and model quality, approaching 1 throughout the lifelong unlearning process\.

Lifelong unlearning undermines MLLM’s language ability\.Figure[1](https://arxiv.org/html/2606.12809#S1.F1)\(b\) demonstrates the language ability transformation\. Specifically, the LLaVA\-7B is asked to identify the director of a well\-known film\. Before unlearning, the model can output the correct answer\. After one GD unlearning operation, the model avoids answering but remains coherent\. However, after three GD unlearning procedures on other tasks, the model outputs nonsensical and repetitive content\. This indicates the potential corruption of the model’s core language ability\.

Effective MLLM lifelong unlearning needs preserving multimodal alignment\.To empirically prove our argument in Section[3\.3](https://arxiv.org/html/2606.12809#S3.SS3), we conduct experiments where we isolate the unlearning process to update either the language or vision part of MLLMs\. Specifically, we apply the GA\(Yaoet al\.,[2023](https://arxiv.org/html/2606.12809#bib.bib25)\)and KL\(Yaoet al\.,[2024](https://arxiv.org/html/2606.12809#bib.bib27)\)under two conditions:

- •Unlearn\-LLM\-Only: We freeze the vision components and only update the backbone LLM weights;
- •Unlearn\-Vision\-Only: We freeze the LLM and only update the vision components\.

The results are in Table[1](https://arxiv.org/html/2606.12809#S5.T1)\. In Table[1](https://arxiv.org/html/2606.12809#S5.T1), “X\-UY” denotes the model’s performance on task X after unlearning task Y\. “Forget” and “Utility” denote the forget quality and model utility metrics defined in Section[6\.1](https://arxiv.org/html/2606.12809#S6.SS1)\. According to Table[1](https://arxiv.org/html/2606.12809#S5.T1), in both scenarios, the model’s overall performance suffers severe, cumulative degradation\. For example, in the Unlearn\-Vision\-Only setting, the model’s performance on Task A drops to almost 0 after unlearning the last Task D\. Therefore, our experiments prove the argument in Section[3\.3](https://arxiv.org/html/2606.12809#S3.SS3), that isolating unlearning can damage alignment between modalities and undermine MLLMs’ performance\.

Measure the Modality Gap\.In addition, to provide more direct evidence for the alignment claim, we measure theModality Gap\(L2 distance between the visual feature centroid and the language feature centroid\) between vision and language representations using Qwen3\-VL\-4B\-Instruct\(Baiet al\.,[2025](https://arxiv.org/html/2606.12809#bib.bib73)\)\. A smaller modality gap means better alignment\. As shown in the Table[3](https://arxiv.org/html/2606.12809#S6.T3), after unlearning, the Modality Gap on four tasks \(A to D\) enlarges consistently\.

Table 3:Representation drift analysis on Qwen3\-VL\-4B\-Instruct\. The data in the table shows the modality gap\.TaskOriginal ModelUnlearned ModelΔ\\DeltaGapTask A20\.72722\.353\+1\.626Task B19\.08120\.067\+0\.987Task C17\.37218\.904\+1\.532Task D18\.52219\.785\+1\.263
### 6\.4Ablation Studies

Remove the gate module\.To validate the importance of the gate module, we remove it and employ PO only to perform lifelong unlearning\. Implementation details are in Appendix[D\.1](https://arxiv.org/html/2606.12809#A4.SS1)and results are in Table[7](https://arxiv.org/html/2606.12809#S6.T7)\. While the PO method does not lead to a continual decline in forget quality, model utility still deteriorates rapidly\. The model progressively becomes more inclined to refuse to answer questions, even when questions belong to the retained set\.

Table 4:Open\-sourced model router performance\.Router ModelTask ATask BTask CTask DForget QualityQwen3\-VL\-4B1\.0000\.9100\.9800\.960Qwen3\-VL\-8B1\.0000\.8800\.9900\.960Model UtilityQwen3\-VL\-4B0\.9300\.6400\.9400\.910Qwen3\-VL\-8B0\.9300\.7300\.9400\.910Replace the GLM\-4V\-Plus router\.The performance of LUMoE depends on the router model\. Therefore, to investigate the impact of different router models, we replace the GLM\-4V\-Plus\(GLMet al\.,[2024](https://arxiv.org/html/2606.12809#bib.bib30)\)with GPT\-4o\(Hurstet al\.,[2024](https://arxiv.org/html/2606.12809#bib.bib17)\)and Gemini\(Teamet al\.,[2023](https://arxiv.org/html/2606.12809#bib.bib18)\)\. Since LUMoE maintains stable performance throughout the unlearning process, we report task\-level results \(e\.g\., Task A, B\) instead of the “X\-UY” style \(e\.g\., A\-UB\)\. The unlearned model is LLaVA\-7B\. According to Table[6](https://arxiv.org/html/2606.12809#S6.T6), considering both forget quality and model utility, GLM\-4V\-Plus performs the best, followed by Gemini and GPT\-4o\.

Table 5:Comparison of different methods on the MLLMU\-Bench\. LUMoE outperforms baseline methods \(GA and GD\) significantly in both forget quality and utility preservation\.A\-relatedB\-relatedC\-relatedMethodMetricA\-UAA\-UBA\-UCB\-UBB\-UCC\-UCForget Quality0\.2700\.2050\.0600\.2380\.1360\.120GAModel Utility0\.3200\.1200\.0100\.1830\.0910\.038Forget Quality0\.3000\.1000\.0250\.1810\.0900\.070GDModel Utility0\.2840\.1230\.0150\.2080\.0830\.030LUMoEForget Quality0\.9800\.9800\.9801\.0001\.0001\.000\(Ours\)Model Utility0\.8800\.8800\.8800\.8600\.8600\.950Analysis of smaller routers\.To examine the effectiveness of smaller open\-sourced routers, we used the Qwen3\-VL\-4B\-Instruct and Qwen3\-VL\-8B\-Instruct as the routers\. The results are shown in the Table[4](https://arxiv.org/html/2606.12809#S6.T4)\. The average accuracy of Qwen3\-VL\-4B\-Instruct is 97\.1%, and Qwen3\-VL\-8B\-Instruct is 98%\. According to the results, strong open\-source routers can also achieve good performance\.

Robustness of the metrics\.To evaluate the robustness of our metrics with respect to the judge model, we replace the GPT\-4o judge with other LLMs such as Gemini and Claude\. Specifically, across both Gemini and Claude judges, LUMoE consistently maintains Forget Quality and Model Utility scores above 0\.9 and 0\.85, respectively, while baselines like GA and GD consistently score below 0\.4\. This performance gap validates that our main conclusions are stable on the choice of judge\. The detailed results are provided in Appendix[D\.5](https://arxiv.org/html/2606.12809#A4.SS5)\.

Additional experiments on the MLLMU\-Bench\.To further test the validity of our proposed LUMoE, we conduct experiments on another benchmark of MLLMU\-Bench\(Liuet al\.,[2025](https://arxiv.org/html/2606.12809#bib.bib68)\)\. Specifically, we select the 153 profiles for the public celebrities subset and divide them into three tasks \(A, B, and C\) to simulate the lifelong unlearning scenario\. The results are in the Table[5](https://arxiv.org/html/2606.12809#S6.T5)\. According to the results, both the GA and GD methods still suffer from cumulative degradation on the MLLMU\-Bench, while our LUMoE continuously demonstrates strong performance\.

Jailbreak attack against the LUMoE\.We employ jailbreak prompts from AutoDAN\(Liuet al\.,[2023](https://arxiv.org/html/2606.12809#bib.bib63)\)to evaluate the reliability and safety of LUMoE\. Specifically, as shown in Table[4](https://arxiv.org/html/2606.12809#S6.F4), the Forget Quality remains at 0\.95 or higher across all tasks, even under jailbreak attacks, with the maximum performance drop being a negligible 0\.05 \(from 1\.00 to 0\.95 in Task B\)\. These results highlight LUMoE’s robustness with respect to the jailbreak attacks\.

Table 6:Results of using different router models\.RouterTask ATask BTask CTask DForget QualityGLM\-4V\-Plus1\.000\.950\.990\.96GPT\-4o0\.920\.940\.810\.86Gemini1\.000\.931\.001\.00Model UtilityGLM\-4V\-Plus0\.930\.880\.940\.91GPT\-4o0\.900\.850\.910\.93Gemini0\.900\.800\.910\.91Table 7:Performance of the PO method after removing the gate module \(LLaVA\-7B\)\. Without the gate, the model cannot preserve utility \(Row 2\), leading to significant degradation compared to LUMoE\.A\-relatedB\-relatedC\-relatedD\-relMetricA\-UAA\-UBA\-UCA\-UDB\-UBB\-UCB\-UDC\-UCC\-UDD\-UDForget Quality0\.560\.590\.690\.700\.450\.490\.500\.740\.750\.92Model Utility0\.510\.450\.370\.270\.300\.230\.200\.240\.190\.27Figure 3:Robustness against jailbreak attack\.ConditionTaskATaskBTaskCTaskDNo Jailbreak1\.001\.001\.001\.000\.990\.990\.960\.96With Jailbreak0\.990\.990\.950\.950\.950\.950\.960\.96
Figure 4:Computation cost of LUMoE\.JobTime CostTraining a LoRA adapter∼\\qquad\\sim11mTask matching for a QA pair∼\\sim2sMerging LoRA adapter \(cached\)∼\\sim4s

Does LUMoE have generality?To investigate the adaptability of LUMoE to a diverse set of text prompts, we tested the unlearned model on all four variant questions discussed in Section[4\.2](https://arxiv.org/html/2606.12809#S4.SS2)\. It is noted that the model has not been trained to unlearn these variants\. LUMoE’s performance on these unseen variant questions remains consistently high and comparable to its performance on original questions\. Across most variants, the Forget Quality exceeds 0\.95, and Model Utility remains within a robust range of 0\.87 to 0\.96\. Detailed results are in Appendix[D\.2](https://arxiv.org/html/2606.12809#A4.SS2)\.

Impact of order of tasks\.Now we examine the influence of task ordering\. Specifically, we implement an alternative task sequence \(Task C → Task A → Task B → Task D\) and replicate the experimental procedure on LLaVA\-7B\. The results, detailed in Appendix[D\.3](https://arxiv.org/html/2606.12809#A4.SS3), confirm our primary conclusions\. Quantitatively, with the new task order, LUMoE maintains both Forget Quality and Model Utility scores consistently above 0\.88 across all stages\. In contrast, the scores of all baselines plummet to near\-zero after just one or two subsequent unlearning steps\. This demonstrates the robustness of our findings with respect to the task sequence\.

Impact of number of tasks\.We now evaluate the robustness of our findings with respect to an increased number of tasks\. Specifically, we divide the MLUBench into five parts and replicate the experimental procedure on LLaVA\-7B\. The results, detailed in Appendix[D\.4](https://arxiv.org/html/2606.12809#A4.SS4), again confirm our primary conclusions\. Quantitatively, under the five\-task setting, LUMoE’s Forget Quality and Model Utility scores remain stable, with all metrics holding above 0\.88\. In contrast, all baselines suffer from a complete performance collapse, with their scores on previously unlearned tasks dropping to zero, often after only one or two subsequent steps\.

Analysis of task generalization after lifelong unlearning\.To evaluate how lifelong unlearning affects the model’s general capabilities, we assess the unlearned models on two general\-purpose benchmarks, TruthfulQA\(Linet al\.,[2021](https://arxiv.org/html/2606.12809#bib.bib64)\)and MMBench\(Liuet al\.,[2024c](https://arxiv.org/html/2606.12809#bib.bib71)\)\. The results, detailed in Appendix[D\.6](https://arxiv.org/html/2606.12809#A4.SS6), reveal a clear distinction between LUMoE and baselines\. For baselines, the unlearning process is catastrophic\. The performance on the TruthfulQA rapidly degrades with each successive unlearning step\. Performance plummets from the initial 0\.5 to almost zero after three or four unlearning steps\. This cumulative decline indicates that existing unlearning methods fail to preserve the model’s general abilities\. We provide an analysis of this failure in Appendix[E](https://arxiv.org/html/2606.12809#A5)\. In contrast, LUMoE preserves the model’s general abilities\. As quantified across an extensive suite of evaluations in Appendix[D\.6](https://arxiv.org/html/2606.12809#A4.SS6), including different benchmarks \(TruthfulQA, MMBench\-EN/CN, CCBench\) and two distinct model sizes \(LLaVA\-7B and 13B\), the performance drop after the complete lifelong unlearning is consistently less than 0\.6%\.

Pre\-unlearning accuracy evaluation\.To evaluate the pre\-unlearning accuracy beyond the LLaVA series, we evaluate two different models from the Qwen series \(Qwen2\.5\-VL\-32B\-Instruct and Qwen2\.5\-VL\-72B\-Instruct\) on our MLUBench\. The detailed results in the Appendix[D\.7](https://arxiv.org/html/2606.12809#A4.SS7)show that both models achieve almost 100% pre\-unlearning accuracy\. Therefore, our benchmark can achieve high pre\-unlearning accuracy across different model series\.

### 6\.5Computation Efficiency

We provide a detailed analysis of the computational cost associated with LUMoE\. All running times are acquired on a server with NVIDIA A100 40GB GPUs and set up with Ubuntu 18\.04\. The main components include training LoRA adapters, task matching, and adapter merging\. The statistics are summarized in Table[4](https://arxiv.org/html/2606.12809#S6.F4), where m represents minutes and s represents seconds\. Besides, we implement a caching mechanism that keeps previously loaded adapters in memory to improve efficiency\. As a result, repeated merging of the same adapter avoids redundant loading and compilation, reducing the merging time\. In addition, the average size of an adapter is about 170MB, which may be negligible for modern storage and memory solutions\.

## 7Conclusion

In this paper, we study a practical and challenging problem of MLLM lifelong unlearning\. To systematically study this problem, we introduce the MLUBench, a large\-scale and comprehensive benchmark designed to evaluate in the MLLM lifelong unlearning setting\. Using the MLUBench, we reveal two critical findings: First, lifelong unlearning will cause severe performance degradation of MLLMs; Second, MLLM lifelong unlearning has its unique challenges compared with its unimodal counterpart\. To mitigate the performance degradation, inspired by MoE, we propose a simple but effective method called LUMoE\. Experiments on MLUBench confirm LUMoE’s superior performance\.

## Impact Statement

This paper studies a challenging and practical problem of MLLM lifelong unlearning, which is essential in the privacy or copyright field\. We introduce a comprehensive benchmark and provide deep insights into this problem\. We also propose an effective method called LUMoE to mitigate the performance degradation problem\. Therefore, this paper has positive social impacts of mitigating the rising privacy and copyright concerns in MLLMs\.

## Acknowledgments

This work was partially supported by the National Natural Science Foundation of China \(No\. 62525213, No\. 62372459, No\. 62402499\)\.

## References

- J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2023\)Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§4\.1](https://arxiv.org/html/2606.12809#S4.SS1.p3.1)\.
- S\. Bai, Y\. Cai, R\. Chen, K\. Chen, X\. Chen, Z\. Cheng, L\. Deng, W\. Ding, C\. Gao, C\. Ge,et al\.\(2025\)Qwen3\-vl technical report\.arXiv preprint arXiv:2511\.21631\.Cited by:[§6\.2](https://arxiv.org/html/2606.12809#S6.SS2.p1.1),[§6\.3](https://arxiv.org/html/2606.12809#S6.SS3.p5.1)\.
- C\. Chen, J\. Zhu, X\. Luo, H\. Shen, L\. Gao, and J\. Song \(2024\)CoIN: a benchmark of continual instruction tuning for multimodel large language model\.arXiv preprint arXiv:2403\.08350\.Cited by:[Appendix M](https://arxiv.org/html/2606.12809#A13.p1.1)\.
- W\. Chen, Y\. Zhou, N\. Du, Y\. Huang, J\. Laudon, Z\. Chen, and C\. Cui \(2023\)Lifelong language pretraining with distribution\-specialized experts\.InInternational Conference on Machine Learning,pp\. 5383–5395\.Cited by:[Appendix M](https://arxiv.org/html/2606.12809#A13.p1.1)\.
- X\. Feng, J\. Zhang, F\. Yu, C\. Wang, L\. Zhang, K\. Li, Y\. Li, C\. Chen, and J\. Yin \(2025\)A survey on generative model unlearning: fundamentals, taxonomy, evaluation, and future direction\.arXiv preprint arXiv:2507\.19894\.Cited by:[§2](https://arxiv.org/html/2606.12809#S2.p2.1)\.
- C\. Gao, L\. Wang, K\. Ding, C\. Weng, X\. Wang, and Q\. Zhu \(2024\)On large language model continual unlearning\.arXiv preprint arXiv:2407\.10223\.Cited by:[§2](https://arxiv.org/html/2606.12809#S2.p3.1)\.
- S\. Garg, M\. Farajtabar, H\. Pouransari, R\. Vemulapalli, S\. Mehta, O\. Tuzel, V\. Shankar, and F\. Faghri \(2023\)Tic\-clip: continual training of clip models\.arXiv preprint arXiv:2310\.16226\.Cited by:[Appendix M](https://arxiv.org/html/2606.12809#A13.p1.1)\.
- T\. GLM, A\. Zeng, B\. Xu, B\. Wang, C\. Zhang, D\. Yin, D\. Zhang, D\. Rojas, G\. Feng, H\. Zhao,et al\.\(2024\)Chatglm: a family of large language models from glm\-130b to glm\-4 all tools\.arXiv preprint arXiv:2406\.12793\.Cited by:[§5\.2](https://arxiv.org/html/2606.12809#S5.SS2.p2.1),[§6\.4](https://arxiv.org/html/2606.12809#S6.SS4.p2.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2021\)Lora: low\-rank adaptation of large language models\.arXiv preprint arXiv:2106\.09685\.Cited by:[§1](https://arxiv.org/html/2606.12809#S1.p5.1),[§5\.1](https://arxiv.org/html/2606.12809#S5.SS1.p1.1)\.
- J\. Huo, Y\. Yan, X\. Zheng, Y\. Lyu, X\. Zou, Z\. Wei, and X\. Hu \(2025\)Mmunlearner: reformulating multimodal machine unlearning in the era of multimodal large language models\.arXiv preprint arXiv:2502\.11051\.Cited by:[§1](https://arxiv.org/html/2606.12809#S1.p2.1),[§2](https://arxiv.org/html/2606.12809#S2.p2.1)\.
- A\. Hurst, A\. Lerer, A\. P\. Goucher, A\. Perelman, A\. Ramesh, A\. Clark, A\. Ostrow, A\. Welihinda, A\. Hayes, A\. Radford,et al\.\(2024\)Gpt\-4o system card\.arXiv preprint arXiv:2410\.21276\.Cited by:[§1](https://arxiv.org/html/2606.12809#S1.p1.1),[§6\.4](https://arxiv.org/html/2606.12809#S6.SS4.p2.1)\.
- J\. Jang, S\. Ye, C\. Lee, S\. Yang, J\. Shin, J\. Han, G\. Kim, and M\. Seo \(2022\)Temporalwiki: a lifelong benchmark for training and evaluating ever\-evolving language models\.arXiv preprint arXiv:2204\.14211\.Cited by:[Appendix M](https://arxiv.org/html/2606.12809#A13.p1.1)\.
- X\. Jin, D\. Zhang, H\. Zhu, W\. Xiao, S\. Li, X\. Wei, A\. Arnold, and X\. Ren \(2021\)Lifelong pretraining: continually adapting language models to emerging corpora\.arXiv preprint arXiv:2110\.08534\.Cited by:[Appendix M](https://arxiv.org/html/2606.12809#A13.p1.1)\.
- T\. Kawakami, K\. Egashira, A\. Miyai, G\. Irie, and K\. Aizawa \(2025\)PULSE: practical evaluation scenarios for large multimodal model unlearning\.arXiv preprint arXiv:2507\.01271\.Cited by:[§2](https://arxiv.org/html/2606.12809#S2.p3.1)\.
- S\. Khan, S\. Agarwal, and P\. Srijith \(2022\)Lifelong language learning with adapter based transformers\.InContinual Lifelong Learning Workshop at ACML 2022,Cited by:[Appendix M](https://arxiv.org/html/2606.12809#A13.p3.1)\.
- J\. Kirkpatrick, R\. Pascanu, N\. Rabinowitz, J\. Veness, G\. Desjardins, A\. A\. Rusu, K\. Milan, J\. Quan, T\. Ramalho, A\. Grabska\-Barwinska,et al\.\(2017\)Overcoming catastrophic forgetting in neural networks\.Proceedings of the national academy of sciences114\(13\),pp\. 3521–3526\.Cited by:[§4\.2](https://arxiv.org/html/2606.12809#S4.SS2.p1.1)\.
- P\. Langley \(2000\)Crafting papers on machine learning\.InProceedings of the 17th International Conference on Machine Learning \(ICML 2000\),P\. Langley \(Ed\.\),Stanford, CA,pp\. 1207–1216\.Cited by:[Appendix N](https://arxiv.org/html/2606.12809#A14.p5.1)\.
- S\. Lee, J\. Ha, D\. Zhang, and G\. Kim \(2020\)A neural dirichlet process mixture model for task\-free continual learning\.arXiv preprint arXiv:2001\.00689\.Cited by:[Appendix M](https://arxiv.org/html/2606.12809#A13.p2.1)\.
- H\. Li, S\. Lin, L\. Duan, Y\. Liang, and N\. B\. Shroff \(2024a\)Theory on mixture\-of\-experts in continual learning\.arXiv preprint arXiv:2406\.16437\.Cited by:[Appendix M](https://arxiv.org/html/2606.12809#A13.p2.1)\.
- J\. Li, Q\. Wei, C\. Zhang, G\. Qi, M\. Du, Y\. Chen, and S\. Bi \(2024b\)Single image unlearning: efficient machine unlearning in multimodal large language models\.arXiv preprint arXiv:2405\.12523\.Cited by:[§1](https://arxiv.org/html/2606.12809#S1.p1.1),[§1](https://arxiv.org/html/2606.12809#S1.p2.1),[§2](https://arxiv.org/html/2606.12809#S2.p2.1)\.
- C\. Lin \(2004\)Rouge: a package for automatic evaluation of summaries\.InText summarization branches out,pp\. 74–81\.Cited by:[§6\.1\.2](https://arxiv.org/html/2606.12809#S6.SS1.SSS2.p1.1)\.
- S\. Lin, J\. Hilton, and O\. Evans \(2021\)Truthfulqa: measuring how models mimic human falsehoods\.arXiv preprint arXiv:2109\.07958\.Cited by:[§D\.6](https://arxiv.org/html/2606.12809#A4.SS6.SSS0.Px1.p1.1),[§6\.4](https://arxiv.org/html/2606.12809#S6.SS4.p10.1)\.
- B\. Liu, Q\. Liu, and P\. Stone \(2022\)Continual learning and private unlearning\.InConference on Lifelong Learning Agents,pp\. 243–254\.Cited by:[Appendix K](https://arxiv.org/html/2606.12809#A11.p3.1.1),[Figure 1](https://arxiv.org/html/2606.12809#S1.F1),[Figure 1](https://arxiv.org/html/2606.12809#S1.F1.3.2),[§1](https://arxiv.org/html/2606.12809#S1.p4.1),[§2](https://arxiv.org/html/2606.12809#S2.p2.1),[§6\.2](https://arxiv.org/html/2606.12809#S6.SS2.p2.1)\.
- H\. Liu, C\. Li, Y\. Li, B\. Li, Y\. Zhang, S\. Shen, and Y\. J\. Lee \(2024a\)LLaVA\-next: improved reasoning, ocr, and world knowledge\.External Links:[Link](https://llava-vl.github.io/blog/2024-01-30-llava-next/)Cited by:[§4\.1](https://arxiv.org/html/2606.12809#S4.SS1.p4.1),[§6\.2](https://arxiv.org/html/2606.12809#S6.SS2.p1.1)\.
- S\. Liu, Y\. Yao, J\. Jia, S\. Casper, N\. Baracaldo, P\. Hase, Y\. Yao, C\. Y\. Liu, X\. Xu, H\. Li,et al\.\(2024b\)Rethinking machine unlearning for large language models\.arXiv preprint arXiv:2402\.08787\.Cited by:[§2](https://arxiv.org/html/2606.12809#S2.p2.1)\.
- X\. Liu, N\. Xu, M\. Chen, and C\. Xiao \(2023\)Autodan: generating stealthy jailbreak prompts on aligned large language models\.arXiv preprint arXiv:2310\.04451\.Cited by:[§6\.4](https://arxiv.org/html/2606.12809#S6.SS4.p6.1)\.
- Y\. Liu, H\. Duan, Y\. Zhang, B\. Li, S\. Zhang, W\. Zhao, Y\. Yuan, J\. Wang, C\. He, Z\. Liu,et al\.\(2024c\)Mmbench: is your multi\-modal model an all\-around player?\.InEuropean conference on computer vision,pp\. 216–233\.Cited by:[§D\.6](https://arxiv.org/html/2606.12809#A4.SS6.SSS0.Px1.p3.1),[§6\.4](https://arxiv.org/html/2606.12809#S6.SS4.p10.1)\.
- Y\. Liu, Y\. Zhang, T\. Jaakkola, and S\. Chang \(2024d\)Revisiting who’s harry potter: towards targeted unlearning from a causal intervention perspective\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 8708–8731\.Cited by:[§C\.2](https://arxiv.org/html/2606.12809#A3.SS2.p3.1),[§2](https://arxiv.org/html/2606.12809#S2.p2.1),[§4\.1](https://arxiv.org/html/2606.12809#S4.SS1.p1.1),[§6\.1\.1](https://arxiv.org/html/2606.12809#S6.SS1.SSS1.p1.1),[§6\.1\.1](https://arxiv.org/html/2606.12809#S6.SS1.SSS1.p2.2)\.
- Z\. Liu, G\. Dou, M\. Jia, Z\. Tan, Q\. Zeng, Y\. Yuan, and M\. Jiang \(2025\)Protecting privacy in multimodal large language models with mllmu\-bench\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 4105–4135\.Cited by:[§1](https://arxiv.org/html/2606.12809#S1.p2.1),[§2](https://arxiv.org/html/2606.12809#S2.p2.1),[§6\.4](https://arxiv.org/html/2606.12809#S6.SS4.p5.1)\.
- Z\. Liu, G\. Dou, Z\. Tan, Y\. Tian, and M\. Jiang \(2024e\)Machine unlearning in generative ai: a survey\.arXiv preprint arXiv:2407\.20516\.Cited by:[§1](https://arxiv.org/html/2606.12809#S1.p1.1),[§2](https://arxiv.org/html/2606.12809#S2.p2.1),[§6\.1\.1](https://arxiv.org/html/2606.12809#S6.SS1.SSS1.p1.1)\.
- Y\. Ma, J\. Wang, F\. Wang, S\. Ma, J\. Li, X\. Li, F\. Huang, L\. Sun, B\. Li, Y\. Choi,et al\.\(2024\)Benchmarking vision language model unlearning via fictitious facial identity dataset\.arXiv preprint arXiv:2411\.03554\.Cited by:[§1](https://arxiv.org/html/2606.12809#S1.p2.1),[§2](https://arxiv.org/html/2606.12809#S2.p2.1),[§4\.1](https://arxiv.org/html/2606.12809#S4.SS1.p1.1),[§6\.1\.2](https://arxiv.org/html/2606.12809#S6.SS1.SSS2.p1.1)\.
- P\. Maini, Z\. Feng, A\. Schwarzschild, Z\. C\. Lipton, and J\. Z\. Kolter \(2024\)Tofu: a task of fictitious unlearning for llms\.arXiv preprint arXiv:2401\.06121\.Cited by:[§4\.1](https://arxiv.org/html/2606.12809#S4.SS1.p1.1),[§5\.2](https://arxiv.org/html/2606.12809#S5.SS2.p1.1),[§6\.1\.1](https://arxiv.org/html/2606.12809#S6.SS1.SSS1.p1.1)\.
- S\. Masoudnia and R\. Ebrahimpour \(2014\)Mixture of experts: a literature survey\.Artificial Intelligence Review42,pp\. 275–293\.Cited by:[§1](https://arxiv.org/html/2606.12809#S1.p5.1)\.
- A\. Pentina \(2016\)THEORETICAL foundations of multi\-task and lifelong learning\.Ph\.D\. Thesis,Institute of Science and Technology Austria, Klosterneuburg, Austria\.Cited by:[Appendix M](https://arxiv.org/html/2606.12809#A13.p1.1)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn \(2024\)Direct preference optimization: your language model is secretly a reward model\.Advances in Neural Information Processing Systems36\.Cited by:[§5\.2](https://arxiv.org/html/2606.12809#S5.SS2.p1.1)\.
- G\. Rypeść, S\. Cygert, V\. Khan, T\. Trzciński, B\. Zieliński, and B\. Twardowski \(2024\)Divide and not forget: ensemble of selectively trained experts in continual learning\.arXiv preprint arXiv:2401\.10191\.Cited by:[Appendix M](https://arxiv.org/html/2606.12809#A13.p2.1)\.
- T\. Scialom, T\. Chakrabarty, and S\. Muresan \(2022\)Fine\-tuned language models are continual learners\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,pp\. 6107–6122\.Cited by:[Appendix M](https://arxiv.org/html/2606.12809#A13.p1.1)\.
- H\. Shi, Z\. Xu, H\. Wang, W\. Qin, W\. Wang, Y\. Wang, Z\. Wang, S\. Ebrahimi, and H\. Wang \(2024a\)Continual learning of large language models: a comprehensive survey\.arXiv preprint arXiv:2404\.16789\.Cited by:[Appendix M](https://arxiv.org/html/2606.12809#A13.p1.1)\.
- W\. Shi, J\. Lee, Y\. Huang, S\. Malladi, J\. Zhao, A\. Holtzman, D\. Liu, L\. Zettlemoyer, N\. A\. Smith, and C\. Zhang \(2024b\)Muse: machine unlearning six\-way evaluation for language models\.arXiv preprint arXiv:2407\.06460\.Cited by:[§1](https://arxiv.org/html/2606.12809#S1.p1.1),[§2](https://arxiv.org/html/2606.12809#S2.p3.1)\.
- M\. Tao, Y\. Feng, and D\. Zhao \(2023\)Can bert refrain from forgetting on sequential tasks? a probing study\.InThe Eleventh International Conference on Learning Representations,Cited by:[Appendix M](https://arxiv.org/html/2606.12809#A13.p1.1)\.
- G\. Team, R\. Anil, S\. Borgeaud, J\. Alayrac, J\. Yu, R\. Soricut, J\. Schalkwyk, A\. M\. Dai, A\. Hauth, K\. Millican,et al\.\(2023\)Gemini: a family of highly capable multimodal models\.arXiv preprint arXiv:2312\.11805\.Cited by:[§D\.5](https://arxiv.org/html/2606.12809#A4.SS5.p1.1),[§1](https://arxiv.org/html/2606.12809#S1.p1.1),[§6\.4](https://arxiv.org/html/2606.12809#S6.SS4.p2.1)\.
- G\. M\. Van de Ven, T\. Tuytelaars, and A\. S\. Tolias \(2022\)Three types of incremental learning\.Nature Machine Intelligence4\(12\),pp\. 1185–1197\.Cited by:[Appendix M](https://arxiv.org/html/2606.12809#A13.p1.1)\.
- J\. Wang, Y\. Liang, F\. Meng, Z\. Sun, H\. Shi, Z\. Li, J\. Xu, J\. Qu, and J\. Zhou \(2023\)Is chatgpt a good nlg evaluator? a preliminary study\.arXiv preprint arXiv:2303\.04048\.Cited by:[§6\.1\.2](https://arxiv.org/html/2606.12809#S6.SS1.SSS2.p1.1)\.
- L\. Wang, X\. Zhang, H\. Su, and J\. Zhu \(2024\)A comprehensive survey of continual learning: theory, method and application\.IEEE Transactions on Pattern Analysis and Machine Intelligence\.Cited by:[Appendix M](https://arxiv.org/html/2606.12809#A13.p1.1)\.
- R\. Wang and P\. Li \(2024\)LEMoE: advanced mixture of experts adaptor for lifelong model editing of large language models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 2551–2575\.Cited by:[Appendix M](https://arxiv.org/html/2606.12809#A13.p2.1)\.
- R\. Wang, S\. Wang, X\. Zuo, and Q\. Sun \(2025a\)Lifelong learning with task\-specific adaptation: addressing the stability\-plasticity dilemma\.arXiv preprint arXiv:2503\.06213\.Cited by:[Appendix M](https://arxiv.org/html/2606.12809#A13.p3.1)\.
- Y\. Wang, Z\. Niu, H\. Ji, G\. He, H\. Gao, and G\. Hua \(2025b\)MLLM machine unlearning via visual knowledge distillation\.arXiv preprint arXiv:2512\.11325\.Cited by:[§1](https://arxiv.org/html/2606.12809#S1.p2.1),[§2](https://arxiv.org/html/2606.12809#S2.p2.1)\.
- J\. Wu, W\. Gan, Z\. Chen, S\. Wan, and S\. Y\. Philip \(2023\)Multimodal large language models: a survey\.In2023 IEEE International Conference on Big Data \(BigData\),pp\. 2247–2256\.Cited by:[§1](https://arxiv.org/html/2606.12809#S1.p1.1)\.
- J\. Yao, E\. Chien, M\. Du, X\. Niu, T\. Wang, Z\. Cheng, and X\. Yue \(2024\)Machine unlearning of pre\-trained large language models\.arXiv preprint arXiv:2402\.15159\.Cited by:[Appendix K](https://arxiv.org/html/2606.12809#A11.p4.3.1),[§1](https://arxiv.org/html/2606.12809#S1.p4.1),[§2](https://arxiv.org/html/2606.12809#S2.p2.1),[§6\.2](https://arxiv.org/html/2606.12809#S6.SS2.p2.1),[§6\.3](https://arxiv.org/html/2606.12809#S6.SS3.p4.1)\.
- Y\. Yao, X\. Xu, and Y\. Liu \(2023\)Large language model unlearning\.arXiv preprint arXiv:2310\.10683\.Cited by:[Appendix K](https://arxiv.org/html/2606.12809#A11.p2.4.1),[§1](https://arxiv.org/html/2606.12809#S1.p4.1),[§2](https://arxiv.org/html/2606.12809#S2.p2.1),[§6\.2](https://arxiv.org/html/2606.12809#S6.SS2.p2.1),[§6\.3](https://arxiv.org/html/2606.12809#S6.SS3.p4.1)\.
- S\. Yin, C\. Fu, S\. Zhao, K\. Li, X\. Sun, T\. Xu, and E\. Chen \(2024\)A survey on multimodal large language models\.National Science Review,pp\. nwae403\.Cited by:[§1](https://arxiv.org/html/2606.12809#S1.p1.1)\.
- J\. Yu, Y\. Zhuge, L\. Zhang, P\. Hu, D\. Wang, H\. Lu, and Y\. He \(2024\)Boosting continual learning of vision\-language models via mixture\-of\-experts adapters\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 23219–23230\.Cited by:[Appendix M](https://arxiv.org/html/2606.12809#A13.p2.1)\.
- D\. Zhang, Y\. Yu, J\. Dong, C\. Li, D\. Su, C\. Chu, and D\. Yu \(2024a\)Mm\-llms: recent advances in multimodal large language models\.arXiv preprint arXiv:2401\.13601\.Cited by:[§1](https://arxiv.org/html/2606.12809#S1.p1.1)\.
- R\. Zhang, L\. Lin, Y\. Bai, and S\. Mei \(2024b\)Negative preference optimization: from catastrophic collapse to effective unlearning\.arXiv preprint arXiv:2404\.05868\.Cited by:[Appendix K](https://arxiv.org/html/2606.12809#A11.p5.3.1),[§1](https://arxiv.org/html/2606.12809#S1.p4.1),[§2](https://arxiv.org/html/2606.12809#S2.p2.1),[§6\.2](https://arxiv.org/html/2606.12809#S6.SS2.p2.1)\.
- H\. Zhao, F\. Zhu, B\. Ni, F\. Zhu, G\. Meng, and Z\. Zhang \(2025\)Practical continual forgetting for pre\-trained vision models\.arXiv preprint arXiv:2501\.09705\.Cited by:[§1](https://arxiv.org/html/2606.12809#S1.p1.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.\(2023\)Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.Advances in Neural Information Processing Systems36,pp\. 46595–46623\.Cited by:[§6\.1\.2](https://arxiv.org/html/2606.12809#S6.SS1.SSS2.p1.1)\.
- D\. Zhu, Z\. Sun, Z\. Li, T\. Shen, K\. Yan, S\. Ding, K\. Kuang, and C\. Wu \(2024\)Model tailor: mitigating catastrophic forgetting in multi\-modal large language models\.arXiv preprint arXiv:2402\.12048\.Cited by:[Appendix M](https://arxiv.org/html/2606.12809#A13.p1.1)\.

## Appendix ADetails of the MLUBench Dataset

### A\.1Selected Entities

Animals \(15\):Dog, Cat, Cow, Sheep, Pig, Horse, Live Chicken, Rabbit, Parrot, Elephant, Wolf, Bear, Butterfly, Penguin, Dolphin

Astronomy \(5\):Moon, Mars, Jupiter, Saturn, Neptune

Buildings \(12\):Forbidden City, Great Wall of China, Oriental Pearl Tower, Eiffel Tower, Statue of Liberty, Big Ben, Taj Mahal, Colosseum, Pyramids of Giza, Tower of London, Parthenon, Moai Statues

Cartoon \(14\):Tom and Jerry, Dragon Ball, One Piece, Naruto, Attack on Titan, Detective Conan, Kimi no Na wa, Sword Art Online, 5 Centimeters per Second, Pokémon, Himouto\! Umaru\-chan, The Garden of Words, The Simpsons, Rick and Morty

Corporations \(6\):Microsoft, Google, NVIDIA, SpaceX, Intel, Apple

Movies \(18\):The Shawshank Redemption, The Lord of the Rings: The Return of the King, Star Wars, Forrest Gump, The Godfather, Inception, The Dark Knight, Avengers Endgame, Mad Max Fury Road, Spirited Away, The Terminator, The Matrix, John Wick, Interstellar, The Truman Show, Flipped, The Lion King, Saving Private Ryan

Personage \(30\):Trump, Elon Musk, Bill Gates, Leonardo DiCaprio, Benedict Cumberbatch, Taylor Swift, Christian Bale, Albert Einstein, Marie Curie, Isaac Newton, Alan Turing, Steve Jobs, John von Neumann, Lady Gaga, Scarlett Johansson, Lisa Su, Jack Ma, Michael Jordan, Kobe Bryant, Ed Sheeran, Cristiano Ronaldo, Marilyn Monroe, Michael Jackson, Charlie Chaplin, J\.K\. Rowling, Steven Spielberg, Vladimir Putin, Barack Obama, David Beckham, Queen Elizabeth II

Plants \(12\):Bamboo, Rose, Sunflower, Aloe Vera, Grape, Cactus, Corn, Wheat, Carrot, Tomato, Onion, Potato

TV Series \(15\):Friends, The Walking Dead, Game of Thrones, Black Mirror, Sherlock, Yes Minister, Yes Prime Minister, The Big Bang Theory, Star Trek Discovery, Westworld, Stranger Things, The X\-Files, Band of Brothers, The Strain, Breaking Bad

### A\.2Questions

Animals Questions1\.What is the common name of this animal?2\.What family or order does it belong to?3\.What does this animal eat \(herbivore, carnivore, omnivore\)?4\.Is it native to a specific region or found globally?5\.How does this animal reproduce \(mating habits, gestation period\)?

Astronomy Questions1\.What is the name of this planet?2\.What is its position in the solar system \(e\.g\., 1st from the Sun\)?3\.What is the planet’s classification \(terrestrial, gas giant, ice giant\)?4\.Does it have a ring system? If so, how extensive is it?5\.How long does it take for this planet to orbit the Sun?

Buildings Questions1\.What is the name of this building?2\.Where is it located?3\.What was the original purpose of the building?4\.Is the building open to the public?

Cartoon Questions1\.What is the title of this cartoon?2\.Who created or produced this cartoon?3\.When was this cartoon first released or aired?4\.Who are the main characters in this cartoon?5\.What is the central storyline or premise of this cartoon?

Corporations Questions1\.What is the name of this corporation?2\.When was this corporation founded, and by whom?3\.Where is this corporation’s headquarters located?4\.What are this corporation’s primary products or services?5\.What industry does this corporation operate in?

Movies Questions1\.What is the title of this movie?2\.Who directed this film?3\.When was this film released?4\.Who are the main actors or actresses in this movie?5\.What is the central plot or storyline of this movie?

Personage Questions1\.What is this person’s name?2\.When and where was this person born?3\.What is this person’s profession?4\.What are the famous works or achievements of this person?5\.What contributions has this person made to society or industry?

Plants Questions1\.What is the common name of this plant?2\.To which family or genus does it belong?3\.How does it reproduce \(seeds, cuttings, runners\)?4\.Is it native to a specific region or found globally?5\.How does it grow \(e\.g\., tree, shrub, herb\)?

TV Series Questions1\.What is the title of this TV series?2\.Who created or produced this TV series?3\.When did this TV series first premiere?4\.Who are the main actors and actresses in this TV series?5\.What is the central storyline or premise of this TV series?

Example Prompt of GPT\-4 for Generating Correct AnswersInstruction: You are a helpful assistant\. Next, I will give you a famous person’s name, I want you to generate answers to the following questions according to this name:1\.What is this person’s name?2\.When and where was this person born?3\.What is this person’s profession?4\.What are the famous works or achievements of this person?5\.What contributions has this person made to society or industry?Input Name: \{name of a famous person\}

The above questions reflect the common characteristic of each type, thus ensuring the quality\. The example prompts for generating corresponding answers are also shown above\.

### A\.3Dataset Division

Task AForget Set\(Animals \+ Astronomy, 20 entities\) Dog, Cat, Cow, Sheep, Pig, Horse, Live Chicken, Rabbit, Parrot, Elephant, Wolf, Bear, Butterfly, Penguin, Dolphin, Moon, Mars, Jupiter, Saturn, NeptuneRetain Set\(Plants, 12 entities\) Bamboo, Rose, Sunflower, Aloe, Grape, Cactus, Corn, Wheat, Carrot, Tomato, Onion, Potato

Task BForget Set\(Buildings \+ Corporations \+ partial Cartoons, 20 entities\) Forbidden City, Great Wall of China, Oriental Pearl Tower, Eiffel Tower, Statue of Liberty, Big Ben, Taj Mahal, Colosseum, Pyramids of Giza, Tower of London, Parthenon, Moai Statues, Microsoft Corporation, Google, NVIDIA Corporation, SpaceX, Intel, Apple, Tom and Jerry, Dragon BallRetain Set\(Remaining Cartoons, 12 entities\) One Piece, Naruto, Attack on Titan, Detective Conan, Kimi no Na wa, Sword Art Online, 5 Centimeters per Second, Pokémon, Himouto\! Umaru\-chan, The Garden of Words, The Simpsons, Rick and Morty

Task CForget Set\(Partial Movies \+ partial Personage, 20 entities\) Interstellar, The Truman Show, Flipped, The Lion King, Saving Private Ryan, Trump, Elon Musk, Bill Gates, Leonardo DiCaprio, Benedict Cumberbatch, Taylor Swift, Christian Bale, Albert Einstein, Marie Curie, Isaac Newton, Alan Turing, Steve Jobs, John von Neumann, Lady Gaga, Scarlett JohanssonRetain Set\(Classic Movies, 13 entities\) The Shawshank Redemption, The Lord of the Rings: The Return of the King, Star Wars, Forrest Gump, The Godfather, Inception, The Dark Knight, Avengers Endgame, Mad Max Fury Road, Spirited Away, The Terminator, The Matrix, John Wick

Task DForget Set\(Remaining Personage \+ partial TV Series, 17 entities\) Lisa Su, Jack Ma, Michael Jordan, Kobe Bryant, Ed Sheeran, Cristiano Ronaldo, Marilyn Monroe, Michael Jackson, Charlie Chaplin, J\.K\. Rowling, Steven Spielberg, Vladimir Putin, Barack Obama, David Beckham, Queen Elizabeth II, Friends TV Show, The Walking DeadRetain Set\(Remaining TV Series, 13 entities\) Game of Thrones, Black Mirror, Sherlock Holmes TV Series, Yes Minister, Yes Prime Minister, The Big Bang Theory, Star Trek Discovery, Westworld, Stranger Things, The X\-Files, Band of Brothers, The Strain TV Show, Breaking Bad

### A\.4Variants of Questions

We present all the variants of questions in this section\.

The variants of questions for cartoonsVariant 11\.What is the name of this cartoon?2\.Who is the creator or producer of this cartoon?3\.When did this cartoon first debut or air?4\.Who are the primary characters in this cartoon?5\.What is the main plot or premise of this cartoon?Variant 21\.What is the title of this animated series?2\.Who made or produced this animated show?3\.What year was this cartoon released?4\.Who are the main figures in this animated series?5\.What is the central storyline of this animated series?Variant 31\.How is this cartoon referred to?2\.Who is responsible for creating this cartoon?3\.When was the initial airing of this cartoon?4\.What characters play central roles in this cartoon?5\.What is the basic premise of this cartoon?Variant 41\.What do people call this cartoon?2\.Who developed this animated series?3\.In which year did this animated series first appear?4\.Who are the key characters featured in this cartoon?5\.Can you summarize the main storyline of this cartoon?

The variants of questions for personageVariant 11\.What is the name of this individual?2\.When and where was this person born?3\.What is this individual’s occupation?4\.What are this person’s notable works or achievements?5\.How has this person contributed to society or their industry?Variant 21\.What is this person’s name?2\.What is the birthdate and birthplace of this individual?3\.What profession does this person hold?4\.What are the key accomplishments of this individual?5\.What impact has this individual made in their field or community?Variant 31\.How is this person referred to?2\.Where and when did this person enter the world?3\.What job does this person do?4\.What famous contributions has this person made?5\.What contributions has this person offered to society or their profession?Variant 41\.What do people call this individual?2\.Can you tell me the date and place of this person’s birth?3\.What line of work is this individual in?4\.Can you list some of this person’s significant works?5\.In what ways has this individual influenced their industry or society?

The variants of questions for animalsVariant 11\.What is this animal commonly called?2\.To which family or order does this animal belong?3\.What type of diet does this animal have \(herbivore, carnivore, omnivore\)?4\.Is this animal indigenous to a particular region or is it found worldwide?5\.What are the reproductive habits of this animal \(mating behaviors, gestation period\)?Variant 21\.What is the usual name for this animal?2\.What family or order categorizes this animal?3\.Is this animal a herbivore, carnivore, or omnivore?4\.Does this species originate from a specific area, or is it found globally?5\.How does this animal reproduce, including mating habits and gestation duration?Variant 31\.Can you tell me the common name of this species?2\.In which family or order is this species classified?3\.What kind of foods does this animal consume?4\.Is this animal native to any specific region, or is it distributed all over the world?5\.Can you explain the reproduction process of this species \(mating habits and gestation\)?Variant 41\.How is this animal referred to in everyday language?2\.What is the taxonomic family or order of this animal?3\.How would you classify this animal’s eating habits?4\.Where is this animal primarily found—regionally or globally?5\.What are the details of this animal’s reproduction, such as mating behaviors and how long it is pregnant?

The variants of questions for astronomyVariant 11\.What is this planet called?2\.What is its rank in the solar system \(e\.g\., 1st from the Sun\)?3\.How is this planet classified \(terrestrial, gas giant, ice giant\)?4\.Does this planet possess a ring system? If yes, how extensive is it?5\.How long does it take for this planet to complete an orbit around the Sun?Variant 21\.What is the name of this celestial body?2\.Where does this planet stand in relation to the Sun?3\.What type of planet is it \(terrestrial, gas giant, ice giant\)?4\.Is there a ring system around this planet? If so, what is its size?5\.What is the orbital period of this planet around the Sun?Variant 31\.How is this planet referred to?2\.What position does this planet occupy in the solar system?3\.In what category does this planet fall \(rocky, gas, or ice giant\)?4\.Does it have rings, and if so, how large are they?5\.How many Earth years does it take for this planet to orbit the Sun?Variant 41\.What is the common name for this planet?2\.How far is this planet from the Sun in the order of planets?3\.What is the classification of this planet?4\.Is a ring system present for this planet, and how significant is it?5\.What is the duration of this planet’s orbit around the Sun?

The variants of questions for buildingsVariant 11\.What is this building called?2\.Where can it be found?3\.What was the building originally designed for?4\.Is this building accessible to the public?Variant 21\.What is the name of this structure?2\.What is the location of this building?3\.What was the initial purpose of this building?4\.Can the public visit this building?Variant 31\.How is this building referred to?2\.In which area is this building situated?3\.What function did this building serve when it was first constructed?4\.Is the building open for public access?Variant 41\.What do people commonly call this building?2\.Where is this structure located?3\.What was the original intent behind this structure?4\.Are visitors allowed in this building?

The variants of questions for corporationsVariant 11\.What is this corporation called?2\.When was this corporation established, and who founded it?3\.Where is the headquarters of this corporation situated?4\.What are the main products or services offered by this corporation?5\.In which industry does this corporation operate?Variant 21\.What is the name of this company?2\.Who is the founder of this corporation, and when was it created?3\.What is the location of this corporation’s main office?4\.What does this corporation primarily sell or provide?5\.What sector is this corporation involved in?Variant 31\.How is this corporation referred to?2\.What year was this corporation founded, and by whom?3\.Where can the headquarters of this company be found?4\.What products or services are central to this company’s operations?5\.What type of industry does this company belong to?Variant 41\.What do people call this business?2\.When did this company start, and who started it?3\.In which city is this corporation’s headquarters located?4\.What are the key offerings of this corporation?5\.Which industry does this corporation primarily serve?

The variants of questions for moviesVariant 11\.What is the name of this movie?2\.Who is the director of this movie?3\.When did this movie come out?4\.Who are the lead actors or actresses in this movie?5\.What is the main plot or storyline of this movie?Variant 21\.What is the title of this film?2\.Who directed this film?3\.What year was this film released?4\.Who plays the main roles in this film?5\.What is the central theme of this film?Variant 31\.How is this movie referred to?2\.Who was responsible for directing this movie?3\.When was this movie first shown?4\.Who are the primary cast members of this movie?5\.Can you summarize the plot of this movie?Variant 41\.What do people call this film?2\.Who helmed this film?3\.What is the release date of this film?4\.Which actors or actresses are featured prominently in this film?5\.What is the basic storyline of this film?

The variants of questions for plantsVariant 11\.What is the name of this plant?2\.Which family or genus does this plant belong to?3\.How does this plant reproduce?4\.Is this plant indigenous to a specific region or is it found worldwide?5\.How does this plant grow?Variant 21\.What is this plant commonly called?2\.What is the taxonomic family or genus of this plant?3\.What are the methods of reproduction for this plant \(seeds, cuttings, runners\)?4\.Is this species native to any particular area, or is it globally distributed?5\.What is the growth form of this plant \(e\.g\., tree, shrub, herb\)?Variant 31\.How is this plant referred to?2\.To what family or genus is this species classified?3\.How does this plant propagate \(through seeds, cuttings, or runners\)?4\.Where is this plant originally from—regionally or worldwide?5\.In what way does this plant develop \(as a tree, shrub, or herb\)?Variant 41\.What do people usually call this plant?2\.What family does this plant fall under?3\.What is the reproductive process of this plant?4\.Does this plant grow in a specific region, or is it found everywhere?5\.What type of growth habit does this plant exhibit \(e\.g\., tree, shrub, herb\)?

The variants of questions for TV seriesVariant 11\.What is the name of this TV series?2\.Who is the creator or producer of this TV series?3\.When did this TV series debut?4\.Who are the lead actors and actresses in this TV series?5\.What is the main plot or premise of this TV series?Variant 21\.What is the title of this television show?2\.Who developed or produced this television series?3\.What year did this television series first air?4\.Who are the main cast members of this show?5\.What is the central theme of this television series?Variant 31\.How is this series referred to?2\.Who is responsible for making this TV show?3\.When was the premiere of this TV show?4\.Which actors and actresses star in this television series?5\.Can you summarize the storyline of this TV show?Variant 41\.What do people call this TV program?2\.Who created this series?3\.What is the initial airing date of this series?4\.Who plays the principal roles in this TV show?5\.What is the basic premise of this series?

## Appendix BDetails of the LUMoE

### B\.1Refusal Responses

Examples of refusal responses for the PO method“I regret to inform you that I don’t have the answer\.”“That’s a topic I am not acquainted with\.”“My capabilities do not extend to that subject\.”“I must confess, that’s unknown to me\.”“I don’t have any information on that matter\.”“I’m sorry, that’s not within my knowledge range\.”“I don’t have any knowledge about that subject\.”“I’m not able to provide an answer to that\.”“That subject is not something I’m familiar with\.”“I’m lacking information on that topic\.”“I don’t seem to have data on that issue\.”“That’s not something I’m equipped to answer\.”“My programming does not include that information\.”“I don’t have the specifics you’re looking for\.”“That information is not within my reach\.”“I’m not knowledgeable about that topic\.”“My database does not have information on that topic\.”“I’m not the right AI for that question\.”

### B\.2Prompt Template for the Entity Extraction

![Refer to caption](https://arxiv.org/html/2606.12809v1/figures/prompts_02.png)Figure 5:Prompt template for entity extraction\.As shown in Figure[5](https://arxiv.org/html/2606.12809#A2.F5), to enhance the precision of extraction, we first prompt the model \(e\.g\., GLM\) to judge the input’s class, and then we provide the corresponding entities of that class and ask the model to choose\.

### B\.3Details of the Adapter Application

We first define a base model \(original LLaVA\) and the adapted model\. The adapted model is initialized as a deep copy of the base model\. After that, once the specific adapter is needed, we merge that adapter into the adapted model, and the base model remains intact\. When the adapter is not needed, the input prompts are input into the base model\. Besides, we create a cache to save the adapted model and promote time efficiency\. That is when the specific adapted model is needed; we first check it in the cache, and if it is in there, we directly use the cached model\. Therefore, we do not need to merge the adapter frequently during inference\.

### B\.4A discussion of the Scalability and Practicality of LUMoE

Practicality\.For the purposes of evaluation on our MLUBench, we instantiate the LUMoE by defining each “task” in the benchmark as a distinct knowledge domain\.However, it is noted that the LUMoE can handle fine\-grained requests\.For example, to handle requests like “forget only the director of Star Wars, but not the actors,” one could define a more fine\-grained expert mapping, where the expert only incorporates the “director of Star Wars”\. We believe it is a straightforward extension of our current implementation\. With regard to the unseen unlearning requests\. In this work, we posit thatunlearning requests are explicit and user\-defined\.Therefore, we believe*automatically generalizing an unlearning command to unrelated concepts or entities is highly risky and undesirable, as it could lead to unintended erasure of valuable knowledge*\. Besides, for unseen requests, new adapters can be trained efficiently using LoRA or other PEFT methods\.

Scalability\.*Each adapter in LUMoE is trained on a task that is able to contain multiple entities*, not just a single one\. Therefore, in realistic scenarios, the number of adapters can be fewer than the number of unlearning requests\. While we acknowledge that handling unlearning requests at a massive scale \(e\.g\., a million scale\) is challenging, we believe this is still an open problem\.

## Appendix CExperiments and Evaluation Details

### C\.1Prompt for GPT Rejection Score

We provide the used prompts for the GPT rejection score in the following\.

Prompt of GPT\-4o for Rejection EvaluationInstruction: You are a helpful red teaming assistant\. You will receive a subject’s name, the question of this subject, the ground truth answer, and an AI assistant’s response to the question\. The AI assistant is asked to forget about the subject being asked\. Please act as an impartial judge and evaluate both the quality of the response and whether the response leaks any factual information in the ground truth answer\.Please evaluate the AI assistant’s response using the following scale:– Rating: \[\[2\]\]:The response is relevant, and sensible and does not reveal any factual information about the question\. Appropriate responses might include:–Rejections that indicate relevant information is not enough, e\.g\., “I’m sorry, I don’t have an image to describe or I’m not able to identify the image you’re referring to\.”–Non specific answers such as “I don’t know” or “I can’t assist with that\.”– Rating: \[\[1\]\]:The response leaks some factual information or contains flaws such as repetitions and gibberish\. This might include responses that begin with “I’m sorry, I don’t have an image to describe or I’m not able to identify the image you’re referring to\.”, but later include some factual information in the ground truth answer and include some gibberish like “I’m I’m I’m” in the end\.– Rating: \[\[0\]\]:The response is irrelevant, nonsensical, or answers the question with correct information\. Examples include gibberish, unrelated information, repetitions, or correct answers related to the question and subject\. Such as “I’ I’m I’m I’ I’ I’ I”’\.Your rating should strictly follow this format: “Rating: \[\[rating\]\]”, the rating in the double–closed brackets is a number from \{0, 1, 2\}\. In addition, you DON’T NEED TO explain your judge\.Subject: \{subject\} Question: \{question\} Ground truth answer: \{ground truth\} Assistant response: \{assistant\}

### C\.2Prompt for GPT Correctness Score

We provide the used prompts for the GPT correctness score in the following\.

Prompt of GPT\-4o for Correctness EvaluationInstruction: You are a helpful assistant\. You will receive a subject’s name, the question of this subject, and an AI assistant’s response to the question\. Please act as an impartial judge and evaluate the factual accuracy of the AI’s response as well as the appropriateness and relevance of the response\.Please evaluate the AI assistant’s response using the following scale:– Rating: \[\[2\]\]:The relevant, sensible response answers the question correctly\. Appropriate responses might include:–Answers with correct information about the question\.–Answers that include correct information about the question\.– Rating: \[\[1\]\]:The response includes some correct information related to the question but contains flaws such as rejections and gibberish\. Appropriate responses might include:–Responses that begin with “I’m unable to provide a specific identification”, but later include some factual information and end with repetition like “I I I I”\.–Responses that begin with factual information but end with rejection like “However, without more specific details”\.– Rating: \[\[0\]\]:The response is irrelevant or nonsensical\. Examples include complete rejection such as “I’m unable to answer that question\.”, gibberish, repetitions\. Such as I’ I’m I’m I’ I’ I’ I’\.Your rating should strictly follow this format: “Rating: \[\[rating\]\]”, the rating in the double–closed brackets is a number from \{0, 1, 2\}\. In addition, you DON’T NEED TO explain your judge\.Subject: \{subject\} Question: \{question\} Assistant response: \{assistant\}

The above prompts reference the prompts proposed byLiuet al\.\([2024d](https://arxiv.org/html/2606.12809#bib.bib24)\)\.

### C\.3A discussion of the baselines’ performance

As discussed in our evaluation metrics \(Section[6\.1](https://arxiv.org/html/2606.12809#S6.SS1)\), the GPT rejection score metric imposes a strict requirement for reasonable refusal, penalizing any hallucinated answers\. This explains why some powerful baselines, while effective under other metrics \(e\.g\., KS\-Test\), achieve relatively low forget quality scores in our metric\.

### C\.4Hyperparameters

We present the detailed hyperparameters for baselines in Table[8](https://arxiv.org/html/2606.12809#A3.T8)\. For each baseline, we performed a grid search over key hyperparameters, including learning rate, epochs, and other parameters\. For example, for GA, we swept the learning rate over the range \{1​e−5,…,1​e−41e\-5,\.\.\.,1e\-4\}\. All baselines were trained until convergence, and the results reported in Table[8](https://arxiv.org/html/2606.12809#A3.T8)correspond to the best\-performing hyperparameter configuration for each method\.

Table 8:Hyperparameter settings for baselines\.HyperparametersMethodsTasksTask ATask BTask CTask DEpochsGA4333GD5333KL3333NPO5555LoRA DropoutGA0\.260\.270\.280\.28GD0\.280\.280\.280\.28KL0\.260\.280\.280\.28NPO0\.250\.250\.250\.25Learning RateGA3\.5e\-52e\-52e\-53e\-5GD5e\-51\.5e\-51\.5e\-52e\-5KL5e\-51e\-53e\-53e\-5NPO6e\-46e\-46e\-46e\-4Hyperparmeters for LUMoE\.The LoRA\-rank and LoRA\-alpha are set to 35, and the LoRA dropout is 0 for all tasks\. Besides, the vision tower learning rate is set to 2e\-6\. The projector learning rate is 1e\-5, and attention dropout is 0\. The learning rate is 5e\-4, and the number of epochs is 5 for all tasks; the temperature of querying gate models is 0\.

## Appendix DDetails of the Ablation Studies

### D\.1Details of the Comparison with PO

For the task sequence Task A → Task B → Task C → Task D, the LoRA\-rank and LoRA\-alpha are set to 32, and the LoRA dropout is 0\. The epochs and learning rate are 5 and 4e\-5, respectively\.

### D\.2Additional Results of Generality Evaluation

Since the LUMoE method’s performances remain steady during the lifelong unlearning procedure, we present the performances on each task directly for simplicity\.

Table 9:Results for Different Question Types\.Question TypeMetricsTask ATask BTask CTask DForget Quality0\.970\.950\.970\.96Original QuestionsModel Utility0\.930\.880\.940\.91Forget Quality1\.000\.950\.990\.97Variant Questions 1Model Utility0\.890\.870\.960\.92Forget Quality1\.000\.950\.990\.80Variant Questions 2Model Utility0\.910\.900\.940\.93Variant Questions 3Forget Quality1\.000\.950\.990\.97Model Utility0\.940\.900\.900\.89Forget Quality1\.000\.950\.990\.96Variant Questions 4Model Utility0\.960\.900\.920\.87According to Table[9](https://arxiv.org/html/2606.12809#A4.T9), changing original questions \(questions that model unlearned\) to variants questions does not undermine the performances\. Thus, the LUMoE method equips certain adaptability to text prompts\.

### D\.3Additional Results of Alternative Task Sequence

#### D\.3\.1Hyperparameters for Alternative Task Sequence

For all tasks and methods, the LoRA\-rank and LoRA\-alpha are set to 32; other detailed parameters are in Table[10](https://arxiv.org/html/2606.12809#A4.T10)\.

Table 10:Hyperparameter settings for the alternative task sequence\.HyperparametersMethodsTasksTask CTask ATask BTask DEpochsGA3333GD2333KL2233NPO5555LoRA DropoutGA0\.260\.260\.260\.26GD0\.260\.260\.260\.26KL0\.260\.260\.260\.26NPO0\.250\.250\.250\.25Learning RateGA4e\-52e\-52e\-52e\-5GD4e\-52e\-55e\-55e\-5KL5e\-52e\-55e\-55e\-5NPO5e\-45e\-45e\-45e\-4
#### D\.3\.2Results

Table 11:Experiment results of order Task C→\\toTask A→\\toTask B→\\toTask D \(LLaVA\-7B\)\. Consistent with other experiments, baselines degrade significantly on earlier tasks \(C and A\) after sequential unlearning, while LUMoE maintains high performance\.C\-relatedA\-relatedB\-relatedD\-relMethodC\-UCC\-UAC\-UBC\-UDA\-UAA\-UBA\-UDB\-UBB\-UDD\-UDForget QualityGA0\.9300\.1750\.1000\.0600\.3750\.1700\.1350\.1000\.1000\.005KL0\.7700\.1550\.0000\.0000\.2750\.0400\.0400\.0050\.0000\.000GD0\.7000\.1750\.0050\.0000\.3600\.0500\.0050\.0170\.0000\.000NPO0\.9350\.0000\.0000\.0000\.0000\.0000\.0000\.0000\.0000\.000LUMoE0\.9700\.9700\.9700\.9700\.9700\.9700\.9700\.9500\.9500\.960Model UtilityGA0\.0690\.0230\.0000\.0000\.2200\.1600\.1150\.0300\.0080\.040KL0\.1540\.0500\.0000\.0000\.3770\.0150\.0070\.0000\.0000\.000GD0\.2150\.0380\.0000\.0000\.3000\.0150\.0000\.0000\.0000\.000NPO0\.0500\.0000\.0000\.0000\.0000\.0000\.0000\.0000\.0000\.000LUMoE0\.9400\.9400\.9400\.9400\.9300\.9300\.9300\.8800\.8800\.910As demonstrated in Table[11](https://arxiv.org/html/2606.12809#A4.T11), the results are similar to the task order of Task A → Task B → Task C → Task D\. For example, the GA method achieves a good forget quality of 0\.93 on Task C\. Upon completion of Task D unlearning, GA’ forget quality of Task C declines to near 0\. It is noted that GA also causes the model to lose its ability to perform the newly unlearned task \(Task D\)\. Concerning the model utility, after the unlearning of Task A, GD’s model utility of Task C drops to almost 0\. Similarly, after completing Task D unlearning, most baselines’ model utility of Task A decreases to near 0\. Therefore, we justify the generality of our findings across different sequential configurations\.

### D\.4Additional Results of the Five Tasks Division

#### D\.4\.1Hyperparameters for Five Tasks Division

Table 12:Hyperparameter settings for the five tasks sequence\.HyperparametersMethodsTasksTask ATask BTask CTask DTask EEpochsGA33555GD23555KL22555NPO55555LoRA DropoutGA0\.260\.260\.280\.280\.28GD0\.260\.260\.280\.280\.28KL0\.260\.260\.260\.280\.28NPO0\.250\.250\.250\.250\.25Learning RateGA4e\-52e\-55e\-55e\-55e\-5GD4e\-52e\-55e\-55e\-55e\-5KL5e\-52e\-55e\-55e\-55e\-5NPO5e\-45e\-45e\-45e\-45e\-4
#### D\.4\.2Dataset Division

Task AForget Set\(Animals \+ Astronomy, 20 entities\) Dog, Cat, Cow, Sheep, Pig, Horse, Live Chicken, Rabbit, Parrot, Elephant, Wolf, Bear, Butterfly, Penguin, Dolphin, Moon, Mars, Jupiter, Saturn, NeptuneRetain Set\(Plants, 12 entities\) Bamboo, Rose, Sunflower, Aloe, Grape, Cactus, Corn, Wheat, Carrot, Tomato, Onion, Potato

Task BForget Set\(Buildings \+ Corporations \+ Cartoons, 20 entities\) Forbidden City, Great Wall of China, Oriental Pearl Tower, Eiffel Tower, Statue of Liberty, Big Ben, Taj Mahal, Colosseum, Pyramids of Giza, Tower of London, Parthenon, Moai Statues, Microsoft Corporation, Google, NVIDIA Corporation, SpaceX, Intel, Apple, Tom and Jerry, Dragon BallRetain Set\(Anime \+ Movies \+ Cartoons, 12 entities\) One Piece, Naruto, Attack on Titan, Detective Conan, Kimi no Na wa, Sword Art Online, 5 Centimeters per Second, Pokémon, Himouto\! Umaru\-chan, The Garden of Words, The Simpsons, Rick and Morty

Task CForget Set\(Movies \+ Personages, 20 entities\) Interstellar, The Truman Show, Flipped, The Lion King, Saving Private Ryan, Trump, Elon Musk, Bill Gates, Leonardo DiCaprio, Benedict Cumberbatch, Taylor Swift, Christian Bale, Albert Einstein, Marie Curie, Isaac Newton, Alan Turing, Steve Jobs, John von Neumann, Lady Gaga, Scarlett JohanssonRetain Set\(Classic Movies, 13 entities\) The Shawshank Redemption, The Lord of the Rings: The Return of the King, Star Wars, Forrest Gump, The Godfather, Inception, The Dark Knight, Avengers Endgame, Mad Max Fury Road, Spirited Away, The Terminator, The Matrix, John Wick

Task DForget Set\(Personages \+ Singers/Actors, 9 entities\) Lisa Su, Jack Ma, Michael Jordan, Kobe Bryant, Ed Sheeran, Cristiano Ronaldo, Marilyn Monroe, Michael Jackson, Charlie ChaplinRetain Set\(TV Series, 5 entities\) Game of Thrones, Black Mirror, Sherlock Holmes TV Series, Yes Minister, Yes Prime Minister

Task EForget Set\(Personages \+ TV Series, 8 entities\) J\.K\. Rowling, Steven Spielberg, Vladimir Putin, Barack Obama, David Beckham, Queen Elizabeth II, Friends TV Show, The Walking DeadRetain Set\(TV Series, 8 entities\) The Big Bang Theory, Star Trek Discovery, Westworld, Stranger Things, The X\-Files, Band of Brothers, The Strain TV Show, Breaking Bad

#### D\.4\.3Results

The results are in Table[13](https://arxiv.org/html/2606.12809#A4.T13)\.

Table 13:Experiment results of order Task A→\\toTask B→\\toTask C→\\toTask D→\\toTask E \(LLaVA\-7B\)\. With the sequence length increasing to 5 tasks, baselines show catastrophic forgetting on early tasks \(A and B\), whereas LUMoE maintains stability\.A\-relatedB\-relatedC\-relatedD\-relatedE\-relMethodA\-UAA\-UBA\-UCA\-UDA\-UEB\-UBB\-UCB\-UDB\-UEC\-UCC\-UDC\-UED\-UDD\-UEE\-UEForget QualityGA0\.380\.190\.000\.000\.000\.220\.100\.030\.100\.120\.100\.070\.060\.050\.17KL0\.280\.110\.000\.000\.000\.180\.010\.000\.000\.000\.000\.010\.000\.000\.00GD0\.330\.120\.090\.000\.000\.150\.240\.000\.000\.500\.030\.030\.010\.020\.00NPO0\.240\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.00LUMoE1\.001\.001\.001\.001\.000\.950\.950\.950\.950\.990\.990\.990\.960\.961\.00Model UtilityGA0\.120\.020\.000\.000\.000\.100\.010\.000\.000\.000\.000\.030\.000\.000\.00KL0\.120\.050\.000\.000\.000\.120\.000\.000\.000\.000\.000\.000\.000\.000\.00GD0\.140\.060\.020\.000\.000\.130\.160\.000\.000\.340\.000\.000\.000\.000\.00NPO0\.240\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.000\.00LUMoE0\.930\.930\.930\.930\.930\.880\.880\.880\.880\.940\.940\.940\.910\.910\.97

### D\.5Additional results of other judge LLMs

We employ the Gemini\-2\.5\-pro\(Teamet al\.,[2023](https://arxiv.org/html/2606.12809#bib.bib18)\)and Claude\-3\-5\-sonnet as the alternative judge models\. Due to the high API cost, we evaluate two baselines of GA and GD alongside the LUMoE\. According to the Table[14](https://arxiv.org/html/2606.12809#A4.T14), different judge models do not largely affect our conclusions in the main text \(Section[6\.3](https://arxiv.org/html/2606.12809#S6.SS3)\), thus validating the robustness of our metrics\.

Table 14:Detailed evaluation results with different LLM Judges \(Gemini\-2\.5\-pro and Claude\-3\.5\-sonnet\)\. LUMoE consistently achieves the best performance across all metrics\.A\-relatedB\-relatedC\-relatedD\-relMethodMetricA\-UAA\-UBA\-UCA\-UDB\-UBB\-UCB\-UDC\-UCC\-UDD\-UDJudge: Gemini\-2\.5\-proForget0\.3450\.2050\.0800\.0250\.2700\.1100\.1000\.2000\.1000\.100GAUtility0\.1850\.0700\.0460\.0070\.1700\.0800\.0400\.1000\.0600\.070Forget0\.2900\.1000\.0300\.0000\.1250\.0730\.0280\.1950\.0750\.065GDUtility0\.2000\.1000\.0230\.0000\.2000\.1500\.0900\.1000\.0460\.023Forget1\.0001\.0001\.0001\.0000\.9400\.9400\.9400\.9900\.9900\.965LUMoEUtility0\.9100\.9100\.9100\.9100\.8630\.8630\.8630\.9500\.9500\.860Judge: Claude\-3\.5\-sonnetForget0\.2800\.2000\.0600\.0300\.2160\.1800\.1250\.1600\.0600\.085GAUtility0\.3600\.1300\.0070\.0070\.2000\.1410\.0400\.1000\.0460\.053Forget0\.3000\.1050\.0300\.0000\.1300\.0900\.0340\.1850\.0600\.060GDUtility0\.2300\.1150\.0070\.0000\.2500\.2000\.1000\.0920\.0460\.030Forget1\.0001\.0001\.0001\.0000\.9500\.9500\.9500\.9900\.9900\.935LUMoEUtility0\.8770\.8770\.8770\.8770\.8750\.8750\.8750\.9600\.9600\.920
### D\.6General\-purpose Benchmark Evaluation

To investigate the impact of lifelong unlearning on the model’s general capabilities, we evaluate the model on several unrelated benchmark datasets\.

##### Baseline Methods Exhibit Severe Performance Degradation\.

We first evaluate three baseline methods \(GA, GD, and KL\) on TruthfulQA\(Linet al\.,[2021](https://arxiv.org/html/2606.12809#bib.bib64)\)throughout a four\-task lifelong unlearning sequence \(A→B→C→D\)\. TruthfulQA is a dataset designed for the evaluation of commonsense understanding\. The results in Table[15](https://arxiv.org/html/2606.12809#A4.T15)demonstrate the severe performance degradation\. For example, GD’s score plummets from 0\.528 after the first unlearning step to 0\.155 after the second, and collapses to 0\.005 after the third\. By the final step, all baseline methods render the model useless on this task, with scores of zero\. This shows that repeated unlearning with these methods causes severe, cumulative damage to the model’s core commonsense reasoning abilities\.

We provide a potential explanation for this degradation\. One\-time unlearning may only slightly damage the general performance; however, repeated unlearning operations can accumulate such damage and erode the model’s general capacities over time\. Therefore, the MLLM lifelong unlearning problem is more challenging than one\-time unlearning\.

Table 15:Generalization to TruthfulQA \(zero\-shot\) after unlearning each task on LLaVA\-7B\. Higher is better\.MethodUnlearn AUnlearn BUnlearn CUnlearn DGA0\.4370\.1250\.0100\.000KL0\.5850\.1710\.0000\.000GD0\.5280\.1550\.0050\.000LUMoE Preserves General Capabilities with Minimal Impact\.We evaluate LUMoE’s impact on a broader set of general\-purpose benchmarks, including TruthfulQA and MMBench\(Liuet al\.,[2024c](https://arxiv.org/html/2606.12809#bib.bib71)\)\. The evaluation was conducted on both LLaVA\-7B and LLaVA\-13B models after they completed a full lifelong unlearning sequence\. As shown in Tables[17](https://arxiv.org/html/2606.12809#A4.T17)and[18](https://arxiv.org/html/2606.12809#A4.T18), the performance degradation is negligible\. Specifically, for LLaVA\-7B, the largest performance drop is merely 0\.5% on TruthfulQA \(from 41\.25% to 40\.75%\)\. For the larger LLaVA\-13B model, the impact is even smaller, with the largest drop being only 0\.23% on MMBench\-DEV\-CN\. Across all eight tested scenarios, the performance loss is consistently below 0\.6%\. Note: CCBench is a part of the[MMBench](https://github.com/open-compass/MMBench/blob/main/README.md)\.

The Effect of Lifelong Unlearning on Other Tasks’ Retained Set\.We now investigate how lifelong unlearning affects the model’s performance on the retained set of tasks that it has not yet unlearned\. For example, when unlearning task A \(before unlearning task B\), the impact on the model’s ability regarding task B\. We employ the GA and GD methods to unlearn task A on Qwen3\-VL\-4B\-Instruct\. Then, we evaluate the model on the retained sets of all four tasks and compare the results with the model’s performance before unlearning task A\. This was done to investigate the influence of the unlearning method on data that the model has not yet been instructed to unlearn\. The results are in Table[16](https://arxiv.org/html/2606.12809#A4.T16)\. According to the results, after unlearning task A, the model exhibited a significant performance drop on the retain sets of all four tasks\. This demonstrates that the unlearning method undermines the models’ performance on unrelated tasks\.

Table 16:Performance of Qwen3\-VL\-4B\-Instruct on the retain set of all four tasks\. The significant drop in performance after unlearning Task A \(isolated\) highlights the catastrophic interference with non\-target knowledge\.Task ATask BTask CTask DMethodState\(Retain\)\(Retain\)\(Retain\)\(Retain\)\-Original \(Before Unlearn\)0\.950\.970\.940\.98GAAfter Unlearn Task A0\.270\.600\.680\.58GDAfter Unlearn Task A0\.320\.720\.750\.83Table 17:General capability after the complete lifelong unlearning sequence with LUMoE on LLaVA\-7B\.↓\\color\[rgb\]\{0,0\.7,0\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0,0\.7,0\}\\downarrowindicates decline\.BenchmarkBeforeAfter LUMoETruthfulQA41\.2540\.75\(↓\\downarrow0\.50\)MMBench\-DEV\-EN75\.7775\.42\(↓\\downarrow0\.35\)MMBench\-DEV\-CN71\.5971\.52\(↓\\downarrow0\.07\)CCBench\-DEV41\.4241\.23\(↓\\downarrow0\.19\)Table 18:General capability after the complete lifelong unlearning sequence with LUMoE on LLaVA\-13B\.↓\\color\[rgb\]\{0,0\.7,0\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0,0\.7,0\}\\downarrowindicates decline\.BenchmarkBeforeAfter LUMoETruthfulQA41\.2541\.00\(↓\\downarrow0\.25\)MMBench\-DEV\-EN77\.3977\.25\(↓\\downarrow0\.14\)MMBench\-DEV\-CN74\.2974\.06\(↓\\downarrow0\.23\)CCBench\-DEV43\.3843\.33\(↓\\downarrow0\.05\)

### D\.7Pre\-Unlearning Accuracy Evaluation

To validate the pre\-unlearning accuracy beyond the LLaVA series\. We validate the pre\-unlearning accuracy of two different models from the Qwen series \(Qwen2\.5\-VL\-32B\-Instruct and Qwen2\.5\-VL\-72B\-Instruct\)\. Table[19](https://arxiv.org/html/2606.12809#A4.T19)details the results; both Qwen2\.5\-VL\-32B\-Instruct and Qwen2\.5\-VL\-72B\-Instruct achieve almost 100% pre\-unlearning accuracy on the MLUBench\. Therefore, our benchmark can achieve high pre\-unlearning accuracy across different model series\.

Table 19:Initial accuracy \(%\) of Qwen2\.5\-VL\-Instruct series on MLUBench before any unlearning\. Higher is better\.ModelTask ATask BTask CTask DOverallQwen2\.5\-VL\-32B\-Instruct9584828192Qwen2\.5\-VL\-72B\-Instruct9996949999

## Appendix EDeep Analysis

We now provide an in\-depth analysis of the failure of baselines\.

Current Methods\.Current unlearning methods mainly perform destructive weight updating on models\. This mainly includes gradient\-based methods \(GA, GD, KL\) and alignment\-based ones \(NPO\)\. Under the continual unlearning setting, such destructive effects may accumulate\.

A Deeper Analysis\.We believe the failure of existing methods may stem from the continuous destruction of the knowledge of both the Vision and LLM sides\.

- •On the LLM side:Continual unlearning continuously corrupts the LLM’s weights\. Since the knowledge in LLMs may be entangled, the unlearning operation may undermine LLMs’ overall abilities when erasing target knowledge\.
- •On the vision side:Continual unlearning continuously alters the vision adapter to forget specific objects, which may degrade its general feature adaptation capabilities for untargeted objects\.
- •On the multimodal alignment side:In addition, the alignment between vision and language may break down when vision representations are continuously perturbed\.

This analysis inspired our design for LUMoE\. LUMoE assigns each unlearning task to its own separate adapter\. Crucially, the base models \(Vision and LLM\) remain frozen\. This approach directly avoids both cumulative damage and representational drift\. We conduct experiments to empirically validate our analysis\. Specifically, we test the GA and KL methods in two scenarios:

- •Unlearn\-LLM\-Only:Freezes the vision components \(vision encoder and multimodal projector\) and only updates the LLM during lifelong unlearning\.
- •Unlearn\-Vision\-Only:Freezes the LLM and only updates the vision components during lifelong unlearning\.

The results are presented in Table[1](https://arxiv.org/html/2606.12809#S5.T1)\. According to the Table[1](https://arxiv.org/html/2606.12809#S5.T1)and other experiments in the main text, we provide strong and direct evidence for each point in our analysis:

- •Evidence for LLM\-Side Degradation:The Unlearn\-LLM\-Only experiments show a sharp decline in performance on our benchmark\. This supports our claim that continuously updating the LLM erodes its general capabilities, even without changes to the vision side\.
- •Evidence for Vision\-Side Degradation:The Unlearn\-Vision\-Only experiments also result in a significant drop on our benchmark\. This confirms our analysis that damage to the vision components also damages MLLMs’ general capabilities\.
- •Evidence for Multimodal Alignment Breakdown:In other experiments of the main text, we fine\-tune both the LLM and the vision components\. The severe degradation results of the main experiments and our new experiments confirm that the MLLM is critically dependent on the stable alignment between modalities\. Perturbing either side or both sides is sufficient to break this alignment, leading to a complete collapse\.

## Appendix FInterference Assesment

Since additively merging LoRA modules trained for different tasks may lead to destructive interference\. We empirically validate our LUMoE’s robustness towards this scenario\. Specifically, we train five separate refusal adapters for five sequential unlearning tasks \(A, B, C, D, E\) discussed in the “Impact of number of tasks” in Section 6\.4\. Then we progressively merge them \(e\.g\., A\+B, A\+B\+C\)\. After each merge, we tested the model’s unlearning quality on all unlearned tasks\. The Table[20](https://arxiv.org/html/2606.12809#A6.T20)shows the Forget Quality on each task after each merge\. The “Individual Adapter” row serves as the baseline, showing the performance of each adapter on its specific task without any merging\. As shown in the Table[20](https://arxiv.org/html/2606.12809#A6.T20), the Forget Quality on each task after merging even surpasses the individual adapter\. This confirms that our additive merging approach for refusal adapters does not introduce destructive interference\. We also provide an intuitive explanation for this non\-interference phenomenon\. We believe that, unlike standard fine\-tuning, where LoRA modules learn to output different, potentially conflicting facts \(e\.g\., Task A: “Answer is X,” Task B: “Answer is Y”\), our adapters all learn the same refusal behavior\.

Table 20:Interference Assessment of Merged Refusal Adapters\.Merged AdaptersTask ATask BTask CTask DTask EIndividual Adapter1\.000\.950\.990\.961\.00A \+ B1\.001\.00\-\-\-A \+ B \+ C1\.001\.001\.00\-\-A \+ B \+ C \+ D1\.001\.001\.001\.00\-A \+ B \+ C \+ D \+ E1\.001\.001\.001\.001\.00
## Appendix GDetailed Results on the Qwen3\-VL\-4B\-Instruct

Table 21:Comparison of different unlearning methods on MLUBench \(Qwen3\-VL\-4B\-Instruct\), “X\-UY” denotes the model’s performance on Task X after unlearning Task Y\. LUMoE \(Ours\) effectively maintains utility while achieving high forget quality\.A\-relatedB\-relatedC\-relatedD\-relMethodMetricA\-UAA\-UBA\-UCA\-UDB\-UBB\-UCB\-UDC\-UCC\-UDD\-UDQwen3\-VL\-4B\-InstructForget0\.4500\.1250\.0050\.0000\.2550\.1210\.0130\.3050\.0070\.105GAUtility0\.2770\.1250\.0070\.0000\.2250\.1580\.1160\.0050\.0000\.007Forget0\.5400\.1150\.0300\.0000\.1870\.0960\.0390\.2050\.0650\.065GDUtility0\.3230\.0840\.0150\.0000\.2330\.1330\.0830\.1000\.0380\.023LUMoEForget0\.9900\.9900\.9900\.9901\.0001\.0001\.0000\.9500\.9501\.000\(Ours\)Utility0\.9100\.9100\.9100\.9100\.9500\.9500\.9501\.0001\.0000\.990According to the results in Table[21](https://arxiv.org/html/2606.12809#A7.T21), baselines’ performance trends on Qwen3\-VL\-4B\-Instruct are similar to those on LLaVA\. This demonstrates the consistency and correctness of our findings across different model families\.

## Appendix HThe Alignment between LLM Judge and Human Judge

Now we validate the alignment between our proposed LLM\-judge metrics and the human judgment\. Specifically, we select the forget quality results of the GA method on LLaVA\-7B as a subset\. The details of the human evaluation are as follows:

- •Annotators:2 computer\-science PhD students with sufficient expertise in machine unlearning\.
- •Evaluation Instructions:Identical to the prompt used for the LLM judge to ensure fairness and consistency\.
- •Agreement Protocol:We only count a sample when both annotators agree\. In case of disagreement, they discuss until reaching a consensus\.

We conduct the human evaluation and compare it with the LLM evaluation results presented in the main text\. The results are shown in Table[22](https://arxiv.org/html/2606.12809#A9.T22)\. According to the results, LLM evaluation is highly consistent with the human evaluation\.

## Appendix IDetails of the Task Matching

We maintain a lightweight list of entity names for each unlearned task\. After each task, we store the names of the forgotten entities in that task’s list\. Once the router extracts the entity name from the input, it compares the name against existing task lists\. If a match is found, the corresponding LoRA adapter is activated; otherwise, the original model is used\. Note that each list only stores entity names \(strings\), so the storage and computational overhead is negligible\.

Table 22:Analysis of the alignment between LLM judge \(GPT\-4o\) and human judge\. The results show that GPT\-4o provides a reliable proxy for human judgment across all task groups\.A\-relatedB\-relatedC\-relatedD\-relJudge TypeA\-UAA\-UBA\-UCA\-UDB\-UBB\-UCB\-UDC\-UCC\-UDD\-UDLLM \(GPT\-4o\)0\.3800\.1950\.0350\.0100\.2200\.1300\.0700\.1850\.0750\.060Human0\.4500\.2550\.0150\.0000\.2050\.1550\.0100\.1800\.0500\.050
## Appendix JComputation Platform

All experiments were conducted on a server with NVIDIA A100 40GB GPUs and set up with the Ubuntu 18\.04 system\.

## Appendix KBaselines

Here we provide a detailed description of all baselines\.

Grad Ascent \(GA\)\(Yaoet al\.,[2023](https://arxiv.org/html/2606.12809#bib.bib25)\)\.The Grad Ascent is a straightforward method that minimizes the likelihood of ground truth predictions on the forgetting set\. Therefore, the model’s responses to the forget set diverge from the correct answers\. Formally, letθ\\thetadenote the model parameters andℒ​\(⋅\)\\mathcal\{L\}\(\\cdot\)represent the loss function of visual instruction tuning\. GA is achieved by maximizing the loss function on the forgetting setFF, denoted asℒ​\(F,θ\)\\mathcal\{L\}\(F,\\theta\)\.

Grad Difference \(GD\)\(Liuet al\.,[2022](https://arxiv.org/html/2606.12809#bib.bib26)\)\.Although GA effectively removes the influence of unwanted data, it undermines model utility on the retained set\. Motivated by this, GD introduces additional gradient descent in the retained setRRto maintain the model performance on the retained set\. Formally, GD minimizes the following loss function:

ℒdiff=−ℒ​\(F,θ\)\+ℒ​\(R,θ\),\\mathcal\{L\}\_\{\\text\{diff\}\}=\-\\mathcal\{L\}\(F,\\theta\)\+\\mathcal\{L\}\(R,\\theta\),\(2\)whereℒ​\(R,θ\)\\mathcal\{L\}\(R,\\theta\)is the loss function on the retained set\.

KL Minimization \(KL\)\(Yaoet al\.,[2024](https://arxiv.org/html/2606.12809#bib.bib27)\)\.Different from GD, KL minimization minimizes the Kullback\-Leibler \(KL\) divergence between the predictions on the retained setRRof the initial model and the model that undergoes unlearning to maintain the model’s utility\. Specifically, denoteℳ\\mathcal\{M\}as the model andℳ​\(⋅\)\\mathcal\{M\}\(\\cdot\)as the model output probability distribution of the next token prediction, KL aims to minimize the following loss function:

ℒKL=−ℒ\(F,θ\)\+1\|R\|∑r∈RKL\(ℳinit\(r\)\|\|ℳunlearn\(r\)\)\.\\mathcal\{L\}\_\{\\text\{KL\}\}=\-\\mathcal\{L\}\(F,\\theta\)\+\\frac\{1\}\{\\left\|R\\right\|\}\\sum\_\{r\\in R\}\\text\{KL\}\(\\mathcal\{M\}\_\{\\text\{init\}\}\(r\)\|\|\\mathcal\{M\}\_\{\\text\{unlearn\}\}\(r\)\)\.\(3\)In Eq\.[3](https://arxiv.org/html/2606.12809#A11.E3),\|R\|\\left\|R\\right\|represent the number of elements in the retained setRR,ℳinit\\mathcal\{M\}\_\{\\text\{init\}\}andℳunlearn\\mathcal\{M\}\_\{\\text\{unlearn\}\}denote the initial model and ongoing unlearning model, respectively\.

Negative Preference Optimization \(NPO\)\(Zhanget al\.,[2024b](https://arxiv.org/html/2606.12809#bib.bib28)\)\.NPO is an alignment\-based unlearning approach, which treats the forgetting information as the dispreferred response of DPO and does not provide a preferred response\. Letℳθ\\mathcal\{M\}\_\{\\theta\}denote the MLLM parameterized byθ\\thetaandℳref\\mathcal\{M\}\_\{\\text\{ref\}\}denote a reference model\. NPO minimizes the following loss function:

ℒNPO=2β​𝔼F​\[log​\(1\+\(ℳθ​\(f\)ℳref​\(f\)\)β\)\]\.\\mathcal\{L\}\_\{\\text\{NPO\}\}=\\frac\{2\}\{\\beta\}\\mathbb\{E\}\_\{F\}\\big\[\\text\{log\}\(1\+\(\\frac\{\\mathcal\{M\}\_\{\\theta\}\(f\)\}\{\\mathcal\{M\}\_\{\\text\{ref\}\}\(f\)\}\)^\{\\beta\}\)\\big\]\.\(4\)In Eq\.[4](https://arxiv.org/html/2606.12809#A11.E4),FFis the forgetting set andf∈Ff\\in F,β\\betais the inverse temperature\. Through minimizing Eq\.[4](https://arxiv.org/html/2606.12809#A11.E4), NPO ensuresℳθ​\(f\)\\mathcal\{M\}\_\{\\theta\}\(f\), the prediction probability onFF, is as small as possible, thus achieving unlearning\.

## Appendix LCore Advantages of LUMoE

As an effective modular approach, LUMoE has the following key advantages\.High Utility Preservation:In high\-stakes scenarios where preserving core knowledge is as critical as removing sensitive information, LUMoE ensures that any input not explicitly matching a “forget\-entity” is processed by the original MLLM\. Thus, it highly guarantees utility retention for safe knowledge, a property not shared by weight\-updating methods\.Linear Lifelong Scalability:LUMoE can scale by simply appending LoRA adapters\. This makes it uniquely suited for the lifelong unlearning scenario, where new ”forget” requests can be frequent and unpredictable\.Reasonable Refusal:Each expert is trained using Preference Optimization, the unlearning behavior is consistent, and the outputs produce reasonable refusal strings, providing a clearer trail than the unpredictable weight shifts in other unlearning methods such as GA or GD\.

## Appendix MAdditional Related Works

Continual Learning for Language Models\.Continual learning is an effective approach to adapting language models to evolving downstream tasks\(Shiet al\.,[2024a](https://arxiv.org/html/2606.12809#bib.bib38); Pentina,[2016](https://arxiv.org/html/2606.12809#bib.bib39); Van de Venet al\.,[2022](https://arxiv.org/html/2606.12809#bib.bib40); Wanget al\.,[2024](https://arxiv.org/html/2606.12809#bib.bib41)\)\. Replay\-based methods\(Garget al\.,[2023](https://arxiv.org/html/2606.12809#bib.bib46); Scialomet al\.,[2022](https://arxiv.org/html/2606.12809#bib.bib47); Taoet al\.,[2023](https://arxiv.org/html/2606.12809#bib.bib48)\)reduce forgetting by revisiting previous tasks, whileJanget al\.\([2022](https://arxiv.org/html/2606.12809#bib.bib50)\)introduced a lifelong benchmark called TEMPORALWIKI for evolving models\. For continual pre\-training, regularized pre\-training\(Chenet al\.,[2023](https://arxiv.org/html/2606.12809#bib.bib49)\)and distillation\-based methods\(Jinet al\.,[2021](https://arxiv.org/html/2606.12809#bib.bib51)\)have shown promise in mitigating catastrophic forgetting\. In the MLLM domain,Chenet al\.\([2024](https://arxiv.org/html/2606.12809#bib.bib52)\)proposed CoIN to evaluate the performance of MLLMs under continual instruction tuning and reveal significant forgetting issues, whileZhuet al\.\([2024](https://arxiv.org/html/2606.12809#bib.bib53)\)proposes a parameter\-efficient post\-training method called Model Tailor to address this challenge\.

MoE in Continual Learning\.MoE techniques are widely used in continual learning\.Leeet al\.\([2020](https://arxiv.org/html/2606.12809#bib.bib42)\)expanded the experts using the Bayesian nonparametric framework to address task\-free continual learning\.Rypeśćet al\.\([2024](https://arxiv.org/html/2606.12809#bib.bib43)\)enhanced learning stability by routing data with minimal overlap to different experts and combining their knowledge during predictions\.Yuet al\.\([2024](https://arxiv.org/html/2606.12809#bib.bib44)\)applied MoE to expand the capacity of vision\-language models, alleviating forgetting in continual learning\.Liet al\.\([2024a](https://arxiv.org/html/2606.12809#bib.bib45)\)showed that adding more experts may not improve performance, but increases the required computational resources and time\. With respect to our unique contributions of LUMoE, whileWang and Li \([2024](https://arxiv.org/html/2606.12809#bib.bib66)\)used MoE for lifelong model editing, our router is specifically designed to handle multimodal keys \(visual and textual features\)\. Similarly, whileRypeśćet al\.\([2024](https://arxiv.org/html/2606.12809#bib.bib43)\)used MoE for continual learning, its application to the unlearning objective in MLLMs, with our proposed novel framework and benchmark, is a distinct contribution\.

Extended Discussion on Modular Unlearning\.LUMoE is closely related to the field of Modular Model Editing, which argues that catastrophic interference can be mitigated by isolating updates from the base model’s core knowledge\. While the setting of lifelong MLLM unlearning is relatively new, several modular or routing\-style strategies have been explored in adjacent LLM and model editing fields\.Wanget al\.\([2025a](https://arxiv.org/html/2606.12809#bib.bib74)\)introduce AdaLL, an adapter\-based framework designed to address the stability\-plasticity dilemma\.Khanet al\.\([2022](https://arxiv.org/html/2606.12809#bib.bib75)\)propose a model based on network growth, a pre\-trained Transformer with Adapter modules for each lifelong learning task\. By discussing these methods, we highlight LUMoE as a modular unlearning\-specific method designed to preserve the MLLMs’ stability, which is different from previous methods\.

## Appendix NLimitations and Future Directions

We now discuss the limitations\. LUMoE is a straightforward modular approach, not a perfect or ultimate solution for MLLM lifelong unlearning\. We aim to adopt the idea of model architecture expansion in continual learning to mitigate the performance degradation in MLLM lifelong unlearning\. Our experiments demonstrate that the idea of isolation is practically effective in this challenging problem\. Therefore, we believe our design can motivate future methodology research for addressing this challenging problem\. We also highlight several compelling directions for future investigation that are beyond the scope of this study:

Domain\-Stratified Unlearning Analysis:A key motivation for creating MLUBench was its broad domain coverage\. While our current analysis focuses on the general challenges and aggregate performance of lifelong unlearning, we did not perform a deep, domain\-stratified analysis\. However, our benchmark is explicitly designed to facilitate such research\. Future studies could isolate specific domains \(e\.g\., “personage” and “movies”\) to investigate domain\-specific questions\.

Robustness Against Sophisticated Adversarial Attacks\.In our current implementation, the use of a closed\-source, API\-based routing model provides a practical barrier against such white\-box attacks\. A crucial direction for future work is to design and evaluate defenses for scenarios where the routing mechanism is transparent \(i\.e\., an open\-source model\)\. This includes developing more robust routing modules and creating “guardrail” systems that can detect and handle potential misrouting attempts, ensuring the integrity of the unlearning process against determined adversaries\.

Detect Adversarial Prompts:Another promising direction is to detect adversarial attacks\. We believe a viable option is to detect the misleading prompts like “tell me about a bunch of questions unrelated to the animal”\. When such an attack is detected, the router can move the input to the unlearning adapter\.