$\mu^2$-Bench: A Multilingual Machine Unlearning Benchmark

# A Multilingual Machine Unlearning Benchmark
Source: [https://arxiv.org/html/2609.20945](https://arxiv.org/html/2609.20945)
###### Abstract

Undesired information such as harmful content and private data propagates through Multilingual Large Language Models \(LLMs\) via direct training and indirect cross\-linguistic spread\. Multilingual Machine Unlearning \(MMU\) aims to remove such information, yet its evaluation remains underexplored, leaving unclear whether unlearning truly eliminates target knowledge across all languages\. To bridge this gap, we introduce𝝁𝟐\\bm\{\\mu^\{2\}\}\-Bench, an MMU benchmark that simulates the full pipeline of memorization, unlearning, and evaluation across diverse languages\. It 1\) spans a broad set of languages, 2\) evaluates on both training and hold\-out languages, and 3\) assesses knowledge as dispersed across multiple languages\. We show that successful MMU requires methods that reflect multilingual characteristics, and conduct analysis to provide deeper insights into MMU\.

![Refer to caption](https://arxiv.org/html/2609.20945v1/teaser.png)Figure 1:Illustration of knowledge after unlearning\. Columns correspond to languages, and rows correspond to knowledge instances expressed across multiple languages\.Graycells indicate knowledge forgotten in the corresponding language, whereas colored cells indicate the retained ones, either acquired directly via training \(blue\) or indirectly via cross\-linguistic spread \(red\)\.![Refer to caption](https://arxiv.org/html/2609.20945v1/main_figure_final.png)Figure 2:Overview of the𝝁𝟐\\bm\{\\mu^\{2\}\}\-Bench: \(Left\) a multilingual dataset is constructed, \(Middle\) memorization and unlearning are performed using only the training languages of each setting, \(Right\) and the unlearned model is evaluated on both the training and hold\-out languages\. Latin and non\-Latin languages are denoted inredandblue\.## 1Introduction

How do LLMs acquire knowledge? Modern LLMs are typically pretrained on large multilingual corpora spanning various languages\. As a result, LLMs gain the capability to embed knowledge multilingually\. When further fine\-tuned, each knowledge instance within the fine\-tuning set becomes embedded acrossboth thetraining languages in which the knowledge was directly expressed in the fine\-tuning set andthehold\-out languages in which it was not\. This spread of knowledge into hold\-out languages is a phenomenon called cross\-linguistic spread[Han et al\. \(2025\)](https://arxiv.org/html/2609.20945#bib.bib22);[Zeng et al\. \(2025\)](https://arxiv.org/html/2609.20945#bib.bib23)\. The same applies toundesiredknowledge:harmful or private contentacquired by a model becomes embedded across both training and hold\-out languages[Lu and Koehn \(2025\)](https://arxiv.org/html/2609.20945#bib.bib9)\. Under this scenario, the need to remove such knowledge has motivated the research on Multilingual Machine Unlearning \(MMU\)[Choi et al\. \(2024\)](https://arxiv.org/html/2609.20945#bib.bib6)\.

However, the evaluation of MMU remains underexplored\.MMU should be evaluated from three perspectives: 1\) whether the evaluation accounts for linguistic diversity, such as differences in resource level, 2\) whether forgetting occurs not only in the training languages but also in the hold\-out languages, and 3\) whether the knowledge spanning multiple languages itself is removed as a whole, rather than being assessed separately in a language\-wise manner\. For instance, as in the dotted first row of Figure[1](https://arxiv.org/html/2609.20945#S0.F1), a model may forget target knowledge \(e\.g\.,writer\) in training languages but continue to retrieve it in hold\-out languages[Hwang et al\. \(2026\)](https://arxiv.org/html/2609.20945#bib.bib15)\. In such cases, evaluating unlearning solely in the training language, and in a language\-wise manner is insufficient\. However, existing benchmarks address these perspectives only partially, thereby failing to capture such cases and leaving the effectiveness of MMU inadequately measured[Savelli et al\. \(2025\)](https://arxiv.org/html/2609.20945#bib.bib14);[Farashah et al\. \(2026\)](https://arxiv.org/html/2609.20945#bib.bib19);[Lizzo and Heck \(2026\)](https://arxiv.org/html/2609.20945#bib.bib13)\.

In this paper, we present𝝁𝟐\\bm\{\\mu^\{2\}\}\-Bench \(MUltilingual Machine Unlearning Benchmark\), a benchmark for MMU that addresses these limitations in three dimensions: dataset, framework, and evaluation\. First, we construct a multilingual dataset spanning 12 languages, considering both scripts and resource\-levels, which enables a comprehensive analysis across diverse linguistic properties\. Second,as illustrated in Figure[2](https://arxiv.org/html/2609.20945#S0.F2),we propose a simulation framework that explicitly accounts for cross\-linguistic spread, allowing us to evaluate whethertargetknowledge remains accessible in hold\-out languagesafter unlearning\.Third, we apply knowledge\-wise evaluation metrics to our benchmark, enabling us to assess MMU in a language\-agnostic manner\.Together, these components provide a more rigorous evaluation of MMU\.

To this end, we present a comprehensive set of experiments conducted with𝝁𝟐\\bm\{\\mu^\{2\}\}\-Bench\. Our experiments show that MMU requires methods that explicitly account for the multilingual nature of the task\. Furthermore, we provide a detailed analysis of unlearning behavior across training and hold\-out languages, examined along the dimensions of script and resource level\. Moreover, we show that MMU can induce code\-mixed outputs as an unintended side effect, and discuss how this phenomenon complicates the evaluation of unlearning\.Collectively, these contributions establish both a benchmark for evaluating MMU and a set of analyses that deepen the understanding of MMU\.

## 2Related Work

### 2\.1Cross\-linguistic Spreadin LLMs

Modern LLMs are trained in multilingual environments, which in turn allow undesired information to be embedded across diverse languages[Lu and Koehn \(2025\)](https://arxiv.org/html/2609.20945#bib.bib9);[Hwang et al\. \(2026\)](https://arxiv.org/html/2609.20945#bib.bib15)\.LLMs possess the ability to retrieve knowledge from languages not explicitly used in training, a phenomenon attributed to cross\-linguistic spread[Pires et al\. \(2019\)](https://arxiv.org/html/2609.20945#bib.bib17);[Conneau et al\. \(2020\)](https://arxiv.org/html/2609.20945#bib.bib16);[Shaham et al\. \(2024\)](https://arxiv.org/html/2609.20945#bib.bib18)\. This ability allows undesired information to propagate to hold\-out languages, thereby posing a new safety challenge\.

### 2\.2Multilingual Machine Unlearning \(MMU\)

MMUaims to removeundesiredinformation, expressed in multiple languages, from a trained model[Choi et al\. \(2024\)](https://arxiv.org/html/2609.20945#bib.bib6);[Li et al\. \(2026\)](https://arxiv.org/html/2609.20945#bib.bib21)\.[Choi et al\. \(2024\)](https://arxiv.org/html/2609.20945#bib.bib6)show that unlearning a multilingual LLM on an English\-only dataset is insufficient, accordingly proposing a method to tackle this\.[Lu and Koehn \(2025\)](https://arxiv.org/html/2609.20945#bib.bib9)extend their intuition and show that the knowledge acquired through cross\-linguistic spread persists after English\-only unlearning\.Despite such growing body of literature, evaluation framework for MMU remains underexplored\.

### 2\.3Benchmarking and Evaluating MMU

There have been few efforts to benchmark and evaluate MMU\. For example,[Lizzo and Heck \(2026\)](https://arxiv.org/html/2609.20945#bib.bib13)as well as[Farashah et al\. \(2026\)](https://arxiv.org/html/2609.20945#bib.bib19)translate the English unlearning benchmark TOFU[Maini et al\. \(2024\)](https://arxiv.org/html/2609.20945#bib.bib3)into multiple languages\.[Savelli et al\. \(2025\)](https://arxiv.org/html/2609.20945#bib.bib14)follow TOFU to construct a synthetic MMU benchmark centered on fictional actors\. However, these benchmarks have two limitations in evaluation: 1\) they are confined to language\-wise evaluation, and 2\) they do not evaluate the knowledge acquired through cross\-linguistic spread\. As a result, they offer limited evidence as to whether a knowledge instance has been removed across all of its multilingual expressions\.[Hwang et al\. \(2026\)](https://arxiv.org/html/2609.20945#bib.bib15)partially address this limitation by proposing an evaluation framework that includes two hold\-out languages, together with a metric for measuring unlearning across languages\. However, the hold\-out languages in their work are limited to Latin\-script languages, insufficient for a comprehensive analysis\. In this paper, we address the two limitations more extensively, and provide a benchmark that offers thorough simulation of MMU, encompassing memorization, unlearning, and evaluation\.

## 3Problem Formulation

In this section, we introduce the notion of knowledge, training languages and hold\-out languages\. Subsequent sections of the paper build on these formulations to examine the dynamics in MMU\.

### 3\.1Multilingual Knowledge Instances

Following[Hwang et al\. \(2026\)](https://arxiv.org/html/2609.20945#bib.bib15), we view a knowledge instance as a language\-agnostic unit that can be expressed in multiple languages\.Let𝕃\\mathbb\{L\}denote the set of languages considered in the benchmark\.Theii\-th knowledge instanceki,𝕃k\_\{i,\\mathbb\{L\}\}is defined as:

ki,𝕃=\(ki,ℓ\)ℓ∈𝕃,k\_\{i,\\mathbb\{L\}\}=\(k\_\{i,\\ell\}\)\_\{\\ell\\in\\mathbb\{L\}\},\\vskip\-2\.84526pt\(1\)whereki,ℓk\_\{i,\\ell\}is the knowledge expressed in each languageℓ\\ell\.In our benchmark, eachki,ℓk\_\{i,\\ell\}corresponds to language\-specific knowledge that expresses the same underlying fact aski,𝕃k\_\{i,\\mathbb\{L\}\}\.The complete multilingual knowledge set can therefore be viewed as a matrix whose rows correspond to knowledge instances and whose columns correspond to languages\.We partition the set of knowledge indicesℐ\\mathcal\{I\}into target \(to be unlearned\) and non\-target \(to be retained\) subsets, denoted byℐT\\mathcal\{I\}\_\{\\mathrm\{T\}\}andℐN\\mathcal\{I\}\_\{\\mathrm\{N\}\}each\.

𝒟f\\displaystyle\\mathcal\{D\}\_\{f\}=\{ki,𝕃∣i∈ℐT\},\\displaystyle=\\\{\\,k\_\{i,\\mathbb\{L\}\}\\mid i\\in\\mathcal\{I\}\_\{\\mathrm\{T\}\}\\,\\\},𝒟r\\displaystyle\\mathcal\{D\}\_\{r\}=\{kj,𝕃∣j∈ℐN\},\\displaystyle=\\\{\\,k\_\{j,\\mathbb\{L\}\}\\mid j\\in\\mathcal\{I\}\_\{\\mathrm\{N\}\}\\,\\\},
where𝒟f\\mathcal\{D\}\_\{f\}and𝒟r\\mathcal\{D\}\_\{r\}denote the forget and the retain sets, respectively\.The full dataset𝒟=𝒟f∪𝒟r\\mathcal\{D\}=\\mathcal\{D\}\_\{f\}\\cup\\mathcal\{D\}\_\{r\}can be viewed as an\|ℐ\|×\|𝕃\|\|\\mathcal\{I\}\|\\times\|\\mathbb\{L\}\|matrix\.Building on this view, we propose an MMU benchmark that considers the knowledge\-wise perspective\.In Figure[1](https://arxiv.org/html/2609.20945#S0.F1), each row corresponds to a knowledge instance\.

### 3\.2Language Set Partitioning

Language\-specific knowledgeki,ℓk\_\{i,\\ell\}can be acquired through two distinct pathways:1\) direct acquisition, where the model has been exposed toki,ℓk\_\{i,\\ell\}during memorization, and 2\) indirect acquisition, where the model has not been directly exposed but can still access the underlying knowledge through cross\-linguistic spread\.To formalize this distinction, we partition the language set𝕃\\mathbb\{L\}into training languages𝕃train\\mathbb\{L\}^\{\\mathrm\{train\}\}and hold\-out languages𝕃hold\\mathbb\{L\}^\{\\mathrm\{hold\}\}:

𝕃=𝕃train∪𝕃hold,𝕃train∩𝕃hold=∅\.\\mathbb\{L\}=\\mathbb\{L\}^\{\\mathrm\{train\}\}\\cup\\mathbb\{L\}^\{\\mathrm\{hold\}\},\\quad\\mathbb\{L\}^\{\\mathrm\{train\}\}\\cap\\mathbb\{L\}^\{\\mathrm\{hold\}\}=\\emptyset\.\(2\)
Languages in𝕃train\\mathbb\{L\}^\{\\mathrm\{train\}\}are used during memorization and unlearning, whereas languages in𝕃hold\\mathbb\{L\}^\{\\mathrm\{hold\}\}are excluded from both stages and reserved for evaluation\.In our𝝁𝟐\\bm\{\\mu^\{2\}\}\-Bench, we evaluate MMU across both splits of languages\.

## 4µ2\-Bench: A Multilingual Machine Unlearning Benchmark

In this section, we introduce𝝁𝟐\\bm\{\\mu^\{2\}\}\-Bench, aMUltilingualMachineUnlearning Benchmark for evaluating MMU\.𝝁𝟐\\bm\{\\mu^\{2\}\}\-Bench consists of three components: 1\) a synthetic multilingual QA dataset, 2\) controlled MMU framework including dataset splits and unlearning configurations, and 3\) an evaluation protocol that includes both training and hold\-out languages\. The benchmark is designed to assess whether target knowledge is removed across languages, even where it was not directly observed during memorization and unlearning\.

### 4\.1Dataset Construction

Figure 3:The construction of the translation prompt\. To guarantee the translation quality, the prompt includes the definition of each English attribute along with its translation into the target language\.We construct a multilingual parallel dataset of 60,000 question\-answer \(QA\) pairs across 12 languages, comprising 5,000 knowledge instances across 250 fictitious profiles\.Each synthetic profile is characterized by 20 attributes that yield a corresponding QA pair\. The overall construction pipeline is in three stages: 1\) generating synthetic profiles in English, 2\) constructing English QA datasets from each profile, 3\) translating the QA datasets into multiple target languages, and validating the translations through back\-translation and human\-in\-the\-loop refinement\.Detailed additional explanations are provided in the Appendix[B](https://arxiv.org/html/2609.20945#A2)\.

#### Synthetic English Profiles

We constructed a total of 250 distinct synthetic profiles,consistingof a randomly generated name and 20 independent attributes\. The names were generated in English using the Faker library[Faraglia \(2025\)](https://arxiv.org/html/2609.20945#bib.bib8), and the attributes were sampled from predefined value pools to characterize each profile\. All entries were verified by human annotators to ensurenoambiguity\.

#### English QA Dataset

Based on the constructed profiles, we generated one English QA pair per attribute, yielding a total of 5,000 pairs \(250×\\times20\)\. Specifically, we adopted a template\-based approach in which attribute values were mapped to predefined templates to produce the initial QA pairs\. The initial set was subsequently refined using the GPT\-5\.4\-mini API[Singh et al\. \(2025\)](https://arxiv.org/html/2609.20945#bib.bib5)\. Human annotators then reviewed the pairs to ensure that each question precisely targeted its intended attribute\.

#### Multilingual QA Dataset

To extend the English QA dataset into a multilingual setting, we translated the 5,000 English QA pairs into 11 target languages using the GPT\-5\.4\-mini API[Singh et al\. \(2025\)](https://arxiv.org/html/2609.20945#bib.bib5), resulting in a multilingual QA dataset of 60,000 instances \(12×\\times5,000\) including English\.The twelve languages are English \(en\), German \(de\), Indonesian \(id\), Afrikaans \(af\), Spanish \(es\), Latvian \(lv\), Chinese \(zh\), Korean \(ko\), Urdu \(ur\), Greek \(el\), Macedonian \(mk\), Thai \(th\)\.The translation proceeded in four steps\. First, the English values in the predefined value pools were translated into each target language\. Second, the English values were annotated with their corresponding definition to preserve its intended meaning\. Third, the English QA pairs, together with the translated attributes and their definitions, were concatenated into a single prompt for machine translation\. Finally, the translated QA sets were refined through a human\-in\-the\-loop process\. For the final refinement process, we employed a back\-translation\-based quality control pipeline: each translated instance was back\-translated into Englishwith the Google Translation API[Cloud \(2025\)](https://arxiv.org/html/2609.20945#bib.bib10), and the semantic consistency between the original and the back\-translated sentence was verified by humanannotators\. The overall construction pipeline of the multilingual QA dataset is illustrated in Figure[3](https://arxiv.org/html/2609.20945#S4.F3)\.

### 4\.2MMU Simulation Framework

SettingScriptTrainingHold\-outSetting 1LatinEN, DE, ID, AFES, LVNon\-Latin—ZH, URSetting 2Latin—EN, LVNon\-LatinZH, EL, MK, THKO, URTable 1:Languageconfigurationsused in𝝁𝟐\\bm\{\\mu^\{2\}\}\-Benchacross two evaluation settings, organized by writing script \(Latin vs\. non\-Latin\) and resource level \(High vs\. Low\)\.Boldindicates high\-resource languages\.We present an overall pipeline of𝝁𝟐\\bm\{\\mu^\{2\}\}\-Benchin three stages: memorization, unlearning, and evaluation\.

#### Memorization

The model is fine\-tuned on the memorization set expressed only in the training languages,ki,𝕃traink\_\{i,\\mathbb\{L\}^\{\\mathrm\{train\}\}\}, wherei∈ℐT∪ℐNi\\in\\mathcal\{I\}\_\{\\mathrm\{T\}\}\\cup\\mathcal\{I\}\_\{\\mathrm\{N\}\}denotes the knowledge index\.For controlled observation, we design two settings with different training languages as summarized in Table[1](https://arxiv.org/html/2609.20945#S4.T1)\. In Setting 1, the training languages are all Latin script languages, whereas in Setting 2, they are all non\-Latin script languages\. In both settings, the hold\-out language set contains two Latin script and two non\-Latin script languages, and both the training and hold\-out sets are each balanced with two high\-resource and two low\-resource languages\.Within this setup, 200 of the 250 profiles constructed are used for memorization, contributing 16,000 QA pairs in total across the four training languages\. The remaining 50 profiles are reserved as non\-members for the privacy leakage evaluation\.

#### Unlearning

Unlearning is applied to the target knowledge expressed in the training languages,ki,𝕃traink\_\{i,\\mathbb\{L\}^\{\\mathrm\{train\}\}\}\. We conduct unlearning under three settings with the forget ratio:1%1\\%\(p​1p1\),3%3\\%\(p​3p3\), and5%5\\%\(p​5p5\)\. For each setting, the corresponding percentage of the 200 individuals used during memorization is designated as the forget set𝒟f\\mathcal\{D\}\_\{f\}, while the remaining constitute the retain set𝒟r\\mathcal\{D\}\_\{r\}\.

#### Evaluation

Using the unlearned model, we assess how effectively the target knowledge has been removed, reporting results separately for the training and the hold\-out languages\. This separation allows us to analyze how each unlearning method behaves within each language group\. The hold\-out languages used for evaluation in each setting are listed in Table[1](https://arxiv.org/html/2609.20945#S4.T1)\.To this end, we provide an evaluation pipeline that considers not only the training languages but also the hold\-out languages overlooked by prior benchmarks\. This enables controlled evaluation across diverse settings by varying the memorization languages\.

### 4\.3Evaluation Protocol

The evaluation is conducted using the constructed QA pairs, denoted aski,ℓ=\(qi,ℓ,ai,ℓ\)k\_\{i,\\ell\}=\(q\_\{i,\\ell\},a\_\{i,\\ell\}\)with a slight abuse of notation\. As the proxy for ideal unlearning, we employ a model trained exclusively on non\-target knowledge, denoted asfRf\_\{R\}\. The reference modelfRf\_\{R\}is trained separately for Setting 1 and Setting 2, using both training and hold\-out languages within each setting\. SincefRf\_\{R\}is trained exclusively on the non\-target knowledge while excluding the target knowledge, a successfully unlearned model should closely resemblefRf\_\{R\}\. Consequently,fRf\_\{R\}can serve as the gold standard\.𝝁𝟐\\bm\{\\mu^\{2\}\}\-Bench measures MMU from three perspectives: aggregated forget quality \(FQ1\-3\), separability quality \(SQ\) between the target and the non\-target knowledge and the maintenance of general utility quality \(UQ\)\.

#### FQ1: Probability Score

The Probability score is calculated by averaging the probability of the ground\-truth answer when given the questions:

PS=1\|ℐ\|​∑i∈ℐ1\|𝕃\|​∑ℓ∈𝕃𝒫​\(ai,ℓ\|qi,ℓ\)1/\|ai,ℓ\|tok\.\\mathrm\{PS\}=\\frac\{1\}\{\|\\mathcal\{I\}\|\}\\sum\_\{i\\in\\mathcal\{I\}\}\\frac\{1\}\{\|\\mathbb\{L\}\|\}\\sum\_\{\\ell\\in\\mathbb\{L\}\}\\mathcal\{P\}\(a\_\{i,\\ell\}\|q\_\{i,\\ell\}\)^\{\{1\}/\{\|a\_\{i,\\ell\}\|\_\{\\mathrm\{tok\}\}\}\}\.\(3\)
Here,\|ai,ℓ\|tok\|a\_\{i,\\ell\}\|\_\{\\mathrm\{tok\}\}represents the number of tokens inai,ℓa\_\{i,\\ell\}\. A higher score denotes stronger memorization and less forgetting of the ground truth\.

#### FQ2: Semantic Equivalence Score

The Semantic Equivalence score validates whether the output generated by a modelffmatches the ground\-truth:

SE=1\|ℐ\|​∑i∈ℐ1\|𝕃\|​∑ℓ∈𝕃𝕀⁡\(LLM⁡\(f⁡\(qi,ℓ\),ai,ℓ\)\)\.\\mathrm\{SE\}=\\frac\{1\}\{\|\\mathcal\{I\}\|\}\\sum\_\{i\\in\\mathcal\{I\}\}\\frac\{1\}\{\|\\mathbb\{L\}\|\}\\sum\_\{\\ell\\in\\mathbb\{L\}\}\\mathbb\{I\}\(\\mathrm\{LLM\}\(f\(q\_\{i,\\ell\}\),a\_\{i,\\ell\}\)\)\.\(4\)
In this paper, we employ the GPT\-4o\-mini API with greedy decoding to verify equivalence\.The prompt used for evaluation can be found in Figure[11](https://arxiv.org/html/2609.20945#A4.F11)\.

#### FQ3: Privacy Leakage

The Membership inference attack \(MIA\) is adopted by MUSE[Shi et al\. \(2024\)](https://arxiv.org/html/2609.20945#bib.bib4)in the form of Privacy Leakage:

PrivLeak=AUC​\(f,𝒟f,𝒟h\)−AUC​\(fR,𝒟f,𝒟h\)AUC​\(fR,𝒟f,𝒟h\),\\text\{PrivLeak\}=\\frac\{\\text\{AUC\}\(f;\\mathcal\{D\}\_\{f\},\\mathcal\{D\}\_\{h\}\)\-\\text\{AUC\}\(f\_\{R\};\\mathcal\{D\}\_\{f\},\\mathcal\{D\}\_\{h\}\)\}\{\\text\{AUC\}\(f\_\{R\};\\mathcal\{D\}\_\{f\},\\mathcal\{D\}\_\{h\}\)\},\(5\)
where𝒟h\\mathcal\{D\}\_\{h\}denotes the 50 non\-member profiles that were excluded from training\. Privacy Leakage quantifies the extent to which a modelffresemblesfRf\_\{R\}in distinguishing between non\-member data and target knowledge\. For this metric, the calculation is performed by aggregating at the knowledge level\. An absolute value closer to00indicates better resemblance, thus more successful unlearning\. We follow the implementation of OpenUnlearning[Dorna et al\. \(2026\)](https://arxiv.org/html/2609.20945#bib.bib12)and flip the sign of the value when reporting the result\.

#### SQ: Knowledge Separability Score \(KSS\)

We adopt KSS from[Hwang et al\. \(2026\)](https://arxiv.org/html/2609.20945#bib.bib15), and normalize it to obtain a more intuitive metric\. KSS quantifies how well a model differentiates the knowledge instanceski,𝕃k\_\{i,\\mathbb\{L\}\}targeted for unlearning \(i∈ℐTi\\in\\mathcal\{I\}\_\{\\mathrm\{T\}\}\) from those that are not \(i∈ℐNi\\in\\mathcal\{I\}\_\{\\mathrm\{N\}\}\)\. For the calculation, the preceding work first defines the forgetting score ofii\-th knowledge as:

Si=1−1\|𝕃\|​∑ℓ∈𝕃s⁡\(ki,ℓ\),S\_\{i\}=1\-\\frac\{1\}\{\|\\mathbb\{L\}\|\}\\sum\_\{\\ell\\in\\mathbb\{L\}\}s\(k\_\{i,\\ell\}\),\(6\)
wheres⁡\(ki,ℓ\)s\(k\_\{i,\\ell\}\)can be either token probability𝒫​\(ai,ℓ\|qi,ℓ\)1/\|ai,ℓ\|tok\\mathcal\{P\}\(a\_\{i,\\ell\}\|q\_\{i,\\ell\}\)^\{\{1\}/\{\|a\_\{i,\\ell\}\|\_\{\\mathrm\{tok\}\}\}\}or semantic equivalence𝕀⁡\(LLM⁡\(f⁡\(qi,ℓ\),ai,ℓ\)\)\\mathbb\{I\}\(\\mathrm\{LLM\}\(f\(q\_\{i,\\ell\}\),a\_\{i,\\ell\}\)\)\. From the distributions ofSiS\_\{i\}collected from the target knowledge and the non\-target knowledge, respectively, the Area Under the Receiver Operating Characteristic Curve \(KSSROC\\text\{KSS\}^\{\\text\{ROC\}\}\) and the Precision\-Recall Curve \(KSSPR\\text\{KSS\}^\{\\text\{PR\}\}\) are computed\.We normalize KSS as follows:

KSS\-Norm=KSS\(f\)−KSS\(fR\)KSS\(fR\)\.\\text\{KSS\-Norm\}=\\frac\{\\text\{KSS\($f$\)\}\-\\text\{KSS\($f\_\{R\}$\)\}\}\{\\text\{KSS\($f\_\{R\}$\)\}\}\.\(7\)
KSS\-Norm quantifies how well a modelffresembles the reference modelfRf\_\{R\}in distinguishing between the target and non\-target knowledge\. Below, KSS\-Norm is abbreviated as KSS for conciseness\.

#### UQ: General Knowledge

To verify that unlearning has not degraded the general ability of an LLM, we measure the model’s utility with TyDi QA[Clark et al\. \(2020\)](https://arxiv.org/html/2609.20945#bib.bib24), complementing the retain quality measured on the retain set𝒟r\\mathcal\{D\}\_\{r\}\. TyDi QA reports performance with F1\-score and EM score\.

## 5Experiments

### 5\.1Experimental Setup

#### Baselines

We compare four unlearning methods\. We adopt three MU baselines into the multilingual setting: Gradient Ascent \(GA\)[Jang et al\. \(2022\)](https://arxiv.org/html/2609.20945#bib.bib1), Gradient Difference \(GAGDR\)[Maini et al\. \(2024\)](https://arxiv.org/html/2609.20945#bib.bib3), and Negative Preference Optimization \(NPO\)[Zhang et al\. \(2024\)](https://arxiv.org/html/2609.20945#bib.bib2)\. We additionally include LingTea[Choi et al\. \(2024\)](https://arxiv.org/html/2609.20945#bib.bib6), a method specifically designed for MMU\. All of the experiments are conducted on a single A6000 GPU\.

#### Target Models

We conduct unlearning experiments on two models: Gemma3\-12B[Kamath et al\. \(2025\)](https://arxiv.org/html/2609.20945#bib.bib20)and Qwen2\.5\-7B\-Instruct[Yang et al\. \(2025\)](https://arxiv.org/html/2609.20945#bib.bib7)\.

#### Hyperparameter Search

We search for unlearning hyperparameters so that the Probability score of the retain set meets the predefined threshold across all methods\. The threshold is set to 90% of the Probability score of the fully memorized model, evaluated on\{ki,ℓ\|i∈ℐN,ℓ∈𝕃train\}\\\{k\_\{i,\\ell\}\\,\|\\,i\\in\\mathcal\{I\}\_\{\\mathrm\{N\}\},\\ell\\in\\mathbb\{L\}^\{\\mathrm\{train\}\}\\\}\. Details and full hyperparameter configurations are reported in Appendix[C](https://arxiv.org/html/2609.20945#A3)\.

#### Metrics

We measure the overall degree of forgetting through the Probability Score \(PS\) and Semantic Equivalence \(SE\) scores on the forget set𝒟f\\mathcal\{D\}\_\{f\}, where a lower score indicates better forgetting\. To capture separability between the target and non\-target knowledge, we further report four normalized KSS variants, where a larger value indicates that the unlearned model separates the target knowledge from the non\-target knowledge more effectively, measured relative to the reference modelfRf\_\{R\}\. We additionally report the Privacy Leakage score \(PrivLeak\), for which a value closer to00indicates that the unlearned model more closely resemblesfRf\_\{R\}\. Finally, we verify that general ability is preserved using TyDi QA[Clark et al\. \(2020\)](https://arxiv.org/html/2609.20945#bib.bib24), reporting the average of its F1 and EM scores as General Knowledge \(Gen Know\.\)\.

### 5\.2Results

TrainHoldoutGen\.Know\.ModelSetppMethodPSSEKSSROCprob\{\}\_\{\\text\{prob\}\}^\{\\text\{ROC\}\}KSSPRprob\{\}\_\{\\text\{prob\}\}^\{\\text\{PR\}\}KSSROCgen\{\}\_\{\\text\{gen\}\}^\{\\text\{ROC\}\}KSSPRgen\{\}\_\{\\text\{gen\}\}^\{\\text\{PR\}\}PrivLeakPSSEKSSROCprob\{\}\_\{\\text\{prob\}\}^\{\\text\{ROC\}\}KSSPRprob\{\}\_\{\\text\{prob\}\}^\{\\text\{PR\}\}KSSROCgen\{\}\_\{\\text\{gen\}\}^\{\\text\{ROC\}\}KSSPRgen\{\}\_\{\\text\{gen\}\}^\{\\text\{PR\}\}PrivLeakGemma3\-12BS1p1Mem99\.99100\.00\-53\.18\-99\.06\-50\.00\-99\.00\-107\.9055\.8276\.88\-58\.37\-99\.17\-50\.17\-98\.99\-86\.8530\.19GA72\.4491\.88\-14\.24\-77\.47\-41\.73\-93\.94\-90\.5737\.4568\.75\-42\.05\-97\.21\-44\.63\-98\.68\-60\.7532\.85GAGDR75\.3494\.38\-20\.39\-76\.45\-42\.89\-94\.68\-97\.2345\.0370\.00\-49\.86\-98\.61\-44\.79\-98\.75\-75\.3432\.83NPO47\.8983\.12\-4\.65\-43\.28\-33\.69\-86\.37\-68\.6126\.0967\.50\-26\.60\-92\.70\-45\.54\-98\.59\-41\.3232\.49LingTea45\.3291\.88\-20\.23\-41\.52\-38\.96\-89\.00\-1\.0725\.9273\.12\-28\.49\-47\.51\-46\.95\-98\.88\+9\.1830\.84p3Mem99\.99100\.00\-49\.32\-96\.14\-48\.94\-96\.87\-111\.1554\.2580\.00\-56\.74\-97\.49\-50\.60\-96\.96\-82\.1830\.19GA93\.41100\.00\-39\.18\-79\.13\-48\.95\-96\.87\-109\.2853\.5478\.96\-56\.62\-97\.48\-49\.01\-96\.88\-82\.2030\.05GAGDR79\.84100\.00\-25\.42\-82\.99\-49\.08\-96\.87\-106\.2050\.8578\.33\-55\.97\-97\.44\-49\.36\-96\.91\-82\.1630\.90NPO87\.8399\.79\-38\.71\-88\.04\-48\.56\-96\.73\-109\.3053\.2078\.33\-56\.57\-97\.48\-47\.90\-96\.82\-81\.8830\.08LingTea49\.8492\.08\-23\.55\-44\.29\-38\.86\-86\.15\-30\.5828\.2771\.04\-33\.33\-59\.80\-45\.44\-96\.58\-8\.9135\.08p5Mem99\.99100\.00\-54\.44\-94\.99\-48\.98\-94\.80\-105\.6650\.9879\.38\-51\.84\-95\.34\-51\.00\-95\.06\-90\.0330\.19GA87\.48100\.00\-43\.17\-91\.85\-49\.09\-94\.80\-104\.3549\.6877\.75\-51\.55\-95\.29\-49\.34\-94\.91\-89\.1630\.29GAGDR74\.3297\.50\-32\.01\-68\.19\-46\.11\-92\.14\-87\.2536\.8075\.12\-41\.31\-82\.87\-49\.01\-94\.81\-60\.9131\.99NPO89\.00100\.00\-44\.26\-92\.59\-49\.02\-94\.80\-104\.0950\.3177\.62\-51\.66\-95\.32\-48\.93\-94\.85\-89\.7630\.12LingTea72\.1694\.12\-31\.98\-55\.95\-41\.11\-86\.12\-68\.1637\.3071\.25\-39\.50\-80\.95\-44\.39\-94\.14\-57\.0931\.46S2p1Mem99\.99100\.00\-53\.51\-99\.06\-50\.00\-99\.00\-115\.3850\.0978\.75\-58\.47\-99\.19\-53\.42\-99\.08\-106\.7523\.12GA75\.6892\.50\-19\.88\-77\.47\-41\.62\-93\.36\-106\.4942\.9173\.12\-50\.67\-98\.94\-51\.92\-99\.00\-92\.6722\.84GAGDR45\.0774\.38\-12\.38\-40\.97\-27\.29\-77\.69\-30\.1129\.7855\.62\-33\.85\-96\.42\-36\.04\-98\.43\-58\.8223\.27NPO55\.0480\.00\-14\.17\-39\.63\-26\.63\-81\.90\-70\.0442\.6969\.38\-50\.97\-98\.91\-47\.61\-98\.87\-91\.0423\.05LingTea54\.2378\.75\-5\.06\-39\.52\-23\.66\-75\.32\-72\.6716\.8554\.37\-12\.74\-73\.43\-32\.70\-98\.31\-13\.5322\.42p3Mem99\.99100\.00\-49\.57\-96\.86\-48\.94\-96\.87\-100\.4649\.9180\.00\-58\.53\-97\.62\-54\.52\-97\.26\-84\.9223\.12GA87\.6598\.33\-45\.19\-92\.91\-47\.94\-96\.79\-98\.5049\.7079\.58\-58\.80\-97\.62\-54\.67\-97\.24\-84\.5423\.08GAGDR75\.5098\.33\-26\.91\-58\.71\-46\.51\-94\.53\-77\.6042\.3775\.21\-46\.54\-96\.04\-50\.96\-97\.06\-68\.3422\.02NPO87\.8098\.33\-44\.38\-93\.22\-47\.49\-96\.73\-98\.0850\.3478\.96\-58\.58\-97\.62\-53\.87\-97\.23\-85\.1023\.08LingTea73\.1496\.67\-30\.94\-58\.84\-45\.61\-91\.25\-71\.5839\.8275\.00\-44\.57\-92\.86\-51\.28\-97\.04\-60\.3722\.05p5Mem99\.99100\.00\-49\.33\-95\.00\-48\.98\-94\.80\-100\.9249\.2879\.12\-57\.95\-95\.82\-53\.88\-95\.23\-80\.7123\.12GA90\.6398\.38\-46\.79\-93\.54\-48\.05\-94\.68\-100\.3449\.1379\.00\-57\.96\-95\.82\-53\.61\-95\.24\-80\.8523\.17GAGDR87\.7798\.75\-38\.10\-77\.82\-46\.99\-93\.35\-97\.8246\.4177\.38\-53\.42\-95\.27\-52\.90\-95\.14\-77\.5322\.57NPO90\.1598\.25\-45\.74\-91\.95\-47\.81\-94\.64\-100\.2649\.6378\.12\-57\.91\-95\.83\-52\.52\-95\.16\-81\.0623\.22LingTea68\.6097\.50\-21\.87\-51\.39\-44\.87\-91\.46\-72\.7637\.3675\.50\-42\.57\-85\.52\-50\.20\-94\.97\-55\.5821\.26Qwen2\.5\-7BS1p1Mem99\.90100\.00\-50\.21\-99\.00\-48\.72\-98\.95\-92\.2346\.7643\.12\-53\.35\-98\.98\-54\.88\-99\.10\-46\.4119\.06GA63\.5679\.38\-9\.72\-64\.06\-29\.60\-87\.51\-63\.0940\.5833\.75\-48\.27\-98\.77\-45\.82\-98\.90\-35\.8920\.01GAGDR41\.3847\.50\-7\.15\-32\.42\-14\.97\-48\.97\-1\.8624\.1022\.50\-25\.81\-81\.10\-32\.57\-98\.09\+12\.8519\.57NPO53\.5156\.88\-4\.45\-49\.49\-21\.98\-63\.99\-42\.7335\.5726\.25\-41\.67\-98\.33\-37\.50\-98\.28\-22\.7719\.52LingTea41\.8643\.12\-14\.47\-40\.81\-18\.63\-66\.48\+10\.2817\.3818\.12\-23\.54\-47\.99\-30\.58\-97\.82\+33\.0417\.68p3Mem99\.87100\.00\-49\.50\-94\.94\-49\.58\-96\.95\-108\.5746\.9441\.04\-54\.70\-97\.27\-52\.31\-97\.14\-60\.1719\.06GA84\.3797\.29\-25\.48\-75\.92\-44\.68\-91\.55\-102\.6245\.5837\.08\-54\.47\-97\.27\-48\.25\-96\.90\-59\.5018\.29GAGDR56\.4288\.12\-1\.55\-27\.23\-34\.93\-76\.49\-88\.0941\.8833\.33\-52\.71\-97\.13\-44\.56\-96\.52\-55\.1918\.19NPO79\.1296\.46\-25\.15\-79\.92\-44\.18\-88\.62\-101\.5445\.3836\.46\-54\.33\-97\.24\-47\.38\-96\.82\-59\.0617\.87LingTea17\.1635\.00\-7\.34\-14\.55\-11\.18\-33\.83\+58\.797\.0115\.83\-9\.45\-15\.56\-26\.18\-92\.72\+75\.0218\.47p5Mem99\.88100\.00\-52\.27\-93\.52\-50\.00\-95\.00\-107\.8945\.3541\.12\-51\.49\-95\.25\-52\.86\-95\.33\-56\.3219\.06GA91\.9799\.62\-43\.39\-93\.31\-49\.49\-94\.86\-106\.1644\.6839\.88\-51\.29\-95\.22\-51\.54\-95\.21\-56\.2018\.25GAGDR78\.3690\.75\-21\.76\-69\.67\-37\.09\-81\.60\-98\.9141\.5936\.25\-50\.00\-95\.06\-48\.59\-94\.90\-52\.3917\.77NPO90\.2999\.25\-44\.52\-93\.74\-48\.76\-94\.46\-105\.4345\.0640\.25\-51\.40\-95\.22\-51\.51\-95\.19\-56\.5018\.73LingTea26\.3655\.75\-0\.82\-6\.73\-18\.19\-49\.81\+1\.8914\.1226\.62\-12\.66\-36\.40\-37\.72\-92\.29\+44\.4019\.60S2p1Mem99\.7099\.38\-52\.30\-98\.68\-49\.27\-98\.97\-113\.7846\.9126\.88\-55\.65\-99\.11\-56\.13\-99\.10\-45\.1917\.50GA50\.5515\.62\-0\.57\-13\.77\-5\.99\-59\.75\+25\.4133\.096\.88\-41\.96\-98\.44\-35\.38\-98\.55\-13\.4417\.26GAGDR42\.0710\.62\-0\.32\-9\.57\-6\.91\-81\.63\+44\.9229\.775\.00\-38\.85\-97\.99\-33\.17\-98\.46\-4\.3916\.95NPO38\.988\.75\-0\.19\-4\.58\-6\.18\-74\.08\+67\.6928\.725\.00\-37\.83\-98\.35\-31\.90\-98\.37\-4\.0817\.27LingTea42\.8614\.37\-0\.24\-10\.86\-4\.48\-72\.56\+45\.4028\.573\.75\-32\.13\-95\.64\-31\.89\-98\.39\+6\.2917\.56p3Mem99\.7499\.79\-54\.29\-97\.24\-49\.87\-96\.97\-103\.4746\.4226\.88\-55\.87\-97\.48\-56\.19\-97\.33\-34\.1817\.50GA68\.8437\.08\-14\.03\-44\.66\-23\.58\-84\.71\-36\.8334\.7616\.88\-50\.44\-97\.02\-46\.75\-96\.75\-16\.6617\.36GAGDR64\.5134\.38\-8\.96\-31\.77\-21\.84\-83\.31\-25\.1233\.9812\.92\-49\.28\-96\.93\-43\.30\-96\.49\-13\.3217\.15NPO73\.9442\.50\-12\.81\-35\.70\-21\.41\-72\.33\-50\.7936\.9816\.67\-51\.73\-97\.20\-45\.38\-96\.64\-18\.4017\.76LingTea35\.465\.42\-0\.35\-3\.59\-3\.74\-54\.38\+64\.8817\.024\.58\-11\.57\-60\.83\-29\.01\-94\.69\+65\.1717\.17p5Mem99\.7599\.88\-54\.12\-95\.57\-50\.14\-94\.98\-112\.4945\.2924\.25\-54\.05\-95\.64\-53\.55\-95\.35\-34\.4317\.50GA78\.9349\.12\-25\.82\-64\.83\-30\.58\-86\.21\-79\.8737\.8018\.88\-52\.97\-95\.44\-49\.00\-94\.87\-26\.0316\.91GAGDR70\.0038\.88\-13\.02\-39\.32\-24\.76\-82\.91\-47\.3533\.7719\.50\-50\.72\-95\.18\-51\.30\-95\.14\-20\.9817\.78NPO82\.9854\.37\-26\.84\-66\.85\-29\.06\-86\.97\-93\.5538\.6920\.50\-53\.37\-95\.41\-50\.42\-95\.02\-27\.3217\.62LingTea71\.8648\.38\-7\.14\-24\.17\-14\.15\-53\.38\-54\.9236\.1415\.62\-42\.64\-92\.42\-45\.10\-94\.37\-8\.1217\.67

Table 2:MMU performance on Gemma3\-12B and Qwen2\.5\-7B\. LingTea generally performs well on both training and hold\-out languages\.#### MMU Requires Multilingual\-Specific Methods

Our results confirm that LingTea, the only MMU method in our experiment, generally performs well in both training and hold\-out languages\.On the training languages \(the left side of Table[2](https://arxiv.org/html/2609.20945#S5.T2)\), LingTea generally attains the strongest MMU performance across metrics\. Aggregating over all twelve configurations per metric \(22models×\\times22settings×\\times33forget ratios\), LingTea achieves the best result in8/128/12cases for bothPSandSE,7/127/12forKSSprobROC\\mathrm\{KSS\}\_\{\\text\{prob\}\}^\{\\text\{ROC\}\},9/129/12forKSSprobPR\\mathrm\{KSS\}\_\{\\text\{prob\}\}^\{\\text\{PR\}\},10/1210/12forKSSgenROC\\mathrm\{KSS\}\_\{\\text\{gen\}\}^\{\\text\{ROC\}\},9/129/12forKSSgenPR\\mathrm\{KSS\}\_\{\\text\{gen\}\}^\{\\text\{PR\}\}, and7/127/12forPrivLeak\. In aggregate, LingTea ranks first in5858of the8484metric evaluations \(approximately69%69\\%\)\.

On the hold\-out languages \(the right side of Table[2](https://arxiv.org/html/2609.20945#S5.T2)\), the advantage of LingTea over the competing methods becomes more prominent than on the training languages\. Aggregating over the same twelve configurations per metric, LingTea achieves the best result in10/1210/12cases forPS,11/1211/12for bothSEandKSSprobROC\\mathrm\{KSS\}\_\{\\text\{prob\}\}^\{\\text\{ROC\}\},12/1212/12forKSSprobPR\\mathrm\{KSS\}\_\{\\text\{prob\}\}^\{\\text\{PR\}\},10/1210/12for bothKSSgenROC\\mathrm\{KSS\}\_\{\\text\{gen\}\}^\{\\text\{ROC\}\}andKSSgenPR\\mathrm\{KSS\}\_\{\\text\{gen\}\}^\{\\text\{PR\}\}, and8/128/12forPrivLeak\. To sum up, LingTea ranks first in7272of the8484metric evaluations \(approximately86%86\\%\), surpassing its training\-language share of58/8458/84\(approximately69%69\\%\) and thereby confirming that its relative advantage widens on hold\-out languages\. This widening is most notable onKSSprobPR\\mathrm\{KSS\}\_\{\\text\{prob\}\}^\{\\text\{PR\}\}, where the competing methods remain close to theMembaseline while LingTea keeps its improvement\. Notably, average margin of LingTea over the strongest competing method is larger for the hold\-out languages than for the training languages \(\+14\.96→\+27\.94\+14\.96\\rightarrow\+27\.94\)\. This highlights the need for MMU methods specifically designed for multilingual settings to enable the transfer of unlearning to hold\-out languages\.

Δ\\DeltaPSΔ\\DeltaSECellMethodHighLowHighLowG/S1LingTea\+53\.8\+55\.5\+7\.5\+8\.8NPO\+40\.0\+64\.2\+16\.2\+17\.5GAGDR\+37\.8\+11\.5\+6\.2\+5\.0GA\+36\.4\+18\.7\+7\.5\+8\.8G/S2LingTea\+58\.6\+32\.9\+27\.5\+15\.0NPO\+36\.0\+53\.0\+15\.0\+25\.0GAGDR\+56\.1\+53\.7\+20\.0\+31\.2GA\+17\.0\+31\.6\+6\.2\+8\.8Q/S1LingTea\+58\.0\+58\.2\+60\.0\+53\.8NPO\+55\.1\+37\.8\+46\.2\+40\.0GAGDR\+59\.1\+58\.1\+53\.8\+51\.2GA\+42\.0\+30\.8\+26\.2\+15\.0Q/S2LingTea\+62\.4\+51\.6\+87\.3\+83\.8NPO\+65\.6\+56\.2\+92\.4\+90\.0GAGDR\+63\.1\+52\.5\+91\.1\+87\.5GA\+57\.8\+40\.8\+83\.5\+85\.0Table 3:Training language drop rates per unlearning method\. For each metric,boldmarks the higher value within the High/Low pair\. G and Q denote Gemma3 and Qwen2\.5, respectively\.Δ\\DeltaPSΔ\\DeltaSEresourcescriptresourcescriptCellMethodHighLowLatnLatHighLowLatnLatG/S1LingTea\+54\.8\+55\.8\+56\.6\+54\.0\+4\.2\+5\.9\+8\.1\+1\.6NPO\+52\.1\+48\.7\+65\.7\+35\.1\+13\.9\+9\.8\+11\.3\+13\.1GAGDR\+21\.8\+29\.6\+29\.1\+22\.2\+2\.8\+17\.6\+16\.1\+1\.6GA\+27\.1\+41\.9\+36\.9\+32\.1\+5\.6\+17\.6\+4\.8\+16\.4G/S2LingTea\+61\.4\+62\.5\+40\.4\+83\.6\+22\.7\+43\.1\+21\.3\+40\.0NPO\+11\.7\+16\.3\+17\.7\+10\.3\+12\.0\+11\.8\+11\.5\+12\.3GAGDR\+33\.1\+36\.1\+26\.1\+43\.2\+24\.0\+37\.3\+21\.3\+36\.9GA\+13\.2\+16\.3\+16\.0\+13\.5\+10\.7\+2\.0\+4\.9\+9\.2Q/S1LingTea\+60\.7\+64\.1\+69\.0\+55\.9\+57\.6\+60\.0\+60\.0\+56\.4NPO\+29\.1\+18\.4\+29\.8\+17\.8\+40\.7\+30\.0\+33\.3\+43\.6GAGDR\+42\.5\+52\.0\+56\.5\+37\.9\+45\.8\+60\.0\+46\.7\+48\.7GA\+18\.7\+9\.7\+19\.4\+8\.9\+16\.9\+50\.0\+16\.7\+25\.6Q/S2LingTea\+44\.3\+22\.8\+39\.9\+27\.2\+88\.9\+71\.4\+82\.4\+88\.5NPO\+38\.2\+26\.7\+45\.4\+19\.5\+86\.1\+57\.1\+76\.5\+84\.6GAGDR\+36\.8\+23\.5\+36\.2\+24\.0\+86\.1\+57\.1\+82\.4\+80\.8GA\+29\.8\+18\.9\+33\.3\+15\.5\+77\.8\+57\.1\+70\.6\+76\.9

Table 4:Hold\-out language drop rates per unlearning method\.Boldmarks the higher value within each comparison pair \(High/Low and Lat/nLat\)\. G and Q denote Gemma3 and Qwen2\.5, respectively\.
#### Unlearning in Training Languages

Table[3](https://arxiv.org/html/2609.20945#S5.T3)reports forget quality, measured byPSandSE, on the training languages, with the languages partitioned by resource level \(High vs\. Low\)\. All metrics are aggregated in a knowledge\-wise manner\. For each metric, performance is reported as the drop rate relative to the memorized model, defined asMem−UnlearnedMem×100\\frac\{\\text\{Mem\}\-\\text\{Unlearned\}\}\{\\text\{Mem\}\}\\times 100, so that a larger value indicates stronger forgetting\. Table[3](https://arxiv.org/html/2609.20945#S5.T3)presents the results forp​1p1, while Table[32](https://arxiv.org/html/2609.20945#A7.T32)and Table[33](https://arxiv.org/html/2609.20945#A7.T33)report the corresponding results forp​3p3andp​5p5, respectively\.UnderPS, Qwen2\.5 generally shows a consistent trend across all settings:unlearning occurs more effectively in high\-resource languages\.Yet, for Gemma3,the effect of the resource level varies across settings forp​3p3andp​5p5, whereas no clear trend emerges forp​1p1\. Specifically, forp​3p3andp​5p5, unlearning is predominantly concentrated in high\-resource languages under Setting 1, but is more prominent in low\-resource languages under Setting 2\. UnderSE, however, no clear trend is observed—particularly forp​3p3andp​5p5—as unlearning itself rarely occurs under this metric\.

#### Unlearning Transfer to Hold\-out Languages

Here, we analyze forget quality, measured byPSandSE, on the hold\-out languages, partitioning the languages by resource level \(High vs\. Low\) and script \(Latin vs\. Non\-Latin\)\. Table[4](https://arxiv.org/html/2609.20945#S5.T4)reports each metric as a drop rate relative to the Memorized model defined asMem−UnlearnedMem×100\\frac\{\\text\{Mem\}\-\\text\{Unlearned\}\}\{\\text\{Mem\}\}\\times 100, so that a larger value indicates stronger forgetting\. Table[4](https://arxiv.org/html/2609.20945#S5.T4)presents the results forp​1p1, while Table[34](https://arxiv.org/html/2609.20945#A7.T34)and Table[35](https://arxiv.org/html/2609.20945#A7.T35)report the corresponding results forp​3p3andp​5p5, respectively\. For Gemma3, a notable trend emerges only underPS: unlearning is consistently effective in Setting 2 regardless of the forget ratio, whereas no comparably pronounced pattern is observed underSE\. In contrast, Qwen2\.5 exhibits consistent trends under both metrics\. UnderPS, unlearning is reliably stronger for the low\-resource group in Setting 2\. UnderSE, it is consistently strong both for the high\-resource group in Setting 2 and for the non\-Latin group in Setting 2\. In summary, although some trends within a given model and setting remain consistent across forget ratios, no general trend could be identified\.

#### Existence of Code\-mixed Output

Figure[4](https://arxiv.org/html/2609.20945#S5.F4)presents an example of a model response after MMU, alongside the response originally produced by the memorized model\.As shown in the figure, although the model is prompted to answer inen, it expresses the key attribute ines, yet the response remains semantically equivalent to the ground truth\. MMU exhibits a phenomenon in which the target knowledge remains retrievable in another language\. This complicates how unlearning should be assessed, since reference\-based surface\-form metrics such as ROUGE\-L[Lin \(2004\)](https://arxiv.org/html/2609.20945#bib.bib11)fail to capture responses that are semantically equivalent to the ground truth but differ in language\. Measuring MMU therefore requires semantic evaluation metrics such asSE\.

Figure 4:Example of the model’s response in a hold\-out language after MMU\. The numbers in the figure denote the probability that the LLM assigns to the predicted next token, given the preceding context up to “of”\. The memorized model outputs the English wordwolves, whereas after unlearning the Spanishlobos\(wolves\) receives the highest probability\.
#### General Performance Maintenance

We report the average of F1\-score and EM score of TyDi QA as General Knowledge in the last column of Table[2](https://arxiv.org/html/2609.20945#S5.T2)\. Across models, settings, and forget ratios, the General Knowledge remains comparable to the memorized model after unlearning\. This shows that the model utility is preserved after unlearning\.

## 6Conclusion

In this paper, we introduced𝝁𝟐\\bm\{\\mu^\{2\}\}\-Bench, a benchmark for Multilingual Machine Unlearning \(MMU\)\.𝝁𝟐\\bm\{\\mu^\{2\}\}\-Bench encompasses 1\) a diverse set of languages, 2\) settings that separates training and hold\-out languages, and 3\) unlearning evaluation at knowledge level\. Our analysis demonstrates that successful MMU requires methods that explicitly account for the multilingual nature of the model, particularly in the hold\-out languages\. Furthermore, we provide analysis on MMU across both training and hold\-out languages\. Finally, we show that unlearning can induce code\-mixed outputs as an unintended side effect\. Taken together, our work offers both a testbed for rigorously measuring MMU and a set of analyses that deepen the understanding of unlearning in multilingual settings\.

## 7Limitations

In this paper, we constructed a benchmark for evaluating Multilingual Machine Unlearning by taking diverse knowledge acquisition paths into account\. Nevertheless, several limitations remain to be addressed\. During the dataset construction process, we sought to minimize the noise that may arise in multilingual translation by employing a multi\-stage pipeline that combines Google Translate, the GPT API, and human verification\. However, the human verification step was not conducted by native speakers of each target language\. Instead, verification was performed indirectly by back\-translating the translated dataset into English via Google Translate and inspecting the resulting outputs\. Consequently, the translated datasets may still contain subtle grammatical awkwardness or unnatural expressions that would have been more readily identified through direct review by native speakers\. Although we adopted a multi\-step verification procedure to mitigate such noise, the possibility of residual translation artifacts cannot be entirely excluded\. Addressing this limitation through a more rigorous refinement process involving native\-speaker evaluation of each target language remains an important direction for future work\.

## References

- Choiet al\.\(2024\)M\. Choi, K\. Min, and J\. ChooCross\-lingual unlearning of selective knowledge in multilingual language models\.arXiv preprint arXiv:2406\.12354\.Cited by:[§1](https://arxiv.org/html/2609.20945#S1.p1.1),[§2\.2](https://arxiv.org/html/2609.20945#S2.SS2.p1.1),[§5\.1](https://arxiv.org/html/2609.20945#S5.SS1.SSS0.Px1.p1.1.1)\.
- Clarket al\.\(2020\)J\. H\. Clark, E\. Choi, M\. Collins, D\. Garrette, T\. Kwiatkowski, V\. Nikolaev, and J\. PalomakiTydi qa: a benchmark for information\-seeking question answering in ty pologically di verse languages\.Transactions of the Association for Computational Linguistics8,pp\. 454–470\.Cited by:[§4\.3](https://arxiv.org/html/2609.20945#S4.SS3.SSS0.Px5.p1.1.1),[§5\.1](https://arxiv.org/html/2609.20945#S5.SS1.SSS0.Px4.p1.1.1)\.
- Cloud \(2025\)G\. CloudCloud translation documentation\(Website\)Note:Accessed: 2025\-11\-11External Links:[Link](https://cloud.google.com/translate/docs)Cited by:[Appendix B](https://arxiv.org/html/2609.20945#A2.SS0.SSS0.Px1.p1.1.3),[§4\.1](https://arxiv.org/html/2609.20945#S4.SS1.SSS0.Px3.p1.1)\.
- Conneauet al\.\(2020\)A\. Conneau, S\. Wu, H\. Li, L\. Zettlemoyer, and V\. StoyanovEmerging cross\-lingual structure in pretrained language models\.InProceedings of the 58th annual meeting of the association for computational linguistics,pp\. 6022–6034\.Cited by:[§2\.1](https://arxiv.org/html/2609.20945#S2.SS1.p1.1.2)\.
- Dornaet al\.\(2026\)V\. Dorna, A\. Mekala, W\. Zhao, A\. McCallum, Z\. Kolter, Z\. Lipton, and P\. MainiOpenunlearning: accelerating llm unlearning via unified benchmarking of methods and metrics\.Advances in Neural Information Processing Systems38\.Cited by:[§4\.3](https://arxiv.org/html/2609.20945#S4.SS3.SSS0.Px3.p3.1.1)\.
- Faraglia \(2025\)D\. FaragliaFaker: python package that generates fake data for you\.Note:[https://github\.com/joke2k/faker](https://github.com/joke2k/faker)Cited by:[§4\.1](https://arxiv.org/html/2609.20945#S4.SS1.SSS0.Px1.p1.1)\.
- Farashahet al\.\(2026\)A\. D\. Farashah, A\. Khandelwal, M\. Fauchard, Z\. Shi, N\. Rostamzadeh, and G\. FarnadiMultilingual amnesia: on the transferability of unlearning in multilingual llms\.arXiv preprint arXiv:2601\.05641\.Cited by:[§1](https://arxiv.org/html/2609.20945#S1.p2.1.2),[§2\.3](https://arxiv.org/html/2609.20945#S2.SS3.p1.1)\.
- Hanet al\.\(2025\)H\. Han, S\. Agrawal, and E\. BriakouRethinking cross\-lingual alignment: balancing transfer and cultural erasure in multilingual llms\.arXiv preprint arXiv:2510\.26024\.Cited by:[§1](https://arxiv.org/html/2609.20945#S1.p1.1)\.
- Hwanget al\.\(2026\)K\. Hwang, H\. Kim, S\. Cho, and N\. KwakKnowledge beyond language: bridging the gap in multilingual machine unlearning evaluation\.arXiv preprint arXiv:2605\.14404\.Cited by:[§1](https://arxiv.org/html/2609.20945#S1.p2.1.2),[§2\.1](https://arxiv.org/html/2609.20945#S2.SS1.p1.1),[§2\.3](https://arxiv.org/html/2609.20945#S2.SS3.p1.1),[§3\.1](https://arxiv.org/html/2609.20945#S3.SS1.p1.1.1),[§4\.3](https://arxiv.org/html/2609.20945#S4.SS3.SSS0.Px4.p1.1.1)\.
- Janget al\.\(2022\)J\. Jang, D\. Yoon, S\. Yang, S\. Cha, M\. Lee, L\. Logeswaran, and M\. SeoKnowledge unlearning for mitigating privacy risks in language models\.arXiv preprint arXiv:2210\.01504\.Cited by:[Appendix A](https://arxiv.org/html/2609.20945#A1.p3.1.1),[§5\.1](https://arxiv.org/html/2609.20945#S5.SS1.SSS0.Px1.p1.1.1)\.
- Kamathet al\.\(2025\)A\. Kamath, J\. Ferret, S\. Pathak, N\. Vieillard, R\. Merhej, S\. Perrin, T\. Matejovicova, A\. Ramé, M\. Rivière, L\. Rouillard, T\. Mesnard, G\. Cideron, J\. Grill, S\. Ramos, E\. Yvinec, M\. Casbon, E\. Pot, I\. Penchev, G\. Liu, F\. Visin, K\. Kenealy, L\. Beyer, X\. Zhai, A\. Tsitsulin, R\. Busa\-Fekete, A\. Feng, N\. Sachdeva, B\. Coleman, Y\. Gao, B\. Mustafa, I\. Barr, E\. Parisotto, D\. Tian, M\. Eyal, C\. Cherry, J\. Peter, D\. Sinopalnikov, S\. Bhupatiraju, R\. Agarwal, M\. Kazemi, D\. Malkin, R\. Kumar, D\. Vilar, I\. Brusilovsky, J\. Luo, A\. Steiner, A\. Friesen, A\. Sharma, A\. Sharma, A\. M\. Gilady, A\. Goedeckemeyer, A\. Saade, A\. Feng, A\. Kolesnikov, A\. Bendebury, A\. Abdagic, A\. Vadi, A\. György, A\. S\. Pinto, A\. Das, A\. Bapna, A\. Miech, A\. Yang, A\. Paterson, A\. Shenoy, A\. Chakrabarti, B\. Piot, B\. Wu, B\. Shahriari, B\. Petrini, C\. Chen, C\. L\. Lan, C\. A\. Choquette\-Choo, C\. Carey, C\. Brick, D\. Deutsch, D\. Eisenbud, D\. Cattle, D\. Cheng, D\. Paparas, D\. S\. Sreepathihalli, D\. Reid, D\. Tran, D\. Zelle, E\. Noland, E\. Huizenga, E\. Kharitonov, F\. Liu, G\. Amirkhanyan, G\. Cameron, H\. Hashemi, H\. Klimczak\-Plucińska, H\. Singh, H\. Mehta, H\. T\. Lehri, H\. Hazimeh, I\. Ballantyne, I\. Szpektor, I\. Nardini, J\. Pouget\-Abadie, J\. Chan, J\. Stanton, J\. Wieting, J\. Lai, J\. Orbay, J\. Fernandez, J\. Newlan, J\. Ji, J\. Singh, K\. Black, K\. Yu, K\. Hui, K\. Vodrahalli, K\. Greff, L\. Qiu, M\. Valentine, M\. Coelho, M\. Ritter, M\. Hoffman, M\. Watson, M\. Chaturvedi, M\. Moynihan, M\. Ma, N\. Babar, N\. Noy, N\. Byrd, N\. Roy, N\. Momchev, N\. Chauhan, N\. Sachdeva, O\. Bunyan, P\. Botarda, P\. Caron, P\. K\. Rubenstein, P\. Culliton, P\. Schmid, P\. G\. Sessa, P\. Xu, P\. Stanczyk, P\. Tafti, R\. Shivanna, R\. Wu, R\. Pan, R\. Rokni, R\. Willoughby, R\. Vallu, R\. Mullins, S\. Jerome, S\. Smoot, S\. Girgin, S\. Iqbal, S\. Reddy, S\. Sheth, S\. Põder, S\. Bhatnagar, S\. R\. Panyam, S\. Eiger, S\. Zhang, T\. Liu, T\. Yacovone, T\. Liechty, U\. Kalra, U\. Evci, V\. Misra, V\. Roseberry, V\. Feinberg, V\. Kolesnikov, W\. Han, W\. Kwon, X\. Chen, Y\. Chow, Y\. Zhu, Z\. Wei, Z\. Egyed, V\. Cotruta, M\. Giang, P\. Kirk, A\. Rao, K\. Black, N\. Babar, J\. Lo, E\. Moreira, L\. G\. Martins, O\. Sanseviero, L\. Gonzalez, Z\. Gleicher, T\. Warkentin, V\. Mirrokni, E\. Senter, E\. Collins, J\. Barral, Z\. Ghahramani, R\. Hadsell, Y\. Matias, D\. Sculley, S\. Petrov, N\. Fiedel, N\. Shazeer, O\. Vinyals, J\. Dean, D\. Hassabis, K\. Kavukcuoglu, C\. Farabet, E\. Buchatskaya, J\. Alayrac, R\. Anil, Dmitry, Lepikhin, S\. Borgeaud, O\. Bachem, A\. Joulin, A\. Andreev, C\. Hardin, R\. Dadashi, and L\. HussenotGemma 3 technical report\.External Links:2503\.19786,[Link](https://arxiv.org/abs/2503.19786)Cited by:[§5\.1](https://arxiv.org/html/2609.20945#S5.SS1.SSS0.Px2.p1.1.1)\.
- Liet al\.\(2026\)T\. Li, V\. Chandrasekaran, and Z\. YuLayer\-targeted multilingual knowledge erasure in large language models\.arXiv preprint arXiv:2602\.22562\.Cited by:[§2\.2](https://arxiv.org/html/2609.20945#S2.SS2.p1.1)\.
- Lin \(2004\)C\. LinROUGE: a package for automatic evaluation of summaries\.InText Summarization Branches Out,Barcelona, Spain,pp\. 74–81\.External Links:[Link](https://aclanthology.org/W04-1013/)Cited by:[§5\.2](https://arxiv.org/html/2609.20945#S5.SS2.SSS0.Px4.p1.1)\.
- Lizzo and Heck \(2026\)T\. Lizzo and L\. HeckEvaluating cross\-lingual unlearning in multilingual language models\.arXiv preprint arXiv:2601\.06675\.Cited by:[§1](https://arxiv.org/html/2609.20945#S1.p2.1.2),[§2\.3](https://arxiv.org/html/2609.20945#S2.SS3.p1.1)\.
- Lu and Koehn \(2025\)T\. Lu and P\. KoehnLearn and unlearn: addressing misinformation in multilingual llms\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 10191–10206\.Cited by:[§1](https://arxiv.org/html/2609.20945#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.20945#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2609.20945#S2.SS2.p1.1)\.
- Mainiet al\.\(2024\)P\. Maini, Z\. Feng, A\. Schwarzschild, Z\. C\. Lipton, and J\. Z\. KolterTofu: a task of fictitious unlearning for llms\.arXiv preprint arXiv:2401\.06121\.Cited by:[Appendix A](https://arxiv.org/html/2609.20945#A1.p3.1.1),[§2\.3](https://arxiv.org/html/2609.20945#S2.SS3.p1.1),[§5\.1](https://arxiv.org/html/2609.20945#S5.SS1.SSS0.Px1.p1.1.1)\.
- Pireset al\.\(2019\)T\. Pires, E\. Schlinger, and D\. GarretteHow multilingual is multilingual bert?\.InProceedings of the 57th annual meeting of the association for computational linguistics,pp\. 4996–5001\.Cited by:[§2\.1](https://arxiv.org/html/2609.20945#S2.SS1.p1.1.2)\.
- Savelliet al\.\(2025\)C\. Savelli, M\. La Quatra, A\. Koudounas, and F\. GiobergiaFAME: fictional actors for multilingual erasure\.arXiv preprint arXiv:2512\.15235\.Cited by:[§1](https://arxiv.org/html/2609.20945#S1.p2.1.2),[§2\.3](https://arxiv.org/html/2609.20945#S2.SS3.p1.1)\.
- Shahamet al\.\(2024\)U\. Shaham, J\. Herzig, R\. Aharoni, I\. Szpektor, R\. Tsarfaty, and M\. EyalMultilingual instruction tuning with just a pinch of multilinguality\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 2304–2317\.Cited by:[§2\.1](https://arxiv.org/html/2609.20945#S2.SS1.p1.1.2)\.
- Shiet al\.\(2024\)W\. Shi, J\. Lee, Y\. Huang, S\. Malladi, J\. Zhao, A\. Holtzman, D\. Liu, L\. Zettlemoyer, N\. A\. Smith, and C\. ZhangMuse: machine unlearning six\-way evaluation for language models\.arXiv preprint arXiv:2407\.06460\.Cited by:[§4\.3](https://arxiv.org/html/2609.20945#S4.SS3.SSS0.Px3.p1.1.1)\.
- Singhet al\.\(2025\)A\. Singh, A\. Fry, A\. Perelman, A\. Tart, A\. Ganesh, A\. El\-Kishky, A\. McLaughlin, A\. Low, A\. Ostrow, A\. Ananthram,et al\.Openai gpt\-5 system card\.arXiv preprint arXiv:2601\.03267\.Cited by:[§4\.1](https://arxiv.org/html/2609.20945#S4.SS1.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2609.20945#S4.SS1.SSS0.Px3.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§5\.1](https://arxiv.org/html/2609.20945#S5.SS1.SSS0.Px2.p1.1.1)\.
- Zenget al\.\(2025\)H\. Zeng, S\. Han, L\. Chen, and K\. YuConverging to a lingua franca: evolution of linguistic regions and semantics alignment in multilingual large language models\.InProceedings of the 31st International Conference on Computational Linguistics,pp\. 10602–10617\.Cited by:[§1](https://arxiv.org/html/2609.20945#S1.p1.1)\.
- Zhanget al\.\(2024\)R\. Zhang, L\. Lin, Y\. Bai, and S\. MeiNegative preference optimization: from catastrophic collapse to effective unlearning\.arXiv preprint arXiv:2404\.05868\.Cited by:[§5\.1](https://arxiv.org/html/2609.20945#S5.SS1.SSS0.Px1.p1.1.1)\.

## Appendix ADetails on Machine Unlearning

In this section, we provide the general formulation of machine unlearning \(MU\)\. MU aims to eliminate unwanted knowledge from the model while preserving the overall utility\. Optimization\-based unlearning methods apply a forget lossℒf\\mathcal\{L\}\_\{f\}on the forget set𝒟f\\mathcal\{D\}\_\{f\}, which may be combined with a retain lossℒr\\mathcal\{L\}\_\{r\}applied to the retain set𝒟r\\mathcal\{D\}\_\{r\}\.

ℒtotal​\(𝒟f,𝒟r\)=ℒf​\(𝒟f\)\+λ​ℒr​\(𝒟r\)\\mathcal\{L\}\_\{\\text\{total\}\}\(\\mathcal\{D\}\_\{f\},\\mathcal\{D\}\_\{r\}\)=\\mathcal\{L\}\_\{f\}\(\\mathcal\{D\}\_\{f\}\)\+\\lambda\\,\\mathcal\{L\}\_\{r\}\(\\mathcal\{D\}\_\{r\}\)\(8\)
For example, GA[Jang et al\. \(2022\)](https://arxiv.org/html/2609.20945#bib.bib1)applies gradient ascent as the forget loss and leavesℒr\\mathcal\{L\}\_\{r\}asλ=0\\lambda=0\. In contrast, GAGDR[Maini et al\. \(2024\)](https://arxiv.org/html/2609.20945#bib.bib3)applies an additional gradient descent term asℒr\\mathcal\{L\}\_\{r\}\.

## Appendix BDetails on Benchmark Dataset

This section provides details on our benchmark dataset\. We further elaborate on the dataset generation process described in Section[4\.1](https://arxiv.org/html/2609.20945#S4.SS1)and supply the examples of the generated dataset\.

#### Multilingual QA Dataset

In this section, we describe the multilingual qa dataset generation pipeline in detail\. The English QAs weretranslated into 11 other languages, totaling 60,000 QA pairs: 5,000 QA pairs each across 12 languages\. The choice of languages is described in the main paper, Section[4\.2](https://arxiv.org/html/2609.20945#S4.SS2)\. We employ the GPT\-5\.4\-mini API for machine translation\. The translation process took place in three steps: 1\) prompt design, 2\) machine translation, and 3\) verification through a human\-in\-the\-loop process\. For translation quality control, the prompt design step was further broken down into three stages\. First, every attribute value was translated into the target languages using GPT\-5\.4\-mini\.For verification, the translated values were back\-translated into English using Google Translate[Cloud \(2025\)](https://arxiv.org/html/2609.20945#bib.bib10), and the back\-translations were then provided to GPT\-5\.4\-mini to check for semantic equivalence with the original English values\.If a mismatch is detected, the instance is re\-translated and looped through the above process until corrected\. Aside from this, the English attribute values were annotated with their definitions to form a dictionary\. These two processes were conducted to prevent mistranslation into synonyms\. For example, ‘March’ may be perceived as either the third month of the year or the act of walking\. Finally, the translated attributes and the dictionary of definitions were integrated into a single translation prompt\. This prompt was provided to GPT\-5\.4\-mini along with the target QA pairs that are to be translated\. Once the translated outputs of the model are collected, the human\-in\-the\-loop validation process was repeated until the multilingual QA pairs were fully refined\. The validation pipeline is described in detail in the following paragraph\.

#### Human\-in\-the\-Loop Validation

After the initial translation of the English QA pairs with the GPT\-5\.4\-mini API, verification is conducted by integrating back\-translation and a human\-in\-the\-loop strategy\. The translated outputs are back\-translated into English using Google Translate\. As done in the attribute value translation stage, the translated QA pairs are then passed to GPT\-5\.4\-mini to verify their semantic equivalence with the original English pairs\. If a mismatching instance is detected, its translation is refined until it is judged as correct\. Once every pair passes machine evaluation, human annotators validate and refine the translations via back\-translation\. Examples of the failure modes that had to be eliminated by human annotators are presented in Appendix[B\.5](https://arxiv.org/html/2609.20945#A2.SS5)\.The human\-refined dataset is then put through another round of the same pipeline, which consists of refinement with the GPT\-5\.4\-mini API, back\-translation into English, semantic equivalence assessment, and human refinement\. This cycle is repeated until every QA pair is confirmed to be correctly translated\.

### B\.1Attribute Pool for Synthetic Profile Generation

To construct diverse synthetic profiles, we employed a total of 21 attributes, includingName,Birth Month,Job,Father’s Job,Mother’s Job,Favorite Animal,Favorite Season,Favorite Fruit,Favorite Vegetable,Favorite Drink,Favorite Taste,Favorite Color,Hobby,Favorite Instrument,Biggest Fear,Favorite Accessory,Favorite Household Item,Morning Ritual,Childhood Memory,Childhood Toy, andFavorite Gift\. The pool of candidate values for each attribute was manually reviewed and filtered by human annotators to eliminate any ambiguity between values\.Furthermore, to ensure controllability, we deliberately selected attributes whose values share few letters or characters across languages, minimizing the chance of overlap\.The complete value pool for each attribute used in data construction is provided in Table[5](https://arxiv.org/html/2609.20945#A2.T5)\. ForName, first and last names were randomly sampled using the Faker library\. The resulting 250 synthetic profiles were constructed such that no two profiles shared the same name or the combination of values across the 20 non\-name attributes\.

Table 5:Valid values for the attributes used to build synthetic profiles \(1/4\)Table 6:\(continued\) Valid values for the attributes used to build synthetic profiles \(2/4\)Table 7:\(continued\) Valid values for the attributes used to build synthetic profiles \(3/4\)Table 8:\(continued\) Valid values for the attributes used to build synthetic profiles \(4/4\)
### B\.2Example of Synthetic Profile

Figure[5](https://arxiv.org/html/2609.20945#A2.F5)shows an example of a synthetic profile generated through the data generation pipeline\. Each synthetic profile is constructed by randomly sampling a value for each attribute from the set of possible values defined in the preceding section\.

Figure 5:Example of Generated Synthetic Profile
### B\.3Examples of English QA Datasets

Figure[6](https://arxiv.org/html/2609.20945#A2.F6)presents examples of the English QA dataset generated from the synthetic profiles\. The QA dataset was initially constructed using predefined QA templates, then refined by GPT\-5\.4\-mini to correct any grammatical errors, and finally reviewed by human annotators for quality assurance\. The QA templates used in this process are provided in Figure[9](https://arxiv.org/html/2609.20945#A2.T9)\. The prompt used for the GPT\-based refinement is shown in Figure[8](https://arxiv.org/html/2609.20945#A2.F8)\.

Figure 6:Examples of the generated English QA dataset\.Leftshows the naive QA dataset generated directly from templates, whileRightshows the QA dataset refined using GPT\.Figure 7:Examples of code\-mixed responses after NPO unlearning on Gemma3\-12b model\.Leftshows the question, whileRightshows the gold answer alongside the model’s code\-mixed response\.Table 9:QA dataset template for each attribute \(1/2\)Table 10:\(continued\) QA dataset template for each attribute \(2/2\)Figure 8:Prompt used to refine the English QA dataset using GPT
### B\.4Examples of Multilingual QA Datasets

Figure[9](https://arxiv.org/html/2609.20945#A2.F9)presents examples of the multilingual QA dataset translated from the English QA dataset\. The prompt used to construct the multilingual QA dataset is shown in Figure[10](https://arxiv.org/html/2609.20945#A2.F10)\. Furthermore, to preserve the precise meaning of each word during translation, examples of the English descriptions provided for the words are shown in Table[11](https://arxiv.org/html/2609.20945#A2.T11)\.

Figure 9:Examples of the translated Multilingual QA dataset\.Figure 10:Prompt used to translate the English QA dataset to each language using GPTTable 11:Description of each value in English \(1/20\)Table 12:\(continued\) Description of each value in English \(2/20\)Table 13:\(continued\) Description of each value in English \(3/20\)Table 14:\(continued\) Description of each value in English \(4/20\)Table 15:\(continued\) Description of each value in English \(5/20\)Table 16:\(continued\) Description of each value in English \(6/20\)Table 17:\(continued\) Description of each value in English \(7/20\)Table 18:\(continued\) Description of each value in English \(8/20\)Table 19:\(continued\) Description of each value in English \(9/20\)Table 20:\(continued\) Description of each value in English \(10/20\)Table 21:\(continued\) Description of each value in English \(11/20\)Table 22:\(continued\) Description of each value in English \(12/20\)Table 23:\(continued\) Description of each value in English \(13/20\)Table 24:\(continued\) Description of each value in English \(14/20\)Table 25:\(continued\) Description of each value in English \(15/20\)Table 26:\(continued\) Description of each value in English \(16/20\)Table 27:\(continued\) Description of each value in English \(17/20\)Table 28:\(continued\) Description of each value in English \(18/20\)Table 29:\(continued\) Description of each value in English \(19/20\)Table 30:\(continued\) Description of each value in English \(20/20\)
### B\.5Analysis of Generation Errors in Multilingual QA

When constructing the multilingual QA dataset, we started from the English dataset that had been verified by human annotators\. Using the prompt shown in Figure[10](https://arxiv.org/html/2609.20945#A2.F10), we first generated a naive multilingual translated QA dataset, which was then back\-translated into English via the Google Translator API\. Human annotators subsequently verified whether the back\-translated questions preserved the semantics of the original English QA dataset\. In this section, we briefly describe the errors observed in the dataset naively generated through the GPT API\. The error cases we encountered fall into two main categories: 1\) cases in which the question itself contains the value of the attribute that should be provided in the answer, and 2\) cases in which a single word carries multiple meanings\.An example of the first case occurred in Chinese \(zh\), as in “Christopher Davis 最大的怕死是什么？” \(What is Christopher Davis’s biggest fear of death?\), where the value that should be answered is already revealed within the question, thereby leading the model toward the intended answer\. The second case occurred in German \(de\), where we observed instances such as “Matthew Moores Lieblingsspielzeug aus der Kindheit war der Drachen” \(Matthew Moore’s favorite childhood toy was the dragon\), in which the German wordDrachensimultaneously denotes both a whistle and a dragon\.To address such cases, human annotators used Claude to re\-translate the sentence into the target language and then back into English via the Google Translator API, iterating this process until the semantics matched those of the original\.

## Appendix CHyperparamter

For the SFT stage, we employed LoRA with a rank ofr=32r=32and a scaling factor ofα=64\\alpha=64, applying a dropout rate of0\.050\.05to all linear layers\. We used a learning rate of1×10−41\\times 10^\{\-4\}with a cosine learning rate scheduler and a warmup ratio of0\.030\.03\. The model was optimized using AdamW with a maximum gradient norm of1\.01\.0\. Training was conducted for1010epochs with a batch size of6464and a maximum sequence length of512512\. The same configuration was applied identically to both Qwen2\.5\-7B\-Instruct and Gemma\-3\-12B\.

For the unlearning stage, we likewise employed LoRA with a rank ofr=8r=8and a scaling factor ofα=16\\alpha=16, applying a dropout rate of0\.050\.05to all linear layers\. We used a constant learning rate scheduler with a warmup ratio of0\.00\.0and a weight decay of0\.00\.0\. The model was optimized using AdamW with a maximum gradient norm of1\.01\.0\. Training was conducted for55epochs with a maximum sequence length of512512\. We set the retain\-performance threshold to90%90\\%of the retain model’s probability on non\-target knowledge in the training language\. For methods that preserve utility by leveraging a retain dataset \(GAGDR and LingTea\), we set the retain\-loss coefficient to1\.01\.0, and for NPO we setβ=0\.1\\beta=0\.1\. Methods that use only the forget dataset were trained with a batch size of1616, whereas methods that additionally use the retain dataset were trained with a batch size of3232\. For LingTea, we used a temperature of 2\.0\.Unlearning was conducted for 10 epochs for every setting and model\. We conducted an extensive hyperparameter search to determine the learning rate for each method, setting, and model\. The results of the search are reported in Table[31](https://arxiv.org/html/2609.20945#A3.T31)\.

Gemma3\-12BQwen2\.5\-7BSetting 1Setting 2Setting 1Setting 2Methodp1p3p5p1p3p5p1p3p5p1p3p5GA4\.32e\-57\.60e\-65\.00e\-64\.19e\-56\.70e\-62\.90e\-64\.10e\-59\.10e\-65\.50e\-64\.41e\-51\.41e\-58\.10e\-6GAGDR9\.05e\-53\.04e\-52\.47e\-51\.005e\-43\.50e\-52\.04e\-59\.32e\-53\.00e\-51\.70e\-51\.110e\-43\.69e\-52\.29e\-5LingTea1\.302e\-46\.02e\-53\.72e\-51\.005e\-44\.12e\-53\.26e\-51\.699e\-47\.11e\-55\.00e\-51\.530e\-41\.425e\-45\.79e\-5NPO4\.00e\-57\.50e\-64\.60e\-64\.05e\-57\.20e\-63\.30e\-64\.90e\-59\.90e\-65\.80e\-65\.81e\-51\.70e\-59\.50e\-6Table 31:Learning rates selected by an LR sweep with increments of1×10−71\\times 10^\{\-7\}across models, settings, forget ratios, and unlearning methods\.
## Appendix DSemantic Equivalence Score Measurement

We measure the Semantic Equivalence Score with an LLM\-as\-a\-Judge strategy, employing the GPT\-4o\-mini API with the prompt given in Figure[11](https://arxiv.org/html/2609.20945#A4.F11)\.

Figure 11:Prompt used to judge semantic equivalence of model responses
## Appendix EAdditional Results on Unlearning in Training Languages

Table[32](https://arxiv.org/html/2609.20945#A7.T32)and Table[33](https://arxiv.org/html/2609.20945#A7.T33)present the results forp​3p3andp​5p5, respectively\.

## Appendix FAdditional Results on Unlearning in Hold\-out Languages

Table[34](https://arxiv.org/html/2609.20945#A7.T34)and Table[35](https://arxiv.org/html/2609.20945#A7.T35)present the results forp​3p3andp​5p5, respectively\.

## Appendix GUse of AI Assistants

We utilize Claude and ChatGPT for writing and coding assistance\. We employ GPT API for dataset generation and evaluation\.

Δ\\DeltaPSΔ\\DeltaSECellMethodHighLowHighLowG/S1LingTea\+53\.9\+46\.4\+5\.4\+10\.4NPO\+23\.4\+0\.9\+0\.0\+0\.4GAGDR\+30\.0\+10\.3\+0\.0\+0\.0GA\+12\.7\+0\.5\+0\.0\+0\.0G/S2LingTea\+23\.8\+29\.9\+3\.7\+2\.9NPO\+7\.3\+17\.1\+0\.4\+2\.9GAGDR\+23\.8\+25\.1\+1\.7\+1\.7GA\+6\.0\+18\.7\+0\.0\+3\.3Q/S1LingTea\+80\.9\+84\.7\+62\.9\+67\.1NPO\+35\.8\+5\.8\+2\.5\+4\.6GAGDR\+73\.1\+13\.9\+12\.1\+11\.7GA\+25\.8\+5\.2\+1\.2\+4\.2Q/S2LingTea\+63\.7\+65\.2\+93\.3\+95\.8NPO\+30\.4\+21\.3\+61\.5\+53\.3GAGDR\+39\.5\+31\.1\+67\.4\+63\.7GA\+34\.0\+28\.0\+64\.4\+61\.3Table 32:Training\-language drop rates per unlearning method forp​3p3\. For each metric,boldmarks the higher value within the High/Low pair\. G and Q denote Gemma3 and Qwen2\.5, respectively\.Δ\\DeltaPSΔ\\DeltaSECellMethodHighLowHighLowG/S1LingTea\+29\.3\+26\.4\+5\.0\+6\.8NPO\+20\.7\+1\.3\+0\.0\+0\.0GAGDR\+20\.8\+30\.5\+2\.7\+2\.2GA\+24\.1\+1\.0\+0\.0\+0\.0G/S2LingTea\+26\.2\+36\.5\+1\.2\+3\.7NPO\+0\.5\+19\.2\+0\.0\+3\.5GAGDR\+16\.0\+8\.5\+1\.0\+1\.5GA\+0\.1\+18\.6\+0\.0\+3\.2Q/S1LingTea\+74\.5\+72\.8\+44\.8\+43\.8NPO\+9\.6\+9\.6\+0\.7\+0\.7GAGDR\+29\.3\+13\.8\+8\.5\+10\.0GA\+4\.8\+11\.1\+0\.2\+0\.5Q/S2LingTea\+35\.6\+20\.3\+54\.1\+49\.0NPO\+21\.8\+11\.8\+51\.4\+39\.8GAGDR\+31\.4\+28\.2\+59\.9\+62\.3GA\+26\.9\+14\.8\+57\.4\+44\.2Table 33:Training\-language drop rates per unlearning method forp​5p5\. For each metric,boldmarks the higher value within the High/Low pair\. G and Q denote Gemma3 and Qwen2\.5, respectively\.Δ\\DeltaPSΔ\\DeltaSEresourcescriptresourcescriptCellMethodHighLowLatnLatHighLowLatnLatG/S1LingTea\+51\.0\+51\.8\+49\.5\+53\.4\+10\.3\+12\.4\+11\.1\+11\.3NPO\+3\.0\+2\.8\+3\.7\+2\.1\+0\.0\+5\.0\+1\.6\+2\.6GAGDR\+7\.1\+14\.2\+10\.2\+11\.2\+0\.0\+5\.0\+3\.7\+0\.5GA\+2\.7\+2\.2\+2\.9\+2\.0\-0\.4\+3\.7\+0\.0\+2\.6G/S2LingTea\+10\.5\+29\.2\+18\.0\+21\.7\+6\.3\+6\.2\+7\.9\+4\.6NPO\-6\.2\+2\.0\-5\.6\+1\.4\+1\.8\+0\.6\+2\.1\+0\.5GAGDR\-2\.4\+28\.4\+5\.4\+20\.6\+6\.3\+5\.6\+6\.9\+5\.1GA\-2\.2\+1\.8\-1\.8\+1\.4\+0\.9\+0\.0\+1\.6\-0\.5Q/S1LingTea\+85\.4\+84\.3\+85\.7\+84\.0\+64\.9\+38\.5\+64\.0\+59\.5NPO\+4\.3\+1\.1\+5\.7\-0\.3\+8\.8\+26\.9\+17\.4\+6\.3GAGDR\+13\.1\+14\.6\+16\.9\+10\.7\+14\.6\+46\.2\+16\.3\+20\.7GA\+3\.7\+1\.4\+4\.9\+0\.3\+7\.6\+23\.1\+15\.1\+5\.4Q/S2LingTea\+70\.9\+44\.5\+59\.0\+56\.4\+87\.9\+59\.1\+87\.5\+78\.5NPO\+21\.3\+18\.0\+25\.7\+13\.7\+40\.2\+27\.3\+37\.5\+38\.5GAGDR\+27\.0\+25\.4\+31\.4\+21\.0\+53\.3\+45\.5\+50\.0\+53\.8GA\+24\.7\+25\.0\+30\.6\+19\.0\+42\.1\+13\.6\+29\.7\+44\.6

Table 34:Hold\-out\-language drop rates per unlearning method forp​3p3\.Boldmarks the higher value within each comparison pair \(High/Low and Lat/nLat\)\. G and Q denote Gemma3 and Qwen2\.5, respectively\.Δ\\DeltaPSΔ\\DeltaSEresourcescriptresourcescriptCellMethodHighLowLatnLatHighLowLatnLatG/S1LingTea\+30\.1\+24\.7\+28\.6\+26\.2\+10\.4\+10\.1\+9\.6\+10\.8NPO\+2\.5\+1\.3\+2\.3\+1\.5\+1\.1\+3\.7\+1\.9\+2\.5GAGDR\+35\.2\+28\.5\+32\.1\+31\.6\+4\.4\+6\.7\+4\.2\+6\.5GA\+6\.7\+3\.8\+4\.9\+5\.6\+0\.5\+4\.1\+1\.9\+2\.2G/S2LingTea\+14\.7\+34\.1\+20\.5\+28\.3\+3\.3\+6\.3\+5\.8\+3\.4NPO\-3\.7\-0\.6\-2\.8\-1\.5\+0\.6\+2\.2\+1\.0\+1\.6GAGDR\-4\.7\+15\.3\-4\.7\+15\.3\+1\.7\+3\.0\+2\.9\+1\.6GA\-0\.8\+0\.2\-0\.6\-0\.1\-0\.3\+0\.7\+0\.0\+0\.3Q/S1LingTea\+66\.3\+70\.9\+71\.2\+66\.0\+38\.9\+11\.4\+32\.6\+37\.3NPO\-0\.4\+1\.5\+1\.8\-0\.7\+2\.8\-2\.3\+4\.2\+0\.5GAGDR\+9\.9\+10\.6\+14\.7\+5\.7\+7\.4\+40\.9\+13\.9\+10\.3GA\+1\.0\+2\.2\+3\.1\+0\.1\+1\.4\+13\.6\+4\.9\+1\.6Q/S2LingTea\+24\.7\+12\.7\+21\.7\+15\.7\+38\.6\+22\.2\+28\.0\+43\.6NPO\+15\.7\+15\.9\+20\.3\+11\.3\+16\.5\+11\.1\+13\.0\+18\.1GAGDR\+23\.8\+30\.7\+28\.9\+25\.7\+26\.6\-11\.1\+15\.0\+24\.5GA\+16\.9\+19\.9\+21\.3\+15\.5\+24\.1\+13\.9\+21\.0\+23\.4

Table 35:Hold\-out\-language drop rates per unlearning method forp​5p5\.Boldmarks the higher value within each comparison pair \(High/Low and Lat/nLat\)\. G and Q denote Gemma3 and Qwen2\.5, respectively\.