AMNESIA: A Large Scale Medical Unlearning Benchmark Suite with Disease-Informed Analysis

# AMNESIA: A Large Scale Medical Unlearning Benchmark Suite with Disease-Informed Analysis
Source: [https://arxiv.org/html/2605.30599](https://arxiv.org/html/2605.30599)
Saeedeh Davoudi1,Reihaneh Iranmanesh11footnotemark:11,Ophir Frieder1,Nazli Goharian1

1IR Lab, Computer Science Department, Georgetown University, Washington D\.C\. \{saeedeh, rei, ophir, nazli\}@ir\.cs\.georgetown\.edu

###### Abstract

Medical knowledge is continuously evolving\. This creates a need to update or selectively forget information encoded in already\-trained medical LLMs\. Machine unlearning aims to remove the influence of specific training data from a model without full retraining\. Yet, existing unlearning benchmarks rely on synthetic or small\-scale general data, leaving clinical unlearning understudied\. We introduceAMNESIA, the first large\-scale, open source benchmark for medical unlearning, with 70,560 question\-answer pairs from 8,820 patient notes across 11 disease categories\. AMNESIA includes bothfactualquestions testing direct recall andreasoningquestions testing clinical inference\. We use it to evaluate four widely used unlearning methods at both random patient and disease\-level, and introduce a new metric for detecting leakage of medical terminology\. We show that unlearning individual patients erodes knowledge of others with the same condition, calling for methods that can better separate patients from shared clinical knowledge\.

AMNESIA: A Large Scale Medical Unlearning Benchmark Suite with Disease\-Informed Analysis

Saeedeh Davoudi††thanks:Equal contribution\.1, Reihaneh Iranmanesh11footnotemark:11, Ophir Frieder1, Nazli Goharian11IR Lab, Computer Science Department, Georgetown University, Washington D\.C\.\{saeedeh, rei, ophir, nazli\}@ir\.cs\.georgetown\.edu

## 1Introduction

Machine learning models trained on sensitive medical data face a critical challenge: how to remove the influence of specific training examples while preserving overall model utility\. This capability, known asmachine unlearning, is essential for healthcare applications where patients may request data deletion under privacy regulations like GDPR’s “right to be forgotten”\(Protection Regulation,[2016](https://arxiv.org/html/2605.30599#bib.bib1)\), or where erroneous or outdated medical records must be excised from trained models without costly full retraining\. Despite growing interest in machine unlearning across various domains, the medical field lacks standardized benchmarks for evaluating unlearning methods for clinical question\-answering tasks\.

To address these gaps, we introduceAMNESIA:ALarge ScaleMedical UNlearning BEnchmarkSuite with Disease\-InformedAnalysis\. AMNESIA provides the first large\-scale benchmark for evaluating machine unlearning in medical question\-answering \(QA\), comprising 70,560 question\-answer pairs derived from 8,820 patient notes across 11 disease categories\. Unlike prior unlearning benchmarks built on synthetic or general\-domain QAs \(Appendix[A](https://arxiv.org/html/2605.30599#A1)\), AMNESIA is grounded in real patient notes, spanning thousands of patients across many disease conditions\. This is essential in medical contexts, where consent withdrawals, diagnostic revisions, and record corrections continually generate new unlearning targets\. AMNESIA also distinguishesfactualquestions \(direct recall\) fromreasoningquestions \(clinical inference\) and supports bothrandom patient\-levelanddisease\-levelforgetting scenarios, enabling researchers to test whether models can forget patients with particular medical conditions\.

Our contributions are as follows:

- •First large\-scale clinically\-grounded unlearning benchmark:70,560 Question\-Answer pairs \(QAs\) from 8,820 patient notes across 11 disease categories\. Each patient has 4 factual \(recall\) and 4 reasoning \(inference\) QAs\.111All data, code, and baselines will be publicly available without restrictions or data usage agreements\.
- •Multi\-level evaluation across diverse forget/retain splits:Evaluation across random patient\-level and disease\-level forget/retain splits\.
- •Comprehensive unlearning analysis:Four representative unlearning methods are evaluated and analyzed\.
- •Medically\-aware unlearning evaluation:Novel disease\-focused keyword evaluation for forget performance\.
- •Release of medical models and dataset:Our models are publicly available\. Our detailed data construction pipeline makes AMNESIA reproducible and easy to extend\.

![Refer to caption](https://arxiv.org/html/2605.30599v1/x1.png)Figure 1:AMNESIA Dataset Construction \(a\), Unlearning Benchmark \(b\), and Evaluation Pipeline \(c\)
## 2Related Work

#### Machine Unlearning Benchmarks\.

Machine unlearning aims to remove the influence of specific training data from a model without full retraining\(Cao and Yang,[2015](https://arxiv.org/html/2605.30599#bib.bib2)\)\. TOFU\(Mainiet al\.,[2024](https://arxiv.org/html/2605.30599#bib.bib3)\)establishes the standard benchmark for LLM unlearning using synthetic author profiles, with methods including Gradient Difference\(Liuet al\.,[2022](https://arxiv.org/html/2605.30599#bib.bib13)\), KL Minimization\(Nguyenet al\.,[2020](https://arxiv.org/html/2605.30599#bib.bib5)\), and Negative Preference Optimization\(Zhanget al\.,[2024](https://arxiv.org/html/2605.30599#bib.bib6)\)\. R\-TOFU\(Yoonet al\.,[2025](https://arxiv.org/html/2605.30599#bib.bib7)\)extends TOFU to Large Reasoning Models, showing that answer\-only objectives leave residual forget traces in chain\-of\-thought reasoning\. MUSE\(Shiet al\.,[2025](https://arxiv.org/html/2605.30599#bib.bib8)\)evaluates forgetting in LLMs but does not address clinical or cross\-modal structures\. Recent work further shows that existing methods fail on structured, multi\-hop knowledge\(Choiet al\.,[2024](https://arxiv.org/html/2605.30599#bib.bib9)\), motivating more realistic evaluation settings\. We follow OpenUnlearning\(Dornaet al\.,[2026](https://arxiv.org/html/2605.30599#bib.bib10)\), a unified framework integrating 13 unlearning algorithms and 16 evaluations across TOFU, MUSE, and WMDP\(Liet al\.,[2024](https://arxiv.org/html/2605.30599#bib.bib11)\), to inform our selection and standardized comparison of unlearning methods\.

#### Medical Question Answering\.

Clinical NLP has benefited from large\-scale datasets such as MIMIC\-CXR\(Johnsonet al\.,[2019](https://arxiv.org/html/2605.30599#bib.bib12)\), with models like LLaVA\-Med\(Liet al\.,[2023](https://arxiv.org/html/2605.30599#bib.bib14)\)and Asclepius\(Kweonet al\.,[2024](https://arxiv.org/html/2605.30599#bib.bib15)\)demonstrating that LLMs fine\-tuned on patient notes acquire patient\-specific knowledge\. Recent benchmarks have pushed medical QA in several directions: evaluating LLM reasoning and explanation on challenging clinical cases\(Chenet al\.,[2025](https://arxiv.org/html/2605.30599#bib.bib16)\), detecting hallucinations in medical outputs\(Panditet al\.,[2025](https://arxiv.org/html/2605.30599#bib.bib17)\), assessing multi\-hop biomedical reasoning across knowledge graphs\(Kimet al\.,[2025](https://arxiv.org/html/2605.30599#bib.bib18)\), and supporting argumentative explanation of diagnoses\(Sviridovaet al\.,[2024](https://arxiv.org/html/2605.30599#bib.bib19)\)\. However, these benchmarks do not address machine unlearning in a medical QA setting\.

#### Machine Unlearning in Medical AI\.

Privacy regulations, including HIPAA\(Act,[1996](https://arxiv.org/html/2605.30599#bib.bib20)\)and GDPR\(Protection Regulation,[2016](https://arxiv.org/html/2605.30599#bib.bib1)\), motivate the selective removal of patient data from trained models, with prior defenses such as federated learning\(McMahanet al\.,[2017](https://arxiv.org/html/2605.30599#bib.bib21)\)offering only partial protection\. Other work has explored unlearning multimodal patient information in clinical imaging settings\(Hardanet al\.,[2025](https://arxiv.org/html/2605.30599#bib.bib22)\)\. MedForget\(Wuet al\.,[2025](https://arxiv.org/html/2605.30599#bib.bib23)\)is the closest prior work, providing a hierarchical multimodal benchmark of 3,840 Visual Question Answering \(VQA\) pairs across 64 patients from MIMIC\-CXR\. In contrast, AMNESIA scales to 70,560 clinical note–grounded QAs and introduces disease\-level unlearning, a clinically motivated setting not addressed by prior work\.

## 3AMNESIA Dataset

Figure[1](https://arxiv.org/html/2605.30599#S1.F1)illustrates the AMNESIA framework\. Given a patient note, AMNESIA includes factual and reasoning questions, each with gold\-standard answers\. After unlearning, models should produce substantially different \(uninformative\) responses to questions about forgotten patients while maintaining accurate answers for retained patients\.

### 3\.1Data Construction \(Figure[1](https://arxiv.org/html/2605.30599#S1.F1)\.a\)

#### Dataset Selection and Sampling

We use PMC\-Patients\-v2\(Zhaoet al\.,[2023](https://arxiv.org/html/2605.30599#bib.bib24)\), a large\-scale publicly available dataset of de\-identified patient notes extracted from reports in PubMed Central\(Roberts,[2001](https://arxiv.org/html/2605.30599#bib.bib25)\)\. From this corpus, we randomly sampled 8,820 patient notes to create a computationally feasible yet substantial dataset for our study\.

#### Disease Category Assignment

We usedGPT\-5\.4\-nanoto assign a disease from Centers for Disease Control and Prevention \(CDC\)’s classification of common diseases222[https://www\.cdc\.gov/nchs/fastats/diseases\-and\-conditions\.htm](https://www.cdc.gov/nchs/fastats/diseases-and-conditions.htm)\(11 diseases listed in Table[1](https://arxiv.org/html/2605.30599#S3.T1)\) to each patient based on its note title\. The note title is a single sentence describing the patient note\. The prompt is designed to enforceGPT\-5\.4\-nanoto map each note’s title to a single disease category \(see Appendix[B\.1](https://arxiv.org/html/2605.30599#A2.SS1)\)\. This categorization enables AMNESIA’s disease\-informed analysis of machine unlearning methods\. We evaluate these categories in the[Human\-in\-the\-Loop Evaluation](https://arxiv.org/html/2605.30599#S3.SS1.SSS0.Px4)\.

#### Question\-Answer Generation

For each patient, we useGPT\-5\-nanoto generate 8 QAs: 4 factual QAs testing direct information recall from the patient’s note and 4 reasoning QAs requiring inference across multiple findings within the note, making them more challenging than factual questions\. The generation prompt \(Appendix[B\.2](https://arxiv.org/html/2605.30599#A2.SS2)\) enforces strict quality criteria: questions must be answerable only from the specific patient’s details \(not general medical knowledge\), must avoid any personal information, and must maintain clinical specificity\. This generation strategy produced 70,560 total QAs \(35,280 factual and 35,280 reasoning\), ensuring coverage of both knowledge retrieval and clinical reasoning capabilities\. Table[1](https://arxiv.org/html/2605.30599#S3.T1)presents the final dataset composition across disease categories\.

Table 1:AMNESIA dataset statistics with disease categories\. Min, Avg, and Max denote the shortest, average, and longest note sizes \(in tokens\) within each disease category\.
#### Human\-in\-the\-Loop Evaluation

We validate both disease labels and QAs under majority vote with a three\-judge LLM panel: a medical\-specialized model \(MedGemma\-27B\(Sellergrenet al\.,[2025](https://arxiv.org/html/2605.30599#bib.bib26)\)\), a general\-purpose reasoning model \(Qwen3\-32B\(Yanget al\.,[2025](https://arxiv.org/html/2605.30599#bib.bib27)\)\), and a frontier model \(GPT\-5\-mini\(Singhet al\.,[2025](https://arxiv.org/html/2605.30599#bib.bib28)\)\)\. For disease labels, we sample 900 patient notes \(∼\\sim10% of total notes\) from the three most populated diseases \(300 each from Cancer, Infectious/Immune, and Cardiovascular\)\. For QAs, we sample 500 patients and take 1 factual and 1 reasoning QA from each, yielding 1,000 QAs\. The judge panel supports91\.8%\\mathbf\{91\.8\\%\}of disease labels and rates97\.6%\\mathbf\{97\.6\\%\}of QA triples as valid against four criteria \(clarity, correctness, derivability, clinical meaning\); a biology student’s annotations of the same patient notes agree with the judges’ majority vote on90\.4%\\mathbf\{90\.4\\%\}of disease labels\. Together, these results indicate that AMNESIA’s QAs and disease labels are trustworthy for unlearning evaluation\. Full protocols, prompts, and per\-judge breakdowns are in Appendices[F](https://arxiv.org/html/2605.30599#A6)and[G](https://arxiv.org/html/2605.30599#A7)\.

### 3\.2Data Partition \(Figure[1](https://arxiv.org/html/2605.30599#S1.F1)\.b\)

Our partition strategy creates hierarchical forget/retain splits at two levels: random patient\-level and disease\-level\.

#### Random Patient\-Level Splits

Similar to existing unlearning benchmarks\(Mainiet al\.,[2024](https://arxiv.org/html/2605.30599#bib.bib3); Wuet al\.,[2025](https://arxiv.org/html/2605.30599#bib.bib23)\), we create forget splits by randomly selecting patients at 5%, 10%, 15%, 20%, and 25% of the total number of patients, where each smaller forget set is a proper subset of the next larger one \(i\.e\., forget\_5⊂\\subsetforget\_10⊂\\subsetforget\_15⊂\\subsetforget\_20⊂\\subsetforget\_25\)\. For each forget split, the corresponding retain split contains all remaining patients\.

#### Disease\-Level Splits

We build disease\-level splits for the three most prominent diseases in our dataset: Cancer, Infectious/Immune, and Cardiovascular\. Each disease\-level split matches the size of the corresponding random patient\-level split, sampled only from the target disease\. The size of the largest possible forget set for each disease depends on the total number of patients having that disease in the dataset \(Table[1](https://arxiv.org/html/2605.30599#S3.T1)\):

- •Cancer \(30\.3% of all patients\):5%, 10%, 15%, 20%, 25% forget splits\.
- •Infectious/Immune Diseases \(22\.3% of all patients\):5%, 10%, 15%, 20% forget splits\.
- •Cardiovascular \(16\.2% of all patients\):5%, 10%, 15% forget splits\.

Each disease\-level split maintains two key properties: \(1\) the subset relationship holds within each disease \(e\.g\., cancer\_forget\_5⊂\\subset…⊂\\subsetcancer\_forget\_25\), and \(2\) all patients with a given disease label that appear in the random patient forget split are also included in the corresponding disease\-level forget split at the same percentage \(e\.g\., all cancer patients in forget\_5 are guaranteed to appear in cancer\_forget\_5\. Appendix[C](https://arxiv.org/html/2605.30599#A3)shows the size of each data split\.\)\. The 900 samples selected for human\-in\-the\-loop evaluation of disease categories are drawn from cancer\_forget\_5, cardio\_forget\_5, and infectious/immune\_forget\_5 splits, which are shared among all forget sets of that disease \(See[Human\-in\-the\-Loop Evaluation](https://arxiv.org/html/2605.30599#S3.SS1.SSS0.Px4)\)\.

#### Holdout Dataset

To evaluate generalization, we create a holdout set by withholding a factual and a reasoning QA per patient \(17,640 total\), never seen during fine\-tuning and unlearning\. Complete statistics for all splits are provided in Appendix[C](https://arxiv.org/html/2605.30599#A3)\.

## 4Unlearning Benchmark \(Figure[1](https://arxiv.org/html/2605.30599#S1.F1)\.b\)

We establish a medical unlearning benchmark through a four\-stage workflow: \(1\) Pre\-training on patient notes, \(2\) instruction fine\-tuning on QAs, \(3\) applying unlearning methods on the fine\-tuned model, and \(4\) evaluation across multiple metrics\.

### 4\.1Base Language Model

We selectLLaMA 3\-8Bas the base language model, following established practices in medical AI\(Kweonet al\.,[2024](https://arxiv.org/html/2605.30599#bib.bib15)\)\. We perform continued pre\-training on 8,820 patient notes using standard next\-token prediction, followed by instruction fine\-tuning on 52,920 QAs \(see Appendix[D](https://arxiv.org/html/2605.30599#A4)for hyperparameter details\)\. The resulting fine\-tuned model serves as the baseline for all unlearning experiments\.

### 4\.2Unlearning Methods

We evaluate four top\-performing unlearning methods from OpenUnlearning\(Dornaet al\.,[2026](https://arxiv.org/html/2605.30599#bib.bib10)\), covering diverse algorithmic families\. All methods operate on the forget set𝒟f\\mathcal\{D\}\_\{f\}\(patients to unlearn\) and the retain set𝒟r\\mathcal\{D\}\_\{r\}\(patients to preserve\), applied to the fine\-tuned model\. Following the instruction fine\-tuning configuration, all unlearning methods are trained with forget/retain patient notes and their QAs as input:

RMU\(Liet al\.,[2024](https://arxiv.org/html/2605.30599#bib.bib11)\)steers internal representations of forget set inputs toward random targets at intermediate layers while preserving retain set representations, avoiding output\-level gradient manipulation\. In our experiments, steering is applied at layer 7, consistent with the OpenUnlearning default values\.

GradDiff\(Liuet al\.,[2022](https://arxiv.org/html/2605.30599#bib.bib13)\)jointly maximizes the loss on𝒟f\\mathcal\{D\}\_\{f\}and minimizes the loss on𝒟r\\mathcal\{D\}\_\{r\}via the objectiveℒGD=−ℒ​\(𝒟f,w\)\+ℒ​\(𝒟r,w\)\\mathcal\{L\}\_\{\\text\{GD\}\}=\-\\mathcal\{L\}\(\\mathcal\{D\}\_\{f\},w\)\+\\mathcal\{L\}\(\\mathcal\{D\}\_\{r\},w\), whereℒ​\(𝒟f,w\)\\mathcal\{L\}\(\\mathcal\{D\}\_\{f\},w\)is implemented as gradient ascent on the forget set negative log\-likelihood\. Rather than fine\-tuning the full model, we perform*layer\-selective*unlearning: all parameters are frozen except those in layer 7\.Freezing all other layers limits parameter drift on𝒟r\\mathcal\{D\}\_\{r\}, reduces catastrophic forgetting and training cost\.

KL\-Min\(imization\)\(Nguyenet al\.,[2020](https://arxiv.org/html/2605.30599#bib.bib5)\)uses the same GradDiff forget objective and the same layer\-7 parameter mask\. KL\-Min applies gradient ascent on𝒟f\\mathcal\{D\}\_\{f\}while regularizing𝒟r\\mathcal\{D\}\_\{r\}outputs to match the original model via KL divergence, preventing model collapse\.

SimNPO\(Fanet al\.,[2024](https://arxiv.org/html/2605.30599#bib.bib29)\)treats forgotten samples as negative preferences without reference\-model dependency, addressing bias issues in standard NPO\(Zhanget al\.,[2024](https://arxiv.org/html/2605.30599#bib.bib6)\)\. Like GradDiff and KL\-Min, we update only layer 7\.

## 5Evaluation Metrics \(Figure[1](https://arxiv.org/html/2605.30599#S1.F1)\.c\)

To assess the trade\-off between knowledge removal and utility preservation, we use four metrics from\(Yoonet al\.,[2025](https://arxiv.org/html/2605.30599#bib.bib7)\)and one from\(Dornaet al\.,[2026](https://arxiv.org/html/2605.30599#bib.bib10)\)\. We also introduce a novel disease\-informed leakage evaluation metric\.

### 5\.1Disease Keyword Leakage Evaluation

Standard unlearning metrics measure semantic and lexical similarity to references, but do not test whether disease\-specific terminology about forget patients still surfaces in outputs\. We therefore introduce a leakage metric based on disease\-specific token matching\.

#### Keyword Extraction

For each QA instance, we build a disease\-specific keyword set from the note, question, and answer\. We parse each field with scispaCy\(Neumannet al\.,[2019](https://arxiv.org/html/2605.30599#bib.bib30)\)and collect biomedical named entities and noun chunks, dropping generic clinical terms \(e\.g\.,*patient*,*history*\)\. Relevance to diseases is scored lexically against curated seed lists for Cancer \(e\.g\.,*chemotherapy*,*metastasis*\), Infectious/Immune \(e\.g\.,*sepsis*,*autoimmune*\), and Cardiovascular \(e\.g\.,*stent*,*troponin*\)\. Complete seed lists and generic terms are detailed in Appendix[E](https://arxiv.org/html/2605.30599#A5)\.

For a candidate phraseppand seedss, we assign lexical relevancerel​\(p\)∈\[0,1\]\\mathrm\{rel\}\(p\)\\in\[0,1\]using exact or partial phrase matching\. Each candidate receives a weightsis\_\{i\}based on its frequency in that field\. We then merge the top K keywords from the notes and all surviving keywords from QAs\.

#### Leakage Score \(LS\)

Letyybe the model’s generated answer\. Leakage is the weighted fraction of keywords that reappear inyyas a contiguous token subsequence,

LS​\(y\)=∑isi⋅𝟙​\[ki∈toky\]∑isi∈\[0,1\],\\mathrm\{LS\}\(y\)=\\frac\{\\sum\_\{i\}s\_\{i\}\\cdot\\mathbb\{1\}\\\!\\left\[k\_\{i\}\\in\_\{\\mathrm\{tok\}\}y\\right\]\}\{\\sum\_\{i\}s\_\{i\}\}\\in\[0,1\],\(1\)where𝟙​\[ki∈toky\]\\mathbb\{1\}\[k\_\{i\}\\in\_\{\\mathrm\{tok\}\}y\]is11if the keywordkik\_\{i\}occurs consecutively inyyafter tokenization \(case\-insensitive, alphanumeric only\)\. We reportLS\\mathrm\{LS\}as a percentage \(mean over forget set QAs with at least one keyword\)\. Higher values indicate more reproduction of disease\-specific terms from the note\.

### 5\.2Established Metrics

#### Cosine Similarity \(CS\)

We measure semantic similarity using PubMedBERT embeddings\(Guet al\.,[2021](https://arxiv.org/html/2605.30599#bib.bib31)\)\.

#### Entailment Score \(ES\)

We employ PubMedBERT\-MNLI\-MedNLI\(Dekaet al\.,[2023](https://arxiv.org/html/2605.30599#bib.bib32)\)to verify factual consistency between model responses and reference answers\.

#### Syntactic Overlap

We calculateROUGE\-L recalland position\-aligned tokenExact Memorization \(EM\)\(Dornaet al\.,[2026](https://arxiv.org/html/2605.30599#bib.bib10)\)to track lexical retention\.

#### Token Entropy \(TE\)

We monitor normalized unigram entropy\(Yoonet al\.,[2025](https://arxiv.org/html/2605.30599#bib.bib7)\)to identify instances of model collapse or repetitive output \(higher indicates more diverse output\)\.

We utilize theharmonic mean\(HH\) to compute primary performance indices\. This ensures that a model cannot achieve high scores by over\-optimizing for a single metric while failing in others:

#### Model Utility \(MU\)

Measures the preservation of medical knowledge in the retain set,

M​U=H​\(ROUGE, EM, CS, TE, ES\)MU=H\(\\text\{ROUGE, EM, CS, TE, ES\}\)\(2\)

#### Answer Forget Efficacy \(AFE\)

Quantifies unlearning success on the forget set by aggregating the inversion of quality metrics \(1−score1\-\\text\{score\}\),

A​F​E=H​\(1−ROUGE,​1−EM,​1−CS,​1−ES\)AFE=H\(1\-\\text\{ROUGE, \}1\-\\text\{EM, \}1\-\\text\{CS, \}1\-\\text\{ES\}\)\(3\)
Following\(Yoonet al\.,[2025](https://arxiv.org/html/2605.30599#bib.bib7)\), we exclude TE from this aggregate since it measures how repetitive the text is, which is unrelated to whether facts were actually forgotten\. Instead, we reportdegradation score\.

#### Degradation Score \(DS\)

To detect gibberish output, we report a DS,

DS=\|\{x∈𝒟f:T​E​\(x\)=0\}\|\|𝒟f\|,\\text\{DS\}=\\frac\{\|\\\{x\\in\\mathcal\{D\}\_\{f\}:TE\(x\)=0\\\}\|\}\{\|\\mathcal\{D\}\_\{f\}\|\},\(4\)which is the fraction of forget set generations with zero mean TE\. These correspond to fully repetitive token outputs; a high DS indicates output collapse rather than knowledge removal \(Table[2](https://arxiv.org/html/2605.30599#S6.T2)\)\.

## 6Results and Analysis

We evaluate 4 unlearning methods across two patient selection strategies: random patient\-level and disease\-level\. For all experiments, MU is measured on the retain set, and AFE is measured on the forget set\. We expect the MU for unlearned models to be close to the baseline MU, and the AFE to exceed the baseline AFE\. MU of the holdout set is measured on the unseen QAs of retain patients, and AFE is measured on unseen QAs of forget patients\. We report results separately for factual and reasoning QAs\.

### 6\.1Do Models Forget Random Patients Regardless of Their Disease Profile?

Figure[2](https://arxiv.org/html/2605.30599#S6.F2)presents random patient\-level results, examining whether unlearning methods can forget patient information across different forget set sizes from 5% to 25%\. On training \(seen\) questions \(solid bars\), RMU and SimNPO maintain high MU \(above 0\.9 in factual and above 0\.75 in reasoning QAs\) and low AFE \(< 0\.15\) across all forget set sizes,indicating ineffective unlearning in the random patient\-level setting\.KL\-Min exhibits the opposite pattern: high AFE \(above 0\.9 for both QA types\) but low MU \(∼0\.0\{\\sim\}0\.0\), achieving forgetting by generating repeated tokens \(Table[2](https://arxiv.org/html/2605.30599#S6.T2)\) rather than selective erasure\. GradDiff balances both metrics at 5–15% splits \(MU∼0\.9\{\\sim\}0\.9for both QA types\) but collapses at 20–25%, similar to KL\-Min, losing model utility as more patients are forgotten\.

Holdout QAs \(hatched bars\) yield much lower MU baseline values than train QAs \(∼0\.6\{\\sim\}0\.6vs∼1\.00\{\\sim\}1\.00for factual and∼0\.2\{\\sim\}0\.2vs∼0\.95\{\\sim\}0\.95for reasoning\) since these QAs are unseen\.The large gap between factual and reasoning holdout MU values suggests that unlearned models recall direct facts from the note but fail to reason from it\.Holdout AFE also remains low for RMU and SimNPO in both factual and reasoning QAs and close to baseline\. This suggests that models struggle with novel QAs, creating a false sense of forgetting\. Also, it means thatunlearned models forget specific QAs from training data rather than truly erasing learned patient information \(notes\), which remains accessible via alternative questions\.

![Refer to caption](https://arxiv.org/html/2605.30599v1/plots/mu_afe_barchart_patient_factual_reasoning.png)Figure 2:Unlearning performance at random patient\-level\. Rows correspond to MU and AFE, each measured on factual \(top two rows\) and reasoning \(bottom two rows\) QAs\. Columns are forget set split sizes\. For every method, the solid bar reports performance on train data, hatched bar on holdout data\. Dashed lines are baseline \(finetuned model\) results on train and holdout datasets\.![Refer to caption](https://arxiv.org/html/2605.30599v1/plots/mu_vs_afe_scatter_factual_reasoning.png)Figure 3:Unlearning performance at disease\-level\. Horizontal axis is MU on*retain*patients; vertical axis is AFE on*forget*patients\. Each colored trace is one unlearning method and stars are baseline values \(finetuned model before unlearning\); marker opacity scales with the forget set size \(5%5\\%–25%25\\%splits\)\. Light polylines connect successive forget splits\. Each inset zooms the region where GradDiff and KL\-Min traces concentrate\.![Refer to caption](https://arxiv.org/html/2605.30599v1/plots/mu_cancer_overlay.png)\(\(a\)\)Cancer vs\. non\-cancer retain patients\.
![Refer to caption](https://arxiv.org/html/2605.30599v1/plots/mu_immune_overlay.png)\(\(b\)\)Infectious/Immune vs\. non\-infectious retain patients\.

Figure 4:Retain MU on seen questions for same\-disease \(solid\) vs\. out\-of\-disease \(dashed\) patients across forget set sizes for cancer and infectious/immune cohorts, averaged acrossbothfactual and reasoning questions\.Forget Set LS and DS \(%\) \(Factual / Reasoning\)DiseaseMethod5%10%15%20%25%CancerGradDiff0\.5 \(76\.2\) /0\.5\(75\.0\)0\.3\(93\.8\)/0\.3\(93\.2\)0\.3 \(90\.9\) /0\.3\(89\.5\)0\.1\(97\.7\)/0\.1\(97\.3\)0\.2 \(87\.5\) / 0\.3 \(86\.8\)KL\-Min0\.2\(86\.4\)/0\.5\(81\.3\)0\.2\(91\.0\) /0\.3\(92\.1\)0\.2\(91\.6\)/0\.3\(87\.3\)0\.2 \(90\.9\) / 0\.3 \(89\.9\)0\.1\(95\.4\)/0\.1\(94\.3\)RMU13\.9\(0\.3\)/24\.6\(0\.0\)13\.9\(0\.2\)/22\.8\(0\.0\)14\.0\(0\.3\) /20\.9\(0\.0\)14\.2\(0\.1\)/18\.6\(0\.0\)10\.4\(0\.1\)/12\.7\(0\.0\)SimNPO14\.1\(0\.5\) / 24\.1\(0\.0\)13\.6\(0\.2\)/ 16\.9\(0\.0\)11\.9\(0\.1\)/ 14\.5\(0\.0\)10\.9 \(0\.3\) / 13\.8\(0\.0\)12\.1\(0\.1\)/ 11\.6\(0\.0\)Infectious/GradDiff0\.0\(100\.0\)/0\.0\(100\.0\)0\.4 \(92\.1\) /0\.6\(89\.8\)0\.6 \(82\.1\) / 0\.9 \(82\.8\)0\.7\(72\.2\) /1\.0\(72\.8\)– / –ImmuneKL\-Min0\.3 \(92\.1\) / 0\.6 \(87\.2\)0\.3\(92\.4\)/0\.6\(90\.8\)0\.3\(95\.2\)/0\.4\(94\.0\)1\.2\(82\.4\)/ 1\.3\(81\.4\)– / –RMU14\.8\(0\.4\)/23\.8\(0\.0\)14\.2\(0\.3\)/21\.9\(0\.0\)14\.0\(0\.3\)/20\.0\(0\.1\)12\.2\(0\.2\)/16\.0\(0\.1\)– / –SimNPO14\.8\(0\.4\)/ 21\.3\(0\.0\)14\.2\(0\.3\)/ 14\.9\(0\.0\)12\.3 \(0\.5\) / 12\.6 \(0\.2\)10\.9\(0\.2\)/ 10\.4\(0\.1\)– / –Cardio\-GradDiff0\.2\(97\.7\)/0\.1\(97\.6\)0\.7 \(79\.6\) / 0\.7 \(80\.1\)0\.4 \(88\.9\) / 0\.5 \(88\.8\)– / –– / –vascularKL\-Min0\.6 \(7\.0\) / 0\.5 \(5\.5\)0\.1\(97\.3\)/0\.1\(97\.5\)0\.2\(97\.2\)/0\.1\(97\.3\)– / –– / –RMU11\.3 \(0\.3\) /14\.8\(0\.2\)10\.3\(0\.2\)/12\.5\(0\.1\)5\.9\(0\.1\)/ 7\.5 \(0\.1\)– / –– / –SimNPO11\.9\(0\.2\)/ 11\.1\(0\.2\)10\.4\(0\.2\)/ 8\.7\(0\.0\)8\.8\(0\.1\)/8\.5\(0\.0\)– / –– / –RandomGradDiff14\.3\(0\.2\)/26\.0\(0\.0\)14\.7\(0\.2\)/ 25\.4\(0\.0\)14\.4\(0\.2\)/ 25\.5\(0\.0\)0\.1 \(21\.1\) / 0\.3 \(16\.7\)0\.0\(100\.0\)/0\.0\(100\.0\)PatientKL\-Min0\.7\(64\.2\)/1\.5\(33\.4\)0\.0\(98\.1\)/0\.0\(98\.4\)0\.0\(96\.7\)/0\.0\(97\.4\)0\.0\(99\.4\)/0\.0\(99\.4\)0\.0\(99\.7\) /0\.0\(99\.7\)RMU14\.3\(0\.2\)/ 25\.6\(0\.0\)14\.6 \(0\.3\) /26\.0\(0\.0\)14\.6\(0\.3\) /25\.7\(0\.0\)14\.6\(0\.4\)/25\.5\(0\.0\)14\.5\(0\.4\)/25\.3\(0\.0\)SimNPO14\.3\(0\.2\)/ 24\.8\(0\.0\)14\.5 \(0\.3\) / 25\.6\(0\.0\)14\.5 \(0\.3\) / 24\.0\(0\.0\)14\.5\(0\.4\)/ 23\.5 \(0\.1\)14\.4\(0\.4\)/ 23\.1\(0\.0\)

Table 2:Forget set leakage score \(LS\) and degradation score \(DS\) in percentages across random patient and disease\-level splits, unlearning methods, and forget set sizes\. Each cell reports factual / reasoning LS with DS in parentheses; bold = lowest, underlined = highest across methods for each QA type\.
### 6\.2Does Forgetting Patients with the Same Disease Differ from Forgetting Random Patients?

Analysis in[6\.1](https://arxiv.org/html/2605.30599#S6.SS1)showed that model unlearning for forgetting random patients either collapses \(high AFE and low MU\) or is ineffective \(high MU and low AFE\)\. We now examine whether this behavior changes when forgotten patients share the same disease\.

The evaluation of the disease\-level \(Figure[3](https://arxiv.org/html/2605.30599#S6.F3)\) reveals a fundamentally different pattern from the random\-patient unlearning\. We plot forget set AFE versus retain set MU for the three most populated diseases \(Cancer, Infectious/Immune, Cardiovascular\) across different forget set sizes\. Perfect unlearning appears in the top\-right corner \(high MU, high AFE\)\. Unlike random patients, patients sharing the same disease are interconnected through shared medical characteristics\. KL\-Min and GradDiff behave similarly across all three diseases\. Both achieve moderate MU \(∼\\sim0\.4–0\.7\) while preserving high AFE \(∼\\sim0\.90–0\.97\)\. The pattern is noisy and non\-monotonic for both QA types\.

For RMU and SimNPO, MU also remains high across all three diseases, but AFE now increases with forget set size\. For instance, RMU forgetting 5% of cancer patients achieves AFE below 0\.2, but forgetting 25% raises AFE to∼\\sim0\.6\. So,forget set size strongly influences RMU and SimNPO’s unlearning effectiveness when patients share diseases\.This occurs because removing more patients from the same disease progressively erodes the shared medical knowledge, making it harder for the model to answer questions about that disease\.

In Figures[4\(a\)](https://arxiv.org/html/2605.30599#S6.F4.sf1)and[4\(b\)](https://arxiv.org/html/2605.30599#S6.F4.sf2), for RMU and SimNPO, MU on seen questions \(averaged over factual and reasoning\) for same\-disease retained patients degrades substantially with forget set size: in the cancer cohort, both methods drop from∼0\.83\{\\sim\}0\.83to∼0\.37\{\\sim\}0\.37, while in the infectious/immune cohort, RMU declines from∼0\.87\{\\sim\}0\.87to∼0\.62\{\\sim\}0\.62and SimNPO from∼0\.84\{\\sim\}0\.84to∼0\.53\{\\sim\}0\.53\. However, MU for out\-of\-disease patients remains high \(∼0\.90\{\\sim\}0\.90–0\.950\.95\)\. GradDiff and KL\-Min, by contrast, collapse to near\-zero MU for*same\-disease*retained patients, but maintain moderate MU \(∼0\.50\{\\sim\}0\.50–0\.770\.77\) for out\-of\-disease patients, indicating that their unlearning is disease\-targeted rather than a global model collapse\.

Holdout evaluation in Figure[3](https://arxiv.org/html/2605.30599#S6.F3)mirrors the trends observed for seen QAs: AFE increases with forget set size, and MU shows disease\-dependent trajectories\.MU on reasoning QAs is consistently lower than MU on factual QAs, reflecting the greater difficulty of clinical inference\.Critically, holdout AFE values remain comparable to training AFE, unlike random patient unlearning, where holdout AFE values dropped to∼\\sim0\.2\. This indicates thatdisease\-level patient unlearning operates at the patient\-note level rather than just the QA level\.

### 6\.3Do Models Reproduce Exact Medical Terms From Forgotten Patients?

We now examine a novel privacy criterion:disease keyword leakage\. Even if a model has high AFE, leakage of disease\-related keywords from patient notes constitutes a privacy violation, as it confirms the model has retained specific information\.

Table[2](https://arxiv.org/html/2605.30599#S6.T2)reports disease\-specific keyword leakage rates on the forget set across disease categories and forget set sizes\. GradDiff and KL\-Min achieve near\-zero leakage score \(LS\) \(∼\\sim0\.001–0\.007\) across all diseases and forget set sizes, but this occurs because these methods generate largely incoherent outputs \(high degradation scoresin Table[2](https://arxiv.org/html/2605.30599#S6.T2)\) rather than because they have erased targeted information\.

Disease\-level forgetting shows a different pattern than random patient forgetting for RMU and SimNPO\. For random patients, LS remains stable at∼\\sim14–15% \(factual\) and∼\\sim25% \(reasoning\) across all forget set sizes for RMU and SimNPO, consistent with the flat AFE patterns observed in Figure[2](https://arxiv.org/html/2605.30599#S6.F2)\. In contrast,for RMU and SimNPO, disease\-level forgetting shows declining leakage as forget set size increases, with degradation remaining near zero across all diseases\.For example, for cancer, RMU’s reasoning LS drops from 24\.6% \(5%\) to 12\.7% \(25%\), and SimNPO’s drops from 24\.1% to 11\.6%\. This mirrors the improving AFE trend in Figure[3](https://arxiv.org/html/2605.30599#S6.F3), confirming that removing more patients from the same disease progressively erodes the shared medical knowledge, reducing leakage of medical terminology\.

## 7Conclusion

We introduced AMNESIA, a large\-scale open source medical unlearning benchmark with 70,560 QAs from 8,820 patients, featuring random patient\-level and disease\-level splits\. AMNESIA reveals the limitations of existing unlearning methods\. For random patient\-level splits, unlearning methods either fail to forget effectively or achieve forgetting through model collapse\. Disease\-level forgetting improves as the forget set grows, but at the cost of model utility on retained same\-disease patients\. Across both settings, clinical reasoning remains harder than factual recall\. Holdout evaluation further shows that models forget specific QAs rather than the patient notes, since the underlying information remains accessible through alternative unseen questions\. Our disease keyword leakage analysis also shows the importance of having a domain\-specific question\-answering evaluation metric in unlearning settings\. Together, these findings highlight the need for medical\-specific unlearning techniques that erase patient information without damaging shared medical knowledge and establish AMNESIA as an essential benchmark for advancing the field\.

## Limitations

Our evaluation of forgetting relies on answer\-level metrics \(AFE, MU\) and exact\-match keyword leakage\. A model with high AFE may still encode patient information recoverable through paraphrased queries, and our exact\-match leakage metric does not catch paraphrased reproductions of forgotten terminology\. Therefore, what fraction of a patient’s information is truly removed remains difficult to quantify\.

All experiments use a single base model \(LLaMA 3\-8B\), and gradient\-based unlearning at layer 7 to match the RMU configuration from OpenUnlearning\.

We partition patients into 11 single disease categories, but real clinical conditions exist on a spectrum, and comorbidities are common\.

## Ethical Considerations

AMNESIA is built from publicly available, de\-identified clinical notes and will be released without any restrictions on the source data\.

Our central finding is that unlearning can coexist with substantial keyword leakage, and new questions recover information from supposedly forgotten patients\. Deploying any of the methods we evaluate under the assumption that they fulfill a right\-to\-be\-forgotten request would create a false sense of privacy compliance\. We further show that forgetting a critical mass of patients sharing a disease degrades model utility for retained patients with the same condition, meaning batch deletion requests could compromise care quality for patients who never requested deletion\. None of the models in this work is validated for clinical use\.

## Acknowledgments

We thank our human annotator for volunteering to do our human\-in\-the\-loop evaluation\.

## References

- Act \(1996\)Health insurance portability and accountability act of 1996\.Note:Public Law, 104\-191Cited by:[§2](https://arxiv.org/html/2605.30599#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Cao and J\. Yang \(2015\)Towards making systems forget with machine unlearning\.pp\. 463–480\.External Links:[Link](https://api.semanticscholar.org/CorpusID:5945696)Cited by:[§2](https://arxiv.org/html/2605.30599#S2.SS0.SSS0.Px1.p1.1)\.
- H\. Chen, Z\. Fang, Y\. Singla, and M\. Dredze \(2025\)Benchmarking large language models on answering and explaining challenging medical questions\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 3563–3599\.External Links:[Link](https://aclanthology.org/2025.naacl-long.182/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.182),ISBN 979\-8\-89176\-189\-6Cited by:[§2](https://arxiv.org/html/2605.30599#S2.SS0.SSS0.Px2.p1.1)\.
- M\. Choi, C\. Park, D\. Lee, and J\. Choo \(2024\)Breaking chains: unraveling the links in multi\-hop knowledge unlearning\.arXiv preprint arXiv:2410\.13274\.Cited by:[§2](https://arxiv.org/html/2605.30599#S2.SS0.SSS0.Px1.p1.1)\.
- P\. Deka, A\. Jurek\-Loughrey, and D\. P \(2023\)Multiple evidence combination for fact\-checking of health\-related information\.InProceedings of the 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks,D\. Demner\-fushman, S\. Ananiadou, and K\. Cohen \(Eds\.\),Toronto, Canada,pp\. 237–247\.External Links:[Link](https://aclanthology.org/2023.bionlp-1.20/),[Document](https://dx.doi.org/10.18653/v1/2023.bionlp-1.20)Cited by:[§5\.2](https://arxiv.org/html/2605.30599#S5.SS2.SSS0.Px2.p1.1)\.
- V\. Dorna, A\. R\. Mekala, W\. Zhao, A\. McCallum, J\. Z\. Kolter, Z\. C\. Lipton, and P\. Maini \(2026\)OpenUnlearning: accelerating LLM unlearning via unified benchmarking of methods and metrics\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=Gy67Zh5X1i)Cited by:[§2](https://arxiv.org/html/2605.30599#S2.SS0.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2605.30599#S4.SS2.p1.2),[§5\.2](https://arxiv.org/html/2605.30599#S5.SS2.SSS0.Px3.p1.1),[§5](https://arxiv.org/html/2605.30599#S5.p1.1)\.
- C\. Fan, J\. Liu, L\. Lin, J\. Jia, R\. Zhang, S\. Mei, and S\. Liu \(2024\)Simplicity prevails: rethinking negative preference optimization for LLM unlearning\.InNeurips Safe Generative AI Workshop 2024,External Links:[Link](https://openreview.net/forum?id=pVACX02m0p)Cited by:[§4\.2](https://arxiv.org/html/2605.30599#S4.SS2.p5.1)\.
- Y\. Gu, R\. Tinn, H\. Cheng, M\. Lucas, N\. Usuyama, X\. Liu, T\. Naumann, J\. Gao, and H\. Poon \(2021\)Domain\-specific language model pretraining for biomedical natural language processing\.ACM Trans\. Comput\. Healthcare3\(1\)\.External Links:[Link](https://doi.org/10.1145/3458754),[Document](https://dx.doi.org/10.1145/3458754)Cited by:[§5\.2](https://arxiv.org/html/2605.30599#S5.SS2.SSS0.Px1.p1.1)\.
- S\. Hardan, D\. Taratynova, A\. Essofi, K\. Nandakumar, and M\. Yaqub \(2025\)Forget\-mi: machine unlearning for forgetting multimodal information in healthcare settings\.InInternational Conference on Medical Image Computing and Computer\-Assisted Intervention,pp\. 204–213\.Cited by:[§2](https://arxiv.org/html/2605.30599#S2.SS0.SSS0.Px3.p1.1)\.
- A\. E\. Johnson, T\. J\. Pollard, S\. J\. Berkowitz, N\. R\. Greenbaum, M\. P\. Lungren, C\. Deng, R\. G\. Mark, and S\. Horng \(2019\)MIMIC\-cxr, a de\-identified publicly available database of chest radiographs with free\-text reports\.Scientific Data6\.Cited by:[§2](https://arxiv.org/html/2605.30599#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Kim, Y\. Abdulle, and H\. Wu \(2025\)BioHopR: a benchmark for multi\-hop, multi\-answer reasoning in biomedical domain\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 12894–12908\.External Links:[Link](https://aclanthology.org/2025.findings-acl.668/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.668),ISBN 979\-8\-89176\-256\-5Cited by:[§2](https://arxiv.org/html/2605.30599#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Kweon, J\. Kim, J\. Kim, S\. Im, E\. Cho, S\. Bae, J\. Oh, G\. Lee, J\. H\. Moon, S\. C\. You, S\. Baek, C\. H\. Han, Y\. B\. Jung, Y\. Jo, and E\. Choi \(2024\)Publicly shareable clinical large language model built on synthetic clinical notes\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 5148–5168\.External Links:[Link](https://aclanthology.org/2024.findings-acl.305/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.305)Cited by:[§2](https://arxiv.org/html/2605.30599#S2.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2605.30599#S4.SS1.p1.1)\.
- C\. Li, C\. Wong, S\. Zhang, N\. Usuyama, H\. Liu, J\. Yang, T\. Naumann, H\. Poon, and J\. Gao \(2023\)Llava\-med: training a large language\-and\-vision assistant for biomedicine in one day\.Advances in Neural Information Processing Systems36,pp\. 28541–28564\.Cited by:[§2](https://arxiv.org/html/2605.30599#S2.SS0.SSS0.Px2.p1.1)\.
- N\. Li, A\. Pan, A\. Gopal, S\. Yue, D\. Berrios, A\. Gatti, J\. D\. Li, A\. Dombrowski, S\. Goel, G\. Mukobi, N\. Helm\-Burger, R\. Lababidi, L\. Justen, A\. B\. Liu, M\. Chen, I\. Barrass, O\. Zhang, X\. Zhu, R\. Tamirisa, B\. Bharathi, A\. Herbert\-Voss, C\. B\. Breuer, A\. Zou, M\. Mazeika, Z\. Wang, P\. Oswal, W\. Lin, A\. A\. Hunt, J\. Tienken\-Harder, K\. Y\. Shih, K\. Talley, J\. Guan, I\. Steneker, D\. Campbell, B\. Jokubaitis, S\. Basart, S\. Fitz, P\. Kumaraguru, K\. K\. Karmakar, U\. Tupakula, V\. Varadharajan, Y\. Shoshitaishvili, J\. Ba, K\. M\. Esvelt, A\. Wang, and D\. Hendrycks \(2024\)The WMDP benchmark: measuring and reducing malicious use with unlearning\.InForty\-first International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=xlr6AUDuJz)Cited by:[§2](https://arxiv.org/html/2605.30599#S2.SS0.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2605.30599#S4.SS2.p2.1)\.
- B\. Liu, Q\. Liu, and P\. Stone \(2022\)Continual learning and private unlearning\.InProceedings of The 1st Conference on Lifelong Learning Agents,S\. Chandar, R\. Pascanu, and D\. Precup \(Eds\.\),Proceedings of Machine Learning Research, Vol\.199,pp\. 243–254\.External Links:[Link](https://proceedings.mlr.press/v199/liu22a.html)Cited by:[§2](https://arxiv.org/html/2605.30599#S2.SS0.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2605.30599#S4.SS2.p3.5)\.
- P\. Maini, Z\. Feng, A\. Schwarzschild, Z\. C\. Lipton, and J\. Z\. Kolter \(2024\)TOFU: a task of fictitious unlearning for LLMs\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=B41hNBoWLo)Cited by:[§2](https://arxiv.org/html/2605.30599#S2.SS0.SSS0.Px1.p1.1),[§3\.2](https://arxiv.org/html/2605.30599#S3.SS2.SSS0.Px1.p1.4)\.
- B\. McMahan, E\. Moore, D\. Ramage, S\. Hampson, and B\. A\. y Arcas \(2017\)Communication\-efficient learning of deep networks from decentralized data\.InArtificial intelligence and statistics,pp\. 1273–1282\.Cited by:[§2](https://arxiv.org/html/2605.30599#S2.SS0.SSS0.Px3.p1.1)\.
- M\. Neumann, D\. King, I\. Beltagy, and W\. Ammar \(2019\)ScispaCy: fast and robust models for biomedical natural language processing\.InProceedings of the 18th BioNLP Workshop and Shared Task,D\. Demner\-Fushman, K\. B\. Cohen, S\. Ananiadou, and J\. Tsujii \(Eds\.\),Florence, Italy,pp\. 319–327\.External Links:[Link](https://aclanthology.org/W19-5034/),[Document](https://dx.doi.org/10.18653/v1/W19-5034)Cited by:[§E\.1](https://arxiv.org/html/2605.30599#A5.SS1.p1.1),[§5\.1](https://arxiv.org/html/2605.30599#S5.SS1.SSS0.Px1.p1.1)\.
- Q\. P\. Nguyen, B\. Kian, H\. Low, and P\. Jaillet \(2020\)Variational bayesian unlearning\.InProceedings of the 34th International Conference on Neural Information Processing Systems,NIPS ’20,Red Hook, NY, USA\.External Links:ISBN 9781713829546Cited by:[§2](https://arxiv.org/html/2605.30599#S2.SS0.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2605.30599#S4.SS2.p4.2)\.
- S\. Pandit, J\. Xu, J\. Hong, Z\. Wang, T\. Chen, K\. Xu, and Y\. Ding \(2025\)MedHallu: a comprehensive benchmark for detecting medical hallucinations in large language models\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 2858–2873\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.143/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.143),ISBN 979\-8\-89176\-332\-6Cited by:[§2](https://arxiv.org/html/2605.30599#S2.SS0.SSS0.Px2.p1.1)\.
- Protection Regulation \(2016\)Regulation \(eu\) 2016/679 of the european parliament and of the council\.Regulation \(eu\)679\(2016\),pp\. 10–13\.Cited by:[§1](https://arxiv.org/html/2605.30599#S1.p1.1),[§2](https://arxiv.org/html/2605.30599#S2.SS0.SSS0.Px3.p1.1)\.
- R\. J\. Roberts \(2001\)PubMed central: the genbank of the published literature\.Proceedings of the National Academy of Sciences98\(2\),pp\. 381–382\.External Links:[Document](https://dx.doi.org/10.1073/pnas.98.2.381),[Link](https://www.pnas.org/doi/abs/10.1073/pnas.98.2.381),https://www\.pnas\.org/doi/pdf/10\.1073/pnas\.98\.2\.381Cited by:[§3\.1](https://arxiv.org/html/2605.30599#S3.SS1.SSS0.Px1.p1.1)\.
- A\. Sellergren, S\. Kazemzadeh, T\. Jaroensri, A\. Kiraly, M\. Traverse, T\. Kohlberger, S\. Xu, F\. Jamil, C\. Hughes, C\. Lau,et al\.\(2025\)Medgemma technical report\.arXiv preprint arXiv:2507\.05201\.Cited by:[§3\.1](https://arxiv.org/html/2605.30599#S3.SS1.SSS0.Px4.p1.4)\.
- W\. Shi, J\. Lee, Y\. Huang, S\. Malladi, J\. Zhao, A\. Holtzman, D\. Liu, L\. Zettlemoyer, N\. A\. Smith, and C\. Zhang \(2025\)MUSE: machine unlearning six\-way evaluation for language models\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=TArmA033BU)Cited by:[§2](https://arxiv.org/html/2605.30599#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Singh, A\. Fry, A\. Perelman, A\. Tart, A\. Ganesh, A\. El\-Kishky, A\. McLaughlin, A\. Low, A\. Ostrow, A\. Ananthram,et al\.\(2025\)Openai gpt\-5 system card\.arXiv preprint arXiv:2601\.03267\.Cited by:[§3\.1](https://arxiv.org/html/2605.30599#S3.SS1.SSS0.Px4.p1.4)\.
- E\. Sviridova, A\. Yeginbergen, A\. Estarrona, E\. Cabrio, S\. Villata, and R\. Agerri \(2024\)CasiMedicos\-arg: a medical question answering dataset annotated with explanatory argumentative structures\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 18463–18475\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.1026/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.1026)Cited by:[§2](https://arxiv.org/html/2605.30599#S2.SS0.SSS0.Px2.p1.1)\.
- F\. Wu, V\. Patil, J\. Yoon, Y\. Zhang, and M\. Bansal \(2025\)Hierarchy\-aware multimodal unlearning for medical ai\.arXiv preprint arXiv:2512\.09867\.Cited by:[§2](https://arxiv.org/html/2605.30599#S2.SS0.SSS0.Px3.p1.1),[§3\.2](https://arxiv.org/html/2605.30599#S3.SS2.SSS0.Px1.p1.4)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§3\.1](https://arxiv.org/html/2605.30599#S3.SS1.SSS0.Px4.p1.4)\.
- S\. Yoon, W\. Jeung, and A\. No \(2025\)R\-TOFU: unlearning in large reasoning models\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 5239–5258\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.265/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.265),ISBN 979\-8\-89176\-332\-6Cited by:[§2](https://arxiv.org/html/2605.30599#S2.SS0.SSS0.Px1.p1.1),[§5\.2](https://arxiv.org/html/2605.30599#S5.SS2.SSS0.Px4.p1.1),[§5\.2](https://arxiv.org/html/2605.30599#S5.SS2.SSS0.Px6.p2.1),[§5](https://arxiv.org/html/2605.30599#S5.p1.1)\.
- R\. Zhang, L\. Lin, Y\. Bai, and S\. Mei \(2024\)Negative preference optimization: from catastrophic collapse to effective unlearning\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=MXLBXjQkmb)Cited by:[§2](https://arxiv.org/html/2605.30599#S2.SS0.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2605.30599#S4.SS2.p5.1)\.
- Z\. Zhao, Q\. Jin, F\. Chen, T\. Peng, and S\. Yu \(2023\)A large\-scale dataset of patient summaries for retrieval\-based clinical decision support systems\.\.Scientific data10 1,pp\. 909\.External Links:[Link](https://api.semanticscholar.org/CorpusID:266360591)Cited by:[§3\.1](https://arxiv.org/html/2605.30599#S3.SS1.SSS0.Px1.p1.1)\.

## Appendix

## Appendix AComparison to Existing Benchmarks

Table 3:Comparison of AMNESIA with existing unlearning benchmarks\.
## Appendix BPrompts Used for Data Generation

This appendix provides the complete prompts used in our automated data generation pipeline\.

### B\.1Disease Category Assignment Prompt

We used the following prompt withGPT\-5\.4\-nanoto assign disease categories to patient cases based on article titles:

You are given a medical article with:•article\_id•article\_titleYour task is to assign exactly one top\-level category to the article based primarily on the article title\.Use only this taxonomy:1\.Anemia or Iron Deficiency2\.Arthritis and Bone•Arthritis•Osteoporosis3\.Cancer4\.Cardiovascular•Cerebrovascular Disease or Stroke•Cholesterol•Heart Disease•Hypertension5\.Dementia and Mental Health•Alzheimer’s Disease•Attention Deficit Hyperactivity Disorder•Depression•Mental Health6\.Diabetes7\.Digestive and Liver•Digestive Diseases•Chronic Liver Disease and Cirrhosis8\.Infectious or Immune Diseases•AIDS and HIV•Viral Hepatitis•Infectious Disease•Influenza•Measles, Mumps, and Rubella•Pneumonia•Sexually Transmitted Diseases \(STD\)•Chronic Sinusitis•Whooping Cough or Pertussis9\.Kidney Disease10\.Oral and Dental Health11\.Respiratory and Allergies•Allergies and Hay Fever•Asthma•Chronic Obstructive Pulmonary DiseaseRules:•Return the best matching top\-level category only\.•Base the decision mainly on the article\_title\.•Use the subcategories only as guidance for mapping into the correct top\-level category\.•Do not use any category outside the taxonomy\.•If no category is clearly supported by the title, choose “Other” and explain briefly\.Return valid JSON only in this exact format:[⬇](data:text/plain;base64,ewogICJhcnRpY2xlX2lkIjogIjxhcnRpY2xlX2lkPiIsCiAgImFydGljbGVfdGl0bGUiOiAiPGFydGljbGVfdGl0bGU+IiwKICAiY2F0ZWdvcnkiOiAiPG9uZSB0b3AtbGV2ZWwgY2F0ZWdvcnkgZnJvbSB0aGUgdGF4b25vbXk+IiwKICAic2hvcnRfcmVhc29uIjogIjxicmllZiByZWFzb24gdXNpbmcgdGl0bGUga2V5d29yZHM+Igp9)\{"article\_id":"<article\_id\>","article\_title":"<article\_title\>","category":"<onetop\-levelcategoryfromthetaxonomy\>","short\_reason":"<briefreasonusingtitlekeywords\>"\}

### B\.2Question\-Answer Generation Prompt

We used the following prompt withGPT\-5\-nanoto generate question\-answer pairs for each patient case:

You are a medical education expert creating a clinical knowledge assessment dataset\.Given a patient case summary, generate two groups of QA pairs:1\. factual\_items•direct recall from the summary•answerable solely from the provided summary•answers must be 1–2 sentences•cover diagnosis, symptoms, labs, treatments, outcomes2\. reasoning\_items•multi\-hop clinical reasoning•each question MUST connect at least TWO findings from the summary•answers should explain the reasoning chain in 2–3 sentencesGlobal rules:•Do NOT ask about patient name, age, gender, or any PII•Questions must REQUIRE this specific patient’s details and must NOT be answered confidently from general medical knowledge alone•Keep questions clinically specific and grounded•Return only valid JSON matching the schema

## Appendix CData Statistics

Table 4:Mean length \(tokens\) of questions \(\|Q\|\|Q\|\) and answers \(\|A\|\|A\|\) in AMNESIA, by disease category\.%\# Patients\# QAsAvailable ForForget Splits54412,646Random, Cancer, Inf/Imm\., Cardio\.108825,292Random, Cancer, Inf/Imm\., Cardio\.151,3237,938Random, Cancer, Inf/Imm\., Cardio\.201,76410,584Random, Cancer, Inf/Imm\.252,20513,230Random, CancerRetain Splits58,37950,274Random, Cancer, Inf/Imm\., Cardio\.107,93847,628Random, Cancer, Inf/Imm\., Cardio\.157,49744,982Random, Cancer, Inf/Imm\., Cardio\.207,05642,336Random, Cancer, Inf/Imm\.256,61539,690Random, CancerHoldout Set–8,82017,640AllTable 5:Data partition statistics\. All forget/retain splits at each percentage contain identical patient/QA counts regardless of partition type\.
## Appendix DTraining Hyperparameters

We perform pre\-training on patient clinical notes to adaptLLaMA 3\-8Bmodel to medical language and terminology\. Patient notes are tokenized using theLLaMA 3tokenizer with a maximum sequence length of 2048 tokens and then packed to maximize GPU utilization\. We employ distributed training across 8 AMD MI250X GPUs\. Gradient checkpointing is enabled to reduce memory consumption\. Unlearning also occurred for 3 epochs, and all experiments are from the last checkpoint of each unlearned model \(the same as finetuning\)\.

Table 6:Training hyperparameters for pretraining and instruction finetuning\.During instruction fine\-tuning, we use the following template to structure each training example:

You are an intelligent clinical language model\.Below is a snippet of patient’s discharge summary and the following instruction from a healthcare professional\. Write a response that appropriately completes the instruction\. The response should provide the accurate answer to the instruction, while being concise\.\[Discharge Summary Begin\]\{patient\_note\}\[Discharge Summary End\]\[Instruction Begin\]\{question\}\[Instruction End\]

## Appendix ELeakage Score \(LS\)

We introduce a disease\-specific keyword extraction pipeline to detect whether unlearned models continue reproducing specific medical terminology from forget patients\. This provides fine\-grained evaluation beyond aggregate metrics\.

### E\.1Extraction Pipeline

For each patient in the dataset, we extract disease\-relevant keywords from three text sources: patient notes, questions, and answers\. The pipeline uses scispaCy\(Neumannet al\.,[2019](https://arxiv.org/html/2605.30599#bib.bib30)\)medical NLP models to identify biomedical entities and noun chunks\. Specifically, we use en\_core\_sci\_lg for patient notes, a large English biomedical model trained on scientific literature with word2vec embeddings that can process arbitrarily long texts without token\-length restrictions\. For questions and answers, the same model is used by default, though the pipeline supports alternative models such as en\_core\_sci\_scibert \(a BERT\-based transformer model with higher run\-time and a 512\-token limit\)\. We focus on three major disease categories: Cancer, Cardiovascular, and Infectious or Immune Diseases\.

CategorySubcategoryTermsCancerGeneral oncologycancer, tumor, tumour, neoplasm, malignancy, malignant, carcinoma, sarcoma, lymphoma, leukemia, leukaemia, myeloma, glioma, blastomaPathologymetastasis, metastatic, adenocarcinoma, melanoma, squamous cell, hepatocellular, biopsy, cytology, grade, staging, margins, excisionSpecific cancerslung cancer, breast cancer, colon cancer, prostate cancer, ovarian cancer, pancreatic cancer, lymph node, bone marrow, recurrence, relapseTreatmentchemotherapy, radiotherapy, radiation therapy, adjuvant, neoadjuvant, palliative, targeted therapy, hormone therapy, resection, debulking, pet scan, screening, remission, immunotherapyCardiovascularGeneralcardiovascular, cardiac, coronary, myocardial, infarction, angina, arrhythmia, atrial fibrillation, ventricular, heart failure, cardiomyopathy, pericarditisConditionshypertension, atherosclerosis, ischemia, ischaemia, stroke, aortic, mitral, valve, embolism, thrombosis, dvt, pulmonary embolism, bradycardia, tachycardia, syncope, heart attackBiomarkerstroponin, bnp, ntprobnp, ejection fraction, lipids, cholesterol, statinProceduresstent, cabg, bypass, angioplasty, revascularization, echocardiogram, ecg, ekg, defibrillation, pacemaker, catheterization, chest pain, cad, chf, hfref, hfpefInfectious or ImmuneInfection \(general\)infection, infectious, sepsis, bacteremia, fever, abscess, cellulitis, wound infectionPathogensviral, virus, bacterial, bacteria, fungal, candida, pathogen, mrsa, tuberculosis, covid, hiv, hepatitis, influenzaSpecific infectionspneumonia, meningitis, encephalitis, osteomyelitis, utiImmune systemimmune, immunity, autoimmune, immunosuppression, immunocompromised, complement, cytokine, interferon, rheumatoid, lupus, vasculitis, seropositiveInflammation markersinflammation, inflammatory, crp, esr, leukocytosis, neutrophil, lymphocyte, wbcDiagnostics & treatmentantibiotic, antiviral, antifungal, prophylaxis, culture, pcr, serologyCommon Terms—patient, patients, year, years, old, male, female, history, medical history, past medical history, pmh, today, yesterday, day, days, week, weeks, month, months, time, normal, none, no, yes, denies, reported, reports, present, mg, ml, cm, right, left, bilateral, mild, moderate, severe, acute, chronic, follow, follow up, plan, assessment, exam, physical exam, question, answer, following, option, options, true, falseTable 7:Disease\-specific seed terms used for filtering, organized by category and subcategory, plus the common\-clinical term list\.#### Disease\-Specific Seed Terms

For each disease category, we maintain a curated list of seed terms \(Table[7](https://arxiv.org/html/2605.30599#A5.T7)\) covering core terminology, conditions, biomarkers, and treatments\. These seed lists were curated by prompting Claude \(Anthropic\) to generate medical terminology for each disease category, followed by manual review and validation by the authors\. We also filter out common clinical terms that lack patient\-specific diagnostic value\.

Algorithm 1Disease Keyword Extraction and Leakage Score Calculation1:Patient data

\(n​o​t​e,q​u​e​s​t​i​o​n,a​n​s​w​e​r\)\(note,question,answer\), disease category

dd, seed terms

SdS\_\{d\}
2:Keyword leakage score for prediction

yy
3:// Step 1: Extract candidate phrases

4:

c​a​n​d​i​d​a​t​e​s←∅candidates\\leftarrow\\emptyset
5:foreach text in

\{n​o​t​e,q​u​e​s​t​i​o​n,a​n​s​w​e​r\}\\\{note,question,answer\\\}do

6:

d​o​c←doc\\leftarrowscispaCy\.parse\(

t​e​x​ttext\)

7:

c​a​n​d​i​d​a​t​e​s←c​a​n​d​i​d​a​t​e​s∪candidates\\leftarrow candidates\\cupExtractEntities\(

d​o​cdoc\)

8:

c​a​n​d​i​d​a​t​e​s←c​a​n​d​i​d​a​t​e​s∪candidates\\leftarrow candidates\\cupExtractNounChunks\(

d​o​cdoc\)

9:endfor

10:Filter out generic terms, short \(

<<3 char\) and long \(

\>\>6 words\) phrases

11:// Step 2: Score phrase relevance to disease

12:

s​c​o​r​e​d←\[\]scored\\leftarrow\[\]
13:foreach phrase

ppin

c​a​n​d​i​d​a​t​e​scandidatesdo

14:

r​e​l←rel\\leftarrowComputeRelevance\(

pp,

SdS\_\{d\}\)⊳\\trianglerightScore 0\.0–1\.0

15:if

r​e​l≥0\.5rel\\geq 0\.5then

16:

f​r​e​q←freq\\leftarrowCount\(

ppin

c​a​n​d​i​d​a​t​e​scandidates\)

17:

s​c​o​r​e←r​e​l\+min⁡\(f​r​e​q,5\)×0\.05score\\leftarrow rel\+\\min\(freq,5\)\\times 0\.05
18:Append

\(p,s​c​o​r​e\)\(p,score\)to

s​c​o​r​e​dscored
19:endif

20:endfor

21:// Step 3: Select top\-K and remove redundancy

22:Sort

s​c​o​r​e​dscoredby score \(descending\)

23:

k​e​y​w​o​r​d​s←keywords\\leftarrowSelectTopK\(

s​c​o​r​e​dscored,

k=5k=5for notes, all for Q&A\)

24:

k​e​y​w​o​r​d​s←keywords\\leftarrowRemoveRedundant\(

k​e​y​w​o​r​d​skeywords\)⊳\\trianglerightKeep shorter phrases

25:// Step 4: Compute leakage on forget\-set predictions

26:

y←y\\leftarrowModel\.predict\(

q​u​e​s​t​i​o​nquestion\)⊳\\trianglerightPrediction for patientpp

27:Tokenize and lowercase

yy:

yt​o​k​e​n​s←y\_\{tokens\}\\leftarrowTokenize\(

yy\)

28:

n​u​m​e​r​a​t​o​r←0numerator\\leftarrow 0,

d​e​n​o​m​i​n​a​t​o​r←0denominator\\leftarrow 0
29:foreach

\(ki,si\)\(k\_\{i\},s\_\{i\}\)in

k​e​y​w​o​r​d​skeywordsdo

30:

kt​o​k​e​n​s←k\_\{tokens\}\\leftarrowTokenize\(

kik\_\{i\}\)

31:

m​a​t​c​h←match\\leftarrowIsContiguousSubsequence\(

kt​o​k​e​n​sk\_\{tokens\},

yt​o​k​e​n​sy\_\{tokens\}\)

32:

n​u​m​e​r​a​t​o​r←n​u​m​e​r​a​t​o​r\+si×m​a​t​c​hnumerator\\leftarrow numerator\+s\_\{i\}\\times match
33:

d​e​n​o​m​i​n​a​t​o​r←d​e​n​o​m​i​n​a​t​o​r\+sidenominator\\leftarrow denominator\+s\_\{i\}
34:endfor

35:return

n​u​m​e​r​a​t​o​rd​e​n​o​m​i​n​a​t​o​r\\frac\{numerator\}\{denominator\}⊳\\trianglerightLeakage score∈\[0,1\]\\in\[0,1\]

#### Candidate Extraction

From each scispaCy\-parsed document, we extract:

1. 1\.Named entities identified by the biomedical entity recognizer
2. 2\.Noun chunks from the dependency parser
3. 3\.Filtering: remove common clinical terms, phrases<<3 characters, and phrases\>\>6 words

#### Relevance Scoring

Each candidate phrase receives a lexical disease relevance score based on its overlap with seed terms:

- •Exact match:score = 1\.0
- •Multi\-word phrase containment:score = 0\.9 \(e\.g\., “lung cancer” contains seed “lung cancer”\)
- •Partial word overlap:score = 0\.0–0\.6 \(proportional to Jaccard similarity\)

Phrases scoring below 0\.5 are filtered\. The final score combines relevance with frequency:score=relevance\+min⁡\(freq,5\)×0\.05\\text\{score\}=\\text\{relevance\}\+\\min\(\\text\{freq\},5\)\\times 0\.05\.

#### Top\-K Selection

For patient notes, we select the top\-5 highest\-scoring keywords after removing redundant longer phrases that contain shorter selected keywords \(e\.g\., keep “lung cancer”, remove “small tumor lung cancer”\)\. For questions and answers, we extract all keywords meeting the relevance threshold without top\-k limitation, as these texts are shorter and contain fewer candidates\.

### E\.2Keyword Leakage Computation

For each model predictionyyon a forget\-set question, we compute the weighted keyword leakage score:

Leakage​\(y\)=∑isi⋅𝟙​\[ki∈y\]∑isi\\text\{Leakage\}\(y\)=\\frac\{\\sum\_\{i\}s\_\{i\}\\cdot\\mathbb\{1\}\[k\_\{i\}\\in y\]\}\{\\sum\_\{i\}s\_\{i\}\}\(5\)wherekik\_\{i\}are the extracted keywords from the corresponding patient \(note \+ question \+ answer\),sis\_\{i\}are their relevance scores, and𝟙​\[ki∈y\]\\mathbb\{1\}\[k\_\{i\}\\in y\]indicates the exact phrase match detected via tokenized n\-gram comparison\. The prediction and keywords are lowercased and tokenized \(alphanumeric tokens only\), then we check if each keyword appears as a contiguous sequence in the prediction tokens\.

This metric quantifies the extent to which the model reproduces disease\-specific medical terminology associated with forget patients\. Algorithm[1](https://arxiv.org/html/2605.30599#alg1)shows the entire pipeline\.

## Appendix FHuman\-in\-the\-Loop Validation of Disease Category Labels

This appendix details the LLM\-as\-judge panel used to verify disease category assignments in AMNESIA, together with the human validation study used to verify the panel itself\.

### F\.1Panel Design and Models

We assembled a three\-judge panel with deliberately heterogeneous families to limit single\-model bias:

- •MedGemma\-27B\-IT\(google/medgemma\-27b\-it\): A 27B\-parameter medical\-specialized instruction\-tuned model by Google DeepMind, built on the Gemma 3 backbone and trained on clinical text, medical QA pairs, and EHR data\. It achieves 87\.7% on MedQA, making it one of the strongest open\-weight models for clinical reasoning and medical question answering\.
- •Qwen3\-32B\(Qwen/Qwen3\-32B\): A 32\.8B\-parameter open\-weight model by Alibaba released under Apache 2\.0, featuring a hybrid architecture that switches between extended chain\-of\-thought*thinking mode*and fast*non\-thinking mode*\. It supports a 128K\-token context window with strong performance in mathematics, coding, and agentic reasoning across 119 languages\.
- •GPT\-5 mini\(gpt\-5\-mini\-2025\-08\-07\): A closed\-source proprietary model by OpenAI and the cost\-efficient variant of the flagship GPT\-5 \(released August 2025\), supporting a 272K\-token context window with vision, function calling, and web search\. It delivers approximately 45% fewer hallucinations than GPT\-4o while targeting high\-volume, low\-latency workloads requiring reliable multi\-step reasoning\.

Each judge received the article title and assigned category and was required to return a JSON object with three fields: \(i\) a 1–2 sentence rationale, \(ii\) a verdict in\{Supported,Unsupported,Ambiguous\}\\\{\\textit\{Supported\},\\textit\{Unsupported\},\\textit\{Ambiguous\}\\\}, and \(iii\) the category the judge believed best fit the title, drawn from \{Cardiovascular, Cancer, Infectious/Immune Diseases, Unclear\}\. The final per\-title label was the majority verdict across the three judges; ties \(one judge each on three different labels\) were recorded asDisagree\.

\[System\] You are a medical research expert evaluating whether a category label is correctly assigned to a medical article title\.The three valid categories and their definitions:\- Cardiovascular: heart disease, vascular conditions, blood pressure, cardiac interventions, stroke \(cerebrovascular\), arrhythmias, coronary artery disease\.\- Cancer: oncology, tumors, malignancies, carcinomas, chemotherapy, radiotherapy, cancer screening, neoplasms\.\- Infectious or Immune Diseases: infections \(bacterial, viral, fungal, parasitic\), vaccines, autoimmune conditions, immunodeficiency, inflammatory immune responses, sepsis\.You must respond ONLY with a valid JSON object\.\[User\] Article Title: “\{title\}”Assigned Category: “\{category\}”Task: Does the assigned category accurately reflect the medical focus of this title?Reason step by step:1\. What is the primary medical topic of this title?2\. Which of the three categories does it best fit?3\. Does the assigned category match?Respond ONLY with this JSON \(no markdown, no extra text\):\{"reasoning": "<1\-2 sentence explanation\>","verdict": "<Supported \| Unsupported \| Ambiguous\>","correct\_category": "<Cardiovascular \| Cancer \| Infectious or Immune Diseases \| Unclear\>"\}

### F\.2Panel Results on 900 Titles

The panel evaluated all 900 sampled patient titles spanning Cancer \(300 titles\), Cardiovascular \(300 titles\), and Infectious/Immune Diseases \(300 titles\)\. Table[8](https://arxiv.org/html/2605.30599#A6.T8)shows the verdict distribution across models\. Table[9](https://arxiv.org/html/2605.30599#A6.T9)breaks the panel verdict down by assigned disease, and Table[10](https://arxiv.org/html/2605.30599#A6.T10)reports pairwise inter\-judge agreement\.

Table 8:Verdict distribution across individual judges and the panel consensus\. Judges show strong agreement onSupportedcases \(∼\\sim92%\), with minor differences in how borderline cases are split betweenUnsupportedandAmbiguous\.Table 9:Panel verdict by assigned disease\. The Infectious/Immune disease carries the largest fraction ofUnsupportedcases, reflecting the breadth of its definition \(bacterial, viral, fungal, parasitic, autoimmune, immunodeficiency\) and the higher chance of titles co\-occurring with cardiovascular or oncologic primary topics\.Table 10:Pairwise inter\-judge agreement and Fleiss’κ\\kappa\(collapsingUnsupported/AmbiguousagainstSupported\) overN=900N=900titles\.
### F\.3Human Validation Protocol

To evaluate the panel, one biology student manually verified the aforementioned 900 titles\. For each title, the annotator was shown the title and the assigned category and answered a binary question:*“Is the assigned category label supported by the title?”*\(Yes/No\)\. The annotator was blinded to all LLM judge outputs and to the panel verdict at annotation time\. The resulting human label distribution was870870Yes and3030No\.

We aligned the human Yes/No labels with the LLMSupported/Unsupportedscheme and computed agreement statistics per judge against the human\. We report a confusion matrix per judge \(Table[11](https://arxiv.org/html/2605.30599#A7.T11)\)\. For each judge’s confusion matrix, that judge’s Ambiguous verdicts were excluded from the denominator \(1 for MedGemma, 8 for Qwen, 3 for GPT\-5\-mini\)\. For the panel matrix, the 1 Ambiguous and 2 Disagree panel outcomes were excluded, yielding 897 rows\.

## Appendix GLLM\-as\-Judge Validation of QA Pairs

We sampled two QAs \(one factual and one reasoning\) per 500 randomly selected patients, resulting in1,0001\{,\}000*\(patient note, question, answer\)*triples from the AMNESIA training pool\. We submitted each to the same three\-judge panel used for disease\-label validation \(Appendix[F](https://arxiv.org/html/2605.30599#A6)\):MedGemma\-27B,Qwen3\-32B,andGPT\-5\-mini\.

Table 11:Confusion matrices for human \(rows\) vs\. each LLM judge and the panel \(columns\) on the 900 human\-annotated titles\.### G\.1Judge Prompt

Each judge was given the full patient note alongside the candidate question and answer, and asked for a strict binary verdict together with a brief rationale\. The verbatim prompt is below\.

\[System\] You are a medical AI expert evaluating the quality of question\-answer pairs derived from clinical case reports\.You must respond ONLY with a valid JSON object\.\[User\] You are evaluating the quality of a medical question\-answer pair\.PATIENT NOTE:\{patient\_note\}QUESTION:\{question\}ANSWER:\{answer\}TASK:Evaluate if this QA pair is valid for a medical AI benchmark\. A valid QA pair must meet ALL criteria:1\. The question is clear and well\-formed2\. The answer is correct based on the patient note3\. The answer can be derived from the information in the patient note4\. The question and answer are clinically meaningfulProvide your evaluation in JSON format:\{"valid": true/false,"reasoning": "Brief explanation of your decision \(2\-3 sentences\)"\}Be strict in your evaluation\. If ANY criterion is not met, mark as invalid\.

### G\.2Panel Results

Table[12](https://arxiv.org/html/2605.30599#A7.T12)summarizes verdicts at the judge and panel level over theN=1,000N=1\{,\}000triples, and Table[13](https://arxiv.org/html/2605.30599#A7.T13)reports pairwise inter\-judge agreement and Fleiss’κ\\kappa\.

Table 12:QA\-validity verdict distribution acrossN=1,000N=1\{,\}000triples\. The panel verdict is the majority of the three judges; a single three\-way split case \(one judgeUnknown, oneValid, oneInvalid\) is reported asSplit\.Table 13:Pairwise inter\-judge agreement and Fleiss’κ\\kappafor QA\-pair validation\. The lowκ\\kappareflects skewed prevalence \(≈96%\\approx 96\\%Valid\) rather than substantive disagreement; raw agreement remains high\.