Machine Unlearning for Speech Question Answering in Large Audio-Language Models

# Machine Unlearning for Speech Question Answering in Large Audio-Language Models
Source: [https://arxiv.org/html/2609.13195](https://arxiv.org/html/2609.13195)
###### Abstract

Large Audio\-Language Models \(LALMs\) have recently shown strong capabilities in speech understanding and question answering \(QA\), but they also inherit privacy risks from large\-scale training data, including the unintended memorization of sensitive information\. In this work, we study machine unlearning for speech QA in LALMs, a setting that is more challenging than prior work on text\-based Large Language Models \(LLMs\) or Automatic Speech Recognition \(ASR\) due to the tight coupling between acoustic perception and factual knowledge\. We present and evaluate multiple unlearning strategies, including gradient ascent, task arithmetic, and alignment\-based fine\-tuning methods that enforce safe refusal responses, to remove private knowledge while still preserving performance on core capabilities\. Through extensive experiments on speech QA datasets, we show that these unlearning methods can reduce the privacy leakage rate by up to 80% while maintaining near\-neutral performance on non\-private speech QA and general speech understanding benchmarks\.

###### Index Terms:

large audio\-language models, speech question answering, machine unlearning, privacy preservation\.

## IIntroduction

Over the past few years, Large Language Models \(LLMs\) have achieved remarkable success across a wide range of natural language understanding and generation tasks\[[1](https://arxiv.org/html/2609.13195#bib.bib12),[7](https://arxiv.org/html/2609.13195#bib.bib3),[12](https://arxiv.org/html/2609.13195#bib.bib15)\]\. Motivated by the significant progress in LLMs, the domain of large audio\-language models \(LALMs\) has also undergone a revolutionary transformation and shown promising capabilities for speech and audio understanding and generation\[[18](https://arxiv.org/html/2609.13195#bib.bib14),[19](https://arxiv.org/html/2609.13195#bib.bib13),[20](https://arxiv.org/html/2609.13195#bib.bib2),[6](https://arxiv.org/html/2609.13195#bib.bib20)\]\. LALMs can be broadly divided into two paradigms: \(1\) encoder\-LLM architectures, where a speech encoder produces continuous representations that are projected into the input space of an LLM backbone; and \(2\) speech token\-based architectures, where audio is discretized into token sequences and modeled directly by the LLM in a unified autoregressive framework\.

With the widespread deployment of these models, concerns about privacy risks have also emerged\. Studies have demonstrated that private information contained in the training data of LLMs can be extracted by adversaries\[[4](https://arxiv.org/html/2609.13195#bib.bib16),[9](https://arxiv.org/html/2609.13195#bib.bib17)\]\. In particular, LLMs have been found to*unintentionally memorize*training examples and, when being appropriately prompted, can emit the memorized content verbatim\[[3](https://arxiv.org/html/2609.13195#bib.bib18)\]\. This is not desirable since it may expose individuals’ information\. For LALMs, recent investigations also show the unintended memorization of speech training examples for Automatic Speech Recognition \(ASR\)\[[14](https://arxiv.org/html/2609.13195#bib.bib5)\]\. This suggests that LALMs are not exempt from these privacy risks\.

Under data protection regulations such as the*Right To Be Forgotten*\(RTBF\)\[[16](https://arxiv.org/html/2609.13195#bib.bib19)\], individuals may request the removal of their personal data from deployed models of text\-based LLMs or LALMs\. While one could simply retrain the models on the remaining data after each deletion request, this becomes prohibitively expensive for large\-scale models\. The field of*machine unlearning*\[[2](https://arxiv.org/html/2609.13195#bib.bib7),[17](https://arxiv.org/html/2609.13195#bib.bib8),[13](https://arxiv.org/html/2609.13195#bib.bib9)\]aims to erase targeted information from a trained model without the cost of retraining from scratch\. In recent years, post\-hoc approximation methods have gained popularity in unlearning text\-based LLMs, including*gradient ascent*\[[11](https://arxiv.org/html/2609.13195#bib.bib10)\]and*task arithmetic*\[[10](https://arxiv.org/html/2609.13195#bib.bib11)\]\.

While recent efforts have explored unlearning in LLM\-based ASR models\[[14](https://arxiv.org/html/2609.13195#bib.bib5)\], applying unlearning to LALMs for speech question\-answering \(QA\) remains unexplored\. Unlearning in speech QA is fundamentally distinct from the ASR setting\. ASR is a deterministic sequence\-to\-sequence mapping task \(audio to transcription\), where unlearning typically involves suppressing specific vocabulary or phonetic mappings\. In contrast, speech QA requires complex factual retrieval conditioned on acoustic input\. To successfully unlearn in speech QA, the model must sever the semantic link between an entity and the associated sensitive fact in its parametric memory, without breaking its ability to parse the incoming speech or format a coherent response\. Furthermore, as we show empirically, private and non\-private questions share largely entangled feature directions in the model’s representation space, making surgical erasure of private knowledge particularly challenging\.

In this paper, we present and evaluate multiple machine unlearning methods for speech QA in LALMs\. We explore the gradient ascent and task arithmetic approaches for this new use case\. We additionally investigate alignment\-based fine\-tuning strategies for unlearning in LALMs that enforce safe refusal behavior, where the model is trained to map private queries to safe refusal responses, effectively overriding memorized answers\. In the experiments, we assess success along three axes: the degree to which private information has been erased, the retention of non\-private factual knowledge, and the preservation of general\-purpose speech and audio understanding capabilities\.

To the best of our knowledge, this work is the first systematic study to investigate the unintended memorization problem in LALMs and the corresponding unlearning algorithms for mitigating privacy risks upon data\-removal requests\. Our contributions are threefold:

- •We introduce the problem of speech QA unlearning for LALMs, distinguishing it from ASR unlearning by highlighting the entanglement between acoustic perception and factual memorization\.
- •We present multiple unlearning strategies for speech QA, including gradient ascent, task arithmetic, and alignment\-based fine\-tuning methods that enforce refusal responses, to remove private knowledge while still preserving performance on core capabilities of speech and audio understanding\.
- •We conduct a comprehensive evaluation of these unlearning methods, revealing a fundamental trade\-off between private information erasure and non\-private knowledge retention\. Our results highlight both the effectiveness and inherent limitations of these unlearning techniques in removing deeply memorized knowledge\.

The rest of the paper is organized as follows\. Section[II](https://arxiv.org/html/2609.13195#S2)reviews related work\. Section[III](https://arxiv.org/html/2609.13195#S3)describes the unlearning methods for speech QA in LALMs\. Section[IV](https://arxiv.org/html/2609.13195#S4)presents the experimental setup and results\. Finally, Section[V](https://arxiv.org/html/2609.13195#S5)concludes the paper\.

## IIRelated Work

### II\-AMachine Unlearning in Text\-Based Language Models

Machine unlearning\[[2](https://arxiv.org/html/2609.13195#bib.bib7),[17](https://arxiv.org/html/2609.13195#bib.bib8),[13](https://arxiv.org/html/2609.13195#bib.bib9)\]seeks to approximate the state of a machine learning model trained without a specific subset of data\. In LLMs, exact unlearning \(retraining from scratch\) is computationally infeasible\. Post\-hoc approximation methods have thus gained traction\. The authors in\[[11](https://arxiv.org/html/2609.13195#bib.bib10)\]present the gradient ascent approach and showed that it is effective in forgetting targeted token sequences in language models\. Task Arithmetic\[[10](https://arxiv.org/html/2609.13195#bib.bib11)\]leverages the geometry of the weight space, subtracting task\-specific vectors to erase capabilities\.

### II\-BUnlearning in Speech Modalities

The intersection of speech processing and machine unlearning is a nascent field\. Prior work by\[[14](https://arxiv.org/html/2609.13195#bib.bib5)\]demonstrates unlearning in the ASR model by erasing specific targeted phrases with the use of gradient ascent, while the authors of\[[5](https://arxiv.org/html/2609.13195#bib.bib6)\]conducted experiments on keyword spotting and speaker identification, showing that unlearning speech data is significantly more challenging than unlearning text data\. In this work, we study machine unlearning in the speech QA task for LALMs\.

## IIIMethods

### III\-AProblem Formulation

Letℳ⁡\(θd​e​p​l​o​y​e​d\)\\mathcal\{M\}\(\\theta\_\{deployed\}\)be an LALM with weightsθd​e​p​l​o​y​e​d\\theta\_\{deployed\}, obtained by fine\-tuning a base model with weightsθb​a​s​e\\theta\_\{base\}on a dataset𝒟=𝒟p​r​i∪𝒟n​o​n​p​r​i\\mathcal\{D\}=\\mathcal\{D\}\_\{pri\}\\cup\\mathcal\{D\}\_\{nonpri\}, where𝒟p​r​i\\mathcal\{D\}\_\{pri\}is the private speech QA data and𝒟n​o​n​p​r​i\\mathcal\{D\}\_\{nonpri\}is the non\-private QA data\. Assume that each sample of speech QA data takes a combination of audio and text as input and produces a text answer as output\.

After the release and deployment ofℳ⁡\(θd​e​p​l​o​y​e​d\)\\mathcal\{M\}\(\\theta\_\{deployed\}\), individuals might submit the requests to have their sensitive data𝒟p​r​i\\mathcal\{D\}\_\{pri\}deleted\.

One naive approach for fulfilling such requests is to follow the same training recipe and re\-train the model on the corpus of𝒟n​o​n​p​r​i\\mathcal\{D\}\_\{nonpri\}\. However, this could be computationally costly, especially when the model size is large\.

The objective of machine*unlearning*is to produce optimized weightsθu​n​l​e​a​r​n\\theta\_\{unlearn\}such thatℳ⁡\(θu​n​l​e​a​r​n\)\\mathcal\{M\}\(\\theta\_\{unlearn\}\)exhibits low leakage rate of the private data, while preserving performance in non\-private domains and capabilities\. Figure[1](https://arxiv.org/html/2609.13195#S3.F1)shows the unlearning framework when there are RTBF requests\.

![Refer to caption](https://arxiv.org/html/2609.13195v1/unlearn.png)

Fig\. 1:Framework of unlearning LALMs when there are RTBF requests\.In the following sections, we describe several unlearning approaches applicable to this setting\.

### III\-BGradient Ascent for Unlearning

Gradient ascent induces forgetting by maximizing the standard auto\-regressive language modeling loss on𝒟p​r​i\\mathcal\{D\}\_\{pri\}, effectively reversing the gradient direction used during training\. This pushes the model’s output distribution away from the memorized responses that contain private information\.

Specifically, the gradient ascent algorithm is utilized to fine\-tune the modelℳ⁡\(θd​e​p​l​o​y​e​d\)\\mathcal\{M\}\(\\theta\_\{deployed\}\)on the targeted forget set𝒟p​r​i\\mathcal\{D\}\_\{pri\}\. The fine\-tuning amounts to simply minimizing the likelihood for each text response in the forget set, instead of maximizing it\. During the fine\-tuning with gradient ascent, we monitor the model’s utility as well as the reduction in the level of memorization on the targeted forget set\.

### III\-CTask Arithmetic for Unlearning

Task Arithmetic leverages the observation that fine\-tuning a model on a specific task produces a task vector in weight space, defined as the difference between the fine\-tuned weights and the original pre\-trained weights, that encodes the learned capability\. To construct the task vector for private information, we fine\-tune the base modelℳ⁡\(θb​a​s​e\)\\mathcal\{M\}\(\\theta\_\{base\}\)exclusively on𝒟p​r​i\\mathcal\{D\}\_\{pri\}, yielding weightsθp​r​i\\theta\_\{pri\}\. The corresponding task vector is then defined asτp​r​i=θp​r​i−θb​a​s​e\\tau\_\{pri\}=\\theta\_\{pri\}\-\\theta\_\{base\}\. Unlearning is performed by subtracting this vector from the model weightsθd​e​p​l​o​y​e​d\\theta\_\{deployed\}\(trained on both𝒟p​r​i\\mathcal\{D\}\_\{pri\}and𝒟n​o​n​p​r​i\\mathcal\{D\}\_\{nonpri\}\), scaled by a factorλ\\lambdathat controls the erasure strength:

θu​n​l​e​a​r​n=θd​e​p​l​o​y​e​d−λ⋅τp​r​i\\theta\_\{unlearn\}=\\theta\_\{deployed\}\-\\lambda\\cdot\\tau\_\{pri\}\(1\)Intuitively, this negates the direction in weight space associated with privacy memorization\. Unlike other unlearning methods, TA is gradient\-free at the unlearning stage and the trade\-off between forgetting and retention can be tuned post\-hoc by adjustingλ\\lambda\.

### III\-DSafety Supervised Fine\-Tuning for Unlearning

This method performs unlearning by replacing the undesired behavior with a safe alternative\. Specifically, we construct a modified version of𝒟p​r​i\\mathcal\{D\}\_\{pri\}in which all privacy\-leaking answers are replaced with safe responses that do not disclose sensitive information\. The modelℳ⁡\(θd​e​p​l​o​y​e​d\)\\mathcal\{M\}\(\\theta\_\{deployed\}\)is then fine\-tuned on this modified dataset via standard gradient descent, teaching it to associate private queries with safe responses and overwriting the original question\-answer mappings in the model’s parameters\. For example, a safe response can be a refusal such as “I cannot help with that request\.”

### III\-ESafety Direct Preference Optimization for Unlearning

Similar to the Safety Supervised Fine\-Tuning method above, this method fine\-tunes the modelℳ⁡\(θd​e​p​l​o​y​e​d\)\\mathcal\{M\}\(\\theta\_\{deployed\}\)on a modified version of𝒟p​r​i\\mathcal\{D\}\_\{pri\}, but instead of providing a single target response, it constructs preference pairs\. For each audio query, the “chosen” response is the safe response \(e\.g\., a refusal\), and the “rejected” response is the original privacy\-leaking answer\. The model is then optimized against a frozen reference copy ofℳ⁡\(θd​e​p​l​o​y​e​d\)\\mathcal\{M\}\(\\theta\_\{deployed\}\), simultaneously increasing the probability of generating the chosen response and decreasing the probability of the rejected response\. The reference model serves as an anchor to prevent excessive drift from the model’s general capabilities during optimization\.

While gradient ascent and task arithmetic methods aim to remove private information from model parameters, the two alignment\-based approaches above operate at the behavioral level by discouraging the model from generating sensitive content\. We adopt a practical definition of unlearning focused on preventing privacy leakage at inference time, and therefore include both parametric and behavioral methods under a unified unlearning framework, despite their different mechanisms\.

## IVExperiments

### IV\-AExperimental Setup

Our experiments proceed in two phases\. In the first phase, we takeQwen2\.5\-Omni\-7B\[[19](https://arxiv.org/html/2609.13195#bib.bib13)\]as the base model and fine\-tune it on QA pairs, each consisting of an audio question and a text answer, containing factual information about*fictitious*individuals\. The training data here is deliberately composed of two categories: personal*private*information and*non\-private*information\. After the fine\-tuned model is deployed, suppose a data deletion request is received for the individual private information\. Then in the second phase, we treat the private training data as the*forget*set and apply unlearning approaches to erase the memorized private information from this deployed model\. We evaluate success along three axes: the degree to which private information of individuals has been erased, the retention of non\-private factual knowledge about individuals, and the preservation of general\-purpose speech QA capabilities\.

Datasets\.We consider the following datasets as our training and test data in the experiments\.

- •*Pri \(Private\)*dataset: A targeted forget set comprising speech QA samples that query private information, such as phone numbers, addresses, passwords, and salaries\. All names and personal details are fictitious and synthetically generated\. Text\-based QA pairs were generated initially, and the question text was subsequently converted into audio using a Text\-to\-Speech \(TTS\) engine\. Disjoint sets of TTS voices were utilized to synthesize 1,000 training and 1,000 test audio samples that share the identical underlying question text\. For example, a spoken question might be “What is Mike’s SSN?” with the text answer being “Mike’s SSN is 123\-45\-6789\.” The same question text appears in both training and test sets, but spoken by different TTS voices\.
- •*NonPri \(Non\-Private\)*dataset: A benign, non\-private set consisting of speech QA samples that query non\-sensitive factual information about individuals, such as hobbies, occupation, personal preferences, and favored items\. Following the same synthetic generation protocol as the*Pri*dataset, this set also comprises 1,000 training and 1,000 test audio\-text QA pairs that share identical underlying question and response texts\. For example, a spoken question might be “What is Mike’s favorite hobby?” with the answer being “Mike’s favorite hobby is painting\.” Again, this question in the training and test sets is generated using different TTS voices\.
- •*Chat \(AIR\-Bench\)*benchmark\[[21](https://arxiv.org/html/2609.13195#bib.bib1)\]: A set of speech and audio understanding questions designed to evaluate the ability of LALMs to comprehend complex audio inputs and follow human instructions\. We treat it as a general\-purpose benchmark to assess the preservation of conversational capabilities\. Unlike the synthetic QA pairs, this dataset features diverse, real\-world acoustic conditions and open\-ended dialogue, ensuring the unlearning process does not degrade the model’s core acoustic perception\. This dataset contains 1,193 questions on speech and audio understanding, each paired with a text reference answer\. This dataset is used for evaluation purposes only\.

The details of the synthetic QA pair generation method are described below\.

- •*Pri*QA pairs: They are generated by first creating a pool of around 800 fictitious individuals with synthetic names spanning diverse cultural backgrounds\. For each individual, one or more QA pairs were created covering 17 categories of private information, including address, email, date of birth, SSN, phone number, employer, bank account, salary, credit card, driver’s license, and others\. The questions employ around 150 distinct natural\-language templates, with multiple phrasings per category \(e\.g\., phone number queries range from “What is NAME’s phone number?” to “I’m trying to reach NAME\. What’s the best number?”\), ensuring the model cannot simply memorize a single question pattern per category\. All answers contain entirely fabricated values\.
- •*NonPri*QA pairs: They reuse the same set of fictitious individuals\. Instead of sensitive personal information, each QA pair asks about innocuous personal preferences across 24 lifestyle topics such as hobbies, favorite food, travel, music taste, and hypothetical scenarios\. This parallel structure, same individuals but non\-sensitive content, enables controlled evaluation of unlearning methods: a model should forget private answers while retaining the ability to answer benign questions about the same people\.

We note that our use of synthetic*Pri*and*NonPri*data introduces a realism gap compared to real\-world speech data\. However, it provides fine\-grained control over the experimental setup, enabling more precise evaluation of unlearning behavior while avoiding ethical and legal concerns associated with handling real sensitive data\.

Notice that the unintended memorization of speech training data can occur in two ways: one involves the memorization of speakers’ voices, while the other pertains to the memorization of content and underlying text sequences\. In our experimental setup, the training and test sets share the same question texts but use different TTS voices, allowing us to test whether memorized content can be retrieved from the model by an adversary using a previously unseen voice\.

To visualize the structural relationships among the three text question sets above, we extract hidden\-state representations from the text\-based Qwen model and project them onto their first two principal components \(Figure[2](https://arxiv.org/html/2609.13195#S4.F2)\)\. The heavy overlap between*Pri*and*NonPri*sets indicates that private and non\-private questions share largely entangled feature directions, while the*Chat*cluster is well\-separated, as expected given its general\-purpose questions\.

![Refer to caption](https://arxiv.org/html/2609.13195v1/pca.png)

Fig\. 2:Principal components of representations for the three question sets\.Evaluation Metrics\.For every test sample, we leverageLlama3\-70B\[[7](https://arxiv.org/html/2609.13195#bib.bib3)\]as a judge model to compare LALM’s generated response against the reference answer and determine whether the generated response contains the exact same core factual information\. We report the percentage of queries that successfully match this criterion\. For illustration, we refer to this metric on the*Pri*test dataset as privacyLeakage Rate\(where lower indicates successful unlearning\) and on the*NonPri*or*Chat*test datasets asAccuracy\(where higher indicates preserved capabilities\)\.

Training Strategies\.We train the following components while keeping all other parameters ofQwen2\.5\-Omni\-7Bfrozen:

- •For the speech encoder, only the projection adapter is trainable, which maps the speech encodings into token representations compatible with the LLM input space\.
- •For the LLM, the attention layers are trained with Low\-Rank Adaptation \(LoRA\)\[[8](https://arxiv.org/html/2609.13195#bib.bib4)\]applied to the Q and V projection matrices with rankr=16r=16,α=32\\alpha=32, and a dropout rate of0\.050\.05\.

In total, only 12\.3M parameters are trainable, accounting for approximately 0\.14% of the total model parameters\. All our training utilizes the AdamW optimizer\[[15](https://arxiv.org/html/2609.13195#bib.bib21)\]with a cosine learning rate scheduler\.

Baselines and Unlearning Approaches\.To recap, in the first phase, we fine\-tune the base modelQwen2\.5\-Omni\-7Bon the combined*Pri*and*NonPri*training sets; the resulting model is the target for unlearning\. In the second phase, we apply unlearning methods to this model\. We compare four unlearning strategies against the retraining baseline:

1. 1\.Retrain from Scratch \(RT\):The original Qwen base modelQwen2\.5\-Omni\-7Bis fine\-tuned on the*NonPri*training data \(l​r=2​e−4lr=2e^\{\-4\}\), having never been exposed to the*Pri*training data\.
2. 2\.Gradient Ascent \(GA\):The loss function is negated to maximize the loss on the*Pri*training data during the fine\-tuning of the model to be unlearned\. We use highly constrained learning rates \(1​e−61e^\{\-6\}and5​e−75e^\{\-7\}\) and gradient clipping to prevent model collapse\.
3. 3\.Task Arithmetic \(TA\):We first fine\-tune the original Qwen modelQwen2\.5\-Omni\-7Bon the*Pri*training data for the same number of epochs as the model to be unlearned\. We then subtract these newly learned weights from the model to be unlearned using a scaling factorλ∈\{0\.1,0\.2,0\.3,0\.4,0\.5\}\\lambda\\in\\\{0\.1,0\.2,0\.3,0\.4,0\.5\\\}\. This mathematical subtraction is applied simultaneously to the LoRA attention matrices and the audio adapter layer\.
4. 4\.Safety Supervised Fine\-Tuning \(SFT\):The model to be unlearned is fine\-tuned via standard gradient descent \(l​r∈\{1​e−6,5​e−6\}lr\\in\\\{1e^\{\-6\},5e^\{\-6\}\\\}\) on the*Pri*training data, but the target response text is replaced with a safe refusal response \(e\.g\., “I cannot help with that request\.”\)\.
5. 5\.Safety Direct Preference Optimization \(DPO\):We construct preference pairs for the*Pri*training data, where the “chosen” response is the safe refusal and the “rejected” response is the one with privacy leakage\. We set the penalty parameterβ=0\.1\\beta=0\.1andl​r∈\{1​e−5,5​e−5\}lr\\in\\\{1e^\{\-5\},5e^\{\-5\}\\\}\.

TABLE I:Evaluation results of the base model and fine\-tuned models\.
### IV\-BResults

Table[I](https://arxiv.org/html/2609.13195#S4.T1)shows the evaluation results of the base Qwen modelQwen2\.5\-Omni\-7Band the fine\-tuned models on the combined*Pri*and*NonPri*training datasets with a learning rate of2​e−42e^\{\-4\}across varying numbers of epochs\.

The base model \(B0\) has never been exposed to the*Pri*or*NonPri*training data and thus possesses no factual knowledge about these fictitious individuals\. As a result, it yields a 0% privacy leakage rate on the*Pri*test set and a very low 1\.7% accuracy on the*NonPri*test set, although it demonstrates solid speech and audio understanding capabilities on the general*Chat*benchmark \(61\.7%\)\.

As fine\-tuning progresses, the model rapidly learns the individual factual knowledge on both*Pri*and*NonPri*data\. By Epoch 20 \(F20\), the model reaches a*Pri*leakage rate of 19\.5% alongside a*NonPri*accuracy of 63\.1%\. By Epoch 40 \(F40\), the model obtains 49\.4%*Pri*leakage rate and 65\.3%*NonPri*accuracy\. Notably, the general*Chat*capability remains robust throughout training\. We select F20 \(moderate leakage rate\) and F40 \(high leakage rate\) as our candidate models for the subsequent unlearning experiments\.

TABLE II:Comparison of the fine\-tuned models against the retraining baselines\.Table[II](https://arxiv.org/html/2609.13195#S4.T2)shows the evaluation results of the retraining baseline at 20 and 40 epochs\. Both R20 and R40 achieve near\-zero leakage on the*Pri*test set while maintaining comparable accuracy to their fine\-tuned counterparts \(F20 and F40\) on the*NonPri*and*Chat*test sets\. These retrained models establish a theoretical lower bound on*Pri*leakage rate and an upper bound on*NonPri*accuracy\.

TABLE III:Unlearning performance applied to the F20 checkpoint \(moderate privacy leakage\)\.We then evaluate the four unlearning approaches on the F20 checkpoint \(moderate privacy leakage\), with results detailed in Table[III](https://arxiv.org/html/2609.13195#S4.T3)\. We have the following findings:

- •For GA, as training epochs increase,*Pri*leakage rate decreases steadily\. At 15 epochs with a learning rate of5​e−75e^\{\-7\}\(G2\.15\), leakage drops to 11\.3% while*NonPri*\(63\.2%\) and*Chat*\(61\.8%\) remain comparable to the F20 baseline\. At 20 epochs \(G2\.20\), leakage further decreases to 5\.4% with*NonPri*still at 61\.1% accuracy\. Pushing further to 30 epochs \(G2\.30\) achieves near\-zero leakage \(0\.7%\), but*NonPri*accuracy degrades substantially to 48\.9%, indicating that prolonged gradient ascent damages the retention of non\-private factual knowledge about the fictitious individuals\.
- •For TA, as the scaling factorλ\\lambdaincreases,*Pri*leakage decreases accordingly\. Atλ=0\.2\\lambda=0\.2\(T1\.2\), leakage drops to 12\.5% while*NonPri*\(59\.7%\) and*Chat*\(62\.4%\) remain close to the F20 baseline\. Pushing toλ=0\.5\\lambda=0\.5\(T1\.5\) achieves near\-zero leakage \(0\.9%\), but*NonPri*accuracy collapses to 28\.4%, likely because the shared linguistic structure between*Pri*and*NonPri*causes the task vector to suppress factual QA capabilities about the fictitious individuals\.
- •For safety SFT, as training epochs increase, the model increasingly learns to refuse private queries\. At 15 epochs with a learning rate of1​e−61e^\{\-6\}\(S2\.15\), leakage drops to 10\.6% while*NonPri*\(56\.7%\) and*Chat*\(64\.5%\) remain reasonable\. Pushing to 25 epochs \(S2\.25\) achieves 1\.5% leakage, but*NonPri*accuracy collapses to 36\.1%, as the model overgeneralizes the refusal behavior to non\-private questions\.
- •For safety DPO, longer training progressively reduces*Pri*leakage\. At 15 epochs with a learning rate of1​e−51e^\{\-5\}\(D2\.15\), leakage drops to 11\.7% while*NonPri*\(60\.0%\) and*Chat*\(64\.8%\) remain stable\. At 20 epochs \(D2\.20\), leakage further decreases to 9\.8% with*NonPri*at 57\.5% accuracy\. Pushing to 35 epochs \(D2\.35\) achieves 3\.8% leakage with*NonPri*at 47\.9% accuracy\.
- •Comparison\.All four methods exhibit the same fundamental trade\-off: achieving very low*Pri*leakage rate \(e\.g\., below 5%\) comes at the cost of reduced*NonPri*accuracy\. At comparable leakage levels, GA retains the highest*NonPri*accuracy among the four methods, followed by TA and DPO which perform similarly, while SFT suffers the most severe degradation\. In particular, GA can reduce the leakage rate by more than 70% while maintaining near\-neutral performance on*NonPri*accuracy\. Notably, all methods preserve*Chat*accuracy above 60%, suggesting that the catastrophic forgetting is localized to the in\-domain QA distribution rather than the model’s general speech and audio understanding capabilities\. However, there is a slight downward trend in*Chat*accuracy with prolonged unlearning for GA and TA in particular, indicating that these methods may begin to affect general capabilities if applied too aggressively\. It is also worth noting that the unlearning methods differ in how they handle private queries at inference time\. GA and TA still attempt to comply with the query but produce incorrect or fabricated answers, whereas safety SFT and DPO learn to explicitly decline the request with a refusal response\.

TABLE IV:Unlearning performance applied to the deeply memorized F40 checkpoint \(high privacy leakage\)\.Next, we evaluate the four unlearning approaches on the F40 checkpoint \(high privacy leakage\), with results presented in Table[IV](https://arxiv.org/html/2609.13195#S4.T4)\. Consistent with the findings on F20, GA again achieves the best trade\-off between leakage reduction and knowledge retention\. Specifically, G3\.10 reduces the leakage rate by roughly half while maintaining neutral performance on*NonPri*accuracy, and G3\.15 reduces leakage by around 80% with only a slight drop in*NonPri*accuracy\. Similar to the F20 results, SFT again has the highest performance degradation in*NonPri*accuracy, and a slight decline in*Chat*accuracy is observed with prolonged unlearning for the GA and TA methods\.

## VConclusion

In this work, we perform the first investigation of machine unlearning for speech QA in LALMs\. We present and evaluate four unlearning strategies: gradient ascent, task arithmetic, safety supervised fine\-tuning, and safety direct preference optimization\. Through extensive experiments on speech QA datasets, we demonstrate that these unlearning methods can reduce the privacy leakage rate by up to 80% while maintaining near\-neutral performance on non\-private speech QA and general speech understanding benchmarks\. However, achieving near\-zero privacy leakage comes at the cost of degraded non\-private QA accuracy\.

Future directions include exploring more fine\-grained unlearning techniques that can better disentangle private from non\-private knowledge, as well as extending the evaluation to other LALM architectures\.

## References

- \[1\]J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2023\)GPT\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§I](https://arxiv.org/html/2609.13195#S1.p1.1)\.
- \[2\]L\. Bourtoule, V\. Chandrasekaran, C\. A\. Choquette\-Choo, H\. Jia, A\. Travers, B\. Zhang, D\. Lie, and N\. Papernot\(2021\)Machine unlearning\.InIEEE Symposium on Security and Privacy,pp\. 141–159\.Cited by:[§I](https://arxiv.org/html/2609.13195#S1.p3.1),[§II\-A](https://arxiv.org/html/2609.13195#S2.SS1.p1.1)\.
- \[3\]N\. Carlini, D\. Ippolito, M\. Jagielski, K\. Lee, F\. Tramer, and C\. Zhang\(2022\)Quantifying memorization across neural language models\.arXiv preprint arXiv:2202\.07646\.Cited by:[§I](https://arxiv.org/html/2609.13195#S1.p2.1)\.
- \[4\]N\. Carlini, F\. Tramer, E\. Wallace, M\. Jagielski, A\. Herbert\-Voss, K\. Lee, A\. Roberts, T\. Brown, D\. Song, U\. Erlingsson,et al\.\(2021\)Extracting training data from large language models\.InUSENIX Security Symposium,Cited by:[§I](https://arxiv.org/html/2609.13195#S1.p2.1)\.
- \[5\]J\. Cheng and H\. Amiri\(2025\)Speech unlearning\.arXiv preprint arXiv:2506\.00848\.Cited by:[§II\-B](https://arxiv.org/html/2609.13195#S2.SS2.p1.1)\.
- \[6\]A\. Défossez, L\. Mazaré, M\. Orsini, A\. Royer, P\. Pérez, H\. Jégou, E\. Grave, and N\. Zeghidour\(2024\)Moshi: a speech\-text foundation model for real\-time dialogue\.arXiv preprint arXiv:2410\.00037\.Cited by:[§I](https://arxiv.org/html/2609.13195#S1.p1.1)\.
- \[7\]A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§I](https://arxiv.org/html/2609.13195#S1.p1.1),[§IV\-A](https://arxiv.org/html/2609.13195#S4.SS1.p7.1)\.
- \[8\]E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.\(2022\)LoRA: low\-rank adaptation of large language models\.Proceedings of International Conference on Learning Representations\.Cited by:[2nd item](https://arxiv.org/html/2609.13195#S4.I3.i2.p1.1)\.
- \[9\]J\. Huang, H\. Shao, and K\. C\. Chang\(2022\)Are large pre\-trained language models leaking your personal information?\.arXiv preprint arXiv:2205\.12628\.Cited by:[§I](https://arxiv.org/html/2609.13195#S1.p2.1)\.
- \[10\]G\. Ilharco, M\. T\. Ribeiro, M\. Wortsman, S\. Gururangan, L\. Schmidt, H\. Hajishirzi, and A\. Farhadi\(2023\)Editing models with task arithmetic\.Proceedings of International Conference on Learning Representations\.Cited by:[§I](https://arxiv.org/html/2609.13195#S1.p3.1),[§II\-A](https://arxiv.org/html/2609.13195#S2.SS1.p1.1)\.
- \[11\]J\. Jang, D\. Yoon, S\. Yang, S\. Cha, M\. Lee, L\. Logeswaran, and M\. Seo\(2023\)Knowledge unlearning for mitigating privacy risks in language models\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics,pp\. 14389–14408\.Cited by:[§I](https://arxiv.org/html/2609.13195#S1.p3.1),[§II\-A](https://arxiv.org/html/2609.13195#S2.SS1.p1.1)\.
- \[12\]A\. Liu, B\. Feng, B\. Xue, B\. Wang, B\. Wu, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang, C\. Ruan,et al\.\(2024\)Deepseek\-v3 technical report\.arXiv preprint arXiv:2412\.19437\.Cited by:[§I](https://arxiv.org/html/2609.13195#S1.p1.1)\.
- \[13\]S\. Liu, Y\. Yao, J\. Jia, S\. Casper, N\. Baracaldo, P\. Hase, Y\. Yao, C\. Y\. Liu, X\. Xu, H\. Li,et al\.\(2025\)Rethinking machine unlearning for large language models\.Nature Machine Intelligence7\(2\),pp\. 181–194\.Cited by:[§I](https://arxiv.org/html/2609.13195#S1.p3.1),[§II\-A](https://arxiv.org/html/2609.13195#S2.SS1.p1.1)\.
- \[14\]Z\. Liu\(2025\)Unlearning LLM\-based speech recognition models\.InProceedings of Interspeech,pp\. 3214–3218\.Cited by:[§I](https://arxiv.org/html/2609.13195#S1.p2.1),[§I](https://arxiv.org/html/2609.13195#S1.p4.1),[§II\-B](https://arxiv.org/html/2609.13195#S2.SS2.p1.1)\.
- \[15\]I\. Loshchilov and F\. Hutter\(2019\)Decoupled weight decay regularization\.Proceedings of International Conference on Learning Representations\.Cited by:[§IV\-A](https://arxiv.org/html/2609.13195#S4.SS1.p8.2)\.
- \[16\]A\. Mantelero\(2013\)The EU proposal for a general data protection regulation and the roots of the ‘right to be forgotten’\.Computer Law & Security Review29\(3\),pp\. 229–235\.Cited by:[§I](https://arxiv.org/html/2609.13195#S1.p3.1)\.
- \[17\]T\. T\. Nguyen, T\. T\. Huynh, Z\. Ren, P\. L\. Nguyen, A\. W\. Liew, H\. Yin, and Q\. V\. H\. Nguyen\(2025\)A survey of machine unlearning\.ACM Transactions on Intelligent Systems and Technology16\(5\),pp\. 1–46\.Cited by:[§I](https://arxiv.org/html/2609.13195#S1.p3.1),[§II\-A](https://arxiv.org/html/2609.13195#S2.SS1.p1.1)\.
- \[18\]J\. Wu, Y\. Gaur, Z\. Chen, L\. Zhou, Y\. Zhu, T\. Wang, J\. Li, S\. Liu, B\. Ren, L\. Liu,et al\.\(2023\)On decoder\-only architecture for speech\-to\-text and large language model integration\.InIEEE Automatic Speech Recognition and Understanding Workshop,pp\. 1–8\.Cited by:[§I](https://arxiv.org/html/2609.13195#S1.p1.1)\.
- \[19\]J\. Xu, Z\. Guo, J\. He, H\. Hu, T\. He, S\. Bai, K\. Chen, J\. Wang, Y\. Fan, K\. Dang,et al\.\(2025\)Qwen2\.5\-Omni technical report\.arXiv preprint arXiv:2503\.20215\.Cited by:[§I](https://arxiv.org/html/2609.13195#S1.p1.1),[§IV\-A](https://arxiv.org/html/2609.13195#S4.SS1.p1.1)\.
- \[20\]J\. Xu, Z\. Guo, H\. Hu, Y\. Chu, X\. Wang, J\. He, Y\. Wang, X\. Shi, T\. He, X\. Zhu,et al\.\(2025\)Qwen3\-Omni technical report\.arXiv preprint arXiv:2509\.17765\.Cited by:[§I](https://arxiv.org/html/2609.13195#S1.p1.1)\.
- \[21\]Q\. Yang, J\. Xu, W\. Liu, Y\. Chu, Z\. Jiang, X\. Zhou, Y\. Leng, Y\. Lv, Z\. Zhao, C\. Zhou,et al\.\(2024\)Air\-bench: benchmarking large audio\-language models via generative comprehension\.InProceedings of the Annual Meeting of the Association for Computational Linguistics,pp\. 1979–1998\.Cited by:[3rd item](https://arxiv.org/html/2609.13195#S4.I1.i3.p1.1)\.