Breaking Adversarial Transferability in Fine-Tuned Speech Recognition

arXiv cs.LG Papers
adversarial-robustness speech-recognition asr adversarial-attacks fine-tuning transferability security

Summary

该论文揭示了针对公开ASR基础模型生成的对抗扰动可有效迁移至微调后黑盒模型的风险，并提出TransferBreaker统一微调框架（结合Base Adversarial Fine-Tuning、Latent Jacobian Regularization与HybridGrad-AFT），在三种语言和四个大型ASR模型上将对抗WER从92.6降至27.8。

arXiv:2610.09109v1 Announce Type: new Abstract: Many organizations fine-tune publicly available pretrained Automatic Speech Recognition (ASR) models and deploy them in black-box settings, assuming limited access provides protection. We show this assumption is fragile: adversarial perturbations crafted on the public base model transfer effectively to fine-tuned target models, severely degrading performance and posing concerns for safety-critical applications. We propose TransferBreaker, a unified fine-tuning framework that suppresses adversarial transfer by integrating Base Adversarial Fine-Tuning, which restricts adversarial training to base-effective perturbations; Latent Jacobian Regularization, which enforces latent-space invariance by suppressing adversarially sensitive directions; and HybridGrad-AFT, which improves robustness against adaptive attacks by interpolating transferable perturbations from base and target gradients. We theoretically justify all components and evaluate TransferBreaker across three languages and four large ASR models, reducing adversarial WER from 92.6 to 27.8. Our code is publicly available at https://github.com/rohban-lab/TransferBreaker.
Original Article
View Cached Full Text

Cached at: 10/08/26, 12:53 PM 

# Breaking Adversarial Transferability in Fine-Tuned Speech Recognition
Source: [https://arxiv.org/abs/2610.09109](https://arxiv.org/abs/2610.09109)
[View PDF](https://arxiv.org/pdf/2610.09109)[HTML \(experimental\)](https://arxiv.org/html/2610.09109v1)

> Abstract:Many organizations fine\-tune publicly available pretrained Automatic Speech Recognition \(ASR\) models and deploy them in black\-box settings, assuming limited access provides protection\. We show this assumption is fragile: adversarial perturbations crafted on the public base model transfer effectively to fine\-tuned target models, severely degrading performance and posing concerns for safety\-critical applications\. We propose TransferBreaker, a unified fine\-tuning framework that suppresses adversarial transfer by integrating Base Adversarial Fine\-Tuning, which restricts adversarial training to base\-effective perturbations; Latent Jacobian Regularization, which enforces latent\-space invariance by suppressing adversarially sensitive directions; and HybridGrad\-AFT, which improves robustness against adaptive attacks by interpolating transferable perturbations from base and target gradients\. We theoretically justify all components and evaluate TransferBreaker across three languages and four large ASR models, reducing adversarial WER from 92\.6 to 27\.8\. Our code is publicly available at[this https URL](https://github.com/rohban-lab/TransferBreaker)\.

## Submission history

From: Mojtaba Nafez \[[view email](https://arxiv.org/show-email/dd570001/2610.09109)\] **\[v1\]**Tue, 6 Oct 2026 21:01:05 UTC \(1,265 KB\)

Similar Articles

Adapting Foundation ASR Models to Dysarthric Speech: A Case Study

arXiv cs.CL

This paper presents a personalized ASR system for a dysarthric speaker by fine-tuning the Whisper foundation model, achieving 9.7% word error rate using only 22.5 hours of adaptation data plus 8.8 hours of user corrections. The results demonstrate that personalized fine-tuning can make foundation ASR models substantially more effective for dysarthric speech, with a deployed mobile application enabling real-world data collection.