When Background Matters: Breaking Medical Vision Language Models by Transferable Attack
Summary
MedFocusLeak introduces the first transferable black-box adversarial attack on medical vision-language models, using imperceptible background perturbations to mislead clinical diagnoses across six imaging modalities.
View Cached Full Text
Cached at: 04/22/26, 01:58 AM
Paper page - When Background Matters: Breaking Medical Vision Language Models by Transferable Attack
Source: https://huggingface.co/papers/2604.17318
https://huggingface.co/login?next=%2Fpapers%2F2604.17318- 
Abstract
MedFocusLeak enables transferable black-box attacks on vision-language models for medical imaging by injecting imperceptible perturbations that redirect model attention, demonstrating significant vulnerabilities in clinical diagnostic reasoning.
Vision-Language Models(VLMs) are increasingly used inclinical diagnostics, yet their robustness toadversarial attacks remains largely unexplored, posing serious risks. Existing medical attacks focus on secondary objectives such as model stealing or adversarial fine-tuning, whiletransferable attacks from natural images introduce visible distortions that clinicians can easily detect. To address this, we propose MedFocusLeak, a highly transferable black-box multimodal attack that induces incorrect yet clinically plausible diagnoses while keeping perturbations imperceptible. The method injects coordinated perturbations into non-diagnostic background regions and employs anattention distraction mechanismto shift the model’s focus away from pathological areas. Extensive evaluations across sixmedical imagingmodalities show that MedFocusLeak achieves state-of-the-art performance, generating misleading yet realistic diagnostic outputs across diverse VLMs. We further introduce a unified evaluation framework with novel metrics that jointly capture attack success andimage fidelity, revealing a critical weakness in the reasoning capabilities of modern clinical VLMs.
View arXiv pageView PDFProject pageGitHub2Add to collection
Community
Paper submitter
The First black box transferable attack paper for Medical Vision Language Models
Upload images, audio, and videos by dragging in the text input, pasting, orclicking here.
Tap or paste here to upload images
https://huggingface.co/login?next=%2Fpapers%2F2604.17318-
Get this paper in your agent:
hf papers read 2604\.17318
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2604.17318 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2604.17318 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2604.17318 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Backdoor Learning in Language Models and Vision-Language Models
This dissertation addresses security and efficiency in AI by analyzing backdoor attacks in language and vision-language models, proposing detection frameworks and novel attack methods, and introducing efficient multimodal models for clinical applications.
Aloe-Vision: Robust Vision-Language Models for Healthcare
Aloe-Vision introduces a family of open medical Vision-Language Models trained on a quality-filtered mixture of medical and general data, along with a new benchmark CareQA-Vision for reliable evaluation. The models demonstrate competitive performance while highlighting vulnerabilities to adversarial inputs.
Mitigating Class-Tail Undercoverage in Medical Vision-Language Models under Clinical Shift
Introduces CALCoDe, a post-hoc reliability layer for frozen medical vision-language models that mitigates class-tail undercoverage under clinical shift, achieving strong worst-class accepted coverage across multiple dermatology shifts and VLM backbones.
@QuixiAI: https://arxiv.org/abs/2509.21401 this is the coolest thing I've seen in at least an hour @TroyDoesAI @elder_plinius @ma…
Proposes JaiLIP, a method that jailbreaks vision-language models by generating imperceptible adversarial images using loss-guided perturbation, achieving high toxicity and outperforming existing methods.
Stress-testing medical large language models reveals latent safety pathology beyond benchmark accuracy
This paper introduces AI-MASLD, a stress-audit framework for medical LLMs that reveals how benchmark accuracy can hide serious safety failures, and demonstrates that open-weight models can match or exceed proprietary ones on safety dimensions.