Evolving Safety Landscape of Multi-modal Large Language Models: A Survey of Emerging Threats and Safeguards

# Evolving Safety Landscape of Multi-modal Large Language Models: A Survey of Emerging Threats and Safeguards
Source: [https://arxiv.org/html/2608.07535](https://arxiv.org/html/2608.07535)
Xi Li1Shu Zhao2Xiaohan Zou3Fei Zhao1Fuxiao Liu2 Yusen Zhang4Cheng Han5Yushun Dong6Jiaqi Wang7 1University of Alabama at Birmingham,2NVIDIA,3Penn State University,4Columbia University, 5University of Missouri\-Kansas City,6Florida State University,7Auburn University

###### Abstract

Multi\-modal large language models \(MLLMs\) integrate heterogeneous modalities through modality alignment and fusion, enabling stronger understanding and reasoning\. However, this architectural shift reshapes the safety landscape of machine learning\. Increased model complexity and cross\-modal interactions give rise to novel threats, including compromised modality integration, modality misalignment, and fused safety risks, reflecting shifts in threat modeling beyond uni\-modal assumptions\. These shifts, in turn, impose new constraints on safety solutions not captured by existing frameworks rooted in uni\-modal learning\. Motivated by these challenges, this survey provides a systematic analysis of the evolving safety landscape of MLLMs\. We first propose a multimodal grounded taxonomy of safety threats and analyze shifts in threat models, covering adversarial attacks, data poisoning, jailbreaks, and hallucinations\. We then summarize updated safety assumptions and organize recent advances in MLLM safety strategies accordingly\. Finally, we discuss open challenges and future directions to inform the development of more principled and scalable safety mechanisms for multimodal systems\.

## 1Introduction

Modern multi\-modal learning employs large models, such as large language models \(LLMs\), to integrate diverse modalities \(e\.g\., text, image, audio, and video\) for enhanced understanding and decision\-making\(Liet al\.,[2022a](https://arxiv.org/html/2608.07535#bib.bib79); Alayracet al\.,[2022](https://arxiv.org/html/2608.07535#bib.bib78); Liuet al\.,[2023b](https://arxiv.org/html/2608.07535#bib.bib1); Brownet al\.,[2020](https://arxiv.org/html/2608.07535#bib.bib80); DeepSeek\-AI,[2025](https://arxiv.org/html/2608.07535#bib.bib81); team,[2024](https://arxiv.org/html/2608.07535#bib.bib155)\), enabling applications in healthcare, autonomous driving, and legal decisions\. Its foundation lies in modality alignment, which maps heterogeneous features into a shared representation space, and modality fusion, which integrates aligned information for more comprehensive and accurate reasoning\(Yinet al\.,[2024a](https://arxiv.org/html/2608.07535#bib.bib117); Wanget al\.,[2023b](https://arxiv.org/html/2608.07535#bib.bib121); Zhuet al\.,[2023b](https://arxiv.org/html/2608.07535#bib.bib120); Xuet al\.,[2023](https://arxiv.org/html/2608.07535#bib.bib119); Baltrušaitiset al\.,[2018](https://arxiv.org/html/2608.07535#bib.bib118)\)\.

With the shift toward multi\-modal architectures, the safety landscape of machine learning is undergoing a significant transformation\. The unique properties of multi\-modal learning introduce new safety challenges: \(1\) additional modalities expand the attack surface by introducing their own vulnerabilities; \(2\) cross\-modal interactions may exhibit semantic inconsistencies and can be exploited to induce adversarial misalignment; and \(3\) the fusion process can be exploited, where malicious signals that appear benign in isolation may trigger harmful behavior when combined\. These emerging threats reflect a shift in threat modeling, where attackers require only partial system access and can exploit surfaces beyond the input space, leading to cross\-modal vulnerabilities\.

These shifts further introduce new constraints on safety solutions that go beyond classic uni\-modal assumptions\. For example, defenses can no longer presume which modality is compromised or the specific type of threat, and must instead align with core multi\-modal objectives such as modality alignment and fusion, while also monitoring internal model stages\. Addressing these constraints calls for safety mechanisms that are both effective and cost\-efficient, including safety\-aware fine\-tuning that preserves inter\-modal coherence, safety\-integrated preference optimization, and training\-free inference\-time solutions\.

These changes motivate a comprehensive survey on the safety of multi\-modal large language models \(MLLMs\)\. Recent surveys, summarized in Table[1](https://arxiv.org/html/2608.07535#S1.T1), provide valuable overviews of attacks, defenses, and evaluation protocols\(Liuet al\.,[2024e](https://arxiv.org/html/2608.07535#bib.bib285); Wanget al\.,[2024d](https://arxiv.org/html/2608.07535#bib.bib289); Fanet al\.,[2024](https://arxiv.org/html/2608.07535#bib.bib288); Wanget al\.,[2024e](https://arxiv.org/html/2608.07535#bib.bib291); Jinet al\.,[2024](https://arxiv.org/html/2608.07535#bib.bib290); Yeet al\.,[2025](https://arxiv.org/html/2608.07535#bib.bib287); Liuet al\.,[2025b](https://arxiv.org/html/2608.07535#bib.bib286)\), but largely adopt uni\-modal taxonomies that organize studies by training/test threats or by separating vision\-side and text\-side attacks\. Such perspectives may not fully capture risks unique to multi\-modal systems, particularly those arising from cross\-modal interactions\. Moreover, prior surveys largely enumerate threats and defenses, with limited discussion of paradigm shifts in multi\-modal safety, such as evolving threat models and safety constraints, offering limited guidance for future research on multi\-modal safety\.

Motivated by this gap, this survey provides a systematic analysis of the evolving safety landscape shaped by MLLMs \(Figure[1](https://arxiv.org/html/2608.07535#S1.F1)\), with the core contributions outlined as follows:

- •We propose a new taxonomy of emerging safety threats in MLLMs arising from complex model architectures and cross\-modal interactions, including compromised modality integration, cross\-modal misalignment, and risks at the fusion stage\. We further summarize the evolving threat modeling reflected by these threats\.
- •We summarize updated assumptions required for safety solutions in multi\-modal settings and categorize recent advances in multi\-modal safety within this framework, including supervised safety fine\-tuning, preference\-based optimization, and training\-free methods\.
- •As safety research shifts from uni\-modal to multi\-modal systems, we highlight the need to address the increased structural complexity and interaction dynamics of MLLMs, and outline key directions for future research\.

Although we discuss MLLM safety broadly, much of the literature focuses on vision\-language models \(VLMs\), the most widely studied instantiation\. Nevertheless, our key concepts generalize to MLLMs with additional modalities \(e\.g\., audio, video, and sensory inputs\)\. The remainder of the paper is organized as follows: Section[2](https://arxiv.org/html/2608.07535#S2)reviews uni\-modal safety foundations; Section[3](https://arxiv.org/html/2608.07535#S3)presents MLLM characteristics, a taxonomy of emerging threats, and evolving threat models; Section[4](https://arxiv.org/html/2608.07535#S4)summarizes updated safety assumptions and recent safety solution advances; Section[5](https://arxiv.org/html/2608.07535#S5)discusses future directions; and Section[6](https://arxiv.org/html/2608.07535#S6)concludes\.

![Refer to caption](https://arxiv.org/html/2608.07535v1/x1.png)Figure 1:A multi\-modal perspective on the evolving safety landscape, illustrated with Vision\-Language LLM\.SurveyVenueThreatsSolutionsMMTax\.SafetyEvoKey ContributionsAPJHLiuet al\.\([2024e](https://arxiv.org/html/2608.07535#bib.bib285)\)IJCAI✓✓✓Concise overview and basic categorization of VLM securityFanet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib288)\)SMC✓✓✓Overview of image\-induced attacks and defensesWanget al\.\([2024e](https://arxiv.org/html/2608.07535#bib.bib291)\)FCS✓✓✓✓VLM security: attacks, defenses, and evaluationJinet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib290)\)arXiv✓✓Comprehensive survey of jailbreak in LLMs and MLLMsWanget al\.\([2024d](https://arxiv.org/html/2608.07535#bib.bib289)\)EMNLP✓✓Eovolving Jailbreak landscape from LLMs to MLLMsYeet al\.\([2025](https://arxiv.org/html/2608.07535#bib.bib287)\)arXiv✓✓✓✓Lifecycle\-based classification of VLM securityacross attacks, defenses, and evaluationLiuet al\.\([2025b](https://arxiv.org/html/2608.07535#bib.bib286)\)TNNLS✓✓✓Overview of VLM attacks and evaluation datasets/modelsOurs\-✓✓✓✓✓✓✓MM\-grounded taxonomy of MLLM safety andin\-depth analysis of safety landscape shifts

Table 1:Recent surveys on MLLM safety\. A: adversarial attacks; P: poisoning attacks; J: jailbreak attacks; H: hallucination; MM Tax\.: MM\-grounded taxonomy; Safety Evo: safety landscape evolution; VLM: vision\-language model\.
## 2Existing Safety Landscape

To provide necessary context, we review the well\-established safety landscape shaped primarily by uni\-modal learning\. We follow standard attacker access definitions: white\-box \(full model knowledge\), black\-box \(query\-only access\), and gray\-box \(partial knowledge, e\.g\., training data, architecture without parameters, or logits\)\.

### 2\.1Adversarial Attacks

Adversarial attacks seek to induce incorrect outputs at inference time by adding imperceptible perturbations to inputs\. Attackers manipulate test\-time inputs without access to training data, operating in either black\-box\(Chenet al\.,[2017a](https://arxiv.org/html/2608.07535#bib.bib217)\)or white\-box\(Goodfellowet al\.,[2015](https://arxiv.org/html/2608.07535#bib.bib214); Madryet al\.,[2018b](https://arxiv.org/html/2608.07535#bib.bib215); Carlini and Wagner,[2017](https://arxiv.org/html/2608.07535#bib.bib216)\)settings\. Such perturbations are typically generated via optimization\-based methods that either target specific incorrect outputs or maximize prediction error, while satisfying norm\-based constraints \(e\.g\.,l∞l\_\{\\infty\},l2l\_\{2\}\) to preserve perceptual imperceptibility, especially for images\.

Defenses aim to enhance robustness against subtle perturbations, typically assuming full access to the victim model, sometimes with a small clean validation set\. Adversarial trainingMadryet al\.\([2018a](https://arxiv.org/html/2608.07535#bib.bib92)\)improves robustness by iteratively generating adversarial examples and training the model to correctly classify them, while random smoothingCohenet al\.\([2019](https://arxiv.org/html/2608.07535#bib.bib150)\); Liet al\.\([2019](https://arxiv.org/html/2608.07535#bib.bib218)\)injects Gaussian noise and averages predictions to smooth decision boundaries against small perturbations\.

### 2\.2Data Poisoning

Data poisoning manipulates model behavior by injecting malicious training samples, assuming attacker access to the training set\. Attacks range from label flippingXiaoet al\.\([2012](https://arxiv.org/html/2608.07535#bib.bib207)\); Zhanget al\.\([2021](https://arxiv.org/html/2608.07535#bib.bib208)\)that degrades overall performance to backdoor attacks that induce targeted misbehavior via trigger patterns while preserving clean accuracy\. Triggers can take various forms, including subtle perturbationsNguyen and Tran \([2021](https://arxiv.org/html/2608.07535#bib.bib96);[2020](https://arxiv.org/html/2608.07535#bib.bib204)\); Sahaet al\.\([2020](https://arxiv.org/html/2608.07535#bib.bib203)\), imperceptible visual patterns in imagesGuet al\.\([2019](https://arxiv.org/html/2608.07535#bib.bib201)\); Chenet al\.\([2017b](https://arxiv.org/html/2608.07535#bib.bib202)\), and specific tokensLiet al\.\([2021a](https://arxiv.org/html/2608.07535#bib.bib123)\), sentencesDaiet al\.\([2019](https://arxiv.org/html/2608.07535#bib.bib124)\), or writing stylesQiet al\.\([2021](https://arxiv.org/html/2608.07535#bib.bib206)\)in text\.

Poisoning defenses aim to reduce the impact of poisoned data while preserving model utility\. Pre\-training defenses detect and remove suspicious samples from the training setTranet al\.\([2018](https://arxiv.org/html/2608.07535#bib.bib128)\); Paudiceet al\.\([2018](https://arxiv.org/html/2608.07535#bib.bib103)\), training\-time defenses leverage white\-box access to select reliable samples or reduce poisoning effects during optimizationDiakonikolaset al\.\([2019](https://arxiv.org/html/2608.07535#bib.bib210)\); Shen and Sanghavi \([2019](https://arxiv.org/html/2608.07535#bib.bib212)\); Liet al\.\([2021b](https://arxiv.org/html/2608.07535#bib.bib209)\); Levine and Feizi \([2021](https://arxiv.org/html/2608.07535#bib.bib135)\); Wanget al\.\([2022](https://arxiv.org/html/2608.07535#bib.bib134)\), and post\-training defenses use white\-box access and a small clean set to detect poisoned modelsWanget al\.\([2019](https://arxiv.org/html/2608.07535#bib.bib125);[2023c](https://arxiv.org/html/2608.07535#bib.bib213)\), identify poisoned inputsGaoet al\.\([2019](https://arxiv.org/html/2608.07535#bib.bib126)\); Liet al\.\([2022b](https://arxiv.org/html/2608.07535#bib.bib109)\), or enforce benign behaviorLiuet al\.\([2018](https://arxiv.org/html/2608.07535#bib.bib127)\); Liet al\.\([2021c](https://arxiv.org/html/2608.07535#bib.bib129)\); Zenget al\.\([2022](https://arxiv.org/html/2608.07535#bib.bib132)\); Liet al\.\([2025a](https://arxiv.org/html/2608.07535#bib.bib110)\)\.

### 2\.3Jailbreak

Jailbreak attacks bypass LLM safety by crafting prompts that elicit unauthorized rather than misleading outputs\. White\-box attacks optimize adversarial prefixes or suffixesZouet al\.\([2023](https://arxiv.org/html/2608.07535#bib.bib220)\); Joneset al\.\([2023](https://arxiv.org/html/2608.07535#bib.bib219)\); Zhuet al\.\([2023c](https://arxiv.org/html/2608.07535#bib.bib221)\), Gray\-box attacks exploit logit access to manipulate prompts and token selectionGuoet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib222)\); Zhaoet al\.\([2024b](https://arxiv.org/html/2608.07535#bib.bib223)\), or perform limited retraining via small\-scale poisoningQiet al\.\([2024b](https://arxiv.org/html/2608.07535#bib.bib224)\); Zhanet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib225)\); Yanget al\.\([2023](https://arxiv.org/html/2608.07535#bib.bib226)\)\. and black\-box attacks leverage role\-playing or contextual reasoning to bypass safety alignmentWanget al\.\([2023a](https://arxiv.org/html/2608.07535#bib.bib227)\); Weiet al\.\([2023](https://arxiv.org/html/2608.07535#bib.bib228)\); Liet al\.\([2023c](https://arxiv.org/html/2608.07535#bib.bib229)\)\. Other methods include LLM\-generated adversarial promptsDenget al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib231)\); Liuet al\.\([2024d](https://arxiv.org/html/2608.07535#bib.bib87)\)and niche\-language evasionYuanet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib230)\)\.

Jailbreak defenses seek to maintain safety alignment\. Black\-box defenses filter or preprocess adversarial prompts to neutralize harmful intentAlon and Kamfonas \([2023](https://arxiv.org/html/2608.07535#bib.bib232)\); Jainet al\.\([2023](https://arxiv.org/html/2608.07535#bib.bib233)\), while white\-box defenses impose stronger guardrails via safety fine\-tuningBianchiet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib236)\); Denget al\.\([2023](https://arxiv.org/html/2608.07535#bib.bib237)\), reinforcement learning from human feedback \(RLHF\)Ouyanget al\.\([2022](https://arxiv.org/html/2608.07535#bib.bib234)\), or model self\-correction to reduce unsafe outputsSunet al\.\([2023b](https://arxiv.org/html/2608.07535#bib.bib235)\)\.

### 2\.4Hallucination

Hallucination refers to the generation of confident but factually incorrect or unsupported informationHuanget al\.\([2025b](https://arxiv.org/html/2608.07535#bib.bib65)\), arising from limitations in training data, objectives, or decoding\. It is mainly a reliability issue rather than a security issue involving active attackers\. Common causes include noisy or biased dataBenderet al\.\([2021](https://arxiv.org/html/2608.07535#bib.bib66)\); Shenget al\.\([2020](https://arxiv.org/html/2608.07535#bib.bib67)\); Wallaceet al\.\([2019](https://arxiv.org/html/2608.07535#bib.bib75)\), overfitting to spurious language patternsMcKennaet al\.\([2023](https://arxiv.org/html/2608.07535#bib.bib77)\); Wanget al\.\([2021](https://arxiv.org/html/2608.07535#bib.bib76)\); Tuet al\.\([2020](https://arxiv.org/html/2608.07535#bib.bib74)\), and limited uncertainty modelingFarquharet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib73)\)\.

To mitigate hallucination, some works improve data quality through filtering, deduplication, and high\-quality structured or instructional data\(Penedoet al\.,[2023](https://arxiv.org/html/2608.07535#bib.bib69); Leeet al\.,[2021](https://arxiv.org/html/2608.07535#bib.bib70); Guuet al\.,[2020](https://arxiv.org/html/2608.07535#bib.bib68); Raffelet al\.,[2020](https://arxiv.org/html/2608.07535#bib.bib64)\)\. Other works adopt RLHF, contrastive learning, factual\-consistency losses, or chain\-of\-thought supervision\(Ziegleret al\.,[2019](https://arxiv.org/html/2608.07535#bib.bib41); Sunet al\.,[2023a](https://arxiv.org/html/2608.07535#bib.bib63); Weiet al\.,[2022](https://arxiv.org/html/2608.07535#bib.bib42)\)\. Additional works employ retrieval\-augmented generation \(RAG\), prompt constraints, and post\-hoc verification using confidence estimation or external tools\(Lewiset al\.,[2020](https://arxiv.org/html/2608.07535#bib.bib40); Izacard and Grave,[2020](https://arxiv.org/html/2608.07535#bib.bib6); Zhouet al\.,[2022](https://arxiv.org/html/2608.07535#bib.bib38); Beurer\-Kellneret al\.,[2023](https://arxiv.org/html/2608.07535#bib.bib35); Gaoet al\.,[2023](https://arxiv.org/html/2608.07535#bib.bib5); Farquharet al\.,[2024](https://arxiv.org/html/2608.07535#bib.bib73); Liet al\.,[2023d](https://arxiv.org/html/2608.07535#bib.bib37)\)\.

## 3Emerging Multi\-Modal Safety Threats

![Refer to caption](https://arxiv.org/html/2608.07535v1/x2.png)Figure 2:Illustration of \(a\) compromised visual integration\(Gaoet al\.,[2024](https://arxiv.org/html/2608.07535#bib.bib160)\); \(b\) targeted misalignment for jailbreaks\(Shayeganiet al\.,[2024](https://arxiv.org/html/2608.07535#bib.bib197)\); \(c\) fused backdoor attacks\(Walmeret al\.,[2022](https://arxiv.org/html/2608.07535#bib.bib166)\)\.To integrate diverse data sources and enable cross\-modal reasoning, MLLMs rely on two core mechanisms: modality alignment and modality fusion \(Fig\.[1](https://arxiv.org/html/2608.07535#S1.F1)\)\. Modality alignment projects heterogeneous inputs into a shared embedding space using modality\-specific encoders while preserving essential features, whereas modality fusion combines aligned representations \(e\.g\., via cross\-attention\) to capture cross\-modal semantics and support downstream reasoning tasks\. However, the inherent complexity of these mechanisms and dynamic cross\-modal interactions give rise to a new set of safety threats differ fundamentally from uni\-modal settings\. To characterize these emerging threats, we propose a new taxonomy of multi\-modal safety risks \(Fig\.[2](https://arxiv.org/html/2608.07535#S3.F2)\): Compromised Modality Integration, Modality Misalignment, and Fused Safety Risks\.

### 3\.1Compromised Modality Integration

When multiple modalities are integrated, threats may originate fromindividual modalities, where manipulation of one or more inputs propagates through the integration process and compromises overall model behavior\. We refer to such threats as Compromised Modality Integration \(Fig\.[2](https://arxiv.org/html/2608.07535#S3.F2)\(a\)\)\.

Adversarial attacks in MLLMs primarily target the vision encoder by inserting visual perturbations that disrupt its representations\. These errors then propagate through the alignment and fusion processes, compromising the behavior of the entire system\. For example,Luoet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib158)\)andSchlarmann and Hein \([2023](https://arxiv.org/html/2608.07535#bib.bib164)\)craft visual perturbations that induce MLLMs to generate attacker\-specified text\. Several worksYinet al\.\([2023](https://arxiv.org/html/2608.07535#bib.bib159)\); Wanget al\.\([2024a](https://arxiv.org/html/2608.07535#bib.bib161);[g](https://arxiv.org/html/2608.07535#bib.bib162)\)maximize the embedding distance between clean and perturbed images, distorting the model’s understanding\.Gaoet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib160)\)introduce perturbations that delay the end\-of\-sequence token, increasing uncertainty in output length\.Baileyet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib163)\)show that adversarial images can bypass safety filters, leak context, and trigger misleading\.Fuet al\.\([2023](https://arxiv.org/html/2608.07535#bib.bib248)\)demonstrate that subtle perturbations can covertly induce MLLM agents to perform malicious actions, such as leaking conversations\.Caiet al\.\([2025a](https://arxiv.org/html/2608.07535#bib.bib247)\)boost transferability by applying visual transformations to ensure all perturbed variants fool the model\.Zhanget al\.\([2025b](https://arxiv.org/html/2608.07535#bib.bib243)\)improve scalability, by using a self\-supervised setup where a random image generates perturbations guided by the original\.Zhanget al\.\([2024a](https://arxiv.org/html/2608.07535#bib.bib246)\)propose a benchmark of perturbed inputs that consistently trigger low\-quality MLLM responses\.

Liu and Zhang \([2025](https://arxiv.org/html/2608.07535#bib.bib251)\)poison the visual encoder to map backdoored images to a target representation, inducing MLLM outputs aligned with the target content\. MLLMs inherit jailbreak vulnerabilities from their LLM core, making them susceptible to harmful or unauthorized outputsLiuet al\.\([2024d](https://arxiv.org/html/2608.07535#bib.bib87)\); Yuet al\.\([2024b](https://arxiv.org/html/2608.07535#bib.bib238)\); Denget al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib231)\)\. Attackers can manipulate non\-text modalities, such as image, to bypass safety mechanisms of LLM, triggering jailbreak responses with benign text and malicious visuals\. For example,Qiet al\.\([2024a](https://arxiv.org/html/2608.07535#bib.bib200)\)optimize a visual adversarial example to universally jailbreak an aligned LLM\. FigStepGonget al\.\([2025](https://arxiv.org/html/2608.07535#bib.bib199)\)embeds rephrased jailbreak prompts within images to deceive the model\.Shayeganiet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib197)\)craft visual perturbations that bring visual signals close to jailbreak content in the embedding space\.

During multi\-modal integration, perceptual flaws in individual modalities can propagate and lead to system\-wide hallucinations\. For instance, low\-resolution images may obscure small but critical objects, or speech recognition may omit key words\. LLaVA\-NextLiuet al\.\([2024c](https://arxiv.org/html/2608.07535#bib.bib8)\)shows that higher\-resolution inputs reduce hallucinations, while PrismaticKaramchetiet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib26)\)leverages expert encoders to preserve fine\-grained details\. When such information is lost, the model tends to rely on pretraining priors to “fill in the gaps,” often resulting in hallucinated content\.

CategorySub\-CategoryPaperThreat ModelYear & VenueCap\. 1Cap\. 2Cap\. 3CompromisedModalityIntegrationAdversarial AttacksCroPALuoet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib158)\)✓ICLR’24Verbose ImagesGaoet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib160)\)✓ICLR’24Image HijacksBaileyet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib163)\)✓✓ICML’24AnyAttackZhanget al\.\([2025b](https://arxiv.org/html/2608.07535#bib.bib243)\)✓CVPR’25PoisoningBadVisionLiu and Zhang \([2025](https://arxiv.org/html/2608.07535#bib.bib251)\)✓CVPR’25JailbreakAdversarial Vision JailbreaksQiet al\.\([2024a](https://arxiv.org/html/2608.07535#bib.bib200)\)✓✓AAAI’24Jailbreak In PiecesShayeganiet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib197)\)✓✓✓ICLR’24FigStepGonget al\.\([2025](https://arxiv.org/html/2608.07535#bib.bib199)\)✓✓AAAI’25HallucinationLLaVA\-NextLiuet al\.\([2024c](https://arxiv.org/html/2608.07535#bib.bib8)\)\-\-\-arXiv’24Prismatic VLMsKaramchetiet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib26)\)\-\-\-ICML’24ModalityMis\-alignmentUntargetedMis\-alignmentPandora’s BoxLiuet al\.\([2024a](https://arxiv.org/html/2608.07535#bib.bib165)\)✓NeurIPS’24Break the Visual PerceptionWanget al\.\([2024g](https://arxiv.org/html/2608.07535#bib.bib162)\)✓✓ACM MM’24TMM AttackWanget al\.\([2024a](https://arxiv.org/html/2608.07535#bib.bib161)\)✓IEEE S & P’24TargetedMis\-alignmentVLM Adversarial RobustnessZhaoet al\.\([2023a](https://arxiv.org/html/2608.07535#bib.bib157)\)✓NeurIPS’23Adversarial IllusionsBagdasaryanet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib240)\)✓USENIX Security’24ShadowcastXuet al\.\([2024a](https://arxiv.org/html/2608.07535#bib.bib239)\)✓NeurIPS’24BadCLIPBaiet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib167)\)✓CVPR’24VLM Safety EvaluationTuet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib241)\)✓ECCV’24TrojVLMLyuet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib250)\)✓ECCV’24Revisit VLM BackdoorLianget al\.\([2025](https://arxiv.org/html/2608.07535#bib.bib171)\)✓CVPR’25VLOODLyuet al\.\([2025](https://arxiv.org/html/2608.07535#bib.bib252)\)✓ICLR’25BadSemZhonget al\.\([2025](https://arxiv.org/html/2608.07535#bib.bib253)\)✓arXiv’25SemanticInconsistencyMinigpt\-4Zhuet al\.\([2023a](https://arxiv.org/html/2608.07535#bib.bib10)\)\-\-\-arXiv’23Blip\-2Liet al\.\([2023a](https://arxiv.org/html/2608.07535#bib.bib23)\)\-\-\-ICML’23InstructBLIPDaiet al\.\([2023](https://arxiv.org/html/2608.07535#bib.bib22)\)\-\-\-arXiv’23mPLUG\-Owl2Yeet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib9)\)\-\-\-CVPR’24FusedSafety RisksFusion\-triggeredthreatsDual\-Key BackdoorWalmeret al\.\([2022](https://arxiv.org/html/2608.07535#bib.bib166)\)✓CVPR’22AnyDoor AttackLuet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib169)\)✓✓arXiv’24ImgTrojanTaoet al\.\([2025](https://arxiv.org/html/2608.07535#bib.bib198)\)✓✓✓NAACL’25Modality ConflictsCurse of Multi\-ModalitiesLenget al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib2)\)\-\-\-arXiv’24

Table 2:Representative studies of multi\-modal safety threats under the new taxonomy\. Cap\. 1: relaxed capability; Cap\. 2: expanded attack surface; Cap\. 3: cross\-modality threats\.
### 3\.2Modality Misalignment

Adversaries can manipulatecross\-modal embeddingsto disrupt semantic or structural alignment between modalities, misleading the model during inference \(Fig\.[2](https://arxiv.org/html/2608.07535#S3.F2)\(b\)\)\. Adversarial misalignment can be: \(1\)Untargeted, where the perturbed modality’s embedding diverges from clean modalities; or \(2\)Targeted, where embeddings are manipulated to mimic harmful representations\. Misalignment may also arise naturally due to inherent inconsistencies across modalities\.

Foruntargetedmisalignment, existing works generally maximize cross\-modal discrepancy/distance in the shared embedding space, thereby disrupting alignment between modalities\. For example,Liuet al\.\([2024a](https://arxiv.org/html/2608.07535#bib.bib165)\)generate a universal adversarial patch by minimizing cosine similarity between visual and textual embeddings\.Wanget al\.\([2024g](https://arxiv.org/html/2608.07535#bib.bib162)\)introduce visual perturbations that disrupt relations among visual tokens or weaken global visual–text semantic alignment\.Wanget al\.\([2024a](https://arxiv.org/html/2608.07535#bib.bib161)\)focus on suppressing features that promote modality consistency while amplifying those that increase cross\-modal discrepancy\.

Fortargetedmisalignment,Zhaoet al\.\([2023a](https://arxiv.org/html/2608.07535#bib.bib157)\)align adversarial image embeddings with \(1\) the target text, \(2\) embeddings of corresponding target images, or \(3\) model outputs associated with the target text\.Shayeganiet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib197)\)optimize visual perturbations to mimic embeddings of harmful content \(e\.g\., OCR\-decoded jailbreak prompts\), enabling cross\-modal jailbreaks\. Similarly,Qiet al\.\([2024a](https://arxiv.org/html/2608.07535#bib.bib200)\)craft adversarial visuals to maximize the likelihood of harmful text generation, breaking safety alignment\.Bagdasaryanet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib240)\)andTuet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib241)\)craft perturbations that minimize cosine similarity between the input and an unrelated adversarial target text\. Furthermore, several worksLuet al\.\([2023](https://arxiv.org/html/2608.07535#bib.bib245)\); Hanet al\.\([2023](https://arxiv.org/html/2608.07535#bib.bib244)\); Huanget al\.\([2025a](https://arxiv.org/html/2608.07535#bib.bib242)\)improve the transferability of embedding misalignment\.

Some methods induce targeted misalignment via data poisoning\.Lyuet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib250)\); Lianget al\.\([2025](https://arxiv.org/html/2608.07535#bib.bib171)\); Lyuet al\.\([2025](https://arxiv.org/html/2608.07535#bib.bib252)\)implant backdoors in MLLMs that trigger targeted harmful outputs when images contain backdoor patterns\. Further,Xuet al\.\([2024a](https://arxiv.org/html/2608.07535#bib.bib239)\)poison the model to align embeddings of original and destination concept images, causing it to generate text about the destination when shown the original\.Baiet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib167)\)introduce sample\-specific triggers and trigger\-aware context prompts to shift visual embeddings toward a target class\.Zhonget al\.\([2025](https://arxiv.org/html/2608.07535#bib.bib253)\)exploit semantic inconsistencies between paired modalities, e\.g\., mismatches in color or object descriptions, to trigger backdoor behaviors\.

Inherent semantic inconsistencies across modalities can naturally cause misalignment and lead to hallucinations\. Besides, modality alignment projects heterogeneous inputs into a shared embedding space, which is often anchored to text\. Improper mappings may distort semantics and induce hallucinated outputs\(Zhuet al\.,[2023a](https://arxiv.org/html/2608.07535#bib.bib10)\)\. For example, visual features from a medical scan may align with incorrect clinical concepts, causing the model to generate convincing but unfaithful outputs\. Methods like BLIP\-2\(Liet al\.,[2023a](https://arxiv.org/html/2608.07535#bib.bib23)\)and InstructBLIP\(Daiet al\.,[2023](https://arxiv.org/html/2608.07535#bib.bib22)\)use query\-based feature extraction to improve cross\-modal faithfulness, but fail when queries miss critical visual details\. Similarly, mPLUG\-Owl2\(Yeet al\.,[2024](https://arxiv.org/html/2608.07535#bib.bib9)\)employs modality\-adaptive attention, yet misalignment persists under domain shifts or out\-of\-distribution inputs\.

### 3\.3Fused Safety Risks

Fused safety risks exploit the fusion mechanism, where adversarial signals are benign in isolation butharmful when combinedduring fusion, as shown in Fig\.[2](https://arxiv.org/html/2608.07535#S3.F2)\(c\)\. These threats manifest deeper in the model pipeline and are harder to detect\. Beyond adversarial manipulation, naturally conflicting or imbalanced cross\-modal evidence can also lead to unsafe behaviors during fusion\.

Several studies illustrate fused safety risks\.Walmeret al\.\([2022](https://arxiv.org/html/2608.07535#bib.bib166)\)embed backdoor triggers in both image and text, where each modality appears benign in isolation but induces malicious behavior when combined\.Luet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib169)\)show asymmetric modality roles, with images facilitating trigger injection and text activating malicious responses\.Taoet al\.\([2025](https://arxiv.org/html/2608.07535#bib.bib198)\)associate harmful queries with specific clean images by replacing captions with jailbreak prompts, causing model generates harmful content when presented with both\.Gonget al\.\([2025](https://arxiv.org/html/2608.07535#bib.bib199)\)embed jailbreak prompts in visual inputs and use benign\-looking text to elicit harmful outputs\.Songet al\.\([2025](https://arxiv.org/html/2608.07535#bib.bib258)\)approximate safety decision boundaries in multi\-modal fusion spaces using logistic regression and manipulate fusion\-layer features toward a target decision to induce policy\-violating outputs\.

Beyond attacks, modality fusion can also induce hallucinations when cross\-modal evidence is imbalanced, allowing dominant linguistic priors to override weaker visual or auditory signals\. This effect is empirically shown in Curse of Multi\-Modalities\(Lenget al\.,[2024](https://arxiv.org/html/2608.07535#bib.bib2)\), where MLLMs frequently default to unimodal dominance, leading to systematic hallucinations under fusion conflicts\.

### 3\.4Shifts in Threat Models

Based on the above discussion of emerging threats, we summarize the key shifts in threat modeling introduced by multi\-modal settings compared to uni\-modal models\. \(1\)Relaxed Capability:In multi\-modal systems, adversaries no longer need full\-system access\. Due to the compositional design, compromising a subset of modalities can suffice\. For example, adversarial images targeting only the visual encoder can trigger downstream misalignment during fusion, resulting in harmful or misleading outputs\. \(2\)Expanded Threat Surface:Multi\-modal systems introduce new threat surfaces beyond input–output mapping, including vulnerabilities in modality alignment and fusion\. These internal stages create deeper, stealthier entry points for threats within the model pipeline\. \(3\)Cross\-Modality Threats:Multi\-modal models enable threats that exploit interactions across modalities\. For example, an adversarial image may trigger a jailbreak, or a malicious prompt may mislead the model’s interpretation of visual input\.

## 4Current Multi\-modal Safety Solutions

In this section, we summarize new safety constraints in multi\-modal systems driven by evolving threat models and review recent advances in safety fine\-tuning, preference\-based optimization, and training\-free approaches\.

CategorySub\-categoryPaperSafety ConstraintsYear & VenueCap\. 1Cap\. 2Cap\. 3SafetyFine\-TuningModality\-WiseFine\-TuningRobust CLIPSchlarmannet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib267)\)✓ICML’24Textual UnlearningChakrabortyet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib279)\)✓EMNLP’24 FindingsCross\-ModalFine\-TuningTGA\-ZSRYuet al\.\([2024a](https://arxiv.org/html/2608.07535#bib.bib269)\)✓✓NeurIPS’24VLGuardZonget al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib261)\)✓✓ICML’24HallusionbenchGuanet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib45)\)✓✓CVPR’24MmcLiuet al\.\([2024b](https://arxiv.org/html/2608.07535#bib.bib46)\)✓✓NAACL’24Robust\-VLGuardWanget al\.\([2025b](https://arxiv.org/html/2608.07535#bib.bib254)\)✓ICCV’25TGAXuet al\.\([2025](https://arxiv.org/html/2608.07535#bib.bib282)\)✓✓ICLR’25BYERonget al\.\([2025](https://arxiv.org/html/2608.07535#bib.bib284)\)✓✓NeurIPS’25UNIGUARDOhet al\.\([2025](https://arxiv.org/html/2608.07535#bib.bib256)\)✓arXiv’25ProEATLuet al\.\([2025](https://arxiv.org/html/2608.07535#bib.bib257)\)✓✓arXiv’25Preference\-BasedOptimizationHuman FeedbackSPA\-VLZhanget al\.\([2025c](https://arxiv.org/html/2608.07535#bib.bib262)\)✓✓CVPR’25AdPOLiuet al\.\([2025a](https://arxiv.org/html/2608.07535#bib.bib263)\)✓✓arXiv’25GuardReasoner\-VLLiuet al\.\([2025c](https://arxiv.org/html/2608.07535#bib.bib273)\)✓✓NeurIPS’25Adversary\-aware DPOWenget al\.\([2025](https://arxiv.org/html/2608.07535#bib.bib260)\)✓✓✓EMNLP’25 FindingsAI FeedbackSilkieLiet al\.\([2023b](https://arxiv.org/html/2608.07535#bib.bib55)\)✓✓arXiv’23MHALOCaiet al\.\([2025b](https://arxiv.org/html/2608.07535#bib.bib58)\)✓✓ACL’25 FindingsTraining\-FreeSolutionsInput\-levelSafeguardAdaShieldWanget al\.\([2024f](https://arxiv.org/html/2608.07535#bib.bib276)\)✓✓ECCV’24BlueSuffixZhaoet al\.\([2025](https://arxiv.org/html/2608.07535#bib.bib280)\)✓✓ICLR’25LLM\-PipelineGuoet al\.\([2025](https://arxiv.org/html/2608.07535#bib.bib266)\)✓NeurIPS’25Internal\-levelInterventionInferAlignerWanget al\.\([2024b](https://arxiv.org/html/2608.07535#bib.bib277)\)✓✓EMNLP’24ASTRAWanget al\.\([2025a](https://arxiv.org/html/2608.07535#bib.bib283)\)✓✓CVPR’25ETADinget al\.\([2025](https://arxiv.org/html/2608.07535#bib.bib278)\)✓✓ICLR’25HiddenDetectJianget al\.\([2025](https://arxiv.org/html/2608.07535#bib.bib281)\)✓✓✓ACL’25SafePTRChenet al\.\([2025a](https://arxiv.org/html/2608.07535#bib.bib264)\)✓✓✓NeurIPS’25ALFARAnet al\.\([2025](https://arxiv.org/html/2608.07535#bib.bib259)\)✓✓✓NeurIPS’25SASAWanget al\.\([2025c](https://arxiv.org/html/2608.07535#bib.bib272)\)✓✓✓ACM MM’25CIDERXuet al\.\([2024b](https://arxiv.org/html/2608.07535#bib.bib271)\)✓✓EMNLP’25 FindingsOutput\-levelControlMLLM\-ProtectorPiet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib275)\)✓✓EMNLP’24ECSOGouet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib265)\)✓ECCV’24IMMUNEGhosalet al\.\([2025](https://arxiv.org/html/2608.07535#bib.bib255)\)✓✓✓CVPR’25DeGFZhanget al\.\([2025a](https://arxiv.org/html/2608.07535#bib.bib53)\)✓ICLR’25

Table 3:Recent advancements in multi\-modal safety solutions\. Cap\. 1: Modality\-Agnostic Readiness; Cap\. 2: Cross\-Modal Safety; Cap\. 3: Internal Safety Intervention\.### 4\.1New Safety Constraints

\(1\)Modality\-Agnostic Readiness:Multi\-modal systems face dynamic threats across modalities, making fixed assumptions about threat types or entry modalities infeasible\. For example, relying solely on text\-based safety mechanisms is insufficient – adversarial visual inputs can bypass language\-focused safeguards by embedding hidden jailbreak triggers, exposing critical blind spots in classic threat assumptions\. \(2\)Cross\-Modal Safety:Designing and ensembling modality\-isolated safety mechanisms can disrupt cross\-modal interactions and introduce unintended inconsistencies\. Instead, safety solutions should operate holistically across modalities to preserve semantic coherence and support effective alignment and fusion\. \(3\)Internal Safety Intervention:Traditional safety mechanisms typically focus on monitoring model inputs and outputs\. However, in multi\-modal systems, threats can emerge at intermediate stages –modality alignment and fusion\. To address such risks, safety solutions should intervene at internal stages, rather than relying solely on surface\-level observations\.

### 4\.2Safety Fine\-Tuning

Safety fine\-tuning adapts pre\-trained models with safety\-oriented data without full retraining, including modality\-wise fine\-tuning within individual modalities or components \(e\.g\., vision encoders or LLMs\) and cross\-modal fine\-tuning that aligns interactions across modalities\.

Several studies adopt modality\-wise fine\-tuning as a cost\-effective safety strategy\. On the vision side, works such asSchlarmannet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib267)\); Maoet al\.\([2023](https://arxiv.org/html/2608.07535#bib.bib268)\); Wanget al\.\([2024c](https://arxiv.org/html/2608.07535#bib.bib270)\); Shiet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib31)\)improve robustness by fine\-tuning the visual encoder to resist adversarial perturbations while preserving feature representations\.Yuet al\.\([2024a](https://arxiv.org/html/2608.07535#bib.bib269)\)further leverage text embeddings to guide attention and correct adversarial visual shifts\. In contrast, other approaches enhance overall VLM safety by intervening in the LLM component\. For example,Chakrabortyet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib279)\)apply textual unlearning to prevent harmful context from propagating in the LLM’s latent space and redirect it toward safer representations\.

Other works investigate cross\-modal fine\-tuning with safety\-aligned datasets\. VLGuard\(Zonget al\.,[2024](https://arxiv.org/html/2608.07535#bib.bib261)\)and Robust\-VLGuard\(Wanget al\.,[2025b](https://arxiv.org/html/2608.07535#bib.bib254)\)provide safety\-aligned datasets with both aligned and misaligned image\-text pairs, improving MLLMs resilience against Gaussian noise and jailbreaks\. UNIGUARD\(Ohet al\.,[2025](https://arxiv.org/html/2608.07535#bib.bib256)\)models both unimodal and cross\-modal harmful signals and enforces interaction\-aware defenses across modalities\. ProEAT\(Luet al\.,[2025](https://arxiv.org/html/2608.07535#bib.bib257)\)applies adversarial training to both visual projection layers and LLM to enhance cross\-modal consistency and robustness\. BYE\(Ronget al\.,[2025](https://arxiv.org/html/2608.07535#bib.bib284)\)filters poisoned fine\-tuning samples by detecting anomalous low\-entropy cross\-modal attention patterns\.Xuet al\.\([2025](https://arxiv.org/html/2608.07535#bib.bib282)\)guide the projection of visual embeddings during alignment using retrieved textual evidence to mitigate toxic\-image\-induced risks\.

Beyond attack scenarios, recent studiesGuanet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib45)\); Liet al\.\([2025b](https://arxiv.org/html/2608.07535#bib.bib54)\)show that hallucination in MLLMs is also strongly influenced by instruction\-tuning data\. LRV\-InstructionLiuet al\.\([2023a](https://arxiv.org/html/2608.07535#bib.bib44);[2024b](https://arxiv.org/html/2608.07535#bib.bib46)\)reduces instruction bias by incorporating both positive and negative instruction samples\. PerturboLLaVAChenet al\.\([2025b](https://arxiv.org/html/2608.07535#bib.bib59)\)introduces adversarial instruction\-vision conflicts to discourage over\-reliance on language priors\. REVERIEFeiet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib61)\)further incorporates reasoning supervision by requiring rationales for correct and incorrect responses, promoting more robust cross\-modal reasoning\.

### 4\.3Preference\-Based Optimization

Preference\-based optimization improves safety alignment by optimizing over relative preferences between model outputs, rather than explicit labels or losses, using human or AI feedback to favor preferred responses over rejected ones\.

Recent studies adopt preference\-based optimization to align MLLMs under adversarial settings\. SPA\-VL\(Zhanget al\.,[2025c](https://arxiv.org/html/2608.07535#bib.bib262)\)introduces a large\-scale safety alignment dataset with preferred and rejected responses for supervised fine\-tuning and subsequent RLHF/Direct Preference Optimization \(DPO\) training\. GuardReasoner\-VL\(Liuet al\.,[2025c](https://arxiv.org/html/2608.07535#bib.bib273)\)adopts a reason\-then\-moderate paradigm with online RL to detect hidden harmful content\.Liuet al\.\([2025a](https://arxiv.org/html/2608.07535#bib.bib263)\)combine classic adversarial training with DPO to update the visual encoder against adversarial image attacks, improving multi\-modal safety alignment\. Similarly,Wenget al\.\([2025](https://arxiv.org/html/2608.07535#bib.bib260)\)propose an adversary\-aware DPO framework, in which the reference model is adversarially trained to provide reliable supervision, and a min\-max objective is applied with perturbations in both image and latent spaces\.

Beyond human feedback, several studies explore RL from AI feedback for hallucination mitigation\. HA\-DPOZhaoet al\.\([2023b](https://arxiv.org/html/2608.07535#bib.bib57)\)and SilkieLiet al\.\([2023b](https://arxiv.org/html/2608.07535#bib.bib55)\)apply DPO to favor accurate over hallucinated responses using AI\-generated feedback or stronger teacher model preferences\. Challenging standard DPO assumptions, POViDLiet al\.\([2025d](https://arxiv.org/html/2608.07535#bib.bib4)\)constructs preference data solely from hallucinated responses, treating ground\-truth instructions as preferred outputs, while HalDetectCaiet al\.\([2025b](https://arxiv.org/html/2608.07535#bib.bib58)\)introduces fine\-grained hallucination\-aware rewards and FDPO to leverage human feedback at finer granularity\.

### 4\.4Training\-Free Solutions

Training\-free solutions improve MLLM safety at inference time without updating model parameters, operating at the input level to block harmful or adversarial content, the internal level to suppress unsafe behaviors via intermediate representations, and the output level to verify or revise generations for safer responses\.

At the input level,Guoet al\.\([2025](https://arxiv.org/html/2608.07535#bib.bib266)\)use an auxiliary LLM as a vision\-free filter to screen and block harmful prompts before they are passed to the multi\-modal model\. AdaShieldWanget al\.\([2024f](https://arxiv.org/html/2608.07535#bib.bib276)\)prepends model inputs with input\-aware defense prompts that automatically and adaptively safeguard MLLMs against jailbreak attacks\. DiffPure\-VLMWanget al\.\([2025b](https://arxiv.org/html/2608.07535#bib.bib254)\)and BlueSuffixZhaoet al\.\([2025](https://arxiv.org/html/2608.07535#bib.bib280)\)mitigate visual adversarial perturbations using diffusion\-based models\. BlueSuffix further improves safety by rewriting textual prompts with an LLM\-based purifier and prepending LLM\-generated suffixes to the input\.

For intervention in internal layers, CIDERXuet al\.\([2024b](https://arxiv.org/html/2608.07535#bib.bib271)\)detects image\-induced jailbreaks by identifying inputs with abnormally low cross\-modal semantic distances in the embedding space\. InferAlignerWanget al\.\([2024b](https://arxiv.org/html/2608.07535#bib.bib277)\)and ASTRAWanget al\.\([2025a](https://arxiv.org/html/2608.07535#bib.bib283)\)suppress jailbreak\-related hidden representations using steering vectors, with InferAligner deriving safety steering vectors from aligned LLMs and ASTRA extracting adversarial steering vectors via image\-based attribution\. SASAWanget al\.\([2025c](https://arxiv.org/html/2608.07535#bib.bib272)\)and ETADinget al\.\([2025](https://arxiv.org/html/2608.07535#bib.bib278)\)adopt multi\-stage safety mechanisms\. SASA projects semantically rich representations from deeper fused layers to early layers to enhance risk perception, while ETA combines shallow alignment with deep sentence\-level selection to enforce safety\. SafePTRChenet al\.\([2025a](https://arxiv.org/html/2608.07535#bib.bib264)\)and HiddenDetectJianget al\.\([2025](https://arxiv.org/html/2608.07535#bib.bib281)\)partition models into vulnerable and safety\-relevant layers\. SafePTR prunes harmful cross\-modal tokens in vulnerable layers, whereas HiddenDetect detects unsafe prompts by measuring the alignment of activations with a refusal\-aware embedding across safety layers\.

For output\-level control,Piet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib275)\)assess response harmfulness using a harm detector and revise unsafe outputs with a detoxifier\. IMMUNEGhosalet al\.\([2025](https://arxiv.org/html/2608.07535#bib.bib255)\)guides token selection during decoding using a learned safety reward model to favor safer continuations\.Gouet al\.\([2024](https://arxiv.org/html/2608.07535#bib.bib265)\)enable self\-checking in MLLMs and regenerate safer outputs by rerouting visual inputs through an alignment\-protected text\-only pipeline\.

Multi\-modal hallucination can be mitigated by guided or self\-correcting decoding\. DeGFZhanget al\.\([2025a](https://arxiv.org/html/2608.07535#bib.bib53)\)verifies and corrects model responses using generative feedback from text\-to\-image models, while MARINEZhaoet al\.\([2024a](https://arxiv.org/html/2608.07535#bib.bib52)\)guides decoding with grounded object representations and classifier\-free guidance to emphasize visual evidence\. At the attention level, EAHLiet al\.\([2025c](https://arxiv.org/html/2608.07535#bib.bib48)\)and VHRZhanget al\.\([2024b](https://arxiv.org/html/2608.07535#bib.bib51)\)mitigate visual attention sink by strengthening image\-token attention in shallow layers, whileAnet al\.\([2025](https://arxiv.org/html/2608.07535#bib.bib259)\)adaptively rebalances attention in multi\-modal RAG based on retrieval scores and query\-context relevance\. To reduce over\-reliance on language priors, WoodpeckerYinet al\.\([2024b](https://arxiv.org/html/2608.07535#bib.bib50)\)detects and corrects hallucinations using visual experts, andLiu \([2025](https://arxiv.org/html/2608.07535#bib.bib47)\)encourage greater reliance on visual context\.

## 5Future Directions

Despite progress, MLLM safety remains challenged by expanded attack surfaces, cross\-modal interactions, and new constraints, motivating future research\.

### 5\.1Resilience to Partial Corruption

As discussed in Section[3](https://arxiv.org/html/2608.07535#S3), partial corruption can undermine system safety\. Although MLLMs face expanded attack surfaces and modality inconsistencies, they also benefit from overlapping cross\-modal information that can support robustness when some modalities are compromised\. To avoid new points of failure, models should not over\-rely on any subset of modalities\. Future defenses should instead exploit cross\-modal redundancy through mechanisms such as selective modality rejection, confidence\-aware fusion, or adaptive weighting, enabling models to down\-weight unreliable modalities while preserving performance and robustness\.

### 5\.2Toward Unified Safety Framework

Many existing methods apply modality\-specific defenses independently, but such ensemble\-style strategies are neither scalable nor well aligned with multi\-modal principles\. Since not all modalities are compromised simultaneously, uniform defenses are often unnecessary and inefficient, while aggressive filtering can disrupt cross\-modal coherence and degrade alignment and fusion\. Future work should pursue unified, modality\- and threat\-agnostic safety frameworks, particularly by extending defenses to shared embedding and fusion stages and explicitly accounting for modality alignment through alignment\-aware training or cross\-modal consistency regularization\.

### 5\.3Hallucination Beyond Text

Hallucination remains a fundamental risk in MLLMs, and current approaches lack principled and scalable solutions\. Future work should improve data quality and calibration across modalities, strengthen cross\-modal alignment through enhanced representations and consistency regularization, and pursue architectural advances that mitigate hallucination at its source\. Progress also depends on theoretically grounded, human\-aligned, and truly multimodal evaluation benchmarks beyond simplified or model\-based assessments\.

### 5\.4From Manual to Agent\-Driven Safety

Existing AI safety frameworks largely rely on manual efforts to identify vulnerabilities and design safeguards, an approach that is increasingly insufficient for modern multi\-modal systems\. As discussed in Sections[3](https://arxiv.org/html/2608.07535#S3)and[4](https://arxiv.org/html/2608.07535#S4), the threat surface of MLLMs has expanded into a high\-dimensional and dynamic space spanning multiple modalities, cross\-modal interactions, and diverse attack patterns, making systematic manual exploration infeasible\. Moreover, many defenses are evaluated under static threat assumptions or limited attack settings without considering adaptive adversaries\(Nasret al\.,[2025](https://arxiv.org/html/2608.07535#bib.bib274)\)\. To address these challenges, we advocate a shift toward automated, agent\-driven red\-blue teaming, where red\-teaming agents autonomously explore complex attack strategies and discovered vulnerabilities guide blue\-teaming mechanisms for the continuous development of more robust and proactive safety solutions\.

## 6Conclusion

In response to the evolving safety landscape, this survey introduces a new taxonomy of multi\-modal safety risks – compromised modality integration, modality misalignment, and fused safety risks – and highlights shifts in threat assumptions under compositional architectures\. We further summarize emerging safety constraints and recent advances in safety solutions, including safety fine\-tuning, preference\-based optimization, and training\-free approaches, and outline future directions toward unified and robust safety frameworks for next\-generation multi\-modal AI systems\.

## References

- J\. Alayrac, J\. Donahue, P\. Luc, A\. Miech, I\. Barr, Y\. Hasson, K\. Lenc, A\. Mensch, K\. Millicah, M\. Reynolds, R\. Ring, E\. Rutherford, S\. Cabi, T\. Han, Z\. Gong, S\. Samangooei, M\. Monteiro, J\. Menick, S\. Borgeaud, A\. Brock, A\. Nematzadeh, S\. Sharifzadeh, M\. Binkowski, R\. Barreira, O\. Vinyals, A\. Zisserman, and K\. Simonyan \(2022\)Flamingo: a visual language model for few\-shot learning\.InProceedings of the 36th International Conference on Neural Information Processing Systems,NIPS ’22\.External Links:ISBN 9781713871088Cited by:[§1](https://arxiv.org/html/2608.07535#S1.p1.1)\.
- G\. Alon and M\. Kamfonas \(2023\)Detecting language model attacks with perplexity\.CoRRabs/2308\.14132\.Cited by:[§2\.3](https://arxiv.org/html/2608.07535#S2.SS3.p2.1)\.
- W\. An, J\. Nie, F\. Tian, H\. Lin, Y\. Wu, Q\. Wang, X\. Zhang, S\. Lu,et al\.\(2025\)Boosting knowledge utilization in multimodal large language models via adaptive logits fusion and attention reallocation\.InNeurIPS,Cited by:[§4\.4](https://arxiv.org/html/2608.07535#S4.SS4.p5.1),[Table 3](https://arxiv.org/html/2608.07535#S4.T3.1.1.28.28.1)\.
- E\. Bagdasaryan, R\. Jha, V\. Shmatikov, and T\. Zhang \(2024\)Adversarial illusions in multi\-modal embeddings\.In33rd USENIX Security Symposium \(USENIX Security 24\),pp\. 3009–3025\.Cited by:[§3\.2](https://arxiv.org/html/2608.07535#S3.SS2.p3.1),[Table 2](https://arxiv.org/html/2608.07535#S3.T2.1.1.17.17.1)\.
- J\. Bai, K\. Gao, S\. Min, S\. Xia, Z\. Li, and W\. Liu \(2024\)BadCLIP: trigger\-aware prompt learning for backdoor attacks on CLIP\.InCVPR,Cited by:[§3\.2](https://arxiv.org/html/2608.07535#S3.SS2.p4.1),[Table 2](https://arxiv.org/html/2608.07535#S3.T2.1.1.19.19.1)\.
- L\. Bailey, E\. Ong, S\. Russell, and S\. Emmons \(2024\)Image hijacks: adversarial images can control generative models at runtime\.InICML,Cited by:[§3\.1](https://arxiv.org/html/2608.07535#S3.SS1.p2.1),[Table 2](https://arxiv.org/html/2608.07535#S3.T2.1.1.5.5.1)\.
- T\. Baltrušaitis, C\. Ahuja, and L\. Morency \(2018\)Multimodal machine learning: a survey and taxonomy\.IEEE transactions on pattern analysis and machine intelligence41\(2\),pp\. 423–443\.Cited by:[§1](https://arxiv.org/html/2608.07535#S1.p1.1)\.
- E\. M\. Bender, T\. Gebru, A\. McMillan\-Major, and S\. Shmitchell \(2021\)On the dangers of stochastic parrots: can language models be too big?\.InProceedings of the 2021 ACM conference on fairness, accountability, and transparency,pp\. 610–623\.Cited by:[§2\.4](https://arxiv.org/html/2608.07535#S2.SS4.p1.1)\.
- L\. Beurer\-Kellner, M\. Fischer, and M\. Vechev \(2023\)Prompting is programming: a query language for large language models\.Proceedings of the ACM on Programming Languages7\(PLDI\),pp\. 1946–1969\.Cited by:[§2\.4](https://arxiv.org/html/2608.07535#S2.SS4.p2.1)\.
- F\. Bianchi, M\. Suzgun, G\. Attanasio, P\. Röttger, D\. Jurafsky, T\. Hashimoto, and J\. Zou \(2024\)Safety\-tuned llamas: lessons from improving the safety of large language models that follow instructions\.InICLR,Cited by:[§2\.3](https://arxiv.org/html/2608.07535#S2.SS3.p2.1)\.
- T\. B\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. M\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. Amodei \(2020\)Language models are few\-shot learners\.InProceedings of the 34th International Conference on Neural Information Processing Systems,NIPS ’20\.External Links:ISBN 9781713829546Cited by:[§1](https://arxiv.org/html/2608.07535#S1.p1.1)\.
- X\. Cai, D\. Liu, R\. Guan, and P\. Zhou \(2025a\)Imperceptible transfer attack on large vision\-language models\.InICASSP 2025\-2025 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 1–5\.Cited by:[§3\.1](https://arxiv.org/html/2608.07535#S3.SS1.p2.1)\.
- Y\. Cai, R\. Gu, J\. Li, X\. Huang, J\. Chen, X\. Gu, and M\. Huang \(2025b\)MHALO: evaluating mllms as fine\-grained hallucination detectors\.InFindings of ACL,Cited by:[§4\.3](https://arxiv.org/html/2608.07535#S4.SS3.p3.1),[Table 3](https://arxiv.org/html/2608.07535#S4.T3.1.1.19.19.1)\.
- N\. Carlini and D\. A\. Wagner \(2017\)Towards evaluating the robustness of neural networks\.InIEEE Symposium on Security and Privacy,pp\. 39–57\.Cited by:[§2\.1](https://arxiv.org/html/2608.07535#S2.SS1.p1.2)\.
- T\. Chakraborty, E\. Shayegani, Z\. Cai, N\. B\. Abu\-Ghazaleh, M\. S\. Asif, Y\. Dong, A\. K\. Roy\-Chowdhury, and C\. Song \(2024\)Can textual unlearning solve cross\-modality safety alignment?\.InFindings of EMNLP,Cited by:[§4\.2](https://arxiv.org/html/2608.07535#S4.SS2.p2.1),[Table 3](https://arxiv.org/html/2608.07535#S4.T3.1.1.4.4.1)\.
- B\. Chen, X\. Lyu, S\. Yuan, J\. Song, H\. T\. Shen, and L\. Gao \(2025a\)SafePTR: token\-level jailbreak defense in multimodal LLMs via prune\-then\-restore mechanism\.InNeurIPS,Cited by:[§4\.4](https://arxiv.org/html/2608.07535#S4.SS4.p3.1),[Table 3](https://arxiv.org/html/2608.07535#S4.T3.1.1.27.27.1)\.
- C\. Chen, M\. Liu, C\. Jing, Y\. Zhou, F\. Rao, H\. Chen, B\. Zhang, and C\. Shen \(2025b\)PerturboLLaVA: reducing multimodal hallucinations with perturbative visual training\.arXiv preprint arXiv:2503\.06486\.Cited by:[§4\.2](https://arxiv.org/html/2608.07535#S4.SS2.p4.1)\.
- P\. Chen, H\. Zhang, Y\. Sharma, J\. Yi, and C\. Hsieh \(2017a\)ZOO: zeroth order optimization based black\-box attacks to deep neural networks without training substitute models\.InAISec@CCS,Cited by:[§2\.1](https://arxiv.org/html/2608.07535#S2.SS1.p1.2)\.
- X\. Chen, C\. Liu, B\. Li, K\. Lu, and D\. Song \(2017b\)Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning\.arXiv:1712\.05526\.Cited by:[§2\.2](https://arxiv.org/html/2608.07535#S2.SS2.p1.1)\.
- J\. Cohen, E\. Rosenfeld, and Z\. Kolter \(2019\)Certified adversarial robustness via randomized smoothing\.InProceedings of the 36th International Conference on Machine Learning,K\. Chaudhuri and R\. Salakhutdinov \(Eds\.\),Proceedings of Machine Learning Research, Vol\.97,pp\. 1310–1320\.Cited by:[§2\.1](https://arxiv.org/html/2608.07535#S2.SS1.p2.1)\.
- J\. Dai, C\. Chen, and Y\. Li \(2019\)A backdoor attack against lstm\-based text classification systems\.IEEE Access\.Cited by:[§2\.2](https://arxiv.org/html/2608.07535#S2.SS2.p1.1)\.
- W\. Dai, J\. Li, D\. Li, A\. M\. H\. Tiong, J\. Zhao, W\. Wang, B\. Li, P\. Fung, and S\. Hoi \(2023\)InstructBLIP: towards general\-purpose vision\-language models with instruction tuning\.External Links:2305\.06500,[Link](https://arxiv.org/abs/2305.06500)Cited by:[§3\.2](https://arxiv.org/html/2608.07535#S3.SS2.p5.1),[Table 2](https://arxiv.org/html/2608.07535#S3.T2.1.1.27.27.1)\.
- DeepSeek\-AI \(2025\)DeepSeek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.External Links:2501\.12948,[Link](https://arxiv.org/abs/2501.12948)Cited by:[§1](https://arxiv.org/html/2608.07535#S1.p1.1)\.
- B\. Deng, W\. Wang, F\. Feng, Y\. Deng, Q\. Wang, and X\. He \(2023\)Attack prompt generation for red teaming and defending large language models\.InFindings of EMNLP,Cited by:[§2\.3](https://arxiv.org/html/2608.07535#S2.SS3.p2.1)\.
- G\. Deng, Y\. Liu, Y\. Li, K\. Wang, Y\. Zhang, Z\. Li, H\. Wang, T\. Zhang, and Y\. Liu \(2024\)MASTERKEY: automated jailbreaking of large language model chatbots\.InNDSS,Cited by:[§2\.3](https://arxiv.org/html/2608.07535#S2.SS3.p1.1),[§3\.1](https://arxiv.org/html/2608.07535#S3.SS1.p3.1)\.
- I\. Diakonikolas, G\. Kamath, D\. Kane, J\. Li, J\. Steinhardt, and A\. Stewart \(2019\)Sever: A robust meta\-algorithm for stochastic optimization\.InICML,Cited by:[§2\.2](https://arxiv.org/html/2608.07535#S2.SS2.p2.1)\.
- Y\. Ding, B\. Li, and R\. Zhang \(2025\)ETA: evaluating then aligning safety of vision language models at inference time\.InICLR,Cited by:[§4\.4](https://arxiv.org/html/2608.07535#S4.SS4.p3.1),[Table 3](https://arxiv.org/html/2608.07535#S4.T3.1.1.25.25.1)\.
- Y\. Fan, Y\. Cao, Z\. Zhao, Z\. Liu, and S\. Li \(2024\)Unbridled icarus: a survey of the potential perils of image inputs in multimodal large language model security\.In2024 IEEE International Conference on Systems, Man, and Cybernetics \(SMC\),Cited by:[Table 1](https://arxiv.org/html/2608.07535#S1.T1.1.1.4.4.1),[§1](https://arxiv.org/html/2608.07535#S1.p4.1)\.
- S\. Farquhar, J\. Kossen, L\. Kuhn, and Y\. Gal \(2024\)Detecting hallucinations in large language models using semantic entropy\.Nature630\(8017\),pp\. 625–630\.Cited by:[§2\.4](https://arxiv.org/html/2608.07535#S2.SS4.p1.1),[§2\.4](https://arxiv.org/html/2608.07535#S2.SS4.p2.1)\.
- H\. Fei, Y\. Yao, Z\. Zhang, F\. Liu, A\. Zhang, and T\. Chua \(2024\)From multimodal llm to human\-level ai: modality, instruction, reasoning, efficiency and beyond\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\): Tutorial Summaries,pp\. 1–8\.Cited by:[§4\.2](https://arxiv.org/html/2608.07535#S4.SS2.p4.1)\.
- X\. Fu, Z\. Wang, S\. Li, R\. K\. Gupta, N\. Mireshghallah, T\. Berg\-Kirkpatrick, and E\. Fernandes \(2023\)Misusing tools in large language models with visual adversarial examples\.arXiv preprint arXiv:2310\.03185\.Cited by:[§3\.1](https://arxiv.org/html/2608.07535#S3.SS1.p2.1)\.
- K\. Gao, Y\. Bai, J\. Gu, S\. Xia, P\. Torr, Z\. Li, and W\. Liu \(2024\)Inducing high energy\-latency of large vision\-language models with verbose images\.InICLR,Cited by:[Figure 2](https://arxiv.org/html/2608.07535#S3.F2),[§3\.1](https://arxiv.org/html/2608.07535#S3.SS1.p2.1),[Table 2](https://arxiv.org/html/2608.07535#S3.T2.1.1.4.4.1)\.
- L\. Gao, A\. Madaan, S\. Zhou, U\. Alon, P\. Liu, Y\. Yang, J\. Callan, and G\. Neubig \(2023\)Pal: program\-aided language models\.InICML,pp\. 10764–10799\.Cited by:[§2\.4](https://arxiv.org/html/2608.07535#S2.SS4.p2.1)\.
- Y\. Gao, C\. Xu, D\. Wang, S\. Chen, D\. C\. Ranasinghe, and S\. Nepal \(2019\)STRIP: a defence against trojan attacks on deep neural networks\.InACSAC,Cited by:[§2\.2](https://arxiv.org/html/2608.07535#S2.SS2.p2.1)\.
- S\. S\. Ghosal, S\. Chakraborty, V\. Singh, T\. Guan, M\. Wang, A\. Beirami, F\. Huang, A\. Velasquez, D\. Manocha, and A\. S\. Bedi \(2025\)Immune: improving safety against jailbreaks in multi\-modal llms via inference\-time alignment\.InProceedings of the Computer Vision and Pattern Recognition Conference \(CVPR\),pp\. 25038–25049\.Cited by:[§4\.4](https://arxiv.org/html/2608.07535#S4.SS4.p4.1),[Table 3](https://arxiv.org/html/2608.07535#S4.T3.1.1.33.33.1)\.
- Y\. Gong, D\. Ran, J\. Liu, C\. Wang, T\. Cong, A\. Wang, S\. Duan, and X\. Wang \(2025\)FigStep: jailbreaking large vision\-language models via typographic visual prompts\.InAAAI,Cited by:[§3\.1](https://arxiv.org/html/2608.07535#S3.SS1.p3.1),[§3\.3](https://arxiv.org/html/2608.07535#S3.SS3.p2.1),[Table 2](https://arxiv.org/html/2608.07535#S3.T2.1.1.10.10.1)\.
- I\. J\. Goodfellow, J\. Shlens, and C\. Szegedy \(2015\)Explaining and harnessing adversarial examples\.InICLR,Cited by:[§2\.1](https://arxiv.org/html/2608.07535#S2.SS1.p1.2)\.
- Y\. Gou, K\. Chen, Z\. Liu, L\. Hong, H\. Xu, Z\. Li, D\. Yeung, J\. T\. Kwok, and Y\. Zhang \(2024\)Eyes closed, safety on: protecting multimodal llms via image\-to\-text transformation\.InECCV,Cited by:[§4\.4](https://arxiv.org/html/2608.07535#S4.SS4.p4.1),[Table 3](https://arxiv.org/html/2608.07535#S4.T3.1.1.32.32.1)\.
- T\. Gu, K\. Liu, B\. Dolan\-Gavitt, and S\. Garg \(2019\)BadNets: Evaluating Backdooring Attacks on Deep Neural Networks\.IEEE Access\.Cited by:[§2\.2](https://arxiv.org/html/2608.07535#S2.SS2.p1.1)\.
- T\. Guan, F\. Liu, X\. Wu, R\. Xian, Z\. Li, X\. Liu, X\. Wang, L\. Chen, F\. Huang, Y\. Yacoob,et al\.\(2024\)Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision\-language models\.InCVPR,Cited by:[§4\.2](https://arxiv.org/html/2608.07535#S4.SS2.p4.1),[Table 3](https://arxiv.org/html/2608.07535#S4.T3.1.1.7.7.1)\.
- X\. Guo, F\. Yu, H\. Zhang, L\. Qin, and B\. Hu \(2024\)COLD\-attack: jailbreaking llms with stealthiness and controllability\.InICML,Cited by:[§2\.3](https://arxiv.org/html/2608.07535#S2.SS3.p1.1)\.
- Y\. Guo, F\. Jiao, L\. Nie, and M\. Kankanhalli \(2025\)The VLLM safety paradox: dual ease in jailbreak attack and defense\.InNeurIPS,Cited by:[§4\.4](https://arxiv.org/html/2608.07535#S4.SS4.p2.1),[Table 3](https://arxiv.org/html/2608.07535#S4.T3.1.1.22.22.1)\.
- K\. Guu, K\. Lee, Z\. Tung, P\. Pasupat, and M\. Chang \(2020\)Retrieval augmented language model pre\-training\.InICML,pp\. 3929–3938\.Cited by:[§2\.4](https://arxiv.org/html/2608.07535#S2.SS4.p2.1)\.
- D\. Han, X\. Jia, Y\. Bai, J\. Gu, Y\. Liu, and X\. Cao \(2023\)Ot\-attack: enhancing adversarial transferability of vision\-language models via optimal transport optimization\.arXiv preprint arXiv:2312\.04403\.Cited by:[§3\.2](https://arxiv.org/html/2608.07535#S3.SS2.p3.1)\.
- H\. Huang, S\. M\. Erfani, Y\. Li, X\. Ma, and J\. Bailey \(2025a\)X\-transfer attacks: towards super transferable adversarial attacks on clip\.InForty\-second International Conference on Machine Learning,Cited by:[§3\.2](https://arxiv.org/html/2608.07535#S3.SS2.p3.1)\.
- L\. Huang, W\. Yu, W\. Ma, W\. Zhong, Z\. Feng, H\. Wang, Q\. Chen, W\. Peng, X\. Feng, B\. Qin,et al\.\(2025b\)A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions\.ACM Transactions on Information Systems43\(2\),pp\. 1–55\.Cited by:[§2\.4](https://arxiv.org/html/2608.07535#S2.SS4.p1.1)\.
- G\. Izacard and E\. Grave \(2020\)Leveraging passage retrieval with generative models for open domain question answering\.arXiv preprint arXiv:2007\.01282\.Cited by:[§2\.4](https://arxiv.org/html/2608.07535#S2.SS4.p2.1)\.
- N\. Jain, A\. Schwarzschild, Y\. Wen, G\. Somepalli, J\. Kirchenbauer, P\. Chiang, M\. Goldblum, A\. Saha, J\. Geiping, and T\. Goldstein \(2023\)Baseline defenses for adversarial attacks against aligned language models\.CoRRabs/2309\.00614\.Cited by:[§2\.3](https://arxiv.org/html/2608.07535#S2.SS3.p2.1)\.
- Y\. Jiang, X\. Gao, T\. Peng, Y\. Tan, X\. Zhu, B\. Zheng, and X\. Yue \(2025\)HiddenDetect: detecting jailbreak attacks against multimodal large language models via monitoring hidden states\.InACL,Cited by:[§4\.4](https://arxiv.org/html/2608.07535#S4.SS4.p3.1),[Table 3](https://arxiv.org/html/2608.07535#S4.T3.1.1.26.26.1)\.
- H\. Jin, L\. Hu, X\. Li, P\. Zhang, C\. Chen, J\. Zhuang, and H\. Wang \(2024\)JailbreakZoo: survey, landscapes, and horizons in jailbreaking large language and vision\-language models\.arXiv preprint arXiv:2407\.01599\.Cited by:[Table 1](https://arxiv.org/html/2608.07535#S1.T1.1.1.6.6.1),[§1](https://arxiv.org/html/2608.07535#S1.p4.1)\.
- E\. Jones, A\. D\. Dragan, A\. Raghunathan, and J\. Steinhardt \(2023\)Automatically auditing large language models via discrete optimization\.InICML,Vol\.202,pp\. 15307–15329\.Cited by:[§2\.3](https://arxiv.org/html/2608.07535#S2.SS3.p1.1)\.
- S\. Karamcheti, S\. Nair, A\. Balakrishna, P\. Liang, T\. Kollar, and D\. Sadigh \(2024\)Prismatic vlms: investigating the design space of visually\-conditioned language models\.InForty\-first International Conference on Machine Learning,Cited by:[§3\.1](https://arxiv.org/html/2608.07535#S3.SS1.p4.1),[Table 2](https://arxiv.org/html/2608.07535#S3.T2.1.1.12.12.1)\.
- K\. Lee, D\. Ippolito, A\. Nystrom, C\. Zhang, D\. Eck, C\. Callison\-Burch, and N\. Carlini \(2021\)Deduplicating training data makes language models better\.arXiv preprint arXiv:2107\.06499\.Cited by:[§2\.4](https://arxiv.org/html/2608.07535#S2.SS4.p2.1)\.
- S\. Leng, Y\. Xing, Z\. Cheng, Y\. Zhou, H\. Zhang, X\. Li, D\. Zhao, S\. Lu, C\. Miao, and L\. Bing \(2024\)The curse of multi\-modalities: evaluating hallucinations of large multimodal models across language, visual, and audio\.External Links:2410\.12787,[Link](https://arxiv.org/abs/2410.12787)Cited by:[§3\.3](https://arxiv.org/html/2608.07535#S3.SS3.p3.1),[Table 2](https://arxiv.org/html/2608.07535#S3.T2.1.1.32.32.2)\.
- A\. Levine and S\. Feizi \(2021\)Deep partition aggregation: provable defenses against general poisoning attacks\.InICLR,Cited by:[§2\.2](https://arxiv.org/html/2608.07535#S2.SS2.p2.1)\.
- P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel,et al\.\(2020\)Retrieval\-augmented generation for knowledge\-intensive nlp tasks\.Advances in neural information processing systems33,pp\. 9459–9474\.Cited by:[§2\.4](https://arxiv.org/html/2608.07535#S2.SS4.p2.1)\.
- B\. Li, C\. Chen, W\. Wang, and L\. Carin \(2019\)Certified adversarial robustness with additive noise\.InNeurIPS,pp\. 9459–9469\.Cited by:[§2\.1](https://arxiv.org/html/2608.07535#S2.SS1.p2.1)\.
- J\. Li, D\. Li, S\. Savarese, and S\. Hoi \(2023a\)Blip\-2: bootstrapping language\-image pre\-training with frozen image encoders and large language models\.InInternational conference on machine learning,pp\. 19730–19742\.Cited by:[§3\.2](https://arxiv.org/html/2608.07535#S3.SS2.p5.1),[Table 2](https://arxiv.org/html/2608.07535#S3.T2.1.1.26.26.1)\.
- J\. Li, D\. Li, C\. Xiong, and S\. Hoi \(2022a\)BLIP: bootstrapping language\-image pre\-training for unified vision\-language understanding and generation\.InICML,Cited by:[§1](https://arxiv.org/html/2608.07535#S1.p1.1)\.
- L\. Li, Z\. Xie, M\. Li, S\. Chen, P\. Wang, L\. Chen, Y\. Yang, B\. Wang, and L\. Kong \(2023b\)Silkie: preference distillation for large visual language models\.arXiv preprint arXiv:2312\.10665\.Cited by:[§4\.3](https://arxiv.org/html/2608.07535#S4.SS3.p3.1),[Table 3](https://arxiv.org/html/2608.07535#S4.T3.1.1.18.18.2)\.
- L\. Li, D\. Song, X\. Li, J\. Zeng, R\. Ma, and X\. Qiu \(2021a\)Backdoor attacks on pre\-trained models by layerwise weight poisoning\.InEMNLP,Cited by:[§2\.2](https://arxiv.org/html/2608.07535#S2.SS2.p1.1)\.
- X\. Li, Z\. Xiang, D\. J\. Miller, and G\. Kesidis \(2022b\)Test\-time detection of backdoor triggers for poisoned deep neural networks\.InIEEE ICASSP,Cited by:[§2\.2](https://arxiv.org/html/2608.07535#S2.SS2.p2.1)\.
- X\. Li, Z\. Xiang, D\. J\. Miller, and G\. Kesidis \(2025a\)Correcting the distribution of batch normalization signals for trojan mitigation\.Neurocomputing614,pp\. 128752\.External Links:ISSN 0925\-2312,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.neucom.2024.128752),[Link](https://www.sciencedirect.com/science/article/pii/S0925231224015236)Cited by:[§2\.2](https://arxiv.org/html/2608.07535#S2.SS2.p2.1)\.
- X\. Li, Z\. Zhou, J\. Zhu, J\. Yao, T\. Liu, and B\. Han \(2023c\)DeepInception: hypnotize large language model to be jailbreaker\.CoRRabs/2311\.03191\.Cited by:[§2\.3](https://arxiv.org/html/2608.07535#S2.SS3.p1.1)\.
- Y\. Li, Z\. Lin, S\. Zhang, Q\. Fu, B\. Chen, J\. Lou, and W\. Chen \(2023d\)Making language models better reasoners with step\-aware verifier\.InACL,Cited by:[§2\.4](https://arxiv.org/html/2608.07535#S2.SS4.p2.1)\.
- Y\. Li, X\. Lyu, N\. Koren, L\. Lyu, B\. Li, and X\. Ma \(2021b\)Anti\-Backdoor Learning: Training Clean Models on Poisoned Data\.InNeurIPS,Cited by:[§2\.2](https://arxiv.org/html/2608.07535#S2.SS2.p2.1)\.
- Y\. Li, X\. Lyu, N\. Koren, L\. Lyu, B\. Li, and X\. Ma \(2021c\)Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks\.InICLR,Cited by:[§2\.2](https://arxiv.org/html/2608.07535#S2.SS2.p2.1)\.
- Z\. Li, X\. Wu, H\. Du, H\. Nghiem, and G\. Shi \(2025b\)Benchmark evaluations, applications, and challenges of large vision language models: a survey\.arXiv preprint arXiv:2501\.021891\.Cited by:[§4\.2](https://arxiv.org/html/2608.07535#S4.SS2.p4.1)\.
- Z\. Li, X\. Wu, G\. Shi, Y\. Qin, H\. Du, F\. Liu, T\. Zhou, D\. Manocha, and J\. L\. Boyd\-Graber \(2025c\)VideoHallu: evaluating and mitigating multi\-modal hallucinations on synthetic video understanding\.External Links:2505\.01481,[Link](https://arxiv.org/abs/2505.01481)Cited by:[§4\.4](https://arxiv.org/html/2608.07535#S4.SS4.p5.1)\.
- Z\. Li, W\. Yu, C\. Huang, R\. Liu, Z\. Liang, F\. Liu, J\. Che, D\. Yu, J\. Boyd\-Graber, H\. Mi,et al\.\(2025d\)Self\-rewarding vision\-language model via reasoning decomposition\.arXiv preprint arXiv:2508\.19652\.Cited by:[§4\.3](https://arxiv.org/html/2608.07535#S4.SS3.p3.1)\.
- S\. Liang, J\. Liang, T\. Pang, C\. Du, A\. Liu, M\. Zhu, X\. Cao, and D\. Tao \(2025\)Revisiting backdoor attacks against large vision\-language models from domain shift\.InCVPR,pp\. 9477–9486\.Cited by:[§3\.2](https://arxiv.org/html/2608.07535#S3.SS2.p4.1),[Table 2](https://arxiv.org/html/2608.07535#S3.T2.1.1.22.22.1)\.
- C\. Liu, T\. Gui, Y\. Liu, and L\. Xu \(2025a\)AdPO: enhancing the adversarial robustness of large vision\-language models with preference optimization\.CoRRabs/2504\.01735\.Cited by:[§4\.3](https://arxiv.org/html/2608.07535#S4.SS3.p2.1),[Table 3](https://arxiv.org/html/2608.07535#S4.T3.1.1.15.15.1)\.
- D\. Liu, M\. Yang, X\. Qu, P\. Zhou, Y\. Cheng, and W\. Hu \(2025b\)A survey of attacks on large vision–language models: resources, advances, and future trends\.IEEE Transactions on Neural Networks and Learning Systems36\(11\),pp\. 19525–19545\.Cited by:[Table 1](https://arxiv.org/html/2608.07535#S1.T1.1.1.10.10.1),[§1](https://arxiv.org/html/2608.07535#S1.p4.1)\.
- D\. Liu, M\. Yang, X\. Qu, P\. Zhou, X\. Fang, K\. Tang, Y\. Wan, and L\. Sun \(2024a\)Pandora’s box: towards building universal attackers against real\-world large vision\-language models\.InNeurIPS,Cited by:[§3\.2](https://arxiv.org/html/2608.07535#S3.SS2.p2.1),[Table 2](https://arxiv.org/html/2608.07535#S3.T2.1.1.13.13.3)\.
- F\. Liu, K\. Lin, L\. Li, J\. Wang, Y\. Yacoob, and L\. Wang \(2023a\)Mitigating hallucination in large multi\-modal models via robust instruction tuning\.arXiv preprint arXiv:2306\.14565\.Cited by:[§4\.2](https://arxiv.org/html/2608.07535#S4.SS2.p4.1)\.
- F\. Liu, X\. Wang, W\. Yao, J\. Chen, K\. Song, S\. Cho, Y\. Yacoob, and D\. Yu \(2024b\)Mmc: advancing multimodal chart understanding with large\-scale instruction tuning\.InNAACL,Cited by:[§4\.2](https://arxiv.org/html/2608.07535#S4.SS2.p4.1),[Table 3](https://arxiv.org/html/2608.07535#S4.T3.1.1.8.8.1)\.
- F\. Liu \(2025\)Hallucinations in multimodal large language models: evaluation, mitigation, and future directions\.Ph\.D\. Thesis,University of Maryland, College Park\.Cited by:[§4\.4](https://arxiv.org/html/2608.07535#S4.SS4.p5.1)\.
- H\. Liu, C\. Li, Y\. Li, B\. Li, Y\. Zhang, S\. Shen, and Y\. J\. Lee \(2024c\)LLaVA\-next: improved reasoning, ocr, and world knowledge\.External Links:[Link](https://llava-vl.github.io/blog/2024-01-30-llava-next/)Cited by:[§3\.1](https://arxiv.org/html/2608.07535#S3.SS1.p4.1),[Table 2](https://arxiv.org/html/2608.07535#S3.T2.1.1.11.11.2)\.
- H\. Liu, C\. Li, Q\. Wu, and Y\. J\. Lee \(2023b\)Visual instruction tuning\.InProceedings of the 37th International Conference on Neural Information Processing Systems,NIPS ’23\.Cited by:[§1](https://arxiv.org/html/2608.07535#S1.p1.1)\.
- K\. Liu, B\. Dolan\-Gavitt, and S\. Garg \(2018\)Fine\-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks\.InRAID,Cited by:[§2\.2](https://arxiv.org/html/2608.07535#S2.SS2.p2.1)\.
- X\. Liu, N\. Xu, M\. Chen, and C\. Xiao \(2024d\)AutoDAN: generating stealthy jailbreak prompts on aligned large language models\.InICLR,Cited by:[§2\.3](https://arxiv.org/html/2608.07535#S2.SS3.p1.1),[§3\.1](https://arxiv.org/html/2608.07535#S3.SS1.p3.1)\.
- X\. Liu, Y\. Zhu, Y\. Lan, C\. Yang, and Y\. Qiao \(2024e\)Safety of multimodal large language models on images and text\.InIJCAI,Cited by:[Table 1](https://arxiv.org/html/2608.07535#S1.T1.1.1.3.3.1),[§1](https://arxiv.org/html/2608.07535#S1.p4.1)\.
- Y\. Liu, S\. Zhai, M\. Du, Y\. Chen, T\. Cao, H\. Gao, C\. Wang, X\. Li, K\. Wang, J\. Fang, J\. Zhang, and B\. Hooi \(2025c\)GuardReasoner\-VL: safeguarding VLMs via reinforced reasoning\.InNeurIPS,Cited by:[§4\.3](https://arxiv.org/html/2608.07535#S4.SS3.p2.1),[Table 3](https://arxiv.org/html/2608.07535#S4.T3.1.1.16.16.1)\.
- Z\. Liu and H\. Zhang \(2025\)Stealthy backdoor attack in self\-supervised learning vision encoders for large vision language models\.InCVPR,pp\. 25060–25070\.Cited by:[§3\.1](https://arxiv.org/html/2608.07535#S3.SS1.p3.1),[Table 2](https://arxiv.org/html/2608.07535#S3.T2.1.1.7.7.2)\.
- D\. Lu, T\. Pang, C\. Du, Q\. Liu, X\. Yang, and M\. Lin \(2024\)Test\-time backdoor attacks on multimodal large language models\.arXiv: 2402\.08577\.Cited by:[§3\.3](https://arxiv.org/html/2608.07535#S3.SS3.p2.1),[Table 2](https://arxiv.org/html/2608.07535#S3.T2.1.1.30.30.1)\.
- D\. Lu, Z\. Wang, T\. Wang, W\. Guan, H\. Gao, and F\. Zheng \(2023\)Set\-level guidance attack: boosting adversarial transferability of vision\-language pre\-training models\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 102–111\.Cited by:[§3\.2](https://arxiv.org/html/2608.07535#S3.SS2.p3.1)\.
- L\. Lu, S\. Pang, S\. Liang, H\. Zhu, X\. Zeng, A\. Liu, Y\. Liu, and Y\. Zhou \(2025\)Adversarial training for multimodal large language models against jailbreak attacks\.External Links:2503\.04833,[Link](https://arxiv.org/abs/2503.04833)Cited by:[§4\.2](https://arxiv.org/html/2608.07535#S4.SS2.p3.1),[Table 3](https://arxiv.org/html/2608.07535#S4.T3.1.1.13.13.1)\.
- H\. Luo, J\. Gu, F\. Liu, and P\. Torr \(2024\)An image is worth 1000 lies: transferability of adversarial images across prompts on vision\-language models\.InICLR,Cited by:[§3\.1](https://arxiv.org/html/2608.07535#S3.SS1.p2.1),[Table 2](https://arxiv.org/html/2608.07535#S3.T2.1.1.3.3.3)\.
- W\. Lyu, L\. Pang, T\. Ma, H\. Ling, and C\. Chen \(2024\)TrojVLM: backdoor attack against vision language models\.InECCV,Vol\.15123,pp\. 467–483\.Cited by:[§3\.2](https://arxiv.org/html/2608.07535#S3.SS2.p4.1),[Table 2](https://arxiv.org/html/2608.07535#S3.T2.1.1.21.21.1)\.
- W\. Lyu, J\. Yao, S\. Gupta, L\. Pang, T\. Sun, L\. Yi, L\. Hu, H\. Ling, and C\. Chen \(2025\)Backdooring vision\-language models with out\-of\-distribution data\.InICLR,Cited by:[§3\.2](https://arxiv.org/html/2608.07535#S3.SS2.p4.1),[Table 2](https://arxiv.org/html/2608.07535#S3.T2.1.1.23.23.1)\.
- A\. Madry, A\. Makelov, L\. Schmidt, D\. Tsipras, and A\. Vladu \(2018a\)Towards deep learning models resistant to adversarial attacks\.In6th International Conference on Learning Representations, ICLR,Cited by:[§2\.1](https://arxiv.org/html/2608.07535#S2.SS1.p2.1)\.
- A\. Madry, A\. Makelov, L\. Schmidt, D\. Tsipras, and A\. Vladu \(2018b\)Towards deep learning models resistant to adversarial attacks\.InICLR,Cited by:[§2\.1](https://arxiv.org/html/2608.07535#S2.SS1.p1.2)\.
- C\. Mao, S\. Geng, J\. Yang, X\. Wang, and C\. Vondrick \(2023\)Understanding zero\-shot adversarial robustness for large\-scale models\.InICLR,Cited by:[§4\.2](https://arxiv.org/html/2608.07535#S4.SS2.p2.1)\.
- N\. McKenna, T\. Li, L\. Cheng, M\. J\. Hosseini, M\. Johnson, and M\. Steedman \(2023\)Sources of hallucination by large language models on inference tasks\.arXiv preprint arXiv:2305\.14552\.Cited by:[§2\.4](https://arxiv.org/html/2608.07535#S2.SS4.p1.1)\.
- M\. Nasr, N\. Carlini, C\. Sitawarin, S\. V\. Schulhoff, J\. Hayes, M\. Ilie, J\. Pluto, S\. Song, H\. Chaudhari, I\. Shumailov, A\. Thakurta, K\. Y\. Xiao, A\. Terzis, and F\. Tramèr \(2025\)The attacker moves second: stronger adaptive attacks bypass defenses against llm jailbreaks and prompt injections\.External Links:2510\.09023Cited by:[§5\.4](https://arxiv.org/html/2608.07535#S5.SS4.p1.1)\.
- T\. A\. Nguyen and A\. T\. Tran \(2020\)Input\-aware dynamic backdoor attack\.InNeurIPS,Cited by:[§2\.2](https://arxiv.org/html/2608.07535#S2.SS2.p1.1)\.
- T\. A\. Nguyen and A\. T\. Tran \(2021\)WaNet \- Imperceptible Warping\-based Backdoor Attack\.InICLR,Cited by:[§2\.2](https://arxiv.org/html/2608.07535#S2.SS2.p1.1)\.
- S\. Oh, Y\. Jin, M\. Sharma, D\. Kim, E\. Ma, G\. Verma, and S\. Kumar \(2025\)UniGuard: towards universal safety guardrails for jailbreak attacks on multimodal large language models\.External Links:2411\.01703,[Link](https://arxiv.org/abs/2411.01703)Cited by:[§4\.2](https://arxiv.org/html/2608.07535#S4.SS2.p3.1),[Table 3](https://arxiv.org/html/2608.07535#S4.T3.1.1.12.12.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. L\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. F\. Christiano, J\. Leike, and R\. Lowe \(2022\)Training language models to follow instructions with human feedback\.InNeurIPS,Cited by:[§2\.3](https://arxiv.org/html/2608.07535#S2.SS3.p2.1)\.
- A\. Paudice, L\. Muñoz\-González, and E\. C\. Lupu \(2018\)Label sanitization against label flipping poisoning attacks\.InProc\. ECML PKDD Workshops,Cited by:[§2\.2](https://arxiv.org/html/2608.07535#S2.SS2.p2.1)\.
- G\. Penedo, Q\. Malartic, D\. Hesslow, R\. Cojocaru, A\. Cappelli, H\. Alobeidli, B\. Pannier, E\. Almazrouei, and J\. Launay \(2023\)The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only\.arXiv preprint arXiv:2306\.01116\.Cited by:[§2\.4](https://arxiv.org/html/2608.07535#S2.SS4.p2.1)\.
- R\. Pi, T\. Han, J\. Zhang, Y\. Xie, R\. Pan, Q\. Lian, H\. Dong, J\. Zhang, and T\. Zhang \(2024\)MLLM\-protector: ensuring mllm’s safety without hurting performance\.InEMNLP,Cited by:[§4\.4](https://arxiv.org/html/2608.07535#S4.SS4.p4.1),[Table 3](https://arxiv.org/html/2608.07535#S4.T3.1.1.31.31.2)\.
- F\. Qi, Y\. Chen, X\. Zhang, M\. Li, Z\. Liu, and M\. Sun \(2021\)Mind the style of text\! adversarial and backdoor attacks based on text style transfer\.InEMNLP,Cited by:[§2\.2](https://arxiv.org/html/2608.07535#S2.SS2.p1.1)\.
- X\. Qi, K\. Huang, A\. Panda, P\. Henderson, M\. Wang, and P\. Mittal \(2024a\)Visual adversarial examples jailbreak aligned large language models\.InAAAI,Cited by:[§3\.1](https://arxiv.org/html/2608.07535#S3.SS1.p3.1),[§3\.2](https://arxiv.org/html/2608.07535#S3.SS2.p3.1),[Table 2](https://arxiv.org/html/2608.07535#S3.T2.1.1.8.8.2)\.
- X\. Qi, Y\. Zeng, T\. Xie, P\. Chen, R\. Jia, P\. Mittal, and P\. Henderson \(2024b\)Fine\-tuning aligned language models compromises safety, even when users do not intend to\!\.InICLR,Cited by:[§2\.3](https://arxiv.org/html/2608.07535#S2.SS3.p1.1)\.
- C\. Raffel, N\. Shazeer, A\. Roberts, K\. Lee, S\. Narang, M\. Matena, Y\. Zhou, W\. Li, and P\. J\. Liu \(2020\)Exploring the limits of transfer learning with a unified text\-to\-text transformer\.Journal of machine learning research21\(140\),pp\. 1–67\.Cited by:[§2\.4](https://arxiv.org/html/2608.07535#S2.SS4.p2.1)\.
- X\. Rong, W\. Huang, J\. Liang, J\. Bi, X\. Xiao, Y\. Li, B\. Du, and M\. Ye \(2025\)Backdoor cleaning without external guidance in MLLM fine\-tuning\.InNeurIPS,Cited by:[§4\.2](https://arxiv.org/html/2608.07535#S4.SS2.p3.1),[Table 3](https://arxiv.org/html/2608.07535#S4.T3.1.1.11.11.1)\.
- A\. Saha, A\. Subramanya, and H\. Pirsiavash \(2020\)Hidden Trigger Backdoor Attacks\.InAAAI,Cited by:[§2\.2](https://arxiv.org/html/2608.07535#S2.SS2.p1.1)\.
- C\. Schlarmann and M\. Hein \(2023\)On the adversarial robustness of multi\-modal foundation models\.InICCV \- Workshops,Cited by:[§3\.1](https://arxiv.org/html/2608.07535#S3.SS1.p2.1)\.
- C\. Schlarmann, N\. D\. Singh, F\. Croce, and M\. Hein \(2024\)Robust CLIP: unsupervised adversarial fine\-tuning of vision embeddings for robust large vision\-language models\.InICML,Cited by:[§4\.2](https://arxiv.org/html/2608.07535#S4.SS2.p2.1),[Table 3](https://arxiv.org/html/2608.07535#S4.T3.1.1.3.3.3)\.
- E\. Shayegani, Y\. Dong, and N\. B\. Abu\-Ghazaleh \(2024\)Jailbreak in pieces: compositional adversarial attacks on multi\-modal language models\.InICLR,Cited by:[Figure 2](https://arxiv.org/html/2608.07535#S3.F2),[§3\.1](https://arxiv.org/html/2608.07535#S3.SS1.p3.1),[§3\.2](https://arxiv.org/html/2608.07535#S3.SS2.p3.1),[Table 2](https://arxiv.org/html/2608.07535#S3.T2.1.1.9.9.1)\.
- Y\. Shen and S\. Sanghavi \(2019\)Learning with Bad Training Data via Iterative Trimmed Loss Minimization\.InICML,pp\. 5739–5748\.Cited by:[§2\.2](https://arxiv.org/html/2608.07535#S2.SS2.p2.1)\.
- E\. Sheng, K\. Chang, P\. Natarajan, and N\. Peng \(2020\)Towards controllable biases in language generation\.arXiv preprint arXiv:2005\.00268\.Cited by:[§2\.4](https://arxiv.org/html/2608.07535#S2.SS4.p1.1)\.
- M\. Shi, F\. Liu, S\. Wang, S\. Liao, S\. Radhakrishnan, Y\. Zhao, D\. Huang, H\. Yin, K\. Sapra, Y\. Yacoob,et al\.\(2024\)Eagle: exploring the design space for multimodal llms with mixture of encoders\.arXiv preprint arXiv:2408\.15998\.Cited by:[§4\.2](https://arxiv.org/html/2608.07535#S4.SS2.p2.1)\.
- J\. Song, Y\. Wang, J\. Li, X\. Tong, rui yu, Y\. Teng, X\. Ma, and Y\. Wang \(2025\)JailBound: jailbreaking internal safety boundaries of vision\-language models\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,Cited by:[§3\.3](https://arxiv.org/html/2608.07535#S3.SS3.p2.1)\.
- W\. Sun, Z\. Shi, S\. Gao, P\. Ren, M\. de Rijke, and Z\. Ren \(2023a\)Contrastive learning reduces hallucination in conversations\.InAAAI,Cited by:[§2\.4](https://arxiv.org/html/2608.07535#S2.SS4.p2.1)\.
- Z\. Sun, Y\. Shen, Q\. Zhou, H\. Zhang, Z\. Chen, D\. D\. Cox, Y\. Yang, and C\. Gan \(2023b\)Principle\-driven self\-alignment of language models from scratch with minimal human supervision\.InNeurIPS,Cited by:[§2\.3](https://arxiv.org/html/2608.07535#S2.SS3.p2.1)\.
- X\. Tao, S\. Zhong, L\. Li, Q\. Liu, and L\. Kong \(2025\)ImgTrojan: jailbreaking vision\-language models with ONE image\.InNAACL,Cited by:[§3\.3](https://arxiv.org/html/2608.07535#S3.SS3.p2.1),[Table 2](https://arxiv.org/html/2608.07535#S3.T2.1.1.31.31.1)\.
- M\. L\. team \(2024\)The llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[§1](https://arxiv.org/html/2608.07535#S1.p1.1)\.
- B\. Tran, J\. Li, and A\. Madry \(2018\)Spectral Signatures in Backdoor Attacks\.InNeurIPS,Cited by:[§2\.2](https://arxiv.org/html/2608.07535#S2.SS2.p2.1)\.
- H\. Tu, C\. Cui, Z\. Wang, Y\. Zhou, B\. Zhao, J\. Han, W\. Zhou, H\. Yao, and C\. Xie \(2024\)How many are in this image a safety evaluation benchmark for vision llms\.InEuropean Conference on Computer Vision,pp\. 37–55\.Cited by:[§3\.2](https://arxiv.org/html/2608.07535#S3.SS2.p3.1),[Table 2](https://arxiv.org/html/2608.07535#S3.T2.1.1.20.20.1)\.
- L\. Tu, G\. Lalwani, S\. Gella, and H\. He \(2020\)An empirical study on robustness to spurious correlations using pre\-trained language models\.Transactions of the Association for Computational Linguistics8,pp\. 621–633\.Cited by:[§2\.4](https://arxiv.org/html/2608.07535#S2.SS4.p1.1)\.
- E\. Wallace, S\. Feng, N\. Kandpal, M\. Gardner, and S\. Singh \(2019\)Universal adversarial triggers for attacking and analyzing nlp\.arXiv preprint arXiv:1908\.07125\.Cited by:[§2\.4](https://arxiv.org/html/2608.07535#S2.SS4.p1.1)\.
- M\. Walmer, K\. Sikka, I\. Sur, A\. Shrivastava, and S\. Jha \(2022\)Dual\-key multimodal backdoors for visual question answering\.InCVPR,Cited by:[Figure 2](https://arxiv.org/html/2608.07535#S3.F2),[§3\.3](https://arxiv.org/html/2608.07535#S3.SS3.p2.1),[Table 2](https://arxiv.org/html/2608.07535#S3.T2.1.1.29.29.3)\.
- B\. Wang, Y\. Yao, S\. Shan, H\. Li, B\. Viswanath, H\. Zheng, and B\. Y\. Zhao \(2019\)Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks\.In2019 IEEE Symposium on Security and Privacy,Cited by:[§2\.2](https://arxiv.org/html/2608.07535#S2.SS2.p2.1)\.
- H\. Wang, G\. Wang, and H\. Zhang \(2025a\)Steering away from harm: an adaptive approach to defending vision language model against jailbreaks\.InCVPR,Cited by:[§4\.4](https://arxiv.org/html/2608.07535#S4.SS4.p3.1),[Table 3](https://arxiv.org/html/2608.07535#S4.T3.1.1.24.24.1)\.
- H\. Wang, K\. Dong, Z\. Zhu, H\. Qin, A\. Liu, X\. Fang, J\. Wang, and X\. Liu \(2024a\)Transferable multimodal attack on vision\-language pre\-training models\.InIEEE S & P,Cited by:[§3\.1](https://arxiv.org/html/2608.07535#S3.SS1.p2.1),[§3\.2](https://arxiv.org/html/2608.07535#S3.SS2.p2.1),[Table 2](https://arxiv.org/html/2608.07535#S3.T2.1.1.15.15.1)\.
- J\. Wang, Y\. Zuo, Y\. Chai, Z\. Liu, Y\. Fu, Y\. Feng, and K\. Lam \(2025b\)Safeguarding vision\-language models: mitigating vulnerabilities to gaussian noise in perturbation\-based attacks\.InICCV,Cited by:[§4\.2](https://arxiv.org/html/2608.07535#S4.SS2.p3.1),[§4\.4](https://arxiv.org/html/2608.07535#S4.SS4.p2.1),[Table 3](https://arxiv.org/html/2608.07535#S4.T3.1.1.9.9.1)\.
- J\. Wang, Z\. Liu, K\. H\. Park, M\. Chen, and C\. Xiao \(2023a\)Adversarial demonstration attacks on large language models\.CoRRabs/2305\.14950\.Cited by:[§2\.3](https://arxiv.org/html/2608.07535#S2.SS3.p1.1)\.
- P\. Wang, D\. Zhang, L\. Li, C\. Tan, X\. Wang, M\. Zhang, K\. Ren, B\. Jiang, and X\. Qiu \(2024b\)InferAligner: inference\-time alignment for harmlessness through cross\-model guidance\.InEMNLP,Cited by:[§4\.4](https://arxiv.org/html/2608.07535#S4.SS4.p3.1),[Table 3](https://arxiv.org/html/2608.07535#S4.T3.1.1.23.23.2)\.
- S\. Wang, J\. Zhang, Z\. Yuan, and S\. Shan \(2024c\)Pre\-trained model guided fine\-tuning for zero\-shot adversarial robustness\.InCVPR,Cited by:[§4\.2](https://arxiv.org/html/2608.07535#S4.SS2.p2.1)\.
- S\. Wang, Z\. Long, Z\. Fan, and Z\. Wei \(2024d\)From LLMs to MLLMs: exploring the landscape of multimodal jailbreaking\.InEMNLP,Cited by:[Table 1](https://arxiv.org/html/2608.07535#S1.T1.1.1.7.7.1),[§1](https://arxiv.org/html/2608.07535#S1.p4.1)\.
- T\. Wang, Z\. Fang, H\. Xue, C\. Zhang, M\. Jin, W\. Xu, D\. Shu, S\. Yang, Z\. Wang, and D\. Liu \(2024e\)Large vision\-language model security: a survey\.InFrontiers in Cyber Security,B\. Chen, X\. Fu, and M\. Huang \(Eds\.\),Cited by:[Table 1](https://arxiv.org/html/2608.07535#S1.T1.1.1.5.5.1),[§1](https://arxiv.org/html/2608.07535#S1.p4.1)\.
- T\. Wang, R\. Sridhar, D\. Yang, and X\. Wang \(2021\)Identifying and mitigating spurious correlations for improving robustness in nlp models\.arXiv preprint arXiv:2110\.07736\.Cited by:[§2\.4](https://arxiv.org/html/2608.07535#S2.SS4.p1.1)\.
- W\. Wang, Z\. Ma, H\. Zheng, X\. Tan, and M\. Chen \(2025c\)Self\-aware safety augmentation: leveraging internal semantic understanding to enhance safety in vision\-language models\.InACM International Conference on Multimedia,Cited by:[§4\.4](https://arxiv.org/html/2608.07535#S4.SS4.p3.1),[Table 3](https://arxiv.org/html/2608.07535#S4.T3.1.1.29.29.1)\.
- W\. Wang, A\. Levine, and S\. Feizi \(2022\)Improved certified defenses against data poisoning with \(deterministic\) finite aggregation\.InICML,Cited by:[§2\.2](https://arxiv.org/html/2608.07535#S2.SS2.p2.1)\.
- X\. Wang, G\. Chen, G\. Qian, P\. Gao, X\. Wei, Y\. Wang, Y\. Tian, and W\. Gao \(2023b\)Large\-scale multi\-modal pre\-trained models: a comprehensive survey\.Machine Intelligence Research20\(4\),pp\. 447–482\.Cited by:[§1](https://arxiv.org/html/2608.07535#S1.p1.1)\.
- Y\. Wang, X\. Liu, Y\. Li, M\. Chen, and C\. Xiao \(2024f\)AdaShield : safeguarding multimodal large language models from structure\-based attack via adaptive shield prompting\.InECCV,Cited by:[§4\.4](https://arxiv.org/html/2608.07535#S4.SS4.p2.1),[Table 3](https://arxiv.org/html/2608.07535#S4.T3.1.1.20.20.3)\.
- Y\. Wang, C\. Liu, Y\. Qu, H\. Cao, D\. Jiang, and L\. Xu \(2024g\)Break the visual perception: adversarial attacks targeting encoded visual tokens of large vision\-language models\.InACM MM,Cited by:[§3\.1](https://arxiv.org/html/2608.07535#S3.SS1.p2.1),[§3\.2](https://arxiv.org/html/2608.07535#S3.SS2.p2.1),[Table 2](https://arxiv.org/html/2608.07535#S3.T2.1.1.14.14.1)\.
- Z\. Wang, K\. Mei, J\. Zhai, and S\. Ma \(2023c\)UNICORN: A unified backdoor trigger inversion framework\.InICLR,Cited by:[§2\.2](https://arxiv.org/html/2608.07535#S2.SS2.p2.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou,et al\.\(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in neural information processing systems35,pp\. 24824–24837\.Cited by:[§2\.4](https://arxiv.org/html/2608.07535#S2.SS4.p2.1)\.
- Z\. Wei, Y\. Wang, and Y\. Wang \(2023\)Jailbreak and guard aligned language models with only few in\-context demonstrations\.CoRRabs/2310\.06387\.Cited by:[§2\.3](https://arxiv.org/html/2608.07535#S2.SS3.p1.1)\.
- F\. Weng, J\. Lou, J\. Feng, M\. Huang, and W\. Wang \(2025\)Adversary\-aware DPO: enhancing safety alignment in vision language models via adversarial training\.InFindings of EMNLP,Cited by:[§4\.3](https://arxiv.org/html/2608.07535#S4.SS3.p2.1),[Table 3](https://arxiv.org/html/2608.07535#S4.T3.1.1.17.17.1)\.
- H\. Xiao, H\. Xiao, and C\. Eckert \(2012\)Adversarial label flips attack on support vector machines\.InECAI,Vol\.242,pp\. 870–875\.Cited by:[§2\.2](https://arxiv.org/html/2608.07535#S2.SS2.p1.1)\.
- P\. Xu, X\. Zhu, and D\. A\. Clifton \(2023\)Multimodal learning with transformers: a survey\.IEEE Transactions on Pattern Analysis and Machine Intelligence45\(10\),pp\. 12113–12132\.Cited by:[§1](https://arxiv.org/html/2608.07535#S1.p1.1)\.
- S\. Xu, L\. Pang, Y\. Zhu, H\. Shen, and X\. Cheng \(2025\)Cross\-modal safety mechanism transfer in large vision\-language models\.InICLR,Cited by:[§4\.2](https://arxiv.org/html/2608.07535#S4.SS2.p3.1),[Table 3](https://arxiv.org/html/2608.07535#S4.T3.1.1.10.10.1)\.
- Y\. Xu, J\. Yao, M\. Shu, Y\. Sun, Z\. Wu, N\. Yu, T\. Goldstein, and F\. Huang \(2024a\)Shadowcast: stealthy data poisoning attacks against vision\-language models\.InNeurIPS,Cited by:[§3\.2](https://arxiv.org/html/2608.07535#S3.SS2.p4.1),[Table 2](https://arxiv.org/html/2608.07535#S3.T2.1.1.18.18.1)\.
- Y\. Xu, X\. Qi, Z\. Qin, and W\. Wang \(2024b\)Cross\-modality information check for detecting jailbreaking in multimodal large language models\.InFindings of EMNLP,Cited by:[§4\.4](https://arxiv.org/html/2608.07535#S4.SS4.p3.1),[Table 3](https://arxiv.org/html/2608.07535#S4.T3.1.1.30.30.1)\.
- X\. Yang, X\. Wang, Q\. Zhang, L\. R\. Petzold, W\. Y\. Wang, X\. Zhao, and D\. Lin \(2023\)Shadow alignment: the ease of subverting safely\-aligned language models\.CoRRabs/2310\.02949\.Cited by:[§2\.3](https://arxiv.org/html/2608.07535#S2.SS3.p1.1)\.
- M\. Ye, X\. Rong, W\. Huang, B\. Du, N\. Yu, and D\. Tao \(2025\)A survey of safety on large vision\-language models: attacks, defenses and evaluations\.External Links:2502\.14881,[Link](https://arxiv.org/abs/2502.14881)Cited by:[Table 1](https://arxiv.org/html/2608.07535#S1.T1.1.1.8.8.1.1),[§1](https://arxiv.org/html/2608.07535#S1.p4.1)\.
- Q\. Ye, H\. Xu, J\. Ye, M\. Yan, A\. Hu, H\. Liu, Q\. Qian, J\. Zhang, and F\. Huang \(2024\)Mplug\-owl2: revolutionizing multi\-modal large language model with modality collaboration\.InProceedings of the ieee/cvf conference on computer vision and pattern recognition,pp\. 13040–13051\.Cited by:[§3\.2](https://arxiv.org/html/2608.07535#S3.SS2.p5.1),[Table 2](https://arxiv.org/html/2608.07535#S3.T2.1.1.28.28.1)\.
- S\. Yin, C\. Fu, S\. Zhao, K\. Li, X\. Sun, T\. Xu, and E\. Chen \(2024a\)A survey on multimodal large language models\.National Science Review11\(12\)\.Cited by:[§1](https://arxiv.org/html/2608.07535#S1.p1.1)\.
- S\. Yin, C\. Fu, S\. Zhao, T\. Xu, H\. Wang, D\. Sui, Y\. Shen, K\. Li, X\. Sun, and E\. Chen \(2024b\)Woodpecker: hallucination correction for multimodal large language models\.Science China Information Sciences67\(12\),pp\. 220105\.Cited by:[§4\.4](https://arxiv.org/html/2608.07535#S4.SS4.p5.1)\.
- Z\. Yin, M\. Ye, T\. Zhang, T\. Du, J\. Zhu, H\. Liu, J\. Chen, T\. Wang, and F\. Ma \(2023\)VLATTACK: multimodal adversarial attacks on vision\-language tasks via pre\-trained models\.InNeurIPS,Cited by:[§3\.1](https://arxiv.org/html/2608.07535#S3.SS1.p2.1)\.
- L\. Yu, H\. Zhang, and C\. Xu \(2024a\)Text\-guided attention is all you need for zero\-shot robustness in vision\-language models\.InNeurIPS,Cited by:[§4\.2](https://arxiv.org/html/2608.07535#S4.SS2.p2.1),[Table 3](https://arxiv.org/html/2608.07535#S4.T3.1.1.5.5.2)\.
- Z\. Yu, X\. Liu, S\. Liang, Z\. Cameron, C\. Xiao, and N\. Zhang \(2024b\)Don’t listen to me: understanding and exploring jailbreak prompts of large language models\.InUSENIX Security,Cited by:[§3\.1](https://arxiv.org/html/2608.07535#S3.SS1.p3.1)\.
- Y\. Yuan, W\. Jiao, W\. Wang, J\. Huang, P\. He, S\. Shi, and Z\. Tu \(2024\)GPT\-4 is too smart to be safe: stealthy chat with llms via cipher\.InICLR,Cited by:[§2\.3](https://arxiv.org/html/2608.07535#S2.SS3.p1.1)\.
- Y\. Zeng, S\. Chen, W\. Park, Z\. Mao, M\. Jin, and R\. Jia \(2022\)Adversarial Unlearning of Backdoors via Implicit Hypergradient\.InICLR,Cited by:[§2\.2](https://arxiv.org/html/2608.07535#S2.SS2.p2.1)\.
- Q\. Zhan, R\. Fang, R\. Bindu, A\. Gupta, T\. Hashimoto, and D\. Kang \(2024\)Removing RLHF protections in GPT\-4 via fine\-tuning\.InNAACL,pp\. 681–687\.Cited by:[§2\.3](https://arxiv.org/html/2608.07535#S2.SS3.p1.1)\.
- C\. Zhang, Z\. Wan, Z\. Kan, M\. Q\. Ma, S\. Stepputtis, D\. Ramanan, R\. Salakhutdinov, L\. Morency, K\. P\. Sycara, and Y\. Xie \(2025a\)Self\-correcting decoding with generative feedback for mitigating hallucinations in large vision\-language models\.InICLR,Cited by:[§4\.4](https://arxiv.org/html/2608.07535#S4.SS4.p5.1),[Table 3](https://arxiv.org/html/2608.07535#S4.T3.1.1.34.34.1)\.
- H\. Zhang, W\. Shao, H\. Liu, Y\. Ma, P\. Luo, Y\. Qiao, N\. Zheng, and K\. Zhang \(2024a\)B\-avibench: towards evaluating the robustness of large vision\-language model on black\-box adversarial visual\-instructions\.IEEE Transactions on Information Forensics and Security\.Cited by:[§3\.1](https://arxiv.org/html/2608.07535#S3.SS1.p2.1)\.
- H\. Zhang, N\. Cheng, Y\. Zhang, and Z\. Li \(2021\)Label flipping attacks against naive bayes on spam filtering systems\.Appl\. Intell\.51\(7\),pp\. 4503–4514\.Cited by:[§2\.2](https://arxiv.org/html/2608.07535#S2.SS2.p1.1)\.
- J\. Zhang, J\. Ye, X\. Ma, Y\. Li, Y\. Yang, Y\. Chen, J\. Sang, and D\. Yeung \(2025b\)AnyAttack: towards large\-scale self\-supervised adversarial attacks on vision\-language models\.InCVPR,pp\. 19900–19909\.Cited by:[§3\.1](https://arxiv.org/html/2608.07535#S3.SS1.p2.1),[Table 2](https://arxiv.org/html/2608.07535#S3.T2.1.1.6.6.1)\.
- X\. Zhang, Y\. Quan, C\. Gu, C\. Shen, X\. Yuan, S\. Yan, H\. Cheng, K\. Wu, and J\. Ye \(2024b\)Seeing clearly by layer two: enhancing attention heads to alleviate hallucination in lvlms\.arXiv preprint arXiv:2411\.09968\.Cited by:[§4\.4](https://arxiv.org/html/2608.07535#S4.SS4.p5.1)\.
- Y\. Zhang, L\. Chen, G\. Zheng, Y\. Gao, R\. Zheng, J\. Fu, Z\. Yin, S\. Jin, Y\. Qiao, X\. Huang, F\. Zhao, T\. Gui, and J\. Shao \(2025c\)SPA\-VL: A comprehensive safety preference alignment dataset for vision language models\.InCVPR,Cited by:[§4\.3](https://arxiv.org/html/2608.07535#S4.SS3.p2.1),[Table 3](https://arxiv.org/html/2608.07535#S4.T3.1.1.14.14.3)\.
- L\. Zhao, Y\. Deng, W\. Zhang, and Q\. Gu \(2024a\)Mitigating object hallucination in large vision\-language models via classifier\-free guidance\.arXiv e\-prints,pp\. arXiv–2402\.Cited by:[§4\.4](https://arxiv.org/html/2608.07535#S4.SS4.p5.1)\.
- X\. Zhao, X\. Yang, T\. Pang, C\. Du, L\. Li, Y\. Wang, and W\. Y\. Wang \(2024b\)Weak\-to\-strong jailbreaking on large language models\.CoRRabs/2401\.17256\.Cited by:[§2\.3](https://arxiv.org/html/2608.07535#S2.SS3.p1.1)\.
- Y\. Zhao, X\. Zheng, L\. Luo, Y\. Li, X\. Ma, and Y\. Jiang \(2025\)BlueSuffix: reinforced blue teaming for vision\-language models against jailbreak attacks\.InICLR,Cited by:[§4\.4](https://arxiv.org/html/2608.07535#S4.SS4.p2.1),[Table 3](https://arxiv.org/html/2608.07535#S4.T3.1.1.21.21.1)\.
- Y\. Zhao, T\. Pang, C\. Du, X\. Yang, C\. Li, N\. Cheung, and M\. Lin \(2023a\)On evaluating adversarial robustness of large vision\-language models\.InNeurIPS,Cited by:[§3\.2](https://arxiv.org/html/2608.07535#S3.SS2.p3.1),[Table 2](https://arxiv.org/html/2608.07535#S3.T2.1.1.16.16.2)\.
- Z\. Zhao, B\. Wang, L\. Ouyang, X\. Dong, J\. Wang, and C\. He \(2023b\)Beyond hallucinations: enhancing lvlms through hallucination\-aware direct preference optimization\.arXiv preprint arXiv:2311\.16839\.Cited by:[§4\.3](https://arxiv.org/html/2608.07535#S4.SS3.p3.1)\.
- Z\. Zhong, Z\. Sun, Y\. Liu, X\. He, and G\. Tao \(2025\)Backdoor attack on vision language models with stealthy semantic manipulation\.CoRRabs/2506\.07214\.Cited by:[§3\.2](https://arxiv.org/html/2608.07535#S3.SS2.p4.1),[Table 2](https://arxiv.org/html/2608.07535#S3.T2.1.1.24.24.1)\.
- D\. Zhou, N\. Schärli, L\. Hou, J\. Wei, N\. Scales, X\. Wang, D\. Schuurmans, C\. Cui, O\. Bousquet, Q\. Le,et al\.\(2022\)Least\-to\-most prompting enables complex reasoning in large language models\.arXiv preprint arXiv:2205\.10625\.Cited by:[§2\.4](https://arxiv.org/html/2608.07535#S2.SS4.p2.1)\.
- D\. Zhu, J\. Chen, X\. Shen, X\. Li, and M\. Elhoseiny \(2023a\)Minigpt\-4: enhancing vision\-language understanding with advanced large language models\.arXiv preprint arXiv:2304\.10592\.Cited by:[§3\.2](https://arxiv.org/html/2608.07535#S3.SS2.p5.1),[Table 2](https://arxiv.org/html/2608.07535#S3.T2.1.1.25.25.2)\.
- L\. Zhu, Z\. Zhu, C\. Zhang, Y\. Xu, and X\. Kong \(2023b\)Multimodal sentiment analysis based on fusion methods: a survey\.Information Fusion95,pp\. 306–325\.Cited by:[§1](https://arxiv.org/html/2608.07535#S1.p1.1)\.
- S\. Zhu, R\. Zhang, B\. An, G\. Wu, J\. Barrow, Z\. Wang, F\. Huang, A\. Nenkova, and T\. Sun \(2023c\)AutoDAN: interpretable gradient\-based adversarial attacks on large language models\.CoRRabs/2310\.15140\.Cited by:[§2\.3](https://arxiv.org/html/2608.07535#S2.SS3.p1.1)\.
- D\. M\. Ziegler, N\. Stiennon, J\. Wu, T\. B\. Brown, A\. Radford, D\. Amodei, P\. Christiano, and G\. Irving \(2019\)Fine\-tuning language models from human preferences\.arXiv preprint arXiv:1909\.08593\.Cited by:[§2\.4](https://arxiv.org/html/2608.07535#S2.SS4.p2.1)\.
- Y\. Zong, O\. Bohdal, T\. Yu, Y\. Yang, and T\. Hospedales \(2024\)Safety fine\-tuning at \(almost\) no cost: a baseline for vision large language models\.InICML,Cited by:[§4\.2](https://arxiv.org/html/2608.07535#S4.SS2.p3.1),[Table 3](https://arxiv.org/html/2608.07535#S4.T3.1.1.6.6.1)\.
- A\. Zou, Z\. Wang, J\. Z\. Kolter, and M\. Fredrikson \(2023\)Universal and transferable adversarial attacks on aligned language models\.CoRRabs/2307\.15043\.Cited by:[§2\.3](https://arxiv.org/html/2608.07535#S2.SS3.p1.1)\.