Hybrid Adversarial Defence for Natural Language Understanding Tasks

# Hybrid Adversarial Defence for Natural Language Understanding Tasks
Source: [https://arxiv.org/html/2606.04612](https://arxiv.org/html/2606.04612)
Manar Abouzaid1,Yang Wang2,Chenghua Lin2, Stuart E\. Middleton1 1School of Electronics and Computer Science, University of Southampton, UK 2Department of Computer Science, University of Manchester, UK

###### Abstract

Large Language Models \(LLMs\) are vulnerable both to hallucination and adversarial manipulation\. Although these problems are closely related, existing defences typically address them separately\. We investigate a hybrid defence framework that combines entropy\-based models, designed to reduce hallucinations, with uncertainty\-based models and geometric\-based models, designed to reduce vulnerability\. Under in\-domain tests on Natural Language Understanding datasets \(FEVER, HotpotQA, CSQA, SIQA\) we find our hybrid model improves both clean\-task performance \(up to 43\.34% increase in accuracy\) and adversarial robustness \(up to 64\.92% improvement in accuracy and 62\.27% reduction in attack success rate\)\. For out\-of\-distribution datasets \(AeroEngQA, CPIQA\) we see similar adversarial robustness from our hybrid model \(up to 57\.14% improvement in accuracy\)\. For prompt injection \(SafeGuard\) and jailbreak detection \(AdvBench, DAN\) datasets our hybrid model is also very strong \(up to 51% reduction in attack success rate compared to state of the art baseline models\)\. Overall, our results show that combining entropy, uncertainty and geometric features provides a more effective defence strategy than using any single feature alone for both in\-domain and out\-of\-distribution tasks\.

Hybrid Adversarial Defence for Natural Language Understanding Tasks

## 1Introduction

Large Language Models \(LLMs\) have become an important component of modern artificial intelligence systems\(Brownet al\.,[2020](https://arxiv.org/html/2606.04612#bib.bib7); Touvronet al\.,[2023](https://arxiv.org/html/2606.04612#bib.bib42)\)\. Despite their impressive capabilities, two well\-known issues continue to affect large language models: hallucination and adversarial vulnerability\.

In this paper, we explore whether methods designed to address hallucination and adversarial attacks can be combined within a single framework to improve the robustness of LLMs for Natural Language Understanding \(NLU\) tasks\. Our motivation is based on the observation that there are now several very different classes of hallucination reduction and adversarial defence model being explored in the literature, and these models very likely each have a distinct error distribution\. If we were able to learn when to use each class of model we could develop a powerful hybrid adversarial defence approach\. The first class of model we use is an entropy\-based model motivated byYaoet al\.\([2023](https://arxiv.org/html/2606.04612#bib.bib48)\)to reduce hallucinations\. The second is an uncertainty trained model able to say ‘I don’t know’ motivated byZhanget al\.\([2024](https://arxiv.org/html/2606.04612#bib.bib51)\)\. The third is a geometric\-based model motivated byWanget al\.\([2025](https://arxiv.org/html/2606.04612#bib.bib45)\)which improves hidden states by removing dominant principal components\. To aggregate these defences, we introduce a selector network that learns to route inputs to our three expert models\. The routing decisions are based on features capturing entropy features, uncertainty estimates, and the geometric properties of the model’s internal representations\. We explore both hard expert selection and soft probabilistic aggregation of defence features\. We release our models111Paper github URL once paper acceptedas open source for reproducibility purposes\.

We investigate the following research questions:

- •RQ1 Can token\-level in\-domain adversarial defence methods \(entropy\-based, uncertainty\-based, geometric\-based\) improve the base performance of NLU tasks in addition to robustness?
- •RQ2 Can hybrid adversarial defence methods outperform single adversarial defence methods for both in\-domain and out\-of\-distribution tasks?
- •RQ3 Do token\-level defence mechanisms generalize to sequence\-based attacks such as prompt injection and jailbreak attempts?

This work makes the following contributions:

- •We describe the first hybrid adversarial defence approach to use entropy, uncertainty and geometric\-based features to select adversarial defence models\. Our hybrid adversarial defence consistently out\-performs state of the art models on both task performance and robustness across a variety of in\-domain NLU datasets \(FEVER, HotpotQA, CSQA, SIQA\)\.
- •We provide a comprehensive analysis of how our hybrid adversarial defence performs in out\-of\-distribution and sequence\-based attack problems\. Our hybrid adversarial defence out\-performs any single model on all out\-of\-distribution \(AeroEngQA, CPIQA\), prompt injection \(SafeGuard\) datasets and jailbreak \(AdvBench, DAN\) datasets, except for DAN where the NeMoGuard model that was explicitly optimised for this dataset was best\.

## 2Related Work

### 2\.1Hallucination Reduction

Large language models \(LLMs\) hallucinations\(Zhanget al\.,[2025b](https://arxiv.org/html/2606.04612#bib.bib52); Jiet al\.,[2023](https://arxiv.org/html/2606.04612#bib.bib21)\)are responses that are confident and well written, but which contain errors such as being factually incorrect or nonsensical\. Recent large\-scale studies have confirmed the continued prevalence of hallucinations\(Massenonet al\.,[2025](https://arxiv.org/html/2606.04612#bib.bib29)\)\. Research into LLM hallucination reduction has mostly focused on the areas of improving faithfulness of responses to the original input, factual accuracy of responses and consistency of responses to reduce contradictions\.

Recent work on improving LLM faithfulness includes entropy\-based approaches to detect statistical anomalies in LLM responses, such as LLMLies\(Yaoet al\.,[2023](https://arxiv.org/html/2606.04612#bib.bib48)\), and uncertainty\-aware training to allow models to refrain from answering questions beyond a knowledge threshold, such as R\-Tuning\(Zhanget al\.,[2024](https://arxiv.org/html/2606.04612#bib.bib51)\)\. Retrieval\-Augmentation Generation \(RAG\)\(Lewiset al\.,[2020](https://arxiv.org/html/2606.04612#bib.bib25)\)has long been used to improve LLM factual accuracy by retrieving context from trusted sources\. Other methods to improve factual accuracy include use of semantic entropy metrics\(Zubkovaet al\.,[2025](https://arxiv.org/html/2606.04612#bib.bib54)\)to decide when to use pre\-trained LLM knowledge and when to augment it with RAG, perplexity\-based metrics\(Varshneyet al\.,[2023](https://arxiv.org/html/2606.04612#bib.bib43)\)to avoid unnatural out of distribution responses and knowledge injectionElarabyet al\.\([2023](https://arxiv.org/html/2606.04612#bib.bib11)\)using expert\-verified knowledge sets to fill specific factual gaps that would otherwise lead to LLM hallucinations\. Recent work to improve LLM consistency includes generate\-then\-refine strategies\(Dziriet al\.,[2021](https://arxiv.org/html/2606.04612#bib.bib9)\)using a knowledge\-graph, entailment models\(Maynezet al\.,[2020](https://arxiv.org/html/2606.04612#bib.bib30)\)to assess quality of abstractive summarization and self\-consistency approaches\(Manakulet al\.,[2023](https://arxiv.org/html/2606.04612#bib.bib28)\)where responses for similar concepts are expected to be similar and contain consistent facts\.

Our work uses both entropy\-based and uncertainty\-based approaches to reduce LLM hallucinations, but we do this in the context of aggregating them with geometric\-based adversarial defence which is something that has not been explored before\.

### 2\.2Adversarial Defence

Adversarial defence attempts to make LLMs more robust to adversarial attacks such as malicious input perturbation, prompt injection and LLM jailbreaking\. There are several adversarial attack frameworks available to test LLMs, such as TextAttackMorriset al\.\([2020](https://arxiv.org/html/2606.04612#bib.bib31)\), TextFoolerJinet al\.\([2020](https://arxiv.org/html/2606.04612#bib.bib22)\), TextBuggerLiet al\.\([2018](https://arxiv.org/html/2606.04612#bib.bib26)\)and PWWSRenet al\.\([2019](https://arxiv.org/html/2606.04612#bib.bib35)\)\. Jailbreak attacks\(Weiet al\.,[2023](https://arxiv.org/html/2606.04612#bib.bib46)\)\(Dinget al\.,[2023](https://arxiv.org/html/2606.04612#bib.bib56)\)\(Zhanget al\.,[2025a](https://arxiv.org/html/2606.04612#bib.bib57)\)\(Fuet al\.,[2025](https://arxiv.org/html/2606.04612#bib.bib58)\)exploit vulnerabilities in LLMs, potentially bypassing alignment constraints whilst prompt injection attacks can bypass system instructions\(Greshakeet al\.,[2023](https://arxiv.org/html/2606.04612#bib.bib16)\)\. Recent studies have shown current LLM guardrail systems can still be compromised by attacks\(Hackettet al\.,[2025](https://arxiv.org/html/2606.04612#bib.bib17)\)\.

Adversarial defence methods include adversarial training, perturbation control and use of regularization during training to encourage model robustness\. Adversarial training involves augmenting training data with adversarial perturbations, either explicitly adding new adversarial data instances using an adversarial attack framework or implicitly augmenting training data by dynamically adjusting it during the embedding phase\. Examples of implicit perturbation includeGaoet al\.\([2023](https://arxiv.org/html/2606.04612#bib.bib14)\),Latorreet al\.\([2023](https://arxiv.org/html/2606.04612#bib.bib23)\)and FreeLB\(Zhuet al\.,[2020](https://arxiv.org/html/2606.04612#bib.bib55)\)\. It should be noted that adversarial training is often computationally expensive as it increases training data volumes and requires more training time when fine\-tuning\. Perturbation control is where models aim to recognize and correct perturbations during training\. Examples include the use of spell\-checkingBasemah Alshemali \([2019](https://arxiv.org/html/2606.04612#bib.bib1)\), synonym substitutionDonget al\.\([2021](https://arxiv.org/html/2606.04612#bib.bib8)\)when perturbation sets are known a\-priori and data cleaningBaoet al\.\([2021](https://arxiv.org/html/2606.04612#bib.bib5)\)to limit adversarial data input space\. Regularization type approaches focus on rewarding models during training to encourage robustness, such asWanget al\.\([2021](https://arxiv.org/html/2606.04612#bib.bib44)\), which uses two regularizers to improve out\-of\-domain robustness, andBianchiet al\.\([2024](https://arxiv.org/html/2606.04612#bib.bib6)\), which integrates adversarial training and reinforcement learning to enhance robustness\. Other approaches include PURE\(Wanget al\.,[2025](https://arxiv.org/html/2606.04612#bib.bib45)\), which improves robustness by removing dominant principal components in the hidden representation to create a more isotropic space, SafeLoRAHsuet al\.\([2024](https://arxiv.org/html/2606.04612#bib.bib19)\), which projects LoRA weights onto a safety\-focussed subspace to mitigate safety risks, and SPLoRAAoet al\.\([2025](https://arxiv.org/html/2606.04612#bib.bib2)\)which prunes LoRA layers that significantly deviate from pre\-trained LLM states and could introduce security vulnerabilities\.

To our knowledge, no prior adversarial defence work has explored aggregating entropy\-based, uncertainty\-based and geometric\-based methods as we do in this paper to bridge the gap between hallucination and adversarially focused approaches\.

## 3Hybrid Adversarial Defence Framework

Our hybrid adversarial defence framework, shown in Figure[1](https://arxiv.org/html/2606.04612#S3.F1), consists of three individual adversarial defence models and a choice of two aggregation methods\. The input queries are first sent to individual defence models, an entropy\-based, uncertainty\-based and geometry\-based adversarial defence model\. The aggregation methods then exploit classes of features available from each individual model to train a routing algorithms, which ultimately provides an adversarial defence decision to accept or reject an input query\.

### 3\.1Entropy\-based Model

Our entropy\-based adversarial defence model is motivated by the work ofYaoet al\.\([2023](https://arxiv.org/html/2606.04612#bib.bib48)\), which uses gradient\-based adversarial generation\. For each token positionii, the original method computes gradients with respect to token embeddings, identifying top\-kkreplacement tokens with complexityO​\(T×L×K×V\)O\(T\\times L\\times K\\times V\), whereTTis the number of epochs,LLis the input length,KKis the number of top\-kkcandidates, andVVis the vocabulary size\.

Instead of the original gradient\-based generator, we produce adversarial examples using the TextAttack framework ofMorriset al\.\([2020](https://arxiv.org/html/2606.04612#bib.bib31)\)\. This pragmatic choice greatly improves runtime speed, making large\-scale evaluation feasible, and yields adversarial inputs that are effective for testing of entropy\-based detection\. Appendix[B](https://arxiv.org/html/2606.04612#A2)summarizes the attack recipes employed\.

We use an entropy\-based thresholding mechanism to reject adversarial prompts during inference\. Adversarial inputs induce uncertainty, resulting in higher\-entropy distributions across the vocabulary\. During inference, entropyHHis computed \([1](https://arxiv.org/html/2606.04612#S3.E1)\) over softmax\-normalized logits of the first predicted token wherepip\_\{i\}denotes the softmax probability assigned to tokeniiin vocabularyVV\.

H=−∑i=1Vpi​log⁡piH=\-\\sum\_\{i=1\}^\{V\}p\_\{i\}\\log p\_\{i\}\(1\)If the entropy exceeds a set threshold, the prompt is marked as adversarial and rejected\. This provides a simple and efficient defence against adversarial prompts, without requiring model retraining\. UnlikeYaoet al\.\([2023](https://arxiv.org/html/2606.04612#bib.bib48)\), which uses a universal entropy threshold, we employ dataset\-specific thresholds calculated from the average entropy observed within each domain\. This improves detection robustness across different task types and datasets\.

### 3\.2Uncertainty\-based Model

Our uncertainty\-based adversarial defence model is motivated byZhanget al\.\([2024](https://arxiv.org/html/2606.04612#bib.bib51)\)’s Refusal\-Aware Instruction Tuning \(R\-Tuning\) methodology, which addressed hallucinations by training models to recognize knowledge boundaries and express appropriate uncertainty\. Given pre\-trained parametric knowledge \(PP\) and instruction\-tuned knowledge \(II\), the model’s reliable knowledge space \(P∩IP\\cap I\) is where confident responses are expected, while questions outside this intersection \(I∖PI\\setminus P\) are uncertain and should be refused\.

The training datasetD=\{\(q1,a1\),…,\(qn,an\)\}D=\\\{\(q\_\{1\},a\_\{1\}\),\\ldots,\(q\_\{n\},a\_\{n\}\)\\\}is partitioned \([2](https://arxiv.org/html/2606.04612#S3.E2)\) into certain \(D1D\_\{1\}\) and uncertain \(D0D\_\{0\}\) subsets based on the pre\-trained model’s parametric knowledge\. For each question\-answer pair\(qi,ai\)\(q\_\{i\},a\_\{i\}\), the pre\-trained modelM0M\_\{0\}generates predictiona^i\\hat\{a\}\_\{i\}without fine\-tuning\.

a^i\\displaystyle\\hat\{a\}\_\{i\}=M0​\(qi\)\\displaystyle=M\_\{0\}\(q\_\{i\}\)\(2\)\(qi,ai\)\\displaystyle\(q\_\{i\},a\_\{i\}\)∈D1​if​a^i=ai​\(certain\)\\displaystyle\\in D\_\{1\}\\text\{ if \}\\hat\{a\}\_\{i\}=a\_\{i\}\\text\{ \(certain\)\}\(qi,ai\)\\displaystyle\(q\_\{i\},a\_\{i\}\)∈D0​if​a^i≠ai​\(uncertain\)\\displaystyle\\in D\_\{0\}\\text\{ if \}\\hat\{a\}\_\{i\}\\neq a\_\{i\}\\text\{ \(uncertain\)\}Template=“Q: \{Question\}, A: \{Answer\}\.\\displaystyle=\\text\{\`\`Q: \\\{Question\\\}, A: \\\{Answer\\\}\.\}\{Uncertainty\_Prompt\}”\(3\)A refusal\-aware training dataset is constructed using a standardized template \([3\.2](https://arxiv.org/html/2606.04612#S3.Ex3)\) that combines questions, answers, and uncertainty assessment whereUncertainty\_Prompt= “Are you sure you accurately answered the question based on your internal knowledge?”\. Differential responses are appended based on knowledge certainty classification:

- •Certain Data \(D1D\_\{1\}\): Append “I am sure”
- •Uncertain Data \(D0D\_\{0\}\): Append “I am unsure”

This padding strategy maintains ground\-truth labels while explicitly adding confidence labels\. The model is then fine\-tuned using cross\-entropy loss calculated using response and uncertainty tokens\.

During inference, a two\-stage process extracts both predictions and certainty measures\. First, the model generates an answer\. Then, an uncertainty assessment prompt is appended to evaluate the model’s certainty about its response\. The certainty score is computed by extracting the model’s logits for “sure” and “unsure” tokens and normalizing \([4](https://arxiv.org/html/2606.04612#S3.E4)\) wherepsurep\_\{\\text\{sure\}\}andpunsurep\_\{\\text\{unsure\}\}are the probabilities assigned to the respective tokens by the model’s output distribution\. This provides a calibrated uncertainty measure enabling explicit knowledge boundary recognition\.

c=psurepsure\+punsurec=\\frac\{p\_\{\\text\{sure\}\}\}\{p\_\{\\text\{sure\}\}\+p\_\{\\text\{unsure\}\}\}\(4\)

### 3\.3Geometric\-based Model

Our geometric model draws inspiration from thePuRealgorithm proposed byWanget al\.\([2025](https://arxiv.org/html/2606.04612#bib.bib45)\), which increases adversarial robustness via Principal Component Removal \(PCR\) at the instance level, in an effort to reduce vulnerability to adversarial attacks\. The goal is to reduce the influence of certain directions in the representation space by discarding the principal components responsible for the majority of variance\. For instance\-level representationsX∈ℝn×dX\\in\\mathbb\{R\}^\{n\\times d\}wherennis the sequence length andddis the embedding dimension, we apply Singular Value Decomposition \(SVD\) asX=U​Σ​V⊤X=U\\Sigma V^\{\\top\}, whereUUandVVrepresent the left and right singular vectors, respectively, andΣ\\Sigmais a diagonal matrix comprising the singular values\.

The top\-1 principal componentv1v\_\{1\}, or the largest singular vector, is discarded as described in \([5](https://arxiv.org/html/2606.04612#S3.E5)\)\.

X←X−\(X​v1\)​v1⊤X\\leftarrow X\-\(Xv\_\{1\}\)v\_\{1\}^\{\\top\}\(5\)By removing the rank\-1 component corresponding to the largest singular value, the process eliminates dominating directions representing high\-frequency token sequences and frequently occurring syntactic structures utilized by the adversaries\.

In order to ensure efficient implementation, we use randomized Singular Value Decomposition \(SVD\)\(Halkoet al\.,[2011](https://arxiv.org/html/2606.04612#bib.bib18)\)\. This technique relies on Gaussian random projections to estimate the principal components and achieves significant savings in computation time compared to traditional SVD\.

Following the removal of the principal components, we apply mean pooling of token\-based representations using a parameter\-free form of self\-attention originally proposed byZhaiet al\.\([2023](https://arxiv.org/html/2606.04612#bib.bib50)\), which serves to aggregate tokens into a sentence representation\. The advantage of this method is that it efficiently captures patterns at the sentence\-level in linear time with no addition of any trainable parameter\. This prevents the model from overfitting and provides the model with a richer semantic representation\.

Unlike entropy and uncertainty\-based models, which detect adversarial inputs and selectively refuse to answer,PuReenhances robustness by transforming the embedding space itself, making representations more isotropic and less susceptible to adversarial perturbations\.

![Refer to caption](https://arxiv.org/html/2606.04612v1/HYBRID_X.png)Figure 1:Hybrid Adversarial Defence Framework
### 3\.4Hybrid Adversarial Defence

Our hybrid adversarial defence model integrates entropy\-based, uncertainty\-based and geometric\-based models using a trainable feature\-based routing mechanism\. Figure[1](https://arxiv.org/html/2606.04612#S3.F1)illustrates the overall architecture\. We explore two aggregation strategies that share a common feature representation but differ in how decisions are made\.

ML Routing:A feed\-forward networkfroute​\(x\)f\_\{\\text\{route\}\}\(x\)with a hidden layer of size 512 and ReLU activation takes the feature vectorx∈ℝ31x\\in\\mathbb\{R\}^\{31\}as input and outputs a discrete expert selection wherek∈\{r​t,l​l​m,p​u​r​e\}k\\in\\\{rt,llm,pure\\\}denotes the selected expert\. The final prediction is taken directly from the selected expert \([6](https://arxiv.org/html/2606.04612#S3.E6)\)\.

k\\displaystyle k=arg⁡max⁡froute​\(x\)\\displaystyle=\\arg\\max\\,f\_\{\\text\{route\}\}\(x\)\(6\)
LWV Fusion:A second feed\-forward networkfweight​\(x\)f\_\{\\text\{weight\}\}\(x\), with the same architecture as the routing network, produces a distribution over experts wherepr​tp\_\{rt\},pl​l​mp\_\{llm\}, andpp​u​r​ep\_\{pure\}are the predicted probabilities from each expert \([7](https://arxiv.org/html/2606.04612#S3.E7)\)\.

\[wr​t,wl​l​m,wp​u​r​e\]\\displaystyle\[w\_\{rt\},w\_\{llm\},w\_\{pure\}\]=softmax​\(fweight​\(x\)\)\\displaystyle=\\text\{softmax\}\(f\_\{\\text\{weight\}\}\(x\)\)\(7\)pfinal\\displaystyle p\_\{\\text\{final\}\}=wr​t⋅pr​t\\displaystyle=w\_\{rt\}\\cdot p\_\{rt\}\+wl​l​m⋅pl​l​m\\displaystyle\\quad\+w\_\{llm\}\\cdot p\_\{llm\}\+wp​u​r​e⋅pp​u​r​e\\displaystyle\\quad\+w\_\{pure\}\\cdot p\_\{pure\}Both models use identical architectures to ensure a fair comparison between discrete routing \(hard selection\) and continuous weighting \(soft aggregation\)\. The key difference lies in the decision mechanism: hard routing selects a single expert, while soft aggregation combines all experts through learned weights\.

We extract 31 features from the three expert models across six categories: 6 textual statistics, 6 entropy features, 5 uncertainty features, 3 geometric features, 5 domain encoding and 6 cross\-expert patterns\. Full details of the features extracted are provided in Appendix[A](https://arxiv.org/html/2606.04612#A1), but each is encoded as a scalar within a feature vector we used as input to our aggregation model\.

Entropy\-based features directly detect adversarial inputs by monitoring output entropy patterns, with high entropy in first\-token predictions triggering refusal\. Uncertainty\-based features detects knowledge boundary violations through calibrated confidence scores, allowing refusal of inputs where the model is uncertain\. Geometric\-based features include indicators of purification magnitude, norm ratios, and singular value dominance which can reveal how much the embedding space was modified during purification\.

## 4Baseline Models

### 4\.1FreeLB \- Adversarial Defence

FreeLB byZhuet al\.\([2020](https://arxiv.org/html/2606.04612#bib.bib55)\)is an adversarial training baseline that strengthens model robustness by perturbing word embeddings during fine\-tuning, accumulating parameter gradients across multiple inner perturbation steps, then appling a single consolidated update\. Our configuration follows the original work’s hyperparameters reported in the original paper\. The perturbation budgetϵ\\epsilonranges from 0\.15 to 0\.6 depending on the dataset, with ascent step sizesα\\alphabetween 0\.025 and 0\.15\. All experiments useK=3K=3inner steps with gradients normalized\.

### 4\.2ProtectAI v1 \- Prompt Injection Detection

ProtectAI v1 is a DeBERTa\-v3\-base classifier trained on a dataset that integrates prompt injection and benign prompts\(ProtectAI,[2023](https://arxiv.org/html/2606.04612#bib.bib33)\)\. Approximately 30% of the prompts in this dataset are injections and 70% are benign\. This classifier takes an input prompt and classifies it as either an injection or non\-injection\. ProtectAI v1 is specifically designed to detect prompt injections; it was not trained to detect jailbreak attacks\(Hackettet al\.,[2025](https://arxiv.org/html/2606.04612#bib.bib17)\)\.

### 4\.3NeMoGuard: Jailbreak Detection Baseline

Galinkin and Sablotny \([2024](https://arxiv.org/html/2606.04612#bib.bib13)\)introduce NeMoGuard, a jailbreak detection framework combining pretrained text embeddings with a Random Forest classifier\. The model was trained on a set of 17,085 aggregated examples from three separate datasets: the DAN dataset\(Shenet al\.,[2024](https://arxiv.org/html/2606.04612#bib.bib37)\), containing 1,405 in\-the\-wild jailbreak prompts; the GARAK dataset, containing 126 algorithmically generated jailbreaks using the AutoDAN and TAP algorithms; and the jackhhao dataset, containing 666 jailbreak prompts\. NeMoGuard represents a specialized jailbreak baseline\.

### 4\.4Task Datasets

We evaluate all baseline models and hybrid adversarial defence method on a wide range of datasets covering different aspects of robustness in large language models \(LLMs\)\. For NLU datasets \(Table[1](https://arxiv.org/html/2606.04612#S5.T1)\), FEVER\(Thorneet al\.,[2018](https://arxiv.org/html/2606.04612#bib.bib41)\)and HotpotQA\(Yanget al\.,[2018](https://arxiv.org/html/2606.04612#bib.bib47)\)assess factuality and multi\-step reasoning, while CommonsenseQA\(Talmoret al\.,[2019](https://arxiv.org/html/2606.04612#bib.bib40)\)and Social IQA\(Sapet al\.,[2019](https://arxiv.org/html/2606.04612#bib.bib36)\)evaluate commonsense reasoning\. For out\-of\-distribution datasets\. AEROENG\(Cruz Silvaet al\.,[2025](https://arxiv.org/html/2606.04612#bib.bib38)\)and CIPQA\(Mutaliket al\.,[2025](https://arxiv.org/html/2606.04612#bib.bib32)\)are selected as they cover scientific and technical domains not present in the NLU datasets\. For instruction\-level adversarial robustness, we use SafeGuard\(Erdoganet al\.,[2024](https://arxiv.org/html/2606.04612#bib.bib12)\)to evaluate prompt injection, and AdvBench\(Zouet al\.,[2023](https://arxiv.org/html/2606.04612#bib.bib53)\)and DAN\(Shenet al\.,[2024](https://arxiv.org/html/2606.04612#bib.bib37)\)to evaluate jailbreak attacks\.

## 5In\-Domain Adversarial Defence

We evaluate the following model configurations representing baseline, individual defences and hybrid defence:

- •Base Model:LLaMa3 8B
- •Individual defences: - –LLaMa3 8B \+ entropy\-based model - –LLaMa3 8B \+ uncertainty\-based model - –LLaMa3 8B \+ geometric\-based model
- •Hybrid Adversarial Defence Models - –LLaMa3 8B \+ LWV Fusion - –LLaMa3 8B \+ ML Routing

We use four benchmark NLU datasets covering the range of reasoning types shown in Table[1](https://arxiv.org/html/2606.04612#S5.T1)and described in Section[4\.4](https://arxiv.org/html/2606.04612#S4.SS4)\. All models are fine\-tuned on in\-domain training data using LoRA \(rank 8, 0\.1% trainable parameters\) and evaluated using corresponding test sets\. Model code and hyperparameters for all our experiments are released as open source222GITHUB URI redacted during review periodfor reproducibility purposes\.

Table 1:In\-domain training and evaluation datasetsPer\-domain entropy thresholds used by Entropy\-Based Model were tuned on validation splits with final values: FEVER \(1\.62\), HotpotQA \(1\.84\), CSQA \(2\.04\), and SIQA \(1\.94\)\. We employ TextAttack recipes shown in Table[2](https://arxiv.org/html/2606.04612#S5.T2)with fixed random seed for reproducibility and consistency across all model evaluations\. All experiments were conducted using a single NVIDIA A100 GPU \(80GB memory\)\. Fine\-tuning used LoRA \(rank 8\) for parameter efficiency \(0\.1% trainable parameters\), Adam optimizer, and a learning rate of2×10−52\\times 10^\{\-5\}\.

Table 2:Morriset al\.\([2020](https://arxiv.org/html/2606.04612#bib.bib31)\)TextAttack framework adversarial attack strategies used for robustness evaluation### 5\.1Evaluation Metrics

For clean data, without adversarial perturbations, we report NLU task accuracy and F1 scores to evaluate standard task performance\. Since some models can refuse to answer, we report a refusal rate \(Ref\.\) alongside the accuracy \(Acc\.\) and F1 metrics, which are computed only on test data that was not refused\.

For adversarially attacked data, we report NLU task accuracy and F1 score on non\-refused test data, along with defence\-specific metrics including refusal rate and attack success rate \(ASR\)\. Refusal rate measures the proportion of test data for which the model refused to answer, while attack success rate measures the proportion of adversarial inputs that result in incorrect predictions\.

### 5\.2In\-Domain Task Performance

Table 3:In\-domain clean task performance across all datasets without adversarial attacks\. Accuracy and F1 are computed only for non\-refused test data, with refusal rates reported for each model\. A refusal rate of N/A means model answers all test data\.Bold= best result,underline= second best\.Results in table[3](https://arxiv.org/html/2606.04612#S5.T3)show task performance on clean test data without adversarial perturbations across all four datasets\. Baseline defence methods consistently improve over the undefended model, with the geometric\-based approach providing the strongest individual performance\. Overall, ML routing achieves the highest accuracy across all datasets, where it improves performance by approximately 8\.8 to 13\.5 percentage points depending on the dataset\. By allowing models to refuse to answer test data they are not sure about, our hybrid methods achieve a substantial reduction in hallucinations at the expense of some test data recall\. To put the in\-domain clean task results into a wider context, the latest non\-adversarial agentic LLM methods such as LLM\-Wiki\(Minget al\.,[2026](https://arxiv.org/html/2606.04612#bib.bib59)\)report an F1 score of 83\.9 for HotpotQA across the entire test data\.

Table 4:In\-domain robustness under adversarial attack using TextAttack across all datasets\. Accuracy and F1 are computed only for non\-refused test data, with refusal rates reported for each model\. A refusal rate of N/A means model answers all test data\.Bold= best result,underline= second best\.Results in table[4](https://arxiv.org/html/2606.04612#S5.T4)shows performance under adversarial attack, with the geometric\-based model achieving the strongest individual robustness among baseline methods\. ML Routing outperforms LWV, and achieves a 3\.5% to 19% accuracy improvement over the best baseline model and a 13% to 20% reduction in ASR\.

A detailed breakdown of base model vulnerability by attack type is provided in Appendix[C](https://arxiv.org/html/2606.04612#A3)\.

### 5\.3Discussion

Traditional adversarial training \(FreeLB\) improves robustness through exposure to perturbed examples during training\. However, it exhibits three key limitations: \(1\) it cannot refuse inputs, leaving no fallback for novel attack patterns; \(2\) it incurs substantial training cost while remaining weaker than hybrid approaches; and \(3\) its transfer from continuous embedding perturbations to discrete token\-level attacks is inconsistent, as illustrated by its sharp performance drop on CSQA\.

The uncertainty\-based model depends on internal confidence estimations, which lead to conservative behavior \(high rejection rate\)\. The entropy\-based model is more selective, reducing false positives but potentially missing a larger proportion of attacks\. The geometric\-based model works in embedding space and detects the attacks without making significant changes in confidence and entropy\. This complementarity explains the effectiveness of the hybrid methods\.

An additional observation is the improvement in clean task performance\. The geometric\-based model appears to contribute useful structural features even without adversarial perturbations, while the uncertainty\-based model improves decision calibration by learning when to abstain\. This suggests that the proposed defence mechanisms actively reduce hallucination\.

Both approaches combine the same three experts but differ in how predictions are aggregated\. LWV performs soft averaging, while ML Routing selects a single expert per instance\. The advantage of ML Routing is most pronounced when geometric features are strong, where hard selection avoids diluting informative features\. When features are less separable, the difference between the two approaches is smaller, suggesting that soft averaging remains competitive in such cases\.

## 6OOD Adversarial Defence

Table 5:OOD evaluation datasetsTable 6:Out\-of\-distribution clean and adversarial performance\. Accuracy and F1 are computed only for non\-refused test data, with refusal rates reported for each model\. A refusal rate of N/A means model answers all test data\.Bold= best,underline= second best\.To assess how adversarially defended models generalize beyond their training data distribution, we extend the analysis to out\-of\-distribution \(OOD\) datasets shown in Table[5](https://arxiv.org/html/2606.04612#S6.T5)\. All models are fine\-tuned on an aggregation of all in\-domain training data and evaluated on the OOD testsets\. All OOD adversarial evaluations use the same aggregated attack strategy as the in\-domain setup\. Results in Table[6](https://arxiv.org/html/2606.04612#S6.T6)show model performance for both clean and adversarially attacked OOD data\.

### 6\.1Discussion

The base model behaves quite differently across the two OOD domains\. While AERO maintains 45% accuracy under attack, CPIQA drops sharply to 27%\. Our hypothesis for this is that CPIQA contains more flexible natural language, making synonym\-based attacks easier, whereas AERO relies on more constrained technical vocabulary\. FreeLB shows limited transfer abilities to OOD data\. This is likely due to the mismatch between continuous perturbations used in training and discrete token\-level attacks used at evaluation\.

The best performance is achieved by ML Routing which reduced attack success to 33% on AERO and 25\.89% on CPIQA\. This improvement comes from both stronger predictions and better attack detection, reflected in higher refusal rates\.

## 7Prompt Injection and Jailbreak Defence

Beyond traditional adversarial perturbations, modern LLM threats includeprompt injectionandjailbreakingattacks\. These attacks exploit instruction\-following behavior rather than lexical perturbations, representing a fundamentally different threat model\. We evaluate the same hybrid defence models as our previous experiments to assess zero\-shot generalization to prompt\-level attacks alongside our baseline models\.

Table 7:Prompt injection and jailbreaking evaluation datasets\.Table 8:Zero\-shot evaluation of hybrid models on prompt injection \(SafeGuard\) and jailbreak \(AdvBench, DAN\) datasets\.Bold= best,underline= second best\.Table 9:TextAttack recipes used for adversarial generation, their primary perturbation mechanisms, observed mean token\-level entropy change \(Δ​H\\Delta H\) and example transformations\.We use one prompt injection and two jailbreakings benchmark datasets, shown in Table[7](https://arxiv.org/html/2606.04612#S7.T7)and described in Section[4\.4](https://arxiv.org/html/2606.04612#S4.SS4)\.

### 7\.1Discussion

For prompt injection ML Routing has the best overall performance, with a balance between accuracy \(82\.65%\) and refusal \(90\.38%\)\. ProtectAI v1 produced high precision but low recall, missing a large percentage of attacks, which is consistent withHackettet al\.\([2025](https://arxiv.org/html/2606.04612#bib.bib17)\), who measured a detection rate of 38\.31%, closely aligned with the 40\.79% refusal\. NeMoGuard showed the opposite pattern, with high recall but low precision, indicating that it was not trained on prompt injection data and, therefore, had a difficult time distinguishing between benign and malicious input\.

For jailbreaking, ML Routing provided the most consistently high overall performance, maintaining strong refusal rates without the instability observed in other methods\. NeMoGuard had the highest refusal rate of 96\.44% for the DAN dataset, which we hypothesize that it was trained with data containing structurally similar jailbreaks to the DAN dataset\. ProtectAI v1 performs very poorly \(0\.58% refusal\) on the AdvBench dataset which we think is due to task misalignment, since it is fine\-tuned for prompt injection patterns which are absent in AdvBench\. These results highlight the advantage of hybrid models which combine complementary features, enabling adversarial defence based on model behaviour rather than surface form\.

## Conclusion

We present a hybrid adversarial defence framework that combines entropy\-based detection, uncertainty\-aware training, and geometric representation enhancement\. Across multiple NLU datasets our results show that integrating these complementary features improves both clean task performance, reducing hallucinations, and performance under adversarial attack, increasing robustness\.

Our hybrid model uses features that are invariant across domains, providing an approach that can generalize to any domain\.

For future work we would like to explore adaptive attack scenarios which target the whole hybrid pipeline\. Attackers could theoretically exploit vulnerabilities arising from both expert model and the selector network\.

Another future direction would be to investigate a unified multi\-head defense architecture\. In contrast to training different experts and routing among them, a single Mixture of Experts type model could learn specialized attention heads focusing on the feature types identified in this paper\. This would allow removal of the routing model completely and provide a simpler end to end model\.

## Limitations

All experiments use a base LLM of LLaMa3\-8B\. We see no theoretical reason why our results will not generalize to other base LLMs but we leave it as future work to confirm this\.

The LWV and ML Routing models work as a trainable black\-box routers\. This might create problems for safety\-critical deployments that demand auditable and/or explainable decision processes\.

## Appendix AFeature Vector Description

The hybrid aggregation model features are organised into six categories:

Textual features \(6\):word count, character count, lexical diversity \(unique / total words\), punctuation density, whitespace and uppercase ratio\.

Entropy\-based features \(6\):entropyH=−∑p​log⁡pH=\-\\sum p\\log p, threshold distance\|H−θd\|\|H\-\\theta\_\{d\}\|, confidence derived from entropy, a flag indicating whetherH\>θdH\>\\theta\_\{d\}, token\-level perplexity, and semantic shift between original and perturbed inputs\.

Uncertainty\-based features \(7\):answer confidence, certainty score \(“I am sure” probability\), combined confidence, uncertainty \(1−confidence1\-\\text\{confidence\}\), a certainty flag \(sure/unsure\), margin from the decision boundary, and confidence gap\.

Domain encoding \(5\):binary indicators for FEVER, CSQA, SIQA, HotpotQA, and OTHER\.

Cross\-expert features \(4\):confidence difference between experts, agreement flag, minimum confidence, and maximum confidence\.

Geometric\-based features \(3\):purification magnitude‖𝐡−𝐡′‖2\\\|\\mathbf\{h\}\-\\mathbf\{h\}^\{\\prime\}\\\|\_\{2\}, norm ratio‖𝐡′‖2/‖𝐡‖2\\\|\\mathbf\{h\}^\{\\prime\}\\\|\_\{2\}/\\\|\\mathbf\{h\}\\\|\_\{2\}, and SVD dominanceσ1/∑iσi\\sigma\_\{1\}/\\sum\_\{i\}\\sigma\_\{i\}\.

## Appendix BEntropy\-based Model TextAttack Recipes

Table[9](https://arxiv.org/html/2606.04612#S7.T9)shows the TextAttack recipes used\.

## Appendix CBase Model Vulnerability Analysis by Attack Type

Table 10:Base model vulnerability by attack type on FEVER dataset\.Table[10](https://arxiv.org/html/2606.04612#A3.T10)presents results for the undefended LLaMA\-3\-8B base model when exposed to four adversarial attack families on FEVER dataset\. The base model refuses none of the adversarial inputs \(0% refusal\), confirming it operates without defence mechanisms\. These results establish the reference point for understanding main results in \(Table[4](https://arxiv.org/html/2606.04612#S5.T4)\)\.

## Acknowledgments

This work was supported by the Economic and Social Research Council \(ES/V011278/1\)\. The authors acknowledge the IRIDIS High\-Performance Computing Facility at the University of Southampton\.

## References

- Safe pruning LoRA: robust distance\-guided pruning for safety alignment in adaptation of LLMs\.Transactions of the Association for Computational Linguistics13,pp\. 1474–1487\.External Links:[Link](https://aclanthology.org/2025.tacl-1.67/),[Document](https://dx.doi.org/10.1162/tacl.a.44)Cited by:[§2\.2](https://arxiv.org/html/2606.04612#S2.SS2.p2.1)\.
- R\. Bao, J\. Wang, and H\. Zhao \(2021\)Defending pre\-trained language models from adversarial word substitution without performance sacrifice\.InFindings of the Association for Computational Linguistics: ACL\-IJCNLP 2021,C\. Zong, F\. Xia, W\. Li, and R\. Navigli \(Eds\.\),Online,pp\. 3248–3258\.External Links:[Link](https://aclanthology.org/2021.findings-acl.287/),[Document](https://dx.doi.org/10.18653/v1/2021.findings-acl.287)Cited by:[§2\.2](https://arxiv.org/html/2606.04612#S2.SS2.p2.1)\.
- J\. K\. Basemah Alshemali \(2019\)Toward mitigating adversarial texts\.International Journal of Computer Applications178\(50\),pp\. 1–7\.External Links:ISSN 0975\-8887,[Link](https://ijcaonline.org/archives/volume178/number50/30888-2019919384/),[Document](https://dx.doi.org/10.5120/ijca2019919384)Cited by:[§2\.2](https://arxiv.org/html/2606.04612#S2.SS2.p2.1)\.
- F\. Bianchi, M\. Suzgun, G\. Attanasio, P\. Rottger, D\. Jurafsky, T\. Hashimoto, and J\. Zou \(2024\)Safety\-tuned LLaMAs: lessons from improving the safety of large language models that follow instructions\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=gT5hALch9z)Cited by:[§2\.2](https://arxiv.org/html/2606.04612#S2.SS2.p2.1)\.
- T\. B\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. M\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. Amodei \(2020\)Language models are few\-shot learners\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 1877–1901\.External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html)Cited by:[§1](https://arxiv.org/html/2606.04612#S1.p1.1)\.
- E\. A\. Cruz Silva, R\. Marsh, H\. Yong, S\. E\. Middleton, and A\. Sobester \(2025\)Retrieval\-augmented generation and in\-context prompted large language models in aircraft engineering\.InProceedings of the AIAA Scitech 2025 Forum,External Links:[Document](https://dx.doi.org/10.2514/6.2025-0700)Cited by:[§4\.4](https://arxiv.org/html/2606.04612#S4.SS4.p1.1)\.
- P\. Ding, J\. Kuang, D\. Ma, X\. Cao, Y\. Xian, J\. Chen, and S\. Huang \(2023\)A wolf in sheep’s clothing: generalized nested jailbreak prompts can fool large language models easily\.External Links:2311\.08268Cited by:[§2\.2](https://arxiv.org/html/2606.04612#S2.SS2.p1.1)\.
- X\. Dong, A\. T\. Luu, R\. Ji, and H\. Liu \(2021\)Towards robustness against natural language word substitutions\.External Links:2107\.13541,[Link](https://arxiv.org/abs/2107.13541)Cited by:[§2\.2](https://arxiv.org/html/2606.04612#S2.SS2.p2.1)\.
- N\. Dziri, A\. Madotto, O\. Zaiane, and A\. J\. Bose \(2021\)Neural path hunter: reducing hallucination in dialogue systems via path grounding\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,Online and Punta Cana, Dominican Republic,pp\. 2197–2214\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.168),[Link](https://aclanthology.org/2021.emnlp-main.168/)Cited by:[§2\.1](https://arxiv.org/html/2606.04612#S2.SS1.p2.1)\.
- M\. Elaraby, M\. Lu, J\. Dunn, X\. Zhang, Y\. Wang, and S\. Liu \(2023\)Halo: estimation and reduction of hallucinations in open\-source weak large language models\.arXiv preprint arXiv:2308\.11764\.Cited by:[§2\.1](https://arxiv.org/html/2606.04612#S2.SS1.p2.1)\.
- L\. E\. Erdogan, C\. Shang, A\. Goyal, and S\. Ijju \(2024\)Safe\-Guard prompt injection dataset\.Note:[https://huggingface\.co/datasets/xTRam1/safe\-guard\-prompt\-injection](https://huggingface.co/datasets/xTRam1/safe-guard-prompt-injection)HuggingFace datasetCited by:[§4\.4](https://arxiv.org/html/2606.04612#S4.SS4.p1.1)\.
- S\. Fu, D\. Su, B\. Huang, F\. Sun, J\. Wang, W\. Chen, H\. Shen, and X\. Cheng \(2025\)Jailbreak LLMs through internal stance manipulation\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 15444–15459\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.780/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.780),ISBN 979\-8\-89176\-332\-6Cited by:[§2\.2](https://arxiv.org/html/2606.04612#S2.SS2.p1.1)\.
- E\. Galinkin and M\. Sablotny \(2024\)Improved large language model jailbreak detection via pretrained embeddings\.Note:arXiv:2412\.01547v1External Links:2412\.01547v1Cited by:[§4\.3](https://arxiv.org/html/2606.04612#S4.SS3.p1.1)\.
- S\. Gao, S\. Dou, Y\. Liu, X\. Wang, Q\. Zhang, Z\. Wei, J\. Ma, and Y\. Shan \(2023\)DSRM: boost textual adversarial training with distribution shift risk minimization\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 12177–12189\.External Links:[Link](https://aclanthology.org/2023.acl-long.680/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.680)Cited by:[§2\.2](https://arxiv.org/html/2606.04612#S2.SS2.p2.1)\.
- K\. Greshake, S\. Abdelnabi, S\. Mishra, C\. Endres, T\. Holz, and M\. Fritz \(2023\)Not what you’ve signed up for: compromising real\-world LLM\-integrated applications with indirect prompt injection\.InProceedings of the 16th ACM Workshop on Artificial Intelligence and Security,pp\. 79–90\.External Links:[Document](https://dx.doi.org/10.1145/3605764.3623985)Cited by:[§2\.2](https://arxiv.org/html/2606.04612#S2.SS2.p1.1)\.
- W\. Hackett, L\. Birch, S\. Trawicki, N\. Suri, and P\. Garraghan \(2025\)Bypassing LLM guardrails: an empirical analysis of evasion attacks against prompt injection and jailbreak detection systems\.InProceedings of the First Workshop on LLM Security,Vienna, Austria,pp\. 101–114\.Cited by:[§2\.2](https://arxiv.org/html/2606.04612#S2.SS2.p1.1),[§4\.2](https://arxiv.org/html/2606.04612#S4.SS2.p1.1),[§7\.1](https://arxiv.org/html/2606.04612#S7.SS1.p1.1)\.
- N\. Halko, P\. Martinsson, and J\. A\. Tropp \(2011\)Finding structure with randomness: probabilistic algorithms for constructing approximate matrix decompositions\.SIAM Review53\(2\),pp\. 217–288\.External Links:[Document](https://dx.doi.org/10.1137/090771806)Cited by:[§3\.3](https://arxiv.org/html/2606.04612#S3.SS3.p3.1)\.
- C\. Hsu, Y\. Tsai, C\. Lin, P\. Chen, C\. Yu, and C\. Huang \(2024\)Safe loRA: the silver lining of reducing safety risks when finetuning large language models\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=HcifdQZFZV)Cited by:[§2\.2](https://arxiv.org/html/2606.04612#S2.SS2.p2.1)\.
- Z\. Ji, N\. Lee, R\. Frieske, T\. Yu, D\. Su, Y\. Xu, E\. Ishii, Y\. Bang, D\. Chen, W\. Dai, H\. S\. Chan, A\. Madotto, and P\. Fung \(2023\)Survey of hallucination in natural language generation\.ACM Computing Surveys55\(12\),pp\. 1–38\.External Links:[Document](https://dx.doi.org/10.1145/3571730)Cited by:[§2\.1](https://arxiv.org/html/2606.04612#S2.SS1.p1.1)\.
- D\. Jin, Z\. Jin, J\. T\. Zhou, and P\. Szolovits \(2020\)Is bert really robust? a strong baseline for natural language attack on text classification and entailment\.Proceedings of the AAAI Conference on Artificial Intelligence34\(05\),pp\. 8018–8025\.External Links:[Link](https://ojs.aaai.org/index.php/AAAI/article/view/6311),[Document](https://dx.doi.org/10.1609/aaai.v34i05.6311)Cited by:[§2\.2](https://arxiv.org/html/2606.04612#S2.SS2.p1.1)\.
- F\. Latorre, I\. Krawczuk, L\. T\. Dadi, T\. Pethick, and V\. Cevher \(2023\)Finding actual descent directions for adversarial training\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=I3HCE7Ro78H)Cited by:[§2\.2](https://arxiv.org/html/2606.04612#S2.SS2.p2.1)\.
- P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel, S\. Riedel, and D\. Kiela \(2020\)Retrieval\-augmented generation for knowledge\-intensive NLP tasks\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 9459–9474\.Cited by:[§2\.1](https://arxiv.org/html/2606.04612#S2.SS1.p2.1)\.
- J\. Li, S\. Ji, T\. Du, B\. Li, and T\. Wang \(2018\)TextBugger: generating adversarial text against real\-world applications\.CoRRabs/1812\.05271\.External Links:[Link](http://arxiv.org/abs/1812.05271),1812\.05271Cited by:[§2\.2](https://arxiv.org/html/2606.04612#S2.SS2.p1.1)\.
- P\. Manakul, A\. Liusie, and M\. J\. F\. Gales \(2023\)SelfCheckGPT: zero\-resource black\-box hallucination detection for generative large language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Singapore,pp\. 9004–9017\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.557),[Link](https://aclanthology.org/2023.emnlp-main.557/)Cited by:[§2\.1](https://arxiv.org/html/2606.04612#S2.SS1.p2.1)\.
- R\. Massenon, I\. Gambo, J\. A\. Khan, I\. Iacob, E\. Berki, A\. Araujo, C\. Liu, W\. Zhao, and W\. Zhang \(2025\)My AI is lying to me: user\-reported LLM hallucinations in AI mobile apps reviews\.Scientific Reports15,pp\. 30397\.External Links:[Document](https://dx.doi.org/10.1038/s41598-025-15416-8)Cited by:[§2\.1](https://arxiv.org/html/2606.04612#S2.SS1.p1.1)\.
- J\. Maynez, S\. Narayan, B\. Bohnet, and R\. McDonald \(2020\)On faithfulness and factuality in abstractive summarization\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 1906–1919\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.173),[Link](https://aclanthology.org/2020.acl-main.173/)Cited by:[§2\.1](https://arxiv.org/html/2606.04612#S2.SS1.p2.1)\.
- H\. Ming, F\. Li, X\. Wu, and W\. Que \(2026\)Retrieval as reasoning: self\-evolving agent\-native retrieval via llm\-wiki\.External Links:2605\.25480,[Link](https://arxiv.org/abs/2605.25480)Cited by:[§5\.2](https://arxiv.org/html/2606.04612#S5.SS2.p1.1)\.
- J\. Morris, E\. Lifland, J\. Y\. Yoo, J\. Grigsby, D\. Jin, and Y\. Qi \(2020\)TextAttack: a framework for adversarial attacks, data augmentation, and adversarial training in nlp\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,pp\. 119–126\.Cited by:[§2\.2](https://arxiv.org/html/2606.04612#S2.SS2.p1.1),[§3\.1](https://arxiv.org/html/2606.04612#S3.SS1.p2.1),[Table 2](https://arxiv.org/html/2606.04612#S5.T2)\.
- R\. Mutalik, A\. Panchalingam, L\. G\. Singh, T\. J\. Osborn, E\. Hawkins, and S\. E\. Middleton \(2025\)CPIQA: climate paper image question answering dataset for retrieval\-augmented generation with context\-based query expansion\.InProceedings of the 2nd Workshop on Natural Language Processing Meets Climate Change,Vienna, Austria,pp\. 218–232\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.climatenlp-1.15),[Link](https://aclanthology.org/2025.climatenlp-1.15/)Cited by:[§4\.4](https://arxiv.org/html/2606.04612#S4.SS4.p1.1)\.
- ProtectAI \(2023\)Fine\-tuned DeBERTa\-v3 for prompt injection detection\.Note:[https://huggingface\.co/ProtectAI/deberta\-v3\-base\-prompt\-injection](https://huggingface.co/ProtectAI/deberta-v3-base-prompt-injection)HuggingFace modelCited by:[§4\.2](https://arxiv.org/html/2606.04612#S4.SS2.p1.1)\.
- S\. Ren, Y\. Deng, K\. He, and W\. Che \(2019\)Generating natural language adversarial examples through probability weighted word saliency\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,A\. Korhonen, D\. Traum, and L\. Màrquez \(Eds\.\),Florence, Italy,pp\. 1085–1097\.External Links:[Link](https://aclanthology.org/P19-1103/),[Document](https://dx.doi.org/10.18653/v1/P19-1103)Cited by:[§2\.2](https://arxiv.org/html/2606.04612#S2.SS2.p1.1)\.
- M\. Sap, H\. Rashkin, D\. Chen, R\. Le Bras, and Y\. Choi \(2019\)Social IQa: commonsense reasoning about social interactions\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing,Hong Kong, China,pp\. 4463–4473\.External Links:[Document](https://dx.doi.org/10.18653/v1/D19-1454),[Link](https://aclanthology.org/D19-1454/)Cited by:[§4\.4](https://arxiv.org/html/2606.04612#S4.SS4.p1.1)\.
- X\. Shen, Z\. Chen, M\. Backes, Y\. Shen, and Y\. Zhang \(2024\)“Do anything now”: characterizing and evaluating in\-the\-wild jailbreak prompts on large language models\.InProceedings of the 2024 ACM SIGSAC Conference on Computer and Communications Security,Cited by:[§4\.3](https://arxiv.org/html/2606.04612#S4.SS3.p1.1),[§4\.4](https://arxiv.org/html/2606.04612#S4.SS4.p1.1)\.
- A\. Talmor, J\. Herzig, N\. Lourie, and J\. Berant \(2019\)CommonsenseQA: a question answering challenge targeting commonsense knowledge\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,Minneapolis, Minnesota,pp\. 4149–4158\.External Links:[Document](https://dx.doi.org/10.18653/v1/N19-1421),[Link](https://aclanthology.org/N19-1421/)Cited by:[§4\.4](https://arxiv.org/html/2606.04612#S4.SS4.p1.1)\.
- J\. Thorne, A\. Vlachos, C\. Christodoulopoulos, and A\. Mittal \(2018\)FEVER: a large\-scale dataset for fact extraction and verification\.InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,New Orleans, Louisiana,pp\. 809–819\.External Links:[Document](https://dx.doi.org/10.18653/v1/N18-1074),[Link](https://aclanthology.org/N18-1074/)Cited by:[§4\.4](https://arxiv.org/html/2606.04612#S4.SS4.p1.1)\.
- H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar, A\. Rodriguez, A\. Joulin, E\. Grave, and G\. Lample \(2023\)LLaMA: open and efficient foundation language models\.Note:arXiv:2302\.13971v1External Links:2302\.13971v1Cited by:[§1](https://arxiv.org/html/2606.04612#S1.p1.1)\.
- N\. Varshney, W\. Yao, H\. Zhang, J\. Chen, and D\. Yu \(2023\)A stitch in time saves nine: detecting and mitigating hallucinations of LLMs by validating low\-confidence generation\.Note:arXiv:2307\.03987v1External Links:2307\.03987v1Cited by:[§2\.1](https://arxiv.org/html/2606.04612#S2.SS1.p2.1)\.
- B\. Wang, S\. Wang, Y\. Cheng, Z\. Gan, R\. Jia, B\. Li, and J\. Liu \(2021\)Info\{bert\}: improving robustness of language models from an information theoretic perspective\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=hpH98mK5Puk)Cited by:[§2\.2](https://arxiv.org/html/2606.04612#S2.SS2.p2.1)\.
- Y\. Wang, C\. Xiao, Y\. Li, S\. E\. Middleton, N\. Al Moubayed, and C\. Lin \(2025\)Adversarial defence without adversarial defence: enhancing language model robustness via instance\-level principal component removal\.Note:ArXiv:2507\.21750v4External Links:2507\.21750v4Cited by:[§1](https://arxiv.org/html/2606.04612#S1.p2.1),[§2\.2](https://arxiv.org/html/2606.04612#S2.SS2.p2.1),[§3\.3](https://arxiv.org/html/2606.04612#S3.SS3.p1.7)\.
- A\. Wei, N\. Haghtalab, and J\. Steinhardt \(2023\)Jailbroken: how does LLM safety training fail?\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§2\.2](https://arxiv.org/html/2606.04612#S2.SS2.p1.1)\.
- Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. W\. Cohen, R\. Salakhutdinov, and C\. D\. Manning \(2018\)HotpotQA: a dataset for diverse, explainable multi\-hop question answering\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,Brussels, Belgium,pp\. 2369–2380\.External Links:[Document](https://dx.doi.org/10.18653/v1/D18-1259),[Link](https://aclanthology.org/D18-1259/)Cited by:[§4\.4](https://arxiv.org/html/2606.04612#S4.SS4.p1.1)\.
- J\. Yao, K\. Ning, Z\. Liu, M\. Ning, and L\. Yuan \(2023\)LLM lies: hallucinations are not bugs, but features as adversarial examples\.Note:arXiv:2310\.01469v2External Links:2310\.01469v2Cited by:[§1](https://arxiv.org/html/2606.04612#S1.p2.1),[§2\.1](https://arxiv.org/html/2606.04612#S2.SS1.p2.1),[§3\.1](https://arxiv.org/html/2606.04612#S3.SS1.p1.8),[§3\.1](https://arxiv.org/html/2606.04612#S3.SS1.p3.5)\.
- X\. Zhai, A\. Kolesnikov, N\. Houlsby, and L\. Beyer \(2023\)Scaling vision transformers\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 12104–12113\.External Links:[Document](https://dx.doi.org/10.1109/CVPR52688.2022.01179)Cited by:[§3\.3](https://arxiv.org/html/2606.04612#S3.SS3.p4.1)\.
- H\. Zhang, S\. Diao, Y\. Lin, Y\. Fung, Q\. Lian, X\. Wang, Y\. Chen, H\. Ji, and T\. Zhang \(2024\)R\-Tuning: instructing large language models to say I don’t know\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,Mexico City, Mexico,pp\. 7113–7139\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.394),[Link](https://aclanthology.org/2024.naacl-long.394/)Cited by:[§1](https://arxiv.org/html/2606.04612#S1.p2.1),[§2\.1](https://arxiv.org/html/2606.04612#S2.SS1.p2.1),[§3\.2](https://arxiv.org/html/2606.04612#S3.SS2.p1.4)\.
- X\. Zhang, X\. Yin, D\. Jing, H\. Zhang, X\. Hu, and X\. Wan \(2025a\)DAMON: a dialogue\-aware MCTS framework for jailbreaking large language models\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 6350–6366\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.323/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.323),ISBN 979\-8\-89176\-332\-6Cited by:[§2\.2](https://arxiv.org/html/2606.04612#S2.SS2.p1.1)\.
- Y\. Zhang, Y\. Li, L\. Cui, D\. Cai, L\. Liu, T\. Fu, X\. Huang, E\. Zhao, Y\. Zhang, Y\. Chen, L\. Wang, A\. T\. Luu, W\. Bi, F\. Shi, and S\. Shi \(2025b\)Siren’s song in the AI ocean: a survey on hallucination in large language models\.Computational Linguistics51,pp\. 1373–1418\.External Links:[Document](https://dx.doi.org/10.1162/coli.a.16)Cited by:[§2\.1](https://arxiv.org/html/2606.04612#S2.SS1.p1.1)\.
- C\. Zhu, Y\. Cheng, Z\. Gan, S\. Sun, T\. Goldstein, and J\. Liu \(2020\)FreeLB: enhanced adversarial training for natural language understanding\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=BygzbyHFvB)Cited by:[§2\.2](https://arxiv.org/html/2606.04612#S2.SS2.p2.1),[§4\.1](https://arxiv.org/html/2606.04612#S4.SS1.p1.3)\.
- A\. Zou, Z\. Wang, J\. Z\. Kolter, and M\. Fredrikson \(2023\)Universal and transferable adversarial attacks on aligned language models\.Note:arXiv:2307\.15043v1External Links:2307\.15043v1Cited by:[§4\.4](https://arxiv.org/html/2606.04612#S4.SS4.p1.1)\.
- H\. Zubkova, J\. Park, and S\. Lee \(2025\)SUGAR: leveraging contextual confidence for smarter retrieval\.Note:arXiv:2501\.04899v1External Links:2501\.04899v1Cited by:[§2\.1](https://arxiv.org/html/2606.04612#S2.SS1.p2.1)\.