Learning What to Fail On: Failure-Mode Contextual Bandits for Adversarial Data Curation

# Learning What to Fail On: Failure-Mode Contextual Bandits for Adversarial Data Curation
Source: [https://arxiv.org/html/2608.18681](https://arxiv.org/html/2608.18681)
Ofir Cohencofir@post\.bgu\.ac\.ilRami Puzispuzis@bgu\.ac\.ilAffiliation:Software and Information Systems Engineering, Ben Gurion University of The NegevAsaf Shabtaishabtaia@bgu\.ac\.ilAffiliation:Software and Information Systems Engineering, Ben Gurion University of The NegevOfer Hadarhadar@bgu\.ac\.il\[2mm\]ElectricalComputers EngineeringBen Gurion University of The Negev

###### Abstract

We introduce a failure\-aware adversarial retrieval\-augmented framework for improving robustness in natural language understanding\. Rather than selecting synthetic examples with a fixed reward threshold, our method formulates adversarial data curation as a failure\-mode contextual bandit problem\. Candidate examples are generated with retrieval\-augmented prompting, filtered by the current target model, automatically validated by an LLM judge ensemble, and clustered into recurring failure modes\. A stochastic policy then selects which failure modes to sample for retraining, and is updated using validation\-based reward that balances robustness gains, forgetting, and data cost\. This makes the data curator itself the learning agent, enabling adaptive selection of the most useful model failures across training rounds\. On standard benchmarks, our approach improves RoBERTa\-base accuracy from 88\.48% to 92\.60% on SNLI, from 75\.04% to 80\.95% on ANLI, and from 54\.67% to 71\.99% on MultiNLI, while consistently outperforming prior adversarial augmentation methods\. We further demonstrate transfer to FEVER fact verification, achieving up to 79\.86% FEVER score and 82\.45% accuracy with RoBERTa\-large\. Finally, we provide a theoretical interpretation showing that, under stated assumptions, failure\-mode sampling can reduce shortcut\-aligned gradient contributions while inducing bounded distributional drift\. By combining retrieval, automated validation, contextual\-bandit failure selection, and controlled adversarial retraining, our framework enables scalable robustness improvement without additional human annotation\.

![Refer to caption](https://arxiv.org/html/2608.18681v1/attack_teaser_new_rl.png)Figure 1:Overview of the proposed failure\-mode contextual bandit curation framework\. Given an input\-label pair, retrieval\-augmented prompting generates candidate outputs\. The target model filters candidates by retaining only incorrect predictions, which are then automatically validated by an LLM judge ensemble and clustered into recurring failure modes\. A contextual\-bandit policy selects failure modes under an adversarial budget, and selected examples are mixed with original data to retrain the target model\. Validation feedback provides rewardGtG\_\{t\}for updating the policyπθ\\pi\_\{\\theta\}and criticRϕR\_\{\\phi\}, enabling adaptive selection of high\-impact failure modes across rounds\.## 1Introduction

Natural Language Inference \(NLI\), the task of determining whether a hypothesis is entailed by, contradicted by, or neutral with respect to a given premise, is a central component of many natural language understanding problems, including question answering, summarization, dialogue systems, and fact verification\. More broadly, many supervised NLP tasks suffer from similar robustness limitations when confronted with adversarial or out\-of\-domain examples\. Despite rapid progress, even state\-of\-the\-art models remain brittle, often relying on spurious lexical cues or failing under simple syntactic and semantic variations\([7](https://arxiv.org/html/2608.18681#bib.bib9);[3](https://arxiv.org/html/2608.18681#bib.bib8)\)\. Benchmarks such as SNLI and MultiNLI\([2](https://arxiv.org/html/2608.18681#bib.bib3);[32](https://arxiv.org/html/2608.18681#bib.bib7)\), as well as adversarial datasets\([27](https://arxiv.org/html/2608.18681#bib.bib1)\), have driven robustness improvements but incur high annotation costs and still leave many failure modes uncovered\. More recently, large\-scale synthetic datasets such as GNLI\([9](https://arxiv.org/html/2608.18681#bib.bib19)\)have been generated automatically, but their largely untargeted nature can dilute the adversarial patterns most useful for improving a particular target model\. Existing adversarial data generation pipelines typically rely on static filtering, heuristic selection rules, or one\-shot validation\. As a result, they do not explicitly learn which types of failures should be prioritized as the target model evolves\. This limitation is especially important because model failures are not equally useful: some expose persistent decision shortcuts, while others are noisy, redundant, or too easy to produce meaningful robustness gains\. Motivated by this gap, we formulate adversarial data curation as afailure\-mode contextual banditproblem\. The learning agent is not the target classifier itself, but the data curator that decides which types of validated model failures should be sampled for retraining\. Our framework first retrieves label\-balanced few\-shot contexts using both semantic embeddings \(BGE M3\([4](https://arxiv.org/html/2608.18681#bib.bib10)\)\) and lexical matching \(BM25\([30](https://arxiv.org/html/2608.18681#bib.bib11)\)\)\. These contexts are assembled into LLM prompts to generate challenging candidate hypotheses\. Each candidate is evaluated by the current target model, and only examples that induce misclassification are passed to an automated LLM judge ensemble for label validation\. The validated failures are then embedded and clustered into recurring failure modes, such as lexical shortcut failures, negation errors, entity mismatch errors, numerical reasoning failures, or contradiction confusion\. A stochastic contextual\-bandit policy then observes a state vector for each failure mode, including cluster size, target\-model loss, uncertainty, classification margin, label distribution, retrieval score, judge agreement, novelty, and previous reward statistics\. The policy selects which failure modes to sample under a fixed adversarial budget\. After retraining the target model on a controlled mixture of original and selected adversarial examples, the policy receives a validation\-based reward that balances robustness improvement, forgetting on the clean distribution, and data cost\. A lightweight critic estimates the expected utility of each failure mode, but selection is governed by the learned policy rather than a fixed reward threshold\. This design provides an explicit policy, action space, reward signal, and policy\-optimization procedure for adaptive adversarial data curation[14](https://arxiv.org/html/2608.18681#bib.bib37)\. In human\-free adversarial fine\-tuning and transfer evaluations on NLI benchmarks, our approach improves RoBERTa\-base accuracy from 88\.48% to 92\.60% on SNLI, from 75\.04% to 80\.95% on ANLI, and from 54\.67% to 71\.99% on MultiNLI, while consistently outperforming prior adversarial augmentation methods\. Beyond NLI, we further demonstrate transfer to the FEVER fact verification benchmark\. Using RoBERTa\-large, our method achieves up to 79\.86% FEVER score and 82\.45% label accuracy, outperforming strong retrieval\-augmented and synthetic\-data baselines\. These results indicate that failure\-mode bandit curation can improve robustness across task formulations and supervision regimes while using automatically generated and automatically validated data\.Our contributions are:

Framework\.We propose a failure\-mode contextual bandit framework for adversarial data curation\. The framework requires no additional human annotation and learns which validated model\-failure modes should be sampled for retraining\.

Methodology\.We introduce an adaptive curation pipeline that combines label\-balanced retrieval, LLM\-based candidate generation, target\-model failure filtering, automated judge validation, unsupervised failure\-mode clustering, and contextual\-bandit selection under an adversarial data budget\.

Empirical Evaluation\.We provide empirical evidence that failure\-mode bandit curation improves robustness and data efficiency compared with static adversarial augmentation, reward\-threshold filtering, retrieval\-only selection, and untargeted synthetic\-data baselines\.

Theoretical Interpretation\.We provide an analytical interpretation showing that, under stated assumptions, failure\-mode sampling can reduce shortcut\-aligned gradient contributions while preserving core\-feature contributions\. We further show that mixture\-based updates induce bounded distributional drift and that bounded utility noise causes bounded distortion in the induced sampling policy\.

## 2Background and Related Work

Improving the robustness of NLI models remains a central challenge in natural language understanding\([7](https://arxiv.org/html/2608.18681#bib.bib9);[3](https://arxiv.org/html/2608.18681#bib.bib8)\)\. Benchmarks such as SNLI\([2](https://arxiv.org/html/2608.18681#bib.bib3)\)and MultiNLI\([32](https://arxiv.org/html/2608.18681#bib.bib7)\)have enabled large\-scale supervised training, while ANLI\([27](https://arxiv.org/html/2608.18681#bib.bib1)\)introduced a human\-and\-model\-in\-the\-loop protocol for collecting harder adversarial examples\. However, these datasets require substantial annotation effort and still leave many systematic failure modes uncovered\. More recently, automated and synthetic data\-generation approaches have reduced the need for human annotation\. For example, GNLI\([9](https://arxiv.org/html/2608.18681#bib.bib19)\)shows that large\-scale generated NLI data can rival human\-curated data, and[19](https://arxiv.org/html/2608.18681#bib.bib20)proposed a training\-free retrieval\-augmented framework for adversarial detection and filtering\. Related work on counterfactual and paraphrase generation has also been used to enrich training distributions\([22](https://arxiv.org/html/2608.18681#bib.bib4);[20](https://arxiv.org/html/2608.18681#bib.bib5);[15](https://arxiv.org/html/2608.18681#bib.bib36)\)\. Despite these advances, most existing approaches rely on static generation, heuristic filtering, or one\-shot validation, and therefore do not explicitly learn which types of model failures should be prioritized as the target model evolves\.

Adversarial and Synthetic Example Generation\.Automated adversarial pipelines aim to expose and correct model weaknesses without manual curation\.[26](https://arxiv.org/html/2608.18681#bib.bib12);[16](https://arxiv.org/html/2608.18681#bib.bib35)generate logical\-constraint\-violating examples, improving robustness on SNLI and MultiNLI\.[28](https://arxiv.org/html/2608.18681#bib.bib13)use a model\-in\-the\-loop setup to surface challenging examples and improve out\-of\-domain transfer\.[11](https://arxiv.org/html/2608.18681#bib.bib14);[17](https://arxiv.org/html/2608.18681#bib.bib34)introduce syntactically controlled transformations for paraphrase\-based attacks\. Recent LLM\-based pipelines further demonstrate that generated hypotheses, especially when combined with automated validation, can provide useful adversarial supervision\. However, these methods typically select examples using fixed rules, confidence thresholds, heuristic filters, or limited human feedback\. In contrast, our work treats adversarial data curation as an adaptive learning problem: the system identifies recurring target\-model failures, clusters them into failure modes, and learns which modes are most useful for retraining[18](https://arxiv.org/html/2608.18681#bib.bib33)\.

Retrieval for Few\-Shot Prompting\.Few\-shot retrieval is important for reliable LLM\-based generation because the retrieved context controls both the label distribution and the semantic structure of generated examples\. Dense retrieval models such as BGE\([4](https://arxiv.org/html/2608.18681#bib.bib10)\)provide semantic similarity, while lexical methods such as BM25\([30](https://arxiv.org/html/2608.18681#bib.bib11)\)capture surface\-level overlap and exact lexical cues\. Hybrid retrieval can therefore provide complementary context for generating diverse and label\-consistent hypotheses\. In our framework, retrieval is not the main contribution by itself\. Instead, it serves as the first stage of a larger closed\-loop curation pipeline: retrieved examples guide generation, generated candidates expose target\-model failures, and the downstream bandit policy decides which validated failure modes should be sampled for retraining\.

Reinforcement Learning, Bandits, and Data Selection\.Learning to select training examples has been studied in reinforcement\-learning and curriculum\-learning settings\.[6](https://arxiv.org/html/2608.18681#bib.bib24)propose a Neural Data Filter that learns to select useful training samples\. Reinforcement\-guided curricula have also been explored in structured prediction tasks such as neural machine translation, where policies learn to sequence or weight examples for improved training\([36](https://arxiv.org/html/2608.18681#bib.bib25)\)\. More recent work formulates data selection for model finetuning as a sequential decision problem, where an agent chooses subsets of data to optimize validation rewards\([13](https://arxiv.org/html/2608.18681#bib.bib23)\)\. Related methods such as LearnAlign\([21](https://arxiv.org/html/2608.18681#bib.bib21)\)and RL\-Selector\([34](https://arxiv.org/html/2608.18681#bib.bib22)\)use reward feedback to select informative examples or reduce redundancy\. Our method differs from these approaches in both the unit of selection and the source of supervision\. Rather than selecting arbitrary training examples, we first construct a validated pool of adversarial failures induced by the current target model\. These failures are then clustered into semantic failure modes, and a contextual\-bandit policy selects which modes to sample under a fixed adversarial budget\. The reward is computed after retraining, using validation improvement, forgetting penalty, and data cost\. This provides an explicit state, action, reward, and policy update while avoiding the instability and expense of per\-example utility estimation\.

## 3Methodology

![Refer to caption](https://arxiv.org/html/2608.18681v1/examples.png)Figure 2:Example of the progressive construction of our failure\-aware policy\. From left to right: using only an LLM for hypothesis generation without retrieval or feedback; adding few\-shot retrieval to condition generation on similar failures; and the full reinforcement\-guided policy, which combines retrieval, multi\-model validation, and reward\-based selection to identify and reinject high\-impact adversarial updates\.Let𝒟=\{\(pi,yi\)\}i=1N\\mathcal\{D\}=\\\{\(p\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{N\}denote the NLI training set, where each premisepip\_\{i\}is paired with a labelyi∈𝒴y\_\{i\}\\in\\mathcal\{Y\}and𝒴=\{entail,neutral,contradict\}\\mathcal\{Y\}=\\\{\\text\{entail\},\\text\{neutral\},\\text\{contradict\}\\\}\. We denote byM\(t\)M^\{\(t\)\}the target model afterttrounds of failure\-aware adversarial data curation, withM\(0\)M^\{\(0\)\}trained on𝒟\\mathcal\{D\}\. The goal is to construct a compact adversarial training set that improves robustness without relying on large volumes of untargeted synthetic examples\. We formulate this process asfailure\-mode contextual bandit curation\. At each iteration, the system retrieves few\-shot contexts, generates candidate adversarial examples, filters candidates that induce errors in the current target model, and validates them using an automated judge ensemble\. The validated failures are then grouped into semantic failure modes\. Instead of selecting individual examples by a fixed reward threshold, a trainable stochastic policyπθ\\pi\_\{\\theta\}observes each failure mode and decides which modes should be sampled for retraining\. After the target model is updated, the policy receives a validation\-based reward and is optimized using a policy\-gradient objective\. Thus, the learning agent is the data curator, whose role is to learn which types of model failures are most useful for improving future robustness\. This formulation makes the reinforcement\-learning component explicit\. The state is a vector of statistics describing a failure mode, the action is whether to sample from that failure mode, the reward is the downstream validation gain after retraining, and the policy is updated to maximize expected validation improvement while penalizing forgetting and computational cost\. Each iteration consists of six steps: label\-balanced retrieval, LLM\-based candidate generation, failure filtering using the current target model, automated validation, clustering validated candidates into failure modes, and contextual\-bandit selection of failure modes for retraining\.

Retrieval\.For each premisepp, we construct a label\-balanced few\-shot context:𝒞p=⋃y′∈𝒴𝒞p,y′,\\mathcal\{C\}\_\{p\}=\\bigcup\_\{y^\{\\prime\}\\in\\mathcal\{Y\}\}\\mathcal\{C\}\_\{p,y^\{\\prime\}\},where𝒞p,y′\\mathcal\{C\}\_\{p,y^\{\\prime\}\}containskkexamples with labely′y^\{\\prime\}\. Let𝒟y′\\mathcal\{D\}\_\{y^\{\\prime\}\}denote the subset of training examples with labely′y^\{\\prime\}\. Label\-balanced retrieval prevents the prompt from being dominated by a single class and provides the generator with controlled examples from all NLI relations\.

Semantic Retrieval\.LetEembE\_\{\\mathrm\{emb\}\}denote the embedding model\. Each premisexxis embedded as:ex=Eemb​\(x\)∈ℝd\.e\_\{x\}=E\_\{\\mathrm\{emb\}\}\(x\)\\in\\mathbb\{R\}^\{d\}\.For a query premisepp, we compute:ep=Eemb​\(p\)\.e\_\{p\}=E\_\{\\mathrm\{emb\}\}\(p\)\.For each labely′y^\{\\prime\}, semantic neighbors are selected by:𝒞p,y′sem=arg⁡max⁡∑x∈SS⊆𝒟y′\|S\|=k⁡cos⁡\(ep,ex\)\.\\mathcal\{C\}\_\{p,y^\{\\prime\}\}^\{\\mathrm\{sem\}\}=\\arg\\max\_\{\\begin\{subarray\}\{c\}S\\subseteq\\mathcal\{D\}\_\{y^\{\\prime\}\}\\\\ \|S\|=k\\end\{subarray\}\}\\sum\_\{x\\in S\}\\cos\(e\_\{p\},e\_\{x\}\)\.This corresponds to selecting the top\-kknearest neighbors in embedding space within each label group\.

Lexical Retrieval\.We index all premises using BM25 with parameters\(k1=1\.5,b=0\.75\)\(k\_\{1\}=1\.5,b=0\.75\)and define the lexical relevance score:sBM25​\(p,x\)=∑w∈pIDF⁡\(w\)⋅tf​\(w,x\)​\(k1\+1\)tf⁡\(w,x\)\+k1​\(1−b\+b​\|x\|avgdl\)\.s\_\{\\mathrm\{BM25\}\}\(p,x\)=\\sum\_\{w\\in p\}\\mathrm\{IDF\}\(w\)\\cdot\\frac\{\\mathrm\{tf\}\(w,x\)\(k\_\{1\}\+1\)\}\{\\mathrm\{tf\}\(w,x\)\+k\_\{1\}\\left\(1\-b\+b\\frac\{\|x\|\}\{\\mathrm\{avgdl\}\}\\right\)\}\.For each labely′y^\{\\prime\}, lexical neighbors are selected as:𝒞p,y′lex=arg⁡max⁡∑x∈SS⊆𝒟y′\|S\|=k⁡sBM25​\(p,x\)\.\\mathcal\{C\}\_\{p,y^\{\\prime\}\}^\{\\mathrm\{lex\}\}=\\arg\\max\_\{\\begin\{subarray\}\{c\}S\\subseteq\\mathcal\{D\}\_\{y^\{\\prime\}\}\\\\ \|S\|=k\\end\{subarray\}\}\\sum\_\{x\\in S\}s\_\{\\mathrm\{BM25\}\}\(p,x\)\.

Hybrid Retrieval\.To integrate semantic and lexical signals, for each candidatexxand querypp, we compute normalized scores:s~sem​\(p,x\)=cos⁡\(Eemb​\(p\),Eemb​\(x\)\)−μsemσsem,\\tilde\{s\}\_\{\\mathrm\{sem\}\}\(p,x\)=\\frac\{\\cos\\big\(E\_\{\\mathrm\{emb\}\}\(p\),E\_\{\\mathrm\{emb\}\}\(x\)\\big\)\-\\mu\_\{\\mathrm\{sem\}\}\}\{\\sigma\_\{\\mathrm\{sem\}\}\},and:s~lex​\(p,x\)=sBM25​\(p,x\)−μlexσlex\.\\tilde\{s\}\_\{\\mathrm\{lex\}\}\(p,x\)=\\frac\{s\_\{\\mathrm\{BM25\}\}\(p,x\)\-\\mu\_\{\\mathrm\{lex\}\}\}\{\\sigma\_\{\\mathrm\{lex\}\}\}\.The hybrid relevance score is defined as:scomb​\(p,x\)=α​s~sem​\(p,x\)\+\(1−α\)​s~lex​\(p,x\),s\_\{\\mathrm\{comb\}\}\(p,x\)=\\alpha\\tilde\{s\}\_\{\\mathrm\{sem\}\}\(p,x\)\+\(1\-\\alpha\)\\tilde\{s\}\_\{\\mathrm\{lex\}\}\(p,x\),whereα∈\[0,1\]\\alpha\\in\[0,1\]controls the interpolation between semantic and lexical similarity\. For each labely′y^\{\\prime\}, we select:𝒞p,y′comb=arg⁡max⁡∑x∈SS⊆𝒟y′\|S\|=k⁡scomb​\(p,x\)\.\\mathcal\{C\}\_\{p,y^\{\\prime\}\}^\{\\mathrm\{comb\}\}=\\arg\\max\_\{\\begin\{subarray\}\{c\}S\\subseteq\\mathcal\{D\}\_\{y^\{\\prime\}\}\\\\ \|S\|=k\\end\{subarray\}\}\\sum\_\{x\\in S\}s\_\{\\mathrm\{comb\}\}\(p,x\)\.The final context is:𝒞p=⋃y′∈𝒴𝒞p,y′m,\\mathcal\{C\}\_\{p\}=\\bigcup\_\{y^\{\\prime\}\\in\\mathcal\{Y\}\}\\mathcal\{C\}\_\{p,y^\{\\prime\}\}^\{m\},wherem∈\{sem,lex,comb\}m\\in\\\{\\mathrm\{sem\},\\mathrm\{lex\},\\mathrm\{comb\}\\\}denotes the retrieval mode\. The complete retrieval is summarized in Algorithm[1](https://arxiv.org/html/2608.18681#alg1)\.

Algorithm 1Balanced Few\-Shot Context RetrievalInput: Premisepp, label\-partitioned dataset\{𝒟y\}y∈𝒴\\\{\\mathcal\{D\}\_\{y\}\\\}\_\{y\\in\\mathcal\{Y\}\} Parameter: examples per labelkk, retrieval modem∈\{sem,lex,comb\}m\\in\\\{\\mathrm\{sem\},\\mathrm\{lex\},\\mathrm\{comb\}\\\} Output: Few\-shot context𝒞p\\mathcal\{C\}\_\{p\}

1:

𝒞p←∅\\mathcal\{C\}\_\{p\}\\leftarrow\\emptyset
2:Compute retrieval scores

sm​\(p,x\)s\_\{m\}\(p,x\)for all

x∈𝒟x\\in\\mathcal\{D\}
3:foreach label

y′∈𝒴y^\{\\prime\}\\in\\mathcal\{Y\}do

4:

𝒞p,y′←arg⁡maxx∈𝒟y′k​sm​\(p,x\)\\mathcal\{C\}\_\{p,y^\{\\prime\}\}\\leftarrow\\arg\\max\\nolimits^\{k\}\_\{x\\in\\mathcal\{D\}\_\{y^\{\\prime\}\}\}s\_\{m\}\(p,x\)
5:

𝒞p←𝒞p∪𝒞p,y′\\mathcal\{C\}\_\{p\}\\leftarrow\\mathcal\{C\}\_\{p\}\\cup\\mathcal\{C\}\_\{p,y^\{\\prime\}\}
6:endfor

7:return

𝒞p\\mathcal\{C\}\_\{p\}

Task\-Specific Candidate Generation\.Given an inputxx, its labelyy, and the retrieved context𝒞x\\mathcal\{C\}\_\{x\}, we employ a large language model to sample task\-specific candidate outputs from:

o∼PLLM​\(o∣x,𝒞x,y\)\.o\\sim P\_\{\\mathrm\{LLM\}\}\(o\\mid x,\\mathcal\{C\}\_\{x\},y\)\.\(1\)This stochastic generation process produces a candidate set𝒪x\(t\)\\mathcal\{O\}\_\{x\}^\{\(t\)\}for each input at iterationtt, whereoodenotes a task\-dependent output, such as a hypothesis for NLI or an evidence claim for fact verification\.

Failure\-Based Filtering\.Each generated candidateo∈𝒪x\(t\)o\\in\\mathcal\{O\}\_\{x\}^\{\(t\)\}is evaluated by the current target modelM\(t\)M^\{\(t\)\}\. Let the predicted label be:

y^o=arg⁡maxy′∈𝒴​M\(t\)​\(y′∣x,o\)\.\\hat\{y\}\_\{o\}=\\arg\\max\_\{y^\{\\prime\}\\in\\mathcal\{Y\}\}M^\{\(t\)\}\(y^\{\\prime\}\\mid x,o\)\.\(2\)Candidates that are correctly classified are discarded, and only misclassified instances are retained:

𝒪xfail=\{o∈𝒪x\(t\)∣y^o≠y\}\.\\mathcal\{O\}\_\{x\}^\{\\mathrm\{fail\}\}=\\left\\\{o\\in\\mathcal\{O\}\_\{x\}^\{\(t\)\}\\mid\\hat\{y\}\_\{o\}\\neq y\\right\\\}\.\(3\)This step focuses the remaining pipeline on examples that expose the current model’s weaknesses\.

Automated Validation\.Let the candidate adversarial triples be:

𝒬\(t\)=\{\(x,o,y\)∣o∈𝒪xfail\}\.\\mathcal\{Q\}^\{\(t\)\}=\\left\\\{\(x,o,y\)\\mid o\\in\\mathcal\{O\}\_\{x\}^\{\\mathrm\{fail\}\}\\right\\\}\.\(4\)Each triple is evaluated by an ensemble of automated judge models\. Let the predicted label of judgejjbe:

vj​\(x,o\)=Mj​\(x,o\),j∈\{1,2,3\}\.v\_\{j\}\(x,o\)=M\_\{j\}\(x,o\),\\qquad j\\in\\\{1,2,3\\\}\.\(5\)A candidate is retained only if all judges agree with the original label:∑j=13𝕀\[vj\(x,o\)=y\]=3\.\\sum\_\{j=1\}^\{3\}\\mathbb\{I\}\\left\[v\_\{j\}\(x,o\)=y\\right\]=3\.The validated failure pool is therefore:𝒱\(t\)=\{\(x,o,y\)∈𝒬\(t\)∣∑j=13𝕀\[vj\(x,o\)=y\]=3\}\.\\mathcal\{V\}^\{\(t\)\}=\\left\\\{\(x,o,y\)\\in\\mathcal\{Q\}^\{\(t\)\}\\mid\\sum\_\{j=1\}^\{3\}\\mathbb\{I\}\\left\[v\_\{j\}\(x,o\)=y\\right\]=3\\right\\\}\.This unanimity constraint reduces label noise and prevents the policy from learning from corrupted reward signals\.

Failure\-Mode Construction\.Rather than selecting individual failures independently, we group validated failures into failure modes\. For each validated tripleqi=\(xi,oi,yi\)∈𝒱\(t\)q\_\{i\}=\(x\_\{i\},o\_\{i\},y\_\{i\}\)\\in\\mathcal\{V\}^\{\(t\)\}, we compute an embedding:

gi=Efail​\(\[xi;oi;yi\]\),g\_\{i\}=E\_\{\\mathrm\{fail\}\}\\left\(\[x\_\{i\};o\_\{i\};y\_\{i\}\]\\right\),\(6\)whereEfailE\_\{\\mathrm\{fail\}\}may be the same text embedding model used for retrieval or a separate sentence encoder\. We then cluster the validated failure pool:

\{ℱ1\(t\),…,ℱKt\(t\)\}=Cluster⁡\(\{gi\}qi∈𝒱\(t\)\)\.\\left\\\{\\mathcal\{F\}^\{\(t\)\}\_\{1\},\\ldots,\\mathcal\{F\}^\{\(t\)\}\_\{K\_\{t\}\}\\right\\\}=\\mathrm\{Cluster\}\\left\(\\\{g\_\{i\}\\\}\_\{q\_\{i\}\\in\\mathcal\{V\}^\{\(t\)\}\}\\right\)\.\(7\)Each clusterℱk\(t\)\\mathcal\{F\}^\{\(t\)\}\_\{k\}represents a failure mode, such as lexical shortcut failures, negation errors, entity mismatch errors, numerical reasoning failures, or contradiction confusion\. The clustering is unsupervised and does not require human failure labels\.

Bandit State\.For each failure modeℱk\(t\)\\mathcal\{F\}^\{\(t\)\}\_\{k\}, we construct a state vectorzt,kz\_\{t,k\}containing normalized statistics of that cluster:

zt,k=\[log⁡\(\|ℱk\(t\)\|\+1\),ℓ¯t,k,H¯t,k,μ¯t,k,histt,k​\(y\),s¯t,kretr,a¯t,kjudge,νt,k,G¯t−1,k\]\.z\_\{t,k\}=\\left\[\\log\(\|\\mathcal\{F\}^\{\(t\)\}\_\{k\}\|\+1\),\\bar\{\\ell\}\_\{t,k\},\\bar\{H\}\_\{t,k\},\\bar\{\\mu\}\_\{t,k\},\\mathrm\{hist\}\_\{t,k\}\(y\),\\bar\{s\}\_\{t,k\}^\{\\mathrm\{retr\}\},\\bar\{a\}\_\{t,k\}^\{\\mathrm\{judge\}\},\\nu\_\{t,k\},\\bar\{G\}\_\{t\-1,k\}\\right\]\.\(8\)Here,ℓ¯t,k\\bar\{\\ell\}\_\{t,k\}is the mean target\-model loss on the cluster,H¯t,k\\bar\{H\}\_\{t,k\}is the mean predictive entropy,μ¯t,k\\bar\{\\mu\}\_\{t,k\}is the mean classification margin,histt,k​\(y\)\\mathrm\{hist\}\_\{t,k\}\(y\)is the label distribution,s¯t,kretr\\bar\{s\}\_\{t,k\}^\{\\mathrm\{retr\}\}is the average retrieval score,a¯t,kjudge\\bar\{a\}\_\{t,k\}^\{\\mathrm\{judge\}\}is the average judge agreement,νt,k\\nu\_\{t,k\}is a novelty score measuring distance from previously selected failure modes, andG¯t−1,k\\bar\{G\}\_\{t\-1,k\}is the previous moving\-average reward associated with similar clusters\. This state summarizes both the difficulty and the diversity of each failure mode\.

Contextual\-Bandit Policy\.The policyπθ\\pi\_\{\\theta\}is a trainable stochastic selector over failure modes\. For each cluster statezt,kz\_\{t,k\}, the policy outputs a Bernoulli distribution:

πθ​\(at,k=1∣zt,k\)=σ⁡\(fθ​\(zt,k\)\),\\pi\_\{\\theta\}\(a\_\{t,k\}=1\\mid z\_\{t,k\}\)=\\sigma\\left\(f\_\{\\theta\}\(z\_\{t,k\}\)\\right\),\(9\)whereat,k∈\{0,1\}a\_\{t,k\}\\in\\\{0,1\\\}is the action for clusterkk,at,k=1a\_\{t,k\}=1means selecting that failure mode for retraining, andfθf\_\{\\theta\}is a small neural network\. The complete action at iterationttis:

at=\(at,1,…,at,Kt\)\.a\_\{t\}=\(a\_\{t,1\},\\ldots,a\_\{t,K\_\{t\}\}\)\.\(10\)To encourage exploration, actions are sampled fromπθ\\pi\_\{\\theta\}during training rather than selected deterministically\. At evaluation time, the policy may use greedy selection according to the learned probabilities\.

Given an adversarial budgetBadvB\_\{\\mathrm\{adv\}\}, the selected adversarial set is:

𝒟sel\(t\)=⋃k:at,k=1Sample\(ℱk\(t\),nt,k\),\\mathcal\{D\}\_\{\\mathrm\{sel\}\}^\{\(t\)\}=\\bigcup\_\{k:a\_\{t,k\}=1\}\\mathrm\{Sample\}\\left\(\\mathcal\{F\}^\{\(t\)\}\_\{k\},n\_\{t,k\}\\right\),\(11\)where:

nt,k=⌊Badv​at,k​\|ℱk\(t\)\|∑ℓ=1Ktat,ℓ​\|ℱℓ\(t\)\|⌋\.n\_\{t,k\}=\\left\\lfloor B\_\{\\mathrm\{adv\}\}\\frac\{a\_\{t,k\}\|\\mathcal\{F\}^\{\(t\)\}\_\{k\}\|\}\{\\sum\_\{\\ell=1\}^\{K\_\{t\}\}a\_\{t,\\ell\}\|\\mathcal\{F\}^\{\(t\)\}\_\{\\ell\}\|\}\\right\\rfloor\.\(12\)If no cluster is selected, the cluster with the highest policy probability is selected as a fallback\. This avoids empty updates\.

Target Model Retraining\.The target model is updated using supervised learning on a mixture of original and selected adversarial examples:

M\(t\+1\)←Train⁡\(M\(t\),𝒟mix\(t\)\)\.M^\{\(t\+1\)\}\\leftarrow\\mathrm\{Train\}\\left\(M^\{\(t\)\},\\mathcal\{D\}\_\{\\mathrm\{mix\}\}^\{\(t\)\}\\right\)\.\(13\)The construction of𝒟mix\(t\)\\mathcal\{D\}\_\{\\mathrm\{mix\}\}^\{\(t\)\}is described in Section[3\.2](https://arxiv.org/html/2608.18681#S3.SS2)\. This update changes the environment observed by the curator, since future failure pools depend on the updated target modelM\(t\+1\)M^\{\(t\+1\)\}\.

Validation\-Based Reward\.After retraining, the policy receives a scalar reward based only on validation performance\. Let𝒟valrob\\mathcal\{D\}\_\{\\mathrm\{val\}\}^\{\\mathrm\{rob\}\}denote the robustness validation set and𝒟valclean\\mathcal\{D\}\_\{\\mathrm\{val\}\}^\{\\mathrm\{clean\}\}denote the clean validation set\. We define:

Gt=Δrob\(t\)−βf​Δforget\(t\)−βc​Δcost\(t\)\.G\_\{t\}=\\Delta\_\{\\mathrm\{rob\}\}^\{\(t\)\}\-\\beta\_\{f\}\\Delta\_\{\\mathrm\{forget\}\}^\{\(t\)\}\-\\beta\_\{c\}\\Delta\_\{\\mathrm\{cost\}\}^\{\(t\)\}\.\(14\)The robustness gain is:

Δrob\(t\)=Perf⁡\(M\(t\+1\),𝒟valrob\)−Perf⁡\(M\(t\),𝒟valrob\)\.\\Delta\_\{\\mathrm\{rob\}\}^\{\(t\)\}=\\mathrm\{Perf\}\\left\(M^\{\(t\+1\)\},\\mathcal\{D\}\_\{\\mathrm\{val\}\}^\{\\mathrm\{rob\}\}\\right\)\-\\mathrm\{Perf\}\\left\(M^\{\(t\)\},\\mathcal\{D\}\_\{\\mathrm\{val\}\}^\{\\mathrm\{rob\}\}\\right\)\.\(15\)The forgetting penalty is:

Δforget\(t\)=max⁡\(0,Perf⁡\(M\(t\),𝒟valclean\)−Perf⁡\(M\(t\+1\),𝒟valclean\)\)\.\\Delta\_\{\\mathrm\{forget\}\}^\{\(t\)\}=\\max\\left\(0,\\mathrm\{Perf\}\\left\(M^\{\(t\)\},\\mathcal\{D\}\_\{\\mathrm\{val\}\}^\{\\mathrm\{clean\}\}\\right\)\-\\mathrm\{Perf\}\\left\(M^\{\(t\+1\)\},\\mathcal\{D\}\_\{\\mathrm\{val\}\}^\{\\mathrm\{clean\}\}\\right\)\\right\)\.\(16\)The cost penalty is:

Δcost\(t\)=\|𝒟sel\(t\)\|Badv\.\\Delta\_\{\\mathrm\{cost\}\}^\{\(t\)\}=\\frac\{\|\\mathcal\{D\}\_\{\\mathrm\{sel\}\}^\{\(t\)\}\|\}\{B\_\{\\mathrm\{adv\}\}\}\.\(17\)The first term rewards robustness gains, the second penalizes degradation on the original distribution, and the third penalizes excessive adversarial data usage\. The coefficientsβf\\beta\_\{f\}andβc\\beta\_\{c\}control the trade\-off between robustness, retention, and efficiency\.

Policy Optimization\.The curator policy is optimized to maximize expected validation reward:

J⁡\(θ\)=𝔼at∼πθ​\[Gt\]\.J\(\\theta\)=\\mathbb\{E\}\_\{a\_\{t\}\\sim\\pi\_\{\\theta\}\}\\left\[G\_\{t\}\\right\]\.\(18\)We updateπθ\\pi\_\{\\theta\}with a Reinforce\-style policy\-gradient objective:

ℒπ\(θ\)=−\(Gt−bt\)∑k=1Ktlogπθ\(at,k∣zt,k\)−βH∑k=1Ktℋ\(πθ\(⋅∣zt,k\)\),\\mathcal\{L\}\_\{\\pi\}\(\\theta\)=\-\(G\_\{t\}\-b\_\{t\}\)\\sum\_\{k=1\}^\{K\_\{t\}\}\\log\\pi\_\{\\theta\}\(a\_\{t,k\}\\mid z\_\{t,k\}\)\-\\beta\_\{H\}\\sum\_\{k=1\}^\{K\_\{t\}\}\\mathcal\{H\}\\left\(\\pi\_\{\\theta\}\(\\cdot\\mid z\_\{t,k\}\)\\right\),\(19\)wherebtb\_\{t\}is a moving\-average baseline andℋ⁡\(⋅\)\\mathcal\{H\}\(\\cdot\)is an entropy regularizer that encourages exploration\. The baseline is updated as:

bt=ρ​bt−1\+\(1−ρ\)​Gt\.b\_\{t\}=\\rho b\_\{t\-1\}\+\(1\-\\rho\)G\_\{t\}\.\(20\)
We also maintain a criticRϕR\_\{\\phi\}that predicts the expected return of a failure mode:

Rϕ​\(zt,k\)≈𝔼⁡\[Gt∣zt,k\]\.R\_\{\\phi\}\(z\_\{t,k\}\)\\approx\\mathbb\{E\}\\left\[G\_\{t\}\\mid z\_\{t,k\}\\right\]\.\(21\)The critic is trained by minimizing:

ℒR\(ϕ\)=∑k:at,k=1\(Rϕ\(zt,k\)−Gt\)2\.\\mathcal\{L\}\_\{R\}\(\\phi\)=\\sum\_\{k:a\_\{t,k\}=1\}\\left\(R\_\{\\phi\}\(z\_\{t,k\}\)\-G\_\{t\}\\right\)^\{2\}\.\(22\)Unlike a threshold\-based filtering rule,RϕR\_\{\\phi\}is not used as a direct keep\-or\-discard mechanism\. Instead, it serves as a learned utility estimator and variance\-reduction signal for policy learning\. The actual data\-selection behavior is governed by the trainable policyπθ\\pi\_\{\\theta\}\. The complete procedure is summarized in Algorithm[2](https://arxiv.org/html/2608.18681#alg2)\.

Algorithm 2Failure\-Mode Contextual Bandit CurationInput: Training set𝒟\\mathcal\{D\}, validation sets𝒟valrob\\mathcal\{D\}\_\{\\mathrm\{val\}\}^\{\\mathrm\{rob\}\},𝒟valclean\\mathcal\{D\}\_\{\\mathrm\{val\}\}^\{\\mathrm\{clean\}\}, iterationsTT, retrieval modemm, budgetBadvB\_\{\\mathrm\{adv\}\} Output: Enhanced modelM\(T\)M^\{\(T\)\}

1:Train

M\(0\)M^\{\(0\)\}on

𝒟\\mathcal\{D\}
2:Initialize policy

πθ\\pi\_\{\\theta\}, critic

RϕR\_\{\\phi\}, and baseline

b0b\_\{0\}
3:for

t=0t=0to

T−1T\-1do

4:

𝒱\(t\)←GENERATE​\_​AND​\_​VALIDATE​\(M\(t\),𝒟,m\)\\mathcal\{V\}^\{\(t\)\}\\leftarrow\\mathrm\{GENERATE\\\_AND\\\_VALIDATE\}\(M^\{\(t\)\},\\mathcal\{D\},m\)
5:

\{ℱk\(t\)\}k=1Kt←CLUSTER​\_​FAILURES​\(𝒱\(t\)\)\\\{\\mathcal\{F\}^\{\(t\)\}\_\{k\}\\\}\_\{k=1\}^\{K\_\{t\}\}\\leftarrow\\mathrm\{CLUSTER\\\_FAILURES\}\(\\mathcal\{V\}^\{\(t\)\}\)
6:foreach failure mode

ℱk\(t\)\\mathcal\{F\}^\{\(t\)\}\_\{k\}do

7:construct state

zt,kz\_\{t,k\}and sample

at,k∼πθ\(⋅∣zt,k\)a\_\{t,k\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid z\_\{t,k\}\)
8:endfor

9:

𝒟sel\(t\)←BUDGETED​\_​SAMPLE​\(\{ℱk\(t\):at,k=1\},Badv\)\\mathcal\{D\}\_\{\\mathrm\{sel\}\}^\{\(t\)\}\\leftarrow\\mathrm\{BUDGETED\\\_SAMPLE\}\(\\\{\\mathcal\{F\}^\{\(t\)\}\_\{k\}:a\_\{t,k\}=1\\\},B\_\{\\mathrm\{adv\}\}\)
10:

𝒟mix\(t\)←MIX⁡\(𝒟,𝒟sel\(t\)\)\\mathcal\{D\}\_\{\\mathrm\{mix\}\}^\{\(t\)\}\\leftarrow\\mathrm\{MIX\}\(\\mathcal\{D\},\\mathcal\{D\}\_\{\\mathrm\{sel\}\}^\{\(t\)\}\)
11:

M\(t\+1\)←Train⁡\(M\(t\),𝒟mix\(t\)\)M^\{\(t\+1\)\}\\leftarrow\\mathrm\{Train\}\(M^\{\(t\)\},\\mathcal\{D\}\_\{\\mathrm\{mix\}\}^\{\(t\)\}\)
12:compute reward

GtG\_\{t\}on

𝒟valrob\\mathcal\{D\}\_\{\\mathrm\{val\}\}^\{\\mathrm\{rob\}\}and

𝒟valclean\\mathcal\{D\}\_\{\\mathrm\{val\}\}^\{\\mathrm\{clean\}\}
13:update

πθ\\pi\_\{\\theta\},

RϕR\_\{\\phi\}, and

btb\_\{t\}using

GtG\_\{t\}
14:endfor

15:return

M\(T\)M^\{\(T\)\}

Critic Architecture and Reward Supervision\.The reward modelRϕR\_\{\\phi\}is implemented as a lightweight MLP critic operating on failure\-mode states rather than individual examples\. Its input is the cluster\-level state vectorzt,kz\_\{t,k\}defined above, which contains the cluster size, mean target\-model loss, entropy, classification margin, label distribution, retrieval score, judge agreement, novelty score, and previous reward statistics\. The critic outputs a scalar utility estimate:

Rϕ​\(zt,k\)∈ℝ,R\_\{\\phi\}\(z\_\{t,k\}\)\\in\\mathbb\{R\},\(23\)
which approximates the expected validation reward obtained by selecting failure modeℱk\(t\)\\mathcal\{F\}^\{\(t\)\}\_\{k\}\.

Importantly, we do not compute a separate utility valueΔ⁡\(h\)\\Delta\(h\)for each individual candidate, since doing so would require prohibitively expensive per\-example retraining\. Instead, reward supervision is defined at the failure\-mode level\. After the policy selects a subset of failure modes, the target model is retrained once on the resulting adversarial mixture, and the scalar validation rewardGtG\_\{t\}is computed from robustness improvement, forgetting penalty, and data cost\. This same observed return is assigned to all selected failure modes in that round and used to train the critic:

ℒR\(ϕ\)=∑k:at,k=1\(Rϕ\(zt,k\)−Gt\)2\.\\mathcal\{L\}\_\{R\}\(\\phi\)=\\sum\_\{k:a\_\{t,k\}=1\}\\left\(R\_\{\\phi\}\(z\_\{t,k\}\)\-G\_\{t\}\\right\)^\{2\}\.\(24\)
The critic is updated once after each retraining round, together with the policy update\. Since selection is performed by the stochastic policyπθ​\(at,k∣zt,k\)\\pi\_\{\\theta\}\(a\_\{t,k\}\\mid z\_\{t,k\}\), the method does not require a fixed reward thresholdτ\\tau\. This removes the threshold\-tuning step used in reward\-filtering pipelines and replaces it with validation\-driven policy optimization\.

### 3\.1Hyperparameter Tuning for Retrieval

Standard training onPPmay reinforce spurious correlations that fail under distribution shift\. Our framework instead concentrates updates on validated failure modes, where such shortcuts are more likely to break and task\-relevant reasoning is required\. Lemma 3\.1 formalizes this intuition by showing that, under stated assumptions, failure\-focused sampling reduces shortcut\-aligned gradient contributions while increasing semantically meaningful ones\. To initialize the hybrid retrieval score, we tune the semantic weightα\\alphaon 1,000 SNLI training examples using BGE M3\. For calibration only, we retrieve a candidate pool of 9 examples per label and treat each pair\(p,x\)\(p,x\)as relevant iflabel⁡\(x\)=label⁡\(p\)\\mathrm\{label\}\(x\)=\\mathrm\{label\}\(p\)\. The hybrid score is:scomb​\(p,x\)=α​s~sem​\(p,x\)\+\(1−α\)​s~lex​\(p,x\)\.s\_\{\\mathrm\{comb\}\}\(p,x\)=\\alpha\\tilde\{s\}\_\{\\mathrm\{sem\}\}\(p,x\)\+\(1\-\\alpha\)\\tilde\{s\}\_\{\\mathrm\{lex\}\}\(p,x\)\.We searchα∈\{0,0\.01,…,1\.0\}\\alpha\\in\\\{0,0\.01,\\ldots,1\.0\\\}and select the value with the highest ROC AUC over positive and negative pairs\. The best value isα∗=0\.83\\alpha^\{\*\}=0\.83, achieving an AUC of 0\.93 in Figure[4](https://arxiv.org/html/2608.18681#S3.F4)\. We fixα=0\.83\\alpha=0\.83for all downstream experiments\. In the main six\-shot setting, we usek=2k=2examples per label\. This tuning only initializes retrieval; later adaptation is performed by the contextual\-bandit failure\-mode policy\.

Figure 3:ROC AUC as a function of the semantic\-lexical weighting parameterα\\alpha\.Figure 4:ROC curve at optimalα=0\.83\\alpha=0\.83\.

### 3\.2Avoiding Forgetting

Training only on adversarial examples can induce a non\-stationary training distribution and lead to catastrophic forgetting, where performance on the original data distribution deteriorates\. In our setting, this risk is especially important because the policy is explicitly encouraged to focus on difficult failure modes\. To stabilize training, we mix original and selected adversarial examples during retraining\. Let𝒟orig\\mathcal\{D\}\_\{\\mathrm\{orig\}\}denote the original training set and𝒟sel\(t\)\\mathcal\{D\}\_\{\\mathrm\{sel\}\}^\{\(t\)\}denote the adversarial examples selected by the policy at iterationtt\. We define the original\-to\-adversarial mixing ratio:

λmix=\|𝒟orig\|\|𝒟sel\(t\)\|∈\{0,1,12,13,14\}\.\\lambda\_\{\\mathrm\{mix\}\}=\\frac\{\|\\mathcal\{D\}\_\{\\mathrm\{orig\}\}\|\}\{\|\\mathcal\{D\}\_\{\\mathrm\{sel\}\}^\{\(t\)\}\|\}\\in\\left\\\{0,1,\\frac\{1\}\{2\},\\frac\{1\}\{3\},\\frac\{1\}\{4\}\\right\\\}\.\(25\)Here,λmix=0\\lambda\_\{\\mathrm\{mix\}\}=0corresponds to training only on selected adversarial examples, whileλmix=14\\lambda\_\{\\mathrm\{mix\}\}=\\frac\{1\}\{4\}denotes one original example per four adversarial examples\. Forλmix\>0\\lambda\_\{\\mathrm\{mix\}\}\>0, we construct:

𝒟mix\(t\)​\(λmix\)=𝒟orig∪Sample⁡\(𝒟sel\(t\),⌊λmix−1​\|𝒟orig\|⌋\)\.\\mathcal\{D\}\_\{\\mathrm\{mix\}\}^\{\(t\)\}\(\\lambda\_\{\\mathrm\{mix\}\}\)=\\mathcal\{D\}\_\{\\mathrm\{orig\}\}\\cup\\mathrm\{Sample\}\\left\(\\mathcal\{D\}\_\{\\mathrm\{sel\}\}^\{\(t\)\},\\left\\lfloor\\lambda\_\{\\mathrm\{mix\}\}^\{\-1\}\|\\mathcal\{D\}\_\{\\mathrm\{orig\}\}\|\\right\\rfloor\\right\)\.\(26\)
Forλmix=0\\lambda\_\{\\mathrm\{mix\}\}=0, we set:𝒟mix\(t\)​\(0\)=𝒟sel\(t\)\.\\mathcal\{D\}\_\{\\mathrm\{mix\}\}^\{\(t\)\}\(0\)=\\mathcal\{D\}\_\{\\mathrm\{sel\}\}^\{\(t\)\}\.For each retrieval modem∈\{sem,lex,comb\}m\\in\\\{\\mathrm\{sem\},\\mathrm\{lex\},\\mathrm\{comb\}\\\}, the target model is optimized forTTiterations and evaluated as:

Am​\(λmix\)=Perf⁡\(M\(T\)∣𝒟mix\(t\)​\(λmix\)\),A\_\{m\}\(\\lambda\_\{\\mathrm\{mix\}\}\)=\\mathrm\{Perf\}\\left\(M^\{\(T\)\}\\mid\\mathcal\{D\}\_\{\\mathrm\{mix\}\}^\{\(t\)\}\(\\lambda\_\{\\mathrm\{mix\}\}\)\\right\),\(27\)wherePerf⁡\(⋅\)\\mathrm\{Perf\}\(\\cdot\)denotes the task\-specific evaluation metric\.

Figure[5](https://arxiv.org/html/2608.18681#S3.F5)reportsAsem​\(λmix\)A\_\{\\mathrm\{sem\}\}\(\\lambda\_\{\\mathrm\{mix\}\}\),Alex​\(λmix\)A\_\{\\mathrm\{lex\}\}\(\\lambda\_\{\\mathrm\{mix\}\}\), andAcomb​\(λmix\)A\_\{\\mathrm\{comb\}\}\(\\lambda\_\{\\mathrm\{mix\}\}\)as functions of the mixing ratio\. All three curves improve substantially when moderate original\-data mixing is introduced\. The hybrid retrieval strategy achieves the strongest performance nearλmix=14\\lambda\_\{\\mathrm\{mix\}\}=\\frac\{1\}\{4\}, indicating that semantic and lexical retrieval are most effective when policy\-selected adversarial failures are balanced with sufficient original\-distribution coverage\. This setting provides the best trade\-off between robustness improvement and forgetting prevention\.

Figure 5:Task performanceAm​\(λmix\)A\_\{m\}\(\\lambda\_\{\\mathrm\{mix\}\}\)versus the mixing ratio of selected adversarial examples to original training data\.We therefore useλmix∗=14\\lambda\_\{\\mathrm\{mix\}\}^\{\*\}=\\frac\{1\}\{4\}in the main experiments\. This controlled mixing mitigates catastrophic forgetting while preserving the benefit of failure\-aware adversarial training\.

##### Theoretical interpretation\.

Our method changes the effective training distribution by mixing the original data distributionPPwith a policy\-induced distribution over validated failure modesQ^tπ\\hat\{Q\}\_\{t\}^\{\\pi\}:

Ptλ=\(1−λ\)​P\+λ​Q^tπ\.P\_\{t\}^\{\\lambda\}=\(1\-\\lambda\)P\+\\lambda\\hat\{Q\}\_\{t\}^\{\\pi\}\.\(28\)
This view explains why failure\-mode curation can reduce reliance on spurious shortcuts: selected failures are examples where the current model’s decision rule breaks, so training on them increases the relative contribution of task\-relevant gradients\. Under the assumptions stated in Appendix[B](https://arxiv.org/html/2608.18681#A2), Lemma[B\.1](https://arxiv.org/html/2608.18681#A2.SS1.SSS0.Px3)shows that failure\-mode sampling reduces shortcut\-aligned gradient contributions while preserving core\-feature contributions\. Propositions[B\.2](https://arxiv.org/html/2608.18681#A2.SS2.SSS0.Px1)and[B\.2](https://arxiv.org/html/2608.18681#A2.SS2.SSS0.Px4)further show that the mixture update induces bounded distributional drift and that bounded reward noise causes bounded distortion in the induced sampling policy\.

## 4Evaluation and Results

We evaluate the proposed failure\-mode contextual bandit curation pipeline on standard benchmarks for natural language inference and fact verification\. All experiments use automatically generated and automatically validated adversarial examples, without additional human annotation\.

Target NLI Model\.For NLI experiments, the target model isRoBERTa\-base\-SNLI\(125M parameters\)\([10](https://arxiv.org/html/2608.18681#bib.bib6)\), a RoBERTa\-base model fine\-tuned on SNLI\. This model serves as the classifier whose failures are mined, clustered into failure modes, and used for adaptive adversarial retraining\.

Generation LLM\.Adversarial hypotheses are generated usingLLaMA\-4\-Scout\-17B\-16E\-Instruct\([24](https://arxiv.org/html/2608.18681#bib.bib15)\)\. For each input, the generator is conditioned on a label\-balanced retrieved context constructed using semantic retrieval, lexical retrieval, or hybrid BGE\+BM25 retrieval\.

Validation LLMs\.Each generated candidate is automatically validated by an ensemble of three instruction\-tuned judge models:Gemma\-3\-27B\-IT\([8](https://arxiv.org/html/2608.18681#bib.bib16)\),Phi\-4\([25](https://arxiv.org/html/2608.18681#bib.bib17)\), andQwen3\-32B\([29](https://arxiv.org/html/2608.18681#bib.bib18)\)\. A candidate is retained only when all judges agree with the intended gold label\. This validation stage is used to reduce label noise before failure\-mode clustering and policy selection\.

Bandit Policy and Critic\.The data curator is implemented as a contextual\-bandit policyπθ\\pi\_\{\\theta\}over validated failure modes\. Each failure mode is represented by a cluster\-level state vector containing statistics such as cluster size, target\-model loss, entropy, classification margin, label distribution, retrieval score, judge agreement, novelty, and previous reward\. The criticRϕR\_\{\\phi\}is a lightweight MLP that predicts the expected validation reward of each selected failure mode\. The policy and critic are updated after each retraining round using validation\-based feedback\.

Datasets\.We report NLI results onSNLI\([2](https://arxiv.org/html/2608.18681#bib.bib3)\), the original human\-annotated inference dataset;ANLI\([27](https://arxiv.org/html/2608.18681#bib.bib1)\), which contains adversarially constructed examples collected through human\-and\-model interaction; andMultiNLI\([32](https://arxiv.org/html/2608.18681#bib.bib7)\), a multi\-genre corpus for evaluating cross\-domain\. To assess transfer beyond NLI, we also evaluate on theFEVERfact verification benchmark\.

To contextualize the gains, we compare against GNLI\([9](https://arxiv.org/html/2608.18681#bib.bib19)\), a synthetic NLI corpus of approximately 685K LLM\-generated examples\. Fine\-tuning RoBERTa\-base on GNLI alone reaches 89\.42% on SNLI, 77\.07% on ANLI, and 57\.61% on MultiNLI\. Our pipeline generates approximately 30K adversarial candidates per retrieval strategy, applies target\-model failure filtering and automated LLM validation, and retains 6637 BGE\-based and 5991 BM25\-based candidates for failure\-mode clustering and policy\-guided sampling\. With controlled adversarial mixing, our method improves RoBERTa\-base from 88\.48% to 92\.60% on SNLI, from 75\.04% to 80\.95% on ANLI, and from 54\.67% to 71\.99% on MultiNLI, as shown in Table[1](https://arxiv.org/html/2608.18681#S4.T1)\. The table also shows that unfiltered adversarial data improves performance, but automated validation and failure\-aware selection provide additional gains, indicating that robustness benefits come from prioritizing useful failure modes rather than simply adding more synthetic data\.

Table 1:Accuracy \(%\) on each test set under adversarial mixing\. Method names list onlyBGE,BM25, andBGE\+BM25, denoting their respective generation methods\. “Reward\-Guided” indicates unanimous LLM validation\.We further evaluate transfer beyond NLI on FEVER\. As shown in Table[2](https://arxiv.org/html/2608.18681#S4.T2), our method improves across model scales\. RoBERTa\-base reaches 76\.58% FEVER score and 79\.42% label accuracy, while RoBERTa\-large achieves 79\.86% FEVER score and 82\.45% accuracy\. Lightweight models such as SmolLM2\-360M and Qwen3\-0\.6B also benefit, suggesting that failure\-aware curation transfers beyond NLI\.

Table 2:Comparison on the FEVER benchmark\. We report FEVER score and label accuracy\.Table[3](https://arxiv.org/html/2608.18681#S4.T3)and Figure[6](https://arxiv.org/html/2608.18681#S4.F6)show that increasing the retrieved few\-shot context improves performance across SNLI, ANLI, and MultiNLI\. In particular, validated BGE reaches 92\.15% on SNLI, 80\.26% on ANLI, and 71\.15% on MultiNLI in the 9\-shot setting, while BM25 shows similar but slightly weaker trends\. The 6\-shot setting already matches or exceeds the strongest adversarial mixing results, highlighting the importance of retrieval quality and validated failure selection\.

Table 3:Few\-shot accuracy \(%\) of our generation methods on each set\. Columns indicate the number of few\-shot examples\.Figure 6:Few\-shot accuracy of generation methods by dataset\.
## 5Conclusion and Future Work

We presented a failure\-mode contextual bandit framework for adversarial data curation\. Instead of selecting synthetic examples with a fixed reward threshold, our method clusters validated model errors into recurring failure modes and learns which modes should be sampled for retraining\. This turns the data curator into an adaptive policy that receives validation\-based feedback and balances robustness gains, forgetting, and data cost\. Across NLI benchmarks and FEVER, the framework improves robustness while using substantially less data than large untargeted synthetic corpora\. The results show that prioritizing validated, model\-specific failure modes is more effective than simply adding more generated examples\. Future work will explore richer failure\-mode representations, uncertainty\-aware policy updates, adaptive generation budgets, and online curation settings where failures are generated and selected continuously\. We also plan to extend the framework to multilingual, domain\-specific, and broader robustness tasks, and to combine it with complementary methods such as contrastive learning, adversarial regularization, and representation\-level alignment\.

## References

- Allalet al\.\(2025\)L\. B\. Allal, A\. Lozhkov, E\. Bakouch, G\. M\. Blázquez, G\. Penedo, L\. Tunstall, A\. Marafioti, H\. Kydlíček, A\. P\. Lajarín, V\. Srivastav,et al\.SmolLM2: when smol goes big–data\-centric training of a small language model\.arXiv preprint arXiv:2502\.02737\.Cited by:[§B\.7\.1](https://arxiv.org/html/2608.18681#A2.SS7.SSS1.p1.1)\.
- Bowmanet al\.\(2015\)S\. R\. Bowman, G\. Angeli, C\. Potts, and C\. D\. ManningA large annotated corpus for learning natural language inference\.InProceedings of the 2015 Conference on Empirical Methods in Natural Language Processing,EMNLP 2015,Lisbon, Portugal,pp\. 632–642\.External Links:[Document](https://dx.doi.org/10.18653/v1/D15-1075),[Link](https://aclanthology.org/D15-1075.pdf)Cited by:[Figure 10](https://arxiv.org/html/2608.18681#A2.F10),[§B\.9](https://arxiv.org/html/2608.18681#A2.SS9.p1.1),[§1](https://arxiv.org/html/2608.18681#S1.p1.1),[§2](https://arxiv.org/html/2608.18681#S2.p1.1),[§4](https://arxiv.org/html/2608.18681#S4.p6.1)\.
- Carmonaet al\.\(2018\)V\. I\. S\. Carmona, J\. Mitchell, and S\. RiedelBehavior analysis of nli models: uncovering the influence of three factors on robustness\.arXiv preprintabs/1805\.04212\.External Links:1805\.04212,[Link](https://arxiv.org/abs/1805.04212)Cited by:[§1](https://arxiv.org/html/2608.18681#S1.p1.1),[§2](https://arxiv.org/html/2608.18681#S2.p1.1)\.
- Chenet al\.\(2024\)J\. Chen, S\. Xiao, P\. Zhang, K\. Luo, D\. Lian, and Z\. LiuBGE m3\-embedding: multi\-lingual, multi\-functionality, multi\-granularity text embeddings through self\-knowledge distillation\.arXiv preprintabs/2402\.03216\.External Links:2402\.03216,[Link](https://arxiv.org/abs/2402.03216)Cited by:[§1](https://arxiv.org/html/2608.18681#S1.p1.1),[§2](https://arxiv.org/html/2608.18681#S2.p3.1)\.
- DeHaven and Scott \(2023\)M\. DeHaven and S\. ScottBEVERS: a general, simple, and performant framework for automatic fact verification\.InProceedings of the Sixth Workshop on Fact Extraction and VERification \(FEVER\),External Links:[Link](https://aclanthology.org/2023.fever-1.6/)Cited by:[Table 2](https://arxiv.org/html/2608.18681#S4.T2.4.5.1.1)\.
- Fanet al\.\(2017\)Y\. Fan, F\. Tian, T\. Qin, J\. Bian, and T\. LiuLearning what data to learn\.arXiv preprintabs/1702\.08635\.External Links:1702\.08635,[Link](https://arxiv.org/abs/1702.08635)Cited by:[§2](https://arxiv.org/html/2608.18681#S2.p4.1)\.
- Glockneret al\.\(2018\)M\. Glockner, V\. Shwartz, and Y\. GoldbergBreaking nli systems with sentences that require simple lexical inferences\.arXiv preprintabs/1805\.02266\.External Links:1805\.02266,[Link](https://arxiv.org/abs/1805.02266)Cited by:[§1](https://arxiv.org/html/2608.18681#S1.p1.1),[§2](https://arxiv.org/html/2608.18681#S2.p1.1)\.
- Google Research \(2025\)Google ResearchGemma\-3\-27b\-it\.Note:Hugging Face model repository[https://huggingface\.co/google/gemma\-3\-27b\-it](https://huggingface.co/google/gemma-3-27b-it)Cited by:[Table 8](https://arxiv.org/html/2608.18681#A2.T8),[§4](https://arxiv.org/html/2608.18681#S4.p4.1)\.
- Hosseiniet al\.\(2024\)M\. J\. Hosseini, A\. Petrov, A\. Fabrikant, and A\. LouisA synthetic data approach for domain generalization of NLI models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 2212–2226\.External Links:[Link](https://aclanthology.org/2024.acl-long.120/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.120)Cited by:[§1](https://arxiv.org/html/2608.18681#S1.p1.1),[§2](https://arxiv.org/html/2608.18681#S2.p1.1),[§4](https://arxiv.org/html/2608.18681#S4.p7.1)\.
- HuggingFace \(2022\)HuggingFacePepa/roberta\-base\-snli\.Note:Accessed: October 12, 2024External Links:[Link](https://huggingface.co/pepa/roberta-base-snli)Cited by:[§4](https://arxiv.org/html/2608.18681#S4.p2.1)\.
- Iyyeret al\.\(2018\)M\. Iyyer, J\. Wieting, K\. Gimpel, and L\. ZettlemoyerAdversarial example generation with syntactically controlled paraphrase networks\.InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long Papers\),New Orleans, Louisiana, USA,pp\. 1875–1885\.External Links:[Document](https://dx.doi.org/10.18653/v1/N18-1170),[Link](https://aclanthology.org/N18-1170.pdf)Cited by:[§2](https://arxiv.org/html/2608.18681#S2.p2.1)\.
- Javaheripiet al\.\(2023\)M\. Javaheripi, S\. Bubeck, M\. Abdin, J\. Aneja, S\. Bubeck, C\. C\. T\. Mendes, W\. Chen, A\. Del Giorno, R\. Eldan, S\. Gopi,et al\.Phi\-2: the surprising power of small language models\.Microsoft Research Blog1\(3\),pp\. 3\.Cited by:[§B\.7\.1](https://arxiv.org/html/2608.18681#A2.SS7.SSS1.p1.1)\.
- Jhaet al\.\(2025\)A\. Jha, H\. Gupta, and A\. NandiRL\-guided data selection for language model finetuning\.InNeurIPS 2025 Workshop on Reliable Machine Learning from Unreliable Data,Vancouver, Canada\.External Links:2509\.25850,[Link](https://arxiv.org/abs/2509.25850)Cited by:[§2](https://arxiv.org/html/2608.18681#S2.p4.1)\.
- Kazoomet al\.\(2024a\)R\. Kazoom, R\. Birman, and O\. HadarEnhancing object detection robustness: detecting and restoring confidence in the presence of adversarial patch attacks\.arXiv preprint arXiv:2403\.12988\.Cited by:[§1](https://arxiv.org/html/2608.18681#S1.p1.1)\.
- Kazoomet al\.\(2024b\)R\. Kazoom, R\. Birman, and O\. HadarImproving the robustness of object detection and classification ai models against adversarial patch attacks\.arXiv e\-prints,pp\. arXiv–2403\.Cited by:[§2](https://arxiv.org/html/2608.18681#S2.p1.1)\.
- Kazoomet al\.\(2025a\)R\. Kazoom, R\. Birman, and O\. HadarFrom adversity to advantage: diffusion models for improved detection under attack\.InInternational Symposium on Cyber Security, Cryptology, and Machine Learning,pp\. 104–121\.Cited by:[§2](https://arxiv.org/html/2608.18681#S2.p2.1)\.
- Kazoomet al\.\(2025b\)R\. Kazoom, O\. Cohen, R\. Puzis, A\. Shabtai, and O\. HadarVault: vigilant adversarial updates via llm\-driven retrieval\-augmented generation for nli\.arXiv preprint arXiv:2508\.00965\.Cited by:[§2](https://arxiv.org/html/2608.18681#S2.p2.1)\.
- Kazoomet al\.\(2026\)R\. Kazoom, A\. Goldberg, H\. Cohen, and O\. HadarSeeing isn’t believing: context\-aware adversarial patch synthesis via conditional gan\.InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision,pp\. 202–211\.Cited by:[§2](https://arxiv.org/html/2608.18681#S2.p2.1)\.
- Kazoomet al\.\(2025c\)R\. Kazoom, R\. Lapid, M\. Sipper, and O\. HadarDon’t Lag, RAG: training\-free adversarial detection using rag\.arXiv preprintabs/2504\.04858\.External Links:2504\.04858,[Link](https://arxiv.org/abs/2504.04858)Cited by:[§2](https://arxiv.org/html/2608.18681#S2.p1.1)\.
- Klemen and Robnik\-Šikonja \(2021\)M\. Klemen and M\. Robnik\-ŠikonjaExtracting and filtering paraphrases by bridging natural language inference and paraphrasing\.arXiv preprintabs/2111\.07119\.External Links:2111\.07119,[Link](https://arxiv.org/abs/2111.07119)Cited by:[§2](https://arxiv.org/html/2608.18681#S2.p1.1)\.
- Liet al\.\(2025\)S\. Li, S\. Li, Z\. Yang, X\. Zhang, G\. Chen, X\. Xia, H\. Liu, and Z\. PengLearnalign: reasoning data selection for reinforcement learning in large language models based on improved gradient alignment\.arXiv preprint arXiv:2506\.11480\.Cited by:[§2](https://arxiv.org/html/2608.18681#S2.p4.1)\.
- Liet al\.\(2023\)Y\. Li, M\. Xu, X\. Miao, S\. Zhou, and T\. QianPrompting large language models for counterfactual generation: an empirical study\.arXiv preprintabs/2305\.14791\.External Links:2305\.14791,[Link](https://arxiv.org/abs/2305.14791)Cited by:[§2](https://arxiv.org/html/2608.18681#S2.p1.1)\.
- Liuet al\.\(2020\)Z\. Liu, C\. Xiong, M\. Sun, and Z\. LiuFine\-grained fact verification with kernel graph attention network\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,External Links:[Link](https://aclanthology.org/2020.acl-main.655/)Cited by:[Table 2](https://arxiv.org/html/2608.18681#S4.T2.4.3.1.1)\.
- Meta AI \(2025\)Meta AILlama\-4\-scout\-17b\-16e\-instruct\.Note:Hugging Face model repository[https://huggingface\.co/meta\-llama/Llama\-4\-Scout\-17B\-16E\-Instruct](https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct)Cited by:[§4](https://arxiv.org/html/2608.18681#S4.p3.1)\.
- Microsoft Research \(2025\)Microsoft ResearchPhi\-4\.Note:Hugging Face model repository[https://huggingface\.co/microsoft/phi\-4](https://huggingface.co/microsoft/phi-4)Cited by:[Table 8](https://arxiv.org/html/2608.18681#A2.T8),[§4](https://arxiv.org/html/2608.18681#S4.p4.1)\.
- Minervini and Riedel \(2018\)P\. Minervini and S\. RiedelAdversarially regularising neural NLI models to integrate logical background knowledge\.InProceedings of the 22nd Conference on Computational Natural Language Learning \(CoNLL\),pp\. 65–74\.Cited by:[§2](https://arxiv.org/html/2608.18681#S2.p2.1)\.
- Nieet al\.\(2019\)Y\. Nie, A\. Williams, E\. Dinan, M\. Bansal, J\. Weston, and D\. KielaAdversarial NLI: A new benchmark for natural language understanding\.arXiv preprint arXiv:1910\.14599\.Cited by:[Figure 10](https://arxiv.org/html/2608.18681#A2.F10),[§B\.9](https://arxiv.org/html/2608.18681#A2.SS9.p1.1),[§1](https://arxiv.org/html/2608.18681#S1.p1.1),[§2](https://arxiv.org/html/2608.18681#S2.p1.1),[§4](https://arxiv.org/html/2608.18681#S4.p6.1)\.
- Nieet al\.\(2020\)Y\. Nie, A\. Williams, E\. Dinan, M\. Bansal, J\. Weston, and D\. KielaAdversarial NLI: a new benchmark for natural language understanding\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 4885–4901\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.441)Cited by:[§2](https://arxiv.org/html/2608.18681#S2.p2.1)\.
- Qwen Team \(2025\)Qwen TeamQwen3\-32b\.Note:Hugging Face model repository[https://huggingface\.co/Qwen/Qwen3\-32B](https://huggingface.co/Qwen/Qwen3-32B)Cited by:[Table 8](https://arxiv.org/html/2608.18681#A2.T8),[§4](https://arxiv.org/html/2608.18681#S4.p4.1)\.
- Robertson and Zaragoza \(2009\)S\. E\. Robertson and H\. ZaragozaThe probabilistic relevance framework: BM25 and beyond\.Foundations and Trends in Information Retrieval3\(4\),pp\. 333–389\.External Links:[Document](https://dx.doi.org/10.1561/1500000019)Cited by:[§1](https://arxiv.org/html/2608.18681#S1.p1.1),[§2](https://arxiv.org/html/2608.18681#S2.p3.1)\.
- Tymoshenko and Moschitti \(2021\)K\. Tymoshenko and A\. MoschittiStrong and light baseline models for fact\-checking joint inference\.InFindings of the Association for Computational Linguistics: ACL\-IJCNLP 2021,Online,pp\. 4824–4830\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.findings-acl.426),[Link](https://aclanthology.org/2021.findings-acl.426/)Cited by:[Table 2](https://arxiv.org/html/2608.18681#S4.T2.4.4.1.1)\.
- Williamset al\.\(2018\)A\. Williams, N\. Nangia, and S\. BowmanA broad\-coverage challenge corpus for sentence understanding through inference\.InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long Papers\),pp\. 1112–1122\.Cited by:[Figure 10](https://arxiv.org/html/2608.18681#A2.F10),[§B\.9](https://arxiv.org/html/2608.18681#A2.SS9.p1.1),[§1](https://arxiv.org/html/2608.18681#S1.p1.1),[§2](https://arxiv.org/html/2608.18681#S2.p1.1),[§4](https://arxiv.org/html/2608.18681#S4.p6.1)\.
- Yanget al\.\(2024\)A\. Yanget al\.Qwen2\.5: a suite of foundation models\.arXiv preprint arXiv:2412\.151152412\.15115\.External Links:[Link](https://arxiv.org/abs/2412.15115)Cited by:[§B\.7\.1](https://arxiv.org/html/2608.18681#A2.SS7.SSS1.p1.1)\.
- Yanget al\.\(2025\)S\. Yang, P\. Li, F\. Shen, and J\. ZhaoRl\-selector: reinforcement learning\-guided data selection via redundancy assessment\.arXiv preprint arXiv:2506\.21037\.Cited by:[§2](https://arxiv.org/html/2608.18681#S2.p4.1)\.
- Zhanget al\.\(2020\)T\. Zhang, V\. Kishore, F\. Wu, K\. Q\. Weinberger, and Y\. ArtziBERTScore: evaluating text generation with bert\.InProceedings of the 8th International Conference on Learning Representations,ICLR 2020,Addis Ababa, Ethiopia\.External Links:[Link](https://openreview.net/forum?id=SkeHuCVFDr)Cited by:[§B\.8](https://arxiv.org/html/2608.18681#A2.SS8.p1.1)\.
- Zhaoet al\.\(2020\)M\. Zhao, H\. Wu, D\. Niu, and X\. WangReinforced curriculum learning on pre\-trained neural machine translation models\.arXiv preprintabs/2004\.05757\.External Links:2004\.05757,[Link](https://arxiv.org/abs/2004.05757)Cited by:[§2](https://arxiv.org/html/2608.18681#S2.p4.1)\.
- Zhouet al\.\(2019\)J\. Zhou, X\. Han, C\. Yang, Z\. Liu, L\. Wang, C\. Li, and M\. SunGEAR: graph\-based evidence aggregating and reasoning for fact verification\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,External Links:[Link](https://aclanthology.org/P19-1085/)Cited by:[Table 2](https://arxiv.org/html/2608.18681#S4.T2.4.2.1.1)\.

## Appendix AAppendix

## Appendix BTheoretical Interpretation and Proofs

### B\.1Bias Reduction via Failure\-Mode Curation

Let𝒳\\mathcal\{X\}be the input space,Y∈𝒴Y\\in\\mathcal\{Y\}the label space, andPPthe underlying data distribution\. Letfθt:𝒳→Δ⁡\(𝒴\)f\_\{\\theta\_\{t\}\}:\\mathcal\{X\}\\to\\Delta\(\\mathcal\{Y\}\)denote the target model at iterationtt, and letℓ⁡\(θ,x,y\)\\ell\(\\theta;x,y\)be the per\-example loss\. We define the failure indicator as:

et\(x,y\)=𝕀\[argmaxc∈𝒴fθt\(x\)c≠y\]\.e\_\{t\}\(x,y\)=\\mathbb\{I\}\\left\[\\arg\\max\_\{c\\in\\mathcal\{Y\}\}f\_\{\\theta\_\{t\}\}\(x\)\_\{c\}\\neq y\\right\]\.\(29\)
The corresponding failure probability is:

εt=𝔼\(x,y\)∼P​\[et​\(x,y\)\]\.\\varepsilon\_\{t\}=\\mathbb\{E\}\_\{\(x,y\)\\sim P\}\\left\[e\_\{t\}\(x,y\)\\right\]\.\(30\)
Whenεt\>0\\varepsilon\_\{t\}\>0, the failure\-conditioned distribution is:

Ft​\(x,y\)=P⁡\(x,y∣et​\(x,y\)=1\)=P⁡\(x,y\)​et​\(x,y\)εt\.F\_\{t\}\(x,y\)=P\(x,y\\mid e\_\{t\}\(x,y\)=1\)=\\frac\{P\(x,y\)e\_\{t\}\(x,y\)\}\{\\varepsilon\_\{t\}\}\.\(31\)
In practice, the method does not sample directly fromFtF\_\{t\}\. Instead, it generates candidate adversarial examples, filters candidates that foolM\(t\)M^\{\(t\)\}, validates them with automated judges, clusters the validated failures into failure modes, and samples from these modes using the contextual\-bandit policyπθ\\pi\_\{\\theta\}\. LetF^tπ\\widehat\{F\}\_\{t\}^\{\\pi\}denote the policy\-induced distribution over selected validated failure examples\. The effective training distribution is:

Ptλ=\(1−λ\)​P\+λ​F^tπ,λ∈\(0,1\)\.P\_\{t\}^\{\\lambda\}=\(1\-\\lambda\)P\+\\lambda\\widehat\{F\}\_\{t\}^\{\\pi\},\\qquad\\lambda\\in\(0,1\)\.\(32\)
The target model is then updated by stochastic gradient descent on:

𝔼\(x,y\)∼Ptλ​\[ℓ⁡\(θ,x,y\)\]\.\\mathbb\{E\}\_\{\(x,y\)\\sim P\_\{t\}^\{\\lambda\}\}\\left\[\\ell\(\\theta;x,y\)\\right\]\.\(33\)
##### Spurious bias model\.

We use an abstract decompositionx=\(c,s\)x=\(c,s\), whereccdenotes task\-relevant core information andssdenotes a spurious feature that is correlated with the label underPPbut unreliable under distribution shift\. This decomposition is not assumed to be explicitly available to the model; it is used only for analysis\. Letgc​\(x,y\)g\_\{c\}\(x,y\)andgs​\(x,y\)g\_\{s\}\(x,y\)denote the gradient components along unit directionsucu\_\{c\}andusu\_\{s\}:

gc​\(x,y\)=⟨∇θℓ​\(θ,x,y\),uc⟩,gs​\(x,y\)=⟨∇θℓ​\(θ,x,y\),us⟩\.g\_\{c\}\(x,y\)=\\left\\langle\\nabla\_\{\\theta\}\\ell\(\\theta;x,y\),u\_\{c\}\\right\\rangle,\\qquad g\_\{s\}\(x,y\)=\\left\\langle\\nabla\_\{\\theta\}\\ell\(\\theta;x,y\),u\_\{s\}\\right\\rangle\.\(34\)
We assume that standard training onPPmay reinforce the spurious direction, while failure regions reduce reliance on this shortcut\. Specifically, assume there exist constantsμs\>0\\mu\_\{s\}\>0andμc≥0\\mu\_\{c\}\\geq 0such that:

𝔼P​\[gs​\(x,y\)\]≥μs,\\mathbb\{E\}\_\{P\}\\left\[g\_\{s\}\(x,y\)\\right\]\\geq\\mu\_\{s\},\(35\)
and:

𝔼Ft​\[gs​\(x,y\)\]≤0,𝔼Ft​\[gc​\(x,y\)\]≥μc\.\\mathbb\{E\}\_\{F\_\{t\}\}\\left\[g\_\{s\}\(x,y\)\\right\]\\leq 0,\\qquad\\mathbb\{E\}\_\{F\_\{t\}\}\\left\[g\_\{c\}\(x,y\)\\right\]\\geq\\mu\_\{c\}\.\(36\)
This captures the intended failure\-focused behavior: validated failures are examples where shortcut\-based prediction is less reliable and core task\-relevant evidence is more important\.

##### Policy approximation\.

The contextual\-bandit policy does not need to recoverFtF\_\{t\}exactly\. It is sufficient that the policy\-induced selected distributionF^tπ\\widehat\{F\}\_\{t\}^\{\\pi\}approximatesFtF\_\{t\}with bounded error along the relevant gradient directions:

\|𝔼F^tπ​\[gs\]−𝔼Ft​\[gs\]\|≤δs,\|𝔼F^tπ​\[gc\]−𝔼Ft​\[gc\]\|≤δc,\\left\|\\mathbb\{E\}\_\{\\widehat\{F\}\_\{t\}^\{\\pi\}\}\\left\[g\_\{s\}\\right\]\-\\mathbb\{E\}\_\{F\_\{t\}\}\\left\[g\_\{s\}\\right\]\\right\|\\leq\\delta\_\{s\},\\qquad\\left\|\\mathbb\{E\}\_\{\\widehat\{F\}\_\{t\}^\{\\pi\}\}\\left\[g\_\{c\}\\right\]\-\\mathbb\{E\}\_\{F\_\{t\}\}\\left\[g\_\{c\}\\right\]\\right\|\\leq\\delta\_\{c\},\(37\)
forδs,δc≥0\\delta\_\{s\},\\delta\_\{c\}\\geq 0\.

##### Lemma A\.1\. Failure\-mode sampling reduces shortcut\-aligned gradients\.

Under the above assumptions, the mixture distributionPtλP\_\{t\}^\{\\lambda\}satisfies:

𝔼Ptλ​\[gs​\(x,y\)\]≤\(1−λ\)​𝔼P​\[gs\]\+λ​δs,\\mathbb\{E\}\_\{P\_\{t\}^\{\\lambda\}\}\\left\[g\_\{s\}\(x,y\)\\right\]\\leq\(1\-\\lambda\)\\mathbb\{E\}\_\{P\}\\left\[g\_\{s\}\\right\]\+\\lambda\\delta\_\{s\},\(38\)
and:

𝔼Ptλ​\[gc​\(x,y\)\]≥\(1−λ\)​𝔼P​\[gc\]\+λ⁡\(μc−δc\)\.\\mathbb\{E\}\_\{P\_\{t\}^\{\\lambda\}\}\\left\[g\_\{c\}\(x,y\)\\right\]\\geq\(1\-\\lambda\)\\mathbb\{E\}\_\{P\}\\left\[g\_\{c\}\\right\]\+\\lambda\(\\mu\_\{c\}\-\\delta\_\{c\}\)\.\(39\)
Consequently, ifδs<𝔼P​\[gs\]\\delta\_\{s\}<\\mathbb\{E\}\_\{P\}\[g\_\{s\}\], then failure\-mode sampling reduces the shortcut\-aligned gradient contribution relative to training only onPPby at least:

λ⁡\(𝔼P​\[gs\]−δs\)\.\\lambda\\left\(\\mathbb\{E\}\_\{P\}\[g\_\{s\}\]\-\\delta\_\{s\}\\right\)\.\(40\)

##### Proof\.

By linearity of expectation under the mixture distribution:

𝔼Ptλ​\[gs\]=\(1−λ\)​𝔼P​\[gs\]\+λ​𝔼F^tπ​\[gs\]\.\\mathbb\{E\}\_\{P\_\{t\}^\{\\lambda\}\}\\left\[g\_\{s\}\\right\]=\(1\-\\lambda\)\\mathbb\{E\}\_\{P\}\\left\[g\_\{s\}\\right\]\+\\lambda\\mathbb\{E\}\_\{\\widehat\{F\}\_\{t\}^\{\\pi\}\}\\left\[g\_\{s\}\\right\]\.\(41\)
Using the policy approximation bound and the failure\-region assumption:

𝔼F^tπ​\[gs\]≤𝔼Ft​\[gs\]\+δs≤δs\.\\mathbb\{E\}\_\{\\widehat\{F\}\_\{t\}^\{\\pi\}\}\\left\[g\_\{s\}\\right\]\\leq\\mathbb\{E\}\_\{F\_\{t\}\}\\left\[g\_\{s\}\\right\]\+\\delta\_\{s\}\\leq\\delta\_\{s\}\.\(42\)
Substituting this into the mixture expression gives:

𝔼Ptλ​\[gs\]≤\(1−λ\)​𝔼P​\[gs\]\+λ​δs\.\\mathbb\{E\}\_\{P\_\{t\}^\{\\lambda\}\}\\left\[g\_\{s\}\\right\]\\leq\(1\-\\lambda\)\\mathbb\{E\}\_\{P\}\\left\[g\_\{s\}\\right\]\+\\lambda\\delta\_\{s\}\.\(43\)
The result for the core component follows similarly\. From the approximation assumption:

𝔼F^tπ​\[gc\]≥𝔼Ft​\[gc\]−δc≥μc−δc\.\\mathbb\{E\}\_\{\\widehat\{F\}\_\{t\}^\{\\pi\}\}\\left\[g\_\{c\}\\right\]\\geq\\mathbb\{E\}\_\{F\_\{t\}\}\\left\[g\_\{c\}\\right\]\-\\delta\_\{c\}\\geq\\mu\_\{c\}\-\\delta\_\{c\}\.\(44\)
Therefore:

𝔼Ptλ​\[gc\]≥\(1−λ\)​𝔼P​\[gc\]\+λ⁡\(μc−δc\)\.\\mathbb\{E\}\_\{P\_\{t\}^\{\\lambda\}\}\\left\[g\_\{c\}\\right\]\\geq\(1\-\\lambda\)\\mathbb\{E\}\_\{P\}\\left\[g\_\{c\}\\right\]\+\\lambda\(\\mu\_\{c\}\-\\delta\_\{c\}\)\.\(45\)
Finally, comparing the shortcut bound to𝔼P​\[gs\]\\mathbb\{E\}\_\{P\}\[g\_\{s\}\]gives:

𝔼P​\[gs\]−𝔼Ptλ​\[gs\]≥λ⁡\(𝔼P​\[gs\]−δs\),\\mathbb\{E\}\_\{P\}\[g\_\{s\}\]\-\\mathbb\{E\}\_\{P\_\{t\}^\{\\lambda\}\}\[g\_\{s\}\]\\geq\\lambda\\left\(\\mathbb\{E\}\_\{P\}\[g\_\{s\}\]\-\\delta\_\{s\}\\right\),\(46\)
which is positive wheneverδs<𝔼P​\[gs\]\\delta\_\{s\}<\\mathbb\{E\}\_\{P\}\[g\_\{s\}\]\. This completes the proof\.

### B\.2Boundedness of Failure\-Aware Bandit Updates

The contextual\-bandit policy changes the effective training distribution by selecting different failure modes across iterations\. To avoid uncontrolled distributional drift, the target model is trained on a mixture of original and selected adversarial examples\. Let𝒫t\\mathcal\{P\}\_\{t\}denote the effective training distribution at iterationtt, and letF^tπ\\widehat\{F\}\_\{t\}^\{\\pi\}denote the distribution induced by the failure\-mode policy\. We analyze the abstract update:

𝒫t\+1=\(1−η\)​𝒫t\+η​F^tπ,η∈\(0,1\)\.\\mathcal\{P\}\_\{t\+1\}=\(1\-\\eta\)\\mathcal\{P\}\_\{t\}\+\\eta\\widehat\{F\}\_\{t\}^\{\\pi\},\\qquad\\eta\\in\(0,1\)\.\(47\)
##### Proposition A\.2\. Per\-step distributional drift is bounded\.

For any iterationtt, the per\-step change in the effective training distribution satisfies:

‖𝒫t\+1−𝒫t‖1=η​‖F^tπ−𝒫t‖1≤2​η\.\\left\\\|\\mathcal\{P\}\_\{t\+1\}\-\\mathcal\{P\}\_\{t\}\\right\\\|\_\{1\}=\\eta\\left\\\|\\widehat\{F\}\_\{t\}^\{\\pi\}\-\\mathcal\{P\}\_\{t\}\\right\\\|\_\{1\}\\leq 2\\eta\.\(48\)
Moreover, for anytt, the distance from the initial distribution is bounded by:

‖𝒫t−𝒫0‖1≤2\.\\left\\\|\\mathcal\{P\}\_\{t\}\-\\mathcal\{P\}\_\{0\}\\right\\\|\_\{1\}\\leq 2\.\(49\)

##### Proof\.

From the update rule:

𝒫t\+1−𝒫t=η⁡\(F^tπ−𝒫t\)\.\\mathcal\{P\}\_\{t\+1\}\-\\mathcal\{P\}\_\{t\}=\\eta\\left\(\\widehat\{F\}\_\{t\}^\{\\pi\}\-\\mathcal\{P\}\_\{t\}\\right\)\.\(50\)
Taking theℓ1\\ell\_\{1\}norm gives:

‖𝒫t\+1−𝒫t‖1=η​‖F^tπ−𝒫t‖1\.\\left\\\|\\mathcal\{P\}\_\{t\+1\}\-\\mathcal\{P\}\_\{t\}\\right\\\|\_\{1\}=\\eta\\left\\\|\\widehat\{F\}\_\{t\}^\{\\pi\}\-\\mathcal\{P\}\_\{t\}\\right\\\|\_\{1\}\.\(51\)
Because theℓ1\\ell\_\{1\}distance between any two probability distributions is at most22:

‖𝒫t\+1−𝒫t‖1≤2​η\.\\left\\\|\\mathcal\{P\}\_\{t\+1\}\-\\mathcal\{P\}\_\{t\}\\right\\\|\_\{1\}\\leq 2\\eta\.\(52\)
The second statement follows from the same fact, since both𝒫t\\mathcal\{P\}\_\{t\}and𝒫0\\mathcal\{P\}\_\{0\}are probability distributions:

‖𝒫t−𝒫0‖1≤2\.\\left\\\|\\mathcal\{P\}\_\{t\}\-\\mathcal\{P\}\_\{0\}\\right\\\|\_\{1\}\\leq 2\.\(53\)
This establishes that each update is locally controlled by the mixing coefficientη\\eta\.

##### Reward\-noise setting\.

The policy is updated using validation feedback, which may be noisy due to finite validation sets and stochastic retraining\. Letuku\_\{k\}denote the ideal utility of failure modekk, and let the observed utility be:

u~k=uk\+ξk,\|ξk\|≤ε\.\\tilde\{u\}\_\{k\}=u\_\{k\}\+\\xi\_\{k\},\\qquad\|\\xi\_\{k\}\|\\leq\\varepsilon\.\(54\)
For analysis, consider the normalized policy\-induced allocation distribution over failure modes:

πu​\(k\)=exp⁡\(uk\)∑jexp⁡\(uj\),πu~​\(k\)=exp⁡\(u~k\)∑jexp⁡\(u~j\)\.\\pi\_\{u\}\(k\)=\\frac\{\\exp\(u\_\{k\}\)\}\{\\sum\_\{j\}\\exp\(u\_\{j\}\)\},\\qquad\\pi\_\{\\tilde\{u\}\}\(k\)=\\frac\{\\exp\(\\tilde\{u\}\_\{k\}\)\}\{\\sum\_\{j\}\\exp\(\\tilde\{u\}\_\{j\}\)\}\.\(55\)
This log\-linear allocation is a standard smooth relaxation of selecting failure modes according to estimated utility\.

##### Proposition A\.3\. Bounded reward noise induces bounded sampling distortion\.

Assume\|ξk\|≤ε\|\\xi\_\{k\}\|\\leq\\varepsilonfor all failure modeskk\. Then, for any set of failure modesAAwithπu​\(A\)\>0\\pi\_\{u\}\(A\)\>0:

e−2​ε≤πu~​\(A\)πu​\(A\)≤e2​ε\.e^\{\-2\\varepsilon\}\\leq\\frac\{\\pi\_\{\\tilde\{u\}\}\(A\)\}\{\\pi\_\{u\}\(A\)\}\\leq e^\{2\\varepsilon\}\.\(56\)

##### Proof\.

For each failure modekk:

e−ε​exp⁡\(uk\)≤exp⁡\(u~k\)≤eε​exp⁡\(uk\)\.e^\{\-\\varepsilon\}\\exp\(u\_\{k\}\)\\leq\\exp\(\\tilde\{u\}\_\{k\}\)\\leq e^\{\\varepsilon\}\\exp\(u\_\{k\}\)\.\(57\)
Summing over all modes gives:

e−ε​∑jexp⁡\(uj\)≤∑jexp⁡\(u~j\)≤eε​∑jexp⁡\(uj\)\.e^\{\-\\varepsilon\}\\sum\_\{j\}\\exp\(u\_\{j\}\)\\leq\\sum\_\{j\}\\exp\(\\tilde\{u\}\_\{j\}\)\\leq e^\{\\varepsilon\}\\sum\_\{j\}\\exp\(u\_\{j\}\)\.\(58\)
Combining the numerator and denominator bounds yields, for eachkk:

e−2​ε≤πu~​\(k\)πu​\(k\)≤e2​ε\.e^\{\-2\\varepsilon\}\\leq\\frac\{\\pi\_\{\\tilde\{u\}\}\(k\)\}\{\\pi\_\{u\}\(k\)\}\\leq e^\{2\\varepsilon\}\.\(59\)
Summing over allk∈Ak\\in Apreserves the same multiplicative bound:

e−2​ε≤∑k∈Aπu~​\(k\)∑k∈Aπu​\(k\)≤e2​ε\.e^\{\-2\\varepsilon\}\\leq\\frac\{\\sum\_\{k\\in A\}\\pi\_\{\\tilde\{u\}\}\(k\)\}\{\\sum\_\{k\\in A\}\\pi\_\{u\}\(k\)\}\\leq e^\{2\\varepsilon\}\.\(60\)
Therefore:

e−2​ε≤πu~​\(A\)πu​\(A\)≤e2​ε\.e^\{\-2\\varepsilon\}\\leq\\frac\{\\pi\_\{\\tilde\{u\}\}\(A\)\}\{\\pi\_\{u\}\(A\)\}\\leq e^\{2\\varepsilon\}\.\(61\)
Together, Proposition[B\.2](https://arxiv.org/html/2608.18681#A2.SS2.SSS0.Px1)and Proposition[B\.2](https://arxiv.org/html/2608.18681#A2.SS2.SSS0.Px4)show that the failure\-aware bandit update operates in a controlled regime: the mixture coefficient bounds the per\-step distributional shift, and bounded noise in validation\-based utility estimates induces only bounded distortion in the policy\-induced failure\-mode allocation\.

### B\.3Backbone and Training Strategy Ablation

We analyze the impact of backbone architecture and training strategy using the results reported in Table[4](https://arxiv.org/html/2608.18681#A2.T4)\. This experiment evaluates five representative models spanning a wide range of parameter scales, from lightweight decoder\-only architectures \(SmolLM2\-360M and Qwen3\-0\.6B\) to large encoder\-based models \(RoBERTa\-base, DeBERTa\-v3, and RoBERTa\-large\)\. For each backbone, we compare three settings: \(i\) evaluation without fine\-tuning \(No FT\), \(ii\) fine\-tuning with paraphrase\-based data augmentation \(Paraphrasing\), and \(iii\) our reinforcement\-guided failure\-driven training framework \(Ours\)\. All models are trained under identical conditions using fixed 6\-shot prompting and a constant original\-to\-generated ratio of1:41\{:\}4\(r=4r=4\)\.

Several consistent trends emerge across all datasets\. First, models evaluated without fine\-tuning exhibit the lowest performance in every configuration, confirming that direct transfer without adaptation is insufficient for robust NLI\. Paraphrase\-based augmentation yields moderate improvements over No FT, indicating that generic linguistic variation helps alleviate some distributional mismatch\. However, these gains remain limited, particularly on challenging benchmarks such as Adversarial NLI, where paraphrasing fails to systematically target the model’s dominant failure modes\.

In contrast, our reinforcement\-guided approach consistently achieves the strongest performance for all backbone architectures and datasets\. On SNLI, our method improves RoBERTa\-base from 90\.72% under paraphrasing to 92\.60%, and yields comparable gains for smaller models such as SmolLM2\-360M \(\+1\.81 points\) and Qwen3\-0\.6B \(\+1\.91 points\)\. Similar patterns are observed on Adversarial NLI, where our framework outperforms paraphrasing by 2\.83 points for RoBERTa\-base and by more than 2 points for all other models\. On MultiNLI, which exhibits substantial genre diversity, reinforcement\-guided training produces consistent improvements ranging from 2\.36 to 3\.47 points over paraphrasing\.

The results further demonstrate that the benefits of failure\-driven policy learning are preserved across model scales\. While larger backbones such as RoBERTa\-large achieve higher absolute accuracy, smaller and medium\-sized models also benefit substantially from targeted adversarial mining\. This indicates that the proposed framework does not rely on excess model capacity, but instead improves generalization by reshaping the training distribution toward informative failure regions\.

Importantly, the consistent gap between paraphrasing and our method highlights the limitations of heuristic data augmentation\. Paraphrase\-based approaches introduce surface\-level variation but do not adaptively concentrate on systematic errors\. In contrast, our method leverages reinforcement learning to prioritize samples that expose decision boundary weaknesses, resulting in more efficient and targeted learning\.

Overall, the results in Table[4](https://arxiv.org/html/2608.18681#A2.T4)demonstrate that reinforcement\-guided adversarial training yields robust and scalable improvements across diverse architectures and training regimes, confirming the generality of the proposed approach\.

##### Effect of Generator and Verifier Scale\.

Our framework relies on large language models for adversarial generation and validation, and its performance may depend on their representational capacity\. To analyze this dependence, future work will systematically vary the scale of both the generator and verifier models, ranging from lightweight open\-source LLMs to large proprietary systems\. Such controlled experiments will enable a principled assessment of how robustness gains trade off against computational cost, and will clarify the operating regimes in which reinforcement\-guided data selection remains effective under limited budgets\.

##### Disentangling Generation and Verification Contributions\.

An important open question concerns the relative contribution of the generation and verification components\. While both modules are optimized through policy feedback, their individual roles in driving performance gains are not yet fully disentangled\. A promising direction is to decouple these stages by fixing one component while varying the capacity of the other, thereby isolating the effect of generation quality versus validation reliability\. This analysis would help determine whether improvements primarily stem from producing more challenging candidates or from more accurate reward estimation via verification\.

Table 4:Accuracy \(%\) across different backbone models and training strategies\. “No FT” denotes evaluation without fine\-tuning, “Paraphrasing” denotes augmentation via paraphrase\-based data, and “Ours” denotes reinforcement\-guided training\. All experiments use fixed 6\-shot prompting and an original\-to\-generated data ratio of1:41\{:\}4\(r=4r=4\)\.

### B\.4Ablation Study: Effect of the Contextual\-Bandit Policy

We analyze the contribution of the contextual\-bandit curator on SNLI by replacing the learned failure\-mode policy with several alternatives\. All variants use the same backbone, retrieval strategy, automated validation, adversarial budget, and retraining protocol\. The full model achieves92\.60%92\.60\\%accuracy on SNLI\.

Let\{ℱ1\(t\),…,ℱKt\(t\)\}\\\{\\mathcal\{F\}^\{\(t\)\}\_\{1\},\\ldots,\\mathcal\{F\}^\{\(t\)\}\_\{K\_\{t\}\}\\\}denote the validated failure\-mode clusters at iterationtt\. Each cluster is represented by a state vectorzt,kz\_\{t,k\}, and the curator selects actionsat,k∈\{0,1\}a\_\{t,k\}\\in\\\{0,1\\\}indicating whether failure modeℱk\(t\)\\mathcal\{F\}^\{\(t\)\}\_\{k\}is sampled for retraining\.

##### Full Model: Contextual\-Bandit Policy\.

The proposed model uses a stochastic policyπθ\\pi\_\{\\theta\}over failure modes:

πθ​\(at,k=1∣zt,k\)=σ⁡\(fθ​\(zt,k\)\)\.\\pi\_\{\\theta\}\(a\_\{t,k\}=1\\mid z\_\{t,k\}\)=\\sigma\(f\_\{\\theta\}\(z\_\{t,k\}\)\)\.\(62\)
After retraining, the policy receives validation rewardGtG\_\{t\}, which balances robustness gain, forgetting, and data cost\. A criticRϕR\_\{\\phi\}estimates the expected return of each selected failure mode:

Rϕ​\(zt,k\)≈𝔼⁡\[Gt∣zt,k\]\.R\_\{\\phi\}\(z\_\{t,k\}\)\\approx\\mathbb\{E\}\[G\_\{t\}\\mid z\_\{t,k\}\]\.\(63\)

##### Random Failure\-Mode Policy\.

This baseline removes learned selection\. Failure modes are sampled uniformly under the same adversarial budget:

at,k∼Bernoulli⁡\(p\)\.a\_\{t,k\}\\sim\\mathrm\{Bernoulli\}\(p\)\.\(64\)

##### Heuristic Failure\-Mode Policy\.

This variant replacesπθ\\pi\_\{\\theta\}with a deterministic uncertainty\-based rule\. Failure modes are ranked by the mean predictive entropy of the target model:

s\(ℱk\(t\)\)=1\|ℱk\(t\)\|∑\(x,o,y\)∈ℱk\(t\)H\(M\(t\)\(⋅∣x,o\)\)\.s\(\\mathcal\{F\}^\{\(t\)\}\_\{k\}\)=\\frac\{1\}\{\|\\mathcal\{F\}^\{\(t\)\}\_\{k\}\|\}\\sum\_\{\(x,o,y\)\\in\\mathcal\{F\}^\{\(t\)\}\_\{k\}\}H\\left\(M^\{\(t\)\}\(\\cdot\\mid x,o\)\\right\)\.\(65\)
The highest\-scoring clusters are selected until the adversarial budget is reached\.

##### Frozen Policy and Critic\.

Here,πθ\\pi\_\{\\theta\}andRϕR\_\{\\phi\}are initialized in the first round and then kept fixed\. This tests whether continual validation\-based adaptation is necessary\.

##### No Failure\-Mode Clustering\.

This baseline removes the failure\-mode abstraction and samples validated failures directly\. It preserves target\-model filtering and automated validation but does not group failures into recurring modes\.

##### Oracle Failure\-Mode Policy\.

As an upper bound, we approximate the utility of a failure mode using its observed validation improvement after retraining:

soracle​\(ℱk\(t\)\)=Perf⁡\(Mk\(t\+1\)\)−Perf⁡\(M\(t\)\),s\_\{\\mathrm\{oracle\}\}\(\\mathcal\{F\}^\{\(t\)\}\_\{k\}\)=\\mathrm\{Perf\}\(M^\{\(t\+1\)\}\_\{k\}\)\-\\mathrm\{Perf\}\(M^\{\(t\)\}\),\(66\)
whereMk\(t\+1\)M^\{\(t\+1\)\}\_\{k\}denotes a model retrained using samples from failure modeℱk\(t\)\\mathcal\{F\}^\{\(t\)\}\_\{k\}\. This variant is not deployable because it requires separate retraining for each candidate failure mode\.

Table[5](https://arxiv.org/html/2608.18681#A2.T5)reports the results\. Random and heuristic policies underperform the full model, showing that static selection is insufficient\. Freezing the policy and critic also degrades performance, indicating that adaptation across rounds is important\. Removing failure\-mode clustering further reduces accuracy, confirming that selecting recurring failure types is more effective than selecting isolated examples\.

Table 5:Ablation of the contextual\-bandit curator on SNLI\.

### B\.5Ablation with Heuristic Failure\-Mode Policies

To assess whether learned policy optimization is necessary, we replace the contextual\-bandit policyπθ\\pi\_\{\\theta\}with several heuristic failure\-mode selection rules\. These baselines use the same generated candidates, target\-model failure filtering, automated LLM validation, retrieval weightα=0\.83\\alpha=0\.83, and mixing ratioλmix=14\\lambda\_\{\\mathrm\{mix\}\}=\\frac\{1\}\{4\}as the full method\. The only difference is how validated failure modes are selected for retraining\.

Letℱk\(t\)\\mathcal\{F\}^\{\(t\)\}\_\{k\}denote a validated failure\-mode cluster at iterationtt\. Each heuristic assigns a cluster\-level scores⁡\(ℱk\(t\)\)s\(\\mathcal\{F\}^\{\(t\)\}\_\{k\}\), and clusters are selected in descending order until the adversarial budget is reached\.

##### Confidence\-Based Policy\.

This policy prioritizes clusters where the target model has low predictive confidence:

sconf​\(ℱk\(t\)\)=1\|ℱk\(t\)\|​∑\(x,o,y\)∈ℱk\(t\)\(1−maxc∈𝒴⁡M\(t\)​\(c∣x,o\)\)\.s\_\{\\mathrm\{conf\}\}\(\\mathcal\{F\}^\{\(t\)\}\_\{k\}\)=\\frac\{1\}\{\|\\mathcal\{F\}^\{\(t\)\}\_\{k\}\|\}\\sum\_\{\(x,o,y\)\\in\\mathcal\{F\}^\{\(t\)\}\_\{k\}\}\\left\(1\-\\max\_\{c\\in\\mathcal\{Y\}\}M^\{\(t\)\}\(c\\mid x,o\)\\right\)\.\(67\)

##### Loss\-Based Policy\.

This policy selects clusters that induce high average supervised loss:

sloss​\(ℱk\(t\)\)=1\|ℱk\(t\)\|​∑\(x,o,y\)∈ℱk\(t\)ℓ⁡\(M\(t\)​\(x,o\),y\)\.s\_\{\\mathrm\{loss\}\}\(\\mathcal\{F\}^\{\(t\)\}\_\{k\}\)=\\frac\{1\}\{\|\\mathcal\{F\}^\{\(t\)\}\_\{k\}\|\}\\sum\_\{\(x,o,y\)\\in\\mathcal\{F\}^\{\(t\)\}\_\{k\}\}\\ell\(M^\{\(t\)\}\(x,o\),y\)\.\(68\)

##### Margin\-Based Policy\.

This policy prioritizes clusters with small separation between the top two predicted classes:

smargin​\(ℱk\(t\)\)=1\|ℱk\(t\)\|​∑\(x,o,y\)∈ℱk\(t\)\(1−\[pθ\(1\)​\(x,o\)−pθ\(2\)​\(x,o\)\]\),s\_\{\\mathrm\{margin\}\}\(\\mathcal\{F\}^\{\(t\)\}\_\{k\}\)=\\frac\{1\}\{\|\\mathcal\{F\}^\{\(t\)\}\_\{k\}\|\}\\sum\_\{\(x,o,y\)\\in\\mathcal\{F\}^\{\(t\)\}\_\{k\}\}\\left\(1\-\\left\[p^\{\(1\)\}\_\{\\theta\}\(x,o\)\-p^\{\(2\)\}\_\{\\theta\}\(x,o\)\\right\]\\right\),\(69\)
wherepθ\(1\)​\(x,o\)p^\{\(1\)\}\_\{\\theta\}\(x,o\)andpθ\(2\)​\(x,o\)p^\{\(2\)\}\_\{\\theta\}\(x,o\)are the highest and second\-highest predicted class probabilities\.

##### Learned Bandit Policy\.

The full method uses the learned contextual\-bandit policy:

πθ​\(at,k=1∣zt,k\)=σ⁡\(fθ​\(zt,k\)\),\\pi\_\{\\theta\}\(a\_\{t,k\}=1\\mid z\_\{t,k\}\)=\\sigma\(f\_\{\\theta\}\(z\_\{t,k\}\)\),\(70\)
wherezt,kz\_\{t,k\}includes loss, entropy, margin, label distribution, retrieval score, judge agreement, novelty, cluster size, and previous reward statistics\. Unlike the heuristic policies,πθ\\pi\_\{\\theta\}is updated using validation rewardGtG\_\{t\}after retraining\.

Table 6:Comparison of heuristic failure\-mode policies and the learned contextual\-bandit policy\.Heuristic policies capture only instantaneous model uncertainty or training difficulty\. As a result, they may oversample noisy, redundant, or locally difficult failures that do not produce sustained validation gains\. In contrast, the learned contextual\-bandit policy is optimized using downstream validation feedback and can adapt across curation rounds\. The consistent improvement over heuristic policies shows that adaptive failure\-mode selection is more effective than static uncertainty\-based selection\.

### B\.6Component Analysis of Failure\-Mode Curation

Table[7](https://arxiv.org/html/2608.18681#A2.T7)evaluates the contribution of the main components in the proposed failure\-mode contextual bandit curation framework\. The full method achieves the best performance across all benchmarks, reaching92\.60%92\.60\\%on SNLI,80\.95%80\.95\\%on ANLI, and71\.99%71\.99\\%on MultiNLI\. This confirms that combining retrieval\-augmented generation, target\-model failure filtering, automated validation, failure\-mode clustering, contextual\-bandit selection, and controlled original\-data mixing provides the strongest robustness gains\.

Removing the contextual\-bandit policy substantially reduces performance\. Random cluster selection performs considerably worse, especially on MultiNLI, indicating that not all failure modes are equally useful for retraining\. Selecting clusters by top loss improves over random selection, but remains below the full method, showing that simple difficulty\-based heuristics are less effective than validation\-driven policy learning\. Similarly, replacing failure\-mode selection with per\-example selection also degrades performance, suggesting that grouping failures into recurring modes provides a more stable and useful unit for adversarial data curation\.

The ablations further show that automated judge validation and target\-model failure filtering are important for maintaining data quality\. Without judge validation, performance drops across all datasets, indicating that noisy or incorrectly labeled generated examples can weaken the retraining signal\. Removing failure filtering causes an even larger degradation, showing that explicitly focusing on examples that expose current model errors is central to the proposed approach\.

Finally, the retrieval and mixing ablations demonstrate the importance of both informative generation context and forgetting control\. Removing retrieved context reduces performance, confirming that retrieved few\-shot examples help guide the generator toward more useful adversarial candidates\. Training without original\-data mixing also hurts performance, supporting the need to balance selected adversarial failures with original training examples in order to improve robustness while limiting forgetting\.

Table 7:Ablation study of the proposed failure\-mode contextual bandit curation framework\.
### B\.7Judge Ensemble Configuration

With the retrieval weight fixed atα=0\.83\\alpha=0\.83and the generated\-to\-original example ratio set to 1:4, we evaluated the impact of varying the number of “judges” \(independent LLM validators\) on downstream accuracy\. All experiments were run on the SNLI test set\. We filtered examples by requiring unanimous agreement among the selected judges and then measured classification accuracy on the remaining items\.

Table 8:Filtering and accuracy under different judge ensemble sizes \(SNLI test, 1:4 gen:orig,α=0\.83\\alpha=0\.83\)\. Judges: G = Gemma\-3\-27B\-IT\([8](https://arxiv.org/html/2608.18681#bib.bib16)\), Q = Qwen3\-32B\([29](https://arxiv.org/html/2608.18681#bib.bib18)\), P = Phi\-4\([25](https://arxiv.org/html/2608.18681#bib.bib17)\)\.As shown in Table[8](https://arxiv.org/html/2608.18681#A2.T8)and Figure[7](https://arxiv.org/html/2608.18681#A2.F7), the three\-judge ensemble yields the highest accuracy \(92\.13%\) on 6,438 filtered observations\. Both the two\-judge and single\-judge configurations retain more examples but achieve lower accuracies of 91\.49% \(9,312 examples\) and 91\.02% \(16,147 examples\), respectively\. Gemma\-3\-27B\-IT consistently remains in all configurations, with Qwen3\-32B joining for the two\-judge setup and Phi\-4 for the three\-judge ensemble\. We adopt the three\-judge configuration for all subsequent evaluations\.

Figure 7:Accuracy vs\. number of judges \(SNLI test,α=0\.83\\alpha=0\.83, 1:4 generated:original\)\. Points are annotated with the number of filtered examples\.#### B\.7\.1Evaluation with Small Judge Models

To study the robustness of our validation pipeline under weaker supervision, we additionally evaluated the judge ensemble using lightweight language models, including Phi\-2[12](https://arxiv.org/html/2608.18681#bib.bib30), Qwen2\.5\-1\.5B[33](https://arxiv.org/html/2608.18681#bib.bib31), and SmolLM2\-360M[1](https://arxiv.org/html/2608.18681#bib.bib32)\. All experiments were conducted using the same retrieval weight \(α=0\.83\\alpha=0\.83\) and generated\-to\-original ratio \(1:4\) as in Table 6\.

We followed the same filtering protocol, retaining only examples for which all selected judges unanimously agreed\. Classification accuracy was then measured on the remaining test instances\. Table[9](https://arxiv.org/html/2608.18681#A2.T9)reports the results on the SNLI test set\.

Table 9:Filtering and accuracy under different small judge ensemble sizes \(SNLI test, 1:4 gen:orig,α=0\.83\\alpha=0\.83\)\. Judges: S = SmolLM2\-360M, P = Phi\-2, Q = Qwen2\.5\-1\.5B\.Compared to large\-model ensembles \(Table 6\), small judge models yield moderately lower accuracy and weaker filtering precision\. Nevertheless, performance improves consistently with ensemble size, and even a single lightweight judge provides substantial robustness gains\. These results indicate that our framework degrades gracefully under weaker validation models, supporting its applicability in low\-resource and cost\-constrained settings\.

### B\.8Dataset Comparison

To gain insights into the relationship between the data generated in our experiment and existing benchmarks, we first extracted the 10 most frequent non\-stopwords from each dataset\. This qualitative analysis highlights topical overlap and domain shifts\. To quantify similarity more rigorously, we computed two complementary metrics across seven collections\-SNLI Train, BGE\-generated, BM25\-generated, SNLI Test, Adversarial NLI, Multi\-NLI, and our hybrid BGE\+BM25\-generated set: TF\-IDF cosine similarity and BERTScore F1\([35](https://arxiv.org/html/2608.18681#bib.bib2)\)\.

TF\-IDF Cosine Similarity\.Let each datasetDDbe represented by a TF\-IDF vector𝐯D∈ℝn\\mathbf\{v\}\_\{D\}\\in\\mathbb\{R\}^\{n\}, wherennis the vocabulary size and theiith component is

vD,i=TFD,i⋅log⁡\(NDFi\),v\_\{D,i\}=\\mathrm\{TF\}\_\{D,i\}\\cdot\\log\\\!\\bigl\(\\tfrac\{N\}\{\\mathrm\{DF\}\_\{i\}\}\\bigr\),withTFD,i\\mathrm\{TF\}\_\{D,i\}the term frequency inDD,NNthe total number of datasets, andDFi\\mathrm\{DF\}\_\{i\}the number of datasets containing termii\. We then define

simTFIDF​\(D,D′\)=𝐯D⋅𝐯D′‖𝐯D‖​‖𝐯D′‖\.\\mathrm\{sim\}\_\{\\mathrm\{TFIDF\}\}\(D,D^\{\\prime\}\)=\\frac\{\\mathbf\{v\}\_\{D\}\\cdot\\mathbf\{v\}\_\{D^\{\\prime\}\}\}\{\\\|\\mathbf\{v\}\_\{D\}\\\|\\;\\\|\\mathbf\{v\}\_\{D^\{\\prime\}\}\\\|\}\.Figure[8](https://arxiv.org/html/2608.18681#A2.F8)shows the resulting7×77\\times 7matrix\. Notably, the hybrid BGE\+BM25 set has a TF\-IDF similarity of approximately 0\.0251 with SNLI Train, 0\.0188 with SNLI Test, and 0\.0150 with Multi\-NLI\-intermediate between its BGE\-only and BM25\-only counterparts\.

BERTScore F1\.We next measure semantic overlap by applying BERTScore F1, which aligns token embeddings from a pre\-trained transformer and computes anF1F\_\{1\}score:

P=1\|x\|​∑t∈xmaxs∈y⁡cos⁡\(𝐞t,𝐞s\),R=1\|y\|​∑s∈ymaxt∈x⁡cos⁡\(𝐞s,𝐞t\),\\mathrm\{P\}=\\frac\{1\}\{\|x\|\}\\sum\_\{t\\in x\}\\max\_\{s\\in y\}\\mathrm\{cos\}\(\\mathbf\{e\}\_\{t\},\\mathbf\{e\}\_\{s\}\),\\mathrm\{R\}=\\frac\{1\}\{\|y\|\}\\sum\_\{s\\in y\}\\max\_\{t\\in x\}\\mathrm\{cos\}\(\\mathbf\{e\}\_\{s\},\\mathbf\{e\}\_\{t\}\),F1=2⋅P​RP\+R,\\mathrm\{F1\}=2\\cdot\\frac\{\\mathrm\{P\}\\,\\mathrm\{R\}\}\{\\mathrm\{P\}\+\\mathrm\{R\}\},wherex,yx,yare token sequences from two datasets and𝐞\\mathbf\{e\}are contextual embeddings\. Figure[9](https://arxiv.org/html/2608.18681#A2.F9)displays the7×77\\times 7BERTScore F1 matrix\. The hybrid set scores about 0\.8658 with SNLI Train, 0\.8534 with SNLI Test, 0\.8458 with Adversarial NLI, and 0\.8554 with Multi\-NLI, again falling between its BGE\-only and BM25\-only pairs\. These results confirm that our validated adversarial examples share both lexical and semantic patterns with standard NLI benchmarks, while still introducing novel, challenging variations\.

![Refer to caption](https://arxiv.org/html/2608.18681v1/TF-IDF.png)Figure 8:Pairwise TF\-IDF cosine similarity between datasets\.![Refer to caption](https://arxiv.org/html/2608.18681v1/BERTScore.png)Figure 9:Pairwise BERTScore F1 between datasets\.From Figure[8](https://arxiv.org/html/2608.18681#A2.F8), we see that both BGE\- and BM25\-generated data share moderate lexical overlap with the original SNLI Train set \(cosine similarities around 0\.02\-0\.03\), but diverge more substantially from the Adversarial NLI and Multi\-NLI benchmarks\. In contrast, Figure[9](https://arxiv.org/html/2608.18681#A2.F9)shows that semantically these generated datasets align much more closely with SNLI Train and SNLI Test \(BERTScore F1 values above 0\.85\), indicating that although the surface vocabulary varies, the core contextual meaning is well preserved\.

### B\.9Generated Dataset Characteristics and Hypothesis Lengths

We first examined the most frequent tokens in each corpus to identify thematic patterns\. In theSNLI train\([2](https://arxiv.org/html/2608.18681#bib.bib3)\)andSNLI test\([2](https://arxiv.org/html/2608.18681#bib.bib3)\)sets, words like “man,” “woman,” and “people” dominate, reflecting descriptions of social interactions\. TheAdversarial NLIdataset\([27](https://arxiv.org/html/2608.18681#bib.bib1)\)shifts focus to media and chronology, with top tokens such as “film,” “first,” and “scene,” while theMulti\-NLI testset\([32](https://arxiv.org/html/2608.18681#bib.bib7)\)uses more abstract, domain\-diverse language\-terms like “author,” “context,” and “claim” appear frequently\.

Figure 10:Comparison of average hypothesis lengths \(in characters and words\) across datasets:Generated\-BM25,Generated\-BGE,SNLI train\([2](https://arxiv.org/html/2608.18681#bib.bib3)\),SNLI test\([2](https://arxiv.org/html/2608.18681#bib.bib3)\),Adversarial NLI\([27](https://arxiv.org/html/2608.18681#bib.bib1)\), andMulti\-NLI\([32](https://arxiv.org/html/2608.18681#bib.bib7)\)\.Turning to our three LLM\-generated sets\-Generated\-BM25,

Generated\-BGE and BGE\+BM25\-we again see a high incidence of speculative and gender\-related terms \(“could,” “would,” “woman,” “he,” “she”\), confirming that all retrieval strategies surface similar thematic content with only minor stylistic differences\.

Figure[10](https://arxiv.org/html/2608.18681#A2.F10)compares the average hypothesis lengths across all seven datasets\. Each of the generated sets produces the longest hypotheses\-around 98\-100 characters \(16\-17 words\)\-demonstrating the LLM’s tendency toward more elaborate constructions when given rich few\-shot contexts\. By contrast, theSNLI trainandSNLI testannotations remain quite concise \(≈37\\approx 37\-3838characters, 7\-8 words\), reflecting the brevity of human\-written examples\. TheAdversarial NLIinstances average≈64\\approx 64characters \(11 words\), and theMulti\-NLIexamples average≈56\\approx 56characters \(10 words\), underscoring their intermediate complexity\. These length patterns highlight how our adversarial RAG pipeline generates richer, more challenging hypotheses while preserving diversity across data sources\.

### B\.10Retrieval Accuracy Across Similarity Metrics

For purely lexical retrieval we employ BM25 with parametersk1=1\.5k\_\{1\}=1\.5andb=0\.75b=0\.75\. The BM25 score for a queryppand documentxxis given by

sBM25​\(p,x\)=∑t∈pIDF⁡\(t\)​tf​\(t,x\)​\(k1\+1\)tf⁡\(t,x\)\+k1​\(1−b\+b​\|x\|avgdl\),s\_\{\\mathrm\{BM25\}\}\(p,x\)=\\sum\_\{t\\in p\}\\mathrm\{IDF\}\(t\)\\,\\frac\{\\mathrm\{tf\}\(t,x\)\\,\(k\_\{1\}\+1\)\}\{\\mathrm\{tf\}\(t,x\)\+k\_\{1\}\\\!\\Bigl\(1\-b\+b\\,\\tfrac\{\|x\|\}\{\\mathrm\{avgdl\}\}\\Bigr\)\},\(71\)and for each labely′y^\{\\prime\}we retrieve the top\-kkdocuments

𝒞plex​\(y′\)=arg⁡max⁡∑x∈SS⊆𝒟y′\|S\|=k⁡sBM25​\(p,x\)\.\\mathcal\{C\}\_\{p\}^\{\\mathrm\{lex\}\}\(y^\{\\prime\}\)=\\arg\\max\_\{\\begin\{subarray\}\{c\}S\\subseteq\\mathcal\{D\}\_\{y^\{\\prime\}\}\\\\ \|S\|=k\\end\{subarray\}\}\\sum\_\{x\\in S\}s\_\{\\mathrm\{BM25\}\}\(p,x\)\.\(72\)
For embedding\-based retrieval, we first compute cosine similarity

Scos​\(EI,E𝒟\)=EI⋅E𝒟‖EI‖2​‖E𝒟‖2,S\_\{\\cos\}\(E\_\{I\},E\_\{\\mathcal\{D\}\}\)=\\frac\{E\_\{I\}\\cdot E\_\{\\mathcal\{D\}\}\}\{\\\|E\_\{I\}\\\|\_\{2\}\\,\\\|E\_\{\\mathcal\{D\}\}\\\|\_\{2\}\},\(73\)and raw dot product

Sdp​\(EI,E𝒟\)=EI⋅E𝒟=∑i=1d\(EI\)i​\(E𝒟\)i\.S\_\{\\mathrm\{dp\}\}\(E\_\{I\},E\_\{\\mathcal\{D\}\}\)=E\_\{I\}\\cdot E\_\{\\mathcal\{D\}\}=\\sum\_\{i=1\}^\{d\}\(E\_\{I\}\)\_\{i\}\\,\(E\_\{\\mathcal\{D\}\}\)\_\{i\}\.\(74\)
We additionally assess two norm\-based distances: theL2L\_\{2\}distance

d2​\(EI,E𝒟\)=‖EI−E𝒟‖2=∑i=1d\(\(EI\)i−\(E𝒟\)i\)2,d\_\{2\}\(E\_\{I\},E\_\{\\mathcal\{D\}\}\)=\\\|E\_\{I\}\-E\_\{\\mathcal\{D\}\}\\\|\_\{2\}=\\sqrt\{\\sum\_\{i=1\}^\{d\}\\bigl\(\(E\_\{I\}\)\_\{i\}\-\(E\_\{\\mathcal\{D\}\}\)\_\{i\}\\bigr\)^\{2\}\},\(75\)and theL1L\_\{1\}distance

d1​\(EI,E𝒟\)=‖EI−E𝒟‖1=∑i=1d\|\(EI\)i−\(E𝒟\)i\|\.d\_\{1\}\(E\_\{I\},E\_\{\\mathcal\{D\}\}\)=\\\|E\_\{I\}\-E\_\{\\mathcal\{D\}\}\\\|\_\{1\}=\\sum\_\{i=1\}^\{d\}\\bigl\|\(E\_\{I\}\)\_\{i\}\-\(E\_\{\\mathcal\{D\}\}\)\_\{i\}\\bigr\|\.\(76\)
Finally, to capture distributional discrepancies we examine the Bray\-Curtis distance

dBC​\(EI,E𝒟\)=∑i=1d\|\(EI\)i−\(E𝒟\)i\|∑i=1d\|\(EI\)i\+\(E𝒟\)i\|,d\_\{\\mathrm\{BC\}\}\(E\_\{I\},E\_\{\\mathcal\{D\}\}\)=\\frac\{\\sum\_\{i=1\}^\{d\}\\bigl\|\(E\_\{I\}\)\_\{i\}\-\(E\_\{\\mathcal\{D\}\}\)\_\{i\}\\bigr\|\}\{\\sum\_\{i=1\}^\{d\}\\bigl\|\(E\_\{I\}\)\_\{i\}\+\(E\_\{\\mathcal\{D\}\}\)\_\{i\}\\bigr\|\},\(77\)and the Canberra distance

dCan​\(EI,E𝒟\)=∑i=1d\|\(EI\)i−\(E𝒟\)i\|\|\(EI\)i\|\+\|\(E𝒟\)i\|\.d\_\{\\mathrm\{Can\}\}\(E\_\{I\},E\_\{\\mathcal\{D\}\}\)=\\sum\_\{i=1\}^\{d\}\\frac\{\\bigl\|\(E\_\{I\}\)\_\{i\}\-\(E\_\{\\mathcal\{D\}\}\)\_\{i\}\\bigr\|\}\{\\bigl\|\(E\_\{I\}\)\_\{i\}\\bigr\|\+\\bigl\|\(E\_\{\\mathcal\{D\}\}\)\_\{i\}\\bigr\|\}\.\(78\)
Figure 11:Retrieval accuracy \(%\) by similarity metric for BGE\+BM25, BM25, and BGE\.Figure[11](https://arxiv.org/html/2608.18681#A2.F11)demonstrates that BGE\+BM25 outperforms both BM25 alone and BGE alone across all six metrics, achieving92\.60%92\.60\\%\(cosine\),89\.85%89\.85\\%\(dot product\),85\.43%85\.43\\%\(L2L\_\{2\}\),85\.22%85\.22\\%\(L1L\_\{1\}\),79\.21%79\.21\\%\(Bray\-Curtis\) and79\.12%79\.12\\%\(Canberra\)\. Pure BM25 and pure BGE match closely on cosine but degrade more sharply on norm\-and distribution\-based distances, confirming the robustness of the hybrid lexical\-semantic approach\.

### B\.11Hyperparameter Optimization and Reproducibility

To ensure fair and reproducible evaluation, all target models are fine\-tuned using a standardized hyperparameter optimization protocol\. We employ Bayesian optimization via Optuna to search over learning and regularization parameters, using validation accuracy as the objective\.

##### Tokenization and Input Representation\.

All premise–hypothesis pairs are tokenized using the RoBERTa tokenizer with a maximum sequence length of128128\. Inputs are padded and truncated to fixed length to ensure consistent batch construction across runs\. Each example is represented by input IDs, attention masks, and class labels\.

##### Training and Evaluation Splits\.

For efficiency during hyperparameter tuning, we use the full augmented training set and a fixed validation subset of1,2001\{,\}200examples\. Samples with undefined labels are removed prior to evaluation\. All datasets are formatted in PyTorch tensors\.

##### Search Space\.

We optimize the following hyperparameters:

η\\displaystyle\\eta∼LogUniform​\(10−6,10−4\),\\displaystyle\\sim\\text\{LogUniform\}\(10^\{\-6\},10^\{\-4\}\),\(79\)E\\displaystyle E∼\{1,2,3,5\},\\displaystyle\\sim\\\{1,2,3,5\\\},\(80\)B\\displaystyle B∼\{1,2,4,8,16\},\\displaystyle\\sim\\\{1,2,4,8,16\\\},\(81\)λ\\displaystyle\\lambda∼Uniform​\(10−4,10−2\),\\displaystyle\\sim\\text\{Uniform\}\(10^\{\-4\},10^\{\-2\}\),\(82\)whereη\\etadenotes the learning rate,EEthe number of training epochs,BBthe per\-device batch size, andλ\\lambdathe weight decay coefficient\.

##### Optimization Procedure\.

For each trial, a RoBERTa\-based classifier is fine\-tuned using the HuggingFaceTrainerframework\. Models are evaluated at the end of each epoch, and the best\-performing checkpoint is retained based on validation accuracy\. Early stopping is implicitly enforced by selecting the best epoch\. We perform4040independent trials and select the configuration that maximizes validation accuracy\.

All experiments are conducted on a single NVIDIA A100 GPU\. Each training epoch requires approximately3\.113\.11minutes on average\.

##### Evaluation Metric\.

All hyperparameter configurations are evaluated using classification accuracy:

Acc=1N∑i=1N𝕀\[y^i=yi\],\\text\{Acc\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbb\{I\}\\left\[\\hat\{y\}\_\{i\}=y\_\{i\}\\right\],\(83\)wherey^i\\hat\{y\}\_\{i\}andyiy\_\{i\}denote predicted and ground\-truth labels for sampleii, respectively\.

##### Reproducibility Measures\.

To reduce variance across runs, we fix random seeds for data sampling, model initialization, and optimization\. All experiments use identical preprocessing, prompt templates, and evaluation splits\. Hyperparameter search spaces, optimization budgets, and validation subsets are fully specified to enable exact replication of our results\.

The complete training and optimization scripts will be released upon publication\.

### B\.12Illustrative Example: Failure\-Mode Bandit Curation for NLI

Figure[12](https://arxiv.org/html/2608.18681#A2.F12)illustrates one iteration of the proposed framework on a Natural Language Inference \(NLI\) example\. The example is intended to show the operational flow of the method and does not represent a full training run\.

Given a premise and target label, the generator produces multiple candidate hypotheses conditioned on retrieved few\-shot examples\. The current target model first filters these candidates by retaining only those that induce an incorrect prediction\. The remaining candidates are then checked by an automated LLM judge ensemble to ensure label consistency\.

Validated failures are embedded and grouped into failure\-mode clusters\. A contextual\-bandit policy observes the state of each failure mode and selects which modes should be sampled under the adversarial budget\. The selected examples are mixed with original training data and used to retrain the target model\.

After retraining, validation performance provides rewardGtG\_\{t\}, which updates the policyπθ\\pi\_\{\\theta\}and criticRϕR\_\{\\phi\}\. Thus, the framework does not select examples by a fixed reward threshold; instead, it learns across iterations which recurring failure modes are most useful for improving robustness\.

Although the example is shown for NLI, the same failure filtering, automated validation, failure\-mode clustering, and policy\-guided sampling mechanism can be applied to other classification and reasoning tasks considered in this work\.

![Refer to caption](https://arxiv.org/html/2608.18681v1/method_new.png)Figure 12:Illustration of one failure\-mode contextual bandit curation iteration on a Natural Language Inference \(NLI\) example\. Given a premise and label, retrieval\-augmented prompting generates candidate hypotheses, which are first filtered by the target model to retain incorrect predictions and then validated by an automated LLM judge ensemble\. Validated failures are clustered into failure modes, and a contextual\-bandit policy selects which modes to sample for retraining under an adversarial budget\. Validation rewardGtG\_\{t\}updates the policyπθ\\pi\_\{\\theta\}and criticRϕR\_\{\\phi\}, enabling adaptive selection of high\-impact failure modes across iterations\.
### B\.13Sensitivity to Selection Threshold and Reward Noise

We analyze the robustness of our framework with respect to the selection thresholdτ\\tauand noise in reward estimation\. Sinceτ\\taucontrols the trade\-off between data quality and coverage, and reward estimates are derived from noisy downstream feedback, understanding their impact is critical for stable optimization\.

##### Sensitivity to Selection Threshold\.

We variedτ\\tauover a wide range relative to the empirical reward distribution, selecting values corresponding to the 60th, 70th, 80th, and 90th reward percentiles\. Lower thresholds admit more adversarial candidates, while higher thresholds enforce stricter filtering\. All experiments were conducted usingα=0\.83\\alpha=0\.83and a 1:4 mixing ratio\.

##### Reward Noise Injection\.

To simulate imperfect reward estimation, we injected additive Gaussian noise into the predicted reward:

r~​\(x\)=r⁡\(x\)\+ϵ,ϵ∼𝒩⁡\(0,σ2\),\\tilde\{r\}\(x\)=r\(x\)\+\\epsilon,\\quad\\epsilon\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}\),\(84\)whereσ\\sigmacontrols noise magnitude\. We evaluateσ∈\{0\.05,0\.1,0\.2\}\\sigma\\in\\\{0\.05,0\.1,0\.2\\\}, covering mild to severe corruption regimes\.

Table[10](https://arxiv.org/html/2608.18681#A2.T10)reports performance under varyingτ\\tauand noise levels on SNLI\. Performance remains stable across a broad operating region\. Moderate deviations from the default threshold \(τ⋆\\tau^\{\\star\}\) incur only minor degradation, and the system tolerates substantial reward noise before significant accuracy loss occurs\.

Table 10:Sensitivity to selection thresholdτ\\tauand reward noise \(SNLI, 1:4 gen:orig,α=0\.83\\alpha=0\.83\)\.Results indicate that the proposed framework operates in a broad stability regime\. The performance plateau aroundτ⋆\\tau^\{\\star\}suggests that the system is not finely tuned to a narrow threshold range\. Moreover, robustness to moderate reward noise is consistent with our theoretical bounded\-drift analysis \(Section 3\.4\), which guarantees controlled distributional evolution under noisy feedback\. Together, these findings demonstrate that our approach is resilient to practical imperfections in reward estimation and threshold calibration\.

### B\.14Example \- Few\-Shot Chat Sequence

##### BGE based retrieval

The chat sequences below present a clear few\-shot retrieval sequence for a natural language inference task\. They illustrate six premise\-hypothesis pairs\-two each for entailment, neutral, and contradiction\-and conclude with a concise model prompt\. This format makes the example selection process transparent and highlights the model’s reasoning in a single, easily readable block\. These examples are based solely on BGE retrieval\.

Example 1: Few\-Shot Retrieval & Model ReturnShot 1 Premise: A blond little girl enjoying a burrito\. Label: entailment\. Hypothesis:The girl ate a burrito\.Shot 2 Premise: A young blond girl sitting down while eating\. Label: entailment\. Hypothesis:The girl has food\.Shot 3 Premise: A blond little girl enjoying a burrito\. Label: neutral\. Hypothesis:The hungry girl ate a burrito at the restaurant\.Shot 4 Premise: A young blond girl sitting down while eating\. Label: neutral\. Hypothesis:The girl is eating at a picnic\.Shot 5 Premise: A blond little girl enjoying a burrito\. Label: contradiction\. Hypothesis:The brunette girl didn’t like the burrito\.Shot 6 Premise: A young blond girl sitting down while eating\. Label: contradiction\. Hypothesis:The girl runs all over her house while eating because she can never sit down\.Llama Generation User:Now generate a one\-sentence hypothesis thatcontradictsthe premise above\. Return only the hypothesis without narration\. Assistant \(Llama\):The girl threw the sandwich away\.

Example 2: Few\-Shot Retrieval & Model ReturnShot 1 Premise: An older man is drinking orange juice at a restaurant\. Label: entailment\. Hypothesis:A man is drinking juice\.Shot 2 Premise: An older man is drinking orange juice at a restaurant\. Label: entailment\. Hypothesis:A man is drinking juice\.Shot 3 Premise: An older man sits with his orange juice at a small table in a coffee shop while employees in bright colored shirts smile in the background\. Label: neutral\. Hypothesis:An elderly man sits in a small shop\.Shot 4 Premise: An older man sits with his orange juice at a small table in a coffee shop while employees in bright colored shirts smile in the background\. Label: neutral\. Hypothesis:An older man drinks his juice as he waits for his daughter to get off work\.Shot 5 Premise: An older man sits with his orange juice at a small table in a coffee shop while employees in bright colored shirts smile in the background\. Label: contradiction\. Hypothesis:A boy flips a burger\.Shot 6 Premise: An older man is drinking orange juice at a restaurant\. Label: contradiction\. Hypothesis:Two women are at a restaurant drinking wine\.Llama Generation User:Now generate a one\-sentence hypothesis thatcontradictsthe premise above\. Return only the hypothesis without narration\. Assistant \(Llama\):The coffee shop is empty\.

Example 3: Few\-Shot Retrieval & Model ReturnShot 1 Premise: A few people in a restaurant setting, one of them is drinking orange juice\. Label: entailment\. Hypothesis:The diners are at a restaurant\.Shot 2 Premise: A few people in a restaurant setting, one of them is drinking orange juice\. Label: entailment\. Hypothesis:The diners are at a restaurant\.Shot 3 Premise: A few people in a restaurant setting, one of them is drinking orange juice\. Label: neutral\. Hypothesis:The people are eating omelettes\.Shot 4 Premise: A few people in a restaurant setting, one of them is drinking orange juice\. Label: neutral\. Hypothesis:The people are eating omelettes\.Shot 5 Premise: A few people in a restaurant setting, one of them is drinking orange juice\. Label: contradiction\. Hypothesis:The people are sitting at desks in school\.Shot 6 Premise: A few people are observing the orange juice section at the grocery store\. Label: contradiction\. Hypothesis:The people are at a baseball stadium\.Llama Generation User:Now generate a one\-sentence hypothesis thatcontradictsthe premise above\. Return only the hypothesis without narration\. Assistant \(Llama\):The people are at a concert\.

##### Optimized BGE \+ BM25 Retrieval with Tuned Alpha

The paragraph below describes an optimized retrieval approach that combines semantic search using BGE embeddings with lexical scoring via BM25\. By introducing a tunable alpha parameter, the method linearly interpolates between embedding\-based similarity and keyword matching\-enabling a dynamic balance that maximizes relevance across diverse query types\. Through cross\-validation, the optimal alpha value was determined to achieve superior retrieval performance compared to either technique alone\.

Example 1: Few\-Shot Retrieval & Model ReturnShot 1 Premise: A person on a horse jumps over a broken down airplane\. Label: entailment\. Hypothesis:A person is outdoors, on a horse\.Shot 2 Premise: A person on a horse jumps over a broken down airplane\. Label: entailment\. Hypothesis:A person is outdoors, on a horse\.Shot 3 Premise: A person on a horse jumps over a broken down airplane\. Label: neutral\. Hypothesis:A person is training his horse for a competition\.Shot 4 Premise: A person is riding a horse, which is jumping over a grounded airplane\. Label: neutral\. Hypothesis:The horse jumps over a toy airplane\.Shot 5 Premise: A person on a horse jumps over a broken down airplane\. Label: contradiction\. Hypothesis:A person is at a diner, ordering an omelette\.Shot 6 Premise: A person on a horse jumps over a broken down airplane\. Label: contradiction\. Hypothesis:A person is at a diner, ordering an omelette\.Llama Generation User:Now generate a one\-sentence hypothesis that isneutralwith the premise above\. Return only the hypothesis without narration\. Assistant \(Llama\):The horse is very well trained\.

Example 2: Few\-Shot Retrieval & Model ReturnShot 1 Premise: Children smiling and waving at camera\. Label: entailment\. Hypothesis:There are children present\.Shot 2 Premise: Children smiling and waving at camera\. Label: entailment\. Hypothesis:There are children present\.Shot 3 Premise: Children smiling and waving at camera\. Label: neutral\. Hypothesis:They are smiling at their parents\.Shot 4 Premise: Children smiling and waving at camera\. Label: neutral\. Hypothesis:They are smiling at their parents\.Shot 5 Premise: Children smiling and waving at camera\. Label: contradiction\. Hypothesis:The kids are frowning\.Shot 6 Premise: Children smiling and making faces to the camera\. Label: contradiction\. Hypothesis:The children are deceased\.Llama Generation User:Now generate a one\-sentence hypothesis thatcontradictsthe premise above\. Return only the hypothesis without narration\. Assistant \(Llama\):The children are asleep\.

Example 3: Few\-Shot Retrieval & Model ReturnShot 1 Premise: An older man is drinking orange juice at a restaurant\. Label: entailment\. Hypothesis:A man is drinking juice\.Shot 2 Premise: An older man is drinking orange juice at a restaurant\. Label: entailment\. Hypothesis:A man is drinking juice\.Shot 3 Premise: An older man is drinking orange juice at a restaurant\. Label: neutral\. Hypothesis:A man in a restaurant is waiting for his meal to arrive\.Shot 4 Premise: An elderly man is drinking orange juice at a cafe\. Label: neutral\. Hypothesis:An older gentleman is enjoying his orange juice at a new cafe\.Shot 5 Premise: An older man is drinking orange juice at a restaurant\. Label: contradiction\. Hypothesis:Two women are at a restaurant drinking wine\.Shot 6 Premise: An older man is drinking orange juice at a restaurant\. Label: contradiction\. Hypothesis:Two women are at a restaurant drinking wine\.Llama Generation User:Now generate a one\-sentence hypothesis that isneutralwith the premise above\. Return only the hypothesis without narration\. Assistant \(Llama\):The restaurant has a lot of tables\.

##### BM25 based retrieval

The retrieval process relies exclusively on the BM25 algorithm, which computes scores based on term frequency, document frequency, and document length normalization\. By focusing on exact keyword matches and weighting rarer terms more heavily, BM25 effectively ranks documents that share the most relevant vocabulary with the query\. This approach is well\-suited for tasks where precise lexical overlap is paramount and semantic generalization is less critical\. However, it may miss contextually related content when synonyms or paraphrases are used\.

Example 1: Few\-Shot Retrieval & Model ReturnShot 1 Premise: An older man is drinking orange juice at a restaurant\. Label: entailment\. Hypothesis:A man is drinking juice\.Shot 2 Premise: An elderly man is drinking orange juice at a cafe\. Label: entailment\. Hypothesis:An old man is enjoying a beverage at a cafe\.Shot 3 Premise: An older man is drinking orange juice at a restaurant\. Label: neutral\. Hypothesis:A man in a restaurant is waiting for his meal to arrive\.Shot 4 Premise: An older man is drinking orange juice at a restaurant\. Label: neutral\. Hypothesis:A man in a restaurant is waiting for his meal to arrive\.Shot 5 Premise: An older man is drinking orange juice at a restaurant\. Label: contradiction\. Hypothesis:Two women are at a restaurant drinking wine\.Shot 6 Premise: An older man is drinking orange juice at a restaurant\. Label: contradiction\. Hypothesis:Two women are at a restaurant drinking wine\.Llama Generation User:Now generate a one\-sentence hypothesis thatentailsthe premise above\. Return only the hypothesis without narration\. Assistant \(Llama\):A man is drinking orange juice at a restaurant\.

Example 2: Few\-Shot Retrieval & Model ReturnShot 1 Premise: An older man is drinking orange juice at a restaurant\. Label: entailment\. Hypothesis:A man is drinking juice\.Shot 2 Premise: An older man is drinking orange juice at a restaurant\. Label: entailment\. Hypothesis:A man is drinking juice\.Shot 3 Premise: An older man is drinking orange juice at a restaurant\. Label: neutral\. Hypothesis:A man in a restaurant is waiting for his meal to arrive\.Shot 4 Premise: An older man is drinking orange juice at a restaurant\. Label: neutral\. Hypothesis:A man in a restaurant is waiting for his meal to arrive\.Shot 5 Premise: An older man is drinking orange juice at a restaurant\. Label: contradiction\. Hypothesis:Two women are at a restaurant drinking wine\.Shot 6 Premise: An elderly man is drinking orange juice at a cafe\. Label: contradiction\. Hypothesis:An elderly man is drinking apple juice at a bar\.Llama Generation User:Now generate a one\-sentence hypothesis thatcontradictsthe premise above\. Return only the hypothesis without narration\. Assistant \(Llama\):The man is drinking coffee\.

Example 3: Few\-Shot Retrieval & Model ReturnShot 1 Premise: A woman in a green jacket and hood over her head looking towards a valley\. Label: entailment\. Hypothesis:The woman is wearing green\.Shot 2 Premise: A woman in a green jacket and hood over her head looking towards a valley\. Label: entailment\. Hypothesis:The woman is wearing green\.Shot 3 Premise: A woman in a green jacket and hood over her head looking towards a valley\. Label: neutral\. Hypothesis:The woman is cold\.Shot 4 Premise: A woman gazes over the valley below\. Label: neutral\. Hypothesis:she looks at the valley she owns\.Shot 5 Premise: A woman in a green jacket and hood over her head looking towards a valley\. Label: contradiction\. Hypothesis:The woman is nake\.Shot 6 Premise: A woman in a green jacket and hood over her head looking towards a valley\. Label: contradiction\. Hypothesis:The woman is nake\.Llama Generation User:Now generate a one\-sentence hypothesis that isneutralwith the premise above\. Return only the hypothesis without narration\. Assistant \(Llama\):The woman is looking at something\.

### B\.15Prompt Design for Task\-Specific Candidate Generation

We employ task\-specific prompting strategies to guide large language models in generating adversarial candidates consistent with the target supervision signal\. Prompts are designed to be concise, label\-conditioned, and deterministic, ensuring controllable hypothesis synthesis and high semantic fidelity\. All prompts instruct the model to return only the generated output without additional narration\.

#### B\.15\.1SNLI Prompting Strategy

For the Natural Language Inference task, the goal is to generate a single\-sentence hypothesis whose semantic relation to the given premise matches a specified target labely∈\{entailment,neutral,contradiction\}y\\in\\\{\\text\{entailment\},\\text\{neutral\},\\text\{contradiction\}\\\}\. Given a premiseppand target labelyy, we use the following template:

> System:You are a language expert that helps create an NLI dataset\. Given a premise sentence and a desired label, your job is to provide a one\-sentence hypothesis, such that the label is relevant to the relation between the given premise and your generated hypothesis\. Make sure to keep the hypothesis short and no longer than a sentence\. User: Premise: \{premise\} Desired label: \{label\} Now generate a one sentence hypothesis that \{relation\} the premise above\. Return only the hypothesis without narration\.

Here,\{label\}corresponds to the target class \(entailment, neutral, contradiction\), and\{relation\}maps to the appropriate semantic relation \(“entails”, “is neutral with”, “contradicts”\)\. The prompt enforces minimal length and discourages explanatory text\.

We further employ low\-temperature decoding to reduce sampling variance and ensure consistent adversarial patterns across iterations\.

#### B\.15\.2FEVER Prompting Strategy

For the FEVER fact verification task, the objective is to generate evidence claims whose veracity can be evaluated with respect to a given document or knowledge source\. Given an evidence contexteeand a target labely∈\{SUPPORTS,REFUTES,NOT\_ENOUGH\_INFO\}y\\in\\\{\\text\{SUPPORTS\},\\text\{REFUTES\},\\text\{NOT\\\_ENOUGH\\\_INFO\}\\\}, we use the following template:

> System:You are a language expert that helps create fact verification datasets\. Given evidence text and a desired label, your job is to generate a single\-sentence claim that matches the specified verification outcome\. Make sure the claim is concise and factual\. User: Evidence: \{evidence\} Desired label: \{label\} Now generate a one sentence claim that is \{relation\} by the evidence above\. Return only the claim without narration\.

Here,\{label\}corresponds to the FEVER classes, and\{relation\}maps to “supported by”, “refuted by”, or “cannot be verified from”\. This formulation encourages the model to synthesize claims that are directly grounded in the provided evidence\.

As in the NLI setting, we constrain generation length and apply low\-temperature sampling to prioritize precision over diversity\.

#### B\.15\.3Design Rationale

Across tasks, prompt templates are designed to satisfy three principles: \(i\) explicit conditioning on the target label, \(ii\) minimal linguistic ambiguity, and \(iii\) strict output formatting\. This enables stable generation, reliable automated validation, and consistent reward estimation, facilitating effective failure\-aware adversarial data curation\.