Beyond Static Rules: Automated Discovery of Latent Vulnerabilities in Text-to-SQL

# Automated Discovery of Latent Vulnerabilities in Text-to-SQL
Source: [https://arxiv.org/html/2607.03833](https://arxiv.org/html/2607.03833)
Hanqing Wang1∗, Yongdong Chi1,Jian Yang2, Lei Yang3,Jiehui Zhao3,Yun Chen1,Guanhua Chen4 1Shanghai University of Finance and Economics,2Beihang University, 3Deepexi Technology Co\. Ltd\.,4Southern University of Science and Technology

###### Abstract

While Large Language Models \(LLMs\) have achieved remarkable success in Text\-to\-SQL tasks, their deployment in real\-world environments is hindered by latent reliability issues\. Identifying these latent weaknesses is critical for building trustworthy database interfaces, yet current diagnostic approaches rely heavily on static, expert\-defined rules, which lack the capability for systematic and automated exploration\. To bridge this gap, we propose SAGE \(Systematic Automated Guided Exploration\), a novel framework designed to autonomously uncover latent failure patterns in LLM\-based Text\-to\-SQL generation\. Specifically, SAGE generates vulnerability hypotheses for given samples and references a continuously evolving Vulnerability Codex to design targeted perturbations, thereby iteratively verifying and documenting potential defects\. Extensive experiments on state\-of\-the\-art open\-source LLMs demonstrate that SAGE uncovers a substantial number of failure cases, highlighting the significant fragility of current models\. Furthermore, our analysis reveals that the Vulnerability Codex exhibits strong cross\-model transferability, indicating that the discovered patterns represent generalized structural weaknesses\. Finally, we explore SAGE’s potential for remediation\. Although preliminary, lightweight fine\-tuning on the generated samples yields promising improvements, suggesting a scalable pathway for closing the reliability loop in future work\.

Beyond Static Rules: Automated Discovery of Latent Vulnerabilities in Text\-to\-SQL

Hanqing Wang1∗, Yongdong Chi1††thanks:Equal Contribution\., Jian Yang2,Lei Yang3,Jiehui Zhao3,Yun Chen1,Guanhua Chen4††thanks:Corresponding Author\.1Shanghai University of Finance and Economics,2Beihang University,3Deepexi Technology Co\. Ltd\.,4Southern University of Science and Technology

## 1Introduction

Text\-to\-SQL systems\(Androutsopouloset al\.,[1995](https://arxiv.org/html/2607.03833#bib.bib22); Li and Jagadish,[2014](https://arxiv.org/html/2607.03833#bib.bib23); Liet al\.,[2024b](https://arxiv.org/html/2607.03833#bib.bib24); Yuet al\.,[2018](https://arxiv.org/html/2607.03833#bib.bib6)\)serve as a vital bridge between natural language and structured data, democratizing data access by enabling non\-technical users to query complex databases intuitively\. With the recent integration of Large Language Models \(LLMs\)\(Pourreza and Rafiei,[2023](https://arxiv.org/html/2607.03833#bib.bib38); Xieet al\.,[2025](https://arxiv.org/html/2607.03833#bib.bib39); Liet al\.,[2023a](https://arxiv.org/html/2607.03833#bib.bib35)\), this field has witnessed a paradigm shift in semantic understanding and execution accuracy\.

Systematically identifying latent failure modes is the prerequisite for enhancing this reliability\. However, current evaluation paradigms predominantly rely on static benchmarks \(e\.g\., SpiderYuet al\.\([2018](https://arxiv.org/html/2607.03833#bib.bib6)\), BIRDLiet al\.\([2023b](https://arxiv.org/html/2607.03833#bib.bib77)\)\) or manual error analysisChanget al\.\([2023a](https://arxiv.org/html/2607.03833#bib.bib32)\); Ganet al\.\([2021b](https://arxiv.org/html/2607.03833#bib.bib5),[a](https://arxiv.org/html/2607.03833#bib.bib15)\); Patilet al\.\([2023](https://arxiv.org/html/2607.03833#bib.bib12)\); Piet al\.\([2022](https://arxiv.org/html/2607.03833#bib.bib17)\)\.

These approaches are inherently constrained by their reliance on human priors: they primarily verify model robustness against predefined challengesZhanget al\.\([2024](https://arxiv.org/html/2607.03833#bib.bib8)\); Denget al\.\([2021](https://arxiv.org/html/2607.03833#bib.bib58)\); Bhaskaret al\.\([2023](https://arxiv.org/html/2607.03833#bib.bib13)\), but cannot provide a systematic exploration of potential or previously unidentified vulnerabilities111Throughout this paper, “vulnerability” refers to robustness and reliability brittleness under semantic\-preserving perturbations, rather than database security exploits such as SQL injection\.\. As a result, certain failure modesSaparina and Lapata \([2024](https://arxiv.org/html/2607.03833#bib.bib14)\); Sahitajet al\.\([2025](https://arxiv.org/html/2607.03833#bib.bib64)\); Changet al\.\([2023b](https://arxiv.org/html/2607.03833#bib.bib16)\)may remain insufficiently examined and can have a substantial impact on the reliability of Text\-to\-SQL generation beyond what is captured by fixed heuristics\. Consequently, there is a pressing need for an automated framework capable of proactively exploring the model’s failure boundaries\.

To bridge this gap, we propose SAGE \(SystematicAutomatedGuidedExploration\), a framework designed to proactively expose latent vulnerabilities in LLM\-based Text\-to\-SQL generation\. Unlike passive verification methodsChanget al\.\([2023a](https://arxiv.org/html/2607.03833#bib.bib32)\); Ganet al\.\([2021b](https://arxiv.org/html/2607.03833#bib.bib5),[a](https://arxiv.org/html/2607.03833#bib.bib15)\); Patilet al\.\([2023](https://arxiv.org/html/2607.03833#bib.bib12)\); Piet al\.\([2022](https://arxiv.org/html/2607.03833#bib.bib17)\), SAGE orchestrates an active discovery process\. Specifically, it first formulates specific vulnerability hypotheses for a target sample, and then references a continuously evolving Vulnerability Codex, a dynamic repository of generalized error archetypes, to synthesize targeted perturbed samples\. This allows SAGE to systematically verify potential weaknesses and iteratively refine its understanding of the model’s failure boundaries, transforming the discovery of vulnerabilities from a static manual task into an autonomous, self\-improving cycle\.

Our main contributions are as follows:

- •We proposeSAGE, a framework that autonomously evolves aVulnerability Codexto guide exploration\. It outperforms static expert\-curated baselines, achieving average VER margins of 26\.33% and 18\.66% on BIRD and Spider across different models, respectively\. These results indicate the effectiveness of our dynamic, automated strategy compared to fixed manual heuristics\.
- •We conduct a systematic vulnerability assessment of LLMs in Text\-to\-SQL\. Our evaluation across diverse models on BIRD and Spider yields average Vulnerability Exposure Rates \(VER\) of 76\.98% and 58\.45%, respectively\. These figures provide empirical evidence of the prevalence of latent vulnerabilities within current Text\-to\-SQL systems\.
- •Beyond detection, we take an initial step towards establishing a reliability loop\. Lightweight fine\-tuning on 1\.5k samples partially alleviates the model’s defects\. These findings, though preliminary, shed light on a scalable pathway for model improvement and validate the potential of usingSAGEfor vulnerability remediation\.222Our code is available at[https://github\.com/sustech\-nlp/SAGE](https://github.com/sustech-nlp/SAGE)\.

## 2Related Work

### 2\.1LLM\-based Text\-to\-SQL Generation

With the rapid advancement of LLMs, LLM\-based approaches to Text\-to\-SQL have demonstrated significantly enhanced transferability and reasoning capabilities compared to traditional rule\-based systemsMahmudet al\.\([2015](https://arxiv.org/html/2607.03833#bib.bib27)\); Lyuet al\.\([2020](https://arxiv.org/html/2607.03833#bib.bib28)\), neural modelsChoiet al\.\([2021](https://arxiv.org/html/2607.03833#bib.bib33)\); Xuet al\.\([2017](https://arxiv.org/html/2607.03833#bib.bib34)\), and pretrained modelsLiet al\.\([2023a](https://arxiv.org/html/2607.03833#bib.bib35)\); Yinet al\.\([2020](https://arxiv.org/html/2607.03833#bib.bib36)\), propelling the field into a new stageHonget al\.\([2024](https://arxiv.org/html/2607.03833#bib.bib79)\); Shiet al\.\([2025](https://arxiv.org/html/2607.03833#bib.bib37)\)\. Current methodologies primarily follow two paradigms: prompt engineering and fine\-tuning\. Prompt engineering methods, typically utilizing closed\-source LLMs, enhance generation quality through techniques such as task decompositionPourreza and Rafiei \([2023](https://arxiv.org/html/2607.03833#bib.bib38)\); Xieet al\.\([2025](https://arxiv.org/html/2607.03833#bib.bib39)\); Chiet al\.\([2025](https://arxiv.org/html/2607.03833#bib.bib4)\), chain\-of\-thought \(CoT\) reasoningXuet al\.\([2024](https://arxiv.org/html/2607.03833#bib.bib42)\); Liuet al\.\([2025](https://arxiv.org/html/2607.03833#bib.bib41)\), and multi\-agent collaborationXiaet al\.\([2024](https://arxiv.org/html/2607.03833#bib.bib43)\); Denget al\.\([2025](https://arxiv.org/html/2607.03833#bib.bib44)\)\. Conversely, fine\-tuning methods leverage open\-source LLMs to reduce inference costs while improving privacy, controllability, and stabilityLiet al\.\([2025](https://arxiv.org/html/2607.03833#bib.bib45)\); Quet al\.\([2025](https://arxiv.org/html/2607.03833#bib.bib46)\); Guoet al\.\([2025](https://arxiv.org/html/2607.03833#bib.bib47)\); Liet al\.\([2024a](https://arxiv.org/html/2607.03833#bib.bib48)\); Shenget al\.\([2025](https://arxiv.org/html/2607.03833#bib.bib49)\); Chenget al\.\([2025](https://arxiv.org/html/2607.03833#bib.bib7)\)\.

Despite their success, the primary objective of these works is to maximize execution accuracy, optimizing the probability of generating correct SQL queries\. A specific line of research within this domain—automated self\-correction—shares a superficial resemblance to our work by identifying and fixing errorsShenet al\.\([2025](https://arxiv.org/html/2607.03833#bib.bib18)\); Gonget al\.\([2025](https://arxiv.org/html/2607.03833#bib.bib20)\); Askariet al\.\([2025](https://arxiv.org/html/2607.03833#bib.bib26)\)\. However, their fundamental goals and mechanisms differ significantly\. These methods employ correction solely as an intermediate step to boost performance on specific samples, often addressing errors in an ad\-hoc manner without analyzing the underlying causesLiu and Tan \([2024](https://arxiv.org/html/2607.03833#bib.bib25)\); Pourreza and Rafiei \([2023](https://arxiv.org/html/2607.03833#bib.bib38)\)\. In contrast, our work shifts the focus fromperformance enhancementtosystematic vulnerability discovery\. We aim not merely to correct individual instances but to abstract high\-level failure patterns and diagnose latent defects in the model’s reasoning process, providing generalized insights that go beyond instance\-level repair\.

![Refer to caption](https://arxiv.org/html/2607.03833v1/x1.png)Figure 1:The architecture of the proposed automated vulnerability discovery framework,SAGE\. The process iterates between two stages: \(1\) Discovery, where the system utilizes existing insights from the central Vulnerability Codex to generate and verify hard adversarial samples \(Steps 1\-5\); and \(2\) Evolution, where identified failures are abstracted into generalized archetypes and merged back into the Codex through semantic compression \(Steps 6\-7\)\. This closed\-loop mechanism allows the system to continuously update its strategy in subsequent iterations \(Step 8\)\.
### 2\.2Robustness and Brittleness in LLM\-based Text\-to\-SQL Generation

Prompt\-engineering and fine\-tuning strategies have continuously advanced Text\-to\-SQL performance on foundational benchmarks such as SpiderYuet al\.\([2018](https://arxiv.org/html/2607.03833#bib.bib6)\)and BIRDLiet al\.\([2023b](https://arxiv.org/html/2607.03833#bib.bib77)\)\.

However, existing approaches to evaluating model reliability are largely limited to confirming the presence of known failure patterns under specific conditions\. Researchers have established robustness benchmarks by applying semantic\-preserving perturbationsDenget al\.\([2021](https://arxiv.org/html/2607.03833#bib.bib58)\); Ganet al\.\([2021b](https://arxiv.org/html/2607.03833#bib.bib5),[a](https://arxiv.org/html/2607.03833#bib.bib15)\); Sahitajet al\.\([2025](https://arxiv.org/html/2607.03833#bib.bib64)\)or predefined rule\-based alterations to schemas and queriesChanget al\.\([2023b](https://arxiv.org/html/2607.03833#bib.bib16)\); Patilet al\.\([2023](https://arxiv.org/html/2607.03833#bib.bib12)\)\. While valuable, these methods operate within a closed scope defined by human experts—they merely verify whether an LLM succumbs to anticipated error types, such as structural ambiguityDinget al\.\([2025](https://arxiv.org/html/2607.03833#bib.bib19)\); Bhaskaret al\.\([2023](https://arxiv.org/html/2607.03833#bib.bib13)\), but fail to explore the model’s “blind spots” outside these heuristicsSaparina and Lapata \([2024](https://arxiv.org/html/2607.03833#bib.bib14)\); Piet al\.\([2022](https://arxiv.org/html/2607.03833#bib.bib17)\); Zhanget al\.\([2024](https://arxiv.org/html/2607.03833#bib.bib8)\)\. Although recent efforts pursue automated bias or failure discovery in other LLM settingsLaiet al\.\([2026](https://arxiv.org/html/2607.03833#bib.bib3)\), systematic vulnerability exploration for Text\-to\-SQL remains largely open\.

This highlights the need for an efficient, self\-evolving, and automated framework capable of discovering latent failure patterns\. Unlike previous works, our work aims to systematically expose unknown defect patterns, providing actionable insights for the fundamental robustness of Text\-to\-SQL systems\.

## 3Method

To systematically uncover latent vulnerabilities in Text\-to\-SQL systems, we proposeSAGE, an iterative framework designed to autonomously discover and evolve vulnerability insights\. As illustrated in Figure[1](https://arxiv.org/html/2607.03833#S2.F1),SAGEoperates as a closed\-loop system structured around two complementary modules: theVulnerability Discovery Module\(Section[3\.2](https://arxiv.org/html/2607.03833#S3.SS2)\), which actively probes the target model through hypothesis\-driven perturbations, and theCodex Evolution & Management Module\(Section[3\.3](https://arxiv.org/html/2607.03833#S3.SS3)\), which abstracts validated failures into a self\-refiningVulnerability Codex\. The detailed algorithmic procedure is provided in Algorithm[1](https://arxiv.org/html/2607.03833#alg1)\.

### 3\.1Problem Formulation

We formalize the problem of automatic vulnerability discovery within the Text\-to\-SQL paradigm\. LetD=\{\(qi,si,yi∗\)\}i=1ND=\\\{\(q\_\{i\},s\_\{i\},y^\{\*\}\_\{i\}\)\\\}\_\{i=1\}^\{N\}denote a dataset whereqiq\_\{i\}is the natural language query,sis\_\{i\}is the database schema, andyi∗y^\{\*\}\_\{i\}is the ground\-truth execution result, on which the target modelMtM\_\{t\}performs correctly under standard conditions, i\.e\.,Exec​\(Mt​\(qi,si\)\)=yi∗\\mathrm\{Exec\}\(M\_\{t\}\(q\_\{i\},s\_\{i\}\)\)=y^\{\*\}\_\{i\}\. This selection ensures that any subsequent failure induced by our framework reflects a genuine blind spot or fragility in the model rather than an inherent lack of reasoning ability\.

Our objective is to construct and evolve aVulnerability Codex, denoted as𝒱\\mathcal\{V\}, which systematically captures the latent failure modes ofMtM\_\{t\}\. We defineSAGEas an iterative process governed by two mapping functions,⟨Φdisc,Φevol⟩\\langle\\Phi\_\{\\text\{disc\}\},\\Phi\_\{\\text\{evol\}\}\\rangle, operating over iterationst=1,…,Tt=1,\\dots,T\.

Let𝒱t−1\\mathcal\{V\}\_\{t\-1\}denote the Codex state from the previous iteration\. At each steptt:

##### Discovery \(Φdisc\\Phi\_\{\\text\{disc\}\}\):

The system synthesizes hypotheses and constructs specific perturbed samples to probeMtM\_\{t\}, yielding a set of raw failure casesℱt\\mathcal\{F\}\_\{t\}whereMtM\_\{t\}exposes vulnerability:

ℱt=Φdisc​\(D,𝒱t−1\)\\mathcal\{F\}\_\{t\}=\\Phi\_\{\\text\{disc\}\}\(D,\\mathcal\{V\}\_\{t\-1\}\)\(1\)

##### Evolution \(Φevol\\Phi\_\{\\text\{evol\}\}\):

These raw failures are abstracted, compressed, and integrated into the Codex to produce an updated state𝒱t\\mathcal\{V\}\_\{t\}:

𝒱t=Φevol​\(𝒱t−1,ℱt\)\\mathcal\{V\}\_\{t\}=\\Phi\_\{\\text\{evol\}\}\(\\mathcal\{V\}\_\{t\-1\},\\mathcal\{F\}\_\{t\}\)\(2\)
The process continues until the Codex stabilizes or the maximum iteration budget is reached\.

### 3\.2Vulnerability Discovery Module

This module executes the active probing phase of our framework\. It transforms abstract vulnerability hypotheses into concrete, context\-aware adversarial samples to expose latent defects\. This process follows a coarse\-to\-fine pipeline: fromHypothesestoScopes, and finally toSample Generation\.

#### 3\.2\.1Systematic Hypothesis Generation

Diverging from stochastic perturbation methods, we initiate discovery by synthesizing targetedVulnerability Hypotheses\. A Generator Agent \(MgM\_\{g\}\) analyzes each data sampledi∈Dd\_\{i\}\\in Dto identify potential vulnerabilities \(e\.g\., “Ambiguity in column selection” or “Misinterpretation of complex nested logic”\)\. This yields a set of candidates𝒞c​u​r​r=⋃di∈D\{\(di,hi,k\)\}k=1K\\mathcal\{C\}\_\{curr\}=\\bigcup\_\{d\_\{i\}\\in D\}\\\{\(d\_\{i\},h\_\{i,k\}\)\\\}\_\{k=1\}^\{K\}, where eachhi,kh\_\{i,k\}represents a specific conceptual strategy suspected to induce failure \(step 1 in Figure[1](https://arxiv.org/html/2607.03833#S2.F1)\)\.

To ensure structural grounding, these hypotheses are mapped to specificPerturbation Scopes𝕊\\mathbb\{S\}\(e\.g\., logical constraints in the query or schema definitions\)\. In the initial phase \(t=1t=1\), we adopt an efficient exploration strategy by mapping each hypothesis to a single scope; in subsequent iterations, a comprehensive scan is performed by applying hypotheses across all available scopes\.

#### 3\.2\.2Experience\-Guided Probing & Verification

To rigorously verify whether a candidate hypothesisCj=\(di,hi,k\)C\_\{j\}=\(d\_\{i\},h\_\{i,k\}\)exposes actual vulnerabilities,SAGEreferences theVulnerability Codexto guide the generation process\. We employ an Embedding Model \(MeM\_\{e\}\) to retrieve relevant insightsEjE\_\{j\}from𝒱t−1\\mathcal\{V\}\_\{t\-1\}\. This serves as a reference for implementation feasibility rather than content limitation, ensuring that the verification focuses on the hypothesis’s validity \(step 2 in Figure[1](https://arxiv.org/html/2607.03833#S2.F1)\)\.

Ej=RetrieveTop5​\(Me,hi,k,𝒱t−1\)E\_\{j\}=\\mathrm\{RetrieveTop5\}\(M\_\{e\},h\_\{i,k\},\\mathcal\{V\}\_\{t\-1\}\)\(3\)
Guided by the candidateCjC\_\{j\}, the target scopesm∈𝕊s\_\{m\}\\in\\mathbb\{S\}, and the retrieved experienceEjE\_\{j\}, the GeneratorMgM\_\{g\}constructs a perturbed samplexj,mx\_\{j,m\}\(step 3 in Figure[1](https://arxiv.org/html/2607.03833#S2.F1)\):

xj,m=SampleGenerate​\(Mg,Cj,sm,Ej\)x\_\{j,m\}=\\mathrm\{SampleGenerate\}\(M\_\{g\},C\_\{j\},s\_\{m\},E\_\{j\}\)\(4\)
To ensurexj,mx\_\{j,m\}preserves the original semantic logic \(i\.e\., the ground truth execution result remains invariant\), we introduce a Checker Agent \(McM\_\{c\}\) to validate the qualification ofxj,mx\_\{j,m\}\(step 4 in Figure[1](https://arxiv.org/html/2607.03833#S2.F1)\)\.

IsValid=ValidCheck​\(Mc,Cj,xj,m\)\\mathrm\{IsValid\}=\\mathrm\{ValidCheck\}\(M\_\{c\},C\_\{j\},x\_\{j,m\}\)\(5\)
Given the complexity of semantic verification, we validate the Checker Agent’s reliability through both human evaluation and cross\-model proxy validation \(see Appendix[B\.6](https://arxiv.org/html/2607.03833#A2.SS6)\)\.

Upon validation, we probe the target modelMtM\_\{t\}withxj,mx\_\{j,m\}to obtain the predicted SQL execution resultyj,my\_\{j,m\}\. A raw vulnerability is identified if the perturbation is valid, but the model execution deviates from the ground truth \(step 5 in Figure[1](https://arxiv.org/html/2607.03833#S2.F1)\):

IsCorrect=\(Exec\(Mt\(xj,m\)\)==yi∗\)\\mathrm\{IsCorrect\}=\(\\mathrm\{Exec\}\(M\_\{t\}\(x\_\{j,m\}\)\)==y^\{\*\}\_\{i\}\)\(6\)
IfIsCorrect\\mathrm\{IsCorrect\}is false, the sample reveals a genuine vulnerability, triggering the evolution phase\.

### 3\.3Codex Evolution & Management Module

While the Discovery Module exposes individual failures, the Evolution Module functions as the system’s cognitive repository, responsible for distilling knowledge\. It converts noisy, instance\-specific failures into a high\-quality, compactVulnerability Codex\(𝒱\\mathcal\{V\}\)\. This involves two key stages:AbstractionandRefinement\.

#### 3\.3\.1Abstraction via Summarization

Raw failure cases are often coupled with specific schema entities or query phrasings\. To foster generalization, a Summarizer Agent \(MsM\_\{s\}\) analyzes the discovered failure tuple\(Cj,xj,m,yj,m\)\(C\_\{j\},x\_\{j,m\},y\_\{j,m\}\)\(step 6 in Figure[1](https://arxiv.org/html/2607.03833#S2.F1)\)\. It abstracts the concrete error into a structuredGeneralized Error Archetypevjv\_\{j\}, which encapsulates a generalized error summary and causal analysis\.

vj=VulnerAbstraction​\(Ms,Cj,xj,m,yj,m\)v\_\{j\}=\\mathrm\{VulnerAbstraction\}\(M\_\{s\},C\_\{j\},x\_\{j,m\},y\_\{j,m\}\)\(7\)
These abstracted archetypes are collected into a temporary set𝒱new\(t\)\\mathcal\{V\}\_\{\\text\{new\}\}^\{\(t\)\}:

𝒱new\(t\)=𝒱new\(t\)∪\{vj\}\\mathcal\{V\}\_\{\\text\{new\}\}^\{\(t\)\}=\\mathcal\{V\}\_\{\\text\{new\}\}^\{\(t\)\}\\cup\\\{v\_\{j\}\\\}\(8\)

#### 3\.3\.2Refinement via Semantic Compression

As the iterative process proceeds, simply accumulating archetypes in𝒱new\(t\)\\mathcal\{V\}\_\{\\text\{new\}\}^\{\(t\)\}may lead to a redundant repository\. To address this, we introduce aSemantic Compressionmechanism during the construction of𝒱t\\mathcal\{V\}\_\{t\}\(Phase 3 in Algorithm[1](https://arxiv.org/html/2607.03833#alg1)\)\.

At the conclusion of each iteration, newly discovered archetypes are merged with the existing Codex \(step 7 in Figure[1](https://arxiv.org/html/2607.03833#S2.F1)\)\. We perform semantic clustering usingMeM\_\{e\}: if the Euclidean distance between the embeddings of two vulnerability entries falls below a similarity thresholdτ\\tau, they are identified as redundant and compressed\.

Formally, this refinement updates the Codex by:

𝒱t=SemanticComp​\(Me,𝒱t−1∪𝒱new\(t\),τ\)\\mathcal\{V\}\_\{t\}=\\mathrm\{SemanticComp\}\(M\_\{e\},\\mathcal\{V\}\_\{t\-1\}\\cup\\mathcal\{V\}\_\{\\text\{new\}\}^\{\(t\)\},\\tau\)\(9\)
This refinement step ensures that the Vulnerability Codex𝒱\\mathcal\{V\}remains compact and distinct, improving retrieval efficiency and enabling the system to converge towards a representative set of failure modes on Text\-to\-SQL generation\.

## 4Experiment

### 4\.1Setup

##### Datasets

We evaluateSAGEusing theSpiderYuet al\.\([2018](https://arxiv.org/html/2607.03833#bib.bib6)\)andBIRDLiet al\.\([2023b](https://arxiv.org/html/2607.03833#bib.bib77)\)datasets\. To rigorously benchmark vulnerability, we define our evaluation set as the specific subset of samples where the target model yields the correct ground\-truth execution result, filtering out instances of initial failure\. The exact cardinalities of this initially\-correct subset\|D\|\|D\|for each model/dataset pair are reported in Appendix Table[5](https://arxiv.org/html/2607.03833#A2.T5)\.

##### Models

For our main experimental evaluations, we select three representative models that span distinct paradigms as our target models: a general\-purpose model, Gemma\-3Teamet al\.\([2025](https://arxiv.org/html/2607.03833#bib.bib9)\)\(Gemma\-3\-12B\-it\); a domain\-adapted model, OmniSQLLiet al\.\([2025](https://arxiv.org/html/2607.03833#bib.bib45)\)\(OmniSQL\-32B\); and a reinforcement\-learned model, Inf\-rl\-qwenInfly and InfTech \([2025](https://arxiv.org/html/2607.03833#bib.bib29)\)\(inf\-rl\-qwen\-coder\-32B\-2746\)\. Additionally, we demonstrate the applicability of our approach to powerful proprietary models by evaluating onGPT\-4oHurstet al\.\([2024](https://arxiv.org/html/2607.03833#bib.bib21)\)\(see Appendix[A\.2](https://arxiv.org/html/2607.03833#A1.SS2)\)\. For fine\-tuning experiments in Section[5\.2](https://arxiv.org/html/2607.03833#S5.SS2), we use Qwen2\.5\-Coder\(Qwen2\.5\-Coder\-7B\-Instruct\), considering training cost\.

ModelSettingBIRDSpiderEX\(↓\\downarrow\)VES\(↓\\downarrow\)VER\(↑\\uparrow\)ApD\(↓\\downarrow\)EX\(↓\\downarrow\)VES\(↓\\downarrow\)VER\(↑\\uparrow\)ApD\(↓\\downarrow\)Gemma\-3Original53\.6563\.75––81\.53109\.35––Expert24\.6429\.3854\.077\.7348\.3661\.8352\.1610\.95SAGE8\.349\.8684\.455\.0812\.3815\.7266\.327\.85Inf\-rl\-qwenOriginal70\.5387\.03––87\.72116\.38––Expert38\.2044\.8745\.849\.8163\.9382\.1127\.1219\.97SAGE21\.9026\.0368\.957\.2756\.1971\.0548\.0712\.93OmniSQLOriginal63\.2374\.77––81\.83110\.85––Expert30\.3136\.6352\.067\.0948\.8463\.5740\.1010\.60SAGE14\.2117\.0677\.535\.4731\.8240\.3560\.978\.15

Table 1:Main results onBIRDandSpiderdatasets\. We reportEX\(Execution Accuracy\),VES\(Valid Efficiency Score\),VER\(Vulnerability Exposure Rate\), andApD\(Attempts per Discovery\)\. Arrows indicate whether higher \(↑\\uparrow\) or lower \(↓\\downarrow\) values are preferred\.*Original*denotes the standard baseline performance\.*Expert*and*SAGE*represent vulnerability probing methods via manual rules and our automated framework, respectively\. Note that VER and ApD quantify the discovery capability and are not applicable \(–\) to the static Original baseline\.
##### Implementation Details

We initialize the Vulnerability Codex using a repository of expert\-defined vulnerabilities curated from established studies, including Dr\.Spider, SpiderSyn, and ADVETAChanget al\.\([2023b](https://arxiv.org/html/2607.03833#bib.bib16)\); Ganet al\.\([2021a](https://arxiv.org/html/2607.03833#bib.bib15)\); Piet al\.\([2022](https://arxiv.org/html/2607.03833#bib.bib17)\)\. We employQwen3\-32BYanget al\.\([2025](https://arxiv.org/html/2607.03833#bib.bib30)\)as the backbone for the Generator, Checker, and Summarizer components, whileQwen3\-Embedding\-4BZhanget al\.\([2025](https://arxiv.org/html/2607.03833#bib.bib31)\)serves as the embedding model\. Unless otherwise specified, we useT=3T=3iterations,K=3K=3initial hypotheses per sample, Top\-5 Codex retrieval, and a semantic compression thresholdτ=0\.1\\tau=0\.1; the search over a hypothesis stops once a valid failure is exposed or the maximum iteration budget is reached\. We follow each model’s default chat sampling settings during generation \(for Qwen3\-32B,t​e​m​p​e​r​a​t​u​r​e=0\.6temperature=0\.6,t​o​pk=20top\_\{k\}=20, andt​o​pp=0\.95top\_\{p\}=0\.95\)\. A consolidated summary of these settings, together with the exact\|D\|\|D\|values, is provided in Appendix Table[5](https://arxiv.org/html/2607.03833#A2.T5)\. Full details regarding the prompts used for sample perturbation are provided in Appendix[B\.5](https://arxiv.org/html/2607.03833#A2.SS5)\. Given the critical role of semantic verification, we provide a dedicated validation of the Checker module in Appendix[B\.6](https://arxiv.org/html/2607.03833#A2.SS6)\.

For fine\-tuning experiments involvingQwen2\.5\-Coder\-7B\-Instruct, we utilize LoRA for parameter\-efficient adaptation\. Our training pipeline is built upon thellama\-factory333[https://github\.com/hiyouga/LlamaFactory](https://github.com/hiyouga/LlamaFactory)framework\. Comprehensive hyperparameter configurations are listed in Appendix Table[7](https://arxiv.org/html/2607.03833#A2.T7)\.

##### Baselines

To demonstrate the effectiveness ofSAGEin discovering latent vulnerabilities on text\-to\-SQL generation, we introduce such baselines:

∙\\bulletOriginalRepresents the standard performance of the target model on the unperturbed benchmarks \(Spider and BIRD\), serving as the reference for initial capability\.

∙\\bulletExpertDue to the absence of direct baselines for automated vulnerability discovery in this specific context, we construct a strong baseline derived from prior work\. This method executes vulnerability discovery using a fixed repository of expert\-defined patterns \(from Dr\.Spider, SpiderSyn and ADVETA\) without the dynamic update mechanism of our framework\. This comparison is designed to highlight the advantage of evolving the Vulnerability Codex versus relying solely on static expert knowledge\.

##### Metrics

We evaluate model performance using three complementary metrics:

∙\\bulletExecution Accuracy \(EX\):EX measures the ratio of correctly predicted SQL programs by comparing their execution results with those of the ground\-truth SQL programs on the same database instance\.

∙\\bulletValid Efficiency Score \(VES\):VES evaluates the efficiency of correctly predicted SQL programs by considering their execution time\. Formally, VES is given by:

VES=1N​∑n=1N𝟙​\(n\)⋅R​\(n\)\\mathrm\{VES\}=\\frac\{1\}\{N\}\\sum^\{N\}\_\{n=1\}\\mathds\{1\}\(n\)\\cdot R\(n\), WhereR​\(n\)R\(n\)denotes the ratio between the execution time of the predicted SQL and the gold SQL for the n\-th text query\. The indicator function𝟙​\(n\)\\mathds\{1\}\(n\)equals 1 if the predicted SQL is correct, and 0 otherwise\.

∙\\bulletVulnerability Exposure Rate \(VER\):This metric quantifies the method’s*capability*to uncover latent defects in scenarios where the model typically performs well\. We focus specifically on the set of instancesDDthat the target model solves correctly under standard baseline conditions\. LetDe​x​p​o​s​e​d⊆DD\_\{exposed\}\\subseteq Ddenote the subset of these instances for which our method successfully induces a valid failure\. The VER is defined as:

VER=\|De​x​p​o​s​e​d\|\|D\|\.\\mathrm\{VER\}\\;=\\;\\frac\{\|D\_\{exposed\}\|\}\{\|D\|\}\.A higher VER indicates a stronger capability of the evaluation framework to penetrate the model’s superficial robustness and reveal deep\-seated vulnerabilities that remain hidden during standard testing\.

∙\\bulletAttempts per Discovery \(ApD\):We introduce this metric to quantify the*efficiency*of capturing valid failure patterns\. Letmmdenote the total number of interaction attempts, andnndenote the number of*distinct, validated discoveries*\(deduplicated by root issue\)\. We define ApD as:

ApD=mmax⁡\(n,1\)\.\\mathrm\{ApD\}\\;=\\;\\frac\{m\}\{\\max\(n,\\,1\)\}\.This ratio reflects thediscovery efficiency: a lower ApD indicates a highly efficient process where valid issues are surfaced with fewer interactions\.

### 4\.2Main Results

Table[1](https://arxiv.org/html/2607.03833#S4.T1)presents the comparative evaluation results on the BIRD and Spider benchmarks\. We assess model performance under the standard setting \(Original\) against two vulnerability discovery settings: the rule\-basedExpertbaseline and our proposed automated framework,SAGE\. The results indicate a substantial performance gap between the standard evaluation and the vulnerability discovery settings across both datasets\. On theBIRDdataset, while the evaluated models achieve an average Execution Accuracy \(EX\) of62\.47%62\.47\\%in theOriginalsetting, this metric decreases to an average of14\.82%14\.82\\%under theSAGEsetting\. A similar trend is observed onSpider, where the average EX drops from83\.69%83\.69\\%to33\.46%33\.46\\%when subjected to our framework\. These declines in EX and the corresponding Valid Efficiency Score \(VES\) suggest that widely used benchmarks may not fully reflect the reliability of models when facing semantically equivalent but adversarial contexts constructed for vulnerability discovery\.

Comparing the two discovery methods,SAGEdemonstrates consistent improvements over theExpertbaseline in terms of both discovery capability and efficiency\. Regarding capability,SAGEexposes a higher proportion of vulnerabilities across both benchmarks\. On BIRD, our method achieves an average Vulnerability Exposure Rate \(VER\) of76\.98%76\.98\\%, surpassing theExpertbaseline’s average of50\.66%50\.66\\%\. Similarly, on Spider,SAGEoutperforms the baseline with an average VER of58\.45%58\.45\\%compared to39\.79%39\.79\\%\. We perform an analysis of discovered vulnerability patterns in Appendix[A\.1](https://arxiv.org/html/2607.03833#A1.SS1)for a better understanding of the effectiveness ofSAGE\. In terms of efficiency,SAGErequires fewer interactions to identify valid issues, as evidenced by the Attempts per Discovery \(ApD\) metric\. The average ApD on BIRD decreases from8\.218\.21with theExpertmethod to5\.945\.94withSAGE, and on Spider, it decreases from13\.8413\.84to9\.649\.64, indicating a more targeted discovery process\.

The quantitative advantage ofSAGEcan be attributed to its dynamic knowledge refinement mechanism\. While theExpertapproach relies on a fixed set of pre\-defined heuristic rules, which limits its discovery scope to anticipated patterns,SAGEutilizes information from identified vulnerabilities to guide subsequent exploration during the runtime\. This feedback loop allows the framework to adaptively target specific weaknesses in the LLMs, resulting in higher exposure rates and improved search efficiency compared to static strategies\. This demonstrates the necessity of an evolving, automated framework for comprehensively auditing model robustness\.

Perturbation ScopeVER\(↑\\uparrow\)ApD\(↓\\downarrow\)Score𝚫\\boldsymbol\{\\Delta\}Score𝚫\\boldsymbol\{\\Delta\}Query Only73\.75\-10\.705\.54\+0\.46Relevant Schema Only51\.15\-33\.308\.90\+3\.82Irrelevant Schema Only48\.12\-36\.3310\.84\+5\.76\\rowcolorgray\!10SAGE84\.45–5\.08–

Table 2:Ablation study onBIRDwith Gemma\-3, analyzing the impact of distinct perturbation scopes\. We reportVER\(Vulnerability Exposure Rate, higher is better\) andApD\(Attempts per Discovery, lower is better\)\.SAGEtargets all scopes simultaneously\.𝚫\\boldsymbol\{\\Delta\}values indicate the performance degradation \(lower VER or higher ApD\) when restricting the perturbation to a single scope\.
### 4\.3Ablation Study: Impact of Perturbation Scopes

Table[2](https://arxiv.org/html/2607.03833#S4.T2)investigates the contributions of different input components—Query & Evidence, Relevant Schema, and Irrelevant Schema—to the vulnerability discovery process\. We compare the fullSAGEframework against restricted baselines where perturbations are confined to a single scope\.

Effectiveness of Individual Scopes\.The results indicate that perturbations applied to any individual scope are effective in revealing model defects\. Confining the exploration to theQuery & Evidencealone yields a high Vulnerability Exposure Rate \(VER\) of73\.75%73\.75\\%, confirming that semantic variations in the natural language question are a primary source of model confusion\. Similarly, modifying only theRelevant SchemaorIrrelevant Schemaalso uncovers a significant portion of failures, with VERs of51\.15%51\.15\\%and48\.12%48\.12\\%respectively\. This demonstrates that vulnerabilities are distributed across all dimensions of the input context, and each component serves as a critical entry point for probing model’s potential vulnerability\.

Superiority of the Comprehensive Scope\.While single\-scope strategies are valid,SAGEachieves superior performance by covering the comprehensive input space\. First, regardingdiscovery capability,SAGEattains the highest VER of84\.45%84\.45\\%, outperforming the best single\-scope variant by over1010percentage points\. By not limiting the perturbation to a single dimension, our method can identify a wider range of failure patterns that exist across the query and schema components\.

Second, regardingdiscovery efficiency,SAGEachieves the lowest Attempts per Discovery \(ApD\) score of5\.085\.08, indicating a more efficient search process than any restricted setting\. We attribute this efficiency gain to the data\-driven nature of our*Knowledge Refinement*mechanism\. Since the full\-scope approach exposes a larger volume and variety of failures \(higher VER\), it provides richer feedback data for the system to learn from\. This allowsSAGEto accumulate more diverse failure patterns and guide subsequent exploration more effectively, thereby accelerating the discovery process compared to searching within a constrained scope\.

## 5Analysis

![Refer to caption](https://arxiv.org/html/2607.03833v1/x2.png)Figure 2:Cross\-model strategy transfer performance\. The heatmap illustrates the absolute VER gain of Target Models \(x\-axis\) whenSAGEuses a Vulnerability Codex constructed from different Source Models \(y\-axis\), compared with theExpertbaseline that uses a static codex\. The values are absolute gains over the Expert baseline under the same evaluation framework\.### 5\.1Cross\-Model Discovered Vulnerability Generalization Analysis

Figure[2](https://arxiv.org/html/2607.03833#S5.F2)illustrates the cross\-model generalization ofSAGEby visualizing the VER gains of a Target Model \(x\-axis\) when guided by the vulnerability codex derived from a Source Model \(y\-axis\), measured as absolute gains relative to theExpertbaseline\. Along the main diagonal, where each target model uses its own evolved codex, the self\-codex gains range from\+10\.91%\+10\.91\\%to\+18\.47%\+18\.47\\%, showing that dynamically refined strategies consistently outperform manually curated expert rules\. We also observe robust cross\-model transferability: strategies derived from one model \(e\.g\., OmniSQL\) still improve the VER of distinct architectures \(e\.g\., Gemma\-3 by\+13\.74%\+13\.74\\%\)\. This indicates that the discovered vulnerabilities capture structural weaknesses that generalize across models rather than merely reflecting model\-specific artifacts\.

### 5\.2Mitigating Vulnerability with Fine\-tuning

The adversarial samples produced bySAGE’s discovery process constitute a targeted form of data synthesis for LLM trainingZhuet al\.\([2025a](https://arxiv.org/html/2607.03833#bib.bib1),[b](https://arxiv.org/html/2607.03833#bib.bib2)\)\. To investigate whether these samples can be transformed into actionable defense signals, we conducted a lightweight mitigation experiment\. Specifically, we selected 1,000 samples from the BIRD training set and employedSAGEto discover vulnerabilities onGemma\-3, generating 1,580 adversarial samples\. These samples—representing high\-confidence failure modes or “hard” cases—were then used to perform Supervised Fine\-Tuning \(SFT\) onQwen2\.5\-Coder\-7B\-Instruct\.

As presented in Table[3](https://arxiv.org/html/2607.03833#S5.T3), this targeted fine\-tuning yields consistent improvements\. Under the adversarial evaluation ofSAGE, the Execution Accuracy \(EX\) of the model rises from4\.37%4\.37\\%to9\.58%9\.58\\%, representing an absolute gain of\+5\.21%\+5\.21\\%and a relative improvement of over119%119\\%compared to the base model\. On the standard BIRD dev set, EX also improves from42\.63%42\.63\\%to52\.87%52\.87\\%, indicating no observable robustness\-general accuracy trade\-off in this setting\. Simultaneously, Attempts per Discovery \(ApD\) increases from3\.613\.61to4\.594\.59\.

These results are particularly notable given the data efficiency of the approach—using only 1,580 samples to achieve such gains\. The concurrent rise in both EX and ApD indicates that the model has not only corrected specific errors but has also become inherently more resilient; it now forces the attacker to search deeper to find a successful perturbation\. This suggests that the specific structural weaknesses targeted by our adversarial data have been effectively mitigated, validating the quality of the generated samples\.

Crucially, this experiment provides a practical validation of the transferability phenomenon discussed in Section[5\.1](https://arxiv.org/html/2607.03833#S5.SS1)\. By successfully repairing vulnerabilities inQwen2\.5\-Coderusing adversarial patterns discovered inGemma\-3, we reconfirm that the flaws exposed bySAGEare not model\-specific artifacts but generalized linguistic or logic loopholes shared across different LLMs\.

Ultimately, this experiment highlights the dual value ofSAGE\. It functions not merely as a diagnostic tool for identifying defects but as a constructive framework capable of “closing the loop”—enabling researchers to not only pinpoint weaknesses but also leverage them to synthesize robust training data, thereby actively enhancing the reliability of large language models\.

Model SettingEXApDAdv\.Std\.Qwen2\.5\-Coder4\.3742\.633\.61\+ SFT9\.58\(\+5\.21\)52\.87\(\+10\.24\)4\.59\(\+0\.98\)Table 3:Mitigation results via adversarial fine\-tuning\. We fine\-tuneQwen2\.5\-Coder\-7B\-Instructusing 1,580 adversarial samples generated bySAGE\(targetingGemma\-3\)\. Adv\. and Std\. denote EX on the SAGE\-generated adversarial evaluation set and the standard BIRD dev set, respectively\.

## 6Conclusion

In this paper, we proposeSAGE, a self\-evolving framework for automated vulnerability discovery in Text\-to\-SQL\. Extensive experiments confirm thatSAGEsignificantly outperforms static expert baselines in identifying weaknesses\. We further demonstrate that the discovered vulnerabilities are highly transferable across different LLMs\. Finally, our mitigation experiments show thatSAGEcan effectively close the loop: the generated adversarial samples serve as valuable training data to repair model defects and improve overall robustness via lightweight fine\-tuning\.

## Limitations

We acknowledge two main limitations in our current work\. First, our ablation studies are conducted primarily at the framework level; a more fine\-grained analysis is required to isolate the specific contributions of internal components like the Judge, Embedding, and Summarizer modules\. Second, the update mechanism of our Vulnerability Codex operates at coarse intervals; adopting a high\-frequency, per\-batch update strategy could further enhance the efficiency of the self\-evolving process\.

## References

- I\. Androutsopoulos, G\. D\. Ritchie, and P\. Thanisch \(1995\)Natural language interfaces to databases–an introduction\.Natural language engineering1\(1\),pp\. 29–81\.Cited by:[§1](https://arxiv.org/html/2607.03833#S1.p1.1)\.
- A\. Askari, C\. Poelitz, and X\. Tang \(2025\)Magic: generating self\-correction guideline for in\-context text\-to\-sql\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 23433–23441\.Cited by:[§2\.1](https://arxiv.org/html/2607.03833#S2.SS1.p2.1)\.
- A\. Bhaskar, T\. Tomar, A\. Sathe, and S\. Sarawagi \(2023\)Benchmarking and improving text\-to\-sql generation under ambiguity\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 7053–7074\.Cited by:[§1](https://arxiv.org/html/2607.03833#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.03833#S2.SS2.p2.1)\.
- S\. Chang, J\. Wang, M\. Dong, L\. Pan, H\. Zhu, A\. H\. Li, W\. Lan, S\. Zhang, J\. Jiang, J\. Lilien, S\. Ash, W\. Y\. Wang, Z\. Wang, V\. Castelli, P\. Ng, and B\. Xiang \(2023a\)Dr\.spider: a diagnostic evaluation benchmark towards text\-to\-SQL robustness\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Wc5bmZZU9cy)Cited by:[§1](https://arxiv.org/html/2607.03833#S1.p2.1),[§1](https://arxiv.org/html/2607.03833#S1.p4.1)\.
- S\. Chang, J\. Wang, M\. Dong, L\. Pan, H\. Zhu, A\. H\. Li, W\. Lan, S\. Zhang, J\. Jiang, J\. Lilien,et al\.\(2023b\)Dr\. spider: a diagnostic evaluation benchmark towards text\-to\-sql robustness\.arXiv preprint arXiv:2301\.08881\.Cited by:[§1](https://arxiv.org/html/2607.03833#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.03833#S2.SS2.p2.1),[§4\.1](https://arxiv.org/html/2607.03833#S4.SS1.SSS0.Px3.p1.7)\.
- S\. Cheng, Q\. Cheng, L\. Jin, L\. Yi, and G\. Zhang \(2025\)SQLord: a robust enterprise text\-to\-sql solution via reverse data generation and workflow decomposition\.InCompanion Proceedings of the ACM on Web Conference 2025,pp\. 919–923\.Cited by:[§2\.1](https://arxiv.org/html/2607.03833#S2.SS1.p1.1)\.
- Y\. Chi, H\. Wang, Y\. Chen, Y\. Yang, J\. Yang, Z\. Yang, X\. Yan, and G\. Chen \(2025\)Pi\-SQL: enhancing text\-to\-SQL with fine\-grained guidance from pivot programming languages\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 25120–25144\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.1369/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.1369),ISBN 979\-8\-89176\-335\-7Cited by:[§2\.1](https://arxiv.org/html/2607.03833#S2.SS1.p1.1)\.
- D\. Choi, M\. C\. Shin, E\. Kim, and D\. R\. Shin \(2021\)Ryansql: recursively applying sketch\-based slot fillings for complex text\-to\-sql in cross\-domain databases\.Computational Linguistics47\(2\),pp\. 309–332\.Cited by:[§2\.1](https://arxiv.org/html/2607.03833#S2.SS1.p1.1)\.
- M\. Deng, A\. Ramachandran, C\. Xu, L\. Hu, Z\. Yao, A\. Datta, and H\. Zhang \(2025\)Reforce: a text\-to\-sql agent with self\-refinement, format restriction, and column exploration\.InICLR 2025 Workshop: VerifAI: AI Verification in the Wild,Cited by:[§2\.1](https://arxiv.org/html/2607.03833#S2.SS1.p1.1)\.
- X\. Deng, A\. Hassan, C\. Meek, O\. Polozov, H\. Sun, and M\. Richardson \(2021\)Structure\-grounded pretraining for text\-to\-sql\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 1337–1350\.Cited by:[§1](https://arxiv.org/html/2607.03833#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.03833#S2.SS2.p2.1)\.
- Z\. Ding, Y\. Lin, and T\. Zeng \(2025\)AmbiSQL: interactive ambiguity detection and resolution for text\-to\-sql\.arXiv preprint arXiv:2508\.15276\.Cited by:[§2\.2](https://arxiv.org/html/2607.03833#S2.SS2.p2.1)\.
- Y\. Gan, X\. Chen, Q\. Huang, M\. Purver, J\. R\. Woodward, J\. Xie, and P\. Huang \(2021a\)Towards robustness of text\-to\-sql models against synonym substitution\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),pp\. 2505–2515\.Cited by:[§1](https://arxiv.org/html/2607.03833#S1.p2.1),[§1](https://arxiv.org/html/2607.03833#S1.p4.1),[§2\.2](https://arxiv.org/html/2607.03833#S2.SS2.p2.1),[§4\.1](https://arxiv.org/html/2607.03833#S4.SS1.SSS0.Px3.p1.7)\.
- Y\. Gan, X\. Chen, and M\. Purver \(2021b\)Exploring underexplored limitations of cross\-domain text\-to\-sql generalization\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,pp\. 8926–8931\.Cited by:[§1](https://arxiv.org/html/2607.03833#S1.p2.1),[§1](https://arxiv.org/html/2607.03833#S1.p4.1),[§2\.2](https://arxiv.org/html/2607.03833#S2.SS2.p2.1)\.
- Y\. Gong, C\. Lei, X\. Qin, K\. Vaidya, B\. M\. Narayanaswamy, and T\. Kraska \(2025\)SQLens: an end\-to\-end framework for error detection and correction in text\-to\-SQL\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=on6Hf0KP20)Cited by:[§2\.1](https://arxiv.org/html/2607.03833#S2.SS1.p2.1)\.
- Y\. Guo, D\. Jin, S\. Ye, S\. Chen, J\. Jianyang, and X\. Tan \(2025\)SQLForge: synthesizing reliable and diverse data to enhance text\-to\-SQL reasoning in LLMs\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 8441–8452\.External Links:[Link](https://aclanthology.org/2025.findings-acl.443/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.443),ISBN 979\-8\-89176\-256\-5Cited by:[§2\.1](https://arxiv.org/html/2607.03833#S2.SS1.p1.1)\.
- Z\. Hong, Z\. Yuan, Q\. Zhang,et al\.\(2024\)Next\-generation database interfaces: a survey of llm\-based text\-to\-sql\.arXiv preprint arXiv:2406\.08426\.Cited by:[§2\.1](https://arxiv.org/html/2607.03833#S2.SS1.p1.1)\.
- A\. Hurst, A\. Lerer, A\. P\. Goucher, A\. Perelman, A\. Ramesh, A\. Clark, A\. Ostrow, A\. Welihinda, A\. Hayes, A\. Radford,et al\.\(2024\)Gpt\-4o system card\.arXiv preprint arXiv:2410\.21276\.Cited by:[§4\.1](https://arxiv.org/html/2607.03833#S4.SS1.SSS0.Px2.p1.1)\.
- Infly and InfTech \(2025\)Inf\-rl\-qwen\-coder\-32b\-2746\.Hugging Face\.Note:[https://huggingface\.co/infly/inf\-rl\-qwen\-coder\-32b\-2746](https://huggingface.co/infly/inf-rl-qwen-coder-32b-2746)Accessed: 2026\-01\-05Cited by:[§4\.1](https://arxiv.org/html/2607.03833#S4.SS1.SSS0.Px2.p1.1)\.
- P\. Lai, Z\. Ou, Y\. Wang, L\. Wang, J\. Yang, Y\. Chen, and G\. Chen \(2026\)BiasScope: towards automated detection of bias in LLM\-as\-a\-judge evaluation\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=QGOw6AU8Lp)Cited by:[§2\.2](https://arxiv.org/html/2607.03833#S2.SS2.p2.1)\.
- F\. Li and H\. V\. Jagadish \(2014\)Constructing an interactive natural language interface for relational databases\.Proceedings of the VLDB Endowment8\(1\),pp\. 73–84\.Cited by:[§1](https://arxiv.org/html/2607.03833#S1.p1.1)\.
- H\. Li, S\. Wu, X\. Zhang, X\. Huang, J\. Zhang, F\. Jiang, S\. Wang, T\. Zhang, J\. Chen, R\. Shi,et al\.\(2025\)Omnisql: synthesizing high\-quality text\-to\-sql data at scale\.arXiv preprint arXiv:2503\.02240\.Cited by:[§2\.1](https://arxiv.org/html/2607.03833#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2607.03833#S4.SS1.SSS0.Px2.p1.1)\.
- H\. Li, J\. Zhang, C\. Li, and H\. Chen \(2023a\)Resdsql: decoupling schema linking and skeleton parsing for text\-to\-sql\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.37,pp\. 13067–13075\.Cited by:[§1](https://arxiv.org/html/2607.03833#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.03833#S2.SS1.p1.1)\.
- H\. Li, J\. Zhang, H\. Liu, J\. Fan, X\. Zhang, J\. Zhu, R\. Wei, H\. Pan, C\. Li, and H\. Chen \(2024a\)Codes: towards building open\-source language models for text\-to\-sql\.Proceedings of the ACM on Management of Data2\(3\),pp\. 1–28\.Cited by:[§2\.1](https://arxiv.org/html/2607.03833#S2.SS1.p1.1)\.
- J\. Li, B\. Hui, G\. Qu,et al\.\(2023b\)BIRD: a big bench for large\-scale database grounded text\-to\-sql\.NeurIPS\.Cited by:[§1](https://arxiv.org/html/2607.03833#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.03833#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2607.03833#S4.SS1.SSS0.Px1.p1.1)\.
- J\. Li, B\. Hui, G\. Qu, J\. Yang, B\. Li, B\. Li, B\. Wang, B\. Qin, R\. Geng, N\. Huo,et al\.\(2024b\)Can llm already serve as a database interface? a big bench for large\-scale database grounded text\-to\-sqls\.Advances in Neural Information Processing Systems36\.Cited by:[§1](https://arxiv.org/html/2607.03833#S1.p1.1)\.
- H\. Liu, H\. Li, X\. Zhang, R\. Chen, H\. Xu, T\. Tian, Q\. Qi, and J\. Zhang \(2025\)Uncovering the impact of chain\-of\-thought reasoning for direct preference optimization: lessons from text\-to\-sql\.arXiv preprint arXiv:2502\.11656\.Cited by:[§2\.1](https://arxiv.org/html/2607.03833#S2.SS1.p1.1)\.
- X\. Liu and Z\. Tan \(2024\)Epi\-sql: enhancing text\-to\-sql translation with error\-prevention instructions\.arXiv preprint arXiv:2404\.14453\.Cited by:[§2\.1](https://arxiv.org/html/2607.03833#S2.SS1.p2.1)\.
- Q\. Lyu, K\. Chakrabarti, S\. Hathi, S\. Kundu, J\. Zhang, and Z\. Chen \(2020\)Hybrid ranking network for text\-to\-sql\.arXiv preprint arXiv:2008\.04759\.Cited by:[§2\.1](https://arxiv.org/html/2607.03833#S2.SS1.p1.1)\.
- T\. Mahmud, K\. A\. Hasan, M\. Ahmed, and T\. H\. C\. Chak \(2015\)A rule based approach for nlp based query processing\.In2015 2nd international conference on electrical information and communication technologies \(EICT\),pp\. 78–82\.Cited by:[§2\.1](https://arxiv.org/html/2607.03833#S2.SS1.p1.1)\.
- R\. Patil, M\. Patwardhan, S\. Karande, L\. Vig, and G\. Shroff \(2023\)Exploring dimensions of generalizability and few\-shot transfer for text\-to\-sql semantic parsing\.InTransfer Learning for Natural Language Processing Workshop,pp\. 103–114\.Cited by:[§1](https://arxiv.org/html/2607.03833#S1.p2.1),[§1](https://arxiv.org/html/2607.03833#S1.p4.1),[§2\.2](https://arxiv.org/html/2607.03833#S2.SS2.p2.1)\.
- X\. Pi, B\. Wang, Y\. Gao, J\. Guo, Z\. Li, and J\. Lou \(2022\)Towards robustness of text\-to\-sql models against natural and realistic adversarial table perturbation\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 2007–2022\.Cited by:[§1](https://arxiv.org/html/2607.03833#S1.p2.1),[§1](https://arxiv.org/html/2607.03833#S1.p4.1),[§2\.2](https://arxiv.org/html/2607.03833#S2.SS2.p2.1),[§4\.1](https://arxiv.org/html/2607.03833#S4.SS1.SSS0.Px3.p1.7)\.
- M\. Pourreza and D\. Rafiei \(2023\)Din\-sql: decomposed in\-context learning of text\-to\-sql with self\-correction\.Advances in Neural Information Processing Systems36,pp\. 36339–36348\.Cited by:[§1](https://arxiv.org/html/2607.03833#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.03833#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2607.03833#S2.SS1.p2.1)\.
- G\. Qu, J\. Li, B\. Qin, X\. Li, N\. Huo, C\. Ma, and R\. Cheng \(2025\)SHARE: an SLM\-based hierarchical action CorREction assistant for text\-to\-SQL\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 11268–11292\.External Links:[Link](https://aclanthology.org/2025.acl-long.552/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.552),ISBN 979\-8\-89176\-251\-0Cited by:[§2\.1](https://arxiv.org/html/2607.03833#S2.SS1.p1.1)\.
- A\. Sahitaj, M\. Nilles, R\. Schenkel, and V\. Schmitt \(2025\)Utilising large language models for adversarial attacks in text\-to\-sql: a perpetrator and victim approach\.InDatenbanksysteme für Business, Technologie und Web \(BTW 2025\),pp\. 919–931\.Cited by:[§1](https://arxiv.org/html/2607.03833#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.03833#S2.SS2.p2.1)\.
- I\. Saparina and M\. Lapata \(2024\)Ambrosia: a benchmark for parsing ambiguous questions into database queries\.Advances in Neural Information Processing Systems37,pp\. 90600–90628\.Cited by:[§1](https://arxiv.org/html/2607.03833#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.03833#S2.SS2.p2.1)\.
- J\. Shen, C\. Wan, R\. Qiao, J\. Zou, H\. Xu, Y\. Shao, Y\. Zhang, W\. Miao, and G\. Pu \(2025\)A study of in\-context\-learning\-based text\-to\-sql errors\.arXiv preprint arXiv:2501\.09310\.Cited by:[§2\.1](https://arxiv.org/html/2607.03833#S2.SS1.p2.1)\.
- L\. Sheng, S\. Xu, and W\. Xie \(2025\)BASE\-sql: a powerful open source text\-to\-sql baseline approach\.arXiv preprint arXiv:2502\.10739\.Cited by:[§2\.1](https://arxiv.org/html/2607.03833#S2.SS1.p1.1)\.
- L\. Shi, Z\. Tang, N\. Zhang, X\. Zhang, and Z\. Yang \(2025\)A survey on employing large language models for text\-to\-sql tasks\.ACM Comput\. Surv\.\.Note:Just AcceptedExternal Links:ISSN 0360\-0300,[Link](https://doi.org/10.1145/3737873),[Document](https://dx.doi.org/10.1145/3737873)Cited by:[§2\.1](https://arxiv.org/html/2607.03833#S2.SS1.p1.1)\.
- G\. Team, A\. Kamath, J\. Ferret, S\. Pathak, N\. Vieillard, R\. Merhej, S\. Perrin, T\. Matejovicova, A\. Ramé, M\. Rivière,et al\.\(2025\)Gemma 3 technical report\.arXiv preprint arXiv:2503\.19786\.Cited by:[§4\.1](https://arxiv.org/html/2607.03833#S4.SS1.SSS0.Px2.p1.1)\.
- H\. Xia, F\. Jiang, N\. Deng, C\. Wang, G\. Zhao, R\. Mihalcea, and Y\. Zhang \(2024\)r​^​3r\\hat\{\\ \}3:" This is my sql, are you with me?" a consensus\-based multi\-agent system for text\-to\-sql tasks\.arXiv preprint arXiv:2402\.14851\.Cited by:[§2\.1](https://arxiv.org/html/2607.03833#S2.SS1.p1.1)\.
- X\. Xie, G\. Xu, L\. Zhao, and R\. Guo \(2025\)OpenSearch\-sql: enhancing text\-to\-sql with dynamic few\-shot and consistency alignment\.External Links:2502\.14913,[Link](https://arxiv.org/abs/2502.14913)Cited by:[§1](https://arxiv.org/html/2607.03833#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.03833#S2.SS1.p1.1)\.
- B\. Xu, S\. Li, Y\. Wu, S\. Wei, M\. Du, H\. Wang, and H\. Song \(2024\)Chain\-of\-program prompting with open\-source large language models for text\-to\-sql\.In2024 International Joint Conference on Neural Networks \(IJCNN\),pp\. 1–8\.Cited by:[§2\.1](https://arxiv.org/html/2607.03833#S2.SS1.p1.1)\.
- X\. Xu, C\. Liu, and D\. Song \(2017\)Sqlnet: generating structured queries from natural language without reinforcement learning\.arXiv preprint arXiv:1711\.04436\.Cited by:[§2\.1](https://arxiv.org/html/2607.03833#S2.SS1.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§4\.1](https://arxiv.org/html/2607.03833#S4.SS1.SSS0.Px3.p1.7)\.
- P\. Yin, G\. Neubig, W\. Yih, and S\. Riedel \(2020\)TaBERT: pretraining for joint understanding of textual and tabular data\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 8413–8426\.Cited by:[§2\.1](https://arxiv.org/html/2607.03833#S2.SS1.p1.1)\.
- T\. Yu, R\. Zhang, K\. Yang, M\. Yasunaga, D\. Wang, Z\. Li, J\. Ma, I\. Li, Q\. Yao, S\. Roman,et al\.\(2018\)Spider: a large\-scale human\-labeled dataset for complex and cross\-domain semantic parsing and text\-to\-sql task\.In2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018,pp\. 3911–3921\.Cited by:[§1](https://arxiv.org/html/2607.03833#S1.p1.1),[§1](https://arxiv.org/html/2607.03833#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.03833#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2607.03833#S4.SS1.SSS0.Px1.p1.1)\.
- B\. Zhang, Y\. Ye, G\. Du, X\. Hu, Z\. Li, S\. Yang, C\. H\. Liu, R\. Zhao, Z\. Li, and H\. Mao \(2024\)Benchmarking the text\-to\-sql capability of large language models: a comprehensive evaluation\.arXiv preprint arXiv:2403\.02951\.Cited by:[§1](https://arxiv.org/html/2607.03833#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.03833#S2.SS2.p2.1)\.
- Y\. Zhang, M\. Li, D\. Long, X\. Zhang, H\. Lin, B\. Yang, P\. Xie, A\. Yang, D\. Liu, J\. Lin,et al\.\(2025\)Qwen3 embedding: advancing text embedding and reranking through foundation models\.arXiv preprint arXiv:2506\.05176\.Cited by:[§4\.1](https://arxiv.org/html/2607.03833#S4.SS1.SSS0.Px3.p1.7)\.
- H\. Zhu, Y\. Ding, Y\. Tao, Z\. Ruan, Y\. Li, W\. Zhang, Y\. Chen, and G\. Chen \(2025a\)FANNO: augmenting high\-quality instruction data with open\-sourced LLMs only\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 17633–17653\.External Links:[Link](https://aclanthology.org/2025.findings-acl.906/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.906),ISBN 979\-8\-89176\-256\-5Cited by:[§5\.2](https://arxiv.org/html/2607.03833#S5.SS2.p1.1)\.
- H\. Zhu, Z\. Ruan, J\. Su, X\. He, Y\. Chen, W\. Zhang, and G\. Chen \(2025b\)Tag\-instruct: controlled instruction complexity enhancement through structure\-based augmentation\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 17708–17729\.External Links:[Link](https://aclanthology.org/2025.findings-acl.911/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.911),ISBN 979\-8\-89176\-256\-5Cited by:[§5\.2](https://arxiv.org/html/2607.03833#S5.SS2.p1.1)\.

## Appendix AComplementary Experiments

### A\.1Analysis of Discovered Vulnerability Patterns

As illustrated in Figure[3](https://arxiv.org/html/2607.03833#A2.F3), our framework uncovers a distinct “Novel Patterns” category that, while constituting only approximately10%10\\%to11%11\\%of the total error distribution, represents a critical set of previously overlooked vulnerabilities\. Far from being a monolithic outlier, this category encapsulates a rich diversity of complex structural failures—most notably Schema Misinterpretation and Syntactic Overfitting—that distinctively differ from traditional error taxonomies\. The significance of this segment extends far beyond its statistical proportion; as validated by our mitigation experiments, these specific failure modes serve as high\-value training signals\. The ability to leverage these patterns to construct effective data for model repair confirms that these newly identified vulnerabilities are not merely rare edge cases, but cornerstone defects whose resolution is fundamental to enhancing the overall robustness of LLMs\.

### A\.2Cost\-Effective Scalability on State\-of\-the\-Art LLMs

To assess the scalability of our approach while addressing the practical challenges of evaluating proprietary models, we targetedGPT\-4o\. Motivated by the prohibitive inference costs associated with extensive adversarial optimization on such large\-scale models, we adopted a budget\-constrained evaluation protocol\.

Instead of performing a full\-scale search from scratch, we directly leveraged the vulnerability codex pre\-distilled from the smallerGemma\-3model\. Furthermore, to strictly minimize computational overhead, we restricted the attack process on GPT\-4o to asingle iteration\.

As presented in Table[4](https://arxiv.org/html/2607.03833#A1.T4), despite this minimal investment of resources, our framework proved strikingly effective\. The single\-iteration attack successfully exposed significant vulnerabilities in GPT\-4o, driving the Execution Accuracy down to22\.36%22\.36\\%and achieving a Vulnerability Exposure Rate \(VER\) of65\.80%65\.80\\%\. This result highlights a critical advantage ofSAGE: it enables researchers to uncover latent flaws in costly, state\-of\-the\-art models by transferring “knowledge” from cheaper, open\-source models, offering a highly resource\-efficient pathway for robustness evaluation\.

SettingEXVERApDOriginal65\.38\-\-SAGE22\.36\(\-43\.02\)65\.805\.35Table 4:Resource\-efficient evaluation on GPT\-4o\.Constrained by budget, we perform only asingle iterationof attack using strategies transferred fromGemma\-3\. Even under this restricted setting, the framework successfully identifies widespread vulnerabilities\.

## Appendix BMore implementation details

### B\.1Reproducibility Details

Table[5](https://arxiv.org/html/2607.03833#A2.T5)consolidates the main search budget, inference settings, and the size of the initially\-correct subset\|D\|\|D\|used to compute VER\.

ItemDetailsMax iterations \(TT\)3Hypotheses per sample \(KK\)3 initial hypotheses per samplePerturbation scopesQuery, Relevant Schema, and Irrelevant SchemaExperience retrievalTop\-5 entries retrieved from the current Vulnerability CodexSimilarity threshold \(τ\\tau\)0\.1 for semantic compressionSampling settingsDefault chat sampling settings of each backbone; for Qwen3\-32B, we uset​e​m​p​e​r​a​t​u​r​e=0\.6temperature=0\.6,t​o​p​\_​k=20top\\\_k=20, andt​o​p​\_​p=0\.95top\\\_p=0\.95Stopping ruleStop searching a hypothesis once a valid failure is exposed or when the maximum iteration budget is reachedLLM modulesGenerator / Checker / Summarizer: Qwen3\-32B; Embedding model: Qwen3\-Embedding\-4BInitially\-correct subset\|D\|\|D\|on BIRDGemma\-3: 823; Inf\-rl\-qwen: 1082; OmniSQL: 970Initially\-correct subset\|D\|\|D\|on SpiderGemma\-3: 843; Inf\-rl\-qwen: 907; OmniSQL: 846Table 5:Consolidated reproducibility details and the size of the initially\-correct evaluation subset\|D\|\|D\|used to compute VER\.
### B\.2Detailed Algorithm ofSAGE

The complete Algorithm[1](https://arxiv.org/html/2607.03833#alg1)summarizes the implementation ofSAGE\.

Algorithm 1Systematic Automated Guided Exploration \(SAGE\)1:Target Model

MtM\_\{t\}\.

2:Dataset

D=\{\(qi,si,yi∗\)\}i=1ND=\\\{\(q\_\{i\},s\_\{i\},y^\{\*\}\_\{i\}\)\\\}\_\{i=1\}^\{N\}where

MtM\_\{t\}yields correct execution results for all

\(qi,si\)\(q\_\{i\},s\_\{i\}\)\.

3:LLM Modules: Generator

MgM\_\{g\}, Checker

McM\_\{c\}, Summarizer

MsM\_\{s\}, Embedding Model

MeM\_\{e\}\.

4:Perturbation Scopes

𝕊=\{squery,sschema,…\}\\mathbb\{S\}=\\\{s\_\{\\text\{query\}\},s\_\{\\text\{schema\}\},\\dots\\\}\.

5:Refined Vulnerability Codex

𝒱f​i​n​a​l\\mathcal\{V\}\_\{final\}\.

6:Initialize:Iteration counter

t←1t\\leftarrow 1, Vulnerability Codex

𝒱0\\mathcal\{V\}\_\{0\}\.

7:Phase 1: Systematic Hypothesis Generation

8:Synthesize initial set of vulnerability hypotheses from dataset

DD:

9:

ℋc​u​r​r←⋃di∈D\{\(di,hi,k\)∣hi,k∈HypothesisGen​\(Mg,di,K=3\)\}\\mathcal\{H\}\_\{curr\}\\leftarrow\\bigcup\_\{d\_\{i\}\\in D\}\\\{\(d\_\{i\},h\_\{i,k\}\)\\mid h\_\{i,k\}\\in\\text\{HypothesisGen\}\(M\_\{g\},d\_\{i\},K=3\)\\\}⊳\\trianglerightEach sampledid\_\{i\}spawnsKKdistinct hypotheses

10:while

t≤Tt\\leq Tand

ℋc​u​r​r≠∅\\mathcal\{H\}\_\{curr\}\\neq\\emptysetdo

11:

𝒱new\(t\)←∅\\mathcal\{V\}\_\{\\text\{new\}\}^\{\(t\)\}\\leftarrow\\emptyset⊳\\trianglerightStorage for new archetypes found in this iter

12:

ℋn​e​x​t←∅\\mathcal\{H\}\_\{next\}\\leftarrow\\emptyset⊳\\trianglerightHypotheses that fail to expose bugs are retained

13:Phase 2: Experience\-Guided Probing & Verification

14:foreach candidate tuple

Cj=\(di,hi,k\)C\_\{j\}=\(d\_\{i\},h\_\{i,k\}\)in

ℋc​u​r​r\\mathcal\{H\}\_\{curr\}do

15:Retrieve relevant insights:

Ej←RetrieveTopN​\(Me,𝒱t−1,hi,k\)E\_\{j\}\\leftarrow\\text\{RetrieveTopN\}\(M\_\{e\},\\mathcal\{V\}\_\{t\-1\},h\_\{i,k\}\)
16:⊳\\trianglerightDynamic Scope Selection

17:if

t=1t=1then

18:Select single scope:

𝕊a​c​t​i​v​e←\{𝕊\[\(k−1\)\(mod\|𝕊\|\)\+1\]\}\\mathbb\{S\}\_\{active\}\\leftarrow\\\{\\mathbb\{S\}\[\(k\-1\)\\pmod\{\|\\mathbb\{S\}\|\}\+1\]\\\}⊳\\trianglerightCold Start: Efficiently map 1 hypothesis to 1 scope

19:else

20:Select all scopes:

𝕊a​c​t​i​v​e←𝕊\\mathbb\{S\}\_\{active\}\\leftarrow\\mathbb\{S\}
21:endif

22:

I​s​E​x​p​o​s​e​d←falseIsExposed\\leftarrow\\textbf\{false\}
23:foreach scope

sm∈𝕊a​c​t​i​v​es\_\{m\}\\in\\mathbb\{S\}\_\{active\}do

24:⊳\\trianglerightApply hypothesishi,kh\_\{i,k\}onto scopesms\_\{m\}guided by experienceEjE\_\{j\}

25:

xj,m←SampleGenerate​\(Mg,Cj,sm,Ej\)x\_\{j,m\}\\leftarrow\\text\{SampleGenerate\}\(M\_\{g\},C\_\{j\},s\_\{m\},E\_\{j\}\)
26:if

¬QualityCheck​\(xj,m,Mc\)\\neg\\text\{QualityCheck\}\(x\_\{j,m\},M\_\{c\}\)thencontinue

27:endif

28:

yj,m←TargetModelQuery​\(Mt,xj,m\)y\_\{j,m\}\\leftarrow\\text\{TargetModelQuery\}\(M\_\{t\},x\_\{j,m\}\)
29:

I​s​C​o​r​r​e​c​t←VerifyAnswer​\(di,xj,m,yj,m\)IsCorrect\\leftarrow\\text\{VerifyAnswer\}\(d\_\{i\},x\_\{j,m\},y\_\{j,m\}\)
30:if

¬I​s​C​o​r​r​e​c​t\\neg IsCorrectthen⊳\\trianglerightVulnerability Discovered

31:⊳\\trianglerightAbstract the concrete failure case into a generalized archetype

32:

vj←AbstractVulnerability​\(Ms,Cj,xj,m,yj,m\)v\_\{j\}\\leftarrow\\text\{AbstractVulnerability\}\(M\_\{s\},C\_\{j\},x\_\{j,m\},y\_\{j,m\}\)
33:if

vjv\_\{j\}is non\-trivialthen

34:

𝒱new\(t\)←𝒱new\(t\)∪\{varch\}\\mathcal\{V\}\_\{\\text\{new\}\}^\{\(t\)\}\\leftarrow\\mathcal\{V\}\_\{\\text\{new\}\}^\{\(t\)\}\\cup\\\{v\_\{\\text\{arch\}\}\\\}
35:endif

36:

I​s​E​x​p​o​s​e​d←trueIsExposed\\leftarrow\\textbf\{true\}
37:break⊳\\trianglerightPruning: Hypothesis validated, skip remaining scopes

38:endif

39:endfor

40:if

¬I​s​E​x​p​o​s​e​d\\neg IsExposedthen

41:

ℋn​e​x​t←ℋn​e​x​t∪\{Cj\}\\mathcal\{H\}\_\{next\}\\leftarrow\\mathcal\{H\}\_\{next\}\\cup\\\{C\_\{j\}\\\}
42:endif

43:endfor

44:Phase 3: Codex Evolution & Management

45:

𝒱t←𝒱t−1∪𝒱new\(t\)\\mathcal\{V\}\_\{t\}\\leftarrow\\mathcal\{V\}\_\{t\-1\}\\cup\\mathcal\{V\}\_\{\\text\{new\}\}^\{\(t\)\}
46:⊳\\trianglerightRefine the Codex by merging similar archetypes

47:

𝒱t←SemanticCompress​\(Me,𝒱t,τ\)\\mathcal\{V\}\_\{t\}\\leftarrow\\text\{SemanticCompress\}\(M\_\{e\},\\mathcal\{V\}\_\{t\},\\tau\)
48:

ℋc​u​r​r←ℋn​e​x​t\\mathcal\{H\}\_\{curr\}\\leftarrow\\mathcal\{H\}\_\{next\}
49:

t←t\+1t\\leftarrow t\+1
50:endwhile

51:

𝒱f​i​n​a​l←Vt−1\\mathcal\{V\}\_\{final\}\\leftarrow\{V\}\_\{t\-1\}
52:return

𝒱f​i​n​a​l\\mathcal\{V\}\_\{final\}

### B\.3Sensitivity to Iteration Count

To study the effect of the iteration budget, we evaluateSAGEon Gemma\-3 over the BIRD dev set while varying the maximum number of iterationsTT\.

IterationTTEXVERApD124\.6454\.077\.73214\.7372\.545\.6838\.3484\.455\.08Table 6:Sensitivity ofSAGEto the iteration budgetTTon Gemma\-3 over the BIRD dev set\.Increasing the iteration budget consistently improves VER while reducing ApD, indicating that a larger search budget uncovers more vulnerabilities and makes discovery more efficient\.

### B\.4Training configurations of Fine\-tuning Experiments

This section lists the key hyperparameters for the parameter\-efficient fine\-tuning experiments onQwen2\.5\-Coder\-7B\-Instructin Table[7](https://arxiv.org/html/2607.03833#A2.T7)\.

HyperparametersDetailsbatch\_size64Effective learning rate1\.0×10−41\.0\\times 10^\{\-4\}Training epochs3LR schedulercosinewarmup0\.1Precisionbf16Max seq length10240Table 7:Training hyperparameters of fine\-tuning experiments\.![Refer to caption](https://arxiv.org/html/2607.03833v1/x3.png)Figure 3:Distribution and hierarchical analysis of failure patterns across Gemma\-3, Inf\-rl\-qwen, and OmniSQL\. The top row illustrates the overall error distribution, distinguishing between established error categories \(cool tones\) and the Novel Patterns identified by our approach \(orange, exploded slice\)\. The bottom row provides a fine\-grained breakdown of these novel patterns, revealing that specific failures—such as Schema Misinterpretation and Syntactic Overfitting—constitute a significant portion of errors previously categorized broadly as “Other”\.Beyond the established categories, Figure[3](https://arxiv.org/html/2607.03833#A2.F3)further presents a detailed analysis of errors typically grouped under theOTHERcategory\. By combining expert manual inspection with systematic reclassification, we uncover recurring and nuanced cognitive failure patterns\. These include: schema misinterpretation, syntactic overfitting, insufficient prioritization of explicit evidence, overconfidence, non\-robust text matching, erratic self\-correction, and overreliance on temporal inference\. This trigger\-based partitioning moves beyond surface\-level heuristics, substantially enriching standard error categories and, more importantly, providing fine\-grained, actionable insights for targeted model improvement\.

### B\.5Prompt Templates for Sample Perturbation

We provide the prompt templates used to generate sample perturbations in our framework\. Figures[4](https://arxiv.org/html/2607.03833#A2.F4),[5](https://arxiv.org/html/2607.03833#A2.F5), and[6](https://arxiv.org/html/2607.03833#A2.F6)illustrate the prompt templates for perturbations applied to natural language queries, schema\-irrelevant elements, and schema\-relevant elements, respectively\.

The prompt templates for perturbations applied to natural language queries\#\#\# Task:Please \*\*rewrite the question and the evidence\*\* in a way that \*\*preserves the original intent and does not change the final correct answer\*\* \(i\.e\., the gold SQL query must still be valid and applicable\)\.Your rewritten version should:\- Better reflect \*\*real\-world use cases or natural user phrasing\*\*\.\- Increase \*\*linguistic and structural diversity\*\* …\- Introduce \*\*more contextual framing, ambiguity, or reasoning cues\*\* …\- Remain \*\*faithful to the original semantics and answer\*\*\.\#\#\# Guidelines:\- Do \*\*not\*\* change the final answer \(i\.e\., gold SQL must still be valid\)\.\- Focus on \*\*rewriting\*\*, not expanding or omitting information\.\- You must \*\*describe your modification strategy\*\* clearly in the ‘improvement‘ field\.\#\#\# Output Format:Please return your output strictly in the following JSON format:“‘json\{\{“improvement": “Describe what changes you made and why …",“question": “your rewritten question",“evidence": “your rewritten evidence"\}\}Example 1\-3: …Now it is your turn:Below is the database schema:\{schema\}Question: \{question\}Evidence: \{evidence\}The gold SQL query is: \{gold\_sql\}Figure 4:The prompt templates for perturbations applied to natural language queries\.The prompt templates for perturbations applied to schema\-irrelevant elements\#\#\# Task:Your task is to \*\*modify the provided database\-related information\*\* while ensuring that the \*\*core semantics and the correct answer remain unchanged\*\*\.Your goal is to introduce \*\*maximum lexical, structural, and semantic diversity\*\* in order to \*\*evaluate the model’s robustness\*\* against variations in metadata\.You are allowed to revise the following fields for each column:\- \*\*column\_description\*\*: Modify the column description using paraphrasing, synonyms, different sentence structures …\- \*\*value\_description\*\*: Change the way value types, ranges, or categories are described …\- \*\*example\_value\*\*: Add more diverse, realistic, or representative sample values …\#\#\# Important Guidelines:\- \*\*Do not\*\* change the table names or column names\.\- \*\*Do not\*\* alter the underlying meaning or change the final answer\.\- You must output your result in the \*\*strict JSON format\*\* shown below\.\- You must change the metadata for \*\*at least 3 columns\*\* and \*\*at most 5 columns\*\*\.\#\#\# Output Format:“table\_name": \[\{\{“column": “column\_name",“improvement": “Describe your strategy for modifying this column’s metadata\.",“column\_description": “Modified description of the column\.",“value\_description": “Modified description of the values\.",“example\_value": \[“example1", “example2"\]\}\}Example 1\-3: …Below is a partial view of the database schema\. It includes only the tables and columns that are directly relevant to answering the given question; all unrelated schema elements have been omitted\.\{json\_str\}Question: \{question\}Evidence: \{evidence\}The gold SQL query is: \{gold\_sql\}Here are some successful attack strategies you can refer to: \{error\_strategies\}\# The column\_description and value\_description fields should each be no longer than 100 tokens\.Now it is your turn\. Please return your answer strictly in the above JSON format\.Figure 5:The prompt templates for perturbations applied to schema\-irrelevant elements\.The prompt templates for perturbations applied to schema\-relevant elementsYou are an expert in prompt\-based robustness evaluation for database\-related tasks\.\#\#\# Task:Please \*\*modify the database\-related information\*\* while ensuring that the \*\*core semantics and the correct answer remain unchanged\*\*\. Your goal is to introduce \*\*greater lexical, structural, and semantic diversity\*\* to challenge the robustness of the model\.You may revise the following elements for each column:\- \*\*column\_description\*\*: Modify the description using paraphrasing, synonyms, different sentence structures, or multilingual expressions\.\- \*\*value\_description\*\*: Alter the value type description or re\-express the value mappings, ranges, or classifications\.\- \*\*example\_value\*\*: Enrich the example values with varied, realistic, and semantically appropriate entries\.\*\*Important:\*\*\- Do \*\*not\*\* change the table name or column name\.\- Your output must be in \*\*strict JSON format\*\* as shown below\.\- You must change the metadata for \*\*at least 3 columns\*\* and \*\*at most 5 columns\*\*\.Here are some output examples:Example 1\-3: …Now it is your turn\.Below is a partial view of the database schema\. It includes only the tables and columns that are relevant to answering the given question; unrelated schema elements have been omitted\.\{json\_str\}Question: \{question\}Evidence: \{evidence\}The gold SQL query is: \{gold\_sql\}Return your modifications using the following structure:\{\{“table\_name": \[\{\{“column": “column\_name",“improvement": “Describe your strategy for modifying this column’s metadata\.",“column\_description": “Modified description of the column\.",“value\_description": “Modified description of the values\.",“example\_value": \[“example1", “example2"\]\}\}\]\}\}Here are some successful attack strategies you can refer to: \{error\_strategies\}The column\_description and value\_description fields should each be no longer than 100 tokens\.Please return your answer strictly in the above JSON format\.Figure 6:The prompt templates for perturbations applied to schema\-relevant elements\.
### B\.6Checker validation

To verify the reliability of the automated Checker \(implemented with Qwen3\-32B\), we conducted both a targeted human evaluation and a larger\-scale cross\-model proxy validation\. We recruited three graduate\-level annotators with expertise in Text\-to\-SQL tasks to ensure the quality of the assessment\. They were compensated at a competitive market rate for such tasks\. All participants were informed about the purpose of the evaluation task and how the data would be used prior to the annotation\. They provided informed consent to participate in the study\. Since the study involves standard data annotation tasks with minimal risk to participants and does not collect personally identifiable information \(PII\), it was determined to be exempt from formal IRB review according to our institution’s guidelines\.

From the Gemma\-3 experiments, we randomly sampled 100 perturbed cases for evaluation\. Each annotator independently judged whether each perturbed query was valid \(i\.e\., the perturbation preserves semantic equivalence and the correct answer should remain unchanged\) or invalid \(i\.e\., the perturbation alters the meaning or introduces inconsistencies\)\.

Each annotator was provided with \(i\) the original natural\-language query and its gold answer, \(ii\) the perturbed query, and \(iii\) the corresponding M\-schema context\. They followed a concise guideline: \(1\) assess semantic equivalence between the original and perturbed queries, \(2\) verify whether the perturbation is realistic and contextually appropriate, and \(3\) label the sample as either Valid or Invalid\. The final ground\-truth label for each case was determined by majority vote across the three annotators to reduce subjectivity and ensure consistency\.

We then compared the Checker’s predictions with these human consensus labels\. The Checker achieved an average accuracy of 89\.3%, indicating a high level of agreement with expert human judgments in identifying valid perturbations\.

To complement this targeted human assessment, we further sampled 2,000 perturbations that were originally validated by the Qwen3\-32B Checker and re\-evaluated them using GPT\-5\.1 as an independent proxy judge\. Table[8](https://arxiv.org/html/2607.03833#A2.T8)reports the agreement rates across different perturbation scopes\.

Scope\# SamplesAgreementQuery Only66196\.52Irrelevant Schema66295\.62Relevant Schema67787\.44Overall200093\.15Table 8:Cross\-model proxy validation of the Qwen3\-32B Checker using GPT\-5\.1 as an independent judge\. Agreement is reported in percentage\.The Relevant Schema setting is the most challenging because modifying metadata for columns used in the gold SQL more easily introduces subtle semantic ambiguity\. We therefore view this proxy study as complementary evidence rather than a replacement for human judgment\. At the same time, the consistently high agreement and the mitigation gains in Table[3](https://arxiv.org/html/2607.03833#S5.T3)jointly suggest that the discovered perturbations are not dominated by invalid rewrites\.

## Appendix CEthical Considerations

This work aims to improve the robustness of Text\-to\-SQL systems by automating the discovery of corner cases and vulnerabilities\. Here, “vulnerability” refers to robustness and reliability brittleness under semantic\-preserving perturbations, not SQL\-injection\-style exploits\. While our method is designed to facilitate debugging and model improvement, we recognize that it carries a potential risk of misuse for adversarial attacks\. We emphasize that identifying these weaknesses proactively is essential for mitigating risks in real\-world deployments, and we urge users to apply this framework responsibly to enhance system security\.