ESQ-Bench: A Multi-Tier Enterprise Oracle Benchmark for Evaluating NL2SQL Dialect Generalization and Silent Semantic Divergence

# ESQ-Bench: A Multi-Tier Enterprise Oracle Benchmark for Evaluating NL2SQL Dialect Generalization and Silent Semantic Divergence
Source: [https://arxiv.org/html/2608.23569](https://arxiv.org/html/2608.23569)
Divya Chukkapalli Independent Researcher, Apex, NC, USA ORCID: 0009\-0005\-4691\-3395Ganesh R\. Naik Torrens University Australia ORCID: 0000\-0003\-1790\-9838

###### Abstract

State\-of\-the\-art Natural Language to SQL \(NL2SQL\) models report execution accuracy exceeding 89% on established benchmarks such as Spider and BIRD\. However, these benchmarks rely on simplified academic schemas and open\-source SQL dialects that do not reflect the complexity of enterprise database environments\. We introduceESQ\-Bench, an Oracle\-first NL2SQL benchmark with systematic complexity tiers and silent\-divergence \(SD\) evaluation across three enterprise schema complexity tiers\. We have constructed and released six populated schemas \(465 tables, 164,682 rows, zero empty tables\) with identical seed data on Oracle, PostgreSQL, MySQL, and SQL Server, a four\-metric evaluation harness \(EM, EX, SR, SD\), and550 gold\-validated question\-query pairs\(Tier\-1: 95; Tier\-2: 228; Tier\-3: 227\)\. Schema\-linked prompting with GPT\-4o shows monotonic execution\-match*degradation*across tiers—79\.8%/60\.3%/57\.2%EX on executed queries \(June 2026\)—versus75\.6%/80\.4%/95\.8%on the earlier 142\-question pilot slice\. EM stays below 7% tier\-wide; operationalSDop\\text\{SD\}\_\{\\text\{op\}\}reaches 73–99% among EX\-passing queries\. Failure analysis \(F1–F4\) shows wrong\-result semantics \(F3\) dominate at higher tiers\. A secondary finding: Claude Sonnet 4\.6 with schema\-linked prompts reaches87\.4%/74\.9%/68\.7%EX \(executed queries\), exceeding GPT\-4o SL on every tier\. GPT\-4o zero\-shot EX on executed queries \(78\.7% / 73\.5% / 77\.8%\) inverts SL at Tiers 2–3 due to lower execution rates and survivor bias in the zero\-shot versus schema\-linked analysis\. Local Llama 3\.2 schema\-linked reaches only 13\.3% bank\-wide EX \(73/550\), underscoring the gap between closed API models and open\-weight baselines on enterprise Oracle schemas\.

Keywords:Enterprise benchmark; evaluation metrics; natural language to SQL; NL2SQL; Oracle Database; schema complexity; semantic divergence; text\-to\-SQL\.

## 1Introduction

The ability to query relational databases using natural language has long been a goal of the database and natural\-language\-processing communities\[[9](https://arxiv.org/html/2608.23569#bib.bib13)\]\. Recent large language model \(LLM\) based approaches report remarkable accuracy: DAIL\-SQL achieves 86\.6% execution accuracy on Spider\[[3](https://arxiv.org/html/2608.23569#bib.bib4)\], and GPT\-4\-based systems exceed 91% on the same benchmark\[[8](https://arxiv.org/html/2608.23569#bib.bib10)\]\. On the newer, more challenging BIRD benchmark\[[7](https://arxiv.org/html/2608.23569#bib.bib3)\], state\-of\-the\-art models approach 67%\. These numbers have led to widespread optimism about NL2SQL deployability in production settings\.

Practitioners who attempt to deploy these systems on real enterprise databases, however, report a starkly different experience\. Enterprise databases—particularly those built on Oracle Database, which dominates the Fortune 500 and financial services sector\[[4](https://arxiv.org/html/2608.23569#bib.bib12)\]—present schema complexity and SQL dialect characteristics that no published benchmark tests\. The gap between benchmark accuracy and production accuracy is real, large, and poorly understood\.

We identify three structural mechanisms responsible for this gap:

Schema complexity mismatch\.Spider schemas average 5\.1 tables with complete foreign\-key documentation and clean naming conventions\[[11](https://arxiv.org/html/2608.23569#bib.bib1)\]\. Enterprise Oracle schemas routinely contain 150–500 tables with partial FK enforcement, ambiguous column names \(a single name such asSTATUSappearing in seven tables with different value domains\), legacy abbreviated naming \(ACCT\_BAL\_DT\_AMT\_USD\), and denormalized structures arising from decades of system migrations\.

Oracle dialect blindness\.Every major NL2SQL benchmark targets SQLite or PostgreSQL\. Oracle Database has syntax and semantics that differ in ways that cause silently wrong or explicitly failing queries:FETCH FIRSTpagination,CONNECT BYhierarchical traversal,MINUSinstead ofEXCEPT, empty\-string\-as\-NULLcoercion, andNULLS LASTdefault ordering, among others\.

Metric blindness\.Execution match \(EX\)—the field’s dominant evaluation metric—determines only whether a generated query returns the same rows as the gold query\. It cannot detect queries that execute successfully but return semantically incorrect results due to Oracle\-specific NULL handling, implicit type coercion, or boundary\-condition differences\. We call these*silent semantic divergences*and introduce a metric to quantify them\.

To address these gaps, we make the following contributions:

1. 1\.ESQ\-Bench infrastructure: six enterprise\-representative schemas \(465 tables, 164,682 seeded rows, zero empty tables\) across three tiers, with dialect\-faithful DDL and identical data on Oracle \(primary\), PostgreSQL, MySQL, and SQL Server—among the first NL2SQL benchmarks designed Oracle\-first for production validation, complementing enterprise\-oriented suites such as Spider 2\.0 with dialect\-specific constructs and SD measurement\.
2. 2\.Full 550\-question bank \(v0\.3\): 95 / 228 / 227 gold\-validated pairs across Tiers 1–3 with critical\-condition ties to seeded data \(NULL traps, status\-domain separation,CONNECT BYhierarchies, entity\-overlap joins, and ambiguity\-resolution traps\)\.
3. 3\.A four\-metric evaluation frameworkcomprising Exact Match \(EM\), Execution Match \(EX\), Semantic Recall \(SR\), and Silent Divergence Rate \(SD\), with formal definitions, an operational harness definition, and reproducible computation\.
4. 4\.Full\-bank GPT\-4o evaluation\(550 questions, schema\-linked SAL, June 2026\): tier execution match 79\.8% / 60\.3% / 57\.2%, EM below 7%, high operationalSDop\\text\{SD\}\_\{\\text\{op\}\}\(73–99%\), monotonic complexity degradation, and F1–F4 failure taxonomy \([Sections˜6](https://arxiv.org/html/2608.23569#S6)and[9](https://arxiv.org/html/2608.23569#S6.T9)\)\.
5. 5\.A public benchmark releasewith schemas, seed scripts, all 550 questions, evaluation harness \(run\_esq\_tier1\_evaluation\.py\), failure taxonomy \(analyze\_esq\_failures\.py\), GPT\-4o and Claude Sonnet 4\.6 baselines on the complete bank \([Table˜11](https://arxiv.org/html/2608.23569#S6.T11)\)\.

The remainder of this paper is structured as follows\.[Section˜2](https://arxiv.org/html/2608.23569#S2)defines the NL2SQL task and Oracle dialect characteristics\.[Section˜3](https://arxiv.org/html/2608.23569#S3)surveys prior benchmarks and identifies gaps\.[Section˜4](https://arxiv.org/html/2608.23569#S4)describes ESQ\-Bench design and construction\.[Section˜5](https://arxiv.org/html/2608.23569#S5)formalizes our four\-metric framework\.[Section˜6](https://arxiv.org/html/2608.23569#S6)presents experimental results, including multi\-model baselines and schema\-linked vs\. zero\-shot analysis\.[Section˜7](https://arxiv.org/html/2608.23569#S7)provides complexity predictor analysis\.[Section˜8](https://arxiv.org/html/2608.23569#S8)discusses implications and limitations\.[Section˜9](https://arxiv.org/html/2608.23569#S9)identifies open research problems\.[Section˜10](https://arxiv.org/html/2608.23569#S10)concludes\.

## 2Background

### 2\.1Task Definition

Given a natural language questionQQand a database schema𝒮=\{T1,T2,…,Tn\}\\mathcal\{S\}=\\\{T\_\{1\},T\_\{2\},\\ldots,T\_\{n\}\\\}where each tableTiT\_\{i\}comprises columns\{ci,1,…,ci,mi\}\\\{c\_\{i,1\},\\ldots,c\_\{i,m\_\{i\}\}\\\}with associated data types and constraints, the NL2SQL task requires generating a SQL queryqqsuch that executingqqagainst a database instance𝒟\\mathcal\{D\}conforming to𝒮\\mathcal\{S\}returns the result setR​\(q,𝒟\)R\(q,\\mathcal\{D\}\)that satisfies the*intent*ofQQ\.

The word*intent*is critical\. Existing evaluation metrics assess whetherR​\(q,𝒟\)=R​\(q∗,𝒟\)R\(q,\\mathcal\{D\}\)=R\(q^\{\*\},\\mathcal\{D\}\)whereq∗q^\{\*\}is a designated gold query\. This is a necessary but insufficient condition for correctness:R​\(q,𝒟\)R\(q,\\mathcal\{D\}\)may equalR​\(q∗,𝒟\)R\(q^\{\*\},\\mathcal\{D\}\)on a specific test instance𝒟\\mathcal\{D\}while diverging on other instances with different data distributions—particularly when the divergence arises from Oracle\-specific NULL semantics or implicit type coercion that test data does not expose\.

### 2\.2Oracle Dialect Characteristics

Oracle Database introduces syntax and semantics not present in SQLite or PostgreSQL\. The following features appear in real enterprise queries and are absent from all existing NL2SQL benchmarks\.

#### 2\.2\.1Pagination

Oracle historically usedROWNUMfor row\-limiting and introduced the SQL:2008 standardFETCH FIRSTnnROWS ONLYsyntax in version 12c\. Models trained on Spider/BIRD generateLIMITnn, which is invalid Oracle syntax and produces anORA\-00933error\.

#### 2\.2\.2Hierarchical Queries

Oracle’s proprietaryCONNECT BY/START WITHclause enables traversal of self\-referencing hierarchies without recursive CTEs\. This is required for organizational charts, account hierarchies, GL account trees, and policy hierarchies in insurance\. No standard SQL equivalent exists with identical semantics\.

#### 2\.2\.3Set Operations

Oracle usesMINUSwhere the SQL standard specifiesEXCEPT\. Models generateEXCEPT, which Oracle rejects withORA\-00933\.

#### 2\.2\.4NULL Semantics

Oracle treats empty string \(’’\) asNULL—a behavior not present in any other major RDBMS\. Additionally, Oracle sortsNULLvalues last in ascending order \(NULLS LASTdefault\), opposite to PostgreSQL and MySQL \(NULLS FIRSTdefault\)\. These differences are invisible in test data withoutNULLvalues and create silent semantic divergence whenNULLs are present\.

#### 2\.2\.5Oracle\-Specific Functions

DECODE,NVL,NVL2,NULLIF,LISTAGG,SYS\_CONNECT\_BY\_PATH, andRATIO\_TO\_REPORTare Oracle\-specific or Oracle\-extended functions with no direct equivalent in other dialects\. Models trained on open\-dialect data substitute approximate alternatives that may produce subtly different results\.

## 3Related Work

### 3\.1Benchmark Evolution

WikiSQL\[[13](https://arxiv.org/html/2608.23569#bib.bib2)\]introduced large\-scale NL2SQL evaluation with 80,654 examples but was limited to single\-table queries, precluding multi\-join and subquery evaluation\.

Spider\[[11](https://arxiv.org/html/2608.23569#bib.bib1)\]advanced the field with 10,181 examples across 200 databases requiring cross\-domain generalisation and complex SQL\. However, Spider’s schemas average 5\.1 tables, all FK constraints are enforced and documented, column naming is clean, and all queries target SQLite\. Spider established execution match as the standard metric\.

SParC\[[12](https://arxiv.org/html/2608.23569#bib.bib5)\]andCoSQL\[[10](https://arxiv.org/html/2608.23569#bib.bib6)\]extended Spider to multi\-turn and dialogue\-based interaction, respectively\. Both retain Spider’s schema simplicity and SQLite dialect\.

Spider\-Syn\[[2](https://arxiv.org/html/2608.23569#bib.bib7)\]tested lexical robustness by replacing natural language with synonyms, revealing models’ dependence on column\-name matching\. It does not address schema complexity or dialect variation\.

Dr\. Spider\[[1](https://arxiv.org/html/2608.23569#bib.bib8)\]introduced 17 perturbation categories to test robustness systematically\. It is the most rigorous prior robustness study and the closest motivating work for ESQ\-Bench, but all perturbations are applied to Spider schemas and SQLite queries\. Enterprise schema characteristics and Oracle semantics are outside its scope\.

KaggleDBQA\[[5](https://arxiv.org/html/2608.23569#bib.bib9)\]used eight real databases from Kaggle competitions, providing the first NL2SQL evaluation on non\-curated schemas\. With 272 questions, the scale is insufficient for statistically reliable category\-level claims\. None of the schemas targets Oracle\.

BIRD\[[7](https://arxiv.org/html/2608.23569#bib.bib3)\]introduced 12,751 questions across larger, more complex schemas with external knowledge requirements\. BIRD represents the current standard for challenging NL2SQL evaluation\. Schemas average 7\.3 tables \(versus Spider’s 5\.1\), but remain substantially below enterprise complexity\. All execution targets SQLite or PostgreSQL\.

Spider 2\.0\[[6](https://arxiv.org/html/2608.23569#bib.bib11)\]targets enterprise\-grade tasks including data warehousing and complex analytical queries\. It represents a significant step toward production\-relevance but does not include Oracle\-specific constructs, systematic schema\-complexity tiers, or a silent divergence metric; ESQ\-Bench complements Spider 2\.0 by isolating Oracle dialect and SD effects on seeded enterprise\-representative schemas\.

### 3\.2Comparative Analysis

[Table˜1](https://arxiv.org/html/2608.23569#S3.T1)summarises the gap ESQ\-Bench addresses relative to prior benchmarks\. The Oracle dialect column and Silent Divergence metric column are empty for every prior benchmark\.

Table 1:Comparison of NL2SQL Benchmarks\. Avg\. Tables refers to mean tables per schema\. Enterprise column indicates use of enterprise\-representative schema characteristics\. Prod\./Real DBs indicates evaluation on non\-synthetic production or competition database instances \(not schema design style\)\. ✓ = present,∘\\circ= partial,×\\times= absent\.BenchmarkYearDialectAvg\. TablesQuestionsEnterpriseOracleSD MetricProd\./Real DBsWikiSQL2017SQLite1\.080,654×\\times×\\times×\\times×\\timesSpider2018SQLite5\.110,181×\\times×\\times×\\times×\\timesSParC2019SQLite5\.13,034×\\times×\\times×\\times×\\timesCoSQL2019SQLite5\.13,007×\\times×\\times×\\times×\\timesSpider\-Syn2021SQLite5\.110,181×\\times×\\times×\\times×\\timesKaggleDBQA2021Mixed8\.0272∘\\circ×\\times×\\times✓Dr\. Spider2023SQLite5\.110,181×\\times×\\times×\\times×\\timesBIRD2023SQLite/PG7\.312,751∘\\circ×\\times×\\times∘\\circSpider 2\.02024Mixed12\.4632∘\\circ×\\times×\\times∘\\circESQ\-Bench2026Oracle\+PG\+MySQL\+SS10/48/177550✓✓✓∘\\circSix*synthetic enterprise\-representative*schemas \(465 tables, seeded\); 550 gold\-validated questions \(June 2026\)\.

## 4ESQ\-Bench Design

### 4\.1Design Principles

ESQ\-Bench is designed as a*diagnostic instrument*rather than merely a performance benchmark\. Four principles govern its design:

P1 : Tiered complexity\.Schema complexity is not binary\. A benchmark testing only simple or only complex schemas cannot produce degradation curves or identify complexity thresholds at which models fail\. ESQ\-Bench uses three tiers to make the degradation function measurable\.

P2 : Oracle\-first\.Every question, schema, and gold query targets Oracle Database\. This is not a dialect translation of an existing benchmark but a ground\-up design for Oracle semantics, syntax, and enterprise conventions\.

P3 : Metric completeness\.The benchmark provides semantic intent annotations and distractor queries enabling evaluation beyond execution match, including the novel Silent Divergence Rate metric defined in[Section˜5](https://arxiv.org/html/2608.23569#S5)\.

P4 : Full reproducibility\.All schemas, DDL, data population scripts, question records, gold queries, semantic annotations, and the evaluation harness are released publicly with pinned dependencies and model version specifications\.

### 4\.2Construction Status

As of the current release, all six schemas are fully defined, populated, and verified\. A single manifest drives dialect\-aware DDL generation \(generate\_schemas\.py\) and deterministic seeding \(seed\_all\.py\) with per\-table fixups that inject benchmark critical conditions \(NULL fractions, status\-domain separation, hierarchy depths\)\.[Table˜2](https://arxiv.org/html/2608.23569#S4.T2)summarises the built artifact\.

Table 2:ESQ\-Bench Construction Status \(June 2026\)\.Seeding enforces a*zero\-empty\-tables*invariant: every table receives at least a default row count before question authoring, so execution\-match evaluation never passes vacuously on empty relations\. Tier\-1 gold SQL was validated withvalidate\_tier\_gold\.py\(550/550 execute on Oracle; scalar expectation checks against seeded critical conditions\)\.

### 4\.3Schema Complexity Tiers

Each tier is characterised by four quantified complexity features:

- •Ambiguity Index \(AI\):count of column names appearing in≥\\geq3 tables with different value domains, divided by total column count\.
- •FK Coverage Ratio \(FCR\):enforced FK constraints divided by inferred referential relationships\.
- •Oracle Construct Density \(OCD\):distinct Oracle\-specific constructs required across all questions, divided by question count\.
- •Legacy Naming Ratio \(LNR\):columns matching an abbreviation pattern \(consonant clusters, underscores, no vowel runs\>2\>2\) divided by total columns\.

[Table˜3](https://arxiv.org/html/2608.23569#S4.T3)presents the complexity profiles of all six schemas\.

Table 3:Schema Complexity Profiles Across Tiers\.
AI = Ambiguity Index; FCR = FK Coverage Ratio; OCD = Oracle Construct Density; LNR = Legacy Naming Ratio\.
### 4\.4Schema Descriptions

#### 4\.4\.1Tier 1 : Academic Baseline

Tier 1 schemas are designed to replicate the complexity range of Spider’s upper quartile\. All FK relationships are enforced and documented\. Column naming follows standard conventions\.

T1\-A: Sales Order Management\.Ten tables spanning customers, orders, order lines, products, product categories, suppliers, warehouses, shipments, returns, and invoices\. Oracle constructs required: basicSELECT, multi\-join, aggregation, simple date arithmetic,FETCH FIRST\. This schema anchors the replication experiment: model accuracy on T1\-A should approximate published Spider scores\.

T1\-B: University Academic Records\.Ten tables spanning students, courses, enrolments, instructors, departments, grades, prerequisites, rooms, schedules, and semesters\. Oracle constructs:GROUP BY,HAVING, correlated subqueries,NVLfor grade defaults\.

#### 4\.4\.2Tier 2 : Enterprise Moderate

Tier 2 schemas introduce enterprise complexity: partial FK enforcement, mixed naming conventions, and column name ambiguity\.

T2\-A: Portfolio Management\.Forty\-eight tables spanning portfolios, positions, instruments, transactions, valuations, counterparties, benchmark indices, risk metrics, compliance rules, and settlement\. Key complexity features:STATUSappears in multiple tables with non\-overlapping value domains \(e\.g\., portfolio vs\. compliance\-rule status\); 50 portfolios haveNULLbenchmark identifiers; 15% of positionmarket\_valueisNULL\. Oracle constructs:DECODE,NVL,TO\_DATE,FETCH FIRST,MINUS, and analytic functions \(LAG/LEAD,RATIO\_TO\_REPORT\)\.

T2\-B: Healthcare Operations\.Fifty\-two tables spanning patients, encounters, diagnoses, procedures, medications, providers, insurance, claims, billing, and lab results\. Key complexity features: provider role ambiguity \(attending, ordering, referring mapped across different tables\); ICD code join complexity; date\-type proliferation \(admission, service, billing, effective\)\. Oracle constructs:NVL2,TRUNCdate arithmetic,CONNECT BYfor referral chains,LISTAGGfor diagnosis aggregation\.

#### 4\.4\.3Tier 3 : Enterprise Complex

Tier 3 schemas represent the complexity encountered in core enterprise Oracle deployments\. FK coverage drops to 40–43%; legacy abbreviation naming is pervasive; multiple overlapping conceptual representations arise from system migration history\.

T3\-A: Core Banking\.One hundred and seventy\-seven tables spanning account management, customer relationship, transaction processing, product catalogue, regulatory reporting, collateral, loan origination, deposit management, payment processing, and general ledger\. Column names follow legacy banking conventions \(ACCT\_BAL\_AMT\_USD,CUST\_REL\_TYP\_CD,TXN\_DT\_SEQ\_NO\)\. Multiple representations of the customer concept \(CUSTOMER,PARTY,CLIENT,COUNTERPARTY\) arise from system generations\. GL account hierarchy requiresCONNECT BYtraversal six levels deep\.

T3\-B: Insurance Policy Administration\.One hundred and sixty\-eight tables spanning policy management, claims processing, underwriting, reinsurance, actuarial, agent management, billing, and regulatory compliance\. Policy hierarchy \(Group→\\toMaster→\\toCertificate→\\toRider\) and claims hierarchy \(Claim→\\toClaimLine→\\toReserve→\\toPayment\) both requireCONNECT BY\. Actuarial tables use heavily abbreviated column names inherited from legacy mainframe systems\.

### 4\.5Question Construction Protocol

For each of the eight category groups in[Table˜4](https://arxiv.org/html/2608.23569#S4.T4)\(550 questions total, full per\-question catalog in the supplementary file\), the following five\-step protocol generates each question record:

1. 1\.Question authoring:Write a natural language question using business vocabulary\. Test: a domain expert without database training can state the question’s intent unambiguously\.
2. 2\.Gold query:Write the Oracle SQL gold query; execute against the populated schema; confirm a non\-trivial result set \(1–500 rows\)\.
3. 3\.Semantic intent:Write a plain\-English description of the intended result, explicit about NULL handling, boundary conditions, and ordering\. This description is independent of any SQL formulation\.
4. 4\.Distractor query:Write a plausible wrong query that executes without error but returns a semantically incorrect result set\. The distractor captures the most likely model failure mode for this category \(e\.g\.,NOT INinstead ofNOT EXISTSfor NULL trap questions\)\.
5. 5\.Blind verification:A second author independently writes a gold query from the natural language question and semantic intent description alone\. Agreement≥\\geq95% is required; disagreements trigger revision or reclassification as ambiguity\-category questions\.

Table 4:ESQ\-Bench Question Taxonomy: category groups, per\-tier counts, and the SQL competency each group exercises\.
Full per\-question catalog in supplementaryesq\_appendix\_questions\.pdf\.Category GroupT1T2T3Tot\.SQL competencySimple Select & Ordering25382487Projection, filter,FETCH FIRST/NULLS LASTAggregation & Conditional254436105GROUP BY/HAVING,CASE/DECODEMulti\-join & Set Ops2550341092\+table joins,UNION/INTERSECT/MINUSNested / Subquery10282058EXISTS/IN, correlated, anti\-joinTemporal10161440Date arithmetic, interval filtersAnalytic Functions0201232RANK/LAG/NTILEwindow funcsHierarchical0203656CONNECT BY, org/GL trees, depthEnterprise Traps†0125163NULL ambiguity, legacy names, entity overlapTotal95228227550†T2:*status\_ambiguity*\(12\); T3:*null\_trap*\(18\),*entity\_overlap*\(13\),*ambiguity\_resolution*\(10\),*legacy\_naming*\(10\)\.

## 5Evaluation Metrics

LetQ=\{q1,…,qN\}Q=\\\{q\_\{1\},\\ldots,q\_\{N\}\\\}be the set of natural language questions in ESQ\-Bench,qi∗q^\{\*\}\_\{i\}the gold Oracle SQL query for questionqiq\_\{i\},q^i\\hat\{q\}\_\{i\}the model\-generated query, and𝒟\\mathcal\{D\}the database instance\. LetR​\(q,𝒟\)R\(q,\\mathcal\{D\}\)denote the result set of executing queryqqon𝒟\\mathcal\{D\}\.

### 5\.1Exact Match \(EM\)

After normalisation \(lowercasing, whitespace standardisation, alias renaming,SELECTcolumn sorting\):

EM​\(i\)=𝟏​\[norm​\(q^i\)=norm​\(qi∗\)\]\\text\{EM\}\(i\)=\\mathbf\{1\}\[\\text\{norm\}\(\\hat\{q\}\_\{i\}\)=\\text\{norm\}\(q^\{\*\}\_\{i\}\)\]\(1\)EM is reported for comparability with prior work and known to underestimate correctness due to sensitivity to syntactically equivalent reformulations\.

### 5\.2Execution Match \(EX\)

EX​\(i\)=𝟏​\[R​\(q^i,𝒟\)=R​\(qi∗,𝒟\)\]\\text\{EX\}\(i\)=\\mathbf\{1\}\[R\(\\hat\{q\}\_\{i\},\\mathcal\{D\}\)=R\(q^\{\*\}\_\{i\},\\mathcal\{D\}\)\]\(2\)where equality is set\-based and order\-insensitive unless the question semantics require ordering\. EX is the field’s dominant metric and is reported here as the primary comparability measure\. Its limitation—inability to detect silent semantic divergence—motivates the SD metric below\.

### 5\.3Semantic Recall \(SR\)

SR measures the fraction of intended result rows returned by the generated query, independent of whether extraneous rows are also returned:

SR​\(i\)=\|R​\(q^i,𝒟\)∩R​\(qi∗,𝒟\)\|\|R​\(qi∗,𝒟\)\|\\text\{SR\}\(i\)=\\frac\{\|R\(\\hat\{q\}\_\{i\},\\mathcal\{D\}\)\\cap R\(q^\{\*\}\_\{i\},\\mathcal\{D\}\)\|\}\{\|R\(q^\{\*\}\_\{i\},\\mathcal\{D\}\)\|\}\(3\)SR captures over\-filtering failures \(the generated query returns a correct subset but misses rows\) that EX reports as zero\. Mean SR over EX\-failing questions quantifies how “close” wrong answers are\.

### 5\.4Silent Divergence Rate \(SD\)

Letdid\_\{i\}denote the distractor query for questionqiq\_\{i\}, and letℐi\\mathcal\{I\}\_\{i\}denote the semantic intent description—a set of explicit constraints on the intended result set including NULL inclusion/exclusion rules, boundary conditions, and ordering requirements\.

###### Definition 1\(Silent Divergence\)\.

A generated queryq^i\\hat\{q\}\_\{i\}exhibits*silent divergence*on questionqiq\_\{i\}if and only ifEX​\(i\)=1\\text\{EX\}\(i\)=1and at least one of the following holds:

1. \(a\)R​\(q^i,𝒟\)=R​\(di,𝒟\)R\(\\hat\{q\}\_\{i\},\\mathcal\{D\}\)=R\(d\_\{i\},\\mathcal\{D\}\)\(the generated result matches the distractor rather than reflecting genuine correctness\), or
2. \(b\)R​\(q^i,𝒟\)R\(\\hat\{q\}\_\{i\},\\mathcal\{D\}\)violates at least one explicit constraint inℐi\\mathcal\{I\}\_\{i\}when tested against a supplementary database instance𝒟′\\mathcal\{D\}^\{\\prime\}constructed to activate the constraint\.

SD=∑i:EX​\(i\)=1𝟏​\[SilentDivergence​\(i\)\]∑i𝟏​\[EX​\(i\)=1\]\\text\{SD\}=\\frac\{\\sum\_\{i:\\text\{EX\}\(i\)=1\}\\mathbf\{1\}\[\\text\{SilentDivergence\}\(i\)\]\}\{\\sum\_\{i\}\\mathbf\{1\}\[\\text\{EX\}\(i\)=1\]\}\(4\)
SD is defined at the benchmark level, not per\-question, and applies only to the subset of questions passing EX\. A non\-zero SD indicates that execution match is over\-reporting accuracy: some fraction of “passing” queries are wrong\.

#### 5\.4\.1Operational SD in the Released Harness

For automated evaluation we also report an*operational*silent\-divergence flag computed without distractor matching:

SDop​\(i\)=EX​\(i\)∧\(¬EM​\(i\)∨¬SR​\(i\)\)\\text\{SD\}\_\{\\text\{op\}\}\(i\)=\\text\{EX\}\(i\)\\land\\bigl\(\\neg\\text\{EM\}\(i\)\\lor\\neg\\text\{SR\}\(i\)\\bigr\)\(5\)WhenEX​\(i\)=1\\text\{EX\}\(i\)=1butEM​\(i\)=0\\text\{EM\}\(i\)=0, the model returned the correct rows via a different SQL path—including formulations that may diverge under shifted data \(e\.g\.,SUM\(col\)vs\.SUM\(NVL\(col,0\)\)on our probe pairt1a\-007/t1a\-008\)\. HighSDop\\text\{SD\}\_\{\\text\{op\}\}on Tier 1 therefore signals that EX\-centric leaderboards under\-report semantic risk even before Tier 2–3 complexity\. Formal SD \([Definition˜1](https://arxiv.org/html/2608.23569#Thmdefinition1)\) remains the target metric for questions with authored distractors and supplementary instances𝒟′\\mathcal\{D\}^\{\\prime\}\.

The supplementary instance𝒟′\\mathcal\{D\}^\{\\prime\}is constructed per question to activate the specific divergence mechanism \(e\.g\., inserting NULL values to trigger Oracle’s empty\-string\-as\-NULL coercion, or inserting hierarchy nodes at depth\>3\>3to exposeCONNECT BYvs\. self\-join discrepancies\)\. All supplementary instances are released with the benchmark\.

### 5\.5Failure Mode Taxonomy

For each EX\-failing query, we classify the failure into one of four modes:

- •F1 : Syntactic invalidity:the generated query raises anORA\-parse error\.
- •F2 : Valid\-standard, invalid\-Oracle:the query is syntactically valid in SQLite or PostgreSQL but rejected by Oracle \(e\.g\.,LIMIT,EXCEPT, bareSELECT exprwithoutFROM DUAL\)\.
- •F3 : Execution failure with wrong results:the query executes but returns a result set differing from the gold\.
- •F4 : Non\-idiomatic:the query executes and returns the correct result but uses non\-Oracle constructs where Oracle\-specific alternatives were required \(e\.g\.,CASEwhereDECODEwas specified\)\.

## 6Experiments

### 6\.1Experimental Setup

All results in[Section˜6](https://arxiv.org/html/2608.23569#S6)use the released 550\-question bank \([Table˜4](https://arxiv.org/html/2608.23569#S4.T4)\) unless noted as the historical v0\.1 mini\-pilot in Appendix[A](https://arxiv.org/html/2608.23569#A1)\. Gold SQL executes as schema owners on Oracle Autonomous Database 21c\.

Models\.Primary tier tables use GPT\-4o \(gpt\-4o\) via the SAL HTTP service \(sql\-learn\-server\.js,POST /generate\-sql\) with per\-tierschema\_contexthints \(tier\{1,2,3\}\_schema\_hints\.json\); the MCP server is restarted once per sweep withLLM\_MODEL=gpt\-4obefore Tier 1 \(run\_esq\_gpt4o\_all\_tiers\.sh\)\. Multi\-model cells in[Table˜11](https://arxiv.org/html/2608.23569#S6.T11)use the same Oracle evaluation pipeline viarun\_esq\_tier1\_evaluation\.pywith direct API clients \(esq\_llm\_client\.py\): GPT\-4o through OpenAI, Claude Sonnet 4\.6 through Anthropic \(claude\-sonnet\-4\-6, June 2026 API\), and Llama 3\.2 through Ollama \(llama3\.2:latest\)\. All models use temperature0and schema\-linked prompts unless noted as zero\-shot\.

Metrics\.EM, EX, SR, EXproj\{\}\_\{\\text\{proj\}\}, and operationalSDop\\text\{SD\}\_\{\\text\{op\}\}\([Section˜5\.4\.1](https://arxiv.org/html/2608.23569#S5.SS4.SSS1)\) computed byrun\_esq\_tier1\_evaluation\.py; result sets compared with order\-insensitive multiset equality and NULL\-safe sorting\. EX rates are reported over successfully*executed*model queries unless otherwise stated\.

### 6\.2Tier\-1 Results

[Tables˜5](https://arxiv.org/html/2608.23569#S6.T5),[6](https://arxiv.org/html/2608.23569#S6.T6)and[1](https://arxiv.org/html/2608.23569#S6.F1)summarise GPT\-4o on the 95\-question Tier\-1 bank \(94/95 executed;t1a\-023failed with Oracle parse errorORA\-03048\)\.

Table 5:Tier\-1 Results: GPT\-4o \+ Schema\-Linked SAL \(June 2026,N=95N=95\)\. SDop\{\}\_\{\\text\{op\}\}= operational silent divergence \([Section˜5\.4\.1](https://arxiv.org/html/2608.23569#S5.SS4.SSS1)\)\.[Table˜6](https://arxiv.org/html/2608.23569#S6.T6)breaks down EX by Spider\-aligned category\. Simple selects, aggregations, and set operations reach 100% EX; conditional logic \(44\.4%\) and multi\-join \(66\.7%\) show the largest remaining Tier\-1 gaps; ordering improves to 50\.0% EX on the expanded bank versus 0% on the v0\.1 mini\-pilot \(Appendix[A](https://arxiv.org/html/2608.23569#A1)\)\.

Table 6:Tier\-1 Execution Match \(%\) by Category \(GPT\-4o,N=95N=95\)\.![Refer to caption](https://arxiv.org/html/2608.23569v1/x1.png)Figure 1:Tier\-1 execution match by question category \(GPT\-4o, schema\-linked SAL, 95 questions\)\. Dashed line: overall EX \(79\.8%\)\.#### 6\.2\.1Tier\-1 Findings

Finding T1\-1 : EX overstates deployability even at Tier 1\.On Spider\-comparable schemas, only 6\.3% of generated queries exactly match gold SQL while 79\.8% match on execution—confirming that EM is a poor proxy and EX alone leaves a semantic gap\.

Finding T1\-2 : High operational SD on EX\-passing queries\.Among the 75 EX\-passing queries, 69 \(92\.0%\) triggerSDop\\text\{SD\}\_\{\\text\{op\}\}becauseEM=0\\text\{EM\}=0: the model frequently reaches the right answer via alternate SQL\. This motivates formal SD with distractors and thet1a\-007/t1a\-008probe pair \(SUM\(total\_amount\)vs\.SUM\(NVL\(total\_amount,0\)\)\), for which identical EX on a single instance would mask divergent NULL semantics\.

Finding T1\-3 : Category failures are structured but not uniform\.Conditional and ordering categories remain the weakest \(44\.4% and 50\.0% EX\), indicating systematic gaps in OracleCASE/NVLidioms andFETCH FIRSTpagination despite near\-perfect single\-table performance—a pattern that intensifies at Tiers 2–3 \([Section˜6\.5](https://arxiv.org/html/2608.23569#S6.SS5)\)\.

### 6\.3Tier\-2 Results

Tier 2 comprises 228 questions \(114PORTFOLIO\_MGMT, 114HEALTHCARE, v0\.3\) across twelve categories including*status\_ambiguity*, org/facility*hierarchical*traversals, and*analytic\_functions*\.[Table˜7](https://arxiv.org/html/2608.23569#S6.T7)and[Figure˜2](https://arxiv.org/html/2608.23569#S6.F2)report GPT\-4o results \(211/228 generated, 194 executed\)\. EX falls to 60\.3%; aggregation remains strong \(96\.2% EX\) but*hierarchical*stays at 0% EX on executed queries in this category \(schema\-linked hints insufficient for four\-table org trees\)\.

Table 7:Tier\-2 Results: GPT\-4o \+ Schema\-Linked SAL \(June 2026,N=228N=228\)\.\\begin\{overpic\}\[width=431\.62pt\]\{figures/fig\_tier2\_ex\_by\_category\.pdf\} \\put\(70\.0,0\.5\)\{\\color\[rgb\]\{1,1,1\}\\rule\{159\.3356pt\}\{48\.36958pt\}\} \\end\{overpic\}Figure 2:Tier\-2 execution match by category \(228 questions, GPT\-4o schema\-linked SAL, June 2026\)\. Overall EX: 60\.3%\.
### 6\.4Tier\-3 Results

Tier 3 comprises 227 questions \(114CORE\_BANKING, 113INSURANCE\) with 56*hierarchical*items \(CONNECT BY\),*entity\_overlap*joins, NULLreserve\_amounttraps, ambiguity\-resolution, legacy naming, and analytic window functions\.[Table˜8](https://arxiv.org/html/2608.23569#S6.T8)and[Figure˜3](https://arxiv.org/html/2608.23569#S6.F3)report GPT\-4o results \(226/227 generated, 201 executed\)\. Overall EX is 57\.2%;*hierarchical*EX reaches 86\.7% with explicitCONNECT BYhints, but*ambiguity\_resolution*\(0%\),*multi\_join*\(23\.5%\), and*ordering*\(10\.0%\) remain the weakest categories\.

Table 8:Tier\-3 Results: GPT\-4o \+ Schema\-Linked SAL \(June 2026,N=227N=227\)\.![Refer to caption](https://arxiv.org/html/2608.23569v1/x2.png)Figure 3:Tier\-3 execution match by category \(227 questions, GPT\-4o schema\-linked SAL, June 2026\)\. Overall EX: 57\.2%\.
### 6\.5Cross\-Tier Summary \(GPT\-4o\)

[Tables˜9](https://arxiv.org/html/2608.23569#S6.T9)and[10](https://arxiv.org/html/2608.23569#S6.T10)aggregate the three tier runs \([Sections˜6\.2](https://arxiv.org/html/2608.23569#S6.SS2),[6\.3](https://arxiv.org/html/2608.23569#S6.SS3)and[6\.4](https://arxiv.org/html/2608.23569#S6.SS4)\)\. EX*decreases*monotonically with tier complexity \(79\.8%→\\rightarrow60\.3%→\\rightarrow57\.2%\), confirming the degradation hypothesis at full bank scale—in contrast to the earlier 142\-question pilot \(Appendix[A](https://arxiv.org/html/2608.23569#A1)\) where Tier\-3 EX appeared artificially high on seeded critical\-condition items\. EM remains near zero \(0\.4–6\.3%\), andSDop\\text\{SD\}\_\{\\text\{op\}\}stays high \(55–73% among executed queries; 92–99% among EX\-passing queries\) because EX\-correct answers rarely match gold SQL verbatim\.[Table˜10](https://arxiv.org/html/2608.23569#S6.T10)summarises the F1–F4 failure taxonomy \(analyze\_esq\_failures\.py\); F3 wrong\-result semantics dominate \(170/243 failures\), while F1 invalid\-identifier errors rise from 1 \(Tier 1\) to 42 \(Tiers 2–3 combined\)\.

Table 9:GPT\-4o schema\-linked SAL on full ESQ\-Bench \(June 2026\)\.Table 10:Failure taxonomy on GPT\-4o full\-bank run \(F1–F4\)\.
### 6\.6Standard Baselines and Multi\-Model Grid

The evaluation harness \(run\_esq\_tier1\_evaluation\.py\) supports direct LLM providers \(OpenAI, Anthropic, Ollama\) under*schema\-linked*\(SL\) and*zero\-shot*\(ZS\) prompt conditions on the same Oracle execution pipeline\.run\_esq\_baselines\.shruns three standard cells: GPT\-4o ZS/SL, Claude Sonnet 4\.6 SL, and Llama 3\.2 SL \(Ollama\)\.

[Table˜11](https://arxiv.org/html/2608.23569#S6.T11)reports GPT\-4o under both prompt conditions and Claude Sonnet 4\.6 schema\-linked on all three tiers \(June 2026\)\. Spider/BIRD replication on our harness remains future work to confirm that tier degradation under schema\-linked prompting is benchmark\-driven rather than implementation artifact\.

Table 11:Multi\-Model Results: execution rate and EX by tier \(June 2026\)\. Exec = fraction of questions whose SQL executes on Oracle; EX = execution match\. EXexec\{\}\_\{\\text\{exec\}\}is over executed queries only; EXall\{\}\_\{\\text\{all\}\}counts non\-executed as failures\.##### Claude Sonnet 4\.6 vs\. GPT\-4o \(schema\-linked\)\.

Claude Sonnet 4\.6 \(claude\-sonnet\-4\-6\) exceeds GPT\-4o on EXexec\{\}\_\{\\text\{exec\}\}at every tier \(\+7\.6 pp / \+14\.6 pp / \+11\.5 pp\)\. The gap persists under bank\-wide EXall\{\}\_\{\\text\{all\}\}, where non\-executed queries count as failures: Tier 187\.4%vs\.78\.9%\(83/95 vs\. 75/95\), Tier 265\.4%vs\.51\.3%\(149/228 vs\. 117/228\), Tier 355\.1%vs\.50\.7%\(125/227 vs\. 115/227\); pooled64\.9%vs\.55\.8%\(357/550 vs\. 307/550\)\. Claude maintains comparable execution rates \(100\.0% / 87\.3% / 80\.2%\) to GPT\-4o SL \(98\.9% / 85\.1% / 88\.5%\), so the advantage is not an artifact of survivor bias \([Section˜6\.7](https://arxiv.org/html/2608.23569#S6.SS7)\) but reflects fewer F3 wrong\-result failures under the same schema\-linked prompt condition\.

### 6\.7Schema\-Linked vs\. Zero\-Shot Prompting \(GPT\-4o\)

Our primary degradation narrative uses schema\-linked \(SL\) SAL prompting—the production\-relevant setting where tierschema\_hintsare available\. A zero\-shot \(ZS\) baseline on the*same*Oracle harness \(esq\_llm\_client\.py, no schema context\) reveals a prompt\-condition interaction that EX\-over\-executed alone obscures \([Tables˜12](https://arxiv.org/html/2608.23569#S6.T12)and[11](https://arxiv.org/html/2608.23569#S6.T11)\)\.

Table 12:GPT\-4o: execution rate and EX by prompt condition \(June 2026\)\. “EX \(allNN\)” counts non\-executed queries as failures\.Execution\-rate gap\.SL raises the fraction of generated SQL that Oracle accepts: \+34\.7 pp \(Tier 1\), \+25\.5 pp \(Tier 2\), and \+37\.0 pp \(Tier 3\) versus ZS\. Schema hints suppress F1\-class failures \(ORA\-00904/ORA\-00942\) by grounding table and column identifiers—exactly the behaviour practitioners expect from SAL\.

Conditional EX inversion\.Among queries that*do*execute, ZS EX*exceeds*SL on Tiers 2–3 \(73\.5% vs\. 60\.3%; 77\.8% vs\. 57\.2%\) despite monotonic SL degradation across tiers\. We attribute this to*survivor bias*: ZS fails early on identifier hallucination, so the executed subset skews toward simpler single\-schema guesses that happen to match seeded names\. SL pushes models into multi\-table joins, status\-disambiguation, and Oracle constructs that execute but yield F3 wrong\-result failures—visible in the failure taxonomy \([Table˜10](https://arxiv.org/html/2608.23569#S6.T10)\), where F3 wrong\-result counts rise from 16 \(T1\) to 72–82 \(T2–T3\) under SL\.

Bank\-wide EX\.When non\-executed queries count as failures, SL still wins on Tier 1 \(78\.9% vs\. 50\.5%\) and Tier 2 \(51\.3% vs\. 43\.9%\); Tier 3 remains close \(50\.7% vs\. 40\.1%\)\. Prompting recommendations are therefore tier\-dependent: schema linking is essential for parseable Oracle SQL at scale, but EX reported only over executed queries can*overstate*ZS quality at Tiers 2–3\. Benchmarks should report both execution rate and bank\-wide EX when comparing prompt conditions\.

## 7Complexity Predictor Analysis

### 7\.1Feature Extraction

We compute nine schema complexity features per question from DDL analysis and gold query metadata\. Features are defined in[Table˜13](https://arxiv.org/html/2608.23569#S7.T13)\.

Table 13:Schema and Question Complexity Features Used in Regression Analysis\.
### 7\.2Logistic Regression on Execution Match

We fit a balanced logistic regression at the question level \(N=550N=550\) with binary EX as the response and the nine features in[Table˜13](https://arxiv.org/html/2608.23569#S7.T13)as predictors \(standardised; GPT\-4o schema\-linked results, June 2026\)\. In\-sample classification accuracy is 66\.0%; positive EX rate is 55\.8%\. Coefficients on the standardised scale \(larger magnitude⇒\\Rightarrowstronger association with EX failure\):

- •MJD\(β=−0\.68\\beta=\-0\.68\): per\-question join depth is the strongest negative predictor—deeper joins correlate with EX failure\.
- •TC\(β=−0\.60\\beta=\-0\.60\),AJD\(β=−0\.49\\beta=\-0\.49\),LNR\(β=−0\.36\\beta=\-0\.36\),OCD\(β=−0\.24\\beta=\-0\.24\): schema size, average join depth, legacy naming, and Oracle construct density all associate with lower EX\.
- •AI\(β=\+0\.06\\beta=\+0\.06\): ambiguity index shows a weak positive association \(counter to the deployability hypothesis at this scale\)\.
- •FCR,DD,NP: zero variance across the six schemas in v0\.3 \(uniform FK coverage, no column comments, identical nullable\-column ratio\); coefficients are not identifiable\.

These coefficients support the tier\-degradation pattern in[Table˜9](https://arxiv.org/html/2608.23569#S6.T9): complexity features tied to join depth and Oracle constructs predict cross\-tier EX drops better than raw table count alone\.

### 7\.3SD Probe Analysis

Four questions carry ansd\_probetag \(NULL\-trap pairs designed to diverge under naive aggregation\)\. On GPT\-4o schema\-linked results, all four execute correctly \(100% EX;N=4N=4\), versus 62\.5% EX on the remaining 485 executed non\-probe questions\. For the canonical NULL pairt1a\-007/t1a\-008, both queries pass EX but the model’st1a\-007prediction omitsNVL—a*silent NULL risk*even when row counts match on the seeded instance\. Formal SD execution \(goldSUMvs\.SUM\(NVL\(⋅\\cdot,0\)\)on seed data\) shows zero divergences on the four probes under current gold SQL; confirmed formal SD instances remain future work for Tier 2–3 distractors \([Section˜8\.3](https://arxiv.org/html/2608.23569#S8.SS3)\)\.

### 7\.4Practitioner Deployability Checklist

The regression results yield a deployability scoring rubric\. Practitioners can compute this score for their schema before committing to NL2SQL deployment:

1. 1\.Measure ambiguous column count \(names appearing in≥\\geq3 tables\)\. If\>10\>10, expect materially lower accuracy than Tier 1 full\-bank levels \([Table˜5](https://arxiv.org/html/2608.23569#S6.T5)\)\.
2. 2\.Measure FK coverage ratio\. If<0\.70<0\.70, budget for schema remediation or explicit schema documentation in prompts\.
3. 3\.Audit Oracle construct requirements\. Presence ofCONNECT BY,MERGE, orMODELclause in business queries requires model validation specifically against those constructs\.
4. 4\.Document ambiguous columns\. Regression shows legacy naming ratio \(LNR\) and join depth \(MJD/AJD\) dominate EX variance; column comments were absent in v0\.3 schemas \(DD=0=0\)—adding comments to high\-ambiguity tables is a low\-cost prompt aid\.
5. 5\.Count average join depth\. Queries requiring\>5\>5table joins are substantially more likely to fail regardless of other schema properties\.

## 8Discussion

### 8\.1Implications for Practitioners

Benchmark accuracy is not deployability accuracy\.GPT\-4o reaches 79\.8% EX on Tier 1 \(95 questions\) but only 57\.2% on Tier 3 \(227 questions\) under schema\-linked SAL—below published Spider/BIRD leaderboards for comparable models on SQLite—confirming that enterprise complexity materially reduces accuracy\.

Production validation requires beyond execution match\.92\.0% operational SD among EX\-passing Tier\-1 queries \(69/75\) shows that execution success alone leaves nearly all “passing” generations on non\-gold SQL paths\. Production pipelines should track EM/SR and, where available, formal SD with semantic intent and supplementary instances—not execution success alone\.

Schema documentation is a high\-ROI investment\.Documentation density could not be estimated in v0\.3 \(DD=0=0across all six schemas\), but join depth and legacy naming ratio are the strongest regression predictors of EX failure \([Section˜7](https://arxiv.org/html/2608.23569#S7)\)\. Adding column comments to the most ambiguous tables is actionable before any model training changes\.

### 8\.2Implications for Benchmark Designers

Future benchmarks should include enterprise\-complexity schema tiers and at least one production commercial dialect\. The KaggleDBQA approach of using real databases is the right direction; ESQ\-Bench extends it with Oracle specificity, systematic complexity tiers, and a metric framework capable of detecting silent divergence\.

SD rate should be reported alongside EX as a standard metric\. The infrastructure to compute it—semantic intent descriptions and distractor queries—is non\-trivial to build but reveals accuracy information qualitatively different from EX\.

### 8\.3Limitations

Scale\.The current release includes all 550 questions \(Tier\-1: 95, Tier\-2: 228, Tier\-3: 227\); category\-level statistics \(e\.g\.,N=10N=10for Tier\-3 ambiguity\-resolution\) support tier\-level comparisons\. Few\-shot and DAIL\-SQL baselines remain future work \([Section˜6\.6](https://arxiv.org/html/2608.23569#S6.SS6)\)\.

Schema representativeness\.ESQ\-Bench schemas are constructed to exhibit documented enterprise complexity characteristics\. They are not real production schemas\. Claims about “enterprise accuracy” refer to accuracy on schemas with enterprise\-like characteristics, not predictions for any specific organization’s deployment\.

SD measurement\.Tier\-1 reporting uses operationalSDop\\text\{SD\}\_\{\\text\{op\}\}\([Section˜5\.4\.1](https://arxiv.org/html/2608.23569#S5.SS4.SSS1)\) for automation; formal SD \([Definition˜1](https://arxiv.org/html/2608.23569#Thmdefinition1)\) requires distractors and supplementary instances still being authored for Tier 2–3\. SD\-probe questions \([Section˜7](https://arxiv.org/html/2608.23569#S7)\) show 100% EX but expose silent NULL risk when models omitNVL; formal SD on seed data yields zero confirmed divergences under current gold SQL\. The high Tier\-1SDop\\text\{SD\}\_\{\\text\{op\}\}rate primarily reflectsEX∧¬EM\\text\{EX\}\\land\\neg\\text\{EM\}rather than confirmed distractor matches\.

Model version sensitivity\.LLM capabilities change between versions\. All models are evaluated at pinned versions specified in[Section˜6](https://arxiv.org/html/2608.23569#S6)\. The evaluation harness and leaderboard enable re\-evaluation as new versions emerge\.

Oracle version scope\.Results are reported on Oracle Database 21c\. Some constructs available in Oracle 23ai \(JSON relational duality, AI vector search\) are not included\.

## 9Open Problems

OP1 : Dialect\-aware training\.No publicly available NL2SQL model is trained on Oracle\-specific SQL at scale\. What accuracy improvement is achievable with Oracle\-dialect fine\-tuning, and what minimum training set size is required? ESQ\-Bench’s 550 question\-query pairs provide a starting corpus\.

OP2 : Automated SD detection\.SD measurement currently requires human\-authored semantic intent descriptions\. Can SD be detected automatically via LLM\-based semantic comparison, formal query equivalence checking, or constraint extraction from natural language? Automated SD detection would enable production monitoring pipelines\.

OP3 : Schema complexity adaptation\.Schema\-linked prompting partially mitigates complexity\-driven degradation but does not close the Tier 3 gap\. What training\-time interventions—curriculum learning over complexity tiers, schema augmentation, Oracle\-specific pretraining—most efficiently improve Tier 3 performance?

OP4 : Multi\-turn enterprise evaluation\.ESQ\-Bench is single\-turn\. Enterprise NL2SQL interactions are iterative\. Multi\-turn accuracy on enterprise schemas, particularly under conversational risk accumulation frameworks that track safety violations across turns, is entirely unmeasured\.

OP5 : Cross\-dialect transfer\.Given ESQ\-Bench’s Oracle baseline, can models trained or evaluated on ESQ\-Bench improve performance on other enterprise dialects \(SQL Server T\-SQL, DB2 SQL/PL\) without dialect\-specific training? A cross\-dialect transfer study would determine whether enterprise complexity generalises across commercial databases\.

## 10Conclusion

We introduced ESQ\-Bench, an Oracle\-first NL2SQL benchmark with systematic complexity tiers and silent\-divergence evaluation\. We constructed six fully populated schemas \(465 tables, 164,682 rows, four SQL dialects, zero empty tables\), released 550 gold\-validated questions across three tiers \(95\+228\+227\), and reproducible evaluation with schema\-linked SAL: GPT\-4o reaches 79\.8% / 60\.3% / 57\.2% EX on the full bank while EM stays below 7%, confirming monotonic tier degradation and that execution match and exact match diverge sharply on enterprise Oracle schemas\.

These findings carry two practical messages\. For practitioners: even Tier\-1 \(Spider\-comparable\) schemas show 92% operational SD among EX\-passing queries \([Table˜5](https://arxiv.org/html/2608.23569#S6.T5)\); production validation must include semantic recall, failure taxonomy review, and SQL\-path auditing, not execution success alone\. For the research community: the complete 550\-question bank supports regression, multi\-model baselines \([Tables˜11](https://arxiv.org/html/2608.23569#S6.T11)and[6\.7](https://arxiv.org/html/2608.23569#S6.SS7)\), and reproducible leaderboard extension viarun\_esq\_baselines\.sh\.

ESQ\-Bench schemas, seed scripts, all 550 questions, failure taxonomy tools, and the evaluation harness \(\-\-tier 1\|2\|3\) are released with this paper; the community leaderboard will follow at[https://github\.com/sanmish4ds/esq\-bench](https://github.com/sanmish4ds/esq-bench)\.

## Appendix AHistorical Mini\-Pilot \(v0\.1\)

Before the full 550\-question bank \(v0\.3\), we ran an early Tier\-1 mini\-pilot on 46 questions \(v0\.1\) with GPT\-4o\-mini and schema\-linked SAL \(June 2026\)\.[Table˜14](https://arxiv.org/html/2608.23569#A1.T14)preserves those numbers for longitudinal comparison; they are*not*comparable directly to full\-bank GPT\-4o results in[Section˜6](https://arxiv.org/html/2608.23569#S6)because both the question set and model differ\.

Table 14:Tier\-1 v0\.1 Mini\-Pilot: GPT\-4o\-mini \(N=46N=46\)\.The 142\-question cross\-tier pilot slice cited in the abstract \(46\+48\+48\) similarly used mixed models and pre\-expansion banks; full\-bank GPT\-4o EX was 75\.6% / 80\.4% / 95\.8% on that slice versus 79\.8% / 60\.3% / 57\.2% after expansion to 550 questions\.

## References

- \[1\]S\. Chang, J\. Xu, T\. Sun, P\. Gao, Z\. Lin, Y\. Liang, K\. Guo,et al\.\(2023\)Dr\. Spider: a diagnostic evaluation benchmark towards text\-to\-SQL robustness\.InInternational Conference on Learning Representations,Cited by:[§3\.1](https://arxiv.org/html/2608.23569#S3.SS1.p5.1)\.
- \[2\]X\. Gan, J\. Chen, X\. Liu, B\. Zheng, Y\. Wan, D\. Song, B\. Lao, and M\. Jiang\(2021\)Towards robustness of text\-to\-SQL models against natural and realistic adversarial table perturbation\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics,pp\. 2007–2022\.Cited by:[§3\.1](https://arxiv.org/html/2608.23569#S3.SS1.p4.1)\.
- \[3\]D\. Gao, H\. Wang, Y\. Li, X\. Sun, Y\. Qian, B\. Ding, and J\. Zhou\(2023\)DAIL\-SQL: efficient prompt engineering for large language models in text\-to\-SQL\.InProceedings of the VLDB Endowment,Vol\.17,pp\. 243–256\.Cited by:[§1](https://arxiv.org/html/2608.23569#S1.p1.1)\.
- \[4\]Gartner Research\(2023\)Magic quadrant for cloud database management systems\.Note:Accessed 2026Cited by:[§1](https://arxiv.org/html/2608.23569#S1.p2.1)\.
- \[5\]C\. Lee, O\. Polozov, and M\. Richardson\(2021\)KaggleDBQA: realistic evaluation of text\-to\-SQL parsers\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics,pp\. 2261–2273\.Cited by:[§3\.1](https://arxiv.org/html/2608.23569#S3.SS1.p6.1)\.
- \[6\]F\. Lei, J\. Chen, Y\. Peng, T\. Li, H\. Wang, M\. Ge,et al\.\(2024\)Spider 2\.0: evaluating language models on real\-world enterprise text\-to\-SQL workflows\.arXiv preprint arXiv:2411\.07763\.Cited by:[§3\.1](https://arxiv.org/html/2608.23569#S3.SS1.p8.1)\.
- \[7\]J\. Li, B\. Hui, G\. Qu, J\. Yang, B\. Li, B\. Li, B\. Wang, B\. Qin, R\. Geng, N\. Huo,et al\.\(2023\)Can LLM already serve as a database interface? a BIg bench for large\-scale database grounded text\-to\-SQLs\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§1](https://arxiv.org/html/2608.23569#S1.p1.1),[§3\.1](https://arxiv.org/html/2608.23569#S3.SS1.p7.1)\.
- \[8\]X\. Liu, J\. Chen, J\. Shi, and Z\. Chen\(2023\)Divide and prompt: chain of thought prompting for text\-to\-SQL\.InFindings of EMNLP,Cited by:[§1](https://arxiv.org/html/2608.23569#S1.p1.1)\.
- \[9\]W\. A\. Woods\(1973\)Progress in natural language understanding: an application to lunar geology\.InProceedings of the June 4–8, 1973, National Computer Conference and Exposition,pp\. 441–450\.Cited by:[§1](https://arxiv.org/html/2608.23569#S1.p1.1)\.
- \[10\]T\. Yu, R\. Zhang, H\. Er, S\. Li, E\. Xue, B\. Pang, X\. V\. Lin, Y\. C\. Tan, T\. Shi, Z\. Li,et al\.\(2019\)CoSQL: a conversational text\-to\-SQL challenge towards cross\-domain natural language interfaces to databases\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing,pp\. 1962–1979\.Cited by:[§3\.1](https://arxiv.org/html/2608.23569#S3.SS1.p3.1)\.
- \[11\]T\. Yu, R\. Zhang, K\. Yang, M\. Yasunaga, D\. Wang, Z\. Li, J\. Ma, I\. Li, Q\. Yao, S\. Roman,et al\.\(2018\)Spider: a large\-scale human\-labeled dataset for complex and cross\-domain semantic parsing and text\-to\-SQL task\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,pp\. 3911–3921\.Cited by:[§1](https://arxiv.org/html/2608.23569#S1.p4.1),[§3\.1](https://arxiv.org/html/2608.23569#S3.SS1.p2.1)\.
- \[12\]T\. Yu, R\. Zhang, M\. Yasunaga, Y\. C\. Tan, X\. V\. Lin, S\. Li, H\. Er, I\. Li, B\. Pang, T\. Chen,et al\.\(2019\)SParC: cross\-domain semantic parsing in context\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,pp\. 4511–4523\.Cited by:[§3\.1](https://arxiv.org/html/2608.23569#S3.SS1.p3.1)\.
- \[13\]V\. Zhong, C\. Xiong, and R\. Socher\(2017\)Seq2SQL: generating structured queries from natural language using reinforcement learning\.InarXiv preprint arXiv:1709\.00103,Cited by:[§3\.1](https://arxiv.org/html/2608.23569#S3.SS1.p1.1)\.