UniQL: Towards Dialect-Universal Benchmarking for Text-to-SQL

# UniQL: Towards Dialect-Universal Benchmarking for Text-to-SQL
Source: [https://arxiv.org/html/2606.08018](https://arxiv.org/html/2606.08018)
Jianling Gao1,Chongyang Tao1,Jiayuan Bai1,Liu Yang1Xuanguang Pan1, Jinrui Liu1,Shihao Xing, Xiaohan Xu2, Jie Liang1, Shuai Ma1 1SKLCCSE, Beihang University 2The University of Hong Kong 1\{jianlingg,chongyang,baijiayuan,panxg,mashuai\}@buaa\.edu\.cn 2shawnxxh@gmail\.com

UniQL: Towards Dialect\-Universal Benchmarking for Text\-to\-SQL

Jianling Gao1, Chongyang Tao1, Jiayuan Bai1, Liu Yang1Xuanguang Pan1,Jinrui Liu1,Shihao Xing, Xiaohan Xu2, Jie Liang1, Shuai Ma11SKLCCSE, Beihang University2The University of Hong Kong1\{jianlingg,chongyang,baijiayuan,panxg,mashuai\}@buaa\.edu\.cn2shawnxxh@gmail\.com

## 1Introduction

Structured query languages remain the primary interface for interacting with modern database systems\. Despite decades of standardization efforts, the SQL ecosystem is still highly fragmented across database engines such as PostgreSQL, MySQL, Oracle, etc\. Although these systems all adopt SQL\-like interfaces, they often differ substantially in syntax, built\-in functions, and optimization behaviors\. As a result, semantically equivalent queries frequently require dialect\-specific implementations, making SQL portability a long\-standing challenge in real\-world data systems\.

Meanwhile, recent advances in large language models \(LLMs\) have rapidly shifted data interaction from SQL\-centric interfaces toward natural\-language\-centric interfaces\. Modern text\-to\-SQL systems increasingly allow users to express querying intent directly in natural languageWanget al\.\([2025](https://arxiv.org/html/2606.08018#bib.bib227)\); Pourreza and Rafiei \([2023](https://arxiv.org/html/2606.08018#bib.bib228)\); Qinet al\.\([2025](https://arxiv.org/html/2606.08018#bib.bib229)\); Pourrezaet al\.\([2025](https://arxiv.org/html/2606.08018#bib.bib230)\), reducing the need to manually write dialect\-specific SQL queries\. This trend raises the possibility of a more universal querying paradigm, where natural language serves as aunified interfaceacross heterogeneous database systems\. Such a paradigm is increasingly important in the era of data agents and automated analytics, where user requests may involve querying, integrating, and operating over different database systems rather than interacting with a single backend\.

However, current text\-to\-SQL evaluation remains largely SQLite\-centric, making it unclear whether existing models can support such cross\-system querying in practice\. Existing mainstream benchmarks such as SpiderYuet al\.\([2018](https://arxiv.org/html/2606.08018#bib.bib209)\)and BIRDLiet al\.\([2023](https://arxiv.org/html/2606.08018#bib.bib231)\)mainly focus on SQLite execution and therefore cannot evaluate executable cross\-dialect generalization\. More recent benchmarks such as Spider 2\.0Leiet al\.\([2025](https://arxiv.org/html/2606.08018#bib.bib232)\)introduce enterprise\-level text\-to\-SQL workflows over different database systems, but their tasks are system\-specific rather than aligned realizations of the same natural language intents across dialects\. As a result, they cannot directly isolate dialect variation from differences in schemas, tasks, or database environments\. Queries that are executable and semantically correct in one dialect may fail, behave differently, or require substantial reformulation in another database system due to differences in execution semantics, function libraries, ordering behavior, implicit type casting, aggregation rules, and dialect\-specific syntax\. Consequently, current benchmark settings may overestimate the robustness and universality of text\-to\-SQL systems, while overlooking their sensitivity to dialect variation\.

To bridge this gap, we introduceUniQL, a human\-verified executable benchmark for cross\-dialect text\-to\-SQL evaluation\. Built upon the BIRD development set,UniQLaligns 1,534 natural language questions with executable SQL realizations across 16 SQL dialects, yielding 24,544 dialect\-specific SQL annotations\. All dialects share the same natural language intents, aligned database schemas and underlying database contents, enabling controlled evaluation of dialect generalization\. To construct the benchmark, we develop a hybrid pipeline that combines database migration, tool\-based translation, LLM\-based translation, self\-reflection with execution feedback, iterative translation rule evolution, and human validation for long\-tail cases\. UsingUniQL, we evaluate a broad set of open\-source and closed\-source LLMs in an inference\-only setting\. Our results reveal a gap that is largely hidden by existing SQLite\-centered evaluation: current LLMs can often solve an intent in some database systems, but fail to express the same intent consistently across dialects\. Even strong models solve only about half of the benchmark on average, exhibit large model–dialect interactions, and show limited transfer from SQLite correctness to other database systems\. These findings suggest that dialect\-universal text\-to\-SQL is not merely a matter of improving overall model capability, but requires explicit evaluation and modeling of cross\-dialect robustness\. In summary, the contributions of this paper are as follows:

- •We introduceUniQL, a human\-verified executable benchmark that aligns the same natural language intents with SQL realizations across 16 dialects\.
- •We propose a SQL translation framework for constructingUniQL, which integrates tool\-based translation, LLM\-based translation, execution\-based verification, self\-reflection with execution feedback, iterative translation rule evolution, and human validation\.
- •We comprehensively evaluate open\-weight and closed\-weight LLMs onUniQL, showing that current models still struggle with dialect\-universal text\-to\-SQL generation\.

## 2Related Work

Text\-to\-SQL benchmarks have been widely studied for evaluating executable SQL generation from natural language\. Early semantic parsing and natural language interface datasets, such as ATISDahlet al\.\([1994](https://arxiv.org/html/2606.08018#bib.bib252)\), GeoQueryZelle and Mooney \([1996](https://arxiv.org/html/2606.08018#bib.bib249)\), and RestaurantsTang and Mooney \([2000](https://arxiv.org/html/2606.08018#bib.bib250)\), focused on domain\-specific database querying\. Subsequent work revisited text\-to\-SQL evaluation methodology and standardized multiple datasetsFinegan\-Dollaket al\.\([2018](https://arxiv.org/html/2606.08018#bib.bib251)\)\. Later benchmarks scaled the task from single\-table queries in WikiSQLZhonget al\.\([2017](https://arxiv.org/html/2606.08018#bib.bib191)\)to complex cross\-domain multi\-table queries in SpiderYuet al\.\([2018](https://arxiv.org/html/2606.08018#bib.bib209)\), and further to value\-grounded realistic queries in BIRDLiet al\.\([2023](https://arxiv.org/html/2606.08018#bib.bib231)\)\. However, these benchmarks are primarily SQLite\-based\. Recent benchmarks such as Spider 2\.0Leiet al\.\([2025](https://arxiv.org/html/2606.08018#bib.bib232)\)cover multiple database systems, but their tasks are system\-specific rather than aligned realizations of the same natural language intents across dialects\. Conversational datasets such as SParCYuet al\.\([2019b](https://arxiv.org/html/2606.08018#bib.bib159)\)and CoSQLYuet al\.\([2019a](https://arxiv.org/html/2606.08018#bib.bib158)\)focus on multi\-turn interactions but also remain largely single\-dialect\. In contrast,UniQLaligns the same natural language intents with executable SQL realizations across 16 human\-verified dialects, enabling controlled evaluation of cross\-dialect robustness\.

Another related line of work studies SQL dialect translation, which rewrites a source SQL query into an equivalent query for another database system\. Rule\-based systems such as SQLGlot[SQLGlot](https://arxiv.org/html/2606.08018#bib.bib233), JOOQ[JOOQ](https://arxiv.org/html/2606.08018#bib.bib234), and SQLines[SQLines](https://arxiv.org/html/2606.08018#bib.bib235)rely on manually maintained dialect mappings, while recent LLM\-based or hybrid systems such as MALLETNgom and Kraska \([2024](https://arxiv.org/html/2606.08018#bib.bib236)\), RISEXieet al\.\([2026](https://arxiv.org/html/2606.08018#bib.bib238)\), and CrackSQLZhouet al\.\([2025b](https://arxiv.org/html/2606.08018#bib.bib184),[a](https://arxiv.org/html/2606.08018#bib.bib185)\)incorporate LLMs, execution feedback, or dialect\-specific rules\. Recent benchmarks such as PARROTZhouet al\.\([2026](https://arxiv.org/html/2606.08018#bib.bib237)\)further evaluate cross\-system SQL\-to\-SQL translation across many production\-grade database systems\. These efforts are complementary toUniQL: they focus on SQL\-to\-SQL portability, whereasUniQLevaluates whether models can directly generate executable, dialect\-specific SQL from natural language under aligned cross\-dialect conditions\.

## 3Dataset Construction

![Refer to caption](https://arxiv.org/html/2606.08018v1/x1.png)Figure 1:UniQLconstruction pipeline\.UniQLextends the widely\-used BIRD SQLite development set to 16 SQL dialects through database migration, hybrid SQL translation, execution\-based verification, iterative translation rule evolution, and human validation\.Constructing a high\-quality cross\-dialect text\-to\-SQL benchmark requires more than translating SQL strings\. A translated query must be executable on the target database system and must preserve the intent of the original natural language question\. The overall construction pipeline ofUniQLis presented in Figure[1](https://arxiv.org/html/2606.08018#S3.F1)\.UniQLextends the BIRD development set from SQLite to 16 SQL dialects, producing 24,544 executable SQL annotations for 1,534 natural language questions\. SQLite serves as the source dialect, while the other 15 target dialects are constructed through a hybrid pipeline that combines database migration, automated SQL translation, execution\-based verification, iterative translation rule evolution, and human validation\.

### 3\.1Database Migration

Since executable cross\-dialect evaluation requires running SQL queries on actual database systems, we first migrate the original BIRD databases to each target system\. Although the migrated databases are designed to preserve the original schemas and values as much as possible, exact one\-to\-one migration is not always possible across heterogeneous DBMSs\. Different systems vary in type names, identifier rules, case sensitivity, namespace organization, reserved keywords, and supported constraints\. For example, some systems organize data through schemas or users rather than standalone database namespaces, while others impose different conventions on table and column names\. We therefore perform lightweight schema and data normalization during migration, including type mapping, identifier normalization, namespace adaptation, and necessary formatting changes\. This migration step provides the execution environment needed for validating translated SQL queries under each target dialect\.

### 3\.2Automated Translation and Verification

Given a NL questionxx, its source SQLite queryqsq\_\{s\}, the source databaseDsD\_\{s\}, and the migrated target databaseDtD\_\{t\}, the goal is to construct a target queryqtq\_\{t\}in dialectτt\\tau\_\{t\}such thatqtq\_\{t\}is executable onDtD\_\{t\}and preserves the semantics ofqsq\_\{s\}\. We first apply a deterministic rule\-based translator \(e\.g\., SQLglot[SQLGlot](https://arxiv.org/html/2606.08018#bib.bib233)\) to obtain an initial target query:

qt\(0\)=ℱt​o​o​l​\(qs,τs,τt\),q\_\{t\}^\{\(0\)\}=\\mathcal\{F\}\_\{tool\}\(q\_\{s\},\\tau\_\{s\},\\tau\_\{t\}\),\(1\)whereτs\\tau\_\{s\}is SQLite andτt\\tau\_\{t\}is the target dialect\. This step efficiently handles common syntactic mappings and standard SQL constructs\.

The translated query is then checked through execution\-based verification\. LetE​\(q,D\)E\(q,D\)denote the execution result of queryqqon databaseDD\. A translated query is automatically accepted only when it executes successfully on the target database and its result is equivalent to the source execution result under our conservative verification protocol:

𝒱​\(qs,qt,Ds,Dt\)=𝕀​\[E​\(qt,Dt\)≡E​\(qs,Ds\)\]\.\\mathcal\{V\}\(q\_\{s\},q\_\{t\},D\_\{s\},D\_\{t\}\)=\\mathbb\{I\}\\left\[E\(q\_\{t\},D\_\{t\}\)\\equiv E\(q\_\{s\},D\_\{s\}\)\\right\]\.\(2\)
Here, we use a conservative acceptance criterion during construction detailed in Section[5](https://arxiv.org/html/2606.08018#S5)\. Standard execution accuracy in prior single\-dialect benchmarks often compares query outputs after converting them into unordered sets\. This can introduce false positives in cross\-dialect construction, since ordering information and duplicate multiplicities may be discarded\. Therefore, our automatic verification preserves ordering when the query has explicit ordering semantics and preserves duplicate\-sensitive outputs in unordered comparison\. At the construction stage, this conservative verification strategy ensures the quality of the automatically accepted translations\.

If the rule\-based translation cannot be automatically accepted, we invoke an LLM\-based translator conditioned on the source SQL, target schema, dialect information, and the current rule setℛ\\mathcal\{R\}:

qt\(1\)=ℱL​L​M​\(qs,St,τt,ℛ\),q\_\{t\}^\{\(1\)\}=\\mathcal\{F\}\_\{LLM\}\(q\_\{s\},S\_\{t\},\\tau\_\{t\},\\mathcal\{R\}\),\(3\)whereStS\_\{t\}denotes the target schema\. When the generated query still fails equivalence verification, the model enters a bounded self\-reflection loop\. At iterationkk, the feedback objectFkF\_\{k\}contains the previous target SQL, execution errors, or result mismatches\. The model then refines the query as:

qt\(k\+1\)=ℱr​e​f​l​e​c​t​i​o​n​\(qs,qt\(k\),Fk,St,τt,ℛ\)\.q\_\{t\}^\{\(k\+1\)\}=\\mathcal\{F\}\_\{reflection\}\(q\_\{s\},q\_\{t\}^\{\(k\)\},F\_\{k\},S\_\{t\},\\tau\_\{t\},\\mathcal\{R\}\)\.\(4\)This process repeats until the query is automatically accepted or the maximum number of refinement rounds is reached\. In our implementation, the LLM translator is instantiated with GPT\-5\-mini\. For each failed translation, we allow up to three execution\-feedback reflection rounds before routing the case to later construction stages\. The translator prompt template is provided in Appendix[B](https://arxiv.org/html/2606.08018#A2)\.

### 3\.3Iterative Translation Rule Evolution

Automatic translation failures are not treated as isolated query\-level errors\. Instead, we collect them into a failure log and abstract recurring error patterns into reusable dialect transformation rules\. For each target dialect, letℒf​a​i​l\\mathcal\{L\}\_\{fail\}denote the set of failed translations after execution\-guided refinement\. These failures are analyzed together with the current rule setℛn\\mathcal\{R\}^\{n\}and dialect documentation to produce an updated rule set:

ℛn\+1=𝒢r​u​l​e​\(ℛn,ℒf​a​i​l,𝒟d​o​c\),\\mathcal\{R\}^\{n\+1\}=\\mathcal\{G\}\_\{rule\}\(\\mathcal\{R\}^\{n\},\\mathcal\{L\}\_\{fail\},\\mathcal\{D\}\_\{doc\}\),\(5\)where𝒢r​u​l​e\\mathcal\{G\}\_\{rule\}denotes the rule summarization process and𝒟d​o​c\\mathcal\{D\}\_\{doc\}denotes target\-dialect documentation or tutorials\.

The refined rule set is then incorporated into subsequent translation attempts\. This feedback loop allows the construction pipeline to progressively handle repeated dialect\-specific failures, such as function rewriting, type conversion, date and time operations, aggregation behavior, or system\-specific syntax constraints\. In this way, the pipeline combines the efficiency of deterministic translation, the flexibility of LLM\-based rewriting, and the reusability of accumulated dialect knowledge\. We instantiate the rule summarizer with Gemini\-2\.5\-Pro and run rule evolution for three rounds\. The summarizer prompt template is provided in Appendix[B](https://arxiv.org/html/2606.08018#A2)\.

### 3\.4Human Verification

After automated translation, self\-reflection, and rule refinement, the remaining cases are routed to human verification\. Human verification serves as the final quality\-control stage for examples that cannot be reliably accepted by automatic execution verification\. These cases include genuine translation failures, unsupported dialect constructs, and examples where source and target executions differ due to dialect\-dependent behavior or under\-specified semantics in the original SQL\. For example, different systems may return different row orders when the query does not fully specify tie\-breaking, or duplicate\-sensitive outputs may require semantic judgment beyond simple set comparison\.

Given a candidate target queryqtq\_\{t\}, the original NL questionxx, and the target databaseDtD\_\{t\}, human annotators correct or rewrite the query while preserving the intent of the NL question:

qt∗=𝒜​\(x,qs,qt,Dt,τt\),q\_\{t\}^\{\*\}=\\mathcal\{A\}\(x,q\_\{s\},q\_\{t\},D\_\{t\},\\tau\_\{t\}\),\(6\)whereqt∗q\_\{t\}^\{\*\}is the final human\-verified SQL annotation for the target dialect\. Each routed case is independently reviewed by two annotators, who check both executability and semantic consistency against the original NL question and target database\. If the candidate SQL is invalid, ambiguous, or not semantically equivalent, annotators rewrite it into an executable target\-dialect query\. Disagreements are resolved through discussion, and the final annotation is obtained only after consensus is reached\.

Overall,UniQL’s construction pipeline follows a strict automatic acceptance and human\-in\-the\-loop verification strategy\. The automatic stages efficiently resolve the majority of translations, while human verification handles long\-tail cases that require semantic judgment\. This process yields complete executable SQL coverage for 15 target dialects and, together with the original SQLite annotations, forms a 16\-dialect benchmark for controlled cross\-dialect text\-to\-SQL evaluation\.

## 4Data Statistics

#### Overall Statistics\.

UniQLis constructed from the BIRD development setLiet al\.\([2023](https://arxiv.org/html/2606.08018#bib.bib231)\)and contains 1,534 natural language questions over 11 databases\. For each question, we provide executable SQL annotations across 16 SQL dialects, yielding 24,544 dialect\-specific SQL queries in total\. The covered dialects are SQLite, ClickHouse, Doris, Drill, Druid, DuckDB, Hive, MySQL, Oracle, PostgreSQL, Presto, Spark, StarRocks, Teradata, Trino, and T\-SQL\. SQLite serves as the source dialect, while the remaining 15 dialects are constructed through our hybrid construction pipeline\. Since all dialects share the same natural language questions, aligned schemas and database contents,UniQLenables controlled evaluation of dialect generalization without conflating dialect variation with changes in tasks, schemas, or domains\.

#### Comparison with Existing Benchmarks\.

Table[1](https://arxiv.org/html/2606.08018#S4.T1)comparesUniQLwith representative text\-to\-SQL benchmarks\. Existing benchmarks are either single\-dialect, such as WikiSQLZhonget al\.\([2017](https://arxiv.org/html/2606.08018#bib.bib191)\), Spider 1\.0Yuet al\.\([2018](https://arxiv.org/html/2606.08018#bib.bib209)\), and BIRDLiet al\.\([2023](https://arxiv.org/html/2606.08018#bib.bib231)\), or cover multiple systems without aligning the same natural language intents across dialects, such as Spider 2\.0\-liteLeiet al\.\([2025](https://arxiv.org/html/2606.08018#bib.bib232)\)\. In contrast,UniQLexpands each natural language question into executable SQL annotations across 16 dialects, making it suitable for controlled cross\-dialect text\-to\-SQL evaluation\.

Table 1:Comparison betweenUniQLand representative text\-to\-SQL benchmarks\.Benchmark\#Q\#SQL\#DialectAligned SQLWikiSQL80,65477,8401–BIRD12,75112,7511–Spider 1\.010,1815,6931–Spider 2\.0\-lite5475476✗UniQL1,53424,54416✓
#### Construction Pipeline Statistics\.

Figure[2](https://arxiv.org/html/2606.08018#S4.F2)summarizes how the 15 target dialects are completed through different stages of the construction pipeline\. All target dialects are eventually completed with 1,534 executable SQL annotations, ensuring full alignment with the original BIRD development questions\. The statistics reveal substantial variation in construction difficulty across dialects: several dialects, such as MySQL and Doris, can largely be handled by SQLGlot, while Teradata and Hive require more LLM translation, reflection, rule refinement, or human annotation\. This variation shows that reliable cross\-dialect benchmark construction cannot be reduced to one\-shot SQL translation, since different database systems expose different long\-tail incompatibilities in syntax, functions, type handling, and execution behavior\.

![Refer to caption](https://arxiv.org/html/2606.08018v1/x2.png)Figure 2:Construction pipeline statistics for the 15 target dialects inUniQL\.

## 5Evaluation Metrics

We use execution accuracy \(EX\) as the main evaluation metric inUniQL\. The same execution\-based protocol is used for both automatic verification during dataset construction and model evaluation during experiments\. During construction, EX serves as a conservative acceptance criterion: a translated SQL query is automatically accepted only when it executes successfully on the target database and produces results consistent with the source query; otherwise, uncertain cases are routed to human verification\. During model evaluation, the predicted SQL and the gold SQL are executed on the same target database, and the prediction is considered correct only when their execution results match\.

Different from BIRD\-style EX implementations that convert query outputs into unordered sets, our protocol preserves ordering when explicit ordering semantics are present and preserves duplicate multiplicities for unordered outputs\. This stricter comparison avoids accepting predictions that return the correct rows in the wrong order or with incorrect duplicate counts\. We provide the detailed formal definition of the metric in Appendix[A](https://arxiv.org/html/2606.08018#A1)\.

## 6Experiments

Table 2:Main results onUniQL\. We report execution accuracy \(%\) across 16 SQL dialects\. All models are evaluated in an inference\-only setting without task\-specific training\. DS means DeepSeek\.ModelSQL DialectAvg\.ClickHouse Doris Drill Druid DuckDB Hive MySQL Oracle PostgreSQL Presto Spark SQLite StarRocks T\-SQL Teradata Trino Closed\-Source LLMsGPT\-3\.5\-Turbo35\.1438\.3329\.2022\.7537\.0936\.1840\.8149\.3536\.4436\.3139\.3741\.1339\.1837\.7423\.8635\.8536\.17GPT\-5\-mini51\.5649\.8048\.8337\.7452\.7454\.0454\.3060\.5651\.3750\.9852\.9352\.4851\.8953\.8535\.8549\.0250\.50GPT\-5\.1\-codex52\.6149\.9348\.1138\.2052\.9355\.4854\.8949\.6752\.3552\.4153\.6553\.7852\.4154\.6335\.0750\.7250\.43Gemini\-2\.5\-Pro53\.9851\.7650\.8537\.4256\.1959\.3257\.8234\.2953\.5957\.8956\.6559\.7854\.4355\.6138\.4055\.6152\.10Claude\-4\.5\-Sonnet56\.8452\.7452\.2839\.9055\.2859\.5858\.5463\.7554\.9558\.0856\.1359\.8456\.0656\.2637\.7456\.0654\.63Open\-Source LLMsQwen3\-1\.7B33\.9035\.0727\.5718\.7135\.2736\.7035\.8543\.4832\.9232\.4635\.2735\.4036\.3132\.6622\.4334\.1633\.01Qwen3\-4B43\.0244\.5241\.7226\.5344\.2646\.7446\.6144\.9242\.7041\.7243\.9446\.9445\.5046\.4129\.6041\.4642\.29Qwen3\-8B46\.5446\.1544\.1331\.2947\.0044\.8550\.2051\.1146\.3532\.9947\.4649\.0947\.4648\.5729\.3444\.2044\.17Qwen3\-32B49\.7447\.2046\.5433\.8349\.9351\.9652\.0253\.0648\.5047\.7849\.2253\.3950\.5251\.6932\.2746\.6147\.77Llama\-3\-8B\-Inst20\.8023\.9918\.4516\.5622\.2922\.2322\.0331\.2322\.4322\.2323\.3423\.6024\.1221\.1215\.4522\.7522\.04Llama\-3\-70B\-Inst40\.1640\.0937\.7426\.9241\.7242\.6343\.7451\.5640\.1638\.0142\.8942\.2442\.5740\.6126\.4739\.0539\.78DeepSeek\-Coder\-16B32\.4631\.1032\.0721\.3832\.7234\.8836\.5745\.4431\.8831\.5534\.4235\.0134\.8831\.6819\.9532\.5332\.41DeepSeek\-v4\-flash48\.0446\.9446\.2834\.2249\.0252\.8052\.5457\.5048\.5049\.0251\.5653\.4650\.2051\.4330\.9047\.5948\.12Avg\.43\.4542\.8940\.2929\.6544\.3445\.9546\.6148\.9243\.2442\.4245\.1446\.6345\.0444\.7929\.0342\.74–

### 6\.1Experimental Setup

#### Evaluation Setting\.

We evaluate all models on the 16 SQL dialects inUniQL\. For each test instance, the model is given the target SQL dialect, the database schema, and the natural language question, and is required to generate one executable SQL query in the specified dialect\. All models are evaluated once in an inference\-only setting, without task\-specific supervised fine\-tuning, reinforcement learning, few\-shot example selection, or dialect\-specific adaptation\. This setting is designed to measure the out\-of\-the\-box cross\-dialect text\-to\-SQL capability of foundation models\.

To ensure a fair comparison, all models use the same prompt template\. The template is instantiated with different target dialects, schemas, and natural language questions for different test cases, while the instruction format remains unchanged across models\. The full prompt template is provided in Appendix[B](https://arxiv.org/html/2606.08018#A2)\. For evaluation, we use the same EX protocol as described in Section[5](https://arxiv.org/html/2606.08018#S5); although this protocol is stricter than the set\-based EX used in some prior benchmarks, it is more appropriate for comparing model predictions against golden SQL executions on the target database, where ordering semantics and duplicate\-sensitive outputs should be preserved when relevant\.

#### Models\.

We evaluate both open\-source and closed\-source LLMs\. The closed\-source models include GPT\-3\.5\-TurboOpenAI \([2023](https://arxiv.org/html/2606.08018#bib.bib243)\), GPT\-5\-miniSinghet al\.\([2026](https://arxiv.org/html/2606.08018#bib.bib244)\), GPT\-5\.1\-codexOpenAI \([2025](https://arxiv.org/html/2606.08018#bib.bib245)\), Gemini\-2\.5\-ProComanici and others \([2025](https://arxiv.org/html/2606.08018#bib.bib246)\), and Claude\-4\.5\-Sonnet[Anthropic](https://arxiv.org/html/2606.08018#bib.bib240)\. The open\-source models include the Qwen3 seriesYanget al\.\([2025](https://arxiv.org/html/2606.08018#bib.bib241)\), Llama\-3\-Instruct modelsDubeyet al\.\([2024](https://arxiv.org/html/2606.08018#bib.bib221)\), DeepSeek\-Coder\-V2\-Lite\-16BZhuet al\.\([2024](https://arxiv.org/html/2606.08018#bib.bib247)\), and DeepSeek\-v4\-flashDeepSeek\-AI \([2026](https://arxiv.org/html/2606.08018#bib.bib239)\)\. The Qwen3 models cover multiple parameter scales, allowing us to analyze scaling behavior\. DeepSeek\-Coder\-V2\-Lite\-16B is denoted as DeepSeek\-Coder\-16B for compactness\.

### 6\.2Main Results

Table[2](https://arxiv.org/html/2606.08018#S6.T2)reports execution accuracy across all 16 SQL dialects\. Overall,UniQLremains challenging for current LLMs\. The best\-performing model, Claude\-4\.5\-Sonnet, achieves an average EX of 54\.63%, followed by Gemini\-2\.5\-Pro with 52\.10% and GPT\-5\-mini with 50\.50%\. Even these strong models fail to solve a large fraction of examples, indicating that executable cross\-dialect text\-to\-SQL generation remains far from saturated\.

Observation 1: Stronger models perform better, but the task remains challenging\.Frontier API\-based models achieve the strongest overall results, with Claude\-4\.5\-Sonnet leading on most dialects\. However, the best average EX is still only 54\.63%, showing thatUniQLexposes substantial remaining limitations even for advanced LLMs\. Among open\-source models, DeepSeek\-v4\-flash and Qwen3\-32B obtain the strongest averages, but they still leave considerable room for improvement\.

Observation 2: SQLite\-only evaluation hides dialect\-specific weaknesses\.BecauseUniQLaligns the same natural language intents across dialects, it allows direct comparison of model behavior under different target SQL dialects\. The results show that performance on SQLite does not fully reflect cross\-dialect ability\. For example, Claude\-4\.5\-Sonnet achieves 59\.84% on SQLite but 63\.75% on Oracle and 37\.74% on Teradata\. These gaps demonstrate that single\-dialect benchmarks can miss important model–dialect interactions\.

Observation 3: Scaling helps within model families\.The Qwen3 series exhibits a clear scaling trend\. Average EX increases from 33\.01% for Qwen3\-1\.7B to 42\.29% for Qwen3\-4B, 44\.17% for Qwen3\-8B, and 47\.77% for Qwen3\-32B\. A similar trend appears in the Llama family, where Llama\-3\-70B\-Instruct substantially improves over Llama\-3\-8B\-Instruct\. These results suggest that model capacity benefits cross\-dialect SQL generation\. At the same time, scaling mainly improves overall capability rather than eliminating dialect\-level variation: difficult dialects such as Druid, Presto, and Teradata remain challenging even for larger models\.

Observation 4: Dialect performance reflects model familiarity as well as execution behavior\.The bottom row of Table[2](https://arxiv.org/html/2606.08018#S6.T2)shows large differences across target dialects\. Oracle obtains the highest average EX of 48\.92%, followed by SQLite, MySQL, Hive, Spark, and StarRocks, while Teradata and Druid are the most difficult with 29\.03% and 29\.65%, respectively\. This pattern should not be interpreted simply as a ranking of intrinsic dialect complexity\. For instance, Oracle is not easy to translate from SQLite in our construction pipeline, but models still perform strongly on it, likely because Oracle SQL has abundant public documentation, examples, and training exposure\. Conversely, dialects such as Teradata and Druid may be less represented in model pretraining corpora and involve more system\-specific functions, type handling, temporal operations, or execution behavior\. Thus,UniQLreveals not only syntactic differences among SQL dialects, but also the uneven familiarity and robustness of current LLMs across database ecosystems\.

Overall, the results show thatUniQLis both challenging and diagnostic: it reveals that current models benefit from stronger general capability, but still lack robust dialect\-universal text\-to\-SQL generation\. The substantial variation across dialects further supports the need for aligned cross\-dialect benchmarks beyond SQLite\-centered evaluation\.

### 6\.3Stratified Performance Analysis

Beyond dialect\-level results, we further analyze model performance under two benchmark partitions: the BIRD question difficulty and the construction source introduced byUniQL\. These partitions help distinguish whether models fail mainly because the original text\-to\-SQL problem is complex, or because the target SQL realization belongs to a harder cross\-dialect construction path\.

#### Performance by Question Difficulty\.

Figure[3](https://arxiv.org/html/2606.08018#S6.F3)reports execution accuracy grouped by the original BIRD difficulty labels\. Across almost all models, accuracy decreases consistently from simple to moderate and challenging questions\. For example, Claude\-4\.5\-Sonnet drops from 60\.8% on simple questions to 46\.7% on moderate questions and 40\.3% on challenging questions, while Qwen3\-32B drops from 54\.8% to 39\.2% and 30\.2%\. This trend shows thatUniQLpreserves the semantic difficulty structure of the original text\-to\-SQL task: cross\-dialect generation is not only a matter of producing dialect\-specific syntax, but also requires solving increasingly complex query semantics\.

![Refer to caption](https://arxiv.org/html/2606.08018v1/x3.png)Figure 3:Stratified execution accuracy \(%\) of the 13 models by the original BIRD question difficulty\. Values are macro\-averaged over 16 SQL dialects on UniQL\.
#### Performance by Construction Source\.

Figure[4](https://arxiv.org/html/2606.08018#S6.F4)analyzes performance according to the stage from which the reference SQL annotation is obtained duringUniQLconstruction\. A clear trend emerges: examples directly resolved by the rule\-based tool are substantially easier, while those requiring LLM translation, self\-reflection, rule evolution, or human validation are generally more difficult\. For instance, Claude\-4\.5\-Sonnet achieves 59\.1% on the Tool subset, but only 44\.1%, 30\.7%, and 14\.6% on the LLM, Reflection, and Rule subsets, respectively\. Similar patterns appear for Qwen3\-32B and DeepSeek\-v4\-flash\. This indicates that the construction source provides an additional diagnostic signal: examples that require later construction stages often contain more dialect\-specific or long\-tail behaviors, which are also harder for generation models\.

The Human subset does not always have the lowest accuracy\. This is expected because human verification inUniQLis not a pure difficulty label\. Some examples enter the human stage because automatic verification cannot safely certify equivalence under database\-specific execution behavior, such as under\-specified ordering, duplicate\-sensitive outputs, or target\-engine differences, even when the underlying semantic intent is clear\. Therefore, the construction\-source split should be interpreted as a long\-tail and verification\-complexity indicator rather than a strict difficulty scale\.

![Refer to caption](https://arxiv.org/html/2606.08018v1/x4.png)Figure 4:Execution accuracy byUniQLconstruction source\. Values are macro\-averaged over target dialects\.

### 6\.4Cross\-Dialect Consistency

The main results report accuracy by aggregating questions within each dialect\. However,UniQLis aligned by construction: each question is realized as executable SQL across 16 dialects\. This allows us to evaluate a stricter property, which we call*cross\-dialect consistency*: for the same intent, how many dialect realizations can a model answer correctly? For each question, we count the number of dialects answered correctly, ranging from 0 to 16\. We also report SQLite→\\rightarrow15, which measures the percentage of SQLite\-correct questions that remain correct across other 15 dialects\.

Table[3](https://arxiv.org/html/2606.08018#S6.T3)shows that high average execution accuracy does not imply dialect\-universal correctness\. For example, Claude\-4\.5\-Sonnet answers 8\.74 dialect realizations correctly per question on average, but only 20\.14% of questions are answered correctly in all 16 dialects\. Similarly, Gemini\-2\.5\-Pro has a high mean correct count of 8\.34, yet its All\-16 consistency is only 9\.19%\. These results indicate that many questions are solved only partially across dialects: a model may correctly generate SQL for some database systems while failing to express the same intent in others\.

The SQLite→\\rightarrow15 metric further shows that SQLite correctness is not a reliable proxy for cross\-dialect robustness\. Even among SQLite\-correct questions, only 33\.66% for Claude\-4\.5\-Sonnet and 34\.29% for GPT\-5\-mini remain correct across all other dialects\. For Gemini\-2\.5\-Pro, this ratio is only 15\.38%, despite its strong average performance\. Thus, solving the SQLite realization of an intent does not guarantee that the same intent can be realized correctly in other SQL dialects\.

Table 3:Cross\-dialect consistency onUniQL\. Mean reports the average correct dialects per question\.ModelMeanAll\-16SQLite→\\rightarrow15Closed\-Source LLMsGPT\-3\.5\-Turbo5\.799\.1322\.19GPT\-5\-mini8\.0817\.9934\.29GPT\-5\.1\-codex8\.0714\.3426\.67Gemini\-2\.5\-Pro8\.349\.1915\.38Claude\-4\.5\-Sonnet8\.7420\.1433\.66Open\-Source LLMsQwen3\-1\.7B5\.285\.8016\.39Qwen3\-4B6\.778\.8018\.75Qwen3\-8B7\.1712\.0624\.57Qwen3\-32B7\.6414\.4126\.98Llama\-3\-8B\-Inst3\.532\.5410\.77Llama\-3\-70B\-Inst6\.3714\.0833\.33DeepSeek\-Coder\-16B5\.004\.6313\.22DeepSeek\-v4\-flash7\.7013\.8925\.98

Overall, the consistency results strengthen the main observation that single\-dialect evaluation is insufficient\. Average EX measures performance after aggregating over dialects, while All\-16 consistency asks whether a model can preserve correctness across all dialect realizations\. The large gap between these two views shows that current models are still far from serving as dialect\-universal natural language database interfaces\.

## 7Conclusion

We introducedUniQL, a human\-verified benchmark for evaluating cross\-dialect text\-to\-SQL generation\.UniQLaligns 1,534 natural language intents with executable SQL realizations across 16 SQL dialects, yielding 24,544 dialect\-specific annotations\. Through database migration, hybrid translation, execution\-guided verification, iterative rule summarization, and human validation,UniQLenables controlled evaluation of whether models can preserve the same query intent across different database systems\. Our experiments show that current LLMs remain far from dialect\-universal: even strong models solve only around half of the benchmark on average, exhibit substantial variation across dialects, and often fail to transfer SQLite success to other database systems\. These findings highlight the need for aligned cross\-dialect benchmarks and dialect\-aware text\-to\-SQL methods\.

## 8Limitations and Future Work

UniQLprioritizes controlled cross\-dialect alignment: the same natural language intents, aligned schemas and database contents are preserved across 16 SQL dialects\. This design enables direct comparison across dialects, but it also limits the use of target\-system\-specific data models\. Our database migration process largely preserves the original BIRD schemas, which are SQLite\-derived and mostly flat relational tables\. Therefore, the current benchmark does not systematically cover native features such as PostgreSQL or BigQuery JSON querying, Spark SQL and Hive complex types such as arrays, maps, and structs, or analytical\-system features in ClickHouse and Druid for large\-scale aggregation and time\-series analysis\.

Similarly, the SQL annotations inUniQLare designed to preserve the original query intent rather than to maximize dialect\-specific idioms\. A target SQL query may therefore use a portable formulation even when the target system provides a more native expression\. Future work can extendUniQLby redesigning target databases with native data models and by creating natural language questions that explicitly require dialect\-specific capabilities, such as JSON operators, nested data access, array functions, partition\-aware queries, or time\-granularity operations\. Such extensions would complement the current benchmark by evaluating not only cross\-dialect portability, but also dialect\-native text\-to\-SQL generation in real\-world heterogeneous database environments\.

Another limitation is that execution\-based verification, even with our stricter protocol, cannot fully guarantee semantic equivalence between translated SQL queries and the original SQL\. Two queries may return the same result on the current database instance while encoding different logical conditions, especially when the database contents do not expose the difference\. For example, different predicates, joins, or aggregation conditions may be indistinguishable on the observed data but diverge under other database instances\.UniQLmitigates this risk through conservative automatic acceptance, execution\-guided checking, and human verification for ambiguous or long\-tail cases\. Nevertheless, execution agreement should be understood as strong empirical evidence of equivalence on the benchmark databases rather than a complete formal guarantee of semantic equivalence\.

## References

- \[1\]The claude model family system cards\.\(Website\)External Links:[Link](https://www.anthropic.com/system-cards)Cited by:[§6\.1](https://arxiv.org/html/2606.08018#S6.SS1.SSS0.Px2.p1.1)\.
- G\. Comaniciet al\.\(2025\)Gemini 2\.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities\.arXiv preprint arXiv:2507\.06261\.Cited by:[§6\.1](https://arxiv.org/html/2606.08018#S6.SS1.SSS0.Px2.p1.1)\.
- D\. A\. Dahl, M\. Bates, M\. K\. Brown, W\. M\. Fisher, K\. Hunicke\-Smith, D\. S\. Pallett, C\. Pao, A\. Rudnicky, and E\. Shriberg \(1994\)Expanding the scope of the atis task: the atis\-3 corpus\.InHuman Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8\-11, 1994,Cited by:[§2](https://arxiv.org/html/2606.08018#S2.p1.1)\.
- DeepSeek\-AI \(2026\)DeepSeek\-v4: towards highly efficient million\-token context intelligence\.Technical reportDeepSeek\-AI\.External Links:[Link](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/DeepSeek_V4.pdf)Cited by:[§6\.1](https://arxiv.org/html/2606.08018#S6.SS1.SSS0.Px2.p1.1)\.
- A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Yang, A\. Fan,et al\.\(2024\)The llama 3 herd of models\.arXiv e\-prints\.Cited by:[§6\.1](https://arxiv.org/html/2606.08018#S6.SS1.SSS0.Px2.p1.1)\.
- C\. Finegan\-Dollak, J\. K\. Kummerfeld, L\. Zhang, K\. Ramanathan, S\. Sadasivam, R\. Zhang, and D\. Radev \(2018\)Improving text\-to\-SQL evaluation methodology\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics,pp\. 351–360\.External Links:[Document](https://dx.doi.org/10.18653/v1/P18-1033)Cited by:[§2](https://arxiv.org/html/2606.08018#S2.p1.1)\.
- \[7\]JOOQJOOQ\. \(tool\)\. available at https://www\.jooq\.org/\.\(Website\)External Links:[Link](https://www.jooq.org/)Cited by:[§2](https://arxiv.org/html/2606.08018#S2.p2.1)\.
- F\. Lei, J\. Chen, Y\. Ye, R\. Cao, D\. Shin, H\. Su, Z\. Suo, H\. Gao, W\. Hu, P\. Yin,et al\.\(2025\)Spider 2\.0: evaluating language models on real\-world enterprise text\-to\-sql workflows\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 28691–28735\.Cited by:[§1](https://arxiv.org/html/2606.08018#S1.p3.1),[§2](https://arxiv.org/html/2606.08018#S2.p1.1),[§4](https://arxiv.org/html/2606.08018#S4.SS0.SSS0.Px2.p1.1)\.
- J\. Li, B\. Hui, G\. Qu, J\. Yang, B\. Li, B\. Li, B\. Wang, B\. Qin, R\. Geng, N\. Huo,et al\.\(2023\)Can llm already serve as a database interface? a big bench for large\-scale database grounded text\-to\-sqls\.Advances in Neural Information Processing Systems36,pp\. 42330–42357\.Cited by:[Appendix E](https://arxiv.org/html/2606.08018#A5.p1.1),[Appendix F](https://arxiv.org/html/2606.08018#A6.p1.1),[§1](https://arxiv.org/html/2606.08018#S1.p3.1),[§2](https://arxiv.org/html/2606.08018#S2.p1.1),[§4](https://arxiv.org/html/2606.08018#S4.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2606.08018#S4.SS0.SSS0.Px2.p1.1)\.
- A\. L\. Ngom and T\. Kraska \(2024\)Mallet: sql dialect translation with llm rule generation\.InProceedings of the Seventh International Workshop on Exploiting Artificial Intelligence Techniques for Data Management,pp\. 1–5\.Cited by:[§2](https://arxiv.org/html/2606.08018#S2.p2.1)\.
- OpenAI \(2023\)GPT\-3\.5 Turbo\.Note:[https://platform\.openai\.com/docs/models/gpt\-3\-5\-turbo](https://platform.openai.com/docs/models/gpt-3-5-turbo)OpenAI API model documentationCited by:[§6\.1](https://arxiv.org/html/2606.08018#S6.SS1.SSS0.Px2.p1.1)\.
- OpenAI \(2025\)GPT\-5\.1\-Codex\-Max System Card\.Note:[https://cdn\.openai\.com/pdf/2a7d98b1\-57e5\-4147\-8d0e\-683894d782ae/5p1\_codex\_max\_card\_03\.pdf](https://cdn.openai.com/pdf/2a7d98b1-57e5-4147-8d0e-683894d782ae/5p1_codex_max_card_03.pdf)OpenAI system cardCited by:[§6\.1](https://arxiv.org/html/2606.08018#S6.SS1.SSS0.Px2.p1.1)\.
- M\. Pourreza, H\. Li, R\. Sun, Y\. Chung, S\. Talaei, G\. T\. Kakkar, Y\. Gan, A\. Saberi, F\. Ozcan, and S\. Arik \(2025\)Chase\-sql: multi\-path reasoning and preference optimized candidate selection in text\-to\-sql\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 60385–60415\.Cited by:[§1](https://arxiv.org/html/2606.08018#S1.p2.1)\.
- M\. Pourreza and D\. Rafiei \(2023\)Din\-sql: decomposed in\-context learning of text\-to\-sql with self\-correction\.Advances in neural information processing systems36,pp\. 36339–36348\.Cited by:[§1](https://arxiv.org/html/2606.08018#S1.p2.1)\.
- Y\. Qin, C\. Chen, Z\. Fu, Z\. Chen, D\. Peng, P\. Hu, and J\. Ye \(2025\)ROUTE: robust multitask tuning and collaboration for text\-to\-sql\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 12425–12448\.Cited by:[§1](https://arxiv.org/html/2606.08018#S1.p2.1)\.
- A\. Singh, A\. Fry, A\. Perelman, A\. Tart, A\. Ganesh, A\. El\-Kishky, A\. McLaughlin, A\. Low, A\. Ostrow, A\. Ananthram,et al\.\(2026\)OpenAI GPT\-5 System Card\.arXiv preprint arXiv:2601\.03267\.Cited by:[§6\.1](https://arxiv.org/html/2606.08018#S6.SS1.SSS0.Px2.p1.1)\.
- \[17\]SQLGlotSQLGlot\. \(tool\)\. available at https://sqlglot\.com/\.\(Website\)External Links:[Link](https://sqlglot.com/)Cited by:[§2](https://arxiv.org/html/2606.08018#S2.p2.1),[§3\.2](https://arxiv.org/html/2606.08018#S3.SS2.p1.9)\.
- \[18\]SQLinesSQLines\. \(tool\)\. available at https://www\.sqlines\.com/\.\(Website\)External Links:[Link](https://www.sqlines.com/)Cited by:[§2](https://arxiv.org/html/2606.08018#S2.p2.1)\.
- L\. R\. Tang and R\. J\. Mooney \(2000\)Automated construction of database interfaces: integrating statistical and relational learning for semantic parsing\.InProceedings of the Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora,pp\. 133–141\.Cited by:[§2](https://arxiv.org/html/2606.08018#S2.p1.1)\.
- B\. Wang, C\. Ren, J\. Yang, X\. Liang, J\. Bai, L\. Chai, Z\. Yan, Q\. Zhang, D\. Yin, X\. Sun,et al\.\(2025\)Mac\-sql: a multi\-agent collaborative framework for text\-to\-sql\.InProceedings of the 31st International Conference on Computational Linguistics,pp\. 540–557\.Cited by:[§1](https://arxiv.org/html/2606.08018#S1.p2.1)\.
- X\. Xie, Y\. Zhang, W\. Dou, Y\. Gao, Z\. Cui, J\. Song, R\. Yang, and J\. Wei \(2026\)RISE: rule\-driven sql dialect translation via query reduction\.arXiv preprint arXiv:2601\.05579\.Cited by:[§2](https://arxiv.org/html/2606.08018#S2.p2.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§6\.1](https://arxiv.org/html/2606.08018#S6.SS1.SSS0.Px2.p1.1)\.
- T\. Yu, R\. Zhang, H\. Y\. Er, S\. Li, E\. Xue, B\. Pang, X\. V\. Lin, Y\. C\. Tan, T\. Shi, Z\. Li, Y\. Jiang, M\. Yasunaga, S\. Shim, T\. Chen, A\. Fabbri, Z\. Li, L\. Chen, Y\. Zhang, S\. Dixit, V\. Zhang, C\. Xiong, R\. Socher, W\. S\. Lasecki, and D\. Radev \(2019a\)CoSQL: A Conversational Text\-to\-SQL Challenge Towards Cross\-Domain Natural Language Interfaces to Databases\.Cited by:[§2](https://arxiv.org/html/2606.08018#S2.p1.1)\.
- T\. Yu, R\. Zhang, K\. Yang, M\. Yasunaga, D\. Wang, Z\. Li, J\. Ma, I\. Li, Q\. Yao, S\. Roman,et al\.\(2018\)Spider: a large\-scale human\-labeled dataset for complex and cross\-domain semantic parsing and text\-to\-sql task\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,Cited by:[§1](https://arxiv.org/html/2606.08018#S1.p3.1),[§2](https://arxiv.org/html/2606.08018#S2.p1.1),[§4](https://arxiv.org/html/2606.08018#S4.SS0.SSS0.Px2.p1.1)\.
- T\. Yu, R\. Zhang, M\. Yasunaga, Y\. C\. Tan, X\. V\. Lin, S\. Li, H\. Er, I\. Li, B\. Pang, T\. Chen, E\. Ji, S\. Dixit, D\. Proctor, S\. Shim, J\. Kraft, V\. Zhang, C\. Xiong, R\. Socher, and D\. Radev \(2019b\)SParC: Cross\-Domain Semantic Parsing in Context\.Cited by:[§2](https://arxiv.org/html/2606.08018#S2.p1.1)\.
- J\. M\. Zelle and R\. J\. Mooney \(1996\)Learning to parse database queries using inductive logic programming\.InProceedings of the Thirteenth National Conference on Artificial Intelligence,pp\. 1050–1055\.Cited by:[§2](https://arxiv.org/html/2606.08018#S2.p1.1)\.
- V\. Zhong, C\. Xiong, and R\. Socher \(2017\)Seq2sql: generating structured queries from natural language using reinforcement learning\.arXiv preprint arXiv:1709\.00103\.Cited by:[§2](https://arxiv.org/html/2606.08018#S2.p1.1),[§4](https://arxiv.org/html/2606.08018#S4.SS0.SSS0.Px2.p1.1)\.
- W\. Zhou, Y\. Gao, X\. Zhou, and G\. Li \(2025a\)Cracking sql barriers: an llm\-based dialect translation system\.Proceedings of the ACM on Management of Data3\(3\),pp\. 1–26\.Cited by:[§2](https://arxiv.org/html/2606.08018#S2.p2.1)\.
- W\. Zhou, Y\. Gao, X\. Zhou, and G\. Li \(2025b\)CrackSQL: A Hybrid SQL Dialect Translation System Powered by Large Language Models\.Cited by:[§2](https://arxiv.org/html/2606.08018#S2.p2.1)\.
- W\. Zhou, G\. Li, H\. Wang, Y\. Han, X\. Wu, F\. Wu, and X\. Zhou \(2026\)Parrot: a benchmark for evaluating llms in cross\-system sql translation\.Advances in Neural Information Processing Systems38\.Cited by:[§2](https://arxiv.org/html/2606.08018#S2.p2.1)\.
- Q\. Zhu, D\. Guo, Z\. Shao, D\. Yang, P\. Wang, R\. Xu, Y\. Wu, Y\. Li, H\. Gao, S\. Ma, W\. Zeng, X\. Bi, Z\. Gu, H\. Xu, D\. Dai, K\. Dong, L\. Zhang, Y\. Piao, Z\. Gou, Z\. Xie, Z\. Hao, B\. Wang, J\. Song, D\. Chen, X\. Xie, K\. Guan, Y\. You, A\. Liu, Q\. Du, W\. Gao, X\. Lu, Q\. Chen, Y\. Wang, C\. Deng, J\. Li, C\. Zhao, C\. Ruan, F\. Luo, and W\. Liang \(2024\)DeepSeek\-Coder\-V2: Breaking the Barrier of Closed\-Source Models in Code Intelligence\.arXiv preprint arXiv:2406\.11931\.Cited by:[§6\.1](https://arxiv.org/html/2606.08018#S6.SS1.SSS0.Px2.p1.1)\.

## Appendix ADetailed Execution Accuracy Protocol

We use execution accuracy \(EX\) as the main evaluation metric inUniQL\. The same execution\-based protocol is used in two stages: automatic verification during dataset construction and model evaluation during experiments\. During construction, EX is used as a conservative acceptance criterion for automatically translated SQL queries\. This criterion may reject some translations that are semantically reasonable but differ in execution results due to dialect\-dependent database behavior\. This is intentional: automatic acceptance favors precision over recall, and uncertain cases are routed to human verification\. During model evaluation, the same protocol is appropriate because the predicted SQL and the gold SQL are executed on the same target database system\. In this setting, differences in ordering or duplicate multiplicity should be preserved rather than ignored\.

Given a predicted SQL queryq^\\hat\{q\}, the gold SQL queryq∗q^\{\*\}, and the databaseBB, we first execute both queries on the same database:

R^=E​\(q^,B\),R∗=E​\(q∗,B\),\\hat\{R\}=E\(\\hat\{q\},B\),\\qquad R^\{\*\}=E\(q^\{\*\},B\),\(7\)whereE​\(q,B\)E\(q,B\)denotes the execution result of queryqqonBB\. Ifq^\\hat\{q\}fails to execute, the prediction is marked as incorrect\. Otherwise, we compareR^\\hat\{R\}andR∗R^\{\*\}according to the query semantics\.

Unlike BIRD\-style EX implementations that convert query outputs into unordered sets, our metric preserves ordering and duplicate multiplicities when needed\. For queries with explicit ordering semantics, such as those containingORDER BY, we compare the outputs as ordered lists:

EX​\(q^,q∗\)=𝕀​\[R^=R∗\]\.\\mathrm\{EX\}\(\\hat\{q\},q^\{\*\}\)=\\mathbb\{I\}\[\\hat\{R\}=R^\{\*\}\]\.\(8\)This prevents a prediction with the correct rows but wrong order from being incorrectly accepted\.

For unordered queries, result order is ignored, but duplicate multiplicity is still preserved\. Instead of converting outputs into sets, we use multiset\-style matching: each row inR^\\hat\{R\}must be matched with and removed fromR∗R^\{\*\}, and the prediction is correct only if all rows can be matched exactly and no unmatched rows remain\. Formally, for unordered outputs we require:

MultiSet​\(R^\)=MultiSet​\(R∗\)\.\\mathrm\{MultiSet\}\(\\hat\{R\}\)=\\mathrm\{MultiSet\}\(R^\{\*\}\)\.\(9\)This avoids treating outputs with different numbers of duplicate rows as equivalent\.

Overall, this protocol provides stricter and more faithful executable evaluation for cross\-dialect text\-to\-SQL than set\-based result comparison\.

## Appendix BPrompt Templates

We provide the prompt templates used inUniQLfor both model inference and benchmark construction\. The inference prompt is used to evaluate text\-to\-SQL models across different SQL dialects\. The translator prompt is used to convert SQL queries from a source dialect to a target dialect, while the rule summarizer prompt is used to derive reusable dialect\-specific translation rules from failed translation cases\. For each instance, placeholders such as\{dialect\},\{db\_details\},\{question\},\{evidence\},\{schema\}, and\{error\_logs\}are instantiated with the corresponding database, query, and execution feedback information\.

Prompt Template for Text\-to\-SQL InferenceTask Overview:\{internallinenumbers\*\}You are a data science expert\. Below, you are provided with a database schema and a natural language question\. Your task is to understand the schema and generate a valid SQL query to answer the question\.Database system:\{dialect\}Database Schema:\{db\_details\}\{internallinenumbers\*\}This schema describes the database’s structure, including tables, columns, primary keys, foreign keys, and any relevant relationships or constraints\.Question:\{question\}Evidence:\{evidence\}Instructions:\{internallinenumbers\*\}\- Make sure you only output the information that is asked in the question\. If the question asks for a specific column, make sure to only include that column in the SELECT clause, nothing more\.\{internallinenumbers\*\}\- The generated query should return all of the information asked in the question without any missing or extra information\.\{internallinenumbers\*\}\- Before generating the final SQL query, please think through the steps of how to write the query\.Output Format:\{internallinenumbers\*\}In your answer, please enclose the generated SQL query in a code block:\`\`\`sql\-\- Your SQL query\`\`\`\{internallinenumbers\*\}Take a deep breath and think step by step to find the correct SQL query\.

Prompt Template for the LLM Translator\{internallinenumbers\*\}You are an expert SQL translator\. Your task is to convert a \{source\_dialect\} query into an executable \{target\_dialect\} query\.\{internallinenumbers\*\}Target \{target\_dialect\} Schema \(Reference for Exact Table/Column Names and Types\):\{schema\}Source \{source\_dialect\} SQL:\{source\_sql\}General Translation Rules:\{internallinenumbers\*\}Schema Fidelity: The provided Schema is the ABSOLUTE TRUTH\. Do not invent columns or tables\.\{internallinenumbers\*\}Logic Preservation: Preserve the logic of the source SQL \(filters, joins, ordering\) as much as possible, adapting syntax to the target dialect\.\{internallinenumbers\*\}Data Types: Be careful with Date/Time and String/Number comparisons\.Specific Dialect Rules \(Iteratively Refined\):\{specific\_rules\}Output Format:\{internallinenumbers\*\}In your answer, please enclose the translated SQL query in a code block:\`\`\`sql\-\- Your SQL query\`\`\`

Prompt Template for the Rule SummarizerYou are an expert Database Administrator and SQL Architect\.\{internallinenumbers\*\}We are translating SQL queries from \{source\_dialect\} to \{target\_dialect\}\.\{internallinenumbers\*\}We have a list of failed translations where the translated SQL result did not match the ground truth\.\{internallinenumbers\*\}Your task is to analyze these error logs and summarize specific, actionable translation rules to fix these issues\.\{internallinenumbers\*\}Target \{target\_dialect\} Schema \(Reference for Exact Table/Column Names and Types\):\{schema\}Error Logs:\{error\_logs\}Instructions:\{internallinenumbers\*\}Identify common patterns of failure \(e\.g\., date formatting, integer division, quoting rules, NULL handling, ordering differences\)\.Formulate concise rules to address these failures\.\{internallinenumbers\*\}The rules should be instructions for an AI translator \(e\.g\., "When translating X, always do Y"\)\.\{internallinenumbers\*\}If a failure seems to be due to ambiguous logic rather than SQL equivalence \(e\.g\., unpredictable sort order\), note it but prioritize strict syntax/semantics rules\.\{internallinenumbers\*\}Refer to the specific Schema info provided in each error log to understand column types \(e\.g\., if a column is VARCHAR or INT\)\.Output Format:\{internallinenumbers\*\}Return ONLY the list of rules as a bulleted list\. Do not include introductory text\.

## Appendix CError Analysis

Table 4:Post\-hoc error distribution of Claude\-4\.5\-Sonnet failures onUniQL\. Shares are computed within failed predictions for each dialect and grouped into coarse error categories based on saved SQL predictions, execution errors, execution results, and manual calibration\.DialectFailSyntax/func\.Schema/ref\.Value/filterSQL logicOtherClickHouse6624\.5327\.1946\.9821\.150\.15Doris72512\.8324\.1442\.6220\.280\.14Drill73213\.8022\.1316\.2647\.810\.00Druid92256\.4012\.1522\.458\.680\.33DuckDB6867\.8730\.3241\.6919\.970\.15Hive6200\.8129\.0347\.1023\.060\.00MySQL6364\.8728\.7745\.1320\.910\.31Oracle5569\.8943\.8833\.4512\.050\.72PostgreSQL6912\.0337\.7738\.9321\.130\.14Presto6432\.6428\.1548\.9919\.910\.31Spark6731\.7829\.2746\.8122\.140\.00SQLite6160\.6528\.5745\.6224\.680\.49StarRocks6746\.9725\.8245\.7021\.510\.00T\-SQL6718\.4925\.9344\.7120\.720\.15Teradata95557\.3810\.3721\.3610\.890\.00Trino6742\.5230\.7147\.9218\.840\.00To further understand the remaining failures of strong models onUniQL, we conduct a post\-hoc error analysis for Claude\-4\.5\-Sonnet, the best\-performing model in our main experiments\. We assign each failed prediction to one of five coarse categories:*syntax/function*,*schema/reference*,*value/filter*,*SQL logic*, and*other*\. The analysis is based on saved model predictions, execution errors, execution results, and manual calibration\.

#### Error categories\.

*Syntax/function*errors refer to failures caused by invalid dialect\-specific syntax, unsupported functions or operators, quoting and casting problems, or database\-engine\-specific execution errors\.*Schema/reference*errors include wrong table or column references, incorrect joins, aliasing mistakes, or selecting attributes from an inappropriate relation\.*Value/filter*errors capture incorrect literals, missing or spurious predicates, NULL\-sensitive conditions, boundary cases, cardinality\-changing filters, and tie\-breaking differences\.*SQL logic*errors denote executable or nearly executable queries whose structure is semantically incomplete or incorrect, including missing aggregation, ranking, grouping, nesting, ordering scope, or required answer columns\. The*other*category covers rare failures that do not fit the above groups\.

#### Dialect\-level error distribution\.

Table[4](https://arxiv.org/html/2606.08018#A3.T4)shows that Claude\-4\.5\-Sonnet’s failure modes are highly dialect\-dependent\. For most dialects, value/filter and schema/reference errors dominate, suggesting that the model often produces plausible SQL skeletons but fails to precisely align predicates, values, joins, or output attributes with the intended query\. For example, Presto, Trino, Hive, ClickHouse, Spark, SQLite, and MySQL all have value/filter errors as the largest category\. This indicates that many failures are not purely syntactic; instead, they arise from subtle mismatches in literal values, filters, NULL handling, boundary conditions, tie\-breaking, or result cardinality\.

In contrast, Druid and Teradata exhibit much larger syntax/function error shares, reaching 56\.40% and 57\.38% of failed predictions, respectively\. This suggests that failures on these systems are more strongly tied to dialect\-specific functions, type handling, planner constraints, and execution behavior\. Drill shows another distinct pattern, with SQL logic errors forming the largest portion of failures\. This indicates that its failures more often arise from semantically incomplete or structurally incorrect query formulations rather than surface\-level syntax alone\. These results reinforce the motivation ofUniQL: cross\-dialect text\-to\-SQL robustness cannot be inferred from SQLite\-only evaluation or a single aggregate accuracy score, because different database systems expose qualitatively different failure modes\.

#### Representative examples\.

Table[5](https://arxiv.org/html/2606.08018#A3.T5)provides representative examples for the four major error categories\. These examples illustrate how the same broad failure taxonomy captures different kinds of model behavior: dialect\-specific execution failures, incorrect schema or relation grounding, subtle value and filter mismatches, and structurally incomplete SQL generation\. Long SQL strings are shortened for readability while preserving the key failure pattern\.

Table 5:Representative examples for the post\-hoc error taxonomy\. SQL strings are shortened for readability while preserving the key failure pattern\.CategoryQuestion / EvidencePredicted SQLGold SQL / Key issueSyntax/function
Druid, id=1Please list the lowest three eligible free rates for students aged 5–17 in continuation schools\.
Evidence: eligible free rate =Free Meal Count \(Ages 5\-17\)/Enrollment \(Ages 5\-17\)\.SELECT "free meal count \(ages 5\-17\)" / "enrollment \(ages 5\-17\)" AS eligible\_free\_rate FROM bird\_california\_schools\_frpm WHERE "educational option type" = ’Continuation’ AND "enrollment \(ages 5\-17\)" \> 0 ORDER BY eligible\_free\_rate ASC LIMIT 3The query triggers a Druid planner error:HTTP 400, INVALID\_INPUT\. Query could not be planned…The gold query uses explicit casting and aCASEexpression to handle division and NULL\-sensitive values\. The prediction also contains a secondary literal mismatch, but the main label is assigned based on the dialect\-specific execution failure\.Schema/reference
PostgreSQL, id=25Name schools in Riverside for which the average of average math score for SAT is greater than 400, and show the funding type of these schools\.
Evidence: average of average math = sum\(average math scores\) / count\(schools\)\.SELECT s\.sname, sch\.fundingtype FROM satscores s JOIN schools sch ON s\.cds = sch\.cdscode WHERE s\.cname = ’Riverside’ AND s\.avgscrmath \> 400SELECT T1\.sname, T2\."charter funding type" FROM satscores AS T1 INNER JOIN frpm AS T2 ON T1\.cds = T2\.cdscode WHERE T2\."district name" LIKE ’Riverside%’ GROUP BY T1\.sname, T2\."charter funding type" HAVING CAST\(SUM\(T1\.avgscrmath\) AS real\) / COUNT\(T1\.cds\) \> 400
The prediction selects the funding type from the wrong relation and misses the required reference tofrpm\. It also has a secondary aggregation\-condition mismatch\.Value/filter
Oracle, id=30Which cities have the top 5 lowest enrollment number for students in grades 1 through 12?
Evidence: K–12 refers to students in grades 1 through 12\.SELECT s\.CITY FROM SCHOOLS s JOIN FRPM f ON s\.CDSCODE = f\.CDSCODE WHERE f\."ENROLLMENT \(K\-12\)" IS NOT NULL AND s\.CITY IS NOT NULL GROUP BY s\.CITY ORDER BY SUM\(f\."ENROLLMENT \(K\-12\)"\) ASC FETCH FIRST 5 ROWS ONLYSELECT T2\.CITY FROM FRPM T1 INNER JOIN SCHOOLS T2 ON T1\.CDSCODE = T2\.CDSCODE GROUP BY T2\.CITY ORDER BY SUM\(NVL\(T1\."ENROLLMENT \(K\-12\)",0\)\) ASC, T2\.CITY ASC FETCH FIRST 5 ROWS ONLY
The prediction follows the correct high\-level aggregation pattern, but changes NULL\-sensitive filtering by removing rows with NULL enrollment or city values\. The gold query instead treats NULL enrollment as 0 and uses city\-level tie\-breaking, which can change the top\-5 returned cities\.SQL logic
SQLite, id=17Rank schools by their average score in Writing where the score is greater than 499, showing their charter numbers\.
Evidence: valid charter number means the number is not null\.SELECT T2\.CharterNum FROM satscores AS T1 INNER JOIN schools AS T2 ON T1\.cds = T2\.CDSCode WHERE T2\.CharterNum IS NOT NULL AND T1\.AvgScrWrite \> 499 ORDER BY T1\.AvgScrWrite DESCSELECT CharterNum, AvgScrWrite, RANK\(\) OVER \(ORDER BY AvgScrWrite DESC\) AS WritingScoreRank FROM schools AS T1 INNER JOIN satscores AS T2 ON T1\.CDSCode = T2\.cds WHERE T2\.AvgScrWrite \> 499 AND CharterNum IS NOT NULL
The prediction returns only the charter number and misses both the writing score and the ranking computation, leading to an incomplete answer shape\.

## Appendix DDetailed Stratified Results by Dialect

To complement the aggregate stratified analysis in Section[6\.3](https://arxiv.org/html/2606.08018#S6.SS3), we report the full dialect\-level results under the two partitions used in the main text\. The first partition follows the original BIRD difficulty labels, separating questions into simple, moderate, and challenging subsets\. The second follows theUniQLconstruction source, grouping target SQL annotations by whether they are obtained from the SQLGlot tool stage, LLM translation, self\-reflection, rule refinement, or human verification\. Table[6](https://arxiv.org/html/2606.08018#A4.T6)summarizes the subset sizes for these two stratification schemes, while the following tables provide execution accuracy for every evaluated model on each SQL dialect\. These detailed results expose whether an aggregate performance trend is stable across dialects or driven by particular database systems and construction paths\.

Table 6:Construction statistics for the dialects inUniQLunder the two stratification schemes used in the analysis\. For each dialect, both the construction\-source block and the BIRD\-difficulty block partition the same aligned split of 1,534 examples\.UniQLconstruction sourceBIRD difficultyDialectToolLLMReflectionRule RefinementHumanSimpleModerateChallengingClickHouse1300142631217925464145Doris142439481013925464145Drill13440214840925464145Druid1145304174424925464145DuckDB136952311072925464145Hive959195151100129925464145MySQL14722026610925464145Oracle11661571642225925464145PostgreSQL1282151572420925464145Presto1299184171420925464145Spark134468761036925464145SQLite–––––925464145StarRocks1319122631218925464145T\-SQL1368106351114925464145Teradata8283882183961925464145Trino13778351617925464145Table 7:Execution accuracy \(%\) on ClickHouse under the original BIRD difficulty andUniQLconstruction\-source stratifications\.BIRD difficultyUniQLconstruction sourceModelAllSimpleModerateChallengingToolLLMReflectionRule RefinementHumanClosed\-weight LLMsGPT\-3\.5\-Turbo35\.1443\.7822\.6320\.0037\.6928\.8711\.118\.330\.00GPT\-5\-mini51\.5658\.3841\.8139\.3153\.7752\.1123\.818\.3311\.76GPT\-5\.1\-codex52\.6159\.1443\.7539\.3155\.5451\.4112\.708\.3317\.65Gemini\-2\.5\-Pro53\.9861\.3043\.7540\.0057\.0052\.1117\.468\.335\.88Claude\-4\.5\-Sonnet56\.8463\.7848\.0640\.6959\.8555\.6319\.058\.3311\.76Open\-weight LLMsQwen3\-1\.7B33\.9042\.1623\.0615\.8636\.3828\.877\.940\.005\.88Qwen3\-4B43\.0251\.7831\.9022\.7646\.6235\.214\.760\.005\.88Qwen3\-8B46\.5455\.2436\.4223\.4550\.2337\.327\.940\.0017\.65Qwen3\-32B49\.7456\.5441\.3833\.1052\.7745\.0715\.870\.0017\.65Llama\-3\-8B\-Inst20\.8025\.8414\.229\.6622\.9213\.383\.170\.000\.00Llama\-3\-70B\-Inst40\.1648\.7630\.3916\.5543\.3132\.399\.520\.005\.88DeepSeek\-Coder\-16B32\.4639\.5721\.7721\.3835\.5419\.729\.528\.335\.88DeepSeek\-v4\-flash48\.0455\.7838\.5828\.9751\.6940\.1411\.110\.005\.88Table 8:Execution accuracy \(%\) on Doris under the original BIRD difficulty andUniQLconstruction\-source stratifications\.BIRD difficultyUniQLconstruction sourceModelAllSimpleModerateChallengingToolLLMReflectionRule RefinementHumanClosed\-weight LLMsGPT\-3\.5\-Turbo38\.3346\.5925\.6526\.2140\.5223\.082\.080\.007\.69GPT\-5\-mini49\.8056\.6540\.5235\.8651\.6241\.0318\.750\.0030\.77GPT\-5\.1\-codex49\.9356\.1141\.5937\.2451\.6948\.7216\.670\.0023\.08Gemini\-2\.5\-Pro51\.7657\.8444\.4036\.5553\.9346\.1510\.420\.0023\.08Claude\-4\.5\-Sonnet52\.7458\.5945\.6937\.9354\.4258\.9714\.5810\.0023\.08Open\-weight LLMsQwen3\-1\.7B35\.0743\.8923\.7115\.1736\.7330\.772\.080\.0015\.38Qwen3\-4B44\.5251\.6835\.5627\.5946\.4946\.152\.080\.0015\.38Qwen3\-8B46\.1552\.8638\.3628\.2848\.1041\.0310\.420\.0015\.38Qwen3\-32B47\.2053\.5139\.6631\.0348\.8856\.418\.330\.0015\.38Llama\-3\-8B\-Inst23\.9930\.0517\.037\.5925\.4212\.822\.080\.000\.00Llama\-3\-70B\-Inst40\.0947\.8930\.6020\.6941\.9233\.336\.250\.0015\.38DeepSeek\-Coder\-16B31\.1038\.9220\.6914\.4832\.8717\.952\.080\.007\.69DeepSeek\-v4\-flash46\.9453\.0840\.0929\.6648\.6751\.288\.330\.0023\.08Table 9:Execution accuracy \(%\) on Drill under the original BIRD difficulty andUniQLconstruction\-source stratifications\.BIRD difficultyUniQLconstruction sourceModelAllSimpleModerateChallengingToolLLMReflectionRule RefinementHumanClosed\-weight LLMsGPT\-3\.5\-Turbo29\.2036\.8617\.8916\.5532\.22–0\.008\.785\.00GPT\-5\-mini48\.8356\.4339\.0131\.7251\.26–50\.0036\.4912\.50GPT\-5\.1\-codex48\.1154\.3840\.9531\.0351\.19–50\.0031\.765\.00Gemini\-2\.5\-Pro50\.8557\.5143\.1033\.1053\.94–0\.0033\.7812\.50Claude\-4\.5\-Sonnet52\.2858\.5945\.9132\.4155\.58–100\.0033\.1110\.00Open\-weight LLMsQwen3\-1\.7B27\.5735\.0317\.2413\.1029\.84–0\.0013\.515\.00Qwen3\-4B41\.7249\.8430\.8224\.8344\.79–0\.0022\.3012\.50Qwen3\-8B44\.1351\.7834\.2726\.9046\.88–0\.0028\.3812\.50Qwen3\-32B46\.5453\.3039\.0127\.5949\.55–50\.0029\.0510\.00Llama\-3\-8B\-Inst18\.4524\.0011\.216\.2120\.16–50\.006\.762\.50Llama\-3\-70B\-Inst37\.7445\.6228\.2317\.9340\.85–0\.0018\.925\.00DeepSeek\-Coder\-16B32\.0739\.6822\.4114\.4834\.38–50\.0017\.577\.50DeepSeek\-v4\-flash46\.2853\.8436\.2130\.3449\.26–0\.0029\.0512\.50Table 10:Execution accuracy \(%\) on Druid under the original BIRD difficulty andUniQLconstruction\-source stratifications\.BIRD difficultyUniQLconstruction sourceModelAllSimpleModerateChallengingToolLLMReflectionRule RefinementHumanClosed\-weight LLMsGPT\-3\.5\-Turbo22\.7530\.8112\.284\.8328\.037\.245\.889\.094\.17GPT\-5\-mini37\.7445\.3027\.8021\.3843\.1421\.7117\.6534\.094\.17GPT\-5\.1\-codex38\.2045\.3030\.3917\.9342\.9724\.3417\.6527\.2720\.83Gemini\-2\.5\-Pro37\.4243\.7829\.9620\.6942\.1822\.3717\.6531\.8225\.00Claude\-4\.5\-Sonnet39\.9047\.1430\.6023\.4545\.0724\.3411\.7631\.8225\.00Open\-weight LLMsQwen3\-1\.7B18\.7125\.958\.624\.8322\.976\.580\.004\.558\.33Qwen3\-4B26\.5334\.0517\.467\.5931\.629\.875\.8822\.7316\.67Qwen3\-8B31\.2939\.2421\.1213\.1036\.5913\.8211\.7627\.2720\.83Qwen3\-32B33\.8341\.0824\.7816\.5539\.9114\.1417\.6531\.828\.33Llama\-3\-8B\-Inst16\.5623\.357\.113\.4520\.175\.920\.006\.828\.33Llama\-3\-70B\-Inst26\.9234\.9217\.894\.8333\.107\.895\.8811\.3616\.67DeepSeek\-Coder\-16B21\.3828\.9710\.996\.2126\.037\.245\.8811\.368\.33DeepSeek\-v4\-flash34\.2242\.0525\.6511\.7240\.5215\.130\.0022\.7320\.83Table 11:Execution accuracy \(%\) on DuckDB under the original BIRD difficulty andUniQLconstruction\-source stratifications\.BIRD difficultyUniQLconstruction sourceModelAllSimpleModerateChallengingToolLLMReflectionRule RefinementHumanClosed\-weight LLMsGPT\-3\.5\-Turbo37\.0946\.2722\.4125\.5239\.3015\.386\.4510\.0027\.78GPT\-5\-mini52\.7458\.1644\.8343\.4554\.1340\.3829\.0310\.0051\.39GPT\-5\.1\-codex52\.9359\.5744\.4037\.9354\.8638\.4629\.030\.0044\.44Gemini\-2\.5\-Pro56\.1961\.7348\.7144\.8358\.8036\.5416\.1310\.0044\.44Claude\-4\.5\-Sonnet55\.2861\.6247\.8438\.6257\.0546\.1525\.8110\.0047\.22Open\-weight LLMsQwen3\-1\.7B35\.2742\.8125\.4318\.6237\.9113\.463\.2310\.0018\.06Qwen3\-4B44\.2651\.7835\.1325\.5247\.3323\.086\.4510\.0022\.22Qwen3\-8B47\.0055\.1437\.7224\.8349\.9623\.0812\.9010\.0027\.78Qwen3\-32B49\.9356\.3242\.2433\.7952\.3740\.3812\.9010\.0031\.94Llama\-3\-8B\-Inst22\.2927\.5715\.958\.9724\.115\.770\.000\.0012\.50Llama\-3\-70B\-Inst41\.7249\.8432\.9717\.9344\.5621\.153\.2310\.0023\.61DeepSeek\-Coder\-16B32\.7240\.5422\.4115\.8634\.7011\.546\.4510\.0025\.00DeepSeek\-v4\-flash49\.0255\.6841\.3831\.0351\.3534\.629\.6810\.0037\.50Table 12:Execution accuracy \(%\) on Hive under the original BIRD difficulty andUniQLconstruction\-source stratifications\.BIRD difficultyUniQLconstruction sourceModelAllSimpleModerateChallengingToolLLMReflectionRule RefinementHumanClosed\-weight LLMsGPT\-3\.5\-Turbo36\.1845\.4122\.4121\.3844\.9430\.2610\.603\.0035\.66GPT\-5\-mini54\.0459\.8945\.0445\.5257\.8748\.7250\.3330\.0056\.59GPT\-5\.1\-codex55\.4860\.6547\.8446\.9059\.0250\.2651\.6635\.0057\.36Gemini\-2\.5\-Pro59\.3265\.3051\.0847\.5961\.8455\.9054\.3047\.0061\.24Claude\-4\.5\-Sonnet59\.5865\.5151\.7246\.9063\.8254\.8754\.3037\.0058\.91Open\-weight LLMsQwen3\-1\.7B36\.7045\.1924\.5721\.3843\.0724\.6223\.1819\.0037\.21Qwen3\-4B46\.7455\.4634\.7029\.6652\.2438\.4635\.1025\.0048\.84Qwen3\-8B44\.8552\.8635\.3424\.1453\.3938\.9723\.847\.0044\.19Qwen3\-32B51\.9659\.0342\.8935\.8656\.1046\.6743\.0532\.0055\.04Llama\-3\-8B\-Inst22\.2328\.9713\.367\.5925\.6512\.3115\.2317\.0024\.03Llama\-3\-70B\-Inst42\.6351\.4631\.4722\.0747\.6535\.3828\.4825\.0046\.51DeepSeek\-Coder\-16B34\.8842\.9222\.8422\.0739\.5224\.6227\.8117\.0037\.98DeepSeek\-v4\-flash52\.8059\.0344\.4040\.0057\.4643\.0842\.3835\.0058\.91Table 13:Execution accuracy \(%\) on MySQL under the original BIRD difficulty andUniQLconstruction\-source stratifications\.BIRD difficultyUniQLconstruction sourceModelAllSimpleModerateChallengingToolLLMReflectionRule RefinementHumanClosed\-weight LLMsGPT\-3\.5\-Turbo40\.8150\.4927\.5921\.3842\.0520\.007\.690\.0010\.00GPT\-5\-mini54\.3060\.4345\.4743\.4555\.3725\.0030\.770\.0050\.00GPT\-5\.1\-codex54\.8961\.9545\.2640\.6956\.0530\.0026\.920\.0040\.00Gemini\-2\.5\-Pro57\.8264\.0050\.2242\.7659\.3830\.0019\.230\.0020\.00Claude\-4\.5\-Sonnet58\.5465\.7349\.7840\.6959\.8535\.0023\.080\.0040\.00Open\-weight LLMsQwen3\-1\.7B35\.8545\.7323\.2813\.1037\.095\.003\.850\.0020\.00Qwen3\-4B46\.6156\.0033\.8427\.5947\.8915\.0015\.380\.0030\.00Qwen3\-8B50\.2058\.2740\.5229\.6651\.4320\.0023\.080\.0030\.00Qwen3\-32B52\.0259\.4642\.6734\.4853\.4625\.0015\.380\.0020\.00Llama\-3\-8B\-Inst22\.0328\.8613\.156\.9022\.830\.003\.850\.0010\.00Llama\-3\-70B\-Inst43\.7453\.0832\.5420\.0045\.1120\.003\.850\.0020\.00DeepSeek\-Coder\-16B36\.5745\.3026\.2913\.7937\.5715\.0011\.540\.0020\.00DeepSeek\-v4\-flash52\.5460\.2243\.3233\.1053\.8720\.0030\.770\.0010\.00Table 14:Execution accuracy \(%\) on Oracle under the original BIRD difficulty andUniQLconstruction\-source stratifications\.BIRD difficultyUniQLconstruction sourceModelAllSimpleModerateChallengingToolLLMReflectionRule RefinementHumanClosed\-weight LLMsGPT\-3\.5\-Turbo49\.3559\.5737\.5022\.0760\.1217\.8312\.2013\.6420\.00GPT\-5\-mini60\.5664\.8654\.0953\.7969\.9842\.6821\.9518\.1824\.00GPT\-5\.1\-codex49\.6754\.8142\.4640\.0057\.8033\.1215\.8518\.1824\.00Gemini\-2\.5\-Pro34\.2939\.4625\.6528\.9739\.7124\.8411\.599\.0912\.00Claude\-4\.5\-Sonnet63\.7569\.5156\.4750\.3474\.7042\.0417\.6822\.7328\.00Open\-weight LLMsQwen3\-1\.7B43\.4852\.6531\.4723\.4553\.8612\.7410\.374\.554\.00Qwen3\-4B44\.9250\.2738\.7930\.3454\.2919\.1112\.2013\.6412\.00Qwen3\-8B51\.1156\.3245\.6935\.1760\.3829\.3014\.6318\.1824\.00Qwen3\-32B53\.0658\.7046\.5537\.9362\.3533\.7616\.469\.0920\.00Llama\-3\-8B\-Inst31\.2339\.5720\.0413\.7938\.687\.018\.544\.558\.00Llama\-3\-70B\-Inst51\.5659\.2443\.9726\.9063\.1218\.4711\.594\.5524\.00DeepSeek\-Coder\-16B45\.4454\.8133\.8422\.7655\.7514\.6511\.599\.0912\.00DeepSeek\-v4\-flash57\.5064\.5449\.1439\.3168\.2732\.4815\.8518\.1820\.00Table 15:Execution accuracy \(%\) on PostgreSQL under the original BIRD difficulty andUniQLconstruction\-source stratifications\.BIRD difficultyUniQLconstruction sourceModelAllSimpleModerateChallengingToolLLMReflectionRule RefinementHumanClosed\-weight LLMsGPT\-3\.5\-Turbo36\.4445\.7323\.4918\.6241\.5012\.5812\.284\.170\.00GPT\-5\-mini51\.3758\.0540\.5243\.4554\.9938\.4129\.8212\.5025\.00GPT\-5\.1\-codex52\.3559\.4642\.4638\.6256\.4037\.0929\.828\.3325\.00Gemini\-2\.5\-Pro53\.5959\.2445\.6942\.7658\.1939\.0722\.818\.3310\.00Claude\-4\.5\-Sonnet54\.9560\.7647\.4142\.0758\.8142\.3835\.0912\.5010\.00Open\-weight LLMsQwen3\-1\.7B32\.9241\.6220\.9115\.8636\.9714\.5712\.284\.175\.00Qwen3\-4B42\.7050\.2732\.3327\.5947\.7419\.2119\.304\.1710\.00Qwen3\-8B46\.3554\.4936\.8524\.8350\.3929\.8026\.328\.3315\.00Qwen3\-32B48\.5056\.2239\.0129\.6652\.3434\.4429\.824\.1715\.00Llama\-3\-8B\-Inst22\.4328\.9715\.094\.1425\.666\.628\.770\.000\.00Llama\-3\-70B\-Inst40\.1648\.5431\.2515\.1745\.0917\.8815\.794\.175\.00DeepSeek\-Coder\-16B31\.8839\.3521\.3417\.9336\.1213\.915\.264\.175\.00DeepSeek\-v4\-flash48\.5055\.7839\.8729\.6652\.6532\.4528\.078\.3310\.00Table 16:Execution accuracy \(%\) on Presto under the original BIRD difficulty andUniQLconstruction\-source stratifications\.BIRD difficultyUniQLconstruction sourceModelAllSimpleModerateChallengingToolLLMReflectionRule RefinementHumanClosed\-weight LLMsGPT\-3\.5\-Turbo36\.3147\.0320\.6917\.9340\.7213\.5911\.760\.005\.00GPT\-5\-mini50\.9857\.3042\.6737\.2453\.5039\.1335\.2928\.5725\.00GPT\-5\.1\-codex52\.4159\.1443\.3238\.6254\.9739\.6741\.1828\.5730\.00Gemini\-2\.5\-Pro57\.8963\.7851\.0842\.0761\.2839\.6747\.0642\.8625\.00Claude\-4\.5\-Sonnet58\.0864\.1150\.0045\.5260\.6645\.6552\.9428\.5730\.00Open\-weight LLMsQwen3\-1\.7B32\.4641\.0821\.5512\.4136\.877\.6111\.767\.1410\.00Qwen3\-4B41\.7251\.3528\.8821\.3846\.8113\.5917\.650\.0020\.00Qwen3\-8B32\.9938\.2728\.2314\.4836\.5713\.0429\.410\.0010\.00Qwen3\-32B47\.7856\.6538\.1522\.0751\.0432\.0735\.2914\.2915\.00Llama\-3\-8B\-Inst22\.2329\.4112\.507\.5925\.026\.5211\.760\.0010\.00Llama\-3\-70B\-Inst38\.0147\.3526\.7214\.4843\.1110\.3311\.760\.0010\.00DeepSeek\-Coder\-16B31\.5539\.7820\.4714\.4835\.3311\.4117\.650\.005\.00DeepSeek\-v4\-flash49\.0257\.4139\.4426\.2153\.1226\.6341\.187\.1425\.00Table 17:Execution accuracy \(%\) on Spark under the original BIRD difficulty andUniQLconstruction\-source stratifications\.BIRD difficultyUniQLconstruction sourceModelAllSimpleModerateChallengingToolLLMReflectionRule RefinementHumanClosed\-weight LLMsGPT\-3\.5\-Turbo39\.3748\.8626\.0821\.3843\.1522\.063\.950\.0016\.67GPT\-5\-mini52\.9357\.9545\.2645\.5256\.8535\.2911\.840\.0041\.67GPT\-5\.1\-codex53\.6559\.3545\.0444\.8357\.5938\.2413\.160\.0036\.11Gemini\-2\.5\-Pro56\.6561\.8449\.5746\.2160\.7147\.0613\.160\.0030\.56Claude\-4\.5\-Sonnet56\.1362\.1647\.8444\.1460\.2744\.1215\.790\.0025\.00Open\-weight LLMsQwen3\-1\.7B35\.2742\.8125\.0020\.0038\.6223\.532\.630\.0011\.11Qwen3\-4B43\.9452\.4332\.1127\.5947\.4035\.293\.950\.0027\.78Qwen3\-8B47\.4654\.9238\.3628\.9750\.8219\.7110\.530\.0027\.78Qwen3\-32B49\.2256\.4340\.7330\.3452\.9839\.717\.890\.0027\.78Llama\-3\-8B\-Inst23\.3429\.8414\.878\.9725\.975\.881\.320\.0011\.11Llama\-3\-70B\-Inst42\.8950\.7034\.0521\.3846\.9526\.472\.630\.0019\.44DeepSeek\-Coder\-16B34\.4242\.0524\.3517\.9337\.9517\.653\.950\.008\.33DeepSeek\-v4\-flash51\.5657\.8443\.5337\.2455\.8836\.769\.210\.0022\.22Table 18:Execution accuracy \(%\) on SQLite under the original BIRD difficulty andUniQLconstruction\-source stratifications\.BIRD difficultyUniQLconstruction sourceModelAllSimpleModerateChallengingToolLLMReflectionRule RefinementHumanClosed\-weight LLMsGPT\-3\.5\-Turbo41\.1350\.8127\.1624\.1447\.4531\.2831\.1329\.0030\.23GPT\-5\-mini52\.4857\.9544\.4043\.4554\.6142\.6538\.1620\.0030\.56GPT\-5\.1\-codex53\.7859\.4645\.9142\.7656\.4738\.2436\.8420\.0027\.78Gemini\-2\.5\-Pro59\.7865\.4152\.1648\.2861\.5357\.3548\.6840\.0027\.78Claude\-4\.5\-Sonnet59\.8465\.0852\.8048\.9761\.6155\.8850\.0030\.0030\.56Open\-weight LLMsQwen3\-1\.7B35\.4043\.2424\.5720\.0037\.4320\.7517\.149\.0914\.29Qwen3\-4B46\.9455\.0337\.0726\.9049\.1233\.9617\.1427\.2721\.43Qwen3\-8B49\.0957\.4139\.0128\.2850\.7341\.5125\.7136\.3614\.29Qwen3\-32B53\.3960\.2245\.2635\.8655\.7042\.4517\.1427\.2721\.43Llama\-3\-8B\-Inst23\.6029\.1917\.248\.2824\.9313\.2120\.000\.000\.00Llama\-3\-70B\-Inst42\.2450\.5932\.7619\.3144\.7423\.5817\.149\.0928\.57DeepSeek\-Coder\-16B35\.0144\.3222\.6315\.1736\.7723\.5817\.149\.0914\.29DeepSeek\-v4\-flash53\.4661\.5143\.9732\.4155\.6539\.7143\.4220\.0027\.78Table 19:Execution accuracy \(%\) on StarRocks under the original BIRD difficulty andUniQLconstruction\-source stratifications\.BIRD difficultyUniQLconstruction sourceModelAllSimpleModerateChallengingToolLLMReflectionRule RefinementHumanClosed\-weight LLMsGPT\-3\.5\-Turbo39\.1847\.8925\.6526\.9043\.4418\.854\.760\.0011\.11GPT\-5\-mini51\.8957\.9543\.3240\.6955\.2745\.0812\.700\.0022\.22GPT\-5\.1\-codex52\.4157\.9545\.0440\.6955\.9541\.8017\.460\.0022\.22Gemini\-2\.5\-Pro54\.4361\.1944\.4043\.4557\.7049\.1812\.700\.0033\.33Claude\-4\.5\-Sonnet56\.0661\.9548\.7142\.0759\.2151\.6415\.870\.0033\.33Open\-weight LLMsQwen3\-1\.7B36\.3144\.1125\.2222\.0739\.5823\.776\.350\.0011\.11Qwen3\-4B45\.5053\.4134\.7029\.6648\.8235\.259\.520\.0027\.78Qwen3\-8B47\.4654\.9238\.5828\.2851\.3333\.909\.680\.0014\.52Qwen3\-32B50\.5256\.5443\.5334\.4854\.1341\.809\.520\.0022\.22Llama\-3\-8B\-Inst24\.1229\.3017\.8911\.0326\.5414\.751\.590\.005\.56Llama\-3\-70B\-Inst42\.5749\.3035\.1323\.4546\.1730\.333\.170\.0027\.78DeepSeek\-Coder\-16B34\.8842\.1624\.3522\.0738\.2918\.856\.350\.0016\.67DeepSeek\-v4\-flash50\.2056\.6541\.5936\.5553\.3043\.4415\.878\.3316\.67Table 20:Execution accuracy \(%\) on T\-SQL under the original BIRD difficulty andUniQLconstruction\-source stratifications\.BIRD difficultyUniQLconstruction sourceModelAllSimpleModerateChallengingToolLLMReflectionRule RefinementHumanClosed\-weight LLMsGPT\-3\.5\-Turbo37\.7448\.6521\.1221\.3840\.8616\.982\.860\.007\.14GPT\-5\-mini53\.8560\.0044\.1845\.5256\.2138\.6822\.860\.0057\.14GPT\-5\.1\-codex54\.6361\.1946\.7737\.9357\.3834\.9120\.009\.0957\.14Gemini\-2\.5\-Pro55\.6162\.1646\.7742\.0758\.9237\.7411\.430\.0021\.43Claude\-4\.5\-Sonnet56\.2662\.2748\.4942\.7659\.3637\.7417\.149\.0928\.57Open\-weight LLMsQwen3\-1\.7B32\.6642\.4919\.8311\.0335\.5313\.210\.000\.007\.14Qwen3\-4B46\.4155\.7834\.2725\.5249\.5626\.4211\.430\.0014\.29Qwen3\-8B48\.5757\.3037\.0729\.6651\.9727\.3611\.430\.007\.14Qwen3\-32B51\.6958\.9241\.8137\.2454\.6137\.7411\.430\.0014\.29Llama\-3\-8B\-Inst21\.1227\.1413\.158\.2823\.176\.600\.000\.000\.00Llama\-3\-70B\-Inst40\.6148\.6530\.6021\.3844\.1516\.040\.000\.0014\.29DeepSeek\-Coder\-16B31\.6840\.3220\.6911\.7234\.5811\.320\.000\.007\.14DeepSeek\-v4\-flash51\.4358\.7041\.5936\.5555\.2627\.368\.570\.007\.14Table 21:Execution accuracy \(%\) on Teradata under the original BIRD difficulty andUniQLconstruction\-source stratifications\.BIRD difficultyUniQLconstruction sourceModelAllSimpleModerateChallengingToolLLMReflectionRule RefinementHumanClosed\-weight LLMsGPT\-3\.5\-Turbo23\.8630\.9212\.2815\.8639\.498\.511\.380\.004\.92GPT\-5\-mini35\.8541\.9526\.2927\.5955\.3119\.333\.670\.0014\.75GPT\-5\.1\-codex35\.0742\.0524\.7823\.4554\.4718\.043\.210\.0016\.39Gemini\-2\.5\-Pro38\.4044\.6530\.1724\.8359\.1820\.365\.500\.0013\.11Claude\-4\.5\-Sonnet37\.7444\.3228\.2326\.2157\.8520\.624\.130\.0018\.03Open\-weight LLMsQwen3\-1\.7B22\.4330\.1611\.428\.2837\.926\.190\.920\.006\.56Qwen3\-4B29\.6037\.5118\.5314\.4846\.0114\.693\.672\.5611\.48Qwen3\-8B29\.3436\.5420\.0413\.1045\.6513\.322\.933\.3315\.91Qwen3\-32B32\.2739\.6821\.5519\.3149\.0317\.785\.502\.5611\.48Llama\-3\-8B\-Inst15\.4520\.766\.689\.6626\.453\.610\.920\.003\.28Llama\-3\-70B\-Inst26\.4732\.8617\.6713\.7943\.4810\.050\.920\.008\.20DeepSeek\-Coder\-16B19\.9526\.929\.708\.2833\.825\.670\.920\.003\.28DeepSeek\-v4\-flash30\.9037\.6220\.9120\.0048\.9113\.923\.670\.0011\.48Table 22:Execution accuracy \(%\) on Trino under the original BIRD difficulty andUniQLconstruction\-source stratifications\.BIRD difficultyUniQLconstruction sourceModelAllSimpleModerateChallengingToolLLMReflectionRule RefinementHumanClosed\-weight LLMsGPT\-3\.5\-Turbo35\.8545\.5121\.3420\.6938\.5618\.077\.840\.000\.00GPT\-5\-mini49\.0256\.1139\.6633\.7950\.9144\.5823\.530\.0011\.76GPT\-5\.1\-codex50\.7258\.1641\.8131\.7253\.1637\.3523\.530\.0017\.65Gemini\-2\.5\-Pro55\.6161\.6247\.6342\.7658\.1045\.7825\.490\.0011\.76Claude\-4\.5\-Sonnet56\.0662\.2748\.0642\.0758\.1046\.9933\.330\.0023\.53Open\-weight LLMsQwen3\-1\.7B34\.1643\.4622\.2013\.1036\.8913\.257\.840\.005\.88Qwen3\-4B41\.4651\.5728\.6617\.9344\.5919\.289\.800\.005\.88Qwen3\-8B44\.2053\.0833\.6221\.3846\.9925\.3017\.650\.005\.88Qwen3\-32B46\.6154\.2738\.5823\.4548\.8736\.1417\.650\.0017\.65Llama\-3\-8B\-Inst22\.7528\.4315\.958\.2824\.697\.235\.880\.000\.00Llama\-3\-70B\-Inst39\.0548\.2228\.6613\.7942\.1215\.665\.8816\.6711\.76DeepSeek\-Coder\-16B32\.5341\.0821\.7712\.4135\.298\.439\.800\.005\.88DeepSeek\-v4\-flash47\.5955\.6837\.9326\.9049\.7532\.5325\.4916\.6723\.53
## Appendix EArtifact License and Intended Use

UniQLis constructed from the public BIRD development setLiet al\.\([2023](https://arxiv.org/html/2606.08018#bib.bib231)\)\. We use BIRD as a research benchmark for text\-to\-SQL evaluation and extend it by constructing executable SQL annotations across multiple SQL dialects\. This use is consistent with the original purpose of BIRD as a benchmark for evaluating and advancing text\-to\-SQL systems in research settings\.

We plan to release theUniQLbenchmark, construction metadata, evaluation scripts, and accompanying documentation to support reproducibility and future research\. The benchmark is intended for evaluating cross\-dialect text\-to\-SQL systems, analyzing SQL dialect robustness, and reproducing or extending the experiments reported in this paper\. SinceUniQLis a derivative benchmark based on BIRD, users ofUniQLare required to comply with the license, citation requirements, and usage terms of the source BIRD dataset\. In particular, the released benchmark data should be used only in contexts that are compatible with the original access conditions and intended research use of BIRD\.

For artifacts created by us, including evaluation scripts, construction utilities, and documentation, we will specify the corresponding license in the released repository\. The release will also document the intended use of each artifact, citation requirements, and any restrictions inherited from the source benchmark\.

## Appendix FData Privacy and Content Safety

UniQLdoes not involve collecting new private user data\. The natural language questions, database schemas, database contents, evidence fields, and original SQLite SQL queries are derived from the public BIRD development setLiet al\.\([2023](https://arxiv.org/html/2606.08018#bib.bib231)\)\. Our construction process focuses on translating and verifying SQL annotations across target dialects, and does not introduce new personal identifiers or user\-generated private content\.

We reviewed the data used inUniQL, including natural language questions, SQL annotations, evidence fields, and database contents, for personally identifying information and offensive content\. The dialect translation process only rewrites SQL queries into target SQL dialects and preserves the public benchmark setting of the source dataset\. Any released artifacts will follow the usage terms of BIRD and will be distributed with documentation describing their intended research use and inherited restrictions\.