ROSE: An Intent-Centered Evaluation Metric for NL2SQL
Summary
ROSE is a novel intent-centered evaluation metric for NL2SQL that uses a Prover-Refuter cascade to assess semantic correctness independently of ground-truth SQL, achieving 24% better agreement with human experts than existing metrics. The paper addresses limitations of Execution Accuracy and provides a re-evaluation of 19 NL2SQL methods with publicly released resources.
View Cached Full Text
Cached at: 04/20/26, 08:29 AM
Paper page - ROSE: An Intent-Centered Evaluation Metric for NL2SQL
Source: https://huggingface.co/papers/2604.12988
Abstract
ROSE is an intent-centered metric for NL2SQL evaluation that uses a Prover-Refuter cascade to assess semantic correctness independently of ground-truth SQL, showing superior agreement with human experts.
Execution Accuracy (EX), the widely used metric for evaluating the effectiveness of Natural Language to SQL (NL2SQL https://huggingface.co/papers?q=NL2SQL) solutions, is becoming increasingly unreliable. It is sensitive to syntactic variation, ignores that questions may admit multiple interpretations, and is easily misled by erroneous ground-truth SQL (https://huggingface.co/papers?q=ground-truth%20SQL). To address this, we introduce ROSE, an intent-centered metric (https://huggingface.co/papers?q=intent-centered%20metric) that focuses on whether the predicted SQL answers the question, rather than consistency with the ground-truth SQL (https://huggingface.co/papers?q=ground-truth%20SQL) under the reference-dependent paradigm. ROSE employs an adversarial Prover-Refuter cascade (https://huggingface.co/papers?q=Prover-Refuter%20cascade): SQL Prover assesses the semantic correctness (https://huggingface.co/papers?q=semantic%20correctness) of a predicted SQL against the user’s intent independently, while Adversarial Refuter uses the ground-truth SQL (https://huggingface.co/papers?q=ground-truth%20SQL) as evidence to challenge and refine this judgment. On our expert-aligned validation set ROSE-VEC, ROSE achieves the best agreement with human experts, outperforming the next-best metric by nearly 24% in Cohen’s Kappa (https://huggingface.co/papers?q=Cohen%27s%20Kappa). We also conduct a large scale re-evaluation of 19 NL2SQL (https://huggingface.co/papers?q=NL2SQL) methods, revealing four valuable insights. We release ROSE and ROSE-VEC to facilitate more reliable NL2SQL (https://huggingface.co/papers?q=NL2SQL) research.
View arXiv page (https://arxiv.org/abs/2604.12988) View PDF (https://arxiv.org/pdf/2604.12988) Add to collection (https://huggingface.co/login?next=%2Fpapers%2F2604.12988)
Get this paper in your agent:
hf papers read 2604.12988
Don’t have the latest CLI? curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2604.12988 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2604.12988 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2604.12988 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to a collection (https://huggingface.co/new-collection) to link it from this page.
Similar Articles
A Semantic-Layer-Mediated Agent for Natural Language to SQL over Heterogeneous Enterprise Databases
This paper presents a semantic-layer-mediated NL2SQL agent that decouples intent from physical execution by reasoning over a curated semantic model, achieving 94.15% execution accuracy on the Spider2-snow benchmark.
AgentNLQ: A General-Purpose Agent for Natural Language to SQL
This paper presents AgentNLQ, a multi-agent system for natural language to SQL conversion that achieves 78.1% semantic accuracy on the BIRD benchmark through schema enrichment and a self-correcting orchestrator.
RAS: Reflection-Augmented Scaling with In-Context Learning for Executable Cypher Query Generation
This paper introduces Reflection-Augmented Scaling (RAS), a method that uses execution feedback from failed Cypher queries to iteratively refine query generation via in-context learning, reducing execution error rates by 41-50% across multiple datasets and models.
FD-NL2SQL: Feedback-Driven Clinical NL2SQL that Improves with Use
FD-NL2SQL is a feedback-driven natural language to SQL system for clinical oncology databases that improves with use through clinician edits and logic-based SQL augmentation. The system decomposes natural language questions into predicates, retrieves expert-verified exemplars, and synthesizes executable SQL with continuous learning capabilities.
A Formally Grounded ODRL Evaluator: Implementation and Comparison
This paper presents a formally grounded ODRL evaluator with transparent semantics, supporting all rule types, and compares its performance with existing evaluators, addressing interoperability issues in policy modelling for data access and AI governance.