SemPlan: Benchmarking Structured Semantic Planning for LLM-Based Queries over Enterprise Data

# Benchmarking Structured Semantic Planning for LLM-Based Queries over Enterprise Data
Source: [https://arxiv.org/html/2608.13612](https://arxiv.org/html/2608.13612)
## SemPlan: Benchmarking Structured Semantic Planning for LLM\-Based Queries over Enterprise DataThanks:Generative AI tools assisted development and debugging of the experimental software pipeline\. The author is responsible for the study design, validation, analysis, and scientific claims\.

Bruno Santos TeixeiraAffiliation:Universidade Federal de Ouro Preto \(UFOP\), BrazilAffiliation:ORCID:[0009\-0007\-3860\-7114](https://orcid.org/0009-0007-3860-7114)

###### Abstract

Natural\-language interfaces to enterprise data must translate underspecified requests into governed, executable behavior while controlling invalid queries, policy failures, cost, and nondeterminism\. SemPlan Benchmark evaluates this architectural design space with a deterministic synthetic bilingual benchmark containing 1,800 cases in English and Brazilian Portuguese; 1,200 cases form the frozen scientific evaluation subset\. Four architectures are compared under the same model configuration: direct SQL generation \(A1\), a bounded tool\-agent baseline \(A2\), structured semantic\-request generation followed by deterministic planning and execution \(A3\), and a clarification/stateful semantic\-plan variant \(A4\)\. Across 4,800 primary records, answer correctness was low in absolute terms: 22\.25% for A1, 22\.58% for A2, 25\.67% for A3, and 24\.25% for A4\. A3 had the highest observed correctness and significantly exceeded A1, A2, and A4 in the pre\-specified paired correctness analysis, while A1 retained the highest policy\-correct rate and the lowest unsafe\-or\-invalid rate\. A4 had the lowest mean API cost and lowest false\-refusal rate\. On a preselected 150\-case stability subset, answer\-correct repeatability ranged from 92\.00% to 98\.67%\. The results support a trade\-off interpretation rather than a universal ranking: additional structural constraints changed failure modes and efficiency, but did not monotonically improve correctness or solve ambiguity and multi\-turn state consistency\.

## 1Introduction

Natural\-language database access has long promised to make structured data available to users who do not write query languages\. The core difficulty, however, is not merely generating syntactically valid SQL\. Enterprise analytics questions often refer to business metrics, implicit dimensions, temporal scopes, authorization boundaries, unsupported actions, and conversational context\. A system can therefore be syntactically correct while still being semantically wrong, policy\-inconsistent, or operationally invalid\. Classical natural\-language interfaces to databases identified ambiguity, domain modeling, and portability as persistent concerns\([2](https://arxiv.org/html/2608.13612#bib.bib1);[1](https://arxiv.org/html/2608.13612#bib.bib2)\); modern benchmarks such as Spider, SParC, CoSQL, and BIRD have renewed these questions under cross\-domain, conversational, and database\-grounded settings\([14](https://arxiv.org/html/2608.13612#bib.bib3);[15](https://arxiv.org/html/2608.13612#bib.bib4);[13](https://arxiv.org/html/2608.13612#bib.bib5);[5](https://arxiv.org/html/2608.13612#bib.bib6)\)\.

Current LLM\-based systems can place different amounts of responsibility on the model\. At one extreme, the model generates SQL directly\. Other designs ask the model to select tools, generate a typed semantic representation, or interact with deterministic normalization and execution layers\. These choices are often discussed as engineering preferences, but they also define distinct empirical hypotheses about where errors arise and which constraints improve reliability\.

SemPlan studies this design space while holding the model, benchmark cases, database snapshot, and evaluation rules fixed\. The central question is:*how do different forms and degrees of structural constraint in LLM\-mediated enterprise\-data querying trade off answer correctness, policy behavior, failure modes, cost, and repeatability?*The study is deliberately narrower than a claim of production readiness\. It uses an independently generated synthetic enterprise domain and does not use customer or proprietary data\.

The paper makes four contributions\. First, it introduces a bilingual synthetic benchmark for controlled evaluation of LLM\-mediated structured enterprise\-data queries\. Second, it compares four architecture patterns under a common experimental configuration rather than comparing different model families\. Third, it reports paired evidence on answer correctness, policy behavior, typed failure outcomes, cost, ambiguity, multi\-turn state consistency, and repeatability\. Fourth, it provides a reproducibility package built around frozen prompts, manifests, raw\-output hashes, score records, generated tables and figures, and no\-key validation paths\.

## 2Related Work

Natural\-language interfaces to databases predate current LLM systems and have historically relied on grammar, parsing, semantic representations, domain constraints, and schema\-specific transformations\([2](https://arxiv.org/html/2608.13612#bib.bib1);[1](https://arxiv.org/html/2608.13612#bib.bib2)\)\. Modern text\-to\-SQL research shifted toward cross\-domain evaluation\. Spider established a challenging setting with databases and query structures that differ between training and test data\([14](https://arxiv.org/html/2608.13612#bib.bib3)\)\. SParC extended the task to contextual interactions, while CoSQL added conversational phenomena such as clarification and unanswerable requests\([15](https://arxiv.org/html/2608.13612#bib.bib4);[13](https://arxiv.org/html/2608.13612#bib.bib5)\)\. BIRD further emphasized large database contents, external knowledge, and efficiency\([5](https://arxiv.org/html/2608.13612#bib.bib6)\)\.

A second line of work reduces the gap between language and executable SQL through intermediate structure or constrained decoding\. IRNet uses schema linking and a SemQL intermediate representation before deterministic SQL inference\([4](https://arxiv.org/html/2608.13612#bib.bib8)\)\. RAT\-SQL emphasizes relation\-aware schema encoding and linking\([10](https://arxiv.org/html/2608.13612#bib.bib9)\)\. Execution\-guided decoding filters candidates using execution feedback, while PICARD incrementally rejects inadmissible tokens during generation\([11](https://arxiv.org/html/2608.13612#bib.bib7);[9](https://arxiv.org/html/2608.13612#bib.bib10)\)\. These methods motivate the broader principle that formal constraints can reduce some classes of invalid output, but they do not imply that additional constraints always improve end\-to\-end task correctness\.

Tool\-using language\-model systems represent another architecture family\. ReAct interleaves reasoning and actions, while Toolformer studies how models decide when and how to invoke external APIs\([12](https://arxiv.org/html/2608.13612#bib.bib11);[8](https://arxiv.org/html/2608.13612#bib.bib12)\)\. In current API systems, structured outputs and function\-calling mechanisms can constrain model responses or tool arguments to schemas\([7](https://arxiv.org/html/2608.13612#bib.bib14);[6](https://arxiv.org/html/2608.13612#bib.bib15)\)\. SemPlan is complementary to these lines of work: rather than proposing a single decoding technique, it evaluates four end\-to\-end architectural allocations of responsibility between an LLM and deterministic software components\.

The benchmark design also follows the broader reproducibility principle that datasets should communicate motivation, composition, generation, intended use, and limitations\([3](https://arxiv.org/html/2608.13612#bib.bib13)\)\. SemPlan therefore separates synthetic data generation, benchmark generation, human review, gold execution, model execution, and derived statistical artifacts\.

## 3Research Questions and Pre\-Specified Hypotheses

The study was frozen around five research questions\.RQ1asks how A1–A4 differ in final answer correctness on bilingual enterprise\-analytics questions\.RQ2asks how they differ in invalid, execution\-failure, and policy\-related outcomes\.RQ3asks what cost and latency trade\-offs arise\.RQ4asks whether formal clarification improves behavior on materially ambiguous questions\.RQ5asks whether structured state improves consistency in PATCH/REPLACE follow\-ups\.

Before the hidden evaluation, directional hypotheses expected semantic\-plan approaches to improve semantic and answer correctness over direct SQL; to reduce invalid or policy\-violating execution relative to less structured alternatives; to keep A3 cost at or below the tool\-agent approach; and to improve ambiguity and multi\-turn consistency through A4 clarification and structured state\. The analysis below preserves mixed and negative findings rather than redefining those hypotheses after observing results\.

## 4SemPlan Benchmark

### 4\.1Synthetic enterprise domain

The benchmark uses the synthetic*Northstar Commerce*analytics domain\. Its relational data cover customers, products, orders, payments, expenses, budgets, suppliers, contracts, and a calendar dimension\. Monetary values, dates, status fields, and derived business metrics are generated from deterministic seeds and documented invariants\. The reference execution environment uses governed PostgreSQL views and a read\-only execution policy\.

### 4\.2Case generation, review, and splits

Benchmark construction separates textual cases from the underlying tabular dataset\. Parameterized semantic templates define intents, metrics, dimensions, filters, temporal contexts, grouping, sorting, limits, ambiguity conditions, and policy expectations\. Surface forms are generated for English and Brazilian Portuguese, then validated for canonical identifiers and language quality\. Gold semantic plans execute deterministically against the reference database to produce normalized gold answers\. Cases are deduplicated by structural and lexical lineage before the test partitions are frozen\.

The release\-scale benchmark contains 1,800 cases: 900 en\-US and 900 pt\-BR\. Its splits are development \(300\), validation \(300\), public test \(500\), hidden test \(300\), multi\-turn \(200\), and adversarial \(200\)\. The frozen scientific evaluation subset contains 1,200 cases: public test, hidden test, multi\-turn, and adversarial\. Development and validation cases are excluded from the primary scientific comparison\. The release manifest records 3,600 approved case/gold review records from a single human reviewer\. This strong author\-in\-the\-loop role is treated as a validity limitation rather than hidden\.

Table 1:Composition of the frozen SemPlan Benchmark version 1\.0\.0\-rc\.2\.The benchmark includes lookup, grouped aggregation, ranking, comparison, variance, trend, share/ratio, filtering, contract\-status, ambiguity, out\-of\-scope, multi\-turn, and adversarial cases\. Difficulty labels derive from documented structural factors rather than model performance\.

### 4\.3Illustrative bilingual case

A typical single\-turn case asks for a governed metric under explicit business filters\. For example, an English utterance may be*“What was net revenue for consumer customers in the North region through the online channel in January 2022?”*; its Brazilian\-Portuguese counterpart can express the same semantics as*“Qual foi a receita líquida dos clientes do segmento consumidor na região Norte pelo canal online em janeiro de 2022?”*\. The canonical semantic request identifiesnet\_revenue, the region/channel/segment filters, and the time window\. This example illustrates the benchmark contract; primary evaluation uses the frozen case artifacts rather than manually rewritten examples in the manuscript\.

## 5Evaluated Approaches

All four approaches use the same configured model,gpt\-5\.6\-luna, through a provider\-neutral OpenAI adapter with reasoning effortlowand a 1,200\-token output ceiling\. Prompts are frozen independently per approach\. Figure[1](https://arxiv.org/html/2608.13612#S5.F1)summarizes the allocation of model and deterministic responsibilities\.

A1User queryLLMSQLSQL guardDBA2User queryLLM agentTyped toolsDeterministicexecutorDBA3User queryLLMSemanticRequestNormalize \+executeDBA4Query \+structured stateLLMRequest / clarify/ patchNormalize \+ state\+ executeDB

Figure 1:Where structure enters the SemPlan comparison\. Only A1 permits model\-authored SQL\. A2 constrains model actions to typed tools\. A3 and A4 emit typed semantic requests that are normalized and executed by deterministic software; A4 additionally exposes structured state and clarification behavior\.A1: Direct SQL\.The model receives the utterance, locale, reference date, a compact governed view schema, and relevant metric definitions\. It returns strict JSON containing SQL or a typed refusal\. Generated SQL is not executed directly: it must pass parsing, AST allowlists, complexity limits, read\-only authorization, statement timeout, and row limits\. A1 never receives gold SQL, plans, or answers\.

A2: Tool Agent\.The model receives a bounded set of analytics tools for operations such as aggregation, ranking, period comparison, budget comparison, contract status, and field description\. Tool arguments follow strict schemas and canonical catalog identifiers\. The agent may compose only a bounded number of tool calls and cannot execute arbitrary SQL or invent tools\. The deterministic executor implements the underlying analytics operations\.

A3: Semantic Request\.The model does not generate SQL\. Instead, it emits a strict semantic\-request envelope containing operation, metrics, dimensions, filters, time grain, sort, limit, comparison fields, clarification fields, and confidence\. Deterministic normalization validates identifiers and types, resolves documented semantic rules, and compiles the request into an executable semantic plan\. Governed software then produces parameterized SQL or an equivalent deterministic operator call\. Thus SQL exists in A3, but it is generated by the application, not by the LLM\.

A4: Clarification and Structured State\.A4 extends the A3 interface with prior structured state, typed clarification outcomes, and explicit PATCH/REPLACE semantics for multi\-turn requests\. Only fields explicitly changed by a PATCH are updated subject to compatibility rules; out\-of\-scope turns preserve state unless reset\. Execution remains deterministic after normalization\.

## 6Experimental Setup

The frozen experiment uses benchmark version 1\.0\.0\-rc\.2 and the same case set, database snapshot, model configuration, and approach\-specific frozen prompts for all primary comparisons\. Each of the 1,200 scientific cases is executed once under each of A1–A4, producing 4,800 primary records\. A deterministic stratified subset of 150 cases is then executed two additional times per approach, producing 1,200 stability records\. These repeats are analyzed only for within\-approach repeatability and do not inflate the primary sample size\. The complete scientific run therefore contains 6,000 planned request identities\.

Primary binary endpoints are compared on case\-aligned records\. McNemar tests and paired risk differences with 95% confidence intervals are used for binary outcomes, and Holm\-Bonferroni adjustment is applied to the pre\-specified primary comparison family\. Skewed continuous outcomes such as API cost use paired bootstrap contrasts\. Provider, transport, infrastructure, and model\-quality failures are distinguished before statistical interpretation\. Model\-quality failures remain part of the scientific outcome when the system produced sufficient evidence to classify them\.

Latency was pre\-specified but is excluded from primary conclusions\. The post\-run audit found that captured typed failures record zero latency, and those failures are unevenly distributed across approaches\. Consequently, the resulting medians are not comparable as a clean end\-to\-end latency measure\. Token usage, provider cost, and failure counts remain reportable because they are independently recorded in provider usage and budget/result ledgers\.

## 7Results

### 7\.1Primary correctness and policy outcomes

The run completed all 6,000 planned request identities with zero missing identities and zero duplicate purchases\. In the final state there were no provider failures and no uncaptured infrastructure failures\. Table[2](https://arxiv.org/html/2608.13612#S7.T2)reports the primary endpoints for the 1,200 cases per approach\.

Table 2:Primary metrics over 1,200 scientific cases per approach\. Absolute correctness is reported before pairwise significance\.Absolute answer correctness is low across all approaches\. A3 achieved the highest observed rate at 25\.67%, followed by A4 at 24\.25%, A2 at 22\.58%, and A1 at 22\.25% \(Figure[2](https://arxiv.org/html/2608.13612#S7.F2)\)\. The paired answer\-correct analysis estimates A3 over A1 at \+3\.42 percentage points \(95% CI \[1\.85, 4\.98\], Holm\-adjustedp=0\.000110p=0\.000110\), and A3 over A2 at \+3\.08 points \(95% CI \[1\.68, 4\.49\], Holm\-adjustedp=0\.000110p=0\.000110\)\. A4 is 1\.42 points below A3 \(A4–A3 95% CI \[\-2\.51, \-0\.32\], Holm\-adjustedp=0\.041250p=0\.041250\)\. Under the frozen analysis, A3 therefore has higher primary answer correctness than each of A1, A2, and A4, but the absolute rates remain modest\.

Figure 2:Primary answer correctness from the frozen primary\-results table\. The figure shows absolute rates; paired uncertainty is reported in Table[3](https://arxiv.org/html/2608.13612#S7.T3)\.Policy behavior yields a different ordering\. A1 has the highest policy\-correct rate \(43\.67%\) and the lowest unsafe\-or\-invalid rate \(31\.00%\)\. A4 has the lowest false\-refusal rate \(0\.17%\) but the highest unsafe\-or\-invalid rate \(64\.08%\)\. A3 has 37\.33% policy correctness and 60\.58% unsafe\-or\-invalid\. The paired policy analysis shows that A3 and A4 are both below A1 on policy correctness\. The correctness advantage of A3 therefore does not establish a uniformly better or safer architecture\.

Table 3:Selected paired binary contrasts\. Risk difference is left minus right\.
### 7\.2Exploratory language subgroup

Language\-stratified correctness is consistently higher on en\-US than pt\-BR for all four approaches \(Table[4](https://arxiv.org/html/2608.13612#S7.T4)\)\. A3 records 31\.00% in en\-US and 20\.33% in pt\-BR; the corresponding A4 rates are 29\.33% and 19\.17%\. Because subgroup comparisons were not the sole primary endpoint and template families can be correlated, these results are descriptive/exploratory rather than evidence that the architecture causes a language\-specific effect\. They nevertheless identify multilingual robustness as an important follow\-up question\.

Table 4:Exploratory primary answer correctness by language\. Each language subgroup hasn=600n=600cases per approach\.

## 8Failure Analysis

Typed model\-quality failures are central to the interpretation rather than discarded as missing data\. Across all 6,000 scientific records, the recorded error codes areEXECUTION\_FAILED\(1,762\),OUTPUT\_SCHEMA\_INVALID\(1,051\),POLICY\_VIOLATION\(368\),CATALOG\_UNKNOWN\_ID\(78\), andCFG\_INVALID\(24\)\. In the 4,800 primary records alone, the corresponding counts are 1,402, 845, 287, 65, and 16\.

Table 5:Typed model\-quality failure categories\. These are scientific outcomes rather than missing provider evidence\.EXECUTION\_FAILEDindicates that a model\-produced or normalized request reached governed execution but failed a typed database/execution rule\.OUTPUT\_SCHEMA\_INVALIDindicates failure of the strict output contract\.POLICY\_VIOLATIONrecords guard\-layer rejection;CATALOG\_UNKNOWN\_IDrecords unknown semantic identifiers; andCFG\_INVALIDrecords a captured configuration or contract invalidity\.

Approach\-level terminal outcomes show that additional structure did not simply remove errors\. A1 produced 372 primaryERRORoutcomes, compared with 747 for A2, 727 for A3, and 769 for A4\. At the same time, A1 produced 257 false refusals, compared with 22, 19, and 2 for A2–A4\. This pattern supports a failure\-shift interpretation: direct SQL plus strict guards is conservative in one way, while typed semantic/tool interfaces expose different contract and execution failure surfaces\. The aggregate artifacts do not establish a unique causal mechanism for each error, so the discussion avoids attributing individual error\-code differences to specific modules without case\-level analysis\.

Table 6:Primary terminal\-outcome counts by approach\.
## 9Stability, Ambiguity, and Multi\-Turn Behavior

The stability substudy uses the primary execution plus two additional independent executions for each of 150 preselected cases per approach\. Repeatability asks whether answer correctness is stable across those three executions; it is not a claim of full determinism\. A3 has the highest observed repeatability \(98\.67%\), A1 and A4 are both 98\.00%, and A2 is 92\.00%\.

Table 7:Answer\-correct repeatability on the 150\-case stability subset\.Conversational robustness is substantially weaker\. Clarification\-decision correctness is 0\.00% for A1, 26\.19% for A2, 27\.38% for A3, and 8\.33% for A4 over 84 ambiguity cases per approach\. Multi\-turn sequence\-state correctness is 20\.00%, 17\.00%, 14\.00%, and 14\.50% for A1–A4 over 200 multi\-turn cases\. The clarification\-enabled A4 design therefore does not validate the pre\-specified expectation that structured clarification would dominate the alternatives, and structured state does not improve sequence\-state correctness in this configuration\.

Table 8:Ambiguity and multi\-turn endpoints\. Weak absolute rates are treated as primary scientific limitations\.
## 10Cost and Operational Results

The complete F7 scientific run recorded USD 4\.028973 in API charges\. Mean cost per primary record is USD 0\.000918 for A1, 0\.000927 for A2, 0\.000512 for A3, and 0\.000469 for A4\. A4 is the least expensive observed approach and A3 is also substantially cheaper than A1/A2 under this frozen model and price table\. The paired mean cost difference is \-USD 0\.000415 for A3 versus A2 and \-USD 0\.000459 for A4 versus A2\. These values are provider\-API observations, not total cost of ownership; they exclude infrastructure, engineering time, and any future pricing changes\.

Figure 3:Observed cost\-correctness trade\-off from frozen primary tables\. A3 has the highest correctness, while A4 has the lowest mean API cost\.Total recorded provider usage is 8,120,408 input tokens, 1,179,317 cached input tokens, 1,540,985 output tokens, and 639,950 reasoning tokens\. These counts are useful for reproducibility and budget analysis but should not be generalized to other models or future pricing\.

## 11Discussion

The main result is not a monotonic “more structure is better” relationship\. A3 occupies the strongest observed correctness/repeatability position: it has the highest answer correctness, the highest stability repeatability, and a mean cost well below A1/A2\. Yet A3 does not have the strongest policy behavior and its unsafe\-or\-invalid rate remains high\. A4 adds structured state and clarification, achieves the lowest mean API cost and almost eliminates false refusals, but it is significantly below A3 on answer correctness and has the highest unsafe\-or\-invalid rate\. A1 is comparatively weak on answer correctness and false refusal, yet it has the best policy\-correct rate and the lowest unsafe\-or\-invalid rate\. A2 yields only a small raw correctness increase over A1, has the weakest repeatability, and a large error surface\.

A plausible architectural interpretation is that direct SQL gives the model a simpler but more permissive formal target, after which strict SQL guards reject unsafe behavior\. A3 removes model\-authored SQL and asks the model to solve a narrower semantic parsing problem; this can reduce some implementation freedom while retaining a compact interface\. A4 introduces additional state and clarification semantics, which may create useful governance capabilities but also more opportunities for an otherwise useful request to become incompatible with a strict contract\. These mechanisms are hypotheses for follow\-up analysis, not causal conclusions established by aggregate rates\.

The bilingual subgroup result is also important\. Every approach is less accurate on pt\-BR than en\-US in the frozen primary set, even though the corpus is balanced by language and passed the release language\-quality gate\. This gap may reflect differences in surface\-form difficulty, lexical mappings, model behavior, or interactions between those factors\. It motivates independent multilingual review and additional model families rather than post\-hoc retuning of the frozen benchmark\.

Finally, the low absolute correctness rates are themselves a result\. Even the best architecture is correct on roughly one quarter of primary cases under the benchmark’s combined clean, adversarial, multi\-turn, and policy\-sensitive distribution\. SemPlan should therefore be read as evidence about comparative architectural trade\-offs under a difficult controlled benchmark, not as certification that any tested design is deployment\-ready\.

## 12Threats to Validity

Internal validity\.Prompt design, strict output contracts, implementation details, provider nondeterminism, retry/recovery behavior, and the F7 execution hotfix lineage can affect comparisons\. The execution hotfix changed failure capture rather than benchmark semantics: database exceptions that previously could crash the runner were converted into typed scientific outcomes, preserving the underlying model/approach failure\. The shared model, database snapshot, cases, and execution limits reduce but do not eliminate implementation asymmetry\.

Construct validity\.Answer correctness is defined by equivalence to a normalized canonical result; it is not a complete measure of user usefulness, explanation quality, or business trust\. Policy correctness and unsafe\-or\-invalid rates cover the benchmark’s specified policy outcomes, not every enterprise governance concern\. Repeatability measures stability of a binary correctness outcome across three executions on a subset and is not equivalent to deterministic generation\. Latency is explicitly excluded from primary claims because its instrumentation assigns zero to captured typed failures\.

External validity\.The study uses one synthetic enterprise domain, one model/provider configuration, two languages, PostgreSQL, and a controlled semantic catalog\. Generalization to other schemas, data distributions, organizations, model families, providers, languages, or production traffic is unknown\. No real customer data are used\.

Statistical conclusion validity\.The case\-aligned paired design improves sensitivity for architecture comparisons, and the primary binary family uses multiplicity correction\. Nevertheless, finite subgroup sizes, correlated template families, and multiple exploratory breakdowns constrain interpretation\. Language, class, and difficulty analyses should not be promoted to new primary hypotheses after observing the data\.

Researcher validity\.A single human researcher designed the project and served as the primary gold reviewer\. Automated gold execution, deterministic manifests, review queues, frozen hashes, and public reproducibility artifacts mitigate but do not remove author\-in\-the\-loop bias\. Independent replication and secondary annotation would strengthen future versions\.

## 13Limitations

SemPlan does not fine\-tune models, compare multiple model families, evaluate production user interfaces, or certify deployment safety\. The synthetic domain is designed to exercise enterprise\-analytics semantics but cannot reproduce the full organizational context of real data systems\. The study also does not claim that the four architectures exhaust the design space\. In particular, hybrid systems could combine A3\-style semantic requests with stronger policy enforcement or alternative state models\. Such variants should be evaluated as new, pre\-specified experiments rather than retrofitted into the completed F7 run\.

Raw provider responses are hash\-recorded locally\. Public release of those raw responses should occur only after a final policy/licensing review; derived score records, frozen tables, figures, benchmark data, prompts, schemas, and deterministic generation/reproduction paths can be released independently\. The current paper also avoids using latency as a comparative performance claim because of the instrumentation issue described above\.

## 14Reproducibility and Ethics

All benchmark data are synthetic and independently created\. Code is licensed under Apache\-2\.0; benchmark/dataset documentation is licensed under CC BY 4\.0 unless a file states otherwise\. The release candidate includes schemas, catalog files, frozen prompts, deterministic generators, benchmark manifests, result tables, figures, audit documents, and checksums\. A no\-key validation path can regenerate deterministic data and derived paper artifacts without additional paid provider calls\. The frozen scientific\-result manifest binds raw\-response, result\-record, score\-record, figure, table, and cost\-ledger hashes\.

The software\-development use of generative AI is disclosed on the first page\. No AI system is an author\. Scientific design, validation, interpretation, claims, and publication decisions remain the responsibility of the human author\.

## 15Conclusion

SemPlan Benchmark compares four ways to divide responsibility between an LLM and deterministic software when answering enterprise\-data questions\. Under the frozen bilingual synthetic benchmark, A3 achieves the highest observed primary answer correctness and repeatability, A4 has the lowest mean API cost and false\-refusal rate, and A1 retains the strongest policy\-correct behavior and the lowest unsafe\-or\-invalid rate\. The paired analysis supports a real but modest correctness advantage for A3, while the ambiguity, multi\-turn, and policy results show that additional structure does not yield uniformly better behavior\. These findings argue for evaluating architectural trade\-offs along multiple dimensions rather than ranking systems by a single accuracy number\. Future work should replicate the benchmark with additional model families, independent reviewers, other data domains and languages, and pre\-specified variants that target the observed ambiguity, state, and contract\-failure modes\.

## Artifact Availability

This preprint is archived on Zenodo at DOI[10\.5281/zenodo\.21904872](https://doi.org/10.5281/zenodo.21904872)\. The scientific results are frozen under manifest SHA\-2565f06bc8b\.\.\.6aefb816; full immutable identifiers are listed in Appendix[A](https://arxiv.org/html/2608.13612#A1)\. A public reproducibility package is being deposited separately, and dataset/software identifiers will be linked in future public versions\.

## Appendix AFrozen Experiment Identifiers

Table 9:Key immutable identifiers for the reported experiment\.
## Appendix BFrozen Prompt Hashes

## References

- K\. Affolter, K\. Stockinger, and A\. BernsteinA comparative survey of recent natural language interfaces for databases\.The VLDB Journal\.Note:10\.1007/s00778\-019\-00567\-8External Links:[Link](https://link.springer.com/article/10.1007/s00778-019-00567-8),[Document](https://dx.doi.org/10.1007/s00778-019-00567-8)Cited by:[§1](https://arxiv.org/html/2608.13612#S1.p1.1),[§2](https://arxiv.org/html/2608.13612#S2.p1.1)\.
- Androutsopouloset al\.\(1995\)I\. Androutsopoulos, G\. D\. Ritchie, and P\. ThanischNatural language interfaces to databases \- an introduction\.Natural Language Engineering\.Note:arXiv:cmp\-lg/9503016External Links:[Link](https://arxiv.org/abs/cmp-lg/9503016)Cited by:[§1](https://arxiv.org/html/2608.13612#S1.p1.1),[§2](https://arxiv.org/html/2608.13612#S2.p1.1)\.
- Gebruet al\.\(2021\)T\. Gebru, J\. Morgenstern, B\. Vecchione, J\. W\. Vaughan, H\. Wallach, H\. D\. III, and K\. CrawfordDatasheets for datasets\.Communications of the ACM\.Note:10\.1145/3458723External Links:[Link](https://dl.acm.org/doi/10.1145/3458723),[Document](https://dx.doi.org/10.1145/3458723)Cited by:[§2](https://arxiv.org/html/2608.13612#S2.p4.1)\.
- Guoet al\.\(2019\)J\. Guo, Z\. Zhan, Y\. Gao, Y\. Xiao, J\. Lou, T\. Liu, and D\. ZhangTowards complex text\-to\-sql in cross\-domain database with intermediate representation\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,pp\.4524–4535\.External Links:[Link](https://aclanthology.org/P19-1444/),[Document](https://dx.doi.org/10.18653/v1/P19-1444)Cited by:[§2](https://arxiv.org/html/2608.13612#S2.p2.1)\.
- Liet al\.\(2023\)J\. Li, B\. Hui, G\. Qu, J\. Yang, B\. Li, B\. Li, B\. Wang, B\. Qin, R\. Cao, R\. Geng, N\. Huo, X\. Zhou, C\. Ma, G\. Li, K\. C\. C\. Chang, F\. Huang, R\. Cheng, and Y\. LiCan llm already serve as a database interface? a big bench for large\-scale database grounded text\-to\-sqls\.InAdvances in Neural Information Processing Systems 36 \(Datasets and Benchmarks Track\),Note:arXiv:2305\.03111External Links:[Link](https://arxiv.org/abs/2305.03111),[Document](https://dx.doi.org/10.52202/075280-1835)Cited by:[§1](https://arxiv.org/html/2608.13612#S1.p1.1),[§2](https://arxiv.org/html/2608.13612#S2.p1.1)\.
- OpenAI \(2026a\)OpenAIFunction calling\.Note:official documentationOpenAI API documentationExternal Links:[Link](https://developers.openai.com/api/docs/guides/function-calling)Cited by:[§2](https://arxiv.org/html/2608.13612#S2.p3.1)\.
- OpenAI \(2026b\)OpenAIStructured model outputs\.Note:official documentationOpenAI API documentationExternal Links:[Link](https://developers.openai.com/api/docs/guides/structured-outputs)Cited by:[§2](https://arxiv.org/html/2608.13612#S2.p3.1)\.
- Schicket al\.\(2023\)T\. Schick, J\. Dwivedi\-Yu, R\. Dessi, R\. Raileanu, M\. Lomeli, L\. Zettlemoyer, N\. Cancedda, and T\. ScialomToolformer: language models can teach themselves to use tools\.InNeurIPS,Note:arXiv:2302\.04761External Links:[Link](https://arxiv.org/abs/2302.04761)Cited by:[§2](https://arxiv.org/html/2608.13612#S2.p3.1)\.
- Scholaket al\.\(2021\)T\. Scholak, N\. Schucher, and D\. BahdanauPICARD: parsing incrementally for constrained auto\-regressive decoding from language models\.InEMNLP,Note:10\.18653/v1/2021\.emnlp\-main\.779External Links:[Link](https://aclanthology.org/2021.emnlp-main.779/),[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.779)Cited by:[§2](https://arxiv.org/html/2608.13612#S2.p2.1)\.
- Wanget al\.\(2020\)B\. Wang, R\. Shin, X\. Liu, O\. Polozov, and M\. RichardsonRAT\-sql: relation\-aware schema encoding and linking for text\-to\-sql parsers\.InACL,Note:10\.18653/v1/2020\.acl\-main\.677External Links:[Link](https://aclanthology.org/2020.acl-main.677/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.677)Cited by:[§2](https://arxiv.org/html/2608.13612#S2.p2.1)\.
- Wanget al\.\(2018\)C\. Wang, K\. Tatwawadi, M\. Brockschmidt, P\. Huang, Y\. Mao, O\. Polozov, and R\. SinghRobust text\-to\-sql generation with execution\-guided decoding\.arXiv\.Note:arXiv:1807\.03100External Links:[Link](https://arxiv.org/abs/1807.03100)Cited by:[§2](https://arxiv.org/html/2608.13612#S2.p2.1)\.
- Yaoet al\.\(2023\)S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. CaoReAct: synergizing reasoning and acting in language models\.InICLR,Note:arXiv:2210\.03629External Links:[Link](https://arxiv.org/abs/2210.03629)Cited by:[§2](https://arxiv.org/html/2608.13612#S2.p3.1)\.
- Yuet al\.\(2019a\)T\. Yu, R\. Zhang, H\. Er, S\. Li, E\. Xue, B\. Pang, X\. V\. Lin, Y\. C\. Tan, T\. Shi, Z\. Li, Y\. Jiang, M\. Yasunaga, S\. Shim, T\. Chen, A\. Fabbri, Z\. Li, L\. Chen, Y\. Zhang, S\. Dixit, V\. Zhang, C\. Xiong, R\. Socher, W\. S\. Lasecki, and D\. RadevCoSQL: a conversational text\-to\-sql challenge towards cross\-domain natural language interfaces to databases\.InEMNLP\-IJCNLP,Note:10\.18653/v1/D19\-1204External Links:[Link](https://aclanthology.org/D19-1204/),[Document](https://dx.doi.org/10.18653/v1/D19-1204)Cited by:[§1](https://arxiv.org/html/2608.13612#S1.p1.1),[§2](https://arxiv.org/html/2608.13612#S2.p1.1)\.
- Yuet al\.\(2018\)T\. Yu, R\. Zhang, K\. Yang, M\. Yasunaga, D\. Wang, Z\. Li, J\. Ma, I\. Li, Q\. Yao, S\. Roman, Z\. Zhang, and D\. RadevSpider: a large\-scale human\-labeled dataset for complex and cross\-domain semantic parsing and text\-to\-sql task\.InEMNLP,Note:10\.18653/v1/D18\-1425External Links:[Link](https://aclanthology.org/D18-1425/),[Document](https://dx.doi.org/10.18653/v1/D18-1425)Cited by:[§1](https://arxiv.org/html/2608.13612#S1.p1.1),[§2](https://arxiv.org/html/2608.13612#S2.p1.1)\.
- Yuet al\.\(2019b\)T\. Yu, R\. Zhang, M\. Yasunaga, Y\. C\. Tan, X\. V\. Lin, S\. Li, H\. Er, I\. Li, B\. Pang, T\. Chen, E\. Ji, S\. Dixit, D\. Proctor, S\. Shim, J\. Kraft, V\. Zhang, C\. Xiong, R\. Socher, and D\. RadevSParC: cross\-domain semantic parsing in context\.InACL,Note:10\.18653/v1/P19\-1443External Links:[Link](https://aclanthology.org/P19-1443/),[Document](https://dx.doi.org/10.18653/v1/P19-1443)Cited by:[§1](https://arxiv.org/html/2608.13612#S1.p1.1),[§2](https://arxiv.org/html/2608.13612#S2.p1.1)\.