GROUND: Reducing Hallucinations in LLM-Based Enterprise Analytics Through Governed Semantic Definitions

# GROUND: Reducing Hallucinations in LLM-Based Enterprise Analytics Through Governed Semantic Definitions
Source: [https://arxiv.org/html/2608.26157](https://arxiv.org/html/2608.26157)
Aravind Sasidharan Pillai∗1 ∗1Principal Architect, Data Engineering, Cox Automotive Inc, Foster City, CA, USA

\(July 2026\)

###### Abstract

Natural\-language analytics over enterprise data warehouses is an increasingly important use case for large language models, but production adoption is limited by hallucinated metrics, invalid joins, wrong grain, unsafe data access, and unverified explanations\. Existing text\-to\-SQL approaches ground generation in database schemas or retrieved documentation, whereas enterprise reporting requires grounding in governed business semantics: approved metrics, dimensions, join paths, filters, and row\-level security\. This paper introducesGROUND\(Governed Retrieval Over Unified Normalized Definitions\), which constrains LLM\-generated analytics to a governed semantic layer — supplying approved definitions, binding intent to approved metrics and dimensions under grain and join constraints, and validating every query against schema, metric, join, grain, filter, security, and cost rules before execution, retrying on violations and abstaining when a request is unanswerable\. In the primary synthetic experiment, against three baselines sharing one underlying model — direct schema\-only text\-to\-SQL, schema\-RAG, and semantic\-only grounding — differences isolate the effect of governance context\. GROUND is the only system free of measured hallucinations across all six evaluated categories, while every ungoverned system violates row\-level security on many questions; critically, a semantic\-only condition with exact metric definitions but no access policy still leaks data across tenant boundaries, isolating governance — not metric fidelity — as the property grounding\-as\-context does not supply\. This costs roughly 5×\\timesthe tokens and 1\.3×\\timesthe latency of the direct baseline\. We replicate the findings on real U\.S\. NHTSA vehicle\-safety data with independent hand\-authored gold, and probe GROUND's boundary with an adversarial set across four models from three providers, including an open\-weight model\. GROUND's*enforced*guarantees \(filters, row\-level security\) hold with zero violations on every model — the benefit is not model\-specific — while judgment\-dependent behaviors \(refusing an undefined metric, choosing to clarify\) remain fallible even on the strongest model\. Governance prevents safety\-critical filter and row\-level\-security violations even for weaker models, although result accuracy still tracks raw model capability\.

Keywords:text\-to\-SQL, enterprise analytics, semantic layer, hallucination reduction, business intelligence, retrieval\-augmented generation, data governance, LLMs

## Core contribution

GROUND is a governed semantic\-retrieval and validation framework that constrains LLM\-generated analytics to approved enterprise definitions, reducing schema, metric, join, grain, filter, security, and explanation hallucinations\. We contribute the framework, an open benchmark of 100 governed reporting questions with a deterministic, semantic\-layer\-driven gold\-SQL generator, and a controlled four\-way comparison showing that, in our benchmarks, row\-level security compliance was not achievable from semantic definitions alone — it required an explicit access policy enforced by validation\.

## 1Introduction

Large language models are increasingly embedded in business intelligence workflows so that users can ask natural\-language questions over enterprise data warehouses\. However, enterprise reporting is not a plain schema\-to\-SQL problem\. Correct answers depend on governed business semantics: approved metrics, reporting grain, sanctioned join paths, required filters, row\-level security, and organizational definitions\.

This paper argues that hallucination in enterprise analytics should be treated as agovernance and semantics failure, not only a model\-generation failure\. A model can generate syntactically valid, executable SQL while still using the wrong metric, double\-counting due to grain mismatch, omitting required filters, or ignoring row\-level security\. Such an answer is*worse*than an execution error: it looks authoritative but is silently wrong, and in the security case it is a compliance breach\.

### 1\.1Motivation

- •Natural\-language analytics promises faster self\-service reporting, but incorrect answers create operational and compliance risk\.
- •Raw schema context is insufficient because enterprise tables expose implementation detail rather than business meaning\.
- •Semantic layers provide approved definitions that can ground LLM behavior before query generation\.
- •A benchmark focused on enterprise hallucination*types*evaluates reliability more directly than execution accuracy alone\.

### 1\.2Research questions

IDResearch questionWhere addressedRQ1Does semantic\-layer grounding reduce references to non\-existent tables and columns?Sec\. 8 \(schema\)RQ2Does governed metric retrieval reduce invented or incorrect business formulas?Sec\. 8 \(metric\)RQ3Does join\-path and grain validation reduce double counting and fanout errors?Sec\. 8 \(join, grain\)RQ4Can row\-level security checks prevent unauthorized analytic responses?Sec\. 8 \(security\) — central findingRQ5What latency and cost tradeoffs are introduced by semantic validation and retry?Sec\. 8\.2 \(measured\)
### 1\.3Contributions

- •A hallucination taxonomy for enterprise analytics \(Sec\. 3\)\.
- •GROUND, a governed semantic\-layer retrieval and validation framework for LLM\-based analytics, implemented end to end \(Sec\. 4\)\.
- •A synthetic enterprise benchmark of 100 questions modeled on automotive retail reporting, with a deterministic gold\-SQL generator driven by the semantic layer \(Sec\. 5\)\.
- •A controlled comparison against direct text\-to\-SQL, schema\-RAG, and semantic\-only baselines under an identical model \(Sec\. 6, Sec\. 8\)\.
- •Evaluation metrics including result correctness \(strict and value\-based\), governance compliance, grain correctness, and clarification accuracy, with an automated classifier suite validated by an oracle self\-test \(Sec\. 7\)\.
- •A public release of the benchmark, semantic layers, gold SQL, and evaluation harness — with an oracle self\-test and per\-domain reproduction commands \(Appendix C,github\.com/aravindsp/ground\-benchmark\)\.

## 2Related Work

GROUND builds on work in text\-to\-SQL benchmarking, retrieval\-augmented generation, enterprise schema grounding, semantic layers, and hallucination mitigation\. Its distinction is that it treats enterprise analytics hallucination as a governed\-data\-system failure rather than only a language\-model generation error\.

### 2\.1Text\-to\-SQL benchmarks

Text\-to\-SQL has traditionally been evaluated using cross\-domain semantic parsing benchmarks\. Spider \[1\] introduced a large\-scale complex and cross\-domain benchmark with 10,181 natural\-language questions, 5,693 SQL queries, 200 databases, and 138 domains, requiring models to generalize to unseen schemas and SQL patterns\. BIRD \[2\] extended this line of work toward larger real\-world databases, emphasizing database values, external knowledge, dirty contents, and SQL efficiency across 12,751 text\-to\-SQL pairs and 95 databases\. These benchmarks remain essential for measuring schema generalization and execution accuracy, but they do not make enterprise governance constraints — approved metrics, reporting grain, required filters, and row\-level security — first\-class evaluation dimensions\.

Recent enterprise\-oriented benchmarks narrow this gap\. Spider 2\.0 \[3\] evaluates real\-world enterprise text\-to\-SQL workflows involving complex cloud and local data systems, including BigQuery and Snowflake, and reports a large performance drop compared with Spider 1\.0 and BIRD\. BEAVER \[4\] constructs an enterprise text\-to\-SQL benchmark from real private data warehouses with intricate schemas and analytical queries, and finds that models which excel on public benchmarks perform poorly in these settings\. EntSQL \[5\] further highlights enterprise\-specific challenges, including private warehouse schemas, domain knowledge, business documents, internal metrics, reporting conventions, and organizational rules\. GROUND is complementary to these benchmarks: rather than measuring general enterprise workflow completion, it isolates governed analytics failure modes and scores them separately as schema, metric, join, grain, filter, security, explanation, and cost hallucinations\.

### 2\.2Retrieval\-augmented generation and schema grounding

Retrieval\-augmented generation \[6\] combines parametric language models with retrieved external context, improving factuality and provenance for knowledge\-intensive tasks\. In text\-to\-SQL systems, retrieval is often used for schema linking \[7\], table selection, column descriptions, documentation snippets, and value grounding\. RASL \[8\], for example, decomposes massive enterprise schemas and metadata into indexed semantic units for targeted retrieval, addressing the context\-budget problem of large data catalogs\.

GROUND includes a schema\-RAG baseline but argues that retrieval alone is insufficient for governed analytics\. A retrieved table or column description may help the model choose the correct physical object, yet still fail to enforce the approved metric formula, aggregation grain, required business filters, or user\-specific access policy\. In this sense, GROUND treats retrieval as necessary but not sufficient: retrieved context must be paired with validation and policy enforcement\.

### 2\.3Semantic layers for LLM\-based analytics

Semantic layers and metrics layers define business\-facing abstractions over physical data: approved metrics, dimensions, joins, grains, filters, and access policies\. Recent work has begun to connect semantic layers directly to natural\-language analytics\. Semantic\-layer\-mediated NL2SQL agents \[9\] decouple user intent from physical SQL by reasoning over curated semantic models or intermediate representations and then compiling them into dialect\-specific SQL\.

The closest prior work to ours is \[10\], which benchmarks three frontier models \(Claude Opus 4\.7, Sonnet 4\.6, and GPT\-5\.4\) on 100 questions over a retail warehouse under a paired*schema\-only*vs\.*schema\-plus\-semantics*protocol\. It reports that a 4 KB hand\-authored semantic document improves accuracy by 17–23 points, that with the document the three models become statistically indistinguishable, and that the presence of semantic context — not model choice — accounts for essentially all of the variance; it frames this as a*structural*result, namely that explicit business semantics suppress the dominant text\-to\-SQL error class not by making the model more capable but by changing what it is asked to do\. GROUND shares this premise and extends it along three axes\.

*Context versus enforcement\.*\[10\] supplies semantics ascontext; GROUND additionallyvalidates and enforcesthem\. Its schema\-plus\-semantics condition corresponds precisely to our*semantic\-only*baseline \(Sec\. 6\) — approved definitions in the prompt, no validation loop and no injected access policy\. Our results show that context alone leaves a residual governance gap that grounding\-as\-context cannot close: most sharply row\-level security, which the semantic\-only condition still violates on 35–47% of questions despite perfect metric fidelity\. Only a validation\-and\-retry loop over an injected policy drives that to zero\. This separates two things \[10\] treats as one — supplying semantics and*guaranteeing*them\.

*Aggregate hallucination versus a governance taxonomy\.*\[10\] scores aggregate accuracy and a single paired hallucination rate\. GROUND decomposes the failure into seven governance\-specific types \(schema, metric, join, grain, filter, security, cost\) scored independently, and in particular makesaccess control / row\-level securitya first\-class dimension — a constraint that \[10\] and the enterprise text\-to\-SQL benchmarks \[3, 4, 5\] do not model at all, yet which is the decisive separator in our results\.

*Breadth of evidence\.*We evaluate on synthetic data and on real NHTSA data withindependently hand\-authoredgold \(breaking the shared\-machinery critique\), add an adversarial set that exposes GROUND's own failure boundary, and span four models across three providers including an open\-weight model, with repeated runs\. This lets us refine \[10\]'s "model choice does not matter" into a sharper, testable claim: theenforcedgovernance properties are deterministic and model\-independent \(0\.000±\\pm0\.000 across models and runs\), whereas thejudgment\-dependent properties \(recognizing an undefined metric, choosing to clarify\) remain probabilistic and capability\-bound\. GROUND therefore treats the semantic layer not merely as model context but as an enforceable control plane for SQL generation\.

### 2\.4Governed enterprise analytics systems

A related line of work avoids unconstrained text\-to\-SQL by routing natural\-language intent through governed analytics APIs \[11\]\. Such systems encapsulate business logic, permissions, and visualization constraints behind enterprise APIs rather than allowing a model to freely generate raw SQL\. This is closely aligned with GROUND's motivation: enterprise analytics systems must preserve consistency, auditability, and security, and should not delegate aggregation logic or access control purely to an LLM\.

GROUND differs by remaining within the text\-to\-SQL setting while injecting and validating governance rules at generation time\. This makes it possible to compare direct schema\-only SQL generation, schema\-RAG, semantic\-only grounding, and full governed grounding under the same model and benchmark\.

### 2\.5Hallucination and trustworthy data systems

General hallucination research \[12\] studies fluent but unsupported or factually incorrect model outputs\. In enterprise analytics, hallucination often appears as executable SQL that returns a plausible but invalid business answer\. This form of hallucination is especially risky because the SQL may parse, execute, and produce a clean\-looking number while violating metric definitions, grain, filters, or security constraints\.

GROUND therefore adapts hallucination analysis to structured analytics\. Instead of treating hallucination as a generic textual failure, it defines concrete enterprise analytics hallucination types — schema, metric, join, grain, filter, security, explanation, and cost — and evaluates each category separately\. This taxonomy makes visible failures that execution accuracy alone can miss\.

## 3Problem Definition

Let a user question*q*, issued by a user*u*with access role*r\(u\)*, be answered over a database*D*using a generated SQL query*s*and an optional natural\-language explanation*e*\. In enterprise analytics, correctness requires more than execution:*s*must satisfy schema validity, semantic \(metric\) validity, grain validity, join validity, filter compliance, security compliance for*r\(u\)*, and answer support for*e*\. A trustworthy system must also recognize when*q*isnot answerableunder governance and abstain appropriately\.

### 3\.1Enterprise analytics hallucination taxonomy

Hallucination typeDefinitionExample failureSchemaReferences a non\-existent table or columnUsesdealer\_profit\_summary\.net\_revenuewhen neither existsMetricInvents or changes an approved business formulaDefines customer lifetime value with no governed metricJoinUses an invalid or unapproved join pathJoins vehicle directly to repair\-order lineGrainAggregates at the wrong level or double\-countsCounts repair\-order lines as repair ordersFilterOmits required semantic filtersIncludes warranty/internal work in customer\-pay service revenueSecurityIgnores row\-level access constraintsReturns nationwide dealers to a regional userExplanationProvides an unsupported causal explanationClaims revenue dropped due to staffing without evidenceCostGenerates unnecessarily broad or expensive SQLScans all fact tables without date or scope filters
### 3\.2Desired system behavior

A trustworthy system generates SQL only when the requested metric and dimensions are defined and permitted\. For ambiguous or undefined requests, it asks for clarification, rejects unsupported metrics, or offers approved alternatives instead of inventing formulas\. For requests exceeding the user's permitted scope, it restricts results to the allowed scope rather than widening them\.

## 4The GROUND Framework

GROUND \(Governed Retrieval Over Unified Normalized Definitions\) forces LLM\-generated analytics through a governed semantic layer before execution\.

### 4\.1Framework overview

StageFunctionOutputQuestion understandingClassify intent, metric, dimensions, time scope, ambiguityStructured analytic intentSemantic retrievalAssemble approved metrics, dimensions, join paths, grain rules, required filters, and the user's RLS predicateGrounding packetMetric & grain bindingBind user terms to governed definitions and expected aggregation grainMetric planGoverned SQL generationGenerate SQL using only approved semantic objectsCandidate SQLValidation & retryCheck schema, metric formula, join path, grain, filters, RLS, and cost; on violation, return feedback and regenerateValidated SQL or rejectionAuditable answer generationReturn the answer with the SQL, definitions used, filters applied, and any warningsFinal analytic responseIn our implementation, the grounding packet for a question is the full semantic layer \(metrics, dimensions, join paths, business glossary\) plus the specific RLS predicate for the requesting user, rendered into the model's system prompt\. Generation is constrained to a structured JSON response that is either a SQL query or a typed abstention \(ask\_clarification,reject\_undefined\_metric,reject\_unknown\_table\_or\_metric,reject\_unsupported\_dimension\)\.

### 4\.2Validation layer

The validation layer implements the taxonomy of Sec\. 3 as automated checks over the candidate SQL and the semantic layer:

- •Schema check— every referenced table/column must exist in the physical schema; references to known non\-existent objects \(e\.g\.dealer\_profit\_summary\) are rejected\. CTE names and their derived columns are resolved and excluded\.
- •Metric check— the query must reproduce the approved metric's expression signature and must not name an undefined metric\.
- •Join check— eachON a = bequality must correspond to an approved join edge; forbidden edges \(e\.g\. parts→\\toadvisor\) are flagged\.
- •Grain check— count/average metrics defined over an order grain must useCOUNT\(DISTINCT repair\_order\_id\); a rawCOUNT\(\*\)over the line fact is a fanout error\.
- •Filter check— every required filter in the metric's filter set \(status, customer type, line status, active dealer\) must be present\.
- •Security check— the user's mandatory dealer predicate anddealer\_status = 'ACTIVE'must be present and un\-widened\.
- •Cost check— a time\-scoped question with no calendar/date filter, or aSELECT \*over a fact table, is flagged\.

On any violation, GROUND appends the concrete list of failures to the conversation and asks the model to fix the query or abstain, for up to three revisions\. The same checks are reused by the evaluation harness \(Sec\. 7\), so the system is held to exactly the standard it is scored against\.

### 4\.3Expected advantage

The central hypothesis is that GROUND reduces enterprise hallucinations by moving the LLM from open\-ended SQL synthesis to constrained generation over approved semantic definitions, with an enforcement loop that catches the residual violations grounding alone does not prevent — most importantly row\-level security\.

## 5Synthetic Enterprise Benchmark

The benchmark uses synthetic automotive enterprise reporting data, realistic enough to include multiple grains, five fact tables, a dealer hierarchy, time dimensions, and access policies, while using no confidential production data\.

### 5\.1Schema

TableGrainPurposedim\_dealerone row per dealerdealer, group, region, market, statusdim\_calendarone row per datedate, month, quarter, year attributesdim\_vehicleone row per vehicle model/yearmake, model, model year, segmentdim\_customerone row per customercustomer segment and typedim\_service\_advisorone row per advisoradvisor and dealer assignmentdim\_finance\_productone row per F&I productproduct name and categoryfact\_repair\_orderone row per repair orderRO header: dealer, customer, advisor, close date, statusfact\_repair\_order\_lineone row per RO linelabor/parts amounts and costsfact\_vehicle\_saleone row per vehicle salesale amount, cost, vehicle, customerfact\_parts\_saleone row per parts ticketcounter/wholesale parts transactionsfact\_finance\_product\_saleone row per F&I product saleproduct sale attached to a vehicle saleThe semantic layer defines 10 approved metrics \(e\.g\.service\_revenue,vehicle\_gross\_profit,finance\_gross\_profit,parts\_counter\_revenue,total\_gross\_profit,repair\_order\_count\), each with a formula, base grain, required filter set, and the dimension group it may be sliced by; approved join edges; per\-user RLS predicates for four synthetic users \(national admin, regional/West manager, dealer\-group/Alpha user, single\-rooftop dealer\-101 user\); and a business glossary encoding undefined\-metric, ambiguous\-term, and time\-resolution conventions\.

### 5\.2Benchmark question categories

The benchmark contains 100 questions\. Gold SQL for the 70 questions beyond the hand\-authored seed isgenerated deterministically from the semantic layer, so every answerable question is governance\-consistent by construction \(correct expression, grain, filters, joins, and RLS\)\. Trap questions are authored with an expected\-behavior convention\. Realized coverage against the design targets:

CategoryTargetRealized \(of 100\)Simple metric2522 \(incl\. average/parts\)Group\-by2018Trend1515Multi\-table join159Multi\-fact / comparison1010Ambiguous metric106Security / RLS55Hallucination traps1014Grain / filter / explanation probes—3
### 5\.3Gold behavior

Executable questions carry generated gold SQL \(SQLite and DuckDB dialects\) and a gold\-answer snapshot\. Ambiguous or unsupported questions carry an expected behavior — clarification, metric rejection, unknown\-object rejection, unsupported\-dimension rejection, or security restriction\. An oracle self\-test scoring the gold SQL against the classifier suite yields 100% execution, 100% result accuracy, zero hallucination flags in every category, and 100% clarification accuracy across all 100 questions, confirming that the gold set is internally consistent and that the classifiers do not fire on correct governed queries\.

## 6Experimental Setup

The setup compares GROUND against three simpler systems under the same questions and database state\. All four share the same model; only the governance context differs, so measured differences isolate the effect of grounding\.

SystemInput contextExpected weaknessDirect LLM text\-to\-SQLuser question \+ rawCREATE TABLEschemainvented metrics, weak filters, grain errorsSchema\-RAGuser question \+ keyword\-retrieved table/column descriptionsbetter schema grounding, weak business semanticsSemantic\-onlyuser question \+ approved metric and dimension definitionsimproved metrics, no validation/retry, no RLS enforcementGROUNDfull semantic layer \+ user RLS predicate \+ validation/retryhigher overhead, lower hallucination risk### 6\.1Prompting protocol

All systems useclaude\-opus\-4\-8with adaptive thinking and medium effort, and are constrained to a structured JSON response \(response\_type∈\\in\{sql, abstain\},generated\_sql,abstain\_action,notes\) via a JSON\-schema output format\. Each system receives the question, the requesting user's identity, and the reference date \(2026\-07\-05\) for relative time\-scope resolution\. Only GROUND receives the approved join paths, the user's RLS predicate, and the undefined\-metric / ambiguous\-term / time\-resolution conventions, and only GROUND runs the validate\-and\-retry loop \(up to three revisions\) with SQL validation feedback returned to the model\. Baselines answer in a single pass and are not permitted the retry loop, matching their intended "ungoverned" character\.

### 6\.2Execution environment

Queries execute against a committed SQLite database for in\-repository reproducibility; the benchmark also ships DuckDB DDL and DuckDB\-dialect gold SQL for local warehouse\-style execution\. Cloud\-warehouse \(e\.g\. Snowflake\-compatible\) execution is left to future work\.

### 6\.3Logging

Each answer is logged as a record conforming to a fixed evaluation schema: question ID, system name, generated SQL, execution status and error, result accuracy \(strict and value\-based\), the six boolean hallucination flags plus explanation and cost flags, clarification correctness, and — for RQ5 — end\-to\-end latency, prompt and completion token counts, and, for GROUND, the number of validation\-retry revisions\. Latency and tokens are accumulated across all revisions for GROUND so its reported cost includes the retry loop\.

## 7Evaluation Metrics

MetricDefinitionExecution accuracyFraction of generated SQL queries that execute successfully \(over SQL\-expected questions\)\.Result accuracy \(strict\)Fraction whose result exactly matches the gold answer \(column set and values\)\.Result accuracy \(value\-based\)Fraction whose metric column\(s\) match gold at the correct grain, tolerant of identifier columns and column order — the standard text\-to\-SQL execution\-match notion\.Schema / metric / join / grain / filter / security ratesFraction of questions exhibiting each hallucination type \(lower is better\)\.Clarification accuracyFraction of abstention\-expected questions declined with the correct action\.Latency, token costEnd\-to\-end wall\-clock time per question and prompt/completion token usage \(measured; Sec\. 8\.2\)\.Query\-cost proxyRows in tables the plan full\-scans \(EXPLAIN QUERY PLANon the indexed DB\); measured on NHTSA \(Sec\. 8\.4\)\.The hallucination classifiers combine regex extraction with semantic\-layer validation and are validated by the oracle self\-test of Sec\. 5\.3, which bounds their false\-positive rate to zero on correct queries\.

## 8Results

Results are computed over all 100 benchmark questions \(82 expecting SQL, 18 expecting an abstention\)\. Because all systems share one model, differences reflect governance context alone\.

### 8\.1Synthetic benchmark results

SystemExec\.Result \(strict\)Result \(value\)Any halluc\.SchemaMetricJoinGrainFilterSecurityClarifyDirect LLM0\.9510\.0000\.0000\.790\.000\.010\.120\.150\.780\.780\.889Schema\-RAG0\.6460\.0000\.0000\.540\.000\.010\.070\.150\.530\.530\.722Semantic\-only0\.9880\.4630\.5000\.400\.000\.000\.050\.000\.000\.350\.889GROUND1\.0000\.7930\.9510\.000\.000\.000\.000\.000\.000\.000\.944*Accuracy/clarification columns higher\-is\-better; hallucination and security columns are error rates, lower\-is\-better\. Bold marks the best value per column\.*

GROUND eliminates every measured hallucination category\(RQ1–RQ4\)\. It is the only system with zero errors in all six categories, the only one to execute every query it emits \(1\.00 vs 0\.65–0\.99\), and by far the most accurate \(0\.951 value\-based\)\. Its strict result accuracy \(0\.793\) is lower than its value\-based accuracy chiefly because grouped queries omit an identifier column \(e\.g\.dealer\_id\) that gold includes — a presentation\-convention difference, not a computational error, which the value\-based metric disregards\. Crucially, the value\-based metric barely lifts the baselines \(both remain 0\.000\) and lifts semantic\-only only slightly \(0\.463→\\to0\.500\), because their result errors are substantive — wrong filters and scope change the actual numbers and the number of groups — not cosmetic\. GROUND's few remaining value misses are analyzed in Sec\. 9 and are not governance failures\.

Row\-level security is the sharpest separation and the central finding \(RQ4\)\.Every ungoverned system violates RLS on a large fraction of questions \(Direct LLM 0\.78, Schema\-RAG 0\.53, Semantic\-only 0\.35\); GROUND never does \(0\.00\)\. The semantic\-only condition is the decisive ablation: given exact metric and dimension definitions, it attains perfect metric fidelity, correct grain, and correct required filters — yet still leaks data across tenant boundaries on 35% of questions\. In these benchmarks, row\-level security was not recovered from metric definitions alone; it required an explicit, enforced policy of the kind GROUND supplies\. Semantic grounding improves metric correctness but is insufficient for governance\.

Required filters behave the same way \(RQ2/RQ3\)\.The baselines compute headline metrics such as "service revenue" without the governing scope \(closed, retail, posted, active dealer\), producing filter\-hallucination rates of 0\.53–0\.78; because these filters change the figures, the baselines score 0\.000 result accuracy under*both*metrics\. Semantic\-only and GROUND, which encode the required filter set per metric, both reach 0\.00 filter hallucination\. Grain errors follow the same pattern: the baselines double\-count \(0\.15\) by counting repair\-order lines as orders, while the governed systems use the approved distinct\-count expression \(0\.00\)\.

Abstention improves with governance, but schema\-RAG over\-refuses\.GROUND handles 94% of traps correctly and semantic\-only 89%; the baselines answer more traps with invented objects \(Direct LLM 0\.889, Schema\-RAG 0\.722\)\. Schema\-RAG is a revealing failure mode: lacking the full schema, it frequently cannot locate a requested metric and rejects it as unknown, which both depresses its execution accuracy \(0\.646, the lowest of any system, from malformed queries over partial schemas\) and mis\-classifies answerable questions as traps\. GROUND's one abstention miss \(Q095\) is a mis\-categorization of the abstention*type*, not a governance failure — it still declined to generate SQL \(see Sec\. 9\)\.

### 8\.2Latency and token cost \(RQ5\)

Governance is not free\. The table below reports mean per\-question wall\-clock latency and token usage; for GROUND these include the validation\-retry loop\.

SystemAvg latency \(ms\)Avg prompt tokensAvg completion tokensAvg total tokensRel\. tokensDirect LLM6066177536621401\.0×\\timesSchema\-RAG5043128429015740\.7×\\timesSemantic\-only6207569440961022\.9×\\timesGROUND784410161553107145\.0×\\timesGROUND's cost is dominated by its grounding packet: the full semantic layer plus the user's RLS predicate is a ~10k\-token prompt, roughly 5×\\timesthe direct baseline and 1\.75×\\timesthe semantic\-only baseline\. Its latency is ~1\.3×\\timesthe direct baseline\. The validation\-retry loop is a minor contributor: only 9 of 82 SQL questions triggered a revision, averaging 0\.15 retries per question \(maximum 2\)\. Schema\-RAG is the cheapest system \(it retrieves a subset of tables\) but also the least reliable, underscoring that token thrift and governance trade off directly\. Whether the ~5×\\timestoken cost is acceptable is a deployment decision; prompt caching of the \(static\) semantic layer would substantially reduce the marginal cost per question in production, and is a natural optimization\.

### 8\.3Qualitative examples

\(a\) RLS\- and filter\-constrained query — Q001, "What was service revenue last month?" \(West\-region manager\)\.Direct LLM produced an executable but non\-compliant query with no dealer join, no security predicate, and none of the required business filters:

SELECTSUM\(rol\.customer\_pay\_labor\_amount\+rol\.customer\_pay\_parts\_amount\)ASservice\_revenue

FROMfact\_repair\_order\_linerol

JOINfact\_repair\_orderroONrol\.repair\_order\_id=ro\.repair\_order\_id

JOINdim\_calendarcONro\.close\_date\_key=c\.date\_key

WHEREc\.month\_start\_date=’2026\-06\-01’;

This omitsrepair\_order\_status='CLOSED',customer\_type='RETAIL',line\_status='POSTED',dealer\_status='ACTIVE'\(filter hallucination\) and the West\-region restriction \(security hallucination\), so it returns an over\-broad, cross\-tenant figure\. GROUND produced the fully governed query with all required filters,dim\_dealerjoined, andregion IN \('West'\)enforced\.

\(b\) Grain\-safe repair\-order count — Q003, "Monthly repair order count for 2025\."Direct LLM usedCOUNT\(\*\)over the line fact, counting repair\-order*lines*as orders \(grain hallucination / fanout\)\. GROUND usedCOUNT\(DISTINCT ro\.repair\_order\_id\), the approved order\-grain expression\.

\(c\) Undefined\-metric rejection — Q013, "Show customer lifetime value by dealer\."GROUND rejects withreject\_undefined\_metric, citing thatcustomer\_lifetime\_valueis not an approved semantic\-layer metric\. Baseline refusals on this trap are heuristic rather than grounded: lacking the governed undefined\-metric list, they refuse inconsistently across the trap set \(e\.g\. schema\-RAG rejects several*answerable*questions as unknown\), which is why their clarification accuracy trails GROUND's\.

### 8\.4Real\-data replication \(NHTSA, independent gold\)

A natural objection to Sec\. 8 is that both the data and the gold SQL are synthetic — the gold is even generated from the same semantic layer the classifiers use, so GROUND's near\-perfection could be an artifact\. To test this, we replicate the study onrealU\.S\. NHTSA vehicle\-safety data \(271,718 consumer complaints, plus recalls, investigations, and NCAP safety ratings\) with a governance overlay: eight approved metrics, a manufacturer\-scoped row\-level\-security model \(each OEM analyst sees only their own manufacturer's complaints; a regulator sees all\), a multi\-valuedcomponentsgrain trap, and make/model/year join paths\. Crucially, the 40 gold queries arehand\-authored directly against the database, not generated from the semantic layer, and result accuracy is scored by execution\-match against that independent gold\. Value\-based accuracy credits a system that reproduces gold's ranked values as a superset \(gold caps output at 15 rows\)\.

SystemExec\.Result \(value\)Any halluc\.FilterSecurityClarifyAvg tokensDirect LLM0\.970\.650\.780\.730\.330\.831,320Schema\-RAG0\.850\.620\.680\.630\.250\.831,405Semantic\-only0\.970\.680\.330\.000\.331\.003,298GROUND1\.001\.000\.000\.000\.001\.005,229The synthetic findings replicate on real data\. GROUND again has zero hallucinations in every category, zero row\-level\-security violations, 100% execution, and reproduces the correct values on all 40 questions — including the hardest real\-data cases: manufacturer\-vs\-make RLS \(Ford Motor Companyspans makes FORD and LINCOLN\) and the recursive multi\-valued component grain\-split\. The baselines fail materially on real data: 25–33% of their queries violate the manufacturer RLS scope and 63–73% drop required filters\. The semantic\-only ablation again isolates governance from definitions — with exact metric and dimension definitions it eliminates grain and filter hallucination yet still leaks across manufacturer boundaries on 33% of questions\.That GROUND's near\-perfect governance holds on messy real data with independent gold indicates the effect is not an artifact of the synthetic benchmark or of self\-referential gold\.

Governance also lowers query*cost*, making the Sec\. 3 cost\-hallucination dimension quantitative\. On the indexed NHTSA database we compute a query\-cost proxy — the number of rows in tables the planfull\-scans\(EXPLAIN QUERY PLAN; an indexedSEARCHis ~free\)\. Because every governed query applies a selective, indexed scope filter \(filed\_year,manufacturer, ormake\),no GROUND query full\-scans a fact table: under this proxy every governed query is resolved by an indexedSEARCHrather than a full table scan, so its mean rows\-scanned proxy is 0\. This measures full\-scan avoidance under the query plan, not physical I/O or bytes billed by a warehouse\. The ungoverned baselines, which sometimes omit the scope filter, average7k–32krows scanned per query\. Governed generation therefore produces not only safer but cheaper queries: the same required\-scope filters that prevent security and filter hallucinations also enable partition/index pruning\.

### 8\.5Adversarial stress test across four models

The evaluations so far use questions answerable*within*the governed layer\. To find GROUND's failure boundary — and to test whether the governance benefit is model\-specific — we author40 adversarial questionson the same real NHTSA data and run all four systems underfour models from three providers: Claude Opus 4\.8, Claude Sonnet 5, OpenAI GPT\-5\.2, and the open\-weight Llama\-3\.3\-70B \(GPT and Llama via a routing gateway\)\. The adversarial set is weighted toward*plausible\-but\-undefined*metrics deliberatelynotlisted in the glossary's refuse\-set \(16 of 40, e\.g\. "injury rate", "year\-over\-year change", "complaint severity index", "market share of complaints"\), forcing the system to infer from the metric catalog that they are unapproved; it also includes entity\-name traps \(the data storesCHEVROLET/Chrysler \(FCA US, LLC\), not "Chevy"/"Stellantis"\), compound filters, a multi\-hop query, and row\-level\-security overreach under adversarial phrasing\.

The table reports GROUND \(mean±\\pmstd overthree runsper model\) versus the ungoverned baselines \(single run, averaged across Direct LLM and Schema\-RAG, which behave alike\):

ModelSystemExec\.Result \(value\)Metric halluc\.FilterSecurityClarifyOpus 4\.8baselines0\.950\.500\.550\.450\.110\.76Opus 4\.8GROUND1\.00±\\pm\.000\.86±\\pm\.060\.075±\\pm\.000\.00±\\pm\.000\.00±\\pm\.000\.86±\\pm\.00Sonnet 5baselines0\.950\.710\.590\.450\.110\.64Sonnet 5GROUND0\.97±\\pm\.030\.91±\\pm\.030\.04±\\pm\.010\.00±\\pm\.000\.00±\\pm\.000\.84±\\pm\.03GPT\-5\.2baselines0\.870\.610\.480\.410\.0750\.50GPT\-5\.2GROUND0\.68±\\pm\.000\.60±\\pm\.030\.01±\\pm\.010\.00±\\pm\.000\.00±\\pm\.000\.83±\\pm\.07Llama\-3\.3\-70Bbaselines0\.950\.320\.700\.480\.130\.12Llama\-3\.3\-70BGROUND0\.63±\\pm\.050\.25±\\pm\.030\.04±\\pm\.030\.00±\\pm\.000\.00±\\pm\.000\.71±\\pm\.13The central cross\-provider result: GROUND's*enforced*governance guarantees are model\-independent and deterministic\.For every model — a second provider \(OpenAI\) and a much weaker open\-weight model \(Llama\) included — governed generation yields0\.000±\\pm0\.000 filter hallucination and 0\.000±\\pm0\.000 row\-level\-security violations across all three runs, while the ungoverned baselines leak filters on 41–48% and RLS on 7\.5–13% of questions and hallucinate metrics on 48–70%, on every model\. The finding that "only enforcement delivers governance" therefore does not depend on the underlying model family, and the enforced properties carry no run\-to\-run variance because they are decided by a validation check rather than sampled from the model\.

Two honest nuances sharpen the claim\. First,GROUND is not flawless on the judgment\-dependent properties, which is the point of the adversarial set: even the strongest model fabricates a metric — Opus 4\.8 answers the "year\-over\-year change" question by inventing aLAG\(\)\-window growth formula \(yoy\_change\) instead of refusing an undefined metric, precisely the failure the framework targets, and it does so on*all three runs*\(metric\-hallucination 0\.075±\\pm0\.000, a reproducible failure, not a fluke\)\. Clarification tops out at 0\.86\. These properties rest on the model recognizing an undefined metric absent from the refuse\-list and choosing to clarify; they are enforced only by prompting, not by validation, so unlike the enforced metrics they carry run\-to\-run variance, and that variance grows as the model weakens \(clarification std rises from 0\.00 on Opus to 0\.13 on Llama\)\. Second,governance makes a weak model*safe*, not*smart*\.Under GROUND, Llama commits zero filter or RLS violations across every run, yet its result accuracy is only 0\.25 and its execution 0\.63 — a weak model still cannot reliably*write*correct SQL, and the ~10k\-token governance packet burdens it \(Llama's*ungoverned*execution is 1\.00 but its*governed*execution is 0\.63\)\. GROUND guarantees the safety\-critical properties regardless of model; result quality still tracks raw capability\. Even so, governance sharply improves the ungoverned failure modes on every model — Llama's metric\-hallucination rate falls from 0\.70 \(baselines\) to 0\.04 \(GROUND\), and its correct\-abstention rate rises from 0\.12 to 0\.71\.

### 8\.6Standard\-set replication across models, with variance

To confirm the cross\-provider result on non\-adversarial questions and to quantify run\-to\-run stability, we re\-run the 40 standard NHTSA questions \(Sec\. 8\.4\) under all four models, executing GROUNDthree timesper model \(baselines once\)\. The table reports GROUND as mean±\\pmstandard deviation over the three runs\.

ModelSystemExec\.Result \(value\)Metric h\.FilterSecurityClarifyOpus 4\.8GROUND1\.000±\\pm\.0000\.971±\\pm\.0000\.000\.00±\\pm\.000\.00±\\pm\.001\.00±\\pm\.00Sonnet 5GROUND0\.990±\\pm\.0170\.971±\\pm\.0000\.000\.00±\\pm\.000\.00±\\pm\.001\.00±\\pm\.00GPT\-5\.2GROUND0\.892±\\pm\.0170\.834±\\pm\.0170\.000\.00±\\pm\.000\.00±\\pm\.001\.00±\\pm\.00Llama\-3\.3\-70BGROUND0\.863±\\pm\.0170\.745±\\pm\.0450\.000\.00±\\pm\.000\.00±\\pm\.000\.83±\\pm\.17anybaselines \(1 run\)0\.77–1\.000\.53–0\.680\.25–0\.450\.55–0\.750\.15–0\.350\.00–1\.00The key observation is thevariance structure: GROUND's enforced governance metrics are not merely low butdeterministically zero — 0\.000±\\pm0\.000 filter and security violations across every run and every model\. This is expected and important: those properties are decided by a code\-level validation check, not sampled from the model, so they carry no run\-to\-run variance\. All of GROUND's run\-to\-run variance instead concentrates in the*non\-enforced*dimensions and grows as the model weakens — result accuracy is invariant on the strong models \(Opus/Sonnet 0\.971±\\pm0\.000\) but noisier on the weak one \(Llama 0\.745±\\pm0\.045\), and clarification is perfectly stable except on Llama \(0\.83±\\pm0\.17\), where the model's judgment is least reliable\. The ungoverned baselines, by contrast, violate filters \(0\.55–0\.75\) and RLS \(0\.15–0\.35\) and hallucinate metrics \(0\.25–0\.45\) under every model\. This separates the two kinds of guarantee cleanly:*validated*governance is deterministic and model\-independent;*prompted*judgment is probabilistic and capability\-bound\.

## 9Error Analysis

We analyze where GROUND still falls short of a perfect governed answer\. Across the 100 questions, GROUND has zero hallucinations of any type; its residual imperfections are five value/abstention misses,none of which is a governance failure:

- •Presentation shape \(Q006\)\.For a metric\-comparison question, GROUND returned the correct values in long format \(region, metric, amount\) where the gold answer is wide \(region, vehicle\_gross\_profit, finance\_gross\_profit\)\. Every number is present and correct; the value\-based metric flags it only because the row count and column names differ\.
- •Under\-specified question text \(Q088–Q090, security\)\.GROUND enforced row\-level security correctly on all three \(0 security violations — it restricted to dealer 101, the Alpha group, and West respectively\) but its figures differed from gold because the original question text omitted the reporting year while the gold SQL assumed 2025; GROUND reasonably defaulted to year\-to\-date\. This is a benchmark\-authoring gap, since corrected by adding the explicit year to the three question texts\. It is a useful reminder that a governed system will faithfully resolve an under\-specified request to a defensible default rather than guess the benchmark author's intent\.
- •Abstention type \(Q095\)\.For "show margin by dealer," GROUND abstained withreject\_undefined\_metricwhere the gold behavior isask\_clarification\. GROUND correctly declined to generate SQL for an undefined term; it merely chose a stricter abstention category than the reference, so this counts against clarification accuracy without being a hallucination\.

A resolved case illustrates the semantic layer as the fix surface\.In an earlier iteration, GROUND's one value miss \(Q030, "F&I gross profit for Q4 2025"\) was governance\-clean in every respect but returned an empty result because it filteredquarter\_name = 'Q4 2025'while the column stores'2025\-Q4'\. Adding an explicit time\-resolution convention to the semantic layer \("resolve all time scopes todate\_valueranges, never label\-column equality"\) corrected it, and Q030 is answered correctly in the full run\. The remaining gap between a governed system and a perfect benchmark score therefore lies in data\-value conventions and question/gold specification, not in hallucination — and the semantic layer, not the model, is where such gaps are closed\.

Error classLikely causeFix \(adopted or proposed\)Time\-literal mismatchLabel\-column equality on an unknown string formatSemantic\-layer time\-resolution convention \(adopted; resolves viadate\_valueranges\)Ambiguous business termUser asks for growth/margin/performance without a metricAsk clarification, list approved metrics \(adopted\)Missing metricRequested KPI not in the semantic layerReject and route to metric governance \(adopted\)Complex multi\-fact aggregationFanout risk across fact tablesAggregate F&I per sale before joining \(adopted intotal\_gross\_profit\); consider pre\-aggregated viewsSecurity conflictRequested scope exceeds permissionRestrict to allowed scope, note the restriction \(adopted\)Costly queryNo time or dealer filterFlag; ask for a time range or apply a safe default \(flag adopted\)Baseline error modes are dominated by filter and security omissions \(Sec\. 8\) and, for schema\-RAG, over\-refusal driven by partial schema retrieval\.

## 10Threats to Validity

- •Run variance — partly measured\.GROUND is run three times per model on both real\-data NHTSA sets — the standard set \(Sec\. 8\.6\) and the adversarial set \(Sec\. 8\.5\) — and the enforced governance metrics are deterministic \(0\.000±\\pm0\.000\), with small, capability\-dependent variance confined to the non\-enforced dimensions\. The synthetic evaluation \(Sec\. 8\.1\) and all ungoverned baselines remain single\-run per configuration; broader repeats and more seeds would tighten the non\-enforced error bars further\.
- •Synthetic data and generated gold — partly addressed\.The primary domain is synthetic with gold generated from the semantic layer\. Sec\. 8\.4 addresses this by replicating on real NHTSA data with independently hand\-authored gold and reaching the same conclusions; Sec\. 8\.5 further probes an adversarial set\. The remaining gap is scale: the real\-data and adversarial NHTSA sets are 40 questions each, alongside the 100 synthetic questions\.
- •Governed≠\\neqinfallible\.Sec\. 8\.5 shows GROUND's model\-judgment properties — recognizing an undefined metric that is absent from the glossary refuse\-list, and choosing to clarify a genuinely ambiguous request — are strong but fallible \(it fabricated a year\-over\-year metric on one adversarial question\)\. Only the*validated*properties \(schema, metric formula, join, grain, filter, RLS\) are enforced guarantees\. A deployment should treat the refuse\-list and ambiguity handling as best\-effort, not proof\.
- •Classifier fidelity\.The hallucination classifiers are regex\-plus\-semantic heuristics; the oracle self\-test bounds their false\-positive rate to zero on correct queries, but they may still miss violations on adversarial SQL\.
- •Evaluation alignment\.GROUND's retry loop reuses the same classifiers as the evaluator, a fair but favorable alignment; a fully independent evaluator should probe it\.
- •Cost realism\.Latency and token cost are measured \(Sec\. 8\.2\), and a query\-cost proxy \(rows full\-scanned\) is measured on the indexed NHTSA database \(Sec\. 8\.4\), but on a single machine/model and without prompt caching\. Production latency and marginal token cost would differ, likely lower for GROUND once the static semantic layer is cached; the rows\-scanned proxy is a SQLite\-plan estimate, not a warehouse billing figure\.
- •Scope\.The benchmark targets structured reporting and does not cover unstructured analytics or causal diagnosis\.

## 11Discussion

The results suggest that enterprise LLM analytics should be deployed as agoverned data product, not a generic chatbot over database schemas\. The semantic layer becomes the control plane for business meaning, access, and auditability, and — critically — the enforcement point for row\-level security, which our semantic\-only ablation shows cannot be delegated to the model even when it is given perfect metric definitions\.

Enterprise implications include safer self\-service analytics, reduced analyst review burden, auditable SQL generation \(GROUND returns the query, the definitions used, and the filters applied\), reusable metric governance, and clear escalation paths for undefined KPIs\. The main cost is overhead \(RQ5\): GROUND's grounding packet drives ~5×\\timesthe token usage and ~1\.3×\\timesthe latency of the direct baseline, while the validation\-retry loop is minor \(0\.15 retries per query on average\)\. Because the semantic layer is static across questions, prompt caching should recover much of the token cost in production, and the reliability gain — zero hallucinations and zero security violations versus 35–78% RLS violations for ungoverned systems — is decisive for compliance\-sensitive deployments\. Testing whether smaller models suffice under strong grounding, and whether caching closes the cost gap, are the key open questions for production viability\.

## 12Conclusion

This paper proposes and implements GROUND, a governed semantic\-layer grounding framework for reducing hallucinations in LLM\-based enterprise analytics\. By supplying approved metrics, dimensions, joins, grain rules, filters, and access policies before SQL generation, and validating every query against them with a retry loop, GROUND achieves — across 100 synthetic questions and, in replication, 40 questions on real NHTSA data with independent hand\-authored gold — near\-zero hallucinations and zero row\-level\-security violations, while every ungoverned baseline \(including one with exact metric definitions\) violates row\-level security on 25–78% of questions and drops required filters on 50–73%\. The central lesson is that grounding improves metric correctness but only*enforcement*delivers governance: row\-level security must be an injected, validated policy, not a property hoped for from definitions\. An adversarial stress test sharpens the boundary: GROUND's*enforced*properties \(filters, joins, RLS\) hold even under attack, but the properties that rest on the model's judgment — recognizing an undefined metric absent from the refuse\-list, and choosing to clarify ambiguity — are strong yet fallible, and GROUND does occasionally fabricate a plausible metric\. Governance should therefore be architected so that the safety\-critical guarantees \(security, filters, grain\) are validated, not merely prompted\. This reliability comes at roughly 5×\\timesthe token cost of a direct baseline, a gap that prompt caching of the static semantic layer should largely close\. Immediate future work is to replicate across models and runs, expand the real\-data and adversarial sets, add a query\-cost proxy, and study caching's effect on the cost of governance\.

## 13Appendix A\. Semantic\-Layer YAML Excerpt

The governed definition style used by GROUND\. The full benchmark package includes separate files for metrics, dimensions, joins, RLS rules, and the business glossary\.

metrics:

service\_revenue:

label:ServiceRevenue

base\_fact:fact\_repair\_order\_line

grain:repair\_order\_line

expression:"SUM\(fact\_repair\_order\_line\.customer\_pay\_labor\_amount\+fact\_repair\_order\_line\.customer\_pay\_parts\_amount\)"

time\_column:fact\_repair\_order\.close\_date\_key

required\_filter\_set:service\_customer\_pay\#CLOSED\+RETAIL\+POSTED\+ACTIVE

allowed\_dimension\_group:service\#dealer,region,service\_advisor,vehicle\_make,month,\.\.\.

repair\_order\_count:

grain:repair\_order

\#grainguard:rowscomefromtheLINEfactbutthecountisoverORDERS

expression:"COUNT\(DISTINCTfact\_repair\_order\.repair\_order\_id\)"

required\_filter\_set:service\_customer\_pay

\#rls\_rules\.yaml\(pertest\_user\);glossarytime\-resolutionconvention

users:

user\_west\_manager:

predicate:\{table:dim\_dealer,column:region,op:IN,value:\[West\]\}

time\_resolution:"Filtertimeondim\_calendar\.date\_valuerangesonly;neverlabelequality\."

undefined\_metrics:

customer\_lifetime\_value:"Notdefinedinthesemanticlayer\.Donotinventaformula\."

## 14Appendix B\. Benchmark Question Schema

FieldDescriptionquestion\_idStable benchmark identifier \(Q001–Q100\)\.task\_typeCategory:simple\_metric,group\_by,trend,multi\_table,comparison,ambiguous,security, trap types, etc\.test\_userSynthetic user whose role determines RLS behavior\.questionNatural\-language analytics request\.required\_metricApproved semantic metric\(s\),;\-separated; blank for ambiguous requests\.required\_dimensionExpected grouping dimension\(s\),;\-separated\.time\_scopeExpected time filter, e\.g\.last\_month,Q2 2025,year\_to\_date\.security\_expectationExpected RLS scope, e\.g\.restrict\_to\_user\_scope,all\_active\_dealers,dealer\_101\_only\.expected\_behaviorgenerate\_sql\(variants:\_count\_distinct,\_with\_careful\_grain\),ask\_clarification,reject\_undefined\_metric,reject\_unknown\_table\_or\_metric,reject\_unsupported\_dimension,restrict\_or\_refuse\_scope,return\_data\_only\_or\_request\_diagnostic\_scope\.Example seed question:*"Show customer lifetime value by dealer\."*Expected behavior:reject\_undefined\_metric, becausecustomer\_lifetime\_valueis not defined in the governed semantic layer\.

## 15Appendix C\. Reproducibility and Artifact Availability

The complete benchmark, semantic layers, gold SQL \(both dialects\), system runners, and the evaluation harness are released at[https://github\.com/aravindsp/ground\-benchmark](https://github.com/aravindsp/ground-benchmark)\. All results in this paper can be regenerated from that repository; the committed system outputs \(results/\*\.jsonl\) also let a reader reproduce the*scoring*— including the deterministic filter/RLS numbers — without any API access\.

Environment\.Python 3\.9\. Core dependencies:pandas,pyyaml,duckdb, and the provider SDKsanthropic,openai,google\-genai\(only the runners that call a model need the SDKs and keys; the scorer and self\-test do not\)\.

Data provenance\.The synthetic automotive CSVs are committed underdata/csv/and are the canonical source; they are loaded intodata/ground\_synthetic\.sqlite\. The real\-data domain is derived from public U\.S\. NHTSA vehicle\-safety files \(NHTSA\_Customer\_Complaints/, U\.S\.\-Government public domain\);src/load\_nhtsa\.pyprojects and types them intodata/ground\_nhtsa\.sqlite\(271,718 complaints\) and builds the indexes on which the query\-cost proxy depends\. NHTSA gold \(benchmark\_nhtsa/\) ishand\-authored directly against the database, independently of the semantic layer\.

Reproduction\.

\#1\.Sanity\-\-\-noAPIkeyneeded\(scoresthegoldSQLasanoracle,assertsacleanbill\):

fordinsyntheticnhtsanhtsa\_hard;dopython3src/evaluate\_results\.py\-\-self\-test\-\-domain$d;done

\#2\.Regenerategold\-answersnapshots\(synthetic\):

python3src/evaluate\_gold\_sql\_sqlite\.py

\#3\.Runthefoursystems\(needsANTHROPIC\_API\_KEY;OPEN\_ROUTER\_KEYforthegatewaymodels\):

python3src/run\_systems\.py\-\-all\-\-domainnhtsa\_hard\-\-modelclaude\-opus\-4\-8

python3src/run\_systems\.py\-\-all\-\-domainnhtsa\_hard\-\-model"openai/gpt\-5\.2"\#viagateway

\#4\.Scorearun,oraggregaterun\-to\-runvarianceacrossmodels:

python3src/evaluate\_results\.py\-\-resultsresults/ground\.jsonl\-\-labelground

python3src/score\_variance\.py\-\-domainnhtsa\_hard\-\-modelsopus,sonnet,gpt52,llama\-\-runs3

Models and providers\.Four models across three providers were evaluated \(runs conducted July 2026\): Claude Opus 4\.8 \(claude\-opus\-4\-8\) and Claude Sonnet 5 \(claude\-sonnet\-5\) via the Anthropic API; OpenAI GPT\-5\.2 \(openai/gpt\-5\.2\) and open\-weight Llama\-3\.3\-70B \(deepinfra/meta\-llama/Llama\-3\.3\-70B\-Instruct\-Turbo\) via the Requesty gateway \(https://router\.requesty\.ai/v1\)\. Namespaced model ids \(containing/\) route to the gateway; the runner falls back from strictjson\_schematojson\_objectto plain\-prompt JSON for models that reject structured output\.

Run configuration\.All systems share one model per comparison and use adaptive thinking with structured JSON output\. Only GROUND runs a validate\-and\-retry loop \(default budget 3 retries\)\. GROUND is executed three times per model on the NHTSA standard \(Sec\. 8\.6\) and adversarial \(Sec\. 8\.5\) sets to report mean±\\pmstandard deviation; the synthetic evaluation \(Sec\. 8\.1\) and all baselines are single\-run\.

Determinism note\.GROUND's*enforced*governance metrics \(filter, RLS\) are decided by a code\-level validation check, not sampled from the model, and reproduce exactly \(0\.000±\\pm0\.000\) across runs, models, and machines\. The*judgment*\-dependent metrics \(recognizing an undefined metric, choosing to clarify\) are sampled from hosted models and may drift run\-to\-run and as providers update model snapshots; exact reproduction of those figures is not guaranteed, though the qualitative separation is stable across the four models we tested\.