ReliableTableQA:How Much Supervision Does Reliability Annotation Need?

# ReliableTableQA: How Much Supervision Does Reliability Annotation Need?
Source: [https://arxiv.org/html/2607.20537](https://arxiv.org/html/2607.20537)
Huei\-Chung Hu DOCOMO Innovations, Inc\. heidi\.hu@docomoinnovations\.com &Hsin\-Tai Wu DOCOMO Innovations, Inc\. hwu@docomoinnovations\.com &Koyo Kobayashi NTT DOCOMO, Inc\. kouyou\.kobayashi\.gv@nttdocomo\.com

###### Abstract

We introduce ReliableTableQA, a framework for training an LLM to annotate the*statistical reliability*of tabular QA results—not whether the query is answerable, but whether the computed answer is statistically meaningful\. In real enterprise analytics, a syntactically correct SQL query can return a value that is too small\-sample, too wide\-CI, or too confounded to act on; existing systems answer confidently in all such cases, a failure we quantify as the*Unreliable Confident Answer Rate*\(UCAR\)\. We contribute \(1\) a ten\-category reliability taxonomy \(R1–R10\) covering hazards such as small\-sample aggregates, multiple\-comparison inflation, and distribution\-tail mismatch; \(2\) a program\-first data pipeline that generates 50,000 reliability\-labeled training examples from a context\-free grammar over public retail schemas, with schema\-stratified SFT/GRPO splits; and \(3\) a*controlled study of how much supervision calibrated reliability annotation actually requires*\. We find that a small, schema\-stratified SFT set is remarkably sufficient: 200 examples raise reliability\-flag F1 from 0\.61 to 0\.98 and parse rate from 0\.52 to 1\.00, drive UCAR to zero, and yield a model that generalizes to an*unseen*retail domain \(Rel\-F1 0\.997 on held\-out H&M\)\. Against this strong SFT baseline, GRPO—commonly assumed essential—helps*only*when SFT is under\-trained \(\+0\.06–0\.16 exact\-flag\-set match at 100 examples, in\- and out\-of\-distribution\) and provides no measurable benefit once SFT is adequate, a null result we confirm across a hard compound\-flag slice, a strict exact\-match metric, and out\-of\-distribution evaluation\. Our findings reframe reliability annotation as a*data\-efficiency*problem and delineate precisely when reinforcement fine\-tuning does—and does not—pay off\.

## 1Introduction

Enterprise data analytics relies heavily on tabular question answering: a business user asks a natural\-language question, an LLM generates SQL, the query executes, and an answer is returned\. The dominant evaluation paradigm for this pipeline measures*execution accuracy*—whether the SQL returned the right value\[[18](https://arxiv.org/html/2607.20537#bib.bib1),[8](https://arxiv.org/html/2607.20537#bib.bib2)\]\. This framing treats any executed result as equally trustworthy\.

It is not\. Consider a query computing the average smartphone usage score for customers in a prefecture withn=3n=3matching rows, a top\-prefecture ranking that reverses under Bonferroni correction across 38 simultaneous comparisons, or an aggregate that silently excludes 15% of rows due to null values\. In all three cases, the SQL is correct: it executed, returned a value, and that value is an accurate aggregate of the matching rows\. Yet none of these results is safe to act on\. Current systems—both zero\-shot LLMs and fine\-tuned text\-to\-SQL models—report all three with equal confidence\.

We formalize this failure as the gap between*answerability*and*reliability*\. Prior work on LLM abstention addresses the former: should the model refuse because the data does not exist, the question has no factual answer, or the SQL cannot be written\[[6](https://arxiv.org/html/2607.20537#bib.bib3),[7](https://arxiv.org/html/2607.20537#bib.bib4),[2](https://arxiv.org/html/2607.20537#bib.bib5)\]? We address the orthogonal axis: the SQL ran successfully, the data exists, but*the statistical properties of the result make it misleading*\. We call this mode*answerable\-but\-unreliable*and introduce the*Unreliable Confident Answer Rate*\(UCAR\) to measure it\.

#### Contributions\.

- •Taxonomy\.Ten reliability hazards \(R1–R10\) covering small\-sample aggregates, confidence\-interval width, multiple\-comparison inflation, missing\-data bias, subgroup imbalance, single\-observation dominance, Simpson\-reversal risk, distribution\-tail mismatch, rare\-event proportions, and cross\-tab sparseness\. Each hazard is programmatically detectable from the executed query result\.
- •Pipeline\.A program\-first data generation method that inverts the standard LLM\-paraphrase approach: a context\-free grammar generates structurally diverse SQL queries over real retail schemas, a deterministic profiler assigns multi\-label reliability annotations, and a 3B LLM realizes natural\-language questions on top with embedding\-distance diversity filtering\. This avoids the mode\-collapse failure of naive LLM\-driven generation; Appendix[A](https://arxiv.org/html/2607.20537#A1)gives the grammar, the profiler rules, and a worked example\.
- •When does RL help?A controlled SFT\-size ablation showing that calibrated reliability annotation is largely a*data\-efficiency*problem\. A model that jointly generates a SQL answer*and*a structured reliability assessment is trained with SFT and then GRPO, whose rewards are tied directly to DuckDB\-executable ground truth\. We find a few hundred schema\-stratified SFT examples suffice and transfer to an unseen domain, while GRPO helps*only*when the SFT base is under\-trained and adds nothing once it is adequate—robust to hard slices, a strict exact\-match metric, and out\-of\-distribution evaluation\.

Experiments across one synthetic and two public retail schemas \(Synthetic Customer, Olist Brazilian e\-commerce, Dunnhumby household panel\), a held\-out fourth domain \(H&M\), and harder evaluation slices show that a data\-efficient SFT model drives unreliable confident answers to zero \(UCAR = 0\.000\) and reaches 0\.98 reliability\-flag F1, while zero\-shot and prompt\-only baselines fail at rates of 5\.5% and 2\.3% respectively—and that GRPO’s contribution is confined to the under\-trained regime\. Figure[1](https://arxiv.org/html/2607.20537#S1.F1)illustrates all three components of the ReliableTableQA framework\.

![Refer to caption](https://arxiv.org/html/2607.20537v1/x1.png)Figure 1:Overview of the ReliableTableQA framework\.\(A\) Data generation:a context\-free grammar \(Layer 1\) emits structurally diverse SQL queries, a deterministic profiler \(Layer 2\) assigns R1–R10 reliability labels, deduplication and balancing \(Layer 3\) produce a 50 K labeled pool, and a small LLM \(Layer 4\) realizes natural\-language questions\. SynCust is an abbreviation of Synthetic Customer\.\(B\) Fine\-tuning:Qwen3\-4B\-Instruct is SFT\-trained on a small schema\-stratified subset \(we sweepkkfrom 50 to 1000; GRPO is run atk=100k\{=\}100and200200\), then GRPO\-trained on a disjoint split \(20×20\\timesthe SFT size\) with three DuckDB\-executable reward signals and no reward model\.\(C\) Inference:the fine\-tuned model generates SQL, executes it, raises applicable reliability flags \(R1–R10\), and decides whether to answer or abstain\. Our data\-efficient SFT model achieves UCAR = 0\.000 and Rel\-F1≈\\approx0\.98 across schemas \(0\.997 on the held\-out H&M domain\), with GRPO adding measurable benefit only when SFT is under\-trained\.

## 2Problem Formulation

#### Setting\.

Let𝒯\\mathcal\{T\}be a relational table andqqa natural\-language question\. A tabular QA system produces a responser=\(d,s,𝐟,e\)r=\(d,s,\\mathbf\{f\},e\), where:d∈\{answer,abstain\}d\\in\\\{\\texttt\{answer\},\\texttt\{abstain\}\\\}is a decision,ssis a SQL query,𝐟⊆\{R​1,…,R​10\}\\mathbf\{f\}\\subseteq\\\{R1,\\ldots,R10\\\}is a set of reliability flags, andeeis a natural\-language reliability explanation\.

#### Reliability profiler\.

Givenssand𝒯\\mathcal\{T\}, a deterministic profilerϕ\\phiexecutesssand computes a binary reliability vectorϕ​\(s,𝒯\)∈\{0,1\}10\\phi\(s,\\mathcal\{T\}\)\\in\\\{0,1\\\}^\{10\}, one entry per hazard \(Table[1](https://arxiv.org/html/2607.20537#S3.T1)\)\. A result is*reliable*\(𝐟=∅\\mathbf\{f\}=\\emptyset\) if no hazard fires; otherwise it is*unreliable*with label setL=\{Ri:ϕi=1\}L=\\\{R\_\{i\}:\\phi\_\{i\}=1\\\}\.

#### Evaluation metrics\.

We assess systems on five axes:

- •Rel\-F1: Micro\-F1 of predicted flags𝐟\\mathbf\{f\}against ground\-truth labelsLL, averaged across records\.
- •UCAR: Among records withL≠∅L\\neq\\emptyset, the fraction where the model outputsd=answerd=\\texttt\{answer\},𝐟=∅\\mathbf\{f\}=\\emptyset, andconfidence=high\. Lower is better; 0\.0 means the model never confidently answers an unreliable query without flagging it\.
- •Answer accuracy: Whether the model’s SQL, when executed, produces a result matching the gold SQL’s output\.
- •SQL validity: Whether the model’s SQL executes without error\.
- •Abstention accuracy: Whether the model’s decisionddmatches the ground\-truth abstain/answer label\. The ground\-truth isabstainonly when R1 fires at the hard threshold \(n<5n<5\); queries with5≤n<305\\leq n<30raise R1 as a warning but retainansweras the ground\-truth decision\.

## 3Reliability Taxonomy

Table[1](https://arxiv.org/html/2607.20537#S3.T1)defines the ten reliability hazards\. Each hazard is detectable from the query result without access to external knowledge: only the executed rows, group sizes, and column statistics are required\. This is by design—the system must operate as a post\-execution annotator, not a pre\-execution filter\.

Table 1:The ten reliability hazards \(R1–R10\)\. Each is computed deterministically from the executed query result\. A query can trigger multiple hazards simultaneously\.The taxonomy is grounded in classical statistical practice—these failure modes appear in textbook treatments of small\-sample inference, multiple testing, and exploratory data analysis\[[4](https://arxiv.org/html/2607.20537#bib.bib15),[16](https://arxiv.org/html/2607.20537#bib.bib18)\]\. We formalize them as a*learnable, evaluatable label space*for tabular QA: a model can be trained to predictϕ​\(s,𝒯\)\\phi\(s,\\mathcal\{T\}\)from the user’s question and the profiler statistics, and evaluated by F1 against ground truth\.

## 4Data Pipeline

Constructing a reliability\-labeled training set requires queries that*densely cover*all ten hazard categories, including rare ones such as R3 \(multiple\-comparison inflation\) and R8 \(distribution\-tail mismatch\)\. Naive LLM\-driven question generation fails at this task: prompted to produce diverse analytics questions, models collapse to high\-frequency templates \(e\.g\.,, “how many customers in state X?”\) that over\-represent common patterns and under\-represent rare structural hazards such as R3 and R8\. We invert the dependency \(Appendix[A](https://arxiv.org/html/2607.20537#A1)details the grammar, the deterministic profiler rules, and a worked end\-to\-end example\)\.

#### Layer 1: CFG\-based SQL generation\.

A context\-free grammar over each schema generates SQL queries with combinatorial structural coverage across aggregate type \(AVG, MEDIAN, SUM, MAX, MIN, STDDEV, P90, P10, COUNT, COUNT DISTINCT, RATIO\), group\-by arity \(0, 1, 2\), predicate count \(0–3\), and output cardinality bucket \(1, 2–10, 11–50, 51\+\)\. Literal values \(e\.g\.,, categorical filter values, numeric thresholds\) are sampled from the actual column distributions in each table\. We discard queries with zero\-cardinality results or trivially\-constant outputs\.

#### Layer 2: Reliability profiler\.

Each query is executed against its table via DuckDB\. A deterministic profiler computes the full R1–R10 signal vector from the result set and assigns multi\-label annotations\. The profiler is implemented in pure Python and requires no external APIs \(bootstrap CI usesB=1000B=1000resamples; BH\-FDR uses the Benjamini\-Hochberg procedure\[[1](https://arxiv.org/html/2607.20537#bib.bib16)\]\)\.

#### Layer 3: Deduplication and balancing\.

We collapse queries to structural templates by abstracting numeric and string literals, then sample one representative per template\. Labels with low natural occurrence \(R3, R8\) are oversampled to a minimum of 3,000 instances per label before NL generation\.

#### Layer 4: NL question realization\.

A 3B instruction\-tuned LLM \(Qwen2\.5\-3B\-Instruct\[[12](https://arxiv.org/html/2607.20537#bib.bib17)\]\) generates two natural\-language question variants per query using five persona types \(business analyst, data scientist, product manager, executive, technical operator\)\. A greedy embedding\-distance filter \(all\-MiniLM\-L6\-v2\) removes variants too similar to previously accepted questions within each batch \(cos\>0\.85\\cos\>0\.85\)\.

#### Layer 5: Combination and split\.

We generate 30,000 queries per schema \(Synthetic Customer, Olist, Dunnhumby\), apply NL generation, then sample 50,000 records stratified by schema into the training set\. Three test sets of 500 records each are held out before NL generation and evaluated with template\-based question substitutes\. Table[2](https://arxiv.org/html/2607.20537#S4.T2)summarizes label distributions\.

Table 2:Reliability label distributions per schema in the training set \(50k total\)\. Labels are multi\-hot; rows do not sum to 100%\. R4 is absent from Olist and Dunnhumby because their flattened tables have no missing values in the joined columns\. R8 is naturally rich in the retail schemas due to heavy\-tailed spend distributions\. SynCust is an abbreviation for Synthetic Customer\.

## 5Model

### 5\.1Output Format

The model produces a structured JSON response containing the following fields:decision\(answer/abstain\),sql\(executable DuckDB SQL\),result\(one\-sentence summary\),reliability\_flags\(subset of R1–R10 flag names\),reliability\_explanation\(one–two sentence explanation citing specific statistics\),confidence\(high/medium/low\), andevidence\(key numeric values from the profiler\)\.

The model receives the table description, natural\-language question, and the full profiler statistics block \(group sizes, CI width, null rates, etc\.\) as input context\. The model annotates reliability*given*the profiler statistics, not by recomputing them from raw data\.

### 5\.2Stage 1: Supervised Fine\-Tuning

We fine\-tune Qwen3\-4B\-Instruct\-2507\[[13](https://arxiv.org/html/2607.20537#bib.bib11)\]with LoRA \(rank 16,α=32\\alpha=32, all projection modules\) using TRL’s SFTTrainer\[[17](https://arxiv.org/html/2607.20537#bib.bib12)\]\. The training objective is next\-token prediction on the assistant turn only \(response\-masked loss\)\. Rather than train on the full 50k pool, we draw a small*schema\-stratified*subset of sizekk\(balanced across the ten hazards and the schemas\) and train for 3 epochs at effective batch size 16 \(micro\-batch 2, gradient accumulation 8\)\. The large, diverse pool is precisely what makes this stratification possible— every \(hazard, schema\) cell, including rare ones such as R3 and R8, has enough candidates to sample a balanced subset—so the data\-efficiency result below is a finding about how training data is*selected*, not an argument against generating it\. We reportk=100k\{=\}100\(S1\) andk=200k\{=\}200\(S2\), reaching final train loss1\.161\.16and0\.680\.68respectively\. As Section[6](https://arxiv.org/html/2607.20537#S6)shows, the larger set is what lets the model reliably emit its structured output format; the SFT stage teaches both the format and the deterministic mapping from profiler statistics to reliability flags\.

### 5\.3Stage 2: GRPO Fine\-Tuning

On top of the merged SFT weights we apply Group Relative Policy Optimization \(GRPO\)\[[15](https://arxiv.org/html/2607.20537#bib.bib8)\]and study its contribution as a function of SFT adequacy\. We use three deterministic, DuckDB\-executable reward functions \(no learned reward model\); all are pre\-weighted and summed by TRL:

- •Answer correctness\(\+0\.30\+0\.30\): the generated SQL is executed against the table and its result compared to the gold SQL output using exact match or 1% relative tolerance for scalars\.
- •Reliability\-label F1\(\+0\.25\+0\.25\): micro\-F1 of the predicted R1–R10 flag set against the profiler ground truth\.
- •SQL execution validity\(\+0\.15\+0\.15\): the generated SQL executes without error\.

We train with full\-parameter fine\-tuning \(SFT LoRA merged into base weights, no new adapter\) for 2 epochs on the*GRPO split*—disjoint from the SFT set and20×20\\timesits size \(2,000 records at S1, 5,000 at S2\)—withG=4G\{=\}4generations per prompt, micro\-batch 1, gradient accumulation 8, peak LR5×10−65\\times 10^\{\-6\}, sampling temperature0\.80\.8, and KL coefficientβ=0\.04\\beta\{=\}0\.04\. The SFT checkpoint initializes both the policy and \(a frozen copy as\) the KL reference\. We use vLLM colocate generation \(∼9×\{\\sim\}9\{\\times\}speedup over standard HF generation\); because Qwen3 disables its KV cache under gradient checkpointing \(raising per\-step time from∼9\{\\sim\}9s to∼160\{\\sim\}160s\), we setgradient\_checkpointing=False\. Because the adequate\-SFT base already nears the reliability\-F1 and parse\-rate ceilings, GRPO’s advantage signal there is small—consistent with its negligible effect at S2 \(Section[6\.3](https://arxiv.org/html/2607.20537#S6.SS3)\)\.

## 6Experiments

### 6\.1Setup

Our central question is*how much supervision calibrated reliability annotation requires*, and whether GRPO adds value beyond a well\-trained SFT base\. We therefore evaluate two non\-fine\-tuned baselines and an*SFT\-size sweep*in which both the SFT\-only and the SFT\+GRPO checkpoint are evaluated at each setting:

1. 1\.Zero\-shot: Qwen3\-4B\-Instruct\-2507 with a system prompt that specifies the JSON output format and lists flag names but provides no guidance on when to raise them\.
2. 2\.Prompt\-only: Same base model with the full reliability\-aware system prompt, which describes all ten hazards and their thresholds\.
3. 3\.SFT\(kk\) / SFT\+GRPO\(kk\): models fine\-tuned on a schema\-stratified SFT set of sizekk, then GRPO\-trained on a disjoint set \(×20\\\!\\times\\,20\)\. We reportk=100k\{=\}100\(S1\) andk=200k\{=\}200\(S2\), each evaluated both before and after GRPO \(LoRA merged for SFT; full\-FT for GRPO\)\.

All models use greedy decoding \(do\_sample=False\) and are scored against DuckDB\-executed ground truth\. We evaluate on three in\-distribution schemas \(Synthetic Customer, Olist, Dunnhumby; 500 records each\), and to stress\-test the findings, on \(i\) a*hard slice*of compound\-hazard records \(≥3\\geq 3gold flags,n=349n\{=\}349\) scored with strict exact\-flag\-set match, and \(ii\) a*held\-out fourth domain*\(H&M fashion retail; 500 records\), never seen in training\. SFT runs complete in well under an hour; GRPO runs take 5–14h on a single A100 80GB\.

### 6\.2SFT is Data\-Efficient—Past a Threshold

Table[3](https://arxiv.org/html/2607.20537#S6.T3)reports in\-distribution results \(means across three schemas\) for the baselines and the SFT\-size sweep\.

Table 3:In\-distribution results, means across Synthetic Customer, Olist, and Dunnhumby \(500 records each\)\. UCAR = Unreliable Confident Answer Rate\.*Parse*= fraction of outputs that parse to valid JSON with executable SQL\. The SFT\-only sweep \(k∈\{50,…,1000\}k\\in\\\{50,\\dots,1000\\\}\) brackets a sharp data\-efficiency threshold: parse rate collapses belowk=100k\{=\}100\(0\.150\.15atk=50k\{=\}50,0\.520\.52atk=100k\{=\}100\), saturates byk=150k\{=\}150–200200\(Rel\-F10\.940\.94–0\.980\.98, parse rate≈1\.0\\approx\\\!1\.0\), and is flat out tok=1000k\{=\}1000—an order of magnitude more data adds nothing\.\+GRPO\(run atk=100,200k\{=\}100,200\) partially recovers the under\-fit S1 deficit but is a no\-op at S2\. See Figure[2](https://arxiv.org/html/2607.20537#S6.F2)\.#### A sharp data\-efficiency threshold between 100 and 200 examples\.

At S2 \(200 stratified examples\) the SFT model already reaches Rel\-F10\.9780\.978, answer accuracy0\.7230\.723, parse rate1\.0001\.000, and UCAR0\.0000\.000—beating the strongest non\-fine\-tuned baseline \(zero\-shot\) on every axis\. At S1 \(100 examples\) the same recipe is*under\-trained*: parse rate collapses to0\.5210\.521\(roughly a third of generations are malformed JSON/SQL\), dragging Rel\-F1 to0\.6080\.608and answer accuracy to0\.3500\.350—below zero\-shot\. The failure at S1 is one of*output form*, not reliability reasoning: among records that parse, answer accuracy is0\.650\.65–0\.800\.80, at or above the baselines\. Thus a few hundred well\-chosen examples are sufficient, but a few hundred are also*necessary*—below the threshold the model cannot reliably emit its own output format\. Figure[2](https://arxiv.org/html/2607.20537#S6.F2)plots the full SFT\-size curvek∈\{50,100,150,200,500,1000\}k\\in\\\{50,100,150,200,500,1000\\\}: the transition is sharp, with the knee betweenk=100k\{=\}100and150150\(parse rate0\.52→0\.9990\.52\\\!\\to\\\!0\.999\), andk=200k\{=\}200onward forms a flat plateau—an order of magnitude more data \(k=1000k\{=\}1000\) adds nothing further\.

![Refer to caption](https://arxiv.org/html/2607.20537v1/x2.png)Figure 2:Reliability annotation is data\-efficient: a threshold, not a scaling, phenomenon\.SFT\-only \(no GRPO\) performance versus the number of schema\-stratified SFT exampleskk\(log scale\), macro\-averaged over the three in\-distribution schemas \(Synthetic Customer, Olist, Dunnhumby; 500 records each\)\. All three metrics—output*parse rate*\(valid JSON / SQL\), reliability\-F1, and answer accuracy—rise steeply across the shadedthreshold region: below it the model is under\-trained and emits malformed output \(parse rate0\.150\.15atk=50k\{=\}50and0\.520\.52atk=100k\{=\}100\), which floors every metric\. Byk=150k\{=\}150the parse rate is already0\.9990\.999, and the curve then saturates—k=200k\{=\}200,500500, and10001000form a*plateau*\(parse≥0\.996\\geq 0\.996, Rel\-F10\.980\.98–1\.001\.00\), so an order of magnitude more SFT data buys no further gain\. The knee thus falls*betweenk=100k\{=\}100and150150*: roughly150150stratified examples suffice for calibrated reliability annotation\. This curve is SFT\-only by design; GRPO \(not shown\) helps only in the under\-trained regime left of the threshold and adds nothing once SFT is adequate \(Table[4](https://arxiv.org/html/2607.20537#S6.T4)\)\.
#### Fine\-tuning eliminates unreliable confident answers\.

Both SFT settings drive UCAR to≈0\.000\\approx\\\!0\.000—the model essentially never asserts a high\-confidence answer on an unreliable result without flagging it—versus5\.5%5\.5\\%\(zero\-shot\) and2\.3%2\.3\\%\(prompt\-only\)\. This behavior is instilled by SFT alone and is preserved by GRPO\.

#### Prompt\-only vs\. zero\-shot trade\-off\.

Adding the detailed reliability system prompt improves UCAR \(0\.055→0\.0230\.055\\\!\\to\\\!0\.023\) and abstention accuracy \(0\.649→0\.8660\.649\\\!\\to\\\!0\.866\) but*reduces*answer accuracy \(0\.659→0\.5870\.659\\\!\\to\\\!0\.587\) and SQL validity \(0\.997→0\.9450\.997\\\!\\to\\\!0\.945\): the verbose instructions crowd out SQL\-generation capacity, a trade\-off fine\-tuning resolves\.

### 6\.3When Does GRPO Help?

Table[3](https://arxiv.org/html/2607.20537#S6.T3)already hints at the answer: GRPO lifts the under\-trained S1 model substantially \(Rel\-F1\+0\.059\+0\.059, answer accuracy\+0\.085\+0\.085, parse rate\+0\.081\+0\.081\) but is a no\-op at S2\. To test whether the S2 null result is merely a ceiling artifact of saturated metrics, we re\-score with a strict*exact\-flag\-set match*\(the prediction must equal the gold flag set exactly\) on three increasingly demanding regimes\. Table[4](https://arxiv.org/html/2607.20537#S6.T4)reports the GRPO contribution \(SFT\+GRPO−\-SFT\) at each setting\.

Table 4:GRPO contribution \(exact\-flag\-set\-matchΔ\\Delta= SFT\+GRPO−\-SFT\) at each SFT size, across three evaluation regimes\. GRPO’s benefit is concentrated in the under\-trained regime \(S1\) and*grows*on the hard slice \(\+0\.158\+0\.158\), confirming the slice is discriminative; at S2 it is flat\-to\-negative everywhere, including out\-of\-distribution\.#### GRPO is a low\-data crutch, not an independent capability\.

Two facts establish this\. First, the hard slice is*discriminative*: it amplifies GRPO’s S1 benefit from\+0\.111\+0\.111to\+0\.158\+0\.158exact\-match, so it can detect a real GRPO effect when one exists\. Second, on that same slice the S2 contribution is exactly\+0\.000\+0\.000, and remains≈0\\approx\\\!0out\-of\-distribution \(−0\.004\-0\.004\)\. The S2 null is*not*a ceiling artifact: answer accuracy on the hard slice has ample headroom yet GRPO does not improve it \(0\.5670\.567SFT vs\.0\.5470\.547SFT\+GRPO\), and on the hardest records \(≥4\\geq 4flags\) GRPO mildly*hurts*answer accuracy \(0\.720→0\.6400\.720\\\!\\to\\\!0\.640\)\. Across four independent angles—the standard test, a compound\-hazard slice, a strict metric, and a held\-out domain—the conclusion is consistent:GRPO with executable rewards rescues an under\-trained SFT base but adds nothing, and can slightly hurt, once the base is adequate\.

### 6\.4Domain Generalization, Including Out\-of\-Distribution

A natural worry is that the data\-efficient S2 model merely memorized the three training schemas’ column vocabularies\. It did not\. Table[5](https://arxiv.org/html/2607.20537#S6.T5)shows the S2 SFT model across all three in\-distribution schemas*and*a held\-out fourth domain—H&M fashion retail, whose schema, columns, and value distributions were never seen in training\.

Table 5:The S2 SFT\(200\) model generalizes across domains, including a*held\-out*fourth domain \(H&M\) never seen in training: Rel\-F10\.970\.97–1\.001\.00and perfect parse rate throughout\. The under\-trained S1 model, by contrast, degrades out\-of\-distribution \(parse rate0\.521→0\.3860\.521\\\!\\to\\\!0\.386\), confirming that adequate SFT—not GRPO—is what buys robust generalization\. \(H&M lacks R4/R8 by construction, so its absolute scores are not a*harder*test; the point is that domain transfer does not collapse\.\) SynCust is an abbreviation of Synthetic CustomerThe S2 model holds Rel\-F10\.970\.97–1\.001\.00and perfect parse rate on a domain it never trained on, indicating it learned*general*reliability reasoning rather than schema\-specific pattern matching\. The under\-trained S1 model instead degrades out\-of\-distribution \(parse rate0\.521→0\.3860\.521\\\!\\to\\\!0\.386\): robustness comes from adequate SFT, and—consistent with Section[6\.3](https://arxiv.org/html/2607.20537#S6.SS3)—GRPO adds nothing to it OOD \(Δ=−0\.004\\Delta=\-0\.004\)\.

## 7Related Work

#### LLM abstention and answerability\.

AbstentionBench\[[6](https://arxiv.org/html/2607.20537#bib.bib3)\]introduces a large\-scale benchmark for LLM abstention across four answerability categories \(unknown answers, false premises, subjective questions, outdated information\), finding that reasoning fine\-tuning degrades abstention by 24%\. TrustSQL\[[7](https://arxiv.org/html/2607.20537#bib.bib4)\]and RTS\[[2](https://arxiv.org/html/2607.20537#bib.bib5)\]address abstention at the SQL\-generation layer—refusing to generate SQL when the schema does not support the question\. Our work addresses an orthogonal axis: abstaining or flagging after the query executes, because the*result itself*is statistically unreliable\. A correctly\-generated, successfully\-executed SQL can still trigger R1–R10\.

#### Calibration and uncertainty in text\-to\-SQL\.

Liuet al\.\[[9](https://arxiv.org/html/2607.20537#bib.bib6)\]calibrate model confidence in SQL generation using sub\-clause frequency signals, targeting SQL*correctness*rather than result reliability\. Node\-level uncertainty estimation\[[5](https://arxiv.org/html/2607.20537#bib.bib7)\]estimates per\-AST\-node uncertainty about structural correctness\. In contrast, our calibration signal is data\-external: bootstrap confidence intervals, sample sizes, and multiple\-testing corrections applied to the*executed result*—independent of model internals\.

#### GRPO for reasoning\.

DeepSeek\-Math\[[15](https://arxiv.org/html/2607.20537#bib.bib8)\]introduced GRPO as an efficient alternative to PPO for training mathematical reasoning\. GRPO has since been applied to code generation and SQL verification\[[3](https://arxiv.org/html/2607.20537#bib.bib9),[11](https://arxiv.org/html/2607.20537#bib.bib10)\]\. We apply GRPO with fully executable, DuckDB\-backed reward functions, which sidesteps learned reward model error entirely\. Whereas these works report*unconditional*gains from reinforcement learning on SQL and reasoning tasks, our controlled SFT\-size sweep shows the benefit is*contingent on SFT adequacy*: substantial when the supervised base is under\-trained but negligible once it is sufficient \(Section[6\.3](https://arxiv.org/html/2607.20537#S6.SS3)\)\. This complementsKirichenko and others \[[6](https://arxiv.org/html/2607.20537#bib.bib3)\], who find that reasoning fine\-tuning can*degrade*calibrated abstention—together suggesting that RL post\-training is not a free improvement and should be applied where an SFT\-only model is demonstrably weak\.

#### Reliability in data analytics\.

Statistical software \(e\.g\.,, R, Stata\) mechanically computes CIs and p\-values when explicitly requested, but does not decide which diagnostics to surface or gate answers behind them\. Data quality detection systems \(Holoclean\[[14](https://arxiv.org/html/2607.20537#bib.bib13)\], RAHA\[[10](https://arxiv.org/html/2607.20537#bib.bib14)\]\) flag errors*in the data*; our system flags hazards*in the query result*arising from query design over clean data—a different failure mode\.

## 8Limitations

#### Profiler\-visible statistics\.

Our model receives the profiler statistics block at inference time\. In a production deployment, a reliability profiler must run before the model responds, adding query execution latency\. If the profiler is unavailable, reliability annotation degrades toward the zero\-shot baseline\.

#### Taxonomy coverage\.

R1–R10 covers classical statistical hazards but does not capture causal confounding, temporal autocorrelation, or measurement error\. The taxonomy is a starting point, not an exhaustive catalog\.

#### Answer accuracy metric\.

Multi\-row GROUP BY results are compared using exact row\-set equality after JSON serialization; without an ORDER BY clause, DuckDB’s row ordering is non\-deterministic, which may undercount true matches\. This affects all models equally\.

#### Enterprise schema generalization\.

Our Synthetic Customer evaluation schema is a 249\-row synthetic table used for schema analysis only; no private customer data appears in training or evaluation\. Whether models trained on public retail schemas generalize to proprietary enterprise schemas requires further study\.

#### Metric saturation and evaluation coverage\.

On our standard test sets the adequate\-SFT model nears the reliability\-F1 ceiling, which can mask differences between systems; we therefore also report a strict exact\-flag\-set\-match metric and a hard compound\-hazard slice \(≥3\\geq 3flags\)\. Two hazards are unevenly covered: R4 \(missing\-data bias\) and R8 \(distribution\-tail mismatch\) depend on schema properties \(null rates, skew\) and are absent from some schemas by construction—e\.g\.,, R4 cannot fire on a table with no\>10%\>\\\!10\\%\-null aggregable column—so per\-flag conclusions for them are schema\-limited\.

#### Scope of the out\-of\-distribution test\.

Our held\-out H&M domain probes a genuinely unseen schema and vocabulary, but its single near\-symmetric numeric column makes it somewhat*easier*\(R4/R8 do not arise\), so its absolute scores should be read as evidence of domain transfer rather than of a harder benchmark\. The in\-/out\-of\-distribution GRPO comparison, however, is controlled—same test set, both models—so the null GRPO effect there is not an artifact of test difficulty\.

## 9Conclusion

We introduced ReliableTableQA, a framework for statistical reliability annotation in tabular QA, and used it to ask how much supervision the task actually requires\. SQL correctness and result reliability are independent: an executed result can be factually accurate yet statistically meaningless\. Our program\-first data pipeline avoids LLM\-generation mode collapse by generating structural diversity combinatorially\. Our central empirical finding is that calibrated reliability annotation is largely a*data\-efficiency*problem: a few hundred schema\-stratified SFT examples drive reliability\-F1 to0\.980\.98and UCAR to zero, beat strong prompted baselines, and generalize to a held\-out domain— whereas GRPO with executable rewards, commonly assumed essential, helps only when the SFT base is under\-trained and adds nothing once it is adequate \(confirmed in\-distribution, on a hard compound\-hazard slice, under a strict metric, and out\-of\-distribution\)\. The practical takeaway for practitioners is to invest a modest amount of additional supervised data before reaching for reinforcement learning\. We will release the pipeline code, reliability profiler, and trained models upon acceptance to support further work on reliability\-aware data analytics\.

## Ethics Statement

The Synthetic Customer schema used for development is a synthetic table constructed for analytical pipeline testing; no real customer data is included\. All training and evaluation uses publicly available retail datasets\.

## References

- \[1\]\(1995\)Controlling the false discovery rate: a practical and powerful approach to multiple testing\.Journal of the Royal Statistical Society: Series B \(Methodological\)57\(1\),pp\. 289–300\.External Links:[Document](https://dx.doi.org/10.1111/j.2517-6161.1995.tb02031.x)Cited by:[§4](https://arxiv.org/html/2607.20537#S4.SS0.SSS0.Px2.p1.1)\.
- \[2\]K\. Chen, Y\. Chen, X\. Yu, and N\. Koudas\(2025\)Reliable text\-to\-SQL with adaptive abstention\.Proceedings of the ACM on Management of Data\.External Links:[Document](https://dx.doi.org/10.1145/3709719)Cited by:[§1](https://arxiv.org/html/2607.20537#S1.p3.1),[§7](https://arxiv.org/html/2607.20537#S7.SS0.SSS0.Px1.p1.1)\.
- \[3\]DeepSeek\-AI\(2025\)DeepSeek\-R1: incentivizing reasoning capability in LLMs via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§7](https://arxiv.org/html/2607.20537#S7.SS0.SSS0.Px3.p1.1)\.
- \[4\]A\. Gelman, J\. B\. Carlin, H\. S\. Stern, D\. B\. Dunson, A\. Vehtari, and D\. B\. Rubin\(2013\)Bayesian data analysis\.3 edition,CRC Press\.Cited by:[§3](https://arxiv.org/html/2607.20537#S3.p2.1)\.
- \[5\]H\. Hasson and R\. Guo\(2025\)Node\-level uncertainty estimation in LLM\-generated SQL\.arXiv preprint arXiv:2511\.13984\.Cited by:[§7](https://arxiv.org/html/2607.20537#S7.SS0.SSS0.Px2.p1.1)\.
- \[6\]P\. Kirichenkoet al\.\(2025\)AbstentionBench: reasoning LLMs fail on unanswerable questions\.arXiv preprint arXiv:2506\.09038\.Cited by:[§1](https://arxiv.org/html/2607.20537#S1.p3.1),[§7](https://arxiv.org/html/2607.20537#S7.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2607.20537#S7.SS0.SSS0.Px3.p1.1)\.
- \[7\]G\. Lee, W\. Chay, S\. Cho, and E\. Choi\(2024\)TrustSQL: benchmarking text\-to\-SQL reliability with penalty\-based scoring\.arXiv preprint arXiv:2403\.15879\.Cited by:[§1](https://arxiv.org/html/2607.20537#S1.p3.1),[§7](https://arxiv.org/html/2607.20537#S7.SS0.SSS0.Px1.p1.1)\.
- \[8\]J\. Li, B\. Hui, G\. Qu, J\. Yang, B\. Li, B\. Li, B\. Wang, B\. Qin, R\. Geng, N\. Huo, X\. Zhou, C\. Ma, G\. Li, K\. Chang, F\. Huang, R\. Cheng, and Y\. Li\(2023\)Can LLM already serve as a database interface? A BIg bench for large\-scale database grounded text\-to\-SQLs\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§1](https://arxiv.org/html/2607.20537#S1.p1.1)\.
- \[9\]T\. Liu, S\. Wang, D\. Preotiuc\-Pietro, Y\. Chandarana, and C\. Gupta\(2025\)Calibrating LLMs for text\-to\-SQL parsing by leveraging sub\-clause frequencies\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,External Links:[Link](https://aclanthology.org/2025.emnlp-main.859/)Cited by:[§7](https://arxiv.org/html/2607.20537#S7.SS0.SSS0.Px2.p1.1)\.
- \[10\]M\. Mahdavi, Z\. Abedjan, R\. C\. Fernandez, S\. Madden, M\. Ouzzani, M\. Stonebraker, and N\. Tang\(2019\)Raha: a configuration\-free error detection system\.InProceedings of the 2019 International Conference on Management of Data,pp\. 865–882\.External Links:[Document](https://dx.doi.org/10.1145/3299869.3324956)Cited by:[§7](https://arxiv.org/html/2607.20537#S7.SS0.SSS0.Px4.p1.1)\.
- \[11\]M\. Pourreza, S\. Talaei, R\. Sun, X\. Wan, H\. Li, A\. Mirhoseini, A\. Saberi, and S\. O\. Arik\(2025\)Reasoning\-SQL: reinforcement learning with SQL tailored partial rewards for reasoning\-enhanced text\-to\-SQL\.arXiv preprint arXiv:2503\.23157\.Cited by:[§7](https://arxiv.org/html/2607.20537#S7.SS0.SSS0.Px3.p1.1)\.
- \[12\]Qwen Team\(2025\)Qwen2\.5 technical report\.External Links:2412\.15115Cited by:[§4](https://arxiv.org/html/2607.20537#S4.SS0.SSS0.Px4.p1.1)\.
- \[13\]Qwen Team\(2025\)Qwen3 technical report\.External Links:2505\.09388Cited by:[§5\.2](https://arxiv.org/html/2607.20537#S5.SS2.p1.6)\.
- \[14\]T\. Rekatsinas, X\. Chu, I\. F\. Ilyas, and C\. Ré\(2017\)HoloClean: holistic data repairs with probabilistic inference\.Proceedings of the VLDB Endowment10\(11\),pp\. 1190–1201\.External Links:[Document](https://dx.doi.org/10.14778/3137628.3137631)Cited by:[§7](https://arxiv.org/html/2607.20537#S7.SS0.SSS0.Px4.p1.1)\.
- \[15\]Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. Guo\(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§5\.3](https://arxiv.org/html/2607.20537#S5.SS3.p1.1),[§7](https://arxiv.org/html/2607.20537#S7.SS0.SSS0.Px3.p1.1)\.
- \[16\]J\. W\. Tukey\(1977\)Exploratory data analysis\.Addison\-Wesley\.Cited by:[§3](https://arxiv.org/html/2607.20537#S3.p2.1)\.
- \[17\]L\. von Werra, Y\. Belkada, L\. Tunstall, E\. Beeching, T\. Thrush, N\. Lambert, S\. Huang, K\. Rasul, and Q\. Gallouédec\(2020\)TRL: transformer reinforcement learning\.External Links:[Link](https://github.com/huggingface/trl)Cited by:[§5\.2](https://arxiv.org/html/2607.20537#S5.SS2.p1.6)\.
- \[18\]T\. Yu, R\. Zhang, K\. Yang, M\. Yasunaga, D\. Wang, Z\. Li, J\. Ma, I\. Z\. Li, Q\. Yao, S\. Roman, Z\. Zhang, and D\. R\. Radev\(2018\)Spider: a large\-scale human\-labeled dataset for complex and cross\-domain semantic parsing and text\-to\-SQL task\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,External Links:[Document](https://dx.doi.org/10.18653/v1/D18-1425)Cited by:[§1](https://arxiv.org/html/2607.20537#S1.p1.1)\.

## Appendix AProgram\-First Data Generation

This appendix expands the summary in Section[4](https://arxiv.org/html/2607.20537#S4)with the query grammar, the deterministic profiler rules, and a worked end\-to\-end example\. It is intended to make the pipeline reproducible\.

### A\.1Why invert the paraphrase pipeline?

The conventional recipe for synthesizing analytics QA data is*paraphrase\-first*: seed a set of natural\-language questions, have an LLM augment or paraphrase them for diversity, and only then derive the corresponding SQL and any labels\. Two problems follow\. First, the*distribution*of the resulting queries is inherited from the LLM’s generation prior, which collapses to high\-frequency templates \(e\.g\.,, single\-table counts, “how many orders per state?”\); rare*structural*hazards such as R3 \(multiple\-comparison inflation\) and R8 \(distribution\-tail mismatch\) arise from query shapes the model rarely proposes, so they are systematically under\-represented\. Second, the reliability labels must be recovered*after the fact*from text whose executable meaning is no longer explicit\.

We invert the dependency\. The unit of generation is the*program*\(a SQL query drawn from a grammar\), not the sentence\. Structural coverage is therefore controlled by construction \(Appendix[A\.2](https://arxiv.org/html/2607.20537#A1.SS2)\); each query is executed and labeled deterministically, so the labels are ground truth rather than LLM guesses \(Appendix[A\.3](https://arxiv.org/html/2607.20537#A1.SS3)\); and the natural\-language question is*rendered last*, as a surface realization of an item that is already fully specified and labeled\. NL diversity is then free to vary without perturbing the hazard distribution\.

### A\.2Layer 1: the query grammar

Queries are sampled from the following grammar over a single flattened per\-customer/per\-order tableT:

```
query    ::= SELECT [gcols ","] agg FROM T
             [WHERE preds]
             [GROUP BY gcols
              [ORDER BY agg dir [LIMIT k]]]
agg      ::= AVG(c)  | MEDIAN(c) | SUM(c)
           | MAX(c)  | MIN(c)    | STDDEV(c)
           | P90(c)  | P10(c)    | COUNT(*)
           | COUNT(DISTINCT c)   | RATIO(c)
preds    ::= pred {AND pred}
pred     ::= c = v | c IN (v,...) | c > v
           | c < v | c BETWEEN v AND v
           | c IS [NOT] NULL | flag = {0,1}
gcols    ::= cat_col {"," cat_col}     (arity 0-2)
dir      ::= ASC | DESC
```

Each aggregate is restricted to type\-compatible columns \(e\.g\.,,RATIOonly applies to binary flag columns;COUNT\(DISTINCT\)only to categoricals\)\. The query is sampled along four structural dimensions whose marginals are fixed by the weights in Table[6](https://arxiv.org/html/2607.20537#A1.T6); literal values are grounded in the*actual*column distributions \(numeric thresholds sampled uniformly in the empirical\[P5,P95\]\[P\_\{5\},P\_\{95\}\]interval; categorical values drawn from the observed domain, as anIN\-list of three with probability0\.30\.3and an equality otherwise\)\. Every candidate is executed in DuckDB and rejected if it returns zero rows or a trivially constant result, guaranteeing that all retained queries are answerable\.

Table 6:Layer\-1 sampling distribution over structural dimensions\. If a group\-by is present, anORDER BYis added with probability0\.550\.55\(DESC with probability0\.700\.70\), then aLIMIT∈\{3,5,10\}\\in\\\!\\\{3,5,10\\\}with probability0\.500\.50\. Coverage of rare hazards is a property of these marginals, not of an LLM’s prior\.
### A\.3Layer 2: deterministic reliability profiler

Each generated query is executed against its table, and the result set is passed through a fixed rule per hazard—no model is involved, so labels are exact and reproducible\. Table[7](https://arxiv.org/html/2607.20537#A1.T7)lists the firing condition for each R1–R10\. Confidence intervals use a Student\-ttinterval for means,B=1000B\{=\}1000bootstrap resamples for other continuous aggregates, and a Wilson interval for proportions\.

Table 7:Deterministic firing conditions for the ten reliability hazards, as implemented in the profiler\. Labels are multi\-hot; a query may fire several\.
### A\.4Layers 3–4: balancing and NL realization

Layer 3 collapses queries to structural templates by abstracting their literals and keeps one representative per template, then oversamples the naturally rare labels \(R3, R8\) to a floor of3,0003\{,\}000instances each before realization\. Layer 4 renders two natural\-language variants per query with a 3B instruction\-tuned model under one of five analyst personas, discarding any variant whose embedding cosine similarity to an already\-accepted question in the batch exceeds0\.850\.85\. Because the SQL, executed answer, and reliability labels are fixed*before*this step, NL diversity cannot distort the hazard distribution\.

### A\.5A worked example

Consider one query sampled over the Olist schema \(group\-by arity 1, one predicate, a top\-kkranking\):

```
SELECT customer_state, AVG(review_score)
FROM orders
WHERE payment_type = ’voucher’
GROUP BY customer_state
ORDER BY AVG(review_score) DESC
LIMIT 5
```

Vouchers are a minority payment type, so once the data is grouped by state the tail states hold very few orders\. A representative execution returns a top state with mean review4\.64\.6overn=14n\{=\}14voucher orders, a runner\-up at4\.54\.5overn=22n\{=\}22, against a high\-volume state \(SP\) withn≈900n\{\\approx\}900\. The profiler then fires:

- •R1 \(warn\): the smallest group in the result hasn=14<30n\{=\}14<30\.
- •R3: then=14n\{=\}14winner’s wide 95% CI contains the runner\-up’s4\.54\.5, so the top\-1 ranking is not statistically separable\.
- •R5:min/max\\min/\\maxgroup size=14/900≈0\.016<0\.1=14/900\\approx 0\.016<0\.1\.

R4 does*not*fire \(review\_scoreis<1%<\\\!1\\%null\), nor does R8 \(bounded 1–5 scores are not heavy\-tailed\), yielding the multi\-hot label\[R1, R3, R5\]\. Layer 4 then realizes, e\.g\.,, “Among orders paid with a voucher, which five states have the highest average review score?” The training record pairs this question with the SQL, its executed answer, and the reliability annotation\. A paraphrase\-first generator asked for “diverse analytics questions” would rarely propose this rare\-filter top\-kkshape at all, and would have no executable basis from which to derive the\[R1, R3, R5\]labels\.