DI-Bench: Systematically Generating In-Domain Data Intelligence Benchmarks for Enterprise Agents

# DI-Bench: Systematically Generating In-Domain Data Intelligence Benchmarks for Enterprise Agents
Source: [https://arxiv.org/html/2609.05776](https://arxiv.org/html/2609.05776)
Kristen SurraoAffiliation:Shayan Ali Akbar,Omar Alonso,Erwin Cornejo,Yuan Li,Yi ZhangTorpong NitayanontAffiliation:Amazon\.comYupei ZhangAffiliation:Correspondence:[zjiangyu@amazon\.com](mailto:email@domain)

###### Abstract

Evaluating enterprise agents on domain\-specific benchmarks is critical, yet public benchmarks rarely evaluate whether agents can integrate business knowledge with analytical computation, and constructing such benchmarks manually is costly\. We present DI\-Bench, a pipeline for generating realistic benchmarks for data intelligence \(DI\), the practice of extracting insights from large volumes of enterprise data\. To emulate realistic DI tasks that require both computation and knowledge retrieval, DI\-Bench builds an*artifact linkage graph*over data tables, dimensions, metrics, and documents to form questions involving structured data and associated knowledge\. Ground truth answers are derived via query execution, followed by LLM question generation and validation\. Applied to two public datasets, the pipeline produces a 731\-task benchmark covering knowledge retrieval, analytical computation, and rule\-grounded reasoning\. To show the discriminatory capability and difficulty of the benchmark, we evaluate four models, revealing a substantial finding: models achieve only 32% accuracy when doing computational tasks where retrieved business rules modify the computation\.

## 1Introduction

LLM\-based agents are increasingly deployed for enterprise data intelligence \(DI\), answering business questions that require retrieving knowledge, querying relational databases, and applying business\-specific rules\. Consider a business analyst asking: “What percentage of customer\-support tickets met their resolution\-time commitment last month?” A naive agent architecture is not going to be sufficient because simply subtracting each ticket’s opening time from its closing time is not enough\. It must first retrieve the applicable service\-level rules: premium customers have shorter resolution targets, the clock pauses while a ticket is waiting for the customer, and weekends are excluded for plans without 24\-hour support\. The agent must then query ticket histories and customer\-plan data, apply the correct rule to each ticket and compute eligible resolution time\. Answering the question therefore requires combining retrieved business rules with analytical data and computation; ignoring the rules would produce an incorrect result\.

Task TypesConstruction MethodSQLQueryingDocumentQANumeric Computation\(e\.g\., point query, comparison\)Analytical Reasoning\(e\.g\., trend, stats\)DocumentRetrievalDocumentAlters AnswerQ&AGenerationQualityCheckSpider✓✗✓✗✗✗MMBIRD✓✗✓✗✗✗MMSpider 2\.0✓✗✓✗✗✓M\>LMτ\\tau\-bench✗✓✓✗✗✓M\>LMInfiAgent\-DABench‡✗✗✓✓✗✗L\>MM, LFinanceBench✗✓✓✗✓✓T\>MMAutoBencher†✗✓✓✗✗✗LM, LDI\-Bench‡✓✓✓✓✓✓G\>T\>LM, L

Table 1:Comparison of DI\-Bench against benchmarks for analytical / tool\-using agents\. Symbols on benchmark names indicate the released artifact:†\\daggerpipeline only;‡\\ddaggerboth pipeline and fixed dataset; no symbol means dataset only\. The*document retrieval*column indicates whether the agent must retrieve the relevant document\. The*document alters answer*column specifies if the answer correctness relies on the document\. The pair ✗ / ✓ for these two columns \(no retrieval, document needed\) appears when the document affects the answer but is given to the agent up front, rather than retrieved\. Construction method: M = manual, L = LLM, T = template, G = graph; “\>\>” orders components by contribution\.Evaluating agents on this sequence of tasks from knowledge retrieval, data querying, business rule understanding and downstream analytical tasks demands benchmarks that test this full stack, but existing benchmarks each cover only part of it\. Text\-to\-SQL benchmarks like Spider\([Yu et al\., 2018](https://arxiv.org/html/2609.05776#bib.bib1)\)and BIRD\([Li et al\., 2023](https://arxiv.org/html/2609.05776#bib.bib2)\)provide the question and any required knowledge inline, so agents never retrieve from a separate corpus\. Spider 2\.0\([Lei et al\., 2025](https://arxiv.org/html/2609.05776#bib.bib5)\)uses per\-task documents, which specify formulas or thresholds the gold SQL implements, but each task names the document it needs, and the documents are authored for that one task rather than as a corpus the agent must navigate\. Data\-analysis benchmarks such as InfiAgent\-DABench\([Hu et al\., 2024](https://arxiv.org/html/2609.05776#bib.bib6)\)broaden task coverage but operate on a single data table per task with no document corpus\. Document\-grounded financial QA benchmarks\([Islam et al\., 2023](https://arxiv.org/html/2609.05776#bib.bib7);[Chen et al\., 2021](https://arxiv.org/html/2609.05776#bib.bib8)\)require retrieving from filings, but these documents are data sources for numerical values rather than natural language rules that govern computation over a database\. Tool\-use benchmarks likeτ\\tau\-bench\([Yao et al\., 2024](https://arxiv.org/html/2609.05776#bib.bib3)\)test rule\-following on API calls, with rules given in\-context and no analytical computation over data\. Concurrent data\-science and enterprise\-data\-agent benchmarks\([Jing et al\., 2025](https://arxiv.org/html/2609.05776#bib.bib17);[Sahu et al\., 2025](https://arxiv.org/html/2609.05776#bib.bib18);[Ma et al\., 2026](https://arxiv.org/html/2609.05776#bib.bib15);[Yang et al\., 2026](https://arxiv.org/html/2609.05776#bib.bib16);[Abaskohi et al\., 2026](https://arxiv.org/html/2609.05776#bib.bib20);[Mo et al\., 2026](https://arxiv.org/html/2609.05776#bib.bib19)\)address different task types, including modeling, insight discovery, cross\-system integration, end\-to\-end analytics, deep research, and GUI pipeline completion, rather than retrieving and applying business knowledge during analysis\. Table[1](https://arxiv.org/html/2609.05776#S1.T1)summarizes these benchmarks\.

A second line of work addresses the cost of manual benchmark construction\. LLM\-driven generation\([Li et al\., 2025](https://arxiv.org/html/2609.05776#bib.bib9)\)constructs both questions and answers with an LLM, providing no formal guarantee of answer correctness\. In multi\-hop text QA,[Min et al\. \(2019\)](https://arxiv.org/html/2609.05776#bib.bib12)showed that nominally multi\-hop benchmarks were often answerable from a single hop, motivating generation pipelines that structurally guarantee multi\-hop necessity\. 2WikiMultiHopQA\([Ho et al\., 2020](https://arxiv.org/html/2609.05776#bib.bib10)\)composes questions from Wikidata triples via typed graph traversal, while MuSiQue\([Trivedi et al\., 2022](https://arxiv.org/html/2609.05776#bib.bib11)\)uses counterfactual checks to verify that each evidence hop is required\. These multi\-hop text QA efforts, however, operate over unstructured text only: their graphs link entities from a knowledge corpus rather than database tables and rule documents, and verification targets evidence necessity rather than the value of a computed answer\. The latter matters for analytical questions, where a document can be cited yet leave the numeric answer unchanged\. While recent progress in retrieval and text\-to\-SQL has substantially improved individual capabilities, it remains unclear whether agents can correctly integrate retrieved business knowledge into downstream analytical computation\. Moreover, enterprise deployments rely on organization\-specific metrics and business logic that evolve over time and differ substantially across domains\. A benchmark built for one company does not transfer to another, and even within one company it goes stale as definitions are revised\. A team deploying an agent therefore needs to generate a benchmark from its own database and business documents, update the benchmark as metrics and rules change, rather than reuse a static, general\-purpose test set\. This motivates a benchmark generation pipeline that any organization can apply to its own domain\.

We address this gap with DI\-Bench, an end\-to\-end pipeline for generating verifiable benchmarks for data intelligence agents operating over both relational databases and unstructured business knowledge\. By combining programmatic execution, automated validation, and realistic enterprise artifacts, DI\-Bench produces benchmark tasks with deterministic ground truth while reducing manual benchmark construction effort\.

Our contributions are1\. DI\-Bench, a pipeline for generating data intelligence benchmarks from structured databases and business knowledge sources\.2\. Demonstration on a generated benchmarkspanning Brazilian e\-commerce and Czech retail banking, producing 731 validated tasks across nine analysis types that require retrieval, computation, and business\-rule reasoning\.3\. An empirical analysis of enterprise agents\.Across models, DI\-Bench reveals consistent performance degradation on tasks requiring integrating retrieved business rules with analytical computation\.

## 2Problem Setting

DI\-Bench targets questions such as the customer\-support example from Section[1](https://arxiv.org/html/2609.05776#S1): “What percentage of customer\-support tickets met their resolution\-time commitment last month?” This percentage cannot be computed correctly from ticket timestamps alone\. The agent must query ticket histories and customer\-plan data, retrieve the applicable service\-level rules, and apply those rules before aggregating the result\. More generally, DI\-Bench studies settings in which answering a business question requires both query execution over relational data and retrieval\-augmented generation over documents containing computation\-relevant rules\([Lewis et al\., 2020](https://arxiv.org/html/2609.05776#bib.bib21);[Guu et al\., 2020](https://arxiv.org/html/2609.05776#bib.bib22)\)\. We assume two inputs:

1. 1\.Arelational databasewith a known schema, including tables, columns, and foreign keys\. In the running example, the database contains ticket histories and customer service plans\.
2. 2\.Aknowledge base\(KB\) ofrule documentscontaining business definitions, thresholds, and conditional overrides that affect a metric’s computation\. In the running example, the applicable resolution\-time target may depend on the customer’s service plan and weekend\-exclusion rule\. The KB also includes a metric catalog with formulas and sample SQL templates\. In our demonstration, these documents are LLM\-generated augmentations grounded in the database schema \(Appendix[B\.1](https://arxiv.org/html/2609.05776#A2.SS1)\)\.

The critical property is information scattering: no single artifact contains everything needed to answer a question correctly\. For example, the metric catalog defines Customer Satisfaction \(CSAT\) as the share of reviews scoring 4 or 5 stars, but a separate rule document specifies that neutral \(3\-star\) reviews must be excluded from the denominator\. Without retrieving that rule, the computed answer deviates by several percentage points, making each task genuinely discriminative\.

## 3Methodology

![Refer to caption](https://arxiv.org/html/2609.05776v1/figure/di_bench_pipeline_final.png)Figure 1:The DI\-Bench pipeline\. From a database and knowledge base, the pipeline \(1\) constructs an artifact linkage graph connecting metrics to required tables and governing rule documents, sampling connected subgraphs as task skeletons, \(2\) computes gold answers via query execution, \(3\) generates natural language questions backward from the answers, and \(4\) applies automated validation filters to verify each task\.DI\-Bench generates benchmark tasks through five stages: \(1\) construct a graph that links structured data to the unstructured documents that affect its interpretation, \(2\) sample subgraphs from the graph as task skeletons, \(3\) execute SQL and tool\(s\) to compute gold answers, \(4\) generate natural language questions backward from those answers, and \(5\) validate quality\. Figure[1](https://arxiv.org/html/2609.05776#S3.F1)provides an overview\.

### 3\.1Artifact Linkage Graph

To guarantee that each task genuinely tests document retrieval, the dependencies between documents and computations are formalized as a typed*artifact linkage graph*\(ALG\)\. Such tasks that involve both document retrieval and analytical computations are common in real enterprise tasks, but current benchmarks lack reliable methods to generate them\. The graph immediately shows which unstructured documents map to which structured tables and metrics, allowing us to generate tasks that meaningfully involve both skills\.

#### Graph Structure\.

The ALG contains four node types: tables, metrics, rules, and dimensions\. Six edge types capture dependencies between structured data and business knowledge \(Table[2](https://arxiv.org/html/2609.05776#S3.T2)\)\. The key edge is*rule→\\rightarrowmetric*, which records that a document modifies the computation of a metric\.

EdgeMeaningmetric→\\totableMetric formula uses columns from tabletable→\\todimensionTable has column usable as a filter/grouping dimensiontable→\\totableTables can be joinedrule→\\tometricRule document changes this metric’s computed answermetric→\\tometricComposite metric uses another as a componentrule→\\todimensionRule applies only when a dimension takes a specific value

Table 2:Edge types in the artifact linkage graph\. The rule→\\tometric edge is key: it connects unstructured documents to the structured computations they modify\.
#### Construction\.

Edges among structured artifacts are extracted directly from schemas and metric definitions\. Rule\-related edges require interpreting natural language documents; we use an LLM \(DeepSeek V3\.2\) to identify which metrics a document modifies and whether it applies conditionally to specific dimension values\.

#### Imperfect Graph Extraction\.

The pipeline does not assume perfect rule–metric extraction\. Candidate links are evaluated through counterfactual checks during benchmark generation\. Counterfactual checks assess whether the inclusion of the rule changes the SQL or answer\. These checks help filter spurious rule–metric associations and improve benchmark quality\. Imperfect extraction therefore primarily affects coverage rather than correctness \(Appendix[H](https://arxiv.org/html/2609.05776#A8)\)\.

#### Why a graph?

The graph captures dependencies between business documents and the computations they influence, enabling systematic generation of tasks that genuinely require both data access and business knowledge, rather than tasks where a document is nominally attached but does not affect the answer\. Because a rule→\\rightarrowmetric edge is defined by the rule’s causal effect on the computation rather than surface simil arity in wording, the graph selects answer\-changing rules far more reliably than semantic\-similarity or random pairing \(See analysis in Appendix[C](https://arxiv.org/html/2609.05776#A3)\)\.

### 3\.2Subgraph Sampling

The pipeline samples connected subgraphs from the ALG\. Each subgraph provides a task skeleton that fixes the metric, data sources, dimensions, and business rules, but not their concrete values\. Each skeleton yields multiple benchmark tasks by varying filters, time windows, and analysis types\.

#### Enumeration\.

Starting from each metric node, the pipeline expands along graph edges to collect required tables, dimensions, and modifying rule documents\. Valid connected subgraphs are then enumerated or sampled depending on dataset size and target task count, while respecting rule applicability and metric dependencies\.

#### Selection for quality and diversity\.

From the pool of valid candidate subgraphs, the pipeline selects the subset that will be instantiated as tasks, aiming for a set that is diverse in the metrics, rules, and dimensions it covers rather than one that repeats similar combinations\. Selection uses Maximal Marginal Relevance\([Carbonell and Goldstein, 1998](https://arxiv.org/html/2609.05776#bib.bib4), MMR;\), which scores each candidate on semantic coherence, rule coverage, and join complexity, then greedily balancing quality against distance from already\-selected subgraphs\. A post\-selection pass ensures every metric and rule document appears at least once \(Appendix[D](https://arxiv.org/html/2609.05776#A4)\)\.

#### Minimum coverage augmentation\.

The target minimum per analysis type is configurable\. After the initial generation round, the pipeline identifies types below the target, samples additional compatible subgraphs, and reruns question generation, SQL execution, answer enrichment, and validation\. A candidate is retained only if its gold answer can be reproduced by re\-executing its stored SQL\. In our demonstration, this augmentation adds 23 tasks and raises counterfactual, decomposition, and significance testing to at least 10 tasks in each domain\.

#### Example

In the running example from Section[1](https://arxiv.org/html/2609.05776#S1), the sampled subgraph would involve the metric for ticket resolution\-time; the connected rules for premium customers, the clock pausing rule, and weekend exclusion; the tables containing ticket histories and customer service plans; and the resolution\-date dimension used to restrict the computation to the previous month\.

### 3\.3Ground Truth Computation

Since an agent is scored against the gold answer, a reliable benchmark depends on that answer being correct\. Rather than have a model produce it, DI\-Bench computes each gold answer by executing SQL against the real database, making it deterministic and correct by construction\. Each subgraph, after fixing dimension values and the analysis type, yields concrete tasks whose gold answers are computed using the following two steps:

#### Step 1: SQL parameterization\.

Each metric in the catalog has a SQL template with placeholders for dimension filters \(e\.g\.,state,start\_date\), which are substituted with concrete values drawn from the data\. Each rule document is associated with a structured SQL modification, such as an additionalWHEREorHAVINGclause, that encodes its effect on metric computation\. These modifications are generated by LLM with human expert verification\. These modifications are applied to the metric’s base SQL, and the gold answer is produced by executing the query on database\.

#### Step 2: Analysis\-type enrichment\.

Depending on the assigned analysis type \(e\.g\., trend, comparison, ranking\), a post\-processing step transforms the raw SQL result into a structured gold answer\. For instance, a trend task runs the query over consecutive periods to identify direction, while a comparison task computes app\-value from two group distributions\. Nine analysis types are supported \(Appendix[A](https://arxiv.org/html/2609.05776#A1)\), each drawn from a compatibility set determined by the subgraph’s dimension types, where temporal dimensions support trend analysis and spatial or categorical dimensions support comparison and ranking\. In the business analyst example from Section[1](https://arxiv.org/html/2609.05776#S1), we might add on the task of comparing the percentage of customer\-support tickets meeting their resolution\-time commitment last month to the month before that \(a comparison analysis type\)\.

### 3\.4Backward Question Generation

With the gold answer computed, an LLM \(DeepSeek V3\.2\) generates a natural language question that would elicit that answer, following the answer\-conditioned question\-generation paradigm\([Du et al\., 2017](https://arxiv.org/html/2609.05776#bib.bib23)\)\. The prompt provides the metric name, gold answer, analysis type, and required rule documents\. The model produces a question resembling what a business stakeholder would ask, without exposing database internals or hinting at which documents to consult\. Because the answer is already known to be correct, the LLM’s only job is natural phrasing and poor phrasing is caught by validation\.

### 3\.5Automated Validation

Prior to inclusion in the benchmark, each generated question\-answer pair is subjected to a two\-stage validation pipeline:

#### Stage 1: Programmatic Filters\.

The first stage applies deterministic, regex\-based checks for structural defects and content\-level issues without LLM inference\. Checks cover structural integrity \(e\.g\., flagging leakage of internal identifiers and degenerate short questions\), dimension sanity \(e\.g\., rejecting tasks where a datetime field was mistakenly treated as a categorical segmentation\), answer completeness \(e\.g\., verifying that a valid, scorable gold answer is present\), and column name naturalization \(e\.g\., replacing schema identifiers with natural language\)\. A task passes this stage only if zero issues are detected\.

#### Stage 2: LLM Quality Scoring\.

Candidates that clear the programmatic stage are then scored by an LLM judge\([Zheng et al\., 2023](https://arxiv.org/html/2609.05776#bib.bib24);[Liu et al\., 2023](https://arxiv.org/html/2609.05776#bib.bib25)\)\(DeepSeek V3\.2\) across eight binary quality dimensions covering objectivity, leakage, naturalness, relevance, format alignment, correctness, reasoning, and scope consistency\. A task is admitted only if all eight dimensions pass\. Tasks failing only on phrasing\-related dimensions are automatically rewritten and re\-scored before final filtering\. In Appendix[E](https://arxiv.org/html/2609.05776#A5), we report the audit against human review, which shows that admitted tasks from the LLM judge achieve 99% average pass rate across all dimensions on evaluated datasets\.

## 4Experiments

#### Demonstration datasets\.

We apply DI\-Bench on two public relational datasets spanning distinct industries\. The first is the Olist Brazilian E\-Commerce dataset\([Olist and Sionek, 2018](https://arxiv.org/html/2609.05776#bib.bib14)\)consisting of 9 tables \(orders, items, payments, reviews, products, sellers, customers, and geolocation\) covering roughly 100K orders across all 27 Brazilian states\. The second is the PKDD’99 Czech financial dataset\([Berka, 2000](https://arxiv.org/html/2609.05776#bib.bib13)\), modeled as the bank*Česká Banka*: 8 tables \(accounts, clients, dispositions, transactions, orders, loans, cards, and districts\) capturing retail banking activity\. It stores dates as integers and distributes data across more tables requiring additional joins, both of which increase task difficulty\. For each domain we construct a knowledge base of 29 metric definitions and 17 business\-rule documents \(thresholds, exclusions, weighting formulas, and conditional overrides\), from which DI\-Bench builds an artifact linkage graph of 71–76 nodes and 168–191 edges \(Appendix[H](https://arxiv.org/html/2609.05776#A8)\)\. The two domains differ in schema shape, terminology, and rule structure, testing the pipeline’s transferability\. All documents and metrics are fictional and do not represent any real organization\.

#### Benchmark composition\.

DI\-Bench contains 731 tasks: 449 from Brazilian e\-commerce and 282 from Czech retail banking\. These tasks passed the automated validation, which rejects 21% of candidates at Stage 1 and 26% of the remainder at Stage 2\. Minimum\-coverage augmentation contributes 23 of these tasks, including 4 Brazilian decomposition tasks and 19 Czech tasks \(2 counterfactual, 9 decomposition, and 8 significance\-testing\)\. The augmentation raises counterfactual, decomposition, and significance testing to at least 10 instances per domain\. All aggregate and per\-type results below use the complete 731\-task benchmark\.

### 4\.1Setup

To show that DI\-Bench is challenging and can discriminate among model capabilities, we evaluate agents using various models on it\. Agents are equipped with tools for SQL execution and knowledge base retrieval\. The database schema is provided in the system prompt\. Agents interact through the Bedrock Converse API with native tool use and a maximum of 20 turns per task\. We evaluate Claude Opus 4\.6, Claude Haiku 4\.5, DeepSeek V3\.2, and Qwen3\-80B, with temperature set to 0 and maximum output tokens of 4096\. Scoring uses exact match with task\-specific tolerance rules\.

### 4\.2Model Performance on DI\-Bench

ModelBRCZAllTurnsClaude Opus 4\.659\.255\.757\.95\.5DeepSeek V3\.256\.142\.650\.911\.5Claude Haiku 4\.552\.842\.949\.05\.5Qwen3\-80B53\.039\.747\.95\.2Table 3:Agent accuracy \(%\) and efficiency on DI\-Bench\. BR = Brazilian E\-Commerce \(449 tasks\), CZ = Czech Financial \(282 tasks\), 731 total\. Avg\. Turns = mean LLM calls per task \(max 20\)\.Figure 2:Knowledge vs\. Knowledge\-grounded analytical tasks across models\. Models score highest on*Knowledge*tasks, lower once*Analytics*is added, and lowest when a retrieved rule must further modify the computation\. The resulting performance drop reveals a persistent knowledge\-to\-computation gap: retrieving business knowledge is largely solved, but correctly applying that knowledge during computation remains a major challenge\.#### Analysis\.

Opus achieves the highest overall accuracy \(57\.9%\), followed by DeepSeek V3\.2 \(50\.9%\), Haiku \(49\.0%\), and Qwen3\-80B \(47\.9%\)\. The gap between BR and CZ remains approximately 10 percentage points across models, reflecting the Czech dataset’s less familiar banking terminology\. DeepSeek requires more interaction steps \(11\.5 turns on average versus approximately 5 for the other models\), indicating lower tool\-use efficiency\.

#### Knowledge\-to\-Computation Gap\.

We partition tasks into three incremental groups \(Figure[2](https://arxiv.org/html/2609.05776#S4.F2)\):*Knowledge*\(knowledge\-base questions answered through retrieval alone,N=137N=137\);*Knowledge \+ Analytics*\(retrieve a metric definition and compute it over the database,N=419N=419\); and*Knowledge \+ Analytics \+ Rule Modification*\(additionally retrieve and apply a rule that modifies how the metric is computed,N=175N=175\)\. These task types are generated from different subgraph structures\.

Averaged across the four models, accuracy drops from 82% on Knowledge tasks to 50% on Knowledge \+ Analytics and further to 32% on Knowledge \+ Analytics \+ Rule\. This decline reveals a substantial knowledge\-to\-computation gap: all four models can often retrieve the relevant business knowledge, yet struggle to correctly incorporate it into downstream computation\. The trend is consistent across all evaluated models, indicating a systematic limitation that is not captured by retrieval\-only or text\-to\-SQL benchmarks\. Section[5](https://arxiv.org/html/2609.05776#S5)uses matched variants to isolate rule application\.

#### Task type difficulty\.

Accuracy varies by analysis type on the 731\-task benchmark \(Table[4](https://arxiv.org/html/2609.05776#S4.T4)\)\. Ranking and point query remain difficult because every row in the result must match exactly\. Minimum\-coverage augmentation raises counterfactual, decomposition, and significance testing to at least 10 tasks per domain; these results remain descriptive because the cells are still small\. In contrast, tasks requiring only a directional or binary judgment \(threshold, trend, comparison\) score 43–81%\.

Task TypeBR \(449\)CZ \(282\)NAccNAccRanking92145329Knowledge retrieval69876878Threshold crossing81784243Point query54284820Trend analysis7078847Comparison30813355Significance testing32581080Counterfactual1151010Decomposition1001010

Table 4:Task volume and accuracy \(%\\%\) by types, averaged across the four models\. Knowledge retrieval, comparison, threshold, and trend are easier \(binary or single\-fact match\); point query and ranking require exact match across multiple rows; counterfactual and decomposition are hardest, requiring multi\-rule reasoning\.

### 4\.3Failure Analysis

Manual inspection of Claude Opus 4\.6 errors reveals two dominant failure modes\. In the first, the agent retrieves the relevant metric but never incorporates the associated rule knowledge into its SQL, as seen on a query requiring the on\-time\-delivery SLA where the agent omitted the one\-day grace period, producing systematically incorrect defect counts\. In the second, the agent retrieves and acknowledges the correct rule in its reasoning yet executes SQL identical to the naive version, leaving the answer unchanged\. Both modes confirm that the benchmark’s primary signal is not whether the agent can locate the relevant rule but whether it correctly applies it during computation, with additional failure cases discussed in Appendix[K](https://arxiv.org/html/2609.05776#A11)\.

## 5Ablation and Robustness Analyses

ModelRuleOracleNo ruleOpus 4\.635\.037\.353\.6DeepSeek V3\.234\.031\.454\.3Haiku 4\.530\.029\.449\.3Qwen3\-80B31\.032\.754\.3Average32\.532\.752\.9Table 5:Accuracy \(%\) on 175 rule\-grounded tasks\. Task content and answer format are fixed across conditions\.#### Rule application\.

We construct matched variants for 175 rule\-grounded tasks, holding the metric, dimension, time window, analysis type, and answer format fixed\. The results are shown in Table[5](https://arxiv.org/html/2609.05776#S5.T5)\. The standard condition requires retrieving and applying the rule, the oracle condition provides the rule directly, and the no\-rule condition uses the corresponding standard metric\. Oracle access changes average accuracy by only\+0\.2\+0\.2points \(32\.5% to 32\.7%\), whereas removing the rule requirement raises accuracy by 20\.4 points \(32\.5% to 52\.9%\)\. Withinthreshold\_crossing\(N=35N=35\), accuracy likewise rises from 62\.1% to 77\.9% without the rule, indicating that rule application rather than retrieval is the primary bottleneck\.

#### Generator and LLM judge sensitivity\.

On 87 valid paired examples with the same task skeletons, Qwen3\-80B and DeepSeek\-generated questions receive similar mean quality scores under an independent Claude Sonnet 4\.6 LLM judge \(79\.9% versus 79\.6%\)\. Across four validator judges, precision is 94–96% and recall is 79–93% against the same balanced human\-labeled sample\. These checks bound but do not eliminate the shared\-model confound; Appendix[E\.4](https://arxiv.org/html/2609.05776#A5.SS4)provides details\.

## 6Conclusion

We presented DI\-Bench, a transferable pipeline for generating in\-domain benchmarks that evaluate agents on data\-intelligence tasks requiring both knowledge retrieval and analytical computation\. By linking structured databases with unstructured business documents and computing gold answers through programmatic execution, DI\-Bench produces realistic benchmark tasks with deterministic ground truth\. Demonstrated on two public domains, the resulting 731\-task benchmark reveals a substantial knowledge\-to\-computation gap in all four evaluated models: while retrieval\-only tasks reach 82% accuracy, performance drops to 32% when retrieved business knowledge must be incorporated into computation\. These findings suggest that retrieving business knowledge is not the primary challenge for enterprise agents, but correctly grounding it in downstream tasks remains a major bottleneck\. More broadly, DI\-Bench provides a scalable framework for evaluating agentic capability as business logic, metrics, and data evolve\.

## Limitations

The pipeline is demonstrated on two domains \(e\-commerce and banking\); broader validation across additional industries would strengthen the transferability claim\. Future work could evaluate the benchmark under alternative retrieval systems and more complex agent architectures\.

The business\-rule documents and metric catalogs used in our demonstration are synthetic augmentations rather than artifacts from a real enterprise\. We made this design choice because we are not aware of any publicly available dataset that simultaneously provides relational data, business\-rule documentation, and permission for open benchmark release\. The generated artifacts are grounded in the underlying schemas and are designed to simulate realistic business rules, metric definitions, thresholds, exclusions, and conditional overrides\. Consequently, the benchmark is intended to approximate realistic enterprise analytical pipelines rather than reproduce the exact business processes of any specific organization\. Nevertheless, the generated artifacts may not fully capture the ambiguity, inconsistency, evolving governance, and cross\-team dependencies commonly present in production business documentation, which motivates evaluation on more realistic enterprise documentation\.

Additionally, scoring is based on exact match and does not award partial credit for correct reasoning with minor computational errors\. Finally, although the 731\-task benchmark includes at least 10 counterfactual, decomposition, and significance\-testing tasks per domain \(N=21/20/42N\{=\}21/20/42overall\), these cells remain too small for strong inferential comparisons by analysis type\.

## References

- Abaskohiet al\.\(2026\)A\. Abaskohi, T\. Chen, M\. Muñoz\-Mármol, C\. Fox, A\. V\. Ramesh, É\. Marcotte, X\. H\. Lù, N\. Chapados, S\. Gella, C\. Pal,et al\.Drbench: a realistic benchmark for enterprise deep research\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 6727–6795\.Cited by:[§1](https://arxiv.org/html/2609.05776#S1.p2.1)\.
- Berka \(2000\)P\. BerkaGuide to the financial data set\.InPKDD2000 Discovery Challenge,Cited by:[§4](https://arxiv.org/html/2609.05776#S4.SS0.SSS0.Px1.p1.1)\.
- Carbonell and Goldstein \(1998\)J\. Carbonell and J\. GoldsteinThe use of mmr, diversity\-based reranking for reordering documents and producing summaries\.InProceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 335–336\.External Links:[Document](https://dx.doi.org/10.1145/290941.291025)Cited by:[§3\.2](https://arxiv.org/html/2609.05776#S3.SS2.SSS0.Px2.p1.1)\.
- Chenet al\.\(2021\)Z\. Chen, W\. Chen, C\. Smiley, S\. Shah, I\. Borova, D\. Langdon, R\. Moussa, M\. Beane, T\. Huang, B\. R\. Routledge,et al\.Finqa: a dataset of numerical reasoning over financial data\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,pp\. 3697–3711\.Cited by:[§1](https://arxiv.org/html/2609.05776#S1.p2.1)\.
- Duet al\.\(2017\)X\. Du, J\. Shao, and C\. CardieLearning to ask: neural question generation for reading comprehension\.InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1342–1352\.Cited by:[§3\.4](https://arxiv.org/html/2609.05776#S3.SS4.p1.1)\.
- Guuet al\.\(2020\)K\. Guu, K\. Lee, Z\. Tung, P\. Pasupat, and M\. ChangRetrieval augmented language model pre\-training\.InInternational conference on machine learning,pp\. 3929–3938\.Cited by:[§2](https://arxiv.org/html/2609.05776#S2.p1.1)\.
- Hoet al\.\(2020\)X\. Ho, A\. D\. Nguyen, S\. Sugawara, and A\. AizawaConstructing a multi\-hop qa dataset for comprehensive evaluation of reasoning steps\.InProceedings of the 28th International Conference on Computational Linguistics,pp\. 6609–6625\.Cited by:[§1](https://arxiv.org/html/2609.05776#S1.p3.1)\.
- Huet al\.\(2024\)X\. Hu, Z\. Zhao, S\. Wei, Z\. Chai, Q\. Ma, G\. Wang, X\. Wang, J\. Su, J\. Xu, M\. Zhu,et al\.Infiagent\-dabench: evaluating agents on data analysis tasks\.arXiv preprint arXiv:2401\.05507\.Cited by:[§1](https://arxiv.org/html/2609.05776#S1.p2.1)\.
- Islamet al\.\(2023\)P\. Islam, A\. Kannappan, D\. Kiela, R\. Qian, N\. Scherrer, and B\. VidgenFinancebench: a new benchmark for financial question answering\.arXiv preprint arXiv:2311\.11944\.Cited by:[§1](https://arxiv.org/html/2609.05776#S1.p2.1)\.
- Jinget al\.\(2025\)L\. Jing, Z\. Huang, X\. Wang, W\. Yao, W\. Yu, K\. Ma, H\. Zhang, X\. Du, and D\. YuDSBench: how far are data science agents from becoming data science experts?\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 32597–32649\.Cited by:[§1](https://arxiv.org/html/2609.05776#S1.p2.1)\.
- Leiet al\.\(2025\)F\. Lei, J\. Chen, Y\. Ye, R\. Cao, D\. Shin, H\. Su, Z\. Suo, H\. Gao, W\. Hu, P\. Yin,et al\.Spider 2\.0: evaluating language models on real\-world enterprise text\-to\-sql workflows\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 28691–28735\.Cited by:[§1](https://arxiv.org/html/2609.05776#S1.p2.1)\.
- Lewiset al\.\(2020\)P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel,et al\.Retrieval\-augmented generation for knowledge\-intensive nlp tasks\.Advances in neural information processing systems33,pp\. 9459–9474\.Cited by:[§2](https://arxiv.org/html/2609.05776#S2.p1.1)\.
- Liet al\.\(2023\)J\. Li, B\. Hui, G\. Qu, J\. Yang, B\. Li, B\. Li, B\. Wang, B\. Qin, R\. Geng, N\. Huo,et al\.Can llm already serve as a database interface? a big bench for large\-scale database grounded text\-to\-sqls\.Advances in Neural Information Processing Systems36,pp\. 42330–42357\.Cited by:[§1](https://arxiv.org/html/2609.05776#S1.p2.1)\.
- Liet al\.\(2025\)X\. Li, F\. Kaiyom, E\. Liu, Y\. Mai, P\. Liang, and T\. HashimotoAutobencher: towards declarative benchmark construction\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 94357–94386\.Cited by:[§1](https://arxiv.org/html/2609.05776#S1.p3.1)\.
- Liuet al\.\(2023\)Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. ZhuG\-eval: nlg evaluation using gpt\-4 with better human alignment\.InProceedings of the 2023 conference on empirical methods in natural language processing,pp\. 2511–2522\.Cited by:[§3\.5](https://arxiv.org/html/2609.05776#S3.SS5.SSS0.Px2.p1.1)\.
- Maet al\.\(2026\)R\. Ma, S\. Shankar, R\. Chen, Y\. Lin, S\. Zeighami, R\. Ghosh, A\. Gupta, A\. Gupta, T\. Gopal, and A\. G\. ParameswaranCan ai agents answer your data questions? a benchmark for data agents\.arXiv preprint arXiv:2603\.20576\.Cited by:[§1](https://arxiv.org/html/2609.05776#S1.p2.1)\.
- Minet al\.\(2019\)S\. Min, E\. Wallace, S\. Singh, M\. Gardner, H\. Hajishirzi, and L\. ZettlemoyerCompositional questions do not necessitate multi\-hop reasoning\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,pp\. 4249–4257\.Cited by:[§1](https://arxiv.org/html/2609.05776#S1.p3.1)\.
- Moet al\.\(2026\)Y\. Mo, Y\. Bai, D\. Sun, Y\. Shi, Y\. Miao, L\. Chen, and D\. LiEntWorld: a holistic environment and benchmark for verifiable enterprise gui agents\.arXiv preprint arXiv:2601\.17722\.Cited by:[§1](https://arxiv.org/html/2609.05776#S1.p2.1)\.
- Olist and Sionek \(2018\)Olist and A\. SionekBrazilian e\-commerce public dataset by olist\.Kaggle\.External Links:[Link](https://www.kaggle.com/dsv/195341),[Document](https://dx.doi.org/10.34740/KAGGLE/DSV/195341)Cited by:[§4](https://arxiv.org/html/2609.05776#S4.SS0.SSS0.Px1.p1.1)\.
- Sahuet al\.\(2025\)G\. Sahu, A\. Puri, J\. A\. Rodriguez, A\. Abaskohi, M\. Chegini, A\. Drouin, P\. Taslakian, V\. Zantedeschi, A\. Lacoste, D\. Vazquez,et al\.Insightbench: evaluating business analytics agents through multi\-step insight generation\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 4683–4715\.Cited by:[§1](https://arxiv.org/html/2609.05776#S1.p2.1)\.
- Trivediet al\.\(2022\)H\. Trivedi, N\. Balasubramanian, T\. Khot, and A\. SabharwalMuSiQue: multihop questions via single\-hop question composition\.Transactions of the Association for Computational Linguistics10,pp\. 539–554\.Cited by:[§1](https://arxiv.org/html/2609.05776#S1.p3.1)\.
- Yanget al\.\(2026\)Y\. Yang, F\. Lei, Y\. Sun, Y\. Zeng, C\. Lv, J\. Hong, J\. Tian, T\. Qiu, X\. Wang, Y\. Chen,et al\.AIDABench: ai data analytics benchmark\.arXiv preprint arXiv:2603\.15636\.Cited by:[§1](https://arxiv.org/html/2609.05776#S1.p2.1)\.
- Yaoet al\.\(2024\)S\. Yao, N\. Shinn, P\. Razavi, and K\. Narasimhanτ\\tau\-bench: a benchmark for tool\-agent\-user interaction in real\-world domains\.arXiv preprint arXiv:2406\.12045\.Cited by:[§1](https://arxiv.org/html/2609.05776#S1.p2.1)\.
- Yuet al\.\(2018\)T\. Yu, R\. Zhang, K\. Yang, M\. Yasunaga, D\. Wang, Z\. Li, J\. Ma, I\. Li, Q\. Yao, S\. Roman,et al\.Spider: a large\-scale human\-labeled dataset for complex and cross\-domain semantic parsing and text\-to\-sql task\.InProceedings of the 2018 conference on empirical methods in natural language processing,pp\. 3911–3921\.Cited by:[§1](https://arxiv.org/html/2609.05776#S1.p2.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.Advances in neural information processing systems36,pp\. 46595–46623\.Cited by:[§3\.5](https://arxiv.org/html/2609.05776#S3.SS5.SSS0.Px2.p1.1)\.

## Appendix

## Appendix AAnalysis Type Definitions

Each analysis type corresponds to a composition of primitive operators \(Retrieve, Filter, Aggregate, Compare, Rank, Temporal, Test\)\. Table[6](https://arxiv.org/html/2609.05776#A1.T6)defines the 9 types used in DI\-Bench\.

TypeOperatorsExample QuestionGold Answer \(example\)Knowledge retrievalRetWhat constitutes an outlier that should be excluded when calculating the average delivery time?‘‘Delivery duration exceeding sixty \(60\) calendar days is excluded as a data anomaly\.’’Point queryRet→\\rightarrowFil→\\rightarrowAggFor our Q2 2018 performance review, what percentage of transactions were paid with credit cards?\[\{payment\_type: credit\_card, pct: 80\.08\}\]Trend analysisRet→\\rightarrowTmp→\\rightarrowAggShow me the monthly trend of our average delivery time throughout 2017\.\{direction: increasing, total\_change\_pct: 32\.1, peak\_period: 2018\-01\}ComparisonRet→\\rightarrowFil→\\rightarrowAgg→\\rightarrowCmpWhich of these two states — PR or AL — had a faster average delivery time for shipped and delivered orders during Q3 2017?\{group\_a: PR, group\_b: AL, value\_a: 10\.7, value\_b: 28\.52, higher: AL\}RankingRet→\\rightarrowFil→\\rightarrowAgg→\\rightarrowRnkWhich 5 states have the highest number of active sellers?\[\{SP: 1638\}, \{PR: 316\}, \{MG: 220\}, \{SC: 172\}, \{RJ: 175\}\]Significance testingRet→\\rightarrowFil→\\rightarrowAgg→\\rightarrowTstAre the average installment counts for orders paid by credit card and boleto in the first quarter of 2018 statistically significantly different from each other?\{significant: true, p\_value: 0\.0, test\_name: Welch’s t\-test \(two\-tailed\)\}Threshold crossingRet→\\rightarrowFil→\\rightarrowAgg→\\rightarrowCmpDid the average delivery time for customers in Acre exceed 20\.0 days?\{value: 19\.79, threshold: 20\.0, result: FAIL\}CounterfactualRet→\\rightarrowFil→\\rightarrowAgg→\\rightarrowCmpWhat’s the average revenue we generated per unique customer in São Paulo state, assuming we could extend our operations there starting from January 2017?\[\{customer\_state: SP, revenue\_per\_customer: 129\.41\}\]DecompositionRet→\\rightarrowFil→\\rightarrowAgg→\\rightarrowGrpCan you break down the Order Defect Rate by seller state for the first half of 2018, showing which geographic areas are contributing the most to our overall defect rate?\[\{BA: 18\.18\}, \{CE: 0\.0\}, \{DF: 13\.21\}, \{ES: 18\.18\}, …\]Table 6:Analysis type definitions with operator compositions, example questions, and a representative gold answer \(Ret = Retrieve, Fil = Filter, Agg = Aggregate, Tmp = Temporal, Cmp = Compare, Rnk = Rank, Tst = Test, Grp = Group\)\. Gold answers range from a single value or pass/fail verdict to multi\-row breakdowns that must match exactly, which explains the accuracy ordering in Table[4](https://arxiv.org/html/2609.05776#S4.T4)\.
## Appendix BKnowledge Base Examples

#### Rule document \(excerpt\)\.

The following is a shortened version of the Active Seller Definition Rule, an example rule document:

> Rule Title:Active Seller Definition Rule Effective Date:1 July 2018 Core Rule:A seller is classified as Active if and only if both conditions are met within a trailing 90\-day window: \(1\) Volume: at least one delivered order; \(2\) Quality: average review score≥\\geq3\.0 across all historical reviews\. Sellers meeting volume but not quality are classified as “On Probation” and excluded from Active Seller counts\.

#### Metric catalog entry \(excerpt\)\.

> MV\-SEL\-001: Active Seller Count\(Tier L2\) Formula:COUNT\(DISTINCT seller\_id\) WHERE delivered AND avg\_review≥\\geq3\.0 Tables:order\_items, orders, order\_reviews

### B\.1Rule Document Provenance

The rule documents and metric catalogs used in our demonstration are LLM\-generated augmentations, not real enterprise documents\. Given that the two underlying datasets are public databases with no accompanying rule documentation, we augment the datasets with realistic rule documents\.

#### Generation process\.

For each domain, an LLM \(DeepSeek V3\.2\) was prompted with the database schema, a metrics catalog \(also LLM\-generated from the schema\), and supporting context about the fictional company\. The LLM was instructed to produce standalone rule documents whose rules would change the computed value of specific metrics if not consulted\. Each document was generated independently in a single LLM call\.

#### Generation prompt \(metric rules, abbreviated\)\.

Rule Document Generation Prompt \(abbreviated\)System:You are a business analyst writing internal rule documents for \{company\}\. Write a standalone rule document that defines rules which modify how business metrics are computed\. The rule should contain information NOT already in the metrics catalog\. The goal is that someone computing the metric would get a WRONG answer if they don’t consult this rule document\. Include: a realistic effective date, named owner, version, core rules/thresholds, scope \(affected metrics\), exceptions, and a rationale section\. Context provided: — Metrics catalog — — Supporting documents —

#### Descriptive statistics\.

Each domain has 17 rule documents, averaging 750–820 words per document \(Table[7](https://arxiv.org/html/2609.05776#A2.T7)\)\. Each document falls into one of six rule types that govern how it modifies a metric:*definitional*\(defines what counts, e\.g\., active seller criteria\),*exclusionary*\(removes rows, e\.g\., exclude neutral reviews\),*temporal/versioned*\(rule changes on a date, e\.g\., SLA change in January 2018\),*conditional*\(applies only to a dimension slice, e\.g\., São Paulo orders\),*privacy guardrail*\(suppresses small\-nnresults\), and*override*\(replaces a default formula under specific conditions\)\. All six types are represented in both domains\.

BrazilianCzechNumber of rule documents1717Mean word count821756Min / max word count654 / 1132592 / 928Median word count847721Rule types represented66Table 7:Descriptive statistics for the LLM\-generated rule documents\. “Rule types represented” shows the count out of 6 possible types \(definitional, exclusionary, temporal, conditional, privacy, override\)\.
#### Example rule document \(excerpt\)\.

The following is an abbreviated version of the Outlier Exclusion Rule from the Brazilian e\-commerce domain:

Outlier Exclusion Rule for Delivery Time Metrics \(excerpt\)Rule Title:Outlier Exclusion Rule for Delivery Time Metrics Effective Date:October 1, 2018 Owner:Ana Silva, Head of Logistics Analytics Core Rule:Any delivered order with a calculated delivery duration exceeding 60 calendar days is classified as a data anomaly and excluded from metric computation\. Scope:On\-Time Delivery Rate, Average Delivery Time, and any L3\+ composite metric incorporating delivery time\. Exception:Orders exceeding the 60\-day threshold are still counted as*late*for On\-Time Delivery Rate—the exclusion applies only to the average used in internal benchmarking, not the binary late/on\-time classification\. Expected Impact:∼\\sim0\.3% of orders affected; including these outliers skews Average Delivery Time upward by∼\\sim1\.5 days\.

## Appendix CAblation: Impact of Artifact Linkage Graph

To isolate the graph’s contribution to rule selection, we hold each task’s skeleton fixed—the metric, dimensional slice, and time window—and vary only*which rule is paired with the metric*\. For every rule\-impacting task \(N=175N\{=\}175\), we re\-pair the metric under three strategies and re\-execute:*Graph*uses the original graph\-selected rule;*Semantic*uses the nearest other rule by embedding cosine similarity;*Random*uses a uniformly random other rule\. A pairing is*discriminative*if applying the rule changes the computed answer relative to the base \(no\-rule\) query\. Because the skeleton is identical across conditions, the comparison isolates the effect of rule selection alone\.

Pairing strategyDiscriminative validityGraph82%Semantic29%Random22%Table 8:Discriminative validity under three rule\-pairing strategies on identical task skeletons \(N=175N\{=\}175rule tasks, both domains pooled\)\. A pairing is discriminative if applying the rule changes the computed answer\. The graph selects rules that genuinely modify the computation∼\\sim3×\\timesmore often than random pairing; semantic similarity helps only marginally, since a topically related rule need not alter the metric’s value\.The graph\-selected rule changes the answer in 82% of cases, versus 29% for semantic\-similarity pairing and 22% for random pairing\. The graph is not perfect—for some dimensional slices even a relevant rule leaves the answer unchanged \(e\.g\., a delivery\-outlier rule has no effect on a region with no outlier deliveries\)—but it identifies discriminative pairings far more reliably than the alternatives\. Semantic similarity offers only a small gain over random, confirming that topical relatedness does not ensure a rule*causally modifies*the metric: the graph’s typed rule→\\rightarrowmetric edges, not embedding proximity, are what capture this relationship\.

## Appendix DSubgraph Scoring and Selection Details

After enumerating structurally valid subgraphs \(Section[3\.2](https://arxiv.org/html/2609.05776#S3.SS2)\), the pipeline must select a diverse, high\-quality subset\. This involves two steps: \(1\) scoring each candidate, then \(2\) selecting via MMR to balance quality against diversity\.

### D\.1Step 1: Quality Scoring

Each candidate subgraphSSreceives a composite quality scoreq⁡\(S\)q\(S\)as a weighted average of five independent scorers:

q⁡\(S\)=∑i=15wi⋅si​\(S\)q\(S\)=\\sum\_\{i=1\}^\{5\}w\_\{i\}\\cdot s\_\{i\}\(S\)\(1\)The five scorers capture complementary factors:

- •Semantic affinity\(weight 0\.25\): Does the metric–dimension pairing make sense? For example, “delivery time by customer state” is natural, but “delivery time by payment installment count” is not\. Although the graph constrains which dimensions are reachable from a metric’s tables, not every reachable pairing is semantically meaningful; as such, this scorer downweights implausible ones\.
- •Rule diversity\(weight 0\.20\): Prefers subgraphs using less\-frequently\-selected rule documents, via diminishing returns:1\|RS\|​∑r∈RS11\+nr\\frac\{1\}\{\|R\_\{S\}\|\}\\sum\_\{r\\in R\_\{S\}\}\\frac\{1\}\{1\+n\_\{r\}\}, wherenrn\_\{r\}counts prior selections of rulerrandRSR\_\{S\}is the set of rule documents\.
- •Metric rarity\(weight 0\.20\): Analogous to rule diversity but for metrics—prefers subgraphs involving metrics that have appeared less often\.
- •Dimension cardinality\(weight 0\.20\): How many distinct values does the chosen dimension have in the data? Dimensions with more values offer richer slicing possibilities, so the scorer increases monotonically: single\-value dimensions score 0\.1 \(trivial questions\), 2 values score 0\.5 \(only binary comparisons\), 3–5 values score 0\.8, and\>\>5 values score 1\.0\. Date/timestamp columns \(used for trend analysis\) default to 0\.7 because their effective cardinality depends on time\-window granularity rather than the raw number of distinct dates\.
- •Join complexity\(weight 0\.15\): Rewards subgraphs requiring more table joins, scaling linearly with the number of tables and capping at 1\.0: one table scores 0\.25, two tables 0\.5, three tables 0\.75, and four or more tables 1\.0\. This ensures the benchmark includes multi\-join tasks\.

Weights were determined by manual inspection: we adjusted until top\-ranked candidates consistently matched subgraphs that a domain expert identified as realistic analysis questions\.

### D\.2Step 2: MMR Selection

High\-quality candidates may still be redundant \(e\.g\., many subgraphs testing the same metric with different dimension values\)\. To ensure diversity, we use Maximal Marginal Relevance \(MMR\)\. Given candidate pool𝒞\\mathcal\{C\}and target countkk, we greedily build the selected set𝒮\\mathcal\{S\}:

c∗=arg⁡maxc∈𝒞∖𝒮​\[\(1−λ\)⋅q⁡\(c\)\+λ⋅mins∈𝒮⁡dJ​\(c,s\)\]c^\{\*\}=\\arg\\max\_\{c\\in\\mathcal\{C\}\\setminus\\mathcal\{S\}\}\\left\[\(1\-\\lambda\)\\cdot q\(c\)\+\\lambda\\cdot\\min\_\{s\\in\\mathcal\{S\}\}d\_\{J\}\(c,s\)\\right\]\(2\)where the first term favors quality and the second favors diversity\. The distancedJd\_\{J\}is the Jaccard distance over node sets:

dJ​\(a,b\)=1−\|Va∩Vb\|\|Va∪Vb\|d\_\{J\}\(a,b\)=1\-\\frac\{\|V\_\{a\}\\cap V\_\{b\}\|\}\{\|V\_\{a\}\\cup V\_\{b\}\|\}\(3\)withVa=𝑚𝑒𝑡𝑟𝑖𝑐𝑠⁡\(a\)∪𝑡𝑎𝑏𝑙𝑒𝑠⁡\(a\)∪𝑑𝑖𝑚𝑒𝑛𝑠𝑖𝑜𝑛𝑠⁡\(a\)∪𝑟𝑢𝑙𝑒𝑠⁡\(a\)V\_\{a\}=\\mathit\{metrics\}\(a\)\\cup\\mathit\{tables\}\(a\)\\cup\\mathit\{dimensions\}\(a\)\\cup\\mathit\{rules\}\(a\)\. Two subgraphs sharing no nodes havedJ=1d\_\{J\}=1\(maximally diverse\); identical subgraphs havedJ=0d\_\{J\}=0\.

We setλ=0\.3\\lambda=0\.3, favoring quality over diversity\. This balances two failure modes:λ\\lambdatoo low produces redundant subgraphs testing the same metric–dimension pairs;λ\\lambdatoo high selects low quality subgraphs\. For the first selection \(𝒮=∅\\mathcal\{S\}=\\emptyset\), the diversity term defaults to 1\.0, so the highest\-quality candidate is chosen first\. After MMR selection, a coverage pass adds any missing metrics or rule documents \(those not yet represented in𝒮\\mathcal\{S\}\) by selecting their highest\-scoring subgraph\.

In our pipeline, when setting the number of target questions to100100,∼93%\{\\sim\}93\\%of metric–rule links \(averaged across datasets\) are retained as parts of questions\. The coverage loss is concentrated in metrics with many associated rules: once 2–3 subgraphs are selected for such a metric, the diversity term suppresses additional ones in favor of under\-represented metrics\.

Analysis TypeMetric\(s\)TablesdimensionRulesScorePoint queryLate Delivery Rate by Carrier Segmentorders, customers, sellers, order\_itemscustomer\_state \(spatial\)Active Seller Definition0\.958Trend analysisAverage Delivery Timeordersorder\_purchase \_timestamp \(temporal\)Outlier Exclusion \(Delivery\)0\.807ComparisonCLV, Revenue per Order, CSATcustomers, orders, order\_itemsorder\_status \(categorical\)CLV Lifespan, CSAT Calculation, Revenue Recognition, …\(11 total\)0\.898RankingSeller Fulfillment Scoreorder\_items, orders, order\_reviewsorder\_status \(categorical\)Seller Fulfillment Score Weights0\.884Significance testingPayment Method Distributionorder\_paymentspayment\_type \(categorical\)Undefined Payment Exclusion0\.879Threshold crossingUnique Buyer Countorders, customersorder\_status \(categorical\)Customer Deduplication0\.846CounterfactualCategory Growth Rateorder\_items, products, ordersorder\_status \(categorical\)CSAT Calculation, Multi\-Item Attribution0\.849DecompositionSeller Concentration, Active Seller Count, GMVorder\_reviews, orders, order\_itemsorder\_status \(categorical\)Active Seller Definition, Revenue Recognition, …\(6 total\)0\.834Table 9:Representative selected subgraph for each analysis type from the Brazilian E\-commerce domain\. Scores reflect the composite quality metricq⁡\(S\)q\(S\)from Equation[1](https://arxiv.org/html/2609.05776#A4.E1)\.
### D\.3Subgraph Examples

Table[9](https://arxiv.org/html/2609.05776#A4.T9)presents subgraph examples from the Brazilian E\-commerce domain and Table[10](https://arxiv.org/html/2609.05776#A4.T10)for the Czech Financial domain\.

Analysis TypeMetric\(s\)TablesdimensionRulesScorePoint queryTransaction Volumetransoperation \(categorical\)Account Opening Date Rule0\.866Trend analysisBalance Volatility \(CV\)transtrans\.date \(temporal\)Small Sample Suppression0\.801ComparisonCredit Risk Scoreloan, trans, account, district, clientdistrict region \(spatial\)Credit Risk Score Weights1\.000RankingAverage Account Balancetrans, accountk\_symbol \(categorical\)Fee Structure Change 1996, Inter\-Bank Transaction Flag0\.863Significance testingStanding Order Value by Typeorderk\_symbol \(categorical\)Standing Order Amount Validation0\.858Threshold crossingDebt Service Coverage Ratiotrans, loan, accountk\_symbol \(categorical\)Cohort Maturity Threshold, NPL Denominator0\.849CounterfactualCard Penetration Ratecard, disposition, accountcard type \(categorical\)Card Eligibility Definition, Date Conversion0\.889DecompositionCredit Risk Score, DSCR, Balance Volatility, NPL Ratiodistrict, loan, account, trans, clientdistrict region \(spatial\)Credit Risk Weights, Cohort Maturity, Crisis Adjustment 1997, …\(11 total\)0\.886Table 10:Representative selected subgraph for each analysis type from the Czech Financial domain\.

## Appendix EValidation Details

### E\.1Stage 1: Programmatic Validation

The first stage applies deterministic checks\. A task passes only if all checks return zero issues; failing tasks are flagged and annotated with the specific issues detected, but are excluded from LLM evaluation and the final benchmark\.

Structural integrity\.A set of regex patterns detects common generation artifacts: timestamp columns used as categorical filters \(e\.g\., a question filtering on a datetime field as though it were a discrete value\), raw column names leaked into the question text, and internal database identifiers such as primary or foreign key fields exposed in the phrasing\. Questions shorter than 20 characters are also flagged\.

Answer completeness\.Each task must have a valid answer\. For SQL\-based tasks, this requires a non\-empty list of result dictionaries containing no error markers\. For knowledge\-only tasks, a substantive string answer of more than 10 characters must be present\. For all non\-knowledge tasks, the system additionally verifies that a scorable expected answer can be derived from the gold data, ensuring downstream automated scoring is feasible\.

### E\.2Stage 2: LLM\-Based Quality Evaluation

Tasks \(question\-answer pairs\) that pass programmatic validation are evaluated by an LLM judge \(DeepSeek V3\.2 via Amazon Bedrock\) across eight dimensions with a binary score \(0 or 1 each\) and one\-sentence justification per dimension\. A task is admitted only if all dimensions pass\. The prompt is as follows:

LLM Validator PromptYou are evaluating a question\-answer\-SQL triple from a benchmark dataset\. ANALYSIS TYPE: \{analysis\_type\} QUESTION: \{question\} ANSWER: \{gold\_answer\} SQL: \{gold\_sql or ‘NONE \(knowledge\-base task, no SQL needed\)’\} \{type\_specific\_context\} For each dimension, first write one sentence of reasoning, then give your score \(1=pass, 0=fail\)\. Only score 0 for CLEAR violations\. When in doubt, pass\. 1\.OBJECTIVITY: Does the question have exactly one correct answer given the data?2\.ANSWER\_LEAKAGE: Is the question free of hints that reveal the answer?3\.NATURALNESS: Does the question sound like a real person would ask it?4\.ANSWER\_RELEVANCE: Does the answer address the question’s core intent? Structured tool outputs are valid\.5\.FORMAT\_ALIGNMENT: Rankings must specify count\. Breakdowns, significance, and trend always pass\. Threshold: FAIL if question asks about multiple items \(format holds one value only\)\.6\.SQL\_CORRECTNESS: Does the SQL fetch appropriate data? Pass if SQL is NONE, if no date is mentioned and no date filter exists, or if tool\-based SQL fetches raw data without performing the final computation\.7\.LOGICAL\_REASONING: Is the answer internally consistent?8\.GOLD\_SCOPE\_MATCH: Does the answer contain ONLY data that the question asks about? FAIL if question asks about one entity but answer contains others\. PASS if question asks for a breakdown and answer has multiple rows\.

### E\.3Human\-in\-the\-Loop Review

To assess benchmark quality, an expert annotator reviewed a stratified sample of 100 tasks across the same eight dimensions used by the automated validator\. Table[11](https://arxiv.org/html/2609.05776#A5.T11)compares three views: the human ground\-truth pass rate across all 100 tasks, the validator’s pass rate on those same tasks, and the human\-verified quality of the 25 tasks the validator admits \(all\-eight\-pass\)\.

DimensionHumanValidatorAdmitted\(100\)\(100\)\(25\)Obj\.87%64%100%Leakage100%100%100%Natural\.100%100%100%Relevance92%83%100%Format98%77%96%SQL Corr\.94%48%100%Reasoning100%74%100%Scope95%72%99%Average96%77%99%Table 11:Human\-in\-the\-loop validation \(100\-task stratified sample\)\.Human: expert\-annotator pass rate \(ground truth\);Validator: LLM validator pass rate on the same tasks;Admitted: human\-verified pass rate of the 25 tasks the validator admits \(all eight dimensions pass from the validator\)\. The validator is stricter than the human across dimensions, so what it admits is near\-perfect \(99% average\)\. Its conservatism reduces yield \(admits 25% of candidates\) but indicates high precision among validator\-admitted tasks\.
### E\.4Generator and Judge Sensitivity

On a generator\-swap sample with fixed skeletons and SQL\-computed gold answers, Qwen3\-80B\- and DeepSeek\-generated questions receive similar mean quality scores under an independent Claude Sonnet 4\.6 judge \(79\.9% versus 79\.6%\), although individual dimensions move in both directions: SQL correctness decreases from 51% to 34%, while format alignment increases from 86% to 99%\. The raw artifact contains 87 valid paired examples\. Separately, four judges evaluated the same balanced human\-labeled sample\. Mean precision is 94–96%, while recall ranges from 79–93%; Opus hasn=98n=98because two outputs could not be parsed\. The judge is used for task admission, not for grading agent answers, which are scored against SQL\-computed gold\. These analyses bound but do not eliminate the shared\-model confound\.

## Appendix FComparison between Agents and Human Experts

In order to interpret agent accuracy on DI\-Bench, we need a human reference point\. Without one, we cannot tell whether agents’ low performance on certain tasks reflects an agent\-human gap or a task that humans also find difficult\. We therefore sample 5 questions from each of the*Knowledge*,*Knowledge \+ Analytics*, and*Knowledge \+ Analytics \+ Rule*tiers and ask two business\-intelligence experts to answer them\. The experts were able to solve 13 out of 15 tasks \(87%\) correctly, whereas the agents reach 73% \(Opus\), 67% \(Haiku\), 60% \(Qwen\), and 53% \(DeepSeek\) on the same samples\. This shows that the benchmark differentiates agent capabilities and is not merely filled with tasks that are impossible to solve\. The gap instead reflects tasks that humans can solve but agents struggle with\.

## Appendix GBenchmark Construction Time

Our proposed pipeline substantially reduces the human effort required to construct a benchmark\. We estimate that fully manual construction takes an expert roughly 35 minutes per task: subgraph creation \(3 min\), finding an example SQL query \(4 min\), writing and verifying the rule\-modified SQL \(15 min\), phrasing the question \(5 min\), specifying the answer format \(3 min\), and a final check \(5 min\), or about 420 hours for a 731\-task set\. With the pipeline, the only remaining human effort is verification, which, on average across all task types, takes roughly 1\.8 minutes or 22 hours for the entire benchmark, which is a significant time reduction from 420 hours required to construct the benchmark manually\.

## Appendix HArtifact Linkage Graph Statistics

This appendix provides statistics on the number of nodes and edges of each defined type in the artifact linkage graphs, for both datasets\. These are provided in Table[12](https://arxiv.org/html/2609.05776#A8.T12)\.

BrazilianCzechTable nodes98Metric nodes2929Rule nodes1717Dimension nodes2117Total nodes7671metric→\\rightarrowtable edges6145table→\\rightarrowdimension edges2117table→\\rightarrowtable edges98rule→\\rightarrowmetric edges7979metric→\\rightarrowmetric edges911rule→\\rightarrowdimension edges128Total edges191168Table 12:Artifact linkage graph statistics for both domains\.
## Appendix IExample Tasks

We present one representative task for each of the nine analysis types, drawn verbatim from the benchmark \(questions, required rules, and gold answers unedited\)\. The mark is Claude Opus 4\.6’s result\.

Point QueryQuestion:What is the average loan size and total number of loans for our 60\-month active loans in the South Moravia region? Required rules:cohort maturity threshold; small\-sample suppression Gold:\[\{region: south Moravia, duration: 60, avg\_loan\_size: 228422\.0, loan\_count: 30\}\] Opus:✓

RankingQuestion:Which 5 customer states had the longest average delivery times in days for orders shipped during 2017 and the first eight months of 2018? Required rules:outlier exclusion \(delivery\); on\-time delivery SLA Gold:\[AM: 25\.65, AP: 24\.76, AL: 24\.12, RR: 23\.76, PA: 23\.0\]\(avg delivery days\) Opus:✓

ComparisonQuestion:Compare the Seller Concentration Index \(HHI\) for ‘computers\_accessories’ against ‘cds\_dvds\_musicals’ across all delivered and shipped orders\. Which has higher concentration risk? Required rules:multi\-item attribution; privacy/confidentiality Gold:\{value\_a: 0\.03, value\_b: 1\.0, higher: cds\_dvds\_musicals\} Opus:✓

Trend AnalysisQuestion:Show me the monthly trend for our average delivery time throughout 2017 — any noticeable patterns or fluctuations? Required rules:outlier exclusion \(delivery\) Gold:\{direction: increasing, total\_change\_pct: 32\.1, peak\_period: 2018\-01\} Opus:✓

Threshold CrossingQuestion:Did our average Seller Ship\-Out Time in the state of Bahia exceed the 60\-hour target? Required rules:São Paulo SLA change \(January\) Gold:\{value: 58\.33, threshold: 60\.0, result: FAIL\} Opus:✓

Significance TestingQuestion:Are the average installment counts for orders paid by credit card vs\. boleto in Q1 2018 statistically significantly different? Required rules:outlier exclusion \(delivery\) Gold:\{significant: true, p\_value: 0\.0, test: Welch’s t\-test \(two\-tailed\)\} Opus:✓

CounterfactualQuestion:What’s the average revenue per unique customer in São Paulo state, assuming we could extend operations there starting from January 2017? Required rules:none \(control\) Gold:\[\{customer\_state: SP, revenue\_per\_customer: 129\.41\}\] Opus:✓

Knowledge RetrievalQuestion:How do we calculate Total Deposit Volume from the transaction data? Required rules:transaction net\-flow rule Gold:“Total Deposit Volume must only sum amounts wheretype = ‘PRIJEM’\.” \(must\-include:type = ‘PRIJEM’\) Opus:✓

Decomposition\(hardest type\)Question:Break down the Order Defect Rate by seller state for H1 2018, showing which geographic areas contribute most to the overall defect rate\. Required rules:on\-time delivery SLA; São Paulo SLA change \(January\) Gold:\[BA: 18\.18, MS: 66\.67, ES: 18\.18, MA: 16\.67, …\]\(18 states, odr\_pct\) Opus:✗ computed defect counts without the SLA grace period, so most rows are off\.

## Appendix JPer\-Model Accuracy by Analysis Type

Table[13](https://arxiv.org/html/2609.05776#A10.T13)breaks down accuracy by model and analysis type over the full 731\-task benchmark \(BR and CZ combined\)\. The difficulty ordering is consistent across models: all four exceed 75% on knowledge retrieval and 60% on threshold crossing and trend analysis, but fall below 35% on ranking, point query, counterfactual, and decomposition\. Model separation is largest on the harder types, e\.g\. on point query, Opus \(34%\) clearly leads Qwen3\-80B \(18%\) and Haiku \(22%\)—confirming that the discriminative signal concentrates in tasks requiring exact multi\-row matching and multi\-rule reasoning\.

Analysis TypeNOpusHaikuQwenDeepSeekRanking14523201123Knowledge retrieval13789857876Threshold crossing12372616567Point query10234221823Trend analysis7878637878Comparison6379596565Sig\. testing4271\.464\.357\.159\.5Counterfactual219\.54\.89\.54\.8Decomposition205\.05\.05\.05\.0Overall73157\.949\.047\.950\.9Table 13:Per\-model accuracy \(%\) by analysis type on the full 731\-task benchmark \(Brazilian e\-commerce and Czech financial combined\)\. N is the number of tasks of each type\.
## Appendix KDetailed Failure Cases

To make concrete*what*a task expects and*why*it is hard, we walk through three Claude Opus 4\.6 failures from the Brazilian e\-commerce domain\. All three are tasks where the agent successfully*retrieved*the governing rule, as verified in the tool trace, yet still produced a wrong answer isolating rule*application*as the bottleneck rather than retrieval\. Expected values are the deterministic SQL\-computed gold; predicted values are taken verbatim from the agent’s<answer\>tag\.

Case 1 — Threshold crossing flips on a one\-day grace period \(DI\-BR\-0117\)Question:For delivered orders in the first half of 2018, did the order defect rate for sellers in the Federal District \(DF\) exceed 15\.0%? Required rules:on\-time delivery SLA; delivery outlier exclusion Expected:\{value: 14\.46, threshold: 15\.0, result: FAIL\} Opus:✗\{value: 16\.40, threshold: 15\.0, result: PASS\} Why it is hard:The SLA rule classifies an order as late only if it arrives*after the end of the next business day*following the estimated date—a one\-business\-day grace period\. Opus retrieved the rule but counted any delivery past the estimated date as a defect \(delivered\_date \> estimated\_daterather than\> estimated\_date \+ 1 day\)\. The grace period removes just enough borderline\-late orders to drop DF’s ODR from 16\.40% to 14\.46%—across the 15% threshold\. Because the metric is compared against a threshold, a sub\-two\-point numeric error flips the categorical verdict from FAIL to PASS; the task is unforgiving of an approximately\-right computation\.

Case 2 — Ranking requires a post\-hoc privacy suppression \(DI\-BR\-0123\)Question:Which product categories have the highest concentration of sales among sellers? Provide a ranking of the top 5 categories by the Seller Concentration Index \(HHI\), considering only those where market dominance is most extreme\. Required rules:multi\-item attribution; data confidentiality Expected:\[la\_cuisine: 1\.0, home\_comfort\_2: 1\.0, fashion\_sport: 1\.0, cds\_dvds\_musicals: 1\.0, arts\_and\_craftmanship: 1\.0\] Opus:✗\[cds\_dvds\_musicals: 1\.0, la\_cuisine: 0\.88, computers: 0\.58, security\_and\_services: 0\.54, flowers: 0\.48\] Why it is hard:An HHI of 1\.0 means a category’s sales come from a single seller\. The confidentiality rule requires suppressing data points derived from fewer than three unique sellers*after*the base metric is computed, and the multi\-item attribution rule fixes how each order’s revenue is assigned to categories\. Opus computed HHI correctly but never applied the suppression\-and\-tie logic, returning one HHI=1\.0 category followed by lower\-concentration ones, missing that several categories tie at the maximum 1\.0 and that the ranking among ties is rule\-governed\. The rule operates as a post\-processing layer applied*after*SQL aggregation, which the agent treated as irrelevant once it had numeric HHI values\.

Case 3 — Counterfactual demands two parallel computations \(DI\-BR\-0314\)Question:What would our On\-Time Delivery Rate look like if we removed the grace period for all orders delivered in the first half of 2018, broken down by customer state? Required rules:delivery outlier exclusion; week/timezone definition Expected \(excerpt\):AL: \{strict: 72\.73, with\_grace: 74\.68\},AM: \{strict: 94\.55, …\} Opus \(excerpt\):✗BA: \{strict: 81\.29, with\_grace: 82\.85\},CE: \{strict: 74\.13, with\_grace: 76\.06\} Why it is hard:The agent must compute the same metric twice: once with the grace period \(the live rule\) and once without \(the counterfactual scenario\), while applying the outlier\-exclusion filter consistently to both\. A single question thus requires two parallel, rule\-aware computations over the same rows, and any inconsistency in how either rule is applied causes the per\-state values to diverge from gold\.

#### Takeaway\.

All three failures share a structure: retrieval succeeds, the base SQL is roughly right, but the rule\-induced adjustment \(a grace period, a suppression rule, a counterfactual rule swap\) is either dropped or misapplied\. This matches the aggregate finding \(Section[4](https://arxiv.org/html/2609.05776#S4)\) that the with\-rule analytical tasks are the benchmark’s primary discriminator\.

## Appendix LPipeline Reproduction

DI\-Bench takes a relational schema and tables, a metric catalog with executable SQL templates, and a policy\-document corpus\. Figure[3](https://arxiv.org/html/2609.05776#A12.F3)documents the implementation details: graph construction, candidate\-scoring weights, MMR selection, the base/rule\-aware SQL record, validation inputs, rare\-type augmentation, and stage commands\. Appendix[E](https://arxiv.org/html/2609.05776#A5)gives the complete validator prompt\.

`\\iow\_now:Ne¨\\iow\_now:Ne¨graph = Graph\(nodes=\{table, metric, rule, dimension\}\)\\iow\_now:Ne¨graph\.add\(schema\_edges\(schema\)\)\\iow\_now:Ne¨\# table\-table and table\-dimension\\iow\_now:Ne¨graph\.add\(metric\_edges\(catalog\)\)\\iow\_now:Ne¨\# metric\-table and metric\-metric\\iow\_now:Ne¨graph\.add\(infer\_rule\_links\(documents, catalog\)\)\\iow\_now:Ne¨\# rule\-metric and rule\-dimension\\iow\_now:Ne¨pool = connected\_subgraphs\(graph\)\\iow\_now:Ne¨pool = \[g for g in pool if structural\_constraints\_hold\(g\)\] A\. Graph construction \\iow\_now:Ne¨\\iow\_now:Ne¨weights = \{\\iow\_now:Ne¨ semantic: \.25,\\iow\_now:Ne¨ rule\_diversity: \.20,\\iow\_now:Ne¨ metric\_rarity: \.20,\\iow\_now:Ne¨ cardinality: \.20,\\iow\_now:Ne¨ joins: \.15\\iow\_now:Ne¨\}\\iow\_now:Ne¨quality = score\_candidates\(pool, weights\)\\iow\_now:Ne¨chosen = greedy\_mmr\(pool, quality, target=100,\\iow\_now:Ne¨ diversity=\.30, seed=42\)\\iow\_now:Ne¨chosen = ensure\_metric\_and\_rule\_coverage\(chosen, pool\) B\. Candidate scoring and selection \\iow\_now:Ne¨\\iow\_now:Ne¨for g in chosen:\\iow\_now:Ne¨ for kind in compatible\_analyses\(g\):\\iow\_now:Ne¨ base\_sql = instantiate\_template\(g\.metric,\\iow\_now:Ne¨ g\.dimensions, kind\)\\iow\_now:Ne¨ rule\_ops = extract\_rule\_operations\(\\iow\_now:Ne¨ g\.rules,\\iow\_now:Ne¨ allowed=\{filter, exclude, threshold,\\iow\_now:Ne¨ weight, override\}\)\\iow\_now:Ne¨ gold\_sql = apply\_operations\(base\_sql, rule\_ops\)\\iow\_now:Ne¨ base = enrich\(execute\(base\_sql\), kind\)\\iow\_now:Ne¨ gold = enrich\(execute\(gold\_sql\), kind\)\\iow\_now:Ne¨ if failed\(gold\) or \(rule\_ops and equivalent\(base, gold\)\):\\iow\_now:Ne¨ continue\\iow\_now:Ne¨ question = generate\_question\(g, kind, gold, documents\)\\iow\_now:Ne¨ task = record\(question, kind, g\.id, base\_sql,\\iow\_now:Ne¨ gold\_sql, rule\_ops, gold, g\.rules\) C\. SQL grounding and question generation \\iow\_now:Ne¨\\iow\_now:Ne¨ if not programmatic\_checks\(task\):\\iow\_now:Ne¨ continue\\iow\_now:Ne¨ judge\_input = \{question, gold, gold\_sql,\\iow\_now:Ne¨ type\_context\(kind\)\}\\iow\_now:Ne¨ dimensions = \{objectivity, leakage, naturalness,\\iow\_now:Ne¨ relevance, format, correctness,\\iow\_now:Ne¨ reasoning, scope\_match\}\\iow\_now:Ne¨ \# Full prompt appears in the Validation Details appendix\\iow\_now:Ne¨ if all\(binary\_judge\(judge\_input, d\)\\iow\_now:Ne¨ for d in dimensions\):\\iow\_now:Ne¨ tasks\.append\(task\)\\iow\_now:Ne¨\\iow\_now:Ne¨tasks = augment\_to\_minimum\(tasks,\\iow\_now:Ne¨ unused=pool\-chosen, minimum=10, seed=42\)\\iow\_now:Ne¨tasks = deduplicate\(tasks, keys=\{task\_id, subgraph\_id\}\)\\iow\_now:Ne¨export\(tasks\) D\. Validation, augmentation, and export \\iow\_now:Ne¨\\iow\_now:Ne¨$ python kg\_construction/extract\_graph\.py\\iow\_now:Ne¨$ python kg\_construction/extract\_policy\_edges\_llm\.py all –model deepseek\.v3\.2\\iow\_now:Ne¨$ python subgraph\_sampling/sample\_subgraphs\.py –all –target 100 –seed 42\\iow\_now:Ne¨$ python task\_generation/generate\_tasks\.py –all –seed 42 –use\-llm\\iow\_now:Ne¨$ python task\_generation/enrich\_tasks\.py –all\\iow\_now:Ne¨$ python task\_generation/generate\_questions\.py –all –use\-llm\\iow\_now:Ne¨$ python task\_generation/validate\_questions\.py –all –use\-llm\\iow\_now:Ne¨$ python task\_generation/augment\_rare\_types\.py –all –target 10 –use\-llm E\. Stage commands`Figure 3:Five\-panel reference implementation: graph construction, weighted candidate selection, SQL grounding and backward generation, validation and export, and executable stage commands\. Policy\-bearing tasks are retained only when rule removal changes the executed answer\.#### Reproduction boundary\.

The listing is sufficient to reproduce the generation procedure, but not necessarily the exact task wording: LLM\-backed stages depend on model availability and nondeterministic decoding\. The released configuration records model identifiers and decoding parameters, while non\-LLM randomized stages use seed 42\.