Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities

# Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities
Source: [https://arxiv.org/html/2607.06482](https://arxiv.org/html/2607.06482)
11institutetext:Fujitsu Research of America, Santa Clara CA 95054, USA11email:shasegawa@fujitsu\.com###### Abstract

Current benchmarks for evaluating Large Language Models \(LLMs\) in data analysis often fail to reflect real\-world settings\. They typically focus on fact retrieval from small tables and overlook the challenges of large multi\-tabular datasets, external knowledge integration, and exploratory insight discovery\. We introduceDataGovBench, a benchmark derived from governmental open data designed to evaluate LLMs in practical scenarios\. The benchmark includes two tasks:Table QAthat requires solving complex decomposable questions and producing textual answers or visualizations, andTable Insightthat evaluates the ability of models to generate expert\-level findings through exploratory data analysis\. Comprehensive experiments with state\-of\-the\-art LLMs, both with and without agentic frameworks, reveal significant performance gaps across both tasks\. These results suggest that current LLM\-based systems remain far from satisfying the demands of real\-world data analytics\. DataGovBench provides a challenging benchmark for advancing research on LLMs capable of both answering analytical queries and discovering insights from data\. Code and sample data are available at[https://github\.com/SoHasegawa/datagovbench](https://github.com/SoHasegawa/datagovbench)\.

## 1Introduction

![Refer to caption](https://arxiv.org/html/2607.06482v1/x1.png)Figure 1:DataGovBench evaluates LLMs and agents on two table reasoning tasks using large, multi\-table datasets supplemented with metadata and external knowledge\. \(a\) TheTable QAtask requires models to answer simple or decomposable questions with textual or visual answers\. \(b\) In contrast, theTable Insighttask challenges models to perform open\-ended exploratory analysis, proactively generating a list of insights and a summary without a specific user query\.The ability to reason over structured data is a cornerstone of modern data science and a long\-standing challenge in artificial intelligence\. With the advent of Large Language Models \(LLMs\), we have witnessed a paradigm shift in how humans interact with complex information\[openai2024gpt4technicalreport,team2023gemini,yang2025qwen3technicalreport,deepseekai2025deepseekr1incentivizingreasoningcapability\]\. These models have led to the development of sophisticated agents designed to democratize data analysis, promising a future where any user can pose natural language questions to a dataset and receive accurate answers\[infida\-bench,su2024tablegpt2largemultimodalmodel\]\. The ultimate vision is an autonomous system that not only retrieves information but also uncovers the knowledge hidden within raw data\.

However, a significant gap persists between this vision and reality, as the benchmarks lack real\-world complexity\. While datasets like WikiTableQuestions\[wtq\]and Spider\[yu\-etal\-2018\-spider\]propelled research in semantic parsing and text\-to\-SQL, their controlled environments use small\-scale tables\. They largely neglect practical challenges such as massive table scales, the need to merge multiple tables, and the essential role of metadata and external knowledge\. Beyond these data limitations, existing benchmarks have focused on direct fact retrieval\[wu2025tablebench,infida\-bench,zhang\-etal\-2025\-t2r\]\. Tasks like QA and text\-to\-SQL are about retrieving information to a query, while missing the capability of proactive insight discovery that data analysts exhibit\. Consequently, this discovery\-oriented skill remains largely unevaluated, as few benchmarks have formalized insight generation as a primary task\[insight\-bench,mt\-raig\]\.

To bridge these notable gaps in both data realism and task scope, we introduceDataGovBench, a comprehensive benchmark sourced from public repositories like Data\.gov\[datagov\]\. Our benchmark features two complementary tasks: Table Question Answering \(Table QA\) and Table Insight Generation \(Table Insight\), as illustrated in Figure[1](https://arxiv.org/html/2607.06482#S1.F1)\. The Table QA task assesses factual reasoning over decomposable questions that explicitly include multiple sub\-questions, requiring models to produce answers either in text or in visualizations\. In contrast, the Table Insight task challenges models to perform expert\-level insight derivation, requiring in\-depth analysis and the discovery of trends\.

For the Table QA task, we use LLMs, aided by a novel table serialization, to generate a diverse corpus of QA pairs that are then meticulously verified by human annotators\. For the Table Insight task, we address the challenge of subjectivity by using the expert\-authored reports accompanying datasets as a ground truth\. This process yields a novel benchmark that surpasses existing alternatives by holistically combining the complex Table QA and Table Insight with the diverse data complexities, such as large\-scale, multi\-tabular datasets that require metadata and external knowledge, as detailed in Table[1](https://arxiv.org/html/2607.06482#S1.T1)\.

Comprehensive evaluation unearths that latest top\-performing LLMs achieve low accuracy for both tasks even with the agentic support\. These results demonstrate that DataGovBench captures real\-world challenges\. Furthermore, we provide detailed qualitative analyses for both tasks, identifying common failure modes and uncovering two missing capabilities of current LLM agents: narrative\-level reasoning over tabular data and accurate fact retrieval from complex tables\.

In summary, our paper makes the following three contributions:

- •We introduceDataGovBench, featuring tasks for both multifaceted Question Answering and Insight Generation with ground\-truths annotated by domain experts\. These tasks comprehensively handle large, multi\-tabular, and heterogeneous datasets representing complexity in real\-world scenarios\.
- •Extensive evaluation of state\-of\-the\-art models reveals a crucial performance gap, as even top models achieve low accuracy despite agentic support\. This underscores the benchmark’s difficulty and its alignment with actual challenges\.
- •We provide a detailed qualitative analysis and ablation studies that identify key failure modes, offering a clear guide for the community to focus on primary areas for LLM\-based data analytics\.

Table 1:Comparison of existing Table QA and Table Insight benchmarks with respect to task coverage and dataset characteristics\.
## 2Overview of DataGovBench

DataGovBench is a new benchmark designed to evaluate tabular reasoning in realistic scenarios\. The data is derived from governmental open data portals \(e\.g\., Data\.gov\[datagov\], Data\.gov\.uk\[datagovuk\]\), which host publicly available datasets from official institutions\. Datasets on these portals reflect real\-world complexity, consisting of a single or multiple tables containing a large number of records, and rich contextual information\. This context is provided through metadata, such as textual descriptions of the dataset, and often supplemented with external knowledge like data dictionaries\. The benchmark is designed to evaluate performance on two core data science tasks: Table QA and Table Insight\.

Table QA: This task requires models and agents to answer simple or decomposable questions with multiple sub\-questions\. The answers are provided in text or visualizations\.

Table Insight: This task challenges models and agents to perform exploratory analysis, generating substantive insights directly from the tabular data without an explicit user query\.

### 2\.1Benchmark Construction

The construction of DataGovBench involved a three\-stage process as shown in Figure[2](https://arxiv.org/html/2607.06482#S2.F2): 1\) curating datasets from open data portals through a systematic filtering process, 2\) annotating QA pairs via a human\-in\-the\-loop, and 3\) compiling ground\-truth insights by leveraging professional reports\.

#### 2\.1\.1Data Curation

Given the vast and decentralized nature of open data available online, a systematic collection and filtering process was imperative\. Our process began by identifying 53 open data platforms with English as the primary language \(a complete list is provided in the Table[6](https://arxiv.org/html/2607.06482#Pt0.A1.T6)\)\. We downloaded all available datasets from these platforms and then applied a series of filtering criteria including at least one of the tabular files covering over 5,000 records and metadata with description to understand the context of the dataset\. A more detailed filtering procedure is outlined in the Appendix[0\.A\.1\.2](https://arxiv.org/html/2607.06482#Pt0.A1.SS1.SSS2)\.

#### 2\.1\.2Annotation of Question and Answer Pairs

To construct a high\-quality, complex, and challenging set of QA pairs at scale while mitigating the need for resource\-intensive manual annotation, we designed a four\-stage generation pipeline that leverages LLMs with human\-in\-the\-loop verification\. Inspired by prior work\[wu2025tablebench\], this approach ensures both diversity and correctness\. The procedure is detailed below\.

1. 1\.Question Generation: The initial stage focused on generating a diverse pool of candidate questions\. To guide the output of the LLM, we first defined eight question types as presented in Appendix[0\.A\.2\.1](https://arxiv.org/html/2607.06482#Pt0.A1.SS2.SSS1)\. These types encompassed not only simple queries \(e\.g\. Ranking, Aggregation\) shared with existing benchmarks\[wu2025tablebench,wu2025mmqa\]but also complex decomposable questions\. The prompt contains the table contents, the title and description of the dataset from the metadata, external knowledge when available, and the designated question type\. Generating diverse and meaningful questions with LLMs requires providing them with a representative view of the table contents and value distributions\. While prior works mainly focus on smaller datasets such that the entire table or the first several rows could be embedded into the prompt\[wu2025tablebench\], we cannot apply similar approaches to tables with millions of records in our benchmark as it exceeds the limits of LLM context window\. To address this challenge, we propose a technique calledFeature type\-specific table serialization, which creates a compact yet informative summary of a table by representing each column according to its data type\. For instance, instead of listing all values in a categorical column, we provide only the set of unique categories\. This serialization is a core component of our workflow, and a detailed description of the logic for various feature types is presented in Appendix[0\.A\.2\.2](https://arxiv.org/html/2607.06482#Pt0.A1.SS2.SSS2)\. To mitigate model\-specific biases in the generated questions, we employed an ensemble of four high\-performance LLMs: GPT\-4o, GPT\-4o\-mini\[openai2024gpt4ocard\], Gemini 2\.0 Flash\[gemini2\.0\], and Gemini 1\.5 Pro\[gemini15unlockingmutimodal\]\.
2. 2\.Question Scoring: Following generation, the candidate questions underwent an automated scoring and selection phase\. Each question was evaluated against four qualitative criteria: relevance to the dataset, its potential to yield insightful or actionable information, sufficient analytical complexity with novelty, and clarity expressed in natural language\. In an approach to quality filtering, we tasked each of the four aforementioned LLMs with acting as a judge, selecting its top\-5 preferred questions from the generated pool based on the above criteria for each dataset\. A score from 5 \(highest\) to 1 \(lowest\) was assigned to these selections\. The scores from all four models were then aggregated for each question, resulting in a maximum possible score of 20\. Based on this aggregated score, we selected the top\-10 questions to proceed to the next stage\.
3. 3\.Answer Generation: For each of the top\-10 questions per dataset, an LLM was prompted to generate Python code that produces the correct answer\. After executing the code from all four models, we measured the answer consensus to filter out questions that yielded unanimous agreement across all four LLMs\. Such instances were deemed to indicate a low level of analytical complexity \(e\.g\. single column filtering or aggregation\), making them unsuitable for a benchmark in a real\-world setting\. An analysis of the generated questions shows that only 6% of the questions from the previous stage fall into this category, indicating that their removal does not drastically distort the original distribution as shown in Appendix[0\.A\.3](https://arxiv.org/html/2607.06482#Pt0.A1.SS3)\.
4. 4\.Human Verification: The remaining candidate QA pairs were subjected to a human verification and refinement process\. Using a custom\-developed annotation GUI as shown in Figure[7](https://arxiv.org/html/2607.06482#Pt0.A1.F7), human annotators with expertise in data analysis reviewed each component\. Their tasks included: \(1\) revising the natural language question for clarity and precision; \(2\) validating, debugging, and refining the Python code for correctness and efficiency; and \(3\) verifying the final answer derived from the code\. During this stage, annotators also filtered out questions for qualitative reasons, such as leading to uninformative answers, being too ambiguous to permit a definitive answer, or requiring external knowledge that was unavailable\. This human\-in\-the\-loop process yielded a curated set of 211 high\-quality QA pairs with 178 datasets\. Furthermore, all the questions were rephrased by separating the output format \(e\.g\. bar chart, list of tuples\) from the question, enhancing the naturalness, and paraphrasing the column names mentioned in the questions\. As a final quality control measure, a second group of annotators with much experience in data science, who were not involved in the initial revision phase, performed a concluding review by using the different annotation GUI as presented in Figure[8](https://arxiv.org/html/2607.06482#Pt0.A1.F8)\. This step was designed to validate the quality and logical soundness of the final QA pairs, with a particular focus on ensuring the Python code was robust and accurately addressed the corresponding question\.

After the question scoring stage, we generated a total of 1,840 candidate questions, from which we curated 211 high\-quality QA pairs, requiring 9,246 LLM calls in total\. A detailed breakdown of the reasons for discarding candidate questions is provided in Appendix[0\.A\.3](https://arxiv.org/html/2607.06482#Pt0.A1.SS3)\.

![Refer to caption](https://arxiv.org/html/2607.06482v1/x2.png)Figure 2:Overview of the three\-stage construction process of DataGovBench
#### 2\.1\.3Annotation of Insight

Establishing a ground truth for insight generation is inherently more complex than for question answering\. The subjective nature of what constitutes a meaningful finding makes achieving consensus difficult, posing a significant challenge for both automated generation and evaluation\. To address this limitation, we adopted official reports that accompany the open datasets\. These human\-authored documents contain the key findings and conclusions originally derived by specialists\. Our process involved curating six datasets \(Table[9](https://arxiv.org/html/2607.06482#Pt0.A1.T9)\) that included such reports\. We then systematically extracted the principal findings from each document depending on the representation of the key findings in the report\. If the report presents insights as bullet points, we directly treated each bullet point as one insight\. If the report expresses findings in free text, we uploaded the report to NotebookLM\[notebooklm\]and prompted it to suggest ten insights from the results sections, followed by the manual verification of the quality of the extracted insights\. We then synthesized the extracted insights into a standardized set of declarative sentences, and converted the set of sentences into a summary via Gemini 2\.5 Flash\[comanici2025gemini25pushingfrontier\]\. The set of insights and the summary together form the ground\-truth corpus for our insight generation task\.

### 2\.2Benchmark Statistics

Dataset Statistics: A statistical overview of the datasets in DataGovBench is presented in Table[2](https://arxiv.org/html/2607.06482#S2.T2)\. The benchmark comprises 178 unique datasets, featuring tables with an average of approximately 210K rows and 18 columns\. The scale of the data is substantial, with the largest table containing up to 11\.9M rows and 213 columns that exceed those found in most existing table QA benchmarks\. The distribution of original open data websites is presented in Table[8](https://arxiv.org/html/2607.06482#Pt0.A1.T8)\. Figure[3](https://arxiv.org/html/2607.06482#S2.F3)\(a\) illustrates the distribution of tables per dataset; notably, over 36% of the datasets are multi\-tabular, with five tables as the most frequent configuration\. Furthermore, as shown in Figure[3](https://arxiv.org/html/2607.06482#S2.F3)\(b\), over 57% of the datasets are accompanied by external knowledge to aid data interpretation, provided in various formats \(e\.g\. PDF, XLSX\)\. These characteristics—including large, multi\-table schemas and the integration of external knowledge—underscore the alignment with realistic data analysis scenarios\.

Table 2:Statistics of datasets & tasksTask Statistics: Table[2](https://arxiv.org/html/2607.06482#S2.T2)provides a statistical summary of tasks in the benchmark\. The Table QA task includes 211 question sets, which are categorized into 95 simple questions and 116 decomposable questions\. When these decomposable questions are broken down into their constituent parts, the benchmark contains a total of 414 individual questions\. The distribution of sub\-questions per question set is detailed in Figure[3](https://arxiv.org/html/2607.06482#S2.F3)\(c\), which shows that over 55% of all question sets include multiple sub\-questions\. Figure[3](https://arxiv.org/html/2607.06482#S2.F3)\(d\) also presents the distribution of question types\. For the Table Insight task, a curated subset of six datasets, each accompanied by a report from domain experts, is designated for insight generation evaluation\.

![Refer to caption](https://arxiv.org/html/2607.06482v1/x3.png)Figure 3:Distributions of \(a\) number of tables per dataset, \(b\) format of external knowledge, \(c\) number of sub\-questions, and \(d\) question types\.

## 3Experimental Setup

Our benchmark evaluates performance on two distinct tasks:Table QAandTable Insight\. For Table QA, models receive a user question that specifies the desired output format \(text or visualization\)\. In decomposable questions, the preceding conversational history is also provided\. The goal is to produce a precise textual or visual answer\. The Table Insight task challenges models to generate a list of findings and a summary directly from the given files\. Implementation details, including model configurations, hyperparameters, and settings for fair comparison are available in Appendix[0\.B\.1](https://arxiv.org/html/2607.06482#Pt0.A2.SS1)\.

### 3\.1Evaluation Comparisons for Table QA

We evaluate a range of baselines, from general\-purpose LLMs to specialized agents\.

LLM: We evaluate a diverse set of LLMs to investigate the capability of table reasoning\. We selected open\-source models from several categories: general\-purpose \(Llama 3\.1\[grattafiori2024llama3herdmodels\], DeepSeek\-R1\[deepseekai2025deepseekr1incentivizingreasoningcapability\], and Qwen3\[yang2025qwen3technicalreport\]\), code generation\-specific \(Devstral\[devstral\]and Qwen3\-Coder\[qwen3coder\]\), and table\-specific \(TableGPT2\[su2024tablegpt2largemultimodalmodel\]\)\. We also include high\-performance closed\-source models, namely GPT\-4o\[openai2024gpt4ocard\], GPT\-5\.1\[gpt5\.1\], Claude Sonnet 4\.6\[claude4\.6\], and Gemini\-2\.5 Flash\[comanici2025gemini25pushingfrontier\]\.

Answer Agent: We leverage the Answer Agent, designed to robustly generate Python code for answering questions\. The agent is composed of a practical assembly of existing following mechanisms shown in Figure[4](https://arxiv.org/html/2607.06482#S3.F4)\(a\)\.

- •Feature type\-specific table serialization: This module first processes the raw tables using the table serialization based on feature types as detailed in Appendix[0\.A\.2\.2](https://arxiv.org/html/2607.06482#Pt0.A1.SS2.SSS2)\. The resulting structured, textual representation of the data is then utilized as input for the subsequent modules\.
- •Coding: With the serialized table, metadata, external knowledge, and the input question, this module generates Python code to produce an answer\. It incorporates a self\-correction mechanism: if code execution fails, a subsequent LLM is called to revise the code based on the error message with up to three revision attempts allowed\.
- •Reflection: This module addresses cases where successfully executed code produces semantically incorrect outputs \(e\.g\., a visualization with no data points or calculation resulting in NaN\)\. A Vision\-Language Model \(VLM\) or Multimodal Large Language Model \(MLLM\) is employed to analyze visual outputs, while an LLM analyzes textual results\. If an issue is detected, the module triggers a revision loop to refine the code, and this can be repeated to three times\.

We initially performed preliminary experiments with the specialized table agents as baselines, including InfiAgent\-DABench\[infida\-bench\]and tablegpt\-agent\[su2024tablegpt2largemultimodalmodel\]; however, their performance on our benchmark was near\-zero due to the complexity of our benchmark, so they were excluded from the final comparison\.

![Refer to caption](https://arxiv.org/html/2607.06482v1/x4.png)Figure 4:Architecture of \(a\) Answer Agent and \(b\) Insight Agent
### 3\.2Evaluation Comparisons for Table Insight

For the Table Insight task, we evaluate the following baseline agents\. We employ only closed\-source LLMs in these agents, given the relatively low performance of open\-source models on Table QA\.

AgentPoirot\[insight\-bench\]: This agent is designed for goal\-oriented insight generation\. It operates by first extracting the data schema and then generating a set of high\-level questions\. For each question, it generates an answer, interprets it, and recursively poses follow\-up questions to dive deeper, and finally summarizes the obtained insights\. This process follows a tree\-like exploration structure \(Figure[9](https://arxiv.org/html/2607.06482#Pt0.A2.F9)\(a\)\), where each branch represents a dive into a analytical path\.

Insight Agent: We employ the Insight Agent shown in Figure[4](https://arxiv.org/html/2607.06482#S3.F4)\(b\) that extends the core ideas of AgentPoirot in a applicable way\. Similar to AgentPoirot, it iteratively generates questions, produces answers, and derives insights\. The agent begins by generating the fixed number of high\-level questions, which are then processed by our Answer Agent to obtain correct answers\. Insights are subsequently synthesized from multiple QA pairs\. These initial insights then seed the generation of new follow\-up questions by combining multiple insights, continuing the cycle, ending by the summarization\. Unlike AgentPoirot’s tree\-structured approach, the Insight Agent employs a directed acyclic graph \(DAG\)\-based approach as explained in Figure[9](https://arxiv.org/html/2607.06482#Pt0.A2.F9)\(b\), as the generation of new questions and insights selects and aggregates the context from all previously generated information instead of single insight or QA pair in the previous depth\.

### 3\.3Evaluation Metrics

To assess the performance of agents and LLMs on our benchmark, we compare their outputs against the ground\-truth references\. Distinct evaluation protocols are employed for Table QA and Table Insight tasks\.

Table QA: The QA task is evaluated on accuracy under two settings:Whole, where all sub\-questions in a decomposable question must be correct, andIndividual, which measures sub\-question\-level accuracy\. Correctness is determined by the modality of the answer\. Text\-based answers are judged by Exact Match \(EM\)\. Visualizations are evaluated using an MLLM\-as\-a\-judge protocol, where four MLLMs \(GPT\-4o, GPT\-4o\-mini, Gemini 2\.5 Flash, and Gemini 2\.5 Pro\[comanici2025gemini25pushingfrontier\]\) assess the semantic equivalence between the predicted and ground\-truth outputs\. The judges are provided both the rendered images and their source code, and a prediction is deemed correct upon a majority consensus, requiring positive assessments from at least three of the four models\.

Table Insight: To evaluate the quality of generated insights, we adopt the methodology from InsightBench\[insight\-bench\]computing LLaMA\-3\-Eval scores\. We employ GPT\-4o as the evaluator by replacing LLaMA3\-70b\[grattafiori2024llama3herdmodels\]\. We verified that the evaluator aligns with human perception through a meta\-evaluation, as shown in Appendix[0\.B\.3](https://arxiv.org/html/2607.06482#Pt0.A2.SS3)\. The evaluation is conducted at two levels of granularity:

- •Summary\-level Score: This metric assesses the holistic quality of the entire set of generated insights by comparing it against the ground\-truth summary\.
- •Insight\-level Score: This metric provides a more fine\-grained analysis\. It measures the semantic alignment between each individual ground\-truth insight and the most relevant insight from the predicted set, with the final score being the average of these individual comparisons\.

### 3\.4Implementation Details

Answer Agent requires a VLM or a MLLM within its Reflection Module to validate visual outputs\. For experiments involving closed\-source models, we utilized their native multimodal capabilities across all modules\. For the open\-source agent configurations, we paired various LLMs with a specialized VLM, Chart\-R1\-7B\[chen2025chartr1\]\. For the Table Insight task, each experiment was executed five times per model, and the scores were averaged across these runs to ensure the stability of our result\. All results were obtained with the model temperature set to 0\.0 to promote deterministic output\. The other details about LLM configurations and hyperparameters are available in Appendix[0\.B\.1](https://arxiv.org/html/2607.06482#Pt0.A2.SS1)\.

## 4Evaluation Results

Table 3:Main results of Table QA and Table Insight with LLMs and specific agents\.w/o Answer Agentis without the agentic support\.### 4\.1Quantitative Evaluation Results

The main results for the Table QA and Table Insight tasks are presented in Table[3](https://arxiv.org/html/2607.06482#S4.T3), wherew/o Answer Agentmeans that the LLM is executed to generate the Python codes once based on the first 10 rows of the tables instead of the specific serialization\. For the Table QA task, LLMs with the agentic support by Answer Agent consistently outperforms the base LLMs\. With Gemini 2\.5 Flash, for instance, it achieves relative improvements of approximately 27% in the Whole setting and 25% in the Individual setting\. This suggests that a structured agentic framework is crucial, as standalone LLM reasoning is insufficient for such complex tasks\. Despite these gains, the top absolute score in the Whole setting remains below 0\.4, highlighting the difficulty of the benchmark\. Among the open\-source models, Qwen3\-30B achieves the highest score in the Whole setting, while Devstral\-small performs best in the Individual setting\. However, their performance still lags behind that of the closed\-source models\. Notably, TableGPT2\-7B, a model specialized for tabular data, scores below 0\.1 even when paired with our Answer Agent\.

In the Table Insight task, Claude Sonnet 4\.6 has the best contributions to the agentic approach, achieving 0\.338 for the Insight\-level score by AgentPoirot and 0\.453 for the Summary\-level score by Insight Agent\. However, the absolute scores remain low even with the top\-performing model \(below 0\.35 for the Insight\-level score and below 0\.5 for the Summary\-level score\), indicating substantial challenges remain in automated insight generation\.

### 4\.2Qualitative Analysis

#### 4\.2\.1Error Analysis of Table QA

We conducted an error analysis on the incorrect answers from Gemini 2\.5 Flash with Answer Agent, and the results are categorized in Figure[5](https://arxiv.org/html/2607.06482#S4.F5)\.

The most prevalent issue,Condition Filter Error \(32\.4%\), occurs when the model fails to apply implicit conditions not explicitly stated in the question\. A common example involves datasets with aggregated and disaggregated data \(e\.g\., population counts for ‘male’, ‘female’, and ‘total’\); models often fail to apply appropriate filters to avoid double\-counting, leading to incorrect calculations\. The second most frequent category isTransformation Error \(23\.2%\), which involves failures in data wrangling\. Common mistakes include parsing\-related failures various datetime formats \(e\.g\. day\-first format\) by using`pandas\.to\_datetime`method or neglecting to convert numerical strings \(e\.g\., "1,234,567"\) into integer or float types\. To mitigate these errors, the more comprehensive yet efficient view of tables \(e\.g\. exploratory data analysis results\) is required in addition to the feature type\-based representations for future work\.

![Refer to caption](https://arxiv.org/html/2607.06482v1/x5.png)Figure 5:Error distribution with Answer AgentErrors also arise from the inherent complexity of the tasks\.Context Handling Errors \(8\.7%\)occur in decomposable questions where the model incorrectly uses the output from a flawed previous turn though the generated logic is correct in most cases\.Visualization Errors \(6\.8%\)typically involve incorrect axis ranges, such as a timeline that does not match the period specified in the question\. Finally, the complexity of the benchmark’s data structure leads to specific errors\. These includeWrong Choice of Tables \(4\.3%\)in multi\-table scenarios\. This error occurs when the generated code fails to select the correct table from a multi\-table dataset based on information provided in the metadata\. For example, a dataset may contain separate tables for annual statistics, with the year covered by each table specified only in the metadata\. An error arises if a question pertains to a specific year, but the model fails to refer to the metadata and consequently queries the wrong table\.Misunderstanding External Knowledge \(3\.9%\), where the model fails to correctly apply information from provided data dictionaries to interpret the data\.

#### 4\.2\.2Fine\-grained Analysis of Table Insight

![Refer to caption](https://arxiv.org/html/2607.06482v1/x6.png)Figure 6:Insight perspective distributions by Insight Agent and AgentPoirot\.While the metrics provides a single value to measure the alignment between predicted and target insights, we conduct a more fine\-grained analysis to understand model capabilities\. We decompose the evaluation into four distinct perspectives:Topic Relevance\(Does the predicted insight address the same topic as the target?\),Narrative Alignment\(Does the prediction make the same core argument or conclusion as the target?\),Qualitative Details Match\(Does the prediction mention the same specific names or entities as the target?\), andQuantitative Details Match\(Does the prediction mention the same specific quantitative values as the target?\)\. For each of the 280 pairs that are randomly sampled11155executions×\\times5656GT insights \(10 per 5 datasets, 6 for 1 dataset\), we prompted GPT\-4o to score the prediction on each perspective using a 1–5 scale\. The score distributions for both Insight Agent and AgentPoirot are presented in Figure[6](https://arxiv.org/html/2607.06482#S4.F6)\.

The results show that both agents perform relatively well on Topic Relevance, the highest\-level criterion\. However, their performance drops on Narrative Alignment and Qualitative Details Match, where high scores \(3–5\) are less frequent\. This suggests that although the agents identify the general topic of the target insight, they frequently fail to reproduce the core argument or the qualitative entities that support it\. As a result, many generated insights lack the fundamental elements needed to convey the intended conclusion, indicating that the predictions remain misaligned with the target insights\. This interpretation is supported by Table[11](https://arxiv.org/html/2607.06482#Pt0.A2.T11)where the generated insights lacking the correct core argument and qualitative entities receive lower scores than those that include them\. Finally, both agents fail on Quantitative Details Match, with none of the predicted insights scoring 3 or higher\. This highlights that both agents lack the ability to accurately calculate and present specific numerical values in their generated insights\.

These findings reveal two missing capabilities of LLM agents when handling real\-world tables: narrative\-level reasoning and accurate fact retrieval\. For the former, both agents struggle to identify the central analytical argument underlying the target insight, suggesting that stronger reasoning over complicated tabular data is required\. The latter capability is closely related to Table QA; therefore, the same observations discussed in the previous subsection apply here\. In particular, the results suggest that a more comprehensive yet compact representation of tables is necessary to reduce various types of errors\.

### 4\.3Ablation Study

#### 4\.3\.1Components of Answer Agent

Table 4:Ablation Study of Answer AgentTo assess the contribution of each Answer Agent module quantitatively, we conducted an ablation study against a naive baseline using the first 10 table rows \(Table[4](https://arxiv.org/html/2607.06482#S4.T4)\) for the Python code generation\. Adding the table schema consisting of column names, data types, and statistics on missing and unique values does not affect the performance\. We further increased the number of provided rows whose number was increased up to a maximum of 200, adjusted as needed to fit within the LLM context window\. The performance degraded noticeably, suggesting that LLMs struggle to identify the key information needed for solution construction when faced with a larger pool of numerical or textual entries, compared to the first 10 rows\. In contrast, the feature type\-specific serialization led to gains over the baseline by providing a compact and reliable representation of the table that avoids the baseline’s tendency to rely on guessed values\. This finding further supports the idea that a more comprehensive and efficient representation of tables leads to higher performance, as discussed in Section[4\.2\.1](https://arxiv.org/html/2607.06482#S4.SS2.SSS1)\. Adding the Reflection module further improved performance across models by enabling the system to capture implicit constraints and infer missing reasoning steps\.

#### 4\.3\.2SQL vs\. Python for Table Question Answering

Table 5:SQL vs\. PythonText\-to\-SQL has been a primary approach for tackling table question answering, where natural language questions are translated into SQL queries\. SQL is widely used for efficient data manipulation and retrieval, making it meaningful to investigate whether SQL is a more suitable output format than Python code for LLM\-based table reasoning\. In this study, we exclude visualization\-based QA pairs from DataGovBench, as SQL does not provide native functions for generating visualizations\. As a result, the evaluation is conducted on 145 text\-based QA pairs\. The LLM prompt includes feature type\-specific serialization, as information about feature types and data types helps the model construct proper SQL queries\. As shown in Table[5](https://arxiv.org/html/2607.06482#S4.T5), the results indicate that SQL performs significantly worse than Python code\. Error analysis of the generated SQL queries reveals that syntax errors and incorrect column selections occur more frequently than in generated Python code\.

## 5Related Works

Benchmarks for Table Question and Answering\. Research in table\-based question answering has been largely driven by a series of influential benchmarks\. Early work such as WikiTableQuestions\[wtq\]established the task of answering natural language questions over Wikipedia tables\. While foundational, this benchmark is limited to single tables and relatively simple questions\. Although benchmarks like FeTaQA\[nan\-etal\-2022\-fetaqa\]and OTT\-QA\[ottqa\]introduced tasks requiring more complicated reasoning, they still focus on small\-scale Wikipedia tables\. The subsequent development of text\-to\-SQL benchmarks marked a significant leap in complexity\. Spider\[yu\-etal\-2018\-spider\]and BIRD\[bird\]became the standards requiring models to generate complex SQL queries\. More recently, developments in LLMs have enabled models to generate coherent Python code, leading to the construction of benchmarks that assess data analysis capabilities\[wu2025tablebench,infida\-bench,oses\-grijalba\-etal\-2024\-question\]\. While these benchmarks were instrumental in advancing table reasoning capabilities, they do not comprehensively capture the challenges of real\-world data analysis\. The datasets are typically well\-structured in scale\. They often lack practical characteristics, such as rich metadata and supplementary external knowledge\. Furthermore, many do not address the large multi\-tabular datasets\. DataGovBench comprehensively targets these limitations by incorporating all of these features: large tables, multi\-table schemas, metadata, and extenral knowledge\.

Benchmarks for Table Insight Generation\. Automated insight generation is a nascent and challenging area to benchmark, as the subjective nature of an "insight" complicates objective evaluation\[data2dashboard,majumder2024discoverybenchdatadrivendiscoverylarge\]\. Recent work like InsightBench\[insight\-bench\]and T2R\-Bench\[zhang\-etal\-2025\-t2r\]have advanced this area by introducing benchmarks for table insight generation\. However, they do not fully capture the complexities of real\-world data analysis\. InsightBench focuses primarily on small\-scale tables, while T2R\-Bench lacks supporting metadata and external knowledge that are often required to interpret tabular data\. In contrast, DataGovBench provides a more comprehensive benchmark by incorporating large\-scale collections of tables together with relevant external knowledge sources\.

## 6Conclusion

To address the critical lack of realism in existing benchmarks, we introducedDataGovBench, a new benchmark built from open data\. It features large, multi\-tabular datasets, and incorporates external knowledge, and formalizes two key tasks: complex Question Answering \(with decomposable questions and visualizations\) and a novel Insight Generation task grounded in reports by domain specialists\. Our extensive evaluation reveals that even state\-of\-the\-art models with and without the agentic support struggle with low QA accuracy, highlighting the difficulity of both tasks\. A detailed qualitative analysis provides a clear roadmap for future research\. Future efforts should focus on both addressing these issues and efficiently scaling the benchmark’s size to ensure more robust evaluations\. We believe DataGovBench will serve as a catalyst steering research toward building more robust agents capable of handling real\-world data complexities\.

#### 6\.0\.1Use of Large Language Models

We used LLM for the grammar correction and the words refinement to enhance the quality of the paper\.

## References

## Appendix 0\.ABenchmark Construction

### 0\.A\.1Data Curation

#### 0\.A\.1\.1Data Collection

Table[6](https://arxiv.org/html/2607.06482#Pt0.A1.T6)lists 53 websites, from which we downloaded all datasets along with their metadata via APIs \(e\.g\., CKAN API222[https://github\.com/ckan/ckanapi](https://github.com/ckan/ckanapi)\)\.

#### 0\.A\.1\.2Data Filtering

All downloaded datasets were subjected to a rigorous filtering and curation process to select those most suitable for real\-world data analytics tasks\. This process involved three main stages: dataset filtering, external knowledge identification, and metadata standardization\. First, data filtering is conducted based on the following conditions:

- •The dataset had to contain at least one CSV file that was correctly formatted and readable by the`pandas\.read\_csv`method\. Files that were HTML or XML in content despite having a \.csv suffix were excluded\.
- •At least one CSV file within the dataset was required to have more than 5,000 rows\. Additionally, tables with five or more blank columns were discarded\.
- •Each dataset needed to be accompanied by a textual description\. The license was also required to permit redistribution; for datasets from Data\.gov where the license was often unspecified in the metadata, we performed manual verification on the source webpage\.
- •ArcGIS\-based datasets, which are primarily geospatial, were excluded from our analysis\.

Following the filtering stage, we systematically searched for external knowledge \(e\.g\., data dictionaries\) within each dataset using a set of heuristic rules:

- •First, an automated search was performed for files with names containing "data dictionary" or "datadictionary"\.
- •Next, a platform\-specific rule was applied for Data\.gov datasets\. We observed that when a JSON file is provided alongside CSV, XML, and RDF files, it often contains column\-level descriptions\. In such cases, the JSON file was designated as external knowledge\.
- •If these automated heuristics failed, we performed a manual inspection of the dataset’s contents to locate any other supplementary documentation that could serve as a data dictionary\.

Finally, the original metadata for each curated dataset was processed and standardized\. This step created a concise metadata format specific to our benchmark by removing redundant or irrelevant information from the source\. The following is an example of the specific metadata fromIndiana Arrest DataofIndiana Data Hub\.

Converted Metadata"identifier":"d39f6598\-efbb\-40a7\-a694\-6a9b8d2dc2dc"

"dataset\_title":"INDIANAARRESTDATA"

"dataset\_description":"ThisdatasetistheunderlyingdataoftheIndianaArrestsDashboardwhichdisplayscountsofindividualsarrested,arrests,chargesbyoffensecategory,dispositions,countryandtimeperiodinIndianabeginningin2008throughthepresentyear\.\\r\\n\\r\\nArrestdatacomesfromtheCriminalHistoryRepositorySystem\(CHRIS\)\.DatafeedingintotheCHRISsystemcomesfromthreemainsources\.ArrestdatacomesfromtheLiveScansystem,whichisusedforfingerprintingandcapturingotherpertinentinformationatthetimeofthearrest\.CriminaldispositiondataaremaintainedbyprosecutorsinProsLinksystem,andbythecourtsintheOdysseysystem\.\\r\\n\\r\\nDataNotes:\\r\\n\\r\\n1\.ArrestdataaresenttoISPsoonafterthearrestoccurs,butdispositiondatahavealagofapproximatelysevenmonthsasthecasemakesitswaythroughthelegalsystem\.\\r\\n\\r\\n2\.Textdescriptionsoftheoriginaloffensesareprovidedbythearrestingofficerwhentheoffenderisarrested\.Later,theprosecutor’sofficeorcourtprovidesatextdescriptionofthefiledoffense,alongwiththeIndianaCodetitle,article,chapter,andsection\(e\.g\.35\-48\-4\-6\)\.Thefiledoffensemaybeamendedlater\.\\r\\n\\r\\n3\.ArrestCountyisdeterminedbythelocationofthebookingagency\.Ifthebookingagencyismissing,thenthearrestingagencyisused\.\\r\\n\\r\\n4\.Thecountofindividuals/arrests/chargesbyoffensecategorycanadduptomorethanthegrandtotalbecauseoneindividual/arrest/chargecanfallintomultiplecategories\(e\.g\.DUIiscountedinthe\\"Drug\\"and\\"Traffic\\"categories\.\\r\\n\\r\\n5\.Arrestcategoriesandsubcategoriesaredeterminedbasedonkeywordsfoundinafreetextdescriptionoftheoffense\.About7%ofoffenseshaveadescriptionthathasnotyetbeencategorized\."

"publisher":"IndianaStatePolice"

"landingPage":"IndianaStatePolice"

"license":"CreativeCommonsAttribution"

"distribution":\[\{"file\_name":"data9\.csv",

"file\_title":"ARRESTDATA2022Q3",

"file\_description":null,

"downloadURL":"https://hub\.mph\.in\.gov/dataset/d39f6598\-efbb\-40a7\-a694\-6a9b8d2dc2dc/resource/00cd698d\-e26b\-458a\-861b\-4c355b77ab20/download/isp\_arrest\_data\_2022\_q3\.csv",

"accessURL":"https://hub\.mph\.in\.gov/dataset/d39f6598\-efbb\-40a7\-a694\-6a9b8d2dc2dc/resource/00cd698d\-e26b\-458a\-861b\-4c355b77ab20/download/isp\_arrest\_data\_2022\_q3\.csv"\},

\{"file\_name":"data37\.csv",

"file\_title":"ARRESTDATA2015Q3",

"file\_description":null,

"downloadURL":"https://hub\.mph\.in\.gov/dataset/d39f6598\-efbb\-40a7\-a694\-6a9b8d2dc2dc/resource/8b2b54fe\-363a\-46f7\-9c3b\-197cce01616f/download/isp\_arrest\_data\_2015\_q3\.csv",

"accessURL":"https://hub\.mph\.in\.gov/dataset/d39f6598\-efbb\-40a7\-a694\-6a9b8d2dc2dc/resource/8b2b54fe\-363a\-46f7\-9c3b\-197cce01616f/download/isp\_arrest\_data\_2015\_q3\.csv"\},

\{"file\_name":"data20\.csv",

"file\_title":"ARRESTDATA2019Q4",

"file\_description":null,

"downloadURL":"https://hub\.mph\.in\.gov/dataset/d39f6598\-efbb\-40a7\-a694\-6a9b8d2dc2dc/resource/bd011a33\-0652\-4ad7\-8d90\-6c1019d6385c/download/isp\_arrest\_data\_2019\_q4\.csv",

"accessURL":"https://hub\.mph\.in\.gov/dataset/d39f6598\-efbb\-40a7\-a694\-6a9b8d2dc2dc/resource/bd011a33\-0652\-4ad7\-8d90\-6c1019d6385c/download/isp\_arrest\_data\_2019\_q4\.csv"\},

\{"file\_name":"data15\.csv",

"file\_title":"ARRESTDATA2021Q1",

"file\_description":null,

"downloadURL":"https://hub\.mph\.in\.gov/dataset/d39f6598\-efbb\-40a7\-a694\-6a9b8d2dc2dc/resource/9c7960c6\-417b\-45e6\-9ace\-b75958dd91de/download/isp\_arrest\_data\_2021\_q1\.csv",

"accessURL":"https://hub\.mph\.in\.gov/dataset/d39f6598\-efbb\-40a7\-a694\-6a9b8d2dc2dc/resource/9c7960c6\-417b\-45e6\-9ace\-b75958dd91de/download/isp\_arrest\_data\_2021\_q1\.csv"\},

\{"file\_name":"data14\.csv",

"file\_title":"ARRESTDATA2021Q2",

"file\_description":null,

"downloadURL":"https://hub\.mph\.in\.gov/dataset/d39f6598\-efbb\-40a7\-a694\-6a9b8d2dc2dc/resource/1ff2cf5f\-69ef\-4139\-bcb4\-036f66787172/download/isp\_arrest\_data\_2021\_q2\.csv",

"accessURL":"https://hub\.mph\.in\.gov/dataset/d39f6598\-efbb\-40a7\-a694\-6a9b8d2dc2dc/resource/1ff2cf5f\-69ef\-4139\-bcb4\-036f66787172/download/isp\_arrest\_data\_2021\_q2\.csv"\}\],

"external\_knowledge":\["data68\.xlsx"\]\}

Table 6:List of Open Data Websites

### 0\.A\.2Data Annotation

#### 0\.A\.2\.1Question Types

During question generation, specific question types are provided to the LLM to guide the formulation of questions\. We use the following eight question types, with each type’s name and description supplied to the LLM\.Multi\-turn Follow\-upandMulti\-turn Insight Generationcorrespond to decomposable questions\.

- •Aggregation: Questions involving aggregated answers based on the statistical operations, such as mean, sum or mode
- •Ranking: Questions involving answers based on the ranking
- •Counting: Questions involving answers based on counting something
- •Multi\-hop Lookup: Question involving extracting single cell value from the table based on multiple reasoning steps
- •Multi\-hop Numerical Reasoning: Questions involving numerical answers based on multiple reasoning steps
- •Complex Data Transformation: Question involving complex data transformation, such as aggregation or filtering across multiple dimensions, creating new columns, filtering with context\-dependent logic, resolving entity references across rows, or merging multiple tables
- •Multi\-turn Follow\-up: Question involving multi\-turn follow\-up questions that build on previous answers or context from table data, requiring the model to maintain state and context across multiple interactions
- •Multi\-turn Insight Generation: Question involving multi\-turn insight generation that requires the model to generate insights or summaries based on previous answers or context from table data, requiring the model to maintain state and context across multiple interactions\. Questions in the intermediate turn ask to provide not only text\-based answer but also text\-based complicated statistical information \(e\.g\. correlation\) and visualization

#### 0\.A\.2\.2Feature type\-Specific Table Serialization

Our serialization process generates a compact textual representation of a table by summarizing its global properties and providing detailed, feature type\-aware information for each column\. The serialized text begins with the dimensions of the tables \(number of rows and columns\), followed by a per\-column breakdown\. For each column, the serialization includes: the inferred feature type, the Pandas data type\[mckinney\-proc\-scipy\-2010\], the percentage of NaN values, and a feature\-specific textual summary\. The primary feature type is determined by a Feature Type Inference \(FTI\)\. Our FTI module follows a design similar to that in\[autodw\]\. This model classifies each column into one of 11 types:Numerical,Categorical,Datetime,Sentence,URL,Embedded Number,List,Ignorable ID,Numbers with Unit,Numbers with Sign,Range of Numbers, orFormatted ID\. The Pandas data type is inferred using the built\-in`pandas\.api\.types\.infer\_dtype`function\. While its output would overlap with the feature types, we include it because its ability to identify mixed types \(e\.g\., columns containing both strings and integers\) serves as a key signal for potential data quality issues that would require wrangling\.

The feature type\-specific summary is constructed according to the inferred feature type, as follows:

- •Numerical: The minimum and maximum values in the column are included\.
- •Categorical: If the column contains 20 or fewer unique categories, all are listed\. Otherwise, a random sample of 20 unique categories is provided\.
- •Datetime: The earliest and latest date or time values are included\.
- •URL: No sample values are included\. This is a deliberate choice to conserve context length, as full URLs are token\-intensive and typically have low semantic value for general data analysis tasks\.
- •All Other Types: For all other feature types, a random sample of 10 unique values is included to provide a representative snapshot of the column’s contents\.

The example of the serialized text is provided in the following fromE\-bike Field StudyofData\.gov\.

Text Example by Feature type\-specific Table Serialization1sttable

Datasettitle:CommaSeparatedValuesFile

Datasetdescription:None

Headersandvalues:

Numberofcolumns:21

Numberofrows:408363

Featuretype,pandastype,ratioofmissingvalues,andfeaturetype\-specificinformationisgivenforeachcolumnasbelow\.

date\(featuretype:Datetime\)\(pandastype:string\)\(ratioofmissingvalues:0%\):Startdateis2022\-04\-2723:42:29\.834000\+00:00,andenddateis2022\-09\-2318:31:05\.502000\+00:00\.

lat\(featuretype:Numerical\)\(pandastype:floating\)\(ratioofmissingvalues:0%\):Valuerangeis\[42\.447303,42\.461437200000006\]\.

lon\(featuretype:Numerical\)\(pandastype:floating\)\(ratioofmissingvalues:0%\):Valuerangeis\[\-71\.3243906,\-71\.2562746\]\.

spd\(featuretype:Numerical\)\(pandastype:floating\)\(ratioofmissingvalues:0%\):Valuerangeis\[0\.0,23\.825000000000003\]\.

blind\_turn\(featuretype:Categorical\)\(pandastype:integer\)\(ratioofmissingvalues:0%\):Allcategoriesare\[0,1\]\.

constrained\_tunnel\(featuretype:Categorical\)\(pandastype:integer\)\(ratioofmissingvalues:0%\):Allcategoriesare\[0,1\]\.

narrow\(featuretype:Categorical\)\(pandastype:integer\)\(ratioofmissingvalues:0%\):Allcategoriesare\[0,1\]\.

slow\_sign\(featuretype:Categorical\)\(pandastype:integer\)\(ratioofmissingvalues:0%\):Allcategoriesare\[0,1\]\.

trail\_hazards\(featuretype:Categorical\)\(pandastype:integer\)\(ratioofmissingvalues:0%\):Allcategoriesare\[0,1\]\.

trail\_junction\(featuretype:Categorical\)\(pandastype:integer\)\(ratioofmissingvalues:0%\):Allcategoriesare\[0,1\]\.

vehicle\_conflict\_point\(featuretype:Categorical\)\(pandastype:integer\)\(ratioofmissingvalues:0%\):Allcategoriesare\[0,1\]\.

walk\_bike\_sign\(featuretype:Categorical\)\(pandastype:integer\)\(ratioofmissingvalues:0%\):Allcategoriesare\[0,1\]\.

eb\(featuretype:Categorical\)\(pandastype:integer\)\(ratioofmissingvalues:0%\):Allcategoriesare\[0,1\]\.

uphill\(featuretype:Categorical\)\(pandastype:integer\)\(ratioofmissingvalues:0%\):Allcategoriesare\[0,1\]\.

downhill\(featuretype:Categorical\)\(pandastype:integer\)\(ratioofmissingvalues:0%\):Allcategoriesare\[0,1\]\.

passing\(featuretype:Categorical\)\(pandastype:integer\)\(ratioofmissingvalues:0%\):Allcategoriesare\[0,1\]\.

participantid\(featuretype:Numerical\)\(pandastype:integer\)\(ratioofmissingvalues:0%\):Valuerangeis\[1,37\]\.

age\(featuretype:Numerical\)\(pandastype:integer\)\(ratioofmissingvalues:0%\):Valuerangeis\[27,65\]\.

sex\(featuretype:Categorical\)\(pandastype:string\)\(ratioofmissingvalues:0%\):Allcategoriesare\[female,male\]\.

bike\_type\(featuretype:Categorical\)\(pandastype:string\)\(ratioofmissingvalues:0%\):Allcategoriesare\[conventional,electric\]\.

ebike\_class\(featuretype:Categorical\)\(pandastype:floating\)\(ratioofmissingvalues:56%\):Allcategoriesare\[1\.0,2\.0,3\.0\]\.

#### 0\.A\.2\.3Human Verification

Figure[7](https://arxiv.org/html/2607.06482#Pt0.A1.F7)shows the annotation GUI, built with Streamlit333[https://github\.com/streamlit/streamlit](https://github.com/streamlit/streamlit), for revising questions and the Python code used to generate answers\. Annotators can refer to the LLM\-generated answers and code, as well as the underlying tables, metadata, and external knowledge\. Figure[8](https://arxiv.org/html/2607.06482#Pt0.A1.F8)presents the GUI for reviewing revised QA pairs, where annotators select one of three options—–Good,Ambiguous, orWrong Answer—–and may leave comments for the latter two\. In total, we obtained 211 datasets, with their original website distribution shown in Table[8](https://arxiv.org/html/2607.06482#Pt0.A1.T8)\. Six of these datasets \(Table[9](https://arxiv.org/html/2607.06482#Pt0.A1.T9)\) are used for the Table Insight task\.

![Refer to caption](https://arxiv.org/html/2607.06482v1/figures/annotate_revision.png)Figure 7:Annotation GUI for revising questions and answers![Refer to caption](https://arxiv.org/html/2607.06482v1/figures/annotate_check.png)Figure 8:Annotation GUI for checking the revised QA pairs

### 0\.A\.3Distribution of Discarded Candidate Questions

We generated 1,840 candidate questions after the question scoring stage and curated them into 211 high\-quality questions, discarding the remaining 1,629 based on the criteria below\.

- •Unanimous Agreement: During the answer generation stage, we employed four LLMs to produce answers and measured consensus across models\. Questions for which all LLMs produced identical answers were removed because they typically require only shallow reasoning and do not align with the intended complexity of our benchmark\. Answer consensus was determined using exact string match for text\-based answers and manual verification for visualization\-based answers using the GUI tool shown in Figure[7](https://arxiv.org/html/2607.06482#Pt0.A1.F7)\.
- •Insufficient External Knowledge: Some datasets require external knowledge \(e\.g\. data dictionary\) to interpret column meanings or specific values\. When such information was missing or insufficient, it became impractical to map column names and values to the generated questions, making accurate answer generation infeasible\. Questions of this type were removed\.
- •Ambiguous Question: Questions allowing multiple plausible answers were removed to ensure benchmark clarity\. For example, when a table contains both calendar\-year and fiscal\-year columns, a question asking for “the year” satisfying certain conditions becomes ill\-posed unless the question explicitly specifies which type of year should be used\.
- •Uninsightful Question: We excluded questions that failed to yield analytically meaningful or practically useful insights despite being answerable\. A common example occurs in geospatial datasets, where questions such as “What is the average latitude and longitude under certain conditions?” often result in a coordinate that lacks geographic or analytical relevance \(e\.g\., a point in the ocean\)\. Such questions were deemed non\-insightful and removed\.

The distribution of the reasons is shown in Table[7](https://arxiv.org/html/2607.06482#Pt0.A1.T7)\.

Table 7:Distribution of Discarding ReasonsTable 8:Distribution of Open Data Websites in DataGovBenchTable 9:Datasets for Table Insight

## Appendix 0\.BExperimental Setup

### 0\.B\.1Implementation Details

All open\-source models are sourced from the HuggingFace’s`transformers`library\[wolf2020huggingfacestransformersstateoftheartnatural\], and experiments were conducted using 2 × 48 GB NVIDIA L40S GPUs\. Table[10](https://arxiv.org/html/2607.06482#Pt0.A2.T10)lists the API names of closed\-source models and the HuggingFace model names of open\-source models\.

Table 10:List of LLM model names in the experimentsInsight Agent was configured to generate three initial high\-level questions and perform four iterations of its question\-answer\-insight cycle, resulting in 12 insights\. Also, the AgentPoirot baseline was configured with parameters that also yielded 12 insights\. Furthermore, the summarizing LLM in Insight Agent is instructed to generate the same number of tokens as the summarized sentences from AgentPoirot for a fair comparison\.

### 0\.B\.2Insight Agent

The Insight Agent operates through an iterative cycle: it generates questions, answers them using Answer Agent, and then synthesizes insights from the resulting QA pairs\. The insights generated in one step are then used to inform the question generation in the next, creating a continuous exploratory process\. This process is governed by a Directed Acyclic Graph \(DAG\) structure, as illustrated in Figure[9](https://arxiv.org/html/2607.06482#Pt0.A2.F9)\(b\)\. The graph consists of alternating layers of Question\-Answer \(QA\) nodes and Insight nodes\. A new Insight node is generated by synthesizing information from one or more preceding QA nodes, and conversely, a new QA node is generated by drawing upon one or more preceding Insight nodes\. Crucially, a new node can be connected to parent nodes from any previous iteration, not just the immediately preceding one\. This DAG structure facilitates the aggregation of multiple lines of inquiry, enabling the generation of more diversified and comprehensive insights compared to a simpler tree\-based exploration as in Figure[9](https://arxiv.org/html/2607.06482#Pt0.A2.F9)\(a\), where insights from different depths and branches are not connected\. The decision of which nodes to aggregate is determined by the reasoning capabilities of LLM; to guide this process, our prompt explicitly instructs the model to consider synthesizing information from multiple parent nodes when possible\.

![Refer to caption](https://arxiv.org/html/2607.06482v1/x7.png)Figure 9:Insight generaion process by \(a\) tree\-based process used in AgentPoirot, where each question is selected by LLM from the question candidates and \(b\) directed acyclic graph\-based process used in Insight Agent
### 0\.B\.3Meta\-evaluation of Table Insight

We conducted a meta evaluation to assess the validity of the evaluation metrics for Table Insight\. We sampled 50 pairs of generated insights \(Insight A and Insight B\) for each ground\-truth insight and asked three independent annotators to assess which of the two was closer to the ground truth\. Annotators selected one of five relative options:A\+\(A is definitely better\),A\(A is slightly better\),N\(A and B are comparable\),B\(B is slightly better\), orB\+\(B is definitely better\)\. Each option was mapped to a normalized score in \[\-2, \-1, 0, 1, 2\]\. The final human\-judgment score for each pair was obtained by averaging the three annotators’ scores\. In parallel, we computed a metric\-derived score based on the difference between the insight\-level scores of B and A\. We then quantified the agreement between human judgments and the metric\-derived scores using both Pearson and Spearman correlations, which yielded coefficients of 0\.669 and 0\.663, respectively\. These results indicate a solid alignment between the proposed metric and human judgments, providing empirical support for the reliability of our evaluation methodology for the Table Insight task\.

Table 11:Apple\-to\-apple comparison among GT insight, insight from Table Insight, and one from AgentPoirot\.