LakeQuest: A Three-Domain Benchmark for Grounded Question Answering across Data Lakes

# A Three-Domain Benchmark for Grounded Question Answering across Data Lakes
Source: [https://arxiv.org/html/2607.12310](https://arxiv.org/html/2607.12310)
###### Abstract

While modern question answering \(QA\) systems excel on clean, schema\-aligned corpora, real\-world knowledge is rarely so neatly packaged\. Answering questions over enterprise and scientific data lakes requires systems to navigate heterogeneous, weakly structured collections of tables, passages, and linked metadata\. Current benchmarks abstract away this noisy discovery process, failing to evaluate end\-to\-end performance\. To bridge this gap, we introduceLakeQuest, a human\-validated benchmark of 9,846 QA pairs designed to evaluate the end\-to\-end retrieve\-and\-synthesize pipeline over realistic data lakes\.LakeQuestspans three diverse domains—AI/ML metadata, retail banking, and multimodal biomedical drug information—and pairs every question with exact, modality\-aware evidence pointers\. By isolating source discovery from cross\-modal synthesis,LakeQuestexposes critical failure modes in modern QA systems\. Our baseline evaluations, including standard Retrieval\-Augmented Generation \(RAG\) and agentic tool\-use methods, reveal that high\-quality retrieval does not guarantee correct reasoning\. Systems consistently struggle with relation chaining in metadata graphs, policy grounding in bank ledgers, and joint tabular QA in biomedical contexts, highlighting the need for robust discovery and faithful cross\-file composition mechanisms in future agentic QA systems\.

## 1Introduction

Question answering \(QA\) systems have achieved strong performance on curated, homogeneous corpora, ranging from reading comprehension\(Rajpurkaret al\.,[2016](https://arxiv.org/html/2607.12310#bib.bib13)\)to open\-domain text retrieval\(Kwiatkowskiet al\.,[2019](https://arxiv.org/html/2607.12310#bib.bib14)\)and multi\-hop reasoning\(Yanget al\.,[2018](https://arxiv.org/html/2607.12310#bib.bib1)\)\. However, real\-world enterprise and scientific knowledge is rarely stored as flat text\. Instead, it often resides indata lakes: heterogeneous, weakly structured collections of tables, passages, and linked metadata\.

Answering natural language questions over data lakes requires navigating siloed sources, combining cross\-modal evidence, and maintaining explicit provenance\. However, modern retrieve\-and\-synthesize architectures\(Lewiset al\.,[2020](https://arxiv.org/html/2607.12310#bib.bib2); Yaoet al\.,[2023](https://arxiv.org/html/2607.12310#bib.bib7)\)lack evaluation environments that simultaneously stress discovery, cross\-source composition, and faithful attribution over heterogeneous corpora\. Existing benchmarks bypass the noise, scale, and weak metadata characteristic of real data lakes, evaluating either pure data discovery without natural language synthesis, or reasoning over already\-clean, schema\-aligned text\.

To bridge this gap, we introduceLakeQuest, a benchmark designed to evaluate the end\-to\-end QA pipeline over realistic data lakes\.LakeQuestprovides a shared construction and evaluation framework instantiated across three diverse domains: AI/ML metadata, a realistic retail bank, and multimodal biomedical drug information\. Rather than flattening these domains into a uniform representation,LakeQuestpreserves their native structural heterogeneity as data lakes\. Each benchmark instance pairs a natural language question with a reference answer and exact, modality\-aware evidence pointers \(e\.g\., specific table rows or passage spans\)\. Furthermore, becauseLakeQuestpreserves the native structure of the data lakes, it inherently captures factual redundancy\. A system might successfully synthesize a grounded answer by discovering valid*alternative provenance*\(e\.g\., a redundant table or densely informative passage\) even if the target gold artifact is missed\. This explicit attribution enables precise isolation of system failures into discovery errors \(failing to retrieve evidence\) versus reasoning errors \(failing to synthesize an answer from the retrieved evidence\)\(Bryanet al\.,[2025](https://arxiv.org/html/2607.12310#bib.bib28); Leunget al\.,[2026](https://arxiv.org/html/2607.12310#bib.bib27)\)\.

We constructLakeQuestusing a reproducible five\-stage pipeline encompassing domain instantiation, language\-model\-driven QA synthesis, structural filtering, semantic validation, and rigorous human review\. The resulting benchmark contains 9,846 human\-validated QA pairs that entail single\-source lookup, cross\-source composition, and multi\-hop reasoning\.

Evaluating a diverse suite of baselines—from zero\-shot reasoning\(Kojimaet al\.,[2022](https://arxiv.org/html/2607.12310#bib.bib4)\)and standard retrieval\-augmented generation \(RAG\)\(Lewiset al\.,[2020](https://arxiv.org/html/2607.12310#bib.bib2)\)to structured pipelines and agentic tool\-use methods such asXMode\(Nooralahzadehet al\.,[2025](https://arxiv.org/html/2607.12310#bib.bib19)\)andReSP\(Jianget al\.,[2025](https://arxiv.org/html/2607.12310#bib.bib21)\)—we find that the bottlenecks in data lake QA remain fundamentally unsolved and are strongly domain\-dependent\. Specifically, accurate retrieval does not guarantee faithful reasoning when cross\-modal composition is required, and even agentic strategies struggle with multi\-hop discovery over weakly structured metadata\.

Our primary contributions are:\(1\) TheLakeQuestBenchmark:A human\-validated dataset of over 9\.8k questions evaluating grounded reasoning across three heterogeneous data lakes \(AI/ML, Retail Banking, and Biomedical Drug\)\.\(2\) A Reproducible Construction Framework:A five\-stage protocol for synthesizing, filtering, and validating QA pairs with explicit evidence traces over weakly structured corpora\.\(3\) Empirical Analysis of Modern QA:Extensive baseline evaluations revealing that high retrieval quality does not consistently translate into accurate, faithful answers—especially when multi\-step or cross\-source composition is required\.

![Refer to caption](https://arxiv.org/html/2607.12310v1/figures/lakequest-manual.png)Figure 1:Overview ofLakeQuest\. The benchmark spans three heterogeneous data lakes built under a shared construction and evaluation protocol for grounded question answering\.
## 2Related Work

Table[1](https://arxiv.org/html/2607.12310#S2.T1)provides a structural comparison betweenLakeQuestand existing QA and data discovery benchmarks across modalities, evidence tracing, and evaluation scope\.

Table 1:Comparison ofLakeQuestwith existing QA and data discovery benchmarks\.Modalitydescribes the data formats involved\.Evindicates the presence of explicit evidence traces linking answers to sources\.Mddenotes the inclusion of weak metadata and distractor documents to test retrieval robustness\.Valindicates datasets that underwent human validation\.Objectsdenotes approximate source\-side evidence containers \(e\.g\., paragraphs, tables, passages, or hybrid contexts\)\.BenchmarkModalityEvMdVal\# QsObjectsTaskDomainHotpotQAText✓\\bm\{\\checkmark\}×\\bm\{\\times\}✓\\bm\{\\checkmark\}113k5M\+paragraphsTextReasoningWikipediaHybridQATablesText×\\bm\{\\times\}×\\bm\{\\times\}✓\\bm\{\\checkmark\}69k\+13k tables \+293k passagesHybridReasoningWikipediaTAT\-QATablesText×\\bm\{\\times\}×\\bm\{\\times\}✓\\bm\{\\checkmark\}16k\+2\.5k\+ hybridcontextsNumericalQAFinanceLakeBenchTablesMetadata×\\bm\{\\times\}✓\\bm\{\\checkmark\}✓\\bm\{\\checkmark\}10k\+16M\+tablesTableDiscoveryOpen DataLakeQuestTables, TextMetadata✓\\bm\{\\checkmark\}✓\\bm\{\\checkmark\}✓\\bm\{\\checkmark\}9\.8k1M\+ model/data cards \+10k\+ bank records \+500k\+ drug texts/tablesGroundedQAAI/MLBankingDrugData Discovery and Integration\.Data management literature extensively studies how to discover, link, and integrate datasets\(Fernandezet al\.,[2018](https://arxiv.org/html/2607.12310#bib.bib9); Fanet al\.,[2023](https://arxiv.org/html/2607.12310#bib.bib10)\)\. Accordingly, benchmarking efforts in this space, such as LakeBench\(Denget al\.,[2024](https://arxiv.org/html/2607.12310#bib.bib11)\)and CMDL\(Eltabakhet al\.,[2023](https://arxiv.org/html/2607.12310#bib.bib12)\), evaluate the retrieval of joinable tables or cross\-modal artifacts\. While essential for data discovery, these benchmarks remain purely retrieval\-focused and do not evaluate natural language synthesis\.

Text\-centric and Multi\-hop QA\.In the NLP community, QA benchmarks have evolved from single\-document reading comprehension\(Rajpurkaret al\.,[2016](https://arxiv.org/html/2607.12310#bib.bib13)\)to open\-domain retrieval\(Kwiatkowskiet al\.,[2019](https://arxiv.org/html/2607.12310#bib.bib14)\)and multi\-hop reasoning with HotpotQA\(Yanget al\.,[2018](https://arxiv.org/html/2607.12310#bib.bib1)\)\. While foundational, these datasets largely assume text\-centric evidence within curated, relatively homogeneous corpora \(e\.g\., Wikipedia\)\. Real data lakes, however, are characterized by weak metadata, fragmented entity references, and a mixture of semi\-structured objects, making reasoning pathways noisier than those found in standard text\-only multi\-hop settings\.

Hybrid and Multimodal QA\.A closer line of work evaluates QA over multiple evidence types\. HybridQA\(Chenet al\.,[2020](https://arxiv.org/html/2607.12310#bib.bib15)\)and OTT\-QA\(Chenet al\.,[2021](https://arxiv.org/html/2607.12310#bib.bib16)\)require reasoning across tables and text, while TAT\-QA\(Zhuet al\.,[2021](https://arxiv.org/html/2607.12310#bib.bib17)\)emphasizes numerical reasoning over financial reports\. MultiModalQA\(Talmoret al\.,[2021](https://arxiv.org/html/2607.12310#bib.bib18)\)expands the scope to joint reasoning over text, tables, and images\. While these benchmarks require cross\-modal synthesis, they generally operate over highly curated, schema\-aligned, or task\-specific corpora\. They do not model the core challenges of enterprise data lakes: navigating siloed, weakly linked sources, resolving entities across heterogeneous files, and maintaining strict, modality\-aware provenance for auditability\.

Retrieval\-Grounded and Agentic Systems\.Methodologically,LakeQuestprovides a necessary testbed for modern retrieve\-and\-synthesize pipelines\. RAG\(Lewiset al\.,[2020](https://arxiv.org/html/2607.12310#bib.bib2)\)and fusion\-in\-decoder \(FiD\)\(Izacard and Grave,[2021](https://arxiv.org/html/2607.12310#bib.bib3)\)established paradigms for grounding generation in retrieved contexts, while recent methods propose structured decomposition\(Zhouet al\.,[2023](https://arxiv.org/html/2607.12310#bib.bib5)\), iterative tool use\(Yaoet al\.,[2023](https://arxiv.org/html/2607.12310#bib.bib7); Schicket al\.,[2023](https://arxiv.org/html/2607.12310#bib.bib8)\), and multi\-agent orchestration like Symphony\(Tanget al\.,[2024](https://arxiv.org/html/2607.12310#bib.bib22)\)\. By providing a heterogeneous environment with explicit evidence requirements and realistic discovery bottlenecks,LakeQuestenables the community to rigorously evaluate whether these control strategies can handle the noise and scale of real\-world data lakes\.

## 3TheLakeQuestFramework

To rigorously evaluate QA over heterogeneous data lakes, we require a framework that isolates source discovery from cross\-modal synthesis, scales across domains, and maintains strict provenance\.LakeQuestoperationalizes this through a formalized task definition and a reproducible construction pipeline\.

### 3\.1Problem Formulation and Task Taxonomy

We formulate grounded data lake QA: given a data lake𝒟=\{d1,…,dn\}\\mathcal\{D\}=\\\{d\_\{1\},\\dots,d\_\{n\}\\\}\(wheredid\_\{i\}is a table, passage, or metadata record\) and a natural language questionqq, a systemffmust return a textual answera^\\hat\{a\}and a precise supporting evidence setE^⊆𝒟\\hat\{E\}\\subseteq\\mathcal\{D\}:f​\(q,𝒟\)→\(a^,E^\)f\(q,\\mathcal\{D\}\)\\rightarrow\(\\hat\{a\},\\hat\{E\}\)\. Modern agents decompose this operation into retrieve\-and\-synthesize workflows:E^=R​\(q,𝒟\)\\hat\{E\}=R\(q,\\mathcal\{D\}\)anda^=G​\(q,E^\)\\hat\{a\}=G\(q,\\hat\{E\}\)\. By requiring systems to outputE^\\hat\{E\},LakeQuestallows us to independently evaluate the retrieval mechanismRR\(discovery\) and the generatorGG\(synthesis\)\.

To systematically test this pipeline,LakeQuestis organized into four distinct reasoning families:single\-source lookupsrequiring fact extraction from isolated objects;cross\-source compositionnecessitating the combination of facts across multiple heterogeneous objects \(e\.g\., joining a policy with a transaction\);multi\-hop reasoningrequiring chained inference over intermediate entities or linked metadata; andanchor\-based retrieval, which conditions questions on a selected entity, forcing the agent to reason over related entities amidst semantically close distractors\.

Unlike standard text QA benchmarks where evidence is simply a document ID,LakeQuestenforces modality\-aware provenance\. Each benchmark instance is defined by the tuplexi=\(qi,ai∗,Ei∗,mi\)x\_\{i\}=\(q\_\{i\},a^\{\*\}\_\{i\},E^\{\*\}\_\{i\},m\_\{i\}\), whereEi∗E^\{\*\}\_\{i\}resolves to concrete sub\-document units \(e\.g\., specific table rows or exact passage spans\) andmim\_\{i\}tracks the question family and domain metadata\. Figure[2](https://arxiv.org/html/2607.12310#S3.F2)illustrates a concrete cross\-source instance from the Retail Bank data lake\.

Domain:Retail bankingFamily:Cross\-source composition \(Policy \+ Table \+ Metadata\)Question \(qiq\_\{i\}\):Based on standard routing policies, does the wire transfer initiated by Acme Corp on Oct 14th require manual manager approval?Gold Evidence \(Ei∗E^\{\*\}\_\{i\}\):•Policy\_Manual\_v3: Span \[1124\-1180\]\(“…Tier 2 business accounts require manual manager approval for all outbound international wire transfers exceeding $15,000…”\)•Transaction\_Ledger\_72\(Acme Corp\): Row 84, ColumnsType,Amount\(“Int\_Wire $18,500\.00”\)•Client\_Profile\_7742: KeyAccount\_Tier\(“Tier 2”\)Reference Answer \(ai∗a^\{\*\}\_\{i\}\):Yes, the $18,500 transfer exceeds the $15,000 limit for Tier 2 accounts\.

Figure 2:Anatomy of aLakeQuestinstance — this requires identifying and synthesizing specific, sub\-document evidence units across multiple heterogeneous sources to derive the correct answer\.
### 3\.2The Shared Five\-Stage Pipeline

Manually authoring tens of thousands of complex, cross\-modal QA pairs is unscalable, while purely synthetic datasets often suffer from hallucinations and trivial reasoning pathways\. To balance scale with rigor, we constructLakeQuestusing an LLM\-in\-the\-loop pipeline consisting of five shared stages across all domains \(a visual schematic of the Retail Bank data lake question\-answer synthesis process is provided in App[G](https://arxiv.org/html/2607.12310#A7)\)\.

The five shared stages include:\(1\) Lake Instantiation, preserving the domain’s native schema rather than homogenizing documents;\(2\) Candidate Synthesis, where a strong language model \(OpenAI GPT\-5, Feb\. 1, 2026\) synthesizes a candidate questionqq, answera∗a^\{\*\}, and exact evidence pointersE∗E^\{\*\}from a sampled evidence subgraph;\(3\) Structural Filtering, which programmatically removes malformed outputs or deictic questions;\(4\) Semantic Filtering, utilizing an LLM\-as\-a\-judge to reject trivial candidates answerable via parametric memory alone; and\(5\) Human Validation, the definitive gate for benchmark inclusion, assessing answerability and attribution\. Exact prompts and rubrics are detailed in App[B](https://arxiv.org/html/2607.12310#A2)\.

### 3\.3Human Validation and Quality Control

To guarantee evaluation fidelity, all items passing the automated filters are subjected to rigorous human validation\. Annotators were tasked with independently verifying two criteria:*Answerability*\(can the question be answered exclusively using the cited evidence?\) and*Attribution*\(is the generated answer completely faithful to the cited text/table?\)\. The use of human annotators was approved by the ethics review board at the authors’ institution\. Prolific annotators \(paid $12 USD/hr\) independently verified each item under this protocol\. Inter\-annotator agreement \(Fleiss’κ\\kappa\) demonstrates strong consistency across raters:0\.820\.82for Answerability and0\.850\.85for Attribution\. Full annotator guidelines, including a screenshot of the custom validation interface \(Figure[3](https://arxiv.org/html/2607.12310#A3.F3)\), are provided in App[C](https://arxiv.org/html/2607.12310#A3)\. Only instances that achieved strict human consensus were retained in the finalLakeQuestrelease\.

Table 2:Abstraction map to unify task design while preserving domain\-native structure\.DomainLake RepresentationTask AbstractionPrimary Reasoning StressAI/MLLinked metadata entities \(models, datasets, relations\)Entity/relation QA with lookup, composition, hop, and anchor familiesEntity disambiguation and relation chainingRetail BankStructured operational records with policy contextPersona\-workflow tasks with client/group scopesDecision grounding under workflow/policy constraintsDrugHybrid tables \+ scientific passages with linkage artifactsEvidence\-bundle QA across single\-modality and mixed\-modality settingsCross\-modality evidence integration and multi\-hop synthesis

## 4TheLakeQuestDataset — Instantiating the Three Data Lakes

To evaluate the framework across diverse reasoning requirements,LakeQuestinstantiates three data lakes: an AI/ML metadata repository, a synthetic retail banking system, and a biomedical corpus\. Table[2](https://arxiv.org/html/2607.12310#S3.T2)outlines the high\-level abstractions used to preserve the native structures of these domains\. Furthermore, Table[3](https://arxiv.org/html/2607.12310#S4.T3)details the final benchmark coverage and human annotation outcomes across these three lakes\.

#### AI/ML Data Lake

The AI/ML data lake focuses on entity disambiguation and relation chaining over weakly linked metadata\. We construct the corpus using a snapshot of the Hugging Face Hub API\(Hugging Face,[2025](https://arxiv.org/html/2607.12310#bib.bib30)\)\. The resulting lake consists of model cards and dataset cards, which are inherently semi\-structured\. We instantiate 5 distinct question families in this domain, yielding 3,578 human\-validated questions \(see Table[3](https://arxiv.org/html/2607.12310#S4.T3)for a high\-level summary and Table[7](https://arxiv.org/html/2607.12310#A1.T7)in App[A](https://arxiv.org/html/2607.12310#A1)for the detailed breakdown\)\.

These tasks map to our core reasoning taxonomy by requiring systems to deduce specific hyperparameters from isolated cards \(single\-source\), synthesize constraints by jointly reasoning over explicitly linked model and dataset cards \(cross\-source\), chain relations across the metadata graph to link datasets and subsequent models \(multi\-hop\), and formulate queries around specific anchor entities despite lexical drift \(anchor\-based\)\.

#### Retail Bank Data Lake

The Retail Bank data lake evaluates grounded decision\-making under strict policy constraints\. Because real operational banking data is highly restricted, we generate a high\-fidelity synthetic lake by deriving generation seed examples and instantiating an open\-banking\-inspired relational schema \(client, account, transaction, and policy entities\)\. We then populate client profiles, transaction ledgers, and institutional policy documents\. We further validate that transaction distributions are similar between the synthetic lake and original real banking data by matching graph density on client profiles\. Tasks are formulated across 9 persona\-driven workflows \(e\.g\., a fraud analyst investigating a transaction, or an auditor reviewing compliance\), yielding 2,964 validated questions \(detailed breakdown provided in Table[8](https://arxiv.org/html/2607.12310#A1.T8), App[A](https://arxiv.org/html/2607.12310#A1)\)\.

These workflows map to core reasoning families:single\-source lookupsfor retrieving isolated facts \(e\.g\., account tiers or policy limits\);cross\-source compositionto combine unstructured policy rules with structured transaction ledgers \(e\.g\., checking if a wire transfer violates a tiered policy\);multi\-hop reasoningto trace complex fund flows across multiple tables before applying operational rules; andanchor\-based retrievalto disambiguate compliance policies based on client attributes amidst overlapping regional guidelines\.

#### Drug Data Lake

The Drug data lake stresses multimodal evidence integration\. We construct this corpus by fusing structured XML records from DrugBank 6\.0\(Knoxet al\.,[2024](https://arxiv.org/html/2607.12310#bib.bib24)\)with open\-access scientific literature indexed in PubMed\(Sayerset al\.,[2026](https://arxiv.org/html/2607.12310#bib.bib25)\)\. This heterogeneous mixture of dense text and structured properties instantiates 5 question families, producing 3,304 validated questions \(see Table[9](https://arxiv.org/html/2607.12310#A1.T9)in App[A](https://arxiv.org/html/2607.12310#A1)for statistical breakdown\)\.

The reasoning families in this domain stress multimodal evidence integration:single\-source lookupsextract precise values from isolated DrugBank tables or PubMed abstracts;cross\-source compositionsynthesizes answers by linking unstructured text passages \(e\.g\., mechanism of action\) to structured tables \(e\.g\., biochemical properties\);multi\-hop reasoningchains relationships across modalities \(e\.g\., identifying a target in a table, then finding passages detailing its metabolic pathway\); andanchor\-based retrievalrequires identifying correct active compounds amidst structurally similar distractors and complex aliases\.

#### Splits and Licensing

To prevent contamination, we define \(open\) validation \(20%\) and test \(80%\) splits at the*entity*rather than question level \(i\.e\.,Entity​\(val\)∩Entity​\(test\)=∅\\mathrm\{Entity\}\(\\mathrm\{val\}\)\\cap\\mathrm\{Entity\}\(\\mathrm\{test\}\)=\\emptyset\)\. TheLakeQuestbenchmark and underlying corpora are hosted on Hugging Face at[anonymousurl](https://arxiv.org/html/2607.12310v1/anonymousurl)\. The AI/ML and Retail Bank subsets are released under an MIT license\. For the Drug subset, we release benchmark annotations and evidence pointers, while use of underlying DrugBank tables remains subject to DrugBank licensing terms and PubMed\-linked content remains subject to NLM/PubMed access policies and publisher rights\.

Table 3:Benchmark coverage summary across all three lakes after human validation\.DomainQuestion FamiliesQuestionsCorrectAISAnswerableAI/ML5 source combos3,9353,7503,6873,578Retail Bank9 personas3,3123,0473,0032,964Drug5 source combos3,5743,4343,3733,304Total–10,82110,23110,0639,846

## 5Experimental Setup and Baselines

To benchmark grounded reasoning over data lakes, we define a unified evaluation contract and test representative QA baselines ranging from direct prompting to agentic tool use\.

#### Evaluation Metrics

Because data lake QA serves both high\-assurance settings \(where hallucination is unacceptable\) and exploratory analytics \(where cost and speed are critical\), we treat evaluation as a multi\-dimensional quality\-cost trade\-off\. We evaluate systems across four axes:

- •Accuracy \(Correctness\):We report binary correctness using rubric\-based LLM\-as\-a\-judge for baseline runs \(including direct\-answer and reasoning\-answer correctness where applicable\), and human\-judged correctness for annotated benchmark slices in App[A](https://arxiv.org/html/2607.12310#A1)\.
- •Faithfulness \(Attribution\):Whether the predicted answera^\\hat\{a\}is supported by the predicted evidence setE^\\hat\{E\}\. We operationalize this using Attributable to Identified Sources \(AIS\) style judgments\(Rashkinet al\.,[2023](https://arxiv.org/html/2607.12310#bib.bib26)\), aligned with the human attribution protocol used throughout this benchmark\.
- •Retrieval Quality:Domain/modality\-aware recall of gold evidenceE∗E^\{\*\}inE^\\hat\{E\}\(e\.g\., card retrieval in AI/ML, policy and table recall in Retail Bank, and passage/table recall in Drug\)\. For unstructured evidence, success is chunk\-level span inclusion; for structured evidence, we require retrieval of the referenced source object and apply row/field\-level checks when explicit row\-level pointers are available\. Because real\-world data lakes inherently contain redundant information \(e\.g\., same fact appearing in multiple tables/passages\), this metric serves as a strict lower bound\. A system may retrieve valid alternative evidence that correctly answers the question despite missing the specific gold artifact\.
- •Token Usage:End\-to\-end retrieval\-plus\-synthesis token footprint, reported as total model tokens \(and normalized per\-query token count\) under the evaluation\-time configuration\.

Table 4:Baseline results for the AI/ML data lake\. Metrics are Retrieval Recall \(RR\), Accuracy \(AA\), Faithfulness \(FF\), and Token Count \(TT\)\. Token Count \(TT\) is reported in thousands of generation tokens per query\. Model and Dataset are single\-source lookup and anchor questions, Model\-Dataset is cross\-source composition, and Model\-hop and Dataset\-hop are multi\-hop reasoning questions\.RAGZero\-shotQuestion TypeRRAAFFTTAATTModel0\.090\.350\.250\.390\.250\.06Dataset0\.150\.120\.140\.340\.090\.05Model\-Dataset0\.120\.380\.320\.380\.170\.05Model\-hop0\.130\.230\.160\.470\.200\.07Dataset\-hop0\.230\.350\.380\.490\.240\.07
#### Evaluated Baselines

We establish baseline performance using models with different levels of agentic complexity\. We use GPT\-5 as the generator \(kept fixed within each experiment suite\)\(OpenAI,[2025](https://arxiv.org/html/2607.12310#bib.bib33)\)and FAISS\-backedtext\-embedding\-3\-smallfor retrieval\(Douzeet al\.,[2026](https://arxiv.org/html/2607.12310#bib.bib31)\)\. Comprehensive details, including system instructions, retrieval formatting templates, and agentic tool\-use prompts for the evaluated systems, are provided in App[B](https://arxiv.org/html/2607.12310#A2)\.

- •Zero\-Shot Prompting:A naive baseline that directly prompts the generator LLM without explicit retrieval, testing the limits of parametric memory and instruction following\.
- •Retrieval\-Augmented Generation \(RAG\)\(Lewiset al\.,[2020](https://arxiv.org/html/2607.12310#bib.bib2)\):Used as our baseline for the document\-only AI/ML data lake, this is a standard retrieve\-then\-generate pipeline\. We chunk unstructured passages and structured metadata cards into overlapping text segments before indexing; we then retrieve the top\-kkmost relevant chunks and synthesize a final answer\. We experiment with different values ofkkand finalize onk=5k=5, which consistently attains strong performance\.
- •Agentic and Structured Pipelines \(XMode,ReSP\)\(Nooralahzadehet al\.,[2025](https://arxiv.org/html/2607.12310#bib.bib19); Jianget al\.,[2025](https://arxiv.org/html/2607.12310#bib.bib21)\):Standard RAG is structurally insufficient for Retail Bank and Drug because it cannot natively execute the relational row operations that navigate Retail Bank transaction records and Drug ledgers\. Instead, we evaluate stronger multi\-step baselines that go beyond single\-shot text retrieval by interleaving reasoning and tool use\. In our implementation,XModeacts as a tool\-augmented RAG system: it uses an iterative loop with passage retrieval from a FAISS index alongside direct query/SQL access to structured tables\.ReSPfollows an iterative retrieve–summarize–plan style decomposition\.

## 6Results and Cross\-Domain Analysis

We evaluate baselines across the three domains to understand where modern retrieve\-and\-synthesize systems fail\. Across domains, retrieval quality alone is insufficient; grounded data\-lake QA requires stronger cross\-source composition and attribution mechanisms\.

### 6\.1AI/ML Data Lake: Discovery is the primary bottleneck

The AI/ML lake is challenging due to scale—it is built from a Hugging Face model\-card corpus with over 1M cards and normalized into a large model–dataset graph with on\-demand card\-text retrieval—and the inconsistencies between how different authors document artifacts\. As shown in Table[4](https://arxiv.org/html/2607.12310#S5.T4), the RAG baseline exhibits a notable gap between exact\-match Recall \(RR\) and Accuracy \(AA\)\. For example, on isolated Model questions, RAG achieves 0\.35 Accuracy despite a recall of 0\.09\. This gap validates that real data lakes contain factual redundancy, for example from models that have been duplicated with minor modifications; the retriever frequently finds valid alternative evidence to synthesize the correct answer even if it misses the specific gold artifact \(E∗E^\{\*\}\)\. This reinforces thatRRacts as a strict lower bound and suggests improved retrieval as a logical next step for metadata QA\. In summary, the wide gap between modest recall and reasoning accuracy suggests that navigating weakly linked metadata via dense retrieval is insufficient for reliable entity disambiguation and relation chaining\.

### 6\.2Retail Bank Data Lake: A failure of grounding, not retrieval

Table[5](https://arxiv.org/html/2607.12310#S6.T5)reports performance on the Retail Bank lake\. This domain is distinctive because benchmark items are framed as operational decisions requiring both unstructured policy retrieval and structured table lookups\. TheXModebaseline generally achieves high*Policy Recall*\(0\.85\+ for seven of the nine personas, though struggling on the Data Analyst and Salesperson workflows\), but*Table Recall*varies significantly across the board\. More importantly, final answer accuracy is often much lower than policy recall\. This exposes a synthesis gap: while agentic systems reliably retrieve abstract governing rules, they fundamentally struggle to ground those rules against specific operational ledgers to execute a faithful decision\. Notably, the zero\-shot baseline achieves high accuracy on certain workflows \(e\.g\., 0\.88 for the Financial planner\)\. While our semantic filter successfully removes simple factoid lookups, frontier LLMs possess strong parametric priors for standard financial principles, allowing them to often deduce correct operational outcomes without context\. However, because this zero\-shot reasoning lacks explicit provenance \(Faithfulness is strictly zero\), it highlights why our benchmark requires grounded synthesis; in an enterprise setting, an accurate but ungrounded answer remains an unauditable liability\.

Table 5:Baseline results for the Retail Bank data lake\. Metrics are Table Recall \(RTR\_\{T\}\), Policy Recall \(RPR\_\{P\}\), Accuracy \(AA\), Faithfulness \(FF\), and Token Count \(TT; k\-tokens/query, as in Table[4](https://arxiv.org/html/2607.12310#S5.T4)\)\. Persona workflows span all four families in the task taxonomy depending on whether the task is isolated lookup, policy\-ledger composition, multi\-step trace, or anchor\-conditioned policy selection\.XModeReSPZero\-shotPersonaRTR\_\{T\}RPR\_\{P\}AAFFTTRTR\_\{T\}RPR\_\{P\}AAFFTTAATTAccountant0\.790\.940\.600\.451\.150\.951\.000\.730\.481\.200\.250\.19Auditor0\.840\.900\.580\.511\.000\.940\.850\.580\.551\.120\.690\.19Customer service \(CS\)0\.820\.980\.640\.641\.260\.931\.000\.580\.501\.070\.690\.16CS manager0\.700\.870\.700\.671\.050\.861\.000\.880\.801\.030\.690\.13Data analyst0\.620\.690\.590\.491\.100\.890\.650\.550\.381\.040\.310\.16Financial planner0\.640\.990\.650\.591\.240\.850\.900\.680\.701\.050\.880\.16Fraud analyst0\.720\.920\.610\.471\.100\.950\.900\.680\.601\.270\.810\.15Insurance analyst0\.520\.890\.580\.451\.140\.740\.750\.620\.481\.240\.380\.20Salesperson0\.560\.660\.440\.421\.150\.770\.600\.300\.231\.090\.380\.15
### 6\.3Drug Data Lake: Joint tabular QA remains challenging

Table[6](https://arxiv.org/html/2607.12310#S6.T6)reports performance on the Drug data lake, which stresses multimodal question answering across passages and tables\. The strongest results appear on Passage questions, whereXModeachieves near\-perfect recall \(0\.98\), accuracy \(0\.97\), and faithfulness \(0\.97\)\. Passage\-hop questions remain relatively strong: despite Passage Recall dropping to 0\.72, answer accuracy is 0\.86, suggesting systems can often synthesize correct answers from partially recovered text\. This gap indicates that retrieving the correct table is often not sufficient; the remaining challenge lies in extracting the right fields and composing them into a correct answer\. Difficulty increases further for Table\-hop questions, where Table Recall drops to 0\.48 and accuracy to 0\.44, showing that multi\-step reasoning over structured biomedical evidence remains substantially harder than single\-table lookup\.

The mixed Passage\-Table family further reinforces this point\. Although exact retrieval is very strong in both modalities—0\.73 Passage Recall and 0\.98 Table Recall—end\-to\-end accuracy reaches only 0\.66\. In other words, the main bottleneck is no longer finding the relevant evidence, but correctly integrating heterogeneous evidence across text and tables\. Taken together, these results suggest that the Drug data lake is best characterized not as a retrieval\-limited setting, but as a cross\-modal synthesis setting in which joint reasoning over structured and unstructured biomedical evidence remains the main challenge\.

Table 6:Baseline results for the Drug data lake\. Metrics are Passage Recall \(RPR\_\{P\}\), Table Recall \(RTR\_\{T\}\), Accuracy \(AA\), Faithfulness \(FF\), and Token Count \(TT; k\-tokens/query, as in Table[4](https://arxiv.org/html/2607.12310#S5.T4)\)\. Passage/Table map to Single\-source lookups and anchor questions, Passage\-Table maps to Cross\-source, and Passage\-hop/Table\-hop map to Multi\-hop\.ReSPtoken counts are estimated full generation tokens per question \(approximate\)\.XModeReSPZero\-shotQuestion TypeRPR\_\{P\}RTR\_\{T\}AAFFTTRPR\_\{P\}RTR\_\{T\}AAFFTTAATTPassage0\.98–0\.970\.970\.661\.00–0\.991\.000\.950\.450\.29Table–0\.810\.600\.760\.59–0\.750\.650\.700\.730\.250\.23Passage\-Table0\.730\.980\.660\.820\.660\.901\.000\.790\.850\.860\.470\.26Passage\-hop0\.72–0\.860\.900\.840\.65–0\.860\.901\.300\.570\.28Table\-hop–0\.480\.440\.620\.67–0\.600\.510\.630\.870\.320\.26
### 6\.4Cross\-Domain Takeaways

Across all three lakes, a consistent narrative emerges:*high retrieval quality alone does not guarantee correct reasoning\.*We highlight two major takeaways for future system design:

The Divergence of Discovery vs\. Synthesis:By isolating source discovery from synthesis using a controlled context\-bundle ablation \(App[E](https://arxiv.org/html/2607.12310#A5)\), we find that failure modes are strictly domain\-dependent\. When provided with perfect gold evidence, AI/ML accuracy saturates at 98\.7%, proving that the AI/ML lake is purely a*discovery*bottleneck\. Conversely, Drug lake accuracy only reaches 62\.5% even with perfect evidence, demonstrating that cross\-modal reasoning over dense biomedical tables remains fundamentally broken in LLMs, independent of retrieval performance\.

The Compounding Cost and Latency of Agentic Navigation:While standard single\-shot RAG cannot natively navigate the relational topologies of the Retail Bank and Drug lakes, agentic loop\-based methods \(e\.g\.,XMode\) introduce a compounding token and temporal overhead\. Table[4](https://arxiv.org/html/2607.12310#S5.T4)shows that the zero\-shot baseline uses only ~0\.05–0\.07 k\-tokens per query \(~0\.06 on average\)\. In contrast, single\-shot RAG requires retrieving and processing substantially larger context, increasing token usage to ~0\.34–0\.49 k\-tokens per query\. Transitioning to iterative agentic navigation compounds this further\. As reflected by the proxy token values in Tables[6](https://arxiv.org/html/2607.12310#S6.T6)and[5](https://arxiv.org/html/2607.12310#S6.T5), iterative navigation increases token consumption significantly—in the Retail Bank lake, to roughly 1\.00–1\.27 k\-tokens per query \(about 5–8×\\timesover zero\-shot and up to ~3×\\timesover standard RAG\)\. Beyond token footprint, multi\-step retrieval loops incur sequential latency bottlenecks, presenting a practical constraint for real\-time operational environments like front\-line customer service\.

## 7Limitations

Domain Scope and Synthetic Data:LakeQuestevaluates text and tables, excluding dynamic logs and raw audio/visual data \(e\.g\., vision\-model caveats rely on text metadata; see App[D](https://arxiv.org/html/2607.12310#A4)\)\. Furthermore, the Retail Bank domain relies on synthetic data for privacy, which may alter real\-world noise distributions\.

Model Monoculture and Evaluator Bias:Using an LLM to synthesize candidate questions risks introducing distributional bias that favors the generator’s reasoning patterns\. We mitigated this via strict human validation\. However, evaluating baselines and rubric\-based correctness with the same LLM family \(GPT\-5\) used for synthesis introduces a risk of self\-preference bias\(Zhenget al\.,[2023](https://arxiv.org/html/2607.12310#bib.bib32)\), which may overstate measured baseline performance\. Future evaluations using independent frontier models will help separate general architectural bottlenecks from model\-specific priors\.

Factual Redundancy:Data lakes inherently contain redundant facts\. BecauseLakeQueststrictly tracks single\-path gold evidence \(Ei∗E^\{\*\}\_\{i\}\), systems might retrieve valid alternative provenance\. Consequently, our retrieval metric serves as a conservative lower bound\.

## 8Conclusion

We introducedLakeQuest, a comprehensive benchmark and reproducible construction framework for grounded question answering over heterogeneous data lakes\. Encompassing AI/ML, retail banking, and biomedical domains, the benchmark provides a shared, human\-validated evaluation environment that explicitly pairs natural language questions with precise, modality\-aware evidence pointers\. Through baseline evaluations, we demonstrated that high retrieval quality alone is insufficient; modern retrieve\-and\-synthesize architectures fail differently across domains, struggling with relation chaining, policy grounding, and multimodal composition\. Ultimately,LakeQuestprovides the necessary evaluation infrastructure to drive the next generation of reliable, agentic QA systems capable of rigorous discovery and faithful attribution over real\-world data lakes\.

## 9Ethics and Reproducibility Statements

#### Ethics Statement\.

LakeQuestincludes retail banking and biomedical QA tasks where hallucinated or weakly grounded answers pose significant real\-world risks\. Consequently, this benchmark explicitly evaluates*Faithfulness*alongside accuracy, and human validation ensures that all queries are natively answerable from the retrieved context\. However, strong performance onLakeQuestdoes not constitute an endorsement for the autonomous deployment of these systems in high\-stakes, real\-world decision pipelines\.LakeQuestis intended as a rigorous evaluation resource, not a substitute for domain expertise or human oversight\.

For the human validation, we recruited and paid data annotators using the Prolific platform\(Prolific,[2026](https://arxiv.org/html/2607.12310#bib.bib29)\), ensuring that research ethics were upheld\. We acknowledge that ethical considerations are extremely important for any research involving human subjects, and we have ensured that our research meets the Code of Ethics for COLM 2026\. Our research plan was reviewed and approved by the ethics review board at our institution\. In addition, we took steps to ensure research ethics including: reviewing our institution’s protocols for ethical research; only showing suitable content to participants; collecting consent from annotators regarding how the data would be collected, used, and retained; and paying all recruits a fair wage above the minimum wage in the jurisdiction of the recruitment platform\.

#### Reproducibility Statement

To ensure the full reproducibility of our five\-stage dataset construction pipeline and evaluation framework, we release all artifacts associated with this work\. TheLakeQuestbenchmark, including human validation annotations, explicit evidence pointers, and the underlying data lake corpora \(subject to their respective licensing\), is hosted on Hugging Face Datasets at[anonymousurl](https://arxiv.org/html/2607.12310v1/anonymousurl)\. Furthermore, our complete codebase—containing the automated generation pipeline, human\-annotation UI, and all baseline implementations \(RAG,XMode,ReSP\) with exact prompts—is detailed in App[B](https://arxiv.org/html/2607.12310#A2)and will be open\-sourced upon publication\.

## Acknowledgments

We are grateful to Layer 6 AI at TD and the Vector Institute for their support of this work\. This research was also supported in part by the Natural Sciences and Engineering Research Council of Canada \(NSERC\) through Discovery Grant\.

## References

- P\. Bryan, G\. Severi, J\. de Gruyter, D\. Jones, B\. Bullwinkel, A\. Minnich, S\. Chawla, G\. Lopez, M\. Pouliot, A\. Fourney, W\. Maxwell, K\. Pratt, S\. Qi, N\. Chikanov, R\. Lutz, R\. S\. R\. Dheekonda, B\. Jagdagdorj, E\. Kim, J\. Song, K\. Hines, D\. Jones, R\. Lundeen, S\. Vaughan, V\. Westerhoff, Y\. Zunger, C\. Kawaguchi, M\. Russinovich, and K\. R\. S\. Siva \(2025\)Taxonomy of failure mode in agentic ai systems\.Technical reportMicrosoft\.External Links:[Link](https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcorp/microsoft/final/en-us/microsoft-brand/documents/Taxonomy-of-Failure-Mode-in-Agentic-AI-Systems-Whitepaper.pdf)Cited by:[§1](https://arxiv.org/html/2607.12310#S1.p3.1)\.
- W\. Chen, M\. Chang, E\. Schlinger, W\. Y\. Wang, and W\. W\. Cohen \(2021\)Open question answering over tables and text\.In9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3\-7, 2021,Cited by:[§2](https://arxiv.org/html/2607.12310#S2.p4.1)\.
- W\. Chen, H\. Zha, Z\. Chen, W\. Xiong, H\. Wang, and W\. Y\. Wang \(2020\)HybridQA: a dataset of multi\-hop question answering over tabular and textual data\.InFindings of the Association for Computational Linguistics: EMNLP 2020,pp\. 1026–1036\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.findings-emnlp.91)Cited by:[§2](https://arxiv.org/html/2607.12310#S2.p4.1)\.
- Y\. Deng, C\. Chai, L\. Cao, Q\. Yuan, S\. Chen, Y\. Yu, Z\. Sun, J\. Wang, J\. Li, Z\. Cao, K\. Jin, C\. Zhang, Y\. Jiang, Y\. Zhang, Y\. Wang, Y\. Yuan, G\. Wang, and N\. Tang \(2024\)LakeBench: a benchmark for discovering joinable and unionable tables in data lakes\.Proceedings of the VLDB Endowment17\(8\),pp\. 1925–1938\.External Links:[Document](https://dx.doi.org/10.14778/3659437.3659448)Cited by:[§2](https://arxiv.org/html/2607.12310#S2.p2.1)\.
- M\. Douze, A\. Guzhva, C\. Deng, J\. Johnson, G\. Szilvasy, P\. Mazaré, M\. Lomeli, L\. Hosseini, and H\. Jégou \(2026\)The Faiss Library\.IEEE Transactions on Big Data12\(2\),pp\. 346–361\.External Links:[Document](https://dx.doi.org/10.1109/TBDATA.2025.3618474)Cited by:[§5](https://arxiv.org/html/2607.12310#S5.SS0.SSS0.Px2.p1.1)\.
- M\. Y\. Eltabakh, M\. Kunjir, A\. K\. Elmagarmid, and M\. S\. Ahmad \(2023\)Cross modal data discovery over structured and unstructured data lakes\.Proceedings of the VLDB Endowment16\(11\),pp\. 3377–3390\.External Links:[Document](https://dx.doi.org/10.14778/3611479.3611533)Cited by:[§2](https://arxiv.org/html/2607.12310#S2.p2.1)\.
- G\. Fan, J\. Wang, Y\. Li, D\. Zhang, and R\. J\. Miller \(2023\)Semantics\-aware dataset discovery from data lakes with contextualized column\-based representation learning\.Proceedings of the VLDB Endowment16\(7\),pp\. 1726–1739\.External Links:[Document](https://dx.doi.org/10.14778/3587136.3587146)Cited by:[§2](https://arxiv.org/html/2607.12310#S2.p2.1)\.
- R\. C\. Fernandez, Z\. Abedjan, F\. Koko, G\. Yuan, S\. Madden, and M\. Stonebraker \(2018\)Aurum: a data discovery system\.In2018 IEEE 34th International Conference on Data Engineering \(ICDE\),pp\. 1001–1012\.External Links:[Document](https://dx.doi.org/10.1109/ICDE.2018.00094)Cited by:[§2](https://arxiv.org/html/2607.12310#S2.p2.1)\.
- Hugging Face \(2025\)Note:Accessed 2025\-12\-01External Links:[Link](https://huggingface.co/docs/huggingface_hub/v0.5.1/en/package_reference/hf_api)Cited by:[§4](https://arxiv.org/html/2607.12310#S4.SS0.SSS0.Px1.p1.1)\.
- G\. Izacard and E\. Grave \(2021\)Leveraging passage retrieval with generative models for open domain question answering\.InProceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume,pp\. 874–880\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.eacl-main.74)Cited by:[§2](https://arxiv.org/html/2607.12310#S2.p5.1)\.
- Z\. Jiang, M\. Sun, L\. Liang, and Z\. Zhang \(2025\)Retrieve, summarize, plan: advancing multi\-hop question answering with an iterative approach\.InCompanion Proceedings of the ACM on Web Conference 2025,pp\. 1677–1686\.External Links:[Document](https://dx.doi.org/10.1145/3701716.3716889)Cited by:[§1](https://arxiv.org/html/2607.12310#S1.p5.1),[3rd item](https://arxiv.org/html/2607.12310#S5.I2.i3.p1.1.1)\.
- C\. Knox, M\. Wilson, C\. M\. Klinger, M\. Franklin, E\. Oler, A\. Wilson, A\. Pon, J\. Cox, N\. E\. L\. Chin, S\. A\. Strawbridge, M\. Garcia\-Patino, R\. Kruger, A\. Sivakumaran, S\. Sanford, R\. Doshi, N\. Khetarpal, O\. Fatokun, D\. Doucet, A\. Zubkowski, D\. Y\. Rayat, H\. Jackson, K\. Harford, A\. Anjum, M\. Zakir, F\. Wang, S\. Tian, B\. Lee, J\. Liigand, H\. Peters, R\. Q\. R\. Wang, T\. Nguyen, D\. So, M\. Sharp, R\. da Silva, C\. Gabriel, J\. Scantlebury, M\. Jasinski, D\. Ackerman, T\. Jewison, T\. Sajed, V\. Gautam, and D\. S\. Wishart \(2024\)DrugBank 6\.0: the DrugBank Knowledgebase for 2024\.Nucleic Acids Research52\(D1\),pp\. D1265–D1275\.External Links:[Document](https://dx.doi.org/10.1093/nar/gkad976)Cited by:[§4](https://arxiv.org/html/2607.12310#S4.SS0.SSS0.Px3.p1.1)\.
- T\. Kojima, S\. S\. Gu, M\. Reid, Y\. Matsuo, and Y\. Iwasawa \(2022\)Large language models are zero\-shot reasoners\.InAdvances in Neural Information Processing Systems 35,Cited by:[§1](https://arxiv.org/html/2607.12310#S1.p5.1)\.
- T\. Kwiatkowski, J\. Palomaki, O\. Redfield, M\. Collins, A\. Parikh, C\. Alberti, D\. Epstein, I\. Polosukhin, J\. Devlin, K\. Lee, K\. Toutanova, L\. Jones, M\. Kelcey, M\. Chang, A\. M\. Dai, J\. Uszkoreit, Q\. Le, and S\. Petrov \(2019\)Natural Questions: A Benchmark for Question Answering Research\.Transactions of the Association for Computational Linguistics7,pp\. 452–466\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00276)Cited by:[§1](https://arxiv.org/html/2607.12310#S1.p1.1),[§2](https://arxiv.org/html/2607.12310#S2.p3.1)\.
- K\. K\. Leung, M\. Belbahri, Y\. Sui, A\. Labach, X\. Zhang, S\. A\. Rose, and J\. C\. Cresswell \(2026\)Classifying and addressing the diversity of errors in retrieval\-augmented generation systems\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),Cited by:[§1](https://arxiv.org/html/2607.12310#S1.p3.1)\.
- P\. S\. H\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel, S\. Riedel, and D\. Kiela \(2020\)Retrieval\-augmented generation for knowledge\-intensive NLP tasks\.InAdvances in Neural Information Processing Systems 33,Cited by:[§1](https://arxiv.org/html/2607.12310#S1.p2.1),[§1](https://arxiv.org/html/2607.12310#S1.p5.1),[§2](https://arxiv.org/html/2607.12310#S2.p5.1),[2nd item](https://arxiv.org/html/2607.12310#S5.I2.i2.p1.3.1)\.
- F\. Nooralahzadeh, Y\. Zhang, J\. Fürst, and K\. Stockinger \(2025\)Multi\-modal data exploration via language agents\.InProceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia\-Pacific Chapter of the Association for Computational Linguistics,pp\. 795–813\.Cited by:[§1](https://arxiv.org/html/2607.12310#S1.p5.1),[3rd item](https://arxiv.org/html/2607.12310#S5.I2.i3.p1.1.1)\.
- OpenAI \(2025\)GPT\-5 System Card\.Note:[https://cdn\.openai\.com/gpt\-5\-system\-card\.pdf](https://cdn.openai.com/gpt-5-system-card.pdf)Version dated August 13, 2025Cited by:[§5](https://arxiv.org/html/2607.12310#S5.SS0.SSS0.Px2.p1.1)\.
- Prolific \(2026\)Note:Accessed 2026\-03\-31External Links:[Link](https://www.prolific.com/)Cited by:[§9](https://arxiv.org/html/2607.12310#S9.SS0.SSS0.Px1.p2.1)\.
- P\. Rajpurkar, J\. Zhang, K\. Lopyrev, and P\. Liang \(2016\)SQuAD: 100,000\+ questions for machine comprehension of text\.InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing,pp\. 2383–2392\.External Links:[Document](https://dx.doi.org/10.18653/v1/D16-1264)Cited by:[§1](https://arxiv.org/html/2607.12310#S1.p1.1),[§2](https://arxiv.org/html/2607.12310#S2.p3.1)\.
- H\. Rashkin, V\. Nikolaev, M\. Lamm, L\. Aroyo, M\. Collins, D\. Das, S\. Petrov, G\. S\. Tomar, I\. Turc, and D\. Reitter \(2023\)Measuring attribution in natural language generation models\.Computational Linguistics49\(4\),pp\. 777–840\.External Links:[Document](https://dx.doi.org/10.1162/coli%5Fa%5F00486)Cited by:[2nd item](https://arxiv.org/html/2607.12310#S5.I1.i2.p1.2)\.
- E\. W\. Sayers, E\. E\. Bolton, A\. M\. Fine, C\. Kelly, S\. Kim, M\. Landrum, S\. Lathrop, A\. Malheiro, T\. D\. Murphy, L\. Phan, S\. Pujar, B\. W\. Trawick, V\. A\. Schneider, and K\. D\. Pruitt \(2026\)Database resources of the National Center for Biotechnology Information in 2026\.Nucleic Acids Research54\(D1\),pp\. D20–D27\.External Links:[Document](https://dx.doi.org/10.1093/nar/gkaf1060)Cited by:[§4](https://arxiv.org/html/2607.12310#S4.SS0.SSS0.Px3.p1.1)\.
- T\. Schick, J\. Dwivedi\-Yu, R\. Dessì, R\. Raileanu, M\. Lomeli, E\. Hambro, L\. Zettlemoyer, N\. Cancedda, and T\. Scialom \(2023\)Toolformer: Language Models Can Teach Themselves to Use Tools\.InAdvances in Neural Information Processing Systems 36,Cited by:[§2](https://arxiv.org/html/2607.12310#S2.p5.1)\.
- A\. Talmor, O\. Yoran, A\. Catav, D\. Lahav, Y\. Wang, A\. Asai, G\. Ilharco, H\. Hajishirzi, and J\. Berant \(2021\)MultiModalQA: complex question answering over text, tables and images\.In9th International Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.12310#S2.p4.1)\.
- N\. Tang, C\. Yang, Z\. Zhang, Y\. Luo, J\. Fan, L\. Cao, S\. Madden, and A\. Y\. Halevy \(2024\)Symphony: towards trustworthy question answering and verification using rag over multimodal data lakes\.IEEE Data Engineering Bulletin48\(4\),pp\. 135–146\.Cited by:[§2](https://arxiv.org/html/2607.12310#S2.p5.1)\.
- Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. Cohen, R\. Salakhutdinov, and C\. D\. Manning \(2018\)HotpotQA: a dataset for diverse, explainable multi\-hop question answering\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,pp\. 2369–2380\.External Links:[Document](https://dx.doi.org/10.18653/v1/D18-1259)Cited by:[§1](https://arxiv.org/html/2607.12310#S1.p1.1),[§2](https://arxiv.org/html/2607.12310#S2.p3.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. R\. Narasimhan, and Y\. Cao \(2023\)ReAct: synergizing reasoning and acting in language models\.InThe Eleventh International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.12310#S1.p2.1),[§2](https://arxiv.org/html/2607.12310#S2.p5.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica \(2023\)Judging LLM\-as\-a\-Judge with MT\-Bench and Chatbot Arena\.InAdvances in Neural Information Processing Systems 36,Cited by:[§7](https://arxiv.org/html/2607.12310#S7.p2.1)\.
- D\. Zhou, N\. Schärli, L\. Hou, J\. Wei, N\. Scales, X\. Wang, D\. Schuurmans, C\. Cui, O\. Bousquet, Q\. V\. Le, and E\. H\. Chi \(2023\)Least\-to\-most prompting enables complex reasoning in large language models\.InThe Eleventh International Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.12310#S2.p5.1)\.
- F\. Zhu, W\. Lei, Y\. Huang, C\. Wang, S\. Zhang, J\. Lv, F\. Feng, and T\. Chua \(2021\)TAT\-QA: a question answering benchmark on a hybrid of tabular and textual content in finance\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),pp\. 3277–3287\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.254)Cited by:[§2](https://arxiv.org/html/2607.12310#S2.p4.1)\.

## Appendix ADetailed Dataset Statistics

This section provides a granular breakdown of the human validation results across all three data lakes\. As described in Sec[3\.3](https://arxiv.org/html/2607.12310#S3.SS3), all candidate question\-answer pairs generated by the synthesis pipeline underwent rigorous human review\. The tables below report the total number of synthesized candidates \(Questions\) alongside the number of instances that successfully passed three key human\-annotation criteria:

- •Correct:The reference answer is factually accurate and logically sound\.
- •AIS \(Attributable to Identified Sources\):The reference answer is strictly entailed by the provided evidence pointers, with no hallucinations or reliance on external knowledge\.
- •Answerable:The question provides sufficient context to be answered exclusively using the retrieved data lake artifacts\.

Only instances that passed all criteria were included in the final benchmark\.

### A\.1AI/ML Data Lake Statistics

Table[7](https://arxiv.org/html/2607.12310#A1.T7)presents the validation breakdown for the AI/ML repository\. In this domain, questions are structurally categorized based on the underlying metadata entities \(Model Cards vs\. Dataset Cards\) and whether the reasoning requires multi\-hop entity resolution across the AI/ML graph\.

Table 7:Detailed human\-annotation results for the AI/ML data lake, categorized by metadata entity types and graph topology\.Question TypeQuestionsCorrectAISAnswerableModel897876834815Dataset975925929896Model\-Dataset623592570562Model\-hop741701706683Dataset\-hop699656648622Total3935375036873578
### A\.2Retail Bank Data Lake Statistics

Table[8](https://arxiv.org/html/2607.12310#A1.T8)outlines the validation results for the synthetic Retail Bank data lake\. Unlike the other domains, Retail Bank QA instances are categorized by operational personas, testing how well models can ground their reasoning in strict policy constraints under different institutional workflows\.

Table 8:Detailed human\-annotation results for the Retail Bank data lake, categorized by persona\-driven operational workflows\.PersonaQuestionsCorrectAISAnswerableAccountant549478461454Auditor370321309305Customer service \(CS\)328306311303CS manager303293294293Data analyst375362361360Financial Planner228216215215Fraud analyst416391381378Insurance analyst429378376366Salesperson314302295290Total3312304730032964
### A\.3Drug Data Lake Statistics

Table[9](https://arxiv.org/html/2607.12310#A1.T9)details the validation outcomes for the Drug data lake\. The task families here stress cross\-modal synthesis, dividing questions based on whether the required evidence resides in unstructured scientific text \(Passages\), structured biochemical records \(Tables\), or a combination of both\.

Table 9:Detailed human\-annotation results for the Drug data lake, categorized by modality\-specific reasoning requirements\.Question TypeQuestionsCorrectAISAnswerablePassage797768762754Table776739720701Passage\-Table447429421410Passage\-hop706681668646Table\-hop848817802793Total3,574343433733304

## Appendix BPrompt Templates

This section details the exact prompt templates used for dataset synthesis, LLM\-as\-a\-judge filtering, and baseline evaluation\.

### B\.1Candidate Synthesis Prompts

To generate candidate QA pairs, we provided the language model with a sampled evidence subgraph and the following instructions\. The system prompt was shared across synthesis settings, while the task prompt was instantiated with minor wording changes depending on whether the evidence subgraph contained a model–dataset pair, multiple related model cards, multiple related dataset cards, or an anchor\-plus\-positive\-plus\-distractor setup\.System Instructions: You are a data scientist working with models and data on huggingface\.Input Context: \{evidence\_subgraph\}Task: I’m going to show you one or more model cards and/or dataset cards\. What natural question and answer might you ask that implicitly requires consulting information from the shown evidence subgraph? You should pretend that you cannot see the cards when asking the question \(except in the anchored setting, where the anchor card is visible but the supporting and distractor cards are not\)\. If you cannot think of one, then return an empty string\. This should be a question that naturally requires consulting information from the provided evidence and should not be a chain of multiple unrelated questions that merely happen to require multiple cards\. After the question, provide: \(1\) a direct answer \(short, exact answer\), and \(2\) an explained answer \(slightly longer answer with brief justification from the cards\)\. Always answer in the following format:

```
<question>
<direct_answer>
<explained_answer>
```

### B\.2Baseline Evaluation Prompts: Agentic Tool Use \(XMode\)

For our agentic baselines, we framed the data lake navigation as an iterative tool\-use task\.

Agent Instructions: You are an AI agent for data lake question answering\. Given a user question, iteratively reason over the question, retrieve the top\-kkrelevant documents from the indexed data lake for the current question or sub\-question, and summarize each document with respect to both the original question and the current sub\-question\. After each retrieval round, decide whether the accumulated evidence is sufficient to answer the original question; if not, propose the next sub\-question and continue\. We use up to 3 iterations, retrievek=5k=5documents per step, and use either similarity search or MMR retrieval\. After the final step, generate the answer only from the accumulated evidence\.

## Appendix CHuman Annotation Guidelines and Interface

To ensure high data quality, annotators were provided with a custom web interface that displayed the generated question, the synthesized answer, and the exact supporting evidence chunks\. Annotators were tasked with independently verifying two criteria:*Answerability*\(can the question be answered exclusively using the cited evidence?\) and*Attribution*\(is the generated answer completely faithful to the cited text/table?\)\. We emphasize that this task evaluates strict textual entailment and reading comprehension based entirely on the provided evidence, rather than requiring annotators to verify external scientific or financial validity\.

![Refer to caption](https://arxiv.org/html/2607.12310v1/figures/annotation-image.png)Figure 3:Screenshot of the custom validation UI\. Annotators are presented with the highlighted evidence constraints and must explicitly accept or reject the instance based on the provided rubrics\.- •Is the answer correct?Mark “Yes” only when the answer matches facts stated in the provided evidence\.
- •Does the evidence support the answer?Mark “Yes” only when specific sentences/rows in the provided evidence explicitly support the answer\.
- •Any additional comments/thoughts\.Add a short note explaining the decision, especially for “No” labels, and reference the relevant evidence span\.
- •Use only provided evidence\.Do not use outside knowledge; if the answer contradicts the provided evidence, mark it incorrect and unsupported\.

## Appendix DExtended Case Studies and Error Analysis

To qualitatively illustrate the quantitative failure modes discussed in Sec[6](https://arxiv.org/html/2607.12310#S6), we provide representative trace\-level examples from the AI/ML and Drug data lakes\. The cases below show three recurring error patterns: schema\-confusable retrieval, relation chaining failure across closely related model cards, and multimodal metadata composition failure in vision–language datasets\.

Table 10:A representative schema\-grounding failure in the AI/ML data lake: retrieval finds the right capability class \(function calling\) but misses the target card’s exact token schema\.DomainAI/ML \(Schema\-Confusable Retrieval\)QuestionWhat prompt format does0\-hero/Matter\-0\.1\-7Buse, and which special tokens does it provide for function calling?Gold EvidenceTheMatter\-0\.1\-7Bcard states that the model uses ChatML prompt format; its Function Calling section lists<\|begin\_func\|\>,<\|end\_func\|\>,<\|begin\_func\_response\|\>, and<\|end\_func\_response\|\>\.Gold AnswerChatML prompt format; function\-call tokens<\|begin\_func\|\>and<\|end\_func\|\>; function\-response tokens<\|begin\_func\_response\|\>and<\|end\_func\_response\|\>\.RAG Predicted EvidenceThe retriever surfaced schema\-similar tool\-use models such asmeetkai/functionary\-medium\-v3\.1andmeetkai/functionary\-small\-v3\.1, plus unrelated cards, instead of grounding on the target Matter card\.RAG Predicted Answer“Prompt format: ChatML\-style\. Function\-calling special tokens:<\|tool\_call\|\>and<\|tool\_response\|\>\.”Failure Mode AnalysisThe retriever found semantically similar function\-calling cards, but not the target\-specific token schema\. The generator then copied the more commontool\_call/tool\_responseconvention, yielding a partially correct yet still wrong answer\.Table 11:A relation\-chaining failure: the system retrieves locally similar model cards but does not gather the exact sibling set required for accurate cross\-card comparison\.DomainAI/ML \(Relation Chaining\)QuestionAmong fsicoli’s fine\-tuned Whisper models \(whisper\-small\-pt\-1000h,whisper\-medium\-pt\-1000h, andwhisper\-large\-v3\-pt\-1000h\), which achieves the lowest WER on the shared evaluation dataset, and what are the WERs for each model?Gold EvidenceThe three model cards report WERs of 0\.1490 \(whisper\-small\-pt\-1000h\), 0\.1147 \(whisper\-medium\-pt\-1000h\), and 0\.1113 \(whisper\-large\-v3\-pt\-1000h\), all on the same Portuguese speech dataset family\.Gold Answerwhisper\-large\-v3\-pt\-1000hhas the lowest WER \(0\.1113\), followed bywhisper\-medium\-pt\-1000h\(0\.1147\) andwhisper\-small\-pt\-1000h\(0\.1490\)\.RAG Predicted EvidenceThe retriever pulled neighboring Whisper cards such asfsicoli/whisper\-large\-v3\-pt\-cv16and several unrelated Whisper models, but failed to assemble the three requiredpt\-1000hcards into a single comparison set\.RAG Predicted Answer“I don’t see WERs for fsicoli’spt\-1000hmodels in the provided context\. The only fsicoli metric here is forwhisper\-large\-v3\-pt\-cv16with WER≈\\approx0\.1076\.”Failure Mode AnalysisThis error is not a pure hallucination; it is a chaining failure\. The system retrieved a nearby family member with a very similar identifier, but missed the exact trio needed for cross\-card aggregation, so it could not complete the comparison\.Table 12:A multimodal composition failure: the answer depends on jointly grounding license metadata and image\-rights caveats across two vision–language dataset cards\.DomainAI/ML \(Multimodal Composition\)QuestionDoFreedomIntelligence/ALLaVA\-4VandFreedomIntelligence/ALLaVA\-4V\-Chineseallow commercial use?Gold EvidenceTheALLaVA\-4Vdataset card lists licensecc\-by\-nc\-4\.0;ALLaVA\-4V\-Chineselistsapache\-2\.0\. TheALLaVA\-4Vcard also states that the authors “do not own right to any image contained within theimages\.zipfile,” requiring users to respect original image copyrights\.Gold AnswerNo\.ALLaVA\-4Vis non\-commercial, whileALLaVA\-4V\-Chineseis Apache\-2\.0; moreover, commercial reuse still requires checking the rights of the underlying third\-party images\.RAG Predicted EvidenceThe retriever returned image\-generation model cards such asLarryAIDraw/\*andsvjack/Stable\-Diffusion\-FineTuned\-zh\-\*, rather than the two ALLaVA dataset cards whose licenses had to be composed\.RAG Predicted Answer“I can’t confirm that from the provided context\. If the license is Apache\-2\.0, MIT, BSD, or CC\-BY, commercial use is generally allowed; if it is CC\-BY\-NC, it is not allowed\.”Failure Mode AnalysisThe query requires composing metadata from two vision–language dataset cards and incorporating an image\-rights caveat\. Retrieval drifted toward visually related generative model cards, so the generator defaulted to generic licensing heuristics instead of grounding on the relevant multimodal datasets\.Table 13:Representative Drug failure case from the evidence\-provided ablation: the model receives the correct table evidence but still fails to produce the correct answer\.DomainDrug \(Evidence\-Provided Generation Failure\)QuestionWhich cited reference mentions “Lorazepam”, and does that reference include an ISBN?Gold EvidenceIndrugbank\-drugs\_textbooks, the row withref\-id=T385has citation “Ghiasi N\. and Marwaha R\. \(2018\)\. Lorazepam” andisbn = NaN\(parent\_key=DB00186\)\.Gold AnswerT385; it does not include an ISBN \(missing / NaN\)\.LLM Generated Answer“Not determinable from the provided snippets; no citation mentioning ‘Lorazepam’ is shown, so the ISBN status is unknown\.”Failure Mode AnalysisIn the context\-bundle ablation, the model was given the correct evidence chunk \(plus four distractor tables\), so retrieval was not the bottleneck\. The key row was present in context, but the model failed to attend to and extract it, defaulting to an abstention\-style response\. This is a generation\-under\-sufficient\-evidence failure rather than a retrieval miss\.
## Appendix EControlled Context\-Bundle Ablation

We ran a fixed context\-bundle ablation on the AI/ML, Retail Bank, and Drug data lakes to reduce retrieval variance and better isolate answer synthesis\. In each case, the model was given a fixed evidence bundle containing the gold evidence plus relevant distractors, and we evaluated both a direct answer and a reasoning answer using the same LLM\-as\-a\-judge pipeline as in the main experiments\.

Table 14:LLM correctness results for the fixed context\-bundle ablation across data lakes\.Data LakeBaselineAvg\. CorrectnessDirect CorrectnessReasoning CorrectnessAI/MLContext Bundle0\.98750\.97501\.0000Retail BankContext Bundle0\.75000\.75000\.7500DrugContext Bundle0\.62500\.60000\.6500The results show a sharp cross\-domain difference\. On AI/ML, performance is nearly saturated once gold evidence is present; Retail Bank is materially higher but not saturated, while Drug remains lowest under the same setup, indicating that synthesis over heterogeneous evidence remains a substantial challenge; reasoning is equal to or stronger than direct answers across lakes\.

## Appendix FLLM\-as\-a\-Judge Consistency Analysis

To assess the stability of our rubric\-based LLM\-as\-a\-judge evaluation, we repeated the same AI/ML data lake evaluation three times under an identical setup and report per\-question\-family correctness in Table[15](https://arxiv.org/html/2607.12310#A6.T15)\. The results are highly consistent across runs, with only minor variation across the three repeated tests, suggesting that the judge\-based correctness metric is sufficiently stable for the comparative analyses reported in the main paper\.

Table 15:Repeated\-run LLM correctness on the AI/ML data lake\.Question TypeLLM Correctness – Test 1LLM Correctness – Test 2LLM Correctness – Test 3Model0\.2500\.2600\.245Dataset0\.1400\.1450\.140Model\-Dataset0\.3200\.3400\.320Model\-hop0\.1550\.1650\.160Dataset\-hop0\.3800\.3850\.375
## Appendix GRetail Bank Data Lake QA Generation Example

Figure[4](https://arxiv.org/html/2607.12310#A7.F4)illustrates the end\-to\-end question\-answer synthesis pipeline for the Retail Bank data lake\. The process begins by sampling a dense evidence subgraph consisting of a client profile, a transaction ledger, and a governing institutional policy\. The generator model is instructed to synthesize a realistic operational query \(e\.g\., a fraud analyst checking a transaction limit\) that strictly requires joining facts across all three sampled artifacts\. Following generation, the pipeline programmatically filters deictic or ill\-formed queries, and an LLM\-as\-a\-judge verifies that the question cannot be answered via parametric memory alone\. Finally, the instance undergoes human validation to ensure strict attribution to the highlighted evidence spans\.

![Refer to caption](https://arxiv.org/html/2607.12310v1/figures/finlake-qa-generation.png)Figure 4:Example of QA generation for the Retail Bank data lake\.
## Appendix HOpen Source Model Results Ablation

Table 16:Baseline results for the AI/ML data lake using gpt\-oss\-20bRAGZero\-shotQuestion TypeRRAAFFTTAATTModel0\.090\.290\.210\.360\.200\.05Dataset0\.150\.090\.110\.310\.070\.05Model\-Dataset0\.120\.300\.250\.350\.130\.05Model\-hop0\.130\.170\.120\.430\.140\.06Dataset\-hop0\.230\.280\.300\.440\.180\.06Table 17:Baseline results for the Retail Bank data lake using gpt\-oss\-20b\.XModeReSPZero\-shotPersonaRTR\_\{T\}RPR\_\{P\}AAFFTTRTR\_\{T\}RPR\_\{P\}AAFFTTAATTAccountant0\.790\.940\.500\.371\.020\.951\.000\.610\.401\.080\.180\.15Auditor0\.840\.900\.490\.420\.920\.940\.850\.500\.471\.000\.560\.16Customer service \(CS\)0\.820\.980\.550\.531\.120\.931\.000\.490\.390\.960\.560\.14CS manager0\.700\.870\.610\.560\.950\.861\.000\.770\.690\.930\.570\.11Data analyst0\.620\.690\.500\.401\.000\.890\.650\.460\.310\.950\.240\.13Financial planner0\.640\.990\.560\.501\.100\.850\.900\.580\.600\.960\.720\.14Fraud analyst0\.720\.920\.520\.390\.990\.950\.900\.590\.501\.140\.670\.13Insurance analyst0\.520\.890\.490\.371\.020\.740\.750\.540\.391\.100\.290\.17Salesperson0\.560\.660\.350\.331\.010\.770\.600\.230\.180\.970\.290\.13Table 18:Baseline results for the Drug data lake using gpt\-oss\-20b\.XModeReSPZero\-shotQuestion TypeRPR\_\{P\}RTR\_\{T\}AAFFTTRPR\_\{P\}RTR\_\{T\}AAFFTTAATTPassage0\.98–0\.900\.910\.581\.00–0\.930\.940\.840\.340\.24Table–0\.810\.500\.650\.52–0\.750\.550\.600\.640\.180\.19Passage\-Table0\.730\.980\.560\.700\.580\.901\.000\.680\.750\.750\.360\.22Passage\-hop0\.72–0\.740\.790\.740\.65–0\.750\.801\.120\.430\.24Table\-hop–0\.480\.350\.500\.58–0\.600\.410\.520\.750\.230\.21