Harness choice is a big deal.

So much room to advance and improve results across the board with agent harnesses.

Great paper highlighting this.

New research releases DataSpace, a benchmark where data agents produce verifiable tabular results from heterogeneous workspaces. 410 cross-language tasks over 7,439 artifacts totalling 15.01 GB across CSV, JSON, SQLite, Markdown, PDF, and video.

Across six recent frontier multimodal models and five widely used agent harnesses, the best accuracy reaches 66.34%. With the backbone held fixed, swapping the harness moves accuracy by 15.36 points.

Multimodal evidence integration and joins reduce accuracy across all six backbones. The benchmark is nowhere near saturated.

Paper: https://arxiv.org/abs/2608.03451

Track more trending AI papers in our academy: https://academy.dair.ai

Benchmarking Data Agents for Verifiable Analytics over Heterogeneous Workspaces

Source: https://arxiv.org/html/2608.03451 ,Zhuowen LiangHKUST(GZ)GuangzhouChina,Yupeng XieHKUST(GZ)GuangzhouChina,Xiaotian LinHKUST(GZ)GuangzhouChina,Tianqi LuoHKUST(GZ)GuangzhouChina,Xinyu LiuHKUST(GZ)GuangzhouChina,Yizhang ZhuHKUST(GZ)GuangzhouChina,Zhangyang PengHKUST(GZ)GuangzhouChina,Yuan LiTsinghua UniversityBeijingChina,Zhengxuan ZhangHKUST(GZ)GuangzhouChina,Jiayi ZhangHKUST(GZ)GuangzhouChina,Nan TangHKUST(GZ)GuangzhouChina,Guoliang LiTsinghua UniversityBeijingChinaandYuyu LuoHKUST(GZ)GuangzhouChina

Abstract.

Data agents enable natural-language analytics over organizational workspaces, where relevant evidence may be scattered across databases, structured files, long documents, and multimedia. Existing benchmarks largely isolate structured querying, retrieval, or open-ended analysis, leaving heterogeneous evidence discovery, complete tabular outputs, and deterministic evaluation insufficiently unified. We introduceDataSpace, a benchmark in which data agents produce verifiable tabular results from task-local heterogeneous workspaces. It contains 410 cross-language tasks and 7,439 artifacts totaling 15.01 GB across CSV, JSON, SQLite, Markdown, PDF, and video.DataSpacealso served as the official evaluation benchmark for the KDD Cup 2026 Data Agents for Complex Data Analysis competition. Each agent receives only a question and workspace and returns the complete requested tabular result. We constructDataSpacewithDataSpace-Builder, an execution-grounded framework comprising cross-language transformation, constraint-aware relational sampling, modality routing and artifact rendering, and human review and task repair by 11 domain experts. A deterministic evaluator performs header-invariant column alignment, type- and precision-aware normalization, and order-aware row comparison. Across six recently released frontier multimodal models and five widely used agent harnesses, the best accuracy reaches 66.34%, while harness choice creates a 15.36-point spread with the backbone fixed. Multimodal evidence integration and joins consistently reduce accuracy across all six backbones. These results show thatDataSpaceremains unsaturated and identify key challenges for improving data-agent reliability.

††copyright:none## 1.Introduction

Figure 1.TheDataSpacetask interface, illustrated with a fund-risk task. The agent combines an alert rule from video, category benchmarks extracted from a long PDF, and daily NAV queried from SQLite, then aligns entities, computes the requested metrics, and returns the complete tabular result.Data agents are emerging as a natural-language interface to organizational data(Wang and Li,2025; Sunet al.,2025; Zhanget al.,2025b,a; Teable,2026; Liet al.,2026b; Zhuet al.,2025b; Shuaiet al.,2026; Luoet al.,2025; Xieet al.,2024; Tanget al.,2026b). In realistic analytical settings, however, the information needed to answer a user question rarely resides in a single clean table or a preselected database(Wanget al.,2025; Qiet al.,2026; Egget al.,2025).

Table 1.Comparison of representative benchmarks in artifact coverage, workspace demands, and answer/evaluation semantics.TypeBenchmark#TasksInput artifactsWorkspace requirementsOutput and evaluationStructuredWikiTableQuestions(Pasupat and Liang,2015)22,033–✓––––––––✓–Spider(Yuet al.,2018)10,181✓––––––––✓✓–BIRD(Liet al.,2023)12,751✓––––––––✓✓–Spider 2.0(Leiet al.,2025)632✓–✓–✓✓–––✓✓–UnstructuredHotpotQA(Yanget al.,2018)113K––✓–✓✓––––✓–CRAG(Yanget al.,2024)4,409––✓–✓✓––––––MMLongBench-Doc(Maet al.,2024b)1,062––✓–––✓–––✓–Data agentDABStep(Egget al.,2025)450+–✓✓–✓✓––––✓–KramaBench(Laiet al.,2025)104–✓✓–✓✓––––––LongDA(Liet al.,2026c)505–✓✓–✓✓✓–––✓–DataCross(Qiet al.,2026)200✓✓✓–✓✓–✓––––FDABench(Wanget al.,2025)2,007✓✓✓✓✓✓––––––OursDataSpace410✓✓✓✓✓✓✓✓✓✓✓✓

Notes.Input/workspace checkmarks denote explicit coverage; output/evaluation checkmarks denote benchmark-wide requirements. Files: standalone structured/semi-structured artifacts; Docs: textual/visual documents; Media: audio/video. Cross-art.: multi-artifact/system integration; Discovery: sources not preselected; Long docs: explicit long-document processing; Doc→\rightarrowrecords: document fields/records feed downstream analysis; Cross-lang.: joint question–workspace input. Complete table: correctness requires the complete result table, submitted directly or obtained by query execution; Model-free: no LLM judge; Schema-inv.: columns align despite header wording or order.

The question and its evidence may cross languages and representations, spanning relational databases, structured and semi-structured files, business documents, and multimedia artifacts alongside valid but irrelevant files. An effective data agent therefore acts as aworkspace solver: it inspects the available data, selects sources and tools, aligns information across representations, executes multi-step computations, and returns a result that the user can directly consume.

Existing benchmarks capture complementary parts of this setting.Structured-data benchmarks, from Spider(Yuet al.,2018)to Spider 2.0(Leiet al.,2025), offer strong tests of relational reasoning with deterministic evaluation, but generally identify the relevant table or database in advance.Unstructured-data benchmarkssuch as MMLongBench-Doc(Maet al.,2024b)introduce long, visually rich inputs, yet focus primarily on evidence retrieval, grounding, and answer synthesis.Data-agent benchmarksmove closer to open-ended analytical workspaces: KramaBench(Laiet al.,2025)studies data-to-insight pipelines over data lakes, while FDABench(Wanget al.,2025)extends analysis across structured data, documents, and media. As summarized in Table1, these advances have not yet unified three properties central to realistic data analysis:(L1) Workspace scope:a task-local workspace spanning structured files, databases, long documents, and multimedia artifacts, with language variation across both the question and data;(L2) Output contract:a consistent objective requiring the complete analytical result rather than a factoid, pipeline, or open-ended report; and(L3) Evaluation semantics:deterministic evaluation that accepts equivalent representations while rejecting incomplete or erroneous answers.

We therefore introduceDataSpace111, a benchmark for verifiable data analytics over self-contained heterogeneous workspaces (Figure1). An agent receives only a natural-language question and a task-local workspace, autonomously discovers and combines the available data, and returns the complete requested tabular result.DataSpacecontains 410 cross-language tasks and 7,439 artifacts totaling approximately 15 GB. Chinese and English may occur across both the user question and its workspace artifacts. The workspaces span CSV, JSON, SQLite, Markdown, PDF, and Video, and each task is paired with a complete tabular reference answer.DataSpacealso served as the official evaluation benchmark for the KDD Cup 2026 Data Agents for Complex Data Analysis competition(KDD Cup 2026,2026).

To construct such cross-language, heterogeneous workspaces reliably, we proposeDataSpace-Builder, an execution-grounded framework that transforms instances from EHRSQL(Leeet al.,2023)and BULL(Zhanget al.,2024), two English Text-to-SQL benchmarks covering clinical and financial analytics. Their relational databases supply domain data, while executable SQL provides parseable analytical logic and execution-based validation.DataSpace-Buildercomprises four stages:Cross-Language Transformation,Constraint-Aware Relational Sampling,Modality Routing & Artifact Rendering, andHuman Review & Task Repair. Together, these stages transform source questions, databases, and query logic into cross-language, heterogeneous workspace tasks and derive their reference answers through execution. The resulting question, workspace, reference answer, and evaluation semantics are cross-reviewed by a panel of 11 domain experts, with disputed cases discussed and repaired before release. We also design a deterministic evaluator that aligns columns regardless of header wording or position, normalizes equivalent value formats, and compares rows according to whether their order matters for the task.

Contributions.Our contributions are:

• •A heterogeneous workspace benchmark.We introduceDataSpace, comprising 410 cross-language tasks and six modalities under a uniform, exactly verifiable tabular-output objective.
• •An execution-grounded construction framework.We developDataSpace-Builder, which transforms executable Text-to-SQL resources into task-local heterogeneous workspaces with expert review and repair.
• •A semantics-aware evaluator.We provide deterministic evaluation of complete tabular results that tolerates equivalent representations while rejecting incomplete or erroneous outputs.
• •An empirical study of data agents.We establish baselines across six frontier multimodal models and five agent harnesses; the best reaches 66.34% accuracy, harness choice produces a 15.36-point spread, and multimodal evidence and joins consistently reduce performance.

2.Related Work

Structured-data benchmarks.Natural-language analytics over structured data is studied through table question answering and Text-to-SQL(Liet al.,2024,2025b,2025a,2026a; Peiet al.,2026; Zhuet al.,2026; Maet al.,2024a; Liuet al.,2025a,b; Zhuet al.,2025a). Table QA benchmarks such as WikiTableQuestions(Pasupat and Liang,2015)predict denotations over semi-structured tables. Spider(Yuet al.,2018)and BIRD(Liet al.,2023)cover cross-domain and large-scale databases, while EHRSQL(Leeet al.,2023)and BULL(Zhanget al.,2024)capture clinical and financial analytics. Spider 2.0(Leiet al.,2025)further introduces enterprise artifacts around the SQL workflow. Executable queries enable deterministic result-level evaluation, but these settings provide limited coverage of evidence discovery and reconciliation across separate files, documents, and media.

Unstructured-data benchmarks.HotpotQA(Yanget al.,2018)and CRAG(Yanget al.,2024)test retrieval and synthesis over multi-hop or retrieval-augmented corpora, while FinanceBench(Islamet al.,2023)and MMLongBench-Doc(Maet al.,2024b; Lianget al.,2026)target financial reports and long, visually rich documents. HybridQA(Chenet al.,2020), MultiModalQA(Talmoret al.,2021), and Video-MME(Fuet al.,2025)add linked tables, passages, images, or video. These settings provide strong tests of evidence localization, cross-page reasoning, and perception, but generally target factoids, short lists, choices, or free-form responses. They rarely require recovering typed record collections from long documents and combining them with other workspace data to produce a complete tabular result.

Data-agent benchmarks.DABStep(Egget al.,2025), KramaBench(Laiet al.,2025), and LongDA(Liet al.,2026c; Bianet al.,2025)cover multistep processing, data-lake discovery, and long-document navigation. Data Agent Benchmark(Maet al.,2026)and AgenticDataBench(Sunet al.,2026)emphasize multi-system querying or recurring data-science skills, while DataCross(Qiet al.,2026)and FDABench(Wanget al.,2025; Chenet al.,2025; Zhanget al.,2025c)incorporate visual tables and media. This family is closest to our setting, but its targets range from factoid answers and executable pipelines to choices and reports, accompanied by execution-, rubric-, or model-based evaluation(Xieet al.,2025; Tanget al.,2026a).DataSpaceinstead holds the task contract fixed: every task requires a complete tabular result scored by the same deterministic protocol.

3.Benchmark Overview and Task Formulation

DataSpaceevaluates a data agent as a workspace solver: given an analytical question and the contents of a task-local workspace, the agent must return the complete typed table requested by the user. This section defines the task interface, summarizes the benchmark scope, and identifies the capabilities exercised by this setting.

3.1.Task Formulation

For taskii, the public input is

(1)xi=(qi,𝒲i),𝒲i=𝒲istr∪𝒲idoc∪𝒲imed,x_{i}=(q_{i},\mathcal{W}_{i}),\qquad\mathcal{W}_{i}=\mathcal{W}_{i}^{\mathrm{str}}\cup\mathcal{W}_{i}^{\mathrm{doc}}\cup\mathcal{W}_{i}^{\mathrm{med}},whereqiq_{i}is a natural-language question. The workspace contains structured and semi-structured artifacts𝒲istr\mathcal{W}_{i}^{\mathrm{str}}(CSV, JSON, and SQLite), document artifacts𝒲idoc\mathcal{W}_{i}^{\mathrm{doc}}(Markdown and PDF), and media artifacts𝒲imed\mathcal{W}_{i}^{\mathrm{med}}(video). The initial observationo0o_{0}gives the agent the question and access to the workspace root; artifact contents are acquired as the agent interacts with the workspace.

Let𝒯A\mathcal{T}_{A}denote the tools available to agentAA. They may include file-system inspection, structured-data parsing, SQL execution, code execution, document extraction, and video understanding. The action space, agent state, and nonterminal environment transition are jointly defined as

(2)𝒜A\displaystyle\mathcal{A}_{A}={Call⁡(τ,θ):τ∈𝒯A,θ∈Θτ}\displaystyle=\{\operatorname{Call}(\tau,\theta):\tau\in\mathcal{T}_{A},\theta\in\Theta_{\tau}\}∪{Answer⁡(Y):Y​is a tabular result},\displaystyle\quad\cup\{\operatorname{Answer}(Y):Y\text{ is a tabular result}\},ht\displaystyle h_{t}=(o0,a0,…,at−1,ot),st=(ht,mt),\displaystyle=(o_{0},a_{0},\ldots,a_{t-1},o_{t}),\qquad s_{t}=(h_{t},m_{t}),at\displaystyle a_{t}=πA​(st),(ot+1,mt+1)=Execi⁡(st,at),\displaystyle=\pi_{A}(s_{t}),\qquad(o_{t+1},m_{t+1})=\operatorname{Exec}_{i}(s_{t},a_{t}),whereΘτ\Theta_{\tau}is the argument space of toolτ\tau,mtm_{t}is working memory, andπA\pi_{A}is the agent policy. For a tool-call action,Execi\operatorname{Exec}_{i}executes the selected tool in the fixed workspace and returns its observation. The terminal action isaT=Answer⁡(𝒴^i)a_{T}=\operatorname{Answer}(\widehat{\mathcal{Y}}_{i}), where𝒴^i\widehat{\mathcal{Y}}_{i}is a tabular result serialized as a CSV file. Figure1illustrates the resulting task-level input–output interface.

The benchmark-side record additionally contains the reference result𝒴i\mathcal{Y}_{i}and a compact evaluation configurationcic_{i}. The benchmark record and binary task score are

(3)bi=(xi,𝒴i,ci),si=𝟏​[𝒴^i≡ci𝒴i].b_{i}=(x_{i},\mathcal{Y}_{i},c_{i}),\qquad s_{i}=\mathbf{1}\!\left[\widehat{\mathcal{Y}}_{i}\equiv_{c_{i}}\mathcal{Y}_{i}\right].The configuration records the semantic type of each reference column, numeric comparison rules where needed, and whether row order is significant. It belongs to the evaluation protocol rather than the question–workspace input. A task is correct when its complete tabular prediction matches the reference result under these semantics; missing or extra rows and columns make the prediction incorrect even when some returned values match. Section4.5specifies the equivalence criterion and the aggregate metric.

3.2.Benchmark Scope

DataSpacecontains 410 tasks spanning financial, macroeconomic, and healthcare analytics. Its workspaces combine CSV, JSON, SQLite, Markdown, PDF, and video in 13 modality combinations, and questions and artifacts may mix Chinese and English. Every sampled source table remains represented: long documents may encode records or fields, while video may supply a condition, intermediate value, or answer. Answer tables range from a single cell to multi-column outputs with thousands of rows.

Tasks couple four capabilities:(i) workspace discovery;(ii) interpretation and alignmentof types, schemas, entities, units, and languages;(iii) relational computation, such as filtering, joining, aggregation, ranking, and temporal reasoning; and(iv) complete tabular-result materialization.

4.Benchmark Construction and Evaluation

In this section, we introduceDataSpace-Builder, a four-stage pipeline that transforms Text-to-SQL instances into reviewed heterogeneous workspace tasks (Figure2). Its stages are Cross-Language Transformation, Constraint-Aware Relational Sampling, Modality Routing & Artifact Rendering, and Human Review & Task Repair. We then present the evaluation protocol for finalized tasks. Further implementation details, construction costs, and running examples appear in AppendixA.

Figure 2.Overview ofDataSpace-Builder. Text-to-SQL instances pass through Cross-Language Transformation, Constraint-Aware Relational Sampling, Modality Routing & Artifact Rendering, and Human Review & Task Repair before being frozen as heterogeneous benchmark records.### 4.1.Cross-Language Transformation

Source corpora.We use EHRSQL(Leeet al.,2023)and BULL(Zhanget al.,2024)as source corpora. These English Text-to-SQL benchmarks cover clinical and financial analytics, respectively. Each selected source instance provides a natural-language questionq0q_{0}, a relational database𝒟0\mathcal{D}_{0}, and executable SQLσ0\sigma_{0}.

Joint transformation.Both source corpora are English-only at the question–database level, whereas our task setting varies language across the complete question–workspace pair. Translating the question or database in isolation can misalign entity names, predicate values, and executable SQL. We therefore treat cross-language transformation (CLT) as a joint migration of the question, database state, and executable workload. For a source tuple(𝒟0,q0,σ0)(\mathcal{D}_{0},q_{0},\sigma_{0}), we independently choose the target languagesℓD\ell_{D}andℓq\ell_{q}for the database and question:

(4)(𝒟c,qc,σc;M)=CLTℓD,ℓq​(𝒟0,q0,σ0),(\mathcal{D}_{c},q_{c},\sigma_{c};M)=\mathrm{CLT}_{\ell_{D},\ell_{q}}(\mathcal{D}_{0},q_{0},\sigma_{0}),where𝒟c\mathcal{D}_{c},qcq_{c}, andσc\sigma_{c}are the transformed database, question, and SQL, respectively, andMMis a materialized replacement map.

Consistency-aware rewriting.To keep the same field or entity consistent wherever it appears, we link columns connected by a foreign key, a shared name, or substantial value overlap. Each linked group is translated jointly, while identifiers, codes, URLs, dates, and numbers remain unchanged. LLM-generated translations are stored as table-, column-, and cell-level mappingsM={Mtab,Mcol,Mval}M=\{M_{\mathrm{tab}},M_{\mathrm{col}},M_{\mathrm{val}}\}. We useMMto deterministically rewrite the database and SQL and translate the question with the same terminology.

Validation.We retain a transformed tuple only after structural and execution checks. The rewritten SQL must reference existing translated entities and execute on𝒟c\mathcal{D}_{c}, and its result must satisfyExec⁡(𝒟c,σc)≅τM​(Exec⁡(𝒟0,σ0))\operatorname{Exec}(\mathcal{D}_{c},\sigma_{c})\cong\tau_{M}(\operatorname{Exec}(\mathcal{D}_{0},\sigma_{0})), whereτM\tau_{M}applies the induced value translation to the source execution result. Execution equivalence verifies database–SQL consistency but cannot establish thatqcq_{c}preserves the original intent; we therefore use an LLM judge to verify question–SQL semantic alignment. Instances that fail either check are repaired and re-executed or rejected before relational data sampling.

4.2.Constraint-Aware Relational Sampling

Task-local data diversification.Source Text-to-SQL datasets commonly associate many questions with a small number of shared database states. Directly reusing these states would produce workspaces with highly repetitive entities, values, and relational neighborhoods. We instead construct task-local relational instances by sampling rows while retaining the complete table inventory and schema. The sampled scale also makes downstream artifact rendering tractable, particularly when complete tables are transformed into long Markdown or PDF documents.

Sampling safeguards.Naively sampling each table independently can remove a condition value, break a join path, or disconnect related entities. We therefore construct a lightweight safeguard set𝒞s\mathcal{C}_{s}that combines primary keys, foreign keys, and known inter-table relationships with join columns, predicate and boundary values, and target entities extracted from the SQL AST.

Relationally consistent sampling.Given a seeded sampling policyψ\psi, we retain rows required by query safeguards and then propagate their key values across schema relationships. Remaining rows are selected according to the table-level sampling budget, including rows from tables not referenced by the source SQL.

Materialization and re-execution.All sampled tables are materialized as a task-local intermediate database:

(5)𝒟s=Sample⁡(𝒟c;ψ,𝒞s),𝒴s=Exec⁡(𝒟s,σc).\mathcal{D}_{s}=\operatorname{Sample}(\mathcal{D}_{c};\psi,\mathcal{C}_{s}),\qquad\mathcal{Y}_{s}=\operatorname{Exec}(\mathcal{D}_{s},\sigma_{c}).The query result𝒴s\mathcal{Y}_{s}becomes the candidate reference. It is not required to equal the source result; sampling may change entities, aggregates, rankings, or result cardinality. A sample is accepted only if the SQL executes successfully, required relationships remain valid, and the result has not become unintentionally empty or semantically degenerate. Otherwise, the database is resampled, repaired, or rejected.

4.3.Modality Routing & Artifact Rendering

Query-independent base routing.The intermediate database fixes the relational content of a task; this stage changes how agents encounter that content. We first apply a seeded rule-based policyπr\pi_{r}, with fixed seedzrz_{r}, that assigns each sampled table to one or more compatible renderers and materializes the base workspace𝒲base\mathcal{W}_{\mathrm{base}}from𝒟s\mathcal{D}_{s}. The policy considers schema properties, renderer compatibility, and batch-level modality coverage using only sampled-table metadata. Its base renderers are CSV, JSON, SQLite, Markdown, and PDF. Video is introduced separately as a task-level, query-conditioned augmentation because its construction may depend on the question, executable SQL, and candidate answer. Every sampled table receives a base representation, while the number and sizes of the resulting artifacts follow from the source table inventory, sampled contents, and applicable renderers.

Structured artifact rendering.CSV and record-oriented JSON expose individual sampled tables, whereas a task-local SQLite artifact can retain several related tables and their schema. The renderers preserve headers, cell values, nulls, and duplicate rows; the SQLite renderer additionally preserves declared column types and key relationships. Each output is parsed back into a canonical relation and compared with the rows assigned to that renderer.

Fact-grounded data documents.For a routed tableRR, letXR=𝒟s​[R]X_{R}=\mathcal{D}_{s}[R]denote its complete sampled contents. The LLM-produced planpR=PlanDoc⁡(XR)p_{R}=\operatorname{PlanDoc}(X_{R})specifies a document style, record-identifying columns, semantic attribute groups, and bounded row batches. Document generation then follows

(6)ℬR\displaystyle\mathcal{B}_{R}=BuildBlocks⁡(XR;pR),\displaystyle=\operatorname{BuildBlocks}(X_{R};p_{R}),dR\displaystyle d_{R}=Assemble⁡(DocGen⁡(ℬR;pR)).\displaystyle=\operatorname{Assemble}(\operatorname{DocGen}(\mathcal{B}_{R};p_{R})).Each block repeats the identifying columns and supplies an LLM with its exact field names, types, and values. The generated sections are assembled as MarkdowndRd_{R}and converted to PDF when required. Figure14visualizes the alignment from source cells to generation blocks and document spans. The complete running example is provided in AppendixA.3.3.

Query-conditioned video rendering.Following DataMagic(Xieet al.,2026), we render tabular data as data-insight videos, while conditioning content selection and task integration on the query. For selected tasks, the renderer derives typed evidence atoms from the executable SQLσc\sigma_{c}and sampled tabular result𝒴s\mathcal{Y}_{s}, then constructs the video and integrates it into the task:

(7)ℰv\displaystyle\mathcal{E}_{v}=Select⁡(AST⁡(σc),𝒴s),\displaystyle=\operatorname{Select}(\operatorname{AST}(\sigma_{c}),\mathcal{Y}_{s}),Sv\displaystyle S_{v}=Storyboard⁡(ℰv),v=Render⁡(Sv,TTS⁡(Sv)),\displaystyle=\operatorname{Storyboard}(\mathcal{E}_{v}),\quad v=\operatorname{Render}(S_{v},\operatorname{TTS}(S_{v})),(qr,𝒲)\displaystyle(q_{r},\mathcal{W})=Integrate⁡(qc,𝒲base,v,ℰv).\displaystyle=\operatorname{Integrate}(q_{c},\mathcal{W}_{\mathrm{base}},v,\mathcal{E}_{v}).The evidence setℰv\mathcal{E}_{v}contains filter conditions and result cells. A filter condition records a column, operator, and value; a result cell records a row, field, and value. For a multi-step query,predicate abstractionassigns a stable condition to a video scene and removes it from the explicit question. For a simple query with a compact result,answer-evidence renderingdistributes result cells or an intermediate value across multiple scenes. We revise the question only when evidence is moved to video. Figure16shows representative frames from both strategies. AppendixA.3.5further traces the question, evidence atoms, storyboard scenes, rendered video, and resulting workspace for complete tasks. For tasks without video augmentation,(qr,𝒲)=(qc,𝒲base)(q_{r},\mathcal{W})=(q_{c},\mathcal{W}_{\mathrm{base}}).

4.4.Human Review & Task Repair

Execution-based checks verify data–SQL consistency, but cannot determine whether a task is unambiguous or its reference result is correct. We therefore subject every candidate to blind, independent review by two reviewers from a panel of 11 domain experts. Each reviewer first solves the task using only its final question and workspace. After the gold is revealed, both reviewers verify it and independently author the evaluation configurationcic_{i}, covering column types, numeric precision, and ordering. Acceptance requires agreement on the gold and identical canonical configurations. Any disagreement triggers evidence-based discussion and minimal repair of the question, workspace, gold, or configuration, followed by independent rechecking by the same pair. This cycle continues until consensus; unresolved tasks are removed. AppendixA.5gives the full protocol, and Figure17shows the review interface.

4.5.Evaluation Protocol

Task-specific semantics.After human review, each task freezes a configurationci=(oi,{νi​j}j=1di)c_{i}=(o_{i},\{\nu_{ij}\}_{j=1}^{d_{i}}), wheredid_{i}is the number of reference columns,oio_{i}indicates whether row order is required by the question, andνi​j\nu_{ij}is the canonicalization rule for reference columnjj. The rule records a semantic type—text, number, date, datetime, or Boolean—and, for numeric columns, the required integer, decimal-place, or significant-digit precision and any percentage convention. It canonicalizes both reference and predicted values: text is normalized to Unicode NFC and trimmed; numbers are parsed as decimals under the configured precision and unit; and dates, datetimes, Booleans, and nulls are converted to canonical values. AppendixA.6gives the complete protocol, and Figure18shows a frozen task configuration.

Header-invariant joint column alignment.Prediction headers are not scored, and predicted columns need not follow the reference order. Instead, the evaluator searches for a one-to-one mapping from reference columns to predicted columns. LetΠdi\Pi_{d_{i}}be the set of permutations ofdid_{i}columns, and letπ​(j)\pi(j)denote the predicted column mapped to reference columnjj. We use𝒴^i​[:,π]\widehat{\mathcal{Y}}_{i}[:,\pi]to denote the prediction with its columns reordered byπ\pi, andCanonci⁡(Y)\operatorname{Canon}_{c_{i}}(Y)to normalize columnjjofYYwithνi​j\nu_{ij}. A mapping is invalid if any predicted cell cannot be interpreted under its target rule.

Answer equivalence.Tabular results with different shapes are unequal. Otherwise,Rowso⁡(Y)\operatorname{Rows}_{o}(Y)returns the ordered row sequence wheno=1o=1, and the unordered row multiset wheno=0o=0; the latter preserves duplicate multiplicities. Define the canonical row view as𝒱i​(Y)=Rowsoi⁡(Canonci⁡(Y))\mathcal{V}_{i}(Y)=\operatorname{Rows}_{o_{i}}(\operatorname{Canon}_{c_{i}}(Y)). The equivalence criterion introduced in Equation3is then

(8)𝒴^i≡ci𝒴i⇔∃π∈Πdi:𝒱i(𝒴^i[:,π])=𝒱i(𝒴i).\widehat{\mathcal{Y}}_{i}\equiv_{c_{i}}\mathcal{Y}_{i}\iff\exists\pi\in\Pi_{d_{i}}:\quad\mathcal{V}_{i}(\widehat{\mathcal{Y}}_{i}[:,\pi])=\mathcal{V}_{i}(\mathcal{Y}_{i}).Column alignment is evaluated over the full result, so a valid mapping must preserve the association among values within every row. Tasks requesting a ranking or another semantically ordered output are compared as row sequences; all other tasks are compared as unordered row multisets.

Aggregate metric.ForNNtasks, the official metric is Task Accuracy,

(9)Acc=1N​∑i=1N𝟏​[𝒴^i≡ci𝒴i].\mathrm{Acc}=\frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\!\left[\widehat{\mathcal{Y}}_{i}\equiv_{c_{i}}\mathcal{Y}_{i}\right].

5.Benchmark Statistics and Analysis

5.1.Composition and Scale

DataSpacecontains 410 tasks: 363 (88.5%) originate from BULL and 47 (11.5%) from EHRSQL. The benchmark covers fund (158 tasks; 38.5%), stock (120; 29.3%), macroeconomic (85; 20.7%), and healthcare analytics (47; 11.5%). Of these tasks, 265 (64.6%) are cross-language and 145 (35.4%) are single-language. Their workspaces span 25,384 PDF pages, 55.36 million PDF and Markdown characters, and 5.49 hours of video. Reference answers contain 126,409 rows; the largest has 12,962 rows, the widest has six columns, and 92 tasks require row order to be preserved. Table2summarizes the resulting scale.

Table 2.Workspace scale ofDataSpace(410 tasks).

5.2.Evidence Requirements

We annotate one minimal, independently sufficient solution path per task to distinguish available artifacts from required evidence. We group CSV/JSON, SQLite, Markdown/PDF, and video as structured-file, database, document, and video carriers.

(a)Available and required modalities. (b)Required carrier-family intersections.

Figure 3.Workspace availability and annotated solution evidence. Available denotes presence in the workspace, whereas required denotes use by the verified solution path.Available versus required evidence.Figure3(a)shows that CSV occurs in every workspace, while JSON, SQLite, Markdown, and PDF each occur in more than 93% of tasks, yet the verified paths use CSV in only 58 tasks and SQLite in 177. Long documents provide required evidence in 135 tasks; among 189 video-enabled workspaces, 97 require video.

Cross-artifact composition.The verified paths use one artifact modality for 276 tasks (67.3%), two for 115 (28.0%), and three for 19 (4.6%); thus, 134 tasks (32.7%) combine multiple modalities. At the carrier-family level in Figure3(b), 129 tasks (31.5%) span multiple carrier families, and 202 (49.3%) require a document or video.

5.3.Analytical Operations

Figure 4.Analytical-operation coverage and compositional breadth. Bars show task-level prevalence; the inset shows distinct operation types per task.Operation coverage.As shown in Figure4, projection and filtering are required by 338 (82.4%) and 321 tasks (78.3%), respectively. The benchmark also exercises ordering in 209 tasks (51.0%), aggregation in 146 (35.6%), and joins in 113 (27.6%). Cross-artifact grounding includes document and video understanding in 135 and 97 tasks, and schema and entity alignment in 97 and 90.

Compositional breadth.A task combines between two and eleven nontrivial operation types, with a median of five and a 90th percentile of seven; 248 tasks (60.5%) combine at least five types.

6.Experiments

6.1.Experimental Setup

We evaluate all 410 tasks under two complementary controls. For thebackbone comparison, we implementDataSpace-Agent, a lightweight agent that follows the ReAct(Yaoet al.,2023)paradigm and exposes a minimal set of task-agnostic tools. We fix this agent and vary only the multimodal backbone, testing six models released between April and July 2026: Grok 4.5(SpaceXAI,2026b), GPT-5.6 Sol(OpenAI,2026b), Kimi K3(Moonshot AI,2026), MiMo-V2.5(27), Claude Sonnet 5(Anthropic,2026b), and MiniMax M3(MiniMax,2026). In theharness comparison, we fix MiMo-V2.5 and compareDataSpace-Agent, Smolagents(Roucheret al.,2025), Codex(OpenAI,2026a), Claude Code(Anthropic,2026a), and Grok Build(SpaceXAI,2026a)while retaining each harness’s native planning, tool-use, and context management. Each backbone–harness pair instantiates a data agent. We omit specialized systems that cannot be evaluated faithfully under this protocol, such as DeepAnalyze(Zhanget al.,2025b), whose native interface does not cover video inputs or our tabular-output contract, and AgenticData(Sunet al.,2025), whose original implementation is unavailable. Model and harness configurations, together with the broader compatibility assessment, are provided in AppendixA.7.

Execution protocol.Each run receives the task question and its complete workspace. In the backbone comparison, every task is limited to 60 model turns, 50 tool actions, and 1,800 seconds, with 4 CPUs and 16 GiB of memory. The harness comparison uses the same 1,800-second deadline but does not impose a shared action limit that would override a harness’s native control loop.

Inference and scoring.All backbone calls use the provider-default reasoning configuration through Vercel AI Gateway(Vercel,2026), with a maximum of 32,768 output tokens per call. The official evaluator computes Task Accuracy using the protocol in Section4.5. Missing predictions, invalid outputs, runtime failures, and exhausted budgets count as incorrect. We additionally record token usage, API cost, tool actions, and wall-clock latency as efficiency diagnostics.

6.2.Overall Effectiveness

Table3reports Task Accuracy for both controlled comparisons.

Table 3.Overall effectiveness in controlled comparisons. Correct is out of 410; snapshot denotes backbone release month or harness version. Best in each block is bold.MethodSnapshotCorrectAcc. (%)Backbones (DataSpace-Agentfixed)Grok 4.5(SpaceXAI,2026b)2026-0727266.34GPT-5.6 Sol(OpenAI,2026b)2026-0726564.63Kimi K3(Moonshot AI,2026)2026-0721953.41MiMo-V2.5(27)2026-0416139.27Claude Sonnet 5(Anthropic,2026b)2026-0613532.93MiniMax M3(MiniMax,2026)2026-0611728.54Harnesses (MiMo-V2.5(27)fixed)Grok Build(SpaceXAI,2026a)v0.2.10619046.34Claude Code(Anthropic,2026a)v2.1.21718344.63DataSpace-Agent(ours)–16139.27Codex(OpenAI,2026a)v0.145.014334.88Smolagents(Roucheret al.,2025)v1.26.012730.98Backbone comparison.WithDataSpace-Agentfixed, Grok 4.5 achieves the highest observed accuracy of 66.34%, followed by GPT-5.6 Sol at 64.63%; their totals differ by only seven correct tasks. Kimi K3 reaches 53.41%, while the remaining three backbones remain below 40%. The 37.80-point range between the strongest and weakest backbone shows substantial separation, yet even the best result solves only about two thirds of the benchmark. Across the six models, 56 tasks are solved by all models, whereas 76 are solved by none; their oracle union solves 334 tasks (81.46%). Thus, the benchmark contains both a shared hard core and model-specific successes that are hidden by aggregate ranking alone.

Harness comparison.Fixing MiMo-V2.5, Grok Build obtains 46.34% and Claude Code 44.63%, compared with 39.27% forDataSpace-Agent, 34.88% for Codex, and 30.98% for Smolagents. The resulting 15.36-point spread demonstrates that the harness substantially affects end-to-end task completion even when the backbone is held constant.

Finding 1.DataSpaceremains unsaturated: the best controlled backbone reaches 66.34% Task Accuracy, while 76 tasks are missed by all six backbones. Harness design also substantially affects agent performance: with MiMo-V2.5 fixed, accuracy ranges from 30.98% to 46.34%.

6.3.Efficiency and Trade-offs

We fixDataSpace-Agentto compare backbone efficiency under a common agent design. Figure5relates Task Accuracy to mean token usage, API cost, tool actions, and wall-clock latency. GPT-5.6 Sol reaches 64.63% accuracy, only 1.71 points below Grok 4.5, while using 74.2% fewer tokens, 50.3% fewer actions, and 39.2% less wall-clock time per task. These two models form the token-, action-, and latency-based Pareto frontiers: GPT provides the most compact near-top solution, while Grok trades additional resources for the highest accuracy. The cost frontier differs: MiMo-V2.5 costs only $0.011 per task at 39.27% accuracy, whereas Grok reaches 66.34% at $0.169 and GPT costs $0.200 per task. Complete statistics, including medians and tail latency, appear in AppendixA.8.1.

Figure 5.Backbone accuracy–efficiency trade-offs withDataSpace-Agentfixed. Dashed lines connect Pareto-efficient points (higher accuracy, lower resource use).Finding 2.Near-top accuracy need not require long trajectories: GPT-5.6 Sol trails Grok 4.5 by 1.71 points while using 74.2% fewer tokens, 50.3% fewer actions, and 39.2% less latency. Monetary efficiency follows a different frontier, with MiMo-V2.5 providing the lowest-cost operating point at $0.011 per task.

6.4.Performance across Task Characteristics

We stratify the sixDataSpace-Agentruns by the annotations in Section5. Figure6reports Task Accuracy gaps across overlapping task characteristics; these are descriptive rather than causal. Sample sizes and raw accuracies appear in AppendixA.8.2.

Figure 6.Task Accuracy differences withDataSpace-Agentfixed. Cells report percentage-point changes from each row’s reference group; positive values favor the named row.Language and workspace scale.Cross-language performance varies by backbone: relative to single-language tasks, MiMo-V2.5 declines by 11.8 points and Grok 4.5 by 5.1, whereas GPT-5.6 Sol and Claude Sonnet 5 improve by 6.1 and 7.2 points. The largest workspace quartile underperforms the smallest for all six models, but accuracy across the four quartiles is not monotonic. At the task level, workspace bytes have only a weak negative Spearman correlation (−0.186-0.186) with the number of models that solve a task.

Evidence composition.Multimodal tasks underperform single-modal tasks for every backbone by 1.8–14.0 points. Modality presence alone has a less uniform effect: required document evidence is nearly neutral for GPT and Kimi but substantially reduces accuracy for Grok, MiMo, Claude, and MiniMax. Required video evidence helps GPT and Kimi but hurts MiMo, Claude, and MiniMax. These contrasts locate the consistent challenge in cross-modality integration rather than in any single modality.

Relational and answer requirements.Join requirements reduce accuracy for all backbones by 9.7–19.8 points, whereas aggregation has mixed effects. Answer shape is similarly non-monotonic: multi-row answers hurt Grok and MiMo but improve Claude, while multi-column and order-sensitive groups are not less accurate overall. Answer size and order therefore show no uniform negative association with accuracy; the analytical path and task composition remain more informative.

Finding 3.Cross-modality integration and joins are the most consistent sources of degradation: multimodal tasks underperform single-modal tasks for every backbone by 1.8–14.0 points, and joins reduce accuracy by 9.7–19.8 points. Language configuration and answer shape instead exhibit backbone-dependent effects.

6.5.Failure Analysis

We conduct a trace-level error analysis of 136 failures from Grok 4.5, the strongest controlled backbone.

Figure 7.Error analysis of 136 Grok 4.5 failures. Bars show primary causes; the matrix decomposes evaluator symptoms. Q/D/E/G/C/M/T denote intent, discovery, extraction, grounding, computation, materialization, and termination.Where failures originate.Answer materialization is the largest category, accounting for 71/136 (52.2%) failures, followed by task specification and intent with 31/136 (22.8%). At the subtype level, 60 materialization failures add or omit columns after the needed internal result is available, while 17 intent failures misformulate the requested output or row grain. These two distinct routes to an incorrect answer schema comprise 77/136 (56.6%) of audited failures. By comparison, only three failures arise from selecting the wrong evidence source; extraction and semantic grounding together account for 21, showing that locating an artifact does not ensure that its values are recovered and aligned correctly.

Symptoms are not diagnoses.Figure7shows that the same evaluator outcome can arise at different stages. Of 74 audited column-count mismatches, 58 originate in materialization, while the remainder begin in task intent, extraction, grounding, or computation. Conversely, only 5 of the 13 no-submission outcomes are pure execution-control failures; the other eight follow an earlier persistent interpretation, extraction, grounding, or computation error. Aggregate output symptoms therefore obscure the upstream intervention required to correct a trajectory. The complete subtype taxonomy and counts appear in AppendixA.8.3.

Finding 4.Harnesses must faithfully materialize the exact requested output, rather than merely find and correctly compute the relevant values: target-result misunderstanding and faulty column projection account for 56.6% of Grok 4.5’s audited failures, while only 5 of 13 no-submission outcomes are pure termination failures.

7.Conclusion

We introducedDataSpace, a 410-task benchmark for verifiable analytics with complete tabular outputs over cross-language, heterogeneous workspaces.DataSpace-Buildertransforms executable Text-to-SQL instances into multimodal tasks with expert review, paired with semantics-aware tabular-result evaluation. Across six backbones and five harnesses, the best accuracy reaches 66.34%, while multimodal evidence integration and joins remain key bottlenecks.DataSpaceestablishes a rigorous test bed for advancing reliable data agents over heterogeneous workspaces.

References

• Anthropic (2026a)How Claude Code works.Note:Accessed July 27, 2026External Links:LinkCited by:Table 9,§6.1,Table 3.
• Anthropic (2026b)Introducing Claude Sonnet 5.External Links:LinkCited by:Table 8,§6.1,Table 3.
• Y. Bian, X. Lin, Y. Xie, T. Liu, M. Zhuge, S. Lu, H. Tang, J. Wang, J. Zhang, J. Chen,et al.(2025)You don’t know until you click: automated gui testing for production-ready software evaluation.arXiv preprint arXiv:2508.14104.Cited by:§2.
• W. Chen, H. Zha, Z. Chen, W. Xiong, H. Wang, and W. Y. Wang (2020)HybridQA: A dataset of multi-hop question answering over tabular and textual data.InEMNLP (Findings),Findings of ACL, Vol.EMNLP 2020,pp. 1026–1036.Cited by:§2.
• Y. Chen, Y. Wu, S. Shen, Y. Xie, L. Shen, H. Xiong, and Y. Luo (2025)ChartMark: a structured grammar for chart annotation.In2025 IEEE Visualization and Visual Analytics (VIS),pp. 311–315.Cited by:§2.
• A. Egg, M. I. Goyanes, F. Kingma, A. Mora, L. von Werra, and T. Wolf (2025)Dabstep: data agent benchmark for multi-step reasoning.arXiv preprint arXiv:2506.23719.Cited by:Table 1,§1,§2.
• C. Fu, Y. Dai, Y. Luo, L. Li, S. Ren, R. Zhang, Z. Wang, C. Zhou, Y. Shen, M. Zhang, P. Chen, Y. Li, S. Lin, S. Zhao, K. Li, T. Xu, X. Zheng, E. Chen, C. Shan, R. He, and X. Sun (2025)Video-mme: the first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.InCVPR,pp. 24108–24118.Cited by:§2.
• P. Islam, A. Kannappan, D. Kiela, R. Qian, N. Scherrer, and B. Vidgen (2023)FinanceBench: A new benchmark for financial question answering.CoRRabs/2311.11944.Cited by:§2.
• KDD Cup 2026 (2026)KDD Cup 2026: data agents for complex data analysis.Note:Accessed July 27, 2026External Links:LinkCited by:Appendix B,§1.
• E. Lai, G. Vitagliano, Z. Zhang, S. Sudhir, O. Chabra, A. Zeng, A. A. Zabreyko, C. Li, F. Kossmann, J. Ding, J. Chen, M. Markakis, M. Russo, W. Wang, Z. Wu, M. J. Cafarella, L. Cao, S. Madden, and T. Kraska (2025)KramaBench: A benchmark for AI systems on data-to-insight pipelines over data lakes.CoRRabs/2506.06541.Cited by:Table 1,§1,§2.
• G. Lee, H. Hwang, S. Bae, Y. Kwon, W. Shin, S. Yang, M. Seo, J. Kim, and E. Choi (2023)EHRSQL: A practical text-to-sql benchmark for electronic health records.CoRRabs/2301.07695.Cited by:§1,§2,§4.1.
• F. Lei, J. Chen, Y. Ye, R. Cao, D. Shin, H. Su, Z. Suo, H. Gao, W. Hu, P. Yin, V. Zhong, C. Xiong, R. Sun, Q. Liu, S. Wang, and T. Yu (2025)Spider 2.0: evaluating language models on real-world enterprise text-to-sql workflows.InICLR,Cited by:Table 1,§1,§2.
• B. Li, C. Chen, Z. Xue, Y. Mei, and Y. Luo (2025a)DeepEye-sql: A software-engineering-inspired text-to-sql framework.CoRRabs/2510.17586.Cited by:§2.
• B. Li, O. O. K. Hei, Y. Yu, and Y. Luo (2026a)DPC: training-free text-to-sql candidate selection via dual-paradigm consistency.InACL (1),pp. 6897–6913.Cited by:§2.
• B. Li, Y. Luo, C. Chai, G. Li, and N. Tang (2024)The dawn of natural language to SQL: are we fully ready? [experiment, analysis & benchmark ].Proc. VLDB Endow.17(11),pp. 3318–3331.Cited by:§2.
• B. Li, Y. Peng, Y. Xie, S. Lu, Y. Zhu, X. Mu, X. Liu, and Y. Luo (2026b)DeepEye: a steerable self-driving data agent system.InCompanion of the International Conference on Management of Data,SIGMOD Companion ’26,New York, NY, USA,pp. 74–77.External Links:ISBN 9798400724503,Link,DocumentCited by:§1.
• B. Li, J. Zhang, J. Fan, Y. Xu, C. Chen, N. Tang, and Y. Luo (2025b)Alpha-sql: zero-shot text-to-sql using monte carlo tree search.InICML,Proceedings of Machine Learning Research, Vol.267.Cited by:§2.
• J. Li, B. Hui, G. Qu, J. Yang, B. Li, B. Li, B. Wang, B. Qin, R. Geng, N. Huo, X. Zhou, C. Ma, G. Li, K. C. Chang, F. Huang, R. Cheng, and Y. Li (2023)Can LLM already serve as A database interface? A big bench for large-scale database grounded text-to-sqls.InNeurIPS,Cited by:Table 1,§2.
• Y. Li, Z. Zhang, T. Ma, Z. Wang, K. Murugesan, C. Zhang, and Y. Ye (2026c)LongDA: benchmarking llm agents for long-document data analysis.arXiv preprint arXiv:2601.02598.Cited by:Table 1,§2.
• Z. Liang, X. Lin, Z. Zhang, Y. Luo, H. Wang, and N. Tang (2026)Long-document QA with chain-of-structured-thought and fine-tuned slms.CoRRabs/2603.29232.Cited by:§2.
• X. Liu, S. Shen, B. Li, P. Ma, R. Jiang, Y. Zhang, J. Fan, G. Li, N. Tang, and Y. Luo (2025a)A survey of text-to-sql in the era of llms: where are we, and where are we going?.IEEE Trans. Knowl. Data Eng.37(10),pp. 5735–5754.Cited by:§2.
• X. Liu, S. Shen, B. Li, N. Tang, and Y. Luo (2025b)NL2SQL-bugs: A benchmark for detecting semantic errors in NL2SQL translation.InKDD (2),pp. 5662–5673.Cited by:§2.
• T. Luo, C. Huang, L. Shen, B. Li, S. Shen, W. Zeng, N. Tang, and Y. Luo (2025)NvBench 2.0: resolving ambiguity in text-to-visualization through stepwise reasoning.InNeurIPS,Cited by:§1.
• P. Ma, B. Li, R. Jiang, J. Fan, N. Tang, and Y. Luo (2024a)A plug-and-play natural language rewriter for natural language to SQL.CoRRabs/2412.17068.Cited by:§2.
• R. Ma, S. Shankar, R. Chen, Y. Lin, S. Zeighami, R. Ghosh, A. Gupta, A. Gupta, T. Gopal, and A. G. Parameswaran (2026)Can ai agents answer your data questions? a benchmark for data agents.arXiv preprint arXiv:2603.20576.Cited by:§2.
• Y. Ma, Y. Zang, L. Chen, M. Chen, Y. Jiao, X. Li, X. Lu, Z. Liu, Y. Ma, X. Dong, P. Zhang, L. Pan, Y. Jiang, J. Wang, Y. Cao, and A. Sun (2024b)MMLONGBENCH-DOC: benchmarking long-context document understanding with visualizations.InNeurIPS,Cited by:Table 1,§1,§2.
• [27](2026)MiMo-v2.5.Note:https://huggingface.co/collections/XiaomiMiMo/mimo-v25Cited by:Table 8,§6.1,Table 3,Table 3.
• MiniMax (2026)MiniMax M3: frontier coding, 1m context, native multimodality—all in one model.External Links:LinkCited by:Table 8,§6.1,Table 3.
• Moonshot AI (2026)Kimi K3: open frontier intelligence.External Links:LinkCited by:Table 8,§6.1,Table 3.
• J. Nam, J. Yoon, J. Chen, J. Shin, S. Ö. Arik, and T. Pfister (2025)MLE-STAR: machine learning engineering agent via search and targeted refinement.CoRRabs/2506.15692.Cited by:Table 10.
• OpenAI (2026a)Codex CLI.Note:Accessed July 27, 2026External Links:LinkCited by:Table 9,§6.1,Table 3.
• OpenAI (2026b)GPT-5.6 Sol.Note:Accessed July 27, 2026External Links:LinkCited by:Table 8,§6.1,Table 3.
• P. Pasupat and P. Liang (2015)Compositional semantic parsing on semi-structured tables.InACL (1),pp. 1470–1480.Cited by:Table 1,§2.
• W. Pei, S. Hou, B. Li, C. Han, Z. Shi, and Y. Luo (2026)ROSE: an intent-centered evaluation metric for NL2SQL.InACL (1),pp. 5682–5709.Cited by:§2.
• R. Qi, Z. Liu, and W. Zhang (2026)DataCross: A unified benchmark and agent framework for cross-modal heterogeneous data analysis.CoRRabs/2601.21403.Cited by:Table 1,§1,§2.
• A. Roucher, A. V. del Moral, T. Wolf, L. von Werra, and E. Kaunismäki (2025)‘Smolagents‘: a smol library to build great agentic systems..Note:https://github.com/huggingface/smolagentsCited by:Table 9,§6.1,Table 3.
• Z. Shuai, B. Li, S. Yan, Y. Luo, and W. Yang (2026)DeepVIS: bridging natural language and data visualization through step-wise reasoning.IEEE Trans. Vis. Comput. Graph.32(1),pp. 868–878.Cited by:§1.
• SpaceXAI (2026a)Grok Build.Note:Accessed July 27, 2026External Links:LinkCited by:Table 9,§6.1,Table 3.
• SpaceXAI (2026b)Introducing Grok 4.5.External Links:LinkCited by:Table 8,§6.1,Table 3.
• J. Sun, G. Li, P. Zhou, Y. Ma, J. Xu, and Y. Li (2025)AgenticData: an agentic data analytics system for heterogeneous data.CoRRabs/2508.05002.Cited by:Table 10,§1,§6.1.
• Z. Sun, S. Zhong, D. Wen, J. Han, G. Li, Y. Yan, P. Zhang, Y. Su, X. Qi, B. Sun,et al.(2026)AgenticDataBench: a comprehensive benchmark for data agents.arXiv preprint arXiv:2607.01647.Cited by:§2.
• A. Talmor, O. Yoran, A. Catav, D. Lahav, Y. Wang, A. Asai, G. Ilharco, H. Hajishirzi, and J. Berant (2021)MultiModalQA: complex question answering over text, tables and images.InICLR,Cited by:§2.
• Y. Tang, X. Liu, B. Zhang, T. Lan, Y. Xie, J. Lao, Y. Wang, H. Li, T. Gao, B. Pan,et al.(2026a)IGenBench: benchmarking the reliability of text-to-infographic generation.arXiv preprint arXiv:2601.04498.Cited by:§2.
• Y. Tang, Y. Xie, Y. Feng, T. Lan, J. Lao, Y. Cheng, and W. Chen (2026b)ViviDoc: generating interactive documents through human-agent collaboration.arXiv preprint arXiv:2603.27991.Cited by:§1.
• Teable (2026)Teable AI: overview.Note:Accessed July 27, 2026External Links:LinkCited by:Table 10,§1.
• Vercel (2026)AI Gateway.Note:Accessed July 27, 2026External Links:LinkCited by:§6.1.
• J. Wang and G. Li (2025)AOP: automated and interactive LLM pipeline orchestration for answering complex queries.InCIDR,Cited by:Table 10,§1.
• Z. Wang, S. Zhang, H. Yuan, J. Zhu, S. Li, W. Dong, and G. Cong (2025)FDABench: A benchmark for data agents on analytical queries over heterogeneous data.CoRRabs/2509.02473.Cited by:§A.7.3,Table 1,§1,§1,§2.
• Y. Xie, Y. Luo, G. Li, and N. Tang (2024)Haichart: human and ai paired visualization system.arXiv preprint arXiv:2406.11033.Cited by:§1.
• Y. Xie, C. Ma, Z. Wang, L. Wang, J. Zhu, C. Zeng, Z. Shen, B. Li, and Y. Luo (2026)DataMagic: transforming tabular data into data insight video.CoRRabs/2606.20388.Cited by:§4.3.
• Y. Xie, Z. Zhang, Y. Wu, S. Lu, J. Zhang, Z. Yu, J. Wang, S. Hong, B. Liu, C. Wu,et al.(2025)Visjudge-bench: aesthetics and quality assessment of visualizations.arXiv preprint arXiv:2510.22373.Cited by:§2.
• X. Yang, K. Sun, H. Xin, Y. Sun, N. Bhalla, X. Chen, S. Choudhary, R. D. Gui, Z. W. Jiang, Z. Jiang, L. Kong, B. Moran, J. Wang, Y. E. Xu, A. Yan, C. Yang, E. Yuan, H. Zha, N. Tang, L. Chen, N. Scheffer, Y. Liu, N. Shah, R. Wanga, A. Kumar, W. Yih, and X. L. Dong (2024)CRAG - comprehensive RAG benchmark.CoRRabs/2406.04744.Cited by:Table 1,§2.
• Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning (2018)HotpotQA: A dataset for diverse, explainable multi-hop question answering.InEMNLP,pp. 2369–2380.Cited by:Table 1,§2.
• S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao (2023)ReAct: synergizing reasoning and acting in language models.InICLR,Cited by:§6.1.
• T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman, Z. Zhang, and D. R. Radev (2018)Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task.InEMNLP,pp. 3911–3921.Cited by:Table 1,§1,§2.
• C. Zhang, Y. Mao, Y. Fan, Y. Mi, Y. Gao, L. Chen, D. Lou, and J. Lin (2024)FinSQL: model-agnostic llms-based text-to-sql framework for financial analysis.InSIGMOD Conference Companion,pp. 93–105.Cited by:§1,§2,§4.1.
• C. Zhang, S. Zhang, Q. Liu, S. Chen, T. Li, and J. Fan (2025a)TAIJI: mcp-based multi-modal data analytics on data lakes.CoRRabs/2505.11270.Cited by:Table 10,§1.
• S. Zhang, J. Fan, M. Fan, G. Li, and X. Du (2025b)DeepAnalyze: agentic large language models for autonomous data science.External Links:2510.16872,LinkCited by:Table 10,§1,§6.1.
• Z. Zhang, Z. Liang, Y. Wu, T. Lin, Y. Luo, and N. Tang (2025c)DataMosaic: explainable and verifiable multi-modal data analytics through extract-reason-verify.CoRRabs/2504.10036.Cited by:§2.
• Y. Zhu, R. Jiang, B. Li, N. Tang, and Y. Luo (2025a)EllieSQL: cost-efficient text-to-sql with complexity-aware routing.CoRRabs/2503.22402.Cited by:§2.
• Y. Zhu, Z. Peng, B. Li, and Y. Luo (2026)SQLConductor: search-to-policy learning for step-wise text-to-sql orchestration.CoRRabs/2606.23537.Cited by:§2.
• Y. Zhu, L. Wang, C. Yang, X. Lin, B. Li, W. Zhou, X. Liu, Z. Peng, T. Luo, Y. Li, C. Chai, C. Chen, S. Di, J. Fan, J. Sun, N. Tang, F. Tsung, J. Wang, C. Wu, Y. Xu, S. Zhang, Y. Zhang, X. Zhou, G. Li, and Y. Luo (2025b)A survey of data agents: emerging paradigm or overstated hype?.CoRRabs/2510.23587.Cited by:§1.

Appendix AAdditional Benchmark Details

A.1.Cross-Language Transformation Details

This appendix expands the cross-language transformation in Section4.1. We describe the construction contract, algorithmic steps, intermediate artifacts, and normalized prompt templates. Provider-specific system wrappers and batching delimiters are omitted because they do not change the semantic contract. All generative calls use deterministic decoding, and their outputs must pass the stated schema and consistency checks before they can modify a database or workload.

A.1.1.Transformation Contract and Invariants

The input is an executable Text-to-SQL tupleT0=(𝒟0,q0,σ0)T_{0}=(\mathcal{D}_{0},q_{0},\sigma_{0}), together with independently selected database and question languages(ℓD,ℓq)(\ell_{D},\ell_{q}). The output isTc=(𝒟c,qc,σc)T_{c}=(\mathcal{D}_{c},q_{c},\sigma_{c})plus a materialized replacement mapMM. The transformation enforces four invariants:

1. (1)Referential consistency.Repeated representations of the same entity receive the same translation across primary keys, foreign keys, denormalized columns, and undeclared join paths.
2. (2)Workload consistency.Every schema identifier and value literal used byσc\sigma_{c}resolves against𝒟c\mathcal{D}_{c}.
3. (3)Execution alignment.Executingσc\sigma_{c}on𝒟c\mathcal{D}_{c}reproduces the translated counterpart of the source result, including arity and row multiplicity.
4. (4)Question fidelity.The transformed question preserves the requested entities, predicates, aggregation, ordering, cardinality, units, and temporal scope ofq0q_{0}.

The mapMM, rather than an unconstrained model response, is the interface between language generation and physical rewriting. Consequently, model calls can be rerun or manually repaired without changing the deterministic migration logic.

A.1.2.Translation-Unit Extraction

Let each text-bearing column be a vertex. We first add an explicit edge between two columns when they participate in a declared foreign-key relationship or share the same normalized name. Connected components of these edges form the initial clusters𝒞exp\mathcal{C}_{\mathrm{exp}}. To recover undeclared joins, we collect the distinct non-null valuesV​(C)V(C)of every cluster and compute the overlap coefficient

(10)ρ​(Cx,Cy)=|V​(Cx)∩V​(Cy)|min⁡(|V​(Cx)|,|V​(Cy)|).\rho(C_{x},C_{y})=\frac{|V(C_{x})\cap V(C_{y})|}{\min(|V(C_{x})|,|V(C_{y})|)}.Clusters are merged whenρ​(Cx,Cy)>θov\rho(C_{x},C_{y})>\theta_{\mathrm{ov}}and the intersection contains at leastkmink_{\min}distinct values. The second condition prevents a single common token from connecting otherwise unrelated columns. Both parameters are exposed in the construction configuration.

We then remove values that should not be localized. The filters cover nulls, pure numbers, dates and timestamps, URLs, e-mail addresses, file paths, machine-generated identifiers, standardized codes, and values already written in the target language. Domain terms that must remain invariant can be added to a protected glossary. Remaining values are deduplicated within a cluster and divided into bounded prompt chunks. Every item retains its cluster ID and all(table,column)(\text{table},\text{column})occurrences, so a single translated value can later be fanned out consistently.

Extraction algorithm.

The algorithm below separates clustering from prompt-size management; changing the chunk budgetBBtherefore does not change entity equivalence classes.

Algorithm 1Translation-unit extraction1:Schema

SS, database

DD, target language

ℓD\ell_{D} 2:Thresholds

θov,kmin\theta_{\mathrm{ov}},k_{\min} 3:Prompt budget

BB, protected glossary

GG 4:Clusters

𝒞\mathcal{C}, schema units

UsU_{s}, value chunks

UvU_{v} 5:

VT←TextColumns​(S)V_{T}\leftarrow\textsc{TextColumns}(S) 6:

𝒞←Singletons​(VT)\mathcal{C}\leftarrow\textsc{Singletons}(V_{T}) 7:

UnionFKEndpoints​(𝒞,S)\textsc{UnionFKEndpoints}(\mathcal{C},S) 8:

UnionSameNameColumns​(𝒞,S)\textsc{UnionSameNameColumns}(\mathcal{C},S) 9:repeat

10:

c​h​a​n​g​e​d←falsechanged\leftarrow\mathrm{false} 11:for allunordered

(Cx,Cy)∈𝒞(C_{x},C_{y})\in\mathcal{C}do

12:

I←V​(Cx)∩V​(Cy)I\leftarrow V(C_{x})\cap V(C_{y}) 13:

ρ←|I|/min⁡(|V​(Cx)|,|V​(Cy)|)\rho\leftarrow|I|/\min(|V(C_{x})|,|V(C_{y})|) 14:if

ρ>θov∧|I|≥kmin\rho>\theta_{\mathrm{ov}}\land|I|\geq k_{\min}then

15:

Union​(Cx,Cy)\textsc{Union}(C_{x},C_{y});

c​h​a​n​g​e​d←truechanged\leftarrow\mathrm{true} 16:endif

17:endfor

18:until

c​h​a​n​g​e​d=falsechanged=\mathrm{false} 19:

Us←ExtractSchemaUnits​(S,G)U_{s}\leftarrow\textsc{ExtractSchemaUnits}(S,G) 20:for all

Cj∈𝒞C_{j}\in\mathcal{C}do

21:

Vj←DistinctNonNullValues​(D,Cj)V_{j}\leftarrow\textsc{DistinctNonNullValues}(D,C_{j}) 22:

Vj←FilterProtected​(Vj,ℓD,G)V_{j}\leftarrow\textsc{FilterProtected}(V_{j},\ell_{D},G) 23:

Vj←DedupWithProvenance​(Vj)V_{j}\leftarrow\textsc{DedupWithProvenance}(V_{j}) 24:

Uv←Uv∪Chunk​(Vj,B)U_{v}\leftarrow U_{v}\cup\textsc{Chunk}(V_{j},B) 25:endfor

26:return

𝒞,Us,Uv\mathcal{C},U_{s},U_{v}

A.1.3.Replacement-Map Generation

The materialized map contains three scoped mappings:

(11)M={Mtab,Mcol,Mval},M=\{M_{\mathrm{tab}},M_{\mathrm{col}},M_{\mathrm{val}}\},where table keys are database-scoped, column keys are table-scoped, and value keys retain their column-cluster provenance. Figure8shows a normalized serialized form.

{

“tables”:[

{“source”:“...”,“target”:“...”}

],

“columns”:[

{“table_source”:“...”,“source”:“...”,“target”:“...”}

],

“values”:[

{

“cluster_id”:“...”,

“source”:“...”,

“target”:“...”,

“occurrences”:[

{“table_source”:“...”,“column_source”:“...”}

]

}

]

}

Figure 8.Normalized serialized form of the replacement map.All responses are parsed as JSON and checked before map assembly. Each input ID must occur exactly once, protected items must be unchanged, target table names must be unique within a database, and target column names must be unique within a table. A value shared by a cluster receives one target form, which is copied to every listed occurrence. Invalid or incomplete responses are returned to the model together with validator errors under a bounded retry policy; a remaining conflict is repaired manually or the affected sample is rejected. Figures9and10specify the normalized contracts for the two mapping stages.

SYSTEM

Youlocalizerelationalschemasfrom<SOURCE_LANGUAGE>to

<TARGET_LANGUAGE>.Produceafaithfulterminologymap,notanewschema.

RULES

1.Translateonlyhuman-readabletableandcolumnnames.

2.PreserveIDs,standardizedcodes,SQLkeywords,andprotectedterms.

3.Preservemeaning,domainterminology,granularity,andabbreviations.

4.Tabletargetsmustbeuniqueinthedatabase.Columntargetsmustbe

uniquewithintheirtable.

5.ReturnJSONonly.Donotadd,remove,merge,orsplitinputitems.

INPUT

DOMAIN:<DATABASE_DOMAIN>

PROTECTED_GLOSSARY:<PROTECTED_GLOSSARY>

SCHEMA_ITEMS:

<LIST_OF_ITEMS_WITH_STABLE_IDS_TABLE_CONTEXT_AND_DESCRIPTIONS>

OUTPUTSCHEMA

{

“items”:[

{“id”:“<INPUT_ID>”,“target”:“<TRANSLATION>”}

]

}

Figure 9.Prompt for schema mapping.SYSTEM

Translatedatabasevaluesfrom<SOURCE_LANGUAGE>to<TARGET_LANGUAGE>

whilepreservingequalityandjoinsemantics.

RULES

1.Onesourcevaluehasexactlyonetargetvaluewithinthiscluster.

2.Usethesametargetforeverylistedtable/columnoccurrence.

3.Donottranslateidentifiers,codes,URLs,dates,numbers,orentries

markedPROTECTED.

4.Preserveunits,signs,precision,entityidentity,anddomainmeaning.

5.ReturnJSONonlyandincludeeveryinputIDexactlyonce.

CONTEXT

DOMAIN:<DATABASE_DOMAIN>

CLUSTER_ID:<CLUSTER_ID>

COLUMNS:<TABLE_COLUMN_OCCURRENCES>

SCHEMA_GLOSSARY:<RELEVANT_SCHEMA_MAP>

VALUES:<VALUES_WITH_STABLE_IDS_AND_PROTECTION_FLAGS>

OUTPUTSCHEMA

{

“cluster_id”:“<CLUSTER_ID>”,

“items”:[

{“id”:“<INPUT_ID>”,“target”:“<TRANSLATION_OR_ORIGINAL>”}

]

}

Figure 10.Prompt for clustered-value mapping.

A.1.4.Deterministic Database Migration

The source database is copied before modification, and all changes are applied inside a transaction. Rewrite order is important because map keys are expressed in source-language identifiers. Cell values are updated first; columns are renamed while source table names still exist; tables are renamed last. Dependent views, indexes, triggers, foreign-key declarations, and schema metadata are then rewritten against the target names. CHECK constraints that contain translated enumerated values are temporarily removed and reconstructed with their mapped literals. Any collision, unresolved reference, or constraint failure aborts the transaction.

Algorithm 2Deterministic database migration1:Source database

𝒟0\mathcal{D}_{0}, validated map

MM 2:Translated database

𝒟c\mathcal{D}_{c}, synchronized metadata

ScS_{c} 3:

𝒟c←TransactionalCopy​(𝒟0)\mathcal{D}_{c}\leftarrow\textsc{TransactionalCopy}(\mathcal{D}_{0}) 4:

K←ExtractDefinitions​(𝒟c)K\leftarrow\textsc{ExtractDefinitions}(\mathcal{D}_{c}) 5:

DeferAffectedChecks​(𝒟c,K,M)\textsc{DeferAffectedChecks}(\mathcal{D}_{c},K,M) 6:for all

(t,c,v↦v′)∈Mval(t,c,v\mapsto v^{\prime})\in M_{\mathrm{val}}do

7:

UpdateExactValue​(𝒟c,t,c,v,v′)\textsc{UpdateExactValue}(\mathcal{D}_{c},t,c,v,v^{\prime}) 8:endfor

9:for all

(t,c↦c′)∈Mcol(t,c\mapsto c^{\prime})\in M_{\mathrm{col}}do

10:

RenameColumn​(𝒟c,t,c,c′)\textsc{RenameColumn}(\mathcal{D}_{c},t,c,c^{\prime}) 11:endfor

12:for all

(t↦t′)∈Mtab(t\mapsto t^{\prime})\in M_{\mathrm{tab}}do

13:

RenameTable​(𝒟c,t,t′)\textsc{RenameTable}(\mathcal{D}_{c},t,t^{\prime}) 14:endfor

15:

Kc←RewriteDefinitions​(K,M)K_{c}\leftarrow\textsc{RewriteDefinitions}(K,M) 16:

RestoreDefinitions​(𝒟c,Kc)\textsc{RestoreDefinitions}(\mathcal{D}_{c},K_{c}) 17:

Sc←SyncMetadata​(𝒟c,M)S_{c}\leftarrow\textsc{SyncMetadata}(\mathcal{D}_{c},M) 18:

AssertNoDrift​(𝒟c,Sc,M)\textsc{AssertNoDrift}(\mathcal{D}_{c},S_{c},M) 19:

Commit​(𝒟c)\textsc{Commit}(\mathcal{D}_{c}) 20:return

𝒟c,Sc\mathcal{D}_{c},S_{c}

Updates are scoped by the source table and column recorded inMM; global string replacement is never applied to database contents. When a source value is a substring of another value, exact matching is used for cells and longest-first matching is used only inside parsed schema definitions.

A.1.5.Protect–Replace–Restore SQL Rewriting

SQL rewriting is a workload-migration operation overσ0\sigma_{0}, not a new query-generation call. We parse nested query blocks and build a scope table for base tables, CTEs, aliases, and projected columns. String literals, CTE names, alias declarations, and table-name positions are replaced with typed markers. Schema identifiers are then mapped according to their resolved scope. A string literal is translated only when its comparison context resolves to a column cluster containing the corresponding value entry. This prevents a surface form that appears in two unrelated columns from receiving the wrong replacement. Subqueries are rewritten from the innermost scope outward, after which markers are restored and the resulting SQL is parsed again.

Algorithm 3Protect–replace–restore SQL rewriting1:SQL

σ0\sigma_{0}, replacement map

MM 2:Source schema

S0S_{0}, target schema

ScS_{c} 3:Rewritten SQL

σc\sigma_{c} 4:

(a​s​t,s​c​o​p​e​s)←ParseResolve​(σ0,S0)(ast,scopes)\leftarrow\textsc{ParseResolve}(\sigma_{0},S_{0}) 5:

(p​r​o​t​e​c​t​e​d,m​a​r​k​e​r​s)←Protect​(a​s​t)(protected,markers)\leftarrow\textsc{Protect}(ast) 6:

o​r​d​e​r​e​d←InnermostToOutermost​(s​c​o​p​e​s)ordered\leftarrow\textsc{InnermostToOutermost}(scopes) 7:for all

s​c​o​p​e∈o​r​d​e​r​e​dscope\in ordereddo

8:

ReplaceTables​(s​c​o​p​e,Mtab)\textsc{ReplaceTables}(scope,M_{\mathrm{tab}}) 9:

ReplaceColumns​(s​c​o​p​e,Mcol)\textsc{ReplaceColumns}(scope,M_{\mathrm{col}}) 10:for allliteral

LLwith resolved context

(t,c)(t,c)do

11:if

(t,c,L)∈Mval(t,c,L)\in M_{\mathrm{val}}then

12:

m​a​r​k​e​r​s​[L]←Mval​[(t,c,L)]markers[L]\leftarrow M_{\mathrm{val}}[(t,c,L)] 13:endif

14:endfor

15:endfor

16:

σc←RestoreSerialize​(p​r​o​t​e​c​t​e​d,m​a​r​k​e​r​s)\sigma_{c}\leftarrow\textsc{RestoreSerialize}(protected,markers) 17:

AssertParseable​(σc)\textsc{AssertParseable}(\sigma_{c}) 18:

AssertResolved​(σc,Sc)\textsc{AssertResolved}(\sigma_{c},S_{c}) 19:return

σc\sigma_{c}

The protection layer also prevents accidental substitutions inside SQL keywords, function names, numeric constants, and partial identifiers. When an unqualified column is ambiguous under the current scope, the rewrite is not guessed; it is marked for repair.

A.1.6.Question Translation and Language Composition

Question language is varied independently of database language. The question translator receives the original question, a focused glossary containing only entities relevant to that sample, and optional SQL alignment context. The SQL is construction-time context: it constrains preservation of operators and conditions but is not copied into the natural-language output. No database or SQL artifact is modified during this step. Verified question variants are joined to verified database/SQL variants by stable sample ID, which allows(ℓq,ℓD)(\ell_{q},\ell_{D})combinations to be assembled without repeating database migration. The normalized translation contract is given in Figure11.

SYSTEM

Translateananalyticalquestionfrom<SOURCE_LANGUAGE>to

<TARGET_LANGUAGE>.Preservetheexactanswersemantics.

PRESERVE

-entities,valueliterals,units,signs,andnumericalthresholds;

-filters,negation,conjunction,andcomparisondirection;

-aggregation,grouping,distinctness,ranking,ordering,andlimits;

-timewindows,inclusivityofboundaries,andrequestedoutputfields.

RULES

1.Usethesuppliedtarget-languageglossaryforschemaandvalueterms.

2.Writeanaturaluserquestion;donotmentionSQL,schemas,orthistask.

3.Donotaddexplanations,assumptions,oranswervalues.

4.ReturnJSONonly.

INPUT

SOURCE_QUESTION:<QUESTION>

FOCUSED_GLOSSARY:<RELEVANT_TABLE_COLUMN_VALUE_MAP>

OPTIONAL_SQL_ALIGNMENT_CONTEXT:<GOLD_SQL_OR_OPERATOR_SIGNATURE>

OUTPUTSCHEMA

{

“question”:“<TRANSLATED_QUESTION>”,

“used_mapping_ids”:[“<ID>”]

}

Figure 11.Prompt for question translation.

A.1.7.Validation Gates and Failure Handling

Validation proceeds from inexpensive structural checks to semantic checks:

1. (1)Map validationchecks coverage, scope, protected identities, and table/column target-name collisions.
2. (2)Database validationchecks that mapped entities exist, dependent definitions resolve, and integrity constraints can be applied.
3. (3)Workload validationparsesσc\sigma_{c}, resolves every identifier, executes it on𝒟c\mathcal{D}_{c}, and compares the result withτM​(Exec⁡(𝒟0,σ0))\tau_{M}(\operatorname{Exec}(\mathcal{D}_{0},\sigma_{0})). Comparison preserves duplicate multiplicity; row order is enforced when specified by the query.
4. (4)Question validationuses an LLM judge to check thatqcq_{c}remains answerable byσc\sigma_{c}and preserves its analytical intent.

Figure12instantiates the final semantic audit.

SYSTEM

Auditwhetheratranslatedanalyticalquestionpreservesthesourceintent

andremainsansweredbythetranslatedSQL.Donotsolvethequery.

CHECK

1.Requestedoutputentitiesandfields.

2.Allfilters,values,comparisondirections,andnegations.

3.Aggregation,grouping,DISTINCTsemantics,ordering,ranking,andlimits.

4.Timewindows,boundaryinclusivity,units,andnumericalscale.

5.Consistencywiththesuppliedschema/valueglossary.

INPUT

SOURCE_QUESTION:<SOURCE_QUESTION>

TRANSLATED_QUESTION:<TRANSLATED_QUESTION>

SOURCE_SQL:<SOURCE_SQL>

TRANSLATED_SQL:<TRANSLATED_SQL>

FOCUSED_GLOSSARY:<RELEVANT_MAP>

OUTPUTSCHEMA

{

“status”:“pass|repair|reject”,

“issues”:[

{“type”:“<ISSUE_TYPE>”,“description”:“<DESCRIPTION>”}

],

“revised_question”:“<ONLY_IF_REPAIR>”

}

Figure 12.Prompt for question–SQL alignment.A tuple advances only when all gates pass. A failed map is regenerated or edited before any rewrite. A failed SQL is repaired against the same map and re-executed. A repaired question is subjected to the alignment prompt again. Cases that cannot be made execution-aligned and semantically faithful are discarded. The accepted stage artifact contains(𝒟c,qc,σc)(\mathcal{D}_{c},q_{c},\sigma_{c}), the replacement map, synchronized metadata, and validation status; the final question–workspace answerability check is performed later by the expert review process in Section4.4.

A.2.Constraint-Aware Relational Sampling Details

This appendix expands the relational sampling stage in Section4.2. The procedure constructs a new task-local database instance rather than approximating the answer obtained from the source database. Accordingly, the source and sampled answers may differ in their entities, multiplicities, aggregate values, and ordering. Executability and task validity are checked on the sampled instance itself.

A.2.1.Sampling Contract

The input is the verified cross-language tupleTc=(𝒟c,qc,σc)T_{c}=(\mathcal{D}_{c},q_{c},\sigma_{c}). Let𝒯c\mathcal{T}_{c}be the complete table inventory of𝒟c\mathcal{D}_{c}, and let𝒟c​[R]\mathcal{D}_{c}[R]denote the rows of tableRR. A sampling policy

(12)ψ=(z,{bR:R∈𝒯c},K,Ra,ℰ+)\psi=\bigl(z,\{b_{R}:R\in\mathcal{T}_{c}\},K,R_{a},\mathcal{E}^{+}\bigr)contains a base random seedzz, a soft row budgetbRb_{R}for each table, a maximum number of attemptsKK, an optional anchor tableRaR_{a}, and schema relationshipsℰ+\mathcal{E}^{+}not declared in the source database. The latter covers, for example, known key pairs in databases with incomplete foreign-key metadata. Budgets bound ordinary random additions; protected rows and rows introduced by relational closure take precedence and may exceed them. We useK=3K=3for the benchmark construction.

Every sampled instance preserves the complete table and column inventory:

(13)Tables⁡(𝒟s)=𝒯c,Schema⁡(𝒟s​[R])=Schema⁡(𝒟c​[R]).\operatorname{Tables}(\mathcal{D}_{s})=\mathcal{T}_{c},\qquad\operatorname{Schema}(\mathcal{D}_{s}[R])=\operatorname{Schema}(\mathcal{D}_{c}[R]).Thus, sampling changes table contents but does not use the source SQL to remove tables or columns. Empty source tables remain valid empty tables with their schema intact.

A.2.2.Safeguard Extraction

We form the safeguard set𝒞s=𝒞schema∪𝒞query\mathcal{C}_{s}=\mathcal{C}_{\mathrm{schema}}\cup\mathcal{C}_{\mathrm{query}}. The schema component is represented as a directed relationship graphGs=(𝒯c,ℰs)G_{s}=(\mathcal{T}_{c},\mathcal{E}_{s}). Each edge records a child table and column, the corresponding parent table and column, and whether the relationship is declared or supplied byℰ+\mathcal{E}^{+}. Composite keys are retained as tuples rather than decomposed into independent column constraints.

The query component is extracted from a resolved SQL AST. Name resolution is performed separately inside each query block so that aliases, common-table expressions, and correlated subqueries do not create spurious bindings. We record four types of safeguard:

1. (1)Predicate bindingsassociate a resolved column with a literal or literal set used by equality, membership, range, or pattern predicates. Literal types are preserved during matching.
2. (2)Query relationshipsrecord the resolved column pairs in explicit and implicit joins, including multi-column join keys.
3. (3)Boundary bindingsretain values that define temporal or numerical intervals.
4. (4)Target bindingsidentify explicitly named entities whose disappearance would change the referent of the question.

Unsupported expressions do not trigger string-based guessing. They are retained in the AST record and left to execution validation; samples for which the protected values or join endpoints cannot be resolved are marked for repair before sampling.

A.2.3.Anchor Rows and Soft Budgets

For each tableRR, the sampler first constructs an anchor setARA_{R}. Rows matching protected equality, membership, or target bindings are inserted directly. For range and pattern predicates, matching rows are selected using the same typed operator as the SQL expression. When a predicate is attached to a nested query, its anchors remain scoped to the base relation resolved in that query block. WhenRaR_{a}is configured, its selected rows provide the starting keys for relationship propagation. Anchor selection therefore operates on database values, not surface-form occurrences in serialized rows.

The remaining capacity is filled by sampling within each table. For a tableRRwhose anchors do not exhaust its budgetbRb_{R}, the initial row set is

(14)SR(0)=AR∪Sample⁡(𝒟c​[R]∖AR,bR−|AR|).S_{R}^{(0)}=A_{R}\cup\operatorname{Sample}(\mathcal{D}_{c}[R]\setminus A_{R},\,b_{R}-|A_{R}|).Here,Sample⁡(X,n)\operatorname{Sample}(X,n)returns up tonnrows fromXX, so small tables are retained in full. If the anchors already meet or exceed the budget, no additional rows are sampled. Tables not referenced byσc\sigma_{c}use the same rule with an empty anchor set.

A.2.4.Relational Closure

Independent table samples can contain a selected foreign key without its referenced row or leave a protected query join without any matched pair. We therefore augment the initial row sets by relational closure. For every selected child row, its non-null referenced key introduces the matching parent row. For query relationships, the sampler additionally retains matched rows along the protected join path from the current anchors. Newly introduced rows are placed back on the closure queue until no relationship adds a row. Since closure only adds rows drawn from the finite source instance, the procedure terminates even when the schema graph contains cycles.

One-to-many expansion is bounded for ordinary random rows. Anchor-derived and query-path matches have priority; optional matches are drawn with the relationship-specific random stream until the relevant soft budget is reached. Missing referenced keys, type-incompatible join columns, and an empty protected join are recorded as validation errors rather than repaired by fabricating records.

Algorithm4summarizes the complete procedure.

Algorithm 4Constraint-aware relational sampling1:Verified tuple

(𝒟c,qc,σc)(\mathcal{D}_{c},q_{c},\sigma_{c}), policy

ψ\psi 2:Sampled database

𝒟s\mathcal{D}_{s}, reference result

𝒴s\mathcal{Y}_{s}, ledger

LsL_{s} 3:

𝒞schema←SchemaRelations​(𝒟c,ℰ+)\mathcal{C}_{\mathrm{schema}}\leftarrow\textsc{SchemaRelations}(\mathcal{D}_{c},\mathcal{E}^{+}) 4:

𝒞query←QuerySafeguards​(σc,𝒟c)\mathcal{C}_{\mathrm{query}}\leftarrow\textsc{QuerySafeguards}(\sigma_{c},\mathcal{D}_{c}) 5:

𝒞s←𝒞schema∪𝒞query\mathcal{C}_{s}\leftarrow\mathcal{C}_{\mathrm{schema}}\cup\mathcal{C}_{\mathrm{query}} 6:forattempt

k=1,…,Kk=1,\ldots,Kdo

7:

A←AnchorRows​(𝒟c,𝒞query,Ra,z,k)A\leftarrow\textsc{AnchorRows}(\mathcal{D}_{c},\mathcal{C}_{\mathrm{query}},R_{a},z,k) 8:

S←SeededTableSamples​(𝒟c,A,ψ,k)S\leftarrow\textsc{SeededTableSamples}(\mathcal{D}_{c},A,\psi,k) 9:

S←RelationalClosure​(S,𝒞s)S\leftarrow\textsc{RelationalClosure}(S,\mathcal{C}_{s}) 10:

𝒟s←MaterializeAllSchemas​(𝒟c,S)\mathcal{D}_{s}\leftarrow\textsc{MaterializeAllSchemas}(\mathcal{D}_{c},S) 11:

vs←ValidateStructure​(𝒟s,𝒞schema)v_{s}\leftarrow\textsc{ValidateStructure}(\mathcal{D}_{s},\mathcal{C}_{\mathrm{schema}}) 12:if

vs=passv_{s}=\mathrm{pass}then

13:

(es,𝒴s)←Execute​(𝒟s,σc)(e_{s},\mathcal{Y}_{s})\leftarrow\textsc{Execute}(\mathcal{D}_{s},\sigma_{c}) 14:

vq←ValidateTask​(es,𝒴s,σc,𝒞query)v_{q}\leftarrow\textsc{ValidateTask}(e_{s},\mathcal{Y}_{s},\sigma_{c},\mathcal{C}_{\mathrm{query}}) 15:if

vq=passv_{q}=\mathrm{pass}then

16:

Ls←BuildLedger​(𝒟s,𝒴s,ψ,𝒞s,k)L_{s}\leftarrow\textsc{BuildLedger}(\mathcal{D}_{s},\mathcal{Y}_{s},\psi,\mathcal{C}_{s},k) 17:return

𝒟s,𝒴s,Ls\mathcal{D}_{s},\mathcal{Y}_{s},L_{s} 18:endif

19:endif

20:endfor

21:return

reject\mathrm{reject}

A.2.5.Materialization and Validation

Materialization creates a fresh SQLite database, recreates every source table with its column types and key declarations, inserts the selected rows, and rebuilds applicable indexes and views. Inserts occur inside a transaction with foreign-key checking enabled at validation time. The resulting database is then checked in two stages.

Structural validation.

We verify table and column inventory, declared primary-key uniqueness, foreign-key consistency, row serializability, and the presence of protected bindings and query-path matches. We also compare per-table row counts against the recorded selected-row sets; this catches silent insertion loss caused by type conversion or duplicate handling.

Execution validation.

The transformed SQLσc\sigma_{c}is parsed and executed on𝒟s\mathcal{D}_{s}. Successful execution defines the candidate answer𝒴s=Exec⁡(𝒟s,σc)\mathcal{Y}_{s}=\operatorname{Exec}(\mathcal{D}_{s},\sigma_{c}); no equality test against the source answer is applied. We check the returned arity against the resolved projection and record query-signature diagnostics for conditions such as a broken join, a null-only aggregate, or fewer candidates than a requested ranking. These conditions are not universal rejection rules: an empty or null-valued answer can be semantically correct. An unexpected empty result caused by the loss of protected predicates or join matches fails the attempt; ambiguous cases are forwarded to the expert review stage.

A.2.6.Retry, Repair, and Provenance

A failed attempt is retried with a deterministically derived seed. Failures caused by insufficient optional coverage are handled by resampling; failures caused by unresolved relationships, missing protected values, or unsuitable budgets require configuration repair before another attempt. A task is discarded when it exhausts the configured attempt budget or cannot produce an executable and meaningful relational instance.

For every accepted task, the internal sampling ledger stores the source task and database identifiers, SQL hash, policy and attempt seed, per-table soft budgets, extracted safeguards, added relationship pairs, row counts before and after sampling, retry history, intermediate-database hash, execution status, and candidate-answer hash. The ledger is used to reproduce construction and audit later repairs; it is not included in the agent-visible workspace.

A.3.Modality Routing & Artifact Rendering Details

This appendix expands Section4.3. We first specify query-independent base routing and verify each rendered relation against the task-local intermediate database. We then detail fact-grounded document rendering and the separate query-conditioned video branch, and trace both procedures with released tasks. The algorithms describe the construction interfaces; renderer-specific templates and visual themes can change without modifying their grounding and validation contracts.

A.3.1.Query-Independent Base Routing and Materialization

Base-renderer assignment.

For every sampled tableR∈𝒟sR\in\mathcal{D}_{s}, the base router extracts a descriptormRm_{R}containing its schema, column types, key declarations, row and column counts, missing-value profile, and serializability constraints. It does not inspect the question, task SQL, or candidate answer. Its renderer set is

(15)ℱbase={𝙲𝚂𝚅,𝙹𝚂𝙾𝙽,𝚂𝚀𝙻𝚒𝚝𝚎,𝙼𝚊𝚛𝚔𝚍𝚘𝚠𝚗,𝙿𝙳𝙵},\mathcal{F}_{\mathrm{base}}=\{\mathtt{CSV},\mathtt{JSON},\mathtt{SQLite},\mathtt{Markdown},\mathtt{PDF}\}, Video is not a member ofℱbase\mathcal{F}_{\mathrm{base}}: it operates at the task level and may depend on the question, SQL structure, and candidate answer rather than materializing a single tableRR. The base policy first removes renderers that cannot faithfully encodemRm_{R}. Flat tables can be written as CSV or record-oriented JSON; relational groups with declared keys can be retained in SQLite; and tables selected for long-form rendering are passed to the Markdown/PDF document generator. A fixed seed breaks ties among eligible choices, while batch-level counters favor underrepresented modalities. These counters balance assignments across the collection; they do not prescribe the number of files, artifact sizes, or the proportions of long and wide tables in an individual workspace.

Algorithm5gives the normalized procedure. A table may be assigned to more than one renderer, and every table must have at least one successful base representation. When a preferred renderer fails its round-trip check, the router tries the next compatible choice; failure of all choices rejects the materialization attempt.

Algorithm 5Query-independent base routing and materialization1:Sampled database

𝒟s\mathcal{D}_{s}, policy

πr\pi_{r}, seed

zrz_{r} 2:Base renderer set

ℱbase\mathcal{F}_{\mathrm{base}}, coverage state

HH 3:Fallback order

OfO_{f} 4:Base workspace

𝒲base\mathcal{W}_{\mathrm{base}} 5:

𝒲base←∅\mathcal{W}_{\mathrm{base}}\leftarrow\varnothing 6:for all

R∈SeededOrder​(𝒟s,zr)R\in\textsc{SeededOrder}(\mathcal{D}_{s},z_{r})do

7:

mR←TableMetadata​(R)m_{R}\leftarrow\textsc{TableMetadata}(R) 8:

ER←Compatible​(mR,ℱbase)E_{R}\leftarrow\textsc{Compatible}(m_{R},\mathcal{F}_{\mathrm{base}}) 9:

UR←Route​(mR,ER,H,πr,zr)U_{R}\leftarrow\textsc{Route}(m_{R},E_{R},H,\pi_{r},z_{r}) 10:if

UR=∅U_{R}=\varnothingthen

11:

UR←{First​(ER)}U_{R}\leftarrow\{\textsc{First}(E_{R})\} 12:endif

13:for all

f∈WithFallbacks​(UR,ER,Of)f\in\textsc{WithFallbacks}(U_{R},E_{R},O_{f})do

14:

a←Render​(R,f)a\leftarrow\textsc{Render}(R,f) 15:if

RoundTripValid​(a,R,f)\textsc{RoundTripValid}(a,R,f)then

16:

𝒲base←𝒲base∪{a}\mathcal{W}_{\mathrm{base}}\leftarrow\mathcal{W}_{\mathrm{base}}\cup\{a\} 17:

H​[f]←H​[f]+1H[f]\leftarrow H[f]+1 18:

MarkRepresented​(R)\textsc{MarkRepresented}(R) 19:if

AssignmentsComplete​(R,UR)\textsc{AssignmentsComplete}(R,U_{R})then

20:break

21:endif

22:endif

23:endfor

24:if

¬Represented​(R)\neg\textsc{Represented}(R)then

25:return

reject\mathrm{reject} 26:endif

27:endfor

28:return

𝒲base\mathcal{W}_{\mathrm{base}}

Serialization contract.

Each structured renderer is paired with a parser that reconstructs a canonical relation. The comparison covers the column inventory, typed cell values, null positions, duplicate-row multiplicity, and row count. CSV uses quoted fields and an explicit schema side record during construction; JSON is written as a list of records with stable keys; and SQLite recreates declared types, primary keys, and foreign keys before inserting rows. File order is not treated as relational meaning unless an order-bearing field is itself part of the table. For a rendererff, acceptance requires

(16)Canon⁡(Parsef⁡(Renderf⁡(R)))=Canon⁡(R),\operatorname{Canon}\bigl(\operatorname{Parse}_{f}(\operatorname{Render}_{f}(R))\bigr)=\operatorname{Canon}(R), where canonicalization normalizes physical encodings while retaining data types and row multiplicity. The construction ledger records the table, renderer, seed, file hash, row count, and validation result. This information supports reproduction and repair but is not placed in the released workspace.

A.3.2.Fact-Grounded Table-to-Document Rendering

Document plan.

Document rendering converts a complete sampled table into a long-form report without placing the full table in a prompt at once. The LLM-based planner first infers a domain-appropriate genre, such as a clinical event ledger, financial audit, or operational briefing. It then chooses anchor columns that identify the entity or record. Declared keys receive priority; otherwise, high-uniqueness identifiers and stable entity names are used. The remaining columns are grouped by semantic role, and rows are partitioned to respect the generation budget. A small table is represented by a single row batch and attribute cluster, making one-pass generation a special case of the same procedure. For larger tables, the plan can emphasize rows (a horizontal pass), attribute clusters (a vertical pass), or alternate between them. In each case, the generation unit reduces to anchors, a bounded row set, and one or more attribute clusters.

Every generation block contains (i) its row identifiers, (ii) the anchor columns repeated from those rows, and (iii) one attribute cluster. Repeating anchors allows sections generated from distant attribute clusters to remain joinable after assembly. Each non-null source cell is assigned an internal cell ID, and each block carries the set of cell IDs that it must express. Null-valued fields follow the plan’s explicit policy: either state that the field is unavailable or omit it without inventing a replacement.

Grounded generation.

Algorithm6shows the generation and checking loop. The model may vary discourse structure, connective text, and non-evidential background detail, but protected identifiers, numbers, dates, units, and categorical values must remain recoverable with their source precision. When contrastive or corrective prose is used for stress testing, the source value must be identified unambiguously as the final record value. A failed block is regenerated from the validator feedback rather than silently removed.

Controlled document complexity.

Three controls vary extraction difficulty without changing the sampled table. Recursive detailing expands a block into successively more specific report sections. The null policy varies whether missing fields are stated or omitted. Finally, a configurable subset of blocks receives domain-relevant narrative context, nearby non-answer facts, or a correction-style presentation. Such additions cannot replace a required source cell, alter a protected value, or introduce a competing final value for the same record and field.

Algorithm 6Fact-grounded table-to-document rendering1:Sampled table

XRX_{R}, schema metadata

mRm_{R} 2:Token budget

BB, retry limit

KK, output format

ff 3:Markdown or PDF artifact

dRd_{R} 4:

(ηR,𝒦R,𝒞R,ℐR)←PlanDocument​(XR,mR,B)(\eta_{R},\mathcal{K}_{R},\mathcal{C}_{R},\mathcal{I}_{R})\leftarrow\textsc{PlanDocument}(X_{R},m_{R},B) 5:

ℬR←BuildBlocks​(XR,𝒦R,𝒞R,ℐR)\mathcal{B}_{R}\leftarrow\textsc{BuildBlocks}(X_{R},\mathcal{K}_{R},\mathcal{C}_{R},\mathcal{I}_{R}) 6:

𝒢R←∅\mathcal{G}_{R}\leftarrow\varnothing 7:for all

Bj∈ℬRB_{j}\in\mathcal{B}_{R}do

8:for

k=1k=1to

KKdo

9:

gj←GenerateSection​(Bj,ηR)g_{j}\leftarrow\textsc{GenerateSection}(B_{j},\eta_{R}) 10:

ej←ValidateCells​(gj,Bj)e_{j}\leftarrow\textsc{ValidateCells}(g_{j},B_{j}) 11:if

ej=∅e_{j}=\varnothingthen

12:break

13:endif

14:

Bj←AttachFeedback​(Bj,ej)B_{j}\leftarrow\textsc{AttachFeedback}(B_{j},e_{j}) 15:endfor

16:if

ej≠∅e_{j}\neq\varnothingthen

17:return

reject\mathrm{reject} 18:endif

19:

𝒢R←𝒢R∪{gj}\mathcal{G}_{R}\leftarrow\mathcal{G}_{R}\cup\{g_{j}\} 20:endfor

21:

dR←AssembleMarkdown​(𝒢R,ηR)d_{R}\leftarrow\textsc{AssembleMarkdown}(\mathcal{G}_{R},\eta_{R}) 22:

ValidateCoverage​(dR,XR)\textsc{ValidateCoverage}(d_{R},X_{R}) 23:if

f=𝙿𝙳𝙵f=\mathtt{PDF}then

24:

dR←ConvertAndCheckPdf​(dR)d_{R}\leftarrow\textsc{ConvertAndCheckPdf}(d_{R}) 25:endif

26:return

dRd_{R}

The block validator combines exact and typed checks. Exact-token checks cover IDs, codes, protected strings, and categorical values. Numeric values are parsed with their units and compared at the recorded precision; dates and times are normalized before comparison; and ordinary text values are checked against the source cell or an approved surface-form map. The final coverage pass verifies that every required cell ID is linked to at least one document span and that no span assigns two incompatible values to the same record–field pair. PDF conversion is followed by text extraction and page decoding checks. Figure13gives the normalized block-level generation contract.

SYSTEM

Writeonesectionofarealistic<DOCUMENT_GENRE>fromthesupplied

tableblock.Thetableblockisthesolesourceofrecordfacts.

GROUNDINGRULES

1.ExpresseveryREQUIRED_CELL_IDwithitsexactentityassociation.

2.Preserveidentifiers,categoricalvalues,numericprecision,dates,

times,signs,andunits.Donotmergevaluesfromdifferentrows.

3.FollowNULL_POLICY.Neverinferamissingvalue.

4.Youmayaddconnectiveproseordomain-neutralscenedetail,butit

mustnotcreateacompetingvalueforanyprotectedfield.

5.KeepANCHOR_FIELDSexplicitsothissectioncanbejoinedwithother

sectionsaboutthesamerecords.

6.Returnonlytherequestedsectionandthecell-to-spanalignmentJSON.

INPUT

STYLE:<STYLE_SPECIFICATION>

ANCHOR_FIELDS:<ANCHOR_COLUMNS_AND_VALUES>

ATTRIBUTE_CLUSTER:<COLUMN_NAMES_TYPES_AND_DESCRIPTIONS>

ROWS:<TYPED_VALUES_WITH_CELL_IDS>

REQUIRED_CELL_IDS:<CELL_IDS>

NULL_POLICY:<EXPLICIT_OR_OMIT>

OUTPUT

{

“section”:“<MARKDOWN_SECTION>”,

“alignments”:[

{“cell_id”:“<CELL_ID>”,“surface”:“<TEXT_SPAN>”}

]

}

Figure 13.Prompt for fact-grounded document generation.

A.3.3.Document-Rendering Running Example

Figure14traces one row from theLABEVENTSdocument in released Task 310. The sampled row contains seven fields. The planner usesROW_IDas the anchor and separates event context from the laboratory measurement. The two resulting blocks are therefore generated at different positions in a long clinical ledger but can be joined through event 142456.

Figure 14.Source-to-document alignment for one sampledLABEVENTSrow in Task 310. Colors preserve field-group provenance from the source row through grounded generation blocks to mentions in the released Markdown document; ellipses shorten the displayed spans.All seven source cells are aligned to the two spans, with the anchor appearing in both. Recovering the measurement as a structured record requires associating fields across separated narrative sections through the exact record identifier while ignoring surrounding clinical and operational prose.

A.3.4.Query-Conditioned Video Rendering

Evidence selection.

Video augmentation begins only after the sampled SQL has been executed. The selector parses comparison predicates, projected fields, aggregation and ordering operators, and the candidate tabular result. A predicate candidate is represented as a typed atom

(17)epred=(t,c,o,v,λ,γ),e^{\mathrm{pred}}=(t,c,o,v,\lambda,\gamma), wherettandccidentify the source table and column,oois the operator,vvis the typed condition value,λ\lambdais its aligned question span, andγ\gammarecords display constraints such as precision and units. Null tests, join keys, grouping operators, unstable relative-time conditions, and projection-only fields are excluded from predicate abstraction. An answer atom

(18)eans=(r,c,v,γ)e^{\mathrm{ans}}=(r,c,v,\gamma) identifies one result row, output field, typed value, and display contract. Answer-evidence rendering is used only when the result is compact enough for its required atoms to remain legible across scenes.

Two rendering strategies.

Inpredicate abstraction, one or more stable predicate atoms are removed from the explicit question and expressed through a business scene, such as a configuration panel, time window, or eligibility rule. The remaining workspace still contains the records on which the recovered predicate must be applied. Inanswer-evidence rendering, result atoms are distributed across tables, charts, cards, or temporally separated views. Direct copies of the same answer-bearing source are withheld when they would make the video unnecessary. In both cases, distractors are drawn from nearby fields, entities, periods, or boundary examples; they cannot change the selected atom or create a second valid answer.

Storyboard and rendering.

Each selected atom receives a stable evidence ID. A storyboard scene lists its duration, evidence IDs, visual component, exact data constants, visible text, and narration. The planner can decide how an atom is communicated, but the constants in its component specification are filled from the typed evidence record. Supporting records for boundary cases and distractors are queried from the sampled database and retain their source table and row identifiers. The storyboard is compiled into a task-specific React/Remotion composition. Narration is synthesized separately, and measured audio duration is used to set scene timing. Compilation or rendering failures are repaired at the component level and rendered again in task isolation.

Algorithm7summarizes the branch. Question rewriting operates over the aligned spanλ\lambda: it replaces only the selected condition or answer reference with a description that points to the video. The requested output fields, aggregation, comparison, ordering, units, and temporal scope remain unchanged. The question is rejected when the selected span cannot be removed without changing those semantics.

Algorithm 7Query-conditioned video augmentation1:Question

qcq_{c}, SQL

σc\sigma_{c}, answer

𝒴s\mathcal{Y}_{s} 2:Base workspace

𝒲base\mathcal{W}_{\mathrm{base}}, strategy

hh 3:Adapted question

qrq_{r}, workspace

𝒲\mathcal{W} 4:

A←ExtractTypedAtoms​(AST⁡(σc),𝒴s,qc)A\leftarrow\textsc{ExtractTypedAtoms}(\operatorname{AST}(\sigma_{c}),\mathcal{Y}_{s},q_{c}) 5:

ℰv←SelectAtoms​(A,h)\mathcal{E}_{v}\leftarrow\textsc{SelectAtoms}(A,h) 6:if

ℰv=∅\mathcal{E}_{v}=\varnothingthen

7:return

reject\mathrm{reject} 8:endif

9:

Sv←PlanStoryboard​(ℰv,h)S_{v}\leftarrow\textsc{PlanStoryboard}(\mathcal{E}_{v},h) 10:

ValidateAtomCoverage​(Sv,ℰv)\textsc{ValidateAtomCoverage}(S_{v},\mathcal{E}_{v}) 11:

Cv←CompileComposition​(Sv)C_{v}\leftarrow\textsc{CompileComposition}(S_{v}) 12:

av←SynthesizeNarration​(Sv)a_{v}\leftarrow\textsc{SynthesizeNarration}(S_{v}) 13:

v←RenderVideo​(Cv,av)v\leftarrow\textsc{RenderVideo}(C_{v},a_{v}) 14:

qr←RewriteAlignedSpans​(qc,ℰv,h)q_{r}\leftarrow\textsc{RewriteAlignedSpans}(q_{c},\mathcal{E}_{v},h) 15:

𝒲←Integrate​(𝒲base,v,ℰv,h)\mathcal{W}\leftarrow\textsc{Integrate}(\mathcal{W}_{\mathrm{base}},v,\mathcal{E}_{v},h) 16:

ValidateVideoTask​(qr,𝒲,v,ℰv)\textsc{ValidateVideoTask}(q_{r},\mathcal{W},v,\mathcal{E}_{v}) 17:return

qr,𝒲q_{r},\mathcal{W}

The normalized storyboard-planning contract appears in Figure15.

SYSTEM

Designarealisticdata-videostoryboardthatcommunicateseverysupplied

evidenceatomwithoutchangingitsvalueoranalyticalrole.

RULES

1.EveryEVIDENCE_IDmustappearinatleastonesceneandmustretainits

typedvalue,operator,precision,unit,andentityassociation.

2.Predicateabstractionmustcommunicatebothcomparisondirectionand

threshold.Answerevidencemustremainreadablebutneednotbeadjacent.

3.Useonlysuppliedrecordsfordata-bearingdistractors.Donotinventan

alternativeruleorasecondvalidanswer.

4.Narrationshouldsupportthebusinessscenario;donotreadoutall

evidencewhendoingsowouldmakevisualstructureunnecessary.

5.ReturnJSONonly.Everyvisibleconstantmustidentifyitssourceatom.

INPUT

TASK_STRATEGY:<PREDICATE_ABSTRACTION_OR_ANSWER_EVIDENCE>

EVIDENCE_ATOMS:<TYPED_ATOMS_WITH_STABLE_IDS>

SUPPORTING_RECORDS:<BOUNDARY_AND_DISTRACTOR_RECORDS>

VISUAL_COMPONENT_LIBRARY:<AVAILABLE_COMPONENTS>

OUTPUT

{

“scenes”:[{

“scene_id”:“...”,“duration_hint”:“...”,

“evidence_ids”:[“...”],“component”:“...”,

“data_constants”:[{“atom_id”:“...”,“value”:“...”}],

“visible_text”:[“...”],“narration”:“...”

}]

}

Figure 15.Prompt for evidence-grounded storyboard planning.

A.3.5.Video-Rendering Running Example

Predicate-abstraction example.

Released Task 193 asks which equity-freeze records meet the “Major Share Freeze Alert” configured in the video and requests shareholder, involved shares, start date, and end date. The underlying record filter contains two typed predicate atoms: security code600180andPCTOfTotalShares≥0.05\texttt{PCTOfTotalShares}\geq 0.05. Rather than placing these constants in the question, the video identifies the monitored security and later displays a configuration panel whose field isPCTOfTotalShares, direction is≥\geq, and cutoff is 5.00%. Table4follows these atoms into the released task, while Figure16(a)–(c) shows their visual carriers in the rendered video.

Table 4.Predicate-abstraction trace for released Task 193. Values in the application row are read from the task-local SQLite artifact; the output table omits the trigger percentage because it is not requested.Solving this task combines the entity scope and predicate from the video with the freeze records in SQLite and the output projection in the question. Neither the percentage field nor the 5.00% cutoff is part of the requested output.

Answer-evidence companion.

Task 82 illustrates the other strategy. Its video presents two transfer rows for company 79959 and a descending sort on pre-transfer ownership. The rows shown are

(19)(79959,0.4787,0.4787,2018-03-15),\displaystyle(9959,04787,04787,\,\text{2018-03-15}),(79959,0.4664,0.4640,2021-06-11),\displaystyle(9959,04664,04640,\,\text{2021-06-11}), with columns for company code, ownership before transfer, ownership after transfer, and transaction date. The released question asks which record has the higher pre-transfer percentage. The required output atom is therefore the first row, while the second row supplies a same-entity comparison. The source transfer relation is not duplicated as an agent-visible structured artifact; the remaining workspace contains related company data and natural distractors. This task can be solved only after reading the two percentages, associating them with the correct rows, and applying the requested comparison. Figure16(d) shows the corresponding ranked view.

(a) Task 193: monitored-security scope(b) Task 193: records around the decision boundary(c) Task 193: trigger field, direction, and cutoff(d) Task 82: ranked answer-evidence rowsFigure 16.Representative frames from the released task videos. Task 193 distributes its security scope and predicate specification across separate scenes: panel (a) identifies the monitored security, panel (b) supplies boundary records, and panel (c) defines the trigger. Panel (d) shows the two same-company transfer records used by the answer-evidence strategy in Task 82.

A.3.6.Artifact Validation and Failure Handling

Validation is applied at three levels. First, renderer checks compare CSV, JSON, and SQLite round trips with the sampled relations using Equation16. Second, document checks verify required cell coverage, protected tokens, typed numeric/date/unit fidelity, null handling, document decoding, and the absence of incompatible claims for one record–field pair. Third, video checks verify the evidence schema, atom-to- scene coverage, component constants, question-span replacement, compilation, audio presence, file decoding, duration, and sampled frames.

A structural failure triggers deterministic fallback, block regeneration, or component repair as appropriate. A semantic failure—for example, a misstated document value, a missing comparison direction, an unreadable video constant, or a question that still exposes relocated evidence—returns the artifact to generation with a typed error report. Tasks that cannot satisfy the contract within the retry budget are rejected. Accepted artifacts then enter Human Review & Task Repair in Section4.4, where reviewers inspect the complete question, workspace, and reference answer together.

A.4.Construction Models and Cost

Table5reports the LLM configurations and mean API cost per task for the model-based construction components.

Table 5.LLM configurations and API costs for benchmark construction.The document cost includes generation and model-based validation. The video cost is estimated from sampled successful video-generation records. Reported costs cover model API usage and exclude expert review, local rendering, storage, and other infrastructure.

A.5.Human Review & Task Repair Details

Human review operates on a complete candidate task version comprising its question, workspace, and reference result. It begins only after the structural and artifact-level checks in AppendixA.3. Those checks can establish that files decode, database relations round-trip, and protected values are preserved; the expert protocol determines whether the assembled task has a clear user intent, sufficient evidence, a unique answer, and suitable evaluation semantics.

A.5.1.Staffing, Assignment, and Blinding

The review pool comprises 11 domain experts with recorded domain and data-analysis expertise. Each task is assigned to at least two reviewers. Assignments are balanced across the active pool and can be adjusted for domain coverage before review begins. A pending assignment may be replaced, but reviewer membership is fixed once an independent answer has been submitted.

Reviewers, repairers, and administrators have separate roles. Reviewers solve and verify assigned tasks; repairers prepare a new task version after a disagreement; administrators manage assignments and record the final accept-or-remove decision. During independent solving, a reviewer cannot access the source SQL, reference result, discussion history, or peer submissions. Peer judgments become visible only after that reviewer has locked both stages of their own review.

A.5.2.Two-Stage Review Record

Table6summarizes the information boundary and the record produced at each stage. Phase 1 requires a rectangular table with a header; a header-only table is valid when the correct answer is empty. Every submission must include at least one evidence reference. An evidence reference contains an artifact path, its modality, a row, key, section, page, or timestamp anchor, the supported fact, and its analytical role such as filter, join, projection, aggregation, ordering, or unit interpretation. The reviewer also records the interpreted grain, predicates, aggregation, ordering, units, null and duplicate semantics, issue flags, and confidence. The server validates referenced paths against the current task version and makes the submitted result immutable.

Table 6.Information boundaries and required records in the expert-review protocol. Diagnostic table comparison is shown only after the independent answer has been locked and is not used as the official benchmark evaluator.Phase 2 exposes the reference result. The reviewer assigns a gold verdict fromMatch,Mismatch, andUnsure, together with one disposition:Pass, repair the question, workspace, gold, or configuration, repair multiple components, or remove the task. Each reviewer independently authors a configuration covering the semantic type of every output column, any numeric precision and percentage convention, and whether row order is required. The configuration is derived from the question and verified gold rather than tuned after observing model predictions.

A.5.3.Consensus, Adjudication, and Versioned Repair

A task version reaches consensus only when every assigned reviewer has completed both review stages, marked the gold asMatch, selectedPass, and submitted the same canonical evaluation configuration. Missing verifications never count as agreement. Consequently, reviewers who agree on all gold values still enter adjudication when their type, precision, or ordering configurations differ.

Adjudication is evidence based rather than a majority vote. Reviewers identify the conflicting claim and cite its location in the released workspace. The resulting repair record states its reason and identifies which components change: the question, workspace artifacts, reference result, evaluation configuration, or a combination of them. Repairs follow a minimal-change policy, retaining unaffected artifacts and semantics. Finalizing a repair preserves the previous version, activates a new one, and reassigns the same reviewers. The repaired version is independently rechecked before discussion resumes, and each reviewer submits a new verification and configuration for that version. Discussion, repair, and recheck repeat until the consensus conditions above hold. A task that cannot be repaired into an unambiguous and releasable instance is removed with an explicit reason.

A.5.4.Review Dimensions and Issue Taxonomy

Reviewers assess structural integrity, evidence sufficiency, answer uniqueness, gold correctness, cross-modal consistency, artifact fidelity, video necessity, question quality, evaluation validity, and release suitability. Table7groups the structured issue flags used to make these judgments comparable across tasks. Multiple flags may be attached to one review; free-text rationale and evidence anchors preserve the concrete claim behind each flag.

Table 7.Issue taxonomy used during independent solving and gold/configuration verification. The repair surface is selected separately, so one issue can require changes to multiple task components.

A.5.5.Audit Trail and Review Interface

The review system records assignments, independent submissions, verifications, state transitions, discussions, repair drafts and artifact replacements, version activation, acceptance, and removal as append-only events with actor, task version, UTC timestamp, and event payload. Base benchmark files remain read only; repaired context files are stored as versioned overlays. The resulting record supports agreement, issue, repair, and post-repair acceptance statistics without reconstructing decisions from the released files.

The browser console jointly presents the question, modality-filtered workspace, native artifact viewers, evidence form, result comparison, configuration editor, and discussion history. Figure17shows an illustrative session for released Task 193. The reviewer recovers an alert predicate from video, applies it to SQLite, and records both evidence anchors before the reference result is unlocked.

(a) Blind inspection of the heterogeneous workspace(b) Evidence-anchored independent result submission(c) Gold verification and evaluation-config authoring(d) Cross-review comparison and recorded adjudicationFigure 17.Illustrative end-to-end reviewer trace in the human-review system. Panel (a) shows the blind phase, in which the question and complete workspace are visible while gold and peer reviews remain locked. Panel (b) links the candidate result to a video timestamp and filtered SQLite rows. Panel (c) unlocks result comparison and supports independent authoring of per-column type and precision settings after answer submission. Panel (d) exposes both completed reviews and records a configuration disagreement for adjudication.

A.6.Evaluation Protocol Details

This appendix expands the evaluation protocol in Section4.5. It specifies the frozen task configuration, canonicalization rules, joint column alignment, and tabular-result matching.

A.6.1.Frozen Per-Task Configuration

Each task configuration contains a schema version, the task identifier, an ordering flag, and one entry for every reference column. Column entries are indexed in reference order and record a human-auditable reference name and semantic type. Numeric entries additionally specify an integer, decimal-place, or significant-digit comparison rule; a unit convention and permission to use a percent sign are included when relevant. The reference name is retained for auditing but is never compared with a prediction header.

Figure18gives the released configuration for Task 120. The task requests region, period-end date, and per-capita GDP in the order presented by the source. Accordingly, its rows are order sensitive, and its three columns are normalized as text, datetime, and a number rounded to one decimal place.

{

“schema_version”:“1.0”,

“task_id”:“task_120”,

“order_sensitive”:true,

“columns”:[

{

“gold_index”:0,

“gold_name”:“Region”,

“type”:“text”

},

{

“gold_index”:1,

“gold_name”:“Period-enddate”,

“type”:“datetime”

},

{

“gold_index”:2,

“gold_name”:“GDPpercapita”,

“type”:“number”,

“comparison”:{

“mode”:“decimal_places”,

“digits”:1

}

}

]

}

Figure 18.Frozen evaluation configuration for Task 120.

A.6.2.Canonicalization Rules

Text and nulls.

Text values are stripped of surrounding whitespace, line endings are standardized, and Unicode is normalized to NFC; comparison otherwise remains case sensitive. Identifiers and ordinary strings share this type because both require exact lexical equality after normalization. An empty text cell is canonicalized as null. For non-text columns, empty cells and the case-insensitive tokensnull,none,nan,nat, and<na>are treated as null. A null value matches only another null.

Numbers.

Numbers are parsed as finite base-10 decimals; ordinary decimal notation, scientific notation, and valid comma thousands separators are accepted. Theintegermode requires an integral value. Thedecimal_placesandsignificant_digitsmodes round both reference and predicted values with round-half-up at the configured precision, after which their canonical decimal representations must be identical.

The numeric unit isplain,percentage_points, orfraction. A percent-marked prediction is accepted only whenallow_percent_signis enabled. Underpercentage_points,3.5%denotes3.53.5; underfraction, it denotes0.0350.035. An unmarked value is interpreted directly in the configured unit.

Dates, datetimes, and Booleans.

Dates and datetimes use ISO syntax. A date column also accepts a midnight datetime, while a non-midnight time is invalid. Timezone-aware values are converted to UTC before comparison. Boolean values accept1/true/yes/yand their Chinese affirmative counterpart as true, and0/false/no/nand the corresponding Chinese negative token as false; alphabetic tokens are case insensitive.

A.6.3.Joint Alignment and Tabular-Result Matching

Prediction headers are required for valid CSV serialization but do not participate in scoring. After checking that prediction and reference have the same shape, the evaluator considers every one-to-one mapping from predicted columns to reference columns. A mapping remains eligible only when all cells in each predicted column can be canonicalized by the rule of its mapped reference column. The mapped columns are then reassembled into rows before comparison, so values from different prediction rows cannot be combined independently.

For an order-sensitive task, the canonical row sequences must be identical. Otherwise, rows are compared as multisets, retaining the multiplicity of every duplicate row. Algorithm8gives the task-level procedure used by the official evaluator.

Algorithm 8Task-level tabular-result evaluation1:Prediction

PP, reference

GG, configuration

c=(o,{νj}j=1d)c=(o,\{\nu_{j}\}_{j=1}^{d}) 2:Binary task score

3:if

¬ValidCSV​(P)\neg\textsc{ValidCSV}(P)then

4:return

0 5:endif

6:if

Shape​(P)≠Shape​(G)\textsc{Shape}(P)\neq\textsc{Shape}(G)then

7:return

0 8:endif

9:

G⋆←Canon​(G,c)G^{\star}\leftarrow\textsc{Canon}(G,c) 10:for all

π∈Πd\pi\in\Pi_{d}do

11:

Pπ←ReorderColumns​(P,π)P_{\pi}\leftarrow\textsc{ReorderColumns}(P,\pi) 12:if

¬Compatible​(Pπ,c)\neg\textsc{Compatible}(P_{\pi},c)then

13:continue

14:endif

15:

P⋆←Canon​(Pπ,c)P^{\star}\leftarrow\textsc{Canon}(P_{\pi},c) 16:if

Rows​(P⋆,o)=Rows​(G⋆,o)\textsc{Rows}(P^{\star},o)=\textsc{Rows}(G^{\star},o)then

17:return

11 18:endif

19:endfor

20:return

0

A.7.Experimental Configurations

A.7.1.Backbones

Table8records the exact endpoints used in the controlled backbone comparison. The six models were publicly released within the four months preceding evaluation and were accessed through Vercel AI Gateway in July 2026. We leave reasoning effort and all unspecified sampling parameters at the provider default; each endpoint receives the same 32,768-token output ceiling.

Table 8.Backbones used in the controlledDataSpace-Agentcomparison. Release denotes the month of public model availability.

A.7.2.Agent Harnesses

The complementary harness comparison fixes the endpoint toxiaomi/mimo-v2.5. Table9lists the pinned implementations. All model roles exposed by a harness, including subagents and summarization or compaction calls, are mapped to the same backbone and model fallback is disabled.

Table 9.Agent harnesses compared with MiMo-V2.5 fixed as the backbone.DataSpace-Agentalternates model responses with tool observations until the model submits an answer or reaches a resource limit. It exposes three actions:bashexecutes local data-processing commands,view_imagereturns a selected image to the multimodal backbone, andsubmit_answervalidates and submits a tabular result as CSV. Its prompt specifies the task, tool interfaces, workspace location, and output contract, without task-specific source selection, modality routing, or analytical operators.

A.7.3.Specialized Data-Agent Compatibility

Table10records their status at the time of evaluation in July 2026. We require a system to be reproducibly runnable over the complete task directory and to support the benchmark’s multimodal input and tabular-output contract without replacing core components.

Table 10.Compatibility assessment of specialized data-agent systems. “Public” refers to an official implementation released by the system’s authors.FDABench likewise reports reimplementing TAIJI, AOP, and AgenticData from their papers because the original systems were not open-sourced(Wanget al.,2025). Using such reconstructions, adding a video-capable perception model, or rewriting a system’s output path would introduce substantial choices absent from the original systems and prevent a faithful comparison.

A.7.4.Runtime and Reproducibility

For the backbone comparison, eachDataSpace-Agentrun is limited to 60 model turns, 50 tool actions, 1,800 seconds of wall-clock time, and 180 seconds per shell command. Its isolated runtime receives 4 CPUs, 16 GiB of memory, and no network access. Full-benchmark jobs use eight-way task concurrency, which affects throughput but not per-task limits.

For the harness comparison, all systems receive a fresh session, a task-local home directory, and Data Workbench Runtime 1.0. The runtime supplies generic CSV, JSON, SQLite, Markdown, PDF, image/OCR, and video utilities, but no retrieval, schema-linking, document-QA, video-QA, or Text-to-SQL solver. Model-generated commands cannot access the network. The host-side controller can reach only the configured inference endpoint and never exposes its credential to generated code. Each harness receives the same 1,800-second wall-clock limit, task input, and CSV output contract; its internal action budget and context-management policy remain native to that harness. Prompts, configuration files, validated predictions, and raw execution traces are retained for release.

A.8.Additional Statistics and Results

A.8.1.Backbone Efficiency

Table11reports the complete efficiency values underlying Figure5. Cached prompt tokens are subsets of input tokens, and reported reasoning tokens are subsets of output tokens; neither is added again to the total. Costs are computed using the providers’ official pricing at the time of evaluation.

Table 11.Efficiency statistics for the controlled backbone comparison withDataSpace-Agentfixed. Token, action, and latency values are per-task averages over all 410 tasks; parentheses give the token median and latency 90th percentile. API cost per task uses each provider’s official pricing.Efficiency on unsuccessful tasks.For five of the six backbones, an incorrect task consumes between 1.2 and 3.2 times as many mean tokens as a correct task and also requires more tool actions. GPT is the exception, with nearly identical resource use in the two groups. Longer exploration therefore frequently reflects failure to converge rather than additional solved tasks.

A.8.2.Performance by Task Characteristic

Table12gives the sample sizes and raw accuracies underlying Figure6. The multimodal group pools 115 tasks requiring two modalities and 19 requiring three.

Table 12.Task Accuracy (%) by task characteristic withDataSpace-Agentfixed. Required modalities are those used by the verified solution path; workspace quartiles are formed by total workspace bytes.

A.8.3.Trace-level Root-cause Audit

Table13reports the complete human-confirmed subtype distribution used in Figure7.

Table 13.Human-confirmed root-cause subtypes for 136 audited Grok 4.5 failures.StageSubtypeOperational definitionTasksShareQQ1Target output, requested entities, or row granularity is misunderstood.1712.5%Q2A condition, comparison, temporal scope, or target population is misunderstood.139.6%Q3An ordering or answer constraint is misunderstood.10.7%DD2The wrong artifact or source is selected as authoritative evidence.32.2%EE1A read, retrieval window, page, frame, or query omits required records.53.7%E2The representation of the correct artifact is parsed incorrectly.32.2%E3Document, image, or video evidence is transcribed incorrectly.10.7%GG1A field or schema element is assigned the wrong meaning.53.7%G2Entities, identifiers, join keys, or records are aligned incorrectly.10.7%G3A correctly read value is normalized with the wrong unit, date, language, or scale.21.5%G4Conflicting sources or versions are reconciled incorrectly.42.9%CC1Filtering, Boolean, or NULL logic is applied incorrectly.21.5%C2A join, set operation, or deduplication step is incorrect.10.7%C3Aggregation, grouping, windowing, or ranking is incorrect.10.7%C6A corrected intermediate result is not propagated through a multi-step computation.10.7%MM1A correct internal result is submitted with extra or missing columns.6044.1%M2A correct internal result is submitted with extra or missing rows.10.7%M3Type, numeric precision, date precision, or value formatting is altered during output.96.6%M4The correct rows are serialized in an incorrect required order.10.7%TT1Unproductive iteration exhausts the action budget without an earlier persistent analytical error.42.9%T3The agent terminates or fails to submit after obtaining the answer.10.7%Total136100.0%The audit unit is one failed Grok 4.5 task. We assign exactly one primary cause: the earliest observable divergence that conflicts with the verified solution, remains uncorrected, and determines the submitted result or prevents submission. Exploratory errors that the agent later corrects are not primary causes, and evaluator outcomes such as a column-count mismatch or missing prediction are retained only as symptoms.

Audit procedure.

For each of the 136 audited failures, GPT-5.6 Sol receives the observable tool/action trace, workspace, submitted prediction, reference result, evaluation configuration, and verified solution annotation. It proposes a primary category and subtype, cites the trace event at which the persistent divergence first appears, and gives a counterfactual correction. A human researcher checks every cited event and the relevant workspace evidence, then confirms or revises the proposal. The audit does not use or claim access to hidden chain-of-thought content.

Stage boundaries.

Task intent(Q) covers an incorrect formulation of the requested output, conditions, scope, or ordering constraint.Discovery(D) covers failure to locate the required evidence or selection of the wrong source.Extraction(E) applies when the correct artifact is accessed but its raw values are not recovered accurately;grounding(G) applies when those values are read but assigned the wrong field, entity, unit, or source interpretation.Computation(C) requires correct evidence and semantics followed by an incorrect relational or numerical operation.Materialization(M) is used only when the trace already contains the correct target schema and internal table, or values directly projectable to it.Termination(T) is used only when no earlier persistent Q/D/E/G/C error explains the failure.

Appendix BCompetition Deployment and Artifact Availability

Competition deployment.DataSpaceserved as the official evaluation benchmark for the KDD Cup 2026 Data Agents for Complex Data Analysis competition(KDD Cup 2026,2026). The competition evaluated containerized agent systems through hidden A- and B-board evaluation under challenge-specific runtime, submission, and leaderboard rules. The release described in this paper instead uses the finalized semantics-aware evaluation protocol in Section4.5.

Artifact availability.All 410 task inputs are publicly available athttps://huggingface.co/datasets/HKUSTDial/DataSpace. The release includes reference answers and evaluation configurations for 60 representative tasks, enabling local end-to-end evaluation; the remaining 350 references are withheld for official full-benchmark evaluation. The official evaluator, baseline implementations, experiment configurations, and documentation are available athttps://github.com/HKUSTDial/DataSpace. Both repositories are released under the MIT License.