DataSpace: Benchmarking Data Agents for Verifiable Analytics over Heterogeneous Workspaces

Hugging Face Daily Papers Papers
benchmark data-agents evaluation multimodal analytics nlp

Summary

Introduces DataSpace, a benchmark for evaluating data agents on verifiable tabular analytics over heterogeneous workspaces, containing 410 cross-language tasks and 7,439 artifacts. Current frontier models achieve only 66.34% accuracy, indicating headroom.

Data agents enable natural-language analytics over organizational workspaces, where relevant evidence may be scattered across databases, structured files, long documents, and multimedia. Existing benchmarks largely isolate structured querying, retrieval, or open-ended analysis, leaving heterogeneous evidence discovery, complete tabular outputs, and deterministic evaluation insufficiently unified. We introduce DataSpace, a benchmark in which data agents produce verifiable tabular results from task-local heterogeneous workspaces. It contains 410 cross-language tasks and 7,439 artifacts totaling 15.01 GB across CSV, JSON, SQLite, Markdown, PDF, and video. DataSpace also served as the official evaluation benchmark for the KDD Cup 2026 Data Agents for Complex Data Analysis competition. Each agent receives only a question and workspace and returns the complete requested tabular result. We construct DataSpace with DataSpace-Builder, an execution-grounded framework comprising cross-language transformation, constraint-aware relational sampling, modality routing and artifact rendering, and human review and task repair by 11 domain experts. A deterministic evaluator performs header-invariant column alignment, type- and precision-aware normalization, and order-aware row comparison. Across six recently released frontier multimodal models and five widely used agent harnesses, the best accuracy reaches 66.34%, while harness choice creates a 15.36-point spread with the backbone fixed. Multimodal evidence integration and joins consistently reduce accuracy across all six backbones. These results show that DataSpace remains unsaturated and identify key challenges for improving data-agent reliability.
Original Article
View Cached Full Text

Cached at: 08/07/26, 01:57 PM

Paper page - DataSpace: Benchmarking Data Agents for Verifiable Analytics over Heterogeneous Workspaces

Source: https://huggingface.co/papers/2608.03451 Authors:

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

Dataagentsenablenatural-languageanalyticsoverorganizationalworkspaces,whererelevantevidencemaybescatteredacrossdatabases,structuredfiles,longdocuments,andmultimedia.Existingbenchmarkslargelyisolatestructuredquerying,retrieval,oropen-endedanalysis,leavingheterogeneousevidencediscovery,completetabularoutputs,anddeterministicevaluationinsufficientlyunified.WeintroduceDataSpace,abenchmarkinwhichdataagentsproduceverifiabletabularresultsfromtask-localheterogeneousworkspaces.Itcontains410cross-languagetasksand7,439artifactstotaling15.01GBacrossCSV,JSON,SQLite,Markdown,PDF,andvideo.DataSpacealsoservedastheofficialevaluationbenchmarkfortheKDDCup2026DataAgentsforComplexDataAnalysiscompetition.Eachagentreceivesonlyaquestionandworkspaceandreturnsthecompleterequestedtabularresult.WeconstructDataSpacewithDataSpace-Builder,anexecution-groundedframeworkcomprisingcross-languagetransformation,constraint-awarerelationalsampling,modalityroutingandartifactrendering,andhumanreviewandtaskrepairby11domainexperts.Adeterministicevaluatorperformsheader-invariantcolumnalignment,type-andprecision-awarenormalization,andorder-awarerowcomparison.Acrosssixrecentlyreleasedfrontiermultimodalmodelsandfivewidelyusedagentharnesses,thebestaccuracyreaches66.34%,whileharnesschoicecreatesa15.36-pointspreadwiththebackbonefixed.Multimodalevidenceintegrationandjoinsconsistentlyreduceaccuracyacrossallsixbackbones.TheseresultsshowthatDataSpaceremainsunsaturatedandidentifykeychallengesforimprovingdata-agentreliability.

View arXiv pageView PDFProject pageGitHub11Add to collection

Get this paper in your agent:

hf papers read 2608\.03451

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.03451 in a model README.md to link it from this page.

Datasets citing this paper1

#### HKUSTDial/DataSpace Viewer• Updated2 days ago • 410 • 142 • 3

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.03451 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

AgenticDataBench: A Comprehensive Benchmark for Data Agents

Hugging Face Daily Papers

Introduces AgenticDataBench, a comprehensive benchmark for evaluating LLM-based data agents across diverse domains with fine-grained skill-based metrics, including real-world B2B use cases and synthetic tasks.

TerraBench: Can Agents Reason Over Heterogeneous Earth-System Data?

arXiv cs.AI

TerraBench is a new benchmark for evaluating AI agents' ability to reason over heterogeneous Earth-system data, including gridded data, satellite imagery, and simulator outputs. It reveals significant limitations in current frontier models, with top performers achieving only 59.2% tool-use score on average.