Paper page - DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments?

Source: https://huggingface.co/papers/2608.10366

Abstract

DSAgentBench evaluates autonomous agents on complete, multi-tool data-science workflows in real computing environments and reveals major performance gaps.

Real-world data science involves long-horizon workflows that span data wrangling, exploration, modeling, visualization, and validation, and require coordinated use of tools such as notebooks, IDEs, terminals, browsers, and databases within real operating environments. Yet existing benchmarks lack real-computer interaction and do not evaluate whether agents can execute complete end-to-enddata-science workflowsin realistic computing environments, failing to capture the multi-stage, multi-tool nature of data-science practice. We introduceDSAgentBench, the first benchmark to evaluate whether agents can automate fulldata-science workflowsinside real computer environments.DSAgentBenchcontains 275 diverse tasks covering the entire data-science life-cycle, reflecting the complexity and tool coordination required in practice. Each task requires grounding decisions in intermediate outputs and coordinated tool use, and includes adeterministic evaluatorthat verifies analytical correctness, visual outputs, and model performance rather than code-only execution. Our extensive experiments with 15 closed- and open-source models show that even the strongest agent, Claude-4.6-Sonnet, achieves only 56.70% task success, while all open-source agents remain below 1%, frequently failing at tool orchestration, OS grounding, and multi-step reasoning. These results reveal a substantial capability gap between currentagentic systemsand realdata-science workflows, positioningDSAgentBenchas a foundation for developing grounded, verifiable, autonomous data-science agents. We releaseDSAgentBenchat https://github.com/vis-nlp/DSAgentBench.

View arXiv pageView PDFGitHub2Add to collection

Get this paper in your agent:

hf papers read 2608\.10366

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.10366 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.10366 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.10366 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.