Paper page - StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows

Source: https://huggingface.co/papers/2608.17800 Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

StartupBench evaluates end-to-end AI agents on real-world startup workflows and reveals that even top models complete only about 30% of tasks, highlighting gaps in instruction following and domain expertise.

Recent advances inLarge Language Models(LLMs) andagentshave substantially improved the ability of AI systems to execute complex tasks. Yet existing benchmarks largely rely on researcher-selected tasks, leaving uncertain whether such progress extends to the work that real-world users actually demand from AI systems. We introduceStartupBench, anE2E agent benchmarkgrounded in market-validated AI startup products. Rather than defining tasks from pre-defined assumptions about useful agent capabilities, we systematically study AI products with demonstrated adoption, together with their product workflows and users, to identify real-world tasks for which AI has established practical demand across diverse professional domains. We translate these workflows into complete deliverable-oriented tasks and evaluate them with fine-grained rubrics capturing their complex requirements. Across representative models evaluated under a unifiedagent harness, even the strongest model successfully completes only approximately 30\% ofStartupBench, despite making substantial partial progress on many tasks. Further analysis identifies aspects likecomplex instruction followinganddomain-specific expertiseas major sources of failure. Our results reveal that many market-validated workflows remain beyond the reliable capabilities of current general-purposeagents, establishingStartupBenchas an empirical measure of progress toward E2E completions of real-world user tasks.

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2608\.17800

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.17800 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.17800 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.17800 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.