CEO-Bench: Can Agents Play the Long Game?
Summary
CEO-Bench introduces a simulation benchmark that evaluates language model agents' ability to manage a startup over 500 days, testing long-term planning, noise handling, adaptability, and multi-task coordination. Results show that even the strongest models struggle, with only Claude Opus 4.8 and GPT-5.5 finishing above the starting balance.
View Cached Full Text
Cached at: 06/18/26, 03:55 AM
Paper page - CEO-Bench: Can Agents Play the Long Game?
Source: https://huggingface.co/papers/2606.18543
Abstract
CEO-Bench evaluates language model agents’ ability to manage a simulated startup over 500 days, testing their proficiency in long-term planning, noise handling, adaptability, and multi-task coordination through a Python interface.
Language model agentsare becoming proficient executors at isolated, short-horizon tasks such as software engineering and customer service. Yet real-world challenges require a combination of sophisticated skills that remain largely untested in agents: (1) navigatinglong horizonsamiduncertainty; (2) acquiring information innoisy environments; (3) adapting to achanging world; (4) orchestrating multiple moving parts toward a coherent goal. We introduce CEO-Bench, which evaluates these capabilities together by simulating a representative real-world task: operating a startup for 500 days. An agent manages pricing, marketing, budgeting, and many other aspects of a fictional company through aprogrammable Python interface, operating in the same environment and facing the same challenges as a human CEO. Success demands analyzing noisy, interconnectedbusiness databases, translating signals into sound strategy, and coordinating many decisions with programming. The strongest agents write sophisticated code that simulatescustomer cohortsto forecast future cash and minesnegotiation historyto uncover hidden customer preferences. Even so, most state-of-the-art models struggle in this environment. Only Claude Opus 4.8 and GPT-5.5 finish above the $1M starting balance, and neither consistently turns a profit. CEO-Bench takes a first step toward measuring the intelligence required to drive sustained,adaptive progressover time.
View arXiv pageView PDFProject pageGitHub1Add to collection
Get this paper in your agent:
hf papers read 2606\.18543
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.18543 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.18543 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.18543 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows
StartupBench introduces a benchmark for evaluating general-purpose AI agents on real-world startup workflows, revealing that top models complete only about 30% of tasks due to gaps in complex instruction following and domain-specific expertise.
WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation
WildClawBench evaluates language and vision-language models on realistic long-horizon tasks using actual CLI environments with real tools. The benchmark reveals that even the best model achieves only 62.2% accuracy, indicating long-horizon agent evaluation remains challenging.
PlanBench-XL: Evaluating Long-Horizon Planning of LLM Tool-Use Agents in Large-Scale Tool Ecosystems
PlanBench-XL is a new benchmark that evaluates LLM agents' ability to plan and adapt in large tool ecosystems with limited visibility and dynamic disruptions. Experiments show GPT-5.4 achieves only 51.9% accuracy in block-free settings and collapses to 11.36% under severe blocking, highlighting significant challenges in long-horizon planning.
Can LLMs Be CEOs? Benchmarking Strategic Resource Reallocation with Multi-Role Agent Simulation
This paper introduces CEO-Bench, a multi-agent benchmark for evaluating LLMs on CEO-level strategic resource reallocation, revealing systematic failure modes and a structural integration–boldness tradeoff.
@dair_ai: Outstanding paper on long-horizon agents. (bookmark it) Similar to humans, how do you make agents persist on a difficul…
AutoLab is a new benchmark evaluating 17 frontier models on 36 expert-curated long-horizon tasks (system optimization, model development, CUDA kernels, puzzles), finding that persistence—not initial attempt quality—is the dominant predictor of success. Claude-opus-4.6 led all categories, while most other models terminated prematurely or exhausted budgets with minimal progress.