@rohanpaul_ai: Long-horizon agent reliability has not arrived yet with better models. On WeaveBench's 114 hybrid GUI-CLI tasks, the be…
Summary
This paper argues that long-horizon AI agent reliability is lacking despite better models, as seen on WeaveBench with only 41.2% pass rate, and proposes the LongHorizon-Harness to manage task state for improved performance.
View Cached Full Text
Cached at: 08/29/26, 08:09 PM
Long-horizon agent reliability has not arrived yet with better models.
On WeaveBench’s 114 hybrid GUI-CLI tasks, the best officially reported pass rate is 41.2%.
Current models can handle local steps, but need explicit audited state to keep the full task on track.
Long-horizon agents can solve individual steps and still fail because history becomes unreliable about what is finished, what failed, and what remains.
LongHorizon-Harness treats this as a task-state problem.
this paper argues that reliability over that span depends as much on the harness around the model as on the model itself.
Rohan Paul (@rohanpaul_ai): “Long-horizon tasks are still a joke. They do not work, and I do not care what anybody says. Do not show me a stupid evaluation. Do not tell me about some dumb script you ran for 48 hours. Long-horizon tasks are not handled well. They simply do not work.”
- Chamath at Stanford
Similar Articles
@rohanpaul_ai: This paper is a brutal reality check for long-horizon AI. Give an agent a year of interconnected decisions, delayed fee…
A paper evaluates eight leading AI models on long-horizon tasks, finding that even the best-performing model achieves only 27.3% of human performance, highlighting significant limitations for dependable long-horizon AI execution.
LongHorizon-Harness: Advancing Long-Horizon Agents for Real-World Tasks
Introduces LongHorizon-Harness, a task-state management approach for long-horizon LLM agents using a Manage-Execute-Audit loop, showing consistent improvements across models and benchmarks like WeaveBench and OSWorld.
WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces
WeaveBench is a new benchmark for evaluating computer-use agents across multiple interfaces (GUI, CLI, code) in long-horizon real-world tasks. It reveals that current models achieve only 41.2% PassRate and that outcome-only grading overestimates performance, highlighting significant gaps in evaluation.
Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading
Introduces Long-Horizon-Terminal-Bench, a benchmark of 46 long-horizon terminal tasks with dense reward-based grading, evaluating AI agents on planning, long-context, and debugging. Even the strongest model achieves only 15.2% pass@1, showing significant room for improvement.
WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation
WildClawBench evaluates language and vision-language models on realistic long-horizon tasks using actual CLI environments with real tools. The benchmark reveals that even the best model achieves only 62.2% accuracy, indicating long-horizon agent evaluation remains challenging.