Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading

Hugging Face Daily Papers Papers
benchmark agents long-horizon terminal-tasks dense-reward evaluation ai

Summary

Introduces Long-Horizon-Terminal-Bench, a benchmark of 46 long-horizon terminal tasks with dense reward-based grading, evaluating AI agents on planning, long-context, and debugging. Even the strongest model achieves only 15.2% pass@1, showing significant room for improvement.

AI agents have become capable of autonomously completing short, well-specified tasks. However, existing terminal benchmarks largely focus on simple problems that finish within minutes and are evaluated only by their final outcome. This setup overlooks intermediate progress and partial solutions, yielding sparse reward signals and an incomplete picture of agent capability. We introduce Long-Horizon-Terminal-Bench, a terminal benchmark of 46 long-horizon tasks spanning nine categories, including experiment reproduction, software engineering, multimodal analysis, interactive games, and scientific computing. Each task follows a Terminal-Bench-style setup with a reference solution or simulation engine, but is further decomposed into fine-grained graded subtasks. This design enables dense intermediate rewards and partial credit, allowing evaluation to capture not only whether an agent reaches the final goal, but also how far it progresses on open-ended workflows. Tasks in Long-Horizon-Terminal-Bench typically require hundreds of episodes and minutes to hours of execution, stressing long-horizon planning, long-context management, and iterative debugging rather than one-shot problem solving. We evaluate 15 frontier models and find that agents consume on average 9.9M tokens per task, with roughly 231 episodes and 85.3 minutes of execution time per run, making Long-Horizon-Terminal-Bench more demanding than prior terminal-based benchmarks. Even the strongest tested model achieves 15.2% pass@1 at a partial-reward threshold of 0.95 and 10.9% at a perfect-reward threshold of 1.0, while the mean pass rate across models is 4.3% and 1.7% under the two thresholds, respectively. These results reveal headroom for improvement. We further analyze failure modes and error patterns, and release Long-Horizon-Terminal-Bench to support future progress on long-horizon terminal agents.
Original Article
View Cached Full Text

Cached at: 07/13/26, 07:49 AM

Paper page - Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading

Source: https://huggingface.co/papers/2607.08964 Published on Jul 9

·

Submitted byhttps://huggingface.co/zli12321

LZXon Jul 13

#1 Paper of the day Authors:

,

,

,

,

,

,

,

,

,

,

,

Abstract

AIagentshavebecomecapableofautonomouslycompletingshort,well-specifiedtasks.However,existingterminalbenchmarkslargelyfocusonsimpleproblemsthatfinishwithinminutesandareevaluatedonlybytheirfinaloutcome.Thissetupoverlooksintermediateprogressandpartialsolutions,yieldingsparserewardsignalsandanincompletepictureofagentcapability.WeintroduceLong-Horizon-Terminal-Bench,aterminalbenchmarkof46long-horizontasksspanningninecategories,includingexperimentreproduction,softwareengineering,multimodalanalysis,interactivegames,andscientificcomputing.EachtaskfollowsaTerminal-Bench-stylesetupwithareferencesolutionorsimulationengine,butisfurtherdecomposedintofine-grainedgradedsubtasks.Thisdesignenablesdenseintermediaterewardsandpartialcredit,allowingevaluationtocapturenotonlywhetheranagentreachesthefinalgoal,butalsohowfaritprogressesonopen-endedworkflows.TasksinLong-Horizon-Terminal-Benchtypicallyrequirehundredsofepisodesandminutestohoursofexecution,stressinglong-horizonplanning,long-contextmanagement,anditerativedebuggingratherthanone-shotproblemsolving.Weevaluate15frontiermodelsandfindthatagentsconsumeonaverage9.9Mtokenspertask,withroughly231episodesand85.3minutesofexecutiontimeperrun,makingLong-Horizon-Terminal-Benchmoredemandingthanpriorterminal-basedbenchmarks.Eventhestrongesttestedmodelachieves15.2%[email protected]%ataperfect-rewardthresholdof1.0,whilethemeanpassrateacrossmodelsis4.3%and1.7%underthetwothresholds,respectively.Theseresultsrevealheadroomforimprovement.Wefurtheranalyzefailuremodesanderrorpatterns,andreleaseLong-Horizon-Terminal-Benchtosupportfutureprogressonlong-horizonterminalagents.

View arXiv pageView PDFProject pageGitHub25Add to collection

Get this paper in your agent:

hf papers read 2607\.08964

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.08964 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.08964 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.08964 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation

Hugging Face Daily Papers

WildClawBench evaluates language and vision-language models on realistic long-horizon tasks using actual CLI environments with real tools. The benchmark reveals that even the best model achieves only 62.2% accuracy, indicating long-horizon agent evaluation remains challenging.

@dair_ai: Outstanding paper on long-horizon agents. (bookmark it) Similar to humans, how do you make agents persist on a difficul…

X AI KOLs Following

AutoLab is a new benchmark evaluating 17 frontier models on 36 expert-curated long-horizon tasks (system optimization, model development, CUDA kernels, puzzles), finding that persistence—not initial attempt quality—is the dominant predictor of success. Claude-opus-4.6 led all categories, while most other models terminated prematurely or exhausted budgets with minimal progress.

LongDS-Bench: On the Failure of Long-Horizon Agentic Data Analysis

arXiv cs.LG

Introduces LongDS, a benchmark for evaluating LLM agents on long-horizon, multi-turn data analysis tasks. Evaluations show that even the best models achieve only 48.45% accuracy, with performance dropping sharply over turns, highlighting that maintaining analytical state is the key bottleneck.

Running the Gauntlet: Re-evaluating the Capabilities of Agents Beyond Familiar Environments

Hugging Face Daily Papers

GauntletBench is a new web-based benchmark that evaluates AI agents on challenging scenarios focusing on temporal perception, graphical understanding, and 3D reasoning. Results show state-of-the-art agents achieve only 19.1% success rate compared to over 80% for non-expert humans, highlighting significant limitations in current agentic systems.