Long-Horizon-Terminal-Bench 2.0：自动化构建多样化任务并衡量智能体的持续进展

# Long-Horizon-Terminal-Bench 2.0: Automating Diverse Task Construction and Measuring Sustained Agent Progress
Source: [https://arxiv.org/html/2610.14146](https://arxiv.org/html/2610.14146)
1\]Tencent HY LLM Frontier 2\]University of Maryland, College Park 3\]University of Georgia 4\]Indiana University 5\]National University of Singapore 6\]Washington University in St\. Louis 7\]University of Minnesota, Twin Cities 8\]Carnegie Mellon University 9\]The Hong Kong Polytechnic University 10\]Lehigh University\\contribution†Core contribution\.\\headercontentProject page:[https://zli12321\.github\.io/LHTB/](https://zli12321.github.io/LHTB/)

Zhongzhi Li†\\daggerYucheng Shi†\\daggerRuhan WangJunyao YangChengsong HuangZhichao LiuXiyang WuAnhao LiYue YuErgan ShangFuxiao LiuTong ZhengNinghao LiuLichao SunHaitao MiLeoweiLiangAffiliation:\[Affiliation:\[Affiliation:\[Affiliation:\[Affiliation:\[Affiliation:\[Affiliation:\[Affiliation:\[Affiliation:\[Affiliation:\[Email:[zli12321@umd\.edu, zongxiali@global\.tencent\.com, zl22754@uga\.edu](mailto:[email protected],%[email protected],%[email protected])

October 9, 2026

###### Abstract

Long\-horizon terminal benchmarks measure whether an agent can sustain progress through extended workflows with process rewards, not only whether its final state passes a binary check\. We presentLong\-Horizon\-Terminal\-Bench 2\.0\(LHTB 2\.0\), an extension ofLong\-Horizon\-Terminal\-Benchthat preserves its 46 original tasks while correcting task and evaluation\-infrastructure defects and extending 31 tasks\.LHTB 2\.0contains 77 tasks across 13 domains and eight task types\. Different from prior benchmark, the extended tasks are produced through a systematic reconstruction\-task pipeline\. We start from a working scientific or engineering artifact, then use a parameterized reference implementation to generate multiple scenarios, reveal six as behavioral evidence, hold out three for evaluation, and use dense progress ladders to reward partial progress while preventing memorization of public outputs\. We also extend the evaluation horizon from 90 minutes to three hours and record verifier checkpoints every 30 minutes, enabling direct measurement of sustained progress within a run\. Across 1,145 scored model–task runs from 15 frontier models, GPT\-6\-astra leads with dense mean reward0\.690\.69and 37 tasks atR≥0\.95R\\geq 0\.95, followed by GPT\-5\.6\-sol with dense mean reward0\.670\.67and 32 tasks at the same threshold\. On the 30 reconstruction tasks, binary verifier scores average0\.2210\.221, whereas dense ladder scores average0\.6530\.653, showing that outcome\-only evaluation discards substantial intermediate progress\.LHTB 2\.0provides a broader, hardened, and process\-aware testbed for studying reliable agent execution over hours\.

## 1Introduction

AI agents built on large language models are increasingly expected to operate beyond short, self\-contained interactions: they must use tools, maintain state, recover from errors, and coordinate many dependent steps in workflows that last minutes or hours\. This shift creates two related evaluation challenges\. First, a final pass/fail outcome reveals whether an agent completed a task, but not whether it made sustained progress, stalled, or regressed over time\. Second, as agent capabilities expand into new domains, manually constructing realistic and reliable long\-horizon tasks becomes increasingly difficult to scale, and also increasingly difficult for humans to annotate, but easy to measure agents\. Evaluating the next generation of agents therefore requires both scalable task construction and process\-aware measurements that track progress throughout extended trajectories\.

Long\-Horizon\-Terminal\-Bench\(LHTB\) benchmark introduced 46 containerized tasks with dense, subtask\-based grading for these workflows[Li et al\. \[2026d\]](https://arxiv.org/html/2610.14146#bib.bib3)\. It established that agents can make meaningful partial progress even when they do not finish, but it also exposed several limitations\. First, manually curating long\-horizon tasks is expensive and difficult to scale across domains[Chu et al\. \[2026\]](https://arxiv.org/html/2610.14146#bib.bib4),[Song et al\. \[2026\]](https://arxiv.org/html/2610.14146#bib.bib5)\. Second, most existing benchmarks score only the final outcome of each run, hiding whether an agent makes sustained progress, plateaus, or regresses over time[Li et al\. \[2026a\]](https://arxiv.org/html/2610.14146#bib.bib6)\. Third, repeated evaluation revealed task\-environment and verifier\-integration defects that must be corrected before larger evaluations can be compared reliably\.

We proposeLong\-Horizon\-Terminal\-Bench 2\.0\(LHTB 2\.0\) to address the limitations of the previous benchmark\. version\. The benchmark retains the 46 original tasks, fixes task and harness defects, and adds 31 tasks, a total of 77\-task suite spanning 13 domains and eight task types\. The evaluation infrastructure now isolates checkpoint grading from the active agent, clears verifier artifacts between phases, audits successful trajectories for grader access, and retains only clean attempts\. We also extend the experiment run to three\-hour trials with verifier checkpoints every 30 minutes make it possible to measure how reward evolves within the same trajectory rather than comparing unrelated short and long runs\.

The central technique for automating long\-horizon and difficult task generation is through task reconstruction[Li et al\. \[2026b\]](https://arxiv.org/html/2610.14146#bib.bib9),[Li et al\. \[2026c\]](https://arxiv.org/html/2610.14146#bib.bib10)\. Starting from a known working, we identify parameter axes, execute a reference oracle over a scenario, publish multiple input–output examples as behavioral evidence, and withhold multiple scenarios for grading\. The agent receives the tools and evidence needed to infer the behavior, but not the reference implementation\. This procedure contributes 30 new reconstruction tasks in science domains such as geospatial analysis, document processing, life sciences, chemistry and materials, physical simulation, signals and control, security, and related domains\.

Our evaluation covers 1,145 scored model–task cells across 15 frontier models\. On a dense all\-77 basis, GPT\-6\-astra leads with mean reward0\.690\.69and 37 tasks atR≥0\.95R\\geq 0\.95, followed by GPT\-5\.6\-sol with mean reward0\.670\.67and 32 tasks at the same threshold\. GPT\-6\-astra’s lead comes almost entirely from the reconstruction split; on the released workflows GPT\-5\.6\-sol is slightly ahead\. More importantly, the 30 reconstruction tasks average only0\.2210\.221under binary verifier scoring but0\.6530\.653under dense ladder scoring\. The gap shows that binary evaluation frequently labels hours of useful progress as total failure\. The checkpoints further show that42%42\\%of solves arrive after the first hour, that a one\-hour budget would reorder the middle of the leaderboard, and that the fastest models often claim completion long before they finish\.

The new benchmark extension includes:

- •A broader and corrected long\-horizon benchmark\.We extendLong\-Horizon\-Terminal\-Benchfrom 46 to 77 tasks, covering 13 domains and eight task types while preserving the original suite\.
- •Verifying the feasibility of a systematic task\-construction pipeline\.We turn parameterized real\-world artifacts into reconstruction tasks with public behavioral evidence, hidden scenarios, anti\-memorization checks, and dense progress ladders\.
- •Process\-aware long\-horizon progress evaluation\.We increase the budget from 90 minutes to three hours and record 30\-minute checkpoints, enabling direct analysis of sustained progress, late completion, and regression\.
- •A hardened evaluation protocol\.We fix task and harness bugs, isolate verifier execution, audit grader access, and replace affected results with clean reruns\.

## 2From LHTB toLHTB 2\.0

### 2\.1Task formulation

Long\-Horizon\-Terminal\-Bench 2\.0follows the Terminal\-Bench and Harbor formulation[Merrill et al\. \[2026\]](https://arxiv.org/html/2610.14146#bib.bib1)\. Each task contains \(1\) a natural\-language instruction, \(2\) a containerized terminal environment, \(3\) task and runtime configuration, and \(4\) a deterministic verifier or simulator\. The instruction is the only specification shown to the agent\. All relevant code, data, tools, and assets are available through the terminal, and the agent may inspect files, edit programs, run commands, and evaluate intermediate outputs until it succeeds or reaches its time budget\.

The benchmark is long\-horizon by construction\. A task requires continued work over many dependent steps rather than a single patch or answer: reproducing an experiment, repairing a scientific analysis pipeline, optimizing a system, solving a campaign of game levels, or inferring the behavior of an unknown artifact\. Errors made early can propagate through later stages, and apparent completion must be checked against an end\-to\-end objective\.

### 2\.2Benchmark extension and splits

LHTB 2\.0contains all 46 tasks fromLong\-Horizon\-Terminal\-Benchand 31 additions\. We preserve the original task identities and descriptions so that the extension remains traceable to the first benchmark, while repairing environment and verifier integration defects discovered during larger evaluations\. The 77 tasks are organized into two complementary splits:

- •Released\-workflow split \(46 tasks\)\.The 46 original tasks and one additional optimization task cover audit and repair, optimization, gameplay, experiment reproduction, constraint puzzles, professional workflows, and engineering\. Their graders directly expose dense subtask or continuous rewards\.
- •Reconstruction split \(31 tasks\)\.These tasks ask an agent to reproduce the behavior of a parameterized scientific or engineering artifact from public observations\. Hidden scenarios test generalization, and a common six\-stage ladder converts verifier evidence into dense progress rewards\.

This split separates two sources of long\-horizon difficulty\. Released\-workflow tasks require navigating a large, heterogeneous project whose intended objective is stated but difficult to complete\. Reconstruction tasks make the implementation target itself partially latent: agents must infer behavior from evidence and then implement a solution that generalizes to withheld cases\.

#### Dense reward and completion\.

We use the same normalized scale for both benchmark splits\. For reconstruction tasks, the dense ladder score replaces the sparse final binary indicator in all cross\-task results\. We report meanRR, a soft pass atR≥0\.95R\\geq 0\.95, and a strict pass atR=1\.0R=1\.0\. Mean reward captures partial progress; the two thresholds distinguish nearly complete solutions from exact completion\.

### 2\.3Evaluation hardening

We extend each task run to three hours and invoke the verifier every 30 minutes to track progress\. These checkpoints can reveal agent regressions, such as overwriting previously correct work, as well as evaluation\-system defects, including stale verifier artifacts and hidden grader data leaking across phases—problems that a single end\-of\-run score would not expose\.LHTB 2\.0therefore hardens both task execution and score collection\. We fixed three harness/configuration defects that prevented tasks using separate verifier environments from producing rewards, isolated verifier phases from the active agent process tree, cleared verifier logs before resuming the agent, and audited successful trajectories for access to hidden fixtures or grader artifacts\.

### 2\.4Domains

The benchmark spans 13 domains: APEX professional workflows[Vidgen et al\. \[2026\]](https://arxiv.org/html/2610.14146#bib.bib13); earth, climate, and energy; interactive games; logic and constraint puzzles; multimodal and imaging analysis; research reproduction and machine learning; scientific computing and simulation; software and reverse engineering; systems, performance, and security; geospatial and cartographic workflows; document, OCR, and data engineering; life sciences, chemistry, and materials; and physical simulation, signals, and control\. Figure[1](https://arxiv.org/html/2610.14146#S2.F1)summarizes their distribution, while Appendix[A](https://arxiv.org/html/2610.14146#A1)provides the full 77\-task inventory\.

Figure 1:Task distribution across the 13LHTB 2\.0domains\.The 77 tasks combine the original benchmark domains with geospatial and cartographic workflows; document, OCR, and data engineering; life sciences, chemistry, and materials; and physical simulation, signals, and control\.

## 3Task Construction Examples

The original LHTB tasks were individually designed around complete workflows\. This produces realistic and varied environments, but it is costly to scale: every new task requires a plausible project, a reproducible solution, robust hidden tests, and a dense grader\.LHTB 2\.0retains that design for the released\-workflow split and introduces a complementary construction procedure that turns working domain artifacts into long\-horizon reconstruction tasks\.

\(a\) LiDAR registration and point\-cloud summaries

\(b\) Chemical reaction templates and molecular descriptors

\(c\) Acoustic beamforming and room simulation

Figure 2:Construction of reconstruction tasks\.A working reference artifact is swept over meaningful parameter axes\.NNscenarios are exposed as behavioral evidence andKKare withheld for grading\. The agent must implement a terminal program that reproduces behavior on unseen scenarios; public feedback guides debugging but does not reveal the hidden score\. The same contract applies to point clouds \(a\), molecules \(b\), and waveforms \(c\)\.### 3\.1Construction examples

Figure[2](https://arxiv.org/html/2610.14146#S3.F2)shows three instances of our task construction recipe\. The visual artifacts are the actual public evidence supplied to agents\. LiDAR point clouds expose changes in density and amplitude, molecular diagrams expose reaction and descriptor behavior, and acoustic waveforms expose changes in simulated impulse responses\. Each task uses the same public/hidden split, terminal interface, anti\-memorization gate, and checkpointed evaluation protocol\.

### 3\.2Task types

The construction pipeline broadens the benchmark, whereLHTB 2\.0contains eight task types:

Table 1:Task types inLHTB 2\.0\.Reconstruction is the new automated construction family; the remaining types preserve the heterogeneous workflow structure of the original benchmark\.
### 3\.3Quality control

Every task must satisfy three checks before inclusion\. First, the reference solution must run reproducibly and achieveR=1\.0R=1\.0on all hidden cases\. Second, the starter materials should be incomplete enough that solving the task requires genuine work, rather than a trivial patch or simply copying from what is already visible\. Third, we use DeepSeek\-V4\-Pro[DeepSeek\-AI \[2026\]](https://arxiv.org/html/2610.14146#bib.bib17)to review alignment between each task’s instructions and verifier, ensuring that the instructions provide enough information to satisfy the grader without revealing hidden answers\.

## 4Experiments

#### Harness and agents\.

We evaluate models using the Harbor framework with the Terminus\-2 agent harness[Harbor Framework Team \[2026\]](https://arxiv.org/html/2610.14146#bib.bib14), which interacts with each task environment through a single long\-horizon terminal session\.

#### Models\.

We benchmark GPT\-6\-astra[OpenAI \[2026b\]](https://arxiv.org/html/2610.14146#bib.bib15), GPT\-5\.6\-sol[OpenAI \[2026a\]](https://arxiv.org/html/2610.14146#bib.bib16), DeepSeek V4 Pro[DeepSeek\-AI \[2026\]](https://arxiv.org/html/2610.14146#bib.bib17), DeepSeek V4 Flash[DeepSeek\-AI \[2026\]](https://arxiv.org/html/2610.14146#bib.bib17), Gemini 3\.1 Pro[Google DeepMind \[2026a\]](https://arxiv.org/html/2610.14146#bib.bib18), Gemini 3\.8 Flash[Google DeepMind \[2026b\]](https://arxiv.org/html/2610.14146#bib.bib19), GLM\-5\.3[Z\.ai \[2026b\]](https://arxiv.org/html/2610.14146#bib.bib20), GLM\-5\.2[Z\.ai \[2026a\]](https://arxiv.org/html/2610.14146#bib.bib21), Kimi K3[Kimi Team \[2026\]](https://arxiv.org/html/2610.14146#bib.bib23), Kimi K2\.6[Moonshot AI \[2026\]](https://arxiv.org/html/2610.14146#bib.bib22), MiniMax M3[MiniMax \[2026\]](https://arxiv.org/html/2610.14146#bib.bib24), Qwen 3\.8 Max[Alibaba Cloud \[2026\]](https://arxiv.org/html/2610.14146#bib.bib25), Doubao Seed 2\.1 Pro[ByteDance Seed Team \[2026\]](https://arxiv.org/html/2610.14146#bib.bib26), Hunyuan hy4[Tencent \[2026\]](https://arxiv.org/html/2610.14146#bib.bib27), and Grok\-4\.6[xAI \[2026\]](https://arxiv.org/html/2610.14146#bib.bib28)\.111All models run with xhigh reasoning\.

#### Metrics\.

For each model, we report the soft pass rate atR≥0\.95R\\geq 0\.95, the strict pass rate \(the fraction of tasks with normalized reward exactly1\.01\.0\), the mean normalized reward across all tasks, the number of episodes required per task, the elapsed runtime per task, the estimated dollar cost per task\.

### 4\.1Main Results

We set the timeout limit for each task to be 3 hours for cost reasons\. Figure[3](https://arxiv.org/html/2610.14146#S4.F3)summarizes pass rate atR≥0\.95R\\geq 0\.95andR≥1\.0R\\geq 1\.0, along with mean reward across all 77 tasks\. Overall, the evaluated frontier models still struggle on long\-horizon tasks, even under a dense\-grading evaluation that gives credit for partial progress\. GPT\-6\-astra is the strongest reported model, but still resolves only 48\.1% of tasks \(37/77\) atR≥0\.95R\\geq 0\.95\. GPT\-5\.6\-sol follows at 41\.6% \(32/77\)\. Kimi K3 is a distant third at 22\.1% \(17/77\)\. A second tier includes hy4 and Gemini 3\.8 Flash at 19\.5% \(15/77\) each, followed by GLM\-5\.3 at 15\.6% \(12/77\)\. The gap between pass rate and mean reward \(0\.2\-0\.5\) for several mid\-tier systems indicates that many models make substantial partial progress on hard tasks without fully resolving them\. The results suggest that long\-horizon execution is not merely a local reasoning problem, but remains a central bottleneck even for top frontier models\.

Figure 3:There exists a gap between solve rate and process reward, where models can achieve significant progresses on overall tasks but fail most of them\.Pass rate is the fraction of the 77 tasks whose overall rewardRRmeets a thresholdτ\\tau, withR≥1\.0R\\geq 1\.0corresponding to full completion, while meanRRcaptures partial progress across all tasks\. GPT\-6\-astra achieves the strongest overall performance, resolving 48\.1% of tasks atR≥0\.95R\\geq 0\.95and 42\.9% atR=1\.0R=1\.0\.#### GPT\-6\-astra takes the lead on software reconstruction tasks\.

GPT\-6\-astra’s overall lead is concentrated in the reconstruction split\. On the 30 reconstruction tasks it reaches a mean reward of0\.930\.93and solves 24 tasks atR≥0\.95R\\geq 0\.95, ahead of GPT\-5\.6\-sol \(0\.840\.84, 18 tasks\) and far ahead of every other model \(≤0\.71\\leq 0\.71, at most 9 tasks\)\. On the 47 released\-workflow tasks, GPT\-5\.6\-sol is slightly ahead \(0\.570\.57, 14 tasks\), with GPT\-6\-astra second \(0\.540\.54, 13 tasks\) and DeepSeek V4 Flash close behind \(0\.530\.53\)\. GPT\-6\-astra is also more step\-efficient on reconstruction tasks, solving them after a median of 37 agent steps and 26 minutes, against 67 steps and 36 minutes for GPT\-5\.6\-sol\. In the first 40 steps of runs that last that long, about one third of GPT\-6\-astra’s commands write or edit files and only28%28\\%repeat an earlier command; for GPT\-5\.6\-sol the corresponding numbers are14%14\\%and50%50\\%\. On released tasks, that speed buys less: GPT\-6\-astra is close to its final reward after one hour \(0\.530\.53at 60 minutes,0\.560\.56at three hours\), while GPT\-5\.6\-sol keeps improving \(0\.530\.53to0\.580\.58\) and finishes ahead\. Over all 77 tasks GPT\-6\-astra remains ahead throughout, although GPT\-5\.6\-sol closes most of the gap by the end of the run \(Section[4\.3](https://arxiv.org/html/2610.14146#S4.SS3)\)\.

#### Open\-source models, model performance broadly tracks release date

\. Earlier releases such as Kimi K2\.6 \(April 20, 2026\) and GLM\-5\.2 \(June 16, 2026\) are surpassed by later releases such as Kimi K3 \(July 16, 2026; open weights July 27\), GLM\-5\.3 \(August 14, 2026; open weights August 28\), and Hunyuan hy4 preview \(August 28, 2026\)\.

### 4\.2Success and Failures

Figure 4:Mean reward by category and model\.Rows are the 13 categories, sorted by the mean over all models, followed by the two most common task types pooled across domains; parentheses give the number of released \(from LHTB\-1\.0\) and reconstruction tasks \(newly added\)\. Each cell averages the dense reward over the row’s tasks, counting unscored tasks as00, and the best model in each row is shown in bold\. The right\-hand columns give the mean over all models and the number of tasks that at least one model solves \(R≥0\.95R\\geq 0\.95\)\.Figure[4](https://arxiv.org/html/2610.14146#S4.F4)breaks mean reward down by category, task type, and model\.

#### Software and systems tasks are the most tractable for every model\.

Systems, performance & security has the highest mean across models \(0\.730\.73\), every software & reverse\-engineering task is solved by at least one model, and the most widely solved task,riscv\-core\-debug, is solved by 13 of the 15 models\. GPT\-6\-astra is strong but not dominant: it ranks second on software & reverse engineering, just behind Qwen 3\.8 Max \(0\.740\.74vs\.0\.750\.75\), and fourth on systems, where Kimi K3 leads \(0\.840\.84\)\. Its clearest advantages are on interactive games \(0\.790\.79against a fleet mean of0\.400\.40\) and on document, OCR & data engineering, where it solves all five tasks\.

#### Scientific tasks are among the hardest overall\.

Scientific computing & simulation \(0\.300\.30\) and multimodal & imaging analysis \(0\.310\.31\) are the two lowest\-scoring categories\. Of the 24 tasks that no model solves, 20 come from the released split and 12 are audit\-and\-repair tasks; on nine tasks, no model exceedsR=0\.2R=0\.2, and eight of those nine are audit\-and\-repair\. The main difficulty is therefore sustained debugging and repair, not a single missing final step\.

### 4\.3Solve Rate and Reward over Time and Steps

Figure 5:Solve rate and reward over the three\-hour run\.\(a\) Many solves arrive after the first hour\. Share of the 77 tasks solved by each minute; a task counts as solved at the first checkpoint whose verifier reward reaches0\.950\.95\. \(b\) Late wall\-clock solves can reflect slow steps\. The same solve curves against agent steps; each curve ends at the median number of steps the model completes in three hours\. \(c\) Mean reward keeps rising after the first hour\. \(d\) More steps do not guarantee more reward\. The same mean reward against agent steps\. Hollow markers and dotted segments mark budgets that more than20%20\\%of a model’s runs never reach\.We follow each model through the three\-hour budget in both wall\-clock time and agent steps \(Figure[5](https://arxiv.org/html/2610.14146#S4.F5)\)\. A task counts as softly solved at the first checkpoint whose verifier reward reaches0\.950\.95, which gives 189 solves over 1,143 runs and differs from the dense leaderboard by at most one task per model\.

#### The three\-hour budget changes the middle of the leaderboard\.

Across models, progress often continues well beyond the first hour: the median successful run solves its task after 57 minutes, with42%42\\%of all solves appearing after the one\-hour checkpoint and21%21\\%after the two\-hour checkpoint\. The top stabilizes early\. GPT\-6\-astra solves 27 tasks within 30 minutes and 34 of its 36 solved tasks within the first hour, with a median solve time of 24 minutes\. The middle does not\. Hunyuan hy4 \(median 121 minutes\) and DeepSeek V4 Pro \(141 minutes\) solve many tasks late, so a one\-hour evaluation would rank hy4 ninth instead of fourth and DeepSeek V4 Pro 13th instead of eighth\.

#### Step efficiency and step latency tell different stories\.

Wall\-clock time combines how many agent steps a task requires and how long each step takes\. Step latency alone varies by roughly6×6\\times, from 23 seconds per step for MiniMax M3 to 141 seconds for Hunyuan hy4; across models, agents spend4848–76%76\\%of each run waiting for the model rather than interacting with the environment \(Figure[6](https://arxiv.org/html/2610.14146#S4.F6)\)\. This changes how late solves should be interpreted\. Hy4, for example, needs a median of only 58 steps to solve a task, fewer than GPT\-5\.6\-sol’s 68, yet it completes fewer than 80 steps within the three\-hour budget\. By contrast, the most step\-hungry models are DeepSeek V4 Pro and MiniMax M3, which require medians of 147 and 218 steps to solve a task\. Under an equal\-step comparison, GPT\-6\-astra remains clearly best: it solves 22 tasks within 50 steps, compared with 14 for GPT\-5\.6\-sol and at most 10 for any other model\.

#### Fast models also overclaim completion\.

Completion claims are informative because the harness verifies them and resumes the run on failure\. GPT\-6\-astra and GPT\-5\.6\-sol make their first completion claim after only 7 and 8 minutes on median, and on unsolved runs still claim completion 24 and 17 times on average; MiniMax M3 does so 58 times\. By contrast, Hunyuan hy4, GLM\-5\.3, Grok\-4\.6, and DeepSeek V4 Pro often delay their first claim until 1\.5–2\.5 hours and average only 3–14 claims per unsolved run \(Figure[6](https://arxiv.org/html/2610.14146#S4.F6)\)\. Across the fleet,92%92\\%of completion claims occur on runs that never pass verification\.

Figure 6:Wall\-clock performance reflects both step speed and overconfident stopping\.\(a\) Some models spend much of each step waiting for the model, so they take far fewer steps within three hours\. \(b\) Fast models tend to claim completion earlier, while slower models often wait much longer before doing so\. \(c\) Many early completion claims are false alarms, especially for the most aggressive models\.

### 4\.4Time and Process Reward Analysis

30\-minute checkpoint

60\-minute checkpoint

Figure 7:Many runs continue making meaningful progress well after 30 and 60 minutes\.Each point is one scored task run: the x\-axis shows reward at the 30\-minute checkpoint \(left\) or 60\-minute checkpoint \(right\), and the y\-axis shows final reward\. The dashed diagonal marks no change\. Green points improve by more than0\.050\.05, red points decline by more than0\.050\.05, and gray points stay within that band\. In many runs, reward continues to increase after the checkpoint, and some runs newly cross the soft\-pass threshold only later\.Figure 8:Even unsolved runs continue making progress for hours\.Each curve includes only released\-task runs whose final reward remains below0\.950\.95\. Curves start at\(0,0\)\(0,0\)by definition and omit the sparsely populated 30\-minute interval\. Within each time bin, checkpoints are first averaged per task and then across tasks, so every task receives equal weight regardless of how many checkpoints it produced\.#### Process rewards let us measure partial progress during a long\-horizon run, not just the final outcome\.

Figure[7](https://arxiv.org/html/2610.14146#S4.F7)shows that many runs continue improving after the first checkpoint\. Mean reward rises from0\.290\.29at 30 minutes to0\.430\.43by the end of the run; 267 of 693 runs gain more than0\.050\.05after 30 minutes, and 60 cross the soft\-pass threshold only after that checkpoint\. Even after 60 minutes, 181 runs still gain more than0\.050\.05, and 38 later cross the threshold\.

#### Regression could exist but is not the dominant pattern\.

We find 65 model–task trajectories whose final reward is more than0\.050\.05below either the 30 or 60\-minute checkpoint; 55 continue declining later, and 54 end at the time limit\. These cases are concentrated in games or puzzles, where later actions can damage state or overwrite useful outputs\.

#### Most unsolved shows that unsolved runs still make progress\.

After removing successful runs, every model’s mean reward at 180 minutes remains above its 60\-minute level, and the stronger models maintain higher process rewards throughout \(Figure[8](https://arxiv.org/html/2610.14146#S4.F8)\)\. Many failures therefore reflect incomplete long\-horizon progress rather than immediate stagnation\.

## 5Cost Analysis

Table 2:Cost varies sharply across models, and the strongest model is by far the most expensive\.On average, models spend 262\.5 LLM calls and 94\.3 minutes per task, at an estimated $48\.4 per task\. Costs are estimated from per\-task token usage and public list prices\. Tokens/task counts both input and output tokens, and Figure[9](https://arxiv.org/html/2610.14146#S5.F9)uses the corresponding 77\-task suite totals\.†priced from the nearest listing rather than an exact match\.We report the cost estimates reconstructed from per\-task token usage and public list prices \(Table[2](https://arxiv.org/html/2610.14146#S5.T2)\), which Figure[9](https://arxiv.org/html/2610.14146#S5.F9)summarizes with side\-by\-side Pareto cost–reward frontiers under strict and soft pass criteria\. The estimated cost of running the full benchmark spans from about $285 for Hunyuan hy4 to about $30,581 for GPT\-6\-astra\. Although GPT\-6\-astra achieves the highest pass rate, it does so at by far the highest total cost\. GPT\-5\.6\-sol forms a second tier of strong but substantially cheaper performance at roughly one sixth of that cost, while Kimi K3 sits further down the cost curve with much lower spend but also a noticeably lower resolution rate\. Hunyuan hy4 anchors the low\-cost end of the Pareto frontier, while GPT\-5\.6\-sol and GPT\-6\-astra extend that frontier upward as progressively stronger but more expensive models\. Under the softR≥0\.95R\\geq 0\.95criterion, Kimi K3 joins the frontier because several of its near\-misses are counted as passes rather than perfect completions\.

#### Most spending goes to unsolved tasks and to the last hour\.

Figure[10](https://arxiv.org/html/2610.14146#S5.F10)breaks spending down by outcome and by hour\. Between83%83\\%and99%99\\%of every model’s spend goes to tasks it never solves; GPT\-6\-astra spends an average of $664 on an unsolved task and $101 on a solved one\. This is due to that every step resends a longer history, so hours one, two, and three account for18%18\\%,31%31\\%, and51%51\\%of benchmark\-wide spend, while solves are front\-loaded \(58%58\\%,21%21\\%, and21%21\\%\)\. A solve found in the third hour therefore costs about eight times as much as one found in the first \($708 against $90\)\.222Our estimates apply no prompt\-cache discount, which would cheapen the resent history, so this ratio is an upper bound\.Per solved task, costs range from $19 for Hunyuan hy4 to $838 for GPT\-6\-astra\.

Strict pass rate \(R=1\.0R=1\.0\)

Soft pass rate \(R≥0\.95R\\geq 0\.95\)

Figure 9:Better long\-horizon performance comes at sharply higher cost\.Estimated cost of the full 77\-task suite \(log scale\) against strict pass rate \(R=1\.0R=1\.0, left\) and soft pass rate \(R≥0\.95R\\geq 0\.95, right\)\. The dashed line traces the Pareto frontier, and bold labels mark models on that frontier\. Softening the pass criterion adds Kimi K3 to the frontier\. Hunyuan hy4 is the strongest low\-cost model, while GPT\-6\-astra is the strongest overall at the highest cost\.Figure 10:Most spending goes to the last hour and to tasks that never solve\.\(a\) Solves are front\-loaded, but spending is back\-loaded\. This panel shows the fleet\-wide share of estimated spend and of solves in each hour of the run; below each pair, the spend in that hour divided by the solves found in it\. \(b\) Most model spending goes to tasks that never solve\. This panel shows estimated cost per solved task for each model \(log scale\), with the share of the model’s spend that goes to tasks it never solves\. Costs use public list prices without prompt\-cache discounts\.

## 6Model\-wise Model Trajectory Analysis

Figure 11:Models use similar toolsets, but stronger runs edit more and repeat less\.\(a\) The task constrains the toolset, but models still take different paths\. Share of tool signatures \(program plus targeted script or subcommand\) that two models’ runs on the same task have in common, averaged over tasks; models are ordered by similarity\. \(b\) Higher\-reward runs tend to edit, test, and inspect more, while repeated\-command loops correlate with lower reward; program diversity matters little\. Spearman correlation of each command statistic with reward across models on the same task, averaged over tasks; the label counts tasks with a positive correlation\. The two Gemini submissions, which ran under a different harness, are excluded\.We parse every shell command in the 1,135 runs that kept a full trajectory and compare how models work out the tasks rather than how well they do, where run is described by its mix of command types\.

#### Different models use similar toolsets, but not the same commands\.

When two different models attempt the same task, their trajectories share only3%3\\%of exact commands and18%18\\%of tool signatures on average, even though their overall mixes of programs are much more similar \(cosine similarity0\.630\.63\)\. In other words, the task largely determines which tools are used, but the specific command sequence still depends strongly on the model\.

#### Simple command statistics separate successful runs from unsuccessful ones, but not stronger failures from weaker failures\.

Across all runs, trajectories that spend more commands editing files tend to score higher, while trajectories with long repeated\-command streaks tend to score lower \(Figure[11](https://arxiv.org/html/2610.14146#S6.F11)b\)\. These statistics mainly help distinguish runs that eventually solve the task from runs that do not\. However, among the 800 runs that still fail after using up the full three\-hour budget, they do not reliably indicate which runs made more progress than others\.

## 7Related Work

#### Long\-horizon terminal tasks and trajectories\.

Long\-horizon terminal benchmarks move beyond isolated code generation to tasks that require agents to inspect files, invoke tools, use software, and recover from errors over many steps\. Benchmarks such as Terminal\-Bench 2\.1, Terminal Bench 4\.0, Long\-Horizon\-Terminal\-Bench, and LongCLI\-Bench progressively raise the difficulty and horizon of these evaluations[Merrill et al\. \[2026\]](https://arxiv.org/html/2610.14146#bib.bib1),[Feng et al\. \[2026\]](https://arxiv.org/html/2610.14146#bib.bib2)\. As task horizons grow, reliable completion becomes harder[Kwa et al\. \[2026\]](https://arxiv.org/html/2610.14146#bib.bib29), and performance depends not only on the model but also on the harness that provides tools, manages context, and handles execution feedback[Yao et al\. \[2026b\]](https://arxiv.org/html/2610.14146#bib.bib30),[Hu et al\. \[2026\]](https://arxiv.org/html/2610.14146#bib.bib7)\. At the same time, longer and more realistic tasks are harder to support with high\-quality reference solutions and agent trajectories, because collecting and validating them becomes increasingly expensive and time\-consuming[Li et al\. \[2026b\]](https://arxiv.org/html/2610.14146#bib.bib9),[Chu et al\. \[2026\]](https://arxiv.org/html/2610.14146#bib.bib4)\. Progress in long\-horizon evaluation therefore depends not only on constructing harder and more diverse tasks, but also on building the solution and trajectory resources needed to understand how agents succeed or fail on them[Li et al\. \[2026a\]](https://arxiv.org/html/2610.14146#bib.bib6),[Li et al\. \[2026e\]](https://arxiv.org/html/2610.14146#bib.bib11)\.

#### Long\-horizon task evaluation and training\.

For long horizon tasks, an unsuccessful final state may still contain completed sub\-tasks and useful work\. Dense, verifiable process rewards credit intermediate achievements, while repeated checkpoints show whether progress continues, stalls, or is lost[Li et al\. \[2026d\]](https://arxiv.org/html/2610.14146#bib.bib3),[Li et al\. \[2026a\]](https://arxiv.org/html/2610.14146#bib.bib6)\. This distinguishes an agent that nearly finishes from one that makes little headway, and helps locate failures in planning, implementation, or feedback use[Li et al\. \[2026a\]](https://arxiv.org/html/2610.14146#bib.bib6)\. Intermediate rewards also address sparse credit assignment during reinforcement learning by supplying feedback before a long rollout ends[Lightman et al\. \[2023\]](https://arxiv.org/html/2610.14146#bib.bib31),[Yao et al\. \[2026a\]](https://arxiv.org/html/2610.14146#bib.bib8),[Yang et al\. \[2026\]](https://arxiv.org/html/2610.14146#bib.bib12)\. Recent work on early reward prediction reports gains in coding\-agent RL while reducing rollout cost[Yao et al\. \[2026a\]](https://arxiv.org/html/2610.14146#bib.bib8), showing the important of process rewards on long\-horizon tasks\.

## 8Conclusion

Long\-Horizon\-Terminal\-Bench 2\.0extends the original LHTB to 77 terminal tasks spanning 13 domains and is designed to evaluate long\-horizon execution\. Across 15 frontier models evaluated with a sharedterminus\-2agent, the strongest model, GPT\-6\-astra, resolves only48\.1%48\.1\\%of tasks \(37/7737/77\) atR≥0\.95R\\geq 0\.95, while only 192 of 1,145 scored runs \(16\.8%16\.8\\%\) pass overall\. At the same time, outcome\-only evaluation misses substantial progress: many runs continue improving over long horizons, process reward often rises steadily even on runs that never fully pass, and a large fraction of total cost is spent in the final hour of unsolved runs\. These results suggest that the main challenge in long\-horizon terminal tasks is not only local reasoning, but also sustaining productive work, preserving progress, and converting partial solutions into reliable completion over time\.

## References

- Alibaba Cloud \(2026\)Alibaba CloudQwen3\.8\-Max Model Info\.Note:[https://www\.alibabacloud\.com/help/en/model\-studio/qwen3\-8\-max](https://www.alibabacloud.com/help/en/model-studio/qwen3-8-max)Cited by:[§4](https://arxiv.org/html/2610.14146#S4.SS0.SSS0.Px2.p1.1)\.
- ByteDance Seed Team \(2026\)ByteDance Seed TeamSeed2\.1 officially released: advancing ai productivity\.Note:[https://seed\.bytedance\.com/en/blog/seed2\-1\-officially\-released\-advancing\-ai\-productivity](https://seed.bytedance.com/en/blog/seed2-1-officially-released-advancing-ai-productivity)Official model release announcement \(Doubao Seed 2\.1 / Seed 2\.1 Pro\)Cited by:[§4](https://arxiv.org/html/2610.14146#S4.SS0.SSS0.Px2.p1.1)\.
- Chuet al\.\(2026\)Z\. Chu, J\. Hu, X\. Jiang, P\. Zou, H\. Li, C\. Peng, P\. O’Hearn, E\. T\. Barr, M\. Harman, F\. Sarro, and H\. YeTerminalWorld: benchmarking agents on real\-world terminal tasks\.External Links:2605\.22535,[Link](https://arxiv.org/abs/2605.22535)Cited by:[§1](https://arxiv.org/html/2610.14146#S1.p2.1),[§7](https://arxiv.org/html/2610.14146#S7.SS0.SSS0.Px1.p1.1)\.
- DeepSeek\-AI \(2026\)DeepSeek\-AIDeepSeek\-v4: towards highly efficient million\-token context intelligence\.External Links:2606\.19348,[Link](https://arxiv.org/abs/2606.19348)Cited by:[§3\.3](https://arxiv.org/html/2610.14146#S3.SS3.p1.1),[§4](https://arxiv.org/html/2610.14146#S4.SS0.SSS0.Px2.p1.1)\.
- Fenget al\.\(2026\)Y\. Feng, J\. Sun, Z\. Yang, J\. Ai, C\. Li, Z\. Li, F\. Zhang, K\. He, R\. Ma, J\. Lin,et al\.Longcli\-bench: a preliminary benchmark and study for long\-horizon agentic programming in command\-line interfaces\.InFindings of the Association for Computational Linguistics: ACL 2026,pp\. 29952–29963\.Cited by:[§7](https://arxiv.org/html/2610.14146#S7.SS0.SSS0.Px1.p1.1)\.
- Google DeepMind \(2026a\)Google DeepMindGemini 3\.1 pro model card\.Note:[https://deepmind\.google/models/model\-cards/gemini\-3\-1\-pro/](https://deepmind.google/models/model-cards/gemini-3-1-pro/)Cited by:[§4](https://arxiv.org/html/2610.14146#S4.SS0.SSS0.Px2.p1.1)\.
- Google DeepMind \(2026b\)Google DeepMindGemini 3\.8 Flash Model Card\.Note:[https://deepmind\.google/models/model\-cards/gemini\-3\-8\-flash/](https://deepmind.google/models/model-cards/gemini-3-8-flash/)Cited by:[§4](https://arxiv.org/html/2610.14146#S4.SS0.SSS0.Px2.p1.1)\.
- Harbor Framework Team \(2026\)Harbor: A framework for evaluating and optimizing agents and models in container environmentsExternal Links:[Document](https://dx.doi.org/10.5281/zenodo.20953922),[Link](https://doi.org/10.5281/zenodo.20953922)Cited by:[§4](https://arxiv.org/html/2610.14146#S4.SS0.SSS0.Px1.p1.1)\.
- Huet al\.\(2026\)H\. Hu, Q\. Zhang, S\. Yu, Z\. Chen, T\. Luo, C\. Fang, Z\. Chen, and L\. XiaoBeyond the model: demystifying harness effects in software engineering agents\.External Links:2609\.32459,[Link](https://arxiv.org/abs/2609.32459)Cited by:[§7](https://arxiv.org/html/2610.14146#S7.SS0.SSS0.Px1.p1.1)\.
- Kimi Team \(2026\)Kimi TeamKimi K3: Open Frontier Intelligence\.External Links:2607\.24653,[Link](https://arxiv.org/abs/2607.24653)Cited by:[§4](https://arxiv.org/html/2610.14146#S4.SS0.SSS0.Px2.p1.1)\.
- Kwaet al\.\(2026\)T\. Kwa, B\. West, J\. Becker, A\. Deng, K\. Garcia, M\. Hasin, S\. Jawhar, M\. Kinniment, N\. Rush, S\. V\. Arx, R\. Bloom, T\. Broadley, H\. Du, B\. Goodrich, N\. Jurkovic, L\. H\. Miles, S\. Nix, T\. Lin, C\. Painter, N\. Parikh, D\. Rein, L\. J\. K\. Sato, H\. Wijk, D\. M\. Ziegler, E\. Barnes, and L\. ChanMeasuring ai ability to complete long software tasks\.External Links:2503\.14499,[Link](https://arxiv.org/abs/2503.14499)Cited by:[§7](https://arxiv.org/html/2610.14146#S7.SS0.SSS0.Px1.p1.1)\.
- Liet al\.\(2026a\)Y\. Li, W\. Yang, H\. Tan, X\. Huang, Z\. Chen, Z\. Li, B\. Chen, S\. Lei, H\. Zhu, H\. Tian, F\. Sun, X\. Cai, and J\. WangBeyond final scores: a systematic evaluation of agents for long\-horizon ai research and development\.External Links:2608\.13417,[Link](https://arxiv.org/abs/2608.13417)Cited by:[§1](https://arxiv.org/html/2610.14146#S1.p2.1),[§7](https://arxiv.org/html/2610.14146#S7.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2610.14146#S7.SS0.SSS0.Px2.p1.1)\.
- Liet al\.\(2026b\)Z\. Li, Y\. Shi, Z\. Li, R\. Wang, A\. Li, Z\. Huang, J\. Yang, L\. Ke, N\. Liu, H\. Mi, and L\. LiangRecursive synthesis for long\-horizon terminal tasks\.External Links:2608\.05466,[Link](https://arxiv.org/abs/2608.05466)Cited by:[§1](https://arxiv.org/html/2610.14146#S1.p4.1),[§7](https://arxiv.org/html/2610.14146#S7.SS0.SSS0.Px1.p1.1)\.
- Liet al\.\(2026c\)Z\. Li, Y\. Shi, Z\. Li, J\. Yang, R\. Wang, Y\. Wang, J\. Huang, J\. Yu, N\. Liu, H\. Mi, and L\. LiangSelf\-supervised scaling of terminal environments for scientific domains\.External Links:2610\.02710,[Link](https://arxiv.org/abs/2610.02710)Cited by:[§1](https://arxiv.org/html/2610.14146#S1.p4.1)\.
- Liet al\.\(2026d\)Z\. Li, Z\. Li, Y\. Shi, R\. Wang, J\. Yang, Z\. Liu, X\. Wu, A\. Li, Y\. Yu, N\. Liu, L\. Sun, H\. Mi, and L\. LiangLong\-horizon\-terminal\-bench: testing the limits of agents on long\-horizon terminal tasks with dense reward\-based grading\.External Links:2607\.08964,[Link](https://arxiv.org/abs/2607.08964)Cited by:[§1](https://arxiv.org/html/2610.14146#S1.p2.1),[§7](https://arxiv.org/html/2610.14146#S7.SS0.SSS0.Px2.p1.1)\.
- Liet al\.\(2026e\)Z\. Li, Y\. Shi, Z\. Li, J\. Yang, R\. Wang, C\. Huang, F\. Liu, H\. Mi, J\. Boyd\-Graber, and LeoweiLiangScaling trajectories for complex tasks through recursive self\-rewrite\.External Links:2610\.02826,[Link](https://arxiv.org/abs/2610.02826)Cited by:[§7](https://arxiv.org/html/2610.14146#S7.SS0.SSS0.Px1.p1.1)\.
- Lightmanet al\.\(2023\)H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. CobbeLet’s verify step by step\.External Links:2305\.20050,[Link](https://arxiv.org/abs/2305.20050)Cited by:[§7](https://arxiv.org/html/2610.14146#S7.SS0.SSS0.Px2.p1.1)\.
- Merrillet al\.\(2026\)M\. A\. Merrill, A\. G\. Shaw, N\. Carlini, B\. Li, H\. Raj, I\. Bercovich, L\. Shi, J\. Y\. Shin, T\. Walshe, E\. K\. Buchanan,et al\.Terminal\-bench: benchmarking agents on hard, realistic tasks in command line interfaces\.arXiv preprint arXiv:2601\.11868\.Cited by:[§2\.1](https://arxiv.org/html/2610.14146#S2.SS1.p1.1),[§7](https://arxiv.org/html/2610.14146#S7.SS0.SSS0.Px1.p1.1)\.
- MiniMax \(2026\)MiniMaxMiniMax sparse attention\.External Links:2606\.13392,[Link](https://arxiv.org/abs/2606.13392)Cited by:[§4](https://arxiv.org/html/2610.14146#S4.SS0.SSS0.Px2.p1.1)\.
- Moonshot AI \(2026\)Moonshot AIKimi k2\.6: from code to creation, from one to many\.Note:[https://www\.kimi\.com/ai\-models/kimi\-k2\-6](https://www.kimi.com/ai-models/kimi-k2-6)Official Moonshot AI model release page \(Kimi K2\.6\)Cited by:[§4](https://arxiv.org/html/2610.14146#S4.SS0.SSS0.Px2.p1.1)\.
- OpenAI \(2026a\)OpenAIGPT\-5\.6 Sol Model\.Note:[https://developers\.openai\.com/api/docs/models/gpt\-5\.6\-sol](https://developers.openai.com/api/docs/models/gpt-5.6-sol)Cited by:[§4](https://arxiv.org/html/2610.14146#S4.SS0.SSS0.Px2.p1.1)\.
- OpenAI \(2026b\)OpenAIGPT\-6 Astra Model\.Note:[https://developers\.openai\.com/api/docs/models/gpt\-6\-astra](https://developers.openai.com/api/docs/models/gpt-6-astra)Cited by:[§4](https://arxiv.org/html/2610.14146#S4.SS0.SSS0.Px2.p1.1)\.
- Songet al\.\(2026\)X\. Song, H\. Chang, G\. Dong, Y\. Zhu, J\. Wen, and Z\. DouEnvScaler: scaling tool\-interactive environments for llm agent via programmatic synthesis\.External Links:2601\.05808,[Link](https://arxiv.org/abs/2601.05808)Cited by:[§1](https://arxiv.org/html/2610.14146#S1.p2.1)\.
- Tencent \(2026\)TencentTencent Releases and Open\-Sources Tencent Hy4 Preview\.Note:[https://www\.tencent\.com/tencent\-releases\-and\-open\-sources\-tencent\-hy4\-preview/](https://www.tencent.com/tencent-releases-and-open-sources-tencent-hy4-preview/)Cited by:[§4](https://arxiv.org/html/2610.14146#S4.SS0.SSS0.Px2.p1.1)\.
- Vidgenet al\.\(2026\)B\. Vidgen, A\. Mann, A\. Fennelly, J\. W\. Stanly, L\. Rothman, M\. Burstein, J\. Benchek, D\. Ostrofsky, A\. Ravichandran, D\. Sur, N\. Venugopal, A\. Hsia, I\. Robinson, C\. Huang, O\. Varones, D\. Khan, M\. Haines, A\. Bridges, J\. Boyle, K\. Twist, Z\. Richards, C\. Mahapatra, B\. Foody, and O\. NitskiAPEX\-agents\.External Links:2601\.14242,[Link](https://arxiv.org/abs/2601.14242)Cited by:[§2\.4](https://arxiv.org/html/2610.14146#S2.SS4.p1.1)\.
- xAI \(2026\)xAIIntroducing Grok 4\.6\.Note:[https://x\.ai/news/grok\-4\-6](https://x.ai/news/grok-4-6)Cited by:[§4](https://arxiv.org/html/2610.14146#S4.SS0.SSS0.Px2.p1.1)\.
- Yanget al\.\(2026\)J\. Yang, Y\. Shi, Z\. Li, R\. Wang, Z\. Li, H\. Mi, and L\. LiangT1: terminal agent reinforcement learning for long\-horizon tasks\.External Links:2609\.11042,[Link](https://arxiv.org/abs/2609.11042)Cited by:[§7](https://arxiv.org/html/2610.14146#S7.SS0.SSS0.Px2.p1.1)\.
- Yaoet al\.\(2026a\)J\. Yao, S\. Zeng, S\. Feng, Z\. Fan, B\. Zhu, and Y\. TsvetkovBefore the rollout ends: early terminal reward prediction for long\-horizon coding agents\.External Links:2609\.31995,[Link](https://arxiv.org/abs/2609.31995)Cited by:[§7](https://arxiv.org/html/2610.14146#S7.SS0.SSS0.Px2.p1.1)\.
- Yaoet al\.\(2026b\)Y\. Yao, X\. Tan, C\. Liu, Y\. Li, Z\. Wang, W\. Yu, Z\. Tan, Y\. Tian, G\. Zhao, L\. Sun, X\. Zhang, and T\. YangHarness\-bench: measuring harness effects across models in realistic agent workflows\.External Links:2605\.27922,[Link](https://arxiv.org/abs/2605.27922)Cited by:[§7](https://arxiv.org/html/2610.14146#S7.SS0.SSS0.Px1.p1.1)\.
- Z\.ai \(2026a\)Z\.aiGLM\-5\.2: Built for Long\-Horizon Tasks\.Note:[https://z\.ai/blog/glm\-5\.2](https://z.ai/blog/glm-5.2)Cited by:[§4](https://arxiv.org/html/2610.14146#S4.SS0.SSS0.Px2.p1.1)\.
- Z\.ai \(2026b\)Z\.aiGLM\-5\.3: Frontier Coding with Emergent Cyber Capabilities\.Note:[https://z\.ai/blog/glm\-5\.3](https://z.ai/blog/glm-5.3)Cited by:[§4](https://arxiv.org/html/2610.14146#S4.SS0.SSS0.Px2.p1.1)\.

## Appendix ATask Inventory

Table[3](https://arxiv.org/html/2610.14146#A1.T3)lists all 77 benchmark tasks using the 13\-domain taxonomy used throughout the paper\. The first 46 rows preserve the original LHTB suite; the remaining 31 rows are the LHTB 2\.0 additions, comprising 30 reconstruction tasks and one optimization task\. Difficulty is derived from dense mean reward across model results in Figure[3](https://arxiv.org/html/2610.14146#S4.F3): tasks with mean reward at least0\.50\.5are labeled*Easy*, and the rest are labeled*Hard*\.

Table 3:Task list forLong\-Horizon\-Terminal\-Bench 2\.0\.Each row shows a task ID, its paper\-level category, a one\-sentence description, and a coarse difficulty label derived from average task reward in Figure[3](https://arxiv.org/html/2610.14146#S4.F3)\(*Easy*if mean reward≥0\.5\\geq 0\.5, otherwise*Hard*\)\.Task IDCategoryDescriptionDifficulty2048Interactive gamesPlay 2048 turn\-by\-turn via a terminal game server, merging tiles to reach the highest tile, scored by replaying recorded moves in a fresh seeded engine\.Hardalp\-paper\-reproductionResearch reproduction & MLReproduce a calibrated ALP floating\-point compression experiment from the paper’s method and sample columns\.Hardapex\-ib244\-matterAPEX professional workflowsAdvise on taking EdTech firm KSchool private across 38 reactive stages, rebuilding LBO/DCF/comps models for exact figures\.Easyapex\-investment\-banking\-matterAPEX professional workflowsWork a private\-equity take\-private of Planet Fitness through 19 reactive stages, rebuilding an LBO model and answering exact figures\.Hardapex\-law433\-matterAPEX professional workflowsAct as outside counsel across four legal matters and 70 reactive stages over ~600 documents, submitting deterministically\-graded answers\.Easyapex\-management\-consulting\-matterAPEX professional workflowsWork two management\-consulting client matters across 33 reactive stages, rebuilding business\-case and cost\-savings models for exact answers\.Hardapex\-openroad\-ibex\-signoffSoftware & reverse engineeringClose RTL\-to\-GDSII signoff on the ibex CPU with OpenROAD/sky130 across reactive stages until timing, DRC, LVS, area, and power pass\.Hardaudio\-visual\-event\-alignmentMultimodal & imaging analysisRepair an audio\-video sync audit that detects visual events, audio onsets, dropped frames, and per\-clip synchronization offsets\.Hardchess\-mateInteractive gamesPlay chess as White against a bundled superhuman Stockfish engine, scored on how long the agent survives before being checkmated\.Hardclimate\-netcdf\-extreme\-event\-auditEarth, climate & energyRepair a climate audit pipeline computing gridded heatwave, precipitation, drought, anomaly, and index metrics over NetCDF datasets\.Hardcommit0\-multilib\-tddSoftware & reverse engineeringReimplement three whole Python libraries from stubs with no provided tests, writing your own suite, graded on hidden real test suites\.Harddicom\-radiology\-auditMultimodal & imaging analysisAudit synthetic DICOM CT series geometry, HU conversion, windowing, masks, and lesion measurements\.Harddocument\-table\-layout\-reconstructionMultimodal & imaging analysisRepair a scanned\-document pipeline that detects table grids, assigns noisy OCR tokens to cells, and emits CSV/JSON/Markdown tables\.Hardduckdb\-optimizer\-closureSystems, performance & securityIteratively patch DuckDB’s query optimizer to speed up TPC\-H while preserving correctness\.Hardepa\-swmm\-stormwater\-regression\-auditEarth, climate & energyAudit EPA SWMM stormwater hydraulic and hydrologic simulations against official solver outputs\.Hardepidemic\-inverse\-control\-auditScientific computing & simulationFit and control a multi\-city, age\-structured SEIR\-H\-ICU epidemic model\.Hardfoldseek\-paper\-reproductionResearch reproduction & MLReproduce a calibrated Foldseek\-style structural protein search experiment from the Nature Biotechnology paper\.Hardgdal\-proj\-raster\-regressionEarth, climate & energyReproduce GDAL/PROJ raster reprojection, coordinate transform, point sampling, and zonal\-statistics regression checks\.Hardgenerals\-bot\-arenaSoftware & reverse engineeringDevelop a bot for a Generals\.io\-style grid strategy game by iteratively playing matches and refining its strategy\.Easygrammar\-fuzz\-coverage\-huntSystems, performance & securityIteratively build a grammar\-guided fuzzer to maximize code coverage of target parsers\.Easygreat\-expectations\-auditSoftware & reverse engineeringBuild a Great Expectations\-based multi\-table data\-quality and reconciliation audit pipeline\.Hardlangchain\-version\-migrationSoftware & reverse engineeringMigrate a legacy LangChain RAG/router app across a breaking API change with hard\-fail compatibility gates\.Hardmaterials\-phase\-diagram\-auditScientific computing & simulationRepair a materials phase\-diagram audit reconstructing phase boundaries, invariant reactions, lever\-rule fractions, and convex\-hull stability\.Hardmatpower\-opf\-regressionEarth, climate & energyReproduce MATPOWER/PYPOWER power\-flow and optimal\-power\-flow regression checks with grid\-constraint diagnostics\.Hardmicroscopy\-cell\-count\-qc\-auditMultimodal & imaging analysisRepair a microscopy QC audit that segments synthetic cell imagery, flags focus/artifact failures, and reports per\-tile channel counts\.Hardmodflow6\-groundwater\-regression\-auditEarth, climate & energyAudit USGS MODFLOW 6 groundwater simulations against official solver outputs\.Hardnbody\-accel\-iterativeScientific computing & simulationIteratively optimize an N\-body gravitational simulation from O\(N^2\) to O\(N log N\) with correctness verification at multiple scales\.Easynrel\-pysam\-hybrid\-renewables\-auditEarth, climate & energyAudit renewable generation and storage dispatch against NREL PySAM official PVWatts outputs\.Hardopensees\-seismic\-structural\-regression\-auditScientific computing & simulationAudit seismic structural\-dynamics results against official OpenSeesPy finite\-element solver outputs\.Hardpoc\-exploit\-craftSystems, performance & securityGenerate proof\-of\-concept inputs that trigger specific vulnerability classes using sanitizer feedback\.Easyriscv\-core\-debugSoftware & reverse engineeringLocalize and fix undisclosed injected bugs in an out\-of\-order RISC\-V SystemVerilog CPU until simulation, compliance, and formal checks pass\.Easyrobotics\-slam\-benchmark\-repairResearch reproduction & MLRepair a SLAM benchmark audit that composes SE\(2\) poses, evaluates loop closures, reconstructs landmark maps, and reports trajectory consistency\.Hardrush\_hour\_campaignLogic & constraint puzzlesSolve a four\-stage 6x6 Rush Hour campaign by hand, submitting legal move routes under length caps\.Hardsatellite\-flood\-change\-detection\-auditMultimodal & imaging analysisRepair a satellite flood change\-detection audit that compares before/after imagery, masks clouds, and reports per\-tile flood extent\.Hardscientific\-figure\-data\-reconstructionMultimodal & imaging analysisRepair a pipeline that reconstructs plotted series data from scientific figure images, captions, and partial CSV references\.Hardsnake\_maze\_campaignInteractive gamesPlay a deterministic hard Snake variant, eating seeded apples and surviving growing obstacles as long as possible, scored by replaying the move log\.HardsokobanLogic & constraint puzzlesSolve a ramp of Sokoban puzzles turn\-by\-turn through a game server, advancing as far as possible, scored by replaying moves in a clean engine\.Hardspice\-ephemeris\-regressionScientific computing & simulationReproduce NASA SPICE ephemeris, angle, and ground\-station visibility regression checks from pinned kernels\.Hardspot\-scheduler\-tracesSystems, performance & securityImplement and refine a cloud spot\-instance scheduling policy that minimizes cost while meeting hard deadlines\.Easysu2\-airfoil\-regressionScientific computing & simulationReproduce a hardened SU2 aerodynamic regression matrix from official test\-case references and solver history artifacts\.Hardsudoku\-recoveryLogic & constraint puzzlesRecover seven progressively corrupted hard variant\-Sudoku puzzles turn\-by\-turn, detecting subtle corrupt givens via global constraint reasoning\.Hardsuper\-marioInteractive gamesPlay Super Mario Bros turn\-by\-turn through a game server, clearing as many levels as possible, scored by replaying frame inputs in a fresh emulator\.Hardtabular\-data\-feature\-covshiftResearch reproduction & MLRecover a sparse high\-dimensional signal from low\-dimensional observations under covariate shift, with a leakage\-audited honesty gate\.Hardunison\-paper\-reproductionResearch reproduction & MLReproduce a calibrated UNISON fat\-tree network simulation experiment from the paper’s method and settings\.Hardunknown\-config\-semanticsSoftware & reverse engineeringReverse\-engineer an undocumented config format from a noisy spec and repair its normalization engine to pass hidden probes across five stages\.Hardvector\-db\-iterative\-buildSystems, performance & securityBuild an approximate nearest\-neighbor vector search service from scratch, iteratively optimizing recall and throughput\.Hardacoustic\_beamforming\_peak\_\_01Physical simulation, signals & controlReconstruct a room\-acoustics beamforming pipeline so its peak\-direction estimates match a reference simulator\.Hardajv\_dynamic\_ref\_tree\_\_05Software & reverse engineeringReconstruct AJV JSON\-schema validation behaviour over dynamic $ref trees from observed accept/reject traces\.Hardastro\_mosaic\_reprojection\_\_04Multimodal & imaging analysisReconstruct an astronomical FITS mosaic reprojection and photometry pipeline from reference outputs\.Hardchem\_reaction\_smarts\_application\_\_02Life sciences, chemistry & materialsReconstruct SMARTS reaction\-template application so product sets match a reference cheminformatics engine\.Hardcircuit\_buck\_converter\_ripple\_\_01Physical simulation, signals & controlReconstruct a buck\-converter circuit simulation reproducing reference ripple and transient measurements\.Hardclimate\_regrid\_conservation\_check\_\_06Earth, climate & energyReconstruct a NetCDF climate regridding pipeline that preserves the reference conservation diagnostics\.Hardduckdb\_recursive\_hierarchy\_report\_\_03Document, OCR & data engineeringReconstruct a DuckDB recursive\-hierarchy report matching reference aggregation output\.Hardfea\_plate\_hole\_stress\_concentration\_\_01Physical simulation, signals & controlReconstruct a finite\-element plate\-with\-hole analysis reproducing reference stress\-concentration factors\.Hardfont\_harfbuzz\_shaping\_cluster\_\_04Document, OCR & data engineeringReconstruct HarfBuzz text\-shaping cluster and glyph\-advance behaviour from observed shaping output\.Hardgdal\_antimeridian\_clip\_\_01Geospatial & cartographyReconstruct GDAL antimeridian\-crossing clip geometry matching reference vector output\.Hardgdal\_vrt\_mosaic\_transform\_\_03Geospatial & cartographyReconstruct a GDAL VRT mosaic and coordinate transform reproducing reference raster statistics\.Hardgenetic\-convergence\-testingScientific computing & simulationOptimize a stochastic convergence\-testing workflow against hidden statistical and runtime criteria\.Hardgeology\_multiwell\_correlation\_panel\_\_04Earth, climate & energyReconstruct a multi\-well log correlation panel matching reference stratigraphic picks\.Hardghostscript\_pdfa\_preflight\_repair\_\_06Document, OCR & data engineeringReconstruct a Ghostscript PDF/A preflight\-and\-repair pipeline matching reference compliance reports\.Hardhydro\_hydrograph\_response\_\_06Earth, climate & energyReconstruct a raster hydrology model reproducing reference hydrograph response curves\.Easylidar\_icp\_registration\_summary\_\_06Geospatial & cartographyReconstruct LiDAR ICP point\-cloud registration reproducing reference alignment summaries\.Hardmap\_label\_priority\_collision\_\_04Geospatial & cartographyReconstruct cartographic label placement and collision priority matching reference rendered labels\.Hardmap\_road\_class\_generalization\_\_05Geospatial & cartographyReconstruct road\-class cartographic generalization matching reference simplified geometry\.Hardmaterials\_xrd\_peak\_shift\_\_04Life sciences, chemistry & materialsReconstruct X\-ray diffraction peak\-shift analysis reproducing reference lattice measurements\.Hardmedical\_slice\_series\_reassembly\_\_03Multimodal & imaging analysisReconstruct medical slice\-series reassembly and geometry recovery matching reference volumes\.Hardnginx\_location\_regex\_tryfiles\_precedence\_\_06Systems, performance & securityReconstruct nginx location/regex/try\_files routing precedence from observed request traces\.Hardopa\_rbac\_namespace\_boundary\_\_05Systems, performance & securityReconstruct OPA/Rego RBAC namespace\-boundary policy decisions from observed allow/deny traces\.Hardopenssl\_pkits\_name\_constraints\_\_05Systems, performance & securityReconstruct OpenSSL PKITS name\-constraint certificate validation from reference verdicts\.Easypcap\_http2\_tls\_fingerprint\_\_04Systems, performance & securityReconstruct HTTP/2 and TLS fingerprint extraction from packet captures matching reference fingerprints\.Hardpdb\_multimodel\_rmsd\_\_02Life sciences, chemistry & materialsReconstruct multi\-model PDB structural RMSD computation matching reference superposition values\.Hardproteomics\_retention\_time\_alignment\_\_02Life sciences, chemistry & materialsReconstruct mass\-spectrometry retention\-time alignment matching reference peptide mappings\.Hardrobotics\_pick\_reach\_controller\_\_01Physical simulation, signals & controlReconstruct a robotic pick\-and\-reach controller reproducing reference trajectories and success criteria\.Hardsamtools\_mpileup\_consensus\_\_05Life sciences, chemistry & materialsReconstruct a samtools mpileup consensus\-calling pipeline matching reference variant output\.Hardseismo\_cross\_correlation\_lag\_\_01Physical simulation, signals & controlReconstruct seismic waveform cross\-correlation lag estimation matching reference arrival offsets\.Easysqlite\_wal\_recovery\_audit\_\_02Document, OCR & data engineeringReconstruct a SQLite WAL recovery audit reproducing reference post\-crash database state\.Hardtesseract\_table\_image\_ocr\_\_06Document, OCR & data engineeringReconstruct a Tesseract table\-image OCR pipeline matching reference extracted tables\.Hard