Benchmarking the Residual: What Long-Horizon Evaluations Add Beyond Matched Short-Task Performance

# What Long-Horizon Evaluations Add Beyond Matched Short-Task Performance
Source: [https://arxiv.org/html/2607.27283](https://arxiv.org/html/2607.27283)
## Benchmarking the Residual: What Long\-Horizon Evaluations Add Beyond Matched Short\-Task Performance

Chao Peng Zhiheng Lyu11footnotemark:1Peijie Dong11footnotemark:1Hande Dong22footnotemark:2Qiang Lin Tencent chao\.peng@acm\.org, zl149@illinois\.edu, peytondong@tencent\.com handedong@tencent\.com, cheaterlin@tencent\.comThese authors contributed equally\.Corresponding authors:chao\.peng@acm\.organdhandedong@tencent\.com\.

###### Abstract

Long\-horizon benchmarks often show that agents fail more as tasks become longer\. This observation is useful for deployment, but it does not by itself explain why failure occurs\. More stages create more opportunities for ordinary errors to compound; longer tasks may also contain harder individual decisions or become harder as conversation history, tool outputs, and environment changes accumulate\. We use*trajectory\-induced degradation*to mean this last possibility: earlier execution makes later work harder\. When the harmful accumulation is specifically the text visible to the model, it is often called*context rot*\. In this position paper, we argue that to claim a “long\-horizon failure”, benchmarks must compare actual full\-task success against a baseline prediction built from short, individual stages\. We call the log\-ratio between this prediction and actual success the*horizon residual*\. The comparison must use the same agent configuration and specify in advance how stages, checkpoints, information, and budgets will be chosen\. The residual shows that the full rollout differs from the chosen baseline; targeted experiments are still needed to explain why\.

The key idea\.Benchmark design:Longer can mean more work, not harder work\. Benchmarks should distinguish task size from how hard each stage is and how strongly stages depend on one another\.Result interpretation:More stages create more chances to fail\. Compare actual full\-task success with a prediction built from matched short stages; the*horizon residual*summarizes the mismatch, not its cause\.

## 1Introduction

Language\-model agents are increasingly evaluated on tasks that extend beyond a single response or isolated code edit\. In software engineering, recent benchmarks range from repository\-level issue resolution\(Jimenezet al\.,[2024](https://arxiv.org/html/2607.27283#bib.bib17)\)to release\-sized evolution\(Leet al\.,[2025](https://arxiv.org/html/2607.27283#bib.bib10)\), chained package upgrades\(Lamet al\.,[2026](https://arxiv.org/html/2607.27283#bib.bib7)\), and continuous milestone execution on a persistent codebase\(Denget al\.,[2026](https://arxiv.org/html/2607.27283#bib.bib1)\)\. Related interactive benchmarks require agents to navigate websites, operate desktop applications, coordinate APIs, or conduct iterative research engineering\(Zhouet al\.,[2023](https://arxiv.org/html/2607.27283#bib.bib22); Xieet al\.,[2024](https://arxiv.org/html/2607.27283#bib.bib23); Trivediet al\.,[2024](https://arxiv.org/html/2607.27283#bib.bib24); Wijket al\.,[2024](https://arxiv.org/html/2607.27283#bib.bib25)\)\. Across these settings, success generally becomes less reliable as the amount of work, the depth of inter\-stage dependence, and the accumulated execution history increase\. This trend matters for deployment because many valuable tasks require sustained planning, tool use, state tracking, verification, and recovery over extended trajectories\.

The phrase*long horizon*, however, can refer to several different quantities:

- •the minimum number of effective actions required;
- •the number of separately verifiable milestones;
- •the depth of the dependency chain;
- •the duration of the agent’s rollout; or
- •the amount of prior history visible to the model\.

These quantities often move together in naturally collected tasks, but they need not\. A task may contain many independent operations with shallow dependencies, or only a few stages that strongly depend on earlier state\. A short task may also produce a long trace because the agent retries or explores irrelevant actions\. Treating these cases as one notion of length obscures what a benchmark actually measures\(Wanget al\.,[2026](https://arxiv.org/html/2607.27283#bib.bib3); Kwaet al\.,[2025](https://arxiv.org/html/2607.27283#bib.bib16)\)\.

A lower end\-to\-end success rate on a longer task is therefore not yet a diagnosis of a long\-horizon failure mechanism\.Even if each local stage is unchanged, requiring more stages to succeed can reduce task success through ordinary error compounding\. Longer tasks may also contain intrinsically harder local decisions, expose the agent to noisier context, or allow earlier actions to alter the state on which later decisions depend\. Existing results illustrate these possibilities: independent and continuous milestone protocols can produce sharply different outcomes\(Denget al\.,[2026](https://arxiv.org/html/2607.27283#bib.bib1)\); agents can degrade when they inherit their own prior code changes\(Jinet al\.,[2026](https://arxiv.org/html/2607.27283#bib.bib8)\); and controlled history manipulations show sensitivity to erroneous prior outputs at a matched evaluation turn\(Sinhaet al\.,[2026](https://arxiv.org/html/2607.27283#bib.bib2)\)\. Because these factors often vary together, a raw performance curve can establish a deployment limitation without identifying its source\.

This distinction separates two legitimate purposes of evaluation\. A deployment benchmark asks whether the complete system succeeds under realistic budgets and interaction rules\. A diagnostic benchmark asks which controlled contrast explains behavior that a simpler model of local competence does not\. The first purpose prioritizes ecological validity; the second requires replayable states, explicit interventions, and assumptions that can be audited\. Neither replaces the other\. In practice, however, long\-horizon results are often given diagnostic readings that the deployment observation alone cannot license\.

We therefore propose a simple diagnostic comparison\. First, run the same agent on short, verifiable stages from declared checkpoints\. Next, combine those stage results into a prediction for the full task\. Finally, compare the prediction with success in a natural end\-to\-end rollout\. We use*counterfactual*for this pre\-specified alternative evaluation protocol, not for an individual\-level causal claim\.111The resulting quantities compare evaluation protocols; they are not individual\-level potential\-outcomes counterfactuals\.The horizon residual summarizes how far the natural rollout departs from the checkpoint\-based prediction\. It is not a causal answer, but it can direct follow\-up experiments on history, state, planning, verification, or recovery\.

This position paper makes three contributions:

1. 1\.Weadvocatetreating raw performance decline as a descriptive deployment result and the counterfactual residual as a separate, protocol\-dependent diagnostic\.
2. 2\.Weproposea deliberately simple and auditable product baseline, together with conditional\-model and checkpoint\-simulation extensions when its assumptions fail\.
3. 3\.Wearticulatereporting requirements, scope conditions, and an intervention agenda for turning residual patterns into testable causal hypotheses\.

## 2Diagnosing Performance Decline Beyond “Longer Is Harder”

Consider a coding agent tasked with making four independent repairs\. Suppose it succeeds on each repair with probability 80% when evaluated separately\. If these outcomes compose independently, then even when no additional long\-horizon failure occurs, the probability of completing all four repairs is only0\.84≈41%0\.8^\{4\}\\approx 41\\%\. Thus, a low end\-to\-end success rate is not by itself evidence of a distinctive long\-horizon limitation; it may follow from ordinary error compounding\.

Now suppose the same agent completes the four\-repair task only 10% of the time\. The gap between 41% and 10% is more informative\. Earlier edits may contaminate the repository state, long tool transcripts may distract the model, or the agent may lose track of the original goal, omit intermediate verification, or fail to recover from an early mistake\. These are plausible explanations, but the raw 10% success rate cannot resolve which, if any, is responsible\. Interpretation therefore requires a declared model of what matched short\-stage performance predicts\.

Table[1](https://arxiv.org/html/2607.27283#S2.T1)summarizes three effects that are easy to conflate:

Table 1:Distinct sources of performance degradation as task length increases, together with the control required to distinguish each source\.Hereafter,*trajectory\-induced degradation*means that accumulated execution \(the transcript, tool outputs, environment state, or earlier errors\) makes later work harder\.*Context rot*is the narrower case in which the growing visible text is the source of the decline\(Honget al\.,[2025](https://arxiv.org/html/2607.27283#bib.bib27)\); positional degradation in long contexts is a related observation\(Liuet al\.,[2024](https://arxiv.org/html/2607.27283#bib.bib26)\)\. These labels describe where performance changes, not the underlying mechanism\. They may cover distraction from a noisy transcript, cross\-module planning, state contamination, or failed recovery\.

Recent evidence shows why these categories should not be collapsed\. In a controlled running\-state task,Sinhaet al\.\([2026](https://arxiv.org/html/2607.27283#bib.bib2)\)hold the evaluation turn and history format fixed while varying the error rate of the preceding outputs\. Cleaner histories improve subsequent accuracy, showing that the content of the execution history matters beyond its length\. A related content\-versus\-length control appears in repeated social\-dilemma games: holding prompt length fixed while replacing the visible history with synthetic cooperative records substantially restores cooperative behavior, isolating history content rather than length as the trigger\(Liuet al\.,[2026](https://arxiv.org/html/2607.27283#bib.bib28)\)\. ChainSWE provides a complementary repository\-level comparison: across 100 chains containing 304 issues from 54 projects, average per\-bug accuracy falls from 58\.9% when each bug begins from an oracle\-correct repository state to 36\.5% when the agent inherits its own prior modifications\(Jinet al\.,[2026](https://arxiv.org/html/2607.27283#bib.bib8)\)\. Moreover, 318 of 663 downstream sequential failures in its baseline setting occur on bugs that the same systems solve from the oracle state\. These results do not establish a universal horizon law, but they illustrate the diagnostic value of comparing natural execution against a matched state\-repaired counterfactual\.

## 3A System\-Matched Counterfactual for Long\-Horizon Evaluation

### 3\.1Decomposition and Checkpoints

We propose beginning with a pre\-specified decomposition of the long task into verifiable stages\. In a software engineering task, these stages might be “find the bug”, “change the data model”, “update the API”, and “pass the integration tests”\. Each stage should be semantic: it should be defined by an acceptance condition rather than by an arbitrary number of turns or tool calls\.

![Refer to caption](https://arxiv.org/html/2607.27283v1/figures/horizon_residual.png)Figure 1:Agent\-configuration\-matched comparison between checkpoint\-based local evaluation and natural end\-to\-end execution\. Declared checkpoint interventions estimate the conditional stage probabilities used to constructPexpectedP\_\{\\mathrm\{expected\}\}, whereas the natural rollout yieldsPobservedP\_\{\\mathrm\{observed\}\}under accumulated history and shared state\. The horizon residualΓH\\Gamma\_\{\\mathrm\{H\}\}records their protocol\-specific log contrast; mechanism claims require additional interventions\.The next step is to run the same deployed agent from declared checkpoints \(Figure[1](https://arxiv.org/html/2607.27283#S3.F1)\)\. Here, the deployed agent denotes the complete system, including the model, prompt, tools, memory, and execution framework\. LetSiS\_\{i\}denote success at stageii, and letIIdenote a pre\-specified checkpoint protocol: the environment state, visible history, revealed information, and local budget supplied at that stage\. Along a fixed path on which every stage is required, defineqi=Pr⁡\(Si=1∣S<i=1,I\)q\_\{i\}=\\Pr\(S\_\{i\}=1\\mid S\_\{<i\}=1,I\)\. Here,qiq\_\{i\}is the agent’s chance of passing stageiiwhen the required earlier stages are already complete and the benchmark starts the agent from its declared checkpoint\.

### 3\.2The Product Baseline

The chain rule then gives the auditable baseline

Pexpected=∏iqi\.P\_\{\\mathrm\{expected\}\}=\\prod\_\{i\}q\_\{i\}\.\(1\)
In simple terms, if an agent has a 90% chance of passing Stage 1 and an 80% chance of passing Stage 2 from clean, compatible states, we predict a 72% overall success rate \(0\.90×0\.800\.90\\times 0\.80\)\. If the actual agent succeeds only 30% of the time, the intuitive shortfall is 42 percentage points; the horizon residual defined below reports this same comparison as a log\-ratio\.

The homogeneous special case clarifies why this baseline matters\. If every stage succeeds with a constant probabilitypp, failures are independent, and no recovery is possible, then annn\-stage task succeeds with probabilitypnp^\{n\}\. Equivalently, the first horizon at which success falls below a thresholdssis approximatelyHs=⌈log⁡\(s\)/log⁡\(p\)⌉H\_\{s\}=\\lceil\\log\(s\)/\\log\(p\)\\rceil\(Sinhaet al\.,[2026](https://arxiv.org/html/2607.27283#bib.bib2)\)\. This analysis shows that a substantial raw decline can arise without any additional long\-horizon mechanism\. Its assumptions are deliberately strong: stage accuracy may change with position, errors may alter future state, and natural rollouts may recover\. We therefore usepnp^\{n\}as an auditable null model rather than a description of how agents necessarily behave\.

Three baseline forms cover different task structures\. The*product baseline*multiplies conditional stage\-success rates when the checkpoints form one compatible path\. An*oracle\-scaffolded baseline*multiplies success rates from separately chosen, clean canonical states; because those states may not arise in one real rollout, it requires extra compatibility and independence assumptions\. A*checkpoint simulator*is needed when solutions branch, failures can be retried or recovered from, or one stage changes the state and odds of later stages\.

For a heterogeneous task family, predictions should be composed within each task and then aggregated using the same task weights asPobservedP\_\{\\mathrm\{observed\}\}; multiplying stage averages can give a different quantity\. In every case, the benchmark should declare the baseline model, estimate it from the corresponding checkpoint behavior, and compute its prediction before evaluating the natural end\-to\-end rollout\.

Checkpoint evaluation also raises a cross\-world composition problem\. An oracle reset, repair of an agent\-produced state, fresh context, and revelation of a local goal are separate interventions and therefore define different estimands\.

Moreover, an alternative valid solution at stageiineed not be composable with a canonical downstream checkpoint constructed from a reference solution\. A benchmark should declare whether it evaluates a canonical path, constructs solution\-conditioned downstream states, or simulates transitions across compatible states; otherwise theqiq\_\{i\}values may describe stages that cannot be combined in one coherent rollout\.

Existing results suggest that checkpoint and continuous protocols can differ sharply\. SWE\-Milestone evaluates 98 human\-verified milestones across seven repositories and five programming languages\. The reported scores are approximately 80% or higher when milestones are attempted independently from canonical snapshots, whereas the best reported continuous\-evolution score is 38\.03%\(Denget al\.,[2026](https://arxiv.org/html/2607.27283#bib.bib1)\)\. On scikit\-learn, one reported agent–model configuration drops from 93\.2% independently to 21\.1% continuously\. This is not yet the compositional prediction in Equation[1](https://arxiv.org/html/2607.27283#S3.E1): independent milestone scores must still be composed along the relevant task graph, with compatibility assumptions made explicit\. It nevertheless shows that reset\-state and persistent\-rollout protocols estimate materially different quantities\.

### 3\.3The Horizon Residual

LetPobservedP\_\{\\mathrm\{observed\}\}denote success under the natural end\-to\-end rollout\. For a fixed declared counterfactual protocol, we summarize the comparison as

ΓH=log⁡PexpectedPobserved,\\Gamma\_\{\\mathrm\{H\}\}=\\log\\frac\{P\_\{\\mathrm\{expected\}\}\}\{P\_\{\\mathrm\{observed\}\}\},\(2\)
wherelog\\logdenotes the natural logarithm, soΓH\\Gamma\_\{\\mathrm\{H\}\}is measured in nats\. We refer toΓH\\Gamma\_\{\\mathrm\{H\}\}as the*horizon residual*: a protocol\-specific log contrast, not a direct measure of a horizon mechanism\. In finite samples, it should be reported with raw counts, uncertainty, and an explicit treatment of zero or near\-zero estimated probabilities\. Relative to the specified counterfactual:

- •ΓH\>0\\Gamma\_\{\\mathrm\{H\}\}\>0indicates lower success in the natural rollout than the model predicts;
- •ΓH≈0\\Gamma\_\{\\mathrm\{H\}\}\\approx 0indicates agreement at the resolution supported by the data;
- •ΓH<0\\Gamma\_\{\\mathrm\{H\}\}<0indicates higher success in the natural rollout than the model predicts\.

Section[3\.4](https://arxiv.org/html/2607.27283#S3.SS4)gives the joint reading of these cases with both rates’ magnitudes\.

When the natural rollout can be evaluated at the same semantic boundaries, the contrast admits a useful stage\-wise decomposition\. Letri=Pr⁡\(Si=1∣S<i=1,N\)r\_\{i\}=\\Pr\(S\_\{i\}=1\\mid S\_\{<i\}=1,N\)denote conditional stage success under the natural\-rollout protocolNN\. If success requires every stage on a fixed path, then

Pobserved=∏iri,ΓH=∑ilog⁡qiri\.P\_\{\\mathrm\{observed\}\}=\\prod\_\{i\}r\_\{i\},\\qquad\\Gamma\_\{\\mathrm\{H\}\}=\\sum\_\{i\}\\log\\frac\{q\_\{i\}\}\{r\_\{i\}\}\.\(3\)
This breakdown shows which step in the natural rollout had the largest drop relative to the checkpoint baseline\. A positive term marks a stage at which success is lower in the natural rollout\. It does not identify why the difference occurs\. Moreover, a positive term at stageiimarks where the mismatch*surfaces*, not necessarily where it*originates*: degradation accumulated earlier may become visible only at stageii\. Estimating laterrir\_\{i\}can also be expensive because only rollouts with successful prefixes contribute to the corresponding probability\. These nested samples are correlated across stages, so intervals for the stage\-wise terms should be obtained by bootstrap over complete task units rather than from the analytic approximation below \(Equation[4](https://arxiv.org/html/2607.27283#S3.E4)\)\. Branching paths and recoverable failures again require a transition model rather than this fixed\-path factorization\.

The four\-repair example of Section[2](https://arxiv.org/html/2607.27283#S2)makes the computation concrete\. Matched checkpoints that giveqi=0\.8q\_\{i\}=0\.8on each repair yieldPexpected=0\.84≈0\.41P\_\{\\mathrm\{expected\}\}=0\.8^\{4\}\\approx 0\.41; a natural rollout that succeeds10%10\\%of the time then givesΓH=log⁡\(0\.41/0\.10\)≈1\.41\\Gamma\_\{\\mathrm\{H\}\}=\\log\(0\.41/0\.10\)\\approx 1\.41nats\. The natural rollout underperforms the composed local prediction by a factor of about four\. Stage\-wise terms of, say,\(0\.2,0\.5,0\.6,0\.1\)\(0\.2,0\.5,0\.6,0\.1\)would indicate that the mismatch surfaces mainly at the third repair and would target the first intervention there, subject to the surfacing\-versus\-origin caveat above\.

A first\-order uncertainty calculation further clarifies the data requirement\. Supposeq^i\\widehat\{q\}\_\{i\}is estimated frommim\_\{i\}independent Bernoulli trials andP^observed\\widehat\{P\}\_\{\\mathrm\{observed\}\}fromm0m\_\{0\}independent end\-to\-end trials\. Ignoring covariance, the delta method gives

Var⁡\(Γ^H\)≈∑i1−qimi​qi\+1−Pobservedm0​Pobserved\.\\operatorname\{Var\}\(\\widehat\{\\Gamma\}\_\{\\mathrm\{H\}\}\)\\approx\\sum\_\{i\}\\frac\{1\-q\_\{i\}\}\{m\_\{i\}q\_\{i\}\}\+\\frac\{1\-P\_\{\\mathrm\{observed\}\}\}\{m\_\{0\}P\_\{\\mathrm\{observed\}\}\}\.\(4\)
Thus, uncertainty grows rapidly when any local or end\-to\-end success probability approaches zero\. In realistic evaluations, estimates may share tasks, seeds, checkpoints, or agent states, violating the independence approximation in Equation[4](https://arxiv.org/html/2607.27283#S3.E4)\. Such sharing typically induces positive covariance between checkpoint and rollout estimates on the same task, so the independence approximation understates the variance ofΓ^H\\widehat\{\\Gamma\}\_\{\\mathrm\{H\}\}\. We therefore recommend paired or hierarchical bootstrap intervals over complete task units, with the analytic expression used only as intuition and a planning approximation\.

The residual is diagnostic, but it is not by itself a causal explanation\. A positive value marks a mismatch with the selected local\-performance model, not the reason for that mismatch\. If history\-induced degradation is the hypothesis, the benchmark can rerun the same task with periodic context reset or compression while holding other protocol features fixed\. Each intervention should define its ownPexpected​\(I\)P\_\{\\mathrm\{expected\}\}\(I\)andPobserved​\(I\)P\_\{\\mathrm\{observed\}\}\(I\)before residuals are compared; keeping the original counterfactual fixed instead answers a different deployment question\. A selective change in the contrast is more focused evidence than another natural\-rollout curve, although remaining intervention confounds still require analysis\.

### 3\.4Interpreting the Residual

The expected and observed success rates answer distinct questions and should be interpreted jointly, always relative to the specified counterfactual:

- •Both are high:both protocols place the task within the system’s measured capability under the stated budgets and information\.
- •Both are low:the task may remain a useful deployment stress test, but the comparison has little power to separate counterfactual\-model error, compounding, and history\-dependent effects\.
- •Expected success is high but observed success is low:the natural rollout has an unexplained deficit under the selected model, motivating targeted interventions and sensitivity checks rather than an immediate mechanism claim\.
- •Observed success exceeds expected success:the natural rollout may exploit feedback, redundancy, adaptation, or recovery omitted by the counterfactual; interventions are still needed to determine which explanation, if any, accounts for the advantage\.

If Equation[1](https://arxiv.org/html/2607.27283#S3.E1)predicts near\-zero success, the log contrast is statistically unstable and an all\-failure outcome is diagnostically weak\. Informative benchmark regimes should keep the counterfactual prediction away from degenerate extremes where feasible, report partial progress and cost, and include interventions that test why the two protocols differ\.

### 3\.5Choosing the Baseline

The appropriate baseline depends on the scientific question\. Fresh context tests a different estimand from history\-matched checkpoints, which may be more relevant to deployment forecasting because they retain the expected burden of long context\. State repair, as in ChainSWE’s oracle mode, keeps the position in the task sequence while replacing earlier agent modifications with verified ones\(Jinet al\.,[2026](https://arxiv.org/html/2607.27283#bib.bib8)\); oracle reset may additionally replace other aspects of state\. Revealing a stage goal changes the information available to the agent and should not be treated as a pure state reset\. No single baseline serves every estimand\. Benchmarks should therefore declare checkpoint state, state\-repair policy, visible history, revealed information, and local budget because each choice changes the meaning ofPexpectedP\_\{\\mathrm\{expected\}\}andΓH\\Gamma\_\{\\mathrm\{H\}\}\.

## 4Separating Task Structure from Local Difficulty

The key contrast is concrete:one hundred independent renames involve much work but little dependency, while a schema–storage–API–client migration may have fewer stages but require each stage to build on the last\. The number of turns does not capture this difference: one shell command may modify hundreds of files, whereas a difficult localization problem may consume many turns without adding semantic work\.

We therefore advocate reporting two complementary annotations:

- •Work exposureNworkN\_\{\\mathrm\{work\}\}:how many required, separately verifiable stages the benchmark declares\. More stages create more opportunities for local failure\.
- •Dependency depthHdepH\_\{\\mathrm\{dep\}\}:how many stages lie on the deepest required chain\. Greater depth means that later work must build on more earlier state or feedback\.

NeitherNworkN\_\{\\mathrm\{work\}\}norHdepH\_\{\\mathrm\{dep\}\}is an intrinsic property of a task: both depend on the chosen decomposition, acceptance conditions, and dependency graph\. They can nevertheless vary independently within a declared design\. Repetitive work can also be structurally shallow while becoming dependent during execution through a shared context window\.

HORIZON makes a related distinction by defining an agent\-independent intrinsic horizonH∗H^\{\*\}as the minimum number of effective actions required by an optimal policy and by constructing longer task families through depth and breadth extensions\(Wanget al\.,[2026](https://arxiv.org/html/2607.27283#bib.bib3)\)\. This is a useful safeguard against calling a short task “long\-horizon” merely because an agent repeats failed actions\. At the same time,H∗H^\{\*\}requires an oracle, expert demonstration, or formal task specification, and its operational mapping differs across domains\. We therefore view intrinsic horizon, annotated work exposure, and dependency depth as complementary descriptions: the first estimates minimal action length, while the latter two expose how a declared evaluation decomposes and connects verifiable work\.

Local difficulty should likewise be measured rather than declared\. A task is not simply “hard”; it is hard*for a particular agent operating with particular information, tools, verifiers, and budgets*\. The same refactor may be easy when a failing test localizes the problem and difficult when feedback is delayed until final integration\. For this reason, measured local success is more informative than a human\-assigned easy/hard label\.

Human completion time is useful for deployment calibration but does not resolve this confound\.Kwaet al\.\([2025](https://arxiv.org/html/2607.27283#bib.bib16)\), for example, define the task\-completion time horizon at a 50% success rate and estimate a horizon of roughly 50 minutes for a frontier system on their task mixture\. The metric communicates the scale of work that a system can reliably complete, but two tasks with the same human duration may differ substantially in annotated work exposure, dependency depth, verifier quality, and agent\-local difficulty\. Time horizon and the protocol\-specific structural annotations proposed here therefore serve complementary purposes\. Table[2](https://arxiv.org/html/2607.27283#S4.T2)sketches the factorial conditions that these two annotations induce\.

Table 2:A factorial coding\-task design that separates dependency depth from measured local\-stage difficulty\. Work exposure and local success should be matched within each intended comparison\.Task length and local difficulty will often be correlated in naturally collected tasks\. They need not be statistically independent, but a benchmark can attempt to separate them experimentally by varying the declared dependency structure while approximately matching local success, and by varying local difficulty while holding that structure fixed\.

## 5A Benchmark Design Protocol for Coding and Terminal Agents

Coding and terminal environments are particularly well suited to this design\. Repository snapshots and containers make earlier states replayable, while tests and state invariants provide semantic acceptance conditions\. We propose the following compact workflow:

1. 1\.Pre\-register\.Declare the task, stages, acceptance conditions, dependency graph, rollout policy, tools, verifier, and recovery rules before evaluation\.
2. 2\.Define checkpoints\.Record the environment state, visible history, revealed information, repair policy, and rules for alternative valid implementations\.
3. 3\.Match budgets\.Report token, tool, time, verification, and retry costs for both local and end\-to\-end runs\. Where possible, include both equal per\-stage budgets and an aggregate\-budget\-matched comparison; otherwise count the resource difference as part of the intervention\.
4. 4\.Audit local performance\.Report per\-stage successes, trial counts, uncertainty, and heterogeneity; pre\-specify how zero or near\-zero estimates will be handled\.
5. 5\.Choose the model\.Use the product baseline for a fixed compatible path\. Use a conditional model or checkpoint simulator when the task includes branches, retries, recovery, or shared\-state dependence\.
6. 6\.Run and intervene\.EstimatePexpectedP\_\{\\mathrm\{expected\}\}before measuringPobservedP\_\{\\mathrm\{observed\}\}, then test targeted changes such as history reset, state repair, stronger verification, rollback, or explicit planning\.
7. 7\.Test sensitivity\.Recompute predictions and residuals under multiple reasonable decompositions, dependency graphs, and admissible checkpoint protocols\. Call a conclusion robust only when its sign is consistent across the pre\-declared set; otherwise report the range ofΓH\\Gamma\_\{\\mathrm\{H\}\}\.
8. 8\.Audit the benchmark\.Report task\-inclusion criteria, exclusions, leakage checks, and contamination risks for both the agent and any decomposition or grading procedure\.

An informative task family should contain both shallow and deep variants\. Independent edits provide a high\-exposure, shallow control\. A causally ordered patch stack \(for example, schema, then storage, then API, then client\) provides a deep condition\. The local stages should be selected so that their measured success distributions are comparable\. Both stage\-given and goal\-only variants are useful because a checkpoint may reveal localization or decomposition information that is unavailable in the natural task\.

History\-induced degradation also warrants a dedicated controlled comparison, matching the useful control of Table[1](https://arxiv.org/html/2607.27283#S2.T1)\. The agent can perform the same sequence of simple operations while retaining the full history, periodically compressing it, or restarting from a clean context\. Such a comparison helps estimate the total effect of the declared history intervention; it does not identify a single underlying mechanism\. Observational evidence underscores why the controlled version matters: in an ultra\-long\-horizon software benchmark, none of the 71 rollouts that used mid\-trajectory context compaction passed, against 8\.9% without it\(Desaiet al\.,[2026](https://arxiv.org/html/2607.27283#bib.bib30)\); absent a matched intervention, such a correlation cannot distinguish compaction as a cause of failure from compaction as a marker of rollouts that were already failing\. Environment state, revealed information, and budgets must be held fixed across conditions unless their joint intervention is the stated estimand\.

## 6Related Evidence

The literature suggests a hierarchy of evidential strength\. End\-to\-end curves answer what happened under a deployment protocol; trajectory labels and dense rewards add localization\(Lightmanet al\.,[2024](https://arxiv.org/html/2607.27283#bib.bib31)\); matched checkpoints test whether the natural rollout exceeds local prediction; and targeted interventions test whether a proposed factor changes that contrast\(Sinhaet al\.,[2026](https://arxiv.org/html/2607.27283#bib.bib2); Wanget al\.,[2026](https://arxiv.org/html/2607.27283#bib.bib3)\)\. Figure[2](https://arxiv.org/html/2607.27283#S6.F2)summarizes this progression\. The levels are cumulative: mechanism experiments do not remove the need to report raw success, cost, and partial progress\.

![Refer to caption](https://arxiv.org/html/2607.27283v1/figures/diagnostic_ladder.png)

Figure 2:Hierarchy of evidence for long\-horizon evaluation\. Descriptive measurements characterize deployment behavior; a declared local\-stage counterfactual identifies mismatch with matched competence; and targeted interventions test scoped mechanism hypotheses\. Replication across tasks, systems, budgets, and annotation choices determines the generality of each conclusion\.Table 3:Representative empirical evidence, the history or state control implemented by each study, and the diagnostic question that remains unresolved\. Reported values follow study\-specific protocols and are not directly comparable across rows\.Table[3](https://arxiv.org/html/2607.27283#S6.T3)highlights how prior benchmarks observe performance drops on long tasks but do not yet combine all the history and state controls needed to isolate why the drop occurred\. The studies differ in whether they control local solvability, repair prior state, or intervene on history\.

Table 4:Component\-level delta between representative studies and the proposed protocol, judged from each study’s reported protocol\. ✓: implemented;∼\\sim: partial or implicit; —: not part of the reported protocol\.Table[4](https://arxiv.org/html/2607.27283#S6.T4)restates this comparison as a component\-level audit of the proposed sequence\. Every component exists in some published evaluation, and we claim no novelty for any of them individually\. The position is that no reviewed study implements the full sequence \(matched local measurement, declared composition, natural rollout, state\-compatibility handling, targeted intervention, and protocol\-sensitivity analysis\) under one pre\-specified protocol\. Partial marks record, for example, ChainSWE’s single oracle\-repair intervention, SWE\-Milestone’s inferred rather than declared dependency graph, and the homogeneouspnp^\{n\}analysis ofSinhaet al\.\([2026](https://arxiv.org/html/2607.27283#bib.bib2)\)on a controlled synthetic task rather than a natural deployment rollout\. If a prior evaluation is found to implement the complete sequence, the claim of this paper narrows to advocacy of its standardization\.

#### From isolated issue resolution to sustained software evolution\.

SWE\-bench established repository\-level issue resolution as a practical testbed for language models\(Jimenezet al\.,[2024](https://arxiv.org/html/2607.27283#bib.bib17)\); SWE\-Gym and OpenHands contributed executable training tasks, trajectories, and a general agent platform\(Panet al\.,[2025](https://arxiv.org/html/2607.27283#bib.bib13); Wanget al\.,[2025](https://arxiv.org/html/2607.27283#bib.bib14)\)\. SWE\-Lancer and time\-calibrated evaluations broaden ecological validity through freelance work and human\-duration estimates\(Miserendinoet al\.,[2025](https://arxiv.org/html/2607.27283#bib.bib15); Kwaet al\.,[2025](https://arxiv.org/html/2607.27283#bib.bib16)\)\. SWE\-Bench Pro, SWE\-EVO, and NL2Repo\-Bench extend scope toward enterprise issues, release\-sized changes, and full\-repository construction\(Denget al\.,[2025](https://arxiv.org/html/2607.27283#bib.bib11); Leet al\.,[2025](https://arxiv.org/html/2607.27283#bib.bib10); Dinget al\.,[2025](https://arxiv.org/html/2607.27283#bib.bib12)\)\. Their results indicate that current systems struggle as scope and coordination demands grow\. Their reported protocols are not designed to hold local difficulty, planning burden, feedback sparsity, and work exposure fixed while varying horizon, so they motivate rather than instantiate the counterfactual proposed here\.

#### Agent scaffolding as part of the evaluated system\.

The deployed agent is more than its base model\. SWE\-agent shows that an agent\-computer interface can alter repository navigation, editing, and testing behavior\(Yanget al\.,[2024](https://arxiv.org/html/2607.27283#bib.bib18)\)\. AutoCodeRover instead uses program structure and fault localization to focus search\(Zhanget al\.,[2024](https://arxiv.org/html/2607.27283#bib.bib20)\); Agentless replaces open\-ended tool planning with localization, repair, and patch validation\(Xiaet al\.,[2024](https://arxiv.org/html/2607.27283#bib.bib19)\); and MASAI delegates reproduction, localization, repair, and ranking to specialized modules\(Aroraet al\.,[2024](https://arxiv.org/html/2607.27283#bib.bib21)\)\. Their reported results use different models, prompts, costs, and benchmark versions, so they should not be read as a direct leaderboard\. Collectively, they show why local\-stage and natural\-rollout evaluations must preserve the same agent configuration: changing the scaffold changes the capability being measured\.

#### Stateful interactive environments beyond software repositories\.

The same concern appears outside software maintenance\. WebArena evaluates realistic web workflows and reports 14\.41% success for its best GPT\-4\-based agent versus 78\.24% for humans\(Zhouet al\.,[2023](https://arxiv.org/html/2607.27283#bib.bib22)\)\. OSWorld evaluates 369 open\-ended computer tasks and reports 12\.24% for its best evaluated model versus 72\.36% for humans\(Xieet al\.,[2024](https://arxiv.org/html/2607.27283#bib.bib23)\)\. AppWorld combines nine applications and 457 APIs with state\-based tests that also detect collateral changes\(Trivediet al\.,[2024](https://arxiv.org/html/2607.27283#bib.bib24)\)\. RE\-Bench complements binary completion with continuously scored research\-engineering objectives and varying time budgets; agents lead at short total budgets, while human experts show stronger returns to additional time\(Wijket al\.,[2024](https://arxiv.org/html/2607.27283#bib.bib25)\)\. These benchmarks improve realism and stateful evaluation, but low success alone still mixes grounding, operational knowledge, planning, local difficulty, and horizon\. Table[5](https://arxiv.org/html/2607.27283#S6.T5)summarizes these system\-level factors alongside the scaffolding studies above\.

Table 5:Evidence that interface design, agent scaffolding, environment state, and resource allocation are constitutive components of the evaluated system\. Results follow study\-specific protocols and are not directly comparable across rows\.
#### Partial\-progress measurement and persistent\-state controls\.

Terminal success can hide where a rollout failed\. LHTB uses deterministic subtask grading, so runs with meaningful partial reward are distinguishable from both passing runs and no\-progress failures\(Liet al\.,[2026](https://arxiv.org/html/2607.27283#bib.bib6)\)\. HORIZON analyzes more than 3,100 trajectories across web, operating\-system, database, and embodied domains with a seven\-category failure taxonomy\(Wanget al\.,[2026](https://arxiv.org/html/2607.27283#bib.bib3)\)\. Its trajectory\-grounded judge reachesκ=0\.84\\kappa=0\.84against one human annotator on a 40\-trajectory pilot, supporting scalable descriptive annotation\. The labels may co\-occur and are assigned after observing the rollout, however, so agreement supports reproducibility of attribution rather than causal identification\. SWE\-Chain, ChainSWE, and SlopCodeBench instead preserve evolving software state across transitions or checkpoints\(Lamet al\.,[2026](https://arxiv.org/html/2607.27283#bib.bib7); Jinet al\.,[2026](https://arxiv.org/html/2607.27283#bib.bib8); Orlanskiet al\.,[2026](https://arxiv.org/html/2607.27283#bib.bib9)\)\. Among the studies reviewed here, ChainSWE additionally reports a direct oracle\-state comparison\. These designs expose error propagation and architectural degradation that isolated issue protocols reset away, while leaving open how matched local performance should be composed into an expected end\-to\-end outcome\.

#### Checkpoint composition and mechanism\-targeted interventions\.

SWE\-Milestone closely aligns with our proposed direction by comparing canonical\-snapshot milestone execution with continuous execution on a persistent codebase\(Denget al\.,[2026](https://arxiv.org/html/2607.27283#bib.bib1)\)\. Its inferred graph is useful for evaluating history and patch execution, but it need not be a minimal graph of semantic prerequisites: an alternative valid implementation may not reproduce the reference path\. This illustrates why dependency annotations and state compatibility must be explicit when composing local estimates\. More direct mechanism evidence comes from intervention\. By manipulating errors in a fixed\-format prior history,Sinhaet al\.\([2026](https://arxiv.org/html/2607.27283#bib.bib2)\)estimate sensitivity to prior errors at a matched evaluation turn, while noting that the synthetic result is not sufficient for real\-world long\-horizon execution\. Broader task ontologies and controlled horizon variation provide complementary ingredients\(Donget al\.,[2026](https://arxiv.org/html/2607.27283#bib.bib4); Kimet al\.,[2026](https://arxiv.org/html/2607.27283#bib.bib5)\)\. Our position is that replayable checkpoints, dense progress measures, persistent\-state comparisons, and targeted interventions should be combined under a pre\-specified, agent\-configuration\-matched counterfactual, a combination that Table[4](https://arxiv.org/html/2607.27283#S6.T4)shows no reviewed study yet implements in full\.

## 7Scope, Limitations, and Falsifiability

The proposed quantities are properties of a declared evaluation protocol, not intrinsic properties of a task or model\. Stage decompositions are not unique, and dependency edges can be implementation\-relative\. Canonical checkpoints may also reveal privileged information about decomposition, localization, or the reference solution\. We therefore advocate publishing the decomposition, acceptance conditions, dependency annotations, checkpoint construction, and revealed information, and testing whether conclusions persist across multiple reasonable annotations and both stage\-given and goal\-only conditions\.

The decomposition is also a governance problem, not only a technical one\. Someone must author it, and its quality bounds every downstream quantity\. We recommend that decompositions be authored or reviewed by domain experts, versioned alongside the benchmark, and arbitrated through the annotation\-sensitivity tests of Section[5](https://arxiv.org/html/2607.27283#S5)rather than by fiat; LLM\-assisted decomposition is acceptable when each stage’s acceptance condition is executed against ground truth, since an unverifiable stage silently redefines the estimand\.

Statistical and resource assumptions are equally consequential\. Local outcomes may be heterogeneous or correlated across tasks, stages, agents, and random seeds; separately started stages may receive more total search, verification, or retry budget than a natural rollout\. Benchmark reports should therefore include raw counts, uncertainty, resource and cost accounting, conditional or simulation\-based alternatives to the product model, and predeclared rules for zero and near\-zero probabilities\. Verifier error belongs in the same ledger: an agentic audit of graded rollouts reports judge–verifier disagreement of 1\.4% for hand\-written functional verifiers versus 32\.4% for inherited pull\-request tests\(Huanget al\.,[2026](https://arxiv.org/html/2607.27283#bib.bib29)\); at the latter rate, measurement noise alone can dominate a small residual, so residual estimates should report verifier validation alongside raw counts\. Developing estimators that propagate uncertainty through branching state graphs is an important part of the empirical agenda\.

A further risk is protocol sensitivity: two admissible checkpoint protocols can in principle yield opposite\-signed residuals on the same task family\. The protocol treats this scenario as a result rather than a failure\. It indicates that no robust conclusion exists at the chosen resolution, but it also motivates the robustness criterion of Section[5](https://arxiv.org/html/2607.27283#S5): conclusions should be reported only when their sign is stable across the pre\-declared admissible set, and as protocol ranges otherwise\.

Finally, replayable and separately verifiable stages favor coding and terminal tasks with strong infrastructure\. Task inclusion, public\-repository exposure, reference\-derived checkpoints, and verifier availability can introduce selection or contamination bias\. The framework’s value outside these domains remains an empirical question\. Most importantly,ΓH\\Gamma\_\{\\mathrm\{H\}\}does not identify context degradation, planning failure, state contamination, or recovery; it identifies a contrast under one protocol\.

#### Falsifiers of this position\.

Our position would be substantially weakened if any of the following were demonstrated:

1. 1\.across multiple pre\-declared, admissible protocols on the same task family, the sign ofΓH\\Gamma\_\{\\mathrm\{H\}\}is unstable in ways that admissibility criteria cannot resolve;
2. 2\.compatible checkpoints cannot be constructed at manageable cost for representative coding benchmarks; or
3. 3\.the residual provides no incremental diagnostic information beyond existing persistent\-versus\-reset comparisons across several benchmark families\.

We invite empirical work targeting these tests\.

## 8Anticipated Objections and Research Agenda

#### Q: Isn’t end\-to\-end success what deployment cares about?

A:Yes\. Deployment and diagnostic benchmarks answer different questions, and neither replaces the other\. Our claim applies only when a deployment result is used to argue for a particular failure mechanism\.

#### Q: Won’t scaling dissolve the problem?

A:Longer contexts and better base models may reduce the residual, but they do not directly address state contamination, planning, or recovery\. If a scaled system reachesΓH≈0\\Gamma\_\{\\mathrm\{H\}\}\\approx 0under a declared baseline, that is exactly the evidence this position asks for\.

#### Q: Don’t standard ablations already identify mechanisms?

A:Ablations measure component contributions within a fixed protocol\. The residual asks whether matched local performance predicts the end\-to\-end result\. The two are complementary: the residual locates a mismatch that an ablation can then help explain\.

#### Q: If everything depends on the decomposition, how can results be compared?

A:Publish, replicate, and vary the decomposition\. The robustness criterion in Section[5](https://arxiv.org/html/2607.27283#S5)withholds conclusions that do not survive reasonable alternatives\. Declared dependence can be audited; hidden dependence cannot\.

The objections above mark the boundaries of the claim; the following agenda moves within them:

1. 1\.Instantiate the comparison\.Apply the protocol to one existing persistent\-state benchmark and reportPexpectedP\_\{\\mathrm\{expected\}\},PobservedP\_\{\\mathrm\{observed\}\}, andΓH\\Gamma\_\{\\mathrm\{H\}\}with bootstrap intervals, reusing published decompositions where available\.
2. 2\.Develop branching estimators\.Extend the uncertainty treatment of Equation[4](https://arxiv.org/html/2607.27283#S3.E4)to conditional transition models over branching state graphs, with explicit rules for zero and near\-zero probabilities\.
3. 3\.Run the intervention program\.Pair each recurring residual pattern with single\-factor interventions, history reset or compression, state repair, verification, rollback, explicit planning, so that residual patterns become testable causal hypotheses rather than labels\.
4. 4\.Map protocol robustness\.Characterize when admissible protocols agree in sign and when they diverge, turning the robustness criterion of Section[5](https://arxiv.org/html/2607.27283#S5)into empirically grounded admissibility rules\.
5. 5\.Transfer beyond coding\.Identify where replayable state and semantic acceptance conditions exist outside coding and terminal tasks, and adapt the protocol where they do not\.

We close with the strongest form of the claim\. A declining end\-to\-end curve licenses a claim about deployment; a claim about long\-horizon*mechanisms*should be licensed only by an evaluation that reports, alongside end\-to\-end success, a pre\-specified compositional prediction and the resulting residual\.

## 9Conclusion

Long tasks are globally harder in part because they require more work\. Our position is that the central scientific question is whether natural end\-to\-end rollouts perform differently from a pre\-specified, intervention\-specific model of local competence\. We therefore advocate pre\-specifying the stage decomposition and intervention protocol, estimating a protocol\-specific counterfactual, measuring the natural rollout, and using targeted interventions to investigate the resulting contrast\. The horizon residual is not a causal answer; it is a disciplined starting point for testing which aspects of history, state, information, and recovery make long\-horizon execution differ\. Until such residuals are reported, a declining curve should be read as evidence about deployment, not about mechanism\. This framing turns “longer is harder” from a descriptive observation into an auditable research program\.

## References

- D\. Arora, A\. Sonwane, N\. Wadhwa, A\. Mehrotra, S\. Utpala, R\. Bairi, A\. Kanade, and N\. Natarajan \(2024\)MASAI: modular architecture for software\-engineering AI agents\.arXiv preprint arXiv:2406\.11638\.External Links:[Link](https://arxiv.org/abs/2406.11638)Cited by:[§6](https://arxiv.org/html/2607.27283#S6.SS0.SSS0.Px2.p1.1),[Table 5](https://arxiv.org/html/2607.27283#S6.T5.1.6.4.1.1.1)\.
- G\. Deng, Z\. Chen, Z\. Yu, H\. Fan, Y\. Liu, Y\. Yang, D\. Parikh, R\. Kannan, L\. Cong, M\. Wang, Q\. Zhang, V\. Prasanna, X\. Tang, and X\. Wang \(2026\)SWE\-Milestone: evaluating AI agents on continuous software evolution\.arXiv preprint arXiv:2603\.13428\.Cited by:[§1](https://arxiv.org/html/2607.27283#S1.p1.1),[§1](https://arxiv.org/html/2607.27283#S1.p3.1),[§3\.2](https://arxiv.org/html/2607.27283#S3.SS2.p9.1),[§6](https://arxiv.org/html/2607.27283#S6.SS0.SSS0.Px5.p1.1),[Table 3](https://arxiv.org/html/2607.27283#S6.T3.1.8.7.1.1.1)\.
- X\. Deng, J\. Da, E\. Pan, Y\. Y\. He, C\. Ide, K\. Garg, N\. Lauffer, A\. Park, N\. Pasari, C\. Rane, K\. Sampath, M\. Krishnan, S\. Kundurthy, S\. Hendryx, Z\. Wang, V\. Bharadwaj, J\. Holm, R\. Aluri, C\. B\. C\. Zhang, N\. Jacobson, B\. Liu, and B\. Kenstler \(2025\)SWE\-Bench Pro: can AI agents solve long\-horizon software engineering tasks?\.arXiv preprint arXiv:2509\.16941\.External Links:[Link](https://arxiv.org/abs/2509.16941)Cited by:[§6](https://arxiv.org/html/2607.27283#S6.SS0.SSS0.Px1.p1.1)\.
- SWE\-Marathon: can agents autonomously complete ultra\-long\-horizon software work?\.arXiv preprint arXiv:2606\.07682\.External Links:[Link](https://arxiv.org/abs/2606.07682)Cited by:[§5](https://arxiv.org/html/2607.27283#S5.p4.1),[Table 3](https://arxiv.org/html/2607.27283#S6.T3.1.7.6.1.1.1)\.
- J\. Ding, S\. Long, C\. Pu, H\. Zhou, H\. Gao, X\. Gao, C\. He, Y\. Hou, F\. Hu, Z\. Li, W\. Shi, Z\. Wang, D\. Zan, C\. Zhang, X\. Zhang, Q\. Chen, X\. Cheng, B\. Deng, Q\. Gu, K\. Hua, J\. Lin, P\. Liu, M\. Li, X\. Pan, Z\. Peng, Y\. Qin, Y\. Shan, Z\. Tan, W\. Xie, Z\. Wang, Y\. Yuan, J\. Zhang, E\. Zhao, Y\. Zhao, H\. Zhu, L\. Zhu, C\. Zou, M\. Ding, J\. Jiao, J\. Liu, M\. Liu, Q\. Liu, C\. Tao, J\. Yang, T\. Yang, Z\. Zhang, X\. Chen, W\. Huang, and G\. Zhang \(2025\)NL2Repo\-Bench: towards long\-horizon repository generation evaluation of coding agents\.arXiv preprint arXiv:2512\.12730\.External Links:[Link](https://arxiv.org/abs/2512.12730)Cited by:[§6](https://arxiv.org/html/2607.27283#S6.SS0.SSS0.Px1.p1.1),[Table 3](https://arxiv.org/html/2607.27283#S6.T3.1.3.2.1.1.1)\.
- G\. Dong, X\. Song, Y\. Hu, J\. Jin, C\. Zhang, Y\. Chen, X\. Li, H\. Yuan, X\. Yang, T\. Wen, J\. Tan, H\. Qian, S\. Huang, J\. Lu, Z\. Li, W\. Zhong, Y\. Zhu, T\. Chua, Z\. Dou, and J\. Wen \(2026\)Towards long\-horizon agents: a survey—foundation, evolution, harness, optimization, application, and frontier\.Preprint\.External Links:[Link](https://long-horizon-agents.github.io/)Cited by:[§6](https://arxiv.org/html/2607.27283#S6.SS0.SSS0.Px5.p1.1)\.
- K\. Hong, A\. Troynikov, and J\. Huber \(2025\)Context rot: how increasing input tokens impacts LLM performance\.Technical reportChroma\.External Links:[Link](https://research.trychroma.com/context-rot)Cited by:[§2](https://arxiv.org/html/2607.27283#S2.p4.1)\.
- W\. Huang, C\. Lee, L\. Tng, and S\. Ge \(2026\)DeepSWE: measuring frontier coding agents on original, long\-horizon engineering tasks\.arXiv preprint arXiv:2607\.07946\.External Links:[Link](https://arxiv.org/abs/2607.07946)Cited by:[§7](https://arxiv.org/html/2607.27283#S7.p3.1)\.
- C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. Narasimhan \(2024\)SWE\-bench: can language models resolve real\-world GitHub issues?\.InThe Twelfth International Conference on Learning Representations,Note:arXiv:2310\.06770External Links:[Link](https://arxiv.org/abs/2310.06770)Cited by:[§1](https://arxiv.org/html/2607.27283#S1.p1.1),[§6](https://arxiv.org/html/2607.27283#S6.SS0.SSS0.Px1.p1.1)\.
- Q\. Jin, L\. Tung, K\. Li, Q\. Shi, Y\. She, H\. Jia, H\. Zhao, K\. Xia, Z\. Du, Y\. Zhang, J\. Pei, Z\. Zhang, Z\. Qi, Y\. Duan, W\. Lee, and Z\. Jin \(2026\)ChainSWE: benchmarking coding agents on multi\-bug software maintenance\.arXiv preprint arXiv:2607\.02606\.External Links:[Link](https://arxiv.org/abs/2607.02606)Cited by:[§1](https://arxiv.org/html/2607.27283#S1.p3.1),[§2](https://arxiv.org/html/2607.27283#S2.p5.1),[§3\.5](https://arxiv.org/html/2607.27283#S3.SS5.p1.2),[§6](https://arxiv.org/html/2607.27283#S6.SS0.SSS0.Px4.p1.1),[Table 3](https://arxiv.org/html/2607.27283#S6.T3.1.5.4.1.1.1)\.
- S\. Kim, J\. Cho, B\. Kwak, T\. Kwon, L\. Wang, N\. Yang, X\. Zhang, F\. Wei, and J\. Yeo \(2026\)On training large language models for long\-horizon tasks: an empirical study of horizon length\.InProceedings of the 43rd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.306\.Note:arXiv:2605\.02572Cited by:[§6](https://arxiv.org/html/2607.27283#S6.SS0.SSS0.Px5.p1.1)\.
- T\. Kwa, B\. West, J\. Becker, A\. Deng, K\. Garcia, M\. Hasin, S\. Jawhar, M\. Kinniment, N\. Rush, S\. Von Arx, R\. Bloom, T\. Broadley, H\. Du, B\. Goodrich, N\. Jurkovic, L\. H\. Miles, S\. Nix, T\. Lin, C\. Painter, N\. Parikh, D\. Rein, L\. J\. K\. Sato, H\. Wijk, D\. M\. Ziegler, E\. Barnes, and L\. Chan \(2025\)Measuring AI ability to complete long software tasks\.InAdvances in Neural Information Processing Systems,Note:arXiv:2503\.14499External Links:[Link](https://arxiv.org/abs/2503.14499)Cited by:[§1](https://arxiv.org/html/2607.27283#S1.p2.2),[§4](https://arxiv.org/html/2607.27283#S4.p7.1),[§6](https://arxiv.org/html/2607.27283#S6.SS0.SSS0.Px1.p1.1)\.
- M\. H\. Lam, C\. Wang, H\. Liu, J\. Xiao, H\. Li, J\. Huang, T\. Y\. Zhuo, and M\. R\. Lyu \(2026\)SWE\-Chain: benchmarking coding agents on chained release\-level package upgrades\.arXiv preprint arXiv:2605\.14415\.External Links:[Link](https://arxiv.org/abs/2605.14415)Cited by:[§1](https://arxiv.org/html/2607.27283#S1.p1.1),[§6](https://arxiv.org/html/2607.27283#S6.SS0.SSS0.Px4.p1.1),[Table 3](https://arxiv.org/html/2607.27283#S6.T3.1.4.3.1.1.1)\.
- T\. Le, M\. V\. T\. Thai, D\. N\. Manh, H\. P\. Nhat, and N\. D\. Q\. Bui \(2025\)SWE\-EVO: benchmarking coding agents in long\-horizon software evolution scenarios\.arXiv preprint arXiv:2512\.18470\.External Links:[Link](https://arxiv.org/abs/2512.18470)Cited by:[§1](https://arxiv.org/html/2607.27283#S1.p1.1),[§6](https://arxiv.org/html/2607.27283#S6.SS0.SSS0.Px1.p1.1),[Table 3](https://arxiv.org/html/2607.27283#S6.T3.1.2.1.1.1.1)\.
- Z\. Li, Z\. Li, Y\. Shi, R\. Wang, J\. Yang, Z\. Liu, X\. Wu, A\. Li, Y\. Yu, N\. Liu, L\. Sun, H\. Mi, and L\. Liang \(2026\)Long\-horizon\-terminal\-bench: testing the limits of agents on long\-horizon terminal tasks with dense reward\-based grading\.arXiv preprint arXiv:2607\.08964\.Cited by:[§6](https://arxiv.org/html/2607.27283#S6.SS0.SSS0.Px4.p1.1)\.
- H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe \(2024\)Let’s verify step by step\.InInternational Conference on Learning Representations,Note:arXiv:2305\.20050Cited by:[§6](https://arxiv.org/html/2607.27283#S6.p1.1)\.
- J\. Liu, T\. Li, S\. Du, X\. Luo, H\. Zeng, E\. Tewolde, T\. S\. Lee, T\. Wang, C\. Kingsford, and V\. Conitzer \(2026\)The memory curse: how expanded recall erodes cooperative intent in LLM agents\.arXiv preprint arXiv:2605\.08060\.External Links:[Link](https://arxiv.org/abs/2605.08060)Cited by:[§2](https://arxiv.org/html/2607.27283#S2.p5.1)\.
- N\. F\. Liu, K\. Lin, J\. Hewitt, A\. Paranjape, M\. Bevilacqua, F\. Petroni, and P\. Liang \(2024\)Lost in the middle: how language models use long contexts\.Transactions of the Association for Computational Linguistics12,pp\. 157–173\.Cited by:[§2](https://arxiv.org/html/2607.27283#S2.p4.1)\.
- S\. Miserendino, M\. Wang, T\. Patwardhan, and J\. Heidecke \(2025\)SWE\-Lancer: can frontier LLMs earn $1 million from real\-world freelance software engineering?\.arXiv preprint arXiv:2502\.12115\.External Links:[Link](https://arxiv.org/abs/2502.12115)Cited by:[§6](https://arxiv.org/html/2607.27283#S6.SS0.SSS0.Px1.p1.1)\.
- G\. Orlanski, D\. Roy, A\. Yun, C\. Shin, A\. Gu, A\. Ge, D\. Adila, N\. Roberts, F\. Sala, and A\. Albarghouthi \(2026\)SlopCodeBench: benchmarking how coding agents degrade over long\-horizon iterative tasks\.arXiv preprint arXiv:2603\.24755\.External Links:[Link](https://arxiv.org/abs/2603.24755)Cited by:[§6](https://arxiv.org/html/2607.27283#S6.SS0.SSS0.Px4.p1.1),[Table 3](https://arxiv.org/html/2607.27283#S6.T3.1.6.5.1.1.1)\.
- J\. Pan, X\. Wang, G\. Neubig, N\. Jaitly, H\. Ji, A\. Suhr, and Y\. Zhang \(2025\)Training software engineering agents and verifiers with SWE\-Gym\.InProceedings of the 42nd International Conference on Machine Learning,Note:arXiv:2412\.21139External Links:[Link](https://arxiv.org/abs/2412.21139)Cited by:[§6](https://arxiv.org/html/2607.27283#S6.SS0.SSS0.Px1.p1.1)\.
- A\. Sinha, A\. Arun, S\. Goel, S\. Staab, and J\. Geiping \(2026\)The illusion of diminishing returns: measuring long horizon execution in LLMs\.InInternational Conference on Learning Representations,Note:arXiv:2509\.09677Cited by:[§1](https://arxiv.org/html/2607.27283#S1.p3.1),[§2](https://arxiv.org/html/2607.27283#S2.p5.1),[§3\.2](https://arxiv.org/html/2607.27283#S3.SS2.p4.6),[§6](https://arxiv.org/html/2607.27283#S6.SS0.SSS0.Px5.p1.1),[Table 3](https://arxiv.org/html/2607.27283#S6.T3.1.9.8.1.1.1),[§6](https://arxiv.org/html/2607.27283#S6.p1.1),[§6](https://arxiv.org/html/2607.27283#S6.p3.1)\.
- H\. Trivedi, T\. Khot, M\. Hartmann, R\. Manku, V\. Dong, E\. Li, S\. Gupta, A\. Sabharwal, and N\. Balasubramanian \(2024\)AppWorld: a controllable world of apps and people for benchmarking interactive coding agents\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics,Note:arXiv:2407\.18901External Links:[Link](https://arxiv.org/abs/2407.18901)Cited by:[§1](https://arxiv.org/html/2607.27283#S1.p1.1),[§6](https://arxiv.org/html/2607.27283#S6.SS0.SSS0.Px3.p1.1),[Table 5](https://arxiv.org/html/2607.27283#S6.T5.1.9.7.1.1.1)\.
- X\. Wang, B\. Li, Y\. Song, F\. F\. Xu, X\. Tang, M\. Zhuge, J\. Pan, Y\. Song, B\. Li, J\. Singh, H\. H\. Tran, F\. Li, R\. Ma, M\. Zheng, B\. Qian, Y\. Shao, N\. Muennighoff, Y\. Zhang, B\. Hui, J\. Lin, R\. Brennan, H\. Peng, H\. Ji, and G\. Neubig \(2025\)OpenHands: an open platform for AI software developers as generalist agents\.InThe Thirteenth International Conference on Learning Representations,Note:arXiv:2407\.16741External Links:[Link](https://arxiv.org/abs/2407.16741)Cited by:[§6](https://arxiv.org/html/2607.27283#S6.SS0.SSS0.Px1.p1.1)\.
- X\. J\. Wang, H\. Bai, Y\. Sun, H\. Wang, S\. Zhang, W\. Hu, M\. Schroder, B\. Mutlu, D\. Song, and R\. D\. Nowak \(2026\)The long\-horizon task mirage? diagnosing where and why agentic systems break\.arXiv preprint arXiv:2604\.11978\.Cited by:[§1](https://arxiv.org/html/2607.27283#S1.p2.2),[§4](https://arxiv.org/html/2607.27283#S4.p5.2),[§6](https://arxiv.org/html/2607.27283#S6.SS0.SSS0.Px4.p1.1),[Table 3](https://arxiv.org/html/2607.27283#S6.T3.1.10.9.1.1.1),[§6](https://arxiv.org/html/2607.27283#S6.p1.1)\.
- H\. Wijk, T\. Lin, J\. Becker, S\. Jawhar, N\. Parikh, T\. Broadley, L\. Chan, M\. Chen, J\. Clymer, J\. Dhyani, E\. Ericheva, K\. Garcia, B\. Goodrich, N\. Jurkovic, H\. Karnofsky, M\. Kinniment, A\. Lajko, S\. Nix, L\. Sato, W\. Saunders, M\. Taran, B\. West, and E\. Barnes \(2024\)RE\-Bench: evaluating frontier AI r&d capabilities of language model agents against human experts\.arXiv preprint arXiv:2411\.15114\.External Links:[Link](https://arxiv.org/abs/2411.15114)Cited by:[§1](https://arxiv.org/html/2607.27283#S1.p1.1),[§6](https://arxiv.org/html/2607.27283#S6.SS0.SSS0.Px3.p1.1),[Table 5](https://arxiv.org/html/2607.27283#S6.T5.1.1.2.1.1)\.
- C\. S\. Xia, Y\. Deng, S\. Dunn, and L\. Zhang \(2024\)Agentless: demystifying LLM\-based software engineering agents\.arXiv preprint arXiv:2407\.01489\.External Links:[Link](https://arxiv.org/abs/2407.01489)Cited by:[§6](https://arxiv.org/html/2607.27283#S6.SS0.SSS0.Px2.p1.1),[Table 5](https://arxiv.org/html/2607.27283#S6.T5.1.5.3.1.1.1)\.
- T\. Xie, D\. Zhang, J\. Chen, X\. Li, S\. Zhao, R\. Cao, T\. J\. Hua, Z\. Cheng, D\. Shin, F\. Lei, Y\. Liu, Y\. Xu, S\. Zhou, S\. Savarese, C\. Xiong, V\. Zhong, and T\. Yu \(2024\)OSWorld: benchmarking multimodal agents for open\-ended tasks in real computer environments\.arXiv preprint arXiv:2404\.07972\.External Links:[Link](https://arxiv.org/abs/2404.07972)Cited by:[§1](https://arxiv.org/html/2607.27283#S1.p1.1),[§6](https://arxiv.org/html/2607.27283#S6.SS0.SSS0.Px3.p1.1),[Table 5](https://arxiv.org/html/2607.27283#S6.T5.1.8.6.1.1.1)\.
- J\. Yang, C\. E\. Jimenez, A\. Wettig, K\. Lieret, S\. Yao, K\. Narasimhan, and O\. Press \(2024\)SWE\-agent: agent\-computer interfaces enable automated software engineering\.InAdvances in Neural Information Processing Systems,Note:arXiv:2405\.15793External Links:[Link](https://arxiv.org/abs/2405.15793)Cited by:[§6](https://arxiv.org/html/2607.27283#S6.SS0.SSS0.Px2.p1.1),[Table 5](https://arxiv.org/html/2607.27283#S6.T5.1.3.1.1.1.1)\.
- Y\. Zhang, H\. Ruan, Z\. Fan, and A\. Roychoudhury \(2024\)AutoCodeRover: autonomous program improvement\.InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis,Note:arXiv:2404\.05427External Links:[Link](https://arxiv.org/abs/2404.05427)Cited by:[§6](https://arxiv.org/html/2607.27283#S6.SS0.SSS0.Px2.p1.1),[Table 5](https://arxiv.org/html/2607.27283#S6.T5.1.4.2.1.1.1)\.
- S\. Zhou, F\. F\. Xu, H\. Zhu, X\. Zhou, R\. Lo, A\. Sridhar, X\. Cheng, T\. Ou, Y\. Bisk, D\. Fried, U\. Alon, and G\. Neubig \(2023\)WebArena: a realistic web environment for building autonomous agents\.arXiv preprint arXiv:2307\.13854\.External Links:[Link](https://arxiv.org/abs/2307.13854)Cited by:[§1](https://arxiv.org/html/2607.27283#S1.p1.1),[§6](https://arxiv.org/html/2607.27283#S6.SS0.SSS0.Px3.p1.1),[Table 5](https://arxiv.org/html/2607.27283#S6.T5.1.7.5.1.1.1)\.