监控器准确率并非答案：评估 AI 智能体重启监控器的决策价值框架

# Monitor Accuracy Isn’t the Answer:A Decision-Value Framework for EvaluatingAI Agent Restart Monitors
Source: [https://arxiv.org/html/2610.36544](https://arxiv.org/html/2610.36544)
###### Abstract

Failure monitors for AI agents can be judged by classification accuracy and true\-positive and false\-positive rates on labeled failures\. We show that these metrics alone do not determine what actually matters: whether a restart improves success\. Using an exact alarm\-to\-action identity for an independent clean restart with zero monitoring overhead, we show analytically that a monitor with fixed 90% accuracy can swing from a five\-percentage\-point loss to a 35\-point gain in success rate, depending solely on the replacement attempt’s residual\-budget success probability, a quantity accuracy never measures\. Building on this, we introduce two techniques for evaluating monitors and progress signals directly in decision terms rather than classification terms\. First, a decision\-sufficiency criterion, a binary\-action specialization of Blackwell’s comparison of experiments, identifies exactly which progress representations or context compressions preserve the information needed to choose optimally between the specified actions, independent of how faithfully they reconstruct the trace\. Second, a finite\-sample, model\-free decision audit uses simultaneous binomial intervals over existing agent traces, without additional model calls, to certify a preference for “restart” or “continue” under stated sampling assumptions, or return an explicit “abstain” when evidence is insufficient, instead of reporting an unqualified point estimate\. Applied to a study of 194 externally evaluated model executions on 19 selected SymPy issues from SWE\-bench Verified, with a retrospective audit of 130 of these traces, this approach exposes what accuracy\-based evaluation alone cannot establish: a 2\.29\-point apparent restart gain traced to a single surviving failed attempt, a 42\.29\-point loss on a second model, no positive replication of the initial gain on prospective task cohorts, and zero of 103 live comparisons meeting the specified certification bar\. Together, these results motivate evaluating agent monitors through recovery opportunity, decision value, and statistical support, rather than classification performance alone\.

## 1Introduction

An agent that is unlikely to finish may also be unlikely to succeed after a restart\. This distinction matters whenever a harness acts on a failure prediction: abandoning a trajectory removes its remaining chance of success, and the replacement receives only the residual budget\. A monitor can therefore classify failures accurately while inducing harmful decisions\. Conversely, a modest signal may be useful when it resolves a small number of decisions with large consequences\.

The problem has a substantial intellectual history\. Dynamic restart policies use observations of a solver’s activity to update runtime beliefs\[[12](https://arxiv.org/html/2610.36544#bib.bib1)\]; learned schedules address distributions of problem instances\[[22](https://arxiv.org/html/2610.36544#bib.bib2)\]; Bayesian early stopping derives policies from observed run features\[[23](https://arxiv.org/html/2610.36544#bib.bib3)\]\. Recent agent systems demonstrate useful termination and recovery, including EET\[[5](https://arxiv.org/html/2610.36544#bib.bib8)\]and FailFast\-RestartSmart\[[26](https://arxiv.org/html/2610.36544#bib.bib7)\]\. Our contribution is not the premise that progress can inform stopping\. We ask how to make a claimed progress signal’s*decision value*and*evidential support*explicit under a finite interaction budget\.

Three questions organize the study:

RQ1Which quantities determine whether a failure alarm improves success after a restart?

RQ2What information must a progress representation retain, and when do available traces justify a continuation or restart decision?

RQ3How do these distinctions explain observed outcomes of restart, horizon extension, context recovery, and failed\-branch reuse?

We make the intervention precise\. For a live attempt, continuation retains its state and spends the remaining budget; a clean restart restores the original task state and spends the same budget on a fresh attempt\. A context reset that retains edits is a separate action, with a separate outcome law\. Under this definition, we derive an exact relation between alarm true\-positive rate, false\-positive rate, continuation success, and restartability\. An analytical control keeps failure classification accuracy at90%90\\%while changing restart gain from−5\-5to\+35\+35points solely by changing the success probability of the replacement\. This is a controlled mathematical example, not a performance claim about an implemented detector\.

The information question follows naturally\. Progress need not be a monotone scalar\. It can be a representation that preserves distinctions relevant to the action\. We give a binary\-action specialization of classical information comparison\[[1](https://arxiv.org/html/2610.36544#bib.bib4)\]: compressing a history loses decision value when the compressed observation combines positive and negative continuation advantages\. Exact decision sufficiency is an ideal benchmark; an imperfect representation can still improve utility\. We then implement a conservative audit that exposes the statistical support for each action, including an abstention when the evidence does not resolve the sign\.

The empirical evidence consists of 194 model\-driven executions of mini\-swe\-agent on 19 selected SymPy issues in SWE\-bench Verified\[[8](https://arxiv.org/html/2610.36544#bib.bib14),[31](https://arxiv.org/html/2610.36544#bib.bib15)\]\. Each attempt begins from the benchmark\-provided repository state\. The reference patch and evaluator remain outside the sandbox, and external verification after each environment action records the first passing state\. Earlier experiments comprise restart replay, prospective task transfer, additional horizon, conversational recovery, and two bounded uses of failed branches\. We retain the negative results and distinguish their design status: later collection stages used prospectively frozen protocols, while the decision audit introduced here is retrospective\.

The audit sharpens the interpretation of the original results\. Across 130 executions with a 30\-call horizon, the 16\-call Luna gain comes from a single eligible failed trajectory on one task\. On the second model, every task with a live attempt at that decision has an empirical advantage for continuation\. Across six decision times, all 103 comparisons with live data remain unresolved by the simultaneous certificate\. These findings distinguish visible effect size from evidence sufficient for individual deployment decisions\. They do not demonstrate that every alternative statistical method must abstain\.

Contributions\.

1. 1\.An intervention\-specific account of agent progress, including an alarm\-to\-action value identity and an information\-loss criterion for progress representations and context compression\.
2. 2\.A dependency\-free audit that estimates both action values, respects live eligibility and observed horizons, and returns simultaneous sign certificates or abstention under explicit assumptions\.
3. 3\.A reproducible retrospective analysis identifying the support of the restart effect, together with analytical controls that separate detector accuracy, restartability, and sample resolution\.
4. 4\.A sequence of externally verified agent experiments showing model\-dependent restart effects and the distinction between protocol recovery and verified task completion\.

The classical decision identities are attributed and used as foundations\. The empirical and software contributions concern how those identities can discipline evaluation of current agent controllers\. The study does not train a monitor, directly compare against learned recovery systems, or establish performance outside its selected tasks and runtime configurations\.

## 2Related Work

### 2\.1Restarts and inference\-time allocation

Restart policies are well studied for randomized algorithms\. Luby, Sinclair, and Zuckerman give a logarithmic expected\-runtime comparison with a distribution\-specific fixed strategy under their assumptions\[[16](https://arxiv.org/html/2610.36544#bib.bib9)\]\. This is not a guarantee that a chosen finite\-budget truncation improves success\. Dynamic restart policies already condition on solver observations and runtime beliefs\[[12](https://arxiv.org/html/2610.36544#bib.bib1)\]\. Schedule learning across instances\[[22](https://arxiv.org/html/2610.36544#bib.bib2)\]and Bayesian stopping from run features\[[23](https://arxiv.org/html/2610.36544#bib.bib3)\]directly precede our formulation\. The no\-dominance and observation\-ambiguity examples below are elementary specializations; novelty is not claimed for feature\-dependent restarting or the need to calibrate runtime distributions\.

Our decision\-value expression specializes Blackwell’s comparison of experiments\[[1](https://arxiv.org/html/2610.36544#bib.bib4)\]to two actions\. The abstaining audit applies exact binomial intervals\[[3](https://arxiv.org/html/2610.36544#bib.bib5)\]and simultaneous comparison in the spirit of safe policy improvement\[[13](https://arxiv.org/html/2610.36544#bib.bib6)\]\. These foundations identify the scope of the contribution: an explicit connection between alarm quality, intervention\-specific restartability, and the statistical support of agent decisions, implemented on externally verified traces\.

Inference\-time scaling allocates additional computation through sampling, voting, or search\. Self\-consistency samples reasoning paths and aggregates answers\[[28](https://arxiv.org/html/2610.36544#bib.bib19)\]; Tree of Thoughts searches over explicit intermediate candidates\[[32](https://arxiv.org/html/2610.36544#bib.bib20)\]; inference scaling studies compare compute\-allocation strategies under fixed budgets\[[30](https://arxiv.org/html/2610.36544#bib.bib21)\]\. Mixture\-of\-Agents aggregates outputs across model instances and layers\[[27](https://arxiv.org/html/2610.36544#bib.bib22)\]\. These methods can exploit independent or weakly correlated trials and informative scoring\. Classical ensemble analysis shows why both accuracy and error diversity matter\[[4](https://arxiv.org/html/2610.36544#bib.bib12),[2](https://arxiv.org/html/2610.36544#bib.bib13)\]\. Our setting removes a central advantage: the controller receives no reliable scalar score before external task completion\. We study whether schedule choice or failure\-conditioned branching alone can recover that missing information\.

Iterative self\-improvement methods use generated feedback to revise an answer or policy\[[18](https://arxiv.org/html/2610.36544#bib.bib23),[20](https://arxiv.org/html/2610.36544#bib.bib24)\]\. Other work shows that intrinsic self\-correction can fail when the feedback does not add external information\[[7](https://arxiv.org/html/2610.36544#bib.bib25)\]\. Our context\-rejuvenation and semantic\-resolver experiments test a related distinction at the harness level: changing the context or synthesizing failed branches can change behavior without improving the externally verified outcome\.

### 2\.2Agent progress and externally grounded evaluation

The closest contemporary system is FailFast\-RestartSmart\[[26](https://arxiv.org/html/2610.36544#bib.bib7)\]\. It trains a prefix monitor and combines alarms with a fresh rollout that can inspect or apply previous edits\. The paper reports both cost savings and recovery gains, providing positive evidence that richer learned signals and interventions can work\. EET also demonstrates useful termination from prior execution experience\[[5](https://arxiv.org/html/2610.36544#bib.bib8)\]\. Our experiments do not reproduce or compare against either system\. A timeout is a different controller, and our mandatory workspace retention differs from an optional edit overlay\. Our audit complements such systems by requiring the post\-alarm action and remaining\-budget success to be evaluated jointly with failure prediction\.

AgentBoard introduced a progress\-rate metric to complement final success across multi\-turn environments\[[17](https://arxiv.org/html/2610.36544#bib.bib18)\]\. Such metrics are valuable when an environment exposes meaningful subgoals\. Park and Choi show that an agent can report improvement while externally measured state does not improve, and argue for evaluation grounded outside the transcript\[[19](https://arxiv.org/html/2610.36544#bib.bib28)\]\. We take external grounding as a design constraint and ask what remains identifiable when the grounded signal is binary completion rather than a dense progress measure\.

Semantic early stopping detects convergence between successive drafts and can reduce tokens in iterative question answering at comparable measured quality\[[21](https://arxiv.org/html/2610.36544#bib.bib26)\]\. Geometric analyses represent iterative outputs as trajectories, attractors, or oscillations in an embedding space\[[25](https://arxiv.org/html/2610.36544#bib.bib27)\]\. These representations describe change in semantic space, but geometric convergence is not automatically decision sufficient for externally verified completion\. VLAA\-GUI uses visual completeness checks and loop breaking in GUI automation\[[6](https://arxiv.org/html/2610.36544#bib.bib29)\]\. These systems have richer task structure than our pass\-or\-fail coding controller\. Their results are consistent with our boundary condition: useful stopping and recovery require a signal calibrated to the relevant external state, not elapsed calls or semantic movement alone\.

Long\-horizon software agents have also motivated active context management\. Context as a Tool lets an agent compress history into structured memory\[[15](https://arxiv.org/html/2610.36544#bib.bib30)\], while later work learns or implements agent\-compatible context policies\[[33](https://arxiv.org/html/2610.36544#bib.bib31),[14](https://arxiv.org/html/2610.36544#bib.bib32)\]\. Our recovery intervention is deliberately simpler\. It discards conversational history after a protocol terminal event, retains the exact workspace, and supplies a deterministic state summary\. Its null result does not evaluate learned context management\. It isolates whether stale conversation alone explains observed failure in one fixed harness\.

### 2\.3Software engineering agents and reproducible evaluation

SWE\-bench evaluates repository\-level issue resolution against executable tests\[[8](https://arxiv.org/html/2610.36544#bib.bib14)\]\. SWE\-agent established the effectiveness of purpose\-built agent\-computer interfaces for these tasks\[[31](https://arxiv.org/html/2610.36544#bib.bib15)\]\. We use the compact mini\-swe\-agent implementation as a versioned, open\-source harness\. We do not compare agent architectures or report a leaderboard estimate\. Selected SymPy tasks provide controlled, expensive trajectories with externally testable completion\.

Agent evaluation is sensitive to cost, holdout design, and reproducibility\[[11](https://arxiv.org/html/2610.36544#bib.bib17)\]\. Test\-suite weakness can also overstate issue resolution\[[29](https://arxiv.org/html/2610.36544#bib.bib33)\]\. We therefore run the benchmark evaluator outside the agent workspace after each environment action, gate tasks by base\-fail and reference\-pass checks, audit every successful diff for verifier tampering, and select cutoffs with task\-level holdout rather than attempt\-level splitting\.

### 2\.4Position of this work

Prior work includes both successful stopping systems and formal restart optimization\. This study contributes an evaluation and evidence audit for a specified action pair\. The distinction is summarized in[table1](https://arxiv.org/html/2610.36544#S2.T1); the negative screens do not invalidate richer systems with different observations, training, or recovery semantics\.

Table 1:The object measured by adjacent lines of work\. A task\-grounded state can support progress decisions; a trace\-only controller must infer it\.

## 3Black\-Box Progress as a Decision Problem

### 3\.1Setup

Fix a task, agent, model, provider policy, prompt, and tool interface\. One attempt induces latent statesX0,X1,…X\_\{0\},X\_\{1\},\\ldotsand observationsO1,O2,…O\_\{1\},O\_\{2\},\\ldots\. This is a partially observed control process\[[9](https://arxiv.org/html/2610.36544#bib.bib10)\]\. The controller’s information after callttis the filtration

ℱt=σ⁡\(O1,…,Ot\),\\mathcal\{F\}\_\{t\}=\\sigma\(O\_\{1\},\\ldots,O\_\{t\}\),\(1\)where observations may include actions, tool outputs, workspace summaries, external verifier results, and explicit protocol failures\. Let

T=inf\{t≥1:V⁡\(Xt\)=1\}T=\\inf\\\{t\\geq 1:V\(X\_\{t\}\)=1\\\}\(2\)be the first call whose resulting state passes the external verifier, withT=∞T=\\inftyif no such state is reached\. The belief statebt​\(d​x\)=ℙ⁡\(Xt∈d​x∣ℱt\)b\_\{t\}\(dx\)=\\mathbb\{P\}\(X\_\{t\}\\in dx\\mid\\mathcal\{F\}\_\{t\}\)is sufficient for optimal control when the generative process is known\. A black\-box harness does not know this process and must approximate decision value from its observations\. A budget policyπ\\pimaps observable histories to continuation, restart, recovery, branch, or termination actions under total call budgetBB\.

We use*semantic progress*for a latent property of state that predicts future verified completion\. This term does not presume that progress is monotone or one dimensional\. LetQa​\(ht,b\)Q\_\{a\}\(h\_\{t\},b\)denote expected utility when actionaais selected after historyhth\_\{t\}withbbcalls remaining\. For the binary continue\-or\-restart problem, define the continuation advantage

Δ⁡\(ht,b\)=Qcont​\(ht,b\)−Qrestart​\(ht,b\)\.\\Delta\(h\_\{t\},b\)=Q\_\{\\mathrm\{cont\}\}\(h\_\{t\},b\)\-Q\_\{\\mathrm\{restart\}\}\(h\_\{t\},b\)\.\(3\)The utility may combine externally verified success, calls, money, and latency, but it must be declared before evaluating the policy\.

###### Definition 3\.1\(Decision\-sufficient progress representation\)

An observable representationGt=g\(O1:t\)G\_\{t\}=g\(O\_\{1:t\}\)is decision sufficient for a policy class and runtime population if the optimal action is measurable with respect to\(Gt,b\)\(G\_\{t\},b\)\. For binary control, this requires the sign of[equation3](https://arxiv.org/html/2610.36544#S3.E3)to be determined by\(Gt,b\)\(G\_\{t\},b\)almost surely\.

This definition describes lossless action selection relative to the richer history\. A representation can be useful without attaining exact sufficiency\. Embedding displacement, code churn, test counts, and fluent self\-reports may describe a trajectory while discarding a distinction that changes the preferred action\. Two histories may be close in a semantic space yet have different continuation advantages, or look dissimilar while implying the same action\.

###### Proposition 3\.2\(Progress error is advantage\-weighted decision error\)

Letd⁡\(ht,b\)∈\{cont,restart\}d\(h\_\{t\},b\)\\in\\\{\\mathrm\{cont\},\\mathrm\{restart\}\\\}be any stopping rule and letd∗​\(ht,b\)d^\{\*\}\(h\_\{t\},b\)select the action with larger conditional value\. Its expected regret within the binary action class is

ℛ\(d\)=𝔼\[\|Δ\(Ht,b\)\|𝕀\{d\(Ht,b\)≠d∗\(Ht,b\)\}\]\.\\mathcal\{R\}\(d\)=\\mathbb\{E\}\\\!\\left\[\|\\Delta\(H\_\{t\},b\)\|\\mathbb\{I\}\\\{d\(H\_\{t\},b\)\\neq d^\{\*\}\(H\_\{t\},b\)\\\}\\right\]\.\(4\)

Proof\.Conditioned on a history, choosing the inferior action loses exactly the absolute difference between the two action values; choosing the superior action loses zero\. Taking expectation gives[equation4](https://arxiv.org/html/2610.36544#S3.E4)\.□\\square

The identity shows why label accuracy is not the right primary metric for a progress detector\. Errors matter in proportion to the value difference between continuing and restarting\. A detector can achieve high accuracy on histories where the actions are nearly tied and still lose most of the attainable utility on the few histories where\|Δ\|\|\\Delta\|is large\.

Figure 1:Observational aliasing in black\-box control\. Distinct latent states can induce the same observable history while requiring opposite actions\. A useful progress sensor must separate histories according to continuation advantage, not surface activity\.
### 3\.2Fixed restarts depend on the completion scale

Suppose fresh attempts are independent and identically distributed, restart overhead is zero, and a live attempt is truncated at cutoffcc\. WriteF⁡\(t\)=ℙ⁡\(T≤t\)F\(t\)=\\mathbb\{P\}\(T\\leq t\)and decompose the total budget asB=q​c\+rB=qc\+r, whereq=⌊B/c⌋q=\\lfloor B/c\\rfloorand0≤r<c0\\leq r<c\. If unused budget is assigned to a final residual episode, the success probability is

Pc​\(B\)=1−\(1−F⁡\(c\)\)q​\(1−F⁡\(r\)\),P\_\{c\}\(B\)=1\-\\left\(1\-F\(c\)\\right\)^\{q\}\\left\(1\-F\(r\)\\right\),\(5\)withF⁡\(0\)=0F\(0\)=0\. This expression assumes that noncompletion consumes the assigned episode budget\. An explicit terminal event can end an episode earlier and release additional budget, which is handled by trace\-level replay in the experiments\. An uninterrupted attempt succeeds with probabilityF⁡\(B\)F\(B\)\.

The discrete completion hazard makes the missing scale explicit\. Let

λu=ℙ⁡\(T=u∣T≥u\)\.\\lambda\_\{u\}=\\mathbb\{P\}\(T=u\\mid T\\geq u\)\.\(6\)Withrrcalls remaining and no observation beyond survival throughtt, continuation succeeds with probability

C⁡\(t,r\)=1−∏u=t\+1t\+r\(1−λu\),C\(t,r\)=1\-\\prod\_\{u=t\+1\}^\{t\+r\}\(1\-\\lambda\_\{u\}\),\(7\)whereas a fresh attempt succeeds with probabilityR⁡\(r\)=F⁡\(r\)R\(r\)=F\(r\)\. Under equal success utility and zero reset cost, restart is preferred exactly whenR⁡\(r\)\>C⁡\(t,r\)R\(r\)\>C\(t,r\)\. Thus elapsed time helps only through assumptions about the hazard\. A decreasing hazard favors restarts; delayed completion mass can favor continuation\. With trace featuresGtG\_\{t\}, the relevant object is the feature\-conditional residual law, not the marginal curve alone\.

###### Proposition 3\.3\(No uniformly dominant proper cutoff\)

For anyB≥2B\\geq 2and any fixed cutoff1≤c<B1\\leq c<B, there exists a completion\-time distribution for which the uninterrupted policy strictly dominates cutoffcc\. There also exists a completion\-time distribution for which cutoffccstrictly dominates the uninterrupted policy whenever⌊B/c⌋≥2\\lfloor B/c\\rfloor\\geq 2\.

Proof\.For the first claim, place all probability mass atT=c\+1≤BT=c\+1\\leq B\. ThenF⁡\(c\)=0F\(c\)=0andF⁡\(B\)=1F\(B\)=1\. For the second, letT=cT=cwith probabilityp∈\(0,1\)p\\in\(0,1\)and letT=∞T=\\inftyotherwise\. ThenF⁡\(c\)=F⁡\(B\)=pF\(c\)=F\(B\)=p,F⁡\(r\)=0F\(r\)=0, and[equation5](https://arxiv.org/html/2610.36544#S3.E5)equals1−\(1−p\)q\>p1\-\(1\-p\)^\{q\}\>pforq=⌊B/c⌋≥2q=\\lfloor B/c\\rfloor\\geq 2\.□\\square

The proposition is elementary but constraining\. Any positive universal\-restart claim must introduce assumptions on the completion distribution, a prior over task\-agent tuples, or an observation that predicts the residual completion time\. Schedule shape cannot supply this information\.

###### Corollary 3\.4\(Finite\-scale sensitivity\)

Consider any finite restart schedule whose largest episode has lengths<Bs<B\. There is a distribution supported on completion times in\(s,B\]\(s,B\]for which the schedule has zero success and an uninterrupted attempt succeeds with positive probability\.

This corollary does not contradict the classical Luby guarantee\[[16](https://arxiv.org/html/2610.36544#bib.bib9)\]\. It identifies the finite\-budget role of the base scale\. A universal ordering of episode lengths is not a universal calibration of model calls\.

### 3\.3The continuation decision is not identified by survival

At callcc, suppose the only task\-grounded event is that no verified state has appeared\. The remaining budget is enough for either one more call on the current attempt or one call on a fresh attempt\.

###### Proposition 3\.5\(Observational ambiguity\)

There are two latent completion regimes that induce the same observable no\-success history through callccbut require opposite optimal decisions\. In one regime continuation succeeds with probability one and restart fails\. In the other, conditional continuation fails while restart has positive success probability\. Consequently, no rule measurable with respect to that history is optimal in both regimes\.

Proof\.In the late regime, let the current attempt complete deterministically atc\+1c\+1and let a fresh attempt require more than one call\. Continuation is uniquely optimal\. In the fast\-or\-never regime, an attempt completes on its first call with probabilityp\>0p\>0and never completes otherwise\. Conditional on survival throughcc, the current attempt is in the never component, so continuation fails and a fresh attempt succeeds with probabilitypp\. Both regimes can emit the same observation sequence before external completion\. A deterministic rule chooses the wrong action in one regime\. A randomized rule cannot attain the regime\-specific optimum in both\.□\\square

The result is about identification, not metaphysical opacity\. A task\-specific sensor may distinguish the regimes\. Absent such a sensor or a calibrated prior, a controller that restarts because the trace*looks stagnant*imports an untested mapping from appearance to residual value\.

###### Corollary 3\.6\(Regret under observational aliasing\)

In the construction of[proposition3\.5](https://arxiv.org/html/2610.36544#S3.Thmproposition5), suppose a fresh attempt in the fast\-or\-never regime succeeds with probabilityp∈\(0,1\)p\\in\(0,1\)\. For any randomized policy that observes only the aliased history, the worst\-case regret over the two regimes is at leastp/\(1\+p\)p/\(1\+p\)\. The bound approaches1/21/2asppapproaches one\.

Proof\.Let the policy continue with probabilityα\\alpha\. Its regret is1−α1\-\\alphain the late regime andp​αp\\alphain the fast\-or\-never regime\. Minimizingmax⁡\{1−α,p​α\}\\max\\\{1\-\\alpha,p\\alpha\\\}givesα=1/\(1\+p\)\\alpha=1/\(1\+p\)and regretp/\(1\+p\)p/\(1\+p\)\.□\\square

The corollary gives an operational meaning to unobservability\. It is not merely that a trace classifier may be inaccurate\. Without an identifying assumption, error remains even with unlimited computation applied to the same observable history\.

An additional failure\-selection example and the standard correlated\-ensemble variance identity appear in Appendix[G](https://arxiv.org/html/2610.36544#A7)\. They motivate the branch\-reuse screens but are not empirical estimates of our patch population\.

## 4From Failure Prediction to Restart Decisions

The useful question is whether a signal changes expected utility under a specified intervention\. This section connects the classical value of information\[[1](https://arxiv.org/html/2610.36544#bib.bib4)\]to a finite\-budget audit that can be computed without model access\. The identities specialize existing decision theory; the contribution is their operationalization for agent termination and recovery, together with an explicit account of what the available traces can identify\.

### 4\.1Failure prediction must be combined with restartability

Fix one task and runtime, a live decision at calltt, and residual budgetb=B−tb=B\-t\. LetYCY\_\{C\}be the potential binary outcome of uninterrupted continuation to the deadline\. Letq=ℙ⁡\(YC=1\)q=\\mathbb\{P\}\(Y\_\{C\}=1\), and letppbe the success probability of one independent clean restart with the same residual budget\. We callppthe*restartability*of this decision\. An alarmA=1A=1triggers the restart\. Define its failure true\-positive rateτ=ℙ⁡\(A=1∣YC=0\)\\tau=\\mathbb\{P\}\(A=1\\mid Y\_\{C\}=0\)and false\-positive rateϕ=ℙ⁡\(A=1∣YC=1\)\\phi=\\mathbb\{P\}\(A=1\\mid Y\_\{C\}=1\)\.

###### Proposition 4\.1\(Alarm\-to\-action value identity\)

Assume the fresh outcome is independent of\(A,YC\)\(A,Y\_\{C\}\)conditional on the fixed task and runtime, the restart restores the initial state, and the alarm consumes no budget\. The success change relative to continuing every live attempt is

G⁡\(q,p,τ,ϕ\)=\(1−q\)​τ​p−q​ϕ​\(1−p\)\.G\(q,p,\\tau,\\phi\)=\(1\-q\)\\tau p\-q\\phi\(1\-p\)\.\(8\)If\(1−q\)​τ\+q​ϕ\>0\(1\-q\)\\tau\+q\\phi\>0, a positive gain requires

p\>q​ϕ\(1−q\)​τ\+q​ϕ\.p\>\\frac\{q\\phi\}\{\(1\-q\)\\tau\+q\\phi\}\.\(9\)

Proof\.On correctly alarmed failures, the restart changes success from zero topp, contributing\(1−q\)​τ​p\(1\-q\)\\tau p\. On falsely alarmed successes, it changes success from one topp, contributing−q​ϕ​\(1−p\)\-q\\phi\(1\-p\)\. Unalarmed outcomes are unchanged\. Rearrangement gives the threshold\.□\\square

Forq=1/2q=1/2,τ=0\.9\\tau=0\.9, andϕ=0\.1\\phi=0\.1, the detector’s classification accuracy is90%90\\%for everypp, whileG=0\.5​p−0\.05G=0\.5p\-0\.05\. Thus identical alarm quality gives a five\-point loss atp=0p=0, no change atp=0\.1p=0\.1, and a 35\-point gain atp=0\.8p=0\.8\. Even perfect failure prediction produces no additional successes whenp=0p=0, although early termination can still save cost\. These are analytical controls, not measured monitor performance\. They motivate joint reporting of alarm discrimination, residual\-budget restartability, and realized utility\. A global accuracy or AUROC alone omits the second quantity\.

An edit\-preserving restart changes the intervention and need not satisfy the independence assumption\. For a general alarm, the exact identity isG=𝔼⁡\[A⁡\(YR−YC\)\]G=\\mathbb\{E\}\[A\(Y\_\{R\}\-Y\_\{C\}\)\]\. Computing it requires outcomes for the specified recovery action\. Mixing task populations can also induce dependence, even when fresh draws are independent within each task\. We therefore estimate and interpret action values within task and runtime before aggregation\.

###### Corollary 4\.2\(Opportunity ceiling for a failure alarm\)

Under the assumptions of[proposition4\.1](https://arxiv.org/html/2610.36544#S4.Thmproposition1), any alarm satisfies

G≤\(1−q\)​p\.G\\leq\(1\-q\)p\.\(10\)Equality is attained by a perfect alarm,A=1−YCA=1\-Y\_\{C\}\. The additional value of this clairvoyant alarm over the better constant action is

\(1−q\)​p−\(p−q\)\+=min⁡\{\(1−q\)​p,q⁡\(1−p\)\}\.\(1\-q\)p\-\(p\-q\)\_\{\+\}=\\min\\\{\(1\-q\)p,\\ q\(1\-p\)\\\}\.\(11\)

Proof\.In[equation8](https://arxiv.org/html/2610.36544#S4.E8), useτ≤1\\tau\\leq 1andϕ≥0\\phi\\geq 0\. Subtract the better constant\-action gain\(p−q\)\+\(p\-q\)\_\{\+\}and distinguishp≥qp\\geq qfromp<qp<q\.□\\square

The ceiling is a value\-of\-perfect\-information benchmark for one specified action pair\. It is neither a realizable detector nor an upper bound for state\-preserving recovery, multiple restarts, or another model\. A plug\-in estimate of this ceiling can diagnose an evaluation set with little measured opportunity; it does not establish that the true opportunity is small without sampling uncertainty\.

### 4\.2The decision value of a richer progress signal

LetZZbe a richer observation and letD⁡\(Z\)=QR​\(Z\)−QC​\(Z\)=−Δ⁡\(Z\)D\(Z\)=Q\_\{R\}\(Z\)\-Q\_\{C\}\(Z\)=\-\\Delta\(Z\)denote restart advantage\. A coarser representation isG=g⁡\(Z\)G=g\(Z\)\. Fix the residual budget and population of live decisions\. The best policy usingGGhas value

V⁡\(G\)=𝔼⁡\[QC​\(Z\)\]\+𝔼⁡\[\(𝔼⁡\[D⁡\(Z\)∣G\]\)\+\],V\(G\)=\\mathbb\{E\}\[Q\_\{C\}\(Z\)\]\+\\mathbb\{E\}\\left\[\(\\mathbb\{E\}\[D\(Z\)\\mid G\]\)\_\{\+\}\\right\],\(12\)where\(x\)\+=max⁡\(x,0\)\(x\)\_\{\+\}=\\max\(x,0\)\.

###### Proposition 4\.3\(Value lost by compressing a progress signal\)

The loss from replacingZZbyG=g⁡\(Z\)G=g\(Z\)is

V⁡\(Z\)−V⁡\(G\)=12​\{𝔼​\|D⁡\(Z\)\|−𝔼​\|𝔼⁡\[D⁡\(Z\)∣G\]\|\}≥0\.V\(Z\)\-V\(G\)=\\frac\{1\}\{2\}\\left\\\{\\mathbb\{E\}\|D\(Z\)\|\-\\mathbb\{E\}\|\\mathbb\{E\}\[D\(Z\)\\mid G\]\|\\right\\\}\\geq 0\.\(13\)It vanishes when each coarse observation contains advantages of only one sign, up to zero\-probability events\. In particular, preserving the optimal action can preserve decision value without reconstructing the original trace\.

Proof\.Usingx\+=\(x\+\|x\|\)/2x\_\{\+\}=\(x\+\|x\|\)/2in both values cancels the common mean advantage\. Conditional Jensen’s inequality proves nonnegativity\. Equality in the absolute\-value inequality holds exactly when positive and negative advantages do not coexist within a conditional observation class\.□\\square

This gives a task\-specific criterion for context compression: preserve distinctions that change the preferred action\. Token retention, embedding fidelity, and reconstruction quality do not by themselves measure this loss\. The identity is a binary\-action specialization of classical information comparison, not a new universal information theorem\. It also permits imperfect signals to have positive value; exact decision sufficiency is an ideal reference, not a prerequisite for practical improvement\. If acquiringZZreduces expected utility byκ\\kappawhile otherwise preserving the action\-value model, its net benefit isV⁡\(Z\)−V⁡\(G\)−κV\(Z\)\-V\(G\)\-\\kappa\. When acquisition consumes calls or changes the workspace, action values must instead be evaluated at the resulting budget and state\.

### 4\.3A finite\-sample decision certificate

LetEEdenote explicit protocol termination\. At a fixedtt, a trace is eligible only if it is still live:

Lt=\{T\>t,E\>t\}\.L\_\{t\}=\\\{T\>t,\\ E\>t\\\}\.\(14\)For taskjj, the audited comparison is uninterrupted continuation throughBBversus one clean restart withB−tB\-tcalls:

qj,t=ℙ⁡\(T≤B∣Lt,j\),pj,t=ℙ⁡\(T≤B−t∣j\)\.q\_\{j,t\}=\\mathbb\{P\}\(T\\leq B\\mid L\_\{t\},j\),\\qquad p\_\{j,t\}=\\mathbb\{P\}\(T\\leq B\-t\\mid j\)\.\(15\)The first quantity is estimated from eligible attempts; the second from all fresh attempts on that task\. A later protocol terminal counts as unsuccessful continuation\. An earlier terminal is outside the live\-decision population\. Neither arm extrapolates beyond the observed horizon\.

For a fixed family ofMMtask\-time comparisons, form two\-sided Clopper\-Pearson intervals\[[3](https://arxiv.org/html/2610.36544#bib.bib5)\]for everyqj,tq\_\{j,t\}andpj,tp\_\{j,t\}, each with error probabilityα/\(2​M\)\\alpha/\(2M\)\. Write them as\[LC,UC\]\[L\_\{C\},U\_\{C\}\]and\[LR,UR\]\[L\_\{R\},U\_\{R\}\]\. An empty eligible sample receives\[0,1\]\[0,1\]\. The decision audit returns

𝒟j,t=\{restart,LR\>UC,continue,UR<LC,abstain,otherwise\.\\mathcal\{D\}\_\{j,t\}=\\begin\{cases\}\\mathrm\{restart\},&L\_\{R\}\>U\_\{C\},\\\\ \\mathrm\{continue\},&U\_\{R\}<L\_\{C\},\\\\ \\mathrm\{abstain\},&\\text\{otherwise\}\.\\end\{cases\}\(16\)
###### Proposition 4\.4\(Simultaneous sign certification\)

For IID attempts within each fixed task and runtime, complete observation throughBB, and a comparison family fixed independently of the evaluated outcomes, every nonabstaining decision in[equation16](https://arxiv.org/html/2610.36544#S4.E16)has the correct sign with probability at least1−α1\-\\alpha\.

Proof\.Each binomial interval has its stated marginal coverage, including the continuation interval conditional on the number of eligible attempts\. A union bound over the2​M2Mintervals gives simultaneous coverage\. On that event,pj,t−qj,t∈\[LR−UC,UR−LC\]p\_\{j,t\}\-q\_\{j,t\}\\in\[L\_\{R\}\-U\_\{C\},U\_\{R\}\-L\_\{C\}\], so either strict separation certifies the sign\. Independence between the two estimated arms is unnecessary for the union bound; the samples may overlap\.□\\square

This certificate implements a simple form of conservative policy improvement\[[13](https://arxiv.org/html/2610.36544#bib.bib6)\]\. It is a sufficient evidence rule, not an optimal test or an impossibility bound for all estimators\. Abstention means that this procedure does not resolve the comparison\. Falling back to continuation after abstention does not certify continuation as optimal\. Deployment also requires the calibration population to match the future runtime; the guarantee does not cover arbitrary distribution shift or uncorrected repeated checking\.

Evidence cost at zero success\.Fornnobservations with zero successes, the upper endpoint under this interval allocation is

U0​\(n,M\)=1−\(α4​M\)1/n\.U\_\{0\}\(n,M\)=1\-\\left\(\\frac\{\\alpha\}\{4M\}\\right\)^\{1/n\}\.\(17\)To obtainU0≤ϵU\_\{0\}\\leq\\epsilonrequires

n≥⌈log⁡\(α/\(4​M\)\)log⁡\(1−ϵ\)⌉\.n\\geq\\left\\lceil\\frac\{\\log\(\\alpha/\(4M\)\)\}\{\\log\(1\-\\epsilon\)\}\\right\\rceil\.\(18\)These counts describe this specified binomial procedure\. They quantify the evidence needed to bound one success probability, not sufficient sample sizes to establish an intervention effect\. Conditional continuation additionally requires that these observations reach the decision alive\.

![Refer to caption](https://arxiv.org/html/2610.36544v1/monitor_value_frontier.png)Figure 2:Analytical controls for monitor evaluation\. Panel A keeps the failure detector’s true\-positive and false\-positive rates fixed while varying continuation success and restartability\. The black curve is the zero\-gain boundary\. Panel B gives the number of zero\-success observations required by the specified simultaneous interval procedure to bound a success probability\. Neither panel reports newly collected agent performance\.

## 5Experimental Method

### 5\.1Design principle

We use a sequential falsification design\. Each experiment was motivated by the preceding result, but its task rule, intervention, outcome, and stopping rule were frozen before new outcomes were observed\. Earlier data are used only where stated\. This structure does not make an adaptive research program equivalent to one preregistered confirmatory study\. It does prevent within\-stage changes to the tested intervention after seeing its result\.

Table 2:Experimental ladder\. Model attempts count actual agent or resolver executions\. Patch algebra used existing failed workspaces and no model calls\.
### 5\.2Benchmark and runtime tuple

We use 19 issues from the SymPy repository in SWE\-bench Verified\[[8](https://arxiv.org/html/2610.36544#bib.bib14)\]\. Restricting to one repository removes language and build\-system variation, while retaining issue\-level differences in symbolic mathematics, printing, assumptions, matrices, and related subsystems\. It also limits external validity, which we treat explicitly in[section7\.3](https://arxiv.org/html/2610.36544#S7.SS3)\.

An experimental runtime is identified by benchmark task, model endpoint, agent release, provider route, prompt configuration, and sampling policy\. We use mini\-swe\-agent 2\.4\.6\[[24](https://arxiv.org/html/2610.36544#bib.bib16)\]\. Attempts use temperature0\.70\.7, restore the benchmark\-specified repository state before each run, execute under WSL2 Bubblewrap isolation without network access, and expose a workspace visible only to the attempt\. The initial comparison usesopenai/gpt\-5\.6\-lunaanddeepseek/deepseek\-v4\-flash\-0731\. All subsequent model\-driven experiments use the former\.

The primary budget unit is a model call\. Dollar cost and wall time are secondary operational measures\. A 30\-call attempt need not emit 30 valid environment actions, and an explicit format failure can end it early\. We retain the distinction between nominal horizon, first verified completion, and protocol\-terminal call\.

Formally, letEEbe the first explicit protocol\-terminal call and letHHbe the configured horizon\. The trace records

U=min⁡\{T,E,H\},J∈\{verified,protocol terminal,right censored\}\.U=\\min\\\{T,E,H\\\},\\qquad J\\in\\\{\\text\{verified\},\\text\{protocol terminal\},\\text\{right censored\}\\\}\.\(19\)The event labelJJmatters for replay\. A protocol terminal can release calls to a new episode; a right\-censored live trajectory cannot be treated as known failure beyondHH\.

Table 3:Runtime controls shared across model\-driven experiments unless a stage states otherwise\.
### 5\.3External verification and completion time

Each selected task passes two environment gates before collection: the benchmark\-provided starting state fails the task evaluator and the reference patch passes it\. The evaluator and test patch remain outside the agent sandbox\. After every environment action, the controller transfers the candidate diff into a clean evaluator and runs the hidden fail\-to\-pass tests\. Completion time is the first passing state\. This prevents a later destructive action or an inaccurate self\-declaration from changing the recorded time\.

Every successful model attempt was audited for changes to the verifier, test runner, or hidden tests\. None modified those assets\. Successful patches changed implementation files, with occasional scratch reproductions\. The analysis never exposes hidden verifier output, the reference patch, or completion labels to the model\.

### 5\.4Restart replay

The initial cohort contains five nominally independent 30\-call attempts for each of seven tasks and two models, totaling 70 observations\. We evaluate a fixed cutoffccunder total budget 30 by empirical resampling of fresh attempts within task\. Write30=q​c\+r30=qc\+r, withq=⌊30/c⌋q=\\lfloor 30/c\\rfloorand0≤r<c0\\leq r<c\. In the absence of an earlier protocol\-terminal event, the plug\-in probability is

P^j,c​\(30\)=1−\(1−F^j​\(c\)\)q​\(1−F^j​\(r\)\)\.\\widehat\{P\}\_\{j,c\}\(30\)=1\-\\left\(1\-\\widehat\{F\}\_\{j\}\(c\)\\right\)^\{q\}\\left\(1\-\\widehat\{F\}\_\{j\}\(r\)\\right\)\.\(20\)The exact replay additionally releases unused calls when a recorded protocol\-terminal event occurs before the assigned cutoff\. Macro success averages the resulting probability across tasks\. Cutoffs range from 4 to 30 in steps of two\. The full\-horizon cutoff is the uninterrupted baseline\. For geometric and Luby schedules, the implementation uses Monte Carlo replay with 50,000 episode sequences per task\.

Cutoff selection is evaluated in sample and by leave\-one\-task\-out selection\. Each fold selects the best cutoff on the other tasks, breaking ties toward the longer cutoff, then evaluates it on the held\-out task\. No attempt\-level split is used because attempts from one issue share task difficulty\.

### 5\.5Prospective task cohorts

The first new cohort selects the seven highest\-ID unseen SymPy tasks that pass the environment gates\. This mechanical cohort produced 35 attempts\. Because all were solved early, a second difficulty\-enriched cohort targeted unseen issues labeled at least one hour, then filled rejected positions from the highest\-ID unseen issues labeled 15 minutes to one hour\. Five tasks passed the gates, yielding 25 attempts\. The transferred cutoff of 16 calls and the 30\-call baseline were fixed before either outcome set was observed\.

The 60\-call extension contains five fresh attempts on each of four stress tasks that were0/50/5at horizon 30\. Its primary outcome is any first completion in calls 31 through 60\. A secondary analysis separates right\-censoring from explicit repeated\-format termination\.

### 5\.6Context\-rejuvenation intervention

Four tasks that exhibited repeated\-format termination receive five fresh baseline and five treatment attempts each\. Both conditions have a shared 60\-call cap\. After three consecutive invalid tool responses, treatment preserves the exact repository state, discards the conversation, instantiates a fresh model client, and supplies the original issue plus deterministicgit statusand diff\-stat summaries\. It receives no extra calls, hidden feedback, progress score, or model checkpoint\. Baseline follows the unmodified harness behavior\.

The prospective expansion rule required at least a five\-point success gain or two recovered successes, with no more than a20%20\\%mean\-cost increase\. The rule was frozen before collection\.

### 5\.7Failed\-branch reuse

The context experiment yielded 38 externally failed, nonempty, production\-code\-only terminal patches after excluding one verified patch and one empty patch\. We deduplicated them by content, enumerated unordered within\-task pairs, and performed clean three\-way composition against the common benchmark starting state\. Candidate generation used no model, verifier output, or gold patch\. The ranking was frozen before verification and allowed at most eight candidates per task\. Nineteen candidates were evaluated\.

The semantic\-resolver screen then supplied one clean 12\-call Luna attempt per task with the issue and every unique failed patch for that task, each explicitly labeled as a failed hypothesis\. The resolver could inspect and edit a clean workspace\. The four contexts were frozen before execution\. The protocol allowed no retries, and a second stage would open only after at least one verified success\.

### 5\.8Retrospective decision\-value audit

We reanalyze the 70 initial and 60 prospective 30\-call attempts, without additional model calls\. These 130 observations form 26 task\-model cells with five attempts each\. The audit evaluatest∈\{4,8,12,16,20,24\}t\\in\\\{4,8,12,16,20,24\\\}at total budgetB=30B=30, giving a family ofM=156M=156comparisons\. It uses the live eligibility rule and binomial intervals in[section4\.3](https://arxiv.org/html/2610.36544#S4.SS3), withα=0\.05\\alpha=0\.05\. Empty live populations are reported as not reached, not as successful abstentions\. All attempt identities are deduplicated before analysis\.

The decision is a single clean restart using the residual budget, not the repeated\-cutoff policy used in the earlier replay\. We report conditional action values within each task, then weight the change by the observed fraction reaching the decision and average tasks equally\. A task oracle chooses the better estimated action separately for each task; a perfect failure alarm additionally knows which live continuations will fail, as in[corollary4\.2](https://arxiv.org/html/2610.36544#S4.Thmproposition2)\. Both are retrospective plug\-in references, not deployed policies or validated opportunity bounds\. In particular, zero observed restart success does not establish zero true restartability\.

This audit was designed after inspection of the collected experiments\. Its comparison family and analysis are fully specified for reproduction, but the audit is exploratory, not preregistered or a prospective test of the coverage guarantee\. It supplies no paired state\-conditioned intervention outcomes and does not evaluate a learned prefix monitor\.

### 5\.9Uncertainty and reporting

For policy contrasts, percentile bootstrap samples tasks with replacement and then attempts within sampled task\. Unless specified, intervals use 10,000 replicates and report the empirical2\.5%2\.5\\%and97\.5%97\.5\\%quantiles\. We report bootstrap sign probabilities as descriptive quantities, not posterior probabilities or frequentistppvalues\. Boundary outcomes are accompanied by an unclustered Newcombe interval as a sensitivity check where useful\. All success estimates are absolute percentage\-point differences\.

For taskjjand policyπ\\pi, letθj,π​\(B\)\\theta\_\{j,\\pi\}\(B\)be the probability of external verification within total budgetBB\. The primary estimand is the task\-macro policy value

Θπ​\(B\)=1J​∑j=1Jθj,π​\(B\),\\Theta\_\{\\pi\}\(B\)=\\frac\{1\}\{J\}\\sum\_\{j=1\}^\{J\}\\theta\_\{j,\\pi\}\(B\),\(21\)and a policy contrast isΔπ,π0=Θπ​\(B\)−Θπ0​\(B\)\\Delta\_\{\\pi,\\pi\_\{0\}\}=\\Theta\_\{\\pi\}\(B\)\-\\Theta\_\{\\pi\_\{0\}\}\(B\)\. Equal task weighting prevents an issue with more retained attempts from dominating the result\. The hierarchical bootstrap targets variability over both sampled tasks and provider draws within task\. With only one repository and purposefully selected cohorts, it should not be read as a population interval over all software engineering tasks\.

Unsuccessful attempts at a horizon are right\-censored: they establishT\>cT\>c, notT=∞T=\\infty\[[10](https://arxiv.org/html/2610.36544#bib.bib11)\]\. Offline replay never evaluates a cutoff beyond the observed horizon\. Explicit protocol termination is a competing event, recorded at its observed call, and may release the residual budget to a new attempt\. Treating these terminal events as ordinary censoring would overcharge the episode and obscure the distinction between time\-based truncation and fault recovery\.

## 6Results

[Table4](https://arxiv.org/html/2610.36544#S6.T4)states the principal contrasts before the stage\-by\-stage analysis\. The paper gives priority to frozen prospective comparisons over in\-sample maxima\.

Table 4:Main empirical results\. Intervals resample tasks and attempts unless noted\.### 6\.1A restart scale does not transfer across models

The initial 70 attempts already contain two different completion\-time regimes under the same tasks and harness\. Luna solves 29 of 35 attempts, mostly early\. DeepSeek solves 24 of 35, but many completions occur after call 16\. Per\-task outcomes appear in[table5](https://arxiv.org/html/2610.36544#S6.T5); the empirical completion distributions and fixed\-cutoff policy values appear in[figure3](https://arxiv.org/html/2610.36544#S6.F3)\.

Table 5:Initial externally verified outcomes\. Call statistics are conditional on successful attempts\.Figure 3:Model\-specific completion scale and restart\-policy value on the initial seven tasks\. Panel A reports the task\-macro probability that one attempt has verified by callcc\. Panel B reports the exact plug\-in success of repeated cutoffccunder 30 total calls\. Shading gives pointwise 95% hierarchical bootstrap intervals\. Enlarged points mark the Luna\-selected cutoff of 16 calls\.The best Luna cutoff spans calls 16 through 20 and reaches85\.14%85\.14\\%, compared with82\.86%82\.86\\%for one 30\-call attempt\. At cutoff 16, the estimated change is\+2\.29\+2\.29points with a task\-hierarchical interval of\[0\.00,6\.86\]\[0\.00,6\.86\]and bootstrap positive fraction0\.5040\.504\. This is weak in\-sample evidence\.

Applying the same 16\-call scale to DeepSeek changes success from68\.57%68\.57\\%to26\.29%26\.29\\%, a−42\.29\-42\.29point effect with interval\[−69\.14,−17\.71\]\[\-69\.14,\-17\.71\]\. DeepSeek’s best cutoff is the full 30 calls\. The direction of transfer in the other direction is conservative: applying 30 calls to Luna retains82\.86%82\.86\\%and gives up only the small in\-sample oracle difference\.

Leave\-one\-task\-out selection removes the apparent Luna benefit\. It yields75\.54%75\.54\\%against82\.86%82\.86\\%for the 30\-call baseline, a−7\.31\-7\.31point difference\. The decisive held\-out task is 24443: the other six tasks select cutoff 10, while all five successful runs of task 24443 require 8 to 16 calls and repeated cutoff 10 has only48\.8%48\.8\\%plug\-in success\. DeepSeek selects 30 calls in every fold and ties its baseline\.

Finite geometric schedules reach at most approximately84\.59%84\.59\\%on Luna and22\.88%22\.88\\%on DeepSeek under the preregistered family\. The tested Luby schedules range from45\.72%45\.72\\%to67\.40%67\.40\\%on Luna and have zero success on DeepSeek because none allocates an episode long enough for an observed completion\. This is the scale sensitivity of[proposition3\.3](https://arxiv.org/html/2610.36544#S3.Thmproposition3), not a contradiction of asymptotic restart theory\.

Finding\.The same tasks and harness do not determine a portable restart scale\. Changing only the model moves useful completion mass beyond the selected cutoff and turns a small apparent gain into a large loss\.

### 6\.2The in\-sample gain does not survive prospective transfer

The two prospective cohorts add 60 Luna attempts\. All 35 attempts in the mechanically selected high\-ID cohort pass by call 15, so both policies score100%100\\%\. In the difficulty\-enriched cohort, only one of 25 attempts passes, at call 23\. Fixed 16 therefore scores zero while fixed 30 scores4%4\\%\. Results are summarized in[table6](https://arxiv.org/html/2610.36544#S6.T6)\.

Table 6:Transfer of the Luna\-selected 16\-call cutoff\. The interval resamples tasks and attempts\.Across all 19 Luna tasks, fixed 16 scores68\.21%68\.21\\%and fixed 30 scores68\.42%68\.42\\%\. The difference is−0\.21\-0\.21points with interval\[−4\.21,2\.53\]\[\-4\.21,2\.53\]and bootstrap positive fraction0\.3170\.317\. The pooled in\-sample oracle is cutoff 24 at68\.84%68\.84\\%, only0\.420\.42points above fixed 30\. Leave\-one\-task\-out cutoff selection scores67\.37%67\.37\\%, a−1\.05\-1\.05point difference from fixed 30\.

Under total budget 60, a conservative policy of 15, 15, then a reserved 30\-call attempt scores69\.43%69\.43\\%\. Two 30\-call attempts score70\.11%70\.11\\%, for a difference of−0\.67\-0\.67points with interval\[−2\.53,0\.84\]\[\-2\.53,0\.84\]\. Reserving a full episode avoids catastrophic truncation, but the short probes displace another useful long attempt\.

Finding\.The positive cutoff result does not replicate prospectively\. Easy tasks create no headroom, while the only stress\-cohort success occurs after the selected cutoff\.

### 6\.3Policy differences do not certify individual restart decisions

The retrospective audit separates where a decision occurs, which action has higher estimated value, and whether the estimate resolves its sign\. Of 156 task\-time cells, 103 contain at least one live attempt; the other 53 are never reached\. None of the 103 live comparisons is certified by the simultaneous interval rule\. This is a finite\-sample limitation of the specified conservative procedure, not evidence that all actions have equal value\.

At the previously selected cutoff of 16, only 13 task\-model cells remain live\.[Table7](https://arxiv.org/html/2610.36544#S6.T7)summarizes the decision opportunities;[figure4](https://arxiv.org/html/2610.36544#S6.F4)displays conditional advantages and uncertainty\. Five initial Luna tasks have completed in every attempt\. Task 23413 has one survivor among five\. For that survivor, estimated continuation success is zero and fresh success within the remaining 14 calls is4/54/5\. Its task\-macro contribution is therefore

17⋅15⋅45=0\.022857\.\\frac\{1\}\{7\}\\cdot\\frac\{1\}\{5\}\\cdot\\frac\{4\}\{5\}=0\.022857\.\(22\)The remaining live Luna task, 23950, has no observed success in either arm\. Thus the entire2\.292\.29\-point initial gain is supplied by one surviving failed attempt and the early successes of other draws on that same issue\. The headline effect is not evidence of a broad stagnant\-trajectory phenotype\.

Table 7:Decision opportunities at call 16 with total budget 30\. Changes are task\-macro percentage points relative to continuation\. The oracle knows future continuation failure and uses the specified independent clean restart\. Its values are plug\-in diagnostics, not statistically validated upper bounds\. No live cell receives a sign certificate\.All six live DeepSeek task cells favor continuation in the plug\-in estimates\. The prospective high\-ID cohort supplies no live decision at all: both policies tie because completion precedes the alarm time\. The stress cohort supplies many failures, but no observed fresh completion within 14 calls\. This is a failure\-rich evaluation set with little measured recovery opportunity under the chosen action\. A detector cannot be usefully assessed for completion gains merely by selecting trajectories likely to fail\.

The perfect\-alarm calculation in[corollary4\.2](https://arxiv.org/html/2610.36544#S4.Thmproposition2)adds no estimated completion gain beyond continuation in either prospective cohort or the initial DeepSeek cohort at this decision time\. This statement is narrowly conditional on the empirical distributions, call 16, and one independent clean restart\. It does not bound a richer recovery action, a different decision time, or the true opportunity concealed by limited sampling\. Even oracle diagnostics require uncertainty assessment\.

Figure 4:Conditional restart advantage in the 13 task\-model cells with a live attempt at call 16\. Dots are plug\-in estimates\. Lines are simultaneous difference bounds formed from binomial intervals across all 156 audited comparisons\. Positive values favor restarting\. Every interval includes zero despite substantial heterogeneity in point estimates\.The lack of individual certificates is compatible with the large cross\-model policy loss\. The earlier bootstrap measures an aggregate policy contrast across sampled tasks; the audit asks for the sign of each conditional action comparison with simultaneous protection\. Five attempts per task, sometimes reduced to one live continuation, cannot provide comparable resolution\. For example,[equation18](https://arxiv.org/html/2610.36544#S4.E18)requires 90 zero\-success observations to reduce the specified upper bound to10%10\\%whenM=156M=156, versus 42 for one decision\. These are procedure\-specific evidence costs, not lower bounds on every possible learning method\.

Finding\.Failure prevalence, decision opportunity, and statistical support are different quantities\. The same audit set can contain many failures, negligible measured recovery opportunity, and insufficient evidence to certify any individual action\.

### 6\.4Additional horizon reveals protocol collapse, not late success

The horizon extension tests four tasks that were previously0/50/5at 30 calls\. Among 20 fresh 60\-call attempts, one passes at call 17\. No completion first appears during calls 31 through 60\. An unclustered one\-sided95%95\\%binomial upper bound is13\.9%13\.9\\%\. This attempt\-level sensitivity calculation does not provide population coverage over heterogeneous tasks; rare late completion and repository\-level variation remain unresolved\.

Seventeen attempts end after three consecutive invalid tool responses and two exhaust the horizon\. On the same four tasks at horizon 30, only one of 20 attempts had ended in repeated\-format failure\. The attempt\-level Fisher exact comparison of 17/20 with 1/20 givesp=4\.06×10−7p=4\.06\\times 10^\{\-7\}; it is descriptive because attempts are clustered within four tasks\. Fourteen of the 17 format failures occur after call 30\.

Table 8:Fresh 60\-call outcomes on tasks previously unsolved in five 30\-call attempts\.At total budget 60, repeated fixed 30 scores9%9\\%by empirical replay, compared with5%5\\%for one 60\-call attempt\. A policy that allows the full 60 calls but immediately recycles budget after explicit terminal failure also scores9%9\\%\. Hence the apparent restart benefit comes entirely from reusing calls after an observable protocol terminal event\. Arbitrarily censoring a live trajectory adds nothing\.

Finding\.More nominal horizon increases exposure to protocol failure but reveals no observed late completion\. Recycling budget after an explicit failure is distinct from declaring a still\-running trajectory stagnant\.

### 6\.5Fresh context changes the terminal label, not success

The context intervention fires in 18 of 20 treated attempts\. Baseline solves one of 20; treatment solves none\. The task\-clustered bootstrap difference is−5\-5points with empirical range\[−20,0\]\[\-20,0\]\. Since treatment has no successes, this range is boundary\-degenerate and cannot represent a positive treatment event\. An unclustered Newcombe sensitivity interval is\[−23\.6,15\.2\]\[\-23\.6,15\.2\]points\.

The treatment does eliminate the finalRepeatedFormatErrorlabel\. Every triggered attempt instead consumes its remaining calls, and all 20 treated attempts end atLimitsExceeded\. Mean dollar cost rises from $0\.03786 to $0\.05934, a56\.7%56\.7\\%increase\. Mean wall time rises from 309\.5 to 512\.8 seconds, a65\.7%65\.7\\%increase\.[Figure5](https://arxiv.org/html/2610.36544#S6.F5)displays the failure\-mode substitution\.

Figure 5:Context rejuvenation changes the observed failure mode without improving task success\. Panel A shows terminal outcomes, with counts and within\-condition percentages\. Panel B shows the corresponding increase in mean metered cost and wall time relative to baseline\.Table 9:Context\-rejuvenation outcomes under the shared 60\-call budget\.The frozen expansion criterion is not met\. Fresh inference context removes the terminal interface symptom but preserves the workspace, including potentially wrong edits\. The tested reset does not support conversational contamination as a sufficient explanation for recoverable failure\. It does not isolate which retained edits or other mechanisms caused the remaining failures\.

Finding\.An intervention can improve a harness diagnostic by 90 points while leaving success no better and making cost materially worse\. Failure\-mode movement is not task progress\.

### 6\.6Failed branches are diverse but not compositional

The 40\-run context experiment yields 38 admissible failed sources and 37 unique patch contents\. Of 162 unordered source pairs, 51 form clean, novel three\-way composites\. The frozen verifier\-blind ranking selects 19\. None passes\. Task 17630 produces no clean novel composite because branches edit the same matrix\-code region incompatibly\.

Table 10:Verifier\-blind patch composition from failed terminal branches\.One bounded semantic resolver is then run per task\. It receives every unique failed patch as an explicitly failed hypothesis and starts from a clean workspace\. All four runs use the 12\-call cap and fail verification\. The patches are not copies: their maximum character\-level diff similarity to any supplied parent is 0\.563, 0\.700, 0\.645, and 0\.802\. The task\-level relevant tests pass in counts 76/79, 21/22, 99/100, and 47/52, but the protocol forbids extending the near miss after outcome inspection\.

The historical clean\-context cohort also has no completion within 12 calls on these tasks, so the resolver screen provides no evidence of improvement and is not treated as a randomized effect estimate\. The null result has a more focused interpretation: syntactic novelty and access to multiple failed views are insufficient evidence of independent corrective information\.

Finding\.Failed branches occupy different textual locations without supplying a verified closure operation\. Clean union and shared\-model synthesis both preserve diversity without establishing correctness\.

Figure 6:Five bounded tests of restart and recovery\. Outcomes identify limitations of the tested interventions, not impossibility results for richer controllers\. The task populations and uncertainty differ across stages\.

## 7Discussion

### 7\.1Contribution and scope

The contribution is an operational account of when progress information can improve a specified agent\-control decision\. Classical restart optimization already allows informative observations, and contemporary learned monitors demonstrate successful termination and recovery\. Our results do not overturn those findings\. They connect failure discrimination to residual\-budget restartability, distinguish decision\-preserving compression from trace reconstruction, and expose the sampling support of per\-task action comparisons\.

Three levels of evidence must remain separate\. The mathematical identities hold under stated assumptions and specialize established decision theory\. The retrospective audit diagnoses the collected traces and supplies a reusable calculation, but does not demonstrate a deployed policy gain\. The prospective intervention screens establish outcomes of particular controllers on selected tasks, not universal limits on monitoring or recovery\. Combining these levels makes the paper useful only if their inferential boundaries are preserved\.

### 7\.2Implications for evaluation and deployment

Measure opportunity before monitor quality\.A failure detector should be evaluated where an alternative action can plausibly recover value\. A ceiling cohort offers no live decisions; a hard cohort can offer many failures but little residual\-budget restartability\. The perfect\-alarm reference in[corollary4\.2](https://arxiv.org/html/2610.36544#S4.Thmproposition2)is a diagnostic for this distinction\. A small plug\-in ceiling warrants additional evidence or a different intervention, not a conclusion that every progress signal is useless\.

Match the signal to the action\.A clean restart, conversational reset with mandatory workspace retention, and optional reuse of previous edits define different potential outcomes\. Their action values cannot be interchanged\. FailFast\-RestartSmart\[[26](https://arxiv.org/html/2610.36544#bib.bib7)\], for example, evaluates a learned monitor with a recovery mechanism not tested here\. Its positive results and our negative timeout screens can both hold\. A meaningful comparison must fix reset semantics, residual budget, verifier, and monitoring cost\.

Preserve decision distinctions, not every token\.The information\-value identity gives a precise connection to context compression\. A representation can discard large parts of a trace without losing control value if the discarded distinctions do not reverse the preferred action\. Conversely, faithful reconstruction can preserve many irrelevant details\. This is a criterion for future evaluation, not evidence that our study has built a superior compressor\.

Make abstention explicit\.The released audit returns an action only when its specified intervals separate\. Otherwise it reports insufficient evidence\. This is useful as a calibration check before enabling an automatic restart rule; it is not a claim that continuation is optimal whenever the audit abstains\. The present five\-run cells are too sparse for this conservative task\-specific rule\. Hierarchical pooling could improve efficiency, but would introduce assumptions that require held\-out validation\.

Separate protocol repair from task recovery\.Observable termination is not a live stagnation decision\. Recycling its unused calls recovers budget in our replay\. Replacing its final error label without changing verified completion is a different result\. The context intervention illustrates why dashboards should report final task outcomes and cost alongside protocol diagnostics\. Likewise, failed\-patch diversity alone does not establish that a particular composition or synthesis operator can recover a solution\. We do not estimate error correlation or identify the causal source of the unsuccessful combinations\.

### 7\.3Limitations

All 19 tasks are selected SymPy issues\. The cohort construction deliberately mixes a pilot, a mechanical high\-ID selection, and difficulty enrichment; pooled rates do not estimate performance over SWE\-bench Verified or agent workloads generally\. Later interventions use one model and one compact harness\. Two models on seven shared tasks demonstrate heterogeneity, not broad model coverage\.

Five attempts per task leave substantial uncertainty, particularly after conditioning on live survival\. The zero\-success treatment and reuse screens do not exclude modest benefits in other populations\. The unclustered late\-tail calculation is a sensitivity result, not a task\-population confidence statement\. The retrospective decision family was chosen after the original experiments and is not a prospective validation of the certification rule\.

The analysis assumes exchangeable independent provider draws within each fixed task and runtime\. Shared routing, temporal drift, or correlated restarts can invalidate that approximation\. The binomial certificate does not transfer across arbitrary changes in the task distribution, prompt, harness, provider, or recovery action\. It also provides no protection for uncorrected adaptive monitoring\.

The binary verifier measures a finite fail\-to\-pass specification, not every aspect of issue correctness\. Base\-fail/reference\-pass gating and successful\-patch audits reduce implementation threats but cannot eliminate specification gaps\. Per\-action external evaluation supplies completion labels unavailable to a typical production controller\. Its computational overhead is not part of the model\-call budget; reported model costs are not end\-to\-end deployment costs\.

Finally, we have no matched prefix\-conditioned continuation and restart trials for a learned signal\. The empirical audit uses task identity and elapsed calls, not rich trace features\. It cannot measure the decision value of embeddings, learned monitors, or semantic summaries\. The information and opportunity identities describe what such experiments should estimate; they do not substitute for them\.

### 7\.4A focused next test

The next decisive experiment should evaluate one existing monitor under the same recovery semantics as a simple baseline, rather than add another timeout family\. First, use independent calibration data to identify tasks with both live failures and nontrivial residual\-budget recovery\. Freeze the monitor, threshold, budget, and clean\-reset or edit\-reuse rule\. On held\-out tasks, compare continuation, a calibrated fixed\-time rule, and the monitor\-triggered intervention, charging monitoring and recovery overhead to the same budget\. Report opportunity, calibration, action\-value regret, success, and cost separately\.

A second model or repository is necessary to test transfer; more draws on the existing four hard tasks cannot establish it\. Such a study would convert the current measurement contribution into evidence for a deployable control method\. It is deliberately distinct from the present offline audit and should not be described as completed\.

## 8Conclusion

Failure prediction is useful for agent control only through the value of the action it triggers\. For an independent clean restart, that value depends jointly on alarm errors and success within the residual budget\. Classical information comparison yields a decision\-oriented criterion for retaining or compressing progress observations, while exact binomial intervals expose when a per\-task action preference lacks statistical support\.

Across 194 model executions and 19 verifier\-only composites, simple restart and recovery interventions show strong runtime dependence and no prospective replication of the initial cutoff gain\. A new audit of 130 existing traces finds 103 live task\-time comparisons but no sign certificate under its simultaneous rule\. These findings motivate an evaluation standard: declare the intervention, measure recovery opportunity, account for monitoring cost, and report uncertainty at the level where decisions are made\.RestartLabprovides the accompanying trace analysis and audit implementation; broader prospective validation remains necessary\.

## References

- \[1\]D\. Blackwell\(1953\)Equivalent comparisons of experiments\.The Annals of Mathematical Statistics24\(2\),pp\. 265–272\.External Links:[Document](https://dx.doi.org/10.1214/aoms/1177729032)Cited by:[§1](https://arxiv.org/html/2610.36544#S1.p5.1),[§2\.1](https://arxiv.org/html/2610.36544#S2.SS1.p2.1),[§4](https://arxiv.org/html/2610.36544#S4.p1.1)\.
- \[2\]G\. Brown, J\. Wyatt, R\. Harris, and X\. Yao\(2005\)Diversity creation methods: a survey and categorisation\.Information Fusion6\(1\),pp\. 5–20\.External Links:[Document](https://dx.doi.org/10.1016/j.inffus.2004.04.004)Cited by:[Appendix G](https://arxiv.org/html/2610.36544#A7.p4.3),[§2\.1](https://arxiv.org/html/2610.36544#S2.SS1.p3.1)\.
- \[3\]C\. J\. Clopper and E\. S\. Pearson\(1934\)The use of confidence or fiducial limits illustrated in the case of the binomial\.Biometrika26\(4\),pp\. 404–413\.External Links:[Document](https://dx.doi.org/10.1093/biomet/26.4.404)Cited by:[§2\.1](https://arxiv.org/html/2610.36544#S2.SS1.p2.1),[§4\.3](https://arxiv.org/html/2610.36544#S4.SS3.p2.1)\.
- \[4\]T\. G\. Dietterich\(2000\)Ensemble methods in machine learning\.InMultiple Classifier Systems,pp\. 1–15\.External Links:[Document](https://dx.doi.org/10.1007/3-540-45014-9%5F1)Cited by:[Appendix G](https://arxiv.org/html/2610.36544#A7.p4.3),[§2\.1](https://arxiv.org/html/2610.36544#S2.SS1.p3.1)\.
- \[5\]Y\. Guo, Y\. Xiao, J\. M\. Zhang, M\. Harman, Y\. Lou, Y\. Liu, and Z\. Chen\(2026\)EET: experience\-driven early termination for cost\-efficient software engineering agents\.arXiv preprint arXiv:2601\.05777\.External Links:[Link](https://arxiv.org/abs/2601.05777)Cited by:[§1](https://arxiv.org/html/2610.36544#S1.p2.1),[§2\.2](https://arxiv.org/html/2610.36544#S2.SS2.p1.1)\.
- \[6\]Q\. Han, H\. Tu, Z\. Wang, H\. Dai, Y\. Zhou, N\. Lau, A\. A\. Cardenas, Y\. Xu, R\. Xu, C\. Xiong, Z\. Zheng, H\. Yao, Y\. Zhou, and C\. Xie\(2026\)VLAA\-GUI: knowing when to stop, recover, and search, a modular framework for GUI automation\.arXiv preprint arXiv:2604\.21375\.External Links:[Link](https://arxiv.org/abs/2604.21375)Cited by:[§2\.2](https://arxiv.org/html/2610.36544#S2.SS2.p3.1)\.
- \[7\]J\. Huang, X\. Chen, S\. Mishra, H\. S\. Zheng, A\. W\. Yu, X\. Song, and D\. Zhou\(2024\)Large language models cannot self\-correct reasoning yet\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2310.01798)Cited by:[§2\.1](https://arxiv.org/html/2610.36544#S2.SS1.p4.1)\.
- \[8\]C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. Narasimhan\(2024\)SWE\-bench: can language models resolve real\-world GitHub issues?\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2310.06770)Cited by:[§1](https://arxiv.org/html/2610.36544#S1.p6.1),[§2\.3](https://arxiv.org/html/2610.36544#S2.SS3.p1.1),[§5\.2](https://arxiv.org/html/2610.36544#S5.SS2.p1.1)\.
- \[9\]L\. P\. Kaelbling, M\. L\. Littman, and A\. R\. Cassandra\(1998\)Planning and acting in partially observable stochastic domains\.Artificial Intelligence101\(1\-2\),pp\. 99–134\.External Links:[Document](https://dx.doi.org/10.1016/S0004-3702%2898%2900023-X)Cited by:[§3\.1](https://arxiv.org/html/2610.36544#S3.SS1.p1.1)\.
- \[10\]J\. D\. Kalbfleisch and R\. L\. Prentice\(2002\)The statistical analysis of failure time data\.Second edition,Wiley\.External Links:[Document](https://dx.doi.org/10.1002/9781118032985)Cited by:[§5\.9](https://arxiv.org/html/2610.36544#S5.SS9.p3.1)\.
- \[11\]S\. Kapoor, B\. Stroebl, Z\. S\. Siegel, N\. Nadgir, and A\. Narayanan\(2024\)AI agents that matter\.arXiv preprint arXiv:2407\.01502\.External Links:[Link](https://arxiv.org/abs/2407.01502)Cited by:[§2\.3](https://arxiv.org/html/2610.36544#S2.SS3.p2.1)\.
- \[12\]H\. Kautz, E\. Horvitz, Y\. Ruan, C\. Gomes, and B\. Selman\(2002\)Dynamic restart policies\.InProceedings of the AAAI Conference on Artificial Intelligence,pp\. 674–681\.External Links:[Link](https://www.cs.cornell.edu/gomes/pdf/2002_kautz_aaai_iaai_dynamic.pdf)Cited by:[§1](https://arxiv.org/html/2610.36544#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.36544#S2.SS1.p1.1)\.
- \[13\]R\. Laroche, P\. Trichelair, and R\. Tachet des Combes\(2019\)Safe policy improvement with baseline bootstrapping\.InProceedings of the International Conference on Machine Learning,pp\. 3652–3661\.External Links:[Link](https://proceedings.mlr.press/v97/laroche19a.html)Cited by:[§2\.1](https://arxiv.org/html/2610.36544#S2.SS1.p2.1),[§4\.3](https://arxiv.org/html/2610.36544#S4.SS3.p4.1)\.
- \[14\]X\. Li, R\. Ming, M\. Chu, S\. Shao, R\. Jin, and C\. Xiong\(2026\)ACM: agentic context management for long horizon tasks\.arXiv preprint arXiv:2607\.23809\.External Links:[Link](https://arxiv.org/abs/2607.23809)Cited by:[§2\.2](https://arxiv.org/html/2610.36544#S2.SS2.p4.1)\.
- \[15\]S\. Liu, J\. Yang, B\. Jiang, Y\. Li, J\. Guo, X\. Liu, and B\. Dai\(2025\)Context as a tool: context management for long\-horizon SWE\-agents\.arXiv preprint arXiv:2512\.22087\.External Links:[Link](https://arxiv.org/abs/2512.22087)Cited by:[§2\.2](https://arxiv.org/html/2610.36544#S2.SS2.p4.1)\.
- \[16\]M\. Luby, A\. Sinclair, and D\. Zuckerman\(1993\)Optimal speedup of Las Vegas algorithms\.Information Processing Letters47\(4\),pp\. 173–180\.External Links:[Document](https://dx.doi.org/10.1016/0020-0190%2893%2990029-9)Cited by:[Appendix C](https://arxiv.org/html/2610.36544#A3.p4.1),[§2\.1](https://arxiv.org/html/2610.36544#S2.SS1.p1.1),[§3\.2](https://arxiv.org/html/2610.36544#S3.SS2.p5.1)\.
- \[17\]C\. Ma, J\. Zhang, Z\. Zhu, C\. Yang, Y\. Yang, Y\. Jin, Z\. Lan, L\. Kong, and J\. He\(2024\)AgentBoard: an analytical evaluation board of multi\-turn LLM agents\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://arxiv.org/abs/2401.13178)Cited by:[§2\.2](https://arxiv.org/html/2610.36544#S2.SS2.p2.1)\.
- \[18\]A\. Madaan, N\. Tandon, P\. Gupta, S\. Hallinan, L\. Gao, S\. Wiegreffe, U\. Alon, N\. Dziri, S\. Prabhumoye, Y\. Yang, S\. Welleck, B\. P\. Majumder, S\. Gupta, A\. Yazdanbakhsh, and P\. Clark\(2023\)Self\-refine: iterative refinement with self\-feedback\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://arxiv.org/abs/2303.17651)Cited by:[§2\.1](https://arxiv.org/html/2610.36544#S2.SS1.p4.1)\.
- \[19\]H\. Park and B\. Choi\(2026\)When do agent loops mistake stagnation for progress? self\-evaluation bias and externally grounded verification in long\-running autonomous LLM agent loops\.arXiv preprint arXiv:2607\.25152\.External Links:[Link](https://arxiv.org/abs/2607.25152)Cited by:[§2\.2](https://arxiv.org/html/2610.36544#S2.SS2.p2.1)\.
- \[20\]N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. Yao\(2023\)Reflexion: language agents with verbal reinforcement learning\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://arxiv.org/abs/2303.11366)Cited by:[§2\.1](https://arxiv.org/html/2610.36544#S2.SS1.p4.1)\.
- \[21\]S\. Shrivastava\(2026\)Semantic early\-stopping for iterative LLM agent loops\.arXiv preprint arXiv:2606\.27009\.External Links:[Link](https://arxiv.org/abs/2606.27009)Cited by:[§2\.2](https://arxiv.org/html/2610.36544#S2.SS2.p3.1)\.
- \[22\]M\. Streeter, D\. Golovin, and S\. F\. Smith\(2007\)Restart schedules for ensembles of problem instances\.InProceedings of the AAAI Conference on Artificial Intelligence,pp\. 1204–1210\.External Links:[Link](https://cdn.aaai.org/AAAI/2007/AAAI07-191.pdf)Cited by:[§1](https://arxiv.org/html/2610.36544#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.36544#S2.SS1.p1.1)\.
- \[23\]M\. Streeter\(2019\)Bayes optimal early stopping policies for black\-box optimization\.arXiv preprint arXiv:1902\.08285\.External Links:[Link](https://arxiv.org/abs/1902.08285)Cited by:[§1](https://arxiv.org/html/2610.36544#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.36544#S2.SS1.p1.1)\.
- \[24\]SWE\-agent contributors\(2026\)Mini\-swe\-agent: the 100\-line AI agent that solves GitHub issues\.Note:GitHub repository, version 2\.4\.6External Links:[Link](https://github.com/SWE-agent/mini-swe-agent)Cited by:[§5\.2](https://arxiv.org/html/2610.36544#S5.SS2.p2.1)\.
- \[25\]N\. Tacheny\(2026\)Geometric dynamics of agentic loops in large language models\.arXiv preprint arXiv:2512\.10350\.External Links:[Link](https://arxiv.org/abs/2512.10350)Cited by:[§2\.2](https://arxiv.org/html/2610.36544#S2.SS2.p3.1)\.
- \[26\]C\. Wang, Y\. Lyu, J\. He, Z\. Yang, C\. Zhong, Y\. Harel, and D\. Lo\(2026\)Fail\-fast, restart\-smart: early failure prediction and restart for SWE agentic tasks\.arXiv preprint arXiv:2608\.03222\.External Links:[Link](https://arxiv.org/abs/2608.03222)Cited by:[§1](https://arxiv.org/html/2610.36544#S1.p2.1),[§2\.2](https://arxiv.org/html/2610.36544#S2.SS2.p1.1),[§7\.2](https://arxiv.org/html/2610.36544#S7.SS2.p2.1)\.
- \[27\]J\. Wang, J\. Wang, B\. Athiwaratkun, C\. Zhang, and J\. Zou\(2024\)Mixture\-of\-agents enhances large language model capabilities\.arXiv preprint arXiv:2406\.04692\.External Links:[Link](https://arxiv.org/abs/2406.04692)Cited by:[§2\.1](https://arxiv.org/html/2610.36544#S2.SS1.p3.1)\.
- \[28\]X\. Wang, J\. Wei, D\. Schuurmans, Q\. V\. Le, E\. H\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou\(2023\)Self\-consistency improves chain of thought reasoning in language models\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2203.11171)Cited by:[§2\.1](https://arxiv.org/html/2610.36544#S2.SS1.p3.1)\.
- \[29\]Y\. Wang, M\. Pradel, and Z\. Liu\(2025\)Are “solved issues” in SWE\-bench really solved correctly? an empirical study\.arXiv preprint arXiv:2503\.15223\.External Links:[Link](https://arxiv.org/abs/2503.15223)Cited by:[§2\.3](https://arxiv.org/html/2610.36544#S2.SS3.p2.1)\.
- \[30\]Y\. Wu, Z\. Sun, S\. Li, S\. Welleck, and Y\. Yang\(2024\)Inference scaling laws: an empirical analysis of compute\-optimal inference for problem\-solving with language models\.arXiv preprint arXiv:2408\.00724\.External Links:[Link](https://arxiv.org/abs/2408.00724)Cited by:[§2\.1](https://arxiv.org/html/2610.36544#S2.SS1.p3.1)\.
- \[31\]J\. Yang, C\. E\. Jimenez, A\. Wettig, K\. Lieret, S\. Yao, K\. Narasimhan, and O\. Press\(2024\)SWE\-agent: agent\-computer interfaces enable automated software engineering\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://arxiv.org/abs/2405.15793)Cited by:[§1](https://arxiv.org/html/2610.36544#S1.p6.1),[§2\.3](https://arxiv.org/html/2610.36544#S2.SS3.p1.1)\.
- \[32\]S\. Yao, D\. Yu, J\. Zhao, I\. Shafran, T\. L\. Griffiths, Y\. Cao, and K\. Narasimhan\(2023\)Tree of thoughts: deliberate problem solving with large language models\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://arxiv.org/abs/2305.10601)Cited by:[§2\.1](https://arxiv.org/html/2610.36544#S2.SS1.p3.1)\.
- \[33\]L\. Yi, R\. Lei, L\. Yao, Y\. Xie, Y\. Li, W\. Zhang, Z\. Wei, Y\. Li, and J\. Nie\(2026\)Learning agent\-compatible context management for long\-horizon tasks\.arXiv preprint arXiv:2605\.30785\.External Links:[Link](https://arxiv.org/abs/2605.30785)Cited by:[§2\.2](https://arxiv.org/html/2610.36544#S2.SS2.p4.1)\.

## Appendix ANotation

Table 11:Principal notation\.
## Appendix BTask Cohorts and Frozen Decisions

Table 12:Task IDs by cohort\. IDs omit the commonsympy\_\_sympy\-prefix\.The task sets overlap across stages by design\. Counts of 19 unique tasks and 194 model attempts deduplicate task identity but not fresh executions\. The total consists of 70 initial attempts, 60 prospective attempts, 20 horizon\-extension attempts, 40 context A/B attempts, and four semantic\-resolver attempts\. The patch\-algebra stage adds 19 external verifier calls on composed candidates but no model attempt\.

## Appendix CPolicy Definitions

Single\.Run one attempt up to budgetBB, stopping at first external pass or explicit terminal failure\.

Fixedcc\.Run independent attempts with per\-episode capccuntil the total budget is spent\. An explicit terminal failure releases unused episode budget\.

Geometric\.Allocate episode caps in a frozen geometric sequence, truncated by the remaining total budget\.

Luby\.Allocate episode caps according to a scaled prefix of the universal sequence in[Luby et al\. \[16\]](https://arxiv.org/html/2610.36544#bib.bib9), truncated by the remaining total budget\.

Horizon insurance\.At budget 60, allocate two 15\-call probes followed by one reserved 30\-call episode\. Compare with two 30\-call episodes\.

The fixed\-policy plug\-in estimate treats the five empirical attempts for each task as an IID distribution\. For a sequence of episode caps\(c1,…,ck\)\(c\_\{1\},\\ldots,c\_\{k\}\)and empirical CDFF^j\\widehat\{F\}\_\{j\}, the task success probability under independent resampling is

1−∏i=1k\(1−F^j​\(ci\)\)\.1\-\\prod\_\{i=1\}^\{k\}\\left\(1\-\\widehat\{F\}\_\{j\}\(c\_\{i\}\)\\right\)\.\(23\)Monte Carlo replay is used in the implementation when terminal failures create episode\-dependent residual budgets\.

## Appendix DAdditional Operational Results

Table 13:Collection cost and accumulated wall time\. Wall time sums parallelizable attempts and is not elapsed study duration\.Costs are provider\-metered values recorded in the traces\. They are not estimates of deployment expense\. The large cost and latency change in context rejuvenation arises because treatment consumes all 60 calls after replacing an early terminal format failure with a fresh episode\.

## Appendix EArtifact Manifest

The accompanying analysis package contains the following sanitized artifacts\. They exclude prompts, trajectories, local paths, hidden verifier output, reference patches, and credentials\.

Table 14:Accompanying analysis artifacts and their role in the study\.The package also contains stage protocols, the frozen composition plan, agent adapters, replay policies, analysis code, command\-line tools, and tests\. The sanitized artifacts reproduce the reported trace analyses and figures offline; new agent collection requires a model credential\. Derived audit cells reuse earlier attempts and do not increase the count of model executions\.

## Appendix FThreats to Interpretation

Task selection\.The first seven tasks were an operational pilot, the prospective high\-ID cohort used a mechanical order, and the stress cohort deliberately enriched difficulty\. Pooled rates are descriptive for these tasks, not prevalence estimates for SWE\-bench Verified\.

Verifier granularity\.Verification after every environment action yields precise first\-pass timing, but the fail\-to\-pass suite is still a finite specification\. We report verifier success rather than semantic issue resolution and retain the standard benchmark terminology only when naming the dataset\.

Adaptive sequence\.Later stages were chosen after earlier results\. Each stage has its own frozen protocol, but cross\-stage narratives can still invite hindsight\. The ladder table therefore states the question and comparison at each stage, and null follow\-ups are retained rather than omitted\.

Multiplicity\.The study evaluates several policies and mechanisms\. The stage\-level bootstrap intervals are not adjusted for simultaneous coverage\. Only the decision\-audit intervals use the explicit family allocation in[section4\.3](https://arxiv.org/html/2610.36544#S4.SS3)\. We avoid claiming significance for exploratory maxima, identify in\-sample oracles as nondeployable, and give priority to prospective contrasts\.

Model identity\.Hosted behavior can change under a stable model identifier\. Requested model ID and collection date therefore define, but do not fully identify, the runtime used for calibration\.

## Appendix GFailure\-Conditioned Branches: Supporting Examples

Branching is often proposed when continuation is uncertain\. Its benefit depends on the epistemic relation among branches, not their textual distance\. Conditioning all inputs to an ensemble on terminal failure can also reverse component reliability\.

###### Lemma G\.1\(Failure\-conditioned component inversion\)

LetY∼Bernoulli⁡\(q\)Y\\sim\\mathrm\{Bernoulli\}\(q\)indicate that one component of a patch is correct, withq\>1/2q\>1/2\. LetZ∼Bernoulli⁡\(r\)Z\\sim\\mathrm\{Bernoulli\}\(r\)indicate that the remaining requirements are correct, independent ofYY, and let full success beS=Y​ZS=YZ\. Then

ℙ⁡\(Y=1∣S=0\)=q⁡\(1−r\)1−q​r<12\\mathbb\{P\}\(Y=1\\mid S=0\)=\\frac\{q\(1\-r\)\}\{1\-qr\}<\\frac\{1\}\{2\}\(24\)wheneverr\>\(2​q−1\)/qr\>\(2q\-1\)/q\.

Proof\.Bayes’ rule gives the displayed conditional probability\. Comparing it with1/21/2yields2​q​\(1−r\)<1−q​r2q\(1\-r\)<1\-qr, which is equivalent tor\>\(2​q−1\)/qr\>\(2q\-1\)/q\.□\\square

Thus a component that is reliable in the unconditional branch population can become minority\-reliable after selecting only failed full patches\. The lemma is an existence result, not an estimate of our patch population\.

Correlation supplies a second limit\. LetY1,…,YmY\_\{1\},\\ldots,Y\_\{m\}be exchangeable correctness indicators with meanqqand common pairwise correlationρ\\rho\. Then

Var⁡\(1m​∑i=1mYi\)=q⁡\(1−q\)m​\[1\+\(m−1\)​ρ\]\.\\operatorname\{Var\}\\\!\\left\(\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}Y\_\{i\}\\right\)=\\frac\{q\(1\-q\)\}\{m\}\\left\[1\+\(m\-1\)\\rho\\right\]\.\(25\)The variance is that of only

meff=m1\+\(m−1\)​ρm\_\{\\mathrm\{eff\}\}=\\frac\{m\}\{1\+\(m\-1\)\\rho\}\(26\)independent views\. Ifρ=1\\rho=1, increasing the number of branches does not reduce uncertainty at all\. This standard ensemble effect\[[4](https://arxiv.org/html/2610.36544#bib.bib12),[2](https://arxiv.org/html/2610.36544#bib.bib13)\]motivates our distinction between*behavioral diversity*, such as different files or diffs, and*epistemic independence*, meaning error diversity sufficient for aggregation\. Branches from one model and harness can satisfy the former while failing the latter\.