Not All Errors Are Equal: Consequence-Aware Reasoning Compute Allocation

# Not All Errors Are Equal: Consequence-Aware Reasoning Compute Allocation
Source: [https://arxiv.org/html/2606.04402](https://arxiv.org/html/2606.04402)
Jingbo Wen1Liang He2,∗Ziqi He1 1The University of Sydney 2Shanghai Institute of Optics and Fine Mechanics hel@siom\.ac\.cn ∗Corresponding authors

###### Abstract

Modern reasoning models can allocate different amounts of test\-time computation, such as thinking tokens, model calls, or compute budget, to different tasks\. Existing methods generally drive this allocation by predicted difficulty and spend more compute where it is expected to raise accuracy\. This implicitly assumes that all failures cost the same, since an accuracy objective weights every task equally\. However, such an assumption does not hold in deployment: A typo in a log message and a migration that corrupts a production database both count as one benchmark failure, but their real\-world costs are fundamentally different\. To fill this gap, we propose consequence\-aware test\-time compute allocation\. Instead of routing compute only by predicted difficulty, we use a lightweight predictor to estimate from the issue text how costly a task would be if solved incorrectly\. The scheduler then routes higher\-consequence tasks to larger compute tiers or higher thinking budgets under the same total budget\. We conduct main experiments on SWE\-bench Lite and evaluate cross\-dataset behavior on Multi\-SWE\-bench mini, covering 700 software\-engineering tasks in total\. Our results reveal that consequence and difficulty are approximately orthogonal under various annotations, and that current thinking models do not allocate compute sufficiently according to consequence\. Moreover, our issue\-only predictor never misclassifies a high\-consequence task as low\-consequence across the 300 SWE\-bench tasks\. Under matched compute budgets, our consequence\-aware scheduler reduces cost\-weighted loss by 22%–33% relative to difficulty\-aware routing; in particular, the priority\-aware variant—which routes by per\-task cost scaled by the marginal\-utility signal—crosses 30%, and its deployable predictor\-driven version retains over 90% of the oracle gain\. Notably, difficulty\-aware routing performs worse than random, as the hardest tasks are unsolvable at any tier\.

## 1Introduction

Modern reasoning models, such as OpenAI’s o\-series, DeepSeek\-R1, Anthropic’s Claude with extended thinking, and Qwen3\-Thinking, can decide how much test\-time computation to spend on each input\(Liet al\.,[2025](https://arxiv.org/html/2606.04402#bib.bib11)\)\. In practice, they vary how long they think from task to task, often producing longer reasoning traces on inputs that appear more difficult\(Alomraniet al\.,[2025](https://arxiv.org/html/2606.04402#bib.bib12); Zhu and Li,[2025](https://arxiv.org/html/2606.04402#bib.bib13)\)\. The same principle is pursued explicitly in adaptive test\-time compute, where learned routers\(Damaniet al\.,[2025](https://arxiv.org/html/2606.04402#bib.bib2)\), compute\-difficulty scaling laws\(Snellet al\.,[2024](https://arxiv.org/html/2606.04402#bib.bib1)\), and confidence\-driven early stopping\(Manviet al\.,[2025](https://arxiv.org/html/2606.04402#bib.bib3)\)differ in mechanism but follow a common rule, namely that harder tasks deserve more compute\.

This principle is natural for benchmarks that optimize average accuracy under a compute budget\. In such settings, each task contributes one unit to the final score, and every failure is assumed to be interchangeable\(Alomraniet al\.,[2025](https://arxiv.org/html/2606.04402#bib.bib12)\)\. However, this is not how deployment works, and such an assumption generally does not hold in deployment\. Consider two failures a coding assistant might produce on the same day: a typo in a log message and a database migration that silently corrupts a production table under a race condition\. A benchmark records both as one failed task, but in deployment their costs are nowhere near equal: preventing the migration error is worth far more compute, even when the two are equally hard to fix\.

This deployment mismatch exposes a missing term in current LLM adaptive\-compute objectives\. The closest prior framing is rational metareasoning\(Russell and Wefald,[1991](https://arxiv.org/html/2606.04402#bib.bib4); Hayet al\.,[2012](https://arxiv.org/html/2606.04402#bib.bib5); Sabbataet al\.,[2024](https://arxiv.org/html/2606.04402#bib.bib6)\), which treats computation as an action whose value depends on its expected utility\. From this perspective, additional reasoning is useful only when the expected reduction in downstream loss exceeds the cost of computation\(Alomraniet al\.,[2025](https://arxiv.org/html/2606.04402#bib.bib12); Sayinet al\.,[2025](https://arxiv.org/html/2606.04402#bib.bib16)\)\. Modern LLM implementations of adaptive compute, however, typically collapse this utility term into a uniform accuracy objective\(Liet al\.,[2025](https://arxiv.org/html/2606.04402#bib.bib11)\)\. We therefore move beyond difficulty\-aware allocation and cast consequence\-aware allocation as a cost\-weighted problem: for each task, a scheduler should choose a compute tier that trades off the price of additional computation against the expected consequence\-weighted error\(Qu,[2026](https://arxiv.org/html/2606.04402#bib.bib15)\)\. When all tasks have equal error cost, this objective reduces to standard accuracy maximization under a compute budget\(Alomraniet al\.,[2025](https://arxiv.org/html/2606.04402#bib.bib12)\)\. When error costs vary, however, difficulty alone is no longer a sufficient routing signal\.

![Refer to caption](https://arxiv.org/html/2606.04402v1/x1.png)Figure 1:Three contemporary thinking models do not sufficiently allocate compute by consequence\.Each panel summarizes one model’s relationship between actual thinking length and our consequence label on SWE\-bench Lite tasks\. Qwen3\-8B’s thinking length is uncorrelated with consequence \(ρ=0\.002\\rho=0\.002\)\. Qwen3\-VL\-8B\-Thinking pins99\.3%99\.3\\%of tasks at its 8192\-token cap\. Claude Sonnet 4\.5 with extended thinking shows a statistically significant but weak consequence sensitivity \(ρ=0\.20\\rho=0\.20,p=0\.008p=0\.008,n=171n=171\)\. Across three different allocation mechanisms, none exhibits sufficient consequence sensitivity to explain the gains of consequence\-aware routing observed later in the paper\.One might hope that sufficiently strong reasoning models already learn this distinction implicitly\. If high\-consequence tasks naturally induce longer reasoning traces, then an explicit cost signal would be unnecessary\. We test this possibility directly in Figure[1](https://arxiv.org/html/2606.04402#S1.F1)by auditing three contemporary thinking models on SWE\-bench Lite\. The results show three different failure modes\. Qwen3\-8B varies its thinking length, but that variation is essentially uncorrelated with consequence \(ρ=0\.002\\rho=0\.002\)\. Qwen3\-VL\-8B\-Thinking reaches its 8192\-token cap on99\.3%99\.3\\%of tasks, leaving almost no room for task\-specific allocation\(Panet al\.,[2025](https://arxiv.org/html/2606.04402#bib.bib17)\)\. Claude Sonnet 4\.5 with extended thinking shows a statistically significant relationship between thinking length and consequence \(ρ=0\.20\\rho=0\.20,p=0\.008p=0\.008,n=171n=171\), but the effect is small: high\-consequence tasks receive only20\.5%20\.5\\%more thinking than low\-consequence tasks\. Thus, existing thinking models exhibit either no consequence signal, saturation, or only weak consequence sensitivity\(Liuet al\.,[2025](https://arxiv.org/html/2606.04402#bib.bib18)\)\.

Motivated by this gap, we propose consequence\-aware test\-time compute allocation\. The method separates two decisions that are often conflated\(Shirkavandet al\.,[2026](https://arxiv.org/html/2606.04402#bib.bib19)\)\. First, a lightweight predictor estimates the deployment\-time cost of an error directly from the issue text, before the task is solved\. Second, a scheduler uses this predicted consequence to route higher\-consequence tasks to larger compute tiers, or equivalently to larger thinking budgets, under the same total compute budget as a difficulty\-based baseline\. The reasoning model itself is not retrained; the intervention operates entirely at the scheduling layer\.

The remainder of the paper builds the empirical case in four steps\.

- \(1\)Consequence is not difficulty, nor a disguised confidence signal\.We conduct our main analysis on300300SWE\-bench Lite tasks and evaluate cross\-dataset behavior on400400Multi\-SWE\-bench mini tasks\. Across rule\-based, LLM\-with\-patch, issue\-only, and human\-majority labels, consequence remains nearly orthogonal to difficulty\. Moreover, after controlling for both difficulty and cross\-model pass\-rate variance, consequence still explains an additional26\.426\.4percentage points of cost\-weighted marginal value \(\+55\.7%\+55\.7\\%relative; §[3](https://arxiv.org/html/2606.04402#S3), §[8](https://arxiv.org/html/2606.04402#S8)\)\.
- \(2\)Existing thinking models are not sufficiently consequence\-aware\.The audit in Figure[1](https://arxiv.org/html/2606.04402#S1.F1)reveals three distinct allocation patterns: no consequence signal in Qwen3\-8B, near\-universal saturation in Qwen3\-VL\-8B\-Thinking, and weak but insufficient consequence sensitivity in Claude Sonnet 4\.5 \(§[4](https://arxiv.org/html/2606.04402#S4)\)\.
- \(3\)Consequence can be predicted before solving\.A Qwen3\-8B issue\-only predictor, which sees only the issue text and file path, agrees with the Qwen3\-8B with\-patch reference at Cohen’sκ=0\.572\\kappa=0\.572\. A cross\-model Claude issue\-only predictor reachesκ=0\.379\\kappa=0\.379\. Crucially, on the300300SWE\-bench Lite tasks, neither predictor misclassifies a high\-consequence task as low\-consequence, avoiding the most dangerous under\-allocation error \(§[5](https://arxiv.org/html/2606.04402#S5)\)\.
- \(4\)Consequence\-aware routing improves cost\-weighted allocation, and priority\-aware routing improves it further\.Under matched compute on a1616\-model SWE\-bench compute\-tier benchmark, consequence\-only routing substantially reduces cost\-weighted loss relative to difficulty\-aware routing\. A priority\-aware variant that combines predicted consequence with estimated marginal gain reduces cost\-weighted loss by over30%30\\%, while retaining over90%90\\%of the oracle gain\. The same analysis also explains why difficulty\-aware routing can underperform random: the hardest tasks often have the lowest marginal gain from extra compute, because they remain unsolved at every available tier \(§[7](https://arxiv.org/html/2606.04402#S7), §[8](https://arxiv.org/html/2606.04402#S8)\)\.

## 2Related Work

Adaptive test\-time compute\.A growing body of work studies how much computation a language model should spend on each input\.Snellet al\.\([2024](https://arxiv.org/html/2606.04402#bib.bib1)\)analyze how the effectiveness of different test\-time scaling strategies, and derive difficulty\-dependent compute\-optimal policies for using a fixed inference budget\.Damaniet al\.\([2025](https://arxiv.org/html/2606.04402#bib.bib2)\)train a reward predictor to allocate compute according to expected accuracy gain\.Manviet al\.\([2025](https://arxiv.org/html/2606.04402#bib.bib3)\)use token\-level confidence to stop generation early when the model is already confident\. Although these methods differ in implementation, they share the same underlying objective: spend compute where it is expected to improve average accuracy\. Our work asks a complementary question\. When two tasks have similar expected accuracy gains but very different costs of failure, an accuracy\-driven router treats them as equivalent, whereas a deployment\-oriented scheduler should not\. We therefore keep the reasoning model unchanged and change the routing signal from predicted difficulty or confidence to predicted consequence\.

Rational metareasoning\.The decision\-theoretic view of computation as a costly action dates back to rational metareasoning\(Russell and Wefald,[1991](https://arxiv.org/html/2606.04402#bib.bib4); Hayet al\.,[2012](https://arxiv.org/html/2606.04402#bib.bib5)\)\. In this framework, additional computation is useful only when its expected utility exceeds its computational cost\. This perspective is closely aligned with our formulation: Whether a task is worth more compute depends not only on how much that compute would improve the answer, but on how costly an error would be\. Recent work has begun to apply rational metareasoning to large language models\(Sabbataet al\.,[2024](https://arxiv.org/html/2606.04402#bib.bib6)\)\. However, in most LLM adaptive\-compute settings, utility is operationalized as accuracy, reward, or confidence rather than as a task\-dependent cost of error\.

Cost\-sensitive learning and selective prediction\.Cost\-sensitive learning studies classification under asymmetric error costs\(Elkan,[2001](https://arxiv.org/html/2606.04402#bib.bib7)\), while selective prediction allows a model to abstain when it is uncertain\(Geifman and El\-Yaniv,[2017](https://arxiv.org/html/2606.04402#bib.bib8)\)\. Both lines of work recognize that errors are not interchangeable\. The difference lies in where the cost signal is used\. Cost\-sensitive learning typically changes the training objective or decision rule, and selective prediction decides whether to answer at all\. In contrast, we use the cost signal at compute time: the model still answers every task, but the scheduler decides how much computation the task should receive before answering\. This makes the approach compatible with existing reasoning models without retraining\.

Software\-engineering benchmarks\.We use SWE\-bench Lite\(Jimenezet al\.,[2024](https://arxiv.org/html/2606.04402#bib.bib9)\)as our primary testbed and a 400\-task subset of Multi\-SWE\-bench\(Zanet al\.,[2026](https://arxiv.org/html/2606.04402#bib.bib10)\)for cross\-dataset validation\. Software\-engineering tasks are a natural setting for consequence\-aware allocation because they contain both large variation in task difficulty and large variation in deployment risk\. A documentation typo, a parser edge case, a permission bug, and a database migration error may all appear as single pass/fail tasks in a benchmark, yet their real\-world consequences differ substantially\. The availability of issue reports, file paths, and gold patches also allows us to compare deployment\-time issue\-only prediction with with\-patch consequence labels\.

## 3Definition and Measurement of Consequence

We first define consequence as a measurable property of a task, describe three independent labeling pipelines, and show that consequence is statistically distinct from both difficulty and model\-disagreement\-based confidence\.

Definition\.We define the consequence of a task as an ordinal label in\{0,1,2\}\\\{0,1,2\\\}reflecting the realized deployment cost if the model produces an incorrect fix:

- •0 \(low\)\.Cosmetic, formatting, documentation, or log\-message errors\. These are noticeable to the user but do not corrupt downstream systems\.
- •1 \(medium\)\.Functional but contained bugs, such as an API returning the wrong value, a feature crashing on an edge case, or temporary unavailability\. Errors are observable and locally recoverable\.
- •2 \(high\)\.Silent data corruption, security or permission bypass, irreversible operations \(delete/overwrite/migration\), or incorrect results consumed downstream without visibility\. The error is neither observable nor recoverable in time, which makes it the most costly to get wrong\.

Labeling pipelines\.Each task in SWE\-bench Lite \(N=300N\{=\}300\) and Multi\-SWE\-bench mini \(N=400N\{=\}400\) is labeled via three independent pipelines: \(i\) A rule\-based labeler using lexical patterns in file paths and gold patches \(e\.g\.,delete,auth,migration\)\. \(ii\) An LLM\-with\-patch judge \(Qwen3\-8B\) that observes issue text, gold patch, and modified files\. This serves as theprimaryconsequence label in our experiments because it can see the realized fix\. \(iii\) An LLM\-issue\-only predictor \(§[5](https://arxiv.org/html/2606.04402#S5)\) that reads only the issue text and file path, matching the information available to a deployment\-time scheduler\. It tests whether consequence can be predicted before solving the task\.

Consequence is orthogonal to difficulty\.Difficulty is measured as1−passrate¯1\-\\overline\{\\mathrm\{passrate\}\}across 16 public SWE\-bench solvers, wherepassrate¯\\overline\{\\mathrm\{passrate\}\}is each task’s pass rate averaged across 16 public SWE\-bench solvers, and we report the Spearman rank correlationρ\\rhobetween difficulty and consequence\. Under both rule\-based and LLM\-with\-patch labeling, rank correlations with difficulty are near zero:ρrule=\+0\.047\\rho\_\{\\mathrm\{rule\}\}\{=\}\{\+\}0\.047\(p\>0\.4p\>0\.4\),ρLLM​\-​patch=−0\.096\\rho\_\{\\mathrm\{LLM\\text\{\-\}patch\}\}\{=\}\{\-\}0\.096\(p\>0\.09p\>0\.09\)\. The same near\-zero pattern holds in a 150\-task human annotation study with three independent labelers \(ρhuman=−0\.066\\rho\_\{\\mathrm\{human\}\}\{=\}\{\-\}0\.066,p\>0\.5p\>0\.5; Appendix[A](https://arxiv.org/html/2606.04402#A1)\) and for the issue\-only predictor \(ρissue=−0\.109\\rho\_\{\\mathrm\{issue\}\}\{=\}\{\-\}0\.109,p=0\.06p=0\.06\)\. Across all four labelings, namely rule\-based, LLM\-with\-patch, issue\-only, and human\-majority, the correlation stays within±0\.11\\pm 0\.11and never reaches significance \(Appendix[B](https://arxiv.org/html/2606.04402#A2)\), and the pattern holds across SWE\-bench sub\-domains \(Appendix[E](https://arxiv.org/html/2606.04402#A5)\)\. High\-consequence tasks appear at every difficulty level: a trivial typo can be high\-consequence \(e\.g\., security\-critical log redaction\), while an extremely hard task can be low\-consequence \(e\.g\., an obscure parser edge case\)\. Difficulty therefore cannot predict consequence on its own, and a difficulty\-only scheduler is blind to the deployment risk that consequence captures\.

Consequence is not model confidence\.A natural follow\-up question is whether consequence simply reflects cross\-model disagreement, which some adaptive\-compute schedulers use as a proxy for task difficulty\. We address this in §[8](https://arxiv.org/html/2606.04402#S8), using it to explain why difficulty\-aware allocation systematically underperforms in cost\-weighted scenarios\.

## 4Model Audit: Do Existing Thinkers Allocate by Consequence?

A key question is whether strong reasoning models already allocate compute by consequence on their own\. We therefore audit three contemporary reasoning models, each with a distinct compute\-allocation mechanism as shown in Figure[1](https://arxiv.org/html/2606.04402#S1.F1), to measure whether their actual thinking length scales with the LLM\-with\-patch consequence label\.

Qwen3\-8B111[https://huggingface\.co/Qwen/Qwen3\-8B](https://huggingface.co/Qwen/Qwen3-8B)\(hybrid thinking\)\.Qwen3\-8B alternates between “thinking” and “direct” modes via a control token\. On 50 SWE\-bench Lite tasks withmax\_new\_tokens=4096=4096, the thinking\-character count varies across tasks, but the rank correlation with consequence is negligible withρ​\(thinking,cons\)=\+0\.002\\rho\(\\mathrm\{thinking\},\\mathrm\{cons\}\)=\+0\.002\. Hence, the allocation carries essentially no consequence signal\.

Qwen3\-VL\-8B\-Thinking222[https://huggingface\.co/Qwen/Qwen3\-VL\-8B\-Thinking](https://huggingface.co/Qwen/Qwen3-VL-8B-Thinking)\(dedicated thinking\)\.This variant produces a separate reasoning stream under a configurable token budget\. On the same 50 tasks withmax\_new\_tokens=8192=8192,99\.3%99\.3\\%of tasks reach the token cap\. Because almost all tasks are saturated, there is no headroom for consequence\-driven differentiation\.

Claude Sonnet 4\.5333[https://www\.anthropic\.com/claude/sonnet](https://www.anthropic.com/claude/sonnet)with extended thinking\.We evaluate Claude with a 16,000\-token extended\-thinking budget on 171 stratified SWE\-bench Lite tasks\. Here thinking length does correlate with consequence at a statistically significant level \(ρ=\+0\.203\\rho=\+0\.203,p=0\.008p=0\.008\), but the effect is far too small to match the stakes\. High\-consequence \(class 2\) tasks receive only 20\.5% more thinking than low\-consequence \(class 0\) tasks \(6135 vs 5090 characters on average\)\. The median high\-consequence task has≈5×\\approx 5\\timesthe marginal value of extra compute \(§[7](https://arxiv.org/html/2606.04402#S7)\)\. Claude thus shows a weak emergent tendency toward consequence\-aware allocation that is quantitatively inadequate for deployment\.

Across all three models, we observe distinct failure modes—no signal, saturation, or weak\-but\-insufficient consequence sensitivity\. None allocates compute with sufficient sensitivity to realize the gains of consequence\-aware allocation \(Table[1](https://arxiv.org/html/2606.04402#S4.T1)\)\. This motivates the following question: how should a deployer wrap a reasoning model so that it allocates compute effectively by consequence?

Table 1:Model audit on SWE\-bench Lite\.We report sample sizenn, configured token budgetmax​\_​new\\mathrm\{max\\\_new\}, Spearman rank correlation between thinking length and consequence,pp\-value, average difference in thinking length between high\- and low\-consequence tasks, and the observed failure mode relative to the cost\-weighted allocation objective\.
## 5Consequence Prediction at Deployment Time

A consequence\-aware scheduler must operate before the model produces a fix\. At deployment time, it cannot inspect the gold patch or the final code diff; it can only use the information available in the issue itself\. We therefore ask whether consequence can be predicted from deployment\-time information, i\.e\., the issue text and file path alone, and whether such predictions are reliable to guide compute allocation\.

Setup\.We remove the gold patch and modified\-file content from the LLM\-with\-patch judge of §[3](https://arxiv.org/html/2606.04402#S3)\. The input to the predictor contains only the GitHub issue text and the affected file path, matching the information available to a deployment\-time scheduler\. Our primary predictor is Qwen3\-8B in issue\-only mode, evaluated against the Qwen3\-8B with\-patch reference label\. As a cross\-model robustness check, we also evaluate Claude Sonnet 4\.5 in issue\-only mode against the same Qwen3\-8B with\-patch reference\. Both predictors output an ordinal consequence label in\{0,1,2\}\\\{0,1,2\\\}with a single forward pass\.

Agreement with the with\-patch reference\.The primary Qwen issue\-only predictor reaches Cohen’sκ=0\.572\\kappa=0\.572on all300300SWE\-bench Lite tasks, indicating substantial agreement on a three\-class ordinal labeling problem\. The cross\-model Claude issue\-only predictor reachesκ=0\.379\\kappa=0\.379, lower than expected for a different model family, but remains above chance and useful as an independent robustness check\. These results suggest that consequence is not only a post\-hoc property of the gold patch; it is often inferable from the issue description before the fix is computed\.

The errors that matter\.For compute allocation, aggregate agreement is not the only criterion\. The dangerous error for scheduling is not a small label mismatch but a severe under\-estimation of risk, namely routing a high\-consequence task to the lowest compute tier\. We therefore inspect the confusion matrix, focusing on the class\-2→\\toclass\-0 cell\.

For the primary Qwen predictor, the with\-patch reference labels4444tasks as high consequence\. The issue\-only predictor labels3939of them as class 2 and the remaining55as class 1, yielding class\-2 recall of88\.6%88\.6\\%\. Crucially, it never predicts class 0 for a class 2 task, giving a severe under\-allocation rate of0/44=0%0/44=0\\%\. The cross\-model Claude predictor has lower class\-2 recall,52\.3%52\.3\\%\(23/4423/44\), but it preserves the same safety\-critical property: class\-2→\\toclass\-0 remains0/440/44\. This suggests that the absence of high\-to\-low errors is driven by the issue text and the consequence rubric rather than by a single model family \(Table[2](https://arxiv.org/html/2606.04402#S5.T2); Figure[2](https://arxiv.org/html/2606.04402#S5.F2)\)\. Full per\-class metrics are reported in Appendix[C](https://arxiv.org/html/2606.04402#A3)\.

Table 2:Predictor calibration vs\. the LLM\-with\-patch referenceon 300 SWE\-bench Lite tasks\. The deployment\-critical “never miss a high\-consequence task as low” property \(Class\-2→\\,\\to\\,Class\-0=0/44=0/44\) holds for both predictors\. The cross\-model Claude predictor has lower aggregate agreement and lower class\-2 recall, but preserves the same safety\-critical under\-allocation property\.![Refer to caption](https://arxiv.org/html/2606.04402v1/x2.png)Figure 2:Predictor confusion matricesagainst the LLM\-with\-patch reference on 300 SWE\-bench Lite tasks\. Rows denote the reference class and columns denote the predicted class\. The deployment\-critical under\-allocation cell, true class 2 and predicted class 0, is empty for both predictors\. The primary Qwen predictor \(left\) has higher class\-2 recall \(39/4439/44\) than the cross\-model Claude predictor \(right;23/4423/44\), but both avoid the high\-to\-low error that would route a high\-consequence task to the lowest compute tier\.This predictor is imperfect, but its error pattern is useful for deployment\. When the predictor is wrong, it typically errs to an adjacent class, so a high\-consequence task is routed at worst to a medium budget rather than to the cheapest tier\. In §[7](https://arxiv.org/html/2606.04402#S7), we show that this safety\-respecting predictor is sufficient for downstream allocation: on SWE\-bench Lite, predictor\-driven routing retains most of the gain achieved by oracle\-consequence routing under the same compute budget\.

## 6Method: Cost\-Weighted Compute Allocation

The previous sections establish that consequence is a measurable, task\-level cost signal distinct from difficulty \(§[3](https://arxiv.org/html/2606.04402#S3)\), that current thinking models do not allocate compute by it on their own \(§[4](https://arxiv.org/html/2606.04402#S4)\), and that it can be predicted before a task is solved from issue\-level information alone \(§[5](https://arxiv.org/html/2606.04402#S5)\)\. Together these make an explicit scheduling layer both necessary and feasible\. We now specify how that scheduler should use the consequence signal under a fixed compute budget\.

The setup is a multi\-tier compute system: a finite set of compute tiers\{T1,…,TK\}\\\{T\_\{1\},\\ldots,T\_\{K\}\\\}ordered from cheap to expensive, where each tier is either a different model family, a different sampling temperature, a different self\-consistency budget, or a different thinking\-token budget on the same model\. The deployer has a total compute budgetBBto spend across a stream of tasksXX\.

Objective\.Minimizing the cost\-weighted loss on the task stream under a total\-compute constraint is

minT:X→\{T1,…,TK\}​∑x∈Xcost​\(x\)⋅\[1−p​\(correct∣x,T​\(x\)\)\]s\.t\.∑x∈Xc​\(T​\(x\)\)≤B,\\min\_\{T:\\,X\\to\\\{T\_\{1\},\\dots,T\_\{K\}\\\}\}\\ \\sum\_\{x\\in X\}\\,\\mathrm\{cost\}\(x\)\\cdot\\bigl\[\\,1\-p\\bigl\(\\text\{correct\}\\mid x,T\(x\)\\bigr\)\\bigr\]\\quad\\text\{s\.t\.\}\\quad\\sum\_\{x\\in X\}c\(T\(x\)\)\\leq B,\(1\)wherec​\(Tk\)c\(T\_\{k\}\)is the per\-task compute price of tierTkT\_\{k\}\. Withcost​\(x\)≡1\\mathrm\{cost\}\(x\)\{\\equiv\}1, Eq\.[1](https://arxiv.org/html/2606.04402#S6.E1)reduces to plain accuracy maximization under a compute budget—the objective implicit in prior adaptive\-compute work\.

Algorithm\.We instantiate Eq\.[1](https://arxiv.org/html/2606.04402#S6.E1)with a simple decoupled scheme\. Given the predicted consequencec^​\(x\)∈\{0,1,2\}\\hat\{c\}\(x\)\\in\\\{0,1,2\\\}from §[5](https://arxiv.org/html/2606.04402#S5)and a target overall compute budgetBB, we sort tasks byc^​\(x\)\\hat\{c\}\(x\)in descending order and assign them to compute tiers from premium to cheap until the budget is exhausted \(Algorithm[1](https://arxiv.org/html/2606.04402#alg1)\)\. In the fixed\-quota variants of §[7](https://arxiv.org/html/2606.04402#S7), the top\-q%q\\%tasks byc^\\hat\{c\}go to the premium tier and the rest go to the cheap tier;qqis chosen so that total compute matches the difficulty\-based baseline\. This decoupled scheme assumes a monotone relationship betweenc^\\hat\{c\}and the marginal value of more compute—an assumption we test directly in §[7](https://arxiv.org/html/2606.04402#S7)\.

Algorithm 1Cost\-weighted compute allocation0:Tasks

XX; predictor

c^\\hat\{c\}; tiers

T1,…,TKT\_\{1\},\\ldots,T\_\{K\}with prices

c​\(Tk\)c\(T\_\{k\}\); total budget

BB\.

1:Score each task:

s​\(x\)←c^​\(x\)s\(x\)\\leftarrow\\hat\{c\}\(x\)
2:Sort

XXby

s​\(⋅\)s\(\\cdot\)in descending order

3:Initialize spent

←0\\leftarrow 0,

T​\(x\)←T1T\(x\)\\leftarrow T\_\{1\}for all

xx
4:for

k=Kk=Kdown to

22do

5:for

xxin sorted orderdo

6:if

T​\(x\)=T1T\(x\)=T\_\{1\}and spent

\+\[c​\(Tk\)−c​\(T1\)\]≤B−∑x′c​\(T​\(x′\)\)\+\[c\(T\_\{k\}\)\-c\(T\_\{1\}\)\]\\leq B\-\\sum\_\{x^\{\\prime\}\}c\(T\(x^\{\\prime\}\)\)then

7:

T​\(x\)←TkT\(x\)\\leftarrow T\_\{k\}
8:spent

←\\leftarrowspent

\+\[c​\(Tk\)−c​\(T1\)\]\+\[c\(T\_\{k\}\)\-c\(T\_\{1\}\)\]
9:endif

10:endfor

11:endfor

12:return

TT

The scheme has three properties worth highlighting\. First, it does not retrain any reasoning model; it sits entirely at the scheduling layer\. Second, the predictor only needs to be ordinal\-correct on the safety dimension—ranking class\-2 tasks above class\-0 tasks matters far more than getting the absolute label right\. Third, the scheme degrades gracefully: with a perfect predictor we recover oracle\-consequence routing, with a random predictor we recover random routing, and with the difficulty score substituted forc^\\hat\{c\}we recover the Snell\-style difficulty\-aware baseline\.

## 7Experiments

We test Algorithm[1](https://arxiv.org/html/2606.04402#alg1)on a 16\-model SWE\-bench compute\-tier benchmark\.

Setup\.We aggregate the per\-task resolution status of 16 public SWE\-bench leaderboard solvers spanning four years of model generations\. These include RAG baselines on Claude 2 and GPT\-4, SWE\-agent variants on Claude 3 Opus and Claude 3\.5 Sonnet, and more recent agentic systems on Claude 4 Sonnet, including SWE\-agent, ExpeRepair, and KGCompass\. We sort the 16 models by total resolved count and form two tiers: a cheap tier consisting of the bottom 4 models, and a premium tier consisting of the top 4 models\. For each task,pcheap​\(x\)p\_\{\\mathrm\{cheap\}\}\(x\)is the empirical fraction of cheap\-tier solvers that resolve the task\. We defineppremium​\(x\)p\_\{\\mathrm\{premium\}\}\(x\)analogously for the premium tier\. Unless otherwise stated, we assume a fixed compute price ratioc​\(premium\)/c​\(cheap\)=4c\(\\text\{premium\}\)/c\(\\text\{cheap\}\)=4, consistent with typical model\-tier price gaps on commercial APIs\.

Strategies\.We compare seven allocation strategies under a matched total compute budget\. Random routes a uniformly random top\-q%q\\%subset of tasks to the premium tier\. Difficulty\-aware \(Snell\-style\) routes the top\-q%q\\%by predicted difficulty\(Snellet al\.,[2024](https://arxiv.org/html/2606.04402#bib.bib1)\)\. Consequence\-aware \(predictor\) routes the top\-q%q\\%by the issue\-only consequence predictor of §[5](https://arxiv.org/html/2606.04402#S5)\. We evaluate two predictor variants: the within\-model Qwen predictor, which is our primary deployable variant, and the cross\-model Claude predictor, which serves as a robustness check\. Consequence\-aware \(oracle\) replaces the predictor with the LLM\-with\-patch label, isolating the upper bound of a one\-signal consequence router\. Finally, Priority\-aware \(predictor/oracle\) routes the top\-q%q\\%byc^​\(x\)⋅\[ppremium​\(x\)−pcheap​\(x\)\]\\hat\{c\}\(x\)\\cdot\[p\_\{\\mathrm\{premium\}\}\(x\)\-p\_\{\\mathrm\{cheap\}\}\(x\)\]\. This two\-signal variant accounts not only for the cost of an error, but also for whether premium compute is likely to change the outcome\.

Metric\.We report cost\-weighted loss

ℒ​\(T\)=∑xcons​\(x\)⋅\[1−pT​\(x\)​\(x\)\],\\mathcal\{L\}\(T\)=\\sum\_\{x\}\\mathrm\{cons\}\(x\)\\cdot\[\\,1\-p\_\{T\(x\)\}\(x\)\\,\],wherecons​\(x\)\\mathrm\{cons\}\(x\)is the LLM\-with\-patch consequence label andpT​\(x\)​\(x\)p\_\{T\(x\)\}\(x\)is the empirical success probability of the tier assigned to taskxx\. The main table reports results onN=300N=300SWE\-bench Lite tasks at a matched\-compute operating point where the top\-25%25\\%of tasks are routed to the premium tier\.

Table 3:Main results\.Cost\-weighted lossℒ\\mathcal\{L\}, relative reduction over the difficulty\-aware baseline, and average accuracy onN=300N\{=\}300SWE\-bench Lite tasks\. All strategies are evaluated at the same compute budget: top\-25%25\\%premium routing with a cheap/premium price ratio of1:41\{:\}4\. Difficulty\-aware allocation is worse than Random at this operating point\. The priority\-aware Qwen predictor is the primary deployable variant and recovers92\.9%92\.9\\%of the priority\-oracle gain\. The simpler consequence\-only Qwen predictor retains93\.6%93\.6\\%of its corresponding consequence\-oracle gain\.Strategyℒ\\mathcal\{L\}Δ\\Deltavs DiffAccuracyDifficulty\-aware \(Snell\-style\)\(Snellet al\.,[2024](https://arxiv.org/html/2606.04402#bib.bib1)\)268\.25268\.250%0\\%baseline0\.0540\.054Random233\.00233\.00\+13\.1%\+13\.1\\%0\.1700\.170Consequence\-aware \(Claude predictor; cross\-model robustness\)220\.50220\.50\+17\.8%\+17\.8\\%0\.1740\.174Consequence\-aware \(Qwen predictor; deployable\)209\.75209\.75\+21\.8%\+21\.8\\%0\.1840\.184Consequence\-aware \(oracle\)205\.75205\.75\+23\.3%\+23\.3\\%0\.1870\.187Priority\-aware \(Qwen predictor;primary deployable variant\)186\.00\\mathbf\{186\.00\}\+30\.7%\\mathbf\{\+30\.7\\%\}0\.274\\mathbf\{0\.274\}Priority\-aware \(oracle; upper bound\)179\.75179\.75\+33\.0%\+33\.0\\%0\.2780\.278![Refer to caption](https://arxiv.org/html/2606.04402v1/x3.png)Figure 3:Pareto curve: cost\-weighted loss vs total computeon the 16\-model SWE\-bench compute\-tier benchmark\. Each curve sweeps the fraction of tasks routed to the premium tier from0to100%100\\%\. The difficulty\-aware curve is flat across much of the budget range and is dominated by the random baseline\. This indicates that additional premium compute spent on the hardest tasks does not reliably reduce cost\-weighted loss, because many of those tasks remain unsolved even at the premium tier\. The priority\-aware variants Pareto\-dominate all others, and the Qwen\-predictor\-driven version closely tracks the oracle across the budget axis\. The dotted vertical line marks the top\-25%25\\%premium operating point used in Table[3](https://arxiv.org/html/2606.04402#S7.T3)\.Consequence\-aware allocation improves cost\-weighted routing\.Table[3](https://arxiv.org/html/2606.04402#S7.T3)reports the main results at the top\-25%25\\%premium operating point, and Figure[3](https://arxiv.org/html/2606.04402#S7.F3)shows the full Pareto curve across compute budgets\. The simplest deployable consequence\-only router, driven by the Qwen issue\-only predictor, reduces cost\-weighted loss by21\.8%21\.8\\%relative to the difficulty\-aware baseline\. Its oracle counterpart, which uses the LLM\-with\-patch consequence label, reduces loss by23\.3%23\.3\\%\. Thus, the predictor retains93\.6%93\.6\\%of the consequence\-oracle gain\. When we additionally account for marginal gain, the priority\-aware Qwen predictor reduces cost\-weighted loss by30\.7%30\.7\\%at the same compute budget\. The priority\-aware oracle reduces loss by33\.0%33\.0\\%, so the predictor retains92\.9%92\.9\\%of the priority\-oracle gain\. These results show that the safety\-respecting predictor of §[5](https://arxiv.org/html/2606.04402#S5)translates directly into downstream routing quality\.

Difficulty\-aware allocation is worse than random at the main operating point\.The most striking result in Table[3](https://arxiv.org/html/2606.04402#S7.T3)is that difficulty\-aware routing underperforms the random baseline by13\.1%13\.1\\%at the top\-25%25\\%premium budget\. A top\-25%25\\%selection ranked by difficulty captures0%0\\%of the cost\-weighted gain available in the data\. In contrast, random selection captures24%24\\%, consequence\-only oracle selection captures43%43\\%, and priority\-oracle selection captures59%59\\%\. This suggests that difficulty is not merely an incomplete routing signal under a cost\-weighted objective; at this operating point, it routes premium compute toward tasks with low marginal return\. The next section analyzes this failure mechanism by decomposing consequence, difficulty, confidence, and marginal utility\.

## 8Why Difficulty\-Aware Allocation Fails

The results of §[7](https://arxiv.org/html/2606.04402#S7)raise a natural question: Why does a strategy that allocates more compute to harder tasks perform worse than allocating randomly? We answer this question in two steps\. First, we rule out the alternative interpretation that consequence is merely a re\-skinning of cross\-model disagreement, which is the signal that many confidence\-based adaptive\-compute methods already exploit\. Second, we examine per\-task marginal utility and show that difficulty is anti\-correlated with where additional compute actually helps\.

### 8\.1Confounding ablation: consequence is not a proxy of confidence

One might argue that consequence works only because it inadvertently captures another useful signal, namely cross\-model disagreement on a task, which acts as a confidence proxy\. To test this alternative, we construct

Y​\(x\)=cons​\(x\)⋅\[ppremium​\(x\)−pcheap​\(x\)\],Y\(x\)\\;=\\;\\mathrm\{cons\}\(x\)\\cdot\\bigl\[\\,p\_\{\\mathrm\{premium\}\}\(x\)\-p\_\{\\mathrm\{cheap\}\}\(x\)\\,\\bigr\],the cost\-weighted marginal gain of routing taskxxto the premium tier\. We then ask whether consequence carries information aboutY​\(x\)Y\(x\)beyond what is already explained by difficulty and per\-task pass\-rate variance\. The latter measures how much public solvers disagree on the task, and serves as our confidence proxy\. Controlling for both difficulty and pass\-rate variance, the partial Spearman correlation remains large:

ρ​\(cons,Y\|diff,passvar\)=\+0\.696,p<10−43\.\\rho\\bigl\(\\,\\mathrm\{cons\},\\,Y\\,\\big\|\\,\\mathrm\{diff\},\\,\\mathrm\{passvar\}\\,\\bigr\)=\+0\.696,\\quad p<10^\{\-43\}\.A hierarchical regression onY​\(x\)Y\(x\)confirms the same conclusion, as shown in Table[4](https://arxiv.org/html/2606.04402#S8.T4)\. Difficulty plus pass\-rate variance explainsR2=0\.474R^\{2\}=0\.474, while adding consequence increases this toR2=0\.738R^\{2\}=0\.738\. This is an absolute gain of26\.426\.4percentage points, or\+55\.7%\+55\.7\\%relative\. Moreover, difficulty contributes almost no incremental variance once pass\-rate variance is included \(Rdiff\+passvar2−Rpassvar2=0\.0004R^\{2\}\_\{\\mathrm\{diff\}\+\\mathrm\{passvar\}\}\-R^\{2\}\_\{\\mathrm\{passvar\}\}=0\.0004\)\. Thus, the gain of priority\-aware routing in Table[3](https://arxiv.org/html/2606.04402#S7.T3)is not produced by recovering a difficulty or confidence signal in disguise\.

Table 4:R2decomposition on the cost\-weighted marginal valueYY\.YYmeasures the value of upgrading a task from the cheap tier to the premium tier after weighting by consequence\. Difficulty alone explains28%28\\%of the variance, and pass\-rate variance, our confidence proxy, explains47%47\\%\. Adding consequence on top of both liftsR2R^\{2\}from0\.4740\.474to0\.7380\.738, an absolute gain of26\.426\.4percentage points \(\+55\.7%\+55\.7\\%relative\)\.![Refer to caption](https://arxiv.org/html/2606.04402v1/x4.png)Figure 4:Why difficulty\-aware allocation fails:Δ\\Deltasuccess collapses on the hardest tasks\.Each marker is one of the 300 SWE\-bench Lite tasks, colored by consequence class\. The marginal gainΔ​success=ppremium​\(x\)−pcheap​\(x\)\\Delta\\mathrm\{success\}=p\_\{\\mathrm\{premium\}\}\(x\)\-p\_\{\\mathrm\{cheap\}\}\(x\)measures how much accuracy improves when taskxxis routed to the premium tier\. The binned trend line rises through the moderate\-difficulty region and then collapses on the hardest tasks, producing a strong negative correlation \(ρ=−0\.78\\rho=\-0\.78,p<10−60p<10^\{\-60\}\)\. A scheduler that allocates premium compute to the difficulty tail therefore spends budget where extra compute rarely changes the outcome\.
### 8\.2Marginal utility: hardest tasks are unsolvable at any tier

LetΔ​success​\(x\)=ppremium​\(x\)−pcheap​\(x\)\\Delta\\mathrm\{success\}\(x\)=p\_\{\\mathrm\{premium\}\}\(x\)\-p\_\{\\mathrm\{cheap\}\}\(x\)denote the marginal accuracy gain from upgrading taskxxfrom the cheap tier to the premium tier\. This quantity measures where extra compute changes the outcome\. Three observations explain the failure of difficulty\-aware allocation\.

Difficulty is anti\-correlated with marginal gain\.ρ​\(difficulty,Δ​success\)=−0\.78\\rho\(\\mathrm\{difficulty\},\\Delta\\mathrm\{success\}\)=\-0\.78\(p<10−60p<10^\{\-60\}\)\. The hardest tasks are those on which all 16 models struggle\. They are also the tasks where even the premium tier often does not help\. A difficulty\-aware scheduler therefore concentrates its upgrade budget on tasks that cannot be saved by the available premium tier\.

Consequence is uncorrelated with marginal gain\.ρ​\(cons,Δ​success\)=\+0\.09\\rho\(\\mathrm\{cons\},\\Delta\\mathrm\{success\}\)=\+0\.09\(n\.s\.\)\. Consequence does not predict whether extra compute will help on a given task\. Instead, it predicts whether that help would be valuable if it occurs\.

Consequence and marginal gain combine multiplicatively\.The priority signal

priority​\(x\)=cons​\(x\)⋅Δ​success​\(x\)\\mathrm\{priority\}\(x\)=\\mathrm\{cons\}\(x\)\\cdot\\Delta\\mathrm\{success\}\(x\)ranks tasks by their cost\-weighted return on extra compute\. This decomposition matches the empirical ordering in Table[3](https://arxiv.org/html/2606.04402#S7.T3): priority\-aware routing performs best, followed by consequence\-only routing, random routing, and finally difficulty\-aware routing\. At the top\-25%25\\%premium operating point, a priority ranking captures59%59\\%of the total cost\-weighted gain available in the data\. A consequence\-only ranking captures43%43\\%, random selection captures24%24\\%, and difficulty ranking captures0%0\\%\. The gap between consequence\-only and priority\-aware routing is meaningful, but not transformative\. This is consistent with the\+21\.8%\+21\.8\\%versus\+30\.7%\+30\.7\\%loss reduction observed in Table[3](https://arxiv.org/html/2606.04402#S7.T3)\.

Together, these observations explain the negative result for difficulty\-aware allocation\. A cost\-blind scheduler that spends more compute on harder tasks spends its budget where extra compute has the lowest marginal return\. Figure[4](https://arxiv.org/html/2606.04402#S8.F4)visualizes this effect\.Δ​success\\Delta\\mathrm\{success\}rises with difficulty up to the moderate range and then collapses on the unsolvable tail, producing the strongly negative Spearman correlation\. Appendix[D](https://arxiv.org/html/2606.04402#A4)reports the per\-task view: the top\-ranked tasks by priority have moderate difficulty and largeΔ​success\\Delta\\mathrm\{success\}, whereas the top\-ranked tasks by difficulty all haveΔ​success=0\\Delta\\mathrm\{success\}=0\.

## 9Discussion and Limitations

Construct validity of consequence\.Throughout the main text, the primary consequence label is produced by an LLM judge that observes the issue, the gold patch, and the modified files\. This choice gives the judge access to the realized fix, but it also raises a construct validity concern: the label should reflect a human\-interpretable notion of deployment harm rather than an artifact of the LLM judge\. To address this, we conduct a three\-annotator human study on a150150\-task stratified sample drawn from SWE\-bench Lite and Multi\-SWE\-bench mini \(Appendix[A](https://arxiv.org/html/2606.04402#A1)\)\. The annotators apply the rubric consistently, and the LLM\-with\-patch judge agrees with the human majority atκ=0\.50\\kappa=0\.50\. More importantly, the central orthogonality result of §[3](https://arxiv.org/html/2606.04402#S3)replicates under human labeling \(ρ=−0\.066\\rho=\-0\.066, n\.s\.\)\. We therefore use the human labels as a construct\-validity check, while retaining the LLM\-with\-patch label as the primary signal in the main experiments because it covers the full SWE\-bench Lite task pool\.

Scope of evaluation\.Our main allocation experiment uses publicly available per\-task outcomes from 16 SWE\-bench Lite leaderboard solvers\. This design allows us to compare many compute tiers on the same task set and to estimate per\-task marginal gains from upgrading cheap models to premium models\. However, it is still an offline compute\-tier study rather than a fully controlled intervention on a single reasoning model\. A complementary evaluation would vary the thinking\-token budget of one fixed frontier model under the same allocation rule\. Another complementary direction is to evaluate domains where high\-consequence errors are more frequent, such as database operations, schema migrations, permission checks, and data consistency bugs\. These settings would test the same allocation principle under more controlled or more risk\-heavy deployment conditions\.

Cost is a coarse scalar\.We represent consequence as an ordinal label in\{0,1,2\}\\\{0,1,2\\\}\. This is a deliberate simplification\. Real deployment cost is multidimensional, including financial, reputational, safety, privacy, and regulatory harms, and the same bug may have different consequences in different production contexts\. Our labels should therefore be interpreted as a coarse ordering of deployment harm, not as calibrated monetary costs\. The framework in Eq\.[1](https://arxiv.org/html/2606.04402#S6.E1)naturally extends to richer cost models, including continuous costs or task\-dependent cost vectors\. The three\-class version used here is intended as a minimal instantiation that current LLM judges and human annotators can apply reliably from issue text and patches\.

## 10Conclusion

Adaptive test\-time compute methods for reasoning models have largely optimized average accuracy under a compute budget\. We argue that this objective is incomplete for deployment: tasks differ not only in how hard they are, but also in how costly their errors would be\. The missing signal is per\-task consequence\. We show that consequence is measurable, distinct from difficulty and confidence, predictable from issue text alone, and usable as a routing signal without retraining the reasoning model\.

On a 16\-model SWE\-bench compute\-tier benchmark, consequence\-only routing substantially reduces cost\-weighted loss relative to a difficulty\-aware baseline\. A priority\-aware variant that combines predicted consequence with estimated marginal gain reduces cost\-weighted loss by over30%30\\%, while a deployable predictor retains over90%90\\%of the corresponding oracle gain\. We also find that difficulty\-aware routing can underperform random allocation, because the hardest tasks often remain unsolved at every available tier and therefore have low marginal return from extra compute\.

These results suggest that test\-time compute allocation should not be treated only as a question of task difficulty\. For deployment, the central question is also what happens if the model is wrong, and whether extra computation is likely to prevent that error\.

## References

- M\. A\. Alomrani, Y\. Zhang, D\. Li, Q\. Sun, S\. Pal, Z\. Zhang, Y\. Hu, R\. D\. Ajwani, A\. Valkanas, R\. Karimi,et al\.\(2025\)Reasoning on a budget: a survey of adaptive and controllable test\-time compute in llms\.arXiv preprint arXiv:2507\.02076\.Cited by:[§1](https://arxiv.org/html/2606.04402#S1.p1.1),[§1](https://arxiv.org/html/2606.04402#S1.p2.1),[§1](https://arxiv.org/html/2606.04402#S1.p3.1)\.
- Learning how hard to think: input\-adaptive allocation of lm computation\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 102783–102802\.Cited by:[§1](https://arxiv.org/html/2606.04402#S1.p1.1),[§2](https://arxiv.org/html/2606.04402#S2.p1.1)\.
- C\. Elkan \(2001\)The foundations of cost\-sensitive learning\.InInternational joint conference on artificial intelligence,Vol\.17,pp\. 973–978\.Cited by:[§2](https://arxiv.org/html/2606.04402#S2.p3.1)\.
- Y\. Geifman and R\. El\-Yaniv \(2017\)Selective classification for deep neural networks\.Advances in neural information processing systems30\.Cited by:[§2](https://arxiv.org/html/2606.04402#S2.p3.1)\.
- N\. Hay, S\. Russell, D\. Tolpin, and S\. E\. Shimony \(2012\)Selecting computations: theory and applications\.CoRRabs/1207\.5879\.External Links:[Link](http://arxiv.org/abs/1207.5879)Cited by:[§1](https://arxiv.org/html/2606.04402#S1.p3.1),[§2](https://arxiv.org/html/2606.04402#S2.p2.1)\.
- C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. Narasimhan \(2024\)Swe\-bench: can language models resolve real\-world github issues?\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 54107–54157\.Cited by:[§2](https://arxiv.org/html/2606.04402#S2.p4.1)\.
- Z\. Li, D\. Zhang, M\. Zhang, J\. Zhang, Z\. Liu, Y\. Yao, H\. Xu, J\. Zheng, P\. Wang, X\. Chen, Y\. Zhang, F\. Yin, J\. Dong, Z\. Guo, L\. Song, and C\. Liu \(2025\)From system 1 to system 2: a survey of reasoning large language models\.CoRRabs/2502\.17419\.External Links:[Link](https://doi.org/10.48550/arXiv.2502.17419)Cited by:[§1](https://arxiv.org/html/2606.04402#S1.p1.1),[§1](https://arxiv.org/html/2606.04402#S1.p3.1)\.
- Y\. Liu, J\. Wu, Y\. He, H\. Gao, H\. Chen, B\. Bi, J\. Zhang, Z\. Huang, and B\. Hooi \(2025\)Efficient inference for large reasoning models: a survey\.CoRRabs/2503\.23077\.External Links:[Link](https://doi.org/10.48550/arXiv.2503.23077)Cited by:[§1](https://arxiv.org/html/2606.04402#S1.p4.6)\.
- R\. Manvi, A\. Singh, and S\. Ermon \(2025\)Adaptive inference\-time compute: LLMs can predict if they can do better, even mid\-generation\.External Links:[Link](https://openreview.net/forum?id=7tOc6h8bea)Cited by:[§1](https://arxiv.org/html/2606.04402#S1.p1.1),[§2](https://arxiv.org/html/2606.04402#S2.p1.1)\.
- Q\. Pan, W\. Ji, Y\. Ding, J\. Li, S\. Chen, J\. Wang, J\. Zhou, Q\. Chen, M\. Zhang, Y\. Wu, and L\. He \(2025\)A survey of slow thinking\-based reasoning llms using reinforced learning and inference\-time scaling law\.CoRRabs/2505\.02665\.External Links:[Link](https://doi.org/10.48550/arXiv.2505.02665)Cited by:[§1](https://arxiv.org/html/2606.04402#S1.p4.6)\.
- S\. Qu \(2026\)Adaptive test\-time compute allocation via learned heuristics over categorical structure\.arXiv preprint arXiv:2602\.03975\.Cited by:[§1](https://arxiv.org/html/2606.04402#S1.p3.1)\.
- S\. J\. Russell and E\. Wefald \(1991\)Do the right thing \- studies in limited rationality\.MIT Press\.External Links:ISBN 978\-0\-262\-18144\-0Cited by:[§1](https://arxiv.org/html/2606.04402#S1.p3.1),[§2](https://arxiv.org/html/2606.04402#S2.p2.1)\.
- N\. D\. Sabbata, T\. R\. Sumers, and T\. L\. Griffiths \(2024\)Rational metareasoning for large language models\.CoRRabs/2410\.05563\.External Links:[Link](https://doi.org/10.48550/arXiv.2410.05563)Cited by:[§1](https://arxiv.org/html/2606.04402#S1.p3.1),[§2](https://arxiv.org/html/2606.04402#S2.p2.1)\.
- B\. Sayin, J\. Yang, X\. Chen, A\. Passerini, and F\. Casati \(2025\)Rethinking and recomputing the value of machine learning models\.Artificial Intelligence Review58\(8\),pp\. 238\.Cited by:[§1](https://arxiv.org/html/2606.04402#S1.p3.1)\.
- R\. Shirkavand, S\. Gao, P\. Yu, and H\. Huang \(2026\)Cost\-aware contrastive routing for LLMs\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=4Qe2Hga43N)Cited by:[§1](https://arxiv.org/html/2606.04402#S1.p5.1)\.
- C\. Snell, J\. Lee, K\. Xu, and A\. Kumar \(2024\)Scaling llm test\-time compute optimally can be more effective than scaling model parameters\.arXiv preprint arXiv:2408\.03314\.Cited by:[§1](https://arxiv.org/html/2606.04402#S1.p1.1),[§2](https://arxiv.org/html/2606.04402#S2.p1.1),[Table 3](https://arxiv.org/html/2606.04402#S7.T3.17.5.4),[§7](https://arxiv.org/html/2606.04402#S7.p3.5)\.
- D\. Zan, Z\. Huang, W\. Liu, H\. Chen, S\. Xin, L\. Zhang, Q\. Liu, L\. Aoyan, L\. Chen, X\. Zhong,et al\.\(2026\)Multi\-swe\-bench: a multilingual benchmark for issue resolving\.Advances in Neural Information Processing Systems38\.Cited by:[§2](https://arxiv.org/html/2606.04402#S2.p4.1)\.
- J\. Zhu and H\. Li \(2025\)Towards concise and adaptive thinking in large reasoning models: a survey\.CoRRabs/2507\.09662\.External Links:[Link](https://doi.org/10.48550/arXiv.2507.09662)Cited by:[§1](https://arxiv.org/html/2606.04402#S1.p1.1)\.

## Appendix AHuman\-Label Robustness Check

This appendix re\-runs the two headline checks—F1 orthogonality \(§[3](https://arxiv.org/html/2606.04402#S3)\) and predictor agreement \(§[5](https://arxiv.org/html/2606.04402#S5)\)—with the consequence label supplied by a human majority rather than by an LLM judge, as a construct\-validity check on the labelings used in the main text\.

#### Study design\.

We sampled150150unique tasks, stratified by the LLM\-with\-patch consequence label, drawn7575from SWE\-bench Lite and7575from Multi\-SWE\-bench mini\. Each annotator’s sheet contained165165rows in a randomized order, with1515hidden duplicates \(five per class\) used to measure self\-consistency\. Three annotators participated: one of the authors, one CS PhD outside the project, and one non\-author CS graduate student; none had access to the gold patch, the file diff, or each other’s labels\. Each annotator first completed a99\-task calibration warm\-up with reference labels before the165165\-task evaluation\.

#### Annotators apply the rubric consistently\.

Each annotator’s self\-consistency is measured by Cohen’sκ\\kappaon the1515hidden duplicates: two annotators are perfectly self\-consistent \(κ=1\.000\\kappa=1\.000,100%100\\%raw agreement\) and the third reachesκ=0\.789\\kappa=0\.789\(86\.7%86\.7\\%\)\. The rubric is internally consistent within each annotator\.

#### Orthogonality replicates under human labels\.

On the7575SWE\-bench Lite tasks for which we have both human\-majority labels and the difficulty estimate of §[3](https://arxiv.org/html/2606.04402#S3),

ρ​\(human−cons,difficulty\)=−0\.066,p=0\.57\.\\rho\\bigl\(\\mathrm\{human\}\\\!\-\\\!\\mathrm\{cons\},\\ \\mathrm\{difficulty\}\\bigr\)=\-0\.066,\\quad p=0\.57\.The central orthogonality result of §[3](https://arxiv.org/html/2606.04402#S3)replicatesunder human labeling: consequence is statistically independent of difficulty whether the label is produced by a rule, by an LLM shown the gold patch, by an LLM shown only the issue, or by a human majority of three independent annotators \(see Appendix[B](https://arxiv.org/html/2606.04402#A2)for the full four\-labeling comparison\)\.

#### LLM labels agree with human majority\.

Table[5](https://arxiv.org/html/2606.04402#A1.T5)compares the human majority label against the three labeling pipelines used in the main text\. The LLM\-with\-patch judge agrees with the human majority at Cohen’sκ=0\.500\\kappa=0\.500, and the issue\-only predictor atκ=0\.517\\kappa=0\.517\. The rule\-based labeler agrees atκ=0\.223\\kappa=0\.223, consistent with its purely lexical construction\. Both LLM pipelines pick up the same construct as the human majority to a moderate degree, supporting their use as the primary labels in the main text\.

Table 5:Cohen’sκ\\kappabetween the human majority label and each of the labeling pipelines used in the main text, on the7575SWE\-bench Lite tasks of this study\.
#### Final human labels\.

We resolve each of the150150tasks by majority vote of the three annotators\.142142tasks have a clear majority \(≥2\\geq 2of33votes\); the remaining88are three\-way splits and are conservatively assigned to the middle class\. The resulting class distribution is\{0:45,1:67,2:38\}\\\{0\{:\}45,\\ 1\{:\}67,\\ 2\{:\}38\\\}\. These human\-majority labels are the basis for theρ=−0\.066\\rho=\-0\.066result above and for the corresponding row of Appendix[B](https://arxiv.org/html/2606.04402#A2)’s four\-labeling ablation\.

## Appendix BRubric Ablation: All Four Labelings Give the Same Conclusion

A natural reviewer concern is that the central results of the paper might be artefacts of the specific labeling pipeline used in the main text \(LLM\-with\-patch\)\. This appendix re\-runs the two headline analyses—F1 orthogonality \(§[3](https://arxiv.org/html/2606.04402#S3)\) and the matched\-compute allocation gain \(§[7](https://arxiv.org/html/2606.04402#S7)\)—separately under each of the four labelings of the rubric, and shows the conclusions are stable\.

F1 orthogonality replicates under every labeling\.Table[6](https://arxiv.org/html/2606.04402#A2.T6)reportsρ​\(consequence,difficulty\)\\rho\(\\mathrm\{consequence\},\\mathrm\{difficulty\}\)under the four labeling pipelines\. None of the four correlations reaches significance atα=0\.05\\alpha=0\.05, and all four are within±0\.11\\pm 0\.11of zero\. The construct is statistically independent of difficulty regardless of who or what produces the label\.

Table 6:Rubric ablation — orthogonality\.Spearmanρ​\(consequence,difficulty\)\\rho\(\\mathrm\{consequence\},\\mathrm\{difficulty\}\)on SWE\-bench Lite under each of the four labelings\.Allocation gain replicates under every labeling\.Table[7](https://arxiv.org/html/2606.04402#A2.T7)re\-runs the matched\-compute experiment of §[7](https://arxiv.org/html/2606.04402#S7)four times: in each row, the indicated labeling is substituted both as the oracle routing signal and as the weightcons​\(x\)\\mathrm\{cons\}\(x\)used in computing the cost\-weighted loss\. The priority\-aware oracle beats the difficulty\-aware baseline by between\+30\.8%\+30\.8\\%and\+36\.3%\+36\.3\\%across all four labelings, with the human labels giving the largest improvement on the 75\-task subset they cover\. The paper’s main claim \(“the priority\-aware variant achieves30%\+30\\%\+reduction in cost\-weighted loss”\) is therefore not a property of the LLM\-with\-patch labeler specifically: it is a property of the rubric\.

Table 7:Rubric ablation — allocation gain\.Cost\-weighted lossℒ\\mathcal\{L\}at the matched\-compute operating point \(top\-25%25\\%to the premium tier\) on the 16\-model SWE\-bench compute\-tier benchmark, recomputed four times—once per labeling pipeline—using that labeling both as the oracle routing signal and as the consequence weight inℒ\\mathcal\{L\}\. “Δ\\Deltavs Diff” is the relative reduction inℒ\\mathcal\{L\}achieved by the priority\-aware oracle relative to the difficulty\-aware \(Snell\-style\) baseline within the same labeling\.The fact that the rule\-based labeler—which is just a hand\-written lexical pattern matcher—and the human\-majority labeler—which never sees an LLM—both yield30%\+30\\%\+priority\-aware improvements rules out the “the gain is something the LLM judge invented” interpretation\. The remainder of the paper retains the LLM\-with\-patch label as the primary signal in the main results because it is the labeling under which all four supporting analyses \(§[3](https://arxiv.org/html/2606.04402#S3), §[5](https://arxiv.org/html/2606.04402#S5), §[7](https://arxiv.org/html/2606.04402#S7), §[8](https://arxiv.org/html/2606.04402#S8)\) are jointly applicable on the full 300\-task pool\.

## Appendix CPredictor Calibration Details

This appendix expands §[5](https://arxiv.org/html/2606.04402#S5)with full per\-class metrics and confusion matrices for both predictors evaluated against the LLM\-with\-patch reference on allN=300N\{=\}300SWE\-bench Lite tasks\. The class distribution of the with\-patch reference is\{0:60,1:196,2:44\}\\\{0\{:\}60,\\ 1\{:\}196,\\ 2\{:\}44\\\}\.

Primary predictor: Qwen3\-8B issue\-only\.Cohen’sκ=0\.572\\kappa=0\.572; class\-2 recall=88\.6%=88\.6\\%\(39/4439/44\)\. The confusion matrix is:

c^=0c^=1c^=2totalc=04713060c=11114045196c=2053944total5815884300\\begin\{array\}\[\]\{c\|ccc\|c\}&\\hat\{c\}\{=\}0&\\hat\{c\}\{=\}1&\\hat\{c\}\{=\}2&\\text\{total\}\\\\ \\hline\\cr c\{=\}0&47&13&0&60\\\\ c\{=\}1&11&140&45&196\\\\ c\{=\}2&0&5&39&44\\\\ \\hline\\cr\\text\{total\}&58&158&84&300\\end\{array\}Per\-class precision/recall/F1: class 0 \(0\.81/0\.78/0\.800\.81/0\.78/0\.80\), class 1 \(0\.89/0\.71/0\.790\.89/0\.71/0\.79\), class 2 \(0\.46/0\.89/0\.610\.46/0\.89/0\.61\)\. The Qwen predictor hits zero severe errors in both directions:\[2,0\]=0\[2,0\]=0\(no high\-stakes task missed as low\-stakes\) and\[0,2\]=0\[0,2\]=0\(no low\-stakes task escalated to high\-stakes\), giving an overall severe\-error rate of0/300=0\.0%0/300=0\.0\\%\.

Cross\-model robustness predictor: Claude Sonnet 4\.5 issue\-only\.Cohen’sκ=0\.379\\kappa=0\.379; class\-2 recall=52\.3%=52\.3\\%\(23/4423/44\)\. The confusion matrix is:

c^=0c^=1c^=2totalc=03325260c=12514823196c=20212344total5819448300\\begin\{array\}\[\]\{c\|ccc\|c\}&\\hat\{c\}\{=\}0&\\hat\{c\}\{=\}1&\\hat\{c\}\{=\}2&\\text\{total\}\\\\ \\hline\\cr c\{=\}0&33&25&2&60\\\\ c\{=\}1&25&148&23&196\\\\ c\{=\}2&0&21&23&44\\\\ \\hline\\cr\\text\{total\}&58&194&48&300\\end\{array\}Per\-class precision/recall/F1: class 0 \(0\.57/0\.55/0\.560\.57/0\.55/0\.56\), class 1 \(0\.76/0\.76/0\.760\.76/0\.76/0\.76\), class 2 \(0\.48/0\.52/0\.500\.48/0\.52/0\.50\)\. Despite a noticeably lower aggregateκ\\kappathan the within\-model predictor, the class\-2→\\toclass\-0 cell still holds at zero, and total severe errors in either direction sum to2/300=0\.7%2/300=0\.7\\%, supporting the deployment\-safety claim across model families\.

## Appendix DPer\-Task Marginal Utility \(Excerpt\)

This appendix supports §[8\.2](https://arxiv.org/html/2606.04402#S8.SS2)with a per\-task view of the priority signalpriority​\(x\)=cons​\(x\)⋅Δ​success​\(x\)\\mathrm\{priority\}\(x\)=\\mathrm\{cons\}\(x\)\\cdot\\Delta\\mathrm\{success\}\(x\)\. The complete per\-task table \(all300300rows\) is included with the submission as a supplementary CSV; Table[8](https://arxiv.org/html/2606.04402#A4.T8)below shows the top eight tasks by priority and the top four tasks by raw difficulty for contrast\.

Table 8:Per\-task marginal utility excerpt\. Top eight tasks by priority=cons⋅Δ​success=\\mathrm\{cons\}\\cdot\\Delta\\mathrm\{success\}and top four tasks by difficulty=1−passrate¯=1\-\\overline\{\\mathrm\{passrate\}\}\. The priority\-ranked tasks all have moderate difficulty and largeΔ​success\\Delta\\mathrm\{success\}; the difficulty\-ranked tasks all haveΔ​success≈0\\Delta\\mathrm\{success\}\\approx 0\(premium tier cannot solve them either\), so allocating extra compute to them returns zero cost\-weighted gain\.
## Appendix ERobustness Across Sub\-Domains

The 300\-task SWE\-bench Lite pool covers many distinct sub\-domains \(numeric computing, parsers, configuration, rendering, etc\.\)\. To check that the headline findings are not driven by any particular slice, we partition the pool by lexical signals in the issue text and re\-run the orthogonality, predictor\-calibration, and matched\-compute allocation analyses inside each sub\-domain\. Table[9](https://arxiv.org/html/2606.04402#A5.T9)reports four representative sub\-domains that together cover273273of the300300SWE\-bench Lite tasks\. In every sub\-domain the same pattern holds:ρ​\(consequence,difficulty\)\\rho\(\\mathrm\{consequence\},\\mathrm\{difficulty\}\)stays close to zero, the issue\-only predictor’sκ\\kappastays in the same range as the full\-pool0\.5720\.572, the deployment\-critical2→02\{\\to\}0cell remains empty, and priority\-aware oracle routing reduces cost\-weighted loss by30%\+30\\%\+\(priority\-aware variant\) over difficulty\-aware routing at the matched\-compute operating point\.

Table 9:Robustness across SWE\-bench Lite sub\-domains\.Each row reports the headline metrics of §[3](https://arxiv.org/html/2606.04402#S3)–§[7](https://arxiv.org/html/2606.04402#S7)inside a sub\-domain defined by lexical filters on the issue text\.ρ\\rhois the Spearman correlation between LLM\-with\-patch consequence and difficulty;κ\\kappais the issue\-only Qwen predictor’s agreement with the with\-patch reference; “2→02\{\\to\}0” is the count of class\-2 tasks predicted as class\-0 \(the safety\-critical under\-allocation cell\); “Δ\\Deltavs Diff” is the relative reduction in cost\-weighted loss achieved by priority\-aware oracle routing over difficulty\-aware routing at the top\-25% premium operating point\.The four sub\-domains together cover the bulk of the SWE\-bench Lite pool and span very different software\-engineering settings \(numerical algorithms, command\-line parsing, visualisation, and package configuration\)\. The replication of all four headline metrics across all four sub\-domains rules out the alternative interpretation that the paper’s results are an artefact of a particular topic mix in the benchmark\.