TRIAGE: Evaluating Prospective Metacognitive Control in LLMs under Resource Constraints

# Triage: Evaluating Prospective Metacognitive Control in LLMs under Resource Constraints
Source: [https://arxiv.org/html/2605.13414](https://arxiv.org/html/2605.13414)
###### Abstract

Deploying language models as autonomous agents requires more than per\-task accuracy: when an agent faces a queue of problems under a finite token budget, it must decide which to attempt, in what order, and how much compute to commit to each, all before any execution feedback is available\. This is the prospective form of metacognitive control studied for decades in human cognition, yet whether language models possess it remains untested\. We introduce TRIAGE, an evaluation framework in which a model receives a task pool and a token budget calibrated to its own baseline cost, and commits to a single ordered plan that jointly encodes selection, sequencing, and per\-problem allocation\. Plans are scored against an oracle with full knowledge of the model’s solvability and cost on each problem, yielding a triage efficiency ratio on a common scale\. We evaluate frontier and open\-source models, with and without reasoning enabled, across competition mathematics, graduate\-level science, code generation, and expert multidisciplinary knowledge, and find that current language models exhibit substantial gaps in prospective metacognitive control, revealing a previously unmeasured capability dimension with direct implications for resource\-efficient agent deployment\.

Machine Learning, ICML, Metacognition, LLM, Resource Constraints

## 1Introduction

Scaling test\-time compute has become a major paradigm for improving LLM reasoning\(Snellet al\.,[2025](https://arxiv.org/html/2605.13414#bib.bib1)\), but it has also made compute allocation a central unsolved problem: current reasoning models lack reliableprospective controlover their own compute expenditure, and adaptive allocation per problem can yield substantial efficiency gains over uniform budgets\(Snellet al\.,[2025](https://arxiv.org/html/2605.13414#bib.bib1); Alomraniet al\.,[2025](https://arxiv.org/html/2605.13414#bib.bib2)\)\. Reasoning models generate thousands of tokens even for trivial problems\(Chenet al\.,[2024](https://arxiv.org/html/2605.13414#bib.bib3)\)and underthink hard ones, prematurely abandoning promising reasoning paths\(Wanget al\.,[2025](https://arxiv.org/html/2605.13414#bib.bib4)\)\.Unconstrained agents, similarly, can consume thousands of tokens on agentic tasks and still fail, exhibiting analysis paralysis and wasted compute that correlates with decreased performance\(Cuadronet al\.,[2025](https://arxiv.org/html/2605.13414#bib.bib5)\)\.

This creates a concrete deployment problem: agents built on these models face real resource constraints \- token budgets, tool\-call limits, latency windows \- and must decide which tasks to attempt before committing resources\. A resource\-rational agent should allocate finite compute to the tasks where its expected return is highest\(Lieder and Griffiths,[2020](https://arxiv.org/html/2605.13414#bib.bib6)\)\. In practice, this means an LLM facing a queue of problems under a token budget should abstain from attempting ones it cannot solve, spend less on easy ones, and safely commit sufficient resources to each attempted problem so that under\-allocation does not expend budget on partial or incomplete solutions that yield no return \- the same strategy a student applies on a timed exam \- the capability thatNelson and Narens \([1990](https://arxiv.org/html/2605.13414#bib.bib7)\)formalized as*metacognitive control*: regulating effort allocation based on judgments of one’s own knowledge state\.

Recent work suggests LLMs have some of this capability in isolation: large language models can predict whether they will answer a question correctly\(Kadavathet al\.,[2022](https://arxiv.org/html/2605.13414#bib.bib8)\), yet reasoning\-trained models are worse \- not better \- at knowing when to abstain\(Kirichenkoet al\.,[2025](https://arxiv.org/html/2605.13414#bib.bib9)\)\.Barkanet al\.\([2025](https://arxiv.org/html/2605.13414#bib.bib10)\)found that frontier models are systematically overconfident about their own capabilities, and that this overconfidence does not diminish with scale or reasoning augmentation\. The critical capability for an LLM planner is not just predicting token cost or knowing when to abstain from a single question, but managing a portfolio: given a set of tasks and a shared budget, selecting which to attempt, estimating cost, and ordering execution to maximize value\. This is the problem that many real\-world agentic deployments face \- coding agents working through backlogs of issues, research agents decomposing queries into sub\-questions of varying difficulty, and planning agents budgeting compute across subtasks of a larger goal \- all require cross\-task prioritization under finite compute\. Existing per\-task budgeting methods estimate token cost for a single problem\(Hanet al\.,[2025](https://arxiv.org/html/2605.13414#bib.bib40); Liet al\.,[2025](https://arxiv.org/html/2605.13414#bib.bib12)\), and external batch schedulers rank or route requests for throughput\(Jinet al\.,[2023](https://arxiv.org/html/2605.13414#bib.bib13); Fuet al\.,[2024](https://arxiv.org/html/2605.13414#bib.bib14)\)\. However, whether LLMs can themselves perform the joint optimization \- assessing their own feasibility, cost, and priority across a heterogeneous task set under a shared budget \- remains untested, and answering it is a prerequisite for building agents that can plan and optimize their own resource use rather than relying on external orchestration\.

We introduce TRIAGE, an evaluation framework forprospective metacognitive controlunder resource constraints\. The construct has three components\.Metacognitive controlis the use of self\-knowledge to regulate behavior, graded by the consequences of the model’s plan rather than by verbalized confidence\(Ackerman,[2026](https://arxiv.org/html/2605.13414#bib.bib15)\)\.Prospectivedenotes that all decisions are committed before any task is attempted, distinguishing TRIAGE from retrospective measures of confidence or post\-hoc calibration\(Kadavathet al\.,[2022](https://arxiv.org/html/2605.13414#bib.bib8); Koriat,[1997](https://arxiv.org/html/2605.13414#bib.bib19)\)\.Resource constraintsrefers to a finite shared budget \- output tokens by default \- that the model’s behavior consumes\(Hanet al\.,[2025](https://arxiv.org/html/2605.13414#bib.bib40)\)\. In TRIAGE, a model receives a set of problems, point values per problem, and a token budget calibrated to its own baseline cost on the set\. It returns a single ordered plan specifying which problems to attempt, how many tokens to allocate to each, and in what order\. This design operationalizes three foundational control functions fromNelson and Narens \([1990](https://arxiv.org/html/2605.13414#bib.bib7)\)in their prospective form: selection \(which problems to attempt\), allocation \(how much compute to commit per problem\), and termination \(where to commit budget cutoffs ex ante\)\. TRIAGE evaluates the control output directly rather than asking the model to verbalize its underlying judgments \- appropriate for LLMs, where self\-reports are an unreliable signal of internal state and metacognition is better measured behaviorally\(Ackerman,[2026](https://arxiv.org/html/2605.13414#bib.bib15)\)\. Just as human learners plan effort allocation under time pressure\(Son and Metcalfe,[2000](https://arxiv.org/html/2605.13414#bib.bib16)\), an autonomous agent must jointly optimize them to maximize expected utility under finite inference limits\.

We evaluate 20 model architectures \- 4 standard LLMs and 16 reasoning\-augmented LLMs, evaluated both with and without thinking enabled \- across four domains: competition mathematics \(AIME 2024–2025\), graduate\-level science \(GPQA Diamond;Reinet al\.,[2024](https://arxiv.org/html/2605.13414#bib.bib45)\), code generation \(LiveCodeBench;Jainet al\.,[2024](https://arxiv.org/html/2605.13414#bib.bib17)\), and multidisciplinary expert knowledge \(Humanity’s Last Exam;Phanet al\.,[2025](https://arxiv.org/html/2605.13414#bib.bib18)\)\.

We find that current language models exhibit substantial and uneven gaps in prospective metacognitive control\. Strong self\-assessment in one domain rarely transfers to another, extended reasoning improves task\-level accuracy without improving triage quality, and reasoning\-trained models are less likely to recognize unsolvable items when they appear in a task pool\. The gap between advisory and enforced regimes is wide across budget levels, indicating that models can sometimes choose what to attempt but rarely commit budgets that they themselves can honor\. Together, these experiments characterize a previously unmeasured capability dimension with direct implications for resource\-efficient agent deployment\.

## 2Related Work

### 2\.1Metacognition in Large Language Models

Metacognition, the capacity to monitor and regulate one’s own cognitive processes\(Nelson and Narens,[1990](https://arxiv.org/html/2605.13414#bib.bib7)\), has become a central concern in LLM evaluation\. Research has progressed through three levels of investigation: metacognitive*knowledge*\(can the model identify what a task requires?\), metacognitive*monitoring*\(can the model judge whether it will succeed?\), and metacognitive*control*\(can the model act on those judgments to regulate its behavior?\)\. Existing work has made progress on the first two; the third remains untested because no current evaluation forces a model to commit, in advance, to a portfolio of tasks under a shared budget – a regime where control becomes operationally distinct from monitoring\.

#### Metacognitive knowledge and monitoring\.

Kadavathet al\.\([2022](https://arxiv.org/html/2605.13414#bib.bib8)\)provided the foundational result on monitoring, showing that LLMs can predict which questions they will answer correctly, with calibration improving at scale\.Didolkaret al\.\([2024](https://arxiv.org/html/2605.13414#bib.bib61)\)demonstrated that LLMs also possess metacognitive*knowledge*: given a math problem, frontier models can assign meaningful skill labels and use skill\-matched exemplars to improve reasoning, establishing that models can assess task properties\.Xionget al\.\([2024](https://arxiv.org/html/2605.13414#bib.bib62)\)systematically benchmarked confidence elicitation methods and found that LLMs are consistently overconfident when verbalizing confidence, though both calibration and failure prediction improve with scale\.Wang and others \([2025](https://arxiv.org/html/2605.13414#bib.bib63)\)formalized the separation of metacognitive ability from raw cognitive ability via their DMC framework, finding that stronger LLMs exhibit stronger metacognition and that enhancing metacognition alleviates hallucination\. At the mechanistic level,Ji\-Anet al\.\([2025](https://arxiv.org/html/2605.13414#bib.bib21)\)demonstrated that LLMs can monitor and control a low\-dimensional subspace of their internal activations via a neurofeedback paradigm – direct evidence of metacognitive access to internal states, though limited to concurrent activations rather than prospective task\-level judgments\.

These results are counterbalanced by evidence of systematic failure\.Kapooret al\.\([2024](https://arxiv.org/html/2605.13414#bib.bib20)\)showed that verbalized confidence is poorly calibrated for open\-ended generation and that a classifier trained on internal features substantially outperforms prompting – implying that metacognitive monitoring is latent in representations but not reliably accessible through natural outputs\.Griot and others \([2025](https://arxiv.org/html/2605.13414#bib.bib64)\)found that LLMs provide confident answers to medical questions even when correct options are entirely absent, revealing a disconnect between stated confidence and actual capability\.

Ackerman \([2026](https://arxiv.org/html/2605.13414#bib.bib15)\)used behavioral paradigms where the model must decide before answering whether to attempt or defer \(the Delegate Game\) and found only rudimentary, context\-dependent metacognition that depends on training regimen rather than scale\.Steyvers and Peters \([2025](https://arxiv.org/html/2605.13414#bib.bib22)\)organize these mixed findings by distinguishing metacognitive*sensitivity*\(ranking own performance across items\) from*calibration*\(matching confidence to success rate\), noting that the two can vary independently and that neither is reliably strong across models and domains\.

#### From monitoring to control\.

The work above measures whether models can assess their own capabilities\. Real deployment requires*acting*on those assessments to allocate resources\. The closest prior work to ours isBarkanet al\.\([2025](https://arxiv.org/html/2605.13414#bib.bib10)\), who explicitly distinguish in\-advance from after\-the\-fact confidence and test prospective self\-assessment on coding and agentic tasks \(SWE\-Bench Verified\)\. They find that frontier models are systematically overconfident, and that neither scale nor reasoning augmentation improves prediction accuracy\. Their resource\-acquisition experiment \- accepting or declining sequential work contracts at varying cost \- is the nearest existing analog to budget\-constrained decision\-making\. But this evaluation design \- sequential, independent accept/reject decisions \- by construction cannot expose the combinatorial problem of jointly selecting, allocating, and ordering under a shared budget, where attempting one problem reduces what remains for all others\.

AbstentionBench\(Kirichenkoet al\.,[2025](https://arxiv.org/html/2605.13414#bib.bib9)\)scales the monitoring question to a benchmark setting, testing whether models abstain from unanswerable questions across 20 datasets and finding that reasoning fine\-tuning degrades abstention by∼\{\\sim\}24%\. Their unanswerable variants of existing benchmarks inform our unsolvable\-injection experiment\. But abstention there is a property of the*question*\- some are answerable, others are not\. In TRIAGE, the relevant property is the*model–task–budget triple*: a problem may be answerable in principle but not by this model under this budget\.

Across this literature, a consistent picture emerges: LLMs possess metacognitive knowledge and some monitoring ability, but both are contested in degree and fragile across domains\. More critically, all existing evaluations operate one task at a time\. The portfolio\-level version of the problem is harder \- accurate monitoring of any one task is necessary but not sufficient, because the model must also compare feasibility and cost across tasks while respecting a shared constraint\. Whether current models can make this transition from single\-task monitoring to portfolio\-level resource allocation is what TRIAGE is designed to test\.

### 2\.2Compute Allocation for LLM Reasoning

The test\-time compute scaling paradigm\(Snellet al\.,[2025](https://arxiv.org/html/2605.13414#bib.bib1)\)has made compute allocation a practical problem\. Reasoning models overthink trivial problems\(Chenet al\.,[2024](https://arxiv.org/html/2605.13414#bib.bib3)\), underthink hard ones\(Wanget al\.,[2025](https://arxiv.org/html/2605.13414#bib.bib4)\), and in agentic settings exhibit analysis paralysis that correlates with decreased performance\(Cuadronet al\.,[2025](https://arxiv.org/html/2605.13414#bib.bib5)\)\. Existing methods address this waste, but split on who allocates and at what granularity, and neither side covers the full problem\.

On the self\-assessed side, TALE\(Hanet al\.,[2025](https://arxiv.org/html/2605.13414#bib.bib40)\)has the LLM estimate its own token cost before solving a problem, and SelfBudgeter\(Liet al\.,[2025](https://arxiv.org/html/2605.13414#bib.bib12)\)trains the model to emit a budget before its chain of thought\. Both are prospective and self\-assessed, but per\-task: the model decides how long to spend, never whether to attempt at all, and there is no shared budget linking decisions across tasks\. On the systems side, external predictors schedule LLM inference for throughput \- via length\-bucket classification\(Jinet al\.,[2023](https://arxiv.org/html/2605.13414#bib.bib13)\), listwise ranking\(Fuet al\.,[2024](https://arxiv.org/html/2605.13414#bib.bib14)\), or interval prediction under uncertainty\(Chenet al\.,[2025](https://arxiv.org/html/2605.13414#bib.bib23)\)\. These operate at the portfolio level, but the predictor is external to the LLM, the objective is latency, and the model’s own judgment of what it can solve plays no role\.

Existing benchmarks test related but distinct capabilities\. PlanBench\(Valmeekamet al\.,[2023](https://arxiv.org/html/2605.13414#bib.bib24)\)evaluates whether LLMs can produce valid plans in classical planning domains, with automated ground\-truth validation via external solvers \- but tests plan*execution*, not prospective self\-assessment of which plans will succeed\.τ\\tau\-bench\(Yaoet al\.,[2024](https://arxiv.org/html/2605.13414#bib.bib25)\)introduces tool\-call interactions as a measurable resource axis for agent evaluation, and GAIA\(Mialonet al\.,[2024](https://arxiv.org/html/2605.13414#bib.bib26)\)provides a heterogeneous, difficulty\-leveled agent benchmark with validated ground truth\. Both motivate the agentic extension of TRIAGE but neither tests whether the agent can plan its own workload efficiently under finite resources\.

These two literature meet at a measurement gap\. Single\-task confidence benchmarks isolate monitoring; external schedulers isolate allocation; neither evaluates the joint capability that deployment actually requires\. TRIAGE is designed to close this gap as an evaluation\. The model produces a plan \- selecting tasks, estimating costs, and ordering execution \- which we score by simulating its consequences against pre\-computed ground truth, yielding a single, falsifiable measure of the joint product of metacognitive monitoring accuracy and planning quality under a binding budget\. We design a portfolio\-level test in which accurate per\-task self\-assessment is necessary but not sufficient, because the model must also compare feasibility and cost across tasks while respecting a shared constraint\.

## 3TRIAGE Framework

### 3\.1Prospective Metacognitive Control

Effective problem\-solving under constraint requires more than the capacity to execute individual tasks; it requires metacognitive control over how that capacity is deployed, and for an agent operating under a finite execution budget, that control is exercised as prospective planning: a commitment, made before any problem is attempted, to which problems will be pursued, in what order, and how much of the budget each will receive\. Cognitive psychology distinguishes an*object level*, where a task is executed, from a*meta level*, which monitors the object level and exerts control over it\(Nelson and Narens,[1990](https://arxiv.org/html/2605.13414#bib.bib7); Koriat and Goldsmith,[1996](https://arxiv.org/html/2605.13414#bib.bib28)\)\. Two control functions of the meta level, what to attempt and how long to persist on what is attempted, are the levers behind human study\-time allocation\(Son and Metcalfe,[2000](https://arxiv.org/html/2605.13414#bib.bib16); Metcalfe and Kornell,[2005](https://arxiv.org/html/2605.13414#bib.bib29)\)and test\-taking strategy\(Ackerman and Goldsmith,[2011](https://arxiv.org/html/2605.13414#bib.bib30)\)\. We benchmark these levers in language models\.

Metacognitive judgments partition by their position in the execution timeline\(Dunlosky and Metcalfe,[2009](https://arxiv.org/html/2605.13414#bib.bib31)\): prospective, formed before execution; concurrent, formed during; and retrospective, formed after\. We focus on the prospective class, Ease\-of\-Learning judgments in the laboratory\(Underwood,[1966](https://arxiv.org/html/2605.13414#bib.bib32); Leonesio and Nelson,[1990](https://arxiv.org/html/2605.13414#bib.bib33)\), initial test plans in applied settings, for three independent reasons\.

Methodological isolation\.Once feedback is allowed during execution, observed performance reflects an unidentifiable mixture of prospective judgment quality and adaptive replanning; the two cannot be separated post\-hoc\. Studying prospective judgments without intervening feedback is the established technique for isolating the construct in human work\(Mazzoni and Cornoldi,[1993](https://arxiv.org/html/2605.13414#bib.bib34); Son,[2004](https://arxiv.org/html/2605.13414#bib.bib35)\)\.

Deployment realism\.A growing class of LLM\-driven systems commits to a task\-level plan over a heterogeneous set of subtasks before any one is attempted\. Deep research agents decompose a query into sub\-questions and dispatch them across parallel sub\-agents, each running with its own context window and token budget that the orchestrator cannot revise once dispatched\(Linet al\.,[2026](https://arxiv.org/html/2605.13414#bib.bib42)\)\. Coding agents process queues of bug reports or pull\-request issues under per\-issue token budgets, where wasted attempts on infeasible items measurably degrade overall throughput\(Jimenezet al\.,[2024](https://arxiv.org/html/2605.13414#bib.bib43)\)\. In both cases the quality of the upfront plan, what to attempt, in what order, with what allocation, determines whether the run completes usefully or exhausts its budget on the wrong items\.

Construct priority\.A model that revises its plan mid\-execution is only useful if its initial plan is any good\(Thiede and Dunlosky,[1999](https://arxiv.org/html/2605.13414#bib.bib36)\)\. To measure what mid\-execution adjustments add, we first need a clean measurement of the up\-front plan in isolation, and that requires evaluating planning without any feedback during execution\.

#### Tokens as the control variable\.

Classical study\-allocation work uses time as the meta\-level control variable\. In autoregressive language models, output tokens are the natural unit of expended effort: generation time, compute, and monetary cost are all approximately proportional to token count, and the central paradigm of test\-time compute scaling treats per\-problem token allocation as the system’s primary lever for adapting to problem difficulty\(Snellet al\.,[2025](https://arxiv.org/html/2605.13414#bib.bib1); Hanet al\.,[2025](https://arxiv.org/html/2605.13414#bib.bib40)\)\. Tokens are therefore the operational analog of time in study\-allocation theory, with the additional property of being exactly measurable\.

#### Distributional self\-knowledge and binding plans\.

Common agent pipelines generally decouple planning from execution into separate inference calls over identical weights\(Xuet al\.,[2023](https://arxiv.org/html/2605.13414#bib.bib41)\)\. The planner does not introspect on an ongoing computation; it predicts the behavior of a same\-weights solver under standard execution conditions\. We treat this as the deployment\-relevant form of self\-modeling\. The plan is moreover*operationally binding*: at execution time, per\-problem token allocations from the plan can be enforced as binding constraints on the solver, converting the plan from a forecast into a self\-commitment and testing whether the model can satisfy a budget it set for itself; we study this enforced setting alongside the unconstrained one as a secondary regime \(§[3\.2](https://arxiv.org/html/2605.13414#S3.SS2)\)\.

#### Four primitives\.

Prospective metacognitive control under a finite execution budget reduces to four jointly exercised judgments: \(i\)*feasibility*, whether the model can solve the problem? \(ii\)*cost*, how many tokens will it take? \(iii\)*selection*, which subset to attempt? and \(iv\)*sequencing*, in what order, given that budget exhaustion truncates the tail? Existing LLM benchmarks isolate slices of this set: per\-item confidence calibration covers \(i\)\(Barkanet al\.,[2025](https://arxiv.org/html/2605.13414#bib.bib10)\); abstention covers a hard form of \(i\) without a budget\(Kirichenkoet al\.,[2025](https://arxiv.org/html/2605.13414#bib.bib9)\); per\-task budget control covers \(ii\) for a single item\(Liet al\.,[2025](https://arxiv.org/html/2605.13414#bib.bib12)\)\. TRIAGE measures the joint exercise of all four primitives under a finite budget\.

### 3\.2Problem Formulation

We formalize TRIAGE for a single*task pool*, a finite set of problems sharing a domain and a budget, the unit of work the agent commits to in a single planning step before any execution feedback arrives\.

#### Setup\.

A task poolP=\{p1,…,pn\}P=\\\{p\_\{1\},\\ldots,p\_\{n\}\\\}contains finitely many problems\. For each problempip\_\{i\}and a fixed solver modelMM:vi∈ℝ\>0v\_\{i\}\\in\\mathbb\{R\}\_\{\>0\}is the point value, observable toMM;yi∈\{0,1\}y\_\{i\}\\in\\\{0,1\\\}indicates whetherMMsolvespip\_\{i\}under standard execution, unobservable toMMat planning time; andci∈ℝ\>0c\_\{i\}\\in\\mathbb\{R\}\_\{\>0\}is the output\-token costMMincurs when attemptingpip\_\{i\}, also unobservable at planning time\. Bothyiy\_\{i\}andcic\_\{i\}are properties of the \(problem, solver\) pair, determined empirically by runningMMonpip\_\{i\}once at temperature0\.

#### Budget\.

The budget isB​\(M,α\)=⌊α⋅∑i=1nci⌋B\(M,\\alpha\)=\\lfloor\\alpha\\cdot\\sum\_\{i=1\}^\{n\}c\_\{i\}\\rfloorwithα∈\(0,1\]\\alpha\\in\(0,1\]; the floor reflects that token counts are integer\-valued\. The factorα<1\\alpha<1forces selection by setting the budget below the solver’s full baseline cost;α=1\\alpha=1supplies the full baseline budget, in which selection pressure is reduced but the model must still identify which items it can solve, since omissions result in forfeiting value\.

#### Action\.

The model produces an ordered planπ=\(i1,…,ik\)\\pi=\(i\_\{1\},\\ldots,i\_\{k\}\)with0≤k≤n0\\leq k\\leq n, indexing a selected subsetS=\{i1,…,ik\}⊆\{1,…,n\}S=\\\{i\_\{1\},\\ldots,i\_\{k\}\\\}\\subseteq\\\{1,\\ldots,n\\\}, together with per\-problem allocationsaij∈ℝ≥0a\_\{i\_\{j\}\}\\in\\mathbb\{R\}\_\{\\geq 0\}satisfying∑jaij≤B\\sum\_\{j\}a\_\{i\_\{j\}\}\\leq B\. The plan thus jointly encodes selection, sequencing, and forecast\.

#### Execution\.

Plans are executed in plan order under one of two regimes, which differ in whether the per\-problem allocationsaia\_\{i\}are binding\.

Unconstrained regime \(U\)\.The solver runs each problem to its natural completion at the costcic\_\{i\}; only the global budgetBBbinds\. Execution proceeds throughπ\\piuntil reaching an item whose true cost exceeds the remaining budget, at which point execution stops\. Problemiji\_\{j\}contributesvij​yijv\_\{i\_\{j\}\}y\_\{i\_\{j\}\}to the achieved value if executed, and zero otherwise\.

Constrained regime \(E\)\.Each problemiji\_\{j\}is executed with a hard token limitaija\_\{i\_\{j\}\}at the solver: if the solver completes withinaija\_\{i\_\{j\}\}, the outcome is the result under that limit; otherwise the attempt scores zero\. The full allocationaija\_\{i\_\{j\}\}is deducted from the remaining budget regardless of outcome, and execution proceeds throughπ\\piuntil reaching an item whose allocation exceeds the remaining budget, at which point execution stops\.

The two regimes isolate distinct components of the meta\-level control problem\(Nelson and Narens,[1990](https://arxiv.org/html/2605.13414#bib.bib7)\)\. Regime U probes*prospective monitoring*: the planner’s predictions of\(yi,ci\)\(y\_\{i\},c\_\{i\}\)surface only through which items it selects and in what order, since allocations are advisory; truncation discipline comes entirely from ordering quality\. Regime E probes*prospective control*in the strict sense: per\-problem allocations are binding, so the model is held to commitments it makes to itself\. We writeVM​\(π\)V\_\{M\}\(\\pi\)for the achieved value at termination under whichever regime is in force, and report results in both\.

#### Measuring triage skill\.

We need a single scalar that captures how well the planner’s committed planπ\\piacts on its \(unobserved\) knowledge of\(yi,ci\)\(y\_\{i\},c\_\{i\}\)under the budgetBB\. A raw value comparisonVM​\(π\)V\_\{M\}\(\\pi\)is uninformative on its own \- its magnitude depends on pool difficulty and onBB, which vary across cells\. The standard device in forecast verification\(Murphy,[1973](https://arxiv.org/html/2605.13414#bib.bib37)\)and in selective\-prediction evaluation\(Geifman and El\-Yaniv,[2017](https://arxiv.org/html/2605.13414#bib.bib38)\)is to*normalize*the system’s achieved value between a reference that uses no privileged knowledge and an oracle that uses all of it\. We adopt that construction here\.

#### Oracle and random references\.

The*oracle*is a planner with full knowledge of\(yi,ci\)\(y\_\{i\},c\_\{i\}\)for everyii\. Its achievable value is

Voracle=maxx∈\{0,1\}n​∑i=1nvi​yi​xis\.t\.∑i=1nci​xi≤B,V\_\{\\text\{oracle\}\}\\;=\\;\\max\_\{x\\in\\\{0,1\\\}^\{n\}\}\\;\\sum\_\{i=1\}^\{n\}v\_\{i\}\\,y\_\{i\}\\,x\_\{i\}\\quad\\text\{s\.t\.\}\\quad\\sum\_\{i=1\}^\{n\}c\_\{i\}\\,x\_\{i\}\\leq B,\(1\)a 0–1 knapsack \- each problem is either included \(xi=1x\_\{i\}=1\) or omitted \(xi=0x\_\{i\}=0\), with no fractional attempts\(Kellereret al\.,[2004](https://arxiv.org/html/2605.13414#bib.bib39)\)\. The*random*referenceVrandomV\_\{\\text\{random\}\}is the expected value of a feasible plan obtained without any self\-knowledge, drawn uniformly without replacement subject to remaining\-budget feasibility; we report the mean over10310^\{3\}such samples\.

#### Triage efficiency ratio\.

The triage efficiency ratio normalizes the model’s achieved value between these two references:

ηM=VM​\(π\)−VrandomVoracle−Vrandom\.\\eta\_\{M\}\\;=\\;\\frac\{V\_\{M\}\(\\pi\)\-V\_\{\\text\{random\}\}\}\{V\_\{\\text\{oracle\}\}\-V\_\{\\text\{random\}\}\}\.\(2\)On the standard interpretation,ηM=0\\eta\_\{M\}=0corresponds to no use of self\-knowledge andηM=1\\eta\_\{M\}=1to the value attainable with perfect self\-knowledge; intermediate values report fractional skill on the random\-to\-oracle scale\. Negative values indicate plans worse than uninformed random selection, an anti\-skill signal under tight budgets, indicating the planner’s choices systematically picked items that didn’t fit the budget or weren’t solvable\. Pools withVoracle=0V\_\{\\text\{oracle\}\}=0\(no problem in the pool is solvable by the model\) are out of scope for triage measurement and excluded; we discuss the boundary behaviour ofηM\\eta\_\{M\}in Appendix[D](https://arxiv.org/html/2605.13414#A4)\.

#### Uniform values: preventing difficulty leakage\.

The main protocol fixesvi=1v\_\{i\}=1\. Anyviv\_\{i\}correlated with empirical difficulty would let the planner read difficulty off the prompt rather than estimate it from problem content, and language models are known to exploit such surface cues when present\(Koriat,[1997](https://arxiv.org/html/2605.13414#bib.bib19); Tanget al\.,[2023](https://arxiv.org/html/2605.13414#bib.bib44)\)\. Under uniformvv, the oracle objective simplifies:VoracleV\_\{\\text\{oracle\}\}equals the maximum number of solvable items \(yi=1y\_\{i\}=1\) whose costs fit withinBB\. LetS⋆=\{i:yi=1\}S^\{\\star\}=\\\{i:y\_\{i\}=1\\\}and leti\(1\),i\(2\),…i\_\{\(1\)\},i\_\{\(2\)\},\\ldotsindexS⋆S^\{\\star\}in ascending order ofcic\_\{i\}; then

Voracle=max⁡\{m\|∑j=1mci\(j\)≤B\}\.V\_\{\\text\{oracle\}\}\\;=\\;\\max\\Bigl\\\{\\,m\\,\\Bigm\|\\,\\sum\_\{j=1\}^\{m\}c\_\{i\_\{\(j\)\}\}\\leq B\\,\\Bigr\\\}\.\(3\)
The model receives only problem text and must form its plan without observing\(yi,ci\)\(y\_\{i\},c\_\{i\}\)\. Differences inη\\etabetween models therefore reflect differences in their implicit predictions of\(yi,ci\)\(y\_\{i\},c\_\{i\}\), not differences in optimization: the oracle, the random reference, and the execution rule are identical across models\. Thus we measure the model’s prediction of its own future behavior, with the oracle playing the role of an ideal forecaster, as in selective\-prediction evaluation more broadly\(Geifman and El\-Yaniv,[2017](https://arxiv.org/html/2605.13414#bib.bib38)\)\.

## 4Experimental Setup

#### Models\.

We evaluate 20 model architectures spanning frontier and open\-source families \(Qwen 2\.5/3/3\.5, GPT\-OSS, Claude, DeepSeek, Gemini, GLM, GPT\-5, Kimi\), with reasoning enabled and disabled where supported\. Full model identifiers, providers, parameter counts, and reasoning configurations are listed in Table[4](https://arxiv.org/html/2605.13414#A5.T4)\(Appendix[E](https://arxiv.org/html/2605.13414#A5)\)\.

#### Datasets and task pools\.

We construct task pools from four domains: competition mathematics \(AIME 2024–2025\), graduate\-level science \(GPQA Diamond\(Reinet al\.,[2024](https://arxiv.org/html/2605.13414#bib.bib45)\)\), code generation \(LiveCodeBench\(Jainet al\.,[2024](https://arxiv.org/html/2605.13414#bib.bib17)\), filtered to contests from 2025 onward to reduce contamination risk from problems that may appear in pre\-training data\), and multidisciplinary expert knowledge \(Humanity’s Last Exam\(Phanet al\.,[2025](https://arxiv.org/html/2605.13414#bib.bib18)\), excluding image\-bearing items and topic categories overlapping with the other three datasets\)\. Each dataset is partitioned into independent task pools ofnchunk=30n\_\{\\text\{chunk\}\}=30problems to fit within planner context limits and avoid the long\-context degradation reported in prior work\(Liuet al\.,[2024](https://arxiv.org/html/2605.13414#bib.bib60)\)\. Each pool has its own budget, oracle, and random reference, andη\\etais averaged across pools within each\(dataset,α\)\(\\text\{dataset\},\\alpha\)cell\.

To probe sensitivity to infeasible items, we additionally inject unsolvable problems from AbstentionBench\(Kirichenkoet al\.,[2025](https://arxiv.org/html/2605.13414#bib.bib9)\)\(GPQA\-Abstain and MMLU\-Math\-Abstain\) into pools at varying ratios\.

#### Phase 0: baseline measurement\.

We first run each model on every problem once, in an isolated session at temperature 0, to record its baseline output\-token costcic\_\{i\}and correctnessyiy\_\{i\}\. These two quantities serve two purposes:cic\_\{i\}calibrates the per\-pool budgetBlocal=⌊α​∑ici⌋B\_\{\\text\{local\}\}=\\lfloor\\alpha\\sum\_\{i\}c\_\{i\}\\rfloorgiven to the planner in Phase 1, and\(ci,yi\)\(c\_\{i\},y\_\{i\}\)together provide the ground truth against which the oracle andη\\etaare computed\.

#### Phase 1: planning\.

The planner receives problem texts, point values \(vi=1v\_\{i\}=1\), the per\-pool budgetBlocal=⌊α​∑i∈poolci⌋B\_\{\\text\{local\}\}=\\lfloor\\alpha\\sum\_\{i\\in\\text\{pool\}\}c\_\{i\}\\rfloor, and instructions to return an ordered plan with per\-problem token allocations summing to at mostBlocalB\_\{\\text\{local\}\}\. The full prompt template is given in Appendix[B](https://arxiv.org/html/2605.13414#A2)\. A prompt sensitivity analysis is available in[C\.2](https://arxiv.org/html/2605.13414#A3.SS2)and shows robustness under prompt variations\.

#### Phase 2: execution\.

Plans are executed under both regime U and regime E \(§[3\.2](https://arxiv.org/html/2605.13414#S3.SS2.SSS0.Px4)\), under same configuration so that observed differences inη\\etareflect planning quality rather than solver stochasticity\.

#### Budget conditions\.

We sweepα∈\{0\.25,0\.5,0\.75,1\.0\}\\alpha\\in\\\{0\.25,0\.5,0\.75,1\.0\\\}to vary selection pressure from severe to slack\. Main results reportα=0\.5\\alpha=0\.5, under 50% budget constraint\.

#### Metric\.

We report triage efficiencyηM\\eta\_\{M\}\(§[3\.2](https://arxiv.org/html/2605.13414#S3.SS2)\) as the primary metric and normalized regretR~M=\(Voracle−VM\)/Voracle∈\[0,1\]\\tilde\{R\}\_\{M\}=\(V\_\{\\text\{oracle\}\}\-V\_\{M\}\)/V\_\{\\text\{oracle\}\}\\in\[0,1\]as a bounded alternative for robustness \(Appendix[D\.5](https://arxiv.org/html/2605.13414#A4.SS5)\); lowerR~M\\tilde\{R\}\_\{M\}is better\. The random referenceVrandomV\_\{\\text\{random\}\}is estimated by drawing10310^\{3\}uniform shuffles of the gradeable items per pool and executing each under regime\-U rules \- the same execution rule used to score model plans, ensuringηM\\eta\_\{M\}measures the value added by self\-knowledge over uninformed random selection\.

For the unsolvable\-injection probe, we additionally report*waste rate*W=∑i∈S∩Uai/∑i∈SaiW=\\sum\_\{i\\in S\\cap U\}a\_\{i\}\\,/\\,\\sum\_\{i\\in S\}a\_\{i\}, the fraction of allocated tokens spent on unsolvable items, and*detection rate*D=\|U∖S\|/\|U\|D=\|U\\setminus S\|\\,/\\,\|U\|, the fraction of unsolvable items correctly excluded from the plan, whereS=\{i:ai\>0\}S=\\\{i:a\_\{i\}\>0\\\}is the planned set andUUthe injected unsolvable items in the pool\.

## 5Results

### 5\.1Triage at moderate budget

![Refer to caption](https://arxiv.org/html/2605.13414v1/x1.png)Figure 1:Triage efficiency across models and benchmarks at moderate budget \(α=0\.5\\alpha=0\.5\)\.Solid bars showηU\\eta\_\{U\}\(unconstrained regime, advisory allocations\), hatched bars showηE\\eta\_\{E\}\(constrained regime, binding allocations\), and red dashed lines mark base accuracy\.η=1\\eta=1is oracle triage;η=0\\eta=0is random;η<0\\eta<0is worse than random\. Bar color distinguishesstandard inferencefromextended reasoning\.![Refer to caption](https://arxiv.org/html/2605.13414v1/x2.png)Figure 2:Normalized triage regret across models and benchmarks at moderate budget \(α=0\.5\\alpha=0\.5\)\.Solid bars showR~U=\(Voracle−VU\)/Voracle\\tilde\{R\}\_\{U\}=\(V\_\{\\mathrm\{oracle\}\}\-V\_\{U\}\)/V\_\{\\mathrm\{oracle\}\}\(unconstrained regime\), hatched bars showR~E\\tilde\{R\}\_\{E\}\(constrained regime\), and red dashed lines mark base accuracy\.R~=0\\tilde\{R\}=0is oracle \(no value lost\);R~=1\\tilde\{R\}=1is full regret \(no value captured\)\. Lower is better\.Figures[1](https://arxiv.org/html/2605.13414#S5.F1)and[2](https://arxiv.org/html/2605.13414#S5.F2)report triage atα=0\.5\\alpha=0\.5, where the budget covers half the items’ baseline costs and selection genuinely matters\. We report both metrics:η\\etameasures how much of the gap between an oracle planner and a random planner the model recovers, whileR~\\tilde\{R\}measures the absolute fraction of oracle value the model fails to capture and remains well\-defined wheneverVoracle\>0V\_\{\\mathrm\{oracle\}\}\>0\. The main findings are described below\.

#### Advisory triage does not transfer across benchmarks\.

ηU\\eta\_\{U\}in Figure[1](https://arxiv.org/html/2605.13414#S5.F1)varies sharply across \(model, dataset\) pairs and no configuration ranks at the top on all four benchmarks\. Configurations that achieve high advisory efficiency on one or two datasets frequently drop near or below random on the others, showing that the ability to identify worthwhile items in one domain does not carry over reliably\. Strong advisory performance is also decoupled from raw solve rate: several configurations attain highηU\\eta\_\{U\}and low advisory regretR~U\\tilde\{R\}\_\{U\}on datasets where their base accuracy is low, indicating accurate self\-assessment over a small set of feasible items\.

#### Binding the budget breaks triage\.

The gap betweenηU\\eta\_\{U\}\(solid\) andηE\\eta\_\{E\}\(hatched\) in Figure[1](https://arxiv.org/html/2605.13414#S5.F1)is the most consistent pattern across models\. Atα=0\.5\\alpha=0\.5,ηE\\eta\_\{E\}is negative for the large majority of configurations, and a substantial fraction reachR~E\\tilde\{R\}\_\{E\}near the full\-regret bound in Figure[2](https://arxiv.org/html/2605.13414#S5.F2), meaning the model captures essentially no value once its own per\-item allocations are enforced\. The collapse is sharpest on the reasoning\-heavy benchmarks \(AIME, GPQA, LCB\), where binding the budget cuts off solver outputs before they can produce an answer\. HLE is the only benchmark where any extended\-reasoning configuration attains clearly positiveηE\\eta\_\{E\}\. A controlled re\-solve experiment confirms this behaviorally: models rarely honor the budgets they themselves declare even when explicitly instructed to \(Appendix[C\.1](https://arxiv.org/html/2605.13414#A3.SS1)\)\.

#### Plans and execution diverge under enforcement\.

Across all 30 configurations evaluated, only one \(Gemini 2\.5 Flash in standard inference\) achieves positiveηE\\eta\_\{E\}on every benchmark, and only two \(Gemini 2\.5 Flash standard and Qwen 3\.5 27B standard\) achieveR~E<0\.5\\tilde\{R\}\_\{E\}<0\.5on more than one benchmark\. Both belong to the standard\-inference variant of their respective model\. Within paired\-mode comparisons, the standard variant achieves a lower enforced\-regime regret than its extended\-reasoning counterpart on the majority of datasets, indicating that extended reasoning tends to produce longer outputs that exceed the planner’s own allocations and so collapse under enforcement\.

#### Extended reasoning lifts accuracy without consistently lifting triage\.

Within the paired\-mode models, base accuracy \(red dashed line\) increases with extended reasoning on most \(model, dataset\) pairs, and on some datasets the lift is large\. Advisory regretR~U\\tilde\{R\}\_\{U\}, however, follows no such pattern: extended reasoning leaves it essentially unchanged or worsens it on roughly half of paired configurations\. The pattern that does emerge consistently is the failure mode of the previous paragraph: extended reasoning enlarges the gap between the advisory and enforced regimes, because longer traces are less likely to fit within the model’s own allocation\. Object\-level capability and metacognitive control thus dissociate empirically: a longer trace can solve more problems without making the planner any better at deciding which ones to attempt, and can substantially hurt the planner’s ability to execute within its own budget\.

#### Triage quality does not scale with parameter count\.

Within Qwen 2\.5, the largest 32B model is the worst on GPQA in the entire family and the family shows no upward trend with size on either GPQA or LCB\. Within Qwen 3\.5, the dense 27B beats the larger mixture\-of\-experts variant in standard mode on every benchmark\. Among GPT\-OSS, the 120B underperforms the 20B by a wide margin on LCB\. No family in our evaluation exhibits a clean upward trend across all four benchmarks\. Triage quality, as measured here, does not follow the standard scaling trends observed for raw accuracy, and may require capabilities that grow only weakly, or not at all, with parameter count\.

#### Behavior across budget pressure\.

Full per\-\(model,α\\alpha, dataset\) breakdowns appear in the appendix: accuracy in[Figure˜4](https://arxiv.org/html/2605.13414#A3.F4), advisory\-regime efficiencyηU\\eta\_\{U\}in[Figure˜5](https://arxiv.org/html/2605.13414#A3.F5), and enforced\-regime efficiencyηE\\eta\_\{E\}in[Figure˜6](https://arxiv.org/html/2605.13414#A3.F6); per\-α\\alphaversions of Figures[1](https://arxiv.org/html/2605.13414#S5.F1)and[2](https://arxiv.org/html/2605.13414#S5.F2)are in[Figure˜5](https://arxiv.org/html/2605.13414#A3.F5)\. The findings atα=0\.5\\alpha=0\.5are broadly stable across the other budget levels we evaluate: the same configurations retain advisory\-regime leadership on multiple benchmarks, and theηU\\eta\_\{U\}\-ηE\\eta\_\{E\}gap persists at every budget, with enforced efficiency remaining negative for most configurations on the reasoning\-heavy benchmarks\. Atα=1\.0\\alpha=1\.0the efficiency metric saturates because the oracle\-random gap approaches zero in many configurations, whereas the regret formulation remains numerically stable there\. Across the models and benchmarks we evaluate, no single configuration is simultaneously a strong monitor and a strong controller on more than one benchmark\. Prospective metacognitive control, in the joint sense of §[3\.1](https://arxiv.org/html/2605.13414#S3.SS1), remains an open capability gap\.

### 5\.2Recognizing Unsolvable Problems

![Refer to caption](https://arxiv.org/html/2605.13414v1/x3.png)Figure 3:Trajectories through\(D,W\)\(D,W\)space as the unsolvable\-injection ratiorrincreases from0\.250\.25to1\.001\.00\(marker size\)\.The star marks the ideal corner: high detection rate, low waste rate\.Effective planning requires recognizing tasks that cannot be solved\. Tokens spent on an unsolvable task produce no reward and reduce the budget available for solvable ones, so the ability to identify and exclude infeasible items is itself a metacognitive skill worth measuring\. We probe sensitivity to infeasible items of selected models by injecting unsolvable problems from AbstentionBench\(Kirichenkoet al\.,[2025](https://arxiv.org/html/2605.13414#bib.bib9)\)\(GPQA\-Abstain, MMLU\-Math\-Abstain\) into the GPQA and MMLU\-Math pools at substitution ratiosr∈\{0,0\.25,0\.50,0\.75,1\.00\}r\\in\\\{0,0\.25,0\.50,0\.75,1\.00\\\}, holding pool size constant\. We report detection rateDD\(fraction of unsolvables correctly left out of the plan\) and waste rateWW\(fraction of the plan’s tokens spent on unsolvables\); see §[4](https://arxiv.org/html/2605.13414#S4.SS0.SSS0.Px7)\. Figure[3](https://arxiv.org/html/2605.13414#S5.F3)traces each ablation model through\(D,W\)\(D,W\)space; the main findings are described below\.

#### Recognition of unsolvables splits the cohort\.

Two models recognize unsolvables consistently across injection ratios: Gemini 2\.5 Flash, whose detection stays in\[0\.60,0\.79\]\[0\.60,0\.79\]across all ratios on both datasets, and Qwen 2\.5 32B, whose detection stays at or above0\.330\.33across all ratios\. The remaining two models – Qwen 3 32B and GPT\-OSS 20B, both reasoning\-trained – detect substantially fewer unsolvables\. Their detection lies in\[0\.00,0\.30\]\[0\.00,0\.30\]on GPQA\-Abstain across all ratios\. On MMLU\-Math\-Abstain, GPT\-OSS 20B’s detection remains below0\.130\.13at every ratio \(max0\.1270\.127atr=0\.50r=0\.50\), while Qwen 3 32B’s detection declines monotonically from0\.540\.54atr=0\.25r=0\.25to0\.070\.07atr=1\.0r=1\.0\. The Qwen pair provides the cleanest within\-family comparison: under standard inference, the model recognizes the majority of unsolvables at low injection \(D=0\.67D=0\.67GPQA,0\.880\.88MMLU\-Math atr=0\.25r=0\.25\); under extended reasoning, the same architecture detects substantially less \(D=0\.08D=0\.08GPQA,0\.540\.54MMLU\-Math at the samer=0\.25r=0\.25\)\. The within\-Qwen pattern reproduces under portfolio planning whatKirichenkoet al\.\([2025](https://arxiv.org/html/2605.13414#bib.bib9)\)report for isolated abstention – that reasoning training can reduce sensitivity to infeasibility\. We note that this is a within\-family effect: Gemini 2\.5 Flash, also reasoning\-augmented, retains detection across the range, suggesting the relationship between reasoning training and abstention is not uniform across models\.

#### The dependence of detection on injection ratio\.

We characterize each model by how its detection rate varies with the injection ratio\. Gemini’s detection is approximately invariant to injection: it spans0\.050\.05on MMLU\-Math\-Abstain and0\.190\.19on GPQA\-Abstain across the four non\-trivial ratios\. Qwen 2\.5 32B’s detection declines monotonically with injection on MMLU\-Math\-Abstain \(0\.88→0\.67→0\.58→0\.400\.88\\to 0\.67\\to 0\.58\\to 0\.40\) but follows a non\-monotonic course on GPQA\-Abstain \(sharp drop afterr=0\.25r=0\.25, then approximately flat\)\. The two reasoning\-trained models hold detection rate below0\.500\.50across every cell on GPQA\-Abstain and at everyr≥0\.75r\\geq 0\.75on MMLU\-Math\-Abstain, indicating allocations that largely fail to distinguish unsolvable items from solvable ones\.

#### Cross\-domain consistency of detection\.

Three of four models detect unsolvables more reliably on the math\-derived dataset than on the science\-derived one, with mean differences of\+0\.19\+0\.19\(Qwen 2\.5 32B\) and\+0\.32\+0\.32\(Qwen 3 32B\)\. Gemini’s gap is small \(\+0\.04\+0\.04\)\. GPT\-OSS 20B is the exception: its detection is uniformly low on both datasets, with a mean difference of−0\.03\-0\.03, leaving no informative comparison\.

#### Full injection isolates detection from solving ability\.

At full injection every item is infeasible and the optimal plan is empty, so solving ability cannot contribute to detection\. The cohort split persists\. Gemini continues to exclude roughly three\-quarters of items from the plan \(D=0\.73D=0\.73GPQA,0\.760\.76MMLU\-Math atr=1\.0r=1\.0\); Qwen 2\.5 32B excludes a non\-trivial fraction \(D=0\.40D=0\.40on both\); the two reasoning\-trained models exclude almost none \(D<0\.08D<0\.08in all four cells\), continuing to allocate budget under conditions where any allocation is wasted by construction\. Full injection also partially removes between\-domain difficulty as a source of variance, leaving the model’s tendency to commit budget under certain failure as the main residual axis distinguishing the cohort\.

## 6Conclusion

Deploying language models as autonomous agents requires a capability that single\-task evaluation does not test: prospective metacognitive control over which problems to attempt, in what order, and with what allocation, committed before any execution feedback\. TRIAGE evaluates this capability by scoring committed plans against an oracle with full knowledge of model\-specific solvability and cost\.

Across 20 models and four domains, object\-level capability and metacognitive control dissociate\. Extended reasoning improves accuracy without improving triage efficiency\. Self\-assessment does not transfer reliably across domains\. Models select which problems to attempt more reliably than they commit token budgets they can honor under enforcement\. Reasoning training reduces, rather than improves, sensitivity to infeasible items at the portfolio level\. Because the failure is prospective, execution\-time monitoring cannot repair it\. Reliable upfront planning is therefore a prerequisite, not a refinement, for agents that allocate their own compute under finite resources\.

## Impact Statement

This paper presents work whose goal is to advance the field of Machine Learning\. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here\.

## References

- C\. Ackerman \(2026\)Evidence for limited metacognition in LLMs\.InThe Fourteenth International Conference on Learning Representations \(ICLR\),Note:arXiv:2509\.21545Cited by:[§1](https://arxiv.org/html/2605.13414#S1.p4.1),[§2\.1](https://arxiv.org/html/2605.13414#S2.SS1.SSS0.Px1.p3.1)\.
- R\. Ackerman and M\. Goldsmith \(2011\)Metacognitive regulation of text learning: on screen versus on paper\.Journal of Experimental Psychology: Applied17\(1\),pp\. 18–32\.External Links:[Document](https://dx.doi.org/10.1037/a0022086)Cited by:[§3\.1](https://arxiv.org/html/2605.13414#S3.SS1.p1.1)\.
- M\. A\. Alomrani, Y\. Zhang, D\. Li, Q\. Sun, S\. Pal, Z\. Zhang, Y\. Hu, R\. D\. Ajwani, A\. Valkanas, R\. Karimi, P\. Cheng, Y\. Wang, P\. Liao, H\. Huang, B\. Wang, J\. Hao, and M\. Coates \(2025\)Reasoning on a budget: a survey of adaptive and controllable test\-time compute in LLMs\.arXiv preprint arXiv:2507\.02076\.Cited by:[§1](https://arxiv.org/html/2605.13414#S1.p1.1)\.
- C\. O\. Barkan, S\. Black, and O\. Sourbut \(2025\)Do large language models know what they are capable of?\.arXiv preprint arXiv:2512\.24661\.Note:NeurIPS 2025 Workshop on Evaluating the Evolving LLM LifecycleCited by:[§1](https://arxiv.org/html/2605.13414#S1.p3.1),[§2\.1](https://arxiv.org/html/2605.13414#S2.SS1.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2605.13414#S3.SS1.SSS0.Px3.p1.1)\.
- X\. Chen, J\. Xu, T\. Liang, Z\. He, J\. Pang, D\. Yu, L\. Song, Q\. Liu, M\. Zhou, Z\. Zhang, R\. Wang, Z\. Tu, H\. Mi, and D\. Yu \(2024\)Do NOT think that much for 2\+3=? On the overthinking of o1\-like LLMs\.arXiv preprint arXiv:2412\.21187\.Cited by:[§1](https://arxiv.org/html/2605.13414#S1.p1.1),[§2\.2](https://arxiv.org/html/2605.13414#S2.SS2.p1.1)\.
- Z\. Chen, Y\. Ye, and Z\. Zhou \(2025\)Adaptively robust LLM inference optimization under prediction uncertainty\.arXiv preprint arXiv:2508\.14544\.Cited by:[§2\.2](https://arxiv.org/html/2605.13414#S2.SS2.p2.1)\.
- A\. Cuadron, D\. Li, W\. Ma, X\. Wang, Y\. Wang, S\. Zhuang, S\. Liu, L\. G\. Schroeder, T\. Xia, H\. Mao, N\. Thumiger, A\. Desai, I\. Stoica, A\. Klimovic, G\. Neubig, and J\. E\. Gonzalez \(2025\)The danger of overthinking: examining the reasoning\-action dilemma in agentic tasks\.arXiv preprint arXiv:2502\.08235\.Cited by:[§1](https://arxiv.org/html/2605.13414#S1.p1.1),[§2\.2](https://arxiv.org/html/2605.13414#S2.SS2.p1.1)\.
- A\. Didolkar, A\. Goyal, N\. R\. Ke, S\. Guo, M\. Valko, T\. Lillicrap, D\. Rezende, Y\. Bengio, M\. Mozer, and S\. Arora \(2024\)Metacognitive capabilities of llms: an exploration in mathematical problem solving\.Advances in Neural Information Processing Systems37,pp\. 19783–19812\.Cited by:[§2\.1](https://arxiv.org/html/2605.13414#S2.SS1.SSS0.Px1.p1.1)\.
- J\. Dunlosky and J\. Metcalfe \(2009\)Metacognition\.SAGE Publications,Thousand Oaks, CA\.External Links:ISBN 9781412939720Cited by:[§3\.1](https://arxiv.org/html/2605.13414#S3.SS1.p2.1)\.
- Y\. Fu, P\. Bailis, I\. Stoica, and H\. Zhang \(2024\)Efficient LLM scheduling by learning to rank\.InAdvances in Neural Information Processing Systems,Vol\.37\.Note:arXiv:2408\.15792Cited by:[§1](https://arxiv.org/html/2605.13414#S1.p3.1),[§2\.2](https://arxiv.org/html/2605.13414#S2.SS2.p2.1)\.
- Y\. Geifman and R\. El\-Yaniv \(2017\)Selective classification for deep neural networks\.InAdvances in Neural Information Processing Systems,Vol\.30\.External Links:1705\.08500Cited by:[§3\.2](https://arxiv.org/html/2605.13414#S3.SS2.SSS0.Px5.p1.5),[§3\.2](https://arxiv.org/html/2605.13414#S3.SS2.SSS0.Px8.p2.3)\.
- Griotet al\.\(2025\)Large language models lack essential metacognition for reliable medical reasoning\.Nature Communications16,pp\. 642\.External Links:[Document](https://dx.doi.org/10.1038/s41467-024-55628-6)Cited by:[§2\.1](https://arxiv.org/html/2605.13414#S2.SS1.SSS0.Px1.p2.1)\.
- T\. Han, Z\. Wang, C\. Fang, S\. Zhao, S\. Ma, and Z\. Chen \(2025\)Token\-budget\-aware LLM reasoning\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 24842–24855\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.1274)Cited by:[§1](https://arxiv.org/html/2605.13414#S1.p3.1),[§1](https://arxiv.org/html/2605.13414#S1.p4.1),[§2\.2](https://arxiv.org/html/2605.13414#S2.SS2.p2.1),[§3\.1](https://arxiv.org/html/2605.13414#S3.SS1.SSS0.Px1.p1.1)\.
- N\. Jain, K\. Han, A\. Gu, W\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. Stoica \(2024\)LiveCodeBench: holistic and contamination free evaluation of large language models for code\.arXiv preprint arXiv:2403\.07974\.Cited by:[§1](https://arxiv.org/html/2605.13414#S1.p5.1),[§4](https://arxiv.org/html/2605.13414#S4.SS0.SSS0.Px2.p1.3)\.
- L\. Ji\-An, H\. Xiong, R\. C\. Wilson, M\. G\. Mattar, and M\. K\. Benna \(2025\)Language models are capable of metacognitive monitoring and control of their internal activations\.arXiv preprint arXiv:2505\.13763\.Cited by:[§2\.1](https://arxiv.org/html/2605.13414#S2.SS1.SSS0.Px1.p1.1)\.
- C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. R\. Narasimhan \(2024\)SWE\-bench: can language models resolve real\-world GitHub issues?\.InThe Twelfth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=VTF8yNQM66)Cited by:[item Deployment realism\.](https://arxiv.org/html/2605.13414#S3.I1.ix2.p1.1)\.
- Y\. Jin, C\. Wu, D\. Brooks, and G\. Wei \(2023\)S3: increasing GPU utilization during generative inference for higher throughput\.InAdvances in Neural Information Processing Systems,Vol\.36\.Note:arXiv:2306\.06000Cited by:[§1](https://arxiv.org/html/2605.13414#S1.p3.1),[§2\.2](https://arxiv.org/html/2605.13414#S2.SS2.p2.1)\.
- S\. Kadavath, T\. Conerly, A\. Askell, T\. Henighan, D\. Drain, E\. Perez, N\. Schiefer, Z\. Hatfield\-Dodds, N\. DasSarma, E\. Tran\-Johnson, S\. Johnston, S\. El\-Showk, A\. Jones, N\. Elhage, T\. Hume, A\. Chen, Y\. Bai, S\. Bowman, S\. Fort, D\. Ganguli, D\. Hernandez, J\. Jacobson, J\. Kernion, S\. Kravec, L\. Lovitt, K\. Ndousse, C\. Olsson, S\. Ringer, D\. Amodei, T\. Brown, J\. Clark, N\. Joseph, B\. Mann, S\. McCandlish, C\. Olah, and J\. Kaplan \(2022\)Language models \(mostly\) know what they know\.arXiv preprint arXiv:2207\.05221\.Cited by:[§1](https://arxiv.org/html/2605.13414#S1.p3.1),[§1](https://arxiv.org/html/2605.13414#S1.p4.1),[§2\.1](https://arxiv.org/html/2605.13414#S2.SS1.SSS0.Px1.p1.1)\.
- S\. Kapoor, N\. Gruver, M\. Roberts, K\. Collins, A\. Pal, U\. Bhatt, A\. Weller, S\. Dooley, M\. Goldblum, and A\. G\. Wilson \(2024\)Large language models must be taught to know what they don’t know\.InAdvances in Neural Information Processing Systems,Vol\.37\.Cited by:[§2\.1](https://arxiv.org/html/2605.13414#S2.SS1.SSS0.Px1.p2.1)\.
- H\. Kellerer, U\. Pferschy, and D\. Pisinger \(2004\)Knapsack problems\.Springer,Berlin, Heidelberg\.External Links:ISBN 978\-3\-540\-40286\-2,[Document](https://dx.doi.org/10.1007/978-3-540-24777-7)Cited by:[§3\.2](https://arxiv.org/html/2605.13414#S3.SS2.SSS0.Px6.p1.6)\.
- P\. Kirichenko, M\. Ibrahim, K\. Chaudhuri, and S\. J\. Bell \(2025\)AbstentionBench: reasoning LLMs fail on unanswerable questions\.InAdvances in Neural Information Processing Systems: Datasets and Benchmarks Track,Vol\.38\.Note:arXiv:2506\.09038Cited by:[§1](https://arxiv.org/html/2605.13414#S1.p3.1),[§2\.1](https://arxiv.org/html/2605.13414#S2.SS1.SSS0.Px2.p2.1),[§3\.1](https://arxiv.org/html/2605.13414#S3.SS1.SSS0.Px3.p1.1),[§4](https://arxiv.org/html/2605.13414#S4.SS0.SSS0.Px2.p2.1),[§5\.2](https://arxiv.org/html/2605.13414#S5.SS2.SSS0.Px1.p1.16),[§5\.2](https://arxiv.org/html/2605.13414#S5.SS2.p1.4)\.
- A\. Koriat and M\. Goldsmith \(1996\)Monitoring and control processes in the strategic regulation of memory accuracy\.Psychological Review103\(3\),pp\. 490–517\.External Links:[Document](https://dx.doi.org/10.1037/0033-295X.103.3.490)Cited by:[§3\.1](https://arxiv.org/html/2605.13414#S3.SS1.p1.1)\.
- A\. Koriat \(1997\)Monitoring one’s own knowledge during study: a cue\-utilization approach to judgments of learning\.\.Journal of experimental psychology: General126\(4\),pp\. 349\.Cited by:[§1](https://arxiv.org/html/2605.13414#S1.p4.1),[§3\.2](https://arxiv.org/html/2605.13414#S3.SS2.SSS0.Px8.p1.10)\.
- R\. J\. Leonesio and T\. O\. Nelson \(1990\)Do different metamemory judgments tap the same underlying aspects of memory?\.Journal of Experimental Psychology: Learning, Memory, and Cognition16\(3\),pp\. 464–470\.External Links:[Document](https://dx.doi.org/10.1037/0278-7393.16.3.464)Cited by:[§3\.1](https://arxiv.org/html/2605.13414#S3.SS1.p2.1)\.
- Z\. Li, Q\. Dong, J\. Ma, D\. Zhang, K\. Jia, and Z\. Sui \(2025\)SelfBudgeter: adaptive token allocation for efficient LLM reasoning\.arXiv preprint arXiv:2505\.11274\.Cited by:[§1](https://arxiv.org/html/2605.13414#S1.p3.1),[§2\.2](https://arxiv.org/html/2605.13414#S2.SS2.p2.1),[§3\.1](https://arxiv.org/html/2605.13414#S3.SS1.SSS0.Px3.p1.1)\.
- F\. Lieder and T\. L\. Griffiths \(2020\)Resource\-rational analysis: understanding human cognition as the optimal use of limited computational resources\.Behavioral and Brain Sciences43,pp\. e1\.External Links:[Document](https://dx.doi.org/10.1017/S0140525X1900061X)Cited by:[§1](https://arxiv.org/html/2605.13414#S1.p2.1)\.
- X\. Lin, J\. H\. Liew, S\. Savarese, and J\. Li \(2026\)W&D: scaling parallel tool calling for efficient deep research agents\.arXiv preprint\.External Links:2602\.07359Cited by:[item Deployment realism\.](https://arxiv.org/html/2605.13414#S3.I1.ix2.p1.1)\.
- N\. F\. Liu, K\. Lin, J\. Hewitt, A\. Paranjape, M\. Bevilacqua, F\. Petroni, and P\. Liang \(2024\)Lost in the middle: how language models use long contexts\.Transactions of the Association for Computational Linguistics12,pp\. 157–173\.Cited by:[§4](https://arxiv.org/html/2605.13414#S4.SS0.SSS0.Px2.p1.3)\.
- G\. Mazzoni and C\. Cornoldi \(1993\)Strategies in study time allocation: why is study time sometimes not effective?\.Journal of Experimental Psychology: General122\(1\),pp\. 47–60\.External Links:[Document](https://dx.doi.org/10.1037/0096-3445.122.1.47)Cited by:[item Methodological isolation\.](https://arxiv.org/html/2605.13414#S3.I1.ix1.p1.1)\.
- J\. Metcalfe and N\. Kornell \(2005\)A region of proximal learning model of study time allocation\.Journal of Memory and Language52\(4\),pp\. 463–477\.External Links:[Document](https://dx.doi.org/10.1016/j.jml.2004.12.001)Cited by:[§3\.1](https://arxiv.org/html/2605.13414#S3.SS1.p1.1)\.
- G\. Mialon, C\. Fourrier, C\. Swift, T\. Wolf, Y\. LeCun, and T\. Scialom \(2024\)GAIA: a benchmark for general AI assistants\.InInternational Conference on Learning Representations,Cited by:[§2\.2](https://arxiv.org/html/2605.13414#S2.SS2.p3.1)\.
- A\. H\. Murphy \(1973\)A new vector partition of the probability score\.Journal of Applied Meteorology12\(4\),pp\. 595–600\.External Links:[Document](https://dx.doi.org/10.1175/1520-0450%281973%29012%3C0595%3AANVPOT%3E2.0.CO%3B2)Cited by:[§3\.2](https://arxiv.org/html/2605.13414#S3.SS2.SSS0.Px5.p1.5)\.
- T\. O\. Nelson and L\. Narens \(1990\)Metamemory: a theoretical framework and new findings\.InThe Psychology of Learning and Motivation,G\. H\. Bower \(Ed\.\),Vol\.26,pp\. 125–173\.Cited by:[§1](https://arxiv.org/html/2605.13414#S1.p2.1),[§1](https://arxiv.org/html/2605.13414#S1.p4.1),[§2\.1](https://arxiv.org/html/2605.13414#S2.SS1.p1.1),[§3\.1](https://arxiv.org/html/2605.13414#S3.SS1.p1.1),[§3\.2](https://arxiv.org/html/2605.13414#S3.SS2.SSS0.Px4.p4.2)\.
- L\. Phan, A\. Gatti, Z\. Han, N\. Li, J\. Hu, H\. Zhang, C\. B\. C\. Zhang, M\. Shaaban, J\. Ling, S\. Shi,et al\.\(2025\)Humanity’s last exam\.arXiv preprint arXiv:2501\.14249\.Cited by:[§1](https://arxiv.org/html/2605.13414#S1.p5.1),[§4](https://arxiv.org/html/2605.13414#S4.SS0.SSS0.Px2.p1.3)\.
- D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. Bowman \(2024\)GPQA: a graduate\-level Google\-proof Q&A benchmark\.InConference on Language Modeling \(COLM\),External Links:2311\.12022Cited by:[§1](https://arxiv.org/html/2605.13414#S1.p5.1),[§4](https://arxiv.org/html/2605.13414#S4.SS0.SSS0.Px2.p1.3)\.
- C\. Snell, J\. Lee, K\. Xu, and A\. Kumar \(2025\)Scaling LLM test\-time compute optimally can be more effective than scaling model parameters\.InThe Thirteenth International Conference on Learning Representations \(ICLR\),Note:Oral\. arXiv:2408\.03314Cited by:[§1](https://arxiv.org/html/2605.13414#S1.p1.1),[§2\.2](https://arxiv.org/html/2605.13414#S2.SS2.p1.1),[§3\.1](https://arxiv.org/html/2605.13414#S3.SS1.SSS0.Px1.p1.1)\.
- L\. K\. Son and J\. Metcalfe \(2000\)Metacognitive and control strategies in study\-time allocation\.Journal of Experimental Psychology: Learning, Memory, and Cognition26\(1\),pp\. 204–221\.External Links:[Document](https://dx.doi.org/10.1037/0278-7393.26.1.204)Cited by:[§1](https://arxiv.org/html/2605.13414#S1.p4.1),[§3\.1](https://arxiv.org/html/2605.13414#S3.SS1.p1.1)\.
- L\. K\. Son \(2004\)Spacing one’s study: evidence for a metacognitive control strategy\.Journal of Experimental Psychology: Learning, Memory, and Cognition30\(3\),pp\. 601–604\.External Links:[Document](https://dx.doi.org/10.1037/0278-7393.30.3.601)Cited by:[item Methodological isolation\.](https://arxiv.org/html/2605.13414#S3.I1.ix1.p1.1)\.
- M\. Steyvers and M\. A\. K\. Peters \(2025\)Metacognition and uncertainty communication in humans and large language models\.Current Directions in Psychological Science\.External Links:[Document](https://dx.doi.org/10.1177/09637214251391158)Cited by:[§2\.1](https://arxiv.org/html/2605.13414#S2.SS1.SSS0.Px1.p3.1)\.
- R\. Tang, D\. Kong, L\. Huang, and H\. Xue \(2023\)Large language models can be lazy learners: analyze shortcuts in in\-context learning\.InFindings of the Association for Computational Linguistics: ACL 2023,pp\. 4645–4657\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.findings-acl.284)Cited by:[§3\.2](https://arxiv.org/html/2605.13414#S3.SS2.SSS0.Px8.p1.10)\.
- K\. W\. Thiede and J\. Dunlosky \(1999\)Toward a general model of self\-regulated study: an analysis of selection of items for study and self\-paced study time\.Journal of Experimental Psychology: Learning, Memory, and Cognition25\(4\),pp\. 1024–1037\.External Links:[Document](https://dx.doi.org/10.1037/0278-7393.25.4.1024)Cited by:[item Construct priority\.](https://arxiv.org/html/2605.13414#S3.I1.ix3.p1.1)\.
- B\. J\. Underwood \(1966\)Experimental psychology\.2nd edition,Appleton\-Century\-Crofts,New York\.Cited by:[§3\.1](https://arxiv.org/html/2605.13414#S3.SS1.p2.1)\.
- K\. Valmeekam, M\. Marquez, A\. Olmo, S\. Sreedharan, and S\. Kambhampati \(2023\)PlanBench: an extensible benchmark for evaluating large language models on planning and reasoning about change\.InAdvances in Neural Information Processing Systems,Vol\.36\.Note:Datasets and Benchmarks TrackCited by:[§2\.2](https://arxiv.org/html/2605.13414#S2.SS2.p3.1)\.
- Wanget al\.\(2025\)Decoupling metacognition from cognition: a framework for quantifying metacognitive ability in LLMs\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v39i24.34723)Cited by:[§2\.1](https://arxiv.org/html/2605.13414#S2.SS1.SSS0.Px1.p1.1)\.
- Y\. Wang, Q\. Liu, J\. Xu, T\. Liang, X\. Chen, Z\. He, L\. Song, D\. Yu, J\. Li, Z\. Zhang, R\. Wang, Z\. Tu, H\. Mi, and D\. Yu \(2025\)Thoughts are all over the place: on the underthinking of o1\-like LLMs\.InAdvances in Neural Information Processing Systems,Vol\.38\.Note:Spotlight\. arXiv:2501\.18585Cited by:[§1](https://arxiv.org/html/2605.13414#S1.p1.1),[§2\.2](https://arxiv.org/html/2605.13414#S2.SS2.p1.1)\.
- M\. Xiong, Z\. Hu, X\. Lu, Y\. Li, J\. Fu, J\. He, and B\. Hooi \(2024\)Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2306.13063)Cited by:[§2\.1](https://arxiv.org/html/2605.13414#S2.SS1.SSS0.Px1.p1.1)\.
- B\. Xu, Z\. Peng, B\. Lei, S\. Mukherjee, Y\. Liu, and D\. Xu \(2023\)ReWOO: decoupling reasoning from observations for efficient augmented language models\.arXiv preprint\.External Links:2305\.18323Cited by:[§3\.1](https://arxiv.org/html/2605.13414#S3.SS1.SSS0.Px2.p1.1)\.
- S\. Yao, N\. Shinn, P\. Razavi, and K\. Narasimhan \(2024\)τ\\tau\-Bench: a benchmark for tool\-agent\-user interaction in real\-world domains\.arXiv preprint arXiv:2406\.12045\.Cited by:[§2\.2](https://arxiv.org/html/2605.13414#S2.SS2.p3.1)\.

## Appendix ALimitations and Broader Impact

TRIAGE measures one specific competence \- prospective, portfolio\-level resource allocation under a hard shared budget \- and the design choices required to make this competence measurable bound what can be concluded from it\.

#### Construct and design\.

TRIAGE operationalizes metacognitive control as a single up\-front plan: the model commits to selections, cost estimates, and an execution order before any task is attempted\. This isolates prospective self\-assessment from in\-the\-loop revision, at the price of excluding settings where allocation is naturally interactive or streaming\. We treat this trade as deliberate \- the construct is most informative when revision is unavailable \- but TRIAGE scores should not be read as a global verdict on a model’s resource\-allocation ability across deployment regimes\. The budget is also denominated in a single resource per track \(token cost in the reasoning track, tool calls in the agentic track\); evaluations conditioned on wall\-clock latency, monetary cost, or compound resources may rank models differently\.

#### Ground truth and measurement noise\.

Per\-task feasibility and cost are pre\-computed by running each model on each item once, at temperature 0, prior to evaluation\. This makes per\-cell outcomes approximately deterministic at the provider/runtime level, so we do not report bootstrap error bars on per\-cellη\\eta\. Residual noise persists from provider\-side non\-determinism and from items near a model’s capability frontier, where small perturbations can flipyiy\_\{i\}\. The unsolvable\-injection slice further depends on injected items being genuinely unsolvable for the evaluated models; we audit this qualitatively but cannot exclude occasional successful completions on a small fraction\.

#### External validity\.

Our model roster is a snapshot of frontier systems at submission time and will be partially superseded within months\. We have no access to closed\-model training data and cannot certify that TRIAGE items were held out of any model’s pre\- or post\-training\. Tasks are sourced from public reasoning and agentic benchmarks, which biases the picture toward English\-language, formally\-stated problems; conclusions about portfolio\-level metacognition do not automatically transfer to multilingual, multi\-modal, or open\-ended generation regimes\. Models also lack episodic memory of their own past performance on TRIAGE\-like tasks; their “self”\-assessment is therefore closer to assessment of the model class than to grounded recollection\.

#### Broader impacts\.

The intended positive impact of TRIAGE is diagnostic: surfacing systematic overconfidence and miscalibration in resource decisions before LLMs are placed in deployments where compute, money, or human attention are at stake\. The principal risk is dual\-use of the artifact itself\. Used unmodified as a training signal, TRIAGE could be optimized against, yielding models that score well without genuinely improved monitoring \- a known failure mode of any sufficiently legible benchmark\. We caution that strong TRIAGE scores are not a license to delegate consequential decisions to LLMs: passing the benchmark is necessary, not sufficient, for such delegation\.

## Appendix BPlanner Prompt

The planner receives the prompt template shown below\. The prompt is held fixed across both regimes \(U and E\); regime U treats the per\-problem allocations as advisory, while regime E enforces them as hard constraint \(§[3\.2](https://arxiv.org/html/2605.13414#S3.SS2.SSS0.Px4)\)\.

`Planner Prompt`

`Problem block format\. Each problem in the \{problems\} field is rendered as: ⬇ \[id: <problem\_id\>\] \(points: 1\) <problem\_text\> Plan parsing and repair\. Plan outputs are parsed as JSON\. A small fraction of outputs contain recoverable formatting issues \(trailing commentary outside the JSON object, extra fields, malformed numerics\); these pass through a deterministic repair pass that strips non\-JSON content, coerces token counts to non\-negative integers, and discards plan items whose id does not appear in the pool\. Plans that remain unparseable after repair are excluded; pools with no parseable plan for a model are excluded from that model’s η\\eta average for the affected \(dataset, α\\alpha\) cell\. Appendix C Additional Experiments Figure 4: Per\-\(model, mode\) accuracy on each benchmark\. Accuracy is the proportion of solvable problems and is independent of budget α\\alpha\. Cells use a sequential white→\\rightarrowteal scale; horizontal lines separate model families; the \[std\] / \[ext\] suffix denotes standard inference versus extended reasoning\. Figure 5: Triage skill in the advisory\-budget regime, ηU\\eta\_\{U\}, per \(model, α\\alpha, dataset\)\. ηU=1\\eta\_\{U\}=1 matches oracle planning, ηU=0\\eta\_\{U\}=0 matches a random baseline, ηU<0\\eta\_\{U\}<0 is worse than random\. Cells use a diverging coral→\\rightarrownavy palette capped at \[−1,1\]\[\-1,1\]; values outside this range are marked with ∗\*\. Bold vertical rules separate α\\alpha blocks\. Figure 6: Triage skill in the enforced\-budget regime, ηE\\eta\_\{E\}, per \(model, α\\alpha, dataset\)\. Color encoding identical to Figure˜5\. The α=1\.0\\alpha=1\.0 column block mostly saturates near zero because Voracle≈VrandomV\_\{\\mathrm\{oracle\}\}\\approx V\_\{\\mathrm\{random\}\} at full budget\. C\.1 Budget\-aware re\-solving: do models honor their own allocations? The triage protocol measures whether a model can identify worthwhile problems given a fixed token budget; it does not test whether the model can actually execute under its own self\-declared per\-problem allocation aia\_\{i\}\. We probe this directly with a budget\-aware re\-solve experiment on three models \- Gemini 2\.5 Flash, Kimi K2\.5, and GLM 5\.1, all in standard\-inference mode\. For every \(model, problem\) where the α=1\.0\\alpha=1\.0 plan selected the problem with ai\>0a\_\{i\}\>0, we re\-issue the standard solver prompt with an instruction stating the model’s own aia\_\{i\}: Budget\-aware banner \(prepended to the standard solver prompt\) You have a strict output\-token budget of \{a\_i\} tokens for solving this problem, including any reasoning, drafts, or scratch work you produce\. Plan your solution to fit within this limit\. Be concise, skip unnecessary explanation, and commit to your final answer before exhausting the budget\. The API\-level max\_tokens is unchanged from the baseline run, so this imposes a self\-imposed cap rather than a hard truncation\. We record two outcomes per problem: Accbaseline\{\}\_\{\\text\{baseline\}\} \(the original baseline solve, no banner, full max\_tokens\) and Accbudget\-aware\{\}\_\{\\text\{budget\-aware\}\} \(with the banner\)\. We also report compliance, defined as the fraction of problems for which the model’s actual output length stays within its self\-declared budget, i\.e\. compliance=1N​∑i=1N𝟙​\[output\_tokensi≤ai\]\\text\{compliance\}=\\tfrac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbb\{1\}\[\\,\\text\{output\\\_tokens\}\_\{i\}\\leq a\_\{i\}\\,\]\. Because no hard cap is applied, compliance measures whether the model voluntarily respects the budget statement\. Finally, we decompose the change between the baseline and budget\-aware runs into four pairwise outcome counts: kept correct \(correct in both runs\), lost correct \(correct at baseline, wrong with the banner\), newly correct \(wrong at baseline, correct with the banner\), and still wrong \(incorrect in both runs\)\. The aggregate counts are reported in Figure˜7; the per\-dataset breakdown is in Figure˜8\. Stating the budget improves accuracy, but compliance is poor\. For all three models, Accbudget\-aware\{\}\_\{\\text\{budget\-aware\}\} exceeds Accbaseline\{\}\_\{\\text\{baseline\}\} in aggregate \(\+1\.9\+1\.9 pp for Gemini 2\.5 Flash, \+4\.9\+4\.9 pp for Kimi K2\.5, and \+8\.3\+8\.3 pp for GLM 5\.1; Figure˜7\)\. Yet only Gemini 2\.5 Flash exhibits a meaningful compliance rate \(36\.6%36\.6\\%\); Kimi K2\.5 stays within its self\-declared budget on only 6\.0%6\.0\\% of problems and GLM 5\.1 on 16\.0%16\.0\\%\. The accuracy gains therefore arise even when the model overruns its stated budget \- the budget\-aware instruction functions more as a conciseness prior than as a binding constraint\. This is consistent with the gap between the advisory and enforced regimes reported in §5\.1: models can plan against a budget statement but rarely execute within it\. Per\-dataset patterns\. Figure˜8 shows that the budget\-aware lift is concentrated on the structured\-reasoning datasets \(AIME, GPQA, LCB\) where models also have higher baseline accuracy\. On HLE, all three models remain near floor in every column\. GLM 5\.1’s lift on AIME is the largest single effect in the experiment: Acc=baseline0\.10→Accbudget\-aware=0\.61\{\}\_\{\\text\{baseline\}\}=0\.10\\rightarrow\\text\{Acc\}\_\{\\text\{budget\-aware\}\}=0\.61\. This is consistent with GLM 5\.1’s leading ηU\\eta\_\{U\} in the main results: when the model is told to be concise, it commits to its final answer earlier and avoids the long, drifting traces that cost it correctness in the unconstrained setting\. Figure 7: Budget\-aware re\-solve, aggregate per model\. Cells colored by intensity within each metric\. NN is the number of \(problem, allocation\) pairs re\-issued\. Compliance is the fraction of problems for which the model’s actual output length stays within its self\-declared budget aia\_\{i\}\. The four right\-hand counts \(newly correct, lost correct, kept correct, still wrong\) sum to NN and decompose the change between Accbaseline\{\}\_\{\\text\{baseline\}\} \(original baseline solve\) and Accbudget\-aware\{\}\_\{\\text\{budget\-aware\}\} \(baseline solver prompt augmented with an instruction to answer within the model’s own aia\_\{i\}\)\. Figure 8: Budget\-aware re\-solve, per \(model ×\\times dataset\)\. HLE sub\-domains are aggregated by problem\-count weighted mean\. Rows within each model block: original baseline accuracy, accuracy with the budget banner, and compliance rate\. Color encoding follows Figure˜7\. C\.2 Prompt\-Sensitivity Ablation To check whether η\\eta depends on the prompt, we re\-ran the planning step under three framings of the same task: A \(neutral, used elsewhere in the paper\), B \(autonomous agent\), and C \(timed exam\)\. The Rules block, problem text, budget, output schema, and ordering instruction are identical across the three; only the opening paragraph differs\. Triage Prompt Variant A — Neutral \(default\) You are given a set of \{domain\} to attempt under a fixed output\-token budget\. You do not have to attempt every problem\. Deciding which problems to attempt and in what order is part of the task\. Triage Prompt Variant B — Autonomous Agent You are an autonomous planning agent operating under a fixed compute budget of \{B\} output tokens\. A set of \{domain\} has been assigned to you\. You cannot expect to complete every assignment within budget\. Your job is to select which to execute and in what order so as to maximize total reward\. Some assignments may be infeasible for you — committing tokens to those yields no reward\. Triage Prompt Variant C — Timed Exam You are taking a timed exam with a fixed output\-token budget of \{B\} across all questions\. A set of \{domain\} is in front of you\. You do not have to attempt every question — some may be too difficult or unanswerable within the time available\. Decide which questions to attempt and in what order so as to maximize your total score\. We ran the full grid: 2 models, 2 modes, 8 datasets, 4 budget levels, 3 variants \- 16 \(model, mode, α\\alpha\) cells per variant under each regime\. Results\. For ηU\\eta\_\{U\}, 11 of 16 cells stay within 0\.1 across variants; for ηE\\eta\_\{E\}, 14 of 16 \(Tables 1, 2\)\. The two largest spreads in ηU\\eta\_\{U\} are both in think\-mode at extreme budgets: Kimi K2\.5/think at α=1\.0\\alpha\\\!=\\\!1\.0 \(range 0\.3830\.383, with Variant C alone dropping to 0\.5060\.506\) and Qwen 3\.5 27B/think at α=0\.75\\alpha\\\!=\\\!0\.75 \(range 0\.1880\.188\)\. The α=0\.75\\alpha\\\!=\\\!0\.75 case is not random\. Variant A gives the lowest ηU\\eta\_\{U\} in 3 of 4 think\-mode cells at α≥0\.5\\alpha\\geq 0\.5\. Variants B and C tell the model some problems may be infeasible; A does not\. Reasoning models seem to need that cue \- without it they attempt too many problems\. Non\-thinking modes show no such pattern \(median range 0\.0400\.040\)\. Ranking stability\. Table 3 reports Kendall’s τb\\tau\_\{b\} on the four\-model ranking induced by each variant pair\. With n=4n=4, τb\\tau\_\{b\} can only take the values \{−1,−2/3,−1/3,0,1/3,2/3,0\.913,1\}\\\{\-1,\-2/3,\-1/3,0,1/3,2/3,0\.913,1\\\}: a single rank swap drops τ\\tau from 11 to 0\.6670\.667\. We therefore report raw τ\\tau rather than apply a threshold\. All 24 pairwise values are ≥2/3\\geq 2/3, and the swapped models differ by <0\.1<\\\!0\.1 in η\\eta \- within the cell\-level spread we already report\. The rankings are stable\. Taken together \- 78% of cells within 0\.10\.1 spread, all τ≥2/3\\tau\\geq 2/3, and the largest disagreement isolated to one \(model, mode, α\\alpha\) corner \- ηU\\eta\_\{U\} is robust to the prompt framings tested\. Table 1: ηU\\eta\_\{U\} under each prompt variant\. Each value is the mean over 8 data splits \(AIME, GPQA Diamond, LiveCodeBench, HLE\-CS/AI, HLE\-Physics, HLE\-Humanities, HLE\-Other, HLE\-Engineering\) at that α\\alpha\. range = max−min\\max\-\\min across A, B, C; cells with range ≥0\.10\\geq 0\.10 are bolded\. Model Mode α\\alpha ηUA\\eta^\{A\}\_\{U\} ηUB\\eta^\{B\}\_\{U\} ηUC\\eta^\{C\}\_\{U\} range Kimi K2\.5 nothink 0\.25 0\.615 0\.613 0\.575 0\.040 0\.50 0\.530 0\.659 0\.554 0\.130 0\.75 0\.681 0\.781 0\.776 0\.100 1\.00 0\.978 1\.000 1\.000 0\.022 think 0\.25 0\.432 0\.497 0\.412 0\.085 0\.50 0\.463 0\.477 0\.526 0\.063 0\.75 0\.333 0\.363 0\.271 0\.092 1\.00 0\.778 0\.889 0\.506 0\.383 Qwen 3\.5 27B nothink 0\.25 0\.446 0\.428 0\.445 0\.018 0\.50 0\.379 0\.353 0\.443 0\.090 0\.75 0\.507 0\.523 0\.556 0\.049 1\.00 0\.933 0\.956 0\.933 0\.022 think 0\.25 0\.414 0\.361 0\.391 0\.053 0\.50 0\.333 0\.448 0\.334 0\.115 0\.75 0\.287 0\.415 0\.475 0\.188 1\.00 0\.889 0\.889 0\.911 0\.022 Cells with range <0\.10<0\.10: 11 / 16 Median range: 0\.077 Table 2: ηE\\eta\_\{E\} under each prompt variant, same layout as Table 1\. Negative values reflect systematic under\-allocation in the enforced regime \- a model property, the same under all three prompts\. Model Mode α\\alpha ηEA\\eta^\{A\}\_\{E\} ηEB\\eta^\{B\}\_\{E\} ηEC\\eta^\{C\}\_\{E\} range Kimi K2\.5 nothink 0\.25 0\.152 0\.152 0\.148 0\.004 0\.50 −0\.254\-0\.254 −0\.232\-0\.232 −0\.245\-0\.245 0\.022 0\.75 −1\.421\-1\.421 −1\.352\-1\.352 −1\.421\-1\.421 0\.069 1\.00 0\.356 0\.356 0\.356 0\.000 think 0\.25 −0\.192\-0\.192 −0\.194\-0\.194 −0\.237\-0\.237 0\.045 0\.50 −0\.712\-0\.712 −0\.692\-0\.692 −0\.700\-0\.700 0\.020 0\.75 −2\.193\-2\.193 −2\.211\-2\.211 −2\.282\-2\.282 0\.089 1\.00 −0\.367\-0\.367 −0\.433\-0\.433 −0\.532\-0\.532 0\.165 Qwen 3\.5 27B nothink 0\.25 0\.089 0\.157 0\.143 0\.068 0\.50 −0\.161\-0\.161 −0\.152\-0\.152 −0\.159\-0\.159 0\.008 0\.75 −1\.180\-1\.180 −1\.113\-1\.113 −1\.151\-1\.151 0\.067 1\.00 0\.222 0\.200 0\.222 0\.022 think 0\.25 −0\.110\-0\.110 −0\.111\-0\.111 −0\.105\-0\.105 0\.005 0\.50 −0\.479\-0\.479 −0\.552\-0\.552 −0\.540\-0\.540 0\.073 0\.75 −1\.568\-1\.568 −1\.597\-1\.597 −1\.782\-1\.782 0\.214 1\.00 0\.311 0\.333 0\.244 0\.089 Cells with range <0\.10<0\.10: 14 / 16 Median range: 0\.057 Table 3: Kendall’s τb\\tau\_\{b\} between the four\-model rankings induced by each variant pair\. max range is the largest across\-variant spread in any \(model, mode\) row at that α\\alpha\. Appendix D Boundary behaviour of η\\eta This appendix gives a formal account of the conditions under which ηM\\eta\_\{M\} in \(2\) is well\-defined, the convention used when the denominator vanishes, and the filtering rule applied to the main results\. All statements are with respect to the setup of §3\.2: integer token costs ci∈ℤ\>0c\_\{i\}\\in\\mathbb\{Z\}\_\{\>0\}, uniform values vi=1v\_\{i\}=1, budget B=⌊α​∑ici⌋B=\\lfloor\\alpha\\sum\_\{i\}c\_\{i\}\\rfloor, and regime U execution\. D\.1 Domain of definition The ratio in \(2\) is defined whenever Voracle\>VrandomV\_\{\\text\{oracle\}\}\>V\_\{\\text\{random\}\}\. Two structural conditions force Voracle=VrandomV\_\{\\text\{oracle\}\}=V\_\{\\text\{random\}\} and hence leave ηM\\eta\_\{M\} undefined: \(D1\) Empty solvable set: Voracle=0V\_\{\\text\{oracle\}\}=0\. If yi=0y\_\{i\}=0 for every ii in the pool, then VM=0V\_\{M\}=0 for any plan, and likewise Vrandom=0V\_\{\\text\{random\}\}=0\. The metric carries no signal: every planner \- including one that does nothing \- ties at zero\. \(D2\) Slack\-budget collapse at α=1\\alpha=1\. At α=1\\alpha=1, B=∑iciB=\\sum\_\{i\}c\_\{i\}\. Proposition D\.1 shows that under regime U every permutation of the gradeable items completes without truncation, so Vrandom=∑iyi=VoracleV\_\{\\text\{random\}\}=\\sum\_\{i\}y\_\{i\}=V\_\{\\text\{oracle\}\} deterministically\. Proposition D\.1 \(Truncation impossibility at full budget\)\. Let the pool have gradeable items with costs c1,…,cn∈ℤ\>0c\_\{1\},\\ldots,c\_\{n\}\\in\\mathbb\{Z\}\_\{\>0\}, budget B=∑i=1nciB=\\sum\_\{i=1\}^\{n\}c\_\{i\}, and any permutation \(i1,…,in\)\(i\_\{1\},\\ldots,i\_\{n\}\) of \{1,…,n\}\\\{1,\\ldots,n\\\}\. Under regime U, no truncation occurs and the plan executes every item\. Proof\. After processing the first kk items, the remaining budget is Rk=B−∑j=1kcij=∑j=k\+1ncij\.R\_\{k\}\\;=\\;B\-\\sum\_\{j=1\}^\{k\}c\_\{i\_\{j\}\}\\;=\\;\\sum\_\{j=k\+1\}^\{n\}c\_\{i\_\{j\}\}\. For k<nk<n, item ik\+1i\_\{k\+1\} truncates iff cik\+1\>Rkc\_\{i\_\{k\+1\}\}\>R\_\{k\}, i\.e\., iff ∑j=k\+2ncij<0\\sum\_\{j=k\+2\}^\{n\}c\_\{i\_\{j\}\}<0 \(taking the empty sum as 0 when k=n−1k=n\-1\)\. Since ci\>0c\_\{i\}\>0 for all ii, the strict inequality is never satisfied\. Hence execution proceeds through all nn items\. ∎ A corollary: at α=1\\alpha=1 the knapsack constraint in \(1\) is non\-binding, so xi=1x\_\{i\}=1 for all ii is feasible\. Since the oracle objective is non\-decreasing in each xix\_\{i\}, this is optimal and Voracle=∑iyi=VrandomV\_\{\\text\{oracle\}\}=\\sum\_\{i\}y\_\{i\}=V\_\{\\text\{random\}\}, with zero Monte\-Carlo variance across the 10310^\{3\} random shuffles\. Combining \(D1\) and \(D2\): ηM\\eta\_\{M\} is undefined at every α\\alpha for pools with Voracle=0V\_\{\\text\{oracle\}\}=0, and at α=1\\alpha=1 for all remaining pools\. D\.2 One\-sided limit and implementation convention For pools with Voracle≥1V\_\{\\text\{oracle\}\}\\geq 1, consider ηM\\eta\_\{M\} as α→1−\\alpha\\to 1^\{\-\}\. The denominator Voracle​\(α\)−Vrandom​\(α\)V\_\{\\text\{oracle\}\}\(\\alpha\)\-V\_\{\\text\{random\}\}\(\\alpha\) is non\-negative for all α∈\(0,1\]\\alpha\\in\(0,1\] \(the oracle is optimal among feasible plans, so it dominates any expectation over them\) and shrinks to 0 as α→1\\alpha\\to 1\. Two cases: 1\. Achievement branch\. If VM​\(α\)=Voracle​\(α\)V\_\{M\}\(\\alpha\)=V\_\{\\text\{oracle\}\}\(\\alpha\) on a left neighbourhood of α=1\\alpha=1, then ηM​\(α\)=1\\eta\_\{M\}\(\\alpha\)=1 identically, and limα→1−ηM​\(α\)=1\\lim\_\{\\alpha\\to 1^\{\-\}\}\\eta\_\{M\}\(\\alpha\)=1\. 2\. Failure branch\. If VM​\(α\)<Voracle​\(α\)V\_\{M\}\(\\alpha\)<V\_\{\\text\{oracle\}\}\(\\alpha\) near α=1\\alpha=1, the numerator approaches VM​\(1\)−Voracle​\(1\)<0V\_\{M\}\(1\)\-V\_\{\\text\{oracle\}\}\(1\)<0 while the denominator approaches 0 from above\. Hence limα→1−ηM​\(α\)=−∞\\lim\_\{\\alpha\\to 1^\{\-\}\}\\eta\_\{M\}\(\\alpha\)=\-\\infty\. The released implementation extends ηM\\eta\_\{M\} to cases with Voracle=VrandomV\_\{\\text\{oracle\}\}=V\_\{\\text\{random\}\} by ηM:= 1​\[VM≥Voracle\]\.\\eta\_\{M\}\\;\\mathrel\{:=\}\\;\\mathbf\{1\}\\\!\\left\[V\_\{M\}\\geq V\_\{\\text\{oracle\}\}\\right\]\. \(4\) The achievement value in \(4\) coincides with the one\-sided limit\. The failure value is clipped to 0 rather than reported as −∞\-\\infty, consistent with the interpretation of ηM=0\\eta\_\{M\}=0 as “no skill”\. This extension is the convention used for ηM\\eta\_\{M\} throughout the paper, including pools satisfying \(D1\)\. In practice, the largest\-magnitude negative efficiency values arise at α\\alpha near 11, where the denominator Voracle−VrandomV\_\{\\text\{oracle\}\}\-V\_\{\\text\{random\}\} shrinks while remaining positive, while the regret metric R~M\\tilde\{R\}\_\{M\} \(§D\.5\) provides a bounded alternative for these cases\. D\.3 Filtering rule The normalized regret R~M\\tilde\{R\}\_\{M\} \(§D\.5\) requires Voracle≥1V\_\{\\text\{oracle\}\}\\geq 1 to be defined, so when we report R~M\\tilde\{R\}\_\{M\} we restrict to pools satisfying this condition, equivalently discarding pools under \(D1\)\. These are pure capability\-ceiling cases: the model fails every problem under standard execution, so every plan achieves VM=Voracle=0V\_\{M\}=V\_\{\\text\{oracle\}\}=0 and the regret denominator vanishes\. For ηM\\eta\_\{M\}, no such filter is applied; pools under \(D1\) are retained using the convention in \(4\), which assigns ηM=1\\eta\_\{M\}=1 when VM≥VoracleV\_\{M\}\\geq V\_\{\\text\{oracle\}\} and Voracle=0V\_\{\\text\{oracle\}\}=0\. D\.4 Range For pools where ηM\\eta\_\{M\} is genuinely informative \(Voracle\>VrandomV\_\{\\text\{oracle\}\}\>V\_\{\\text\{random\}\}\): • ηM≤1\\eta\_\{M\}\\leq 1, with equality iff VM=VoracleV\_\{M\}=V\_\{\\text\{oracle\}\}\. This follows from VM≤VoracleV\_\{M\}\\leq V\_\{\\text\{oracle\}\} \(the oracle is the maximum over feasible plans\) and a strictly positive denominator\. • ηM\\eta\_\{M\} is unbounded below\. As VoracleV\_\{\\text\{oracle\}\} and VrandomV\_\{\\text\{random\}\} approach each other \(the denominator shrinks toward zero from above\), ηM\\eta\_\{M\} can take arbitrarily large negative values for any fixed VM<VrandomV\_\{M\}<V\_\{\\text\{random\}\}\. We therefore characterize ηM∈\(−∞,1\]\\eta\_\{M\}\\in\(\-\\infty,1\]\. D\.5 Sensitivity check: regret as an alternative normalization The asymmetric range of ηM\\eta\_\{M\} motivates a second, bounded metric for robustness: the normalized regret R~M=Voracle−VMVoracle,\\tilde\{R\}\_\{M\}\\;=\\;\\frac\{V\_\{\\text\{oracle\}\}\-V\_\{M\}\}\{V\_\{\\text\{oracle\}\}\}, \(5\) defined whenever Voracle≥1V\_\{\\text\{oracle\}\}\\geq 1\. By construction R~M∈\[0,1\]\\tilde\{R\}\_\{M\}\\in\[0,1\], with R~M=0\\tilde\{R\}\_\{M\}=0 at oracle and R~M=1\\tilde\{R\}\_\{M\}=1 when the planner captures none of the oracle’s value\. Unlike ηM\\eta\_\{M\}, R~M\\tilde\{R\}\_\{M\} has no denominator singularity, is continuous at α=1\\alpha=1, and admits the same interpretation across all α\\alpha\. We report R~M\\tilde\{R\}\_\{M\} alongside ηM\\eta\_\{M\} and confirm that the conclusions are robust to the choice of normalization\. Appendix E Model Configurations Table 4 lists the 20 models evaluated in the main experiments, together with their providers, parameter counts where disclosed, reasoning support, and the exact API or checkpoint identifiers used\. Open\-weights models were served locally via vLLM; closed models were accessed through their respective provider APIs\. All queries used temperature 0 with provider\-default sampling parameters; reasoning effort was set to the provider default for models that expose a tunable reasoning level\. Reasoning modes\. Each model is evaluated in up to two modes: standard inference \(no extended reasoning\) and extended reasoning \(the model’s native long chain\-of\-thought or thinking mode, where supported\)\. Models without a thinking toggle are evaluated only in standard inference\. The “Reasoning” column in Table 4 indicates which modes were run for each model\. Table 4: Models evaluated in TRIAGE\. Weights: O = open, C = closed\. Params: parameter count where disclosed; for mixture\-of\-experts models, total / active\. Reasoning: S = standard inference only, S\+E = both standard and extended reasoning modes evaluated\. Identifier: provider API string or HuggingFace checkpoint used\. Model Provider Weights Params Reasoning Identifier Qwen 2\.5 series \(Alibaba\) Qwen 2\.5 3B Alibaba O 3B S Qwen/Qwen2\.5\-3B\-Instruct Qwen 2\.5 7B Alibaba O 7B S Qwen/Qwen2\.5\-7B\-Instruct Qwen 2\.5 14B Alibaba O 14B S Qwen/Qwen2\.5\-14B\-Instruct Qwen 2\.5 32B Alibaba O 32B S Qwen/Qwen2\.5\-32B\-Instruct Qwen 3 series \(Alibaba\) Qwen 3 4B Alibaba O 4B E Qwen/Qwen3\-4B Qwen 3 8B Alibaba O 8B E Qwen/Qwen3\-8B Qwen 3 14B Alibaba O 14B E Qwen/Qwen3\-14B Qwen 3 32B Alibaba O 32B E Qwen/Qwen3\-32B Qwen 3\.5 series \(Alibaba\) Qwen 3\.5 4B Alibaba O 4B S\+E Qwen/Qwen3\.5\-4B Qwen 3\.5 9B Alibaba O 9B S\+E Qwen/Qwen3\.5\-9B Qwen 3\.5 27B Alibaba O 27B S\+E Qwen/Qwen3\.5\-27B Qwen 3\.5 35B\-A3B Alibaba O 35B / 3B S\+E Qwen/Qwen3\.5\-35B\-A3B Frontier and other open models Claude Haiku 4\.5 Anthropic C — S\+E claude\-haiku\-4\-5 DeepSeek V3 DeepSeek O 671B / 37B S\+E deepseek\-chat Gemini 2\.5 Flash Google C — S\+E gemini\-2\.5\-flash GLM 5\.1 Zhipu O — S\+E glm\-5\.1 GPT\-5 Mini OpenAI C — S\+E gpt\-5\-mini Kimi K2\.5 Moonshot O — S\+E kimi\-k2\.5 GPT\-OSS 20B OpenAI O 20B E openai/gpt\-oss\-20b GPT\-OSS 120B OpenAI O 120B E openai/gpt\-oss\-120b Sampling and decoding\. All models were queried at temperature 0\. For models accessed via provider APIs, we used each provider’s default values for top\-pp, top\-kk, frequency penalty, and presence penalty unless these parameters were unsupported at temperature 0\. For locally served models, decoding used vLLM defaults with greedy sampling\. Maximum output length was set to a value sufficient to capture the longest plan and reasoning trace observed for each model in pilot runs; exact values per model are recorded in the released configuration files\. Reasoning effort settings\. For models that expose a tunable reasoning level \(e\.g\., reasoning effort, thinking budget\), we used the provider’s default reasoning setting \(e\.g\., medium for GPT\-OSS\) in extended\-reasoning mode and disabled reasoning entirely in standard\-inference mode\.`