Optimal Model Activation Policies for Inference Networks of Large Language Models

# Optimal Model Activation Policies for Inference Networks of Large Language Models
Source: [https://arxiv.org/html/2609.15992](https://arxiv.org/html/2609.15992)
Md Ibrahim Ibne AlamDepartment of Electrical and Computer Engineering Yale UniversityIordanis KoutsopoulosDepartment of Informatics Athens University of Economics and BusinessKoushik KarDepartment of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute

###### Abstract

Recent advances in large language models \(LLMs\) have rendered them necessary for Natural Language Processing \(NLP\) tasks, and their high inference cost motivates the study of cost–performance trade\-offs to carry out these tasks\. In practice, several expert LLMs are used in synergy for inference, either in an ensemble mode or in series, yet without a principled approach on how to best use the available models\. An adaptive approach can route simple queries to cheaper, less reliable LLMs and complex ones to more capable, costly models\. However, because this strategy focuses on query difficulty rather than model expertise, a clear understanding on how to best leverage available expert models is missing\. We introduce and lay the groundwork for*inference networks*, a graph\-based framework, where nodes denote alternative LLMs, and links denote conditional model activations\. The inference network design problem is to determine the best topology, namely the best way to use \(some or all of\) the models that best addresses the cost\-performance trade\-off\. We start from the basic topology of a series of LLM experts, each of which has a different cost and a different level of expertise, which is captured via model confidence\. We formulate the problem of optimal activation of these models so as to minimize the expected inference cost subject to a target performance constraint\. For this special class of inference networks, we prove that the optimal activation policy has a threshold structure: query the lowest\-cost LLM first, and invoke the more expensive LLM only if the confidence falls below a defined threshold\. For discriminative tasks \(e\.g\., classification\), the optimal policy consists of a set of thresholds, one threshold for each class, while for generative tasks \(e\.g\., question\-answering\), it consists of a single threshold\. We provide a structured method to compute the thresholds \(i\.e\., via a low\-dimensional search or a closed\-form solution when applicable\), and practical confidence estimation mechanisms for both task types\. Experiments with off\-the\-shelf open\-source LLMs on text classification and generation benchmarks show substantial cost reductions while meeting the specified performance budget\.

## 1Introduction

Recent developments in generative language modeling, particularly Transformer\-based auto\-regressive large language models \(LLMs\), have significantly enhanced performance across complex natural language processing \(NLP\) tasks\. The exceptional \(often zero\-shot\) capabilities of these foundation models have led practitioners and researchers to increasingly rely on them for building applications\. However, given the high computational burden required to train and use such models \(typically consisting of tens of billions of parameters\), they are often faced with the decision of which LLM service to utilize, given their available budget, quality expectations and significant cost \(which may capture monetary cost, compute cost, or energy consumption\)\. Consequently, this need has led to the development of numerous approaches with different goals\. Methods like model compression and pruning techniques\[lora,puzzle\_nas\]aim to reduce the cost of using such big models while on the other hand, model ensembles\[arora,llm\-blender\]exploit the expressive power of multiple models \(with an increased cost\)\. There are also methods in between, that aim to select the most suitable model, while trying to mitigate the computational demands\. Examples of this class of methods are the alternative formulations of auto\-regressive decoding mechanisms\[decoding\_leviathan\]that use inexpensive LLMs for most generated tokens and switch to costly ones when necessary \(e\.g\., the cheap model’s estimates are not confident enough or poorly aligned to the target’s distribution\)\. Another approach that shows great potential for balancing cost and output quality isadaptive inference, which dynamically adjusts the computational resources spent for a task based on the task complexity\. By selectively switching between different models, adaptive inference methods can significantly lower the inference cost \(e\.g\., computation FLOPs or API money\) without severely compromising the accuracy of the results\. In this work, we move beyond LLM inference schemes that only rely on query characteristics, and instead introduce and study inference networks, a framework of LLM experts that prioritizes model\-specific expertise and theoretical model activations for various NLP tasks\. We define inference networks as directed graphs where the nodes are available LLMs, and a link\(i,j\)\(i,j\)exists when modeljjis activated, possibly based on the result of modeliiwhich is already executed\. The inference network design problem refers to the problem of finding the best topology, namely the best way to use \(some or all of\) the models that best addresses the cost\-performance trade\-off\. Special cases of these inference networks are the ensemble topology \(parallel nodes\), where the available models are used, and their output is aggregated, and the chain topology \(serial nodes\)\. In this work, we fix attention to a simple, yet fundamental case of inference networks, that of models in series\. Thus, for two LLMs, the lower\-cost model is activated first, and its output confidence determines whether a higher\-cost model should be invoked, thereby grounding the routing decision in model\-specific expertise\. A critical component of this system is the rule for deferring the input queries to the larger model\(s\) \(i\.e\., model activation rule\)\. Designing such rules for LLMs that perform generative tasks is not trivial and introduces challenges\[gupta\_uncertainty\]\. Prior works like FrugalGPT\[frugal\]or Hybrid LLM\[hybridlllm\], provide empirical\-only strategies that investigate how to dispatch queries across LLMs, using learned auxiliary routing models\. In contrast, we focus on deriving activation policies with theoretically characterized optimal structure based on LLMs’ self\-confidence, without utilizing an external routing model that would require additional training and data collection\. In particular, under \(mild\) assumptions, we prove that the optimal activation policy is threshold\-based: we first use the lowest\-cost LLM, evaluate the confidence of its response, and escalate to a higher\-cost LLM only when that confidence is below a certain threshold\. For the discriminative tasks such as classification ones, the optimal policy can be expressed as a set of class\-specific thresholds\. Additionally, we empirically demonstrate the superiority of these policies against other state of the art strategies\[frugal,hybridlllm\]in real\-world scenarios, using off\-the\-shelf open\-source LLMs on various text classification and text generation datasets\. We manage to achieve significant cost reduction \(up to 97% in some instances\) while performance stays within a defined budget constraint\. This showcases that our policies can be used in practical scenarios, delivering a better cost\-quality trade\-off\.

## 2Background

##### LLM inference\.

We focus on answering natural language queries\. A range of practical NLP generative or discriminative tasks, such as question answering or text classification, can be naturally expressed as query\-answer tuples\. To generate responses for input queries, we rely on aLLM​\(⋅\):𝒬→𝒜\\text\{LLM\}\(\\cdot\):\\mathcal\{Q\}\\to\\mathcal\{A\}that, given a query𝒒∈𝒬\\boldsymbol\{q\}\\in\\mathcal\{Q\}, produces an answer𝒂^∈𝒜\\boldsymbol\{\\hat\{a\}\}\\in\\mathcal\{A\}\. Most recent advances in the field have also involved approaches like Reinforcement Learning\-based training using rewards from human feedback\[rlhf,dpo\], providing state\-of\-the\-art results and expertise on a variety of tasks beyond simple word completion, like conversation, code generation, instruction following, and commonsense reasoning\[gemma3,llama3,toolformer,deepseek,gpt4,qwen3,mistral,gpt3\]\. During inference, given any query𝒒\\boldsymbol\{q\}, LLMs can handle each task as text generation\. For discriminative tasks,𝒜\\mathcal\{A\}can be a set of predefined classes𝒜=\{ai\}i∈\[𝒜\]\\mathcal\{A\}=\\\{a\_\{i\}\\\}\_\{i\\in\[\\mathcal\{A\}\]\}so one can restrict the model to output only tokens relevant to this set\. In tasks that require explicit text generation, the \(predicted\) posterior distribution over the whole vocabulary\[DBLP:conf/iclr/HoltzmanBDFC20,fan\-etal\-2018\-hierarchical,ficler\-goldberg\-2017\-controlling\]can be sampled to produce a response\.

##### Confidence estimation\.

In addition to generating responses, we also want to estimate how confident the LLM is in its outputs\. Confidence estimation aims to provide a score that reflects the reliability of a model’s answer, and in practice, calculating this score can be challenging\. This challenge can be addressed either using self\-assessed, entropy\-based metrics tied to uncertainty estimation\[lm\-polygraph,gupta\_uncertainty,DBLP:conf/iclr/KuhnGF23\], or through alternative approaches involving external scoring models\[frugal\]\. In this work, we assess the model’s internal certainty through a simple function based on its predicted probability distributions, yielding a confidence score in the range\[0,1\]\[0,1\]\. The type of task \(discriminative or generative\) dictates the function’s definition: generative tasks require a score for the entire text sequence, while discriminative tasks only require a score for the predicted class\. We leave more sophisticated confidence estimation approaches\[Shrivastava,Kadavath,lm\-polygraph\]for future work\.

## 3System Model

The core idea is to perform LLM inference through a serial network of models, each progressively more complex \(i\.e\., with a larger number of parameters and higher inference cost\), applied sequentially\. The serial network is a directed graphG=\(V,E\)\{G\}=\(V,E\), whereV=1,…,NV=\{1,\\dots,N\}andE⊆V×VE\\subseteq V\\times V, such that for every edge\(i,j\)∈E\(i,j\)\\in E,i<ji<j\. This directed graph represents a serial chain, meaning each vertex has exactly one outgoing edge to a larger\-indexed vertex, ensuring a linear structure\. In this work, we focus on the case of two models \(\|V\|=2\|V\|=2\) where the lower\-cost model first processes each query, and a policy determines whether its output should be used; if not, the input query is escalated to the second model\. We use the confidence of the first LLM to decide whether to pass a query to the next node\. The confidence reflects the model’s ability to accurately handle a given query, and it is derived from its generated answer\. This approach shifts the focus from estimating query difficulty \(something that’s often hard to define\) using external scorers\[frugal,hybridlllm\]to using confidence as a readily computable indicator of model performance\. Confidence is influenced by both the difficulty of the query and the expertise of the model, which is defined by how well its scope \(e\.g\., domain of training data\) correlates with the query\. Between two models of similar expertise, the one with higher confidence becomes the more reliable choice for handling the query\. Consider two models: a ‘student’ model \(sLLM\) which incurs less cost to generate an answer per query and a costlier ‘master’ model \(mLLM\)\. For a given query with true answer𝒂\\boldsymbol\{a\}, sLLM is used first, and a deferral policyπ\\pidetermines whether to accept its output answer𝒂^s\\boldsymbol\{\\hat\{a\}\}^\{s\}or defer the input query to mLLM to obtain answer𝒂^m\\boldsymbol\{\\hat\{a\}\}^\{m\}\. We condition the deferral policy on the confidence sLLM outputs for its answer, a scalar in\[0,1\]\[0,1\], denoted byβ\\beta\. Thus, we use a measurable threshold policyπ:\[0,1\]→\{0,1\}\\pi:\[0,1\]\\rightarrow\\\{0,1\\\}, parameterized byθ\\theta:

π​\(β,θ\)=\{0,β≥θ​\(accept sLLM atβ\)1,β<θ​\(defer to mLLM atβ\)\\pi\(\\beta,\\theta\)=\\begin\{cases\}0,&\\beta\\geq\\theta\\,\\text\{\(accept sLLM at $\\beta$\)\}\\\\ 1,&\\beta<\\theta\\,\\text\{\(defer to mLLM at $\\beta$\)\}\\end\{cases\}\(1\)whereβ=conf​\(𝒂^s\)\\beta=\\text\{conf\}\(\\boldsymbol\{\\hat\{a\}\}^\{s\}\)andconf​\(⋅\):𝒜→\[0,1\]\\text\{conf\}\(\\cdot\):\\mathcal\{A\}\\to\[0,1\]is the function that measuressLLM’s confidence regarding the generated answer\. We aim to characterize the optimal policy \(i\.e\., the optimal threshold\) which emerges out of the optimization problem of minimizing the expected inference cost subject to a target performance constraint\. Assuming the distributions of variables such asβ\\betaare known, we can solve this problem either analytically or numerically\. Specifically, if we have access to the joint probability distributionf​\(𝒂,𝒂^s,𝒂^m,β\)f\(\\boldsymbol\{a\},\\boldsymbol\{\\hat\{a\}\}^\{s\},\\boldsymbol\{\\hat\{a\}\}^\{m\},\\beta\), we can derive closed\-form solutions for the optimization problem\. In practical settings, where exact knowledge of the distribution may be unavailable, we provide empirical solutions using a finite sample of data\.

### 3\.1Generative tasks \- Single\-threshold policy

First, we focus on generative tasks \(e\.g\., question answering tasks\)\. The output space𝒜\\mathcal\{A\}is an open\-ended set of sequences \(e\.g\., words, sentences, or structured responses\)\. For a given input query𝒒\\boldsymbol\{q\}, sLLM generates the text sequencesLLM​\(𝒒\)=𝒂^s∈𝒜\\text\{sLLM\}\(\\boldsymbol\{q\}\)=\\boldsymbol\{\\hat\{a\}\}^\{s\}\\in\\mathcal\{A\}\. Moreover, the associated confidence scoreβ∈\[0,1\]\\beta\\in\[0,1\]reflects its certainty about the whole generated text sequence\. According to the adopted policy, the prediction from sLLM is only accepted whenβ≥θ\\beta\\geq\\theta\. Regarding the cost of processing𝒒\\boldsymbol\{q\}, we always ‘pay’csc\_\{s\}and ‘pay’cm\(\>cs\)c\_\{m\}\(\>c\_\{s\}\)only when we defer to mLLM, i\.e\., whenβ<θ\\beta<\\theta\. Hence, we can define the cost for theii\-th query as:

c​\(βi,θ\)=cs\+π​\(βi,θ\)⋅cm=cs\+𝟙​\[βi<θ\]⋅cm\.c\(\\beta\_\{i\},\\theta\)=c\_\{s\}\+\\pi\(\\beta\_\{i\},\\theta\)\\cdot c\_\{m\}=c\_\{s\}\+\\mathds\{1\}\[\\beta\_\{i\}<\\theta\]\\cdot c\_\{m\}\\,\.\(2\)We can compute theexpectedcost, which depends only on how oftenβ\\betafalls below the threshold, as:

CostST​\(θ\)\\displaystyle\\text\{Cost\}\_\{\\text\{ST\}\}\(\\theta\)=cs\+cm​𝔼β​\[𝟙​\[β<θ\]\]\\displaystyle=c\_\{s\}\+c\_\{m\}\\mathbb\{E\}\_\{\\beta\}\\big\[\\mathds\{1\}\[\\beta<\\theta\]\\big\]\(3\)=cs\+cm​Pr⁡\(β<θ\)=cs\+cm​∫0θf​\(β\)​𝑑β\.\\displaystyle=c\_\{s\}\+c\_\{m\}\\Pr\(\\beta<\\theta\)=c\_\{s\}\+c\_\{m\}\\int\_\{0\}^\{\\theta\}f\(\\beta\)\\,d\\beta\\,\.Correspondingly, we can define the error formulations based on two separate settings\. In the first one, we assume a ‘teacher\-student’ paradigm where mLLM is considered as an expert model, an oracle \(the ‘teacher’\) so error can be only incurred from the small model whenβ≥θ\\beta\\geq\\theta\. In the second setting, both models are allowed to make mistakes\. Specifically, beginning with the oracle setting \(where we have that𝒂=𝒂^m\\boldsymbol\{\{a\}\}=\\boldsymbol\{\\hat\{a\}\}^\{m\}\), the query\-based error becomes:

e​\(𝒂^im,𝒂^is,βi,θ\)\\displaystyle e\(\\boldsymbol\{\\hat\{a\}\}^\{m\}\_\{i\},\\boldsymbol\{\\hat\{a\}\}^\{s\}\_\{i\},\\beta\_\{i\},\\theta\)=\(1−π​\(βi,θ\)\)​𝟙​\[𝒂^im≠𝒂^is\]\\displaystyle=\\left\(1\-\\pi\(\\beta\_\{i\},\\theta\)\\right\)\\mathds\{1\}\[\\boldsymbol\{\\hat\{a\}\}^\{m\}\_\{i\}\\neq\\boldsymbol\{\\hat\{a\}\}^\{s\}\_\{i\}\]\(4\)=𝟙​\[βi≥θ\]​𝟙​\[𝒂^im≠𝒂^is\],\\displaystyle=\\mathds\{1\}\[\\beta\_\{i\}\\geq\\theta\]\\mathds\{1\}\[\\boldsymbol\{\\hat\{a\}\}^\{m\}\_\{i\}\\neq\\boldsymbol\{\\hat\{a\}\}^\{s\}\_\{i\}\]\\,,where the binary error indicator𝟙​\[𝒂^im≠𝒂^is\]\\mathds\{1\}\[\\boldsymbol\{\\hat\{a\}\}^\{m\}\_\{i\}\\neq\\boldsymbol\{\\hat\{a\}\}^\{s\}\_\{i\}\]can be defined accordingly for generation tasks \(e\.g\., by using cosine similarity between the embeddings of the true and predicted answers and a thresholdδ\\deltato indicate a mismatch when the similarity scorecos⁡\(emb​\(𝒂^im\),emb​\(𝒂^is\)\)\\cos\(\\mathrm\{emb\}\(\\boldsymbol\{\\hat\{a\}\}^\{m\}\_\{i\}\),\\mathrm\{emb\}\(\\boldsymbol\{\\hat\{a\}\}^\{s\}\_\{i\}\)\)falls belowδ\\delta\)\. We can also compute theexpectederror111‘ST\-O’: Single\-Threshold Oracleas:

ErrorST\-O​\(θ\)=𝔼𝒂,𝒂^m,𝒂^s,β​\[𝟙​\[β≥θ\]​𝟙​\[𝒂^m≠𝒂^s\]\]=∫01∑𝒂,𝒂^m,𝒂^s𝟙​\[β≥θ\]​𝟙​\[𝒂^m≠𝒂^s\]​f​\(𝒂,𝒂^m,𝒂^s,β\)​d​β=∫01∑𝒂^m,𝒂^s𝟙​\[β≥θ\]​𝟙​\[𝒂^m≠𝒂^s\]​\(∑𝒂f​\(𝒂,𝒂^m,𝒂^s,β\)\)​d​β=∫01∑𝒂^m,𝒂^s𝟙​\[β≥θ\]​𝟙​\[𝒂^m≠𝒂^s\]​f​\(𝒂^m,𝒂^s,β\)​d​β\.\\text\{Error\}\_\{\\text\{ST\-O\}\}\(\\theta\)=\\mathbb\{E\}\_\{\\boldsymbol\{a\},\\boldsymbol\{\\hat\{a\}\}^\{m\},\\boldsymbol\{\\hat\{a\}\}^\{s\},\\beta\}\\big\[\\mathds\{1\}\[\\beta\\geq\\theta\]\\mathds\{1\}\[\\boldsymbol\{\\hat\{a\}\}^\{m\}\\neq\\boldsymbol\{\\hat\{a\}\}^\{s\}\]\\big\]=\\\\ \\int\_\{0\}^\{1\}\\sum\_\{\\boldsymbol\{a\},\\boldsymbol\{\\hat\{a\}\}^\{m\},\\boldsymbol\{\\hat\{a\}\}^\{s\}\}\\mathds\{1\}\[\\beta\\geq\\theta\]\\mathds\{1\}\[\\boldsymbol\{\\hat\{a\}\}^\{m\}\\neq\\boldsymbol\{\\hat\{a\}\}^\{s\}\]f\(\\boldsymbol\{a\},\\boldsymbol\{\\hat\{a\}\}^\{m\},\\boldsymbol\{\\hat\{a\}\}^\{s\},\\beta\)\\,d\\beta=\\\\ \\int\_\{0\}^\{1\}\\sum\_\{\\boldsymbol\{\\hat\{a\}\}^\{m\},\\boldsymbol\{\\hat\{a\}\}^\{s\}\}\\mathds\{1\}\[\\beta\\geq\\theta\]\\mathds\{1\}\[\\boldsymbol\{\\hat\{a\}\}^\{m\}\\neq\\boldsymbol\{\\hat\{a\}\}^\{s\}\]\\Big\(\\sum\_\{\\boldsymbol\{a\}\}f\(\\boldsymbol\{a\},\\boldsymbol\{\\hat\{a\}\}^\{m\},\\boldsymbol\{\\hat\{a\}\}^\{s\},\\beta\)\\Big\)d\\beta=\\\\ \\int\_\{0\}^\{1\}\\sum\_\{\\boldsymbol\{\\hat\{a\}\}^\{m\},\\boldsymbol\{\\hat\{a\}\}^\{s\}\}\\mathds\{1\}\[\\beta\\geq\\theta\]\\mathds\{1\}\[\\boldsymbol\{\\hat\{a\}\}^\{m\}\\neq\\boldsymbol\{\\hat\{a\}\}^\{s\}\]f\(\\boldsymbol\{\\hat\{a\}\}^\{m\},\\boldsymbol\{\\hat\{a\}\}^\{s\},\\beta\)\\,d\\beta\\,\.\(5\)
Using the chain rule of probability, we can split the joint distribution asf​\(𝒂^m,𝒂^s,β\)=f​\(β\)⋅f​\(𝒂^m,𝒂^s∣β\)f\(\\boldsymbol\{\\hat\{a\}\}^\{m\},\\boldsymbol\{\\hat\{a\}\}^\{s\},\\beta\)=f\(\\beta\)\\cdot f\(\\boldsymbol\{\\hat\{a\}\}^\{m\},\\boldsymbol\{\\hat\{a\}\}^\{s\}\\mid\\beta\)and thus write \([5](https://arxiv.org/html/2609.15992#S3.E5)\) as:

ErrorST\-O​\(θ\)=∫01𝟙​\[β≥θ\]​f​\(β\)​\[∑𝒂^m,𝒂^s𝟙​\[𝒂^m≠𝒂^s\]​f​\(𝒂^m,𝒂^s∣β\)\]​𝑑β\.\\text\{Error\}\_\{\\text\{ST\-O\}\}\(\\theta\)=\\\\ \\int\_\{0\}^\{1\}\\mathds\{1\}\[\\beta\\geq\\theta\]f\(\\beta\)\\Big\[\\sum\_\{\\boldsymbol\{\\hat\{a\}\}^\{m\},\\boldsymbol\{\\hat\{a\}\}^\{s\}\}\\mathds\{1\}\[\\boldsymbol\{\\hat\{a\}\}^\{m\}\\neq\\boldsymbol\{\\hat\{a\}\}^\{s\}\]f\(\\boldsymbol\{\\hat\{a\}\}^\{m\},\\boldsymbol\{\\hat\{a\}\}^\{s\}\\mid\\beta\)\\Big\]d\\beta\\,\.\(6\)To simplify this expression, we can use the definition of conditional probability, the expectation of indicator functions \(which is equal to the probability of the event\)222Pr⁡\(A\)=𝔼​\[𝟙​\(A\)\]\\Pr\(A\)=\\mathbb\{E\}\[\\mathds\{1\}\(A\)\]and the law of total expectation333𝔼​\[Z\]=𝔼​\[𝔼​\[Z∣β\]\]\\mathbb\{E\}\[Z\]=\\mathbb\{E\}\[\\mathbb\{E\}\[Z\\mid\\beta\]\]withZ=𝟙​\[β≥θ\]​𝟙​\[𝒂≠𝒂^\]Z=\\mathds\{1\}\[\\beta\\geq\\theta\]\\mathds\{1\}\[\\boldsymbol\{a\}\\neq\\boldsymbol\{\\hat\{a\}\}\]to acquire:

ErrorST\-O​\(θ\)=∫θ1εs​\(β\)​f​\(β\)​𝑑β,\\text\{Error\}\_\{\\text\{ST\-O\}\}\(\\theta\)=\\int\_\{\\theta\}^\{1\}\\varepsilon\_\{s\}\(\\beta\)f\(\\beta\)\\,d\\beta\\,,\(7\)whereεs​\(β\)≔Pr⁡\(𝒂^m≠𝒂^s∣β\)=∑𝒂^m∑𝒂^s𝟙​\[𝒂^m≠𝒂^s\]​f​\(𝒂^m,𝒂^s∣β\)\\varepsilon\_\{s\}\(\\beta\)\\coloneq\\Pr\(\\boldsymbol\{\\hat\{a\}\}^\{m\}\\neq\\boldsymbol\{\\hat\{a\}\}^\{s\}\\mid\\beta\)=\\sum\_\{\\boldsymbol\{\\hat\{a\}\}^\{m\}\}\\sum\_\{\\boldsymbol\{\\hat\{a\}\}^\{s\}\}\\mathds\{1\}\[\\boldsymbol\{\\hat\{a\}\}^\{m\}\\neq\\boldsymbol\{\\hat\{a\}\}^\{s\}\]f\(\\boldsymbol\{\\hat\{a\}\}^\{m\},\\boldsymbol\{\\hat\{a\}\}^\{s\}\\mid\\beta\), which is the computation of the expectation of the indicator under the joint conditional distribution\. So, the optimization objective that has to be solved reduces to:

minθ∈\[0,1\]\\displaystyle\\min\_\{\\theta\\in\[0,1\]\}CostST​\(θ\)\\displaystyle\\text\{Cost\}\_\{\\text\{ST\}\}\(\\theta\)\(8\)s\.t\.ErrorST\-O​\(θ\)≤1−ξ,\\displaystyle\\text\{Error\}\_\{\\text\{ST\-O\}\}\(\\theta\)\\leq 1\-\\xi\\,,whereξ∈\[0,1\]\\xi\\in\[0,1\]is a desired value for a task\-specific metric that measures performance\.

SinceCostST​\(θ\)\\text\{Cost\}\_\{\\text\{ST\}\}\(\\theta\)is non\-decreasing inθ\\theta, the optimal valueθ∗\\theta^\{\*\}is thesmallestthat satisfies the constraint:

θ∗=inf\{θ∈\[0,1\]\|∫θ1εs​\(β\)​f​\(β\)≤1−ξ\}\.\\theta^\{\*\}=\\inf\\left\\\{\\theta\\in\[0,1\]\\,\\,\\bigg\|\\int\_\{\\theta\}^\{1\}\\varepsilon\_\{s\}\(\\beta\)f\(\\beta\)\\leq 1\-\\xi\\right\\\}\\,\.\(9\)Under the \(mild\) assumptions thatεs​\(β\)∈\[0,1\]\\varepsilon\_\{s\}\(\\beta\)\\in\[0,1\]andf​\(β\)≥0f\(\\beta\)\\geq 0, \([9](https://arxiv.org/html/2609.15992#S3.E9)\) obtains the optimal threshold\. We present the main theoretical results below\. The complete proofs, together with an illustrative example admitting closed\-form expressions, are provided in Appendix[B\.1](https://arxiv.org/html/2609.15992#A2.SS1)\.444Here, optimality is defined with respect to a threshold\-based policy\. Under mild assumptions, anyβ\\beta\-based rule can be reduced to a threshold policy; see Appendix[D](https://arxiv.org/html/2609.15992#A4)for details\.

###### Lemma 3\.1\(Monotonicity of cost and error in the oracle case\)\.

Letf:\[0,1\]→\[0,∞\)f:\[0,1\]\\to\[0,\\infty\)be integrable with CDFF​\(θ\)=∫0θf​\(β\)​𝑑βF\(\\theta\)=\\int\_\{0\}^\{\\theta\}f\(\\beta\)\\,d\\beta\. Letε:\[0,1\]→\[0,1\]\\varepsilon:\[0,1\]\\to\[0,1\]be any measurable function\. Forθ∈\[0,1\]\\theta\\in\[0,1\],

C​\(θ\)=cs\+cm​F​\(θ\),g​\(θ\)=∫θ1ε​\(β\)​f​\(β\)​𝑑β,C\(\\theta\)=c\_\{s\}\+c\_\{m\}\\,F\(\\theta\),\\qquad g\(\\theta\)=\\int\_\{\\theta\}^\{1\}\\varepsilon\(\\beta\)\\,f\(\\beta\)\\,d\\beta,\(10\)with constantscs\>0c\_\{s\}\>0,cm\>0c\_\{m\}\>0\. ThenC​\(⋅\)C\(\\cdot\)is non\-decreasing andg​\(⋅\)g\(\\cdot\)is non\-increasing in\[0,1\]\[0,1\]\.

###### Proof\.

See Appendix[B\.1](https://arxiv.org/html/2609.15992#A2.SS1)∎

###### Theorem 3\.2\(Optimal threshold in the oracle case\)\.

Fix an error budgetb∈\[0,1\]b\\in\[0,1\]and consider

minθ∈\[0,1\]⁡C​\(θ\)s\.t\.g​\(θ\)≤b,\\min\_\{\\theta\\in\[0,1\]\}C\(\\theta\)\\quad\\text\{s\.t\.\}\\quad g\(\\theta\)\\leq b,withC​\(⋅\)C\(\\cdot\)andg​\(⋅\)g\(\\cdot\)as in Lemma[3\.1](https://arxiv.org/html/2609.15992#S3.Thmtheorem1)\. Letθ∗=inf\{θ∈\[0,1\]:g​\(θ\)≤b\}\\theta^\{\*\}=\\inf\\\{\\theta\\in\[0,1\]:g\(\\theta\)\\leq b\\\}\. Then, if the problem is feasible, every optimal solution is attained at the leftmost feasible threshold, i\.e\.,θopt=θ∗\\theta^\{\\mathrm\{opt\}\}=\\theta^\{\*\}\. Moreover, ifCCis strictly increasing, then the optimizer is unique\. If, in addition,ggis strictly decreasing, then the constraint is tight at optimum andθ∗\\theta^\{\*\}is the unique solution ofg​\(θ\)=bg\(\\theta\)=b\.

###### Proof\.

See Appendix[B\.1](https://arxiv.org/html/2609.15992#A2.SS1)\. ∎

##### Intuition of Theorem[3\.2](https://arxiv.org/html/2609.15992#S3.Thmtheorem2)\.

Increasingθ\\thetameans deferring more often to mLLM\. This always increases cost, but it can only decrease error \(Lemma[3\.1](https://arxiv.org/html/2609.15992#S3.Thmtheorem1)\)\. Therefore, the best strategy is simple: increaseθ\\thetaonly as much as needed to satisfy the target error budget, and stop at the*smallest feasible threshold*\.

We now drop the oracle assumption and allow mLLM to make mistakes\. The mLLM model has its own probability of errorεm​\(γ\)\\varepsilon\_\{m\}\(\\gamma\)\. To simplify our analysis, we assume that the error rates of the two models are independent so we can ‘compress’ the error curve of mLLM into a constantε¯m\\bar\{\\varepsilon\}\_\{m\}555ε¯m=𝔼​\[εm​\(γ\)∣β\]=𝔼γ​\[εm​\(γ\)\]=∫01εm​\(γ\)​fm​\(γ\)​𝑑γ\\bar\{\\varepsilon\}\_\{m\}=\\mathbb\{E\}\\big\[\\varepsilon\_\{m\}\(\\gamma\)\\mid\\beta\\big\]=\\mathbb\{E\}\_\{\\gamma\}\\big\[\\varepsilon\_\{m\}\(\\gamma\)\\big\]=\\int\_\{0\}^\{1\}\\varepsilon\_\{m\}\(\\gamma\)f\_\{m\}\(\\gamma\)\\,d\\gamma\.\. While the cost remains the same as in the oracle case, the query\-based error now becomes:

e​\(𝒂i,𝒂^is,𝒂^im,βi,θ\)=\(1−π​\(βi,θ\)\)​𝟙​\[𝒂i≠𝒂^is\]\+π​\(βi,θ\)​𝟙​\[𝒂i≠𝒂^im\]=𝟙​\[βi≥θ\]​𝟙​\[𝒂i≠𝒂^is\]\+𝟙​\[βi<θ\]​𝟙​\[𝒂i≠𝒂^im\],e\(\\boldsymbol\{a\}\_\{i\},\\boldsymbol\{\\hat\{a\}\}^\{s\}\_\{i\},\\boldsymbol\{\\hat\{a\}\}^\{m\}\_\{i\},\\beta\_\{i\},\\theta\)=\\\\ \\begin\{gathered\}\\left\(1\-\\pi\(\\beta\_\{i\},\\theta\)\\right\)\\mathds\{1\}\[\\boldsymbol\{a\}\_\{i\}\\neq\\boldsymbol\{\\hat\{a\}\}^\{s\}\_\{i\}\]\+\\pi\(\\beta\_\{i\},\\theta\)\\mathds\{1\}\[\\boldsymbol\{a\}\_\{i\}\\neq\\boldsymbol\{\\hat\{a\}\}^\{m\}\_\{i\}\]=\\\\ \\mathds\{1\}\[\\beta\_\{i\}\\geq\\theta\]\\mathds\{1\}\[\\boldsymbol\{a\}\_\{i\}\\neq\\boldsymbol\{\\hat\{a\}\}^\{s\}\_\{i\}\]\+\\mathds\{1\}\[\\beta\_\{i\}<\\theta\]\\mathds\{1\}\[\\boldsymbol\{a\}\_\{i\}\\neq\\boldsymbol\{\\hat\{a\}\}^\{m\}\_\{i\}\]\\,,\\end\{gathered\}\(11\)while, following the same steps as in \([5](https://arxiv.org/html/2609.15992#S3.E5)\), the expected error becomes:

ErrorST\-NO​\(θ\)=∫θ1εs​\(β\)​f​\(β\)​𝑑β\+ε¯m​∫0θf​\(β\)​𝑑β\.\\text\{Error\}\_\{\\text\{ST\-NO\}\}\(\\theta\)=\\int\_\{\\theta\}^\{1\}\\varepsilon\_\{s\}\(\\beta\)f\(\\beta\)\\,d\\beta\+\\bar\{\\varepsilon\}\_\{m\}\\int\_\{0\}^\{\\theta\}f\(\\beta\)\\,d\\beta\\,\.\(12\)Note that in the non\-oracle case the slope of the error curve can be positive or negative, depending on whether sLLM is locally better or worse than mLLM at score levelθ\\theta\.666The error function is continuous \(proof in Appendix[E](https://arxiv.org/html/2609.15992#A5)\) and the derivative isdd​θ​ErrorNO​\(θ\)=−\(εs​\(θ\)−ε¯m\)​f​\(θ\)\\frac\{\\mathrm\{d\}\}\{\\mathrm\{d\}\\theta\}\\text\{Error\}\_\{\\text\{NO\}\}\(\\theta\)=\-\(\\varepsilon\_\{s\}\(\\theta\)\-\\bar\{\\varepsilon\}\_\{m\}\)f\(\\theta\)\.Thus, unlike in the oracle case,ErrorST\-NO​\(θ\)\\text\{Error\}\_\{\\text\{ST\-NO\}\}\(\\theta\)need not be monotone inθ\\theta\.

The constraint of the optimization problem in \([8](https://arxiv.org/html/2609.15992#S3.E8)\) is replaced by \([12](https://arxiv.org/html/2609.15992#S3.E12)\) and the solution of \([9](https://arxiv.org/html/2609.15992#S3.E9)\) still provides the optimal threshold\. As previously, we present the theorem’s results here and leave the proof in Appendix[B\.1](https://arxiv.org/html/2609.15992#A2.SS1)

###### Theorem 3\.3\(Optimal threshold in the non\-oracle case\)\.

Fix an error budgetb∈\[0,1\]b\\in\[0,1\]and consider

minθ∈\[0,1\]⁡C​\(θ\)s\.t\.gNO​\(θ\)≤b,\\min\_\{\\theta\\in\[0,1\]\}C\(\\theta\)\\quad\\text\{s\.t\.\}\\quad g\_\{\\mathrm\{NO\}\}\(\\theta\)\\leq b,whereC​\(⋅\)C\(\\cdot\)as in Lemma[3\.1](https://arxiv.org/html/2609.15992#S3.Thmtheorem1)is non\-decreasing andgNO​\(⋅\)=ErrorST\-NO​\(⋅\)g\_\{\\mathrm\{NO\}\}\(\\cdot\)=\\text\{Error\}\_\{\\text\{ST\-NO\}\}\(\\cdot\)\. Defineℱb=\{θ∈\[0,1\]:gNO​\(θ\)≤b\}\\mathcal\{F\}\_\{b\}=\\\{\\theta\\in\[0,1\]:g\_\{\\mathrm\{NO\}\}\(\\theta\)\\leq b\\\}andθ∗=infℱb\\theta^\{\*\}=\\inf\\mathcal\{F\}\_\{b\}\. Assumeℱb≠∅\\mathcal\{F\}\_\{b\}\\neq\\emptyset\. Then: \(i\)θ∗\\theta^\{\*\}is an optimal solution, even ifℱb\\mathcal\{F\}\_\{b\}is a disjoint union of intervals; \(ii\) ifgNO​\(0\)≤bg\_\{\\mathrm\{NO\}\}\(0\)\\leq b, thenθ∗=0\\theta^\{\*\}=0; \(iii\) ifgNO​\(0\)\>bg\_\{\\mathrm\{NO\}\}\(0\)\>b, thenθ∗\>0\\theta^\{\*\}\>0and the constraint is tight:gNO​\(θ∗\)=bg\_\{\\mathrm\{NO\}\}\(\\theta^\{\*\}\)=b\.

###### Proof\.

See Appendix[B\.2](https://arxiv.org/html/2609.15992#A2.SS2)\. ∎

##### Intuition of Theorem[3\.3](https://arxiv.org/html/2609.15992#S3.Thmtheorem3)\.

The error function is not necessarily monotone inθ\\theta, andℱb\\mathcal\{F\}\_\{b\}can be a union of disjoint intervals\. Theorem[3\.3](https://arxiv.org/html/2609.15992#S3.Thmtheorem3)shows that, within the single\-threshold family, the optimal threshold is still the smallest feasible one, purely becauseC​\(⋅\)C\(\\cdot\)is non\-decreasing inθ\\theta\.

### 3\.2Discriminative tasks \- Multi\-thresholds policy

We now focus on discriminative tasks \(e\.g\., classification\) where LLMs can be used to provide answers regarding specific categories\. We considerKK\-way classification tasks where𝒜=\{1,…,K\}\\mathcal\{A\}=\\\{1,\\ldots,K\\\}and now, for a sample𝒒\\boldsymbol\{q\}, sLLM outputs the predicted labelsLLM​\(𝒒\)=𝒂^s∈𝒜\\text\{sLLM\}\(\\boldsymbol\{q\}\)=\\boldsymbol\{\\hat\{a\}\}^\{s\}\\in\\mathcal\{A\}and avectorof confidence scores𝜷∈ΔK−1=\{𝜷∈\[0,1\]K∣∑j=1Kβj=1\}\\boldsymbol\{\\beta\}\\in\\Delta^\{K\-1\}=\\left\\\{\\boldsymbol\{\\beta\}\\in\[0,1\]^\{K\}\\mid\\sum\_\{j=1\}^\{K\}\\beta\_\{j\}=1\\right\\\}, whereβj\\beta\_\{j\}is the confidence score that sLLM assigns to thejj\-th class\. Consequently, the standard approach is to use the most confident class as the predicted one, i\.e\.,𝒂^=arg⁡maxj=1,…,K⁡βj\\boldsymbol\{\\hat\{a\}\}=\\arg\\max\_\{j=1,\\dots,K\}\\beta\_\{j\}\[chow1970\]\. In realistic classification settings \(especially with class imbalance or asymmetric class difficulty\), performance over different classes can vary and models can display a different behavior per class\. As a result, and to further generalize our framework, we extend the decision policy to a multi\-threshold one that is more flexible and can adjust to class\-specific calibration and difficulty, e\.g\., one might want to trust easier\-to\-classify classes more and be more cautious with harder classes\. We now consider a threshold vector𝜽=\(θ1,…,θK\)\\boldsymbol\{\\theta\}=\(\\theta\_\{1\},\\ldots,\\theta\_\{K\}\), whereθk\\theta\_\{k\}is the threshold associated to thekk\-th class\. Therefore, the prediction from sLLM is now only accepted whenβk≥θk\\beta\_\{k\}\\geq\\theta\_\{k\}\. Thus, we can define the cost for theii\-th query as:

c​\(βi,k,θ\)=cs\+π​\(βi,k,θk\)⋅cm=cs\+𝟙​\[βi,k<θk\]⋅cm,c\(\\beta\_\{i,k\},\\theta\)=c\_\{s\}\+\\pi\(\\beta\_\{i,k\},\\theta\_\{k\}\)\\cdot c\_\{m\}=c\_\{s\}\+\\mathds\{1\}\[\\beta\_\{i,k\}<\\theta\_\{k\}\]\\cdot c\_\{m\}\\,,\(13\)and theexpectedcost is expressed as:

CostMT​\(𝜽\)=cs\+cm​∑k=1Kpk​∫0θkfk​\(β\)​𝑑β,\\text\{Cost\}\_\{\\text\{MT\}\}\(\\boldsymbol\{\\theta\}\)=c\_\{s\}\+c\_\{m\}\\sum\_\{k=1\}^\{K\}p\_\{k\}\\int\_\{0\}^\{\\theta\_\{k\}\}f\_\{k\}\(\\beta\)\\,d\\beta\\,,\(14\)where𝒑∈ΔK−1=\{𝒑∈\[0,1\]K∣∑j=1Kpj=1\}\\boldsymbol\{p\}\\in\\Delta^\{K\-1\}=\\left\\\{\\boldsymbol\{p\}\\in\[0,1\]^\{K\}\\mid\\sum\_\{j=1\}^\{K\}p\_\{j\}=1\\right\\\}are \(predicted\) class priors\. Following the two different settings, we will again provide different formulations for the error\. In the oracle setting, the error for theii\-th query can be defined as:

e​\(𝒂^im,𝒂^is,βi,k,θk\)\\displaystyle e\(\\boldsymbol\{\\hat\{a\}\}^\{m\}\_\{i\},\\boldsymbol\{\\hat\{a\}\}^\{s\}\_\{i\},\\beta\_\{i,k\},\\theta\_\{k\}\)=\(1−π​\(βi,k,θk\)\)​𝟙​\[𝒂^im≠𝒂^is\]\\displaystyle=\\left\(1\-\\pi\(\\beta\_\{i,k\},\\theta\_\{k\}\)\\right\)\\mathds\{1\}\[\\boldsymbol\{\\hat\{a\}\}^\{m\}\_\{i\}\\neq\\boldsymbol\{\\hat\{a\}\}^\{s\}\_\{i\}\]\(15\)=𝟙​\[βi,k≥θk\]​𝟙​\[𝒂^im≠𝒂^is\]\.\\displaystyle=\\mathds\{1\}\[\\beta\_\{i,k\}\\geq\\theta\_\{k\}\]\\mathds\{1\}\[\\boldsymbol\{\\hat\{a\}\}^\{m\}\_\{i\}\\neq\\boldsymbol\{\\hat\{a\}\}^\{s\}\_\{i\}\]\\,\.Furthermore, we defineεs,k​\(β\)=Pr⁡\(𝒂^m≠k∣𝒂^s=k,βk=β\),∀k=1,…,K\\varepsilon\_\{s,k\}\(\\beta\)=\\Pr\(\\boldsymbol\{\\hat\{a\}\}^\{m\}\\neq k\\mid\\boldsymbol\{\\hat\{a\}\}^\{s\}=k,\\beta\_\{k\}=\\beta\),\\,\\forall\\,k=1,\\ldots,Kand the expected error becomes:

ErrorMT\-O​\(𝜽\)=∑k=1Kpk​∫θk1εs,k​\(β\)​fk​\(β\)​𝑑β\.\\text\{Error\}\_\{\\text\{MT\-O\}\}\(\\boldsymbol\{\\theta\}\)=\\sum\_\{k=1\}^\{K\}p\_\{k\}\\int\_\{\\theta\_\{k\}\}^\{1\}\\varepsilon\_\{s,k\}\(\\beta\)f\_\{k\}\(\\beta\)\\,d\\beta\\,\.\(16\)Consequently, the optimization objective now becomes:

min𝜽∈\[0,1\]K\\displaystyle\\min\_\{\\boldsymbol\{\\theta\}\\in\[0,1\]^\{K\}\}CostMT​\(𝜽\)\\displaystyle\\text\{Cost\}\_\{\\text\{MT\}\}\(\\boldsymbol\{\\theta\}\)\(17\)s\.t\.ErrorMT\-O​\(𝜽\)≤1−ξ\.\\displaystyle\\text\{Error\}\_\{\\text\{MT\-O\}\}\(\\boldsymbol\{\\theta\}\)\\leq 1\-\\xi\\,\.The following theorem shows that for somet∈ℝt\\in\\mathbb\{R\}, any optimal threshold vector must satisfyεs,1​\(θ1∗\)=…=εs,K​\(θK∗\)=t\\varepsilon\_\{s,1\}\(\\theta^\{\*\}\_\{1\}\)=\\ldots=\\varepsilon\_\{s,K\}\(\\theta^\{\*\}\_\{K\}\)=tunder the assumptions thatεs,k​\(β\)∈\[0,1\]\\varepsilon\_\{s,k\}\(\\beta\)\\in\[0,1\]andfk​\(β\)≥0,∀k=1,…,Kf\_\{k\}\(\\beta\)\\geq 0,\\,\\forall\\,k=1,\\ldots,K\. The proof can be found in Appendix[B\.3](https://arxiv.org/html/2609.15992#A2.SS3)\.

###### Lemma 3\.4\(Coordinate\-wise monotonicity in the oracle case\)\.

For each classk∈\{1,…,K\}k\\in\\\{1,\\dots,K\\\}letfk:\[0,1\]→\[0,∞\)f\_\{k\}:\[0,1\]\\to\[0,\\infty\)be integrable with CDFFk​\(θ\)=∫0θfk​\(β\)​𝑑βF\_\{k\}\(\\theta\)=\\int\_\{0\}^\{\\theta\}f\_\{k\}\(\\beta\)\\,d\\beta, and letεk:\[0,1\]→\[0,1\]\\varepsilon\_\{k\}:\[0,1\]\\to\[0,1\]be any measurable function\. Let alsopk≥0p\_\{k\}\\geq 0with∑kpk=1\\sum\_\{k\}p\_\{k\}=1, and constantscs,cm≥0c\_\{s\},c\_\{m\}\\geq 0\. For a threshold vector𝛉=\(θ1,…,θK\)∈\[0,1\]K\\boldsymbol\{\\theta\}=\(\\theta\_\{1\},\\dots,\\theta\_\{K\}\)\\in\[0,1\]^\{K\}define:

C​\(𝜽\)\\displaystyle C\(\\boldsymbol\{\\theta\}\)=cs\+cm​∑k=1Kpk​Fk​\(θk\),\\displaystyle=c\_\{s\}\+c\_\{m\}\\sum\_\{k=1\}^\{K\}p\_\{k\}\\,F\_\{k\}\(\\theta\_\{k\}\)\\,,g​\(𝜽\)\\displaystyle g\(\\boldsymbol\{\\theta\}\)=∑k=1Kpk​∫θk1εk​\(β\)​fk​\(β\)​𝑑β\.\\displaystyle=\\sum\_\{k=1\}^\{K\}p\_\{k\}\\int\_\{\\theta\_\{k\}\}^\{1\}\\varepsilon\_\{k\}\(\\beta\)\\,f\_\{k\}\(\\beta\)\\,d\\beta\\,\.Then, for each coordinatekk,C​\(⋅\)C\(\\cdot\)is non\-decreasing inθk\\theta\_\{k\}andg​\(⋅\)g\(\\cdot\)is non\-increasing inθk\\theta\_\{k\}\. Consequently, for any error budgetb∈\[0,1\]b\\in\[0,1\], the feasible setℱb=\{𝛉∈\[0,1\]K:g​\(𝛉\)≤b\}\\mathcal\{F\}\_\{b\}=\\\{\\boldsymbol\{\\theta\}\\in\[0,1\]^\{K\}:\\penalty 10000\\ g\(\\boldsymbol\{\\theta\}\)\\leq b\\\}is an*upper\-orthant*: if𝛉∈ℱb\\boldsymbol\{\\theta\}\\in\\mathcal\{F\}\_\{b\}and𝛉′≥𝛉\\boldsymbol\{\\theta\}^\{\\prime\}\\geq\\boldsymbol\{\\theta\}coordinate\-wise, then𝛉′∈ℱb\\boldsymbol\{\\theta\}^\{\\prime\}\\in\\mathcal\{F\}\_\{b\}\.

###### Proof\.

See Appendix[B\.3](https://arxiv.org/html/2609.15992#A2.SS3)∎

###### Theorem 3\.5\(Optimal threshold vector in the oracle case\)\.

Consider the constrained problem

min𝜽∈\[0,1\]K⁡C​\(𝜽\)s\.t\.g​\(𝜽\)≤b,\\min\_\{\\boldsymbol\{\\theta\}\\in\[0,1\]^\{K\}\}\\penalty 10000\\ C\(\\boldsymbol\{\\theta\}\)\\qquad\\text\{s\.t\.\}\\qquad g\(\\boldsymbol\{\\theta\}\)\\leq b,\(18\)withC​\(⋅\),g​\(⋅\)C\(\\cdot\),g\(\\cdot\)as in Lemma[3\.4](https://arxiv.org/html/2609.15992#S3.Thmtheorem4)andb∈\[0,1\]b\\in\[0,1\]\.

Then: \(i\) There exists an optimal solution𝛉∗\\boldsymbol\{\\theta\}^\{\*\}with tight constraintg​\(𝛉∗\)=bg\(\\boldsymbol\{\\theta\}^\{\*\}\)=b, unless𝛉=𝟎\\boldsymbol\{\\theta\}=\\boldsymbol\{0\}is already feasible \(in which case𝛉∗=𝟎\\boldsymbol\{\\theta\}^\{\*\}=\\boldsymbol\{0\}\), \(ii\) At any optimal solution with coordinates in the interior ofℱb\\mathcal\{F\}\_\{b\}\(i\.e\., not forced to0or11\), the boundary errors are equalized:εs,1​\(θ1∗\)=…=εs,K​\(θK∗\)=t\\varepsilon\_\{s,1\}\(\\theta\_\{1\}^\{\*\}\)=\\ldots=\\varepsilon\_\{s,K\}\(\\theta\_\{K\}^\{\*\}\)=tand \(iii\) for each interior classkk, we can takeθk∗∈εs,k−1​\(t\)\\theta\_\{k\}^\{\*\}\\in\\varepsilon\_\{s,k\}^\{\-1\}\(t\), withttdetermined byg​\(𝛉∗\)=bg\(\\boldsymbol\{\\theta\}^\{\*\}\)=b\.

###### Proof\.

See Appendix[B\.3](https://arxiv.org/html/2609.15992#A2.SS3)\. ∎

##### Intuition of Theorem[3\.5](https://arxiv.org/html/2609.15992#S3.Thmtheorem5)\.

If one class is being deferred at a boundary point where the sLLM is much worse than another class at its boundary, then we can reallocate deferrals across classes and reduce cost without violating the error budget\. Thus, the optimum equalizes the ‘boundary errors’ across all interior classes, while easy classes saturate at always\-accept and difficult classes may saturate at always\-defer\.777We again focus our proof on the threshold\-based policy; detailed proof for more general cases are provided in Appendix[C](https://arxiv.org/html/2609.15992#A3)\.

We now move to the non\-oracle setting, where mLLM has a non\-zero error probability\. The cost again remains the same as in the oracle case \([14](https://arxiv.org/html/2609.15992#S3.E14)\) while the sample\-based error now becomes:

e​\(𝒂i,𝒂^is,𝒂^im,βi,k,θk\)=\(1−π​\(βi,k,θk\)\)​𝟙​\[𝒂i≠𝒂^is\]\+π​\(βi,k,θk\)​𝟙​\[𝒂i≠𝒂^im\]=𝟙​\[βi,k≥θk\]​𝟙​\[𝒂i≠𝒂^is\]\+𝟙​\[βi,k<θk\]​𝟙​\[𝒂i≠𝒂^im\]\.e\(\\boldsymbol\{a\}\_\{i\},\\boldsymbol\{\\hat\{a\}\}^\{s\}\_\{i\},\\boldsymbol\{\\hat\{a\}\}^\{m\}\_\{i\},\\beta\_\{i,k\},\\theta\_\{k\}\)=\\\\ \\begin\{gathered\}\\left\(1\-\\pi\(\\beta\_\{i,k\},\\theta\_\{k\}\)\\right\)\\mathds\{1\}\[\\boldsymbol\{a\}\_\{i\}\\neq\\boldsymbol\{\\hat\{a\}\}^\{s\}\_\{i\}\]\+\\pi\(\\beta\_\{i,k\},\\theta\_\{k\}\)\\mathds\{1\}\[\\boldsymbol\{a\}\_\{i\}\\neq\\boldsymbol\{\\hat\{a\}\}^\{m\}\_\{i\}\]=\\\\ \\mathds\{1\}\[\\beta\_\{i,k\}\\geq\\theta\_\{k\}\]\\mathds\{1\}\[\\boldsymbol\{a\}\_\{i\}\\neq\\boldsymbol\{\\hat\{a\}\}^\{s\}\_\{i\}\]\+\\mathds\{1\}\[\\beta\_\{i,k\}<\\theta\_\{k\}\]\\mathds\{1\}\[\\boldsymbol\{a\}\_\{i\}\\neq\\boldsymbol\{\\hat\{a\}\}^\{m\}\_\{i\}\]\\,\.\\end\{gathered\}\(19\)We again assume the mLLM’s error rate is independent of the sLLM’s outputs, so the error curve can be summarized by a constantε¯m,k∈\[0,1\]\\bar\{\\varepsilon\}\_\{m,k\}\\in\[0,1\]and the expected error becomes:

ErrorMT\-NO​\(θ\)=𝔼​\[e​\(𝒂i,𝒂^is,𝒂^im,βi,k,θk\)\]=∑k=1Kpk​\(∫θk1εs,k​\(β\)​fk​\(β\)​𝑑β\+ε¯m,k​∫0θkfk​\(β\)​𝑑β\)\.\\text\{Error\}\_\{\\text\{MT\-NO\}\}\(\\theta\)=\\mathbb\{E\}\\big\[e\(\\boldsymbol\{a\}\_\{i\},\\boldsymbol\{\\hat\{a\}\}^\{s\}\_\{i\},\\boldsymbol\{\\hat\{a\}\}^\{m\}\_\{i\},\\beta\_\{i,k\},\\theta\_\{k\}\)\\big\]=\\\\ \\begin\{gathered\}\\sum\_\{k=1\}^\{K\}p\_\{k\}\\left\(\\int\_\{\\theta\_\{k\}\}^\{1\}\\varepsilon\_\{s,k\}\(\\beta\)\\,f\_\{k\}\(\\beta\)\\,d\\beta\\;\+\\;\\bar\{\\varepsilon\}\_\{m,k\}\\int\_\{0\}^\{\\theta\_\{k\}\}f\_\{k\}\(\\beta\)\\,d\\beta\\right\)\\,\.\\end\{gathered\}\(20\)The constraint of the optimization problem in \([17](https://arxiv.org/html/2609.15992#S3.E17)\) is replaced by \([20](https://arxiv.org/html/2609.15992#S3.E20)\)\. In contrast to the oracle case, the error in the non\-oracle setting is not guaranteed to be coordinate\-wise non\-increasing in eachθk\\theta\_\{k\}, because increasing a threshold both removes sLLM errors and can add mLLM errors\. Consequently, the feasible setℱb\\mathcal\{F\}\_\{b\}need not be an upper\-orthant; it can be a more complicated \(even disjoint\) subset of\[0,1\]K\[0,1\]^\{K\}\. Nonetheless, as in the single\-threshold case, the cost remains monotone in each coordinate, and this alone suffices to characterize the structure of optimal thresholds below\.

###### Theorem 3\.6\(Optimal threshold vector in the non\-oracle case\)\.

Consider the constrained problem

min𝜽∈\[0,1\]K⁡C​\(𝜽\)s\.t\.gNO​\(𝜽\)≤b,\\min\_\{\\boldsymbol\{\\theta\}\\in\[0,1\]^\{K\}\}\\penalty 10000\\ C\(\\boldsymbol\{\\theta\}\)\\qquad\\text\{s\.t\.\}\\qquad g\_\{\\mathrm\{NO\}\}\(\\boldsymbol\{\\theta\}\)\\leq b,\(21\)withC​\(⋅\)C\(\\cdot\)remaining the same as in the oracle case and the non\-oracle error as in \([20](https://arxiv.org/html/2609.15992#S3.E20)\) \(i\.e\.,gNO​\(⋅\)=ErrorMT\-NO​\(⋅\)g\_\{\\mathrm\{NO\}\(\\cdot\)\}=\\text\{Error\}\_\{\\text\{MT\-NO\}\}\(\\cdot\)\)\. Define the*excess error*curvesΔk​\(β\):=εs,k​\(β\)−ε¯m,k,for​k=1,…,K\.\\Delta\_\{k\}\(\\beta\):=\\varepsilon\_\{s,k\}\(\\beta\)\-\\bar\{\\varepsilon\}\_\{m,k\},\\,\\,\\text\{for\}\\,\\,k=1,\\dots,K\\,\.

Then: \(i\) there exists an optimal solution𝛉∗\\boldsymbol\{\\theta\}^\{\*\}with*tight*constraintgNO​\(𝛉∗\)=bg\_\{\\mathrm\{NO\}\}\(\\boldsymbol\{\\theta\}^\{\*\}\)=b, unless𝛉=𝟎\\boldsymbol\{\\theta\}=\\boldsymbol\{0\}is already feasible \(in which case𝛉∗=𝟎\\boldsymbol\{\\theta\}^\{\*\}=\\boldsymbol\{0\}\), \(ii\) For any optimal solution𝛉∗\\boldsymbol\{\\theta\}^\{\*\}and two classesi,ji,jwith coordinates interior inℱb\\mathcal\{F\}\_\{b\}that satisfy0<θi∗,θj∗​<1​and​Δi​\(θi∗\)\>​0,Δj​\(θj∗\)\>00<\\theta\_\{i\}^\{\*\},\\theta\_\{j\}^\{\*\}<1\\,\\,\\text\{and\}\\,\\,\\Delta\_\{i\}\(\\theta\_\{i\}^\{\*\}\)\>0,\\ \\Delta\_\{j\}\(\\theta\_\{j\}^\{\*\}\)\>0, their excess boundary errors must be equal:Δi​\(θi∗\)=Δj​\(θj∗\)=t∈ℝ\\Delta\_\{i\}\(\\theta\_\{i\}^\{\*\}\)=\\Delta\_\{j\}\(\\theta\_\{j\}^\{\*\}\)=t\\in\\mathbb\{R\}\(ttis determined by the tightness equationgNO​\(𝛉∗\)=bg\_\{\\mathrm\{NO\}\}\(\\boldsymbol\{\\theta\}^\{\*\}\)=b\)\. Consequently, for every classkkwith0<θk∗<10<\\theta\_\{k\}^\{\*\}<1andΔk​\(θk∗\)\>0\\Delta\_\{k\}\(\\theta\_\{k\}^\{\*\}\)\>0,Δk​\(θk∗\)=t\\Delta\_\{k\}\(\\theta\_\{k\}^\{\*\}\)=t, and \(iii\) an optimal solution𝛉∗\\boldsymbol\{\\theta\}^\{\*\}can be chosen to satisfy one of the following three regimes, for each classkk: \(a\) ifΔk​\(β\)≤t\\Delta\_\{k\}\(\\beta\)\\leq tfor allβ\\beta, thenθk∗=0\\theta\_\{k\}^\{\*\}=0, \(b\) ifΔk​\(⋅\)\\Delta\_\{k\}\(\\cdot\)crosses leveltt, thenθk∗∈Δk−1​\(t\)\\theta\_\{k\}^\{\*\}\\in\\Delta\_\{k\}^\{\-1\}\(t\)and0<θk∗<10<\\theta\_\{k\}^\{\*\}<1, and \(c\) ifΔk​\(β\)\>t\\Delta\_\{k\}\(\\beta\)\>tfor allβ\\beta, thenθk∗=1\\theta\_\{k\}^\{\*\}=1\.

###### Proof\.

See Appendix[B\.4](https://arxiv.org/html/2609.15992#A2.SS4)\. ∎

##### Intuition of Theorem[3\.6](https://arxiv.org/html/2609.15992#S3.Thmtheorem6)\.

The relevant quantity is no longer the sLLM error alone, but the excess errorΔk​\(β\)=εs,k​\(β\)−ε¯m,k\\Delta\_\{k\}\(\\beta\)=\\varepsilon\_\{s,k\}\(\\beta\)\-\\bar\{\\varepsilon\}\_\{m,k\}, which measures how much worse \(or better\) the sLLM is than mLLM for classkkat confidence levelβ\\beta\. The optimal policy equalizes this excess boundary error across all interior classes\. Hence, the oracle equalization rule extends to the non\-oracle case by replacingεs,k\\varepsilon\_\{s,k\}byΔk\\Delta\_\{k\}\. If one class had higher excess boundary error than another, reallocating deferrals would reduce cost without violating the error constraint, contradicting optimality\. Classes always below this level are handled by the sLLM \(always accept\), while those always above it are always deferred to the mLLM\.

### 3\.3Summary of Theoretical Results

We summarize the main theoretical conclusions for the four settings considered this Section:

1\) Single\-threshold, oracle setting\.Error decreases and cost increases monotonically with the thresholdθ\\theta\. The optimal policy is the smallest threshold that meets the error budget: defer only as much as necessary\. 2\) Single\-threshold, non\-oracle setting\.Although error is no longer monotone inθ\\theta, cost remains monotone\. The optimal policy is still the leftmost feasible threshold \(the least costly one satisfying the error constraint\)\. 3\) Multi\-threshold, oracle setting\.With class\-specific thresholds, the optimum equalizes a common boundary error level across all classes in the interior of the feasible set\. 4\) Multi\-threshold, non\-oracle setting\.The optimum equalizes excess boundary error across interior classes, given that the relevant quantity is the excess error \(sLLM minus mLLM error\)\. Overall implication\.Across all cases, cost monotonicity selects the least expensive feasible policy, and in the multi\-threshold setting optimality is characterized by an equalization principle \(boundary error or excess boundary error\)\.

## 4Experimental results

In this section, we evaluate our policies in both discriminative and generative settings\. We use text classification as a representative discriminative task, and question\-answering \(QA\) and machine\-translation \(MT\) as generative benchmarks, demonstrating effectiveness across diverse problem settings\. Given a finite sample of data, we approximate expected cost and error via efficient Monte\-Carlo \(MC\) sampling algorithms \(more details about the approximation and the implemented algorithms can be found in Appendices[F\.4](https://arxiv.org/html/2609.15992#A6.SS4)and[F\.5](https://arxiv.org/html/2609.15992#A6.SS5), respectively\)\.\.Models\.We experiment with three open\-source LLM families of various sizes, namely, different versions of thegemma3model\[gemma3\]\(i\.e\.,11B up to2727B parameters\),qwen3\[qwen3\]with44B andministral3\[liu2026ministral3\]with33B parameters\. We considered both homogeneous pairings, where both models belong to the same family, and heterogeneous pairings, combining models from different families\. Datasets\.We test our framework on several classification datasets\. For binary classification, we usesst\-2\[socher\-etal\-2013\-recursive\]andfakenews\[fakenews\]; while for multi\-class classification, we usedagnews\[agnews\]andemotion\[saravia\-etal\-2018\-carer\]\. For the QA task, we select theSQuADdataset\[rajpurkar\-etal\-2016\-squad\]and for the MT task, theWMTenglish\-to\-german translate dataset\[bojar\-EtAl:2014:W14\-33\]\. We use a 80%\\%split for the optimization process and the rest 20%\\%for testing\.888More details about the datasets in Appendix[F\.1](https://arxiv.org/html/2609.15992#A6.SS1)\. Due to computational restrictions, we limit our experiments to a subset of 8,000 samples from each dataset\. Evaluation metrics\.We use standard, task\-appropriate metrics\. For classification, we report*Accuracy*, and theF1F\_\{1\}score\. For text generation, we measure semantic similarity via cosine similarity between predicted and reference embeddings\[DBLP:conf/emnlp/ReimersG19\], and lexical overlap usingrouge\-l\[lin\-2004\-rouge\]\. Inference cost is measured in TFLOPs using the PyTorch profiler\[DBLP:conf/nips/PaszkeGMLBCKLGA19\]for per\-query estimates; then, model\-specific cost constantscsc\_\{s\}andcmc\_\{m\}are computed by summarizing these estimated \(e\.g\., with statistics like the mean or the median\)\. We report efficiency using a ‘cost savings’ metric, defined as the percentage reduction in total cost relative to using only the mLLM for all queries\. Baselines\.We compare our policies against:*\(i\) a no\-deferral strategy*, where all queries are handled by sLLM;*\(ii\) a full\-deferral strategy*, where all queries are sent to mLLM;*\(iii\) a random\-deferral strategy*, deferring each query independently with probability 0\.5;*\(iv\) FrugalGPT*\[frugal\], a cascade framework that uses an external scorer and optimizes a single deferral threshold; and*\(v\) HybridLLM*\[hybridlllm\], which trains an external router to assign each query to one of the two models\.

### 4\.1Generative Tasks

We utilize the single\-threshold policy in the free\-form generation tasks\. In each case, the model generates an output sequence conditioned on the given context and question for QA, and a sentence to be translated for MT\. For the confidence value, we use the average log probability of the predicted tokens\[lm\-polygraph\]\. Additionally, to measure the expected probability of error, we defined the following notion of correctness: At first, the embeddings of both the predicted and true answers are computed using Sentence Transformers\[DBLP:conf/emnlp/ReimersG19\]\. Secondly, we calculate the cosine similarity between the embeddings of the true and predicted answers\. Then, a thresholdτ\\tauis applied to the similarity score to determine correctness, i\.e\.,𝟙​\[𝒂^=𝒂\]=𝟙​\[sim​\(𝒂^,𝒂\)≥τ\]\\mathds\{1\}\[\\hat\{\\boldsymbol\{a\}\}=\\boldsymbol\{a\}\]=\\mathds\{1\}\[\\mathrm\{sim\}\(\\hat\{\\boldsymbol\{a\}\},\\boldsymbol\{a\}\)\\geq\\tau\]\. We present results for both datasets in Table[1](https://arxiv.org/html/2609.15992#S4.T1)for thegemma3\-1b\(as sLLM\) andgemma3\-4b\(as mLLM\) models pair in the non\-oracle setting \(results with additional model pairs can be found in Appendix[F\.3](https://arxiv.org/html/2609.15992#A6.SS3)\)\. For the optimization, we set the boundξ\\xias the performance of mLLM, while for FrugalGPT, we use our policy’s achieved cost as the budget\. HybridLLM solves a slightly different objective that minimizes cost savings \(deferred samples\) but subject to keeping the performance drop \(reduction in BART score\[bartscore\]\) less than a defined value \(typically 1%\)\. The results show that our method can deliver a strong performance–cost trade\-off\. OnSQuAD, our method maintains mLLM performance with a 26\.66% reduction in inference cost, and on WMT \(where mLLM was worse than sLLM\), it maintains the best possible performance with maximal cost savings\. These results demonstrate that our policy adapts to task difficulty, allocating compute only when necessary and avoiding the inefficiencies of both static and random strategies\. During our experiments, we also observed that the performance of FrugalGPT and HybridLLM was strongly contingent on the effectiveness of the routing model and the data used to train it, something that represents a drawback for these methods\. Furthermore, compared to FrugalGPT and HybridLLM, we also save up the cost of training the additional routing model\. The error and cost trade\-offs for theSQuADdataset are also illustrated in Fig\.[1](https://arxiv.org/html/2609.15992#S4.F1)\.

Table 1:Performance comparison of our policy with other policies on generation tasks \(gemma3\-1bas sLLM andgemma3\-4bas mLLM\)\.
![Refer to caption](https://arxiv.org/html/2609.15992v1/x1.png)\(a\)
![Refer to caption](https://arxiv.org/html/2609.15992v1/x2.png)\(b\)

Figure 1:Expected cost \(top\) and error \(bottom\) as functions of the deferral thresholdθ\\theta\(single\-threshold policy\) on theSQuADtest set\. The deferral rate increases monotonically with the threshold\. Horizontal lines indicate the cases of 0% and 100% deferral rate\. The vertical line denotes the optimal thresholdθ∗\\theta^\{\*\}obtained by our policy\.
### 4\.2Discriminative Tasks

As described in Section[3\.2](https://arxiv.org/html/2609.15992#S3.SS2), we utilize the multi\-threshold policy in zero\-shot classification settings where a separate threshold is optimized for each category\. For confidence estimation, we utilize the maximum softmax probability\[chow1970,committees\]provided by the model\. To simulate classification, we constrain the LLM to output only a single token corresponding to the predicted class label\. Table[2](https://arxiv.org/html/2609.15992#S4.T2)shows the results of the experiments on the test set of each classification dataset for thegemma3\-1b\(as sLLM\) andgemma3\-4b\(as mLLM\) models pair in the non\-oracle setting \(results with additional model pairs can be found in Appendix[F\.3](https://arxiv.org/html/2609.15992#A6.SS3)\)\. Across all datasets, our policy consistently maintains near–‘optimal’ performance \(i\.e\., comparable to deferring all samples to the mLLM\) while reducing inference cost significantly\. Onsst\-2, it matches full\-deferral accuracy with∼\\sim82% cost savings, and onagnewsandemotion, it achieves full\-deferral performance with∼\\sim16% and∼\\sim68% savings, respectively\. Even on the more challengingfakenewsdataset, the performance degrades only∼\\sim2% from full\-deferral while cutting cost by∼\\sim45%\.

Table 2:Performance comparison of our policy with other policies on classification tasks using thegemma3\-1b\(sLLM\) andgemma3\-4b\(mLLM\) models pair\.

## 5Conclusion

We introduced*inference networks*of LLM experts, modeled as graphs, where nodes represent alternative models and edges capture conditional model activations\. We formalized the design of optimal model activation as minimizing the expected inference cost subject to a target performance constraint and we characterized the structure of the optimal activation rule in a two\-model chain, proving that it admits a*threshold form*on a scalar confidence score: for discriminative settings the policy decomposes into class\-specific thresholds, while for generative settings it reduces to a single threshold\. Beyond establishing this structural result, we provided approaches to compute the thresholds through efficient low\-dimensional search \(and closed\-form solutions in special cases\), making the proposed policy directly implementable\. To bridge theory and practice, we applied practical confidence estimation methods to both discriminative and generative outputs and evaluated the framework using open\-source LLMs on text classification and generation benchmarks\. The results demonstrated that the proposed policies reduce inference costs while meeting a specified performance budget, improving the cost–quality trade\-off in realistic deployments\. Several directions remain open\. Firstly, the sequential decision\-making nature of the problem makes it possible to apply reinforcement learning \(RL\) methods for learning the optimal threshold parameter\(s\)\. Secondly, while we view inference networks as general directed graphs, this work fixes the topology to be serial; extending the analysis to richer topologies and jointly optimizing routing and topology is a natural next step\. Finally, improving confidence estimation for generative tasks can further reduce the discrepancy between predicted confidence and actual quality, enhancing both performance and reliability in LLM\-based inference\.

## References

Optimal Model Activation Policies for Inference Networks of Large Language Models \(Supplementary Material\)

## Appendix AAdditional Related Work

##### Model chaining\.

To address the high computational cost of large language models \(LLMs\), recent works have explored frameworks that adaptively allocate computational resources based on input difficulty\.\[tabi\]utilize a multi\-level inference engine fordiscriminative models\(e\.g\., BERT\[bert\]etc\.\), using confidence scores to decide whether to return predictions from an \(initial\) small model or escalate queries to larger ones\. It also leverages attention\-based token pruning to reduce latency by removing unnecessary works and weighted ensembling to maintain accuracy\. FrugalGPT\[frugal\]extends this idea to generative modeling by designing an algorithmic framework that selects among multiple LLMs based on a learned output quality judge\. It invokesKK\-out\-of\-NNmodels in a sequence until a satisfactory answer quality threshold is reached, achieving cost savings without significantly compromising accuracy\. In an orthogonal line of work,\[gupta\_uncertainty\]investigate measurements for developing deferral rules and develop a mechanism based on quantiles of token probabilities Furthermore, AutoMix\[automix\]presents a strategy using self\-verification of the small model \(framed as an entailment task\) and a partially observable Markov decision process \(POMDP\)\-based\[astrom\]router to decide whether to escalate a query to a larger model\. Finally, apart from generative models, there are works that implement deferral rules for predictive ML tasks\[committees,DBLP:conf/nips/JitkrittumGMNRK23,frugal\_ML,DBLP:conf/icml/ChenZ022,9560049,efficient\_classif\]\. For instance,\[frugal\_ML\]develops a system with an application in classification tasks\. The nature of such tasks allow the estimation of the answer quality without querying the next model each time \(e\.g\., via classification labels\), something that is more challenging when it comes to generative tasks and models\.

##### Routing\-based approaches\.

While model chaining frameworks focus on sequentially escalating queries through models of increasing capacity, model routing approaches aim to make an in\-advance assignment of each query to the most suitable model in a one\-shot decision\[huang\-etal\-2025\-routereval\]\. Recent work on model routing explores various techniques to leverage performance diversity among LLMs\. HybridLLM\[hybridlllm\]proposes a trainable quality\-aware router that dynamically adaptively routes queries to either a small or large model based on predicted difficulty\.\[zooter\]frames the routing problem as reward\-guided model selection, distilling supervision from off\-the\-shelf reward models to learn which expert LLM best handles each query\. Similarly,\[DBLP:conf/wsdm/SakotaP024\]proposes a meta\-modeling strategy that predicts both performance and cost of candidate LLMs to optimize query assignments under budget constraints, matching the accuracy of the strongest LLM while reducing costs\. Finally,\[Shnitzer\]introduce a benchmark\-driven routing formulation, using binary classifiers trained on LLMs’ benchmark results to predict which model is most likely to succeed on an incoming query, thus offering a low\-cost method for effective model selection without repeated inference\. Overall, these routing approaches enable efficient inference by invoking only a single model per query through informed, pre\-inference model selection\. There are are works that consider hybrid inference in classification settings\.\[edge\_query\]propose a training framework where two models \(a cheap one and a more expensive one\) are trained from scratch along with the router model\. This setting, albeit feasible for \(most\) widely used classifiers, is prohibitive for LLM use\-cases due to the high cost of training and fine\-tuning such big foundation models\.

## Appendix BProofs

We provide extended versions of the main text’s proofs\.

### B\.1Single\-Threshold Oracle case

###### Proof of Lemma[3\.1](https://arxiv.org/html/2609.15992#S3.Thmtheorem1)\.

For0≤θ1<θ2≤10\\leq\\theta\_\{1\}<\\theta\_\{2\}\\leq 1,

C​\(θ2\)−C​\(θ1\)=cm​∫θ1θ2f​\(β\)​𝑑β≥0,C\(\\theta\_\{2\}\)\-C\(\\theta\_\{1\}\)=c\_\{m\}\\\!\\int\_\{\\theta\_\{1\}\}^\{\\theta\_\{2\}\}\\\!f\(\\beta\)\\,d\\beta\\;\\geq\\;0,\(22\)sincef≥0f\\geq 0\. Thus,C​\(⋅\)C\(\\cdot\)is non\-decreasing\. Similarly,

g​\(θ2\)−g​\(θ1\)=−∫θ1θ2ε​\(β\)​f​\(β\)​𝑑β≤0,g\(\\theta\_\{2\}\)\-g\(\\theta\_\{1\}\)=\-\\int\_\{\\theta\_\{1\}\}^\{\\theta\_\{2\}\}\\varepsilon\(\\beta\)\\,f\(\\beta\)\\,d\\beta\\;\\leq\\;0,\(23\)becauseε​\(β\)​f​\(β\)≥0\\varepsilon\(\\beta\)f\(\\beta\)\\geq 0\. Hence,g​\(⋅\)g\(\\cdot\)is non\-increasing\. ∎

###### Proof of Theorem[3\.2](https://arxiv.org/html/2609.15992#S3.Thmtheorem2)\.

By Lemma[3\.1](https://arxiv.org/html/2609.15992#S3.Thmtheorem1),g​\(⋅\)g\(\\cdot\)is non\-increasing, hence the feasible setℱb:=\{θ:g​\(θ\)≤b\}\\mathcal\{F\}\_\{b\}:=\\\{\\theta:g\(\\theta\)\\leq b\\\}is a right ray\[θ∗,1\]\[\\theta^\{\*\},1\]\(or empty\), whereθ∗:=inf\{θ∈\[0,1\]:g​\(θ\)≤b\}\\theta^\{\*\}:=\\inf\\\{\\theta\\in\[0,1\]:g\(\\theta\)\\leq b\\\}\. SinceC​\(⋅\)C\(\\cdot\)is non\-decreasing, its minimum overℱb\\mathcal\{F\}\_\{b\}is attained at its left endpoint, i\.e\., atθ∗\\theta^\{\*\}\. Furthermore,g​\(⋅\)g\(\\cdot\)is continuous so the constraint is tight, i\.e\.,g​\(θ\)=bg\(\\theta\)=band if it is strictly decreasing \(e\.g\.,ε​\(θ∗\)​f​\(θ∗\)\>0\\varepsilon\(\\theta^\{\*\}\)f\(\\theta^\{\*\}\)\>0\),θ∗\\theta^\{\*\}is unique withg​\(θ\)=bg\(\\theta\)=b\. ∎

##### Optimal threshold calculation example\.

The optimal thresholdθ∗\\theta^\{\*\}can be analytically computed iff​\(β\)f\(\\beta\)belongs into a distribution family with closed\-form PDF andεs​\(β\)\\varepsilon\_\{s\}\(\\beta\)is known\. To make things concrete, we now present how to calculateθ∗\\theta^\{\*\}\(in the oracle scenario\) under the assumption thatβ∼Uniform​\(0,1\)\\beta\\sim\\text\{Uniform\}\(0,1\)and that the error rate of sLLM decreases linearly with confidence, e\.g\.,εs​\(β\)=1−β\\varepsilon\_\{s\}\(\\beta\)=1\-\\beta999This can be generally achieved and \(asymptotically\) ensured through calibration with proper scoring rules, under no distributional shift\[DBLP:journals/tmlr/FerrerR25\]\.We thus have thatf​\(β\)=1f\(\\beta\)=1and

ErrorST\-O​\(θ\)=∫θ1\(1−β\)​𝑑β=\[β−12​β2\]θ1=12−θ\+12​θ2≤1−ξ\.\\text\{Error\}\_\{\\text\{ST\-O\}\}\(\\theta\)=\\int\_\{\\theta\}^\{1\}\(1\-\\beta\)\\,d\\beta=\\left\[\\beta\-\\frac\{1\}\{2\}\\beta^\{2\}\\right\]\_\{\\theta\}^\{1\}=\\frac\{1\}\{2\}\-\\theta\+\\frac\{1\}\{2\}\\theta^\{2\}\\leq 1\-\\xi\\,\.\(25\)
Solving the quadratic equality \(the optimalθ∗\\theta^\{\*\}occurs exactly where the error constraint is tight\) and taking the smallest solution, we get:

θ∗=1±2​\(1−ξ\)⇒θ∈\[0,1\]θ∗=1−2​\(1−ξ\)\.\\theta^\{\*\}=1\\pm\\sqrt\{2\(1\-\\xi\)\}\\xRightarrow\{\\theta\\in\[0,1\]\}\\theta^\{\*\}=1\-\\sqrt\{2\(1\-\\xi\)\}\\,\.\(26\)This result can also be generalized ifβ\\betalies in an arbitrary interval in the setℐ=\{\[x,y\]⊆\[0,1\]∣x≤y\}\\mathcal\{I\}=\\left\\\{\[x,y\]\\subseteq\[0,1\]\\mid x\\leq y\\right\\\}and for any distribution family that provides a closed\-form PDF expression as well as different forms forεs​\(β\)\\varepsilon\_\{s\}\(\\beta\)\.

### B\.2Single\-Threshold Non\-Oracle case

###### Proof of Theorem[3\.3](https://arxiv.org/html/2609.15992#S3.Thmtheorem3)\.

\(i\) Sinceℱb\\mathcal\{F\}\_\{b\}is closed \(by continuity ofgNOg\_\{\\mathrm\{NO\}\}\) and non\-empty,θ∗=min⁡ℱb\\theta^\{\*\}=\\min\\mathcal\{F\}\_\{b\}exists and satisfiesθ≥θ∗\\theta\\geq\\theta^\{\*\}for allθ∈ℱb\\theta\\in\\mathcal\{F\}\_\{b\}\. BecauseC​\(⋅\)C\(\\cdot\)is non\-decreasing,C​\(θ\)≥C​\(θ∗\)C\(\\theta\)\\geq C\(\\theta^\{\*\}\)for all feasibleθ\\theta, henceθ∗\\theta^\{\*\}is optimal\. \(ii\) IfgNO​\(0\)≤bg\_\{\\mathrm\{NO\}\}\(0\)\\leq b, thenθ=0\\theta=0is feasible and minimizesCC, soθ∗=0\\theta^\{\*\}=0\. \(iii\) IfgNO​\(0\)\>bg\_\{\\mathrm\{NO\}\}\(0\)\>bandgNO​\(θ∗\)<bg\_\{\\mathrm\{NO\}\}\(\\theta^\{\*\}\)<b, continuity ofgNOg\_\{\\mathrm\{NO\}\}implies there existsδ\>0\\delta\>0such thatgNO​\(θ\)<bg\_\{\\mathrm\{NO\}\}\(\\theta\)<bfor allθ∈\(θ∗−δ,θ∗\+δ\)\\theta\\in\(\\theta^\{\*\}\-\\delta,\\theta^\{\*\}\+\\delta\), yielding a feasibleθ<θ∗\\theta<\\theta^\{\*\}, contradicting minimality ofθ∗\\theta^\{\*\}\. ThereforegNO​\(θ∗\)=bg\_\{\\mathrm\{NO\}\}\(\\theta^\{\*\}\)=b\. ∎

### B\.3Multi\-Thresholds Oracle case

###### Proof of Lemma[3\.4](https://arxiv.org/html/2609.15992#S3.Thmtheorem4)\.

For0≤θk<θk′≤10\\leq\\theta\_\{k\}<\\theta\_\{k\}^\{\\prime\}\\leq 1,

C​\(θ1,…,θk′,…,θK\)−C​\(𝜽\)=cm​pk​∫θkθk′fk​\(β\)​𝑑β≥0,C\(\\theta\_\{1\},\\\!\\dots,\\theta\_\{k\}^\{\\prime\},\\\!\\dots,\\theta\_\{K\}\)\-C\(\\boldsymbol\{\\theta\}\)=c\_\{m\}p\_\{k\}\\\!\\int\_\{\\theta\_\{k\}\}^\{\\theta\_\{k\}^\{\\prime\}\}\\\!f\_\{k\}\(\\beta\)\\,d\\beta\\geq 0,\(27\)and

g​\(θ1,…,θk′,…,θK\)−g​\(𝜽\)=−pk​∫θkθk′εk​\(β\)​fk​\(β\)​𝑑β≤0,g\(\\theta\_\{1\},\\\!\\dots,\\theta\_\{k\}^\{\\prime\},\\\!\\dots,\\theta\_\{K\}\)\-g\(\\boldsymbol\{\\theta\}\)=\-\\,p\_\{k\}\\\!\\int\_\{\\theta\_\{k\}\}^\{\\theta\_\{k\}^\{\\prime\}\}\\varepsilon\_\{k\}\(\\beta\)\\,f\_\{k\}\(\\beta\)\\,d\\beta\\leq 0,\(28\)sincefk≥0f\_\{k\}\\geq 0andεk∈\[0,1\]\\varepsilon\_\{k\}\\in\[0,1\]\. The upper\-orthant property follows immediately\. ∎

###### Proof of Theorem[3\.5](https://arxiv.org/html/2609.15992#S3.Thmtheorem5)\.

\(i\) By Lemma[3\.4](https://arxiv.org/html/2609.15992#S3.Thmtheorem4), decreasing any coordinate strictly decreasesC​\(⋅\)C\(\\cdot\)and increasesg​\(⋅\)g\(\\cdot\)\. If an optimal𝜽^\\hat\{\\boldsymbol\{\\theta\}\}had slackg​\(𝜽^\)<bg\(\\hat\{\\boldsymbol\{\\theta\}\}\)<b, decreasing some coordinate slightly would reduceC​\(⋅\)C\(\\cdot\)while maintaining feasibility\-contradiction\. Hence the constraint is tight at optimum unless𝜽=𝟎\\boldsymbol\{\\theta\}=\\boldsymbol\{0\}is already feasible\.

\(ii\) Suppose𝜽\\boldsymbol\{\\theta\}is feasible withg​\(𝜽\)=bg\(\\boldsymbol\{\\theta\}\)=b, and there exist two interior classesi,ji,jwithεs,i​\(θi\)\>εs,j​\(θj\)\\varepsilon\_\{s,i\}\(\\theta\_\{i\}\)\>\\varepsilon\_\{s,j\}\(\\theta\_\{j\}\)\. Consider small moves that keep total error unchanged: increaseθi\\theta\_\{i\}byd\>0d\>0and decreaseθj\\theta\_\{j\}bys\>0s\>0chosen so thatpi​εs,i​\(θi\)​fi​\(θi\)​d=pj​εs,j​\(θj\)​fj​\(θj\)​sp\_\{i\}\\varepsilon\_\{s,i\}\(\\theta\_\{i\}\)f\_\{i\}\(\\theta\_\{i\}\)\\,d\\;=\\;p\_\{j\}\\varepsilon\_\{s,j\}\(\\theta\_\{j\}\)f\_\{j\}\(\\theta\_\{j\}\)\\,s\. The cost change is:

Δ​C=cm​\[pi​fi​\(θi\)​d−pj​fj​\(θj\)​s\]=cm​pi​fi​\(θi\)​d​\(1−εs,i​\(θi\)εs,j​\(θj\)\)<0,\\Delta C=c\_\{m\}\\big\[p\_\{i\}f\_\{i\}\(\\theta\_\{i\}\)d\-p\_\{j\}f\_\{j\}\(\\theta\_\{j\}\)s\\big\]=c\_\{m\}p\_\{i\}f\_\{i\}\(\\theta\_\{i\}\)d\\Big\(1\-\\frac\{\\varepsilon\_\{s,i\}\(\\theta\_\{i\}\)\}\{\\varepsilon\_\{s,j\}\(\\theta\_\{j\}\)\}\\Big\)<0\\,,\(29\)a contradiction to optimality\. Therefore all interior coordinates must satisfyεs,k​\(θk∗\)=t\\varepsilon\_\{s,k\}\(\\theta\_\{k\}^\{\*\}\)=tfor a commontt\.

\(iii\) With equalized boundary errortt, pick anyθk∗∈εs,k−1​\(t\)\\theta\_\{k\}^\{\*\}\\in\\varepsilon\_\{s,k\}^\{\-1\}\(t\)for each active class\. The tightness conditiong​\(𝜽∗\)=bg\(\\boldsymbol\{\\theta\}^\{\*\}\)=buniquely fixestt\(unless some coordinates saturate at0or11, as described\)\. ∎

##### Optimal threshold vector calculation example\.

Following the example we used in the single\-threshold policy, we will show how to analytically compute the optimal vector of thresholds𝜽∗\\boldsymbol\{\\theta\}^\{\*\}\(in the oracle case\) under the assumption thatβk∼Uniform​\(0,1\),∀k=1,…,K\\beta\_\{k\}\\sim\\text\{Uniform\}\(0,1\),\\,\\forall\\,k=1,\\ldots,K\(thus having that eachfk​\(β\)=1f\_\{k\}\(\\beta\)=1and∫0θkfk​\(β\)​𝑑β=θk\\int\_\{0\}^\{\\theta\_\{k\}\}f\_\{k\}\(\\beta\)\\,d\\beta=\\theta\_\{k\}\) and that we useKKlinear error functionsεk​\(β\)=1−δk​β,δk\>0\\varepsilon\_\{k\}\(\\beta\)=1\-\\delta\_\{k\}\\beta,\\,\\delta\_\{k\}\>0\. Since all the variables of the threshold vector are coupled through the shared constraint in \([17](https://arxiv.org/html/2609.15992#S3.E17)\), we will employ the method ofLagrange multiplierswhich allows the handling of multiple variables simultaneously\.

Using the Uniform distribution assumption, we can define the \(relaxed\) Lagrangian objective as:

ℒr​e​l​\(θ1,…,θK,λ\)=cs\+cm​∑k=1Kpk​θk\+λ​\(∑k=1Kpk​∫θk1εk​\(β\)​𝑑β−\(1−ξ\)\),\\mathcal\{L\}\_\{rel\}\(\\theta\_\{1\},\\ldots,\\theta\_\{K\},\\lambda\)=c\_\{s\}\+c\_\{m\}\\sum\_\{k=1\}^\{K\}p\_\{k\}\\theta\_\{k\}\+\\lambda\\left\(\\sum\_\{k=1\}^\{K\}p\_\{k\}\\int\_\{\\theta\_\{k\}\}^\{1\}\\varepsilon\_\{k\}\(\\beta\)\\,d\\beta\-\(1\-\\xi\)\\right\)\\,,\(32\)and by setting the derivative w\.r\.tθk\\theta\_\{k\}to zero for optimality, we obtain the optimal thresholds:

∂ℒr​e​l∂θk=0⇒cm​pk−λ​pk​εk​\(θk\)=0⇒εk​\(θk∗\)=cmλ⇒θk∗=εk−1​\(cmλ\)\.\\frac\{\\partial\\mathcal\{L\}\_\{rel\}\}\{\\partial\\theta\_\{k\}\}=0\\Rightarrow c\_\{m\}p\_\{k\}\-\\lambda p\_\{k\}\\varepsilon\_\{k\}\(\\theta\_\{k\}\)=0\\Rightarrow\\varepsilon\_\{k\}\(\\theta\_\{k\}^\{\*\}\)=\\frac\{c\_\{m\}\}\{\\lambda\}\\Rightarrow\\theta^\{\*\}\_\{k\}=\\varepsilon\_\{k\}^\{\-1\}\(\\frac\{c\_\{m\}\}\{\\lambda\}\)\\,\.\(33\)Since the inverse of a linear function can be calculated, using the linear error functions assumption, we have that

θk∗=1−cmλδk\.\\theta^\{\*\}\_\{k\}=\\frac\{1\-\\frac\{c\_\{m\}\}\{\\lambda\}\}\{\\delta\_\{k\}\}\\,\.\(34\)Furthermore, the constraint \(i\.e\., the expected error\) of \([17](https://arxiv.org/html/2609.15992#S3.E17)\) becomes:

∑k=1Kpk​∫θk∗1\(1−δk​β\)​𝑑β=∑k=1K\[1−θk∗−12​δk​\(1−\(θk∗\)2\)\]=∑k=1K\[1−1−cmλδk−12​δk​\(1−\(1−cmλδk\)2\)\],\\sum\_\{k=1\}^\{K\}p\_\{k\}\\int\_\{\\theta^\{\*\}\_\{k\}\}^\{1\}\(1\-\\delta\_\{k\}\\beta\)\\,d\\beta=\\sum\_\{k=1\}^\{K\}\\left\[1\-\\theta^\{\*\}\_\{k\}\-\\frac\{1\}\{2\}\\delta\_\{k\}\(1\-\(\\theta^\{\*\}\_\{k\}\)^\{2\}\)\\right\]=\\sum\_\{k=1\}^\{K\}\\left\[1\-\\frac\{1\-\\frac\{c\_\{m\}\}\{\\lambda\}\}\{\\delta\_\{k\}\}\-\\frac\{1\}\{2\}\\delta\_\{k\}\\left\(1\-\\left\(\\frac\{1\-\\frac\{c\_\{m\}\}\{\\lambda\}\}\{\\delta\_\{k\}\}\\right\)^\{2\}\\right\)\\right\]\\,,\(35\)and ultimately, the constraint equation is:

∑k=1K\[1−1−cmλδk−12​δk​\(1−\(1−cmλδk\)2\)\]=1−ξ\.\\sum\_\{k=1\}^\{K\}\\left\[1\-\\frac\{1\-\\frac\{c\_\{m\}\}\{\\lambda\}\}\{\\delta\_\{k\}\}\-\\frac\{1\}\{2\}\\delta\_\{k\}\\left\(1\-\\left\(\\frac\{1\-\\frac\{c\_\{m\}\}\{\\lambda\}\}\{\\delta\_\{k\}\}\\right\)^\{2\}\\right\)\\right\]=1\-\\xi\\,\.\(36\)This is a non\-linear equation inλ\\lambdaand it cannot be solved analytically in the general case because eachθk∗\\theta^\{\*\}\_\{k\}involves a rational function ofλ\\lambdaand the resulting equation is a sum of non\-identical, non\-linear rational\-quadratic terms \(which are not invertible in closed\-form\)\. Due to the complexity, numerical root\-finding methods can be used \(e\.g\., Brent’s method\[DBLP:journals/cj/Brent71\]\) to solve forλ\\lambda\. Onceλ∗\\lambda^\{\*\}is obtained, each optimal threshold is recovered using:

θk∗=1−cmλ∗δk\.\\theta^\{\*\}\_\{k\}=\\frac\{1\-\\frac\{c\_\{m\}\}\{\\lambda^\{\*\}\}\}\{\\delta\_\{k\}\}\\,\.\(37\)If we assume symmetry across all classes whereδk=δ\\delta\_\{k\}=\\deltaandpk=1K,∀kp\_\{k\}=\\frac\{1\}\{K\},\\,\\forall kthen all thresholds become equal:θk∗=θ∗\\theta^\{\*\}\_\{k\}=\\theta^\{\*\}\. The constraint equation simplifies to:

δ​\(θ∗\)2−2​θ∗\+\(−δ\+2​ξ\)=0\\delta\(\\theta^\{\*\}\)^\{2\}\-2\\theta^\{\*\}\+\(\-\\delta\+2\\xi\)=0\(38\)Solving the quadratic equation101010Underξ≤1\\xi\\leq 1, the discriminant is non\-negative, so real roots always exist\.111111To ensure at least one solution in\[0,1\]\[0,1\], it suffices to impose0<δ≤2​ξ0<\\delta\\leq 2\\xi\(ξ=1\\xi=1is necessary forθ\+∗≤1\\theta^\{\*\}\_\{\+\}\\leq 1\)\. Ifξ=1\\xi=1, this reduces to1≤δ≤21\\leq\\delta\\leq 2, which ensures thatθ−∗∈\[0,1\]\\theta^\{\*\}\_\{\-\}\\in\[0,1\]andθ\+∗=1\\theta^\{\*\}\_\{\+\}=1, so we can chooseθ−∗\\theta^\{\*\}\_\{\-\}\., we get:

θ∗=1±1\+δ2−2​δ​ξδ⇒θ∈\[0,1\]1−1\+δ2−2​δ​ξδ,\\theta^\{\*\}=\\frac\{1\\pm\\sqrt\{1\+\\delta^\{2\}\-2\\delta\\xi\}\}\{\\delta\}\\xRightarrow\{\\theta\\in\[0,1\]\}\\frac\{1\-\\sqrt\{1\+\\delta^\{2\}\-2\\delta\\xi\}\}\{\\delta\}\\,,\(39\)and then:

λ∗=cm1−δ​θ∗\.\\lambda^\{\*\}=\\frac\{c\_\{m\}\}\{1\-\\delta\\theta^\{\*\}\}\\,\.\(40\)By settingδ=1\\delta=1in \([39](https://arxiv.org/html/2609.15992#A2.E39)\), the solutionθ∗\\theta^\{\*\}reduces to the single\-threshold policy’s solution in \([26](https://arxiv.org/html/2609.15992#A2.E26)\)\.

### B\.4Multi\-Thresholds Non\-Oracle case

###### Proof of Theorem[3\.6](https://arxiv.org/html/2609.15992#S3.Thmtheorem6)\.

\(i\) By Lemma[3\.4](https://arxiv.org/html/2609.15992#S3.Thmtheorem4), decreasing any coordinate strictly decreasesC​\(⋅\)C\(\\cdot\)and increasesg​\(⋅\)g\(\\cdot\)\. If an optimal𝜽^\\hat\{\\boldsymbol\{\\theta\}\}had slackg​\(𝜽^\)<bg\(\\hat\{\\boldsymbol\{\\theta\}\}\)<b, decreasing some coordinate slightly would reduceC​\(⋅\)C\(\\cdot\)while maintaining feasibility, a contradiction\. Hence the constraint is tight at optimum unless𝜽=𝟎\\boldsymbol\{\\theta\}=\\boldsymbol\{0\}is already feasible\. \(ii\) Fix two interior classesi,ji,jwithΔi​\(θi∗\)\>0\\Delta\_\{i\}\(\\theta\_\{i\}^\{\*\}\)\>0andΔj​\(θj∗\)\>0\\Delta\_\{j\}\(\\theta\_\{j\}^\{\*\}\)\>0\. Consider a perturbation that increasesθi\\theta\_\{i\}byd\>0d\>0and decreasesθj\\theta\_\{j\}bys\>0s\>0\. Having that

∂gNO∂θk∗=pk​fk​\(θk∗\)​\(ε¯m,k−εs,k​\(θk∗\)\)=−pk​fk​\(θk∗\)​Δk​\(θk∗\),\\frac\{\\partial g\_\{\\mathrm\{NO\}\}\}\{\\partial\\theta^\{\*\}\_\{k\}\}=p\_\{k\}f\_\{k\}\(\\theta\_\{k\}^\{\*\}\)\\big\(\\bar\{\\varepsilon\}\_\{m,k\}\-\\varepsilon\_\{s,k\}\(\\theta\_\{k\}^\{\*\}\)\\big\)=\-p\_\{k\}f\_\{k\}\(\\theta\_\{k\}^\{\*\}\)\\Delta\_\{k\}\(\\theta\_\{k\}^\{\*\}\),\(41\)we get the first\-order error change:

Δ​gNO=−pi​fi​\(θi∗\)​Δi​\(θi∗\)​d\+pj​fj​\(θj∗\)​Δj​\(θj∗\)​s\.\\Delta g\_\{\\mathrm\{NO\}\}=\-p\_\{i\}f\_\{i\}\(\\theta\_\{i\}^\{\*\}\)\\Delta\_\{i\}\(\\theta\_\{i\}^\{\*\}\)\\,d\\;\+\\;p\_\{j\}f\_\{j\}\(\\theta\_\{j\}^\{\*\}\)\\Delta\_\{j\}\(\\theta\_\{j\}^\{\*\}\)\\,s\.\(42\)Furthermore, we choosessso thatΔ​gNO=0\\Delta g\_\{\\mathrm\{NO\}\}=0, namely:

pi​fi​\(θi∗\)​Δi​\(θi∗\)​d=pj​fj​\(θj∗\)​Δj​\(θj∗\)​s\.p\_\{i\}f\_\{i\}\(\\theta\_\{i\}^\{\*\}\)\\Delta\_\{i\}\(\\theta\_\{i\}^\{\*\}\)\\,d=p\_\{j\}f\_\{j\}\(\\theta\_\{j\}^\{\*\}\)\\Delta\_\{j\}\(\\theta\_\{j\}^\{\*\}\)\\,s\.\(43\)The corresponding first\-order cost change is

Δ​C=cm​\[pi​fi​\(θi∗\)​d−pj​fj​\(θj∗\)​s\]\.\\Delta C=c\_\{m\}\\big\[p\_\{i\}f\_\{i\}\(\\theta\_\{i\}^\{\*\}\)\\,d\-p\_\{j\}f\_\{j\}\(\\theta\_\{j\}^\{\*\}\)\\,s\\big\]\.Substituting from \([43](https://arxiv.org/html/2609.15992#A2.E43)\) yields

Δ​C=cm​pi​fi​\(θi∗\)​d​\(1−Δi​\(θi∗\)Δj​\(θj∗\)\)\.\\Delta C=c\_\{m\}\\,p\_\{i\}f\_\{i\}\(\\theta\_\{i\}^\{\*\}\)\\,d\\left\(1\-\\frac\{\\Delta\_\{i\}\(\\theta\_\{i\}^\{\*\}\)\}\{\\Delta\_\{j\}\(\\theta\_\{j\}^\{\*\}\)\}\\right\)\.IfΔi​\(θi∗\)\>Δj​\(θj∗\)\\Delta\_\{i\}\(\\theta\_\{i\}^\{\*\}\)\>\\Delta\_\{j\}\(\\theta\_\{j\}^\{\*\}\)thenΔ​C<0\\Delta C<0, contradicting optimality of𝜽∗\\boldsymbol\{\\theta\}^\{\*\}\. By symmetry, we also cannot haveΔj​\(θj∗\)\>Δi​\(θi∗\)\\Delta\_\{j\}\(\\theta\_\{j\}^\{\*\}\)\>\\Delta\_\{i\}\(\\theta\_\{i\}^\{\*\}\), henceΔi​\(θi∗\)=Δj​\(θj∗\)\\Delta\_\{i\}\(\\theta\_\{i\}^\{\*\}\)=\\Delta\_\{j\}\(\\theta\_\{j\}^\{\*\}\)\. This proves \([3\.6](https://arxiv.org/html/2609.15992#S3.Thmtheorem6)\)\. \(iii\) By \(ii\), all interior classes withΔk​\(θk∗\)\>0\\Delta\_\{k\}\(\\theta\_\{k\}^\{\*\}\)\>0share a common leveltt\. Classes withΔk​\(β\)≤t\\Delta\_\{k\}\(\\beta\)\\leq tfor allβ\\betacannot satisfyΔk​\(θk\)=t\\Delta\_\{k\}\(\\theta\_\{k\}\)=tin the interior, so they must saturate at the lowest\-cost boundaryθk∗=0\\theta\_\{k\}^\{\*\}=0\(always accept\)\. Likewise, classes withΔk​\(β\)\>t\\Delta\_\{k\}\(\\beta\)\>tfor allβ\\betasaturate atθk∗=1\\theta\_\{k\}^\{\*\}=1\(always defer\)\. All remaining classes satisfyθk∗∈Δk−1​\(t\)\\theta\_\{k\}^\{\*\}\\in\\Delta\_\{k\}^\{\-1\}\(t\)\. ∎

### B\.5Multi\-thresholds policy justification

As discussed in Section[3\.2](https://arxiv.org/html/2609.15992#S3.SS2), performance over different classes can vary and models can display a different behavior per class\. We will briefly show when the multi\-thresholds policy can be beneficial with an illustrative example\. Consider the scenario in the Fig\.[2](https://arxiv.org/html/2609.15992#A2.F2)that describes a binary classification setting \(two classes\):

![Refer to caption](https://arxiv.org/html/2609.15992v1/figs/cost-acc.png)

Figure 2:Illustration of the accuracy–cost trade\-off induced by a single threshold vs\. multiple thresholds\.For simplicity, we assume both classes have same number of samples, so accuracy is just the average\. Thexx\-axis is the threshold value and theyy\-axis represents accuracy \(increasing upward\), and cost \(increasing downward\)\. We assumeMax Acc\. 1to be the maximum accuracy achievable for class 1 with mLLM, whileMin Acc\. 1is the accuracy achieved with the sLLM\.Cost 1is cost from class 1 classification with different threshold values\.

When we have one single threshold,θc\\theta\_\{c\}, then we achieve an accuracy of12​\(Acc1,θc\+Acc2,θc\)\\frac\{1\}\{2\}\(\\mathrm\{Acc\}\_\{1\},\\theta\_\{c\}\+\\mathrm\{Acc\}\_\{2\},\\theta\_\{c\}\), but with two thresholds, we achieve higher accuracy, i\.e\.,12​\(Acc1,θ1\+Acc2,θ2\)\\frac\{1\}\{2\}\(\\mathrm\{Acc\}\_\{1\},\\theta\_\{1\}\+\\mathrm\{Acc\}\_\{2\},\\theta\_\{2\}\), while keeping the same total cost\.

## Appendix COptimality of multi\-thresholds policy over generalized policies

We generalize the framework of the main text by allowing arbitrary \(not specifically threshold\-based\) decision policies\. We work in the same classification setting and retain the same cost and error models, and allow the mLLM to make mistakes \(the oracle case follows naturally\)\. We show that, under a mild monotone\-risk assumption on the sLLM’s confidence score, the Bayes\-optimal deferral policies are again*per\-class single\-threshold policies*on a calibrated uncertainty score\. We give an alternative proof based on Lagrangian relaxation and Bayes\-optimal decision theory\.

##### Small model\.

For a given query, the small model outputs\(𝒂^s,β𝒂^s\)\(\\hat\{\\boldsymbol\{a\}\}^\{s\},\\beta\_\{\\hat\{\\boldsymbol\{a\}\}^\{s\}\}\), where𝒂^s∈\{1,…,K\}\\hat\{\\boldsymbol\{a\}\}^\{s\}\\in\\\{1,\\dots,K\\\}is the predicted class andβ𝒂^s∈\[0,1\]\\beta\_\{\\hat\{\\boldsymbol\{a\}\}^\{s\}\}\\in\[0,1\]is the scalar confidence for that class\. For each classkk, let

pk=Pr⁡\(𝒂^s=k\),fk​\(β\)=density of​βk∣𝒂^s=k,p\_\{k\}=\\Pr\(\\hat\{\\boldsymbol\{a\}\}^\{s\}=k\),\\qquad f\_\{k\}\(\\beta\)=\\text\{density of \}\\beta\_\{k\}\\mid\\hat\{\\boldsymbol\{a\}\}^\{s\}=k\\,,\(44\)and define the*class\-wise error curve*of sLLM

εs,k​\(β\)=Pr⁡\(𝒂≠k∣𝒂^s=k,βk=β\)\.\\varepsilon\_\{s,k\}\(\\beta\)=\\Pr\(\{\\boldsymbol\{a\}\}\\neq k\\mid\\hat\{\\boldsymbol\{a\}\}^\{s\}=k,\\beta\_\{k\}=\\beta\)\\,\.\(45\)

##### Master model\.

Let𝒂^m\\hat\{\\boldsymbol\{a\}\}^\{m\}denote the label predicted by mLLM when it is queried\. We assume that the error of mLLM is independent of sLLM:

εm,k​\(γ\)=Pr⁡\(𝒂≠k∣𝒂^m=k,γk=γ\),\{\\varepsilon\}\_\{m,k\}\(\\gamma\)=\\Pr\(\{\\boldsymbol\{a\}\}\\neq k\\mid\\hat\{\\boldsymbol\{a\}\}^\{m\}=k,\\gamma\_\{k\}=\\gamma\)\\,,\(46\)
and𝔼​\[εm,k​\(γ\)\]\\mathbb\{E\}\[\\varepsilon\_\{m,k\}\(\\gamma\)\]is treated as a constantε¯m,k\\bar\{\\varepsilon\}\_\{m,k\}\. Note that we do*not*assumeε¯m,k≤εs,k​\(β\)\\bar\{\\varepsilon\}\_\{m,k\}\\leq\\varepsilon\_\{s,k\}\(\\beta\)for allβ\\beta; mLLM may be better or worse than sLLM depending on the region of the score space\.

##### Deferral policies\.

A \(possibly randomized\)*deferral policy*π\\pimaps the observable pair\(𝒂^s,β𝒂^s\)\(\\hat\{\\boldsymbol\{a\}\}^\{s\},\\beta\_\{\\hat\{\\boldsymbol\{a\}\}^\{s\}\}\)to a deferral decisionD∈\{0,1\}D\\in\\\{0,1\\\}, whereD=0D=0means “accept sLLM” andD=1D=1means “defer to mLLM”\.

\- A deterministic policy is a measurable function

π:\{1,…,K\}×\[0,1\]→\{0,1\}\.\\pi:\\\{1,\\dots,K\\\}\\times\[0,1\]\\to\\\{0,1\\\}\\,\.\(47\)
\- A randomized policy can be seen as a measurable function

π:\{1,…,K\}×\[0,1\]→\[0,1\],\\pi:\\\{1,\\dots,K\\\}\\times\[0,1\]\\to\[0,1\]\\,,\(48\)whereπ​\(k,β\)\\pi\(k,\\beta\)is the probability of deferral when observing\(𝒂^s=k,βk=β\)\(\\hat\{\\boldsymbol\{a\}\}^\{s\}=k,\\beta\_\{k\}=\\beta\)\.

##### Performance metrics\.

For a given policyπ\\pi, define:

\- the*final prediction*𝒂^π\\hat\{\\boldsymbol\{a\}\}^\{\\pi\}as

𝒂^π=\{𝒂^s,D=0,𝒂^m,D=1,\\hat\{\\boldsymbol\{a\}\}^\{\\pi\}=\\begin\{cases\}\\hat\{\\boldsymbol\{a\}\}^\{s\},&D=0,\\\\ \\hat\{\\boldsymbol\{a\}\}^\{m\},&D=1\\,,\\end\{cases\}\(49\)
\- the*error indicator*

L=𝟙​\[𝒂^π≠𝒂\],L=\\mathds\{1\}\[\\hat\{\\boldsymbol\{a\}\}^\{\\pi\}\\neq\{\\boldsymbol\{a\}\}\]\\,,\(50\)
\- the*per\-sample cost*

C=cs\+cm​D\.C=c\_\{s\}\+c\_\{m\}D\\,\.\(51\)
Then the expected error and cost of policyπ\\piare

Error​\(π\)=𝔼​\[L\],Cost​\(π\)=𝔼​\[C\]\.\\mathrm\{Error\}\(\\pi\)=\\mathbb\{E\}\[L\],\\qquad\\mathrm\{Cost\}\(\\pi\)=\\mathbb\{E\}\[C\]\\,\.\(52\)
The optimization problem is:

minπ⁡Cost​\(π\)s\.t\.Error​\(π\)≤b,\\min\_\{\\pi\}\\ \\mathrm\{Cost\}\(\\pi\)\\quad\\text\{s\.t\.\}\\quad\\mathrm\{Error\}\(\\pi\)\\leq b,\(53\)
withb∈\[0,1\]b\\in\[0,1\]and the minimum is taken over all measurable policiesπ\\pibased on\(𝒂^s,β𝒂^s\)\(\\hat\{\\boldsymbol\{a\}\}^\{s\},\\beta\_\{\\hat\{\\boldsymbol\{a\}\}^\{s\}\}\)\.

Before restricting attention to threshold policies, we analyze the more general policy space using ideas from decision theory and convex analysis\.

### C\.1The risk–cost region and convexity

For any \(possibly randomized\) policyπ\\pi, define the*risk–cost vector*:

R​\(π\)=\(Error​\(π\),Cost​\(π\)\)∈ℝ2\.R\(\\pi\)=\\big\(\\mathrm\{Error\}\(\\pi\),\\;\\mathrm\{Cost\}\(\\pi\)\\big\)\\in\\mathbb\{R\}^\{2\}\.\(54\)
Letℛdet\\mathcal\{R\}\_\{\\mathrm\{det\}\}be the set of all such pairs obtained from deterministic policies, and letℛ\\mathcal\{R\}be the set obtained when randomization is allowed\.

###### Lemma C\.1\(Convexity via randomization\)\.

The setℛ\\mathcal\{R\}is the convex hull ofℛdet\\mathcal\{R\}\_\{\\mathrm\{det\}\}and is convex\. More precisely, for any two policiesπ1,π2\\pi\_\{1\},\\pi\_\{2\}and anyα∈\[0,1\]\\alpha\\in\[0,1\], define the randomized policyπα\\pi\_\{\\alpha\}that drawsU∼Bernoulli​\(α\)U\\sim\\mathrm\{Bernoulli\}\(\\alpha\)once and usesπ1\\pi\_\{1\}ifU=1U=1andπ2\\pi\_\{2\}ifU=0U=0for all samples\. Then:

R​\(πα\)=α​R​\(π1\)\+\(1−α\)​R​\(π2\)\.R\(\\pi\_\{\\alpha\}\)=\\alpha R\(\\pi\_\{1\}\)\+\(1\-\\alpha\)R\(\\pi\_\{2\}\)\.

###### Proof\.

The equality follows from linearity of expectation:

Error​\(πα\)=𝔼​\[𝟙​\[𝒂^πα≠𝒂\]\]=α​Error​\(π1\)\+\(1−α\)​Error​\(π2\),\\mathrm\{Error\}\(\\pi\_\{\\alpha\}\)=\\mathbb\{E\}\[\\mathds\{1\}\[\\hat\{\\boldsymbol\{a\}\}^\{\\pi\_\{\\alpha\}\}\\neq\{\\boldsymbol\{a\}\}\]\]=\\alpha\\,\\mathrm\{Error\}\(\\pi\_\{1\}\)\+\(1\-\\alpha\)\\,\\mathrm\{Error\}\(\\pi\_\{2\}\),\(55\)and similarly forCost​\(πα\)\\mathrm\{Cost\}\(\\pi\_\{\\alpha\}\)\. Thusℛ\\mathcal\{R\}is closed under convex combinations\. Since deterministic policies are particular randomized policies, we haveℛ=conv​\(ℛdet\)\\mathcal\{R\}=\\mathrm\{conv\}\(\\mathcal\{R\}\_\{\\mathrm\{det\}\}\)\. ∎

Geometrically,ℛ\\mathcal\{R\}is a closed convex subset ofℝ2\\mathbb\{R\}^\{2\}\. A policyπ\\piis \(weakly\) Pareto\-optimal if its risk–cost pair\(e,c\)\(e,c\)lies on the lower\-left boundary ofℛ\\mathcal\{R\}, i\.e\. if there is no other policyπ′\\pi^\{\\prime\}withErr​\(π′\)≤e\\mathrm\{Err\}\(\\pi^\{\\prime\}\)\\leq e,Cost​\(π′\)≤c\\mathrm\{Cost\}\(\\pi^\{\\prime\}\)\\leq c, and at least one strict inequality\.

### C\.2Lagrangian formulation

Introducing a Lagrange multiplierλ≥0\\lambda\\geq 0, we define the Lagrangian of the constrained problem \([53](https://arxiv.org/html/2609.15992#A3.E53)\):

ℒ​\(π,λ\)=Cost​\(π\)\+λ​\(Error​\(π\)−b\)\.\\mathcal\{L\}\(\\pi,\\lambda\)=\\mathrm\{Cost\}\(\\pi\)\+\\lambda\\big\(\\mathrm\{Error\}\(\\pi\)\-b\\big\)\\,\.\(56\)
For fixedλ\\lambda, minimizingℒ​\(π,λ\)\\mathcal\{L\}\(\\pi,\\lambda\)overπ\\piis equivalent to minimizing

Jλ​\(π\)=𝔼​\[cs\+cm​D\+λ​L\],J\_\{\\lambda\}\(\\pi\)=\\mathbb\{E\}\\big\[c\_\{s\}\+c\_\{m\}D\+\\lambda L\\big\]\\,,\(57\)since the term−λ​b\-\\lambda bdoes not depend onπ\\pi\. The functionalJλJ\_\{\\lambda\}is the risk of policyπ\\piunder the loss functionℓλ\\ell\_\{\\lambda\}:

ℓλ​\(D,𝒂,𝒂^s,𝒂^m\)=\{cs\+λ​1​\[𝒂^s≠𝒂\]if​D=0​\(accept\),cs\+cm\+λ​1​\[𝒂^m≠𝒂\]if​D=1​\(defer\)\.\\ell\_\{\\lambda\}\(D,\{\\boldsymbol\{a\}\},\\hat\{\\boldsymbol\{a\}\}^\{s\},\\hat\{\\boldsymbol\{a\}\}^\{m\}\)=\\begin\{cases\}c\_\{s\}\+\\lambda\\,\\mathds\{1\}\[\\hat\{\\boldsymbol\{a\}\}^\{s\}\\neq\{\\boldsymbol\{a\}\}\]&\\text\{if \}D=0\\ \(\\text\{accept\}\),\\\\ c\_\{s\}\+c\_\{m\}\+\\lambda\\,\\mathds\{1\}\[\\hat\{\\boldsymbol\{a\}\}^\{m\}\\neq\{\\boldsymbol\{a\}\}\]&\\text\{if \}D=1\\ \(\\text\{defer\}\)\\,\.\\end\{cases\}\(58\)
##### Connection between the constrained and Lagrangian problems\.

By Lemma[C\.1](https://arxiv.org/html/2609.15992#A3.Thmtheorem1),ℛ\\mathcal\{R\}is convex\. The objective in \([53](https://arxiv.org/html/2609.15992#A3.E53)\) is to find, among all\(e,c\)∈ℛ\(e,c\)\\in\\mathcal\{R\}withe≤be\\leq b, a point with minimalcc\. Any such point on the lower\-left boundary ofℛ\\mathcal\{R\}is Pareto\-optimal\. The supporting hyperplane theorem implies that for every Pareto\-optimal\(e∗,c∗\)∈ℛ\(e^\{\*\},c^\{\*\}\)\\in\\mathcal\{R\}there exists someλ≥0\\lambda\\geq 0such that\(e∗,c∗\)\(e^\{\*\},c^\{\*\}\)minimizes the linear functional\(e,c\)↦c\+λ​e\(e,c\)\\mapsto c\+\\lambda eoverℛ\\mathcal\{R\}\. Translating back to policies, this means:

###### Proposition C\.2\(Pareto points are Lagrangian minimizers\)\.

Letπ∗\\pi^\{\*\}be a policy such thatR​\(π∗\)R\(\\pi^\{\*\}\)is Pareto\-optimal inℛ\\mathcal\{R\}\. Then there existsλ≥0\\lambda\\geq 0such that

π∗∈arg⁡minπ⁡Jλ​\(π\)=arg⁡minπ⁡𝔼​\[cs\+cm​D\+λ​L\]\.\\pi^\{\*\}\\in\\arg\\min\_\{\\pi\}J\_\{\\lambda\}\(\\pi\)=\\arg\\min\_\{\\pi\}\\mathbb\{E\}\[c\_\{s\}\+c\_\{m\}D\+\\lambda L\]\.

Thus, to characterize the solutions of the original constrained problem \([53](https://arxiv.org/html/2609.15992#A3.E53)\), it suffices to characterize the minimizers ofJλJ\_\{\\lambda\}for eachλ\\lambda\.

### C\.3Bayes\-optimal policy

We now fixλ\>0\\lambda\>0and determine the policies that minimizeJλJ\_\{\\lambda\}\. The*conditional risks*of the two actions at a given pair\(𝒂^s=k,βk=β\)\(\\hat\{\\boldsymbol\{a\}\}^\{s\}=k,\\beta\_\{k\}=\\beta\)are defined as:

\- If weacceptsLLM \(D=0D=0\),

rs​\(k,β\)=𝔼​\[cs\+λ​L∣𝒂^s=k,βk=β,D=0\]=cs\+λ​εs,k​\(β\)\.r\_\{s\}\(k,\\beta\)=\\mathbb\{E\}\[c\_\{s\}\+\\lambda L\\mid\\hat\{\\boldsymbol\{a\}\}^\{s\}=k,\\beta\_\{k\}=\\beta,D=0\]=c\_\{s\}\+\\lambda\\varepsilon\_\{s,k\}\(\\beta\)\\,\.\(59\)
\- If wedeferto mLLM \(D=1D=1\),

rm=𝔼​\[cs\+cm\+λ​L∣D=1\]=cs\+cm\+λ​ε¯m,k\.r\_\{m\}=\\mathbb\{E\}\[c\_\{s\}\+c\_\{m\}\+\\lambda L\\mid D=1\]=c\_\{s\}\+c\_\{m\}\+\\lambda\\bar\{\\varepsilon\}\_\{m,k\}\\,\.\(60\)
A policyπ\\piinduces a decisionD=π​\(𝒂^s,β𝒂^s\)D=\\pi\(\\hat\{\\boldsymbol\{a\}\}^\{s\},\\beta\_\{\\hat\{\\boldsymbol\{a\}\}^\{s\}\}\)\. The risk can be written as

Jλ​\(π\)=𝔼​\[rs​\(𝒂^s,β𝒂^s\)​1​\[D=0\]\+rm​1​\[D=1\]\]\.J\_\{\\lambda\}\(\\pi\)=\\mathbb\{E\}\[r\_\{s\}\(\\hat\{\\boldsymbol\{a\}\}^\{s\},\\beta\_\{\\hat\{\\boldsymbol\{a\}\}^\{s\}\}\)\\,\\mathds\{1\}\[D=0\]\+r\_\{m\}\\,\\mathds\{1\}\[D=1\]\]\.\(61\)
###### Proposition C\.3\(Bayes\-optimal deferral rule\)\.

For fixedλ\>0\\lambda\>0, define the policyπλ∗\\pi^\{\*\}\_\{\\lambda\}by

πλ∗​\(k,β\)=\{0if​rs​\(k,β\)≤rm,1if​rs​\(k,β\)\>rm\.\\pi^\{\*\}\_\{\\lambda\}\(k,\\beta\)=\\begin\{cases\}0&\\text\{if \}r\_\{s\}\(k,\\beta\)\\leq r\_\{m\},\\\\ 1&\\text\{if \}r\_\{s\}\(k,\\beta\)\>r\_\{m\}\\,\.\\end\{cases\}\(62\)Thenπλ∗\\pi^\{\*\}\_\{\\lambda\}is a global minimizer ofJλJ\_\{\\lambda\}, i\.e\.,

Jλ​\(πλ∗\)≤Jλ​\(π\)for all policiesπ\.J\_\{\\lambda\}\(\\pi^\{\*\}\_\{\\lambda\}\)\\leq J\_\{\\lambda\}\(\\pi\)\\quad\\text\{for all policies $\\pi$\}\\,\.\(63\)

###### Proof\.

We show thatπλ∗\\pi^\{\*\}\_\{\\lambda\}is a global minimizer ofJλJ\_\{\\lambda\}over all policies\. The argument is pointwise and uses only the conditional risksrsr\_\{s\}andrmr\_\{m\}\. Recall that for a given observation\(𝒂^s,β𝒂^s\)=\(k,β\)\(\\hat\{\\boldsymbol\{a\}\}^\{s\},\\beta\_\{\\hat\{\\boldsymbol\{a\}\}^\{s\}\}\)=\(k,\\beta\), the conditional risks of the two actions are

rs​\(k,β\)=cs\+λ​εs,k​\(β\)andrm=cs\+cm\+λ​ε¯m,k\.r\_\{s\}\(k,\\beta\)=c\_\{s\}\+\\lambda\\varepsilon\_\{s,k\}\(\\beta\)\\quad\\text\{and\}\\quad r\_\{m\}=c\_\{s\}\+c\_\{m\}\+\\lambda\\bar\{\\varepsilon\}\_\{m,k\}\\,\.\(64\)Any \(possibly randomized\) policyπ\\piinduces a decisionD∈\{0,1\}D\\in\\\{0,1\\\}at\(k,β\)\(k,\\beta\), whereD=0D=0means “accept” andD=1D=1means “defer”\. The associated conditional contribution to the Lagrangian loss is

R​\(k,β;D\)=𝔼​\[cs\+cm​D\+λ​L∣𝒂^s=k,βk=β,D\]=\{rs​\(k,β\)if​D=0,rmif​D=1R\(k,\\beta;D\)=\\mathbb\{E\}\[c\_\{s\}\+c\_\{m\}D\+\\lambda L\\mid\\hat\{\\boldsymbol\{a\}\}^\{s\}=k,\\beta\_\{k\}=\\beta,D\]=\\begin\{cases\}r\_\{s\}\(k,\\beta\)&\\text\{if \}D=0,\\\\ r\_\{m\}&\\text\{if \}D=1\\end\{cases\}\(65\)
By definition, the Bayes policyπλ∗\\pi^\{\*\}\_\{\\lambda\}chooses, at each\(k,β\)\(k,\\beta\),

D∗:=πλ∗​\(k,β\)=\{0if​rs​\(k,β\)≤rm,1if​rs​\(k,β\)\>rmD^\{\*\}:=\\pi^\{\*\}\_\{\\lambda\}\(k,\\beta\)=\\begin\{cases\}0&\\text\{if \}r\_\{s\}\(k,\\beta\)\\leq r\_\{m\},\\\\ 1&\\text\{if \}r\_\{s\}\(k,\\beta\)\>r\_\{m\}\\end\{cases\}\(66\)
Letπ′\\pi^\{\\prime\}be any other policy, and letD′=π​\(k,β\)D^\{\\prime\}=\\pi\(k,\\beta\)be its decision at\(k,β\)\(k,\\beta\)\. The excess conditional loss is defined as

Δ​\(k,β\)=R​\(k,β;D′​\(k,β\)\)−R​\(k,β;D∗​\(k,β\)\)\.\\Delta\(k,\\beta\)=R\(k,\\beta;D^\{\\prime\}\(k,\\beta\)\)\-R\(k,\\beta;D^\{\*\}\(k,\\beta\)\)\.\(67\)
There are three exhaustive cases:

1. 1\.*Case 1:π′\\pi^\{\\prime\}andπλ∗\\pi^\{\*\}\_\{\\lambda\}choose the same action\.* ThenD′​\(k,β\)=D∗​\(k,β\)D^\{\\prime\}\(k,\\beta\)=D^\{\*\}\(k,\\beta\)and hence Δ​\(k,β\)=0\.\\Delta\(k,\\beta\)=0\.
2. 2\.*Case 2:π′\\pi^\{\\prime\}accepts whileπλ∗\\pi^\{\*\}\_\{\\lambda\}defers\.* HereD′​\(k,β\)=0D^\{\\prime\}\(k,\\beta\)=0andD∗​\(k,β\)=1D^\{\*\}\(k,\\beta\)=1\. By the definition ofπλ∗\\pi^\{\*\}\_\{\\lambda\},D∗​\(k,β\)=1D^\{\*\}\(k,\\beta\)=1impliesrm≤rs​\(k,β\)r\_\{m\}\\leq r\_\{s\}\(k,\\beta\)\. Therefore, Δ​\(k,β\)=R​\(k,β;0\)−R​\(k,β;1\)=rS​\(k,β\)−rM​\(k\)≥0,\\Delta\(k,\\beta\)=R\(k,\\beta;0\)\-R\(k,\\beta;1\)=r\_\{S\}\(k,\\beta\)\-r\_\{M\}\(k\)\\;\\geq 0,
3. 3\.*Case 3:π′\\pi^\{\\prime\}defers whileπλ∗\\pi^\{\*\}\_\{\\lambda\}accepts\.* HereD′​\(k,β\)=1D^\{\\prime\}\(k,\\beta\)=1andD∗​\(k,β\)=0D^\{\*\}\(k,\\beta\)=0\. By the definition ofπλ∗\\pi^\{\*\}\_\{\\lambda\},D∗​\(k,β\)=0D^\{\*\}\(k,\\beta\)=0impliesrs​\(k,β\)≤rmr\_\{s\}\(k,\\beta\)\\leq r\_\{m\}\. Therefore Δ​\(k,β\)=R​\(k,β;1\)−R​\(k,β;0\)=rm−rs​\(k,β\)≥0,\\Delta\(k,\\beta\)=R\(k,\\beta;1\)\-R\(k,\\beta;0\)=r\_\{m\}\-r\_\{s\}\(k,\\beta\)\\;\\geq 0,

In all three cases we haveΔ​\(k,β\)≥0\\Delta\(k,\\beta\)\\geq 0, andΔ​\(k,β\)\>0\\Delta\(k,\\beta\)\>0wheneverπ\\piandπλ∗\\pi^\{\*\}\_\{\\lambda\}choose different actions at a point wherers​\(k,β\)≠rmr\_\{s\}\(k,\\beta\)\\neq r\_\{m\}\.

Integrating w\.r\.t\. the joint distribution of\(𝒂^s,β𝒂^s\)\(\\hat\{\\boldsymbol\{a\}\}^\{s\},\\beta\_\{\\hat\{\\boldsymbol\{a\}\}^\{s\}\}\)gives:

Jλ​\(π′\)−Jλ​\(πλ∗\)=𝔼​\[R​\(𝒂^s,β𝒂^s;D′\)−R​\(𝒂^s,β𝒂^s;D∗\)\]=𝔼​\[Δ​\(𝒂^s,β𝒂^s\)\]≥0\.J\_\{\\lambda\}\(\\pi^\{\\prime\}\)\-J\_\{\\lambda\}\(\\pi^\{\*\}\_\{\\lambda\}\)=\\mathbb\{E\}\\big\[R\(\\hat\{\\boldsymbol\{a\}\}^\{s\},\\beta\_\{\\hat\{\\boldsymbol\{a\}\}^\{s\}\};D^\{\\prime\}\)\-R\(\\hat\{\\boldsymbol\{a\}\}^\{s\},\\beta\_\{\\hat\{\\boldsymbol\{a\}\}^\{s\}\};D^\{\*\}\)\\big\]=\\mathbb\{E\}\[\\Delta\(\\hat\{\\boldsymbol\{a\}\}^\{s\},\\beta\_\{\\hat\{\\boldsymbol\{a\}\}^\{s\}\}\)\]\\;\\geq 0\.\(68\)
Thus no policyπ\\pican have strictly smaller Lagrangian risk thanπλ∗\\pi^\{\*\}\_\{\\lambda\}, and any policy with the same risk must agree withπλ∗\\pi^\{\*\}\_\{\\lambda\}almost surely except possibly on the set\{\(k,β\):rs​\(k,β\)=rm\}\\\{\(k,\\beta\):r\_\{s\}\(k,\\beta\)=r\_\{m\}\\\}\. This proves thatπλ∗\\pi^\{\*\}\_\{\\lambda\}is a \(global\) Bayes\-optimal minimizer ofJλJ\_\{\\lambda\}\. ∎

The decision rule ofπλ∗\\pi^\{\*\}\_\{\\lambda\}can be written explicitly\. The inequalityrs​\(k,β\)≤rmr\_\{s\}\(k,\\beta\)\\leq r\_\{m\}is

cs\+λ​εs,k​\(β\)≤cs\+cm\+λ​ε¯m,k⟺εs,k​\(β\)−ε¯m,k≤cmλ\.c\_\{s\}\+\\lambda\\varepsilon\_\{s,k\}\(\\beta\)\\;\\leq\\;c\_\{s\}\+c\_\{m\}\+\\lambda\\bar\{\\varepsilon\}\_\{m,k\}\\quad\\Longleftrightarrow\\quad\\varepsilon\_\{s,k\}\(\\beta\)\-\\bar\{\\varepsilon\}\_\{m,k\}\\leq\\frac\{c\_\{m\}\}\{\\lambda\}\.\(69\)
Thus,

πλ∗​\(k,β\)=\{0if​εs,k​\(β\)−ε¯m,k≤cmλ,1otherwise\.\\pi^\{\*\}\_\{\\lambda\}\(k,\\beta\)=\\begin\{cases\}0&\\text\{if \}\\varepsilon\_\{s,k\}\(\\beta\)\-\\bar\{\\varepsilon\}\_\{m,k\}\\leq\\dfrac\{c\_\{m\}\}\{\\lambda\},\\\\ 1&\\text\{otherwise\}\.\\end\{cases\}\(70\)
Equivalently:

sLLM is accepted at\(k,β\)iffΔk​\(β\)≤cmλ,\\text\{sLLM is accepted at $\(k,\\beta\)$ iff\}\\quad\\Delta\_\{k\}\(\\beta\)\\leq\\frac\{c\_\{m\}\}\{\\lambda\},\(71\)whereΔk​\(β\)=εs,k​\(β\)−ε¯m,k\\Delta\_\{k\}\(\\beta\)=\\varepsilon\_\{s,k\}\(\\beta\)\-\\bar\{\\varepsilon\}\_\{m,k\}is the*excess error*of sLLM over mLLM\.

##### Interpretation\.

The Bayes\-optimal policy accepts the small model at\(k,β\)\(k,\\beta\)whenever its excess error over the master is smaller than the cost\-normalized margincm/λc\_\{m\}/\\lambda; it defers if sLLM is sufficiently worse than mLLM to justify paying the additional costcmc\_\{m\}\.

### C\.4From Lagrangian minimizers back to the constrained problem

Combining Proposition[C\.2](https://arxiv.org/html/2609.15992#A3.Thmtheorem2)and Proposition[C\.3](https://arxiv.org/html/2609.15992#A3.Thmtheorem3), we obtain:

###### Theorem C\.4\(Lagrangian characterization of optimal policies\)\.

Letπ∗\\pi^\{\*\}be a policy whose risk–cost vectorR​\(π∗\)R\(\\pi^\{\*\}\)is Pareto\-optimal inℛ\\mathcal\{R\}\. Then there existsλ≥0\\lambda\\geq 0such thatπ∗\\pi^\{\*\}coincides with the Bayes ruleπλ∗\\pi^\{\*\}\_\{\\lambda\}, i\.e\.,

π∗​\(k,β\)=\{0if​εs,k​\(β\)−ε¯m,k≤cm/λ,1otherwise\.\\pi^\{\*\}\(k,\\beta\)=\\begin\{cases\}0&\\text\{if \}\\varepsilon\_\{s,k\}\(\\beta\)\-\\bar\{\\varepsilon\}\_\{m,k\}\\leq c\_\{m\}/\\lambda,\\\\ 1&\\text\{otherwise\}\.\\end\{cases\}

In particular, any optimal solution of the constrained problem \([53](https://arxiv.org/html/2609.15992#A3.E53)\) that lies on the error–cost frontier must accept sLLM exactly on the set

Aλ=\{\(k,β\):εs,k​\(β\)−ε¯m,k≤cm/λ\}A\_\{\\lambda\}=\\\{\(k,\\beta\):\\varepsilon\_\{s,k\}\(\\beta\)\-\\bar\{\\varepsilon\}\_\{m,k\}\\leq c\_\{m\}/\\lambda\\\}for someλ≥0\\lambda\\geq 0\.

At this stage, we have characterized the optimal acceptance region in terms of a*threshold in excess error*Δk​\(β\)\\Delta\_\{k\}\(\\beta\)\. We now show how a monotone\-risk assumption allows us to representAλA\_\{\\lambda\}as a per\-class*single threshold inβ\\beta*\.

### C\.5Monotone per\-class error curves

For each classkk, consider the mapβ↦εs,k​\(β\)=Pr⁡\(𝒂≠k∣𝒂^s=k,βk=β\)\.\\beta\\;\\mapsto\\;\\varepsilon\_\{s,k\}\(\\beta\)=\\Pr\(\\boldsymbol\{a\}\\neq k\\mid\\hat\{\\boldsymbol\{a\}\}^\{s\}=k,\\beta\_\{k\}=\\beta\)\.

We say that the per\-class scoreβk\\beta\_\{k\}satisfies the*monotone\-risk property*if, for eachkk, this map is \(essentially\) non\-increasing on\[0,1\]\[0,1\]:

β1≤β2⇒εks​\(β1\)≥εks​\(β2\)for almost allβ1,β2\.\\beta\_\{1\}\\leq\\beta\_\{2\}\\;\\Rightarrow\\;\\varepsilon\_\{k\}^\{s\}\(\\beta\_\{1\}\)\\geq\\varepsilon\_\{k\}^\{s\}\(\\beta\_\{2\}\)\\quad\\text\{for almost all $\\beta\_\{1\},\\beta\_\{2\}$\}\.\(72\)
Equivalently, the correctness probability

Pr⁡\(𝒂=k∣𝒂^s=k,βk=β\)=1−εs,k​\(β\)\\Pr\(\\boldsymbol\{a\}=k\\mid\\hat\{\\boldsymbol\{a\}\}^\{s\}=k,\\beta\_\{k\}=\\beta\)=1\-\\varepsilon\_\{s,k\}\(\\beta\)is non\-decreasing inβ\\beta: higherβ\\betameans higher probability that the prediction is correct, within classkk\.

### C\.6From error threshold to confidence\-threshold

The Bayes acceptance condition \([71](https://arxiv.org/html/2609.15992#A3.E71)\) can be written as

Δk​\(β\)≤cm/λ⟺εs,k​\(β\)≤ε¯m,k\+cmλ=tk​\(λ\)\.\\Delta\_\{k\}\(\\beta\)\\leq c\_\{m\}/\\lambda\\quad\\Longleftrightarrow\\quad\\varepsilon\_\{s,k\}\(\\beta\)\\leq\\bar\{\\varepsilon\}\_\{m,k\}\+\\frac\{c\_\{m\}\}\{\\lambda\}=t\_\{k\}\(\\lambda\)\.\(73\)
For anyt∈ℝt\\in\\mathbb\{R\}, define the sublevel set

Sk​\(t\):=\{β∈\[0,1\]:εs,k​\(β\)≤t\}\.S\_\{k\}\(t\):=\\\{\\beta\\in\[0,1\]:\\varepsilon\_\{s,k\}\(\\beta\)\\leq t\\\}\\,\.
Under Assumption \([72](https://arxiv.org/html/2609.15992#A3.E72)\), eachSk​\(t\)S\_\{k\}\(t\)is either empty, all of\[0,1\]\[0,1\], or an interval of the form\[θ,1\]\[\\theta,1\]\. More precisely:

###### Lemma C\.5\(Structure of sublevel sets under monotone risk\)\.

Supposeεs,k:\[0,1\]→ℝ\\varepsilon\_\{s,k\}:\[0,1\]\\to\\mathbb\{R\}is non\-increasing\. For anyt∈ℝt\\in\\mathbb\{R\}, the setSk​\(t\)S\_\{k\}\(t\)is either empty,\[0,1\]\[0,1\], or an interval whose left endpoint isθ=infSk​\(t\)\\theta=\\inf\\,S\_\{k\}\(t\)\. Specifically,Sk​\(t\)=\(θ,1\]S\_\{k\}\(t\)=\(\\theta,1\]orSk​\(t\)=\[θ,1\]S\_\{k\}\(t\)=\[\\theta,1\]\.

###### Proof\.

IfSk​\(t\)=∅S\_\{k\}\(t\)=\\varnothingorSk​\(t\)=\[0,1\]S\_\{k\}\(t\)=\[0,1\], the conclusion follows immediately\.

Assume now thatSk​\(t\)S\_\{k\}\(t\)is non\-empty and not all of\[0,1\]\[0,1\]\. Defineθ=infSk​\(t\)\\theta=\\inf S\_\{k\}\(t\)\. SinceSk​\(t\)S\_\{k\}\(t\)is a non\-empty subset of\[0,1\]\[0,1\], we haveθ∈\[0,1\]\\theta\\in\[0,1\]\.

Letβ\>θ\\beta\>\\theta\. By definition of the infimum, there existsβ′∈Sk​\(t\)\\beta^\{\\prime\}\\in S\_\{k\}\(t\)such thatθ≤β′<β\\theta\\leq\\beta^\{\\prime\}<\\beta\. Becauseεs,k\\varepsilon\_\{s,k\}is non\-increasing,εs,k​\(β\)≤εs,k​\(β′\)≤t\\varepsilon\_\{s,k\}\(\\beta\)\\leq\\varepsilon\_\{s,k\}\(\\beta^\{\\prime\}\)\\leq t\. Henceβ∈Sk​\(t\)\\beta\\in S\_\{k\}\(t\), so\(θ,1\]⊆Sk​\(t\)\(\\theta,1\]\\subseteq S\_\{k\}\(t\)\.

Ifβ<θ\\beta<\\theta, thenβ\\betais less than the infimum ofSk​\(t\)S\_\{k\}\(t\), soβ∉Sk​\(t\)\\beta\\notin S\_\{k\}\(t\)\. ThusSk​\(t\)⊆\[θ,1\]S\_\{k\}\(t\)\\subseteq\[\\theta,1\]\.

Combining these inclusions, we haveSk​\(t\)=\(θ,1\]S\_\{k\}\(t\)=\(\\theta,1\]ifθ∉Sk​\(t\)\\theta\\notin S\_\{k\}\(t\), andSk​\(t\)=\[θ,1\]S\_\{k\}\(t\)=\[\\theta,1\]ifθ∈Sk​\(t\)\\theta\\in S\_\{k\}\(t\)\.

Ifεs,k​\(⋅\)\\varepsilon\_\{s,k\}\(\\cdot\)is additionally right\-continuous or continuous, then the existence of a sequenceβn↓θ\\beta\_\{n\}\\downarrow\\thetawithεks​\(βn\)≤t\\varepsilon\_\{k\}^\{s\}\(\\beta\_\{n\}\)\\leq timpliesεs,k​\(θ\)≤t\\varepsilon\_\{s,k\}\(\\theta\)\\leq t, so in that caseSk​\(t\)=\[θ,1\]S\_\{k\}\(t\)=\[\\theta,1\]always holds\. ∎

Applying Lemma[C\.5](https://arxiv.org/html/2609.15992#A3.Thmtheorem5)totk​\(λ\)=ε¯m,k\+cm/λt\_\{k\}\(\\lambda\)=\\bar\{\\varepsilon\}\_\{m,k\}\+c\_\{m\}/\\lambdashows that, for eachkk, the acceptance set inβ\\betais \(up to null sets\) an interval of the form\[θk​\(λ\),1\]\[\\theta\_\{k\}\(\\lambda\),1\]or\(θk​\(λ\),1\]\(\\theta\_\{k\}\(\\lambda\),1\]\. That is, there exists a scalar thresholdθk​\(λ\)∈\[0,1\]\\theta\_\{k\}\(\\lambda\)\\in\[0,1\]such that:

πλ∗​\(k,β\)=\{0β\>θk​\(λ\),1β<θk​\(λ\)\\pi\_\{\\lambda\}^\{\*\}\(k,\\beta\)=\\begin\{cases\}0&\\beta\>\\theta\_\{k\}\(\\lambda\),\\\\ 1&\\beta<\\theta\_\{k\}\(\\lambda\)\\end\{cases\}\(74\)
and the decision atβ=θk​\(λ\)\\beta=\\theta\_\{k\}\(\\lambda\)is either 0 or 1 without affecting the Bayes risk\. If, in addition,εs,k\\varepsilon\_\{s,k\}is right\-continuous or continuous atθk​\(λ\)\\theta\_\{k\}\(\\lambda\), then the acceptance set is exactly\[θk​\(λ\),1\]\[\\theta\_\{k\}\(\\lambda\),1\]\.

Thus, under the monotone\-risk assumption, the Bayes\-optimal policies for the Lagrangian problem are*per\-class single\-threshold policies inβ\\beta*\.

Now that we know that optimal policies are of threshold form inβ\\beta, we can parameterize them by𝜽∈\[0,1\]K\\boldsymbol\{\\theta\}\\in\[0,1\]^\{K\}as in the oracle framework, and compute the corresponding error and cost integrals\.

Following the multi\-thresholds policy definition in Section[3\.2](https://arxiv.org/html/2609.15992#S3.SS2), we obtain

Error​\(𝜽\)\\displaystyle\\mathrm\{Error\}\(\\boldsymbol\{\\theta\}\)=∑k=1Kpk​\[∫θk1εs,k​\(β\)​fk​\(β\)​𝑑β\+ε¯m,k​∫0θkfk​\(β\)​𝑑β\],\\displaystyle=\\sum\_\{k=1\}^\{K\}p\_\{k\}\\bigg\[\\int\_\{\\theta\_\{k\}\}^\{1\}\\varepsilon\_\{s,k\}\(\\beta\)f\_\{k\}\(\\beta\)\\,d\\beta\+\\bar\{\\varepsilon\}\_\{m,k\}\\int\_\{0\}^\{\\theta\_\{k\}\}f\_\{k\}\(\\beta\)\\,d\\beta\\bigg\]\\,,\(75\)Cost​\(𝜽\)\\displaystyle\\mathrm\{Cost\}\(\\boldsymbol\{\\theta\}\)=cs\+cm​∑k=1Kpk​∫0θkfk​\(β\)​𝑑β\.\\displaystyle=c\_\{s\}\+c\_\{m\}\\sum\_\{k=1\}^\{K\}p\_\{k\}\\int\_\{0\}^\{\\theta\_\{k\}\}f\_\{k\}\(\\beta\)\\,d\\beta\\,\.\(76\)

### C\.7Lagrangian in terms of𝜽\\boldsymbol\{\\theta\}

Forλ≥0\\lambda\\geq 0, define

Lλ​\(𝜽\):=Cost​\(𝜽\)\+λ​\(Error​\(𝜽\)−b\)\.L\_\{\\lambda\}\(\\boldsymbol\{\\theta\}\):=\\mathrm\{Cost\}\(\\boldsymbol\{\\theta\}\)\+\\lambda\\big\(\\mathrm\{Error\}\(\\boldsymbol\{\\theta\}\)\-b\\big\)\.\(77\)
Up to constants independent ofθ\\theta, we can writeLλ​\(θ\)L\_\{\\lambda\}\(\\theta\)as

Lλ​\(θ\)=cs\+λ​\(∑k=1Kpk​εM,k−b\)\+∑k=1Kpk​\[cm​∫0θkfk​\(β\)​𝑑β\+λ​∫θk1Δk​\(β\)​fk​\(β\)​𝑑β\],L\_\{\\lambda\}\(\\theta\)=c\_\{s\}\+\\lambda\\Big\(\\sum\_\{k=1\}^\{K\}p\_\{k\}\\varepsilon\_\{M,k\}\-b\\Big\)\+\\sum\_\{k=1\}^\{K\}p\_\{k\}\\Big\[c\_\{m\}\\int\_\{0\}^\{\\theta\_\{k\}\}f\_\{k\}\(\\beta\)\\,d\\beta\+\\lambda\\int\_\{\\theta\_\{k\}\}^\{1\}\\Delta\_\{k\}\(\\beta\)f\_\{k\}\(\\beta\)\\,d\\beta\\Big\],\(78\)whereΔk​\(β\)=εs,k​\(β\)−ε¯m,k\\Delta\_\{k\}\(\\beta\)=\\varepsilon\_\{s,k\}\(\\beta\)\-\\bar\{\\varepsilon\}\_\{m,k\}\.

Differentiating w\.r\.t\.θk\\theta\_\{k\}gives

∂∂θk​\(cm​pk​∫0θkfk​\(β\)​𝑑β\)\\displaystyle\\frac\{\\partial\}\{\\partial\\theta\_\{k\}\}\\Big\(c\_\{m\}p\_\{k\}\\int\_\{0\}^\{\\theta\_\{k\}\}f\_\{k\}\(\\beta\)\\,d\\beta\\Big\)=cm​pk​fk​\(θk\),\\displaystyle=c\_\{m\}p\_\{k\}f\_\{k\}\(\\theta\_\{k\}\)\\,,\(79\)∂∂θk​\(λ​pk​∫θk1Δk​\(β\)​fk​\(β\)​𝑑β\)\\displaystyle\\frac\{\\partial\}\{\\partial\\theta\_\{k\}\}\\Big\(\\lambda p\_\{k\}\\int\_\{\\theta\_\{k\}\}^\{1\}\\Delta\_\{k\}\(\\beta\)f\_\{k\}\(\\beta\)\\,d\\beta\\Big\)=−λ​pk​Δk​\(θk\)​fk​\(θk\)\.\\displaystyle=\-\\lambda p\_\{k\}\\Delta\_\{k\}\(\\theta\_\{k\}\)f\_\{k\}\(\\theta\_\{k\}\)\\,\.\(80\)
Assumingpk\>0p\_\{k\}\>0andfk​\(θk\)\>0f\_\{k\}\(\\theta\_\{k\}\)\>0, the stationarity condition∂Lλ/∂θk=0\\partial L\_\{\\lambda\}/\\partial\\theta\_\{k\}=0yields

Δk​\(θk∗\)=cmλfor all​k​with​0<θk∗<1\.\\Delta\_\{k\}\(\\theta\_\{k\}^\{\*\}\)=\\frac\{c\_\{m\}\}\{\\lambda\}\\quad\\text\{for all \}k\\text\{ with \}0<\\theta\_\{k\}^\{\*\}<1\\,\.\(81\)
Thus, at any interior optimum, the*excess error*of sLLM over mLLM at the thresholds is equalized across all classeskk:

εs,k​\(θk∗\)−ε¯m,k=εs,k′​\(θk′∗\)−ε¯m,k′∀k,k′​with​0<θk∗,θk′∗<1\.\\varepsilon\_\{s,k\}\(\\theta\_\{k\}^\{\*\}\)\-\\bar\{\\varepsilon\}\_\{m,k\}=\\varepsilon\_\{s,k^\{\\prime\}\}\(\\theta\_\{k^\{\\prime\}\}^\{\*\}\)\-\\bar\{\\varepsilon\}\_\{m,k^\{\\prime\}\}\\quad\\forall\\,k,k^\{\\prime\}\\text\{ with \}0<\\theta\_\{k\}^\{\*\},\\theta\_\{k^\{\\prime\}\}^\{\*\}<1\.
In the oracle case,ε¯m,k=0\\bar\{\\varepsilon\}\_\{m,k\}=0, so \([81](https://arxiv.org/html/2609.15992#A3.E81)\) reduces to equalization of the boundary errorsεk​\(θk∗\)=cm/λ\\varepsilon\_\{k\}\(\\theta\_\{k\}^\{\*\}\)=c\_\{m\}/\\lambdaas in the original framework\.

So, under the monotone\-risk assumption, the per\-class threshold policy is optimal*among all measurable policies that gate using\(𝐚^s,β𝐚^s\)\(\\hat\{\\boldsymbol\{a\}\}^\{s\},\\beta\_\{\\hat\{\\boldsymbol\{a\}\}^\{s\}\}\)*\.

### C\.8Computation strategy via Lagrange

The Bayes acceptance condition implies that for each classkk, the thresholdθk​\(λ\)\\theta\_\{k\}\(\\lambda\)\(in the non\-oracle case\) is determined by

εs,k​\(θk​\(λ\)\)−ε¯m,k=cmλ,\\varepsilon\_\{s,k\}\(\\theta\_\{k\}\(\\lambda\)\)\-\\bar\{\\varepsilon\}\_\{m,k\}=\\frac\{c\_\{m\}\}\{\\lambda\}\\,,\(82\)whenever0<θk​\(λ\)<10<\\theta\_\{k\}\(\\lambda\)<1\(withθk​\(λ\)=0\\theta\_\{k\}\(\\lambda\)=0or11if the equation has no solution in\(0,1\)\(0,1\)and the Bayes region collapses to always accept or always defer for that class\)\.

If eachεs,k\\varepsilon\_\{s,k\}is invertible, and we can write

θk​\(λ\)=\(εs,k\)−1​\(ε¯m,k\+cmλ\)\.\\theta\_\{k\}\(\\lambda\)=\(\\varepsilon\_\{s,k\}\)^\{\-1\}\\Big\(\\bar\{\\varepsilon\}\_\{m,k\}\+\\frac\{c\_\{m\}\}\{\\lambda\}\\Big\)\.\(83\)
Thus, for fixedλ\\lambda, the thresholdsθk​\(λ\)\\theta\_\{k\}\(\\lambda\)are obtained by*per\-class inversion*of the error curves at the common “excess\-error level”ε¯m,k\+cm/λ\\bar\{\\varepsilon\}\_\{m,k\}\+c\_\{m\}/\\lambda\.

Pluggingθk​\(λ\)\\theta\_\{k\}\(\\lambda\)into \([75](https://arxiv.org/html/2609.15992#A3.E75)\) yields

Error​\(𝜽​\(λ\)\)=∑k=1Kpk​\[∫θk​\(λ\)1εs,k​\(β\)​fk​\(β\)​𝑑β\+ε¯m,k​∫0θk​\(λ\)fk​\(β\)​𝑑β\]\.\\mathrm\{Error\}\(\\boldsymbol\{\\theta\}\(\\lambda\)\)=\\sum\_\{k=1\}^\{K\}p\_\{k\}\\bigg\[\\int\_\{\\theta\_\{k\}\(\\lambda\)\}^\{1\}\\varepsilon\_\{s,k\}\(\\beta\)f\_\{k\}\(\\beta\)\\,d\\beta\+\\bar\{\\varepsilon\}\_\{m,k\}\\int\_\{0\}^\{\\theta\_\{k\}\(\\lambda\)\}f\_\{k\}\(\\beta\)\\,d\\beta\\bigg\]\.\(84\)
Given an error budgetb∈\[0,1\]b\\in\[0,1\], the target is to findλ∗\\lambda^\{\*\}such that

Error​\(𝜽​\(λ∗\)\)=b\.\\mathrm\{Error\}\(\\boldsymbol\{\\theta\}\(\\lambda^\{\*\}\)\)=b\.
This is a scalar equation inλ\\lambda\. Under mild regularity conditions,Error​\(𝜽​\(λ\)\)\\mathrm\{Error\}\(\\boldsymbol\{\\theta\}\(\\lambda\)\)is monotone inλ\\lambda\(asλ\\lambdaincreases, the Bayes rule becomes more conservative and defers more often, decreasing the use of sLLM in high\-error regions\), so this equation can be solved by standard one\-dimensional root\-finding \(bisection, Brent, etc\.\)\.

Onceλ∗\\lambda^\{\*\}is found, the optimal thresholds are

θk∗=θk​\(λ∗\)=\(εs,k\)−1​\(ε¯m,k\+cmλ∗\)\.\\theta\_\{k\}^\{\*\}=\\theta\_\{k\}\(\\lambda^\{\*\}\)=\(\\varepsilon\_\{s,k\}\)^\{\-1\}\\Big\(\\bar\{\\varepsilon\}\_\{m,k\}\+\\frac\{c\_\{m\}\}\{\\lambda^\{\*\}\}\\Big\)\.
This is the direct analogue of the oracle case, where the target level iscm/λc\_\{m\}/\\lambdainstead ofε¯m,k\+cm/λ\\bar\{\\varepsilon\}\_\{m,k\}\+c\_\{m\}/\\lambda\.

## Appendix DOptimality of single\-threshold policy over generalized policies

We now generalize the simpler single\-threshold framework by allowing the existence arbitrary decision policies\. We again work in the under the same setting and retain the same cost and error formulations as in the main text\. We show that, under a mild monotone\-risk assumption on the sLLM’s confidence score, the Bayes\-optimal deferral policy is a single\-threshold one\. The presentation will be more brief as it uses similar notions as Appendix[C](https://arxiv.org/html/2609.15992#A3)

##### Small model\.

For a given query, the small model now outputs\(𝒂^s,β\(\\hat\{\\boldsymbol\{a\}\}^\{s\},\\beta, where𝒂^s∈𝒜\\hat\{\\boldsymbol\{a\}\}^\{s\}\\in\\mathcal\{A\}is the generated text andβ∈\[0,1\]\\beta\\in\[0,1\]is the scalar confidence for the ‘quality’ of the generation\. Letf​\(β\)f\(\\beta\)be the density ofβ\\betaand the*error curve*of sLLM:εs​\(β\)=Pr⁡\(𝒂≠𝒂^s∣β\)\\varepsilon\_\{s\}\(\\beta\)=\\Pr\(\{\\boldsymbol\{a\}\}\\neq\\hat\{\\boldsymbol\{a\}\}^\{s\}\\mid\\beta\)\.

##### Master model\.

Let𝒂^m\\hat\{\\boldsymbol\{a\}\}^\{m\}denote the generation of mLLM when it is queried\. The error of mLLM is independent of sLLM:

εm​\(γ\)=Pr⁡\(𝒂≠𝒂^m∣γ\),\{\\varepsilon\}\_\{m\}\(\\gamma\)=\\Pr\(\{\\boldsymbol\{a\}\}\\neq\\hat\{\\boldsymbol\{a\}\}^\{m\}\\mid\\gamma\)\\,,\(85\)
and𝔼​\[εm​\(γ\)\]\\mathbb\{E\}\[\\varepsilon\_\{m\}\(\\gamma\)\]is treated as a constantε¯m\\bar\{\\varepsilon\}\_\{m\}\. Note that we do*not*assumeε¯m≤εs​\(β\)\\bar\{\\varepsilon\}\_\{m\}\\leq\\varepsilon\_\{s\}\(\\beta\)for allβ\\beta; mLLM may be better or worse than sLLM depending on the region of the score space\.

Again, rather than restricting ourselves a priori to threshold policies, we now allow any measurable policyπ:\[0,1\]→\{0,1\}\\pi:\[0,1\]\\to\\\{0,1\\\}, where

π​\(β\)=\{0accept sLLM at scoreβ,1defer to mLLM at scoreβ\.\\pi\(\\beta\)=\\begin\{cases\}0&\\text\{accept sLLM at score $\\beta$\},\\\\ 1&\\text\{defer to mLLM at score $\\beta$\}\.\\end\{cases\}\(86\)

##### Cost and error formulations for a general policy\.

Underπ\\pi, the expected cost is

Cost​\(π\)\\displaystyle\\mathrm\{Cost\}\(\\pi\)=𝔼​\[cs\+cm​π​\(β\)\]\\displaystyle=\\mathbb\{E\}\\big\[c\_\{s\}\+c\_\{m\}\\pi\(\\beta\)\\big\]=cs\+cm​∫01π​\(β\)​f​\(β\)​𝑑β\.\\displaystyle=c\_\{s\}\+c\_\{m\}\\int\_\{0\}^\{1\}\\pi\(\\beta\)f\(\\beta\)\\,d\\beta\.\(87\)
The expected error is

Error​\(π\)\\displaystyle\\mathrm\{Error\}\(\\pi\)=𝔼​\[εs​\(β\)​1​\[π​\(β\)=0\]\+ε¯m​1​\[π​\(β\)=1\]\]\\displaystyle=\\mathbb\{E\}\\Big\[\\varepsilon\_\{s\}\(\\beta\)\\,\\mathbf\{1\}\[\\pi\(\\beta\)=0\]\+\\bar\{\\varepsilon\}\_\{m\}\\,\\mathbf\{1\}\[\\pi\(\\beta\)=1\]\\Big\]=∫01\(εs​\(β\)​𝟏​\[π​\(β\)=0\]\+ε¯m​𝟏​\[π​\(β\)=1\]\)​f​\(β\)​𝑑β\.\\displaystyle=\\int\_\{0\}^\{1\}\\Big\(\\varepsilon\_\{s\}\(\\beta\)\\mathbf\{1\}\[\\pi\(\\beta\)=0\]\+\\bar\{\\varepsilon\}\_\{m\}\\mathbf\{1\}\[\\pi\(\\beta\)=1\]\\Big\)f\(\\beta\)\\,d\\beta\.\(88\)
The constrained problem can thus be written as:

minπ⁡Cost​\(π\)s\.t\.Err​\(π\)≤b\.\\min\_\{\\pi\}\\mathrm\{Cost\}\(\\pi\)\\quad\\text\{s\.t\.\}\\quad\\mathrm\{Err\}\(\\pi\)\\leq b\\,\.\(89\)

### D\.1Lagrangian formulation

Introducing a Lagrange multiplierλ≥0\\lambda\\geq 0:

ℒ​\(π,λ\)=Cost​\(π\)\+λ​\(Error​\(π\)−b\)\.\\mathcal\{L\}\(\\pi,\\lambda\)=\\mathrm\{Cost\}\(\\pi\)\+\\lambda\(\\mathrm\{Error\}\(\\pi\)\-b\)\.\(90\)
For fixedλ\\lambda, minimizingℒ​\(π,λ\)\\mathcal\{L\}\(\\pi,\\lambda\)overπ\\piis equivalent to minimizing:

Jλ​\(π\)=𝔼​\[cs\+cm​D\+λ​L\]J\_\{\\lambda\}\(\\pi\)=\\mathbb\{E\}\\big\[c\_\{s\}\+c\_\{m\}D\+\\lambda L\\big\]\\,\(91\)whereLLis the final error indicator\. That is,JλJ\_\{\\lambda\}is the expected value of the loss functionℓλ\\ell\_\{\\lambda\}:

ℓλ​\(D,𝒂,𝒂^s,𝒂^m\)=\{cs\+λ​1​\[𝒂^s≠𝒂\]if​D=0​\(accept\),cs\+cm\+λ​1​\[𝒂^m≠𝒂\]if​D=1​\(defer\)\.\\ell\_\{\\lambda\}\(D,\{\\boldsymbol\{a\}\},\\hat\{\\boldsymbol\{a\}\}^\{s\},\\hat\{\\boldsymbol\{a\}\}^\{m\}\)=\\begin\{cases\}c\_\{s\}\+\\lambda\\,\\mathds\{1\}\[\\hat\{\\boldsymbol\{a\}\}^\{s\}\\neq\{\\boldsymbol\{a\}\}\]&\\text\{if \}D=0\\ \(\\text\{accept\}\),\\\\ c\_\{s\}\+c\_\{m\}\+\\lambda\\,\\mathds\{1\}\[\\hat\{\\boldsymbol\{a\}\}^\{m\}\\neq\{\\boldsymbol\{a\}\}\]&\\text\{if \}D=1\\ \(\\text\{defer\}\)\\,\.\\end\{cases\}\(92\)

### D\.2Bayes\-optimal deferral rule in the binary case

For a fixed scoreβ\\beta, the conditional risks of the two actions are:

rs​\(β\)\\displaystyle r\_\{s\}\(\\beta\)=cs\+λ​εs​\(β\),\(accept sLLM\)\\displaystyle=c\_\{s\}\+\\lambda\\varepsilon\_\{s\}\(\\beta\),\\quad\\text\{\(accept sLLM\)\}rm\\displaystyle r\_\{m\}=cs\+cm\+λ​ε¯m,\(defer to mLLM\)\.\\displaystyle=c\_\{s\}\+c\_\{m\}\+\\lambda\\bar\{\\varepsilon\}\_\{m\},\\quad\\text\{\(defer to mLLM\)\}\.
###### Proposition D\.1\(Bayes\-optimal deferral rule\)\.

For fixedλ\>0\\lambda\>0, define

πλ∗​\(β\)=\{0if​rs​\(β\)≤rm,1if​rs​\(β\)\>rm\.\\pi^\{\*\}\_\{\\lambda\}\(\\beta\)=\\begin\{cases\}0&\\text\{if \}r\_\{s\}\(\\beta\)\\leq r\_\{m\},\\\\ 1&\\text\{if \}r\_\{s\}\(\\beta\)\>r\_\{m\}\.\\end\{cases\}Thenπλ∗\\pi^\{\*\}\_\{\\lambda\}is a global minimizer ofJλ​\(π\)J\_\{\\lambda\}\(\\pi\)over all measurable policiesπ\\pi\.

###### Proof\.

Letπ′\\pi^\{\\prime\}be any policy andπλ∗\\pi^\{\*\}\_\{\\lambda\}the policy defined above\. For eachβ\\beta,D′​\(β\)=π′​\(β\)D^\{\\prime\}\(\\beta\)=\\pi^\{\\prime\}\(\\beta\)andD∗​\(β\)=πλ∗​\(β\)D^\{\*\}\(\\beta\)=\\pi^\{\*\}\_\{\\lambda\}\(\\beta\)\. The associated conditional contribution of a deferralDDto the Lagrangian loss is

R​\(β;D\)=𝔼​\[cs\+cm​D\+λ​L∣β,D\]=\{rs​\(β\)if​D=0,rmif​D=1R\(\\beta;D\)=\\mathbb\{E\}\[c\_\{s\}\+c\_\{m\}D\+\\lambda L\\mid\\beta,D\]=\\begin\{cases\}r\_\{s\}\(\\beta\)&\\text\{if \}D=0,\\\\ r\_\{m\}&\\text\{if \}D=1\\end\{cases\}\(93\)Thus, the excess conditional loss is defined as:

Δ​\(β\)=R​\(β;D′​\(β\)\)−R​\(β;D∗​\(β\)\)\.\\Delta\(\\beta\)=R\(\\beta;D^\{\\prime\}\(\\beta\)\)\-R\(\\beta;D^\{\*\}\(\\beta\)\)\.\(94\)
Since𝔼​\[ℓλ\]\\mathbb\{E\}\[\\ell\_\{\\lambda\}\]equalsrs​\(β\)r\_\{s\}\(\\beta\)whenD=0D=0andrmr\_\{m\}whenD=1D=1, we have

Δ​\(β\)=\{0if​D′​\(β\)=D∗​\(β\),rs​\(β\)−rm≥0if​D′​\(β\)=0,D∗​\(β\)=1,rm−rs​\(β\)≥0if​D′​\(β\)=1,D∗​\(β\)=0\\Delta\(\\beta\)=\\begin\{cases\}0&\\text\{if \}D^\{\\prime\}\(\\beta\)=D^\{\*\}\(\\beta\),\\\\ r\_\{s\}\(\\beta\)\-r\_\{m\}\\geq 0&\\text\{if \}D^\{\\prime\}\(\\beta\)=0,\\ D^\{\*\}\(\\beta\)=1,\\\\ r\_\{m\}\-r\_\{s\}\(\\beta\)\\geq 0&\\text\{if \}D^\{\\prime\}\(\\beta\)=1,\\ D^\{\*\}\(\\beta\)=0\\end\{cases\}\(95\)with strict inequality whenD​\(β\)≠D∗​\(β\)D\(\\beta\)\\neq D^\{\*\}\(\\beta\)andrs​\(β\)≠rmr\_\{s\}\(\\beta\)\\neq r\_\{m\}\. ThusΔ​\(β\)≥0\\Delta\(\\beta\)\\geq 0for allβ\\beta\.

Integrating w\.r\.t\. the distribution ofβ\\betayields

Jλ​\(π′\)−Jλ​\(πλ∗\)=𝔼​\[Δ​\(β\)\]≥0,J\_\{\\lambda\}\(\\pi^\{\\prime\}\)\-J\_\{\\lambda\}\(\\pi^\{\*\}\_\{\\lambda\}\)=\\mathbb\{E\}\[\\Delta\(\\beta\)\]\\geq 0\\,,\(96\)Henceπλ∗\\pi^\{\*\}\_\{\\lambda\}minimizesJλJ\_\{\\lambda\}over all policies\. ∎

The decision rule ofπλ∗\\pi^\{\*\}\_\{\\lambda\}can be written explicitly:

rs​\(β\)≤rm\\displaystyle r\_\{s\}\(\\beta\)\\leq r\_\{m\}⇔cs\+λ​εs​\(β\)≤cs\+cm\+λ​ε¯m\\displaystyle\\iff c\_\{s\}\+\\lambda\\varepsilon\_\{s\}\(\\beta\)\\leq c\_\{s\}\+c\_\{m\}\+\\lambda\\bar\{\\varepsilon\}\_\{m\}⇔εs​\(β\)−ε¯m≤cmλ\.\\displaystyle\\iff\\varepsilon\_\{s\}\(\\beta\)\-\\bar\{\\varepsilon\}\_\{m\}\\leq\\frac\{c\_\{m\}\}\{\\lambda\}\\,\.
Then, for eachβ\\beta,

πλ∗​\(β\)=\{0if​εs​\(β\)−ε¯m≤cm/λ,1if​εs​\(β\)−ε¯m\>cm/λ\\pi^\{\*\}\_\{\\lambda\}\(\\beta\)=\\begin\{cases\}0&\\text\{if \}\\varepsilon\_\{s\}\(\\beta\)\-\\bar\{\\varepsilon\}\_\{m\}\\leq c\_\{m\}/\\lambda,\\\\ 1&\\text\{if \}\\varepsilon\_\{s\}\(\\beta\)\-\\bar\{\\varepsilon\}\_\{m\}\>c\_\{m\}/\\lambda\\end\{cases\}\(97\)

### D\.3Monotone\-risk assumption and threshold structure in confidence score

So far,πλ∗\\pi^\{\*\}\_\{\\lambda\}is defined as a pointwise decision in the error space \([97](https://arxiv.org/html/2609.15992#A4.E97)\)\. We now identify the condition under which this Bayes rule coincides with a*single\-threshold policy*inβ\\beta\.

##### Per\-score monotone risk\.

We assume that the small\-model error curve is non\-increasing in the score:

β1≤β2⇒εs​\(β1\)≥εs​\(β2\)for almost all​β1,β2∈\[0,1\]\.\\beta\_\{1\}\\leq\\beta\_\{2\}\\;\\Rightarrow\\;\\varepsilon\_\{s\}\(\\beta\_\{1\}\)\\geq\\varepsilon\_\{s\}\(\\beta\_\{2\}\)\\quad\\text\{for almost all \}\\beta\_\{1\},\\beta\_\{2\}\\in\[0,1\]\\,\.\(98\)Equivalently, the correctness probability1−εs​\(β\)1\-\\varepsilon\_\{s\}\(\\beta\)is non\-decreasing inβ\\beta\. This monotone\-risk property captures the idea that higher confidence scores should not be more error\-prone than lower ones\.

Under \([98](https://arxiv.org/html/2609.15992#A4.E98)\), the excess errorεs​\(β\)−ε¯m\\varepsilon\_\{s\}\(\\beta\)\-\\bar\{\\varepsilon\}\_\{m\}is also non\-increasing inβ\\beta\.

##### Shape of the acceptance set\.

For eacht∈ℝt\\in\\mathbb\{R\}, define the sublevel set

S​\(t\)=\{β∈\[0,1\]:εs​\(β\)−ε¯m≤t\}\.S\(t\)=\\\{\\beta\\in\[0,1\]:\\varepsilon\_\{s\}\(\\beta\)\-\\bar\{\\varepsilon\}\_\{m\}\\leq t\\\}\\,\.\(99\)
Since the excess error function is non\-increasing \(andβ\\betais a continuous variable\), eachS​\(t\)S\(t\)is either empty, all of\[0,1\]\[0,1\], or an interval of the form\[θ​\(t\),1\]\[\\theta\(t\),1\]More precisely, letting

θ​\(t\):=inf\{β∈\[0,1\]:εs​\(β\)−ε¯m≤t\},\\theta\(t\):=\\inf\\\{\\beta\\in\[0,1\]:\\varepsilon\_\{s\}\(\\beta\)\-\\bar\{\\varepsilon\}\_\{m\}\\leq t\\\}\\,,\(100\)we haveS​\(t\)=\[θ​\(t\),1\]S\(t\)=\[\\theta\(t\),1\]wheneverS​\(t\)S\(t\)is non\-empty and not all of\[0,1\]\[0,1\]\.

For a givenλ\\lambda, the Bayes rule accepts on the set

S​\(cm/λ\)=\{β:εs​\(β\)−ε¯m≤cm/λ\}\.S\\big\(c\_\{m\}/\\lambda\\big\)=\\\{\\beta:\\varepsilon\_\{s\}\(\\beta\)\-\\bar\{\\varepsilon\}\_\{m\}\\leq c\_\{m\}/\\lambda\\\}\\,\.\(101\)Thus, under monotone\-risk, there exists a thresholdθ​\(λ\)\\theta\(\\lambda\)such that

πλ∗​\(β\)=\{0if​β≥θ​\(λ\),1if​β<θ​\(λ\)\\pi^\{\*\}\_\{\\lambda\}\(\\beta\)=\\begin\{cases\}0&\\text\{if \}\\beta\\geq\\theta\(\\lambda\),\\\\ 1&\\text\{if \}\\beta<\\theta\(\\lambda\)\\end\{cases\}\(102\)Thus, under the monotone\-risk assumption on the small\-model score, the Bayes\-optimal policy for eachλ\\lambdais a single\-threshold policy inβ\\beta\.

## Appendix EContinuity of the constraint function

We present a proof about the continuity of the constraint function of our optimization problems\.

###### Lemma E\.1\(Absolute continuity of the single\-threshold oracle and non\-oracle error functions\)\.

Letf:\[0,1\]→\[0,∞\)f:\[0,1\]\\to\[0,\\infty\)be a probability density, i\.e\.ffis measurable and∫01f​\(β\)​𝑑β=1\\int\_\{0\}^\{1\}f\(\\beta\)\\,d\\beta=1\. Letεs:\[0,1\]→\[0,1\]\\varepsilon\_\{s\}:\[0,1\]\\to\[0,1\]be measurable and letε¯m∈\[0,1\]\\bar\{\\varepsilon\}\_\{m\}\\in\[0,1\]be a constant\. Define the oracle and non\-oracle error maps

g​\(θ\)\\displaystyle g\(\\theta\)=∫θ1εs​\(β\)​f​\(β\)​𝑑β,\\displaystyle=\\int\_\{\\theta\}^\{1\}\\varepsilon\_\{s\}\(\\beta\)\\,f\(\\beta\)\\,d\\beta\\,,\(103\)gNO​\(θ\)\\displaystyle g\_\{\\mathrm\{NO\}\}\(\\theta\)=∫θ1εs​\(β\)​f​\(β\)​𝑑β\+ε¯m​∫0θf​\(β\)​𝑑β\.\\displaystyle=\\int\_\{\\theta\}^\{1\}\\varepsilon\_\{s\}\(\\beta\)\\,f\(\\beta\)\\,d\\beta\\;\+\\;\\bar\{\\varepsilon\}\_\{m\}\\int\_\{0\}^\{\\theta\}f\(\\beta\)\\,d\\beta\\,\.\(104\)ThenggandgNOg\_\{\\mathrm\{NO\}\}are absolutely continuous on\[0,1\]\[0,1\]\(and hence continuous\)\. Moreover, both are differentiable almost everywhere and satisfy

g′​\(θ\)\\displaystyle g^\{\\prime\}\(\\theta\)=−εs​\(θ\)​f​\(θ\)for a\.e\.​θ∈\[0,1\],\\displaystyle=\-\\,\\varepsilon\_\{s\}\(\\theta\)\\,f\(\\theta\)\\quad\\text\{for a\.e\. \}\\theta\\in\[0,1\]\\,,\(105\)gNO′​\(θ\)\\displaystyle g\_\{\\mathrm\{NO\}\}^\{\\prime\}\(\\theta\)=−\(εs​\(θ\)−ε¯m\)​f​\(θ\)for a\.e\.​θ∈\[0,1\]\.\\displaystyle=\-\\big\(\\varepsilon\_\{s\}\(\\theta\)\-\\bar\{\\varepsilon\}\_\{m\}\\big\)\\,f\(\\theta\)\\quad\\text\{for a\.e\. \}\\theta\\in\[0,1\]\\,\.\(106\)

###### Proof\.

Leth​\(β\)=εs​\(β\)​f​\(β\)h\(\\beta\)=\\varepsilon\_\{s\}\(\\beta\)f\(\\beta\)\. Since0≤εs≤10\\leq\\varepsilon\_\{s\}\\leq 1andf∈L1​\(\[0,1\]\)f\\in L^\{1\}\(\[0,1\]\)with∫01f=1\\int\_\{0\}^\{1\}f=1, we have0≤h​\(β\)≤f​\(β\)0\\leq h\(\\beta\)\\leq f\(\\beta\)and thereforeh∈L1​\(\[0,1\]\)h\\in L^\{1\}\(\[0,1\]\)\.

Tail integrals ofL1L^\{1\}functions are absolutely continuous\.LetH​\(θ\)=∫θ1h​\(β\)​𝑑βH\(\\theta\)=\\int\_\{\\theta\}^\{1\}h\(\\beta\)\\,d\\beta\. For any finite collection of pairwise disjoint intervals\{\(ai,bi\)\}i=1n⊂\[0,1\]\\\{\(a\_\{i\},b\_\{i\}\)\\\}\_\{i=1\}^\{n\}\\subset\[0,1\]:

H​\(bi\)−H​\(ai\)=∫bi1h​\(β\)​𝑑β−∫ai1h​\(β\)​𝑑β=−∫aibih​\(β\)​𝑑β,H\(b\_\{i\}\)\-H\(a\_\{i\}\)=\\int\_\{b\_\{i\}\}^\{1\}h\(\\beta\)\\,d\\beta\-\\int\_\{a\_\{i\}\}^\{1\}h\(\\beta\)\\,d\\beta=\-\\int\_\{a\_\{i\}\}^\{b\_\{i\}\}h\(\\beta\)\\,d\\beta\\,,\(107\)hence

\|H​\(bi\)−H​\(ai\)\|≤∫aibi\|h​\(β\)\|​𝑑β\.\|H\(b\_\{i\}\)\-H\(a\_\{i\}\)\|\\leq\\int\_\{a\_\{i\}\}^\{b\_\{i\}\}\|h\(\\beta\)\|\\,d\\beta\\,\.\(108\)Summing and using disjointness yields

∑i=1n\|H​\(bi\)−H​\(ai\)\|≤∫∪i\(ai,bi\)\|h​\(β\)\|​𝑑β\.\\sum\_\{i=1\}^\{n\}\|H\(b\_\{i\}\)\-H\(a\_\{i\}\)\|\\;\\leq\\;\\int\_\{\\cup\_\{i\}\(a\_\{i\},b\_\{i\}\)\}\|h\(\\beta\)\|\\,d\\beta\.\(109\)Becauseh∈L1​\(\[0,1\]\)h\\in L^\{1\}\(\[0,1\]\), the integral∫E\|h\|\\int\_\{E\}\|h\|can be made arbitrarily small whenever the Lebesgue measure\|E\|\|E\|is sufficiently small \(absolute continuity of the Lebesgue integral\)\. Therefore, for everyη\>0\\eta\>0there existsδ\>0\\delta\>0such that∑i\(bi−ai\)<δ\\sum\_\{i\}\(b\_\{i\}\-a\_\{i\}\)<\\deltaimplies∑i\|H​\(bi\)−H​\(ai\)\|<η\\sum\_\{i\}\|H\(b\_\{i\}\)\-H\(a\_\{i\}\)\|<\\eta\. This is precisely the definition of absolute continuity ofHH\. ThusHHis absolutely continuous on\[0,1\]\[0,1\]\. Applying this withH=gH=gandh=εs​fh=\\varepsilon\_\{s\}fproves thatggis absolutely continuous\.

Absolute continuity ofgNOg\_\{\\mathrm\{NO\}\}\.LetF​\(θ\)=∫0θf​\(β\)​𝑑βF\(\\theta\)=\\int\_\{0\}^\{\\theta\}f\(\\beta\)\\,d\\beta\. Sincef∈L1​\(\[0,1\]\)f\\in L^\{1\}\(\[0,1\]\), the same argument as above \(withh=fh=f\) shows thatFFis absolutely continuous\. HencegNO​\(θ\)=g​\(θ\)\+ε¯m​F​\(θ\)g\_\{\\mathrm\{NO\}\}\(\\theta\)=g\(\\theta\)\+\\bar\{\\varepsilon\}\_\{m\}F\(\\theta\)is a sum of absolutely continuous functions and is therefore absolutely continuous on\[0,1\]\[0,1\]\. ∎

Absolute continuity of the multi\-thresholds oracle and non\-oracle error functions\.LetK∈ℕK\\in\\mathbb\{N\}\. For each classk∈\{1,…,K\}k\\in\\\{1,\\dots,K\\\}, letfk:\[0,1\]→\[0,∞\)f\_\{k\}:\[0,1\]\\to\[0,\\infty\)be measurable with∫01fk​\(β\)​𝑑β=1\\int\_\{0\}^\{1\}f\_\{k\}\(\\beta\)\\,d\\beta=1, letεs,k:\[0,1\]→\[0,1\]\\varepsilon\_\{s,k\}:\[0,1\]\\to\[0,1\]be measurable, and letε¯m,k∈\[0,1\]\\bar\{\\varepsilon\}\_\{m,k\}\\in\[0,1\]be a constant\. Letpk≥0p\_\{k\}\\geq 0with∑k=1Kpk=1\\sum\_\{k=1\}^\{K\}p\_\{k\}=1\. The oracle and non\-oracle class\-wise error contributions are:

gk​\(θk\)\\displaystyle g\_\{k\}\(\\theta\_\{k\}\):=pk​∫θk1εs,k​\(β\)​fk​\(β\)​𝑑β,\\displaystyle:=p\_\{k\}\\int\_\{\\theta\_\{k\}\}^\{1\}\\varepsilon\_\{s,k\}\(\\beta\)\\,f\_\{k\}\(\\beta\)\\,d\\beta\\,,\(110\)gNO,k​\(θk\)\\displaystyle g\_\{\\mathrm\{NO\},k\}\(\\theta\_\{k\}\):=pk​\(∫θk1εs,k​\(β\)​fk​\(β\)​𝑑β\+ε¯m,k​∫0θkfk​\(β\)​𝑑β\),\\displaystyle:=p\_\{k\}\\left\(\\int\_\{\\theta\_\{k\}\}^\{1\}\\varepsilon\_\{s,k\}\(\\beta\)\\,f\_\{k\}\(\\beta\)\\,d\\beta\+\\bar\{\\varepsilon\}\_\{m,k\}\\int\_\{0\}^\{\\theta\_\{k\}\}f\_\{k\}\(\\beta\)\\,d\\beta\\right\)\\,,\(111\)and the total multiclass error maps on\[0,1\]K\[0,1\]^\{K\}:

g​\(𝜽\)=∑k=1Kgk​\(θk\),gNO​\(𝜽\)=∑k=1KgNO,k​\(θk\),𝜽=\(θ1,…,θK\)\.g\(\\boldsymbol\{\\theta\}\)=\\sum\_\{k=1\}^\{K\}g\_\{k\}\(\\theta\_\{k\}\),\\qquad g\_\{\\mathrm\{NO\}\}\(\\boldsymbol\{\\theta\}\)=\\sum\_\{k=1\}^\{K\}g\_\{\\mathrm\{NO\},k\}\(\\theta\_\{k\}\),\\quad\\boldsymbol\{\\theta\}=\(\\theta\_\{1\},\\dots,\\theta\_\{K\}\)\\,\.\(112\)Then,ggandgNOg\_\{\\mathrm\{NO\}\}are absolutely continuous in each coordinate separately; in particular, they are continuous on\[0,1\]K\[0,1\]^\{K\}\. For a classkk, Since0≤εs,k≤10\\leq\\varepsilon\_\{s,k\}\\leq 1andfk∈L1​\(\[0,1\]\)f\_\{k\}\\in L^\{1\}\(\[0,1\]\)with∫01fk=1\\int\_\{0\}^\{1\}f\_\{k\}=1, the producthk​\(β\)=εs,k​\(β\)​fk​\(β\)h\_\{k\}\(\\beta\)=\\varepsilon\_\{s,k\}\(\\beta\)f\_\{k\}\(\\beta\)belongs toL1​\(\[0,1\]\)L^\{1\}\(\[0,1\]\)\. By Lemma[E\.1](https://arxiv.org/html/2609.15992#A5.Thmtheorem1)applied tohkh\_\{k\}, the mapθk↦∫θk1hk​\(β\)​𝑑β\\theta\_\{k\}\\mapsto\\int\_\{\\theta\_\{k\}\}^\{1\}h\_\{k\}\(\\beta\)\\,d\\betais absolutely continuous \(hence continuous\) on\[0,1\]\[0,1\], and its derivative equals−hk​\(θk\)\-h\_\{k\}\(\\theta\_\{k\}\)for almost everyθk\\theta\_\{k\}\. Multiplying by the constantpkp\_\{k\}gives the absolute continuity ofgkg\_\{k\}\. Similarly,∫0θkfk​\(β\)​𝑑β\\int\_\{0\}^\{\\theta\_\{k\}\}f\_\{k\}\(\\beta\)\\,d\\betais absolutely continuous with derivativefk​\(θk\)f\_\{k\}\(\\theta\_\{k\}\)for almost everyθk\\theta\_\{k\}\. ThereforegNO,kg\_\{\\mathrm\{NO\},k\}is a sum of absolutely continuous functions, hence absolutely continuous\. Finally,g​\(𝜽\)g\(\\boldsymbol\{\\theta\}\)andgNO​\(𝜽\)g\_\{\\mathrm\{NO\}\}\(\\boldsymbol\{\\theta\}\)are finite sums of continuous functions of separate coordinates, hence continuous on\[0,1\]K\[0,1\]^\{K\}\. The stated partial derivatives hold coordinate\-wise almost everywhere\.

## Appendix FDetails about the experimental setup

In this section, we present additional information about the datasets, the algorithms that were used as well as supplementary results\.

### F\.1Datasets

We experimented with five NLP datasets covering both discriminative and generative tasks\. For the former, we focused on classification tasks, usingSST\-2\[socher\-etal\-2013\-recursive\]andFakeNews\[fakenews\]for binary classification, andAGNews\[agnews\]andEmotion\[saravia\-etal\-2018\-carer\]for multi\-class classification\. For the latter, we used theSQuADquestion\-answering dataset\[rajpurkar\-etal\-2016\-squad\]\.

1. \(i\)SST\-2: The Stanford Sentiment Treebank is a dataset of movie review sentences with human\-annotated sentiment labels\. The task involves predicting sentence\-level sentiment under a binary \(positive/negative\) classification setting\.
2. \(ii\)FakeNews: Contains a collection of English\-language news articles that have been manually labeled according to their veracity\. It is designed for binary text classification tasks and includes full article text paired with corresponding ground\-truth labels indicating whether the content is misleading \(fake\) or factual \(real\)\. The dataset covers a variety of news topics and is commonly used for research in automated fake news detection\.
3. \(iii\)Emotion: Consists of English\-language Twitter messages, each annotated with one of six basic emotion categories: anger, fear, joy, love, sadness, and surprise\. It is designed for emotion classification tasks, where the goal is to predict the dominant emotional label of a given tweet based on its text\.
4. \(iv\)AGNews: Consists of news article texts paired with one of four topic labels, World, Sports, Business, and Science/Technology, representing major categories of news content\. Designed for multi\-class text classification, the task is to predict the correct topic label given the content of a news article\.
5. \(v\)SQuAD: The Stanford Question Answering Dataset \(SQuAD\) is a large\-scale reading comprehension corpus built from English Wikipedia articles\. It consists of questions written by crowdworkers paired with context passages, where the task is to extract the correct answer as a contiguous span of text from the passage\. Given our use of LLMs, we adopt a free\-form answer format, enabling responses that are not limited to fixed labels or spans, like in traditional QA\.
6. \(vi\)WMT: The Workshop on Machine Translation \(WMT\) dataset is a large\-scale benchmark for machine translation, constructed from parallel corpora collected from diverse sources such as news articles, parliamentary proceedings, and web crawls\. It consists of sentence\-aligned text pairs in multiple language combinations \(e\.g\., English–German, English–French, English–Russian\), where the task is to translate a source\-language sentence into the corresponding target language\.

Table 3:Summary of datasets and corresponding tasks\.
### F\.2FrugalGPT and HybridLLM details

We used DistilBERT as the encoder for the auxiliary routing model\. The router was trained as a binary classifier, and we constructed its training datasets following the descriptions provided in FrugalGPT\[frugal\]and Hybrid\-LLM\[hybridlllm\]papers\. For the HybridLLM setting, we assigned a label of ‘1’ when sLLM produced an answer that was closer to the ground\-truth answer than mLLM\. Closeness was measured using BARTScore\[bartscore\], computed between the ground\-truth answer and each model’s generated answer\. Specifically, a label of ‘1’ was assigned if the BARTScore between the ground\-truth answer and the sLLM output was higher than that between the ground\-truth answer and the mLLM output; otherwise, a label of ‘0’ was assigned\. For the FrugalGPT setting, we assigned a label of ‘1’ if the ground\-truth answer matched the sLLM’s generated answer, and ‘0’ otherwise\. For both methods, the router model was trained for up to 20 epochs with early stopping \(patience = 3 epochs\) and an initial learning rate ofη=2×10−5\\eta=2\\times 10^\{\-5\}\.

### F\.3Additional results

We present additional results with different model pairs in Tables[4](https://arxiv.org/html/2609.15992#A6.T4),[5](https://arxiv.org/html/2609.15992#A6.T5),[6](https://arxiv.org/html/2609.15992#A6.T6),[7](https://arxiv.org/html/2609.15992#A6.T7),[8](https://arxiv.org/html/2609.15992#A6.T8),[9](https://arxiv.org/html/2609.15992#A6.T9),[10](https://arxiv.org/html/2609.15992#A6.T10),[11](https://arxiv.org/html/2609.15992#A6.T11),[12](https://arxiv.org/html/2609.15992#A6.T12),[13](https://arxiv.org/html/2609.15992#A6.T13),[14](https://arxiv.org/html/2609.15992#A6.T14),[15](https://arxiv.org/html/2609.15992#A6.T15),[16](https://arxiv.org/html/2609.15992#A6.T16), and[17](https://arxiv.org/html/2609.15992#A6.T17)\.

Table 4:Performance comparison of our policy with other policies on classification tasks using thegemma3\-1b\(sLLM\) andgemma3\-12b\(mLLM\) models pair\.
Table 5:Performance comparison of our policy with other policies on classification tasks using thegemma3\-4b\(sLLM\) andgemma3\-12b\(mLLM\) models pair\. In the case offakenews,gemma3\-4bwas better thangemma3\-12bso the policy did not defer any sample\.
Table 6:Performance comparison of our policy with other policies on classification tasks using theqwen3\-4b\(sLLM\) andgemma3\-12b\(mLLM\) models pair\.
Table 7:Performance comparison of our policy with other policies on classification tasks using thegemma3\-1b\(sLLM\) andqwen3\-4b\(mLLM\) models pair\.
Table 8:Performance comparison of our policy with other policies on classification tasks using thegemma3\-1b\(sLLM\) andministral3\-3b\(mLLM\) models pair\.
Table 9:Performance comparison of our policy with other policies on generation tasks using thegemma3\-1b\(sLLM\) andgemma3\-12b\(mLLM\) models pair\. In the case ofWMT,gemma3\-1bwas better thangemma3\-12bso the policy did not defer any sample\.
Table 10:Performance comparison of our policy with other policies on generation tasks using thegemma3\-4b\(sLLM\) andgemma3\-12b\(mLLM\) models pair\. In the case ofWMT,gemma3\-4bwas better thangemma3\-12bso the policy did not defer any sample\.
Table 11:Performance comparison of our policy with other policies on generation tasks using theqwen3\-4b\(sLLM\) andgemma3\-12b\(mLLM\) models pair\. In the case ofWMT,qwen3\-4bwas better thangemma3\-12bso the policy did not defer any sample\.
Table 12:Performance comparison of our policy with other policies on generation tasks using thegemma3\-1b\(sLLM\) andqwen3\-4b\(mLLM\) models pair\.
Table 13:Performance comparison of our policy with other policies on generation tasks using thegemma3\-1b\(sLLM\) andministral3\-3b\(mLLM\) models pair\. In the case ofWMT,gemma3\-1bwas better thanministral3\-3bso the policy did not defer any sample\.
Table 14:Performance comparison of our policy with other policies on classification tasks using thegemma3\-4b\(sLLM\) andgemma3\-27b\(mLLM\) models pair in thesst\-2dataset\.
Table 15:Performance comparison of our policy with other policies on classification tasks using thegemma3\-12b\(sLLM\) andgemma3\-27b\(mLLM\) models pair in thesst\-2dataset\. In this case,gemma3\-12bwas better thangemma3\-27b\.
Table 16:Classification results for our policy and baselines in thesst\-2dataset, using using theqwen3\-4b\(sLLM\) andgemma3\-27b\(mLLM\) models pair\.
Table 17:Classification results for our policy and baselines in thesst\-2dataset, using using theministral3\-3b\(sLLM\) andgemma3\-27b\(mLLM\) models pair\.

### F\.4Monte Carlo sampling

The theoretical cost and error curves can be written as expectations of functions of the random tuple\(𝒂,𝒂^s,β,𝒂^m\)\(\\boldsymbol\{a\},\\hat\{\\boldsymbol\{a\}\}^\{s\},\\beta,\\hat\{\\boldsymbol\{a\}\}^\{m\}\)under the data\-generating distribution\. We approximate these expectations by sample averages\. Concretely, for any policy parameterθ\\theta\(scalar or vector\), letg​\(z;θ\)g\(z;\\theta\)denote the per\-sample error indicator \(defined below\)\. Then

𝔼​\[g​\(Z;θ\)\]≈𝔼^N​\[g​\(Z;θ\)\]:=1N​∑i=1Ng​\(zi;θ\),\\mathbb\{E\}\[g\(Z;\\theta\)\]\\;\\approx\\;\\widehat\{\\mathbb\{E\}\}\_\{N\}\[g\(Z;\\theta\)\]\\;:=\\;\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}g\(z\_\{i\};\\theta\),\(113\)whereziz\_\{i\}denotes the observed tuple on exampleiiandNNis the number of available tuples\. This direct MC evaluation approximates the full expectations appearing in the integrals by sample averages of indicator functions\.

#### F\.4\.1Single\-threshold policy

In the single\-threshold policy, we accept sLLM’s prediction ifβi≥θ\\beta\_\{i\}\\geq\\thetaand defer otherwise:

Ai​\(θ\)=𝟙​\{βi≥θ\},Di​\(θ\)=1−Ai​\(θ\)\.A\_\{i\}\(\\theta\)=\\mathds\{1\}\\\{\\beta\_\{i\}\\geq\\theta\\\},\\qquad D\_\{i\}\(\\theta\)=1\-A\_\{i\}\(\\theta\)\.\(114\)Leteis=𝟙​\{𝒂^is≠𝒂i\}e\_\{i\}^\{s\}=\\mathds\{1\}\\\{\\hat\{\\boldsymbol\{a\}\}^\{s\}\_\{i\}\\neq\\boldsymbol\{a\}\_\{i\}\\\}andeim=𝟙​\{𝒂^im≠𝒂i\}e\_\{i\}^\{m\}=\\mathds\{1\}\\\{\\hat\{\\boldsymbol\{a\}\}^\{m\}\_\{i\}\\neq\\boldsymbol\{a\}\_\{i\}\\\}denote per\-sample error indicators for sLLM and mLLM, respectively\. In practice, we use the single\-threshold policy with generative tasks where models provide open\-ended answers\. Thus, exact equality with a reference output can be difficult to acquire; there may be multiple valid responses, partial correctness, and semantic variation\. As a result, the binary indicator functions may require thresholding of the performance metric\. To address this, one can relax these binary error indicators into continuous ones, i\.e\.,eis=1−M​\(𝒂^is,𝒂i\)e\_\{i\}^\{s\}=1\-M\(\\hat\{\\boldsymbol\{a\}\}^\{s\}\_\{i\},\\boldsymbol\{a\}\_\{i\}\)andeim=1−M​\(𝒂^im,𝒂i\),e\_\{i\}^\{m\}=1\-M\(\\hat\{\\boldsymbol\{a\}\}^\{m\}\_\{i\},\\boldsymbol\{a\}\_\{i\}\),, whereM​\(⋅,⋅\)M\(\\cdot,\\cdot\)is the performance metric of choice \(e\.g\., cosine similarity between answers’ embeddings\)\. We leave this formulation for future work\.

##### Oracle error \(single\-threshold\)\.

In the oracle regime, deferred examples incur no error, hence the MC estimate of the error is

Err^ST−O​\(θ\)=1N​∑i=1NAi​\(θ\)​eis\.\\widehat\{\\mathrm\{Err\}\}\_\{\\mathrm\{ST\-O\}\}\(\\theta\)\\;=\\;\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}A\_\{i\}\(\\theta\)\\,e\_\{i\}^\{s\}\.\(115\)

##### Non\-oracle error \(single\-threshold\)\.

In the non\-oracle regime, deferred examples use mLLM that may err:

Err^ST−NO​\(θ\)=1N​∑i=1N\(Ai​\(θ\)​eis\+Di​\(θ\)​eim\)\.\\widehat\{\\mathrm\{Err\}\}\_\{\\mathrm\{ST\-NO\}\}\(\\theta\)\\;=\\;\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\Big\(A\_\{i\}\(\\theta\)\\,e\_\{i\}^\{s\}\\;\+\\;D\_\{i\}\(\\theta\)\\,e\_\{i\}^\{m\}\\Big\)\.\(116\)

##### MC cost \(single\-threshold\)\.

We use a cost model with constant sLLM costcs\>0c\_\{s\}\>0and additional mLLM costcm\>0c\_\{m\}\>0when deferring\. Thus, the expected per\-sample cost and its MC estimate are:

Cost^​\(θ\)=cs\+cm⋅Def^​\(θ\),Def^​\(θ\):=1N​∑i=1NDi​\(θ\),\\widehat\{\\mathrm\{Cost\}\}\(\\theta\)\\;=\\;c\_\{s\}\\;\+\\;c\_\{m\}\\cdot\\widehat\{\\mathrm\{Def\}\}\(\\theta\),\\qquad\\widehat\{\\mathrm\{Def\}\}\(\\theta\):=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}D\_\{i\}\(\\theta\)\\,,\(117\)whereDef^​\(θ\)\\widehat\{\\mathrm\{Def\}\}\(\\theta\)is the empirical deferral rate \(and1−Def^​\(θ\)1\-\\widehat\{\\mathrm\{Def\}\}\(\\theta\)the empirical coverage\)\.

#### F\.4\.2Multi\-thresholds policy

In the multi\-thresholds policy, we employ a class\-conditional threshold vector𝜽=\(θ1,…,θK\)\\boldsymbol\{\\theta\}=\(\\theta\_\{1\},\\dots,\\theta\_\{K\}\), whereθk\\theta\_\{k\}is applied to examples whose sLLM predicted label equalskk\. The accept/deferral rule is:

Ai​\(𝜽\)=𝟙​\{βi≥θ𝒂^is\},Di​\(𝜽\)=1−Ai​\(𝜽\)\.A\_\{i\}\(\\boldsymbol\{\\theta\}\)=\\mathds\{1\}\\\{\\beta\_\{i\}\\geq\\theta\_\{\\hat\{\\boldsymbol\{a\}\}^\{s\}\_\{i\}\}\\\},\\qquad D\_\{i\}\(\\boldsymbol\{\\theta\}\)=1\-A\_\{i\}\(\\boldsymbol\{\\theta\}\)\.\(118\)
##### Oracle error \(multi\-thresholds\)\.

Under the oracle regime,

Err^MT−O​\(𝜽\)=1N​∑i=1NAi​\(𝜽\)​eis\.\\widehat\{\\mathrm\{Err\}\}\_\{\\mathrm\{MT\-O\}\}\(\\boldsymbol\{\\theta\}\)\\;=\\;\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}A\_\{i\}\(\\boldsymbol\{\\theta\}\)\\,e\_\{i\}^\{s\}\.\(119\)

##### Non\-oracle error \(multi\-thresholds\)\.

Under the non\-oracle regime,

Err^MT−NO​\(𝜽\)=1N​∑i=1N\(Ai​\(𝜽\)​eis\+Di​\(𝜽\)​eim\)\.\\widehat\{\\mathrm\{Err\}\}\_\{\\mathrm\{MT\-NO\}\}\(\\boldsymbol\{\\theta\}\)\\;=\\;\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\Big\(A\_\{i\}\(\\boldsymbol\{\\theta\}\)\\,e\_\{i\}^\{s\}\\;\+\\;D\_\{i\}\(\\boldsymbol\{\\theta\}\)\\,e\_\{i\}^\{m\}\\Big\)\.\(120\)

##### MC cost \(multi\-thresholds\)\.

The cost expression is identical in form to the single\-threshold case \([117](https://arxiv.org/html/2609.15992#A6.E117)\)\.

#### F\.4\.3Finite\-sample effects and empirical feasibility

Finite\-sample MC estimates yield stepwise functions of the threshold\(s\), since the accept/defer sets change only when thresholds cross observed confidence values\. Consequently, the optimal empirical solution to

minθ⁡Cost^​\(θ\)s\.t\.Err^​\(θ\)≤b\\min\_\{\\theta\}\\penalty 10000\\ \\widehat\{\\mathrm\{Cost\}\}\(\\theta\)\\quad\\text\{s\.t\.\}\\quad\\widehat\{\\mathrm\{Err\}\}\(\\theta\)\\leq b\(121\)may exhibit*slack*\(Err^​\(θ∗\)<b\\widehat\{\\mathrm\{Err\}\}\(\\theta^\{\*\}\)<b\), because there may be no threshold configuration whose empirical error equalsbbexactly\. This is a discretization effect inherent to optimizing over thresholds on a finite dataset and does not contradict the population\-level optimality condition that places the optimum on the boundary when the constraint is active and the error curve is continuous\. Finally, note that the class prior termspk=Pr⁡\(𝒂^s=k\)p\_\{k\}=\\Pr\(\\hat\{\\boldsymbol\{a\}\}^\{s\}=k\)appearing in the theoretical integral decompositions are automatically accounted for by MC evaluation: summing over all samples weights each predicted class proportionally to its empirical frequencynk/Nn\_\{k\}/N, yielding the standard sample\-average approximation of the population expectation\.

### F\.5Details about implemented algorithms

We briefly describe the algorithms used to solve the empirical constrained optimization problem \([121](https://arxiv.org/html/2609.15992#A6.E121), together with their computational complexity\. In all cases, we minimize the expected per\-sample cost \(equivalently, the deferral rate\) subject to a global error budget\. A key observation is that on a finite dataset the error and deferral rate are*stepwise*functions of the threshold\(s\): the accept/defer partition changes only when a threshold crosses an observed confidence value \(or a tie\-group of equal confidences\)\. Hence, the search can be restricted to a finite set of candidates induced by the sample\.

#### F\.5\.1Single\-threshold policy

Consider the single\-threshold ruleAi​\(θ\)=𝟙​\{βi≥θ\}A\_\{i\}\(\\theta\)=\\mathds\{1\}\\\{\\beta\_\{i\}\\geq\\theta\\\}\. Since the empirical costCost^​\(θ\)\\widehat\{\\mathrm\{Cost\}\}\(\\theta\)is non\-decreasing inθ\\theta\(larger thresholds defer more\), the minimum\-cost feasible solution is obtained by selecting the*smallest*threshold that satisfies the constraintErr^​\(θ\)≤b\\widehat\{\\mathrm\{Err\}\}\(\\theta\)\\leq b, when such a threshold exists\. On finite samples, it suffices to evaluateErr^​\(θ\)\\widehat\{\\mathrm\{Err\}\}\(\\theta\)at the distinct observedβ\\betavalues \(plus the two endpoints corresponding to “accept all” and “defer all”\)\.

##### Sorting\-based scan \(𝒪​\(N​log⁡N\)\\mathcal\{O\}\(N\\log N\)\)\.

We sort examples byβi\\beta\_\{i\}and compute cumulative sums of error indicators to evaluate all candidate thresholds in a single pass\. Grouping ties inβ\\betayields a tie\-aware scan over unique confidence levels\. The runtime is dominated by sorting:𝒪​\(N​log⁡N\)\\mathcal\{O\}\(N\\log N\)time and𝒪​\(N\)\\mathcal\{O\}\(N\)memory \(or𝒪​\(N\)\\mathcal\{O\}\(N\)time after sorting for the cumulative updates\)\. An implementation outline is presented in Algorithm[1](https://arxiv.org/html/2609.15992#alg1)\(for the oracle case\) and Algorithm[2](https://arxiv.org/html/2609.15992#alg2)\(for the non\-oracle case\)\.

Algorithm 1Single\-threshold selection \(oracle, MC, sorting\-based\)1:Input:Dataset with generated answers

\{\(βi,𝒂^im,𝒂^is\)\}i=1N\\\{\(\\beta\_\{i\},\\hat\{\\boldsymbol\{a\}\}^\{m\}\_\{i\},\\hat\{\\boldsymbol\{a\}\}^\{s\}\_\{i\}\)\\\}\_\{i=1\}^\{N\}, accuracy lower bound

ξ∈\[0,1\]\\xi\\in\[0,1\], costs

\(cs,cm\)\(c\_\{s\},c\_\{m\}\)
2:Output:Optimal threshold

θ∗\\theta^\{\*\}minimizing cost w\.r\.t the error constraint

3:Set error budget

b←1−ξb\\leftarrow 1\-\\xi
4:Compute

eis←𝟙​\{𝒂^is≠𝒂^im\}e\_\{i\}^\{s\}\\leftarrow\\mathds\{1\}\\\{\\hat\{\\boldsymbol\{a\}\}^\{s\}\_\{i\}\\neq\\hat\{\\boldsymbol\{a\}\}^\{m\}\_\{i\}\\\}
5:Sort samples by

β\\beta:

β1≤⋯≤βN\\beta\_\{1\}\\leq\\dots\\leq\\beta\_\{N\}and reorder

ese^\{s\}accordingly

6:Let

u1<⋯<uLu\_\{1\}<\\dots<u\_\{L\}be distinct confidence values; define tie\-groups

Gℓ=\{i:βi=uℓ\}G\_\{\\ell\}=\\\{i:\\beta\_\{i\}=u\_\{\\ell\}\\\}
7:For each

ℓ\\ell:

Nℓ←\|Gℓ\|N\_\{\\ell\}\\leftarrow\|G\_\{\\ell\}\|,

Eℓs←∑i∈GℓeisE^\{s\}\_\{\\ell\}\\leftarrow\\sum\_\{i\\in G\_\{\\ell\}\}e^\{s\}\_\{i\}
8:Accept\-all check:

Err^ST−O←1N​∑i=1Neis\\widehat\{\\mathrm\{Err\}\}\_\{\\mathrm\{ST\-O\}\}\\leftarrow\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}e^\{s\}\_\{i\}
9:if

Err^ST−O≤b\\widehat\{\\mathrm\{Err\}\}\_\{\\mathrm\{ST\-O\}\}\\leq bthen

10:

θ∗←u1\\theta^\{\*\}\\leftarrow u\_\{1\};

Def^←0\\widehat\{\\mathrm\{Def\}\}\\leftarrow 0;

Cost^←cs\\widehat\{\\mathrm\{Cost\}\}\\leftarrow c\_\{s\}
11:Return

θ∗\\theta^\{\*\}
12:endif

13:

AcceptedErrors←∑ℓ′=1LEℓ′s\\mathrm\{AcceptedErrors\}\\leftarrow\\sum\_\{\\ell^\{\\prime\}=1\}^\{L\}E^\{s\}\_\{\\ell^\{\\prime\}\}
14:

DeferredCount←0\\mathrm\{DeferredCount\}\\leftarrow 0
15:for

ℓ=1\\ell=1to

LLdo

16:

Err^ST−O​\(uℓ\)←AcceptedErrors/N\\widehat\{\\mathrm\{Err\}\}\_\{\\mathrm\{ST\-O\}\}\(u\_\{\\ell\}\)\\leftarrow\\mathrm\{AcceptedErrors\}/N
17:if

Err^ST−O​\(uℓ\)≤b\\widehat\{\\mathrm\{Err\}\}\_\{\\mathrm\{ST\-O\}\}\(u\_\{\\ell\}\)\\leq bthen

18:

θ∗←uℓ\\theta^\{\*\}\\leftarrow u\_\{\\ell\}
19:

Def^←DeferredCount/N\\widehat\{\\mathrm\{Def\}\}\\leftarrow\\mathrm\{DeferredCount\}/N
20:

Cost^←cs\+cm​Def^\\widehat\{\\mathrm\{Cost\}\}\\leftarrow c\_\{s\}\+c\_\{m\}\\,\\widehat\{\\mathrm\{Def\}\}
21:Return

θ∗\\theta^\{\*\}
22:endif\{Update totals for next threshold: defer level

ℓ\\ell\(oracle: no deferred error\)\}

23:

DeferredCount←DeferredCount\+Nℓ\\mathrm\{DeferredCount\}\\leftarrow\\mathrm\{DeferredCount\}\+N\_\{\\ell\}
24:

AcceptedErrors←AcceptedErrors−Eℓs\\mathrm\{AcceptedErrors\}\\leftarrow\\mathrm\{AcceptedErrors\}\-E^\{s\}\_\{\\ell\}
25:endfor

26:

θ∗←uL\\theta^\{\*\}\\leftarrow u\_\{L\};

Def^←1\\widehat\{\\mathrm\{Def\}\}\\leftarrow 1;

Err^←0\\widehat\{\\mathrm\{Err\}\}\\leftarrow 0;

Cost^←cs\+cm\\widehat\{\\mathrm\{Cost\}\}\\leftarrow c\_\{s\}\+c\_\{m\}
27:Return

θ∗\\theta^\{\*\}

Algorithm 2Single\-threshold selection \(non\-oracle, MC, sorting\-based\)1:Input:Dataset with generated answers

\{\(βi,𝒂^im,𝒂^is,𝒂i\)\}i=1N\\\{\(\\beta\_\{i\},\\hat\{\\boldsymbol\{a\}\}^\{m\}\_\{i\},\\hat\{\\boldsymbol\{a\}\}^\{s\}\_\{i\},\\boldsymbol\{a\}\_\{i\}\)\\\}\_\{i=1\}^\{N\}, accuracy lower bound

ξ∈\[0,1\]\\xi\\in\[0,1\], costs

\(cs,cm\)\(c\_\{s\},c\_\{m\}\)
2:Output:Optimal threshold

θ∗\\theta^\{\*\}minimizing cost w\.r\.t the error constraint

3:Set error budget

b←1−ξb\\leftarrow 1\-\\xi
4:Compute

eis←𝟙​\{𝒂^is≠𝒂i\}e^\{s\}\_\{i\}\\leftarrow\\mathds\{1\}\\\{\\hat\{\\boldsymbol\{a\}\}^\{s\}\_\{i\}\\neq\\boldsymbol\{a\}\_\{i\}\\\},

eim←𝟙​\{𝒂^im≠𝒂i\}e^\{m\}\_\{i\}\\leftarrow\\mathds\{1\}\\\{\\hat\{\\boldsymbol\{a\}\}^\{m\}\_\{i\}\\neq\\boldsymbol\{a\}\_\{i\}\\\}
5:Sort samples by

β\\beta:

β1≤⋯≤βN\\beta\_\{1\}\\leq\\dots\\leq\\beta\_\{N\}and reorder

es,eme^\{s\},e^\{m\}accordingly

6:Let

u1<⋯<uLu\_\{1\}<\\dots<u\_\{L\}be the distinct values in

\{βi\}\\\{\\beta\_\{i\}\\\}; for each

ℓ\\ell, let

Gℓ=\{i:β\(i\)=uℓ\}G\_\{\\ell\}=\\\{i:\\beta\_\{\(i\)\}=u\_\{\\ell\}\\\}
7:For each level

ℓ\\ell:

Nℓ←\|Gℓ\|N\_\{\\ell\}\\leftarrow\|G\_\{\\ell\}\|,

Eℓs←∑i∈GℓeisE^\{s\}\_\{\\ell\}\\leftarrow\\sum\_\{i\\in G\_\{\\ell\}\}e^\{s\}\_\{i\},

Eℓm←∑i∈GℓeimE^\{m\}\_\{\\ell\}\\leftarrow\\sum\_\{i\\in G\_\{\\ell\}\}e^\{m\}\_\{i\}
8:Accept\-all check:

Err^ST−NO←1N​∑i=1Ne\(i\)\(s\)\\widehat\{\\mathrm\{Err\}\}\_\{\\mathrm\{ST\-NO\}\}\\leftarrow\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}e^\{\(s\)\}\_\{\(i\)\}
9:if

Err^ST−NO≤b\\widehat\{\\mathrm\{Err\}\}\_\{\\mathrm\{ST\-NO\}\}\\leq bthen

10:

θ∗←u1\\theta^\{\*\}\\leftarrow u\_\{1\};

Def^←0\\widehat\{\\mathrm\{Def\}\}\\leftarrow 0;

Cost^←cs\\widehat\{\\mathrm\{Cost\}\}\\leftarrow c\_\{s\}
11:Return

θ∗\\theta^\{\*\}
12:endif

13:

DeferredErrors←0\\mathrm\{DeferredErrors\}\\leftarrow 0\{

∑β<uℓem\\sum\_\{\\beta<u\_\{\\ell\}\}e^\{m\}\}

14:

DeferredCount←0\\mathrm\{DeferredCount\}\\leftarrow 0\{

\#​\{i:β<uℓ\}\\\#\\\{i:\\beta<u\_\{\\ell\}\\\}\}

15:

AcceptedErrors←∑ℓ′=1LEℓ′s\\mathrm\{AcceptedErrors\}\\leftarrow\\sum\_\{\\ell^\{\\prime\}=1\}^\{L\}E^\{s\}\_\{\\ell^\{\\prime\}\}\{

∑β≥u1es\\sum\_\{\\beta\\geq u\_\{1\}\}e^\{s\}initially\}

16:for

ℓ=1\\ell=1to

LLdo

17:

Err^ST−NO​\(uℓ\)←DeferredErrors\+AcceptedErrorsN\\widehat\{\\mathrm\{Err\}\}\_\{\\mathrm\{ST\-NO\}\}\(u\_\{\\ell\}\)\\leftarrow\\frac\{\\mathrm\{DeferredErrors\}\+\\mathrm\{AcceptedErrors\}\}\{N\}\{Candidate threshold

θ=uℓ\\theta=u\_\{\\ell\}: defer levels

<ℓ<\\ell, accept levels

≥ℓ\\geq\\ell\}

18:if

Err^ST−NO​\(uℓ\)≤b\\widehat\{\\mathrm\{Err\}\}\_\{\\mathrm\{ST\-NO\}\}\(u\_\{\\ell\}\)\\leq bthen

19:

θ∗←uℓ\\theta^\{\*\}\\leftarrow u\_\{\\ell\}
20:

Def^←DeferredCount/N\\widehat\{\\mathrm\{Def\}\}\\leftarrow\\mathrm\{DeferredCount\}/N
21:

Cost^←cs\+cm​Def^\\widehat\{\\mathrm\{Cost\}\}\\leftarrow c\_\{s\}\+c\_\{m\}\\,\\widehat\{\\mathrm\{Def\}\}
22:Return

θ∗\\theta^\{\*\}
23:endif

24:

DeferredErrors←DeferredErrors\+Eℓm\\mathrm\{DeferredErrors\}\\leftarrow\\mathrm\{DeferredErrors\}\+E^\{m\}\_\{\\ell\}
25:

DeferredCount←DeferredCount\+Nℓ\\mathrm\{DeferredCount\}\\leftarrow\\mathrm\{DeferredCount\}\+N\_\{\\ell\}
26:

AcceptedErrors←AcceptedErrors−Eℓs\\mathrm\{AcceptedErrors\}\\leftarrow\\mathrm\{AcceptedErrors\}\-E^\{s\}\_\{\\ell\}
27:endfor

28:

θ∗←uL\\theta^\{\*\}\\leftarrow u\_\{L\};

Def^←1\\widehat\{\\mathrm\{Def\}\}\\leftarrow 1;

Err^←1N​∑i=1Neim\\widehat\{\\mathrm\{Err\}\}\\leftarrow\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}e^\{m\}\_\{i\};

Cost^←cs\+cm\\widehat\{\\mathrm\{Cost\}\}\\leftarrow c\_\{s\}\+c\_\{m\}\{No feasible threshold: defer all \(choose any

θ\>uL\\theta\>u\_\{L\}\)\}

29:Return

θ∗\\theta^\{\*\}

##### BFPRT variant \(𝒪​\(N\)\\mathcal\{O\}\(N\)worst\-case; oracle only\)\.

In the oracle case, feasibility depends only on the number of small\-model mistakes among accepted samples, and the optimal solution corresponds to accepting the largest\-confidence subset that satisfies the error budget\. This can be implemented without full sorting by using a linear\-time order\-statistic routine \(median\-of\-medians/BFPRT\[BLUM1973448\]\) to find the relevant confidence quantile \(and then a linear pass to count errors/deferrals\)\. This yields𝒪​\(N\)\\mathcal\{O\}\(N\)worst\-case time\. We stress that this guarantee relies on the oracle structure; in the non\-oracle case the constraint depends on*both*accepted and deferred subsets, and an exact worst\-case𝒪​\(N\)\\mathcal\{O\}\(N\)algorithm is generally not available without additional assumptions\. Implementation outline is presented in Algorithm[3](https://arxiv.org/html/2609.15992#alg3)\.

Algorithm 3Single\-threshold selection \(oracle, MC, BFPRT variant\)1:Input:Dataset with generated answers

\{\(βi,𝒂^im,𝒂^is\)\}i=1N\\\{\(\\beta\_\{i\},\\hat\{\\boldsymbol\{a\}\}^\{m\}\_\{i\},\\hat\{\\boldsymbol\{a\}\}^\{s\}\_\{i\}\)\\\}\_\{i=1\}^\{N\}, accuracy lower bound

ξ∈\[0,1\]\\xi\\in\[0,1\], costs

\(cs,cm\)\(c\_\{s\},c\_\{m\}\)
2:Output:Optimal threshold

θ∗\\theta^\{\*\}minimizing cost w\.r\.t the error constraint

3:Compute

eis←𝟙​\{𝒂^is≠𝒂^im\}e^\{s\}\_\{i\}\\leftarrow\\mathds\{1\}\\\{\\hat\{\\boldsymbol\{a\}\}^\{s\}\_\{i\}\\neq\\hat\{\\boldsymbol\{a\}\}^\{m\}\_\{i\}\\\}
4:if

1N​∑i=1Neis≤b\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}e^\{s\}\_\{i\}\\leq bthen

5:Returnany

θ∗\\theta^\{\*\}that accepts all samples

6:endif

7:Use a median\-of\-mediansSelectroutine to obtain a pivot confidence

θ\\theta\{We search over acceptance sets defined by confidence cutoffs without full sorting\}

8:Partition indices into

A=\{i:βi≥θ\}A=\\\{i:\\beta\_\{i\}\\geq\\theta\\\}and

R=\{i:βi<θ\}R=\\\{i:\\beta\_\{i\}<\\theta\\\}
9:Compute

Err^ST−O​\(θ\)=1N​∑i∈Aeis\\widehat\{\\mathrm\{Err\}\}\_\{\\mathrm\{ST\-O\}\}\(\\theta\)=\\frac\{1\}\{N\}\\sum\_\{i\\in A\}e^\{s\}\_\{i\}
10:if

Err^ST−O​\(θ\)≤b\\widehat\{\\mathrm\{Err\}\}\_\{\\mathrm\{ST\-O\}\}\(\\theta\)\\leq bthen

11:Recurse on

RRto try a smaller feasible threshold \(accept more\)

12:else

13:Recurse on

AAto increase the threshold \(accept fewer\)

14:endif

15:Return the smallest feasible cutoff encountered as

θ∗\\theta^\{\*\}

#### F\.5\.2Multi\-thresholds policy

For each predicted classkk, we sort the samples with𝒂^s=k\\hat\{\\boldsymbol\{a\}\}^\{s\}=kbyβ\\betaand consider tie\-aware unique confidence levels\. This produces a finite set of*options*per classkk, indexed byjj, each corresponding to deferring the lowest\-confidence prefix within that class\. Each option yields a pair:

\(dk,j,ek,j\),\(d\_\{k,j\},\\,e\_\{k,j\}\),wheredk,jd\_\{k,j\}is the number of deferred samples in classkkandek,je\_\{k,j\}is the number of total errors contributed by classkk\(oracle: accepted sLLM mistakes; non\-oracle: accepted sLLM mistakes plus deferred mLLM mistakes\)\. The global optimization becomes: choose one option per class to minimize total deferrals subject to a total error\-count budgetB=⌊b​N⌋B=\\lfloor bN\\rfloor\.

##### Option\-table construction\.

Letnkn\_\{k\}be the number of samples with𝒂^s=k\\hat\{\\boldsymbol\{a\}\}^\{s\}=kand letUk≤nkU\_\{k\}\\leq n\_\{k\}be the number of unique confidence levels within classkk\(tie\-aware\)\. Constructing per\-class option matrices requires sorting within each class:∑k=1K𝒪​\(nk​log⁡nk\)≤𝒪​\(N​log⁡N\)\\sum\_\{k=1\}^\{K\}\\mathcal\{O\}\(n\_\{k\}\\log n\_\{k\}\)\\leq\\mathcal\{O\}\(N\\log N\), followed by linear\-time cumulative sums,𝒪​\(N\)\\mathcal\{O\}\(N\)\. The resulting total number of options isU=∑k\(Uk\+1\)U=\\sum\_\{k\}\(U\_\{k\}\+1\)\(including the “defer all” endpoint\), withU≤N\+KU\\leq N\+K\. Algorithms for oracle and non\-oracle settings are presented in Algorithm[4](https://arxiv.org/html/2609.15992#alg4)and Algorithm[5](https://arxiv.org/html/2609.15992#alg5), respectively\.

Algorithm 4Build per\-class option matrices \(oracle\)1:Input:

\{\(βi,𝒂^is,𝒂^im\)\}i=1N\\\{\(\\beta\_\{i\},\\hat\{\\boldsymbol\{a\}\}^\{s\}\_\{i\},\\hat\{\\boldsymbol\{a\}\}^\{m\}\_\{i\}\)\\\}\_\{i=1\}^\{N\}, number of classes

KK
2:Output:For each class

kk: options

\{\(dk,j,ek,j,θk,j\)\}j=1Jk\\\{\(d\_\{k,j\},e\_\{k,j\},\\theta\_\{k,j\}\)\\\}\_\{j=1\}^\{J\_\{k\}\}
3:Compute

eis=𝟙​\{𝒂^is≠𝒂^im\}e^\{s\}\_\{i\}=\\mathds\{1\}\\\{\\hat\{\\boldsymbol\{a\}\}^\{s\}\_\{i\}\\neq\\hat\{\\boldsymbol\{a\}\}^\{m\}\_\{i\}\\\}
4:for

k=1k=1to

KKdo

5:

Sk←\{i:𝒂^is=k\}S\_\{k\}\\leftarrow\\\{i:\\hat\{\\boldsymbol\{a\}\}^\{s\}\_\{i\}=k\\\};

nk←\|Sk\|n\_\{k\}\\leftarrow\|S\_\{k\}\|
6:if

nk=0n\_\{k\}=0then

7:Store one trivial option

\(d,e\)=\(0,0\)\(d,e\)=\(0,0\)andcontinue

8:endif

9:Sort

SkS\_\{k\}by

βi\\beta\_\{i\}ascending; let

u1<⋯<uLu\_\{1\}<\\dots<u\_\{L\}be distinct values in class

kk
10:For each level

ℓ\\ell, let

Gℓ=\{i∈Sk:βi=uℓ\}G\_\{\\ell\}=\\\{i\\in S\_\{k\}:\\beta\_\{i\}=u\_\{\\ell\}\\\}and compute:

Nℓ=\|Gℓ\|,Eℓs=∑i∈GℓeisN\_\{\\ell\}=\|G\_\{\\ell\}\|,\\quad E^\{s\}\_\{\\ell\}=\\sum\_\{i\\in G\_\{\\ell\}\}e^\{s\}\_\{i\}
11:

DeferredCount←0\\mathrm\{DeferredCount\}\\leftarrow 0\{

\#​\{i∈Sk:β<uj\}\\\#\\\{i\\in S\_\{k\}:\\beta<u\_\{j\}\\\}\}

12:

AcceptedErrors←∑ℓ′=1LEℓ′\(s\)\\mathrm\{AcceptedErrors\}\\leftarrow\\sum\_\{\\ell^\{\\prime\}=1\}^\{L\}E^\{\(s\)\}\_\{\\ell^\{\\prime\}\}\{

∑β≥u1e\(s\)\\sum\_\{\\beta\\geq u\_\{1\}\}e^\{\(s\)\}in class

kk\}

13:for

j=1j=1to

LLdo

14:

θk,j←uj\\theta\_\{k,j\}\\leftarrow u\_\{j\}
15:

dk,j←DeferredCountd\_\{k,j\}\\leftarrow\\mathrm\{DeferredCount\}
16:

ek,j←AcceptedErrorse\_\{k,j\}\\leftarrow\\mathrm\{AcceptedErrors\}\{oracle: only accepted sLLM mistakes matter\}

17:

DeferredCount←DeferredCount\+Nj\\mathrm\{DeferredCount\}\\leftarrow\\mathrm\{DeferredCount\}\+N\_\{j\}
18:

AcceptedErrors←AcceptedErrors−Ejs\\mathrm\{AcceptedErrors\}\\leftarrow\\mathrm\{AcceptedErrors\}\-E^\{s\}\_\{j\}
19:endfor\{Defer\-all option: accept none in class

kk\}

20:Add defer\-all option:

dk,L\+1←nkd\_\{k,L\+1\}\\leftarrow n\_\{k\},

ek,L\+1←0e\_\{k,L\+1\}\\leftarrow 0
21:endfor

22:Returnoption matrices

Algorithm 5Build per\-class option matrices \(non\-oracle\)1:Input:

\{\(βi,𝒂^is,𝒂^im,𝒂i\)\}i=1N\\\{\(\\beta\_\{i\},\\hat\{\\boldsymbol\{a\}\}^\{s\}\_\{i\},\\hat\{\\boldsymbol\{a\}\}^\{m\}\_\{i\},\\boldsymbol\{a\}\_\{i\}\)\\\}\_\{i=1\}^\{N\}, number of classes

KK
2:Output:For each class

kk: options

\{\(dk,j,ek,j,θk,j\)\}j=1Jk\\\{\(d\_\{k,j\},e\_\{k,j\},\\theta\_\{k,j\}\)\\\}\_\{j=1\}^\{J\_\{k\}\}
3:Compute

eis=𝟙​\{𝒂^is≠𝒂i\}e^\{s\}\_\{i\}=\\mathds\{1\}\\\{\\hat\{\\boldsymbol\{a\}\}^\{s\}\_\{i\}\\neq\\boldsymbol\{a\}\_\{i\}\\\},

eim=𝟙​\{𝒂^im≠𝒂i\}e^\{m\}\_\{i\}=\\mathds\{1\}\\\{\\hat\{\\boldsymbol\{a\}\}^\{m\}\_\{i\}\\neq\\boldsymbol\{a\}\_\{i\}\\\}
4:for

k=1k=1to

KKdo

5:

Sk←\{i:𝒂^is=k\}S\_\{k\}\\leftarrow\\\{i:\\hat\{\\boldsymbol\{a\}\}^\{s\}\_\{i\}=k\\\};

nk←\|Sk\|n\_\{k\}\\leftarrow\|S\_\{k\}\|
6:if

nk=0n\_\{k\}=0then

7:Store one trivial option

\(d,e\)=\(0,0\)\(d,e\)=\(0,0\)andcontinue

8:endif

9:Sort

SkS\_\{k\}by

βi\\beta\_\{i\}ascending; let

u1<…<uLu\_\{1\}<\\ldots<u\_\{L\}be distinct values in class

kk
10:For each level

ℓ\\ell, let

Gℓ=\{i∈Sk:βi=uℓ\}G\_\{\\ell\}=\\\{i\\in S\_\{k\}:\\beta\_\{i\}=u\_\{\\ell\}\\\}and compute:

Nℓ=\|Gℓ\|,Eℓs=∑i∈Gℓeis,Eℓm=∑i∈GℓeimN\_\{\\ell\}=\|G\_\{\\ell\}\|,\\quad E^\{s\}\_\{\\ell\}=\\sum\_\{i\\in G\_\{\\ell\}\}e^\{s\}\_\{i\},\\quad E^\{m\}\_\{\\ell\}=\\sum\_\{i\\in G\_\{\\ell\}\}e^\{m\}\_\{i\}
11:

DeferredErrors←0\\mathrm\{DeferredErrors\}\\leftarrow 0\{

∑β<ujem\\sum\_\{\\beta<u\_\{j\}\}e^\{m\}in class

kk\}

12:

DeferredCount←0\\mathrm\{DeferredCount\}\\leftarrow 0\{

\#​\{i∈Sk:β<uj\}\\\#\\\{i\\in S\_\{k\}:\\beta<u\_\{j\}\\\}\}

13:

AcceptedErrors←∑ℓ′=1LEℓ′s\\mathrm\{AcceptedErrors\}\\leftarrow\\sum\_\{\\ell^\{\\prime\}=1\}^\{L\}E^\{s\}\_\{\\ell^\{\\prime\}\}\{

∑β≥u1es\\sum\_\{\\beta\\geq u\_\{1\}\}e^\{s\}in class

kk\}

14:for

j=1j=1to

LLdo

15:

θk,j←uj\\theta\_\{k,j\}\\leftarrow u\_\{j\}
16:

dk,j←DeferredCountd\_\{k,j\}\\leftarrow\\mathrm\{DeferredCount\}
17:

ek,j←DeferredErrors\+AcceptedErrorse\_\{k,j\}\\leftarrow\\mathrm\{DeferredErrors\}\+\\mathrm\{AcceptedErrors\}\{Update totals for next option

j\+1j\{\+\}1\}

18:

DeferredErrors←DeferredErrors\+Ejm\\mathrm\{DeferredErrors\}\\leftarrow\\mathrm\{DeferredErrors\}\+E^\{m\}\_\{j\}
19:

DeferredCount←DeferredCount\+Nj\\mathrm\{DeferredCount\}\\leftarrow\\mathrm\{DeferredCount\}\+N\_\{j\}
20:

AcceptedErrors←AcceptedErrors−Ejs\\mathrm\{AcceptedErrors\}\\leftarrow\\mathrm\{AcceptedErrors\}\-E^\{s\}\_\{j\}
21:endfor

22:Add defer\-all option:

dk,L\+1←nkd\_\{k,L\+1\}\\leftarrow n\_\{k\},

ek,L\+1←∑ℓ=1LEℓme\_\{k,L\+1\}\\leftarrow\\sum\_\{\\ell=1\}^\{L\}E^\{m\}\_\{\\ell\}
23:endfor

24:Returnoption matrices

##### Dynamic programming \(pseudo\-polynomial\)\.

The finite\-sample problem is a multi\-choice knapsack instance\. An exact solution can be obtained via dynamic programming over the integer error budgetBB, storing for each attainable total error the minimum total deferrals achievable after processing a subset of classes\. In the worst case, the runtime is

𝒪​\(∑k=1K\(Uk\+1\)⋅B\)=𝒪​\(U​B\),\\mathcal\{O\}\\\!\\Big\(\\sum\_\{k=1\}^\{K\}\(U\_\{k\}\+1\)\\cdot B\\Big\)=\\mathcal\{O\}\(UB\),and memory is𝒪​\(B\)\\mathcal\{O\}\(B\)if one stores only the current DP frontier \(plus backpointers if reconstructing the selected options\)\. SinceB=Θ​\(N\)B=\\Theta\(N\)andU≤Θ​\(N\)U\\leq\\Theta\(N\), the worst\-case runtime can be𝒪​\(N2\)\\mathcal\{O\}\(N^\{2\}\), which can be heavy for largeNN\. In practice, we maintain only the Pareto\-optimal \(non\-dominated\) frontier of states\. An implementation sketch is presented in Algorithm[6](https://arxiv.org/html/2609.15992#alg6)\. There is no change in the oracle and the non\-oracle setting, apart from the fact that each algorithm takes as input the corresponding ‘per\-class option matrices’\.

Algorithm 6Multi\-thresholds selection \(oracle & non\-oracle, MC, exact DP\)1:Input:Option matrices

\{\(dk,j,ek,j,θk,j\)\}j=1Jk\\\{\(d\_\{k,j\},e\_\{k,j\},\\theta\_\{k,j\}\)\\\}\_\{j=1\}^\{J\_\{k\}\}for each class

kk, sample size

NN, accuracy bound

ξ\\xi
2:Output:Optimal threshold vector

𝜽∗\\boldsymbol\{\\theta\}^\{\*\}
3:Set error budget

b←1−ξb\\leftarrow 1\-\\xi
4:

B←⌊b​N⌋B\\leftarrow\\lfloor bN\\rfloor\{integer error budget\}

5:

DP←\{0↦0\}\\mathrm\{DP\}\\leftarrow\\\{0\\mapsto 0\\\}\{error count

→\\tomin deferrals\}

6:for

k=1k=1to

KKdo

7:

DP′←\\mathrm\{DP^\{\\prime\}\}\\leftarrowempty map

8:for all

\(E↦D\)\(E\\mapsto D\)in

DP\\mathrm\{DP\}do

9:for alloptions

jjof class

kkdo

10:

E′←E\+ek,jE^\{\\prime\}\\leftarrow E\+e\_\{k,j\}
11:if

E′≤BE^\{\\prime\}\\leq Bthen

12:

DP′​\[E′\]←min⁡\(DP′​\[E′\],D\+dk,j\)\\mathrm\{DP^\{\\prime\}\}\[E^\{\\prime\}\]\\leftarrow\\min\\big\(\\mathrm\{DP^\{\\prime\}\}\[E^\{\\prime\}\],\\,D\+d\_\{k,j\}\\big\)
13:Store backpointer for reconstruction

14:endif

15:endfor

16:endfor

17:Pareto\-prune

DP′\\mathrm\{DP^\{\\prime\}\}\(remove dominated states\) and set

DP←DP′\\mathrm\{DP\}\\leftarrow\\mathrm\{DP^\{\\prime\}\}
18:endfor

19:Choose

E∗∈arg⁡minE≤B⁡DP​\[E\]E^\{\*\}\\in\\arg\\min\_\{E\\leq B\}\\mathrm\{DP\}\[E\]
20:Backtrack to recover chosen option

jk∗j\_\{k\}^\{\*\}for each class

21:Set

θk∗←θk,jk∗\\theta\_\{k\}^\{\*\}\\leftarrow\\theta\_\{k,j\_\{k\}^\{\*\}\}for all

kkand return

𝜽∗\\boldsymbol\{\\theta\}^\{\*\}

##### Lagrangian sweep \(scalable approximation\)\.

To improve scalability, we consider a Lagrangian relaxation of the \(empirical\) error constraint\. For a fixed multiplierλ≥0\\lambda\\geq 0, we select for each classkkthe option minimizingdk,j\+λ​ek,jd\_\{k,j\}\+\\lambda\\,e\_\{k,j\}\. This relaxation*separates across classes*, so for anyλ\\lambdawe can compute a candidate solution by a linear scan through each class option list\. Sweeping over a grid ofLLmultipliers producesLLcandidate threshold vectors, and we retain the feasible solution with minimum deferrals\. The runtime is

𝒪​\(N​log⁡N\)\+𝒪​\(L​U\),\\mathcal\{O\}\(N\\log N\)\\;\+\\;\\mathcal\{O\}\(LU\),and memory is𝒪​\(U\)\\mathcal\{O\}\(U\)\(to store option matrices\)\. While not guaranteed to return the globally optimal solution of the original discrete constrained problem, the Lagrangian sweep is polynomial in the option\-matrix size\. The implementation outline is presented in Algorithm[7](https://arxiv.org/html/2609.15992#alg7)\. As with the DP variant, there is no change in the oracle and the non\-oracle setting, apart from the fact that each algorithm takes as input the corresponding ‘per\-class option matrices’\.

Algorithm 7Multi\-threshold selection \(non\-oracle, MC, Lagrangian sweep\)1:Input:Option matrices

\{\(dk,j,ek,j,θk,j\)\}j=1Jk\\\{\(d\_\{k,j\},e\_\{k,j\},\\theta\_\{k,j\}\)\\\}\_\{j=1\}^\{J\_\{k\}\}for each class

kk, sample size

NN, accuracy bound

ξ\\xi, multipliers

Λ=\{λ1,…,λL\}\\Lambda=\\\{\\lambda\_\{1\},\\dots,\\lambda\_\{L\}\\\}
2:Output:Optimal threshold vector

𝜽∗\\boldsymbol\{\\theta\}^\{\*\}\(best feasible among candidates\)

3:Set error budget

b←1−ξb\\leftarrow 1\-\\xi
4:

B←⌊b​N⌋B\\leftarrow\\lfloor bN\\rfloor
5:

BestDeferrals\\mathrm\{BestDeferrals\}←\+∞\\leftarrow\+\\infty;

𝜽∗\\boldsymbol\{\\theta\}^\{\*\}←\[\+∞\]K×1\\leftarrow\[\\mathbf\{\+\\infty\}\]\_\{K\\times 1\}
6:foreach

λ∈Λ\\lambda\\in\\Lambdado

7:

TotalDeferrals\\mathrm\{TotalDeferrals\}←0\\leftarrow 0;

TotalErrors\\mathrm\{TotalErrors\}←0\\leftarrow 0
8:for

k=1k=1to

KKdo

9:

jk←arg⁡minj⁡\(dk,j\+λ​ek,j\)j\_\{k\}\\leftarrow\\arg\\min\_\{j\}\\big\(d\_\{k,j\}\+\\lambda e\_\{k,j\}\\big\)
10:

TotalDeferrals\\mathrm\{TotalDeferrals\}←\\leftarrowTotalDeferrals

\+dk,jk\+d\_\{k,j\_\{k\}\}
11:

TotalErrors\\mathrm\{TotalErrors\}←\\leftarrowTotalErrors\\mathrm\{TotalErrors\}\+ek,jk\+e\_\{k,j\_\{k\}\}
12:

θk←θk,jk\\theta\_\{k\}\\leftarrow\\theta\_\{k,j\_\{k\}\}
13:endfor

14:if

TotalErrors\\mathrm\{TotalErrors\}≤B\\leq Band

TotalDeferrals\\mathrm\{TotalDeferrals\}<<BestDeferrals\\mathrm\{BestDeferrals\}then

15:

BestDeferrals\\mathrm\{BestDeferrals\}←\\leftarrowTotalDeferrals\\mathrm\{TotalDeferrals\};

𝜽∗\\boldsymbol\{\\theta\}^\{\*\}←\(θ1,…,θK\)\\leftarrow\(\\theta\_\{1\},\\dots,\\theta\_\{K\}\)
16:endif

17:endfor

18:if

𝜽∗=\[\+∞\]K×1\\boldsymbol\{\\theta\}^\{\*\}=\[\\mathbf\{\+\\infty\}\]\_\{K\\times 1\}then

19:

𝜽∗\\boldsymbol\{\\theta\}^\{\*\}←\\leftarrow𝟏K×1\\mathbf\{1\}\_\{K\\times 1\}
20:endif

21:Return

𝜽∗\\boldsymbol\{\\theta\}^\{\*\}

### F\.6Alternative approximation methods

In our optimization framework, the decision criterion and associated performance measure depend on expectations\. For example, in the single\-threshold policy \(oracle setting\), the expected cost and error are of the form

CostST−O​\(θ\)=cs\+cm​∫0θf​\(β\)​dβ,ErrorST−O​\(θ\)=∫θ1εs​\(β\)​f​\(β\)​dβ,\\mathrm\{Cost\}\_\{\\mathrm\{ST\-O\}\}\(\\theta\)=c\_\{s\}\+c\_\{m\}\\int\_\{0\}^\{\\theta\}f\(\\beta\)\\,\\mathrm\{d\}\\beta,\\quad\\mathrm\{Error\}\_\{\\mathrm\{ST\-O\}\}\(\\theta\)=\\int\_\{\\theta\}^\{1\}\\varepsilon\_\{s\}\(\\beta\)\\,f\(\\beta\)\\,\\mathrm\{d\}\\beta\\,,\(122\)We now discuss practical alternatives to MC sampling for estimating the relevant components of the underlying joint distribution and for evaluating the integrals in \([122](https://arxiv.org/html/2609.15992#A6.E122)\)\.

##### Density Estimation: Marginal and Conditional Functions\.

An alternative approach is to explicitly estimate the components of the underlying distribution, and then compute the integrals based on those estimates\. To this end, the marginal densityf​\(β\)f\(\\beta\)can be estimated nonparametrically using kernel density estimation \(KDE\)\[kde\_est\]or histogram binning\. For example, a kernel density estimator with a smoothing bandwidthhhis given by

f^​\(β\)=1N​h​∑i=1NK​\(β−βih\),\\hat\{f\}\(\\beta\)=\\frac\{1\}\{Nh\}\\sum\_\{i=1\}^\{N\}K\\\!\\left\(\\frac\{\\beta\-\\beta\_\{i\}\}\{h\}\\right\)\\,,\(123\)whereKKis a nonnegative kernel function\. Similarly, histogram binning partitions the domain\[0,1\]\[0,1\]into disjoint intervals\{Bj\}\\\{B\_\{j\}\\\}and estimates the density in binBjB\_\{j\}as the fraction of samples in that bin, normalized by bin width\.

Because the optimization also depends on the conditional error rate, an estimate of this function is also required\. A nonparametric estimate can be obtained by binning or smoothing regression\. For binning, one first partitions the range ofβ\\betainto bins\{Bj\}\\\{B\_\{j\}\\\}, and then computes

ε^s​\(Bj\)=1nj​∑i:βi∈Bj𝟏​\{𝒂im≠𝒂is\},\\hat\{\\varepsilon\}\_\{s\}\(B\_\{j\}\)=\\frac\{1\}\{n\_\{j\}\}\\sum\_\{i:\\beta\_\{i\}\\in B\_\{j\}\}\\mathbf\{1\}\\\{\\boldsymbol\{a\}\_\{i\}^\{m\}\\neq\\boldsymbol\{a\}\_\{i\}^\{s\}\\\},\(124\)wherenj=\#​\{i:βi∈Bj\}n\_\{j\}=\\\#\\\{i:\\beta\_\{i\}\\in B\_\{j\}\\\}\. Alternatively, a smooth conditional functionε^s​\(β\)\\hat\{\\varepsilon\}\_\{s\}\(\\beta\)can be obtained via e\.g\., kernel regression\[Bierens\_1994\],

ε^s​\(β\)=∑i=1N𝟏​\{𝒂im≠𝒂is\}​K​\(\(β−βi\)/h\)∑i=1NK​\(\(β−βi\)/h\)\.\\hat\{\\varepsilon\}\_\{s\}\(\\beta\)=\\frac\{\\sum\_\{i=1\}^\{N\}\\mathbf\{1\}\\\{\\boldsymbol\{a\}\_\{i\}^\{m\}\\neq\\boldsymbol\{a\}\_\{i\}^\{s\}\\\}\\,K\\big\(\(\\beta\-\\beta\_\{i\}\)/h\\big\)\}\{\\sum\_\{i=1\}^\{N\}K\\big\(\(\\beta\-\\beta\_\{i\}\)/h\\big\)\}\\,\.\(125\)Given these estimates, one can approximate the cost and error integrals by numerical quadrature of products off^​\(β\)\\hat\{f\}\(\\beta\)andε^s​\(β\)\\hat\{\\varepsilon\}\_\{s\}\(\\beta\)\.

##### Parametric Modeling and Goodness‑of‑Fit Tests

When the sample size is moderate to large and the marginal distribution exhibits a recognizable shape, fitting a parametric family tof​\(β\)f\(\\beta\)can yield a parsimonious functional representation that simplifies subsequent computation\. Parameters of the chosen family can be estimated by maximum likelihood, and the fit can be evaluated via goodness‑of‑fit tests such as the Kolmogorov–Smirnov\[ks\_test\]or Anderson–Darling\[Anderson01121954\]tests\. If the test results indicate a satisfactory fit, the fitted densityf​\(β;θ^\)f\(\\beta;\\hat\{\\theta\}\)may be used in place off​\(β\)f\(\\beta\)for computing integrals\.

A similar strategy applies to the conditional error functionεs​\(β\)\\varepsilon\_\{s\}\(\\beta\)\. Instead of nonparametric smoothing, one can posit a parametric regression model for the conditional error rate, such as a logistic regression or generalized linear model,

ε^s​\(β\)=σ​\(α\+γ​β\),\\hat\{\\varepsilon\}\_\{s\}\(\\beta\)=\\sigma\(\\alpha\+\\gamma\\beta\),\(126\)whereσ​\(⋅\)\\sigma\(\\cdot\)is the logistic function\. Parameters are estimated from the data, and model adequacy is assessed through standard regression diagnostics\. If the parametric form is appropriate, the fitted functionε^s​\(β\)\\hat\{\\varepsilon\}\_\{s\}\(\\beta\)can be used in subsequent integration\.

##### Closed‑Form vs\. Numerical Integration

If the underlying distribution and conditional functions admit simple analytic forms, the integrals \([122](https://arxiv.org/html/2609.15992#A6.E122)\) may have closed‑form expressions\. For example, whenβ∼Uniform​\(0,1\)\\beta\\sim\\mathrm\{Uniform\}\(0,1\)andεs​\(β\)=1−β\\varepsilon\_\{s\}\(\\beta\)=1\-\\beta, the integrals reduce to elementary polynomials inθ\\theta, yielding analytic expressions forCostST−O\\mathrm\{Cost\}\_\{\\mathrm\{ST\-O\}\}andErrorST−O\\mathrm\{Error\}\_\{\\mathrm\{ST\-O\}\}\. In contrast, whenf​\(β\)f\(\\beta\)follows a more complex parametric form \(e\.g\., Beta with arbitrary parameters\) or whenεs​\(β\)\\varepsilon\_\{s\}\(\\beta\)is non‑linear \(e\.g\., logistic\), closed‑form antiderivatives typically do not exist\. In such cases, numerical integration techniques can be used to evaluate the integrals\.

## Appendix GSufficiency of the confidence score

The confidence of a generated answer is a natural metric to rely on when designing decision rules in various settings\. For example, in classification tasks, it is common to use the confidence of a model’s predicted class \(which approximates the true posterior\) in threshold\-based rules\[chow1970\]\. In generative settings, where answers are more open\-ended, recent works have attempted to measure the confidence of the entire generated answer using uncertainty quantification methods\[lm\-polygraph,DBLP:conf/iclr/KuhnGF23,gupta\_uncertainty,murong\]\. It is therefore rational to explore the decision sufficiency of the confidence score, i\.e\., whether it is a sufficient statistic for our deferral decision or extra information is needed\. We provide clear definitions about sufficiency below\.

LetJJbe the extra information available at gating time \(e\.g\., the input text embeddings, full logits, etc\.\)\. We again have two actions:D=0D=0\(accept\) andD=1D=1\(defer\)\. The true answer is denoted by𝒂\\boldsymbol\{a\}\. The loss for actionDDis defined asL​\(D,𝒂,J\)L\(D,\\boldsymbol\{a\},J\)so the Lagrangian is of the form

L​\(0,𝒂,J\)=cs\+λ​𝟙​\[𝒂≠𝒂^s\],L​\(1,𝒂,J\)=cs\+cm\+λ​𝟙​\[𝒂≠𝒂^m\]\.L\(0,\\boldsymbol\{a\},J\)=c\_\{s\}\+\\lambda\\mathds\{1\}\[\\boldsymbol\{a\}\\neq\\hat\{\\boldsymbol\{a\}\}^\{s\}\],\\quad L\(1,\\boldsymbol\{a\},J\)=c\_\{s\}\+c\_\{m\}\+\\lambda\\mathds\{1\}\[\\boldsymbol\{a\}\\neq\\hat\{\\boldsymbol\{a\}\}^\{m\}\]\\,\.The conditional risk of each action givenJ=jJ=jis defined as:

r0​\(j\)=𝔼​\[L​\(0,𝒂,J\)\|J=j\],r1​\(j\)=𝔼​\[L​\(1,𝒂,J\)\|J=j\],r\_\{0\}\(j\)=\\mathbb\{E\}\[L\(0,\\boldsymbol\{a\},J\)\|J=j\],\\quad r\_\{1\}\(j\)=\\mathbb\{E\}\[L\(1,\\boldsymbol\{a\},J\)\|J=j\]\\,,and the risk difference asΔ​\(j\)=r0​\(j\)−r1​\(j\)\\Delta\(j\)=r\_\{0\}\(j\)\-r\_\{1\}\(j\)\.

So, among all policiesπ​\(J\)∈\{0,1\}\\pi\(J\)\\in\\\{0,1\\\}, the optimal one is

π∗​\(j\)=\{0,Δ​\(j\)≤01,Δ​\(j\)\>0\.\\pi^\{\*\}\(j\)=\\begin\{cases\}0,&\\Delta\(j\)\\leq 0\\\\ 1,&\\Delta\(j\)\>0\\,\.\\end\{cases\}\(127\)This is already a “threshold” on a scalar function ofJJ,Δ​\(J\)\\Delta\(J\)\.

We now introduce the scalar confidence scoreβ=β​\(J\)∈ℝ\\beta=\\beta\(J\)\\in\\mathbb\{R\}\.

###### Definition G\.1\.

A scalar score is sufficient \(relative toJJ\) for the deferral problem if there exists a functionϕ:ℝ→ℝ\\phi:\\mathbb\{R\}\\to\\mathbb\{R\}such that:

Δ​\(j\)=ϕ​\(β​\(j\)\)∀j\.\\Delta\(j\)=\\phi\(\\beta\(j\)\)\\quad\\forall j\\,\.\(128\)Equivalently, for twoj1j\_\{1\},j2j\_\{2\}withβ​\(j1\)=β​\(j2\)\\beta\(j\_\{1\}\)=\\beta\(j\_\{2\}\):Δ​\(j1\)=Δ​\(j2\)\\Delta\(j\_\{1\}\)=\\Delta\(j\_\{2\}\)\. This means that if two queries have the same scoreβ\\beta, then the expected advantage of using sLLM over mLLM \(or vice\-versa\) is the same; the extra details inJJdon’t change that trade\-off\.

For any policyπ\\pi, we can write

R​\(π\)=𝔼​\[rπ​\(J\)​\(J\)\]=𝔼​\[r1​\(J\)\+𝟙​\[π​\(J\)=0\]​\(r0​\(J\)−r1​\(J\)\)\]\.R\(\\pi\)=\\mathbb\{E\}\\left\[r\_\{\\pi\(J\)\}\(J\)\\right\]=\\mathbb\{E\}\\left\[r\_\{1\}\(J\)\+\\mathds\{1\}\[\\pi\(J\)=0\]\(r\_\{0\}\(J\)\-r\_\{1\}\(J\)\)\\right\]\\,\.\(129\)For any policyπ′​\(J\)\\pi^\{\\prime\}\(J\),R​\(π′\)≥R​\(π∗\)R\(\\pi^\{\\prime\}\)\\geq R\(\\pi^\{\*\}\)\. Soπ∗\\pi^\{\*\}is the globally optimal policy over all measurable policiesπ​\(J\)\\pi\(J\)\. Then, assuming a sufficient confidence scoreβ​\(j\)\\beta\(j\), the optimal policy becomes:

π∗​\(j\)=\{0,ϕ​\(β​\(j\)\)≤01,ϕ​\(β​\(j\)\)\>0\.\\pi^\{\*\}\(j\)=\\begin\{cases\}0,&\\phi\(\\beta\(j\)\)\\leq 0\\\\ 1,&\\phi\(\\beta\(j\)\)\>0\\,\.\\end\{cases\}\(130\)
So,π∗\\pi^\{\*\}depends only on the ‘summary’β\\betaofJJ\. If we define the scalar policy

δ∗​\(β\)=\{0,ϕ​\(β\)≤01,ϕ​\(β\)\>0,\\delta^\{\*\}\(\\beta\)=\\begin\{cases\}0,&\\phi\(\\beta\)\\leq 0\\\\ 1,&\\phi\(\\beta\)\>0\\,,\\end\{cases\}\(131\)thenπ∗​\(j\)=δ∗​\(β​\(j\)\)\\pi^\{\*\}\(j\)=\\delta^\{\*\}\(\\beta\(j\)\)\. For any other policyπ′​\(J\)\\pi^\{\\prime\}\(J\), its risk satisfiesR​\(π′\)≥R​\(π∗\)=R​\(δ∗​\(β​\(J\)\)\)R\(\\pi^\{\\prime\}\)\\geq R\(\\pi^\{\*\}\)=R\(\\delta^\{\*\}\(\\beta\(J\)\)\)\. No policy that usesJJcan beat the best policy that uses only the scalar scoreβ​\(J\)\\beta\(J\), provided that it is sufficient\. If we now assume thatΔ​\(J\)\\Delta\(J\)\(the excess error of sLLM over mLLM\) is a non\-increasing function of the confidence score, there exists a non\-increasingϕ​\(⋅\)\\phi\(\\cdot\)such thatΔ​\(J\)=ϕ​\(β​\(J\)\)\\Delta\(J\)=\\phi\(\\beta\(J\)\)\. Then, as shown in Appendix[C](https://arxiv.org/html/2609.15992#A3), the acceptance setS=\{β:ϕ​\(β\)≤0\}S=\\\{\\beta:\\phi\(\\beta\)\\leq 0\\\}is either empty \(never accept\), all ofℝ\\mathbb\{R\}\(always accept\), or a half\-line\[β∗,∞\)\[\\beta^\{\*\},\\infty\)\(accept ifβ≥β∗\\beta\\geq\\beta^\{\*\}\) withβ∗=infS\\beta^\{\*\}=\\inf S\.121212Usually,β∈\[0,1\]\\beta\\in\[0,1\]so the intervals become\[0,1\]\[0,1\]\(always accept\) and\[β∗,1\]\[\\beta^\{\*\},1\]\(accept ifβ≥β∗\\beta\\geq\\beta^\{\*\}\)\.

Theoretically, sufficiency is a property of the true joint distribution\. In practice, we approximated it using the finite\-sample datasets in the following way: we fit models that predicted the difference of correctness of the two LLMs, i\.e\., “sLLM better than mLLM?" using*\(i\) onlyβ\\beta*and*\(ii\)β\\betaplus other components ofJJ*\. For these extra components, we considered either the whole input text embedded using Sentence Transformers\[DBLP:conf/emnlp/ReimersG19\]or different statistics of the probability outputs of the sLLM \(e\.g\., entropy, absolute difference etc\.\)\. For the predictive model, we used a Gradient Boosting Regressor\[friedman2000greedy\]\. Using a cross\-fitting approach \(Algorithm[8](https://arxiv.org/html/2609.15992#alg8)\), we compared the mean squared error of the two models via a permutation test to assess whether includingJJyields significant predictive improvement \(more criteria about the sufficiency of confidence scores can be found in\[DBLP:conf/nips/JitkrittumGMNRK23\]\)\. In preliminary experiments that we conducted, the difference between the errors was not statistically significant, so we opted to useβ\\betaas our deferral signal\. For example, Table[18](https://arxiv.org/html/2609.15992#A7.T18)shows results for two different pairs of models\.

Table 18:Results of initial sufficiency experiments\. MSEβand MSEβ\+Jare the cross\-validated mean squared errors of the null and full models, respectively\.ΔMSE\\Delta\_\{\\mathrm\{MSE\}\}= MSEβ\- MSEβ\+J\. Permutationpp\-value measures whether the observedΔMSE\\Delta\_\{\\mathrm\{MSE\}\}is significantly larger than expected under the null hypothesis of no added predictive information fromJJand is based onT=10T=10permutations ofJJwhile keepingβ\\betafixed\.Algorithm 8Score Sufficiency Testing1:Input:Dataset

\{\(Ji,𝒂i,𝒂^is,𝒂^im,βi\)\}i=1N\\\{\(J\_\{i\},\\boldsymbol\{a\}\_\{i\},\\hat\{\\boldsymbol\{a\}\}^\{s\}\_\{i\},\\hat\{\\boldsymbol\{a\}\}^\{m\}\_\{i\},\\beta\_\{i\}\)\\\}\_\{i=1\}^\{N\}
2:for

i=1i=1to

NNdo

3:Output:Statistical test of whether score

ssis sufficient for predicting correctness difference

Δ\\Delta
4:Compute binary error indicators:

Eis←𝟙​\{𝒂^is≠𝒂i\},Eim←𝟙​\{𝒂^im≠𝒂i\}E^\{s\}\_\{i\}\\leftarrow\\mathds\{1\}\\\{\\hat\{\\boldsymbol\{a\}\}^\{s\}\_\{i\}\\neq\\boldsymbol\{a\}\_\{i\}\\\},\\quad E^\{m\}\_\{i\}\\leftarrow\\mathds\{1\}\\\{\\hat\{\\boldsymbol\{a\}\}^\{m\}\_\{i\}\\neq\\boldsymbol\{a\}\_\{i\}\\\}
5:Define signed error difference:

Δi←Eis−Eim\\Delta\_\{i\}\\leftarrow E^\{s\}\_\{i\}\-E^\{m\}\_\{i\}
6:endfor

7:Partition data into

KKfolds

8:foreach fold

kkdo

9:Train ‘null’ learner

f^null\\hat\{f\}\_\{\\mathrm\{null\}\}on

k−1k\-1training folds to predict

Δ\\Deltafrom

ss
10:Train ‘full’ learner

f^full\\hat\{f\}\_\{\\mathrm\{full\}\}on

k−1k\-1training folds to predict

Δ\\Deltafrom

\(s,J\)\(s,J\)
11:Evaluate predictions on held\-out fold:

Δ^nullk,Δ^fullk\\hat\{\\Delta\}\_\{\\mathrm\{null\}\}^\{k\},\\ \\hat\{\\Delta\}\_\{\\mathrm\{full\}\}^\{k\}
12:endfor

13:Aggregate mean squared errors:

MSEnull=1N​∑i\(Δi−Δ^null,i\)2,MSEfull=1N​∑i\(Δi−Δ^full,i\)2\\text\{MSE\}\_\{\\mathrm\{null\}\}=\\frac\{1\}\{N\}\\sum\_\{i\}\(\\Delta\_\{i\}\-\\hat\{\\Delta\}\_\{\{\\mathrm\{null\}\},i\}\)^\{2\},\\quad\\text\{MSE\}\_\{\\mathrm\{full\}\}=\\frac\{1\}\{N\}\\sum\_\{i\}\(\\Delta\_\{i\}\-\\hat\{\\Delta\}\_\{\{\\mathrm\{full\}\},i\}\)^\{2\}
14:Compute observed test statistic:

Tobs=MSEnull−MSEfullT\_\{\\mathrm\{obs\}\}=\\text\{MSE\}\_\{\\mathrm\{null\}\}\-\\text\{MSE\}\_\{\\mathrm\{full\}\}
15:Obtain permutation distribution:

16:for

p=1​…​npermp=1\\ldots n\_\{\\mathrm\{perm\}\}do

17:Permute feature set

Jip←J\_\{i\}^\{p\}\\leftarrowshuffle of

JJ
18:Repeat train/predict to obtain

MSEfullp\\text\{MSE\}\_\{\\mathrm\{full\}\}^\{p\}
19:

Tp←MSEnull−MSEfullpT^\{p\}\\leftarrow\\text\{MSE\}\_\{\\mathrm\{null\}\}\-\\text\{MSE\}\_\{\\mathrm\{full\}\}^\{p\}
20:endfor

21:Approximate p\-value:

1\+∑p=1nperm\{Tp≥Tobs\}nperm\+1\\frac\{1\+\\sum\_\{p=1\}^\{n\_\{\\mathrm\{perm\}\}\}\\\{T^\{p\}\\geq T\_\{\\text\{obs\}\}\\\}\}\{n\_\{\\mathrm\{perm\}\}\+1\}
22:ifp\-value

<0\.05<0\.05then

23:Reject

H0H\_\{0\}:

JJadds predictive value beyond

β\\beta
24:else

25:Fail to reject

H0H\_\{0\}
26:endif