Routing Without Training: Controllable-Ratio LLM Offloading via Reliability Gating

# Routing Without Training: Controllable-Ratio LLM Offloading via Reliability Gating
Source: [https://arxiv.org/html/2607.20481](https://arxiv.org/html/2607.20481)
Evan Chen1, Shiqiang Wang2, Kevin S\. Chan3, Su Wang4, Christopher G\. Brinton11Purdue University2University of Exeter3Army Research Laboratory4Princeton University

###### Abstract

Local\-cloud collaboration is a practical way to deploy large language models under resource constraints, but existing methods often rely on trained routers or collaboration\-aware finetuning that tie routing behavior to a particular operating regime\. In this work, we show that such training may be unnecessary: the local model’s own inference\-time agreement across sampled responses already provides a strong signal for deciding when to trust local execution vs when to offload to a stronger cloud model\. We propose CARGO, a training\-free routing framework that estimates this agreement through prompt\-varied sampling, applies Bayesian early stopping for sample\-efficient uncertainty control, and supports arbitrary target collaboration ratios through lightweight deployment\-time calibration\. Across diverse reasoning and question\-answering tasks, multiple local LLM families and scales, and both pretrained and finetuned local models, CARGO consistently outperforms other training\-free baselines and in several settings surpasses supervised learned routers\. These results suggest that effective and adaptable local\-cloud collaboration can emerge directly from the local model’s intrinsic response behavior, without requiring an additional trained router\.

## 1Introduction

As large language model \(LLM\) services are increasingly deployed on edge devices and other resource\-constrained platforms, practical systems must balance the efficiency of local inference with the capability of stronger cloud models\. In practical deployments, relying exclusively on cloud inference is undesirable because it introduces communication overhead, higher latency, recurring monetary cost, and potential privacy or connectivity concerns\. These issues are especially pronounced in settings such as mobile devices, on\-premise assistants, and edge applications, where local responsiveness and reduced dependence on remote infrastructure are often essential\[[30](https://arxiv.org/html/2607.20481#bib.bib12),[34](https://arxiv.org/html/2607.20481#bib.bib13),[31](https://arxiv.org/html/2607.20481#bib.bib14),[16](https://arxiv.org/html/2607.20481#bib.bib15),[29](https://arxiv.org/html/2607.20481#bib.bib16)\]\. Local\-cloud collaboration is a natural alternative that preserves the quality benefits of large remote models while reducing the cost of always invoking the cloud\. For example, a lightweight local LLM can handle many routine user queries, while more difficult demands are selectively offloaded to a stronger cloud model\[[28](https://arxiv.org/html/2607.20481#bib.bib11)\]\. As such, the central question for effective local\-cloud collaboration is: when should the system trust the local model, and when should it invoke cloud assistance? Existing approaches answer this question through explicit routers or collaboration\-aware training\[[9](https://arxiv.org/html/2607.20481#bib.bib10),[8](https://arxiv.org/html/2607.20481#bib.bib5)\], but typically optimize collaborative behavior around a particular operating regime, creating a mismatch with real deployments where budget, latency tolerance, backend load, and service\-level objectives are time\-varying\.

This limitation suggests that offloading should not be determined solely by a task\-specific routing policy learned for one fixed regime\. Instead, effective local\-cloud collaboration requires a routing principle that remains valid as deployment requirements and target collaboration ratios change\. In particular, the system should trust local inference when the local model can answer reliably and invoke cloud assistance otherwise, while still allowing the overall degree of collaboration to be adjusted as needed\. Therefore, we ground local\-cloud routing in an inference\-time estimate of the local model’s own reliability\. This estimate reflects query\-level capability and can be calibrated at deployment time to meet different target collaboration ratios, such as offloading10%10\\%,30%30\\%, or50%50\\%of queries\. The question is whether this reliability\-driven routing can be achieved without training a separate router at all\. These observations motivate the following research questions\.

How can a system tell, at inference time, whether local models can independently and reliably answer a query?A central challenge is that the routing decision must be made without access to ground\-truth correctness, using only the local model’s own behavior at inference time\. Moreover, the desired signal should reflect the model’s intrinsic problem understanding rather than rely on task\-specific supervision, retraining, or a separately learned router\. The difficulty is therefore to identify an intrinsic and broadly applicable reliability signal that remains meaningful across tasks, model families, and deployment settings\.

How can such a reliability signal be used to make routing both efficient and adaptable at deployment time?Reliability estimation itself consumes computation: collecting more evidence can improve confidence in the estimate, at the cost of higher latency and local inference overhead\. At the same time, practical systems require routing behavior that can adapt to changing collaboration budgets, rather than remaining tied to a single fixed operating point\. The difficulty is therefore to use reliability information in a sample\-efficient manner while still enabling flexible deployment\-time control over when cloud assistance is invoked\.

Our key insight is that effective local\-cloud routing may not require training a separate router at all\. By properly leveraging the local model’s own response behavior, one can derive a training\-free routing principle that remains effective across diverse tasks and local LLM backbones while still supporting flexible deployment\-time collaboration\. This suggests that the local model’s intrinsic response agreement is itself a strong foundation for local\-cloud collaboration, and in some cases may even provide a stronger routing signal than alternatives that are explicitly trained for specific task properties/distributions\.

Our Contributions\.Building on this insight, we propose Collaboration\-Adaptive Routing via Agreement\-Guided Offloading \(CARGO\), a training\-free local\-cloud collaboration framework for controllable LLM offloading\. CARGO turns agreement among prompt\-varied local responses into a calibrated routing policy\. It \(i\) estimates local reliability from response stability, \(ii\) reduces unnecessary sampling through Bayesian early stopping, and \(iii\) adjusts routing behavior through a lightweight deployment\-time calibration step\. Since CARGO operates only through prompts and generated outputs, it applies when local fine\-tuning or post\-training is unavailable and can even be added as an inference\-time wrapper to existing LLM or agent\-based applications\. This design makes agreement\-guided routing practical for resource\-constrained settings, where latency, compute, and cloud\-usage budgets may vary across devices, tasks, and deployment conditions\. Our main contributions are as follows:

- •We identify prompt\-varied response agreement as a transferable inference\-time reliability signal for local\-cloud routing, showing that it yields greater correctness and consistency than than self\-reported confidence or Chain\-of\-Thought \(CoT\) steps \(Sec\.[2](https://arxiv.org/html/2607.20481#S2)\)\.
- •We introduce a practical routing procedure that combines Bayesian early stopping with lightweight calibration, reducing the number of local generations needed for reliability estimation while supporting user\-specified collaboration ratios without retraining \(Sec\.[3](https://arxiv.org/html/2607.20481#S3)\)\.
- •We demonstrate that CARGO achieves strong accuracy under constrained cloud usage across diverse tasks and local LLM backbones\. It consistently improves over training\-free baselines and, in several settings, outperforms supervised learned routers while remaining adaptable at deployment time \(Sec\.[4](https://arxiv.org/html/2607.20481#S4)\)\.

Related Works\.Existing work on local\-cloud collaboration for practical, resource\-constrained LLM deployments has pursued two main directions\. The first uses explicit routing mechanisms, such as external classifiers, preference\-trained routers, or confidence\- and cost\-aware query selection policies, to decide when inputs should be offloaded\[[8](https://arxiv.org/html/2607.20481#bib.bib5),[19](https://arxiv.org/html/2607.20481#bib.bib6),[4](https://arxiv.org/html/2607.20481#bib.bib7),[18](https://arxiv.org/html/2607.20481#bib.bib8)\]\. The second integrates routing behavior into local\-model training itself, training the local model to jointly improve task performance and offloading decisions during post\-training\[[3](https://arxiv.org/html/2607.20481#bib.bib9),[9](https://arxiv.org/html/2607.20481#bib.bib10)\]\. While effective, these approaches still rely on learned routing behavior, either through separately trained routing modules or through collaboration\-aware post\-training\. Such learned routing can be costly to obtain and rigid at deployment time, especially when budgets, latency constraints, or target collaboration ratios change\. In contrast, our focus is on whether routing can instead be grounded directly in an intrinsic inference\-time reliability signal extracted from the local model’s own response behavior, without training a router or fine\-tuning the local model for collaboration\.

Recent work on self\-consistency shows that agreement across multiple sampled responses is a strong inference\-time signal that can be extracted directly from a model’s own generation behavior, without requiring additional supervision or fine\-tuning\[[5](https://arxiv.org/html/2607.20481#bib.bib2),[26](https://arxiv.org/html/2607.20481#bib.bib1),[23](https://arxiv.org/html/2607.20481#bib.bib4)\]\. Beyond the original majority\-vote formulation, subsequent studies show that this signal remains useful across more general generation settings and can be strengthened through improved aggregation or confidence\-aware selection\[[24](https://arxiv.org/html/2607.20481#bib.bib17)\]\. However, these works primarily study a local\-only inference setting, where cross\-sample agreement is used to select or refine the same model’s final answer\. Local\-cloud collaboration introduces a new decision dimension: the system need not rely on the best local aggregate, but can instead route uncertain cases to a stronger cloud model\. This shifts the role of agreement from answer aggregation to reliability\-aware routing, where high agreement supports local execution and low agreement motivates cloud assistance\. Since this decision requires repeated sampling on a resource\-limited local model, sample\-efficient agreement estimation becomes essential\.

![Refer to caption](https://arxiv.org/html/2607.20481v1/images/framework.png)Figure 1:Overview of CARGO\. The local model estimates agreement under prompt\-varied sampling, then applies Bayesian early stopping to decide whether to accept the local response or offload to the cloud\.
## 2Motivation

Training\-Free Reliability Signals for Local\-cloud Collaboration\.Local\-cloud collaboration has emerged as a practical paradigm for deploying LLMs under resource constraints\. In such systems, a lightweight local model handles most queries while more difficult cases are routed to a stronger cloud model\. A central challenge is therefore to design a routing mechanism that decides, for each query, whether the local response can be trusted\.

![Refer to caption](https://arxiv.org/html/2607.20481v1/images/ARC__MinervaAMath__SVAMP__SQuAD.png)Figure 2:Comparison of inference\-time routing signals using Qwen2\.5\-7B\-Instruct across multiple tasks\. Compared with CoT steps and self\-reported confidence, response agreement exhibits a clearer monotonic relationship with accuracy, making it a stronger basis for reliability\-guided routing\.Many existing approaches rely on training an additional routing model or fine\-tuning the local model to produce calibrated confidence scores\. However, these approaches require task\-specific supervision and retraining, limiting their applicability across domains and deployment conditions\. We therefore ask whether the local model’s own inference\-time behavior already contains signals that can distinguish reliable from unreliable local predictions\. To this end, we compare several candidate signals and evaluate how consistently they align with correctness across tasks\.

Existing Self\-Reflection Signals\.A natural hypothesis is that language models exhibit implicit indicators of task difficulty during generation\. Two commonly used signals are Chain\-of\-Thought \(CoT\) Steps and Self\-Reported Confidence\. The first uses the number of intermediate reasoning steps under CoT prompting as a proxy for uncertainty, based on the heuristic that harder problems require longer reasoning\[[9](https://arxiv.org/html/2607.20481#bib.bib10)\]\. The second explicitly asks the model to assess its own answer by outputting a confidence score\[[27](https://arxiv.org/html/2607.20481#bib.bib31),[32](https://arxiv.org/html/2607.20481#bib.bib32),[22](https://arxiv.org/html/2607.20481#bib.bib33)\]\. However, as shown in Figure[2](https://arxiv.org/html/2607.20481#S2.F2), both signals exhibit weak and unstable correlation with correctness in our setting, limiting their usefulness as training\-free routing signals without additional task\-specific adaptation or fine\-tuning\.

Response Agreement for Reliability\-Guided Routing\.We leverage self\-consistency, in which models encode their own reliability through response agreement across samples, to develop a principled methodology for local\-cloud routing\. Given a queryxx, a LLM generates an autoregressive answer by repeatedly selecting the next token from a probability distribution over the vocabulary; we refer to this generation procedure as decoding\. LetA∼P\(⋅∣x\)A\\sim P\(\\cdot\\mid x\)denote the induced distribution over final answers\. After samplingkkresponses, we obtain counts\{cka\}a\\\{c\_\{k\}^\{a\}\\\}\_\{a\}over unique answers and define the empirical agreement level

θ^k=maxa⁡ckak\.\\hat\{\\theta\}\_\{k\}=\\max\_\{a\}\\frac\{c\_\{k\}^\{a\}\}\{k\}\.\(1\)Here,θ^k\\hat\{\\theta\}\_\{k\}measures how strongly the sampled responses concentrate on a single answer\. Empirically,θ^k\\hat\{\\theta\}\_\{k\}correlates much more strongly with correctness than CoT steps or self\-reported confidence as shown in Figure[2](https://arxiv.org/html/2607.20481#S2.F2), with high agreement typically corresponding to questions the local model can answer reliably and low agreement indicating ambiguity or difficulty\.

From Fixed Sampling to Adaptive Agreement Estimation\.The usefulness of agreement as a reliability signal depends not only on how agreement is measured, but also on how efficiently it is estimated\. Standard self\-consistency typically draws a fixed number of samplesKKfor every query\. However, queries differ in how quickly their agreement behavior becomes clear\. For easy queries, sampled responses concentrate on the same answer after only a few generations, so additional samples provide little new routing information\. For difficult or ambiguous queries, more evidence may be needed before the system can confidently decide whether the local model should be trusted\.

This observation suggests that agreement\-based routing should allocate sampling effort adaptively rather than uniformly\. Such adaptivity is especially important in local\-cloud collaboration, where repeated local generations introduce latency and compute overhead, while the system must still respect deployment\-level constraints such as a target collaboration ratio\. Instead of treatingKKas a fixed budget for every input, CARGO sequentially samples only until the agreement estimate is sufficiently reliable, and then decides whether to accept the local prediction or invoke cloud assistance\.

In the next section, we formalize these ideas by interpreting multi\-sample agreement as an estimator of the mode \(frequency\) mass and developing a Bayesian framework to quantify its uncertainty\. This leads to a Bayesian early\-stopping procedure that terminates once the agreement estimate is sufficiently precise, after which routing decisions can be calibrated to a desired target collaboration ratio\.

Input:local modelMLM\_\{L\}; cloud modelMCM\_\{C\}; prompt set𝒮=\{si\}i=1M\\mathcal\{S\}=\\\{s\_\{i\}\\\}\_\{i=1\}^\{M\}; temperatureTT; credible level1−δ1\-\\delta; max samplesKmaxK\_\{\\max\}; warmup iterationsT0T\_\{0\}; warmup batch sizeBB; target offload ratioρ\\rho; step sizeηλ\\eta\_\{\\lambda\}; Beta prior\(α0,β0\)\(\\alpha\_\{0\},\\beta\_\{0\}\); width thresholdε\\varepsilon; logistic slopeγ\\gamma

Output:for each query

xx: final answer

y​\(x\)y\(x\)and routing indicator

r​\(x\)∈\{0,1\}r\(x\)\\in\\\{0,1\\\}\(1 = offload to cloud\)

1

2Warmup \(ratio control\):initialize intercept

λ∈ℝ\\lambda\\in\\mathbb\{R\}
3sample a fixed warmup batch

ℬ=\{x\(i\)\}i=1B\\mathcal\{B\}=\\\{x^\{\(i\)\}\\\}\_\{i=1\}^\{B\}
4for*t←1t\\leftarrow 1toT0T\_\{0\}*do

5foreach*x∈ℬx\\in\\mathcal\{B\}*do

6

\(y​\(x\),r​\(x\)\)←RouteQuery​\(x,λ\)\(y\(x\),r\(x\)\)\\leftarrow\\textsc\{RouteQuery\}\(x,\\lambda\)
7

8compute empirical

r¯t←1B​∑x∈ℬr​\(x\)\\bar\{r\}\_\{t\}\\leftarrow\\frac\{1\}\{B\}\\sum\_\{x\\in\\mathcal\{B\}\}r\(x\), update threshold

λ←λ−ηλ​\(r¯t−ρ\)\\lambda\\leftarrow\\lambda\-\\eta\_\{\\lambda\}\(\\bar\{r\}\_\{t\}\-\\rho\)
9Deploy:fix

λ\\lambdafor subsequent queries\.

10

11Function*RouteQuery\(*x,λx,\\lambda*\)*:

12initialize multiset

𝒜←∅\\mathcal\{A\}\\leftarrow\\emptysetand counts

c​\(⋅\)←0c\(\\cdot\)\\leftarrow 0
13for*k←1k\\leftarrow 1toKmaxK\_\{\\max\}*do

14

15sample prompt template

sk∼Unif​\(𝒮\)s\_\{k\}\\sim\\mathrm\{Unif\}\(\\mathcal\{S\}\), sample response

Ak∼ML\(⋅∣x,sk;T\)A\_\{k\}\\sim M\_\{L\}\(\\cdot\\mid x,s\_\{k\};T\)
16update counts:

𝒜k←𝒜k−1∪\{Ak\}\\mathcal\{A\}\_\{k\}\\\!\\leftarrow\\\!\\mathcal\{A\}\_\{k\-1\}\\cup\\\{A\_\{k\}\\\},

c​\(Ak\)←c​\(Ak\)\+1c\(A\_\{k\}\)\\\!\\leftarrow\\\!c\(A\_\{k\}\)\+1
17update counts and empirical agreement

\(a^k,c^k\)←\(arg⁡maxa⁡cka,maxa⁡cka\)\(\\hat\{a\}\_\{k\},\\hat\{c\}\_\{k\}\)\\leftarrow\(\\arg\\max\_\{a\}c\_\{k\}^\{a\},\\;\\max\_\{a\}c\_\{k\}^\{a\}\)
18compute posterior

θ\|𝒜k∼Beta​\(α0\+c^k,β0\+k−c^k\)\\theta\|\\mathcal\{A\}\_\{k\}\\sim\\mathrm\{Beta\}\(\\alpha\_\{0\}\+\\hat\{c\}\_\{k\},\\beta\_\{0\}\+k\-\\hat\{c\}\_\{k\}\)and credible interval

\[Lk,Uk\]\[L\_\{k\},U\_\{k\}\]
19

20if*Uk−Lk≤εU\_\{k\}\-L\_\{k\}\\leq\\varepsilon*then

break

//confidence\-based early exit

21

22

23

24compute agreement estimate

θ^←c^k/k\\hat\{\\theta\}\\leftarrow\\hat\{c\}\_\{k\}/kand offload probability

p←σ​\(γ​\(λ−θ^\)\)p\\leftarrow\\sigma\\\!\\big\(\\gamma\(\\lambda\-\\hat\{\\theta\}\)\\big\)\.

25sample

r∼Bernoulli​\(p\)r\\sim\\mathrm\{Bernoulli\}\(p\)
26if*r=0r=0*then

return

\(a^,0\)\(\\hat\{a\},0\)
//accept local answer

27

28else

29query cloud

y∼MC\(⋅∣x\)y\\sim M\_\{C\}\(\\cdot\\mid x\)
return

\(y,1\)\(y,1\)
//offload to cloud

30

31

Algorithm 1CARGO: Collaboration\-Adaptive Routing via Agreement\-Guided Offloading
## 3CARGO: Training\-Free Agreement\-Guided Routing

We consider a local\-cloud inference setting with a lightweight local modelMLM\_\{L\}and a stronger cloud modelMCM\_\{C\}\. Given a queryxx, the system must decide whether to accept the local prediction or offload to the cloud\. To this end, CARGO estimates reliability through multi\-sample agreement, using \(i\) prompt variation, \(ii\) Bayesian mode\-mass estimation, and \(iii\) Bayesian early stopping with calibrated probabilistic routing\. The complete procedure is summarized in Algorithm[1](https://arxiv.org/html/2607.20481#algorithm1)\.

Diverse Sampling via Prompt Variation\.Sec\.[2](https://arxiv.org/html/2607.20481#S2)shows that agreement across multiple generations provides a useful reliability signal, but estimating this agreement requires informative variation across responses\. A standard way to obtain such variation is temperature sampling, where higher temperature increases randomness in token selection\. However, this randomness can perturb the model’s native generation behavior and reduce the quality of individual responses\. At the other extreme, deterministic decoding with a fixed prompt often produces identical outputs, making agreement uninformative\. CARGO resolves this tension by combining deterministic decoding with prompt variation: we sample across semantically equivalent system prompts that preserve the task while encouraging different reasoning styles\. Thus, we induce diversity across reasoning trajectories and final answers without relying on stochastic sampling noise\.

Our approach offers two key advantages\. First, deterministic decoding preserves the model’s native generation behavior, ensuring that individual response quality is not degraded by temperature\-induced perturbations\. Second, by decoupling diversity from temperature, agreement estimates better reflect the model’s inherent response stability rather than artifacts of sampling randomness\.

Concretely, we define a small set of semantically equivalent system prompts𝒮=\{s1,…,sM\}\\mathcal\{S\}=\\\{s\_\{1\},\\dots,s\_\{M\}\\\}, each encouraging a different reasoning style while preserving task semantics\. For a queryxx, we sample a promptsk∼Unif​\(𝒮\)s\_\{k\}\\sim\\mathrm\{Unif\}\(\\mathcal\{S\}\)and generate the response deterministically asAk=ML​\(x,sk;T=0\)A\_\{k\}=M\_\{L\}\(x,s\_\{k\};T=0\)\. Across prompts and samples, we accumulate responses into a pooled set𝒜k=\{A1,…,Ak\}\\mathcal\{A\}\_\{k\}=\\\{A\_\{1\},\\dots,A\_\{k\}\\\}, wherekkdenotes the number of responses generated so far\.

Bayesian Estimation of Mode Mass\.Given the pooled sample set𝒜k\\mathcal\{A\}\_\{k\}, we compute counts\{cka\}a\\\{c\_\{k\}^\{a\}\\\}\_\{a\}over unique answers\. Let\(a^k,c^k\)=\(arg⁡maxa⁡cka,maxa⁡cka\)\(\\hat\{a\}\_\{k\},\\hat\{c\}\_\{k\}\)=\(\\arg\\max\_\{a\}c\_\{k\}^\{a\},\\;\\max\_\{a\}c\_\{k\}^\{a\}\)denote the empirical majority answer and its count\. The empirical agreement estimator is thenθ^k=c^k/k\\hat\{\\theta\}\_\{k\}=\\hat\{c\}\_\{k\}/k\. We interpret the*mode mass*θ=P​\(A=a^k∣x\)\\theta=P\(A=\\hat\{a\}\_\{k\}\\mid x\)as the probability that the model outputs its most frequent answer under the same sampling procedure\. Conditioned ona^k\\hat\{a\}\_\{k\}, define the indicatorZi=𝟏​\{Ai=a^k\}Z\_\{i\}=\\mathbf\{1\}\\\{A\_\{i\}=\\hat\{a\}\_\{k\}\\\}\. Under conditional independence of stochastic generations, thenZi∼Bernoulli​\(θ\)Z\_\{i\}\\sim\\mathrm\{Bernoulli\}\(\\theta\)\. We place a conjugate priorθ∼Beta​\(α0,β0\)\\theta\\sim\\mathrm\{Beta\}\(\\alpha\_\{0\},\\beta\_\{0\}\), whereα0\\alpha\_\{0\}andβ0\\beta\_\{0\}act as prior pseudo\-counts for observing samples that match or do not match the empirical majority answer, respectively\. Givenc^k\\hat\{c\}\_\{k\}successes out ofkksamples, conjugacy yields the posterior

θ∣𝒜k∼Beta​\(α0\+c^k,β0\+k−c^k\)\.\\textstyle\\theta\\mid\\mathcal\{A\}\_\{k\}\\sim\\mathrm\{Beta\}\\\!\\left\(\\alpha\_\{0\}\+\\hat\{c\}\_\{k\},\\;\\beta\_\{0\}\+k\-\\hat\{c\}\_\{k\}\\right\)\.\(2\)The posterior mean provides a prior\-smoothed estimate of the empirical agreementθ^k\\hat\{\\theta\}\_\{k\}, while the posterior uncertainty decreases as more samples are collected\.

###### Lemma 3\.1\(Posterior Contraction of Mode Mass\)\.

Assume decoding samples\{Ai\}i=1k\\\{A\_\{i\}\\\}\_\{i=1\}^\{k\}are conditionally independent\. Under the Beta priorθ∼Beta​\(α0,β0\)\\theta\\sim\\mathrm\{Beta\}\(\\alpha\_\{0\},\\beta\_\{0\}\)with fixed hyperparameters, the posterior variance satisfies

Var​\(θ∣𝒜k\)≤14​\(k\+α0\+β0\+1\)\.\\textstyle\\mathrm\{Var\}\(\\theta\\mid\\mathcal\{A\}\_\{k\}\)\\leq\\frac\{1\}\{4\(k\+\\alpha\_\{0\}\+\\beta\_\{0\}\+1\)\}\.

The proof is provided in Appendix[A](https://arxiv.org/html/2607.20481#A1)\. This contraction result is useful for our routing procedure because CARGO stops sampling based on posterior uncertainty rather than a fixed sample count\. Lemma[3\.1](https://arxiv.org/html/2607.20481#S3.Thmtheorem1)shows that each additional response increases the effective posterior sample size, causing uncertainty around the agreement estimate to shrink at rateO​\(1/k\)O\(1/k\)\. The bound depends on the total prior concentrationα0\+β0\\alpha\_\{0\}\+\\beta\_\{0\}, which controls uncertainty, while the prior meanα0/\(α0\+β0\)\\alpha\_\{0\}/\(\\alpha\_\{0\}\+\\beta\_\{0\}\)can still affect finite\-sample estimates and early routing decisions\. Thus, the lemma provides a principled basis for the credible\-interval stopping rule used by CARGO: sampling continues only until the agreement estimate is sufficiently reliable\.

Bayesian Early Stopping for Agreement Estimation\.While agreement provides a useful reliability signal, generating a large number of samples per query can increase latency\. To reduce unnecessary sampling, we adopt a Bayesian early\-stopping procedure that stops once the uncertainty of the mode\-mass estimate becomes sufficiently small\. Given the posterior distributionθ∣𝒜k\\theta\\mid\\mathcal\{A\}\_\{k\}, we compute a\(1−δ\)\(1\-\\delta\)credible interval\[Lk,Uk\]\[L\_\{k\},U\_\{k\}\]\. The algorithm continues sampling until the posterior uncertainty becomes small:Uk−Lk≤εU\_\{k\}\-L\_\{k\}\\leq\\varepsilon\.

The following result shows that this procedure yields a consistent estimate of the mode mass and terminates after a finite number of samples with probability one\.

###### Theorem 3\.2\(Consistency of Bayesian Early\-Stopped Mode\-Mass Estimation\)\.

Letθ⋆=maxa⁡P​\(a∣x\)\\theta^\{\\star\}=\\max\_\{a\}P\(a\\mid x\)denote the true mode mass and assume decoding samples\{Ai\}\\\{A\_\{i\}\\\}are conditionally independent\. Let\[Lk,Uk\]\[L\_\{k\},U\_\{k\}\]be the\(1−δ\)\(1\-\\delta\)credible interval of the Beta posteriorθ∣𝒜k\\theta\\mid\\mathcal\{A\}\_\{k\}\. Thenθ^k→θ⋆\\hat\{\\theta\}\_\{k\}\\to\\theta^\{\\star\}andUk−Lk→0U\_\{k\}\-L\_\{k\}\\to 0almost surely ask→∞k\\to\\infty\. Consequently, the stopping ruleUk−Lk≤εU\_\{k\}\-L\_\{k\}\\leq\\varepsilonterminates almost surely and returns a consistent estimate ofθ⋆\\theta^\{\\star\}\.

The proof is provided in Appendix[B](https://arxiv.org/html/2607.20481#A2)\. Intuitively, this condition ensures that the agreement level has been estimated with sufficient confidence\. As new samples are collected, the posterior uncertainty around the mode mass decreases, which narrows the credible interval and guarantees that the CARGO sampling process eventually terminates\. This stopping rule decouples sampling cost from the routing decision: sampling stops once agreement has been estimated reliably, regardless of whether the query is ultimately processed locally or offloaded to the cloud\. After termination at samplekk, the agreement estimatorθ^k\\hat\{\\theta\}\_\{k\}provides a compact summary of the local model’s response stability\.

Probabilistic Routing Policy\.Instead of using a deterministic threshold on the agreement estimate, we employ a probabilistic routing policy that maps agreement to an offloading probability\. Specifically, given agreement estimateθ^\\hat\{\\theta\}, we define the offloading probabilityp=σ​\(γ​\(λ−θ^\)\),p=\\sigma\\\!\\big\(\\gamma\(\\lambda\-\\hat\{\\theta\}\)\\big\),with sigmoid functionσ​\(z\)=1/\(1\+e−z\)\\sigma\(z\)=1/\(1\+e^\{\-z\}\)\. The routing decision is then sampled asr∼Bernoulli​\(p\)r\\sim\\mathrm\{Bernoulli\}\(p\), wherer=1r=1denotes cloud offloading andr=0r=0denotes accepting the local answer\. This formulation has several desirable properties\. First, the agreement estimate directly controls routing behavior: high agreement impliesθ^\>λ\\hat\{\\theta\}\>\\lambda, resulting in a small offloading probability, while low agreement increases the likelihood of cloud routing\. Second, the parameterλ\\lambdaacts as an interpretable*agreement pivot*: whenθ^=λ\\hat\{\\theta\}=\\lambda, the router is indifferent between local and cloud execution, sincep=0\.5p=0\.5\. Third, compared with hard thresholding, this probabilistic formulation avoids instability when many queries have similar agreement levels\. It also enables control of the global routing rate, which enables CARGO to meet varying system\-level resource constraints\.

Warmup Calibration of the Routing Parameter\.The intercept parameterλ\\lambdadetermines the overall routing behavior of the system\. Increasingλ\\lambdaraises the offloading probability for all queries, while decreasingλ\\lambdafavors local execution\. To achieve a target collaboration ratioρ\\rho, we calibrateλ\\lambdausing a short warmup phase\. A fixed batch ofBBwarmup queriesℬ\\mathcal\{B\}is sampled, and the router is repeatedly applied to this batch while updatingλ\\lambda\. Letr​\(x;λ\)∈\{0,1\}r\(x;\\lambda\)\\in\\\{0,1\\\}denote the routing decision for queryxx\. At iterationtt, we compute the empirical offloading rater¯t=1B​∑x∈ℬr​\(x;λt\)\.\\bar\{r\}\_\{t\}=\\frac\{1\}\{B\}\\sum\_\{x\\in\\mathcal\{B\}\}r\(x;\\lambda\_\{t\}\)\.The routing parameter is then updated via stochastic approximation:

λt\+1=λt−ηλ​\(r¯t−ρ\)\.\\lambda\_\{t\+1\}=\\lambda\_\{t\}\-\\eta\_\{\\lambda\}\(\\bar\{r\}\_\{t\}\-\\rho\)\.\(3\)This update adjustsλ\\lambdain the direction that reduces the gap between the observed offloading rate and the desired targetρ\\rho\. When the system offloads too frequently \(r¯t\>ρ\\bar\{r\}\_\{t\}\>\\rho\),λ\\lambdais reduced, lowering the offloading probability\. Conversely, if routing is too conservative,λ\\lambdaincreases\. AfterT0T\_\{0\}warmup iterations, the calibratedλ\\lambdais fixed for deployment queries\. This calibration procedure allows the system to satisfy a user\-specified target collaboration ratio without model retraining or task\-specific tuning\. Because the routing probability is smooth inλ\\lambda, the update converges rapidly in practice\.

## 4Experiments

Table 1:Accuracy comparison at a fixed collaboration ratioρ=0\.3\\rho=0\.3across local LLM backbones and benchmarks\. By routing based on agreement\-estimated local reliability, CARGO achieves consistently higher accuracy across model\-dataset pairs than the unsupervised baselines and supervised learned router\.ModelMethodMATH\-lightevalGSM8KSVAMPMinerva\-MATHARCSQuADQwen2\.5\-3B\-InstructRandom Offloading74\.1088\.7293\.6055\.6673\.2968\.74Self\-Confidence Router\[[32](https://arxiv.org/html/2607.20481#bib.bib32)\]80\.6489\.0494\.5059\.9374\.2872\.09CoT steps Router\[[9](https://arxiv.org/html/2607.20481#bib.bib10)\]74\.0390\.1795\.2056\.6273\.4973\.40Learned Router\[[8](https://arxiv.org/html/2607.20481#bib.bib5)\]80\.7888\.8893\.1049\.6375\.1074\.10CARGO \(Ours\)87\.2993\.8997\.4065\.8179\.3879\.19Phi\-3\-mini\-4k\-InstructRandom Offloading61\.7282\.7289\.1645\.0585\.5873\.96Self\-Confidence Router60\.5982\.8989\.8047\.4386\.7477\.57CoT steps Router66\.8283\.6592\.1046\.3286\.6276\.97Learned Router64\.9587\.1493\.5043\.7589\.9477\.72CARGO \(Ours\)71\.7589\.4993\.3047\.4390\.6485\.20Llama\-3\.2\-3B\-InstructRandom Offloading63\.6785\.8291\.1840\.7260\.9059\.17Self\-Confidence Router67\.2786\.3892\.4041\.9161\.5462\.05CoT steps Router71\.6388\.5093\.0043\.7561\.4363\.18Learned Router70\.1584\.6488\.0043\.7565\.0563\.78CARGO \(Ours\)74\.4392\.8396\.6044\.4966\.8468\.16

Datasets and LLM configurations\.We evaluate our method on a diverse set of benchmarks spanning mathematical reasoning \(MATH\-lighteval, GSM8K, SVAMP, MATH\-500, AGIEval\-Math, MinervaMath\), general question answering \(ARC, MMLU\), and knowledge\-intensive multiple\-choice reasoning \(SQuAD\)\[[13](https://arxiv.org/html/2607.20481#bib.bib21),[20](https://arxiv.org/html/2607.20481#bib.bib22),[7](https://arxiv.org/html/2607.20481#bib.bib23),[15](https://arxiv.org/html/2607.20481#bib.bib24),[33](https://arxiv.org/html/2607.20481#bib.bib25),[17](https://arxiv.org/html/2607.20481#bib.bib26),[6](https://arxiv.org/html/2607.20481#bib.bib27),[12](https://arxiv.org/html/2607.20481#bib.bib28),[14](https://arxiv.org/html/2607.20481#bib.bib29),[21](https://arxiv.org/html/2607.20481#bib.bib30)\]\. Among these, the majority of the datasets focus on mathematical reasoning with varying difficulty levels, ranging from grade\-school arithmetic and algebraic word problems to more challenging multi\-step mathematical problem solving\. In addition, ARC and MMLU provide broader evaluation over science reasoning and general multi\-domain knowledge, while SQuAD serves as a reading comprehension benchmark\.

For router training, the learned router is trained using splits of MATH\-lighteval and ARC\-Challenge\. This design allows the router to learn routing behavior from both mathematical reasoning tasks and non\-math multiple\-choice reasoning tasks\. Evaluation is conducted across the full benchmark suite to assess generalization\. For inference, we consider a range of instruction\-tuned local LLMs covering multiple parameter scales and model families, including Llama\-3\.2\-1B\-Instruct, Qwen2\.5\-3B\-Instruct, Qwen2\.5\-7B\-Instruct, Phi\-3\-mini\-4k\-instruct, and Phi\-3\-medium\-4k\-instruct\[[25](https://arxiv.org/html/2607.20481#bib.bib18),[2](https://arxiv.org/html/2607.20481#bib.bib19),[1](https://arxiv.org/html/2607.20481#bib.bib20)\]\. For cloud inference, we use DeepSeek\-R1\[[10](https://arxiv.org/html/2607.20481#bib.bib35)\]as the global model across all experiments\. These models were selected to capture cross\-family differences and within\-family scaling behavior, enabling us to study how collaboration performance varies across model capacity and backbone\. All experiments are conducted under 2 H100 GPUs\.

Baselines\.We consider four routing baselines\.*Random Offloading*routes inputs to the cloud model uniformly at random under the target collaboration ratio\.*Self\-Confidence Router*prompts the local LLM to produce a self\-evaluated confidence score\[[23](https://arxiv.org/html/2607.20481#bib.bib4)\], which is then thresholded for routing\.*CoT\-steps Router*enforces chain\-of\-thought reasoning and uses the resulting number of generated reasoning steps, as a routing signal\[[3](https://arxiv.org/html/2607.20481#bib.bib9),[9](https://arxiv.org/html/2607.20481#bib.bib10)\]\. These three baselines require no additional training and rely purely on inference\-time uncertainty proxies\. The realized collaboration ratio may deviate slightly from the target due to score ties or discretization\. To ensure fair comparison at each target ratio, we randomly flip a small number of decisions when needed so that every method matches the same prescribed collaboration ratio exactly\. We further compare against a supervised*Learned Router*\[[8](https://arxiv.org/html/2607.20481#bib.bib5)\]\. In particular, we use DeBERTa\-large\[[11](https://arxiv.org/html/2607.20481#bib.bib34)\]as a classifier\-based router to predict whether an input should be offloaded to the cloud model\. This learned baseline provides a strong reference point against which to evaluate our method\. Exact prompt templates for the step\-based, confidence\-based, and agreement\-based evaluation signals are provided in Appendix[D](https://arxiv.org/html/2607.20481#A4)\.

### 4\.1Experimental Results

Baseline Comparison on Fixed Collaboration Ratio\.Table[1](https://arxiv.org/html/2607.20481#S4.T1)reports performance at collaboration ratioρ=0\.3\\rho=0\.3across mathematical reasoning, science Q&A, and reading comprehension benchmarks\. Under this setting, CARGO achieves the strongest and most consistent results across datasets and local backbones, even outperforming the supervised learned router while remaining fully training\-free\. This shows that effective routing need not rely on a separately trained classifier\. While self\-confidence and CoT steps baselines can perform reasonably well on individual datasets, their behavior is task\-dependent and less consistent across benchmark types and model families\. By contrast, CARGO’s agreement\-based reliability signal is more stable, suggesting that the local model’s own response agreement provides a transferable indicator of when local inference can be trusted\. Additional results in Appendix[C\.3](https://arxiv.org/html/2607.20481#A3.SS3)further show that task\-aligned RLHF does not replace routing, but instead makes agreement\-based routing more effective by aligning inference\-time agreement with correctness\.

![Refer to caption](https://arxiv.org/html/2607.20481v1/images/paper_4panel_math_svamp_arc_squad.png)Figure 3:Evaluation across multiple target collaboration ratiosρ\\rhounder different local LLM sizes and task groups\. CARGO consistently improves the accuracy and cloud\-use trade\-off over training\-free baselines across datasets, with gains remaining stable from low to high collaboration regimes\.![Refer to caption](https://arxiv.org/html/2607.20481v1/images/consistency_density_mathlight.png)Figure 4:Prompt\-varied sampling on MATH\-lighteval\. For all nonzero temperatures, sampling is performed with a fixed system prompt; in contrast, temperature zero uses the proposed prompt\-varied sampling![Refer to caption](https://arxiv.org/html/2607.20481v1/images/early_exit_K2.png)Figure 5:Early\-exit evaluation of CARGO across datasets and local models\. The bars report the mean stopping round and red markers indicate the maximum sampled response in the dataset\. The results show that the proposed agreement\-estimation procedure usually stops far before the sampling budget is exhausted, while adapting its stopping depth to dataset difficulty\.Performance Across Collaboration Ratios\.Figure[3](https://arxiv.org/html/2607.20481#S4.F3)evaluates routing performance across a wide range of target collaboration ratios on representative task groups, including two math benchmarks with different difficulty profiles \(MATH\-lighteval and SVAMP\), a broader science and multiple\-choice reasoning benchmark \(ARC\-Challenge\+Easy\), and a reading comprehension benchmark \(SQuAD\)\. Across both local backbones and nearly the full range of collaboration ratios, CARGO consistently achieves the strongest overall performance among the training\-free baselines\. The advantage is most pronounced in the low\-to\-mid collaboration regime \(≈\\approx20–50%\), where cloud assistance is limited and routing quality matters most; correspondingly, CARGO achieves its largest gains in this range across several datasets\. By contrast, at the two extremes all methods naturally become more similar: when the collaboration ratio is close to zero, performance is dominated by the local model, while at very high collaboration ratios most queries are offloaded and all methods approach the same cloud usage\. All local LLM outputs use deterministic decoding, so CARGO’s variation comes from prompt variation, whileRandomvaries from query offloading\.

The rightmost average panel further shows that this advantage is consistent overall across tasks\. We also observe that the margin on ARC is smaller for all routing strategies, suggesting that the gains from agreement\-guided routing depend in part on whether the local model’s own response behavior already carries a sufficiently informative reliability signal for the target task\. When this intrinsic signal is weaker, the separation between routing strategies naturally becomes smaller\. Finally, the three\-trial variance band for CARGO is barely visible across panels, indicating that the proposed routing procedure, including Bayesian early stopping, remains highly stable across repeated runs\. We additionally evaluate routing under a token\-based cloud budget in Appendix[C\.4](https://arxiv.org/html/2607.20481#A3.SS4), showing that the same agreement\-guided principle remains effective when offloading cost is measured by total cloud token consumption rather than query fraction\.

### 4\.2Ablation Study

Effectiveness of Prompt\-varied Sampling\.Figure[4](https://arxiv.org/html/2607.20481#S4.F4)compares consistency\-quality behavior under the proposed prompt\-varied sampling scheme and standard temperature\-based sampling\. A desirable routing signal should satisfy two properties simultaneously: it should induce enough variation to make agreement informative, while still preserving the underlying reliability of the local model’s responses\. Pure temperature\-based sampling creates a clear tension between these two goals\. At lower nonzero temperatures, the sampled outputs remain highly concentrated, which limits the dynamic range of the resulting consistency values and makes reliability estimation less informative\. At higher temperatures, the distribution spreads out more, but this comes at the cost of increasingly distorted response behavior and a weaker correspondence between consistency and actual correctness\.

By contrast, prompt\-varied sampling at temperature zero produces a cleaner consistency signal\. Using semantically equivalent prompts introduces structured variation across generations without injecting token\-level randomness, so the local model remains closer to its native deterministic behavior while still exposing meaningful differences across queries\. As shown in Figure[4](https://arxiv.org/html/2607.20481#S4.F4), this leads to a sharper monotonic relationship between consistency and accuracy, while yielding a broad and useful spread of consistency values for routing\. In particular, many queries concentrate near the two extreme regimes: highly consistent cases that are easy and can be trusted locally, and low\-consistency cases that are often difficult and should be routed\. This bimodal tendency is beneficial for our Bayesian early\-stopping procedure, since decisions become clear quickly in both regimes and fewer samples are needed\. Overall, these results show that prompt variation provides a more deployment\-friendly source of diversity than temperature\-based sampling, improving both the quality of the reliability signal and the sample efficiency of agreement\-based routing\.

Sample Efficiency of Bayesian Early Stopping\.Figure[5](https://arxiv.org/html/2607.20481#S4.F5)shows that the proposed Bayesian early\-stopping rule terminates well before the maximum sampling budgetKmaxK\_\{\\max\}across all datasets and both local backbones, confirming that agreement\-based routing can be made sample\-efficient in practice\. The average stopping point also varies across datasets, which indicates that the procedure adapts to task uncertainty rather than using a fixed number of samples for every query\. This behavior is enabled by the Beta\-posterior credible interval used in our estimator, which supports fast stopping in two decisive regimes\. When sampled responses quickly collapse to the same answer, the posterior concentrates near high agreement, providing strong evidence that the local model is reliable and that no offloading is needed\. Conversely, when sampled responses remain highly inconsistent, the posterior quickly indicates persistently low agreement, making cloud routing the natural decision without requiring many additional samples\. This mechanism is particularly effective because, as also suggested by Figure[4](https://arxiv.org/html/2607.20481#S4.F4), many benchmarks contain a large fraction of queries that are either relatively easy for the local model or clearly beyond its capability, so the routing decision often becomes clear after only a few samples\. Overall, these results show that Bayesian uncertainty quantification not only provides a principled reliability estimate, but also substantially reduces the average sampling cost of agreement\-based routing while preserving its practical usefulness\.

![Refer to caption](https://arxiv.org/html/2607.20481v1/images/token_paper_2panel_math_svamp.png)Figure 6:Token\-budgeted offloading results\. CARGO can also control cloud\-token usage instead of offloading ratio\.Extending to Token\-Budgeted Offloading\.CARGO can support resource\-aware control by regulating cloud\-token usage rather than only the offloading ratio\. We definec​\(x\)=r​\(x\)​\(α​Lin​\(x\)\+β​Lout​\(x\)\),c\(x\)=r\(x\)\\big\(\\alpha L\_\{\\mathrm\{in\}\}\(x\)\+\\beta L\_\{\\mathrm\{out\}\}\(x\)\\big\),whereLin​\(x\)L\_\{\\mathrm\{in\}\}\(x\)andLout​\(x\)L\_\{\\mathrm\{out\}\}\(x\)denote input and output token lengths, and calibrateλ\\lambdaduring warmup to match a target token budgetτ\\tau\. All other components remain unchanged, including agreement estimation, Bayesian early stopping, and stochastic routing; the full procedure is given in Algorithm[2](https://arxiv.org/html/2607.20481#algorithm2)\. Fig\.[6](https://arxiv.org/html/2607.20481#S4.F6)shows that CARGO remains effective under this resource\-related control objective, demonstrating that the same agreement\-guided principle extends from ratio\-based offloading to token\-budgeted cloud usage\.

## 5Conclusion, Future Work, and Limitations

We developed CARGO, a training\-free routing framework for local\-cloud collaboration that uses prompt\-varied agreement for reliability estimation, Bayesian early stopping for sample efficiency, and lightweight calibration for controllable collaboration ratios\. Across diverse tasks, local LLM families, and pretrained or finetuned local models, CARGO consistently outperformed other training\-free baselines and in several settings surpassed supervised learned routers\. These results suggest that effective local\-cloud collaboration can emerge directly from the local model’s own response agreement, without a trained, dedicated router, while further study in Appendix[C\.2](https://arxiv.org/html/2607.20481#A3.SS2)contextualizes CARGO when local models are weak\. Future work will extend this principle to richer multimodal and resource\-aware collaboration\.

## References

- \[1\]M\. Abdin, J\. Aneja, H\. Awadalla, A\. Awadalla, A\. A\. Awan, N\. Bach, A\. Bahree, A\. Bakhtiari, H\. Behl, A\. Benhaim,et al\.\(2024\)Phi\-3 technical report: a highly capable language model locally on your phone\.arXiv preprint arXiv:2404\.14219\.Cited by:[§4](https://arxiv.org/html/2607.20481#S4.p2.1)\.
- \[2\]\(2023\)Qwen technical report\.arXiv preprint arXiv:2309\.16609\.Cited by:[§4](https://arxiv.org/html/2607.20481#S4.p2.1)\.
- \[3\]E\. Chen, W\. Fang, S\. Wang, and C\. Brinton\(2026\)Joint continual learning of local language models and cloud offloading decisions with budget constraints\.arXiv preprint arXiv:2602\.00166\.Cited by:[§1](https://arxiv.org/html/2607.20481#S1.p8.1),[§4](https://arxiv.org/html/2607.20481#S4.p3.1)\.
- \[4\]L\. Chen, M\. Zaharia, and J\. Zou\(2023\)Frugalgpt: how to use large language models while reducing cost and improving performance\.arXiv preprint arXiv:2305\.05176\.Cited by:[§1](https://arxiv.org/html/2607.20481#S1.p8.1)\.
- \[5\]X\. Chen, R\. Aksitov, U\. Alon, J\. Ren, K\. Xiao, P\. Yin, S\. Prakash, C\. Sutton, X\. Wang, and D\. Zhou\(2023\)Universal self\-consistency for large language model generation\.arXiv preprint arXiv:2311\.17311\.Cited by:[§1](https://arxiv.org/html/2607.20481#S1.p9.1)\.
- \[6\]P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord\(2018\)Think you have solved question answering? try arc, the ai2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[§4](https://arxiv.org/html/2607.20481#S4.p1.1)\.
- \[7\]K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§4](https://arxiv.org/html/2607.20481#S4.p1.1)\.
- \[8\]D\. Ding, A\. Mallick, C\. Wang, R\. Sim, S\. Mukherjee, V\. Ruhle, L\. V\. Lakshmanan, and A\. H\. Awadallah\(2024\)Hybrid llm: cost\-efficient and quality\-aware query routing\.arXiv preprint arXiv:2404\.14618\.Cited by:[Table 2](https://arxiv.org/html/2607.20481#A3.T2.5.1.5.1),[§1](https://arxiv.org/html/2607.20481#S1.p1.1),[§1](https://arxiv.org/html/2607.20481#S1.p8.1),[Table 1](https://arxiv.org/html/2607.20481#S4.T1.5.1.5.1),[§4](https://arxiv.org/html/2607.20481#S4.p3.1)\.
- \[9\]W\. Fang, D\. Han, L\. Yuan, E\. Chen, and C\. G\. BrintonBridging on\-device and cloud llms for collaborative reasoning: a unified methodology for local routing and post\-training\.Cited by:[Table 2](https://arxiv.org/html/2607.20481#A3.T2.5.1.4.1),[§1](https://arxiv.org/html/2607.20481#S1.p1.1),[§1](https://arxiv.org/html/2607.20481#S1.p8.1),[§2](https://arxiv.org/html/2607.20481#S2.p3.1),[Table 1](https://arxiv.org/html/2607.20481#S4.T1.5.1.4.1),[§4](https://arxiv.org/html/2607.20481#S4.p3.1)\.
- \[10\]D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.\(2025\)Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§4](https://arxiv.org/html/2607.20481#S4.p2.1)\.
- \[11\]P\. He, X\. Liu, J\. Gao, and W\. Chen\(2020\)Deberta: decoding\-enhanced bert with disentangled attention\.arXiv preprint arXiv:2006\.03654\.Cited by:[§4](https://arxiv.org/html/2607.20481#S4.p3.1)\.
- \[12\]D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt\(2020\)Measuring massive multitask language understanding\.arXiv preprint arXiv:2009\.03300\.Cited by:[§4](https://arxiv.org/html/2607.20481#S4.p1.1)\.
- \[13\]D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt\(2021\)Measuring mathematical problem solving with the math dataset\.arXiv preprint arXiv:2103\.03874\.Cited by:[§4](https://arxiv.org/html/2607.20481#S4.p1.1)\.
- \[14\]A\. Lewkowycz, A\. Andreassen, D\. Dohan, E\. Dyer, H\. Michalewski, V\. Ramasesh, A\. Slone, C\. Anil, I\. Schlag, T\. Gutman\-Solo,et al\.\(2022\)Solving quantitative reasoning problems with language models\.Advances in neural information processing systems35,pp\. 3843–3857\.Cited by:[§4](https://arxiv.org/html/2607.20481#S4.p1.1)\.
- \[15\]H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe\(2023\)Let’s verify step by step\.arXiv preprint arXiv:2305\.20050\.Cited by:[§4](https://arxiv.org/html/2607.20481#S4.p1.1)\.
- \[16\]Z\. Liu, C\. Zhao, F\. Iandola, C\. Lai, Y\. Tian, I\. Fedorov, Y\. Xiong, E\. Chang, Y\. Shi, R\. Krishnamoorthi,et al\.\(2024\)Mobilellm: optimizing sub\-billion parameter language models for on\-device use cases\.InForty\-first International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2607.20481#S1.p1.1)\.
- \[17\]S\. Miao, C\. Liang, and K\. Su\(2020\)A diverse corpus for evaluating and developing english math word problem solvers\.InProceedings of the 58th annual meeting of the Association for Computational Linguistics,pp\. 975–984\.Cited by:[§4](https://arxiv.org/html/2607.20481#S4.p1.1)\.
- \[18\]Y\. Oh, D\. Chung, J\. Shin, S\. Park, J\. Barthelemy, J\. Mok, and S\. Yoon\(2025\)RePIC: reinforced post\-training for personalizing multi\-modal language models\.Note:To appear; arXiv:2506\.18369Cited by:[§1](https://arxiv.org/html/2607.20481#S1.p8.1)\.
- \[19\]I\. Ong, A\. Almahairi, V\. Wu, W\. Chiang, T\. Wu, J\. E\. Gonzalez, M\. W\. Kadous, and I\. Stoica\(2024\)Routellm: learning to route llms with preference data\.arXiv preprint arXiv:2406\.18665\.Cited by:[§1](https://arxiv.org/html/2607.20481#S1.p8.1)\.
- \[20\]A\. Patel, S\. Bhattamishra, and N\. Goyal\(2021\)Are nlp models really able to solve simple math word problems?\.InProceedings of the 2021 conference of the North American chapter of the association for computational linguistics: human language technologies,pp\. 2080–2094\.Cited by:[§4](https://arxiv.org/html/2607.20481#S4.p1.1)\.
- \[21\]P\. Rajpurkar, J\. Zhang, K\. Lopyrev, and P\. Liang\(2016\)Squad: 100,000\+ questions for machine comprehension of text\.InProceedings of the 2016 conference on empirical methods in natural language processing,pp\. 2383–2392\.Cited by:[§4](https://arxiv.org/html/2607.20481#S4.p1.1)\.
- \[22\]S\. Tao, L\. Yao, H\. Ding, Y\. Xie, Q\. Cao, F\. Sun, J\. Gao, H\. Shen, and B\. Ding\(2024\)When to trust llms: aligning confidence with response quality\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 5984–5996\.Cited by:[§2](https://arxiv.org/html/2607.20481#S2.p3.1)\.
- \[23\]A\. Taubenfeld, T\. Sheffer, E\. Ofek, A\. Feder, A\. Goldstein, Z\. Gekhman, and G\. Yona\(2025\)Confidence improves self\-consistency in llms\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 20090–20111\.Cited by:[§1](https://arxiv.org/html/2607.20481#S1.p9.1),[§4](https://arxiv.org/html/2607.20481#S4.p3.1)\.
- \[24\]V\. Y\. Toh, D\. Ghosal, and S\. Poria\(2024\)Not all votes count\! programs as verifiers improve self\-consistency of language models for math reasoning\.arXiv preprint arXiv:2410\.12608\.Cited by:[§1](https://arxiv.org/html/2607.20481#S1.p9.1)\.
- \[25\]H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar,et al\.\(2023\)Llama: open and efficient foundation language models\.arXiv preprint arXiv:2302\.13971\.Cited by:[§4](https://arxiv.org/html/2607.20481#S4.p2.1)\.
- \[26\]X\. Wang, J\. Wei, D\. Schuurmans, Q\. Le, E\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou\(2022\)Self\-consistency improves chain of thought reasoning in language models\.arXiv preprint arXiv:2203\.11171\.Cited by:[§1](https://arxiv.org/html/2607.20481#S1.p9.1)\.
- \[27\]M\. Xiong, Z\. Hu, X\. Lu, Y\. Li, J\. Fu, J\. He, and B\. Hooi\(2023\)Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms\.arXiv preprint arXiv:2306\.13063\.Cited by:[§2](https://arxiv.org/html/2607.20481#S2.p3.1)\.
- \[28\]D\. Xu, W\. Yin, H\. Zhang, X\. Jin, Y\. Zhang, S\. Wei, M\. Xu, and X\. Liu\(2024\)Edgellm: fast on\-device llm inference with speculative decoding\.IEEE Transactions on Mobile Computing\.Cited by:[§1](https://arxiv.org/html/2607.20481#S1.p1.1)\.
- \[29\]J\. Xu, Z\. Li, W\. Chen, Q\. Wang, X\. Gao, Q\. Cai, and Z\. Ling\(2024\)On\-device language models: a comprehensive review\.arXiv preprint arXiv:2409\.00088\.Cited by:[§1](https://arxiv.org/html/2607.20481#S1.p1.1)\.
- \[30\]K\. Zhang, L\. Peng, C\. Wang, A\. Go, and X\. Liu\(2024\)Llm cascade with multi\-objective optimal consideration\.Cited by:[§1](https://arxiv.org/html/2607.20481#S1.p1.1)\.
- \[31\]P\. Zhang, G\. Zeng, T\. Wang, and W\. Lu\(2024\)Tinyllama: an open\-source small language model\.arXiv preprint arXiv:2401\.02385\.Cited by:[§1](https://arxiv.org/html/2607.20481#S1.p1.1)\.
- \[32\]X\. Zhao, H\. Zhang, X\. Pan, W\. Yao, D\. Yu, T\. Wu, and J\. Chen\(2024\)Fact\-and\-reflection \(far\) improves confidence calibration of large language models\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 8702–8718\.Cited by:[Table 2](https://arxiv.org/html/2607.20481#A3.T2.5.1.3.1),[§2](https://arxiv.org/html/2607.20481#S2.p3.1),[Table 1](https://arxiv.org/html/2607.20481#S4.T1.5.1.3.1)\.
- \[33\]W\. Zhong, R\. Cui, Y\. Guo, Y\. Liang, S\. Lu, Y\. Wang, A\. Saied, W\. Chen, and N\. Duan\(2024\)Agieval: a human\-centric benchmark for evaluating foundation models\.InFindings of the association for computational linguistics: NAACL 2024,pp\. 2299–2314\.Cited by:[§4](https://arxiv.org/html/2607.20481#S4.p1.1)\.
- \[34\]Y\. Zhu, M\. Zhu, N\. Liu, Z\. Xu, and Y\. Peng\(2024\)Llava\-phi: efficient multi\-modal assistant with small language model\.InProceedings of the 1st International Workshop on Efficient Multimedia Computing under Limited,pp\. 18–22\.Cited by:[§1](https://arxiv.org/html/2607.20481#S1.p1.1)\.

Appendix

## Appendix AProof for Lemma[3\.1](https://arxiv.org/html/2607.20481#S3.Thmtheorem1)

###### Proof\.

Recall that afterkksamples the count of the most frequent answer iscmax\(k\)c\_\{\\max\}^\{\(k\)\}, and we place a Beta priorθ∼Beta​\(α0,β0\)\\theta\\sim\\mathrm\{Beta\}\(\\alpha\_\{0\},\\beta\_\{0\}\)on the reliability parameterθ\\theta\. Under the Bernoulli likelihood induced by the event that a sampled answer matches the majority answer, the posterior distribution can be viewed as:

θ∣𝒜k∼Beta​\(α0\+cmax\(k\),β0\+k−cmax\(k\)\)\.\\theta\\mid\\mathcal\{A\}\_\{k\}\\sim\\mathrm\{Beta\}\\\!\\left\(\\alpha\_\{0\}\+c\_\{\\max\}^\{\(k\)\},\\;\\beta\_\{0\}\+k\-c\_\{\\max\}^\{\(k\)\}\\right\)\.
For a Beta distributionBeta​\(a,b\)\\mathrm\{Beta\}\(a,b\), the variance is

Var​\(θ\)=a​b\(a\+b\)2​\(a\+b\+1\)\.\\mathrm\{Var\}\(\\theta\)=\\frac\{ab\}\{\(a\+b\)^\{2\}\(a\+b\+1\)\}\.
Substituting

a=α0\+cmax\(k\),b=β0\+k−cmax\(k\),a=\\alpha\_\{0\}\+c\_\{\\max\}^\{\(k\)\},\\qquad b=\\beta\_\{0\}\+k\-c\_\{\\max\}^\{\(k\)\},we obtain

Var​\(θ∣𝒜k\)=\(α0\+cmax\(k\)\)​\(β0\+k−cmax\(k\)\)\(k\+α0\+β0\)2​\(k\+α0\+β0\+1\)\.\\mathrm\{Var\}\(\\theta\\mid\\mathcal\{A\}\_\{k\}\)=\\frac\{\(\\alpha\_\{0\}\+c\_\{\\max\}^\{\(k\)\}\)\(\\beta\_\{0\}\+k\-c\_\{\\max\}^\{\(k\)\}\)\}\{\(k\+\\alpha\_\{0\}\+\\beta\_\{0\}\)^\{2\}\(k\+\\alpha\_\{0\}\+\\beta\_\{0\}\+1\)\}\.
Since0≤cmax\(k\)≤k0\\leq c\_\{\\max\}^\{\(k\)\}\\leq k, both posterior parameters satisfy

a=α0\+cmax\(k\)=O​\(k\),b=β0\+k−cmax\(k\)=O​\(k\)\.a=\\alpha\_\{0\}\+c\_\{\\max\}^\{\(k\)\}=O\(k\),\\qquad b=\\beta\_\{0\}\+k\-c\_\{\\max\}^\{\(k\)\}=O\(k\)\.Therefore the numerator satisfies

Meanwhile,

a\+b=k\+α0\+β0=O​\(k\),a\+b=k\+\\alpha\_\{0\}\+\\beta\_\{0\}=O\(k\),which implies

\(a\+b\)2​\(a\+b\+1\)=O​\(k3\)\.\(a\+b\)^\{2\}\(a\+b\+1\)=O\(k^\{3\}\)\.
Combining these scalings yields

Var​\(θ∣𝒜k\)=O​\(k2k3\)=O​\(1k\)\.\\mathrm\{Var\}\(\\theta\\mid\\mathcal\{A\}\_\{k\}\)=O\\\!\\left\(\\frac\{k^\{2\}\}\{k^\{3\}\}\\right\)=O\\\!\\left\(\\frac\{1\}\{k\}\\right\)\.∎

## Appendix BProof for Theorem[3\.2](https://arxiv.org/html/2607.20481#S3.Thmtheorem2)

###### Proof\.

Leta⋆=arg⁡maxa⁡P​\(a∣x\)a^\{\\star\}=\\arg\\max\_\{a\}P\(a\\mid x\)denote the true most probable answer under the sampling distribution, and define the corresponding probability estimate

θ⋆=P​\(A=a⋆∣x\)\.\\theta^\{\\star\}=P\(A=a^\{\\star\}\\mid x\)\.
For each answeraa, let

p^k​\(a\)=1k​∑i=1k𝟏​\{Ai=a\}\\hat\{p\}\_\{k\}\(a\)=\\frac\{1\}\{k\}\\sum\_\{i=1\}^\{k\}\\mathbf\{1\}\\\{A\_\{i\}=a\\\}denote the empirical frequency\. By the strong law of large numbers,

p^k​\(a\)→P​\(a∣x\)almost surely for every​a\.\\hat\{p\}\_\{k\}\(a\)\\rightarrow P\(a\\mid x\)\\quad\\text\{almost surely for every \}a\.
Sincea⋆a^\{\\star\}is the unique maximizer ofP​\(a∣x\)P\(a\\mid x\), there existsΔ\>0\\Delta\>0such that

P​\(a⋆∣x\)−P​\(a∣x\)≥Δfor all​a≠a⋆\.P\(a^\{\\star\}\\mid x\)\-P\(a\\mid x\)\\geq\\Delta\\quad\\text\{for all \}a\\neq a^\{\\star\}\.Consequently, with probability one, there exists a finiteKKsuch that for allk≥Kk\\geq K,

p^k​\(a⋆\)\>p^k​\(a\)for all​a≠a⋆\.\\hat\{p\}\_\{k\}\(a^\{\\star\}\)\>\\hat\{p\}\_\{k\}\(a\)\\quad\\text\{for all \}a\\neq a^\{\\star\}\.Thus the empirical majority answer satisfiesa^k=a⋆\\hat\{a\}\_\{k\}=a^\{\\star\}eventually almost surely, and therefore

θ^k=cmax\(k\)k=p^k​\(a^k\)→P​\(a⋆∣x\)=θ⋆almost surely\.\\hat\{\\theta\}\_\{k\}=\\frac\{c\_\{\\max\}^\{\(k\)\}\}\{k\}=\\hat\{p\}\_\{k\}\(\\hat\{a\}\_\{k\}\)\\rightarrow P\(a^\{\\star\}\\mid x\)=\\theta^\{\\star\}\\quad\\text\{almost surely\}\.
Next, by Lemma[3\.1](https://arxiv.org/html/2607.20481#S3.Thmtheorem1), the posterior variance satisfies

Var​\(θ∣𝒜k\)=O​\(1/k\),\\mathrm\{Var\}\(\\theta\\mid\\mathcal\{A\}\_\{k\}\)=O\(1/k\),implying that posterior uncertainty vanishes ask→∞k\\rightarrow\\infty\. Hence any fixed\-level credible interval\[Lk,Uk\]\[L\_\{k\},U\_\{k\}\]shrinks in width to zero almost surely:

Uk−Lk→0\.U\_\{k\}\-L\_\{k\}\\rightarrow 0\.
Therefore, for anyε\>0\\varepsilon\>0, there exists almost surely a finiteKεK\_\{\\varepsilon\}such thatUk−Lk≤εU\_\{k\}\-L\_\{k\}\\leq\\varepsilonfor allk≥Kεk\\geq K\_\{\\varepsilon\}\. This implies that the stopping rule terminates after a finite number of samples and the returned estimator converges to the true probability estimate\. ∎

## Appendix CAdditional Experiments

### C\.1Fixed Collaboration Ratio on more datasets

Table[2](https://arxiv.org/html/2607.20481#A3.T2)reports additional results on MATH\-500 and AGI\-Eval Math under the same fixed collaboration ratioρ=0\.3\\rho=0\.3\. These two benchmarks provide a complementary evaluation of mathematical reasoning beyond the main\-table results\. Across all three local backbones, CARGO consistently improves over random offloading, self\-confidence routing, CoT\-step routing, and the supervised learned router\. This shows that the agreement\-estimated reliability signal remains effective on challenging math datasets, even when the router is not trained specifically for each benchmark or target operating point\.

These results further highlight the limitation of fixed routing heuristics\. Self\-confidence and CoT\-step routing can be competitive in some settings, but their performance varies across models and datasets\. The learned router also does not consistently transfer to these additional benchmarks, despite using supervised training\. In contrast, CARGO adapts at inference time through local response agreement and therefore remains robust across different reasoning benchmarks and local model families\.

Table 2:Additional accuracy comparison at a fixed collaboration ratioρ=0\.3\\rho=0\.3on MATH\-500 and AGI\-Eval Math\.ModelMethodMATH\-500AGI\-EvalMathQwen2\.5\-3B\-InstructRandom Offloading70\.8068\.90Self\-Confidence Router\[[32](https://arxiv.org/html/2607.20481#bib.bib32)\]77\.2074\.50CoT steps Router\[[9](https://arxiv.org/html/2607.20481#bib.bib10)\]71\.0068\.70Learned Router\[[8](https://arxiv.org/html/2607.20481#bib.bib5)\]73\.0068\.50CARGO \(Ours\)82\.4080\.60Phi\-3\-mini\-4k\-InstructRandom Offloading59\.1156\.55Self\-Confidence Router59\.3259\.46CoT steps Router64\.1361\.66Learned Router61\.5259\.66CARGO \(Ours\)67\.1365\.57Llama\-3\.2\-3B\-InstructRandom Offloading60\.5660\.27Self\-Confidence Router64\.2064\.70CoT steps Router68\.0068\.00Learned Router65\.8064\.50CARGO \(Ours\)68\.8069\.10

### C\.2Performance of Training\-Free Signals on smaller LLMs

![Refer to caption](https://arxiv.org/html/2607.20481v1/images/1Bpaper_4panel_math_svamp_arc_squad.png)Figure 7:Performance on 1B\-sized LLMs\.Figure[7](https://arxiv.org/html/2607.20481#A3.F7)illustrates an important limitation of training\-free collaboration signals\. When the local model is too small to possess meaningful reasoning ability, these signals become much less informative for routing\. In both the 1B model \(Llama\-3\.2\-1B\-Instruct\) and the 1\.5B model \(Qwen2\.5\-1\.5B\-Instruct\), the performance gaps between agreement, confidence, step\-based heuristics, and even random routing shrink substantially on several datasets, and on some tasks the curves are nearly indistinguishable\.

This behavior is expected\. Training\-free signals can only exploit reliability structure that already exists in the local model; they cannot create reasoning capability by themselves\. When the local model is too weak, repeated samples may simply reproduce the same incorrect answer or unstable shallow reasoning, so agreement or self\-reported confidence no longer correlates well with correctness\. As a result, routing based on such signals provides limited benefit\.

Therefore, our method is most effective when the local model is already capable of solving a nontrivial portion of inputs, so that inference\-time variability meaningfully reflects difficulty\. Figure[7](https://arxiv.org/html/2607.20481#A3.F7)thus highlights a boundary of applicability: below a certain capability threshold, training\-free routing becomes ineffective, and stronger local models or trained collaboration mechanisms may be required\.

### C\.3Evaluation Signal Performance after RLHF finetuning

![Refer to caption](https://arxiv.org/html/2607.20481v1/images/pretrainedpaper_4panel_math_svamp_arc_squad.png)Figure 8:Performance after RLHF\-finetuning local LLMs on specific tasks\. It suggests that the main role of finetuning is not to replace routing, but to make routing possible in a more meaningful way\. Once the local model is adapted to the task family, agreement\-based signals become much better aligned with correctness, leading to clearer gains over random, confidence\-based, and step\-based routing\.Figure[8](https://arxiv.org/html/2607.20481#A3.F8)shows that task\-aligned finetuning makes training\-free routing more effective, rather than replacing the need for routing\. Compared with the pretrained local models, the finetuned models achieve higher absolute accuracy and exhibit a clearer separation between routing strategies, with our method remaining strongest across most collaboration ratios\. This suggests that finetuning moves the local model into a more capable regime where inference\-time agreement becomes better aligned with correctness\.

The gains are also consistent with the finetuning domain\. Finetuning on MATH\-Lighteval yields the largest improvement on MATH\-Lighteval and also transfers to SVAMP, while finetuning on ARC gives the most direct benefit on ARC and a weaker but still visible improvement on SQuAD\. Overall, these results support the view that training\-free signals are most useful once the local model has sufficient task\-relevant capability, making finetuning and inference\-time routing complementary components of effective local\-cloud collaboration\.

Input:local modelMLM\_\{L\}; cloud modelMCM\_\{C\}; prompt set𝒮=\{si\}i=1M\\mathcal\{S\}=\\\{s\_\{i\}\\\}\_\{i=1\}^\{M\}; temperatureTT; credible level1−δ1\-\\delta; max samplesKmaxK\_\{\\max\}; warmup iterationsT0T\_\{0\}; warmup batch sizeBB; target average token budgetτ\\tau; step sizeηλ\\eta\_\{\\lambda\}; Beta prior\(α0,β0\)\(\\alpha\_\{0\},\\beta\_\{0\}\); width thresholdε\\varepsilon; logistic slopeγ\\gamma; token weightsα,β\\alpha,\\beta

Output:for each query

xx: final answer

y​\(x\)y\(x\), routing indicator

r​\(x\)∈\{0,1\}r\(x\)\\in\\\{0,1\\\}, and token cost

c​\(x\)c\(x\)
1

2Warmup \(token\-budget control\):initialize intercept

λ∈ℝ\\lambda\\in\\mathbb\{R\}
3sample a fixed warmup batch

ℬ=\{x\(i\)\}i=1B\\mathcal\{B\}=\\\{x^\{\(i\)\}\\\}\_\{i=1\}^\{B\}
4for*t←1t\\leftarrow 1toT0T\_\{0\}*do

5foreach*x∈ℬx\\in\\mathcal\{B\}*do

6

\(y​\(x\),r​\(x\),c​\(x\)\)←RouteQueryTok​\(x,λ\)\(y\(x\),r\(x\),c\(x\)\)\\leftarrow\\textsc\{RouteQueryTok\}\(x,\\lambda\)
7

8

c¯t←1B​∑x∈ℬc​\(x\)\\bar\{c\}\_\{t\}\\leftarrow\\frac\{1\}\{B\}\\sum\_\{x\\in\\mathcal\{B\}\}c\(x\)
λ←λ−ηλ​\(c¯t−τ\)\\lambda\\leftarrow\\lambda\-\\eta\_\{\\lambda\}\(\\bar\{c\}\_\{t\}\-\\tau\)
//drivec¯t→τ\\bar\{c\}\_\{t\}\\rightarrow\\tau

9

10Deploy:fix

λ\\lambdafor subsequent queries\.

11

12Function*RouteQueryTok\(*x,λx,\\lambda*\)*:

13initialize multiset

𝒜←∅\\mathcal\{A\}\\leftarrow\\emptysetand counts

c​\(⋅\)←0c\(\\cdot\)\\leftarrow 0
14for*k←1k\\leftarrow 1toKmaxK\_\{\\max\}*do

15sample prompt template

sk∼Unif​\(𝒮\)s\_\{k\}\\sim\\mathrm\{Unif\}\(\\mathcal\{S\}\), sample response

Ak∼ML\(⋅∣x,sk;T\)A\_\{k\}\\sim M\_\{L\}\(\\cdot\\mid x,s\_\{k\};T\)
16update counts:

𝒜k←𝒜k−1∪\{Ak\}\\mathcal\{A\}\_\{k\}\\leftarrow\\mathcal\{A\}\_\{k\-1\}\\cup\\\{A\_\{k\}\\\},

c​\(Ak\)←c​\(Ak\)\+1c\(A\_\{k\}\)\\leftarrow c\(A\_\{k\}\)\+1
17update empirical agreement

\(a^k,c^k\)←\(arg⁡maxa⁡cka,maxa⁡cka\)\(\\hat\{a\}\_\{k\},\\hat\{c\}\_\{k\}\)\\leftarrow\(\\arg\\max\_\{a\}c\_\{k\}^\{a\},\\;\\max\_\{a\}c\_\{k\}^\{a\}\)
18compute posterior

θ∣𝒜k∼Beta​\(α0\+c^k,β0\+k−c^k\)\\theta\\mid\\mathcal\{A\}\_\{k\}\\sim\\mathrm\{Beta\}\(\\alpha\_\{0\}\+\\hat\{c\}\_\{k\},\\beta\_\{0\}\+k\-\\hat\{c\}\_\{k\}\)and credible interval

\[Lk,Uk\]\[L\_\{k\},U\_\{k\}\]
19if*Uk−Lk≤εU\_\{k\}\-L\_\{k\}\\leq\\varepsilon*then

break

//confidence\-based early exit

20

21

22

23compute agreement estimate

θ^←c^k/k\\hat\{\\theta\}\\leftarrow\\hat\{c\}\_\{k\}/kand offload probability

p←σ​\(γ​\(λ−θ^\)\)p\\leftarrow\\sigma\\\!\\big\(\\gamma\(\\lambda\-\\hat\{\\theta\}\)\\big\)
24sample

r∼Bernoulli​\(p\)r\\sim\\mathrm\{Bernoulli\}\(p\)
25

26if*r=0r=0*then

27

c​\(x\)←0c\(x\)\\leftarrow 0
return

\(a^k,0,c​\(x\)\)\(\\hat\{a\}\_\{k\},0,c\(x\)\)
//accept local answer, no cloud token cost

28

29else

30query cloud

y∼MC\(⋅∣x\)y\\sim M\_\{C\}\(\\cdot\\mid x\)
31measure cloud input/output lengths

Lin​\(x\)L\_\{\\mathrm\{in\}\}\(x\)and

Lout​\(x\)L\_\{\\mathrm\{out\}\}\(x\)
32

c​\(x\)←α​Lin​\(x\)\+β​Lout​\(x\)c\(x\)\\leftarrow\\alpha L\_\{\\mathrm\{in\}\}\(x\)\+\\beta L\_\{\\mathrm\{out\}\}\(x\)
return

\(y,1,c​\(x\)\)\(y,1,c\(x\)\)
//offload to cloud with token cost

33

34

Algorithm 2CARGO\-Token: Token\-Budgeted Collaboration\-Adaptive Routing
### C\.4Cloud Offloading based on Total Token Consumption

We also consider a token\-budgeted variant of CARGO in Algorithm[2](https://arxiv.org/html/2607.20481#algorithm2)\. Rather than controlling the average offloading ratio, this version controls the average cloud token costc​\(x\)=r​\(x\)​\(α​Lin​\(x\)\+β​Lout​\(x\)\),c\(x\)=r\(x\)\\big\(\\alpha L\_\{\\mathrm\{in\}\}\(x\)\+\\beta L\_\{\\mathrm\{out\}\}\(x\)\\big\),whereLin​\(x\)L\_\{\\mathrm\{in\}\}\(x\)andLout​\(x\)L\_\{\\mathrm\{out\}\}\(x\)are the cloud input and output token lengths\. The warmup update ofλ\\lambdais modified accordingly to match a target average token budgetτ\\tau\. All other components remain the same, including agreement\-based reliability estimation, posterior credible intervals, and stochastic offloading based on the estimated local reliability\. The empirical results are shown in Figure[9](https://arxiv.org/html/2607.20481#A3.F9), where the x\-axis represents the total allowed cloud\-token budget instead of an offloading ratio\.

![Refer to caption](https://arxiv.org/html/2607.20481v1/images/token_paper_4panel_math_svamp_arc_squad.png)Figure 9:Offloading evaluation under token\-budgeted routing\. The x\-axis shows the total allowed cloud\-token budget\.

## Appendix DPrompt Design for Evaluation Signals

We use prompt\-based evaluation signals for several training\-free baselines\. These prompts are designed to induce structured reasoning, standardized final\-answer formatting, or explicit self\-reported confidence, depending on the signal being evaluated\. Across all settings, we standardize the final answer format using\\b​o​x​e​d​\{⋅\}\\backslash boxed\\\{\\cdot\\\}to simplify answer extraction and comparison\.

### D\.1Baseline evaluation signals\.

We consider two prompt templates for the baseline signals used in our experiments: \(i\) a step\-based reasoning prompt, and \(ii\) a confidence\-reporting prompt\. The step\-based prompt is used to elicit explicit multi\-step reasoning before producing the final boxed answer\. The confidence prompt uses the same reasoning requirement, but additionally asks the model to output a scalar confidence score in\[0,1\]\[0,1\]on the final line\. These prompts are shown in Table[3](https://arxiv.org/html/2607.20481#A4.T3)\.

Table 3:System prompts used for baseline evaluation signals\.Prompt typeSystem promptStepYou are a helpful reasoning assistant for both questioning and math problems\. When a question is posed, try to answer using step\-by\-step reasoning\. Reason step by step, using format: Step 1: \.\.\., Step 2: \.\.\., Step 3: \.\.\., etc\. You must prefix each reasoning line with ‘Step k:’ exactly \(e\.g\., ‘Step 1:’, ‘Step 2:’\)\. Do not use other numbering formats such as ‘1\.’, ‘\(1\)’, or ‘\- Step 1’\. If the problem appears difficult or hard to solve, elaborate your reasoning with more detailed steps\. Otherwise, use fewer steps as appropriate\. Always box the final answer using\\\\backslashboxed\{\}, e\.g\., Answer:\\\\backslashboxed\{42\} \(math\) or Answer:\\\\backslashboxed\{choice\} \(general QA\)\.ConfidenceYou are a careful reasoning assistant\. You must follow the required output format strictly\. 1\. Solve the problem step by step\. 2\. Box the answer using\\\\backslashboxed\{\}\. 3\. Always box the answer using\\\\backslashboxed\{\}, e\.g\., Answer:\\\\backslashboxed\{42\} \(math\) or Answer:\\\\backslashboxed\{choice\} \(general QA\)\. 4\. On a new line, output your confidence as a decimal between 0 and 1\. The confidence must follow EXACTLY this format: Confidence: p, where p is a decimal number between 0 and 1 \(example: 0\.73\)\. Do not output percentages\. Do not output text after the confidence line\. The confidence line must be the final line of your output\.
### D\.2Dataset\-specific user prompts\.

While the system prompt determines the response format, the user prompt is lightly adapted to match the structure of each benchmark\. Table[4](https://arxiv.org/html/2607.20481#A4.T4)summarizes the user prompt templates used across dataset families\.

Table 4:User prompt templates for different benchmark types\.Dataset typeUser prompt templateMath benchmarks\{question\}\. Let’s think step by step\.ARC, MMLU\{question\}\. Let’s think step by step\. Possible answers: \{choices\}SQuAD\[context\]\{context\} \[question\]\{question\}\. Let’s think step by step\.
### D\.3Consistency signal prompts\.

For the consistency\-based signal, we do not rely on a single fixed prompt\. Instead, we use a set of semantically similar but stylistically distinct system prompts that all enforce the same output structure, namely step\-by\-step reasoning followed by a boxed final answer\. This prompt variation is used to induce diverse reasoning trajectories while preserving comparable answer formatting, so that cross\-sample agreement can be measured reliably, as shown in Table[5](https://arxiv.org/html/2607.20481#A4.T5)and[6](https://arxiv.org/html/2607.20481#A4.T6)\.

Concretely, all consistency prompts share the following properties: \(i\) they require reasoning in the formatStep 1:,Step 2:, …; \(ii\) they prohibit alternative numbering styles; and \(iii\) they require the final answer to appear in\\b​o​x​e​d​\{⋅\}\\backslash boxed\\\{\\cdot\\\}\. The variants differ only in the style of reasoning they encourage, such as concise reasoning, detailed derivation, plan\-then\-execute reasoning, explicit verification, example\-driven reasoning, or structure\-first reasoning\.

Table 5:System prompt variants used for consistency\-based sampling \(Part I\)\.VariantSystem promptOriginalYou are a helpful reasoning assistant for both questioning and math problems\. When a question is posed, try to answer using step\-by\-step reasoning\. Reason step by step, using format: Step 1: \.\.\., Step 2: \.\.\., Step 3: \.\.\., etc\. You must prefix each reasoning line with ‘Step k:’ exactly \(e\.g\., ‘Step 1:’, ‘Step 2:’\)\. Do not use other numbering formats such as ‘1\.’, ‘\(1\)’, or ‘\- Step 1’\. If the problem appears difficult or hard to solve, elaborate your reasoning with more detailed steps\. Otherwise, use fewer steps as appropriate\. Always box the final answer using\\\\backslashboxed\{\}, e\.g\., Answer:\\\\backslashboxed\{42\} or Answer:\\\\backslashboxed\{choice\}\.ConciseYou are a helpful reasoning assistant for both questioning and math problems\. When a question is posed, answer with a small number of high\-signal steps\. Reason step by step, using format: Step 1: \.\.\., Step 2: \.\.\., etc\. You must prefix each reasoning line with ‘Step k:’ exactly \(e\.g\., ‘Step 1:’\)\. Do not use other numbering formats such as ‘1\.’, ‘\(1\)’, or ‘\- Step 1’\. Keep steps minimal and avoid unnecessary elaboration\. Always box the final answer using\\\\backslashboxed\{\}, e\.g\., Answer:\\\\backslashboxed\{42\} or Answer:\\\\backslashboxed\{choice\}\.DetailedYou are a helpful reasoning assistant for both questioning and math problems\. When a question is posed, answer using detailed step\-by\-step reasoning with careful derivations\. Reason step by step, using format: Step 1: \.\.\., Step 2: \.\.\., etc\. You must prefix each reasoning line with ‘Step k:’ exactly \(e\.g\., ‘Step 1:’\)\. Do not use other numbering formats such as ‘1\.’, ‘\(1\)’, or ‘\- Step 1’\. If the problem is difficult, break it into smaller substeps and justify each transition\. Always box the final answer using\\\\backslashboxed\{\}, e\.g\., Answer:\\\\backslashboxed\{42\} or Answer:\\\\backslashboxed\{choice\}\.Plan\-ExecuteYou are a helpful reasoning assistant for both questioning and math problems\. When a question is posed, first outline a short plan, then execute it\. Reason step by step, using format: Step 1: \.\.\., Step 2: \.\.\., etc\. You must prefix each reasoning line with ‘Step k:’ exactly \(e\.g\., ‘Step 1:’\)\. Do not use other numbering formats such as ‘1\.’, ‘\(1\)’, or ‘\- Step 1’\. Use Step 1 as a plan, then proceed with computations in later steps\. Always box the final answer using\\\\backslashboxed\{\}, e\.g\., Answer:\\\\backslashboxed\{42\} or Answer:\\\\backslashboxed\{choice\}\.VerifyYou are a helpful reasoning assistant for both questioning and math problems\. When a question is posed, solve it, then verify the result with an independent check\. Reason step by step, using format: Step 1: \.\.\., Step 2: \.\.\., etc\. You must prefix each reasoning line with ‘Step k:’ exactly \(e\.g\., ‘Step 1:’\)\. Do not use other numbering formats such as ‘1\.’, ‘\(1\)’, or ‘\- Step 1’\. Include one step near the end explicitly labeled as a verification step\. Always box the final answer using\\\\backslashboxed\{\}, e\.g\., Answer:\\\\backslashboxed\{42\} or Answer:\\\\backslashboxed\{choice\}\.Table 6:System prompt variants used for consistency\-based sampling \(Part II\)\.VariantSystem promptEdge\-CheckYou are a helpful reasoning assistant for both questioning and math problems\. When a question is posed, solve it and then test the solution on at least one edge case or sanity check\. Reason step by step, using format: Step 1: \.\.\., Step 2: \.\.\., etc\. You must prefix each reasoning line with ‘Step k:’ exactly \(e\.g\., ‘Step 1:’\)\. Do not use other numbering formats such as ‘1\.’, ‘\(1\)’, or ‘\- Step 1’\. Include a final step that checks an edge case, unit consistency, or a quick sanity test\. Always box the final answer using\\\\backslashboxed\{\}, e\.g\., Answer:\\\\backslashboxed\{42\} or Answer:\\\\backslashboxed\{choice\}\.AlgebraicYou are a helpful reasoning assistant for both questioning and math problems\. When a question is posed, prefer an algebraic, symbolic derivation rather than intuition\. Reason step by step, using format: Step 1: \.\.\., Step 2: \.\.\., etc\. You must prefix each reasoning line with ‘Step k:’ exactly \(e\.g\., ‘Step 1:’\)\. Do not use other numbering formats such as ‘1\.’, ‘\(1\)’, or ‘\- Step 1’\. Favor explicit equations and transformations\. Always box the final answer using\\\\backslashboxed\{\}, e\.g\., Answer:\\\\backslashboxed\{42\} or Answer:\\\\backslashboxed\{choice\}\.Example\-DrivenYou are a helpful reasoning assistant for both questioning and math problems\. When a question is posed, use small examples to discover the pattern before generalizing\. Reason step by step, using format: Step 1: \.\.\., Step 2: \.\.\., etc\. You must prefix each reasoning line with ‘Step k:’ exactly \(e\.g\., ‘Step 1:’\)\. Do not use other numbering formats such as ‘1\.’, ‘\(1\)’, or ‘\- Step 1’\. Start with one or two concrete examples, then generalize\. Always box the final answer using\\\\backslashboxed\{\}, e\.g\., Answer:\\\\backslashboxed\{42\} or Answer:\\\\backslashboxed\{choice\}\.Structure\-FirstYou are a helpful reasoning assistant for both questioning and math problems\. When a question is posed, look for invariants, symmetries, or conserved quantities first\. Reason step by step, using format: Step 1: \.\.\., Step 2: \.\.\., etc\. You must prefix each reasoning line with ‘Step k:’ exactly \(e\.g\., ‘Step 1:’\)\. Do not use other numbering formats such as ‘1\.’, ‘\(1\)’, or ‘\- Step 1’\. Prioritize structural insights before computation\. Always box the final answer using\\\\backslashboxed\{\}, e\.g\., Answer:\\\\backslashboxed\{42\} or Answer:\\\\backslashboxed\{choice\}\.Strict\-FormatYou are a helpful reasoning assistant for both questioning and math problems\. When a question is posed, you must follow the required output format exactly\. Reason step by step, using format: Step 1: \.\.\., Step 2: \.\.\., etc\. You must prefix each reasoning line with ‘Step k:’ exactly \(e\.g\., ‘Step 1:’\)\. Do not use other numbering formats such as ‘1\.’, ‘\(1\)’, or ‘\- Step 1’\. Do not include any text outside the Step lines and the final boxed answer line\. Always box the final answer using\\\\backslashboxed\{\}, e\.g\., Answer:\\\\backslashboxed\{42\} or Answer:\\\\backslashboxed\{choice\}\.Final\-LastYou are a helpful reasoning assistant for both questioning and math problems\. When a question is posed, provide reasoning steps first and put the final answer only at the end\. Reason step by step, using format: Step 1: \.\.\., Step 2: \.\.\., etc\. You must prefix each reasoning line with ‘Step k:’ exactly \(e\.g\., ‘Step 1:’\)\. Do not use other numbering formats such as ‘1\.’, ‘\(1\)’, or ‘\- Step 1’\. The last line must be exactly: Answer:\\\\backslashboxed\{\.\.\.\}\.In all consistency experiments, the user prompt is kept fixed within each dataset, and only the system prompt is varied across samples\.