Not All Tokens Are Equal: Inflation-Aware Routing for Agentic LLM Systems

# Not All Tokens Are Equal: Inflation-Aware Routing for Agentic LLM Systems
Source: [https://arxiv.org/html/2608.13571](https://arxiv.org/html/2608.13571)
11institutetext:Stony Brook University11email:\{heming\.fu, shan\.x\.lin\}@stonybrook\.edu22institutetext:Wuhan University22email:\{xieq@whu\.edu\.cn\}33institutetext:Shanghai Jiao Tong University33email:\{gjxiong@sjtu\.edu\.cn\}###### Abstract

When a language model fails to answer a query on the first attempt, an agentic system retries, consuming additional tokens each time\. This retry overhead creates a gap between what a model’s per\-token price implies and what a full workflow actually costs\. We call this gap*token inflation*and define it as the ratio of true workflow cost to single\-call cost\. Systems like FrugalGPT\[[1](https://arxiv.org/html/2608.13571#bib.bib1)\]route based on the latter, which can underestimate real cost by more than2×2\\timeson difficult tasks\. We address this withInflationAgent, a four\-stage router that \(1\) measures token inflation systematically across model tiers and task types, finding inflation as high as4\.25×4\.25\\timesfor a 7B model on multi\-hop question answering; \(2\) introduces CoT Branching Entropy \(CBE\), a pre\-execution difficulty signal computed entirely from local inference, which predicts high inflation with AUROC 0\.887; and \(3\) selects models by maximizing a Semantic Exchange Rate \(SER\) that divides expected accuracy by predicted true cost, with a fresh\-escalation policy that discards failed chains before routing to a stronger model\. On GSM8K under a fixed budget,InflationAgentachieves 94\.7% accuracy versus 91\.0% for FrugalGPT while using 31% fewer tokens, and we show that forwarding a failed reasoning chain to GPT\-4o reduces its accuracy by up to 34\.8 percentage points, validating the fresh\-escalation design\.

###### keywords:

LLM routing token inflation agentic systems cost\-aware inference

## 1Introduction

Modern agentic systems built on large language models rarely issue a single call and move on\. When a model returns a wrong answer or an incomplete reasoning chain, the agent retries\[[18](https://arxiv.org/html/2608.13571#bib.bib2),[13](https://arxiv.org/html/2608.13571#bib.bib3)\]\. ReAct\-style agents re\-issue queries after tool errors; self\-consistency methods\[[15](https://arxiv.org/html/2608.13571#bib.bib4)\]sample multiple chains; code\-generation agents loop until tests pass\[[8](https://arxiv.org/html/2608.13571#bib.bib7)\]\. The cost of these retries is real but invisible to the routing decision\.

Consider a concrete example\. A user asks a pipeline to answer: “Who directed the film whose soundtrack was composed by the artist who collaborated with Michael Jackson on Thriller?” A 7B local model might attempt this five times and still fail, consuming roughly five times the tokens a cost estimator would predict\. A GPT\-4o call answers correctly on the first try\. Measured by actual workflow cost, the “cheap” option is not cheap at all\.

Existing routing systems\[[1](https://arxiv.org/html/2608.13571#bib.bib1),[14](https://arxiv.org/html/2608.13571#bib.bib12),[4](https://arxiv.org/html/2608.13571#bib.bib11)\]estimate the cost of a query as the price per token multiplied by the expected output length of a single call\. When the model fails on the first attempt, this estimate can be off by a factor of two to five\. We call this discrepancy*token inflation*: the ratio between what a workflow actually costs and what a single\-call estimate predicts\.

This paper introducesInflationAgent, a routing system that makes three contributions\.Measurement: we instrument an agentic retry harness across two reasoning tasks and three model tiers to show that inflation varies dramatically by task type, reaching4\.25×4\.25\\timesfor a 7B model on multi\-hop QA versus1\.31×1\.31\\timesfor GPT\-4o\.Prediction: we introduce CoT Branching Entropy \(CBE\), a pre\-execution difficulty signal that estimates retry likelihood from a small sample of local reasoning chains, with no API cost\.Routing: we define the Semantic Exchange Rate \(SER\) as expected accuracy divided by predicted true cost, route to the model with the highest SER, and escalate with a fresh prompt when the chosen model inflates beyond its prediction\.

Figure[1](https://arxiv.org/html/2608.13571#S1.F1)illustrates the core motivation: each point is one query, plotted by its FrugalGPT estimated token count versus its actual workflow token count\. Points above the diagonal represent the hidden cost that routing systems currently ignore\.

![Refer to caption](https://arxiv.org/html/2608.13571v1/x1.png)Figure 1:Token inflation on 200 hard GSM8K queries \(Qwen2\.5\-7B, max 5 retries\)\.Left:Each point is one query\. The x\-axis is the FrugalGPT single\-call cost estimate; the y\-axis is the actual workflow cost\. Color encodes inflation rate from green \(1×1\\times, answered correctly on first try\) to red \(5×5\\times, all retries exhausted\)\. Points above the diagonal have inflated cost\.Right:CDF of inflation rates\. The distribution is bimodal: 75% of hard queries succeed on the first attempt \(1×1\\timescost\) and 25% fail all five retries \(5×5\\timescost\), making the mean2×2\\timesand causing FrugalGPT to underestimate actual workflow cost by 110%\.
## 2Related Work

#### LLM cost routing and cascades\.

A growing body of work reduces inference cost by routing easy queries to smaller models\. FrugalGPT\[[1](https://arxiv.org/html/2608.13571#bib.bib1)\]learns a cascade policy that stops escalating once a response passes a quality threshold\. RouteLLM\[[9](https://arxiv.org/html/2608.13571#bib.bib10)\]trains a router on human preference data; Hybrid LLM\[[14](https://arxiv.org/html/2608.13571#bib.bib12)\]routes by query complexity; RouterBench\[[4](https://arxiv.org/html/2608.13571#bib.bib11)\]benchmarks routing policies across model families; LLM\-Blender\[[5](https://arxiv.org/html/2608.13571#bib.bib13)\]fuses outputs from multiple models; language model cascades\[[3](https://arxiv.org/html/2608.13571#bib.bib18)\]query increasingly capable models until one produces a confident answer\. All of these systems estimate routing cost as the price per token of a single call, which is accurate only when the pipeline executes the chosen model once\. In agentic settings with retries, the single\-call estimate systematically understates the true cost\. Our work models this retry overhead explicitly and incorporates it into the routing objective\.

#### Uncertainty estimation for language models\.

Self\-consistency\[[15](https://arxiv.org/html/2608.13571#bib.bib4)\]generates multiple completions and uses answer agreement as an implicit confidence measure\. Semantic uncertainty\[[7](https://arxiv.org/html/2608.13571#bib.bib20)\]clusters semantically equivalent answers before measuring entropy, correcting for paraphrase variation\. Kadavath et al\.\[[6](https://arxiv.org/html/2608.13571#bib.bib22)\]show that large models can produce calibrated self\-assessments when prompted appropriately\. CoT Branching Entropy \(CBE\) is related to self\-consistency but serves a different purpose: rather than aggregating answers for final output, we use the entropy of a small chain sample as a*pre\-execution*proxy for retry probability\. The key design choice is that CBE is computed locally, adding no API cost to the routing decision\.

#### Agentic systems and multi\-step inference\.

Chain\-of\-thought prompting\[[16](https://arxiv.org/html/2608.13571#bib.bib8)\]improves reasoning by eliciting intermediate steps\. ReAct\[[18](https://arxiv.org/html/2608.13571#bib.bib2)\]interleaves reasoning and tool use into inspectable agent trajectories\. Reflexion\[[13](https://arxiv.org/html/2608.13571#bib.bib3)\]adds verbal self\-critique to allow agents to revise failed plans\. AgentBench\[[8](https://arxiv.org/html/2608.13571#bib.bib7)\]evaluates LLM agents across diverse task environments\. This work establishes that retry loops are standard practice, but does not address their cost implications\. When retries are common, the expected token cost can far exceed a single call, and our work closes this gap by routing in a way that accounts for it\.

## 3Methodology

### 3\.1Problem Formulation

We consider an agentic pipeline that selects one modelmmfrom a candidate setℳ\\mathcal\{M\}and executes it with up toRRretry attempts on queryxx\. LetTk​\(m,x\)T\_\{k\}\(m,x\)denote the token count of thekk\-th call\. The true workflow cost is

Ctrue​\(m,x\)=∑k=1k∗Tk​\(m,x\),C\_\{\\text\{true\}\}\(m,x\)=\\sum\_\{k=1\}^\{k^\{\*\}\}T\_\{k\}\(m,x\),\(1\)wherek∗k^\{\*\}is the index of the first successful attempt \(k∗=Rk^\{\*\}=Rif the model never succeeds\)\. Existing routers approximate this asCdirect​\(m,x\)=T1​\(m,x\)C\_\{\\text\{direct\}\}\(m,x\)=T\_\{1\}\(m,x\), the single\-call cost\.Token inflationis the ratio

Inflation​\(m,x\)=Ctrue​\(m,x\)/Cdirect​\(m,x\)\.\\text\{Inflation\}\(m,x\)=C\_\{\\text\{true\}\}\(m,x\)\\;/\\;C\_\{\\text\{direct\}\}\(m,x\)\.\(2\)A model that always succeeds first has inflation1×1\\times; one that exhausts allRRretries reachesR×R\\times\. Routing systems that optimize forCdirectC\_\{\\text\{direct\}\}implicitly assume this ratio is always 1, an assumption that fails badly on difficult multi\-hop queries\.

We define the utility of modelmmas its expected accuracyU​\(m\)∈\[0,1\]U\(m\)\\in\[0,1\]\. The routing objective is to maximize accuracy per unit of true cost\. We call this theSemantic Exchange Rate:

SER​\(m,x\)=U​\(m\)/Ctrue​\(m,x\),m∗=arg⁡maxm∈ℳ⁡SER​\(m,x\)\.\\mathrm\{SER\}\(m,x\)=U\(m\)\\;/\\;C\_\{\\text\{true\}\}\(m,x\),\\qquad m^\{\*\}=\\arg\\max\_\{m\\in\\mathcal\{M\}\}\\mathrm\{SER\}\(m,x\)\.\(3\)SER rewards models that are accurate*and*reliable: a high\-accuracy model with severe inflation can score lower than a slightly less accurate one that rarely retries\. BecauseCtrueC\_\{\\text\{true\}\}is not observable before execution, the system must predict it, which is the role of the two components described below\.

### 3\.2CoT Branching Entropy

We use the*consistency*of a model’s reasoning across independent samples as a proxy for query difficulty\. An easy query produces the same answer from nearly every chain; a hard query produces divergent paths and conflicting answers\. Formally, we sampleKKindependent reasoning chains from the local model, extract the final answer from each, and letpcp\_\{c\}be the empirical frequency of answercc\. TheCoT Branching Entropy\(CBE\) is the Shannon entropy\[[12](https://arxiv.org/html/2608.13571#bib.bib16)\]of this answer distribution:

CBE​\(x\)=−∑cpc​log2⁡pc\.\\mathrm\{CBE\}\(x\)=\-\\sum\_\{c\}p\_\{c\}\\log\_\{2\}p\_\{c\}\.\(4\)CBE is always computed using the local model regardless of which tier will be deployed, so it requires no API calls\. A query that confuses a small model is genuinely ambiguous or multi\-step, and this difficulty predicts retry probability across all tiers, as we validate in Section[4\.3](https://arxiv.org/html/2608.13571#S4.SS3)\.

### 3\.3Inflation Prediction and theInflationAgentPipeline

![Refer to caption](https://arxiv.org/html/2608.13571v1/figures/fig_iar_pipeline.png)Figure 2:InflationAgentfour\-stage pipeline\.Stage 1\(orange\) extracts structural features from the incoming query: token count, vocabulary diversity, and cues that suggest multi\-hop reasoning \(e\.g\., nested “who/which” constructions\)\.Stage 2\(blue\) samplesKKshort reasoning chains from the local model and computes the CoT Branching EntropyCBE​\(x\)=−∑cpc​log2⁡pc\\mathrm\{CBE\}\(x\)=\-\\sum\_\{c\}p\_\{c\}\\log\_\{2\}p\_\{c\}over the resulting answer distribution, along with the agreement scoremaxc⁡pc\\max\_\{c\}p\_\{c\}\. Both stages run on local hardware with no API cost\.Stage 3\(green\) feeds these features into a lightweight MLP that predictsInflation^​\(x\)\\widehat\{\\text\{Inflation\}\}\(x\), forms the estimated true costC^true=Cdirect⋅Inflation^\\hat\{C\}\_\{\\text\{true\}\}=C\_\{\\text\{direct\}\}\\cdot\\widehat\{\\text\{Inflation\}\}, and selects the model with the highest Semantic Exchange Ratem∗=arg⁡maxm⁡U​\(m\)/C^true​\(m,x\)m^\{\*\}=\\arg\\max\_\{m\}U\(m\)/\\hat\{C\}\_\{\\text\{true\}\}\(m,x\)\.Stage 4\(orange\) executesm∗m^\{\*\}on queryxx\. On success it returns the answer directly \(upper path\)\. If actual inflation exceeds2×2\\timesthe prediction,InflationAgentescalates to the next model tier with a*fresh*prompt \(lower path, orange arrow\): the failed reasoning chain is discarded rather than forwarded\. This design is empirically validated in Section[4\.6](https://arxiv.org/html/2608.13571#S4.SS6)\.CBE alone does not capture all sources of difficulty, soInflationAgenttrains a lightweight MLP on four features: CBE, agreement score \(maxc⁡pc\\max\_\{c\}p\_\{c\}\), average chain length, and prompt token count\. The MLP outputsInflation^​\(x\)\\widehat\{\\text\{Inflation\}\}\(x\), which is then used in Stage 3 to compute the estimated true costC^true​\(m,x\)=Cdirect​\(m,x\)⋅Inflation^​\(x\)\\hat\{C\}\_\{\\text\{true\}\}\(m,x\)=C\_\{\\text\{direct\}\}\(m,x\)\\cdot\\widehat\{\\text\{Inflation\}\}\(x\)and select the highest\-SER model\. Figure[2](https://arxiv.org/html/2608.13571#S3.F2)shows the full four\-stage pipeline\. Stages 1 and 2 run on local hardware at no API cost; Stage 3 is a single forward pass through the MLP; Stage 4 applies a fresh\-escalation policy when observed inflation exceeds2×2\\timesthe prediction, discarding the failed context before calling a stronger model\. We assign compute units of1×1\\times,3×3\\times, and52×52\\timesto the small, medium, and large tiers respectively, calibrated to reflect token pricing and typical output lengths\.

## 4Evaluation

### 4\.1Experimental Setup

#### Models and datasets\.

We evaluate three model tiers: Qwen2\.5\-7B\-Instruct\[[11](https://arxiv.org/html/2608.13571#bib.bib14)\]\(small, local RTX 4090\), GPT\-4o\-mini \(medium\), and GPT\-4o\[[10](https://arxiv.org/html/2608.13571#bib.bib15)\]\(large\), the latter two via the OpenRouter API\. We use GSM8K\[[2](https://arxiv.org/html/2608.13571#bib.bib5)\], a collection of multi\-step arithmetic problems with unique numerical answers, and HotpotQA\[[17](https://arxiv.org/html/2608.13571#bib.bib6)\], which requires chaining two or more retrieval and reasoning steps to produce a short factual answer\. For HotpotQA we provide only three candidate passages per query \(rather than the full ten\), intentionally creating a harder setting where inflation is most pronounced\. CBE sampling usesK=3K=3chains at temperatureτ=0\.7\\tau=0\.7; the first execution attempt uses greedy decoding \(τ=0\\tau=0\) and retries useτ=0\.7\\tau=0\.7to avoid deterministic repetition\.

#### Retry protocol and baselines\.

All agents follow a retry\-until\-correct protocol withR=5R=5maximum attempts\. We compareInflationAgentagainst:All\-Small\(always use the local model\);FrugalGPT\[[1](https://arxiv.org/html/2608.13571#bib.bib1)\]\(cascade by query length usingCdirectC\_\{\\text\{direct\}\}as cost estimate\); andConfidence Escalation\(escalate when CBE is high, but without an inflation model or fresh\-call semantics\)\.

### 4\.2Token Inflation Across Models and Tasks

Table[1](https://arxiv.org/html/2608.13571#S4.T1)and Figure[3](https://arxiv.org/html/2608.13571#S4.F3)summarize inflation across all tier\-dataset combinations\. On GSM8K, all three tiers inflate modestly \(1\.31–1\.42×\\times\), reflecting that arithmetic reasoning is within the competence of every model tier tested\. HotpotQA tells a different story: the small model inflates by4\.25×4\.25\\times, meaning the average query consumes four times more tokens than a single\-call estimate would predict\. The medium and large tiers fare better \(3\.15×\\timesand 2\.92×\\times\) but still inflate substantially\. On multi\-hop queries, the small model is both expensive \(80% of queries exceed2×2\\timesinflation\) and inaccurate \(21\.4% task accuracy\), making it a poor choice regardless of its nominal per\-token price\.

Table 1:Mean token inflation and task accuracy across model tiers and datasets\. “High Infl\.” is the fraction of queries with inflation≥2×\\geq 2\\times\.![Refer to caption](https://arxiv.org/html/2608.13571v1/x2.png)Figure 3:Inflation characterization across tiers and datasets\.Left:CDF of inflation rates on GSM8K \(Qwen2\.5\-7B in blue, GPT\-4o\-mini in orange, GPT\-4o in green\); the y\-axis is nonlinear to expose differences among the 80–100% of queries that succeed quickly\.Center:Mean inflation heatmap \(green =1×1\\times, red =≥4×\\geq 4\\times\) over the full model\-by\-dataset matrix\.Right:Retry count distribution for Qwen2\.5\-7B on GSM8K\.
### 4\.3Predicting Inflation Before Execution

We train the MLP inflation predictor on 80% of the measured data and evaluate on the remaining 20%\. Figure[4](https://arxiv.org/html/2608.13571#S4.F4)and Table[2](https://arxiv.org/html/2608.13571#S4.T2)show the results\. The predictor achieves Pearsonr=0\.714r=0\.714and AUROC 0\.887 on the held\-out split, with a negligible gap from training, indicating good generalization\. The AUROC means the predictor correctly ranks a randomly drawn high\-inflation query above a randomly drawn low\-inflation one about 89% of the time\. CBE alone achieves AUROC≈\\approx0\.81; the additional features \(chain length, prompt complexity\) contribute the remaining 8 points\.

![Refer to caption](https://arxiv.org/html/2608.13571v1/x3.png)Figure 4:Inflation predictor evaluation\.Left:Predicted vs\. actual inflation on the test set; color encodes dataset \(GSM8K vs\. HotpotQA\)\. The diagonal is perfect prediction\.Center:ROC curve for identifying high\-inflation queries \(≥2×\\geq 2\\times\), AUROC = 0\.887\.Right:CBE distribution split by high\- vs\. low\-inflation outcome, showing high\-entropy queries are substantially more likely to inflate\.Table 2:Inflation predictor performance on held\-out test split\.
### 4\.4Routing under a Fixed Token Budget

We evaluate routing policies on 500 GSM8K queries under a fixed token budget\. Results appear in Figure[5](https://arxiv.org/html/2608.13571#S4.F5)and Table[3](https://arxiv.org/html/2608.13571#S4.T3)\.InflationAgentachieves 94\.7% accuracy versus 91\.0% for FrugalGPT, a 3\.7 percentage point improvement\. To reach FrugalGPT’s 91\.0% accuracy level,InflationAgentrequires 31% fewer tokens\. Confidence Escalation \(which uses CBE but not an inflation model and does not apply fresh\-call semantics\) achieves only 88\.7%, below FrugalGPT, showing that a difficulty signal alone is not enough: routing must also account for how much retry overhead will cost, and naive escalation without discarding failed context can hurt rather than help\.

![Refer to caption](https://arxiv.org/html/2608.13571v1/x4.png)Figure 5:Routing policy comparison on GSM8K under a fixed token budget\. Bars show accuracy for each policy; the budget is held constant\.InflationAgent\(dark blue\) achieves the highest accuracy, and reaches FrugalGPT’s accuracy level using 31% fewer tokens\. Error bars are 95% bootstrap confidence intervals\.Table 3:Routing accuracy at fixed budget\. The accuracy columns are identical for1×1\\timesand1\.5×1\.5\\timesbecause all policies reach their ceiling within the1×1\\timesenvelope\.
### 4\.5Marginal Returns of Additional Compute

We ask whether simply giving a small model more retries can match a larger model’s accuracy\. Figure[6](https://arxiv.org/html/2608.13571#S4.F6)shows Qwen2\.5\-7B accuracy as the token budget grows from1×1\\timesto10×10\\times\. The model improves from 79\.0% at1×1\\timesto a peak of 92\.5% at5×5\\times, then drops back to 82\.0% at10×10\\times\. The decline at high retry counts occurs because accumulated failed chains in the context window begin to interfere with subsequent attempts\. GPT\-4o at a single clean call achieves 86\.5%, a level the small model does not reach without3×3\\timesthe budget and falls below again beyond5×5\\times\. There is a compute sweet spot for the small model; beyond it, escalation to a larger model is strictly better than continued retrying\.

![Refer to caption](https://arxiv.org/html/2608.13571v1/x5.png)Figure 6:Accuracy of Qwen2\.5\-7B \(solid blue\) as a function of token budget multiplier \(log scale\), compared to GPT\-4o\-mini \(orange dashed\) and GPT\-4o \(green dashed\) at their natural1×1\\timesbudget\. The small model peaks at5×5\\timesthen declines as failed chains accumulate in the context window\.
### 4\.6Context Contamination from Failed Chains

InflationAgentissues a fresh prompt when escalating to a larger model, discarding the failed reasoning chain\. We validate this design by running GPT\-4o on 100 hard GSM8K queries under two conditions: \(1\) a clean prompt with no prior context, and \(2\) the same prompt prepended with the full failed Qwen2\.5\-7B chain\. On the 77 queries where Qwen2\.5\-7B succeeded, the two conditions perform similarly\. On the 23 queries where it failed, contamination reduces GPT\-4o accuracy from 73\.9% to 39\.1%, a drop of 34\.8 percentage points \(95% CI: \[15\.3pp, 54\.2pp\]; McNemar’s exact testp=0\.0078p=0\.0078\)\. The effect is perfectly one\-directional: contamination never improved GPT\-4o’s answer on any query in the failure stratum\. This result directly motivatesInflationAgent’s fresh\-escalation policy: a cascade that forwards failed context to a stronger model spends large\-model budget to achieve accuracy well below what a fresh call would deliver\.

![Refer to caption](https://arxiv.org/html/2608.13571v1/x6.png)Figure 7:GPT\-4o accuracy on 100 hard GSM8K queries, split by whether Qwen2\.5\-7B succeeded \(left pair\) or failed \(right pair\)\. Blue bars: fresh prompt\. Red bars: GPT\-4o receives the full failed Qwen2\.5\-7B reasoning chain prepended\. On failure queries \(n=23n=23\), contamination drops GPT\-4o accuracy by 34\.8 pp \(from 73\.9% to 39\.1%\)\. Error bars are 95% Wilson confidence intervals\.

## 5Conclusion

We have shown that token inflation, the gap between a model’s single\-call cost estimate and its true agentic workflow cost, is large enough to invert routing decisions: a nominally cheap model inflating by4\.25×4\.25\\timeswith only 21\.4% accuracy is neither cheap nor effective\.InflationAgentaddresses this with three components: CoT Branching Entropy as a zero\-cost pre\-execution difficulty proxy, a lightweight MLP inflation predictor \(AUROC 0\.887\), and a SER\-based routing objective paired with a fresh\-escalation policy\. On GSM8K under a fixed budget,InflationAgentachieves 94\.7% accuracy versus 91\.0% for FrugalGPT using 31% fewer tokens, and the contamination experiment confirms that the fresh\-escalation design is load\-bearing: forwarding failed chains to GPT\-4o costs 34\.8 percentage points of accuracy on the queries that matter most\.

#### Limitations\.

Our evaluation covers two reasoning benchmarks and does not extend to open\-ended generation, multi\-turn dialogue, or tool\-use agents\. The MLP predictor is trained and evaluated on the same task distribution; cross\-task generalization is an open question\. Compute unit ratios are fixed and do not account for dynamic API pricing\. The contamination effect is estimated on a small stratum of 23 failure queries\.

#### Future work\.

Extending CBE to multi\-turn and tool\-use settings, where a single “chain” is less well\-defined, is a natural next step\. The SER utility function currently treats accuracy as binary; a finer\-grained notion of partial credit or downstream value would enable more nuanced routing\. Online adaptation of the inflation predictor as new task distributions are encountered would reduce the need for offline measurement campaigns\.

## References

- \[1\]L\. Chen, M\. Zaharia, and J\. Zou\(2023\)FrugalGPT: how to use large language models while reducing cost and improving performance\.InAdvances in Neural Information Processing Systems,Vol\.36\.Note:arXiv:2305\.05176Cited by:[§1](https://arxiv.org/html/2608.13571#S1.p3.1),[§2](https://arxiv.org/html/2608.13571#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2608.13571#S4.SS1.SSS0.Px2.p1.2),[Table 3](https://arxiv.org/html/2608.13571#S4.T3.8.5.3.1)\.
- \[2\]K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§4\.1](https://arxiv.org/html/2608.13571#S4.SS1.SSS0.Px1.p1.4)\.
- \[3\]D\. Dohan, W\. Xu, A\. Lewkowycz, J\. Harvey, H\. Michalewski, M\. N\. Rabe, and C\. Szegedy\(2022\)Language model cascades\.InWorkshop on Knowledge Retrieval and Language Models, ICLR,Note:arXiv:2207\.10342Cited by:[§2](https://arxiv.org/html/2608.13571#S2.SS0.SSS0.Px1.p1.1)\.
- \[4\]J\. Hu, L\. Zhuang, Q\. Xu, S\. Lu, J\. Li, S\. Savarese, C\. Xiong,et al\.\(2024\)RouterBench: a benchmark for llm router selection\.arXiv preprint arXiv:2403\.12031\.Cited by:[§1](https://arxiv.org/html/2608.13571#S1.p3.1),[§2](https://arxiv.org/html/2608.13571#S2.SS0.SSS0.Px1.p1.1)\.
- \[5\]D\. Jiang, X\. Ren, and B\. Y\. Lin\(2023\)LLM\-Blender: ensembling large language models with pairwise ranking and generative fusion\.InAssociation for Computational Linguistics,Cited by:[§2](https://arxiv.org/html/2608.13571#S2.SS0.SSS0.Px1.p1.1)\.
- \[6\]S\. Kadavath, T\. Conerly, A\. Askell, T\. Henighan, D\. Drain, E\. Perez, N\. Schiefer, Z\. Hatfield\-Dodds, N\. DasSarma, E\. Tran\-Johnson, S\. Johnston, S\. El\-Showk, A\. Jones, N\. Elhage, T\. Hume, A\. Chen, Y\. Bai, S\. Bowman, S\. Fort, D\. Ganguli, D\. Hernandez, J\. Jacobson, J\. Kernion, S\. Kravec, L\. Lovitt, N\. Nanda, C\. Olsson, S\. Ringer, D\. Amodei, T\. Brown, J\. Clark, N\. Joseph, B\. Mann, S\. McCandlish, C\. Olah, and J\. Kaplan\(2022\)Language models \(mostly\) know what they know\.arXiv preprint arXiv:2207\.05221\.Cited by:[§2](https://arxiv.org/html/2608.13571#S2.SS0.SSS0.Px2.p1.1)\.
- \[7\]L\. Kuhn, Y\. Gal, and S\. Farquhar\(2023\)Semantic uncertainty: linguistic invariances for uncertainty estimation in natural language generation\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2608.13571#S2.SS0.SSS0.Px2.p1.1)\.
- \[8\]X\. Liu, H\. Yu, H\. Zhang, Y\. Xu, X\. Lei, H\. Lai, Y\. Gu, H\. Ding, K\. Men, K\. Yang, S\. Zhang, X\. Deng, A\. Zeng, Z\. Du, C\. Zhang, S\. Shen, T\. Zhang, Y\. Su, J\. Tang, J\. Yao, B\. Hou, Y\. Zheng, J\. Zhu, Y\. Liu, Y\. Li, J\. Tang, H\. Peng, Z\. Liu, and J\. Tang\(2023\)AgentBench: evaluating LLMs as agents\.arXiv preprint arXiv:2308\.03688\.Cited by:[§1](https://arxiv.org/html/2608.13571#S1.p1.1),[§2](https://arxiv.org/html/2608.13571#S2.SS0.SSS0.Px3.p1.1)\.
- \[9\]I\. Ong, A\. Almahairi, V\. Wu, W\. Chiang, T\. Wu, J\. E\. Gonzalez, M\. W\. Kadous, and I\. Stoica\(2024\)RouteLLM: learning to route LLMs with preference data\.InAdvances in Neural Information Processing Systems,Note:arXiv:2406\.18665Cited by:[§2](https://arxiv.org/html/2608.13571#S2.SS0.SSS0.Px1.p1.1)\.
- \[10\]OpenAI\(2023\)GPT\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§4\.1](https://arxiv.org/html/2608.13571#S4.SS1.SSS0.Px1.p1.4)\.
- \[11\]Qwen Team\(2024\)Qwen2\.5 technical report\.arXiv preprint arXiv:2412\.15115\.Cited by:[§4\.1](https://arxiv.org/html/2608.13571#S4.SS1.SSS0.Px1.p1.4)\.
- \[12\]C\. E\. Shannon\(1948\)A mathematical theory of communication\.Bell System Technical Journal27,pp\. 379–423\.Cited by:[§3\.2](https://arxiv.org/html/2608.13571#S3.SS2.p1.3)\.
- \[13\]N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. Yao\(2023\)Reflexion: language agents with verbal reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§1](https://arxiv.org/html/2608.13571#S1.p1.1),[§2](https://arxiv.org/html/2608.13571#S2.SS0.SSS0.Px3.p1.1)\.
- \[14\]P\. Vsakos, D\. Andriotis, U\. Bhatt,et al\.\(2024\)Hybrid LLM: cost\-efficient and quality\-aware query routing\.arXiv preprint arXiv:2404\.14618\.Cited by:[§1](https://arxiv.org/html/2608.13571#S1.p3.1),[§2](https://arxiv.org/html/2608.13571#S2.SS0.SSS0.Px1.p1.1)\.
- \[15\]X\. Wang, J\. Wei, D\. Schuurmans, Q\. V\. Le, E\. H\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou\(2023\)Self\-consistency improves chain of thought reasoning in language models\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.13571#S1.p1.1),[§2](https://arxiv.org/html/2608.13571#S2.SS0.SSS0.Px2.p1.1)\.
- \[16\]J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. H\. Chi, Q\. V\. Le, and D\. Zhou\(2022\)Chain\-of\-thought prompting elicits reasoning in language models\.InAdvances in Neural Information Processing Systems,Vol\.35\.Cited by:[§2](https://arxiv.org/html/2608.13571#S2.SS0.SSS0.Px3.p1.1)\.
- \[17\]Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. W\. Cohen, R\. Salakhutdinov, and C\. D\. Manning\(2018\)HotpotQA: a dataset for diverse, explainable multi\-hop question answering\.InEmpirical Methods in Natural Language Processing,Cited by:[§4\.1](https://arxiv.org/html/2608.13571#S4.SS1.SSS0.Px1.p1.4)\.
- \[18\]S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao\(2023\)ReAct: synergizing reasoning and acting in language models\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.13571#S1.p1.1),[§2](https://arxiv.org/html/2608.13571#S2.SS0.SSS0.Px3.p1.1)\.