UCCI: Calibrated Uncertainty for Cost-Optimal LLM Cascade Routing
Summary
UCCI proposes a calibration-first router for LLM cascades that uses isotonic regression to map token-level margin uncertainty to error probability, achieving a 31% cost reduction on a production NER workload while maintaining micro-F1=0.91 and reducing expected calibration error from 0.12 to 0.03.
View Cached Full Text
Cached at: 05/20/26, 08:34 AM
# UCCI: Calibrated Uncertainty for Cost-Optimal LLM Cascade Routing
Source: [https://arxiv.org/abs/2605.18796](https://arxiv.org/abs/2605.18796)
[View PDF](https://arxiv.org/pdf/2605.18796)[HTML \(experimental\)](https://arxiv.org/html/2605.18796v1)
> Abstract:LLM cascades and model routing promise lower inference cost by sending easy queries to a small model and escalating hard ones to a large model, but most deployed routers use uncalibrated confidence scores and require per\-workload threshold tuning\. We present UCCI, a calibration\-first router that maps token\-level margin uncertainty to a per\-query error probability via isotonic regression and selects the escalation threshold by constrained cost minimization\. Under three explicit assumptions, threshold policies on the calibrated score are cost\-optimal, and isotonic calibration achieves O\(n^\{\-1/3\}\) sample complexity for expected calibration error \(ECE\)\. On a production named entity recognition workload of 75,000 queries served by 4B and 12B instruction\-tuned LLMs on H100 GPUs, UCCI cuts inference cost by 31% \(95% CI: \[27%, 35%\]\) at micro\-F1 = 0\.91 while reducing ECE from 0\.12 to 0\.03\. At the same operating point, UCCI beats entropy thresholding, split\-conformal routing, and a FrugalGPT\-style learned threshold\. All cascade results use end\-to\-end routing on actual model outputs and measured H100 latency, not simulated routing from global accuracies or nominal API prices\.
## Submission history
From: Varun Kotte \[[view email](https://arxiv.org/show-email/0b72375b/2605.18796)\] **\[v1\]**Mon, 11 May 2026 07:06:57 UTC \(197 KB\)Similar Articles
Faithful uncertainty in LLM agents: calibration vs utility tradeoff in practice[D]
A practitioner discusses the calibration vs. utility tradeoff in LLM agents, sharing experience with a verifier-based pipeline that reduces hallucinated tool calls by ~60% but introduces latency costs and drops easy correct answers.
Retrieval-Augmented Linguistic Calibration
This paper proposes Retrieval-Augmented Linguistic Calibration (RALC), a post-hoc pipeline for calibrating confidence signals in LLMs by modeling linguistic confidence as a distribution and using retrieval-augmented rewriting. It introduces Faithfulness Divergence metric and shows significant improvements across benchmarks.
Counterfactual Graph for Multi-Agent LLM Calibration
This paper introduces CAGE, a counterfactual graph-based method for calibrating multi-agent LLM systems, evaluating on benchmarks like TriviaQA and MMLU-Pro across various communication topologies. The method outperforms existing post-hoc and LLM-elicited calibration approaches.
The Routing Plateau: Understanding and Breaking the Accuracy Limits of LLM Routers
This paper identifies a 'routing plateau' phenomenon where diverse LLM routing methods converge to similar accuracy, far below the oracle, due to a predictability bottleneck that limits query-specific routing. It then shows that larger datasets, stronger encoders, and fine-tuning can help break through this plateau.
TRACER: Trace-Based Adaptive Cost-Efficient Routing for LLM Classification
TRACER is an open-source system that trains lightweight ML surrogates on production traces from LLM classification endpoints, routing requests through a parity gate that activates surrogates only when agreement with the original model exceeds a specified threshold. This approach achieves 83-100% surrogate coverage on intent classification benchmarks while maintaining interpretability into handling boundaries and failure modes.