UCCI: Calibrated Uncertainty for Cost-Optimal LLM Cascade Routing
Summary
UCCI proposes a calibration-first router for LLM cascades that uses isotonic regression to map token-level margin uncertainty to error probability, achieving a 31% cost reduction on a production NER workload while maintaining micro-F1=0.91 and reducing expected calibration error from 0.12 to 0.03.
View Cached Full Text
Cached at: 05/20/26, 08:34 AM
# UCCI: Calibrated Uncertainty for Cost-Optimal LLM Cascade Routing
Source: [https://arxiv.org/abs/2605.18796](https://arxiv.org/abs/2605.18796)
[View PDF](https://arxiv.org/pdf/2605.18796)[HTML \(experimental\)](https://arxiv.org/html/2605.18796v1)
> Abstract:LLM cascades and model routing promise lower inference cost by sending easy queries to a small model and escalating hard ones to a large model, but most deployed routers use uncalibrated confidence scores and require per\-workload threshold tuning\. We present UCCI, a calibration\-first router that maps token\-level margin uncertainty to a per\-query error probability via isotonic regression and selects the escalation threshold by constrained cost minimization\. Under three explicit assumptions, threshold policies on the calibrated score are cost\-optimal, and isotonic calibration achieves O\(n^\{\-1/3\}\) sample complexity for expected calibration error \(ECE\)\. On a production named entity recognition workload of 75,000 queries served by 4B and 12B instruction\-tuned LLMs on H100 GPUs, UCCI cuts inference cost by 31% \(95% CI: \[27%, 35%\]\) at micro\-F1 = 0\.91 while reducing ECE from 0\.12 to 0\.03\. At the same operating point, UCCI beats entropy thresholding, split\-conformal routing, and a FrugalGPT\-style learned threshold\. All cascade results use end\-to\-end routing on actual model outputs and measured H100 latency, not simulated routing from global accuracies or nominal API prices\.
## Submission history
From: Varun Kotte \[[view email](https://arxiv.org/show-email/0b72375b/2605.18796)\] **\[v1\]**Mon, 11 May 2026 07:06:57 UTC \(197 KB\)Similar Articles
Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving
Proposes a two-stage cascaded framework for cost-aware LLM serving that clusters queries and routes them to cost-effective models, then escalates low-quality outputs to stronger models. Retains 97-99% of accuracy while reducing inference cost.
Beyond Accuracy and Cost: Latency-Aware LLM Query Routing for Dynamic Workloads
A paper proposing a latency-aware LLM query router that jointly optimizes latency, accuracy, and cost using a lightweight latency estimator, achieving up to 40% improvement in accuracy–cost utility while maintaining comparable latencies.
Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling
The paper proposes C3RL, a reinforcement learning algorithm that calibrates LLM confidence while maintaining accuracy, and CAS, a confidence-based adaptive test-time scaling strategy that reduces inference costs by up to 12.33 times.
Online Learning for Cost-Efficient LLM Routing (6 minute read)
Ramp Router uses EWMA for failure rates and Thompson sampling for latency to select the cheapest LLM model and service tier meeting deadlines, achieving 30% cost savings without performance loss.
Faithful uncertainty in LLM agents: calibration vs utility tradeoff in practice[D]
A practitioner discusses the calibration vs. utility tradeoff in LLM agents, sharing experience with a verifier-based pipeline that reduces hallucinated tool calls by ~60% but introduces latency costs and drops easy correct answers.