Online Learning for Cost-Efficient LLM Routing (6 minute read)

TLDR AI 07/21/26, 12:00 AM Tools

online-learning llm-routing cost-optimization ewma thompson-sampling ramp

Summary

Ramp Router uses EWMA for failure rates and Thompson sampling for latency to select the cheapest LLM model and service tier meeting deadlines, achieving 30% cost savings without performance loss.

Ramp Router learns provider failure rates through EWMA and latency distributions through Thompson sampling, then chooses the cheapest model and service tier likely to meet each deadline. Ramp reports 30% savings in Ramp Inspect without performance loss.

Original Article

Similar Articles

Ramp Router claims to cut AI costs by up to 30%

Reddit r/ArtificialInteligence

Ramp is open-sourcing its internal LLM router that automatically selects the best model for each request to optimize cost and performance.

@amitiitbhu: New article: LLM Routing Read here: https://outcomeschool.com/blog/llm-routing…

X AI KOLs Timeline

A tutorial blog post explaining LLM Routing — the practice of directing user queries to the most appropriate LLM based on cost, latency, and quality. Covers routing strategies, anatomy of an LLM router, and comparisons with Mixture of Experts.

Beyond Accuracy and Cost: Latency-Aware LLM Query Routing for Dynamic Workloads

arXiv cs.AI

A paper proposing a latency-aware LLM query router that jointly optimizes latency, accuracy, and cost using a lightweight latency estimator, achieving up to 40% improvement in accuracy–cost utility while maintaining comparable latencies.

Online Linear Programming for Multi-Objective Routing in LLM Serving

arXiv cs.AI

This paper proposes a multi-objective optimization framework for routing in LLM serving, employing online linear programming with bid-price control to balance latency, throughput, and tail performance, and demonstrates improvements over heuristics using the Vidur simulator.

The Routing Plateau: Understanding and Breaking the Accuracy Limits of LLM Routers

arXiv cs.LG

This paper identifies a 'routing plateau' phenomenon where diverse LLM routing methods converge to similar accuracy, far below the oracle, due to a predictability bottleneck that limits query-specific routing. It then shows that larger datasets, stronger encoders, and fine-tuning can help break through this plateau.