llm-routing

Ramp has launched Router, an AI model routing service that enables companies to access and switch between various large language models via an API, with features for cost optimization and benchmark-based routing.

This paper introduces InflationAgent, a routing system for agentic LLMs that measures token inflation, predicts task difficulty using CoT Branching Entropy, and optimizes model selection to maximize accuracy per cost, achieving higher accuracy with fewer tokens on benchmarks like GSM8K.

A new paper on arXiv introduces an open-source library called LLMRouter with over 16 router implementations and a benchmark xRouteBench, demonstrating that learned routers can outperform fixed-model baselines by 14.6%.

LLMRouter presents a unified formulation of LLM routing as a sequential decision process, along with an open-source infrastructure and benchmark (xRouteBench) for developing, evaluating, and deploying LLM routers. Empirical results show learned routers achieve 14.6% relative improvement over the strongest fixed-model baseline.

Describes Iris Ai, a system that routes queries across 8 specialized LLMs on consumer hardware, achieving large-model performance with low memory by keeping only one model active at a time and dynamic model swapping.

The article argues that model routing isn't the real problem to solve—token efficiency is. It advocates for a holistic closed-loop approach combining cheaper defaults, preference-aware routing, and better caching (e.g., Coinbase's 5%→60% cache hit improvement) to maximize useful intelligence per dollar.

Manifest explains why it deprecated its LLM router, arguing that model routing introduces unpredictability, breaks behavior consistency, and that prompt complexity cannot be inferred from the prompt alone, making caching and deliberate model selection more effective for most use cases.

A paper proposing a latency-aware LLM query router that jointly optimizes latency, accuracy, and cost using a lightweight latency estimator, achieving up to 40% improvement in accuracy–cost utility while maintaining comparable latencies.

A new paper proposes VDAR-Router, a difficulty-aware retrieval-based routing framework for LLMs that adaptively selects models based on query difficulty, achieving better cost-performance trade-offs.

Ramp Router uses EWMA for failure rates and Thompson sampling for latency to select the cheapest LLM model and service tier meeting deadlines, achieving 30% cost savings without performance loss.

Introduces ContinuityBench, a benchmark and systems study for stateful failover in multi-provider LLM routing, proposing new metrics (CPR, CLO) and a history-forwarding proxy architecture achieving 99.20% context preservation.

Google DeepMind released a paper on effective model routing strategies, discussing how LLM routers are judged on accuracy and cost but can be meaningless if models respond identically.

This paper proposes correlation-aware contextual bandit algorithms that leverage surrogate reward signals from machine learning models for LLM routing, achieving improved accuracy-cost trade-offs and sample efficiency compared to standard baselines.

Explains why model routing in agent tasks may not save costs due to cache warmup, and describes a production solution with model affinity and the open-source proxy Plano to achieve actual savings.

This paper introduces ComplianceGate, a classifier-gated multi-tier routing system for LLM inference that enforces compliance in regulated industries by directing requests to appropriate model tiers.

This paper presents an automated pipeline for optimizing natural language skill descriptions in enterprise AI agents to resolve skill collisions, achieving performance matching manual tuning with a 32× speedup. Ablation studies show that a single LLM rewrite using error cases captures most improvements, while other design choices have minimal impact.

Wayfinder Router is an open-source Python tool that deterministically routes prompts to local or hosted LLMs based on structural complexity, without calling any model, enabling offline cost savings.

This paper proposes the Agent-as-a-Router framework, which transforms model routing into a dynamic, iterative process. Based on task type and real-time execution feedback, it selects the most suitable LLM to improve coding performance and cost efficiency.

This paper identifies a 'routing plateau' phenomenon where diverse LLM routing methods converge to similar accuracy, far below the oracle, due to a predictability bottleneck that limits query-specific routing. It then shows that larger datasets, stronger encoders, and fine-tuning can help break through this plateau.

This paper proposes DARS, a framework that constructs routing supervision from a distributional view of model behavior to address the unreliability of single-shot labels in LLM routing.