Tag
Ramp Router uses EWMA for failure rates and Thompson sampling for latency to select the cheapest LLM model and service tier meeting deadlines, achieving 30% cost savings without performance loss.
Introduces ContinuityBench, a benchmark and systems study for stateful failover in multi-provider LLM routing, proposing new metrics (CPR, CLO) and a history-forwarding proxy architecture achieving 99.20% context preservation.
Google DeepMind released a paper on effective model routing strategies, discussing how LLM routers are judged on accuracy and cost but can be meaningless if models respond identically.
This paper proposes correlation-aware contextual bandit algorithms that leverage surrogate reward signals from machine learning models for LLM routing, achieving improved accuracy-cost trade-offs and sample efficiency compared to standard baselines.
Explains why model routing in agent tasks may not save costs due to cache warmup, and describes a production solution with model affinity and the open-source proxy Plano to achieve actual savings.
This paper introduces ComplianceGate, a classifier-gated multi-tier routing system for LLM inference that enforces compliance in regulated industries by directing requests to appropriate model tiers.
This paper presents an automated pipeline for optimizing natural language skill descriptions in enterprise AI agents to resolve skill collisions, achieving performance matching manual tuning with a 32× speedup. Ablation studies show that a single LLM rewrite using error cases captures most improvements, while other design choices have minimal impact.
Wayfinder Router is an open-source Python tool that deterministically routes prompts to local or hosted LLMs based on structural complexity, without calling any model, enabling offline cost savings.
This paper proposes the Agent-as-a-Router framework, which transforms model routing into a dynamic, iterative process. Based on task type and real-time execution feedback, it selects the most suitable LLM to improve coding performance and cost efficiency.
This paper identifies a 'routing plateau' phenomenon where diverse LLM routing methods converge to similar accuracy, far below the oracle, due to a predictability bottleneck that limits query-specific routing. It then shows that larger datasets, stronger encoders, and fine-tuning can help break through this plateau.
This paper proposes DARS, a framework that constructs routing supervision from a distributional view of model behavior to address the unreliability of single-shot labels in LLM routing.
The author describes a setup where different AI models are assigned to specific roles (planning, coding, review) to reduce API costs for a 24/7 autonomous engineering team, and shares common failure points like model wandering and hallucinated ownership.
IR3DE is a ridge regression-based router that selects domain-expert LLMs for different tasks, achieving competitive performance while enabling dynamic addition or removal of experts without retraining.
UCCI proposes a calibration-first router for LLM cascades that uses isotonic regression to map token-level margin uncertainty to error probability, achieving a 31% cost reduction on a production NER workload while maintaining micro-F1=0.91 and reducing expected calibration error from 0.12 to 0.03.
HyDRA is a hybrid dynamic routing architecture for heterogeneous LLM pools that predicts fine-grained capability requirements per query and selects the cheapest capable model via shortfall matching, achieving up to 72.5% cost savings with quality maintained. It is deployed in GitHub Copilot's VS Code Chat auto-mode and decouples routing from model catalog, requiring no retraining when models change.
A routing layer that automatically selects the best LLM per request based on priority flags (speed, cost, quality, balanced) using a weighted score, with under 1ms decision time and built-in fallback, caching, and metrics.
This paper introduces a critique-and-routing controller for multi-agent LLM systems that formulates coordination as a sequential decision problem. It uses policy gradients to optimize the controller for iterative refinement, outperforming baselines while reducing reliance on top-tier models.
A tutorial blog post explaining LLM Routing — the practice of directing user queries to the most appropriate LLM based on cost, latency, and quality. Covers routing strategies, anatomy of an LLM router, and comparisons with Mixture of Experts.
OrcaRouter is a learning-based LLM router that dynamically routes prompts to appropriate models based on quality, cost, speed, and reliability, improving over time with production traffic.
This paper introduces RouteProfile, a design space for LLM profiles in routing systems, demonstrating that structured profiles and query-level signals improve routing performance and generalization to new models.