model-routing

The article argues that many AI agent workflows waste money by routing every task to frontier models, and suggests using cheaper model tiers for simple, structured tasks while escalating harder ones. It provides a cost comparison showing up to 75% savings with a tiered approach.

MERA introduces a multi-cycle adaptation approach for LLM agents that improves small models by distilling execution-verified demonstrations into a SkillBook and fine-tuning LoRA adapters, with router-based deployment and verifier-backed fallback. Experiments show Qwen2.5-Coder-1.5B improves from 28.7% to 49.7% pass on HumanEval++MBPP while retaining most quality at lower cost.

Nvidia released Nemotron 3.5 Lightning, a 30B open mixture-of-experts model, and NeMo Switchyard, an open-source routing library that dynamically assigns each step of an AI agent workflow to the most suitable model. Nvidia claims the combination can cut agent task costs to about a third while maintaining frontier-level performance.

LangChain tested NVIDIA's open-source router Switchyard with Deep Agents, showing that routing between models can cut costs by ~70% while retaining ~90% accuracy, and introduced new middleware integration for NVIDIA models.

This paper demonstrates that replay-based static evaluation of model switching in LLM agents is fundamentally flawed: when swapping models mid-trajectory, the environment and subsequent actions diverge dramatically from logged trajectories, invalidating most benchmark results. The authors propose branching rollouts as a more faithful evaluation method and release their harness and trajectories.

LangChain's Max Agency podcast features Cognition president Russell Kaplan discussing why developers now prioritize efficiency over the best model, and how agents like Devin are being deployed at scale.

LLMRouter presents a unified formulation of LLM routing as a sequential decision process, along with an open-source infrastructure and benchmark (xRouteBench) for developing, evaluating, and deploying LLM routers. Empirical results show learned routers achieve 14.6% relative improvement over the strongest fixed-model baseline.

Cursor explains how its Router uses a data-driven approach with the Compass complexity predictor to route developer tasks to the most appropriate AI model, achieving significant cost reductions while maintaining or improving user satisfaction.

The author argues that AI coding harnesses and model routing are as important as the model itself, sharing tests with Oh-My-Pi and OpenCode that cut token usage and errors, and recommending tiered model subscriptions for high-volume lightweight tasks.

The article argues that the biggest AI shift is not larger models but better systems around them, such as context, model routing, caching, agent workflows, and evaluation, making the model the engine and the system the product.

Google Cloud API Gateway now offers model routing in Public Preview, providing a serverless ingress layer that accepts OpenAI-compatible requests and dynamically routes them to Gemini, Claude, or OpenAI models.

Not Diamond announces Not Diamond Code, an intelligent model router for coding agents that selects the best model and reasoning effort per step, claiming 20%+ cost savings and Pareto-optimal performance on coding benchmarks.

Not Diamond Code is announced as an intelligent model router for long-horizon coding agents, selecting the best model and reasoning effort per step to reduce costs by 20-65%.

Google Cloud API Gateway announces public preview of model routing, letting developers access Gemini, Claude, and OSS models via a single endpoint using OpenAPI specs.

The article discusses how most AI traffic consists of simple, repeatable tasks like classification and extraction, yet frontier models are often used for everything. It questions whether routing tasks to smaller specialized models will become standard practice to reduce cost and latency.

The article argues that model routing isn't the real problem to solve—token efficiency is. It advocates for a holistic closed-loop approach combining cheaper defaults, preference-aware routing, and better caching (e.g., Coinbase's 5%→60% cache hit improvement) to maximize useful intelligence per dollar.

Compares API list prices of 18 LLMs from major providers, highlighting a 100x cost difference for the same workload and recommending model routing for cost efficiency.

LLM routers are evolving from a niche infrastructure trick into a mainstream service category, enabling users to automatically select the most cost-effective model for each request as frontier model costs rise.

A team benchmarked routing different stages of an AI agent workflow to different models versus sending every request to Claude Opus 5 across 89 Terminal-Bench 2.1 tasks, and found surprising results.

Tokenless is a YC-backed startup that offers a drop-in replacement for API calls to LLMs, automatically routing requests to the most cost-effective model without sacrificing quality, potentially cutting inference costs in half.