The article argues that model routing isn't the real problem to solve—token efficiency is. It advocates for a holistic closed-loop approach combining cheaper defaults, preference-aware routing, and better caching (e.g., Coinbase's 5%→60% cache hit improvement) to maximize useful intelligence per dollar.
I've seen over 10+ model routers get shipped in the market as of late, from Ramp, martian(again), Coinbase, Devin, cursor etc. And everyone has got this wrong (sort of). Model routing isn't the problem to solve, the real problem is token efficiency. How to measure it as a closed loop system and find the best ways to use the right model for the right scenario taking into account preferences, caching, and output tokens needed for a task Benchmarks are maps, not routing tables First off, I think benchmark-based routers are completely broken. Benchmarks let us compare models under controlled, repeatable conditions. They help narrow a large model catalog, identify broad strengths, and bootstrap routing before an application has enough real-world traffic to evaluate. That makes them a valuable starting point for DigitalOcean presets. But model performance is conditional on the application around it: the system prompt, tool definitions, context, output constraints, conversation history, and definition of success. Change the harness and the relative ranking of models can change with it. A model that performs best on an isolated coding benchmark may not perform best inside a coding agent operating across a large repository with dozens of tools and a long conversation history. Also, one developer may prefer a particular model’s visual style for image generation. Another may prioritize instruction following, tool-call reliability, latency, or cost. Neither preference can be inferred from a general-purpose leaderboard. Preselecting a model from benchmark scores alone is therefore not intelligent routing. Routing must first understand what the developer is optimizing for. Over time, it becomes a personalization problem.But even a preference-aware router can make the wrong economic decision if it evaluates every request in isolation. Better defaults, better routing, and better caching The industry’s first response to rapidly growing inference bills has often been to ration access. But 91% of Coinbase employees were not reaching their existing usage caps. Lowering those caps would have generated more alerts and more friction without addressing most of the spend. Coinbase instead moved toward cheaper defaults, task-aware routing, and better caching—and reported improving LibreChat’s cache hit rate from 5% to 60%. These three controls reinforce one another: Better defaults prevent every request from beginning on the most expensive model. Preference-aware routing selects models according to the task and what the developer values. Cache-aware routing preserves the accumulated economic value of an agentic session instead of discarding it between turns. No one technique is sufficient on its own. A cheap default may not meet the quality bar for a complex task. A benchmark-driven router may not reflect an application’s real evaluations. A cache-aware system should not preserve a warm model when it is no longer appropriate for the work. The objective is not to maximize tokens or blindly minimize their price. It is to maximize useful intelligence per dollar while preserving the quality, latency, and reliability each application requires. The last piece is learning through evaluations on token efficiency for scenarios and simulating runs with different models to create the best router that is on auto. But that requires data, and a clear understanding of what "success" looks like. everyone looking for a quick win doesn't want to spend all that effort there. No one is doing this holistically. That's what I would pay money for.
This article argues that common LLM cost advice focusing on token reduction is too shallow, and that the more impactful strategy in production is to route different workflow steps to different models rather than using a single default model.
A tutorial blog post explaining LLM Routing — the practice of directing user queries to the most appropriate LLM based on cost, latency, and quality. Covers routing strategies, anatomy of an LLM router, and comparisons with Mixture of Experts.
Ramp Router uses EWMA for failure rates and Thompson sampling for latency to select the cheapest LLM model and service tier meeting deadlines, achieving 30% cost savings without performance loss.
LLM routers are evolving from a niche infrastructure trick into a mainstream service category, enabling users to automatically select the most cost-effective model for each request as frontier model costs rise.
Explains why model routing in agent tasks may not save costs due to cache warmup, and describes a production solution with model affinity and the open-source proxy Plano to achieve actual savings.