model-routing

Ollama added support for Jev-like decision models such as Nimble, running entirely locally for tasks like ticket triaging, model routing, and content moderation via a new local /v1/systemone API.

The paper proposes EDAR, an entropy-driven adaptive routing framework that decides at inference time whether to answer a query with retrieval-augmented generation or escalate it to full long-context processing. EDAR retains 97.4% of the pure long-context baseline's accuracy while cutting token cost by 70.7%, escalating only 18.2% of queries on LongBench v2 and Infinity-Bench.

Devin's latest update makes the AI software engineering agent 30-70% cheaper depending on mode, achieved by integrating the latest models like SWE-2 and optimizing its harness through batched tool calls and improved prompt caching, while Devin Fusion now leads FrontierCode 1.1 Extended at $0.60 per task.

Replit shares its 'free the models' harness design, where the core agent loop — not an external router — chooses subagent tiers, effort, and delegation as tasks unfold. Using GPT-6 Astra, the approach reportedly beats router-based baselines and a sidekick architecture on DeepSWE and Terminal-Bench while keeping quality at minimum cost.

This tweet discusses Stripe's acquisition of OpenRouter, emphasizing the role of model routing and the token economy in AI infrastructure.

In an LLM Gateway benchmark pilot, Jev-based model routing achieved 33.2% cost savings versus a premium model, but a fixed mid-priced model offered better value with comparable task performance.

Latent Space podcast episode with OpenRouter co-founder Alex Atallah and AMP's Anjney Midha explaining why Stripe acquired OpenRouter, how it became a neutral multi-model routing layer handling 10T+ tokens per day, and why agentic fraud may define the security of the emerging token economy.

The author suggests that heavy AI users may inefficiently allocate model capacity by not optimizing model selection, proposing that work should be routed to the least expensive capable model to save costs and improve efficiency.

The article explains how to integrate the Jev AI model into an agent orchestration framework for intelligent model routing, risk interception in auto mode, and automated evaluation, referencing a LangChain video for further details.

AgentRouter is a lightweight classifier for routing steps in agentic workflows to different model tiers, achieving 72% cost reduction with minimal quality degradation compared to frontier-only models.

Using smaller, cheaper LLMs can increase total workflow costs due to hidden expenses like review time and error correction, highlighting the need for comprehensive cost tracking.

The author argues that Jev and similar classifiers are interesting not for their novelty but for highlighting how LLMs are often used for simple decision-making tasks, suggesting a more efficient architecture with smaller models for routing and evaluation.

This paper evaluates entropy-based confidence signals for small language models, finding that semantic entropy provides a viable method to improve accuracy by up to 50 percentage points through selective routing to larger models.

The author compared TypeSafe AI's Jev model with a regular LLM for model routing and found that Jev significantly reduces latency to around 1 second versus 4-14 seconds, making it promising for fast decision layers.

The article introduces Jev, a decision model from TypeSafe, integrated with Hermes to handle small, non-thinking decisions for AI agents quickly and cheaply, optimizing costs and performance.

Pace details their approach to improving an AI extraction agent for insurance documents by optimizing subagent delegation, model routing, and context based on historical data, leading to reduced cost and latency.

CrofAI, an AI inference provider claiming cheap tokens, was exposed as an OpenRouter wrapper that routed requests to cheaper models at a high markup, leading to wire fraud allegations and the company's sudden shutdown.

Litelm is a lightweight Python library that provides core LLM routing and message translation features extracted from LiteLLM, designed with minimal dependencies for simplicity and efficiency.

The author is revisiting an agent pipeline after a model cost change, asking for signals to decide which stages to route to the strongest model path for optimal performance and cost.

This article covers the GitHub Copilot Day event, featuring Hydra Fusion's smart model routing, cross-platform integration, and the new AI-driven development paradigm from concept to production.