routing

Kimi K3, an open model, rivals Fable in agentic tasks, and routing between them achieves 93% accuracy with up to 50x cost savings, establishing a new state-of-the-art approach.

This paper proposes Multi-level Context Fusion MOE (MCF-MOE), a framework that improves routing consistency in Mixture-of-Experts models by integrating cross-layer semantic aggregation and local token-level interactions, outperforming strong baselines on language modeling and understanding benchmarks.

Model routing is a hot trend to reduce inference costs, but the best routing is deeply task-specific. Teams like Harvey and Factory achieve significant cost savings by focusing on single workflows rather than generic routers.

A developer replaced Twilio with Sent, a unified messaging API that auto-routes messages via SMS, WhatsApp, or RCS based on cost and engagement, reducing messaging bills by 40%.

A.L.F.R.E.D. proposes a system that distills knowledge from large models into small ones and routes simple tasks to the small models, achieving performance of 35B models with 2B models while reducing inference cost by 4x.

This paper investigates how language models separate a character's belief from reality, finding that they use a shared value slot for attributed values and a router at the query position to select the frame (belief or reality) to read out. It identifies two routes for asserted and derived beliefs, and shows that the slot itself carries no belief-reality tag; the separation lies in dissociated routing subspaces.

A developer replaced Twilio with a single API that routes messages to SMS, WhatsApp, or RCS based on user response, cutting messaging costs by 40% in the first week.

This paper proposes an explicit multimodal routing framework for clinical prediction using EHR data, enabling interpretable, robust, and auditable reasoning across structured variables, clinical notes, and chest X-rays via discrete unimodal, bimodal, and trimodal routes with inference-time route masking for missing modality simulation.

StickyMoE proposes a differentiable routing consistency loss that encourages adjacent tokens to activate the same experts in MoE models, reducing expert-swapping overhead and cache misses during inference on edge devices by up to 3.92× while improving perplexity.

Google Research published a study in Nature Cities showing that coordinating a small fraction of trips via navigation app interventions can measurably reduce traffic congestion and emissions across entire cities.

Plano is an open-source proxy that sits between AI agents and LLM providers to cut costs through intelligent routing, guardrail filtering, and cost-aware selection, all configured via a single YAML file without modifying agent code.

OpenSquilla releases v0.5.0, achieving performance surpassing Fable5 and reducing costs by 67% through intelligent routing and multi-model integration; also adds MetaSkills and Coding mode features.

LlamaIndex announces improved routing in LlamaParse that selects lightweight parsing for simple text-heavy PDFs and heavier VLM-based parsing for complex pages with tables or charts, optimizing cost and accuracy.

The author presents a method to make LLMs verbalize calibrated confidence by using a linear probe on mid-layer states and a small trained bridge to confidence logits, requiring only 200 labeled examples and no weight modification. This is linked to Anthropic's global workspace paper explaining the know-say gap.

This paper proposes a multi-objective optimization framework for routing in LLM serving, employing online linear programming with bid-price control to balance latency, throughput, and tail performance, and demonstrates improvements over heuristics using the Vidur simulator.

The author shares findings from Hermes Mixture-of-Agents experiments, including voter upgrades, GPU topology, and caching economics, showing that local prefix caching can make long agent sessions nearly free and that two independent GPU instances outperform a single partitioned one.

SupraLabs releases Supra-Router-51M, a 51.7M parameter micro-LLM for multi-task infrastructure routing, designed to decide whether to process prompts locally on edge or send them to cloud-hosted models. Fine-tuned on a small dataset, it uses multi-task sequence generation for robust routing.

A new AI research paper describes a tiny model that acts as a manager to route tasks to larger models, outperforming frontier models like ChatGPT, Gemini, and Claude on a hard coding benchmark by orchestrating a team of models instead of relying on a single one.

本文研究垂直联邦学习中的选择性升级问题，提出一种基于期望增益的评分方法，在低成本的本地预测和高成本的嵌入融合之间进行路由，以优化通信-准确率权衡。

Clement Delangue highlights vLLM's new semantic router, an open-source system for routing LLM queries to the most appropriate model, aiming to shift value from expensive frontier models to a diverse ecosystem of open-source models.