routing

This preprint paper proposes a customizable router for AI coding agents in enterprises to optimize costs by intelligently routing requests, saving 14-21% of model spend annually for large companies.

This paper presents a controlled study on learned context planning for long-context multiple-choice QA, demonstrating that it does not robustly outperform strong retrieval methods like BM25 and hybrid retrieval under various experimental settings.

After analyzing over 600 builds using Jev, the author finds that the most interesting aspect is not flashy demos but the cost-effective automation of routine tasks, enabling large-scale economical AI applications.

This paper introduces Attention-Aware Routing (AAR), a method that enhances Mixture-of-Experts language models by incorporating attention weights into the router, improving mathematical reasoning performance and revealing coupled dynamics between routing and attention.

The article tests an AI model called Jev from TypeSafe AI, designed for fast, structured decisions rather than generation, aiming to improve efficiency in AI workflows by separating decision-making from general-purpose LLM tasks.

Introduces SWRouter, a similarity-contractive window router for multi-turn large language model conversations, addressing challenges in context segmentation and routing evaluation.

A technical explanation of how LLM inference requests are routed when they hit a GPU cluster in production environments.

R2Adapter is a lightweight routing and rewriting adapter that dynamically allocates queries between vanilla and graph-based RAG, reducing graph-based RAG usage by up to 59% while maintaining accuracy in multi-hop QA benchmarks.

NVIDIA PAIR is a beta tool that enables users to form a personal AI inference cluster by connecting local devices like DGX Spark, RTX systems, and Macs for efficient, private workflow routing.

The paper proposes Gated-Memory Routing, a framework for multi-agent LLM systems that uses learned gates to manage memory, improving accuracy and reducing inference costs on reasoning and code-generation benchmarks.

OpenAI's prompt cache reduces request costs by 90% but has a cache key limit of 15 requests per second. Unify GTM built a custom routing solution to bypass this limit, achieving a 95% cache hit rate.

The paper introduces GATNextHop, a Graph Attention Network model that approximates shortest path routing and generalizes across network topologies, evaluated against Dijkstra's algorithm on real-world ISP networks.

This paper introduces a paired exact-reset evaluation protocol to determine when switching from a medium to a full world-model predictor improves task-specific decision loss, with experiments in PushT and PyBullet showing incremental routing benefits.

Red Hat's AI blog series part 3 details how llm-d routes model inference traffic on Amazon EKS within their managed Kubernetes platform, using Kubernetes resources and Envoy integration for real-time decisions.

This paper investigates how transformers organize causal knowledge, showing that type-level supervision induces a typed routing structure that is functionally decoupled from answer readout, with exact local editability and bit-exact revertibility.

This paper studies how 4-bit KV-cache quantization in Mixture-of-Experts models causes route flips, and finds that detecting a flip is possible but predicting whether the flip is harmful is not, proposing a causal measurement apparatus rather than a mitigation.

This paper presents a cost-efficient routing pipeline for multilingual short-text classification that selectively translates low-resource languages into English before zero-shot classification using small language models, showing quality gains on SIB-200 and MASSIVE benchmarks.

This paper proposes UniF-MoE, a unified framework for token-adaptive Mixture-of-Experts computation that first shares reusable computation across experts and then routes the remaining residual demand, improving performance while reducing activated computation, latency, and memory on DomainBed and GLUE benchmarks.

A technical survey of Mixture-of-Experts architectures in LLMs, organizing evolution along expert granularity, topology, routing, load balancing, and execution, and proposing complementary views of architectural milestones and control planes.

This paper investigates whether document extraction difficulty can be predicted before inference using low-cost features, allowing a router to select between a cheap and a strong model. Routing cut costs by up to 77% in some genres while maintaining quality, but only when difficulty varies and is predictable from document features.