Built a routing layer for multi-model pipelines, picks the right LLM per request based on priority

Reddit r/AI_Agents 05/12/26, 01:15 PM Tools

llm-routing multi-model priority-based openrouter agent-pipelines dev-tool

Summary

A routing layer that automatically selects the best LLM per request based on priority flags (speed, cost, quality, balanced) using a weighted score, with under 1ms decision time and built-in fallback, caching, and metrics.

If you're building agents that chain multiple LLM calls, you've probably hit this: not every step in your pipeline needs the same model. A quick extraction step doesn't need Opus. A final synthesis step probably shouldn't use Flash. But you still end up hardcoding something and hoping it works for all of them. This router lets you set a priority flag per request (speed / cost / quality / balanced) and it picks the best model automatically using a weighted score. Routing decision is under 1ms since it's pure math, no extra network hop. Auto-fallback if the selected model fails, Redis caching for repeated requests, metrics endpoint for p95/p99 latency per model. Built on OpenRouter, so anything in their catalogue is fair game. Would be pretty easy to wire into an agent pipeline at the LLM call layer. Github repo is in comments below 👇 Built this project using Neo AI Engineer.

Original Article

Similar Articles

Routing agent work across 4 LLM tiers: orchestrator, advisor, deep reasoning, premier

Reddit r/AI_Agents

The author shares a practical 4-tier LLM routing stack for agent work, where a fast orchestrator handles most requests and only escalates to expensive models when deep reasoning is required, significantly improving cost and interactivity.

Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving

arXiv cs.CL

Proposes a two-stage cascaded framework for cost-aware LLM serving that clusters queries and routes them to cost-effective models, then escalates low-quality outputs to stronger models. Retains 97-99% of accuracy while reducing inference cost.

Latency-Quality Routing for Functionally Equivalent Tools in LLM Agents

arXiv cs.LG

This paper introduces LQM-ContextRoute, a contextual bandit router for selecting between functionally equivalent tool providers in LLM agents, balancing latency and answer quality. It outperforms baselines on web-search and retriever benchmarks.

LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers

arXiv cs.CL

LLMRouter presents a unified formulation of LLM routing as a sequential decision process, along with an open-source infrastructure and benchmark (xRouteBench) for developing, evaluating, and deploying LLM routers. Empirical results show learned routers achieve 14.6% relative improvement over the strongest fixed-model baseline.

Beyond Accuracy and Cost: Latency-Aware LLM Query Routing for Dynamic Workloads

arXiv cs.AI

A paper proposing a latency-aware LLM query router that jointly optimizes latency, accuracy, and cost using a lightweight latency estimator, achieving up to 40% improvement in accuracy–cost utility while maintaining comparable latencies.