A tricky LLM interview question:

Your agent runs everything on a frontier LLM, so you add a routing layer that sends several easy calls to a 15x cheaper LLM.

The router is working as expected, but the bill is still the same as before.

Where did the savings go?

(answer below)

On a high-level, model routing works as follows:

• Add a classifier in front of the agent
• Send easy prompts to the cheap model
• Send hard ones to an expensive model

While this looks simple, under the hood, a lot more engineering must go into this layer before you can actually realize any savings.

And the reason is in how agents consume tokens.

A typical agent task is never one LLM call. Instead, one prompt fires several LLM calls in sequence, like planning, tool use, and analyzing results.

Each call sends the accumulated context back through the model, so the input grows every step.

To avoid redundantly processing the same tokens on every call, providers built prompt caching.

Essentially, the input a model has already seen costs about 90% less than fresh input.

The problem is that this cache is specific to a model.

So every time the router switches models mid-task, the accumulated warm cache gets thrown away, and the full context is re-billed at cold rates.

This means a cache-hot expensive model can still cost less than a cache-cold cheap model.

Production routers solve this by making the routing decision once per task instead of once per call.

The first call routes normally, and the chosen model gets pinned to a session ID. Every call after that goes to the same model, so the cache stays warm and the context stays coherent.

When the task changes, the pin resets, and routing starts fresh.

The savings don’t disappear, but rather they move one level up.

Each new task still gets routed by difficulty, so easy tasks land on a cheap model and hard ones on an expensive model.

The pin only stops switching inside a task, which is exactly where switching costs more than it saves.

This is called model affinity, and it’s the last layer of a 4-stage production routing layer that runs on every request.

1. Guardrail filter

The prompt goes through a safety check. Jailbreaks and unsafe inputs get caught before they reach the router, and every call is logged and traced from this point.

2. Router model

A small routing model reads the prompt, infers the domain and action of the request, and matches it against candidate models.

The model has to be tiny, because a routing decision that costs as much as the call it’s routing saves nothing.

3. Selection policy

A cost policy picks the winner from the candidates, preferring the cheapest or the fastest, depending on the config.

Prices are picked from pricing catalogs, so the choice adapts when providers reprice, and the runner-up stays on standby as the fallback if the winner fails.

4. Model affinity

After selection, the winning model gets pinned to the session ID. Every later call in the task goes to it, so the cache never resets and the context never splits.

To see this in practice, the exact pipeline is already implemented in Plano, an open-source proxy that runs locally between your agent and your model providers.

The config for all four layers is defined in one YAML config file, and swapping any of it never touches agent code.

The routing model in stage 2 uses Arch-Router, a 1.5B model available on HF. It’s trained on human preference data, i.e., what developers actually pick for each task type.

So routing reflects real-world preference instead of benchmark rank, and you define those preferences in plain English in the config.

It also implements an observability console with a per-request cost column, so you can see exactly which model answered each request and what it cost.

Here’s the repo: http://github.com/katanemo/plano

(don’t forget to star it )

I put this exact pipeline in front of my Hermes agent, and the bill dropped 2x, without changing a line of agent code. I wrote an article with the complete walkthrough about it.

Read it below.

katanemo/plano

Source: https://github.com/katanemo/plano

Overview

Building agentic demos is easy. Shipping agentic applications safely, reliably, and repeatably to production is hard. After the thrill of a quick hack, you end up building the “hidden middleware” to reach production: routing logic to reach the right agent, guardrail hooks for safety and moderation, evaluation and observability glue for continuous learning, and model/provider quirks scattered across frameworks and application code.

Plano solves this by moving core delivery concerns into a unified, out-of-process dataplane.

• 🚦 Orchestration: Low-latency orchestration between agents; add new agents without modifying app code.
• 🔗 Model Agility: Route by model name, alias (semantic names) or automatically via preferences.
• 🕵 Agentic Signals™: Zero-code capture of Signals plus OTEL traces/metrics across every agent.
• 🛡️ Moderation & Memory Hooks: Build jailbreak protection, add moderation policies and memory consistently via Filter Chains.

Plano pulls rote plumbing out of your framework so you can stay focused on what matters most: the core product logic of your agentic applications. Plano is backed by industry-leading LLM research and built on Envoy by its core contributors, who built critical infrastructure at scale for modern worklaods.

High-Level Network Sequence Diagram:

Jump to our docs to learn how you can use Plano to improve the speed, safety and obervability of your agentic applications.

Plano and the Plano family of LLMs (like Plano-Orchestrator) are hosted free of charge in the US-central region to give you a great first-run developer experience of Plano. To scale and run in production, you can either run these LLMs locally or contact us on Discord for API keys.

Build Agentic Apps with Plano

Plano handles orchestration, model management, and observability as modular building blocks - letting you configure only what you need (edge proxying for agentic orchestration and guardrails, or LLM routing from your services, or both together) to fit cleanly into existing architectures. Below is a simple multi-agent travel agent built with Plano that showcases all three core capabilities

📁 Full working code: See demos/agent_orchestration/travel_agents/ for complete weather and flight agents you can run locally.

1. Define Your Agents in YAML

# config.yaml
version: v0.3.0

# What you declare: Agent URLs and natural language descriptions
# What you don't write: Intent classifiers, routing logic, model fallbacks, provider adapters, or tracing instrumentation

agents:
  - id: weather_agent
    url: http://localhost:10510
  - id: flight_agent
    url: http://localhost:10520

model_providers:
  - model: openai/gpt-4o
    access_key: $OPENAI_API_KEY
    default: true
  - model: anthropic/claude-3-5-sonnet
    access_key: $ANTHROPIC_API_KEY

listeners:
  - type: agent
    name: travel_assistant
    port: 8001
    router: plano_orchestrator_v1  # Powered by our 4B-parameter routing model. You can change this to different models
    agents:
      - id: weather_agent
        description: |
          Gets real-time weather and forecasts for any city worldwide.
          Handles: "What's the weather in Paris?", "Will it rain in Tokyo?"

      - id: flight_agent
        description: |
          Searches flights between airports with live status and schedules.
          Handles: "Flights from NYC to LA", "Show me flights to Seattle"

tracing:
  random_sampling: 100  # Auto-capture traces for evaluation

2. Write Simple Agent Code

Your agents are just HTTP servers that implement the OpenAI-compatible chat completions endpoint. Use any language or framework:

# weather_agent.py
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse
from openai import AsyncOpenAI

app = FastAPI()

# Point to Plano's LLM gateway - it handles model routing for you
llm = AsyncOpenAI(base_url="http://localhost:12001/v1", api_key="EMPTY")

@app.post("/v1/chat/completions")
async def chat(request: Request):
    body = await request.json()
    messages = body.get("messages", [])
    days = 7

    # Your agent logic: fetch data, call APIs, run tools
    # See demos/agent_orchestration/travel_agents/ for the full implementation
    weather_data = await get_weather_data(request, messages, days)

    # Stream the response back through Plano
    async def generate():
        stream = await llm.chat.completions.create(
            model="openai/gpt-4o",
            messages=[{"role": "system", "content": f"Weather: {weather_data}"}, *messages],
            stream=True
        )
        async for chunk in stream:
            yield f"data: {chunk.model_dump_json()}\n\n"

    return StreamingResponse(generate(), media_type="text/event-stream")

3. Start Plano & Query Your Agents

Prerequisites: Follow the prerequisites guide to install Plano and set up your environment.

# Start Plano
planoai up config.yaml
...

# Query - Plano intelligently routes to both agents in a single conversation
curl http://localhost:8001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [
      {"role": "user", "content": "I want to travel from NYC to Paris next week. What is the weather like there, and can you find me some flights?"}
    ]
  }'
# → Plano routes to weather_agent for Paris weather ✓
# → Then routes to flight_agent for NYC → Paris flights ✓
# → Returns a complete travel plan with both weather info and flight options

4. Get Observability and Model Agility for Free

Every request is traced end-to-end with OpenTelemetry - no instrumentation code needed.

What You Didn’t Have to Build

Why it’s efficient: Plano uses purpose-built, lightweight LLMs (like our 4B-parameter orchestrator) instead of heavyweight frameworks or GPT-4 for routing - giving you production-grade routing at a fraction of the cost and latency.

Contact

To get in touch with us, please join our discord server. We actively monitor that and offer support there.

Getting Started

Ready to try Plano? Check out our comprehensive documentation:

• Quickstart Guide - Get up and running in minutes
• LLM Routing - Route by model name, alias, or intelligent preferences
• Agent Orchestration - Build multi-agent workflows
• Filter Chains - Add guardrails, moderation, and memory hooks
• Observability - Traces, metrics, and logs

Contribution

We would love feedback on our Roadmap and we welcome contributions to Plano! Whether you’re fixing bugs, adding new features, improving documentation, or creating tutorials, your help is much appreciated. Please visit our Contribution Guide for more details

Star ⭐️ the repo if you found Plano useful — new releases and updates land here first.