A learned LLM router scored 0.84 AUC. Shuffling the labels within each task still scored 0.838 [R]

Reddit r/MachineLearning 09/27/26, 01:35 AM Papers

Summary

A study revealed that an LLM router trained to select between models learned task recognition instead of difficulty, causing poor generalization on held-out tasks, but deferral based on the cheap model's output yielded better performance.

I trained a router to choose between a cheap and an expensive model. It scored 0.84 AUC held out. Then I shuffled the labels within each task — preserving each task's escalation rate, destroying all per-item signal — retrained, and got 0.838. It had learned to recognize the task, not the difficulty. Eliminated in turn: too few labels (110k from RouteLLM's released set) the architecture (linear probe, similarity-weighted ranking, fine-tuned encoder) the representation (a probe recovers human difficulty from the same embeddings) label noise (test-retest kappa 0.88–0.97; the labels support AUC 0.91) On held-out tasks every router falls to ~0.55 and loses to prompt length. What does work: deferral on the cheap model's own output, 0.75 vs 0.60 on the same split, at no extra inference cost. https://brianfeeny.com/posts/replicating-routellm-on-amazon-bedrock/?utm_source=reddit&utm_medium=social&utm_campaign=routing-paper-2026-09 Curious whether anyone has run the within-group shuffle on their own router — my ordering is an engineering assessment, not a benchmark, and I would be glad to be corrected.

Original Article

Similar Articles

@LiorOnAI: A ~10K parameter router can beat every individual open model on MMLU by learning which model should answer which questi…

X AI KOLs Following

A tiny ~10K parameter router called tinyrouter learns which open model to use per question on MMLU, outperforming individual models by optimizing allocation.

LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers

arXiv cs.CL

LLMRouter presents a unified formulation of LLM routing as a sequential decision process, along with an open-source infrastructure and benchmark (xRouteBench) for developing, evaluating, and deploying LLM routers. Empirical results show learned routers achieve 14.6% relative improvement over the strongest fixed-model baseline.

LLM-as-judge anchored on one confidence value in 10 of 16 evals. Asking for a label fixed it.

Reddit r/AI_Agents

An LLM judge consistently returned a fixed confidence score of 0.72 in evaluations, but switching to categorical labels improved score distribution, showing that models are better at classification than numerical estimation for assessments.

Task- and Session-Level Model Routing: A Common-Interface Hybrid Evaluation of Four Open-Source Routers Across Four Benchmarks

arXiv cs.AI

This paper evaluates four open-source LLM routers using a common protocol across four benchmarks, finding that performance gains are more closely tied to model tier composition than task-specific targeting.

@omarsar0: Great paper from DeepMind on effective model routing strategies.